2024-02-01T10:00:19.885 INFO:root:teuthology version: 0.0.1.dev253+g6128cc3 2024-02-01T10:00:19.885 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_yuvalif@teuthology --description rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zones supported-random-distro$/{ubuntu_latest} tasks/test_multi} --verbose --archive /home/teuthworker/archive/yuvalif-2024-01-30_08:46:48-rgw-wip-end2end-tracing-distro-default-smithi/7537587 --name yuvalif-2024-01-30_08:46:48-rgw-wip-end2end-tracing-distro-default-smithi -- /home/teuthworker/archive/yuvalif-2024-01-30_08:46:48-rgw-wip-end2end-tracing-distro-default-smithi/7537587/orig.config.yaml 2024-02-01T10:00:19.933 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-01T10:00:20.024 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuvalif-2024-01-30_08:46:48-rgw-wip-end2end-tracing-distro-default-smithi/7537587 branch: wip-end2end-tracing description: rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zones supported-random-distro$/{ubuntu_latest} tasks/test_multi} email: null first_in_suite: false job_id: '7537587' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: yuvalif-2024-01-30_08:46:48-rgw-wip-end2end-tracing-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-end2end-tracing ceph: conf: client: debug rgw: 20 debug rgw sync: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= rgw curl low speed time: 300 rgw data log num shards: 4 rgw md log max shards: 4 rgw sync data inject err probability: 0.1 rgw sync log trim interval: 0 rgw sync meta inject err probability: 0.1 rgw sync obj etag verify: true setgroup: ceph setuser: ceph client.0: rgw data notify interval msec: 0 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 rocksdb delete range threshold: 0 osd.0: osd_max_omap_entries_per_request: 10 osd.1: osd_max_omap_entries_per_request: 1000 osd.2: osd_max_omap_entries_per_request: 10000 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) - \(PG_DEGRADED\) - \(POOL_APP_NOT_ENABLED\) - not have an application enabled sha1: 5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24 rgw: compression type: random frontend: beast rgw-multisite: realm: is default: true name: test-realm zonegroups: - enabled_features: - resharding endpoints: - c1.client.0 is_default: true is_master: true name: test-zonegroup zones: - endpoints: - c1.client.0 is_default: true is_master: true name: test-zone1 - endpoints: - c2.client.0 is_default: true name: test-zone2 - endpoints: - c1.client.1 name: test-zone3 rgw-multisite-tests: args: - tests.py workunit: branch: wip-end2end-tracing sha1: 5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24 owner: scheduled_yuvalif@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - c1.mon.a - c1.mgr.x - c1.osd.0 - c1.osd.1 - c1.osd.2 - c1.client.0 - c1.client.1 - - c2.mon.a - c2.mgr.x - c2.osd.0 - c2.osd.1 - c2.osd.2 - c2.client.0 - c2.client.1 seed: 8557 sha1: 5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24 sleep_before_teardown: 0 subset: 1/3 suite: rgw suite_branch: wip-end2end-tracing suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24 targets: smithi148.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBH5ce3/FCp+PGNOXXdX8il8f4jkJaHpplftAcDQSFX3tHjR+D85kVrtsDWCNIGnihkTWiK/q2ovxDNYhkN4nd0g= smithi187.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLmrn1ce5MSq4PQkPQy/SrK5OHVanVF/Lx8xwqrWintcWDpX5+8boxWTRV5B7sthtEqb55YGOwOBChsT+yODK3Y= tasks: - install: null - ceph: cluster: c1 - ceph: cluster: c2 - rgw: c1.client.0: port: 8000 c1.client.1: port: 8001 c2.client.0: port: 8000 c2.client.1: port: 8001 - rgw-multisite: null - rgw-multisite-tests: config: reconfigure_delay: 60 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6128cc3ecb49b7d62475e3595041c19b5326ca7c timestamp: 2024-01-30_08:46:48 tube: smithi user: yuvalif verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-02-01T10:00:20.024 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa; will attempt to use it 2024-02-01T10:00:20.024 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/tasks 2024-02-01T10:00:20.025 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-02-01T10:00:20.026 INFO:teuthology.task.internal:Checking packages... 2024-02-01T10:00:20.051 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24' 2024-02-01T10:00:20.051 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-01T10:00:20.051 INFO:teuthology.packaging:ref: None 2024-02-01T10:00:20.051 INFO:teuthology.packaging:tag: None 2024-02-01T10:00:20.051 INFO:teuthology.packaging:branch: wip-end2end-tracing 2024-02-01T10:00:20.051 INFO:teuthology.packaging:sha1: 5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24 2024-02-01T10:00:20.051 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-end2end-tracing 2024-02-01T10:00:20.199 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-931-g5a7faa11-1jammy 2024-02-01T10:00:20.201 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-02-01T10:00:20.212 INFO:teuthology.task.internal:no buildpackages task found 2024-02-01T10:00:20.212 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-02-01T10:00:20.235 INFO:teuthology.task.internal:Saving configuration 2024-02-01T10:00:20.249 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-02-01T10:00:20.257 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-01T10:00:20.283 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi148.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuvalif-2024-01-30_08:46:48-rgw-wip-end2end-tracing-distro-default-smithi/7537587', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-01 09:54:39.089625', 'locked_by': 'scheduled_yuvalif@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIh7WFzx1evOVEDNmn5Uc9Zbsz7SeS19Ntj4wYwg6LQc'} 2024-02-01T10:00:20.308 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi187.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuvalif-2024-01-30_08:46:48-rgw-wip-end2end-tracing-distro-default-smithi/7537587', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-01 09:54:39.088225', 'locked_by': 'scheduled_yuvalif@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAII1fPod+M2YGpsLFrdtiJzeSDYFdhhfLTdLK15oSICTW'} 2024-02-01T10:00:20.308 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-02-01T10:00:20.316 INFO:teuthology.task.internal:roles: ubuntu@smithi148.front.sepia.ceph.com - ['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'] 2024-02-01T10:00:20.317 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1'] 2024-02-01T10:00:20.317 INFO:teuthology.run_tasks:Running task console_log... 2024-02-01T10:00:20.496 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fc7ab5a21f0>, signals=[15]) 2024-02-01T10:00:20.497 INFO:teuthology.run_tasks:Running task internal.connect... 2024-02-01T10:00:20.505 INFO:teuthology.task.internal:Opening connections... 2024-02-01T10:00:20.506 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi148.front.sepia.ceph.com 2024-02-01T10:00:20.507 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-01T10:00:20.637 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi187.front.sepia.ceph.com 2024-02-01T10:00:20.638 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-01T10:00:20.725 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-02-01T10:00:20.734 DEBUG:teuthology.orchestra.run.smithi148:> uname -m 2024-02-01T10:00:20.740 INFO:teuthology.orchestra.run.smithi148.stdout:x86_64 2024-02-01T10:00:20.740 DEBUG:teuthology.orchestra.run.smithi148:> cat /etc/os-release 2024-02-01T10:00:20.786 INFO:teuthology.orchestra.run.smithi148.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-02-01T10:00:20.786 INFO:teuthology.orchestra.run.smithi148.stdout:NAME="Ubuntu" 2024-02-01T10:00:20.786 INFO:teuthology.orchestra.run.smithi148.stdout:VERSION_ID="22.04" 2024-02-01T10:00:20.787 INFO:teuthology.orchestra.run.smithi148.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-02-01T10:00:20.787 INFO:teuthology.orchestra.run.smithi148.stdout:VERSION_CODENAME=jammy 2024-02-01T10:00:20.787 INFO:teuthology.orchestra.run.smithi148.stdout:ID=ubuntu 2024-02-01T10:00:20.787 INFO:teuthology.orchestra.run.smithi148.stdout:ID_LIKE=debian 2024-02-01T10:00:20.787 INFO:teuthology.orchestra.run.smithi148.stdout:HOME_URL="https://www.ubuntu.com/" 2024-02-01T10:00:20.787 INFO:teuthology.orchestra.run.smithi148.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-02-01T10:00:20.787 INFO:teuthology.orchestra.run.smithi148.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-02-01T10:00:20.787 INFO:teuthology.orchestra.run.smithi148.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-02-01T10:00:20.787 INFO:teuthology.orchestra.run.smithi148.stdout:UBUNTU_CODENAME=jammy 2024-02-01T10:00:20.788 INFO:teuthology.lock.ops:Updating smithi148.front.sepia.ceph.com on lock server 2024-02-01T10:00:20.815 DEBUG:teuthology.orchestra.run.smithi187:> uname -m 2024-02-01T10:00:20.821 INFO:teuthology.orchestra.run.smithi187.stdout:x86_64 2024-02-01T10:00:20.821 DEBUG:teuthology.orchestra.run.smithi187:> cat /etc/os-release 2024-02-01T10:00:20.866 INFO:teuthology.orchestra.run.smithi187.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-02-01T10:00:20.866 INFO:teuthology.orchestra.run.smithi187.stdout:NAME="Ubuntu" 2024-02-01T10:00:20.867 INFO:teuthology.orchestra.run.smithi187.stdout:VERSION_ID="22.04" 2024-02-01T10:00:20.867 INFO:teuthology.orchestra.run.smithi187.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-02-01T10:00:20.867 INFO:teuthology.orchestra.run.smithi187.stdout:VERSION_CODENAME=jammy 2024-02-01T10:00:20.867 INFO:teuthology.orchestra.run.smithi187.stdout:ID=ubuntu 2024-02-01T10:00:20.867 INFO:teuthology.orchestra.run.smithi187.stdout:ID_LIKE=debian 2024-02-01T10:00:20.867 INFO:teuthology.orchestra.run.smithi187.stdout:HOME_URL="https://www.ubuntu.com/" 2024-02-01T10:00:20.867 INFO:teuthology.orchestra.run.smithi187.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-02-01T10:00:20.867 INFO:teuthology.orchestra.run.smithi187.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-02-01T10:00:20.867 INFO:teuthology.orchestra.run.smithi187.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-02-01T10:00:20.867 INFO:teuthology.orchestra.run.smithi187.stdout:UBUNTU_CODENAME=jammy 2024-02-01T10:00:20.867 INFO:teuthology.lock.ops:Updating smithi187.front.sepia.ceph.com on lock server 2024-02-01T10:00:20.889 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-02-01T10:00:20.903 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-02-01T10:00:20.915 INFO:teuthology.task.internal:Checking for old test directory... 2024-02-01T10:00:20.915 DEBUG:teuthology.orchestra.run.smithi148:> test '!' -e /home/ubuntu/cephtest 2024-02-01T10:00:20.921 DEBUG:teuthology.orchestra.run.smithi187:> test '!' -e /home/ubuntu/cephtest 2024-02-01T10:00:20.926 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-02-01T10:00:20.937 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-02-01T10:00:20.937 DEBUG:teuthology.orchestra.run.smithi148:> test -z $(ls -A /var/lib/ceph) 2024-02-01T10:00:20.968 DEBUG:teuthology.orchestra.run.smithi187:> test -z $(ls -A /var/lib/ceph) 2024-02-01T10:00:21.049 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-02-01T10:00:21.108 INFO:teuthology.run_tasks:Running task kernel... 2024-02-01T10:00:21.125 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-02-01T10:00:21.125 INFO:teuthology.task.kernel:config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-02-01T10:00:21.126 DEBUG:teuthology.orchestra.run.smithi148:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-01T10:00:21.126 DEBUG:teuthology.orchestra.run.smithi187:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-01T10:00:21.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-01T10:00:21.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-01T10:00:21.132 DEBUG:teuthology.orchestra.run.smithi187:> uname -r 2024-02-01T10:00:21.133 DEBUG:teuthology.orchestra.run.smithi148:> uname -r 2024-02-01T10:00:21.180 INFO:teuthology.orchestra.run.smithi148.stdout:5.15.0-56-generic 2024-02-01T10:00:21.180 INFO:teuthology.orchestra.run.smithi187.stdout:5.15.0-56-generic 2024-02-01T10:00:21.180 INFO:teuthology.task.kernel:Running kernel on smithi148: 5.15.0-56-generic 2024-02-01T10:00:21.180 DEBUG:teuthology.orchestra.run.smithi148:> sudo apt-get clean 2024-02-01T10:00:21.181 INFO:teuthology.task.kernel:Running kernel on smithi187: 5.15.0-56-generic 2024-02-01T10:00:21.181 DEBUG:teuthology.orchestra.run.smithi187:> sudo apt-get clean 2024-02-01T10:00:21.303 DEBUG:teuthology.orchestra.run.smithi148:> sudo apt-get update 2024-02-01T10:00:21.315 DEBUG:teuthology.orchestra.run.smithi187:> sudo apt-get update 2024-02-01T10:00:21.478 INFO:teuthology.orchestra.run.smithi148.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-02-01T10:00:21.478 INFO:teuthology.orchestra.run.smithi148.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-02-01T10:00:21.478 INFO:teuthology.orchestra.run.smithi148.stdout:Get:3 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-02-01T10:00:21.558 INFO:teuthology.orchestra.run.smithi148.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-02-01T10:00:21.591 INFO:teuthology.orchestra.run.smithi187.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-02-01T10:00:21.592 INFO:teuthology.orchestra.run.smithi187.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-02-01T10:00:21.657 INFO:teuthology.orchestra.run.smithi187.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-02-01T10:00:21.728 INFO:teuthology.orchestra.run.smithi148.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,109 kB] 2024-02-01T10:00:21.802 INFO:teuthology.orchestra.run.smithi148.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [560 kB] 2024-02-01T10:00:21.831 INFO:teuthology.orchestra.run.smithi148.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [393 kB] 2024-02-01T10:00:21.836 INFO:teuthology.orchestra.run.smithi148.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [207 kB] 2024-02-01T10:00:21.839 INFO:teuthology.orchestra.run.smithi148.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-02-01T10:00:21.839 INFO:teuthology.orchestra.run.smithi148.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,313 kB] 2024-02-01T10:00:21.861 INFO:teuthology.orchestra.run.smithi148.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [33.2 kB] 2024-02-01T10:00:21.861 INFO:teuthology.orchestra.run.smithi148.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [214 kB] 2024-02-01T10:00:21.864 INFO:teuthology.orchestra.run.smithi148.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-02-01T10:00:21.864 INFO:teuthology.orchestra.run.smithi148.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [837 kB] 2024-02-01T10:00:21.870 INFO:teuthology.orchestra.run.smithi148.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,325 kB] 2024-02-01T10:00:21.874 INFO:teuthology.orchestra.run.smithi148.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [588 kB] 2024-02-01T10:00:21.880 INFO:teuthology.orchestra.run.smithi148.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [160 kB] 2024-02-01T10:00:21.882 INFO:teuthology.orchestra.run.smithi148.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-02-01T10:00:21.905 INFO:teuthology.orchestra.run.smithi148.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-02-01T10:00:21.905 INFO:teuthology.orchestra.run.smithi148.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [267 kB] 2024-02-01T10:00:21.905 INFO:teuthology.orchestra.run.smithi148.stdout:Get:21 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.1 kB] 2024-02-01T10:00:21.906 INFO:teuthology.orchestra.run.smithi148.stdout:Get:22 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,476 B] 2024-02-01T10:00:21.906 INFO:teuthology.orchestra.run.smithi148.stdout:Get:23 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-02-01T10:00:21.911 INFO:teuthology.orchestra.run.smithi148.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-02-01T10:00:21.912 INFO:teuthology.orchestra.run.smithi148.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [34.8 kB] 2024-02-01T10:00:21.913 INFO:teuthology.orchestra.run.smithi148.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,342 kB] 2024-02-01T10:00:21.930 INFO:teuthology.orchestra.run.smithi148.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [220 kB] 2024-02-01T10:00:21.933 INFO:teuthology.orchestra.run.smithi148.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-02-01T10:00:21.933 INFO:teuthology.orchestra.run.smithi148.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [686 kB] 2024-02-01T10:00:21.982 INFO:teuthology.orchestra.run.smithi148.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,042 kB] 2024-02-01T10:00:21.992 INFO:teuthology.orchestra.run.smithi148.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [235 kB] 2024-02-01T10:00:21.995 INFO:teuthology.orchestra.run.smithi148.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-02-01T10:00:21.995 INFO:teuthology.orchestra.run.smithi148.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.1 kB] 2024-02-01T10:00:21.996 INFO:teuthology.orchestra.run.smithi148.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,164 B] 2024-02-01T10:00:21.996 INFO:teuthology.orchestra.run.smithi148.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.1 kB] 2024-02-01T10:00:21.996 INFO:teuthology.orchestra.run.smithi148.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-02-01T10:00:21.997 INFO:teuthology.orchestra.run.smithi148.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [33.9 kB] 2024-02-01T10:00:21.997 INFO:teuthology.orchestra.run.smithi148.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [41.7 kB] 2024-02-01T10:00:22.004 INFO:teuthology.orchestra.run.smithi148.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [10.5 kB] 2024-02-01T10:00:22.018 INFO:teuthology.orchestra.run.smithi187.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,109 kB] 2024-02-01T10:00:22.018 INFO:teuthology.orchestra.run.smithi187.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-02-01T10:00:22.023 INFO:teuthology.orchestra.run.smithi148.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-02-01T10:00:22.024 INFO:teuthology.orchestra.run.smithi148.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [24.2 kB] 2024-02-01T10:00:22.024 INFO:teuthology.orchestra.run.smithi148.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [13.4 kB] 2024-02-01T10:00:22.024 INFO:teuthology.orchestra.run.smithi148.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-02-01T10:00:22.025 INFO:teuthology.orchestra.run.smithi148.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-02-01T10:00:22.156 INFO:teuthology.orchestra.run.smithi187.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [560 kB] 2024-02-01T10:00:22.345 INFO:teuthology.orchestra.run.smithi187.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,325 kB] 2024-02-01T10:00:22.415 INFO:teuthology.orchestra.run.smithi187.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [393 kB] 2024-02-01T10:00:22.435 INFO:teuthology.orchestra.run.smithi187.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [207 kB] 2024-02-01T10:00:22.446 INFO:teuthology.orchestra.run.smithi187.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-02-01T10:00:22.446 INFO:teuthology.orchestra.run.smithi187.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,313 kB] 2024-02-01T10:00:22.484 INFO:teuthology.orchestra.run.smithi187.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [267 kB] 2024-02-01T10:00:22.499 INFO:teuthology.orchestra.run.smithi187.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-02-01T10:00:22.500 INFO:teuthology.orchestra.run.smithi187.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,342 kB] 2024-02-01T10:00:22.534 INFO:teuthology.orchestra.run.smithi187.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [33.2 kB] 2024-02-01T10:00:22.535 INFO:teuthology.orchestra.run.smithi187.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [214 kB] 2024-02-01T10:00:22.576 INFO:teuthology.orchestra.run.smithi187.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [34.8 kB] 2024-02-01T10:00:22.576 INFO:teuthology.orchestra.run.smithi187.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-02-01T10:00:22.576 INFO:teuthology.orchestra.run.smithi187.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [837 kB] 2024-02-01T10:00:22.577 INFO:teuthology.orchestra.run.smithi187.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [220 kB] 2024-02-01T10:00:22.581 INFO:teuthology.orchestra.run.smithi187.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-02-01T10:00:22.590 INFO:teuthology.orchestra.run.smithi187.stdout:Get:22 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [588 kB] 2024-02-01T10:00:22.744 INFO:teuthology.orchestra.run.smithi187.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,042 kB] 2024-02-01T10:00:22.764 INFO:teuthology.orchestra.run.smithi187.stdout:Get:24 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [160 kB] 2024-02-01T10:00:23.165 INFO:teuthology.orchestra.run.smithi187.stdout:Get:25 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-02-01T10:00:23.171 INFO:teuthology.orchestra.run.smithi187.stdout:Get:26 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.1 kB] 2024-02-01T10:00:23.185 INFO:teuthology.orchestra.run.smithi187.stdout:Get:27 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-02-01T10:00:23.186 INFO:teuthology.orchestra.run.smithi187.stdout:Get:28 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,476 B] 2024-02-01T10:00:23.188 INFO:teuthology.orchestra.run.smithi187.stdout:Get:29 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-02-01T10:00:23.367 INFO:teuthology.orchestra.run.smithi187.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [686 kB] 2024-02-01T10:00:23.405 INFO:teuthology.orchestra.run.smithi187.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [235 kB] 2024-02-01T10:00:23.418 INFO:teuthology.orchestra.run.smithi187.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-02-01T10:00:23.419 INFO:teuthology.orchestra.run.smithi187.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.1 kB] 2024-02-01T10:00:23.422 INFO:teuthology.orchestra.run.smithi187.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,164 B] 2024-02-01T10:00:23.422 INFO:teuthology.orchestra.run.smithi187.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.1 kB] 2024-02-01T10:00:23.422 INFO:teuthology.orchestra.run.smithi187.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-02-01T10:00:23.423 INFO:teuthology.orchestra.run.smithi187.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [33.9 kB] 2024-02-01T10:00:23.425 INFO:teuthology.orchestra.run.smithi187.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [41.7 kB] 2024-02-01T10:00:23.450 INFO:teuthology.orchestra.run.smithi187.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [10.5 kB] 2024-02-01T10:00:23.614 INFO:teuthology.orchestra.run.smithi187.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-02-01T10:00:23.697 INFO:teuthology.orchestra.run.smithi187.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [13.4 kB] 2024-02-01T10:00:23.702 INFO:teuthology.orchestra.run.smithi187.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [24.2 kB] 2024-02-01T10:00:23.820 INFO:teuthology.orchestra.run.smithi187.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-02-01T10:00:23.872 INFO:teuthology.orchestra.run.smithi187.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-02-01T10:00:27.431 INFO:teuthology.orchestra.run.smithi148.stdout:Fetched 11.2 MB in 2s (6,708 kB/s) 2024-02-01T10:00:28.104 INFO:teuthology.orchestra.run.smithi187.stdout:Fetched 11.2 MB in 2s (4,522 kB/s) 2024-02-01T10:00:28.754 INFO:teuthology.orchestra.run.smithi148.stdout:Reading package lists... 2024-02-01T10:00:28.772 DEBUG:teuthology.orchestra.run.smithi148:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-02-01T10:00:28.823 INFO:teuthology.orchestra.run.smithi148.stdout:Reading package lists... 2024-02-01T10:00:29.000 INFO:teuthology.orchestra.run.smithi148.stdout:Building dependency tree... 2024-02-01T10:00:29.000 INFO:teuthology.orchestra.run.smithi148.stdout:Reading state information... 2024-02-01T10:00:29.151 INFO:teuthology.orchestra.run.smithi148.stdout:The following packages were automatically installed and are no longer required: 2024-02-01T10:00:29.152 INFO:teuthology.orchestra.run.smithi148.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-02-01T10:00:29.152 INFO:teuthology.orchestra.run.smithi148.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-02-01T10:00:29.152 INFO:teuthology.orchestra.run.smithi148.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-02-01T10:00:29.152 INFO:teuthology.orchestra.run.smithi148.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-02-01T10:00:29.152 INFO:teuthology.orchestra.run.smithi148.stdout: python2.7-minimal 2024-02-01T10:00:29.153 INFO:teuthology.orchestra.run.smithi148.stdout:Use 'sudo apt autoremove' to remove them. 2024-02-01T10:00:29.153 INFO:teuthology.orchestra.run.smithi148.stdout:The following additional packages will be installed: 2024-02-01T10:00:29.154 INFO:teuthology.orchestra.run.smithi148.stdout: linux-generic linux-headers-5.15.0-92 linux-headers-5.15.0-92-generic 2024-02-01T10:00:29.154 INFO:teuthology.orchestra.run.smithi148.stdout: linux-headers-generic linux-image-5.15.0-92-generic 2024-02-01T10:00:29.154 INFO:teuthology.orchestra.run.smithi148.stdout: linux-modules-5.15.0-92-generic linux-modules-extra-5.15.0-92-generic 2024-02-01T10:00:29.155 INFO:teuthology.orchestra.run.smithi148.stdout:Suggested packages: 2024-02-01T10:00:29.155 INFO:teuthology.orchestra.run.smithi148.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-02-01T10:00:29.155 INFO:teuthology.orchestra.run.smithi148.stdout:Recommended packages: 2024-02-01T10:00:29.156 INFO:teuthology.orchestra.run.smithi148.stdout: thermald 2024-02-01T10:00:29.203 INFO:teuthology.orchestra.run.smithi148.stdout:The following NEW packages will be installed: 2024-02-01T10:00:29.203 INFO:teuthology.orchestra.run.smithi148.stdout: linux-headers-5.15.0-92 linux-headers-5.15.0-92-generic 2024-02-01T10:00:29.203 INFO:teuthology.orchestra.run.smithi148.stdout: linux-image-5.15.0-92-generic linux-modules-5.15.0-92-generic 2024-02-01T10:00:29.204 INFO:teuthology.orchestra.run.smithi148.stdout: linux-modules-extra-5.15.0-92-generic 2024-02-01T10:00:29.204 INFO:teuthology.orchestra.run.smithi148.stdout:The following packages will be upgraded: 2024-02-01T10:00:29.205 INFO:teuthology.orchestra.run.smithi148.stdout: linux-generic linux-headers-generic linux-image-generic 2024-02-01T10:00:29.383 INFO:teuthology.orchestra.run.smithi148.stdout:3 upgraded, 5 newly installed, 0 to remove and 281 not upgraded. 2024-02-01T10:00:29.383 INFO:teuthology.orchestra.run.smithi148.stdout:Need to get 113 MB of archives. 2024-02-01T10:00:29.383 INFO:teuthology.orchestra.run.smithi148.stdout:After this operation, 582 MB of additional disk space will be used. 2024-02-01T10:00:29.383 INFO:teuthology.orchestra.run.smithi148.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-92-generic amd64 5.15.0-92.102 [22.5 MB] 2024-02-01T10:00:29.488 INFO:teuthology.orchestra.run.smithi187.stdout:Reading package lists... 2024-02-01T10:00:29.505 DEBUG:teuthology.orchestra.run.smithi187:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-02-01T10:00:29.564 INFO:teuthology.orchestra.run.smithi187.stdout:Reading package lists... 2024-02-01T10:00:29.755 INFO:teuthology.orchestra.run.smithi187.stdout:Building dependency tree... 2024-02-01T10:00:29.756 INFO:teuthology.orchestra.run.smithi187.stdout:Reading state information... 2024-02-01T10:00:29.913 INFO:teuthology.orchestra.run.smithi187.stdout:The following packages were automatically installed and are no longer required: 2024-02-01T10:00:29.913 INFO:teuthology.orchestra.run.smithi187.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-02-01T10:00:29.913 INFO:teuthology.orchestra.run.smithi187.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-02-01T10:00:29.913 INFO:teuthology.orchestra.run.smithi187.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-02-01T10:00:29.914 INFO:teuthology.orchestra.run.smithi187.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-02-01T10:00:29.914 INFO:teuthology.orchestra.run.smithi187.stdout: python2.7-minimal 2024-02-01T10:00:29.914 INFO:teuthology.orchestra.run.smithi187.stdout:Use 'sudo apt autoremove' to remove them. 2024-02-01T10:00:29.914 INFO:teuthology.orchestra.run.smithi187.stdout:The following additional packages will be installed: 2024-02-01T10:00:29.914 INFO:teuthology.orchestra.run.smithi187.stdout: linux-generic linux-headers-5.15.0-92 linux-headers-5.15.0-92-generic 2024-02-01T10:00:29.914 INFO:teuthology.orchestra.run.smithi187.stdout: linux-headers-generic linux-image-5.15.0-92-generic 2024-02-01T10:00:29.915 INFO:teuthology.orchestra.run.smithi187.stdout: linux-modules-5.15.0-92-generic linux-modules-extra-5.15.0-92-generic 2024-02-01T10:00:29.916 INFO:teuthology.orchestra.run.smithi187.stdout:Suggested packages: 2024-02-01T10:00:29.916 INFO:teuthology.orchestra.run.smithi187.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-02-01T10:00:29.916 INFO:teuthology.orchestra.run.smithi187.stdout:Recommended packages: 2024-02-01T10:00:29.916 INFO:teuthology.orchestra.run.smithi187.stdout: thermald 2024-02-01T10:00:29.956 INFO:teuthology.orchestra.run.smithi187.stdout:The following NEW packages will be installed: 2024-02-01T10:00:29.956 INFO:teuthology.orchestra.run.smithi187.stdout: linux-headers-5.15.0-92 linux-headers-5.15.0-92-generic 2024-02-01T10:00:29.957 INFO:teuthology.orchestra.run.smithi187.stdout: linux-image-5.15.0-92-generic linux-modules-5.15.0-92-generic 2024-02-01T10:00:29.957 INFO:teuthology.orchestra.run.smithi187.stdout: linux-modules-extra-5.15.0-92-generic 2024-02-01T10:00:29.957 INFO:teuthology.orchestra.run.smithi187.stdout:The following packages will be upgraded: 2024-02-01T10:00:29.958 INFO:teuthology.orchestra.run.smithi187.stdout: linux-generic linux-headers-generic linux-image-generic 2024-02-01T10:00:30.014 INFO:teuthology.orchestra.run.smithi187.stdout:3 upgraded, 5 newly installed, 0 to remove and 281 not upgraded. 2024-02-01T10:00:30.014 INFO:teuthology.orchestra.run.smithi187.stdout:Need to get 113 MB of archives. 2024-02-01T10:00:30.014 INFO:teuthology.orchestra.run.smithi187.stdout:After this operation, 582 MB of additional disk space will be used. 2024-02-01T10:00:30.014 INFO:teuthology.orchestra.run.smithi187.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-92-generic amd64 5.15.0-92.102 [22.5 MB] 2024-02-01T10:00:30.527 INFO:teuthology.orchestra.run.smithi187.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-92-generic amd64 5.15.0-92.102 [11.5 MB] 2024-02-01T10:00:30.629 INFO:teuthology.orchestra.run.smithi187.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-92-generic amd64 5.15.0-92.102 [63.8 MB] 2024-02-01T10:00:30.798 INFO:teuthology.orchestra.run.smithi148.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-92-generic amd64 5.15.0-92.102 [11.5 MB] 2024-02-01T10:00:31.128 INFO:teuthology.orchestra.run.smithi148.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-92-generic amd64 5.15.0-92.102 [63.8 MB] 2024-02-01T10:00:31.560 INFO:teuthology.orchestra.run.smithi187.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.92.89 [1,700 B] 2024-02-01T10:00:31.560 INFO:teuthology.orchestra.run.smithi187.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.92.89 [2,498 B] 2024-02-01T10:00:31.561 INFO:teuthology.orchestra.run.smithi187.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-92 all 5.15.0-92.102 [12.3 MB] 2024-02-01T10:00:31.688 INFO:teuthology.orchestra.run.smithi187.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-92-generic amd64 5.15.0-92.102 [2,859 kB] 2024-02-01T10:00:31.716 INFO:teuthology.orchestra.run.smithi187.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.92.89 [2,346 B] 2024-02-01T10:00:32.092 INFO:teuthology.orchestra.run.smithi187.stdout:Fetched 113 MB in 2s (64.5 MB/s) 2024-02-01T10:00:32.310 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package linux-modules-5.15.0-92-generic. 2024-02-01T10:00:33.299 INFO:teuthology.orchestra.run.smithi148.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.92.89 [1,700 B] 2024-02-01T10:00:33.299 INFO:teuthology.orchestra.run.smithi148.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.92.89 [2,498 B] 2024-02-01T10:00:33.300 INFO:teuthology.orchestra.run.smithi148.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-92 all 5.15.0-92.102 [12.3 MB] 2024-02-01T10:00:34.458 INFO:teuthology.orchestra.run.smithi187.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-02-01T10:00:34.462 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../0-linux-modules-5.15.0-92-generic_5.15.0-92.102_amd64.deb ... 2024-02-01T10:00:34.567 INFO:teuthology.orchestra.run.smithi148.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-92-generic amd64 5.15.0-92.102 [2,859 kB] 2024-02-01T10:00:34.677 INFO:teuthology.orchestra.run.smithi148.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.92.89 [2,346 B] 2024-02-01T10:00:34.749 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking linux-modules-5.15.0-92-generic (5.15.0-92.102) ... 2024-02-01T10:00:35.066 INFO:teuthology.orchestra.run.smithi148.stdout:Fetched 113 MB in 5s (20.7 MB/s) 2024-02-01T10:00:35.262 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package linux-modules-5.15.0-92-generic. 2024-02-01T10:00:36.866 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package linux-image-5.15.0-92-generic. 2024-02-01T10:00:36.879 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../1-linux-image-5.15.0-92-generic_5.15.0-92.102_amd64.deb ... 2024-02-01T10:00:36.967 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking linux-image-5.15.0-92-generic (5.15.0-92.102) ... 2024-02-01T10:00:37.272 INFO:teuthology.orchestra.run.smithi148.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-02-01T10:00:37.277 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../0-linux-modules-5.15.0-92-generic_5.15.0-92.102_amd64.deb ... 2024-02-01T10:00:37.342 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking linux-modules-5.15.0-92-generic (5.15.0-92.102) ... 2024-02-01T10:00:37.430 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-92-generic. 2024-02-01T10:00:37.450 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-92-generic_5.15.0-92.102_amd64.deb ... 2024-02-01T10:00:37.488 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking linux-modules-extra-5.15.0-92-generic (5.15.0-92.102) ... 2024-02-01T10:00:39.712 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package linux-image-5.15.0-92-generic. 2024-02-01T10:00:39.733 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../1-linux-image-5.15.0-92-generic_5.15.0-92.102_amd64.deb ... 2024-02-01T10:00:39.809 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking linux-image-5.15.0-92-generic (5.15.0-92.102) ... 2024-02-01T10:00:40.299 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-92-generic. 2024-02-01T10:00:40.319 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-92-generic_5.15.0-92.102_amd64.deb ... 2024-02-01T10:00:40.348 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking linux-modules-extra-5.15.0-92-generic (5.15.0-92.102) ... 2024-02-01T10:00:44.056 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../3-linux-generic_5.15.0.92.89_amd64.deb ... 2024-02-01T10:00:44.451 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking linux-generic (5.15.0.92.89) over (5.15.0.56.54) ... 2024-02-01T10:00:44.767 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.92.89_amd64.deb ... 2024-02-01T10:00:44.897 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking linux-image-generic (5.15.0.92.89) over (5.15.0.56.54) ... 2024-02-01T10:00:45.199 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package linux-headers-5.15.0-92. 2024-02-01T10:00:45.211 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../5-linux-headers-5.15.0-92_5.15.0-92.102_all.deb ... 2024-02-01T10:00:45.271 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking linux-headers-5.15.0-92 (5.15.0-92.102) ... 2024-02-01T10:00:46.689 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../3-linux-generic_5.15.0.92.89_amd64.deb ... 2024-02-01T10:00:46.816 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking linux-generic (5.15.0.92.89) over (5.15.0.56.54) ... 2024-02-01T10:00:47.174 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.92.89_amd64.deb ... 2024-02-01T10:00:47.286 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking linux-image-generic (5.15.0.92.89) over (5.15.0.56.54) ... 2024-02-01T10:00:47.555 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package linux-headers-5.15.0-92. 2024-02-01T10:00:47.566 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../5-linux-headers-5.15.0-92_5.15.0-92.102_all.deb ... 2024-02-01T10:00:47.605 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking linux-headers-5.15.0-92 (5.15.0-92.102) ... 2024-02-01T10:00:50.720 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package linux-headers-5.15.0-92-generic. 2024-02-01T10:00:50.757 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../6-linux-headers-5.15.0-92-generic_5.15.0-92.102_amd64.deb ... 2024-02-01T10:00:50.794 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking linux-headers-5.15.0-92-generic (5.15.0-92.102) ... 2024-02-01T10:00:53.128 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package linux-headers-5.15.0-92-generic. 2024-02-01T10:00:53.165 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../6-linux-headers-5.15.0-92-generic_5.15.0-92.102_amd64.deb ... 2024-02-01T10:00:53.203 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking linux-headers-5.15.0-92-generic (5.15.0-92.102) ... 2024-02-01T10:00:53.399 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.92.89_amd64.deb ... 2024-02-01T10:00:53.521 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking linux-headers-generic (5.15.0.92.89) over (5.15.0.56.54) ... 2024-02-01T10:00:53.863 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up linux-headers-5.15.0-92 (5.15.0-92.102) ... 2024-02-01T10:00:53.984 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up linux-headers-5.15.0-92-generic (5.15.0-92.102) ... 2024-02-01T10:00:54.104 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up linux-headers-generic (5.15.0.92.89) ... 2024-02-01T10:00:54.205 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up linux-image-5.15.0-92-generic (5.15.0-92.102) ... 2024-02-01T10:00:55.511 INFO:teuthology.orchestra.run.smithi187.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-02-01T10:00:55.512 INFO:teuthology.orchestra.run.smithi187.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-02-01T10:00:55.512 INFO:teuthology.orchestra.run.smithi187.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-92-generic 2024-02-01T10:00:55.512 INFO:teuthology.orchestra.run.smithi187.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-92-generic 2024-02-01T10:00:55.666 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up linux-modules-5.15.0-92-generic (5.15.0-92.102) ... 2024-02-01T10:00:55.780 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.92.89_amd64.deb ... 2024-02-01T10:00:55.906 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking linux-headers-generic (5.15.0.92.89) over (5.15.0.56.54) ... 2024-02-01T10:00:56.263 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up linux-headers-5.15.0-92 (5.15.0-92.102) ... 2024-02-01T10:00:56.385 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up linux-headers-5.15.0-92-generic (5.15.0-92.102) ... 2024-02-01T10:00:56.528 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up linux-headers-generic (5.15.0.92.89) ... 2024-02-01T10:00:56.656 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up linux-image-5.15.0-92-generic (5.15.0-92.102) ... 2024-02-01T10:00:57.592 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up linux-modules-extra-5.15.0-92-generic (5.15.0-92.102) ... 2024-02-01T10:00:58.034 INFO:teuthology.orchestra.run.smithi148.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-02-01T10:00:58.034 INFO:teuthology.orchestra.run.smithi148.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-02-01T10:00:58.035 INFO:teuthology.orchestra.run.smithi148.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-92-generic 2024-02-01T10:00:58.035 INFO:teuthology.orchestra.run.smithi148.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-92-generic 2024-02-01T10:00:58.174 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up linux-modules-5.15.0-92-generic (5.15.0-92.102) ... 2024-02-01T10:00:59.002 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up linux-image-generic (5.15.0.92.89) ... 2024-02-01T10:00:59.127 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up linux-generic (5.15.0.92.89) ... 2024-02-01T10:00:59.254 INFO:teuthology.orchestra.run.smithi187.stdout:Processing triggers for linux-image-5.15.0-92-generic (5.15.0-92.102) ... 2024-02-01T10:00:59.327 INFO:teuthology.orchestra.run.smithi187.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-02-01T10:00:59.327 INFO:teuthology.orchestra.run.smithi187.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-92-generic 2024-02-01T10:00:59.567 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up linux-modules-extra-5.15.0-92-generic (5.15.0-92.102) ... 2024-02-01T10:01:00.960 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up linux-image-generic (5.15.0.92.89) ... 2024-02-01T10:01:01.086 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up linux-generic (5.15.0.92.89) ... 2024-02-01T10:01:01.212 INFO:teuthology.orchestra.run.smithi148.stdout:Processing triggers for linux-image-5.15.0-92-generic (5.15.0-92.102) ... 2024-02-01T10:01:01.280 INFO:teuthology.orchestra.run.smithi148.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-02-01T10:01:01.280 INFO:teuthology.orchestra.run.smithi148.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-92-generic 2024-02-01T10:01:13.769 INFO:teuthology.orchestra.run.smithi187.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-02-01T10:01:13.769 INFO:teuthology.orchestra.run.smithi187.stdout:Sourcing file `/etc/default/grub' 2024-02-01T10:01:13.780 INFO:teuthology.orchestra.run.smithi187.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-02-01T10:01:13.806 INFO:teuthology.orchestra.run.smithi187.stdout:Generating grub configuration file ... 2024-02-01T10:01:14.139 INFO:teuthology.orchestra.run.smithi187.stdout:Found linux image: /boot/vmlinuz-5.15.0-92-generic 2024-02-01T10:01:14.152 INFO:teuthology.orchestra.run.smithi187.stdout:Found initrd image: /boot/initrd.img-5.15.0-92-generic 2024-02-01T10:01:14.529 INFO:teuthology.orchestra.run.smithi187.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-02-01T10:01:14.533 INFO:teuthology.orchestra.run.smithi187.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-02-01T10:01:14.609 INFO:teuthology.orchestra.run.smithi187.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-02-01T10:01:14.616 INFO:teuthology.orchestra.run.smithi187.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-02-01T10:01:14.780 INFO:teuthology.orchestra.run.smithi187.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-02-01T10:01:14.780 INFO:teuthology.orchestra.run.smithi187.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-02-01T10:01:14.780 INFO:teuthology.orchestra.run.smithi187.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-02-01T10:01:14.817 INFO:teuthology.orchestra.run.smithi187.stdout:done 2024-02-01T10:01:15.299 DEBUG:teuthology.orchestra.run.smithi187:> dpkg -s linux-image-generic 2024-02-01T10:01:15.319 INFO:teuthology.orchestra.run.smithi187.stdout:Package: linux-image-generic 2024-02-01T10:01:15.319 INFO:teuthology.orchestra.run.smithi187.stdout:Status: install ok installed 2024-02-01T10:01:15.319 INFO:teuthology.orchestra.run.smithi187.stdout:Priority: optional 2024-02-01T10:01:15.320 INFO:teuthology.orchestra.run.smithi187.stdout:Section: kernel 2024-02-01T10:01:15.320 INFO:teuthology.orchestra.run.smithi187.stdout:Installed-Size: 21 2024-02-01T10:01:15.320 INFO:teuthology.orchestra.run.smithi187.stdout:Maintainer: Ubuntu Kernel Team 2024-02-01T10:01:15.320 INFO:teuthology.orchestra.run.smithi187.stdout:Architecture: amd64 2024-02-01T10:01:15.320 INFO:teuthology.orchestra.run.smithi187.stdout:Source: linux-meta 2024-02-01T10:01:15.320 INFO:teuthology.orchestra.run.smithi187.stdout:Version: 5.15.0.92.89 2024-02-01T10:01:15.320 INFO:teuthology.orchestra.run.smithi187.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-92), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2024-02-01T10:01:15.320 INFO:teuthology.orchestra.run.smithi187.stdout:Depends: linux-image-5.15.0-92-generic, linux-modules-extra-5.15.0-92-generic, linux-firmware, intel-microcode, amd64-microcode 2024-02-01T10:01:15.320 INFO:teuthology.orchestra.run.smithi187.stdout:Recommends: thermald 2024-02-01T10:01:15.320 INFO:teuthology.orchestra.run.smithi187.stdout:Description: Generic Linux kernel image 2024-02-01T10:01:15.320 INFO:teuthology.orchestra.run.smithi187.stdout: This package will always depend on the latest generic kernel image 2024-02-01T10:01:15.320 INFO:teuthology.orchestra.run.smithi187.stdout: available. 2024-02-01T10:01:15.321 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-92-generic 2024-02-01T10:01:15.321 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-01T10:01:15.321 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-01T10:01:15.321 INFO:teuthology.task.kernel:Installing distro kernel on c2.mon.a... 2024-02-01T10:01:15.321 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi187.front.sepia.ceph.com, path=None, version=distro) 2024-02-01T10:01:15.321 DEBUG:teuthology.orchestra.run.smithi187:> sudo apt-get clean 2024-02-01T10:01:15.462 INFO:teuthology.orchestra.run.smithi148.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-02-01T10:01:15.462 INFO:teuthology.orchestra.run.smithi148.stdout:Sourcing file `/etc/default/grub' 2024-02-01T10:01:15.477 INFO:teuthology.orchestra.run.smithi148.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-02-01T10:01:15.488 DEBUG:teuthology.orchestra.run.smithi187:> sudo apt-get update 2024-02-01T10:01:15.498 INFO:teuthology.orchestra.run.smithi148.stdout:Generating grub configuration file ... 2024-02-01T10:01:15.727 INFO:teuthology.orchestra.run.smithi187.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-02-01T10:01:15.727 INFO:teuthology.orchestra.run.smithi187.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-02-01T10:01:15.758 INFO:teuthology.orchestra.run.smithi148.stdout:Found linux image: /boot/vmlinuz-5.15.0-92-generic 2024-02-01T10:01:15.774 INFO:teuthology.orchestra.run.smithi148.stdout:Found initrd image: /boot/initrd.img-5.15.0-92-generic 2024-02-01T10:01:15.799 INFO:teuthology.orchestra.run.smithi187.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-02-01T10:01:15.881 INFO:teuthology.orchestra.run.smithi187.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-02-01T10:01:16.059 INFO:teuthology.orchestra.run.smithi148.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-02-01T10:01:16.065 INFO:teuthology.orchestra.run.smithi148.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-02-01T10:01:16.137 INFO:teuthology.orchestra.run.smithi148.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-02-01T10:01:16.143 INFO:teuthology.orchestra.run.smithi148.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-02-01T10:01:16.318 INFO:teuthology.orchestra.run.smithi148.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-02-01T10:01:16.318 INFO:teuthology.orchestra.run.smithi148.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-02-01T10:01:16.318 INFO:teuthology.orchestra.run.smithi148.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-02-01T10:01:16.360 INFO:teuthology.orchestra.run.smithi148.stdout:done 2024-02-01T10:01:16.802 DEBUG:teuthology.orchestra.run.smithi148:> dpkg -s linux-image-generic 2024-02-01T10:01:16.821 INFO:teuthology.orchestra.run.smithi148.stdout:Package: linux-image-generic 2024-02-01T10:01:16.821 INFO:teuthology.orchestra.run.smithi148.stdout:Status: install ok installed 2024-02-01T10:01:16.822 INFO:teuthology.orchestra.run.smithi148.stdout:Priority: optional 2024-02-01T10:01:16.822 INFO:teuthology.orchestra.run.smithi148.stdout:Section: kernel 2024-02-01T10:01:16.822 INFO:teuthology.orchestra.run.smithi148.stdout:Installed-Size: 21 2024-02-01T10:01:16.822 INFO:teuthology.orchestra.run.smithi148.stdout:Maintainer: Ubuntu Kernel Team 2024-02-01T10:01:16.822 INFO:teuthology.orchestra.run.smithi148.stdout:Architecture: amd64 2024-02-01T10:01:16.822 INFO:teuthology.orchestra.run.smithi148.stdout:Source: linux-meta 2024-02-01T10:01:16.822 INFO:teuthology.orchestra.run.smithi148.stdout:Version: 5.15.0.92.89 2024-02-01T10:01:16.822 INFO:teuthology.orchestra.run.smithi148.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-92), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2024-02-01T10:01:16.822 INFO:teuthology.orchestra.run.smithi148.stdout:Depends: linux-image-5.15.0-92-generic, linux-modules-extra-5.15.0-92-generic, linux-firmware, intel-microcode, amd64-microcode 2024-02-01T10:01:16.822 INFO:teuthology.orchestra.run.smithi148.stdout:Recommends: thermald 2024-02-01T10:01:16.822 INFO:teuthology.orchestra.run.smithi148.stdout:Description: Generic Linux kernel image 2024-02-01T10:01:16.822 INFO:teuthology.orchestra.run.smithi148.stdout: This package will always depend on the latest generic kernel image 2024-02-01T10:01:16.822 INFO:teuthology.orchestra.run.smithi148.stdout: available. 2024-02-01T10:01:16.823 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-92-generic 2024-02-01T10:01:16.823 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-01T10:01:16.823 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-01T10:01:16.823 INFO:teuthology.task.kernel:Installing distro kernel on c1.mon.a... 2024-02-01T10:01:16.823 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi148.front.sepia.ceph.com, path=None, version=distro) 2024-02-01T10:01:16.823 DEBUG:teuthology.orchestra.run.smithi148:> sudo apt-get clean 2024-02-01T10:01:17.013 DEBUG:teuthology.orchestra.run.smithi148:> sudo apt-get update 2024-02-01T10:01:17.057 INFO:teuthology.orchestra.run.smithi187.stdout:Reading package lists... 2024-02-01T10:01:17.075 DEBUG:teuthology.orchestra.run.smithi187:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-02-01T10:01:17.127 INFO:teuthology.orchestra.run.smithi187.stdout:Reading package lists... 2024-02-01T10:01:17.136 INFO:teuthology.orchestra.run.smithi148.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-02-01T10:01:17.155 INFO:teuthology.orchestra.run.smithi148.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-02-01T10:01:17.179 INFO:teuthology.orchestra.run.smithi148.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-02-01T10:01:17.243 INFO:teuthology.orchestra.run.smithi148.stdout:Hit:4 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-02-01T10:01:17.311 INFO:teuthology.orchestra.run.smithi187.stdout:Building dependency tree... 2024-02-01T10:01:17.311 INFO:teuthology.orchestra.run.smithi187.stdout:Reading state information... 2024-02-01T10:01:17.456 INFO:teuthology.orchestra.run.smithi187.stdout:linux-image-generic is already the newest version (5.15.0.92.89). 2024-02-01T10:01:17.456 INFO:teuthology.orchestra.run.smithi187.stdout:The following packages were automatically installed and are no longer required: 2024-02-01T10:01:17.457 INFO:teuthology.orchestra.run.smithi187.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-02-01T10:01:17.457 INFO:teuthology.orchestra.run.smithi187.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-02-01T10:01:17.457 INFO:teuthology.orchestra.run.smithi187.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-02-01T10:01:17.457 INFO:teuthology.orchestra.run.smithi187.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-02-01T10:01:17.457 INFO:teuthology.orchestra.run.smithi187.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-02-01T10:01:17.457 INFO:teuthology.orchestra.run.smithi187.stdout:Use 'sudo apt autoremove' to remove them. 2024-02-01T10:01:17.493 INFO:teuthology.orchestra.run.smithi187.stdout:0 upgraded, 0 newly installed, 0 to remove and 281 not upgraded. 2024-02-01T10:01:17.494 DEBUG:teuthology.orchestra.run.smithi187:> dpkg -s linux-image-generic 2024-02-01T10:01:17.553 INFO:teuthology.orchestra.run.smithi187.stdout:Package: linux-image-generic 2024-02-01T10:01:17.553 INFO:teuthology.orchestra.run.smithi187.stdout:Status: install ok installed 2024-02-01T10:01:17.554 INFO:teuthology.orchestra.run.smithi187.stdout:Priority: optional 2024-02-01T10:01:17.554 INFO:teuthology.orchestra.run.smithi187.stdout:Section: kernel 2024-02-01T10:01:17.554 INFO:teuthology.orchestra.run.smithi187.stdout:Installed-Size: 21 2024-02-01T10:01:17.554 INFO:teuthology.orchestra.run.smithi187.stdout:Maintainer: Ubuntu Kernel Team 2024-02-01T10:01:17.554 INFO:teuthology.orchestra.run.smithi187.stdout:Architecture: amd64 2024-02-01T10:01:17.554 INFO:teuthology.orchestra.run.smithi187.stdout:Source: linux-meta 2024-02-01T10:01:17.554 INFO:teuthology.orchestra.run.smithi187.stdout:Version: 5.15.0.92.89 2024-02-01T10:01:17.554 INFO:teuthology.orchestra.run.smithi187.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-92), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2024-02-01T10:01:17.554 INFO:teuthology.orchestra.run.smithi187.stdout:Depends: linux-image-5.15.0-92-generic, linux-modules-extra-5.15.0-92-generic, linux-firmware, intel-microcode, amd64-microcode 2024-02-01T10:01:17.554 INFO:teuthology.orchestra.run.smithi187.stdout:Recommends: thermald 2024-02-01T10:01:17.554 INFO:teuthology.orchestra.run.smithi187.stdout:Description: Generic Linux kernel image 2024-02-01T10:01:17.555 INFO:teuthology.orchestra.run.smithi187.stdout: This package will always depend on the latest generic kernel image 2024-02-01T10:01:17.555 INFO:teuthology.orchestra.run.smithi187.stdout: available. 2024-02-01T10:01:17.555 DEBUG:teuthology.orchestra.run.smithi187:> mktemp 2024-02-01T10:01:17.598 INFO:teuthology.orchestra.run.smithi187.stdout:/tmp/tmp.3H6SNRuIgM 2024-02-01T10:01:17.599 DEBUG:teuthology.orchestra.run.smithi187:> sudo cp /boot/grub/grub.cfg /tmp/tmp.3H6SNRuIgM 2024-02-01T10:01:17.655 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod 0666 /tmp/tmp.3H6SNRuIgM 2024-02-01T10:01:17.775 DEBUG:teuthology.orchestra.remote:smithi187:/tmp/tmp.3H6SNRuIgM is 10KB 2024-02-01T10:01:17.830 DEBUG:teuthology.orchestra.run.smithi187:> rm -fr /tmp/tmp.3H6SNRuIgM 2024-02-01T10:01:17.835 DEBUG:teuthology.orchestra.run.smithi187:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-02-01T10:01:17.891 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-02-01T10:01:17.891 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-02-01T10:01:17.891 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-02-01T10:01:17.962 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-92-generic 2024-02-01T10:01:17.962 DEBUG:teuthology.orchestra.run.smithi187:> sudo update-grub 2024-02-01T10:01:18.475 INFO:teuthology.orchestra.run.smithi148.stdout:Reading package lists... 2024-02-01T10:01:18.493 DEBUG:teuthology.orchestra.run.smithi148:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-02-01T10:01:18.553 INFO:teuthology.orchestra.run.smithi148.stdout:Reading package lists... 2024-02-01T10:01:18.748 INFO:teuthology.orchestra.run.smithi148.stdout:Building dependency tree... 2024-02-01T10:01:18.749 INFO:teuthology.orchestra.run.smithi148.stdout:Reading state information... 2024-02-01T10:01:18.901 INFO:teuthology.orchestra.run.smithi148.stdout:linux-image-generic is already the newest version (5.15.0.92.89). 2024-02-01T10:01:18.901 INFO:teuthology.orchestra.run.smithi148.stdout:The following packages were automatically installed and are no longer required: 2024-02-01T10:01:18.901 INFO:teuthology.orchestra.run.smithi148.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-02-01T10:01:18.901 INFO:teuthology.orchestra.run.smithi148.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-02-01T10:01:18.901 INFO:teuthology.orchestra.run.smithi148.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-02-01T10:01:18.902 INFO:teuthology.orchestra.run.smithi148.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-02-01T10:01:18.902 INFO:teuthology.orchestra.run.smithi148.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-02-01T10:01:18.902 INFO:teuthology.orchestra.run.smithi148.stdout:Use 'sudo apt autoremove' to remove them. 2024-02-01T10:01:18.938 INFO:teuthology.orchestra.run.smithi148.stdout:0 upgraded, 0 newly installed, 0 to remove and 281 not upgraded. 2024-02-01T10:01:18.939 DEBUG:teuthology.orchestra.run.smithi148:> dpkg -s linux-image-generic 2024-02-01T10:01:18.955 INFO:teuthology.orchestra.run.smithi148.stdout:Package: linux-image-generic 2024-02-01T10:01:18.955 INFO:teuthology.orchestra.run.smithi148.stdout:Status: install ok installed 2024-02-01T10:01:18.955 INFO:teuthology.orchestra.run.smithi148.stdout:Priority: optional 2024-02-01T10:01:18.955 INFO:teuthology.orchestra.run.smithi148.stdout:Section: kernel 2024-02-01T10:01:18.955 INFO:teuthology.orchestra.run.smithi148.stdout:Installed-Size: 21 2024-02-01T10:01:18.955 INFO:teuthology.orchestra.run.smithi148.stdout:Maintainer: Ubuntu Kernel Team 2024-02-01T10:01:18.955 INFO:teuthology.orchestra.run.smithi148.stdout:Architecture: amd64 2024-02-01T10:01:18.955 INFO:teuthology.orchestra.run.smithi148.stdout:Source: linux-meta 2024-02-01T10:01:18.956 INFO:teuthology.orchestra.run.smithi148.stdout:Version: 5.15.0.92.89 2024-02-01T10:01:18.956 INFO:teuthology.orchestra.run.smithi148.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-92), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2024-02-01T10:01:18.956 INFO:teuthology.orchestra.run.smithi148.stdout:Depends: linux-image-5.15.0-92-generic, linux-modules-extra-5.15.0-92-generic, linux-firmware, intel-microcode, amd64-microcode 2024-02-01T10:01:18.956 INFO:teuthology.orchestra.run.smithi148.stdout:Recommends: thermald 2024-02-01T10:01:18.956 INFO:teuthology.orchestra.run.smithi148.stdout:Description: Generic Linux kernel image 2024-02-01T10:01:18.956 INFO:teuthology.orchestra.run.smithi148.stdout: This package will always depend on the latest generic kernel image 2024-02-01T10:01:18.956 INFO:teuthology.orchestra.run.smithi148.stdout: available. 2024-02-01T10:01:18.956 DEBUG:teuthology.orchestra.run.smithi148:> mktemp 2024-02-01T10:01:19.001 INFO:teuthology.orchestra.run.smithi148.stdout:/tmp/tmp.EaWF2sLMrW 2024-02-01T10:01:19.002 DEBUG:teuthology.orchestra.run.smithi148:> sudo cp /boot/grub/grub.cfg /tmp/tmp.EaWF2sLMrW 2024-02-01T10:01:19.058 DEBUG:teuthology.orchestra.run.smithi148:> sudo chmod 0666 /tmp/tmp.EaWF2sLMrW 2024-02-01T10:01:19.183 DEBUG:teuthology.orchestra.remote:smithi148:/tmp/tmp.EaWF2sLMrW is 10KB 2024-02-01T10:01:19.236 DEBUG:teuthology.orchestra.run.smithi148:> rm -fr /tmp/tmp.EaWF2sLMrW 2024-02-01T10:01:19.243 DEBUG:teuthology.orchestra.run.smithi148:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-02-01T10:01:19.269 INFO:teuthology.orchestra.run.smithi187.stderr:Sourcing file `/etc/default/grub' 2024-02-01T10:01:19.270 INFO:teuthology.orchestra.run.smithi187.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-02-01T10:01:19.272 INFO:teuthology.orchestra.run.smithi187.stderr:Generating grub configuration file ... 2024-02-01T10:01:19.303 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-02-01T10:01:19.303 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-02-01T10:01:19.303 DEBUG:teuthology.orchestra.run.smithi148:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-02-01T10:01:19.370 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-92-generic 2024-02-01T10:01:19.370 DEBUG:teuthology.orchestra.run.smithi148:> sudo update-grub 2024-02-01T10:01:19.514 INFO:teuthology.orchestra.run.smithi187.stderr:Found linux image: /boot/vmlinuz-5.15.0-92-generic 2024-02-01T10:01:19.520 INFO:teuthology.orchestra.run.smithi187.stderr:Found initrd image: /boot/initrd.img-5.15.0-92-generic 2024-02-01T10:01:19.811 INFO:teuthology.orchestra.run.smithi187.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-02-01T10:01:19.816 INFO:teuthology.orchestra.run.smithi187.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-02-01T10:01:19.858 INFO:teuthology.orchestra.run.smithi187.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-02-01T10:01:19.864 INFO:teuthology.orchestra.run.smithi187.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-02-01T10:01:19.971 INFO:teuthology.orchestra.run.smithi187.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-02-01T10:01:19.972 INFO:teuthology.orchestra.run.smithi187.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-02-01T10:01:19.972 INFO:teuthology.orchestra.run.smithi187.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-02-01T10:01:19.992 INFO:teuthology.orchestra.run.smithi187.stderr:done 2024-02-01T10:01:19.994 DEBUG:teuthology.orchestra.run.smithi187:> sudo shutdown -r now 2024-02-01T10:01:20.574 INFO:teuthology.orchestra.run.smithi148.stderr:Sourcing file `/etc/default/grub' 2024-02-01T10:01:20.575 INFO:teuthology.orchestra.run.smithi148.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-02-01T10:01:20.579 INFO:teuthology.orchestra.run.smithi148.stderr:Generating grub configuration file ... 2024-02-01T10:01:20.766 INFO:teuthology.orchestra.run.smithi148.stderr:Found linux image: /boot/vmlinuz-5.15.0-92-generic 2024-02-01T10:01:20.773 INFO:teuthology.orchestra.run.smithi148.stderr:Found initrd image: /boot/initrd.img-5.15.0-92-generic 2024-02-01T10:01:21.004 INFO:teuthology.orchestra.run.smithi148.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-02-01T10:01:21.010 INFO:teuthology.orchestra.run.smithi148.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-02-01T10:01:21.052 INFO:teuthology.orchestra.run.smithi148.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-02-01T10:01:21.057 INFO:teuthology.orchestra.run.smithi148.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-02-01T10:01:21.191 INFO:teuthology.orchestra.run.smithi148.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-02-01T10:01:21.191 INFO:teuthology.orchestra.run.smithi148.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-02-01T10:01:21.192 INFO:teuthology.orchestra.run.smithi148.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-02-01T10:01:21.211 INFO:teuthology.orchestra.run.smithi148.stderr:done 2024-02-01T10:01:21.214 DEBUG:teuthology.orchestra.run.smithi148:> sudo shutdown -r now 2024-02-01T10:01:50.024 INFO:teuthology.task.kernel:Checking client c2.mon.a for new kernel version... 2024-02-01T10:01:50.025 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2024-02-01T10:01:50.025 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-01T10:01:51.218 INFO:teuthology.task.kernel:Checking client c1.mon.a for new kernel version... 2024-02-01T10:01:51.220 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi148.front.sepia.ceph.com' 2024-02-01T10:01:51.222 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-01T10:02:08.577 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.187 2024-02-01T10:02:09.600 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.148 2024-02-01T10:02:17.586 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2024-02-01T10:02:17.587 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-01T10:02:18.602 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi148.front.sepia.ceph.com' 2024-02-01T10:02:18.602 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-01T10:02:20.668 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.187 2024-02-01T10:02:21.660 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.148 2024-02-01T10:02:32.680 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2024-02-01T10:02:32.681 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-01T10:02:33.662 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi148.front.sepia.ceph.com' 2024-02-01T10:02:33.663 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-01T10:02:51.068 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.187 2024-02-01T10:02:52.092 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.148 2024-02-01T10:03:06.083 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2024-02-01T10:03:06.084 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-01T10:03:06.458 DEBUG:teuthology.orchestra.run.smithi187:> true 2024-02-01T10:03:07.046 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi187.front.sepia.ceph.com' 2024-02-01T10:03:07.046 INFO:teuthology.task.kernel:Checking kernel version of c2.mon.a, want "5.15.0-92-generic"... 2024-02-01T10:03:07.046 DEBUG:teuthology.orchestra.run.smithi187:> uname -r 2024-02-01T10:03:07.092 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi148.front.sepia.ceph.com' 2024-02-01T10:03:07.093 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-01T10:03:07.095 INFO:teuthology.orchestra.run.smithi187.stdout:5.15.0-92-generic 2024-02-01T10:03:07.096 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-92-generic vs 5.15.0-92-generic 2024-02-01T10:03:07.096 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-01T10:03:07.097 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-02-01T10:03:07.633 DEBUG:teuthology.orchestra.run.smithi148:> true 2024-02-01T10:03:08.098 INFO:teuthology.task.kernel:Enabling kdb on c2.mon.a... 2024-02-01T10:03:08.098 DEBUG:teuthology.orchestra.run.smithi187:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-01T10:03:08.438 INFO:teuthology.orchestra.run.smithi187.stdout:ttyS1 2024-02-01T10:03:08.459 DEBUG:teuthology.parallel:result is None 2024-02-01T10:03:08.610 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi148.front.sepia.ceph.com' 2024-02-01T10:03:08.611 INFO:teuthology.task.kernel:Checking kernel version of c1.mon.a, want "5.15.0-92-generic"... 2024-02-01T10:03:08.611 DEBUG:teuthology.orchestra.run.smithi148:> uname -r 2024-02-01T10:03:08.659 INFO:teuthology.orchestra.run.smithi148.stdout:5.15.0-92-generic 2024-02-01T10:03:08.660 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-92-generic vs 5.15.0-92-generic 2024-02-01T10:03:08.660 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-01T10:03:08.660 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-02-01T10:03:09.661 INFO:teuthology.task.kernel:Enabling kdb on c1.mon.a... 2024-02-01T10:03:09.661 DEBUG:teuthology.orchestra.run.smithi148:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-01T10:03:09.779 INFO:teuthology.orchestra.run.smithi148.stdout:ttyS1 2024-02-01T10:03:09.806 DEBUG:teuthology.parallel:result is None 2024-02-01T10:03:09.806 INFO:teuthology.run_tasks:Running task internal.base... 2024-02-01T10:03:09.812 INFO:teuthology.task.internal:Creating test directory... 2024-02-01T10:03:09.812 DEBUG:teuthology.orchestra.run.smithi148:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-01T10:03:09.814 DEBUG:teuthology.orchestra.run.smithi187:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-01T10:03:09.820 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-02-01T10:03:09.826 INFO:teuthology.run_tasks:Running task internal.archive... 2024-02-01T10:03:09.886 INFO:teuthology.task.internal:Creating archive directory... 2024-02-01T10:03:09.887 DEBUG:teuthology.orchestra.run.smithi148:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-01T10:03:09.889 DEBUG:teuthology.orchestra.run.smithi187:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-01T10:03:10.029 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-02-01T10:03:10.035 INFO:teuthology.task.internal:Enabling coredump saving... 2024-02-01T10:03:10.035 DEBUG:teuthology.orchestra.run.smithi148:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-01T10:03:10.038 DEBUG:teuthology.orchestra.run.smithi187:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-01T10:03:10.057 INFO:teuthology.orchestra.run.smithi187.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-01T10:03:10.065 INFO:teuthology.orchestra.run.smithi187.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-01T10:03:10.073 INFO:teuthology.orchestra.run.smithi148.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-01T10:03:10.080 INFO:teuthology.orchestra.run.smithi148.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-01T10:03:10.082 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-02-01T10:03:10.088 INFO:teuthology.task.internal:Configuring sudo... 2024-02-01T10:03:10.088 DEBUG:teuthology.orchestra.run.smithi148:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-01T10:03:10.125 DEBUG:teuthology.orchestra.run.smithi187:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-01T10:03:10.138 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-02-01T10:03:10.147 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-02-01T10:03:10.147 DEBUG:teuthology.orchestra.run.smithi148:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-01T10:03:10.177 DEBUG:teuthology.orchestra.run.smithi187:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-01T10:03:10.185 DEBUG:teuthology.orchestra.run.smithi148:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-01T10:03:10.228 DEBUG:teuthology.orchestra.run.smithi148:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-01T10:03:10.276 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-02-01T10:03:10.276 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-01T10:03:10.363 DEBUG:teuthology.orchestra.run.smithi187:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-01T10:03:10.369 DEBUG:teuthology.orchestra.run.smithi187:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-01T10:03:10.418 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-02-01T10:03:10.418 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-01T10:03:10.489 DEBUG:teuthology.orchestra.run.smithi148:> sudo service rsyslog restart 2024-02-01T10:03:10.491 DEBUG:teuthology.orchestra.run.smithi187:> sudo service rsyslog restart 2024-02-01T10:03:10.590 INFO:teuthology.run_tasks:Running task internal.timer... 2024-02-01T10:03:10.599 INFO:teuthology.task.internal:Starting timer... 2024-02-01T10:03:10.599 INFO:teuthology.run_tasks:Running task pcp... 2024-02-01T10:03:10.609 INFO:teuthology.run_tasks:Running task selinux... 2024-02-01T10:03:10.617 DEBUG:teuthology.task.selinux:Excluding smithi148: OS 'ubuntu' does not support SELinux 2024-02-01T10:03:10.617 DEBUG:teuthology.task.selinux:Excluding smithi187: OS 'ubuntu' does not support SELinux 2024-02-01T10:03:10.617 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-02-01T10:03:10.618 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-02-01T10:03:10.618 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-02-01T10:03:10.618 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-02-01T10:03:10.627 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-02-01T10:03:10.647 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-02-01T10:03:10.648 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi148.front.sepia.ceph.com,smithi187.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-02-01T10:10:00.746 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi148.front.sepia.ceph.com'), Remote(name='ubuntu@smithi187.front.sepia.ceph.com')] 2024-02-01T10:10:00.747 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi148.front.sepia.ceph.com' 2024-02-01T10:10:00.748 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-01T10:10:00.830 DEBUG:teuthology.orchestra.run.smithi148:> true 2024-02-01T10:10:00.908 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi148.front.sepia.ceph.com' 2024-02-01T10:10:00.909 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2024-02-01T10:10:00.909 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-01T10:10:01.006 DEBUG:teuthology.orchestra.run.smithi187:> true 2024-02-01T10:10:01.084 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi187.front.sepia.ceph.com' 2024-02-01T10:10:01.084 INFO:teuthology.run_tasks:Running task clock... 2024-02-01T10:10:01.096 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-02-01T10:10:01.097 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-01T10:10:01.097 DEBUG:teuthology.orchestra.run.smithi148:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-01T10:10:01.099 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-01T10:10:01.099 DEBUG:teuthology.orchestra.run.smithi187:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-01T10:10:01.127 INFO:teuthology.orchestra.run.smithi148.stdout: 1 Feb 10:10:01 ntpd[16883]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-02-01T10:10:01.127 INFO:teuthology.orchestra.run.smithi148.stdout: 1 Feb 10:10:01 ntpd[16883]: Command line: ntpd -gq 2024-02-01T10:10:01.127 INFO:teuthology.orchestra.run.smithi148.stdout: 1 Feb 10:10:01 ntpd[16883]: ---------------------------------------------------- 2024-02-01T10:10:01.127 INFO:teuthology.orchestra.run.smithi148.stdout: 1 Feb 10:10:01 ntpd[16883]: ntp-4 is maintained by Network Time Foundation, 2024-02-01T10:10:01.128 INFO:teuthology.orchestra.run.smithi148.stdout: 1 Feb 10:10:01 ntpd[16883]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-02-01T10:10:01.128 INFO:teuthology.orchestra.run.smithi148.stdout: 1 Feb 10:10:01 ntpd[16883]: corporation. Support and training for ntp-4 are 2024-02-01T10:10:01.128 INFO:teuthology.orchestra.run.smithi148.stdout: 1 Feb 10:10:01 ntpd[16883]: available at https://www.nwtime.org/support 2024-02-01T10:10:01.128 INFO:teuthology.orchestra.run.smithi148.stdout: 1 Feb 10:10:01 ntpd[16883]: ---------------------------------------------------- 2024-02-01T10:10:01.128 INFO:teuthology.orchestra.run.smithi148.stdout: 1 Feb 10:10:01 ntpd[16883]: proto: precision = 0.058 usec (-24) 2024-02-01T10:10:01.128 INFO:teuthology.orchestra.run.smithi148.stdout: 1 Feb 10:10:01 ntpd[16883]: basedate set to 2022-02-04 2024-02-01T10:10:01.128 INFO:teuthology.orchestra.run.smithi148.stdout: 1 Feb 10:10:01 ntpd[16883]: gps base set to 2022-02-06 (week 2196) 2024-02-01T10:10:01.128 INFO:teuthology.orchestra.run.smithi148.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-02-01T10:10:01.129 INFO:teuthology.orchestra.run.smithi148.stdout: 1 Feb 10:10:01 ntpd[16883]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-02-01T10:10:01.129 INFO:teuthology.orchestra.run.smithi148.stdout: 1 Feb 10:10:01 ntpd[16883]: restrict ::: KOD does nothing without LIMITED. 2024-02-01T10:10:01.129 INFO:teuthology.orchestra.run.smithi148.stderr:restrict ::: KOD does nothing without LIMITED. 2024-02-01T10:10:01.130 INFO:teuthology.orchestra.run.smithi148.stdout: 1 Feb 10:10:01 ntpd[16883]: Listen and drop on 0 v6wildcard [::]:123 2024-02-01T10:10:01.130 INFO:teuthology.orchestra.run.smithi148.stdout: 1 Feb 10:10:01 ntpd[16883]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-02-01T10:10:01.130 INFO:teuthology.orchestra.run.smithi148.stdout: 1 Feb 10:10:01 ntpd[16883]: Listen normally on 2 lo 127.0.0.1:123 2024-02-01T10:10:01.130 INFO:teuthology.orchestra.run.smithi148.stdout: 1 Feb 10:10:01 ntpd[16883]: Listen normally on 3 enp3s0f1 172.21.15.148:123 2024-02-01T10:10:01.130 INFO:teuthology.orchestra.run.smithi148.stdout: 1 Feb 10:10:01 ntpd[16883]: Listen normally on 4 lo [::1]:123 2024-02-01T10:10:01.130 INFO:teuthology.orchestra.run.smithi148.stdout: 1 Feb 10:10:01 ntpd[16883]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7257%5]:123 2024-02-01T10:10:01.130 INFO:teuthology.orchestra.run.smithi148.stdout: 1 Feb 10:10:01 ntpd[16883]: Listening on routing socket on fd #22 for interface updates 2024-02-01T10:10:01.152 INFO:teuthology.orchestra.run.smithi187.stdout: 1 Feb 10:10:01 ntpd[16883]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-02-01T10:10:01.152 INFO:teuthology.orchestra.run.smithi187.stdout: 1 Feb 10:10:01 ntpd[16883]: Command line: ntpd -gq 2024-02-01T10:10:01.153 INFO:teuthology.orchestra.run.smithi187.stdout: 1 Feb 10:10:01 ntpd[16883]: ---------------------------------------------------- 2024-02-01T10:10:01.153 INFO:teuthology.orchestra.run.smithi187.stdout: 1 Feb 10:10:01 ntpd[16883]: ntp-4 is maintained by Network Time Foundation, 2024-02-01T10:10:01.153 INFO:teuthology.orchestra.run.smithi187.stdout: 1 Feb 10:10:01 ntpd[16883]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-02-01T10:10:01.153 INFO:teuthology.orchestra.run.smithi187.stdout: 1 Feb 10:10:01 ntpd[16883]: corporation. Support and training for ntp-4 are 2024-02-01T10:10:01.153 INFO:teuthology.orchestra.run.smithi187.stdout: 1 Feb 10:10:01 ntpd[16883]: available at https://www.nwtime.org/support 2024-02-01T10:10:01.153 INFO:teuthology.orchestra.run.smithi187.stdout: 1 Feb 10:10:01 ntpd[16883]: ---------------------------------------------------- 2024-02-01T10:10:01.153 INFO:teuthology.orchestra.run.smithi187.stdout: 1 Feb 10:10:01 ntpd[16883]: proto: precision = 0.057 usec (-24) 2024-02-01T10:10:01.153 INFO:teuthology.orchestra.run.smithi187.stdout: 1 Feb 10:10:01 ntpd[16883]: basedate set to 2022-02-04 2024-02-01T10:10:01.154 INFO:teuthology.orchestra.run.smithi187.stdout: 1 Feb 10:10:01 ntpd[16883]: gps base set to 2022-02-06 (week 2196) 2024-02-01T10:10:01.154 INFO:teuthology.orchestra.run.smithi187.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-02-01T10:10:01.154 INFO:teuthology.orchestra.run.smithi187.stdout: 1 Feb 10:10:01 ntpd[16883]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-02-01T10:10:01.154 INFO:teuthology.orchestra.run.smithi187.stdout: 1 Feb 10:10:01 ntpd[16883]: restrict ::: KOD does nothing without LIMITED. 2024-02-01T10:10:01.154 INFO:teuthology.orchestra.run.smithi187.stderr:restrict ::: KOD does nothing without LIMITED. 2024-02-01T10:10:01.155 INFO:teuthology.orchestra.run.smithi187.stdout: 1 Feb 10:10:01 ntpd[16883]: Listen and drop on 0 v6wildcard [::]:123 2024-02-01T10:10:01.155 INFO:teuthology.orchestra.run.smithi187.stdout: 1 Feb 10:10:01 ntpd[16883]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-02-01T10:10:01.156 INFO:teuthology.orchestra.run.smithi187.stdout: 1 Feb 10:10:01 ntpd[16883]: Listen normally on 2 lo 127.0.0.1:123 2024-02-01T10:10:01.156 INFO:teuthology.orchestra.run.smithi187.stdout: 1 Feb 10:10:01 ntpd[16883]: Listen normally on 3 enp3s0f1 172.21.15.187:123 2024-02-01T10:10:01.156 INFO:teuthology.orchestra.run.smithi187.stdout: 1 Feb 10:10:01 ntpd[16883]: Listen normally on 4 lo [::1]:123 2024-02-01T10:10:01.156 INFO:teuthology.orchestra.run.smithi187.stdout: 1 Feb 10:10:01 ntpd[16883]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:77b1%5]:123 2024-02-01T10:10:01.156 INFO:teuthology.orchestra.run.smithi187.stdout: 1 Feb 10:10:01 ntpd[16883]: Listening on routing socket on fd #22 for interface updates 2024-02-01T10:10:02.128 INFO:teuthology.orchestra.run.smithi148.stderr: 1 Feb 10:10:02 ntpd[16883]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-01T10:10:02.128 INFO:teuthology.orchestra.run.smithi148.stderr: 1 Feb 10:10:02 ntpd[16883]: can't open /var/log/ntpstats/rawstats.20240201: Permission denied 2024-02-01T10:10:02.128 INFO:teuthology.orchestra.run.smithi148.stderr: 1 Feb 10:10:02 ntpd[16883]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-01T10:10:02.128 INFO:teuthology.orchestra.run.smithi148.stderr: 1 Feb 10:10:02 ntpd[16883]: can't open /var/log/ntpstats/rawstats.20240201: Permission denied 2024-02-01T10:10:02.128 INFO:teuthology.orchestra.run.smithi148.stderr: 1 Feb 10:10:02 ntpd[16883]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-01T10:10:02.128 INFO:teuthology.orchestra.run.smithi148.stderr: 1 Feb 10:10:02 ntpd[16883]: can't open /var/log/ntpstats/rawstats.20240201: Permission denied 2024-02-01T10:10:02.129 INFO:teuthology.orchestra.run.smithi148.stderr: 1 Feb 10:10:02 ntpd[16883]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-01T10:10:02.129 INFO:teuthology.orchestra.run.smithi148.stderr: 1 Feb 10:10:02 ntpd[16883]: can't open /var/log/ntpstats/peerstats.20240201: Permission denied 2024-02-01T10:10:02.152 INFO:teuthology.orchestra.run.smithi187.stderr: 1 Feb 10:10:02 ntpd[16883]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-01T10:10:02.153 INFO:teuthology.orchestra.run.smithi187.stderr: 1 Feb 10:10:02 ntpd[16883]: can't open /var/log/ntpstats/rawstats.20240201: Permission denied 2024-02-01T10:10:02.153 INFO:teuthology.orchestra.run.smithi187.stderr: 1 Feb 10:10:02 ntpd[16883]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-01T10:10:02.153 INFO:teuthology.orchestra.run.smithi187.stderr: 1 Feb 10:10:02 ntpd[16883]: can't open /var/log/ntpstats/rawstats.20240201: Permission denied 2024-02-01T10:10:02.153 INFO:teuthology.orchestra.run.smithi187.stderr: 1 Feb 10:10:02 ntpd[16883]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-01T10:10:02.153 INFO:teuthology.orchestra.run.smithi187.stderr: 1 Feb 10:10:02 ntpd[16883]: can't open /var/log/ntpstats/peerstats.20240201: Permission denied 2024-02-01T10:10:03.153 INFO:teuthology.orchestra.run.smithi187.stderr: 1 Feb 10:10:03 ntpd[16883]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-01T10:10:03.153 INFO:teuthology.orchestra.run.smithi187.stderr: 1 Feb 10:10:03 ntpd[16883]: can't open /var/log/ntpstats/rawstats.20240201: Permission denied 2024-02-01T10:10:03.153 INFO:teuthology.orchestra.run.smithi187.stderr: 1 Feb 10:10:03 ntpd[16883]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-01T10:10:03.153 INFO:teuthology.orchestra.run.smithi187.stderr: 1 Feb 10:10:03 ntpd[16883]: can't open /var/log/ntpstats/rawstats.20240201: Permission denied 2024-02-01T10:10:03.153 INFO:teuthology.orchestra.run.smithi187.stderr: 1 Feb 10:10:03 ntpd[16883]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-01T10:10:03.153 INFO:teuthology.orchestra.run.smithi187.stderr: 1 Feb 10:10:03 ntpd[16883]: can't open /var/log/ntpstats/peerstats.20240201: Permission denied 2024-02-01T10:10:04.129 INFO:teuthology.orchestra.run.smithi148.stderr: 1 Feb 10:10:04 ntpd[16883]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-01T10:10:04.129 INFO:teuthology.orchestra.run.smithi148.stderr: 1 Feb 10:10:04 ntpd[16883]: can't open /var/log/ntpstats/rawstats.20240201: Permission denied 2024-02-01T10:10:04.129 INFO:teuthology.orchestra.run.smithi148.stderr: 1 Feb 10:10:04 ntpd[16883]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-01T10:10:04.129 INFO:teuthology.orchestra.run.smithi148.stderr: 1 Feb 10:10:04 ntpd[16883]: can't open /var/log/ntpstats/rawstats.20240201: Permission denied 2024-02-01T10:10:04.129 INFO:teuthology.orchestra.run.smithi148.stderr: 1 Feb 10:10:04 ntpd[16883]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-01T10:10:04.130 INFO:teuthology.orchestra.run.smithi148.stderr: 1 Feb 10:10:04 ntpd[16883]: can't open /var/log/ntpstats/rawstats.20240201: Permission denied 2024-02-01T10:10:04.130 INFO:teuthology.orchestra.run.smithi148.stderr: 1 Feb 10:10:04 ntpd[16883]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-01T10:10:04.130 INFO:teuthology.orchestra.run.smithi148.stderr: 1 Feb 10:10:04 ntpd[16883]: can't open /var/log/ntpstats/rawstats.20240201: Permission denied 2024-02-01T10:10:04.130 INFO:teuthology.orchestra.run.smithi148.stderr: 1 Feb 10:10:04 ntpd[16883]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-01T10:10:04.130 INFO:teuthology.orchestra.run.smithi148.stderr: 1 Feb 10:10:04 ntpd[16883]: can't open /var/log/ntpstats/peerstats.20240201: Permission denied 2024-02-01T10:10:04.130 INFO:teuthology.orchestra.run.smithi148.stderr: 1 Feb 10:10:04 ntpd[16883]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-01T10:10:04.130 INFO:teuthology.orchestra.run.smithi148.stderr: 1 Feb 10:10:04 ntpd[16883]: can't open /var/log/ntpstats/rawstats.20240201: Permission denied 2024-02-01T10:10:04.130 INFO:teuthology.orchestra.run.smithi148.stderr: 1 Feb 10:10:04 ntpd[16883]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-01T10:10:04.130 INFO:teuthology.orchestra.run.smithi148.stderr: 1 Feb 10:10:04 ntpd[16883]: can't open /var/log/ntpstats/peerstats.20240201: Permission denied 2024-02-01T10:10:04.130 INFO:teuthology.orchestra.run.smithi148.stderr: 1 Feb 10:10:04 ntpd[16883]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-01T10:10:04.130 INFO:teuthology.orchestra.run.smithi148.stderr: 1 Feb 10:10:04 ntpd[16883]: can't open /var/log/ntpstats/rawstats.20240201: Permission denied 2024-02-01T10:10:04.130 INFO:teuthology.orchestra.run.smithi148.stderr: 1 Feb 10:10:04 ntpd[16883]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-01T10:10:04.130 INFO:teuthology.orchestra.run.smithi148.stderr: 1 Feb 10:10:04 ntpd[16883]: can't open /var/log/ntpstats/peerstats.20240201: Permission denied 2024-02-01T10:10:04.153 INFO:teuthology.orchestra.run.smithi187.stderr: 1 Feb 10:10:04 ntpd[16883]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-01T10:10:04.153 INFO:teuthology.orchestra.run.smithi187.stderr: 1 Feb 10:10:04 ntpd[16883]: can't open /var/log/ntpstats/rawstats.20240201: Permission denied 2024-02-01T10:10:04.153 INFO:teuthology.orchestra.run.smithi187.stderr: 1 Feb 10:10:04 ntpd[16883]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-01T10:10:04.154 INFO:teuthology.orchestra.run.smithi187.stderr: 1 Feb 10:10:04 ntpd[16883]: can't open /var/log/ntpstats/rawstats.20240201: Permission denied 2024-02-01T10:10:04.154 INFO:teuthology.orchestra.run.smithi187.stderr: 1 Feb 10:10:04 ntpd[16883]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-01T10:10:04.154 INFO:teuthology.orchestra.run.smithi187.stderr: 1 Feb 10:10:04 ntpd[16883]: can't open /var/log/ntpstats/rawstats.20240201: Permission denied 2024-02-01T10:10:04.154 INFO:teuthology.orchestra.run.smithi187.stderr: 1 Feb 10:10:04 ntpd[16883]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-01T10:10:04.154 INFO:teuthology.orchestra.run.smithi187.stderr: 1 Feb 10:10:04 ntpd[16883]: can't open /var/log/ntpstats/peerstats.20240201: Permission denied 2024-02-01T10:10:04.154 INFO:teuthology.orchestra.run.smithi187.stderr: 1 Feb 10:10:04 ntpd[16883]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-01T10:10:04.154 INFO:teuthology.orchestra.run.smithi187.stderr: 1 Feb 10:10:04 ntpd[16883]: can't open /var/log/ntpstats/rawstats.20240201: Permission denied 2024-02-01T10:10:04.154 INFO:teuthology.orchestra.run.smithi187.stderr: 1 Feb 10:10:04 ntpd[16883]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-01T10:10:04.154 INFO:teuthology.orchestra.run.smithi187.stderr: 1 Feb 10:10:04 ntpd[16883]: can't open /var/log/ntpstats/peerstats.20240201: Permission denied 2024-02-01T10:10:05.153 INFO:teuthology.orchestra.run.smithi187.stderr: 1 Feb 10:10:05 ntpd[16883]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-01T10:10:05.153 INFO:teuthology.orchestra.run.smithi187.stderr: 1 Feb 10:10:05 ntpd[16883]: can't open /var/log/ntpstats/rawstats.20240201: Permission denied 2024-02-01T10:10:05.153 INFO:teuthology.orchestra.run.smithi187.stderr: 1 Feb 10:10:05 ntpd[16883]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-01T10:10:05.153 INFO:teuthology.orchestra.run.smithi187.stderr: 1 Feb 10:10:05 ntpd[16883]: can't open /var/log/ntpstats/rawstats.20240201: Permission denied 2024-02-01T10:10:05.153 INFO:teuthology.orchestra.run.smithi187.stderr: 1 Feb 10:10:05 ntpd[16883]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-01T10:10:05.153 INFO:teuthology.orchestra.run.smithi187.stderr: 1 Feb 10:10:05 ntpd[16883]: can't open /var/log/ntpstats/peerstats.20240201: Permission denied 2024-02-01T10:10:06.129 INFO:teuthology.orchestra.run.smithi148.stderr: 1 Feb 10:10:06 ntpd[16883]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-01T10:10:06.129 INFO:teuthology.orchestra.run.smithi148.stderr: 1 Feb 10:10:06 ntpd[16883]: can't open /var/log/ntpstats/rawstats.20240201: Permission denied 2024-02-01T10:10:06.129 INFO:teuthology.orchestra.run.smithi148.stderr: 1 Feb 10:10:06 ntpd[16883]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-01T10:10:06.129 INFO:teuthology.orchestra.run.smithi148.stderr: 1 Feb 10:10:06 ntpd[16883]: can't open /var/log/ntpstats/rawstats.20240201: Permission denied 2024-02-01T10:10:06.129 INFO:teuthology.orchestra.run.smithi148.stderr: 1 Feb 10:10:06 ntpd[16883]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-01T10:10:06.129 INFO:teuthology.orchestra.run.smithi148.stderr: 1 Feb 10:10:06 ntpd[16883]: can't open /var/log/ntpstats/rawstats.20240201: Permission denied 2024-02-01T10:10:06.129 INFO:teuthology.orchestra.run.smithi148.stderr: 1 Feb 10:10:06 ntpd[16883]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-01T10:10:06.129 INFO:teuthology.orchestra.run.smithi148.stderr: 1 Feb 10:10:06 ntpd[16883]: can't open /var/log/ntpstats/rawstats.20240201: Permission denied 2024-02-01T10:10:06.129 INFO:teuthology.orchestra.run.smithi148.stderr: 1 Feb 10:10:06 ntpd[16883]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-01T10:10:06.129 INFO:teuthology.orchestra.run.smithi148.stderr: 1 Feb 10:10:06 ntpd[16883]: can't open /var/log/ntpstats/peerstats.20240201: Permission denied 2024-02-01T10:10:06.129 INFO:teuthology.orchestra.run.smithi148.stderr: 1 Feb 10:10:06 ntpd[16883]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-01T10:10:06.130 INFO:teuthology.orchestra.run.smithi148.stderr: 1 Feb 10:10:06 ntpd[16883]: can't open /var/log/ntpstats/rawstats.20240201: Permission denied 2024-02-01T10:10:06.130 INFO:teuthology.orchestra.run.smithi148.stderr: 1 Feb 10:10:06 ntpd[16883]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-01T10:10:06.130 INFO:teuthology.orchestra.run.smithi148.stderr: 1 Feb 10:10:06 ntpd[16883]: can't open /var/log/ntpstats/peerstats.20240201: Permission denied 2024-02-01T10:10:06.130 INFO:teuthology.orchestra.run.smithi148.stderr: 1 Feb 10:10:06 ntpd[16883]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-01T10:10:06.130 INFO:teuthology.orchestra.run.smithi148.stderr: 1 Feb 10:10:06 ntpd[16883]: can't open /var/log/ntpstats/rawstats.20240201: Permission denied 2024-02-01T10:10:06.130 INFO:teuthology.orchestra.run.smithi148.stderr: 1 Feb 10:10:06 ntpd[16883]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-01T10:10:06.130 INFO:teuthology.orchestra.run.smithi148.stderr: 1 Feb 10:10:06 ntpd[16883]: can't open /var/log/ntpstats/peerstats.20240201: Permission denied 2024-02-01T10:10:06.153 INFO:teuthology.orchestra.run.smithi187.stderr: 1 Feb 10:10:06 ntpd[16883]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-01T10:10:06.154 INFO:teuthology.orchestra.run.smithi187.stderr: 1 Feb 10:10:06 ntpd[16883]: can't open /var/log/ntpstats/rawstats.20240201: Permission denied 2024-02-01T10:10:06.154 INFO:teuthology.orchestra.run.smithi187.stderr: 1 Feb 10:10:06 ntpd[16883]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-01T10:10:06.154 INFO:teuthology.orchestra.run.smithi187.stderr: 1 Feb 10:10:06 ntpd[16883]: can't open /var/log/ntpstats/rawstats.20240201: Permission denied 2024-02-01T10:10:06.154 INFO:teuthology.orchestra.run.smithi187.stderr: 1 Feb 10:10:06 ntpd[16883]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-01T10:10:06.154 INFO:teuthology.orchestra.run.smithi187.stderr: 1 Feb 10:10:06 ntpd[16883]: can't open /var/log/ntpstats/rawstats.20240201: Permission denied 2024-02-01T10:10:06.154 INFO:teuthology.orchestra.run.smithi187.stderr: 1 Feb 10:10:06 ntpd[16883]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-01T10:10:06.154 INFO:teuthology.orchestra.run.smithi187.stderr: 1 Feb 10:10:06 ntpd[16883]: can't open /var/log/ntpstats/rawstats.20240201: Permission denied 2024-02-01T10:10:06.154 INFO:teuthology.orchestra.run.smithi187.stderr: 1 Feb 10:10:06 ntpd[16883]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-01T10:10:06.154 INFO:teuthology.orchestra.run.smithi187.stderr: 1 Feb 10:10:06 ntpd[16883]: can't open /var/log/ntpstats/peerstats.20240201: Permission denied 2024-02-01T10:10:06.154 INFO:teuthology.orchestra.run.smithi187.stderr: 1 Feb 10:10:06 ntpd[16883]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-01T10:10:06.154 INFO:teuthology.orchestra.run.smithi187.stderr: 1 Feb 10:10:06 ntpd[16883]: can't open /var/log/ntpstats/rawstats.20240201: Permission denied 2024-02-01T10:10:06.154 INFO:teuthology.orchestra.run.smithi187.stderr: 1 Feb 10:10:06 ntpd[16883]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-01T10:10:06.154 INFO:teuthology.orchestra.run.smithi187.stderr: 1 Feb 10:10:06 ntpd[16883]: can't open /var/log/ntpstats/peerstats.20240201: Permission denied 2024-02-01T10:10:07.153 INFO:teuthology.orchestra.run.smithi187.stderr: 1 Feb 10:10:07 ntpd[16883]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-01T10:10:07.153 INFO:teuthology.orchestra.run.smithi187.stderr: 1 Feb 10:10:07 ntpd[16883]: can't open /var/log/ntpstats/rawstats.20240201: Permission denied 2024-02-01T10:10:07.153 INFO:teuthology.orchestra.run.smithi187.stderr: 1 Feb 10:10:07 ntpd[16883]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-01T10:10:07.153 INFO:teuthology.orchestra.run.smithi187.stderr: 1 Feb 10:10:07 ntpd[16883]: can't open /var/log/ntpstats/rawstats.20240201: Permission denied 2024-02-01T10:10:07.153 INFO:teuthology.orchestra.run.smithi187.stderr: 1 Feb 10:10:07 ntpd[16883]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-01T10:10:07.153 INFO:teuthology.orchestra.run.smithi187.stderr: 1 Feb 10:10:07 ntpd[16883]: can't open /var/log/ntpstats/peerstats.20240201: Permission denied 2024-02-01T10:10:08.128 INFO:teuthology.orchestra.run.smithi148.stderr: 1 Feb 10:10:08 ntpd[16883]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-01T10:10:08.129 INFO:teuthology.orchestra.run.smithi148.stderr: 1 Feb 10:10:08 ntpd[16883]: can't open /var/log/ntpstats/rawstats.20240201: Permission denied 2024-02-01T10:10:08.129 INFO:teuthology.orchestra.run.smithi148.stderr: 1 Feb 10:10:08 ntpd[16883]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-01T10:10:08.129 INFO:teuthology.orchestra.run.smithi148.stderr: 1 Feb 10:10:08 ntpd[16883]: can't open /var/log/ntpstats/rawstats.20240201: Permission denied 2024-02-01T10:10:08.129 INFO:teuthology.orchestra.run.smithi148.stderr: 1 Feb 10:10:08 ntpd[16883]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-01T10:10:08.129 INFO:teuthology.orchestra.run.smithi148.stderr: 1 Feb 10:10:08 ntpd[16883]: can't open /var/log/ntpstats/rawstats.20240201: Permission denied 2024-02-01T10:10:08.129 INFO:teuthology.orchestra.run.smithi148.stderr: 1 Feb 10:10:08 ntpd[16883]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-01T10:10:08.129 INFO:teuthology.orchestra.run.smithi148.stderr: 1 Feb 10:10:08 ntpd[16883]: can't open /var/log/ntpstats/rawstats.20240201: Permission denied 2024-02-01T10:10:08.129 INFO:teuthology.orchestra.run.smithi148.stderr: 1 Feb 10:10:08 ntpd[16883]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-01T10:10:08.129 INFO:teuthology.orchestra.run.smithi148.stderr: 1 Feb 10:10:08 ntpd[16883]: can't open /var/log/ntpstats/peerstats.20240201: Permission denied 2024-02-01T10:10:08.129 INFO:teuthology.orchestra.run.smithi148.stderr: 1 Feb 10:10:08 ntpd[16883]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-01T10:10:08.129 INFO:teuthology.orchestra.run.smithi148.stderr: 1 Feb 10:10:08 ntpd[16883]: can't open /var/log/ntpstats/rawstats.20240201: Permission denied 2024-02-01T10:10:08.129 INFO:teuthology.orchestra.run.smithi148.stderr: 1 Feb 10:10:08 ntpd[16883]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-01T10:10:08.129 INFO:teuthology.orchestra.run.smithi148.stderr: 1 Feb 10:10:08 ntpd[16883]: can't open /var/log/ntpstats/peerstats.20240201: Permission denied 2024-02-01T10:10:08.130 INFO:teuthology.orchestra.run.smithi148.stderr: 1 Feb 10:10:08 ntpd[16883]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-01T10:10:08.130 INFO:teuthology.orchestra.run.smithi148.stderr: 1 Feb 10:10:08 ntpd[16883]: can't open /var/log/ntpstats/rawstats.20240201: Permission denied 2024-02-01T10:10:08.130 INFO:teuthology.orchestra.run.smithi148.stderr: 1 Feb 10:10:08 ntpd[16883]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-01T10:10:08.130 INFO:teuthology.orchestra.run.smithi148.stderr: 1 Feb 10:10:08 ntpd[16883]: can't open /var/log/ntpstats/peerstats.20240201: Permission denied 2024-02-01T10:10:08.130 INFO:teuthology.orchestra.run.smithi148.stdout: 1 Feb 10:10:08 ntpd[16883]: ntpd: time slew -0.002757 s 2024-02-01T10:10:08.130 INFO:teuthology.orchestra.run.smithi148.stdout:ntpd: time slew -0.002757s 2024-02-01T10:10:08.130 INFO:teuthology.orchestra.run.smithi148.stderr: 1 Feb 10:10:08 ntpd[16883]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-02-01T10:10:08.131 INFO:teuthology.orchestra.run.smithi148.stderr: 1 Feb 10:10:08 ntpd[16883]: can't open /var/log/ntpstats/loopstats.20240201: Permission denied 2024-02-01T10:10:08.154 INFO:teuthology.orchestra.run.smithi187.stderr: 1 Feb 10:10:08 ntpd[16883]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-01T10:10:08.154 INFO:teuthology.orchestra.run.smithi187.stderr: 1 Feb 10:10:08 ntpd[16883]: can't open /var/log/ntpstats/rawstats.20240201: Permission denied 2024-02-01T10:10:08.155 INFO:teuthology.orchestra.run.smithi187.stderr: 1 Feb 10:10:08 ntpd[16883]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-01T10:10:08.155 INFO:teuthology.orchestra.run.smithi187.stderr: 1 Feb 10:10:08 ntpd[16883]: can't open /var/log/ntpstats/rawstats.20240201: Permission denied 2024-02-01T10:10:08.155 INFO:teuthology.orchestra.run.smithi187.stderr: 1 Feb 10:10:08 ntpd[16883]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-01T10:10:08.155 INFO:teuthology.orchestra.run.smithi187.stderr: 1 Feb 10:10:08 ntpd[16883]: can't open /var/log/ntpstats/rawstats.20240201: Permission denied 2024-02-01T10:10:08.155 INFO:teuthology.orchestra.run.smithi187.stderr: 1 Feb 10:10:08 ntpd[16883]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-01T10:10:08.155 INFO:teuthology.orchestra.run.smithi187.stderr: 1 Feb 10:10:08 ntpd[16883]: can't open /var/log/ntpstats/peerstats.20240201: Permission denied 2024-02-01T10:10:08.155 INFO:teuthology.orchestra.run.smithi187.stderr: 1 Feb 10:10:08 ntpd[16883]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-01T10:10:08.155 INFO:teuthology.orchestra.run.smithi187.stderr: 1 Feb 10:10:08 ntpd[16883]: can't open /var/log/ntpstats/rawstats.20240201: Permission denied 2024-02-01T10:10:08.155 INFO:teuthology.orchestra.run.smithi187.stderr: 1 Feb 10:10:08 ntpd[16883]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-01T10:10:08.155 INFO:teuthology.orchestra.run.smithi187.stderr: 1 Feb 10:10:08 ntpd[16883]: can't open /var/log/ntpstats/peerstats.20240201: Permission denied 2024-02-01T10:10:08.155 INFO:teuthology.orchestra.run.smithi187.stdout: 1 Feb 10:10:08 ntpd[16883]: ntpd: time slew -0.001301 s 2024-02-01T10:10:08.156 INFO:teuthology.orchestra.run.smithi187.stdout:ntpd: time slew -0.001301s 2024-02-01T10:10:08.156 INFO:teuthology.orchestra.run.smithi187.stderr: 1 Feb 10:10:08 ntpd[16883]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-02-01T10:10:08.156 INFO:teuthology.orchestra.run.smithi187.stderr: 1 Feb 10:10:08 ntpd[16883]: can't open /var/log/ntpstats/loopstats.20240201: Permission denied 2024-02-01T10:10:08.208 INFO:teuthology.orchestra.run.smithi187.stdout: remote refid st t when poll reach delay offset jitter 2024-02-01T10:10:08.208 INFO:teuthology.orchestra.run.smithi187.stdout:============================================================================== 2024-02-01T10:10:08.208 INFO:teuthology.orchestra.run.smithi187.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-02-01T10:10:08.208 INFO:teuthology.orchestra.run.smithi187.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-02-01T10:10:08.208 INFO:teuthology.orchestra.run.smithi187.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-02-01T10:10:08.208 INFO:teuthology.orchestra.run.smithi187.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-02-01T10:10:08.235 INFO:teuthology.orchestra.run.smithi148.stdout: remote refid st t when poll reach delay offset jitter 2024-02-01T10:10:08.235 INFO:teuthology.orchestra.run.smithi148.stdout:============================================================================== 2024-02-01T10:10:08.235 INFO:teuthology.orchestra.run.smithi148.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-02-01T10:10:08.235 INFO:teuthology.orchestra.run.smithi148.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-02-01T10:10:08.235 INFO:teuthology.orchestra.run.smithi148.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-02-01T10:10:08.236 INFO:teuthology.orchestra.run.smithi148.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-02-01T10:10:08.236 INFO:teuthology.run_tasks:Running task install... 2024-02-01T10:10:08.245 DEBUG:teuthology.task.install:project ceph 2024-02-01T10:10:08.245 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24'}} 2024-02-01T10:10:08.245 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24'} 2024-02-01T10:10:08.245 INFO:teuthology.task.install:Using flavor: default 2024-02-01T10:10:08.251 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-02-01T10:10:08.251 INFO:teuthology.task.install:extra packages: [] 2024-02-01T10:10:08.252 DEBUG:teuthology.orchestra.run.smithi148:> sudo apt-key list | grep Ceph 2024-02-01T10:10:08.253 DEBUG:teuthology.orchestra.run.smithi187:> sudo apt-key list | grep Ceph 2024-02-01T10:10:08.307 INFO:teuthology.orchestra.run.smithi187.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-02-01T10:10:08.335 INFO:teuthology.orchestra.run.smithi187.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-02-01T10:10:08.335 INFO:teuthology.orchestra.run.smithi187.stdout:uid [ unknown] Ceph.com (release key) 2024-02-01T10:10:08.335 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-02-01T10:10:08.336 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24 2024-02-01T10:10:08.340 INFO:teuthology.orchestra.run.smithi148.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-02-01T10:10:08.361 INFO:teuthology.orchestra.run.smithi148.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-02-01T10:10:08.361 INFO:teuthology.orchestra.run.smithi148.stdout:uid [ unknown] Ceph.com (release key) 2024-02-01T10:10:08.361 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-02-01T10:10:08.361 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24 2024-02-01T10:10:08.481 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/wip-end2end-tracing/5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/ubuntu/jammy/flavors/default/ 2024-02-01T10:10:08.481 INFO:teuthology.task.install.deb:Package version is 19.0.0-931-g5a7faa11-1jammy 2024-02-01T10:10:08.582 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/wip-end2end-tracing/5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/ubuntu/jammy/flavors/default/ 2024-02-01T10:10:08.582 INFO:teuthology.task.install.deb:Package version is 19.0.0-931-g5a7faa11-1jammy 2024-02-01T10:10:08.650 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-02-01T10:10:08.650 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-02-01T10:10:08.664 DEBUG:teuthology.orchestra.run.smithi187:> sudo apt-get update 2024-02-01T10:10:08.691 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-02-01T10:10:08.692 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-02-01T10:10:08.706 DEBUG:teuthology.orchestra.run.smithi148:> sudo apt-get update 2024-02-01T10:10:08.840 INFO:teuthology.orchestra.run.smithi187.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-02-01T10:10:08.850 INFO:teuthology.orchestra.run.smithi187.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-02-01T10:10:08.869 INFO:teuthology.orchestra.run.smithi187.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-02-01T10:10:08.874 INFO:teuthology.orchestra.run.smithi148.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-02-01T10:10:08.874 INFO:teuthology.orchestra.run.smithi148.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-02-01T10:10:08.887 INFO:teuthology.orchestra.run.smithi148.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-02-01T10:10:08.888 INFO:teuthology.orchestra.run.smithi187.stdout:Ign:4 https://4.chacra.ceph.com/r/ceph/wip-end2end-tracing/5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/ubuntu/jammy/flavors/default jammy InRelease 2024-02-01T10:10:08.907 INFO:teuthology.orchestra.run.smithi148.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-02-01T10:10:08.910 INFO:teuthology.orchestra.run.smithi187.stdout:Get:5 https://4.chacra.ceph.com/r/ceph/wip-end2end-tracing/5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/ubuntu/jammy/flavors/default jammy Release [6,978 B] 2024-02-01T10:10:08.924 INFO:teuthology.orchestra.run.smithi148.stdout:Ign:5 https://4.chacra.ceph.com/r/ceph/wip-end2end-tracing/5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/ubuntu/jammy/flavors/default jammy InRelease 2024-02-01T10:10:08.932 INFO:teuthology.orchestra.run.smithi187.stdout:Ign:6 https://4.chacra.ceph.com/r/ceph/wip-end2end-tracing/5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/ubuntu/jammy/flavors/default jammy Release.gpg 2024-02-01T10:10:08.950 INFO:teuthology.orchestra.run.smithi148.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/wip-end2end-tracing/5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/ubuntu/jammy/flavors/default jammy Release [6,978 B] 2024-02-01T10:10:08.955 INFO:teuthology.orchestra.run.smithi187.stdout:Hit:7 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-02-01T10:10:08.963 INFO:teuthology.orchestra.run.smithi187.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/wip-end2end-tracing/5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,399 B] 2024-02-01T10:10:08.972 INFO:teuthology.orchestra.run.smithi148.stdout:Ign:7 https://4.chacra.ceph.com/r/ceph/wip-end2end-tracing/5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/ubuntu/jammy/flavors/default jammy Release.gpg 2024-02-01T10:10:09.010 INFO:teuthology.orchestra.run.smithi148.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/wip-end2end-tracing/5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,399 B] 2024-02-01T10:10:09.010 INFO:teuthology.orchestra.run.smithi187.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/wip-end2end-tracing/5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.8 kB] 2024-02-01T10:10:09.040 INFO:teuthology.orchestra.run.smithi148.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/wip-end2end-tracing/5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.8 kB] 2024-02-01T10:10:09.192 INFO:teuthology.orchestra.run.smithi187.stdout:Fetched 28.2 kB in 0s (81.3 kB/s) 2024-02-01T10:10:09.217 INFO:teuthology.orchestra.run.smithi148.stdout:Fetched 28.2 kB in 0s (84.9 kB/s) 2024-02-01T10:10:10.101 INFO:teuthology.orchestra.run.smithi187.stdout:Reading package lists... 2024-02-01T10:10:10.116 DEBUG:teuthology.orchestra.run.smithi187:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-931-g5a7faa11-1jammy cephadm=19.0.0-931-g5a7faa11-1jammy ceph-mds=19.0.0-931-g5a7faa11-1jammy ceph-mgr=19.0.0-931-g5a7faa11-1jammy ceph-common=19.0.0-931-g5a7faa11-1jammy ceph-fuse=19.0.0-931-g5a7faa11-1jammy ceph-test=19.0.0-931-g5a7faa11-1jammy ceph-volume=19.0.0-931-g5a7faa11-1jammy radosgw=19.0.0-931-g5a7faa11-1jammy python3-rados=19.0.0-931-g5a7faa11-1jammy python3-rgw=19.0.0-931-g5a7faa11-1jammy python3-cephfs=19.0.0-931-g5a7faa11-1jammy python3-rbd=19.0.0-931-g5a7faa11-1jammy libcephfs2=19.0.0-931-g5a7faa11-1jammy libcephfs-dev=19.0.0-931-g5a7faa11-1jammy librados2=19.0.0-931-g5a7faa11-1jammy librbd1=19.0.0-931-g5a7faa11-1jammy rbd-fuse=19.0.0-931-g5a7faa11-1jammy 2024-02-01T10:10:10.140 INFO:teuthology.orchestra.run.smithi148.stdout:Reading package lists... 2024-02-01T10:10:10.155 DEBUG:teuthology.orchestra.run.smithi148:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-931-g5a7faa11-1jammy cephadm=19.0.0-931-g5a7faa11-1jammy ceph-mds=19.0.0-931-g5a7faa11-1jammy ceph-mgr=19.0.0-931-g5a7faa11-1jammy ceph-common=19.0.0-931-g5a7faa11-1jammy ceph-fuse=19.0.0-931-g5a7faa11-1jammy ceph-test=19.0.0-931-g5a7faa11-1jammy ceph-volume=19.0.0-931-g5a7faa11-1jammy radosgw=19.0.0-931-g5a7faa11-1jammy python3-rados=19.0.0-931-g5a7faa11-1jammy python3-rgw=19.0.0-931-g5a7faa11-1jammy python3-cephfs=19.0.0-931-g5a7faa11-1jammy python3-rbd=19.0.0-931-g5a7faa11-1jammy libcephfs2=19.0.0-931-g5a7faa11-1jammy libcephfs-dev=19.0.0-931-g5a7faa11-1jammy librados2=19.0.0-931-g5a7faa11-1jammy librbd1=19.0.0-931-g5a7faa11-1jammy rbd-fuse=19.0.0-931-g5a7faa11-1jammy 2024-02-01T10:10:10.180 INFO:teuthology.orchestra.run.smithi187.stdout:Reading package lists... 2024-02-01T10:10:10.215 INFO:teuthology.orchestra.run.smithi148.stdout:Reading package lists... 2024-02-01T10:10:10.370 INFO:teuthology.orchestra.run.smithi187.stdout:Building dependency tree... 2024-02-01T10:10:10.371 INFO:teuthology.orchestra.run.smithi187.stdout:Reading state information... 2024-02-01T10:10:10.404 INFO:teuthology.orchestra.run.smithi148.stdout:Building dependency tree... 2024-02-01T10:10:10.405 INFO:teuthology.orchestra.run.smithi148.stdout:Reading state information... 2024-02-01T10:10:10.547 INFO:teuthology.orchestra.run.smithi187.stdout:The following packages were automatically installed and are no longer required: 2024-02-01T10:10:10.547 INFO:teuthology.orchestra.run.smithi187.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-02-01T10:10:10.547 INFO:teuthology.orchestra.run.smithi187.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-02-01T10:10:10.547 INFO:teuthology.orchestra.run.smithi187.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-02-01T10:10:10.548 INFO:teuthology.orchestra.run.smithi187.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-02-01T10:10:10.548 INFO:teuthology.orchestra.run.smithi187.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-02-01T10:10:10.548 INFO:teuthology.orchestra.run.smithi187.stdout:Use 'sudo apt autoremove' to remove them. 2024-02-01T10:10:10.548 INFO:teuthology.orchestra.run.smithi187.stdout:The following additional packages will be installed: 2024-02-01T10:10:10.549 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-02-01T10:10:10.549 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-02-01T10:10:10.549 INFO:teuthology.orchestra.run.smithi187.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-02-01T10:10:10.549 INFO:teuthology.orchestra.run.smithi187.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-02-01T10:10:10.550 INFO:teuthology.orchestra.run.smithi187.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-02-01T10:10:10.550 INFO:teuthology.orchestra.run.smithi187.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-02-01T10:10:10.550 INFO:teuthology.orchestra.run.smithi187.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-02-01T10:10:10.550 INFO:teuthology.orchestra.run.smithi187.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-02-01T10:10:10.550 INFO:teuthology.orchestra.run.smithi187.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-02-01T10:10:10.550 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-02-01T10:10:10.550 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-02-01T10:10:10.550 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-02-01T10:10:10.550 INFO:teuthology.orchestra.run.smithi187.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-02-01T10:10:10.550 INFO:teuthology.orchestra.run.smithi187.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-02-01T10:10:10.551 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-02-01T10:10:10.551 INFO:teuthology.orchestra.run.smithi187.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-02-01T10:10:10.551 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-02-01T10:10:10.551 INFO:teuthology.orchestra.run.smithi187.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-02-01T10:10:10.551 INFO:teuthology.orchestra.run.smithi187.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-02-01T10:10:10.551 INFO:teuthology.orchestra.run.smithi187.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-02-01T10:10:10.551 INFO:teuthology.orchestra.run.smithi187.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-02-01T10:10:10.551 INFO:teuthology.orchestra.run.smithi187.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-02-01T10:10:10.551 INFO:teuthology.orchestra.run.smithi187.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-02-01T10:10:10.551 INFO:teuthology.orchestra.run.smithi187.stdout: xmlstarlet zip 2024-02-01T10:10:10.551 INFO:teuthology.orchestra.run.smithi187.stdout:Suggested packages: 2024-02-01T10:10:10.552 INFO:teuthology.orchestra.run.smithi187.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-02-01T10:10:10.552 INFO:teuthology.orchestra.run.smithi187.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-02-01T10:10:10.552 INFO:teuthology.orchestra.run.smithi187.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-02-01T10:10:10.552 INFO:teuthology.orchestra.run.smithi187.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-02-01T10:10:10.552 INFO:teuthology.orchestra.run.smithi187.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-02-01T10:10:10.552 INFO:teuthology.orchestra.run.smithi187.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-02-01T10:10:10.552 INFO:teuthology.orchestra.run.smithi187.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-02-01T10:10:10.552 INFO:teuthology.orchestra.run.smithi187.stdout:Recommended packages: 2024-02-01T10:10:10.552 INFO:teuthology.orchestra.run.smithi187.stdout: btrfs-tools 2024-02-01T10:10:10.578 INFO:teuthology.orchestra.run.smithi148.stdout:The following packages were automatically installed and are no longer required: 2024-02-01T10:10:10.578 INFO:teuthology.orchestra.run.smithi148.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-02-01T10:10:10.579 INFO:teuthology.orchestra.run.smithi148.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-02-01T10:10:10.579 INFO:teuthology.orchestra.run.smithi148.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-02-01T10:10:10.579 INFO:teuthology.orchestra.run.smithi148.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-02-01T10:10:10.579 INFO:teuthology.orchestra.run.smithi148.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-02-01T10:10:10.579 INFO:teuthology.orchestra.run.smithi148.stdout:Use 'sudo apt autoremove' to remove them. 2024-02-01T10:10:10.580 INFO:teuthology.orchestra.run.smithi148.stdout:The following additional packages will be installed: 2024-02-01T10:10:10.580 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-02-01T10:10:10.580 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-02-01T10:10:10.580 INFO:teuthology.orchestra.run.smithi148.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-02-01T10:10:10.580 INFO:teuthology.orchestra.run.smithi148.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-02-01T10:10:10.581 INFO:teuthology.orchestra.run.smithi148.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-02-01T10:10:10.581 INFO:teuthology.orchestra.run.smithi148.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-02-01T10:10:10.582 INFO:teuthology.orchestra.run.smithi148.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-02-01T10:10:10.582 INFO:teuthology.orchestra.run.smithi148.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-02-01T10:10:10.582 INFO:teuthology.orchestra.run.smithi148.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-02-01T10:10:10.582 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-02-01T10:10:10.582 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-02-01T10:10:10.582 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-02-01T10:10:10.582 INFO:teuthology.orchestra.run.smithi148.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-02-01T10:10:10.582 INFO:teuthology.orchestra.run.smithi148.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-02-01T10:10:10.582 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-02-01T10:10:10.582 INFO:teuthology.orchestra.run.smithi148.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-02-01T10:10:10.582 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-02-01T10:10:10.582 INFO:teuthology.orchestra.run.smithi148.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-02-01T10:10:10.582 INFO:teuthology.orchestra.run.smithi148.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-02-01T10:10:10.583 INFO:teuthology.orchestra.run.smithi148.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-02-01T10:10:10.583 INFO:teuthology.orchestra.run.smithi148.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-02-01T10:10:10.583 INFO:teuthology.orchestra.run.smithi148.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-02-01T10:10:10.583 INFO:teuthology.orchestra.run.smithi148.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-02-01T10:10:10.583 INFO:teuthology.orchestra.run.smithi148.stdout: xmlstarlet zip 2024-02-01T10:10:10.583 INFO:teuthology.orchestra.run.smithi148.stdout:Suggested packages: 2024-02-01T10:10:10.583 INFO:teuthology.orchestra.run.smithi148.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-02-01T10:10:10.583 INFO:teuthology.orchestra.run.smithi148.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-02-01T10:10:10.583 INFO:teuthology.orchestra.run.smithi148.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-02-01T10:10:10.583 INFO:teuthology.orchestra.run.smithi148.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-02-01T10:10:10.584 INFO:teuthology.orchestra.run.smithi148.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-02-01T10:10:10.584 INFO:teuthology.orchestra.run.smithi148.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-02-01T10:10:10.584 INFO:teuthology.orchestra.run.smithi148.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-02-01T10:10:10.584 INFO:teuthology.orchestra.run.smithi148.stdout:Recommended packages: 2024-02-01T10:10:10.584 INFO:teuthology.orchestra.run.smithi148.stdout: btrfs-tools 2024-02-01T10:10:10.656 INFO:teuthology.orchestra.run.smithi187.stdout:The following NEW packages will be installed: 2024-02-01T10:10:10.657 INFO:teuthology.orchestra.run.smithi187.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-02-01T10:10:10.657 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-02-01T10:10:10.657 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-02-01T10:10:10.657 INFO:teuthology.orchestra.run.smithi187.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-02-01T10:10:10.657 INFO:teuthology.orchestra.run.smithi187.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-02-01T10:10:10.657 INFO:teuthology.orchestra.run.smithi187.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-02-01T10:10:10.657 INFO:teuthology.orchestra.run.smithi187.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-02-01T10:10:10.657 INFO:teuthology.orchestra.run.smithi187.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-02-01T10:10:10.657 INFO:teuthology.orchestra.run.smithi187.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-02-01T10:10:10.657 INFO:teuthology.orchestra.run.smithi187.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-02-01T10:10:10.657 INFO:teuthology.orchestra.run.smithi187.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-02-01T10:10:10.657 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-02-01T10:10:10.658 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-02-01T10:10:10.658 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-02-01T10:10:10.658 INFO:teuthology.orchestra.run.smithi187.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-02-01T10:10:10.658 INFO:teuthology.orchestra.run.smithi187.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-02-01T10:10:10.658 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-02-01T10:10:10.658 INFO:teuthology.orchestra.run.smithi187.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-02-01T10:10:10.658 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-02-01T10:10:10.658 INFO:teuthology.orchestra.run.smithi187.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-02-01T10:10:10.658 INFO:teuthology.orchestra.run.smithi187.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-02-01T10:10:10.658 INFO:teuthology.orchestra.run.smithi187.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-02-01T10:10:10.658 INFO:teuthology.orchestra.run.smithi187.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-02-01T10:10:10.658 INFO:teuthology.orchestra.run.smithi187.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-02-01T10:10:10.658 INFO:teuthology.orchestra.run.smithi187.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-02-01T10:10:10.659 INFO:teuthology.orchestra.run.smithi187.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-02-01T10:10:10.685 INFO:teuthology.orchestra.run.smithi148.stdout:The following NEW packages will be installed: 2024-02-01T10:10:10.686 INFO:teuthology.orchestra.run.smithi148.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-02-01T10:10:10.686 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-02-01T10:10:10.686 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-02-01T10:10:10.686 INFO:teuthology.orchestra.run.smithi148.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-02-01T10:10:10.686 INFO:teuthology.orchestra.run.smithi148.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-02-01T10:10:10.686 INFO:teuthology.orchestra.run.smithi148.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-02-01T10:10:10.687 INFO:teuthology.orchestra.run.smithi148.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-02-01T10:10:10.688 INFO:teuthology.orchestra.run.smithi148.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-02-01T10:10:10.688 INFO:teuthology.orchestra.run.smithi148.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-02-01T10:10:10.688 INFO:teuthology.orchestra.run.smithi148.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-02-01T10:10:10.688 INFO:teuthology.orchestra.run.smithi148.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-02-01T10:10:10.688 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-02-01T10:10:10.688 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-02-01T10:10:10.688 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-02-01T10:10:10.688 INFO:teuthology.orchestra.run.smithi148.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-02-01T10:10:10.688 INFO:teuthology.orchestra.run.smithi148.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-02-01T10:10:10.688 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-02-01T10:10:10.688 INFO:teuthology.orchestra.run.smithi148.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-02-01T10:10:10.688 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-02-01T10:10:10.689 INFO:teuthology.orchestra.run.smithi148.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-02-01T10:10:10.689 INFO:teuthology.orchestra.run.smithi148.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-02-01T10:10:10.689 INFO:teuthology.orchestra.run.smithi148.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-02-01T10:10:10.689 INFO:teuthology.orchestra.run.smithi148.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-02-01T10:10:10.689 INFO:teuthology.orchestra.run.smithi148.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-02-01T10:10:10.689 INFO:teuthology.orchestra.run.smithi148.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-02-01T10:10:10.689 INFO:teuthology.orchestra.run.smithi148.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-02-01T10:10:10.722 INFO:teuthology.orchestra.run.smithi187.stdout:0 upgraded, 118 newly installed, 0 to remove and 269 not upgraded. 2024-02-01T10:10:10.722 INFO:teuthology.orchestra.run.smithi187.stdout:Need to get 175 MB of archives. 2024-02-01T10:10:10.722 INFO:teuthology.orchestra.run.smithi187.stdout:After this operation, 727 MB of additional disk space will be used. 2024-02-01T10:10:10.722 INFO:teuthology.orchestra.run.smithi187.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-02-01T10:10:10.813 INFO:teuthology.orchestra.run.smithi187.stdout:Get:2 https://4.chacra.ceph.com/r/ceph/wip-end2end-tracing/5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-931-g5a7faa11-1jammy [3,514 kB] 2024-02-01T10:10:10.828 INFO:teuthology.orchestra.run.smithi187.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-02-01T10:10:10.831 INFO:teuthology.orchestra.run.smithi187.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-02-01T10:10:10.843 INFO:teuthology.orchestra.run.smithi148.stdout:0 upgraded, 118 newly installed, 0 to remove and 269 not upgraded. 2024-02-01T10:10:10.843 INFO:teuthology.orchestra.run.smithi148.stdout:Need to get 175 MB of archives. 2024-02-01T10:10:10.843 INFO:teuthology.orchestra.run.smithi148.stdout:After this operation, 727 MB of additional disk space will be used. 2024-02-01T10:10:10.843 INFO:teuthology.orchestra.run.smithi148.stdout:Get:1 https://4.chacra.ceph.com/r/ceph/wip-end2end-tracing/5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-931-g5a7faa11-1jammy [3,514 kB] 2024-02-01T10:10:10.852 INFO:teuthology.orchestra.run.smithi187.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-02-01T10:10:10.919 INFO:teuthology.orchestra.run.smithi187.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-02-01T10:10:10.923 INFO:teuthology.orchestra.run.smithi187.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-02-01T10:10:10.935 INFO:teuthology.orchestra.run.smithi187.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-02-01T10:10:10.939 INFO:teuthology.orchestra.run.smithi187.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-02-01T10:10:10.940 INFO:teuthology.orchestra.run.smithi187.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-02-01T10:10:10.940 INFO:teuthology.orchestra.run.smithi187.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-02-01T10:10:10.949 INFO:teuthology.orchestra.run.smithi187.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-02-01T10:10:10.951 INFO:teuthology.orchestra.run.smithi187.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-02-01T10:10:10.953 INFO:teuthology.orchestra.run.smithi187.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-02-01T10:10:10.954 INFO:teuthology.orchestra.run.smithi187.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-02-01T10:10:10.969 INFO:teuthology.orchestra.run.smithi187.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-02-01T10:10:10.972 INFO:teuthology.orchestra.run.smithi187.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.1 [174 kB] 2024-02-01T10:10:10.974 INFO:teuthology.orchestra.run.smithi187.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-02-01T10:10:10.975 INFO:teuthology.orchestra.run.smithi187.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-02-01T10:10:10.976 INFO:teuthology.orchestra.run.smithi187.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-02-01T10:10:10.990 INFO:teuthology.orchestra.run.smithi187.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-02-01T10:10:10.991 INFO:teuthology.orchestra.run.smithi187.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-02-01T10:10:10.991 INFO:teuthology.orchestra.run.smithi187.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-02-01T10:10:10.991 INFO:teuthology.orchestra.run.smithi187.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-02-01T10:10:10.992 INFO:teuthology.orchestra.run.smithi187.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-02-01T10:10:11.010 INFO:teuthology.orchestra.run.smithi187.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-02-01T10:10:11.011 INFO:teuthology.orchestra.run.smithi187.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-02-01T10:10:11.011 INFO:teuthology.orchestra.run.smithi187.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-02-01T10:10:11.013 INFO:teuthology.orchestra.run.smithi187.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-02-01T10:10:11.014 INFO:teuthology.orchestra.run.smithi187.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-02-01T10:10:11.031 INFO:teuthology.orchestra.run.smithi187.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-02-01T10:10:11.031 INFO:teuthology.orchestra.run.smithi187.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-02-01T10:10:11.032 INFO:teuthology.orchestra.run.smithi187.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-02-01T10:10:11.032 INFO:teuthology.orchestra.run.smithi187.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-02-01T10:10:11.033 INFO:teuthology.orchestra.run.smithi187.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-02-01T10:10:11.052 INFO:teuthology.orchestra.run.smithi187.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-02-01T10:10:11.052 INFO:teuthology.orchestra.run.smithi187.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-02-01T10:10:11.053 INFO:teuthology.orchestra.run.smithi187.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-02-01T10:10:11.058 INFO:teuthology.orchestra.run.smithi187.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-02-01T10:10:11.058 INFO:teuthology.orchestra.run.smithi187.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-02-01T10:10:11.072 INFO:teuthology.orchestra.run.smithi187.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-02-01T10:10:11.073 INFO:teuthology.orchestra.run.smithi187.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-02-01T10:10:11.074 INFO:teuthology.orchestra.run.smithi187.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-02-01T10:10:11.076 INFO:teuthology.orchestra.run.smithi187.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-02-01T10:10:11.077 INFO:teuthology.orchestra.run.smithi187.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-02-01T10:10:11.096 INFO:teuthology.orchestra.run.smithi187.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-02-01T10:10:11.096 INFO:teuthology.orchestra.run.smithi187.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-02-01T10:10:11.096 INFO:teuthology.orchestra.run.smithi187.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-02-01T10:10:11.097 INFO:teuthology.orchestra.run.smithi187.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-02-01T10:10:11.097 INFO:teuthology.orchestra.run.smithi187.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-02-01T10:10:11.113 INFO:teuthology.orchestra.run.smithi187.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-02-01T10:10:11.166 INFO:teuthology.orchestra.run.smithi187.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-02-01T10:10:11.167 INFO:teuthology.orchestra.run.smithi187.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-02-01T10:10:11.168 INFO:teuthology.orchestra.run.smithi187.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-02-01T10:10:11.169 INFO:teuthology.orchestra.run.smithi187.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-02-01T10:10:11.170 INFO:teuthology.orchestra.run.smithi187.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-02-01T10:10:11.187 INFO:teuthology.orchestra.run.smithi187.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-02-01T10:10:11.189 INFO:teuthology.orchestra.run.smithi187.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-02-01T10:10:11.189 INFO:teuthology.orchestra.run.smithi187.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-02-01T10:10:11.205 INFO:teuthology.orchestra.run.smithi187.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-02-01T10:10:11.206 INFO:teuthology.orchestra.run.smithi187.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-02-01T10:10:11.216 INFO:teuthology.orchestra.run.smithi187.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-02-01T10:10:11.217 INFO:teuthology.orchestra.run.smithi187.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-02-01T10:10:11.217 INFO:teuthology.orchestra.run.smithi187.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-02-01T10:10:11.218 INFO:teuthology.orchestra.run.smithi187.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-02-01T10:10:11.218 INFO:teuthology.orchestra.run.smithi187.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-02-01T10:10:11.218 INFO:teuthology.orchestra.run.smithi187.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-02-01T10:10:11.221 INFO:teuthology.orchestra.run.smithi187.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-02-01T10:10:11.237 INFO:teuthology.orchestra.run.smithi187.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-02-01T10:10:11.238 INFO:teuthology.orchestra.run.smithi187.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-02-01T10:10:11.239 INFO:teuthology.orchestra.run.smithi187.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-02-01T10:10:11.258 INFO:teuthology.orchestra.run.smithi187.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-02-01T10:10:11.260 INFO:teuthology.orchestra.run.smithi187.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-02-01T10:10:11.261 INFO:teuthology.orchestra.run.smithi187.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-02-01T10:10:11.262 INFO:teuthology.orchestra.run.smithi187.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-02-01T10:10:11.267 INFO:teuthology.orchestra.run.smithi187.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-02-01T10:10:11.267 INFO:teuthology.orchestra.run.smithi187.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-02-01T10:10:11.270 INFO:teuthology.orchestra.run.smithi187.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-02-01T10:10:11.279 INFO:teuthology.orchestra.run.smithi187.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-02-01T10:10:11.279 INFO:teuthology.orchestra.run.smithi187.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-02-01T10:10:11.279 INFO:teuthology.orchestra.run.smithi187.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-02-01T10:10:11.299 INFO:teuthology.orchestra.run.smithi187.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-02-01T10:10:11.300 INFO:teuthology.orchestra.run.smithi187.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-02-01T10:10:11.307 INFO:teuthology.orchestra.run.smithi187.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-02-01T10:10:11.308 INFO:teuthology.orchestra.run.smithi187.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-02-01T10:10:11.308 INFO:teuthology.orchestra.run.smithi187.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-02-01T10:10:11.309 INFO:teuthology.orchestra.run.smithi187.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-02-01T10:10:11.309 INFO:teuthology.orchestra.run.smithi187.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-02-01T10:10:11.412 INFO:teuthology.orchestra.run.smithi187.stdout:Get:89 https://4.chacra.ceph.com/r/ceph/wip-end2end-tracing/5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-931-g5a7faa11-1jammy [3,217 kB] 2024-02-01T10:10:11.423 INFO:teuthology.orchestra.run.smithi148.stdout:Get:2 https://4.chacra.ceph.com/r/ceph/wip-end2end-tracing/5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-931-g5a7faa11-1jammy [3,217 kB] 2024-02-01T10:10:11.625 INFO:teuthology.orchestra.run.smithi187.stdout:Get:90 https://4.chacra.ceph.com/r/ceph/wip-end2end-tracing/5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-931-g5a7faa11-1jammy [710 kB] 2024-02-01T10:10:11.682 INFO:teuthology.orchestra.run.smithi187.stdout:Get:91 https://4.chacra.ceph.com/r/ceph/wip-end2end-tracing/5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-931-g5a7faa11-1jammy [356 kB] 2024-02-01T10:10:11.708 INFO:teuthology.orchestra.run.smithi187.stdout:Get:92 https://4.chacra.ceph.com/r/ceph/wip-end2end-tracing/5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-931-g5a7faa11-1jammy [32.8 kB] 2024-02-01T10:10:11.715 INFO:teuthology.orchestra.run.smithi187.stdout:Get:93 https://4.chacra.ceph.com/r/ceph/wip-end2end-tracing/5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-931-g5a7faa11-1jammy [180 kB] 2024-02-01T10:10:11.724 INFO:teuthology.orchestra.run.smithi187.stdout:Get:94 https://4.chacra.ceph.com/r/ceph/wip-end2end-tracing/5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-931-g5a7faa11-1jammy [64.1 kB] 2024-02-01T10:10:11.725 INFO:teuthology.orchestra.run.smithi187.stdout:Get:95 https://4.chacra.ceph.com/r/ceph/wip-end2end-tracing/5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-931-g5a7faa11-1jammy [330 kB] 2024-02-01T10:10:11.739 INFO:teuthology.orchestra.run.smithi187.stdout:Get:96 https://4.chacra.ceph.com/r/ceph/wip-end2end-tracing/5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-931-g5a7faa11-1jammy [6,554 kB] 2024-02-01T10:10:11.767 INFO:teuthology.orchestra.run.smithi148.stdout:Get:3 https://4.chacra.ceph.com/r/ceph/wip-end2end-tracing/5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-931-g5a7faa11-1jammy [710 kB] 2024-02-01T10:10:11.803 INFO:teuthology.orchestra.run.smithi148.stdout:Get:4 https://4.chacra.ceph.com/r/ceph/wip-end2end-tracing/5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-931-g5a7faa11-1jammy [356 kB] 2024-02-01T10:10:11.831 INFO:teuthology.orchestra.run.smithi148.stdout:Get:5 https://4.chacra.ceph.com/r/ceph/wip-end2end-tracing/5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-931-g5a7faa11-1jammy [32.8 kB] 2024-02-01T10:10:11.832 INFO:teuthology.orchestra.run.smithi148.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/wip-end2end-tracing/5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-931-g5a7faa11-1jammy [180 kB] 2024-02-01T10:10:11.855 INFO:teuthology.orchestra.run.smithi148.stdout:Get:7 https://4.chacra.ceph.com/r/ceph/wip-end2end-tracing/5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-931-g5a7faa11-1jammy [64.1 kB] 2024-02-01T10:10:11.857 INFO:teuthology.orchestra.run.smithi148.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/wip-end2end-tracing/5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-931-g5a7faa11-1jammy [330 kB] 2024-02-01T10:10:11.887 INFO:teuthology.orchestra.run.smithi148.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/wip-end2end-tracing/5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-931-g5a7faa11-1jammy [6,554 kB] 2024-02-01T10:10:12.126 INFO:teuthology.orchestra.run.smithi187.stdout:Get:97 https://4.chacra.ceph.com/r/ceph/wip-end2end-tracing/5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-931-g5a7faa11-1jammy [112 kB] 2024-02-01T10:10:12.138 INFO:teuthology.orchestra.run.smithi187.stdout:Get:98 https://4.chacra.ceph.com/r/ceph/wip-end2end-tracing/5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-931-g5a7faa11-1jammy [467 kB] 2024-02-01T10:10:12.165 INFO:teuthology.orchestra.run.smithi187.stdout:Get:99 https://4.chacra.ceph.com/r/ceph/wip-end2end-tracing/5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-931-g5a7faa11-1jammy [23.7 MB] 2024-02-01T10:10:12.359 INFO:teuthology.orchestra.run.smithi148.stdout:Get:10 https://4.chacra.ceph.com/r/ceph/wip-end2end-tracing/5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-931-g5a7faa11-1jammy [112 kB] 2024-02-01T10:10:12.361 INFO:teuthology.orchestra.run.smithi148.stdout:Get:11 https://4.chacra.ceph.com/r/ceph/wip-end2end-tracing/5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-931-g5a7faa11-1jammy [467 kB] 2024-02-01T10:10:12.397 INFO:teuthology.orchestra.run.smithi148.stdout:Get:12 https://4.chacra.ceph.com/r/ceph/wip-end2end-tracing/5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-931-g5a7faa11-1jammy [23.7 MB] 2024-02-01T10:10:13.731 INFO:teuthology.orchestra.run.smithi187.stdout:Get:100 https://4.chacra.ceph.com/r/ceph/wip-end2end-tracing/5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-931-g5a7faa11-1jammy [5,059 kB] 2024-02-01T10:10:13.932 INFO:teuthology.orchestra.run.smithi187.stdout:Get:101 https://4.chacra.ceph.com/r/ceph/wip-end2end-tracing/5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-931-g5a7faa11-1jammy [240 kB] 2024-02-01T10:10:13.938 INFO:teuthology.orchestra.run.smithi187.stdout:Get:102 https://4.chacra.ceph.com/r/ceph/wip-end2end-tracing/5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-931-g5a7faa11-1jammy [124 kB] 2024-02-01T10:10:13.942 INFO:teuthology.orchestra.run.smithi187.stdout:Get:103 https://4.chacra.ceph.com/r/ceph/wip-end2end-tracing/5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-931-g5a7faa11-1jammy [1,519 kB] 2024-02-01T10:10:14.012 INFO:teuthology.orchestra.run.smithi148.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-02-01T10:10:14.049 INFO:teuthology.orchestra.run.smithi187.stdout:Get:104 https://4.chacra.ceph.com/r/ceph/wip-end2end-tracing/5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-931-g5a7faa11-1jammy [6,235 kB] 2024-02-01T10:10:14.210 INFO:teuthology.orchestra.run.smithi148.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-02-01T10:10:14.210 INFO:teuthology.orchestra.run.smithi148.stdout:Get:15 https://4.chacra.ceph.com/r/ceph/wip-end2end-tracing/5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-931-g5a7faa11-1jammy [5,059 kB] 2024-02-01T10:10:14.212 INFO:teuthology.orchestra.run.smithi148.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-02-01T10:10:14.217 INFO:teuthology.orchestra.run.smithi148.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-02-01T10:10:14.295 INFO:teuthology.orchestra.run.smithi148.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-02-01T10:10:14.300 INFO:teuthology.orchestra.run.smithi148.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-02-01T10:10:14.312 INFO:teuthology.orchestra.run.smithi148.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-02-01T10:10:14.317 INFO:teuthology.orchestra.run.smithi148.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-02-01T10:10:14.317 INFO:teuthology.orchestra.run.smithi148.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-02-01T10:10:14.318 INFO:teuthology.orchestra.run.smithi148.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-02-01T10:10:14.327 INFO:teuthology.orchestra.run.smithi148.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-02-01T10:10:14.329 INFO:teuthology.orchestra.run.smithi148.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-02-01T10:10:14.331 INFO:teuthology.orchestra.run.smithi148.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-02-01T10:10:14.333 INFO:teuthology.orchestra.run.smithi148.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-02-01T10:10:14.338 INFO:teuthology.orchestra.run.smithi148.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-02-01T10:10:14.340 INFO:teuthology.orchestra.run.smithi148.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.1 [174 kB] 2024-02-01T10:10:14.343 INFO:teuthology.orchestra.run.smithi148.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-02-01T10:10:14.358 INFO:teuthology.orchestra.run.smithi148.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-02-01T10:10:14.359 INFO:teuthology.orchestra.run.smithi148.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-02-01T10:10:14.359 INFO:teuthology.orchestra.run.smithi148.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-02-01T10:10:14.360 INFO:teuthology.orchestra.run.smithi148.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-02-01T10:10:14.361 INFO:teuthology.orchestra.run.smithi148.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-02-01T10:10:14.361 INFO:teuthology.orchestra.run.smithi148.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-02-01T10:10:14.362 INFO:teuthology.orchestra.run.smithi148.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-02-01T10:10:14.378 INFO:teuthology.orchestra.run.smithi148.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-02-01T10:10:14.378 INFO:teuthology.orchestra.run.smithi148.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-02-01T10:10:14.379 INFO:teuthology.orchestra.run.smithi148.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-02-01T10:10:14.397 INFO:teuthology.orchestra.run.smithi187.stdout:Get:105 https://4.chacra.ceph.com/r/ceph/wip-end2end-tracing/5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-931-g5a7faa11-1jammy [22.7 MB] 2024-02-01T10:10:14.397 INFO:teuthology.orchestra.run.smithi148.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-02-01T10:10:14.398 INFO:teuthology.orchestra.run.smithi148.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-02-01T10:10:14.398 INFO:teuthology.orchestra.run.smithi148.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-02-01T10:10:14.399 INFO:teuthology.orchestra.run.smithi148.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-02-01T10:10:14.400 INFO:teuthology.orchestra.run.smithi148.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-02-01T10:10:14.400 INFO:teuthology.orchestra.run.smithi148.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-02-01T10:10:14.401 INFO:teuthology.orchestra.run.smithi148.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-02-01T10:10:14.416 INFO:teuthology.orchestra.run.smithi148.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-02-01T10:10:14.417 INFO:teuthology.orchestra.run.smithi148.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-02-01T10:10:14.418 INFO:teuthology.orchestra.run.smithi148.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-02-01T10:10:14.424 INFO:teuthology.orchestra.run.smithi148.stdout:Get:51 https://4.chacra.ceph.com/r/ceph/wip-end2end-tracing/5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-931-g5a7faa11-1jammy [240 kB] 2024-02-01T10:10:14.436 INFO:teuthology.orchestra.run.smithi148.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-02-01T10:10:14.437 INFO:teuthology.orchestra.run.smithi148.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-02-01T10:10:14.437 INFO:teuthology.orchestra.run.smithi148.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-02-01T10:10:14.438 INFO:teuthology.orchestra.run.smithi148.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-02-01T10:10:14.439 INFO:teuthology.orchestra.run.smithi148.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-02-01T10:10:14.442 INFO:teuthology.orchestra.run.smithi148.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-02-01T10:10:14.442 INFO:teuthology.orchestra.run.smithi148.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-02-01T10:10:14.456 INFO:teuthology.orchestra.run.smithi148.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-02-01T10:10:14.457 INFO:teuthology.orchestra.run.smithi148.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-02-01T10:10:14.459 INFO:teuthology.orchestra.run.smithi148.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-02-01T10:10:14.467 INFO:teuthology.orchestra.run.smithi148.stdout:Get:62 https://4.chacra.ceph.com/r/ceph/wip-end2end-tracing/5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-931-g5a7faa11-1jammy [124 kB] 2024-02-01T10:10:14.471 INFO:teuthology.orchestra.run.smithi148.stdout:Get:63 https://4.chacra.ceph.com/r/ceph/wip-end2end-tracing/5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-931-g5a7faa11-1jammy [1,519 kB] 2024-02-01T10:10:14.475 INFO:teuthology.orchestra.run.smithi148.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-02-01T10:10:14.476 INFO:teuthology.orchestra.run.smithi148.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-02-01T10:10:14.479 INFO:teuthology.orchestra.run.smithi148.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-02-01T10:10:14.532 INFO:teuthology.orchestra.run.smithi148.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-02-01T10:10:14.533 INFO:teuthology.orchestra.run.smithi148.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-02-01T10:10:14.534 INFO:teuthology.orchestra.run.smithi148.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-02-01T10:10:14.534 INFO:teuthology.orchestra.run.smithi148.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-02-01T10:10:14.535 INFO:teuthology.orchestra.run.smithi148.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-02-01T10:10:14.556 INFO:teuthology.orchestra.run.smithi148.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-02-01T10:10:14.558 INFO:teuthology.orchestra.run.smithi148.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-02-01T10:10:14.558 INFO:teuthology.orchestra.run.smithi148.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-02-01T10:10:14.559 INFO:teuthology.orchestra.run.smithi148.stdout:Get:75 https://4.chacra.ceph.com/r/ceph/wip-end2end-tracing/5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-931-g5a7faa11-1jammy [6,235 kB] 2024-02-01T10:10:14.579 INFO:teuthology.orchestra.run.smithi148.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-02-01T10:10:14.579 INFO:teuthology.orchestra.run.smithi148.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-02-01T10:10:14.579 INFO:teuthology.orchestra.run.smithi148.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-02-01T10:10:14.580 INFO:teuthology.orchestra.run.smithi148.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-02-01T10:10:14.580 INFO:teuthology.orchestra.run.smithi148.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-02-01T10:10:14.590 INFO:teuthology.orchestra.run.smithi148.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-02-01T10:10:14.590 INFO:teuthology.orchestra.run.smithi148.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-02-01T10:10:14.591 INFO:teuthology.orchestra.run.smithi148.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-02-01T10:10:14.594 INFO:teuthology.orchestra.run.smithi148.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-02-01T10:10:14.609 INFO:teuthology.orchestra.run.smithi148.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-02-01T10:10:14.610 INFO:teuthology.orchestra.run.smithi148.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-02-01T10:10:14.611 INFO:teuthology.orchestra.run.smithi148.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-02-01T10:10:14.614 INFO:teuthology.orchestra.run.smithi148.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-02-01T10:10:14.617 INFO:teuthology.orchestra.run.smithi148.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-02-01T10:10:14.618 INFO:teuthology.orchestra.run.smithi148.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-02-01T10:10:14.629 INFO:teuthology.orchestra.run.smithi148.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-02-01T10:10:14.633 INFO:teuthology.orchestra.run.smithi148.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-02-01T10:10:14.634 INFO:teuthology.orchestra.run.smithi148.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-02-01T10:10:14.637 INFO:teuthology.orchestra.run.smithi148.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-02-01T10:10:14.648 INFO:teuthology.orchestra.run.smithi148.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-02-01T10:10:14.649 INFO:teuthology.orchestra.run.smithi148.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-02-01T10:10:14.649 INFO:teuthology.orchestra.run.smithi148.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-02-01T10:10:14.651 INFO:teuthology.orchestra.run.smithi148.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-02-01T10:10:14.651 INFO:teuthology.orchestra.run.smithi148.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-02-01T10:10:14.658 INFO:teuthology.orchestra.run.smithi148.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-02-01T10:10:14.668 INFO:teuthology.orchestra.run.smithi148.stdout:Get:101 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-02-01T10:10:14.668 INFO:teuthology.orchestra.run.smithi148.stdout:Get:102 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-02-01T10:10:14.670 INFO:teuthology.orchestra.run.smithi148.stdout:Get:103 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-02-01T10:10:14.671 INFO:teuthology.orchestra.run.smithi148.stdout:Get:104 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-02-01T10:10:15.015 INFO:teuthology.orchestra.run.smithi148.stdout:Get:105 https://4.chacra.ceph.com/r/ceph/wip-end2end-tracing/5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-931-g5a7faa11-1jammy [22.7 MB] 2024-02-01T10:10:15.887 INFO:teuthology.orchestra.run.smithi187.stdout:Get:106 https://4.chacra.ceph.com/r/ceph/wip-end2end-tracing/5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-931-g5a7faa11-1jammy [14.0 kB] 2024-02-01T10:10:15.888 INFO:teuthology.orchestra.run.smithi187.stdout:Get:107 https://4.chacra.ceph.com/r/ceph/wip-end2end-tracing/5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-931-g5a7faa11-1jammy [866 kB] 2024-02-01T10:10:15.906 INFO:teuthology.orchestra.run.smithi187.stdout:Get:108 https://4.chacra.ceph.com/r/ceph/wip-end2end-tracing/5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-931-g5a7faa11-1jammy [2,186 kB] 2024-02-01T10:10:15.979 INFO:teuthology.orchestra.run.smithi187.stdout:Get:109 https://4.chacra.ceph.com/r/ceph/wip-end2end-tracing/5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-931-g5a7faa11-1jammy [771 kB] 2024-02-01T10:10:16.037 INFO:teuthology.orchestra.run.smithi187.stdout:Get:110 https://4.chacra.ceph.com/r/ceph/wip-end2end-tracing/5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-931-g5a7faa11-1jammy [139 kB] 2024-02-01T10:10:16.043 INFO:teuthology.orchestra.run.smithi187.stdout:Get:111 https://4.chacra.ceph.com/r/ceph/wip-end2end-tracing/5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-931-g5a7faa11-1jammy [1,876 kB] 2024-02-01T10:10:16.140 INFO:teuthology.orchestra.run.smithi187.stdout:Get:112 https://4.chacra.ceph.com/r/ceph/wip-end2end-tracing/5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-931-g5a7faa11-1jammy [8,625 kB] 2024-02-01T10:10:16.676 INFO:teuthology.orchestra.run.smithi187.stdout:Get:113 https://4.chacra.ceph.com/r/ceph/wip-end2end-tracing/5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-931-g5a7faa11-1jammy [14.2 kB] 2024-02-01T10:10:16.677 INFO:teuthology.orchestra.run.smithi187.stdout:Get:114 https://4.chacra.ceph.com/r/ceph/wip-end2end-tracing/5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-931-g5a7faa11-1jammy [49.9 MB] 2024-02-01T10:10:16.734 INFO:teuthology.orchestra.run.smithi148.stdout:Get:106 https://4.chacra.ceph.com/r/ceph/wip-end2end-tracing/5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-931-g5a7faa11-1jammy [14.0 kB] 2024-02-01T10:10:16.734 INFO:teuthology.orchestra.run.smithi148.stdout:Get:107 https://4.chacra.ceph.com/r/ceph/wip-end2end-tracing/5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-931-g5a7faa11-1jammy [866 kB] 2024-02-01T10:10:16.751 INFO:teuthology.orchestra.run.smithi148.stdout:Get:108 https://4.chacra.ceph.com/r/ceph/wip-end2end-tracing/5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-931-g5a7faa11-1jammy [2,186 kB] 2024-02-01T10:10:16.869 INFO:teuthology.orchestra.run.smithi148.stdout:Get:109 https://4.chacra.ceph.com/r/ceph/wip-end2end-tracing/5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-931-g5a7faa11-1jammy [771 kB] 2024-02-01T10:10:16.970 INFO:teuthology.orchestra.run.smithi148.stdout:Get:110 https://4.chacra.ceph.com/r/ceph/wip-end2end-tracing/5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-931-g5a7faa11-1jammy [139 kB] 2024-02-01T10:10:16.973 INFO:teuthology.orchestra.run.smithi148.stdout:Get:111 https://4.chacra.ceph.com/r/ceph/wip-end2end-tracing/5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-931-g5a7faa11-1jammy [1,876 kB] 2024-02-01T10:10:17.079 INFO:teuthology.orchestra.run.smithi148.stdout:Get:112 https://4.chacra.ceph.com/r/ceph/wip-end2end-tracing/5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-931-g5a7faa11-1jammy [8,625 kB] 2024-02-01T10:10:17.812 INFO:teuthology.orchestra.run.smithi148.stdout:Get:113 https://4.chacra.ceph.com/r/ceph/wip-end2end-tracing/5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-931-g5a7faa11-1jammy [14.2 kB] 2024-02-01T10:10:17.813 INFO:teuthology.orchestra.run.smithi148.stdout:Get:114 https://4.chacra.ceph.com/r/ceph/wip-end2end-tracing/5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-931-g5a7faa11-1jammy [49.9 MB] 2024-02-01T10:10:19.619 INFO:teuthology.orchestra.run.smithi187.stdout:Get:115 https://4.chacra.ceph.com/r/ceph/wip-end2end-tracing/5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-931-g5a7faa11-1jammy [121 kB] 2024-02-01T10:10:19.622 INFO:teuthology.orchestra.run.smithi187.stdout:Get:116 https://4.chacra.ceph.com/r/ceph/wip-end2end-tracing/5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-931-g5a7faa11-1jammy [38.4 kB] 2024-02-01T10:10:19.623 INFO:teuthology.orchestra.run.smithi187.stdout:Get:117 https://4.chacra.ceph.com/r/ceph/wip-end2end-tracing/5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-931-g5a7faa11-1jammy [13.0 MB] 2024-02-01T10:10:20.149 INFO:teuthology.orchestra.run.smithi187.stdout:Get:118 https://4.chacra.ceph.com/r/ceph/wip-end2end-tracing/5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-931-g5a7faa11-1jammy [91.5 kB] 2024-02-01T10:10:20.708 INFO:teuthology.orchestra.run.smithi187.stdout:Fetched 175 MB in 9s (18.5 MB/s) 2024-02-01T10:10:20.857 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-02-01T10:10:20.956 INFO:teuthology.orchestra.run.smithi187.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165989 files and directories currently installed.) 2024-02-01T10:10:20.961 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-02-01T10:10:21.039 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-02-01T10:10:21.089 INFO:teuthology.orchestra.run.smithi148.stdout:Get:115 https://4.chacra.ceph.com/r/ceph/wip-end2end-tracing/5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-931-g5a7faa11-1jammy [121 kB] 2024-02-01T10:10:21.091 INFO:teuthology.orchestra.run.smithi148.stdout:Get:116 https://4.chacra.ceph.com/r/ceph/wip-end2end-tracing/5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-931-g5a7faa11-1jammy [38.4 kB] 2024-02-01T10:10:21.155 INFO:teuthology.orchestra.run.smithi148.stdout:Get:117 https://4.chacra.ceph.com/r/ceph/wip-end2end-tracing/5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-931-g5a7faa11-1jammy [13.0 MB] 2024-02-01T10:10:21.342 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-02-01T10:10:21.362 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-02-01T10:10:21.400 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-02-01T10:10:21.465 INFO:teuthology.orchestra.run.smithi148.stdout:Get:118 https://4.chacra.ceph.com/r/ceph/wip-end2end-tracing/5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-931-g5a7faa11-1jammy [91.5 kB] 2024-02-01T10:10:21.737 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-02-01T10:10:21.763 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-02-01T10:10:21.820 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-02-01T10:10:22.018 INFO:teuthology.orchestra.run.smithi148.stdout:Fetched 175 MB in 11s (16.3 MB/s) 2024-02-01T10:10:22.177 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-02-01T10:10:22.207 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-02-01T10:10:22.233 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-02-01T10:10:22.273 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-02-01T10:10:22.276 INFO:teuthology.orchestra.run.smithi148.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165989 files and directories currently installed.) 2024-02-01T10:10:22.282 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-02-01T10:10:22.350 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-02-01T10:10:22.645 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-02-01T10:10:22.672 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-02-01T10:10:22.685 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-02-01T10:10:22.711 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-02-01T10:10:22.711 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-02-01T10:10:22.751 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-02-01T10:10:23.056 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-02-01T10:10:23.082 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-02-01T10:10:23.096 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-02-01T10:10:23.122 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-02-01T10:10:23.123 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-02-01T10:10:23.162 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-02-01T10:10:23.491 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-02-01T10:10:23.517 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-02-01T10:10:23.549 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-02-01T10:10:23.568 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-02-01T10:10:23.595 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-02-01T10:10:23.643 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-02-01T10:10:23.852 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package librados2. 2024-02-01T10:10:23.878 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../007-librados2_19.0.0-931-g5a7faa11-1jammy_amd64.deb ... 2024-02-01T10:10:23.918 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking librados2 (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:10:24.038 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-02-01T10:10:24.064 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-02-01T10:10:24.105 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-02-01T10:10:24.371 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package librbd1. 2024-02-01T10:10:24.397 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../008-librbd1_19.0.0-931-g5a7faa11-1jammy_amd64.deb ... 2024-02-01T10:10:24.458 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-02-01T10:10:24.483 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-02-01T10:10:24.496 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking librbd1 (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:10:24.524 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-02-01T10:10:24.944 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-02-01T10:10:24.958 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package libcephfs2. 2024-02-01T10:10:24.959 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-02-01T10:10:24.984 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-931-g5a7faa11-1jammy_amd64.deb ... 2024-02-01T10:10:24.994 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-02-01T10:10:25.024 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking libcephfs2 (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:10:25.268 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package python3-rados. 2024-02-01T10:10:25.271 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package librados2. 2024-02-01T10:10:25.283 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../010-python3-rados_19.0.0-931-g5a7faa11-1jammy_amd64.deb ... 2024-02-01T10:10:25.298 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../007-librados2_19.0.0-931-g5a7faa11-1jammy_amd64.deb ... 2024-02-01T10:10:25.318 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking python3-rados (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:10:25.330 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking librados2 (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:10:25.637 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-02-01T10:10:25.664 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-931-g5a7faa11-1jammy_all.deb ... 2024-02-01T10:10:25.706 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking python3-ceph-argparse (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:10:25.774 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package librbd1. 2024-02-01T10:10:25.800 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../008-librbd1_19.0.0-931-g5a7faa11-1jammy_amd64.deb ... 2024-02-01T10:10:25.840 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking librbd1 (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:10:25.940 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package python3-cephfs. 2024-02-01T10:10:25.955 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-931-g5a7faa11-1jammy_amd64.deb ... 2024-02-01T10:10:25.998 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking python3-cephfs (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:10:26.293 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package libcephfs2. 2024-02-01T10:10:26.309 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package python3-ceph-common. 2024-02-01T10:10:26.319 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-931-g5a7faa11-1jammy_amd64.deb ... 2024-02-01T10:10:26.336 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-931-g5a7faa11-1jammy_all.deb ... 2024-02-01T10:10:26.360 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking libcephfs2 (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:10:26.376 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking python3-ceph-common (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:10:26.612 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-rados. 2024-02-01T10:10:26.627 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../010-python3-rados_19.0.0-931-g5a7faa11-1jammy_amd64.deb ... 2024-02-01T10:10:26.662 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-rados (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:10:26.671 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package python3-wcwidth. 2024-02-01T10:10:26.696 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-02-01T10:10:26.729 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-02-01T10:10:26.957 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-02-01T10:10:26.983 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-931-g5a7faa11-1jammy_all.deb ... 2024-02-01T10:10:27.023 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-ceph-argparse (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:10:27.057 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package python3-prettytable. 2024-02-01T10:10:27.083 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-02-01T10:10:27.123 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-02-01T10:10:27.276 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-cephfs. 2024-02-01T10:10:27.291 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-931-g5a7faa11-1jammy_amd64.deb ... 2024-02-01T10:10:27.334 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-cephfs (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:10:27.367 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package python3-rbd. 2024-02-01T10:10:27.383 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-931-g5a7faa11-1jammy_amd64.deb ... 2024-02-01T10:10:27.425 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking python3-rbd (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:10:27.637 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-ceph-common. 2024-02-01T10:10:27.663 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-931-g5a7faa11-1jammy_all.deb ... 2024-02-01T10:10:27.703 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-ceph-common (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:10:27.770 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-02-01T10:10:27.797 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-02-01T10:10:27.836 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-02-01T10:10:27.990 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-wcwidth. 2024-02-01T10:10:28.016 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-02-01T10:10:28.048 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-02-01T10:10:28.097 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-02-01T10:10:28.112 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-02-01T10:10:28.155 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-02-01T10:10:28.343 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-prettytable. 2024-02-01T10:10:28.358 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-02-01T10:10:28.401 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-02-01T10:10:28.417 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-02-01T10:10:28.432 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-02-01T10:10:28.467 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-02-01T10:10:28.687 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-rbd. 2024-02-01T10:10:28.713 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-931-g5a7faa11-1jammy_amd64.deb ... 2024-02-01T10:10:28.753 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-rbd (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:10:28.787 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package lua5.1. 2024-02-01T10:10:28.810 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-02-01T10:10:28.853 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-02-01T10:10:29.098 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-02-01T10:10:29.124 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-02-01T10:10:29.164 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-02-01T10:10:29.173 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package lua-any. 2024-02-01T10:10:29.189 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-02-01T10:10:29.231 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking lua-any (27ubuntu1) ... 2024-02-01T10:10:29.433 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-02-01T10:10:29.460 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-02-01T10:10:29.709 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-02-01T10:10:29.742 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package zip. 2024-02-01T10:10:29.758 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-02-01T10:10:29.792 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking zip (3.0-12build2) ... 2024-02-01T10:10:30.020 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-02-01T10:10:30.035 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-02-01T10:10:30.078 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-02-01T10:10:30.078 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package unzip. 2024-02-01T10:10:30.105 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.1_amd64.deb ... 2024-02-01T10:10:30.145 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking unzip (6.0-26ubuntu3.1) ... 2024-02-01T10:10:30.381 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package lua5.1. 2024-02-01T10:10:30.407 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-02-01T10:10:30.448 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-02-01T10:10:30.507 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package luarocks. 2024-02-01T10:10:30.522 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-02-01T10:10:30.582 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-02-01T10:10:30.767 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package lua-any. 2024-02-01T10:10:30.783 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-02-01T10:10:30.826 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking lua-any (27ubuntu1) ... 2024-02-01T10:10:30.931 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package librgw2. 2024-02-01T10:10:30.958 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../025-librgw2_19.0.0-931-g5a7faa11-1jammy_amd64.deb ... 2024-02-01T10:10:30.998 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking librgw2 (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:10:31.078 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package zip. 2024-02-01T10:10:31.093 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-02-01T10:10:31.137 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking zip (3.0-12build2) ... 2024-02-01T10:10:31.456 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package unzip. 2024-02-01T10:10:31.483 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.1_amd64.deb ... 2024-02-01T10:10:31.523 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking unzip (6.0-26ubuntu3.1) ... 2024-02-01T10:10:31.525 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package python3-rgw. 2024-02-01T10:10:31.541 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-931-g5a7faa11-1jammy_amd64.deb ... 2024-02-01T10:10:31.584 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking python3-rgw (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:10:31.903 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package luarocks. 2024-02-01T10:10:31.904 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package liboath0:amd64. 2024-02-01T10:10:31.917 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-02-01T10:10:31.929 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-02-01T10:10:31.951 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-02-01T10:10:31.969 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-02-01T10:10:32.239 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package libradosstriper1. 2024-02-01T10:10:32.254 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-931-g5a7faa11-1jammy_amd64.deb ... 2024-02-01T10:10:32.289 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking libradosstriper1 (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:10:32.343 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package librgw2. 2024-02-01T10:10:32.370 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../025-librgw2_19.0.0-931-g5a7faa11-1jammy_amd64.deb ... 2024-02-01T10:10:32.409 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking librgw2 (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:10:32.711 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package ceph-common. 2024-02-01T10:10:32.738 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../029-ceph-common_19.0.0-931-g5a7faa11-1jammy_amd64.deb ... 2024-02-01T10:10:32.778 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking ceph-common (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:10:32.953 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-rgw. 2024-02-01T10:10:32.979 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-931-g5a7faa11-1jammy_amd64.deb ... 2024-02-01T10:10:33.020 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-rgw (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:10:33.323 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package liboath0:amd64. 2024-02-01T10:10:33.349 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-02-01T10:10:33.389 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-02-01T10:10:33.675 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package libradosstriper1. 2024-02-01T10:10:33.701 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-931-g5a7faa11-1jammy_amd64.deb ... 2024-02-01T10:10:33.744 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking libradosstriper1 (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:10:34.017 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package ceph-base. 2024-02-01T10:10:34.043 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../030-ceph-base_19.0.0-931-g5a7faa11-1jammy_amd64.deb ... 2024-02-01T10:10:34.093 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking ceph-base (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:10:34.189 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package ceph-common. 2024-02-01T10:10:34.215 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../029-ceph-common_19.0.0-931-g5a7faa11-1jammy_amd64.deb ... 2024-02-01T10:10:34.255 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking ceph-common (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:10:34.646 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-02-01T10:10:34.673 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-02-01T10:10:34.704 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-02-01T10:10:34.982 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package python3-cheroot. 2024-02-01T10:10:35.009 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-02-01T10:10:35.049 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-02-01T10:10:35.302 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-02-01T10:10:35.329 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-02-01T10:10:35.369 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-02-01T10:10:35.502 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package ceph-base. 2024-02-01T10:10:35.529 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../030-ceph-base_19.0.0-931-g5a7faa11-1jammy_amd64.deb ... 2024-02-01T10:10:35.579 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking ceph-base (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:10:35.672 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package python3-jaraco.text. 2024-02-01T10:10:35.698 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-02-01T10:10:35.738 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-02-01T10:10:36.041 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-02-01T10:10:36.068 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-02-01T10:10:36.107 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-02-01T10:10:36.140 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-02-01T10:10:36.155 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-02-01T10:10:36.190 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-02-01T10:10:36.353 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package python3-tempora. 2024-02-01T10:10:36.379 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-02-01T10:10:36.411 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-02-01T10:10:36.451 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-cheroot. 2024-02-01T10:10:36.477 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-02-01T10:10:36.517 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-02-01T10:10:36.714 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package python3-portend. 2024-02-01T10:10:36.740 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-02-01T10:10:36.780 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-02-01T10:10:36.837 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-02-01T10:10:36.863 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-02-01T10:10:36.904 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-02-01T10:10:37.092 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-02-01T10:10:37.118 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-02-01T10:10:37.167 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-02-01T10:10:37.206 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-jaraco.text. 2024-02-01T10:10:37.232 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-02-01T10:10:37.273 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-02-01T10:10:37.444 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package python3-cherrypy3. 2024-02-01T10:10:37.471 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-02-01T10:10:37.503 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-02-01T10:10:37.576 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-02-01T10:10:37.602 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-02-01T10:10:37.642 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-02-01T10:10:37.832 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package python3-natsort. 2024-02-01T10:10:37.858 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-02-01T10:10:37.898 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-02-01T10:10:37.929 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-tempora. 2024-02-01T10:10:37.955 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-02-01T10:10:37.987 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-02-01T10:10:38.235 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package python3-logutils. 2024-02-01T10:10:38.261 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-02-01T10:10:38.273 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-portend. 2024-02-01T10:10:38.299 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-02-01T10:10:38.301 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-02-01T10:10:38.340 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-02-01T10:10:38.604 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package python3-markupsafe. 2024-02-01T10:10:38.631 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-02-01T10:10:38.659 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-02-01T10:10:38.671 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-02-01T10:10:38.685 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-02-01T10:10:38.725 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-02-01T10:10:38.974 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package python3-mako. 2024-02-01T10:10:39.000 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-02-01T10:10:39.045 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-cherrypy3. 2024-02-01T10:10:39.048 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-02-01T10:10:39.071 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-02-01T10:10:39.112 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-02-01T10:10:39.344 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package python3-simplegeneric. 2024-02-01T10:10:39.371 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-02-01T10:10:39.402 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-02-01T10:10:39.474 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-natsort. 2024-02-01T10:10:39.500 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-02-01T10:10:39.540 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-02-01T10:10:39.663 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package python3-singledispatch. 2024-02-01T10:10:39.690 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-02-01T10:10:39.730 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-02-01T10:10:39.835 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-logutils. 2024-02-01T10:10:39.861 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-02-01T10:10:39.901 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-02-01T10:10:40.024 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package python3-webob. 2024-02-01T10:10:40.051 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-02-01T10:10:40.091 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-02-01T10:10:40.404 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-markupsafe. 2024-02-01T10:10:40.431 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-02-01T10:10:40.444 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package python3-waitress. 2024-02-01T10:10:40.462 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-02-01T10:10:40.471 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-02-01T10:10:40.514 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-02-01T10:10:40.740 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-mako. 2024-02-01T10:10:40.766 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-02-01T10:10:40.797 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package python3-tempita. 2024-02-01T10:10:40.807 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-02-01T10:10:40.824 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-02-01T10:10:40.863 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-02-01T10:10:41.109 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package python3-paste. 2024-02-01T10:10:41.135 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-simplegeneric. 2024-02-01T10:10:41.136 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-02-01T10:10:41.161 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-02-01T10:10:41.175 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-02-01T10:10:41.201 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-02-01T10:10:41.513 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-singledispatch. 2024-02-01T10:10:41.529 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-02-01T10:10:41.539 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-02-01T10:10:41.544 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-02-01T10:10:41.579 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-02-01T10:10:41.587 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-02-01T10:10:41.865 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package python3-pastedeploy. 2024-02-01T10:10:41.873 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-webob. 2024-02-01T10:10:41.891 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-02-01T10:10:41.900 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-02-01T10:10:41.931 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-02-01T10:10:41.940 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-02-01T10:10:42.210 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package python3-webtest. 2024-02-01T10:10:42.236 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-02-01T10:10:42.251 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-waitress. 2024-02-01T10:10:42.276 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-02-01T10:10:42.278 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-02-01T10:10:42.321 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-02-01T10:10:42.587 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package python3-pecan. 2024-02-01T10:10:42.614 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-02-01T10:10:42.621 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-tempita. 2024-02-01T10:10:42.647 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-02-01T10:10:42.654 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-02-01T10:10:42.687 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-02-01T10:10:42.957 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-paste. 2024-02-01T10:10:42.983 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-02-01T10:10:43.009 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package python3-certifi. 2024-02-01T10:10:43.015 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-02-01T10:10:43.037 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-02-01T10:10:43.084 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-02-01T10:10:43.343 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-02-01T10:10:43.358 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-02-01T10:10:43.401 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-02-01T10:10:43.412 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package python3-idna. 2024-02-01T10:10:43.439 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2024-02-01T10:10:43.478 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking python3-idna (3.3-1) ... 2024-02-01T10:10:43.679 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-pastedeploy. 2024-02-01T10:10:43.706 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-02-01T10:10:43.746 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-02-01T10:10:43.757 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package python3-urllib3. 2024-02-01T10:10:43.783 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-02-01T10:10:43.815 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-02-01T10:10:44.007 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-webtest. 2024-02-01T10:10:44.033 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-02-01T10:10:44.102 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package python3-requests. 2024-02-01T10:10:44.128 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-02-01T10:10:44.332 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-02-01T10:10:44.576 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-02-01T10:10:45.152 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-pecan. 2024-02-01T10:10:45.178 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-02-01T10:10:45.188 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package python3-werkzeug. 2024-02-01T10:10:45.215 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-02-01T10:10:45.218 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-02-01T10:10:45.255 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-02-01T10:10:45.524 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-02-01T10:10:45.548 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-931-g5a7faa11-1jammy_all.deb ... 2024-02-01T10:10:45.591 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking ceph-mgr-modules-core (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:10:45.607 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-certifi. 2024-02-01T10:10:45.633 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-02-01T10:10:45.673 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-02-01T10:10:45.946 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-02-01T10:10:45.973 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-931-g5a7faa11-1jammy_amd64.deb ... 2024-02-01T10:10:46.021 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-idna. 2024-02-01T10:10:46.029 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:10:46.044 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2024-02-01T10:10:46.084 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-idna (3.3-1) ... 2024-02-01T10:10:46.316 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package python3-bcrypt. 2024-02-01T10:10:46.342 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-02-01T10:10:46.370 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-urllib3. 2024-02-01T10:10:46.382 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-02-01T10:10:46.386 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-02-01T10:10:46.420 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-02-01T10:10:46.685 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package python3-openssl. 2024-02-01T10:10:46.712 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-02-01T10:10:46.723 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-requests. 2024-02-01T10:10:46.750 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-02-01T10:10:46.752 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-02-01T10:10:46.781 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-02-01T10:10:47.059 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-werkzeug. 2024-02-01T10:10:47.086 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-02-01T10:10:47.088 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package ceph-mgr. 2024-02-01T10:10:47.115 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-931-g5a7faa11-1jammy_amd64.deb ... 2024-02-01T10:10:47.126 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-02-01T10:10:47.155 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking ceph-mgr (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:10:47.404 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-02-01T10:10:47.430 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-931-g5a7faa11-1jammy_all.deb ... 2024-02-01T10:10:47.462 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking ceph-mgr-modules-core (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:10:47.500 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package ceph-mon. 2024-02-01T10:10:47.526 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-931-g5a7faa11-1jammy_amd64.deb ... 2024-02-01T10:10:47.558 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking ceph-mon (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:10:47.817 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-02-01T10:10:47.843 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-931-g5a7faa11-1jammy_amd64.deb ... 2024-02-01T10:10:47.883 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:10:48.162 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package ceph-osd. 2024-02-01T10:10:48.189 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-931-g5a7faa11-1jammy_amd64.deb ... 2024-02-01T10:10:48.194 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-bcrypt. 2024-02-01T10:10:48.221 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-02-01T10:10:48.228 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking ceph-osd (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:10:48.261 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-02-01T10:10:48.555 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-openssl. 2024-02-01T10:10:48.571 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-02-01T10:10:48.614 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-02-01T10:10:48.942 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package ceph-mgr. 2024-02-01T10:10:48.969 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-931-g5a7faa11-1jammy_amd64.deb ... 2024-02-01T10:10:49.008 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking ceph-mgr (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:10:49.307 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package ceph. 2024-02-01T10:10:49.322 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../066-ceph_19.0.0-931-g5a7faa11-1jammy_amd64.deb ... 2024-02-01T10:10:49.353 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package ceph-mon. 2024-02-01T10:10:49.356 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking ceph (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:10:49.380 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-931-g5a7faa11-1jammy_amd64.deb ... 2024-02-01T10:10:49.411 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking ceph-mon (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:10:49.643 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package ceph-fuse. 2024-02-01T10:10:49.670 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-931-g5a7faa11-1jammy_amd64.deb ... 2024-02-01T10:10:49.709 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking ceph-fuse (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:10:50.023 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package ceph-osd. 2024-02-01T10:10:50.050 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-931-g5a7faa11-1jammy_amd64.deb ... 2024-02-01T10:10:50.063 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package ceph-mds. 2024-02-01T10:10:50.089 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking ceph-osd (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:10:50.090 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-931-g5a7faa11-1jammy_amd64.deb ... 2024-02-01T10:10:50.129 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking ceph-mds (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:10:50.524 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package cephadm. 2024-02-01T10:10:50.551 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../069-cephadm_19.0.0-931-g5a7faa11-1jammy_amd64.deb ... 2024-02-01T10:10:50.590 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking cephadm (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:10:50.860 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package python3-asyncssh. 2024-02-01T10:10:50.887 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-02-01T10:10:50.918 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-02-01T10:10:51.168 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package ceph. 2024-02-01T10:10:51.183 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../066-ceph_19.0.0-931-g5a7faa11-1jammy_amd64.deb ... 2024-02-01T10:10:51.226 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking ceph (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:10:51.238 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package python-babel-localedata. 2024-02-01T10:10:51.265 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-02-01T10:10:51.305 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-02-01T10:10:51.512 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package ceph-fuse. 2024-02-01T10:10:51.539 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-931-g5a7faa11-1jammy_amd64.deb ... 2024-02-01T10:10:51.570 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking ceph-fuse (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:10:51.949 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package ceph-mds. 2024-02-01T10:10:51.964 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-931-g5a7faa11-1jammy_amd64.deb ... 2024-02-01T10:10:51.998 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking ceph-mds (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:10:52.301 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package python3-babel. 2024-02-01T10:10:52.329 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-02-01T10:10:52.334 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package cephadm. 2024-02-01T10:10:52.361 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../069-cephadm_19.0.0-931-g5a7faa11-1jammy_amd64.deb ... 2024-02-01T10:10:52.367 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-02-01T10:10:52.401 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking cephadm (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:10:52.712 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-asyncssh. 2024-02-01T10:10:52.721 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package python3-jinja2. 2024-02-01T10:10:52.738 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-02-01T10:10:52.748 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-02-01T10:10:52.778 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-02-01T10:10:52.787 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-02-01T10:10:53.098 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python-babel-localedata. 2024-02-01T10:10:53.107 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-02-01T10:10:53.113 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-02-01T10:10:53.134 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-931-g5a7faa11-1jammy_all.deb ... 2024-02-01T10:10:53.156 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-02-01T10:10:53.173 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking ceph-mgr-cephadm (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:10:53.519 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package python3-repoze.lru. 2024-02-01T10:10:53.546 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-02-01T10:10:53.585 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-02-01T10:10:53.880 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package python3-routes. 2024-02-01T10:10:53.907 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-02-01T10:10:53.946 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-02-01T10:10:54.152 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-babel. 2024-02-01T10:10:54.180 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-02-01T10:10:54.219 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-02-01T10:10:54.275 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-02-01T10:10:54.302 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-931-g5a7faa11-1jammy_all.deb ... 2024-02-01T10:10:54.341 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking ceph-mgr-dashboard (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:10:54.555 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-jinja2. 2024-02-01T10:10:54.582 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-02-01T10:10:54.622 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-02-01T10:10:54.761 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-02-01T10:10:54.778 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-02-01T10:10:54.819 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-02-01T10:10:54.958 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-02-01T10:10:54.973 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-931-g5a7faa11-1jammy_all.deb ... 2024-02-01T10:10:55.008 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking ceph-mgr-cephadm (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:10:55.315 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package python3-joblib. 2024-02-01T10:10:55.342 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-02-01T10:10:55.373 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-02-01T10:10:55.386 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-repoze.lru. 2024-02-01T10:10:55.413 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-02-01T10:10:55.452 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-02-01T10:10:55.702 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-02-01T10:10:55.705 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-routes. 2024-02-01T10:10:55.717 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-02-01T10:10:55.721 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-02-01T10:10:55.751 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-02-01T10:10:55.763 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-02-01T10:10:56.038 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package python3-sklearn. 2024-02-01T10:10:56.065 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-02-01T10:10:56.066 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-02-01T10:10:56.082 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-931-g5a7faa11-1jammy_all.deb ... 2024-02-01T10:10:56.104 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-02-01T10:10:56.117 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking ceph-mgr-dashboard (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:10:56.536 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-02-01T10:10:56.552 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-02-01T10:10:56.594 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-02-01T10:10:56.822 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-02-01T10:10:56.849 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-931-g5a7faa11-1jammy_all.deb ... 2024-02-01T10:10:56.973 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:10:57.106 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-joblib. 2024-02-01T10:10:57.133 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-02-01T10:10:57.173 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-02-01T10:10:57.535 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-02-01T10:10:57.561 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-02-01T10:10:57.601 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-02-01T10:10:57.904 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-sklearn. 2024-02-01T10:10:57.919 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package python3-cachetools. 2024-02-01T10:10:57.930 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-02-01T10:10:57.946 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-02-01T10:10:57.970 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-02-01T10:10:57.985 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-02-01T10:10:58.247 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package python3-pyasn1. 2024-02-01T10:10:58.274 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-02-01T10:10:58.305 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-02-01T10:10:58.617 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-02-01T10:10:58.644 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-02-01T10:10:58.662 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-02-01T10:10:58.683 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-02-01T10:10:58.689 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-931-g5a7faa11-1jammy_all.deb ... 2024-02-01T10:10:58.814 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:10:58.978 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package python3-rsa. 2024-02-01T10:10:59.005 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-02-01T10:10:59.044 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking python3-rsa (4.8-1) ... 2024-02-01T10:10:59.357 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package python3-google-auth. 2024-02-01T10:10:59.383 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-02-01T10:10:59.423 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-02-01T10:10:59.742 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-cachetools. 2024-02-01T10:10:59.743 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-02-01T10:10:59.770 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-02-01T10:10:59.770 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-02-01T10:10:59.809 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-02-01T10:10:59.810 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-02-01T10:11:00.071 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package python3-websocket. 2024-02-01T10:11:00.078 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-pyasn1. 2024-02-01T10:11:00.091 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-02-01T10:11:00.098 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-02-01T10:11:00.119 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-02-01T10:11:00.138 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-02-01T10:11:00.457 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-02-01T10:11:00.484 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-02-01T10:11:00.523 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-02-01T10:11:00.733 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package python3-kubernetes. 2024-02-01T10:11:00.760 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-02-01T10:11:00.800 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-02-01T10:11:00.826 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-rsa. 2024-02-01T10:11:00.853 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-02-01T10:11:00.892 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-rsa (4.8-1) ... 2024-02-01T10:11:01.213 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-google-auth. 2024-02-01T10:11:01.240 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-02-01T10:11:01.279 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-02-01T10:11:01.481 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-02-01T10:11:01.509 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-931-g5a7faa11-1jammy_all.deb ... 2024-02-01T10:11:01.547 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:11:01.583 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-02-01T10:11:01.610 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-02-01T10:11:01.649 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-02-01T10:11:01.825 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package libonig5:amd64. 2024-02-01T10:11:01.852 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-02-01T10:11:01.891 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-02-01T10:11:01.935 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-websocket. 2024-02-01T10:11:01.963 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-02-01T10:11:01.993 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-02-01T10:11:02.203 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package libjq1:amd64. 2024-02-01T10:11:02.230 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-02-01T10:11:02.270 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-02-01T10:11:02.322 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-kubernetes. 2024-02-01T10:11:02.349 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-02-01T10:11:02.388 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-02-01T10:11:02.489 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package jq. 2024-02-01T10:11:02.505 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-02-01T10:11:02.547 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-02-01T10:11:02.775 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package socat. 2024-02-01T10:11:02.802 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-02-01T10:11:02.841 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-02-01T10:11:03.077 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-02-01T10:11:03.105 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-931-g5a7faa11-1jammy_all.deb ... 2024-02-01T10:11:03.111 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package xmlstarlet. 2024-02-01T10:11:03.128 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-02-01T10:11:03.143 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:11:03.161 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-02-01T10:11:03.463 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package libonig5:amd64. 2024-02-01T10:11:03.491 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-02-01T10:11:03.499 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package ceph-test. 2024-02-01T10:11:03.515 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../097-ceph-test_19.0.0-931-g5a7faa11-1jammy_amd64.deb ... 2024-02-01T10:11:03.529 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-02-01T10:11:03.549 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking ceph-test (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:11:03.841 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package libjq1:amd64. 2024-02-01T10:11:03.868 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-02-01T10:11:03.907 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-02-01T10:11:04.168 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package jq. 2024-02-01T10:11:04.184 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-02-01T10:11:04.226 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-02-01T10:11:04.479 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package socat. 2024-02-01T10:11:04.507 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-02-01T10:11:04.554 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-02-01T10:11:04.865 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package xmlstarlet. 2024-02-01T10:11:04.893 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-02-01T10:11:04.932 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-02-01T10:11:05.320 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package ceph-test. 2024-02-01T10:11:05.347 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../097-ceph-test_19.0.0-931-g5a7faa11-1jammy_amd64.deb ... 2024-02-01T10:11:05.386 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking ceph-test (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:11:05.854 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package ceph-volume. 2024-02-01T10:11:05.881 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-931-g5a7faa11-1jammy_all.deb ... 2024-02-01T10:11:05.920 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking ceph-volume (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:11:06.216 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package libcephfs-dev. 2024-02-01T10:11:06.244 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-931-g5a7faa11-1jammy_amd64.deb ... 2024-02-01T10:11:06.282 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking libcephfs-dev (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:11:06.585 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package lua-socket:amd64. 2024-02-01T10:11:06.603 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-02-01T10:11:06.644 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-02-01T10:11:06.932 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package lua-sec:amd64. 2024-02-01T10:11:06.948 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-02-01T10:11:06.981 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-02-01T10:11:07.353 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package nvme-cli. 2024-02-01T10:11:07.380 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-02-01T10:11:07.419 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-02-01T10:11:07.690 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package ceph-volume. 2024-02-01T10:11:07.718 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-931-g5a7faa11-1jammy_all.deb ... 2024-02-01T10:11:07.757 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking ceph-volume (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:11:07.858 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package pkg-config. 2024-02-01T10:11:07.885 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-02-01T10:11:07.924 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-02-01T10:11:08.019 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package libcephfs-dev. 2024-02-01T10:11:08.046 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-931-g5a7faa11-1jammy_amd64.deb ... 2024-02-01T10:11:08.077 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking libcephfs-dev (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:11:08.161 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-02-01T10:11:08.177 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-02-01T10:11:08.219 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-02-01T10:11:08.413 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package lua-socket:amd64. 2024-02-01T10:11:08.441 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-02-01T10:11:08.480 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-02-01T10:11:08.589 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package python3-iniconfig. 2024-02-01T10:11:08.617 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-02-01T10:11:08.656 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-02-01T10:11:08.826 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package lua-sec:amd64. 2024-02-01T10:11:08.853 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-02-01T10:11:08.892 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-02-01T10:11:08.958 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package python3-pastescript. 2024-02-01T10:11:08.986 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-02-01T10:11:09.025 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-02-01T10:11:09.230 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package nvme-cli. 2024-02-01T10:11:09.257 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-02-01T10:11:09.288 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-02-01T10:11:09.314 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package python3-pluggy. 2024-02-01T10:11:09.341 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-02-01T10:11:09.380 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-02-01T10:11:09.676 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package pkg-config. 2024-02-01T10:11:09.692 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package python3-psutil. 2024-02-01T10:11:09.704 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-02-01T10:11:09.719 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-02-01T10:11:09.742 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-02-01T10:11:09.758 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-02-01T10:11:09.978 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-02-01T10:11:10.006 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-02-01T10:11:10.045 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-02-01T10:11:10.069 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package python3-py. 2024-02-01T10:11:10.097 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-02-01T10:11:10.135 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking python3-py (1.10.0-1) ... 2024-02-01T10:11:10.440 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-iniconfig. 2024-02-01T10:11:10.447 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package python3-pygments. 2024-02-01T10:11:10.456 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-02-01T10:11:10.475 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-02-01T10:11:10.492 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-02-01T10:11:10.514 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-02-01T10:11:10.792 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-pastescript. 2024-02-01T10:11:10.820 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-02-01T10:11:10.859 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-02-01T10:11:10.984 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package python3-pyinotify. 2024-02-01T10:11:11.012 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-02-01T10:11:11.051 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-02-01T10:11:11.162 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-pluggy. 2024-02-01T10:11:11.189 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-02-01T10:11:11.228 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-02-01T10:11:11.362 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package python3-toml. 2024-02-01T10:11:11.389 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-02-01T10:11:11.437 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-02-01T10:11:11.511 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-psutil. 2024-02-01T10:11:11.536 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-02-01T10:11:11.567 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-02-01T10:11:11.740 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package python3-pytest. 2024-02-01T10:11:11.767 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-02-01T10:11:11.806 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-02-01T10:11:11.920 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-py. 2024-02-01T10:11:11.947 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-02-01T10:11:11.986 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-py (1.10.0-1) ... 2024-02-01T10:11:12.176 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package python3-simplejson. 2024-02-01T10:11:12.204 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-02-01T10:11:12.243 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-02-01T10:11:12.290 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-pygments. 2024-02-01T10:11:12.317 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-02-01T10:11:12.348 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-02-01T10:11:12.521 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-02-01T10:11:12.548 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-02-01T10:11:12.596 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-02-01T10:11:12.794 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-pyinotify. 2024-02-01T10:11:12.821 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-02-01T10:11:12.852 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-02-01T10:11:13.146 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-toml. 2024-02-01T10:11:13.174 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-02-01T10:11:13.212 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-02-01T10:11:13.299 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package radosgw. 2024-02-01T10:11:13.327 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../116-radosgw_19.0.0-931-g5a7faa11-1jammy_amd64.deb ... 2024-02-01T10:11:13.365 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking radosgw (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:11:13.507 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-pytest. 2024-02-01T10:11:13.535 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-02-01T10:11:13.573 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-02-01T10:11:13.910 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-simplejson. 2024-02-01T10:11:13.938 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-02-01T10:11:13.968 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-02-01T10:11:14.143 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package rbd-fuse. 2024-02-01T10:11:14.159 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-931-g5a7faa11-1jammy_amd64.deb ... 2024-02-01T10:11:14.196 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-02-01T10:11:14.201 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking rbd-fuse (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:11:14.223 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-02-01T10:11:14.254 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-02-01T10:11:14.481 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-02-01T10:11:14.731 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-02-01T10:11:14.858 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-02-01T10:11:14.992 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package radosgw. 2024-02-01T10:11:15.020 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../116-radosgw_19.0.0-931-g5a7faa11-1jammy_amd64.deb ... 2024-02-01T10:11:15.074 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking radosgw (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:11:15.124 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-02-01T10:11:15.423 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-02-01T10:11:15.830 INFO:teuthology.orchestra.run.smithi187.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-02-01T10:11:15.869 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package rbd-fuse. 2024-02-01T10:11:15.893 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up cephadm (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:11:15.897 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-931-g5a7faa11-1jammy_amd64.deb ... 2024-02-01T10:11:15.936 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking rbd-fuse (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:11:16.231 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-02-01T10:11:16.444 INFO:teuthology.orchestra.run.smithi187.stdout:Adding system user cephadm....done 2024-02-01T10:11:16.491 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-02-01T10:11:16.574 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-02-01T10:11:16.616 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-02-01T10:11:16.842 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-02-01T10:11:16.879 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-02-01T10:11:17.084 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-02-01T10:11:17.186 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-02-01T10:11:17.211 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-02-01T10:11:17.495 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-02-01T10:11:17.605 INFO:teuthology.orchestra.run.smithi148.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-02-01T10:11:17.746 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-02-01T10:11:17.855 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up python3-py (1.10.0-1) ... 2024-02-01T10:11:17.894 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up cephadm (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:11:18.141 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-02-01T10:11:18.462 INFO:teuthology.orchestra.run.smithi148.stdout:Adding system user cephadm....done 2024-02-01T10:11:18.492 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-02-01T10:11:18.642 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-02-01T10:11:18.735 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-02-01T10:11:18.927 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-02-01T10:11:19.011 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up unzip (6.0-26ubuntu3.1) ... 2024-02-01T10:11:19.177 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-02-01T10:11:19.303 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-02-01T10:11:19.313 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-02-01T10:11:19.546 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-02-01T10:11:19.555 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-02-01T10:11:19.789 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-02-01T10:11:19.807 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-02-01T10:11:19.899 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-py (1.10.0-1) ... 2024-02-01T10:11:20.057 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up python3-ceph-argparse (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:11:20.183 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-02-01T10:11:20.367 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-02-01T10:11:20.493 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-02-01T10:11:20.552 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-02-01T10:11:20.619 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-02-01T10:11:20.753 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-02-01T10:11:20.811 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-02-01T10:11:21.079 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up unzip (6.0-26ubuntu3.1) ... 2024-02-01T10:11:21.129 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-02-01T10:11:21.339 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-02-01T10:11:21.574 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-02-01T10:11:21.791 INFO:teuthology.orchestra.run.smithi187.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-02-01T10:11:21.820 INFO:teuthology.orchestra.run.smithi187.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-02-01T10:11:21.825 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-02-01T10:11:21.882 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-02-01T10:11:22.008 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-02-01T10:11:22.326 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-ceph-argparse (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:11:22.376 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-02-01T10:11:22.502 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-02-01T10:11:22.593 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-02-01T10:11:22.719 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-02-01T10:11:22.812 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-02-01T10:11:22.845 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-02-01T10:11:22.971 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-02-01T10:11:23.071 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-02-01T10:11:23.230 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-02-01T10:11:23.372 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up zip (3.0-12build2) ... 2024-02-01T10:11:23.490 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-02-01T10:11:23.592 INFO:teuthology.orchestra.run.smithi148.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-02-01T10:11:23.629 INFO:teuthology.orchestra.run.smithi148.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-02-01T10:11:23.692 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-02-01T10:11:23.818 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-02-01T10:11:24.169 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-02-01T10:11:24.183 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-02-01T10:11:24.295 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-02-01T10:11:24.434 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-02-01T10:11:24.544 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-02-01T10:11:24.554 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-02-01T10:11:24.653 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-02-01T10:11:24.814 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-02-01T10:11:24.937 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-02-01T10:11:25.115 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up zip (3.0-12build2) ... 2024-02-01T10:11:25.225 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-02-01T10:11:25.339 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-02-01T10:11:25.598 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-02-01T10:11:25.876 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-02-01T10:11:25.974 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-02-01T10:11:26.136 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-02-01T10:11:26.237 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-02-01T10:11:26.242 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-02-01T10:11:26.338 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-02-01T10:11:26.577 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-02-01T10:11:26.630 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-02-01T10:11:26.837 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-02-01T10:11:26.963 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up python3-ceph-common (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:11:27.007 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-02-01T10:11:27.258 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-02-01T10:11:27.264 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up python3-idna (3.3-1) ... 2024-02-01T10:11:27.590 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-02-01T10:11:27.643 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-02-01T10:11:27.936 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-02-01T10:11:28.262 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-02-01T10:11:28.538 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-02-01T10:11:28.656 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-ceph-common (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:11:28.796 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-02-01T10:11:28.935 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-idna (3.3-1) ... 2024-02-01T10:11:29.055 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-02-01T10:11:29.190 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up python3-toml (0.10.2-1) ... 2024-02-01T10:11:29.250 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-02-01T10:11:29.466 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-02-01T10:11:29.592 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-02-01T10:11:29.718 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-02-01T10:11:30.011 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-02-01T10:11:30.270 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-02-01T10:11:30.372 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-02-01T10:11:30.521 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-02-01T10:11:30.632 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-02-01T10:11:30.648 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-02-01T10:11:30.766 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-toml (0.10.2-1) ... 2024-02-01T10:11:30.940 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-02-01T10:11:31.034 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-02-01T10:11:31.174 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-02-01T10:11:31.180 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-02-01T10:11:31.295 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-02-01T10:11:31.426 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-02-01T10:11:31.588 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-02-01T10:11:31.872 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-02-01T10:11:31.910 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-02-01T10:11:32.132 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-02-01T10:11:32.153 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-02-01T10:11:32.257 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-02-01T10:11:32.404 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-02-01T10:11:32.530 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-02-01T10:11:32.567 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-02-01T10:11:32.823 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-02-01T10:11:32.835 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-02-01T10:11:32.948 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-02-01T10:11:33.094 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-02-01T10:11:33.236 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-02-01T10:11:33.329 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-02-01T10:11:33.534 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-02-01T10:11:33.579 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-02-01T10:11:33.768 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-02-01T10:11:33.847 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-02-01T10:11:33.920 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-02-01T10:11:33.973 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-02-01T10:11:34.045 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-02-01T10:11:34.136 INFO:teuthology.orchestra.run.smithi187.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-02-01T10:11:34.258 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-02-01T10:11:34.380 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-02-01T10:11:34.384 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-02-01T10:11:34.506 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-02-01T10:11:34.676 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-02-01T10:11:34.908 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-02-01T10:11:34.961 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-02-01T10:11:35.175 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up lua-any (27ubuntu1) ... 2024-02-01T10:11:35.212 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-02-01T10:11:35.301 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up python3-portend (3.0.0-1) ... 2024-02-01T10:11:35.355 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-02-01T10:11:35.481 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-02-01T10:11:35.561 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-02-01T10:11:35.578 INFO:teuthology.orchestra.run.smithi148.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-02-01T10:11:35.681 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-02-01T10:11:35.833 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-02-01T10:11:35.959 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-02-01T10:11:35.996 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-02-01T10:11:36.289 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-02-01T10:11:36.360 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-02-01T10:11:36.557 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-02-01T10:11:36.594 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up lua-any (27ubuntu1) ... 2024-02-01T10:11:36.683 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up python3-rsa (4.8-1) ... 2024-02-01T10:11:36.712 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-portend (3.0.0-1) ... 2024-02-01T10:11:36.942 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-02-01T10:11:36.971 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-02-01T10:11:37.097 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-02-01T10:11:37.210 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-02-01T10:11:37.415 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-02-01T10:11:37.530 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-02-01T10:11:37.717 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-02-01T10:11:37.915 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-02-01T10:11:38.001 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-02-01T10:11:38.102 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-rsa (4.8-1) ... 2024-02-01T10:11:38.183 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-02-01T10:11:38.370 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-02-01T10:11:38.519 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-02-01T10:11:38.629 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-02-01T10:11:38.645 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up librados2 (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:11:38.771 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up libsqlite3-mod-ceph (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:11:38.897 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-02-01T10:11:38.966 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-02-01T10:11:39.208 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up libcephfs2 (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:11:39.334 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up libradosstriper1 (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:11:39.351 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-02-01T10:11:39.460 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-02-01T10:11:39.619 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-02-01T10:11:39.737 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up librbd1 (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:11:39.863 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up ceph-mgr-modules-core (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:11:39.963 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-02-01T10:11:39.989 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up ceph-fuse (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:11:40.097 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up librados2 (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:11:40.201 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-02-01T10:11:40.201 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-02-01T10:11:40.232 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up libsqlite3-mod-ceph (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:11:40.358 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-02-01T10:11:40.678 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up libcephfs2 (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:11:40.757 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up libcephfs-dev (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:11:40.779 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up libradosstriper1 (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:11:40.883 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up python3-rados (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:11:40.904 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-02-01T10:11:41.009 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up librgw2 (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:11:41.110 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-02-01T10:11:41.172 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up librbd1 (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:11:41.299 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up ceph-mgr-modules-core (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:11:41.425 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up ceph-fuse (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:11:41.639 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-02-01T10:11:41.639 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-02-01T10:11:42.159 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up libcephfs-dev (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:11:42.277 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-rados (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:11:42.297 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up python3-rbd (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:11:42.378 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up librgw2 (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:11:42.406 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up rbd-fuse (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:11:42.479 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-02-01T10:11:42.507 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up python3-rgw (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:11:42.624 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up python3-cephfs (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:11:42.750 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up ceph-common (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:11:43.301 INFO:teuthology.orchestra.run.smithi187.stdout:Adding group ceph....done 2024-02-01T10:11:43.662 INFO:teuthology.orchestra.run.smithi187.stdout:Adding system user ceph....done 2024-02-01T10:11:43.675 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-rbd (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:11:43.801 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up rbd-fuse (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:11:43.837 INFO:teuthology.orchestra.run.smithi187.stdout:Setting system user ceph properties....done 2024-02-01T10:11:43.854 INFO:teuthology.orchestra.run.smithi187.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-02-01T10:11:43.927 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-rgw (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:11:43.990 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-02-01T10:11:44.044 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-cephfs (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:11:44.145 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up ceph-common (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:11:44.327 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-02-01T10:11:44.654 INFO:teuthology.orchestra.run.smithi148.stdout:Adding group ceph....done 2024-02-01T10:11:45.083 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up ceph-test (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:11:45.089 INFO:teuthology.orchestra.run.smithi148.stdout:Adding system user ceph....done 2024-02-01T10:11:45.212 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up radosgw (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:11:45.266 INFO:teuthology.orchestra.run.smithi148.stdout:Setting system user ceph properties....done 2024-02-01T10:11:45.282 INFO:teuthology.orchestra.run.smithi148.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-02-01T10:11:45.421 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-02-01T10:11:45.753 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-02-01T10:11:45.776 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-02-01T10:11:45.777 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-02-01T10:11:46.221 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up ceph-test (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:11:46.231 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up ceph-base (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:11:46.347 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up radosgw (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:11:46.545 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-02-01T10:11:46.819 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-02-01T10:11:46.819 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-02-01T10:11:47.008 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up ceph-mds (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:11:47.274 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-02-01T10:11:47.274 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-02-01T10:11:47.299 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up ceph-base (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:11:47.645 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-02-01T10:11:47.746 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up ceph-mgr (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:11:47.941 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-02-01T10:11:47.941 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-02-01T10:11:48.109 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up ceph-mds (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:11:48.328 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-02-01T10:11:48.328 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-02-01T10:11:48.388 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up ceph-osd (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:11:48.644 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-02-01T10:11:48.644 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-02-01T10:11:48.794 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up ceph-mgr (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:11:49.008 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-02-01T10:11:49.009 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-02-01T10:11:49.107 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up ceph-mgr-k8sevents (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:11:49.233 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:11:49.375 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up ceph-mon (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:11:49.495 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up ceph-osd (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:11:49.593 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-02-01T10:11:49.593 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-02-01T10:11:49.770 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-02-01T10:11:49.770 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-02-01T10:11:50.068 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up ceph-mgr-cephadm (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:11:50.228 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up ceph (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:11:50.238 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up ceph-mgr-k8sevents (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:11:50.345 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up ceph-mgr-dashboard (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:11:50.365 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:11:50.471 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up ceph-volume (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:11:50.516 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up ceph-mon (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:11:50.734 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-02-01T10:11:50.734 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-02-01T10:11:50.799 INFO:teuthology.orchestra.run.smithi187.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-02-01T10:11:50.908 INFO:teuthology.orchestra.run.smithi187.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-02-01T10:11:51.061 INFO:teuthology.orchestra.run.smithi187.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-02-01T10:11:51.192 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up ceph-mgr-cephadm (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:11:51.334 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up ceph (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:11:51.631 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up ceph-mgr-dashboard (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:11:51.807 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up ceph-volume (19.0.0-931-g5a7faa11-1jammy) ... 2024-02-01T10:11:52.153 INFO:teuthology.orchestra.run.smithi187.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-02-01T10:11:52.160 INFO:teuthology.orchestra.run.smithi148.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-02-01T10:11:52.260 INFO:teuthology.orchestra.run.smithi148.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-02-01T10:11:52.412 INFO:teuthology.orchestra.run.smithi148.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-02-01T10:11:53.248 INFO:teuthology.orchestra.run.smithi187.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-02-01T10:11:53.254 DEBUG:teuthology.parallel:result is None 2024-02-01T10:11:53.797 INFO:teuthology.orchestra.run.smithi148.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-02-01T10:11:54.431 INFO:teuthology.orchestra.run.smithi148.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-02-01T10:11:54.435 DEBUG:teuthology.parallel:result is None 2024-02-01T10:11:54.436 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24 2024-02-01T10:11:54.580 DEBUG:teuthology.orchestra.run.smithi148:> dpkg-query -W -f '${Version}' ceph 2024-02-01T10:11:54.599 INFO:teuthology.orchestra.run.smithi148.stdout:19.0.0-931-g5a7faa11-1jammy 2024-02-01T10:11:54.600 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-931-g5a7faa11-1jammy 2024-02-01T10:11:54.600 INFO:teuthology.task.install:The correct ceph version 19.0.0-931-g5a7faa11-1jammy is installed. 2024-02-01T10:11:54.602 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24 2024-02-01T10:11:54.747 DEBUG:teuthology.orchestra.run.smithi187:> dpkg-query -W -f '${Version}' ceph 2024-02-01T10:11:54.766 INFO:teuthology.orchestra.run.smithi187.stdout:19.0.0-931-g5a7faa11-1jammy 2024-02-01T10:11:54.766 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-931-g5a7faa11-1jammy 2024-02-01T10:11:54.766 INFO:teuthology.task.install:The correct ceph version 19.0.0-931-g5a7faa11-1jammy is installed. 2024-02-01T10:11:54.768 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-02-01T10:11:54.768 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-02-01T10:11:54.768 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-01T10:11:54.781 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-02-01T10:11:54.782 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-01T10:11:54.819 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-02-01T10:11:54.819 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-02-01T10:11:54.819 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/usr/bin/daemon-helper 2024-02-01T10:11:54.835 DEBUG:teuthology.orchestra.run.smithi148:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-01T10:11:54.891 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-02-01T10:11:54.891 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/usr/bin/daemon-helper 2024-02-01T10:11:54.903 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-01T10:11:54.959 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-02-01T10:11:54.959 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-02-01T10:11:54.959 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-01T10:11:54.973 DEBUG:teuthology.orchestra.run.smithi148:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-01T10:11:55.027 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-02-01T10:11:55.027 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-01T10:11:55.040 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-01T10:11:55.094 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-02-01T10:11:55.095 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-02-01T10:11:55.095 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/usr/bin/stdin-killer 2024-02-01T10:11:55.109 DEBUG:teuthology.orchestra.run.smithi148:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-01T10:11:55.163 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-02-01T10:11:55.163 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/usr/bin/stdin-killer 2024-02-01T10:11:55.176 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-01T10:11:55.230 INFO:teuthology.run_tasks:Running task ceph... 2024-02-01T10:11:55.320 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-02-01T10:11:55.320 DEBUG:teuthology.orchestra.run.smithi148:> sudo chmod 777 /var/log/ceph 2024-02-01T10:11:55.323 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod 777 /var/log/ceph 2024-02-01T10:11:55.336 INFO:tasks.ceph:Disabling ceph logrotate... 2024-02-01T10:11:55.336 DEBUG:teuthology.orchestra.run.smithi148:> sudo rm -f -- /etc/logrotate.d/ceph 2024-02-01T10:11:55.377 DEBUG:teuthology.orchestra.run.smithi187:> sudo rm -f -- /etc/logrotate.d/ceph 2024-02-01T10:11:55.389 INFO:tasks.ceph:Creating extra log directories... 2024-02-01T10:11:55.390 DEBUG:teuthology.orchestra.run.smithi148:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-02-01T10:11:55.433 DEBUG:teuthology.orchestra.run.smithi187:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-02-01T10:11:55.465 INFO:tasks.ceph:Creating ceph cluster c1... 2024-02-01T10:11:55.465 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'cpu_profile': set(), 'cluster': 'c1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-02-01T10:11:55.465 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuvalif-2024-01-30_08:46:48-rgw-wip-end2end-tracing-distro-default-smithi/7537587', 'branch': 'wip-end2end-tracing', 'description': 'rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zones supported-random-distro$/{ubuntu_latest} tasks/test_multi}', 'email': None, 'first_in_suite': False, 'job_id': '7537587', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuvalif-2024-01-30_08:46:48-rgw-wip-end2end-tracing-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-end2end-tracing'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': '5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'enabled_features': ['resharding'], 'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}, {'endpoints': ['c1.client.1'], 'name': 'test-zone3'}]}]}, 'rgw-multisite-tests': {'args': ['tests.py']}, 'workunit': {'branch': 'wip-end2end-tracing', 'sha1': '5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24'}}, 'owner': 'scheduled_yuvalif@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'seed': 8557, 'sha1': '5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24', 'sleep_before_teardown': 0, 'subset': '1/3', 'suite': 'rgw', 'suite_branch': 'wip-end2end-tracing', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24', 'targets': {'smithi148.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBH5ce3/FCp+PGNOXXdX8il8f4jkJaHpplftAcDQSFX3tHjR+D85kVrtsDWCNIGnihkTWiK/q2ovxDNYhkN4nd0g=', 'smithi187.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLmrn1ce5MSq4PQkPQy/SrK5OHVanVF/Lx8xwqrWintcWDpX5+8boxWTRV5B7sthtEqb55YGOwOBChsT+yODK3Y='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': '5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24'}}, {'ceph': {'cluster': 'c2'}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '6128cc3ecb49b7d62475e3595041c19b5326ca7c', 'timestamp': '2024-01-30_08:46:48', 'tube': 'smithi', 'user': 'yuvalif', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-02-01T10:11:55.467 DEBUG:teuthology.orchestra.run.smithi148:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2024-02-01T10:11:55.489 DEBUG:teuthology.orchestra.run.smithi187:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2024-02-01T10:11:55.495 DEBUG:teuthology.orchestra.run.smithi148:> sudo install -d -m0777 -- /var/run/ceph 2024-02-01T10:11:55.537 DEBUG:teuthology.orchestra.run.smithi187:> sudo install -d -m0777 -- /var/run/ceph 2024-02-01T10:11:55.553 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-02-01T10:11:55.553 DEBUG:teuthology.orchestra.run.smithi148:> dd if=/scratch_devs of=/dev/stdout 2024-02-01T10:11:55.597 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-01T10:11:55.597 DEBUG:teuthology.orchestra.run.smithi148:> stat /dev/vg_nvme/lv_1 2024-02-01T10:11:55.644 INFO:teuthology.orchestra.run.smithi148.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-01T10:11:55.644 INFO:teuthology.orchestra.run.smithi148.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-01T10:11:55.644 INFO:teuthology.orchestra.run.smithi148.stdout:Device: 5h/5d Inode: 872 Links: 1 2024-02-01T10:11:55.644 INFO:teuthology.orchestra.run.smithi148.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-01T10:11:55.645 INFO:teuthology.orchestra.run.smithi148.stdout:Access: 2024-02-01 10:06:43.609335209 +0000 2024-02-01T10:11:55.645 INFO:teuthology.orchestra.run.smithi148.stdout:Modify: 2024-02-01 10:06:43.357234139 +0000 2024-02-01T10:11:55.645 INFO:teuthology.orchestra.run.smithi148.stdout:Change: 2024-02-01 10:06:43.357234139 +0000 2024-02-01T10:11:55.645 INFO:teuthology.orchestra.run.smithi148.stdout: Birth: - 2024-02-01T10:11:55.645 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-01T10:11:55.698 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records in 2024-02-01T10:11:55.698 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records out 2024-02-01T10:11:55.698 INFO:teuthology.orchestra.run.smithi148.stderr:512 bytes copied, 0.000316747 s, 1.6 MB/s 2024-02-01T10:11:55.700 DEBUG:teuthology.orchestra.run.smithi148:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-01T10:11:55.749 DEBUG:teuthology.orchestra.run.smithi148:> stat /dev/vg_nvme/lv_2 2024-02-01T10:11:55.796 INFO:teuthology.orchestra.run.smithi148.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-01T10:11:55.796 INFO:teuthology.orchestra.run.smithi148.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-01T10:11:55.796 INFO:teuthology.orchestra.run.smithi148.stdout:Device: 5h/5d Inode: 882 Links: 1 2024-02-01T10:11:55.796 INFO:teuthology.orchestra.run.smithi148.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-01T10:11:55.796 INFO:teuthology.orchestra.run.smithi148.stdout:Access: 2024-02-01 10:06:44.085525037 +0000 2024-02-01T10:11:55.796 INFO:teuthology.orchestra.run.smithi148.stdout:Modify: 2024-02-01 10:06:43.837426652 +0000 2024-02-01T10:11:55.796 INFO:teuthology.orchestra.run.smithi148.stdout:Change: 2024-02-01 10:06:43.837426652 +0000 2024-02-01T10:11:55.797 INFO:teuthology.orchestra.run.smithi148.stdout: Birth: - 2024-02-01T10:11:55.797 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-01T10:11:55.850 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records in 2024-02-01T10:11:55.851 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records out 2024-02-01T10:11:55.851 INFO:teuthology.orchestra.run.smithi148.stderr:512 bytes copied, 0.000344182 s, 1.5 MB/s 2024-02-01T10:11:55.852 DEBUG:teuthology.orchestra.run.smithi148:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-01T10:11:55.905 DEBUG:teuthology.orchestra.run.smithi148:> stat /dev/vg_nvme/lv_3 2024-02-01T10:11:55.952 INFO:teuthology.orchestra.run.smithi148.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-01T10:11:55.952 INFO:teuthology.orchestra.run.smithi148.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-01T10:11:55.952 INFO:teuthology.orchestra.run.smithi148.stdout:Device: 5h/5d Inode: 906 Links: 1 2024-02-01T10:11:55.952 INFO:teuthology.orchestra.run.smithi148.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-01T10:11:55.952 INFO:teuthology.orchestra.run.smithi148.stdout:Access: 2024-02-01 10:06:44.541701997 +0000 2024-02-01T10:11:55.952 INFO:teuthology.orchestra.run.smithi148.stdout:Modify: 2024-02-01 10:06:44.317615069 +0000 2024-02-01T10:11:55.952 INFO:teuthology.orchestra.run.smithi148.stdout:Change: 2024-02-01 10:06:44.317615069 +0000 2024-02-01T10:11:55.952 INFO:teuthology.orchestra.run.smithi148.stdout: Birth: - 2024-02-01T10:11:55.952 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-01T10:11:56.006 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records in 2024-02-01T10:11:56.006 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records out 2024-02-01T10:11:56.006 INFO:teuthology.orchestra.run.smithi148.stderr:512 bytes copied, 0.000373413 s, 1.4 MB/s 2024-02-01T10:11:56.007 DEBUG:teuthology.orchestra.run.smithi148:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-01T10:11:56.057 DEBUG:teuthology.orchestra.run.smithi148:> stat /dev/vg_nvme/lv_4 2024-02-01T10:11:56.104 INFO:teuthology.orchestra.run.smithi148.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-01T10:11:56.104 INFO:teuthology.orchestra.run.smithi148.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-01T10:11:56.104 INFO:teuthology.orchestra.run.smithi148.stdout:Device: 5h/5d Inode: 921 Links: 1 2024-02-01T10:11:56.104 INFO:teuthology.orchestra.run.smithi148.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-01T10:11:56.104 INFO:teuthology.orchestra.run.smithi148.stdout:Access: 2024-02-01 10:06:45.013885015 +0000 2024-02-01T10:11:56.105 INFO:teuthology.orchestra.run.smithi148.stdout:Modify: 2024-02-01 10:06:44.757785819 +0000 2024-02-01T10:11:56.105 INFO:teuthology.orchestra.run.smithi148.stdout:Change: 2024-02-01 10:06:44.757785819 +0000 2024-02-01T10:11:56.105 INFO:teuthology.orchestra.run.smithi148.stdout: Birth: - 2024-02-01T10:11:56.105 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-01T10:11:56.158 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records in 2024-02-01T10:11:56.158 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records out 2024-02-01T10:11:56.159 INFO:teuthology.orchestra.run.smithi148.stderr:512 bytes copied, 0.000352699 s, 1.5 MB/s 2024-02-01T10:11:56.159 DEBUG:teuthology.orchestra.run.smithi148:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-01T10:11:56.209 INFO:tasks.ceph:osd dev map: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2024-02-01T10:11:56.209 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi148.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-02-01T10:11:56.209 INFO:tasks.ceph:Generating config... 2024-02-01T10:11:56.210 INFO:tasks.ceph:[client] debug rgw = 20 2024-02-01T10:11:56.210 INFO:tasks.ceph:[client] debug rgw sync = 20 2024-02-01T10:11:56.210 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2024-02-01T10:11:56.210 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2024-02-01T10:11:56.210 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2024-02-01T10:11:56.211 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2024-02-01T10:11:56.211 INFO:tasks.ceph:[client] rgw data log num shards = 4 2024-02-01T10:11:56.211 INFO:tasks.ceph:[client] rgw md log max shards = 4 2024-02-01T10:11:56.211 INFO:tasks.ceph:[client] rgw sync data inject err probability = 0.1 2024-02-01T10:11:56.211 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2024-02-01T10:11:56.211 INFO:tasks.ceph:[client] rgw sync meta inject err probability = 0.1 2024-02-01T10:11:56.211 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2024-02-01T10:11:56.211 INFO:tasks.ceph:[client] setgroup = ceph 2024-02-01T10:11:56.211 INFO:tasks.ceph:[client] setuser = ceph 2024-02-01T10:11:56.211 INFO:tasks.ceph:[client.0] rgw data notify interval msec = 0 2024-02-01T10:11:56.211 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-02-01T10:11:56.211 INFO:tasks.ceph:[mgr] debug ms = 1 2024-02-01T10:11:56.211 INFO:tasks.ceph:[mon] debug mon = 20 2024-02-01T10:11:56.211 INFO:tasks.ceph:[mon] debug ms = 1 2024-02-01T10:11:56.211 INFO:tasks.ceph:[mon] debug paxos = 20 2024-02-01T10:11:56.212 INFO:tasks.ceph:[osd] debug ms = 1 2024-02-01T10:11:56.212 INFO:tasks.ceph:[osd] debug osd = 20 2024-02-01T10:11:56.212 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2024-02-01T10:11:56.212 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2024-02-01T10:11:56.212 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2024-02-01T10:11:56.212 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2024-02-01T10:11:56.212 INFO:tasks.ceph:Setting up c1.mon.a... 2024-02-01T10:11:56.212 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c1.keyring 2024-02-01T10:11:56.282 INFO:teuthology.orchestra.run.smithi148.stdout:creating /etc/ceph/c1.keyring 2024-02-01T10:11:56.293 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c1.keyring 2024-02-01T10:11:56.373 DEBUG:teuthology.orchestra.run.smithi148:> sudo chmod 0644 /etc/ceph/c1.keyring 2024-02-01T10:11:56.426 DEBUG:tasks.ceph:Ceph mon addresses: [('c1.mon.a', '172.21.15.148')] 2024-02-01T10:11:56.427 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.148'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'client.0': {'rgw data notify interval msec': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2024-02-01T10:11:56.427 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-02-01T10:11:56.427 DEBUG:teuthology.orchestra.run.smithi148:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-02-01T10:11:56.473 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.148 --print /home/ubuntu/cephtest/c1.monmap 2024-02-01T10:11:56.552 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:11:56.552 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:11:56.553 INFO:teuthology.orchestra.run.smithi148.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c1.monmap 2024-02-01T10:11:56.553 INFO:teuthology.orchestra.run.smithi148.stdout:monmaptool: generated fsid 0c83d209-61c1-46c3-b2f4-1dd4a32516a3 2024-02-01T10:11:56.553 INFO:teuthology.orchestra.run.smithi148.stdout:setting min_mon_release = quincy 2024-02-01T10:11:56.553 INFO:teuthology.orchestra.run.smithi148.stdout:epoch 0 2024-02-01T10:11:56.553 INFO:teuthology.orchestra.run.smithi148.stdout:fsid 0c83d209-61c1-46c3-b2f4-1dd4a32516a3 2024-02-01T10:11:56.553 INFO:teuthology.orchestra.run.smithi148.stdout:last_changed 2024-02-01T10:11:56.555269+0000 2024-02-01T10:11:56.553 INFO:teuthology.orchestra.run.smithi148.stdout:created 2024-02-01T10:11:56.555269+0000 2024-02-01T10:11:56.553 INFO:teuthology.orchestra.run.smithi148.stdout:min_mon_release 17 (quincy) 2024-02-01T10:11:56.553 INFO:teuthology.orchestra.run.smithi148.stdout:election_strategy: 1 2024-02-01T10:11:56.553 INFO:teuthology.orchestra.run.smithi148.stdout:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2024-02-01T10:11:56.553 INFO:teuthology.orchestra.run.smithi148.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c1.monmap (1 monitors) 2024-02-01T10:11:56.555 DEBUG:teuthology.orchestra.run.smithi148:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-02-01T10:11:56.600 INFO:tasks.ceph:Writing /etc/ceph/c1.conf for FSID 0c83d209-61c1-46c3-b2f4-1dd4a32516a3... 2024-02-01T10:11:56.601 DEBUG:teuthology.orchestra.run.smithi148:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2024-02-01T10:11:56.645 DEBUG:teuthology.orchestra.run.smithi187:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2024-02-01T10:11:56.673 INFO:teuthology.orchestra.run.smithi148.stdout:[global] 2024-02-01T10:11:56.674 INFO:teuthology.orchestra.run.smithi148.stdout: chdir = "" 2024-02-01T10:11:56.674 INFO:teuthology.orchestra.run.smithi148.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-02-01T10:11:56.674 INFO:teuthology.orchestra.run.smithi148.stdout: auth supported = cephx 2024-02-01T10:11:56.674 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-02-01T10:11:56.674 INFO:teuthology.orchestra.run.smithi148.stdout: filestore xattr use omap = true 2024-02-01T10:11:56.674 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-02-01T10:11:56.674 INFO:teuthology.orchestra.run.smithi148.stdout: mon clock drift allowed = 1.000 2024-02-01T10:11:56.674 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-02-01T10:11:56.674 INFO:teuthology.orchestra.run.smithi148.stdout: osd crush chooseleaf type = 0 2024-02-01T10:11:56.674 INFO:teuthology.orchestra.run.smithi148.stdout: auth debug = true 2024-02-01T10:11:56.674 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-02-01T10:11:56.674 INFO:teuthology.orchestra.run.smithi148.stdout: ms die on old message = true 2024-02-01T10:11:56.674 INFO:teuthology.orchestra.run.smithi148.stdout: ms die on bug = true 2024-02-01T10:11:56.674 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-02-01T10:11:56.675 INFO:teuthology.orchestra.run.smithi148.stdout: mon max pg per osd = 10000 # >= luminous 2024-02-01T10:11:56.675 INFO:teuthology.orchestra.run.smithi148.stdout: mon pg warn max object skew = 0 2024-02-01T10:11:56.675 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-02-01T10:11:56.675 INFO:teuthology.orchestra.run.smithi148.stdout: # disable pg_autoscaler by default for new pools 2024-02-01T10:11:56.675 INFO:teuthology.orchestra.run.smithi148.stdout: osd_pool_default_pg_autoscale_mode = off 2024-02-01T10:11:56.675 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-02-01T10:11:56.675 INFO:teuthology.orchestra.run.smithi148.stdout: osd pool default size = 2 2024-02-01T10:11:56.675 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-02-01T10:11:56.675 INFO:teuthology.orchestra.run.smithi148.stdout: mon osd allow primary affinity = true 2024-02-01T10:11:56.675 INFO:teuthology.orchestra.run.smithi148.stdout: mon osd allow pg remap = true 2024-02-01T10:11:56.675 INFO:teuthology.orchestra.run.smithi148.stdout: mon warn on legacy crush tunables = false 2024-02-01T10:11:56.675 INFO:teuthology.orchestra.run.smithi148.stdout: mon warn on crush straw calc version zero = false 2024-02-01T10:11:56.675 INFO:teuthology.orchestra.run.smithi148.stdout: mon warn on no sortbitwise = false 2024-02-01T10:11:56.675 INFO:teuthology.orchestra.run.smithi148.stdout: mon warn on osd down out interval zero = false 2024-02-01T10:11:56.676 INFO:teuthology.orchestra.run.smithi148.stdout: mon warn on too few osds = false 2024-02-01T10:11:56.676 INFO:teuthology.orchestra.run.smithi148.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-02-01T10:11:56.676 INFO:teuthology.orchestra.run.smithi148.stdout: mon_warn_on_pool_no_redundancy = false 2024-02-01T10:11:56.676 INFO:teuthology.orchestra.run.smithi148.stdout: mon_allow_pool_size_one = true 2024-02-01T10:11:56.676 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-02-01T10:11:56.676 INFO:teuthology.orchestra.run.smithi148.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-02-01T10:11:56.676 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-02-01T10:11:56.676 INFO:teuthology.orchestra.run.smithi148.stdout: osd default data pool replay window = 5 2024-02-01T10:11:56.676 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-02-01T10:11:56.676 INFO:teuthology.orchestra.run.smithi148.stdout: mon allow pool delete = true 2024-02-01T10:11:56.676 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-02-01T10:11:56.676 INFO:teuthology.orchestra.run.smithi148.stdout: mon cluster log file level = debug 2024-02-01T10:11:56.676 INFO:teuthology.orchestra.run.smithi148.stdout: debug asserts on shutdown = true 2024-02-01T10:11:56.676 INFO:teuthology.orchestra.run.smithi148.stdout: mon health detail to clog = false 2024-02-01T10:11:56.677 INFO:teuthology.orchestra.run.smithi148.stdout: mon host = 172.21.15.148 2024-02-01T10:11:56.677 INFO:teuthology.orchestra.run.smithi148.stdout: fsid = 0c83d209-61c1-46c3-b2f4-1dd4a32516a3 2024-02-01T10:11:56.677 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-02-01T10:11:56.677 INFO:teuthology.orchestra.run.smithi148.stdout:[osd] 2024-02-01T10:11:56.677 INFO:teuthology.orchestra.run.smithi148.stdout: osd journal size = 100 2024-02-01T10:11:56.677 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-02-01T10:11:56.677 INFO:teuthology.orchestra.run.smithi148.stdout: osd scrub load threshold = 5.0 2024-02-01T10:11:56.677 INFO:teuthology.orchestra.run.smithi148.stdout: osd scrub max interval = 600 2024-02-01T10:11:56.677 INFO:teuthology.orchestra.run.smithi148.stdout: osd mclock profile = high_recovery_ops 2024-02-01T10:11:56.677 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-02-01T10:11:56.677 INFO:teuthology.orchestra.run.smithi148.stdout: osd recover clone overlap = true 2024-02-01T10:11:56.677 INFO:teuthology.orchestra.run.smithi148.stdout: osd recovery max chunk = 1048576 2024-02-01T10:11:56.677 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-02-01T10:11:56.677 INFO:teuthology.orchestra.run.smithi148.stdout: osd debug shutdown = true 2024-02-01T10:11:56.678 INFO:teuthology.orchestra.run.smithi148.stdout: osd debug op order = true 2024-02-01T10:11:56.678 INFO:teuthology.orchestra.run.smithi148.stdout: osd debug verify stray on activate = true 2024-02-01T10:11:56.678 INFO:teuthology.orchestra.run.smithi148.stdout: osd debug trim objects = true 2024-02-01T10:11:56.678 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-02-01T10:11:56.678 INFO:teuthology.orchestra.run.smithi148.stdout: osd open classes on start = true 2024-02-01T10:11:56.678 INFO:teuthology.orchestra.run.smithi148.stdout: osd debug pg log writeout = true 2024-02-01T10:11:56.678 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-02-01T10:11:56.678 INFO:teuthology.orchestra.run.smithi148.stdout: osd deep scrub update digest min age = 30 2024-02-01T10:11:56.678 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-02-01T10:11:56.678 INFO:teuthology.orchestra.run.smithi148.stdout: osd map max advance = 10 2024-02-01T10:11:56.678 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-02-01T10:11:56.678 INFO:teuthology.orchestra.run.smithi148.stdout: journal zero on create = true 2024-02-01T10:11:56.678 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-02-01T10:11:56.678 INFO:teuthology.orchestra.run.smithi148.stdout: filestore ondisk finisher threads = 3 2024-02-01T10:11:56.678 INFO:teuthology.orchestra.run.smithi148.stdout: filestore apply finisher threads = 3 2024-02-01T10:11:56.679 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-02-01T10:11:56.679 INFO:teuthology.orchestra.run.smithi148.stdout: bdev debug aio = true 2024-02-01T10:11:56.679 INFO:teuthology.orchestra.run.smithi148.stdout: osd debug misdirected ops = true 2024-02-01T10:11:56.679 INFO:teuthology.orchestra.run.smithi148.stdout: debug ms = 1 2024-02-01T10:11:56.679 INFO:teuthology.orchestra.run.smithi148.stdout: debug osd = 20 2024-02-01T10:11:56.679 INFO:teuthology.orchestra.run.smithi148.stdout: rocksdb delete range threshold = 0 2024-02-01T10:11:56.679 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-02-01T10:11:56.679 INFO:teuthology.orchestra.run.smithi148.stdout:[mgr] 2024-02-01T10:11:56.679 INFO:teuthology.orchestra.run.smithi148.stdout: debug ms = 1 2024-02-01T10:11:56.679 INFO:teuthology.orchestra.run.smithi148.stdout: debug mgr = 20 2024-02-01T10:11:56.679 INFO:teuthology.orchestra.run.smithi148.stdout: debug mon = 20 2024-02-01T10:11:56.679 INFO:teuthology.orchestra.run.smithi148.stdout: debug auth = 20 2024-02-01T10:11:56.679 INFO:teuthology.orchestra.run.smithi148.stdout: mon reweight min pgs per osd = 4 2024-02-01T10:11:56.679 INFO:teuthology.orchestra.run.smithi148.stdout: mon reweight min bytes per osd = 10 2024-02-01T10:11:56.680 INFO:teuthology.orchestra.run.smithi148.stdout: mgr/telemetry/nag = false 2024-02-01T10:11:56.680 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-02-01T10:11:56.680 INFO:teuthology.orchestra.run.smithi148.stdout:[mon] 2024-02-01T10:11:56.680 INFO:teuthology.orchestra.run.smithi148.stdout: debug ms = 1 2024-02-01T10:11:56.680 INFO:teuthology.orchestra.run.smithi148.stdout: debug mon = 20 2024-02-01T10:11:56.680 INFO:teuthology.orchestra.run.smithi148.stdout: debug paxos = 20 2024-02-01T10:11:56.680 INFO:teuthology.orchestra.run.smithi148.stdout: debug auth = 20 2024-02-01T10:11:56.680 INFO:teuthology.orchestra.run.smithi148.stdout: mon data avail warn = 5 2024-02-01T10:11:56.680 INFO:teuthology.orchestra.run.smithi148.stdout: mon mgr mkfs grace = 240 2024-02-01T10:11:56.680 INFO:teuthology.orchestra.run.smithi148.stdout: mon reweight min pgs per osd = 4 2024-02-01T10:11:56.680 INFO:teuthology.orchestra.run.smithi148.stdout: mon osd reporter subtree level = osd 2024-02-01T10:11:56.680 INFO:teuthology.orchestra.run.smithi148.stdout: mon osd prime pg temp = true 2024-02-01T10:11:56.680 INFO:teuthology.orchestra.run.smithi148.stdout: mon reweight min bytes per osd = 10 2024-02-01T10:11:56.680 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-02-01T10:11:56.680 INFO:teuthology.orchestra.run.smithi148.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-02-01T10:11:56.681 INFO:teuthology.orchestra.run.smithi148.stdout: auth mon ticket ttl = 660 # 11m 2024-02-01T10:11:56.681 INFO:teuthology.orchestra.run.smithi148.stdout: auth service ticket ttl = 240 # 4m 2024-02-01T10:11:56.681 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-02-01T10:11:56.681 INFO:teuthology.orchestra.run.smithi148.stdout: # don't complain about insecure global_id in the test suite 2024-02-01T10:11:56.681 INFO:teuthology.orchestra.run.smithi148.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-02-01T10:11:56.681 INFO:teuthology.orchestra.run.smithi148.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-02-01T10:11:56.681 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-02-01T10:11:56.681 INFO:teuthology.orchestra.run.smithi148.stdout: # 1m isn't quite enough 2024-02-01T10:11:56.681 INFO:teuthology.orchestra.run.smithi148.stdout: mon_down_mkfs_grace = 2m 2024-02-01T10:11:56.681 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-02-01T10:11:56.681 INFO:teuthology.orchestra.run.smithi148.stdout: mon_warn_on_filestore_osds = false 2024-02-01T10:11:56.681 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-02-01T10:11:56.681 INFO:teuthology.orchestra.run.smithi148.stdout:[client] 2024-02-01T10:11:56.681 INFO:teuthology.orchestra.run.smithi148.stdout: rgw cache enabled = true 2024-02-01T10:11:56.681 INFO:teuthology.orchestra.run.smithi148.stdout: rgw enable ops log = true 2024-02-01T10:11:56.682 INFO:teuthology.orchestra.run.smithi148.stdout: rgw enable usage log = true 2024-02-01T10:11:56.682 INFO:teuthology.orchestra.run.smithi148.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-02-01T10:11:56.682 INFO:teuthology.orchestra.run.smithi148.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-02-01T10:11:56.682 INFO:teuthology.orchestra.run.smithi148.stdout: debug rgw = 20 2024-02-01T10:11:56.682 INFO:teuthology.orchestra.run.smithi148.stdout: debug rgw sync = 20 2024-02-01T10:11:56.682 INFO:teuthology.orchestra.run.smithi148.stdout: rgw crypt require ssl = False 2024-02-01T10:11:56.682 INFO:teuthology.orchestra.run.smithi148.stdout: rgw crypt s3 kms backend = testing 2024-02-01T10:11:56.682 INFO:teuthology.orchestra.run.smithi148.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2024-02-01T10:11:56.682 INFO:teuthology.orchestra.run.smithi148.stdout: rgw curl low speed time = 300 2024-02-01T10:11:56.682 INFO:teuthology.orchestra.run.smithi148.stdout: rgw data log num shards = 4 2024-02-01T10:11:56.682 INFO:teuthology.orchestra.run.smithi148.stdout: rgw md log max shards = 4 2024-02-01T10:11:56.682 INFO:teuthology.orchestra.run.smithi148.stdout: rgw sync data inject err probability = 0.1 2024-02-01T10:11:56.682 INFO:teuthology.orchestra.run.smithi148.stdout: rgw sync log trim interval = 0 2024-02-01T10:11:56.683 INFO:teuthology.orchestra.run.smithi148.stdout: rgw sync meta inject err probability = 0.1 2024-02-01T10:11:56.683 INFO:teuthology.orchestra.run.smithi148.stdout: rgw sync obj etag verify = True 2024-02-01T10:11:56.683 INFO:teuthology.orchestra.run.smithi148.stdout: setgroup = ceph 2024-02-01T10:11:56.683 INFO:teuthology.orchestra.run.smithi148.stdout: setuser = ceph 2024-02-01T10:11:56.683 INFO:teuthology.orchestra.run.smithi148.stdout:[mon.a] 2024-02-01T10:11:56.683 INFO:teuthology.orchestra.run.smithi148.stdout:[client.0] 2024-02-01T10:11:56.683 INFO:teuthology.orchestra.run.smithi148.stdout: rgw data notify interval msec = 0 2024-02-01T10:11:56.683 INFO:teuthology.orchestra.run.smithi148.stdout:[osd.0] 2024-02-01T10:11:56.683 INFO:teuthology.orchestra.run.smithi148.stdout: osd_max_omap_entries_per_request = 10 2024-02-01T10:11:56.683 INFO:teuthology.orchestra.run.smithi148.stdout:[osd.1] 2024-02-01T10:11:56.683 INFO:teuthology.orchestra.run.smithi148.stdout: osd_max_omap_entries_per_request = 1000 2024-02-01T10:11:56.683 INFO:teuthology.orchestra.run.smithi148.stdout:[osd.2] 2024-02-01T10:11:56.683 INFO:teuthology.orchestra.run.smithi148.stdout: osd_max_omap_entries_per_request = 10000 2024-02-01T10:11:56.684 INFO:teuthology.orchestra.run.smithi187.stdout:[global] 2024-02-01T10:11:56.684 INFO:teuthology.orchestra.run.smithi187.stdout: chdir = "" 2024-02-01T10:11:56.684 INFO:teuthology.orchestra.run.smithi187.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-02-01T10:11:56.684 INFO:teuthology.orchestra.run.smithi187.stdout: auth supported = cephx 2024-02-01T10:11:56.684 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-02-01T10:11:56.684 INFO:teuthology.orchestra.run.smithi187.stdout: filestore xattr use omap = true 2024-02-01T10:11:56.685 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-02-01T10:11:56.685 INFO:teuthology.orchestra.run.smithi187.stdout: mon clock drift allowed = 1.000 2024-02-01T10:11:56.685 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-02-01T10:11:56.685 INFO:teuthology.orchestra.run.smithi187.stdout: osd crush chooseleaf type = 0 2024-02-01T10:11:56.685 INFO:teuthology.orchestra.run.smithi187.stdout: auth debug = true 2024-02-01T10:11:56.685 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-02-01T10:11:56.685 INFO:teuthology.orchestra.run.smithi187.stdout: ms die on old message = true 2024-02-01T10:11:56.685 INFO:teuthology.orchestra.run.smithi187.stdout: ms die on bug = true 2024-02-01T10:11:56.685 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-02-01T10:11:56.685 INFO:teuthology.orchestra.run.smithi187.stdout: mon max pg per osd = 10000 # >= luminous 2024-02-01T10:11:56.685 INFO:teuthology.orchestra.run.smithi187.stdout: mon pg warn max object skew = 0 2024-02-01T10:11:56.685 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-02-01T10:11:56.685 INFO:teuthology.orchestra.run.smithi187.stdout: # disable pg_autoscaler by default for new pools 2024-02-01T10:11:56.685 INFO:teuthology.orchestra.run.smithi187.stdout: osd_pool_default_pg_autoscale_mode = off 2024-02-01T10:11:56.686 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-02-01T10:11:56.686 INFO:teuthology.orchestra.run.smithi187.stdout: osd pool default size = 2 2024-02-01T10:11:56.686 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-02-01T10:11:56.686 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd allow primary affinity = true 2024-02-01T10:11:56.686 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd allow pg remap = true 2024-02-01T10:11:56.686 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on legacy crush tunables = false 2024-02-01T10:11:56.686 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on crush straw calc version zero = false 2024-02-01T10:11:56.686 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on no sortbitwise = false 2024-02-01T10:11:56.686 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on osd down out interval zero = false 2024-02-01T10:11:56.686 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on too few osds = false 2024-02-01T10:11:56.686 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-02-01T10:11:56.686 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_pool_no_redundancy = false 2024-02-01T10:11:56.686 INFO:teuthology.orchestra.run.smithi187.stdout: mon_allow_pool_size_one = true 2024-02-01T10:11:56.686 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-02-01T10:11:56.686 INFO:teuthology.orchestra.run.smithi187.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-02-01T10:11:56.687 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-02-01T10:11:56.687 INFO:teuthology.orchestra.run.smithi187.stdout: osd default data pool replay window = 5 2024-02-01T10:11:56.687 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-02-01T10:11:56.687 INFO:teuthology.orchestra.run.smithi187.stdout: mon allow pool delete = true 2024-02-01T10:11:56.687 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-02-01T10:11:56.687 INFO:teuthology.orchestra.run.smithi187.stdout: mon cluster log file level = debug 2024-02-01T10:11:56.687 INFO:teuthology.orchestra.run.smithi187.stdout: debug asserts on shutdown = true 2024-02-01T10:11:56.687 INFO:teuthology.orchestra.run.smithi187.stdout: mon health detail to clog = false 2024-02-01T10:11:56.687 INFO:teuthology.orchestra.run.smithi187.stdout: mon host = 172.21.15.148 2024-02-01T10:11:56.687 INFO:teuthology.orchestra.run.smithi187.stdout: fsid = 0c83d209-61c1-46c3-b2f4-1dd4a32516a3 2024-02-01T10:11:56.687 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-02-01T10:11:56.687 INFO:teuthology.orchestra.run.smithi187.stdout:[osd] 2024-02-01T10:11:56.687 INFO:teuthology.orchestra.run.smithi187.stdout: osd journal size = 100 2024-02-01T10:11:56.687 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-02-01T10:11:56.688 INFO:teuthology.orchestra.run.smithi187.stdout: osd scrub load threshold = 5.0 2024-02-01T10:11:56.688 INFO:teuthology.orchestra.run.smithi187.stdout: osd scrub max interval = 600 2024-02-01T10:11:56.688 INFO:teuthology.orchestra.run.smithi187.stdout: osd mclock profile = high_recovery_ops 2024-02-01T10:11:56.688 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-02-01T10:11:56.688 INFO:teuthology.orchestra.run.smithi187.stdout: osd recover clone overlap = true 2024-02-01T10:11:56.688 INFO:teuthology.orchestra.run.smithi187.stdout: osd recovery max chunk = 1048576 2024-02-01T10:11:56.688 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-02-01T10:11:56.688 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug shutdown = true 2024-02-01T10:11:56.688 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug op order = true 2024-02-01T10:11:56.688 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug verify stray on activate = true 2024-02-01T10:11:56.688 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug trim objects = true 2024-02-01T10:11:56.688 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-02-01T10:11:56.688 INFO:teuthology.orchestra.run.smithi187.stdout: osd open classes on start = true 2024-02-01T10:11:56.689 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug pg log writeout = true 2024-02-01T10:11:56.689 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-02-01T10:11:56.689 INFO:teuthology.orchestra.run.smithi187.stdout: osd deep scrub update digest min age = 30 2024-02-01T10:11:56.689 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-02-01T10:11:56.689 INFO:teuthology.orchestra.run.smithi187.stdout: osd map max advance = 10 2024-02-01T10:11:56.689 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-02-01T10:11:56.689 INFO:teuthology.orchestra.run.smithi187.stdout: journal zero on create = true 2024-02-01T10:11:56.689 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-02-01T10:11:56.689 INFO:teuthology.orchestra.run.smithi187.stdout: filestore ondisk finisher threads = 3 2024-02-01T10:11:56.689 INFO:teuthology.orchestra.run.smithi187.stdout: filestore apply finisher threads = 3 2024-02-01T10:11:56.689 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-02-01T10:11:56.689 INFO:teuthology.orchestra.run.smithi187.stdout: bdev debug aio = true 2024-02-01T10:11:56.689 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug misdirected ops = true 2024-02-01T10:11:56.689 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2024-02-01T10:11:56.689 INFO:teuthology.orchestra.run.smithi187.stdout: debug osd = 20 2024-02-01T10:11:56.690 INFO:teuthology.orchestra.run.smithi187.stdout: rocksdb delete range threshold = 0 2024-02-01T10:11:56.690 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-02-01T10:11:56.690 INFO:teuthology.orchestra.run.smithi187.stdout:[mgr] 2024-02-01T10:11:56.690 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2024-02-01T10:11:56.690 INFO:teuthology.orchestra.run.smithi187.stdout: debug mgr = 20 2024-02-01T10:11:56.690 INFO:teuthology.orchestra.run.smithi187.stdout: debug mon = 20 2024-02-01T10:11:56.690 INFO:teuthology.orchestra.run.smithi187.stdout: debug auth = 20 2024-02-01T10:11:56.690 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min pgs per osd = 4 2024-02-01T10:11:56.690 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min bytes per osd = 10 2024-02-01T10:11:56.690 INFO:teuthology.orchestra.run.smithi187.stdout: mgr/telemetry/nag = false 2024-02-01T10:11:56.690 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-02-01T10:11:56.690 INFO:teuthology.orchestra.run.smithi187.stdout:[mon] 2024-02-01T10:11:56.690 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2024-02-01T10:11:56.690 INFO:teuthology.orchestra.run.smithi187.stdout: debug mon = 20 2024-02-01T10:11:56.690 INFO:teuthology.orchestra.run.smithi187.stdout: debug paxos = 20 2024-02-01T10:11:56.691 INFO:teuthology.orchestra.run.smithi187.stdout: debug auth = 20 2024-02-01T10:11:56.691 INFO:teuthology.orchestra.run.smithi187.stdout: mon data avail warn = 5 2024-02-01T10:11:56.691 INFO:teuthology.orchestra.run.smithi187.stdout: mon mgr mkfs grace = 240 2024-02-01T10:11:56.691 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min pgs per osd = 4 2024-02-01T10:11:56.691 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd reporter subtree level = osd 2024-02-01T10:11:56.691 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd prime pg temp = true 2024-02-01T10:11:56.691 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min bytes per osd = 10 2024-02-01T10:11:56.691 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-02-01T10:11:56.691 INFO:teuthology.orchestra.run.smithi187.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-02-01T10:11:56.691 INFO:teuthology.orchestra.run.smithi187.stdout: auth mon ticket ttl = 660 # 11m 2024-02-01T10:11:56.691 INFO:teuthology.orchestra.run.smithi187.stdout: auth service ticket ttl = 240 # 4m 2024-02-01T10:11:56.691 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-02-01T10:11:56.691 INFO:teuthology.orchestra.run.smithi187.stdout: # don't complain about insecure global_id in the test suite 2024-02-01T10:11:56.691 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-02-01T10:11:56.692 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-02-01T10:11:56.692 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-02-01T10:11:56.692 INFO:teuthology.orchestra.run.smithi187.stdout: # 1m isn't quite enough 2024-02-01T10:11:56.692 INFO:teuthology.orchestra.run.smithi187.stdout: mon_down_mkfs_grace = 2m 2024-02-01T10:11:56.692 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-02-01T10:11:56.692 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_filestore_osds = false 2024-02-01T10:11:56.692 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-02-01T10:11:56.692 INFO:teuthology.orchestra.run.smithi187.stdout:[client] 2024-02-01T10:11:56.692 INFO:teuthology.orchestra.run.smithi187.stdout: rgw cache enabled = true 2024-02-01T10:11:56.692 INFO:teuthology.orchestra.run.smithi187.stdout: rgw enable ops log = true 2024-02-01T10:11:56.692 INFO:teuthology.orchestra.run.smithi187.stdout: rgw enable usage log = true 2024-02-01T10:11:56.692 INFO:teuthology.orchestra.run.smithi187.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-02-01T10:11:56.692 INFO:teuthology.orchestra.run.smithi187.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-02-01T10:11:56.692 INFO:teuthology.orchestra.run.smithi187.stdout: debug rgw = 20 2024-02-01T10:11:56.693 INFO:teuthology.orchestra.run.smithi187.stdout: debug rgw sync = 20 2024-02-01T10:11:56.693 INFO:teuthology.orchestra.run.smithi187.stdout: rgw crypt require ssl = False 2024-02-01T10:11:56.693 INFO:teuthology.orchestra.run.smithi187.stdout: rgw crypt s3 kms backend = testing 2024-02-01T10:11:56.693 INFO:teuthology.orchestra.run.smithi187.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2024-02-01T10:11:56.693 INFO:teuthology.orchestra.run.smithi187.stdout: rgw curl low speed time = 300 2024-02-01T10:11:56.693 INFO:teuthology.orchestra.run.smithi187.stdout: rgw data log num shards = 4 2024-02-01T10:11:56.693 INFO:teuthology.orchestra.run.smithi187.stdout: rgw md log max shards = 4 2024-02-01T10:11:56.693 INFO:teuthology.orchestra.run.smithi187.stdout: rgw sync data inject err probability = 0.1 2024-02-01T10:11:56.693 INFO:teuthology.orchestra.run.smithi187.stdout: rgw sync log trim interval = 0 2024-02-01T10:11:56.693 INFO:teuthology.orchestra.run.smithi187.stdout: rgw sync meta inject err probability = 0.1 2024-02-01T10:11:56.693 INFO:teuthology.orchestra.run.smithi187.stdout: rgw sync obj etag verify = True 2024-02-01T10:11:56.693 INFO:teuthology.orchestra.run.smithi187.stdout: setgroup = ceph 2024-02-01T10:11:56.693 INFO:teuthology.orchestra.run.smithi187.stdout: setuser = ceph 2024-02-01T10:11:56.693 INFO:teuthology.orchestra.run.smithi187.stdout:[mon.a] 2024-02-01T10:11:56.693 INFO:teuthology.orchestra.run.smithi187.stdout:[client.0] 2024-02-01T10:11:56.694 INFO:teuthology.orchestra.run.smithi187.stdout: rgw data notify interval msec = 0 2024-02-01T10:11:56.694 INFO:teuthology.orchestra.run.smithi187.stdout:[osd.0] 2024-02-01T10:11:56.694 INFO:teuthology.orchestra.run.smithi187.stdout: osd_max_omap_entries_per_request = 10 2024-02-01T10:11:56.694 INFO:teuthology.orchestra.run.smithi187.stdout:[osd.1] 2024-02-01T10:11:56.694 INFO:teuthology.orchestra.run.smithi187.stdout: osd_max_omap_entries_per_request = 1000 2024-02-01T10:11:56.694 INFO:teuthology.orchestra.run.smithi187.stdout:[osd.2] 2024-02-01T10:11:56.694 INFO:teuthology.orchestra.run.smithi187.stdout: osd_max_omap_entries_per_request = 10000 2024-02-01T10:11:56.695 INFO:tasks.ceph:Creating admin key on c1.mon.a... 2024-02-01T10:11:56.696 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c1.keyring 2024-02-01T10:11:56.789 INFO:tasks.ceph:Copying monmap to all nodes... 2024-02-01T10:11:56.789 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-02-01T10:11:56.789 DEBUG:teuthology.orchestra.run.smithi148:> dd if=/etc/ceph/c1.keyring of=/dev/stdout 2024-02-01T10:11:56.796 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-02-01T10:11:56.796 DEBUG:teuthology.orchestra.run.smithi148:> dd if=/home/ubuntu/cephtest/c1.monmap of=/dev/stdout 2024-02-01T10:11:56.844 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi148.front.sepia.ceph.com 2024-02-01T10:11:56.844 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-02-01T10:11:56.844 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/etc/ceph/c1.keyring 2024-02-01T10:11:56.844 DEBUG:teuthology.orchestra.run.smithi148:> sudo chmod 0644 /etc/ceph/c1.keyring 2024-02-01T10:11:56.907 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-02-01T10:11:56.907 DEBUG:teuthology.orchestra.run.smithi148:> dd of=/home/ubuntu/cephtest/c1.monmap 2024-02-01T10:11:56.957 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi187.front.sepia.ceph.com 2024-02-01T10:11:56.957 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-02-01T10:11:56.957 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/etc/ceph/c1.keyring 2024-02-01T10:11:56.957 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod 0644 /etc/ceph/c1.keyring 2024-02-01T10:11:56.980 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-02-01T10:11:56.980 DEBUG:teuthology.orchestra.run.smithi187:> dd of=/home/ubuntu/cephtest/c1.monmap 2024-02-01T10:11:57.028 INFO:tasks.ceph:Setting up mon nodes... 2024-02-01T10:11:57.028 INFO:tasks.ceph:Setting up mgr nodes... 2024-02-01T10:11:57.028 DEBUG:teuthology.orchestra.run.smithi148:> sudo mkdir -p /var/lib/ceph/mgr/c1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c1-x/keyring 2024-02-01T10:11:57.070 INFO:teuthology.orchestra.run.smithi148.stdout:creating /var/lib/ceph/mgr/c1-x/keyring 2024-02-01T10:11:57.075 INFO:tasks.ceph:Setting up mds nodes... 2024-02-01T10:11:57.075 INFO:tasks.ceph_client:Setting up client nodes... 2024-02-01T10:11:57.075 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c1.client.0.keyring && sudo chmod 0644 /etc/ceph/c1.client.0.keyring 2024-02-01T10:11:57.149 INFO:teuthology.orchestra.run.smithi148.stdout:creating /etc/ceph/c1.client.0.keyring 2024-02-01T10:11:57.160 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c1.client.1.keyring && sudo chmod 0644 /etc/ceph/c1.client.1.keyring 2024-02-01T10:11:57.234 INFO:teuthology.orchestra.run.smithi148.stdout:creating /etc/ceph/c1.client.1.keyring 2024-02-01T10:11:57.244 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-02-01T10:11:57.244 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi148.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-02-01T10:11:57.245 DEBUG:teuthology.orchestra.run.smithi148:> sudo mkdir -p /var/lib/ceph/osd/c1-0 2024-02-01T10:11:57.298 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2024-02-01T10:11:57.298 INFO:tasks.ceph:role: c1.osd.0 2024-02-01T10:11:57.298 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi148.front.sepia.ceph.com 2024-02-01T10:11:57.299 DEBUG:teuthology.orchestra.run.smithi148:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-02-01T10:11:57.359 INFO:teuthology.orchestra.run.smithi148.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-02-01T10:11:57.359 INFO:teuthology.orchestra.run.smithi148.stdout: = sectsz=512 attr=2, projid32bit=1 2024-02-01T10:11:57.359 INFO:teuthology.orchestra.run.smithi148.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-02-01T10:11:57.359 INFO:teuthology.orchestra.run.smithi148.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-02-01T10:11:57.360 INFO:teuthology.orchestra.run.smithi148.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-02-01T10:11:57.360 INFO:teuthology.orchestra.run.smithi148.stdout: = sunit=0 swidth=0 blks 2024-02-01T10:11:57.360 INFO:teuthology.orchestra.run.smithi148.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-02-01T10:11:57.360 INFO:teuthology.orchestra.run.smithi148.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-02-01T10:11:57.360 INFO:teuthology.orchestra.run.smithi148.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-02-01T10:11:57.360 INFO:teuthology.orchestra.run.smithi148.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-02-01T10:11:57.821 INFO:teuthology.orchestra.run.smithi148.stdout:Discarding blocks...Done. 2024-02-01T10:11:57.823 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi148.front.sepia.ceph.com -o noatime 2024-02-01T10:11:57.823 DEBUG:teuthology.orchestra.run.smithi148:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c1-0 2024-02-01T10:11:57.916 DEBUG:teuthology.orchestra.run.smithi148:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-0 2024-02-01T10:11:57.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-01T10:11:57.926 INFO:teuthology.orchestra.run.smithi148.stderr:sudo: /sbin/restorecon: command not found 2024-02-01T10:11:57.926 DEBUG:teuthology.orchestra.run.smithi148:> sudo mkdir -p /var/lib/ceph/osd/c1-1 2024-02-01T10:11:57.977 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2024-02-01T10:11:57.977 INFO:tasks.ceph:role: c1.osd.1 2024-02-01T10:11:57.978 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi148.front.sepia.ceph.com 2024-02-01T10:11:57.978 DEBUG:teuthology.orchestra.run.smithi148:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-02-01T10:11:58.040 INFO:teuthology.orchestra.run.smithi148.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-02-01T10:11:58.040 INFO:teuthology.orchestra.run.smithi148.stdout: = sectsz=512 attr=2, projid32bit=1 2024-02-01T10:11:58.040 INFO:teuthology.orchestra.run.smithi148.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-02-01T10:11:58.040 INFO:teuthology.orchestra.run.smithi148.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-02-01T10:11:58.041 INFO:teuthology.orchestra.run.smithi148.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-02-01T10:11:58.041 INFO:teuthology.orchestra.run.smithi148.stdout: = sunit=0 swidth=0 blks 2024-02-01T10:11:58.041 INFO:teuthology.orchestra.run.smithi148.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-02-01T10:11:58.041 INFO:teuthology.orchestra.run.smithi148.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-02-01T10:11:58.041 INFO:teuthology.orchestra.run.smithi148.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-02-01T10:11:58.041 INFO:teuthology.orchestra.run.smithi148.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-02-01T10:11:58.503 INFO:teuthology.orchestra.run.smithi148.stdout:Discarding blocks...Done. 2024-02-01T10:11:58.505 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi148.front.sepia.ceph.com -o noatime 2024-02-01T10:11:58.506 DEBUG:teuthology.orchestra.run.smithi148:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c1-1 2024-02-01T10:11:58.646 DEBUG:teuthology.orchestra.run.smithi148:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-1 2024-02-01T10:11:58.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-01T10:11:58.658 INFO:teuthology.orchestra.run.smithi148.stderr:sudo: /sbin/restorecon: command not found 2024-02-01T10:11:58.659 DEBUG:teuthology.orchestra.run.smithi148:> sudo mkdir -p /var/lib/ceph/osd/c1-2 2024-02-01T10:11:58.712 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2024-02-01T10:11:58.712 INFO:tasks.ceph:role: c1.osd.2 2024-02-01T10:11:58.712 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi148.front.sepia.ceph.com 2024-02-01T10:11:58.713 DEBUG:teuthology.orchestra.run.smithi148:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-02-01T10:11:58.776 INFO:teuthology.orchestra.run.smithi148.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-02-01T10:11:58.776 INFO:teuthology.orchestra.run.smithi148.stdout: = sectsz=512 attr=2, projid32bit=1 2024-02-01T10:11:58.776 INFO:teuthology.orchestra.run.smithi148.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-02-01T10:11:58.776 INFO:teuthology.orchestra.run.smithi148.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-02-01T10:11:58.776 INFO:teuthology.orchestra.run.smithi148.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-02-01T10:11:58.777 INFO:teuthology.orchestra.run.smithi148.stdout: = sunit=0 swidth=0 blks 2024-02-01T10:11:58.777 INFO:teuthology.orchestra.run.smithi148.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-02-01T10:11:58.777 INFO:teuthology.orchestra.run.smithi148.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-02-01T10:11:58.777 INFO:teuthology.orchestra.run.smithi148.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-02-01T10:11:58.777 INFO:teuthology.orchestra.run.smithi148.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-02-01T10:11:59.272 INFO:teuthology.orchestra.run.smithi148.stdout:Discarding blocks...Done. 2024-02-01T10:11:59.274 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi148.front.sepia.ceph.com -o noatime 2024-02-01T10:11:59.274 DEBUG:teuthology.orchestra.run.smithi148:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c1-2 2024-02-01T10:11:59.423 DEBUG:teuthology.orchestra.run.smithi148:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-2 2024-02-01T10:11:59.434 INFO:teuthology.orchestra.run.smithi148.stderr:sudo: /sbin/restorecon: command not found 2024-02-01T10:11:59.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-01T10:11:59.435 DEBUG:teuthology.orchestra.run.smithi148:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c1.monmap 2024-02-01T10:11:59.549 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:11:59.548+0000 7fae45f807c0 -1 auth: error reading file: /var/lib/ceph/osd/c1-0/keyring: can't open /var/lib/ceph/osd/c1-0/keyring: (2) No such file or directory 2024-02-01T10:11:59.550 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:11:59.548+0000 7fae45f807c0 -1 created new key in keyring /var/lib/ceph/osd/c1-0/keyring 2024-02-01T10:11:59.550 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:11:59.548+0000 7fae45f807c0 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2024-02-01T10:11:59.550 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:11:59.548+0000 7fae45f807c0 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2024-02-01T10:11:59.550 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:11:59.548+0000 7fae45f807c0 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2024-02-01T10:11:59.550 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:11:59.548+0000 7fae45f807c0 -1 bluestore(/var/lib/ceph/osd/c1-0) _read_fsid unparsable uuid 2024-02-01T10:12:01.760 DEBUG:teuthology.orchestra.run.smithi148:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-0 2024-02-01T10:12:01.774 DEBUG:teuthology.orchestra.run.smithi148:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c1.monmap 2024-02-01T10:12:01.852 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:12:01.852+0000 7fde9f3b37c0 -1 auth: error reading file: /var/lib/ceph/osd/c1-1/keyring: can't open /var/lib/ceph/osd/c1-1/keyring: (2) No such file or directory 2024-02-01T10:12:01.853 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:12:01.852+0000 7fde9f3b37c0 -1 created new key in keyring /var/lib/ceph/osd/c1-1/keyring 2024-02-01T10:12:01.853 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:12:01.852+0000 7fde9f3b37c0 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2024-02-01T10:12:01.853 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:12:01.852+0000 7fde9f3b37c0 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2024-02-01T10:12:01.853 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:12:01.852+0000 7fde9f3b37c0 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2024-02-01T10:12:01.853 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:12:01.852+0000 7fde9f3b37c0 -1 bluestore(/var/lib/ceph/osd/c1-1) _read_fsid unparsable uuid 2024-02-01T10:12:04.022 DEBUG:teuthology.orchestra.run.smithi148:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-1 2024-02-01T10:12:04.076 DEBUG:teuthology.orchestra.run.smithi148:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c1.monmap 2024-02-01T10:12:04.157 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:12:04.155+0000 7f12f0e2d7c0 -1 auth: error reading file: /var/lib/ceph/osd/c1-2/keyring: can't open /var/lib/ceph/osd/c1-2/keyring: (2) No such file or directory 2024-02-01T10:12:04.158 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:12:04.155+0000 7f12f0e2d7c0 -1 created new key in keyring /var/lib/ceph/osd/c1-2/keyring 2024-02-01T10:12:04.158 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:12:04.155+0000 7f12f0e2d7c0 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2024-02-01T10:12:04.158 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:12:04.155+0000 7f12f0e2d7c0 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2024-02-01T10:12:04.158 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:12:04.155+0000 7f12f0e2d7c0 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2024-02-01T10:12:04.158 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:12:04.155+0000 7f12f0e2d7c0 -1 bluestore(/var/lib/ceph/osd/c1-2) _read_fsid unparsable uuid 2024-02-01T10:12:06.338 DEBUG:teuthology.orchestra.run.smithi148:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-2 2024-02-01T10:12:06.393 INFO:tasks.ceph:Reading keys from all nodes... 2024-02-01T10:12:06.446 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-02-01T10:12:06.446 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/var/lib/ceph/mgr/c1-x/keyring of=/dev/stdout 2024-02-01T10:12:06.459 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-02-01T10:12:06.459 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/var/lib/ceph/osd/c1-0/keyring of=/dev/stdout 2024-02-01T10:12:06.511 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-02-01T10:12:06.511 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/var/lib/ceph/osd/c1-1/keyring of=/dev/stdout 2024-02-01T10:12:06.563 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-02-01T10:12:06.563 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/var/lib/ceph/osd/c1-2/keyring of=/dev/stdout 2024-02-01T10:12:06.614 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-02-01T10:12:06.614 DEBUG:teuthology.orchestra.run.smithi148:> dd if=/etc/ceph/c1.client.0.keyring of=/dev/stdout 2024-02-01T10:12:06.660 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-02-01T10:12:06.660 DEBUG:teuthology.orchestra.run.smithi148:> dd if=/etc/ceph/c1.client.1.keyring of=/dev/stdout 2024-02-01T10:12:06.708 INFO:tasks.ceph:Adding keys to all mons... 2024-02-01T10:12:06.708 DEBUG:teuthology.orchestra.run.smithi148:> sudo tee -a /etc/ceph/c1.keyring 2024-02-01T10:12:06.761 INFO:teuthology.orchestra.run.smithi148.stdout:[mgr.x] 2024-02-01T10:12:06.762 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQBtbrtlikNjBBAA3PCN0/PsZtLXKJtFEqC+mg== 2024-02-01T10:12:06.762 INFO:teuthology.orchestra.run.smithi148.stdout:[osd.0] 2024-02-01T10:12:06.762 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQBvbrtlUkLpIBAA0duJkjWAhHsOK/sEvBstHA== 2024-02-01T10:12:06.762 INFO:teuthology.orchestra.run.smithi148.stdout:[osd.1] 2024-02-01T10:12:06.762 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQBxbrtll6v9MhAAxTbVEgvh9T3ihLYEdBVeAg== 2024-02-01T10:12:06.762 INFO:teuthology.orchestra.run.smithi148.stdout:[osd.2] 2024-02-01T10:12:06.762 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQB0brtldH2CCRAAjvrvjZk1PGNv6te9tKZUlw== 2024-02-01T10:12:06.762 INFO:teuthology.orchestra.run.smithi148.stdout:[client.0] 2024-02-01T10:12:06.762 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQBtbrtlCS4LCRAAD1NixNmmBypLW5V44+w9Sg== 2024-02-01T10:12:06.762 INFO:teuthology.orchestra.run.smithi148.stdout:[client.1] 2024-02-01T10:12:06.762 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQBtbrtlgfMmDhAAluevpNK9e2pCBHSuLn+yGA== 2024-02-01T10:12:06.763 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-02-01T10:12:06.851 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-01T10:12:06.934 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-01T10:12:07.020 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-01T10:12:07.063 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-02-01T10:12:07.152 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-02-01T10:12:07.195 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-02-01T10:12:07.195 DEBUG:teuthology.orchestra.run.smithi148:> sudo mkdir -p /var/lib/ceph/mon/c1-a 2024-02-01T10:12:07.249 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c1 --mkfs -i a --monmap /home/ubuntu/cephtest/c1.monmap --keyring /etc/ceph/c1.keyring 2024-02-01T10:12:07.335 DEBUG:teuthology.orchestra.run.smithi148:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c1-a 2024-02-01T10:12:07.389 DEBUG:teuthology.orchestra.run.smithi148:> rm -- /home/ubuntu/cephtest/c1.monmap 2024-02-01T10:12:07.435 INFO:tasks.ceph:Starting mon daemons in cluster c1... 2024-02-01T10:12:07.435 INFO:tasks.ceph.c1.mon.a:Restarting daemon 2024-02-01T10:12:07.435 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c1 -i a 2024-02-01T10:12:07.481 INFO:tasks.ceph.c1.mon.a:Started 2024-02-01T10:12:07.481 INFO:tasks.ceph:Starting mgr daemons in cluster c1... 2024-02-01T10:12:07.481 INFO:tasks.ceph.c1.mgr.x:Restarting daemon 2024-02-01T10:12:07.482 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c1 -i x 2024-02-01T10:12:07.484 INFO:tasks.ceph.c1.mgr.x:Started 2024-02-01T10:12:07.484 INFO:tasks.ceph:Setting crush tunables to default 2024-02-01T10:12:07.484 DEBUG:teuthology.orchestra.run.smithi148:> sudo ceph --cluster c1 osd crush tunables default 2024-02-01T10:12:07.725 INFO:teuthology.orchestra.run.smithi148.stderr:adjusted tunables profile to default 2024-02-01T10:12:07.743 INFO:tasks.ceph:check_enable_crimson: False 2024-02-01T10:12:07.743 INFO:tasks.ceph:Starting osd daemons in cluster c1... 2024-02-01T10:12:07.743 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-02-01T10:12:07.744 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/var/lib/ceph/osd/c1-0/fsid of=/dev/stdout 2024-02-01T10:12:07.753 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-02-01T10:12:07.753 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/var/lib/ceph/osd/c1-1/fsid of=/dev/stdout 2024-02-01T10:12:07.804 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-02-01T10:12:07.804 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/var/lib/ceph/osd/c1-2/fsid of=/dev/stdout 2024-02-01T10:12:07.813 INFO:tasks.ceph.c1.mgr.x.smithi148.stderr:2024-02-01T10:12:07.811+0000 7f750cd01180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-01T10:12:07.822 DEBUG:teuthology.orchestra.run.smithi148:> sudo ceph --cluster c1 osd new 7561f0b9-54ba-4c3e-8caf-82c93f166f82 0 2024-02-01T10:12:08.060 INFO:teuthology.orchestra.run.smithi148.stdout:0 2024-02-01T10:12:08.075 DEBUG:teuthology.orchestra.run.smithi148:> sudo ceph --cluster c1 osd new 6ad3baa8-a606-4551-b81a-d59bec96546c 1 2024-02-01T10:12:08.268 INFO:teuthology.orchestra.run.smithi148.stdout:1 2024-02-01T10:12:08.284 DEBUG:teuthology.orchestra.run.smithi148:> sudo ceph --cluster c1 osd new c92e0b1a-24c8-4f65-b1d9-0e85da0662a1 2 2024-02-01T10:12:08.315 INFO:tasks.ceph.c1.mgr.x.smithi148.stderr:2024-02-01T10:12:08.315+0000 7f750cd01180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-01T10:12:08.474 INFO:teuthology.orchestra.run.smithi148.stdout:2 2024-02-01T10:12:08.489 INFO:tasks.ceph.c1.osd.0:Restarting daemon 2024-02-01T10:12:08.490 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 0 2024-02-01T10:12:08.531 INFO:tasks.ceph.c1.mgr.x.smithi148.stderr:2024-02-01T10:12:08.527+0000 7f750cd01180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-01T10:12:08.532 INFO:tasks.ceph.c1.osd.0:Started 2024-02-01T10:12:08.533 INFO:tasks.ceph.c1.osd.1:Restarting daemon 2024-02-01T10:12:08.533 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 1 2024-02-01T10:12:08.535 INFO:tasks.ceph.c1.osd.1:Started 2024-02-01T10:12:08.535 INFO:tasks.ceph.c1.osd.2:Restarting daemon 2024-02-01T10:12:08.535 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 2 2024-02-01T10:12:08.537 INFO:tasks.ceph.c1.osd.2:Started 2024-02-01T10:12:08.538 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2024-02-01T10:12:08.707 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-02-01T10:12:08.707 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":5,"fsid":"0c83d209-61c1-46c3-b2f4-1dd4a32516a3","created":"2024-02-01T10:12:07.611947+0000","modified":"2024-02-01T10:12:08.475740+0000","last_up_change":"0.000000","last_in_change":"2024-02-01T10:12:08.475740+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7561f0b9-54ba-4c3e-8caf-82c93f166f82","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6ad3baa8-a606-4551-b81a-d59bec96546c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c92e0b1a-24c8-4f65-b1d9-0e85da0662a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-01T10:12:08.722 INFO:tasks.ceph.ceph_manager.c1:[] 2024-02-01T10:12:08.723 INFO:tasks.ceph:Waiting for OSDs to come up 2024-02-01T10:12:09.025 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2024-02-01T10:12:09.097 INFO:tasks.ceph.c1.mgr.x.smithi148.stderr:2024-02-01T10:12:09.095+0000 7f750cd01180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-01T10:12:09.148 INFO:tasks.ceph.c1.osd.0.smithi148.stderr:2024-02-01T10:12:09.147+0000 7f15b20947c0 -1 Falling back to public interface 2024-02-01T10:12:09.153 INFO:tasks.ceph.c1.mgr.x.smithi148.stderr:2024-02-01T10:12:09.151+0000 7f750cd01180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-01T10:12:09.160 INFO:tasks.ceph.c1.osd.2.smithi148.stderr:2024-02-01T10:12:09.159+0000 7f9dc554f7c0 -1 Falling back to public interface 2024-02-01T10:12:09.180 INFO:tasks.ceph.c1.osd.1.smithi148.stderr:2024-02-01T10:12:09.179+0000 7faf05b687c0 -1 Falling back to public interface 2024-02-01T10:12:09.199 INFO:teuthology.misc.health.smithi148.stdout: 2024-02-01T10:12:09.200 INFO:teuthology.misc.health.smithi148.stdout:{"epoch":5,"fsid":"0c83d209-61c1-46c3-b2f4-1dd4a32516a3","created":"2024-02-01T10:12:07.611947+0000","modified":"2024-02-01T10:12:08.475740+0000","last_up_change":"0.000000","last_in_change":"2024-02-01T10:12:08.475740+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7561f0b9-54ba-4c3e-8caf-82c93f166f82","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6ad3baa8-a606-4551-b81a-d59bec96546c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c92e0b1a-24c8-4f65-b1d9-0e85da0662a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-01T10:12:09.211 INFO:tasks.ceph.c1.mgr.x.smithi148.stderr:2024-02-01T10:12:09.207+0000 7f750cd01180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-01T10:12:09.214 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-02-01T10:12:09.288 INFO:tasks.ceph.c1.mgr.x.smithi148.stderr:2024-02-01T10:12:09.287+0000 7f750cd01180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-01T10:12:09.339 INFO:tasks.ceph.c1.mgr.x.smithi148.stderr:2024-02-01T10:12:09.335+0000 7f750cd01180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-01T10:12:09.414 INFO:tasks.ceph.c1.mgr.x.smithi148.stderr:2024-02-01T10:12:09.411+0000 7f750cd01180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-01T10:12:09.466 INFO:tasks.ceph.c1.mgr.x.smithi148.stderr:2024-02-01T10:12:09.463+0000 7f750cd01180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-01T10:12:09.608 INFO:tasks.ceph.c1.mgr.x.smithi148.stderr:2024-02-01T10:12:09.607+0000 7f750cd01180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-01T10:12:09.667 INFO:tasks.ceph.c1.mgr.x.smithi148.stderr:2024-02-01T10:12:09.663+0000 7f750cd01180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-01T10:12:10.334 INFO:tasks.ceph.c1.mgr.x.smithi148.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-02-01T10:12:10.334 INFO:tasks.ceph.c1.mgr.x.smithi148.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-02-01T10:12:10.334 INFO:tasks.ceph.c1.mgr.x.smithi148.stderr: from numpy import show_config as show_numpy_config 2024-02-01T10:12:10.408 INFO:tasks.ceph.c1.mgr.x.smithi148.stderr:2024-02-01T10:12:10.407+0000 7f750cd01180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-01T10:12:10.574 INFO:tasks.ceph.c1.mgr.x.smithi148.stderr:2024-02-01T10:12:10.571+0000 7f750cd01180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-01T10:12:10.726 INFO:tasks.ceph.c1.mgr.x.smithi148.stderr:2024-02-01T10:12:10.723+0000 7f750cd01180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-01T10:12:10.789 INFO:tasks.ceph.c1.mgr.x.smithi148.stderr:2024-02-01T10:12:10.787+0000 7f750cd01180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-01T10:12:10.911 INFO:tasks.ceph.c1.mgr.x.smithi148.stderr:2024-02-01T10:12:10.907+0000 7f750cd01180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-01T10:12:11.505 INFO:tasks.ceph.c1.mgr.x.smithi148.stderr:2024-02-01T10:12:11.503+0000 7f750cd01180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-01T10:12:11.581 INFO:tasks.ceph.c1.mgr.x.smithi148.stderr:2024-02-01T10:12:11.579+0000 7f750cd01180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-01T10:12:11.753 INFO:tasks.ceph.c1.mgr.x.smithi148.stderr:2024-02-01T10:12:11.751+0000 7f750cd01180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-01T10:12:11.826 INFO:tasks.ceph.c1.mgr.x.smithi148.stderr:2024-02-01T10:12:11.823+0000 7f750cd01180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-01T10:12:11.997 INFO:tasks.ceph.c1.osd.2.smithi148.stderr:2024-02-01T10:12:11.995+0000 7f9dc554f7c0 -1 osd.2 0 log_to_monitors true 2024-02-01T10:12:12.115 INFO:tasks.ceph.c1.mgr.x.smithi148.stderr:2024-02-01T10:12:12.111+0000 7f750cd01180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-01T10:12:12.169 INFO:tasks.ceph.c1.osd.0.smithi148.stderr:2024-02-01T10:12:12.167+0000 7f15b20947c0 -1 osd.0 0 log_to_monitors true 2024-02-01T10:12:12.178 INFO:tasks.ceph.c1.mgr.x.smithi148.stderr:2024-02-01T10:12:12.175+0000 7f750cd01180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-01T10:12:12.229 INFO:tasks.ceph.c1.osd.1.smithi148.stderr:2024-02-01T10:12:12.227+0000 7faf05b687c0 -1 osd.1 0 log_to_monitors true 2024-02-01T10:12:13.277 INFO:tasks.ceph.c1.mgr.x.smithi148.stderr:2024-02-01T10:12:13.275+0000 7f74b4449640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-02-01T10:12:13.669 INFO:tasks.ceph.c1.osd.2.smithi148.stderr:2024-02-01T10:12:13.667+0000 7f9dc14f8640 -1 osd.2 0 waiting for initial osdmap 2024-02-01T10:12:13.689 INFO:tasks.ceph.c1.osd.2.smithi148.stderr:2024-02-01T10:12:13.687+0000 7f9db9333640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-01T10:12:15.084 INFO:tasks.ceph.c1.osd.0.smithi148.stderr:2024-02-01T10:12:15.083+0000 7f15ae03d640 -1 osd.0 0 waiting for initial osdmap 2024-02-01T10:12:15.130 INFO:tasks.ceph.c1.osd.0.smithi148.stderr:2024-02-01T10:12:15.127+0000 7f15a5e78640 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-01T10:12:15.150 INFO:tasks.ceph.c1.osd.1.smithi148.stderr:2024-02-01T10:12:15.147+0000 7faf01b11640 -1 osd.1 0 waiting for initial osdmap 2024-02-01T10:12:15.177 INFO:tasks.ceph.c1.osd.1.smithi148.stderr:2024-02-01T10:12:15.175+0000 7faef994c640 -1 osd.1 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-01T10:12:15.518 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2024-02-01T10:12:15.810 INFO:teuthology.misc.health.smithi148.stdout: 2024-02-01T10:12:15.810 INFO:teuthology.misc.health.smithi148.stdout:{"epoch":9,"fsid":"0c83d209-61c1-46c3-b2f4-1dd4a32516a3","created":"2024-02-01T10:12:07.611947+0000","modified":"2024-02-01T10:12:15.311603+0000","last_up_change":"2024-02-01T10:12:15.311603+0000","last_in_change":"2024-02-01T10:12:08.475740+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7561f0b9-54ba-4c3e-8caf-82c93f166f82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":3784929296},{"type":"v1","addr":"172.21.15.148:6801","nonce":3784929296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":3784929296},{"type":"v1","addr":"172.21.15.148:6803","nonce":3784929296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":3784929296},{"type":"v1","addr":"172.21.15.148:6807","nonce":3784929296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":3784929296},{"type":"v1","addr":"172.21.15.148:6805","nonce":3784929296}]},"public_addr":"172.21.15.148:6801/3784929296","cluster_addr":"172.21.15.148:6803/3784929296","heartbeat_back_addr":"172.21.15.148:6807/3784929296","heartbeat_front_addr":"172.21.15.148:6805/3784929296","state":["exists","up"]},{"osd":1,"uuid":"6ad3baa8-a606-4551-b81a-d59bec96546c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":730989348},{"type":"v1","addr":"172.21.15.148:6817","nonce":730989348}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":730989348},{"type":"v1","addr":"172.21.15.148:6819","nonce":730989348}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":730989348},{"type":"v1","addr":"172.21.15.148:6823","nonce":730989348}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":730989348},{"type":"v1","addr":"172.21.15.148:6821","nonce":730989348}]},"public_addr":"172.21.15.148:6817/730989348","cluster_addr":"172.21.15.148:6819/730989348","heartbeat_back_addr":"172.21.15.148:6823/730989348","heartbeat_front_addr":"172.21.15.148:6821/730989348","state":["exists","up"]},{"osd":2,"uuid":"c92e0b1a-24c8-4f65-b1d9-0e85da0662a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":1948417304},{"type":"v1","addr":"172.21.15.148:6809","nonce":1948417304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":1948417304},{"type":"v1","addr":"172.21.15.148:6811","nonce":1948417304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":1948417304},{"type":"v1","addr":"172.21.15.148:6815","nonce":1948417304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":1948417304},{"type":"v1","addr":"172.21.15.148:6813","nonce":1948417304}]},"public_addr":"172.21.15.148:6809/1948417304","cluster_addr":"172.21.15.148:6811/1948417304","heartbeat_back_addr":"172.21.15.148:6815/1948417304","heartbeat_front_addr":"172.21.15.148:6813/1948417304","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-01T10:12:12.971319+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-01T10:12:15.827 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-02-01T10:12:15.827 INFO:tasks.ceph:Creating RBD pool 2024-02-01T10:12:15.827 DEBUG:teuthology.orchestra.run.smithi148:> sudo ceph --cluster c1 osd pool create rbd 8 2024-02-01T10:12:17.076 INFO:teuthology.orchestra.run.smithi148.stderr:pool 'rbd' created 2024-02-01T10:12:17.101 DEBUG:teuthology.orchestra.run.smithi148:> sudo ceph --cluster c1 osd pool application enable rbd rbd --yes-i-really-mean-it 2024-02-01T10:12:18.078 INFO:teuthology.orchestra.run.smithi148.stderr:enabled application 'rbd' on pool 'rbd' 2024-02-01T10:12:18.113 INFO:tasks.ceph:Starting mds daemons in cluster c1... 2024-02-01T10:12:18.113 INFO:tasks.ceph:Waiting until c1 daemons up and pgs clean... 2024-02-01T10:12:18.113 INFO:tasks.ceph.ceph_manager.c1:waiting for mgr available 2024-02-01T10:12:18.114 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 mgr dump --format=json 2024-02-01T10:12:18.114 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-02-01T10:12:18.461 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-02-01T10:12:18.478 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":4,"flags":0,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6824","nonce":75369387},{"type":"v1","addr":"172.21.15.148:6825","nonce":75369387}]},"active_addr":"172.21.15.148:6825/75369387","active_change":"2024-02-01T10:12:12.199492+0000","active_mgr_features":4540701272860131327,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.148:0","nonce":3406206158}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.148:0","nonce":630805001}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.148:0","nonce":3289281454}]}]} 2024-02-01T10:12:18.480 INFO:tasks.ceph.ceph_manager.c1:mgr available! 2024-02-01T10:12:18.480 INFO:tasks.ceph.ceph_manager.c1:waiting for all up 2024-02-01T10:12:18.480 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2024-02-01T10:12:18.791 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-02-01T10:12:18.791 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":11,"fsid":"0c83d209-61c1-46c3-b2f4-1dd4a32516a3","created":"2024-02-01T10:12:07.611947+0000","modified":"2024-02-01T10:12:18.079031+0000","last_up_change":"2024-02-01T10:12:15.311603+0000","last_in_change":"2024-02-01T10:12:08.475740+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-02-01T10:12:16.155584+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-02-01T10:12:16.274031+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7561f0b9-54ba-4c3e-8caf-82c93f166f82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":3784929296},{"type":"v1","addr":"172.21.15.148:6801","nonce":3784929296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":3784929296},{"type":"v1","addr":"172.21.15.148:6803","nonce":3784929296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":3784929296},{"type":"v1","addr":"172.21.15.148:6807","nonce":3784929296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":3784929296},{"type":"v1","addr":"172.21.15.148:6805","nonce":3784929296}]},"public_addr":"172.21.15.148:6801/3784929296","cluster_addr":"172.21.15.148:6803/3784929296","heartbeat_back_addr":"172.21.15.148:6807/3784929296","heartbeat_front_addr":"172.21.15.148:6805/3784929296","state":["exists","up"]},{"osd":1,"uuid":"6ad3baa8-a606-4551-b81a-d59bec96546c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":730989348},{"type":"v1","addr":"172.21.15.148:6817","nonce":730989348}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":730989348},{"type":"v1","addr":"172.21.15.148:6819","nonce":730989348}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":730989348},{"type":"v1","addr":"172.21.15.148:6823","nonce":730989348}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":730989348},{"type":"v1","addr":"172.21.15.148:6821","nonce":730989348}]},"public_addr":"172.21.15.148:6817/730989348","cluster_addr":"172.21.15.148:6819/730989348","heartbeat_back_addr":"172.21.15.148:6823/730989348","heartbeat_front_addr":"172.21.15.148:6821/730989348","state":["exists","up"]},{"osd":2,"uuid":"c92e0b1a-24c8-4f65-b1d9-0e85da0662a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":1948417304},{"type":"v1","addr":"172.21.15.148:6809","nonce":1948417304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":1948417304},{"type":"v1","addr":"172.21.15.148:6811","nonce":1948417304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":1948417304},{"type":"v1","addr":"172.21.15.148:6815","nonce":1948417304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":1948417304},{"type":"v1","addr":"172.21.15.148:6813","nonce":1948417304}]},"public_addr":"172.21.15.148:6809/1948417304","cluster_addr":"172.21.15.148:6811/1948417304","heartbeat_back_addr":"172.21.15.148:6815/1948417304","heartbeat_front_addr":"172.21.15.148:6813/1948417304","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-01T10:12:13.158227+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-01T10:12:13.238813+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-01T10:12:12.971319+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-01T10:12:18.810 INFO:tasks.ceph.ceph_manager.c1:all up! 2024-02-01T10:12:18.810 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2024-02-01T10:12:19.101 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-02-01T10:12:19.101 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":12,"fsid":"0c83d209-61c1-46c3-b2f4-1dd4a32516a3","created":"2024-02-01T10:12:07.611947+0000","modified":"2024-02-01T10:12:19.086939+0000","last_up_change":"2024-02-01T10:12:15.311603+0000","last_in_change":"2024-02-01T10:12:08.475740+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-02-01T10:12:16.155584+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-02-01T10:12:16.274031+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7561f0b9-54ba-4c3e-8caf-82c93f166f82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":3784929296},{"type":"v1","addr":"172.21.15.148:6801","nonce":3784929296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":3784929296},{"type":"v1","addr":"172.21.15.148:6803","nonce":3784929296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":3784929296},{"type":"v1","addr":"172.21.15.148:6807","nonce":3784929296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":3784929296},{"type":"v1","addr":"172.21.15.148:6805","nonce":3784929296}]},"public_addr":"172.21.15.148:6801/3784929296","cluster_addr":"172.21.15.148:6803/3784929296","heartbeat_back_addr":"172.21.15.148:6807/3784929296","heartbeat_front_addr":"172.21.15.148:6805/3784929296","state":["exists","up"]},{"osd":1,"uuid":"6ad3baa8-a606-4551-b81a-d59bec96546c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":730989348},{"type":"v1","addr":"172.21.15.148:6817","nonce":730989348}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":730989348},{"type":"v1","addr":"172.21.15.148:6819","nonce":730989348}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":730989348},{"type":"v1","addr":"172.21.15.148:6823","nonce":730989348}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":730989348},{"type":"v1","addr":"172.21.15.148:6821","nonce":730989348}]},"public_addr":"172.21.15.148:6817/730989348","cluster_addr":"172.21.15.148:6819/730989348","heartbeat_back_addr":"172.21.15.148:6823/730989348","heartbeat_front_addr":"172.21.15.148:6821/730989348","state":["exists","up"]},{"osd":2,"uuid":"c92e0b1a-24c8-4f65-b1d9-0e85da0662a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":1948417304},{"type":"v1","addr":"172.21.15.148:6809","nonce":1948417304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":1948417304},{"type":"v1","addr":"172.21.15.148:6811","nonce":1948417304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":1948417304},{"type":"v1","addr":"172.21.15.148:6815","nonce":1948417304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":1948417304},{"type":"v1","addr":"172.21.15.148:6813","nonce":1948417304}]},"public_addr":"172.21.15.148:6809/1948417304","cluster_addr":"172.21.15.148:6811/1948417304","heartbeat_back_addr":"172.21.15.148:6815/1948417304","heartbeat_front_addr":"172.21.15.148:6813/1948417304","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-01T10:12:13.158227+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-01T10:12:13.238813+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-01T10:12:12.971319+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-01T10:12:19.120 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.0 flush_pg_stats 2024-02-01T10:12:19.121 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.1 flush_pg_stats 2024-02-01T10:12:19.121 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.2 flush_pg_stats 2024-02-01T10:12:19.279 INFO:teuthology.orchestra.run.smithi148.stdout:38654705668 2024-02-01T10:12:19.280 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2024-02-01T10:12:19.285 INFO:teuthology.orchestra.run.smithi148.stdout:38654705667 2024-02-01T10:12:19.285 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2024-02-01T10:12:19.294 INFO:teuthology.orchestra.run.smithi148.stdout:34359738372 2024-02-01T10:12:19.294 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2024-02-01T10:12:19.569 INFO:teuthology.orchestra.run.smithi148.stdout:38654705667 2024-02-01T10:12:19.586 INFO:tasks.ceph.ceph_manager.c1:need seq 38654705668 got 38654705667 for osd.1 2024-02-01T10:12:19.628 INFO:teuthology.orchestra.run.smithi148.stdout:34359738370 2024-02-01T10:12:19.632 INFO:teuthology.orchestra.run.smithi148.stdout:38654705666 2024-02-01T10:12:19.646 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738372 got 34359738370 for osd.2 2024-02-01T10:12:19.652 INFO:tasks.ceph.ceph_manager.c1:need seq 38654705667 got 38654705666 for osd.0 2024-02-01T10:12:20.587 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2024-02-01T10:12:20.647 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2024-02-01T10:12:20.652 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2024-02-01T10:12:20.888 INFO:teuthology.orchestra.run.smithi148.stdout:38654705667 2024-02-01T10:12:20.905 INFO:tasks.ceph.ceph_manager.c1:need seq 38654705668 got 38654705667 for osd.1 2024-02-01T10:12:20.938 INFO:teuthology.orchestra.run.smithi148.stdout:34359738370 2024-02-01T10:12:20.955 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738372 got 34359738370 for osd.2 2024-02-01T10:12:20.970 INFO:teuthology.orchestra.run.smithi148.stdout:38654705666 2024-02-01T10:12:20.986 INFO:tasks.ceph.ceph_manager.c1:need seq 38654705667 got 38654705666 for osd.0 2024-02-01T10:12:21.906 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2024-02-01T10:12:21.958 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2024-02-01T10:12:21.988 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2024-02-01T10:12:22.205 INFO:teuthology.orchestra.run.smithi148.stdout:38654705668 2024-02-01T10:12:22.223 INFO:tasks.ceph.ceph_manager.c1:need seq 38654705668 got 38654705668 for osd.1 2024-02-01T10:12:22.224 DEBUG:teuthology.parallel:result is None 2024-02-01T10:12:22.259 INFO:teuthology.orchestra.run.smithi148.stdout:34359738372 2024-02-01T10:12:22.276 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738372 got 34359738372 for osd.2 2024-02-01T10:12:22.276 DEBUG:teuthology.parallel:result is None 2024-02-01T10:12:22.301 INFO:teuthology.orchestra.run.smithi148.stdout:38654705668 2024-02-01T10:12:22.318 INFO:tasks.ceph.ceph_manager.c1:need seq 38654705667 got 38654705668 for osd.0 2024-02-01T10:12:22.318 DEBUG:teuthology.parallel:result is None 2024-02-01T10:12:22.318 INFO:tasks.ceph.ceph_manager.c1:waiting for clean 2024-02-01T10:12:22.318 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2024-02-01T10:12:22.621 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-02-01T10:12:22.621 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2024-02-01T10:12:22.638 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2024-02-01T10:12:22.204734+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":82508,"kb_used_data":1448,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":314490292,"statfs":{"total":322122547200,"available":322038059008,"internally_reserved":0,"allocated":1482752,"data_stored":1255520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":82963791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000551"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-02-01T10:12:20.804096+0000","last_change":"2024-02-01T10:12:18.092001+0000","last_active":"2024-02-01T10:12:20.804096+0000","last_peered":"2024-02-01T10:12:20.804096+0000","last_clean":"2024-02-01T10:12:20.804096+0000","last_became_active":"2024-02-01T10:12:18.091589+0000","last_became_peered":"2024-02-01T10:12:18.091589+0000","last_unstale":"2024-02-01T10:12:20.804096+0000","last_undegraded":"2024-02-01T10:12:20.804096+0000","last_fullsized":"2024-02-01T10:12:20.804096+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-01T10:12:17.072834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-01T10:12:17.072834+0000","last_clean_scrub_stamp":"2024-02-01T10:12:17.072834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-02T12:57:54.185947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":12,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-02-01T10:12:20.803961+0000","last_change":"2024-02-01T10:12:18.093505+0000","last_active":"2024-02-01T10:12:20.803961+0000","last_peered":"2024-02-01T10:12:20.803961+0000","last_clean":"2024-02-01T10:12:20.803961+0000","last_became_active":"2024-02-01T10:12:18.093108+0000","last_became_peered":"2024-02-01T10:12:18.093108+0000","last_unstale":"2024-02-01T10:12:20.803961+0000","last_undegraded":"2024-02-01T10:12:20.803961+0000","last_fullsized":"2024-02-01T10:12:20.803961+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-01T10:12:17.072834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-01T10:12:17.072834+0000","last_clean_scrub_stamp":"2024-02-01T10:12:17.072834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-02T19:50:17.941428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":12,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-02-01T10:12:19.096669+0000","last_change":"2024-02-01T10:12:18.094357+0000","last_active":"2024-02-01T10:12:19.096669+0000","last_peered":"2024-02-01T10:12:19.096669+0000","last_clean":"2024-02-01T10:12:19.096669+0000","last_became_active":"2024-02-01T10:12:18.094012+0000","last_became_peered":"2024-02-01T10:12:18.094012+0000","last_unstale":"2024-02-01T10:12:19.096669+0000","last_undegraded":"2024-02-01T10:12:19.096669+0000","last_fullsized":"2024-02-01T10:12:19.096669+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-01T10:12:17.072834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-01T10:12:17.072834+0000","last_clean_scrub_stamp":"2024-02-01T10:12:17.072834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-02T13:13:37.556812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":12,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-02-01T10:12:20.804226+0000","last_change":"2024-02-01T10:12:18.093168+0000","last_active":"2024-02-01T10:12:20.804226+0000","last_peered":"2024-02-01T10:12:20.804226+0000","last_clean":"2024-02-01T10:12:20.804226+0000","last_became_active":"2024-02-01T10:12:18.092812+0000","last_became_peered":"2024-02-01T10:12:18.092812+0000","last_unstale":"2024-02-01T10:12:20.804226+0000","last_undegraded":"2024-02-01T10:12:20.804226+0000","last_fullsized":"2024-02-01T10:12:20.804226+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-01T10:12:17.072834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-01T10:12:17.072834+0000","last_clean_scrub_stamp":"2024-02-01T10:12:17.072834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-02T19:18:56.868870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"11'97","reported_seq":139,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-02-01T10:12:19.096681+0000","last_change":"2024-02-01T10:12:18.092459+0000","last_active":"2024-02-01T10:12:19.096681+0000","last_peered":"2024-02-01T10:12:19.096681+0000","last_clean":"2024-02-01T10:12:19.096681+0000","last_became_active":"2024-02-01T10:12:18.092087+0000","last_became_peered":"2024-02-01T10:12:18.092087+0000","last_unstale":"2024-02-01T10:12:19.096681+0000","last_undegraded":"2024-02-01T10:12:19.096681+0000","last_fullsized":"2024-02-01T10:12:19.096681+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-01T10:12:17.072834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-01T10:12:17.072834+0000","last_clean_scrub_stamp":"2024-02-01T10:12:17.072834+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-02T10:49:34.969226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":12,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-02-01T10:12:20.804147+0000","last_change":"2024-02-01T10:12:18.091014+0000","last_active":"2024-02-01T10:12:20.804147+0000","last_peered":"2024-02-01T10:12:20.804147+0000","last_clean":"2024-02-01T10:12:20.804147+0000","last_became_active":"2024-02-01T10:12:18.090688+0000","last_became_peered":"2024-02-01T10:12:18.090688+0000","last_unstale":"2024-02-01T10:12:20.804147+0000","last_undegraded":"2024-02-01T10:12:20.804147+0000","last_fullsized":"2024-02-01T10:12:20.804147+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-01T10:12:17.072834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-01T10:12:17.072834+0000","last_clean_scrub_stamp":"2024-02-01T10:12:17.072834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-02T18:04:48.931779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":12,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-02-01T10:12:20.803891+0000","last_change":"2024-02-01T10:12:18.091610+0000","last_active":"2024-02-01T10:12:20.803891+0000","last_peered":"2024-02-01T10:12:20.803891+0000","last_clean":"2024-02-01T10:12:20.803891+0000","last_became_active":"2024-02-01T10:12:18.091241+0000","last_became_peered":"2024-02-01T10:12:18.091241+0000","last_unstale":"2024-02-01T10:12:20.803891+0000","last_undegraded":"2024-02-01T10:12:20.803891+0000","last_fullsized":"2024-02-01T10:12:20.803891+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-01T10:12:17.072834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-01T10:12:17.072834+0000","last_clean_scrub_stamp":"2024-02-01T10:12:17.072834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-02T15:52:24.979309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":12,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-02-01T10:12:19.096437+0000","last_change":"2024-02-01T10:12:18.094476+0000","last_active":"2024-02-01T10:12:19.096437+0000","last_peered":"2024-02-01T10:12:19.096437+0000","last_clean":"2024-02-01T10:12:19.096437+0000","last_became_active":"2024-02-01T10:12:18.094056+0000","last_became_peered":"2024-02-01T10:12:18.094056+0000","last_unstale":"2024-02-01T10:12:19.096437+0000","last_undegraded":"2024-02-01T10:12:19.096437+0000","last_fullsized":"2024-02-01T10:12:19.096437+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-01T10:12:17.072834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-01T10:12:17.072834+0000","last_clean_scrub_stamp":"2024-02-01T10:12:17.072834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-02T22:03:55.323553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-02-01T10:12:20.803888+0000","last_change":"2024-02-01T10:12:18.095010+0000","last_active":"2024-02-01T10:12:20.803888+0000","last_peered":"2024-02-01T10:12:20.803888+0000","last_clean":"2024-02-01T10:12:20.803888+0000","last_became_active":"2024-02-01T10:12:18.094679+0000","last_became_peered":"2024-02-01T10:12:18.094679+0000","last_unstale":"2024-02-01T10:12:20.803888+0000","last_undegraded":"2024-02-01T10:12:20.803888+0000","last_fullsized":"2024-02-01T10:12:20.803888+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-01T10:12:17.072834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-01T10:12:17.072834+0000","last_clean_scrub_stamp":"2024-02-01T10:12:17.072834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-02T18:57:08.865523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":9,"seq":38654705669,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27692,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829908,"statfs":{"total":107374182400,"available":107345825792,"internally_reserved":0,"allocated":692224,"data_stored":615296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38500000000000001}]}]},{"osd":0,"up_from":9,"seq":38654705668,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27124,"kb_used_data":96,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830476,"statfs":{"total":107374182400,"available":107346407424,"internally_reserved":0,"allocated":98304,"data_stored":24928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.253}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32300000000000001}]}]},{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27692,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829908,"statfs":{"total":107374182400,"available":107345825792,"internally_reserved":0,"allocated":692224,"data_stored":615296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-01T10:12:22.639 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2024-02-01T10:12:22.897 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-02-01T10:12:22.898 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2024-02-01T10:12:22.914 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2024-02-01T10:12:22.204734+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":82508,"kb_used_data":1448,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":314490292,"statfs":{"total":322122547200,"available":322038059008,"internally_reserved":0,"allocated":1482752,"data_stored":1255520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":82963791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000551"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-02-01T10:12:20.804096+0000","last_change":"2024-02-01T10:12:18.092001+0000","last_active":"2024-02-01T10:12:20.804096+0000","last_peered":"2024-02-01T10:12:20.804096+0000","last_clean":"2024-02-01T10:12:20.804096+0000","last_became_active":"2024-02-01T10:12:18.091589+0000","last_became_peered":"2024-02-01T10:12:18.091589+0000","last_unstale":"2024-02-01T10:12:20.804096+0000","last_undegraded":"2024-02-01T10:12:20.804096+0000","last_fullsized":"2024-02-01T10:12:20.804096+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-01T10:12:17.072834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-01T10:12:17.072834+0000","last_clean_scrub_stamp":"2024-02-01T10:12:17.072834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-02T12:57:54.185947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":12,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-02-01T10:12:20.803961+0000","last_change":"2024-02-01T10:12:18.093505+0000","last_active":"2024-02-01T10:12:20.803961+0000","last_peered":"2024-02-01T10:12:20.803961+0000","last_clean":"2024-02-01T10:12:20.803961+0000","last_became_active":"2024-02-01T10:12:18.093108+0000","last_became_peered":"2024-02-01T10:12:18.093108+0000","last_unstale":"2024-02-01T10:12:20.803961+0000","last_undegraded":"2024-02-01T10:12:20.803961+0000","last_fullsized":"2024-02-01T10:12:20.803961+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-01T10:12:17.072834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-01T10:12:17.072834+0000","last_clean_scrub_stamp":"2024-02-01T10:12:17.072834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-02T19:50:17.941428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":12,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-02-01T10:12:19.096669+0000","last_change":"2024-02-01T10:12:18.094357+0000","last_active":"2024-02-01T10:12:19.096669+0000","last_peered":"2024-02-01T10:12:19.096669+0000","last_clean":"2024-02-01T10:12:19.096669+0000","last_became_active":"2024-02-01T10:12:18.094012+0000","last_became_peered":"2024-02-01T10:12:18.094012+0000","last_unstale":"2024-02-01T10:12:19.096669+0000","last_undegraded":"2024-02-01T10:12:19.096669+0000","last_fullsized":"2024-02-01T10:12:19.096669+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-01T10:12:17.072834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-01T10:12:17.072834+0000","last_clean_scrub_stamp":"2024-02-01T10:12:17.072834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-02T13:13:37.556812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":12,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-02-01T10:12:20.804226+0000","last_change":"2024-02-01T10:12:18.093168+0000","last_active":"2024-02-01T10:12:20.804226+0000","last_peered":"2024-02-01T10:12:20.804226+0000","last_clean":"2024-02-01T10:12:20.804226+0000","last_became_active":"2024-02-01T10:12:18.092812+0000","last_became_peered":"2024-02-01T10:12:18.092812+0000","last_unstale":"2024-02-01T10:12:20.804226+0000","last_undegraded":"2024-02-01T10:12:20.804226+0000","last_fullsized":"2024-02-01T10:12:20.804226+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-01T10:12:17.072834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-01T10:12:17.072834+0000","last_clean_scrub_stamp":"2024-02-01T10:12:17.072834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-02T19:18:56.868870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"11'97","reported_seq":139,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-02-01T10:12:19.096681+0000","last_change":"2024-02-01T10:12:18.092459+0000","last_active":"2024-02-01T10:12:19.096681+0000","last_peered":"2024-02-01T10:12:19.096681+0000","last_clean":"2024-02-01T10:12:19.096681+0000","last_became_active":"2024-02-01T10:12:18.092087+0000","last_became_peered":"2024-02-01T10:12:18.092087+0000","last_unstale":"2024-02-01T10:12:19.096681+0000","last_undegraded":"2024-02-01T10:12:19.096681+0000","last_fullsized":"2024-02-01T10:12:19.096681+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-01T10:12:17.072834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-01T10:12:17.072834+0000","last_clean_scrub_stamp":"2024-02-01T10:12:17.072834+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-02T10:49:34.969226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":12,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-02-01T10:12:20.804147+0000","last_change":"2024-02-01T10:12:18.091014+0000","last_active":"2024-02-01T10:12:20.804147+0000","last_peered":"2024-02-01T10:12:20.804147+0000","last_clean":"2024-02-01T10:12:20.804147+0000","last_became_active":"2024-02-01T10:12:18.090688+0000","last_became_peered":"2024-02-01T10:12:18.090688+0000","last_unstale":"2024-02-01T10:12:20.804147+0000","last_undegraded":"2024-02-01T10:12:20.804147+0000","last_fullsized":"2024-02-01T10:12:20.804147+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-01T10:12:17.072834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-01T10:12:17.072834+0000","last_clean_scrub_stamp":"2024-02-01T10:12:17.072834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-02T18:04:48.931779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":12,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-02-01T10:12:20.803891+0000","last_change":"2024-02-01T10:12:18.091610+0000","last_active":"2024-02-01T10:12:20.803891+0000","last_peered":"2024-02-01T10:12:20.803891+0000","last_clean":"2024-02-01T10:12:20.803891+0000","last_became_active":"2024-02-01T10:12:18.091241+0000","last_became_peered":"2024-02-01T10:12:18.091241+0000","last_unstale":"2024-02-01T10:12:20.803891+0000","last_undegraded":"2024-02-01T10:12:20.803891+0000","last_fullsized":"2024-02-01T10:12:20.803891+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-01T10:12:17.072834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-01T10:12:17.072834+0000","last_clean_scrub_stamp":"2024-02-01T10:12:17.072834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-02T15:52:24.979309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":12,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-02-01T10:12:19.096437+0000","last_change":"2024-02-01T10:12:18.094476+0000","last_active":"2024-02-01T10:12:19.096437+0000","last_peered":"2024-02-01T10:12:19.096437+0000","last_clean":"2024-02-01T10:12:19.096437+0000","last_became_active":"2024-02-01T10:12:18.094056+0000","last_became_peered":"2024-02-01T10:12:18.094056+0000","last_unstale":"2024-02-01T10:12:19.096437+0000","last_undegraded":"2024-02-01T10:12:19.096437+0000","last_fullsized":"2024-02-01T10:12:19.096437+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-01T10:12:17.072834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-01T10:12:17.072834+0000","last_clean_scrub_stamp":"2024-02-01T10:12:17.072834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-02T22:03:55.323553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-02-01T10:12:20.803888+0000","last_change":"2024-02-01T10:12:18.095010+0000","last_active":"2024-02-01T10:12:20.803888+0000","last_peered":"2024-02-01T10:12:20.803888+0000","last_clean":"2024-02-01T10:12:20.803888+0000","last_became_active":"2024-02-01T10:12:18.094679+0000","last_became_peered":"2024-02-01T10:12:18.094679+0000","last_unstale":"2024-02-01T10:12:20.803888+0000","last_undegraded":"2024-02-01T10:12:20.803888+0000","last_fullsized":"2024-02-01T10:12:20.803888+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-01T10:12:17.072834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-01T10:12:17.072834+0000","last_clean_scrub_stamp":"2024-02-01T10:12:17.072834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-02T18:57:08.865523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":9,"seq":38654705669,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27692,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829908,"statfs":{"total":107374182400,"available":107345825792,"internally_reserved":0,"allocated":692224,"data_stored":615296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38500000000000001}]}]},{"osd":0,"up_from":9,"seq":38654705668,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27124,"kb_used_data":96,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830476,"statfs":{"total":107374182400,"available":107346407424,"internally_reserved":0,"allocated":98304,"data_stored":24928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.253}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32300000000000001}]}]},{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27692,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829908,"statfs":{"total":107374182400,"available":107345825792,"internally_reserved":0,"allocated":692224,"data_stored":615296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-01T10:12:22.915 INFO:tasks.ceph.ceph_manager.c1:clean! 2024-02-01T10:12:22.915 INFO:tasks.ceph:Waiting until ceph cluster c1 is healthy... 2024-02-01T10:12:22.915 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy 2024-02-01T10:12:22.915 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 health --format=json 2024-02-01T10:12:23.207 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-02-01T10:12:23.207 INFO:teuthology.orchestra.run.smithi148.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-02-01T10:12:23.224 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy done 2024-02-01T10:12:23.224 INFO:teuthology.run_tasks:Running task ceph... 2024-02-01T10:12:23.233 INFO:tasks.ceph:Creating ceph cluster c2... 2024-02-01T10:12:23.233 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'cpu_profile': set(), 'cluster': 'c2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-02-01T10:12:23.234 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuvalif-2024-01-30_08:46:48-rgw-wip-end2end-tracing-distro-default-smithi/7537587', 'branch': 'wip-end2end-tracing', 'description': 'rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zones supported-random-distro$/{ubuntu_latest} tasks/test_multi}', 'email': None, 'first_in_suite': False, 'job_id': '7537587', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuvalif-2024-01-30_08:46:48-rgw-wip-end2end-tracing-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-end2end-tracing'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': '5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'enabled_features': ['resharding'], 'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}, {'endpoints': ['c1.client.1'], 'name': 'test-zone3'}]}]}, 'rgw-multisite-tests': {'args': ['tests.py']}, 'workunit': {'branch': 'wip-end2end-tracing', 'sha1': '5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24'}}, 'owner': 'scheduled_yuvalif@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'seed': 8557, 'sha1': '5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24', 'sleep_before_teardown': 0, 'subset': '1/3', 'suite': 'rgw', 'suite_branch': 'wip-end2end-tracing', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24', 'targets': {'smithi148.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBH5ce3/FCp+PGNOXXdX8il8f4jkJaHpplftAcDQSFX3tHjR+D85kVrtsDWCNIGnihkTWiK/q2ovxDNYhkN4nd0g=', 'smithi187.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLmrn1ce5MSq4PQkPQy/SrK5OHVanVF/Lx8xwqrWintcWDpX5+8boxWTRV5B7sthtEqb55YGOwOBChsT+yODK3Y='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': '5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24'}}, {'ceph': {'cluster': 'c2', 'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': '5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24'}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '6128cc3ecb49b7d62475e3595041c19b5326ca7c', 'timestamp': '2024-01-30_08:46:48', 'tube': 'smithi', 'user': 'yuvalif', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-02-01T10:12:23.234 DEBUG:teuthology.orchestra.run.smithi148:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2024-02-01T10:12:23.236 DEBUG:teuthology.orchestra.run.smithi187:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2024-02-01T10:12:23.243 DEBUG:teuthology.orchestra.run.smithi148:> sudo install -d -m0777 -- /var/run/ceph 2024-02-01T10:12:23.284 DEBUG:teuthology.orchestra.run.smithi187:> sudo install -d -m0777 -- /var/run/ceph 2024-02-01T10:12:23.300 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-02-01T10:12:23.300 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/scratch_devs of=/dev/stdout 2024-02-01T10:12:23.347 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-01T10:12:23.347 DEBUG:teuthology.orchestra.run.smithi187:> stat /dev/vg_nvme/lv_1 2024-02-01T10:12:23.395 INFO:teuthology.orchestra.run.smithi187.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-01T10:12:23.396 INFO:teuthology.orchestra.run.smithi187.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-01T10:12:23.396 INFO:teuthology.orchestra.run.smithi187.stdout:Device: 5h/5d Inode: 869 Links: 1 2024-02-01T10:12:23.396 INFO:teuthology.orchestra.run.smithi187.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-01T10:12:23.396 INFO:teuthology.orchestra.run.smithi187.stdout:Access: 2024-02-01 10:06:43.656800321 +0000 2024-02-01T10:12:23.396 INFO:teuthology.orchestra.run.smithi187.stdout:Modify: 2024-02-01 10:06:43.408802534 +0000 2024-02-01T10:12:23.396 INFO:teuthology.orchestra.run.smithi187.stdout:Change: 2024-02-01 10:06:43.408802534 +0000 2024-02-01T10:12:23.396 INFO:teuthology.orchestra.run.smithi187.stdout: Birth: - 2024-02-01T10:12:23.396 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-01T10:12:23.453 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records in 2024-02-01T10:12:23.453 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records out 2024-02-01T10:12:23.453 INFO:teuthology.orchestra.run.smithi187.stderr:512 bytes copied, 0.000444767 s, 1.2 MB/s 2024-02-01T10:12:23.455 DEBUG:teuthology.orchestra.run.smithi187:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-01T10:12:23.504 DEBUG:teuthology.orchestra.run.smithi187:> stat /dev/vg_nvme/lv_2 2024-02-01T10:12:23.551 INFO:teuthology.orchestra.run.smithi187.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-01T10:12:23.551 INFO:teuthology.orchestra.run.smithi187.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-01T10:12:23.551 INFO:teuthology.orchestra.run.smithi187.stdout:Device: 5h/5d Inode: 885 Links: 1 2024-02-01T10:12:23.551 INFO:teuthology.orchestra.run.smithi187.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-01T10:12:23.551 INFO:teuthology.orchestra.run.smithi187.stdout:Access: 2024-02-01 10:06:44.136796034 +0000 2024-02-01T10:12:23.551 INFO:teuthology.orchestra.run.smithi187.stdout:Modify: 2024-02-01 10:06:43.872798393 +0000 2024-02-01T10:12:23.551 INFO:teuthology.orchestra.run.smithi187.stdout:Change: 2024-02-01 10:06:43.872798393 +0000 2024-02-01T10:12:23.551 INFO:teuthology.orchestra.run.smithi187.stdout: Birth: - 2024-02-01T10:12:23.552 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-01T10:12:23.605 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records in 2024-02-01T10:12:23.605 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records out 2024-02-01T10:12:23.605 INFO:teuthology.orchestra.run.smithi187.stderr:512 bytes copied, 0.00038585 s, 1.3 MB/s 2024-02-01T10:12:23.606 DEBUG:teuthology.orchestra.run.smithi187:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-01T10:12:23.656 DEBUG:teuthology.orchestra.run.smithi187:> stat /dev/vg_nvme/lv_3 2024-02-01T10:12:23.703 INFO:teuthology.orchestra.run.smithi187.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-01T10:12:23.703 INFO:teuthology.orchestra.run.smithi187.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-01T10:12:23.703 INFO:teuthology.orchestra.run.smithi187.stdout:Device: 5h/5d Inode: 896 Links: 1 2024-02-01T10:12:23.703 INFO:teuthology.orchestra.run.smithi187.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-01T10:12:23.703 INFO:teuthology.orchestra.run.smithi187.stdout:Access: 2024-02-01 10:06:44.628791643 +0000 2024-02-01T10:12:23.703 INFO:teuthology.orchestra.run.smithi187.stdout:Modify: 2024-02-01 10:06:44.356794070 +0000 2024-02-01T10:12:23.703 INFO:teuthology.orchestra.run.smithi187.stdout:Change: 2024-02-01 10:06:44.356794070 +0000 2024-02-01T10:12:23.703 INFO:teuthology.orchestra.run.smithi187.stdout: Birth: - 2024-02-01T10:12:23.703 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-01T10:12:23.757 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records in 2024-02-01T10:12:23.757 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records out 2024-02-01T10:12:23.757 INFO:teuthology.orchestra.run.smithi187.stderr:512 bytes copied, 0.000354851 s, 1.4 MB/s 2024-02-01T10:12:23.758 DEBUG:teuthology.orchestra.run.smithi187:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-01T10:12:23.808 DEBUG:teuthology.orchestra.run.smithi187:> stat /dev/vg_nvme/lv_4 2024-02-01T10:12:23.854 INFO:teuthology.orchestra.run.smithi187.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-01T10:12:23.855 INFO:teuthology.orchestra.run.smithi187.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-01T10:12:23.855 INFO:teuthology.orchestra.run.smithi187.stdout:Device: 5h/5d Inode: 919 Links: 1 2024-02-01T10:12:23.855 INFO:teuthology.orchestra.run.smithi187.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-01T10:12:23.855 INFO:teuthology.orchestra.run.smithi187.stdout:Access: 2024-02-01 10:06:45.168786821 +0000 2024-02-01T10:12:23.855 INFO:teuthology.orchestra.run.smithi187.stdout:Modify: 2024-02-01 10:06:44.892789288 +0000 2024-02-01T10:12:23.855 INFO:teuthology.orchestra.run.smithi187.stdout:Change: 2024-02-01 10:06:44.892789288 +0000 2024-02-01T10:12:23.855 INFO:teuthology.orchestra.run.smithi187.stdout: Birth: - 2024-02-01T10:12:23.856 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-01T10:12:23.908 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records in 2024-02-01T10:12:23.909 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records out 2024-02-01T10:12:23.909 INFO:teuthology.orchestra.run.smithi187.stderr:512 bytes copied, 0.000375421 s, 1.4 MB/s 2024-02-01T10:12:23.910 DEBUG:teuthology.orchestra.run.smithi187:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-01T10:12:23.961 INFO:tasks.ceph:osd dev map: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2024-02-01T10:12:23.961 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi187.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-02-01T10:12:23.961 INFO:tasks.ceph:Generating config... 2024-02-01T10:12:23.962 INFO:tasks.ceph:[client] debug rgw = 20 2024-02-01T10:12:23.962 INFO:tasks.ceph:[client] debug rgw sync = 20 2024-02-01T10:12:23.962 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2024-02-01T10:12:23.963 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2024-02-01T10:12:23.963 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2024-02-01T10:12:23.963 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2024-02-01T10:12:23.963 INFO:tasks.ceph:[client] rgw data log num shards = 4 2024-02-01T10:12:23.963 INFO:tasks.ceph:[client] rgw md log max shards = 4 2024-02-01T10:12:23.963 INFO:tasks.ceph:[client] rgw sync data inject err probability = 0.1 2024-02-01T10:12:23.963 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2024-02-01T10:12:23.963 INFO:tasks.ceph:[client] rgw sync meta inject err probability = 0.1 2024-02-01T10:12:23.963 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2024-02-01T10:12:23.963 INFO:tasks.ceph:[client] setgroup = ceph 2024-02-01T10:12:23.963 INFO:tasks.ceph:[client] setuser = ceph 2024-02-01T10:12:23.963 INFO:tasks.ceph:[client.0] rgw data notify interval msec = 0 2024-02-01T10:12:23.963 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-02-01T10:12:23.963 INFO:tasks.ceph:[mgr] debug ms = 1 2024-02-01T10:12:23.964 INFO:tasks.ceph:[mon] debug mon = 20 2024-02-01T10:12:23.964 INFO:tasks.ceph:[mon] debug ms = 1 2024-02-01T10:12:23.964 INFO:tasks.ceph:[mon] debug paxos = 20 2024-02-01T10:12:23.964 INFO:tasks.ceph:[osd] debug ms = 1 2024-02-01T10:12:23.964 INFO:tasks.ceph:[osd] debug osd = 20 2024-02-01T10:12:23.964 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2024-02-01T10:12:23.964 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2024-02-01T10:12:23.964 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2024-02-01T10:12:23.964 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2024-02-01T10:12:23.964 INFO:tasks.ceph:Setting up c2.mon.a... 2024-02-01T10:12:23.964 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c2.keyring 2024-02-01T10:12:24.034 INFO:teuthology.orchestra.run.smithi187.stdout:creating /etc/ceph/c2.keyring 2024-02-01T10:12:24.050 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c2.keyring 2024-02-01T10:12:24.123 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod 0644 /etc/ceph/c2.keyring 2024-02-01T10:12:24.183 DEBUG:tasks.ceph:Ceph mon addresses: [('c2.mon.a', '172.21.15.187')] 2024-02-01T10:12:24.184 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.187'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'client.0': {'rgw data notify interval msec': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2024-02-01T10:12:24.185 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-02-01T10:12:24.185 DEBUG:teuthology.orchestra.run.smithi187:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-02-01T10:12:24.236 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.187 --print /home/ubuntu/cephtest/c2.monmap 2024-02-01T10:12:24.315 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:12:24.315 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:12:24.315 INFO:teuthology.orchestra.run.smithi187.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c2.monmap 2024-02-01T10:12:24.315 INFO:teuthology.orchestra.run.smithi187.stdout:monmaptool: generated fsid 94d64532-9ed5-4e0f-80ec-57730a638832 2024-02-01T10:12:24.315 INFO:teuthology.orchestra.run.smithi187.stdout:setting min_mon_release = quincy 2024-02-01T10:12:24.316 INFO:teuthology.orchestra.run.smithi187.stdout:epoch 0 2024-02-01T10:12:24.316 INFO:teuthology.orchestra.run.smithi187.stdout:fsid 94d64532-9ed5-4e0f-80ec-57730a638832 2024-02-01T10:12:24.316 INFO:teuthology.orchestra.run.smithi187.stdout:last_changed 2024-02-01T10:12:24.317526+0000 2024-02-01T10:12:24.316 INFO:teuthology.orchestra.run.smithi187.stdout:created 2024-02-01T10:12:24.317526+0000 2024-02-01T10:12:24.316 INFO:teuthology.orchestra.run.smithi187.stdout:min_mon_release 17 (quincy) 2024-02-01T10:12:24.316 INFO:teuthology.orchestra.run.smithi187.stdout:election_strategy: 1 2024-02-01T10:12:24.316 INFO:teuthology.orchestra.run.smithi187.stdout:0: [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] mon.a 2024-02-01T10:12:24.316 INFO:teuthology.orchestra.run.smithi187.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c2.monmap (1 monitors) 2024-02-01T10:12:24.317 DEBUG:teuthology.orchestra.run.smithi187:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-02-01T10:12:24.362 INFO:tasks.ceph:Writing /etc/ceph/c2.conf for FSID 94d64532-9ed5-4e0f-80ec-57730a638832... 2024-02-01T10:12:24.363 DEBUG:teuthology.orchestra.run.smithi148:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2024-02-01T10:12:24.366 DEBUG:teuthology.orchestra.run.smithi187:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2024-02-01T10:12:24.409 INFO:teuthology.orchestra.run.smithi148.stdout:[global] 2024-02-01T10:12:24.409 INFO:teuthology.orchestra.run.smithi148.stdout: chdir = "" 2024-02-01T10:12:24.409 INFO:teuthology.orchestra.run.smithi148.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-02-01T10:12:24.409 INFO:teuthology.orchestra.run.smithi148.stdout: auth supported = cephx 2024-02-01T10:12:24.409 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-02-01T10:12:24.410 INFO:teuthology.orchestra.run.smithi148.stdout: filestore xattr use omap = true 2024-02-01T10:12:24.410 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-02-01T10:12:24.410 INFO:teuthology.orchestra.run.smithi148.stdout: mon clock drift allowed = 1.000 2024-02-01T10:12:24.410 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-02-01T10:12:24.410 INFO:teuthology.orchestra.run.smithi148.stdout: osd crush chooseleaf type = 0 2024-02-01T10:12:24.410 INFO:teuthology.orchestra.run.smithi148.stdout: auth debug = true 2024-02-01T10:12:24.410 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-02-01T10:12:24.410 INFO:teuthology.orchestra.run.smithi148.stdout: ms die on old message = true 2024-02-01T10:12:24.410 INFO:teuthology.orchestra.run.smithi148.stdout: ms die on bug = true 2024-02-01T10:12:24.410 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-02-01T10:12:24.410 INFO:teuthology.orchestra.run.smithi148.stdout: mon max pg per osd = 10000 # >= luminous 2024-02-01T10:12:24.410 INFO:teuthology.orchestra.run.smithi148.stdout: mon pg warn max object skew = 0 2024-02-01T10:12:24.410 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-02-01T10:12:24.410 INFO:teuthology.orchestra.run.smithi148.stdout: # disable pg_autoscaler by default for new pools 2024-02-01T10:12:24.411 INFO:teuthology.orchestra.run.smithi148.stdout: osd_pool_default_pg_autoscale_mode = off 2024-02-01T10:12:24.411 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-02-01T10:12:24.411 INFO:teuthology.orchestra.run.smithi148.stdout: osd pool default size = 2 2024-02-01T10:12:24.411 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-02-01T10:12:24.411 INFO:teuthology.orchestra.run.smithi148.stdout: mon osd allow primary affinity = true 2024-02-01T10:12:24.411 INFO:teuthology.orchestra.run.smithi148.stdout: mon osd allow pg remap = true 2024-02-01T10:12:24.411 INFO:teuthology.orchestra.run.smithi148.stdout: mon warn on legacy crush tunables = false 2024-02-01T10:12:24.411 INFO:teuthology.orchestra.run.smithi148.stdout: mon warn on crush straw calc version zero = false 2024-02-01T10:12:24.411 INFO:teuthology.orchestra.run.smithi148.stdout: mon warn on no sortbitwise = false 2024-02-01T10:12:24.411 INFO:teuthology.orchestra.run.smithi148.stdout: mon warn on osd down out interval zero = false 2024-02-01T10:12:24.411 INFO:teuthology.orchestra.run.smithi148.stdout: mon warn on too few osds = false 2024-02-01T10:12:24.412 INFO:teuthology.orchestra.run.smithi148.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-02-01T10:12:24.412 INFO:teuthology.orchestra.run.smithi148.stdout: mon_warn_on_pool_no_redundancy = false 2024-02-01T10:12:24.412 INFO:teuthology.orchestra.run.smithi148.stdout: mon_allow_pool_size_one = true 2024-02-01T10:12:24.412 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-02-01T10:12:24.412 INFO:teuthology.orchestra.run.smithi148.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-02-01T10:12:24.412 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-02-01T10:12:24.412 INFO:teuthology.orchestra.run.smithi148.stdout: osd default data pool replay window = 5 2024-02-01T10:12:24.412 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-02-01T10:12:24.412 INFO:teuthology.orchestra.run.smithi148.stdout: mon allow pool delete = true 2024-02-01T10:12:24.412 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-02-01T10:12:24.412 INFO:teuthology.orchestra.run.smithi148.stdout: mon cluster log file level = debug 2024-02-01T10:12:24.413 INFO:teuthology.orchestra.run.smithi148.stdout: debug asserts on shutdown = true 2024-02-01T10:12:24.413 INFO:teuthology.orchestra.run.smithi148.stdout: mon health detail to clog = false 2024-02-01T10:12:24.413 INFO:teuthology.orchestra.run.smithi148.stdout: mon host = 172.21.15.187 2024-02-01T10:12:24.413 INFO:teuthology.orchestra.run.smithi148.stdout: fsid = 94d64532-9ed5-4e0f-80ec-57730a638832 2024-02-01T10:12:24.413 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-02-01T10:12:24.413 INFO:teuthology.orchestra.run.smithi148.stdout:[osd] 2024-02-01T10:12:24.413 INFO:teuthology.orchestra.run.smithi148.stdout: osd journal size = 100 2024-02-01T10:12:24.413 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-02-01T10:12:24.413 INFO:teuthology.orchestra.run.smithi148.stdout: osd scrub load threshold = 5.0 2024-02-01T10:12:24.413 INFO:teuthology.orchestra.run.smithi148.stdout: osd scrub max interval = 600 2024-02-01T10:12:24.413 INFO:teuthology.orchestra.run.smithi148.stdout: osd mclock profile = high_recovery_ops 2024-02-01T10:12:24.413 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-02-01T10:12:24.413 INFO:teuthology.orchestra.run.smithi148.stdout: osd recover clone overlap = true 2024-02-01T10:12:24.413 INFO:teuthology.orchestra.run.smithi148.stdout: osd recovery max chunk = 1048576 2024-02-01T10:12:24.413 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-02-01T10:12:24.414 INFO:teuthology.orchestra.run.smithi148.stdout: osd debug shutdown = true 2024-02-01T10:12:24.414 INFO:teuthology.orchestra.run.smithi148.stdout: osd debug op order = true 2024-02-01T10:12:24.414 INFO:teuthology.orchestra.run.smithi148.stdout: osd debug verify stray on activate = true 2024-02-01T10:12:24.414 INFO:teuthology.orchestra.run.smithi148.stdout: osd debug trim objects = true 2024-02-01T10:12:24.414 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-02-01T10:12:24.414 INFO:teuthology.orchestra.run.smithi148.stdout: osd open classes on start = true 2024-02-01T10:12:24.414 INFO:teuthology.orchestra.run.smithi148.stdout: osd debug pg log writeout = true 2024-02-01T10:12:24.414 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-02-01T10:12:24.414 INFO:teuthology.orchestra.run.smithi148.stdout: osd deep scrub update digest min age = 30 2024-02-01T10:12:24.414 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-02-01T10:12:24.414 INFO:teuthology.orchestra.run.smithi148.stdout: osd map max advance = 10 2024-02-01T10:12:24.414 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-02-01T10:12:24.415 INFO:teuthology.orchestra.run.smithi148.stdout: journal zero on create = true 2024-02-01T10:12:24.415 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-02-01T10:12:24.415 INFO:teuthology.orchestra.run.smithi148.stdout: filestore ondisk finisher threads = 3 2024-02-01T10:12:24.415 INFO:teuthology.orchestra.run.smithi148.stdout: filestore apply finisher threads = 3 2024-02-01T10:12:24.415 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-02-01T10:12:24.415 INFO:teuthology.orchestra.run.smithi148.stdout: bdev debug aio = true 2024-02-01T10:12:24.415 INFO:teuthology.orchestra.run.smithi148.stdout: osd debug misdirected ops = true 2024-02-01T10:12:24.415 INFO:teuthology.orchestra.run.smithi148.stdout: debug ms = 1 2024-02-01T10:12:24.415 INFO:teuthology.orchestra.run.smithi148.stdout: debug osd = 20 2024-02-01T10:12:24.415 INFO:teuthology.orchestra.run.smithi148.stdout: rocksdb delete range threshold = 0 2024-02-01T10:12:24.415 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-02-01T10:12:24.415 INFO:teuthology.orchestra.run.smithi148.stdout:[mgr] 2024-02-01T10:12:24.415 INFO:teuthology.orchestra.run.smithi148.stdout: debug ms = 1 2024-02-01T10:12:24.416 INFO:teuthology.orchestra.run.smithi148.stdout: debug mgr = 20 2024-02-01T10:12:24.416 INFO:teuthology.orchestra.run.smithi148.stdout: debug mon = 20 2024-02-01T10:12:24.416 INFO:teuthology.orchestra.run.smithi148.stdout: debug auth = 20 2024-02-01T10:12:24.416 INFO:teuthology.orchestra.run.smithi148.stdout: mon reweight min pgs per osd = 4 2024-02-01T10:12:24.416 INFO:teuthology.orchestra.run.smithi148.stdout: mon reweight min bytes per osd = 10 2024-02-01T10:12:24.416 INFO:teuthology.orchestra.run.smithi148.stdout: mgr/telemetry/nag = false 2024-02-01T10:12:24.416 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-02-01T10:12:24.416 INFO:teuthology.orchestra.run.smithi148.stdout:[mon] 2024-02-01T10:12:24.416 INFO:teuthology.orchestra.run.smithi148.stdout: debug ms = 1 2024-02-01T10:12:24.416 INFO:teuthology.orchestra.run.smithi148.stdout: debug mon = 20 2024-02-01T10:12:24.416 INFO:teuthology.orchestra.run.smithi148.stdout: debug paxos = 20 2024-02-01T10:12:24.416 INFO:teuthology.orchestra.run.smithi148.stdout: debug auth = 20 2024-02-01T10:12:24.416 INFO:teuthology.orchestra.run.smithi148.stdout: mon data avail warn = 5 2024-02-01T10:12:24.416 INFO:teuthology.orchestra.run.smithi148.stdout: mon mgr mkfs grace = 240 2024-02-01T10:12:24.417 INFO:teuthology.orchestra.run.smithi148.stdout: mon reweight min pgs per osd = 4 2024-02-01T10:12:24.417 INFO:teuthology.orchestra.run.smithi148.stdout: mon osd reporter subtree level = osd 2024-02-01T10:12:24.417 INFO:teuthology.orchestra.run.smithi148.stdout: mon osd prime pg temp = true 2024-02-01T10:12:24.417 INFO:teuthology.orchestra.run.smithi148.stdout: mon reweight min bytes per osd = 10 2024-02-01T10:12:24.417 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-02-01T10:12:24.417 INFO:teuthology.orchestra.run.smithi148.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-02-01T10:12:24.417 INFO:teuthology.orchestra.run.smithi148.stdout: auth mon ticket ttl = 660 # 11m 2024-02-01T10:12:24.417 INFO:teuthology.orchestra.run.smithi148.stdout: auth service ticket ttl = 240 # 4m 2024-02-01T10:12:24.417 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-02-01T10:12:24.417 INFO:teuthology.orchestra.run.smithi148.stdout: # don't complain about insecure global_id in the test suite 2024-02-01T10:12:24.417 INFO:teuthology.orchestra.run.smithi148.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-02-01T10:12:24.417 INFO:teuthology.orchestra.run.smithi148.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-02-01T10:12:24.417 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-02-01T10:12:24.417 INFO:teuthology.orchestra.run.smithi148.stdout: # 1m isn't quite enough 2024-02-01T10:12:24.418 INFO:teuthology.orchestra.run.smithi148.stdout: mon_down_mkfs_grace = 2m 2024-02-01T10:12:24.418 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-02-01T10:12:24.418 INFO:teuthology.orchestra.run.smithi148.stdout: mon_warn_on_filestore_osds = false 2024-02-01T10:12:24.418 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-02-01T10:12:24.418 INFO:teuthology.orchestra.run.smithi148.stdout:[client] 2024-02-01T10:12:24.418 INFO:teuthology.orchestra.run.smithi148.stdout: rgw cache enabled = true 2024-02-01T10:12:24.418 INFO:teuthology.orchestra.run.smithi148.stdout: rgw enable ops log = true 2024-02-01T10:12:24.418 INFO:teuthology.orchestra.run.smithi148.stdout: rgw enable usage log = true 2024-02-01T10:12:24.418 INFO:teuthology.orchestra.run.smithi148.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-02-01T10:12:24.418 INFO:teuthology.orchestra.run.smithi148.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-02-01T10:12:24.418 INFO:teuthology.orchestra.run.smithi148.stdout: debug rgw = 20 2024-02-01T10:12:24.418 INFO:teuthology.orchestra.run.smithi148.stdout: debug rgw sync = 20 2024-02-01T10:12:24.418 INFO:teuthology.orchestra.run.smithi148.stdout: rgw crypt require ssl = False 2024-02-01T10:12:24.418 INFO:teuthology.orchestra.run.smithi148.stdout: rgw crypt s3 kms backend = testing 2024-02-01T10:12:24.419 INFO:teuthology.orchestra.run.smithi148.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2024-02-01T10:12:24.419 INFO:teuthology.orchestra.run.smithi148.stdout: rgw curl low speed time = 300 2024-02-01T10:12:24.419 INFO:teuthology.orchestra.run.smithi148.stdout: rgw data log num shards = 4 2024-02-01T10:12:24.419 INFO:teuthology.orchestra.run.smithi148.stdout: rgw md log max shards = 4 2024-02-01T10:12:24.419 INFO:teuthology.orchestra.run.smithi148.stdout: rgw sync data inject err probability = 0.1 2024-02-01T10:12:24.419 INFO:teuthology.orchestra.run.smithi148.stdout: rgw sync log trim interval = 0 2024-02-01T10:12:24.419 INFO:teuthology.orchestra.run.smithi148.stdout: rgw sync meta inject err probability = 0.1 2024-02-01T10:12:24.419 INFO:teuthology.orchestra.run.smithi148.stdout: rgw sync obj etag verify = True 2024-02-01T10:12:24.419 INFO:teuthology.orchestra.run.smithi148.stdout: setgroup = ceph 2024-02-01T10:12:24.419 INFO:teuthology.orchestra.run.smithi148.stdout: setuser = ceph 2024-02-01T10:12:24.419 INFO:teuthology.orchestra.run.smithi148.stdout:[mon.a] 2024-02-01T10:12:24.419 INFO:teuthology.orchestra.run.smithi148.stdout:[client.0] 2024-02-01T10:12:24.419 INFO:teuthology.orchestra.run.smithi148.stdout: rgw data notify interval msec = 0 2024-02-01T10:12:24.419 INFO:teuthology.orchestra.run.smithi148.stdout:[osd.0] 2024-02-01T10:12:24.419 INFO:teuthology.orchestra.run.smithi148.stdout: osd_max_omap_entries_per_request = 10 2024-02-01T10:12:24.420 INFO:teuthology.orchestra.run.smithi148.stdout:[osd.1] 2024-02-01T10:12:24.420 INFO:teuthology.orchestra.run.smithi148.stdout: osd_max_omap_entries_per_request = 1000 2024-02-01T10:12:24.420 INFO:teuthology.orchestra.run.smithi148.stdout:[osd.2] 2024-02-01T10:12:24.420 INFO:teuthology.orchestra.run.smithi148.stdout: osd_max_omap_entries_per_request = 10000 2024-02-01T10:12:24.436 INFO:teuthology.orchestra.run.smithi187.stdout:[global] 2024-02-01T10:12:24.437 INFO:teuthology.orchestra.run.smithi187.stdout: chdir = "" 2024-02-01T10:12:24.437 INFO:teuthology.orchestra.run.smithi187.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-02-01T10:12:24.437 INFO:teuthology.orchestra.run.smithi187.stdout: auth supported = cephx 2024-02-01T10:12:24.437 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-02-01T10:12:24.437 INFO:teuthology.orchestra.run.smithi187.stdout: filestore xattr use omap = true 2024-02-01T10:12:24.437 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-02-01T10:12:24.437 INFO:teuthology.orchestra.run.smithi187.stdout: mon clock drift allowed = 1.000 2024-02-01T10:12:24.437 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-02-01T10:12:24.437 INFO:teuthology.orchestra.run.smithi187.stdout: osd crush chooseleaf type = 0 2024-02-01T10:12:24.437 INFO:teuthology.orchestra.run.smithi187.stdout: auth debug = true 2024-02-01T10:12:24.437 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-02-01T10:12:24.437 INFO:teuthology.orchestra.run.smithi187.stdout: ms die on old message = true 2024-02-01T10:12:24.437 INFO:teuthology.orchestra.run.smithi187.stdout: ms die on bug = true 2024-02-01T10:12:24.437 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-02-01T10:12:24.438 INFO:teuthology.orchestra.run.smithi187.stdout: mon max pg per osd = 10000 # >= luminous 2024-02-01T10:12:24.438 INFO:teuthology.orchestra.run.smithi187.stdout: mon pg warn max object skew = 0 2024-02-01T10:12:24.438 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-02-01T10:12:24.438 INFO:teuthology.orchestra.run.smithi187.stdout: # disable pg_autoscaler by default for new pools 2024-02-01T10:12:24.438 INFO:teuthology.orchestra.run.smithi187.stdout: osd_pool_default_pg_autoscale_mode = off 2024-02-01T10:12:24.438 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-02-01T10:12:24.438 INFO:teuthology.orchestra.run.smithi187.stdout: osd pool default size = 2 2024-02-01T10:12:24.438 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-02-01T10:12:24.438 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd allow primary affinity = true 2024-02-01T10:12:24.438 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd allow pg remap = true 2024-02-01T10:12:24.438 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on legacy crush tunables = false 2024-02-01T10:12:24.438 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on crush straw calc version zero = false 2024-02-01T10:12:24.438 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on no sortbitwise = false 2024-02-01T10:12:24.438 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on osd down out interval zero = false 2024-02-01T10:12:24.439 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on too few osds = false 2024-02-01T10:12:24.439 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-02-01T10:12:24.439 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_pool_no_redundancy = false 2024-02-01T10:12:24.439 INFO:teuthology.orchestra.run.smithi187.stdout: mon_allow_pool_size_one = true 2024-02-01T10:12:24.439 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-02-01T10:12:24.439 INFO:teuthology.orchestra.run.smithi187.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-02-01T10:12:24.439 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-02-01T10:12:24.439 INFO:teuthology.orchestra.run.smithi187.stdout: osd default data pool replay window = 5 2024-02-01T10:12:24.439 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-02-01T10:12:24.439 INFO:teuthology.orchestra.run.smithi187.stdout: mon allow pool delete = true 2024-02-01T10:12:24.439 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-02-01T10:12:24.439 INFO:teuthology.orchestra.run.smithi187.stdout: mon cluster log file level = debug 2024-02-01T10:12:24.439 INFO:teuthology.orchestra.run.smithi187.stdout: debug asserts on shutdown = true 2024-02-01T10:12:24.439 INFO:teuthology.orchestra.run.smithi187.stdout: mon health detail to clog = false 2024-02-01T10:12:24.439 INFO:teuthology.orchestra.run.smithi187.stdout: mon host = 172.21.15.187 2024-02-01T10:12:24.440 INFO:teuthology.orchestra.run.smithi187.stdout: fsid = 94d64532-9ed5-4e0f-80ec-57730a638832 2024-02-01T10:12:24.440 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-02-01T10:12:24.440 INFO:teuthology.orchestra.run.smithi187.stdout:[osd] 2024-02-01T10:12:24.440 INFO:teuthology.orchestra.run.smithi187.stdout: osd journal size = 100 2024-02-01T10:12:24.440 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-02-01T10:12:24.440 INFO:teuthology.orchestra.run.smithi187.stdout: osd scrub load threshold = 5.0 2024-02-01T10:12:24.440 INFO:teuthology.orchestra.run.smithi187.stdout: osd scrub max interval = 600 2024-02-01T10:12:24.440 INFO:teuthology.orchestra.run.smithi187.stdout: osd mclock profile = high_recovery_ops 2024-02-01T10:12:24.440 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-02-01T10:12:24.441 INFO:teuthology.orchestra.run.smithi187.stdout: osd recover clone overlap = true 2024-02-01T10:12:24.441 INFO:teuthology.orchestra.run.smithi187.stdout: osd recovery max chunk = 1048576 2024-02-01T10:12:24.441 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-02-01T10:12:24.441 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug shutdown = true 2024-02-01T10:12:24.441 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug op order = true 2024-02-01T10:12:24.441 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug verify stray on activate = true 2024-02-01T10:12:24.441 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug trim objects = true 2024-02-01T10:12:24.441 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-02-01T10:12:24.441 INFO:teuthology.orchestra.run.smithi187.stdout: osd open classes on start = true 2024-02-01T10:12:24.441 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug pg log writeout = true 2024-02-01T10:12:24.441 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-02-01T10:12:24.441 INFO:teuthology.orchestra.run.smithi187.stdout: osd deep scrub update digest min age = 30 2024-02-01T10:12:24.441 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-02-01T10:12:24.441 INFO:teuthology.orchestra.run.smithi187.stdout: osd map max advance = 10 2024-02-01T10:12:24.441 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-02-01T10:12:24.442 INFO:teuthology.orchestra.run.smithi187.stdout: journal zero on create = true 2024-02-01T10:12:24.442 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-02-01T10:12:24.442 INFO:teuthology.orchestra.run.smithi187.stdout: filestore ondisk finisher threads = 3 2024-02-01T10:12:24.442 INFO:teuthology.orchestra.run.smithi187.stdout: filestore apply finisher threads = 3 2024-02-01T10:12:24.442 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-02-01T10:12:24.442 INFO:teuthology.orchestra.run.smithi187.stdout: bdev debug aio = true 2024-02-01T10:12:24.442 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug misdirected ops = true 2024-02-01T10:12:24.442 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2024-02-01T10:12:24.442 INFO:teuthology.orchestra.run.smithi187.stdout: debug osd = 20 2024-02-01T10:12:24.442 INFO:teuthology.orchestra.run.smithi187.stdout: rocksdb delete range threshold = 0 2024-02-01T10:12:24.442 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-02-01T10:12:24.443 INFO:teuthology.orchestra.run.smithi187.stdout:[mgr] 2024-02-01T10:12:24.443 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2024-02-01T10:12:24.443 INFO:teuthology.orchestra.run.smithi187.stdout: debug mgr = 20 2024-02-01T10:12:24.443 INFO:teuthology.orchestra.run.smithi187.stdout: debug mon = 20 2024-02-01T10:12:24.443 INFO:teuthology.orchestra.run.smithi187.stdout: debug auth = 20 2024-02-01T10:12:24.443 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min pgs per osd = 4 2024-02-01T10:12:24.443 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min bytes per osd = 10 2024-02-01T10:12:24.443 INFO:teuthology.orchestra.run.smithi187.stdout: mgr/telemetry/nag = false 2024-02-01T10:12:24.443 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-02-01T10:12:24.443 INFO:teuthology.orchestra.run.smithi187.stdout:[mon] 2024-02-01T10:12:24.443 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2024-02-01T10:12:24.443 INFO:teuthology.orchestra.run.smithi187.stdout: debug mon = 20 2024-02-01T10:12:24.443 INFO:teuthology.orchestra.run.smithi187.stdout: debug paxos = 20 2024-02-01T10:12:24.443 INFO:teuthology.orchestra.run.smithi187.stdout: debug auth = 20 2024-02-01T10:12:24.444 INFO:teuthology.orchestra.run.smithi187.stdout: mon data avail warn = 5 2024-02-01T10:12:24.444 INFO:teuthology.orchestra.run.smithi187.stdout: mon mgr mkfs grace = 240 2024-02-01T10:12:24.444 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min pgs per osd = 4 2024-02-01T10:12:24.444 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd reporter subtree level = osd 2024-02-01T10:12:24.444 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd prime pg temp = true 2024-02-01T10:12:24.444 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min bytes per osd = 10 2024-02-01T10:12:24.444 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-02-01T10:12:24.444 INFO:teuthology.orchestra.run.smithi187.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-02-01T10:12:24.444 INFO:teuthology.orchestra.run.smithi187.stdout: auth mon ticket ttl = 660 # 11m 2024-02-01T10:12:24.444 INFO:teuthology.orchestra.run.smithi187.stdout: auth service ticket ttl = 240 # 4m 2024-02-01T10:12:24.444 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-02-01T10:12:24.444 INFO:teuthology.orchestra.run.smithi187.stdout: # don't complain about insecure global_id in the test suite 2024-02-01T10:12:24.444 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-02-01T10:12:24.444 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-02-01T10:12:24.445 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-02-01T10:12:24.445 INFO:teuthology.orchestra.run.smithi187.stdout: # 1m isn't quite enough 2024-02-01T10:12:24.445 INFO:teuthology.orchestra.run.smithi187.stdout: mon_down_mkfs_grace = 2m 2024-02-01T10:12:24.445 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-02-01T10:12:24.445 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_filestore_osds = false 2024-02-01T10:12:24.445 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-02-01T10:12:24.445 INFO:teuthology.orchestra.run.smithi187.stdout:[client] 2024-02-01T10:12:24.445 INFO:teuthology.orchestra.run.smithi187.stdout: rgw cache enabled = true 2024-02-01T10:12:24.445 INFO:teuthology.orchestra.run.smithi187.stdout: rgw enable ops log = true 2024-02-01T10:12:24.445 INFO:teuthology.orchestra.run.smithi187.stdout: rgw enable usage log = true 2024-02-01T10:12:24.445 INFO:teuthology.orchestra.run.smithi187.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-02-01T10:12:24.445 INFO:teuthology.orchestra.run.smithi187.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-02-01T10:12:24.445 INFO:teuthology.orchestra.run.smithi187.stdout: debug rgw = 20 2024-02-01T10:12:24.445 INFO:teuthology.orchestra.run.smithi187.stdout: debug rgw sync = 20 2024-02-01T10:12:24.446 INFO:teuthology.orchestra.run.smithi187.stdout: rgw crypt require ssl = False 2024-02-01T10:12:24.446 INFO:teuthology.orchestra.run.smithi187.stdout: rgw crypt s3 kms backend = testing 2024-02-01T10:12:24.446 INFO:teuthology.orchestra.run.smithi187.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2024-02-01T10:12:24.446 INFO:teuthology.orchestra.run.smithi187.stdout: rgw curl low speed time = 300 2024-02-01T10:12:24.446 INFO:teuthology.orchestra.run.smithi187.stdout: rgw data log num shards = 4 2024-02-01T10:12:24.446 INFO:teuthology.orchestra.run.smithi187.stdout: rgw md log max shards = 4 2024-02-01T10:12:24.446 INFO:teuthology.orchestra.run.smithi187.stdout: rgw sync data inject err probability = 0.1 2024-02-01T10:12:24.446 INFO:teuthology.orchestra.run.smithi187.stdout: rgw sync log trim interval = 0 2024-02-01T10:12:24.446 INFO:teuthology.orchestra.run.smithi187.stdout: rgw sync meta inject err probability = 0.1 2024-02-01T10:12:24.446 INFO:teuthology.orchestra.run.smithi187.stdout: rgw sync obj etag verify = True 2024-02-01T10:12:24.446 INFO:teuthology.orchestra.run.smithi187.stdout: setgroup = ceph 2024-02-01T10:12:24.447 INFO:teuthology.orchestra.run.smithi187.stdout: setuser = ceph 2024-02-01T10:12:24.447 INFO:teuthology.orchestra.run.smithi187.stdout:[mon.a] 2024-02-01T10:12:24.447 INFO:teuthology.orchestra.run.smithi187.stdout:[client.0] 2024-02-01T10:12:24.447 INFO:teuthology.orchestra.run.smithi187.stdout: rgw data notify interval msec = 0 2024-02-01T10:12:24.447 INFO:teuthology.orchestra.run.smithi187.stdout:[osd.0] 2024-02-01T10:12:24.447 INFO:teuthology.orchestra.run.smithi187.stdout: osd_max_omap_entries_per_request = 10 2024-02-01T10:12:24.447 INFO:teuthology.orchestra.run.smithi187.stdout:[osd.1] 2024-02-01T10:12:24.447 INFO:teuthology.orchestra.run.smithi187.stdout: osd_max_omap_entries_per_request = 1000 2024-02-01T10:12:24.447 INFO:teuthology.orchestra.run.smithi187.stdout:[osd.2] 2024-02-01T10:12:24.447 INFO:teuthology.orchestra.run.smithi187.stdout: osd_max_omap_entries_per_request = 10000 2024-02-01T10:12:24.449 INFO:tasks.ceph:Creating admin key on c2.mon.a... 2024-02-01T10:12:24.449 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c2.keyring 2024-02-01T10:12:24.537 INFO:tasks.ceph:Copying monmap to all nodes... 2024-02-01T10:12:24.538 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-02-01T10:12:24.539 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/etc/ceph/c2.keyring of=/dev/stdout 2024-02-01T10:12:24.545 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-02-01T10:12:24.545 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/home/ubuntu/cephtest/c2.monmap of=/dev/stdout 2024-02-01T10:12:24.596 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi148.front.sepia.ceph.com 2024-02-01T10:12:24.596 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-02-01T10:12:24.596 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/etc/ceph/c2.keyring 2024-02-01T10:12:24.596 DEBUG:teuthology.orchestra.run.smithi148:> sudo chmod 0644 /etc/ceph/c2.keyring 2024-02-01T10:12:24.619 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-02-01T10:12:24.620 DEBUG:teuthology.orchestra.run.smithi148:> dd of=/home/ubuntu/cephtest/c2.monmap 2024-02-01T10:12:24.668 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi187.front.sepia.ceph.com 2024-02-01T10:12:24.668 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-02-01T10:12:24.668 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/etc/ceph/c2.keyring 2024-02-01T10:12:24.668 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod 0644 /etc/ceph/c2.keyring 2024-02-01T10:12:24.691 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-02-01T10:12:24.691 DEBUG:teuthology.orchestra.run.smithi187:> dd of=/home/ubuntu/cephtest/c2.monmap 2024-02-01T10:12:24.739 INFO:tasks.ceph:Setting up mon nodes... 2024-02-01T10:12:24.740 INFO:tasks.ceph:Setting up mgr nodes... 2024-02-01T10:12:24.740 DEBUG:teuthology.orchestra.run.smithi187:> sudo mkdir -p /var/lib/ceph/mgr/c2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c2-x/keyring 2024-02-01T10:12:24.823 INFO:teuthology.orchestra.run.smithi187.stdout:creating /var/lib/ceph/mgr/c2-x/keyring 2024-02-01T10:12:24.826 INFO:tasks.ceph:Setting up mds nodes... 2024-02-01T10:12:24.826 INFO:tasks.ceph_client:Setting up client nodes... 2024-02-01T10:12:24.826 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c2.client.0.keyring && sudo chmod 0644 /etc/ceph/c2.client.0.keyring 2024-02-01T10:12:24.900 INFO:teuthology.orchestra.run.smithi187.stdout:creating /etc/ceph/c2.client.0.keyring 2024-02-01T10:12:24.911 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c2.client.1.keyring && sudo chmod 0644 /etc/ceph/c2.client.1.keyring 2024-02-01T10:12:24.986 INFO:teuthology.orchestra.run.smithi187.stdout:creating /etc/ceph/c2.client.1.keyring 2024-02-01T10:12:24.997 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-02-01T10:12:24.998 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi148.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi187.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-02-01T10:12:24.998 DEBUG:teuthology.orchestra.run.smithi187:> sudo mkdir -p /var/lib/ceph/osd/c2-0 2024-02-01T10:12:25.050 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2024-02-01T10:12:25.050 INFO:tasks.ceph:role: c2.osd.0 2024-02-01T10:12:25.050 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi187.front.sepia.ceph.com 2024-02-01T10:12:25.050 DEBUG:teuthology.orchestra.run.smithi187:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-02-01T10:12:25.116 INFO:teuthology.orchestra.run.smithi187.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-02-01T10:12:25.116 INFO:teuthology.orchestra.run.smithi187.stdout: = sectsz=512 attr=2, projid32bit=1 2024-02-01T10:12:25.116 INFO:teuthology.orchestra.run.smithi187.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-02-01T10:12:25.116 INFO:teuthology.orchestra.run.smithi187.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-02-01T10:12:25.116 INFO:teuthology.orchestra.run.smithi187.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-02-01T10:12:25.116 INFO:teuthology.orchestra.run.smithi187.stdout: = sunit=0 swidth=0 blks 2024-02-01T10:12:25.116 INFO:teuthology.orchestra.run.smithi187.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-02-01T10:12:25.116 INFO:teuthology.orchestra.run.smithi187.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-02-01T10:12:25.116 INFO:teuthology.orchestra.run.smithi187.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-02-01T10:12:25.117 INFO:teuthology.orchestra.run.smithi187.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-02-01T10:12:25.570 INFO:teuthology.orchestra.run.smithi187.stdout:Discarding blocks...Done. 2024-02-01T10:12:25.572 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi187.front.sepia.ceph.com -o noatime 2024-02-01T10:12:25.572 DEBUG:teuthology.orchestra.run.smithi187:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c2-0 2024-02-01T10:12:25.669 DEBUG:teuthology.orchestra.run.smithi187:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-0 2024-02-01T10:12:25.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-01T10:12:25.679 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: /sbin/restorecon: command not found 2024-02-01T10:12:25.680 DEBUG:teuthology.orchestra.run.smithi187:> sudo mkdir -p /var/lib/ceph/osd/c2-1 2024-02-01T10:12:25.734 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2024-02-01T10:12:25.734 INFO:tasks.ceph:role: c2.osd.1 2024-02-01T10:12:25.734 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi187.front.sepia.ceph.com 2024-02-01T10:12:25.734 DEBUG:teuthology.orchestra.run.smithi187:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-02-01T10:12:25.798 INFO:teuthology.orchestra.run.smithi187.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-02-01T10:12:25.799 INFO:teuthology.orchestra.run.smithi187.stdout: = sectsz=512 attr=2, projid32bit=1 2024-02-01T10:12:25.799 INFO:teuthology.orchestra.run.smithi187.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-02-01T10:12:25.799 INFO:teuthology.orchestra.run.smithi187.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-02-01T10:12:25.799 INFO:teuthology.orchestra.run.smithi187.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-02-01T10:12:25.799 INFO:teuthology.orchestra.run.smithi187.stdout: = sunit=0 swidth=0 blks 2024-02-01T10:12:25.799 INFO:teuthology.orchestra.run.smithi187.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-02-01T10:12:25.799 INFO:teuthology.orchestra.run.smithi187.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-02-01T10:12:25.799 INFO:teuthology.orchestra.run.smithi187.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-02-01T10:12:25.799 INFO:teuthology.orchestra.run.smithi187.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-02-01T10:12:26.270 INFO:teuthology.orchestra.run.smithi187.stdout:Discarding blocks...Done. 2024-02-01T10:12:26.272 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi187.front.sepia.ceph.com -o noatime 2024-02-01T10:12:26.272 DEBUG:teuthology.orchestra.run.smithi187:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c2-1 2024-02-01T10:12:26.408 DEBUG:teuthology.orchestra.run.smithi187:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-1 2024-02-01T10:12:26.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-01T10:12:26.419 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: /sbin/restorecon: command not found 2024-02-01T10:12:26.420 DEBUG:teuthology.orchestra.run.smithi187:> sudo mkdir -p /var/lib/ceph/osd/c2-2 2024-02-01T10:12:26.474 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2024-02-01T10:12:26.474 INFO:tasks.ceph:role: c2.osd.2 2024-02-01T10:12:26.474 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi187.front.sepia.ceph.com 2024-02-01T10:12:26.474 DEBUG:teuthology.orchestra.run.smithi187:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-02-01T10:12:26.538 INFO:teuthology.orchestra.run.smithi187.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-02-01T10:12:26.538 INFO:teuthology.orchestra.run.smithi187.stdout: = sectsz=512 attr=2, projid32bit=1 2024-02-01T10:12:26.538 INFO:teuthology.orchestra.run.smithi187.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-02-01T10:12:26.538 INFO:teuthology.orchestra.run.smithi187.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-02-01T10:12:26.538 INFO:teuthology.orchestra.run.smithi187.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-02-01T10:12:26.539 INFO:teuthology.orchestra.run.smithi187.stdout: = sunit=0 swidth=0 blks 2024-02-01T10:12:26.539 INFO:teuthology.orchestra.run.smithi187.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-02-01T10:12:26.539 INFO:teuthology.orchestra.run.smithi187.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-02-01T10:12:26.539 INFO:teuthology.orchestra.run.smithi187.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-02-01T10:12:26.539 INFO:teuthology.orchestra.run.smithi187.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-02-01T10:12:27.025 INFO:teuthology.orchestra.run.smithi187.stdout:Discarding blocks...Done. 2024-02-01T10:12:27.027 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi187.front.sepia.ceph.com -o noatime 2024-02-01T10:12:27.027 DEBUG:teuthology.orchestra.run.smithi187:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c2-2 2024-02-01T10:12:27.179 DEBUG:teuthology.orchestra.run.smithi187:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-2 2024-02-01T10:12:27.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-01T10:12:27.190 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: /sbin/restorecon: command not found 2024-02-01T10:12:27.191 DEBUG:teuthology.orchestra.run.smithi187:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c2.monmap 2024-02-01T10:12:27.304 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:12:27.304+0000 7fa06604e7c0 -1 auth: error reading file: /var/lib/ceph/osd/c2-0/keyring: can't open /var/lib/ceph/osd/c2-0/keyring: (2) No such file or directory 2024-02-01T10:12:27.305 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:12:27.304+0000 7fa06604e7c0 -1 created new key in keyring /var/lib/ceph/osd/c2-0/keyring 2024-02-01T10:12:27.305 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:12:27.304+0000 7fa06604e7c0 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2024-02-01T10:12:27.305 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:12:27.304+0000 7fa06604e7c0 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2024-02-01T10:12:27.305 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:12:27.304+0000 7fa06604e7c0 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2024-02-01T10:12:27.305 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:12:27.304+0000 7fa06604e7c0 -1 bluestore(/var/lib/ceph/osd/c2-0) _read_fsid unparsable uuid 2024-02-01T10:12:29.472 DEBUG:teuthology.orchestra.run.smithi187:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-0 2024-02-01T10:12:29.486 DEBUG:teuthology.orchestra.run.smithi187:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c2.monmap 2024-02-01T10:12:29.569 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:12:29.568+0000 7f7eb4a4b7c0 -1 auth: error reading file: /var/lib/ceph/osd/c2-1/keyring: can't open /var/lib/ceph/osd/c2-1/keyring: (2) No such file or directory 2024-02-01T10:12:29.570 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:12:29.568+0000 7f7eb4a4b7c0 -1 created new key in keyring /var/lib/ceph/osd/c2-1/keyring 2024-02-01T10:12:29.570 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:12:29.568+0000 7f7eb4a4b7c0 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2024-02-01T10:12:29.570 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:12:29.568+0000 7f7eb4a4b7c0 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2024-02-01T10:12:29.570 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:12:29.568+0000 7f7eb4a4b7c0 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2024-02-01T10:12:29.570 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:12:29.568+0000 7f7eb4a4b7c0 -1 bluestore(/var/lib/ceph/osd/c2-1) _read_fsid unparsable uuid 2024-02-01T10:12:31.760 DEBUG:teuthology.orchestra.run.smithi187:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-1 2024-02-01T10:12:31.814 DEBUG:teuthology.orchestra.run.smithi187:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c2.monmap 2024-02-01T10:12:31.900 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:12:31.900+0000 7f86b23ee7c0 -1 auth: error reading file: /var/lib/ceph/osd/c2-2/keyring: can't open /var/lib/ceph/osd/c2-2/keyring: (2) No such file or directory 2024-02-01T10:12:31.901 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:12:31.900+0000 7f86b23ee7c0 -1 created new key in keyring /var/lib/ceph/osd/c2-2/keyring 2024-02-01T10:12:31.901 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:12:31.900+0000 7f86b23ee7c0 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2024-02-01T10:12:31.901 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:12:31.900+0000 7f86b23ee7c0 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2024-02-01T10:12:31.901 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:12:31.900+0000 7f86b23ee7c0 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2024-02-01T10:12:31.902 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:12:31.900+0000 7f86b23ee7c0 -1 bluestore(/var/lib/ceph/osd/c2-2) _read_fsid unparsable uuid 2024-02-01T10:12:34.073 DEBUG:teuthology.orchestra.run.smithi187:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-2 2024-02-01T10:12:34.127 INFO:tasks.ceph:Reading keys from all nodes... 2024-02-01T10:12:34.128 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-02-01T10:12:34.128 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/var/lib/ceph/mgr/c2-x/keyring of=/dev/stdout 2024-02-01T10:12:34.182 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-02-01T10:12:34.182 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/var/lib/ceph/osd/c2-0/keyring of=/dev/stdout 2024-02-01T10:12:34.238 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-02-01T10:12:34.238 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/var/lib/ceph/osd/c2-1/keyring of=/dev/stdout 2024-02-01T10:12:34.294 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-02-01T10:12:34.294 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/var/lib/ceph/osd/c2-2/keyring of=/dev/stdout 2024-02-01T10:12:34.346 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-02-01T10:12:34.346 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/etc/ceph/c2.client.0.keyring of=/dev/stdout 2024-02-01T10:12:34.395 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-02-01T10:12:34.395 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/etc/ceph/c2.client.1.keyring of=/dev/stdout 2024-02-01T10:12:34.444 INFO:tasks.ceph:Adding keys to all mons... 2024-02-01T10:12:34.444 DEBUG:teuthology.orchestra.run.smithi187:> sudo tee -a /etc/ceph/c2.keyring 2024-02-01T10:12:34.498 INFO:teuthology.orchestra.run.smithi187.stdout:[mgr.x] 2024-02-01T10:12:34.498 INFO:teuthology.orchestra.run.smithi187.stdout: key = AQCIbrtlqIQwMRAAAny8/lV3Pw+/dcCYLskQvw== 2024-02-01T10:12:34.498 INFO:teuthology.orchestra.run.smithi187.stdout:[osd.0] 2024-02-01T10:12:34.498 INFO:teuthology.orchestra.run.smithi187.stdout: key = AQCLbrtlD8BAEhAAAZifbvgXjtDVlQ2SR84GZg== 2024-02-01T10:12:34.498 INFO:teuthology.orchestra.run.smithi187.stdout:[osd.1] 2024-02-01T10:12:34.498 INFO:teuthology.orchestra.run.smithi187.stdout: key = AQCNbrtlA1ALIhAACBW0w+iBxrgQay0rj/5Z+w== 2024-02-01T10:12:34.498 INFO:teuthology.orchestra.run.smithi187.stdout:[osd.2] 2024-02-01T10:12:34.498 INFO:teuthology.orchestra.run.smithi187.stdout: key = AQCPbrtlLEbRNRAA6mzHuFcJ/XoN5zYNtKm4yA== 2024-02-01T10:12:34.498 INFO:teuthology.orchestra.run.smithi187.stdout:[client.0] 2024-02-01T10:12:34.498 INFO:teuthology.orchestra.run.smithi187.stdout: key = AQCIbrtlDBjPNRAA2rH6o2zdyXATN8FbFCHsRA== 2024-02-01T10:12:34.498 INFO:teuthology.orchestra.run.smithi187.stdout:[client.1] 2024-02-01T10:12:34.498 INFO:teuthology.orchestra.run.smithi187.stdout: key = AQCIbrtl/PjyOhAAO9XG9s+SahdJzN1U4yrMZQ== 2024-02-01T10:12:34.499 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-02-01T10:12:34.588 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-01T10:12:34.628 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-01T10:12:34.709 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-01T10:12:34.790 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-02-01T10:12:34.879 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-02-01T10:12:34.960 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-02-01T10:12:34.961 DEBUG:teuthology.orchestra.run.smithi187:> sudo mkdir -p /var/lib/ceph/mon/c2-a 2024-02-01T10:12:35.013 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c2 --mkfs -i a --monmap /home/ubuntu/cephtest/c2.monmap --keyring /etc/ceph/c2.keyring 2024-02-01T10:12:35.101 DEBUG:teuthology.orchestra.run.smithi187:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c2-a 2024-02-01T10:12:35.116 DEBUG:teuthology.orchestra.run.smithi187:> rm -- /home/ubuntu/cephtest/c2.monmap 2024-02-01T10:12:35.159 INFO:tasks.ceph:Starting mon daemons in cluster c2... 2024-02-01T10:12:35.159 INFO:tasks.ceph.c2.mon.a:Restarting daemon 2024-02-01T10:12:35.159 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c2 -i a 2024-02-01T10:12:35.204 INFO:tasks.ceph.c2.mon.a:Started 2024-02-01T10:12:35.204 INFO:tasks.ceph:Starting mgr daemons in cluster c2... 2024-02-01T10:12:35.204 INFO:tasks.ceph.c2.mgr.x:Restarting daemon 2024-02-01T10:12:35.204 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c2 -i x 2024-02-01T10:12:35.206 INFO:tasks.ceph.c2.mgr.x:Started 2024-02-01T10:12:35.206 INFO:tasks.ceph:Setting crush tunables to default 2024-02-01T10:12:35.206 DEBUG:teuthology.orchestra.run.smithi187:> sudo ceph --cluster c2 osd crush tunables default 2024-02-01T10:12:35.442 INFO:teuthology.orchestra.run.smithi187.stderr:adjusted tunables profile to default 2024-02-01T10:12:35.460 INFO:tasks.ceph:check_enable_crimson: False 2024-02-01T10:12:35.460 INFO:tasks.ceph:Starting osd daemons in cluster c2... 2024-02-01T10:12:35.461 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-02-01T10:12:35.461 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/var/lib/ceph/osd/c2-0/fsid of=/dev/stdout 2024-02-01T10:12:35.471 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-02-01T10:12:35.471 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/var/lib/ceph/osd/c2-1/fsid of=/dev/stdout 2024-02-01T10:12:35.524 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-02-01T10:12:35.524 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/var/lib/ceph/osd/c2-2/fsid of=/dev/stdout 2024-02-01T10:12:35.547 INFO:tasks.ceph.c2.mgr.x.smithi187.stderr:2024-02-01T10:12:35.548+0000 7f8f86fc4180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-01T10:12:35.556 DEBUG:teuthology.orchestra.run.smithi187:> sudo ceph --cluster c2 osd new c7150bad-9637-4fd0-a5b1-3a82ce15afe7 0 2024-02-01T10:12:35.798 INFO:teuthology.orchestra.run.smithi187.stdout:0 2024-02-01T10:12:35.817 DEBUG:teuthology.orchestra.run.smithi187:> sudo ceph --cluster c2 osd new c5fef800-a312-4f36-9eb5-b4d0435f0b9c 1 2024-02-01T10:12:36.015 INFO:teuthology.orchestra.run.smithi187.stdout:1 2024-02-01T10:12:36.033 DEBUG:teuthology.orchestra.run.smithi187:> sudo ceph --cluster c2 osd new 90c21075-9f63-44fa-a85c-b8968dd4f3c3 2 2024-02-01T10:12:36.067 INFO:tasks.ceph.c2.mgr.x.smithi187.stderr:2024-02-01T10:12:36.068+0000 7f8f86fc4180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-01T10:12:36.231 INFO:teuthology.orchestra.run.smithi187.stdout:2 2024-02-01T10:12:36.250 INFO:tasks.ceph.c2.osd.0:Restarting daemon 2024-02-01T10:12:36.250 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 0 2024-02-01T10:12:36.276 INFO:tasks.ceph.c2.mgr.x.smithi187.stderr:2024-02-01T10:12:36.276+0000 7f8f86fc4180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-01T10:12:36.278 INFO:tasks.ceph.c2.osd.0:Started 2024-02-01T10:12:36.278 INFO:tasks.ceph.c2.osd.1:Restarting daemon 2024-02-01T10:12:36.278 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 1 2024-02-01T10:12:36.280 INFO:tasks.ceph.c2.osd.1:Started 2024-02-01T10:12:36.280 INFO:tasks.ceph.c2.osd.2:Restarting daemon 2024-02-01T10:12:36.280 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 2 2024-02-01T10:12:36.282 INFO:tasks.ceph.c2.osd.2:Started 2024-02-01T10:12:36.282 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2024-02-01T10:12:36.471 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-02-01T10:12:36.471 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":5,"fsid":"94d64532-9ed5-4e0f-80ec-57730a638832","created":"2024-02-01T10:12:35.337929+0000","modified":"2024-02-01T10:12:36.232711+0000","last_up_change":"0.000000","last_in_change":"2024-02-01T10:12:36.232711+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c7150bad-9637-4fd0-a5b1-3a82ce15afe7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5fef800-a312-4f36-9eb5-b4d0435f0b9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"90c21075-9f63-44fa-a85c-b8968dd4f3c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-01T10:12:36.488 INFO:tasks.ceph.ceph_manager.c2:[] 2024-02-01T10:12:36.489 INFO:tasks.ceph:Waiting for OSDs to come up 2024-02-01T10:12:36.790 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2024-02-01T10:12:36.883 INFO:tasks.ceph.c2.mgr.x.smithi187.stderr:2024-02-01T10:12:36.884+0000 7f8f86fc4180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-01T10:12:36.911 INFO:tasks.ceph.c2.osd.0.smithi187.stderr:2024-02-01T10:12:36.912+0000 7efcfdd0d7c0 -1 Falling back to public interface 2024-02-01T10:12:36.912 INFO:tasks.ceph.c2.osd.1.smithi187.stderr:2024-02-01T10:12:36.912+0000 7fd23afe77c0 -1 Falling back to public interface 2024-02-01T10:12:36.912 INFO:tasks.ceph.c2.osd.1.smithi187.stderr:2024-02-01T10:12:36.912+0000 7fd239fa7640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6801/0: (98) Address already in use 2024-02-01T10:12:36.915 INFO:tasks.ceph.c2.osd.2.smithi187.stderr:2024-02-01T10:12:36.916+0000 7f6bbdcac7c0 -1 Falling back to public interface 2024-02-01T10:12:36.942 INFO:tasks.ceph.c2.mgr.x.smithi187.stderr:2024-02-01T10:12:36.940+0000 7f8f86fc4180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-01T10:12:36.970 INFO:teuthology.misc.health.smithi187.stdout: 2024-02-01T10:12:36.970 INFO:teuthology.misc.health.smithi187.stdout:{"epoch":5,"fsid":"94d64532-9ed5-4e0f-80ec-57730a638832","created":"2024-02-01T10:12:35.337929+0000","modified":"2024-02-01T10:12:36.232711+0000","last_up_change":"0.000000","last_in_change":"2024-02-01T10:12:36.232711+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c7150bad-9637-4fd0-a5b1-3a82ce15afe7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5fef800-a312-4f36-9eb5-b4d0435f0b9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"90c21075-9f63-44fa-a85c-b8968dd4f3c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-01T10:12:36.985 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-02-01T10:12:36.997 INFO:tasks.ceph.c2.mgr.x.smithi187.stderr:2024-02-01T10:12:36.996+0000 7f8f86fc4180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-01T10:12:37.074 INFO:tasks.ceph.c2.mgr.x.smithi187.stderr:2024-02-01T10:12:37.072+0000 7f8f86fc4180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-01T10:12:37.125 INFO:tasks.ceph.c2.mgr.x.smithi187.stderr:2024-02-01T10:12:37.124+0000 7f8f86fc4180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-01T10:12:37.201 INFO:tasks.ceph.c2.mgr.x.smithi187.stderr:2024-02-01T10:12:37.200+0000 7f8f86fc4180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-01T10:12:37.253 INFO:tasks.ceph.c2.mgr.x.smithi187.stderr:2024-02-01T10:12:37.252+0000 7f8f86fc4180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-01T10:12:37.394 INFO:tasks.ceph.c2.mgr.x.smithi187.stderr:2024-02-01T10:12:37.392+0000 7f8f86fc4180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-01T10:12:37.453 INFO:tasks.ceph.c2.mgr.x.smithi187.stderr:2024-02-01T10:12:37.452+0000 7f8f86fc4180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-01T10:12:38.132 INFO:tasks.ceph.c2.mgr.x.smithi187.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-02-01T10:12:38.132 INFO:tasks.ceph.c2.mgr.x.smithi187.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-02-01T10:12:38.133 INFO:tasks.ceph.c2.mgr.x.smithi187.stderr: from numpy import show_config as show_numpy_config 2024-02-01T10:12:38.175 INFO:tasks.ceph.c2.mgr.x.smithi187.stderr:2024-02-01T10:12:38.176+0000 7f8f86fc4180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-01T10:12:38.315 INFO:tasks.ceph.c2.mgr.x.smithi187.stderr:2024-02-01T10:12:38.316+0000 7f8f86fc4180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-01T10:12:38.478 INFO:tasks.ceph.c2.mgr.x.smithi187.stderr:2024-02-01T10:12:38.476+0000 7f8f86fc4180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-01T10:12:38.561 INFO:tasks.ceph.c2.mgr.x.smithi187.stderr:2024-02-01T10:12:38.560+0000 7f8f86fc4180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-01T10:12:38.695 INFO:tasks.ceph.c2.mgr.x.smithi187.stderr:2024-02-01T10:12:38.696+0000 7f8f86fc4180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-01T10:12:39.300 INFO:tasks.ceph.c2.mgr.x.smithi187.stderr:2024-02-01T10:12:39.299+0000 7f8f86fc4180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-01T10:12:39.375 INFO:tasks.ceph.c2.mgr.x.smithi187.stderr:2024-02-01T10:12:39.375+0000 7f8f86fc4180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-01T10:12:39.548 INFO:tasks.ceph.c2.mgr.x.smithi187.stderr:2024-02-01T10:12:39.547+0000 7f8f86fc4180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-01T10:12:39.623 INFO:tasks.ceph.c2.mgr.x.smithi187.stderr:2024-02-01T10:12:39.619+0000 7f8f86fc4180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-01T10:12:39.692 INFO:tasks.ceph.c2.osd.2.smithi187.stderr:2024-02-01T10:12:39.691+0000 7f6bbdcac7c0 -1 osd.2 0 log_to_monitors true 2024-02-01T10:12:39.917 INFO:tasks.ceph.c2.mgr.x.smithi187.stderr:2024-02-01T10:12:39.915+0000 7f8f86fc4180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-01T10:12:39.975 INFO:tasks.ceph.c2.osd.1.smithi187.stderr:2024-02-01T10:12:39.975+0000 7fd23afe77c0 -1 osd.1 0 log_to_monitors true 2024-02-01T10:12:39.983 INFO:tasks.ceph.c2.mgr.x.smithi187.stderr:2024-02-01T10:12:39.983+0000 7f8f86fc4180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-01T10:12:39.985 INFO:tasks.ceph.c2.osd.0.smithi187.stderr:2024-02-01T10:12:39.983+0000 7efcfdd0d7c0 -1 osd.0 0 log_to_monitors true 2024-02-01T10:12:41.024 INFO:tasks.ceph.c2.mgr.x.smithi187.stderr:2024-02-01T10:12:41.023+0000 7f8f2e811640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-02-01T10:12:41.300 INFO:tasks.ceph.c2.osd.2.smithi187.stderr:2024-02-01T10:12:41.299+0000 7f6bb9c55640 -1 osd.2 0 waiting for initial osdmap 2024-02-01T10:12:41.319 INFO:tasks.ceph.c2.osd.2.smithi187.stderr:2024-02-01T10:12:41.319+0000 7f6bb1a90640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-01T10:12:42.831 INFO:tasks.ceph.c2.osd.1.smithi187.stderr:2024-02-01T10:12:42.831+0000 7fd236f90640 -1 osd.1 0 waiting for initial osdmap 2024-02-01T10:12:42.849 INFO:tasks.ceph.c2.osd.0.smithi187.stderr:2024-02-01T10:12:42.847+0000 7efcf9cb6640 -1 osd.0 0 waiting for initial osdmap 2024-02-01T10:12:42.855 INFO:tasks.ceph.c2.osd.1.smithi187.stderr:2024-02-01T10:12:42.855+0000 7fd22edcb640 -1 osd.1 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-01T10:12:42.880 INFO:tasks.ceph.c2.osd.0.smithi187.stderr:2024-02-01T10:12:42.879+0000 7efcf1af1640 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-01T10:12:43.288 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2024-02-01T10:12:43.578 INFO:teuthology.misc.health.smithi187.stdout: 2024-02-01T10:12:43.578 INFO:teuthology.misc.health.smithi187.stdout:{"epoch":9,"fsid":"94d64532-9ed5-4e0f-80ec-57730a638832","created":"2024-02-01T10:12:35.337929+0000","modified":"2024-02-01T10:12:43.073636+0000","last_up_change":"2024-02-01T10:12:43.073636+0000","last_in_change":"2024-02-01T10:12:36.232711+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c7150bad-9637-4fd0-a5b1-3a82ce15afe7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":3373160298},{"type":"v1","addr":"172.21.15.187:6801","nonce":3373160298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6803","nonce":3373160298},{"type":"v1","addr":"172.21.15.187:6805","nonce":3373160298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6811","nonce":3373160298},{"type":"v1","addr":"172.21.15.187:6813","nonce":3373160298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6807","nonce":3373160298},{"type":"v1","addr":"172.21.15.187:6809","nonce":3373160298}]},"public_addr":"172.21.15.187:6801/3373160298","cluster_addr":"172.21.15.187:6805/3373160298","heartbeat_back_addr":"172.21.15.187:6813/3373160298","heartbeat_front_addr":"172.21.15.187:6809/3373160298","state":["exists","up"]},{"osd":1,"uuid":"c5fef800-a312-4f36-9eb5-b4d0435f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":4226563690},{"type":"v1","addr":"172.21.15.187:6804","nonce":4226563690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":4226563690},{"type":"v1","addr":"172.21.15.187:6808","nonce":4226563690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":4226563690},{"type":"v1","addr":"172.21.15.187:6815","nonce":4226563690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":4226563690},{"type":"v1","addr":"172.21.15.187:6812","nonce":4226563690}]},"public_addr":"172.21.15.187:6804/4226563690","cluster_addr":"172.21.15.187:6808/4226563690","heartbeat_back_addr":"172.21.15.187:6815/4226563690","heartbeat_front_addr":"172.21.15.187:6812/4226563690","state":["exists","up"]},{"osd":2,"uuid":"90c21075-9f63-44fa-a85c-b8968dd4f3c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":1607205092},{"type":"v1","addr":"172.21.15.187:6817","nonce":1607205092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":1607205092},{"type":"v1","addr":"172.21.15.187:6819","nonce":1607205092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":1607205092},{"type":"v1","addr":"172.21.15.187:6823","nonce":1607205092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":1607205092},{"type":"v1","addr":"172.21.15.187:6821","nonce":1607205092}]},"public_addr":"172.21.15.187:6817/1607205092","cluster_addr":"172.21.15.187:6819/1607205092","heartbeat_back_addr":"172.21.15.187:6823/1607205092","heartbeat_front_addr":"172.21.15.187:6821/1607205092","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-01T10:12:40.717380+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-01T10:12:43.596 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-02-01T10:12:43.596 INFO:tasks.ceph:Creating RBD pool 2024-02-01T10:12:43.597 DEBUG:teuthology.orchestra.run.smithi187:> sudo ceph --cluster c2 osd pool create rbd 8 2024-02-01T10:12:44.773 INFO:teuthology.orchestra.run.smithi187.stderr:pool 'rbd' created 2024-02-01T10:12:44.798 DEBUG:teuthology.orchestra.run.smithi187:> sudo ceph --cluster c2 osd pool application enable rbd rbd --yes-i-really-mean-it 2024-02-01T10:12:45.776 INFO:teuthology.orchestra.run.smithi187.stderr:enabled application 'rbd' on pool 'rbd' 2024-02-01T10:12:45.809 INFO:tasks.ceph:Starting mds daemons in cluster c2... 2024-02-01T10:12:45.809 INFO:tasks.ceph:Waiting until c2 daemons up and pgs clean... 2024-02-01T10:12:45.809 INFO:tasks.ceph.ceph_manager.c2:waiting for mgr available 2024-02-01T10:12:45.809 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 mgr dump --format=json 2024-02-01T10:12:45.810 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-02-01T10:12:46.152 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-02-01T10:12:46.169 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":4,"flags":0,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":2117054634},{"type":"v1","addr":"172.21.15.187:6825","nonce":2117054634}]},"active_addr":"172.21.15.187:6825/2117054634","active_change":"2024-02-01T10:12:40.005668+0000","active_mgr_features":4540701272860131327,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.187:0","nonce":1363297237}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.187:0","nonce":3056081041}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.187:0","nonce":3814592826}]}]} 2024-02-01T10:12:46.174 INFO:tasks.ceph.ceph_manager.c2:mgr available! 2024-02-01T10:12:46.174 INFO:tasks.ceph.ceph_manager.c2:waiting for all up 2024-02-01T10:12:46.174 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2024-02-01T10:12:46.460 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-02-01T10:12:46.460 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":11,"fsid":"94d64532-9ed5-4e0f-80ec-57730a638832","created":"2024-02-01T10:12:35.337929+0000","modified":"2024-02-01T10:12:45.774439+0000","last_up_change":"2024-02-01T10:12:43.073636+0000","last_in_change":"2024-02-01T10:12:36.232711+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-02-01T10:12:43.923865+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-02-01T10:12:44.078254+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c7150bad-9637-4fd0-a5b1-3a82ce15afe7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":3373160298},{"type":"v1","addr":"172.21.15.187:6801","nonce":3373160298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6803","nonce":3373160298},{"type":"v1","addr":"172.21.15.187:6805","nonce":3373160298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6811","nonce":3373160298},{"type":"v1","addr":"172.21.15.187:6813","nonce":3373160298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6807","nonce":3373160298},{"type":"v1","addr":"172.21.15.187:6809","nonce":3373160298}]},"public_addr":"172.21.15.187:6801/3373160298","cluster_addr":"172.21.15.187:6805/3373160298","heartbeat_back_addr":"172.21.15.187:6813/3373160298","heartbeat_front_addr":"172.21.15.187:6809/3373160298","state":["exists","up"]},{"osd":1,"uuid":"c5fef800-a312-4f36-9eb5-b4d0435f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":4226563690},{"type":"v1","addr":"172.21.15.187:6804","nonce":4226563690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":4226563690},{"type":"v1","addr":"172.21.15.187:6808","nonce":4226563690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":4226563690},{"type":"v1","addr":"172.21.15.187:6815","nonce":4226563690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":4226563690},{"type":"v1","addr":"172.21.15.187:6812","nonce":4226563690}]},"public_addr":"172.21.15.187:6804/4226563690","cluster_addr":"172.21.15.187:6808/4226563690","heartbeat_back_addr":"172.21.15.187:6815/4226563690","heartbeat_front_addr":"172.21.15.187:6812/4226563690","state":["exists","up"]},{"osd":2,"uuid":"90c21075-9f63-44fa-a85c-b8968dd4f3c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":1607205092},{"type":"v1","addr":"172.21.15.187:6817","nonce":1607205092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":1607205092},{"type":"v1","addr":"172.21.15.187:6819","nonce":1607205092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":1607205092},{"type":"v1","addr":"172.21.15.187:6823","nonce":1607205092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":1607205092},{"type":"v1","addr":"172.21.15.187:6821","nonce":1607205092}]},"public_addr":"172.21.15.187:6817/1607205092","cluster_addr":"172.21.15.187:6819/1607205092","heartbeat_back_addr":"172.21.15.187:6823/1607205092","heartbeat_front_addr":"172.21.15.187:6821/1607205092","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-01T10:12:40.971222+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-01T10:12:40.934641+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-01T10:12:40.717380+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-01T10:12:46.478 INFO:tasks.ceph.ceph_manager.c2:all up! 2024-02-01T10:12:46.478 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2024-02-01T10:12:46.764 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-02-01T10:12:46.764 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":11,"fsid":"94d64532-9ed5-4e0f-80ec-57730a638832","created":"2024-02-01T10:12:35.337929+0000","modified":"2024-02-01T10:12:45.774439+0000","last_up_change":"2024-02-01T10:12:43.073636+0000","last_in_change":"2024-02-01T10:12:36.232711+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-02-01T10:12:43.923865+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-02-01T10:12:44.078254+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c7150bad-9637-4fd0-a5b1-3a82ce15afe7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":3373160298},{"type":"v1","addr":"172.21.15.187:6801","nonce":3373160298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6803","nonce":3373160298},{"type":"v1","addr":"172.21.15.187:6805","nonce":3373160298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6811","nonce":3373160298},{"type":"v1","addr":"172.21.15.187:6813","nonce":3373160298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6807","nonce":3373160298},{"type":"v1","addr":"172.21.15.187:6809","nonce":3373160298}]},"public_addr":"172.21.15.187:6801/3373160298","cluster_addr":"172.21.15.187:6805/3373160298","heartbeat_back_addr":"172.21.15.187:6813/3373160298","heartbeat_front_addr":"172.21.15.187:6809/3373160298","state":["exists","up"]},{"osd":1,"uuid":"c5fef800-a312-4f36-9eb5-b4d0435f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":4226563690},{"type":"v1","addr":"172.21.15.187:6804","nonce":4226563690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":4226563690},{"type":"v1","addr":"172.21.15.187:6808","nonce":4226563690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":4226563690},{"type":"v1","addr":"172.21.15.187:6815","nonce":4226563690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":4226563690},{"type":"v1","addr":"172.21.15.187:6812","nonce":4226563690}]},"public_addr":"172.21.15.187:6804/4226563690","cluster_addr":"172.21.15.187:6808/4226563690","heartbeat_back_addr":"172.21.15.187:6815/4226563690","heartbeat_front_addr":"172.21.15.187:6812/4226563690","state":["exists","up"]},{"osd":2,"uuid":"90c21075-9f63-44fa-a85c-b8968dd4f3c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":1607205092},{"type":"v1","addr":"172.21.15.187:6817","nonce":1607205092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":1607205092},{"type":"v1","addr":"172.21.15.187:6819","nonce":1607205092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":1607205092},{"type":"v1","addr":"172.21.15.187:6823","nonce":1607205092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":1607205092},{"type":"v1","addr":"172.21.15.187:6821","nonce":1607205092}]},"public_addr":"172.21.15.187:6817/1607205092","cluster_addr":"172.21.15.187:6819/1607205092","heartbeat_back_addr":"172.21.15.187:6823/1607205092","heartbeat_front_addr":"172.21.15.187:6821/1607205092","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-01T10:12:40.971222+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-01T10:12:40.934641+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-01T10:12:40.717380+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-01T10:12:46.783 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.0 flush_pg_stats 2024-02-01T10:12:46.783 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.1 flush_pg_stats 2024-02-01T10:12:46.784 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.2 flush_pg_stats 2024-02-01T10:12:46.945 INFO:teuthology.orchestra.run.smithi187.stdout:38654705668 2024-02-01T10:12:46.945 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2024-02-01T10:12:46.954 INFO:teuthology.orchestra.run.smithi187.stdout:38654705667 2024-02-01T10:12:46.955 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2024-02-01T10:12:46.956 INFO:teuthology.orchestra.run.smithi187.stdout:34359738370 2024-02-01T10:12:46.956 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2024-02-01T10:12:47.234 INFO:teuthology.orchestra.run.smithi187.stdout:38654705667 2024-02-01T10:12:47.251 INFO:tasks.ceph.ceph_manager.c2:need seq 38654705668 got 38654705667 for osd.1 2024-02-01T10:12:47.280 INFO:teuthology.orchestra.run.smithi187.stdout:38654705666 2024-02-01T10:12:47.282 INFO:teuthology.orchestra.run.smithi187.stdout:34359738369 2024-02-01T10:12:47.299 INFO:tasks.ceph.ceph_manager.c2:need seq 38654705667 got 38654705666 for osd.0 2024-02-01T10:12:47.301 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738370 got 34359738369 for osd.2 2024-02-01T10:12:48.252 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2024-02-01T10:12:48.300 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2024-02-01T10:12:48.301 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2024-02-01T10:12:48.580 INFO:teuthology.orchestra.run.smithi187.stdout:38654705667 2024-02-01T10:12:48.584 INFO:teuthology.orchestra.run.smithi187.stdout:34359738369 2024-02-01T10:12:48.601 INFO:tasks.ceph.ceph_manager.c2:need seq 38654705668 got 38654705667 for osd.1 2024-02-01T10:12:48.603 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738370 got 34359738369 for osd.2 2024-02-01T10:12:48.611 INFO:teuthology.orchestra.run.smithi187.stdout:38654705666 2024-02-01T10:12:48.628 INFO:tasks.ceph.ceph_manager.c2:need seq 38654705667 got 38654705666 for osd.0 2024-02-01T10:12:49.602 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2024-02-01T10:12:49.603 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2024-02-01T10:12:49.629 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2024-02-01T10:12:49.894 INFO:teuthology.orchestra.run.smithi187.stdout:34359738371 2024-02-01T10:12:49.899 INFO:teuthology.orchestra.run.smithi187.stdout:38654705668 2024-02-01T10:12:49.914 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738370 got 34359738371 for osd.2 2024-02-01T10:12:49.914 DEBUG:teuthology.parallel:result is None 2024-02-01T10:12:49.919 INFO:tasks.ceph.ceph_manager.c2:need seq 38654705668 got 38654705668 for osd.1 2024-02-01T10:12:49.919 DEBUG:teuthology.parallel:result is None 2024-02-01T10:12:49.933 INFO:teuthology.orchestra.run.smithi187.stdout:38654705667 2024-02-01T10:12:49.950 INFO:tasks.ceph.ceph_manager.c2:need seq 38654705667 got 38654705667 for osd.0 2024-02-01T10:12:49.950 DEBUG:teuthology.parallel:result is None 2024-02-01T10:12:49.950 INFO:tasks.ceph.ceph_manager.c2:waiting for clean 2024-02-01T10:12:49.950 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2024-02-01T10:12:50.260 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-02-01T10:12:50.260 INFO:teuthology.orchestra.run.smithi187.stderr:dumped all 2024-02-01T10:12:50.276 INFO:teuthology.orchestra.run.smithi187.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2024-02-01T10:12:50.011144+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":6,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":83024,"kb_used_data":1464,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":314489776,"statfs":{"total":322122547200,"available":322037530624,"internally_reserved":0,"allocated":1499136,"data_stored":1263216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":82963791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000487"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-02-01T10:12:47.198094+0000","last_change":"2024-02-01T10:12:45.788731+0000","last_active":"2024-02-01T10:12:47.198094+0000","last_peered":"2024-02-01T10:12:47.198094+0000","last_clean":"2024-02-01T10:12:47.198094+0000","last_became_active":"2024-02-01T10:12:45.788389+0000","last_became_peered":"2024-02-01T10:12:45.788389+0000","last_unstale":"2024-02-01T10:12:47.198094+0000","last_undegraded":"2024-02-01T10:12:47.198094+0000","last_fullsized":"2024-02-01T10:12:47.198094+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-01T10:12:44.768683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-01T10:12:44.768683+0000","last_clean_scrub_stamp":"2024-02-01T10:12:44.768683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-02T19:18:12.294663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":12,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-02-01T10:12:47.198001+0000","last_change":"2024-02-01T10:12:45.792040+0000","last_active":"2024-02-01T10:12:47.198001+0000","last_peered":"2024-02-01T10:12:47.198001+0000","last_clean":"2024-02-01T10:12:47.198001+0000","last_became_active":"2024-02-01T10:12:45.790610+0000","last_became_peered":"2024-02-01T10:12:45.790610+0000","last_unstale":"2024-02-01T10:12:47.198001+0000","last_undegraded":"2024-02-01T10:12:47.198001+0000","last_fullsized":"2024-02-01T10:12:47.198001+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-01T10:12:44.768683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-01T10:12:44.768683+0000","last_clean_scrub_stamp":"2024-02-01T10:12:44.768683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-02T16:10:52.444579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":12,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-02-01T10:12:46.791168+0000","last_change":"2024-02-01T10:12:45.789539+0000","last_active":"2024-02-01T10:12:46.791168+0000","last_peered":"2024-02-01T10:12:46.791168+0000","last_clean":"2024-02-01T10:12:46.791168+0000","last_became_active":"2024-02-01T10:12:45.789079+0000","last_became_peered":"2024-02-01T10:12:45.789079+0000","last_unstale":"2024-02-01T10:12:46.791168+0000","last_undegraded":"2024-02-01T10:12:46.791168+0000","last_fullsized":"2024-02-01T10:12:46.791168+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-01T10:12:44.768683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-01T10:12:44.768683+0000","last_clean_scrub_stamp":"2024-02-01T10:12:44.768683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-02T15:06:29.621736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":12,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-02-01T10:12:47.197880+0000","last_change":"2024-02-01T10:12:45.792453+0000","last_active":"2024-02-01T10:12:47.197880+0000","last_peered":"2024-02-01T10:12:47.197880+0000","last_clean":"2024-02-01T10:12:47.197880+0000","last_became_active":"2024-02-01T10:12:45.791564+0000","last_became_peered":"2024-02-01T10:12:45.791564+0000","last_unstale":"2024-02-01T10:12:47.197880+0000","last_undegraded":"2024-02-01T10:12:47.197880+0000","last_fullsized":"2024-02-01T10:12:47.197880+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-01T10:12:44.768683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-01T10:12:44.768683+0000","last_clean_scrub_stamp":"2024-02-01T10:12:44.768683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-02T17:57:03.604151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"11'97","reported_seq":139,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-02-01T10:12:46.790637+0000","last_change":"2024-02-01T10:12:45.792850+0000","last_active":"2024-02-01T10:12:46.790637+0000","last_peered":"2024-02-01T10:12:46.790637+0000","last_clean":"2024-02-01T10:12:46.790637+0000","last_became_active":"2024-02-01T10:12:45.792449+0000","last_became_peered":"2024-02-01T10:12:45.792449+0000","last_unstale":"2024-02-01T10:12:46.790637+0000","last_undegraded":"2024-02-01T10:12:46.790637+0000","last_fullsized":"2024-02-01T10:12:46.790637+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-01T10:12:44.768683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-01T10:12:44.768683+0000","last_clean_scrub_stamp":"2024-02-01T10:12:44.768683+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-02T12:33:56.259108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":12,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-02-01T10:12:47.197848+0000","last_change":"2024-02-01T10:12:45.788818+0000","last_active":"2024-02-01T10:12:47.197848+0000","last_peered":"2024-02-01T10:12:47.197848+0000","last_clean":"2024-02-01T10:12:47.197848+0000","last_became_active":"2024-02-01T10:12:45.788477+0000","last_became_peered":"2024-02-01T10:12:45.788477+0000","last_unstale":"2024-02-01T10:12:47.197848+0000","last_undegraded":"2024-02-01T10:12:47.197848+0000","last_fullsized":"2024-02-01T10:12:47.197848+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-01T10:12:44.768683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-01T10:12:44.768683+0000","last_clean_scrub_stamp":"2024-02-01T10:12:44.768683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-02T12:34:10.071043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":12,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-02-01T10:12:47.199812+0000","last_change":"2024-02-01T10:12:45.799194+0000","last_active":"2024-02-01T10:12:47.199812+0000","last_peered":"2024-02-01T10:12:47.199812+0000","last_clean":"2024-02-01T10:12:47.199812+0000","last_became_active":"2024-02-01T10:12:45.798743+0000","last_became_peered":"2024-02-01T10:12:45.798743+0000","last_unstale":"2024-02-01T10:12:47.199812+0000","last_undegraded":"2024-02-01T10:12:47.199812+0000","last_fullsized":"2024-02-01T10:12:47.199812+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-01T10:12:44.768683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-01T10:12:44.768683+0000","last_clean_scrub_stamp":"2024-02-01T10:12:44.768683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-02T11:03:12.274895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":12,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-02-01T10:12:46.790780+0000","last_change":"2024-02-01T10:12:45.794169+0000","last_active":"2024-02-01T10:12:46.790780+0000","last_peered":"2024-02-01T10:12:46.790780+0000","last_clean":"2024-02-01T10:12:46.790780+0000","last_became_active":"2024-02-01T10:12:45.793726+0000","last_became_peered":"2024-02-01T10:12:45.793726+0000","last_unstale":"2024-02-01T10:12:46.790780+0000","last_undegraded":"2024-02-01T10:12:46.790780+0000","last_fullsized":"2024-02-01T10:12:46.790780+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-01T10:12:44.768683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-01T10:12:44.768683+0000","last_clean_scrub_stamp":"2024-02-01T10:12:44.768683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-02T13:14:10.106065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-02-01T10:12:47.198427+0000","last_change":"2024-02-01T10:12:45.791425+0000","last_active":"2024-02-01T10:12:47.198427+0000","last_peered":"2024-02-01T10:12:47.198427+0000","last_clean":"2024-02-01T10:12:47.198427+0000","last_became_active":"2024-02-01T10:12:45.790709+0000","last_became_peered":"2024-02-01T10:12:45.790709+0000","last_unstale":"2024-02-01T10:12:47.198427+0000","last_undegraded":"2024-02-01T10:12:47.198427+0000","last_fullsized":"2024-02-01T10:12:47.198427+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-01T10:12:44.768683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-01T10:12:44.768683+0000","last_clean_scrub_stamp":"2024-02-01T10:12:44.768683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-02T19:26:49.751985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27704,"kb_used_data":684,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829896,"statfs":{"total":107374182400,"available":107345813504,"internally_reserved":0,"allocated":700416,"data_stored":619144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63300000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57999999999999996}]}]},{"osd":0,"up_from":9,"seq":38654705668,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27124,"kb_used_data":104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830476,"statfs":{"total":107374182400,"available":107346407424,"internally_reserved":0,"allocated":106496,"data_stored":28776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.629}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60399999999999998}]}]},{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":28196,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829404,"statfs":{"total":107374182400,"available":107345309696,"internally_reserved":0,"allocated":692224,"data_stored":615296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56299999999999994}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47599999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-01T10:12:50.277 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2024-02-01T10:12:50.544 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-02-01T10:12:50.545 INFO:teuthology.orchestra.run.smithi187.stderr:dumped all 2024-02-01T10:12:50.561 INFO:teuthology.orchestra.run.smithi187.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2024-02-01T10:12:50.011144+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":6,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":83024,"kb_used_data":1464,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":314489776,"statfs":{"total":322122547200,"available":322037530624,"internally_reserved":0,"allocated":1499136,"data_stored":1263216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":82963791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000487"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-02-01T10:12:47.198094+0000","last_change":"2024-02-01T10:12:45.788731+0000","last_active":"2024-02-01T10:12:47.198094+0000","last_peered":"2024-02-01T10:12:47.198094+0000","last_clean":"2024-02-01T10:12:47.198094+0000","last_became_active":"2024-02-01T10:12:45.788389+0000","last_became_peered":"2024-02-01T10:12:45.788389+0000","last_unstale":"2024-02-01T10:12:47.198094+0000","last_undegraded":"2024-02-01T10:12:47.198094+0000","last_fullsized":"2024-02-01T10:12:47.198094+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-01T10:12:44.768683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-01T10:12:44.768683+0000","last_clean_scrub_stamp":"2024-02-01T10:12:44.768683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-02T19:18:12.294663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":12,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-02-01T10:12:47.198001+0000","last_change":"2024-02-01T10:12:45.792040+0000","last_active":"2024-02-01T10:12:47.198001+0000","last_peered":"2024-02-01T10:12:47.198001+0000","last_clean":"2024-02-01T10:12:47.198001+0000","last_became_active":"2024-02-01T10:12:45.790610+0000","last_became_peered":"2024-02-01T10:12:45.790610+0000","last_unstale":"2024-02-01T10:12:47.198001+0000","last_undegraded":"2024-02-01T10:12:47.198001+0000","last_fullsized":"2024-02-01T10:12:47.198001+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-01T10:12:44.768683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-01T10:12:44.768683+0000","last_clean_scrub_stamp":"2024-02-01T10:12:44.768683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-02T16:10:52.444579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":12,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-02-01T10:12:46.791168+0000","last_change":"2024-02-01T10:12:45.789539+0000","last_active":"2024-02-01T10:12:46.791168+0000","last_peered":"2024-02-01T10:12:46.791168+0000","last_clean":"2024-02-01T10:12:46.791168+0000","last_became_active":"2024-02-01T10:12:45.789079+0000","last_became_peered":"2024-02-01T10:12:45.789079+0000","last_unstale":"2024-02-01T10:12:46.791168+0000","last_undegraded":"2024-02-01T10:12:46.791168+0000","last_fullsized":"2024-02-01T10:12:46.791168+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-01T10:12:44.768683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-01T10:12:44.768683+0000","last_clean_scrub_stamp":"2024-02-01T10:12:44.768683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-02T15:06:29.621736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":12,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-02-01T10:12:47.197880+0000","last_change":"2024-02-01T10:12:45.792453+0000","last_active":"2024-02-01T10:12:47.197880+0000","last_peered":"2024-02-01T10:12:47.197880+0000","last_clean":"2024-02-01T10:12:47.197880+0000","last_became_active":"2024-02-01T10:12:45.791564+0000","last_became_peered":"2024-02-01T10:12:45.791564+0000","last_unstale":"2024-02-01T10:12:47.197880+0000","last_undegraded":"2024-02-01T10:12:47.197880+0000","last_fullsized":"2024-02-01T10:12:47.197880+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-01T10:12:44.768683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-01T10:12:44.768683+0000","last_clean_scrub_stamp":"2024-02-01T10:12:44.768683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-02T17:57:03.604151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"11'97","reported_seq":139,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-02-01T10:12:46.790637+0000","last_change":"2024-02-01T10:12:45.792850+0000","last_active":"2024-02-01T10:12:46.790637+0000","last_peered":"2024-02-01T10:12:46.790637+0000","last_clean":"2024-02-01T10:12:46.790637+0000","last_became_active":"2024-02-01T10:12:45.792449+0000","last_became_peered":"2024-02-01T10:12:45.792449+0000","last_unstale":"2024-02-01T10:12:46.790637+0000","last_undegraded":"2024-02-01T10:12:46.790637+0000","last_fullsized":"2024-02-01T10:12:46.790637+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-01T10:12:44.768683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-01T10:12:44.768683+0000","last_clean_scrub_stamp":"2024-02-01T10:12:44.768683+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-02T12:33:56.259108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":12,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-02-01T10:12:47.197848+0000","last_change":"2024-02-01T10:12:45.788818+0000","last_active":"2024-02-01T10:12:47.197848+0000","last_peered":"2024-02-01T10:12:47.197848+0000","last_clean":"2024-02-01T10:12:47.197848+0000","last_became_active":"2024-02-01T10:12:45.788477+0000","last_became_peered":"2024-02-01T10:12:45.788477+0000","last_unstale":"2024-02-01T10:12:47.197848+0000","last_undegraded":"2024-02-01T10:12:47.197848+0000","last_fullsized":"2024-02-01T10:12:47.197848+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-01T10:12:44.768683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-01T10:12:44.768683+0000","last_clean_scrub_stamp":"2024-02-01T10:12:44.768683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-02T12:34:10.071043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":12,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-02-01T10:12:47.199812+0000","last_change":"2024-02-01T10:12:45.799194+0000","last_active":"2024-02-01T10:12:47.199812+0000","last_peered":"2024-02-01T10:12:47.199812+0000","last_clean":"2024-02-01T10:12:47.199812+0000","last_became_active":"2024-02-01T10:12:45.798743+0000","last_became_peered":"2024-02-01T10:12:45.798743+0000","last_unstale":"2024-02-01T10:12:47.199812+0000","last_undegraded":"2024-02-01T10:12:47.199812+0000","last_fullsized":"2024-02-01T10:12:47.199812+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-01T10:12:44.768683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-01T10:12:44.768683+0000","last_clean_scrub_stamp":"2024-02-01T10:12:44.768683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-02T11:03:12.274895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":12,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-02-01T10:12:46.790780+0000","last_change":"2024-02-01T10:12:45.794169+0000","last_active":"2024-02-01T10:12:46.790780+0000","last_peered":"2024-02-01T10:12:46.790780+0000","last_clean":"2024-02-01T10:12:46.790780+0000","last_became_active":"2024-02-01T10:12:45.793726+0000","last_became_peered":"2024-02-01T10:12:45.793726+0000","last_unstale":"2024-02-01T10:12:46.790780+0000","last_undegraded":"2024-02-01T10:12:46.790780+0000","last_fullsized":"2024-02-01T10:12:46.790780+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-01T10:12:44.768683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-01T10:12:44.768683+0000","last_clean_scrub_stamp":"2024-02-01T10:12:44.768683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-02T13:14:10.106065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-02-01T10:12:47.198427+0000","last_change":"2024-02-01T10:12:45.791425+0000","last_active":"2024-02-01T10:12:47.198427+0000","last_peered":"2024-02-01T10:12:47.198427+0000","last_clean":"2024-02-01T10:12:47.198427+0000","last_became_active":"2024-02-01T10:12:45.790709+0000","last_became_peered":"2024-02-01T10:12:45.790709+0000","last_unstale":"2024-02-01T10:12:47.198427+0000","last_undegraded":"2024-02-01T10:12:47.198427+0000","last_fullsized":"2024-02-01T10:12:47.198427+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-01T10:12:44.768683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-01T10:12:44.768683+0000","last_clean_scrub_stamp":"2024-02-01T10:12:44.768683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-02T19:26:49.751985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27704,"kb_used_data":684,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829896,"statfs":{"total":107374182400,"available":107345813504,"internally_reserved":0,"allocated":700416,"data_stored":619144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63300000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57999999999999996}]}]},{"osd":0,"up_from":9,"seq":38654705668,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27124,"kb_used_data":104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830476,"statfs":{"total":107374182400,"available":107346407424,"internally_reserved":0,"allocated":106496,"data_stored":28776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.629}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60399999999999998}]}]},{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":28196,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829404,"statfs":{"total":107374182400,"available":107345309696,"internally_reserved":0,"allocated":692224,"data_stored":615296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56299999999999994}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47599999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-01T10:12:50.562 INFO:tasks.ceph.ceph_manager.c2:clean! 2024-02-01T10:12:50.562 INFO:tasks.ceph:Waiting until ceph cluster c2 is healthy... 2024-02-01T10:12:50.562 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy 2024-02-01T10:12:50.562 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 health --format=json 2024-02-01T10:12:50.852 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-02-01T10:12:50.852 INFO:teuthology.orchestra.run.smithi187.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-02-01T10:12:50.869 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy done 2024-02-01T10:12:50.870 INFO:teuthology.run_tasks:Running task rgw... 2024-02-01T10:12:50.884 DEBUG:tasks.rgw:config is {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}} 2024-02-01T10:12:50.884 DEBUG:tasks.rgw:client list is dict_keys(['c1.client.0', 'c1.client.1', 'c2.client.0', 'c2.client.1']) 2024-02-01T10:12:50.885 INFO:tasks.rgw:Creating data pools 2024-02-01T10:12:50.885 DEBUG:tasks.rgw:Obtaining remote for client c1.client.0 2024-02-01T10:12:50.885 DEBUG:teuthology.orchestra.run.smithi148:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2024-02-01T10:12:51.594 INFO:teuthology.orchestra.run.smithi148.stderr:pool 'default.rgw.buckets.data' created 2024-02-01T10:12:51.629 DEBUG:teuthology.orchestra.run.smithi148:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2024-02-01T10:12:52.604 INFO:teuthology.orchestra.run.smithi148.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2024-02-01T10:12:52.647 DEBUG:teuthology.orchestra.run.smithi148:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2024-02-01T10:12:53.607 INFO:teuthology.orchestra.run.smithi148.stderr:pool 'default.rgw.buckets.index' created 2024-02-01T10:12:53.642 DEBUG:teuthology.orchestra.run.smithi148:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2024-02-01T10:12:54.615 INFO:teuthology.orchestra.run.smithi148.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2024-02-01T10:12:54.652 DEBUG:tasks.rgw:Obtaining remote for client c1.client.1 2024-02-01T10:12:54.653 DEBUG:teuthology.orchestra.run.smithi148:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2024-02-01T10:12:54.947 INFO:teuthology.orchestra.run.smithi148.stderr:pool 'default.rgw.buckets.data' already exists 2024-02-01T10:12:54.966 DEBUG:teuthology.orchestra.run.smithi148:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2024-02-01T10:12:55.629 INFO:teuthology.orchestra.run.smithi148.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2024-02-01T10:12:55.654 DEBUG:teuthology.orchestra.run.smithi148:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2024-02-01T10:12:55.945 INFO:teuthology.orchestra.run.smithi148.stderr:pool 'default.rgw.buckets.index' already exists 2024-02-01T10:12:55.963 DEBUG:teuthology.orchestra.run.smithi148:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2024-02-01T10:12:56.630 INFO:teuthology.orchestra.run.smithi148.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2024-02-01T10:12:56.655 DEBUG:tasks.rgw:Obtaining remote for client c2.client.0 2024-02-01T10:12:56.655 DEBUG:teuthology.orchestra.run.smithi187:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2024-02-01T10:12:57.304 INFO:teuthology.orchestra.run.smithi187.stderr:pool 'default.rgw.buckets.data' created 2024-02-01T10:12:57.338 DEBUG:teuthology.orchestra.run.smithi187:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2024-02-01T10:12:58.310 INFO:teuthology.orchestra.run.smithi187.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2024-02-01T10:12:58.365 DEBUG:teuthology.orchestra.run.smithi187:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2024-02-01T10:12:59.318 INFO:teuthology.orchestra.run.smithi187.stderr:pool 'default.rgw.buckets.index' created 2024-02-01T10:12:59.351 DEBUG:teuthology.orchestra.run.smithi187:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2024-02-01T10:13:00.323 INFO:teuthology.orchestra.run.smithi187.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2024-02-01T10:13:00.362 DEBUG:tasks.rgw:Obtaining remote for client c2.client.1 2024-02-01T10:13:00.362 DEBUG:teuthology.orchestra.run.smithi187:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2024-02-01T10:13:00.676 INFO:teuthology.orchestra.run.smithi187.stderr:pool 'default.rgw.buckets.data' already exists 2024-02-01T10:13:00.695 DEBUG:teuthology.orchestra.run.smithi187:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2024-02-01T10:13:01.324 INFO:teuthology.orchestra.run.smithi187.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2024-02-01T10:13:01.349 DEBUG:teuthology.orchestra.run.smithi187:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2024-02-01T10:13:01.632 INFO:teuthology.orchestra.run.smithi187.stderr:pool 'default.rgw.buckets.index' already exists 2024-02-01T10:13:01.650 DEBUG:teuthology.orchestra.run.smithi187:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2024-02-01T10:13:02.334 INFO:teuthology.orchestra.run.smithi187.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2024-02-01T10:13:02.359 DEBUG:tasks.rgw:Pools created 2024-02-01T10:13:02.359 INFO:tasks.rgw:Configuring compression type = random 2024-02-01T10:13:02.359 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'list'] 2024-02-01T10:13:02.359 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'list'] 2024-02-01T10:13:02.360 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user list 2024-02-01T10:13:02.422 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:13:02.422 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:13:03.739 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:03.739+0000 7f947f02b980 20 rados->read ofs=0 len=0 2024-02-01T10:13:03.740 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:03.739+0000 7f947f02b980 20 rados_obj.operate() r=-2 bl.length=0 2024-02-01T10:13:03.740 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:03.739+0000 7f947f02b980 20 realm 2024-02-01T10:13:03.740 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:03.739+0000 7f947f02b980 20 rados->read ofs=0 len=0 2024-02-01T10:13:03.741 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:03.739+0000 7f947f02b980 20 rados_obj.operate() r=-2 bl.length=0 2024-02-01T10:13:03.741 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:03.739+0000 7f947f02b980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-02-01T10:13:03.741 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:03.739+0000 7f947f02b980 20 rados->read ofs=0 len=0 2024-02-01T10:13:03.741 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:03.739+0000 7f947f02b980 20 rados_obj.operate() r=-2 bl.length=0 2024-02-01T10:13:03.741 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:03.739+0000 7f947f02b980 20 rados->read ofs=0 len=0 2024-02-01T10:13:03.741 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:03.739+0000 7f947f02b980 20 rados_obj.operate() r=0 bl.length=46 2024-02-01T10:13:03.741 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:03.739+0000 7f947f02b980 20 rados->read ofs=0 len=0 2024-02-01T10:13:03.741 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:03.739+0000 7f947f02b980 20 rados_obj.operate() r=0 bl.length=874 2024-02-01T10:13:03.742 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:03.739+0000 7f947f02b980 20 searching for the correct realm 2024-02-01T10:13:03.752 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:03.751+0000 7f947f02b980 20 RGWRados::pool_iterate: got zone_info.0b737f8c-eed6-42ec-8b9e-a02467818b0e 2024-02-01T10:13:03.752 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:03.751+0000 7f947f02b980 20 RGWRados::pool_iterate: got zonegroup_info.adf21334-61e1-4708-b003-aff334eb28c9 2024-02-01T10:13:03.752 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:03.751+0000 7f947f02b980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-02-01T10:13:03.753 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:03.751+0000 7f947f02b980 20 RGWRados::pool_iterate: got default.zone. 2024-02-01T10:13:03.753 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:03.751+0000 7f947f02b980 20 RGWRados::pool_iterate: got zone_names.default 2024-02-01T10:13:03.753 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:03.751+0000 7f947f02b980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-02-01T10:13:03.753 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:03.751+0000 7f947f02b980 20 rados->read ofs=0 len=0 2024-02-01T10:13:03.753 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:03.751+0000 7f947f02b980 20 rados_obj.operate() r=-2 bl.length=0 2024-02-01T10:13:03.753 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:03.751+0000 7f947f02b980 20 rados->read ofs=0 len=0 2024-02-01T10:13:03.754 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:03.751+0000 7f947f02b980 20 rados_obj.operate() r=0 bl.length=46 2024-02-01T10:13:03.754 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:03.751+0000 7f947f02b980 20 rados->read ofs=0 len=0 2024-02-01T10:13:03.754 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:03.751+0000 7f947f02b980 20 rados_obj.operate() r=0 bl.length=398 2024-02-01T10:13:03.754 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:03.751+0000 7f947f02b980 20 zone default found 2024-02-01T10:13:03.754 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:03.751+0000 7f947f02b980 4 Realm: () 2024-02-01T10:13:03.754 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:03.751+0000 7f947f02b980 4 ZoneGroup: default (adf21334-61e1-4708-b003-aff334eb28c9) 2024-02-01T10:13:03.755 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:03.751+0000 7f947f02b980 4 Zone: default (0b737f8c-eed6-42ec-8b9e-a02467818b0e) 2024-02-01T10:13:03.755 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:03.751+0000 7f947f02b980 10 cannot find current period zonegroup using local zonegroup configuration 2024-02-01T10:13:03.755 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:03.751+0000 7f947f02b980 20 zonegroup default 2024-02-01T10:13:03.755 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:03.751+0000 7f947f02b980 20 rados->read ofs=0 len=0 2024-02-01T10:13:03.755 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:03.751+0000 7f947f02b980 20 rados_obj.operate() r=-2 bl.length=0 2024-02-01T10:13:03.755 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:03.751+0000 7f947f02b980 20 rados->read ofs=0 len=0 2024-02-01T10:13:05.699 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:05.699+0000 7f947f02b980 20 rados_obj.operate() r=-2 bl.length=0 2024-02-01T10:13:05.699 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:05.699+0000 7f947f02b980 20 rados->read ofs=0 len=0 2024-02-01T10:13:05.702 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:05.699+0000 7f947f02b980 20 rados_obj.operate() r=-2 bl.length=0 2024-02-01T10:13:05.702 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:05.699+0000 7f947f02b980 20 started sync module instance, tier type = 2024-02-01T10:13:05.702 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:05.699+0000 7f947f02b980 20 started zone id=0b737f8c-eed6-42ec-8b9e-a02467818b0e (name=default) with tier type = 2024-02-01T10:13:07.772 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:07.771+0000 7f947f02b980 20 add_watcher() i=0 2024-02-01T10:13:07.772 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:07.771+0000 7f947f02b980 20 add_watcher() i=1 2024-02-01T10:13:07.772 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:07.771+0000 7f947f02b980 20 add_watcher() i=2 2024-02-01T10:13:07.772 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:07.771+0000 7f947f02b980 20 add_watcher() i=3 2024-02-01T10:13:07.772 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:07.771+0000 7f947f02b980 20 add_watcher() i=4 2024-02-01T10:13:07.772 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:07.771+0000 7f947f02b980 20 add_watcher() i=5 2024-02-01T10:13:07.772 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:07.771+0000 7f947f02b980 20 add_watcher() i=6 2024-02-01T10:13:07.773 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:07.771+0000 7f947f02b980 20 add_watcher() i=7 2024-02-01T10:13:07.773 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:07.771+0000 7f947f02b980 2 all 8 watchers are set, enabling cache 2024-02-01T10:13:07.773 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:07.771+0000 7f947f02b980 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(const DoutPrefixProvider*, optional_yield):410: oid=data_loggenerations_metadata not found 2024-02-01T10:13:07.773 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:07.771+0000 7f947f02b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.0 2024-02-01T10:13:07.774 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:07.771+0000 7f947f02b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-02-01T10:13:07.774 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:07.771+0000 7f947f02b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.0 does not exist 2024-02-01T10:13:07.774 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:07.771+0000 7f947f02b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.1 2024-02-01T10:13:07.774 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:07.771+0000 7f947f02b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-02-01T10:13:07.774 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:07.775+0000 7f947f02b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.1 does not exist 2024-02-01T10:13:07.774 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:07.775+0000 7f947f02b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.2 2024-02-01T10:13:07.774 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:07.775+0000 7f947f02b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-02-01T10:13:07.775 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:07.775+0000 7f947f02b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.2 does not exist 2024-02-01T10:13:07.775 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:07.775+0000 7f947f02b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.3 2024-02-01T10:13:07.775 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:07.775+0000 7f947f02b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-02-01T10:13:07.775 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:07.775+0000 7f947f02b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.3 does not exist 2024-02-01T10:13:07.775 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:07.775+0000 7f947f02b980 20 static int rgw::cls::fifo::FIFO::create(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):1210 entering 2024-02-01T10:13:07.777 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:07.775+0000 7f947f02b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-02-01T10:13:07.783 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:07.783+0000 7f932efe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-02-01T10:13:07.783 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:07.783+0000 7f947f02b980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-02-01T10:13:07.784 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:07.783+0000 7f947f02b980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-02-01T10:13:10.730 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:10.727+0000 7f947f02b980 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2024-02-01T10:13:10.730 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:10.727+0000 7f947f02b980 5 note: GC not initialized 2024-02-01T10:13:10.738 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:10.735+0000 7f932dfe3640 20 reqs_thread_entry: start 2024-02-01T10:13:10.738 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:10.735+0000 7f947f02b980 20 init_complete bucket index max shards: 11 2024-02-01T10:13:10.738 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:10.739+0000 7f947f02b980 20 Filter name: none 2024-02-01T10:13:10.738 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:10.739+0000 7f928f7fe640 20 reqs_thread_entry: start 2024-02-01T10:13:10.767 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:10.767+0000 7f932efe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-02-01T10:13:10.769 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:10.767+0000 7f947f02b980 20 remove_watcher() i=0 2024-02-01T10:13:10.769 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:10.767+0000 7f947f02b980 2 removed watcher, disabling cache 2024-02-01T10:13:10.771 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:10.771+0000 7f947f02b980 20 remove_watcher() i=1 2024-02-01T10:13:10.773 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:10.771+0000 7f947f02b980 20 remove_watcher() i=2 2024-02-01T10:13:10.774 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:10.771+0000 7f947f02b980 20 remove_watcher() i=3 2024-02-01T10:13:10.775 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:10.775+0000 7f947f02b980 20 remove_watcher() i=4 2024-02-01T10:13:10.777 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:10.775+0000 7f947f02b980 20 remove_watcher() i=5 2024-02-01T10:13:10.778 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:10.779+0000 7f947f02b980 20 remove_watcher() i=6 2024-02-01T10:13:10.780 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:10.779+0000 7f947f02b980 20 remove_watcher() i=7 2024-02-01T10:13:10.796 INFO:teuthology.orchestra.run.smithi148.stdout:[] 2024-02-01T10:13:10.797 DEBUG:tasks.util.rgw: json result: [] 2024-02-01T10:13:10.797 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-02-01T10:13:10.797 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-02-01T10:13:10.797 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2024-02-01T10:13:10.837 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:13:10.838 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:13:10.866 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:10.867+0000 7f128cca5980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-02-01T10:13:10.866 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:10.867+0000 7f128cca5980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-02-01T10:13:10.869 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:10.867+0000 7f114cfe1640 20 reqs_thread_entry: start 2024-02-01T10:13:10.886 INFO:teuthology.orchestra.run.smithi148.stdout:{"id":"0b737f8c-eed6-42ec-8b9e-a02467818b0e","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"","notif_pool":"default.rgw.log:notif"} 2024-02-01T10:13:10.886 DEBUG:tasks.util.rgw: json result: {'id': '0b737f8c-eed6-42ec-8b9e-a02467818b0e', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2024-02-01T10:13:10.887 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['user', 'list'] 2024-02-01T10:13:10.887 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'user', 'list'] 2024-02-01T10:13:10.887 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 user list 2024-02-01T10:13:10.958 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:13:10.958 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:13:10.995 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:10.991+0000 7f74a05ef980 20 rados->read ofs=0 len=0 2024-02-01T10:13:10.995 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:10.991+0000 7f74a05ef980 20 rados_obj.operate() r=-2 bl.length=0 2024-02-01T10:13:10.995 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:10.991+0000 7f74a05ef980 20 realm 2024-02-01T10:13:10.995 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:10.991+0000 7f74a05ef980 20 rados->read ofs=0 len=0 2024-02-01T10:13:10.995 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:10.991+0000 7f74a05ef980 20 rados_obj.operate() r=-2 bl.length=0 2024-02-01T10:13:10.995 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:10.991+0000 7f74a05ef980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-02-01T10:13:10.995 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:10.991+0000 7f74a05ef980 20 rados->read ofs=0 len=0 2024-02-01T10:13:10.996 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:10.991+0000 7f74a05ef980 20 rados_obj.operate() r=-2 bl.length=0 2024-02-01T10:13:10.996 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:10.991+0000 7f74a05ef980 20 rados->read ofs=0 len=0 2024-02-01T10:13:10.996 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:10.995+0000 7f74a05ef980 20 rados_obj.operate() r=0 bl.length=46 2024-02-01T10:13:10.996 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:10.995+0000 7f74a05ef980 20 rados->read ofs=0 len=0 2024-02-01T10:13:10.996 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:10.995+0000 7f74a05ef980 20 rados_obj.operate() r=0 bl.length=890 2024-02-01T10:13:10.996 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:10.995+0000 7f74a05ef980 20 searching for the correct realm 2024-02-01T10:13:11.004 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:11.003+0000 7f74a05ef980 20 RGWRados::pool_iterate: got zone_info.0b737f8c-eed6-42ec-8b9e-a02467818b0e 2024-02-01T10:13:11.004 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:11.003+0000 7f74a05ef980 20 RGWRados::pool_iterate: got zonegroup_info.adf21334-61e1-4708-b003-aff334eb28c9 2024-02-01T10:13:11.005 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:11.003+0000 7f74a05ef980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-02-01T10:13:11.005 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:11.003+0000 7f74a05ef980 20 RGWRados::pool_iterate: got default.zone. 2024-02-01T10:13:11.005 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:11.003+0000 7f74a05ef980 20 RGWRados::pool_iterate: got zone_names.default 2024-02-01T10:13:11.005 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:11.003+0000 7f74a05ef980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-02-01T10:13:11.005 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:11.003+0000 7f74a05ef980 20 rados->read ofs=0 len=0 2024-02-01T10:13:11.005 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:11.003+0000 7f74a05ef980 20 rados_obj.operate() r=-2 bl.length=0 2024-02-01T10:13:11.005 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:11.003+0000 7f74a05ef980 20 rados->read ofs=0 len=0 2024-02-01T10:13:11.005 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:11.003+0000 7f74a05ef980 20 rados_obj.operate() r=0 bl.length=46 2024-02-01T10:13:11.005 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:11.003+0000 7f74a05ef980 20 rados->read ofs=0 len=0 2024-02-01T10:13:11.007 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:11.003+0000 7f74a05ef980 20 rados_obj.operate() r=0 bl.length=398 2024-02-01T10:13:11.007 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:11.003+0000 7f74a05ef980 20 zone default found 2024-02-01T10:13:11.007 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:11.003+0000 7f74a05ef980 4 Realm: () 2024-02-01T10:13:11.007 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:11.003+0000 7f74a05ef980 4 ZoneGroup: default (adf21334-61e1-4708-b003-aff334eb28c9) 2024-02-01T10:13:11.007 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:11.003+0000 7f74a05ef980 4 Zone: default (0b737f8c-eed6-42ec-8b9e-a02467818b0e) 2024-02-01T10:13:11.007 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:11.003+0000 7f74a05ef980 10 cannot find current period zonegroup using local zonegroup configuration 2024-02-01T10:13:11.007 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:11.003+0000 7f74a05ef980 20 zonegroup default 2024-02-01T10:13:11.007 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:11.003+0000 7f74a05ef980 20 rados->read ofs=0 len=0 2024-02-01T10:13:11.007 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:11.003+0000 7f74a05ef980 20 rados_obj.operate() r=-2 bl.length=0 2024-02-01T10:13:11.008 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:11.003+0000 7f74a05ef980 20 rados->read ofs=0 len=0 2024-02-01T10:13:11.008 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:11.003+0000 7f74a05ef980 20 rados_obj.operate() r=-2 bl.length=0 2024-02-01T10:13:11.008 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:11.003+0000 7f74a05ef980 20 rados->read ofs=0 len=0 2024-02-01T10:13:11.008 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:11.003+0000 7f74a05ef980 20 rados_obj.operate() r=-2 bl.length=0 2024-02-01T10:13:11.008 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:11.003+0000 7f74a05ef980 20 started sync module instance, tier type = 2024-02-01T10:13:11.008 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:11.003+0000 7f74a05ef980 20 started zone id=0b737f8c-eed6-42ec-8b9e-a02467818b0e (name=default) with tier type = 2024-02-01T10:13:11.025 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:11.023+0000 7f74a05ef980 20 add_watcher() i=0 2024-02-01T10:13:11.025 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:11.023+0000 7f74a05ef980 20 add_watcher() i=1 2024-02-01T10:13:11.025 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:11.023+0000 7f74a05ef980 20 add_watcher() i=2 2024-02-01T10:13:11.026 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:11.023+0000 7f74a05ef980 20 add_watcher() i=3 2024-02-01T10:13:11.026 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:11.023+0000 7f74a05ef980 20 add_watcher() i=4 2024-02-01T10:13:11.026 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:11.023+0000 7f74a05ef980 20 add_watcher() i=5 2024-02-01T10:13:11.026 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:11.023+0000 7f74a05ef980 20 add_watcher() i=6 2024-02-01T10:13:11.028 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:11.027+0000 7f74a05ef980 20 add_watcher() i=7 2024-02-01T10:13:11.028 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:11.027+0000 7f74a05ef980 2 all 8 watchers are set, enabling cache 2024-02-01T10:13:11.030 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:11.027+0000 7f7354fe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-02-01T10:13:11.030 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:11.027+0000 7f74a05ef980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-02-01T10:13:11.031 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:11.027+0000 7f74a05ef980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-02-01T10:13:11.031 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:11.027+0000 7f74a05ef980 5 note: GC not initialized 2024-02-01T10:13:11.031 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:11.027+0000 7f72cf7fe640 20 reqs_thread_entry: start 2024-02-01T10:13:11.031 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:11.027+0000 7f74a05ef980 20 init_complete bucket index max shards: 11 2024-02-01T10:13:11.031 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:11.027+0000 7f74a05ef980 20 Filter name: none 2024-02-01T10:13:11.031 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:11.027+0000 7f72cd7fa640 20 reqs_thread_entry: start 2024-02-01T10:13:11.039 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:11.039+0000 7f7354fe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-02-01T10:13:11.043 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:11.043+0000 7f74a05ef980 20 remove_watcher() i=0 2024-02-01T10:13:11.043 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:11.043+0000 7f74a05ef980 2 removed watcher, disabling cache 2024-02-01T10:13:11.044 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:11.043+0000 7f74a05ef980 20 remove_watcher() i=1 2024-02-01T10:13:11.049 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:11.047+0000 7f74a05ef980 20 remove_watcher() i=2 2024-02-01T10:13:11.050 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:11.051+0000 7f74a05ef980 20 remove_watcher() i=3 2024-02-01T10:13:11.053 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:11.051+0000 7f74a05ef980 20 remove_watcher() i=4 2024-02-01T10:13:11.054 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:11.051+0000 7f74a05ef980 20 remove_watcher() i=5 2024-02-01T10:13:11.056 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:11.055+0000 7f74a05ef980 20 remove_watcher() i=6 2024-02-01T10:13:11.057 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:11.055+0000 7f74a05ef980 20 remove_watcher() i=7 2024-02-01T10:13:11.070 INFO:teuthology.orchestra.run.smithi148.stdout:[] 2024-02-01T10:13:11.070 DEBUG:tasks.util.rgw: json result: [] 2024-02-01T10:13:11.070 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-02-01T10:13:11.071 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-02-01T10:13:11.071 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2024-02-01T10:13:11.149 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:13:11.149 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:13:11.178 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:11.179+0000 7fd13d8a7980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-02-01T10:13:11.178 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:11.179+0000 7fd13d8a7980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-02-01T10:13:11.181 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:11.179+0000 7fcfa1ffb640 20 reqs_thread_entry: start 2024-02-01T10:13:11.199 INFO:teuthology.orchestra.run.smithi148.stdout:{"id":"0b737f8c-eed6-42ec-8b9e-a02467818b0e","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"","notif_pool":"default.rgw.log:notif"} 2024-02-01T10:13:11.200 DEBUG:tasks.util.rgw: json result: {'id': '0b737f8c-eed6-42ec-8b9e-a02467818b0e', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2024-02-01T10:13:11.200 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['user', 'list'] 2024-02-01T10:13:11.200 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'user', 'list'] 2024-02-01T10:13:11.200 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 user list 2024-02-01T10:13:11.317 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:13:11.317 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:13:13.433 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:13.431+0000 7faafa950980 20 rados->read ofs=0 len=0 2024-02-01T10:13:13.434 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:13.435+0000 7faafa950980 20 rados_obj.operate() r=-2 bl.length=0 2024-02-01T10:13:13.434 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:13.435+0000 7faafa950980 20 realm 2024-02-01T10:13:13.434 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:13.435+0000 7faafa950980 20 rados->read ofs=0 len=0 2024-02-01T10:13:13.435 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:13.435+0000 7faafa950980 20 rados_obj.operate() r=-2 bl.length=0 2024-02-01T10:13:13.435 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:13.435+0000 7faafa950980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-02-01T10:13:13.435 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:13.435+0000 7faafa950980 20 rados->read ofs=0 len=0 2024-02-01T10:13:13.435 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:13.435+0000 7faafa950980 20 rados_obj.operate() r=-2 bl.length=0 2024-02-01T10:13:13.435 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:13.435+0000 7faafa950980 20 rados->read ofs=0 len=0 2024-02-01T10:13:13.435 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:13.435+0000 7faafa950980 20 rados_obj.operate() r=0 bl.length=46 2024-02-01T10:13:13.435 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:13.435+0000 7faafa950980 20 rados->read ofs=0 len=0 2024-02-01T10:13:13.436 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:13.435+0000 7faafa950980 20 rados_obj.operate() r=0 bl.length=874 2024-02-01T10:13:13.436 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:13.435+0000 7faafa950980 20 searching for the correct realm 2024-02-01T10:13:13.446 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:13.447+0000 7faafa950980 20 RGWRados::pool_iterate: got zone_info.d365cf07-b4f5-4e49-91b9-d941d0e743a7 2024-02-01T10:13:13.446 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:13.447+0000 7faafa950980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-02-01T10:13:13.446 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:13.447+0000 7faafa950980 20 RGWRados::pool_iterate: got default.zone. 2024-02-01T10:13:13.446 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:13.447+0000 7faafa950980 20 RGWRados::pool_iterate: got zonegroup_info.e8e31cec-345e-4b67-9a89-b1b6bc00ed71 2024-02-01T10:13:13.446 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:13.447+0000 7faafa950980 20 RGWRados::pool_iterate: got zone_names.default 2024-02-01T10:13:13.447 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:13.447+0000 7faafa950980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-02-01T10:13:13.447 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:13.447+0000 7faafa950980 20 rados->read ofs=0 len=0 2024-02-01T10:13:13.447 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:13.447+0000 7faafa950980 20 rados_obj.operate() r=-2 bl.length=0 2024-02-01T10:13:13.447 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:13.447+0000 7faafa950980 20 rados->read ofs=0 len=0 2024-02-01T10:13:13.447 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:13.447+0000 7faafa950980 20 rados_obj.operate() r=0 bl.length=46 2024-02-01T10:13:13.448 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:13.447+0000 7faafa950980 20 rados->read ofs=0 len=0 2024-02-01T10:13:13.448 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:13.447+0000 7faafa950980 20 rados_obj.operate() r=0 bl.length=398 2024-02-01T10:13:13.448 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:13.447+0000 7faafa950980 20 zone default found 2024-02-01T10:13:13.448 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:13.447+0000 7faafa950980 4 Realm: () 2024-02-01T10:13:13.448 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:13.447+0000 7faafa950980 4 ZoneGroup: default (e8e31cec-345e-4b67-9a89-b1b6bc00ed71) 2024-02-01T10:13:13.448 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:13.447+0000 7faafa950980 4 Zone: default (d365cf07-b4f5-4e49-91b9-d941d0e743a7) 2024-02-01T10:13:13.448 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:13.447+0000 7faafa950980 10 cannot find current period zonegroup using local zonegroup configuration 2024-02-01T10:13:13.448 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:13.447+0000 7faafa950980 20 zonegroup default 2024-02-01T10:13:13.448 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:13.447+0000 7faafa950980 20 rados->read ofs=0 len=0 2024-02-01T10:13:13.448 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:13.447+0000 7faafa950980 20 rados_obj.operate() r=-2 bl.length=0 2024-02-01T10:13:13.448 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:13.447+0000 7faafa950980 20 rados->read ofs=0 len=0 2024-02-01T10:13:15.424 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:15.423+0000 7faafa950980 20 rados_obj.operate() r=-2 bl.length=0 2024-02-01T10:13:15.424 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:15.423+0000 7faafa950980 20 rados->read ofs=0 len=0 2024-02-01T10:13:15.424 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:15.423+0000 7faafa950980 20 rados_obj.operate() r=-2 bl.length=0 2024-02-01T10:13:15.425 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:15.423+0000 7faafa950980 20 started sync module instance, tier type = 2024-02-01T10:13:15.425 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:15.423+0000 7faafa950980 20 started zone id=d365cf07-b4f5-4e49-91b9-d941d0e743a7 (name=default) with tier type = 2024-02-01T10:13:17.458 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:17.459+0000 7faafa950980 20 add_watcher() i=0 2024-02-01T10:13:17.459 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:17.459+0000 7faafa950980 20 add_watcher() i=1 2024-02-01T10:13:17.459 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:17.459+0000 7faafa950980 20 add_watcher() i=2 2024-02-01T10:13:17.459 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:17.459+0000 7faafa950980 20 add_watcher() i=3 2024-02-01T10:13:17.459 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:17.459+0000 7faafa950980 20 add_watcher() i=4 2024-02-01T10:13:17.459 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:17.459+0000 7faafa950980 20 add_watcher() i=5 2024-02-01T10:13:17.459 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:17.459+0000 7faafa950980 20 add_watcher() i=6 2024-02-01T10:13:17.461 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:17.459+0000 7faafa950980 20 add_watcher() i=7 2024-02-01T10:13:17.461 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:17.459+0000 7faafa950980 2 all 8 watchers are set, enabling cache 2024-02-01T10:13:17.462 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:17.459+0000 7faafa950980 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(const DoutPrefixProvider*, optional_yield):410: oid=data_loggenerations_metadata not found 2024-02-01T10:13:17.462 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:17.459+0000 7faafa950980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.0 2024-02-01T10:13:17.462 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:17.459+0000 7faafa950980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-02-01T10:13:17.462 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:17.459+0000 7faafa950980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.0 does not exist 2024-02-01T10:13:17.462 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:17.459+0000 7faafa950980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.1 2024-02-01T10:13:17.462 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:17.459+0000 7faafa950980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-02-01T10:13:17.463 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:17.463+0000 7faafa950980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.1 does not exist 2024-02-01T10:13:17.463 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:17.463+0000 7faafa950980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.2 2024-02-01T10:13:17.463 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:17.463+0000 7faafa950980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-02-01T10:13:17.463 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:17.463+0000 7faafa950980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.2 does not exist 2024-02-01T10:13:17.463 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:17.463+0000 7faafa950980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.3 2024-02-01T10:13:17.463 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:17.463+0000 7faafa950980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-02-01T10:13:17.463 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:17.463+0000 7faafa950980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.3 does not exist 2024-02-01T10:13:17.463 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:17.463+0000 7faafa950980 20 static int rgw::cls::fifo::FIFO::create(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):1210 entering 2024-02-01T10:13:17.465 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:17.463+0000 7faafa950980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-02-01T10:13:17.471 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:17.471+0000 7fa9aafdd640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-02-01T10:13:17.471 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:17.471+0000 7faafa950980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-02-01T10:13:17.471 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:17.471+0000 7faafa950980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-02-01T10:13:20.422 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.419+0000 7faafa950980 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2024-02-01T10:13:20.422 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.419+0000 7faafa950980 5 note: GC not initialized 2024-02-01T10:13:20.495 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.495+0000 7fa9a9fdb640 20 reqs_thread_entry: start 2024-02-01T10:13:20.496 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.495+0000 7faafa950980 20 init_complete bucket index max shards: 11 2024-02-01T10:13:20.496 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.495+0000 7faafa950980 20 Filter name: none 2024-02-01T10:13:20.496 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.495+0000 7fa90b7fe640 20 reqs_thread_entry: start 2024-02-01T10:13:20.524 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.523+0000 7fa9aafdd640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-02-01T10:13:20.526 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.527+0000 7faafa950980 20 remove_watcher() i=0 2024-02-01T10:13:20.527 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.527+0000 7faafa950980 2 removed watcher, disabling cache 2024-02-01T10:13:20.528 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.527+0000 7faafa950980 20 remove_watcher() i=1 2024-02-01T10:13:20.530 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.531+0000 7faafa950980 20 remove_watcher() i=2 2024-02-01T10:13:20.533 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.531+0000 7faafa950980 20 remove_watcher() i=3 2024-02-01T10:13:20.535 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.535+0000 7faafa950980 20 remove_watcher() i=4 2024-02-01T10:13:20.537 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.535+0000 7faafa950980 20 remove_watcher() i=5 2024-02-01T10:13:20.538 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.539+0000 7faafa950980 20 remove_watcher() i=6 2024-02-01T10:13:20.539 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.539+0000 7faafa950980 20 remove_watcher() i=7 2024-02-01T10:13:20.555 INFO:teuthology.orchestra.run.smithi187.stdout:[] 2024-02-01T10:13:20.556 DEBUG:tasks.util.rgw: json result: [] 2024-02-01T10:13:20.556 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-02-01T10:13:20.556 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-02-01T10:13:20.556 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2024-02-01T10:13:20.597 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:13:20.597 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:13:20.625 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.623+0000 7fcda13b1980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-02-01T10:13:20.626 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.623+0000 7fcda13b1980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-02-01T10:13:20.628 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.627+0000 7fcc5cff9640 20 reqs_thread_entry: start 2024-02-01T10:13:20.644 INFO:teuthology.orchestra.run.smithi187.stdout:{"id":"d365cf07-b4f5-4e49-91b9-d941d0e743a7","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"","notif_pool":"default.rgw.log:notif"} 2024-02-01T10:13:20.645 DEBUG:tasks.util.rgw: json result: {'id': 'd365cf07-b4f5-4e49-91b9-d941d0e743a7', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2024-02-01T10:13:20.645 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['user', 'list'] 2024-02-01T10:13:20.645 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'user', 'list'] 2024-02-01T10:13:20.645 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 user list 2024-02-01T10:13:20.713 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:13:20.713 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:13:20.746 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.743+0000 7fbb60c3f980 20 rados->read ofs=0 len=0 2024-02-01T10:13:20.747 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.747+0000 7fbb60c3f980 20 rados_obj.operate() r=-2 bl.length=0 2024-02-01T10:13:20.747 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.747+0000 7fbb60c3f980 20 realm 2024-02-01T10:13:20.747 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.747+0000 7fbb60c3f980 20 rados->read ofs=0 len=0 2024-02-01T10:13:20.747 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.747+0000 7fbb60c3f980 20 rados_obj.operate() r=-2 bl.length=0 2024-02-01T10:13:20.747 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.747+0000 7fbb60c3f980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-02-01T10:13:20.747 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.747+0000 7fbb60c3f980 20 rados->read ofs=0 len=0 2024-02-01T10:13:20.747 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.747+0000 7fbb60c3f980 20 rados_obj.operate() r=-2 bl.length=0 2024-02-01T10:13:20.747 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.747+0000 7fbb60c3f980 20 rados->read ofs=0 len=0 2024-02-01T10:13:20.748 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.747+0000 7fbb60c3f980 20 rados_obj.operate() r=0 bl.length=46 2024-02-01T10:13:20.748 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.747+0000 7fbb60c3f980 20 rados->read ofs=0 len=0 2024-02-01T10:13:20.748 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.747+0000 7fbb60c3f980 20 rados_obj.operate() r=0 bl.length=890 2024-02-01T10:13:20.748 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.747+0000 7fbb60c3f980 20 searching for the correct realm 2024-02-01T10:13:20.757 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.755+0000 7fbb60c3f980 20 RGWRados::pool_iterate: got zone_info.d365cf07-b4f5-4e49-91b9-d941d0e743a7 2024-02-01T10:13:20.758 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.755+0000 7fbb60c3f980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-02-01T10:13:20.758 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.755+0000 7fbb60c3f980 20 RGWRados::pool_iterate: got default.zone. 2024-02-01T10:13:20.758 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.755+0000 7fbb60c3f980 20 RGWRados::pool_iterate: got zonegroup_info.e8e31cec-345e-4b67-9a89-b1b6bc00ed71 2024-02-01T10:13:20.758 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.755+0000 7fbb60c3f980 20 RGWRados::pool_iterate: got zone_names.default 2024-02-01T10:13:20.758 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.755+0000 7fbb60c3f980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-02-01T10:13:20.758 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.755+0000 7fbb60c3f980 20 rados->read ofs=0 len=0 2024-02-01T10:13:20.758 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.755+0000 7fbb60c3f980 20 rados_obj.operate() r=-2 bl.length=0 2024-02-01T10:13:20.758 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.755+0000 7fbb60c3f980 20 rados->read ofs=0 len=0 2024-02-01T10:13:20.758 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.755+0000 7fbb60c3f980 20 rados_obj.operate() r=0 bl.length=46 2024-02-01T10:13:20.758 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.755+0000 7fbb60c3f980 20 rados->read ofs=0 len=0 2024-02-01T10:13:20.759 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.759+0000 7fbb60c3f980 20 rados_obj.operate() r=0 bl.length=398 2024-02-01T10:13:20.760 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.759+0000 7fbb60c3f980 20 zone default found 2024-02-01T10:13:20.760 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.759+0000 7fbb60c3f980 4 Realm: () 2024-02-01T10:13:20.760 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.759+0000 7fbb60c3f980 4 ZoneGroup: default (e8e31cec-345e-4b67-9a89-b1b6bc00ed71) 2024-02-01T10:13:20.760 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.759+0000 7fbb60c3f980 4 Zone: default (d365cf07-b4f5-4e49-91b9-d941d0e743a7) 2024-02-01T10:13:20.760 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.759+0000 7fbb60c3f980 10 cannot find current period zonegroup using local zonegroup configuration 2024-02-01T10:13:20.760 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.759+0000 7fbb60c3f980 20 zonegroup default 2024-02-01T10:13:20.760 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.759+0000 7fbb60c3f980 20 rados->read ofs=0 len=0 2024-02-01T10:13:20.760 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.759+0000 7fbb60c3f980 20 rados_obj.operate() r=-2 bl.length=0 2024-02-01T10:13:20.760 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.759+0000 7fbb60c3f980 20 rados->read ofs=0 len=0 2024-02-01T10:13:20.760 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.759+0000 7fbb60c3f980 20 rados_obj.operate() r=-2 bl.length=0 2024-02-01T10:13:20.760 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.759+0000 7fbb60c3f980 20 rados->read ofs=0 len=0 2024-02-01T10:13:20.760 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.759+0000 7fbb60c3f980 20 rados_obj.operate() r=-2 bl.length=0 2024-02-01T10:13:20.760 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.759+0000 7fbb60c3f980 20 started sync module instance, tier type = 2024-02-01T10:13:20.761 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.759+0000 7fbb60c3f980 20 started zone id=d365cf07-b4f5-4e49-91b9-d941d0e743a7 (name=default) with tier type = 2024-02-01T10:13:20.780 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.779+0000 7fbb60c3f980 20 add_watcher() i=0 2024-02-01T10:13:20.780 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.779+0000 7fbb60c3f980 20 add_watcher() i=1 2024-02-01T10:13:20.780 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.779+0000 7fbb60c3f980 20 add_watcher() i=2 2024-02-01T10:13:20.781 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.779+0000 7fbb60c3f980 20 add_watcher() i=3 2024-02-01T10:13:20.781 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.779+0000 7fbb60c3f980 20 add_watcher() i=4 2024-02-01T10:13:20.781 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.779+0000 7fbb60c3f980 20 add_watcher() i=5 2024-02-01T10:13:20.781 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.779+0000 7fbb60c3f980 20 add_watcher() i=6 2024-02-01T10:13:20.781 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.779+0000 7fbb60c3f980 20 add_watcher() i=7 2024-02-01T10:13:20.781 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.779+0000 7fbb60c3f980 2 all 8 watchers are set, enabling cache 2024-02-01T10:13:20.784 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.783+0000 7fba14fe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-02-01T10:13:20.784 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.783+0000 7fbb60c3f980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-02-01T10:13:20.784 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.783+0000 7fbb60c3f980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-02-01T10:13:20.784 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.783+0000 7fbb60c3f980 5 note: GC not initialized 2024-02-01T10:13:20.784 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.783+0000 7fb98f7fe640 20 reqs_thread_entry: start 2024-02-01T10:13:20.784 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.783+0000 7fbb60c3f980 20 init_complete bucket index max shards: 11 2024-02-01T10:13:20.784 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.783+0000 7fbb60c3f980 20 Filter name: none 2024-02-01T10:13:20.784 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.783+0000 7fb98d7fa640 20 reqs_thread_entry: start 2024-02-01T10:13:20.792 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.791+0000 7fba14fe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-02-01T10:13:20.795 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.795+0000 7fbb60c3f980 20 remove_watcher() i=0 2024-02-01T10:13:20.795 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.795+0000 7fbb60c3f980 2 removed watcher, disabling cache 2024-02-01T10:13:20.796 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.795+0000 7fbb60c3f980 20 remove_watcher() i=1 2024-02-01T10:13:20.797 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.795+0000 7fbb60c3f980 20 remove_watcher() i=2 2024-02-01T10:13:20.799 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.799+0000 7fbb60c3f980 20 remove_watcher() i=3 2024-02-01T10:13:20.799 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.799+0000 7fbb60c3f980 20 remove_watcher() i=4 2024-02-01T10:13:20.800 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.799+0000 7fbb60c3f980 20 remove_watcher() i=5 2024-02-01T10:13:20.801 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.799+0000 7fbb60c3f980 20 remove_watcher() i=6 2024-02-01T10:13:20.802 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.803+0000 7fbb60c3f980 20 remove_watcher() i=7 2024-02-01T10:13:20.815 INFO:teuthology.orchestra.run.smithi187.stdout:[] 2024-02-01T10:13:20.815 DEBUG:tasks.util.rgw: json result: [] 2024-02-01T10:13:20.815 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-02-01T10:13:20.815 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-02-01T10:13:20.815 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2024-02-01T10:13:20.894 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:13:20.895 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:13:20.922 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.919+0000 7f8f72060980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-02-01T10:13:20.922 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.919+0000 7f8f72060980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-02-01T10:13:20.924 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:20.923+0000 7f8dddffb640 20 reqs_thread_entry: start 2024-02-01T10:13:20.942 INFO:teuthology.orchestra.run.smithi187.stdout:{"id":"d365cf07-b4f5-4e49-91b9-d941d0e743a7","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"","notif_pool":"default.rgw.log:notif"} 2024-02-01T10:13:20.942 DEBUG:tasks.util.rgw: json result: {'id': 'd365cf07-b4f5-4e49-91b9-d941d0e743a7', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2024-02-01T10:13:20.942 INFO:tasks.rgw:Starting rgw... 2024-02-01T10:13:20.942 INFO:tasks.rgw:rgw c1.client.0 config is {'port': 8000} 2024-02-01T10:13:20.942 INFO:tasks.rgw:Using beast as radosgw frontend 2024-02-01T10:13:20.943 INFO:tasks.rgw.c1.client.0:Restarting daemon 2024-02-01T10:13:20.943 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2024-02-01T10:13:20.945 INFO:tasks.rgw.c1.client.0:Started 2024-02-01T10:13:20.945 INFO:tasks.rgw:rgw c1.client.1 config is {'port': 8001} 2024-02-01T10:13:20.946 INFO:tasks.rgw:Using beast as radosgw frontend 2024-02-01T10:13:20.946 INFO:tasks.rgw.c1.client.1:Restarting daemon 2024-02-01T10:13:20.946 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2024-02-01T10:13:20.948 INFO:tasks.rgw.c1.client.1:Started 2024-02-01T10:13:20.948 INFO:tasks.rgw:rgw c2.client.0 config is {'port': 8000} 2024-02-01T10:13:20.948 INFO:tasks.rgw:Using beast as radosgw frontend 2024-02-01T10:13:20.948 INFO:tasks.rgw.c2.client.0:Restarting daemon 2024-02-01T10:13:20.949 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2024-02-01T10:13:20.988 INFO:tasks.rgw.c2.client.0:Started 2024-02-01T10:13:20.988 INFO:tasks.rgw:rgw c2.client.1 config is {'port': 8001} 2024-02-01T10:13:20.988 INFO:tasks.rgw:Using beast as radosgw frontend 2024-02-01T10:13:20.988 INFO:tasks.rgw.c2.client.1:Restarting daemon 2024-02-01T10:13:20.988 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2024-02-01T10:13:20.990 INFO:tasks.rgw.c2.client.1:Started 2024-02-01T10:13:20.990 INFO:tasks.rgw:Polling c1.client.0 until it starts accepting connections on http://smithi148.front.sepia.ceph.com:8000/ 2024-02-01T10:13:20.991 DEBUG:teuthology.orchestra.run.smithi148:> curl http://smithi148.front.sepia.ceph.com:8000/ 2024-02-01T10:13:21.017 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-02-01T10:13:21.017 INFO:teuthology.orchestra.run.smithi148.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-02-01T10:13:21.017 INFO:teuthology.orchestra.run.smithi148.stderr: Dload Upload Total Spent Left Speed 2024-02-01T10:13:21.017 INFO:teuthology.orchestra.run.smithi148.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2024-02-01T10:13:21.017 INFO:teuthology.orchestra.run.smithi148.stderr:curl: (7) Failed to connect to smithi148.front.sepia.ceph.com port 8000 after 0 ms: Connection refused 2024-02-01T10:13:22.018 DEBUG:teuthology.orchestra.run.smithi148:> curl http://smithi148.front.sepia.ceph.com:8000/ 2024-02-01T10:13:22.030 INFO:teuthology.orchestra.run.smithi148.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-02-01T10:13:22.030 INFO:teuthology.orchestra.run.smithi148.stderr: Dload Upload Total Spent Left Speed 2024-02-01T10:13:22.032 INFO:teuthology.orchestra.run.smithi148.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 75940 0 --:--:-- --:--:-- --:--:-- 104k 2024-02-01T10:13:22.033 INFO:teuthology.orchestra.run.smithi148.stdout:anonymous 2024-02-01T10:13:22.034 INFO:tasks.rgw:Polling c1.client.1 until it starts accepting connections on http://smithi148.front.sepia.ceph.com:8001/ 2024-02-01T10:13:22.034 DEBUG:teuthology.orchestra.run.smithi148:> curl http://smithi148.front.sepia.ceph.com:8001/ 2024-02-01T10:13:22.083 INFO:teuthology.orchestra.run.smithi148.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-02-01T10:13:22.083 INFO:teuthology.orchestra.run.smithi148.stderr: Dload Upload Total Spent Left Speed 2024-02-01T10:13:22.086 INFO:teuthology.orchestra.run.smithi148.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 73187 0 --:--:-- --:--:-- --:--:-- 104k 2024-02-01T10:13:22.086 INFO:teuthology.orchestra.run.smithi148.stdout:anonymous 2024-02-01T10:13:22.087 INFO:tasks.rgw:Polling c2.client.0 until it starts accepting connections on http://smithi187.front.sepia.ceph.com:8000/ 2024-02-01T10:13:22.087 DEBUG:teuthology.orchestra.run.smithi187:> curl http://smithi187.front.sepia.ceph.com:8000/ 2024-02-01T10:13:22.109 INFO:teuthology.orchestra.run.smithi187.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-02-01T10:13:22.109 INFO:teuthology.orchestra.run.smithi187.stderr: Dload Upload Total Spent Left Speed 2024-02-01T10:13:22.111 INFO:teuthology.orchestra.run.smithi187.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 75272 0 --:--:-- --:--:-- --:--:-- 104k 2024-02-01T10:13:22.112 INFO:teuthology.orchestra.run.smithi187.stdout:anonymous 2024-02-01T10:13:22.112 INFO:tasks.rgw:Polling c2.client.1 until it starts accepting connections on http://smithi187.front.sepia.ceph.com:8001/ 2024-02-01T10:13:22.113 DEBUG:teuthology.orchestra.run.smithi187:> curl http://smithi187.front.sepia.ceph.com:8001/ 2024-02-01T10:13:22.163 INFO:teuthology.orchestra.run.smithi187.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-02-01T10:13:22.164 INFO:teuthology.orchestra.run.smithi187.stderr: Dload Upload Total Spent Left Speed 2024-02-01T10:13:22.166 INFO:teuthology.orchestra.run.smithi187.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 71691 0 --:--:-- --:--:-- --:--:-- 104k 2024-02-01T10:13:22.168 INFO:teuthology.orchestra.run.smithi187.stdout:anonymous 2024-02-01T10:13:22.168 INFO:teuthology.run_tasks:Running task rgw-multisite... 2024-02-01T10:13:23.211 INFO:tasks.rgw_multisite:creating realm.. 2024-02-01T10:13:23.211 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:13:23.211 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:13:23.211 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 realm create --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-02-01T10:13:23.249 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:13:23.249 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:13:23.306 INFO:teuthology.orchestra.run.smithi148.stdout:{"id":"c7bb4449-4a82-4742-a9c6-c7422f73e18f","name":"test-realm","current_period":"a5912b1d-745a-45aa-9170-9d6654159908","epoch":1} 2024-02-01T10:13:23.307 INFO:tasks.rgw_multisite:creating master zonegroup.. 2024-02-01T10:13:23.307 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:13:23.307 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:13:23.307 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup set --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --cluster c1 --debug-rgw 0 --debug-ms 0 2024-02-01T10:13:23.386 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:13:23.386 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:13:23.417 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:23.415+0000 7f5ba4411980 -1 old format 2024-02-01T10:13:23.434 INFO:teuthology.orchestra.run.smithi148.stdout:{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":true,"endpoints":["http://smithi148.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"","zones":[],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":[]}],"default_placement":"default-placement","realm_id":"c7bb4449-4a82-4742-a9c6-c7422f73e18f","sync_policy":{"groups":[]},"enabled_features":["resharding"]} 2024-02-01T10:13:23.434 INFO:tasks.rgw_multisite:creating master zone.. 2024-02-01T10:13:23.435 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', '1C56UIO2VKZWOSJZ', '--secret', '4X9hEalkm5sn0VTq71G3qlDBF1FRjy9C', '--endpoints', 'http://smithi148.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:13:23.435 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', '1C56UIO2VKZWOSJZ', '--secret', '4X9hEalkm5sn0VTq71G3qlDBF1FRjy9C', '--endpoints', 'http://smithi148.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:13:23.435 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --master --access-key 1C56UIO2VKZWOSJZ --secret 4X9hEalkm5sn0VTq71G3qlDBF1FRjy9C --endpoints http://smithi148.front.sepia.ceph.com:8000 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-02-01T10:13:23.524 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:13:23.524 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:13:23.556 INFO:teuthology.orchestra.run.smithi148.stderr:NOTICE: set zone's realm_id=c7bb4449-4a82-4742-a9c6-c7422f73e18f 2024-02-01T10:13:23.584 INFO:teuthology.orchestra.run.smithi148.stdout:{"id":"ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01","name":"test-zone1","domain_root":"test-zone1.rgw.meta:root","control_pool":"test-zone1.rgw.control","gc_pool":"test-zone1.rgw.log:gc","lc_pool":"test-zone1.rgw.log:lc","log_pool":"test-zone1.rgw.log","intent_log_pool":"test-zone1.rgw.log:intent","usage_log_pool":"test-zone1.rgw.log:usage","roles_pool":"test-zone1.rgw.meta:roles","reshard_pool":"test-zone1.rgw.log:reshard","user_keys_pool":"test-zone1.rgw.meta:users.keys","user_email_pool":"test-zone1.rgw.meta:users.email","user_swift_pool":"test-zone1.rgw.meta:users.swift","user_uid_pool":"test-zone1.rgw.meta:users.uid","otp_pool":"test-zone1.rgw.otp","system_key":{"access_key":"1C56UIO2VKZWOSJZ","secret_key":"4X9hEalkm5sn0VTq71G3qlDBF1FRjy9C"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone1.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone1.rgw.buckets.data"}},"data_extra_pool":"test-zone1.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"c7bb4449-4a82-4742-a9c6-c7422f73e18f","notif_pool":"test-zone1.rgw.log:notif"} 2024-02-01T10:13:23.585 DEBUG:teuthology.orchestra.run.smithi148:> sudo ceph osd pool create test-zone1.rgw.buckets.data 64 64 --cluster c1 2024-02-01T10:13:24.595 INFO:teuthology.orchestra.run.smithi148.stderr:pool 'test-zone1.rgw.buckets.data' created 2024-02-01T10:13:24.629 DEBUG:teuthology.orchestra.run.smithi148:> sudo ceph osd pool application enable test-zone1.rgw.buckets.data rgw --cluster c1 2024-02-01T10:13:25.604 INFO:teuthology.orchestra.run.smithi148.stderr:enabled application 'rgw' on pool 'test-zone1.rgw.buckets.data' 2024-02-01T10:13:25.646 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:13:25.646 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:13:25.646 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2024-02-01T10:13:25.694 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:13:25.694 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:13:25.739 INFO:teuthology.orchestra.run.smithi148.stdout:{"id":"ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01","name":"test-zone1","domain_root":"test-zone1.rgw.meta:root","control_pool":"test-zone1.rgw.control","gc_pool":"test-zone1.rgw.log:gc","lc_pool":"test-zone1.rgw.log:lc","log_pool":"test-zone1.rgw.log","intent_log_pool":"test-zone1.rgw.log:intent","usage_log_pool":"test-zone1.rgw.log:usage","roles_pool":"test-zone1.rgw.meta:roles","reshard_pool":"test-zone1.rgw.log:reshard","user_keys_pool":"test-zone1.rgw.meta:users.keys","user_email_pool":"test-zone1.rgw.meta:users.email","user_swift_pool":"test-zone1.rgw.meta:users.swift","user_uid_pool":"test-zone1.rgw.meta:users.uid","otp_pool":"test-zone1.rgw.otp","system_key":{"access_key":"1C56UIO2VKZWOSJZ","secret_key":"4X9hEalkm5sn0VTq71G3qlDBF1FRjy9C"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone1.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone1.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone1.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"c7bb4449-4a82-4742-a9c6-c7422f73e18f","notif_pool":"test-zone1.rgw.log:notif"} 2024-02-01T10:13:25.739 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:13:25.739 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:13:25.739 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-02-01T10:13:25.818 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:13:25.818 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:13:25.857 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:25.855+0000 7ff7b09e5980 0 period (a5912b1d-745a-45aa-9170-9d6654159908 does not have zone ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01 configured 2024-02-01T10:13:32.757 INFO:teuthology.orchestra.run.smithi148.stdout:{"id":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","epoch":1,"predecessor_uuid":"a5912b1d-745a-45aa-9170-9d6654159908","sync_status":[],"period_map":{"id":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":true,"endpoints":["http://smithi148.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01","zones":[{"id":"ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01","name":"test-zone1","endpoints":["http://smithi148.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":false,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"c7bb4449-4a82-4742-a9c6-c7422f73e18f","sync_policy":{"groups":[]},"enabled_features":["resharding"]}],"short_zone_ids":[{"key":"ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01","val":3362809119}]},"master_zonegroup":"test-zonegroup","master_zone":"ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"bucket_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"anonymous_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false}},"realm_id":"c7bb4449-4a82-4742-a9c6-c7422f73e18f","realm_epoch":2} 2024-02-01T10:13:32.758 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2024-02-01T10:13:32.758 INFO:teuthology.orchestra.run:waiting for 300 2024-02-01T10:13:32.799 INFO:tasks.rgw.c1.client.0.smithi148.stdout:2024-02-01T10:13:32.798+0000 7fc0c7136640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground (PID: 29191) UID: 0 2024-02-01T10:13:32.799 INFO:tasks.rgw.c1.client.0.smithi148.stdout:2024-02-01T10:13:32.798+0000 7fc0ca97ea40 -1 shutting down 2024-02-01T10:13:36.183 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2024-02-01T10:13:38.859 INFO:tasks.rgw.c1.client.0:Stopped 2024-02-01T10:13:38.859 INFO:tasks.rgw.c1.client.0:Restarting daemon 2024-02-01T10:13:38.860 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2024-02-01T10:13:38.863 INFO:tasks.rgw.c1.client.0:Started 2024-02-01T10:13:38.863 DEBUG:teuthology.orchestra.run.smithi148:> curl http://smithi148.front.sepia.ceph.com:8000 2024-02-01T10:13:38.874 INFO:teuthology.orchestra.run.smithi148.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-02-01T10:13:38.874 INFO:teuthology.orchestra.run.smithi148.stderr: Dload Upload Total Spent Left Speed 2024-02-01T10:13:38.875 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-02-01T10:13:38.876 INFO:teuthology.orchestra.run.smithi148.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2024-02-01T10:13:38.876 INFO:teuthology.orchestra.run.smithi148.stderr:curl: (7) Failed to connect to smithi148.front.sepia.ceph.com port 8000 after 0 ms: Connection refused 2024-02-01T10:13:39.877 DEBUG:teuthology.orchestra.run.smithi148:> curl http://smithi148.front.sepia.ceph.com:8000 2024-02-01T10:13:39.889 INFO:teuthology.orchestra.run.smithi148.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-02-01T10:13:39.889 INFO:teuthology.orchestra.run.smithi148.stderr: Dload Upload Total Spent Left Speed 2024-02-01T10:13:39.891 INFO:teuthology.orchestra.run.smithi148.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 74877 0 --:--:-- --:--:-- --:--:-- 104k 2024-02-01T10:13:39.892 INFO:teuthology.orchestra.run.smithi148.stdout:anonymous 2024-02-01T10:13:39.892 INFO:tasks.rgw_multisite:creating admin user.. 2024-02-01T10:13:39.893 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', '1C56UIO2VKZWOSJZ', '--secret', '4X9hEalkm5sn0VTq71G3qlDBF1FRjy9C', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:13:39.893 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', '1C56UIO2VKZWOSJZ', '--secret', '4X9hEalkm5sn0VTq71G3qlDBF1FRjy9C', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:13:39.893 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid realm-admin --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Realm Admin' --system --access-key 1C56UIO2VKZWOSJZ --secret 4X9hEalkm5sn0VTq71G3qlDBF1FRjy9C --cluster c1 --debug-rgw 0 --debug-ms 0 2024-02-01T10:13:39.983 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:13:39.983 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:13:40.040 INFO:teuthology.orchestra.run.smithi148.stdout:{"user_id":"realm-admin","display_name":"Realm Admin","email":"","suspended":0,"max_buckets":1000,"subusers":[],"keys":[{"user":"realm-admin","access_key":"1C56UIO2VKZWOSJZ","secret_key":"4X9hEalkm5sn0VTq71G3qlDBF1FRjy9C"}],"swift_keys":[],"caps":[],"op_mask":"read, write, delete","system":true,"default_placement":"","default_storage_class":"","placement_tags":[],"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"temp_url_keys":[],"type":"rgw","mfa_ids":[]} 2024-02-01T10:13:40.061 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2024-02-01T10:13:40.061 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi148.front.sepia.ceph.com:8000', '--access-key', '1C56UIO2VKZWOSJZ', '--secret', '4X9hEalkm5sn0VTq71G3qlDBF1FRjy9C', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:13:40.061 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi148.front.sepia.ceph.com:8000', '--access-key', '1C56UIO2VKZWOSJZ', '--secret', '4X9hEalkm5sn0VTq71G3qlDBF1FRjy9C', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:13:40.062 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi148.front.sepia.ceph.com:8000 --access-key 1C56UIO2VKZWOSJZ --secret 4X9hEalkm5sn0VTq71G3qlDBF1FRjy9C --cluster c2 --debug-rgw 0 --debug-ms 0 2024-02-01T10:13:40.107 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:13:40.107 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:13:40.176 INFO:teuthology.orchestra.run.smithi187.stdout:{"id":"c7bb4449-4a82-4742-a9c6-c7422f73e18f","name":"test-realm","current_period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","epoch":2} 2024-02-01T10:13:40.176 INFO:tasks.rgw_multisite:creating zone.. 2024-02-01T10:13:40.176 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', '1C56UIO2VKZWOSJZ', '--secret', '4X9hEalkm5sn0VTq71G3qlDBF1FRjy9C', '--endpoints', 'http://smithi187.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:13:40.176 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', '1C56UIO2VKZWOSJZ', '--secret', '4X9hEalkm5sn0VTq71G3qlDBF1FRjy9C', '--endpoints', 'http://smithi187.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:13:40.177 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --access-key 1C56UIO2VKZWOSJZ --secret 4X9hEalkm5sn0VTq71G3qlDBF1FRjy9C --endpoints http://smithi187.front.sepia.ceph.com:8000 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-02-01T10:13:40.215 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:13:40.216 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:13:40.249 INFO:teuthology.orchestra.run.smithi187.stderr:NOTICE: set zone's realm_id=c7bb4449-4a82-4742-a9c6-c7422f73e18f 2024-02-01T10:13:40.279 INFO:teuthology.orchestra.run.smithi187.stdout:{"id":"01fb6d04-f0ef-49b2-9ebc-c1b530925485","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"1C56UIO2VKZWOSJZ","secret_key":"4X9hEalkm5sn0VTq71G3qlDBF1FRjy9C"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"c7bb4449-4a82-4742-a9c6-c7422f73e18f","notif_pool":"test-zone2.rgw.log:notif"} 2024-02-01T10:13:40.280 DEBUG:teuthology.orchestra.run.smithi187:> sudo ceph osd pool create test-zone2.rgw.buckets.data 64 64 --cluster c2 2024-02-01T10:13:41.318 INFO:teuthology.orchestra.run.smithi187.stderr:pool 'test-zone2.rgw.buckets.data' created 2024-02-01T10:13:41.357 DEBUG:teuthology.orchestra.run.smithi187:> sudo ceph osd pool application enable test-zone2.rgw.buckets.data rgw --cluster c2 2024-02-01T10:13:41.893 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 has been restored 2024-02-01T10:13:42.332 INFO:teuthology.orchestra.run.smithi187.stderr:enabled application 'rgw' on pool 'test-zone2.rgw.buckets.data' 2024-02-01T10:13:42.364 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:13:42.364 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:13:42.364 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2024-02-01T10:13:42.421 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:13:42.421 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:13:42.465 INFO:teuthology.orchestra.run.smithi187.stdout:{"id":"01fb6d04-f0ef-49b2-9ebc-c1b530925485","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"1C56UIO2VKZWOSJZ","secret_key":"4X9hEalkm5sn0VTq71G3qlDBF1FRjy9C"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"c7bb4449-4a82-4742-a9c6-c7422f73e18f","notif_pool":"test-zone2.rgw.log:notif"} 2024-02-01T10:13:42.466 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:13:42.466 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:13:42.466 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2024-02-01T10:13:42.497 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:13:42.497 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:13:42.533 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:13:42.530+0000 7fe13fabf980 0 period (5dc4efbd-eccd-47a8-9471-ae4a209ec978 does not have zone 01fb6d04-f0ef-49b2-9ebc-c1b530925485 configured 2024-02-01T10:13:49.451 INFO:teuthology.orchestra.run.smithi187.stderr:Sending period to new master zone ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01 2024-02-01T10:13:49.538 INFO:teuthology.orchestra.run.smithi187.stdout:{"id":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","epoch":2,"predecessor_uuid":"a5912b1d-745a-45aa-9170-9d6654159908","sync_status":[],"period_map":{"id":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":true,"endpoints":["http://smithi148.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01","zones":[{"id":"01fb6d04-f0ef-49b2-9ebc-c1b530925485","name":"test-zone2","endpoints":["http://smithi187.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]},{"id":"ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01","name":"test-zone1","endpoints":["http://smithi148.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"c7bb4449-4a82-4742-a9c6-c7422f73e18f","sync_policy":{"groups":[]},"enabled_features":["resharding"]}],"short_zone_ids":[{"key":"01fb6d04-f0ef-49b2-9ebc-c1b530925485","val":1553472663},{"key":"ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01","val":3362809119}]},"master_zonegroup":"test-zonegroup","master_zone":"ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"bucket_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"anonymous_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false}},"realm_id":"c7bb4449-4a82-4742-a9c6-c7422f73e18f","realm_epoch":2} 2024-02-01T10:13:49.539 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2024-02-01T10:13:49.539 INFO:teuthology.orchestra.run:waiting for 300 2024-02-01T10:13:49.583 INFO:tasks.rgw.c2.client.0.smithi187.stdout:2024-02-01T10:13:49.582+0000 7fbfaa2ed640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground (PID: 28960) UID: 0 2024-02-01T10:13:49.583 INFO:tasks.rgw.c2.client.0.smithi187.stdout:2024-02-01T10:13:49.582+0000 7fbfadb35a40 -1 shutting down 2024-02-01T10:13:53.479 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2024-02-01T10:13:55.640 INFO:tasks.rgw.c2.client.0:Stopped 2024-02-01T10:13:55.641 INFO:tasks.rgw.c2.client.0:Restarting daemon 2024-02-01T10:13:55.641 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2024-02-01T10:13:55.644 INFO:tasks.rgw.c2.client.0:Started 2024-02-01T10:13:55.644 DEBUG:teuthology.orchestra.run.smithi187:> curl http://smithi187.front.sepia.ceph.com:8000 2024-02-01T10:13:55.657 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-02-01T10:13:55.657 INFO:teuthology.orchestra.run.smithi187.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-02-01T10:13:55.657 INFO:teuthology.orchestra.run.smithi187.stderr: Dload Upload Total Spent Left Speed 2024-02-01T10:13:55.657 INFO:teuthology.orchestra.run.smithi187.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2024-02-01T10:13:55.658 INFO:teuthology.orchestra.run.smithi187.stderr:curl: (7) Failed to connect to smithi187.front.sepia.ceph.com port 8000 after 0 ms: Connection refused 2024-02-01T10:13:56.658 DEBUG:teuthology.orchestra.run.smithi187:> curl http://smithi187.front.sepia.ceph.com:8000 2024-02-01T10:13:56.670 INFO:teuthology.orchestra.run.smithi187.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-02-01T10:13:56.670 INFO:teuthology.orchestra.run.smithi187.stderr: Dload Upload Total Spent Left Speed 2024-02-01T10:13:56.673 INFO:teuthology.orchestra.run.smithi187.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 64071 0 --:--:-- --:--:-- --:--:-- 71333 2024-02-01T10:13:56.674 INFO:teuthology.orchestra.run.smithi187.stdout:anonymous 2024-02-01T10:13:56.674 INFO:tasks.rgw_multisite:creating zone.. 2024-02-01T10:13:56.674 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', '1C56UIO2VKZWOSJZ', '--secret', '4X9hEalkm5sn0VTq71G3qlDBF1FRjy9C', '--endpoints', 'http://smithi148.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:13:56.674 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', '1C56UIO2VKZWOSJZ', '--secret', '4X9hEalkm5sn0VTq71G3qlDBF1FRjy9C', '--endpoints', 'http://smithi148.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:13:56.675 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key 1C56UIO2VKZWOSJZ --secret 4X9hEalkm5sn0VTq71G3qlDBF1FRjy9C --endpoints http://smithi148.front.sepia.ceph.com:8001 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-02-01T10:13:56.718 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:13:56.719 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:13:56.753 INFO:teuthology.orchestra.run.smithi148.stderr:NOTICE: set zone's realm_id=c7bb4449-4a82-4742-a9c6-c7422f73e18f 2024-02-01T10:13:56.779 INFO:teuthology.orchestra.run.smithi148.stdout:{"id":"2ea581c3-3f48-49d7-a989-68024c5b8eac","name":"test-zone3","domain_root":"test-zone3.rgw.meta:root","control_pool":"test-zone3.rgw.control","gc_pool":"test-zone3.rgw.log:gc","lc_pool":"test-zone3.rgw.log:lc","log_pool":"test-zone3.rgw.log","intent_log_pool":"test-zone3.rgw.log:intent","usage_log_pool":"test-zone3.rgw.log:usage","roles_pool":"test-zone3.rgw.meta:roles","reshard_pool":"test-zone3.rgw.log:reshard","user_keys_pool":"test-zone3.rgw.meta:users.keys","user_email_pool":"test-zone3.rgw.meta:users.email","user_swift_pool":"test-zone3.rgw.meta:users.swift","user_uid_pool":"test-zone3.rgw.meta:users.uid","otp_pool":"test-zone3.rgw.otp","system_key":{"access_key":"1C56UIO2VKZWOSJZ","secret_key":"4X9hEalkm5sn0VTq71G3qlDBF1FRjy9C"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone3.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone3.rgw.buckets.data"}},"data_extra_pool":"test-zone3.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"c7bb4449-4a82-4742-a9c6-c7422f73e18f","notif_pool":"test-zone3.rgw.log:notif"} 2024-02-01T10:13:56.780 DEBUG:teuthology.orchestra.run.smithi148:> sudo ceph osd pool create test-zone3.rgw.buckets.data 64 64 --cluster c1 2024-02-01T10:13:57.617 INFO:teuthology.orchestra.run.smithi148.stderr:pool 'test-zone3.rgw.buckets.data' created 2024-02-01T10:13:57.650 DEBUG:teuthology.orchestra.run.smithi148:> sudo ceph osd pool application enable test-zone3.rgw.buckets.data rgw --cluster c1 2024-02-01T10:13:58.632 INFO:teuthology.orchestra.run.smithi148.stderr:enabled application 'rgw' on pool 'test-zone3.rgw.buckets.data' 2024-02-01T10:13:58.677 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:13:58.678 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:13:58.678 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2024-02-01T10:13:58.728 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:13:58.728 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:13:58.772 INFO:teuthology.orchestra.run.smithi148.stdout:{"id":"2ea581c3-3f48-49d7-a989-68024c5b8eac","name":"test-zone3","domain_root":"test-zone3.rgw.meta:root","control_pool":"test-zone3.rgw.control","gc_pool":"test-zone3.rgw.log:gc","lc_pool":"test-zone3.rgw.log:lc","log_pool":"test-zone3.rgw.log","intent_log_pool":"test-zone3.rgw.log:intent","usage_log_pool":"test-zone3.rgw.log:usage","roles_pool":"test-zone3.rgw.meta:roles","reshard_pool":"test-zone3.rgw.log:reshard","user_keys_pool":"test-zone3.rgw.meta:users.keys","user_email_pool":"test-zone3.rgw.meta:users.email","user_swift_pool":"test-zone3.rgw.meta:users.swift","user_uid_pool":"test-zone3.rgw.meta:users.uid","otp_pool":"test-zone3.rgw.otp","system_key":{"access_key":"1C56UIO2VKZWOSJZ","secret_key":"4X9hEalkm5sn0VTq71G3qlDBF1FRjy9C"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone3.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone3.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone3.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"c7bb4449-4a82-4742-a9c6-c7422f73e18f","notif_pool":"test-zone3.rgw.log:notif"} 2024-02-01T10:13:58.773 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:13:58.773 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:13:58.773 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-02-01T10:13:58.804 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:13:58.804 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:13:58.840 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:13:58.841+0000 7f6c84c82980 0 period (5dc4efbd-eccd-47a8-9471-ae4a209ec978 does not have zone 2ea581c3-3f48-49d7-a989-68024c5b8eac configured 2024-02-01T10:13:59.189 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2024-02-01T10:14:05.777 INFO:teuthology.orchestra.run.smithi148.stderr:Sending period to new master zone ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01 2024-02-01T10:14:05.833 INFO:teuthology.orchestra.run.smithi148.stdout:{"id":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","epoch":3,"predecessor_uuid":"a5912b1d-745a-45aa-9170-9d6654159908","sync_status":[],"period_map":{"id":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":true,"endpoints":["http://smithi148.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01","zones":[{"id":"01fb6d04-f0ef-49b2-9ebc-c1b530925485","name":"test-zone2","endpoints":["http://smithi187.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]},{"id":"2ea581c3-3f48-49d7-a989-68024c5b8eac","name":"test-zone3","endpoints":["http://smithi148.front.sepia.ceph.com:8001"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]},{"id":"ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01","name":"test-zone1","endpoints":["http://smithi148.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"c7bb4449-4a82-4742-a9c6-c7422f73e18f","sync_policy":{"groups":[]},"enabled_features":["resharding"]}],"short_zone_ids":[{"key":"01fb6d04-f0ef-49b2-9ebc-c1b530925485","val":1553472663},{"key":"2ea581c3-3f48-49d7-a989-68024c5b8eac","val":1173357353},{"key":"ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01","val":3362809119}]},"master_zonegroup":"test-zonegroup","master_zone":"ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"bucket_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"anonymous_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false}},"realm_id":"c7bb4449-4a82-4742-a9c6-c7422f73e18f","realm_epoch":2} 2024-02-01T10:14:05.833 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2024-02-01T10:14:05.834 INFO:teuthology.orchestra.run:waiting for 300 2024-02-01T10:14:05.876 INFO:tasks.rgw.c1.client.1.smithi148.stdout:2024-02-01T10:14:05.876+0000 7fd53df69640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground (PID: 29193) UID: 0 2024-02-01T10:14:05.876 INFO:tasks.rgw.c1.client.1.smithi148.stdout:2024-02-01T10:14:05.876+0000 7fd5417b1a40 -1 shutting down 2024-02-01T10:14:10.326 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2024-02-01T10:14:11.935 INFO:tasks.rgw.c1.client.1:Stopped 2024-02-01T10:14:11.936 INFO:tasks.rgw.c1.client.1:Restarting daemon 2024-02-01T10:14:11.936 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2024-02-01T10:14:11.939 INFO:tasks.rgw.c1.client.1:Started 2024-02-01T10:14:11.939 DEBUG:teuthology.orchestra.run.smithi148:> curl http://smithi148.front.sepia.ceph.com:8001 2024-02-01T10:14:11.951 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-02-01T10:14:11.951 INFO:teuthology.orchestra.run.smithi148.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-02-01T10:14:11.952 INFO:teuthology.orchestra.run.smithi148.stderr: Dload Upload Total Spent Left Speed 2024-02-01T10:14:11.952 INFO:teuthology.orchestra.run.smithi148.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2024-02-01T10:14:11.952 INFO:teuthology.orchestra.run.smithi148.stderr:curl: (7) Failed to connect to smithi148.front.sepia.ceph.com port 8001 after 0 ms: Connection refused 2024-02-01T10:14:12.953 DEBUG:teuthology.orchestra.run.smithi148:> curl http://smithi148.front.sepia.ceph.com:8001 2024-02-01T10:14:12.964 INFO:teuthology.orchestra.run.smithi148.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-02-01T10:14:12.964 INFO:teuthology.orchestra.run.smithi148.stderr: Dload Upload Total Spent Left Speed 2024-02-01T10:14:12.967 INFO:teuthology.orchestra.run.smithi148.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 66294 0 --:--:-- --:--:-- --:--:-- 71333 2024-02-01T10:14:12.968 INFO:teuthology.orchestra.run.smithi148.stdout:anonymous 2024-02-01T10:14:12.968 INFO:tasks.rgw_multisite:rgw multisite configuration completed 2024-02-01T10:14:12.969 INFO:teuthology.run_tasks:Running task rgw-multisite-tests... 2024-02-01T10:14:12.981 INFO:tasks.rgw_multisite_tests:importing tests from /home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/../src/test/rgw/rgw_multi 2024-02-01T10:14:13.020 INFO:tasks.rgw_multisite_tests:creating test user.. 2024-02-01T10:14:13.023 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--caps', 'roles=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:14:13.023 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--caps', 'roles=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:14:13.024 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid rgw-multisite-test-user --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Multisite Test User' --gen-access-key --gen-secret --caps 'roles=*' --cluster c1 --debug-rgw 0 --debug-ms 0 2024-02-01T10:14:13.072 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:14:13.072 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:14:13.132 INFO:teuthology.orchestra.run.smithi148.stdout:{"user_id":"rgw-multisite-test-user","display_name":"Multisite Test User","email":"","suspended":0,"max_buckets":1000,"subusers":[],"keys":[{"user":"rgw-multisite-test-user","access_key":"1IMYR3NXFV2H92HDXL01","secret_key":"9CskT2Rd1uXIwlPwcJhH0OfxgH2YGGYceoskhM5e"}],"swift_keys":[],"caps":[{"type":"roles","perm":"*"}],"op_mask":"read, write, delete","default_placement":"","default_storage_class":"","placement_tags":[],"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"temp_url_keys":[],"type":"rgw","mfa_ids":[]} 2024-02-01T10:14:13.152 INFO:rgw_multi.tests:meta checkpoint 2024-02-01T10:14:13.152 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:14:13.153 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:14:13.153 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:14:13.181 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:14:13.182 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:14:13.229 INFO:teuthology.orchestra.run.smithi148.stderr:No --period given, using current period=5dc4efbd-eccd-47a8-9471-ae4a209ec978 2024-02-01T10:14:13.248 INFO:teuthology.orchestra.run.smithi148.stdout:[{"marker":"1_1706782453.132619_30.1","last_update":"2024-02-01T10:14:13.132619Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2024-02-01T10:14:13.249 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-02-01T10:14:13.249 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:14:13.249 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:14:13.249 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:14:13.291 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:14:13.291 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:14:13.348 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:14:13.346+0000 7f5c27787980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:14:13.349 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:14:13.346+0000 7f5c27787980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:14:13.371 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:14:13Z"} 2024-02-01T10:14:13.372 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1706782453.132619_30.1 target= 2024-02-01T10:14:16.033 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2024-02-01T10:14:18.372 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:14:18.373 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:14:18.373 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:14:18.416 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:14:18.416 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:14:18.473 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:14:18.469+0000 7f2c5b587980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:14:18.473 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:14:18.469+0000 7f2c5b587980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:14:18.495 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:14:18Z"} 2024-02-01T10:14:18.495 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1706782453.132619_30.1 target= 2024-02-01T10:14:23.497 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:14:23.497 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:14:23.497 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:14:23.539 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:14:23.539 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:14:23.594 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:14:23.589+0000 7f39dccaa980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:14:23.595 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:14:23.593+0000 7f39dccaa980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:14:23.617 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:14:23Z"} 2024-02-01T10:14:23.618 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1706782453.132619_30.1 target= 2024-02-01T10:14:28.618 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:14:28.619 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:14:28.619 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:14:28.662 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:14:28.662 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:14:28.716 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:14:28.713+0000 7f490aa5c980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:14:28.716 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:14:28.713+0000 7f490aa5c980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:14:28.739 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:14:28Z"} 2024-02-01T10:14:28.739 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1706782453.132619_30.1 target= 2024-02-01T10:14:33.740 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:14:33.740 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:14:33.740 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:14:33.782 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:14:33.783 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:14:33.843 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:14:33.841+0000 7fa367b47980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:14:33.844 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:14:33.841+0000 7fa367b47980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:14:33.867 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:14:33Z"} 2024-02-01T10:14:33.867 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1706782453.132619_30.1 target= 2024-02-01T10:14:38.868 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:14:38.869 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:14:38.869 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:14:38.912 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:14:38.912 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:14:38.965 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:14:38.961+0000 7fe97ad35980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:14:38.966 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:14:38.961+0000 7fe97ad35980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:14:38.988 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:14:38Z"} 2024-02-01T10:14:38.988 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1706782453.132619_30.1 target= 2024-02-01T10:14:43.989 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:14:43.989 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:14:43.990 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:14:44.031 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:14:44.032 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:14:44.088 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:14:44.085+0000 7f04ae387980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:14:44.089 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:14:44.085+0000 7f04ae387980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:14:44.111 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:14:44Z"} 2024-02-01T10:14:44.112 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1706782453.132619_30.1 target= 2024-02-01T10:14:49.113 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:14:49.113 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:14:49.114 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:14:49.155 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:14:49.156 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:14:49.209 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:14:49.205+0000 7fc803d13980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:14:49.210 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:14:49.205+0000 7fc803d13980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:14:49.232 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:14:49Z"} 2024-02-01T10:14:49.232 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1706782453.132619_30.1 target= 2024-02-01T10:14:54.233 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:14:54.233 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:14:54.234 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:14:54.277 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:14:54.277 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:14:54.334 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:14:54.329+0000 7f3d551bd980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:14:54.335 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:14:54.333+0000 7f3d551bd980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:14:54.359 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:14:54Z"} 2024-02-01T10:14:54.359 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1706782453.132619_30.1 target= 2024-02-01T10:14:59.360 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:14:59.362 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:14:59.362 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:14:59.407 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:14:59.407 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:14:59.462 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:14:59.457+0000 7f6e2ee71980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:14:59.463 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:14:59.461+0000 7f6e2ee71980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:14:59.485 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:14:59Z"} 2024-02-01T10:14:59.485 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1706782453.132619_30.1 target= 2024-02-01T10:15:04.486 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:15:04.486 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:15:04.486 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:15:04.530 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:15:04.531 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:15:04.583 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:15:04.581+0000 7f3d6b601980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:15:04.584 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:15:04.581+0000 7f3d6b601980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:15:04.606 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:15:04Z"} 2024-02-01T10:15:04.606 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1706782453.132619_30.1 target= 2024-02-01T10:15:09.607 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:15:09.607 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:15:09.607 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:15:09.651 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:15:09.651 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:15:09.705 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:15:09.701+0000 7fd376b50980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:15:09.706 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:15:09.701+0000 7fd376b50980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:15:09.728 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:15:09Z"} 2024-02-01T10:15:09.728 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1706782453.132619_30.1 target= 2024-02-01T10:15:14.729 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:15:14.729 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:15:14.730 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:15:14.771 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:15:14.772 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:15:14.825 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:15:14.821+0000 7f28df069980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:15:14.826 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:15:14.821+0000 7f28df069980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:15:14.848 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:15:14Z"} 2024-02-01T10:15:14.848 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1706782453.132619_30.1 target= 2024-02-01T10:15:19.849 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:15:19.849 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:15:19.850 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:15:19.892 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:15:19.892 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:15:19.947 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:15:19.944+0000 7f4d3aca3980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:15:19.948 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:15:19.944+0000 7f4d3aca3980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:15:19.970 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:15:19Z"} 2024-02-01T10:15:19.971 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1706782453.132619_30.1 target= 2024-02-01T10:15:24.972 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:15:24.972 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:15:24.972 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:15:25.016 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:15:25.016 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:15:25.071 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:15:25.068+0000 7fb869849980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:15:25.072 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:15:25.068+0000 7fb869849980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:15:25.094 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:15:25Z"} 2024-02-01T10:15:25.095 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1706782453.132619_30.1 target= 2024-02-01T10:15:30.095 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:15:30.096 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:15:30.096 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:15:30.138 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:15:30.138 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:15:30.192 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:15:30.188+0000 7f9384bc0980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:15:30.192 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:15:30.188+0000 7f9384bc0980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:15:30.214 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:15:30Z"} 2024-02-01T10:15:30.214 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1706782453.132619_30.1 target= 2024-02-01T10:15:35.215 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:15:35.215 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:15:35.216 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:15:35.259 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:15:35.259 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:15:35.312 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:15:35.308+0000 7f2174524980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:15:35.313 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:15:35.308+0000 7f2174524980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:15:35.335 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:15:35Z"} 2024-02-01T10:15:35.336 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1706782453.132619_30.1 target= 2024-02-01T10:15:40.337 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:15:40.337 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:15:40.337 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:15:40.381 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:15:40.381 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:15:40.437 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:15:40.436+0000 7f5be3c6a980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:15:40.438 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:15:40.436+0000 7f5be3c6a980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:15:40.463 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:15:40Z"} 2024-02-01T10:15:40.463 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1706782453.132619_30.1 target= 2024-02-01T10:15:45.464 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:15:45.464 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:15:45.464 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:15:45.508 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:15:45.508 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:15:45.561 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:15:45.560+0000 7f765cba1980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:15:45.562 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:15:45.560+0000 7f765cba1980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:15:45.584 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:15:45Z"} 2024-02-01T10:15:45.584 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1706782453.132619_30.1 target= 2024-02-01T10:15:50.585 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:15:50.585 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:15:50.585 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:15:50.628 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:15:50.628 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:15:50.682 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:15:50.680+0000 7fe4d62da980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:15:50.683 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:15:50.684+0000 7fe4d62da980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:15:50.706 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:15:50Z"} 2024-02-01T10:15:50.706 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1706782453.132619_30.1 target= 2024-02-01T10:15:55.707 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:15:55.708 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:15:55.708 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:15:55.750 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:15:55.750 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:15:55.804 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:15:55.804+0000 7f8bf60ba980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:15:55.805 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:15:55.804+0000 7f8bf60ba980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:15:55.827 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:15:55Z"} 2024-02-01T10:15:55.828 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1706782453.132619_30.1 target= 2024-02-01T10:16:00.829 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:16:00.829 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:16:00.830 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:16:00.872 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:16:00.872 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:16:00.925 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:16:00.924+0000 7f8a21480980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:16:00.926 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:16:00.928+0000 7f8a21480980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:16:00.949 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:16:00Z"} 2024-02-01T10:16:00.949 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1706782453.132619_30.1 target= 2024-02-01T10:16:05.950 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:16:05.950 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:16:05.951 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:16:05.995 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:16:05.995 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:16:06.051 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:16:06.052+0000 7f21a02f5980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:16:06.052 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:16:06.052+0000 7f21a02f5980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:16:06.075 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:16:06Z"} 2024-02-01T10:16:06.076 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1706782453.132619_30.1 target= 2024-02-01T10:16:11.077 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:16:11.077 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:16:11.077 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:16:11.119 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:16:11.119 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:16:11.174 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:16:11.176+0000 7fd500db7980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:16:11.175 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:16:11.176+0000 7fd500db7980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:16:11.198 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:16:11Z"} 2024-02-01T10:16:11.199 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-02-01T10:16:11.199 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-02-01T10:16:11.199 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:16:11.199 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:16:11.199 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:16:11.241 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:16:11.241 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:16:11.297 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:16:11.297+0000 7f4e61f7c980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:16:11.298 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:16:11.297+0000 7f4e61f7c980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:16:11.320 INFO:teuthology.orchestra.run.smithi148.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:16:11Z"} 2024-02-01T10:16:11.320 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-02-01T10:16:11.320 INFO:rgw_multi.tests:meta checkpoint 2024-02-01T10:16:11.320 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:16:11.320 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:16:11.321 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:16:11.396 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:16:11.396 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:16:11.448 INFO:teuthology.orchestra.run.smithi148.stderr:No --period given, using current period=5dc4efbd-eccd-47a8-9471-ae4a209ec978 2024-02-01T10:16:11.470 INFO:teuthology.orchestra.run.smithi148.stdout:[{"marker":"1_1706782453.132619_30.1","last_update":"2024-02-01T10:14:13.132619Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2024-02-01T10:16:11.471 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-02-01T10:16:11.471 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:16:11.471 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:16:11.471 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:16:11.513 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:16:11.513 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:16:11.567 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:16:11.568+0000 7f735246c980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:16:11.567 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:16:11.568+0000 7f735246c980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:16:11.589 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:16:11Z"} 2024-02-01T10:16:11.590 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-02-01T10:16:11.590 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-02-01T10:16:11.590 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:16:11.590 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:16:11.590 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:16:11.631 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:16:11.631 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:16:11.686 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:16:11.685+0000 7f6a4a06e980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:16:11.687 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:16:11.685+0000 7f6a4a06e980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:16:11.711 INFO:teuthology.orchestra.run.smithi148.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:16:11Z"} 2024-02-01T10:16:11.711 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-02-01T10:16:11.712 INFO:tasks.rgw_multisite_tests:running rgw multisite tests on '/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/../src/test/rgw/rgw_multi' with args=['tests.py'] 2024-02-01T10:16:12.196 DEBUG:boto:Using access key provided by client. 2024-02-01T10:16:12.196 DEBUG:boto:Using secret key provided by client. 2024-02-01T10:16:12.197 DEBUG:boto:Using access key provided by client. 2024-02-01T10:16:12.197 DEBUG:boto:Using secret key provided by client. 2024-02-01T10:16:12.197 DEBUG:boto:Using access key provided by client. 2024-02-01T10:16:12.197 DEBUG:boto:Using secret key provided by client. 2024-02-01T10:16:12.198 DEBUG:boto:Using access key provided by client. 2024-02-01T10:16:12.198 DEBUG:boto:Using secret key provided by client. 2024-02-01T10:16:12.198 DEBUG:boto:Using access key provided by client. 2024-02-01T10:16:12.198 DEBUG:boto:Using secret key provided by client. 2024-02-01T10:16:12.199 DEBUG:boto:Using access key provided by client. 2024-02-01T10:16:12.199 DEBUG:boto:Using secret key provided by client. 2024-02-01T10:16:12.199 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=vclfcn-1 2024-02-01T10:16:12.199 DEBUG:boto:path=/vclfcn-1/ 2024-02-01T10:16:12.199 DEBUG:boto:auth_path=/vclfcn-1/ 2024-02-01T10:16:12.199 DEBUG:boto:Method: PUT 2024-02-01T10:16:12.200 DEBUG:boto:Path: /vclfcn-1/ 2024-02-01T10:16:12.200 DEBUG:boto:Data: 2024-02-01T10:16:12.200 DEBUG:boto:Headers: {} 2024-02-01T10:16:12.200 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:12.200 DEBUG:boto:Port: 8000 2024-02-01T10:16:12.200 DEBUG:boto:Params: {} 2024-02-01T10:16:12.200 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T10:16:12.200 DEBUG:boto:Token: None 2024-02-01T10:16:12.200 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:16:12 GMT /vclfcn-1/ 2024-02-01T10:16:12.201 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:afI0p9f9igPRktG96xMV2jgoFqc= 2024-02-01T10:16:12.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:afI0p9f9igPRktG96xMV2jgoFqc=', 'Content-Length': '0'} 2024-02-01T10:16:14.756 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000095aedf93c0f8038-0065bb6f6c-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:16:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:14.756 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=vclfcn-2 2024-02-01T10:16:14.756 DEBUG:boto:path=/vclfcn-2/ 2024-02-01T10:16:14.756 DEBUG:boto:auth_path=/vclfcn-2/ 2024-02-01T10:16:14.757 DEBUG:boto:Method: PUT 2024-02-01T10:16:14.757 DEBUG:boto:Path: /vclfcn-2/ 2024-02-01T10:16:14.757 DEBUG:boto:Data: 2024-02-01T10:16:14.757 DEBUG:boto:Headers: {} 2024-02-01T10:16:14.757 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:16:14.757 DEBUG:boto:Port: 8000 2024-02-01T10:16:14.757 DEBUG:boto:Params: {} 2024-02-01T10:16:14.757 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T10:16:14.757 DEBUG:boto:Token: None 2024-02-01T10:16:14.758 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:16:14 GMT /vclfcn-2/ 2024-02-01T10:16:14.758 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mHPmWklRS1BxgkU671Y8+rOOkOo= 2024-02-01T10:16:14.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mHPmWklRS1BxgkU671Y8+rOOkOo=', 'Content-Length': '0'} 2024-02-01T10:16:17.436 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003f1a06d60f068187-0065bb6f6e-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:16:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:17.437 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=vclfcn-3 2024-02-01T10:16:17.437 DEBUG:boto:path=/vclfcn-3/ 2024-02-01T10:16:17.437 DEBUG:boto:auth_path=/vclfcn-3/ 2024-02-01T10:16:17.437 DEBUG:boto:Method: PUT 2024-02-01T10:16:17.437 DEBUG:boto:Path: /vclfcn-3/ 2024-02-01T10:16:17.437 DEBUG:boto:Data: 2024-02-01T10:16:17.437 DEBUG:boto:Headers: {} 2024-02-01T10:16:17.437 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:16:17.437 DEBUG:boto:Port: 8001 2024-02-01T10:16:17.437 DEBUG:boto:Params: {} 2024-02-01T10:16:17.438 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-02-01T10:16:17.438 DEBUG:boto:Token: None 2024-02-01T10:16:17.438 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:16:17 GMT /vclfcn-3/ 2024-02-01T10:16:17.438 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:MKMdCv5zSiACFEa245F67uR9/f4= 2024-02-01T10:16:17.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:MKMdCv5zSiACFEa245F67uR9/f4=', 'Content-Length': '0'} 2024-02-01T10:16:17.500 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fc713c4d6a9aebfa-0065bb6f71-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:16:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:17.501 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:16:17.501 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:16:17.501 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:16:17.541 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:16:17.541 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:16:17.600 INFO:teuthology.orchestra.run.smithi148.stderr:No --period given, using current period=5dc4efbd-eccd-47a8-9471-ae4a209ec978 2024-02-01T10:16:17.623 INFO:teuthology.orchestra.run.smithi148.stdout:[{"marker":"1_1706782453.132619_30.1","last_update":"2024-02-01T10:14:13.132619Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1706782577.476062_55.1","last_update":"2024-02-01T10:16:17.476062Z"}] 2024-02-01T10:16:17.624 DEBUG:rgw_multi.tests:master meta markers={0: '1_1706782453.132619_30.1', 1: '', 2: '', 3: '1_1706782577.476062_55.1'} 2024-02-01T10:16:17.624 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-02-01T10:16:17.624 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:16:17.624 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:16:17.624 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:16:17.666 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:16:17.666 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:16:17.720 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:16:17.720+0000 7f8d42539980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:16:17.721 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:16:17.720+0000 7f8d42539980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:16:17.743 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782574.724766_44.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:14.724766Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:16:17Z"} 2024-02-01T10:16:17.744 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782574.724766_44.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:14.724766Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:16:17Z"} 2024-02-01T10:16:17.744 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782574.724766_44.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:14.724766Z', 'realm_epoch': 2}}] 2024-02-01T10:16:17.744 DEBUG:rgw_multi.tests:log_status={0: '1_1706782453.132619_30.1', 1: '', 2: '', 3: '1_1706782577.476062_55.1'} 2024-02-01T10:16:17.744 DEBUG:rgw_multi.tests:sync_status={0: '1_1706782453.132619_30.1', 1: '', 2: '', 3: '1_1706782574.724766_44.1'} 2024-02-01T10:16:17.744 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1706782577.476062_55.1 target=1_1706782574.724766_44.1 2024-02-01T10:16:22.745 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:16:22.746 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:16:22.746 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:16:22.789 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:16:22.790 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:16:22.847 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:16:22.848+0000 7fa834c4b980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:16:22.848 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:16:22.848+0000 7fa834c4b980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:16:22.870 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782577.476062_55.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:17.476062Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:16:22Z"} 2024-02-01T10:16:22.870 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782577.476062_55.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:17.476062Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:16:22Z"} 2024-02-01T10:16:22.871 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782577.476062_55.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:17.476062Z', 'realm_epoch': 2}}] 2024-02-01T10:16:22.871 DEBUG:rgw_multi.tests:log_status={0: '1_1706782453.132619_30.1', 1: '', 2: '', 3: '1_1706782577.476062_55.1'} 2024-02-01T10:16:22.871 DEBUG:rgw_multi.tests:sync_status={0: '1_1706782453.132619_30.1', 1: '', 2: '', 3: '1_1706782577.476062_55.1'} 2024-02-01T10:16:22.871 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-02-01T10:16:22.871 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-02-01T10:16:22.871 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:16:22.871 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:16:22.872 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:16:22.914 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:16:22.914 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:16:22.968 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:16:22.969+0000 7ff0b2fb5980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:16:22.969 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:16:22.969+0000 7ff0b2fb5980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:16:22.991 INFO:teuthology.orchestra.run.smithi148.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782577.476062_55.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:17.476062Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:16:22Z"} 2024-02-01T10:16:22.992 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782577.476062_55.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:17.476062Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:16:22Z"} 2024-02-01T10:16:22.992 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782577.476062_55.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:17.476062Z', 'realm_epoch': 2}}] 2024-02-01T10:16:22.992 DEBUG:rgw_multi.tests:log_status={0: '1_1706782453.132619_30.1', 1: '', 2: '', 3: '1_1706782577.476062_55.1'} 2024-02-01T10:16:22.992 DEBUG:rgw_multi.tests:sync_status={0: '1_1706782453.132619_30.1', 1: '', 2: '', 3: '1_1706782577.476062_55.1'} 2024-02-01T10:16:22.992 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-02-01T10:16:22.992 DEBUG:boto:path=/vclfcn-1/ 2024-02-01T10:16:22.992 DEBUG:boto:auth_path=/vclfcn-1/ 2024-02-01T10:16:22.993 DEBUG:boto:Method: HEAD 2024-02-01T10:16:22.993 DEBUG:boto:Path: /vclfcn-1/ 2024-02-01T10:16:22.993 DEBUG:boto:Data: 2024-02-01T10:16:22.993 DEBUG:boto:Headers: {} 2024-02-01T10:16:22.993 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:22.993 DEBUG:boto:Port: 8000 2024-02-01T10:16:22.993 DEBUG:boto:Params: {} 2024-02-01T10:16:22.993 DEBUG:boto:Token: None 2024-02-01T10:16:22.993 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:22 GMT /vclfcn-1/ 2024-02-01T10:16:22.994 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZGI49IM9DrVW6lhGHweYJ4fAqdQ= 2024-02-01T10:16:22.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZGI49IM9DrVW6lhGHweYJ4fAqdQ=', 'Content-Length': '0'} 2024-02-01T10:16:22.996 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e786e7947583ae0c-0065bb6f76-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:16:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:22.997 DEBUG:boto:path=/vclfcn-2/ 2024-02-01T10:16:22.997 DEBUG:boto:auth_path=/vclfcn-2/ 2024-02-01T10:16:22.997 DEBUG:boto:Method: HEAD 2024-02-01T10:16:22.997 DEBUG:boto:Path: /vclfcn-2/ 2024-02-01T10:16:22.997 DEBUG:boto:Data: 2024-02-01T10:16:22.997 DEBUG:boto:Headers: {} 2024-02-01T10:16:22.997 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:22.997 DEBUG:boto:Port: 8000 2024-02-01T10:16:22.998 DEBUG:boto:Params: {} 2024-02-01T10:16:22.998 DEBUG:boto:Token: None 2024-02-01T10:16:22.998 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:22 GMT /vclfcn-2/ 2024-02-01T10:16:22.999 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:i1lDAMAx0hBUaqje2aaxBTtdxnc= 2024-02-01T10:16:22.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:i1lDAMAx0hBUaqje2aaxBTtdxnc=', 'Content-Length': '0'} 2024-02-01T10:16:23.001 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003ab174c3460c8a83-0065bb6f77-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:16:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:23.001 DEBUG:boto:path=/vclfcn-3/ 2024-02-01T10:16:23.001 DEBUG:boto:auth_path=/vclfcn-3/ 2024-02-01T10:16:23.002 DEBUG:boto:Method: HEAD 2024-02-01T10:16:23.002 DEBUG:boto:Path: /vclfcn-3/ 2024-02-01T10:16:23.002 DEBUG:boto:Data: 2024-02-01T10:16:23.002 DEBUG:boto:Headers: {} 2024-02-01T10:16:23.002 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:23.003 DEBUG:boto:Port: 8000 2024-02-01T10:16:23.003 DEBUG:boto:Params: {} 2024-02-01T10:16:23.003 DEBUG:boto:Token: None 2024-02-01T10:16:23.003 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:23 GMT /vclfcn-3/ 2024-02-01T10:16:23.004 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:iDV0MHFWLagxzIFMl+TSiMkio8E= 2024-02-01T10:16:23.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iDV0MHFWLagxzIFMl+TSiMkio8E=', 'Content-Length': '0'} 2024-02-01T10:16:23.006 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a8ab1f0bd25c9605-0065bb6f77-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:16:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:23.006 DEBUG:boto:path=/vclfcn-1/ 2024-02-01T10:16:23.006 DEBUG:boto:auth_path=/vclfcn-1/ 2024-02-01T10:16:23.006 DEBUG:boto:Method: HEAD 2024-02-01T10:16:23.007 DEBUG:boto:Path: /vclfcn-1/ 2024-02-01T10:16:23.007 DEBUG:boto:Data: 2024-02-01T10:16:23.007 DEBUG:boto:Headers: {} 2024-02-01T10:16:23.007 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:16:23.007 DEBUG:boto:Port: 8000 2024-02-01T10:16:23.008 DEBUG:boto:Params: {} 2024-02-01T10:16:23.008 DEBUG:boto:Token: None 2024-02-01T10:16:23.008 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:23 GMT /vclfcn-1/ 2024-02-01T10:16:23.008 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wfHziKNgv0UM9Xh47gP0MobSyac= 2024-02-01T10:16:23.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wfHziKNgv0UM9Xh47gP0MobSyac=', 'Content-Length': '0'} 2024-02-01T10:16:23.012 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001d287833a335ba97-0065bb6f77-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:16:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:23.012 DEBUG:boto:path=/vclfcn-2/ 2024-02-01T10:16:23.013 DEBUG:boto:auth_path=/vclfcn-2/ 2024-02-01T10:16:23.013 DEBUG:boto:Method: HEAD 2024-02-01T10:16:23.013 DEBUG:boto:Path: /vclfcn-2/ 2024-02-01T10:16:23.013 DEBUG:boto:Data: 2024-02-01T10:16:23.013 DEBUG:boto:Headers: {} 2024-02-01T10:16:23.013 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:16:23.013 DEBUG:boto:Port: 8000 2024-02-01T10:16:23.013 DEBUG:boto:Params: {} 2024-02-01T10:16:23.013 DEBUG:boto:Token: None 2024-02-01T10:16:23.014 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:23 GMT /vclfcn-2/ 2024-02-01T10:16:23.014 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Wm+ZrntzTmrRNIXVmHMFMl/HP7Y= 2024-02-01T10:16:23.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Wm+ZrntzTmrRNIXVmHMFMl/HP7Y=', 'Content-Length': '0'} 2024-02-01T10:16:23.018 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002908d842e33979c1-0065bb6f77-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:16:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:23.018 DEBUG:boto:path=/vclfcn-3/ 2024-02-01T10:16:23.018 DEBUG:boto:auth_path=/vclfcn-3/ 2024-02-01T10:16:23.018 DEBUG:boto:Method: HEAD 2024-02-01T10:16:23.018 DEBUG:boto:Path: /vclfcn-3/ 2024-02-01T10:16:23.018 DEBUG:boto:Data: 2024-02-01T10:16:23.018 DEBUG:boto:Headers: {} 2024-02-01T10:16:23.018 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:16:23.018 DEBUG:boto:Port: 8000 2024-02-01T10:16:23.018 DEBUG:boto:Params: {} 2024-02-01T10:16:23.019 DEBUG:boto:Token: None 2024-02-01T10:16:23.019 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:23 GMT /vclfcn-3/ 2024-02-01T10:16:23.019 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:iDV0MHFWLagxzIFMl+TSiMkio8E= 2024-02-01T10:16:23.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iDV0MHFWLagxzIFMl+TSiMkio8E=', 'Content-Length': '0'} 2024-02-01T10:16:23.023 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f414316ee325e395-0065bb6f77-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:16:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:23.023 DEBUG:boto:path=/vclfcn-1/ 2024-02-01T10:16:23.024 DEBUG:boto:auth_path=/vclfcn-1/ 2024-02-01T10:16:23.024 DEBUG:boto:Method: HEAD 2024-02-01T10:16:23.024 DEBUG:boto:Path: /vclfcn-1/ 2024-02-01T10:16:23.024 DEBUG:boto:Data: 2024-02-01T10:16:23.024 DEBUG:boto:Headers: {} 2024-02-01T10:16:23.025 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:16:23.025 DEBUG:boto:Port: 8001 2024-02-01T10:16:23.025 DEBUG:boto:Params: {} 2024-02-01T10:16:23.025 DEBUG:boto:Token: None 2024-02-01T10:16:23.025 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:23 GMT /vclfcn-1/ 2024-02-01T10:16:23.026 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wfHziKNgv0UM9Xh47gP0MobSyac= 2024-02-01T10:16:23.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wfHziKNgv0UM9Xh47gP0MobSyac=', 'Content-Length': '0'} 2024-02-01T10:16:23.028 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000020cd59979035113c-0065bb6f77-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:16:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:23.028 DEBUG:boto:path=/vclfcn-2/ 2024-02-01T10:16:23.028 DEBUG:boto:auth_path=/vclfcn-2/ 2024-02-01T10:16:23.029 DEBUG:boto:Method: HEAD 2024-02-01T10:16:23.029 DEBUG:boto:Path: /vclfcn-2/ 2024-02-01T10:16:23.029 DEBUG:boto:Data: 2024-02-01T10:16:23.029 DEBUG:boto:Headers: {} 2024-02-01T10:16:23.029 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:16:23.029 DEBUG:boto:Port: 8001 2024-02-01T10:16:23.029 DEBUG:boto:Params: {} 2024-02-01T10:16:23.029 DEBUG:boto:Token: None 2024-02-01T10:16:23.029 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:23 GMT /vclfcn-2/ 2024-02-01T10:16:23.029 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Wm+ZrntzTmrRNIXVmHMFMl/HP7Y= 2024-02-01T10:16:23.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Wm+ZrntzTmrRNIXVmHMFMl/HP7Y=', 'Content-Length': '0'} 2024-02-01T10:16:23.031 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000098e9af534846d107-0065bb6f77-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:16:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:23.032 DEBUG:boto:path=/vclfcn-3/ 2024-02-01T10:16:23.032 DEBUG:boto:auth_path=/vclfcn-3/ 2024-02-01T10:16:23.032 DEBUG:boto:Method: HEAD 2024-02-01T10:16:23.032 DEBUG:boto:Path: /vclfcn-3/ 2024-02-01T10:16:23.032 DEBUG:boto:Data: 2024-02-01T10:16:23.032 DEBUG:boto:Headers: {} 2024-02-01T10:16:23.032 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:16:23.032 DEBUG:boto:Port: 8001 2024-02-01T10:16:23.032 DEBUG:boto:Params: {} 2024-02-01T10:16:23.033 DEBUG:boto:Token: None 2024-02-01T10:16:23.033 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:23 GMT /vclfcn-3/ 2024-02-01T10:16:23.033 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:iDV0MHFWLagxzIFMl+TSiMkio8E= 2024-02-01T10:16:23.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iDV0MHFWLagxzIFMl+TSiMkio8E=', 'Content-Length': '0'} 2024-02-01T10:16:23.035 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a5d067c293593cfe-0065bb6f77-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:16:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:23.035 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_create ... ok 2024-02-01T10:16:23.036 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=vclfcn-4 2024-02-01T10:16:23.036 DEBUG:boto:path=/vclfcn-4/ 2024-02-01T10:16:23.036 DEBUG:boto:auth_path=/vclfcn-4/ 2024-02-01T10:16:23.036 DEBUG:boto:Method: PUT 2024-02-01T10:16:23.036 DEBUG:boto:Path: /vclfcn-4/ 2024-02-01T10:16:23.037 DEBUG:boto:Data: 2024-02-01T10:16:23.037 DEBUG:boto:Headers: {} 2024-02-01T10:16:23.037 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:23.037 DEBUG:boto:Port: 8000 2024-02-01T10:16:23.037 DEBUG:boto:Params: {} 2024-02-01T10:16:23.037 DEBUG:boto:Token: None 2024-02-01T10:16:23.037 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:16:23 GMT /vclfcn-4/ 2024-02-01T10:16:23.037 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:emERkahfHTGRaNPKr8wNuhIEIbg= 2024-02-01T10:16:23.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:emERkahfHTGRaNPKr8wNuhIEIbg=', 'Content-Length': '0'} 2024-02-01T10:16:23.062 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d17ea22b5b9a8a73-0065bb6f77-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:16:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:23.062 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=vclfcn-5 2024-02-01T10:16:23.062 DEBUG:boto:path=/vclfcn-5/ 2024-02-01T10:16:23.062 DEBUG:boto:auth_path=/vclfcn-5/ 2024-02-01T10:16:23.062 DEBUG:boto:Method: PUT 2024-02-01T10:16:23.062 DEBUG:boto:Path: /vclfcn-5/ 2024-02-01T10:16:23.063 DEBUG:boto:Data: 2024-02-01T10:16:23.063 DEBUG:boto:Headers: {} 2024-02-01T10:16:23.063 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:16:23.063 DEBUG:boto:Port: 8000 2024-02-01T10:16:23.063 DEBUG:boto:Params: {} 2024-02-01T10:16:23.063 DEBUG:boto:Token: None 2024-02-01T10:16:23.063 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:16:23 GMT /vclfcn-5/ 2024-02-01T10:16:23.063 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sDB2w7RUKaBjM8Ffl+9S2WkSkpA= 2024-02-01T10:16:23.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sDB2w7RUKaBjM8Ffl+9S2WkSkpA=', 'Content-Length': '0'} 2024-02-01T10:16:23.113 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000002dcb375c3a1450c-0065bb6f77-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:16:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:23.113 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=vclfcn-6 2024-02-01T10:16:23.113 DEBUG:boto:path=/vclfcn-6/ 2024-02-01T10:16:23.113 DEBUG:boto:auth_path=/vclfcn-6/ 2024-02-01T10:16:23.113 DEBUG:boto:Method: PUT 2024-02-01T10:16:23.113 DEBUG:boto:Path: /vclfcn-6/ 2024-02-01T10:16:23.114 DEBUG:boto:Data: 2024-02-01T10:16:23.114 DEBUG:boto:Headers: {} 2024-02-01T10:16:23.114 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:16:23.114 DEBUG:boto:Port: 8001 2024-02-01T10:16:23.114 DEBUG:boto:Params: {} 2024-02-01T10:16:23.114 DEBUG:boto:Token: None 2024-02-01T10:16:23.114 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:16:23 GMT /vclfcn-6/ 2024-02-01T10:16:23.114 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:g+DfbhO/kVj8KwcR5MOnA0xJJJw= 2024-02-01T10:16:23.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:g+DfbhO/kVj8KwcR5MOnA0xJJJw=', 'Content-Length': '0'} 2024-02-01T10:16:23.164 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000bd0a6f54fbff7caa-0065bb6f77-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:16:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:23.165 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:16:23.165 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:16:23.165 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:16:23.205 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:16:23.206 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:16:23.259 INFO:teuthology.orchestra.run.smithi148.stderr:No --period given, using current period=5dc4efbd-eccd-47a8-9471-ae4a209ec978 2024-02-01T10:16:23.281 INFO:teuthology.orchestra.run.smithi148.stdout:[{"marker":"1_1706782453.132619_30.1","last_update":"2024-02-01T10:14:13.132619Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1706782583.148165_67.1","last_update":"2024-02-01T10:16:23.148165Z"}] 2024-02-01T10:16:23.281 DEBUG:rgw_multi.tests:master meta markers={0: '1_1706782453.132619_30.1', 1: '', 2: '', 3: '1_1706782583.148165_67.1'} 2024-02-01T10:16:23.282 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-02-01T10:16:23.282 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:16:23.282 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:16:23.282 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:16:23.327 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:16:23.327 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:16:23.386 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:16:23.384+0000 7f071c427980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:16:23.387 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:16:23.388+0000 7f071c427980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:16:23.410 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782583.148165_67.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:23.148165Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:16:23Z"} 2024-02-01T10:16:23.410 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782583.148165_67.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:23.148165Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:16:23Z"} 2024-02-01T10:16:23.410 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782583.148165_67.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:23.148165Z', 'realm_epoch': 2}}] 2024-02-01T10:16:23.410 DEBUG:rgw_multi.tests:log_status={0: '1_1706782453.132619_30.1', 1: '', 2: '', 3: '1_1706782583.148165_67.1'} 2024-02-01T10:16:23.411 DEBUG:rgw_multi.tests:sync_status={0: '1_1706782453.132619_30.1', 1: '', 2: '', 3: '1_1706782583.148165_67.1'} 2024-02-01T10:16:23.411 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-02-01T10:16:23.411 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-02-01T10:16:23.411 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:16:23.411 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:16:23.411 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:16:23.452 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:16:23.452 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:16:23.507 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:16:23.505+0000 7f1a5770a980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:16:23.508 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:16:23.509+0000 7f1a5770a980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:16:23.530 INFO:teuthology.orchestra.run.smithi148.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782583.148165_67.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:23.148165Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:16:23Z"} 2024-02-01T10:16:23.531 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782583.148165_67.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:23.148165Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:16:23Z"} 2024-02-01T10:16:23.531 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782583.148165_67.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:23.148165Z', 'realm_epoch': 2}}] 2024-02-01T10:16:23.532 DEBUG:rgw_multi.tests:log_status={0: '1_1706782453.132619_30.1', 1: '', 2: '', 3: '1_1706782583.148165_67.1'} 2024-02-01T10:16:23.532 DEBUG:rgw_multi.tests:sync_status={0: '1_1706782453.132619_30.1', 1: '', 2: '', 3: '1_1706782583.148165_67.1'} 2024-02-01T10:16:23.532 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-02-01T10:16:23.532 DEBUG:boto:path=/vclfcn-4/ 2024-02-01T10:16:23.532 DEBUG:boto:auth_path=/vclfcn-4/ 2024-02-01T10:16:23.532 DEBUG:boto:Method: HEAD 2024-02-01T10:16:23.532 DEBUG:boto:Path: /vclfcn-4/ 2024-02-01T10:16:23.532 DEBUG:boto:Data: 2024-02-01T10:16:23.532 DEBUG:boto:Headers: {} 2024-02-01T10:16:23.532 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:23.533 DEBUG:boto:Port: 8000 2024-02-01T10:16:23.533 DEBUG:boto:Params: {} 2024-02-01T10:16:23.533 DEBUG:boto:Token: None 2024-02-01T10:16:23.533 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:23 GMT /vclfcn-4/ 2024-02-01T10:16:23.533 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/d7DdcKFv6oNs/ZTGOu9egdeKFo= 2024-02-01T10:16:23.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/d7DdcKFv6oNs/ZTGOu9egdeKFo=', 'Content-Length': '0'} 2024-02-01T10:16:23.535 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f340fb3a915b5a0f-0065bb6f77-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:16:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:23.536 DEBUG:boto:path=/vclfcn-5/ 2024-02-01T10:16:23.536 DEBUG:boto:auth_path=/vclfcn-5/ 2024-02-01T10:16:23.536 DEBUG:boto:Method: HEAD 2024-02-01T10:16:23.536 DEBUG:boto:Path: /vclfcn-5/ 2024-02-01T10:16:23.536 DEBUG:boto:Data: 2024-02-01T10:16:23.536 DEBUG:boto:Headers: {} 2024-02-01T10:16:23.536 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:23.536 DEBUG:boto:Port: 8000 2024-02-01T10:16:23.536 DEBUG:boto:Params: {} 2024-02-01T10:16:23.536 DEBUG:boto:Token: None 2024-02-01T10:16:23.537 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:23 GMT /vclfcn-5/ 2024-02-01T10:16:23.537 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:CrCp2EHDmzXvsyyHqLsF+UOBWpk= 2024-02-01T10:16:23.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:CrCp2EHDmzXvsyyHqLsF+UOBWpk=', 'Content-Length': '0'} 2024-02-01T10:16:23.539 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a294c3e29f403aed-0065bb6f77-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:16:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:23.539 DEBUG:boto:path=/vclfcn-6/ 2024-02-01T10:16:23.539 DEBUG:boto:auth_path=/vclfcn-6/ 2024-02-01T10:16:23.539 DEBUG:boto:Method: HEAD 2024-02-01T10:16:23.539 DEBUG:boto:Path: /vclfcn-6/ 2024-02-01T10:16:23.539 DEBUG:boto:Data: 2024-02-01T10:16:23.539 DEBUG:boto:Headers: {} 2024-02-01T10:16:23.540 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:23.540 DEBUG:boto:Port: 8000 2024-02-01T10:16:23.540 DEBUG:boto:Params: {} 2024-02-01T10:16:23.540 DEBUG:boto:Token: None 2024-02-01T10:16:23.540 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:23 GMT /vclfcn-6/ 2024-02-01T10:16:23.540 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GkD94pG23HuJhzdK9S2OpUShwts= 2024-02-01T10:16:23.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GkD94pG23HuJhzdK9S2OpUShwts=', 'Content-Length': '0'} 2024-02-01T10:16:23.542 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004089e4b737e70ab1-0065bb6f77-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:16:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:23.542 DEBUG:boto:path=/vclfcn-4/ 2024-02-01T10:16:23.543 DEBUG:boto:auth_path=/vclfcn-4/ 2024-02-01T10:16:23.543 DEBUG:boto:Method: HEAD 2024-02-01T10:16:23.543 DEBUG:boto:Path: /vclfcn-4/ 2024-02-01T10:16:23.543 DEBUG:boto:Data: 2024-02-01T10:16:23.543 DEBUG:boto:Headers: {} 2024-02-01T10:16:23.543 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:16:23.543 DEBUG:boto:Port: 8000 2024-02-01T10:16:23.543 DEBUG:boto:Params: {} 2024-02-01T10:16:23.544 DEBUG:boto:Token: None 2024-02-01T10:16:23.544 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:23 GMT /vclfcn-4/ 2024-02-01T10:16:23.544 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/d7DdcKFv6oNs/ZTGOu9egdeKFo= 2024-02-01T10:16:23.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/d7DdcKFv6oNs/ZTGOu9egdeKFo=', 'Content-Length': '0'} 2024-02-01T10:16:23.548 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000002c3417a36f29fbf-0065bb6f77-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:16:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:23.548 DEBUG:boto:path=/vclfcn-5/ 2024-02-01T10:16:23.548 DEBUG:boto:auth_path=/vclfcn-5/ 2024-02-01T10:16:23.548 DEBUG:boto:Method: HEAD 2024-02-01T10:16:23.548 DEBUG:boto:Path: /vclfcn-5/ 2024-02-01T10:16:23.548 DEBUG:boto:Data: 2024-02-01T10:16:23.549 DEBUG:boto:Headers: {} 2024-02-01T10:16:23.549 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:16:23.549 DEBUG:boto:Port: 8000 2024-02-01T10:16:23.549 DEBUG:boto:Params: {} 2024-02-01T10:16:23.549 DEBUG:boto:Token: None 2024-02-01T10:16:23.550 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:23 GMT /vclfcn-5/ 2024-02-01T10:16:23.550 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:CrCp2EHDmzXvsyyHqLsF+UOBWpk= 2024-02-01T10:16:23.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:CrCp2EHDmzXvsyyHqLsF+UOBWpk=', 'Content-Length': '0'} 2024-02-01T10:16:23.553 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002af5076ccda15a38-0065bb6f77-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:16:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:23.554 DEBUG:boto:path=/vclfcn-6/ 2024-02-01T10:16:23.555 DEBUG:boto:auth_path=/vclfcn-6/ 2024-02-01T10:16:23.555 DEBUG:boto:Method: HEAD 2024-02-01T10:16:23.555 DEBUG:boto:Path: /vclfcn-6/ 2024-02-01T10:16:23.555 DEBUG:boto:Data: 2024-02-01T10:16:23.555 DEBUG:boto:Headers: {} 2024-02-01T10:16:23.555 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:16:23.555 DEBUG:boto:Port: 8000 2024-02-01T10:16:23.555 DEBUG:boto:Params: {} 2024-02-01T10:16:23.555 DEBUG:boto:Token: None 2024-02-01T10:16:23.555 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:23 GMT /vclfcn-6/ 2024-02-01T10:16:23.556 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GkD94pG23HuJhzdK9S2OpUShwts= 2024-02-01T10:16:23.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GkD94pG23HuJhzdK9S2OpUShwts=', 'Content-Length': '0'} 2024-02-01T10:16:23.560 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a0dda046e9171a12-0065bb6f77-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:16:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:23.560 DEBUG:boto:path=/vclfcn-4/ 2024-02-01T10:16:23.560 DEBUG:boto:auth_path=/vclfcn-4/ 2024-02-01T10:16:23.560 DEBUG:boto:Method: HEAD 2024-02-01T10:16:23.560 DEBUG:boto:Path: /vclfcn-4/ 2024-02-01T10:16:23.560 DEBUG:boto:Data: 2024-02-01T10:16:23.560 DEBUG:boto:Headers: {} 2024-02-01T10:16:23.561 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:16:23.561 DEBUG:boto:Port: 8001 2024-02-01T10:16:23.561 DEBUG:boto:Params: {} 2024-02-01T10:16:23.561 DEBUG:boto:Token: None 2024-02-01T10:16:23.561 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:23 GMT /vclfcn-4/ 2024-02-01T10:16:23.561 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/d7DdcKFv6oNs/ZTGOu9egdeKFo= 2024-02-01T10:16:23.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/d7DdcKFv6oNs/ZTGOu9egdeKFo=', 'Content-Length': '0'} 2024-02-01T10:16:23.564 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000388609f06dd5ae4b-0065bb6f77-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:16:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:23.564 DEBUG:boto:path=/vclfcn-5/ 2024-02-01T10:16:23.564 DEBUG:boto:auth_path=/vclfcn-5/ 2024-02-01T10:16:23.564 DEBUG:boto:Method: HEAD 2024-02-01T10:16:23.564 DEBUG:boto:Path: /vclfcn-5/ 2024-02-01T10:16:23.565 DEBUG:boto:Data: 2024-02-01T10:16:23.565 DEBUG:boto:Headers: {} 2024-02-01T10:16:23.565 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:16:23.565 DEBUG:boto:Port: 8001 2024-02-01T10:16:23.565 DEBUG:boto:Params: {} 2024-02-01T10:16:23.565 DEBUG:boto:Token: None 2024-02-01T10:16:23.565 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:23 GMT /vclfcn-5/ 2024-02-01T10:16:23.565 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:CrCp2EHDmzXvsyyHqLsF+UOBWpk= 2024-02-01T10:16:23.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:CrCp2EHDmzXvsyyHqLsF+UOBWpk=', 'Content-Length': '0'} 2024-02-01T10:16:23.567 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000092f4b347327b89cd-0065bb6f77-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:16:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:23.568 DEBUG:boto:path=/vclfcn-6/ 2024-02-01T10:16:23.568 DEBUG:boto:auth_path=/vclfcn-6/ 2024-02-01T10:16:23.568 DEBUG:boto:Method: HEAD 2024-02-01T10:16:23.568 DEBUG:boto:Path: /vclfcn-6/ 2024-02-01T10:16:23.568 DEBUG:boto:Data: 2024-02-01T10:16:23.568 DEBUG:boto:Headers: {} 2024-02-01T10:16:23.568 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:16:23.568 DEBUG:boto:Port: 8001 2024-02-01T10:16:23.568 DEBUG:boto:Params: {} 2024-02-01T10:16:23.568 DEBUG:boto:Token: None 2024-02-01T10:16:23.569 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:23 GMT /vclfcn-6/ 2024-02-01T10:16:23.569 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GkD94pG23HuJhzdK9S2OpUShwts= 2024-02-01T10:16:23.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GkD94pG23HuJhzdK9S2OpUShwts=', 'Content-Length': '0'} 2024-02-01T10:16:23.571 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002a0f768d92235bd7-0065bb6f77-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:16:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:23.571 DEBUG:boto:path=/vclfcn-4/ 2024-02-01T10:16:23.571 DEBUG:boto:auth_path=/vclfcn-4/ 2024-02-01T10:16:23.571 DEBUG:boto:Method: PUT 2024-02-01T10:16:23.571 DEBUG:boto:Path: /vclfcn-4/ 2024-02-01T10:16:23.572 DEBUG:boto:Data: 2024-02-01T10:16:23.572 DEBUG:boto:Headers: {} 2024-02-01T10:16:23.572 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:23.572 DEBUG:boto:Port: 8000 2024-02-01T10:16:23.572 DEBUG:boto:Params: {} 2024-02-01T10:16:23.572 DEBUG:boto:Token: None 2024-02-01T10:16:23.572 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:16:23 GMT /vclfcn-4/ 2024-02-01T10:16:23.573 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:emERkahfHTGRaNPKr8wNuhIEIbg= 2024-02-01T10:16:23.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:emERkahfHTGRaNPKr8wNuhIEIbg=', 'Content-Length': '0'} 2024-02-01T10:16:23.595 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000ed28cef06fb5af0-0065bb6f77-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:16:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:23.595 DEBUG:boto:path=/vclfcn-5/ 2024-02-01T10:16:23.595 DEBUG:boto:auth_path=/vclfcn-5/ 2024-02-01T10:16:23.595 DEBUG:boto:Method: PUT 2024-02-01T10:16:23.595 DEBUG:boto:Path: /vclfcn-5/ 2024-02-01T10:16:23.595 DEBUG:boto:Data: 2024-02-01T10:16:23.595 DEBUG:boto:Headers: {} 2024-02-01T10:16:23.596 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:23.596 DEBUG:boto:Port: 8000 2024-02-01T10:16:23.596 DEBUG:boto:Params: {} 2024-02-01T10:16:23.596 DEBUG:boto:Token: None 2024-02-01T10:16:23.596 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:16:23 GMT /vclfcn-5/ 2024-02-01T10:16:23.597 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sDB2w7RUKaBjM8Ffl+9S2WkSkpA= 2024-02-01T10:16:23.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sDB2w7RUKaBjM8Ffl+9S2WkSkpA=', 'Content-Length': '0'} 2024-02-01T10:16:23.618 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d7c12223324f768f-0065bb6f77-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:16:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:23.619 DEBUG:boto:path=/vclfcn-6/ 2024-02-01T10:16:23.619 DEBUG:boto:auth_path=/vclfcn-6/ 2024-02-01T10:16:23.619 DEBUG:boto:Method: PUT 2024-02-01T10:16:23.619 DEBUG:boto:Path: /vclfcn-6/ 2024-02-01T10:16:23.619 DEBUG:boto:Data: 2024-02-01T10:16:23.619 DEBUG:boto:Headers: {} 2024-02-01T10:16:23.619 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:23.620 DEBUG:boto:Port: 8000 2024-02-01T10:16:23.620 DEBUG:boto:Params: {} 2024-02-01T10:16:23.620 DEBUG:boto:Token: None 2024-02-01T10:16:23.620 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:16:23 GMT /vclfcn-6/ 2024-02-01T10:16:23.620 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:g+DfbhO/kVj8KwcR5MOnA0xJJJw= 2024-02-01T10:16:23.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:g+DfbhO/kVj8KwcR5MOnA0xJJJw=', 'Content-Length': '0'} 2024-02-01T10:16:23.653 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000016cd0dfa856d92f2-0065bb6f77-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:16:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:23.653 DEBUG:boto:path=/vclfcn-4/ 2024-02-01T10:16:23.653 DEBUG:boto:auth_path=/vclfcn-4/ 2024-02-01T10:16:23.654 DEBUG:boto:Method: PUT 2024-02-01T10:16:23.654 DEBUG:boto:Path: /vclfcn-4/ 2024-02-01T10:16:23.654 DEBUG:boto:Data: 2024-02-01T10:16:23.654 DEBUG:boto:Headers: {} 2024-02-01T10:16:23.654 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:16:23.654 DEBUG:boto:Port: 8000 2024-02-01T10:16:23.654 DEBUG:boto:Params: {} 2024-02-01T10:16:23.654 DEBUG:boto:Token: None 2024-02-01T10:16:23.655 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:16:23 GMT /vclfcn-4/ 2024-02-01T10:16:23.655 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:emERkahfHTGRaNPKr8wNuhIEIbg= 2024-02-01T10:16:23.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:emERkahfHTGRaNPKr8wNuhIEIbg=', 'Content-Length': '0'} 2024-02-01T10:16:23.708 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000abdddb08353925ca-0065bb6f77-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:16:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:23.709 DEBUG:boto:path=/vclfcn-5/ 2024-02-01T10:16:23.709 DEBUG:boto:auth_path=/vclfcn-5/ 2024-02-01T10:16:23.709 DEBUG:boto:Method: PUT 2024-02-01T10:16:23.709 DEBUG:boto:Path: /vclfcn-5/ 2024-02-01T10:16:23.709 DEBUG:boto:Data: 2024-02-01T10:16:23.709 DEBUG:boto:Headers: {} 2024-02-01T10:16:23.709 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:16:23.709 DEBUG:boto:Port: 8000 2024-02-01T10:16:23.709 DEBUG:boto:Params: {} 2024-02-01T10:16:23.710 DEBUG:boto:Token: None 2024-02-01T10:16:23.710 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:16:23 GMT /vclfcn-5/ 2024-02-01T10:16:23.710 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sDB2w7RUKaBjM8Ffl+9S2WkSkpA= 2024-02-01T10:16:23.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sDB2w7RUKaBjM8Ffl+9S2WkSkpA=', 'Content-Length': '0'} 2024-02-01T10:16:23.802 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b87c2bda5e584715-0065bb6f77-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:16:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:23.803 DEBUG:boto:path=/vclfcn-6/ 2024-02-01T10:16:23.803 DEBUG:boto:auth_path=/vclfcn-6/ 2024-02-01T10:16:23.803 DEBUG:boto:Method: PUT 2024-02-01T10:16:23.803 DEBUG:boto:Path: /vclfcn-6/ 2024-02-01T10:16:23.803 DEBUG:boto:Data: 2024-02-01T10:16:23.803 DEBUG:boto:Headers: {} 2024-02-01T10:16:23.803 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:16:23.803 DEBUG:boto:Port: 8000 2024-02-01T10:16:23.804 DEBUG:boto:Params: {} 2024-02-01T10:16:23.804 DEBUG:boto:Token: None 2024-02-01T10:16:23.804 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:16:23 GMT /vclfcn-6/ 2024-02-01T10:16:23.804 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:g+DfbhO/kVj8KwcR5MOnA0xJJJw= 2024-02-01T10:16:23.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:g+DfbhO/kVj8KwcR5MOnA0xJJJw=', 'Content-Length': '0'} 2024-02-01T10:16:23.910 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000000ce6806dae5a7da-0065bb6f77-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:16:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:23.910 DEBUG:boto:path=/vclfcn-4/ 2024-02-01T10:16:23.910 DEBUG:boto:auth_path=/vclfcn-4/ 2024-02-01T10:16:23.911 DEBUG:boto:Method: PUT 2024-02-01T10:16:23.911 DEBUG:boto:Path: /vclfcn-4/ 2024-02-01T10:16:23.911 DEBUG:boto:Data: 2024-02-01T10:16:23.911 DEBUG:boto:Headers: {} 2024-02-01T10:16:23.911 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:16:23.911 DEBUG:boto:Port: 8001 2024-02-01T10:16:23.911 DEBUG:boto:Params: {} 2024-02-01T10:16:23.911 DEBUG:boto:Token: None 2024-02-01T10:16:23.911 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:16:23 GMT /vclfcn-4/ 2024-02-01T10:16:23.911 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:emERkahfHTGRaNPKr8wNuhIEIbg= 2024-02-01T10:16:23.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:emERkahfHTGRaNPKr8wNuhIEIbg=', 'Content-Length': '0'} 2024-02-01T10:16:23.978 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002f5044b4c05ced80-0065bb6f77-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:16:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:23.978 DEBUG:boto:path=/vclfcn-5/ 2024-02-01T10:16:23.979 DEBUG:boto:auth_path=/vclfcn-5/ 2024-02-01T10:16:23.979 DEBUG:boto:Method: PUT 2024-02-01T10:16:23.979 DEBUG:boto:Path: /vclfcn-5/ 2024-02-01T10:16:23.979 DEBUG:boto:Data: 2024-02-01T10:16:23.979 DEBUG:boto:Headers: {} 2024-02-01T10:16:23.979 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:16:23.979 DEBUG:boto:Port: 8001 2024-02-01T10:16:23.979 DEBUG:boto:Params: {} 2024-02-01T10:16:23.979 DEBUG:boto:Token: None 2024-02-01T10:16:23.980 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:16:23 GMT /vclfcn-5/ 2024-02-01T10:16:23.980 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sDB2w7RUKaBjM8Ffl+9S2WkSkpA= 2024-02-01T10:16:23.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sDB2w7RUKaBjM8Ffl+9S2WkSkpA=', 'Content-Length': '0'} 2024-02-01T10:16:24.089 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000085f5c19a7b1e5b1c-0065bb6f77-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:16:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:24.089 DEBUG:boto:path=/vclfcn-6/ 2024-02-01T10:16:24.089 DEBUG:boto:auth_path=/vclfcn-6/ 2024-02-01T10:16:24.089 DEBUG:boto:Method: PUT 2024-02-01T10:16:24.090 DEBUG:boto:Path: /vclfcn-6/ 2024-02-01T10:16:24.090 DEBUG:boto:Data: 2024-02-01T10:16:24.090 DEBUG:boto:Headers: {} 2024-02-01T10:16:24.090 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:16:24.090 DEBUG:boto:Port: 8001 2024-02-01T10:16:24.090 DEBUG:boto:Params: {} 2024-02-01T10:16:24.090 DEBUG:boto:Token: None 2024-02-01T10:16:24.090 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:16:24 GMT /vclfcn-6/ 2024-02-01T10:16:24.090 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:N38lAryEgfHEL1SdrDCHqHRTtz4= 2024-02-01T10:16:24.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:N38lAryEgfHEL1SdrDCHqHRTtz4=', 'Content-Length': '0'} 2024-02-01T10:16:24.204 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000828a0d31e0aa763f-0065bb6f78-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:16:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:24.204 DEBUG:boto:path=/vclfcn-4/ 2024-02-01T10:16:24.204 DEBUG:boto:auth_path=/vclfcn-4/ 2024-02-01T10:16:24.205 DEBUG:boto:Method: HEAD 2024-02-01T10:16:24.205 DEBUG:boto:Path: /vclfcn-4/ 2024-02-01T10:16:24.205 DEBUG:boto:Data: 2024-02-01T10:16:24.205 DEBUG:boto:Headers: {} 2024-02-01T10:16:24.205 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:24.205 DEBUG:boto:Port: 8000 2024-02-01T10:16:24.205 DEBUG:boto:Params: {} 2024-02-01T10:16:24.205 DEBUG:boto:Token: None 2024-02-01T10:16:24.205 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:24 GMT /vclfcn-4/ 2024-02-01T10:16:24.206 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tm53Zr89hf03B/rbsodvETEy8Vc= 2024-02-01T10:16:24.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tm53Zr89hf03B/rbsodvETEy8Vc=', 'Content-Length': '0'} 2024-02-01T10:16:24.210 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a0347d762b6257a0-0065bb6f78-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:16:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:24.211 DEBUG:boto:path=/vclfcn-5/ 2024-02-01T10:16:24.211 DEBUG:boto:auth_path=/vclfcn-5/ 2024-02-01T10:16:24.211 DEBUG:boto:Method: HEAD 2024-02-01T10:16:24.211 DEBUG:boto:Path: /vclfcn-5/ 2024-02-01T10:16:24.211 DEBUG:boto:Data: 2024-02-01T10:16:24.211 DEBUG:boto:Headers: {} 2024-02-01T10:16:24.211 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:24.211 DEBUG:boto:Port: 8000 2024-02-01T10:16:24.211 DEBUG:boto:Params: {} 2024-02-01T10:16:24.211 DEBUG:boto:Token: None 2024-02-01T10:16:24.212 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:24 GMT /vclfcn-5/ 2024-02-01T10:16:24.212 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+RbpctKCPjqnh1hHUCM4/V/ZxXg= 2024-02-01T10:16:24.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+RbpctKCPjqnh1hHUCM4/V/ZxXg=', 'Content-Length': '0'} 2024-02-01T10:16:24.217 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e732e551e012bd57-0065bb6f78-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:16:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:24.217 DEBUG:boto:path=/vclfcn-6/ 2024-02-01T10:16:24.217 DEBUG:boto:auth_path=/vclfcn-6/ 2024-02-01T10:16:24.217 DEBUG:boto:Method: HEAD 2024-02-01T10:16:24.217 DEBUG:boto:Path: /vclfcn-6/ 2024-02-01T10:16:24.217 DEBUG:boto:Data: 2024-02-01T10:16:24.218 DEBUG:boto:Headers: {} 2024-02-01T10:16:24.218 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:24.218 DEBUG:boto:Port: 8000 2024-02-01T10:16:24.218 DEBUG:boto:Params: {} 2024-02-01T10:16:24.218 DEBUG:boto:Token: None 2024-02-01T10:16:24.218 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:24 GMT /vclfcn-6/ 2024-02-01T10:16:24.218 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:il/kv1pWsCKknEay5pFZ2Xwuc1o= 2024-02-01T10:16:24.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:il/kv1pWsCKknEay5pFZ2Xwuc1o=', 'Content-Length': '0'} 2024-02-01T10:16:24.222 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000511a98f18df52418-0065bb6f78-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:16:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:24.223 DEBUG:boto:path=/vclfcn-4/ 2024-02-01T10:16:24.223 DEBUG:boto:auth_path=/vclfcn-4/ 2024-02-01T10:16:24.223 DEBUG:boto:Method: HEAD 2024-02-01T10:16:24.223 DEBUG:boto:Path: /vclfcn-4/ 2024-02-01T10:16:24.223 DEBUG:boto:Data: 2024-02-01T10:16:24.223 DEBUG:boto:Headers: {} 2024-02-01T10:16:24.223 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:16:24.223 DEBUG:boto:Port: 8000 2024-02-01T10:16:24.223 DEBUG:boto:Params: {} 2024-02-01T10:16:24.223 DEBUG:boto:Token: None 2024-02-01T10:16:24.224 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:24 GMT /vclfcn-4/ 2024-02-01T10:16:24.224 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tm53Zr89hf03B/rbsodvETEy8Vc= 2024-02-01T10:16:24.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tm53Zr89hf03B/rbsodvETEy8Vc=', 'Content-Length': '0'} 2024-02-01T10:16:24.229 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007bdf3cd98aacaaac-0065bb6f78-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:16:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:24.229 DEBUG:boto:path=/vclfcn-5/ 2024-02-01T10:16:24.229 DEBUG:boto:auth_path=/vclfcn-5/ 2024-02-01T10:16:24.229 DEBUG:boto:Method: HEAD 2024-02-01T10:16:24.229 DEBUG:boto:Path: /vclfcn-5/ 2024-02-01T10:16:24.229 DEBUG:boto:Data: 2024-02-01T10:16:24.229 DEBUG:boto:Headers: {} 2024-02-01T10:16:24.230 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:16:24.230 DEBUG:boto:Port: 8000 2024-02-01T10:16:24.230 DEBUG:boto:Params: {} 2024-02-01T10:16:24.230 DEBUG:boto:Token: None 2024-02-01T10:16:24.230 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:24 GMT /vclfcn-5/ 2024-02-01T10:16:24.230 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+RbpctKCPjqnh1hHUCM4/V/ZxXg= 2024-02-01T10:16:24.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+RbpctKCPjqnh1hHUCM4/V/ZxXg=', 'Content-Length': '0'} 2024-02-01T10:16:24.235 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c180c1dad26e2fa1-0065bb6f78-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:16:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:24.235 DEBUG:boto:path=/vclfcn-6/ 2024-02-01T10:16:24.235 DEBUG:boto:auth_path=/vclfcn-6/ 2024-02-01T10:16:24.235 DEBUG:boto:Method: HEAD 2024-02-01T10:16:24.235 DEBUG:boto:Path: /vclfcn-6/ 2024-02-01T10:16:24.236 DEBUG:boto:Data: 2024-02-01T10:16:24.236 DEBUG:boto:Headers: {} 2024-02-01T10:16:24.236 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:16:24.236 DEBUG:boto:Port: 8000 2024-02-01T10:16:24.236 DEBUG:boto:Params: {} 2024-02-01T10:16:24.236 DEBUG:boto:Token: None 2024-02-01T10:16:24.236 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:24 GMT /vclfcn-6/ 2024-02-01T10:16:24.236 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:il/kv1pWsCKknEay5pFZ2Xwuc1o= 2024-02-01T10:16:24.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:il/kv1pWsCKknEay5pFZ2Xwuc1o=', 'Content-Length': '0'} 2024-02-01T10:16:24.241 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dc30d7eaa49f1d47-0065bb6f78-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:16:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:24.242 DEBUG:boto:path=/vclfcn-4/ 2024-02-01T10:16:24.242 DEBUG:boto:auth_path=/vclfcn-4/ 2024-02-01T10:16:24.242 DEBUG:boto:Method: HEAD 2024-02-01T10:16:24.242 DEBUG:boto:Path: /vclfcn-4/ 2024-02-01T10:16:24.242 DEBUG:boto:Data: 2024-02-01T10:16:24.242 DEBUG:boto:Headers: {} 2024-02-01T10:16:24.242 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:16:24.242 DEBUG:boto:Port: 8001 2024-02-01T10:16:24.242 DEBUG:boto:Params: {} 2024-02-01T10:16:24.243 DEBUG:boto:Token: None 2024-02-01T10:16:24.243 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:24 GMT /vclfcn-4/ 2024-02-01T10:16:24.243 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tm53Zr89hf03B/rbsodvETEy8Vc= 2024-02-01T10:16:24.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tm53Zr89hf03B/rbsodvETEy8Vc=', 'Content-Length': '0'} 2024-02-01T10:16:24.248 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000047937435251889a-0065bb6f78-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:16:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:24.248 DEBUG:boto:path=/vclfcn-5/ 2024-02-01T10:16:24.248 DEBUG:boto:auth_path=/vclfcn-5/ 2024-02-01T10:16:24.249 DEBUG:boto:Method: HEAD 2024-02-01T10:16:24.249 DEBUG:boto:Path: /vclfcn-5/ 2024-02-01T10:16:24.249 DEBUG:boto:Data: 2024-02-01T10:16:24.249 DEBUG:boto:Headers: {} 2024-02-01T10:16:24.249 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:16:24.249 DEBUG:boto:Port: 8001 2024-02-01T10:16:24.249 DEBUG:boto:Params: {} 2024-02-01T10:16:24.249 DEBUG:boto:Token: None 2024-02-01T10:16:24.249 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:24 GMT /vclfcn-5/ 2024-02-01T10:16:24.249 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+RbpctKCPjqnh1hHUCM4/V/ZxXg= 2024-02-01T10:16:24.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+RbpctKCPjqnh1hHUCM4/V/ZxXg=', 'Content-Length': '0'} 2024-02-01T10:16:24.254 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000070d5038d9e662b6-0065bb6f78-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:16:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:24.255 DEBUG:boto:path=/vclfcn-6/ 2024-02-01T10:16:24.255 DEBUG:boto:auth_path=/vclfcn-6/ 2024-02-01T10:16:24.255 DEBUG:boto:Method: HEAD 2024-02-01T10:16:24.255 DEBUG:boto:Path: /vclfcn-6/ 2024-02-01T10:16:24.255 DEBUG:boto:Data: 2024-02-01T10:16:24.255 DEBUG:boto:Headers: {} 2024-02-01T10:16:24.255 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:16:24.255 DEBUG:boto:Port: 8001 2024-02-01T10:16:24.255 DEBUG:boto:Params: {} 2024-02-01T10:16:24.256 DEBUG:boto:Token: None 2024-02-01T10:16:24.256 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:24 GMT /vclfcn-6/ 2024-02-01T10:16:24.256 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:il/kv1pWsCKknEay5pFZ2Xwuc1o= 2024-02-01T10:16:24.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:il/kv1pWsCKknEay5pFZ2Xwuc1o=', 'Content-Length': '0'} 2024-02-01T10:16:24.260 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002fa0e82a9b518fef-0065bb6f78-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:16:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:24.261 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:16:24.261 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:16:24.261 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:16:24.301 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:16:24.301 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:16:24.359 INFO:teuthology.orchestra.run.smithi148.stderr:No --period given, using current period=5dc4efbd-eccd-47a8-9471-ae4a209ec978 2024-02-01T10:16:24.385 INFO:teuthology.orchestra.run.smithi148.stdout:[{"marker":"1_1706782453.132619_30.1","last_update":"2024-02-01T10:14:13.132619Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1706782584.123490_103.1","last_update":"2024-02-01T10:16:24.123490Z"}] 2024-02-01T10:16:24.385 DEBUG:rgw_multi.tests:master meta markers={0: '1_1706782453.132619_30.1', 1: '', 2: '', 3: '1_1706782584.123490_103.1'} 2024-02-01T10:16:24.385 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-02-01T10:16:24.385 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:16:24.386 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:16:24.386 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:16:24.430 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:16:24.430 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:16:24.483 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:16:24.484+0000 7f869daa5980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:16:24.484 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:16:24.484+0000 7f869daa5980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:16:24.506 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782583.671310_80.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:23.671310Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:16:24Z"} 2024-02-01T10:16:24.506 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782583.671310_80.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:23.671310Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:16:24Z"} 2024-02-01T10:16:24.507 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782583.671310_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:23.671310Z', 'realm_epoch': 2}}] 2024-02-01T10:16:24.507 DEBUG:rgw_multi.tests:log_status={0: '1_1706782453.132619_30.1', 1: '', 2: '', 3: '1_1706782584.123490_103.1'} 2024-02-01T10:16:24.507 DEBUG:rgw_multi.tests:sync_status={0: '1_1706782453.132619_30.1', 1: '', 2: '', 3: '1_1706782583.671310_80.1'} 2024-02-01T10:16:24.507 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1706782584.123490_103.1 target=1_1706782583.671310_80.1 2024-02-01T10:16:29.508 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:16:29.508 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:16:29.508 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:16:29.552 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:16:29.552 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:16:29.605 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:16:29.604+0000 7f301d90b980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:16:29.605 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:16:29.604+0000 7f301d90b980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:16:29.629 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782584.123490_103.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:24.123490Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:16:29Z"} 2024-02-01T10:16:29.629 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782584.123490_103.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:24.123490Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:16:29Z"} 2024-02-01T10:16:29.629 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782584.123490_103.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:24.123490Z', 'realm_epoch': 2}}] 2024-02-01T10:16:29.629 DEBUG:rgw_multi.tests:log_status={0: '1_1706782453.132619_30.1', 1: '', 2: '', 3: '1_1706782584.123490_103.1'} 2024-02-01T10:16:29.629 DEBUG:rgw_multi.tests:sync_status={0: '1_1706782453.132619_30.1', 1: '', 2: '', 3: '1_1706782584.123490_103.1'} 2024-02-01T10:16:29.629 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-02-01T10:16:29.630 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-02-01T10:16:29.630 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:16:29.630 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:16:29.630 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:16:29.676 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:16:29.676 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:16:29.732 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:16:29.733+0000 7f78dd02c980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:16:29.733 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:16:29.733+0000 7f78dd02c980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:16:29.757 INFO:teuthology.orchestra.run.smithi148.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782584.123490_103.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:24.123490Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:16:29Z"} 2024-02-01T10:16:29.757 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782584.123490_103.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:24.123490Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:16:29Z"} 2024-02-01T10:16:29.757 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782584.123490_103.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:24.123490Z', 'realm_epoch': 2}}] 2024-02-01T10:16:29.757 DEBUG:rgw_multi.tests:log_status={0: '1_1706782453.132619_30.1', 1: '', 2: '', 3: '1_1706782584.123490_103.1'} 2024-02-01T10:16:29.757 DEBUG:rgw_multi.tests:sync_status={0: '1_1706782453.132619_30.1', 1: '', 2: '', 3: '1_1706782584.123490_103.1'} 2024-02-01T10:16:29.758 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-02-01T10:16:29.758 DEBUG:boto:path=/vclfcn-4/ 2024-02-01T10:16:29.758 DEBUG:boto:auth_path=/vclfcn-4/ 2024-02-01T10:16:29.758 DEBUG:boto:Method: HEAD 2024-02-01T10:16:29.758 DEBUG:boto:Path: /vclfcn-4/ 2024-02-01T10:16:29.758 DEBUG:boto:Data: 2024-02-01T10:16:29.758 DEBUG:boto:Headers: {} 2024-02-01T10:16:29.758 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:29.758 DEBUG:boto:Port: 8000 2024-02-01T10:16:29.758 DEBUG:boto:Params: {} 2024-02-01T10:16:29.759 DEBUG:boto:Token: None 2024-02-01T10:16:29.759 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:29 GMT /vclfcn-4/ 2024-02-01T10:16:29.759 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:m6pkHYfTt6YfbuuQ/TyeZ5exYxM= 2024-02-01T10:16:29.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:29 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:m6pkHYfTt6YfbuuQ/TyeZ5exYxM=', 'Content-Length': '0'} 2024-02-01T10:16:29.761 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008cad52d45b7f0ac4-0065bb6f7d-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:16:29 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:29.762 DEBUG:boto:path=/vclfcn-5/ 2024-02-01T10:16:29.762 DEBUG:boto:auth_path=/vclfcn-5/ 2024-02-01T10:16:29.762 DEBUG:boto:Method: HEAD 2024-02-01T10:16:29.762 DEBUG:boto:Path: /vclfcn-5/ 2024-02-01T10:16:29.762 DEBUG:boto:Data: 2024-02-01T10:16:29.762 DEBUG:boto:Headers: {} 2024-02-01T10:16:29.762 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:29.762 DEBUG:boto:Port: 8000 2024-02-01T10:16:29.762 DEBUG:boto:Params: {} 2024-02-01T10:16:29.763 DEBUG:boto:Token: None 2024-02-01T10:16:29.763 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:29 GMT /vclfcn-5/ 2024-02-01T10:16:29.763 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HyvfkVQ/gqOr9Lz4nY+CKcQCpfo= 2024-02-01T10:16:29.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:29 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HyvfkVQ/gqOr9Lz4nY+CKcQCpfo=', 'Content-Length': '0'} 2024-02-01T10:16:29.765 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000adfab5b71577eb8b-0065bb6f7d-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:16:29 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:29.765 DEBUG:boto:path=/vclfcn-6/ 2024-02-01T10:16:29.765 DEBUG:boto:auth_path=/vclfcn-6/ 2024-02-01T10:16:29.765 DEBUG:boto:Method: HEAD 2024-02-01T10:16:29.765 DEBUG:boto:Path: /vclfcn-6/ 2024-02-01T10:16:29.765 DEBUG:boto:Data: 2024-02-01T10:16:29.765 DEBUG:boto:Headers: {} 2024-02-01T10:16:29.765 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:29.765 DEBUG:boto:Port: 8000 2024-02-01T10:16:29.766 DEBUG:boto:Params: {} 2024-02-01T10:16:29.766 DEBUG:boto:Token: None 2024-02-01T10:16:29.766 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:29 GMT /vclfcn-6/ 2024-02-01T10:16:29.766 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:4WR9SDIH4EKrVNv11RFGllkSJsc= 2024-02-01T10:16:29.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:29 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:4WR9SDIH4EKrVNv11RFGllkSJsc=', 'Content-Length': '0'} 2024-02-01T10:16:29.768 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000012ef0ef44dc1c53f-0065bb6f7d-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:16:29 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:29.768 DEBUG:boto:path=/vclfcn-4/ 2024-02-01T10:16:29.768 DEBUG:boto:auth_path=/vclfcn-4/ 2024-02-01T10:16:29.768 DEBUG:boto:Method: HEAD 2024-02-01T10:16:29.768 DEBUG:boto:Path: /vclfcn-4/ 2024-02-01T10:16:29.768 DEBUG:boto:Data: 2024-02-01T10:16:29.769 DEBUG:boto:Headers: {} 2024-02-01T10:16:29.769 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:16:29.769 DEBUG:boto:Port: 8000 2024-02-01T10:16:29.769 DEBUG:boto:Params: {} 2024-02-01T10:16:29.769 DEBUG:boto:Token: None 2024-02-01T10:16:29.769 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:29 GMT /vclfcn-4/ 2024-02-01T10:16:29.769 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:m6pkHYfTt6YfbuuQ/TyeZ5exYxM= 2024-02-01T10:16:29.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:29 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:m6pkHYfTt6YfbuuQ/TyeZ5exYxM=', 'Content-Length': '0'} 2024-02-01T10:16:29.773 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ddd3f12413acd614-0065bb6f7d-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:16:29 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:29.773 DEBUG:boto:path=/vclfcn-5/ 2024-02-01T10:16:29.774 DEBUG:boto:auth_path=/vclfcn-5/ 2024-02-01T10:16:29.774 DEBUG:boto:Method: HEAD 2024-02-01T10:16:29.774 DEBUG:boto:Path: /vclfcn-5/ 2024-02-01T10:16:29.774 DEBUG:boto:Data: 2024-02-01T10:16:29.774 DEBUG:boto:Headers: {} 2024-02-01T10:16:29.774 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:16:29.774 DEBUG:boto:Port: 8000 2024-02-01T10:16:29.774 DEBUG:boto:Params: {} 2024-02-01T10:16:29.774 DEBUG:boto:Token: None 2024-02-01T10:16:29.774 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:29 GMT /vclfcn-5/ 2024-02-01T10:16:29.775 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HyvfkVQ/gqOr9Lz4nY+CKcQCpfo= 2024-02-01T10:16:29.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:29 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HyvfkVQ/gqOr9Lz4nY+CKcQCpfo=', 'Content-Length': '0'} 2024-02-01T10:16:29.778 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007b19f575de1e1c9f-0065bb6f7d-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:16:29 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:29.779 DEBUG:boto:path=/vclfcn-6/ 2024-02-01T10:16:29.779 DEBUG:boto:auth_path=/vclfcn-6/ 2024-02-01T10:16:29.779 DEBUG:boto:Method: HEAD 2024-02-01T10:16:29.779 DEBUG:boto:Path: /vclfcn-6/ 2024-02-01T10:16:29.779 DEBUG:boto:Data: 2024-02-01T10:16:29.779 DEBUG:boto:Headers: {} 2024-02-01T10:16:29.779 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:16:29.779 DEBUG:boto:Port: 8000 2024-02-01T10:16:29.779 DEBUG:boto:Params: {} 2024-02-01T10:16:29.779 DEBUG:boto:Token: None 2024-02-01T10:16:29.780 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:29 GMT /vclfcn-6/ 2024-02-01T10:16:29.780 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:4WR9SDIH4EKrVNv11RFGllkSJsc= 2024-02-01T10:16:29.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:29 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:4WR9SDIH4EKrVNv11RFGllkSJsc=', 'Content-Length': '0'} 2024-02-01T10:16:29.784 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000560f924dd85f936b-0065bb6f7d-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:16:29 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:29.784 DEBUG:boto:path=/vclfcn-4/ 2024-02-01T10:16:29.784 DEBUG:boto:auth_path=/vclfcn-4/ 2024-02-01T10:16:29.784 DEBUG:boto:Method: HEAD 2024-02-01T10:16:29.784 DEBUG:boto:Path: /vclfcn-4/ 2024-02-01T10:16:29.784 DEBUG:boto:Data: 2024-02-01T10:16:29.784 DEBUG:boto:Headers: {} 2024-02-01T10:16:29.784 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:16:29.784 DEBUG:boto:Port: 8001 2024-02-01T10:16:29.784 DEBUG:boto:Params: {} 2024-02-01T10:16:29.785 DEBUG:boto:Token: None 2024-02-01T10:16:29.785 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:29 GMT /vclfcn-4/ 2024-02-01T10:16:29.785 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:m6pkHYfTt6YfbuuQ/TyeZ5exYxM= 2024-02-01T10:16:29.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:29 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:m6pkHYfTt6YfbuuQ/TyeZ5exYxM=', 'Content-Length': '0'} 2024-02-01T10:16:29.787 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005e75273fc0f47ffc-0065bb6f7d-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:16:29 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:29.787 DEBUG:boto:path=/vclfcn-5/ 2024-02-01T10:16:29.787 DEBUG:boto:auth_path=/vclfcn-5/ 2024-02-01T10:16:29.787 DEBUG:boto:Method: HEAD 2024-02-01T10:16:29.788 DEBUG:boto:Path: /vclfcn-5/ 2024-02-01T10:16:29.788 DEBUG:boto:Data: 2024-02-01T10:16:29.788 DEBUG:boto:Headers: {} 2024-02-01T10:16:29.788 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:16:29.788 DEBUG:boto:Port: 8001 2024-02-01T10:16:29.788 DEBUG:boto:Params: {} 2024-02-01T10:16:29.788 DEBUG:boto:Token: None 2024-02-01T10:16:29.788 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:29 GMT /vclfcn-5/ 2024-02-01T10:16:29.788 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HyvfkVQ/gqOr9Lz4nY+CKcQCpfo= 2024-02-01T10:16:29.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:29 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HyvfkVQ/gqOr9Lz4nY+CKcQCpfo=', 'Content-Length': '0'} 2024-02-01T10:16:29.791 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000056cd1ec60e0b96dd-0065bb6f7d-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:16:29 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:29.791 DEBUG:boto:path=/vclfcn-6/ 2024-02-01T10:16:29.791 DEBUG:boto:auth_path=/vclfcn-6/ 2024-02-01T10:16:29.791 DEBUG:boto:Method: HEAD 2024-02-01T10:16:29.791 DEBUG:boto:Path: /vclfcn-6/ 2024-02-01T10:16:29.791 DEBUG:boto:Data: 2024-02-01T10:16:29.791 DEBUG:boto:Headers: {} 2024-02-01T10:16:29.791 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:16:29.791 DEBUG:boto:Port: 8001 2024-02-01T10:16:29.791 DEBUG:boto:Params: {} 2024-02-01T10:16:29.792 DEBUG:boto:Token: None 2024-02-01T10:16:29.792 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:29 GMT /vclfcn-6/ 2024-02-01T10:16:29.792 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:4WR9SDIH4EKrVNv11RFGllkSJsc= 2024-02-01T10:16:29.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:29 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:4WR9SDIH4EKrVNv11RFGllkSJsc=', 'Content-Length': '0'} 2024-02-01T10:16:29.794 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000a7a5bcb2457b613-0065bb6f7d-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:16:29 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:29.794 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_recreate ... ok 2024-02-01T10:16:29.795 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=vclfcn-7 2024-02-01T10:16:29.795 DEBUG:boto:path=/vclfcn-7/ 2024-02-01T10:16:29.795 DEBUG:boto:auth_path=/vclfcn-7/ 2024-02-01T10:16:29.795 DEBUG:boto:Method: PUT 2024-02-01T10:16:29.796 DEBUG:boto:Path: /vclfcn-7/ 2024-02-01T10:16:29.796 DEBUG:boto:Data: 2024-02-01T10:16:29.796 DEBUG:boto:Headers: {} 2024-02-01T10:16:29.796 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:29.796 DEBUG:boto:Port: 8000 2024-02-01T10:16:29.796 DEBUG:boto:Params: {} 2024-02-01T10:16:29.796 DEBUG:boto:Token: None 2024-02-01T10:16:29.796 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:16:29 GMT /vclfcn-7/ 2024-02-01T10:16:29.796 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:4F76NFJvnxD1aYn8wlbT/rZh32g= 2024-02-01T10:16:29.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:29 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:4F76NFJvnxD1aYn8wlbT/rZh32g=', 'Content-Length': '0'} 2024-02-01T10:16:29.812 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005d6092e16915400f-0065bb6f7d-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:16:29 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:29.812 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=vclfcn-8 2024-02-01T10:16:29.812 DEBUG:boto:path=/vclfcn-8/ 2024-02-01T10:16:29.813 DEBUG:boto:auth_path=/vclfcn-8/ 2024-02-01T10:16:29.813 DEBUG:boto:Method: PUT 2024-02-01T10:16:29.813 DEBUG:boto:Path: /vclfcn-8/ 2024-02-01T10:16:29.813 DEBUG:boto:Data: 2024-02-01T10:16:29.813 DEBUG:boto:Headers: {} 2024-02-01T10:16:29.813 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:16:29.813 DEBUG:boto:Port: 8000 2024-02-01T10:16:29.813 DEBUG:boto:Params: {} 2024-02-01T10:16:29.813 DEBUG:boto:Token: None 2024-02-01T10:16:29.814 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:16:29 GMT /vclfcn-8/ 2024-02-01T10:16:29.814 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:X7Ty7SQOnSiKcFDUdGRko7DUvY0= 2024-02-01T10:16:29.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:29 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:X7Ty7SQOnSiKcFDUdGRko7DUvY0=', 'Content-Length': '0'} 2024-02-01T10:16:29.858 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ede4df7b6288925a-0065bb6f7d-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:16:29 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:29.858 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=vclfcn-9 2024-02-01T10:16:29.859 DEBUG:boto:path=/vclfcn-9/ 2024-02-01T10:16:29.859 DEBUG:boto:auth_path=/vclfcn-9/ 2024-02-01T10:16:29.859 DEBUG:boto:Method: PUT 2024-02-01T10:16:29.859 DEBUG:boto:Path: /vclfcn-9/ 2024-02-01T10:16:29.859 DEBUG:boto:Data: 2024-02-01T10:16:29.859 DEBUG:boto:Headers: {} 2024-02-01T10:16:29.859 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:16:29.859 DEBUG:boto:Port: 8001 2024-02-01T10:16:29.859 DEBUG:boto:Params: {} 2024-02-01T10:16:29.859 DEBUG:boto:Token: None 2024-02-01T10:16:29.860 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:16:29 GMT /vclfcn-9/ 2024-02-01T10:16:29.860 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:IPpgMxSnzBrm/NBpnakH9Jj1z4U= 2024-02-01T10:16:29.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:29 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:IPpgMxSnzBrm/NBpnakH9Jj1z4U=', 'Content-Length': '0'} 2024-02-01T10:16:29.916 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005eff3a6290a8f1a8-0065bb6f7d-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:16:29 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:29.916 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:16:29.916 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:16:29.917 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:16:29.964 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:16:29.964 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:16:30.023 INFO:teuthology.orchestra.run.smithi148.stderr:No --period given, using current period=5dc4efbd-eccd-47a8-9471-ae4a209ec978 2024-02-01T10:16:30.045 INFO:teuthology.orchestra.run.smithi148.stdout:[{"marker":"1_1706782453.132619_30.1","last_update":"2024-02-01T10:14:13.132619Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1706782589.893748_115.1","last_update":"2024-02-01T10:16:29.893748Z"}] 2024-02-01T10:16:30.045 DEBUG:rgw_multi.tests:master meta markers={0: '1_1706782453.132619_30.1', 1: '', 2: '', 3: '1_1706782589.893748_115.1'} 2024-02-01T10:16:30.045 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-02-01T10:16:30.046 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:16:30.046 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:16:30.046 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:16:30.088 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:16:30.088 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:16:30.141 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:16:30.140+0000 7fb97933e980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:16:30.142 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:16:30.140+0000 7fb97933e980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:16:30.165 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782589.804979_104.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:29.804979Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:16:30Z"} 2024-02-01T10:16:30.166 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782589.804979_104.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:29.804979Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:16:30Z"} 2024-02-01T10:16:30.166 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782589.804979_104.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:29.804979Z', 'realm_epoch': 2}}] 2024-02-01T10:16:30.166 DEBUG:rgw_multi.tests:log_status={0: '1_1706782453.132619_30.1', 1: '', 2: '', 3: '1_1706782589.893748_115.1'} 2024-02-01T10:16:30.166 DEBUG:rgw_multi.tests:sync_status={0: '1_1706782453.132619_30.1', 1: '', 2: '', 3: '1_1706782589.804979_104.1'} 2024-02-01T10:16:30.166 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1706782589.893748_115.1 target=1_1706782589.804979_104.1 2024-02-01T10:16:35.167 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:16:35.167 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:16:35.168 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:16:35.210 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:16:35.210 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:16:35.267 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:16:35.268+0000 7f36785b4980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:16:35.268 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:16:35.268+0000 7f36785b4980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:16:35.290 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782589.893748_115.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:29.893748Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:16:35Z"} 2024-02-01T10:16:35.291 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782589.893748_115.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:29.893748Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:16:35Z"} 2024-02-01T10:16:35.291 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782589.893748_115.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:29.893748Z', 'realm_epoch': 2}}] 2024-02-01T10:16:35.291 DEBUG:rgw_multi.tests:log_status={0: '1_1706782453.132619_30.1', 1: '', 2: '', 3: '1_1706782589.893748_115.1'} 2024-02-01T10:16:35.291 DEBUG:rgw_multi.tests:sync_status={0: '1_1706782453.132619_30.1', 1: '', 2: '', 3: '1_1706782589.893748_115.1'} 2024-02-01T10:16:35.291 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-02-01T10:16:35.291 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-02-01T10:16:35.291 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:16:35.292 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:16:35.292 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:16:35.335 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:16:35.335 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:16:35.388 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:16:35.389+0000 7f257d58d980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:16:35.389 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:16:35.389+0000 7f257d58d980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:16:35.413 INFO:teuthology.orchestra.run.smithi148.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782589.893748_115.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:29.893748Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:16:35Z"} 2024-02-01T10:16:35.414 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782589.893748_115.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:29.893748Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:16:35Z"} 2024-02-01T10:16:35.414 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782589.893748_115.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:29.893748Z', 'realm_epoch': 2}}] 2024-02-01T10:16:35.414 DEBUG:rgw_multi.tests:log_status={0: '1_1706782453.132619_30.1', 1: '', 2: '', 3: '1_1706782589.893748_115.1'} 2024-02-01T10:16:35.414 DEBUG:rgw_multi.tests:sync_status={0: '1_1706782453.132619_30.1', 1: '', 2: '', 3: '1_1706782589.893748_115.1'} 2024-02-01T10:16:35.414 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-02-01T10:16:35.414 DEBUG:boto:path=/vclfcn-7/ 2024-02-01T10:16:35.414 DEBUG:boto:auth_path=/vclfcn-7/ 2024-02-01T10:16:35.415 DEBUG:boto:Method: HEAD 2024-02-01T10:16:35.415 DEBUG:boto:Path: /vclfcn-7/ 2024-02-01T10:16:35.415 DEBUG:boto:Data: 2024-02-01T10:16:35.415 DEBUG:boto:Headers: {} 2024-02-01T10:16:35.415 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:35.415 DEBUG:boto:Port: 8000 2024-02-01T10:16:35.415 DEBUG:boto:Params: {} 2024-02-01T10:16:35.415 DEBUG:boto:Token: None 2024-02-01T10:16:35.416 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:35 GMT /vclfcn-7/ 2024-02-01T10:16:35.416 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/8ohlhWBvcJIZAqZZDgxBCWUrc0= 2024-02-01T10:16:35.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/8ohlhWBvcJIZAqZZDgxBCWUrc0=', 'Content-Length': '0'} 2024-02-01T10:16:35.418 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c60b629392bf54c4-0065bb6f83-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:16:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:35.418 DEBUG:boto:path=/vclfcn-8/ 2024-02-01T10:16:35.418 DEBUG:boto:auth_path=/vclfcn-8/ 2024-02-01T10:16:35.419 DEBUG:boto:Method: HEAD 2024-02-01T10:16:35.419 DEBUG:boto:Path: /vclfcn-8/ 2024-02-01T10:16:35.419 DEBUG:boto:Data: 2024-02-01T10:16:35.419 DEBUG:boto:Headers: {} 2024-02-01T10:16:35.419 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:35.419 DEBUG:boto:Port: 8000 2024-02-01T10:16:35.419 DEBUG:boto:Params: {} 2024-02-01T10:16:35.419 DEBUG:boto:Token: None 2024-02-01T10:16:35.419 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:35 GMT /vclfcn-8/ 2024-02-01T10:16:35.420 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1l2hchMKp7KJD7tayAV7l2VLzQE= 2024-02-01T10:16:35.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1l2hchMKp7KJD7tayAV7l2VLzQE=', 'Content-Length': '0'} 2024-02-01T10:16:35.421 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000353e822a60ae19b2-0065bb6f83-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:16:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:35.422 DEBUG:boto:path=/vclfcn-9/ 2024-02-01T10:16:35.422 DEBUG:boto:auth_path=/vclfcn-9/ 2024-02-01T10:16:35.422 DEBUG:boto:Method: HEAD 2024-02-01T10:16:35.422 DEBUG:boto:Path: /vclfcn-9/ 2024-02-01T10:16:35.422 DEBUG:boto:Data: 2024-02-01T10:16:35.422 DEBUG:boto:Headers: {} 2024-02-01T10:16:35.422 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:35.422 DEBUG:boto:Port: 8000 2024-02-01T10:16:35.422 DEBUG:boto:Params: {} 2024-02-01T10:16:35.423 DEBUG:boto:Token: None 2024-02-01T10:16:35.423 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:35 GMT /vclfcn-9/ 2024-02-01T10:16:35.423 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:51WNlU4dMc7BxzGMIE+duNZFJkk= 2024-02-01T10:16:35.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:51WNlU4dMc7BxzGMIE+duNZFJkk=', 'Content-Length': '0'} 2024-02-01T10:16:35.425 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000349b5cc4b5625934-0065bb6f83-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:16:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:35.425 DEBUG:boto:path=/vclfcn-7/ 2024-02-01T10:16:35.425 DEBUG:boto:auth_path=/vclfcn-7/ 2024-02-01T10:16:35.425 DEBUG:boto:Method: HEAD 2024-02-01T10:16:35.425 DEBUG:boto:Path: /vclfcn-7/ 2024-02-01T10:16:35.425 DEBUG:boto:Data: 2024-02-01T10:16:35.425 DEBUG:boto:Headers: {} 2024-02-01T10:16:35.425 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:16:35.425 DEBUG:boto:Port: 8000 2024-02-01T10:16:35.426 DEBUG:boto:Params: {} 2024-02-01T10:16:35.426 DEBUG:boto:Token: None 2024-02-01T10:16:35.426 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:35 GMT /vclfcn-7/ 2024-02-01T10:16:35.426 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/8ohlhWBvcJIZAqZZDgxBCWUrc0= 2024-02-01T10:16:35.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/8ohlhWBvcJIZAqZZDgxBCWUrc0=', 'Content-Length': '0'} 2024-02-01T10:16:35.430 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000096c6fbcd6861581a-0065bb6f83-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:16:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:35.430 DEBUG:boto:path=/vclfcn-8/ 2024-02-01T10:16:35.430 DEBUG:boto:auth_path=/vclfcn-8/ 2024-02-01T10:16:35.430 DEBUG:boto:Method: HEAD 2024-02-01T10:16:35.430 DEBUG:boto:Path: /vclfcn-8/ 2024-02-01T10:16:35.430 DEBUG:boto:Data: 2024-02-01T10:16:35.430 DEBUG:boto:Headers: {} 2024-02-01T10:16:35.431 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:16:35.431 DEBUG:boto:Port: 8000 2024-02-01T10:16:35.431 DEBUG:boto:Params: {} 2024-02-01T10:16:35.431 DEBUG:boto:Token: None 2024-02-01T10:16:35.431 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:35 GMT /vclfcn-8/ 2024-02-01T10:16:35.431 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1l2hchMKp7KJD7tayAV7l2VLzQE= 2024-02-01T10:16:35.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1l2hchMKp7KJD7tayAV7l2VLzQE=', 'Content-Length': '0'} 2024-02-01T10:16:35.435 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c26e60b52f67eb3e-0065bb6f83-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:16:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:35.435 DEBUG:boto:path=/vclfcn-9/ 2024-02-01T10:16:35.435 DEBUG:boto:auth_path=/vclfcn-9/ 2024-02-01T10:16:35.435 DEBUG:boto:Method: HEAD 2024-02-01T10:16:35.435 DEBUG:boto:Path: /vclfcn-9/ 2024-02-01T10:16:35.435 DEBUG:boto:Data: 2024-02-01T10:16:35.435 DEBUG:boto:Headers: {} 2024-02-01T10:16:35.436 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:16:35.436 DEBUG:boto:Port: 8000 2024-02-01T10:16:35.436 DEBUG:boto:Params: {} 2024-02-01T10:16:35.436 DEBUG:boto:Token: None 2024-02-01T10:16:35.436 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:35 GMT /vclfcn-9/ 2024-02-01T10:16:35.436 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:51WNlU4dMc7BxzGMIE+duNZFJkk= 2024-02-01T10:16:35.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:51WNlU4dMc7BxzGMIE+duNZFJkk=', 'Content-Length': '0'} 2024-02-01T10:16:35.440 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005636305ba66a2bf9-0065bb6f83-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:16:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:35.440 DEBUG:boto:path=/vclfcn-7/ 2024-02-01T10:16:35.440 DEBUG:boto:auth_path=/vclfcn-7/ 2024-02-01T10:16:35.441 DEBUG:boto:Method: HEAD 2024-02-01T10:16:35.441 DEBUG:boto:Path: /vclfcn-7/ 2024-02-01T10:16:35.441 DEBUG:boto:Data: 2024-02-01T10:16:35.441 DEBUG:boto:Headers: {} 2024-02-01T10:16:35.441 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:16:35.441 DEBUG:boto:Port: 8001 2024-02-01T10:16:35.441 DEBUG:boto:Params: {} 2024-02-01T10:16:35.442 DEBUG:boto:Token: None 2024-02-01T10:16:35.442 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:35 GMT /vclfcn-7/ 2024-02-01T10:16:35.442 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/8ohlhWBvcJIZAqZZDgxBCWUrc0= 2024-02-01T10:16:35.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/8ohlhWBvcJIZAqZZDgxBCWUrc0=', 'Content-Length': '0'} 2024-02-01T10:16:35.444 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000262a13ea905dba9-0065bb6f83-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:16:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:35.445 DEBUG:boto:path=/vclfcn-8/ 2024-02-01T10:16:35.445 DEBUG:boto:auth_path=/vclfcn-8/ 2024-02-01T10:16:35.445 DEBUG:boto:Method: HEAD 2024-02-01T10:16:35.445 DEBUG:boto:Path: /vclfcn-8/ 2024-02-01T10:16:35.445 DEBUG:boto:Data: 2024-02-01T10:16:35.445 DEBUG:boto:Headers: {} 2024-02-01T10:16:35.445 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:16:35.445 DEBUG:boto:Port: 8001 2024-02-01T10:16:35.445 DEBUG:boto:Params: {} 2024-02-01T10:16:35.445 DEBUG:boto:Token: None 2024-02-01T10:16:35.446 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:35 GMT /vclfcn-8/ 2024-02-01T10:16:35.446 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1l2hchMKp7KJD7tayAV7l2VLzQE= 2024-02-01T10:16:35.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1l2hchMKp7KJD7tayAV7l2VLzQE=', 'Content-Length': '0'} 2024-02-01T10:16:35.448 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003f117d512e2fb20c-0065bb6f83-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:16:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:35.448 DEBUG:boto:path=/vclfcn-9/ 2024-02-01T10:16:35.448 DEBUG:boto:auth_path=/vclfcn-9/ 2024-02-01T10:16:35.449 DEBUG:boto:Method: HEAD 2024-02-01T10:16:35.449 DEBUG:boto:Path: /vclfcn-9/ 2024-02-01T10:16:35.449 DEBUG:boto:Data: 2024-02-01T10:16:35.449 DEBUG:boto:Headers: {} 2024-02-01T10:16:35.449 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:16:35.449 DEBUG:boto:Port: 8001 2024-02-01T10:16:35.450 DEBUG:boto:Params: {} 2024-02-01T10:16:35.450 DEBUG:boto:Token: None 2024-02-01T10:16:35.450 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:35 GMT /vclfcn-9/ 2024-02-01T10:16:35.450 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:51WNlU4dMc7BxzGMIE+duNZFJkk= 2024-02-01T10:16:35.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:51WNlU4dMc7BxzGMIE+duNZFJkk=', 'Content-Length': '0'} 2024-02-01T10:16:35.452 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c629d5a8b07057f5-0065bb6f83-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:16:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:35.452 DEBUG:boto:path=/vclfcn-7/ 2024-02-01T10:16:35.452 DEBUG:boto:auth_path=/vclfcn-7/ 2024-02-01T10:16:35.453 DEBUG:boto:Method: DELETE 2024-02-01T10:16:35.453 DEBUG:boto:Path: /vclfcn-7/ 2024-02-01T10:16:35.453 DEBUG:boto:Data: 2024-02-01T10:16:35.453 DEBUG:boto:Headers: {} 2024-02-01T10:16:35.453 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:35.453 DEBUG:boto:Port: 8000 2024-02-01T10:16:35.453 DEBUG:boto:Params: {} 2024-02-01T10:16:35.453 DEBUG:boto:Token: None 2024-02-01T10:16:35.453 DEBUG:boto:StringToSign: DELETE Thu, 01 Feb 2024 10:16:35 GMT /vclfcn-7/ 2024-02-01T10:16:35.454 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:a92ePcNvmue05dILj4APibl8+As= 2024-02-01T10:16:35.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:a92ePcNvmue05dILj4APibl8+As=', 'Content-Length': '0'} 2024-02-01T10:16:35.485 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004261d5fdc0c9de76-0065bb6f83-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:35.485 DEBUG:boto:path=/vclfcn-8/ 2024-02-01T10:16:35.485 DEBUG:boto:auth_path=/vclfcn-8/ 2024-02-01T10:16:35.485 DEBUG:boto:Method: DELETE 2024-02-01T10:16:35.485 DEBUG:boto:Path: /vclfcn-8/ 2024-02-01T10:16:35.485 DEBUG:boto:Data: 2024-02-01T10:16:35.485 DEBUG:boto:Headers: {} 2024-02-01T10:16:35.485 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:16:35.486 DEBUG:boto:Port: 8000 2024-02-01T10:16:35.486 DEBUG:boto:Params: {} 2024-02-01T10:16:35.486 DEBUG:boto:Token: None 2024-02-01T10:16:35.486 DEBUG:boto:StringToSign: DELETE Thu, 01 Feb 2024 10:16:35 GMT /vclfcn-8/ 2024-02-01T10:16:35.486 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/ut9nbN7eGEDoGxoHBfW1uk1Sck= 2024-02-01T10:16:35.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/ut9nbN7eGEDoGxoHBfW1uk1Sck=', 'Content-Length': '0'} 2024-02-01T10:16:35.601 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000055b4405b030a89cf-0065bb6f83-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:35.602 DEBUG:boto:path=/vclfcn-9/ 2024-02-01T10:16:35.602 DEBUG:boto:auth_path=/vclfcn-9/ 2024-02-01T10:16:35.602 DEBUG:boto:Method: DELETE 2024-02-01T10:16:35.602 DEBUG:boto:Path: /vclfcn-9/ 2024-02-01T10:16:35.602 DEBUG:boto:Data: 2024-02-01T10:16:35.602 DEBUG:boto:Headers: {} 2024-02-01T10:16:35.602 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:16:35.602 DEBUG:boto:Port: 8001 2024-02-01T10:16:35.602 DEBUG:boto:Params: {} 2024-02-01T10:16:35.603 DEBUG:boto:Token: None 2024-02-01T10:16:35.603 DEBUG:boto:StringToSign: DELETE Thu, 01 Feb 2024 10:16:35 GMT /vclfcn-9/ 2024-02-01T10:16:35.603 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:UcbJLEEqr7Z+9voF8XJ6B1wGT4Y= 2024-02-01T10:16:35.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:UcbJLEEqr7Z+9voF8XJ6B1wGT4Y=', 'Content-Length': '0'} 2024-02-01T10:16:35.732 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f7b6b22817fdc7c9-0065bb6f83-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:35.732 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:16:35.732 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:16:35.732 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:16:35.773 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:16:35.774 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:16:35.829 INFO:teuthology.orchestra.run.smithi148.stderr:No --period given, using current period=5dc4efbd-eccd-47a8-9471-ae4a209ec978 2024-02-01T10:16:35.853 INFO:teuthology.orchestra.run.smithi148.stdout:[{"marker":"1_1706782453.132619_30.1","last_update":"2024-02-01T10:14:13.132619Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1706782595.670952_121.1","last_update":"2024-02-01T10:16:35.670952Z"}] 2024-02-01T10:16:35.854 DEBUG:rgw_multi.tests:master meta markers={0: '1_1706782453.132619_30.1', 1: '', 2: '', 3: '1_1706782595.670952_121.1'} 2024-02-01T10:16:35.854 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-02-01T10:16:35.854 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:16:35.854 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:16:35.855 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:16:35.899 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:16:35.899 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:16:35.952 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:16:35.952+0000 7f13e4fee980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:16:35.953 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:16:35.952+0000 7f13e4fee980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:16:35.976 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:16:35Z"} 2024-02-01T10:16:35.976 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:16:35Z"} 2024-02-01T10:16:35.976 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:16:35.976 DEBUG:rgw_multi.tests:log_status={0: '1_1706782453.132619_30.1', 1: '', 2: '', 3: '1_1706782595.670952_121.1'} 2024-02-01T10:16:35.977 DEBUG:rgw_multi.tests:sync_status={0: '1_1706782453.132619_30.1', 1: '', 2: '', 3: '1_1706782595.670952_121.1'} 2024-02-01T10:16:35.977 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-02-01T10:16:35.977 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-02-01T10:16:35.977 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:16:35.977 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:16:35.977 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:16:36.020 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:16:36.020 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:16:36.074 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:16:36.073+0000 7f8ce1aa9980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:16:36.075 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:16:36.073+0000 7f8ce1aa9980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:16:36.098 INFO:teuthology.orchestra.run.smithi148.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:16:36Z"} 2024-02-01T10:16:36.098 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:16:36Z"} 2024-02-01T10:16:36.098 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:16:36.098 DEBUG:rgw_multi.tests:log_status={0: '1_1706782453.132619_30.1', 1: '', 2: '', 3: '1_1706782595.670952_121.1'} 2024-02-01T10:16:36.098 DEBUG:rgw_multi.tests:sync_status={0: '1_1706782453.132619_30.1', 1: '', 2: '', 3: '1_1706782595.670952_121.1'} 2024-02-01T10:16:36.099 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-02-01T10:16:36.099 DEBUG:boto:path=/vclfcn-7/ 2024-02-01T10:16:36.099 DEBUG:boto:auth_path=/vclfcn-7/ 2024-02-01T10:16:36.099 DEBUG:boto:Method: HEAD 2024-02-01T10:16:36.099 DEBUG:boto:Path: /vclfcn-7/ 2024-02-01T10:16:36.099 DEBUG:boto:Data: 2024-02-01T10:16:36.099 DEBUG:boto:Headers: {} 2024-02-01T10:16:36.099 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:36.099 DEBUG:boto:Port: 8000 2024-02-01T10:16:36.099 DEBUG:boto:Params: {} 2024-02-01T10:16:36.100 DEBUG:boto:Token: None 2024-02-01T10:16:36.100 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:36 GMT /vclfcn-7/ 2024-02-01T10:16:36.100 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:6NfX5w5xlRuuPrlQPqtn44lpuTg= 2024-02-01T10:16:36.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:6NfX5w5xlRuuPrlQPqtn44lpuTg=', 'Content-Length': '0'} 2024-02-01T10:16:36.101 DEBUG:boto:Response headers: [('Content-Length', '250'), ('x-amz-request-id', 'tx000007488ab6f9f3d5c81-0065bb6f84-4351-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:36.101 DEBUG:boto:path=/vclfcn-8/ 2024-02-01T10:16:36.101 DEBUG:boto:auth_path=/vclfcn-8/ 2024-02-01T10:16:36.102 DEBUG:boto:Method: HEAD 2024-02-01T10:16:36.102 DEBUG:boto:Path: /vclfcn-8/ 2024-02-01T10:16:36.102 DEBUG:boto:Data: 2024-02-01T10:16:36.102 DEBUG:boto:Headers: {} 2024-02-01T10:16:36.102 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:36.102 DEBUG:boto:Port: 8000 2024-02-01T10:16:36.102 DEBUG:boto:Params: {} 2024-02-01T10:16:36.102 DEBUG:boto:Token: None 2024-02-01T10:16:36.102 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:36 GMT /vclfcn-8/ 2024-02-01T10:16:36.102 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:hoGWvHgLsFu/1UCUVBMTk2183TE= 2024-02-01T10:16:36.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:hoGWvHgLsFu/1UCUVBMTk2183TE=', 'Content-Length': '0'} 2024-02-01T10:16:36.103 DEBUG:boto:Response headers: [('Content-Length', '250'), ('x-amz-request-id', 'tx00000e081b964c761bcf8-0065bb6f84-4351-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:36.103 DEBUG:boto:path=/vclfcn-9/ 2024-02-01T10:16:36.104 DEBUG:boto:auth_path=/vclfcn-9/ 2024-02-01T10:16:36.104 DEBUG:boto:Method: HEAD 2024-02-01T10:16:36.104 DEBUG:boto:Path: /vclfcn-9/ 2024-02-01T10:16:36.104 DEBUG:boto:Data: 2024-02-01T10:16:36.104 DEBUG:boto:Headers: {} 2024-02-01T10:16:36.104 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:36.104 DEBUG:boto:Port: 8000 2024-02-01T10:16:36.104 DEBUG:boto:Params: {} 2024-02-01T10:16:36.104 DEBUG:boto:Token: None 2024-02-01T10:16:36.104 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:36 GMT /vclfcn-9/ 2024-02-01T10:16:36.105 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5ULytCCrZ6YnYLCJVWb+SucFl0k= 2024-02-01T10:16:36.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5ULytCCrZ6YnYLCJVWb+SucFl0k=', 'Content-Length': '0'} 2024-02-01T10:16:36.106 DEBUG:boto:Response headers: [('Content-Length', '250'), ('x-amz-request-id', 'tx000004fd0940dce0f4e83-0065bb6f84-4351-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:36.106 DEBUG:boto:path=/vclfcn-7/ 2024-02-01T10:16:36.106 DEBUG:boto:auth_path=/vclfcn-7/ 2024-02-01T10:16:36.106 DEBUG:boto:Method: HEAD 2024-02-01T10:16:36.106 DEBUG:boto:Path: /vclfcn-7/ 2024-02-01T10:16:36.106 DEBUG:boto:Data: 2024-02-01T10:16:36.106 DEBUG:boto:Headers: {} 2024-02-01T10:16:36.106 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:16:36.106 DEBUG:boto:Port: 8000 2024-02-01T10:16:36.107 DEBUG:boto:Params: {} 2024-02-01T10:16:36.107 DEBUG:boto:Token: None 2024-02-01T10:16:36.107 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:36 GMT /vclfcn-7/ 2024-02-01T10:16:36.107 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:6NfX5w5xlRuuPrlQPqtn44lpuTg= 2024-02-01T10:16:36.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:6NfX5w5xlRuuPrlQPqtn44lpuTg=', 'Content-Length': '0'} 2024-02-01T10:16:36.110 DEBUG:boto:Response headers: [('Content-Length', '250'), ('x-amz-request-id', 'tx00000cd87098302ea2c2c-0065bb6f84-4307-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:36.110 DEBUG:boto:path=/vclfcn-8/ 2024-02-01T10:16:36.110 DEBUG:boto:auth_path=/vclfcn-8/ 2024-02-01T10:16:36.110 DEBUG:boto:Method: HEAD 2024-02-01T10:16:36.110 DEBUG:boto:Path: /vclfcn-8/ 2024-02-01T10:16:36.110 DEBUG:boto:Data: 2024-02-01T10:16:36.111 DEBUG:boto:Headers: {} 2024-02-01T10:16:36.111 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:16:36.111 DEBUG:boto:Port: 8000 2024-02-01T10:16:36.111 DEBUG:boto:Params: {} 2024-02-01T10:16:36.111 DEBUG:boto:Token: None 2024-02-01T10:16:36.111 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:36 GMT /vclfcn-8/ 2024-02-01T10:16:36.111 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:hoGWvHgLsFu/1UCUVBMTk2183TE= 2024-02-01T10:16:36.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:hoGWvHgLsFu/1UCUVBMTk2183TE=', 'Content-Length': '0'} 2024-02-01T10:16:36.113 DEBUG:boto:Response headers: [('Content-Length', '250'), ('x-amz-request-id', 'tx00000724c3be78cb284e6-0065bb6f84-4307-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:36.113 DEBUG:boto:path=/vclfcn-9/ 2024-02-01T10:16:36.113 DEBUG:boto:auth_path=/vclfcn-9/ 2024-02-01T10:16:36.113 DEBUG:boto:Method: HEAD 2024-02-01T10:16:36.113 DEBUG:boto:Path: /vclfcn-9/ 2024-02-01T10:16:36.113 DEBUG:boto:Data: 2024-02-01T10:16:36.113 DEBUG:boto:Headers: {} 2024-02-01T10:16:36.113 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:16:36.113 DEBUG:boto:Port: 8000 2024-02-01T10:16:36.113 DEBUG:boto:Params: {} 2024-02-01T10:16:36.114 DEBUG:boto:Token: None 2024-02-01T10:16:36.114 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:36 GMT /vclfcn-9/ 2024-02-01T10:16:36.114 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5ULytCCrZ6YnYLCJVWb+SucFl0k= 2024-02-01T10:16:36.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5ULytCCrZ6YnYLCJVWb+SucFl0k=', 'Content-Length': '0'} 2024-02-01T10:16:36.116 DEBUG:boto:Response headers: [('Content-Length', '250'), ('x-amz-request-id', 'tx000005a7f9a9b7501a6df-0065bb6f84-4307-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:36.116 DEBUG:boto:path=/vclfcn-7/ 2024-02-01T10:16:36.116 DEBUG:boto:auth_path=/vclfcn-7/ 2024-02-01T10:16:36.117 DEBUG:boto:Method: HEAD 2024-02-01T10:16:36.117 DEBUG:boto:Path: /vclfcn-7/ 2024-02-01T10:16:36.117 DEBUG:boto:Data: 2024-02-01T10:16:36.117 DEBUG:boto:Headers: {} 2024-02-01T10:16:36.117 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:16:36.117 DEBUG:boto:Port: 8001 2024-02-01T10:16:36.117 DEBUG:boto:Params: {} 2024-02-01T10:16:36.117 DEBUG:boto:Token: None 2024-02-01T10:16:36.117 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:36 GMT /vclfcn-7/ 2024-02-01T10:16:36.117 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:6NfX5w5xlRuuPrlQPqtn44lpuTg= 2024-02-01T10:16:36.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:6NfX5w5xlRuuPrlQPqtn44lpuTg=', 'Content-Length': '0'} 2024-02-01T10:16:36.119 DEBUG:boto:Response headers: [('Content-Length', '250'), ('x-amz-request-id', 'tx000007bbc6027a9aa0aba-0065bb6f84-4358-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:36.119 DEBUG:boto:path=/vclfcn-8/ 2024-02-01T10:16:36.120 DEBUG:boto:auth_path=/vclfcn-8/ 2024-02-01T10:16:36.120 DEBUG:boto:Method: HEAD 2024-02-01T10:16:36.120 DEBUG:boto:Path: /vclfcn-8/ 2024-02-01T10:16:36.120 DEBUG:boto:Data: 2024-02-01T10:16:36.120 DEBUG:boto:Headers: {} 2024-02-01T10:16:36.120 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:16:36.120 DEBUG:boto:Port: 8001 2024-02-01T10:16:36.120 DEBUG:boto:Params: {} 2024-02-01T10:16:36.120 DEBUG:boto:Token: None 2024-02-01T10:16:36.120 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:36 GMT /vclfcn-8/ 2024-02-01T10:16:36.121 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:hoGWvHgLsFu/1UCUVBMTk2183TE= 2024-02-01T10:16:36.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:hoGWvHgLsFu/1UCUVBMTk2183TE=', 'Content-Length': '0'} 2024-02-01T10:16:36.122 DEBUG:boto:Response headers: [('Content-Length', '250'), ('x-amz-request-id', 'tx00000827fe738ac4d113b-0065bb6f84-4358-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:36.122 DEBUG:boto:path=/vclfcn-9/ 2024-02-01T10:16:36.122 DEBUG:boto:auth_path=/vclfcn-9/ 2024-02-01T10:16:36.122 DEBUG:boto:Method: HEAD 2024-02-01T10:16:36.122 DEBUG:boto:Path: /vclfcn-9/ 2024-02-01T10:16:36.123 DEBUG:boto:Data: 2024-02-01T10:16:36.123 DEBUG:boto:Headers: {} 2024-02-01T10:16:36.123 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:16:36.123 DEBUG:boto:Port: 8001 2024-02-01T10:16:36.123 DEBUG:boto:Params: {} 2024-02-01T10:16:36.123 DEBUG:boto:Token: None 2024-02-01T10:16:36.123 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:36 GMT /vclfcn-9/ 2024-02-01T10:16:36.123 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5ULytCCrZ6YnYLCJVWb+SucFl0k= 2024-02-01T10:16:36.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5ULytCCrZ6YnYLCJVWb+SucFl0k=', 'Content-Length': '0'} 2024-02-01T10:16:36.125 DEBUG:boto:Response headers: [('Content-Length', '250'), ('x-amz-request-id', 'tx00000e9bcf9bd446fa985-0065bb6f84-4358-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:36.125 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_remove ... ok 2024-02-01T10:16:36.126 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=vclfcn-10 2024-02-01T10:16:36.126 DEBUG:boto:path=/vclfcn-10/ 2024-02-01T10:16:36.126 DEBUG:boto:auth_path=/vclfcn-10/ 2024-02-01T10:16:36.126 DEBUG:boto:Method: PUT 2024-02-01T10:16:36.126 DEBUG:boto:Path: /vclfcn-10/ 2024-02-01T10:16:36.126 DEBUG:boto:Data: 2024-02-01T10:16:36.126 DEBUG:boto:Headers: {} 2024-02-01T10:16:36.127 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:36.127 DEBUG:boto:Port: 8000 2024-02-01T10:16:36.127 DEBUG:boto:Params: {} 2024-02-01T10:16:36.127 DEBUG:boto:Token: None 2024-02-01T10:16:36.127 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:16:36 GMT /vclfcn-10/ 2024-02-01T10:16:36.127 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:hllXYnLDmQCukAuR/3l5wFMC7Hg= 2024-02-01T10:16:36.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:hllXYnLDmQCukAuR/3l5wFMC7Hg=', 'Content-Length': '0'} 2024-02-01T10:16:36.148 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009f247082261dbc01-0065bb6f84-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:36.148 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=vclfcn-11 2024-02-01T10:16:36.148 DEBUG:boto:path=/vclfcn-11/ 2024-02-01T10:16:36.148 DEBUG:boto:auth_path=/vclfcn-11/ 2024-02-01T10:16:36.148 DEBUG:boto:Method: PUT 2024-02-01T10:16:36.148 DEBUG:boto:Path: /vclfcn-11/ 2024-02-01T10:16:36.148 DEBUG:boto:Data: 2024-02-01T10:16:36.148 DEBUG:boto:Headers: {} 2024-02-01T10:16:36.148 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:16:36.149 DEBUG:boto:Port: 8000 2024-02-01T10:16:36.149 DEBUG:boto:Params: {} 2024-02-01T10:16:36.149 DEBUG:boto:Token: None 2024-02-01T10:16:36.149 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:16:36 GMT /vclfcn-11/ 2024-02-01T10:16:36.149 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:F5haetNXXHEsyMAhwvB+kZ2ESt8= 2024-02-01T10:16:36.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:F5haetNXXHEsyMAhwvB+kZ2ESt8=', 'Content-Length': '0'} 2024-02-01T10:16:36.197 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001b00489ef33eb84e-0065bb6f84-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:36.197 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=vclfcn-12 2024-02-01T10:16:36.198 DEBUG:boto:path=/vclfcn-12/ 2024-02-01T10:16:36.198 DEBUG:boto:auth_path=/vclfcn-12/ 2024-02-01T10:16:36.198 DEBUG:boto:Method: PUT 2024-02-01T10:16:36.198 DEBUG:boto:Path: /vclfcn-12/ 2024-02-01T10:16:36.198 DEBUG:boto:Data: 2024-02-01T10:16:36.198 DEBUG:boto:Headers: {} 2024-02-01T10:16:36.198 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:16:36.198 DEBUG:boto:Port: 8001 2024-02-01T10:16:36.198 DEBUG:boto:Params: {} 2024-02-01T10:16:36.199 DEBUG:boto:Token: None 2024-02-01T10:16:36.199 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:16:36 GMT /vclfcn-12/ 2024-02-01T10:16:36.199 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:xEoLn8V9CmLSfTdIJ9NFYefTpCg= 2024-02-01T10:16:36.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:xEoLn8V9CmLSfTdIJ9NFYefTpCg=', 'Content-Length': '0'} 2024-02-01T10:16:36.254 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ae7ccb524c20bd84-0065bb6f84-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:36.255 DEBUG:boto:path=/vclfcn-10/ 2024-02-01T10:16:36.255 DEBUG:boto:auth_path=/vclfcn-10/ 2024-02-01T10:16:36.255 DEBUG:boto:Method: HEAD 2024-02-01T10:16:36.255 DEBUG:boto:Path: /vclfcn-10/ 2024-02-01T10:16:36.255 DEBUG:boto:Data: 2024-02-01T10:16:36.255 DEBUG:boto:Headers: {} 2024-02-01T10:16:36.255 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:36.255 DEBUG:boto:Port: 8000 2024-02-01T10:16:36.255 DEBUG:boto:Params: {} 2024-02-01T10:16:36.255 DEBUG:boto:Token: None 2024-02-01T10:16:36.256 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:36 GMT /vclfcn-10/ 2024-02-01T10:16:36.256 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:dE824M82OPAUFOKhxAQ8AzvAOqs= 2024-02-01T10:16:36.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dE824M82OPAUFOKhxAQ8AzvAOqs=', 'Content-Length': '0'} 2024-02-01T10:16:36.260 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002b9234a669351b9e-0065bb6f84-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:36.260 DEBUG:boto:path=/vclfcn-10/myobj 2024-02-01T10:16:36.260 DEBUG:boto:auth_path=/vclfcn-10/myobj 2024-02-01T10:16:36.260 DEBUG:boto:Method: PUT 2024-02-01T10:16:36.261 DEBUG:boto:Path: /vclfcn-10/myobj 2024-02-01T10:16:36.261 DEBUG:boto:Data: 2024-02-01T10:16:36.261 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-02-01T10:16:36.261 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:36.261 DEBUG:boto:Port: 8000 2024-02-01T10:16:36.261 DEBUG:boto:Params: {} 2024-02-01T10:16:36.261 DEBUG:boto:Token: None 2024-02-01T10:16:36.261 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 01 Feb 2024 10:16:36 GMT /vclfcn-10/myobj 2024-02-01T10:16:36.261 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Y6HxcmjYXdZvHOFdub2RJJOtTpE= 2024-02-01T10:16:36.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:16:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Y6HxcmjYXdZvHOFdub2RJJOtTpE='} 2024-02-01T10:16:36.284 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009bda2765af3b00e4-0065bb6f84-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:36.284 DEBUG:boto:path=/vclfcn-10/ 2024-02-01T10:16:36.284 DEBUG:boto:auth_path=/vclfcn-10/ 2024-02-01T10:16:36.284 DEBUG:boto:Method: HEAD 2024-02-01T10:16:36.284 DEBUG:boto:Path: /vclfcn-10/ 2024-02-01T10:16:36.284 DEBUG:boto:Data: 2024-02-01T10:16:36.284 DEBUG:boto:Headers: {} 2024-02-01T10:16:36.284 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:36.284 DEBUG:boto:Port: 8000 2024-02-01T10:16:36.284 DEBUG:boto:Params: {} 2024-02-01T10:16:36.285 DEBUG:boto:Token: None 2024-02-01T10:16:36.285 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:36 GMT /vclfcn-10/ 2024-02-01T10:16:36.285 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:dE824M82OPAUFOKhxAQ8AzvAOqs= 2024-02-01T10:16:36.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dE824M82OPAUFOKhxAQ8AzvAOqs=', 'Content-Length': '0'} 2024-02-01T10:16:36.289 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000951f1de9074f44da-0065bb6f84-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:36.289 DEBUG:boto:path=/vclfcn-10/_myobj 2024-02-01T10:16:36.289 DEBUG:boto:auth_path=/vclfcn-10/_myobj 2024-02-01T10:16:36.289 DEBUG:boto:Method: PUT 2024-02-01T10:16:36.289 DEBUG:boto:Path: /vclfcn-10/_myobj 2024-02-01T10:16:36.289 DEBUG:boto:Data: 2024-02-01T10:16:36.290 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-02-01T10:16:36.290 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:36.290 DEBUG:boto:Port: 8000 2024-02-01T10:16:36.290 DEBUG:boto:Params: {} 2024-02-01T10:16:36.290 DEBUG:boto:Token: None 2024-02-01T10:16:36.290 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 01 Feb 2024 10:16:36 GMT /vclfcn-10/_myobj 2024-02-01T10:16:36.290 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:VSfbl4AgDvJA8k0sLF1BwYTs3sg= 2024-02-01T10:16:36.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:16:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:VSfbl4AgDvJA8k0sLF1BwYTs3sg='} 2024-02-01T10:16:36.336 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000eb0628f916c3062f-0065bb6f84-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:36.337 DEBUG:boto:path=/vclfcn-10/ 2024-02-01T10:16:36.337 DEBUG:boto:auth_path=/vclfcn-10/ 2024-02-01T10:16:36.337 DEBUG:boto:Method: HEAD 2024-02-01T10:16:36.337 DEBUG:boto:Path: /vclfcn-10/ 2024-02-01T10:16:36.337 DEBUG:boto:Data: 2024-02-01T10:16:36.337 DEBUG:boto:Headers: {} 2024-02-01T10:16:36.337 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:36.337 DEBUG:boto:Port: 8000 2024-02-01T10:16:36.337 DEBUG:boto:Params: {} 2024-02-01T10:16:36.337 DEBUG:boto:Token: None 2024-02-01T10:16:36.338 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:36 GMT /vclfcn-10/ 2024-02-01T10:16:36.338 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:dE824M82OPAUFOKhxAQ8AzvAOqs= 2024-02-01T10:16:36.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dE824M82OPAUFOKhxAQ8AzvAOqs=', 'Content-Length': '0'} 2024-02-01T10:16:36.344 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000285c160849e52301-0065bb6f84-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:36.344 DEBUG:boto:path=/vclfcn-10/%3A 2024-02-01T10:16:36.344 DEBUG:boto:auth_path=/vclfcn-10/%3A 2024-02-01T10:16:36.344 DEBUG:boto:Method: PUT 2024-02-01T10:16:36.344 DEBUG:boto:Path: /vclfcn-10/%3A 2024-02-01T10:16:36.345 DEBUG:boto:Data: 2024-02-01T10:16:36.345 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-02-01T10:16:36.345 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:36.345 DEBUG:boto:Port: 8000 2024-02-01T10:16:36.345 DEBUG:boto:Params: {} 2024-02-01T10:16:36.345 DEBUG:boto:Token: None 2024-02-01T10:16:36.345 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 01 Feb 2024 10:16:36 GMT /vclfcn-10/%3A 2024-02-01T10:16:36.345 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:oz+RXDhkJfvS+CaZ8doGceVeB2E= 2024-02-01T10:16:36.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:16:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:oz+RXDhkJfvS+CaZ8doGceVeB2E='} 2024-02-01T10:16:36.367 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ddc7b33d4c75bc41-0065bb6f84-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:36.367 DEBUG:boto:path=/vclfcn-10/ 2024-02-01T10:16:36.367 DEBUG:boto:auth_path=/vclfcn-10/ 2024-02-01T10:16:36.367 DEBUG:boto:Method: HEAD 2024-02-01T10:16:36.367 DEBUG:boto:Path: /vclfcn-10/ 2024-02-01T10:16:36.368 DEBUG:boto:Data: 2024-02-01T10:16:36.368 DEBUG:boto:Headers: {} 2024-02-01T10:16:36.368 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:36.368 DEBUG:boto:Port: 8000 2024-02-01T10:16:36.368 DEBUG:boto:Params: {} 2024-02-01T10:16:36.368 DEBUG:boto:Token: None 2024-02-01T10:16:36.368 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:36 GMT /vclfcn-10/ 2024-02-01T10:16:36.368 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:dE824M82OPAUFOKhxAQ8AzvAOqs= 2024-02-01T10:16:36.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dE824M82OPAUFOKhxAQ8AzvAOqs=', 'Content-Length': '0'} 2024-02-01T10:16:36.373 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000c28c89c627260dc-0065bb6f84-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:36.373 DEBUG:boto:path=/vclfcn-10/%26 2024-02-01T10:16:36.373 DEBUG:boto:auth_path=/vclfcn-10/%26 2024-02-01T10:16:36.373 DEBUG:boto:Method: PUT 2024-02-01T10:16:36.373 DEBUG:boto:Path: /vclfcn-10/%26 2024-02-01T10:16:36.373 DEBUG:boto:Data: 2024-02-01T10:16:36.374 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-02-01T10:16:36.374 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:36.374 DEBUG:boto:Port: 8000 2024-02-01T10:16:36.374 DEBUG:boto:Params: {} 2024-02-01T10:16:36.374 DEBUG:boto:Token: None 2024-02-01T10:16:36.374 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 01 Feb 2024 10:16:36 GMT /vclfcn-10/%26 2024-02-01T10:16:36.374 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:XlmgEn7tienZlTqSKG/lRKW/5fA= 2024-02-01T10:16:36.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:16:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:XlmgEn7tienZlTqSKG/lRKW/5fA='} 2024-02-01T10:16:36.420 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000086e62f05c89bbe00-0065bb6f84-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:36.420 DEBUG:boto:path=/vclfcn-11/ 2024-02-01T10:16:36.421 DEBUG:boto:auth_path=/vclfcn-11/ 2024-02-01T10:16:36.421 DEBUG:boto:Method: HEAD 2024-02-01T10:16:36.421 DEBUG:boto:Path: /vclfcn-11/ 2024-02-01T10:16:36.421 DEBUG:boto:Data: 2024-02-01T10:16:36.421 DEBUG:boto:Headers: {} 2024-02-01T10:16:36.421 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:16:36.422 DEBUG:boto:Port: 8000 2024-02-01T10:16:36.422 DEBUG:boto:Params: {} 2024-02-01T10:16:36.422 DEBUG:boto:Token: None 2024-02-01T10:16:36.422 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:36 GMT /vclfcn-11/ 2024-02-01T10:16:36.422 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:evqlc3JcECZ822PzCmA23g16dxU= 2024-02-01T10:16:36.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:evqlc3JcECZ822PzCmA23g16dxU=', 'Content-Length': '0'} 2024-02-01T10:16:36.426 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000024ee41ae987bf11e-0065bb6f84-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:36.426 DEBUG:boto:path=/vclfcn-11/myobj 2024-02-01T10:16:36.427 DEBUG:boto:auth_path=/vclfcn-11/myobj 2024-02-01T10:16:36.427 DEBUG:boto:Method: PUT 2024-02-01T10:16:36.427 DEBUG:boto:Path: /vclfcn-11/myobj 2024-02-01T10:16:36.427 DEBUG:boto:Data: 2024-02-01T10:16:36.427 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-02-01T10:16:36.427 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:16:36.427 DEBUG:boto:Port: 8000 2024-02-01T10:16:36.427 DEBUG:boto:Params: {} 2024-02-01T10:16:36.427 DEBUG:boto:Token: None 2024-02-01T10:16:36.428 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 01 Feb 2024 10:16:36 GMT /vclfcn-11/myobj 2024-02-01T10:16:36.428 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:hVw/2UkqmdNaJ3dK/db/YSac12Q= 2024-02-01T10:16:36.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:16:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:hVw/2UkqmdNaJ3dK/db/YSac12Q='} 2024-02-01T10:16:36.451 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007ed0325731f31e1d-0065bb6f84-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:36.451 DEBUG:boto:path=/vclfcn-11/ 2024-02-01T10:16:36.451 DEBUG:boto:auth_path=/vclfcn-11/ 2024-02-01T10:16:36.451 DEBUG:boto:Method: HEAD 2024-02-01T10:16:36.451 DEBUG:boto:Path: /vclfcn-11/ 2024-02-01T10:16:36.452 DEBUG:boto:Data: 2024-02-01T10:16:36.452 DEBUG:boto:Headers: {} 2024-02-01T10:16:36.452 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:16:36.452 DEBUG:boto:Port: 8000 2024-02-01T10:16:36.452 DEBUG:boto:Params: {} 2024-02-01T10:16:36.452 DEBUG:boto:Token: None 2024-02-01T10:16:36.452 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:36 GMT /vclfcn-11/ 2024-02-01T10:16:36.452 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:evqlc3JcECZ822PzCmA23g16dxU= 2024-02-01T10:16:36.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:evqlc3JcECZ822PzCmA23g16dxU=', 'Content-Length': '0'} 2024-02-01T10:16:36.456 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c096ef6768263ece-0065bb6f84-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:36.457 DEBUG:boto:path=/vclfcn-11/_myobj 2024-02-01T10:16:36.457 DEBUG:boto:auth_path=/vclfcn-11/_myobj 2024-02-01T10:16:36.457 DEBUG:boto:Method: PUT 2024-02-01T10:16:36.457 DEBUG:boto:Path: /vclfcn-11/_myobj 2024-02-01T10:16:36.457 DEBUG:boto:Data: 2024-02-01T10:16:36.457 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-02-01T10:16:36.458 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:16:36.458 DEBUG:boto:Port: 8000 2024-02-01T10:16:36.458 DEBUG:boto:Params: {} 2024-02-01T10:16:36.458 DEBUG:boto:Token: None 2024-02-01T10:16:36.458 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 01 Feb 2024 10:16:36 GMT /vclfcn-11/_myobj 2024-02-01T10:16:36.458 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/r+6UEhYY2w/eD1OGHPIa2HjpAo= 2024-02-01T10:16:36.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:16:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/r+6UEhYY2w/eD1OGHPIa2HjpAo='} 2024-02-01T10:16:36.500 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f1d5e62a5ca011fb-0065bb6f84-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:36.500 DEBUG:boto:path=/vclfcn-11/ 2024-02-01T10:16:36.500 DEBUG:boto:auth_path=/vclfcn-11/ 2024-02-01T10:16:36.501 DEBUG:boto:Method: HEAD 2024-02-01T10:16:36.501 DEBUG:boto:Path: /vclfcn-11/ 2024-02-01T10:16:36.501 DEBUG:boto:Data: 2024-02-01T10:16:36.501 DEBUG:boto:Headers: {} 2024-02-01T10:16:36.501 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:16:36.501 DEBUG:boto:Port: 8000 2024-02-01T10:16:36.501 DEBUG:boto:Params: {} 2024-02-01T10:16:36.501 DEBUG:boto:Token: None 2024-02-01T10:16:36.502 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:36 GMT /vclfcn-11/ 2024-02-01T10:16:36.502 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:evqlc3JcECZ822PzCmA23g16dxU= 2024-02-01T10:16:36.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:evqlc3JcECZ822PzCmA23g16dxU=', 'Content-Length': '0'} 2024-02-01T10:16:36.506 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f7a3fcde7fd0c3f7-0065bb6f84-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:36.506 DEBUG:boto:path=/vclfcn-11/%3A 2024-02-01T10:16:36.506 DEBUG:boto:auth_path=/vclfcn-11/%3A 2024-02-01T10:16:36.507 DEBUG:boto:Method: PUT 2024-02-01T10:16:36.507 DEBUG:boto:Path: /vclfcn-11/%3A 2024-02-01T10:16:36.507 DEBUG:boto:Data: 2024-02-01T10:16:36.507 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-02-01T10:16:36.507 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:16:36.507 DEBUG:boto:Port: 8000 2024-02-01T10:16:36.507 DEBUG:boto:Params: {} 2024-02-01T10:16:36.507 DEBUG:boto:Token: None 2024-02-01T10:16:36.508 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 01 Feb 2024 10:16:36 GMT /vclfcn-11/%3A 2024-02-01T10:16:36.508 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:eiF73EimgNKrwwheBvJvFj+q6oM= 2024-02-01T10:16:36.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:16:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:eiF73EimgNKrwwheBvJvFj+q6oM='} 2024-02-01T10:16:36.527 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003cb1e31d12a48095-0065bb6f84-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:36.527 DEBUG:boto:path=/vclfcn-11/ 2024-02-01T10:16:36.527 DEBUG:boto:auth_path=/vclfcn-11/ 2024-02-01T10:16:36.527 DEBUG:boto:Method: HEAD 2024-02-01T10:16:36.527 DEBUG:boto:Path: /vclfcn-11/ 2024-02-01T10:16:36.527 DEBUG:boto:Data: 2024-02-01T10:16:36.528 DEBUG:boto:Headers: {} 2024-02-01T10:16:36.528 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:16:36.528 DEBUG:boto:Port: 8000 2024-02-01T10:16:36.528 DEBUG:boto:Params: {} 2024-02-01T10:16:36.528 DEBUG:boto:Token: None 2024-02-01T10:16:36.528 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:36 GMT /vclfcn-11/ 2024-02-01T10:16:36.528 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:evqlc3JcECZ822PzCmA23g16dxU= 2024-02-01T10:16:36.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:evqlc3JcECZ822PzCmA23g16dxU=', 'Content-Length': '0'} 2024-02-01T10:16:36.532 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000076a1bbcf0ddcf8c3-0065bb6f84-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:36.532 DEBUG:boto:path=/vclfcn-11/%26 2024-02-01T10:16:36.532 DEBUG:boto:auth_path=/vclfcn-11/%26 2024-02-01T10:16:36.533 DEBUG:boto:Method: PUT 2024-02-01T10:16:36.533 DEBUG:boto:Path: /vclfcn-11/%26 2024-02-01T10:16:36.533 DEBUG:boto:Data: 2024-02-01T10:16:36.533 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-02-01T10:16:36.533 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:16:36.533 DEBUG:boto:Port: 8000 2024-02-01T10:16:36.533 DEBUG:boto:Params: {} 2024-02-01T10:16:36.533 DEBUG:boto:Token: None 2024-02-01T10:16:36.534 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 01 Feb 2024 10:16:36 GMT /vclfcn-11/%26 2024-02-01T10:16:36.534 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8yqWTMDbo/NS4Kn2boHgkXTClP0= 2024-02-01T10:16:36.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:16:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8yqWTMDbo/NS4Kn2boHgkXTClP0='} 2024-02-01T10:16:36.576 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000046593b217f0e8af0-0065bb6f84-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:36.576 DEBUG:boto:path=/vclfcn-12/ 2024-02-01T10:16:36.576 DEBUG:boto:auth_path=/vclfcn-12/ 2024-02-01T10:16:36.577 DEBUG:boto:Method: HEAD 2024-02-01T10:16:36.579 DEBUG:boto:Path: /vclfcn-12/ 2024-02-01T10:16:36.579 DEBUG:boto:Data: 2024-02-01T10:16:36.579 DEBUG:boto:Headers: {} 2024-02-01T10:16:36.579 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:16:36.579 DEBUG:boto:Port: 8001 2024-02-01T10:16:36.580 DEBUG:boto:Params: {} 2024-02-01T10:16:36.580 DEBUG:boto:Token: None 2024-02-01T10:16:36.580 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:36 GMT /vclfcn-12/ 2024-02-01T10:16:36.580 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5tnWP8omhBZ5iPAa/Fn8XSIHF6I= 2024-02-01T10:16:36.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5tnWP8omhBZ5iPAa/Fn8XSIHF6I=', 'Content-Length': '0'} 2024-02-01T10:16:36.584 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000558776550ab711e5-0065bb6f84-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:36.585 DEBUG:boto:path=/vclfcn-12/myobj 2024-02-01T10:16:36.585 DEBUG:boto:auth_path=/vclfcn-12/myobj 2024-02-01T10:16:36.585 DEBUG:boto:Method: PUT 2024-02-01T10:16:36.585 DEBUG:boto:Path: /vclfcn-12/myobj 2024-02-01T10:16:36.585 DEBUG:boto:Data: 2024-02-01T10:16:36.585 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-02-01T10:16:36.585 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:16:36.585 DEBUG:boto:Port: 8001 2024-02-01T10:16:36.585 DEBUG:boto:Params: {} 2024-02-01T10:16:36.586 DEBUG:boto:Token: None 2024-02-01T10:16:36.586 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 01 Feb 2024 10:16:36 GMT /vclfcn-12/myobj 2024-02-01T10:16:36.586 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZG7lKkZjUFcYC7/fXmtzvch4bcU= 2024-02-01T10:16:36.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:16:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZG7lKkZjUFcYC7/fXmtzvch4bcU='} 2024-02-01T10:16:36.628 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000021fe315f66be889f-0065bb6f84-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:36.628 DEBUG:boto:path=/vclfcn-12/ 2024-02-01T10:16:36.628 DEBUG:boto:auth_path=/vclfcn-12/ 2024-02-01T10:16:36.629 DEBUG:boto:Method: HEAD 2024-02-01T10:16:36.629 DEBUG:boto:Path: /vclfcn-12/ 2024-02-01T10:16:36.629 DEBUG:boto:Data: 2024-02-01T10:16:36.629 DEBUG:boto:Headers: {} 2024-02-01T10:16:36.629 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:16:36.629 DEBUG:boto:Port: 8001 2024-02-01T10:16:36.629 DEBUG:boto:Params: {} 2024-02-01T10:16:36.629 DEBUG:boto:Token: None 2024-02-01T10:16:36.630 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:36 GMT /vclfcn-12/ 2024-02-01T10:16:36.630 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5tnWP8omhBZ5iPAa/Fn8XSIHF6I= 2024-02-01T10:16:36.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5tnWP8omhBZ5iPAa/Fn8XSIHF6I=', 'Content-Length': '0'} 2024-02-01T10:16:36.634 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000381fbac7f5c112ad-0065bb6f84-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:36.634 DEBUG:boto:path=/vclfcn-12/_myobj 2024-02-01T10:16:36.634 DEBUG:boto:auth_path=/vclfcn-12/_myobj 2024-02-01T10:16:36.634 DEBUG:boto:Method: PUT 2024-02-01T10:16:36.634 DEBUG:boto:Path: /vclfcn-12/_myobj 2024-02-01T10:16:36.634 DEBUG:boto:Data: 2024-02-01T10:16:36.634 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-02-01T10:16:36.634 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:16:36.635 DEBUG:boto:Port: 8001 2024-02-01T10:16:36.635 DEBUG:boto:Params: {} 2024-02-01T10:16:36.635 DEBUG:boto:Token: None 2024-02-01T10:16:36.635 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 01 Feb 2024 10:16:36 GMT /vclfcn-12/_myobj 2024-02-01T10:16:36.635 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:TuCEMrs0QS99Wuh+Pt8rcD0I7kU= 2024-02-01T10:16:36.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:16:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:TuCEMrs0QS99Wuh+Pt8rcD0I7kU='} 2024-02-01T10:16:36.658 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001a41779e97234d8b-0065bb6f84-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:36.658 DEBUG:boto:path=/vclfcn-12/ 2024-02-01T10:16:36.659 DEBUG:boto:auth_path=/vclfcn-12/ 2024-02-01T10:16:36.659 DEBUG:boto:Method: HEAD 2024-02-01T10:16:36.659 DEBUG:boto:Path: /vclfcn-12/ 2024-02-01T10:16:36.659 DEBUG:boto:Data: 2024-02-01T10:16:36.659 DEBUG:boto:Headers: {} 2024-02-01T10:16:36.659 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:16:36.659 DEBUG:boto:Port: 8001 2024-02-01T10:16:36.659 DEBUG:boto:Params: {} 2024-02-01T10:16:36.659 DEBUG:boto:Token: None 2024-02-01T10:16:36.660 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:36 GMT /vclfcn-12/ 2024-02-01T10:16:36.660 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5tnWP8omhBZ5iPAa/Fn8XSIHF6I= 2024-02-01T10:16:36.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5tnWP8omhBZ5iPAa/Fn8XSIHF6I=', 'Content-Length': '0'} 2024-02-01T10:16:36.663 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007e349eae686af3a3-0065bb6f84-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:36.664 DEBUG:boto:path=/vclfcn-12/%3A 2024-02-01T10:16:36.664 DEBUG:boto:auth_path=/vclfcn-12/%3A 2024-02-01T10:16:36.664 DEBUG:boto:Method: PUT 2024-02-01T10:16:36.664 DEBUG:boto:Path: /vclfcn-12/%3A 2024-02-01T10:16:36.664 DEBUG:boto:Data: 2024-02-01T10:16:36.664 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-02-01T10:16:36.664 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:16:36.664 DEBUG:boto:Port: 8001 2024-02-01T10:16:36.665 DEBUG:boto:Params: {} 2024-02-01T10:16:36.665 DEBUG:boto:Token: None 2024-02-01T10:16:36.665 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 01 Feb 2024 10:16:36 GMT /vclfcn-12/%3A 2024-02-01T10:16:36.665 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Wd6JJO4/aFSmb1L1nHYbSHpIf9M= 2024-02-01T10:16:36.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:16:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Wd6JJO4/aFSmb1L1nHYbSHpIf9M='} 2024-02-01T10:16:36.708 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000492c9ebb7c21265a-0065bb6f84-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:36.708 DEBUG:boto:path=/vclfcn-12/ 2024-02-01T10:16:36.709 DEBUG:boto:auth_path=/vclfcn-12/ 2024-02-01T10:16:36.709 DEBUG:boto:Method: HEAD 2024-02-01T10:16:36.709 DEBUG:boto:Path: /vclfcn-12/ 2024-02-01T10:16:36.709 DEBUG:boto:Data: 2024-02-01T10:16:36.709 DEBUG:boto:Headers: {} 2024-02-01T10:16:36.709 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:16:36.709 DEBUG:boto:Port: 8001 2024-02-01T10:16:36.709 DEBUG:boto:Params: {} 2024-02-01T10:16:36.710 DEBUG:boto:Token: None 2024-02-01T10:16:36.710 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:36 GMT /vclfcn-12/ 2024-02-01T10:16:36.710 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5tnWP8omhBZ5iPAa/Fn8XSIHF6I= 2024-02-01T10:16:36.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5tnWP8omhBZ5iPAa/Fn8XSIHF6I=', 'Content-Length': '0'} 2024-02-01T10:16:36.714 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a736356d930532db-0065bb6f84-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:36.715 DEBUG:boto:path=/vclfcn-12/%26 2024-02-01T10:16:36.715 DEBUG:boto:auth_path=/vclfcn-12/%26 2024-02-01T10:16:36.715 DEBUG:boto:Method: PUT 2024-02-01T10:16:36.715 DEBUG:boto:Path: /vclfcn-12/%26 2024-02-01T10:16:36.715 DEBUG:boto:Data: 2024-02-01T10:16:36.715 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-02-01T10:16:36.715 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:16:36.715 DEBUG:boto:Port: 8001 2024-02-01T10:16:36.715 DEBUG:boto:Params: {} 2024-02-01T10:16:36.716 DEBUG:boto:Token: None 2024-02-01T10:16:36.716 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 01 Feb 2024 10:16:36 GMT /vclfcn-12/%26 2024-02-01T10:16:36.716 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:y3vcvbESNqs/vT2G5v1Xw68OGj0= 2024-02-01T10:16:36.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:16:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:y3vcvbESNqs/vT2G5v1Xw68OGj0='} 2024-02-01T10:16:36.728 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004e981260895e089b-0065bb6f84-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:36.728 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:16:36.728 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:16:36.728 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:16:36.770 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:16:36.770 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:16:36.824 INFO:teuthology.orchestra.run.smithi148.stderr:No --period given, using current period=5dc4efbd-eccd-47a8-9471-ae4a209ec978 2024-02-01T10:16:36.847 INFO:teuthology.orchestra.run.smithi148.stdout:[{"marker":"1_1706782453.132619_30.1","last_update":"2024-02-01T10:14:13.132619Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1706782596.230502_44.1","last_update":"2024-02-01T10:16:36.230502Z"},{"marker":"1_1706782595.670952_121.1","last_update":"2024-02-01T10:16:35.670952Z"}] 2024-02-01T10:16:36.848 DEBUG:rgw_multi.tests:master meta markers={0: '1_1706782453.132619_30.1', 1: '', 2: '1_1706782596.230502_44.1', 3: '1_1706782595.670952_121.1'} 2024-02-01T10:16:36.848 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-02-01T10:16:36.848 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:16:36.848 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:16:36.849 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:16:36.891 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:16:36.891 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:16:36.946 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:16:36.944+0000 7ff03b275980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:16:36.946 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:16:36.948+0000 7ff03b275980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:16:36.969 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706782596.142883_35.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:36.142883Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:16:36Z"} 2024-02-01T10:16:36.970 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706782596.142883_35.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:36.142883Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:16:36Z"} 2024-02-01T10:16:36.970 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706782596.142883_35.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:36.142883Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:16:36.970 DEBUG:rgw_multi.tests:log_status={0: '1_1706782453.132619_30.1', 1: '', 2: '1_1706782596.230502_44.1', 3: '1_1706782595.670952_121.1'} 2024-02-01T10:16:36.970 DEBUG:rgw_multi.tests:sync_status={0: '1_1706782453.132619_30.1', 1: '', 2: '1_1706782596.142883_35.1', 3: '1_1706782595.670952_121.1'} 2024-02-01T10:16:36.970 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1706782596.230502_44.1 target=1_1706782596.142883_35.1 2024-02-01T10:16:41.971 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:16:41.972 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:16:41.972 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:16:42.016 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:16:42.016 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:16:42.070 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:16:42.072+0000 7fb0e499c980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:16:42.071 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:16:42.072+0000 7fb0e499c980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:16:42.094 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706782596.230502_44.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:36.230502Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:16:42Z"} 2024-02-01T10:16:42.094 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706782596.230502_44.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:36.230502Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:16:42Z"} 2024-02-01T10:16:42.095 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706782596.230502_44.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:36.230502Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:16:42.095 DEBUG:rgw_multi.tests:log_status={0: '1_1706782453.132619_30.1', 1: '', 2: '1_1706782596.230502_44.1', 3: '1_1706782595.670952_121.1'} 2024-02-01T10:16:42.095 DEBUG:rgw_multi.tests:sync_status={0: '1_1706782453.132619_30.1', 1: '', 2: '1_1706782596.230502_44.1', 3: '1_1706782595.670952_121.1'} 2024-02-01T10:16:42.095 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-02-01T10:16:42.095 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-02-01T10:16:42.095 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:16:42.095 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:16:42.096 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:16:42.138 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:16:42.139 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:16:42.193 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:16:42.192+0000 7f0c2743b980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:16:42.193 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:16:42.192+0000 7f0c2743b980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:16:42.215 INFO:teuthology.orchestra.run.smithi148.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706782596.230502_44.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:36.230502Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:16:42Z"} 2024-02-01T10:16:42.216 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706782596.230502_44.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:36.230502Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:16:42Z"} 2024-02-01T10:16:42.216 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706782596.230502_44.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:36.230502Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:16:42.216 DEBUG:rgw_multi.tests:log_status={0: '1_1706782453.132619_30.1', 1: '', 2: '1_1706782596.230502_44.1', 3: '1_1706782595.670952_121.1'} 2024-02-01T10:16:42.216 DEBUG:rgw_multi.tests:sync_status={0: '1_1706782453.132619_30.1', 1: '', 2: '1_1706782596.230502_44.1', 3: '1_1706782595.670952_121.1'} 2024-02-01T10:16:42.216 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-02-01T10:16:42.217 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:16:42.217 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:16:42.217 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket vclfcn-10 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-02-01T10:16:42.266 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:16:42.267 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:16:42.328 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:16:42.328+0000 7f5f4bd0d980 1 waiting to reach incremental sync.. 2024-02-01T10:16:46.261 INFO:tasks.rgw.c2.client.0.smithi187.stdout:2024-02-01T10:16:46.260+0000 7f7bead57640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-02-01T10:16:47.329 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:16:47.328+0000 7f5f4bd0d980 1 waiting to reach incremental sync.. 2024-02-01T10:16:52.332 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:16:52.332+0000 7f5f4bd0d980 1 bucket sync caught up with source: 2024-02-01T10:16:52.332 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, 00000000001.13.6, , 00000000001.18.6, , , , 00000000001.20.6, 00000000001.7.6, , ] 2024-02-01T10:16:52.332 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, 00000000001.13.6, , 00000000001.18.6, , , , 00000000001.20.6, 00000000001.7.6, , ] 2024-02-01T10:16:52.332 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:16:52.332+0000 7f5f4bd0d980 0 bucket checkpoint complete 2024-02-01T10:16:52.378 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-10 zones={test-zone2, test-zone1} 2024-02-01T10:16:52.378 DEBUG:boto:path=/vclfcn-10/ 2024-02-01T10:16:52.378 DEBUG:boto:auth_path=/vclfcn-10/ 2024-02-01T10:16:52.378 DEBUG:boto:Method: HEAD 2024-02-01T10:16:52.379 DEBUG:boto:Path: /vclfcn-10/ 2024-02-01T10:16:52.379 DEBUG:boto:Data: 2024-02-01T10:16:52.379 DEBUG:boto:Headers: {} 2024-02-01T10:16:52.379 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:16:52.379 DEBUG:boto:Port: 8000 2024-02-01T10:16:52.379 DEBUG:boto:Params: {} 2024-02-01T10:16:52.379 DEBUG:boto:Token: None 2024-02-01T10:16:52.379 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:52 GMT /vclfcn-10/ 2024-02-01T10:16:52.380 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:yt+rZjotmFdu/D1wOaImTfpwAas= 2024-02-01T10:16:52.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:yt+rZjotmFdu/D1wOaImTfpwAas=', 'Content-Length': '0'} 2024-02-01T10:16:52.384 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007a9a55f7ea9c87b0-0065bb6f94-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:16:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:52.384 DEBUG:boto:path=/vclfcn-10/ 2024-02-01T10:16:52.384 DEBUG:boto:auth_path=/vclfcn-10/ 2024-02-01T10:16:52.384 DEBUG:boto:Method: HEAD 2024-02-01T10:16:52.384 DEBUG:boto:Path: /vclfcn-10/ 2024-02-01T10:16:52.384 DEBUG:boto:Data: 2024-02-01T10:16:52.385 DEBUG:boto:Headers: {} 2024-02-01T10:16:52.385 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:52.385 DEBUG:boto:Port: 8000 2024-02-01T10:16:52.385 DEBUG:boto:Params: {} 2024-02-01T10:16:52.385 DEBUG:boto:Token: None 2024-02-01T10:16:52.385 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:52 GMT /vclfcn-10/ 2024-02-01T10:16:52.385 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:yt+rZjotmFdu/D1wOaImTfpwAas= 2024-02-01T10:16:52.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:yt+rZjotmFdu/D1wOaImTfpwAas=', 'Content-Length': '0'} 2024-02-01T10:16:52.387 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a1c5aea26b8413f3-0065bb6f94-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:16:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:52.387 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:16:52.388 DEBUG:boto:path=/vclfcn-10/ 2024-02-01T10:16:52.388 DEBUG:boto:auth_path=/vclfcn-10/ 2024-02-01T10:16:52.388 DEBUG:boto:path=/vclfcn-10/?versions&max-keys=999 2024-02-01T10:16:52.388 DEBUG:boto:auth_path=/vclfcn-10/?versions&max-keys=999 2024-02-01T10:16:52.388 DEBUG:boto:Method: GET 2024-02-01T10:16:52.388 DEBUG:boto:Path: /vclfcn-10/?versions&max-keys=999 2024-02-01T10:16:52.388 DEBUG:boto:Data: 2024-02-01T10:16:52.388 DEBUG:boto:Headers: {} 2024-02-01T10:16:52.388 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:16:52.389 DEBUG:boto:Port: 8000 2024-02-01T10:16:52.389 DEBUG:boto:Params: {} 2024-02-01T10:16:52.389 DEBUG:boto:Token: None 2024-02-01T10:16:52.389 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:52 GMT /vclfcn-10/?versions 2024-02-01T10:16:52.389 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:LiGhc9Fz5vhHvAbY6j2wU+dYyTM= 2024-02-01T10:16:52.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:LiGhc9Fz5vhHvAbY6j2wU+dYyTM=', 'Content-Length': '0'} 2024-02-01T10:16:52.393 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4c8561985fb5a9e-0065bb6f94-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:52.393 DEBUG:boto:b'vclfcn-10999false&nulltrue2024-02-01T10:16:36.379Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-02-01T10:16:36.351Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-02-01T10:16:36.295Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-02-01T10:16:36.267Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:16:52.395 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-02-01T10:16:52.395 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-02-01T10:16:52.395 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-02-01T10:16:52.395 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-02-01T10:16:52.395 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:16:52.395 DEBUG:boto:path=/vclfcn-10/ 2024-02-01T10:16:52.396 DEBUG:boto:auth_path=/vclfcn-10/ 2024-02-01T10:16:52.396 DEBUG:boto:path=/vclfcn-10/?versions&max-keys=999 2024-02-01T10:16:52.396 DEBUG:boto:auth_path=/vclfcn-10/?versions&max-keys=999 2024-02-01T10:16:52.396 DEBUG:boto:Method: GET 2024-02-01T10:16:52.396 DEBUG:boto:Path: /vclfcn-10/?versions&max-keys=999 2024-02-01T10:16:52.396 DEBUG:boto:Data: 2024-02-01T10:16:52.396 DEBUG:boto:Headers: {} 2024-02-01T10:16:52.396 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:52.396 DEBUG:boto:Port: 8000 2024-02-01T10:16:52.396 DEBUG:boto:Params: {} 2024-02-01T10:16:52.397 DEBUG:boto:Token: None 2024-02-01T10:16:52.397 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:52 GMT /vclfcn-10/?versions 2024-02-01T10:16:52.397 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:LiGhc9Fz5vhHvAbY6j2wU+dYyTM= 2024-02-01T10:16:52.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:LiGhc9Fz5vhHvAbY6j2wU+dYyTM=', 'Content-Length': '0'} 2024-02-01T10:16:52.399 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f42bc0b4f8461fe9-0065bb6f94-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:52.399 DEBUG:boto:b'vclfcn-10999false&nulltrue2024-02-01T10:16:36.379Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-02-01T10:16:36.351Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-02-01T10:16:36.295Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-02-01T10:16:36.267Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:16:52.400 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-02-01T10:16:52.400 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-02-01T10:16:52.400 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-02-01T10:16:52.400 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-02-01T10:16:52.400 DEBUG:boto:path=/vclfcn-10/ 2024-02-01T10:16:52.400 DEBUG:boto:auth_path=/vclfcn-10/ 2024-02-01T10:16:52.400 DEBUG:boto:path=/vclfcn-10/?versions&max-keys=999 2024-02-01T10:16:52.400 DEBUG:boto:auth_path=/vclfcn-10/?versions&max-keys=999 2024-02-01T10:16:52.401 DEBUG:boto:Method: GET 2024-02-01T10:16:52.401 DEBUG:boto:Path: /vclfcn-10/?versions&max-keys=999 2024-02-01T10:16:52.401 DEBUG:boto:Data: 2024-02-01T10:16:52.401 DEBUG:boto:Headers: {} 2024-02-01T10:16:52.401 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:16:52.401 DEBUG:boto:Port: 8000 2024-02-01T10:16:52.401 DEBUG:boto:Params: {} 2024-02-01T10:16:52.401 DEBUG:boto:Token: None 2024-02-01T10:16:52.401 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:52 GMT /vclfcn-10/?versions 2024-02-01T10:16:52.401 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:LiGhc9Fz5vhHvAbY6j2wU+dYyTM= 2024-02-01T10:16:52.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:LiGhc9Fz5vhHvAbY6j2wU+dYyTM=', 'Content-Length': '0'} 2024-02-01T10:16:52.404 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087050b419c3a1b78-0065bb6f94-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:52.405 DEBUG:boto:b'vclfcn-10999false&nulltrue2024-02-01T10:16:36.379Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-02-01T10:16:36.351Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-02-01T10:16:36.295Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-02-01T10:16:36.267Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:16:52.405 DEBUG:boto:path=/vclfcn-10/ 2024-02-01T10:16:52.405 DEBUG:boto:auth_path=/vclfcn-10/ 2024-02-01T10:16:52.405 DEBUG:boto:path=/vclfcn-10/?versions&max-keys=999 2024-02-01T10:16:52.405 DEBUG:boto:auth_path=/vclfcn-10/?versions&max-keys=999 2024-02-01T10:16:52.405 DEBUG:boto:Method: GET 2024-02-01T10:16:52.406 DEBUG:boto:Path: /vclfcn-10/?versions&max-keys=999 2024-02-01T10:16:52.406 DEBUG:boto:Data: 2024-02-01T10:16:52.406 DEBUG:boto:Headers: {} 2024-02-01T10:16:52.406 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:52.406 DEBUG:boto:Port: 8000 2024-02-01T10:16:52.406 DEBUG:boto:Params: {} 2024-02-01T10:16:52.406 DEBUG:boto:Token: None 2024-02-01T10:16:52.406 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:52 GMT /vclfcn-10/?versions 2024-02-01T10:16:52.406 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:LiGhc9Fz5vhHvAbY6j2wU+dYyTM= 2024-02-01T10:16:52.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:LiGhc9Fz5vhHvAbY6j2wU+dYyTM=', 'Content-Length': '0'} 2024-02-01T10:16:52.409 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052b2ee7277bee005-0065bb6f94-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:52.409 DEBUG:boto:b'vclfcn-10999false&nulltrue2024-02-01T10:16:36.379Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-02-01T10:16:36.351Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-02-01T10:16:36.295Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-02-01T10:16:36.267Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:16:52.410 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-02-01T10:16:52.410 DEBUG:boto:path=/vclfcn-10/%26 2024-02-01T10:16:52.411 DEBUG:boto:auth_path=/vclfcn-10/%26 2024-02-01T10:16:52.411 DEBUG:boto:path=/vclfcn-10/%26?versionId=null 2024-02-01T10:16:52.412 DEBUG:boto:auth_path=/vclfcn-10/%26?versionId=null 2024-02-01T10:16:52.412 DEBUG:boto:Method: GET 2024-02-01T10:16:52.412 DEBUG:boto:Path: /vclfcn-10/%26?versionId=null 2024-02-01T10:16:52.412 DEBUG:boto:Data: 2024-02-01T10:16:52.412 DEBUG:boto:Headers: {} 2024-02-01T10:16:52.412 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:16:52.412 DEBUG:boto:Port: 8000 2024-02-01T10:16:52.412 DEBUG:boto:Params: {} 2024-02-01T10:16:52.413 DEBUG:boto:Token: None 2024-02-01T10:16:52.413 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:52 GMT /vclfcn-10/%26?versionId=null 2024-02-01T10:16:52.413 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Qm0mUpnxZ9NcppGU8LUFvpxsFHw= 2024-02-01T10:16:52.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Qm0mUpnxZ9NcppGU8LUFvpxsFHw=', 'Content-Length': '0'} 2024-02-01T10:16:52.416 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000df5473ac459e2aee-0065bb6f94-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:52.416 DEBUG:boto:path=/vclfcn-10/%26 2024-02-01T10:16:52.416 DEBUG:boto:auth_path=/vclfcn-10/%26 2024-02-01T10:16:52.417 DEBUG:boto:path=/vclfcn-10/%26?versionId=null 2024-02-01T10:16:52.417 DEBUG:boto:auth_path=/vclfcn-10/%26?versionId=null 2024-02-01T10:16:52.417 DEBUG:boto:Method: GET 2024-02-01T10:16:52.417 DEBUG:boto:Path: /vclfcn-10/%26?versionId=null 2024-02-01T10:16:52.417 DEBUG:boto:Data: 2024-02-01T10:16:52.417 DEBUG:boto:Headers: {} 2024-02-01T10:16:52.417 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:52.417 DEBUG:boto:Port: 8000 2024-02-01T10:16:52.417 DEBUG:boto:Params: {} 2024-02-01T10:16:52.417 DEBUG:boto:Token: None 2024-02-01T10:16:52.418 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:52 GMT /vclfcn-10/%26?versionId=null 2024-02-01T10:16:52.418 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Qm0mUpnxZ9NcppGU8LUFvpxsFHw= 2024-02-01T10:16:52.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Qm0mUpnxZ9NcppGU8LUFvpxsFHw=', 'Content-Length': '0'} 2024-02-01T10:16:52.420 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000837d1cfd08022a8c-0065bb6f94-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:52.420 DEBUG:boto:path=/vclfcn-10/%26 2024-02-01T10:16:52.420 DEBUG:boto:auth_path=/vclfcn-10/%26 2024-02-01T10:16:52.420 DEBUG:boto:path=/vclfcn-10/%26?versionId=null 2024-02-01T10:16:52.420 DEBUG:boto:auth_path=/vclfcn-10/%26?versionId=null 2024-02-01T10:16:52.421 DEBUG:boto:Method: HEAD 2024-02-01T10:16:52.421 DEBUG:boto:Path: /vclfcn-10/%26?versionId=null 2024-02-01T10:16:52.421 DEBUG:boto:Data: 2024-02-01T10:16:52.421 DEBUG:boto:Headers: {} 2024-02-01T10:16:52.421 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:16:52.421 DEBUG:boto:Port: 8000 2024-02-01T10:16:52.421 DEBUG:boto:Params: {} 2024-02-01T10:16:52.421 DEBUG:boto:Token: None 2024-02-01T10:16:52.421 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:52 GMT /vclfcn-10/%26?versionId=null 2024-02-01T10:16:52.422 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:oRI/tX8iZjuzH+mgjeTVtG31Bb0= 2024-02-01T10:16:52.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:oRI/tX8iZjuzH+mgjeTVtG31Bb0=', 'Content-Length': '0'} 2024-02-01T10:16:52.424 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b9b7ad99acf0c834-0065bb6f94-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:52.424 DEBUG:boto:path=/vclfcn-10/%26 2024-02-01T10:16:52.424 DEBUG:boto:auth_path=/vclfcn-10/%26 2024-02-01T10:16:52.424 DEBUG:boto:path=/vclfcn-10/%26?versionId=null 2024-02-01T10:16:52.425 DEBUG:boto:auth_path=/vclfcn-10/%26?versionId=null 2024-02-01T10:16:52.425 DEBUG:boto:Method: HEAD 2024-02-01T10:16:52.425 DEBUG:boto:Path: /vclfcn-10/%26?versionId=null 2024-02-01T10:16:52.425 DEBUG:boto:Data: 2024-02-01T10:16:52.425 DEBUG:boto:Headers: {} 2024-02-01T10:16:52.425 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:52.425 DEBUG:boto:Port: 8000 2024-02-01T10:16:52.425 DEBUG:boto:Params: {} 2024-02-01T10:16:52.425 DEBUG:boto:Token: None 2024-02-01T10:16:52.425 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:52 GMT /vclfcn-10/%26?versionId=null 2024-02-01T10:16:52.425 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:oRI/tX8iZjuzH+mgjeTVtG31Bb0= 2024-02-01T10:16:52.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:oRI/tX8iZjuzH+mgjeTVtG31Bb0=', 'Content-Length': '0'} 2024-02-01T10:16:52.427 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b00af3347da0ec70-0065bb6f94-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:52.428 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-02-01T10:16:52.428 DEBUG:boto:path=/vclfcn-10/%26 2024-02-01T10:16:52.428 DEBUG:boto:auth_path=/vclfcn-10/%26 2024-02-01T10:16:52.428 DEBUG:boto:path=/vclfcn-10/%26?versionId=null 2024-02-01T10:16:52.428 DEBUG:boto:auth_path=/vclfcn-10/%26?versionId=null 2024-02-01T10:16:52.428 DEBUG:boto:Method: GET 2024-02-01T10:16:52.428 DEBUG:boto:Path: /vclfcn-10/%26?versionId=null 2024-02-01T10:16:52.428 DEBUG:boto:Data: 2024-02-01T10:16:52.428 DEBUG:boto:Headers: {} 2024-02-01T10:16:52.429 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:16:52.429 DEBUG:boto:Port: 8000 2024-02-01T10:16:52.429 DEBUG:boto:Params: {} 2024-02-01T10:16:52.429 DEBUG:boto:Token: None 2024-02-01T10:16:52.429 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:52 GMT /vclfcn-10/%26?versionId=null 2024-02-01T10:16:52.429 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Qm0mUpnxZ9NcppGU8LUFvpxsFHw= 2024-02-01T10:16:52.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Qm0mUpnxZ9NcppGU8LUFvpxsFHw=', 'Content-Length': '0'} 2024-02-01T10:16:52.431 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000553339bf575d0dd8-0065bb6f94-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:52.476 DEBUG:boto:path=/vclfcn-10/%26 2024-02-01T10:16:52.476 DEBUG:boto:auth_path=/vclfcn-10/%26 2024-02-01T10:16:52.476 DEBUG:boto:path=/vclfcn-10/%26?versionId=null 2024-02-01T10:16:52.476 DEBUG:boto:auth_path=/vclfcn-10/%26?versionId=null 2024-02-01T10:16:52.477 DEBUG:boto:Method: GET 2024-02-01T10:16:52.477 DEBUG:boto:Path: /vclfcn-10/%26?versionId=null 2024-02-01T10:16:52.477 DEBUG:boto:Data: 2024-02-01T10:16:52.477 DEBUG:boto:Headers: {} 2024-02-01T10:16:52.477 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:52.477 DEBUG:boto:Port: 8000 2024-02-01T10:16:52.477 DEBUG:boto:Params: {} 2024-02-01T10:16:52.477 DEBUG:boto:Token: None 2024-02-01T10:16:52.477 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:52 GMT /vclfcn-10/%26?versionId=null 2024-02-01T10:16:52.478 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Qm0mUpnxZ9NcppGU8LUFvpxsFHw= 2024-02-01T10:16:52.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Qm0mUpnxZ9NcppGU8LUFvpxsFHw=', 'Content-Length': '0'} 2024-02-01T10:16:52.480 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000043ba32bedbcb72a1-0065bb6f94-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:52.481 DEBUG:boto:path=/vclfcn-10/ 2024-02-01T10:16:52.481 DEBUG:boto:auth_path=/vclfcn-10/ 2024-02-01T10:16:52.481 DEBUG:boto:path=/vclfcn-10/?prefix=%26 2024-02-01T10:16:52.481 DEBUG:boto:auth_path=/vclfcn-10/?prefix=%26 2024-02-01T10:16:52.481 DEBUG:boto:Method: GET 2024-02-01T10:16:52.481 DEBUG:boto:Path: /vclfcn-10/?prefix=%26 2024-02-01T10:16:52.481 DEBUG:boto:Data: 2024-02-01T10:16:52.482 DEBUG:boto:Headers: {} 2024-02-01T10:16:52.482 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:16:52.482 DEBUG:boto:Port: 8000 2024-02-01T10:16:52.482 DEBUG:boto:Params: {} 2024-02-01T10:16:52.482 DEBUG:boto:Token: None 2024-02-01T10:16:52.482 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:52 GMT /vclfcn-10/ 2024-02-01T10:16:52.482 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Pgkc0sSASjZxVxFnlTuL1OitXOU= 2024-02-01T10:16:52.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Pgkc0sSASjZxVxFnlTuL1OitXOU=', 'Content-Length': '0'} 2024-02-01T10:16:52.486 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000531a96f020edcd36-0065bb6f94-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:52.486 DEBUG:boto:b'vclfcn-10&1000false&2024-02-01T10:16:36.379Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:16:52.487 DEBUG:boto:path=/vclfcn-10/ 2024-02-01T10:16:52.487 DEBUG:boto:auth_path=/vclfcn-10/ 2024-02-01T10:16:52.487 DEBUG:boto:path=/vclfcn-10/?prefix=%26 2024-02-01T10:16:52.487 DEBUG:boto:auth_path=/vclfcn-10/?prefix=%26 2024-02-01T10:16:52.487 DEBUG:boto:Method: GET 2024-02-01T10:16:52.487 DEBUG:boto:Path: /vclfcn-10/?prefix=%26 2024-02-01T10:16:52.488 DEBUG:boto:Data: 2024-02-01T10:16:52.488 DEBUG:boto:Headers: {} 2024-02-01T10:16:52.488 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:52.488 DEBUG:boto:Port: 8000 2024-02-01T10:16:52.488 DEBUG:boto:Params: {} 2024-02-01T10:16:52.488 DEBUG:boto:Token: None 2024-02-01T10:16:52.488 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:52 GMT /vclfcn-10/ 2024-02-01T10:16:52.488 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Pgkc0sSASjZxVxFnlTuL1OitXOU= 2024-02-01T10:16:52.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Pgkc0sSASjZxVxFnlTuL1OitXOU=', 'Content-Length': '0'} 2024-02-01T10:16:52.493 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2d3c947f13a795f-0065bb6f94-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:52.493 DEBUG:boto:b'vclfcn-10&1000false&2024-02-01T10:16:36.379Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:16:52.493 DEBUG:boto:path=/vclfcn-10/%26 2024-02-01T10:16:52.494 DEBUG:boto:auth_path=/vclfcn-10/%26 2024-02-01T10:16:52.494 DEBUG:boto:Method: HEAD 2024-02-01T10:16:52.494 DEBUG:boto:Path: /vclfcn-10/%26 2024-02-01T10:16:52.494 DEBUG:boto:Data: 2024-02-01T10:16:52.494 DEBUG:boto:Headers: {} 2024-02-01T10:16:52.494 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:16:52.494 DEBUG:boto:Port: 8000 2024-02-01T10:16:52.494 DEBUG:boto:Params: {} 2024-02-01T10:16:52.494 DEBUG:boto:Token: None 2024-02-01T10:16:52.495 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:52 GMT /vclfcn-10/%26 2024-02-01T10:16:52.495 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:2zRxkZwQvNc8sCndvEDse9WAko8= 2024-02-01T10:16:52.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2zRxkZwQvNc8sCndvEDse9WAko8=', 'Content-Length': '0'} 2024-02-01T10:16:52.498 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aa2ef302b7ef41f9-0065bb6f94-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:52.498 DEBUG:boto:path=/vclfcn-10/%26 2024-02-01T10:16:52.498 DEBUG:boto:auth_path=/vclfcn-10/%26 2024-02-01T10:16:52.498 DEBUG:boto:Method: HEAD 2024-02-01T10:16:52.498 DEBUG:boto:Path: /vclfcn-10/%26 2024-02-01T10:16:52.498 DEBUG:boto:Data: 2024-02-01T10:16:52.498 DEBUG:boto:Headers: {} 2024-02-01T10:16:52.499 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:52.499 DEBUG:boto:Port: 8000 2024-02-01T10:16:52.499 DEBUG:boto:Params: {} 2024-02-01T10:16:52.499 DEBUG:boto:Token: None 2024-02-01T10:16:52.499 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:52 GMT /vclfcn-10/%26 2024-02-01T10:16:52.499 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:2zRxkZwQvNc8sCndvEDse9WAko8= 2024-02-01T10:16:52.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2zRxkZwQvNc8sCndvEDse9WAko8=', 'Content-Length': '0'} 2024-02-01T10:16:52.502 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000053b5dc95c273e1d2-0065bb6f94-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:52.502 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-02-01T10:16:52.502 DEBUG:boto:path=/vclfcn-10/%26 2024-02-01T10:16:52.502 DEBUG:boto:auth_path=/vclfcn-10/%26 2024-02-01T10:16:52.503 DEBUG:boto:Method: GET 2024-02-01T10:16:52.503 DEBUG:boto:Path: /vclfcn-10/%26 2024-02-01T10:16:52.503 DEBUG:boto:Data: 2024-02-01T10:16:52.503 DEBUG:boto:Headers: {} 2024-02-01T10:16:52.503 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:16:52.503 DEBUG:boto:Port: 8000 2024-02-01T10:16:52.503 DEBUG:boto:Params: {} 2024-02-01T10:16:52.503 DEBUG:boto:Token: None 2024-02-01T10:16:52.503 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:52 GMT /vclfcn-10/%26 2024-02-01T10:16:52.503 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ts+M2/z03g7WLPyndQ9dzrmwqzA= 2024-02-01T10:16:52.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ts+M2/z03g7WLPyndQ9dzrmwqzA=', 'Content-Length': '0'} 2024-02-01T10:16:52.506 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000dfdb41cde813762-0065bb6f94-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:52.548 DEBUG:boto:path=/vclfcn-10/%26 2024-02-01T10:16:52.548 DEBUG:boto:auth_path=/vclfcn-10/%26 2024-02-01T10:16:52.548 DEBUG:boto:Method: GET 2024-02-01T10:16:52.549 DEBUG:boto:Path: /vclfcn-10/%26 2024-02-01T10:16:52.549 DEBUG:boto:Data: 2024-02-01T10:16:52.549 DEBUG:boto:Headers: {} 2024-02-01T10:16:52.549 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:52.549 DEBUG:boto:Port: 8000 2024-02-01T10:16:52.549 DEBUG:boto:Params: {} 2024-02-01T10:16:52.549 DEBUG:boto:Token: None 2024-02-01T10:16:52.549 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:52 GMT /vclfcn-10/%26 2024-02-01T10:16:52.549 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ts+M2/z03g7WLPyndQ9dzrmwqzA= 2024-02-01T10:16:52.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ts+M2/z03g7WLPyndQ9dzrmwqzA=', 'Content-Length': '0'} 2024-02-01T10:16:52.553 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006046bf31084b0229-0065bb6f94-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:52.553 DEBUG:boto:path=/vclfcn-10/ 2024-02-01T10:16:52.553 DEBUG:boto:auth_path=/vclfcn-10/ 2024-02-01T10:16:52.553 DEBUG:boto:path=/vclfcn-10/?prefix=%26 2024-02-01T10:16:52.553 DEBUG:boto:auth_path=/vclfcn-10/?prefix=%26 2024-02-01T10:16:52.553 DEBUG:boto:Method: GET 2024-02-01T10:16:52.554 DEBUG:boto:Path: /vclfcn-10/?prefix=%26 2024-02-01T10:16:52.554 DEBUG:boto:Data: 2024-02-01T10:16:52.554 DEBUG:boto:Headers: {} 2024-02-01T10:16:52.554 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:16:52.554 DEBUG:boto:Port: 8000 2024-02-01T10:16:52.554 DEBUG:boto:Params: {} 2024-02-01T10:16:52.554 DEBUG:boto:Token: None 2024-02-01T10:16:52.554 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:52 GMT /vclfcn-10/ 2024-02-01T10:16:52.554 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Pgkc0sSASjZxVxFnlTuL1OitXOU= 2024-02-01T10:16:52.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Pgkc0sSASjZxVxFnlTuL1OitXOU=', 'Content-Length': '0'} 2024-02-01T10:16:52.559 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd684fc6e0fd581e-0065bb6f94-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:52.559 DEBUG:boto:b'vclfcn-10&1000false&2024-02-01T10:16:36.379Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:16:52.560 DEBUG:boto:path=/vclfcn-10/ 2024-02-01T10:16:52.560 DEBUG:boto:auth_path=/vclfcn-10/ 2024-02-01T10:16:52.560 DEBUG:boto:path=/vclfcn-10/?prefix=%26 2024-02-01T10:16:52.560 DEBUG:boto:auth_path=/vclfcn-10/?prefix=%26 2024-02-01T10:16:52.560 DEBUG:boto:Method: GET 2024-02-01T10:16:52.560 DEBUG:boto:Path: /vclfcn-10/?prefix=%26 2024-02-01T10:16:52.560 DEBUG:boto:Data: 2024-02-01T10:16:52.560 DEBUG:boto:Headers: {} 2024-02-01T10:16:52.560 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:52.560 DEBUG:boto:Port: 8000 2024-02-01T10:16:52.561 DEBUG:boto:Params: {} 2024-02-01T10:16:52.561 DEBUG:boto:Token: None 2024-02-01T10:16:52.561 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:52 GMT /vclfcn-10/ 2024-02-01T10:16:52.561 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Pgkc0sSASjZxVxFnlTuL1OitXOU= 2024-02-01T10:16:52.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Pgkc0sSASjZxVxFnlTuL1OitXOU=', 'Content-Length': '0'} 2024-02-01T10:16:52.565 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa8a29981f3ce2b5-0065bb6f94-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:52.566 DEBUG:boto:b'vclfcn-10&1000false&2024-02-01T10:16:36.379Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:16:52.566 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-02-01T10:16:52.566 DEBUG:boto:path=/vclfcn-10/%3A 2024-02-01T10:16:52.566 DEBUG:boto:auth_path=/vclfcn-10/%3A 2024-02-01T10:16:52.567 DEBUG:boto:path=/vclfcn-10/%3A?versionId=null 2024-02-01T10:16:52.567 DEBUG:boto:auth_path=/vclfcn-10/%3A?versionId=null 2024-02-01T10:16:52.567 DEBUG:boto:Method: GET 2024-02-01T10:16:52.567 DEBUG:boto:Path: /vclfcn-10/%3A?versionId=null 2024-02-01T10:16:52.567 DEBUG:boto:Data: 2024-02-01T10:16:52.567 DEBUG:boto:Headers: {} 2024-02-01T10:16:52.567 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:16:52.567 DEBUG:boto:Port: 8000 2024-02-01T10:16:52.567 DEBUG:boto:Params: {} 2024-02-01T10:16:52.568 DEBUG:boto:Token: None 2024-02-01T10:16:52.568 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:52 GMT /vclfcn-10/%3A?versionId=null 2024-02-01T10:16:52.568 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Gk5ceUTOk+GpwAUzz+BtDKxYEY0= 2024-02-01T10:16:52.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Gk5ceUTOk+GpwAUzz+BtDKxYEY0=', 'Content-Length': '0'} 2024-02-01T10:16:52.571 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001d87c13f01217926-0065bb6f94-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:52.572 DEBUG:boto:path=/vclfcn-10/%3A 2024-02-01T10:16:52.572 DEBUG:boto:auth_path=/vclfcn-10/%3A 2024-02-01T10:16:52.572 DEBUG:boto:path=/vclfcn-10/%3A?versionId=null 2024-02-01T10:16:52.572 DEBUG:boto:auth_path=/vclfcn-10/%3A?versionId=null 2024-02-01T10:16:52.572 DEBUG:boto:Method: GET 2024-02-01T10:16:52.573 DEBUG:boto:Path: /vclfcn-10/%3A?versionId=null 2024-02-01T10:16:52.573 DEBUG:boto:Data: 2024-02-01T10:16:52.573 DEBUG:boto:Headers: {} 2024-02-01T10:16:52.573 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:52.573 DEBUG:boto:Port: 8000 2024-02-01T10:16:52.573 DEBUG:boto:Params: {} 2024-02-01T10:16:52.573 DEBUG:boto:Token: None 2024-02-01T10:16:52.573 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:52 GMT /vclfcn-10/%3A?versionId=null 2024-02-01T10:16:52.573 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Gk5ceUTOk+GpwAUzz+BtDKxYEY0= 2024-02-01T10:16:52.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Gk5ceUTOk+GpwAUzz+BtDKxYEY0=', 'Content-Length': '0'} 2024-02-01T10:16:52.577 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000044fda5dcbd331499-0065bb6f94-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:52.577 DEBUG:boto:path=/vclfcn-10/%3A 2024-02-01T10:16:52.577 DEBUG:boto:auth_path=/vclfcn-10/%3A 2024-02-01T10:16:52.577 DEBUG:boto:path=/vclfcn-10/%3A?versionId=null 2024-02-01T10:16:52.577 DEBUG:boto:auth_path=/vclfcn-10/%3A?versionId=null 2024-02-01T10:16:52.577 DEBUG:boto:Method: HEAD 2024-02-01T10:16:52.577 DEBUG:boto:Path: /vclfcn-10/%3A?versionId=null 2024-02-01T10:16:52.577 DEBUG:boto:Data: 2024-02-01T10:16:52.578 DEBUG:boto:Headers: {} 2024-02-01T10:16:52.578 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:16:52.578 DEBUG:boto:Port: 8000 2024-02-01T10:16:52.578 DEBUG:boto:Params: {} 2024-02-01T10:16:52.578 DEBUG:boto:Token: None 2024-02-01T10:16:52.578 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:52 GMT /vclfcn-10/%3A?versionId=null 2024-02-01T10:16:52.578 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Z+Zo8P9ViktSoYOx74AH78dTb4k= 2024-02-01T10:16:52.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Z+Zo8P9ViktSoYOx74AH78dTb4k=', 'Content-Length': '0'} 2024-02-01T10:16:52.581 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b02bae74c6e90eb7-0065bb6f94-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:52.581 DEBUG:boto:path=/vclfcn-10/%3A 2024-02-01T10:16:52.582 DEBUG:boto:auth_path=/vclfcn-10/%3A 2024-02-01T10:16:52.582 DEBUG:boto:path=/vclfcn-10/%3A?versionId=null 2024-02-01T10:16:52.582 DEBUG:boto:auth_path=/vclfcn-10/%3A?versionId=null 2024-02-01T10:16:52.582 DEBUG:boto:Method: HEAD 2024-02-01T10:16:52.582 DEBUG:boto:Path: /vclfcn-10/%3A?versionId=null 2024-02-01T10:16:52.582 DEBUG:boto:Data: 2024-02-01T10:16:52.582 DEBUG:boto:Headers: {} 2024-02-01T10:16:52.582 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:52.582 DEBUG:boto:Port: 8000 2024-02-01T10:16:52.582 DEBUG:boto:Params: {} 2024-02-01T10:16:52.582 DEBUG:boto:Token: None 2024-02-01T10:16:52.583 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:52 GMT /vclfcn-10/%3A?versionId=null 2024-02-01T10:16:52.583 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Z+Zo8P9ViktSoYOx74AH78dTb4k= 2024-02-01T10:16:52.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Z+Zo8P9ViktSoYOx74AH78dTb4k=', 'Content-Length': '0'} 2024-02-01T10:16:52.585 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cac8ab69cdbbe97c-0065bb6f94-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:52.586 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-02-01T10:16:52.586 DEBUG:boto:path=/vclfcn-10/%3A 2024-02-01T10:16:52.586 DEBUG:boto:auth_path=/vclfcn-10/%3A 2024-02-01T10:16:52.586 DEBUG:boto:path=/vclfcn-10/%3A?versionId=null 2024-02-01T10:16:52.586 DEBUG:boto:auth_path=/vclfcn-10/%3A?versionId=null 2024-02-01T10:16:52.586 DEBUG:boto:Method: GET 2024-02-01T10:16:52.586 DEBUG:boto:Path: /vclfcn-10/%3A?versionId=null 2024-02-01T10:16:52.586 DEBUG:boto:Data: 2024-02-01T10:16:52.586 DEBUG:boto:Headers: {} 2024-02-01T10:16:52.586 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:16:52.587 DEBUG:boto:Port: 8000 2024-02-01T10:16:52.587 DEBUG:boto:Params: {} 2024-02-01T10:16:52.587 DEBUG:boto:Token: None 2024-02-01T10:16:52.587 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:52 GMT /vclfcn-10/%3A?versionId=null 2024-02-01T10:16:52.587 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Gk5ceUTOk+GpwAUzz+BtDKxYEY0= 2024-02-01T10:16:52.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Gk5ceUTOk+GpwAUzz+BtDKxYEY0=', 'Content-Length': '0'} 2024-02-01T10:16:52.590 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e9d4dbcf040638d2-0065bb6f94-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:52.636 DEBUG:boto:path=/vclfcn-10/%3A 2024-02-01T10:16:52.636 DEBUG:boto:auth_path=/vclfcn-10/%3A 2024-02-01T10:16:52.637 DEBUG:boto:path=/vclfcn-10/%3A?versionId=null 2024-02-01T10:16:52.637 DEBUG:boto:auth_path=/vclfcn-10/%3A?versionId=null 2024-02-01T10:16:52.637 DEBUG:boto:Method: GET 2024-02-01T10:16:52.637 DEBUG:boto:Path: /vclfcn-10/%3A?versionId=null 2024-02-01T10:16:52.637 DEBUG:boto:Data: 2024-02-01T10:16:52.637 DEBUG:boto:Headers: {} 2024-02-01T10:16:52.637 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:52.637 DEBUG:boto:Port: 8000 2024-02-01T10:16:52.637 DEBUG:boto:Params: {} 2024-02-01T10:16:52.637 DEBUG:boto:Token: None 2024-02-01T10:16:52.638 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:52 GMT /vclfcn-10/%3A?versionId=null 2024-02-01T10:16:52.638 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Gk5ceUTOk+GpwAUzz+BtDKxYEY0= 2024-02-01T10:16:52.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Gk5ceUTOk+GpwAUzz+BtDKxYEY0=', 'Content-Length': '0'} 2024-02-01T10:16:52.641 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001fed042183a0fa3a-0065bb6f94-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:52.641 DEBUG:boto:path=/vclfcn-10/ 2024-02-01T10:16:52.641 DEBUG:boto:auth_path=/vclfcn-10/ 2024-02-01T10:16:52.642 DEBUG:boto:path=/vclfcn-10/?prefix=%3A 2024-02-01T10:16:52.642 DEBUG:boto:auth_path=/vclfcn-10/?prefix=%3A 2024-02-01T10:16:52.642 DEBUG:boto:Method: GET 2024-02-01T10:16:52.642 DEBUG:boto:Path: /vclfcn-10/?prefix=%3A 2024-02-01T10:16:52.642 DEBUG:boto:Data: 2024-02-01T10:16:52.642 DEBUG:boto:Headers: {} 2024-02-01T10:16:52.642 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:16:52.642 DEBUG:boto:Port: 8000 2024-02-01T10:16:52.642 DEBUG:boto:Params: {} 2024-02-01T10:16:52.642 DEBUG:boto:Token: None 2024-02-01T10:16:52.642 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:52 GMT /vclfcn-10/ 2024-02-01T10:16:52.643 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Pgkc0sSASjZxVxFnlTuL1OitXOU= 2024-02-01T10:16:52.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Pgkc0sSASjZxVxFnlTuL1OitXOU=', 'Content-Length': '0'} 2024-02-01T10:16:52.647 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001890955e163aa204-0065bb6f94-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:52.647 DEBUG:boto:b'vclfcn-10:1000false:2024-02-01T10:16:36.351Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:16:52.648 DEBUG:boto:path=/vclfcn-10/ 2024-02-01T10:16:52.648 DEBUG:boto:auth_path=/vclfcn-10/ 2024-02-01T10:16:52.648 DEBUG:boto:path=/vclfcn-10/?prefix=%3A 2024-02-01T10:16:52.648 DEBUG:boto:auth_path=/vclfcn-10/?prefix=%3A 2024-02-01T10:16:52.648 DEBUG:boto:Method: GET 2024-02-01T10:16:52.648 DEBUG:boto:Path: /vclfcn-10/?prefix=%3A 2024-02-01T10:16:52.648 DEBUG:boto:Data: 2024-02-01T10:16:52.648 DEBUG:boto:Headers: {} 2024-02-01T10:16:52.649 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:52.649 DEBUG:boto:Port: 8000 2024-02-01T10:16:52.649 DEBUG:boto:Params: {} 2024-02-01T10:16:52.649 DEBUG:boto:Token: None 2024-02-01T10:16:52.649 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:52 GMT /vclfcn-10/ 2024-02-01T10:16:52.649 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Pgkc0sSASjZxVxFnlTuL1OitXOU= 2024-02-01T10:16:52.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Pgkc0sSASjZxVxFnlTuL1OitXOU=', 'Content-Length': '0'} 2024-02-01T10:16:52.655 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc5bcbe96be43c64-0065bb6f94-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:52.655 DEBUG:boto:b'vclfcn-10:1000false:2024-02-01T10:16:36.351Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:16:52.656 DEBUG:boto:path=/vclfcn-10/%3A 2024-02-01T10:16:52.656 DEBUG:boto:auth_path=/vclfcn-10/%3A 2024-02-01T10:16:52.656 DEBUG:boto:Method: HEAD 2024-02-01T10:16:52.656 DEBUG:boto:Path: /vclfcn-10/%3A 2024-02-01T10:16:52.656 DEBUG:boto:Data: 2024-02-01T10:16:52.656 DEBUG:boto:Headers: {} 2024-02-01T10:16:52.656 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:16:52.656 DEBUG:boto:Port: 8000 2024-02-01T10:16:52.657 DEBUG:boto:Params: {} 2024-02-01T10:16:52.657 DEBUG:boto:Token: None 2024-02-01T10:16:52.657 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:52 GMT /vclfcn-10/%3A 2024-02-01T10:16:52.657 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RlmvNYoAbQGD1mAXNMd8Z4Vk0kY= 2024-02-01T10:16:52.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RlmvNYoAbQGD1mAXNMd8Z4Vk0kY=', 'Content-Length': '0'} 2024-02-01T10:16:52.660 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002b7c8366493bdd62-0065bb6f94-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:52.660 DEBUG:boto:path=/vclfcn-10/%3A 2024-02-01T10:16:52.660 DEBUG:boto:auth_path=/vclfcn-10/%3A 2024-02-01T10:16:52.660 DEBUG:boto:Method: HEAD 2024-02-01T10:16:52.661 DEBUG:boto:Path: /vclfcn-10/%3A 2024-02-01T10:16:52.661 DEBUG:boto:Data: 2024-02-01T10:16:52.661 DEBUG:boto:Headers: {} 2024-02-01T10:16:52.661 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:52.661 DEBUG:boto:Port: 8000 2024-02-01T10:16:52.661 DEBUG:boto:Params: {} 2024-02-01T10:16:52.661 DEBUG:boto:Token: None 2024-02-01T10:16:52.661 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:52 GMT /vclfcn-10/%3A 2024-02-01T10:16:52.661 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RlmvNYoAbQGD1mAXNMd8Z4Vk0kY= 2024-02-01T10:16:52.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RlmvNYoAbQGD1mAXNMd8Z4Vk0kY=', 'Content-Length': '0'} 2024-02-01T10:16:52.664 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000982daad35ba0869-0065bb6f94-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:52.664 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-02-01T10:16:52.664 DEBUG:boto:path=/vclfcn-10/%3A 2024-02-01T10:16:52.664 DEBUG:boto:auth_path=/vclfcn-10/%3A 2024-02-01T10:16:52.665 DEBUG:boto:Method: GET 2024-02-01T10:16:52.665 DEBUG:boto:Path: /vclfcn-10/%3A 2024-02-01T10:16:52.665 DEBUG:boto:Data: 2024-02-01T10:16:52.665 DEBUG:boto:Headers: {} 2024-02-01T10:16:52.665 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:16:52.665 DEBUG:boto:Port: 8000 2024-02-01T10:16:52.665 DEBUG:boto:Params: {} 2024-02-01T10:16:52.665 DEBUG:boto:Token: None 2024-02-01T10:16:52.665 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:52 GMT /vclfcn-10/%3A 2024-02-01T10:16:52.665 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:c6wD3eTdM1MGPMRIWeGYfp6YgA0= 2024-02-01T10:16:52.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:c6wD3eTdM1MGPMRIWeGYfp6YgA0=', 'Content-Length': '0'} 2024-02-01T10:16:52.668 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000307064a783ff83e6-0065bb6f94-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:52.712 DEBUG:boto:path=/vclfcn-10/%3A 2024-02-01T10:16:52.712 DEBUG:boto:auth_path=/vclfcn-10/%3A 2024-02-01T10:16:52.712 DEBUG:boto:Method: GET 2024-02-01T10:16:52.712 DEBUG:boto:Path: /vclfcn-10/%3A 2024-02-01T10:16:52.712 DEBUG:boto:Data: 2024-02-01T10:16:52.713 DEBUG:boto:Headers: {} 2024-02-01T10:16:52.713 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:52.713 DEBUG:boto:Port: 8000 2024-02-01T10:16:52.713 DEBUG:boto:Params: {} 2024-02-01T10:16:52.713 DEBUG:boto:Token: None 2024-02-01T10:16:52.713 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:52 GMT /vclfcn-10/%3A 2024-02-01T10:16:52.713 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:c6wD3eTdM1MGPMRIWeGYfp6YgA0= 2024-02-01T10:16:52.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:c6wD3eTdM1MGPMRIWeGYfp6YgA0=', 'Content-Length': '0'} 2024-02-01T10:16:52.716 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000db2c74431a5b9751-0065bb6f94-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:52.716 DEBUG:boto:path=/vclfcn-10/ 2024-02-01T10:16:52.716 DEBUG:boto:auth_path=/vclfcn-10/ 2024-02-01T10:16:52.717 DEBUG:boto:path=/vclfcn-10/?prefix=%3A 2024-02-01T10:16:52.717 DEBUG:boto:auth_path=/vclfcn-10/?prefix=%3A 2024-02-01T10:16:52.717 DEBUG:boto:Method: GET 2024-02-01T10:16:52.717 DEBUG:boto:Path: /vclfcn-10/?prefix=%3A 2024-02-01T10:16:52.717 DEBUG:boto:Data: 2024-02-01T10:16:52.717 DEBUG:boto:Headers: {} 2024-02-01T10:16:52.717 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:16:52.717 DEBUG:boto:Port: 8000 2024-02-01T10:16:52.717 DEBUG:boto:Params: {} 2024-02-01T10:16:52.717 DEBUG:boto:Token: None 2024-02-01T10:16:52.718 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:52 GMT /vclfcn-10/ 2024-02-01T10:16:52.718 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Pgkc0sSASjZxVxFnlTuL1OitXOU= 2024-02-01T10:16:52.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Pgkc0sSASjZxVxFnlTuL1OitXOU=', 'Content-Length': '0'} 2024-02-01T10:16:52.722 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c20c7e827c683ae-0065bb6f94-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:52.722 DEBUG:boto:b'vclfcn-10:1000false:2024-02-01T10:16:36.351Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:16:52.723 DEBUG:boto:path=/vclfcn-10/ 2024-02-01T10:16:52.723 DEBUG:boto:auth_path=/vclfcn-10/ 2024-02-01T10:16:52.723 DEBUG:boto:path=/vclfcn-10/?prefix=%3A 2024-02-01T10:16:52.723 DEBUG:boto:auth_path=/vclfcn-10/?prefix=%3A 2024-02-01T10:16:52.723 DEBUG:boto:Method: GET 2024-02-01T10:16:52.723 DEBUG:boto:Path: /vclfcn-10/?prefix=%3A 2024-02-01T10:16:52.723 DEBUG:boto:Data: 2024-02-01T10:16:52.723 DEBUG:boto:Headers: {} 2024-02-01T10:16:52.723 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:52.724 DEBUG:boto:Port: 8000 2024-02-01T10:16:52.724 DEBUG:boto:Params: {} 2024-02-01T10:16:52.724 DEBUG:boto:Token: None 2024-02-01T10:16:52.724 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:52 GMT /vclfcn-10/ 2024-02-01T10:16:52.724 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Pgkc0sSASjZxVxFnlTuL1OitXOU= 2024-02-01T10:16:52.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Pgkc0sSASjZxVxFnlTuL1OitXOU=', 'Content-Length': '0'} 2024-02-01T10:16:52.729 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f1c705f4bba1e23-0065bb6f94-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:52.776 DEBUG:boto:b'vclfcn-10:1000false:2024-02-01T10:16:36.351Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:16:52.776 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-02-01T10:16:52.777 DEBUG:boto:path=/vclfcn-10/_myobj 2024-02-01T10:16:52.777 DEBUG:boto:auth_path=/vclfcn-10/_myobj 2024-02-01T10:16:52.777 DEBUG:boto:path=/vclfcn-10/_myobj?versionId=null 2024-02-01T10:16:52.777 DEBUG:boto:auth_path=/vclfcn-10/_myobj?versionId=null 2024-02-01T10:16:52.778 DEBUG:boto:Method: GET 2024-02-01T10:16:52.778 DEBUG:boto:Path: /vclfcn-10/_myobj?versionId=null 2024-02-01T10:16:52.778 DEBUG:boto:Data: 2024-02-01T10:16:52.778 DEBUG:boto:Headers: {} 2024-02-01T10:16:52.778 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:16:52.778 DEBUG:boto:Port: 8000 2024-02-01T10:16:52.778 DEBUG:boto:Params: {} 2024-02-01T10:16:52.778 DEBUG:boto:Token: None 2024-02-01T10:16:52.778 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:52 GMT /vclfcn-10/_myobj?versionId=null 2024-02-01T10:16:52.779 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:iDjEA1blCEq9/9ZBWm5EA2BwjoM= 2024-02-01T10:16:52.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iDjEA1blCEq9/9ZBWm5EA2BwjoM=', 'Content-Length': '0'} 2024-02-01T10:16:52.782 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a11c260a799ccf55-0065bb6f94-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:52.782 DEBUG:boto:path=/vclfcn-10/_myobj 2024-02-01T10:16:52.782 DEBUG:boto:auth_path=/vclfcn-10/_myobj 2024-02-01T10:16:52.783 DEBUG:boto:path=/vclfcn-10/_myobj?versionId=null 2024-02-01T10:16:52.783 DEBUG:boto:auth_path=/vclfcn-10/_myobj?versionId=null 2024-02-01T10:16:52.783 DEBUG:boto:Method: GET 2024-02-01T10:16:52.783 DEBUG:boto:Path: /vclfcn-10/_myobj?versionId=null 2024-02-01T10:16:52.783 DEBUG:boto:Data: 2024-02-01T10:16:52.783 DEBUG:boto:Headers: {} 2024-02-01T10:16:52.783 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:52.783 DEBUG:boto:Port: 8000 2024-02-01T10:16:52.783 DEBUG:boto:Params: {} 2024-02-01T10:16:52.783 DEBUG:boto:Token: None 2024-02-01T10:16:52.783 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:52 GMT /vclfcn-10/_myobj?versionId=null 2024-02-01T10:16:52.784 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:iDjEA1blCEq9/9ZBWm5EA2BwjoM= 2024-02-01T10:16:52.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iDjEA1blCEq9/9ZBWm5EA2BwjoM=', 'Content-Length': '0'} 2024-02-01T10:16:52.787 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000302848f488fae8de-0065bb6f94-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:52.787 DEBUG:boto:path=/vclfcn-10/_myobj 2024-02-01T10:16:52.787 DEBUG:boto:auth_path=/vclfcn-10/_myobj 2024-02-01T10:16:52.787 DEBUG:boto:path=/vclfcn-10/_myobj?versionId=null 2024-02-01T10:16:52.787 DEBUG:boto:auth_path=/vclfcn-10/_myobj?versionId=null 2024-02-01T10:16:52.787 DEBUG:boto:Method: HEAD 2024-02-01T10:16:52.787 DEBUG:boto:Path: /vclfcn-10/_myobj?versionId=null 2024-02-01T10:16:52.788 DEBUG:boto:Data: 2024-02-01T10:16:52.788 DEBUG:boto:Headers: {} 2024-02-01T10:16:52.788 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:16:52.788 DEBUG:boto:Port: 8000 2024-02-01T10:16:52.788 DEBUG:boto:Params: {} 2024-02-01T10:16:52.788 DEBUG:boto:Token: None 2024-02-01T10:16:52.788 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:52 GMT /vclfcn-10/_myobj?versionId=null 2024-02-01T10:16:52.788 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Wws3aec7L6GAnI7EjWKin/IiCNA= 2024-02-01T10:16:52.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Wws3aec7L6GAnI7EjWKin/IiCNA=', 'Content-Length': '0'} 2024-02-01T10:16:52.791 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bb8a5310fe808378-0065bb6f94-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:52.792 DEBUG:boto:path=/vclfcn-10/_myobj 2024-02-01T10:16:52.792 DEBUG:boto:auth_path=/vclfcn-10/_myobj 2024-02-01T10:16:52.792 DEBUG:boto:path=/vclfcn-10/_myobj?versionId=null 2024-02-01T10:16:52.792 DEBUG:boto:auth_path=/vclfcn-10/_myobj?versionId=null 2024-02-01T10:16:52.792 DEBUG:boto:Method: HEAD 2024-02-01T10:16:52.792 DEBUG:boto:Path: /vclfcn-10/_myobj?versionId=null 2024-02-01T10:16:52.792 DEBUG:boto:Data: 2024-02-01T10:16:52.792 DEBUG:boto:Headers: {} 2024-02-01T10:16:52.792 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:52.792 DEBUG:boto:Port: 8000 2024-02-01T10:16:52.792 DEBUG:boto:Params: {} 2024-02-01T10:16:52.793 DEBUG:boto:Token: None 2024-02-01T10:16:52.793 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:52 GMT /vclfcn-10/_myobj?versionId=null 2024-02-01T10:16:52.793 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Wws3aec7L6GAnI7EjWKin/IiCNA= 2024-02-01T10:16:52.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Wws3aec7L6GAnI7EjWKin/IiCNA=', 'Content-Length': '0'} 2024-02-01T10:16:52.795 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000099668e1ed2615ada-0065bb6f94-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:52.796 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-02-01T10:16:52.796 DEBUG:boto:path=/vclfcn-10/_myobj 2024-02-01T10:16:52.796 DEBUG:boto:auth_path=/vclfcn-10/_myobj 2024-02-01T10:16:52.796 DEBUG:boto:path=/vclfcn-10/_myobj?versionId=null 2024-02-01T10:16:52.796 DEBUG:boto:auth_path=/vclfcn-10/_myobj?versionId=null 2024-02-01T10:16:52.796 DEBUG:boto:Method: GET 2024-02-01T10:16:52.796 DEBUG:boto:Path: /vclfcn-10/_myobj?versionId=null 2024-02-01T10:16:52.796 DEBUG:boto:Data: 2024-02-01T10:16:52.797 DEBUG:boto:Headers: {} 2024-02-01T10:16:52.797 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:16:52.797 DEBUG:boto:Port: 8000 2024-02-01T10:16:52.797 DEBUG:boto:Params: {} 2024-02-01T10:16:52.797 DEBUG:boto:Token: None 2024-02-01T10:16:52.797 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:52 GMT /vclfcn-10/_myobj?versionId=null 2024-02-01T10:16:52.797 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:iDjEA1blCEq9/9ZBWm5EA2BwjoM= 2024-02-01T10:16:52.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iDjEA1blCEq9/9ZBWm5EA2BwjoM=', 'Content-Length': '0'} 2024-02-01T10:16:52.800 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005106e4ca73131bbe-0065bb6f94-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:52.800 DEBUG:boto:path=/vclfcn-10/_myobj 2024-02-01T10:16:52.801 DEBUG:boto:auth_path=/vclfcn-10/_myobj 2024-02-01T10:16:52.801 DEBUG:boto:path=/vclfcn-10/_myobj?versionId=null 2024-02-01T10:16:52.801 DEBUG:boto:auth_path=/vclfcn-10/_myobj?versionId=null 2024-02-01T10:16:52.801 DEBUG:boto:Method: GET 2024-02-01T10:16:52.801 DEBUG:boto:Path: /vclfcn-10/_myobj?versionId=null 2024-02-01T10:16:52.801 DEBUG:boto:Data: 2024-02-01T10:16:52.801 DEBUG:boto:Headers: {} 2024-02-01T10:16:52.801 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:52.801 DEBUG:boto:Port: 8000 2024-02-01T10:16:52.802 DEBUG:boto:Params: {} 2024-02-01T10:16:52.802 DEBUG:boto:Token: None 2024-02-01T10:16:52.802 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:52 GMT /vclfcn-10/_myobj?versionId=null 2024-02-01T10:16:52.802 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:iDjEA1blCEq9/9ZBWm5EA2BwjoM= 2024-02-01T10:16:52.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iDjEA1blCEq9/9ZBWm5EA2BwjoM=', 'Content-Length': '0'} 2024-02-01T10:16:52.805 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b11391ae3336e91e-0065bb6f94-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:52.805 DEBUG:boto:path=/vclfcn-10/ 2024-02-01T10:16:52.805 DEBUG:boto:auth_path=/vclfcn-10/ 2024-02-01T10:16:52.805 DEBUG:boto:path=/vclfcn-10/?prefix=_myobj 2024-02-01T10:16:52.805 DEBUG:boto:auth_path=/vclfcn-10/?prefix=_myobj 2024-02-01T10:16:52.805 DEBUG:boto:Method: GET 2024-02-01T10:16:52.805 DEBUG:boto:Path: /vclfcn-10/?prefix=_myobj 2024-02-01T10:16:52.805 DEBUG:boto:Data: 2024-02-01T10:16:52.806 DEBUG:boto:Headers: {} 2024-02-01T10:16:52.806 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:16:52.806 DEBUG:boto:Port: 8000 2024-02-01T10:16:52.806 DEBUG:boto:Params: {} 2024-02-01T10:16:52.806 DEBUG:boto:Token: None 2024-02-01T10:16:52.806 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:52 GMT /vclfcn-10/ 2024-02-01T10:16:52.806 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Pgkc0sSASjZxVxFnlTuL1OitXOU= 2024-02-01T10:16:52.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Pgkc0sSASjZxVxFnlTuL1OitXOU=', 'Content-Length': '0'} 2024-02-01T10:16:52.811 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000815447de8325f204-0065bb6f94-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:52.852 DEBUG:boto:b'vclfcn-10_myobj1000false_myobj2024-02-01T10:16:36.295Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:16:52.853 DEBUG:boto:path=/vclfcn-10/ 2024-02-01T10:16:52.853 DEBUG:boto:auth_path=/vclfcn-10/ 2024-02-01T10:16:52.853 DEBUG:boto:path=/vclfcn-10/?prefix=_myobj 2024-02-01T10:16:52.853 DEBUG:boto:auth_path=/vclfcn-10/?prefix=_myobj 2024-02-01T10:16:52.853 DEBUG:boto:Method: GET 2024-02-01T10:16:52.853 DEBUG:boto:Path: /vclfcn-10/?prefix=_myobj 2024-02-01T10:16:52.854 DEBUG:boto:Data: 2024-02-01T10:16:52.854 DEBUG:boto:Headers: {} 2024-02-01T10:16:52.854 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:52.854 DEBUG:boto:Port: 8000 2024-02-01T10:16:52.854 DEBUG:boto:Params: {} 2024-02-01T10:16:52.854 DEBUG:boto:Token: None 2024-02-01T10:16:52.854 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:52 GMT /vclfcn-10/ 2024-02-01T10:16:52.854 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Pgkc0sSASjZxVxFnlTuL1OitXOU= 2024-02-01T10:16:52.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Pgkc0sSASjZxVxFnlTuL1OitXOU=', 'Content-Length': '0'} 2024-02-01T10:16:52.859 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010b7fcb7fb8187fc-0065bb6f94-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:52.859 DEBUG:boto:b'vclfcn-10_myobj1000false_myobj2024-02-01T10:16:36.295Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:16:52.859 DEBUG:boto:path=/vclfcn-10/_myobj 2024-02-01T10:16:52.859 DEBUG:boto:auth_path=/vclfcn-10/_myobj 2024-02-01T10:16:52.859 DEBUG:boto:Method: HEAD 2024-02-01T10:16:52.859 DEBUG:boto:Path: /vclfcn-10/_myobj 2024-02-01T10:16:52.860 DEBUG:boto:Data: 2024-02-01T10:16:52.860 DEBUG:boto:Headers: {} 2024-02-01T10:16:52.860 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:16:52.860 DEBUG:boto:Port: 8000 2024-02-01T10:16:52.860 DEBUG:boto:Params: {} 2024-02-01T10:16:52.860 DEBUG:boto:Token: None 2024-02-01T10:16:52.860 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:52 GMT /vclfcn-10/_myobj 2024-02-01T10:16:52.860 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+d5bBdkZg5anKs493Itpgbxmtxg= 2024-02-01T10:16:52.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+d5bBdkZg5anKs493Itpgbxmtxg=', 'Content-Length': '0'} 2024-02-01T10:16:52.863 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003db1366f84158d4e-0065bb6f94-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:52.863 DEBUG:boto:path=/vclfcn-10/_myobj 2024-02-01T10:16:52.864 DEBUG:boto:auth_path=/vclfcn-10/_myobj 2024-02-01T10:16:52.864 DEBUG:boto:Method: HEAD 2024-02-01T10:16:52.864 DEBUG:boto:Path: /vclfcn-10/_myobj 2024-02-01T10:16:52.864 DEBUG:boto:Data: 2024-02-01T10:16:52.864 DEBUG:boto:Headers: {} 2024-02-01T10:16:52.864 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:52.864 DEBUG:boto:Port: 8000 2024-02-01T10:16:52.864 DEBUG:boto:Params: {} 2024-02-01T10:16:52.864 DEBUG:boto:Token: None 2024-02-01T10:16:52.864 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:52 GMT /vclfcn-10/_myobj 2024-02-01T10:16:52.864 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+d5bBdkZg5anKs493Itpgbxmtxg= 2024-02-01T10:16:52.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+d5bBdkZg5anKs493Itpgbxmtxg=', 'Content-Length': '0'} 2024-02-01T10:16:52.867 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b814217502f354d3-0065bb6f94-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:52.868 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-02-01T10:16:52.868 DEBUG:boto:path=/vclfcn-10/_myobj 2024-02-01T10:16:52.868 DEBUG:boto:auth_path=/vclfcn-10/_myobj 2024-02-01T10:16:52.868 DEBUG:boto:Method: GET 2024-02-01T10:16:52.868 DEBUG:boto:Path: /vclfcn-10/_myobj 2024-02-01T10:16:52.868 DEBUG:boto:Data: 2024-02-01T10:16:52.868 DEBUG:boto:Headers: {} 2024-02-01T10:16:52.868 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:16:52.868 DEBUG:boto:Port: 8000 2024-02-01T10:16:52.868 DEBUG:boto:Params: {} 2024-02-01T10:16:52.869 DEBUG:boto:Token: None 2024-02-01T10:16:52.869 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:52 GMT /vclfcn-10/_myobj 2024-02-01T10:16:52.869 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cgC9woHfrtQzz/3+6wbGGgXRk/0= 2024-02-01T10:16:52.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cgC9woHfrtQzz/3+6wbGGgXRk/0=', 'Content-Length': '0'} 2024-02-01T10:16:52.872 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007b4868de3099f284-0065bb6f94-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:52.872 DEBUG:boto:path=/vclfcn-10/_myobj 2024-02-01T10:16:52.872 DEBUG:boto:auth_path=/vclfcn-10/_myobj 2024-02-01T10:16:52.872 DEBUG:boto:Method: GET 2024-02-01T10:16:52.873 DEBUG:boto:Path: /vclfcn-10/_myobj 2024-02-01T10:16:52.873 DEBUG:boto:Data: 2024-02-01T10:16:52.873 DEBUG:boto:Headers: {} 2024-02-01T10:16:52.873 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:52.873 DEBUG:boto:Port: 8000 2024-02-01T10:16:52.873 DEBUG:boto:Params: {} 2024-02-01T10:16:52.873 DEBUG:boto:Token: None 2024-02-01T10:16:52.873 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:52 GMT /vclfcn-10/_myobj 2024-02-01T10:16:52.873 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cgC9woHfrtQzz/3+6wbGGgXRk/0= 2024-02-01T10:16:52.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cgC9woHfrtQzz/3+6wbGGgXRk/0=', 'Content-Length': '0'} 2024-02-01T10:16:52.876 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000438534942e6a3aa2-0065bb6f94-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:52.877 DEBUG:boto:path=/vclfcn-10/ 2024-02-01T10:16:52.877 DEBUG:boto:auth_path=/vclfcn-10/ 2024-02-01T10:16:52.877 DEBUG:boto:path=/vclfcn-10/?prefix=_myobj 2024-02-01T10:16:52.877 DEBUG:boto:auth_path=/vclfcn-10/?prefix=_myobj 2024-02-01T10:16:52.877 DEBUG:boto:Method: GET 2024-02-01T10:16:52.877 DEBUG:boto:Path: /vclfcn-10/?prefix=_myobj 2024-02-01T10:16:52.877 DEBUG:boto:Data: 2024-02-01T10:16:52.877 DEBUG:boto:Headers: {} 2024-02-01T10:16:52.877 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:16:52.877 DEBUG:boto:Port: 8000 2024-02-01T10:16:52.878 DEBUG:boto:Params: {} 2024-02-01T10:16:52.878 DEBUG:boto:Token: None 2024-02-01T10:16:52.878 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:52 GMT /vclfcn-10/ 2024-02-01T10:16:52.878 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Pgkc0sSASjZxVxFnlTuL1OitXOU= 2024-02-01T10:16:52.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Pgkc0sSASjZxVxFnlTuL1OitXOU=', 'Content-Length': '0'} 2024-02-01T10:16:52.882 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb789412c4a40883-0065bb6f94-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:52.882 DEBUG:boto:b'vclfcn-10_myobj1000false_myobj2024-02-01T10:16:36.295Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:16:52.883 DEBUG:boto:path=/vclfcn-10/ 2024-02-01T10:16:52.883 DEBUG:boto:auth_path=/vclfcn-10/ 2024-02-01T10:16:52.883 DEBUG:boto:path=/vclfcn-10/?prefix=_myobj 2024-02-01T10:16:52.883 DEBUG:boto:auth_path=/vclfcn-10/?prefix=_myobj 2024-02-01T10:16:52.883 DEBUG:boto:Method: GET 2024-02-01T10:16:52.883 DEBUG:boto:Path: /vclfcn-10/?prefix=_myobj 2024-02-01T10:16:52.883 DEBUG:boto:Data: 2024-02-01T10:16:52.884 DEBUG:boto:Headers: {} 2024-02-01T10:16:52.884 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:52.884 DEBUG:boto:Port: 8000 2024-02-01T10:16:52.884 DEBUG:boto:Params: {} 2024-02-01T10:16:52.884 DEBUG:boto:Token: None 2024-02-01T10:16:52.884 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:52 GMT /vclfcn-10/ 2024-02-01T10:16:52.884 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Pgkc0sSASjZxVxFnlTuL1OitXOU= 2024-02-01T10:16:52.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Pgkc0sSASjZxVxFnlTuL1OitXOU=', 'Content-Length': '0'} 2024-02-01T10:16:52.888 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5c376aab03ce820-0065bb6f94-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:52.932 DEBUG:boto:b'vclfcn-10_myobj1000false_myobj2024-02-01T10:16:36.295Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:16:52.933 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-02-01T10:16:52.933 DEBUG:boto:path=/vclfcn-10/myobj 2024-02-01T10:16:52.933 DEBUG:boto:auth_path=/vclfcn-10/myobj 2024-02-01T10:16:52.933 DEBUG:boto:path=/vclfcn-10/myobj?versionId=null 2024-02-01T10:16:52.933 DEBUG:boto:auth_path=/vclfcn-10/myobj?versionId=null 2024-02-01T10:16:52.933 DEBUG:boto:Method: GET 2024-02-01T10:16:52.933 DEBUG:boto:Path: /vclfcn-10/myobj?versionId=null 2024-02-01T10:16:52.933 DEBUG:boto:Data: 2024-02-01T10:16:52.933 DEBUG:boto:Headers: {} 2024-02-01T10:16:52.934 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:16:52.934 DEBUG:boto:Port: 8000 2024-02-01T10:16:52.934 DEBUG:boto:Params: {} 2024-02-01T10:16:52.934 DEBUG:boto:Token: None 2024-02-01T10:16:52.934 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:52 GMT /vclfcn-10/myobj?versionId=null 2024-02-01T10:16:52.934 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:SlIMT7cS1rhumzyLA81QlU4+2iE= 2024-02-01T10:16:52.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:SlIMT7cS1rhumzyLA81QlU4+2iE=', 'Content-Length': '0'} 2024-02-01T10:16:52.937 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000550f9ce393cd221b-0065bb6f94-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:52.938 DEBUG:boto:path=/vclfcn-10/myobj 2024-02-01T10:16:52.938 DEBUG:boto:auth_path=/vclfcn-10/myobj 2024-02-01T10:16:52.938 DEBUG:boto:path=/vclfcn-10/myobj?versionId=null 2024-02-01T10:16:52.939 DEBUG:boto:auth_path=/vclfcn-10/myobj?versionId=null 2024-02-01T10:16:52.939 DEBUG:boto:Method: GET 2024-02-01T10:16:52.939 DEBUG:boto:Path: /vclfcn-10/myobj?versionId=null 2024-02-01T10:16:52.939 DEBUG:boto:Data: 2024-02-01T10:16:52.939 DEBUG:boto:Headers: {} 2024-02-01T10:16:52.940 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:52.940 DEBUG:boto:Port: 8000 2024-02-01T10:16:52.940 DEBUG:boto:Params: {} 2024-02-01T10:16:52.940 DEBUG:boto:Token: None 2024-02-01T10:16:52.941 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:52 GMT /vclfcn-10/myobj?versionId=null 2024-02-01T10:16:52.941 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:SlIMT7cS1rhumzyLA81QlU4+2iE= 2024-02-01T10:16:52.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:SlIMT7cS1rhumzyLA81QlU4+2iE=', 'Content-Length': '0'} 2024-02-01T10:16:52.944 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000043924298605c53d7-0065bb6f94-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:52.944 DEBUG:boto:path=/vclfcn-10/myobj 2024-02-01T10:16:52.945 DEBUG:boto:auth_path=/vclfcn-10/myobj 2024-02-01T10:16:52.945 DEBUG:boto:path=/vclfcn-10/myobj?versionId=null 2024-02-01T10:16:52.945 DEBUG:boto:auth_path=/vclfcn-10/myobj?versionId=null 2024-02-01T10:16:52.945 DEBUG:boto:Method: HEAD 2024-02-01T10:16:52.946 DEBUG:boto:Path: /vclfcn-10/myobj?versionId=null 2024-02-01T10:16:52.946 DEBUG:boto:Data: 2024-02-01T10:16:52.946 DEBUG:boto:Headers: {} 2024-02-01T10:16:52.946 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:16:52.946 DEBUG:boto:Port: 8000 2024-02-01T10:16:52.947 DEBUG:boto:Params: {} 2024-02-01T10:16:52.947 DEBUG:boto:Token: None 2024-02-01T10:16:52.947 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:52 GMT /vclfcn-10/myobj?versionId=null 2024-02-01T10:16:52.947 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:prOwKF/f8C3R0NSIlZ7H/mdGFYU= 2024-02-01T10:16:52.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:prOwKF/f8C3R0NSIlZ7H/mdGFYU=', 'Content-Length': '0'} 2024-02-01T10:16:52.950 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bd3271178076868c-0065bb6f94-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:52.951 DEBUG:boto:path=/vclfcn-10/myobj 2024-02-01T10:16:52.951 DEBUG:boto:auth_path=/vclfcn-10/myobj 2024-02-01T10:16:52.951 DEBUG:boto:path=/vclfcn-10/myobj?versionId=null 2024-02-01T10:16:52.952 DEBUG:boto:auth_path=/vclfcn-10/myobj?versionId=null 2024-02-01T10:16:52.952 DEBUG:boto:Method: HEAD 2024-02-01T10:16:52.952 DEBUG:boto:Path: /vclfcn-10/myobj?versionId=null 2024-02-01T10:16:52.952 DEBUG:boto:Data: 2024-02-01T10:16:52.952 DEBUG:boto:Headers: {} 2024-02-01T10:16:52.953 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:52.953 DEBUG:boto:Port: 8000 2024-02-01T10:16:52.953 DEBUG:boto:Params: {} 2024-02-01T10:16:52.953 DEBUG:boto:Token: None 2024-02-01T10:16:52.954 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:52 GMT /vclfcn-10/myobj?versionId=null 2024-02-01T10:16:52.954 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:prOwKF/f8C3R0NSIlZ7H/mdGFYU= 2024-02-01T10:16:52.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:prOwKF/f8C3R0NSIlZ7H/mdGFYU=', 'Content-Length': '0'} 2024-02-01T10:16:52.957 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004d4ebdc623e08320-0065bb6f94-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:52.957 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-02-01T10:16:52.957 DEBUG:boto:path=/vclfcn-10/myobj 2024-02-01T10:16:52.958 DEBUG:boto:auth_path=/vclfcn-10/myobj 2024-02-01T10:16:52.958 DEBUG:boto:path=/vclfcn-10/myobj?versionId=null 2024-02-01T10:16:52.958 DEBUG:boto:auth_path=/vclfcn-10/myobj?versionId=null 2024-02-01T10:16:52.958 DEBUG:boto:Method: GET 2024-02-01T10:16:52.958 DEBUG:boto:Path: /vclfcn-10/myobj?versionId=null 2024-02-01T10:16:52.958 DEBUG:boto:Data: 2024-02-01T10:16:52.958 DEBUG:boto:Headers: {} 2024-02-01T10:16:52.958 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:16:52.958 DEBUG:boto:Port: 8000 2024-02-01T10:16:52.958 DEBUG:boto:Params: {} 2024-02-01T10:16:52.958 DEBUG:boto:Token: None 2024-02-01T10:16:52.959 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:52 GMT /vclfcn-10/myobj?versionId=null 2024-02-01T10:16:52.959 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:SlIMT7cS1rhumzyLA81QlU4+2iE= 2024-02-01T10:16:52.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:SlIMT7cS1rhumzyLA81QlU4+2iE=', 'Content-Length': '0'} 2024-02-01T10:16:52.963 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004f6f2bd4d99202a5-0065bb6f94-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:53.004 DEBUG:boto:path=/vclfcn-10/myobj 2024-02-01T10:16:53.004 DEBUG:boto:auth_path=/vclfcn-10/myobj 2024-02-01T10:16:53.004 DEBUG:boto:path=/vclfcn-10/myobj?versionId=null 2024-02-01T10:16:53.004 DEBUG:boto:auth_path=/vclfcn-10/myobj?versionId=null 2024-02-01T10:16:53.005 DEBUG:boto:Method: GET 2024-02-01T10:16:53.005 DEBUG:boto:Path: /vclfcn-10/myobj?versionId=null 2024-02-01T10:16:53.005 DEBUG:boto:Data: 2024-02-01T10:16:53.005 DEBUG:boto:Headers: {} 2024-02-01T10:16:53.005 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:53.005 DEBUG:boto:Port: 8000 2024-02-01T10:16:53.005 DEBUG:boto:Params: {} 2024-02-01T10:16:53.005 DEBUG:boto:Token: None 2024-02-01T10:16:53.005 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:53 GMT /vclfcn-10/myobj?versionId=null 2024-02-01T10:16:53.006 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sxKU9eN8gpJoOnexBpKq9gsTXLo= 2024-02-01T10:16:53.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sxKU9eN8gpJoOnexBpKq9gsTXLo=', 'Content-Length': '0'} 2024-02-01T10:16:53.009 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c331c986e6876d15-0065bb6f95-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:53.009 DEBUG:boto:path=/vclfcn-10/ 2024-02-01T10:16:53.009 DEBUG:boto:auth_path=/vclfcn-10/ 2024-02-01T10:16:53.009 DEBUG:boto:path=/vclfcn-10/?prefix=myobj 2024-02-01T10:16:53.009 DEBUG:boto:auth_path=/vclfcn-10/?prefix=myobj 2024-02-01T10:16:53.009 DEBUG:boto:Method: GET 2024-02-01T10:16:53.010 DEBUG:boto:Path: /vclfcn-10/?prefix=myobj 2024-02-01T10:16:53.010 DEBUG:boto:Data: 2024-02-01T10:16:53.010 DEBUG:boto:Headers: {} 2024-02-01T10:16:53.010 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:16:53.010 DEBUG:boto:Port: 8000 2024-02-01T10:16:53.010 DEBUG:boto:Params: {} 2024-02-01T10:16:53.010 DEBUG:boto:Token: None 2024-02-01T10:16:53.010 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:53 GMT /vclfcn-10/ 2024-02-01T10:16:53.010 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gxA919U/ZvYiLbGrFtIj/OtV1Jg= 2024-02-01T10:16:53.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gxA919U/ZvYiLbGrFtIj/OtV1Jg=', 'Content-Length': '0'} 2024-02-01T10:16:53.015 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009fe12c3670b91aca-0065bb6f95-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:53.015 DEBUG:boto:b'vclfcn-10myobj1000falsemyobj2024-02-01T10:16:36.267Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:16:53.016 DEBUG:boto:path=/vclfcn-10/ 2024-02-01T10:16:53.016 DEBUG:boto:auth_path=/vclfcn-10/ 2024-02-01T10:16:53.016 DEBUG:boto:path=/vclfcn-10/?prefix=myobj 2024-02-01T10:16:53.016 DEBUG:boto:auth_path=/vclfcn-10/?prefix=myobj 2024-02-01T10:16:53.016 DEBUG:boto:Method: GET 2024-02-01T10:16:53.016 DEBUG:boto:Path: /vclfcn-10/?prefix=myobj 2024-02-01T10:16:53.016 DEBUG:boto:Data: 2024-02-01T10:16:53.016 DEBUG:boto:Headers: {} 2024-02-01T10:16:53.016 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:53.016 DEBUG:boto:Port: 8000 2024-02-01T10:16:53.016 DEBUG:boto:Params: {} 2024-02-01T10:16:53.017 DEBUG:boto:Token: None 2024-02-01T10:16:53.017 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:53 GMT /vclfcn-10/ 2024-02-01T10:16:53.017 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gxA919U/ZvYiLbGrFtIj/OtV1Jg= 2024-02-01T10:16:53.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gxA919U/ZvYiLbGrFtIj/OtV1Jg=', 'Content-Length': '0'} 2024-02-01T10:16:53.021 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a614319bd73ba0fd-0065bb6f95-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:53.064 DEBUG:boto:b'vclfcn-10myobj1000falsemyobj2024-02-01T10:16:36.267Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:16:53.065 DEBUG:boto:path=/vclfcn-10/myobj 2024-02-01T10:16:53.065 DEBUG:boto:auth_path=/vclfcn-10/myobj 2024-02-01T10:16:53.065 DEBUG:boto:Method: HEAD 2024-02-01T10:16:53.065 DEBUG:boto:Path: /vclfcn-10/myobj 2024-02-01T10:16:53.065 DEBUG:boto:Data: 2024-02-01T10:16:53.065 DEBUG:boto:Headers: {} 2024-02-01T10:16:53.065 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:16:53.065 DEBUG:boto:Port: 8000 2024-02-01T10:16:53.065 DEBUG:boto:Params: {} 2024-02-01T10:16:53.065 DEBUG:boto:Token: None 2024-02-01T10:16:53.066 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:53 GMT /vclfcn-10/myobj 2024-02-01T10:16:53.066 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1TwCrXD/lVlIYOOv1d4zYH3BVHQ= 2024-02-01T10:16:53.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1TwCrXD/lVlIYOOv1d4zYH3BVHQ=', 'Content-Length': '0'} 2024-02-01T10:16:53.068 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bad5bf2e62a0dbc6-0065bb6f95-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:53.069 DEBUG:boto:path=/vclfcn-10/myobj 2024-02-01T10:16:53.069 DEBUG:boto:auth_path=/vclfcn-10/myobj 2024-02-01T10:16:53.069 DEBUG:boto:Method: HEAD 2024-02-01T10:16:53.069 DEBUG:boto:Path: /vclfcn-10/myobj 2024-02-01T10:16:53.069 DEBUG:boto:Data: 2024-02-01T10:16:53.069 DEBUG:boto:Headers: {} 2024-02-01T10:16:53.069 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:53.069 DEBUG:boto:Port: 8000 2024-02-01T10:16:53.070 DEBUG:boto:Params: {} 2024-02-01T10:16:53.070 DEBUG:boto:Token: None 2024-02-01T10:16:53.070 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:53 GMT /vclfcn-10/myobj 2024-02-01T10:16:53.070 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1TwCrXD/lVlIYOOv1d4zYH3BVHQ= 2024-02-01T10:16:53.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1TwCrXD/lVlIYOOv1d4zYH3BVHQ=', 'Content-Length': '0'} 2024-02-01T10:16:53.073 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000beb39aef08362888-0065bb6f95-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:53.074 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-02-01T10:16:53.074 DEBUG:boto:path=/vclfcn-10/myobj 2024-02-01T10:16:53.074 DEBUG:boto:auth_path=/vclfcn-10/myobj 2024-02-01T10:16:53.074 DEBUG:boto:Method: GET 2024-02-01T10:16:53.075 DEBUG:boto:Path: /vclfcn-10/myobj 2024-02-01T10:16:53.075 DEBUG:boto:Data: 2024-02-01T10:16:53.075 DEBUG:boto:Headers: {} 2024-02-01T10:16:53.075 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:16:53.075 DEBUG:boto:Port: 8000 2024-02-01T10:16:53.075 DEBUG:boto:Params: {} 2024-02-01T10:16:53.075 DEBUG:boto:Token: None 2024-02-01T10:16:53.076 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:53 GMT /vclfcn-10/myobj 2024-02-01T10:16:53.076 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:dpRgrQgzVUz5DuHNFRKfvTiOsBg= 2024-02-01T10:16:53.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dpRgrQgzVUz5DuHNFRKfvTiOsBg=', 'Content-Length': '0'} 2024-02-01T10:16:53.080 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000756540ebbf7f48e1-0065bb6f95-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:53.124 DEBUG:boto:path=/vclfcn-10/myobj 2024-02-01T10:16:53.125 DEBUG:boto:auth_path=/vclfcn-10/myobj 2024-02-01T10:16:53.125 DEBUG:boto:Method: GET 2024-02-01T10:16:53.125 DEBUG:boto:Path: /vclfcn-10/myobj 2024-02-01T10:16:53.125 DEBUG:boto:Data: 2024-02-01T10:16:53.125 DEBUG:boto:Headers: {} 2024-02-01T10:16:53.125 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:53.125 DEBUG:boto:Port: 8000 2024-02-01T10:16:53.125 DEBUG:boto:Params: {} 2024-02-01T10:16:53.126 DEBUG:boto:Token: None 2024-02-01T10:16:53.126 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:53 GMT /vclfcn-10/myobj 2024-02-01T10:16:53.126 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:dpRgrQgzVUz5DuHNFRKfvTiOsBg= 2024-02-01T10:16:53.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dpRgrQgzVUz5DuHNFRKfvTiOsBg=', 'Content-Length': '0'} 2024-02-01T10:16:53.130 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bfcd0df05f6aeaf2-0065bb6f95-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:53.130 DEBUG:boto:path=/vclfcn-10/ 2024-02-01T10:16:53.130 DEBUG:boto:auth_path=/vclfcn-10/ 2024-02-01T10:16:53.131 DEBUG:boto:path=/vclfcn-10/?prefix=myobj 2024-02-01T10:16:53.131 DEBUG:boto:auth_path=/vclfcn-10/?prefix=myobj 2024-02-01T10:16:53.131 DEBUG:boto:Method: GET 2024-02-01T10:16:53.131 DEBUG:boto:Path: /vclfcn-10/?prefix=myobj 2024-02-01T10:16:53.131 DEBUG:boto:Data: 2024-02-01T10:16:53.131 DEBUG:boto:Headers: {} 2024-02-01T10:16:53.131 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:16:53.131 DEBUG:boto:Port: 8000 2024-02-01T10:16:53.132 DEBUG:boto:Params: {} 2024-02-01T10:16:53.132 DEBUG:boto:Token: None 2024-02-01T10:16:53.132 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:53 GMT /vclfcn-10/ 2024-02-01T10:16:53.132 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gxA919U/ZvYiLbGrFtIj/OtV1Jg= 2024-02-01T10:16:53.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gxA919U/ZvYiLbGrFtIj/OtV1Jg=', 'Content-Length': '0'} 2024-02-01T10:16:53.136 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f780e7c801e227d2-0065bb6f95-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:53.137 DEBUG:boto:b'vclfcn-10myobj1000falsemyobj2024-02-01T10:16:36.267Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:16:53.138 DEBUG:boto:path=/vclfcn-10/ 2024-02-01T10:16:53.138 DEBUG:boto:auth_path=/vclfcn-10/ 2024-02-01T10:16:53.138 DEBUG:boto:path=/vclfcn-10/?prefix=myobj 2024-02-01T10:16:53.138 DEBUG:boto:auth_path=/vclfcn-10/?prefix=myobj 2024-02-01T10:16:53.138 DEBUG:boto:Method: GET 2024-02-01T10:16:53.138 DEBUG:boto:Path: /vclfcn-10/?prefix=myobj 2024-02-01T10:16:53.139 DEBUG:boto:Data: 2024-02-01T10:16:53.139 DEBUG:boto:Headers: {} 2024-02-01T10:16:53.139 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:53.139 DEBUG:boto:Port: 8000 2024-02-01T10:16:53.139 DEBUG:boto:Params: {} 2024-02-01T10:16:53.140 DEBUG:boto:Token: None 2024-02-01T10:16:53.140 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:53 GMT /vclfcn-10/ 2024-02-01T10:16:53.140 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gxA919U/ZvYiLbGrFtIj/OtV1Jg= 2024-02-01T10:16:53.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gxA919U/ZvYiLbGrFtIj/OtV1Jg=', 'Content-Length': '0'} 2024-02-01T10:16:53.145 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb941c7203615c8c-0065bb6f95-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:53.146 DEBUG:boto:b'vclfcn-10myobj1000falsemyobj2024-02-01T10:16:36.267Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:16:53.146 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-10 zones={test-zone2, test-zone1} 2024-02-01T10:16:53.147 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:16:53.147 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:16:53.147 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-10 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:16:53.199 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:16:53.199 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:16:53.261 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:16:53.260+0000 7fc64cde0980 1 bucket sync caught up with source: 2024-02-01T10:16:53.261 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, 00000000001.13.6, , 00000000001.18.6, , , , 00000000001.20.6, 00000000001.7.6, , ] 2024-02-01T10:16:53.261 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, 00000000001.13.6, , 00000000001.18.6, , , , 00000000001.20.6, 00000000001.7.6, , ] 2024-02-01T10:16:53.261 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:16:53.260+0000 7fc64cde0980 0 bucket checkpoint complete 2024-02-01T10:16:53.282 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-10 zones={test-zone3, test-zone1} 2024-02-01T10:16:53.282 DEBUG:boto:path=/vclfcn-10/ 2024-02-01T10:16:53.282 DEBUG:boto:auth_path=/vclfcn-10/ 2024-02-01T10:16:53.282 DEBUG:boto:Method: HEAD 2024-02-01T10:16:53.283 DEBUG:boto:Path: /vclfcn-10/ 2024-02-01T10:16:53.283 DEBUG:boto:Data: 2024-02-01T10:16:53.283 DEBUG:boto:Headers: {} 2024-02-01T10:16:53.283 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:16:53.283 DEBUG:boto:Port: 8001 2024-02-01T10:16:53.283 DEBUG:boto:Params: {} 2024-02-01T10:16:53.283 DEBUG:boto:Token: None 2024-02-01T10:16:53.283 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:53 GMT /vclfcn-10/ 2024-02-01T10:16:53.284 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RJZ+QVCYi1pawTTjVNQ3rWL5K94= 2024-02-01T10:16:53.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RJZ+QVCYi1pawTTjVNQ3rWL5K94=', 'Content-Length': '0'} 2024-02-01T10:16:53.286 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d52422689a10d6d2-0065bb6f95-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:16:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:53.286 DEBUG:boto:path=/vclfcn-10/ 2024-02-01T10:16:53.286 DEBUG:boto:auth_path=/vclfcn-10/ 2024-02-01T10:16:53.287 DEBUG:boto:Method: HEAD 2024-02-01T10:16:53.287 DEBUG:boto:Path: /vclfcn-10/ 2024-02-01T10:16:53.288 DEBUG:boto:Data: 2024-02-01T10:16:53.288 DEBUG:boto:Headers: {} 2024-02-01T10:16:53.288 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:53.288 DEBUG:boto:Port: 8000 2024-02-01T10:16:53.288 DEBUG:boto:Params: {} 2024-02-01T10:16:53.289 DEBUG:boto:Token: None 2024-02-01T10:16:53.289 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:53 GMT /vclfcn-10/ 2024-02-01T10:16:53.289 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RJZ+QVCYi1pawTTjVNQ3rWL5K94= 2024-02-01T10:16:53.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RJZ+QVCYi1pawTTjVNQ3rWL5K94=', 'Content-Length': '0'} 2024-02-01T10:16:53.291 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000934c354d93507388-0065bb6f95-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:16:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:53.292 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:16:53.292 DEBUG:boto:path=/vclfcn-10/ 2024-02-01T10:16:53.292 DEBUG:boto:auth_path=/vclfcn-10/ 2024-02-01T10:16:53.292 DEBUG:boto:path=/vclfcn-10/?versions&max-keys=999 2024-02-01T10:16:53.292 DEBUG:boto:auth_path=/vclfcn-10/?versions&max-keys=999 2024-02-01T10:16:53.292 DEBUG:boto:Method: GET 2024-02-01T10:16:53.292 DEBUG:boto:Path: /vclfcn-10/?versions&max-keys=999 2024-02-01T10:16:53.292 DEBUG:boto:Data: 2024-02-01T10:16:53.293 DEBUG:boto:Headers: {} 2024-02-01T10:16:53.293 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:16:53.293 DEBUG:boto:Port: 8001 2024-02-01T10:16:53.293 DEBUG:boto:Params: {} 2024-02-01T10:16:53.293 DEBUG:boto:Token: None 2024-02-01T10:16:53.293 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:53 GMT /vclfcn-10/?versions 2024-02-01T10:16:53.293 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:xyHBi4bmgiwZUEWQQDbdeddLj7o= 2024-02-01T10:16:53.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:xyHBi4bmgiwZUEWQQDbdeddLj7o=', 'Content-Length': '0'} 2024-02-01T10:16:53.296 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec9b91a20081c3a9-0065bb6f95-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:53.296 DEBUG:boto:b'vclfcn-10999false&nulltrue2024-02-01T10:16:36.379Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-02-01T10:16:36.351Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-02-01T10:16:36.295Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-02-01T10:16:36.267Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:16:53.297 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-02-01T10:16:53.297 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-02-01T10:16:53.297 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-02-01T10:16:53.297 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-02-01T10:16:53.297 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:16:53.297 DEBUG:boto:path=/vclfcn-10/ 2024-02-01T10:16:53.297 DEBUG:boto:auth_path=/vclfcn-10/ 2024-02-01T10:16:53.297 DEBUG:boto:path=/vclfcn-10/?versions&max-keys=999 2024-02-01T10:16:53.297 DEBUG:boto:auth_path=/vclfcn-10/?versions&max-keys=999 2024-02-01T10:16:53.297 DEBUG:boto:Method: GET 2024-02-01T10:16:53.298 DEBUG:boto:Path: /vclfcn-10/?versions&max-keys=999 2024-02-01T10:16:53.298 DEBUG:boto:Data: 2024-02-01T10:16:53.298 DEBUG:boto:Headers: {} 2024-02-01T10:16:53.298 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:53.298 DEBUG:boto:Port: 8000 2024-02-01T10:16:53.298 DEBUG:boto:Params: {} 2024-02-01T10:16:53.298 DEBUG:boto:Token: None 2024-02-01T10:16:53.298 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:53 GMT /vclfcn-10/?versions 2024-02-01T10:16:53.298 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:xyHBi4bmgiwZUEWQQDbdeddLj7o= 2024-02-01T10:16:53.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:xyHBi4bmgiwZUEWQQDbdeddLj7o=', 'Content-Length': '0'} 2024-02-01T10:16:53.301 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c66bebbd94d7794b-0065bb6f95-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:53.301 DEBUG:boto:b'vclfcn-10999false&nulltrue2024-02-01T10:16:36.379Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-02-01T10:16:36.351Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-02-01T10:16:36.295Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-02-01T10:16:36.267Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:16:53.301 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-02-01T10:16:53.301 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-02-01T10:16:53.301 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-02-01T10:16:53.301 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-02-01T10:16:53.302 DEBUG:boto:path=/vclfcn-10/ 2024-02-01T10:16:53.302 DEBUG:boto:auth_path=/vclfcn-10/ 2024-02-01T10:16:53.302 DEBUG:boto:path=/vclfcn-10/?versions&max-keys=999 2024-02-01T10:16:53.302 DEBUG:boto:auth_path=/vclfcn-10/?versions&max-keys=999 2024-02-01T10:16:53.302 DEBUG:boto:Method: GET 2024-02-01T10:16:53.302 DEBUG:boto:Path: /vclfcn-10/?versions&max-keys=999 2024-02-01T10:16:53.302 DEBUG:boto:Data: 2024-02-01T10:16:53.302 DEBUG:boto:Headers: {} 2024-02-01T10:16:53.302 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:16:53.302 DEBUG:boto:Port: 8001 2024-02-01T10:16:53.303 DEBUG:boto:Params: {} 2024-02-01T10:16:53.303 DEBUG:boto:Token: None 2024-02-01T10:16:53.303 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:53 GMT /vclfcn-10/?versions 2024-02-01T10:16:53.303 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:xyHBi4bmgiwZUEWQQDbdeddLj7o= 2024-02-01T10:16:53.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:xyHBi4bmgiwZUEWQQDbdeddLj7o=', 'Content-Length': '0'} 2024-02-01T10:16:53.305 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000006951d00b4f1ec0b-0065bb6f95-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:53.305 DEBUG:boto:b'vclfcn-10999false&nulltrue2024-02-01T10:16:36.379Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-02-01T10:16:36.351Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-02-01T10:16:36.295Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-02-01T10:16:36.267Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:16:53.306 DEBUG:boto:path=/vclfcn-10/ 2024-02-01T10:16:53.306 DEBUG:boto:auth_path=/vclfcn-10/ 2024-02-01T10:16:53.306 DEBUG:boto:path=/vclfcn-10/?versions&max-keys=999 2024-02-01T10:16:53.306 DEBUG:boto:auth_path=/vclfcn-10/?versions&max-keys=999 2024-02-01T10:16:53.306 DEBUG:boto:Method: GET 2024-02-01T10:16:53.307 DEBUG:boto:Path: /vclfcn-10/?versions&max-keys=999 2024-02-01T10:16:53.307 DEBUG:boto:Data: 2024-02-01T10:16:53.307 DEBUG:boto:Headers: {} 2024-02-01T10:16:53.307 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:53.307 DEBUG:boto:Port: 8000 2024-02-01T10:16:53.307 DEBUG:boto:Params: {} 2024-02-01T10:16:53.307 DEBUG:boto:Token: None 2024-02-01T10:16:53.307 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:53 GMT /vclfcn-10/?versions 2024-02-01T10:16:53.308 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:xyHBi4bmgiwZUEWQQDbdeddLj7o= 2024-02-01T10:16:53.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:xyHBi4bmgiwZUEWQQDbdeddLj7o=', 'Content-Length': '0'} 2024-02-01T10:16:53.310 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000001d69070d3ef958d-0065bb6f95-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:53.310 DEBUG:boto:b'vclfcn-10999false&nulltrue2024-02-01T10:16:36.379Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-02-01T10:16:36.351Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-02-01T10:16:36.295Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-02-01T10:16:36.267Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:16:53.311 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-02-01T10:16:53.311 DEBUG:boto:path=/vclfcn-10/%26 2024-02-01T10:16:53.311 DEBUG:boto:auth_path=/vclfcn-10/%26 2024-02-01T10:16:53.311 DEBUG:boto:path=/vclfcn-10/%26?versionId=null 2024-02-01T10:16:53.311 DEBUG:boto:auth_path=/vclfcn-10/%26?versionId=null 2024-02-01T10:16:53.311 DEBUG:boto:Method: GET 2024-02-01T10:16:53.311 DEBUG:boto:Path: /vclfcn-10/%26?versionId=null 2024-02-01T10:16:53.311 DEBUG:boto:Data: 2024-02-01T10:16:53.311 DEBUG:boto:Headers: {} 2024-02-01T10:16:53.312 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:16:53.312 DEBUG:boto:Port: 8001 2024-02-01T10:16:53.312 DEBUG:boto:Params: {} 2024-02-01T10:16:53.312 DEBUG:boto:Token: None 2024-02-01T10:16:53.312 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:53 GMT /vclfcn-10/%26?versionId=null 2024-02-01T10:16:53.312 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:2b6Rpns2G/TIPWH6STsN2wFuQNg= 2024-02-01T10:16:53.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2b6Rpns2G/TIPWH6STsN2wFuQNg=', 'Content-Length': '0'} 2024-02-01T10:16:53.314 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f792b16ca1359ef1-0065bb6f95-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:53.364 DEBUG:boto:path=/vclfcn-10/%26 2024-02-01T10:16:53.364 DEBUG:boto:auth_path=/vclfcn-10/%26 2024-02-01T10:16:53.364 DEBUG:boto:path=/vclfcn-10/%26?versionId=null 2024-02-01T10:16:53.364 DEBUG:boto:auth_path=/vclfcn-10/%26?versionId=null 2024-02-01T10:16:53.364 DEBUG:boto:Method: GET 2024-02-01T10:16:53.364 DEBUG:boto:Path: /vclfcn-10/%26?versionId=null 2024-02-01T10:16:53.364 DEBUG:boto:Data: 2024-02-01T10:16:53.364 DEBUG:boto:Headers: {} 2024-02-01T10:16:53.365 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:53.365 DEBUG:boto:Port: 8000 2024-02-01T10:16:53.365 DEBUG:boto:Params: {} 2024-02-01T10:16:53.365 DEBUG:boto:Token: None 2024-02-01T10:16:53.365 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:53 GMT /vclfcn-10/%26?versionId=null 2024-02-01T10:16:53.365 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:2b6Rpns2G/TIPWH6STsN2wFuQNg= 2024-02-01T10:16:53.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2b6Rpns2G/TIPWH6STsN2wFuQNg=', 'Content-Length': '0'} 2024-02-01T10:16:53.368 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000944477bde5133a8c-0065bb6f95-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:53.368 DEBUG:boto:path=/vclfcn-10/%26 2024-02-01T10:16:53.368 DEBUG:boto:auth_path=/vclfcn-10/%26 2024-02-01T10:16:53.369 DEBUG:boto:path=/vclfcn-10/%26?versionId=null 2024-02-01T10:16:53.369 DEBUG:boto:auth_path=/vclfcn-10/%26?versionId=null 2024-02-01T10:16:53.369 DEBUG:boto:Method: HEAD 2024-02-01T10:16:53.369 DEBUG:boto:Path: /vclfcn-10/%26?versionId=null 2024-02-01T10:16:53.369 DEBUG:boto:Data: 2024-02-01T10:16:53.369 DEBUG:boto:Headers: {} 2024-02-01T10:16:53.369 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:16:53.369 DEBUG:boto:Port: 8001 2024-02-01T10:16:53.369 DEBUG:boto:Params: {} 2024-02-01T10:16:53.369 DEBUG:boto:Token: None 2024-02-01T10:16:53.370 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:53 GMT /vclfcn-10/%26?versionId=null 2024-02-01T10:16:53.370 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Tde2hLXfFzN3HN8IXh2sSxd1Weg= 2024-02-01T10:16:53.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Tde2hLXfFzN3HN8IXh2sSxd1Weg=', 'Content-Length': '0'} 2024-02-01T10:16:53.372 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000094eba7d223015cff-0065bb6f95-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:53.372 DEBUG:boto:path=/vclfcn-10/%26 2024-02-01T10:16:53.372 DEBUG:boto:auth_path=/vclfcn-10/%26 2024-02-01T10:16:53.373 DEBUG:boto:path=/vclfcn-10/%26?versionId=null 2024-02-01T10:16:53.373 DEBUG:boto:auth_path=/vclfcn-10/%26?versionId=null 2024-02-01T10:16:53.373 DEBUG:boto:Method: HEAD 2024-02-01T10:16:53.373 DEBUG:boto:Path: /vclfcn-10/%26?versionId=null 2024-02-01T10:16:53.373 DEBUG:boto:Data: 2024-02-01T10:16:53.373 DEBUG:boto:Headers: {} 2024-02-01T10:16:53.373 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:53.373 DEBUG:boto:Port: 8000 2024-02-01T10:16:53.373 DEBUG:boto:Params: {} 2024-02-01T10:16:53.373 DEBUG:boto:Token: None 2024-02-01T10:16:53.373 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:53 GMT /vclfcn-10/%26?versionId=null 2024-02-01T10:16:53.374 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Tde2hLXfFzN3HN8IXh2sSxd1Weg= 2024-02-01T10:16:53.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Tde2hLXfFzN3HN8IXh2sSxd1Weg=', 'Content-Length': '0'} 2024-02-01T10:16:53.376 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d2fbbb50cda20151-0065bb6f95-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:53.376 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-02-01T10:16:53.376 DEBUG:boto:path=/vclfcn-10/%26 2024-02-01T10:16:53.377 DEBUG:boto:auth_path=/vclfcn-10/%26 2024-02-01T10:16:53.377 DEBUG:boto:path=/vclfcn-10/%26?versionId=null 2024-02-01T10:16:53.377 DEBUG:boto:auth_path=/vclfcn-10/%26?versionId=null 2024-02-01T10:16:53.377 DEBUG:boto:Method: GET 2024-02-01T10:16:53.377 DEBUG:boto:Path: /vclfcn-10/%26?versionId=null 2024-02-01T10:16:53.377 DEBUG:boto:Data: 2024-02-01T10:16:53.377 DEBUG:boto:Headers: {} 2024-02-01T10:16:53.377 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:16:53.377 DEBUG:boto:Port: 8001 2024-02-01T10:16:53.377 DEBUG:boto:Params: {} 2024-02-01T10:16:53.377 DEBUG:boto:Token: None 2024-02-01T10:16:53.378 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:53 GMT /vclfcn-10/%26?versionId=null 2024-02-01T10:16:53.378 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:2b6Rpns2G/TIPWH6STsN2wFuQNg= 2024-02-01T10:16:53.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2b6Rpns2G/TIPWH6STsN2wFuQNg=', 'Content-Length': '0'} 2024-02-01T10:16:53.381 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f27154c993296990-0065bb6f95-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:53.381 DEBUG:boto:path=/vclfcn-10/%26 2024-02-01T10:16:53.381 DEBUG:boto:auth_path=/vclfcn-10/%26 2024-02-01T10:16:53.382 DEBUG:boto:path=/vclfcn-10/%26?versionId=null 2024-02-01T10:16:53.382 DEBUG:boto:auth_path=/vclfcn-10/%26?versionId=null 2024-02-01T10:16:53.382 DEBUG:boto:Method: GET 2024-02-01T10:16:53.383 DEBUG:boto:Path: /vclfcn-10/%26?versionId=null 2024-02-01T10:16:53.383 DEBUG:boto:Data: 2024-02-01T10:16:53.383 DEBUG:boto:Headers: {} 2024-02-01T10:16:53.383 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:53.383 DEBUG:boto:Port: 8000 2024-02-01T10:16:53.383 DEBUG:boto:Params: {} 2024-02-01T10:16:53.383 DEBUG:boto:Token: None 2024-02-01T10:16:53.383 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:53 GMT /vclfcn-10/%26?versionId=null 2024-02-01T10:16:53.383 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:2b6Rpns2G/TIPWH6STsN2wFuQNg= 2024-02-01T10:16:53.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2b6Rpns2G/TIPWH6STsN2wFuQNg=', 'Content-Length': '0'} 2024-02-01T10:16:53.386 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b7058f0c3fd18809-0065bb6f95-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:53.386 DEBUG:boto:path=/vclfcn-10/ 2024-02-01T10:16:53.386 DEBUG:boto:auth_path=/vclfcn-10/ 2024-02-01T10:16:53.387 DEBUG:boto:path=/vclfcn-10/?prefix=%26 2024-02-01T10:16:53.387 DEBUG:boto:auth_path=/vclfcn-10/?prefix=%26 2024-02-01T10:16:53.387 DEBUG:boto:Method: GET 2024-02-01T10:16:53.387 DEBUG:boto:Path: /vclfcn-10/?prefix=%26 2024-02-01T10:16:53.387 DEBUG:boto:Data: 2024-02-01T10:16:53.387 DEBUG:boto:Headers: {} 2024-02-01T10:16:53.387 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:16:53.387 DEBUG:boto:Port: 8001 2024-02-01T10:16:53.387 DEBUG:boto:Params: {} 2024-02-01T10:16:53.387 DEBUG:boto:Token: None 2024-02-01T10:16:53.388 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:53 GMT /vclfcn-10/ 2024-02-01T10:16:53.388 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gxA919U/ZvYiLbGrFtIj/OtV1Jg= 2024-02-01T10:16:53.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gxA919U/ZvYiLbGrFtIj/OtV1Jg=', 'Content-Length': '0'} 2024-02-01T10:16:53.392 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e791186c2f7412e1-0065bb6f95-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:53.436 DEBUG:boto:b'vclfcn-10&1000false&2024-02-01T10:16:36.379Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:16:53.437 DEBUG:boto:path=/vclfcn-10/ 2024-02-01T10:16:53.437 DEBUG:boto:auth_path=/vclfcn-10/ 2024-02-01T10:16:53.437 DEBUG:boto:path=/vclfcn-10/?prefix=%26 2024-02-01T10:16:53.437 DEBUG:boto:auth_path=/vclfcn-10/?prefix=%26 2024-02-01T10:16:53.437 DEBUG:boto:Method: GET 2024-02-01T10:16:53.437 DEBUG:boto:Path: /vclfcn-10/?prefix=%26 2024-02-01T10:16:53.437 DEBUG:boto:Data: 2024-02-01T10:16:53.437 DEBUG:boto:Headers: {} 2024-02-01T10:16:53.437 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:53.437 DEBUG:boto:Port: 8000 2024-02-01T10:16:53.438 DEBUG:boto:Params: {} 2024-02-01T10:16:53.438 DEBUG:boto:Token: None 2024-02-01T10:16:53.438 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:53 GMT /vclfcn-10/ 2024-02-01T10:16:53.438 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gxA919U/ZvYiLbGrFtIj/OtV1Jg= 2024-02-01T10:16:53.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gxA919U/ZvYiLbGrFtIj/OtV1Jg=', 'Content-Length': '0'} 2024-02-01T10:16:53.442 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a78e61f24b49df02-0065bb6f95-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:53.442 DEBUG:boto:b'vclfcn-10&1000false&2024-02-01T10:16:36.379Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:16:53.443 DEBUG:boto:path=/vclfcn-10/%26 2024-02-01T10:16:53.443 DEBUG:boto:auth_path=/vclfcn-10/%26 2024-02-01T10:16:53.443 DEBUG:boto:Method: HEAD 2024-02-01T10:16:53.443 DEBUG:boto:Path: /vclfcn-10/%26 2024-02-01T10:16:53.443 DEBUG:boto:Data: 2024-02-01T10:16:53.443 DEBUG:boto:Headers: {} 2024-02-01T10:16:53.443 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:16:53.443 DEBUG:boto:Port: 8001 2024-02-01T10:16:53.443 DEBUG:boto:Params: {} 2024-02-01T10:16:53.444 DEBUG:boto:Token: None 2024-02-01T10:16:53.444 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:53 GMT /vclfcn-10/%26 2024-02-01T10:16:53.444 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ADT0biNbY3191c1R44m5EFfLckI= 2024-02-01T10:16:53.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ADT0biNbY3191c1R44m5EFfLckI=', 'Content-Length': '0'} 2024-02-01T10:16:53.447 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c7a3035b1fd9e691-0065bb6f95-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:53.447 DEBUG:boto:path=/vclfcn-10/%26 2024-02-01T10:16:53.447 DEBUG:boto:auth_path=/vclfcn-10/%26 2024-02-01T10:16:53.447 DEBUG:boto:Method: HEAD 2024-02-01T10:16:53.447 DEBUG:boto:Path: /vclfcn-10/%26 2024-02-01T10:16:53.447 DEBUG:boto:Data: 2024-02-01T10:16:53.447 DEBUG:boto:Headers: {} 2024-02-01T10:16:53.448 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:53.448 DEBUG:boto:Port: 8000 2024-02-01T10:16:53.448 DEBUG:boto:Params: {} 2024-02-01T10:16:53.448 DEBUG:boto:Token: None 2024-02-01T10:16:53.448 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:53 GMT /vclfcn-10/%26 2024-02-01T10:16:53.448 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ADT0biNbY3191c1R44m5EFfLckI= 2024-02-01T10:16:53.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ADT0biNbY3191c1R44m5EFfLckI=', 'Content-Length': '0'} 2024-02-01T10:16:53.451 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000009b64ab196eab76b-0065bb6f95-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:53.451 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-02-01T10:16:53.451 DEBUG:boto:path=/vclfcn-10/%26 2024-02-01T10:16:53.451 DEBUG:boto:auth_path=/vclfcn-10/%26 2024-02-01T10:16:53.451 DEBUG:boto:Method: GET 2024-02-01T10:16:53.451 DEBUG:boto:Path: /vclfcn-10/%26 2024-02-01T10:16:53.451 DEBUG:boto:Data: 2024-02-01T10:16:53.451 DEBUG:boto:Headers: {} 2024-02-01T10:16:53.452 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:16:53.452 DEBUG:boto:Port: 8001 2024-02-01T10:16:53.452 DEBUG:boto:Params: {} 2024-02-01T10:16:53.452 DEBUG:boto:Token: None 2024-02-01T10:16:53.452 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:53 GMT /vclfcn-10/%26 2024-02-01T10:16:53.452 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:k+maamXBse7IxUV86ZSKiHqju1U= 2024-02-01T10:16:53.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:k+maamXBse7IxUV86ZSKiHqju1U=', 'Content-Length': '0'} 2024-02-01T10:16:53.455 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000af8bd259cb119d45-0065bb6f95-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:53.455 DEBUG:boto:path=/vclfcn-10/%26 2024-02-01T10:16:53.455 DEBUG:boto:auth_path=/vclfcn-10/%26 2024-02-01T10:16:53.455 DEBUG:boto:Method: GET 2024-02-01T10:16:53.455 DEBUG:boto:Path: /vclfcn-10/%26 2024-02-01T10:16:53.456 DEBUG:boto:Data: 2024-02-01T10:16:53.456 DEBUG:boto:Headers: {} 2024-02-01T10:16:53.456 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:53.456 DEBUG:boto:Port: 8000 2024-02-01T10:16:53.456 DEBUG:boto:Params: {} 2024-02-01T10:16:53.456 DEBUG:boto:Token: None 2024-02-01T10:16:53.456 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:53 GMT /vclfcn-10/%26 2024-02-01T10:16:53.456 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:k+maamXBse7IxUV86ZSKiHqju1U= 2024-02-01T10:16:53.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:k+maamXBse7IxUV86ZSKiHqju1U=', 'Content-Length': '0'} 2024-02-01T10:16:53.459 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000035a6c514590f2fc1-0065bb6f95-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:53.500 DEBUG:boto:path=/vclfcn-10/ 2024-02-01T10:16:53.500 DEBUG:boto:auth_path=/vclfcn-10/ 2024-02-01T10:16:53.500 DEBUG:boto:path=/vclfcn-10/?prefix=%26 2024-02-01T10:16:53.501 DEBUG:boto:auth_path=/vclfcn-10/?prefix=%26 2024-02-01T10:16:53.501 DEBUG:boto:Method: GET 2024-02-01T10:16:53.501 DEBUG:boto:Path: /vclfcn-10/?prefix=%26 2024-02-01T10:16:53.501 DEBUG:boto:Data: 2024-02-01T10:16:53.501 DEBUG:boto:Headers: {} 2024-02-01T10:16:53.501 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:16:53.501 DEBUG:boto:Port: 8001 2024-02-01T10:16:53.501 DEBUG:boto:Params: {} 2024-02-01T10:16:53.501 DEBUG:boto:Token: None 2024-02-01T10:16:53.502 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:53 GMT /vclfcn-10/ 2024-02-01T10:16:53.502 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gxA919U/ZvYiLbGrFtIj/OtV1Jg= 2024-02-01T10:16:53.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gxA919U/ZvYiLbGrFtIj/OtV1Jg=', 'Content-Length': '0'} 2024-02-01T10:16:53.506 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008895c2041d94ba7-0065bb6f95-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:53.506 DEBUG:boto:b'vclfcn-10&1000false&2024-02-01T10:16:36.379Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:16:53.507 DEBUG:boto:path=/vclfcn-10/ 2024-02-01T10:16:53.507 DEBUG:boto:auth_path=/vclfcn-10/ 2024-02-01T10:16:53.507 DEBUG:boto:path=/vclfcn-10/?prefix=%26 2024-02-01T10:16:53.507 DEBUG:boto:auth_path=/vclfcn-10/?prefix=%26 2024-02-01T10:16:53.507 DEBUG:boto:Method: GET 2024-02-01T10:16:53.507 DEBUG:boto:Path: /vclfcn-10/?prefix=%26 2024-02-01T10:16:53.507 DEBUG:boto:Data: 2024-02-01T10:16:53.507 DEBUG:boto:Headers: {} 2024-02-01T10:16:53.507 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:53.508 DEBUG:boto:Port: 8000 2024-02-01T10:16:53.508 DEBUG:boto:Params: {} 2024-02-01T10:16:53.508 DEBUG:boto:Token: None 2024-02-01T10:16:53.508 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:53 GMT /vclfcn-10/ 2024-02-01T10:16:53.508 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gxA919U/ZvYiLbGrFtIj/OtV1Jg= 2024-02-01T10:16:53.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gxA919U/ZvYiLbGrFtIj/OtV1Jg=', 'Content-Length': '0'} 2024-02-01T10:16:53.512 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd25b86e7a4bb13d-0065bb6f95-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:53.513 DEBUG:boto:b'vclfcn-10&1000false&2024-02-01T10:16:36.379Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:16:53.513 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-02-01T10:16:53.514 DEBUG:boto:path=/vclfcn-10/%3A 2024-02-01T10:16:53.514 DEBUG:boto:auth_path=/vclfcn-10/%3A 2024-02-01T10:16:53.514 DEBUG:boto:path=/vclfcn-10/%3A?versionId=null 2024-02-01T10:16:53.514 DEBUG:boto:auth_path=/vclfcn-10/%3A?versionId=null 2024-02-01T10:16:53.514 DEBUG:boto:Method: GET 2024-02-01T10:16:53.514 DEBUG:boto:Path: /vclfcn-10/%3A?versionId=null 2024-02-01T10:16:53.514 DEBUG:boto:Data: 2024-02-01T10:16:53.514 DEBUG:boto:Headers: {} 2024-02-01T10:16:53.514 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:16:53.514 DEBUG:boto:Port: 8001 2024-02-01T10:16:53.515 DEBUG:boto:Params: {} 2024-02-01T10:16:53.515 DEBUG:boto:Token: None 2024-02-01T10:16:53.515 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:53 GMT /vclfcn-10/%3A?versionId=null 2024-02-01T10:16:53.515 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:i1hyMcSjMNraE/cwTnKkiz1lOe8= 2024-02-01T10:16:53.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:i1hyMcSjMNraE/cwTnKkiz1lOe8=', 'Content-Length': '0'} 2024-02-01T10:16:53.518 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a52760d6e3f27818-0065bb6f95-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:53.518 DEBUG:boto:path=/vclfcn-10/%3A 2024-02-01T10:16:53.519 DEBUG:boto:auth_path=/vclfcn-10/%3A 2024-02-01T10:16:53.519 DEBUG:boto:path=/vclfcn-10/%3A?versionId=null 2024-02-01T10:16:53.519 DEBUG:boto:auth_path=/vclfcn-10/%3A?versionId=null 2024-02-01T10:16:53.519 DEBUG:boto:Method: GET 2024-02-01T10:16:53.519 DEBUG:boto:Path: /vclfcn-10/%3A?versionId=null 2024-02-01T10:16:53.519 DEBUG:boto:Data: 2024-02-01T10:16:53.519 DEBUG:boto:Headers: {} 2024-02-01T10:16:53.519 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:53.519 DEBUG:boto:Port: 8000 2024-02-01T10:16:53.519 DEBUG:boto:Params: {} 2024-02-01T10:16:53.519 DEBUG:boto:Token: None 2024-02-01T10:16:53.520 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:53 GMT /vclfcn-10/%3A?versionId=null 2024-02-01T10:16:53.520 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:i1hyMcSjMNraE/cwTnKkiz1lOe8= 2024-02-01T10:16:53.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:i1hyMcSjMNraE/cwTnKkiz1lOe8=', 'Content-Length': '0'} 2024-02-01T10:16:53.523 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000918bcad31b34fad3-0065bb6f95-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:53.523 DEBUG:boto:path=/vclfcn-10/%3A 2024-02-01T10:16:53.523 DEBUG:boto:auth_path=/vclfcn-10/%3A 2024-02-01T10:16:53.524 DEBUG:boto:path=/vclfcn-10/%3A?versionId=null 2024-02-01T10:16:53.524 DEBUG:boto:auth_path=/vclfcn-10/%3A?versionId=null 2024-02-01T10:16:53.524 DEBUG:boto:Method: HEAD 2024-02-01T10:16:53.524 DEBUG:boto:Path: /vclfcn-10/%3A?versionId=null 2024-02-01T10:16:53.524 DEBUG:boto:Data: 2024-02-01T10:16:53.524 DEBUG:boto:Headers: {} 2024-02-01T10:16:53.524 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:16:53.524 DEBUG:boto:Port: 8001 2024-02-01T10:16:53.524 DEBUG:boto:Params: {} 2024-02-01T10:16:53.524 DEBUG:boto:Token: None 2024-02-01T10:16:53.525 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:53 GMT /vclfcn-10/%3A?versionId=null 2024-02-01T10:16:53.525 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cxxbvjA1LG2By/cW0GlJoXMUa6Y= 2024-02-01T10:16:53.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cxxbvjA1LG2By/cW0GlJoXMUa6Y=', 'Content-Length': '0'} 2024-02-01T10:16:53.528 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000056b1a0197446b364-0065bb6f95-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:53.528 DEBUG:boto:path=/vclfcn-10/%3A 2024-02-01T10:16:53.528 DEBUG:boto:auth_path=/vclfcn-10/%3A 2024-02-01T10:16:53.528 DEBUG:boto:path=/vclfcn-10/%3A?versionId=null 2024-02-01T10:16:53.529 DEBUG:boto:auth_path=/vclfcn-10/%3A?versionId=null 2024-02-01T10:16:53.529 DEBUG:boto:Method: HEAD 2024-02-01T10:16:53.529 DEBUG:boto:Path: /vclfcn-10/%3A?versionId=null 2024-02-01T10:16:53.529 DEBUG:boto:Data: 2024-02-01T10:16:53.529 DEBUG:boto:Headers: {} 2024-02-01T10:16:53.529 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:53.529 DEBUG:boto:Port: 8000 2024-02-01T10:16:53.529 DEBUG:boto:Params: {} 2024-02-01T10:16:53.529 DEBUG:boto:Token: None 2024-02-01T10:16:53.529 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:53 GMT /vclfcn-10/%3A?versionId=null 2024-02-01T10:16:53.530 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cxxbvjA1LG2By/cW0GlJoXMUa6Y= 2024-02-01T10:16:53.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cxxbvjA1LG2By/cW0GlJoXMUa6Y=', 'Content-Length': '0'} 2024-02-01T10:16:53.532 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000101f8e734f049a13-0065bb6f95-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:53.533 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-02-01T10:16:53.533 DEBUG:boto:path=/vclfcn-10/%3A 2024-02-01T10:16:53.533 DEBUG:boto:auth_path=/vclfcn-10/%3A 2024-02-01T10:16:53.533 DEBUG:boto:path=/vclfcn-10/%3A?versionId=null 2024-02-01T10:16:53.533 DEBUG:boto:auth_path=/vclfcn-10/%3A?versionId=null 2024-02-01T10:16:53.533 DEBUG:boto:Method: GET 2024-02-01T10:16:53.533 DEBUG:boto:Path: /vclfcn-10/%3A?versionId=null 2024-02-01T10:16:53.533 DEBUG:boto:Data: 2024-02-01T10:16:53.533 DEBUG:boto:Headers: {} 2024-02-01T10:16:53.533 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:16:53.533 DEBUG:boto:Port: 8001 2024-02-01T10:16:53.534 DEBUG:boto:Params: {} 2024-02-01T10:16:53.534 DEBUG:boto:Token: None 2024-02-01T10:16:53.534 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:53 GMT /vclfcn-10/%3A?versionId=null 2024-02-01T10:16:53.534 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:i1hyMcSjMNraE/cwTnKkiz1lOe8= 2024-02-01T10:16:53.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:i1hyMcSjMNraE/cwTnKkiz1lOe8=', 'Content-Length': '0'} 2024-02-01T10:16:53.537 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cc50b2fddaef197c-0065bb6f95-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:53.584 DEBUG:boto:path=/vclfcn-10/%3A 2024-02-01T10:16:53.584 DEBUG:boto:auth_path=/vclfcn-10/%3A 2024-02-01T10:16:53.585 DEBUG:boto:path=/vclfcn-10/%3A?versionId=null 2024-02-01T10:16:53.585 DEBUG:boto:auth_path=/vclfcn-10/%3A?versionId=null 2024-02-01T10:16:53.585 DEBUG:boto:Method: GET 2024-02-01T10:16:53.585 DEBUG:boto:Path: /vclfcn-10/%3A?versionId=null 2024-02-01T10:16:53.585 DEBUG:boto:Data: 2024-02-01T10:16:53.585 DEBUG:boto:Headers: {} 2024-02-01T10:16:53.585 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:53.585 DEBUG:boto:Port: 8000 2024-02-01T10:16:53.585 DEBUG:boto:Params: {} 2024-02-01T10:16:53.586 DEBUG:boto:Token: None 2024-02-01T10:16:53.586 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:53 GMT /vclfcn-10/%3A?versionId=null 2024-02-01T10:16:53.586 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:i1hyMcSjMNraE/cwTnKkiz1lOe8= 2024-02-01T10:16:53.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:i1hyMcSjMNraE/cwTnKkiz1lOe8=', 'Content-Length': '0'} 2024-02-01T10:16:53.589 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001c01d5e008f9d120-0065bb6f95-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:53.590 DEBUG:boto:path=/vclfcn-10/ 2024-02-01T10:16:53.590 DEBUG:boto:auth_path=/vclfcn-10/ 2024-02-01T10:16:53.590 DEBUG:boto:path=/vclfcn-10/?prefix=%3A 2024-02-01T10:16:53.590 DEBUG:boto:auth_path=/vclfcn-10/?prefix=%3A 2024-02-01T10:16:53.590 DEBUG:boto:Method: GET 2024-02-01T10:16:53.591 DEBUG:boto:Path: /vclfcn-10/?prefix=%3A 2024-02-01T10:16:53.591 DEBUG:boto:Data: 2024-02-01T10:16:53.591 DEBUG:boto:Headers: {} 2024-02-01T10:16:53.591 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:16:53.591 DEBUG:boto:Port: 8001 2024-02-01T10:16:53.592 DEBUG:boto:Params: {} 2024-02-01T10:16:53.592 DEBUG:boto:Token: None 2024-02-01T10:16:53.592 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:53 GMT /vclfcn-10/ 2024-02-01T10:16:53.593 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gxA919U/ZvYiLbGrFtIj/OtV1Jg= 2024-02-01T10:16:53.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gxA919U/ZvYiLbGrFtIj/OtV1Jg=', 'Content-Length': '0'} 2024-02-01T10:16:53.597 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006536ecdadd15f6b8-0065bb6f95-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:53.598 DEBUG:boto:b'vclfcn-10:1000false:2024-02-01T10:16:36.351Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:16:53.598 DEBUG:boto:path=/vclfcn-10/ 2024-02-01T10:16:53.599 DEBUG:boto:auth_path=/vclfcn-10/ 2024-02-01T10:16:53.599 DEBUG:boto:path=/vclfcn-10/?prefix=%3A 2024-02-01T10:16:53.599 DEBUG:boto:auth_path=/vclfcn-10/?prefix=%3A 2024-02-01T10:16:53.599 DEBUG:boto:Method: GET 2024-02-01T10:16:53.600 DEBUG:boto:Path: /vclfcn-10/?prefix=%3A 2024-02-01T10:16:53.600 DEBUG:boto:Data: 2024-02-01T10:16:53.600 DEBUG:boto:Headers: {} 2024-02-01T10:16:53.600 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:53.600 DEBUG:boto:Port: 8000 2024-02-01T10:16:53.601 DEBUG:boto:Params: {} 2024-02-01T10:16:53.601 DEBUG:boto:Token: None 2024-02-01T10:16:53.601 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:53 GMT /vclfcn-10/ 2024-02-01T10:16:53.602 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gxA919U/ZvYiLbGrFtIj/OtV1Jg= 2024-02-01T10:16:53.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gxA919U/ZvYiLbGrFtIj/OtV1Jg=', 'Content-Length': '0'} 2024-02-01T10:16:53.606 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001824f0ef61f88e25-0065bb6f95-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:53.648 DEBUG:boto:b'vclfcn-10:1000false:2024-02-01T10:16:36.351Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:16:53.649 DEBUG:boto:path=/vclfcn-10/%3A 2024-02-01T10:16:53.649 DEBUG:boto:auth_path=/vclfcn-10/%3A 2024-02-01T10:16:53.649 DEBUG:boto:Method: HEAD 2024-02-01T10:16:53.649 DEBUG:boto:Path: /vclfcn-10/%3A 2024-02-01T10:16:53.649 DEBUG:boto:Data: 2024-02-01T10:16:53.649 DEBUG:boto:Headers: {} 2024-02-01T10:16:53.649 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:16:53.650 DEBUG:boto:Port: 8001 2024-02-01T10:16:53.650 DEBUG:boto:Params: {} 2024-02-01T10:16:53.650 DEBUG:boto:Token: None 2024-02-01T10:16:53.650 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:53 GMT /vclfcn-10/%3A 2024-02-01T10:16:53.651 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:uzXU4Ea9rO9TdXsBHM8yFesMnzo= 2024-02-01T10:16:53.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:uzXU4Ea9rO9TdXsBHM8yFesMnzo=', 'Content-Length': '0'} 2024-02-01T10:16:53.654 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000af6c8c8dc14fca87-0065bb6f95-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:53.654 DEBUG:boto:path=/vclfcn-10/%3A 2024-02-01T10:16:53.655 DEBUG:boto:auth_path=/vclfcn-10/%3A 2024-02-01T10:16:53.655 DEBUG:boto:Method: HEAD 2024-02-01T10:16:53.655 DEBUG:boto:Path: /vclfcn-10/%3A 2024-02-01T10:16:53.655 DEBUG:boto:Data: 2024-02-01T10:16:53.655 DEBUG:boto:Headers: {} 2024-02-01T10:16:53.656 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:53.656 DEBUG:boto:Port: 8000 2024-02-01T10:16:53.656 DEBUG:boto:Params: {} 2024-02-01T10:16:53.656 DEBUG:boto:Token: None 2024-02-01T10:16:53.656 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:53 GMT /vclfcn-10/%3A 2024-02-01T10:16:53.656 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:uzXU4Ea9rO9TdXsBHM8yFesMnzo= 2024-02-01T10:16:53.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:uzXU4Ea9rO9TdXsBHM8yFesMnzo=', 'Content-Length': '0'} 2024-02-01T10:16:53.659 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c48909736f762ff8-0065bb6f95-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:53.659 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-02-01T10:16:53.659 DEBUG:boto:path=/vclfcn-10/%3A 2024-02-01T10:16:53.660 DEBUG:boto:auth_path=/vclfcn-10/%3A 2024-02-01T10:16:53.660 DEBUG:boto:Method: GET 2024-02-01T10:16:53.660 DEBUG:boto:Path: /vclfcn-10/%3A 2024-02-01T10:16:53.660 DEBUG:boto:Data: 2024-02-01T10:16:53.660 DEBUG:boto:Headers: {} 2024-02-01T10:16:53.661 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:16:53.661 DEBUG:boto:Port: 8001 2024-02-01T10:16:53.661 DEBUG:boto:Params: {} 2024-02-01T10:16:53.661 DEBUG:boto:Token: None 2024-02-01T10:16:53.662 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:53 GMT /vclfcn-10/%3A 2024-02-01T10:16:53.662 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tN2Ahssqydw8sYbjmt6sjWN6gWA= 2024-02-01T10:16:53.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tN2Ahssqydw8sYbjmt6sjWN6gWA=', 'Content-Length': '0'} 2024-02-01T10:16:53.665 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f35da51fc494e704-0065bb6f95-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:53.666 DEBUG:boto:path=/vclfcn-10/%3A 2024-02-01T10:16:53.666 DEBUG:boto:auth_path=/vclfcn-10/%3A 2024-02-01T10:16:53.666 DEBUG:boto:Method: GET 2024-02-01T10:16:53.667 DEBUG:boto:Path: /vclfcn-10/%3A 2024-02-01T10:16:53.667 DEBUG:boto:Data: 2024-02-01T10:16:53.667 DEBUG:boto:Headers: {} 2024-02-01T10:16:53.667 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:53.668 DEBUG:boto:Port: 8000 2024-02-01T10:16:53.668 DEBUG:boto:Params: {} 2024-02-01T10:16:53.668 DEBUG:boto:Token: None 2024-02-01T10:16:53.668 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:53 GMT /vclfcn-10/%3A 2024-02-01T10:16:53.669 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tN2Ahssqydw8sYbjmt6sjWN6gWA= 2024-02-01T10:16:53.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tN2Ahssqydw8sYbjmt6sjWN6gWA=', 'Content-Length': '0'} 2024-02-01T10:16:53.673 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000040c558a135674a92-0065bb6f95-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:53.673 DEBUG:boto:path=/vclfcn-10/ 2024-02-01T10:16:53.673 DEBUG:boto:auth_path=/vclfcn-10/ 2024-02-01T10:16:53.674 DEBUG:boto:path=/vclfcn-10/?prefix=%3A 2024-02-01T10:16:53.674 DEBUG:boto:auth_path=/vclfcn-10/?prefix=%3A 2024-02-01T10:16:53.674 DEBUG:boto:Method: GET 2024-02-01T10:16:53.674 DEBUG:boto:Path: /vclfcn-10/?prefix=%3A 2024-02-01T10:16:53.675 DEBUG:boto:Data: 2024-02-01T10:16:53.675 DEBUG:boto:Headers: {} 2024-02-01T10:16:53.675 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:16:53.675 DEBUG:boto:Port: 8001 2024-02-01T10:16:53.675 DEBUG:boto:Params: {} 2024-02-01T10:16:53.676 DEBUG:boto:Token: None 2024-02-01T10:16:53.676 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:53 GMT /vclfcn-10/ 2024-02-01T10:16:53.676 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gxA919U/ZvYiLbGrFtIj/OtV1Jg= 2024-02-01T10:16:53.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gxA919U/ZvYiLbGrFtIj/OtV1Jg=', 'Content-Length': '0'} 2024-02-01T10:16:53.682 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008aac94f5661f0940-0065bb6f95-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:53.724 DEBUG:boto:b'vclfcn-10:1000false:2024-02-01T10:16:36.351Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:16:53.725 DEBUG:boto:path=/vclfcn-10/ 2024-02-01T10:16:53.725 DEBUG:boto:auth_path=/vclfcn-10/ 2024-02-01T10:16:53.725 DEBUG:boto:path=/vclfcn-10/?prefix=%3A 2024-02-01T10:16:53.725 DEBUG:boto:auth_path=/vclfcn-10/?prefix=%3A 2024-02-01T10:16:53.725 DEBUG:boto:Method: GET 2024-02-01T10:16:53.725 DEBUG:boto:Path: /vclfcn-10/?prefix=%3A 2024-02-01T10:16:53.725 DEBUG:boto:Data: 2024-02-01T10:16:53.725 DEBUG:boto:Headers: {} 2024-02-01T10:16:53.725 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:53.726 DEBUG:boto:Port: 8000 2024-02-01T10:16:53.726 DEBUG:boto:Params: {} 2024-02-01T10:16:53.726 DEBUG:boto:Token: None 2024-02-01T10:16:53.726 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:53 GMT /vclfcn-10/ 2024-02-01T10:16:53.726 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gxA919U/ZvYiLbGrFtIj/OtV1Jg= 2024-02-01T10:16:53.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gxA919U/ZvYiLbGrFtIj/OtV1Jg=', 'Content-Length': '0'} 2024-02-01T10:16:53.731 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3d21754fcaf09cd-0065bb6f95-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:53.731 DEBUG:boto:b'vclfcn-10:1000false:2024-02-01T10:16:36.351Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:16:53.731 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-02-01T10:16:53.732 DEBUG:boto:path=/vclfcn-10/_myobj 2024-02-01T10:16:53.732 DEBUG:boto:auth_path=/vclfcn-10/_myobj 2024-02-01T10:16:53.732 DEBUG:boto:path=/vclfcn-10/_myobj?versionId=null 2024-02-01T10:16:53.732 DEBUG:boto:auth_path=/vclfcn-10/_myobj?versionId=null 2024-02-01T10:16:53.732 DEBUG:boto:Method: GET 2024-02-01T10:16:53.732 DEBUG:boto:Path: /vclfcn-10/_myobj?versionId=null 2024-02-01T10:16:53.732 DEBUG:boto:Data: 2024-02-01T10:16:53.732 DEBUG:boto:Headers: {} 2024-02-01T10:16:53.732 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:16:53.732 DEBUG:boto:Port: 8001 2024-02-01T10:16:53.732 DEBUG:boto:Params: {} 2024-02-01T10:16:53.733 DEBUG:boto:Token: None 2024-02-01T10:16:53.733 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:53 GMT /vclfcn-10/_myobj?versionId=null 2024-02-01T10:16:53.733 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:XRe/jcZ5JcAe2YvihSYVSEIVyPY= 2024-02-01T10:16:53.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:XRe/jcZ5JcAe2YvihSYVSEIVyPY=', 'Content-Length': '0'} 2024-02-01T10:16:53.736 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eb51f99a786ff9ac-0065bb6f95-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:53.736 DEBUG:boto:path=/vclfcn-10/_myobj 2024-02-01T10:16:53.736 DEBUG:boto:auth_path=/vclfcn-10/_myobj 2024-02-01T10:16:53.736 DEBUG:boto:path=/vclfcn-10/_myobj?versionId=null 2024-02-01T10:16:53.737 DEBUG:boto:auth_path=/vclfcn-10/_myobj?versionId=null 2024-02-01T10:16:53.737 DEBUG:boto:Method: GET 2024-02-01T10:16:53.737 DEBUG:boto:Path: /vclfcn-10/_myobj?versionId=null 2024-02-01T10:16:53.737 DEBUG:boto:Data: 2024-02-01T10:16:53.737 DEBUG:boto:Headers: {} 2024-02-01T10:16:53.737 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:53.737 DEBUG:boto:Port: 8000 2024-02-01T10:16:53.737 DEBUG:boto:Params: {} 2024-02-01T10:16:53.737 DEBUG:boto:Token: None 2024-02-01T10:16:53.737 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:53 GMT /vclfcn-10/_myobj?versionId=null 2024-02-01T10:16:53.738 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:XRe/jcZ5JcAe2YvihSYVSEIVyPY= 2024-02-01T10:16:53.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:XRe/jcZ5JcAe2YvihSYVSEIVyPY=', 'Content-Length': '0'} 2024-02-01T10:16:53.740 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000352ec9c8d0c8e8f3-0065bb6f95-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:53.741 DEBUG:boto:path=/vclfcn-10/_myobj 2024-02-01T10:16:53.741 DEBUG:boto:auth_path=/vclfcn-10/_myobj 2024-02-01T10:16:53.741 DEBUG:boto:path=/vclfcn-10/_myobj?versionId=null 2024-02-01T10:16:53.741 DEBUG:boto:auth_path=/vclfcn-10/_myobj?versionId=null 2024-02-01T10:16:53.741 DEBUG:boto:Method: HEAD 2024-02-01T10:16:53.741 DEBUG:boto:Path: /vclfcn-10/_myobj?versionId=null 2024-02-01T10:16:53.741 DEBUG:boto:Data: 2024-02-01T10:16:53.741 DEBUG:boto:Headers: {} 2024-02-01T10:16:53.741 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:16:53.742 DEBUG:boto:Port: 8001 2024-02-01T10:16:53.742 DEBUG:boto:Params: {} 2024-02-01T10:16:53.742 DEBUG:boto:Token: None 2024-02-01T10:16:53.742 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:53 GMT /vclfcn-10/_myobj?versionId=null 2024-02-01T10:16:53.742 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:2zWi5dXKSiqkzf/5ghZs9NzE3dY= 2024-02-01T10:16:53.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2zWi5dXKSiqkzf/5ghZs9NzE3dY=', 'Content-Length': '0'} 2024-02-01T10:16:53.745 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000008683d2d45700047-0065bb6f95-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:53.745 DEBUG:boto:path=/vclfcn-10/_myobj 2024-02-01T10:16:53.745 DEBUG:boto:auth_path=/vclfcn-10/_myobj 2024-02-01T10:16:53.745 DEBUG:boto:path=/vclfcn-10/_myobj?versionId=null 2024-02-01T10:16:53.745 DEBUG:boto:auth_path=/vclfcn-10/_myobj?versionId=null 2024-02-01T10:16:53.746 DEBUG:boto:Method: HEAD 2024-02-01T10:16:53.746 DEBUG:boto:Path: /vclfcn-10/_myobj?versionId=null 2024-02-01T10:16:53.746 DEBUG:boto:Data: 2024-02-01T10:16:53.746 DEBUG:boto:Headers: {} 2024-02-01T10:16:53.746 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:53.746 DEBUG:boto:Port: 8000 2024-02-01T10:16:53.746 DEBUG:boto:Params: {} 2024-02-01T10:16:53.746 DEBUG:boto:Token: None 2024-02-01T10:16:53.746 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:53 GMT /vclfcn-10/_myobj?versionId=null 2024-02-01T10:16:53.746 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:2zWi5dXKSiqkzf/5ghZs9NzE3dY= 2024-02-01T10:16:53.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2zWi5dXKSiqkzf/5ghZs9NzE3dY=', 'Content-Length': '0'} 2024-02-01T10:16:53.749 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008e18637e8df8db5c-0065bb6f95-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:53.749 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-02-01T10:16:53.749 DEBUG:boto:path=/vclfcn-10/_myobj 2024-02-01T10:16:53.750 DEBUG:boto:auth_path=/vclfcn-10/_myobj 2024-02-01T10:16:53.750 DEBUG:boto:path=/vclfcn-10/_myobj?versionId=null 2024-02-01T10:16:53.750 DEBUG:boto:auth_path=/vclfcn-10/_myobj?versionId=null 2024-02-01T10:16:53.750 DEBUG:boto:Method: GET 2024-02-01T10:16:53.750 DEBUG:boto:Path: /vclfcn-10/_myobj?versionId=null 2024-02-01T10:16:53.750 DEBUG:boto:Data: 2024-02-01T10:16:53.750 DEBUG:boto:Headers: {} 2024-02-01T10:16:53.750 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:16:53.750 DEBUG:boto:Port: 8001 2024-02-01T10:16:53.750 DEBUG:boto:Params: {} 2024-02-01T10:16:53.750 DEBUG:boto:Token: None 2024-02-01T10:16:53.751 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:53 GMT /vclfcn-10/_myobj?versionId=null 2024-02-01T10:16:53.751 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:XRe/jcZ5JcAe2YvihSYVSEIVyPY= 2024-02-01T10:16:53.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:XRe/jcZ5JcAe2YvihSYVSEIVyPY=', 'Content-Length': '0'} 2024-02-01T10:16:53.754 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000af304a965a2353e4-0065bb6f95-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:53.796 DEBUG:boto:path=/vclfcn-10/_myobj 2024-02-01T10:16:53.796 DEBUG:boto:auth_path=/vclfcn-10/_myobj 2024-02-01T10:16:53.796 DEBUG:boto:path=/vclfcn-10/_myobj?versionId=null 2024-02-01T10:16:53.796 DEBUG:boto:auth_path=/vclfcn-10/_myobj?versionId=null 2024-02-01T10:16:53.797 DEBUG:boto:Method: GET 2024-02-01T10:16:53.797 DEBUG:boto:Path: /vclfcn-10/_myobj?versionId=null 2024-02-01T10:16:53.797 DEBUG:boto:Data: 2024-02-01T10:16:53.797 DEBUG:boto:Headers: {} 2024-02-01T10:16:53.797 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:53.797 DEBUG:boto:Port: 8000 2024-02-01T10:16:53.797 DEBUG:boto:Params: {} 2024-02-01T10:16:53.797 DEBUG:boto:Token: None 2024-02-01T10:16:53.797 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:53 GMT /vclfcn-10/_myobj?versionId=null 2024-02-01T10:16:53.797 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:XRe/jcZ5JcAe2YvihSYVSEIVyPY= 2024-02-01T10:16:53.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:XRe/jcZ5JcAe2YvihSYVSEIVyPY=', 'Content-Length': '0'} 2024-02-01T10:16:53.800 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b03d41904a44a73c-0065bb6f95-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:53.801 DEBUG:boto:path=/vclfcn-10/ 2024-02-01T10:16:53.801 DEBUG:boto:auth_path=/vclfcn-10/ 2024-02-01T10:16:53.801 DEBUG:boto:path=/vclfcn-10/?prefix=_myobj 2024-02-01T10:16:53.801 DEBUG:boto:auth_path=/vclfcn-10/?prefix=_myobj 2024-02-01T10:16:53.801 DEBUG:boto:Method: GET 2024-02-01T10:16:53.801 DEBUG:boto:Path: /vclfcn-10/?prefix=_myobj 2024-02-01T10:16:53.801 DEBUG:boto:Data: 2024-02-01T10:16:53.801 DEBUG:boto:Headers: {} 2024-02-01T10:16:53.801 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:16:53.802 DEBUG:boto:Port: 8001 2024-02-01T10:16:53.802 DEBUG:boto:Params: {} 2024-02-01T10:16:53.802 DEBUG:boto:Token: None 2024-02-01T10:16:53.802 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:53 GMT /vclfcn-10/ 2024-02-01T10:16:53.802 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gxA919U/ZvYiLbGrFtIj/OtV1Jg= 2024-02-01T10:16:53.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gxA919U/ZvYiLbGrFtIj/OtV1Jg=', 'Content-Length': '0'} 2024-02-01T10:16:53.807 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c3185939dabc7c3-0065bb6f95-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:53.808 DEBUG:boto:b'vclfcn-10_myobj1000false_myobj2024-02-01T10:16:36.295Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:16:53.808 DEBUG:boto:path=/vclfcn-10/ 2024-02-01T10:16:53.808 DEBUG:boto:auth_path=/vclfcn-10/ 2024-02-01T10:16:53.808 DEBUG:boto:path=/vclfcn-10/?prefix=_myobj 2024-02-01T10:16:53.808 DEBUG:boto:auth_path=/vclfcn-10/?prefix=_myobj 2024-02-01T10:16:53.809 DEBUG:boto:Method: GET 2024-02-01T10:16:53.809 DEBUG:boto:Path: /vclfcn-10/?prefix=_myobj 2024-02-01T10:16:53.809 DEBUG:boto:Data: 2024-02-01T10:16:53.809 DEBUG:boto:Headers: {} 2024-02-01T10:16:53.809 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:53.809 DEBUG:boto:Port: 8000 2024-02-01T10:16:53.809 DEBUG:boto:Params: {} 2024-02-01T10:16:53.809 DEBUG:boto:Token: None 2024-02-01T10:16:53.809 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:53 GMT /vclfcn-10/ 2024-02-01T10:16:53.809 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gxA919U/ZvYiLbGrFtIj/OtV1Jg= 2024-02-01T10:16:53.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gxA919U/ZvYiLbGrFtIj/OtV1Jg=', 'Content-Length': '0'} 2024-02-01T10:16:53.813 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c6c8787e0bb7cc5-0065bb6f95-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:53.814 DEBUG:boto:b'vclfcn-10_myobj1000false_myobj2024-02-01T10:16:36.295Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:16:53.814 DEBUG:boto:path=/vclfcn-10/_myobj 2024-02-01T10:16:53.814 DEBUG:boto:auth_path=/vclfcn-10/_myobj 2024-02-01T10:16:53.814 DEBUG:boto:Method: HEAD 2024-02-01T10:16:53.814 DEBUG:boto:Path: /vclfcn-10/_myobj 2024-02-01T10:16:53.814 DEBUG:boto:Data: 2024-02-01T10:16:53.814 DEBUG:boto:Headers: {} 2024-02-01T10:16:53.815 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:16:53.815 DEBUG:boto:Port: 8001 2024-02-01T10:16:53.815 DEBUG:boto:Params: {} 2024-02-01T10:16:53.815 DEBUG:boto:Token: None 2024-02-01T10:16:53.815 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:53 GMT /vclfcn-10/_myobj 2024-02-01T10:16:53.815 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:oxTtwo9OYae+N2oIkx4J0eCEv9I= 2024-02-01T10:16:53.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:oxTtwo9OYae+N2oIkx4J0eCEv9I=', 'Content-Length': '0'} 2024-02-01T10:16:53.818 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000bf12248d83f9200-0065bb6f95-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:53.818 DEBUG:boto:path=/vclfcn-10/_myobj 2024-02-01T10:16:53.818 DEBUG:boto:auth_path=/vclfcn-10/_myobj 2024-02-01T10:16:53.819 DEBUG:boto:Method: HEAD 2024-02-01T10:16:53.819 DEBUG:boto:Path: /vclfcn-10/_myobj 2024-02-01T10:16:53.819 DEBUG:boto:Data: 2024-02-01T10:16:53.819 DEBUG:boto:Headers: {} 2024-02-01T10:16:53.819 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:53.819 DEBUG:boto:Port: 8000 2024-02-01T10:16:53.819 DEBUG:boto:Params: {} 2024-02-01T10:16:53.819 DEBUG:boto:Token: None 2024-02-01T10:16:53.819 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:53 GMT /vclfcn-10/_myobj 2024-02-01T10:16:53.819 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:oxTtwo9OYae+N2oIkx4J0eCEv9I= 2024-02-01T10:16:53.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:oxTtwo9OYae+N2oIkx4J0eCEv9I=', 'Content-Length': '0'} 2024-02-01T10:16:53.822 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a5597425124d387d-0065bb6f95-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:53.822 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-02-01T10:16:53.822 DEBUG:boto:path=/vclfcn-10/_myobj 2024-02-01T10:16:53.822 DEBUG:boto:auth_path=/vclfcn-10/_myobj 2024-02-01T10:16:53.823 DEBUG:boto:Method: GET 2024-02-01T10:16:53.823 DEBUG:boto:Path: /vclfcn-10/_myobj 2024-02-01T10:16:53.823 DEBUG:boto:Data: 2024-02-01T10:16:53.823 DEBUG:boto:Headers: {} 2024-02-01T10:16:53.823 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:16:53.823 DEBUG:boto:Port: 8001 2024-02-01T10:16:53.823 DEBUG:boto:Params: {} 2024-02-01T10:16:53.823 DEBUG:boto:Token: None 2024-02-01T10:16:53.823 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:53 GMT /vclfcn-10/_myobj 2024-02-01T10:16:53.823 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:B1MufN52R7wPTjWgsu9WESMk80o= 2024-02-01T10:16:53.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:B1MufN52R7wPTjWgsu9WESMk80o=', 'Content-Length': '0'} 2024-02-01T10:16:53.826 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d825e0b15cf13602-0065bb6f95-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:53.872 DEBUG:boto:path=/vclfcn-10/_myobj 2024-02-01T10:16:53.872 DEBUG:boto:auth_path=/vclfcn-10/_myobj 2024-02-01T10:16:53.872 DEBUG:boto:Method: GET 2024-02-01T10:16:53.873 DEBUG:boto:Path: /vclfcn-10/_myobj 2024-02-01T10:16:53.873 DEBUG:boto:Data: 2024-02-01T10:16:53.873 DEBUG:boto:Headers: {} 2024-02-01T10:16:53.873 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:53.873 DEBUG:boto:Port: 8000 2024-02-01T10:16:53.873 DEBUG:boto:Params: {} 2024-02-01T10:16:53.873 DEBUG:boto:Token: None 2024-02-01T10:16:53.873 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:53 GMT /vclfcn-10/_myobj 2024-02-01T10:16:53.873 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:B1MufN52R7wPTjWgsu9WESMk80o= 2024-02-01T10:16:53.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:B1MufN52R7wPTjWgsu9WESMk80o=', 'Content-Length': '0'} 2024-02-01T10:16:53.876 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005171ba793d4d6dc2-0065bb6f95-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:53.877 DEBUG:boto:path=/vclfcn-10/ 2024-02-01T10:16:53.877 DEBUG:boto:auth_path=/vclfcn-10/ 2024-02-01T10:16:53.877 DEBUG:boto:path=/vclfcn-10/?prefix=_myobj 2024-02-01T10:16:53.877 DEBUG:boto:auth_path=/vclfcn-10/?prefix=_myobj 2024-02-01T10:16:53.877 DEBUG:boto:Method: GET 2024-02-01T10:16:53.877 DEBUG:boto:Path: /vclfcn-10/?prefix=_myobj 2024-02-01T10:16:53.877 DEBUG:boto:Data: 2024-02-01T10:16:53.877 DEBUG:boto:Headers: {} 2024-02-01T10:16:53.877 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:16:53.878 DEBUG:boto:Port: 8001 2024-02-01T10:16:53.878 DEBUG:boto:Params: {} 2024-02-01T10:16:53.878 DEBUG:boto:Token: None 2024-02-01T10:16:53.878 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:53 GMT /vclfcn-10/ 2024-02-01T10:16:53.878 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gxA919U/ZvYiLbGrFtIj/OtV1Jg= 2024-02-01T10:16:53.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gxA919U/ZvYiLbGrFtIj/OtV1Jg=', 'Content-Length': '0'} 2024-02-01T10:16:53.882 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045377501e2e8e486-0065bb6f95-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:53.882 DEBUG:boto:b'vclfcn-10_myobj1000false_myobj2024-02-01T10:16:36.295Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:16:53.882 DEBUG:boto:path=/vclfcn-10/ 2024-02-01T10:16:53.883 DEBUG:boto:auth_path=/vclfcn-10/ 2024-02-01T10:16:53.883 DEBUG:boto:path=/vclfcn-10/?prefix=_myobj 2024-02-01T10:16:53.883 DEBUG:boto:auth_path=/vclfcn-10/?prefix=_myobj 2024-02-01T10:16:53.883 DEBUG:boto:Method: GET 2024-02-01T10:16:53.883 DEBUG:boto:Path: /vclfcn-10/?prefix=_myobj 2024-02-01T10:16:53.883 DEBUG:boto:Data: 2024-02-01T10:16:53.883 DEBUG:boto:Headers: {} 2024-02-01T10:16:53.883 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:53.883 DEBUG:boto:Port: 8000 2024-02-01T10:16:53.883 DEBUG:boto:Params: {} 2024-02-01T10:16:53.884 DEBUG:boto:Token: None 2024-02-01T10:16:53.884 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:53 GMT /vclfcn-10/ 2024-02-01T10:16:53.884 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gxA919U/ZvYiLbGrFtIj/OtV1Jg= 2024-02-01T10:16:53.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gxA919U/ZvYiLbGrFtIj/OtV1Jg=', 'Content-Length': '0'} 2024-02-01T10:16:53.888 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000466c4618c32c1b69-0065bb6f95-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:53.932 DEBUG:boto:b'vclfcn-10_myobj1000false_myobj2024-02-01T10:16:36.295Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:16:53.933 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-02-01T10:16:53.933 DEBUG:boto:path=/vclfcn-10/myobj 2024-02-01T10:16:53.933 DEBUG:boto:auth_path=/vclfcn-10/myobj 2024-02-01T10:16:53.933 DEBUG:boto:path=/vclfcn-10/myobj?versionId=null 2024-02-01T10:16:53.933 DEBUG:boto:auth_path=/vclfcn-10/myobj?versionId=null 2024-02-01T10:16:53.933 DEBUG:boto:Method: GET 2024-02-01T10:16:53.933 DEBUG:boto:Path: /vclfcn-10/myobj?versionId=null 2024-02-01T10:16:53.933 DEBUG:boto:Data: 2024-02-01T10:16:53.933 DEBUG:boto:Headers: {} 2024-02-01T10:16:53.933 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:16:53.934 DEBUG:boto:Port: 8001 2024-02-01T10:16:53.934 DEBUG:boto:Params: {} 2024-02-01T10:16:53.934 DEBUG:boto:Token: None 2024-02-01T10:16:53.934 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:53 GMT /vclfcn-10/myobj?versionId=null 2024-02-01T10:16:53.934 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sxKU9eN8gpJoOnexBpKq9gsTXLo= 2024-02-01T10:16:53.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sxKU9eN8gpJoOnexBpKq9gsTXLo=', 'Content-Length': '0'} 2024-02-01T10:16:53.937 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000664f743c3b73fe87-0065bb6f95-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:53.938 DEBUG:boto:path=/vclfcn-10/myobj 2024-02-01T10:16:53.938 DEBUG:boto:auth_path=/vclfcn-10/myobj 2024-02-01T10:16:53.938 DEBUG:boto:path=/vclfcn-10/myobj?versionId=null 2024-02-01T10:16:53.938 DEBUG:boto:auth_path=/vclfcn-10/myobj?versionId=null 2024-02-01T10:16:53.938 DEBUG:boto:Method: GET 2024-02-01T10:16:53.938 DEBUG:boto:Path: /vclfcn-10/myobj?versionId=null 2024-02-01T10:16:53.938 DEBUG:boto:Data: 2024-02-01T10:16:53.939 DEBUG:boto:Headers: {} 2024-02-01T10:16:53.939 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:53.939 DEBUG:boto:Port: 8000 2024-02-01T10:16:53.939 DEBUG:boto:Params: {} 2024-02-01T10:16:53.939 DEBUG:boto:Token: None 2024-02-01T10:16:53.939 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:53 GMT /vclfcn-10/myobj?versionId=null 2024-02-01T10:16:53.939 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sxKU9eN8gpJoOnexBpKq9gsTXLo= 2024-02-01T10:16:53.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sxKU9eN8gpJoOnexBpKq9gsTXLo=', 'Content-Length': '0'} 2024-02-01T10:16:53.942 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009eea291dc2bcab38-0065bb6f95-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:53.943 DEBUG:boto:path=/vclfcn-10/myobj 2024-02-01T10:16:53.943 DEBUG:boto:auth_path=/vclfcn-10/myobj 2024-02-01T10:16:53.943 DEBUG:boto:path=/vclfcn-10/myobj?versionId=null 2024-02-01T10:16:53.943 DEBUG:boto:auth_path=/vclfcn-10/myobj?versionId=null 2024-02-01T10:16:53.943 DEBUG:boto:Method: HEAD 2024-02-01T10:16:53.943 DEBUG:boto:Path: /vclfcn-10/myobj?versionId=null 2024-02-01T10:16:53.943 DEBUG:boto:Data: 2024-02-01T10:16:53.943 DEBUG:boto:Headers: {} 2024-02-01T10:16:53.943 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:16:53.943 DEBUG:boto:Port: 8001 2024-02-01T10:16:53.944 DEBUG:boto:Params: {} 2024-02-01T10:16:53.944 DEBUG:boto:Token: None 2024-02-01T10:16:53.944 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:53 GMT /vclfcn-10/myobj?versionId=null 2024-02-01T10:16:53.944 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:UEXEP50NFe/1FZMEJS/VFpPCrEk= 2024-02-01T10:16:53.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:UEXEP50NFe/1FZMEJS/VFpPCrEk=', 'Content-Length': '0'} 2024-02-01T10:16:53.947 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c95b800086f6a0ac-0065bb6f95-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:53.947 DEBUG:boto:path=/vclfcn-10/myobj 2024-02-01T10:16:53.947 DEBUG:boto:auth_path=/vclfcn-10/myobj 2024-02-01T10:16:53.947 DEBUG:boto:path=/vclfcn-10/myobj?versionId=null 2024-02-01T10:16:53.948 DEBUG:boto:auth_path=/vclfcn-10/myobj?versionId=null 2024-02-01T10:16:53.948 DEBUG:boto:Method: HEAD 2024-02-01T10:16:53.948 DEBUG:boto:Path: /vclfcn-10/myobj?versionId=null 2024-02-01T10:16:53.948 DEBUG:boto:Data: 2024-02-01T10:16:53.948 DEBUG:boto:Headers: {} 2024-02-01T10:16:53.948 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:53.948 DEBUG:boto:Port: 8000 2024-02-01T10:16:53.948 DEBUG:boto:Params: {} 2024-02-01T10:16:53.949 DEBUG:boto:Token: None 2024-02-01T10:16:53.949 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:53 GMT /vclfcn-10/myobj?versionId=null 2024-02-01T10:16:53.949 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:UEXEP50NFe/1FZMEJS/VFpPCrEk= 2024-02-01T10:16:53.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:UEXEP50NFe/1FZMEJS/VFpPCrEk=', 'Content-Length': '0'} 2024-02-01T10:16:53.951 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000066abc21c91fc634b-0065bb6f95-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:53.952 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-02-01T10:16:53.952 DEBUG:boto:path=/vclfcn-10/myobj 2024-02-01T10:16:53.952 DEBUG:boto:auth_path=/vclfcn-10/myobj 2024-02-01T10:16:53.952 DEBUG:boto:path=/vclfcn-10/myobj?versionId=null 2024-02-01T10:16:53.952 DEBUG:boto:auth_path=/vclfcn-10/myobj?versionId=null 2024-02-01T10:16:53.952 DEBUG:boto:Method: GET 2024-02-01T10:16:53.952 DEBUG:boto:Path: /vclfcn-10/myobj?versionId=null 2024-02-01T10:16:53.952 DEBUG:boto:Data: 2024-02-01T10:16:53.953 DEBUG:boto:Headers: {} 2024-02-01T10:16:53.953 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:16:53.953 DEBUG:boto:Port: 8001 2024-02-01T10:16:53.953 DEBUG:boto:Params: {} 2024-02-01T10:16:53.953 DEBUG:boto:Token: None 2024-02-01T10:16:53.953 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:53 GMT /vclfcn-10/myobj?versionId=null 2024-02-01T10:16:53.953 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sxKU9eN8gpJoOnexBpKq9gsTXLo= 2024-02-01T10:16:53.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sxKU9eN8gpJoOnexBpKq9gsTXLo=', 'Content-Length': '0'} 2024-02-01T10:16:53.956 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000010640cc36f740d79-0065bb6f95-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:54.001 DEBUG:boto:path=/vclfcn-10/myobj 2024-02-01T10:16:54.001 DEBUG:boto:auth_path=/vclfcn-10/myobj 2024-02-01T10:16:54.001 DEBUG:boto:path=/vclfcn-10/myobj?versionId=null 2024-02-01T10:16:54.001 DEBUG:boto:auth_path=/vclfcn-10/myobj?versionId=null 2024-02-01T10:16:54.001 DEBUG:boto:Method: GET 2024-02-01T10:16:54.001 DEBUG:boto:Path: /vclfcn-10/myobj?versionId=null 2024-02-01T10:16:54.002 DEBUG:boto:Data: 2024-02-01T10:16:54.002 DEBUG:boto:Headers: {} 2024-02-01T10:16:54.002 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:54.002 DEBUG:boto:Port: 8000 2024-02-01T10:16:54.002 DEBUG:boto:Params: {} 2024-02-01T10:16:54.003 DEBUG:boto:Token: None 2024-02-01T10:16:54.003 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:54 GMT /vclfcn-10/myobj?versionId=null 2024-02-01T10:16:54.003 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:dC7WztFCD+JUgqDP3D6wAq+bAlI= 2024-02-01T10:16:54.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dC7WztFCD+JUgqDP3D6wAq+bAlI=', 'Content-Length': '0'} 2024-02-01T10:16:54.006 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d654a8ee46ee4cc7-0065bb6f96-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:54.007 DEBUG:boto:path=/vclfcn-10/ 2024-02-01T10:16:54.007 DEBUG:boto:auth_path=/vclfcn-10/ 2024-02-01T10:16:54.007 DEBUG:boto:path=/vclfcn-10/?prefix=myobj 2024-02-01T10:16:54.007 DEBUG:boto:auth_path=/vclfcn-10/?prefix=myobj 2024-02-01T10:16:54.007 DEBUG:boto:Method: GET 2024-02-01T10:16:54.007 DEBUG:boto:Path: /vclfcn-10/?prefix=myobj 2024-02-01T10:16:54.007 DEBUG:boto:Data: 2024-02-01T10:16:54.008 DEBUG:boto:Headers: {} 2024-02-01T10:16:54.008 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:16:54.008 DEBUG:boto:Port: 8001 2024-02-01T10:16:54.008 DEBUG:boto:Params: {} 2024-02-01T10:16:54.008 DEBUG:boto:Token: None 2024-02-01T10:16:54.008 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:54 GMT /vclfcn-10/ 2024-02-01T10:16:54.008 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RdUKjmXYD8oWcT3G8xohU4JxP/A= 2024-02-01T10:16:54.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RdUKjmXYD8oWcT3G8xohU4JxP/A=', 'Content-Length': '0'} 2024-02-01T10:16:54.013 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f16b11d7932a2bf-0065bb6f96-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:54.060 DEBUG:boto:b'vclfcn-10myobj1000falsemyobj2024-02-01T10:16:36.267Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:16:54.061 DEBUG:boto:path=/vclfcn-10/ 2024-02-01T10:16:54.061 DEBUG:boto:auth_path=/vclfcn-10/ 2024-02-01T10:16:54.061 DEBUG:boto:path=/vclfcn-10/?prefix=myobj 2024-02-01T10:16:54.061 DEBUG:boto:auth_path=/vclfcn-10/?prefix=myobj 2024-02-01T10:16:54.061 DEBUG:boto:Method: GET 2024-02-01T10:16:54.061 DEBUG:boto:Path: /vclfcn-10/?prefix=myobj 2024-02-01T10:16:54.061 DEBUG:boto:Data: 2024-02-01T10:16:54.061 DEBUG:boto:Headers: {} 2024-02-01T10:16:54.061 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:54.061 DEBUG:boto:Port: 8000 2024-02-01T10:16:54.061 DEBUG:boto:Params: {} 2024-02-01T10:16:54.062 DEBUG:boto:Token: None 2024-02-01T10:16:54.062 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:54 GMT /vclfcn-10/ 2024-02-01T10:16:54.062 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RdUKjmXYD8oWcT3G8xohU4JxP/A= 2024-02-01T10:16:54.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RdUKjmXYD8oWcT3G8xohU4JxP/A=', 'Content-Length': '0'} 2024-02-01T10:16:54.066 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1dca66252ef439a-0065bb6f96-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:54.067 DEBUG:boto:b'vclfcn-10myobj1000falsemyobj2024-02-01T10:16:36.267Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:16:54.067 DEBUG:boto:path=/vclfcn-10/myobj 2024-02-01T10:16:54.067 DEBUG:boto:auth_path=/vclfcn-10/myobj 2024-02-01T10:16:54.067 DEBUG:boto:Method: HEAD 2024-02-01T10:16:54.067 DEBUG:boto:Path: /vclfcn-10/myobj 2024-02-01T10:16:54.068 DEBUG:boto:Data: 2024-02-01T10:16:54.068 DEBUG:boto:Headers: {} 2024-02-01T10:16:54.068 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:16:54.068 DEBUG:boto:Port: 8001 2024-02-01T10:16:54.068 DEBUG:boto:Params: {} 2024-02-01T10:16:54.068 DEBUG:boto:Token: None 2024-02-01T10:16:54.068 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:54 GMT /vclfcn-10/myobj 2024-02-01T10:16:54.068 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:pEe+lV9pXBY58VUa6n9RkO2ZqUc= 2024-02-01T10:16:54.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:pEe+lV9pXBY58VUa6n9RkO2ZqUc=', 'Content-Length': '0'} 2024-02-01T10:16:54.071 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000af03efb41acf26bd-0065bb6f96-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:54.071 DEBUG:boto:path=/vclfcn-10/myobj 2024-02-01T10:16:54.071 DEBUG:boto:auth_path=/vclfcn-10/myobj 2024-02-01T10:16:54.072 DEBUG:boto:Method: HEAD 2024-02-01T10:16:54.072 DEBUG:boto:Path: /vclfcn-10/myobj 2024-02-01T10:16:54.072 DEBUG:boto:Data: 2024-02-01T10:16:54.072 DEBUG:boto:Headers: {} 2024-02-01T10:16:54.072 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:54.072 DEBUG:boto:Port: 8000 2024-02-01T10:16:54.072 DEBUG:boto:Params: {} 2024-02-01T10:16:54.072 DEBUG:boto:Token: None 2024-02-01T10:16:54.072 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:16:54 GMT /vclfcn-10/myobj 2024-02-01T10:16:54.072 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:pEe+lV9pXBY58VUa6n9RkO2ZqUc= 2024-02-01T10:16:54.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:pEe+lV9pXBY58VUa6n9RkO2ZqUc=', 'Content-Length': '0'} 2024-02-01T10:16:54.075 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cd5363be06287205-0065bb6f96-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:54.075 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-02-01T10:16:54.076 DEBUG:boto:path=/vclfcn-10/myobj 2024-02-01T10:16:54.076 DEBUG:boto:auth_path=/vclfcn-10/myobj 2024-02-01T10:16:54.076 DEBUG:boto:Method: GET 2024-02-01T10:16:54.076 DEBUG:boto:Path: /vclfcn-10/myobj 2024-02-01T10:16:54.076 DEBUG:boto:Data: 2024-02-01T10:16:54.076 DEBUG:boto:Headers: {} 2024-02-01T10:16:54.076 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:16:54.076 DEBUG:boto:Port: 8001 2024-02-01T10:16:54.076 DEBUG:boto:Params: {} 2024-02-01T10:16:54.076 DEBUG:boto:Token: None 2024-02-01T10:16:54.077 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:54 GMT /vclfcn-10/myobj 2024-02-01T10:16:54.077 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:aUeDUOwLqpzkWsdrX767QqDNmHg= 2024-02-01T10:16:54.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:aUeDUOwLqpzkWsdrX767QqDNmHg=', 'Content-Length': '0'} 2024-02-01T10:16:54.079 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005bb344b83dfdab7d-0065bb6f96-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:54.120 DEBUG:boto:path=/vclfcn-10/myobj 2024-02-01T10:16:54.120 DEBUG:boto:auth_path=/vclfcn-10/myobj 2024-02-01T10:16:54.120 DEBUG:boto:Method: GET 2024-02-01T10:16:54.120 DEBUG:boto:Path: /vclfcn-10/myobj 2024-02-01T10:16:54.121 DEBUG:boto:Data: 2024-02-01T10:16:54.121 DEBUG:boto:Headers: {} 2024-02-01T10:16:54.121 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:54.121 DEBUG:boto:Port: 8000 2024-02-01T10:16:54.121 DEBUG:boto:Params: {} 2024-02-01T10:16:54.121 DEBUG:boto:Token: None 2024-02-01T10:16:54.121 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:54 GMT /vclfcn-10/myobj 2024-02-01T10:16:54.121 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:aUeDUOwLqpzkWsdrX767QqDNmHg= 2024-02-01T10:16:54.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:aUeDUOwLqpzkWsdrX767QqDNmHg=', 'Content-Length': '0'} 2024-02-01T10:16:54.124 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a7d453502a04be76-0065bb6f96-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:54.124 DEBUG:boto:path=/vclfcn-10/ 2024-02-01T10:16:54.125 DEBUG:boto:auth_path=/vclfcn-10/ 2024-02-01T10:16:54.125 DEBUG:boto:path=/vclfcn-10/?prefix=myobj 2024-02-01T10:16:54.125 DEBUG:boto:auth_path=/vclfcn-10/?prefix=myobj 2024-02-01T10:16:54.125 DEBUG:boto:Method: GET 2024-02-01T10:16:54.125 DEBUG:boto:Path: /vclfcn-10/?prefix=myobj 2024-02-01T10:16:54.125 DEBUG:boto:Data: 2024-02-01T10:16:54.125 DEBUG:boto:Headers: {} 2024-02-01T10:16:54.125 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:16:54.125 DEBUG:boto:Port: 8001 2024-02-01T10:16:54.126 DEBUG:boto:Params: {} 2024-02-01T10:16:54.126 DEBUG:boto:Token: None 2024-02-01T10:16:54.126 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:54 GMT /vclfcn-10/ 2024-02-01T10:16:54.126 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RdUKjmXYD8oWcT3G8xohU4JxP/A= 2024-02-01T10:16:54.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RdUKjmXYD8oWcT3G8xohU4JxP/A=', 'Content-Length': '0'} 2024-02-01T10:16:54.131 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf97fe8da93510c3-0065bb6f96-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:54.132 DEBUG:boto:b'vclfcn-10myobj1000falsemyobj2024-02-01T10:16:36.267Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:16:54.132 DEBUG:boto:path=/vclfcn-10/ 2024-02-01T10:16:54.132 DEBUG:boto:auth_path=/vclfcn-10/ 2024-02-01T10:16:54.132 DEBUG:boto:path=/vclfcn-10/?prefix=myobj 2024-02-01T10:16:54.132 DEBUG:boto:auth_path=/vclfcn-10/?prefix=myobj 2024-02-01T10:16:54.132 DEBUG:boto:Method: GET 2024-02-01T10:16:54.133 DEBUG:boto:Path: /vclfcn-10/?prefix=myobj 2024-02-01T10:16:54.133 DEBUG:boto:Data: 2024-02-01T10:16:54.133 DEBUG:boto:Headers: {} 2024-02-01T10:16:54.133 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:16:54.133 DEBUG:boto:Port: 8000 2024-02-01T10:16:54.133 DEBUG:boto:Params: {} 2024-02-01T10:16:54.133 DEBUG:boto:Token: None 2024-02-01T10:16:54.133 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:16:54 GMT /vclfcn-10/ 2024-02-01T10:16:54.133 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RdUKjmXYD8oWcT3G8xohU4JxP/A= 2024-02-01T10:16:54.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:16:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RdUKjmXYD8oWcT3G8xohU4JxP/A=', 'Content-Length': '0'} 2024-02-01T10:16:54.138 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003918f32945a5445c-0065bb6f96-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:16:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:16:54.138 DEBUG:boto:b'vclfcn-10myobj1000falsemyobj2024-02-01T10:16:36.267Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:16:54.139 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-10 zones={test-zone3, test-zone1} 2024-02-01T10:16:54.139 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:16:54.139 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:16:54.139 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-11 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:16:54.190 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:16:54.190 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:16:54.253 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:16:54.252+0000 7f871e8b4980 1 waiting to reach incremental sync.. 2024-02-01T10:16:59.255 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:16:59.252+0000 7f871e8b4980 1 waiting to reach incremental sync.. 2024-02-01T10:17:04.256 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:17:04.256+0000 7f871e8b4980 1 waiting to reach incremental sync.. 2024-02-01T10:17:09.257 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:17:09.256+0000 7f871e8b4980 1 waiting to reach incremental sync.. 2024-02-01T10:17:14.258 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:17:14.256+0000 7f871e8b4980 1 waiting to reach incremental sync.. 2024-02-01T10:17:19.259 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:17:19.256+0000 7f871e8b4980 1 waiting to reach incremental sync.. 2024-02-01T10:17:24.260 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:17:24.260+0000 7f871e8b4980 1 waiting to reach incremental sync.. 2024-02-01T10:17:29.261 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:17:29.260+0000 7f871e8b4980 1 waiting to reach incremental sync.. 2024-02-01T10:17:34.262 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:17:34.260+0000 7f871e8b4980 1 waiting to reach incremental sync.. 2024-02-01T10:17:39.263 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:17:39.259+0000 7f871e8b4980 1 waiting to reach incremental sync.. 2024-02-01T10:17:44.264 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:17:44.263+0000 7f871e8b4980 1 waiting to reach incremental sync.. 2024-02-01T10:17:49.265 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:17:49.263+0000 7f871e8b4980 1 waiting to reach incremental sync.. 2024-02-01T10:17:54.267 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:17:54.263+0000 7f871e8b4980 1 waiting to reach incremental sync.. 2024-02-01T10:17:59.268 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:17:59.267+0000 7f871e8b4980 1 waiting to reach incremental sync.. 2024-02-01T10:18:04.268 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:18:04.267+0000 7f871e8b4980 1 waiting to reach incremental sync.. 2024-02-01T10:18:09.272 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:18:09.271+0000 7f871e8b4980 1 bucket sync caught up with source: 2024-02-01T10:18:09.272 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, 00000000001.31.6, , 00000000001.12.6, , , , 00000000001.14.6, 00000000001.6.6, , ] 2024-02-01T10:18:09.272 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, 00000000001.31.6, , 00000000001.12.6, , , , 00000000001.14.6, 00000000001.6.6, , ] 2024-02-01T10:18:09.272 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:18:09.271+0000 7f871e8b4980 0 bucket checkpoint complete 2024-02-01T10:18:09.311 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-11 zones={test-zone1, test-zone2} 2024-02-01T10:18:09.311 DEBUG:boto:path=/vclfcn-11/ 2024-02-01T10:18:09.311 DEBUG:boto:auth_path=/vclfcn-11/ 2024-02-01T10:18:09.311 DEBUG:boto:Method: HEAD 2024-02-01T10:18:09.312 DEBUG:boto:Path: /vclfcn-11/ 2024-02-01T10:18:09.312 DEBUG:boto:Data: 2024-02-01T10:18:09.312 DEBUG:boto:Headers: {} 2024-02-01T10:18:09.312 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:09.312 DEBUG:boto:Port: 8000 2024-02-01T10:18:09.312 DEBUG:boto:Params: {} 2024-02-01T10:18:09.313 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T10:18:09.313 DEBUG:boto:Token: None 2024-02-01T10:18:09.314 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:09 GMT /vclfcn-11/ 2024-02-01T10:18:09.314 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:anp9yXn6cJgW+20yq1MVxl+HZKc= 2024-02-01T10:18:09.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:anp9yXn6cJgW+20yq1MVxl+HZKc=', 'Content-Length': '0'} 2024-02-01T10:18:09.323 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005e679f72508d1607-0065bb6fe1-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:18:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:09.323 DEBUG:boto:path=/vclfcn-11/ 2024-02-01T10:18:09.323 DEBUG:boto:auth_path=/vclfcn-11/ 2024-02-01T10:18:09.323 DEBUG:boto:Method: HEAD 2024-02-01T10:18:09.323 DEBUG:boto:Path: /vclfcn-11/ 2024-02-01T10:18:09.323 DEBUG:boto:Data: 2024-02-01T10:18:09.323 DEBUG:boto:Headers: {} 2024-02-01T10:18:09.324 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:09.324 DEBUG:boto:Port: 8000 2024-02-01T10:18:09.324 DEBUG:boto:Params: {} 2024-02-01T10:18:09.324 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T10:18:09.324 DEBUG:boto:Token: None 2024-02-01T10:18:09.325 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:09 GMT /vclfcn-11/ 2024-02-01T10:18:09.325 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:anp9yXn6cJgW+20yq1MVxl+HZKc= 2024-02-01T10:18:09.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:anp9yXn6cJgW+20yq1MVxl+HZKc=', 'Content-Length': '0'} 2024-02-01T10:18:09.331 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009868b609236de31f-0065bb6fe1-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:18:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:09.331 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:18:09.331 DEBUG:boto:path=/vclfcn-11/ 2024-02-01T10:18:09.331 DEBUG:boto:auth_path=/vclfcn-11/ 2024-02-01T10:18:09.331 DEBUG:boto:path=/vclfcn-11/?versions&max-keys=999 2024-02-01T10:18:09.332 DEBUG:boto:auth_path=/vclfcn-11/?versions&max-keys=999 2024-02-01T10:18:09.332 DEBUG:boto:Method: GET 2024-02-01T10:18:09.332 DEBUG:boto:Path: /vclfcn-11/?versions&max-keys=999 2024-02-01T10:18:09.332 DEBUG:boto:Data: 2024-02-01T10:18:09.332 DEBUG:boto:Headers: {} 2024-02-01T10:18:09.332 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:09.332 DEBUG:boto:Port: 8000 2024-02-01T10:18:09.332 DEBUG:boto:Params: {} 2024-02-01T10:18:09.332 DEBUG:boto:Token: None 2024-02-01T10:18:09.333 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:09 GMT /vclfcn-11/?versions 2024-02-01T10:18:09.333 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nSQBNumyACibSPofrLsy9jkhvoA= 2024-02-01T10:18:09.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nSQBNumyACibSPofrLsy9jkhvoA=', 'Content-Length': '0'} 2024-02-01T10:18:09.336 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003deccac112346340-0065bb6fe1-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:09.337 DEBUG:boto:b'vclfcn-11999false&nulltrue2024-02-01T10:16:36.539Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-02-01T10:16:36.512Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-02-01T10:16:36.463Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-02-01T10:16:36.434Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:09.338 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-02-01T10:18:09.338 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-02-01T10:18:09.338 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-02-01T10:18:09.338 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-02-01T10:18:09.338 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:18:09.338 DEBUG:boto:path=/vclfcn-11/ 2024-02-01T10:18:09.338 DEBUG:boto:auth_path=/vclfcn-11/ 2024-02-01T10:18:09.339 DEBUG:boto:path=/vclfcn-11/?versions&max-keys=999 2024-02-01T10:18:09.339 DEBUG:boto:auth_path=/vclfcn-11/?versions&max-keys=999 2024-02-01T10:18:09.339 DEBUG:boto:Method: GET 2024-02-01T10:18:09.339 DEBUG:boto:Path: /vclfcn-11/?versions&max-keys=999 2024-02-01T10:18:09.339 DEBUG:boto:Data: 2024-02-01T10:18:09.339 DEBUG:boto:Headers: {} 2024-02-01T10:18:09.339 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:09.339 DEBUG:boto:Port: 8000 2024-02-01T10:18:09.340 DEBUG:boto:Params: {} 2024-02-01T10:18:09.340 DEBUG:boto:Token: None 2024-02-01T10:18:09.340 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:09 GMT /vclfcn-11/?versions 2024-02-01T10:18:09.340 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nSQBNumyACibSPofrLsy9jkhvoA= 2024-02-01T10:18:09.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nSQBNumyACibSPofrLsy9jkhvoA=', 'Content-Length': '0'} 2024-02-01T10:18:09.344 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cba877bd93fbf6aa-0065bb6fe1-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:09.345 DEBUG:boto:b'vclfcn-11999false&nulltrue2024-02-01T10:16:36.539Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-02-01T10:16:36.512Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-02-01T10:16:36.463Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-02-01T10:16:36.434Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:09.345 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-02-01T10:18:09.345 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-02-01T10:18:09.345 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-02-01T10:18:09.345 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-02-01T10:18:09.346 DEBUG:boto:path=/vclfcn-11/ 2024-02-01T10:18:09.346 DEBUG:boto:auth_path=/vclfcn-11/ 2024-02-01T10:18:09.346 DEBUG:boto:path=/vclfcn-11/?versions&max-keys=999 2024-02-01T10:18:09.346 DEBUG:boto:auth_path=/vclfcn-11/?versions&max-keys=999 2024-02-01T10:18:09.346 DEBUG:boto:Method: GET 2024-02-01T10:18:09.346 DEBUG:boto:Path: /vclfcn-11/?versions&max-keys=999 2024-02-01T10:18:09.346 DEBUG:boto:Data: 2024-02-01T10:18:09.346 DEBUG:boto:Headers: {} 2024-02-01T10:18:09.346 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:09.346 DEBUG:boto:Port: 8000 2024-02-01T10:18:09.347 DEBUG:boto:Params: {} 2024-02-01T10:18:09.347 DEBUG:boto:Token: None 2024-02-01T10:18:09.347 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:09 GMT /vclfcn-11/?versions 2024-02-01T10:18:09.347 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nSQBNumyACibSPofrLsy9jkhvoA= 2024-02-01T10:18:09.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nSQBNumyACibSPofrLsy9jkhvoA=', 'Content-Length': '0'} 2024-02-01T10:18:09.351 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf5b28d29640e987-0065bb6fe1-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:09.351 DEBUG:boto:b'vclfcn-11999false&nulltrue2024-02-01T10:16:36.539Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-02-01T10:16:36.512Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-02-01T10:16:36.463Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-02-01T10:16:36.434Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:09.351 DEBUG:boto:path=/vclfcn-11/ 2024-02-01T10:18:09.352 DEBUG:boto:auth_path=/vclfcn-11/ 2024-02-01T10:18:09.352 DEBUG:boto:path=/vclfcn-11/?versions&max-keys=999 2024-02-01T10:18:09.352 DEBUG:boto:auth_path=/vclfcn-11/?versions&max-keys=999 2024-02-01T10:18:09.352 DEBUG:boto:Method: GET 2024-02-01T10:18:09.352 DEBUG:boto:Path: /vclfcn-11/?versions&max-keys=999 2024-02-01T10:18:09.352 DEBUG:boto:Data: 2024-02-01T10:18:09.352 DEBUG:boto:Headers: {} 2024-02-01T10:18:09.352 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:09.352 DEBUG:boto:Port: 8000 2024-02-01T10:18:09.353 DEBUG:boto:Params: {} 2024-02-01T10:18:09.353 DEBUG:boto:Token: None 2024-02-01T10:18:09.353 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:09 GMT /vclfcn-11/?versions 2024-02-01T10:18:09.353 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nSQBNumyACibSPofrLsy9jkhvoA= 2024-02-01T10:18:09.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nSQBNumyACibSPofrLsy9jkhvoA=', 'Content-Length': '0'} 2024-02-01T10:18:09.357 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e6a5a4016ef341f-0065bb6fe1-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:09.358 DEBUG:boto:b'vclfcn-11999false&nulltrue2024-02-01T10:16:36.539Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-02-01T10:16:36.512Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-02-01T10:16:36.463Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-02-01T10:16:36.434Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:09.358 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-02-01T10:18:09.358 DEBUG:boto:path=/vclfcn-11/%26 2024-02-01T10:18:09.358 DEBUG:boto:auth_path=/vclfcn-11/%26 2024-02-01T10:18:09.359 DEBUG:boto:path=/vclfcn-11/%26?versionId=null 2024-02-01T10:18:09.359 DEBUG:boto:auth_path=/vclfcn-11/%26?versionId=null 2024-02-01T10:18:09.359 DEBUG:boto:Method: GET 2024-02-01T10:18:09.359 DEBUG:boto:Path: /vclfcn-11/%26?versionId=null 2024-02-01T10:18:09.359 DEBUG:boto:Data: 2024-02-01T10:18:09.359 DEBUG:boto:Headers: {} 2024-02-01T10:18:09.359 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:09.359 DEBUG:boto:Port: 8000 2024-02-01T10:18:09.359 DEBUG:boto:Params: {} 2024-02-01T10:18:09.360 DEBUG:boto:Token: None 2024-02-01T10:18:09.360 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:09 GMT /vclfcn-11/%26?versionId=null 2024-02-01T10:18:09.360 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NoPOTx9KBFmdm2B99Xn++B0lIxo= 2024-02-01T10:18:09.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NoPOTx9KBFmdm2B99Xn++B0lIxo=', 'Content-Length': '0'} 2024-02-01T10:18:09.363 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005e9ce81c24231562-0065bb6fe1-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:09.363 DEBUG:boto:path=/vclfcn-11/%26 2024-02-01T10:18:09.363 DEBUG:boto:auth_path=/vclfcn-11/%26 2024-02-01T10:18:09.363 DEBUG:boto:path=/vclfcn-11/%26?versionId=null 2024-02-01T10:18:09.364 DEBUG:boto:auth_path=/vclfcn-11/%26?versionId=null 2024-02-01T10:18:09.364 DEBUG:boto:Method: GET 2024-02-01T10:18:09.364 DEBUG:boto:Path: /vclfcn-11/%26?versionId=null 2024-02-01T10:18:09.364 DEBUG:boto:Data: 2024-02-01T10:18:09.364 DEBUG:boto:Headers: {} 2024-02-01T10:18:09.364 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:09.364 DEBUG:boto:Port: 8000 2024-02-01T10:18:09.364 DEBUG:boto:Params: {} 2024-02-01T10:18:09.364 DEBUG:boto:Token: None 2024-02-01T10:18:09.365 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:09 GMT /vclfcn-11/%26?versionId=null 2024-02-01T10:18:09.365 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NoPOTx9KBFmdm2B99Xn++B0lIxo= 2024-02-01T10:18:09.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NoPOTx9KBFmdm2B99Xn++B0lIxo=', 'Content-Length': '0'} 2024-02-01T10:18:09.368 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000adc86ab4ca8c36c4-0065bb6fe1-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:09.408 DEBUG:boto:path=/vclfcn-11/%26 2024-02-01T10:18:09.408 DEBUG:boto:auth_path=/vclfcn-11/%26 2024-02-01T10:18:09.408 DEBUG:boto:path=/vclfcn-11/%26?versionId=null 2024-02-01T10:18:09.408 DEBUG:boto:auth_path=/vclfcn-11/%26?versionId=null 2024-02-01T10:18:09.408 DEBUG:boto:Method: HEAD 2024-02-01T10:18:09.409 DEBUG:boto:Path: /vclfcn-11/%26?versionId=null 2024-02-01T10:18:09.409 DEBUG:boto:Data: 2024-02-01T10:18:09.409 DEBUG:boto:Headers: {} 2024-02-01T10:18:09.409 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:09.409 DEBUG:boto:Port: 8000 2024-02-01T10:18:09.409 DEBUG:boto:Params: {} 2024-02-01T10:18:09.409 DEBUG:boto:Token: None 2024-02-01T10:18:09.409 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:09 GMT /vclfcn-11/%26?versionId=null 2024-02-01T10:18:09.409 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:3t3rpQPYjTcTbapvq+dzYzISejI= 2024-02-01T10:18:09.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:3t3rpQPYjTcTbapvq+dzYzISejI=', 'Content-Length': '0'} 2024-02-01T10:18:09.412 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b3a72dcafac4e325-0065bb6fe1-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:09.413 DEBUG:boto:path=/vclfcn-11/%26 2024-02-01T10:18:09.413 DEBUG:boto:auth_path=/vclfcn-11/%26 2024-02-01T10:18:09.413 DEBUG:boto:path=/vclfcn-11/%26?versionId=null 2024-02-01T10:18:09.413 DEBUG:boto:auth_path=/vclfcn-11/%26?versionId=null 2024-02-01T10:18:09.413 DEBUG:boto:Method: HEAD 2024-02-01T10:18:09.413 DEBUG:boto:Path: /vclfcn-11/%26?versionId=null 2024-02-01T10:18:09.413 DEBUG:boto:Data: 2024-02-01T10:18:09.413 DEBUG:boto:Headers: {} 2024-02-01T10:18:09.413 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:09.413 DEBUG:boto:Port: 8000 2024-02-01T10:18:09.413 DEBUG:boto:Params: {} 2024-02-01T10:18:09.414 DEBUG:boto:Token: None 2024-02-01T10:18:09.414 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:09 GMT /vclfcn-11/%26?versionId=null 2024-02-01T10:18:09.414 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:3t3rpQPYjTcTbapvq+dzYzISejI= 2024-02-01T10:18:09.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:3t3rpQPYjTcTbapvq+dzYzISejI=', 'Content-Length': '0'} 2024-02-01T10:18:09.416 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009222209d349025d1-0065bb6fe1-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:09.417 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-02-01T10:18:09.417 DEBUG:boto:path=/vclfcn-11/%26 2024-02-01T10:18:09.417 DEBUG:boto:auth_path=/vclfcn-11/%26 2024-02-01T10:18:09.417 DEBUG:boto:path=/vclfcn-11/%26?versionId=null 2024-02-01T10:18:09.417 DEBUG:boto:auth_path=/vclfcn-11/%26?versionId=null 2024-02-01T10:18:09.417 DEBUG:boto:Method: GET 2024-02-01T10:18:09.417 DEBUG:boto:Path: /vclfcn-11/%26?versionId=null 2024-02-01T10:18:09.417 DEBUG:boto:Data: 2024-02-01T10:18:09.418 DEBUG:boto:Headers: {} 2024-02-01T10:18:09.418 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:09.418 DEBUG:boto:Port: 8000 2024-02-01T10:18:09.418 DEBUG:boto:Params: {} 2024-02-01T10:18:09.418 DEBUG:boto:Token: None 2024-02-01T10:18:09.418 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:09 GMT /vclfcn-11/%26?versionId=null 2024-02-01T10:18:09.418 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NoPOTx9KBFmdm2B99Xn++B0lIxo= 2024-02-01T10:18:09.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NoPOTx9KBFmdm2B99Xn++B0lIxo=', 'Content-Length': '0'} 2024-02-01T10:18:09.421 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000518cd12d21bfd840-0065bb6fe1-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:09.464 DEBUG:boto:path=/vclfcn-11/%26 2024-02-01T10:18:09.464 DEBUG:boto:auth_path=/vclfcn-11/%26 2024-02-01T10:18:09.464 DEBUG:boto:path=/vclfcn-11/%26?versionId=null 2024-02-01T10:18:09.464 DEBUG:boto:auth_path=/vclfcn-11/%26?versionId=null 2024-02-01T10:18:09.465 DEBUG:boto:Method: GET 2024-02-01T10:18:09.465 DEBUG:boto:Path: /vclfcn-11/%26?versionId=null 2024-02-01T10:18:09.465 DEBUG:boto:Data: 2024-02-01T10:18:09.465 DEBUG:boto:Headers: {} 2024-02-01T10:18:09.465 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:09.465 DEBUG:boto:Port: 8000 2024-02-01T10:18:09.465 DEBUG:boto:Params: {} 2024-02-01T10:18:09.465 DEBUG:boto:Token: None 2024-02-01T10:18:09.465 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:09 GMT /vclfcn-11/%26?versionId=null 2024-02-01T10:18:09.466 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NoPOTx9KBFmdm2B99Xn++B0lIxo= 2024-02-01T10:18:09.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NoPOTx9KBFmdm2B99Xn++B0lIxo=', 'Content-Length': '0'} 2024-02-01T10:18:09.469 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a1e176cfaafb39b7-0065bb6fe1-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:09.469 DEBUG:boto:path=/vclfcn-11/ 2024-02-01T10:18:09.469 DEBUG:boto:auth_path=/vclfcn-11/ 2024-02-01T10:18:09.469 DEBUG:boto:path=/vclfcn-11/?prefix=%26 2024-02-01T10:18:09.469 DEBUG:boto:auth_path=/vclfcn-11/?prefix=%26 2024-02-01T10:18:09.470 DEBUG:boto:Method: GET 2024-02-01T10:18:09.470 DEBUG:boto:Path: /vclfcn-11/?prefix=%26 2024-02-01T10:18:09.470 DEBUG:boto:Data: 2024-02-01T10:18:09.470 DEBUG:boto:Headers: {} 2024-02-01T10:18:09.470 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:09.470 DEBUG:boto:Port: 8000 2024-02-01T10:18:09.470 DEBUG:boto:Params: {} 2024-02-01T10:18:09.470 DEBUG:boto:Token: None 2024-02-01T10:18:09.470 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:09 GMT /vclfcn-11/ 2024-02-01T10:18:09.470 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/GfGIqwjxtU9GSGOfvBZe1JBZvs= 2024-02-01T10:18:09.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/GfGIqwjxtU9GSGOfvBZe1JBZvs=', 'Content-Length': '0'} 2024-02-01T10:18:09.475 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000061fcaaaf9e62d1fb-0065bb6fe1-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:09.475 DEBUG:boto:b'vclfcn-11&1000false&2024-02-01T10:16:36.539Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:09.476 DEBUG:boto:path=/vclfcn-11/ 2024-02-01T10:18:09.476 DEBUG:boto:auth_path=/vclfcn-11/ 2024-02-01T10:18:09.476 DEBUG:boto:path=/vclfcn-11/?prefix=%26 2024-02-01T10:18:09.476 DEBUG:boto:auth_path=/vclfcn-11/?prefix=%26 2024-02-01T10:18:09.476 DEBUG:boto:Method: GET 2024-02-01T10:18:09.477 DEBUG:boto:Path: /vclfcn-11/?prefix=%26 2024-02-01T10:18:09.477 DEBUG:boto:Data: 2024-02-01T10:18:09.477 DEBUG:boto:Headers: {} 2024-02-01T10:18:09.477 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:09.477 DEBUG:boto:Port: 8000 2024-02-01T10:18:09.477 DEBUG:boto:Params: {} 2024-02-01T10:18:09.477 DEBUG:boto:Token: None 2024-02-01T10:18:09.477 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:09 GMT /vclfcn-11/ 2024-02-01T10:18:09.477 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/GfGIqwjxtU9GSGOfvBZe1JBZvs= 2024-02-01T10:18:09.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/GfGIqwjxtU9GSGOfvBZe1JBZvs=', 'Content-Length': '0'} 2024-02-01T10:18:09.481 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3913024139e11f7-0065bb6fe1-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:09.482 DEBUG:boto:b'vclfcn-11&1000false&2024-02-01T10:16:36.539Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:09.482 DEBUG:boto:path=/vclfcn-11/%26 2024-02-01T10:18:09.482 DEBUG:boto:auth_path=/vclfcn-11/%26 2024-02-01T10:18:09.482 DEBUG:boto:Method: HEAD 2024-02-01T10:18:09.482 DEBUG:boto:Path: /vclfcn-11/%26 2024-02-01T10:18:09.482 DEBUG:boto:Data: 2024-02-01T10:18:09.483 DEBUG:boto:Headers: {} 2024-02-01T10:18:09.483 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:09.483 DEBUG:boto:Port: 8000 2024-02-01T10:18:09.483 DEBUG:boto:Params: {} 2024-02-01T10:18:09.483 DEBUG:boto:Token: None 2024-02-01T10:18:09.483 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:09 GMT /vclfcn-11/%26 2024-02-01T10:18:09.483 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:bV02zSXfbM0u4QgQDA0jcSA5MWw= 2024-02-01T10:18:09.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:bV02zSXfbM0u4QgQDA0jcSA5MWw=', 'Content-Length': '0'} 2024-02-01T10:18:09.486 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a4bbef71c9dd0178-0065bb6fe1-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:09.486 DEBUG:boto:path=/vclfcn-11/%26 2024-02-01T10:18:09.486 DEBUG:boto:auth_path=/vclfcn-11/%26 2024-02-01T10:18:09.486 DEBUG:boto:Method: HEAD 2024-02-01T10:18:09.486 DEBUG:boto:Path: /vclfcn-11/%26 2024-02-01T10:18:09.487 DEBUG:boto:Data: 2024-02-01T10:18:09.487 DEBUG:boto:Headers: {} 2024-02-01T10:18:09.487 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:09.487 DEBUG:boto:Port: 8000 2024-02-01T10:18:09.487 DEBUG:boto:Params: {} 2024-02-01T10:18:09.487 DEBUG:boto:Token: None 2024-02-01T10:18:09.487 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:09 GMT /vclfcn-11/%26 2024-02-01T10:18:09.487 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:bV02zSXfbM0u4QgQDA0jcSA5MWw= 2024-02-01T10:18:09.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:bV02zSXfbM0u4QgQDA0jcSA5MWw=', 'Content-Length': '0'} 2024-02-01T10:18:09.490 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000044132c3afd2e467a-0065bb6fe1-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:09.490 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-02-01T10:18:09.491 DEBUG:boto:path=/vclfcn-11/%26 2024-02-01T10:18:09.491 DEBUG:boto:auth_path=/vclfcn-11/%26 2024-02-01T10:18:09.491 DEBUG:boto:Method: GET 2024-02-01T10:18:09.491 DEBUG:boto:Path: /vclfcn-11/%26 2024-02-01T10:18:09.491 DEBUG:boto:Data: 2024-02-01T10:18:09.491 DEBUG:boto:Headers: {} 2024-02-01T10:18:09.491 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:09.491 DEBUG:boto:Port: 8000 2024-02-01T10:18:09.491 DEBUG:boto:Params: {} 2024-02-01T10:18:09.491 DEBUG:boto:Token: None 2024-02-01T10:18:09.492 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:09 GMT /vclfcn-11/%26 2024-02-01T10:18:09.492 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7YmQSOEdPUvGEayqHTQJ/eSAeww= 2024-02-01T10:18:09.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7YmQSOEdPUvGEayqHTQJ/eSAeww=', 'Content-Length': '0'} 2024-02-01T10:18:09.494 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000021f6f19c2f05513a-0065bb6fe1-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:09.536 DEBUG:boto:path=/vclfcn-11/%26 2024-02-01T10:18:09.536 DEBUG:boto:auth_path=/vclfcn-11/%26 2024-02-01T10:18:09.536 DEBUG:boto:Method: GET 2024-02-01T10:18:09.537 DEBUG:boto:Path: /vclfcn-11/%26 2024-02-01T10:18:09.537 DEBUG:boto:Data: 2024-02-01T10:18:09.537 DEBUG:boto:Headers: {} 2024-02-01T10:18:09.537 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:09.537 DEBUG:boto:Port: 8000 2024-02-01T10:18:09.537 DEBUG:boto:Params: {} 2024-02-01T10:18:09.537 DEBUG:boto:Token: None 2024-02-01T10:18:09.537 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:09 GMT /vclfcn-11/%26 2024-02-01T10:18:09.538 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7YmQSOEdPUvGEayqHTQJ/eSAeww= 2024-02-01T10:18:09.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7YmQSOEdPUvGEayqHTQJ/eSAeww=', 'Content-Length': '0'} 2024-02-01T10:18:09.541 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eb858b7beafe8990-0065bb6fe1-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:09.541 DEBUG:boto:path=/vclfcn-11/ 2024-02-01T10:18:09.542 DEBUG:boto:auth_path=/vclfcn-11/ 2024-02-01T10:18:09.542 DEBUG:boto:path=/vclfcn-11/?prefix=%26 2024-02-01T10:18:09.542 DEBUG:boto:auth_path=/vclfcn-11/?prefix=%26 2024-02-01T10:18:09.542 DEBUG:boto:Method: GET 2024-02-01T10:18:09.542 DEBUG:boto:Path: /vclfcn-11/?prefix=%26 2024-02-01T10:18:09.542 DEBUG:boto:Data: 2024-02-01T10:18:09.542 DEBUG:boto:Headers: {} 2024-02-01T10:18:09.542 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:09.542 DEBUG:boto:Port: 8000 2024-02-01T10:18:09.542 DEBUG:boto:Params: {} 2024-02-01T10:18:09.543 DEBUG:boto:Token: None 2024-02-01T10:18:09.543 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:09 GMT /vclfcn-11/ 2024-02-01T10:18:09.543 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/GfGIqwjxtU9GSGOfvBZe1JBZvs= 2024-02-01T10:18:09.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/GfGIqwjxtU9GSGOfvBZe1JBZvs=', 'Content-Length': '0'} 2024-02-01T10:18:09.547 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf2de87ce1409022-0065bb6fe1-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:09.548 DEBUG:boto:b'vclfcn-11&1000false&2024-02-01T10:16:36.539Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:09.548 DEBUG:boto:path=/vclfcn-11/ 2024-02-01T10:18:09.548 DEBUG:boto:auth_path=/vclfcn-11/ 2024-02-01T10:18:09.548 DEBUG:boto:path=/vclfcn-11/?prefix=%26 2024-02-01T10:18:09.548 DEBUG:boto:auth_path=/vclfcn-11/?prefix=%26 2024-02-01T10:18:09.548 DEBUG:boto:Method: GET 2024-02-01T10:18:09.548 DEBUG:boto:Path: /vclfcn-11/?prefix=%26 2024-02-01T10:18:09.549 DEBUG:boto:Data: 2024-02-01T10:18:09.549 DEBUG:boto:Headers: {} 2024-02-01T10:18:09.549 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:09.549 DEBUG:boto:Port: 8000 2024-02-01T10:18:09.549 DEBUG:boto:Params: {} 2024-02-01T10:18:09.549 DEBUG:boto:Token: None 2024-02-01T10:18:09.549 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:09 GMT /vclfcn-11/ 2024-02-01T10:18:09.549 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/GfGIqwjxtU9GSGOfvBZe1JBZvs= 2024-02-01T10:18:09.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/GfGIqwjxtU9GSGOfvBZe1JBZvs=', 'Content-Length': '0'} 2024-02-01T10:18:09.553 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000618d0cdee05d401e-0065bb6fe1-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:09.553 DEBUG:boto:b'vclfcn-11&1000false&2024-02-01T10:16:36.539Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:09.554 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-02-01T10:18:09.554 DEBUG:boto:path=/vclfcn-11/%3A 2024-02-01T10:18:09.554 DEBUG:boto:auth_path=/vclfcn-11/%3A 2024-02-01T10:18:09.554 DEBUG:boto:path=/vclfcn-11/%3A?versionId=null 2024-02-01T10:18:09.554 DEBUG:boto:auth_path=/vclfcn-11/%3A?versionId=null 2024-02-01T10:18:09.554 DEBUG:boto:Method: GET 2024-02-01T10:18:09.554 DEBUG:boto:Path: /vclfcn-11/%3A?versionId=null 2024-02-01T10:18:09.554 DEBUG:boto:Data: 2024-02-01T10:18:09.554 DEBUG:boto:Headers: {} 2024-02-01T10:18:09.555 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:09.555 DEBUG:boto:Port: 8000 2024-02-01T10:18:09.555 DEBUG:boto:Params: {} 2024-02-01T10:18:09.555 DEBUG:boto:Token: None 2024-02-01T10:18:09.555 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:09 GMT /vclfcn-11/%3A?versionId=null 2024-02-01T10:18:09.555 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:u8Tq1pXi+Ed79fT10sJqjA31f+0= 2024-02-01T10:18:09.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:u8Tq1pXi+Ed79fT10sJqjA31f+0=', 'Content-Length': '0'} 2024-02-01T10:18:09.558 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008465a3f6c84adc19-0065bb6fe1-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:09.558 DEBUG:boto:path=/vclfcn-11/%3A 2024-02-01T10:18:09.559 DEBUG:boto:auth_path=/vclfcn-11/%3A 2024-02-01T10:18:09.559 DEBUG:boto:path=/vclfcn-11/%3A?versionId=null 2024-02-01T10:18:09.559 DEBUG:boto:auth_path=/vclfcn-11/%3A?versionId=null 2024-02-01T10:18:09.559 DEBUG:boto:Method: GET 2024-02-01T10:18:09.559 DEBUG:boto:Path: /vclfcn-11/%3A?versionId=null 2024-02-01T10:18:09.559 DEBUG:boto:Data: 2024-02-01T10:18:09.559 DEBUG:boto:Headers: {} 2024-02-01T10:18:09.559 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:09.559 DEBUG:boto:Port: 8000 2024-02-01T10:18:09.559 DEBUG:boto:Params: {} 2024-02-01T10:18:09.560 DEBUG:boto:Token: None 2024-02-01T10:18:09.560 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:09 GMT /vclfcn-11/%3A?versionId=null 2024-02-01T10:18:09.560 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:u8Tq1pXi+Ed79fT10sJqjA31f+0= 2024-02-01T10:18:09.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:u8Tq1pXi+Ed79fT10sJqjA31f+0=', 'Content-Length': '0'} 2024-02-01T10:18:09.562 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006f2171823cc20fcf-0065bb6fe1-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:09.563 DEBUG:boto:path=/vclfcn-11/%3A 2024-02-01T10:18:09.563 DEBUG:boto:auth_path=/vclfcn-11/%3A 2024-02-01T10:18:09.563 DEBUG:boto:path=/vclfcn-11/%3A?versionId=null 2024-02-01T10:18:09.563 DEBUG:boto:auth_path=/vclfcn-11/%3A?versionId=null 2024-02-01T10:18:09.563 DEBUG:boto:Method: HEAD 2024-02-01T10:18:09.563 DEBUG:boto:Path: /vclfcn-11/%3A?versionId=null 2024-02-01T10:18:09.563 DEBUG:boto:Data: 2024-02-01T10:18:09.563 DEBUG:boto:Headers: {} 2024-02-01T10:18:09.564 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:09.564 DEBUG:boto:Port: 8000 2024-02-01T10:18:09.564 DEBUG:boto:Params: {} 2024-02-01T10:18:09.564 DEBUG:boto:Token: None 2024-02-01T10:18:09.564 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:09 GMT /vclfcn-11/%3A?versionId=null 2024-02-01T10:18:09.564 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:D4yUsaAuhfXNzYUUx5kAjct/U4c= 2024-02-01T10:18:09.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:D4yUsaAuhfXNzYUUx5kAjct/U4c=', 'Content-Length': '0'} 2024-02-01T10:18:09.567 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000836cd3c52dd5584b-0065bb6fe1-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:09.567 DEBUG:boto:path=/vclfcn-11/%3A 2024-02-01T10:18:09.568 DEBUG:boto:auth_path=/vclfcn-11/%3A 2024-02-01T10:18:09.568 DEBUG:boto:path=/vclfcn-11/%3A?versionId=null 2024-02-01T10:18:09.568 DEBUG:boto:auth_path=/vclfcn-11/%3A?versionId=null 2024-02-01T10:18:09.568 DEBUG:boto:Method: HEAD 2024-02-01T10:18:09.568 DEBUG:boto:Path: /vclfcn-11/%3A?versionId=null 2024-02-01T10:18:09.568 DEBUG:boto:Data: 2024-02-01T10:18:09.568 DEBUG:boto:Headers: {} 2024-02-01T10:18:09.568 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:09.568 DEBUG:boto:Port: 8000 2024-02-01T10:18:09.568 DEBUG:boto:Params: {} 2024-02-01T10:18:09.569 DEBUG:boto:Token: None 2024-02-01T10:18:09.569 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:09 GMT /vclfcn-11/%3A?versionId=null 2024-02-01T10:18:09.569 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:D4yUsaAuhfXNzYUUx5kAjct/U4c= 2024-02-01T10:18:09.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:D4yUsaAuhfXNzYUUx5kAjct/U4c=', 'Content-Length': '0'} 2024-02-01T10:18:09.571 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004460b3293fa31195-0065bb6fe1-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:09.572 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-02-01T10:18:09.572 DEBUG:boto:path=/vclfcn-11/%3A 2024-02-01T10:18:09.572 DEBUG:boto:auth_path=/vclfcn-11/%3A 2024-02-01T10:18:09.572 DEBUG:boto:path=/vclfcn-11/%3A?versionId=null 2024-02-01T10:18:09.572 DEBUG:boto:auth_path=/vclfcn-11/%3A?versionId=null 2024-02-01T10:18:09.572 DEBUG:boto:Method: GET 2024-02-01T10:18:09.572 DEBUG:boto:Path: /vclfcn-11/%3A?versionId=null 2024-02-01T10:18:09.572 DEBUG:boto:Data: 2024-02-01T10:18:09.573 DEBUG:boto:Headers: {} 2024-02-01T10:18:09.573 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:09.573 DEBUG:boto:Port: 8000 2024-02-01T10:18:09.573 DEBUG:boto:Params: {} 2024-02-01T10:18:09.573 DEBUG:boto:Token: None 2024-02-01T10:18:09.573 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:09 GMT /vclfcn-11/%3A?versionId=null 2024-02-01T10:18:09.573 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:u8Tq1pXi+Ed79fT10sJqjA31f+0= 2024-02-01T10:18:09.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:u8Tq1pXi+Ed79fT10sJqjA31f+0=', 'Content-Length': '0'} 2024-02-01T10:18:09.576 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003e5e011421ba90b2-0065bb6fe1-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:09.620 DEBUG:boto:path=/vclfcn-11/%3A 2024-02-01T10:18:09.620 DEBUG:boto:auth_path=/vclfcn-11/%3A 2024-02-01T10:18:09.620 DEBUG:boto:path=/vclfcn-11/%3A?versionId=null 2024-02-01T10:18:09.620 DEBUG:boto:auth_path=/vclfcn-11/%3A?versionId=null 2024-02-01T10:18:09.621 DEBUG:boto:Method: GET 2024-02-01T10:18:09.621 DEBUG:boto:Path: /vclfcn-11/%3A?versionId=null 2024-02-01T10:18:09.621 DEBUG:boto:Data: 2024-02-01T10:18:09.621 DEBUG:boto:Headers: {} 2024-02-01T10:18:09.621 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:09.621 DEBUG:boto:Port: 8000 2024-02-01T10:18:09.621 DEBUG:boto:Params: {} 2024-02-01T10:18:09.621 DEBUG:boto:Token: None 2024-02-01T10:18:09.622 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:09 GMT /vclfcn-11/%3A?versionId=null 2024-02-01T10:18:09.622 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:u8Tq1pXi+Ed79fT10sJqjA31f+0= 2024-02-01T10:18:09.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:u8Tq1pXi+Ed79fT10sJqjA31f+0=', 'Content-Length': '0'} 2024-02-01T10:18:09.625 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008281bf211e5173f2-0065bb6fe1-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:09.625 DEBUG:boto:path=/vclfcn-11/ 2024-02-01T10:18:09.625 DEBUG:boto:auth_path=/vclfcn-11/ 2024-02-01T10:18:09.625 DEBUG:boto:path=/vclfcn-11/?prefix=%3A 2024-02-01T10:18:09.626 DEBUG:boto:auth_path=/vclfcn-11/?prefix=%3A 2024-02-01T10:18:09.626 DEBUG:boto:Method: GET 2024-02-01T10:18:09.626 DEBUG:boto:Path: /vclfcn-11/?prefix=%3A 2024-02-01T10:18:09.626 DEBUG:boto:Data: 2024-02-01T10:18:09.626 DEBUG:boto:Headers: {} 2024-02-01T10:18:09.626 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:09.626 DEBUG:boto:Port: 8000 2024-02-01T10:18:09.626 DEBUG:boto:Params: {} 2024-02-01T10:18:09.626 DEBUG:boto:Token: None 2024-02-01T10:18:09.626 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:09 GMT /vclfcn-11/ 2024-02-01T10:18:09.627 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/GfGIqwjxtU9GSGOfvBZe1JBZvs= 2024-02-01T10:18:09.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/GfGIqwjxtU9GSGOfvBZe1JBZvs=', 'Content-Length': '0'} 2024-02-01T10:18:09.631 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6a49f6da92e7a65-0065bb6fe1-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:09.672 DEBUG:boto:b'vclfcn-11:1000false:2024-02-01T10:16:36.512Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:09.673 DEBUG:boto:path=/vclfcn-11/ 2024-02-01T10:18:09.673 DEBUG:boto:auth_path=/vclfcn-11/ 2024-02-01T10:18:09.673 DEBUG:boto:path=/vclfcn-11/?prefix=%3A 2024-02-01T10:18:09.673 DEBUG:boto:auth_path=/vclfcn-11/?prefix=%3A 2024-02-01T10:18:09.673 DEBUG:boto:Method: GET 2024-02-01T10:18:09.673 DEBUG:boto:Path: /vclfcn-11/?prefix=%3A 2024-02-01T10:18:09.673 DEBUG:boto:Data: 2024-02-01T10:18:09.673 DEBUG:boto:Headers: {} 2024-02-01T10:18:09.673 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:09.674 DEBUG:boto:Port: 8000 2024-02-01T10:18:09.674 DEBUG:boto:Params: {} 2024-02-01T10:18:09.674 DEBUG:boto:Token: None 2024-02-01T10:18:09.674 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:09 GMT /vclfcn-11/ 2024-02-01T10:18:09.674 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/GfGIqwjxtU9GSGOfvBZe1JBZvs= 2024-02-01T10:18:09.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/GfGIqwjxtU9GSGOfvBZe1JBZvs=', 'Content-Length': '0'} 2024-02-01T10:18:09.678 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d9d45874a0b322a-0065bb6fe1-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:09.679 DEBUG:boto:b'vclfcn-11:1000false:2024-02-01T10:16:36.512Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:09.679 DEBUG:boto:path=/vclfcn-11/%3A 2024-02-01T10:18:09.679 DEBUG:boto:auth_path=/vclfcn-11/%3A 2024-02-01T10:18:09.679 DEBUG:boto:Method: HEAD 2024-02-01T10:18:09.679 DEBUG:boto:Path: /vclfcn-11/%3A 2024-02-01T10:18:09.679 DEBUG:boto:Data: 2024-02-01T10:18:09.679 DEBUG:boto:Headers: {} 2024-02-01T10:18:09.680 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:09.680 DEBUG:boto:Port: 8000 2024-02-01T10:18:09.680 DEBUG:boto:Params: {} 2024-02-01T10:18:09.680 DEBUG:boto:Token: None 2024-02-01T10:18:09.680 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:09 GMT /vclfcn-11/%3A 2024-02-01T10:18:09.680 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:TCOXDFRy5DPhHNrY3YMmVbePzWE= 2024-02-01T10:18:09.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:TCOXDFRy5DPhHNrY3YMmVbePzWE=', 'Content-Length': '0'} 2024-02-01T10:18:09.683 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d6ca19f681b8118f-0065bb6fe1-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:09.683 DEBUG:boto:path=/vclfcn-11/%3A 2024-02-01T10:18:09.684 DEBUG:boto:auth_path=/vclfcn-11/%3A 2024-02-01T10:18:09.684 DEBUG:boto:Method: HEAD 2024-02-01T10:18:09.684 DEBUG:boto:Path: /vclfcn-11/%3A 2024-02-01T10:18:09.684 DEBUG:boto:Data: 2024-02-01T10:18:09.684 DEBUG:boto:Headers: {} 2024-02-01T10:18:09.684 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:09.684 DEBUG:boto:Port: 8000 2024-02-01T10:18:09.684 DEBUG:boto:Params: {} 2024-02-01T10:18:09.684 DEBUG:boto:Token: None 2024-02-01T10:18:09.684 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:09 GMT /vclfcn-11/%3A 2024-02-01T10:18:09.685 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:TCOXDFRy5DPhHNrY3YMmVbePzWE= 2024-02-01T10:18:09.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:TCOXDFRy5DPhHNrY3YMmVbePzWE=', 'Content-Length': '0'} 2024-02-01T10:18:09.687 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a56ba05164288147-0065bb6fe1-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:09.688 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-02-01T10:18:09.688 DEBUG:boto:path=/vclfcn-11/%3A 2024-02-01T10:18:09.688 DEBUG:boto:auth_path=/vclfcn-11/%3A 2024-02-01T10:18:09.688 DEBUG:boto:Method: GET 2024-02-01T10:18:09.688 DEBUG:boto:Path: /vclfcn-11/%3A 2024-02-01T10:18:09.688 DEBUG:boto:Data: 2024-02-01T10:18:09.688 DEBUG:boto:Headers: {} 2024-02-01T10:18:09.689 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:09.689 DEBUG:boto:Port: 8000 2024-02-01T10:18:09.689 DEBUG:boto:Params: {} 2024-02-01T10:18:09.689 DEBUG:boto:Token: None 2024-02-01T10:18:09.689 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:09 GMT /vclfcn-11/%3A 2024-02-01T10:18:09.689 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:f1y/Z9DIslulWzeLXuRuF2BP8hw= 2024-02-01T10:18:09.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:f1y/Z9DIslulWzeLXuRuF2BP8hw=', 'Content-Length': '0'} 2024-02-01T10:18:09.692 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002afe63351a4d7bb7-0065bb6fe1-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:09.736 DEBUG:boto:path=/vclfcn-11/%3A 2024-02-01T10:18:09.736 DEBUG:boto:auth_path=/vclfcn-11/%3A 2024-02-01T10:18:09.737 DEBUG:boto:Method: GET 2024-02-01T10:18:09.737 DEBUG:boto:Path: /vclfcn-11/%3A 2024-02-01T10:18:09.737 DEBUG:boto:Data: 2024-02-01T10:18:09.737 DEBUG:boto:Headers: {} 2024-02-01T10:18:09.737 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:09.737 DEBUG:boto:Port: 8000 2024-02-01T10:18:09.737 DEBUG:boto:Params: {} 2024-02-01T10:18:09.737 DEBUG:boto:Token: None 2024-02-01T10:18:09.738 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:09 GMT /vclfcn-11/%3A 2024-02-01T10:18:09.738 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:f1y/Z9DIslulWzeLXuRuF2BP8hw= 2024-02-01T10:18:09.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:f1y/Z9DIslulWzeLXuRuF2BP8hw=', 'Content-Length': '0'} 2024-02-01T10:18:09.741 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000724afaed6815cea1-0065bb6fe1-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:09.742 DEBUG:boto:path=/vclfcn-11/ 2024-02-01T10:18:09.742 DEBUG:boto:auth_path=/vclfcn-11/ 2024-02-01T10:18:09.742 DEBUG:boto:path=/vclfcn-11/?prefix=%3A 2024-02-01T10:18:09.742 DEBUG:boto:auth_path=/vclfcn-11/?prefix=%3A 2024-02-01T10:18:09.742 DEBUG:boto:Method: GET 2024-02-01T10:18:09.743 DEBUG:boto:Path: /vclfcn-11/?prefix=%3A 2024-02-01T10:18:09.743 DEBUG:boto:Data: 2024-02-01T10:18:09.743 DEBUG:boto:Headers: {} 2024-02-01T10:18:09.743 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:09.743 DEBUG:boto:Port: 8000 2024-02-01T10:18:09.743 DEBUG:boto:Params: {} 2024-02-01T10:18:09.743 DEBUG:boto:Token: None 2024-02-01T10:18:09.743 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:09 GMT /vclfcn-11/ 2024-02-01T10:18:09.744 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/GfGIqwjxtU9GSGOfvBZe1JBZvs= 2024-02-01T10:18:09.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/GfGIqwjxtU9GSGOfvBZe1JBZvs=', 'Content-Length': '0'} 2024-02-01T10:18:09.748 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000daa672f3e0f3874c-0065bb6fe1-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:09.796 DEBUG:boto:b'vclfcn-11:1000false:2024-02-01T10:16:36.512Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:09.797 DEBUG:boto:path=/vclfcn-11/ 2024-02-01T10:18:09.797 DEBUG:boto:auth_path=/vclfcn-11/ 2024-02-01T10:18:09.797 DEBUG:boto:path=/vclfcn-11/?prefix=%3A 2024-02-01T10:18:09.797 DEBUG:boto:auth_path=/vclfcn-11/?prefix=%3A 2024-02-01T10:18:09.797 DEBUG:boto:Method: GET 2024-02-01T10:18:09.797 DEBUG:boto:Path: /vclfcn-11/?prefix=%3A 2024-02-01T10:18:09.797 DEBUG:boto:Data: 2024-02-01T10:18:09.797 DEBUG:boto:Headers: {} 2024-02-01T10:18:09.797 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:09.797 DEBUG:boto:Port: 8000 2024-02-01T10:18:09.798 DEBUG:boto:Params: {} 2024-02-01T10:18:09.798 DEBUG:boto:Token: None 2024-02-01T10:18:09.798 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:09 GMT /vclfcn-11/ 2024-02-01T10:18:09.798 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/GfGIqwjxtU9GSGOfvBZe1JBZvs= 2024-02-01T10:18:09.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/GfGIqwjxtU9GSGOfvBZe1JBZvs=', 'Content-Length': '0'} 2024-02-01T10:18:09.803 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2e794e6d8265689-0065bb6fe1-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:09.803 DEBUG:boto:b'vclfcn-11:1000false:2024-02-01T10:16:36.512Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:09.803 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-02-01T10:18:09.803 DEBUG:boto:path=/vclfcn-11/_myobj 2024-02-01T10:18:09.803 DEBUG:boto:auth_path=/vclfcn-11/_myobj 2024-02-01T10:18:09.803 DEBUG:boto:path=/vclfcn-11/_myobj?versionId=null 2024-02-01T10:18:09.804 DEBUG:boto:auth_path=/vclfcn-11/_myobj?versionId=null 2024-02-01T10:18:09.804 DEBUG:boto:Method: GET 2024-02-01T10:18:09.804 DEBUG:boto:Path: /vclfcn-11/_myobj?versionId=null 2024-02-01T10:18:09.804 DEBUG:boto:Data: 2024-02-01T10:18:09.804 DEBUG:boto:Headers: {} 2024-02-01T10:18:09.804 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:09.804 DEBUG:boto:Port: 8000 2024-02-01T10:18:09.804 DEBUG:boto:Params: {} 2024-02-01T10:18:09.804 DEBUG:boto:Token: None 2024-02-01T10:18:09.804 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:09 GMT /vclfcn-11/_myobj?versionId=null 2024-02-01T10:18:09.805 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:LihypPNshOu+UX0PnwpLatfK5RE= 2024-02-01T10:18:09.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:LihypPNshOu+UX0PnwpLatfK5RE=', 'Content-Length': '0'} 2024-02-01T10:18:09.808 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f3607428b5565aa1-0065bb6fe1-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:09.848 DEBUG:boto:path=/vclfcn-11/_myobj 2024-02-01T10:18:09.848 DEBUG:boto:auth_path=/vclfcn-11/_myobj 2024-02-01T10:18:09.848 DEBUG:boto:path=/vclfcn-11/_myobj?versionId=null 2024-02-01T10:18:09.848 DEBUG:boto:auth_path=/vclfcn-11/_myobj?versionId=null 2024-02-01T10:18:09.849 DEBUG:boto:Method: GET 2024-02-01T10:18:09.849 DEBUG:boto:Path: /vclfcn-11/_myobj?versionId=null 2024-02-01T10:18:09.849 DEBUG:boto:Data: 2024-02-01T10:18:09.849 DEBUG:boto:Headers: {} 2024-02-01T10:18:09.849 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:09.849 DEBUG:boto:Port: 8000 2024-02-01T10:18:09.849 DEBUG:boto:Params: {} 2024-02-01T10:18:09.849 DEBUG:boto:Token: None 2024-02-01T10:18:09.849 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:09 GMT /vclfcn-11/_myobj?versionId=null 2024-02-01T10:18:09.850 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:LihypPNshOu+UX0PnwpLatfK5RE= 2024-02-01T10:18:09.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:LihypPNshOu+UX0PnwpLatfK5RE=', 'Content-Length': '0'} 2024-02-01T10:18:09.852 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000043934a6267a4f23c-0065bb6fe1-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:09.853 DEBUG:boto:path=/vclfcn-11/_myobj 2024-02-01T10:18:09.853 DEBUG:boto:auth_path=/vclfcn-11/_myobj 2024-02-01T10:18:09.853 DEBUG:boto:path=/vclfcn-11/_myobj?versionId=null 2024-02-01T10:18:09.853 DEBUG:boto:auth_path=/vclfcn-11/_myobj?versionId=null 2024-02-01T10:18:09.853 DEBUG:boto:Method: HEAD 2024-02-01T10:18:09.853 DEBUG:boto:Path: /vclfcn-11/_myobj?versionId=null 2024-02-01T10:18:09.853 DEBUG:boto:Data: 2024-02-01T10:18:09.854 DEBUG:boto:Headers: {} 2024-02-01T10:18:09.854 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:09.854 DEBUG:boto:Port: 8000 2024-02-01T10:18:09.854 DEBUG:boto:Params: {} 2024-02-01T10:18:09.854 DEBUG:boto:Token: None 2024-02-01T10:18:09.854 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:09 GMT /vclfcn-11/_myobj?versionId=null 2024-02-01T10:18:09.854 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:VZDkfSWwafZcy1D5YJ3LBXSFee4= 2024-02-01T10:18:09.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:VZDkfSWwafZcy1D5YJ3LBXSFee4=', 'Content-Length': '0'} 2024-02-01T10:18:09.857 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000040cb31f2354c8f02-0065bb6fe1-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:09.857 DEBUG:boto:path=/vclfcn-11/_myobj 2024-02-01T10:18:09.858 DEBUG:boto:auth_path=/vclfcn-11/_myobj 2024-02-01T10:18:09.858 DEBUG:boto:path=/vclfcn-11/_myobj?versionId=null 2024-02-01T10:18:09.858 DEBUG:boto:auth_path=/vclfcn-11/_myobj?versionId=null 2024-02-01T10:18:09.858 DEBUG:boto:Method: HEAD 2024-02-01T10:18:09.858 DEBUG:boto:Path: /vclfcn-11/_myobj?versionId=null 2024-02-01T10:18:09.858 DEBUG:boto:Data: 2024-02-01T10:18:09.858 DEBUG:boto:Headers: {} 2024-02-01T10:18:09.858 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:09.858 DEBUG:boto:Port: 8000 2024-02-01T10:18:09.859 DEBUG:boto:Params: {} 2024-02-01T10:18:09.859 DEBUG:boto:Token: None 2024-02-01T10:18:09.859 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:09 GMT /vclfcn-11/_myobj?versionId=null 2024-02-01T10:18:09.859 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:VZDkfSWwafZcy1D5YJ3LBXSFee4= 2024-02-01T10:18:09.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:VZDkfSWwafZcy1D5YJ3LBXSFee4=', 'Content-Length': '0'} 2024-02-01T10:18:09.862 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000070c982d8b8365803-0065bb6fe1-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:09.862 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-02-01T10:18:09.862 DEBUG:boto:path=/vclfcn-11/_myobj 2024-02-01T10:18:09.863 DEBUG:boto:auth_path=/vclfcn-11/_myobj 2024-02-01T10:18:09.863 DEBUG:boto:path=/vclfcn-11/_myobj?versionId=null 2024-02-01T10:18:09.863 DEBUG:boto:auth_path=/vclfcn-11/_myobj?versionId=null 2024-02-01T10:18:09.863 DEBUG:boto:Method: GET 2024-02-01T10:18:09.863 DEBUG:boto:Path: /vclfcn-11/_myobj?versionId=null 2024-02-01T10:18:09.863 DEBUG:boto:Data: 2024-02-01T10:18:09.863 DEBUG:boto:Headers: {} 2024-02-01T10:18:09.863 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:09.863 DEBUG:boto:Port: 8000 2024-02-01T10:18:09.863 DEBUG:boto:Params: {} 2024-02-01T10:18:09.864 DEBUG:boto:Token: None 2024-02-01T10:18:09.864 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:09 GMT /vclfcn-11/_myobj?versionId=null 2024-02-01T10:18:09.864 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:LihypPNshOu+UX0PnwpLatfK5RE= 2024-02-01T10:18:09.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:LihypPNshOu+UX0PnwpLatfK5RE=', 'Content-Length': '0'} 2024-02-01T10:18:09.867 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000055fb5b25fa4e29c6-0065bb6fe1-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:09.908 DEBUG:boto:path=/vclfcn-11/_myobj 2024-02-01T10:18:09.908 DEBUG:boto:auth_path=/vclfcn-11/_myobj 2024-02-01T10:18:09.908 DEBUG:boto:path=/vclfcn-11/_myobj?versionId=null 2024-02-01T10:18:09.909 DEBUG:boto:auth_path=/vclfcn-11/_myobj?versionId=null 2024-02-01T10:18:09.909 DEBUG:boto:Method: GET 2024-02-01T10:18:09.909 DEBUG:boto:Path: /vclfcn-11/_myobj?versionId=null 2024-02-01T10:18:09.909 DEBUG:boto:Data: 2024-02-01T10:18:09.909 DEBUG:boto:Headers: {} 2024-02-01T10:18:09.909 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:09.909 DEBUG:boto:Port: 8000 2024-02-01T10:18:09.909 DEBUG:boto:Params: {} 2024-02-01T10:18:09.909 DEBUG:boto:Token: None 2024-02-01T10:18:09.910 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:09 GMT /vclfcn-11/_myobj?versionId=null 2024-02-01T10:18:09.910 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:LihypPNshOu+UX0PnwpLatfK5RE= 2024-02-01T10:18:09.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:LihypPNshOu+UX0PnwpLatfK5RE=', 'Content-Length': '0'} 2024-02-01T10:18:09.913 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002fc0f5f60ba4af63-0065bb6fe1-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:09.913 DEBUG:boto:path=/vclfcn-11/ 2024-02-01T10:18:09.913 DEBUG:boto:auth_path=/vclfcn-11/ 2024-02-01T10:18:09.914 DEBUG:boto:path=/vclfcn-11/?prefix=_myobj 2024-02-01T10:18:09.914 DEBUG:boto:auth_path=/vclfcn-11/?prefix=_myobj 2024-02-01T10:18:09.914 DEBUG:boto:Method: GET 2024-02-01T10:18:09.914 DEBUG:boto:Path: /vclfcn-11/?prefix=_myobj 2024-02-01T10:18:09.914 DEBUG:boto:Data: 2024-02-01T10:18:09.914 DEBUG:boto:Headers: {} 2024-02-01T10:18:09.914 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:09.914 DEBUG:boto:Port: 8000 2024-02-01T10:18:09.914 DEBUG:boto:Params: {} 2024-02-01T10:18:09.914 DEBUG:boto:Token: None 2024-02-01T10:18:09.915 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:09 GMT /vclfcn-11/ 2024-02-01T10:18:09.915 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/GfGIqwjxtU9GSGOfvBZe1JBZvs= 2024-02-01T10:18:09.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/GfGIqwjxtU9GSGOfvBZe1JBZvs=', 'Content-Length': '0'} 2024-02-01T10:18:09.919 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a64df3d8f6533f80-0065bb6fe1-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:09.964 DEBUG:boto:b'vclfcn-11_myobj1000false_myobj2024-02-01T10:16:36.463Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:09.965 DEBUG:boto:path=/vclfcn-11/ 2024-02-01T10:18:09.965 DEBUG:boto:auth_path=/vclfcn-11/ 2024-02-01T10:18:09.965 DEBUG:boto:path=/vclfcn-11/?prefix=_myobj 2024-02-01T10:18:09.965 DEBUG:boto:auth_path=/vclfcn-11/?prefix=_myobj 2024-02-01T10:18:09.965 DEBUG:boto:Method: GET 2024-02-01T10:18:09.965 DEBUG:boto:Path: /vclfcn-11/?prefix=_myobj 2024-02-01T10:18:09.965 DEBUG:boto:Data: 2024-02-01T10:18:09.965 DEBUG:boto:Headers: {} 2024-02-01T10:18:09.965 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:09.965 DEBUG:boto:Port: 8000 2024-02-01T10:18:09.966 DEBUG:boto:Params: {} 2024-02-01T10:18:09.966 DEBUG:boto:Token: None 2024-02-01T10:18:09.966 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:09 GMT /vclfcn-11/ 2024-02-01T10:18:09.966 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/GfGIqwjxtU9GSGOfvBZe1JBZvs= 2024-02-01T10:18:09.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/GfGIqwjxtU9GSGOfvBZe1JBZvs=', 'Content-Length': '0'} 2024-02-01T10:18:09.971 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008771defdec08c5e5-0065bb6fe1-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:09.971 DEBUG:boto:b'vclfcn-11_myobj1000false_myobj2024-02-01T10:16:36.463Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:09.971 DEBUG:boto:path=/vclfcn-11/_myobj 2024-02-01T10:18:09.971 DEBUG:boto:auth_path=/vclfcn-11/_myobj 2024-02-01T10:18:09.971 DEBUG:boto:Method: HEAD 2024-02-01T10:18:09.972 DEBUG:boto:Path: /vclfcn-11/_myobj 2024-02-01T10:18:09.972 DEBUG:boto:Data: 2024-02-01T10:18:09.972 DEBUG:boto:Headers: {} 2024-02-01T10:18:09.972 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:09.972 DEBUG:boto:Port: 8000 2024-02-01T10:18:09.972 DEBUG:boto:Params: {} 2024-02-01T10:18:09.972 DEBUG:boto:Token: None 2024-02-01T10:18:09.972 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:09 GMT /vclfcn-11/_myobj 2024-02-01T10:18:09.972 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0eGmNg9BnY91iYrIy4z+n4YMMbI= 2024-02-01T10:18:09.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0eGmNg9BnY91iYrIy4z+n4YMMbI=', 'Content-Length': '0'} 2024-02-01T10:18:09.975 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000006f5998d3091d28b-0065bb6fe1-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:09.975 DEBUG:boto:path=/vclfcn-11/_myobj 2024-02-01T10:18:09.976 DEBUG:boto:auth_path=/vclfcn-11/_myobj 2024-02-01T10:18:09.976 DEBUG:boto:Method: HEAD 2024-02-01T10:18:09.976 DEBUG:boto:Path: /vclfcn-11/_myobj 2024-02-01T10:18:09.976 DEBUG:boto:Data: 2024-02-01T10:18:09.976 DEBUG:boto:Headers: {} 2024-02-01T10:18:09.976 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:09.976 DEBUG:boto:Port: 8000 2024-02-01T10:18:09.976 DEBUG:boto:Params: {} 2024-02-01T10:18:09.976 DEBUG:boto:Token: None 2024-02-01T10:18:09.977 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:09 GMT /vclfcn-11/_myobj 2024-02-01T10:18:09.977 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0eGmNg9BnY91iYrIy4z+n4YMMbI= 2024-02-01T10:18:09.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0eGmNg9BnY91iYrIy4z+n4YMMbI=', 'Content-Length': '0'} 2024-02-01T10:18:09.979 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000908c961bb975d642-0065bb6fe1-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:09.979 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-02-01T10:18:09.980 DEBUG:boto:path=/vclfcn-11/_myobj 2024-02-01T10:18:09.980 DEBUG:boto:auth_path=/vclfcn-11/_myobj 2024-02-01T10:18:09.980 DEBUG:boto:Method: GET 2024-02-01T10:18:09.980 DEBUG:boto:Path: /vclfcn-11/_myobj 2024-02-01T10:18:09.980 DEBUG:boto:Data: 2024-02-01T10:18:09.980 DEBUG:boto:Headers: {} 2024-02-01T10:18:09.980 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:09.980 DEBUG:boto:Port: 8000 2024-02-01T10:18:09.980 DEBUG:boto:Params: {} 2024-02-01T10:18:09.980 DEBUG:boto:Token: None 2024-02-01T10:18:09.981 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:09 GMT /vclfcn-11/_myobj 2024-02-01T10:18:09.982 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GtL5kPYGsoe9bsw5+mvUIKaC1uE= 2024-02-01T10:18:09.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GtL5kPYGsoe9bsw5+mvUIKaC1uE=', 'Content-Length': '0'} 2024-02-01T10:18:09.985 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001ca1d3cd962a0291-0065bb6fe1-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:10.028 DEBUG:boto:path=/vclfcn-11/_myobj 2024-02-01T10:18:10.028 DEBUG:boto:auth_path=/vclfcn-11/_myobj 2024-02-01T10:18:10.028 DEBUG:boto:Method: GET 2024-02-01T10:18:10.029 DEBUG:boto:Path: /vclfcn-11/_myobj 2024-02-01T10:18:10.029 DEBUG:boto:Data: 2024-02-01T10:18:10.029 DEBUG:boto:Headers: {} 2024-02-01T10:18:10.029 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:10.029 DEBUG:boto:Port: 8000 2024-02-01T10:18:10.029 DEBUG:boto:Params: {} 2024-02-01T10:18:10.029 DEBUG:boto:Token: None 2024-02-01T10:18:10.029 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:10 GMT /vclfcn-11/_myobj 2024-02-01T10:18:10.030 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:pHPaWdTSAbq7Rnmq/7t7cUlawbo= 2024-02-01T10:18:10.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:pHPaWdTSAbq7Rnmq/7t7cUlawbo=', 'Content-Length': '0'} 2024-02-01T10:18:10.033 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002c116669247005d9-0065bb6fe2-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:10.033 DEBUG:boto:path=/vclfcn-11/ 2024-02-01T10:18:10.033 DEBUG:boto:auth_path=/vclfcn-11/ 2024-02-01T10:18:10.033 DEBUG:boto:path=/vclfcn-11/?prefix=_myobj 2024-02-01T10:18:10.033 DEBUG:boto:auth_path=/vclfcn-11/?prefix=_myobj 2024-02-01T10:18:10.033 DEBUG:boto:Method: GET 2024-02-01T10:18:10.033 DEBUG:boto:Path: /vclfcn-11/?prefix=_myobj 2024-02-01T10:18:10.033 DEBUG:boto:Data: 2024-02-01T10:18:10.034 DEBUG:boto:Headers: {} 2024-02-01T10:18:10.034 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:10.034 DEBUG:boto:Port: 8000 2024-02-01T10:18:10.034 DEBUG:boto:Params: {} 2024-02-01T10:18:10.034 DEBUG:boto:Token: None 2024-02-01T10:18:10.034 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:10 GMT /vclfcn-11/ 2024-02-01T10:18:10.034 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:M9IZDpl/mkiHoyqfgRxTQVr04Ro= 2024-02-01T10:18:10.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:M9IZDpl/mkiHoyqfgRxTQVr04Ro=', 'Content-Length': '0'} 2024-02-01T10:18:10.038 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b98d16e3d582156c-0065bb6fe2-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:10.080 DEBUG:boto:b'vclfcn-11_myobj1000false_myobj2024-02-01T10:16:36.463Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:10.081 DEBUG:boto:path=/vclfcn-11/ 2024-02-01T10:18:10.081 DEBUG:boto:auth_path=/vclfcn-11/ 2024-02-01T10:18:10.081 DEBUG:boto:path=/vclfcn-11/?prefix=_myobj 2024-02-01T10:18:10.081 DEBUG:boto:auth_path=/vclfcn-11/?prefix=_myobj 2024-02-01T10:18:10.081 DEBUG:boto:Method: GET 2024-02-01T10:18:10.081 DEBUG:boto:Path: /vclfcn-11/?prefix=_myobj 2024-02-01T10:18:10.081 DEBUG:boto:Data: 2024-02-01T10:18:10.081 DEBUG:boto:Headers: {} 2024-02-01T10:18:10.081 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:10.082 DEBUG:boto:Port: 8000 2024-02-01T10:18:10.082 DEBUG:boto:Params: {} 2024-02-01T10:18:10.082 DEBUG:boto:Token: None 2024-02-01T10:18:10.082 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:10 GMT /vclfcn-11/ 2024-02-01T10:18:10.082 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:M9IZDpl/mkiHoyqfgRxTQVr04Ro= 2024-02-01T10:18:10.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:M9IZDpl/mkiHoyqfgRxTQVr04Ro=', 'Content-Length': '0'} 2024-02-01T10:18:10.086 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063464626a837a8b3-0065bb6fe2-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:10.087 DEBUG:boto:b'vclfcn-11_myobj1000false_myobj2024-02-01T10:16:36.463Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:10.087 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-02-01T10:18:10.087 DEBUG:boto:path=/vclfcn-11/myobj 2024-02-01T10:18:10.087 DEBUG:boto:auth_path=/vclfcn-11/myobj 2024-02-01T10:18:10.087 DEBUG:boto:path=/vclfcn-11/myobj?versionId=null 2024-02-01T10:18:10.088 DEBUG:boto:auth_path=/vclfcn-11/myobj?versionId=null 2024-02-01T10:18:10.088 DEBUG:boto:Method: GET 2024-02-01T10:18:10.088 DEBUG:boto:Path: /vclfcn-11/myobj?versionId=null 2024-02-01T10:18:10.088 DEBUG:boto:Data: 2024-02-01T10:18:10.088 DEBUG:boto:Headers: {} 2024-02-01T10:18:10.088 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:10.088 DEBUG:boto:Port: 8000 2024-02-01T10:18:10.088 DEBUG:boto:Params: {} 2024-02-01T10:18:10.088 DEBUG:boto:Token: None 2024-02-01T10:18:10.089 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:10 GMT /vclfcn-11/myobj?versionId=null 2024-02-01T10:18:10.089 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:g6zwl9Y7t8JaIyZQqs4TRO2H9I4= 2024-02-01T10:18:10.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:g6zwl9Y7t8JaIyZQqs4TRO2H9I4=', 'Content-Length': '0'} 2024-02-01T10:18:10.092 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000468d42ebae152989-0065bb6fe2-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:10.136 DEBUG:boto:path=/vclfcn-11/myobj 2024-02-01T10:18:10.137 DEBUG:boto:auth_path=/vclfcn-11/myobj 2024-02-01T10:18:10.137 DEBUG:boto:path=/vclfcn-11/myobj?versionId=null 2024-02-01T10:18:10.137 DEBUG:boto:auth_path=/vclfcn-11/myobj?versionId=null 2024-02-01T10:18:10.137 DEBUG:boto:Method: GET 2024-02-01T10:18:10.137 DEBUG:boto:Path: /vclfcn-11/myobj?versionId=null 2024-02-01T10:18:10.137 DEBUG:boto:Data: 2024-02-01T10:18:10.137 DEBUG:boto:Headers: {} 2024-02-01T10:18:10.137 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:10.137 DEBUG:boto:Port: 8000 2024-02-01T10:18:10.137 DEBUG:boto:Params: {} 2024-02-01T10:18:10.139 DEBUG:boto:Token: None 2024-02-01T10:18:10.139 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:10 GMT /vclfcn-11/myobj?versionId=null 2024-02-01T10:18:10.139 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:g6zwl9Y7t8JaIyZQqs4TRO2H9I4= 2024-02-01T10:18:10.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:g6zwl9Y7t8JaIyZQqs4TRO2H9I4=', 'Content-Length': '0'} 2024-02-01T10:18:10.143 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006cd5476e1935a7ac-0065bb6fe2-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:10.143 DEBUG:boto:path=/vclfcn-11/myobj 2024-02-01T10:18:10.143 DEBUG:boto:auth_path=/vclfcn-11/myobj 2024-02-01T10:18:10.143 DEBUG:boto:path=/vclfcn-11/myobj?versionId=null 2024-02-01T10:18:10.143 DEBUG:boto:auth_path=/vclfcn-11/myobj?versionId=null 2024-02-01T10:18:10.144 DEBUG:boto:Method: HEAD 2024-02-01T10:18:10.144 DEBUG:boto:Path: /vclfcn-11/myobj?versionId=null 2024-02-01T10:18:10.144 DEBUG:boto:Data: 2024-02-01T10:18:10.144 DEBUG:boto:Headers: {} 2024-02-01T10:18:10.144 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:10.144 DEBUG:boto:Port: 8000 2024-02-01T10:18:10.144 DEBUG:boto:Params: {} 2024-02-01T10:18:10.144 DEBUG:boto:Token: None 2024-02-01T10:18:10.145 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:10 GMT /vclfcn-11/myobj?versionId=null 2024-02-01T10:18:10.145 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GZZ2AiBvGhQNAcLxUffkknuveFM= 2024-02-01T10:18:10.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GZZ2AiBvGhQNAcLxUffkknuveFM=', 'Content-Length': '0'} 2024-02-01T10:18:10.148 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004b655a87a8a0d599-0065bb6fe2-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:10.148 DEBUG:boto:path=/vclfcn-11/myobj 2024-02-01T10:18:10.148 DEBUG:boto:auth_path=/vclfcn-11/myobj 2024-02-01T10:18:10.148 DEBUG:boto:path=/vclfcn-11/myobj?versionId=null 2024-02-01T10:18:10.148 DEBUG:boto:auth_path=/vclfcn-11/myobj?versionId=null 2024-02-01T10:18:10.148 DEBUG:boto:Method: HEAD 2024-02-01T10:18:10.148 DEBUG:boto:Path: /vclfcn-11/myobj?versionId=null 2024-02-01T10:18:10.149 DEBUG:boto:Data: 2024-02-01T10:18:10.149 DEBUG:boto:Headers: {} 2024-02-01T10:18:10.149 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:10.149 DEBUG:boto:Port: 8000 2024-02-01T10:18:10.149 DEBUG:boto:Params: {} 2024-02-01T10:18:10.149 DEBUG:boto:Token: None 2024-02-01T10:18:10.149 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:10 GMT /vclfcn-11/myobj?versionId=null 2024-02-01T10:18:10.149 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GZZ2AiBvGhQNAcLxUffkknuveFM= 2024-02-01T10:18:10.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GZZ2AiBvGhQNAcLxUffkknuveFM=', 'Content-Length': '0'} 2024-02-01T10:18:10.152 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fa7221b14c3f64d6-0065bb6fe2-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:10.152 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-02-01T10:18:10.152 DEBUG:boto:path=/vclfcn-11/myobj 2024-02-01T10:18:10.153 DEBUG:boto:auth_path=/vclfcn-11/myobj 2024-02-01T10:18:10.153 DEBUG:boto:path=/vclfcn-11/myobj?versionId=null 2024-02-01T10:18:10.153 DEBUG:boto:auth_path=/vclfcn-11/myobj?versionId=null 2024-02-01T10:18:10.153 DEBUG:boto:Method: GET 2024-02-01T10:18:10.153 DEBUG:boto:Path: /vclfcn-11/myobj?versionId=null 2024-02-01T10:18:10.153 DEBUG:boto:Data: 2024-02-01T10:18:10.153 DEBUG:boto:Headers: {} 2024-02-01T10:18:10.153 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:10.153 DEBUG:boto:Port: 8000 2024-02-01T10:18:10.153 DEBUG:boto:Params: {} 2024-02-01T10:18:10.153 DEBUG:boto:Token: None 2024-02-01T10:18:10.154 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:10 GMT /vclfcn-11/myobj?versionId=null 2024-02-01T10:18:10.154 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:g6zwl9Y7t8JaIyZQqs4TRO2H9I4= 2024-02-01T10:18:10.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:g6zwl9Y7t8JaIyZQqs4TRO2H9I4=', 'Content-Length': '0'} 2024-02-01T10:18:10.157 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005fa02bb29aaaa36d-0065bb6fe2-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:10.200 DEBUG:boto:path=/vclfcn-11/myobj 2024-02-01T10:18:10.200 DEBUG:boto:auth_path=/vclfcn-11/myobj 2024-02-01T10:18:10.200 DEBUG:boto:path=/vclfcn-11/myobj?versionId=null 2024-02-01T10:18:10.201 DEBUG:boto:auth_path=/vclfcn-11/myobj?versionId=null 2024-02-01T10:18:10.201 DEBUG:boto:Method: GET 2024-02-01T10:18:10.201 DEBUG:boto:Path: /vclfcn-11/myobj?versionId=null 2024-02-01T10:18:10.201 DEBUG:boto:Data: 2024-02-01T10:18:10.201 DEBUG:boto:Headers: {} 2024-02-01T10:18:10.201 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:10.201 DEBUG:boto:Port: 8000 2024-02-01T10:18:10.201 DEBUG:boto:Params: {} 2024-02-01T10:18:10.201 DEBUG:boto:Token: None 2024-02-01T10:18:10.202 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:10 GMT /vclfcn-11/myobj?versionId=null 2024-02-01T10:18:10.202 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:g6zwl9Y7t8JaIyZQqs4TRO2H9I4= 2024-02-01T10:18:10.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:g6zwl9Y7t8JaIyZQqs4TRO2H9I4=', 'Content-Length': '0'} 2024-02-01T10:18:10.205 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005c2b5ab6d3f54309-0065bb6fe2-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:10.205 DEBUG:boto:path=/vclfcn-11/ 2024-02-01T10:18:10.205 DEBUG:boto:auth_path=/vclfcn-11/ 2024-02-01T10:18:10.205 DEBUG:boto:path=/vclfcn-11/?prefix=myobj 2024-02-01T10:18:10.205 DEBUG:boto:auth_path=/vclfcn-11/?prefix=myobj 2024-02-01T10:18:10.206 DEBUG:boto:Method: GET 2024-02-01T10:18:10.206 DEBUG:boto:Path: /vclfcn-11/?prefix=myobj 2024-02-01T10:18:10.206 DEBUG:boto:Data: 2024-02-01T10:18:10.206 DEBUG:boto:Headers: {} 2024-02-01T10:18:10.206 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:10.206 DEBUG:boto:Port: 8000 2024-02-01T10:18:10.206 DEBUG:boto:Params: {} 2024-02-01T10:18:10.206 DEBUG:boto:Token: None 2024-02-01T10:18:10.207 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:10 GMT /vclfcn-11/ 2024-02-01T10:18:10.207 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:M9IZDpl/mkiHoyqfgRxTQVr04Ro= 2024-02-01T10:18:10.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:M9IZDpl/mkiHoyqfgRxTQVr04Ro=', 'Content-Length': '0'} 2024-02-01T10:18:10.211 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004af427589ab6fc8f-0065bb6fe2-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:10.211 DEBUG:boto:b'vclfcn-11myobj1000falsemyobj2024-02-01T10:16:36.434Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:10.212 DEBUG:boto:path=/vclfcn-11/ 2024-02-01T10:18:10.212 DEBUG:boto:auth_path=/vclfcn-11/ 2024-02-01T10:18:10.212 DEBUG:boto:path=/vclfcn-11/?prefix=myobj 2024-02-01T10:18:10.212 DEBUG:boto:auth_path=/vclfcn-11/?prefix=myobj 2024-02-01T10:18:10.212 DEBUG:boto:Method: GET 2024-02-01T10:18:10.213 DEBUG:boto:Path: /vclfcn-11/?prefix=myobj 2024-02-01T10:18:10.213 DEBUG:boto:Data: 2024-02-01T10:18:10.213 DEBUG:boto:Headers: {} 2024-02-01T10:18:10.213 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:10.213 DEBUG:boto:Port: 8000 2024-02-01T10:18:10.213 DEBUG:boto:Params: {} 2024-02-01T10:18:10.213 DEBUG:boto:Token: None 2024-02-01T10:18:10.213 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:10 GMT /vclfcn-11/ 2024-02-01T10:18:10.213 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:M9IZDpl/mkiHoyqfgRxTQVr04Ro= 2024-02-01T10:18:10.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:M9IZDpl/mkiHoyqfgRxTQVr04Ro=', 'Content-Length': '0'} 2024-02-01T10:18:10.218 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d60627ea8437f54e-0065bb6fe2-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:10.264 DEBUG:boto:b'vclfcn-11myobj1000falsemyobj2024-02-01T10:16:36.434Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:10.265 DEBUG:boto:path=/vclfcn-11/myobj 2024-02-01T10:18:10.265 DEBUG:boto:auth_path=/vclfcn-11/myobj 2024-02-01T10:18:10.265 DEBUG:boto:Method: HEAD 2024-02-01T10:18:10.265 DEBUG:boto:Path: /vclfcn-11/myobj 2024-02-01T10:18:10.265 DEBUG:boto:Data: 2024-02-01T10:18:10.265 DEBUG:boto:Headers: {} 2024-02-01T10:18:10.265 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:10.265 DEBUG:boto:Port: 8000 2024-02-01T10:18:10.265 DEBUG:boto:Params: {} 2024-02-01T10:18:10.266 DEBUG:boto:Token: None 2024-02-01T10:18:10.266 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:10 GMT /vclfcn-11/myobj 2024-02-01T10:18:10.266 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1BC/0Da7YwuZIIambdDkitHKcVQ= 2024-02-01T10:18:10.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1BC/0Da7YwuZIIambdDkitHKcVQ=', 'Content-Length': '0'} 2024-02-01T10:18:10.269 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ab28f17ded6c052d-0065bb6fe2-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:10.269 DEBUG:boto:path=/vclfcn-11/myobj 2024-02-01T10:18:10.269 DEBUG:boto:auth_path=/vclfcn-11/myobj 2024-02-01T10:18:10.269 DEBUG:boto:Method: HEAD 2024-02-01T10:18:10.269 DEBUG:boto:Path: /vclfcn-11/myobj 2024-02-01T10:18:10.270 DEBUG:boto:Data: 2024-02-01T10:18:10.270 DEBUG:boto:Headers: {} 2024-02-01T10:18:10.270 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:10.270 DEBUG:boto:Port: 8000 2024-02-01T10:18:10.270 DEBUG:boto:Params: {} 2024-02-01T10:18:10.270 DEBUG:boto:Token: None 2024-02-01T10:18:10.270 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:10 GMT /vclfcn-11/myobj 2024-02-01T10:18:10.270 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1BC/0Da7YwuZIIambdDkitHKcVQ= 2024-02-01T10:18:10.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1BC/0Da7YwuZIIambdDkitHKcVQ=', 'Content-Length': '0'} 2024-02-01T10:18:10.273 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003e3e6d9d70240b2b-0065bb6fe2-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:10.273 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-02-01T10:18:10.274 DEBUG:boto:path=/vclfcn-11/myobj 2024-02-01T10:18:10.274 DEBUG:boto:auth_path=/vclfcn-11/myobj 2024-02-01T10:18:10.274 DEBUG:boto:Method: GET 2024-02-01T10:18:10.274 DEBUG:boto:Path: /vclfcn-11/myobj 2024-02-01T10:18:10.274 DEBUG:boto:Data: 2024-02-01T10:18:10.274 DEBUG:boto:Headers: {} 2024-02-01T10:18:10.274 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:10.274 DEBUG:boto:Port: 8000 2024-02-01T10:18:10.274 DEBUG:boto:Params: {} 2024-02-01T10:18:10.275 DEBUG:boto:Token: None 2024-02-01T10:18:10.275 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:10 GMT /vclfcn-11/myobj 2024-02-01T10:18:10.275 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NeoyuscdP+KLHRIJRy4JEAN9meQ= 2024-02-01T10:18:10.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NeoyuscdP+KLHRIJRy4JEAN9meQ=', 'Content-Length': '0'} 2024-02-01T10:18:10.278 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c4c9dec72f5d94dd-0065bb6fe2-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:10.324 DEBUG:boto:path=/vclfcn-11/myobj 2024-02-01T10:18:10.324 DEBUG:boto:auth_path=/vclfcn-11/myobj 2024-02-01T10:18:10.324 DEBUG:boto:Method: GET 2024-02-01T10:18:10.325 DEBUG:boto:Path: /vclfcn-11/myobj 2024-02-01T10:18:10.325 DEBUG:boto:Data: 2024-02-01T10:18:10.325 DEBUG:boto:Headers: {} 2024-02-01T10:18:10.325 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:10.325 DEBUG:boto:Port: 8000 2024-02-01T10:18:10.325 DEBUG:boto:Params: {} 2024-02-01T10:18:10.325 DEBUG:boto:Token: None 2024-02-01T10:18:10.325 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:10 GMT /vclfcn-11/myobj 2024-02-01T10:18:10.325 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NeoyuscdP+KLHRIJRy4JEAN9meQ= 2024-02-01T10:18:10.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NeoyuscdP+KLHRIJRy4JEAN9meQ=', 'Content-Length': '0'} 2024-02-01T10:18:10.328 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000717103329f4f80b3-0065bb6fe2-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:10.329 DEBUG:boto:path=/vclfcn-11/ 2024-02-01T10:18:10.329 DEBUG:boto:auth_path=/vclfcn-11/ 2024-02-01T10:18:10.329 DEBUG:boto:path=/vclfcn-11/?prefix=myobj 2024-02-01T10:18:10.329 DEBUG:boto:auth_path=/vclfcn-11/?prefix=myobj 2024-02-01T10:18:10.329 DEBUG:boto:Method: GET 2024-02-01T10:18:10.329 DEBUG:boto:Path: /vclfcn-11/?prefix=myobj 2024-02-01T10:18:10.329 DEBUG:boto:Data: 2024-02-01T10:18:10.329 DEBUG:boto:Headers: {} 2024-02-01T10:18:10.330 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:10.330 DEBUG:boto:Port: 8000 2024-02-01T10:18:10.330 DEBUG:boto:Params: {} 2024-02-01T10:18:10.330 DEBUG:boto:Token: None 2024-02-01T10:18:10.330 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:10 GMT /vclfcn-11/ 2024-02-01T10:18:10.330 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:M9IZDpl/mkiHoyqfgRxTQVr04Ro= 2024-02-01T10:18:10.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:M9IZDpl/mkiHoyqfgRxTQVr04Ro=', 'Content-Length': '0'} 2024-02-01T10:18:10.335 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a21c03b332114b0-0065bb6fe2-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:10.335 DEBUG:boto:b'vclfcn-11myobj1000falsemyobj2024-02-01T10:16:36.434Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:10.335 DEBUG:boto:path=/vclfcn-11/ 2024-02-01T10:18:10.336 DEBUG:boto:auth_path=/vclfcn-11/ 2024-02-01T10:18:10.336 DEBUG:boto:path=/vclfcn-11/?prefix=myobj 2024-02-01T10:18:10.336 DEBUG:boto:auth_path=/vclfcn-11/?prefix=myobj 2024-02-01T10:18:10.336 DEBUG:boto:Method: GET 2024-02-01T10:18:10.336 DEBUG:boto:Path: /vclfcn-11/?prefix=myobj 2024-02-01T10:18:10.336 DEBUG:boto:Data: 2024-02-01T10:18:10.337 DEBUG:boto:Headers: {} 2024-02-01T10:18:10.337 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:10.337 DEBUG:boto:Port: 8000 2024-02-01T10:18:10.337 DEBUG:boto:Params: {} 2024-02-01T10:18:10.337 DEBUG:boto:Token: None 2024-02-01T10:18:10.337 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:10 GMT /vclfcn-11/ 2024-02-01T10:18:10.337 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:M9IZDpl/mkiHoyqfgRxTQVr04Ro= 2024-02-01T10:18:10.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:M9IZDpl/mkiHoyqfgRxTQVr04Ro=', 'Content-Length': '0'} 2024-02-01T10:18:10.342 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eda7cec3915394eb-0065bb6fe2-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:10.342 DEBUG:boto:b'vclfcn-11myobj1000falsemyobj2024-02-01T10:16:36.434Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:10.342 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-11 zones={test-zone1, test-zone2} 2024-02-01T10:18:10.343 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:18:10.343 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:18:10.343 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-11 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:18:10.393 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:18:10.393 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:18:10.456 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:18:10.455+0000 7f0e0f4cf980 1 bucket sync caught up with source: 2024-02-01T10:18:10.456 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, 00000000001.31.6, , 00000000001.12.6, , , , 00000000001.14.6, 00000000001.6.6, , ] 2024-02-01T10:18:10.456 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, 00000000001.31.6, , 00000000001.12.6, , , , 00000000001.14.6, 00000000001.6.6, , ] 2024-02-01T10:18:10.456 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:18:10.455+0000 7f0e0f4cf980 0 bucket checkpoint complete 2024-02-01T10:18:10.478 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-11 zones={test-zone3, test-zone2} 2024-02-01T10:18:10.478 DEBUG:boto:path=/vclfcn-11/ 2024-02-01T10:18:10.478 DEBUG:boto:auth_path=/vclfcn-11/ 2024-02-01T10:18:10.479 DEBUG:boto:Method: HEAD 2024-02-01T10:18:10.479 DEBUG:boto:Path: /vclfcn-11/ 2024-02-01T10:18:10.479 DEBUG:boto:Data: 2024-02-01T10:18:10.479 DEBUG:boto:Headers: {} 2024-02-01T10:18:10.479 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:10.479 DEBUG:boto:Port: 8001 2024-02-01T10:18:10.479 DEBUG:boto:Params: {} 2024-02-01T10:18:10.479 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-02-01T10:18:10.480 DEBUG:boto:Token: None 2024-02-01T10:18:10.480 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:10 GMT /vclfcn-11/ 2024-02-01T10:18:10.480 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:l/zZSp3CXhhPE4qvkLNaCEl4CPI= 2024-02-01T10:18:10.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:l/zZSp3CXhhPE4qvkLNaCEl4CPI=', 'Content-Length': '0'} 2024-02-01T10:18:10.484 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cd2374dbf10fbe80-0065bb6fe2-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:10.485 DEBUG:boto:path=/vclfcn-11/ 2024-02-01T10:18:10.485 DEBUG:boto:auth_path=/vclfcn-11/ 2024-02-01T10:18:10.485 DEBUG:boto:Method: HEAD 2024-02-01T10:18:10.485 DEBUG:boto:Path: /vclfcn-11/ 2024-02-01T10:18:10.485 DEBUG:boto:Data: 2024-02-01T10:18:10.485 DEBUG:boto:Headers: {} 2024-02-01T10:18:10.485 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:10.485 DEBUG:boto:Port: 8000 2024-02-01T10:18:10.486 DEBUG:boto:Params: {} 2024-02-01T10:18:10.486 DEBUG:boto:Token: None 2024-02-01T10:18:10.486 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:10 GMT /vclfcn-11/ 2024-02-01T10:18:10.486 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:l/zZSp3CXhhPE4qvkLNaCEl4CPI= 2024-02-01T10:18:10.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:l/zZSp3CXhhPE4qvkLNaCEl4CPI=', 'Content-Length': '0'} 2024-02-01T10:18:10.490 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002a44bb4963b56e66-0065bb6fe2-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:10.490 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:18:10.490 DEBUG:boto:path=/vclfcn-11/ 2024-02-01T10:18:10.490 DEBUG:boto:auth_path=/vclfcn-11/ 2024-02-01T10:18:10.491 DEBUG:boto:path=/vclfcn-11/?versions&max-keys=999 2024-02-01T10:18:10.491 DEBUG:boto:auth_path=/vclfcn-11/?versions&max-keys=999 2024-02-01T10:18:10.491 DEBUG:boto:Method: GET 2024-02-01T10:18:10.491 DEBUG:boto:Path: /vclfcn-11/?versions&max-keys=999 2024-02-01T10:18:10.491 DEBUG:boto:Data: 2024-02-01T10:18:10.491 DEBUG:boto:Headers: {} 2024-02-01T10:18:10.491 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:10.492 DEBUG:boto:Port: 8001 2024-02-01T10:18:10.492 DEBUG:boto:Params: {} 2024-02-01T10:18:10.492 DEBUG:boto:Token: None 2024-02-01T10:18:10.492 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:10 GMT /vclfcn-11/?versions 2024-02-01T10:18:10.492 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:j5xYE0X8I1VqXS+d6/7TxPuSVUc= 2024-02-01T10:18:10.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:j5xYE0X8I1VqXS+d6/7TxPuSVUc=', 'Content-Length': '0'} 2024-02-01T10:18:10.495 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000032fc930dfc4363a4-0065bb6fe2-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:10.495 DEBUG:boto:b'vclfcn-11999false&nulltrue2024-02-01T10:16:36.539Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-02-01T10:16:36.512Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-02-01T10:16:36.463Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-02-01T10:16:36.434Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:10.495 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-02-01T10:18:10.495 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-02-01T10:18:10.495 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-02-01T10:18:10.496 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-02-01T10:18:10.496 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:18:10.496 DEBUG:boto:path=/vclfcn-11/ 2024-02-01T10:18:10.496 DEBUG:boto:auth_path=/vclfcn-11/ 2024-02-01T10:18:10.496 DEBUG:boto:path=/vclfcn-11/?versions&max-keys=999 2024-02-01T10:18:10.496 DEBUG:boto:auth_path=/vclfcn-11/?versions&max-keys=999 2024-02-01T10:18:10.496 DEBUG:boto:Method: GET 2024-02-01T10:18:10.496 DEBUG:boto:Path: /vclfcn-11/?versions&max-keys=999 2024-02-01T10:18:10.496 DEBUG:boto:Data: 2024-02-01T10:18:10.496 DEBUG:boto:Headers: {} 2024-02-01T10:18:10.497 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:10.497 DEBUG:boto:Port: 8000 2024-02-01T10:18:10.497 DEBUG:boto:Params: {} 2024-02-01T10:18:10.497 DEBUG:boto:Token: None 2024-02-01T10:18:10.497 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:10 GMT /vclfcn-11/?versions 2024-02-01T10:18:10.497 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:j5xYE0X8I1VqXS+d6/7TxPuSVUc= 2024-02-01T10:18:10.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:j5xYE0X8I1VqXS+d6/7TxPuSVUc=', 'Content-Length': '0'} 2024-02-01T10:18:10.501 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008271048589fdf06e-0065bb6fe2-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:10.548 DEBUG:boto:b'vclfcn-11999false&nulltrue2024-02-01T10:16:36.539Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-02-01T10:16:36.512Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-02-01T10:16:36.463Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-02-01T10:16:36.434Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:10.549 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-02-01T10:18:10.549 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-02-01T10:18:10.549 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-02-01T10:18:10.549 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-02-01T10:18:10.549 DEBUG:boto:path=/vclfcn-11/ 2024-02-01T10:18:10.549 DEBUG:boto:auth_path=/vclfcn-11/ 2024-02-01T10:18:10.549 DEBUG:boto:path=/vclfcn-11/?versions&max-keys=999 2024-02-01T10:18:10.549 DEBUG:boto:auth_path=/vclfcn-11/?versions&max-keys=999 2024-02-01T10:18:10.549 DEBUG:boto:Method: GET 2024-02-01T10:18:10.550 DEBUG:boto:Path: /vclfcn-11/?versions&max-keys=999 2024-02-01T10:18:10.550 DEBUG:boto:Data: 2024-02-01T10:18:10.550 DEBUG:boto:Headers: {} 2024-02-01T10:18:10.550 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:10.550 DEBUG:boto:Port: 8001 2024-02-01T10:18:10.550 DEBUG:boto:Params: {} 2024-02-01T10:18:10.550 DEBUG:boto:Token: None 2024-02-01T10:18:10.550 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:10 GMT /vclfcn-11/?versions 2024-02-01T10:18:10.550 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:j5xYE0X8I1VqXS+d6/7TxPuSVUc= 2024-02-01T10:18:10.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:j5xYE0X8I1VqXS+d6/7TxPuSVUc=', 'Content-Length': '0'} 2024-02-01T10:18:10.554 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a50f62c015fd81ff-0065bb6fe2-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:10.554 DEBUG:boto:b'vclfcn-11999false&nulltrue2024-02-01T10:16:36.539Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-02-01T10:16:36.512Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-02-01T10:16:36.463Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-02-01T10:16:36.434Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:10.555 DEBUG:boto:path=/vclfcn-11/ 2024-02-01T10:18:10.555 DEBUG:boto:auth_path=/vclfcn-11/ 2024-02-01T10:18:10.555 DEBUG:boto:path=/vclfcn-11/?versions&max-keys=999 2024-02-01T10:18:10.555 DEBUG:boto:auth_path=/vclfcn-11/?versions&max-keys=999 2024-02-01T10:18:10.555 DEBUG:boto:Method: GET 2024-02-01T10:18:10.555 DEBUG:boto:Path: /vclfcn-11/?versions&max-keys=999 2024-02-01T10:18:10.555 DEBUG:boto:Data: 2024-02-01T10:18:10.556 DEBUG:boto:Headers: {} 2024-02-01T10:18:10.556 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:10.556 DEBUG:boto:Port: 8000 2024-02-01T10:18:10.556 DEBUG:boto:Params: {} 2024-02-01T10:18:10.556 DEBUG:boto:Token: None 2024-02-01T10:18:10.556 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:10 GMT /vclfcn-11/?versions 2024-02-01T10:18:10.556 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:j5xYE0X8I1VqXS+d6/7TxPuSVUc= 2024-02-01T10:18:10.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:j5xYE0X8I1VqXS+d6/7TxPuSVUc=', 'Content-Length': '0'} 2024-02-01T10:18:10.560 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000062c9436a8baea9db-0065bb6fe2-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:10.561 DEBUG:boto:b'vclfcn-11999false&nulltrue2024-02-01T10:16:36.539Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-02-01T10:16:36.512Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-02-01T10:16:36.463Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-02-01T10:16:36.434Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:10.561 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-02-01T10:18:10.561 DEBUG:boto:path=/vclfcn-11/%26 2024-02-01T10:18:10.561 DEBUG:boto:auth_path=/vclfcn-11/%26 2024-02-01T10:18:10.562 DEBUG:boto:path=/vclfcn-11/%26?versionId=null 2024-02-01T10:18:10.562 DEBUG:boto:auth_path=/vclfcn-11/%26?versionId=null 2024-02-01T10:18:10.562 DEBUG:boto:Method: GET 2024-02-01T10:18:10.562 DEBUG:boto:Path: /vclfcn-11/%26?versionId=null 2024-02-01T10:18:10.562 DEBUG:boto:Data: 2024-02-01T10:18:10.562 DEBUG:boto:Headers: {} 2024-02-01T10:18:10.562 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:10.562 DEBUG:boto:Port: 8001 2024-02-01T10:18:10.562 DEBUG:boto:Params: {} 2024-02-01T10:18:10.562 DEBUG:boto:Token: None 2024-02-01T10:18:10.563 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:10 GMT /vclfcn-11/%26?versionId=null 2024-02-01T10:18:10.563 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:eeKy2pPUopBEdLe4vd/mEN/IWd4= 2024-02-01T10:18:10.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:eeKy2pPUopBEdLe4vd/mEN/IWd4=', 'Content-Length': '0'} 2024-02-01T10:18:10.566 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000691fd7a72e907f58-0065bb6fe2-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:10.566 DEBUG:boto:path=/vclfcn-11/%26 2024-02-01T10:18:10.566 DEBUG:boto:auth_path=/vclfcn-11/%26 2024-02-01T10:18:10.566 DEBUG:boto:path=/vclfcn-11/%26?versionId=null 2024-02-01T10:18:10.566 DEBUG:boto:auth_path=/vclfcn-11/%26?versionId=null 2024-02-01T10:18:10.566 DEBUG:boto:Method: GET 2024-02-01T10:18:10.566 DEBUG:boto:Path: /vclfcn-11/%26?versionId=null 2024-02-01T10:18:10.567 DEBUG:boto:Data: 2024-02-01T10:18:10.567 DEBUG:boto:Headers: {} 2024-02-01T10:18:10.567 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:10.567 DEBUG:boto:Port: 8000 2024-02-01T10:18:10.567 DEBUG:boto:Params: {} 2024-02-01T10:18:10.567 DEBUG:boto:Token: None 2024-02-01T10:18:10.567 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:10 GMT /vclfcn-11/%26?versionId=null 2024-02-01T10:18:10.567 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:eeKy2pPUopBEdLe4vd/mEN/IWd4= 2024-02-01T10:18:10.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:eeKy2pPUopBEdLe4vd/mEN/IWd4=', 'Content-Length': '0'} 2024-02-01T10:18:10.570 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008aca0f41043905b1-0065bb6fe2-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:10.570 DEBUG:boto:path=/vclfcn-11/%26 2024-02-01T10:18:10.571 DEBUG:boto:auth_path=/vclfcn-11/%26 2024-02-01T10:18:10.571 DEBUG:boto:path=/vclfcn-11/%26?versionId=null 2024-02-01T10:18:10.571 DEBUG:boto:auth_path=/vclfcn-11/%26?versionId=null 2024-02-01T10:18:10.571 DEBUG:boto:Method: HEAD 2024-02-01T10:18:10.571 DEBUG:boto:Path: /vclfcn-11/%26?versionId=null 2024-02-01T10:18:10.571 DEBUG:boto:Data: 2024-02-01T10:18:10.571 DEBUG:boto:Headers: {} 2024-02-01T10:18:10.571 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:10.571 DEBUG:boto:Port: 8001 2024-02-01T10:18:10.571 DEBUG:boto:Params: {} 2024-02-01T10:18:10.572 DEBUG:boto:Token: None 2024-02-01T10:18:10.572 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:10 GMT /vclfcn-11/%26?versionId=null 2024-02-01T10:18:10.572 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:K3sq7GVZVzzs/x4jKiL2DAb8oAE= 2024-02-01T10:18:10.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:K3sq7GVZVzzs/x4jKiL2DAb8oAE=', 'Content-Length': '0'} 2024-02-01T10:18:10.574 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005f4a51f5136db9ce-0065bb6fe2-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:10.574 DEBUG:boto:path=/vclfcn-11/%26 2024-02-01T10:18:10.575 DEBUG:boto:auth_path=/vclfcn-11/%26 2024-02-01T10:18:10.575 DEBUG:boto:path=/vclfcn-11/%26?versionId=null 2024-02-01T10:18:10.575 DEBUG:boto:auth_path=/vclfcn-11/%26?versionId=null 2024-02-01T10:18:10.575 DEBUG:boto:Method: HEAD 2024-02-01T10:18:10.575 DEBUG:boto:Path: /vclfcn-11/%26?versionId=null 2024-02-01T10:18:10.575 DEBUG:boto:Data: 2024-02-01T10:18:10.575 DEBUG:boto:Headers: {} 2024-02-01T10:18:10.575 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:10.575 DEBUG:boto:Port: 8000 2024-02-01T10:18:10.575 DEBUG:boto:Params: {} 2024-02-01T10:18:10.576 DEBUG:boto:Token: None 2024-02-01T10:18:10.576 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:10 GMT /vclfcn-11/%26?versionId=null 2024-02-01T10:18:10.576 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:K3sq7GVZVzzs/x4jKiL2DAb8oAE= 2024-02-01T10:18:10.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:K3sq7GVZVzzs/x4jKiL2DAb8oAE=', 'Content-Length': '0'} 2024-02-01T10:18:10.579 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000396aa631abe05ea5-0065bb6fe2-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:10.579 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-02-01T10:18:10.579 DEBUG:boto:path=/vclfcn-11/%26 2024-02-01T10:18:10.579 DEBUG:boto:auth_path=/vclfcn-11/%26 2024-02-01T10:18:10.579 DEBUG:boto:path=/vclfcn-11/%26?versionId=null 2024-02-01T10:18:10.580 DEBUG:boto:auth_path=/vclfcn-11/%26?versionId=null 2024-02-01T10:18:10.580 DEBUG:boto:Method: GET 2024-02-01T10:18:10.580 DEBUG:boto:Path: /vclfcn-11/%26?versionId=null 2024-02-01T10:18:10.580 DEBUG:boto:Data: 2024-02-01T10:18:10.580 DEBUG:boto:Headers: {} 2024-02-01T10:18:10.580 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:10.580 DEBUG:boto:Port: 8001 2024-02-01T10:18:10.580 DEBUG:boto:Params: {} 2024-02-01T10:18:10.580 DEBUG:boto:Token: None 2024-02-01T10:18:10.580 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:10 GMT /vclfcn-11/%26?versionId=null 2024-02-01T10:18:10.581 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:eeKy2pPUopBEdLe4vd/mEN/IWd4= 2024-02-01T10:18:10.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:eeKy2pPUopBEdLe4vd/mEN/IWd4=', 'Content-Length': '0'} 2024-02-01T10:18:10.583 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006867c1f9c64271f4-0065bb6fe2-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:10.628 DEBUG:boto:path=/vclfcn-11/%26 2024-02-01T10:18:10.628 DEBUG:boto:auth_path=/vclfcn-11/%26 2024-02-01T10:18:10.628 DEBUG:boto:path=/vclfcn-11/%26?versionId=null 2024-02-01T10:18:10.629 DEBUG:boto:auth_path=/vclfcn-11/%26?versionId=null 2024-02-01T10:18:10.629 DEBUG:boto:Method: GET 2024-02-01T10:18:10.629 DEBUG:boto:Path: /vclfcn-11/%26?versionId=null 2024-02-01T10:18:10.629 DEBUG:boto:Data: 2024-02-01T10:18:10.629 DEBUG:boto:Headers: {} 2024-02-01T10:18:10.629 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:10.629 DEBUG:boto:Port: 8000 2024-02-01T10:18:10.629 DEBUG:boto:Params: {} 2024-02-01T10:18:10.629 DEBUG:boto:Token: None 2024-02-01T10:18:10.629 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:10 GMT /vclfcn-11/%26?versionId=null 2024-02-01T10:18:10.630 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:eeKy2pPUopBEdLe4vd/mEN/IWd4= 2024-02-01T10:18:10.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:eeKy2pPUopBEdLe4vd/mEN/IWd4=', 'Content-Length': '0'} 2024-02-01T10:18:10.633 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007ce37a4901567c72-0065bb6fe2-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:10.633 DEBUG:boto:path=/vclfcn-11/ 2024-02-01T10:18:10.633 DEBUG:boto:auth_path=/vclfcn-11/ 2024-02-01T10:18:10.633 DEBUG:boto:path=/vclfcn-11/?prefix=%26 2024-02-01T10:18:10.633 DEBUG:boto:auth_path=/vclfcn-11/?prefix=%26 2024-02-01T10:18:10.633 DEBUG:boto:Method: GET 2024-02-01T10:18:10.633 DEBUG:boto:Path: /vclfcn-11/?prefix=%26 2024-02-01T10:18:10.633 DEBUG:boto:Data: 2024-02-01T10:18:10.634 DEBUG:boto:Headers: {} 2024-02-01T10:18:10.634 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:10.634 DEBUG:boto:Port: 8001 2024-02-01T10:18:10.634 DEBUG:boto:Params: {} 2024-02-01T10:18:10.634 DEBUG:boto:Token: None 2024-02-01T10:18:10.634 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:10 GMT /vclfcn-11/ 2024-02-01T10:18:10.634 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:M9IZDpl/mkiHoyqfgRxTQVr04Ro= 2024-02-01T10:18:10.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:M9IZDpl/mkiHoyqfgRxTQVr04Ro=', 'Content-Length': '0'} 2024-02-01T10:18:10.638 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ea1b86d7c1fa8cf-0065bb6fe2-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:10.639 DEBUG:boto:b'vclfcn-11&1000false&2024-02-01T10:16:36.539Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:10.639 DEBUG:boto:path=/vclfcn-11/ 2024-02-01T10:18:10.639 DEBUG:boto:auth_path=/vclfcn-11/ 2024-02-01T10:18:10.639 DEBUG:boto:path=/vclfcn-11/?prefix=%26 2024-02-01T10:18:10.639 DEBUG:boto:auth_path=/vclfcn-11/?prefix=%26 2024-02-01T10:18:10.639 DEBUG:boto:Method: GET 2024-02-01T10:18:10.639 DEBUG:boto:Path: /vclfcn-11/?prefix=%26 2024-02-01T10:18:10.640 DEBUG:boto:Data: 2024-02-01T10:18:10.640 DEBUG:boto:Headers: {} 2024-02-01T10:18:10.640 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:10.640 DEBUG:boto:Port: 8000 2024-02-01T10:18:10.640 DEBUG:boto:Params: {} 2024-02-01T10:18:10.640 DEBUG:boto:Token: None 2024-02-01T10:18:10.640 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:10 GMT /vclfcn-11/ 2024-02-01T10:18:10.640 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:M9IZDpl/mkiHoyqfgRxTQVr04Ro= 2024-02-01T10:18:10.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:M9IZDpl/mkiHoyqfgRxTQVr04Ro=', 'Content-Length': '0'} 2024-02-01T10:18:10.644 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000bdbc81364ea1df3-0065bb6fe2-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:10.645 DEBUG:boto:b'vclfcn-11&1000false&2024-02-01T10:16:36.539Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:10.645 DEBUG:boto:path=/vclfcn-11/%26 2024-02-01T10:18:10.645 DEBUG:boto:auth_path=/vclfcn-11/%26 2024-02-01T10:18:10.645 DEBUG:boto:Method: HEAD 2024-02-01T10:18:10.645 DEBUG:boto:Path: /vclfcn-11/%26 2024-02-01T10:18:10.645 DEBUG:boto:Data: 2024-02-01T10:18:10.645 DEBUG:boto:Headers: {} 2024-02-01T10:18:10.645 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:10.646 DEBUG:boto:Port: 8001 2024-02-01T10:18:10.646 DEBUG:boto:Params: {} 2024-02-01T10:18:10.646 DEBUG:boto:Token: None 2024-02-01T10:18:10.646 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:10 GMT /vclfcn-11/%26 2024-02-01T10:18:10.646 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WRXgj7WJzejSloShQyH+YTXZJbw= 2024-02-01T10:18:10.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WRXgj7WJzejSloShQyH+YTXZJbw=', 'Content-Length': '0'} 2024-02-01T10:18:10.649 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001bc9072d66c8b9cd-0065bb6fe2-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:10.649 DEBUG:boto:path=/vclfcn-11/%26 2024-02-01T10:18:10.649 DEBUG:boto:auth_path=/vclfcn-11/%26 2024-02-01T10:18:10.649 DEBUG:boto:Method: HEAD 2024-02-01T10:18:10.649 DEBUG:boto:Path: /vclfcn-11/%26 2024-02-01T10:18:10.650 DEBUG:boto:Data: 2024-02-01T10:18:10.650 DEBUG:boto:Headers: {} 2024-02-01T10:18:10.650 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:10.650 DEBUG:boto:Port: 8000 2024-02-01T10:18:10.650 DEBUG:boto:Params: {} 2024-02-01T10:18:10.650 DEBUG:boto:Token: None 2024-02-01T10:18:10.650 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:10 GMT /vclfcn-11/%26 2024-02-01T10:18:10.650 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WRXgj7WJzejSloShQyH+YTXZJbw= 2024-02-01T10:18:10.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WRXgj7WJzejSloShQyH+YTXZJbw=', 'Content-Length': '0'} 2024-02-01T10:18:10.653 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000070f63d572892fc42-0065bb6fe2-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:10.653 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-02-01T10:18:10.653 DEBUG:boto:path=/vclfcn-11/%26 2024-02-01T10:18:10.654 DEBUG:boto:auth_path=/vclfcn-11/%26 2024-02-01T10:18:10.654 DEBUG:boto:Method: GET 2024-02-01T10:18:10.654 DEBUG:boto:Path: /vclfcn-11/%26 2024-02-01T10:18:10.654 DEBUG:boto:Data: 2024-02-01T10:18:10.654 DEBUG:boto:Headers: {} 2024-02-01T10:18:10.654 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:10.654 DEBUG:boto:Port: 8001 2024-02-01T10:18:10.654 DEBUG:boto:Params: {} 2024-02-01T10:18:10.654 DEBUG:boto:Token: None 2024-02-01T10:18:10.654 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:10 GMT /vclfcn-11/%26 2024-02-01T10:18:10.655 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:e5wxUeGZSoKj9pbOaYcI0b0RcYs= 2024-02-01T10:18:10.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:e5wxUeGZSoKj9pbOaYcI0b0RcYs=', 'Content-Length': '0'} 2024-02-01T10:18:10.657 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f7d3f8651b654c89-0065bb6fe2-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:10.704 DEBUG:boto:path=/vclfcn-11/%26 2024-02-01T10:18:10.704 DEBUG:boto:auth_path=/vclfcn-11/%26 2024-02-01T10:18:10.704 DEBUG:boto:Method: GET 2024-02-01T10:18:10.704 DEBUG:boto:Path: /vclfcn-11/%26 2024-02-01T10:18:10.705 DEBUG:boto:Data: 2024-02-01T10:18:10.705 DEBUG:boto:Headers: {} 2024-02-01T10:18:10.705 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:10.705 DEBUG:boto:Port: 8000 2024-02-01T10:18:10.705 DEBUG:boto:Params: {} 2024-02-01T10:18:10.705 DEBUG:boto:Token: None 2024-02-01T10:18:10.705 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:10 GMT /vclfcn-11/%26 2024-02-01T10:18:10.705 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:e5wxUeGZSoKj9pbOaYcI0b0RcYs= 2024-02-01T10:18:10.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:e5wxUeGZSoKj9pbOaYcI0b0RcYs=', 'Content-Length': '0'} 2024-02-01T10:18:10.708 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b39f25395c0ff48e-0065bb6fe2-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:10.709 DEBUG:boto:path=/vclfcn-11/ 2024-02-01T10:18:10.709 DEBUG:boto:auth_path=/vclfcn-11/ 2024-02-01T10:18:10.709 DEBUG:boto:path=/vclfcn-11/?prefix=%26 2024-02-01T10:18:10.709 DEBUG:boto:auth_path=/vclfcn-11/?prefix=%26 2024-02-01T10:18:10.709 DEBUG:boto:Method: GET 2024-02-01T10:18:10.709 DEBUG:boto:Path: /vclfcn-11/?prefix=%26 2024-02-01T10:18:10.709 DEBUG:boto:Data: 2024-02-01T10:18:10.709 DEBUG:boto:Headers: {} 2024-02-01T10:18:10.709 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:10.709 DEBUG:boto:Port: 8001 2024-02-01T10:18:10.710 DEBUG:boto:Params: {} 2024-02-01T10:18:10.710 DEBUG:boto:Token: None 2024-02-01T10:18:10.710 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:10 GMT /vclfcn-11/ 2024-02-01T10:18:10.710 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:M9IZDpl/mkiHoyqfgRxTQVr04Ro= 2024-02-01T10:18:10.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:M9IZDpl/mkiHoyqfgRxTQVr04Ro=', 'Content-Length': '0'} 2024-02-01T10:18:10.714 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1e78f380b3c6592-0065bb6fe2-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:10.715 DEBUG:boto:b'vclfcn-11&1000false&2024-02-01T10:16:36.539Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:10.715 DEBUG:boto:path=/vclfcn-11/ 2024-02-01T10:18:10.715 DEBUG:boto:auth_path=/vclfcn-11/ 2024-02-01T10:18:10.715 DEBUG:boto:path=/vclfcn-11/?prefix=%26 2024-02-01T10:18:10.715 DEBUG:boto:auth_path=/vclfcn-11/?prefix=%26 2024-02-01T10:18:10.715 DEBUG:boto:Method: GET 2024-02-01T10:18:10.716 DEBUG:boto:Path: /vclfcn-11/?prefix=%26 2024-02-01T10:18:10.716 DEBUG:boto:Data: 2024-02-01T10:18:10.716 DEBUG:boto:Headers: {} 2024-02-01T10:18:10.716 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:10.716 DEBUG:boto:Port: 8000 2024-02-01T10:18:10.716 DEBUG:boto:Params: {} 2024-02-01T10:18:10.716 DEBUG:boto:Token: None 2024-02-01T10:18:10.716 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:10 GMT /vclfcn-11/ 2024-02-01T10:18:10.716 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:M9IZDpl/mkiHoyqfgRxTQVr04Ro= 2024-02-01T10:18:10.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:M9IZDpl/mkiHoyqfgRxTQVr04Ro=', 'Content-Length': '0'} 2024-02-01T10:18:10.721 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000073460d09aaed5fba-0065bb6fe2-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:10.764 DEBUG:boto:b'vclfcn-11&1000false&2024-02-01T10:16:36.539Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:10.765 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-02-01T10:18:10.765 DEBUG:boto:path=/vclfcn-11/%3A 2024-02-01T10:18:10.765 DEBUG:boto:auth_path=/vclfcn-11/%3A 2024-02-01T10:18:10.765 DEBUG:boto:path=/vclfcn-11/%3A?versionId=null 2024-02-01T10:18:10.765 DEBUG:boto:auth_path=/vclfcn-11/%3A?versionId=null 2024-02-01T10:18:10.765 DEBUG:boto:Method: GET 2024-02-01T10:18:10.766 DEBUG:boto:Path: /vclfcn-11/%3A?versionId=null 2024-02-01T10:18:10.766 DEBUG:boto:Data: 2024-02-01T10:18:10.766 DEBUG:boto:Headers: {} 2024-02-01T10:18:10.766 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:10.766 DEBUG:boto:Port: 8001 2024-02-01T10:18:10.766 DEBUG:boto:Params: {} 2024-02-01T10:18:10.766 DEBUG:boto:Token: None 2024-02-01T10:18:10.766 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:10 GMT /vclfcn-11/%3A?versionId=null 2024-02-01T10:18:10.767 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QTlTtZyIYfmKwymJnYMtUsr5L9w= 2024-02-01T10:18:10.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QTlTtZyIYfmKwymJnYMtUsr5L9w=', 'Content-Length': '0'} 2024-02-01T10:18:10.770 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000085b4db3a3279a9b-0065bb6fe2-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:10.770 DEBUG:boto:path=/vclfcn-11/%3A 2024-02-01T10:18:10.770 DEBUG:boto:auth_path=/vclfcn-11/%3A 2024-02-01T10:18:10.770 DEBUG:boto:path=/vclfcn-11/%3A?versionId=null 2024-02-01T10:18:10.770 DEBUG:boto:auth_path=/vclfcn-11/%3A?versionId=null 2024-02-01T10:18:10.770 DEBUG:boto:Method: GET 2024-02-01T10:18:10.771 DEBUG:boto:Path: /vclfcn-11/%3A?versionId=null 2024-02-01T10:18:10.771 DEBUG:boto:Data: 2024-02-01T10:18:10.771 DEBUG:boto:Headers: {} 2024-02-01T10:18:10.771 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:10.771 DEBUG:boto:Port: 8000 2024-02-01T10:18:10.771 DEBUG:boto:Params: {} 2024-02-01T10:18:10.771 DEBUG:boto:Token: None 2024-02-01T10:18:10.771 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:10 GMT /vclfcn-11/%3A?versionId=null 2024-02-01T10:18:10.772 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QTlTtZyIYfmKwymJnYMtUsr5L9w= 2024-02-01T10:18:10.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QTlTtZyIYfmKwymJnYMtUsr5L9w=', 'Content-Length': '0'} 2024-02-01T10:18:10.774 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000054b21b831f915a56-0065bb6fe2-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:10.775 DEBUG:boto:path=/vclfcn-11/%3A 2024-02-01T10:18:10.775 DEBUG:boto:auth_path=/vclfcn-11/%3A 2024-02-01T10:18:10.775 DEBUG:boto:path=/vclfcn-11/%3A?versionId=null 2024-02-01T10:18:10.775 DEBUG:boto:auth_path=/vclfcn-11/%3A?versionId=null 2024-02-01T10:18:10.775 DEBUG:boto:Method: HEAD 2024-02-01T10:18:10.775 DEBUG:boto:Path: /vclfcn-11/%3A?versionId=null 2024-02-01T10:18:10.775 DEBUG:boto:Data: 2024-02-01T10:18:10.775 DEBUG:boto:Headers: {} 2024-02-01T10:18:10.776 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:10.776 DEBUG:boto:Port: 8001 2024-02-01T10:18:10.776 DEBUG:boto:Params: {} 2024-02-01T10:18:10.776 DEBUG:boto:Token: None 2024-02-01T10:18:10.776 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:10 GMT /vclfcn-11/%3A?versionId=null 2024-02-01T10:18:10.776 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RytG+2MVIqauwCwawfep6N0j/9U= 2024-02-01T10:18:10.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RytG+2MVIqauwCwawfep6N0j/9U=', 'Content-Length': '0'} 2024-02-01T10:18:10.779 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000605f25308b8a3c9d-0065bb6fe2-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:10.779 DEBUG:boto:path=/vclfcn-11/%3A 2024-02-01T10:18:10.779 DEBUG:boto:auth_path=/vclfcn-11/%3A 2024-02-01T10:18:10.779 DEBUG:boto:path=/vclfcn-11/%3A?versionId=null 2024-02-01T10:18:10.780 DEBUG:boto:auth_path=/vclfcn-11/%3A?versionId=null 2024-02-01T10:18:10.780 DEBUG:boto:Method: HEAD 2024-02-01T10:18:10.780 DEBUG:boto:Path: /vclfcn-11/%3A?versionId=null 2024-02-01T10:18:10.780 DEBUG:boto:Data: 2024-02-01T10:18:10.780 DEBUG:boto:Headers: {} 2024-02-01T10:18:10.780 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:10.780 DEBUG:boto:Port: 8000 2024-02-01T10:18:10.780 DEBUG:boto:Params: {} 2024-02-01T10:18:10.780 DEBUG:boto:Token: None 2024-02-01T10:18:10.781 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:10 GMT /vclfcn-11/%3A?versionId=null 2024-02-01T10:18:10.781 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RytG+2MVIqauwCwawfep6N0j/9U= 2024-02-01T10:18:10.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RytG+2MVIqauwCwawfep6N0j/9U=', 'Content-Length': '0'} 2024-02-01T10:18:10.784 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001c79151a30cd0804-0065bb6fe2-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:10.784 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-02-01T10:18:10.784 DEBUG:boto:path=/vclfcn-11/%3A 2024-02-01T10:18:10.784 DEBUG:boto:auth_path=/vclfcn-11/%3A 2024-02-01T10:18:10.784 DEBUG:boto:path=/vclfcn-11/%3A?versionId=null 2024-02-01T10:18:10.784 DEBUG:boto:auth_path=/vclfcn-11/%3A?versionId=null 2024-02-01T10:18:10.785 DEBUG:boto:Method: GET 2024-02-01T10:18:10.785 DEBUG:boto:Path: /vclfcn-11/%3A?versionId=null 2024-02-01T10:18:10.785 DEBUG:boto:Data: 2024-02-01T10:18:10.785 DEBUG:boto:Headers: {} 2024-02-01T10:18:10.785 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:10.785 DEBUG:boto:Port: 8001 2024-02-01T10:18:10.785 DEBUG:boto:Params: {} 2024-02-01T10:18:10.785 DEBUG:boto:Token: None 2024-02-01T10:18:10.785 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:10 GMT /vclfcn-11/%3A?versionId=null 2024-02-01T10:18:10.785 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QTlTtZyIYfmKwymJnYMtUsr5L9w= 2024-02-01T10:18:10.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QTlTtZyIYfmKwymJnYMtUsr5L9w=', 'Content-Length': '0'} 2024-02-01T10:18:10.788 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000019e549aad0fa28a4-0065bb6fe2-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:10.832 DEBUG:boto:path=/vclfcn-11/%3A 2024-02-01T10:18:10.832 DEBUG:boto:auth_path=/vclfcn-11/%3A 2024-02-01T10:18:10.832 DEBUG:boto:path=/vclfcn-11/%3A?versionId=null 2024-02-01T10:18:10.832 DEBUG:boto:auth_path=/vclfcn-11/%3A?versionId=null 2024-02-01T10:18:10.832 DEBUG:boto:Method: GET 2024-02-01T10:18:10.832 DEBUG:boto:Path: /vclfcn-11/%3A?versionId=null 2024-02-01T10:18:10.832 DEBUG:boto:Data: 2024-02-01T10:18:10.832 DEBUG:boto:Headers: {} 2024-02-01T10:18:10.833 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:10.833 DEBUG:boto:Port: 8000 2024-02-01T10:18:10.833 DEBUG:boto:Params: {} 2024-02-01T10:18:10.833 DEBUG:boto:Token: None 2024-02-01T10:18:10.833 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:10 GMT /vclfcn-11/%3A?versionId=null 2024-02-01T10:18:10.833 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QTlTtZyIYfmKwymJnYMtUsr5L9w= 2024-02-01T10:18:10.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QTlTtZyIYfmKwymJnYMtUsr5L9w=', 'Content-Length': '0'} 2024-02-01T10:18:10.836 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000005aad6868f291dd7-0065bb6fe2-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:10.836 DEBUG:boto:path=/vclfcn-11/ 2024-02-01T10:18:10.836 DEBUG:boto:auth_path=/vclfcn-11/ 2024-02-01T10:18:10.836 DEBUG:boto:path=/vclfcn-11/?prefix=%3A 2024-02-01T10:18:10.837 DEBUG:boto:auth_path=/vclfcn-11/?prefix=%3A 2024-02-01T10:18:10.837 DEBUG:boto:Method: GET 2024-02-01T10:18:10.837 DEBUG:boto:Path: /vclfcn-11/?prefix=%3A 2024-02-01T10:18:10.837 DEBUG:boto:Data: 2024-02-01T10:18:10.837 DEBUG:boto:Headers: {} 2024-02-01T10:18:10.837 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:10.837 DEBUG:boto:Port: 8001 2024-02-01T10:18:10.837 DEBUG:boto:Params: {} 2024-02-01T10:18:10.837 DEBUG:boto:Token: None 2024-02-01T10:18:10.837 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:10 GMT /vclfcn-11/ 2024-02-01T10:18:10.837 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:M9IZDpl/mkiHoyqfgRxTQVr04Ro= 2024-02-01T10:18:10.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:M9IZDpl/mkiHoyqfgRxTQVr04Ro=', 'Content-Length': '0'} 2024-02-01T10:18:10.842 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ad670b71daae922-0065bb6fe2-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:10.842 DEBUG:boto:b'vclfcn-11:1000false:2024-02-01T10:16:36.512Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:10.843 DEBUG:boto:path=/vclfcn-11/ 2024-02-01T10:18:10.843 DEBUG:boto:auth_path=/vclfcn-11/ 2024-02-01T10:18:10.843 DEBUG:boto:path=/vclfcn-11/?prefix=%3A 2024-02-01T10:18:10.843 DEBUG:boto:auth_path=/vclfcn-11/?prefix=%3A 2024-02-01T10:18:10.843 DEBUG:boto:Method: GET 2024-02-01T10:18:10.843 DEBUG:boto:Path: /vclfcn-11/?prefix=%3A 2024-02-01T10:18:10.843 DEBUG:boto:Data: 2024-02-01T10:18:10.843 DEBUG:boto:Headers: {} 2024-02-01T10:18:10.843 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:10.843 DEBUG:boto:Port: 8000 2024-02-01T10:18:10.843 DEBUG:boto:Params: {} 2024-02-01T10:18:10.844 DEBUG:boto:Token: None 2024-02-01T10:18:10.844 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:10 GMT /vclfcn-11/ 2024-02-01T10:18:10.844 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:M9IZDpl/mkiHoyqfgRxTQVr04Ro= 2024-02-01T10:18:10.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:M9IZDpl/mkiHoyqfgRxTQVr04Ro=', 'Content-Length': '0'} 2024-02-01T10:18:10.849 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000005805bde461d6f76-0065bb6fe2-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:10.849 DEBUG:boto:b'vclfcn-11:1000false:2024-02-01T10:16:36.512Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:10.849 DEBUG:boto:path=/vclfcn-11/%3A 2024-02-01T10:18:10.849 DEBUG:boto:auth_path=/vclfcn-11/%3A 2024-02-01T10:18:10.850 DEBUG:boto:Method: HEAD 2024-02-01T10:18:10.850 DEBUG:boto:Path: /vclfcn-11/%3A 2024-02-01T10:18:10.850 DEBUG:boto:Data: 2024-02-01T10:18:10.850 DEBUG:boto:Headers: {} 2024-02-01T10:18:10.850 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:10.850 DEBUG:boto:Port: 8001 2024-02-01T10:18:10.850 DEBUG:boto:Params: {} 2024-02-01T10:18:10.850 DEBUG:boto:Token: None 2024-02-01T10:18:10.851 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:10 GMT /vclfcn-11/%3A 2024-02-01T10:18:10.851 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QlmaJTh1+z3D0YmaQrdr6Oe9KPM= 2024-02-01T10:18:10.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QlmaJTh1+z3D0YmaQrdr6Oe9KPM=', 'Content-Length': '0'} 2024-02-01T10:18:10.854 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000758eded717562ff9-0065bb6fe2-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:10.854 DEBUG:boto:path=/vclfcn-11/%3A 2024-02-01T10:18:10.854 DEBUG:boto:auth_path=/vclfcn-11/%3A 2024-02-01T10:18:10.854 DEBUG:boto:Method: HEAD 2024-02-01T10:18:10.854 DEBUG:boto:Path: /vclfcn-11/%3A 2024-02-01T10:18:10.854 DEBUG:boto:Data: 2024-02-01T10:18:10.855 DEBUG:boto:Headers: {} 2024-02-01T10:18:10.855 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:10.855 DEBUG:boto:Port: 8000 2024-02-01T10:18:10.855 DEBUG:boto:Params: {} 2024-02-01T10:18:10.855 DEBUG:boto:Token: None 2024-02-01T10:18:10.855 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:10 GMT /vclfcn-11/%3A 2024-02-01T10:18:10.855 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QlmaJTh1+z3D0YmaQrdr6Oe9KPM= 2024-02-01T10:18:10.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QlmaJTh1+z3D0YmaQrdr6Oe9KPM=', 'Content-Length': '0'} 2024-02-01T10:18:10.858 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ff4d97ed81304b29-0065bb6fe2-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:10.858 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-02-01T10:18:10.859 DEBUG:boto:path=/vclfcn-11/%3A 2024-02-01T10:18:10.859 DEBUG:boto:auth_path=/vclfcn-11/%3A 2024-02-01T10:18:10.859 DEBUG:boto:Method: GET 2024-02-01T10:18:10.859 DEBUG:boto:Path: /vclfcn-11/%3A 2024-02-01T10:18:10.859 DEBUG:boto:Data: 2024-02-01T10:18:10.859 DEBUG:boto:Headers: {} 2024-02-01T10:18:10.859 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:10.859 DEBUG:boto:Port: 8001 2024-02-01T10:18:10.859 DEBUG:boto:Params: {} 2024-02-01T10:18:10.860 DEBUG:boto:Token: None 2024-02-01T10:18:10.860 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:10 GMT /vclfcn-11/%3A 2024-02-01T10:18:10.860 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ozKpJUQs6mehLRTEx57hSC/LEKs= 2024-02-01T10:18:10.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ozKpJUQs6mehLRTEx57hSC/LEKs=', 'Content-Length': '0'} 2024-02-01T10:18:10.863 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f65b3ff3e81e275d-0065bb6fe2-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:10.904 DEBUG:boto:path=/vclfcn-11/%3A 2024-02-01T10:18:10.904 DEBUG:boto:auth_path=/vclfcn-11/%3A 2024-02-01T10:18:10.904 DEBUG:boto:Method: GET 2024-02-01T10:18:10.904 DEBUG:boto:Path: /vclfcn-11/%3A 2024-02-01T10:18:10.904 DEBUG:boto:Data: 2024-02-01T10:18:10.905 DEBUG:boto:Headers: {} 2024-02-01T10:18:10.905 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:10.905 DEBUG:boto:Port: 8000 2024-02-01T10:18:10.905 DEBUG:boto:Params: {} 2024-02-01T10:18:10.905 DEBUG:boto:Token: None 2024-02-01T10:18:10.905 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:10 GMT /vclfcn-11/%3A 2024-02-01T10:18:10.905 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ozKpJUQs6mehLRTEx57hSC/LEKs= 2024-02-01T10:18:10.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ozKpJUQs6mehLRTEx57hSC/LEKs=', 'Content-Length': '0'} 2024-02-01T10:18:10.908 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000caa31ea80bda34e7-0065bb6fe2-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:10.909 DEBUG:boto:path=/vclfcn-11/ 2024-02-01T10:18:10.909 DEBUG:boto:auth_path=/vclfcn-11/ 2024-02-01T10:18:10.909 DEBUG:boto:path=/vclfcn-11/?prefix=%3A 2024-02-01T10:18:10.909 DEBUG:boto:auth_path=/vclfcn-11/?prefix=%3A 2024-02-01T10:18:10.909 DEBUG:boto:Method: GET 2024-02-01T10:18:10.909 DEBUG:boto:Path: /vclfcn-11/?prefix=%3A 2024-02-01T10:18:10.909 DEBUG:boto:Data: 2024-02-01T10:18:10.909 DEBUG:boto:Headers: {} 2024-02-01T10:18:10.909 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:10.909 DEBUG:boto:Port: 8001 2024-02-01T10:18:10.909 DEBUG:boto:Params: {} 2024-02-01T10:18:10.910 DEBUG:boto:Token: None 2024-02-01T10:18:10.910 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:10 GMT /vclfcn-11/ 2024-02-01T10:18:10.910 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:M9IZDpl/mkiHoyqfgRxTQVr04Ro= 2024-02-01T10:18:10.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:M9IZDpl/mkiHoyqfgRxTQVr04Ro=', 'Content-Length': '0'} 2024-02-01T10:18:10.914 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a9ce6d89b7c3316-0065bb6fe2-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:10.914 DEBUG:boto:b'vclfcn-11:1000false:2024-02-01T10:16:36.512Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:10.915 DEBUG:boto:path=/vclfcn-11/ 2024-02-01T10:18:10.915 DEBUG:boto:auth_path=/vclfcn-11/ 2024-02-01T10:18:10.915 DEBUG:boto:path=/vclfcn-11/?prefix=%3A 2024-02-01T10:18:10.915 DEBUG:boto:auth_path=/vclfcn-11/?prefix=%3A 2024-02-01T10:18:10.915 DEBUG:boto:Method: GET 2024-02-01T10:18:10.915 DEBUG:boto:Path: /vclfcn-11/?prefix=%3A 2024-02-01T10:18:10.915 DEBUG:boto:Data: 2024-02-01T10:18:10.915 DEBUG:boto:Headers: {} 2024-02-01T10:18:10.915 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:10.916 DEBUG:boto:Port: 8000 2024-02-01T10:18:10.916 DEBUG:boto:Params: {} 2024-02-01T10:18:10.916 DEBUG:boto:Token: None 2024-02-01T10:18:10.916 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:10 GMT /vclfcn-11/ 2024-02-01T10:18:10.916 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:M9IZDpl/mkiHoyqfgRxTQVr04Ro= 2024-02-01T10:18:10.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:M9IZDpl/mkiHoyqfgRxTQVr04Ro=', 'Content-Length': '0'} 2024-02-01T10:18:10.920 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d69bc92040f73184-0065bb6fe2-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:10.920 DEBUG:boto:b'vclfcn-11:1000false:2024-02-01T10:16:36.512Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:10.920 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-02-01T10:18:10.921 DEBUG:boto:path=/vclfcn-11/_myobj 2024-02-01T10:18:10.921 DEBUG:boto:auth_path=/vclfcn-11/_myobj 2024-02-01T10:18:10.921 DEBUG:boto:path=/vclfcn-11/_myobj?versionId=null 2024-02-01T10:18:10.921 DEBUG:boto:auth_path=/vclfcn-11/_myobj?versionId=null 2024-02-01T10:18:10.921 DEBUG:boto:Method: GET 2024-02-01T10:18:10.921 DEBUG:boto:Path: /vclfcn-11/_myobj?versionId=null 2024-02-01T10:18:10.921 DEBUG:boto:Data: 2024-02-01T10:18:10.921 DEBUG:boto:Headers: {} 2024-02-01T10:18:10.921 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:10.921 DEBUG:boto:Port: 8001 2024-02-01T10:18:10.921 DEBUG:boto:Params: {} 2024-02-01T10:18:10.922 DEBUG:boto:Token: None 2024-02-01T10:18:10.922 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:10 GMT /vclfcn-11/_myobj?versionId=null 2024-02-01T10:18:10.922 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:JyOlnz3ChI0iWW1ijCMil0gZySs= 2024-02-01T10:18:10.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:JyOlnz3ChI0iWW1ijCMil0gZySs=', 'Content-Length': '0'} 2024-02-01T10:18:10.925 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000c1fc28bc1bbece8-0065bb6fe2-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:10.968 DEBUG:boto:path=/vclfcn-11/_myobj 2024-02-01T10:18:10.968 DEBUG:boto:auth_path=/vclfcn-11/_myobj 2024-02-01T10:18:10.968 DEBUG:boto:path=/vclfcn-11/_myobj?versionId=null 2024-02-01T10:18:10.969 DEBUG:boto:auth_path=/vclfcn-11/_myobj?versionId=null 2024-02-01T10:18:10.969 DEBUG:boto:Method: GET 2024-02-01T10:18:10.969 DEBUG:boto:Path: /vclfcn-11/_myobj?versionId=null 2024-02-01T10:18:10.969 DEBUG:boto:Data: 2024-02-01T10:18:10.969 DEBUG:boto:Headers: {} 2024-02-01T10:18:10.969 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:10.969 DEBUG:boto:Port: 8000 2024-02-01T10:18:10.969 DEBUG:boto:Params: {} 2024-02-01T10:18:10.969 DEBUG:boto:Token: None 2024-02-01T10:18:10.969 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:10 GMT /vclfcn-11/_myobj?versionId=null 2024-02-01T10:18:10.970 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:JyOlnz3ChI0iWW1ijCMil0gZySs= 2024-02-01T10:18:10.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:JyOlnz3ChI0iWW1ijCMil0gZySs=', 'Content-Length': '0'} 2024-02-01T10:18:10.973 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a022d0791ac0df6c-0065bb6fe2-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:10.973 DEBUG:boto:path=/vclfcn-11/_myobj 2024-02-01T10:18:10.973 DEBUG:boto:auth_path=/vclfcn-11/_myobj 2024-02-01T10:18:10.973 DEBUG:boto:path=/vclfcn-11/_myobj?versionId=null 2024-02-01T10:18:10.973 DEBUG:boto:auth_path=/vclfcn-11/_myobj?versionId=null 2024-02-01T10:18:10.973 DEBUG:boto:Method: HEAD 2024-02-01T10:18:10.974 DEBUG:boto:Path: /vclfcn-11/_myobj?versionId=null 2024-02-01T10:18:10.974 DEBUG:boto:Data: 2024-02-01T10:18:10.974 DEBUG:boto:Headers: {} 2024-02-01T10:18:10.974 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:10.974 DEBUG:boto:Port: 8001 2024-02-01T10:18:10.974 DEBUG:boto:Params: {} 2024-02-01T10:18:10.974 DEBUG:boto:Token: None 2024-02-01T10:18:10.974 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:10 GMT /vclfcn-11/_myobj?versionId=null 2024-02-01T10:18:10.974 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:qKKQckYzIBwWvTLOMS0e8sVpWss= 2024-02-01T10:18:10.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:qKKQckYzIBwWvTLOMS0e8sVpWss=', 'Content-Length': '0'} 2024-02-01T10:18:10.977 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007ac2b2349d7a687c-0065bb6fe2-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:10.978 DEBUG:boto:path=/vclfcn-11/_myobj 2024-02-01T10:18:10.978 DEBUG:boto:auth_path=/vclfcn-11/_myobj 2024-02-01T10:18:10.978 DEBUG:boto:path=/vclfcn-11/_myobj?versionId=null 2024-02-01T10:18:10.978 DEBUG:boto:auth_path=/vclfcn-11/_myobj?versionId=null 2024-02-01T10:18:10.978 DEBUG:boto:Method: HEAD 2024-02-01T10:18:10.978 DEBUG:boto:Path: /vclfcn-11/_myobj?versionId=null 2024-02-01T10:18:10.978 DEBUG:boto:Data: 2024-02-01T10:18:10.978 DEBUG:boto:Headers: {} 2024-02-01T10:18:10.978 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:10.978 DEBUG:boto:Port: 8000 2024-02-01T10:18:10.978 DEBUG:boto:Params: {} 2024-02-01T10:18:10.979 DEBUG:boto:Token: None 2024-02-01T10:18:10.979 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:10 GMT /vclfcn-11/_myobj?versionId=null 2024-02-01T10:18:10.979 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:qKKQckYzIBwWvTLOMS0e8sVpWss= 2024-02-01T10:18:10.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:qKKQckYzIBwWvTLOMS0e8sVpWss=', 'Content-Length': '0'} 2024-02-01T10:18:10.982 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000effe0e66e0d2a714-0065bb6fe2-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:10.983 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-02-01T10:18:10.983 DEBUG:boto:path=/vclfcn-11/_myobj 2024-02-01T10:18:10.983 DEBUG:boto:auth_path=/vclfcn-11/_myobj 2024-02-01T10:18:10.983 DEBUG:boto:path=/vclfcn-11/_myobj?versionId=null 2024-02-01T10:18:10.983 DEBUG:boto:auth_path=/vclfcn-11/_myobj?versionId=null 2024-02-01T10:18:10.983 DEBUG:boto:Method: GET 2024-02-01T10:18:10.983 DEBUG:boto:Path: /vclfcn-11/_myobj?versionId=null 2024-02-01T10:18:10.984 DEBUG:boto:Data: 2024-02-01T10:18:10.984 DEBUG:boto:Headers: {} 2024-02-01T10:18:10.984 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:10.984 DEBUG:boto:Port: 8001 2024-02-01T10:18:10.984 DEBUG:boto:Params: {} 2024-02-01T10:18:10.985 DEBUG:boto:Token: None 2024-02-01T10:18:10.985 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:10 GMT /vclfcn-11/_myobj?versionId=null 2024-02-01T10:18:10.985 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:JyOlnz3ChI0iWW1ijCMil0gZySs= 2024-02-01T10:18:10.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:JyOlnz3ChI0iWW1ijCMil0gZySs=', 'Content-Length': '0'} 2024-02-01T10:18:10.988 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003f0e063be73f4551-0065bb6fe2-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:11.028 DEBUG:boto:path=/vclfcn-11/_myobj 2024-02-01T10:18:11.028 DEBUG:boto:auth_path=/vclfcn-11/_myobj 2024-02-01T10:18:11.028 DEBUG:boto:path=/vclfcn-11/_myobj?versionId=null 2024-02-01T10:18:11.028 DEBUG:boto:auth_path=/vclfcn-11/_myobj?versionId=null 2024-02-01T10:18:11.028 DEBUG:boto:Method: GET 2024-02-01T10:18:11.029 DEBUG:boto:Path: /vclfcn-11/_myobj?versionId=null 2024-02-01T10:18:11.029 DEBUG:boto:Data: 2024-02-01T10:18:11.029 DEBUG:boto:Headers: {} 2024-02-01T10:18:11.029 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:11.029 DEBUG:boto:Port: 8000 2024-02-01T10:18:11.029 DEBUG:boto:Params: {} 2024-02-01T10:18:11.029 DEBUG:boto:Token: None 2024-02-01T10:18:11.029 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:11 GMT /vclfcn-11/_myobj?versionId=null 2024-02-01T10:18:11.029 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:c0si5SkfLNFkVDpR1m07iq6dPjo= 2024-02-01T10:18:11.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:c0si5SkfLNFkVDpR1m07iq6dPjo=', 'Content-Length': '0'} 2024-02-01T10:18:11.032 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000041432975bba26600-0065bb6fe3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:11.033 DEBUG:boto:path=/vclfcn-11/ 2024-02-01T10:18:11.033 DEBUG:boto:auth_path=/vclfcn-11/ 2024-02-01T10:18:11.033 DEBUG:boto:path=/vclfcn-11/?prefix=_myobj 2024-02-01T10:18:11.033 DEBUG:boto:auth_path=/vclfcn-11/?prefix=_myobj 2024-02-01T10:18:11.033 DEBUG:boto:Method: GET 2024-02-01T10:18:11.033 DEBUG:boto:Path: /vclfcn-11/?prefix=_myobj 2024-02-01T10:18:11.033 DEBUG:boto:Data: 2024-02-01T10:18:11.033 DEBUG:boto:Headers: {} 2024-02-01T10:18:11.033 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:11.033 DEBUG:boto:Port: 8001 2024-02-01T10:18:11.034 DEBUG:boto:Params: {} 2024-02-01T10:18:11.034 DEBUG:boto:Token: None 2024-02-01T10:18:11.034 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:11 GMT /vclfcn-11/ 2024-02-01T10:18:11.034 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sBHs6LszJk95INcVIha7qruzAlc= 2024-02-01T10:18:11.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sBHs6LszJk95INcVIha7qruzAlc=', 'Content-Length': '0'} 2024-02-01T10:18:11.039 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ccd355aed9f2b42c-0065bb6fe3-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:11.084 DEBUG:boto:b'vclfcn-11_myobj1000false_myobj2024-02-01T10:16:36.463Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:11.085 DEBUG:boto:path=/vclfcn-11/ 2024-02-01T10:18:11.085 DEBUG:boto:auth_path=/vclfcn-11/ 2024-02-01T10:18:11.085 DEBUG:boto:path=/vclfcn-11/?prefix=_myobj 2024-02-01T10:18:11.085 DEBUG:boto:auth_path=/vclfcn-11/?prefix=_myobj 2024-02-01T10:18:11.085 DEBUG:boto:Method: GET 2024-02-01T10:18:11.085 DEBUG:boto:Path: /vclfcn-11/?prefix=_myobj 2024-02-01T10:18:11.085 DEBUG:boto:Data: 2024-02-01T10:18:11.085 DEBUG:boto:Headers: {} 2024-02-01T10:18:11.085 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:11.085 DEBUG:boto:Port: 8000 2024-02-01T10:18:11.085 DEBUG:boto:Params: {} 2024-02-01T10:18:11.086 DEBUG:boto:Token: None 2024-02-01T10:18:11.086 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:11 GMT /vclfcn-11/ 2024-02-01T10:18:11.086 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sBHs6LszJk95INcVIha7qruzAlc= 2024-02-01T10:18:11.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sBHs6LszJk95INcVIha7qruzAlc=', 'Content-Length': '0'} 2024-02-01T10:18:11.091 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df5abe387f955b6c-0065bb6fe3-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:11.091 DEBUG:boto:b'vclfcn-11_myobj1000false_myobj2024-02-01T10:16:36.463Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:11.091 DEBUG:boto:path=/vclfcn-11/_myobj 2024-02-01T10:18:11.092 DEBUG:boto:auth_path=/vclfcn-11/_myobj 2024-02-01T10:18:11.092 DEBUG:boto:Method: HEAD 2024-02-01T10:18:11.092 DEBUG:boto:Path: /vclfcn-11/_myobj 2024-02-01T10:18:11.092 DEBUG:boto:Data: 2024-02-01T10:18:11.092 DEBUG:boto:Headers: {} 2024-02-01T10:18:11.092 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:11.092 DEBUG:boto:Port: 8001 2024-02-01T10:18:11.092 DEBUG:boto:Params: {} 2024-02-01T10:18:11.092 DEBUG:boto:Token: None 2024-02-01T10:18:11.093 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:11 GMT /vclfcn-11/_myobj 2024-02-01T10:18:11.093 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:SPi1gl+dLMk/D6w2fwYFMHjbGl8= 2024-02-01T10:18:11.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:SPi1gl+dLMk/D6w2fwYFMHjbGl8=', 'Content-Length': '0'} 2024-02-01T10:18:11.096 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009092a4675486dc41-0065bb6fe3-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:11.096 DEBUG:boto:path=/vclfcn-11/_myobj 2024-02-01T10:18:11.096 DEBUG:boto:auth_path=/vclfcn-11/_myobj 2024-02-01T10:18:11.096 DEBUG:boto:Method: HEAD 2024-02-01T10:18:11.096 DEBUG:boto:Path: /vclfcn-11/_myobj 2024-02-01T10:18:11.096 DEBUG:boto:Data: 2024-02-01T10:18:11.096 DEBUG:boto:Headers: {} 2024-02-01T10:18:11.097 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:11.097 DEBUG:boto:Port: 8000 2024-02-01T10:18:11.097 DEBUG:boto:Params: {} 2024-02-01T10:18:11.097 DEBUG:boto:Token: None 2024-02-01T10:18:11.097 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:11 GMT /vclfcn-11/_myobj 2024-02-01T10:18:11.097 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:SPi1gl+dLMk/D6w2fwYFMHjbGl8= 2024-02-01T10:18:11.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:SPi1gl+dLMk/D6w2fwYFMHjbGl8=', 'Content-Length': '0'} 2024-02-01T10:18:11.100 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000015abc0ba27453ba4-0065bb6fe3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:11.100 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-02-01T10:18:11.100 DEBUG:boto:path=/vclfcn-11/_myobj 2024-02-01T10:18:11.100 DEBUG:boto:auth_path=/vclfcn-11/_myobj 2024-02-01T10:18:11.100 DEBUG:boto:Method: GET 2024-02-01T10:18:11.101 DEBUG:boto:Path: /vclfcn-11/_myobj 2024-02-01T10:18:11.101 DEBUG:boto:Data: 2024-02-01T10:18:11.101 DEBUG:boto:Headers: {} 2024-02-01T10:18:11.101 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:11.101 DEBUG:boto:Port: 8001 2024-02-01T10:18:11.101 DEBUG:boto:Params: {} 2024-02-01T10:18:11.101 DEBUG:boto:Token: None 2024-02-01T10:18:11.101 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:11 GMT /vclfcn-11/_myobj 2024-02-01T10:18:11.101 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:CzAg17K+03G7ErzAi6hyZgUxgKc= 2024-02-01T10:18:11.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:CzAg17K+03G7ErzAi6hyZgUxgKc=', 'Content-Length': '0'} 2024-02-01T10:18:11.104 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000be2048bdfa911121-0065bb6fe3-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:11.148 DEBUG:boto:path=/vclfcn-11/_myobj 2024-02-01T10:18:11.148 DEBUG:boto:auth_path=/vclfcn-11/_myobj 2024-02-01T10:18:11.148 DEBUG:boto:Method: GET 2024-02-01T10:18:11.149 DEBUG:boto:Path: /vclfcn-11/_myobj 2024-02-01T10:18:11.149 DEBUG:boto:Data: 2024-02-01T10:18:11.149 DEBUG:boto:Headers: {} 2024-02-01T10:18:11.149 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:11.149 DEBUG:boto:Port: 8000 2024-02-01T10:18:11.149 DEBUG:boto:Params: {} 2024-02-01T10:18:11.149 DEBUG:boto:Token: None 2024-02-01T10:18:11.149 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:11 GMT /vclfcn-11/_myobj 2024-02-01T10:18:11.149 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:CzAg17K+03G7ErzAi6hyZgUxgKc= 2024-02-01T10:18:11.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:CzAg17K+03G7ErzAi6hyZgUxgKc=', 'Content-Length': '0'} 2024-02-01T10:18:11.152 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d54e4c4f39b228c4-0065bb6fe3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:11.152 DEBUG:boto:path=/vclfcn-11/ 2024-02-01T10:18:11.153 DEBUG:boto:auth_path=/vclfcn-11/ 2024-02-01T10:18:11.153 DEBUG:boto:path=/vclfcn-11/?prefix=_myobj 2024-02-01T10:18:11.153 DEBUG:boto:auth_path=/vclfcn-11/?prefix=_myobj 2024-02-01T10:18:11.153 DEBUG:boto:Method: GET 2024-02-01T10:18:11.153 DEBUG:boto:Path: /vclfcn-11/?prefix=_myobj 2024-02-01T10:18:11.153 DEBUG:boto:Data: 2024-02-01T10:18:11.153 DEBUG:boto:Headers: {} 2024-02-01T10:18:11.153 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:11.154 DEBUG:boto:Port: 8001 2024-02-01T10:18:11.154 DEBUG:boto:Params: {} 2024-02-01T10:18:11.154 DEBUG:boto:Token: None 2024-02-01T10:18:11.154 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:11 GMT /vclfcn-11/ 2024-02-01T10:18:11.154 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sBHs6LszJk95INcVIha7qruzAlc= 2024-02-01T10:18:11.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sBHs6LszJk95INcVIha7qruzAlc=', 'Content-Length': '0'} 2024-02-01T10:18:11.158 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005216d32de8aaf018-0065bb6fe3-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:11.200 DEBUG:boto:b'vclfcn-11_myobj1000false_myobj2024-02-01T10:16:36.463Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:11.201 DEBUG:boto:path=/vclfcn-11/ 2024-02-01T10:18:11.201 DEBUG:boto:auth_path=/vclfcn-11/ 2024-02-01T10:18:11.201 DEBUG:boto:path=/vclfcn-11/?prefix=_myobj 2024-02-01T10:18:11.201 DEBUG:boto:auth_path=/vclfcn-11/?prefix=_myobj 2024-02-01T10:18:11.201 DEBUG:boto:Method: GET 2024-02-01T10:18:11.201 DEBUG:boto:Path: /vclfcn-11/?prefix=_myobj 2024-02-01T10:18:11.201 DEBUG:boto:Data: 2024-02-01T10:18:11.201 DEBUG:boto:Headers: {} 2024-02-01T10:18:11.201 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:11.202 DEBUG:boto:Port: 8000 2024-02-01T10:18:11.202 DEBUG:boto:Params: {} 2024-02-01T10:18:11.202 DEBUG:boto:Token: None 2024-02-01T10:18:11.202 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:11 GMT /vclfcn-11/ 2024-02-01T10:18:11.202 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sBHs6LszJk95INcVIha7qruzAlc= 2024-02-01T10:18:11.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sBHs6LszJk95INcVIha7qruzAlc=', 'Content-Length': '0'} 2024-02-01T10:18:11.206 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2710f7d0f9ec911-0065bb6fe3-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:11.207 DEBUG:boto:b'vclfcn-11_myobj1000false_myobj2024-02-01T10:16:36.463Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:11.207 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-02-01T10:18:11.207 DEBUG:boto:path=/vclfcn-11/myobj 2024-02-01T10:18:11.207 DEBUG:boto:auth_path=/vclfcn-11/myobj 2024-02-01T10:18:11.207 DEBUG:boto:path=/vclfcn-11/myobj?versionId=null 2024-02-01T10:18:11.208 DEBUG:boto:auth_path=/vclfcn-11/myobj?versionId=null 2024-02-01T10:18:11.208 DEBUG:boto:Method: GET 2024-02-01T10:18:11.208 DEBUG:boto:Path: /vclfcn-11/myobj?versionId=null 2024-02-01T10:18:11.208 DEBUG:boto:Data: 2024-02-01T10:18:11.208 DEBUG:boto:Headers: {} 2024-02-01T10:18:11.208 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:11.208 DEBUG:boto:Port: 8001 2024-02-01T10:18:11.208 DEBUG:boto:Params: {} 2024-02-01T10:18:11.208 DEBUG:boto:Token: None 2024-02-01T10:18:11.208 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:11 GMT /vclfcn-11/myobj?versionId=null 2024-02-01T10:18:11.209 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:JP07vx5VuroF6uscCMPvcTH0ovc= 2024-02-01T10:18:11.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:JP07vx5VuroF6uscCMPvcTH0ovc=', 'Content-Length': '0'} 2024-02-01T10:18:11.212 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d2d1dcd3d4adba04-0065bb6fe3-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:11.212 DEBUG:boto:path=/vclfcn-11/myobj 2024-02-01T10:18:11.212 DEBUG:boto:auth_path=/vclfcn-11/myobj 2024-02-01T10:18:11.212 DEBUG:boto:path=/vclfcn-11/myobj?versionId=null 2024-02-01T10:18:11.212 DEBUG:boto:auth_path=/vclfcn-11/myobj?versionId=null 2024-02-01T10:18:11.213 DEBUG:boto:Method: GET 2024-02-01T10:18:11.213 DEBUG:boto:Path: /vclfcn-11/myobj?versionId=null 2024-02-01T10:18:11.213 DEBUG:boto:Data: 2024-02-01T10:18:11.213 DEBUG:boto:Headers: {} 2024-02-01T10:18:11.213 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:11.213 DEBUG:boto:Port: 8000 2024-02-01T10:18:11.213 DEBUG:boto:Params: {} 2024-02-01T10:18:11.213 DEBUG:boto:Token: None 2024-02-01T10:18:11.213 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:11 GMT /vclfcn-11/myobj?versionId=null 2024-02-01T10:18:11.214 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:JP07vx5VuroF6uscCMPvcTH0ovc= 2024-02-01T10:18:11.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:JP07vx5VuroF6uscCMPvcTH0ovc=', 'Content-Length': '0'} 2024-02-01T10:18:11.216 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000870a5dc956ff4d28-0065bb6fe3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:11.217 DEBUG:boto:path=/vclfcn-11/myobj 2024-02-01T10:18:11.217 DEBUG:boto:auth_path=/vclfcn-11/myobj 2024-02-01T10:18:11.217 DEBUG:boto:path=/vclfcn-11/myobj?versionId=null 2024-02-01T10:18:11.217 DEBUG:boto:auth_path=/vclfcn-11/myobj?versionId=null 2024-02-01T10:18:11.217 DEBUG:boto:Method: HEAD 2024-02-01T10:18:11.217 DEBUG:boto:Path: /vclfcn-11/myobj?versionId=null 2024-02-01T10:18:11.217 DEBUG:boto:Data: 2024-02-01T10:18:11.217 DEBUG:boto:Headers: {} 2024-02-01T10:18:11.217 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:11.218 DEBUG:boto:Port: 8001 2024-02-01T10:18:11.218 DEBUG:boto:Params: {} 2024-02-01T10:18:11.218 DEBUG:boto:Token: None 2024-02-01T10:18:11.218 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:11 GMT /vclfcn-11/myobj?versionId=null 2024-02-01T10:18:11.218 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cjlbhy5I7AxqGvylpxEf5XDYfOk= 2024-02-01T10:18:11.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cjlbhy5I7AxqGvylpxEf5XDYfOk=', 'Content-Length': '0'} 2024-02-01T10:18:11.221 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001ec74ea6e2dc927e-0065bb6fe3-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:11.221 DEBUG:boto:path=/vclfcn-11/myobj 2024-02-01T10:18:11.222 DEBUG:boto:auth_path=/vclfcn-11/myobj 2024-02-01T10:18:11.222 DEBUG:boto:path=/vclfcn-11/myobj?versionId=null 2024-02-01T10:18:11.222 DEBUG:boto:auth_path=/vclfcn-11/myobj?versionId=null 2024-02-01T10:18:11.222 DEBUG:boto:Method: HEAD 2024-02-01T10:18:11.222 DEBUG:boto:Path: /vclfcn-11/myobj?versionId=null 2024-02-01T10:18:11.222 DEBUG:boto:Data: 2024-02-01T10:18:11.222 DEBUG:boto:Headers: {} 2024-02-01T10:18:11.222 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:11.222 DEBUG:boto:Port: 8000 2024-02-01T10:18:11.222 DEBUG:boto:Params: {} 2024-02-01T10:18:11.222 DEBUG:boto:Token: None 2024-02-01T10:18:11.223 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:11 GMT /vclfcn-11/myobj?versionId=null 2024-02-01T10:18:11.223 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cjlbhy5I7AxqGvylpxEf5XDYfOk= 2024-02-01T10:18:11.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cjlbhy5I7AxqGvylpxEf5XDYfOk=', 'Content-Length': '0'} 2024-02-01T10:18:11.226 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001ba92d2e1105f751-0065bb6fe3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:11.226 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-02-01T10:18:11.226 DEBUG:boto:path=/vclfcn-11/myobj 2024-02-01T10:18:11.226 DEBUG:boto:auth_path=/vclfcn-11/myobj 2024-02-01T10:18:11.227 DEBUG:boto:path=/vclfcn-11/myobj?versionId=null 2024-02-01T10:18:11.227 DEBUG:boto:auth_path=/vclfcn-11/myobj?versionId=null 2024-02-01T10:18:11.227 DEBUG:boto:Method: GET 2024-02-01T10:18:11.227 DEBUG:boto:Path: /vclfcn-11/myobj?versionId=null 2024-02-01T10:18:11.227 DEBUG:boto:Data: 2024-02-01T10:18:11.227 DEBUG:boto:Headers: {} 2024-02-01T10:18:11.227 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:11.227 DEBUG:boto:Port: 8001 2024-02-01T10:18:11.227 DEBUG:boto:Params: {} 2024-02-01T10:18:11.227 DEBUG:boto:Token: None 2024-02-01T10:18:11.228 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:11 GMT /vclfcn-11/myobj?versionId=null 2024-02-01T10:18:11.228 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:JP07vx5VuroF6uscCMPvcTH0ovc= 2024-02-01T10:18:11.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:JP07vx5VuroF6uscCMPvcTH0ovc=', 'Content-Length': '0'} 2024-02-01T10:18:11.231 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000847ae822b05314e7-0065bb6fe3-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:11.272 DEBUG:boto:path=/vclfcn-11/myobj 2024-02-01T10:18:11.272 DEBUG:boto:auth_path=/vclfcn-11/myobj 2024-02-01T10:18:11.272 DEBUG:boto:path=/vclfcn-11/myobj?versionId=null 2024-02-01T10:18:11.273 DEBUG:boto:auth_path=/vclfcn-11/myobj?versionId=null 2024-02-01T10:18:11.273 DEBUG:boto:Method: GET 2024-02-01T10:18:11.273 DEBUG:boto:Path: /vclfcn-11/myobj?versionId=null 2024-02-01T10:18:11.273 DEBUG:boto:Data: 2024-02-01T10:18:11.273 DEBUG:boto:Headers: {} 2024-02-01T10:18:11.273 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:11.273 DEBUG:boto:Port: 8000 2024-02-01T10:18:11.273 DEBUG:boto:Params: {} 2024-02-01T10:18:11.273 DEBUG:boto:Token: None 2024-02-01T10:18:11.274 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:11 GMT /vclfcn-11/myobj?versionId=null 2024-02-01T10:18:11.274 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:JP07vx5VuroF6uscCMPvcTH0ovc= 2024-02-01T10:18:11.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:JP07vx5VuroF6uscCMPvcTH0ovc=', 'Content-Length': '0'} 2024-02-01T10:18:11.277 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000b728cf96d40ba03-0065bb6fe3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:11.277 DEBUG:boto:path=/vclfcn-11/ 2024-02-01T10:18:11.278 DEBUG:boto:auth_path=/vclfcn-11/ 2024-02-01T10:18:11.278 DEBUG:boto:path=/vclfcn-11/?prefix=myobj 2024-02-01T10:18:11.278 DEBUG:boto:auth_path=/vclfcn-11/?prefix=myobj 2024-02-01T10:18:11.278 DEBUG:boto:Method: GET 2024-02-01T10:18:11.278 DEBUG:boto:Path: /vclfcn-11/?prefix=myobj 2024-02-01T10:18:11.278 DEBUG:boto:Data: 2024-02-01T10:18:11.278 DEBUG:boto:Headers: {} 2024-02-01T10:18:11.278 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:11.278 DEBUG:boto:Port: 8001 2024-02-01T10:18:11.278 DEBUG:boto:Params: {} 2024-02-01T10:18:11.279 DEBUG:boto:Token: None 2024-02-01T10:18:11.279 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:11 GMT /vclfcn-11/ 2024-02-01T10:18:11.279 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sBHs6LszJk95INcVIha7qruzAlc= 2024-02-01T10:18:11.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sBHs6LszJk95INcVIha7qruzAlc=', 'Content-Length': '0'} 2024-02-01T10:18:11.284 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000195c66c69d4de630-0065bb6fe3-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:11.324 DEBUG:boto:b'vclfcn-11myobj1000falsemyobj2024-02-01T10:16:36.434Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:11.325 DEBUG:boto:path=/vclfcn-11/ 2024-02-01T10:18:11.325 DEBUG:boto:auth_path=/vclfcn-11/ 2024-02-01T10:18:11.325 DEBUG:boto:path=/vclfcn-11/?prefix=myobj 2024-02-01T10:18:11.325 DEBUG:boto:auth_path=/vclfcn-11/?prefix=myobj 2024-02-01T10:18:11.325 DEBUG:boto:Method: GET 2024-02-01T10:18:11.325 DEBUG:boto:Path: /vclfcn-11/?prefix=myobj 2024-02-01T10:18:11.325 DEBUG:boto:Data: 2024-02-01T10:18:11.325 DEBUG:boto:Headers: {} 2024-02-01T10:18:11.325 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:11.326 DEBUG:boto:Port: 8000 2024-02-01T10:18:11.326 DEBUG:boto:Params: {} 2024-02-01T10:18:11.326 DEBUG:boto:Token: None 2024-02-01T10:18:11.326 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:11 GMT /vclfcn-11/ 2024-02-01T10:18:11.326 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sBHs6LszJk95INcVIha7qruzAlc= 2024-02-01T10:18:11.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sBHs6LszJk95INcVIha7qruzAlc=', 'Content-Length': '0'} 2024-02-01T10:18:11.331 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d466021398ab6ed-0065bb6fe3-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:11.331 DEBUG:boto:b'vclfcn-11myobj1000falsemyobj2024-02-01T10:16:36.434Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:11.331 DEBUG:boto:path=/vclfcn-11/myobj 2024-02-01T10:18:11.331 DEBUG:boto:auth_path=/vclfcn-11/myobj 2024-02-01T10:18:11.331 DEBUG:boto:Method: HEAD 2024-02-01T10:18:11.331 DEBUG:boto:Path: /vclfcn-11/myobj 2024-02-01T10:18:11.332 DEBUG:boto:Data: 2024-02-01T10:18:11.332 DEBUG:boto:Headers: {} 2024-02-01T10:18:11.332 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:11.332 DEBUG:boto:Port: 8001 2024-02-01T10:18:11.332 DEBUG:boto:Params: {} 2024-02-01T10:18:11.332 DEBUG:boto:Token: None 2024-02-01T10:18:11.332 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:11 GMT /vclfcn-11/myobj 2024-02-01T10:18:11.333 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HCnyeCyixur2dOBT1lyzgFca6h0= 2024-02-01T10:18:11.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HCnyeCyixur2dOBT1lyzgFca6h0=', 'Content-Length': '0'} 2024-02-01T10:18:11.336 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c3a506fa80498d14-0065bb6fe3-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:11.336 DEBUG:boto:path=/vclfcn-11/myobj 2024-02-01T10:18:11.336 DEBUG:boto:auth_path=/vclfcn-11/myobj 2024-02-01T10:18:11.336 DEBUG:boto:Method: HEAD 2024-02-01T10:18:11.336 DEBUG:boto:Path: /vclfcn-11/myobj 2024-02-01T10:18:11.336 DEBUG:boto:Data: 2024-02-01T10:18:11.337 DEBUG:boto:Headers: {} 2024-02-01T10:18:11.337 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:11.337 DEBUG:boto:Port: 8000 2024-02-01T10:18:11.337 DEBUG:boto:Params: {} 2024-02-01T10:18:11.337 DEBUG:boto:Token: None 2024-02-01T10:18:11.337 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:11 GMT /vclfcn-11/myobj 2024-02-01T10:18:11.337 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HCnyeCyixur2dOBT1lyzgFca6h0= 2024-02-01T10:18:11.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HCnyeCyixur2dOBT1lyzgFca6h0=', 'Content-Length': '0'} 2024-02-01T10:18:11.340 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008198156748ec353e-0065bb6fe3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:11.340 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-02-01T10:18:11.340 DEBUG:boto:path=/vclfcn-11/myobj 2024-02-01T10:18:11.340 DEBUG:boto:auth_path=/vclfcn-11/myobj 2024-02-01T10:18:11.340 DEBUG:boto:Method: GET 2024-02-01T10:18:11.341 DEBUG:boto:Path: /vclfcn-11/myobj 2024-02-01T10:18:11.341 DEBUG:boto:Data: 2024-02-01T10:18:11.341 DEBUG:boto:Headers: {} 2024-02-01T10:18:11.341 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:11.341 DEBUG:boto:Port: 8001 2024-02-01T10:18:11.341 DEBUG:boto:Params: {} 2024-02-01T10:18:11.341 DEBUG:boto:Token: None 2024-02-01T10:18:11.342 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:11 GMT /vclfcn-11/myobj 2024-02-01T10:18:11.342 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:H26/ZEjZ08wNADvdDpYcg6aeXG0= 2024-02-01T10:18:11.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:H26/ZEjZ08wNADvdDpYcg6aeXG0=', 'Content-Length': '0'} 2024-02-01T10:18:11.344 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007bda569c8770e605-0065bb6fe3-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:11.388 DEBUG:boto:path=/vclfcn-11/myobj 2024-02-01T10:18:11.388 DEBUG:boto:auth_path=/vclfcn-11/myobj 2024-02-01T10:18:11.388 DEBUG:boto:Method: GET 2024-02-01T10:18:11.389 DEBUG:boto:Path: /vclfcn-11/myobj 2024-02-01T10:18:11.389 DEBUG:boto:Data: 2024-02-01T10:18:11.389 DEBUG:boto:Headers: {} 2024-02-01T10:18:11.389 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:11.389 DEBUG:boto:Port: 8000 2024-02-01T10:18:11.389 DEBUG:boto:Params: {} 2024-02-01T10:18:11.389 DEBUG:boto:Token: None 2024-02-01T10:18:11.389 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:11 GMT /vclfcn-11/myobj 2024-02-01T10:18:11.390 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:H26/ZEjZ08wNADvdDpYcg6aeXG0= 2024-02-01T10:18:11.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:H26/ZEjZ08wNADvdDpYcg6aeXG0=', 'Content-Length': '0'} 2024-02-01T10:18:11.392 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004a692d0b5474f919-0065bb6fe3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:11.393 DEBUG:boto:path=/vclfcn-11/ 2024-02-01T10:18:11.393 DEBUG:boto:auth_path=/vclfcn-11/ 2024-02-01T10:18:11.393 DEBUG:boto:path=/vclfcn-11/?prefix=myobj 2024-02-01T10:18:11.393 DEBUG:boto:auth_path=/vclfcn-11/?prefix=myobj 2024-02-01T10:18:11.393 DEBUG:boto:Method: GET 2024-02-01T10:18:11.393 DEBUG:boto:Path: /vclfcn-11/?prefix=myobj 2024-02-01T10:18:11.393 DEBUG:boto:Data: 2024-02-01T10:18:11.393 DEBUG:boto:Headers: {} 2024-02-01T10:18:11.393 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:11.393 DEBUG:boto:Port: 8001 2024-02-01T10:18:11.394 DEBUG:boto:Params: {} 2024-02-01T10:18:11.394 DEBUG:boto:Token: None 2024-02-01T10:18:11.394 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:11 GMT /vclfcn-11/ 2024-02-01T10:18:11.394 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sBHs6LszJk95INcVIha7qruzAlc= 2024-02-01T10:18:11.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sBHs6LszJk95INcVIha7qruzAlc=', 'Content-Length': '0'} 2024-02-01T10:18:11.398 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca4b007b5af0b630-0065bb6fe3-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:11.399 DEBUG:boto:b'vclfcn-11myobj1000falsemyobj2024-02-01T10:16:36.434Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:11.399 DEBUG:boto:path=/vclfcn-11/ 2024-02-01T10:18:11.399 DEBUG:boto:auth_path=/vclfcn-11/ 2024-02-01T10:18:11.399 DEBUG:boto:path=/vclfcn-11/?prefix=myobj 2024-02-01T10:18:11.400 DEBUG:boto:auth_path=/vclfcn-11/?prefix=myobj 2024-02-01T10:18:11.400 DEBUG:boto:Method: GET 2024-02-01T10:18:11.400 DEBUG:boto:Path: /vclfcn-11/?prefix=myobj 2024-02-01T10:18:11.400 DEBUG:boto:Data: 2024-02-01T10:18:11.400 DEBUG:boto:Headers: {} 2024-02-01T10:18:11.400 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:11.400 DEBUG:boto:Port: 8000 2024-02-01T10:18:11.400 DEBUG:boto:Params: {} 2024-02-01T10:18:11.400 DEBUG:boto:Token: None 2024-02-01T10:18:11.400 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:11 GMT /vclfcn-11/ 2024-02-01T10:18:11.401 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sBHs6LszJk95INcVIha7qruzAlc= 2024-02-01T10:18:11.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sBHs6LszJk95INcVIha7qruzAlc=', 'Content-Length': '0'} 2024-02-01T10:18:11.405 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ff9e2c1dd4596ff-0065bb6fe3-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:11.405 DEBUG:boto:b'vclfcn-11myobj1000falsemyobj2024-02-01T10:16:36.434Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:11.406 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-11 zones={test-zone3, test-zone2} 2024-02-01T10:18:11.406 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:18:11.406 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:18:11.406 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-12 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:18:11.457 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:18:11.457 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:18:11.517 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:18:11.515+0000 7f4ddb0c8980 1 bucket sync caught up with source: 2024-02-01T10:18:11.517 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, 00000000001.13.6, , 00000000001.10.6, , , , 00000000001.14.6, 00000000001.9.6, , ] 2024-02-01T10:18:11.517 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, 00000000001.13.6, , 00000000001.10.6, , , , 00000000001.14.6, 00000000001.9.6, , ] 2024-02-01T10:18:11.517 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:18:11.515+0000 7f4ddb0c8980 0 bucket checkpoint complete 2024-02-01T10:18:11.539 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-12 zones={test-zone1, test-zone3} 2024-02-01T10:18:11.539 DEBUG:boto:path=/vclfcn-12/ 2024-02-01T10:18:11.539 DEBUG:boto:auth_path=/vclfcn-12/ 2024-02-01T10:18:11.539 DEBUG:boto:Method: HEAD 2024-02-01T10:18:11.539 DEBUG:boto:Path: /vclfcn-12/ 2024-02-01T10:18:11.539 DEBUG:boto:Data: 2024-02-01T10:18:11.539 DEBUG:boto:Headers: {} 2024-02-01T10:18:11.540 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:11.540 DEBUG:boto:Port: 8000 2024-02-01T10:18:11.540 DEBUG:boto:Params: {} 2024-02-01T10:18:11.540 DEBUG:boto:Token: None 2024-02-01T10:18:11.540 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:11 GMT /vclfcn-12/ 2024-02-01T10:18:11.540 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:JfkMSm1zNK9iy7oyH8bB7/1F9BA= 2024-02-01T10:18:11.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:JfkMSm1zNK9iy7oyH8bB7/1F9BA=', 'Content-Length': '0'} 2024-02-01T10:18:11.543 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000857d263120ba71a1-0065bb6fe3-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:18:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:11.543 DEBUG:boto:path=/vclfcn-12/ 2024-02-01T10:18:11.543 DEBUG:boto:auth_path=/vclfcn-12/ 2024-02-01T10:18:11.543 DEBUG:boto:Method: HEAD 2024-02-01T10:18:11.543 DEBUG:boto:Path: /vclfcn-12/ 2024-02-01T10:18:11.543 DEBUG:boto:Data: 2024-02-01T10:18:11.543 DEBUG:boto:Headers: {} 2024-02-01T10:18:11.543 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:11.543 DEBUG:boto:Port: 8001 2024-02-01T10:18:11.543 DEBUG:boto:Params: {} 2024-02-01T10:18:11.544 DEBUG:boto:Token: None 2024-02-01T10:18:11.544 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:11 GMT /vclfcn-12/ 2024-02-01T10:18:11.544 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:JfkMSm1zNK9iy7oyH8bB7/1F9BA= 2024-02-01T10:18:11.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:JfkMSm1zNK9iy7oyH8bB7/1F9BA=', 'Content-Length': '0'} 2024-02-01T10:18:11.546 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000065315e51838aa819-0065bb6fe3-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:18:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:11.546 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:18:11.546 DEBUG:boto:path=/vclfcn-12/ 2024-02-01T10:18:11.546 DEBUG:boto:auth_path=/vclfcn-12/ 2024-02-01T10:18:11.546 DEBUG:boto:path=/vclfcn-12/?versions&max-keys=999 2024-02-01T10:18:11.546 DEBUG:boto:auth_path=/vclfcn-12/?versions&max-keys=999 2024-02-01T10:18:11.547 DEBUG:boto:Method: GET 2024-02-01T10:18:11.547 DEBUG:boto:Path: /vclfcn-12/?versions&max-keys=999 2024-02-01T10:18:11.547 DEBUG:boto:Data: 2024-02-01T10:18:11.547 DEBUG:boto:Headers: {} 2024-02-01T10:18:11.547 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:11.547 DEBUG:boto:Port: 8000 2024-02-01T10:18:11.547 DEBUG:boto:Params: {} 2024-02-01T10:18:11.547 DEBUG:boto:Token: None 2024-02-01T10:18:11.547 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:11 GMT /vclfcn-12/?versions 2024-02-01T10:18:11.547 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:VeHQr8sctRKPH0jIIK1mrYtZu5Q= 2024-02-01T10:18:11.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:VeHQr8sctRKPH0jIIK1mrYtZu5Q=', 'Content-Length': '0'} 2024-02-01T10:18:11.550 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d311e4b92b612c0-0065bb6fe3-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:11.550 DEBUG:boto:b'vclfcn-12999false&nulltrue2024-02-01T10:16:36.721Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-02-01T10:16:36.670Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-02-01T10:16:36.640Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-02-01T10:16:36.593Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:11.551 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-02-01T10:18:11.551 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-02-01T10:18:11.551 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-02-01T10:18:11.551 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-02-01T10:18:11.551 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:18:11.551 DEBUG:boto:path=/vclfcn-12/ 2024-02-01T10:18:11.551 DEBUG:boto:auth_path=/vclfcn-12/ 2024-02-01T10:18:11.551 DEBUG:boto:path=/vclfcn-12/?versions&max-keys=999 2024-02-01T10:18:11.551 DEBUG:boto:auth_path=/vclfcn-12/?versions&max-keys=999 2024-02-01T10:18:11.552 DEBUG:boto:Method: GET 2024-02-01T10:18:11.553 DEBUG:boto:Path: /vclfcn-12/?versions&max-keys=999 2024-02-01T10:18:11.553 DEBUG:boto:Data: 2024-02-01T10:18:11.553 DEBUG:boto:Headers: {} 2024-02-01T10:18:11.553 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:11.553 DEBUG:boto:Port: 8001 2024-02-01T10:18:11.553 DEBUG:boto:Params: {} 2024-02-01T10:18:11.553 DEBUG:boto:Token: None 2024-02-01T10:18:11.553 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:11 GMT /vclfcn-12/?versions 2024-02-01T10:18:11.554 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:VeHQr8sctRKPH0jIIK1mrYtZu5Q= 2024-02-01T10:18:11.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:VeHQr8sctRKPH0jIIK1mrYtZu5Q=', 'Content-Length': '0'} 2024-02-01T10:18:11.557 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce49781d39b6f377-0065bb6fe3-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:11.557 DEBUG:boto:b'vclfcn-12999false&nulltrue2024-02-01T10:16:36.721Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-02-01T10:16:36.670Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-02-01T10:16:36.640Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-02-01T10:16:36.593Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:11.557 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-02-01T10:18:11.558 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-02-01T10:18:11.558 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-02-01T10:18:11.558 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-02-01T10:18:11.558 DEBUG:boto:path=/vclfcn-12/ 2024-02-01T10:18:11.558 DEBUG:boto:auth_path=/vclfcn-12/ 2024-02-01T10:18:11.558 DEBUG:boto:path=/vclfcn-12/?versions&max-keys=999 2024-02-01T10:18:11.558 DEBUG:boto:auth_path=/vclfcn-12/?versions&max-keys=999 2024-02-01T10:18:11.558 DEBUG:boto:Method: GET 2024-02-01T10:18:11.558 DEBUG:boto:Path: /vclfcn-12/?versions&max-keys=999 2024-02-01T10:18:11.558 DEBUG:boto:Data: 2024-02-01T10:18:11.559 DEBUG:boto:Headers: {} 2024-02-01T10:18:11.559 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:11.559 DEBUG:boto:Port: 8000 2024-02-01T10:18:11.559 DEBUG:boto:Params: {} 2024-02-01T10:18:11.559 DEBUG:boto:Token: None 2024-02-01T10:18:11.559 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:11 GMT /vclfcn-12/?versions 2024-02-01T10:18:11.559 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:VeHQr8sctRKPH0jIIK1mrYtZu5Q= 2024-02-01T10:18:11.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:VeHQr8sctRKPH0jIIK1mrYtZu5Q=', 'Content-Length': '0'} 2024-02-01T10:18:11.561 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b138facd62f799e-0065bb6fe3-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:11.561 DEBUG:boto:b'vclfcn-12999false&nulltrue2024-02-01T10:16:36.721Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-02-01T10:16:36.670Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-02-01T10:16:36.640Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-02-01T10:16:36.593Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:11.562 DEBUG:boto:path=/vclfcn-12/ 2024-02-01T10:18:11.562 DEBUG:boto:auth_path=/vclfcn-12/ 2024-02-01T10:18:11.562 DEBUG:boto:path=/vclfcn-12/?versions&max-keys=999 2024-02-01T10:18:11.562 DEBUG:boto:auth_path=/vclfcn-12/?versions&max-keys=999 2024-02-01T10:18:11.562 DEBUG:boto:Method: GET 2024-02-01T10:18:11.562 DEBUG:boto:Path: /vclfcn-12/?versions&max-keys=999 2024-02-01T10:18:11.563 DEBUG:boto:Data: 2024-02-01T10:18:11.563 DEBUG:boto:Headers: {} 2024-02-01T10:18:11.563 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:11.563 DEBUG:boto:Port: 8001 2024-02-01T10:18:11.563 DEBUG:boto:Params: {} 2024-02-01T10:18:11.563 DEBUG:boto:Token: None 2024-02-01T10:18:11.563 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:11 GMT /vclfcn-12/?versions 2024-02-01T10:18:11.563 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:VeHQr8sctRKPH0jIIK1mrYtZu5Q= 2024-02-01T10:18:11.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:VeHQr8sctRKPH0jIIK1mrYtZu5Q=', 'Content-Length': '0'} 2024-02-01T10:18:11.565 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000abe4f9d2dc09ed80-0065bb6fe3-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:11.566 DEBUG:boto:b'vclfcn-12999false&nulltrue2024-02-01T10:16:36.721Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-02-01T10:16:36.670Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-02-01T10:16:36.640Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-02-01T10:16:36.593Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:11.566 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-02-01T10:18:11.566 DEBUG:boto:path=/vclfcn-12/%26 2024-02-01T10:18:11.566 DEBUG:boto:auth_path=/vclfcn-12/%26 2024-02-01T10:18:11.566 DEBUG:boto:path=/vclfcn-12/%26?versionId=null 2024-02-01T10:18:11.566 DEBUG:boto:auth_path=/vclfcn-12/%26?versionId=null 2024-02-01T10:18:11.567 DEBUG:boto:Method: GET 2024-02-01T10:18:11.567 DEBUG:boto:Path: /vclfcn-12/%26?versionId=null 2024-02-01T10:18:11.567 DEBUG:boto:Data: 2024-02-01T10:18:11.567 DEBUG:boto:Headers: {} 2024-02-01T10:18:11.567 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:11.567 DEBUG:boto:Port: 8000 2024-02-01T10:18:11.567 DEBUG:boto:Params: {} 2024-02-01T10:18:11.567 DEBUG:boto:Token: None 2024-02-01T10:18:11.567 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:11 GMT /vclfcn-12/%26?versionId=null 2024-02-01T10:18:11.567 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:PEtl4HS0DeejUZb2hcb4es/eBFo= 2024-02-01T10:18:11.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:PEtl4HS0DeejUZb2hcb4es/eBFo=', 'Content-Length': '0'} 2024-02-01T10:18:11.569 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006874ae3066dedee8-0065bb6fe3-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:11.570 DEBUG:boto:path=/vclfcn-12/%26 2024-02-01T10:18:11.570 DEBUG:boto:auth_path=/vclfcn-12/%26 2024-02-01T10:18:11.570 DEBUG:boto:path=/vclfcn-12/%26?versionId=null 2024-02-01T10:18:11.570 DEBUG:boto:auth_path=/vclfcn-12/%26?versionId=null 2024-02-01T10:18:11.570 DEBUG:boto:Method: GET 2024-02-01T10:18:11.570 DEBUG:boto:Path: /vclfcn-12/%26?versionId=null 2024-02-01T10:18:11.570 DEBUG:boto:Data: 2024-02-01T10:18:11.570 DEBUG:boto:Headers: {} 2024-02-01T10:18:11.571 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:11.571 DEBUG:boto:Port: 8001 2024-02-01T10:18:11.571 DEBUG:boto:Params: {} 2024-02-01T10:18:11.571 DEBUG:boto:Token: None 2024-02-01T10:18:11.571 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:11 GMT /vclfcn-12/%26?versionId=null 2024-02-01T10:18:11.571 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:PEtl4HS0DeejUZb2hcb4es/eBFo= 2024-02-01T10:18:11.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:PEtl4HS0DeejUZb2hcb4es/eBFo=', 'Content-Length': '0'} 2024-02-01T10:18:11.573 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c72f074a24565630-0065bb6fe3-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:11.616 DEBUG:boto:path=/vclfcn-12/%26 2024-02-01T10:18:11.616 DEBUG:boto:auth_path=/vclfcn-12/%26 2024-02-01T10:18:11.616 DEBUG:boto:path=/vclfcn-12/%26?versionId=null 2024-02-01T10:18:11.617 DEBUG:boto:auth_path=/vclfcn-12/%26?versionId=null 2024-02-01T10:18:11.617 DEBUG:boto:Method: HEAD 2024-02-01T10:18:11.617 DEBUG:boto:Path: /vclfcn-12/%26?versionId=null 2024-02-01T10:18:11.617 DEBUG:boto:Data: 2024-02-01T10:18:11.617 DEBUG:boto:Headers: {} 2024-02-01T10:18:11.617 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:11.617 DEBUG:boto:Port: 8000 2024-02-01T10:18:11.617 DEBUG:boto:Params: {} 2024-02-01T10:18:11.617 DEBUG:boto:Token: None 2024-02-01T10:18:11.618 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:11 GMT /vclfcn-12/%26?versionId=null 2024-02-01T10:18:11.618 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:yX9azKVAIxgg4RS/3FIxs+oRSAw= 2024-02-01T10:18:11.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:yX9azKVAIxgg4RS/3FIxs+oRSAw=', 'Content-Length': '0'} 2024-02-01T10:18:11.621 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000edee487735f99d96-0065bb6fe3-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:11.621 DEBUG:boto:path=/vclfcn-12/%26 2024-02-01T10:18:11.621 DEBUG:boto:auth_path=/vclfcn-12/%26 2024-02-01T10:18:11.622 DEBUG:boto:path=/vclfcn-12/%26?versionId=null 2024-02-01T10:18:11.622 DEBUG:boto:auth_path=/vclfcn-12/%26?versionId=null 2024-02-01T10:18:11.622 DEBUG:boto:Method: HEAD 2024-02-01T10:18:11.622 DEBUG:boto:Path: /vclfcn-12/%26?versionId=null 2024-02-01T10:18:11.622 DEBUG:boto:Data: 2024-02-01T10:18:11.622 DEBUG:boto:Headers: {} 2024-02-01T10:18:11.622 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:11.622 DEBUG:boto:Port: 8001 2024-02-01T10:18:11.622 DEBUG:boto:Params: {} 2024-02-01T10:18:11.622 DEBUG:boto:Token: None 2024-02-01T10:18:11.623 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:11 GMT /vclfcn-12/%26?versionId=null 2024-02-01T10:18:11.623 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:yX9azKVAIxgg4RS/3FIxs+oRSAw= 2024-02-01T10:18:11.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:yX9azKVAIxgg4RS/3FIxs+oRSAw=', 'Content-Length': '0'} 2024-02-01T10:18:11.625 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008bf2409eeb1a660d-0065bb6fe3-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:11.625 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-02-01T10:18:11.625 DEBUG:boto:path=/vclfcn-12/%26 2024-02-01T10:18:11.625 DEBUG:boto:auth_path=/vclfcn-12/%26 2024-02-01T10:18:11.625 DEBUG:boto:path=/vclfcn-12/%26?versionId=null 2024-02-01T10:18:11.625 DEBUG:boto:auth_path=/vclfcn-12/%26?versionId=null 2024-02-01T10:18:11.626 DEBUG:boto:Method: GET 2024-02-01T10:18:11.626 DEBUG:boto:Path: /vclfcn-12/%26?versionId=null 2024-02-01T10:18:11.626 DEBUG:boto:Data: 2024-02-01T10:18:11.626 DEBUG:boto:Headers: {} 2024-02-01T10:18:11.626 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:11.626 DEBUG:boto:Port: 8000 2024-02-01T10:18:11.626 DEBUG:boto:Params: {} 2024-02-01T10:18:11.626 DEBUG:boto:Token: None 2024-02-01T10:18:11.626 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:11 GMT /vclfcn-12/%26?versionId=null 2024-02-01T10:18:11.626 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:PEtl4HS0DeejUZb2hcb4es/eBFo= 2024-02-01T10:18:11.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:PEtl4HS0DeejUZb2hcb4es/eBFo=', 'Content-Length': '0'} 2024-02-01T10:18:11.629 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002832b5cc11b09d7f-0065bb6fe3-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:11.672 DEBUG:boto:path=/vclfcn-12/%26 2024-02-01T10:18:11.672 DEBUG:boto:auth_path=/vclfcn-12/%26 2024-02-01T10:18:11.672 DEBUG:boto:path=/vclfcn-12/%26?versionId=null 2024-02-01T10:18:11.673 DEBUG:boto:auth_path=/vclfcn-12/%26?versionId=null 2024-02-01T10:18:11.673 DEBUG:boto:Method: GET 2024-02-01T10:18:11.673 DEBUG:boto:Path: /vclfcn-12/%26?versionId=null 2024-02-01T10:18:11.673 DEBUG:boto:Data: 2024-02-01T10:18:11.673 DEBUG:boto:Headers: {} 2024-02-01T10:18:11.673 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:11.673 DEBUG:boto:Port: 8001 2024-02-01T10:18:11.673 DEBUG:boto:Params: {} 2024-02-01T10:18:11.673 DEBUG:boto:Token: None 2024-02-01T10:18:11.673 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:11 GMT /vclfcn-12/%26?versionId=null 2024-02-01T10:18:11.674 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:PEtl4HS0DeejUZb2hcb4es/eBFo= 2024-02-01T10:18:11.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:PEtl4HS0DeejUZb2hcb4es/eBFo=', 'Content-Length': '0'} 2024-02-01T10:18:11.677 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b7409242838517e1-0065bb6fe3-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:11.677 DEBUG:boto:path=/vclfcn-12/ 2024-02-01T10:18:11.677 DEBUG:boto:auth_path=/vclfcn-12/ 2024-02-01T10:18:11.677 DEBUG:boto:path=/vclfcn-12/?prefix=%26 2024-02-01T10:18:11.677 DEBUG:boto:auth_path=/vclfcn-12/?prefix=%26 2024-02-01T10:18:11.677 DEBUG:boto:Method: GET 2024-02-01T10:18:11.677 DEBUG:boto:Path: /vclfcn-12/?prefix=%26 2024-02-01T10:18:11.678 DEBUG:boto:Data: 2024-02-01T10:18:11.678 DEBUG:boto:Headers: {} 2024-02-01T10:18:11.678 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:11.678 DEBUG:boto:Port: 8000 2024-02-01T10:18:11.678 DEBUG:boto:Params: {} 2024-02-01T10:18:11.678 DEBUG:boto:Token: None 2024-02-01T10:18:11.678 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:11 GMT /vclfcn-12/ 2024-02-01T10:18:11.678 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:CTVz9ejk+2+7QsJSkOhDszZprtY= 2024-02-01T10:18:11.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:CTVz9ejk+2+7QsJSkOhDszZprtY=', 'Content-Length': '0'} 2024-02-01T10:18:11.682 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f09862340650c7f4-0065bb6fe3-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:11.683 DEBUG:boto:b'vclfcn-12&1000false&2024-02-01T10:16:36.721Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:11.683 DEBUG:boto:path=/vclfcn-12/ 2024-02-01T10:18:11.683 DEBUG:boto:auth_path=/vclfcn-12/ 2024-02-01T10:18:11.683 DEBUG:boto:path=/vclfcn-12/?prefix=%26 2024-02-01T10:18:11.684 DEBUG:boto:auth_path=/vclfcn-12/?prefix=%26 2024-02-01T10:18:11.684 DEBUG:boto:Method: GET 2024-02-01T10:18:11.684 DEBUG:boto:Path: /vclfcn-12/?prefix=%26 2024-02-01T10:18:11.684 DEBUG:boto:Data: 2024-02-01T10:18:11.684 DEBUG:boto:Headers: {} 2024-02-01T10:18:11.684 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:11.684 DEBUG:boto:Port: 8001 2024-02-01T10:18:11.684 DEBUG:boto:Params: {} 2024-02-01T10:18:11.684 DEBUG:boto:Token: None 2024-02-01T10:18:11.684 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:11 GMT /vclfcn-12/ 2024-02-01T10:18:11.685 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:CTVz9ejk+2+7QsJSkOhDszZprtY= 2024-02-01T10:18:11.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:CTVz9ejk+2+7QsJSkOhDszZprtY=', 'Content-Length': '0'} 2024-02-01T10:18:11.688 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b25c338d3d0b138b-0065bb6fe3-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:11.689 DEBUG:boto:b'vclfcn-12&1000false&2024-02-01T10:16:36.721Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:11.689 DEBUG:boto:path=/vclfcn-12/%26 2024-02-01T10:18:11.689 DEBUG:boto:auth_path=/vclfcn-12/%26 2024-02-01T10:18:11.689 DEBUG:boto:Method: HEAD 2024-02-01T10:18:11.689 DEBUG:boto:Path: /vclfcn-12/%26 2024-02-01T10:18:11.689 DEBUG:boto:Data: 2024-02-01T10:18:11.689 DEBUG:boto:Headers: {} 2024-02-01T10:18:11.690 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:11.690 DEBUG:boto:Port: 8000 2024-02-01T10:18:11.690 DEBUG:boto:Params: {} 2024-02-01T10:18:11.690 DEBUG:boto:Token: None 2024-02-01T10:18:11.690 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:11 GMT /vclfcn-12/%26 2024-02-01T10:18:11.690 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Rky277maYj6wkfk4VGFVu3Gsles= 2024-02-01T10:18:11.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Rky277maYj6wkfk4VGFVu3Gsles=', 'Content-Length': '0'} 2024-02-01T10:18:11.693 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e9c1a8662b2aa164-0065bb6fe3-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:11.693 DEBUG:boto:path=/vclfcn-12/%26 2024-02-01T10:18:11.693 DEBUG:boto:auth_path=/vclfcn-12/%26 2024-02-01T10:18:11.693 DEBUG:boto:Method: HEAD 2024-02-01T10:18:11.693 DEBUG:boto:Path: /vclfcn-12/%26 2024-02-01T10:18:11.693 DEBUG:boto:Data: 2024-02-01T10:18:11.694 DEBUG:boto:Headers: {} 2024-02-01T10:18:11.694 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:11.694 DEBUG:boto:Port: 8001 2024-02-01T10:18:11.694 DEBUG:boto:Params: {} 2024-02-01T10:18:11.694 DEBUG:boto:Token: None 2024-02-01T10:18:11.694 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:11 GMT /vclfcn-12/%26 2024-02-01T10:18:11.694 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Rky277maYj6wkfk4VGFVu3Gsles= 2024-02-01T10:18:11.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Rky277maYj6wkfk4VGFVu3Gsles=', 'Content-Length': '0'} 2024-02-01T10:18:11.697 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fdce62a57a270ad0-0065bb6fe3-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:11.697 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-02-01T10:18:11.697 DEBUG:boto:path=/vclfcn-12/%26 2024-02-01T10:18:11.697 DEBUG:boto:auth_path=/vclfcn-12/%26 2024-02-01T10:18:11.697 DEBUG:boto:Method: GET 2024-02-01T10:18:11.698 DEBUG:boto:Path: /vclfcn-12/%26 2024-02-01T10:18:11.698 DEBUG:boto:Data: 2024-02-01T10:18:11.698 DEBUG:boto:Headers: {} 2024-02-01T10:18:11.698 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:11.698 DEBUG:boto:Port: 8000 2024-02-01T10:18:11.698 DEBUG:boto:Params: {} 2024-02-01T10:18:11.698 DEBUG:boto:Token: None 2024-02-01T10:18:11.698 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:11 GMT /vclfcn-12/%26 2024-02-01T10:18:11.698 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:R+BuMn1g8Bar/PhMZTehHIylfeE= 2024-02-01T10:18:11.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:R+BuMn1g8Bar/PhMZTehHIylfeE=', 'Content-Length': '0'} 2024-02-01T10:18:11.701 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a52ad0a327266eb9-0065bb6fe3-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:11.748 DEBUG:boto:path=/vclfcn-12/%26 2024-02-01T10:18:11.748 DEBUG:boto:auth_path=/vclfcn-12/%26 2024-02-01T10:18:11.748 DEBUG:boto:Method: GET 2024-02-01T10:18:11.748 DEBUG:boto:Path: /vclfcn-12/%26 2024-02-01T10:18:11.749 DEBUG:boto:Data: 2024-02-01T10:18:11.749 DEBUG:boto:Headers: {} 2024-02-01T10:18:11.749 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:11.749 DEBUG:boto:Port: 8001 2024-02-01T10:18:11.749 DEBUG:boto:Params: {} 2024-02-01T10:18:11.749 DEBUG:boto:Token: None 2024-02-01T10:18:11.749 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:11 GMT /vclfcn-12/%26 2024-02-01T10:18:11.749 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:R+BuMn1g8Bar/PhMZTehHIylfeE= 2024-02-01T10:18:11.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:R+BuMn1g8Bar/PhMZTehHIylfeE=', 'Content-Length': '0'} 2024-02-01T10:18:11.753 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b88c45ab2035b090-0065bb6fe3-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:11.753 DEBUG:boto:path=/vclfcn-12/ 2024-02-01T10:18:11.753 DEBUG:boto:auth_path=/vclfcn-12/ 2024-02-01T10:18:11.753 DEBUG:boto:path=/vclfcn-12/?prefix=%26 2024-02-01T10:18:11.753 DEBUG:boto:auth_path=/vclfcn-12/?prefix=%26 2024-02-01T10:18:11.753 DEBUG:boto:Method: GET 2024-02-01T10:18:11.754 DEBUG:boto:Path: /vclfcn-12/?prefix=%26 2024-02-01T10:18:11.754 DEBUG:boto:Data: 2024-02-01T10:18:11.754 DEBUG:boto:Headers: {} 2024-02-01T10:18:11.754 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:11.754 DEBUG:boto:Port: 8000 2024-02-01T10:18:11.754 DEBUG:boto:Params: {} 2024-02-01T10:18:11.754 DEBUG:boto:Token: None 2024-02-01T10:18:11.754 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:11 GMT /vclfcn-12/ 2024-02-01T10:18:11.754 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:CTVz9ejk+2+7QsJSkOhDszZprtY= 2024-02-01T10:18:11.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:CTVz9ejk+2+7QsJSkOhDszZprtY=', 'Content-Length': '0'} 2024-02-01T10:18:11.759 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086a851030e1e4988-0065bb6fe3-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:11.760 DEBUG:boto:b'vclfcn-12&1000false&2024-02-01T10:16:36.721Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:11.760 DEBUG:boto:path=/vclfcn-12/ 2024-02-01T10:18:11.760 DEBUG:boto:auth_path=/vclfcn-12/ 2024-02-01T10:18:11.760 DEBUG:boto:path=/vclfcn-12/?prefix=%26 2024-02-01T10:18:11.760 DEBUG:boto:auth_path=/vclfcn-12/?prefix=%26 2024-02-01T10:18:11.761 DEBUG:boto:Method: GET 2024-02-01T10:18:11.761 DEBUG:boto:Path: /vclfcn-12/?prefix=%26 2024-02-01T10:18:11.761 DEBUG:boto:Data: 2024-02-01T10:18:11.761 DEBUG:boto:Headers: {} 2024-02-01T10:18:11.761 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:11.761 DEBUG:boto:Port: 8001 2024-02-01T10:18:11.761 DEBUG:boto:Params: {} 2024-02-01T10:18:11.761 DEBUG:boto:Token: None 2024-02-01T10:18:11.762 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:11 GMT /vclfcn-12/ 2024-02-01T10:18:11.762 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:CTVz9ejk+2+7QsJSkOhDszZprtY= 2024-02-01T10:18:11.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:CTVz9ejk+2+7QsJSkOhDszZprtY=', 'Content-Length': '0'} 2024-02-01T10:18:11.766 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c614872dc331f2f-0065bb6fe3-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:11.766 DEBUG:boto:b'vclfcn-12&1000false&2024-02-01T10:16:36.721Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:11.767 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-02-01T10:18:11.767 DEBUG:boto:path=/vclfcn-12/%3A 2024-02-01T10:18:11.767 DEBUG:boto:auth_path=/vclfcn-12/%3A 2024-02-01T10:18:11.767 DEBUG:boto:path=/vclfcn-12/%3A?versionId=null 2024-02-01T10:18:11.767 DEBUG:boto:auth_path=/vclfcn-12/%3A?versionId=null 2024-02-01T10:18:11.767 DEBUG:boto:Method: GET 2024-02-01T10:18:11.768 DEBUG:boto:Path: /vclfcn-12/%3A?versionId=null 2024-02-01T10:18:11.768 DEBUG:boto:Data: 2024-02-01T10:18:11.768 DEBUG:boto:Headers: {} 2024-02-01T10:18:11.768 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:11.768 DEBUG:boto:Port: 8000 2024-02-01T10:18:11.768 DEBUG:boto:Params: {} 2024-02-01T10:18:11.768 DEBUG:boto:Token: None 2024-02-01T10:18:11.768 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:11 GMT /vclfcn-12/%3A?versionId=null 2024-02-01T10:18:11.768 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:T875Utl7/zHkUw4dUefq3NygV/s= 2024-02-01T10:18:11.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:T875Utl7/zHkUw4dUefq3NygV/s=', 'Content-Length': '0'} 2024-02-01T10:18:11.771 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c4154b9a13db0222-0065bb6fe3-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:11.812 DEBUG:boto:path=/vclfcn-12/%3A 2024-02-01T10:18:11.812 DEBUG:boto:auth_path=/vclfcn-12/%3A 2024-02-01T10:18:11.812 DEBUG:boto:path=/vclfcn-12/%3A?versionId=null 2024-02-01T10:18:11.813 DEBUG:boto:auth_path=/vclfcn-12/%3A?versionId=null 2024-02-01T10:18:11.813 DEBUG:boto:Method: GET 2024-02-01T10:18:11.813 DEBUG:boto:Path: /vclfcn-12/%3A?versionId=null 2024-02-01T10:18:11.813 DEBUG:boto:Data: 2024-02-01T10:18:11.813 DEBUG:boto:Headers: {} 2024-02-01T10:18:11.813 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:11.813 DEBUG:boto:Port: 8001 2024-02-01T10:18:11.813 DEBUG:boto:Params: {} 2024-02-01T10:18:11.813 DEBUG:boto:Token: None 2024-02-01T10:18:11.814 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:11 GMT /vclfcn-12/%3A?versionId=null 2024-02-01T10:18:11.814 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:T875Utl7/zHkUw4dUefq3NygV/s= 2024-02-01T10:18:11.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:T875Utl7/zHkUw4dUefq3NygV/s=', 'Content-Length': '0'} 2024-02-01T10:18:11.817 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009386d33da313eb24-0065bb6fe3-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:11.817 DEBUG:boto:path=/vclfcn-12/%3A 2024-02-01T10:18:11.817 DEBUG:boto:auth_path=/vclfcn-12/%3A 2024-02-01T10:18:11.817 DEBUG:boto:path=/vclfcn-12/%3A?versionId=null 2024-02-01T10:18:11.817 DEBUG:boto:auth_path=/vclfcn-12/%3A?versionId=null 2024-02-01T10:18:11.818 DEBUG:boto:Method: HEAD 2024-02-01T10:18:11.818 DEBUG:boto:Path: /vclfcn-12/%3A?versionId=null 2024-02-01T10:18:11.818 DEBUG:boto:Data: 2024-02-01T10:18:11.818 DEBUG:boto:Headers: {} 2024-02-01T10:18:11.818 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:11.818 DEBUG:boto:Port: 8000 2024-02-01T10:18:11.818 DEBUG:boto:Params: {} 2024-02-01T10:18:11.818 DEBUG:boto:Token: None 2024-02-01T10:18:11.818 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:11 GMT /vclfcn-12/%3A?versionId=null 2024-02-01T10:18:11.818 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:eoXrFMOnbMX6BLdIIKIrYuzZxwg= 2024-02-01T10:18:11.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:eoXrFMOnbMX6BLdIIKIrYuzZxwg=', 'Content-Length': '0'} 2024-02-01T10:18:11.821 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000013bc9773b604796d-0065bb6fe3-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:11.821 DEBUG:boto:path=/vclfcn-12/%3A 2024-02-01T10:18:11.821 DEBUG:boto:auth_path=/vclfcn-12/%3A 2024-02-01T10:18:11.822 DEBUG:boto:path=/vclfcn-12/%3A?versionId=null 2024-02-01T10:18:11.822 DEBUG:boto:auth_path=/vclfcn-12/%3A?versionId=null 2024-02-01T10:18:11.822 DEBUG:boto:Method: HEAD 2024-02-01T10:18:11.822 DEBUG:boto:Path: /vclfcn-12/%3A?versionId=null 2024-02-01T10:18:11.822 DEBUG:boto:Data: 2024-02-01T10:18:11.822 DEBUG:boto:Headers: {} 2024-02-01T10:18:11.822 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:11.822 DEBUG:boto:Port: 8001 2024-02-01T10:18:11.822 DEBUG:boto:Params: {} 2024-02-01T10:18:11.822 DEBUG:boto:Token: None 2024-02-01T10:18:11.822 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:11 GMT /vclfcn-12/%3A?versionId=null 2024-02-01T10:18:11.823 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:eoXrFMOnbMX6BLdIIKIrYuzZxwg= 2024-02-01T10:18:11.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:eoXrFMOnbMX6BLdIIKIrYuzZxwg=', 'Content-Length': '0'} 2024-02-01T10:18:11.825 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002b31f6e70dca4360-0065bb6fe3-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:11.826 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-02-01T10:18:11.826 DEBUG:boto:path=/vclfcn-12/%3A 2024-02-01T10:18:11.826 DEBUG:boto:auth_path=/vclfcn-12/%3A 2024-02-01T10:18:11.826 DEBUG:boto:path=/vclfcn-12/%3A?versionId=null 2024-02-01T10:18:11.826 DEBUG:boto:auth_path=/vclfcn-12/%3A?versionId=null 2024-02-01T10:18:11.826 DEBUG:boto:Method: GET 2024-02-01T10:18:11.826 DEBUG:boto:Path: /vclfcn-12/%3A?versionId=null 2024-02-01T10:18:11.826 DEBUG:boto:Data: 2024-02-01T10:18:11.826 DEBUG:boto:Headers: {} 2024-02-01T10:18:11.826 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:11.827 DEBUG:boto:Port: 8000 2024-02-01T10:18:11.827 DEBUG:boto:Params: {} 2024-02-01T10:18:11.827 DEBUG:boto:Token: None 2024-02-01T10:18:11.827 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:11 GMT /vclfcn-12/%3A?versionId=null 2024-02-01T10:18:11.827 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:T875Utl7/zHkUw4dUefq3NygV/s= 2024-02-01T10:18:11.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:T875Utl7/zHkUw4dUefq3NygV/s=', 'Content-Length': '0'} 2024-02-01T10:18:11.830 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e29e9d88ac7a1d8b-0065bb6fe3-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:11.872 DEBUG:boto:path=/vclfcn-12/%3A 2024-02-01T10:18:11.872 DEBUG:boto:auth_path=/vclfcn-12/%3A 2024-02-01T10:18:11.872 DEBUG:boto:path=/vclfcn-12/%3A?versionId=null 2024-02-01T10:18:11.873 DEBUG:boto:auth_path=/vclfcn-12/%3A?versionId=null 2024-02-01T10:18:11.873 DEBUG:boto:Method: GET 2024-02-01T10:18:11.873 DEBUG:boto:Path: /vclfcn-12/%3A?versionId=null 2024-02-01T10:18:11.873 DEBUG:boto:Data: 2024-02-01T10:18:11.873 DEBUG:boto:Headers: {} 2024-02-01T10:18:11.873 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:11.873 DEBUG:boto:Port: 8001 2024-02-01T10:18:11.873 DEBUG:boto:Params: {} 2024-02-01T10:18:11.873 DEBUG:boto:Token: None 2024-02-01T10:18:11.873 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:11 GMT /vclfcn-12/%3A?versionId=null 2024-02-01T10:18:11.874 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:T875Utl7/zHkUw4dUefq3NygV/s= 2024-02-01T10:18:11.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:T875Utl7/zHkUw4dUefq3NygV/s=', 'Content-Length': '0'} 2024-02-01T10:18:11.876 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000014720fb3c701fca7-0065bb6fe3-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:11.877 DEBUG:boto:path=/vclfcn-12/ 2024-02-01T10:18:11.877 DEBUG:boto:auth_path=/vclfcn-12/ 2024-02-01T10:18:11.877 DEBUG:boto:path=/vclfcn-12/?prefix=%3A 2024-02-01T10:18:11.877 DEBUG:boto:auth_path=/vclfcn-12/?prefix=%3A 2024-02-01T10:18:11.877 DEBUG:boto:Method: GET 2024-02-01T10:18:11.877 DEBUG:boto:Path: /vclfcn-12/?prefix=%3A 2024-02-01T10:18:11.877 DEBUG:boto:Data: 2024-02-01T10:18:11.878 DEBUG:boto:Headers: {} 2024-02-01T10:18:11.878 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:11.878 DEBUG:boto:Port: 8000 2024-02-01T10:18:11.878 DEBUG:boto:Params: {} 2024-02-01T10:18:11.878 DEBUG:boto:Token: None 2024-02-01T10:18:11.878 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:11 GMT /vclfcn-12/ 2024-02-01T10:18:11.878 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:CTVz9ejk+2+7QsJSkOhDszZprtY= 2024-02-01T10:18:11.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:CTVz9ejk+2+7QsJSkOhDszZprtY=', 'Content-Length': '0'} 2024-02-01T10:18:11.882 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed650a99d3a5a428-0065bb6fe3-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:11.882 DEBUG:boto:b'vclfcn-12:1000false:2024-02-01T10:16:36.670Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:11.883 DEBUG:boto:path=/vclfcn-12/ 2024-02-01T10:18:11.883 DEBUG:boto:auth_path=/vclfcn-12/ 2024-02-01T10:18:11.883 DEBUG:boto:path=/vclfcn-12/?prefix=%3A 2024-02-01T10:18:11.883 DEBUG:boto:auth_path=/vclfcn-12/?prefix=%3A 2024-02-01T10:18:11.883 DEBUG:boto:Method: GET 2024-02-01T10:18:11.883 DEBUG:boto:Path: /vclfcn-12/?prefix=%3A 2024-02-01T10:18:11.883 DEBUG:boto:Data: 2024-02-01T10:18:11.884 DEBUG:boto:Headers: {} 2024-02-01T10:18:11.884 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:11.884 DEBUG:boto:Port: 8001 2024-02-01T10:18:11.884 DEBUG:boto:Params: {} 2024-02-01T10:18:11.884 DEBUG:boto:Token: None 2024-02-01T10:18:11.884 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:11 GMT /vclfcn-12/ 2024-02-01T10:18:11.884 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:CTVz9ejk+2+7QsJSkOhDszZprtY= 2024-02-01T10:18:11.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:CTVz9ejk+2+7QsJSkOhDszZprtY=', 'Content-Length': '0'} 2024-02-01T10:18:11.888 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b30e2256fdab3700-0065bb6fe3-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:11.932 DEBUG:boto:b'vclfcn-12:1000false:2024-02-01T10:16:36.670Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:11.933 DEBUG:boto:path=/vclfcn-12/%3A 2024-02-01T10:18:11.933 DEBUG:boto:auth_path=/vclfcn-12/%3A 2024-02-01T10:18:11.933 DEBUG:boto:Method: HEAD 2024-02-01T10:18:11.933 DEBUG:boto:Path: /vclfcn-12/%3A 2024-02-01T10:18:11.933 DEBUG:boto:Data: 2024-02-01T10:18:11.933 DEBUG:boto:Headers: {} 2024-02-01T10:18:11.933 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:11.933 DEBUG:boto:Port: 8000 2024-02-01T10:18:11.933 DEBUG:boto:Params: {} 2024-02-01T10:18:11.934 DEBUG:boto:Token: None 2024-02-01T10:18:11.934 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:11 GMT /vclfcn-12/%3A 2024-02-01T10:18:11.934 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+imIOBWjB3MuEBJGWcqph07fQyY= 2024-02-01T10:18:11.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+imIOBWjB3MuEBJGWcqph07fQyY=', 'Content-Length': '0'} 2024-02-01T10:18:11.937 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a9dfc2b49b53611c-0065bb6fe3-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:11.937 DEBUG:boto:path=/vclfcn-12/%3A 2024-02-01T10:18:11.937 DEBUG:boto:auth_path=/vclfcn-12/%3A 2024-02-01T10:18:11.937 DEBUG:boto:Method: HEAD 2024-02-01T10:18:11.937 DEBUG:boto:Path: /vclfcn-12/%3A 2024-02-01T10:18:11.937 DEBUG:boto:Data: 2024-02-01T10:18:11.937 DEBUG:boto:Headers: {} 2024-02-01T10:18:11.938 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:11.938 DEBUG:boto:Port: 8001 2024-02-01T10:18:11.938 DEBUG:boto:Params: {} 2024-02-01T10:18:11.938 DEBUG:boto:Token: None 2024-02-01T10:18:11.938 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:11 GMT /vclfcn-12/%3A 2024-02-01T10:18:11.938 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+imIOBWjB3MuEBJGWcqph07fQyY= 2024-02-01T10:18:11.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+imIOBWjB3MuEBJGWcqph07fQyY=', 'Content-Length': '0'} 2024-02-01T10:18:11.941 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006359524fecf227a0-0065bb6fe3-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:11.941 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-02-01T10:18:11.941 DEBUG:boto:path=/vclfcn-12/%3A 2024-02-01T10:18:11.941 DEBUG:boto:auth_path=/vclfcn-12/%3A 2024-02-01T10:18:11.941 DEBUG:boto:Method: GET 2024-02-01T10:18:11.941 DEBUG:boto:Path: /vclfcn-12/%3A 2024-02-01T10:18:11.941 DEBUG:boto:Data: 2024-02-01T10:18:11.942 DEBUG:boto:Headers: {} 2024-02-01T10:18:11.942 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:11.942 DEBUG:boto:Port: 8000 2024-02-01T10:18:11.942 DEBUG:boto:Params: {} 2024-02-01T10:18:11.942 DEBUG:boto:Token: None 2024-02-01T10:18:11.942 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:11 GMT /vclfcn-12/%3A 2024-02-01T10:18:11.942 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8MtDCTOF+fRC/qVDwPMZ5rAZ3dI= 2024-02-01T10:18:11.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8MtDCTOF+fRC/qVDwPMZ5rAZ3dI=', 'Content-Length': '0'} 2024-02-01T10:18:11.945 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003dc988c87d40e621-0065bb6fe3-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:11.988 DEBUG:boto:path=/vclfcn-12/%3A 2024-02-01T10:18:11.991 DEBUG:boto:auth_path=/vclfcn-12/%3A 2024-02-01T10:18:11.991 DEBUG:boto:Method: GET 2024-02-01T10:18:11.991 DEBUG:boto:Path: /vclfcn-12/%3A 2024-02-01T10:18:11.991 DEBUG:boto:Data: 2024-02-01T10:18:11.991 DEBUG:boto:Headers: {} 2024-02-01T10:18:11.991 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:11.991 DEBUG:boto:Port: 8001 2024-02-01T10:18:11.991 DEBUG:boto:Params: {} 2024-02-01T10:18:11.992 DEBUG:boto:Token: None 2024-02-01T10:18:11.992 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:11 GMT /vclfcn-12/%3A 2024-02-01T10:18:11.992 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8MtDCTOF+fRC/qVDwPMZ5rAZ3dI= 2024-02-01T10:18:11.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8MtDCTOF+fRC/qVDwPMZ5rAZ3dI=', 'Content-Length': '0'} 2024-02-01T10:18:11.995 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a1a7f79fae46e59c-0065bb6fe3-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:11.996 DEBUG:boto:path=/vclfcn-12/ 2024-02-01T10:18:11.996 DEBUG:boto:auth_path=/vclfcn-12/ 2024-02-01T10:18:11.996 DEBUG:boto:path=/vclfcn-12/?prefix=%3A 2024-02-01T10:18:11.996 DEBUG:boto:auth_path=/vclfcn-12/?prefix=%3A 2024-02-01T10:18:11.996 DEBUG:boto:Method: GET 2024-02-01T10:18:11.996 DEBUG:boto:Path: /vclfcn-12/?prefix=%3A 2024-02-01T10:18:11.997 DEBUG:boto:Data: 2024-02-01T10:18:11.997 DEBUG:boto:Headers: {} 2024-02-01T10:18:11.997 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:11.997 DEBUG:boto:Port: 8000 2024-02-01T10:18:11.997 DEBUG:boto:Params: {} 2024-02-01T10:18:11.997 DEBUG:boto:Token: None 2024-02-01T10:18:11.997 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:11 GMT /vclfcn-12/ 2024-02-01T10:18:11.997 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:CTVz9ejk+2+7QsJSkOhDszZprtY= 2024-02-01T10:18:11.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:CTVz9ejk+2+7QsJSkOhDszZprtY=', 'Content-Length': '0'} 2024-02-01T10:18:12.002 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac3ee8d37b564581-0065bb6fe3-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:12.044 DEBUG:boto:b'vclfcn-12:1000false:2024-02-01T10:16:36.670Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:12.045 DEBUG:boto:path=/vclfcn-12/ 2024-02-01T10:18:12.045 DEBUG:boto:auth_path=/vclfcn-12/ 2024-02-01T10:18:12.045 DEBUG:boto:path=/vclfcn-12/?prefix=%3A 2024-02-01T10:18:12.045 DEBUG:boto:auth_path=/vclfcn-12/?prefix=%3A 2024-02-01T10:18:12.045 DEBUG:boto:Method: GET 2024-02-01T10:18:12.045 DEBUG:boto:Path: /vclfcn-12/?prefix=%3A 2024-02-01T10:18:12.046 DEBUG:boto:Data: 2024-02-01T10:18:12.046 DEBUG:boto:Headers: {} 2024-02-01T10:18:12.046 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:12.046 DEBUG:boto:Port: 8001 2024-02-01T10:18:12.046 DEBUG:boto:Params: {} 2024-02-01T10:18:12.046 DEBUG:boto:Token: None 2024-02-01T10:18:12.046 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:12 GMT /vclfcn-12/ 2024-02-01T10:18:12.046 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HjHdiPPd6aPwDPCSTh+P8ZN/iyI= 2024-02-01T10:18:12.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HjHdiPPd6aPwDPCSTh+P8ZN/iyI=', 'Content-Length': '0'} 2024-02-01T10:18:12.050 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000709dc1efc7d99b22-0065bb6fe4-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:12.051 DEBUG:boto:b'vclfcn-12:1000false:2024-02-01T10:16:36.670Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:12.051 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-02-01T10:18:12.051 DEBUG:boto:path=/vclfcn-12/_myobj 2024-02-01T10:18:12.051 DEBUG:boto:auth_path=/vclfcn-12/_myobj 2024-02-01T10:18:12.051 DEBUG:boto:path=/vclfcn-12/_myobj?versionId=null 2024-02-01T10:18:12.052 DEBUG:boto:auth_path=/vclfcn-12/_myobj?versionId=null 2024-02-01T10:18:12.052 DEBUG:boto:Method: GET 2024-02-01T10:18:12.052 DEBUG:boto:Path: /vclfcn-12/_myobj?versionId=null 2024-02-01T10:18:12.052 DEBUG:boto:Data: 2024-02-01T10:18:12.052 DEBUG:boto:Headers: {} 2024-02-01T10:18:12.052 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:12.052 DEBUG:boto:Port: 8000 2024-02-01T10:18:12.052 DEBUG:boto:Params: {} 2024-02-01T10:18:12.053 DEBUG:boto:Token: None 2024-02-01T10:18:12.053 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:12 GMT /vclfcn-12/_myobj?versionId=null 2024-02-01T10:18:12.053 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cjmBB855mOl5vU25QYR8F0YHqrw= 2024-02-01T10:18:12.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cjmBB855mOl5vU25QYR8F0YHqrw=', 'Content-Length': '0'} 2024-02-01T10:18:12.056 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004a4683030d6dbaa6-0065bb6fe4-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:12.056 DEBUG:boto:path=/vclfcn-12/_myobj 2024-02-01T10:18:12.057 DEBUG:boto:auth_path=/vclfcn-12/_myobj 2024-02-01T10:18:12.057 DEBUG:boto:path=/vclfcn-12/_myobj?versionId=null 2024-02-01T10:18:12.057 DEBUG:boto:auth_path=/vclfcn-12/_myobj?versionId=null 2024-02-01T10:18:12.057 DEBUG:boto:Method: GET 2024-02-01T10:18:12.057 DEBUG:boto:Path: /vclfcn-12/_myobj?versionId=null 2024-02-01T10:18:12.057 DEBUG:boto:Data: 2024-02-01T10:18:12.057 DEBUG:boto:Headers: {} 2024-02-01T10:18:12.057 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:12.057 DEBUG:boto:Port: 8001 2024-02-01T10:18:12.057 DEBUG:boto:Params: {} 2024-02-01T10:18:12.058 DEBUG:boto:Token: None 2024-02-01T10:18:12.058 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:12 GMT /vclfcn-12/_myobj?versionId=null 2024-02-01T10:18:12.058 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cjmBB855mOl5vU25QYR8F0YHqrw= 2024-02-01T10:18:12.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cjmBB855mOl5vU25QYR8F0YHqrw=', 'Content-Length': '0'} 2024-02-01T10:18:12.061 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000edf8c2bb15917fa-0065bb6fe4-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:12.061 DEBUG:boto:path=/vclfcn-12/_myobj 2024-02-01T10:18:12.061 DEBUG:boto:auth_path=/vclfcn-12/_myobj 2024-02-01T10:18:12.061 DEBUG:boto:path=/vclfcn-12/_myobj?versionId=null 2024-02-01T10:18:12.061 DEBUG:boto:auth_path=/vclfcn-12/_myobj?versionId=null 2024-02-01T10:18:12.062 DEBUG:boto:Method: HEAD 2024-02-01T10:18:12.062 DEBUG:boto:Path: /vclfcn-12/_myobj?versionId=null 2024-02-01T10:18:12.062 DEBUG:boto:Data: 2024-02-01T10:18:12.062 DEBUG:boto:Headers: {} 2024-02-01T10:18:12.062 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:12.062 DEBUG:boto:Port: 8000 2024-02-01T10:18:12.062 DEBUG:boto:Params: {} 2024-02-01T10:18:12.062 DEBUG:boto:Token: None 2024-02-01T10:18:12.062 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:12 GMT /vclfcn-12/_myobj?versionId=null 2024-02-01T10:18:12.063 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1pixGaAXjruAX65Dz1xLCDB6dVc= 2024-02-01T10:18:12.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1pixGaAXjruAX65Dz1xLCDB6dVc=', 'Content-Length': '0'} 2024-02-01T10:18:12.065 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000035ff20bba9806507-0065bb6fe4-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:12.066 DEBUG:boto:path=/vclfcn-12/_myobj 2024-02-01T10:18:12.066 DEBUG:boto:auth_path=/vclfcn-12/_myobj 2024-02-01T10:18:12.066 DEBUG:boto:path=/vclfcn-12/_myobj?versionId=null 2024-02-01T10:18:12.066 DEBUG:boto:auth_path=/vclfcn-12/_myobj?versionId=null 2024-02-01T10:18:12.066 DEBUG:boto:Method: HEAD 2024-02-01T10:18:12.066 DEBUG:boto:Path: /vclfcn-12/_myobj?versionId=null 2024-02-01T10:18:12.066 DEBUG:boto:Data: 2024-02-01T10:18:12.066 DEBUG:boto:Headers: {} 2024-02-01T10:18:12.067 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:12.067 DEBUG:boto:Port: 8001 2024-02-01T10:18:12.067 DEBUG:boto:Params: {} 2024-02-01T10:18:12.067 DEBUG:boto:Token: None 2024-02-01T10:18:12.067 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:12 GMT /vclfcn-12/_myobj?versionId=null 2024-02-01T10:18:12.067 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1pixGaAXjruAX65Dz1xLCDB6dVc= 2024-02-01T10:18:12.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1pixGaAXjruAX65Dz1xLCDB6dVc=', 'Content-Length': '0'} 2024-02-01T10:18:12.070 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005b30db78dc6fc01c-0065bb6fe4-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:12.070 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-02-01T10:18:12.071 DEBUG:boto:path=/vclfcn-12/_myobj 2024-02-01T10:18:12.071 DEBUG:boto:auth_path=/vclfcn-12/_myobj 2024-02-01T10:18:12.071 DEBUG:boto:path=/vclfcn-12/_myobj?versionId=null 2024-02-01T10:18:12.071 DEBUG:boto:auth_path=/vclfcn-12/_myobj?versionId=null 2024-02-01T10:18:12.071 DEBUG:boto:Method: GET 2024-02-01T10:18:12.071 DEBUG:boto:Path: /vclfcn-12/_myobj?versionId=null 2024-02-01T10:18:12.071 DEBUG:boto:Data: 2024-02-01T10:18:12.071 DEBUG:boto:Headers: {} 2024-02-01T10:18:12.071 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:12.071 DEBUG:boto:Port: 8000 2024-02-01T10:18:12.071 DEBUG:boto:Params: {} 2024-02-01T10:18:12.072 DEBUG:boto:Token: None 2024-02-01T10:18:12.072 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:12 GMT /vclfcn-12/_myobj?versionId=null 2024-02-01T10:18:12.072 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cjmBB855mOl5vU25QYR8F0YHqrw= 2024-02-01T10:18:12.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cjmBB855mOl5vU25QYR8F0YHqrw=', 'Content-Length': '0'} 2024-02-01T10:18:12.074 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001121f02251b164b2-0065bb6fe4-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:12.116 DEBUG:boto:path=/vclfcn-12/_myobj 2024-02-01T10:18:12.116 DEBUG:boto:auth_path=/vclfcn-12/_myobj 2024-02-01T10:18:12.117 DEBUG:boto:path=/vclfcn-12/_myobj?versionId=null 2024-02-01T10:18:12.117 DEBUG:boto:auth_path=/vclfcn-12/_myobj?versionId=null 2024-02-01T10:18:12.117 DEBUG:boto:Method: GET 2024-02-01T10:18:12.117 DEBUG:boto:Path: /vclfcn-12/_myobj?versionId=null 2024-02-01T10:18:12.117 DEBUG:boto:Data: 2024-02-01T10:18:12.117 DEBUG:boto:Headers: {} 2024-02-01T10:18:12.117 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:12.118 DEBUG:boto:Port: 8001 2024-02-01T10:18:12.118 DEBUG:boto:Params: {} 2024-02-01T10:18:12.118 DEBUG:boto:Token: None 2024-02-01T10:18:12.118 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:12 GMT /vclfcn-12/_myobj?versionId=null 2024-02-01T10:18:12.118 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cjmBB855mOl5vU25QYR8F0YHqrw= 2024-02-01T10:18:12.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cjmBB855mOl5vU25QYR8F0YHqrw=', 'Content-Length': '0'} 2024-02-01T10:18:12.122 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c8a4510414f7d573-0065bb6fe4-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:12.122 DEBUG:boto:path=/vclfcn-12/ 2024-02-01T10:18:12.122 DEBUG:boto:auth_path=/vclfcn-12/ 2024-02-01T10:18:12.122 DEBUG:boto:path=/vclfcn-12/?prefix=_myobj 2024-02-01T10:18:12.122 DEBUG:boto:auth_path=/vclfcn-12/?prefix=_myobj 2024-02-01T10:18:12.122 DEBUG:boto:Method: GET 2024-02-01T10:18:12.123 DEBUG:boto:Path: /vclfcn-12/?prefix=_myobj 2024-02-01T10:18:12.123 DEBUG:boto:Data: 2024-02-01T10:18:12.123 DEBUG:boto:Headers: {} 2024-02-01T10:18:12.123 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:12.123 DEBUG:boto:Port: 8000 2024-02-01T10:18:12.123 DEBUG:boto:Params: {} 2024-02-01T10:18:12.123 DEBUG:boto:Token: None 2024-02-01T10:18:12.124 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:12 GMT /vclfcn-12/ 2024-02-01T10:18:12.124 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HjHdiPPd6aPwDPCSTh+P8ZN/iyI= 2024-02-01T10:18:12.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HjHdiPPd6aPwDPCSTh+P8ZN/iyI=', 'Content-Length': '0'} 2024-02-01T10:18:12.128 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000020940c562d1c3792-0065bb6fe4-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:12.129 DEBUG:boto:b'vclfcn-12_myobj1000false_myobj2024-02-01T10:16:36.640Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:12.129 DEBUG:boto:path=/vclfcn-12/ 2024-02-01T10:18:12.129 DEBUG:boto:auth_path=/vclfcn-12/ 2024-02-01T10:18:12.129 DEBUG:boto:path=/vclfcn-12/?prefix=_myobj 2024-02-01T10:18:12.129 DEBUG:boto:auth_path=/vclfcn-12/?prefix=_myobj 2024-02-01T10:18:12.130 DEBUG:boto:Method: GET 2024-02-01T10:18:12.130 DEBUG:boto:Path: /vclfcn-12/?prefix=_myobj 2024-02-01T10:18:12.130 DEBUG:boto:Data: 2024-02-01T10:18:12.130 DEBUG:boto:Headers: {} 2024-02-01T10:18:12.130 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:12.130 DEBUG:boto:Port: 8001 2024-02-01T10:18:12.130 DEBUG:boto:Params: {} 2024-02-01T10:18:12.130 DEBUG:boto:Token: None 2024-02-01T10:18:12.130 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:12 GMT /vclfcn-12/ 2024-02-01T10:18:12.130 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HjHdiPPd6aPwDPCSTh+P8ZN/iyI= 2024-02-01T10:18:12.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HjHdiPPd6aPwDPCSTh+P8ZN/iyI=', 'Content-Length': '0'} 2024-02-01T10:18:12.134 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e81e26540657d082-0065bb6fe4-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:12.135 DEBUG:boto:b'vclfcn-12_myobj1000false_myobj2024-02-01T10:16:36.640Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:12.135 DEBUG:boto:path=/vclfcn-12/_myobj 2024-02-01T10:18:12.135 DEBUG:boto:auth_path=/vclfcn-12/_myobj 2024-02-01T10:18:12.135 DEBUG:boto:Method: HEAD 2024-02-01T10:18:12.135 DEBUG:boto:Path: /vclfcn-12/_myobj 2024-02-01T10:18:12.135 DEBUG:boto:Data: 2024-02-01T10:18:12.135 DEBUG:boto:Headers: {} 2024-02-01T10:18:12.136 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:12.136 DEBUG:boto:Port: 8000 2024-02-01T10:18:12.136 DEBUG:boto:Params: {} 2024-02-01T10:18:12.136 DEBUG:boto:Token: None 2024-02-01T10:18:12.136 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:12 GMT /vclfcn-12/_myobj 2024-02-01T10:18:12.136 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ufJBP9F71eMOuuZ07FCdnD7hqBw= 2024-02-01T10:18:12.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ufJBP9F71eMOuuZ07FCdnD7hqBw=', 'Content-Length': '0'} 2024-02-01T10:18:12.139 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003faf340b9047f417-0065bb6fe4-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:12.140 DEBUG:boto:path=/vclfcn-12/_myobj 2024-02-01T10:18:12.140 DEBUG:boto:auth_path=/vclfcn-12/_myobj 2024-02-01T10:18:12.140 DEBUG:boto:Method: HEAD 2024-02-01T10:18:12.140 DEBUG:boto:Path: /vclfcn-12/_myobj 2024-02-01T10:18:12.140 DEBUG:boto:Data: 2024-02-01T10:18:12.140 DEBUG:boto:Headers: {} 2024-02-01T10:18:12.140 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:12.140 DEBUG:boto:Port: 8001 2024-02-01T10:18:12.140 DEBUG:boto:Params: {} 2024-02-01T10:18:12.140 DEBUG:boto:Token: None 2024-02-01T10:18:12.141 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:12 GMT /vclfcn-12/_myobj 2024-02-01T10:18:12.141 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ufJBP9F71eMOuuZ07FCdnD7hqBw= 2024-02-01T10:18:12.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ufJBP9F71eMOuuZ07FCdnD7hqBw=', 'Content-Length': '0'} 2024-02-01T10:18:12.144 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000011cdf09c87a7f6a6-0065bb6fe4-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:12.144 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-02-01T10:18:12.144 DEBUG:boto:path=/vclfcn-12/_myobj 2024-02-01T10:18:12.144 DEBUG:boto:auth_path=/vclfcn-12/_myobj 2024-02-01T10:18:12.144 DEBUG:boto:Method: GET 2024-02-01T10:18:12.144 DEBUG:boto:Path: /vclfcn-12/_myobj 2024-02-01T10:18:12.144 DEBUG:boto:Data: 2024-02-01T10:18:12.145 DEBUG:boto:Headers: {} 2024-02-01T10:18:12.145 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:12.145 DEBUG:boto:Port: 8000 2024-02-01T10:18:12.145 DEBUG:boto:Params: {} 2024-02-01T10:18:12.145 DEBUG:boto:Token: None 2024-02-01T10:18:12.145 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:12 GMT /vclfcn-12/_myobj 2024-02-01T10:18:12.145 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:SLtl/0Nb2f+hBI2eQmp2Kt4Oa1I= 2024-02-01T10:18:12.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:SLtl/0Nb2f+hBI2eQmp2Kt4Oa1I=', 'Content-Length': '0'} 2024-02-01T10:18:12.148 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007240c838ad308744-0065bb6fe4-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:12.192 DEBUG:boto:path=/vclfcn-12/_myobj 2024-02-01T10:18:12.192 DEBUG:boto:auth_path=/vclfcn-12/_myobj 2024-02-01T10:18:12.192 DEBUG:boto:Method: GET 2024-02-01T10:18:12.193 DEBUG:boto:Path: /vclfcn-12/_myobj 2024-02-01T10:18:12.193 DEBUG:boto:Data: 2024-02-01T10:18:12.193 DEBUG:boto:Headers: {} 2024-02-01T10:18:12.193 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:12.193 DEBUG:boto:Port: 8001 2024-02-01T10:18:12.193 DEBUG:boto:Params: {} 2024-02-01T10:18:12.193 DEBUG:boto:Token: None 2024-02-01T10:18:12.193 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:12 GMT /vclfcn-12/_myobj 2024-02-01T10:18:12.193 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:SLtl/0Nb2f+hBI2eQmp2Kt4Oa1I= 2024-02-01T10:18:12.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:SLtl/0Nb2f+hBI2eQmp2Kt4Oa1I=', 'Content-Length': '0'} 2024-02-01T10:18:12.197 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000000d26193e4c0fed-0065bb6fe4-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:12.197 DEBUG:boto:path=/vclfcn-12/ 2024-02-01T10:18:12.197 DEBUG:boto:auth_path=/vclfcn-12/ 2024-02-01T10:18:12.197 DEBUG:boto:path=/vclfcn-12/?prefix=_myobj 2024-02-01T10:18:12.197 DEBUG:boto:auth_path=/vclfcn-12/?prefix=_myobj 2024-02-01T10:18:12.197 DEBUG:boto:Method: GET 2024-02-01T10:18:12.197 DEBUG:boto:Path: /vclfcn-12/?prefix=_myobj 2024-02-01T10:18:12.197 DEBUG:boto:Data: 2024-02-01T10:18:12.198 DEBUG:boto:Headers: {} 2024-02-01T10:18:12.198 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:12.198 DEBUG:boto:Port: 8000 2024-02-01T10:18:12.198 DEBUG:boto:Params: {} 2024-02-01T10:18:12.198 DEBUG:boto:Token: None 2024-02-01T10:18:12.198 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:12 GMT /vclfcn-12/ 2024-02-01T10:18:12.198 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HjHdiPPd6aPwDPCSTh+P8ZN/iyI= 2024-02-01T10:18:12.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HjHdiPPd6aPwDPCSTh+P8ZN/iyI=', 'Content-Length': '0'} 2024-02-01T10:18:12.202 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9e9220969d16eb8-0065bb6fe4-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:12.244 DEBUG:boto:b'vclfcn-12_myobj1000false_myobj2024-02-01T10:16:36.640Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:12.245 DEBUG:boto:path=/vclfcn-12/ 2024-02-01T10:18:12.245 DEBUG:boto:auth_path=/vclfcn-12/ 2024-02-01T10:18:12.245 DEBUG:boto:path=/vclfcn-12/?prefix=_myobj 2024-02-01T10:18:12.245 DEBUG:boto:auth_path=/vclfcn-12/?prefix=_myobj 2024-02-01T10:18:12.245 DEBUG:boto:Method: GET 2024-02-01T10:18:12.245 DEBUG:boto:Path: /vclfcn-12/?prefix=_myobj 2024-02-01T10:18:12.245 DEBUG:boto:Data: 2024-02-01T10:18:12.245 DEBUG:boto:Headers: {} 2024-02-01T10:18:12.245 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:12.245 DEBUG:boto:Port: 8001 2024-02-01T10:18:12.246 DEBUG:boto:Params: {} 2024-02-01T10:18:12.246 DEBUG:boto:Token: None 2024-02-01T10:18:12.246 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:12 GMT /vclfcn-12/ 2024-02-01T10:18:12.246 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HjHdiPPd6aPwDPCSTh+P8ZN/iyI= 2024-02-01T10:18:12.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HjHdiPPd6aPwDPCSTh+P8ZN/iyI=', 'Content-Length': '0'} 2024-02-01T10:18:12.250 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000345de5972fa19d4e-0065bb6fe4-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:12.250 DEBUG:boto:b'vclfcn-12_myobj1000false_myobj2024-02-01T10:16:36.640Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:12.251 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-02-01T10:18:12.251 DEBUG:boto:path=/vclfcn-12/myobj 2024-02-01T10:18:12.251 DEBUG:boto:auth_path=/vclfcn-12/myobj 2024-02-01T10:18:12.251 DEBUG:boto:path=/vclfcn-12/myobj?versionId=null 2024-02-01T10:18:12.251 DEBUG:boto:auth_path=/vclfcn-12/myobj?versionId=null 2024-02-01T10:18:12.251 DEBUG:boto:Method: GET 2024-02-01T10:18:12.251 DEBUG:boto:Path: /vclfcn-12/myobj?versionId=null 2024-02-01T10:18:12.252 DEBUG:boto:Data: 2024-02-01T10:18:12.252 DEBUG:boto:Headers: {} 2024-02-01T10:18:12.252 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:12.252 DEBUG:boto:Port: 8000 2024-02-01T10:18:12.252 DEBUG:boto:Params: {} 2024-02-01T10:18:12.252 DEBUG:boto:Token: None 2024-02-01T10:18:12.252 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:12 GMT /vclfcn-12/myobj?versionId=null 2024-02-01T10:18:12.252 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Detm9cnn5epbJFYRyO3pNxcdPV4= 2024-02-01T10:18:12.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Detm9cnn5epbJFYRyO3pNxcdPV4=', 'Content-Length': '0'} 2024-02-01T10:18:12.255 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000052fa71caa4a0dfbe-0065bb6fe4-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:12.256 DEBUG:boto:path=/vclfcn-12/myobj 2024-02-01T10:18:12.256 DEBUG:boto:auth_path=/vclfcn-12/myobj 2024-02-01T10:18:12.256 DEBUG:boto:path=/vclfcn-12/myobj?versionId=null 2024-02-01T10:18:12.256 DEBUG:boto:auth_path=/vclfcn-12/myobj?versionId=null 2024-02-01T10:18:12.256 DEBUG:boto:Method: GET 2024-02-01T10:18:12.256 DEBUG:boto:Path: /vclfcn-12/myobj?versionId=null 2024-02-01T10:18:12.256 DEBUG:boto:Data: 2024-02-01T10:18:12.257 DEBUG:boto:Headers: {} 2024-02-01T10:18:12.257 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:12.257 DEBUG:boto:Port: 8001 2024-02-01T10:18:12.257 DEBUG:boto:Params: {} 2024-02-01T10:18:12.257 DEBUG:boto:Token: None 2024-02-01T10:18:12.257 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:12 GMT /vclfcn-12/myobj?versionId=null 2024-02-01T10:18:12.257 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Detm9cnn5epbJFYRyO3pNxcdPV4= 2024-02-01T10:18:12.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Detm9cnn5epbJFYRyO3pNxcdPV4=', 'Content-Length': '0'} 2024-02-01T10:18:12.261 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007321a986817c92b7-0065bb6fe4-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:12.308 DEBUG:boto:path=/vclfcn-12/myobj 2024-02-01T10:18:12.308 DEBUG:boto:auth_path=/vclfcn-12/myobj 2024-02-01T10:18:12.308 DEBUG:boto:path=/vclfcn-12/myobj?versionId=null 2024-02-01T10:18:12.308 DEBUG:boto:auth_path=/vclfcn-12/myobj?versionId=null 2024-02-01T10:18:12.309 DEBUG:boto:Method: HEAD 2024-02-01T10:18:12.309 DEBUG:boto:Path: /vclfcn-12/myobj?versionId=null 2024-02-01T10:18:12.309 DEBUG:boto:Data: 2024-02-01T10:18:12.309 DEBUG:boto:Headers: {} 2024-02-01T10:18:12.309 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:12.309 DEBUG:boto:Port: 8000 2024-02-01T10:18:12.309 DEBUG:boto:Params: {} 2024-02-01T10:18:12.309 DEBUG:boto:Token: None 2024-02-01T10:18:12.309 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:12 GMT /vclfcn-12/myobj?versionId=null 2024-02-01T10:18:12.310 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:qI2JeCZ/zb9JaKVngDDCPLcXMeo= 2024-02-01T10:18:12.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:qI2JeCZ/zb9JaKVngDDCPLcXMeo=', 'Content-Length': '0'} 2024-02-01T10:18:12.313 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000516e4d817b8d060e-0065bb6fe4-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:12.313 DEBUG:boto:path=/vclfcn-12/myobj 2024-02-01T10:18:12.314 DEBUG:boto:auth_path=/vclfcn-12/myobj 2024-02-01T10:18:12.314 DEBUG:boto:path=/vclfcn-12/myobj?versionId=null 2024-02-01T10:18:12.314 DEBUG:boto:auth_path=/vclfcn-12/myobj?versionId=null 2024-02-01T10:18:12.314 DEBUG:boto:Method: HEAD 2024-02-01T10:18:12.314 DEBUG:boto:Path: /vclfcn-12/myobj?versionId=null 2024-02-01T10:18:12.314 DEBUG:boto:Data: 2024-02-01T10:18:12.314 DEBUG:boto:Headers: {} 2024-02-01T10:18:12.314 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:12.314 DEBUG:boto:Port: 8001 2024-02-01T10:18:12.314 DEBUG:boto:Params: {} 2024-02-01T10:18:12.315 DEBUG:boto:Token: None 2024-02-01T10:18:12.315 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:12 GMT /vclfcn-12/myobj?versionId=null 2024-02-01T10:18:12.315 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:qI2JeCZ/zb9JaKVngDDCPLcXMeo= 2024-02-01T10:18:12.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:qI2JeCZ/zb9JaKVngDDCPLcXMeo=', 'Content-Length': '0'} 2024-02-01T10:18:12.318 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f78131d427dbf4f0-0065bb6fe4-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:12.318 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-02-01T10:18:12.318 DEBUG:boto:path=/vclfcn-12/myobj 2024-02-01T10:18:12.318 DEBUG:boto:auth_path=/vclfcn-12/myobj 2024-02-01T10:18:12.318 DEBUG:boto:path=/vclfcn-12/myobj?versionId=null 2024-02-01T10:18:12.318 DEBUG:boto:auth_path=/vclfcn-12/myobj?versionId=null 2024-02-01T10:18:12.318 DEBUG:boto:Method: GET 2024-02-01T10:18:12.319 DEBUG:boto:Path: /vclfcn-12/myobj?versionId=null 2024-02-01T10:18:12.319 DEBUG:boto:Data: 2024-02-01T10:18:12.319 DEBUG:boto:Headers: {} 2024-02-01T10:18:12.319 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:12.319 DEBUG:boto:Port: 8000 2024-02-01T10:18:12.319 DEBUG:boto:Params: {} 2024-02-01T10:18:12.319 DEBUG:boto:Token: None 2024-02-01T10:18:12.319 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:12 GMT /vclfcn-12/myobj?versionId=null 2024-02-01T10:18:12.319 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Detm9cnn5epbJFYRyO3pNxcdPV4= 2024-02-01T10:18:12.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Detm9cnn5epbJFYRyO3pNxcdPV4=', 'Content-Length': '0'} 2024-02-01T10:18:12.322 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ea28c44d8c402939-0065bb6fe4-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:12.364 DEBUG:boto:path=/vclfcn-12/myobj 2024-02-01T10:18:12.364 DEBUG:boto:auth_path=/vclfcn-12/myobj 2024-02-01T10:18:12.364 DEBUG:boto:path=/vclfcn-12/myobj?versionId=null 2024-02-01T10:18:12.364 DEBUG:boto:auth_path=/vclfcn-12/myobj?versionId=null 2024-02-01T10:18:12.365 DEBUG:boto:Method: GET 2024-02-01T10:18:12.365 DEBUG:boto:Path: /vclfcn-12/myobj?versionId=null 2024-02-01T10:18:12.365 DEBUG:boto:Data: 2024-02-01T10:18:12.365 DEBUG:boto:Headers: {} 2024-02-01T10:18:12.365 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:12.365 DEBUG:boto:Port: 8001 2024-02-01T10:18:12.365 DEBUG:boto:Params: {} 2024-02-01T10:18:12.365 DEBUG:boto:Token: None 2024-02-01T10:18:12.365 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:12 GMT /vclfcn-12/myobj?versionId=null 2024-02-01T10:18:12.366 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Detm9cnn5epbJFYRyO3pNxcdPV4= 2024-02-01T10:18:12.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Detm9cnn5epbJFYRyO3pNxcdPV4=', 'Content-Length': '0'} 2024-02-01T10:18:12.368 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005274f7543060d574-0065bb6fe4-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:12.369 DEBUG:boto:path=/vclfcn-12/ 2024-02-01T10:18:12.369 DEBUG:boto:auth_path=/vclfcn-12/ 2024-02-01T10:18:12.369 DEBUG:boto:path=/vclfcn-12/?prefix=myobj 2024-02-01T10:18:12.369 DEBUG:boto:auth_path=/vclfcn-12/?prefix=myobj 2024-02-01T10:18:12.369 DEBUG:boto:Method: GET 2024-02-01T10:18:12.369 DEBUG:boto:Path: /vclfcn-12/?prefix=myobj 2024-02-01T10:18:12.369 DEBUG:boto:Data: 2024-02-01T10:18:12.369 DEBUG:boto:Headers: {} 2024-02-01T10:18:12.369 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:12.370 DEBUG:boto:Port: 8000 2024-02-01T10:18:12.370 DEBUG:boto:Params: {} 2024-02-01T10:18:12.370 DEBUG:boto:Token: None 2024-02-01T10:18:12.370 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:12 GMT /vclfcn-12/ 2024-02-01T10:18:12.370 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HjHdiPPd6aPwDPCSTh+P8ZN/iyI= 2024-02-01T10:18:12.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HjHdiPPd6aPwDPCSTh+P8ZN/iyI=', 'Content-Length': '0'} 2024-02-01T10:18:12.374 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000069ba2416d047a6b1-0065bb6fe4-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:12.420 DEBUG:boto:b'vclfcn-12myobj1000falsemyobj2024-02-01T10:16:36.593Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:12.421 DEBUG:boto:path=/vclfcn-12/ 2024-02-01T10:18:12.421 DEBUG:boto:auth_path=/vclfcn-12/ 2024-02-01T10:18:12.421 DEBUG:boto:path=/vclfcn-12/?prefix=myobj 2024-02-01T10:18:12.421 DEBUG:boto:auth_path=/vclfcn-12/?prefix=myobj 2024-02-01T10:18:12.421 DEBUG:boto:Method: GET 2024-02-01T10:18:12.421 DEBUG:boto:Path: /vclfcn-12/?prefix=myobj 2024-02-01T10:18:12.421 DEBUG:boto:Data: 2024-02-01T10:18:12.421 DEBUG:boto:Headers: {} 2024-02-01T10:18:12.421 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:12.422 DEBUG:boto:Port: 8001 2024-02-01T10:18:12.422 DEBUG:boto:Params: {} 2024-02-01T10:18:12.422 DEBUG:boto:Token: None 2024-02-01T10:18:12.422 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:12 GMT /vclfcn-12/ 2024-02-01T10:18:12.422 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HjHdiPPd6aPwDPCSTh+P8ZN/iyI= 2024-02-01T10:18:12.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HjHdiPPd6aPwDPCSTh+P8ZN/iyI=', 'Content-Length': '0'} 2024-02-01T10:18:12.426 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d63f9f4c7cf1007-0065bb6fe4-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:12.426 DEBUG:boto:b'vclfcn-12myobj1000falsemyobj2024-02-01T10:16:36.593Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:12.427 DEBUG:boto:path=/vclfcn-12/myobj 2024-02-01T10:18:12.427 DEBUG:boto:auth_path=/vclfcn-12/myobj 2024-02-01T10:18:12.427 DEBUG:boto:Method: HEAD 2024-02-01T10:18:12.427 DEBUG:boto:Path: /vclfcn-12/myobj 2024-02-01T10:18:12.427 DEBUG:boto:Data: 2024-02-01T10:18:12.427 DEBUG:boto:Headers: {} 2024-02-01T10:18:12.427 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:12.427 DEBUG:boto:Port: 8000 2024-02-01T10:18:12.427 DEBUG:boto:Params: {} 2024-02-01T10:18:12.428 DEBUG:boto:Token: None 2024-02-01T10:18:12.428 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:12 GMT /vclfcn-12/myobj 2024-02-01T10:18:12.428 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:3cw68n53peXYjqlrYuJuuMjcA7k= 2024-02-01T10:18:12.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:3cw68n53peXYjqlrYuJuuMjcA7k=', 'Content-Length': '0'} 2024-02-01T10:18:12.430 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b3f8163bf919e722-0065bb6fe4-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:12.431 DEBUG:boto:path=/vclfcn-12/myobj 2024-02-01T10:18:12.431 DEBUG:boto:auth_path=/vclfcn-12/myobj 2024-02-01T10:18:12.431 DEBUG:boto:Method: HEAD 2024-02-01T10:18:12.431 DEBUG:boto:Path: /vclfcn-12/myobj 2024-02-01T10:18:12.431 DEBUG:boto:Data: 2024-02-01T10:18:12.431 DEBUG:boto:Headers: {} 2024-02-01T10:18:12.431 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:12.431 DEBUG:boto:Port: 8001 2024-02-01T10:18:12.431 DEBUG:boto:Params: {} 2024-02-01T10:18:12.432 DEBUG:boto:Token: None 2024-02-01T10:18:12.432 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:12 GMT /vclfcn-12/myobj 2024-02-01T10:18:12.432 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:3cw68n53peXYjqlrYuJuuMjcA7k= 2024-02-01T10:18:12.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:3cw68n53peXYjqlrYuJuuMjcA7k=', 'Content-Length': '0'} 2024-02-01T10:18:12.435 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000f7473fc0f03cc58-0065bb6fe4-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:12.435 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-02-01T10:18:12.435 DEBUG:boto:path=/vclfcn-12/myobj 2024-02-01T10:18:12.435 DEBUG:boto:auth_path=/vclfcn-12/myobj 2024-02-01T10:18:12.435 DEBUG:boto:Method: GET 2024-02-01T10:18:12.435 DEBUG:boto:Path: /vclfcn-12/myobj 2024-02-01T10:18:12.435 DEBUG:boto:Data: 2024-02-01T10:18:12.435 DEBUG:boto:Headers: {} 2024-02-01T10:18:12.436 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:12.436 DEBUG:boto:Port: 8000 2024-02-01T10:18:12.436 DEBUG:boto:Params: {} 2024-02-01T10:18:12.436 DEBUG:boto:Token: None 2024-02-01T10:18:12.436 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:12 GMT /vclfcn-12/myobj 2024-02-01T10:18:12.436 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HwtapRBy5j/8GJFQIGbJsl9VVp0= 2024-02-01T10:18:12.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HwtapRBy5j/8GJFQIGbJsl9VVp0=', 'Content-Length': '0'} 2024-02-01T10:18:12.439 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000831867d762335b28-0065bb6fe4-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:12.480 DEBUG:boto:path=/vclfcn-12/myobj 2024-02-01T10:18:12.480 DEBUG:boto:auth_path=/vclfcn-12/myobj 2024-02-01T10:18:12.480 DEBUG:boto:Method: GET 2024-02-01T10:18:12.481 DEBUG:boto:Path: /vclfcn-12/myobj 2024-02-01T10:18:12.481 DEBUG:boto:Data: 2024-02-01T10:18:12.481 DEBUG:boto:Headers: {} 2024-02-01T10:18:12.481 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:12.481 DEBUG:boto:Port: 8001 2024-02-01T10:18:12.481 DEBUG:boto:Params: {} 2024-02-01T10:18:12.481 DEBUG:boto:Token: None 2024-02-01T10:18:12.481 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:12 GMT /vclfcn-12/myobj 2024-02-01T10:18:12.481 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HwtapRBy5j/8GJFQIGbJsl9VVp0= 2024-02-01T10:18:12.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HwtapRBy5j/8GJFQIGbJsl9VVp0=', 'Content-Length': '0'} 2024-02-01T10:18:12.485 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000043f90462133ab6a5-0065bb6fe4-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:12.485 DEBUG:boto:path=/vclfcn-12/ 2024-02-01T10:18:12.485 DEBUG:boto:auth_path=/vclfcn-12/ 2024-02-01T10:18:12.486 DEBUG:boto:path=/vclfcn-12/?prefix=myobj 2024-02-01T10:18:12.486 DEBUG:boto:auth_path=/vclfcn-12/?prefix=myobj 2024-02-01T10:18:12.486 DEBUG:boto:Method: GET 2024-02-01T10:18:12.486 DEBUG:boto:Path: /vclfcn-12/?prefix=myobj 2024-02-01T10:18:12.486 DEBUG:boto:Data: 2024-02-01T10:18:12.486 DEBUG:boto:Headers: {} 2024-02-01T10:18:12.486 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:12.486 DEBUG:boto:Port: 8000 2024-02-01T10:18:12.486 DEBUG:boto:Params: {} 2024-02-01T10:18:12.486 DEBUG:boto:Token: None 2024-02-01T10:18:12.487 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:12 GMT /vclfcn-12/ 2024-02-01T10:18:12.487 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HjHdiPPd6aPwDPCSTh+P8ZN/iyI= 2024-02-01T10:18:12.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HjHdiPPd6aPwDPCSTh+P8ZN/iyI=', 'Content-Length': '0'} 2024-02-01T10:18:12.491 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080085e19ca948ebd-0065bb6fe4-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:12.491 DEBUG:boto:b'vclfcn-12myobj1000falsemyobj2024-02-01T10:16:36.593Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:12.491 DEBUG:boto:path=/vclfcn-12/ 2024-02-01T10:18:12.491 DEBUG:boto:auth_path=/vclfcn-12/ 2024-02-01T10:18:12.491 DEBUG:boto:path=/vclfcn-12/?prefix=myobj 2024-02-01T10:18:12.491 DEBUG:boto:auth_path=/vclfcn-12/?prefix=myobj 2024-02-01T10:18:12.492 DEBUG:boto:Method: GET 2024-02-01T10:18:12.492 DEBUG:boto:Path: /vclfcn-12/?prefix=myobj 2024-02-01T10:18:12.492 DEBUG:boto:Data: 2024-02-01T10:18:12.492 DEBUG:boto:Headers: {} 2024-02-01T10:18:12.492 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:12.492 DEBUG:boto:Port: 8001 2024-02-01T10:18:12.492 DEBUG:boto:Params: {} 2024-02-01T10:18:12.492 DEBUG:boto:Token: None 2024-02-01T10:18:12.492 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:12 GMT /vclfcn-12/ 2024-02-01T10:18:12.493 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HjHdiPPd6aPwDPCSTh+P8ZN/iyI= 2024-02-01T10:18:12.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HjHdiPPd6aPwDPCSTh+P8ZN/iyI=', 'Content-Length': '0'} 2024-02-01T10:18:12.497 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000645445929ff7c300-0065bb6fe4-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:12.540 DEBUG:boto:b'vclfcn-12myobj1000falsemyobj2024-02-01T10:16:36.593Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:12.541 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-12 zones={test-zone1, test-zone3} 2024-02-01T10:18:12.541 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:18:12.541 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:18:12.541 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket vclfcn-12 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-02-01T10:18:12.592 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:18:12.592 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:18:12.656 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:18:12.655+0000 7fe80ba9b980 1 bucket sync caught up with source: 2024-02-01T10:18:12.656 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, 00000000001.13.6, , 00000000001.10.6, , , , 00000000001.14.6, 00000000001.9.6, , ] 2024-02-01T10:18:12.656 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, 00000000001.13.6, , 00000000001.10.6, , , , 00000000001.14.6, 00000000001.9.6, , ] 2024-02-01T10:18:12.656 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:18:12.655+0000 7fe80ba9b980 0 bucket checkpoint complete 2024-02-01T10:18:12.678 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-12 zones={test-zone2, test-zone3} 2024-02-01T10:18:12.678 DEBUG:boto:path=/vclfcn-12/ 2024-02-01T10:18:12.678 DEBUG:boto:auth_path=/vclfcn-12/ 2024-02-01T10:18:12.679 DEBUG:boto:Method: HEAD 2024-02-01T10:18:12.679 DEBUG:boto:Path: /vclfcn-12/ 2024-02-01T10:18:12.679 DEBUG:boto:Data: 2024-02-01T10:18:12.679 DEBUG:boto:Headers: {} 2024-02-01T10:18:12.679 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:12.679 DEBUG:boto:Port: 8000 2024-02-01T10:18:12.679 DEBUG:boto:Params: {} 2024-02-01T10:18:12.679 DEBUG:boto:Token: None 2024-02-01T10:18:12.679 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:12 GMT /vclfcn-12/ 2024-02-01T10:18:12.680 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/24REa0mgI2EVFDmDlwCBevWsQY= 2024-02-01T10:18:12.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/24REa0mgI2EVFDmDlwCBevWsQY=', 'Content-Length': '0'} 2024-02-01T10:18:12.682 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c3cdaf8d5a4d3f16-0065bb6fe4-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:18:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:12.682 DEBUG:boto:path=/vclfcn-12/ 2024-02-01T10:18:12.682 DEBUG:boto:auth_path=/vclfcn-12/ 2024-02-01T10:18:12.682 DEBUG:boto:Method: HEAD 2024-02-01T10:18:12.682 DEBUG:boto:Path: /vclfcn-12/ 2024-02-01T10:18:12.682 DEBUG:boto:Data: 2024-02-01T10:18:12.683 DEBUG:boto:Headers: {} 2024-02-01T10:18:12.683 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:12.683 DEBUG:boto:Port: 8001 2024-02-01T10:18:12.683 DEBUG:boto:Params: {} 2024-02-01T10:18:12.683 DEBUG:boto:Token: None 2024-02-01T10:18:12.683 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:12 GMT /vclfcn-12/ 2024-02-01T10:18:12.683 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/24REa0mgI2EVFDmDlwCBevWsQY= 2024-02-01T10:18:12.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/24REa0mgI2EVFDmDlwCBevWsQY=', 'Content-Length': '0'} 2024-02-01T10:18:12.687 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000015f8b67a9f45c718-0065bb6fe4-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:18:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:12.688 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:18:12.688 DEBUG:boto:path=/vclfcn-12/ 2024-02-01T10:18:12.688 DEBUG:boto:auth_path=/vclfcn-12/ 2024-02-01T10:18:12.688 DEBUG:boto:path=/vclfcn-12/?versions&max-keys=999 2024-02-01T10:18:12.688 DEBUG:boto:auth_path=/vclfcn-12/?versions&max-keys=999 2024-02-01T10:18:12.688 DEBUG:boto:Method: GET 2024-02-01T10:18:12.688 DEBUG:boto:Path: /vclfcn-12/?versions&max-keys=999 2024-02-01T10:18:12.688 DEBUG:boto:Data: 2024-02-01T10:18:12.689 DEBUG:boto:Headers: {} 2024-02-01T10:18:12.689 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:12.689 DEBUG:boto:Port: 8000 2024-02-01T10:18:12.689 DEBUG:boto:Params: {} 2024-02-01T10:18:12.689 DEBUG:boto:Token: None 2024-02-01T10:18:12.689 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:12 GMT /vclfcn-12/?versions 2024-02-01T10:18:12.689 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Wm7L6oV+UJDWTzps01LqRBu+e1A= 2024-02-01T10:18:12.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Wm7L6oV+UJDWTzps01LqRBu+e1A=', 'Content-Length': '0'} 2024-02-01T10:18:12.692 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035235d52f4fb2ccb-0065bb6fe4-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:12.692 DEBUG:boto:b'vclfcn-12999false&nulltrue2024-02-01T10:16:36.721Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-02-01T10:16:36.670Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-02-01T10:16:36.640Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-02-01T10:16:36.593Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:12.692 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-02-01T10:18:12.692 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-02-01T10:18:12.693 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-02-01T10:18:12.693 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-02-01T10:18:12.693 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:18:12.694 DEBUG:boto:path=/vclfcn-12/ 2024-02-01T10:18:12.694 DEBUG:boto:auth_path=/vclfcn-12/ 2024-02-01T10:18:12.694 DEBUG:boto:path=/vclfcn-12/?versions&max-keys=999 2024-02-01T10:18:12.694 DEBUG:boto:auth_path=/vclfcn-12/?versions&max-keys=999 2024-02-01T10:18:12.694 DEBUG:boto:Method: GET 2024-02-01T10:18:12.694 DEBUG:boto:Path: /vclfcn-12/?versions&max-keys=999 2024-02-01T10:18:12.694 DEBUG:boto:Data: 2024-02-01T10:18:12.694 DEBUG:boto:Headers: {} 2024-02-01T10:18:12.694 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:12.694 DEBUG:boto:Port: 8001 2024-02-01T10:18:12.694 DEBUG:boto:Params: {} 2024-02-01T10:18:12.694 DEBUG:boto:Token: None 2024-02-01T10:18:12.695 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:12 GMT /vclfcn-12/?versions 2024-02-01T10:18:12.695 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Wm7L6oV+UJDWTzps01LqRBu+e1A= 2024-02-01T10:18:12.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Wm7L6oV+UJDWTzps01LqRBu+e1A=', 'Content-Length': '0'} 2024-02-01T10:18:12.699 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f87e8f3197df89d5-0065bb6fe4-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:12.699 DEBUG:boto:b'vclfcn-12999false&nulltrue2024-02-01T10:16:36.721Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-02-01T10:16:36.670Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-02-01T10:16:36.640Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-02-01T10:16:36.593Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:12.699 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-02-01T10:18:12.699 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-02-01T10:18:12.700 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-02-01T10:18:12.700 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-02-01T10:18:12.700 DEBUG:boto:path=/vclfcn-12/ 2024-02-01T10:18:12.700 DEBUG:boto:auth_path=/vclfcn-12/ 2024-02-01T10:18:12.700 DEBUG:boto:path=/vclfcn-12/?versions&max-keys=999 2024-02-01T10:18:12.700 DEBUG:boto:auth_path=/vclfcn-12/?versions&max-keys=999 2024-02-01T10:18:12.700 DEBUG:boto:Method: GET 2024-02-01T10:18:12.700 DEBUG:boto:Path: /vclfcn-12/?versions&max-keys=999 2024-02-01T10:18:12.701 DEBUG:boto:Data: 2024-02-01T10:18:12.701 DEBUG:boto:Headers: {} 2024-02-01T10:18:12.701 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:12.701 DEBUG:boto:Port: 8000 2024-02-01T10:18:12.701 DEBUG:boto:Params: {} 2024-02-01T10:18:12.701 DEBUG:boto:Token: None 2024-02-01T10:18:12.701 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:12 GMT /vclfcn-12/?versions 2024-02-01T10:18:12.702 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Wm7L6oV+UJDWTzps01LqRBu+e1A= 2024-02-01T10:18:12.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Wm7L6oV+UJDWTzps01LqRBu+e1A=', 'Content-Length': '0'} 2024-02-01T10:18:12.704 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000465f256187a30176-0065bb6fe4-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:12.704 DEBUG:boto:b'vclfcn-12999false&nulltrue2024-02-01T10:16:36.721Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-02-01T10:16:36.670Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-02-01T10:16:36.640Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-02-01T10:16:36.593Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:12.704 DEBUG:boto:path=/vclfcn-12/ 2024-02-01T10:18:12.705 DEBUG:boto:auth_path=/vclfcn-12/ 2024-02-01T10:18:12.705 DEBUG:boto:path=/vclfcn-12/?versions&max-keys=999 2024-02-01T10:18:12.705 DEBUG:boto:auth_path=/vclfcn-12/?versions&max-keys=999 2024-02-01T10:18:12.705 DEBUG:boto:Method: GET 2024-02-01T10:18:12.706 DEBUG:boto:Path: /vclfcn-12/?versions&max-keys=999 2024-02-01T10:18:12.706 DEBUG:boto:Data: 2024-02-01T10:18:12.706 DEBUG:boto:Headers: {} 2024-02-01T10:18:12.706 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:12.706 DEBUG:boto:Port: 8001 2024-02-01T10:18:12.706 DEBUG:boto:Params: {} 2024-02-01T10:18:12.706 DEBUG:boto:Token: None 2024-02-01T10:18:12.707 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:12 GMT /vclfcn-12/?versions 2024-02-01T10:18:12.707 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Wm7L6oV+UJDWTzps01LqRBu+e1A= 2024-02-01T10:18:12.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Wm7L6oV+UJDWTzps01LqRBu+e1A=', 'Content-Length': '0'} 2024-02-01T10:18:12.711 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004660f60575ed3ce0-0065bb6fe4-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:12.711 DEBUG:boto:b'vclfcn-12999false&nulltrue2024-02-01T10:16:36.721Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-02-01T10:16:36.670Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-02-01T10:16:36.640Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-02-01T10:16:36.593Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:12.712 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-02-01T10:18:12.712 DEBUG:boto:path=/vclfcn-12/%26 2024-02-01T10:18:12.712 DEBUG:boto:auth_path=/vclfcn-12/%26 2024-02-01T10:18:12.712 DEBUG:boto:path=/vclfcn-12/%26?versionId=null 2024-02-01T10:18:12.713 DEBUG:boto:auth_path=/vclfcn-12/%26?versionId=null 2024-02-01T10:18:12.713 DEBUG:boto:Method: GET 2024-02-01T10:18:12.713 DEBUG:boto:Path: /vclfcn-12/%26?versionId=null 2024-02-01T10:18:12.713 DEBUG:boto:Data: 2024-02-01T10:18:12.713 DEBUG:boto:Headers: {} 2024-02-01T10:18:12.713 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:12.713 DEBUG:boto:Port: 8000 2024-02-01T10:18:12.713 DEBUG:boto:Params: {} 2024-02-01T10:18:12.713 DEBUG:boto:Token: None 2024-02-01T10:18:12.713 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:12 GMT /vclfcn-12/%26?versionId=null 2024-02-01T10:18:12.714 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:qNIg6kZBfSKgegz1U4i5M/GMw2M= 2024-02-01T10:18:12.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:qNIg6kZBfSKgegz1U4i5M/GMw2M=', 'Content-Length': '0'} 2024-02-01T10:18:12.716 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c8a556ad9b2934f3-0065bb6fe4-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:12.716 DEBUG:boto:path=/vclfcn-12/%26 2024-02-01T10:18:12.716 DEBUG:boto:auth_path=/vclfcn-12/%26 2024-02-01T10:18:12.716 DEBUG:boto:path=/vclfcn-12/%26?versionId=null 2024-02-01T10:18:12.717 DEBUG:boto:auth_path=/vclfcn-12/%26?versionId=null 2024-02-01T10:18:12.717 DEBUG:boto:Method: GET 2024-02-01T10:18:12.717 DEBUG:boto:Path: /vclfcn-12/%26?versionId=null 2024-02-01T10:18:12.717 DEBUG:boto:Data: 2024-02-01T10:18:12.717 DEBUG:boto:Headers: {} 2024-02-01T10:18:12.717 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:12.717 DEBUG:boto:Port: 8001 2024-02-01T10:18:12.717 DEBUG:boto:Params: {} 2024-02-01T10:18:12.717 DEBUG:boto:Token: None 2024-02-01T10:18:12.718 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:12 GMT /vclfcn-12/%26?versionId=null 2024-02-01T10:18:12.718 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:qNIg6kZBfSKgegz1U4i5M/GMw2M= 2024-02-01T10:18:12.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:qNIg6kZBfSKgegz1U4i5M/GMw2M=', 'Content-Length': '0'} 2024-02-01T10:18:12.721 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000524a578e544c95b2-0065bb6fe4-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:12.764 DEBUG:boto:path=/vclfcn-12/%26 2024-02-01T10:18:12.764 DEBUG:boto:auth_path=/vclfcn-12/%26 2024-02-01T10:18:12.764 DEBUG:boto:path=/vclfcn-12/%26?versionId=null 2024-02-01T10:18:12.764 DEBUG:boto:auth_path=/vclfcn-12/%26?versionId=null 2024-02-01T10:18:12.764 DEBUG:boto:Method: HEAD 2024-02-01T10:18:12.764 DEBUG:boto:Path: /vclfcn-12/%26?versionId=null 2024-02-01T10:18:12.765 DEBUG:boto:Data: 2024-02-01T10:18:12.765 DEBUG:boto:Headers: {} 2024-02-01T10:18:12.765 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:12.765 DEBUG:boto:Port: 8000 2024-02-01T10:18:12.765 DEBUG:boto:Params: {} 2024-02-01T10:18:12.765 DEBUG:boto:Token: None 2024-02-01T10:18:12.765 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:12 GMT /vclfcn-12/%26?versionId=null 2024-02-01T10:18:12.765 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:2yagC36pOeu5LO3BnYP3ijUAD90= 2024-02-01T10:18:12.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2yagC36pOeu5LO3BnYP3ijUAD90=', 'Content-Length': '0'} 2024-02-01T10:18:12.768 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d6eb097565cb4967-0065bb6fe4-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:12.769 DEBUG:boto:path=/vclfcn-12/%26 2024-02-01T10:18:12.769 DEBUG:boto:auth_path=/vclfcn-12/%26 2024-02-01T10:18:12.769 DEBUG:boto:path=/vclfcn-12/%26?versionId=null 2024-02-01T10:18:12.769 DEBUG:boto:auth_path=/vclfcn-12/%26?versionId=null 2024-02-01T10:18:12.769 DEBUG:boto:Method: HEAD 2024-02-01T10:18:12.769 DEBUG:boto:Path: /vclfcn-12/%26?versionId=null 2024-02-01T10:18:12.769 DEBUG:boto:Data: 2024-02-01T10:18:12.769 DEBUG:boto:Headers: {} 2024-02-01T10:18:12.769 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:12.769 DEBUG:boto:Port: 8001 2024-02-01T10:18:12.770 DEBUG:boto:Params: {} 2024-02-01T10:18:12.770 DEBUG:boto:Token: None 2024-02-01T10:18:12.770 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:12 GMT /vclfcn-12/%26?versionId=null 2024-02-01T10:18:12.770 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:2yagC36pOeu5LO3BnYP3ijUAD90= 2024-02-01T10:18:12.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2yagC36pOeu5LO3BnYP3ijUAD90=', 'Content-Length': '0'} 2024-02-01T10:18:12.773 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000068960d1584137c3c-0065bb6fe4-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:12.773 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-02-01T10:18:12.773 DEBUG:boto:path=/vclfcn-12/%26 2024-02-01T10:18:12.773 DEBUG:boto:auth_path=/vclfcn-12/%26 2024-02-01T10:18:12.773 DEBUG:boto:path=/vclfcn-12/%26?versionId=null 2024-02-01T10:18:12.773 DEBUG:boto:auth_path=/vclfcn-12/%26?versionId=null 2024-02-01T10:18:12.773 DEBUG:boto:Method: GET 2024-02-01T10:18:12.774 DEBUG:boto:Path: /vclfcn-12/%26?versionId=null 2024-02-01T10:18:12.774 DEBUG:boto:Data: 2024-02-01T10:18:12.774 DEBUG:boto:Headers: {} 2024-02-01T10:18:12.774 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:12.774 DEBUG:boto:Port: 8000 2024-02-01T10:18:12.774 DEBUG:boto:Params: {} 2024-02-01T10:18:12.774 DEBUG:boto:Token: None 2024-02-01T10:18:12.774 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:12 GMT /vclfcn-12/%26?versionId=null 2024-02-01T10:18:12.774 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:qNIg6kZBfSKgegz1U4i5M/GMw2M= 2024-02-01T10:18:12.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:qNIg6kZBfSKgegz1U4i5M/GMw2M=', 'Content-Length': '0'} 2024-02-01T10:18:12.778 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e19dfbec8f98916c-0065bb6fe4-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:12.778 DEBUG:boto:path=/vclfcn-12/%26 2024-02-01T10:18:12.778 DEBUG:boto:auth_path=/vclfcn-12/%26 2024-02-01T10:18:12.778 DEBUG:boto:path=/vclfcn-12/%26?versionId=null 2024-02-01T10:18:12.778 DEBUG:boto:auth_path=/vclfcn-12/%26?versionId=null 2024-02-01T10:18:12.778 DEBUG:boto:Method: GET 2024-02-01T10:18:12.778 DEBUG:boto:Path: /vclfcn-12/%26?versionId=null 2024-02-01T10:18:12.779 DEBUG:boto:Data: 2024-02-01T10:18:12.779 DEBUG:boto:Headers: {} 2024-02-01T10:18:12.779 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:12.779 DEBUG:boto:Port: 8001 2024-02-01T10:18:12.779 DEBUG:boto:Params: {} 2024-02-01T10:18:12.779 DEBUG:boto:Token: None 2024-02-01T10:18:12.779 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:12 GMT /vclfcn-12/%26?versionId=null 2024-02-01T10:18:12.779 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:qNIg6kZBfSKgegz1U4i5M/GMw2M= 2024-02-01T10:18:12.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:qNIg6kZBfSKgegz1U4i5M/GMw2M=', 'Content-Length': '0'} 2024-02-01T10:18:12.782 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000bd1402afe52db33-0065bb6fe4-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:12.782 DEBUG:boto:path=/vclfcn-12/ 2024-02-01T10:18:12.782 DEBUG:boto:auth_path=/vclfcn-12/ 2024-02-01T10:18:12.782 DEBUG:boto:path=/vclfcn-12/?prefix=%26 2024-02-01T10:18:12.783 DEBUG:boto:auth_path=/vclfcn-12/?prefix=%26 2024-02-01T10:18:12.783 DEBUG:boto:Method: GET 2024-02-01T10:18:12.783 DEBUG:boto:Path: /vclfcn-12/?prefix=%26 2024-02-01T10:18:12.783 DEBUG:boto:Data: 2024-02-01T10:18:12.783 DEBUG:boto:Headers: {} 2024-02-01T10:18:12.783 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:12.783 DEBUG:boto:Port: 8000 2024-02-01T10:18:12.783 DEBUG:boto:Params: {} 2024-02-01T10:18:12.783 DEBUG:boto:Token: None 2024-02-01T10:18:12.783 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:12 GMT /vclfcn-12/ 2024-02-01T10:18:12.784 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HjHdiPPd6aPwDPCSTh+P8ZN/iyI= 2024-02-01T10:18:12.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HjHdiPPd6aPwDPCSTh+P8ZN/iyI=', 'Content-Length': '0'} 2024-02-01T10:18:12.787 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092439d7fe1266892-0065bb6fe4-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:12.828 DEBUG:boto:b'vclfcn-12&1000false&2024-02-01T10:16:36.721Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:12.829 DEBUG:boto:path=/vclfcn-12/ 2024-02-01T10:18:12.829 DEBUG:boto:auth_path=/vclfcn-12/ 2024-02-01T10:18:12.829 DEBUG:boto:path=/vclfcn-12/?prefix=%26 2024-02-01T10:18:12.829 DEBUG:boto:auth_path=/vclfcn-12/?prefix=%26 2024-02-01T10:18:12.829 DEBUG:boto:Method: GET 2024-02-01T10:18:12.829 DEBUG:boto:Path: /vclfcn-12/?prefix=%26 2024-02-01T10:18:12.829 DEBUG:boto:Data: 2024-02-01T10:18:12.829 DEBUG:boto:Headers: {} 2024-02-01T10:18:12.829 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:12.829 DEBUG:boto:Port: 8001 2024-02-01T10:18:12.830 DEBUG:boto:Params: {} 2024-02-01T10:18:12.830 DEBUG:boto:Token: None 2024-02-01T10:18:12.830 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:12 GMT /vclfcn-12/ 2024-02-01T10:18:12.830 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HjHdiPPd6aPwDPCSTh+P8ZN/iyI= 2024-02-01T10:18:12.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HjHdiPPd6aPwDPCSTh+P8ZN/iyI=', 'Content-Length': '0'} 2024-02-01T10:18:12.834 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1e8ecd49e87f69a-0065bb6fe4-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:12.834 DEBUG:boto:b'vclfcn-12&1000false&2024-02-01T10:16:36.721Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:12.835 DEBUG:boto:path=/vclfcn-12/%26 2024-02-01T10:18:12.835 DEBUG:boto:auth_path=/vclfcn-12/%26 2024-02-01T10:18:12.835 DEBUG:boto:Method: HEAD 2024-02-01T10:18:12.835 DEBUG:boto:Path: /vclfcn-12/%26 2024-02-01T10:18:12.836 DEBUG:boto:Data: 2024-02-01T10:18:12.836 DEBUG:boto:Headers: {} 2024-02-01T10:18:12.836 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:12.836 DEBUG:boto:Port: 8000 2024-02-01T10:18:12.836 DEBUG:boto:Params: {} 2024-02-01T10:18:12.836 DEBUG:boto:Token: None 2024-02-01T10:18:12.836 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:12 GMT /vclfcn-12/%26 2024-02-01T10:18:12.836 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RgCiExvX4Ci2jy1z41L00Syoaeg= 2024-02-01T10:18:12.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RgCiExvX4Ci2jy1z41L00Syoaeg=', 'Content-Length': '0'} 2024-02-01T10:18:12.840 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000985759d93c3cad2d-0065bb6fe4-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:12.840 DEBUG:boto:path=/vclfcn-12/%26 2024-02-01T10:18:12.840 DEBUG:boto:auth_path=/vclfcn-12/%26 2024-02-01T10:18:12.840 DEBUG:boto:Method: HEAD 2024-02-01T10:18:12.840 DEBUG:boto:Path: /vclfcn-12/%26 2024-02-01T10:18:12.840 DEBUG:boto:Data: 2024-02-01T10:18:12.840 DEBUG:boto:Headers: {} 2024-02-01T10:18:12.840 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:12.840 DEBUG:boto:Port: 8001 2024-02-01T10:18:12.841 DEBUG:boto:Params: {} 2024-02-01T10:18:12.841 DEBUG:boto:Token: None 2024-02-01T10:18:12.841 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:12 GMT /vclfcn-12/%26 2024-02-01T10:18:12.841 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RgCiExvX4Ci2jy1z41L00Syoaeg= 2024-02-01T10:18:12.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RgCiExvX4Ci2jy1z41L00Syoaeg=', 'Content-Length': '0'} 2024-02-01T10:18:12.844 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ff5b241df3d7a2bc-0065bb6fe4-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:12.844 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-02-01T10:18:12.844 DEBUG:boto:path=/vclfcn-12/%26 2024-02-01T10:18:12.844 DEBUG:boto:auth_path=/vclfcn-12/%26 2024-02-01T10:18:12.844 DEBUG:boto:Method: GET 2024-02-01T10:18:12.844 DEBUG:boto:Path: /vclfcn-12/%26 2024-02-01T10:18:12.844 DEBUG:boto:Data: 2024-02-01T10:18:12.845 DEBUG:boto:Headers: {} 2024-02-01T10:18:12.845 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:12.845 DEBUG:boto:Port: 8000 2024-02-01T10:18:12.845 DEBUG:boto:Params: {} 2024-02-01T10:18:12.845 DEBUG:boto:Token: None 2024-02-01T10:18:12.845 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:12 GMT /vclfcn-12/%26 2024-02-01T10:18:12.845 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GmWo+3k9bO01hCiSrwtPBNetBjA= 2024-02-01T10:18:12.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GmWo+3k9bO01hCiSrwtPBNetBjA=', 'Content-Length': '0'} 2024-02-01T10:18:12.848 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000079dda2522422e898-0065bb6fe4-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:12.848 DEBUG:boto:path=/vclfcn-12/%26 2024-02-01T10:18:12.848 DEBUG:boto:auth_path=/vclfcn-12/%26 2024-02-01T10:18:12.848 DEBUG:boto:Method: GET 2024-02-01T10:18:12.849 DEBUG:boto:Path: /vclfcn-12/%26 2024-02-01T10:18:12.849 DEBUG:boto:Data: 2024-02-01T10:18:12.849 DEBUG:boto:Headers: {} 2024-02-01T10:18:12.849 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:12.849 DEBUG:boto:Port: 8001 2024-02-01T10:18:12.849 DEBUG:boto:Params: {} 2024-02-01T10:18:12.849 DEBUG:boto:Token: None 2024-02-01T10:18:12.849 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:12 GMT /vclfcn-12/%26 2024-02-01T10:18:12.849 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GmWo+3k9bO01hCiSrwtPBNetBjA= 2024-02-01T10:18:12.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GmWo+3k9bO01hCiSrwtPBNetBjA=', 'Content-Length': '0'} 2024-02-01T10:18:12.852 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cf8ca551f7442515-0065bb6fe4-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:12.900 DEBUG:boto:path=/vclfcn-12/ 2024-02-01T10:18:12.900 DEBUG:boto:auth_path=/vclfcn-12/ 2024-02-01T10:18:12.900 DEBUG:boto:path=/vclfcn-12/?prefix=%26 2024-02-01T10:18:12.901 DEBUG:boto:auth_path=/vclfcn-12/?prefix=%26 2024-02-01T10:18:12.901 DEBUG:boto:Method: GET 2024-02-01T10:18:12.901 DEBUG:boto:Path: /vclfcn-12/?prefix=%26 2024-02-01T10:18:12.901 DEBUG:boto:Data: 2024-02-01T10:18:12.901 DEBUG:boto:Headers: {} 2024-02-01T10:18:12.901 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:12.901 DEBUG:boto:Port: 8000 2024-02-01T10:18:12.901 DEBUG:boto:Params: {} 2024-02-01T10:18:12.901 DEBUG:boto:Token: None 2024-02-01T10:18:12.902 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:12 GMT /vclfcn-12/ 2024-02-01T10:18:12.902 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HjHdiPPd6aPwDPCSTh+P8ZN/iyI= 2024-02-01T10:18:12.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HjHdiPPd6aPwDPCSTh+P8ZN/iyI=', 'Content-Length': '0'} 2024-02-01T10:18:12.906 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c32dd3e95eb2d50a-0065bb6fe4-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:12.907 DEBUG:boto:b'vclfcn-12&1000false&2024-02-01T10:16:36.721Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:12.907 DEBUG:boto:path=/vclfcn-12/ 2024-02-01T10:18:12.907 DEBUG:boto:auth_path=/vclfcn-12/ 2024-02-01T10:18:12.908 DEBUG:boto:path=/vclfcn-12/?prefix=%26 2024-02-01T10:18:12.908 DEBUG:boto:auth_path=/vclfcn-12/?prefix=%26 2024-02-01T10:18:12.908 DEBUG:boto:Method: GET 2024-02-01T10:18:12.908 DEBUG:boto:Path: /vclfcn-12/?prefix=%26 2024-02-01T10:18:12.908 DEBUG:boto:Data: 2024-02-01T10:18:12.908 DEBUG:boto:Headers: {} 2024-02-01T10:18:12.908 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:12.908 DEBUG:boto:Port: 8001 2024-02-01T10:18:12.908 DEBUG:boto:Params: {} 2024-02-01T10:18:12.908 DEBUG:boto:Token: None 2024-02-01T10:18:12.909 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:12 GMT /vclfcn-12/ 2024-02-01T10:18:12.909 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HjHdiPPd6aPwDPCSTh+P8ZN/iyI= 2024-02-01T10:18:12.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HjHdiPPd6aPwDPCSTh+P8ZN/iyI=', 'Content-Length': '0'} 2024-02-01T10:18:12.913 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000073a0d64e3a29dd0a-0065bb6fe4-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:12.913 DEBUG:boto:b'vclfcn-12&1000false&2024-02-01T10:16:36.721Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:12.914 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-02-01T10:18:12.914 DEBUG:boto:path=/vclfcn-12/%3A 2024-02-01T10:18:12.914 DEBUG:boto:auth_path=/vclfcn-12/%3A 2024-02-01T10:18:12.914 DEBUG:boto:path=/vclfcn-12/%3A?versionId=null 2024-02-01T10:18:12.914 DEBUG:boto:auth_path=/vclfcn-12/%3A?versionId=null 2024-02-01T10:18:12.914 DEBUG:boto:Method: GET 2024-02-01T10:18:12.914 DEBUG:boto:Path: /vclfcn-12/%3A?versionId=null 2024-02-01T10:18:12.914 DEBUG:boto:Data: 2024-02-01T10:18:12.915 DEBUG:boto:Headers: {} 2024-02-01T10:18:12.915 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:12.915 DEBUG:boto:Port: 8000 2024-02-01T10:18:12.915 DEBUG:boto:Params: {} 2024-02-01T10:18:12.915 DEBUG:boto:Token: None 2024-02-01T10:18:12.915 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:12 GMT /vclfcn-12/%3A?versionId=null 2024-02-01T10:18:12.915 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9GwERSGQFp/jNidBCiE3cZHSPxY= 2024-02-01T10:18:12.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9GwERSGQFp/jNidBCiE3cZHSPxY=', 'Content-Length': '0'} 2024-02-01T10:18:12.918 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000afeb6dee53519414-0065bb6fe4-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:12.918 DEBUG:boto:path=/vclfcn-12/%3A 2024-02-01T10:18:12.919 DEBUG:boto:auth_path=/vclfcn-12/%3A 2024-02-01T10:18:12.919 DEBUG:boto:path=/vclfcn-12/%3A?versionId=null 2024-02-01T10:18:12.919 DEBUG:boto:auth_path=/vclfcn-12/%3A?versionId=null 2024-02-01T10:18:12.919 DEBUG:boto:Method: GET 2024-02-01T10:18:12.919 DEBUG:boto:Path: /vclfcn-12/%3A?versionId=null 2024-02-01T10:18:12.919 DEBUG:boto:Data: 2024-02-01T10:18:12.919 DEBUG:boto:Headers: {} 2024-02-01T10:18:12.919 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:12.919 DEBUG:boto:Port: 8001 2024-02-01T10:18:12.919 DEBUG:boto:Params: {} 2024-02-01T10:18:12.920 DEBUG:boto:Token: None 2024-02-01T10:18:12.920 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:12 GMT /vclfcn-12/%3A?versionId=null 2024-02-01T10:18:12.920 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9GwERSGQFp/jNidBCiE3cZHSPxY= 2024-02-01T10:18:12.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9GwERSGQFp/jNidBCiE3cZHSPxY=', 'Content-Length': '0'} 2024-02-01T10:18:12.923 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009a358ef3fba49840-0065bb6fe4-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:12.923 DEBUG:boto:path=/vclfcn-12/%3A 2024-02-01T10:18:12.923 DEBUG:boto:auth_path=/vclfcn-12/%3A 2024-02-01T10:18:12.923 DEBUG:boto:path=/vclfcn-12/%3A?versionId=null 2024-02-01T10:18:12.923 DEBUG:boto:auth_path=/vclfcn-12/%3A?versionId=null 2024-02-01T10:18:12.923 DEBUG:boto:Method: HEAD 2024-02-01T10:18:12.923 DEBUG:boto:Path: /vclfcn-12/%3A?versionId=null 2024-02-01T10:18:12.923 DEBUG:boto:Data: 2024-02-01T10:18:12.924 DEBUG:boto:Headers: {} 2024-02-01T10:18:12.924 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:12.924 DEBUG:boto:Port: 8000 2024-02-01T10:18:12.924 DEBUG:boto:Params: {} 2024-02-01T10:18:12.924 DEBUG:boto:Token: None 2024-02-01T10:18:12.924 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:12 GMT /vclfcn-12/%3A?versionId=null 2024-02-01T10:18:12.924 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fr9TIgCox7PrsJM/P+zdWp42H5Y= 2024-02-01T10:18:12.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fr9TIgCox7PrsJM/P+zdWp42H5Y=', 'Content-Length': '0'} 2024-02-01T10:18:12.927 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000019a765f1938e6845-0065bb6fe4-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:12.927 DEBUG:boto:path=/vclfcn-12/%3A 2024-02-01T10:18:12.927 DEBUG:boto:auth_path=/vclfcn-12/%3A 2024-02-01T10:18:12.928 DEBUG:boto:path=/vclfcn-12/%3A?versionId=null 2024-02-01T10:18:12.928 DEBUG:boto:auth_path=/vclfcn-12/%3A?versionId=null 2024-02-01T10:18:12.928 DEBUG:boto:Method: HEAD 2024-02-01T10:18:12.928 DEBUG:boto:Path: /vclfcn-12/%3A?versionId=null 2024-02-01T10:18:12.928 DEBUG:boto:Data: 2024-02-01T10:18:12.928 DEBUG:boto:Headers: {} 2024-02-01T10:18:12.928 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:12.928 DEBUG:boto:Port: 8001 2024-02-01T10:18:12.928 DEBUG:boto:Params: {} 2024-02-01T10:18:12.928 DEBUG:boto:Token: None 2024-02-01T10:18:12.929 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:12 GMT /vclfcn-12/%3A?versionId=null 2024-02-01T10:18:12.929 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fr9TIgCox7PrsJM/P+zdWp42H5Y= 2024-02-01T10:18:12.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fr9TIgCox7PrsJM/P+zdWp42H5Y=', 'Content-Length': '0'} 2024-02-01T10:18:12.931 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000015a93390584c3123-0065bb6fe4-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:12.932 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-02-01T10:18:12.932 DEBUG:boto:path=/vclfcn-12/%3A 2024-02-01T10:18:12.932 DEBUG:boto:auth_path=/vclfcn-12/%3A 2024-02-01T10:18:12.932 DEBUG:boto:path=/vclfcn-12/%3A?versionId=null 2024-02-01T10:18:12.932 DEBUG:boto:auth_path=/vclfcn-12/%3A?versionId=null 2024-02-01T10:18:12.932 DEBUG:boto:Method: GET 2024-02-01T10:18:12.933 DEBUG:boto:Path: /vclfcn-12/%3A?versionId=null 2024-02-01T10:18:12.933 DEBUG:boto:Data: 2024-02-01T10:18:12.933 DEBUG:boto:Headers: {} 2024-02-01T10:18:12.933 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:12.933 DEBUG:boto:Port: 8000 2024-02-01T10:18:12.933 DEBUG:boto:Params: {} 2024-02-01T10:18:12.933 DEBUG:boto:Token: None 2024-02-01T10:18:12.933 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:12 GMT /vclfcn-12/%3A?versionId=null 2024-02-01T10:18:12.933 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9GwERSGQFp/jNidBCiE3cZHSPxY= 2024-02-01T10:18:12.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9GwERSGQFp/jNidBCiE3cZHSPxY=', 'Content-Length': '0'} 2024-02-01T10:18:12.936 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003d7d2916d2356804-0065bb6fe4-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:12.980 DEBUG:boto:path=/vclfcn-12/%3A 2024-02-01T10:18:12.980 DEBUG:boto:auth_path=/vclfcn-12/%3A 2024-02-01T10:18:12.980 DEBUG:boto:path=/vclfcn-12/%3A?versionId=null 2024-02-01T10:18:12.981 DEBUG:boto:auth_path=/vclfcn-12/%3A?versionId=null 2024-02-01T10:18:12.981 DEBUG:boto:Method: GET 2024-02-01T10:18:12.981 DEBUG:boto:Path: /vclfcn-12/%3A?versionId=null 2024-02-01T10:18:12.981 DEBUG:boto:Data: 2024-02-01T10:18:12.981 DEBUG:boto:Headers: {} 2024-02-01T10:18:12.981 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:12.981 DEBUG:boto:Port: 8001 2024-02-01T10:18:12.981 DEBUG:boto:Params: {} 2024-02-01T10:18:12.981 DEBUG:boto:Token: None 2024-02-01T10:18:12.982 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:12 GMT /vclfcn-12/%3A?versionId=null 2024-02-01T10:18:12.982 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9GwERSGQFp/jNidBCiE3cZHSPxY= 2024-02-01T10:18:12.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9GwERSGQFp/jNidBCiE3cZHSPxY=', 'Content-Length': '0'} 2024-02-01T10:18:12.985 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bdaab109abf21aa1-0065bb6fe4-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:12.985 DEBUG:boto:path=/vclfcn-12/ 2024-02-01T10:18:12.985 DEBUG:boto:auth_path=/vclfcn-12/ 2024-02-01T10:18:12.986 DEBUG:boto:path=/vclfcn-12/?prefix=%3A 2024-02-01T10:18:12.986 DEBUG:boto:auth_path=/vclfcn-12/?prefix=%3A 2024-02-01T10:18:12.986 DEBUG:boto:Method: GET 2024-02-01T10:18:12.986 DEBUG:boto:Path: /vclfcn-12/?prefix=%3A 2024-02-01T10:18:12.986 DEBUG:boto:Data: 2024-02-01T10:18:12.986 DEBUG:boto:Headers: {} 2024-02-01T10:18:12.986 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:12.986 DEBUG:boto:Port: 8000 2024-02-01T10:18:12.986 DEBUG:boto:Params: {} 2024-02-01T10:18:12.986 DEBUG:boto:Token: None 2024-02-01T10:18:12.987 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:12 GMT /vclfcn-12/ 2024-02-01T10:18:12.987 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HjHdiPPd6aPwDPCSTh+P8ZN/iyI= 2024-02-01T10:18:12.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HjHdiPPd6aPwDPCSTh+P8ZN/iyI=', 'Content-Length': '0'} 2024-02-01T10:18:12.991 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008082f47128f2cb06-0065bb6fe4-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:12.991 DEBUG:boto:b'vclfcn-12:1000false:2024-02-01T10:16:36.670Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:12.992 DEBUG:boto:path=/vclfcn-12/ 2024-02-01T10:18:12.992 DEBUG:boto:auth_path=/vclfcn-12/ 2024-02-01T10:18:12.992 DEBUG:boto:path=/vclfcn-12/?prefix=%3A 2024-02-01T10:18:12.992 DEBUG:boto:auth_path=/vclfcn-12/?prefix=%3A 2024-02-01T10:18:12.992 DEBUG:boto:Method: GET 2024-02-01T10:18:12.992 DEBUG:boto:Path: /vclfcn-12/?prefix=%3A 2024-02-01T10:18:12.992 DEBUG:boto:Data: 2024-02-01T10:18:12.992 DEBUG:boto:Headers: {} 2024-02-01T10:18:12.992 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:12.992 DEBUG:boto:Port: 8001 2024-02-01T10:18:12.993 DEBUG:boto:Params: {} 2024-02-01T10:18:12.993 DEBUG:boto:Token: None 2024-02-01T10:18:12.993 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:12 GMT /vclfcn-12/ 2024-02-01T10:18:12.993 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HjHdiPPd6aPwDPCSTh+P8ZN/iyI= 2024-02-01T10:18:12.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HjHdiPPd6aPwDPCSTh+P8ZN/iyI=', 'Content-Length': '0'} 2024-02-01T10:18:12.997 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005321a4b1bf020d52-0065bb6fe4-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:13.040 DEBUG:boto:b'vclfcn-12:1000false:2024-02-01T10:16:36.670Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:13.041 DEBUG:boto:path=/vclfcn-12/%3A 2024-02-01T10:18:13.041 DEBUG:boto:auth_path=/vclfcn-12/%3A 2024-02-01T10:18:13.041 DEBUG:boto:Method: HEAD 2024-02-01T10:18:13.041 DEBUG:boto:Path: /vclfcn-12/%3A 2024-02-01T10:18:13.041 DEBUG:boto:Data: 2024-02-01T10:18:13.041 DEBUG:boto:Headers: {} 2024-02-01T10:18:13.041 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:13.042 DEBUG:boto:Port: 8000 2024-02-01T10:18:13.042 DEBUG:boto:Params: {} 2024-02-01T10:18:13.042 DEBUG:boto:Token: None 2024-02-01T10:18:13.042 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:13 GMT /vclfcn-12/%3A 2024-02-01T10:18:13.042 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7lzB82MRSiOVXPCA8F8Xra9epvI= 2024-02-01T10:18:13.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7lzB82MRSiOVXPCA8F8Xra9epvI=', 'Content-Length': '0'} 2024-02-01T10:18:13.045 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000049f0bbcf8fb2caeb-0065bb6fe5-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:13.045 DEBUG:boto:path=/vclfcn-12/%3A 2024-02-01T10:18:13.045 DEBUG:boto:auth_path=/vclfcn-12/%3A 2024-02-01T10:18:13.045 DEBUG:boto:Method: HEAD 2024-02-01T10:18:13.046 DEBUG:boto:Path: /vclfcn-12/%3A 2024-02-01T10:18:13.046 DEBUG:boto:Data: 2024-02-01T10:18:13.046 DEBUG:boto:Headers: {} 2024-02-01T10:18:13.046 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:13.046 DEBUG:boto:Port: 8001 2024-02-01T10:18:13.046 DEBUG:boto:Params: {} 2024-02-01T10:18:13.046 DEBUG:boto:Token: None 2024-02-01T10:18:13.046 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:13 GMT /vclfcn-12/%3A 2024-02-01T10:18:13.046 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7lzB82MRSiOVXPCA8F8Xra9epvI= 2024-02-01T10:18:13.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7lzB82MRSiOVXPCA8F8Xra9epvI=', 'Content-Length': '0'} 2024-02-01T10:18:13.049 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000bf11d0a558f126f-0065bb6fe5-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:13.049 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-02-01T10:18:13.050 DEBUG:boto:path=/vclfcn-12/%3A 2024-02-01T10:18:13.050 DEBUG:boto:auth_path=/vclfcn-12/%3A 2024-02-01T10:18:13.050 DEBUG:boto:Method: GET 2024-02-01T10:18:13.050 DEBUG:boto:Path: /vclfcn-12/%3A 2024-02-01T10:18:13.050 DEBUG:boto:Data: 2024-02-01T10:18:13.050 DEBUG:boto:Headers: {} 2024-02-01T10:18:13.050 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:13.050 DEBUG:boto:Port: 8000 2024-02-01T10:18:13.050 DEBUG:boto:Params: {} 2024-02-01T10:18:13.050 DEBUG:boto:Token: None 2024-02-01T10:18:13.051 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:13 GMT /vclfcn-12/%3A 2024-02-01T10:18:13.051 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:AdZMfQUAh+FPHch0u/mCa+L3sV8= 2024-02-01T10:18:13.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:AdZMfQUAh+FPHch0u/mCa+L3sV8=', 'Content-Length': '0'} 2024-02-01T10:18:13.054 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000697c9fc3c3c8ed74-0065bb6fe5-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:13.054 DEBUG:boto:path=/vclfcn-12/%3A 2024-02-01T10:18:13.054 DEBUG:boto:auth_path=/vclfcn-12/%3A 2024-02-01T10:18:13.054 DEBUG:boto:Method: GET 2024-02-01T10:18:13.054 DEBUG:boto:Path: /vclfcn-12/%3A 2024-02-01T10:18:13.054 DEBUG:boto:Data: 2024-02-01T10:18:13.054 DEBUG:boto:Headers: {} 2024-02-01T10:18:13.055 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:13.055 DEBUG:boto:Port: 8001 2024-02-01T10:18:13.055 DEBUG:boto:Params: {} 2024-02-01T10:18:13.055 DEBUG:boto:Token: None 2024-02-01T10:18:13.055 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:13 GMT /vclfcn-12/%3A 2024-02-01T10:18:13.055 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:AdZMfQUAh+FPHch0u/mCa+L3sV8= 2024-02-01T10:18:13.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:AdZMfQUAh+FPHch0u/mCa+L3sV8=', 'Content-Length': '0'} 2024-02-01T10:18:13.058 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f62e67b4687f2654-0065bb6fe5-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:13.058 DEBUG:boto:path=/vclfcn-12/ 2024-02-01T10:18:13.059 DEBUG:boto:auth_path=/vclfcn-12/ 2024-02-01T10:18:13.059 DEBUG:boto:path=/vclfcn-12/?prefix=%3A 2024-02-01T10:18:13.059 DEBUG:boto:auth_path=/vclfcn-12/?prefix=%3A 2024-02-01T10:18:13.059 DEBUG:boto:Method: GET 2024-02-01T10:18:13.059 DEBUG:boto:Path: /vclfcn-12/?prefix=%3A 2024-02-01T10:18:13.059 DEBUG:boto:Data: 2024-02-01T10:18:13.059 DEBUG:boto:Headers: {} 2024-02-01T10:18:13.059 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:13.059 DEBUG:boto:Port: 8000 2024-02-01T10:18:13.059 DEBUG:boto:Params: {} 2024-02-01T10:18:13.060 DEBUG:boto:Token: None 2024-02-01T10:18:13.060 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:13 GMT /vclfcn-12/ 2024-02-01T10:18:13.060 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:EOaK3E32jM46ePAZZBVpXu7grPs= 2024-02-01T10:18:13.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:EOaK3E32jM46ePAZZBVpXu7grPs=', 'Content-Length': '0'} 2024-02-01T10:18:13.064 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005814096036987856-0065bb6fe5-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:13.104 DEBUG:boto:b'vclfcn-12:1000false:2024-02-01T10:16:36.670Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:13.105 DEBUG:boto:path=/vclfcn-12/ 2024-02-01T10:18:13.105 DEBUG:boto:auth_path=/vclfcn-12/ 2024-02-01T10:18:13.105 DEBUG:boto:path=/vclfcn-12/?prefix=%3A 2024-02-01T10:18:13.105 DEBUG:boto:auth_path=/vclfcn-12/?prefix=%3A 2024-02-01T10:18:13.105 DEBUG:boto:Method: GET 2024-02-01T10:18:13.105 DEBUG:boto:Path: /vclfcn-12/?prefix=%3A 2024-02-01T10:18:13.105 DEBUG:boto:Data: 2024-02-01T10:18:13.106 DEBUG:boto:Headers: {} 2024-02-01T10:18:13.106 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:13.106 DEBUG:boto:Port: 8001 2024-02-01T10:18:13.106 DEBUG:boto:Params: {} 2024-02-01T10:18:13.106 DEBUG:boto:Token: None 2024-02-01T10:18:13.106 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:13 GMT /vclfcn-12/ 2024-02-01T10:18:13.106 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:EOaK3E32jM46ePAZZBVpXu7grPs= 2024-02-01T10:18:13.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:EOaK3E32jM46ePAZZBVpXu7grPs=', 'Content-Length': '0'} 2024-02-01T10:18:13.111 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000400b5501cfc79f38-0065bb6fe5-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:13.111 DEBUG:boto:b'vclfcn-12:1000false:2024-02-01T10:16:36.670Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:13.112 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-02-01T10:18:13.112 DEBUG:boto:path=/vclfcn-12/_myobj 2024-02-01T10:18:13.112 DEBUG:boto:auth_path=/vclfcn-12/_myobj 2024-02-01T10:18:13.113 DEBUG:boto:path=/vclfcn-12/_myobj?versionId=null 2024-02-01T10:18:13.113 DEBUG:boto:auth_path=/vclfcn-12/_myobj?versionId=null 2024-02-01T10:18:13.113 DEBUG:boto:Method: GET 2024-02-01T10:18:13.113 DEBUG:boto:Path: /vclfcn-12/_myobj?versionId=null 2024-02-01T10:18:13.113 DEBUG:boto:Data: 2024-02-01T10:18:13.113 DEBUG:boto:Headers: {} 2024-02-01T10:18:13.113 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:13.113 DEBUG:boto:Port: 8000 2024-02-01T10:18:13.113 DEBUG:boto:Params: {} 2024-02-01T10:18:13.114 DEBUG:boto:Token: None 2024-02-01T10:18:13.114 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:13 GMT /vclfcn-12/_myobj?versionId=null 2024-02-01T10:18:13.114 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tljTjdNAW9wT0byxL+tG0tFlLJI= 2024-02-01T10:18:13.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tljTjdNAW9wT0byxL+tG0tFlLJI=', 'Content-Length': '0'} 2024-02-01T10:18:13.117 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ea0db552abc3638f-0065bb6fe5-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:13.117 DEBUG:boto:path=/vclfcn-12/_myobj 2024-02-01T10:18:13.117 DEBUG:boto:auth_path=/vclfcn-12/_myobj 2024-02-01T10:18:13.117 DEBUG:boto:path=/vclfcn-12/_myobj?versionId=null 2024-02-01T10:18:13.118 DEBUG:boto:auth_path=/vclfcn-12/_myobj?versionId=null 2024-02-01T10:18:13.118 DEBUG:boto:Method: GET 2024-02-01T10:18:13.118 DEBUG:boto:Path: /vclfcn-12/_myobj?versionId=null 2024-02-01T10:18:13.118 DEBUG:boto:Data: 2024-02-01T10:18:13.118 DEBUG:boto:Headers: {} 2024-02-01T10:18:13.118 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:13.118 DEBUG:boto:Port: 8001 2024-02-01T10:18:13.118 DEBUG:boto:Params: {} 2024-02-01T10:18:13.118 DEBUG:boto:Token: None 2024-02-01T10:18:13.119 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:13 GMT /vclfcn-12/_myobj?versionId=null 2024-02-01T10:18:13.119 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tljTjdNAW9wT0byxL+tG0tFlLJI= 2024-02-01T10:18:13.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tljTjdNAW9wT0byxL+tG0tFlLJI=', 'Content-Length': '0'} 2024-02-01T10:18:13.122 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cecf799273328078-0065bb6fe5-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:13.164 DEBUG:boto:path=/vclfcn-12/_myobj 2024-02-01T10:18:13.164 DEBUG:boto:auth_path=/vclfcn-12/_myobj 2024-02-01T10:18:13.164 DEBUG:boto:path=/vclfcn-12/_myobj?versionId=null 2024-02-01T10:18:13.165 DEBUG:boto:auth_path=/vclfcn-12/_myobj?versionId=null 2024-02-01T10:18:13.165 DEBUG:boto:Method: HEAD 2024-02-01T10:18:13.165 DEBUG:boto:Path: /vclfcn-12/_myobj?versionId=null 2024-02-01T10:18:13.165 DEBUG:boto:Data: 2024-02-01T10:18:13.165 DEBUG:boto:Headers: {} 2024-02-01T10:18:13.165 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:13.165 DEBUG:boto:Port: 8000 2024-02-01T10:18:13.165 DEBUG:boto:Params: {} 2024-02-01T10:18:13.165 DEBUG:boto:Token: None 2024-02-01T10:18:13.166 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:13 GMT /vclfcn-12/_myobj?versionId=null 2024-02-01T10:18:13.166 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zLgWCr8jvrgThPExvnrUhXooZtg= 2024-02-01T10:18:13.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zLgWCr8jvrgThPExvnrUhXooZtg=', 'Content-Length': '0'} 2024-02-01T10:18:13.169 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000003c743950f2f0d98-0065bb6fe5-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:13.169 DEBUG:boto:path=/vclfcn-12/_myobj 2024-02-01T10:18:13.169 DEBUG:boto:auth_path=/vclfcn-12/_myobj 2024-02-01T10:18:13.169 DEBUG:boto:path=/vclfcn-12/_myobj?versionId=null 2024-02-01T10:18:13.169 DEBUG:boto:auth_path=/vclfcn-12/_myobj?versionId=null 2024-02-01T10:18:13.169 DEBUG:boto:Method: HEAD 2024-02-01T10:18:13.169 DEBUG:boto:Path: /vclfcn-12/_myobj?versionId=null 2024-02-01T10:18:13.170 DEBUG:boto:Data: 2024-02-01T10:18:13.170 DEBUG:boto:Headers: {} 2024-02-01T10:18:13.170 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:13.170 DEBUG:boto:Port: 8001 2024-02-01T10:18:13.170 DEBUG:boto:Params: {} 2024-02-01T10:18:13.170 DEBUG:boto:Token: None 2024-02-01T10:18:13.170 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:13 GMT /vclfcn-12/_myobj?versionId=null 2024-02-01T10:18:13.170 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zLgWCr8jvrgThPExvnrUhXooZtg= 2024-02-01T10:18:13.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zLgWCr8jvrgThPExvnrUhXooZtg=', 'Content-Length': '0'} 2024-02-01T10:18:13.173 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006ebdf39eb6944126-0065bb6fe5-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:13.173 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-02-01T10:18:13.174 DEBUG:boto:path=/vclfcn-12/_myobj 2024-02-01T10:18:13.174 DEBUG:boto:auth_path=/vclfcn-12/_myobj 2024-02-01T10:18:13.174 DEBUG:boto:path=/vclfcn-12/_myobj?versionId=null 2024-02-01T10:18:13.174 DEBUG:boto:auth_path=/vclfcn-12/_myobj?versionId=null 2024-02-01T10:18:13.174 DEBUG:boto:Method: GET 2024-02-01T10:18:13.174 DEBUG:boto:Path: /vclfcn-12/_myobj?versionId=null 2024-02-01T10:18:13.174 DEBUG:boto:Data: 2024-02-01T10:18:13.174 DEBUG:boto:Headers: {} 2024-02-01T10:18:13.175 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:13.175 DEBUG:boto:Port: 8000 2024-02-01T10:18:13.175 DEBUG:boto:Params: {} 2024-02-01T10:18:13.175 DEBUG:boto:Token: None 2024-02-01T10:18:13.175 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:13 GMT /vclfcn-12/_myobj?versionId=null 2024-02-01T10:18:13.175 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tljTjdNAW9wT0byxL+tG0tFlLJI= 2024-02-01T10:18:13.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tljTjdNAW9wT0byxL+tG0tFlLJI=', 'Content-Length': '0'} 2024-02-01T10:18:13.178 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001a1853b6fddf3450-0065bb6fe5-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:13.178 DEBUG:boto:path=/vclfcn-12/_myobj 2024-02-01T10:18:13.178 DEBUG:boto:auth_path=/vclfcn-12/_myobj 2024-02-01T10:18:13.179 DEBUG:boto:path=/vclfcn-12/_myobj?versionId=null 2024-02-01T10:18:13.179 DEBUG:boto:auth_path=/vclfcn-12/_myobj?versionId=null 2024-02-01T10:18:13.179 DEBUG:boto:Method: GET 2024-02-01T10:18:13.179 DEBUG:boto:Path: /vclfcn-12/_myobj?versionId=null 2024-02-01T10:18:13.179 DEBUG:boto:Data: 2024-02-01T10:18:13.179 DEBUG:boto:Headers: {} 2024-02-01T10:18:13.179 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:13.179 DEBUG:boto:Port: 8001 2024-02-01T10:18:13.180 DEBUG:boto:Params: {} 2024-02-01T10:18:13.180 DEBUG:boto:Token: None 2024-02-01T10:18:13.180 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:13 GMT /vclfcn-12/_myobj?versionId=null 2024-02-01T10:18:13.180 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tljTjdNAW9wT0byxL+tG0tFlLJI= 2024-02-01T10:18:13.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tljTjdNAW9wT0byxL+tG0tFlLJI=', 'Content-Length': '0'} 2024-02-01T10:18:13.183 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000056950facce1d6e6a-0065bb6fe5-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:13.183 DEBUG:boto:path=/vclfcn-12/ 2024-02-01T10:18:13.184 DEBUG:boto:auth_path=/vclfcn-12/ 2024-02-01T10:18:13.184 DEBUG:boto:path=/vclfcn-12/?prefix=_myobj 2024-02-01T10:18:13.184 DEBUG:boto:auth_path=/vclfcn-12/?prefix=_myobj 2024-02-01T10:18:13.184 DEBUG:boto:Method: GET 2024-02-01T10:18:13.184 DEBUG:boto:Path: /vclfcn-12/?prefix=_myobj 2024-02-01T10:18:13.184 DEBUG:boto:Data: 2024-02-01T10:18:13.184 DEBUG:boto:Headers: {} 2024-02-01T10:18:13.184 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:13.184 DEBUG:boto:Port: 8000 2024-02-01T10:18:13.185 DEBUG:boto:Params: {} 2024-02-01T10:18:13.185 DEBUG:boto:Token: None 2024-02-01T10:18:13.185 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:13 GMT /vclfcn-12/ 2024-02-01T10:18:13.185 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:EOaK3E32jM46ePAZZBVpXu7grPs= 2024-02-01T10:18:13.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:EOaK3E32jM46ePAZZBVpXu7grPs=', 'Content-Length': '0'} 2024-02-01T10:18:13.189 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c188cd5d72a54ae2-0065bb6fe5-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:13.236 DEBUG:boto:b'vclfcn-12_myobj1000false_myobj2024-02-01T10:16:36.640Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:13.237 DEBUG:boto:path=/vclfcn-12/ 2024-02-01T10:18:13.237 DEBUG:boto:auth_path=/vclfcn-12/ 2024-02-01T10:18:13.237 DEBUG:boto:path=/vclfcn-12/?prefix=_myobj 2024-02-01T10:18:13.237 DEBUG:boto:auth_path=/vclfcn-12/?prefix=_myobj 2024-02-01T10:18:13.237 DEBUG:boto:Method: GET 2024-02-01T10:18:13.237 DEBUG:boto:Path: /vclfcn-12/?prefix=_myobj 2024-02-01T10:18:13.237 DEBUG:boto:Data: 2024-02-01T10:18:13.238 DEBUG:boto:Headers: {} 2024-02-01T10:18:13.238 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:13.238 DEBUG:boto:Port: 8001 2024-02-01T10:18:13.238 DEBUG:boto:Params: {} 2024-02-01T10:18:13.238 DEBUG:boto:Token: None 2024-02-01T10:18:13.238 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:13 GMT /vclfcn-12/ 2024-02-01T10:18:13.238 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:EOaK3E32jM46ePAZZBVpXu7grPs= 2024-02-01T10:18:13.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:EOaK3E32jM46ePAZZBVpXu7grPs=', 'Content-Length': '0'} 2024-02-01T10:18:13.243 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084e2bdfdf0c63945-0065bb6fe5-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:13.243 DEBUG:boto:b'vclfcn-12_myobj1000false_myobj2024-02-01T10:16:36.640Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:13.244 DEBUG:boto:path=/vclfcn-12/_myobj 2024-02-01T10:18:13.244 DEBUG:boto:auth_path=/vclfcn-12/_myobj 2024-02-01T10:18:13.244 DEBUG:boto:Method: HEAD 2024-02-01T10:18:13.244 DEBUG:boto:Path: /vclfcn-12/_myobj 2024-02-01T10:18:13.244 DEBUG:boto:Data: 2024-02-01T10:18:13.244 DEBUG:boto:Headers: {} 2024-02-01T10:18:13.244 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:13.245 DEBUG:boto:Port: 8000 2024-02-01T10:18:13.245 DEBUG:boto:Params: {} 2024-02-01T10:18:13.245 DEBUG:boto:Token: None 2024-02-01T10:18:13.245 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:13 GMT /vclfcn-12/_myobj 2024-02-01T10:18:13.245 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gst4jA5Yw8YFasH90YPyaAlu83E= 2024-02-01T10:18:13.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gst4jA5Yw8YFasH90YPyaAlu83E=', 'Content-Length': '0'} 2024-02-01T10:18:13.248 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000032697ed70e676627-0065bb6fe5-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:13.249 DEBUG:boto:path=/vclfcn-12/_myobj 2024-02-01T10:18:13.249 DEBUG:boto:auth_path=/vclfcn-12/_myobj 2024-02-01T10:18:13.249 DEBUG:boto:Method: HEAD 2024-02-01T10:18:13.249 DEBUG:boto:Path: /vclfcn-12/_myobj 2024-02-01T10:18:13.249 DEBUG:boto:Data: 2024-02-01T10:18:13.249 DEBUG:boto:Headers: {} 2024-02-01T10:18:13.249 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:13.249 DEBUG:boto:Port: 8001 2024-02-01T10:18:13.249 DEBUG:boto:Params: {} 2024-02-01T10:18:13.250 DEBUG:boto:Token: None 2024-02-01T10:18:13.250 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:13 GMT /vclfcn-12/_myobj 2024-02-01T10:18:13.250 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gst4jA5Yw8YFasH90YPyaAlu83E= 2024-02-01T10:18:13.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gst4jA5Yw8YFasH90YPyaAlu83E=', 'Content-Length': '0'} 2024-02-01T10:18:13.253 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007398b6630f7b206e-0065bb6fe5-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:13.253 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-02-01T10:18:13.253 DEBUG:boto:path=/vclfcn-12/_myobj 2024-02-01T10:18:13.253 DEBUG:boto:auth_path=/vclfcn-12/_myobj 2024-02-01T10:18:13.253 DEBUG:boto:Method: GET 2024-02-01T10:18:13.253 DEBUG:boto:Path: /vclfcn-12/_myobj 2024-02-01T10:18:13.253 DEBUG:boto:Data: 2024-02-01T10:18:13.253 DEBUG:boto:Headers: {} 2024-02-01T10:18:13.254 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:13.254 DEBUG:boto:Port: 8000 2024-02-01T10:18:13.254 DEBUG:boto:Params: {} 2024-02-01T10:18:13.254 DEBUG:boto:Token: None 2024-02-01T10:18:13.254 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:13 GMT /vclfcn-12/_myobj 2024-02-01T10:18:13.254 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:P0kkeOOk9Egdpo+cJDSburDQvvA= 2024-02-01T10:18:13.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:P0kkeOOk9Egdpo+cJDSburDQvvA=', 'Content-Length': '0'} 2024-02-01T10:18:13.257 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000037854d73432b3a22-0065bb6fe5-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:13.257 DEBUG:boto:path=/vclfcn-12/_myobj 2024-02-01T10:18:13.257 DEBUG:boto:auth_path=/vclfcn-12/_myobj 2024-02-01T10:18:13.258 DEBUG:boto:Method: GET 2024-02-01T10:18:13.258 DEBUG:boto:Path: /vclfcn-12/_myobj 2024-02-01T10:18:13.258 DEBUG:boto:Data: 2024-02-01T10:18:13.258 DEBUG:boto:Headers: {} 2024-02-01T10:18:13.258 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:13.258 DEBUG:boto:Port: 8001 2024-02-01T10:18:13.258 DEBUG:boto:Params: {} 2024-02-01T10:18:13.258 DEBUG:boto:Token: None 2024-02-01T10:18:13.258 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:13 GMT /vclfcn-12/_myobj 2024-02-01T10:18:13.259 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:P0kkeOOk9Egdpo+cJDSburDQvvA= 2024-02-01T10:18:13.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:P0kkeOOk9Egdpo+cJDSburDQvvA=', 'Content-Length': '0'} 2024-02-01T10:18:13.261 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000832deee56c888f4a-0065bb6fe5-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:13.304 DEBUG:boto:path=/vclfcn-12/ 2024-02-01T10:18:13.304 DEBUG:boto:auth_path=/vclfcn-12/ 2024-02-01T10:18:13.304 DEBUG:boto:path=/vclfcn-12/?prefix=_myobj 2024-02-01T10:18:13.305 DEBUG:boto:auth_path=/vclfcn-12/?prefix=_myobj 2024-02-01T10:18:13.305 DEBUG:boto:Method: GET 2024-02-01T10:18:13.305 DEBUG:boto:Path: /vclfcn-12/?prefix=_myobj 2024-02-01T10:18:13.305 DEBUG:boto:Data: 2024-02-01T10:18:13.305 DEBUG:boto:Headers: {} 2024-02-01T10:18:13.305 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:13.305 DEBUG:boto:Port: 8000 2024-02-01T10:18:13.305 DEBUG:boto:Params: {} 2024-02-01T10:18:13.305 DEBUG:boto:Token: None 2024-02-01T10:18:13.306 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:13 GMT /vclfcn-12/ 2024-02-01T10:18:13.306 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:EOaK3E32jM46ePAZZBVpXu7grPs= 2024-02-01T10:18:13.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:EOaK3E32jM46ePAZZBVpXu7grPs=', 'Content-Length': '0'} 2024-02-01T10:18:13.310 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b1e906a9abcedcf-0065bb6fe5-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:13.310 DEBUG:boto:b'vclfcn-12_myobj1000false_myobj2024-02-01T10:16:36.640Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:13.310 DEBUG:boto:path=/vclfcn-12/ 2024-02-01T10:18:13.310 DEBUG:boto:auth_path=/vclfcn-12/ 2024-02-01T10:18:13.311 DEBUG:boto:path=/vclfcn-12/?prefix=_myobj 2024-02-01T10:18:13.311 DEBUG:boto:auth_path=/vclfcn-12/?prefix=_myobj 2024-02-01T10:18:13.311 DEBUG:boto:Method: GET 2024-02-01T10:18:13.311 DEBUG:boto:Path: /vclfcn-12/?prefix=_myobj 2024-02-01T10:18:13.311 DEBUG:boto:Data: 2024-02-01T10:18:13.311 DEBUG:boto:Headers: {} 2024-02-01T10:18:13.311 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:13.311 DEBUG:boto:Port: 8001 2024-02-01T10:18:13.311 DEBUG:boto:Params: {} 2024-02-01T10:18:13.311 DEBUG:boto:Token: None 2024-02-01T10:18:13.311 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:13 GMT /vclfcn-12/ 2024-02-01T10:18:13.312 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:EOaK3E32jM46ePAZZBVpXu7grPs= 2024-02-01T10:18:13.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:EOaK3E32jM46ePAZZBVpXu7grPs=', 'Content-Length': '0'} 2024-02-01T10:18:13.316 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a68b09834a37f82-0065bb6fe5-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:13.316 DEBUG:boto:b'vclfcn-12_myobj1000false_myobj2024-02-01T10:16:36.640Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:13.316 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-02-01T10:18:13.317 DEBUG:boto:path=/vclfcn-12/myobj 2024-02-01T10:18:13.317 DEBUG:boto:auth_path=/vclfcn-12/myobj 2024-02-01T10:18:13.317 DEBUG:boto:path=/vclfcn-12/myobj?versionId=null 2024-02-01T10:18:13.317 DEBUG:boto:auth_path=/vclfcn-12/myobj?versionId=null 2024-02-01T10:18:13.317 DEBUG:boto:Method: GET 2024-02-01T10:18:13.317 DEBUG:boto:Path: /vclfcn-12/myobj?versionId=null 2024-02-01T10:18:13.317 DEBUG:boto:Data: 2024-02-01T10:18:13.317 DEBUG:boto:Headers: {} 2024-02-01T10:18:13.317 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:13.317 DEBUG:boto:Port: 8000 2024-02-01T10:18:13.317 DEBUG:boto:Params: {} 2024-02-01T10:18:13.318 DEBUG:boto:Token: None 2024-02-01T10:18:13.318 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:13 GMT /vclfcn-12/myobj?versionId=null 2024-02-01T10:18:13.318 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jN0mzvfqdEcMolq0SRMMqsAhgP0= 2024-02-01T10:18:13.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jN0mzvfqdEcMolq0SRMMqsAhgP0=', 'Content-Length': '0'} 2024-02-01T10:18:13.321 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c4509870fd513d85-0065bb6fe5-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:13.364 DEBUG:boto:path=/vclfcn-12/myobj 2024-02-01T10:18:13.364 DEBUG:boto:auth_path=/vclfcn-12/myobj 2024-02-01T10:18:13.364 DEBUG:boto:path=/vclfcn-12/myobj?versionId=null 2024-02-01T10:18:13.364 DEBUG:boto:auth_path=/vclfcn-12/myobj?versionId=null 2024-02-01T10:18:13.365 DEBUG:boto:Method: GET 2024-02-01T10:18:13.365 DEBUG:boto:Path: /vclfcn-12/myobj?versionId=null 2024-02-01T10:18:13.365 DEBUG:boto:Data: 2024-02-01T10:18:13.365 DEBUG:boto:Headers: {} 2024-02-01T10:18:13.365 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:13.365 DEBUG:boto:Port: 8001 2024-02-01T10:18:13.365 DEBUG:boto:Params: {} 2024-02-01T10:18:13.365 DEBUG:boto:Token: None 2024-02-01T10:18:13.365 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:13 GMT /vclfcn-12/myobj?versionId=null 2024-02-01T10:18:13.366 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jN0mzvfqdEcMolq0SRMMqsAhgP0= 2024-02-01T10:18:13.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jN0mzvfqdEcMolq0SRMMqsAhgP0=', 'Content-Length': '0'} 2024-02-01T10:18:13.369 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b38f2bacaabeaabb-0065bb6fe5-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:13.369 DEBUG:boto:path=/vclfcn-12/myobj 2024-02-01T10:18:13.369 DEBUG:boto:auth_path=/vclfcn-12/myobj 2024-02-01T10:18:13.369 DEBUG:boto:path=/vclfcn-12/myobj?versionId=null 2024-02-01T10:18:13.369 DEBUG:boto:auth_path=/vclfcn-12/myobj?versionId=null 2024-02-01T10:18:13.370 DEBUG:boto:Method: HEAD 2024-02-01T10:18:13.370 DEBUG:boto:Path: /vclfcn-12/myobj?versionId=null 2024-02-01T10:18:13.370 DEBUG:boto:Data: 2024-02-01T10:18:13.370 DEBUG:boto:Headers: {} 2024-02-01T10:18:13.370 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:13.370 DEBUG:boto:Port: 8000 2024-02-01T10:18:13.370 DEBUG:boto:Params: {} 2024-02-01T10:18:13.370 DEBUG:boto:Token: None 2024-02-01T10:18:13.370 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:13 GMT /vclfcn-12/myobj?versionId=null 2024-02-01T10:18:13.370 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Gxtnlg1IbZ8vk1FGf+upfGOypL4= 2024-02-01T10:18:13.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Gxtnlg1IbZ8vk1FGf+upfGOypL4=', 'Content-Length': '0'} 2024-02-01T10:18:13.373 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cee11f2c800a2d0e-0065bb6fe5-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:13.373 DEBUG:boto:path=/vclfcn-12/myobj 2024-02-01T10:18:13.373 DEBUG:boto:auth_path=/vclfcn-12/myobj 2024-02-01T10:18:13.374 DEBUG:boto:path=/vclfcn-12/myobj?versionId=null 2024-02-01T10:18:13.374 DEBUG:boto:auth_path=/vclfcn-12/myobj?versionId=null 2024-02-01T10:18:13.374 DEBUG:boto:Method: HEAD 2024-02-01T10:18:13.374 DEBUG:boto:Path: /vclfcn-12/myobj?versionId=null 2024-02-01T10:18:13.374 DEBUG:boto:Data: 2024-02-01T10:18:13.374 DEBUG:boto:Headers: {} 2024-02-01T10:18:13.374 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:13.374 DEBUG:boto:Port: 8001 2024-02-01T10:18:13.374 DEBUG:boto:Params: {} 2024-02-01T10:18:13.375 DEBUG:boto:Token: None 2024-02-01T10:18:13.375 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:13 GMT /vclfcn-12/myobj?versionId=null 2024-02-01T10:18:13.375 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Gxtnlg1IbZ8vk1FGf+upfGOypL4= 2024-02-01T10:18:13.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Gxtnlg1IbZ8vk1FGf+upfGOypL4=', 'Content-Length': '0'} 2024-02-01T10:18:13.378 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a02beab4c5cee728-0065bb6fe5-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:13.378 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-02-01T10:18:13.378 DEBUG:boto:path=/vclfcn-12/myobj 2024-02-01T10:18:13.378 DEBUG:boto:auth_path=/vclfcn-12/myobj 2024-02-01T10:18:13.378 DEBUG:boto:path=/vclfcn-12/myobj?versionId=null 2024-02-01T10:18:13.378 DEBUG:boto:auth_path=/vclfcn-12/myobj?versionId=null 2024-02-01T10:18:13.379 DEBUG:boto:Method: GET 2024-02-01T10:18:13.379 DEBUG:boto:Path: /vclfcn-12/myobj?versionId=null 2024-02-01T10:18:13.379 DEBUG:boto:Data: 2024-02-01T10:18:13.379 DEBUG:boto:Headers: {} 2024-02-01T10:18:13.379 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:13.379 DEBUG:boto:Port: 8000 2024-02-01T10:18:13.379 DEBUG:boto:Params: {} 2024-02-01T10:18:13.380 DEBUG:boto:Token: None 2024-02-01T10:18:13.380 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:13 GMT /vclfcn-12/myobj?versionId=null 2024-02-01T10:18:13.380 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jN0mzvfqdEcMolq0SRMMqsAhgP0= 2024-02-01T10:18:13.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jN0mzvfqdEcMolq0SRMMqsAhgP0=', 'Content-Length': '0'} 2024-02-01T10:18:13.383 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fc7410195dcd88f5-0065bb6fe5-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:13.383 DEBUG:boto:path=/vclfcn-12/myobj 2024-02-01T10:18:13.384 DEBUG:boto:auth_path=/vclfcn-12/myobj 2024-02-01T10:18:13.384 DEBUG:boto:path=/vclfcn-12/myobj?versionId=null 2024-02-01T10:18:13.384 DEBUG:boto:auth_path=/vclfcn-12/myobj?versionId=null 2024-02-01T10:18:13.384 DEBUG:boto:Method: GET 2024-02-01T10:18:13.384 DEBUG:boto:Path: /vclfcn-12/myobj?versionId=null 2024-02-01T10:18:13.384 DEBUG:boto:Data: 2024-02-01T10:18:13.384 DEBUG:boto:Headers: {} 2024-02-01T10:18:13.384 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:13.385 DEBUG:boto:Port: 8001 2024-02-01T10:18:13.385 DEBUG:boto:Params: {} 2024-02-01T10:18:13.385 DEBUG:boto:Token: None 2024-02-01T10:18:13.385 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:13 GMT /vclfcn-12/myobj?versionId=null 2024-02-01T10:18:13.385 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jN0mzvfqdEcMolq0SRMMqsAhgP0= 2024-02-01T10:18:13.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jN0mzvfqdEcMolq0SRMMqsAhgP0=', 'Content-Length': '0'} 2024-02-01T10:18:13.388 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000caeb09dcee005183-0065bb6fe5-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:13.432 DEBUG:boto:path=/vclfcn-12/ 2024-02-01T10:18:13.432 DEBUG:boto:auth_path=/vclfcn-12/ 2024-02-01T10:18:13.433 DEBUG:boto:path=/vclfcn-12/?prefix=myobj 2024-02-01T10:18:13.433 DEBUG:boto:auth_path=/vclfcn-12/?prefix=myobj 2024-02-01T10:18:13.433 DEBUG:boto:Method: GET 2024-02-01T10:18:13.433 DEBUG:boto:Path: /vclfcn-12/?prefix=myobj 2024-02-01T10:18:13.433 DEBUG:boto:Data: 2024-02-01T10:18:13.433 DEBUG:boto:Headers: {} 2024-02-01T10:18:13.433 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:13.433 DEBUG:boto:Port: 8000 2024-02-01T10:18:13.433 DEBUG:boto:Params: {} 2024-02-01T10:18:13.433 DEBUG:boto:Token: None 2024-02-01T10:18:13.434 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:13 GMT /vclfcn-12/ 2024-02-01T10:18:13.434 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:EOaK3E32jM46ePAZZBVpXu7grPs= 2024-02-01T10:18:13.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:EOaK3E32jM46ePAZZBVpXu7grPs=', 'Content-Length': '0'} 2024-02-01T10:18:13.439 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017577aab830f91aa-0065bb6fe5-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:13.440 DEBUG:boto:b'vclfcn-12myobj1000falsemyobj2024-02-01T10:16:36.593Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:13.440 DEBUG:boto:path=/vclfcn-12/ 2024-02-01T10:18:13.440 DEBUG:boto:auth_path=/vclfcn-12/ 2024-02-01T10:18:13.440 DEBUG:boto:path=/vclfcn-12/?prefix=myobj 2024-02-01T10:18:13.440 DEBUG:boto:auth_path=/vclfcn-12/?prefix=myobj 2024-02-01T10:18:13.441 DEBUG:boto:Method: GET 2024-02-01T10:18:13.441 DEBUG:boto:Path: /vclfcn-12/?prefix=myobj 2024-02-01T10:18:13.441 DEBUG:boto:Data: 2024-02-01T10:18:13.441 DEBUG:boto:Headers: {} 2024-02-01T10:18:13.441 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:13.441 DEBUG:boto:Port: 8001 2024-02-01T10:18:13.441 DEBUG:boto:Params: {} 2024-02-01T10:18:13.441 DEBUG:boto:Token: None 2024-02-01T10:18:13.441 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:13 GMT /vclfcn-12/ 2024-02-01T10:18:13.442 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:EOaK3E32jM46ePAZZBVpXu7grPs= 2024-02-01T10:18:13.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:EOaK3E32jM46ePAZZBVpXu7grPs=', 'Content-Length': '0'} 2024-02-01T10:18:13.446 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001db56d1b6d46fc92-0065bb6fe5-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:13.446 DEBUG:boto:b'vclfcn-12myobj1000falsemyobj2024-02-01T10:16:36.593Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:13.446 DEBUG:boto:path=/vclfcn-12/myobj 2024-02-01T10:18:13.447 DEBUG:boto:auth_path=/vclfcn-12/myobj 2024-02-01T10:18:13.447 DEBUG:boto:Method: HEAD 2024-02-01T10:18:13.447 DEBUG:boto:Path: /vclfcn-12/myobj 2024-02-01T10:18:13.447 DEBUG:boto:Data: 2024-02-01T10:18:13.447 DEBUG:boto:Headers: {} 2024-02-01T10:18:13.447 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:13.447 DEBUG:boto:Port: 8000 2024-02-01T10:18:13.447 DEBUG:boto:Params: {} 2024-02-01T10:18:13.447 DEBUG:boto:Token: None 2024-02-01T10:18:13.447 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:13 GMT /vclfcn-12/myobj 2024-02-01T10:18:13.448 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Bq2JYUA9o0pfXLNeCnH8F/AWUn0= 2024-02-01T10:18:13.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Bq2JYUA9o0pfXLNeCnH8F/AWUn0=', 'Content-Length': '0'} 2024-02-01T10:18:13.450 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003a8c872d55cc1b01-0065bb6fe5-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:13.451 DEBUG:boto:path=/vclfcn-12/myobj 2024-02-01T10:18:13.451 DEBUG:boto:auth_path=/vclfcn-12/myobj 2024-02-01T10:18:13.451 DEBUG:boto:Method: HEAD 2024-02-01T10:18:13.451 DEBUG:boto:Path: /vclfcn-12/myobj 2024-02-01T10:18:13.451 DEBUG:boto:Data: 2024-02-01T10:18:13.451 DEBUG:boto:Headers: {} 2024-02-01T10:18:13.451 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:13.451 DEBUG:boto:Port: 8001 2024-02-01T10:18:13.451 DEBUG:boto:Params: {} 2024-02-01T10:18:13.451 DEBUG:boto:Token: None 2024-02-01T10:18:13.452 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:13 GMT /vclfcn-12/myobj 2024-02-01T10:18:13.452 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Bq2JYUA9o0pfXLNeCnH8F/AWUn0= 2024-02-01T10:18:13.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Bq2JYUA9o0pfXLNeCnH8F/AWUn0=', 'Content-Length': '0'} 2024-02-01T10:18:13.454 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000055cfe2b62f1c4adf-0065bb6fe5-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:13.455 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-02-01T10:18:13.455 DEBUG:boto:path=/vclfcn-12/myobj 2024-02-01T10:18:13.455 DEBUG:boto:auth_path=/vclfcn-12/myobj 2024-02-01T10:18:13.455 DEBUG:boto:Method: GET 2024-02-01T10:18:13.455 DEBUG:boto:Path: /vclfcn-12/myobj 2024-02-01T10:18:13.455 DEBUG:boto:Data: 2024-02-01T10:18:13.455 DEBUG:boto:Headers: {} 2024-02-01T10:18:13.455 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:13.455 DEBUG:boto:Port: 8000 2024-02-01T10:18:13.456 DEBUG:boto:Params: {} 2024-02-01T10:18:13.456 DEBUG:boto:Token: None 2024-02-01T10:18:13.456 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:13 GMT /vclfcn-12/myobj 2024-02-01T10:18:13.456 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RPtZWXRLbrzB+nTTY++MOYUjsv4= 2024-02-01T10:18:13.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RPtZWXRLbrzB+nTTY++MOYUjsv4=', 'Content-Length': '0'} 2024-02-01T10:18:13.459 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000544a73cef6c9c3e4-0065bb6fe5-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:13.500 DEBUG:boto:path=/vclfcn-12/myobj 2024-02-01T10:18:13.500 DEBUG:boto:auth_path=/vclfcn-12/myobj 2024-02-01T10:18:13.500 DEBUG:boto:Method: GET 2024-02-01T10:18:13.501 DEBUG:boto:Path: /vclfcn-12/myobj 2024-02-01T10:18:13.501 DEBUG:boto:Data: 2024-02-01T10:18:13.501 DEBUG:boto:Headers: {} 2024-02-01T10:18:13.501 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:13.501 DEBUG:boto:Port: 8001 2024-02-01T10:18:13.501 DEBUG:boto:Params: {} 2024-02-01T10:18:13.501 DEBUG:boto:Token: None 2024-02-01T10:18:13.501 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:13 GMT /vclfcn-12/myobj 2024-02-01T10:18:13.501 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RPtZWXRLbrzB+nTTY++MOYUjsv4= 2024-02-01T10:18:13.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RPtZWXRLbrzB+nTTY++MOYUjsv4=', 'Content-Length': '0'} 2024-02-01T10:18:13.504 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:16:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003cf93a4c28e0c174-0065bb6fe5-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:13.505 DEBUG:boto:path=/vclfcn-12/ 2024-02-01T10:18:13.505 DEBUG:boto:auth_path=/vclfcn-12/ 2024-02-01T10:18:13.505 DEBUG:boto:path=/vclfcn-12/?prefix=myobj 2024-02-01T10:18:13.505 DEBUG:boto:auth_path=/vclfcn-12/?prefix=myobj 2024-02-01T10:18:13.505 DEBUG:boto:Method: GET 2024-02-01T10:18:13.505 DEBUG:boto:Path: /vclfcn-12/?prefix=myobj 2024-02-01T10:18:13.505 DEBUG:boto:Data: 2024-02-01T10:18:13.505 DEBUG:boto:Headers: {} 2024-02-01T10:18:13.505 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:13.506 DEBUG:boto:Port: 8000 2024-02-01T10:18:13.506 DEBUG:boto:Params: {} 2024-02-01T10:18:13.506 DEBUG:boto:Token: None 2024-02-01T10:18:13.506 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:13 GMT /vclfcn-12/ 2024-02-01T10:18:13.506 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:EOaK3E32jM46ePAZZBVpXu7grPs= 2024-02-01T10:18:13.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:EOaK3E32jM46ePAZZBVpXu7grPs=', 'Content-Length': '0'} 2024-02-01T10:18:13.510 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000299567ecf95eff75-0065bb6fe5-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:13.510 DEBUG:boto:b'vclfcn-12myobj1000falsemyobj2024-02-01T10:16:36.593Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:13.511 DEBUG:boto:path=/vclfcn-12/ 2024-02-01T10:18:13.511 DEBUG:boto:auth_path=/vclfcn-12/ 2024-02-01T10:18:13.511 DEBUG:boto:path=/vclfcn-12/?prefix=myobj 2024-02-01T10:18:13.511 DEBUG:boto:auth_path=/vclfcn-12/?prefix=myobj 2024-02-01T10:18:13.511 DEBUG:boto:Method: GET 2024-02-01T10:18:13.511 DEBUG:boto:Path: /vclfcn-12/?prefix=myobj 2024-02-01T10:18:13.511 DEBUG:boto:Data: 2024-02-01T10:18:13.511 DEBUG:boto:Headers: {} 2024-02-01T10:18:13.511 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:13.512 DEBUG:boto:Port: 8001 2024-02-01T10:18:13.512 DEBUG:boto:Params: {} 2024-02-01T10:18:13.512 DEBUG:boto:Token: None 2024-02-01T10:18:13.512 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:13 GMT /vclfcn-12/ 2024-02-01T10:18:13.512 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:EOaK3E32jM46ePAZZBVpXu7grPs= 2024-02-01T10:18:13.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:EOaK3E32jM46ePAZZBVpXu7grPs=', 'Content-Length': '0'} 2024-02-01T10:18:13.516 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd7880436e7a6d8c-0065bb6fe5-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:13.560 DEBUG:boto:b'vclfcn-12myobj1000falsemyobj2024-02-01T10:16:36.593Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:13.561 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-12 zones={test-zone2, test-zone3} 2024-02-01T10:18:13.562 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_object_sync ... ok 2024-02-01T10:18:13.563 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=vclfcn-13 2024-02-01T10:18:13.564 DEBUG:boto:path=/vclfcn-13/ 2024-02-01T10:18:13.564 DEBUG:boto:auth_path=/vclfcn-13/ 2024-02-01T10:18:13.564 DEBUG:boto:Method: PUT 2024-02-01T10:18:13.564 DEBUG:boto:Path: /vclfcn-13/ 2024-02-01T10:18:13.564 DEBUG:boto:Data: 2024-02-01T10:18:13.564 DEBUG:boto:Headers: {} 2024-02-01T10:18:13.564 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:13.564 DEBUG:boto:Port: 8000 2024-02-01T10:18:13.564 DEBUG:boto:Params: {} 2024-02-01T10:18:13.567 DEBUG:boto:Token: None 2024-02-01T10:18:13.568 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:18:13 GMT /vclfcn-13/ 2024-02-01T10:18:13.568 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/mSAPVOF2hsXFX11PqRW2GZW67Q= 2024-02-01T10:18:13.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/mSAPVOF2hsXFX11PqRW2GZW67Q=', 'Content-Length': '0'} 2024-02-01T10:18:13.601 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000007e672f4b570a1ed-0065bb6fe5-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:13.601 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=vclfcn-14 2024-02-01T10:18:13.601 DEBUG:boto:path=/vclfcn-14/ 2024-02-01T10:18:13.602 DEBUG:boto:auth_path=/vclfcn-14/ 2024-02-01T10:18:13.602 DEBUG:boto:Method: PUT 2024-02-01T10:18:13.602 DEBUG:boto:Path: /vclfcn-14/ 2024-02-01T10:18:13.602 DEBUG:boto:Data: 2024-02-01T10:18:13.602 DEBUG:boto:Headers: {} 2024-02-01T10:18:13.602 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:13.602 DEBUG:boto:Port: 8000 2024-02-01T10:18:13.602 DEBUG:boto:Params: {} 2024-02-01T10:18:13.602 DEBUG:boto:Token: None 2024-02-01T10:18:13.603 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:18:13 GMT /vclfcn-14/ 2024-02-01T10:18:13.603 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:dodffDwouhafbdI1R++Z96Wcufg= 2024-02-01T10:18:13.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dodffDwouhafbdI1R++Z96Wcufg=', 'Content-Length': '0'} 2024-02-01T10:18:13.660 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000048789db8ab9451c3-0065bb6fe5-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:13.660 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=vclfcn-15 2024-02-01T10:18:13.661 DEBUG:boto:path=/vclfcn-15/ 2024-02-01T10:18:13.661 DEBUG:boto:auth_path=/vclfcn-15/ 2024-02-01T10:18:13.661 DEBUG:boto:Method: PUT 2024-02-01T10:18:13.661 DEBUG:boto:Path: /vclfcn-15/ 2024-02-01T10:18:13.661 DEBUG:boto:Data: 2024-02-01T10:18:13.661 DEBUG:boto:Headers: {} 2024-02-01T10:18:13.661 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:13.661 DEBUG:boto:Port: 8001 2024-02-01T10:18:13.661 DEBUG:boto:Params: {} 2024-02-01T10:18:13.661 DEBUG:boto:Token: None 2024-02-01T10:18:13.662 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:18:13 GMT /vclfcn-15/ 2024-02-01T10:18:13.662 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5ttgO+VwTtPR4/eFpcXDffy2Qfo= 2024-02-01T10:18:13.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5ttgO+VwTtPR4/eFpcXDffy2Qfo=', 'Content-Length': '0'} 2024-02-01T10:18:13.727 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007d4097cb425acc0b-0065bb6fe5-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:13.727 DEBUG:boto:path=/vclfcn-13/ 2024-02-01T10:18:13.727 DEBUG:boto:auth_path=/vclfcn-13/ 2024-02-01T10:18:13.727 DEBUG:boto:Method: HEAD 2024-02-01T10:18:13.727 DEBUG:boto:Path: /vclfcn-13/ 2024-02-01T10:18:13.727 DEBUG:boto:Data: 2024-02-01T10:18:13.727 DEBUG:boto:Headers: {} 2024-02-01T10:18:13.728 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:13.728 DEBUG:boto:Port: 8000 2024-02-01T10:18:13.728 DEBUG:boto:Params: {} 2024-02-01T10:18:13.728 DEBUG:boto:Token: None 2024-02-01T10:18:13.728 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:13 GMT /vclfcn-13/ 2024-02-01T10:18:13.728 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Oe8JAtCK/7CG4hufrMtuxeCch8M= 2024-02-01T10:18:13.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Oe8JAtCK/7CG4hufrMtuxeCch8M=', 'Content-Length': '0'} 2024-02-01T10:18:13.732 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001dbb56f2f5a5db63-0065bb6fe5-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:13.733 DEBUG:boto:path=/vclfcn-13/myobj 2024-02-01T10:18:13.733 DEBUG:boto:auth_path=/vclfcn-13/myobj 2024-02-01T10:18:13.733 DEBUG:boto:Method: PUT 2024-02-01T10:18:13.733 DEBUG:boto:Path: /vclfcn-13/myobj 2024-02-01T10:18:13.733 DEBUG:boto:Data: 2024-02-01T10:18:13.733 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-02-01T10:18:13.733 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:13.733 DEBUG:boto:Port: 8000 2024-02-01T10:18:13.734 DEBUG:boto:Params: {} 2024-02-01T10:18:13.734 DEBUG:boto:Token: None 2024-02-01T10:18:13.734 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 01 Feb 2024 10:18:13 GMT /vclfcn-13/myobj 2024-02-01T10:18:13.734 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:3qLT9mljqmTDPR0O8UPHqkytkk8= 2024-02-01T10:18:13.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:18:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:3qLT9mljqmTDPR0O8UPHqkytkk8='} 2024-02-01T10:18:13.780 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000091c74da0a057f2f7-0065bb6fe5-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:13.780 DEBUG:boto:path=/vclfcn-14/ 2024-02-01T10:18:13.781 DEBUG:boto:auth_path=/vclfcn-14/ 2024-02-01T10:18:13.781 DEBUG:boto:Method: HEAD 2024-02-01T10:18:13.781 DEBUG:boto:Path: /vclfcn-14/ 2024-02-01T10:18:13.781 DEBUG:boto:Data: 2024-02-01T10:18:13.781 DEBUG:boto:Headers: {} 2024-02-01T10:18:13.781 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:13.781 DEBUG:boto:Port: 8000 2024-02-01T10:18:13.781 DEBUG:boto:Params: {} 2024-02-01T10:18:13.781 DEBUG:boto:Token: None 2024-02-01T10:18:13.782 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:13 GMT /vclfcn-14/ 2024-02-01T10:18:13.782 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:4CagLAUtay3wlfKtaMZFvK5Fmhc= 2024-02-01T10:18:13.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:4CagLAUtay3wlfKtaMZFvK5Fmhc=', 'Content-Length': '0'} 2024-02-01T10:18:13.786 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000080082813aedfdf01-0065bb6fe5-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:13.787 DEBUG:boto:path=/vclfcn-14/myobj 2024-02-01T10:18:13.787 DEBUG:boto:auth_path=/vclfcn-14/myobj 2024-02-01T10:18:13.787 DEBUG:boto:Method: PUT 2024-02-01T10:18:13.787 DEBUG:boto:Path: /vclfcn-14/myobj 2024-02-01T10:18:13.787 DEBUG:boto:Data: 2024-02-01T10:18:13.787 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-02-01T10:18:13.787 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:13.787 DEBUG:boto:Port: 8000 2024-02-01T10:18:13.788 DEBUG:boto:Params: {} 2024-02-01T10:18:13.788 DEBUG:boto:Token: None 2024-02-01T10:18:13.788 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 01 Feb 2024 10:18:13 GMT /vclfcn-14/myobj 2024-02-01T10:18:13.788 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cT63DZCTxD+z97bOk/qheYIYSTk= 2024-02-01T10:18:13.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:18:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cT63DZCTxD+z97bOk/qheYIYSTk='} 2024-02-01T10:18:13.810 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000a6746bf5b0ebe2d-0065bb6fe5-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:13.811 DEBUG:boto:path=/vclfcn-15/ 2024-02-01T10:18:13.811 DEBUG:boto:auth_path=/vclfcn-15/ 2024-02-01T10:18:13.811 DEBUG:boto:Method: HEAD 2024-02-01T10:18:13.811 DEBUG:boto:Path: /vclfcn-15/ 2024-02-01T10:18:13.811 DEBUG:boto:Data: 2024-02-01T10:18:13.811 DEBUG:boto:Headers: {} 2024-02-01T10:18:13.811 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:13.811 DEBUG:boto:Port: 8001 2024-02-01T10:18:13.811 DEBUG:boto:Params: {} 2024-02-01T10:18:13.812 DEBUG:boto:Token: None 2024-02-01T10:18:13.812 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:13 GMT /vclfcn-15/ 2024-02-01T10:18:13.812 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:MkWBLYNS8XvgTrSqJ04nwCst4gw= 2024-02-01T10:18:13.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:MkWBLYNS8XvgTrSqJ04nwCst4gw=', 'Content-Length': '0'} 2024-02-01T10:18:13.815 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009e8f09ce075fef60-0065bb6fe5-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:13.815 DEBUG:boto:path=/vclfcn-15/myobj 2024-02-01T10:18:13.815 DEBUG:boto:auth_path=/vclfcn-15/myobj 2024-02-01T10:18:13.815 DEBUG:boto:Method: PUT 2024-02-01T10:18:13.815 DEBUG:boto:Path: /vclfcn-15/myobj 2024-02-01T10:18:13.815 DEBUG:boto:Data: 2024-02-01T10:18:13.816 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-02-01T10:18:13.816 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:13.816 DEBUG:boto:Port: 8001 2024-02-01T10:18:13.816 DEBUG:boto:Params: {} 2024-02-01T10:18:13.816 DEBUG:boto:Token: None 2024-02-01T10:18:13.816 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 01 Feb 2024 10:18:13 GMT /vclfcn-15/myobj 2024-02-01T10:18:13.816 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:LKnOKsvU5QF6h4D/MF6DJ0rXpsg= 2024-02-01T10:18:13.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:18:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:LKnOKsvU5QF6h4D/MF6DJ0rXpsg='} 2024-02-01T10:18:13.827 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d33a95f689afe5e8-0065bb6fe5-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:13.828 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:18:13.828 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:18:13.828 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:18:13.866 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:18:13.866 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:18:13.922 INFO:teuthology.orchestra.run.smithi148.stderr:No --period given, using current period=5dc4efbd-eccd-47a8-9471-ae4a209ec978 2024-02-01T10:18:13.945 INFO:teuthology.orchestra.run.smithi148.stdout:[{"marker":"1_1706782453.132619_30.1","last_update":"2024-02-01T10:14:13.132619Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1706782693.703781_64.1","last_update":"2024-02-01T10:18:13.703781Z"},{"marker":"1_1706782595.670952_121.1","last_update":"2024-02-01T10:16:35.670952Z"}] 2024-02-01T10:18:13.945 DEBUG:rgw_multi.tests:master meta markers={0: '1_1706782453.132619_30.1', 1: '', 2: '1_1706782693.703781_64.1', 3: '1_1706782595.670952_121.1'} 2024-02-01T10:18:13.945 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-02-01T10:18:13.946 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:18:13.946 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:18:13.946 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:18:13.989 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:18:13.989 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:18:14.042 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:18:14.043+0000 7fd8ed87d980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:18:14.043 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:18:14.043+0000 7fd8ed87d980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:18:14.066 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706782693.629469_58.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:18:13.629469Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:18:14Z"} 2024-02-01T10:18:14.067 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706782693.629469_58.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:18:13.629469Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:18:14Z"} 2024-02-01T10:18:14.067 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706782693.629469_58.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:18:13.629469Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:18:14.067 DEBUG:rgw_multi.tests:log_status={0: '1_1706782453.132619_30.1', 1: '', 2: '1_1706782693.703781_64.1', 3: '1_1706782595.670952_121.1'} 2024-02-01T10:18:14.067 DEBUG:rgw_multi.tests:sync_status={0: '1_1706782453.132619_30.1', 1: '', 2: '1_1706782693.629469_58.1', 3: '1_1706782595.670952_121.1'} 2024-02-01T10:18:14.067 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1706782693.703781_64.1 target=1_1706782693.629469_58.1 2024-02-01T10:18:19.068 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:18:19.069 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:18:19.069 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:18:19.111 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:18:19.111 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:18:19.165 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:18:19.163+0000 7f9f0e733980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:18:19.166 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:18:19.167+0000 7f9f0e733980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:18:19.188 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706782693.703781_64.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:18:13.703781Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:18:19Z"} 2024-02-01T10:18:19.189 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706782693.703781_64.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:18:13.703781Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:18:19Z"} 2024-02-01T10:18:19.189 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706782693.703781_64.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:18:13.703781Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:18:19.189 DEBUG:rgw_multi.tests:log_status={0: '1_1706782453.132619_30.1', 1: '', 2: '1_1706782693.703781_64.1', 3: '1_1706782595.670952_121.1'} 2024-02-01T10:18:19.189 DEBUG:rgw_multi.tests:sync_status={0: '1_1706782453.132619_30.1', 1: '', 2: '1_1706782693.703781_64.1', 3: '1_1706782595.670952_121.1'} 2024-02-01T10:18:19.189 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-02-01T10:18:19.189 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-02-01T10:18:19.190 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:18:19.190 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:18:19.190 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:18:19.238 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:18:19.238 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:18:19.293 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:18:19.291+0000 7f7417bfe980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:18:19.294 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:18:19.291+0000 7f7417bfe980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:18:19.316 INFO:teuthology.orchestra.run.smithi148.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706782693.703781_64.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:18:13.703781Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:18:19Z"} 2024-02-01T10:18:19.316 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706782693.703781_64.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:18:13.703781Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:18:19Z"} 2024-02-01T10:18:19.316 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706782693.703781_64.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:18:13.703781Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:18:19.316 DEBUG:rgw_multi.tests:log_status={0: '1_1706782453.132619_30.1', 1: '', 2: '1_1706782693.703781_64.1', 3: '1_1706782595.670952_121.1'} 2024-02-01T10:18:19.317 DEBUG:rgw_multi.tests:sync_status={0: '1_1706782453.132619_30.1', 1: '', 2: '1_1706782693.703781_64.1', 3: '1_1706782595.670952_121.1'} 2024-02-01T10:18:19.317 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-02-01T10:18:19.317 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:18:19.317 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:18:19.317 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket vclfcn-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-02-01T10:18:19.366 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:18:19.366 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:18:19.430 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:18:19.431+0000 7f4264301980 1 waiting to reach incremental sync.. 2024-02-01T10:18:24.431 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:18:24.431+0000 7f4264301980 1 waiting to reach incremental sync.. 2024-02-01T10:18:29.435 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:18:29.435+0000 7f4264301980 1 bucket sync caught up with source: 2024-02-01T10:18:29.435 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, 00000000001.13.6, , , , , , , , , ] 2024-02-01T10:18:29.435 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, 00000000001.13.6, , , , , , , , , ] 2024-02-01T10:18:29.435 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:18:29.435+0000 7f4264301980 0 bucket checkpoint complete 2024-02-01T10:18:29.469 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-13 zones={test-zone2, test-zone1} 2024-02-01T10:18:29.470 DEBUG:boto:path=/vclfcn-13/ 2024-02-01T10:18:29.470 DEBUG:boto:auth_path=/vclfcn-13/ 2024-02-01T10:18:29.470 DEBUG:boto:Method: HEAD 2024-02-01T10:18:29.470 DEBUG:boto:Path: /vclfcn-13/ 2024-02-01T10:18:29.470 DEBUG:boto:Data: 2024-02-01T10:18:29.470 DEBUG:boto:Headers: {} 2024-02-01T10:18:29.470 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:29.470 DEBUG:boto:Port: 8000 2024-02-01T10:18:29.470 DEBUG:boto:Params: {} 2024-02-01T10:18:29.470 DEBUG:boto:Token: None 2024-02-01T10:18:29.471 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:29 GMT /vclfcn-13/ 2024-02-01T10:18:29.471 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:EnCR5QcARGXaZD57QzoyxTa/Sd0= 2024-02-01T10:18:29.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:29 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:EnCR5QcARGXaZD57QzoyxTa/Sd0=', 'Content-Length': '0'} 2024-02-01T10:18:29.474 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007966fa51c9de1679-0065bb6ff5-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:18:29 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:29.474 DEBUG:boto:path=/vclfcn-13/ 2024-02-01T10:18:29.474 DEBUG:boto:auth_path=/vclfcn-13/ 2024-02-01T10:18:29.474 DEBUG:boto:Method: HEAD 2024-02-01T10:18:29.474 DEBUG:boto:Path: /vclfcn-13/ 2024-02-01T10:18:29.475 DEBUG:boto:Data: 2024-02-01T10:18:29.475 DEBUG:boto:Headers: {} 2024-02-01T10:18:29.475 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:29.475 DEBUG:boto:Port: 8000 2024-02-01T10:18:29.475 DEBUG:boto:Params: {} 2024-02-01T10:18:29.475 DEBUG:boto:Token: None 2024-02-01T10:18:29.475 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:29 GMT /vclfcn-13/ 2024-02-01T10:18:29.475 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:EnCR5QcARGXaZD57QzoyxTa/Sd0= 2024-02-01T10:18:29.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:29 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:EnCR5QcARGXaZD57QzoyxTa/Sd0=', 'Content-Length': '0'} 2024-02-01T10:18:29.480 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f06b006b4028ea51-0065bb6ff5-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:18:29 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:29.480 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:18:29.480 DEBUG:boto:path=/vclfcn-13/ 2024-02-01T10:18:29.480 DEBUG:boto:auth_path=/vclfcn-13/ 2024-02-01T10:18:29.480 DEBUG:boto:path=/vclfcn-13/?versions&max-keys=999 2024-02-01T10:18:29.480 DEBUG:boto:auth_path=/vclfcn-13/?versions&max-keys=999 2024-02-01T10:18:29.480 DEBUG:boto:Method: GET 2024-02-01T10:18:29.481 DEBUG:boto:Path: /vclfcn-13/?versions&max-keys=999 2024-02-01T10:18:29.481 DEBUG:boto:Data: 2024-02-01T10:18:29.481 DEBUG:boto:Headers: {} 2024-02-01T10:18:29.481 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:29.481 DEBUG:boto:Port: 8000 2024-02-01T10:18:29.481 DEBUG:boto:Params: {} 2024-02-01T10:18:29.481 DEBUG:boto:Token: None 2024-02-01T10:18:29.481 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:29 GMT /vclfcn-13/?versions 2024-02-01T10:18:29.481 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:N1ZykG7l7uUvM5MbwWoRwAI8xe0= 2024-02-01T10:18:29.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:29 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:N1ZykG7l7uUvM5MbwWoRwAI8xe0=', 'Content-Length': '0'} 2024-02-01T10:18:29.484 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024076e6db7a89c67-0065bb6ff5-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:29 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:29.528 DEBUG:boto:b'vclfcn-13999falsemyobjnulltrue2024-02-01T10:18:13.740Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:29.529 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-02-01T10:18:29.529 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:18:29.529 DEBUG:boto:path=/vclfcn-13/ 2024-02-01T10:18:29.529 DEBUG:boto:auth_path=/vclfcn-13/ 2024-02-01T10:18:29.529 DEBUG:boto:path=/vclfcn-13/?versions&max-keys=999 2024-02-01T10:18:29.529 DEBUG:boto:auth_path=/vclfcn-13/?versions&max-keys=999 2024-02-01T10:18:29.529 DEBUG:boto:Method: GET 2024-02-01T10:18:29.529 DEBUG:boto:Path: /vclfcn-13/?versions&max-keys=999 2024-02-01T10:18:29.529 DEBUG:boto:Data: 2024-02-01T10:18:29.530 DEBUG:boto:Headers: {} 2024-02-01T10:18:29.530 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:29.530 DEBUG:boto:Port: 8000 2024-02-01T10:18:29.530 DEBUG:boto:Params: {} 2024-02-01T10:18:29.530 DEBUG:boto:Token: None 2024-02-01T10:18:29.530 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:29 GMT /vclfcn-13/?versions 2024-02-01T10:18:29.530 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:N1ZykG7l7uUvM5MbwWoRwAI8xe0= 2024-02-01T10:18:29.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:29 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:N1ZykG7l7uUvM5MbwWoRwAI8xe0=', 'Content-Length': '0'} 2024-02-01T10:18:29.535 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2dd2abf00d17094-0065bb6ff5-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:29 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:29.535 DEBUG:boto:b'vclfcn-13999falsemyobjnulltrue2024-02-01T10:18:13.740Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:29.535 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-02-01T10:18:29.535 DEBUG:boto:path=/vclfcn-13/ 2024-02-01T10:18:29.535 DEBUG:boto:auth_path=/vclfcn-13/ 2024-02-01T10:18:29.535 DEBUG:boto:path=/vclfcn-13/?versions&max-keys=999 2024-02-01T10:18:29.536 DEBUG:boto:auth_path=/vclfcn-13/?versions&max-keys=999 2024-02-01T10:18:29.536 DEBUG:boto:Method: GET 2024-02-01T10:18:29.536 DEBUG:boto:Path: /vclfcn-13/?versions&max-keys=999 2024-02-01T10:18:29.536 DEBUG:boto:Data: 2024-02-01T10:18:29.536 DEBUG:boto:Headers: {} 2024-02-01T10:18:29.536 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:29.536 DEBUG:boto:Port: 8000 2024-02-01T10:18:29.536 DEBUG:boto:Params: {} 2024-02-01T10:18:29.536 DEBUG:boto:Token: None 2024-02-01T10:18:29.536 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:29 GMT /vclfcn-13/?versions 2024-02-01T10:18:29.537 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:N1ZykG7l7uUvM5MbwWoRwAI8xe0= 2024-02-01T10:18:29.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:29 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:N1ZykG7l7uUvM5MbwWoRwAI8xe0=', 'Content-Length': '0'} 2024-02-01T10:18:29.541 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f4f6063eb45cd24-0065bb6ff5-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:29 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:29.541 DEBUG:boto:b'vclfcn-13999falsemyobjnulltrue2024-02-01T10:18:13.740Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:29.541 DEBUG:boto:path=/vclfcn-13/ 2024-02-01T10:18:29.541 DEBUG:boto:auth_path=/vclfcn-13/ 2024-02-01T10:18:29.541 DEBUG:boto:path=/vclfcn-13/?versions&max-keys=999 2024-02-01T10:18:29.541 DEBUG:boto:auth_path=/vclfcn-13/?versions&max-keys=999 2024-02-01T10:18:29.542 DEBUG:boto:Method: GET 2024-02-01T10:18:29.542 DEBUG:boto:Path: /vclfcn-13/?versions&max-keys=999 2024-02-01T10:18:29.542 DEBUG:boto:Data: 2024-02-01T10:18:29.542 DEBUG:boto:Headers: {} 2024-02-01T10:18:29.542 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:29.542 DEBUG:boto:Port: 8000 2024-02-01T10:18:29.542 DEBUG:boto:Params: {} 2024-02-01T10:18:29.542 DEBUG:boto:Token: None 2024-02-01T10:18:29.542 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:29 GMT /vclfcn-13/?versions 2024-02-01T10:18:29.542 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:N1ZykG7l7uUvM5MbwWoRwAI8xe0= 2024-02-01T10:18:29.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:29 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:N1ZykG7l7uUvM5MbwWoRwAI8xe0=', 'Content-Length': '0'} 2024-02-01T10:18:29.547 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad33785ce77b20bb-0065bb6ff5-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:29 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:29.547 DEBUG:boto:b'vclfcn-13999falsemyobjnulltrue2024-02-01T10:18:13.740Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:29.547 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-02-01T10:18:29.547 DEBUG:boto:path=/vclfcn-13/myobj 2024-02-01T10:18:29.547 DEBUG:boto:auth_path=/vclfcn-13/myobj 2024-02-01T10:18:29.547 DEBUG:boto:path=/vclfcn-13/myobj?versionId=null 2024-02-01T10:18:29.547 DEBUG:boto:auth_path=/vclfcn-13/myobj?versionId=null 2024-02-01T10:18:29.548 DEBUG:boto:Method: GET 2024-02-01T10:18:29.548 DEBUG:boto:Path: /vclfcn-13/myobj?versionId=null 2024-02-01T10:18:29.548 DEBUG:boto:Data: 2024-02-01T10:18:29.548 DEBUG:boto:Headers: {} 2024-02-01T10:18:29.548 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:29.548 DEBUG:boto:Port: 8000 2024-02-01T10:18:29.548 DEBUG:boto:Params: {} 2024-02-01T10:18:29.548 DEBUG:boto:Token: None 2024-02-01T10:18:29.548 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:29 GMT /vclfcn-13/myobj?versionId=null 2024-02-01T10:18:29.548 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GvrPmLq5LytxzeFYaQl59je++PQ= 2024-02-01T10:18:29.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:29 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GvrPmLq5LytxzeFYaQl59je++PQ=', 'Content-Length': '0'} 2024-02-01T10:18:29.552 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d88336473ab7b5be-0065bb6ff5-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:29 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:29.552 DEBUG:boto:path=/vclfcn-13/myobj 2024-02-01T10:18:29.552 DEBUG:boto:auth_path=/vclfcn-13/myobj 2024-02-01T10:18:29.552 DEBUG:boto:path=/vclfcn-13/myobj?versionId=null 2024-02-01T10:18:29.552 DEBUG:boto:auth_path=/vclfcn-13/myobj?versionId=null 2024-02-01T10:18:29.552 DEBUG:boto:Method: GET 2024-02-01T10:18:29.552 DEBUG:boto:Path: /vclfcn-13/myobj?versionId=null 2024-02-01T10:18:29.553 DEBUG:boto:Data: 2024-02-01T10:18:29.553 DEBUG:boto:Headers: {} 2024-02-01T10:18:29.553 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:29.553 DEBUG:boto:Port: 8000 2024-02-01T10:18:29.553 DEBUG:boto:Params: {} 2024-02-01T10:18:29.553 DEBUG:boto:Token: None 2024-02-01T10:18:29.553 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:29 GMT /vclfcn-13/myobj?versionId=null 2024-02-01T10:18:29.553 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GvrPmLq5LytxzeFYaQl59je++PQ= 2024-02-01T10:18:29.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:29 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GvrPmLq5LytxzeFYaQl59je++PQ=', 'Content-Length': '0'} 2024-02-01T10:18:29.556 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000afdf97fa4169df80-0065bb6ff5-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:29 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:29.557 DEBUG:boto:path=/vclfcn-13/myobj 2024-02-01T10:18:29.557 DEBUG:boto:auth_path=/vclfcn-13/myobj 2024-02-01T10:18:29.557 DEBUG:boto:path=/vclfcn-13/myobj?versionId=null 2024-02-01T10:18:29.557 DEBUG:boto:auth_path=/vclfcn-13/myobj?versionId=null 2024-02-01T10:18:29.557 DEBUG:boto:Method: HEAD 2024-02-01T10:18:29.557 DEBUG:boto:Path: /vclfcn-13/myobj?versionId=null 2024-02-01T10:18:29.557 DEBUG:boto:Data: 2024-02-01T10:18:29.557 DEBUG:boto:Headers: {} 2024-02-01T10:18:29.557 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:29.557 DEBUG:boto:Port: 8000 2024-02-01T10:18:29.557 DEBUG:boto:Params: {} 2024-02-01T10:18:29.558 DEBUG:boto:Token: None 2024-02-01T10:18:29.558 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:29 GMT /vclfcn-13/myobj?versionId=null 2024-02-01T10:18:29.558 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9A7UHlbZcUvsNPJ794dbnX+Ufdc= 2024-02-01T10:18:29.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:29 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9A7UHlbZcUvsNPJ794dbnX+Ufdc=', 'Content-Length': '0'} 2024-02-01T10:18:29.561 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000af78e28a14e0db11-0065bb6ff5-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:29 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:29.561 DEBUG:boto:path=/vclfcn-13/myobj 2024-02-01T10:18:29.561 DEBUG:boto:auth_path=/vclfcn-13/myobj 2024-02-01T10:18:29.561 DEBUG:boto:path=/vclfcn-13/myobj?versionId=null 2024-02-01T10:18:29.561 DEBUG:boto:auth_path=/vclfcn-13/myobj?versionId=null 2024-02-01T10:18:29.561 DEBUG:boto:Method: HEAD 2024-02-01T10:18:29.561 DEBUG:boto:Path: /vclfcn-13/myobj?versionId=null 2024-02-01T10:18:29.562 DEBUG:boto:Data: 2024-02-01T10:18:29.562 DEBUG:boto:Headers: {} 2024-02-01T10:18:29.562 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:29.562 DEBUG:boto:Port: 8000 2024-02-01T10:18:29.562 DEBUG:boto:Params: {} 2024-02-01T10:18:29.562 DEBUG:boto:Token: None 2024-02-01T10:18:29.562 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:29 GMT /vclfcn-13/myobj?versionId=null 2024-02-01T10:18:29.562 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9A7UHlbZcUvsNPJ794dbnX+Ufdc= 2024-02-01T10:18:29.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:29 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9A7UHlbZcUvsNPJ794dbnX+Ufdc=', 'Content-Length': '0'} 2024-02-01T10:18:29.565 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003381add0bf6ee98b-0065bb6ff5-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:29 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:29.565 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-02-01T10:18:29.566 DEBUG:boto:path=/vclfcn-13/myobj 2024-02-01T10:18:29.566 DEBUG:boto:auth_path=/vclfcn-13/myobj 2024-02-01T10:18:29.566 DEBUG:boto:path=/vclfcn-13/myobj?versionId=null 2024-02-01T10:18:29.566 DEBUG:boto:auth_path=/vclfcn-13/myobj?versionId=null 2024-02-01T10:18:29.566 DEBUG:boto:Method: GET 2024-02-01T10:18:29.566 DEBUG:boto:Path: /vclfcn-13/myobj?versionId=null 2024-02-01T10:18:29.566 DEBUG:boto:Data: 2024-02-01T10:18:29.566 DEBUG:boto:Headers: {} 2024-02-01T10:18:29.566 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:29.566 DEBUG:boto:Port: 8000 2024-02-01T10:18:29.566 DEBUG:boto:Params: {} 2024-02-01T10:18:29.567 DEBUG:boto:Token: None 2024-02-01T10:18:29.567 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:29 GMT /vclfcn-13/myobj?versionId=null 2024-02-01T10:18:29.567 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GvrPmLq5LytxzeFYaQl59je++PQ= 2024-02-01T10:18:29.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:29 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GvrPmLq5LytxzeFYaQl59je++PQ=', 'Content-Length': '0'} 2024-02-01T10:18:29.569 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004bf8ffc5e0dd95c2-0065bb6ff5-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:29 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:29.615 DEBUG:boto:path=/vclfcn-13/myobj 2024-02-01T10:18:29.615 DEBUG:boto:auth_path=/vclfcn-13/myobj 2024-02-01T10:18:29.615 DEBUG:boto:path=/vclfcn-13/myobj?versionId=null 2024-02-01T10:18:29.616 DEBUG:boto:auth_path=/vclfcn-13/myobj?versionId=null 2024-02-01T10:18:29.616 DEBUG:boto:Method: GET 2024-02-01T10:18:29.616 DEBUG:boto:Path: /vclfcn-13/myobj?versionId=null 2024-02-01T10:18:29.616 DEBUG:boto:Data: 2024-02-01T10:18:29.616 DEBUG:boto:Headers: {} 2024-02-01T10:18:29.616 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:29.616 DEBUG:boto:Port: 8000 2024-02-01T10:18:29.616 DEBUG:boto:Params: {} 2024-02-01T10:18:29.616 DEBUG:boto:Token: None 2024-02-01T10:18:29.617 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:29 GMT /vclfcn-13/myobj?versionId=null 2024-02-01T10:18:29.617 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GvrPmLq5LytxzeFYaQl59je++PQ= 2024-02-01T10:18:29.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:29 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GvrPmLq5LytxzeFYaQl59je++PQ=', 'Content-Length': '0'} 2024-02-01T10:18:29.620 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009b33275609eccb98-0065bb6ff5-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:29 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:29.621 DEBUG:boto:path=/vclfcn-13/ 2024-02-01T10:18:29.621 DEBUG:boto:auth_path=/vclfcn-13/ 2024-02-01T10:18:29.621 DEBUG:boto:path=/vclfcn-13/?prefix=myobj 2024-02-01T10:18:29.621 DEBUG:boto:auth_path=/vclfcn-13/?prefix=myobj 2024-02-01T10:18:29.621 DEBUG:boto:Method: GET 2024-02-01T10:18:29.621 DEBUG:boto:Path: /vclfcn-13/?prefix=myobj 2024-02-01T10:18:29.621 DEBUG:boto:Data: 2024-02-01T10:18:29.621 DEBUG:boto:Headers: {} 2024-02-01T10:18:29.621 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:29.621 DEBUG:boto:Port: 8000 2024-02-01T10:18:29.621 DEBUG:boto:Params: {} 2024-02-01T10:18:29.622 DEBUG:boto:Token: None 2024-02-01T10:18:29.622 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:29 GMT /vclfcn-13/ 2024-02-01T10:18:29.622 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Vjj+/hQojFVphQbHcCFZklloPME= 2024-02-01T10:18:29.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:29 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Vjj+/hQojFVphQbHcCFZklloPME=', 'Content-Length': '0'} 2024-02-01T10:18:29.626 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d5b1d937c79567c-0065bb6ff5-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:29 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:29.626 DEBUG:boto:b'vclfcn-13myobj1000falsemyobj2024-02-01T10:18:13.740Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:29.626 DEBUG:boto:path=/vclfcn-13/ 2024-02-01T10:18:29.627 DEBUG:boto:auth_path=/vclfcn-13/ 2024-02-01T10:18:29.627 DEBUG:boto:path=/vclfcn-13/?prefix=myobj 2024-02-01T10:18:29.627 DEBUG:boto:auth_path=/vclfcn-13/?prefix=myobj 2024-02-01T10:18:29.627 DEBUG:boto:Method: GET 2024-02-01T10:18:29.627 DEBUG:boto:Path: /vclfcn-13/?prefix=myobj 2024-02-01T10:18:29.627 DEBUG:boto:Data: 2024-02-01T10:18:29.627 DEBUG:boto:Headers: {} 2024-02-01T10:18:29.627 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:29.627 DEBUG:boto:Port: 8000 2024-02-01T10:18:29.627 DEBUG:boto:Params: {} 2024-02-01T10:18:29.627 DEBUG:boto:Token: None 2024-02-01T10:18:29.628 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:29 GMT /vclfcn-13/ 2024-02-01T10:18:29.628 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Vjj+/hQojFVphQbHcCFZklloPME= 2024-02-01T10:18:29.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:29 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Vjj+/hQojFVphQbHcCFZklloPME=', 'Content-Length': '0'} 2024-02-01T10:18:29.632 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004cef8c6cd21c375-0065bb6ff5-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:29 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:29.633 DEBUG:boto:b'vclfcn-13myobj1000falsemyobj2024-02-01T10:18:13.740Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:29.633 DEBUG:boto:path=/vclfcn-13/myobj 2024-02-01T10:18:29.633 DEBUG:boto:auth_path=/vclfcn-13/myobj 2024-02-01T10:18:29.633 DEBUG:boto:Method: HEAD 2024-02-01T10:18:29.633 DEBUG:boto:Path: /vclfcn-13/myobj 2024-02-01T10:18:29.633 DEBUG:boto:Data: 2024-02-01T10:18:29.633 DEBUG:boto:Headers: {} 2024-02-01T10:18:29.634 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:29.634 DEBUG:boto:Port: 8000 2024-02-01T10:18:29.634 DEBUG:boto:Params: {} 2024-02-01T10:18:29.634 DEBUG:boto:Token: None 2024-02-01T10:18:29.634 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:29 GMT /vclfcn-13/myobj 2024-02-01T10:18:29.634 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:bwBB2zyncpYlwuvnNVz976e2TMw= 2024-02-01T10:18:29.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:29 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:bwBB2zyncpYlwuvnNVz976e2TMw=', 'Content-Length': '0'} 2024-02-01T10:18:29.637 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f0e73facf871b83f-0065bb6ff5-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:29 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:29.637 DEBUG:boto:path=/vclfcn-13/myobj 2024-02-01T10:18:29.637 DEBUG:boto:auth_path=/vclfcn-13/myobj 2024-02-01T10:18:29.638 DEBUG:boto:Method: HEAD 2024-02-01T10:18:29.638 DEBUG:boto:Path: /vclfcn-13/myobj 2024-02-01T10:18:29.638 DEBUG:boto:Data: 2024-02-01T10:18:29.638 DEBUG:boto:Headers: {} 2024-02-01T10:18:29.638 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:29.638 DEBUG:boto:Port: 8000 2024-02-01T10:18:29.638 DEBUG:boto:Params: {} 2024-02-01T10:18:29.638 DEBUG:boto:Token: None 2024-02-01T10:18:29.638 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:29 GMT /vclfcn-13/myobj 2024-02-01T10:18:29.639 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:bwBB2zyncpYlwuvnNVz976e2TMw= 2024-02-01T10:18:29.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:29 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:bwBB2zyncpYlwuvnNVz976e2TMw=', 'Content-Length': '0'} 2024-02-01T10:18:29.641 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f1423beea1d4280b-0065bb6ff5-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:29 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:29.641 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-02-01T10:18:29.641 DEBUG:boto:path=/vclfcn-13/myobj 2024-02-01T10:18:29.642 DEBUG:boto:auth_path=/vclfcn-13/myobj 2024-02-01T10:18:29.642 DEBUG:boto:Method: GET 2024-02-01T10:18:29.642 DEBUG:boto:Path: /vclfcn-13/myobj 2024-02-01T10:18:29.642 DEBUG:boto:Data: 2024-02-01T10:18:29.642 DEBUG:boto:Headers: {} 2024-02-01T10:18:29.642 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:29.642 DEBUG:boto:Port: 8000 2024-02-01T10:18:29.642 DEBUG:boto:Params: {} 2024-02-01T10:18:29.642 DEBUG:boto:Token: None 2024-02-01T10:18:29.643 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:29 GMT /vclfcn-13/myobj 2024-02-01T10:18:29.643 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:geBBNRGEpjZcWMGFxRbT7aV5XiQ= 2024-02-01T10:18:29.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:29 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:geBBNRGEpjZcWMGFxRbT7aV5XiQ=', 'Content-Length': '0'} 2024-02-01T10:18:29.645 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002f060e82b12dacc0-0065bb6ff5-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:29 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:29.688 DEBUG:boto:path=/vclfcn-13/myobj 2024-02-01T10:18:29.688 DEBUG:boto:auth_path=/vclfcn-13/myobj 2024-02-01T10:18:29.688 DEBUG:boto:Method: GET 2024-02-01T10:18:29.689 DEBUG:boto:Path: /vclfcn-13/myobj 2024-02-01T10:18:29.689 DEBUG:boto:Data: 2024-02-01T10:18:29.689 DEBUG:boto:Headers: {} 2024-02-01T10:18:29.689 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:29.689 DEBUG:boto:Port: 8000 2024-02-01T10:18:29.689 DEBUG:boto:Params: {} 2024-02-01T10:18:29.689 DEBUG:boto:Token: None 2024-02-01T10:18:29.689 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:29 GMT /vclfcn-13/myobj 2024-02-01T10:18:29.690 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:geBBNRGEpjZcWMGFxRbT7aV5XiQ= 2024-02-01T10:18:29.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:29 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:geBBNRGEpjZcWMGFxRbT7aV5XiQ=', 'Content-Length': '0'} 2024-02-01T10:18:29.692 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ac08aa7d29c1fcc8-0065bb6ff5-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:29 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:29.693 DEBUG:boto:path=/vclfcn-13/ 2024-02-01T10:18:29.693 DEBUG:boto:auth_path=/vclfcn-13/ 2024-02-01T10:18:29.693 DEBUG:boto:path=/vclfcn-13/?prefix=myobj 2024-02-01T10:18:29.693 DEBUG:boto:auth_path=/vclfcn-13/?prefix=myobj 2024-02-01T10:18:29.693 DEBUG:boto:Method: GET 2024-02-01T10:18:29.693 DEBUG:boto:Path: /vclfcn-13/?prefix=myobj 2024-02-01T10:18:29.694 DEBUG:boto:Data: 2024-02-01T10:18:29.694 DEBUG:boto:Headers: {} 2024-02-01T10:18:29.694 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:29.694 DEBUG:boto:Port: 8000 2024-02-01T10:18:29.694 DEBUG:boto:Params: {} 2024-02-01T10:18:29.694 DEBUG:boto:Token: None 2024-02-01T10:18:29.694 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:29 GMT /vclfcn-13/ 2024-02-01T10:18:29.694 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Vjj+/hQojFVphQbHcCFZklloPME= 2024-02-01T10:18:29.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:29 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Vjj+/hQojFVphQbHcCFZklloPME=', 'Content-Length': '0'} 2024-02-01T10:18:29.699 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000759963e3fc764cc6-0065bb6ff5-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:29 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:29.699 DEBUG:boto:b'vclfcn-13myobj1000falsemyobj2024-02-01T10:18:13.740Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:29.699 DEBUG:boto:path=/vclfcn-13/ 2024-02-01T10:18:29.699 DEBUG:boto:auth_path=/vclfcn-13/ 2024-02-01T10:18:29.700 DEBUG:boto:path=/vclfcn-13/?prefix=myobj 2024-02-01T10:18:29.700 DEBUG:boto:auth_path=/vclfcn-13/?prefix=myobj 2024-02-01T10:18:29.700 DEBUG:boto:Method: GET 2024-02-01T10:18:29.700 DEBUG:boto:Path: /vclfcn-13/?prefix=myobj 2024-02-01T10:18:29.700 DEBUG:boto:Data: 2024-02-01T10:18:29.700 DEBUG:boto:Headers: {} 2024-02-01T10:18:29.700 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:29.700 DEBUG:boto:Port: 8000 2024-02-01T10:18:29.700 DEBUG:boto:Params: {} 2024-02-01T10:18:29.700 DEBUG:boto:Token: None 2024-02-01T10:18:29.700 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:29 GMT /vclfcn-13/ 2024-02-01T10:18:29.701 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Vjj+/hQojFVphQbHcCFZklloPME= 2024-02-01T10:18:29.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:29 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Vjj+/hQojFVphQbHcCFZklloPME=', 'Content-Length': '0'} 2024-02-01T10:18:29.706 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000224990a28e479cf0-0065bb6ff5-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:29 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:29.748 DEBUG:boto:b'vclfcn-13myobj1000falsemyobj2024-02-01T10:18:13.740Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:29.748 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-13 zones={test-zone2, test-zone1} 2024-02-01T10:18:29.749 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:18:29.749 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:18:29.749 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:18:29.799 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:18:29.800 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:18:29.858 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:18:29.854+0000 7f3b2e576980 1 waiting to reach incremental sync.. 2024-02-01T10:18:34.861 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:18:34.858+0000 7f3b2e576980 1 bucket sync caught up with source: 2024-02-01T10:18:34.861 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, 00000000001.13.6, , , , , , , , , ] 2024-02-01T10:18:34.861 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, 00000000001.13.6, , , , , , , , , ] 2024-02-01T10:18:34.861 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:18:34.858+0000 7f3b2e576980 0 bucket checkpoint complete 2024-02-01T10:18:34.901 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-13 zones={test-zone3, test-zone1} 2024-02-01T10:18:34.901 DEBUG:boto:path=/vclfcn-13/ 2024-02-01T10:18:34.902 DEBUG:boto:auth_path=/vclfcn-13/ 2024-02-01T10:18:34.902 DEBUG:boto:Method: HEAD 2024-02-01T10:18:34.902 DEBUG:boto:Path: /vclfcn-13/ 2024-02-01T10:18:34.902 DEBUG:boto:Data: 2024-02-01T10:18:34.902 DEBUG:boto:Headers: {} 2024-02-01T10:18:34.902 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:34.902 DEBUG:boto:Port: 8001 2024-02-01T10:18:34.902 DEBUG:boto:Params: {} 2024-02-01T10:18:34.902 DEBUG:boto:Token: None 2024-02-01T10:18:34.903 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:34 GMT /vclfcn-13/ 2024-02-01T10:18:34.903 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ag3uTSekQsq3SzjDH6gTa+GIgMc= 2024-02-01T10:18:34.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:34 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ag3uTSekQsq3SzjDH6gTa+GIgMc=', 'Content-Length': '0'} 2024-02-01T10:18:34.906 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000053fc96a5881b9915-0065bb6ffa-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:18:34 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:34.906 DEBUG:boto:path=/vclfcn-13/ 2024-02-01T10:18:34.906 DEBUG:boto:auth_path=/vclfcn-13/ 2024-02-01T10:18:34.906 DEBUG:boto:Method: HEAD 2024-02-01T10:18:34.907 DEBUG:boto:Path: /vclfcn-13/ 2024-02-01T10:18:34.907 DEBUG:boto:Data: 2024-02-01T10:18:34.907 DEBUG:boto:Headers: {} 2024-02-01T10:18:34.907 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:34.907 DEBUG:boto:Port: 8000 2024-02-01T10:18:34.907 DEBUG:boto:Params: {} 2024-02-01T10:18:34.907 DEBUG:boto:Token: None 2024-02-01T10:18:34.907 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:34 GMT /vclfcn-13/ 2024-02-01T10:18:34.907 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ag3uTSekQsq3SzjDH6gTa+GIgMc= 2024-02-01T10:18:34.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:34 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ag3uTSekQsq3SzjDH6gTa+GIgMc=', 'Content-Length': '0'} 2024-02-01T10:18:34.910 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000696746d8d11cf870-0065bb6ffa-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:18:34 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:34.910 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:18:34.911 DEBUG:boto:path=/vclfcn-13/ 2024-02-01T10:18:34.911 DEBUG:boto:auth_path=/vclfcn-13/ 2024-02-01T10:18:34.911 DEBUG:boto:path=/vclfcn-13/?versions&max-keys=999 2024-02-01T10:18:34.911 DEBUG:boto:auth_path=/vclfcn-13/?versions&max-keys=999 2024-02-01T10:18:34.911 DEBUG:boto:Method: GET 2024-02-01T10:18:34.911 DEBUG:boto:Path: /vclfcn-13/?versions&max-keys=999 2024-02-01T10:18:34.911 DEBUG:boto:Data: 2024-02-01T10:18:34.911 DEBUG:boto:Headers: {} 2024-02-01T10:18:34.911 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:34.911 DEBUG:boto:Port: 8001 2024-02-01T10:18:34.911 DEBUG:boto:Params: {} 2024-02-01T10:18:34.912 DEBUG:boto:Token: None 2024-02-01T10:18:34.912 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:34 GMT /vclfcn-13/?versions 2024-02-01T10:18:34.912 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:XFVDHxmGU23UFJkRVDPQcInUS90= 2024-02-01T10:18:34.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:34 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:XFVDHxmGU23UFJkRVDPQcInUS90=', 'Content-Length': '0'} 2024-02-01T10:18:34.915 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000140f0ee200f8498d-0065bb6ffa-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:34 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:34.915 DEBUG:boto:b'vclfcn-13999falsemyobjnulltrue2024-02-01T10:18:13.740Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:34.916 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-02-01T10:18:34.916 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:18:34.916 DEBUG:boto:path=/vclfcn-13/ 2024-02-01T10:18:34.916 DEBUG:boto:auth_path=/vclfcn-13/ 2024-02-01T10:18:34.916 DEBUG:boto:path=/vclfcn-13/?versions&max-keys=999 2024-02-01T10:18:34.916 DEBUG:boto:auth_path=/vclfcn-13/?versions&max-keys=999 2024-02-01T10:18:34.916 DEBUG:boto:Method: GET 2024-02-01T10:18:34.917 DEBUG:boto:Path: /vclfcn-13/?versions&max-keys=999 2024-02-01T10:18:34.917 DEBUG:boto:Data: 2024-02-01T10:18:34.917 DEBUG:boto:Headers: {} 2024-02-01T10:18:34.917 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:34.917 DEBUG:boto:Port: 8000 2024-02-01T10:18:34.917 DEBUG:boto:Params: {} 2024-02-01T10:18:34.917 DEBUG:boto:Token: None 2024-02-01T10:18:34.917 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:34 GMT /vclfcn-13/?versions 2024-02-01T10:18:34.917 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:XFVDHxmGU23UFJkRVDPQcInUS90= 2024-02-01T10:18:34.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:34 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:XFVDHxmGU23UFJkRVDPQcInUS90=', 'Content-Length': '0'} 2024-02-01T10:18:34.920 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002af507d0cdd4a320-0065bb6ffa-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:34 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:34.921 DEBUG:boto:b'vclfcn-13999falsemyobjnulltrue2024-02-01T10:18:13.740Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:34.921 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-02-01T10:18:34.921 DEBUG:boto:path=/vclfcn-13/ 2024-02-01T10:18:34.921 DEBUG:boto:auth_path=/vclfcn-13/ 2024-02-01T10:18:34.921 DEBUG:boto:path=/vclfcn-13/?versions&max-keys=999 2024-02-01T10:18:34.921 DEBUG:boto:auth_path=/vclfcn-13/?versions&max-keys=999 2024-02-01T10:18:34.922 DEBUG:boto:Method: GET 2024-02-01T10:18:34.922 DEBUG:boto:Path: /vclfcn-13/?versions&max-keys=999 2024-02-01T10:18:34.922 DEBUG:boto:Data: 2024-02-01T10:18:34.922 DEBUG:boto:Headers: {} 2024-02-01T10:18:34.922 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:34.922 DEBUG:boto:Port: 8001 2024-02-01T10:18:34.922 DEBUG:boto:Params: {} 2024-02-01T10:18:34.922 DEBUG:boto:Token: None 2024-02-01T10:18:34.922 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:34 GMT /vclfcn-13/?versions 2024-02-01T10:18:34.922 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:XFVDHxmGU23UFJkRVDPQcInUS90= 2024-02-01T10:18:34.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:34 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:XFVDHxmGU23UFJkRVDPQcInUS90=', 'Content-Length': '0'} 2024-02-01T10:18:34.926 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ca713bc27663d90-0065bb6ffa-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:34 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:34.926 DEBUG:boto:b'vclfcn-13999falsemyobjnulltrue2024-02-01T10:18:13.740Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:34.926 DEBUG:boto:path=/vclfcn-13/ 2024-02-01T10:18:34.926 DEBUG:boto:auth_path=/vclfcn-13/ 2024-02-01T10:18:34.926 DEBUG:boto:path=/vclfcn-13/?versions&max-keys=999 2024-02-01T10:18:34.926 DEBUG:boto:auth_path=/vclfcn-13/?versions&max-keys=999 2024-02-01T10:18:34.927 DEBUG:boto:Method: GET 2024-02-01T10:18:34.927 DEBUG:boto:Path: /vclfcn-13/?versions&max-keys=999 2024-02-01T10:18:34.927 DEBUG:boto:Data: 2024-02-01T10:18:34.927 DEBUG:boto:Headers: {} 2024-02-01T10:18:34.927 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:34.927 DEBUG:boto:Port: 8000 2024-02-01T10:18:34.927 DEBUG:boto:Params: {} 2024-02-01T10:18:34.927 DEBUG:boto:Token: None 2024-02-01T10:18:34.927 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:34 GMT /vclfcn-13/?versions 2024-02-01T10:18:34.927 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:XFVDHxmGU23UFJkRVDPQcInUS90= 2024-02-01T10:18:34.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:34 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:XFVDHxmGU23UFJkRVDPQcInUS90=', 'Content-Length': '0'} 2024-02-01T10:18:34.930 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009466eb14900dd530-0065bb6ffa-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:34 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:34.931 DEBUG:boto:b'vclfcn-13999falsemyobjnulltrue2024-02-01T10:18:13.740Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:34.931 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-02-01T10:18:34.931 DEBUG:boto:path=/vclfcn-13/myobj 2024-02-01T10:18:34.931 DEBUG:boto:auth_path=/vclfcn-13/myobj 2024-02-01T10:18:34.931 DEBUG:boto:path=/vclfcn-13/myobj?versionId=null 2024-02-01T10:18:34.931 DEBUG:boto:auth_path=/vclfcn-13/myobj?versionId=null 2024-02-01T10:18:34.932 DEBUG:boto:Method: GET 2024-02-01T10:18:34.932 DEBUG:boto:Path: /vclfcn-13/myobj?versionId=null 2024-02-01T10:18:34.932 DEBUG:boto:Data: 2024-02-01T10:18:34.932 DEBUG:boto:Headers: {} 2024-02-01T10:18:34.932 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:34.932 DEBUG:boto:Port: 8001 2024-02-01T10:18:34.932 DEBUG:boto:Params: {} 2024-02-01T10:18:34.932 DEBUG:boto:Token: None 2024-02-01T10:18:34.932 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:34 GMT /vclfcn-13/myobj?versionId=null 2024-02-01T10:18:34.932 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RqMOEnGYjbSFw/6SHx4DBohg7K0= 2024-02-01T10:18:34.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:34 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RqMOEnGYjbSFw/6SHx4DBohg7K0=', 'Content-Length': '0'} 2024-02-01T10:18:34.935 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f2048cd049633027-0065bb6ffa-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:34 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:34.935 DEBUG:boto:path=/vclfcn-13/myobj 2024-02-01T10:18:34.935 DEBUG:boto:auth_path=/vclfcn-13/myobj 2024-02-01T10:18:34.935 DEBUG:boto:path=/vclfcn-13/myobj?versionId=null 2024-02-01T10:18:34.936 DEBUG:boto:auth_path=/vclfcn-13/myobj?versionId=null 2024-02-01T10:18:34.936 DEBUG:boto:Method: GET 2024-02-01T10:18:34.936 DEBUG:boto:Path: /vclfcn-13/myobj?versionId=null 2024-02-01T10:18:34.936 DEBUG:boto:Data: 2024-02-01T10:18:34.936 DEBUG:boto:Headers: {} 2024-02-01T10:18:34.936 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:34.936 DEBUG:boto:Port: 8000 2024-02-01T10:18:34.936 DEBUG:boto:Params: {} 2024-02-01T10:18:34.936 DEBUG:boto:Token: None 2024-02-01T10:18:34.937 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:34 GMT /vclfcn-13/myobj?versionId=null 2024-02-01T10:18:34.937 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RqMOEnGYjbSFw/6SHx4DBohg7K0= 2024-02-01T10:18:34.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:34 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RqMOEnGYjbSFw/6SHx4DBohg7K0=', 'Content-Length': '0'} 2024-02-01T10:18:34.939 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000aeed2375e9b7768-0065bb6ffa-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:34 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:34.939 DEBUG:boto:path=/vclfcn-13/myobj 2024-02-01T10:18:34.939 DEBUG:boto:auth_path=/vclfcn-13/myobj 2024-02-01T10:18:34.940 DEBUG:boto:path=/vclfcn-13/myobj?versionId=null 2024-02-01T10:18:34.940 DEBUG:boto:auth_path=/vclfcn-13/myobj?versionId=null 2024-02-01T10:18:34.940 DEBUG:boto:Method: HEAD 2024-02-01T10:18:34.940 DEBUG:boto:Path: /vclfcn-13/myobj?versionId=null 2024-02-01T10:18:34.940 DEBUG:boto:Data: 2024-02-01T10:18:34.940 DEBUG:boto:Headers: {} 2024-02-01T10:18:34.940 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:34.940 DEBUG:boto:Port: 8001 2024-02-01T10:18:34.940 DEBUG:boto:Params: {} 2024-02-01T10:18:34.940 DEBUG:boto:Token: None 2024-02-01T10:18:34.941 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:34 GMT /vclfcn-13/myobj?versionId=null 2024-02-01T10:18:34.941 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:74agnMLS6u6xeOqP54T0Lp1a9Y4= 2024-02-01T10:18:34.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:34 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:74agnMLS6u6xeOqP54T0Lp1a9Y4=', 'Content-Length': '0'} 2024-02-01T10:18:34.943 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b502263a6c31c344-0065bb6ffa-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:34 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:34.943 DEBUG:boto:path=/vclfcn-13/myobj 2024-02-01T10:18:34.943 DEBUG:boto:auth_path=/vclfcn-13/myobj 2024-02-01T10:18:34.943 DEBUG:boto:path=/vclfcn-13/myobj?versionId=null 2024-02-01T10:18:34.944 DEBUG:boto:auth_path=/vclfcn-13/myobj?versionId=null 2024-02-01T10:18:34.944 DEBUG:boto:Method: HEAD 2024-02-01T10:18:34.944 DEBUG:boto:Path: /vclfcn-13/myobj?versionId=null 2024-02-01T10:18:34.944 DEBUG:boto:Data: 2024-02-01T10:18:34.944 DEBUG:boto:Headers: {} 2024-02-01T10:18:34.944 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:34.944 DEBUG:boto:Port: 8000 2024-02-01T10:18:34.944 DEBUG:boto:Params: {} 2024-02-01T10:18:34.944 DEBUG:boto:Token: None 2024-02-01T10:18:34.944 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:34 GMT /vclfcn-13/myobj?versionId=null 2024-02-01T10:18:34.945 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:74agnMLS6u6xeOqP54T0Lp1a9Y4= 2024-02-01T10:18:34.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:34 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:74agnMLS6u6xeOqP54T0Lp1a9Y4=', 'Content-Length': '0'} 2024-02-01T10:18:34.947 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000075729dd719e865dc-0065bb6ffa-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:34 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:34.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-02-01T10:18:34.947 DEBUG:boto:path=/vclfcn-13/myobj 2024-02-01T10:18:34.947 DEBUG:boto:auth_path=/vclfcn-13/myobj 2024-02-01T10:18:34.947 DEBUG:boto:path=/vclfcn-13/myobj?versionId=null 2024-02-01T10:18:34.947 DEBUG:boto:auth_path=/vclfcn-13/myobj?versionId=null 2024-02-01T10:18:34.947 DEBUG:boto:Method: GET 2024-02-01T10:18:34.947 DEBUG:boto:Path: /vclfcn-13/myobj?versionId=null 2024-02-01T10:18:34.948 DEBUG:boto:Data: 2024-02-01T10:18:34.948 DEBUG:boto:Headers: {} 2024-02-01T10:18:34.948 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:34.948 DEBUG:boto:Port: 8001 2024-02-01T10:18:34.948 DEBUG:boto:Params: {} 2024-02-01T10:18:34.948 DEBUG:boto:Token: None 2024-02-01T10:18:34.948 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:34 GMT /vclfcn-13/myobj?versionId=null 2024-02-01T10:18:34.948 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RqMOEnGYjbSFw/6SHx4DBohg7K0= 2024-02-01T10:18:34.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:34 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RqMOEnGYjbSFw/6SHx4DBohg7K0=', 'Content-Length': '0'} 2024-02-01T10:18:34.951 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c310d39aff28d461-0065bb6ffa-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:34 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:34.951 DEBUG:boto:path=/vclfcn-13/myobj 2024-02-01T10:18:34.951 DEBUG:boto:auth_path=/vclfcn-13/myobj 2024-02-01T10:18:34.951 DEBUG:boto:path=/vclfcn-13/myobj?versionId=null 2024-02-01T10:18:34.951 DEBUG:boto:auth_path=/vclfcn-13/myobj?versionId=null 2024-02-01T10:18:34.951 DEBUG:boto:Method: GET 2024-02-01T10:18:34.951 DEBUG:boto:Path: /vclfcn-13/myobj?versionId=null 2024-02-01T10:18:34.952 DEBUG:boto:Data: 2024-02-01T10:18:34.952 DEBUG:boto:Headers: {} 2024-02-01T10:18:34.952 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:34.952 DEBUG:boto:Port: 8000 2024-02-01T10:18:34.952 DEBUG:boto:Params: {} 2024-02-01T10:18:34.952 DEBUG:boto:Token: None 2024-02-01T10:18:34.956 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:34 GMT /vclfcn-13/myobj?versionId=null 2024-02-01T10:18:34.956 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RqMOEnGYjbSFw/6SHx4DBohg7K0= 2024-02-01T10:18:34.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:34 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RqMOEnGYjbSFw/6SHx4DBohg7K0=', 'Content-Length': '0'} 2024-02-01T10:18:34.959 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ba0ffbd2f67e9008-0065bb6ffa-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:34 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:34.959 DEBUG:boto:path=/vclfcn-13/ 2024-02-01T10:18:34.959 DEBUG:boto:auth_path=/vclfcn-13/ 2024-02-01T10:18:34.959 DEBUG:boto:path=/vclfcn-13/?prefix=myobj 2024-02-01T10:18:34.959 DEBUG:boto:auth_path=/vclfcn-13/?prefix=myobj 2024-02-01T10:18:34.959 DEBUG:boto:Method: GET 2024-02-01T10:18:34.960 DEBUG:boto:Path: /vclfcn-13/?prefix=myobj 2024-02-01T10:18:34.960 DEBUG:boto:Data: 2024-02-01T10:18:34.960 DEBUG:boto:Headers: {} 2024-02-01T10:18:34.960 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:34.960 DEBUG:boto:Port: 8001 2024-02-01T10:18:34.960 DEBUG:boto:Params: {} 2024-02-01T10:18:34.960 DEBUG:boto:Token: None 2024-02-01T10:18:34.960 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:34 GMT /vclfcn-13/ 2024-02-01T10:18:34.960 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:weCDvU3nnjjHswuHv6bmYUu4UQE= 2024-02-01T10:18:34.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:34 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:weCDvU3nnjjHswuHv6bmYUu4UQE=', 'Content-Length': '0'} 2024-02-01T10:18:34.964 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006502202a491479fd-0065bb6ffa-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:34 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:34.964 DEBUG:boto:b'vclfcn-13myobj1000falsemyobj2024-02-01T10:18:13.740Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:34.965 DEBUG:boto:path=/vclfcn-13/ 2024-02-01T10:18:34.965 DEBUG:boto:auth_path=/vclfcn-13/ 2024-02-01T10:18:34.965 DEBUG:boto:path=/vclfcn-13/?prefix=myobj 2024-02-01T10:18:34.965 DEBUG:boto:auth_path=/vclfcn-13/?prefix=myobj 2024-02-01T10:18:34.965 DEBUG:boto:Method: GET 2024-02-01T10:18:34.965 DEBUG:boto:Path: /vclfcn-13/?prefix=myobj 2024-02-01T10:18:34.965 DEBUG:boto:Data: 2024-02-01T10:18:34.965 DEBUG:boto:Headers: {} 2024-02-01T10:18:34.965 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:34.965 DEBUG:boto:Port: 8000 2024-02-01T10:18:34.966 DEBUG:boto:Params: {} 2024-02-01T10:18:34.966 DEBUG:boto:Token: None 2024-02-01T10:18:34.966 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:34 GMT /vclfcn-13/ 2024-02-01T10:18:34.966 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:weCDvU3nnjjHswuHv6bmYUu4UQE= 2024-02-01T10:18:34.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:34 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:weCDvU3nnjjHswuHv6bmYUu4UQE=', 'Content-Length': '0'} 2024-02-01T10:18:34.969 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef3f0c84eba574fc-0065bb6ffa-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:34 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:34.969 DEBUG:boto:b'vclfcn-13myobj1000falsemyobj2024-02-01T10:18:13.740Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:34.970 DEBUG:boto:path=/vclfcn-13/myobj 2024-02-01T10:18:34.970 DEBUG:boto:auth_path=/vclfcn-13/myobj 2024-02-01T10:18:34.970 DEBUG:boto:Method: HEAD 2024-02-01T10:18:34.970 DEBUG:boto:Path: /vclfcn-13/myobj 2024-02-01T10:18:34.970 DEBUG:boto:Data: 2024-02-01T10:18:34.970 DEBUG:boto:Headers: {} 2024-02-01T10:18:34.970 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:34.970 DEBUG:boto:Port: 8001 2024-02-01T10:18:34.970 DEBUG:boto:Params: {} 2024-02-01T10:18:34.970 DEBUG:boto:Token: None 2024-02-01T10:18:34.971 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:34 GMT /vclfcn-13/myobj 2024-02-01T10:18:34.971 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8X6Na7p5Owx7MSfg4oxvQxQ7EqI= 2024-02-01T10:18:34.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:34 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8X6Na7p5Owx7MSfg4oxvQxQ7EqI=', 'Content-Length': '0'} 2024-02-01T10:18:34.973 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003a9876f3f4f1fdee-0065bb6ffa-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:34 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:34.973 DEBUG:boto:path=/vclfcn-13/myobj 2024-02-01T10:18:34.973 DEBUG:boto:auth_path=/vclfcn-13/myobj 2024-02-01T10:18:34.973 DEBUG:boto:Method: HEAD 2024-02-01T10:18:34.974 DEBUG:boto:Path: /vclfcn-13/myobj 2024-02-01T10:18:34.974 DEBUG:boto:Data: 2024-02-01T10:18:34.974 DEBUG:boto:Headers: {} 2024-02-01T10:18:34.974 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:34.974 DEBUG:boto:Port: 8000 2024-02-01T10:18:34.974 DEBUG:boto:Params: {} 2024-02-01T10:18:34.974 DEBUG:boto:Token: None 2024-02-01T10:18:34.974 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:34 GMT /vclfcn-13/myobj 2024-02-01T10:18:34.974 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8X6Na7p5Owx7MSfg4oxvQxQ7EqI= 2024-02-01T10:18:34.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:34 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8X6Na7p5Owx7MSfg4oxvQxQ7EqI=', 'Content-Length': '0'} 2024-02-01T10:18:34.977 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001287b0534a74074b-0065bb6ffa-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:34 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:34.977 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-02-01T10:18:34.977 DEBUG:boto:path=/vclfcn-13/myobj 2024-02-01T10:18:34.978 DEBUG:boto:auth_path=/vclfcn-13/myobj 2024-02-01T10:18:34.978 DEBUG:boto:Method: GET 2024-02-01T10:18:34.978 DEBUG:boto:Path: /vclfcn-13/myobj 2024-02-01T10:18:34.978 DEBUG:boto:Data: 2024-02-01T10:18:34.978 DEBUG:boto:Headers: {} 2024-02-01T10:18:34.978 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:34.978 DEBUG:boto:Port: 8001 2024-02-01T10:18:34.978 DEBUG:boto:Params: {} 2024-02-01T10:18:34.978 DEBUG:boto:Token: None 2024-02-01T10:18:34.978 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:34 GMT /vclfcn-13/myobj 2024-02-01T10:18:34.979 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jSydTD1pQmGzakUa8D2CO+Y2hI4= 2024-02-01T10:18:34.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:34 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jSydTD1pQmGzakUa8D2CO+Y2hI4=', 'Content-Length': '0'} 2024-02-01T10:18:34.981 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f52cd1edce65466b-0065bb6ffa-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:34 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:35.028 DEBUG:boto:path=/vclfcn-13/myobj 2024-02-01T10:18:35.028 DEBUG:boto:auth_path=/vclfcn-13/myobj 2024-02-01T10:18:35.028 DEBUG:boto:Method: GET 2024-02-01T10:18:35.029 DEBUG:boto:Path: /vclfcn-13/myobj 2024-02-01T10:18:35.029 DEBUG:boto:Data: 2024-02-01T10:18:35.029 DEBUG:boto:Headers: {} 2024-02-01T10:18:35.029 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:35.029 DEBUG:boto:Port: 8000 2024-02-01T10:18:35.029 DEBUG:boto:Params: {} 2024-02-01T10:18:35.029 DEBUG:boto:Token: None 2024-02-01T10:18:35.029 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:35 GMT /vclfcn-13/myobj 2024-02-01T10:18:35.029 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:slFtfsQt5EEg8yvtk4K6MRHLM3Q= 2024-02-01T10:18:35.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:slFtfsQt5EEg8yvtk4K6MRHLM3Q=', 'Content-Length': '0'} 2024-02-01T10:18:35.032 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000de4d049f2ba4691e-0065bb6ffb-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:35.033 DEBUG:boto:path=/vclfcn-13/ 2024-02-01T10:18:35.033 DEBUG:boto:auth_path=/vclfcn-13/ 2024-02-01T10:18:35.033 DEBUG:boto:path=/vclfcn-13/?prefix=myobj 2024-02-01T10:18:35.033 DEBUG:boto:auth_path=/vclfcn-13/?prefix=myobj 2024-02-01T10:18:35.033 DEBUG:boto:Method: GET 2024-02-01T10:18:35.033 DEBUG:boto:Path: /vclfcn-13/?prefix=myobj 2024-02-01T10:18:35.033 DEBUG:boto:Data: 2024-02-01T10:18:35.033 DEBUG:boto:Headers: {} 2024-02-01T10:18:35.033 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:35.034 DEBUG:boto:Port: 8001 2024-02-01T10:18:35.034 DEBUG:boto:Params: {} 2024-02-01T10:18:35.034 DEBUG:boto:Token: None 2024-02-01T10:18:35.034 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:35 GMT /vclfcn-13/ 2024-02-01T10:18:35.034 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GURxUzflz0puYFH73bZIVrf+9x0= 2024-02-01T10:18:35.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GURxUzflz0puYFH73bZIVrf+9x0=', 'Content-Length': '0'} 2024-02-01T10:18:35.038 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d873b401c6927d17-0065bb6ffb-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:35.038 DEBUG:boto:b'vclfcn-13myobj1000falsemyobj2024-02-01T10:18:13.740Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:35.038 DEBUG:boto:path=/vclfcn-13/ 2024-02-01T10:18:35.038 DEBUG:boto:auth_path=/vclfcn-13/ 2024-02-01T10:18:35.038 DEBUG:boto:path=/vclfcn-13/?prefix=myobj 2024-02-01T10:18:35.039 DEBUG:boto:auth_path=/vclfcn-13/?prefix=myobj 2024-02-01T10:18:35.039 DEBUG:boto:Method: GET 2024-02-01T10:18:35.039 DEBUG:boto:Path: /vclfcn-13/?prefix=myobj 2024-02-01T10:18:35.039 DEBUG:boto:Data: 2024-02-01T10:18:35.039 DEBUG:boto:Headers: {} 2024-02-01T10:18:35.039 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:35.039 DEBUG:boto:Port: 8000 2024-02-01T10:18:35.039 DEBUG:boto:Params: {} 2024-02-01T10:18:35.039 DEBUG:boto:Token: None 2024-02-01T10:18:35.039 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:35 GMT /vclfcn-13/ 2024-02-01T10:18:35.040 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GURxUzflz0puYFH73bZIVrf+9x0= 2024-02-01T10:18:35.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GURxUzflz0puYFH73bZIVrf+9x0=', 'Content-Length': '0'} 2024-02-01T10:18:35.044 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a88980971491bb3-0065bb6ffb-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:35.084 DEBUG:boto:b'vclfcn-13myobj1000falsemyobj2024-02-01T10:18:13.740Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:35.084 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-13 zones={test-zone3, test-zone1} 2024-02-01T10:18:35.085 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:18:35.085 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:18:35.085 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:18:35.135 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:18:35.135 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:18:35.199 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:18:35.198+0000 7f52add4a980 1 bucket sync caught up with source: 2024-02-01T10:18:35.200 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, 00000000001.10.6, , , , , , , , , ] 2024-02-01T10:18:35.200 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, 00000000001.10.6, , , , , , , , , ] 2024-02-01T10:18:35.200 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:18:35.198+0000 7f52add4a980 0 bucket checkpoint complete 2024-02-01T10:18:35.222 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-14 zones={test-zone1, test-zone2} 2024-02-01T10:18:35.222 DEBUG:boto:path=/vclfcn-14/ 2024-02-01T10:18:35.222 DEBUG:boto:auth_path=/vclfcn-14/ 2024-02-01T10:18:35.222 DEBUG:boto:Method: HEAD 2024-02-01T10:18:35.222 DEBUG:boto:Path: /vclfcn-14/ 2024-02-01T10:18:35.222 DEBUG:boto:Data: 2024-02-01T10:18:35.223 DEBUG:boto:Headers: {} 2024-02-01T10:18:35.223 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:35.223 DEBUG:boto:Port: 8000 2024-02-01T10:18:35.223 DEBUG:boto:Params: {} 2024-02-01T10:18:35.223 DEBUG:boto:Token: None 2024-02-01T10:18:35.223 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:35 GMT /vclfcn-14/ 2024-02-01T10:18:35.223 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:naJXyAJ9hwPmMhefifG5KDL673U= 2024-02-01T10:18:35.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:naJXyAJ9hwPmMhefifG5KDL673U=', 'Content-Length': '0'} 2024-02-01T10:18:35.226 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f8d00de46343c2e7-0065bb6ffb-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:18:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:35.226 DEBUG:boto:path=/vclfcn-14/ 2024-02-01T10:18:35.226 DEBUG:boto:auth_path=/vclfcn-14/ 2024-02-01T10:18:35.226 DEBUG:boto:Method: HEAD 2024-02-01T10:18:35.226 DEBUG:boto:Path: /vclfcn-14/ 2024-02-01T10:18:35.227 DEBUG:boto:Data: 2024-02-01T10:18:35.227 DEBUG:boto:Headers: {} 2024-02-01T10:18:35.227 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:35.227 DEBUG:boto:Port: 8000 2024-02-01T10:18:35.227 DEBUG:boto:Params: {} 2024-02-01T10:18:35.227 DEBUG:boto:Token: None 2024-02-01T10:18:35.227 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:35 GMT /vclfcn-14/ 2024-02-01T10:18:35.227 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:naJXyAJ9hwPmMhefifG5KDL673U= 2024-02-01T10:18:35.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:naJXyAJ9hwPmMhefifG5KDL673U=', 'Content-Length': '0'} 2024-02-01T10:18:35.231 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000018441365599ae661-0065bb6ffb-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:18:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:35.232 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:18:35.232 DEBUG:boto:path=/vclfcn-14/ 2024-02-01T10:18:35.232 DEBUG:boto:auth_path=/vclfcn-14/ 2024-02-01T10:18:35.232 DEBUG:boto:path=/vclfcn-14/?versions&max-keys=999 2024-02-01T10:18:35.232 DEBUG:boto:auth_path=/vclfcn-14/?versions&max-keys=999 2024-02-01T10:18:35.232 DEBUG:boto:Method: GET 2024-02-01T10:18:35.232 DEBUG:boto:Path: /vclfcn-14/?versions&max-keys=999 2024-02-01T10:18:35.232 DEBUG:boto:Data: 2024-02-01T10:18:35.232 DEBUG:boto:Headers: {} 2024-02-01T10:18:35.233 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:35.233 DEBUG:boto:Port: 8000 2024-02-01T10:18:35.233 DEBUG:boto:Params: {} 2024-02-01T10:18:35.233 DEBUG:boto:Token: None 2024-02-01T10:18:35.233 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:35 GMT /vclfcn-14/?versions 2024-02-01T10:18:35.233 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mM5mGKPJoZyWkcbjXzvVuvektWY= 2024-02-01T10:18:35.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mM5mGKPJoZyWkcbjXzvVuvektWY=', 'Content-Length': '0'} 2024-02-01T10:18:35.235 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009cb5dc115c30fdd8-0065bb6ffb-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:35.236 DEBUG:boto:b'vclfcn-14999falsemyobjnulltrue2024-02-01T10:18:13.794Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:35.236 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-02-01T10:18:35.236 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:18:35.236 DEBUG:boto:path=/vclfcn-14/ 2024-02-01T10:18:35.236 DEBUG:boto:auth_path=/vclfcn-14/ 2024-02-01T10:18:35.236 DEBUG:boto:path=/vclfcn-14/?versions&max-keys=999 2024-02-01T10:18:35.237 DEBUG:boto:auth_path=/vclfcn-14/?versions&max-keys=999 2024-02-01T10:18:35.237 DEBUG:boto:Method: GET 2024-02-01T10:18:35.237 DEBUG:boto:Path: /vclfcn-14/?versions&max-keys=999 2024-02-01T10:18:35.237 DEBUG:boto:Data: 2024-02-01T10:18:35.237 DEBUG:boto:Headers: {} 2024-02-01T10:18:35.237 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:35.237 DEBUG:boto:Port: 8000 2024-02-01T10:18:35.237 DEBUG:boto:Params: {} 2024-02-01T10:18:35.237 DEBUG:boto:Token: None 2024-02-01T10:18:35.237 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:35 GMT /vclfcn-14/?versions 2024-02-01T10:18:35.237 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mM5mGKPJoZyWkcbjXzvVuvektWY= 2024-02-01T10:18:35.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mM5mGKPJoZyWkcbjXzvVuvektWY=', 'Content-Length': '0'} 2024-02-01T10:18:35.241 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eaf8f7a352b3c145-0065bb6ffb-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:35.242 DEBUG:boto:b'vclfcn-14999falsemyobjnulltrue2024-02-01T10:18:13.794Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:35.242 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-02-01T10:18:35.242 DEBUG:boto:path=/vclfcn-14/ 2024-02-01T10:18:35.242 DEBUG:boto:auth_path=/vclfcn-14/ 2024-02-01T10:18:35.242 DEBUG:boto:path=/vclfcn-14/?versions&max-keys=999 2024-02-01T10:18:35.242 DEBUG:boto:auth_path=/vclfcn-14/?versions&max-keys=999 2024-02-01T10:18:35.242 DEBUG:boto:Method: GET 2024-02-01T10:18:35.243 DEBUG:boto:Path: /vclfcn-14/?versions&max-keys=999 2024-02-01T10:18:35.243 DEBUG:boto:Data: 2024-02-01T10:18:35.243 DEBUG:boto:Headers: {} 2024-02-01T10:18:35.243 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:35.243 DEBUG:boto:Port: 8000 2024-02-01T10:18:35.243 DEBUG:boto:Params: {} 2024-02-01T10:18:35.243 DEBUG:boto:Token: None 2024-02-01T10:18:35.243 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:35 GMT /vclfcn-14/?versions 2024-02-01T10:18:35.243 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mM5mGKPJoZyWkcbjXzvVuvektWY= 2024-02-01T10:18:35.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mM5mGKPJoZyWkcbjXzvVuvektWY=', 'Content-Length': '0'} 2024-02-01T10:18:35.246 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007682cce7b0abb4de-0065bb6ffb-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:35.246 DEBUG:boto:b'vclfcn-14999falsemyobjnulltrue2024-02-01T10:18:13.794Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:35.246 DEBUG:boto:path=/vclfcn-14/ 2024-02-01T10:18:35.246 DEBUG:boto:auth_path=/vclfcn-14/ 2024-02-01T10:18:35.246 DEBUG:boto:path=/vclfcn-14/?versions&max-keys=999 2024-02-01T10:18:35.246 DEBUG:boto:auth_path=/vclfcn-14/?versions&max-keys=999 2024-02-01T10:18:35.247 DEBUG:boto:Method: GET 2024-02-01T10:18:35.247 DEBUG:boto:Path: /vclfcn-14/?versions&max-keys=999 2024-02-01T10:18:35.247 DEBUG:boto:Data: 2024-02-01T10:18:35.247 DEBUG:boto:Headers: {} 2024-02-01T10:18:35.247 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:35.247 DEBUG:boto:Port: 8000 2024-02-01T10:18:35.247 DEBUG:boto:Params: {} 2024-02-01T10:18:35.247 DEBUG:boto:Token: None 2024-02-01T10:18:35.247 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:35 GMT /vclfcn-14/?versions 2024-02-01T10:18:35.247 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mM5mGKPJoZyWkcbjXzvVuvektWY= 2024-02-01T10:18:35.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mM5mGKPJoZyWkcbjXzvVuvektWY=', 'Content-Length': '0'} 2024-02-01T10:18:35.251 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a7a9ca2c598bade-0065bb6ffb-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:35.252 DEBUG:boto:b'vclfcn-14999falsemyobjnulltrue2024-02-01T10:18:13.794Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:35.252 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-02-01T10:18:35.252 DEBUG:boto:path=/vclfcn-14/myobj 2024-02-01T10:18:35.252 DEBUG:boto:auth_path=/vclfcn-14/myobj 2024-02-01T10:18:35.252 DEBUG:boto:path=/vclfcn-14/myobj?versionId=null 2024-02-01T10:18:35.252 DEBUG:boto:auth_path=/vclfcn-14/myobj?versionId=null 2024-02-01T10:18:35.252 DEBUG:boto:Method: GET 2024-02-01T10:18:35.252 DEBUG:boto:Path: /vclfcn-14/myobj?versionId=null 2024-02-01T10:18:35.253 DEBUG:boto:Data: 2024-02-01T10:18:35.253 DEBUG:boto:Headers: {} 2024-02-01T10:18:35.253 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:35.253 DEBUG:boto:Port: 8000 2024-02-01T10:18:35.253 DEBUG:boto:Params: {} 2024-02-01T10:18:35.253 DEBUG:boto:Token: None 2024-02-01T10:18:35.253 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:35 GMT /vclfcn-14/myobj?versionId=null 2024-02-01T10:18:35.253 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:OpQmr6H9WTVsggoyPHyxIAITlNo= 2024-02-01T10:18:35.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:OpQmr6H9WTVsggoyPHyxIAITlNo=', 'Content-Length': '0'} 2024-02-01T10:18:35.255 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008b714fc1ca936d01-0065bb6ffb-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:35.255 DEBUG:boto:path=/vclfcn-14/myobj 2024-02-01T10:18:35.256 DEBUG:boto:auth_path=/vclfcn-14/myobj 2024-02-01T10:18:35.256 DEBUG:boto:path=/vclfcn-14/myobj?versionId=null 2024-02-01T10:18:35.256 DEBUG:boto:auth_path=/vclfcn-14/myobj?versionId=null 2024-02-01T10:18:35.256 DEBUG:boto:Method: GET 2024-02-01T10:18:35.256 DEBUG:boto:Path: /vclfcn-14/myobj?versionId=null 2024-02-01T10:18:35.256 DEBUG:boto:Data: 2024-02-01T10:18:35.256 DEBUG:boto:Headers: {} 2024-02-01T10:18:35.256 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:35.256 DEBUG:boto:Port: 8000 2024-02-01T10:18:35.256 DEBUG:boto:Params: {} 2024-02-01T10:18:35.256 DEBUG:boto:Token: None 2024-02-01T10:18:35.257 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:35 GMT /vclfcn-14/myobj?versionId=null 2024-02-01T10:18:35.257 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:OpQmr6H9WTVsggoyPHyxIAITlNo= 2024-02-01T10:18:35.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:OpQmr6H9WTVsggoyPHyxIAITlNo=', 'Content-Length': '0'} 2024-02-01T10:18:35.259 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b01a99db846e8dc6-0065bb6ffb-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:35.260 DEBUG:boto:path=/vclfcn-14/myobj 2024-02-01T10:18:35.260 DEBUG:boto:auth_path=/vclfcn-14/myobj 2024-02-01T10:18:35.260 DEBUG:boto:path=/vclfcn-14/myobj?versionId=null 2024-02-01T10:18:35.260 DEBUG:boto:auth_path=/vclfcn-14/myobj?versionId=null 2024-02-01T10:18:35.260 DEBUG:boto:Method: HEAD 2024-02-01T10:18:35.260 DEBUG:boto:Path: /vclfcn-14/myobj?versionId=null 2024-02-01T10:18:35.260 DEBUG:boto:Data: 2024-02-01T10:18:35.260 DEBUG:boto:Headers: {} 2024-02-01T10:18:35.260 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:35.261 DEBUG:boto:Port: 8000 2024-02-01T10:18:35.261 DEBUG:boto:Params: {} 2024-02-01T10:18:35.261 DEBUG:boto:Token: None 2024-02-01T10:18:35.261 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:35 GMT /vclfcn-14/myobj?versionId=null 2024-02-01T10:18:35.261 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:eQwkFRc624Riu3VsB2YDoRvsbf4= 2024-02-01T10:18:35.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:eQwkFRc624Riu3VsB2YDoRvsbf4=', 'Content-Length': '0'} 2024-02-01T10:18:35.263 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009db1684d5daa3c32-0065bb6ffb-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:35.263 DEBUG:boto:path=/vclfcn-14/myobj 2024-02-01T10:18:35.263 DEBUG:boto:auth_path=/vclfcn-14/myobj 2024-02-01T10:18:35.263 DEBUG:boto:path=/vclfcn-14/myobj?versionId=null 2024-02-01T10:18:35.263 DEBUG:boto:auth_path=/vclfcn-14/myobj?versionId=null 2024-02-01T10:18:35.263 DEBUG:boto:Method: HEAD 2024-02-01T10:18:35.263 DEBUG:boto:Path: /vclfcn-14/myobj?versionId=null 2024-02-01T10:18:35.263 DEBUG:boto:Data: 2024-02-01T10:18:35.264 DEBUG:boto:Headers: {} 2024-02-01T10:18:35.264 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:35.264 DEBUG:boto:Port: 8000 2024-02-01T10:18:35.264 DEBUG:boto:Params: {} 2024-02-01T10:18:35.264 DEBUG:boto:Token: None 2024-02-01T10:18:35.264 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:35 GMT /vclfcn-14/myobj?versionId=null 2024-02-01T10:18:35.264 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:eQwkFRc624Riu3VsB2YDoRvsbf4= 2024-02-01T10:18:35.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:eQwkFRc624Riu3VsB2YDoRvsbf4=', 'Content-Length': '0'} 2024-02-01T10:18:35.267 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000013361041b7f2ce6f-0065bb6ffb-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:35.267 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-02-01T10:18:35.267 DEBUG:boto:path=/vclfcn-14/myobj 2024-02-01T10:18:35.267 DEBUG:boto:auth_path=/vclfcn-14/myobj 2024-02-01T10:18:35.267 DEBUG:boto:path=/vclfcn-14/myobj?versionId=null 2024-02-01T10:18:35.267 DEBUG:boto:auth_path=/vclfcn-14/myobj?versionId=null 2024-02-01T10:18:35.268 DEBUG:boto:Method: GET 2024-02-01T10:18:35.268 DEBUG:boto:Path: /vclfcn-14/myobj?versionId=null 2024-02-01T10:18:35.268 DEBUG:boto:Data: 2024-02-01T10:18:35.268 DEBUG:boto:Headers: {} 2024-02-01T10:18:35.268 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:35.268 DEBUG:boto:Port: 8000 2024-02-01T10:18:35.268 DEBUG:boto:Params: {} 2024-02-01T10:18:35.268 DEBUG:boto:Token: None 2024-02-01T10:18:35.268 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:35 GMT /vclfcn-14/myobj?versionId=null 2024-02-01T10:18:35.268 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:OpQmr6H9WTVsggoyPHyxIAITlNo= 2024-02-01T10:18:35.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:OpQmr6H9WTVsggoyPHyxIAITlNo=', 'Content-Length': '0'} 2024-02-01T10:18:35.271 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000037603f56c23c903f-0065bb6ffb-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:35.312 DEBUG:boto:path=/vclfcn-14/myobj 2024-02-01T10:18:35.312 DEBUG:boto:auth_path=/vclfcn-14/myobj 2024-02-01T10:18:35.312 DEBUG:boto:path=/vclfcn-14/myobj?versionId=null 2024-02-01T10:18:35.312 DEBUG:boto:auth_path=/vclfcn-14/myobj?versionId=null 2024-02-01T10:18:35.313 DEBUG:boto:Method: GET 2024-02-01T10:18:35.313 DEBUG:boto:Path: /vclfcn-14/myobj?versionId=null 2024-02-01T10:18:35.313 DEBUG:boto:Data: 2024-02-01T10:18:35.313 DEBUG:boto:Headers: {} 2024-02-01T10:18:35.313 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:35.313 DEBUG:boto:Port: 8000 2024-02-01T10:18:35.313 DEBUG:boto:Params: {} 2024-02-01T10:18:35.313 DEBUG:boto:Token: None 2024-02-01T10:18:35.313 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:35 GMT /vclfcn-14/myobj?versionId=null 2024-02-01T10:18:35.314 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:OpQmr6H9WTVsggoyPHyxIAITlNo= 2024-02-01T10:18:35.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:OpQmr6H9WTVsggoyPHyxIAITlNo=', 'Content-Length': '0'} 2024-02-01T10:18:35.317 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003f8d04da88403cca-0065bb6ffb-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:35.317 DEBUG:boto:path=/vclfcn-14/ 2024-02-01T10:18:35.317 DEBUG:boto:auth_path=/vclfcn-14/ 2024-02-01T10:18:35.317 DEBUG:boto:path=/vclfcn-14/?prefix=myobj 2024-02-01T10:18:35.317 DEBUG:boto:auth_path=/vclfcn-14/?prefix=myobj 2024-02-01T10:18:35.317 DEBUG:boto:Method: GET 2024-02-01T10:18:35.318 DEBUG:boto:Path: /vclfcn-14/?prefix=myobj 2024-02-01T10:18:35.318 DEBUG:boto:Data: 2024-02-01T10:18:35.318 DEBUG:boto:Headers: {} 2024-02-01T10:18:35.318 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:35.318 DEBUG:boto:Port: 8000 2024-02-01T10:18:35.318 DEBUG:boto:Params: {} 2024-02-01T10:18:35.318 DEBUG:boto:Token: None 2024-02-01T10:18:35.318 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:35 GMT /vclfcn-14/ 2024-02-01T10:18:35.318 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tDe1Ts8PaHe0WXxBOmiz2PN75k8= 2024-02-01T10:18:35.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tDe1Ts8PaHe0WXxBOmiz2PN75k8=', 'Content-Length': '0'} 2024-02-01T10:18:35.323 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab21d3ae471bf705-0065bb6ffb-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:35.323 DEBUG:boto:b'vclfcn-14myobj1000falsemyobj2024-02-01T10:18:13.794Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:35.323 DEBUG:boto:path=/vclfcn-14/ 2024-02-01T10:18:35.323 DEBUG:boto:auth_path=/vclfcn-14/ 2024-02-01T10:18:35.324 DEBUG:boto:path=/vclfcn-14/?prefix=myobj 2024-02-01T10:18:35.324 DEBUG:boto:auth_path=/vclfcn-14/?prefix=myobj 2024-02-01T10:18:35.324 DEBUG:boto:Method: GET 2024-02-01T10:18:35.324 DEBUG:boto:Path: /vclfcn-14/?prefix=myobj 2024-02-01T10:18:35.324 DEBUG:boto:Data: 2024-02-01T10:18:35.324 DEBUG:boto:Headers: {} 2024-02-01T10:18:35.324 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:35.324 DEBUG:boto:Port: 8000 2024-02-01T10:18:35.324 DEBUG:boto:Params: {} 2024-02-01T10:18:35.324 DEBUG:boto:Token: None 2024-02-01T10:18:35.325 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:35 GMT /vclfcn-14/ 2024-02-01T10:18:35.325 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tDe1Ts8PaHe0WXxBOmiz2PN75k8= 2024-02-01T10:18:35.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tDe1Ts8PaHe0WXxBOmiz2PN75k8=', 'Content-Length': '0'} 2024-02-01T10:18:35.329 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b8577dde0ebf01c-0065bb6ffb-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:35.372 DEBUG:boto:b'vclfcn-14myobj1000falsemyobj2024-02-01T10:18:13.794Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:35.373 DEBUG:boto:path=/vclfcn-14/myobj 2024-02-01T10:18:35.373 DEBUG:boto:auth_path=/vclfcn-14/myobj 2024-02-01T10:18:35.373 DEBUG:boto:Method: HEAD 2024-02-01T10:18:35.373 DEBUG:boto:Path: /vclfcn-14/myobj 2024-02-01T10:18:35.373 DEBUG:boto:Data: 2024-02-01T10:18:35.373 DEBUG:boto:Headers: {} 2024-02-01T10:18:35.373 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:35.373 DEBUG:boto:Port: 8000 2024-02-01T10:18:35.373 DEBUG:boto:Params: {} 2024-02-01T10:18:35.374 DEBUG:boto:Token: None 2024-02-01T10:18:35.374 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:35 GMT /vclfcn-14/myobj 2024-02-01T10:18:35.374 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:rRpfMdImLewvuGvdHt4/1T5Ju+o= 2024-02-01T10:18:35.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:rRpfMdImLewvuGvdHt4/1T5Ju+o=', 'Content-Length': '0'} 2024-02-01T10:18:35.377 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000003acc3a1780c769a-0065bb6ffb-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:35.377 DEBUG:boto:path=/vclfcn-14/myobj 2024-02-01T10:18:35.377 DEBUG:boto:auth_path=/vclfcn-14/myobj 2024-02-01T10:18:35.377 DEBUG:boto:Method: HEAD 2024-02-01T10:18:35.377 DEBUG:boto:Path: /vclfcn-14/myobj 2024-02-01T10:18:35.377 DEBUG:boto:Data: 2024-02-01T10:18:35.377 DEBUG:boto:Headers: {} 2024-02-01T10:18:35.378 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:35.378 DEBUG:boto:Port: 8000 2024-02-01T10:18:35.378 DEBUG:boto:Params: {} 2024-02-01T10:18:35.378 DEBUG:boto:Token: None 2024-02-01T10:18:35.378 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:35 GMT /vclfcn-14/myobj 2024-02-01T10:18:35.378 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:rRpfMdImLewvuGvdHt4/1T5Ju+o= 2024-02-01T10:18:35.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:rRpfMdImLewvuGvdHt4/1T5Ju+o=', 'Content-Length': '0'} 2024-02-01T10:18:35.381 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e0af73d5bb0174d1-0065bb6ffb-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:35.381 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-02-01T10:18:35.381 DEBUG:boto:path=/vclfcn-14/myobj 2024-02-01T10:18:35.381 DEBUG:boto:auth_path=/vclfcn-14/myobj 2024-02-01T10:18:35.382 DEBUG:boto:Method: GET 2024-02-01T10:18:35.382 DEBUG:boto:Path: /vclfcn-14/myobj 2024-02-01T10:18:35.382 DEBUG:boto:Data: 2024-02-01T10:18:35.382 DEBUG:boto:Headers: {} 2024-02-01T10:18:35.382 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:35.382 DEBUG:boto:Port: 8000 2024-02-01T10:18:35.382 DEBUG:boto:Params: {} 2024-02-01T10:18:35.382 DEBUG:boto:Token: None 2024-02-01T10:18:35.382 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:35 GMT /vclfcn-14/myobj 2024-02-01T10:18:35.383 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/pJWAqzDl1FLVz1JOwp0xEair0U= 2024-02-01T10:18:35.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/pJWAqzDl1FLVz1JOwp0xEair0U=', 'Content-Length': '0'} 2024-02-01T10:18:35.385 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000089f9d1ba388b26d5-0065bb6ffb-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:35.428 DEBUG:boto:path=/vclfcn-14/myobj 2024-02-01T10:18:35.428 DEBUG:boto:auth_path=/vclfcn-14/myobj 2024-02-01T10:18:35.428 DEBUG:boto:Method: GET 2024-02-01T10:18:35.428 DEBUG:boto:Path: /vclfcn-14/myobj 2024-02-01T10:18:35.428 DEBUG:boto:Data: 2024-02-01T10:18:35.429 DEBUG:boto:Headers: {} 2024-02-01T10:18:35.429 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:35.429 DEBUG:boto:Port: 8000 2024-02-01T10:18:35.429 DEBUG:boto:Params: {} 2024-02-01T10:18:35.429 DEBUG:boto:Token: None 2024-02-01T10:18:35.429 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:35 GMT /vclfcn-14/myobj 2024-02-01T10:18:35.429 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/pJWAqzDl1FLVz1JOwp0xEair0U= 2024-02-01T10:18:35.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/pJWAqzDl1FLVz1JOwp0xEair0U=', 'Content-Length': '0'} 2024-02-01T10:18:35.432 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009fe687702ff42794-0065bb6ffb-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:35.433 DEBUG:boto:path=/vclfcn-14/ 2024-02-01T10:18:35.433 DEBUG:boto:auth_path=/vclfcn-14/ 2024-02-01T10:18:35.433 DEBUG:boto:path=/vclfcn-14/?prefix=myobj 2024-02-01T10:18:35.433 DEBUG:boto:auth_path=/vclfcn-14/?prefix=myobj 2024-02-01T10:18:35.433 DEBUG:boto:Method: GET 2024-02-01T10:18:35.433 DEBUG:boto:Path: /vclfcn-14/?prefix=myobj 2024-02-01T10:18:35.433 DEBUG:boto:Data: 2024-02-01T10:18:35.434 DEBUG:boto:Headers: {} 2024-02-01T10:18:35.434 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:35.434 DEBUG:boto:Port: 8000 2024-02-01T10:18:35.434 DEBUG:boto:Params: {} 2024-02-01T10:18:35.434 DEBUG:boto:Token: None 2024-02-01T10:18:35.434 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:35 GMT /vclfcn-14/ 2024-02-01T10:18:35.434 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tDe1Ts8PaHe0WXxBOmiz2PN75k8= 2024-02-01T10:18:35.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tDe1Ts8PaHe0WXxBOmiz2PN75k8=', 'Content-Length': '0'} 2024-02-01T10:18:35.438 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084403204942915ff-0065bb6ffb-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:35.480 DEBUG:boto:b'vclfcn-14myobj1000falsemyobj2024-02-01T10:18:13.794Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:35.481 DEBUG:boto:path=/vclfcn-14/ 2024-02-01T10:18:35.481 DEBUG:boto:auth_path=/vclfcn-14/ 2024-02-01T10:18:35.481 DEBUG:boto:path=/vclfcn-14/?prefix=myobj 2024-02-01T10:18:35.481 DEBUG:boto:auth_path=/vclfcn-14/?prefix=myobj 2024-02-01T10:18:35.481 DEBUG:boto:Method: GET 2024-02-01T10:18:35.481 DEBUG:boto:Path: /vclfcn-14/?prefix=myobj 2024-02-01T10:18:35.481 DEBUG:boto:Data: 2024-02-01T10:18:35.481 DEBUG:boto:Headers: {} 2024-02-01T10:18:35.481 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:35.481 DEBUG:boto:Port: 8000 2024-02-01T10:18:35.481 DEBUG:boto:Params: {} 2024-02-01T10:18:35.482 DEBUG:boto:Token: None 2024-02-01T10:18:35.482 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:35 GMT /vclfcn-14/ 2024-02-01T10:18:35.482 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tDe1Ts8PaHe0WXxBOmiz2PN75k8= 2024-02-01T10:18:35.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tDe1Ts8PaHe0WXxBOmiz2PN75k8=', 'Content-Length': '0'} 2024-02-01T10:18:35.486 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000586cde66c5486012-0065bb6ffb-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:35.486 DEBUG:boto:b'vclfcn-14myobj1000falsemyobj2024-02-01T10:18:13.794Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:35.487 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-14 zones={test-zone1, test-zone2} 2024-02-01T10:18:35.487 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:18:35.487 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:18:35.487 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:18:35.537 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:18:35.537 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:18:35.601 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:18:35.598+0000 7f17758be980 1 bucket sync caught up with source: 2024-02-01T10:18:35.602 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, 00000000001.10.6, , , , , , , , , ] 2024-02-01T10:18:35.602 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, 00000000001.10.6, , , , , , , , , ] 2024-02-01T10:18:35.602 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:18:35.598+0000 7f17758be980 0 bucket checkpoint complete 2024-02-01T10:18:35.623 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-14 zones={test-zone3, test-zone2} 2024-02-01T10:18:35.624 DEBUG:boto:path=/vclfcn-14/ 2024-02-01T10:18:35.624 DEBUG:boto:auth_path=/vclfcn-14/ 2024-02-01T10:18:35.624 DEBUG:boto:Method: HEAD 2024-02-01T10:18:35.624 DEBUG:boto:Path: /vclfcn-14/ 2024-02-01T10:18:35.624 DEBUG:boto:Data: 2024-02-01T10:18:35.624 DEBUG:boto:Headers: {} 2024-02-01T10:18:35.624 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:35.624 DEBUG:boto:Port: 8001 2024-02-01T10:18:35.624 DEBUG:boto:Params: {} 2024-02-01T10:18:35.625 DEBUG:boto:Token: None 2024-02-01T10:18:35.625 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:35 GMT /vclfcn-14/ 2024-02-01T10:18:35.625 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:naJXyAJ9hwPmMhefifG5KDL673U= 2024-02-01T10:18:35.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:naJXyAJ9hwPmMhefifG5KDL673U=', 'Content-Length': '0'} 2024-02-01T10:18:35.627 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c35f5a8344e48bc1-0065bb6ffb-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:18:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:35.628 DEBUG:boto:path=/vclfcn-14/ 2024-02-01T10:18:35.628 DEBUG:boto:auth_path=/vclfcn-14/ 2024-02-01T10:18:35.628 DEBUG:boto:Method: HEAD 2024-02-01T10:18:35.628 DEBUG:boto:Path: /vclfcn-14/ 2024-02-01T10:18:35.628 DEBUG:boto:Data: 2024-02-01T10:18:35.628 DEBUG:boto:Headers: {} 2024-02-01T10:18:35.628 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:35.628 DEBUG:boto:Port: 8000 2024-02-01T10:18:35.628 DEBUG:boto:Params: {} 2024-02-01T10:18:35.628 DEBUG:boto:Token: None 2024-02-01T10:18:35.629 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:35 GMT /vclfcn-14/ 2024-02-01T10:18:35.629 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:naJXyAJ9hwPmMhefifG5KDL673U= 2024-02-01T10:18:35.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:naJXyAJ9hwPmMhefifG5KDL673U=', 'Content-Length': '0'} 2024-02-01T10:18:35.633 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000001b51b8dab73a2cd-0065bb6ffb-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:18:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:35.633 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:18:35.633 DEBUG:boto:path=/vclfcn-14/ 2024-02-01T10:18:35.633 DEBUG:boto:auth_path=/vclfcn-14/ 2024-02-01T10:18:35.633 DEBUG:boto:path=/vclfcn-14/?versions&max-keys=999 2024-02-01T10:18:35.633 DEBUG:boto:auth_path=/vclfcn-14/?versions&max-keys=999 2024-02-01T10:18:35.633 DEBUG:boto:Method: GET 2024-02-01T10:18:35.633 DEBUG:boto:Path: /vclfcn-14/?versions&max-keys=999 2024-02-01T10:18:35.634 DEBUG:boto:Data: 2024-02-01T10:18:35.634 DEBUG:boto:Headers: {} 2024-02-01T10:18:35.634 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:35.634 DEBUG:boto:Port: 8001 2024-02-01T10:18:35.634 DEBUG:boto:Params: {} 2024-02-01T10:18:35.634 DEBUG:boto:Token: None 2024-02-01T10:18:35.634 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:35 GMT /vclfcn-14/?versions 2024-02-01T10:18:35.634 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mM5mGKPJoZyWkcbjXzvVuvektWY= 2024-02-01T10:18:35.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mM5mGKPJoZyWkcbjXzvVuvektWY=', 'Content-Length': '0'} 2024-02-01T10:18:35.636 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec9e506268e968d1-0065bb6ffb-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:35.637 DEBUG:boto:b'vclfcn-14999falsemyobjnulltrue2024-02-01T10:18:13.794Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:35.637 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-02-01T10:18:35.637 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:18:35.637 DEBUG:boto:path=/vclfcn-14/ 2024-02-01T10:18:35.637 DEBUG:boto:auth_path=/vclfcn-14/ 2024-02-01T10:18:35.637 DEBUG:boto:path=/vclfcn-14/?versions&max-keys=999 2024-02-01T10:18:35.638 DEBUG:boto:auth_path=/vclfcn-14/?versions&max-keys=999 2024-02-01T10:18:35.638 DEBUG:boto:Method: GET 2024-02-01T10:18:35.638 DEBUG:boto:Path: /vclfcn-14/?versions&max-keys=999 2024-02-01T10:18:35.638 DEBUG:boto:Data: 2024-02-01T10:18:35.638 DEBUG:boto:Headers: {} 2024-02-01T10:18:35.638 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:35.638 DEBUG:boto:Port: 8000 2024-02-01T10:18:35.638 DEBUG:boto:Params: {} 2024-02-01T10:18:35.638 DEBUG:boto:Token: None 2024-02-01T10:18:35.638 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:35 GMT /vclfcn-14/?versions 2024-02-01T10:18:35.639 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mM5mGKPJoZyWkcbjXzvVuvektWY= 2024-02-01T10:18:35.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mM5mGKPJoZyWkcbjXzvVuvektWY=', 'Content-Length': '0'} 2024-02-01T10:18:35.643 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab0846192c45cad9-0065bb6ffb-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:35.644 DEBUG:boto:b'vclfcn-14999falsemyobjnulltrue2024-02-01T10:18:13.794Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:35.644 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-02-01T10:18:35.644 DEBUG:boto:path=/vclfcn-14/ 2024-02-01T10:18:35.644 DEBUG:boto:auth_path=/vclfcn-14/ 2024-02-01T10:18:35.644 DEBUG:boto:path=/vclfcn-14/?versions&max-keys=999 2024-02-01T10:18:35.644 DEBUG:boto:auth_path=/vclfcn-14/?versions&max-keys=999 2024-02-01T10:18:35.645 DEBUG:boto:Method: GET 2024-02-01T10:18:35.645 DEBUG:boto:Path: /vclfcn-14/?versions&max-keys=999 2024-02-01T10:18:35.645 DEBUG:boto:Data: 2024-02-01T10:18:35.645 DEBUG:boto:Headers: {} 2024-02-01T10:18:35.645 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:35.645 DEBUG:boto:Port: 8001 2024-02-01T10:18:35.645 DEBUG:boto:Params: {} 2024-02-01T10:18:35.645 DEBUG:boto:Token: None 2024-02-01T10:18:35.645 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:35 GMT /vclfcn-14/?versions 2024-02-01T10:18:35.646 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mM5mGKPJoZyWkcbjXzvVuvektWY= 2024-02-01T10:18:35.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mM5mGKPJoZyWkcbjXzvVuvektWY=', 'Content-Length': '0'} 2024-02-01T10:18:35.648 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b0ffd560143a0dc-0065bb6ffb-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:35.648 DEBUG:boto:b'vclfcn-14999falsemyobjnulltrue2024-02-01T10:18:13.794Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:35.648 DEBUG:boto:path=/vclfcn-14/ 2024-02-01T10:18:35.648 DEBUG:boto:auth_path=/vclfcn-14/ 2024-02-01T10:18:35.649 DEBUG:boto:path=/vclfcn-14/?versions&max-keys=999 2024-02-01T10:18:35.649 DEBUG:boto:auth_path=/vclfcn-14/?versions&max-keys=999 2024-02-01T10:18:35.649 DEBUG:boto:Method: GET 2024-02-01T10:18:35.649 DEBUG:boto:Path: /vclfcn-14/?versions&max-keys=999 2024-02-01T10:18:35.649 DEBUG:boto:Data: 2024-02-01T10:18:35.649 DEBUG:boto:Headers: {} 2024-02-01T10:18:35.649 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:35.649 DEBUG:boto:Port: 8000 2024-02-01T10:18:35.650 DEBUG:boto:Params: {} 2024-02-01T10:18:35.650 DEBUG:boto:Token: None 2024-02-01T10:18:35.650 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:35 GMT /vclfcn-14/?versions 2024-02-01T10:18:35.650 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mM5mGKPJoZyWkcbjXzvVuvektWY= 2024-02-01T10:18:35.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mM5mGKPJoZyWkcbjXzvVuvektWY=', 'Content-Length': '0'} 2024-02-01T10:18:35.654 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000000e9445521658227-0065bb6ffb-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:35.700 DEBUG:boto:b'vclfcn-14999falsemyobjnulltrue2024-02-01T10:18:13.794Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:35.700 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-02-01T10:18:35.701 DEBUG:boto:path=/vclfcn-14/myobj 2024-02-01T10:18:35.701 DEBUG:boto:auth_path=/vclfcn-14/myobj 2024-02-01T10:18:35.701 DEBUG:boto:path=/vclfcn-14/myobj?versionId=null 2024-02-01T10:18:35.701 DEBUG:boto:auth_path=/vclfcn-14/myobj?versionId=null 2024-02-01T10:18:35.701 DEBUG:boto:Method: GET 2024-02-01T10:18:35.701 DEBUG:boto:Path: /vclfcn-14/myobj?versionId=null 2024-02-01T10:18:35.701 DEBUG:boto:Data: 2024-02-01T10:18:35.701 DEBUG:boto:Headers: {} 2024-02-01T10:18:35.701 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:35.702 DEBUG:boto:Port: 8001 2024-02-01T10:18:35.702 DEBUG:boto:Params: {} 2024-02-01T10:18:35.702 DEBUG:boto:Token: None 2024-02-01T10:18:35.702 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:35 GMT /vclfcn-14/myobj?versionId=null 2024-02-01T10:18:35.702 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:OpQmr6H9WTVsggoyPHyxIAITlNo= 2024-02-01T10:18:35.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:OpQmr6H9WTVsggoyPHyxIAITlNo=', 'Content-Length': '0'} 2024-02-01T10:18:35.705 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fccbe1c7e77c7140-0065bb6ffb-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:35.706 DEBUG:boto:path=/vclfcn-14/myobj 2024-02-01T10:18:35.706 DEBUG:boto:auth_path=/vclfcn-14/myobj 2024-02-01T10:18:35.706 DEBUG:boto:path=/vclfcn-14/myobj?versionId=null 2024-02-01T10:18:35.706 DEBUG:boto:auth_path=/vclfcn-14/myobj?versionId=null 2024-02-01T10:18:35.706 DEBUG:boto:Method: GET 2024-02-01T10:18:35.706 DEBUG:boto:Path: /vclfcn-14/myobj?versionId=null 2024-02-01T10:18:35.706 DEBUG:boto:Data: 2024-02-01T10:18:35.706 DEBUG:boto:Headers: {} 2024-02-01T10:18:35.706 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:35.706 DEBUG:boto:Port: 8000 2024-02-01T10:18:35.706 DEBUG:boto:Params: {} 2024-02-01T10:18:35.707 DEBUG:boto:Token: None 2024-02-01T10:18:35.707 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:35 GMT /vclfcn-14/myobj?versionId=null 2024-02-01T10:18:35.707 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:OpQmr6H9WTVsggoyPHyxIAITlNo= 2024-02-01T10:18:35.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:OpQmr6H9WTVsggoyPHyxIAITlNo=', 'Content-Length': '0'} 2024-02-01T10:18:35.710 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008c64754afd11a43a-0065bb6ffb-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:35.710 DEBUG:boto:path=/vclfcn-14/myobj 2024-02-01T10:18:35.710 DEBUG:boto:auth_path=/vclfcn-14/myobj 2024-02-01T10:18:35.710 DEBUG:boto:path=/vclfcn-14/myobj?versionId=null 2024-02-01T10:18:35.711 DEBUG:boto:auth_path=/vclfcn-14/myobj?versionId=null 2024-02-01T10:18:35.711 DEBUG:boto:Method: HEAD 2024-02-01T10:18:35.711 DEBUG:boto:Path: /vclfcn-14/myobj?versionId=null 2024-02-01T10:18:35.711 DEBUG:boto:Data: 2024-02-01T10:18:35.711 DEBUG:boto:Headers: {} 2024-02-01T10:18:35.711 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:35.711 DEBUG:boto:Port: 8001 2024-02-01T10:18:35.711 DEBUG:boto:Params: {} 2024-02-01T10:18:35.711 DEBUG:boto:Token: None 2024-02-01T10:18:35.711 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:35 GMT /vclfcn-14/myobj?versionId=null 2024-02-01T10:18:35.712 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:eQwkFRc624Riu3VsB2YDoRvsbf4= 2024-02-01T10:18:35.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:eQwkFRc624Riu3VsB2YDoRvsbf4=', 'Content-Length': '0'} 2024-02-01T10:18:35.714 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000481e462456a602f9-0065bb6ffb-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:35.715 DEBUG:boto:path=/vclfcn-14/myobj 2024-02-01T10:18:35.715 DEBUG:boto:auth_path=/vclfcn-14/myobj 2024-02-01T10:18:35.715 DEBUG:boto:path=/vclfcn-14/myobj?versionId=null 2024-02-01T10:18:35.715 DEBUG:boto:auth_path=/vclfcn-14/myobj?versionId=null 2024-02-01T10:18:35.715 DEBUG:boto:Method: HEAD 2024-02-01T10:18:35.715 DEBUG:boto:Path: /vclfcn-14/myobj?versionId=null 2024-02-01T10:18:35.715 DEBUG:boto:Data: 2024-02-01T10:18:35.715 DEBUG:boto:Headers: {} 2024-02-01T10:18:35.715 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:35.715 DEBUG:boto:Port: 8000 2024-02-01T10:18:35.716 DEBUG:boto:Params: {} 2024-02-01T10:18:35.716 DEBUG:boto:Token: None 2024-02-01T10:18:35.716 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:35 GMT /vclfcn-14/myobj?versionId=null 2024-02-01T10:18:35.716 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:eQwkFRc624Riu3VsB2YDoRvsbf4= 2024-02-01T10:18:35.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:eQwkFRc624Riu3VsB2YDoRvsbf4=', 'Content-Length': '0'} 2024-02-01T10:18:35.718 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008c1dc18cbaa73360-0065bb6ffb-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:35.719 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-02-01T10:18:35.719 DEBUG:boto:path=/vclfcn-14/myobj 2024-02-01T10:18:35.719 DEBUG:boto:auth_path=/vclfcn-14/myobj 2024-02-01T10:18:35.719 DEBUG:boto:path=/vclfcn-14/myobj?versionId=null 2024-02-01T10:18:35.719 DEBUG:boto:auth_path=/vclfcn-14/myobj?versionId=null 2024-02-01T10:18:35.719 DEBUG:boto:Method: GET 2024-02-01T10:18:35.719 DEBUG:boto:Path: /vclfcn-14/myobj?versionId=null 2024-02-01T10:18:35.720 DEBUG:boto:Data: 2024-02-01T10:18:35.720 DEBUG:boto:Headers: {} 2024-02-01T10:18:35.720 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:35.720 DEBUG:boto:Port: 8001 2024-02-01T10:18:35.720 DEBUG:boto:Params: {} 2024-02-01T10:18:35.720 DEBUG:boto:Token: None 2024-02-01T10:18:35.720 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:35 GMT /vclfcn-14/myobj?versionId=null 2024-02-01T10:18:35.720 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:OpQmr6H9WTVsggoyPHyxIAITlNo= 2024-02-01T10:18:35.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:OpQmr6H9WTVsggoyPHyxIAITlNo=', 'Content-Length': '0'} 2024-02-01T10:18:35.723 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c7f48a47ef762e67-0065bb6ffb-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:35.768 DEBUG:boto:path=/vclfcn-14/myobj 2024-02-01T10:18:35.768 DEBUG:boto:auth_path=/vclfcn-14/myobj 2024-02-01T10:18:35.768 DEBUG:boto:path=/vclfcn-14/myobj?versionId=null 2024-02-01T10:18:35.769 DEBUG:boto:auth_path=/vclfcn-14/myobj?versionId=null 2024-02-01T10:18:35.769 DEBUG:boto:Method: GET 2024-02-01T10:18:35.769 DEBUG:boto:Path: /vclfcn-14/myobj?versionId=null 2024-02-01T10:18:35.769 DEBUG:boto:Data: 2024-02-01T10:18:35.769 DEBUG:boto:Headers: {} 2024-02-01T10:18:35.769 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:35.769 DEBUG:boto:Port: 8000 2024-02-01T10:18:35.769 DEBUG:boto:Params: {} 2024-02-01T10:18:35.769 DEBUG:boto:Token: None 2024-02-01T10:18:35.769 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:35 GMT /vclfcn-14/myobj?versionId=null 2024-02-01T10:18:35.770 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:OpQmr6H9WTVsggoyPHyxIAITlNo= 2024-02-01T10:18:35.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:OpQmr6H9WTVsggoyPHyxIAITlNo=', 'Content-Length': '0'} 2024-02-01T10:18:35.773 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d38c393367d14cde-0065bb6ffb-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:35.773 DEBUG:boto:path=/vclfcn-14/ 2024-02-01T10:18:35.773 DEBUG:boto:auth_path=/vclfcn-14/ 2024-02-01T10:18:35.773 DEBUG:boto:path=/vclfcn-14/?prefix=myobj 2024-02-01T10:18:35.773 DEBUG:boto:auth_path=/vclfcn-14/?prefix=myobj 2024-02-01T10:18:35.773 DEBUG:boto:Method: GET 2024-02-01T10:18:35.773 DEBUG:boto:Path: /vclfcn-14/?prefix=myobj 2024-02-01T10:18:35.773 DEBUG:boto:Data: 2024-02-01T10:18:35.774 DEBUG:boto:Headers: {} 2024-02-01T10:18:35.774 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:35.774 DEBUG:boto:Port: 8001 2024-02-01T10:18:35.774 DEBUG:boto:Params: {} 2024-02-01T10:18:35.774 DEBUG:boto:Token: None 2024-02-01T10:18:35.774 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:35 GMT /vclfcn-14/ 2024-02-01T10:18:35.774 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tDe1Ts8PaHe0WXxBOmiz2PN75k8= 2024-02-01T10:18:35.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tDe1Ts8PaHe0WXxBOmiz2PN75k8=', 'Content-Length': '0'} 2024-02-01T10:18:35.778 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000807e97baec3e9ac0-0065bb6ffb-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:35.779 DEBUG:boto:b'vclfcn-14myobj1000falsemyobj2024-02-01T10:18:13.794Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:35.779 DEBUG:boto:path=/vclfcn-14/ 2024-02-01T10:18:35.779 DEBUG:boto:auth_path=/vclfcn-14/ 2024-02-01T10:18:35.779 DEBUG:boto:path=/vclfcn-14/?prefix=myobj 2024-02-01T10:18:35.779 DEBUG:boto:auth_path=/vclfcn-14/?prefix=myobj 2024-02-01T10:18:35.780 DEBUG:boto:Method: GET 2024-02-01T10:18:35.780 DEBUG:boto:Path: /vclfcn-14/?prefix=myobj 2024-02-01T10:18:35.780 DEBUG:boto:Data: 2024-02-01T10:18:35.780 DEBUG:boto:Headers: {} 2024-02-01T10:18:35.780 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:35.780 DEBUG:boto:Port: 8000 2024-02-01T10:18:35.780 DEBUG:boto:Params: {} 2024-02-01T10:18:35.780 DEBUG:boto:Token: None 2024-02-01T10:18:35.780 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:35 GMT /vclfcn-14/ 2024-02-01T10:18:35.780 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tDe1Ts8PaHe0WXxBOmiz2PN75k8= 2024-02-01T10:18:35.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tDe1Ts8PaHe0WXxBOmiz2PN75k8=', 'Content-Length': '0'} 2024-02-01T10:18:35.785 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d35daf1609b2d82-0065bb6ffb-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:35.828 DEBUG:boto:b'vclfcn-14myobj1000falsemyobj2024-02-01T10:18:13.794Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:35.828 DEBUG:boto:path=/vclfcn-14/myobj 2024-02-01T10:18:35.829 DEBUG:boto:auth_path=/vclfcn-14/myobj 2024-02-01T10:18:35.829 DEBUG:boto:Method: HEAD 2024-02-01T10:18:35.829 DEBUG:boto:Path: /vclfcn-14/myobj 2024-02-01T10:18:35.829 DEBUG:boto:Data: 2024-02-01T10:18:35.829 DEBUG:boto:Headers: {} 2024-02-01T10:18:35.829 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:35.829 DEBUG:boto:Port: 8001 2024-02-01T10:18:35.829 DEBUG:boto:Params: {} 2024-02-01T10:18:35.829 DEBUG:boto:Token: None 2024-02-01T10:18:35.830 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:35 GMT /vclfcn-14/myobj 2024-02-01T10:18:35.830 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:rRpfMdImLewvuGvdHt4/1T5Ju+o= 2024-02-01T10:18:35.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:rRpfMdImLewvuGvdHt4/1T5Ju+o=', 'Content-Length': '0'} 2024-02-01T10:18:35.833 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e8e1f61a7244742c-0065bb6ffb-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:35.833 DEBUG:boto:path=/vclfcn-14/myobj 2024-02-01T10:18:35.833 DEBUG:boto:auth_path=/vclfcn-14/myobj 2024-02-01T10:18:35.833 DEBUG:boto:Method: HEAD 2024-02-01T10:18:35.833 DEBUG:boto:Path: /vclfcn-14/myobj 2024-02-01T10:18:35.833 DEBUG:boto:Data: 2024-02-01T10:18:35.833 DEBUG:boto:Headers: {} 2024-02-01T10:18:35.833 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:35.833 DEBUG:boto:Port: 8000 2024-02-01T10:18:35.834 DEBUG:boto:Params: {} 2024-02-01T10:18:35.834 DEBUG:boto:Token: None 2024-02-01T10:18:35.834 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:35 GMT /vclfcn-14/myobj 2024-02-01T10:18:35.834 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:rRpfMdImLewvuGvdHt4/1T5Ju+o= 2024-02-01T10:18:35.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:rRpfMdImLewvuGvdHt4/1T5Ju+o=', 'Content-Length': '0'} 2024-02-01T10:18:35.837 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000089f094a47b9d7e15-0065bb6ffb-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:35.837 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-02-01T10:18:35.837 DEBUG:boto:path=/vclfcn-14/myobj 2024-02-01T10:18:35.837 DEBUG:boto:auth_path=/vclfcn-14/myobj 2024-02-01T10:18:35.837 DEBUG:boto:Method: GET 2024-02-01T10:18:35.838 DEBUG:boto:Path: /vclfcn-14/myobj 2024-02-01T10:18:35.838 DEBUG:boto:Data: 2024-02-01T10:18:35.838 DEBUG:boto:Headers: {} 2024-02-01T10:18:35.838 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:35.838 DEBUG:boto:Port: 8001 2024-02-01T10:18:35.838 DEBUG:boto:Params: {} 2024-02-01T10:18:35.838 DEBUG:boto:Token: None 2024-02-01T10:18:35.838 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:35 GMT /vclfcn-14/myobj 2024-02-01T10:18:35.838 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/pJWAqzDl1FLVz1JOwp0xEair0U= 2024-02-01T10:18:35.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/pJWAqzDl1FLVz1JOwp0xEair0U=', 'Content-Length': '0'} 2024-02-01T10:18:35.841 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c2fb37c76a2c4d02-0065bb6ffb-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:35.884 DEBUG:boto:path=/vclfcn-14/myobj 2024-02-01T10:18:35.884 DEBUG:boto:auth_path=/vclfcn-14/myobj 2024-02-01T10:18:35.884 DEBUG:boto:Method: GET 2024-02-01T10:18:35.885 DEBUG:boto:Path: /vclfcn-14/myobj 2024-02-01T10:18:35.885 DEBUG:boto:Data: 2024-02-01T10:18:35.885 DEBUG:boto:Headers: {} 2024-02-01T10:18:35.885 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:35.885 DEBUG:boto:Port: 8000 2024-02-01T10:18:35.885 DEBUG:boto:Params: {} 2024-02-01T10:18:35.885 DEBUG:boto:Token: None 2024-02-01T10:18:35.885 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:35 GMT /vclfcn-14/myobj 2024-02-01T10:18:35.885 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/pJWAqzDl1FLVz1JOwp0xEair0U= 2024-02-01T10:18:35.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/pJWAqzDl1FLVz1JOwp0xEair0U=', 'Content-Length': '0'} 2024-02-01T10:18:35.888 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007b4ec33472aa1166-0065bb6ffb-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:35.889 DEBUG:boto:path=/vclfcn-14/ 2024-02-01T10:18:35.889 DEBUG:boto:auth_path=/vclfcn-14/ 2024-02-01T10:18:35.889 DEBUG:boto:path=/vclfcn-14/?prefix=myobj 2024-02-01T10:18:35.889 DEBUG:boto:auth_path=/vclfcn-14/?prefix=myobj 2024-02-01T10:18:35.889 DEBUG:boto:Method: GET 2024-02-01T10:18:35.889 DEBUG:boto:Path: /vclfcn-14/?prefix=myobj 2024-02-01T10:18:35.889 DEBUG:boto:Data: 2024-02-01T10:18:35.889 DEBUG:boto:Headers: {} 2024-02-01T10:18:35.889 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:35.889 DEBUG:boto:Port: 8001 2024-02-01T10:18:35.890 DEBUG:boto:Params: {} 2024-02-01T10:18:35.890 DEBUG:boto:Token: None 2024-02-01T10:18:35.890 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:35 GMT /vclfcn-14/ 2024-02-01T10:18:35.890 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tDe1Ts8PaHe0WXxBOmiz2PN75k8= 2024-02-01T10:18:35.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tDe1Ts8PaHe0WXxBOmiz2PN75k8=', 'Content-Length': '0'} 2024-02-01T10:18:35.894 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043be19c15c2280de-0065bb6ffb-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:35.936 DEBUG:boto:b'vclfcn-14myobj1000falsemyobj2024-02-01T10:18:13.794Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:35.936 DEBUG:boto:path=/vclfcn-14/ 2024-02-01T10:18:35.937 DEBUG:boto:auth_path=/vclfcn-14/ 2024-02-01T10:18:35.937 DEBUG:boto:path=/vclfcn-14/?prefix=myobj 2024-02-01T10:18:35.937 DEBUG:boto:auth_path=/vclfcn-14/?prefix=myobj 2024-02-01T10:18:35.937 DEBUG:boto:Method: GET 2024-02-01T10:18:35.937 DEBUG:boto:Path: /vclfcn-14/?prefix=myobj 2024-02-01T10:18:35.937 DEBUG:boto:Data: 2024-02-01T10:18:35.937 DEBUG:boto:Headers: {} 2024-02-01T10:18:35.937 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:35.937 DEBUG:boto:Port: 8000 2024-02-01T10:18:35.937 DEBUG:boto:Params: {} 2024-02-01T10:18:35.937 DEBUG:boto:Token: None 2024-02-01T10:18:35.938 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:35 GMT /vclfcn-14/ 2024-02-01T10:18:35.938 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tDe1Ts8PaHe0WXxBOmiz2PN75k8= 2024-02-01T10:18:35.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tDe1Ts8PaHe0WXxBOmiz2PN75k8=', 'Content-Length': '0'} 2024-02-01T10:18:35.942 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dec7c1e99dfc40a5-0065bb6ffb-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:35.943 DEBUG:boto:b'vclfcn-14myobj1000falsemyobj2024-02-01T10:18:13.794Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:35.943 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-14 zones={test-zone3, test-zone2} 2024-02-01T10:18:35.943 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:18:35.943 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:18:35.943 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:18:35.995 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:18:35.995 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:18:36.058 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:18:36.054+0000 7f8fcf1e5980 1 bucket sync caught up with source: 2024-02-01T10:18:36.058 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, 00000000001.12.6, , , , , , , , , ] 2024-02-01T10:18:36.058 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, 00000000001.12.6, , , , , , , , , ] 2024-02-01T10:18:36.058 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:18:36.054+0000 7f8fcf1e5980 0 bucket checkpoint complete 2024-02-01T10:18:36.081 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-15 zones={test-zone1, test-zone3} 2024-02-01T10:18:36.081 DEBUG:boto:path=/vclfcn-15/ 2024-02-01T10:18:36.082 DEBUG:boto:auth_path=/vclfcn-15/ 2024-02-01T10:18:36.082 DEBUG:boto:Method: HEAD 2024-02-01T10:18:36.082 DEBUG:boto:Path: /vclfcn-15/ 2024-02-01T10:18:36.082 DEBUG:boto:Data: 2024-02-01T10:18:36.082 DEBUG:boto:Headers: {} 2024-02-01T10:18:36.082 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:36.082 DEBUG:boto:Port: 8000 2024-02-01T10:18:36.082 DEBUG:boto:Params: {} 2024-02-01T10:18:36.082 DEBUG:boto:Token: None 2024-02-01T10:18:36.083 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:36 GMT /vclfcn-15/ 2024-02-01T10:18:36.083 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jk/JL1ux23WGtQtyK0/Sy+kqi+I= 2024-02-01T10:18:36.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jk/JL1ux23WGtQtyK0/Sy+kqi+I=', 'Content-Length': '0'} 2024-02-01T10:18:36.085 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ebb705bd4adb238c-0065bb6ffc-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:18:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:36.085 DEBUG:boto:path=/vclfcn-15/ 2024-02-01T10:18:36.085 DEBUG:boto:auth_path=/vclfcn-15/ 2024-02-01T10:18:36.085 DEBUG:boto:Method: HEAD 2024-02-01T10:18:36.086 DEBUG:boto:Path: /vclfcn-15/ 2024-02-01T10:18:36.086 DEBUG:boto:Data: 2024-02-01T10:18:36.086 DEBUG:boto:Headers: {} 2024-02-01T10:18:36.086 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:36.086 DEBUG:boto:Port: 8001 2024-02-01T10:18:36.086 DEBUG:boto:Params: {} 2024-02-01T10:18:36.086 DEBUG:boto:Token: None 2024-02-01T10:18:36.086 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:36 GMT /vclfcn-15/ 2024-02-01T10:18:36.086 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jk/JL1ux23WGtQtyK0/Sy+kqi+I= 2024-02-01T10:18:36.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jk/JL1ux23WGtQtyK0/Sy+kqi+I=', 'Content-Length': '0'} 2024-02-01T10:18:36.089 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000529304151997485d-0065bb6ffc-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:18:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:36.089 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:18:36.089 DEBUG:boto:path=/vclfcn-15/ 2024-02-01T10:18:36.089 DEBUG:boto:auth_path=/vclfcn-15/ 2024-02-01T10:18:36.089 DEBUG:boto:path=/vclfcn-15/?versions&max-keys=999 2024-02-01T10:18:36.089 DEBUG:boto:auth_path=/vclfcn-15/?versions&max-keys=999 2024-02-01T10:18:36.089 DEBUG:boto:Method: GET 2024-02-01T10:18:36.089 DEBUG:boto:Path: /vclfcn-15/?versions&max-keys=999 2024-02-01T10:18:36.090 DEBUG:boto:Data: 2024-02-01T10:18:36.090 DEBUG:boto:Headers: {} 2024-02-01T10:18:36.090 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:36.090 DEBUG:boto:Port: 8000 2024-02-01T10:18:36.090 DEBUG:boto:Params: {} 2024-02-01T10:18:36.090 DEBUG:boto:Token: None 2024-02-01T10:18:36.090 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:36 GMT /vclfcn-15/?versions 2024-02-01T10:18:36.090 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:I1ft7S6zbwwU5eoSKtL9w+uXDJ4= 2024-02-01T10:18:36.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:I1ft7S6zbwwU5eoSKtL9w+uXDJ4=', 'Content-Length': '0'} 2024-02-01T10:18:36.092 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fce746c9f417a9f0-0065bb6ffc-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:36.093 DEBUG:boto:b'vclfcn-15999falsemyobjnulltrue2024-02-01T10:18:13.821Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:36.093 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-02-01T10:18:36.093 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:18:36.093 DEBUG:boto:path=/vclfcn-15/ 2024-02-01T10:18:36.093 DEBUG:boto:auth_path=/vclfcn-15/ 2024-02-01T10:18:36.094 DEBUG:boto:path=/vclfcn-15/?versions&max-keys=999 2024-02-01T10:18:36.094 DEBUG:boto:auth_path=/vclfcn-15/?versions&max-keys=999 2024-02-01T10:18:36.094 DEBUG:boto:Method: GET 2024-02-01T10:18:36.094 DEBUG:boto:Path: /vclfcn-15/?versions&max-keys=999 2024-02-01T10:18:36.094 DEBUG:boto:Data: 2024-02-01T10:18:36.094 DEBUG:boto:Headers: {} 2024-02-01T10:18:36.094 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:36.094 DEBUG:boto:Port: 8001 2024-02-01T10:18:36.094 DEBUG:boto:Params: {} 2024-02-01T10:18:36.094 DEBUG:boto:Token: None 2024-02-01T10:18:36.095 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:36 GMT /vclfcn-15/?versions 2024-02-01T10:18:36.095 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:I1ft7S6zbwwU5eoSKtL9w+uXDJ4= 2024-02-01T10:18:36.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:I1ft7S6zbwwU5eoSKtL9w+uXDJ4=', 'Content-Length': '0'} 2024-02-01T10:18:36.097 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050468d4b4eb3d343-0065bb6ffc-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:36.097 DEBUG:boto:b'vclfcn-15999falsemyobjnulltrue2024-02-01T10:18:13.821Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:36.097 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-02-01T10:18:36.097 DEBUG:boto:path=/vclfcn-15/ 2024-02-01T10:18:36.098 DEBUG:boto:auth_path=/vclfcn-15/ 2024-02-01T10:18:36.098 DEBUG:boto:path=/vclfcn-15/?versions&max-keys=999 2024-02-01T10:18:36.098 DEBUG:boto:auth_path=/vclfcn-15/?versions&max-keys=999 2024-02-01T10:18:36.098 DEBUG:boto:Method: GET 2024-02-01T10:18:36.098 DEBUG:boto:Path: /vclfcn-15/?versions&max-keys=999 2024-02-01T10:18:36.098 DEBUG:boto:Data: 2024-02-01T10:18:36.098 DEBUG:boto:Headers: {} 2024-02-01T10:18:36.098 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:36.098 DEBUG:boto:Port: 8000 2024-02-01T10:18:36.099 DEBUG:boto:Params: {} 2024-02-01T10:18:36.099 DEBUG:boto:Token: None 2024-02-01T10:18:36.099 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:36 GMT /vclfcn-15/?versions 2024-02-01T10:18:36.099 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:I1ft7S6zbwwU5eoSKtL9w+uXDJ4= 2024-02-01T10:18:36.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:I1ft7S6zbwwU5eoSKtL9w+uXDJ4=', 'Content-Length': '0'} 2024-02-01T10:18:36.101 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000000af0973c46f183c-0065bb6ffc-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:36.144 DEBUG:boto:b'vclfcn-15999falsemyobjnulltrue2024-02-01T10:18:13.821Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:36.145 DEBUG:boto:path=/vclfcn-15/ 2024-02-01T10:18:36.145 DEBUG:boto:auth_path=/vclfcn-15/ 2024-02-01T10:18:36.145 DEBUG:boto:path=/vclfcn-15/?versions&max-keys=999 2024-02-01T10:18:36.145 DEBUG:boto:auth_path=/vclfcn-15/?versions&max-keys=999 2024-02-01T10:18:36.145 DEBUG:boto:Method: GET 2024-02-01T10:18:36.145 DEBUG:boto:Path: /vclfcn-15/?versions&max-keys=999 2024-02-01T10:18:36.145 DEBUG:boto:Data: 2024-02-01T10:18:36.145 DEBUG:boto:Headers: {} 2024-02-01T10:18:36.145 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:36.145 DEBUG:boto:Port: 8001 2024-02-01T10:18:36.146 DEBUG:boto:Params: {} 2024-02-01T10:18:36.146 DEBUG:boto:Token: None 2024-02-01T10:18:36.146 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:36 GMT /vclfcn-15/?versions 2024-02-01T10:18:36.146 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:I1ft7S6zbwwU5eoSKtL9w+uXDJ4= 2024-02-01T10:18:36.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:I1ft7S6zbwwU5eoSKtL9w+uXDJ4=', 'Content-Length': '0'} 2024-02-01T10:18:36.149 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d79bf9e5241ec63-0065bb6ffc-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:36.149 DEBUG:boto:b'vclfcn-15999falsemyobjnulltrue2024-02-01T10:18:13.821Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:36.149 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-02-01T10:18:36.150 DEBUG:boto:path=/vclfcn-15/myobj 2024-02-01T10:18:36.150 DEBUG:boto:auth_path=/vclfcn-15/myobj 2024-02-01T10:18:36.150 DEBUG:boto:path=/vclfcn-15/myobj?versionId=null 2024-02-01T10:18:36.150 DEBUG:boto:auth_path=/vclfcn-15/myobj?versionId=null 2024-02-01T10:18:36.150 DEBUG:boto:Method: GET 2024-02-01T10:18:36.150 DEBUG:boto:Path: /vclfcn-15/myobj?versionId=null 2024-02-01T10:18:36.151 DEBUG:boto:Data: 2024-02-01T10:18:36.151 DEBUG:boto:Headers: {} 2024-02-01T10:18:36.151 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:36.151 DEBUG:boto:Port: 8000 2024-02-01T10:18:36.151 DEBUG:boto:Params: {} 2024-02-01T10:18:36.151 DEBUG:boto:Token: None 2024-02-01T10:18:36.151 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:36 GMT /vclfcn-15/myobj?versionId=null 2024-02-01T10:18:36.152 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:emi2EuS/xiME1VyT2HwfimsgmA0= 2024-02-01T10:18:36.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:emi2EuS/xiME1VyT2HwfimsgmA0=', 'Content-Length': '0'} 2024-02-01T10:18:36.154 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e38af3bae0e69ef1-0065bb6ffc-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:36.154 DEBUG:boto:path=/vclfcn-15/myobj 2024-02-01T10:18:36.154 DEBUG:boto:auth_path=/vclfcn-15/myobj 2024-02-01T10:18:36.154 DEBUG:boto:path=/vclfcn-15/myobj?versionId=null 2024-02-01T10:18:36.154 DEBUG:boto:auth_path=/vclfcn-15/myobj?versionId=null 2024-02-01T10:18:36.154 DEBUG:boto:Method: GET 2024-02-01T10:18:36.154 DEBUG:boto:Path: /vclfcn-15/myobj?versionId=null 2024-02-01T10:18:36.154 DEBUG:boto:Data: 2024-02-01T10:18:36.155 DEBUG:boto:Headers: {} 2024-02-01T10:18:36.155 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:36.155 DEBUG:boto:Port: 8001 2024-02-01T10:18:36.155 DEBUG:boto:Params: {} 2024-02-01T10:18:36.155 DEBUG:boto:Token: None 2024-02-01T10:18:36.155 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:36 GMT /vclfcn-15/myobj?versionId=null 2024-02-01T10:18:36.155 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:emi2EuS/xiME1VyT2HwfimsgmA0= 2024-02-01T10:18:36.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:emi2EuS/xiME1VyT2HwfimsgmA0=', 'Content-Length': '0'} 2024-02-01T10:18:36.157 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d9947203db4470d3-0065bb6ffc-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:36.200 DEBUG:boto:path=/vclfcn-15/myobj 2024-02-01T10:18:36.200 DEBUG:boto:auth_path=/vclfcn-15/myobj 2024-02-01T10:18:36.200 DEBUG:boto:path=/vclfcn-15/myobj?versionId=null 2024-02-01T10:18:36.200 DEBUG:boto:auth_path=/vclfcn-15/myobj?versionId=null 2024-02-01T10:18:36.201 DEBUG:boto:Method: HEAD 2024-02-01T10:18:36.201 DEBUG:boto:Path: /vclfcn-15/myobj?versionId=null 2024-02-01T10:18:36.201 DEBUG:boto:Data: 2024-02-01T10:18:36.201 DEBUG:boto:Headers: {} 2024-02-01T10:18:36.201 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:36.201 DEBUG:boto:Port: 8000 2024-02-01T10:18:36.201 DEBUG:boto:Params: {} 2024-02-01T10:18:36.201 DEBUG:boto:Token: None 2024-02-01T10:18:36.202 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:36 GMT /vclfcn-15/myobj?versionId=null 2024-02-01T10:18:36.202 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:R7lhur5Afyg0hLyn/zBRNmJGAA4= 2024-02-01T10:18:36.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:R7lhur5Afyg0hLyn/zBRNmJGAA4=', 'Content-Length': '0'} 2024-02-01T10:18:36.204 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000019af01f3fa96c886-0065bb6ffc-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:36.205 DEBUG:boto:path=/vclfcn-15/myobj 2024-02-01T10:18:36.205 DEBUG:boto:auth_path=/vclfcn-15/myobj 2024-02-01T10:18:36.205 DEBUG:boto:path=/vclfcn-15/myobj?versionId=null 2024-02-01T10:18:36.205 DEBUG:boto:auth_path=/vclfcn-15/myobj?versionId=null 2024-02-01T10:18:36.205 DEBUG:boto:Method: HEAD 2024-02-01T10:18:36.205 DEBUG:boto:Path: /vclfcn-15/myobj?versionId=null 2024-02-01T10:18:36.205 DEBUG:boto:Data: 2024-02-01T10:18:36.206 DEBUG:boto:Headers: {} 2024-02-01T10:18:36.206 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:36.206 DEBUG:boto:Port: 8001 2024-02-01T10:18:36.206 DEBUG:boto:Params: {} 2024-02-01T10:18:36.206 DEBUG:boto:Token: None 2024-02-01T10:18:36.206 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:36 GMT /vclfcn-15/myobj?versionId=null 2024-02-01T10:18:36.206 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:R7lhur5Afyg0hLyn/zBRNmJGAA4= 2024-02-01T10:18:36.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:R7lhur5Afyg0hLyn/zBRNmJGAA4=', 'Content-Length': '0'} 2024-02-01T10:18:36.209 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000849e8f2b791b7bf-0065bb6ffc-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:36.209 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-02-01T10:18:36.210 DEBUG:boto:path=/vclfcn-15/myobj 2024-02-01T10:18:36.210 DEBUG:boto:auth_path=/vclfcn-15/myobj 2024-02-01T10:18:36.210 DEBUG:boto:path=/vclfcn-15/myobj?versionId=null 2024-02-01T10:18:36.210 DEBUG:boto:auth_path=/vclfcn-15/myobj?versionId=null 2024-02-01T10:18:36.210 DEBUG:boto:Method: GET 2024-02-01T10:18:36.210 DEBUG:boto:Path: /vclfcn-15/myobj?versionId=null 2024-02-01T10:18:36.210 DEBUG:boto:Data: 2024-02-01T10:18:36.210 DEBUG:boto:Headers: {} 2024-02-01T10:18:36.210 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:36.210 DEBUG:boto:Port: 8000 2024-02-01T10:18:36.210 DEBUG:boto:Params: {} 2024-02-01T10:18:36.211 DEBUG:boto:Token: None 2024-02-01T10:18:36.211 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:36 GMT /vclfcn-15/myobj?versionId=null 2024-02-01T10:18:36.211 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:emi2EuS/xiME1VyT2HwfimsgmA0= 2024-02-01T10:18:36.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:emi2EuS/xiME1VyT2HwfimsgmA0=', 'Content-Length': '0'} 2024-02-01T10:18:36.214 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000204151a0bfa75e46-0065bb6ffc-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:36.214 DEBUG:boto:path=/vclfcn-15/myobj 2024-02-01T10:18:36.214 DEBUG:boto:auth_path=/vclfcn-15/myobj 2024-02-01T10:18:36.214 DEBUG:boto:path=/vclfcn-15/myobj?versionId=null 2024-02-01T10:18:36.214 DEBUG:boto:auth_path=/vclfcn-15/myobj?versionId=null 2024-02-01T10:18:36.215 DEBUG:boto:Method: GET 2024-02-01T10:18:36.215 DEBUG:boto:Path: /vclfcn-15/myobj?versionId=null 2024-02-01T10:18:36.215 DEBUG:boto:Data: 2024-02-01T10:18:36.215 DEBUG:boto:Headers: {} 2024-02-01T10:18:36.215 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:36.215 DEBUG:boto:Port: 8001 2024-02-01T10:18:36.215 DEBUG:boto:Params: {} 2024-02-01T10:18:36.215 DEBUG:boto:Token: None 2024-02-01T10:18:36.215 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:36 GMT /vclfcn-15/myobj?versionId=null 2024-02-01T10:18:36.216 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:emi2EuS/xiME1VyT2HwfimsgmA0= 2024-02-01T10:18:36.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:emi2EuS/xiME1VyT2HwfimsgmA0=', 'Content-Length': '0'} 2024-02-01T10:18:36.219 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000029432df2cdef0d37-0065bb6ffc-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:36.219 DEBUG:boto:path=/vclfcn-15/ 2024-02-01T10:18:36.219 DEBUG:boto:auth_path=/vclfcn-15/ 2024-02-01T10:18:36.219 DEBUG:boto:path=/vclfcn-15/?prefix=myobj 2024-02-01T10:18:36.219 DEBUG:boto:auth_path=/vclfcn-15/?prefix=myobj 2024-02-01T10:18:36.220 DEBUG:boto:Method: GET 2024-02-01T10:18:36.220 DEBUG:boto:Path: /vclfcn-15/?prefix=myobj 2024-02-01T10:18:36.220 DEBUG:boto:Data: 2024-02-01T10:18:36.220 DEBUG:boto:Headers: {} 2024-02-01T10:18:36.220 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:36.220 DEBUG:boto:Port: 8000 2024-02-01T10:18:36.220 DEBUG:boto:Params: {} 2024-02-01T10:18:36.220 DEBUG:boto:Token: None 2024-02-01T10:18:36.221 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:36 GMT /vclfcn-15/ 2024-02-01T10:18:36.221 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:JzwPZp3sA9AKorDf9yrD7Pnv5ko= 2024-02-01T10:18:36.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:JzwPZp3sA9AKorDf9yrD7Pnv5ko=', 'Content-Length': '0'} 2024-02-01T10:18:36.225 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004585cb618fc0fc97-0065bb6ffc-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:36.268 DEBUG:boto:b'vclfcn-15myobj1000falsemyobj2024-02-01T10:18:13.821Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:36.269 DEBUG:boto:path=/vclfcn-15/ 2024-02-01T10:18:36.269 DEBUG:boto:auth_path=/vclfcn-15/ 2024-02-01T10:18:36.269 DEBUG:boto:path=/vclfcn-15/?prefix=myobj 2024-02-01T10:18:36.269 DEBUG:boto:auth_path=/vclfcn-15/?prefix=myobj 2024-02-01T10:18:36.269 DEBUG:boto:Method: GET 2024-02-01T10:18:36.269 DEBUG:boto:Path: /vclfcn-15/?prefix=myobj 2024-02-01T10:18:36.269 DEBUG:boto:Data: 2024-02-01T10:18:36.269 DEBUG:boto:Headers: {} 2024-02-01T10:18:36.269 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:36.269 DEBUG:boto:Port: 8001 2024-02-01T10:18:36.269 DEBUG:boto:Params: {} 2024-02-01T10:18:36.270 DEBUG:boto:Token: None 2024-02-01T10:18:36.270 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:36 GMT /vclfcn-15/ 2024-02-01T10:18:36.270 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:JzwPZp3sA9AKorDf9yrD7Pnv5ko= 2024-02-01T10:18:36.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:JzwPZp3sA9AKorDf9yrD7Pnv5ko=', 'Content-Length': '0'} 2024-02-01T10:18:36.274 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000abea2d6c06f222bf-0065bb6ffc-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:36.275 DEBUG:boto:b'vclfcn-15myobj1000falsemyobj2024-02-01T10:18:13.821Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:36.275 DEBUG:boto:path=/vclfcn-15/myobj 2024-02-01T10:18:36.275 DEBUG:boto:auth_path=/vclfcn-15/myobj 2024-02-01T10:18:36.275 DEBUG:boto:Method: HEAD 2024-02-01T10:18:36.275 DEBUG:boto:Path: /vclfcn-15/myobj 2024-02-01T10:18:36.275 DEBUG:boto:Data: 2024-02-01T10:18:36.276 DEBUG:boto:Headers: {} 2024-02-01T10:18:36.276 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:36.276 DEBUG:boto:Port: 8000 2024-02-01T10:18:36.276 DEBUG:boto:Params: {} 2024-02-01T10:18:36.276 DEBUG:boto:Token: None 2024-02-01T10:18:36.276 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:36 GMT /vclfcn-15/myobj 2024-02-01T10:18:36.276 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:DbmaJPf9kIMGxEiQ4PXtXdQi11E= 2024-02-01T10:18:36.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:DbmaJPf9kIMGxEiQ4PXtXdQi11E=', 'Content-Length': '0'} 2024-02-01T10:18:36.279 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000062a243cb0b9a2718-0065bb6ffc-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:36.279 DEBUG:boto:path=/vclfcn-15/myobj 2024-02-01T10:18:36.279 DEBUG:boto:auth_path=/vclfcn-15/myobj 2024-02-01T10:18:36.279 DEBUG:boto:Method: HEAD 2024-02-01T10:18:36.280 DEBUG:boto:Path: /vclfcn-15/myobj 2024-02-01T10:18:36.280 DEBUG:boto:Data: 2024-02-01T10:18:36.280 DEBUG:boto:Headers: {} 2024-02-01T10:18:36.280 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:36.280 DEBUG:boto:Port: 8001 2024-02-01T10:18:36.280 DEBUG:boto:Params: {} 2024-02-01T10:18:36.280 DEBUG:boto:Token: None 2024-02-01T10:18:36.280 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:36 GMT /vclfcn-15/myobj 2024-02-01T10:18:36.280 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:DbmaJPf9kIMGxEiQ4PXtXdQi11E= 2024-02-01T10:18:36.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:DbmaJPf9kIMGxEiQ4PXtXdQi11E=', 'Content-Length': '0'} 2024-02-01T10:18:36.283 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b85967048991df15-0065bb6ffc-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:36.283 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-02-01T10:18:36.283 DEBUG:boto:path=/vclfcn-15/myobj 2024-02-01T10:18:36.284 DEBUG:boto:auth_path=/vclfcn-15/myobj 2024-02-01T10:18:36.284 DEBUG:boto:Method: GET 2024-02-01T10:18:36.284 DEBUG:boto:Path: /vclfcn-15/myobj 2024-02-01T10:18:36.284 DEBUG:boto:Data: 2024-02-01T10:18:36.284 DEBUG:boto:Headers: {} 2024-02-01T10:18:36.284 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:36.284 DEBUG:boto:Port: 8000 2024-02-01T10:18:36.284 DEBUG:boto:Params: {} 2024-02-01T10:18:36.284 DEBUG:boto:Token: None 2024-02-01T10:18:36.285 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:36 GMT /vclfcn-15/myobj 2024-02-01T10:18:36.285 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Icfsd0Vcbk+eW2pzzGstMlBtqzY= 2024-02-01T10:18:36.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Icfsd0Vcbk+eW2pzzGstMlBtqzY=', 'Content-Length': '0'} 2024-02-01T10:18:36.287 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008d0c2b89ffce901c-0065bb6ffc-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:36.332 DEBUG:boto:path=/vclfcn-15/myobj 2024-02-01T10:18:36.332 DEBUG:boto:auth_path=/vclfcn-15/myobj 2024-02-01T10:18:36.332 DEBUG:boto:Method: GET 2024-02-01T10:18:36.333 DEBUG:boto:Path: /vclfcn-15/myobj 2024-02-01T10:18:36.333 DEBUG:boto:Data: 2024-02-01T10:18:36.333 DEBUG:boto:Headers: {} 2024-02-01T10:18:36.333 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:36.333 DEBUG:boto:Port: 8001 2024-02-01T10:18:36.333 DEBUG:boto:Params: {} 2024-02-01T10:18:36.333 DEBUG:boto:Token: None 2024-02-01T10:18:36.333 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:36 GMT /vclfcn-15/myobj 2024-02-01T10:18:36.333 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Icfsd0Vcbk+eW2pzzGstMlBtqzY= 2024-02-01T10:18:36.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Icfsd0Vcbk+eW2pzzGstMlBtqzY=', 'Content-Length': '0'} 2024-02-01T10:18:36.336 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002aa8939c18442e25-0065bb6ffc-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:36.337 DEBUG:boto:path=/vclfcn-15/ 2024-02-01T10:18:36.337 DEBUG:boto:auth_path=/vclfcn-15/ 2024-02-01T10:18:36.337 DEBUG:boto:path=/vclfcn-15/?prefix=myobj 2024-02-01T10:18:36.337 DEBUG:boto:auth_path=/vclfcn-15/?prefix=myobj 2024-02-01T10:18:36.337 DEBUG:boto:Method: GET 2024-02-01T10:18:36.337 DEBUG:boto:Path: /vclfcn-15/?prefix=myobj 2024-02-01T10:18:36.337 DEBUG:boto:Data: 2024-02-01T10:18:36.337 DEBUG:boto:Headers: {} 2024-02-01T10:18:36.338 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:36.338 DEBUG:boto:Port: 8000 2024-02-01T10:18:36.338 DEBUG:boto:Params: {} 2024-02-01T10:18:36.338 DEBUG:boto:Token: None 2024-02-01T10:18:36.338 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:36 GMT /vclfcn-15/ 2024-02-01T10:18:36.338 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:JzwPZp3sA9AKorDf9yrD7Pnv5ko= 2024-02-01T10:18:36.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:JzwPZp3sA9AKorDf9yrD7Pnv5ko=', 'Content-Length': '0'} 2024-02-01T10:18:36.342 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b87536794cc5fe4-0065bb6ffc-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:36.342 DEBUG:boto:b'vclfcn-15myobj1000falsemyobj2024-02-01T10:18:13.821Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:36.343 DEBUG:boto:path=/vclfcn-15/ 2024-02-01T10:18:36.343 DEBUG:boto:auth_path=/vclfcn-15/ 2024-02-01T10:18:36.343 DEBUG:boto:path=/vclfcn-15/?prefix=myobj 2024-02-01T10:18:36.343 DEBUG:boto:auth_path=/vclfcn-15/?prefix=myobj 2024-02-01T10:18:36.343 DEBUG:boto:Method: GET 2024-02-01T10:18:36.343 DEBUG:boto:Path: /vclfcn-15/?prefix=myobj 2024-02-01T10:18:36.343 DEBUG:boto:Data: 2024-02-01T10:18:36.343 DEBUG:boto:Headers: {} 2024-02-01T10:18:36.343 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:36.343 DEBUG:boto:Port: 8001 2024-02-01T10:18:36.343 DEBUG:boto:Params: {} 2024-02-01T10:18:36.344 DEBUG:boto:Token: None 2024-02-01T10:18:36.344 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:36 GMT /vclfcn-15/ 2024-02-01T10:18:36.344 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:JzwPZp3sA9AKorDf9yrD7Pnv5ko= 2024-02-01T10:18:36.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:JzwPZp3sA9AKorDf9yrD7Pnv5ko=', 'Content-Length': '0'} 2024-02-01T10:18:36.348 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054a2d3edc9856a35-0065bb6ffc-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:36.349 DEBUG:boto:b'vclfcn-15myobj1000falsemyobj2024-02-01T10:18:13.821Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:36.349 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-15 zones={test-zone1, test-zone3} 2024-02-01T10:18:36.349 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:18:36.349 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:18:36.350 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket vclfcn-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-02-01T10:18:36.403 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:18:36.403 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:18:36.470 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:18:36.471+0000 7f778eec2980 1 bucket sync caught up with source: 2024-02-01T10:18:36.471 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, 00000000001.12.6, , , , , , , , , ] 2024-02-01T10:18:36.471 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, 00000000001.12.6, , , , , , , , , ] 2024-02-01T10:18:36.471 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:18:36.471+0000 7f778eec2980 0 bucket checkpoint complete 2024-02-01T10:18:36.492 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-15 zones={test-zone2, test-zone3} 2024-02-01T10:18:36.492 DEBUG:boto:path=/vclfcn-15/ 2024-02-01T10:18:36.493 DEBUG:boto:auth_path=/vclfcn-15/ 2024-02-01T10:18:36.493 DEBUG:boto:Method: HEAD 2024-02-01T10:18:36.493 DEBUG:boto:Path: /vclfcn-15/ 2024-02-01T10:18:36.493 DEBUG:boto:Data: 2024-02-01T10:18:36.493 DEBUG:boto:Headers: {} 2024-02-01T10:18:36.494 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:36.494 DEBUG:boto:Port: 8000 2024-02-01T10:18:36.494 DEBUG:boto:Params: {} 2024-02-01T10:18:36.494 DEBUG:boto:Token: None 2024-02-01T10:18:36.494 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:36 GMT /vclfcn-15/ 2024-02-01T10:18:36.494 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jk/JL1ux23WGtQtyK0/Sy+kqi+I= 2024-02-01T10:18:36.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jk/JL1ux23WGtQtyK0/Sy+kqi+I=', 'Content-Length': '0'} 2024-02-01T10:18:36.497 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d430791723fd2fc9-0065bb6ffc-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:18:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:36.497 DEBUG:boto:path=/vclfcn-15/ 2024-02-01T10:18:36.497 DEBUG:boto:auth_path=/vclfcn-15/ 2024-02-01T10:18:36.497 DEBUG:boto:Method: HEAD 2024-02-01T10:18:36.497 DEBUG:boto:Path: /vclfcn-15/ 2024-02-01T10:18:36.497 DEBUG:boto:Data: 2024-02-01T10:18:36.498 DEBUG:boto:Headers: {} 2024-02-01T10:18:36.498 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:36.498 DEBUG:boto:Port: 8001 2024-02-01T10:18:36.498 DEBUG:boto:Params: {} 2024-02-01T10:18:36.498 DEBUG:boto:Token: None 2024-02-01T10:18:36.498 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:36 GMT /vclfcn-15/ 2024-02-01T10:18:36.498 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jk/JL1ux23WGtQtyK0/Sy+kqi+I= 2024-02-01T10:18:36.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jk/JL1ux23WGtQtyK0/Sy+kqi+I=', 'Content-Length': '0'} 2024-02-01T10:18:36.503 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000239f5b5bd7d0e2c8-0065bb6ffc-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:18:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:36.503 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:18:36.503 DEBUG:boto:path=/vclfcn-15/ 2024-02-01T10:18:36.503 DEBUG:boto:auth_path=/vclfcn-15/ 2024-02-01T10:18:36.504 DEBUG:boto:path=/vclfcn-15/?versions&max-keys=999 2024-02-01T10:18:36.504 DEBUG:boto:auth_path=/vclfcn-15/?versions&max-keys=999 2024-02-01T10:18:36.504 DEBUG:boto:Method: GET 2024-02-01T10:18:36.504 DEBUG:boto:Path: /vclfcn-15/?versions&max-keys=999 2024-02-01T10:18:36.504 DEBUG:boto:Data: 2024-02-01T10:18:36.505 DEBUG:boto:Headers: {} 2024-02-01T10:18:36.505 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:36.505 DEBUG:boto:Port: 8000 2024-02-01T10:18:36.505 DEBUG:boto:Params: {} 2024-02-01T10:18:36.505 DEBUG:boto:Token: None 2024-02-01T10:18:36.505 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:36 GMT /vclfcn-15/?versions 2024-02-01T10:18:36.505 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:I1ft7S6zbwwU5eoSKtL9w+uXDJ4= 2024-02-01T10:18:36.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:I1ft7S6zbwwU5eoSKtL9w+uXDJ4=', 'Content-Length': '0'} 2024-02-01T10:18:36.508 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000044b59fc24d41c126-0065bb6ffc-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:36.508 DEBUG:boto:b'vclfcn-15999falsemyobjnulltrue2024-02-01T10:18:13.821Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:36.508 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-02-01T10:18:36.508 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:18:36.509 DEBUG:boto:path=/vclfcn-15/ 2024-02-01T10:18:36.509 DEBUG:boto:auth_path=/vclfcn-15/ 2024-02-01T10:18:36.509 DEBUG:boto:path=/vclfcn-15/?versions&max-keys=999 2024-02-01T10:18:36.509 DEBUG:boto:auth_path=/vclfcn-15/?versions&max-keys=999 2024-02-01T10:18:36.509 DEBUG:boto:Method: GET 2024-02-01T10:18:36.509 DEBUG:boto:Path: /vclfcn-15/?versions&max-keys=999 2024-02-01T10:18:36.509 DEBUG:boto:Data: 2024-02-01T10:18:36.509 DEBUG:boto:Headers: {} 2024-02-01T10:18:36.509 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:36.509 DEBUG:boto:Port: 8001 2024-02-01T10:18:36.509 DEBUG:boto:Params: {} 2024-02-01T10:18:36.510 DEBUG:boto:Token: None 2024-02-01T10:18:36.510 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:36 GMT /vclfcn-15/?versions 2024-02-01T10:18:36.510 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:I1ft7S6zbwwU5eoSKtL9w+uXDJ4= 2024-02-01T10:18:36.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:I1ft7S6zbwwU5eoSKtL9w+uXDJ4=', 'Content-Length': '0'} 2024-02-01T10:18:36.514 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ab6ce5ca4735610-0065bb6ffc-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:36.514 DEBUG:boto:b'vclfcn-15999falsemyobjnulltrue2024-02-01T10:18:13.821Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:36.515 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-02-01T10:18:36.515 DEBUG:boto:path=/vclfcn-15/ 2024-02-01T10:18:36.515 DEBUG:boto:auth_path=/vclfcn-15/ 2024-02-01T10:18:36.515 DEBUG:boto:path=/vclfcn-15/?versions&max-keys=999 2024-02-01T10:18:36.515 DEBUG:boto:auth_path=/vclfcn-15/?versions&max-keys=999 2024-02-01T10:18:36.515 DEBUG:boto:Method: GET 2024-02-01T10:18:36.515 DEBUG:boto:Path: /vclfcn-15/?versions&max-keys=999 2024-02-01T10:18:36.515 DEBUG:boto:Data: 2024-02-01T10:18:36.515 DEBUG:boto:Headers: {} 2024-02-01T10:18:36.516 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:36.516 DEBUG:boto:Port: 8000 2024-02-01T10:18:36.516 DEBUG:boto:Params: {} 2024-02-01T10:18:36.516 DEBUG:boto:Token: None 2024-02-01T10:18:36.516 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:36 GMT /vclfcn-15/?versions 2024-02-01T10:18:36.516 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:I1ft7S6zbwwU5eoSKtL9w+uXDJ4= 2024-02-01T10:18:36.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:I1ft7S6zbwwU5eoSKtL9w+uXDJ4=', 'Content-Length': '0'} 2024-02-01T10:18:36.518 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000048014693f77d733c-0065bb6ffc-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:36.518 DEBUG:boto:b'vclfcn-15999falsemyobjnulltrue2024-02-01T10:18:13.821Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:36.519 DEBUG:boto:path=/vclfcn-15/ 2024-02-01T10:18:36.519 DEBUG:boto:auth_path=/vclfcn-15/ 2024-02-01T10:18:36.519 DEBUG:boto:path=/vclfcn-15/?versions&max-keys=999 2024-02-01T10:18:36.519 DEBUG:boto:auth_path=/vclfcn-15/?versions&max-keys=999 2024-02-01T10:18:36.519 DEBUG:boto:Method: GET 2024-02-01T10:18:36.519 DEBUG:boto:Path: /vclfcn-15/?versions&max-keys=999 2024-02-01T10:18:36.519 DEBUG:boto:Data: 2024-02-01T10:18:36.519 DEBUG:boto:Headers: {} 2024-02-01T10:18:36.519 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:36.519 DEBUG:boto:Port: 8001 2024-02-01T10:18:36.520 DEBUG:boto:Params: {} 2024-02-01T10:18:36.520 DEBUG:boto:Token: None 2024-02-01T10:18:36.520 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:36 GMT /vclfcn-15/?versions 2024-02-01T10:18:36.520 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:I1ft7S6zbwwU5eoSKtL9w+uXDJ4= 2024-02-01T10:18:36.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:I1ft7S6zbwwU5eoSKtL9w+uXDJ4=', 'Content-Length': '0'} 2024-02-01T10:18:36.524 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1b53f581dd11ae8-0065bb6ffc-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:36.568 DEBUG:boto:b'vclfcn-15999falsemyobjnulltrue2024-02-01T10:18:13.821Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:36.569 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-02-01T10:18:36.569 DEBUG:boto:path=/vclfcn-15/myobj 2024-02-01T10:18:36.569 DEBUG:boto:auth_path=/vclfcn-15/myobj 2024-02-01T10:18:36.569 DEBUG:boto:path=/vclfcn-15/myobj?versionId=null 2024-02-01T10:18:36.569 DEBUG:boto:auth_path=/vclfcn-15/myobj?versionId=null 2024-02-01T10:18:36.569 DEBUG:boto:Method: GET 2024-02-01T10:18:36.569 DEBUG:boto:Path: /vclfcn-15/myobj?versionId=null 2024-02-01T10:18:36.569 DEBUG:boto:Data: 2024-02-01T10:18:36.569 DEBUG:boto:Headers: {} 2024-02-01T10:18:36.570 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:36.570 DEBUG:boto:Port: 8000 2024-02-01T10:18:36.570 DEBUG:boto:Params: {} 2024-02-01T10:18:36.570 DEBUG:boto:Token: None 2024-02-01T10:18:36.570 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:36 GMT /vclfcn-15/myobj?versionId=null 2024-02-01T10:18:36.570 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:emi2EuS/xiME1VyT2HwfimsgmA0= 2024-02-01T10:18:36.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:emi2EuS/xiME1VyT2HwfimsgmA0=', 'Content-Length': '0'} 2024-02-01T10:18:36.574 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000081732306d636c3b3-0065bb6ffc-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:36.574 DEBUG:boto:path=/vclfcn-15/myobj 2024-02-01T10:18:36.574 DEBUG:boto:auth_path=/vclfcn-15/myobj 2024-02-01T10:18:36.574 DEBUG:boto:path=/vclfcn-15/myobj?versionId=null 2024-02-01T10:18:36.574 DEBUG:boto:auth_path=/vclfcn-15/myobj?versionId=null 2024-02-01T10:18:36.574 DEBUG:boto:Method: GET 2024-02-01T10:18:36.574 DEBUG:boto:Path: /vclfcn-15/myobj?versionId=null 2024-02-01T10:18:36.575 DEBUG:boto:Data: 2024-02-01T10:18:36.575 DEBUG:boto:Headers: {} 2024-02-01T10:18:36.575 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:36.575 DEBUG:boto:Port: 8001 2024-02-01T10:18:36.575 DEBUG:boto:Params: {} 2024-02-01T10:18:36.575 DEBUG:boto:Token: None 2024-02-01T10:18:36.575 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:36 GMT /vclfcn-15/myobj?versionId=null 2024-02-01T10:18:36.575 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:emi2EuS/xiME1VyT2HwfimsgmA0= 2024-02-01T10:18:36.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:emi2EuS/xiME1VyT2HwfimsgmA0=', 'Content-Length': '0'} 2024-02-01T10:18:36.578 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007e79783b37a4fdac-0065bb6ffc-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:36.578 DEBUG:boto:path=/vclfcn-15/myobj 2024-02-01T10:18:36.579 DEBUG:boto:auth_path=/vclfcn-15/myobj 2024-02-01T10:18:36.579 DEBUG:boto:path=/vclfcn-15/myobj?versionId=null 2024-02-01T10:18:36.579 DEBUG:boto:auth_path=/vclfcn-15/myobj?versionId=null 2024-02-01T10:18:36.579 DEBUG:boto:Method: HEAD 2024-02-01T10:18:36.579 DEBUG:boto:Path: /vclfcn-15/myobj?versionId=null 2024-02-01T10:18:36.579 DEBUG:boto:Data: 2024-02-01T10:18:36.579 DEBUG:boto:Headers: {} 2024-02-01T10:18:36.579 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:36.579 DEBUG:boto:Port: 8000 2024-02-01T10:18:36.579 DEBUG:boto:Params: {} 2024-02-01T10:18:36.579 DEBUG:boto:Token: None 2024-02-01T10:18:36.580 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:36 GMT /vclfcn-15/myobj?versionId=null 2024-02-01T10:18:36.580 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:R7lhur5Afyg0hLyn/zBRNmJGAA4= 2024-02-01T10:18:36.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:R7lhur5Afyg0hLyn/zBRNmJGAA4=', 'Content-Length': '0'} 2024-02-01T10:18:36.583 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006070c5d195d50e39-0065bb6ffc-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:36.583 DEBUG:boto:path=/vclfcn-15/myobj 2024-02-01T10:18:36.583 DEBUG:boto:auth_path=/vclfcn-15/myobj 2024-02-01T10:18:36.583 DEBUG:boto:path=/vclfcn-15/myobj?versionId=null 2024-02-01T10:18:36.583 DEBUG:boto:auth_path=/vclfcn-15/myobj?versionId=null 2024-02-01T10:18:36.583 DEBUG:boto:Method: HEAD 2024-02-01T10:18:36.583 DEBUG:boto:Path: /vclfcn-15/myobj?versionId=null 2024-02-01T10:18:36.583 DEBUG:boto:Data: 2024-02-01T10:18:36.583 DEBUG:boto:Headers: {} 2024-02-01T10:18:36.584 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:36.584 DEBUG:boto:Port: 8001 2024-02-01T10:18:36.584 DEBUG:boto:Params: {} 2024-02-01T10:18:36.584 DEBUG:boto:Token: None 2024-02-01T10:18:36.584 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:36 GMT /vclfcn-15/myobj?versionId=null 2024-02-01T10:18:36.584 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:R7lhur5Afyg0hLyn/zBRNmJGAA4= 2024-02-01T10:18:36.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:R7lhur5Afyg0hLyn/zBRNmJGAA4=', 'Content-Length': '0'} 2024-02-01T10:18:36.587 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000097c3ff80aa694581-0065bb6ffc-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:36.587 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-02-01T10:18:36.587 DEBUG:boto:path=/vclfcn-15/myobj 2024-02-01T10:18:36.587 DEBUG:boto:auth_path=/vclfcn-15/myobj 2024-02-01T10:18:36.587 DEBUG:boto:path=/vclfcn-15/myobj?versionId=null 2024-02-01T10:18:36.588 DEBUG:boto:auth_path=/vclfcn-15/myobj?versionId=null 2024-02-01T10:18:36.588 DEBUG:boto:Method: GET 2024-02-01T10:18:36.588 DEBUG:boto:Path: /vclfcn-15/myobj?versionId=null 2024-02-01T10:18:36.588 DEBUG:boto:Data: 2024-02-01T10:18:36.588 DEBUG:boto:Headers: {} 2024-02-01T10:18:36.588 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:36.588 DEBUG:boto:Port: 8000 2024-02-01T10:18:36.588 DEBUG:boto:Params: {} 2024-02-01T10:18:36.591 DEBUG:boto:Token: None 2024-02-01T10:18:36.591 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:36 GMT /vclfcn-15/myobj?versionId=null 2024-02-01T10:18:36.591 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:emi2EuS/xiME1VyT2HwfimsgmA0= 2024-02-01T10:18:36.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:emi2EuS/xiME1VyT2HwfimsgmA0=', 'Content-Length': '0'} 2024-02-01T10:18:36.594 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006fe70cdc94604843-0065bb6ffc-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:36.636 DEBUG:boto:path=/vclfcn-15/myobj 2024-02-01T10:18:36.636 DEBUG:boto:auth_path=/vclfcn-15/myobj 2024-02-01T10:18:36.636 DEBUG:boto:path=/vclfcn-15/myobj?versionId=null 2024-02-01T10:18:36.637 DEBUG:boto:auth_path=/vclfcn-15/myobj?versionId=null 2024-02-01T10:18:36.637 DEBUG:boto:Method: GET 2024-02-01T10:18:36.637 DEBUG:boto:Path: /vclfcn-15/myobj?versionId=null 2024-02-01T10:18:36.637 DEBUG:boto:Data: 2024-02-01T10:18:36.637 DEBUG:boto:Headers: {} 2024-02-01T10:18:36.637 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:36.638 DEBUG:boto:Port: 8001 2024-02-01T10:18:36.638 DEBUG:boto:Params: {} 2024-02-01T10:18:36.638 DEBUG:boto:Token: None 2024-02-01T10:18:36.638 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:36 GMT /vclfcn-15/myobj?versionId=null 2024-02-01T10:18:36.638 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:emi2EuS/xiME1VyT2HwfimsgmA0= 2024-02-01T10:18:36.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:emi2EuS/xiME1VyT2HwfimsgmA0=', 'Content-Length': '0'} 2024-02-01T10:18:36.643 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cbb4f17179de40bf-0065bb6ffc-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:36.643 DEBUG:boto:path=/vclfcn-15/ 2024-02-01T10:18:36.643 DEBUG:boto:auth_path=/vclfcn-15/ 2024-02-01T10:18:36.643 DEBUG:boto:path=/vclfcn-15/?prefix=myobj 2024-02-01T10:18:36.644 DEBUG:boto:auth_path=/vclfcn-15/?prefix=myobj 2024-02-01T10:18:36.644 DEBUG:boto:Method: GET 2024-02-01T10:18:36.644 DEBUG:boto:Path: /vclfcn-15/?prefix=myobj 2024-02-01T10:18:36.644 DEBUG:boto:Data: 2024-02-01T10:18:36.644 DEBUG:boto:Headers: {} 2024-02-01T10:18:36.644 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:36.644 DEBUG:boto:Port: 8000 2024-02-01T10:18:36.644 DEBUG:boto:Params: {} 2024-02-01T10:18:36.644 DEBUG:boto:Token: None 2024-02-01T10:18:36.644 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:36 GMT /vclfcn-15/ 2024-02-01T10:18:36.645 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:JzwPZp3sA9AKorDf9yrD7Pnv5ko= 2024-02-01T10:18:36.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:JzwPZp3sA9AKorDf9yrD7Pnv5ko=', 'Content-Length': '0'} 2024-02-01T10:18:36.649 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a6aafd1bad0ac0b-0065bb6ffc-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:36.649 DEBUG:boto:b'vclfcn-15myobj1000falsemyobj2024-02-01T10:18:13.821Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:36.649 DEBUG:boto:path=/vclfcn-15/ 2024-02-01T10:18:36.649 DEBUG:boto:auth_path=/vclfcn-15/ 2024-02-01T10:18:36.649 DEBUG:boto:path=/vclfcn-15/?prefix=myobj 2024-02-01T10:18:36.649 DEBUG:boto:auth_path=/vclfcn-15/?prefix=myobj 2024-02-01T10:18:36.650 DEBUG:boto:Method: GET 2024-02-01T10:18:36.650 DEBUG:boto:Path: /vclfcn-15/?prefix=myobj 2024-02-01T10:18:36.650 DEBUG:boto:Data: 2024-02-01T10:18:36.650 DEBUG:boto:Headers: {} 2024-02-01T10:18:36.650 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:36.650 DEBUG:boto:Port: 8001 2024-02-01T10:18:36.650 DEBUG:boto:Params: {} 2024-02-01T10:18:36.650 DEBUG:boto:Token: None 2024-02-01T10:18:36.650 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:36 GMT /vclfcn-15/ 2024-02-01T10:18:36.651 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:JzwPZp3sA9AKorDf9yrD7Pnv5ko= 2024-02-01T10:18:36.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:JzwPZp3sA9AKorDf9yrD7Pnv5ko=', 'Content-Length': '0'} 2024-02-01T10:18:36.656 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043e0f213bae7405b-0065bb6ffc-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:36.700 DEBUG:boto:b'vclfcn-15myobj1000falsemyobj2024-02-01T10:18:13.821Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:36.701 DEBUG:boto:path=/vclfcn-15/myobj 2024-02-01T10:18:36.701 DEBUG:boto:auth_path=/vclfcn-15/myobj 2024-02-01T10:18:36.701 DEBUG:boto:Method: HEAD 2024-02-01T10:18:36.701 DEBUG:boto:Path: /vclfcn-15/myobj 2024-02-01T10:18:36.701 DEBUG:boto:Data: 2024-02-01T10:18:36.701 DEBUG:boto:Headers: {} 2024-02-01T10:18:36.701 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:36.702 DEBUG:boto:Port: 8000 2024-02-01T10:18:36.702 DEBUG:boto:Params: {} 2024-02-01T10:18:36.702 DEBUG:boto:Token: None 2024-02-01T10:18:36.702 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:36 GMT /vclfcn-15/myobj 2024-02-01T10:18:36.702 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:DbmaJPf9kIMGxEiQ4PXtXdQi11E= 2024-02-01T10:18:36.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:DbmaJPf9kIMGxEiQ4PXtXdQi11E=', 'Content-Length': '0'} 2024-02-01T10:18:36.705 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ab310e689690607e-0065bb6ffc-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:36.705 DEBUG:boto:path=/vclfcn-15/myobj 2024-02-01T10:18:36.705 DEBUG:boto:auth_path=/vclfcn-15/myobj 2024-02-01T10:18:36.705 DEBUG:boto:Method: HEAD 2024-02-01T10:18:36.706 DEBUG:boto:Path: /vclfcn-15/myobj 2024-02-01T10:18:36.706 DEBUG:boto:Data: 2024-02-01T10:18:36.706 DEBUG:boto:Headers: {} 2024-02-01T10:18:36.706 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:36.706 DEBUG:boto:Port: 8001 2024-02-01T10:18:36.706 DEBUG:boto:Params: {} 2024-02-01T10:18:36.706 DEBUG:boto:Token: None 2024-02-01T10:18:36.706 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:36 GMT /vclfcn-15/myobj 2024-02-01T10:18:36.706 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:DbmaJPf9kIMGxEiQ4PXtXdQi11E= 2024-02-01T10:18:36.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:DbmaJPf9kIMGxEiQ4PXtXdQi11E=', 'Content-Length': '0'} 2024-02-01T10:18:36.709 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000033744756e01edc39-0065bb6ffc-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:36.709 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-02-01T10:18:36.709 DEBUG:boto:path=/vclfcn-15/myobj 2024-02-01T10:18:36.710 DEBUG:boto:auth_path=/vclfcn-15/myobj 2024-02-01T10:18:36.710 DEBUG:boto:Method: GET 2024-02-01T10:18:36.710 DEBUG:boto:Path: /vclfcn-15/myobj 2024-02-01T10:18:36.710 DEBUG:boto:Data: 2024-02-01T10:18:36.710 DEBUG:boto:Headers: {} 2024-02-01T10:18:36.710 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:36.710 DEBUG:boto:Port: 8000 2024-02-01T10:18:36.710 DEBUG:boto:Params: {} 2024-02-01T10:18:36.710 DEBUG:boto:Token: None 2024-02-01T10:18:36.711 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:36 GMT /vclfcn-15/myobj 2024-02-01T10:18:36.711 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Icfsd0Vcbk+eW2pzzGstMlBtqzY= 2024-02-01T10:18:36.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Icfsd0Vcbk+eW2pzzGstMlBtqzY=', 'Content-Length': '0'} 2024-02-01T10:18:36.714 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000173f8002b6dce35d-0065bb6ffc-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:36.714 DEBUG:boto:path=/vclfcn-15/myobj 2024-02-01T10:18:36.714 DEBUG:boto:auth_path=/vclfcn-15/myobj 2024-02-01T10:18:36.714 DEBUG:boto:Method: GET 2024-02-01T10:18:36.714 DEBUG:boto:Path: /vclfcn-15/myobj 2024-02-01T10:18:36.715 DEBUG:boto:Data: 2024-02-01T10:18:36.715 DEBUG:boto:Headers: {} 2024-02-01T10:18:36.715 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:36.715 DEBUG:boto:Port: 8001 2024-02-01T10:18:36.715 DEBUG:boto:Params: {} 2024-02-01T10:18:36.715 DEBUG:boto:Token: None 2024-02-01T10:18:36.715 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:36 GMT /vclfcn-15/myobj 2024-02-01T10:18:36.715 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Icfsd0Vcbk+eW2pzzGstMlBtqzY= 2024-02-01T10:18:36.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Icfsd0Vcbk+eW2pzzGstMlBtqzY=', 'Content-Length': '0'} 2024-02-01T10:18:36.718 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003feb178bc7023754-0065bb6ffc-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:36.718 DEBUG:boto:path=/vclfcn-15/ 2024-02-01T10:18:36.719 DEBUG:boto:auth_path=/vclfcn-15/ 2024-02-01T10:18:36.719 DEBUG:boto:path=/vclfcn-15/?prefix=myobj 2024-02-01T10:18:36.719 DEBUG:boto:auth_path=/vclfcn-15/?prefix=myobj 2024-02-01T10:18:36.719 DEBUG:boto:Method: GET 2024-02-01T10:18:36.719 DEBUG:boto:Path: /vclfcn-15/?prefix=myobj 2024-02-01T10:18:36.719 DEBUG:boto:Data: 2024-02-01T10:18:36.719 DEBUG:boto:Headers: {} 2024-02-01T10:18:36.719 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:18:36.719 DEBUG:boto:Port: 8000 2024-02-01T10:18:36.719 DEBUG:boto:Params: {} 2024-02-01T10:18:36.719 DEBUG:boto:Token: None 2024-02-01T10:18:36.720 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:36 GMT /vclfcn-15/ 2024-02-01T10:18:36.720 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:JzwPZp3sA9AKorDf9yrD7Pnv5ko= 2024-02-01T10:18:36.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:JzwPZp3sA9AKorDf9yrD7Pnv5ko=', 'Content-Length': '0'} 2024-02-01T10:18:36.724 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000068b56e0b1bc46c1-0065bb6ffc-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:36.764 DEBUG:boto:b'vclfcn-15myobj1000falsemyobj2024-02-01T10:18:13.821Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:36.765 DEBUG:boto:path=/vclfcn-15/ 2024-02-01T10:18:36.765 DEBUG:boto:auth_path=/vclfcn-15/ 2024-02-01T10:18:36.765 DEBUG:boto:path=/vclfcn-15/?prefix=myobj 2024-02-01T10:18:36.765 DEBUG:boto:auth_path=/vclfcn-15/?prefix=myobj 2024-02-01T10:18:36.765 DEBUG:boto:Method: GET 2024-02-01T10:18:36.765 DEBUG:boto:Path: /vclfcn-15/?prefix=myobj 2024-02-01T10:18:36.765 DEBUG:boto:Data: 2024-02-01T10:18:36.765 DEBUG:boto:Headers: {} 2024-02-01T10:18:36.765 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:18:36.765 DEBUG:boto:Port: 8001 2024-02-01T10:18:36.765 DEBUG:boto:Params: {} 2024-02-01T10:18:36.766 DEBUG:boto:Token: None 2024-02-01T10:18:36.766 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:18:36 GMT /vclfcn-15/ 2024-02-01T10:18:36.766 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:JzwPZp3sA9AKorDf9yrD7Pnv5ko= 2024-02-01T10:18:36.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:JzwPZp3sA9AKorDf9yrD7Pnv5ko=', 'Content-Length': '0'} 2024-02-01T10:18:36.770 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e920b3181f1cce33-0065bb6ffc-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:36.771 DEBUG:boto:b'vclfcn-15myobj1000falsemyobj2024-02-01T10:18:13.821Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:18:36.771 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-15 zones={test-zone2, test-zone3} 2024-02-01T10:18:36.771 DEBUG:boto:path=/vclfcn-13/ 2024-02-01T10:18:36.771 DEBUG:boto:auth_path=/vclfcn-13/ 2024-02-01T10:18:36.772 DEBUG:boto:Method: HEAD 2024-02-01T10:18:36.772 DEBUG:boto:Path: /vclfcn-13/ 2024-02-01T10:18:36.772 DEBUG:boto:Data: 2024-02-01T10:18:36.772 DEBUG:boto:Headers: {} 2024-02-01T10:18:36.772 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:36.772 DEBUG:boto:Port: 8000 2024-02-01T10:18:36.772 DEBUG:boto:Params: {} 2024-02-01T10:18:36.772 DEBUG:boto:Token: None 2024-02-01T10:18:36.772 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:36 GMT /vclfcn-13/ 2024-02-01T10:18:36.773 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:g5t8afRIldXGIftyb6ngnHTNjkQ= 2024-02-01T10:18:36.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:g5t8afRIldXGIftyb6ngnHTNjkQ=', 'Content-Length': '0'} 2024-02-01T10:18:36.777 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e2d3e28cfe315e8f-0065bb6ffc-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:18:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:36.778 DEBUG:boto:path=/vclfcn-13/myobj 2024-02-01T10:18:36.778 DEBUG:boto:auth_path=/vclfcn-13/myobj 2024-02-01T10:18:36.778 DEBUG:boto:Method: HEAD 2024-02-01T10:18:36.778 DEBUG:boto:Path: /vclfcn-13/myobj 2024-02-01T10:18:36.778 DEBUG:boto:Data: 2024-02-01T10:18:36.778 DEBUG:boto:Headers: {} 2024-02-01T10:18:36.778 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:36.778 DEBUG:boto:Port: 8000 2024-02-01T10:18:36.778 DEBUG:boto:Params: {} 2024-02-01T10:18:36.778 DEBUG:boto:Token: None 2024-02-01T10:18:36.779 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:18:36 GMT /vclfcn-13/myobj 2024-02-01T10:18:36.779 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wX138BceZUgi8qSEhtVJbT5mI6Y= 2024-02-01T10:18:36.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wX138BceZUgi8qSEhtVJbT5mI6Y=', 'Content-Length': '0'} 2024-02-01T10:18:36.781 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000004a9420df5ba887a-0065bb6ffc-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:36.781 DEBUG:boto:path=/vclfcn-13/myobj 2024-02-01T10:18:36.781 DEBUG:boto:auth_path=/vclfcn-13/myobj 2024-02-01T10:18:36.782 DEBUG:boto:Method: DELETE 2024-02-01T10:18:36.782 DEBUG:boto:Path: /vclfcn-13/myobj 2024-02-01T10:18:36.782 DEBUG:boto:Data: 2024-02-01T10:18:36.782 DEBUG:boto:Headers: {} 2024-02-01T10:18:36.782 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:18:36.782 DEBUG:boto:Port: 8000 2024-02-01T10:18:36.782 DEBUG:boto:Params: {} 2024-02-01T10:18:36.782 DEBUG:boto:Token: None 2024-02-01T10:18:36.782 DEBUG:boto:StringToSign: DELETE Thu, 01 Feb 2024 10:18:36 GMT /vclfcn-13/myobj 2024-02-01T10:18:36.782 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sVkHwspnM/xJCn1qlrr9R5e6VAQ= 2024-02-01T10:18:36.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:18:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sVkHwspnM/xJCn1qlrr9R5e6VAQ=', 'Content-Length': '0'} 2024-02-01T10:18:36.791 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000401abd52b69d39b2-0065bb6ffc-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:18:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:18:36.791 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:18:36.791 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:18:36.792 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket vclfcn-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-02-01T10:18:36.846 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:18:36.846 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:18:36.909 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:18:36.907+0000 7f5fe81a2980 1 waiting for incremental sync to catch up: 2024-02-01T10:18:36.909 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, 00000000001.13.6, , , , , , , , , ] 2024-02-01T10:18:36.909 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, 00000000002.15.6, , , , , , , , , ] 2024-02-01T10:18:41.911 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:18:41.911+0000 7f5fe81a2980 1 waiting for incremental sync to catch up: 2024-02-01T10:18:41.911 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, 00000000001.13.6, , , , , , , , , ] 2024-02-01T10:18:41.912 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, 00000000002.15.6, , , , , , , , , ] 2024-02-01T10:18:46.914 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:18:46.914+0000 7f5fe81a2980 1 waiting for incremental sync to catch up: 2024-02-01T10:18:46.914 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, 00000000001.13.6, , , , , , , , , ] 2024-02-01T10:18:46.914 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, 00000000002.15.6, , , , , , , , , ] 2024-02-01T10:18:51.916 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:18:51.914+0000 7f5fe81a2980 1 waiting for incremental sync to catch up: 2024-02-01T10:18:51.916 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, 00000000001.13.6, , , , , , , , , ] 2024-02-01T10:18:51.916 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, 00000000002.15.6, , , , , , , , , ] 2024-02-01T10:18:56.918 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:18:56.918+0000 7f5fe81a2980 1 waiting for incremental sync to catch up: 2024-02-01T10:18:56.919 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, 00000000001.13.6, , , , , , , , , ] 2024-02-01T10:18:56.919 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, 00000000002.15.6, , , , , , , , , ] 2024-02-01T10:19:01.921 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:19:01.918+0000 7f5fe81a2980 1 waiting for incremental sync to catch up: 2024-02-01T10:19:01.921 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, 00000000001.13.6, , , , , , , , , ] 2024-02-01T10:19:01.921 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, 00000000002.15.6, , , , , , , , , ] 2024-02-01T10:19:06.923 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:19:06.922+0000 7f5fe81a2980 1 waiting for incremental sync to catch up: 2024-02-01T10:19:06.923 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, 00000000001.13.6, , , , , , , , , ] 2024-02-01T10:19:06.923 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, 00000000002.15.6, , , , , , , , , ] 2024-02-01T10:19:11.925 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:19:11.922+0000 7f5fe81a2980 1 bucket sync caught up with source: 2024-02-01T10:19:11.925 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, 00000000002.15.6, , , , , , , , , ] 2024-02-01T10:19:11.926 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, 00000000002.15.6, , , , , , , , , ] 2024-02-01T10:19:11.926 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:19:11.922+0000 7f5fe81a2980 0 bucket checkpoint complete 2024-02-01T10:19:11.963 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-13 zones={test-zone2, test-zone1} 2024-02-01T10:19:11.964 DEBUG:boto:path=/vclfcn-13/ 2024-02-01T10:19:11.964 DEBUG:boto:auth_path=/vclfcn-13/ 2024-02-01T10:19:11.964 DEBUG:boto:Method: HEAD 2024-02-01T10:19:11.964 DEBUG:boto:Path: /vclfcn-13/ 2024-02-01T10:19:11.964 DEBUG:boto:Data: 2024-02-01T10:19:11.964 DEBUG:boto:Headers: {} 2024-02-01T10:19:11.964 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:19:11.965 DEBUG:boto:Port: 8000 2024-02-01T10:19:11.965 DEBUG:boto:Params: {} 2024-02-01T10:19:11.965 DEBUG:boto:Token: None 2024-02-01T10:19:11.965 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:19:11 GMT /vclfcn-13/ 2024-02-01T10:19:11.965 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tGhCYWaYQJRCOmDHUm+o0tVI0R0= 2024-02-01T10:19:11.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:19:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tGhCYWaYQJRCOmDHUm+o0tVI0R0=', 'Content-Length': '0'} 2024-02-01T10:19:11.969 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000086404a39b494c04-0065bb701f-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:19:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:19:11.969 DEBUG:boto:path=/vclfcn-13/ 2024-02-01T10:19:11.970 DEBUG:boto:auth_path=/vclfcn-13/ 2024-02-01T10:19:11.970 DEBUG:boto:Method: HEAD 2024-02-01T10:19:11.970 DEBUG:boto:Path: /vclfcn-13/ 2024-02-01T10:19:11.970 DEBUG:boto:Data: 2024-02-01T10:19:11.970 DEBUG:boto:Headers: {} 2024-02-01T10:19:11.970 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:19:11.970 DEBUG:boto:Port: 8000 2024-02-01T10:19:11.970 DEBUG:boto:Params: {} 2024-02-01T10:19:11.970 DEBUG:boto:Token: None 2024-02-01T10:19:11.970 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:19:11 GMT /vclfcn-13/ 2024-02-01T10:19:11.971 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tGhCYWaYQJRCOmDHUm+o0tVI0R0= 2024-02-01T10:19:11.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:19:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tGhCYWaYQJRCOmDHUm+o0tVI0R0=', 'Content-Length': '0'} 2024-02-01T10:19:11.974 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d2f82040f43dfead-0065bb701f-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:19:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:19:11.975 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:19:11.975 DEBUG:boto:path=/vclfcn-13/ 2024-02-01T10:19:11.975 DEBUG:boto:auth_path=/vclfcn-13/ 2024-02-01T10:19:11.975 DEBUG:boto:path=/vclfcn-13/?versions&max-keys=999 2024-02-01T10:19:11.975 DEBUG:boto:auth_path=/vclfcn-13/?versions&max-keys=999 2024-02-01T10:19:11.975 DEBUG:boto:Method: GET 2024-02-01T10:19:11.975 DEBUG:boto:Path: /vclfcn-13/?versions&max-keys=999 2024-02-01T10:19:11.975 DEBUG:boto:Data: 2024-02-01T10:19:11.976 DEBUG:boto:Headers: {} 2024-02-01T10:19:11.976 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:19:11.976 DEBUG:boto:Port: 8000 2024-02-01T10:19:11.976 DEBUG:boto:Params: {} 2024-02-01T10:19:11.976 DEBUG:boto:Token: None 2024-02-01T10:19:11.976 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:19:11 GMT /vclfcn-13/?versions 2024-02-01T10:19:11.976 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:s1X7dpXb4rhsZ8ErQ+iI9BDo1Jc= 2024-02-01T10:19:11.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:19:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:s1X7dpXb4rhsZ8ErQ+iI9BDo1Jc=', 'Content-Length': '0'} 2024-02-01T10:19:11.979 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca241cb7e925d505-0065bb701f-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:19:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:19:11.979 DEBUG:boto:b'vclfcn-13999false' 2024-02-01T10:19:11.980 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:19:11.980 DEBUG:boto:path=/vclfcn-13/ 2024-02-01T10:19:11.980 DEBUG:boto:auth_path=/vclfcn-13/ 2024-02-01T10:19:11.980 DEBUG:boto:path=/vclfcn-13/?versions&max-keys=999 2024-02-01T10:19:11.980 DEBUG:boto:auth_path=/vclfcn-13/?versions&max-keys=999 2024-02-01T10:19:11.981 DEBUG:boto:Method: GET 2024-02-01T10:19:11.981 DEBUG:boto:Path: /vclfcn-13/?versions&max-keys=999 2024-02-01T10:19:11.981 DEBUG:boto:Data: 2024-02-01T10:19:11.981 DEBUG:boto:Headers: {} 2024-02-01T10:19:11.981 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:19:11.981 DEBUG:boto:Port: 8000 2024-02-01T10:19:11.981 DEBUG:boto:Params: {} 2024-02-01T10:19:11.981 DEBUG:boto:Token: None 2024-02-01T10:19:11.981 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:19:11 GMT /vclfcn-13/?versions 2024-02-01T10:19:11.981 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:s1X7dpXb4rhsZ8ErQ+iI9BDo1Jc= 2024-02-01T10:19:11.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:19:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:s1X7dpXb4rhsZ8ErQ+iI9BDo1Jc=', 'Content-Length': '0'} 2024-02-01T10:19:11.985 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000036b9bd78dda66b68-0065bb701f-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:19:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:19:11.986 DEBUG:boto:b'vclfcn-13999false' 2024-02-01T10:19:11.986 DEBUG:boto:path=/vclfcn-13/ 2024-02-01T10:19:11.986 DEBUG:boto:auth_path=/vclfcn-13/ 2024-02-01T10:19:11.986 DEBUG:boto:path=/vclfcn-13/?versions&max-keys=999 2024-02-01T10:19:11.987 DEBUG:boto:auth_path=/vclfcn-13/?versions&max-keys=999 2024-02-01T10:19:11.987 DEBUG:boto:Method: GET 2024-02-01T10:19:11.987 DEBUG:boto:Path: /vclfcn-13/?versions&max-keys=999 2024-02-01T10:19:11.987 DEBUG:boto:Data: 2024-02-01T10:19:11.987 DEBUG:boto:Headers: {} 2024-02-01T10:19:11.987 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:19:11.987 DEBUG:boto:Port: 8000 2024-02-01T10:19:11.987 DEBUG:boto:Params: {} 2024-02-01T10:19:11.987 DEBUG:boto:Token: None 2024-02-01T10:19:11.987 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:19:11 GMT /vclfcn-13/?versions 2024-02-01T10:19:11.988 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:s1X7dpXb4rhsZ8ErQ+iI9BDo1Jc= 2024-02-01T10:19:11.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:19:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:s1X7dpXb4rhsZ8ErQ+iI9BDo1Jc=', 'Content-Length': '0'} 2024-02-01T10:19:11.991 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000527a1b95536ad4e9-0065bb701f-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:19:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:19:11.991 DEBUG:boto:b'vclfcn-13999false' 2024-02-01T10:19:11.991 DEBUG:boto:path=/vclfcn-13/ 2024-02-01T10:19:11.992 DEBUG:boto:auth_path=/vclfcn-13/ 2024-02-01T10:19:11.992 DEBUG:boto:path=/vclfcn-13/?versions&max-keys=999 2024-02-01T10:19:11.992 DEBUG:boto:auth_path=/vclfcn-13/?versions&max-keys=999 2024-02-01T10:19:11.992 DEBUG:boto:Method: GET 2024-02-01T10:19:11.992 DEBUG:boto:Path: /vclfcn-13/?versions&max-keys=999 2024-02-01T10:19:11.992 DEBUG:boto:Data: 2024-02-01T10:19:11.992 DEBUG:boto:Headers: {} 2024-02-01T10:19:11.992 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:19:11.992 DEBUG:boto:Port: 8000 2024-02-01T10:19:11.992 DEBUG:boto:Params: {} 2024-02-01T10:19:11.993 DEBUG:boto:Token: None 2024-02-01T10:19:11.993 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:19:11 GMT /vclfcn-13/?versions 2024-02-01T10:19:11.993 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:s1X7dpXb4rhsZ8ErQ+iI9BDo1Jc= 2024-02-01T10:19:11.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:19:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:s1X7dpXb4rhsZ8ErQ+iI9BDo1Jc=', 'Content-Length': '0'} 2024-02-01T10:19:11.997 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009371431314127f45-0065bb701f-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:19:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:19:11.997 DEBUG:boto:b'vclfcn-13999false' 2024-02-01T10:19:11.997 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-13 zones={test-zone2, test-zone1} 2024-02-01T10:19:11.998 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:19:11.998 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:19:11.998 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:19:12.048 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:19:12.048 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:19:12.110 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:19:12.106+0000 7fb304de7980 1 bucket sync caught up with source: 2024-02-01T10:19:12.110 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, 00000000002.15.6, , , , , , , , , ] 2024-02-01T10:19:12.111 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, 00000000002.15.6, , , , , , , , , ] 2024-02-01T10:19:12.111 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:19:12.106+0000 7fb304de7980 0 bucket checkpoint complete 2024-02-01T10:19:12.132 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-13 zones={test-zone3, test-zone1} 2024-02-01T10:19:12.132 DEBUG:boto:path=/vclfcn-13/ 2024-02-01T10:19:12.132 DEBUG:boto:auth_path=/vclfcn-13/ 2024-02-01T10:19:12.132 DEBUG:boto:Method: HEAD 2024-02-01T10:19:12.132 DEBUG:boto:Path: /vclfcn-13/ 2024-02-01T10:19:12.132 DEBUG:boto:Data: 2024-02-01T10:19:12.133 DEBUG:boto:Headers: {} 2024-02-01T10:19:12.133 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:19:12.133 DEBUG:boto:Port: 8001 2024-02-01T10:19:12.133 DEBUG:boto:Params: {} 2024-02-01T10:19:12.133 DEBUG:boto:Token: None 2024-02-01T10:19:12.133 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:19:12 GMT /vclfcn-13/ 2024-02-01T10:19:12.133 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:M3nTpPi7mfPKnve9zbERD0/E1/k= 2024-02-01T10:19:12.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:19:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:M3nTpPi7mfPKnve9zbERD0/E1/k=', 'Content-Length': '0'} 2024-02-01T10:19:12.135 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009d5e35dc32cdd433-0065bb7020-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:19:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:19:12.136 DEBUG:boto:path=/vclfcn-13/ 2024-02-01T10:19:12.136 DEBUG:boto:auth_path=/vclfcn-13/ 2024-02-01T10:19:12.136 DEBUG:boto:Method: HEAD 2024-02-01T10:19:12.136 DEBUG:boto:Path: /vclfcn-13/ 2024-02-01T10:19:12.136 DEBUG:boto:Data: 2024-02-01T10:19:12.136 DEBUG:boto:Headers: {} 2024-02-01T10:19:12.136 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:19:12.136 DEBUG:boto:Port: 8000 2024-02-01T10:19:12.136 DEBUG:boto:Params: {} 2024-02-01T10:19:12.137 DEBUG:boto:Token: None 2024-02-01T10:19:12.137 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:19:12 GMT /vclfcn-13/ 2024-02-01T10:19:12.137 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:M3nTpPi7mfPKnve9zbERD0/E1/k= 2024-02-01T10:19:12.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:19:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:M3nTpPi7mfPKnve9zbERD0/E1/k=', 'Content-Length': '0'} 2024-02-01T10:19:12.139 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bd0d785d8a7f72a2-0065bb7020-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:19:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:19:12.139 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:19:12.139 DEBUG:boto:path=/vclfcn-13/ 2024-02-01T10:19:12.140 DEBUG:boto:auth_path=/vclfcn-13/ 2024-02-01T10:19:12.140 DEBUG:boto:path=/vclfcn-13/?versions&max-keys=999 2024-02-01T10:19:12.140 DEBUG:boto:auth_path=/vclfcn-13/?versions&max-keys=999 2024-02-01T10:19:12.140 DEBUG:boto:Method: GET 2024-02-01T10:19:12.140 DEBUG:boto:Path: /vclfcn-13/?versions&max-keys=999 2024-02-01T10:19:12.140 DEBUG:boto:Data: 2024-02-01T10:19:12.140 DEBUG:boto:Headers: {} 2024-02-01T10:19:12.140 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:19:12.140 DEBUG:boto:Port: 8001 2024-02-01T10:19:12.140 DEBUG:boto:Params: {} 2024-02-01T10:19:12.140 DEBUG:boto:Token: None 2024-02-01T10:19:12.141 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:19:12 GMT /vclfcn-13/?versions 2024-02-01T10:19:12.141 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vi03a3EIKjWxskhc1Sb512QHlWQ= 2024-02-01T10:19:12.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:19:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vi03a3EIKjWxskhc1Sb512QHlWQ=', 'Content-Length': '0'} 2024-02-01T10:19:12.143 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000049c9ac7e8791ea59-0065bb7020-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:19:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:19:12.143 DEBUG:boto:b'vclfcn-13999false' 2024-02-01T10:19:12.143 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:19:12.143 DEBUG:boto:path=/vclfcn-13/ 2024-02-01T10:19:12.143 DEBUG:boto:auth_path=/vclfcn-13/ 2024-02-01T10:19:12.143 DEBUG:boto:path=/vclfcn-13/?versions&max-keys=999 2024-02-01T10:19:12.143 DEBUG:boto:auth_path=/vclfcn-13/?versions&max-keys=999 2024-02-01T10:19:12.144 DEBUG:boto:Method: GET 2024-02-01T10:19:12.144 DEBUG:boto:Path: /vclfcn-13/?versions&max-keys=999 2024-02-01T10:19:12.144 DEBUG:boto:Data: 2024-02-01T10:19:12.144 DEBUG:boto:Headers: {} 2024-02-01T10:19:12.144 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:19:12.144 DEBUG:boto:Port: 8000 2024-02-01T10:19:12.144 DEBUG:boto:Params: {} 2024-02-01T10:19:12.144 DEBUG:boto:Token: None 2024-02-01T10:19:12.144 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:19:12 GMT /vclfcn-13/?versions 2024-02-01T10:19:12.145 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vi03a3EIKjWxskhc1Sb512QHlWQ= 2024-02-01T10:19:12.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:19:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vi03a3EIKjWxskhc1Sb512QHlWQ=', 'Content-Length': '0'} 2024-02-01T10:19:12.147 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000dc7407ff19a74c8-0065bb7020-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:19:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:19:12.147 DEBUG:boto:b'vclfcn-13999false' 2024-02-01T10:19:12.147 DEBUG:boto:path=/vclfcn-13/ 2024-02-01T10:19:12.147 DEBUG:boto:auth_path=/vclfcn-13/ 2024-02-01T10:19:12.147 DEBUG:boto:path=/vclfcn-13/?versions&max-keys=999 2024-02-01T10:19:12.147 DEBUG:boto:auth_path=/vclfcn-13/?versions&max-keys=999 2024-02-01T10:19:12.147 DEBUG:boto:Method: GET 2024-02-01T10:19:12.148 DEBUG:boto:Path: /vclfcn-13/?versions&max-keys=999 2024-02-01T10:19:12.148 DEBUG:boto:Data: 2024-02-01T10:19:12.148 DEBUG:boto:Headers: {} 2024-02-01T10:19:12.148 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:19:12.148 DEBUG:boto:Port: 8001 2024-02-01T10:19:12.148 DEBUG:boto:Params: {} 2024-02-01T10:19:12.148 DEBUG:boto:Token: None 2024-02-01T10:19:12.148 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:19:12 GMT /vclfcn-13/?versions 2024-02-01T10:19:12.148 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vi03a3EIKjWxskhc1Sb512QHlWQ= 2024-02-01T10:19:12.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:19:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vi03a3EIKjWxskhc1Sb512QHlWQ=', 'Content-Length': '0'} 2024-02-01T10:19:12.151 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000093eead4de5b2bb4-0065bb7020-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:19:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:19:12.151 DEBUG:boto:b'vclfcn-13999false' 2024-02-01T10:19:12.151 DEBUG:boto:path=/vclfcn-13/ 2024-02-01T10:19:12.151 DEBUG:boto:auth_path=/vclfcn-13/ 2024-02-01T10:19:12.151 DEBUG:boto:path=/vclfcn-13/?versions&max-keys=999 2024-02-01T10:19:12.151 DEBUG:boto:auth_path=/vclfcn-13/?versions&max-keys=999 2024-02-01T10:19:12.151 DEBUG:boto:Method: GET 2024-02-01T10:19:12.152 DEBUG:boto:Path: /vclfcn-13/?versions&max-keys=999 2024-02-01T10:19:12.152 DEBUG:boto:Data: 2024-02-01T10:19:12.152 DEBUG:boto:Headers: {} 2024-02-01T10:19:12.152 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:19:12.152 DEBUG:boto:Port: 8000 2024-02-01T10:19:12.152 DEBUG:boto:Params: {} 2024-02-01T10:19:12.152 DEBUG:boto:Token: None 2024-02-01T10:19:12.152 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:19:12 GMT /vclfcn-13/?versions 2024-02-01T10:19:12.152 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vi03a3EIKjWxskhc1Sb512QHlWQ= 2024-02-01T10:19:12.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:19:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vi03a3EIKjWxskhc1Sb512QHlWQ=', 'Content-Length': '0'} 2024-02-01T10:19:12.154 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050101e8b644b8f8b-0065bb7020-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:19:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:19:12.155 DEBUG:boto:b'vclfcn-13999false' 2024-02-01T10:19:12.155 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-13 zones={test-zone3, test-zone1} 2024-02-01T10:19:12.155 DEBUG:boto:path=/vclfcn-14/ 2024-02-01T10:19:12.155 DEBUG:boto:auth_path=/vclfcn-14/ 2024-02-01T10:19:12.155 DEBUG:boto:Method: HEAD 2024-02-01T10:19:12.155 DEBUG:boto:Path: /vclfcn-14/ 2024-02-01T10:19:12.155 DEBUG:boto:Data: 2024-02-01T10:19:12.155 DEBUG:boto:Headers: {} 2024-02-01T10:19:12.156 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:19:12.156 DEBUG:boto:Port: 8000 2024-02-01T10:19:12.156 DEBUG:boto:Params: {} 2024-02-01T10:19:12.156 DEBUG:boto:Token: None 2024-02-01T10:19:12.156 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:19:12 GMT /vclfcn-14/ 2024-02-01T10:19:12.156 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:uv8ccT9ZeyXEX6gGWGVSYmAzNwk= 2024-02-01T10:19:12.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:19:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:uv8ccT9ZeyXEX6gGWGVSYmAzNwk=', 'Content-Length': '0'} 2024-02-01T10:19:12.160 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000085a1796d41d2351e-0065bb7020-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:19:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:19:12.160 DEBUG:boto:path=/vclfcn-14/myobj 2024-02-01T10:19:12.160 DEBUG:boto:auth_path=/vclfcn-14/myobj 2024-02-01T10:19:12.160 DEBUG:boto:Method: HEAD 2024-02-01T10:19:12.161 DEBUG:boto:Path: /vclfcn-14/myobj 2024-02-01T10:19:12.161 DEBUG:boto:Data: 2024-02-01T10:19:12.161 DEBUG:boto:Headers: {} 2024-02-01T10:19:12.161 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:19:12.161 DEBUG:boto:Port: 8000 2024-02-01T10:19:12.161 DEBUG:boto:Params: {} 2024-02-01T10:19:12.161 DEBUG:boto:Token: None 2024-02-01T10:19:12.161 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:19:12 GMT /vclfcn-14/myobj 2024-02-01T10:19:12.161 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:rgPjsMlxHigg2Qbw7KGUsPNr83M= 2024-02-01T10:19:12.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:19:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:rgPjsMlxHigg2Qbw7KGUsPNr83M=', 'Content-Length': '0'} 2024-02-01T10:19:12.164 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000db0bedce6c26da5d-0065bb7020-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:19:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:19:12.165 DEBUG:boto:path=/vclfcn-14/myobj 2024-02-01T10:19:12.165 DEBUG:boto:auth_path=/vclfcn-14/myobj 2024-02-01T10:19:12.165 DEBUG:boto:Method: DELETE 2024-02-01T10:19:12.165 DEBUG:boto:Path: /vclfcn-14/myobj 2024-02-01T10:19:12.165 DEBUG:boto:Data: 2024-02-01T10:19:12.165 DEBUG:boto:Headers: {} 2024-02-01T10:19:12.165 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:19:12.166 DEBUG:boto:Port: 8000 2024-02-01T10:19:12.166 DEBUG:boto:Params: {} 2024-02-01T10:19:12.166 DEBUG:boto:Token: None 2024-02-01T10:19:12.166 DEBUG:boto:StringToSign: DELETE Thu, 01 Feb 2024 10:19:12 GMT /vclfcn-14/myobj 2024-02-01T10:19:12.166 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:CmTNUMBaNk9wim0HH0tXNDgtpP4= 2024-02-01T10:19:12.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:19:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:CmTNUMBaNk9wim0HH0tXNDgtpP4=', 'Content-Length': '0'} 2024-02-01T10:19:12.178 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fa14332620074f48-0065bb7020-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:19:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:19:12.178 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:19:12.178 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:19:12.179 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:19:12.226 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:19:12.226 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:19:12.295 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:19:12.294+0000 7eff48ab1980 1 waiting for incremental sync to catch up: 2024-02-01T10:19:12.296 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, 00000000001.10.6, , , , , , , , , ] 2024-02-01T10:19:12.296 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, 00000000002.14.6, , , , , , , , , ] 2024-02-01T10:19:17.298 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:19:17.294+0000 7eff48ab1980 1 waiting for incremental sync to catch up: 2024-02-01T10:19:17.298 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, 00000000001.10.6, , , , , , , , , ] 2024-02-01T10:19:17.298 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, 00000000002.14.6, , , , , , , , , ] 2024-02-01T10:19:22.300 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:19:22.302+0000 7eff48ab1980 1 waiting for incremental sync to catch up: 2024-02-01T10:19:22.301 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, 00000000001.10.6, , , , , , , , , ] 2024-02-01T10:19:22.301 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, 00000000002.14.6, , , , , , , , , ] 2024-02-01T10:19:27.303 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:19:27.306+0000 7eff48ab1980 1 bucket sync caught up with source: 2024-02-01T10:19:27.303 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, 00000000002.14.6, , , , , , , , , ] 2024-02-01T10:19:27.303 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, 00000000002.14.6, , , , , , , , , ] 2024-02-01T10:19:27.303 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:19:27.306+0000 7eff48ab1980 0 bucket checkpoint complete 2024-02-01T10:19:27.341 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-14 zones={test-zone1, test-zone2} 2024-02-01T10:19:27.341 DEBUG:boto:path=/vclfcn-14/ 2024-02-01T10:19:27.342 DEBUG:boto:auth_path=/vclfcn-14/ 2024-02-01T10:19:27.342 DEBUG:boto:Method: HEAD 2024-02-01T10:19:27.342 DEBUG:boto:Path: /vclfcn-14/ 2024-02-01T10:19:27.342 DEBUG:boto:Data: 2024-02-01T10:19:27.342 DEBUG:boto:Headers: {} 2024-02-01T10:19:27.342 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:19:27.342 DEBUG:boto:Port: 8000 2024-02-01T10:19:27.342 DEBUG:boto:Params: {} 2024-02-01T10:19:27.343 DEBUG:boto:Token: None 2024-02-01T10:19:27.343 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:19:27 GMT /vclfcn-14/ 2024-02-01T10:19:27.343 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0x0HV3cmGfQOkWQCsXEP8EhCHgM= 2024-02-01T10:19:27.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:19:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0x0HV3cmGfQOkWQCsXEP8EhCHgM=', 'Content-Length': '0'} 2024-02-01T10:19:27.346 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e1fb2b63d9d22f31-0065bb702f-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:19:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:19:27.347 DEBUG:boto:path=/vclfcn-14/ 2024-02-01T10:19:27.347 DEBUG:boto:auth_path=/vclfcn-14/ 2024-02-01T10:19:27.347 DEBUG:boto:Method: HEAD 2024-02-01T10:19:27.347 DEBUG:boto:Path: /vclfcn-14/ 2024-02-01T10:19:27.347 DEBUG:boto:Data: 2024-02-01T10:19:27.347 DEBUG:boto:Headers: {} 2024-02-01T10:19:27.347 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:19:27.347 DEBUG:boto:Port: 8000 2024-02-01T10:19:27.347 DEBUG:boto:Params: {} 2024-02-01T10:19:27.347 DEBUG:boto:Token: None 2024-02-01T10:19:27.348 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:19:27 GMT /vclfcn-14/ 2024-02-01T10:19:27.348 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0x0HV3cmGfQOkWQCsXEP8EhCHgM= 2024-02-01T10:19:27.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:19:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0x0HV3cmGfQOkWQCsXEP8EhCHgM=', 'Content-Length': '0'} 2024-02-01T10:19:27.352 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000085f460de4637ef1d-0065bb702f-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:19:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:19:27.352 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:19:27.353 DEBUG:boto:path=/vclfcn-14/ 2024-02-01T10:19:27.353 DEBUG:boto:auth_path=/vclfcn-14/ 2024-02-01T10:19:27.353 DEBUG:boto:path=/vclfcn-14/?versions&max-keys=999 2024-02-01T10:19:27.353 DEBUG:boto:auth_path=/vclfcn-14/?versions&max-keys=999 2024-02-01T10:19:27.353 DEBUG:boto:Method: GET 2024-02-01T10:19:27.353 DEBUG:boto:Path: /vclfcn-14/?versions&max-keys=999 2024-02-01T10:19:27.353 DEBUG:boto:Data: 2024-02-01T10:19:27.354 DEBUG:boto:Headers: {} 2024-02-01T10:19:27.354 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:19:27.354 DEBUG:boto:Port: 8000 2024-02-01T10:19:27.354 DEBUG:boto:Params: {} 2024-02-01T10:19:27.354 DEBUG:boto:Token: None 2024-02-01T10:19:27.354 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:19:27 GMT /vclfcn-14/?versions 2024-02-01T10:19:27.354 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YbpDudHxzwq7a/cE5quuN1VRtTQ= 2024-02-01T10:19:27.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:19:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YbpDudHxzwq7a/cE5quuN1VRtTQ=', 'Content-Length': '0'} 2024-02-01T10:19:27.357 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000378772bddcdf212d-0065bb702f-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:19:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:19:27.400 DEBUG:boto:b'vclfcn-14999false' 2024-02-01T10:19:27.401 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:19:27.401 DEBUG:boto:path=/vclfcn-14/ 2024-02-01T10:19:27.401 DEBUG:boto:auth_path=/vclfcn-14/ 2024-02-01T10:19:27.401 DEBUG:boto:path=/vclfcn-14/?versions&max-keys=999 2024-02-01T10:19:27.401 DEBUG:boto:auth_path=/vclfcn-14/?versions&max-keys=999 2024-02-01T10:19:27.401 DEBUG:boto:Method: GET 2024-02-01T10:19:27.401 DEBUG:boto:Path: /vclfcn-14/?versions&max-keys=999 2024-02-01T10:19:27.402 DEBUG:boto:Data: 2024-02-01T10:19:27.402 DEBUG:boto:Headers: {} 2024-02-01T10:19:27.402 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:19:27.402 DEBUG:boto:Port: 8000 2024-02-01T10:19:27.402 DEBUG:boto:Params: {} 2024-02-01T10:19:27.402 DEBUG:boto:Token: None 2024-02-01T10:19:27.402 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:19:27 GMT /vclfcn-14/?versions 2024-02-01T10:19:27.402 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YbpDudHxzwq7a/cE5quuN1VRtTQ= 2024-02-01T10:19:27.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:19:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YbpDudHxzwq7a/cE5quuN1VRtTQ=', 'Content-Length': '0'} 2024-02-01T10:19:27.406 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001dc33b1d01af4d8f-0065bb702f-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:19:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:19:27.407 DEBUG:boto:b'vclfcn-14999false' 2024-02-01T10:19:27.407 DEBUG:boto:path=/vclfcn-14/ 2024-02-01T10:19:27.407 DEBUG:boto:auth_path=/vclfcn-14/ 2024-02-01T10:19:27.407 DEBUG:boto:path=/vclfcn-14/?versions&max-keys=999 2024-02-01T10:19:27.407 DEBUG:boto:auth_path=/vclfcn-14/?versions&max-keys=999 2024-02-01T10:19:27.407 DEBUG:boto:Method: GET 2024-02-01T10:19:27.408 DEBUG:boto:Path: /vclfcn-14/?versions&max-keys=999 2024-02-01T10:19:27.408 DEBUG:boto:Data: 2024-02-01T10:19:27.408 DEBUG:boto:Headers: {} 2024-02-01T10:19:27.408 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:19:27.408 DEBUG:boto:Port: 8000 2024-02-01T10:19:27.408 DEBUG:boto:Params: {} 2024-02-01T10:19:27.408 DEBUG:boto:Token: None 2024-02-01T10:19:27.408 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:19:27 GMT /vclfcn-14/?versions 2024-02-01T10:19:27.408 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YbpDudHxzwq7a/cE5quuN1VRtTQ= 2024-02-01T10:19:27.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:19:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YbpDudHxzwq7a/cE5quuN1VRtTQ=', 'Content-Length': '0'} 2024-02-01T10:19:27.412 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ac86c63ad668bf8-0065bb702f-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:19:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:19:27.412 DEBUG:boto:b'vclfcn-14999false' 2024-02-01T10:19:27.412 DEBUG:boto:path=/vclfcn-14/ 2024-02-01T10:19:27.412 DEBUG:boto:auth_path=/vclfcn-14/ 2024-02-01T10:19:27.413 DEBUG:boto:path=/vclfcn-14/?versions&max-keys=999 2024-02-01T10:19:27.413 DEBUG:boto:auth_path=/vclfcn-14/?versions&max-keys=999 2024-02-01T10:19:27.413 DEBUG:boto:Method: GET 2024-02-01T10:19:27.413 DEBUG:boto:Path: /vclfcn-14/?versions&max-keys=999 2024-02-01T10:19:27.413 DEBUG:boto:Data: 2024-02-01T10:19:27.413 DEBUG:boto:Headers: {} 2024-02-01T10:19:27.413 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:19:27.413 DEBUG:boto:Port: 8000 2024-02-01T10:19:27.413 DEBUG:boto:Params: {} 2024-02-01T10:19:27.413 DEBUG:boto:Token: None 2024-02-01T10:19:27.413 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:19:27 GMT /vclfcn-14/?versions 2024-02-01T10:19:27.414 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YbpDudHxzwq7a/cE5quuN1VRtTQ= 2024-02-01T10:19:27.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:19:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YbpDudHxzwq7a/cE5quuN1VRtTQ=', 'Content-Length': '0'} 2024-02-01T10:19:27.418 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a229297bf022e3b8-0065bb702f-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:19:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:19:27.418 DEBUG:boto:b'vclfcn-14999false' 2024-02-01T10:19:27.418 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-14 zones={test-zone1, test-zone2} 2024-02-01T10:19:27.419 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:19:27.419 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:19:27.419 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:19:27.470 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:19:27.470 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:19:27.536 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:19:27.538+0000 7f668eb53980 1 bucket sync caught up with source: 2024-02-01T10:19:27.536 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, 00000000002.14.6, , , , , , , , , ] 2024-02-01T10:19:27.536 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, 00000000002.14.6, , , , , , , , , ] 2024-02-01T10:19:27.537 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:19:27.538+0000 7f668eb53980 0 bucket checkpoint complete 2024-02-01T10:19:27.557 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-14 zones={test-zone3, test-zone2} 2024-02-01T10:19:27.558 DEBUG:boto:path=/vclfcn-14/ 2024-02-01T10:19:27.558 DEBUG:boto:auth_path=/vclfcn-14/ 2024-02-01T10:19:27.558 DEBUG:boto:Method: HEAD 2024-02-01T10:19:27.558 DEBUG:boto:Path: /vclfcn-14/ 2024-02-01T10:19:27.558 DEBUG:boto:Data: 2024-02-01T10:19:27.558 DEBUG:boto:Headers: {} 2024-02-01T10:19:27.558 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:19:27.558 DEBUG:boto:Port: 8001 2024-02-01T10:19:27.558 DEBUG:boto:Params: {} 2024-02-01T10:19:27.559 DEBUG:boto:Token: None 2024-02-01T10:19:27.559 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:19:27 GMT /vclfcn-14/ 2024-02-01T10:19:27.559 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0x0HV3cmGfQOkWQCsXEP8EhCHgM= 2024-02-01T10:19:27.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:19:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0x0HV3cmGfQOkWQCsXEP8EhCHgM=', 'Content-Length': '0'} 2024-02-01T10:19:27.561 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000db4662c7e6e85738-0065bb702f-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:19:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:19:27.561 DEBUG:boto:path=/vclfcn-14/ 2024-02-01T10:19:27.561 DEBUG:boto:auth_path=/vclfcn-14/ 2024-02-01T10:19:27.562 DEBUG:boto:Method: HEAD 2024-02-01T10:19:27.562 DEBUG:boto:Path: /vclfcn-14/ 2024-02-01T10:19:27.562 DEBUG:boto:Data: 2024-02-01T10:19:27.562 DEBUG:boto:Headers: {} 2024-02-01T10:19:27.562 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:19:27.562 DEBUG:boto:Port: 8000 2024-02-01T10:19:27.562 DEBUG:boto:Params: {} 2024-02-01T10:19:27.562 DEBUG:boto:Token: None 2024-02-01T10:19:27.562 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:19:27 GMT /vclfcn-14/ 2024-02-01T10:19:27.562 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0x0HV3cmGfQOkWQCsXEP8EhCHgM= 2024-02-01T10:19:27.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:19:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0x0HV3cmGfQOkWQCsXEP8EhCHgM=', 'Content-Length': '0'} 2024-02-01T10:19:27.566 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000045b80e9156a9b705-0065bb702f-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:19:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:19:27.566 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:19:27.567 DEBUG:boto:path=/vclfcn-14/ 2024-02-01T10:19:27.567 DEBUG:boto:auth_path=/vclfcn-14/ 2024-02-01T10:19:27.567 DEBUG:boto:path=/vclfcn-14/?versions&max-keys=999 2024-02-01T10:19:27.567 DEBUG:boto:auth_path=/vclfcn-14/?versions&max-keys=999 2024-02-01T10:19:27.567 DEBUG:boto:Method: GET 2024-02-01T10:19:27.567 DEBUG:boto:Path: /vclfcn-14/?versions&max-keys=999 2024-02-01T10:19:27.567 DEBUG:boto:Data: 2024-02-01T10:19:27.567 DEBUG:boto:Headers: {} 2024-02-01T10:19:27.567 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:19:27.567 DEBUG:boto:Port: 8001 2024-02-01T10:19:27.568 DEBUG:boto:Params: {} 2024-02-01T10:19:27.568 DEBUG:boto:Token: None 2024-02-01T10:19:27.568 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:19:27 GMT /vclfcn-14/?versions 2024-02-01T10:19:27.568 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YbpDudHxzwq7a/cE5quuN1VRtTQ= 2024-02-01T10:19:27.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:19:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YbpDudHxzwq7a/cE5quuN1VRtTQ=', 'Content-Length': '0'} 2024-02-01T10:19:27.570 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092c3b66492ea88a2-0065bb702f-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:19:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:19:27.612 DEBUG:boto:b'vclfcn-14999false' 2024-02-01T10:19:27.612 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:19:27.613 DEBUG:boto:path=/vclfcn-14/ 2024-02-01T10:19:27.613 DEBUG:boto:auth_path=/vclfcn-14/ 2024-02-01T10:19:27.613 DEBUG:boto:path=/vclfcn-14/?versions&max-keys=999 2024-02-01T10:19:27.613 DEBUG:boto:auth_path=/vclfcn-14/?versions&max-keys=999 2024-02-01T10:19:27.613 DEBUG:boto:Method: GET 2024-02-01T10:19:27.613 DEBUG:boto:Path: /vclfcn-14/?versions&max-keys=999 2024-02-01T10:19:27.613 DEBUG:boto:Data: 2024-02-01T10:19:27.613 DEBUG:boto:Headers: {} 2024-02-01T10:19:27.613 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:19:27.613 DEBUG:boto:Port: 8000 2024-02-01T10:19:27.614 DEBUG:boto:Params: {} 2024-02-01T10:19:27.614 DEBUG:boto:Token: None 2024-02-01T10:19:27.614 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:19:27 GMT /vclfcn-14/?versions 2024-02-01T10:19:27.614 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YbpDudHxzwq7a/cE5quuN1VRtTQ= 2024-02-01T10:19:27.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:19:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YbpDudHxzwq7a/cE5quuN1VRtTQ=', 'Content-Length': '0'} 2024-02-01T10:19:27.618 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e55dcb274c1e655-0065bb702f-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:19:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:19:27.619 DEBUG:boto:b'vclfcn-14999false' 2024-02-01T10:19:27.619 DEBUG:boto:path=/vclfcn-14/ 2024-02-01T10:19:27.619 DEBUG:boto:auth_path=/vclfcn-14/ 2024-02-01T10:19:27.619 DEBUG:boto:path=/vclfcn-14/?versions&max-keys=999 2024-02-01T10:19:27.619 DEBUG:boto:auth_path=/vclfcn-14/?versions&max-keys=999 2024-02-01T10:19:27.619 DEBUG:boto:Method: GET 2024-02-01T10:19:27.619 DEBUG:boto:Path: /vclfcn-14/?versions&max-keys=999 2024-02-01T10:19:27.619 DEBUG:boto:Data: 2024-02-01T10:19:27.620 DEBUG:boto:Headers: {} 2024-02-01T10:19:27.620 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:19:27.620 DEBUG:boto:Port: 8001 2024-02-01T10:19:27.620 DEBUG:boto:Params: {} 2024-02-01T10:19:27.620 DEBUG:boto:Token: None 2024-02-01T10:19:27.620 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:19:27 GMT /vclfcn-14/?versions 2024-02-01T10:19:27.620 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YbpDudHxzwq7a/cE5quuN1VRtTQ= 2024-02-01T10:19:27.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:19:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YbpDudHxzwq7a/cE5quuN1VRtTQ=', 'Content-Length': '0'} 2024-02-01T10:19:27.624 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000096dcc45014481569-0065bb702f-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:19:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:19:27.624 DEBUG:boto:b'vclfcn-14999false' 2024-02-01T10:19:27.624 DEBUG:boto:path=/vclfcn-14/ 2024-02-01T10:19:27.624 DEBUG:boto:auth_path=/vclfcn-14/ 2024-02-01T10:19:27.624 DEBUG:boto:path=/vclfcn-14/?versions&max-keys=999 2024-02-01T10:19:27.625 DEBUG:boto:auth_path=/vclfcn-14/?versions&max-keys=999 2024-02-01T10:19:27.625 DEBUG:boto:Method: GET 2024-02-01T10:19:27.625 DEBUG:boto:Path: /vclfcn-14/?versions&max-keys=999 2024-02-01T10:19:27.625 DEBUG:boto:Data: 2024-02-01T10:19:27.625 DEBUG:boto:Headers: {} 2024-02-01T10:19:27.625 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:19:27.625 DEBUG:boto:Port: 8000 2024-02-01T10:19:27.625 DEBUG:boto:Params: {} 2024-02-01T10:19:27.625 DEBUG:boto:Token: None 2024-02-01T10:19:27.625 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:19:27 GMT /vclfcn-14/?versions 2024-02-01T10:19:27.626 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YbpDudHxzwq7a/cE5quuN1VRtTQ= 2024-02-01T10:19:27.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:19:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YbpDudHxzwq7a/cE5quuN1VRtTQ=', 'Content-Length': '0'} 2024-02-01T10:19:27.630 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001025117022a3ac95-0065bb702f-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:19:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:19:27.630 DEBUG:boto:b'vclfcn-14999false' 2024-02-01T10:19:27.630 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-14 zones={test-zone3, test-zone2} 2024-02-01T10:19:27.630 DEBUG:boto:path=/vclfcn-15/ 2024-02-01T10:19:27.630 DEBUG:boto:auth_path=/vclfcn-15/ 2024-02-01T10:19:27.630 DEBUG:boto:Method: HEAD 2024-02-01T10:19:27.630 DEBUG:boto:Path: /vclfcn-15/ 2024-02-01T10:19:27.631 DEBUG:boto:Data: 2024-02-01T10:19:27.631 DEBUG:boto:Headers: {} 2024-02-01T10:19:27.631 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:19:27.631 DEBUG:boto:Port: 8001 2024-02-01T10:19:27.631 DEBUG:boto:Params: {} 2024-02-01T10:19:27.631 DEBUG:boto:Token: None 2024-02-01T10:19:27.631 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:19:27 GMT /vclfcn-15/ 2024-02-01T10:19:27.631 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lGiehrsdKiNr/d1EhDpIWUZHy+c= 2024-02-01T10:19:27.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:19:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lGiehrsdKiNr/d1EhDpIWUZHy+c=', 'Content-Length': '0'} 2024-02-01T10:19:27.635 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d1a0b136d325c669-0065bb702f-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:19:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:19:27.635 DEBUG:boto:path=/vclfcn-15/myobj 2024-02-01T10:19:27.635 DEBUG:boto:auth_path=/vclfcn-15/myobj 2024-02-01T10:19:27.635 DEBUG:boto:Method: HEAD 2024-02-01T10:19:27.635 DEBUG:boto:Path: /vclfcn-15/myobj 2024-02-01T10:19:27.635 DEBUG:boto:Data: 2024-02-01T10:19:27.635 DEBUG:boto:Headers: {} 2024-02-01T10:19:27.635 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:19:27.636 DEBUG:boto:Port: 8001 2024-02-01T10:19:27.636 DEBUG:boto:Params: {} 2024-02-01T10:19:27.636 DEBUG:boto:Token: None 2024-02-01T10:19:27.636 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:19:27 GMT /vclfcn-15/myobj 2024-02-01T10:19:27.636 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cLpECDm6g0GaVopo6rKu5lTXXyM= 2024-02-01T10:19:27.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:19:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cLpECDm6g0GaVopo6rKu5lTXXyM=', 'Content-Length': '0'} 2024-02-01T10:19:27.638 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:18:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fc46825801188a57-0065bb702f-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:19:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:19:27.639 DEBUG:boto:path=/vclfcn-15/myobj 2024-02-01T10:19:27.639 DEBUG:boto:auth_path=/vclfcn-15/myobj 2024-02-01T10:19:27.639 DEBUG:boto:Method: DELETE 2024-02-01T10:19:27.639 DEBUG:boto:Path: /vclfcn-15/myobj 2024-02-01T10:19:27.639 DEBUG:boto:Data: 2024-02-01T10:19:27.639 DEBUG:boto:Headers: {} 2024-02-01T10:19:27.639 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:19:27.639 DEBUG:boto:Port: 8001 2024-02-01T10:19:27.639 DEBUG:boto:Params: {} 2024-02-01T10:19:27.640 DEBUG:boto:Token: None 2024-02-01T10:19:27.640 DEBUG:boto:StringToSign: DELETE Thu, 01 Feb 2024 10:19:27 GMT /vclfcn-15/myobj 2024-02-01T10:19:27.640 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:6+lSqAFFXxm4w0wSQ6VVfqhEiJs= 2024-02-01T10:19:27.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:19:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:6+lSqAFFXxm4w0wSQ6VVfqhEiJs=', 'Content-Length': '0'} 2024-02-01T10:19:27.649 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000db899907df18c5fd-0065bb702f-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:19:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:19:27.649 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:19:27.649 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:19:27.650 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:19:27.701 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:19:27.701 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:19:27.761 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:19:27.762+0000 7ff826330980 1 waiting for incremental sync to catch up: 2024-02-01T10:19:27.761 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, 00000000001.12.6, , , , , , , , , ] 2024-02-01T10:19:27.761 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, 00000000002.30.6, , , , , , , , , ] 2024-02-01T10:19:32.764 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:19:32.766+0000 7ff826330980 1 waiting for incremental sync to catch up: 2024-02-01T10:19:32.764 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, 00000000001.12.6, , , , , , , , , ] 2024-02-01T10:19:32.764 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, 00000000002.30.6, , , , , , , , , ] 2024-02-01T10:19:37.766 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:19:37.766+0000 7ff826330980 1 waiting for incremental sync to catch up: 2024-02-01T10:19:37.766 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, 00000000001.12.6, , , , , , , , , ] 2024-02-01T10:19:37.766 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, 00000000002.30.6, , , , , , , , , ] 2024-02-01T10:19:42.769 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:19:42.770+0000 7ff826330980 1 waiting for incremental sync to catch up: 2024-02-01T10:19:42.769 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, 00000000001.12.6, , , , , , , , , ] 2024-02-01T10:19:42.769 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, 00000000002.30.6, , , , , , , , , ] 2024-02-01T10:19:47.771 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:19:47.774+0000 7ff826330980 1 bucket sync caught up with source: 2024-02-01T10:19:47.771 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, 00000000002.30.6, , , , , , , , , ] 2024-02-01T10:19:47.771 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, 00000000002.30.6, , , , , , , , , ] 2024-02-01T10:19:47.771 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:19:47.774+0000 7ff826330980 0 bucket checkpoint complete 2024-02-01T10:19:47.810 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-15 zones={test-zone1, test-zone3} 2024-02-01T10:19:47.810 DEBUG:boto:path=/vclfcn-15/ 2024-02-01T10:19:47.810 DEBUG:boto:auth_path=/vclfcn-15/ 2024-02-01T10:19:47.810 DEBUG:boto:Method: HEAD 2024-02-01T10:19:47.810 DEBUG:boto:Path: /vclfcn-15/ 2024-02-01T10:19:47.810 DEBUG:boto:Data: 2024-02-01T10:19:47.810 DEBUG:boto:Headers: {} 2024-02-01T10:19:47.810 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:19:47.811 DEBUG:boto:Port: 8000 2024-02-01T10:19:47.811 DEBUG:boto:Params: {} 2024-02-01T10:19:47.811 DEBUG:boto:Token: None 2024-02-01T10:19:47.811 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:19:47 GMT /vclfcn-15/ 2024-02-01T10:19:47.811 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZYZ/8Ho773oKfp5jDOQ7A3Sm5MA= 2024-02-01T10:19:47.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:19:47 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZYZ/8Ho773oKfp5jDOQ7A3Sm5MA=', 'Content-Length': '0'} 2024-02-01T10:19:47.815 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a1e1feccf0b8ffd9-0065bb7043-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:19:47 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:19:47.815 DEBUG:boto:path=/vclfcn-15/ 2024-02-01T10:19:47.816 DEBUG:boto:auth_path=/vclfcn-15/ 2024-02-01T10:19:47.816 DEBUG:boto:Method: HEAD 2024-02-01T10:19:47.816 DEBUG:boto:Path: /vclfcn-15/ 2024-02-01T10:19:47.816 DEBUG:boto:Data: 2024-02-01T10:19:47.816 DEBUG:boto:Headers: {} 2024-02-01T10:19:47.816 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:19:47.816 DEBUG:boto:Port: 8001 2024-02-01T10:19:47.816 DEBUG:boto:Params: {} 2024-02-01T10:19:47.816 DEBUG:boto:Token: None 2024-02-01T10:19:47.816 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:19:47 GMT /vclfcn-15/ 2024-02-01T10:19:47.817 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZYZ/8Ho773oKfp5jDOQ7A3Sm5MA= 2024-02-01T10:19:47.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:19:47 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZYZ/8Ho773oKfp5jDOQ7A3Sm5MA=', 'Content-Length': '0'} 2024-02-01T10:19:47.820 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000020e5b7430f2ee768-0065bb7043-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:19:47 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:19:47.820 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:19:47.820 DEBUG:boto:path=/vclfcn-15/ 2024-02-01T10:19:47.821 DEBUG:boto:auth_path=/vclfcn-15/ 2024-02-01T10:19:47.821 DEBUG:boto:path=/vclfcn-15/?versions&max-keys=999 2024-02-01T10:19:47.821 DEBUG:boto:auth_path=/vclfcn-15/?versions&max-keys=999 2024-02-01T10:19:47.821 DEBUG:boto:Method: GET 2024-02-01T10:19:47.821 DEBUG:boto:Path: /vclfcn-15/?versions&max-keys=999 2024-02-01T10:19:47.821 DEBUG:boto:Data: 2024-02-01T10:19:47.821 DEBUG:boto:Headers: {} 2024-02-01T10:19:47.821 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:19:47.821 DEBUG:boto:Port: 8000 2024-02-01T10:19:47.821 DEBUG:boto:Params: {} 2024-02-01T10:19:47.821 DEBUG:boto:Token: None 2024-02-01T10:19:47.822 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:19:47 GMT /vclfcn-15/?versions 2024-02-01T10:19:47.822 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:EyyyHcYyUGSJUQFR4bnwEKUgWNo= 2024-02-01T10:19:47.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:19:47 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:EyyyHcYyUGSJUQFR4bnwEKUgWNo=', 'Content-Length': '0'} 2024-02-01T10:19:47.825 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000452eeef183d7f3de-0065bb7043-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:19:47 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:19:47.825 DEBUG:boto:b'vclfcn-15999false' 2024-02-01T10:19:47.826 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:19:47.826 DEBUG:boto:path=/vclfcn-15/ 2024-02-01T10:19:47.826 DEBUG:boto:auth_path=/vclfcn-15/ 2024-02-01T10:19:47.826 DEBUG:boto:path=/vclfcn-15/?versions&max-keys=999 2024-02-01T10:19:47.826 DEBUG:boto:auth_path=/vclfcn-15/?versions&max-keys=999 2024-02-01T10:19:47.826 DEBUG:boto:Method: GET 2024-02-01T10:19:47.826 DEBUG:boto:Path: /vclfcn-15/?versions&max-keys=999 2024-02-01T10:19:47.826 DEBUG:boto:Data: 2024-02-01T10:19:47.826 DEBUG:boto:Headers: {} 2024-02-01T10:19:47.826 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:19:47.827 DEBUG:boto:Port: 8001 2024-02-01T10:19:47.827 DEBUG:boto:Params: {} 2024-02-01T10:19:47.827 DEBUG:boto:Token: None 2024-02-01T10:19:47.827 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:19:47 GMT /vclfcn-15/?versions 2024-02-01T10:19:47.827 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:EyyyHcYyUGSJUQFR4bnwEKUgWNo= 2024-02-01T10:19:47.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:19:47 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:EyyyHcYyUGSJUQFR4bnwEKUgWNo=', 'Content-Length': '0'} 2024-02-01T10:19:47.830 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000061fd08714e77f06a-0065bb7043-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:19:47 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:19:47.830 DEBUG:boto:b'vclfcn-15999false' 2024-02-01T10:19:47.830 DEBUG:boto:path=/vclfcn-15/ 2024-02-01T10:19:47.831 DEBUG:boto:auth_path=/vclfcn-15/ 2024-02-01T10:19:47.831 DEBUG:boto:path=/vclfcn-15/?versions&max-keys=999 2024-02-01T10:19:47.831 DEBUG:boto:auth_path=/vclfcn-15/?versions&max-keys=999 2024-02-01T10:19:47.831 DEBUG:boto:Method: GET 2024-02-01T10:19:47.831 DEBUG:boto:Path: /vclfcn-15/?versions&max-keys=999 2024-02-01T10:19:47.831 DEBUG:boto:Data: 2024-02-01T10:19:47.831 DEBUG:boto:Headers: {} 2024-02-01T10:19:47.831 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:19:47.831 DEBUG:boto:Port: 8000 2024-02-01T10:19:47.831 DEBUG:boto:Params: {} 2024-02-01T10:19:47.831 DEBUG:boto:Token: None 2024-02-01T10:19:47.832 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:19:47 GMT /vclfcn-15/?versions 2024-02-01T10:19:47.832 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:EyyyHcYyUGSJUQFR4bnwEKUgWNo= 2024-02-01T10:19:47.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:19:47 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:EyyyHcYyUGSJUQFR4bnwEKUgWNo=', 'Content-Length': '0'} 2024-02-01T10:19:47.834 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000151696e427bea12f-0065bb7043-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:19:47 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:19:47.835 DEBUG:boto:b'vclfcn-15999false' 2024-02-01T10:19:47.835 DEBUG:boto:path=/vclfcn-15/ 2024-02-01T10:19:47.835 DEBUG:boto:auth_path=/vclfcn-15/ 2024-02-01T10:19:47.835 DEBUG:boto:path=/vclfcn-15/?versions&max-keys=999 2024-02-01T10:19:47.835 DEBUG:boto:auth_path=/vclfcn-15/?versions&max-keys=999 2024-02-01T10:19:47.835 DEBUG:boto:Method: GET 2024-02-01T10:19:47.835 DEBUG:boto:Path: /vclfcn-15/?versions&max-keys=999 2024-02-01T10:19:47.836 DEBUG:boto:Data: 2024-02-01T10:19:47.836 DEBUG:boto:Headers: {} 2024-02-01T10:19:47.836 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:19:47.836 DEBUG:boto:Port: 8001 2024-02-01T10:19:47.836 DEBUG:boto:Params: {} 2024-02-01T10:19:47.836 DEBUG:boto:Token: None 2024-02-01T10:19:47.836 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:19:47 GMT /vclfcn-15/?versions 2024-02-01T10:19:47.836 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:EyyyHcYyUGSJUQFR4bnwEKUgWNo= 2024-02-01T10:19:47.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:19:47 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:EyyyHcYyUGSJUQFR4bnwEKUgWNo=', 'Content-Length': '0'} 2024-02-01T10:19:47.839 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6288e7c68a6b41e-0065bb7043-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:19:47 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:19:47.880 DEBUG:boto:b'vclfcn-15999false' 2024-02-01T10:19:47.880 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-15 zones={test-zone1, test-zone3} 2024-02-01T10:19:47.881 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:19:47.881 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:19:47.881 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket vclfcn-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-02-01T10:19:47.932 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:19:47.933 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:19:47.995 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:19:47.994+0000 7f575c6a2980 1 bucket sync caught up with source: 2024-02-01T10:19:47.995 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, 00000000002.30.6, , , , , , , , , ] 2024-02-01T10:19:47.995 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, 00000000002.30.6, , , , , , , , , ] 2024-02-01T10:19:47.995 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:19:47.994+0000 7f575c6a2980 0 bucket checkpoint complete 2024-02-01T10:19:48.018 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-15 zones={test-zone2, test-zone3} 2024-02-01T10:19:48.018 DEBUG:boto:path=/vclfcn-15/ 2024-02-01T10:19:48.018 DEBUG:boto:auth_path=/vclfcn-15/ 2024-02-01T10:19:48.018 DEBUG:boto:Method: HEAD 2024-02-01T10:19:48.018 DEBUG:boto:Path: /vclfcn-15/ 2024-02-01T10:19:48.018 DEBUG:boto:Data: 2024-02-01T10:19:48.018 DEBUG:boto:Headers: {} 2024-02-01T10:19:48.018 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:19:48.018 DEBUG:boto:Port: 8000 2024-02-01T10:19:48.019 DEBUG:boto:Params: {} 2024-02-01T10:19:48.019 DEBUG:boto:Token: None 2024-02-01T10:19:48.019 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:19:48 GMT /vclfcn-15/ 2024-02-01T10:19:48.019 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1pAXBOGNQ8ms0599QTkmHmiwJTE= 2024-02-01T10:19:48.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:19:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1pAXBOGNQ8ms0599QTkmHmiwJTE=', 'Content-Length': '0'} 2024-02-01T10:19:48.021 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000083d5e42c8f990d22-0065bb7044-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:19:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:19:48.022 DEBUG:boto:path=/vclfcn-15/ 2024-02-01T10:19:48.022 DEBUG:boto:auth_path=/vclfcn-15/ 2024-02-01T10:19:48.022 DEBUG:boto:Method: HEAD 2024-02-01T10:19:48.022 DEBUG:boto:Path: /vclfcn-15/ 2024-02-01T10:19:48.022 DEBUG:boto:Data: 2024-02-01T10:19:48.022 DEBUG:boto:Headers: {} 2024-02-01T10:19:48.022 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:19:48.022 DEBUG:boto:Port: 8001 2024-02-01T10:19:48.022 DEBUG:boto:Params: {} 2024-02-01T10:19:48.023 DEBUG:boto:Token: None 2024-02-01T10:19:48.023 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:19:48 GMT /vclfcn-15/ 2024-02-01T10:19:48.023 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1pAXBOGNQ8ms0599QTkmHmiwJTE= 2024-02-01T10:19:48.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:19:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1pAXBOGNQ8ms0599QTkmHmiwJTE=', 'Content-Length': '0'} 2024-02-01T10:19:48.027 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a307a2db888e1a5d-0065bb7044-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:19:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:19:48.027 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:19:48.027 DEBUG:boto:path=/vclfcn-15/ 2024-02-01T10:19:48.027 DEBUG:boto:auth_path=/vclfcn-15/ 2024-02-01T10:19:48.028 DEBUG:boto:path=/vclfcn-15/?versions&max-keys=999 2024-02-01T10:19:48.028 DEBUG:boto:auth_path=/vclfcn-15/?versions&max-keys=999 2024-02-01T10:19:48.028 DEBUG:boto:Method: GET 2024-02-01T10:19:48.028 DEBUG:boto:Path: /vclfcn-15/?versions&max-keys=999 2024-02-01T10:19:48.028 DEBUG:boto:Data: 2024-02-01T10:19:48.028 DEBUG:boto:Headers: {} 2024-02-01T10:19:48.028 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:19:48.028 DEBUG:boto:Port: 8000 2024-02-01T10:19:48.028 DEBUG:boto:Params: {} 2024-02-01T10:19:48.028 DEBUG:boto:Token: None 2024-02-01T10:19:48.029 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:19:48 GMT /vclfcn-15/?versions 2024-02-01T10:19:48.029 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8BTWFqrQwXndnHgZnHtotuqjkAI= 2024-02-01T10:19:48.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:19:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8BTWFqrQwXndnHgZnHtotuqjkAI=', 'Content-Length': '0'} 2024-02-01T10:19:48.031 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007761642858696fe7-0065bb7044-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:19:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:19:48.076 DEBUG:boto:b'vclfcn-15999false' 2024-02-01T10:19:48.077 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:19:48.077 DEBUG:boto:path=/vclfcn-15/ 2024-02-01T10:19:48.077 DEBUG:boto:auth_path=/vclfcn-15/ 2024-02-01T10:19:48.077 DEBUG:boto:path=/vclfcn-15/?versions&max-keys=999 2024-02-01T10:19:48.078 DEBUG:boto:auth_path=/vclfcn-15/?versions&max-keys=999 2024-02-01T10:19:48.078 DEBUG:boto:Method: GET 2024-02-01T10:19:48.078 DEBUG:boto:Path: /vclfcn-15/?versions&max-keys=999 2024-02-01T10:19:48.078 DEBUG:boto:Data: 2024-02-01T10:19:48.078 DEBUG:boto:Headers: {} 2024-02-01T10:19:48.079 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:19:48.079 DEBUG:boto:Port: 8001 2024-02-01T10:19:48.079 DEBUG:boto:Params: {} 2024-02-01T10:19:48.080 DEBUG:boto:Token: None 2024-02-01T10:19:48.080 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:19:48 GMT /vclfcn-15/?versions 2024-02-01T10:19:48.080 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8BTWFqrQwXndnHgZnHtotuqjkAI= 2024-02-01T10:19:48.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:19:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8BTWFqrQwXndnHgZnHtotuqjkAI=', 'Content-Length': '0'} 2024-02-01T10:19:48.086 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c98108b046c1f20d-0065bb7044-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:19:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:19:48.086 DEBUG:boto:b'vclfcn-15999false' 2024-02-01T10:19:48.086 DEBUG:boto:path=/vclfcn-15/ 2024-02-01T10:19:48.086 DEBUG:boto:auth_path=/vclfcn-15/ 2024-02-01T10:19:48.086 DEBUG:boto:path=/vclfcn-15/?versions&max-keys=999 2024-02-01T10:19:48.087 DEBUG:boto:auth_path=/vclfcn-15/?versions&max-keys=999 2024-02-01T10:19:48.087 DEBUG:boto:Method: GET 2024-02-01T10:19:48.087 DEBUG:boto:Path: /vclfcn-15/?versions&max-keys=999 2024-02-01T10:19:48.087 DEBUG:boto:Data: 2024-02-01T10:19:48.087 DEBUG:boto:Headers: {} 2024-02-01T10:19:48.092 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:19:48.092 DEBUG:boto:Port: 8000 2024-02-01T10:19:48.092 DEBUG:boto:Params: {} 2024-02-01T10:19:48.093 DEBUG:boto:Token: None 2024-02-01T10:19:48.093 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:19:48 GMT /vclfcn-15/?versions 2024-02-01T10:19:48.093 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8BTWFqrQwXndnHgZnHtotuqjkAI= 2024-02-01T10:19:48.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:19:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8BTWFqrQwXndnHgZnHtotuqjkAI=', 'Content-Length': '0'} 2024-02-01T10:19:48.097 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c606c6bcb5b5c72-0065bb7044-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:19:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:19:48.097 DEBUG:boto:b'vclfcn-15999false' 2024-02-01T10:19:48.097 DEBUG:boto:path=/vclfcn-15/ 2024-02-01T10:19:48.097 DEBUG:boto:auth_path=/vclfcn-15/ 2024-02-01T10:19:48.097 DEBUG:boto:path=/vclfcn-15/?versions&max-keys=999 2024-02-01T10:19:48.097 DEBUG:boto:auth_path=/vclfcn-15/?versions&max-keys=999 2024-02-01T10:19:48.098 DEBUG:boto:Method: GET 2024-02-01T10:19:48.098 DEBUG:boto:Path: /vclfcn-15/?versions&max-keys=999 2024-02-01T10:19:48.098 DEBUG:boto:Data: 2024-02-01T10:19:48.098 DEBUG:boto:Headers: {} 2024-02-01T10:19:48.098 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:19:48.098 DEBUG:boto:Port: 8001 2024-02-01T10:19:48.098 DEBUG:boto:Params: {} 2024-02-01T10:19:48.098 DEBUG:boto:Token: None 2024-02-01T10:19:48.098 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:19:48 GMT /vclfcn-15/?versions 2024-02-01T10:19:48.098 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8BTWFqrQwXndnHgZnHtotuqjkAI= 2024-02-01T10:19:48.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:19:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8BTWFqrQwXndnHgZnHtotuqjkAI=', 'Content-Length': '0'} 2024-02-01T10:19:48.103 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003050616c19172212-0065bb7044-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:19:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:19:48.104 DEBUG:boto:b'vclfcn-15999false' 2024-02-01T10:19:48.104 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-15 zones={test-zone2, test-zone3} 2024-02-01T10:19:48.105 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_object_delete ... ok 2024-02-01T10:19:48.106 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=vclfcn-16 2024-02-01T10:19:48.107 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:19:48.107 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:19:48.107 DEBUG:boto:Method: PUT 2024-02-01T10:19:48.107 DEBUG:boto:Path: /vclfcn-16/ 2024-02-01T10:19:48.107 DEBUG:boto:Data: 2024-02-01T10:19:48.107 DEBUG:boto:Headers: {} 2024-02-01T10:19:48.107 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:19:48.108 DEBUG:boto:Port: 8000 2024-02-01T10:19:48.108 DEBUG:boto:Params: {} 2024-02-01T10:19:48.108 DEBUG:boto:Token: None 2024-02-01T10:19:48.108 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:19:48 GMT /vclfcn-16/ 2024-02-01T10:19:48.108 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sEALcz2cY3e78I7BswwEzv3Q9Ns= 2024-02-01T10:19:48.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:19:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sEALcz2cY3e78I7BswwEzv3Q9Ns=', 'Content-Length': '0'} 2024-02-01T10:19:48.144 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006f71ee7903f89488-0065bb7044-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:19:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:19:48.144 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=vclfcn-17 2024-02-01T10:19:48.145 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:19:48.145 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:19:48.145 DEBUG:boto:Method: PUT 2024-02-01T10:19:48.145 DEBUG:boto:Path: /vclfcn-17/ 2024-02-01T10:19:48.145 DEBUG:boto:Data: 2024-02-01T10:19:48.145 DEBUG:boto:Headers: {} 2024-02-01T10:19:48.145 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:19:48.145 DEBUG:boto:Port: 8000 2024-02-01T10:19:48.145 DEBUG:boto:Params: {} 2024-02-01T10:19:48.146 DEBUG:boto:Token: None 2024-02-01T10:19:48.146 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:19:48 GMT /vclfcn-17/ 2024-02-01T10:19:48.146 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:oqBrSZZPPb1sRpdR0r4tRzXFyrg= 2024-02-01T10:19:48.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:19:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:oqBrSZZPPb1sRpdR0r4tRzXFyrg=', 'Content-Length': '0'} 2024-02-01T10:19:48.203 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fe229d1bac6a94e8-0065bb7044-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:19:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:19:48.204 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=vclfcn-18 2024-02-01T10:19:48.204 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:19:48.204 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:19:48.204 DEBUG:boto:Method: PUT 2024-02-01T10:19:48.204 DEBUG:boto:Path: /vclfcn-18/ 2024-02-01T10:19:48.204 DEBUG:boto:Data: 2024-02-01T10:19:48.204 DEBUG:boto:Headers: {} 2024-02-01T10:19:48.204 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:19:48.205 DEBUG:boto:Port: 8001 2024-02-01T10:19:48.205 DEBUG:boto:Params: {} 2024-02-01T10:19:48.205 DEBUG:boto:Token: None 2024-02-01T10:19:48.205 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:19:48 GMT /vclfcn-18/ 2024-02-01T10:19:48.205 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:pCQolkhqmY13lba12FVtKoMXiX8= 2024-02-01T10:19:48.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:19:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:pCQolkhqmY13lba12FVtKoMXiX8=', 'Content-Length': '0'} 2024-02-01T10:19:48.256 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000029f4b5a3d9b01bbd-0065bb7044-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:19:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:19:48.257 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:19:48.257 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:19:48.257 DEBUG:boto:path=/vclfcn-16/?versioning 2024-02-01T10:19:48.257 DEBUG:boto:auth_path=/vclfcn-16/?versioning 2024-02-01T10:19:48.257 DEBUG:boto:Method: PUT 2024-02-01T10:19:48.257 DEBUG:boto:Path: /vclfcn-16/?versioning 2024-02-01T10:19:48.257 DEBUG:boto:Data: Enabled Disabled 2024-02-01T10:19:48.257 DEBUG:boto:Headers: {} 2024-02-01T10:19:48.257 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:19:48.257 DEBUG:boto:Port: 8000 2024-02-01T10:19:48.257 DEBUG:boto:Params: {} 2024-02-01T10:19:48.258 DEBUG:boto:Token: None 2024-02-01T10:19:48.258 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:19:48 GMT /vclfcn-16/?versioning 2024-02-01T10:19:48.258 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:MDsJ3j3m61K0akH7Fb4KQN72bU8= 2024-02-01T10:19:48.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:19:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:MDsJ3j3m61K0akH7Fb4KQN72bU8=', 'Content-Length': '228'} 2024-02-01T10:19:48.265 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e8f57d607622607c-0065bb7044-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:19:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:19:48.266 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:19:48.266 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:19:48.266 DEBUG:boto:path=/vclfcn-17/?versioning 2024-02-01T10:19:48.266 DEBUG:boto:auth_path=/vclfcn-17/?versioning 2024-02-01T10:19:48.266 DEBUG:boto:Method: PUT 2024-02-01T10:19:48.266 DEBUG:boto:Path: /vclfcn-17/?versioning 2024-02-01T10:19:48.266 DEBUG:boto:Data: Enabled Disabled 2024-02-01T10:19:48.267 DEBUG:boto:Headers: {} 2024-02-01T10:19:48.267 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:19:48.267 DEBUG:boto:Port: 8000 2024-02-01T10:19:48.267 DEBUG:boto:Params: {} 2024-02-01T10:19:48.267 DEBUG:boto:Token: None 2024-02-01T10:19:48.267 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:19:48 GMT /vclfcn-17/?versioning 2024-02-01T10:19:48.267 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fo3jZd1lRV1v1xaVGu5JG1JNzqw= 2024-02-01T10:19:48.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:19:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fo3jZd1lRV1v1xaVGu5JG1JNzqw=', 'Content-Length': '228'} 2024-02-01T10:19:48.286 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a9b3e6bf9d176d1d-0065bb7044-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:19:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:19:48.287 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:19:48.287 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:19:48.287 DEBUG:boto:path=/vclfcn-18/?versioning 2024-02-01T10:19:48.287 DEBUG:boto:auth_path=/vclfcn-18/?versioning 2024-02-01T10:19:48.287 DEBUG:boto:Method: PUT 2024-02-01T10:19:48.287 DEBUG:boto:Path: /vclfcn-18/?versioning 2024-02-01T10:19:48.287 DEBUG:boto:Data: Enabled Disabled 2024-02-01T10:19:48.287 DEBUG:boto:Headers: {} 2024-02-01T10:19:48.287 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:19:48.288 DEBUG:boto:Port: 8001 2024-02-01T10:19:48.288 DEBUG:boto:Params: {} 2024-02-01T10:19:48.288 DEBUG:boto:Token: None 2024-02-01T10:19:48.288 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:19:48 GMT /vclfcn-18/?versioning 2024-02-01T10:19:48.288 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BdNNsAW00juXZI4Gc/Dhi56yvJQ= 2024-02-01T10:19:48.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:19:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BdNNsAW00juXZI4Gc/Dhi56yvJQ=', 'Content-Length': '228'} 2024-02-01T10:19:48.300 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fffc4ba999a9b5da-0065bb7044-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:19:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:19:48.301 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:19:48.301 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:19:48.301 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:19:48.339 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:19:48.339 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:19:48.393 INFO:teuthology.orchestra.run.smithi148.stderr:No --period given, using current period=5dc4efbd-eccd-47a8-9471-ae4a209ec978 2024-02-01T10:19:48.418 INFO:teuthology.orchestra.run.smithi148.stdout:[{"marker":"1_1706782453.132619_30.1","last_update":"2024-02-01T10:14:13.132619Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1706782788.299861_85.1","last_update":"2024-02-01T10:19:48.299861Z"},{"marker":"1_1706782595.670952_121.1","last_update":"2024-02-01T10:16:35.670952Z"}] 2024-02-01T10:19:48.418 DEBUG:rgw_multi.tests:master meta markers={0: '1_1706782453.132619_30.1', 1: '', 2: '1_1706782788.299861_85.1', 3: '1_1706782595.670952_121.1'} 2024-02-01T10:19:48.419 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-02-01T10:19:48.419 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:19:48.419 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:19:48.419 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:19:48.467 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:19:48.467 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:19:48.530 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:19:48.530+0000 7fefe8c60980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:19:48.531 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:19:48.530+0000 7fefe8c60980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:19:48.554 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706782788.225397_76.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:19:48.225397Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:19:48Z"} 2024-02-01T10:19:48.554 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706782788.225397_76.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:19:48.225397Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:19:48Z"} 2024-02-01T10:19:48.554 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706782788.225397_76.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:19:48.225397Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:19:48.555 DEBUG:rgw_multi.tests:log_status={0: '1_1706782453.132619_30.1', 1: '', 2: '1_1706782788.299861_85.1', 3: '1_1706782595.670952_121.1'} 2024-02-01T10:19:48.555 DEBUG:rgw_multi.tests:sync_status={0: '1_1706782453.132619_30.1', 1: '', 2: '1_1706782788.225397_76.1', 3: '1_1706782595.670952_121.1'} 2024-02-01T10:19:48.555 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1706782788.299861_85.1 target=1_1706782788.225397_76.1 2024-02-01T10:19:53.556 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:19:53.556 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:19:53.556 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:19:53.599 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:19:53.599 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:19:53.656 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:19:53.654+0000 7fc999d4b980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:19:53.657 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:19:53.658+0000 7fc999d4b980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:19:53.682 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706782788.299861_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:19:48.299861Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:19:53Z"} 2024-02-01T10:19:53.682 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706782788.299861_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:19:48.299861Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:19:53Z"} 2024-02-01T10:19:53.682 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706782788.299861_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:19:48.299861Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:19:53.683 DEBUG:rgw_multi.tests:log_status={0: '1_1706782453.132619_30.1', 1: '', 2: '1_1706782788.299861_85.1', 3: '1_1706782595.670952_121.1'} 2024-02-01T10:19:53.683 DEBUG:rgw_multi.tests:sync_status={0: '1_1706782453.132619_30.1', 1: '', 2: '1_1706782788.299861_85.1', 3: '1_1706782595.670952_121.1'} 2024-02-01T10:19:53.683 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-02-01T10:19:53.683 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-02-01T10:19:53.683 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:19:53.683 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:19:53.683 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:19:53.732 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:19:53.732 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:19:53.792 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:19:53.794+0000 7fe07be35980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:19:53.793 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:19:53.794+0000 7fe07be35980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:19:53.816 INFO:teuthology.orchestra.run.smithi148.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706782788.125323_68.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:19:48.125323Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:19:53Z"} 2024-02-01T10:19:53.816 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706782788.125323_68.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:19:48.125323Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:19:53Z"} 2024-02-01T10:19:53.816 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706782788.125323_68.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:19:48.125323Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:19:53.817 DEBUG:rgw_multi.tests:log_status={0: '1_1706782453.132619_30.1', 1: '', 2: '1_1706782788.299861_85.1', 3: '1_1706782595.670952_121.1'} 2024-02-01T10:19:53.817 DEBUG:rgw_multi.tests:sync_status={0: '1_1706782453.132619_30.1', 1: '', 2: '1_1706782788.125323_68.1', 3: '1_1706782595.670952_121.1'} 2024-02-01T10:19:53.817 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1706782788.299861_85.1 target=1_1706782788.125323_68.1 2024-02-01T10:19:58.818 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:19:58.818 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:19:58.818 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:19:58.861 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:19:58.861 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:19:58.917 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:19:58.918+0000 7f5b7040d980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:19:58.919 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:19:58.922+0000 7f5b7040d980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:19:58.942 INFO:teuthology.orchestra.run.smithi148.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706782788.299861_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:19:48.299861Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:19:58Z"} 2024-02-01T10:19:58.943 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706782788.299861_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:19:48.299861Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:19:58Z"} 2024-02-01T10:19:58.943 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706782788.299861_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:19:48.299861Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:19:58.943 DEBUG:rgw_multi.tests:log_status={0: '1_1706782453.132619_30.1', 1: '', 2: '1_1706782788.299861_85.1', 3: '1_1706782595.670952_121.1'} 2024-02-01T10:19:58.943 DEBUG:rgw_multi.tests:sync_status={0: '1_1706782453.132619_30.1', 1: '', 2: '1_1706782788.299861_85.1', 3: '1_1706782595.670952_121.1'} 2024-02-01T10:19:58.943 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-02-01T10:19:58.943 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:19:58.944 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:19:58.944 DEBUG:boto:Method: HEAD 2024-02-01T10:19:58.944 DEBUG:boto:Path: /vclfcn-16/ 2024-02-01T10:19:58.944 DEBUG:boto:Data: 2024-02-01T10:19:58.944 DEBUG:boto:Headers: {} 2024-02-01T10:19:58.944 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:19:58.944 DEBUG:boto:Port: 8000 2024-02-01T10:19:58.944 DEBUG:boto:Params: {} 2024-02-01T10:19:58.947 DEBUG:boto:Token: None 2024-02-01T10:19:58.947 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:19:58 GMT /vclfcn-16/ 2024-02-01T10:19:58.947 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:EcfdisU895mmGEhiq0qQ8VJsgnA= 2024-02-01T10:19:58.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:19:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:EcfdisU895mmGEhiq0qQ8VJsgnA=', 'Content-Length': '0'} 2024-02-01T10:19:58.950 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000690e7d14c833596d-0065bb704e-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:19:58.950 DEBUG:boto:path=/vclfcn-16/dummy 2024-02-01T10:19:58.951 DEBUG:boto:auth_path=/vclfcn-16/dummy 2024-02-01T10:19:58.951 DEBUG:boto:Method: PUT 2024-02-01T10:19:58.951 DEBUG:boto:Path: /vclfcn-16/dummy 2024-02-01T10:19:58.951 DEBUG:boto:Data: 2024-02-01T10:19:58.951 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-02-01T10:19:58.951 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:19:58.951 DEBUG:boto:Port: 8000 2024-02-01T10:19:58.951 DEBUG:boto:Params: {} 2024-02-01T10:19:58.952 DEBUG:boto:Token: None 2024-02-01T10:19:58.952 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Thu, 01 Feb 2024 10:19:58 GMT /vclfcn-16/dummy 2024-02-01T10:19:58.952 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Wej3kp9ZkFyW8rww/pOT81BHmjs= 2024-02-01T10:19:58.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:19:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Wej3kp9ZkFyW8rww/pOT81BHmjs='} 2024-02-01T10:19:58.964 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'rQRpBowIFAsvxVDn92H30shYnKoGt4n'), ('x-amz-request-id', 'tx00000755af72df8c9bdea-0065bb704e-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:19:58.964 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:19:58.964 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:19:58.964 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket vclfcn-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-02-01T10:19:59.016 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:19:59.016 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:19:59.079 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:19:59.078+0000 7fe302c46980 1 waiting to reach incremental sync.. 2024-02-01T10:20:04.080 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:20:04.078+0000 7fe302c46980 1 waiting to reach incremental sync.. 2024-02-01T10:20:09.084 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:20:09.082+0000 7fe302c46980 1 bucket sync caught up with source: 2024-02-01T10:20:09.084 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, , , , , 00000000002.17.11, , , , , ] 2024-02-01T10:20:09.084 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, , , , , 00000000002.17.11, , , , , ] 2024-02-01T10:20:09.084 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:20:09.082+0000 7fe302c46980 0 bucket checkpoint complete 2024-02-01T10:20:09.125 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:20:09.125 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:20:09.126 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:20:09.176 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:20:09.177 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:20:09.235 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:20:09.237+0000 7f68110c4980 1 waiting to reach incremental sync.. 2024-02-01T10:20:14.239 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:20:14.241+0000 7f68110c4980 1 bucket sync caught up with source: 2024-02-01T10:20:14.239 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, , , , , 00000000002.17.11, , , , , ] 2024-02-01T10:20:14.240 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, , , , , 00000000002.17.11, , , , , ] 2024-02-01T10:20:14.240 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:20:14.241+0000 7f68110c4980 0 bucket checkpoint complete 2024-02-01T10:20:14.282 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:20:14.282 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:20:14.282 DEBUG:boto:Method: HEAD 2024-02-01T10:20:14.283 DEBUG:boto:Path: /vclfcn-17/ 2024-02-01T10:20:14.283 DEBUG:boto:Data: 2024-02-01T10:20:14.283 DEBUG:boto:Headers: {} 2024-02-01T10:20:14.283 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:20:14.283 DEBUG:boto:Port: 8000 2024-02-01T10:20:14.283 DEBUG:boto:Params: {} 2024-02-01T10:20:14.283 DEBUG:boto:Token: None 2024-02-01T10:20:14.283 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:20:14 GMT /vclfcn-17/ 2024-02-01T10:20:14.284 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lDMGt8AP/+e4w1QMYzVunYEjN1s= 2024-02-01T10:20:14.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:20:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lDMGt8AP/+e4w1QMYzVunYEjN1s=', 'Content-Length': '0'} 2024-02-01T10:20:14.288 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004157d53fd5846c79-0065bb705e-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:20:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:20:14.289 DEBUG:boto:path=/vclfcn-17/dummy 2024-02-01T10:20:14.289 DEBUG:boto:auth_path=/vclfcn-17/dummy 2024-02-01T10:20:14.289 DEBUG:boto:Method: PUT 2024-02-01T10:20:14.289 DEBUG:boto:Path: /vclfcn-17/dummy 2024-02-01T10:20:14.289 DEBUG:boto:Data: 2024-02-01T10:20:14.289 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-02-01T10:20:14.289 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:20:14.289 DEBUG:boto:Port: 8000 2024-02-01T10:20:14.290 DEBUG:boto:Params: {} 2024-02-01T10:20:14.290 DEBUG:boto:Token: None 2024-02-01T10:20:14.290 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Thu, 01 Feb 2024 10:20:14 GMT /vclfcn-17/dummy 2024-02-01T10:20:14.290 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zVd9DOdTY61S1vx5vKnxjEuwD5w= 2024-02-01T10:20:14.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:20:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zVd9DOdTY61S1vx5vKnxjEuwD5w='} 2024-02-01T10:20:14.316 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'hQXZXSD45t2CSRVMJOLMBXih54iMPmJ'), ('x-amz-request-id', 'tx0000043d9254fbaddc3ab-0065bb705e-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:20:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:20:14.316 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:20:14.317 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:20:14.317 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:20:14.365 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:20:14.366 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:20:14.427 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:20:14.429+0000 7ff86c1d2980 1 waiting to reach incremental sync.. 2024-02-01T10:20:19.428 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:20:19.429+0000 7ff86c1d2980 1 waiting to reach incremental sync.. 2024-02-01T10:20:24.429 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:20:24.429+0000 7ff86c1d2980 1 waiting to reach incremental sync.. 2024-02-01T10:20:29.433 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:20:29.433+0000 7ff86c1d2980 1 bucket sync caught up with source: 2024-02-01T10:20:29.433 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, , , , , 00000000002.30.11, , , , , ] 2024-02-01T10:20:29.433 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, , , , , 00000000002.30.11, , , , , ] 2024-02-01T10:20:29.433 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:20:29.433+0000 7ff86c1d2980 0 bucket checkpoint complete 2024-02-01T10:20:29.474 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:20:29.474 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:20:29.474 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:20:29.522 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:20:29.522 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:20:29.584 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:20:29.585+0000 7f2445027980 1 waiting to reach incremental sync.. 2024-02-01T10:20:34.588 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:20:34.589+0000 7f2445027980 1 bucket sync caught up with source: 2024-02-01T10:20:34.588 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, , , , , 00000000003.32.10, , , , , ] 2024-02-01T10:20:34.588 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, , , , , 00000000003.32.10, , , , , ] 2024-02-01T10:20:34.588 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:20:34.589+0000 7f2445027980 0 bucket checkpoint complete 2024-02-01T10:20:34.630 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:20:34.631 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:20:34.631 DEBUG:boto:Method: HEAD 2024-02-01T10:20:34.631 DEBUG:boto:Path: /vclfcn-18/ 2024-02-01T10:20:34.631 DEBUG:boto:Data: 2024-02-01T10:20:34.631 DEBUG:boto:Headers: {} 2024-02-01T10:20:34.631 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:20:34.631 DEBUG:boto:Port: 8001 2024-02-01T10:20:34.631 DEBUG:boto:Params: {} 2024-02-01T10:20:34.631 DEBUG:boto:Token: None 2024-02-01T10:20:34.632 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:20:34 GMT /vclfcn-18/ 2024-02-01T10:20:34.632 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ebg3HCCEBC/8bXhH9/e7Bfyv2nI= 2024-02-01T10:20:34.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:20:34 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ebg3HCCEBC/8bXhH9/e7Bfyv2nI=', 'Content-Length': '0'} 2024-02-01T10:20:34.636 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000022d2dcfe780ac415-0065bb7072-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:20:34 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:20:34.636 DEBUG:boto:path=/vclfcn-18/dummy 2024-02-01T10:20:34.636 DEBUG:boto:auth_path=/vclfcn-18/dummy 2024-02-01T10:20:34.636 DEBUG:boto:Method: PUT 2024-02-01T10:20:34.637 DEBUG:boto:Path: /vclfcn-18/dummy 2024-02-01T10:20:34.637 DEBUG:boto:Data: 2024-02-01T10:20:34.637 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-02-01T10:20:34.637 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:20:34.637 DEBUG:boto:Port: 8001 2024-02-01T10:20:34.637 DEBUG:boto:Params: {} 2024-02-01T10:20:34.637 DEBUG:boto:Token: None 2024-02-01T10:20:34.637 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Thu, 01 Feb 2024 10:20:34 GMT /vclfcn-18/dummy 2024-02-01T10:20:34.637 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fsH+M8wdKskvrqMPOijo6WPM+Ps= 2024-02-01T10:20:34.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:20:34 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fsH+M8wdKskvrqMPOijo6WPM+Ps='} 2024-02-01T10:20:34.658 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'txgLlJWZVbWbjqLWqkOtt87dUoD4zun'), ('x-amz-request-id', 'tx00000f1b7c855eaa7b4ef-0065bb7072-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:20:34 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:20:34.658 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:20:34.659 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:20:34.659 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:20:34.707 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:20:34.707 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:20:34.770 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:20:34.773+0000 7f135afe0980 1 waiting to reach incremental sync.. 2024-02-01T10:20:39.770 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:20:39.773+0000 7f135afe0980 1 waiting to reach incremental sync.. 2024-02-01T10:20:44.771 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:20:44.773+0000 7f135afe0980 1 waiting to reach incremental sync.. 2024-02-01T10:20:49.775 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:20:49.777+0000 7f135afe0980 1 bucket sync caught up with source: 2024-02-01T10:20:49.775 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, , , , , 00000000002.68.11, , , , , ] 2024-02-01T10:20:49.775 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, , , , , 00000000002.68.11, , , , , ] 2024-02-01T10:20:49.776 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:20:49.777+0000 7f135afe0980 0 bucket checkpoint complete 2024-02-01T10:20:49.814 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:20:49.815 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:20:49.815 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket vclfcn-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-02-01T10:20:49.865 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:20:49.865 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:20:49.928 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:20:49.925+0000 7f6cc99ba980 1 bucket sync caught up with source: 2024-02-01T10:20:49.928 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, , , , , 00000000002.68.11, , , , , ] 2024-02-01T10:20:49.928 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, , , , , 00000000002.68.11, , , , , ] 2024-02-01T10:20:49.928 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:20:49.925+0000 7f6cc99ba980 0 bucket checkpoint complete 2024-02-01T10:20:49.950 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:20:49.950 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:20:49.950 DEBUG:boto:Method: HEAD 2024-02-01T10:20:49.950 DEBUG:boto:Path: /vclfcn-16/ 2024-02-01T10:20:49.951 DEBUG:boto:Data: 2024-02-01T10:20:49.951 DEBUG:boto:Headers: {} 2024-02-01T10:20:49.951 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:20:49.951 DEBUG:boto:Port: 8000 2024-02-01T10:20:49.951 DEBUG:boto:Params: {} 2024-02-01T10:20:49.951 DEBUG:boto:Token: None 2024-02-01T10:20:49.951 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:20:49 GMT /vclfcn-16/ 2024-02-01T10:20:49.951 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jghOmkLmO38SF1/rXmyJLH06X9Q= 2024-02-01T10:20:49.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:20:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jghOmkLmO38SF1/rXmyJLH06X9Q=', 'Content-Length': '0'} 2024-02-01T10:20:49.956 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000962f0edc7e3786d4-0065bb7081-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:20:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:20:49.956 DEBUG:boto:path=/vclfcn-16/obj-test-zone1 2024-02-01T10:20:49.956 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1 2024-02-01T10:20:49.956 DEBUG:boto:Method: PUT 2024-02-01T10:20:49.957 DEBUG:boto:Path: /vclfcn-16/obj-test-zone1 2024-02-01T10:20:49.957 DEBUG:boto:Data: 2024-02-01T10:20:49.957 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-02-01T10:20:49.957 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:20:49.957 DEBUG:boto:Port: 8000 2024-02-01T10:20:49.957 DEBUG:boto:Params: {} 2024-02-01T10:20:49.957 DEBUG:boto:Token: None 2024-02-01T10:20:49.957 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 01 Feb 2024 10:20:49 GMT /vclfcn-16/obj-test-zone1 2024-02-01T10:20:49.958 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:AiEmuNCxt6eWcBomwg9qhA4USRU= 2024-02-01T10:20:49.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:20:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:AiEmuNCxt6eWcBomwg9qhA4USRU='} 2024-02-01T10:20:49.982 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'lS9-sMOvyR-VcD.47BkPLymGmIyfWmf'), ('x-amz-request-id', 'tx00000fbfb2865dc49b416-0065bb7081-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:20:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:20:49.982 DEBUG:rgw_multi.tests:version1 id=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:20:49.982 DEBUG:boto:path=/vclfcn-16/obj-test-zone1 2024-02-01T10:20:49.982 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1 2024-02-01T10:20:49.982 DEBUG:boto:Method: PUT 2024-02-01T10:20:49.983 DEBUG:boto:Path: /vclfcn-16/obj-test-zone1 2024-02-01T10:20:49.983 DEBUG:boto:Data: 2024-02-01T10:20:49.983 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-02-01T10:20:49.983 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:20:49.983 DEBUG:boto:Port: 8000 2024-02-01T10:20:49.983 DEBUG:boto:Params: {} 2024-02-01T10:20:49.983 DEBUG:boto:Token: None 2024-02-01T10:20:49.983 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Thu, 01 Feb 2024 10:20:49 GMT /vclfcn-16/obj-test-zone1 2024-02-01T10:20:49.983 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:SckvjIVBnEnazENebK5YESFcEus= 2024-02-01T10:20:49.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:20:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:SckvjIVBnEnazENebK5YESFcEus='} 2024-02-01T10:20:50.033 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'sOHikgSt4F-gL1hrhqAPg5EaJO1lYBO'), ('x-amz-request-id', 'tx000003d198218b6f32832-0065bb7081-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:20:50.033 DEBUG:rgw_multi.tests:version2 id=sOHikgSt4F-gL1hrhqAPg5EaJO1lYBO 2024-02-01T10:20:50.033 DEBUG:boto:path=/vclfcn-16/obj-test-zone1 2024-02-01T10:20:50.033 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1 2024-02-01T10:20:50.033 DEBUG:boto:path=/vclfcn-16/obj-test-zone1?versionId=sOHikgSt4F-gL1hrhqAPg5EaJO1lYBO 2024-02-01T10:20:50.033 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1?versionId=sOHikgSt4F-gL1hrhqAPg5EaJO1lYBO 2024-02-01T10:20:50.033 DEBUG:boto:Method: DELETE 2024-02-01T10:20:50.033 DEBUG:boto:Path: /vclfcn-16/obj-test-zone1?versionId=sOHikgSt4F-gL1hrhqAPg5EaJO1lYBO 2024-02-01T10:20:50.034 DEBUG:boto:Data: 2024-02-01T10:20:50.034 DEBUG:boto:Headers: {} 2024-02-01T10:20:50.034 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:20:50.034 DEBUG:boto:Port: 8000 2024-02-01T10:20:50.034 DEBUG:boto:Params: {} 2024-02-01T10:20:50.034 DEBUG:boto:Token: None 2024-02-01T10:20:50.034 DEBUG:boto:StringToSign: DELETE Thu, 01 Feb 2024 10:20:50 GMT /vclfcn-16/obj-test-zone1?versionId=sOHikgSt4F-gL1hrhqAPg5EaJO1lYBO 2024-02-01T10:20:50.035 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:32ZG3JFsdP+bHbP+ke8K+4GN8yE= 2024-02-01T10:20:50.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:20:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:32ZG3JFsdP+bHbP+ke8K+4GN8yE=', 'Content-Length': '0'} 2024-02-01T10:20:50.059 DEBUG:boto:Response headers: [('x-amz-version-id', 'sOHikgSt4F-gL1hrhqAPg5EaJO1lYBO'), ('x-amz-request-id', 'tx0000040dde48a4c17f705-0065bb7082-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:20:50.059 DEBUG:boto:path=/vclfcn-16/obj-test-zone1 2024-02-01T10:20:50.059 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1 2024-02-01T10:20:50.059 DEBUG:boto:Method: PUT 2024-02-01T10:20:50.060 DEBUG:boto:Path: /vclfcn-16/obj-test-zone1 2024-02-01T10:20:50.060 DEBUG:boto:Data: 2024-02-01T10:20:50.060 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-02-01T10:20:50.060 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:20:50.060 DEBUG:boto:Port: 8000 2024-02-01T10:20:50.060 DEBUG:boto:Params: {} 2024-02-01T10:20:50.060 DEBUG:boto:Token: None 2024-02-01T10:20:50.060 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Thu, 01 Feb 2024 10:20:50 GMT /vclfcn-16/obj-test-zone1 2024-02-01T10:20:50.061 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nQcgXYxVcozJ3nYDCEAOi9KV23s= 2024-02-01T10:20:50.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:20:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nQcgXYxVcozJ3nYDCEAOi9KV23s='} 2024-02-01T10:20:50.082 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'BRDjacHxvr7jhm8Vnkq5vDx.g.pVe1c'), ('x-amz-request-id', 'tx0000066bc587b13ba362d-0065bb7082-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:20:50.082 DEBUG:rgw_multi.tests:version3 id=BRDjacHxvr7jhm8Vnkq5vDx.g.pVe1c 2024-02-01T10:20:50.083 DEBUG:boto:path=/vclfcn-16/obj-test-zone1 2024-02-01T10:20:50.083 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1 2024-02-01T10:20:50.083 DEBUG:boto:path=/vclfcn-16/obj-test-zone1?versionId=BRDjacHxvr7jhm8Vnkq5vDx.g.pVe1c 2024-02-01T10:20:50.083 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1?versionId=BRDjacHxvr7jhm8Vnkq5vDx.g.pVe1c 2024-02-01T10:20:50.083 DEBUG:boto:Method: DELETE 2024-02-01T10:20:50.083 DEBUG:boto:Path: /vclfcn-16/obj-test-zone1?versionId=BRDjacHxvr7jhm8Vnkq5vDx.g.pVe1c 2024-02-01T10:20:50.083 DEBUG:boto:Data: 2024-02-01T10:20:50.083 DEBUG:boto:Headers: {} 2024-02-01T10:20:50.083 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:20:50.083 DEBUG:boto:Port: 8000 2024-02-01T10:20:50.084 DEBUG:boto:Params: {} 2024-02-01T10:20:50.084 DEBUG:boto:Token: None 2024-02-01T10:20:50.084 DEBUG:boto:StringToSign: DELETE Thu, 01 Feb 2024 10:20:50 GMT /vclfcn-16/obj-test-zone1?versionId=BRDjacHxvr7jhm8Vnkq5vDx.g.pVe1c 2024-02-01T10:20:50.084 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:L1Shs+F0dzUDSftBqE/3wsQZCQc= 2024-02-01T10:20:50.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:20:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:L1Shs+F0dzUDSftBqE/3wsQZCQc=', 'Content-Length': '0'} 2024-02-01T10:20:50.107 DEBUG:boto:Response headers: [('x-amz-version-id', 'BRDjacHxvr7jhm8Vnkq5vDx.g.pVe1c'), ('x-amz-request-id', 'tx00000accccd2cc033d2cc-0065bb7082-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:20:50.107 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:20:50.107 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:20:50.107 DEBUG:boto:Method: HEAD 2024-02-01T10:20:50.108 DEBUG:boto:Path: /vclfcn-16/ 2024-02-01T10:20:50.108 DEBUG:boto:Data: 2024-02-01T10:20:50.108 DEBUG:boto:Headers: {} 2024-02-01T10:20:50.108 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:20:50.108 DEBUG:boto:Port: 8000 2024-02-01T10:20:50.108 DEBUG:boto:Params: {} 2024-02-01T10:20:50.108 DEBUG:boto:Token: None 2024-02-01T10:20:50.108 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:20:50 GMT /vclfcn-16/ 2024-02-01T10:20:50.108 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:V/wieOXSldUtmJzlk2JMcy/pj8A= 2024-02-01T10:20:50.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:20:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:V/wieOXSldUtmJzlk2JMcy/pj8A=', 'Content-Length': '0'} 2024-02-01T10:20:50.113 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e7dfc71b09961ec8-0065bb7082-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:20:50.113 DEBUG:boto:path=/vclfcn-16/obj-test-zone2 2024-02-01T10:20:50.113 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2 2024-02-01T10:20:50.113 DEBUG:boto:Method: PUT 2024-02-01T10:20:50.113 DEBUG:boto:Path: /vclfcn-16/obj-test-zone2 2024-02-01T10:20:50.114 DEBUG:boto:Data: 2024-02-01T10:20:50.114 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-02-01T10:20:50.114 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:20:50.115 DEBUG:boto:Port: 8000 2024-02-01T10:20:50.115 DEBUG:boto:Params: {} 2024-02-01T10:20:50.115 DEBUG:boto:Token: None 2024-02-01T10:20:50.116 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 01 Feb 2024 10:20:50 GMT /vclfcn-16/obj-test-zone2 2024-02-01T10:20:50.116 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:O9NPO7gA9jRokA2de3pK/gWKUMM= 2024-02-01T10:20:50.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:20:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:O9NPO7gA9jRokA2de3pK/gWKUMM='} 2024-02-01T10:20:50.164 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'mE6urSxcrRA9wdyuToa41.w2aTia9p1'), ('x-amz-request-id', 'tx0000029101fca9eee7d73-0065bb7082-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:20:50.165 DEBUG:rgw_multi.tests:version1 id=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:20:50.165 DEBUG:boto:path=/vclfcn-16/obj-test-zone2 2024-02-01T10:20:50.165 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2 2024-02-01T10:20:50.165 DEBUG:boto:Method: PUT 2024-02-01T10:20:50.165 DEBUG:boto:Path: /vclfcn-16/obj-test-zone2 2024-02-01T10:20:50.165 DEBUG:boto:Data: 2024-02-01T10:20:50.165 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-02-01T10:20:50.165 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:20:50.165 DEBUG:boto:Port: 8000 2024-02-01T10:20:50.166 DEBUG:boto:Params: {} 2024-02-01T10:20:50.166 DEBUG:boto:Token: None 2024-02-01T10:20:50.166 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Thu, 01 Feb 2024 10:20:50 GMT /vclfcn-16/obj-test-zone2 2024-02-01T10:20:50.166 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:2LagNujZACWUNkvi2O3lY+sBKJE= 2024-02-01T10:20:50.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:20:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2LagNujZACWUNkvi2O3lY+sBKJE='} 2024-02-01T10:20:50.187 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'c490oGgvrwL3X5qVcaYbCq3o3j0FrvL'), ('x-amz-request-id', 'tx00000499503f7061bdc78-0065bb7082-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:20:50.187 DEBUG:rgw_multi.tests:version2 id=c490oGgvrwL3X5qVcaYbCq3o3j0FrvL 2024-02-01T10:20:50.187 DEBUG:boto:path=/vclfcn-16/obj-test-zone2 2024-02-01T10:20:50.188 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2 2024-02-01T10:20:50.188 DEBUG:boto:path=/vclfcn-16/obj-test-zone2?versionId=c490oGgvrwL3X5qVcaYbCq3o3j0FrvL 2024-02-01T10:20:50.188 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2?versionId=c490oGgvrwL3X5qVcaYbCq3o3j0FrvL 2024-02-01T10:20:50.188 DEBUG:boto:Method: DELETE 2024-02-01T10:20:50.188 DEBUG:boto:Path: /vclfcn-16/obj-test-zone2?versionId=c490oGgvrwL3X5qVcaYbCq3o3j0FrvL 2024-02-01T10:20:50.188 DEBUG:boto:Data: 2024-02-01T10:20:50.188 DEBUG:boto:Headers: {} 2024-02-01T10:20:50.188 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:20:50.188 DEBUG:boto:Port: 8000 2024-02-01T10:20:50.188 DEBUG:boto:Params: {} 2024-02-01T10:20:50.189 DEBUG:boto:Token: None 2024-02-01T10:20:50.189 DEBUG:boto:StringToSign: DELETE Thu, 01 Feb 2024 10:20:50 GMT /vclfcn-16/obj-test-zone2?versionId=c490oGgvrwL3X5qVcaYbCq3o3j0FrvL 2024-02-01T10:20:50.189 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:v/l225ZbxL2ZVdoZw2Ci4B5Bc/M= 2024-02-01T10:20:50.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:20:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:v/l225ZbxL2ZVdoZw2Ci4B5Bc/M=', 'Content-Length': '0'} 2024-02-01T10:20:50.211 DEBUG:boto:Response headers: [('x-amz-version-id', 'c490oGgvrwL3X5qVcaYbCq3o3j0FrvL'), ('x-amz-request-id', 'tx0000019d6f866f940deed-0065bb7082-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:20:50.212 DEBUG:boto:path=/vclfcn-16/obj-test-zone2 2024-02-01T10:20:50.212 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2 2024-02-01T10:20:50.212 DEBUG:boto:Method: PUT 2024-02-01T10:20:50.212 DEBUG:boto:Path: /vclfcn-16/obj-test-zone2 2024-02-01T10:20:50.212 DEBUG:boto:Data: 2024-02-01T10:20:50.212 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-02-01T10:20:50.212 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:20:50.212 DEBUG:boto:Port: 8000 2024-02-01T10:20:50.213 DEBUG:boto:Params: {} 2024-02-01T10:20:50.213 DEBUG:boto:Token: None 2024-02-01T10:20:50.213 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Thu, 01 Feb 2024 10:20:50 GMT /vclfcn-16/obj-test-zone2 2024-02-01T10:20:50.213 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ws7SQ5d2tbmwFjwz7B2UM/e95Qs= 2024-02-01T10:20:50.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:20:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ws7SQ5d2tbmwFjwz7B2UM/e95Qs='} 2024-02-01T10:20:50.256 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'b4b5mgNIKv0wbfMfprYgXUSK4kbmWQ8'), ('x-amz-request-id', 'tx00000983d05d03b9fc778-0065bb7082-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:20:50.256 DEBUG:rgw_multi.tests:version3 id=b4b5mgNIKv0wbfMfprYgXUSK4kbmWQ8 2024-02-01T10:20:50.257 DEBUG:boto:path=/vclfcn-16/obj-test-zone2 2024-02-01T10:20:50.257 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2 2024-02-01T10:20:50.257 DEBUG:boto:path=/vclfcn-16/obj-test-zone2?versionId=b4b5mgNIKv0wbfMfprYgXUSK4kbmWQ8 2024-02-01T10:20:50.257 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2?versionId=b4b5mgNIKv0wbfMfprYgXUSK4kbmWQ8 2024-02-01T10:20:50.257 DEBUG:boto:Method: DELETE 2024-02-01T10:20:50.257 DEBUG:boto:Path: /vclfcn-16/obj-test-zone2?versionId=b4b5mgNIKv0wbfMfprYgXUSK4kbmWQ8 2024-02-01T10:20:50.257 DEBUG:boto:Data: 2024-02-01T10:20:50.257 DEBUG:boto:Headers: {} 2024-02-01T10:20:50.257 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:20:50.257 DEBUG:boto:Port: 8000 2024-02-01T10:20:50.257 DEBUG:boto:Params: {} 2024-02-01T10:20:50.258 DEBUG:boto:Token: None 2024-02-01T10:20:50.258 DEBUG:boto:StringToSign: DELETE Thu, 01 Feb 2024 10:20:50 GMT /vclfcn-16/obj-test-zone2?versionId=b4b5mgNIKv0wbfMfprYgXUSK4kbmWQ8 2024-02-01T10:20:50.258 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:J5CVXPxcZ+16IT8rVshTOsUW8kY= 2024-02-01T10:20:50.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:20:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:J5CVXPxcZ+16IT8rVshTOsUW8kY=', 'Content-Length': '0'} 2024-02-01T10:20:50.281 DEBUG:boto:Response headers: [('x-amz-version-id', 'b4b5mgNIKv0wbfMfprYgXUSK4kbmWQ8'), ('x-amz-request-id', 'tx00000ff769a178926ab1d-0065bb7082-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:20:50.282 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:20:50.282 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:20:50.282 DEBUG:boto:Method: HEAD 2024-02-01T10:20:50.282 DEBUG:boto:Path: /vclfcn-16/ 2024-02-01T10:20:50.282 DEBUG:boto:Data: 2024-02-01T10:20:50.282 DEBUG:boto:Headers: {} 2024-02-01T10:20:50.282 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:20:50.282 DEBUG:boto:Port: 8001 2024-02-01T10:20:50.282 DEBUG:boto:Params: {} 2024-02-01T10:20:50.283 DEBUG:boto:Token: None 2024-02-01T10:20:50.283 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:20:50 GMT /vclfcn-16/ 2024-02-01T10:20:50.283 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:V/wieOXSldUtmJzlk2JMcy/pj8A= 2024-02-01T10:20:50.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:20:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:V/wieOXSldUtmJzlk2JMcy/pj8A=', 'Content-Length': '0'} 2024-02-01T10:20:50.288 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000481b7bf6e8628af5-0065bb7082-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:20:50.288 DEBUG:boto:path=/vclfcn-16/obj-test-zone3 2024-02-01T10:20:50.289 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3 2024-02-01T10:20:50.289 DEBUG:boto:Method: PUT 2024-02-01T10:20:50.289 DEBUG:boto:Path: /vclfcn-16/obj-test-zone3 2024-02-01T10:20:50.289 DEBUG:boto:Data: 2024-02-01T10:20:50.289 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-02-01T10:20:50.289 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:20:50.289 DEBUG:boto:Port: 8001 2024-02-01T10:20:50.289 DEBUG:boto:Params: {} 2024-02-01T10:20:50.289 DEBUG:boto:Token: None 2024-02-01T10:20:50.290 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 01 Feb 2024 10:20:50 GMT /vclfcn-16/obj-test-zone3 2024-02-01T10:20:50.290 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Eh+8jVgjmI1dpSppX0RSLFTZSmY= 2024-02-01T10:20:50.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:20:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Eh+8jVgjmI1dpSppX0RSLFTZSmY='} 2024-02-01T10:20:50.332 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP'), ('x-amz-request-id', 'tx00000e11089740521d086-0065bb7082-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:20:50.333 DEBUG:rgw_multi.tests:version1 id=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:20:50.333 DEBUG:boto:path=/vclfcn-16/obj-test-zone3 2024-02-01T10:20:50.333 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3 2024-02-01T10:20:50.333 DEBUG:boto:Method: PUT 2024-02-01T10:20:50.333 DEBUG:boto:Path: /vclfcn-16/obj-test-zone3 2024-02-01T10:20:50.333 DEBUG:boto:Data: 2024-02-01T10:20:50.333 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-02-01T10:20:50.334 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:20:50.334 DEBUG:boto:Port: 8001 2024-02-01T10:20:50.334 DEBUG:boto:Params: {} 2024-02-01T10:20:50.334 DEBUG:boto:Token: None 2024-02-01T10:20:50.334 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Thu, 01 Feb 2024 10:20:50 GMT /vclfcn-16/obj-test-zone3 2024-02-01T10:20:50.334 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WfDf/08YpCW5tX50WB5kEUqcSqE= 2024-02-01T10:20:50.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:20:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WfDf/08YpCW5tX50WB5kEUqcSqE='} 2024-02-01T10:20:50.356 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'Rs7p-9UKMSAUpXxy6zb8nqKZUE.mGjH'), ('x-amz-request-id', 'tx00000954ff20991f937f1-0065bb7082-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:20:50.356 DEBUG:rgw_multi.tests:version2 id=Rs7p-9UKMSAUpXxy6zb8nqKZUE.mGjH 2024-02-01T10:20:50.356 DEBUG:boto:path=/vclfcn-16/obj-test-zone3 2024-02-01T10:20:50.357 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3 2024-02-01T10:20:50.357 DEBUG:boto:path=/vclfcn-16/obj-test-zone3?versionId=Rs7p-9UKMSAUpXxy6zb8nqKZUE.mGjH 2024-02-01T10:20:50.357 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3?versionId=Rs7p-9UKMSAUpXxy6zb8nqKZUE.mGjH 2024-02-01T10:20:50.357 DEBUG:boto:Method: DELETE 2024-02-01T10:20:50.357 DEBUG:boto:Path: /vclfcn-16/obj-test-zone3?versionId=Rs7p-9UKMSAUpXxy6zb8nqKZUE.mGjH 2024-02-01T10:20:50.357 DEBUG:boto:Data: 2024-02-01T10:20:50.357 DEBUG:boto:Headers: {} 2024-02-01T10:20:50.357 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:20:50.357 DEBUG:boto:Port: 8001 2024-02-01T10:20:50.358 DEBUG:boto:Params: {} 2024-02-01T10:20:50.358 DEBUG:boto:Token: None 2024-02-01T10:20:50.358 DEBUG:boto:StringToSign: DELETE Thu, 01 Feb 2024 10:20:50 GMT /vclfcn-16/obj-test-zone3?versionId=Rs7p-9UKMSAUpXxy6zb8nqKZUE.mGjH 2024-02-01T10:20:50.358 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0zGIbeho+0WQtiiWde8KHji1kYw= 2024-02-01T10:20:50.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:20:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0zGIbeho+0WQtiiWde8KHji1kYw=', 'Content-Length': '0'} 2024-02-01T10:20:50.382 DEBUG:boto:Response headers: [('x-amz-version-id', 'Rs7p-9UKMSAUpXxy6zb8nqKZUE.mGjH'), ('x-amz-request-id', 'tx00000d22f9d271f31ca5b-0065bb7082-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:20:50.382 DEBUG:boto:path=/vclfcn-16/obj-test-zone3 2024-02-01T10:20:50.382 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3 2024-02-01T10:20:50.382 DEBUG:boto:Method: PUT 2024-02-01T10:20:50.382 DEBUG:boto:Path: /vclfcn-16/obj-test-zone3 2024-02-01T10:20:50.383 DEBUG:boto:Data: 2024-02-01T10:20:50.383 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-02-01T10:20:50.383 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:20:50.383 DEBUG:boto:Port: 8001 2024-02-01T10:20:50.383 DEBUG:boto:Params: {} 2024-02-01T10:20:50.383 DEBUG:boto:Token: None 2024-02-01T10:20:50.383 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Thu, 01 Feb 2024 10:20:50 GMT /vclfcn-16/obj-test-zone3 2024-02-01T10:20:50.383 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:c2oF30qnLUoPNBnSdcMvPljxxTY= 2024-02-01T10:20:50.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:20:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:c2oF30qnLUoPNBnSdcMvPljxxTY='} 2024-02-01T10:20:50.432 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'CYgmIsauenRqDdyrHTLb7O2bRAkqZrL'), ('x-amz-request-id', 'tx000003cb528e5404c0468-0065bb7082-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:20:50.433 DEBUG:rgw_multi.tests:version3 id=CYgmIsauenRqDdyrHTLb7O2bRAkqZrL 2024-02-01T10:20:50.433 DEBUG:boto:path=/vclfcn-16/obj-test-zone3 2024-02-01T10:20:50.433 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3 2024-02-01T10:20:50.433 DEBUG:boto:path=/vclfcn-16/obj-test-zone3?versionId=CYgmIsauenRqDdyrHTLb7O2bRAkqZrL 2024-02-01T10:20:50.433 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3?versionId=CYgmIsauenRqDdyrHTLb7O2bRAkqZrL 2024-02-01T10:20:50.433 DEBUG:boto:Method: DELETE 2024-02-01T10:20:50.433 DEBUG:boto:Path: /vclfcn-16/obj-test-zone3?versionId=CYgmIsauenRqDdyrHTLb7O2bRAkqZrL 2024-02-01T10:20:50.433 DEBUG:boto:Data: 2024-02-01T10:20:50.433 DEBUG:boto:Headers: {} 2024-02-01T10:20:50.433 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:20:50.434 DEBUG:boto:Port: 8001 2024-02-01T10:20:50.434 DEBUG:boto:Params: {} 2024-02-01T10:20:50.434 DEBUG:boto:Token: None 2024-02-01T10:20:50.434 DEBUG:boto:StringToSign: DELETE Thu, 01 Feb 2024 10:20:50 GMT /vclfcn-16/obj-test-zone3?versionId=CYgmIsauenRqDdyrHTLb7O2bRAkqZrL 2024-02-01T10:20:50.434 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:LiPrXY9tDkOIawHRkP7AMWj4o/g= 2024-02-01T10:20:50.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:20:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:LiPrXY9tDkOIawHRkP7AMWj4o/g=', 'Content-Length': '0'} 2024-02-01T10:20:50.459 DEBUG:boto:Response headers: [('x-amz-version-id', 'CYgmIsauenRqDdyrHTLb7O2bRAkqZrL'), ('x-amz-request-id', 'tx00000724607695e0ade74-0065bb7082-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:20:50.459 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:20:50.460 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:20:50.460 DEBUG:boto:Method: HEAD 2024-02-01T10:20:50.460 DEBUG:boto:Path: /vclfcn-17/ 2024-02-01T10:20:50.460 DEBUG:boto:Data: 2024-02-01T10:20:50.460 DEBUG:boto:Headers: {} 2024-02-01T10:20:50.460 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:20:50.460 DEBUG:boto:Port: 8000 2024-02-01T10:20:50.460 DEBUG:boto:Params: {} 2024-02-01T10:20:50.460 DEBUG:boto:Token: None 2024-02-01T10:20:50.461 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:20:50 GMT /vclfcn-17/ 2024-02-01T10:20:50.461 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FBZjUJX4dGli65F4t5NdXGJHqV4= 2024-02-01T10:20:50.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:20:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FBZjUJX4dGli65F4t5NdXGJHqV4=', 'Content-Length': '0'} 2024-02-01T10:20:50.465 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fd6bae3b5b4ca5c7-0065bb7082-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:20:50.465 DEBUG:boto:path=/vclfcn-17/obj-test-zone1 2024-02-01T10:20:50.465 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1 2024-02-01T10:20:50.465 DEBUG:boto:Method: PUT 2024-02-01T10:20:50.465 DEBUG:boto:Path: /vclfcn-17/obj-test-zone1 2024-02-01T10:20:50.465 DEBUG:boto:Data: 2024-02-01T10:20:50.465 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-02-01T10:20:50.465 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:20:50.466 DEBUG:boto:Port: 8000 2024-02-01T10:20:50.466 DEBUG:boto:Params: {} 2024-02-01T10:20:50.466 DEBUG:boto:Token: None 2024-02-01T10:20:50.466 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 01 Feb 2024 10:20:50 GMT /vclfcn-17/obj-test-zone1 2024-02-01T10:20:50.466 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9ajRUryedNIbqS5kluHcAGOHvDA= 2024-02-01T10:20:50.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:20:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9ajRUryedNIbqS5kluHcAGOHvDA='} 2024-02-01T10:20:50.491 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'xMxAGR4A-xEILepnzFPRgrWNTOLvitz'), ('x-amz-request-id', 'tx00000d85e51e7cf30d37d-0065bb7082-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:20:50.491 DEBUG:rgw_multi.tests:version1 id=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:20:50.491 DEBUG:boto:path=/vclfcn-17/obj-test-zone1 2024-02-01T10:20:50.492 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1 2024-02-01T10:20:50.492 DEBUG:boto:Method: PUT 2024-02-01T10:20:50.492 DEBUG:boto:Path: /vclfcn-17/obj-test-zone1 2024-02-01T10:20:50.492 DEBUG:boto:Data: 2024-02-01T10:20:50.492 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-02-01T10:20:50.492 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:20:50.492 DEBUG:boto:Port: 8000 2024-02-01T10:20:50.492 DEBUG:boto:Params: {} 2024-02-01T10:20:50.492 DEBUG:boto:Token: None 2024-02-01T10:20:50.493 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Thu, 01 Feb 2024 10:20:50 GMT /vclfcn-17/obj-test-zone1 2024-02-01T10:20:50.493 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:o8rZyShYHA5usGBjcjaGjtldLPk= 2024-02-01T10:20:50.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:20:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:o8rZyShYHA5usGBjcjaGjtldLPk='} 2024-02-01T10:20:50.515 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'NCYHv2wmQl8hPhZZa.4k7jPDW6q-XaH'), ('x-amz-request-id', 'tx00000ada7fb30281aeb30-0065bb7082-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:20:50.515 DEBUG:rgw_multi.tests:version2 id=NCYHv2wmQl8hPhZZa.4k7jPDW6q-XaH 2024-02-01T10:20:50.516 DEBUG:boto:path=/vclfcn-17/obj-test-zone1 2024-02-01T10:20:50.516 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1 2024-02-01T10:20:50.516 DEBUG:boto:path=/vclfcn-17/obj-test-zone1?versionId=NCYHv2wmQl8hPhZZa.4k7jPDW6q-XaH 2024-02-01T10:20:50.516 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1?versionId=NCYHv2wmQl8hPhZZa.4k7jPDW6q-XaH 2024-02-01T10:20:50.516 DEBUG:boto:Method: DELETE 2024-02-01T10:20:50.516 DEBUG:boto:Path: /vclfcn-17/obj-test-zone1?versionId=NCYHv2wmQl8hPhZZa.4k7jPDW6q-XaH 2024-02-01T10:20:50.516 DEBUG:boto:Data: 2024-02-01T10:20:50.516 DEBUG:boto:Headers: {} 2024-02-01T10:20:50.516 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:20:50.516 DEBUG:boto:Port: 8000 2024-02-01T10:20:50.516 DEBUG:boto:Params: {} 2024-02-01T10:20:50.517 DEBUG:boto:Token: None 2024-02-01T10:20:50.517 DEBUG:boto:StringToSign: DELETE Thu, 01 Feb 2024 10:20:50 GMT /vclfcn-17/obj-test-zone1?versionId=NCYHv2wmQl8hPhZZa.4k7jPDW6q-XaH 2024-02-01T10:20:50.517 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ztW7ikfu8hYKvWCRdjgIdpIZmR8= 2024-02-01T10:20:50.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:20:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ztW7ikfu8hYKvWCRdjgIdpIZmR8=', 'Content-Length': '0'} 2024-02-01T10:20:50.540 DEBUG:boto:Response headers: [('x-amz-version-id', 'NCYHv2wmQl8hPhZZa.4k7jPDW6q-XaH'), ('x-amz-request-id', 'tx000004624199dbf8c4e91-0065bb7082-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:20:50.541 DEBUG:boto:path=/vclfcn-17/obj-test-zone1 2024-02-01T10:20:50.541 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1 2024-02-01T10:20:50.541 DEBUG:boto:Method: PUT 2024-02-01T10:20:50.541 DEBUG:boto:Path: /vclfcn-17/obj-test-zone1 2024-02-01T10:20:50.541 DEBUG:boto:Data: 2024-02-01T10:20:50.541 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-02-01T10:20:50.541 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:20:50.541 DEBUG:boto:Port: 8000 2024-02-01T10:20:50.541 DEBUG:boto:Params: {} 2024-02-01T10:20:50.542 DEBUG:boto:Token: None 2024-02-01T10:20:50.542 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Thu, 01 Feb 2024 10:20:50 GMT /vclfcn-17/obj-test-zone1 2024-02-01T10:20:50.542 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ayt1NOTP28GIzsxG1hDxLffinmQ= 2024-02-01T10:20:50.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:20:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ayt1NOTP28GIzsxG1hDxLffinmQ='} 2024-02-01T10:20:50.584 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '7SPNxG5D485GoUyk5IRd9Pmpo.QET3H'), ('x-amz-request-id', 'tx00000f740286f4d408e7b-0065bb7082-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:20:50.585 DEBUG:rgw_multi.tests:version3 id=7SPNxG5D485GoUyk5IRd9Pmpo.QET3H 2024-02-01T10:20:50.585 DEBUG:boto:path=/vclfcn-17/obj-test-zone1 2024-02-01T10:20:50.585 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1 2024-02-01T10:20:50.585 DEBUG:boto:path=/vclfcn-17/obj-test-zone1?versionId=7SPNxG5D485GoUyk5IRd9Pmpo.QET3H 2024-02-01T10:20:50.585 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1?versionId=7SPNxG5D485GoUyk5IRd9Pmpo.QET3H 2024-02-01T10:20:50.585 DEBUG:boto:Method: DELETE 2024-02-01T10:20:50.585 DEBUG:boto:Path: /vclfcn-17/obj-test-zone1?versionId=7SPNxG5D485GoUyk5IRd9Pmpo.QET3H 2024-02-01T10:20:50.585 DEBUG:boto:Data: 2024-02-01T10:20:50.585 DEBUG:boto:Headers: {} 2024-02-01T10:20:50.585 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:20:50.585 DEBUG:boto:Port: 8000 2024-02-01T10:20:50.586 DEBUG:boto:Params: {} 2024-02-01T10:20:50.586 DEBUG:boto:Token: None 2024-02-01T10:20:50.586 DEBUG:boto:StringToSign: DELETE Thu, 01 Feb 2024 10:20:50 GMT /vclfcn-17/obj-test-zone1?versionId=7SPNxG5D485GoUyk5IRd9Pmpo.QET3H 2024-02-01T10:20:50.586 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:XNj+HScvYpFaGSBkqnWeFIAFVW0= 2024-02-01T10:20:50.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:20:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:XNj+HScvYpFaGSBkqnWeFIAFVW0=', 'Content-Length': '0'} 2024-02-01T10:20:50.610 DEBUG:boto:Response headers: [('x-amz-version-id', '7SPNxG5D485GoUyk5IRd9Pmpo.QET3H'), ('x-amz-request-id', 'tx00000ee6c7cc0ef36997c-0065bb7082-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:20:50.610 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:20:50.610 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:20:50.611 DEBUG:boto:Method: HEAD 2024-02-01T10:20:50.611 DEBUG:boto:Path: /vclfcn-17/ 2024-02-01T10:20:50.611 DEBUG:boto:Data: 2024-02-01T10:20:50.611 DEBUG:boto:Headers: {} 2024-02-01T10:20:50.611 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:20:50.611 DEBUG:boto:Port: 8000 2024-02-01T10:20:50.611 DEBUG:boto:Params: {} 2024-02-01T10:20:50.611 DEBUG:boto:Token: None 2024-02-01T10:20:50.611 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:20:50 GMT /vclfcn-17/ 2024-02-01T10:20:50.611 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FBZjUJX4dGli65F4t5NdXGJHqV4= 2024-02-01T10:20:50.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:20:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FBZjUJX4dGli65F4t5NdXGJHqV4=', 'Content-Length': '0'} 2024-02-01T10:20:50.617 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000048bb33c7d19d5f5d-0065bb7082-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:20:50.617 DEBUG:boto:path=/vclfcn-17/obj-test-zone2 2024-02-01T10:20:50.617 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2 2024-02-01T10:20:50.617 DEBUG:boto:Method: PUT 2024-02-01T10:20:50.617 DEBUG:boto:Path: /vclfcn-17/obj-test-zone2 2024-02-01T10:20:50.617 DEBUG:boto:Data: 2024-02-01T10:20:50.618 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-02-01T10:20:50.618 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:20:50.618 DEBUG:boto:Port: 8000 2024-02-01T10:20:50.618 DEBUG:boto:Params: {} 2024-02-01T10:20:50.618 DEBUG:boto:Token: None 2024-02-01T10:20:50.618 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 01 Feb 2024 10:20:50 GMT /vclfcn-17/obj-test-zone2 2024-02-01T10:20:50.618 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cKSfjEvW4k5jtj331LqgdSffLik= 2024-02-01T10:20:50.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:20:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cKSfjEvW4k5jtj331LqgdSffLik='} 2024-02-01T10:20:50.643 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1'), ('x-amz-request-id', 'tx000003e39435dca07564b-0065bb7082-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:20:50.643 DEBUG:rgw_multi.tests:version1 id=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:20:50.643 DEBUG:boto:path=/vclfcn-17/obj-test-zone2 2024-02-01T10:20:50.644 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2 2024-02-01T10:20:50.644 DEBUG:boto:Method: PUT 2024-02-01T10:20:50.644 DEBUG:boto:Path: /vclfcn-17/obj-test-zone2 2024-02-01T10:20:50.644 DEBUG:boto:Data: 2024-02-01T10:20:50.644 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-02-01T10:20:50.644 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:20:50.644 DEBUG:boto:Port: 8000 2024-02-01T10:20:50.644 DEBUG:boto:Params: {} 2024-02-01T10:20:50.644 DEBUG:boto:Token: None 2024-02-01T10:20:50.645 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Thu, 01 Feb 2024 10:20:50 GMT /vclfcn-17/obj-test-zone2 2024-02-01T10:20:50.645 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YoV70DjwQVsyT3k7rmicrqHuZVM= 2024-02-01T10:20:50.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:20:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YoV70DjwQVsyT3k7rmicrqHuZVM='} 2024-02-01T10:20:50.692 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'miU1t6efrISkwvcBJJSigCe9jb9zU.m'), ('x-amz-request-id', 'tx000000eeab4d86a636108-0065bb7082-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:20:50.693 DEBUG:rgw_multi.tests:version2 id=miU1t6efrISkwvcBJJSigCe9jb9zU.m 2024-02-01T10:20:50.693 DEBUG:boto:path=/vclfcn-17/obj-test-zone2 2024-02-01T10:20:50.693 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2 2024-02-01T10:20:50.693 DEBUG:boto:path=/vclfcn-17/obj-test-zone2?versionId=miU1t6efrISkwvcBJJSigCe9jb9zU.m 2024-02-01T10:20:50.693 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2?versionId=miU1t6efrISkwvcBJJSigCe9jb9zU.m 2024-02-01T10:20:50.693 DEBUG:boto:Method: DELETE 2024-02-01T10:20:50.693 DEBUG:boto:Path: /vclfcn-17/obj-test-zone2?versionId=miU1t6efrISkwvcBJJSigCe9jb9zU.m 2024-02-01T10:20:50.693 DEBUG:boto:Data: 2024-02-01T10:20:50.693 DEBUG:boto:Headers: {} 2024-02-01T10:20:50.693 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:20:50.694 DEBUG:boto:Port: 8000 2024-02-01T10:20:50.694 DEBUG:boto:Params: {} 2024-02-01T10:20:50.694 DEBUG:boto:Token: None 2024-02-01T10:20:50.694 DEBUG:boto:StringToSign: DELETE Thu, 01 Feb 2024 10:20:50 GMT /vclfcn-17/obj-test-zone2?versionId=miU1t6efrISkwvcBJJSigCe9jb9zU.m 2024-02-01T10:20:50.694 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:eqi39WBM5ZmnetBDVWW47OvsweY= 2024-02-01T10:20:50.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:20:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:eqi39WBM5ZmnetBDVWW47OvsweY=', 'Content-Length': '0'} 2024-02-01T10:20:50.718 DEBUG:boto:Response headers: [('x-amz-version-id', 'miU1t6efrISkwvcBJJSigCe9jb9zU.m'), ('x-amz-request-id', 'tx00000e3cb206c4a525f02-0065bb7082-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:20:50.718 DEBUG:boto:path=/vclfcn-17/obj-test-zone2 2024-02-01T10:20:50.718 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2 2024-02-01T10:20:50.718 DEBUG:boto:Method: PUT 2024-02-01T10:20:50.718 DEBUG:boto:Path: /vclfcn-17/obj-test-zone2 2024-02-01T10:20:50.719 DEBUG:boto:Data: 2024-02-01T10:20:50.719 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-02-01T10:20:50.719 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:20:50.719 DEBUG:boto:Port: 8000 2024-02-01T10:20:50.719 DEBUG:boto:Params: {} 2024-02-01T10:20:50.719 DEBUG:boto:Token: None 2024-02-01T10:20:50.719 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Thu, 01 Feb 2024 10:20:50 GMT /vclfcn-17/obj-test-zone2 2024-02-01T10:20:50.719 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:DhaVmKAFnvforJv6jOAJ6DoWw68= 2024-02-01T10:20:50.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:20:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:DhaVmKAFnvforJv6jOAJ6DoWw68='} 2024-02-01T10:20:50.740 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'v43.LA0WP2kT-PzgKeeIe6WWXQPJt.k'), ('x-amz-request-id', 'tx00000d7f28c6d89fc0e25-0065bb7082-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:20:50.741 DEBUG:rgw_multi.tests:version3 id=v43.LA0WP2kT-PzgKeeIe6WWXQPJt.k 2024-02-01T10:20:50.741 DEBUG:boto:path=/vclfcn-17/obj-test-zone2 2024-02-01T10:20:50.741 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2 2024-02-01T10:20:50.741 DEBUG:boto:path=/vclfcn-17/obj-test-zone2?versionId=v43.LA0WP2kT-PzgKeeIe6WWXQPJt.k 2024-02-01T10:20:50.741 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2?versionId=v43.LA0WP2kT-PzgKeeIe6WWXQPJt.k 2024-02-01T10:20:50.741 DEBUG:boto:Method: DELETE 2024-02-01T10:20:50.741 DEBUG:boto:Path: /vclfcn-17/obj-test-zone2?versionId=v43.LA0WP2kT-PzgKeeIe6WWXQPJt.k 2024-02-01T10:20:50.742 DEBUG:boto:Data: 2024-02-01T10:20:50.742 DEBUG:boto:Headers: {} 2024-02-01T10:20:50.742 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:20:50.742 DEBUG:boto:Port: 8000 2024-02-01T10:20:50.742 DEBUG:boto:Params: {} 2024-02-01T10:20:50.742 DEBUG:boto:Token: None 2024-02-01T10:20:50.743 DEBUG:boto:StringToSign: DELETE Thu, 01 Feb 2024 10:20:50 GMT /vclfcn-17/obj-test-zone2?versionId=v43.LA0WP2kT-PzgKeeIe6WWXQPJt.k 2024-02-01T10:20:50.743 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nM0Wk0VLI9NvGSdsEdtbNalVsgg= 2024-02-01T10:20:50.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:20:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nM0Wk0VLI9NvGSdsEdtbNalVsgg=', 'Content-Length': '0'} 2024-02-01T10:20:50.766 DEBUG:boto:Response headers: [('x-amz-version-id', 'v43.LA0WP2kT-PzgKeeIe6WWXQPJt.k'), ('x-amz-request-id', 'tx0000094153a932b1e8aed-0065bb7082-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:20:50.766 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:20:50.766 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:20:50.766 DEBUG:boto:Method: HEAD 2024-02-01T10:20:50.766 DEBUG:boto:Path: /vclfcn-17/ 2024-02-01T10:20:50.766 DEBUG:boto:Data: 2024-02-01T10:20:50.767 DEBUG:boto:Headers: {} 2024-02-01T10:20:50.767 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:20:50.767 DEBUG:boto:Port: 8001 2024-02-01T10:20:50.767 DEBUG:boto:Params: {} 2024-02-01T10:20:50.767 DEBUG:boto:Token: None 2024-02-01T10:20:50.767 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:20:50 GMT /vclfcn-17/ 2024-02-01T10:20:50.767 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FBZjUJX4dGli65F4t5NdXGJHqV4= 2024-02-01T10:20:50.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:20:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FBZjUJX4dGli65F4t5NdXGJHqV4=', 'Content-Length': '0'} 2024-02-01T10:20:50.771 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e448a4cca95804e8-0065bb7082-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:20:50.772 DEBUG:boto:path=/vclfcn-17/obj-test-zone3 2024-02-01T10:20:50.772 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3 2024-02-01T10:20:50.772 DEBUG:boto:Method: PUT 2024-02-01T10:20:50.772 DEBUG:boto:Path: /vclfcn-17/obj-test-zone3 2024-02-01T10:20:50.772 DEBUG:boto:Data: 2024-02-01T10:20:50.772 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-02-01T10:20:50.772 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:20:50.772 DEBUG:boto:Port: 8001 2024-02-01T10:20:50.773 DEBUG:boto:Params: {} 2024-02-01T10:20:50.773 DEBUG:boto:Token: None 2024-02-01T10:20:50.773 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 01 Feb 2024 10:20:50 GMT /vclfcn-17/obj-test-zone3 2024-02-01T10:20:50.773 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:rU6sYhSXrYnWAiE2JP88kpoKpFc= 2024-02-01T10:20:50.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:20:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:rU6sYhSXrYnWAiE2JP88kpoKpFc='} 2024-02-01T10:20:50.796 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'YE736OwOOXbZJHJe7d8YvcNV1LxwcQb'), ('x-amz-request-id', 'tx00000720f8b086ca54732-0065bb7082-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:20:50.796 DEBUG:rgw_multi.tests:version1 id=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:20:50.796 DEBUG:boto:path=/vclfcn-17/obj-test-zone3 2024-02-01T10:20:50.797 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3 2024-02-01T10:20:50.797 DEBUG:boto:Method: PUT 2024-02-01T10:20:50.797 DEBUG:boto:Path: /vclfcn-17/obj-test-zone3 2024-02-01T10:20:50.797 DEBUG:boto:Data: 2024-02-01T10:20:50.797 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-02-01T10:20:50.797 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:20:50.797 DEBUG:boto:Port: 8001 2024-02-01T10:20:50.797 DEBUG:boto:Params: {} 2024-02-01T10:20:50.797 DEBUG:boto:Token: None 2024-02-01T10:20:50.798 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Thu, 01 Feb 2024 10:20:50 GMT /vclfcn-17/obj-test-zone3 2024-02-01T10:20:50.798 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:DA5WV+HoYCaZj0z7snd726WJi20= 2024-02-01T10:20:50.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:20:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:DA5WV+HoYCaZj0z7snd726WJi20='} 2024-02-01T10:20:50.821 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'HKG1HqJyAOxcRCUazBWPaC3CPccRqZP'), ('x-amz-request-id', 'tx00000993d275c96d41ef6-0065bb7082-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:20:50.821 DEBUG:rgw_multi.tests:version2 id=HKG1HqJyAOxcRCUazBWPaC3CPccRqZP 2024-02-01T10:20:50.821 DEBUG:boto:path=/vclfcn-17/obj-test-zone3 2024-02-01T10:20:50.821 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3 2024-02-01T10:20:50.822 DEBUG:boto:path=/vclfcn-17/obj-test-zone3?versionId=HKG1HqJyAOxcRCUazBWPaC3CPccRqZP 2024-02-01T10:20:50.822 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3?versionId=HKG1HqJyAOxcRCUazBWPaC3CPccRqZP 2024-02-01T10:20:50.822 DEBUG:boto:Method: DELETE 2024-02-01T10:20:50.822 DEBUG:boto:Path: /vclfcn-17/obj-test-zone3?versionId=HKG1HqJyAOxcRCUazBWPaC3CPccRqZP 2024-02-01T10:20:50.822 DEBUG:boto:Data: 2024-02-01T10:20:50.822 DEBUG:boto:Headers: {} 2024-02-01T10:20:50.822 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:20:50.822 DEBUG:boto:Port: 8001 2024-02-01T10:20:50.822 DEBUG:boto:Params: {} 2024-02-01T10:20:50.823 DEBUG:boto:Token: None 2024-02-01T10:20:50.823 DEBUG:boto:StringToSign: DELETE Thu, 01 Feb 2024 10:20:50 GMT /vclfcn-17/obj-test-zone3?versionId=HKG1HqJyAOxcRCUazBWPaC3CPccRqZP 2024-02-01T10:20:50.823 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/E+Xu+Trh/3oTXk2BHwV+UqxZvk= 2024-02-01T10:20:50.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:20:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/E+Xu+Trh/3oTXk2BHwV+UqxZvk=', 'Content-Length': '0'} 2024-02-01T10:20:50.845 DEBUG:boto:Response headers: [('x-amz-version-id', 'HKG1HqJyAOxcRCUazBWPaC3CPccRqZP'), ('x-amz-request-id', 'tx00000a0749e740ecddb4e-0065bb7082-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:20:50.846 DEBUG:boto:path=/vclfcn-17/obj-test-zone3 2024-02-01T10:20:50.846 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3 2024-02-01T10:20:50.846 DEBUG:boto:Method: PUT 2024-02-01T10:20:50.846 DEBUG:boto:Path: /vclfcn-17/obj-test-zone3 2024-02-01T10:20:50.846 DEBUG:boto:Data: 2024-02-01T10:20:50.847 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-02-01T10:20:50.847 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:20:50.847 DEBUG:boto:Port: 8001 2024-02-01T10:20:50.847 DEBUG:boto:Params: {} 2024-02-01T10:20:50.847 DEBUG:boto:Token: None 2024-02-01T10:20:50.847 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Thu, 01 Feb 2024 10:20:50 GMT /vclfcn-17/obj-test-zone3 2024-02-01T10:20:50.847 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1jyDBZgxCJkOlggbLTg2wzxo7aw= 2024-02-01T10:20:50.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:20:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1jyDBZgxCJkOlggbLTg2wzxo7aw='} 2024-02-01T10:20:50.896 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'k1rUhRJCBD.wrYidB--t8iyqlM.gTVs'), ('x-amz-request-id', 'tx00000aca02c625d05fa3b-0065bb7082-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:20:50.897 DEBUG:rgw_multi.tests:version3 id=k1rUhRJCBD.wrYidB--t8iyqlM.gTVs 2024-02-01T10:20:50.897 DEBUG:boto:path=/vclfcn-17/obj-test-zone3 2024-02-01T10:20:50.897 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3 2024-02-01T10:20:50.897 DEBUG:boto:path=/vclfcn-17/obj-test-zone3?versionId=k1rUhRJCBD.wrYidB--t8iyqlM.gTVs 2024-02-01T10:20:50.898 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3?versionId=k1rUhRJCBD.wrYidB--t8iyqlM.gTVs 2024-02-01T10:20:50.898 DEBUG:boto:Method: DELETE 2024-02-01T10:20:50.898 DEBUG:boto:Path: /vclfcn-17/obj-test-zone3?versionId=k1rUhRJCBD.wrYidB--t8iyqlM.gTVs 2024-02-01T10:20:50.898 DEBUG:boto:Data: 2024-02-01T10:20:50.898 DEBUG:boto:Headers: {} 2024-02-01T10:20:50.899 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:20:50.899 DEBUG:boto:Port: 8001 2024-02-01T10:20:50.899 DEBUG:boto:Params: {} 2024-02-01T10:20:50.899 DEBUG:boto:Token: None 2024-02-01T10:20:50.900 DEBUG:boto:StringToSign: DELETE Thu, 01 Feb 2024 10:20:50 GMT /vclfcn-17/obj-test-zone3?versionId=k1rUhRJCBD.wrYidB--t8iyqlM.gTVs 2024-02-01T10:20:50.900 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:K1IZwZRt1972YbHQMbOqCOzYVGQ= 2024-02-01T10:20:50.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:20:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:K1IZwZRt1972YbHQMbOqCOzYVGQ=', 'Content-Length': '0'} 2024-02-01T10:20:50.924 DEBUG:boto:Response headers: [('x-amz-version-id', 'k1rUhRJCBD.wrYidB--t8iyqlM.gTVs'), ('x-amz-request-id', 'tx00000f314f319fc785efd-0065bb7082-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:20:50.924 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:20:50.925 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:20:50.925 DEBUG:boto:Method: HEAD 2024-02-01T10:20:50.925 DEBUG:boto:Path: /vclfcn-18/ 2024-02-01T10:20:50.925 DEBUG:boto:Data: 2024-02-01T10:20:50.925 DEBUG:boto:Headers: {} 2024-02-01T10:20:50.926 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:20:50.926 DEBUG:boto:Port: 8000 2024-02-01T10:20:50.926 DEBUG:boto:Params: {} 2024-02-01T10:20:50.926 DEBUG:boto:Token: None 2024-02-01T10:20:50.927 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:20:50 GMT /vclfcn-18/ 2024-02-01T10:20:50.927 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:U30N6b0GU5cfY4cIuk+v9Qnmv9M= 2024-02-01T10:20:50.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:20:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:U30N6b0GU5cfY4cIuk+v9Qnmv9M=', 'Content-Length': '0'} 2024-02-01T10:20:50.931 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000032314323b7998b30-0065bb7082-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:20:50.932 DEBUG:boto:path=/vclfcn-18/obj-test-zone1 2024-02-01T10:20:50.932 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1 2024-02-01T10:20:50.932 DEBUG:boto:Method: PUT 2024-02-01T10:20:50.932 DEBUG:boto:Path: /vclfcn-18/obj-test-zone1 2024-02-01T10:20:50.933 DEBUG:boto:Data: 2024-02-01T10:20:50.933 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-02-01T10:20:50.933 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:20:50.933 DEBUG:boto:Port: 8000 2024-02-01T10:20:50.933 DEBUG:boto:Params: {} 2024-02-01T10:20:50.934 DEBUG:boto:Token: None 2024-02-01T10:20:50.934 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 01 Feb 2024 10:20:50 GMT /vclfcn-18/obj-test-zone1 2024-02-01T10:20:50.934 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:M+CNNEeOVYv5XUBm8O4Lca7Hk+w= 2024-02-01T10:20:50.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:20:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:M+CNNEeOVYv5XUBm8O4Lca7Hk+w='} 2024-02-01T10:20:50.960 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK'), ('x-amz-request-id', 'tx00000698e20153d81b995-0065bb7082-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:20:50.961 DEBUG:rgw_multi.tests:version1 id=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:20:50.961 DEBUG:boto:path=/vclfcn-18/obj-test-zone1 2024-02-01T10:20:50.962 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1 2024-02-01T10:20:50.962 DEBUG:boto:Method: PUT 2024-02-01T10:20:50.962 DEBUG:boto:Path: /vclfcn-18/obj-test-zone1 2024-02-01T10:20:50.962 DEBUG:boto:Data: 2024-02-01T10:20:50.963 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-02-01T10:20:50.963 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:20:50.963 DEBUG:boto:Port: 8000 2024-02-01T10:20:50.963 DEBUG:boto:Params: {} 2024-02-01T10:20:50.963 DEBUG:boto:Token: None 2024-02-01T10:20:50.964 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Thu, 01 Feb 2024 10:20:50 GMT /vclfcn-18/obj-test-zone1 2024-02-01T10:20:50.964 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QWSvhoismoja/TwuMsCiOzSJtzE= 2024-02-01T10:20:50.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:20:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QWSvhoismoja/TwuMsCiOzSJtzE='} 2024-02-01T10:20:51.008 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'MJCZFeyzpqPbXC.GEZyA59tqoe6cRda'), ('x-amz-request-id', 'tx0000011f97e18dda59250-0065bb7082-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:20:51.009 DEBUG:rgw_multi.tests:version2 id=MJCZFeyzpqPbXC.GEZyA59tqoe6cRda 2024-02-01T10:20:51.009 DEBUG:boto:path=/vclfcn-18/obj-test-zone1 2024-02-01T10:20:51.009 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1 2024-02-01T10:20:51.009 DEBUG:boto:path=/vclfcn-18/obj-test-zone1?versionId=MJCZFeyzpqPbXC.GEZyA59tqoe6cRda 2024-02-01T10:20:51.009 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1?versionId=MJCZFeyzpqPbXC.GEZyA59tqoe6cRda 2024-02-01T10:20:51.009 DEBUG:boto:Method: DELETE 2024-02-01T10:20:51.009 DEBUG:boto:Path: /vclfcn-18/obj-test-zone1?versionId=MJCZFeyzpqPbXC.GEZyA59tqoe6cRda 2024-02-01T10:20:51.009 DEBUG:boto:Data: 2024-02-01T10:20:51.009 DEBUG:boto:Headers: {} 2024-02-01T10:20:51.010 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:20:51.010 DEBUG:boto:Port: 8000 2024-02-01T10:20:51.010 DEBUG:boto:Params: {} 2024-02-01T10:20:51.010 DEBUG:boto:Token: None 2024-02-01T10:20:51.010 DEBUG:boto:StringToSign: DELETE Thu, 01 Feb 2024 10:20:51 GMT /vclfcn-18/obj-test-zone1?versionId=MJCZFeyzpqPbXC.GEZyA59tqoe6cRda 2024-02-01T10:20:51.010 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:M7xLlzz1IdNyDrSLHFi+fRWyYx8= 2024-02-01T10:20:51.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:20:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:M7xLlzz1IdNyDrSLHFi+fRWyYx8=', 'Content-Length': '0'} 2024-02-01T10:20:51.033 DEBUG:boto:Response headers: [('x-amz-version-id', 'MJCZFeyzpqPbXC.GEZyA59tqoe6cRda'), ('x-amz-request-id', 'tx00000d5d31c02565ecc0a-0065bb7083-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:20:51.033 DEBUG:boto:path=/vclfcn-18/obj-test-zone1 2024-02-01T10:20:51.033 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1 2024-02-01T10:20:51.034 DEBUG:boto:Method: PUT 2024-02-01T10:20:51.034 DEBUG:boto:Path: /vclfcn-18/obj-test-zone1 2024-02-01T10:20:51.034 DEBUG:boto:Data: 2024-02-01T10:20:51.034 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-02-01T10:20:51.034 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:20:51.034 DEBUG:boto:Port: 8000 2024-02-01T10:20:51.034 DEBUG:boto:Params: {} 2024-02-01T10:20:51.034 DEBUG:boto:Token: None 2024-02-01T10:20:51.034 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Thu, 01 Feb 2024 10:20:51 GMT /vclfcn-18/obj-test-zone1 2024-02-01T10:20:51.035 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YYWcFpOnihNQ3b0EjYwfCoRR+h0= 2024-02-01T10:20:51.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:20:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YYWcFpOnihNQ3b0EjYwfCoRR+h0='} 2024-02-01T10:20:51.056 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'zyYgFGitF4LUOz41DuOg2wII-WcaZRY'), ('x-amz-request-id', 'tx000005a96f25d6da94b0b-0065bb7083-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:20:51.056 DEBUG:rgw_multi.tests:version3 id=zyYgFGitF4LUOz41DuOg2wII-WcaZRY 2024-02-01T10:20:51.056 DEBUG:boto:path=/vclfcn-18/obj-test-zone1 2024-02-01T10:20:51.056 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1 2024-02-01T10:20:51.056 DEBUG:boto:path=/vclfcn-18/obj-test-zone1?versionId=zyYgFGitF4LUOz41DuOg2wII-WcaZRY 2024-02-01T10:20:51.057 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1?versionId=zyYgFGitF4LUOz41DuOg2wII-WcaZRY 2024-02-01T10:20:51.057 DEBUG:boto:Method: DELETE 2024-02-01T10:20:51.057 DEBUG:boto:Path: /vclfcn-18/obj-test-zone1?versionId=zyYgFGitF4LUOz41DuOg2wII-WcaZRY 2024-02-01T10:20:51.057 DEBUG:boto:Data: 2024-02-01T10:20:51.057 DEBUG:boto:Headers: {} 2024-02-01T10:20:51.057 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:20:51.057 DEBUG:boto:Port: 8000 2024-02-01T10:20:51.057 DEBUG:boto:Params: {} 2024-02-01T10:20:51.057 DEBUG:boto:Token: None 2024-02-01T10:20:51.057 DEBUG:boto:StringToSign: DELETE Thu, 01 Feb 2024 10:20:51 GMT /vclfcn-18/obj-test-zone1?versionId=zyYgFGitF4LUOz41DuOg2wII-WcaZRY 2024-02-01T10:20:51.058 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/qJm+uIah3qZYLp4/PmkifmYzrA= 2024-02-01T10:20:51.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:20:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/qJm+uIah3qZYLp4/PmkifmYzrA=', 'Content-Length': '0'} 2024-02-01T10:20:51.080 DEBUG:boto:Response headers: [('x-amz-version-id', 'zyYgFGitF4LUOz41DuOg2wII-WcaZRY'), ('x-amz-request-id', 'tx00000a0e781967a3de997-0065bb7083-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:20:51.081 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:20:51.081 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:20:51.081 DEBUG:boto:Method: HEAD 2024-02-01T10:20:51.081 DEBUG:boto:Path: /vclfcn-18/ 2024-02-01T10:20:51.081 DEBUG:boto:Data: 2024-02-01T10:20:51.081 DEBUG:boto:Headers: {} 2024-02-01T10:20:51.081 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:20:51.082 DEBUG:boto:Port: 8000 2024-02-01T10:20:51.082 DEBUG:boto:Params: {} 2024-02-01T10:20:51.082 DEBUG:boto:Token: None 2024-02-01T10:20:51.082 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:20:51 GMT /vclfcn-18/ 2024-02-01T10:20:51.082 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:w4jCQC9tMU/EU1n5E5Z0Etq9WbQ= 2024-02-01T10:20:51.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:20:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:w4jCQC9tMU/EU1n5E5Z0Etq9WbQ=', 'Content-Length': '0'} 2024-02-01T10:20:51.086 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009bd6a3fa61f7014d-0065bb7083-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:20:51.086 DEBUG:boto:path=/vclfcn-18/obj-test-zone2 2024-02-01T10:20:51.087 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2 2024-02-01T10:20:51.087 DEBUG:boto:Method: PUT 2024-02-01T10:20:51.087 DEBUG:boto:Path: /vclfcn-18/obj-test-zone2 2024-02-01T10:20:51.087 DEBUG:boto:Data: 2024-02-01T10:20:51.087 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-02-01T10:20:51.087 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:20:51.087 DEBUG:boto:Port: 8000 2024-02-01T10:20:51.087 DEBUG:boto:Params: {} 2024-02-01T10:20:51.087 DEBUG:boto:Token: None 2024-02-01T10:20:51.087 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 01 Feb 2024 10:20:51 GMT /vclfcn-18/obj-test-zone2 2024-02-01T10:20:51.088 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:xCnAvE4fr444FIL+I9B2Aq8+W/Y= 2024-02-01T10:20:51.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:20:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:xCnAvE4fr444FIL+I9B2Aq8+W/Y='} 2024-02-01T10:20:51.113 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD'), ('x-amz-request-id', 'tx0000036b0261472683ad1-0065bb7083-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:20:51.113 DEBUG:rgw_multi.tests:version1 id=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:20:51.114 DEBUG:boto:path=/vclfcn-18/obj-test-zone2 2024-02-01T10:20:51.114 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2 2024-02-01T10:20:51.114 DEBUG:boto:Method: PUT 2024-02-01T10:20:51.114 DEBUG:boto:Path: /vclfcn-18/obj-test-zone2 2024-02-01T10:20:51.114 DEBUG:boto:Data: 2024-02-01T10:20:51.114 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-02-01T10:20:51.114 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:20:51.114 DEBUG:boto:Port: 8000 2024-02-01T10:20:51.114 DEBUG:boto:Params: {} 2024-02-01T10:20:51.114 DEBUG:boto:Token: None 2024-02-01T10:20:51.115 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Thu, 01 Feb 2024 10:20:51 GMT /vclfcn-18/obj-test-zone2 2024-02-01T10:20:51.115 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gsSr0ZutdGTTZabrGEEO3FI4Ayc= 2024-02-01T10:20:51.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:20:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gsSr0ZutdGTTZabrGEEO3FI4Ayc='} 2024-02-01T10:20:51.136 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'dRFINd5C3wqd9vSe3hVbazyoohOwmdD'), ('x-amz-request-id', 'tx0000007410d1e62846dc2-0065bb7083-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:20:51.137 DEBUG:rgw_multi.tests:version2 id=dRFINd5C3wqd9vSe3hVbazyoohOwmdD 2024-02-01T10:20:51.137 DEBUG:boto:path=/vclfcn-18/obj-test-zone2 2024-02-01T10:20:51.137 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2 2024-02-01T10:20:51.137 DEBUG:boto:path=/vclfcn-18/obj-test-zone2?versionId=dRFINd5C3wqd9vSe3hVbazyoohOwmdD 2024-02-01T10:20:51.137 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2?versionId=dRFINd5C3wqd9vSe3hVbazyoohOwmdD 2024-02-01T10:20:51.137 DEBUG:boto:Method: DELETE 2024-02-01T10:20:51.137 DEBUG:boto:Path: /vclfcn-18/obj-test-zone2?versionId=dRFINd5C3wqd9vSe3hVbazyoohOwmdD 2024-02-01T10:20:51.137 DEBUG:boto:Data: 2024-02-01T10:20:51.137 DEBUG:boto:Headers: {} 2024-02-01T10:20:51.138 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:20:51.138 DEBUG:boto:Port: 8000 2024-02-01T10:20:51.138 DEBUG:boto:Params: {} 2024-02-01T10:20:51.138 DEBUG:boto:Token: None 2024-02-01T10:20:51.138 DEBUG:boto:StringToSign: DELETE Thu, 01 Feb 2024 10:20:51 GMT /vclfcn-18/obj-test-zone2?versionId=dRFINd5C3wqd9vSe3hVbazyoohOwmdD 2024-02-01T10:20:51.138 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ml4OSSyJ3IMiVb3D3300dA1ULts= 2024-02-01T10:20:51.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:20:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ml4OSSyJ3IMiVb3D3300dA1ULts=', 'Content-Length': '0'} 2024-02-01T10:20:51.161 DEBUG:boto:Response headers: [('x-amz-version-id', 'dRFINd5C3wqd9vSe3hVbazyoohOwmdD'), ('x-amz-request-id', 'tx0000010e7aff3f2030b85-0065bb7083-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:20:51.161 DEBUG:boto:path=/vclfcn-18/obj-test-zone2 2024-02-01T10:20:51.161 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2 2024-02-01T10:20:51.161 DEBUG:boto:Method: PUT 2024-02-01T10:20:51.161 DEBUG:boto:Path: /vclfcn-18/obj-test-zone2 2024-02-01T10:20:51.162 DEBUG:boto:Data: 2024-02-01T10:20:51.162 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-02-01T10:20:51.162 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:20:51.162 DEBUG:boto:Port: 8000 2024-02-01T10:20:51.162 DEBUG:boto:Params: {} 2024-02-01T10:20:51.162 DEBUG:boto:Token: None 2024-02-01T10:20:51.162 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Thu, 01 Feb 2024 10:20:51 GMT /vclfcn-18/obj-test-zone2 2024-02-01T10:20:51.162 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:OOAFP79iK0lJlUEFvuYTipgL7KQ= 2024-02-01T10:20:51.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:20:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:OOAFP79iK0lJlUEFvuYTipgL7KQ='} 2024-02-01T10:20:51.208 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'WNrRIC4PveTYI6ueqAUrzMwTE2J7EAJ'), ('x-amz-request-id', 'tx0000038f3a28ba29b5e2f-0065bb7083-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:20:51.209 DEBUG:rgw_multi.tests:version3 id=WNrRIC4PveTYI6ueqAUrzMwTE2J7EAJ 2024-02-01T10:20:51.209 DEBUG:boto:path=/vclfcn-18/obj-test-zone2 2024-02-01T10:20:51.209 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2 2024-02-01T10:20:51.209 DEBUG:boto:path=/vclfcn-18/obj-test-zone2?versionId=WNrRIC4PveTYI6ueqAUrzMwTE2J7EAJ 2024-02-01T10:20:51.209 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2?versionId=WNrRIC4PveTYI6ueqAUrzMwTE2J7EAJ 2024-02-01T10:20:51.209 DEBUG:boto:Method: DELETE 2024-02-01T10:20:51.209 DEBUG:boto:Path: /vclfcn-18/obj-test-zone2?versionId=WNrRIC4PveTYI6ueqAUrzMwTE2J7EAJ 2024-02-01T10:20:51.209 DEBUG:boto:Data: 2024-02-01T10:20:51.209 DEBUG:boto:Headers: {} 2024-02-01T10:20:51.209 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:20:51.210 DEBUG:boto:Port: 8000 2024-02-01T10:20:51.210 DEBUG:boto:Params: {} 2024-02-01T10:20:51.210 DEBUG:boto:Token: None 2024-02-01T10:20:51.210 DEBUG:boto:StringToSign: DELETE Thu, 01 Feb 2024 10:20:51 GMT /vclfcn-18/obj-test-zone2?versionId=WNrRIC4PveTYI6ueqAUrzMwTE2J7EAJ 2024-02-01T10:20:51.210 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NF6VeJtWoF9f3THSTn9zlTQowL4= 2024-02-01T10:20:51.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:20:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NF6VeJtWoF9f3THSTn9zlTQowL4=', 'Content-Length': '0'} 2024-02-01T10:20:51.233 DEBUG:boto:Response headers: [('x-amz-version-id', 'WNrRIC4PveTYI6ueqAUrzMwTE2J7EAJ'), ('x-amz-request-id', 'tx000005422e8d4af3d67d2-0065bb7083-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:20:51.234 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:20:51.234 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:20:51.234 DEBUG:boto:Method: HEAD 2024-02-01T10:20:51.234 DEBUG:boto:Path: /vclfcn-18/ 2024-02-01T10:20:51.234 DEBUG:boto:Data: 2024-02-01T10:20:51.234 DEBUG:boto:Headers: {} 2024-02-01T10:20:51.234 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:20:51.234 DEBUG:boto:Port: 8001 2024-02-01T10:20:51.234 DEBUG:boto:Params: {} 2024-02-01T10:20:51.235 DEBUG:boto:Token: None 2024-02-01T10:20:51.235 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:20:51 GMT /vclfcn-18/ 2024-02-01T10:20:51.235 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:w4jCQC9tMU/EU1n5E5Z0Etq9WbQ= 2024-02-01T10:20:51.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:20:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:w4jCQC9tMU/EU1n5E5Z0Etq9WbQ=', 'Content-Length': '0'} 2024-02-01T10:20:51.239 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000084c79af58600bf34-0065bb7083-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:20:51.239 DEBUG:boto:path=/vclfcn-18/obj-test-zone3 2024-02-01T10:20:51.239 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3 2024-02-01T10:20:51.240 DEBUG:boto:Method: PUT 2024-02-01T10:20:51.240 DEBUG:boto:Path: /vclfcn-18/obj-test-zone3 2024-02-01T10:20:51.240 DEBUG:boto:Data: 2024-02-01T10:20:51.240 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-02-01T10:20:51.240 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:20:51.240 DEBUG:boto:Port: 8001 2024-02-01T10:20:51.240 DEBUG:boto:Params: {} 2024-02-01T10:20:51.240 DEBUG:boto:Token: None 2024-02-01T10:20:51.240 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 01 Feb 2024 10:20:51 GMT /vclfcn-18/obj-test-zone3 2024-02-01T10:20:51.240 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:EtlhyPey0AZnc2Tinxkq1LaRSXs= 2024-02-01T10:20:51.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:20:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:EtlhyPey0AZnc2Tinxkq1LaRSXs='} 2024-02-01T10:20:51.262 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N'), ('x-amz-request-id', 'tx00000187ab74edf46c498-0065bb7083-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:20:51.263 DEBUG:rgw_multi.tests:version1 id=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:20:51.263 DEBUG:boto:path=/vclfcn-18/obj-test-zone3 2024-02-01T10:20:51.263 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3 2024-02-01T10:20:51.263 DEBUG:boto:Method: PUT 2024-02-01T10:20:51.263 DEBUG:boto:Path: /vclfcn-18/obj-test-zone3 2024-02-01T10:20:51.263 DEBUG:boto:Data: 2024-02-01T10:20:51.263 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-02-01T10:20:51.263 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:20:51.263 DEBUG:boto:Port: 8001 2024-02-01T10:20:51.264 DEBUG:boto:Params: {} 2024-02-01T10:20:51.264 DEBUG:boto:Token: None 2024-02-01T10:20:51.264 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Thu, 01 Feb 2024 10:20:51 GMT /vclfcn-18/obj-test-zone3 2024-02-01T10:20:51.264 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8cubHq7LS2OV5A2N1bHE5pT8hO8= 2024-02-01T10:20:51.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:20:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8cubHq7LS2OV5A2N1bHE5pT8hO8='} 2024-02-01T10:20:51.308 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'kKilFRz4njfKpdTjqgJnG1JFj2dNxmQ'), ('x-amz-request-id', 'tx00000d8171db6cae6aa0a-0065bb7083-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:20:51.308 DEBUG:rgw_multi.tests:version2 id=kKilFRz4njfKpdTjqgJnG1JFj2dNxmQ 2024-02-01T10:20:51.309 DEBUG:boto:path=/vclfcn-18/obj-test-zone3 2024-02-01T10:20:51.309 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3 2024-02-01T10:20:51.309 DEBUG:boto:path=/vclfcn-18/obj-test-zone3?versionId=kKilFRz4njfKpdTjqgJnG1JFj2dNxmQ 2024-02-01T10:20:51.309 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3?versionId=kKilFRz4njfKpdTjqgJnG1JFj2dNxmQ 2024-02-01T10:20:51.309 DEBUG:boto:Method: DELETE 2024-02-01T10:20:51.309 DEBUG:boto:Path: /vclfcn-18/obj-test-zone3?versionId=kKilFRz4njfKpdTjqgJnG1JFj2dNxmQ 2024-02-01T10:20:51.309 DEBUG:boto:Data: 2024-02-01T10:20:51.309 DEBUG:boto:Headers: {} 2024-02-01T10:20:51.309 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:20:51.309 DEBUG:boto:Port: 8001 2024-02-01T10:20:51.310 DEBUG:boto:Params: {} 2024-02-01T10:20:51.310 DEBUG:boto:Token: None 2024-02-01T10:20:51.310 DEBUG:boto:StringToSign: DELETE Thu, 01 Feb 2024 10:20:51 GMT /vclfcn-18/obj-test-zone3?versionId=kKilFRz4njfKpdTjqgJnG1JFj2dNxmQ 2024-02-01T10:20:51.310 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:MPk9qY+ncEL/YJVUIf9RX+7mkHE= 2024-02-01T10:20:51.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:20:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:MPk9qY+ncEL/YJVUIf9RX+7mkHE=', 'Content-Length': '0'} 2024-02-01T10:20:51.332 DEBUG:boto:Response headers: [('x-amz-version-id', 'kKilFRz4njfKpdTjqgJnG1JFj2dNxmQ'), ('x-amz-request-id', 'tx00000f5e943f9733a20ff-0065bb7083-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:20:51.332 DEBUG:boto:path=/vclfcn-18/obj-test-zone3 2024-02-01T10:20:51.333 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3 2024-02-01T10:20:51.333 DEBUG:boto:Method: PUT 2024-02-01T10:20:51.333 DEBUG:boto:Path: /vclfcn-18/obj-test-zone3 2024-02-01T10:20:51.333 DEBUG:boto:Data: 2024-02-01T10:20:51.333 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-02-01T10:20:51.333 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:20:51.333 DEBUG:boto:Port: 8001 2024-02-01T10:20:51.333 DEBUG:boto:Params: {} 2024-02-01T10:20:51.333 DEBUG:boto:Token: None 2024-02-01T10:20:51.334 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Thu, 01 Feb 2024 10:20:51 GMT /vclfcn-18/obj-test-zone3 2024-02-01T10:20:51.334 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/KD/x9m0r+mqDwJzhfBcIy1FV/A= 2024-02-01T10:20:51.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:20:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/KD/x9m0r+mqDwJzhfBcIy1FV/A='} 2024-02-01T10:20:51.355 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'kw4YpbdbX4jOow1c4LPEFQ833gDZFX3'), ('x-amz-request-id', 'tx000004ac7faf07338ee52-0065bb7083-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:20:51.355 DEBUG:rgw_multi.tests:version3 id=kw4YpbdbX4jOow1c4LPEFQ833gDZFX3 2024-02-01T10:20:51.356 DEBUG:boto:path=/vclfcn-18/obj-test-zone3 2024-02-01T10:20:51.356 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3 2024-02-01T10:20:51.356 DEBUG:boto:path=/vclfcn-18/obj-test-zone3?versionId=kw4YpbdbX4jOow1c4LPEFQ833gDZFX3 2024-02-01T10:20:51.356 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3?versionId=kw4YpbdbX4jOow1c4LPEFQ833gDZFX3 2024-02-01T10:20:51.356 DEBUG:boto:Method: DELETE 2024-02-01T10:20:51.356 DEBUG:boto:Path: /vclfcn-18/obj-test-zone3?versionId=kw4YpbdbX4jOow1c4LPEFQ833gDZFX3 2024-02-01T10:20:51.356 DEBUG:boto:Data: 2024-02-01T10:20:51.356 DEBUG:boto:Headers: {} 2024-02-01T10:20:51.356 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:20:51.356 DEBUG:boto:Port: 8001 2024-02-01T10:20:51.357 DEBUG:boto:Params: {} 2024-02-01T10:20:51.357 DEBUG:boto:Token: None 2024-02-01T10:20:51.357 DEBUG:boto:StringToSign: DELETE Thu, 01 Feb 2024 10:20:51 GMT /vclfcn-18/obj-test-zone3?versionId=kw4YpbdbX4jOow1c4LPEFQ833gDZFX3 2024-02-01T10:20:51.357 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:uOjfzRHmgrPflqkyZhIW3mRa/tI= 2024-02-01T10:20:51.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:20:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:uOjfzRHmgrPflqkyZhIW3mRa/tI=', 'Content-Length': '0'} 2024-02-01T10:20:51.379 DEBUG:boto:Response headers: [('x-amz-version-id', 'kw4YpbdbX4jOow1c4LPEFQ833gDZFX3'), ('x-amz-request-id', 'tx000002b96aeaa34f02b25-0065bb7083-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:20:51.380 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=vclfcn-16 2024-02-01T10:20:51.380 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:20:51.380 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:20:51.380 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket vclfcn-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-02-01T10:20:51.432 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:20:51.432 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:20:51.496 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:20:51.493+0000 7fb0d2cf2980 1 waiting for incremental sync to catch up: 2024-02-01T10:20:51.496 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, , , , , 00000000005.23.10, , , , , ] 2024-02-01T10:20:51.496 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, 00000000010.30.6, , , , 00000000005.23.10, , , , , ] 2024-02-01T10:20:56.499 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:20:56.497+0000 7fb0d2cf2980 1 waiting for incremental sync to catch up: 2024-02-01T10:20:56.499 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, , , , , 00000000005.23.10, , , , , ] 2024-02-01T10:20:56.499 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, 00000000010.30.6, , , , 00000000005.23.10, , , , , ] 2024-02-01T10:21:01.501 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:21:01.501+0000 7fb0d2cf2980 1 waiting for incremental sync to catch up: 2024-02-01T10:21:01.501 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, , , , , 00000000005.23.10, , , , , ] 2024-02-01T10:21:01.501 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, 00000000010.30.6, , , , 00000000005.23.10, , , , , ] 2024-02-01T10:21:06.503 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:21:06.501+0000 7fb0d2cf2980 1 waiting for incremental sync to catch up: 2024-02-01T10:21:06.504 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, , , , , 00000000005.23.10, , , , , ] 2024-02-01T10:21:06.504 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, 00000000010.30.6, , , , 00000000005.23.10, , , , , ] 2024-02-01T10:21:11.506 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:21:11.505+0000 7fb0d2cf2980 1 bucket sync caught up with source: 2024-02-01T10:21:11.507 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, 00000000010.30.6, , 00000000003.20.10, , 00000000007.27.11, , , , , ] 2024-02-01T10:21:11.507 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, 00000000010.30.6, , , , 00000000005.23.10, , , , , ] 2024-02-01T10:21:11.507 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:21:11.505+0000 7fb0d2cf2980 0 bucket checkpoint complete 2024-02-01T10:21:11.550 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=vclfcn-16 2024-02-01T10:21:11.550 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:21:11.550 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:21:11.551 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:21:11.602 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:21:11.602 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:21:11.662 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:21:11.665+0000 7fa4641fd980 1 waiting for incremental sync to catch up: 2024-02-01T10:21:11.662 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, 00000000010.30.6, , , , 00000000005.23.10, , , , , ] 2024-02-01T10:21:11.662 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, 00000000010.30.6, , 00000000003.20.10, , 00000000007.27.11, , , , , ] 2024-02-01T10:21:16.665 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:21:16.669+0000 7fa4641fd980 1 bucket sync caught up with source: 2024-02-01T10:21:16.666 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, 00000000010.30.6, , 00000000003.20.10, , 00000000007.27.11, , , , , ] 2024-02-01T10:21:16.666 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, 00000000010.30.6, , 00000000003.20.10, , 00000000007.27.11, , , , , ] 2024-02-01T10:21:16.666 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:21:16.669+0000 7fa4641fd980 0 bucket checkpoint complete 2024-02-01T10:21:16.702 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=vclfcn-16 2024-02-01T10:21:16.702 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:21:16.702 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:21:16.703 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:21:16.750 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:21:16.751 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:21:16.816 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:21:16.817+0000 7f05048f2980 1 waiting for incremental sync to catch up: 2024-02-01T10:21:16.816 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, , , 00000000010.31.6, , 00000000004.25.10, , , , , ] 2024-02-01T10:21:16.816 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, 00000000003.22.10, , 00000000010.31.6, , 00000000007.31.10, , , , , ] 2024-02-01T10:21:21.818 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:21:21.821+0000 7f05048f2980 1 waiting for incremental sync to catch up: 2024-02-01T10:21:21.818 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, , , 00000000010.31.6, , 00000000004.25.10, , , , , ] 2024-02-01T10:21:21.819 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, 00000000003.22.10, , 00000000010.31.6, , 00000000007.31.10, , , , , ] 2024-02-01T10:21:26.821 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:21:26.821+0000 7f05048f2980 1 bucket sync caught up with source: 2024-02-01T10:21:26.821 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, 00000000003.22.10, , 00000000010.31.6, , 00000000007.31.10, , , , , ] 2024-02-01T10:21:26.821 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, 00000000003.22.10, , 00000000010.31.6, , 00000000007.31.10, , , , , ] 2024-02-01T10:21:26.821 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:21:26.821+0000 7f05048f2980 0 bucket checkpoint complete 2024-02-01T10:21:26.866 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=vclfcn-16 2024-02-01T10:21:26.866 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:21:26.866 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:21:26.867 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:21:26.916 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:21:26.916 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:21:26.979 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:21:26.981+0000 7f51f9527980 1 waiting for incremental sync to catch up: 2024-02-01T10:21:26.979 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, 00000000003.22.10, , 00000000010.31.6, , 00000000004.25.10, , , , , ] 2024-02-01T10:21:26.979 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, 00000000003.22.10, , 00000000010.31.6, , 00000000007.31.10, , , , , ] 2024-02-01T10:21:31.981 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:21:31.985+0000 7f51f9527980 1 waiting for incremental sync to catch up: 2024-02-01T10:21:31.981 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, 00000000003.22.10, , 00000000010.31.6, , 00000000004.25.10, , , , , ] 2024-02-01T10:21:31.981 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, 00000000003.22.10, , 00000000010.31.6, , 00000000007.31.10, , , , , ] 2024-02-01T10:21:36.983 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:21:36.985+0000 7f51f9527980 1 bucket sync caught up with source: 2024-02-01T10:21:36.983 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, 00000000003.22.10, , 00000000010.31.6, , 00000000007.31.10, , , , , ] 2024-02-01T10:21:36.984 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, 00000000003.22.10, , 00000000010.31.6, , 00000000007.31.10, , , , , ] 2024-02-01T10:21:36.984 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:21:36.985+0000 7f51f9527980 0 bucket checkpoint complete 2024-02-01T10:21:37.020 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=vclfcn-16 2024-02-01T10:21:37.020 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:21:37.020 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:21:37.020 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:21:37.069 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:21:37.070 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:21:37.130 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:21:37.133+0000 7fdfa904f980 1 bucket sync caught up with source: 2024-02-01T10:21:37.130 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, 00000000003.50.10, , 00000000003.28.10, , 00000000014.59.6, , , , , ] 2024-02-01T10:21:37.131 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, 00000000003.50.10, , 00000000003.28.10, , 00000000014.59.6, , , , , ] 2024-02-01T10:21:37.131 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:21:37.133+0000 7fdfa904f980 0 bucket checkpoint complete 2024-02-01T10:21:37.153 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=vclfcn-16 2024-02-01T10:21:37.153 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:21:37.153 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:21:37.154 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket vclfcn-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-02-01T10:21:37.204 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:21:37.204 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:21:37.268 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:21:37.265+0000 7fe8a34ae980 1 bucket sync caught up with source: 2024-02-01T10:21:37.268 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, 00000000003.50.10, , 00000000003.28.10, , 00000000014.59.6, , , , , ] 2024-02-01T10:21:37.268 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, 00000000003.50.10, , 00000000003.28.10, , 00000000014.59.6, , , , , ] 2024-02-01T10:21:37.268 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:21:37.265+0000 7fe8a34ae980 0 bucket checkpoint complete 2024-02-01T10:21:37.290 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-16 zones={test-zone2, test-zone1} 2024-02-01T10:21:37.291 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:21:37.291 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:21:37.291 DEBUG:boto:Method: HEAD 2024-02-01T10:21:37.291 DEBUG:boto:Path: /vclfcn-16/ 2024-02-01T10:21:37.291 DEBUG:boto:Data: 2024-02-01T10:21:37.291 DEBUG:boto:Headers: {} 2024-02-01T10:21:37.291 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:21:37.292 DEBUG:boto:Port: 8000 2024-02-01T10:21:37.292 DEBUG:boto:Params: {} 2024-02-01T10:21:37.292 DEBUG:boto:Token: None 2024-02-01T10:21:37.292 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:21:37 GMT /vclfcn-16/ 2024-02-01T10:21:37.293 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ChhmF/OD2qc2j8lJDoXk6ur9fcY= 2024-02-01T10:21:37.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ChhmF/OD2qc2j8lJDoXk6ur9fcY=', 'Content-Length': '0'} 2024-02-01T10:21:37.295 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003b30a091fe39e036-0065bb70b1-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:21:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:37.295 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:21:37.295 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:21:37.296 DEBUG:boto:Method: HEAD 2024-02-01T10:21:37.296 DEBUG:boto:Path: /vclfcn-16/ 2024-02-01T10:21:37.296 DEBUG:boto:Data: 2024-02-01T10:21:37.296 DEBUG:boto:Headers: {} 2024-02-01T10:21:37.296 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:21:37.296 DEBUG:boto:Port: 8000 2024-02-01T10:21:37.296 DEBUG:boto:Params: {} 2024-02-01T10:21:37.296 DEBUG:boto:Token: None 2024-02-01T10:21:37.296 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:21:37 GMT /vclfcn-16/ 2024-02-01T10:21:37.296 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ChhmF/OD2qc2j8lJDoXk6ur9fcY= 2024-02-01T10:21:37.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ChhmF/OD2qc2j8lJDoXk6ur9fcY=', 'Content-Length': '0'} 2024-02-01T10:21:37.300 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a0907e9eda926733-0065bb70b1-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:21:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:37.301 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:21:37.301 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:21:37.301 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:21:37.301 DEBUG:boto:path=/vclfcn-16/?versions&max-keys=999 2024-02-01T10:21:37.301 DEBUG:boto:auth_path=/vclfcn-16/?versions&max-keys=999 2024-02-01T10:21:37.301 DEBUG:boto:Method: GET 2024-02-01T10:21:37.301 DEBUG:boto:Path: /vclfcn-16/?versions&max-keys=999 2024-02-01T10:21:37.301 DEBUG:boto:Data: 2024-02-01T10:21:37.301 DEBUG:boto:Headers: {} 2024-02-01T10:21:37.301 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:21:37.302 DEBUG:boto:Port: 8000 2024-02-01T10:21:37.302 DEBUG:boto:Params: {} 2024-02-01T10:21:37.302 DEBUG:boto:Token: None 2024-02-01T10:21:37.302 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:37 GMT /vclfcn-16/?versions 2024-02-01T10:21:37.302 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:f9s/44lnT6us5BhOKyMN3VSh1lo= 2024-02-01T10:21:37.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:f9s/44lnT6us5BhOKyMN3VSh1lo=', 'Content-Length': '0'} 2024-02-01T10:21:37.304 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000553eb14638d7afea-0065bb70b1-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:37.305 DEBUG:boto:b'vclfcn-16999falsedummyrQRpBowIFAsvxVDn92H30shYnKoGt4ntrue2024-02-01T10:19:58.958Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1lS9-sMOvyR-VcD.47BkPLymGmIyfWmftrue2024-02-01T10:20:49.965Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2mE6urSxcrRA9wdyuToa41.w2aTia9p1true2024-02-01T10:20:50.122Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjPtrue2024-02-01T10:20:50.297Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:21:37.305 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-02-01T10:21:37.306 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-02-01T10:21:37.306 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-02-01T10:21:37.306 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-02-01T10:21:37.306 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:21:37.306 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:21:37.306 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:21:37.306 DEBUG:boto:path=/vclfcn-16/?versions&max-keys=999 2024-02-01T10:21:37.306 DEBUG:boto:auth_path=/vclfcn-16/?versions&max-keys=999 2024-02-01T10:21:37.306 DEBUG:boto:Method: GET 2024-02-01T10:21:37.306 DEBUG:boto:Path: /vclfcn-16/?versions&max-keys=999 2024-02-01T10:21:37.307 DEBUG:boto:Data: 2024-02-01T10:21:37.307 DEBUG:boto:Headers: {} 2024-02-01T10:21:37.307 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:21:37.307 DEBUG:boto:Port: 8000 2024-02-01T10:21:37.307 DEBUG:boto:Params: {} 2024-02-01T10:21:37.307 DEBUG:boto:Token: None 2024-02-01T10:21:37.307 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:37 GMT /vclfcn-16/?versions 2024-02-01T10:21:37.307 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:f9s/44lnT6us5BhOKyMN3VSh1lo= 2024-02-01T10:21:37.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:f9s/44lnT6us5BhOKyMN3VSh1lo=', 'Content-Length': '0'} 2024-02-01T10:21:37.311 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7cc247597ad742e-0065bb70b1-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:37.311 DEBUG:boto:b'vclfcn-16999falsedummyrQRpBowIFAsvxVDn92H30shYnKoGt4ntrue2024-02-01T10:19:58.958Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1lS9-sMOvyR-VcD.47BkPLymGmIyfWmftrue2024-02-01T10:20:49.965Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2mE6urSxcrRA9wdyuToa41.w2aTia9p1true2024-02-01T10:20:50.122Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjPtrue2024-02-01T10:20:50.297Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:21:37.312 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-02-01T10:21:37.312 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-02-01T10:21:37.312 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-02-01T10:21:37.312 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-02-01T10:21:37.312 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:21:37.312 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:21:37.312 DEBUG:boto:path=/vclfcn-16/?versions&max-keys=999 2024-02-01T10:21:37.313 DEBUG:boto:auth_path=/vclfcn-16/?versions&max-keys=999 2024-02-01T10:21:37.313 DEBUG:boto:Method: GET 2024-02-01T10:21:37.313 DEBUG:boto:Path: /vclfcn-16/?versions&max-keys=999 2024-02-01T10:21:37.313 DEBUG:boto:Data: 2024-02-01T10:21:37.313 DEBUG:boto:Headers: {} 2024-02-01T10:21:37.313 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:21:37.313 DEBUG:boto:Port: 8000 2024-02-01T10:21:37.313 DEBUG:boto:Params: {} 2024-02-01T10:21:37.313 DEBUG:boto:Token: None 2024-02-01T10:21:37.313 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:37 GMT /vclfcn-16/?versions 2024-02-01T10:21:37.313 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:f9s/44lnT6us5BhOKyMN3VSh1lo= 2024-02-01T10:21:37.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:f9s/44lnT6us5BhOKyMN3VSh1lo=', 'Content-Length': '0'} 2024-02-01T10:21:37.316 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000506067b710f1f8f0-0065bb70b1-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:37.316 DEBUG:boto:b'vclfcn-16999falsedummyrQRpBowIFAsvxVDn92H30shYnKoGt4ntrue2024-02-01T10:19:58.958Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1lS9-sMOvyR-VcD.47BkPLymGmIyfWmftrue2024-02-01T10:20:49.965Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2mE6urSxcrRA9wdyuToa41.w2aTia9p1true2024-02-01T10:20:50.122Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjPtrue2024-02-01T10:20:50.297Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:21:37.317 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:21:37.317 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:21:37.317 DEBUG:boto:path=/vclfcn-16/?versions&max-keys=999 2024-02-01T10:21:37.317 DEBUG:boto:auth_path=/vclfcn-16/?versions&max-keys=999 2024-02-01T10:21:37.317 DEBUG:boto:Method: GET 2024-02-01T10:21:37.317 DEBUG:boto:Path: /vclfcn-16/?versions&max-keys=999 2024-02-01T10:21:37.317 DEBUG:boto:Data: 2024-02-01T10:21:37.317 DEBUG:boto:Headers: {} 2024-02-01T10:21:37.317 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:21:37.317 DEBUG:boto:Port: 8000 2024-02-01T10:21:37.317 DEBUG:boto:Params: {} 2024-02-01T10:21:37.318 DEBUG:boto:Token: None 2024-02-01T10:21:37.318 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:37 GMT /vclfcn-16/?versions 2024-02-01T10:21:37.318 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:f9s/44lnT6us5BhOKyMN3VSh1lo= 2024-02-01T10:21:37.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:f9s/44lnT6us5BhOKyMN3VSh1lo=', 'Content-Length': '0'} 2024-02-01T10:21:37.322 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004a69a0518f95add-0065bb70b1-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:37.322 DEBUG:boto:b'vclfcn-16999falsedummyrQRpBowIFAsvxVDn92H30shYnKoGt4ntrue2024-02-01T10:19:58.958Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1lS9-sMOvyR-VcD.47BkPLymGmIyfWmftrue2024-02-01T10:20:49.965Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2mE6urSxcrRA9wdyuToa41.w2aTia9p1true2024-02-01T10:20:50.122Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjPtrue2024-02-01T10:20:50.297Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:21:37.323 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-02-01T10:21:37.323 DEBUG:boto:path=/vclfcn-16/dummy 2024-02-01T10:21:37.323 DEBUG:boto:auth_path=/vclfcn-16/dummy 2024-02-01T10:21:37.323 DEBUG:boto:path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:37.323 DEBUG:boto:auth_path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:37.324 DEBUG:boto:Method: GET 2024-02-01T10:21:37.324 DEBUG:boto:Path: /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:37.324 DEBUG:boto:Data: 2024-02-01T10:21:37.324 DEBUG:boto:Headers: {} 2024-02-01T10:21:37.324 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:21:37.324 DEBUG:boto:Port: 8000 2024-02-01T10:21:37.324 DEBUG:boto:Params: {} 2024-02-01T10:21:37.324 DEBUG:boto:Token: None 2024-02-01T10:21:37.327 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:37 GMT /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:37.327 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:4XKc+VaY6TvoI4LaJTkYhHyc/uo= 2024-02-01T10:21:37.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:4XKc+VaY6TvoI4LaJTkYhHyc/uo=', 'Content-Length': '0'} 2024-02-01T10:21:37.329 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:19:58 GMT'), ('x-amz-version-id', 'rQRpBowIFAsvxVDn92H30shYnKoGt4n'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000909564f63ddec66-0065bb70b1-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:37.329 DEBUG:boto:path=/vclfcn-16/dummy 2024-02-01T10:21:37.329 DEBUG:boto:auth_path=/vclfcn-16/dummy 2024-02-01T10:21:37.329 DEBUG:boto:path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:37.330 DEBUG:boto:auth_path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:37.330 DEBUG:boto:Method: GET 2024-02-01T10:21:37.330 DEBUG:boto:Path: /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:37.330 DEBUG:boto:Data: 2024-02-01T10:21:37.330 DEBUG:boto:Headers: {} 2024-02-01T10:21:37.330 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:21:37.330 DEBUG:boto:Port: 8000 2024-02-01T10:21:37.330 DEBUG:boto:Params: {} 2024-02-01T10:21:37.330 DEBUG:boto:Token: None 2024-02-01T10:21:37.330 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:37 GMT /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:37.331 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:4XKc+VaY6TvoI4LaJTkYhHyc/uo= 2024-02-01T10:21:37.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:4XKc+VaY6TvoI4LaJTkYhHyc/uo=', 'Content-Length': '0'} 2024-02-01T10:21:37.333 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:19:58 GMT'), ('x-amz-version-id', 'rQRpBowIFAsvxVDn92H30shYnKoGt4n'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004dc2dded31e5c3dd-0065bb70b1-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:37.334 DEBUG:boto:path=/vclfcn-16/dummy 2024-02-01T10:21:37.334 DEBUG:boto:auth_path=/vclfcn-16/dummy 2024-02-01T10:21:37.334 DEBUG:boto:path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:37.334 DEBUG:boto:auth_path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:37.334 DEBUG:boto:Method: HEAD 2024-02-01T10:21:37.334 DEBUG:boto:Path: /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:37.334 DEBUG:boto:Data: 2024-02-01T10:21:37.334 DEBUG:boto:Headers: {} 2024-02-01T10:21:37.334 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:21:37.335 DEBUG:boto:Port: 8000 2024-02-01T10:21:37.335 DEBUG:boto:Params: {} 2024-02-01T10:21:37.335 DEBUG:boto:Token: None 2024-02-01T10:21:37.335 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:21:37 GMT /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:37.335 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:KPAsykN28gAq0AgrC9/ptZyAqAY= 2024-02-01T10:21:37.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:KPAsykN28gAq0AgrC9/ptZyAqAY=', 'Content-Length': '0'} 2024-02-01T10:21:37.338 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:19:58 GMT'), ('x-amz-version-id', 'rQRpBowIFAsvxVDn92H30shYnKoGt4n'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c48e7deb3ba30108-0065bb70b1-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:37.338 DEBUG:boto:path=/vclfcn-16/dummy 2024-02-01T10:21:37.338 DEBUG:boto:auth_path=/vclfcn-16/dummy 2024-02-01T10:21:37.338 DEBUG:boto:path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:37.338 DEBUG:boto:auth_path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:37.338 DEBUG:boto:Method: HEAD 2024-02-01T10:21:37.338 DEBUG:boto:Path: /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:37.338 DEBUG:boto:Data: 2024-02-01T10:21:37.339 DEBUG:boto:Headers: {} 2024-02-01T10:21:37.339 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:21:37.339 DEBUG:boto:Port: 8000 2024-02-01T10:21:37.339 DEBUG:boto:Params: {} 2024-02-01T10:21:37.339 DEBUG:boto:Token: None 2024-02-01T10:21:37.339 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:21:37 GMT /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:37.339 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:KPAsykN28gAq0AgrC9/ptZyAqAY= 2024-02-01T10:21:37.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:KPAsykN28gAq0AgrC9/ptZyAqAY=', 'Content-Length': '0'} 2024-02-01T10:21:37.342 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:19:58 GMT'), ('x-amz-version-id', 'rQRpBowIFAsvxVDn92H30shYnKoGt4n'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003cc2ce0dbfc17bab-0065bb70b1-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:37.342 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-02-01T10:21:37.342 DEBUG:boto:path=/vclfcn-16/dummy 2024-02-01T10:21:37.342 DEBUG:boto:auth_path=/vclfcn-16/dummy 2024-02-01T10:21:37.342 DEBUG:boto:path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:37.342 DEBUG:boto:auth_path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:37.342 DEBUG:boto:Method: GET 2024-02-01T10:21:37.343 DEBUG:boto:Path: /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:37.343 DEBUG:boto:Data: 2024-02-01T10:21:37.343 DEBUG:boto:Headers: {} 2024-02-01T10:21:37.343 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:21:37.343 DEBUG:boto:Port: 8000 2024-02-01T10:21:37.343 DEBUG:boto:Params: {} 2024-02-01T10:21:37.343 DEBUG:boto:Token: None 2024-02-01T10:21:37.343 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:37 GMT /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:37.343 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:4XKc+VaY6TvoI4LaJTkYhHyc/uo= 2024-02-01T10:21:37.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:4XKc+VaY6TvoI4LaJTkYhHyc/uo=', 'Content-Length': '0'} 2024-02-01T10:21:37.346 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:19:58 GMT'), ('x-amz-version-id', 'rQRpBowIFAsvxVDn92H30shYnKoGt4n'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a425720c1c31d107-0065bb70b1-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:37.346 DEBUG:boto:path=/vclfcn-16/dummy 2024-02-01T10:21:37.346 DEBUG:boto:auth_path=/vclfcn-16/dummy 2024-02-01T10:21:37.346 DEBUG:boto:path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:37.346 DEBUG:boto:auth_path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:37.347 DEBUG:boto:Method: GET 2024-02-01T10:21:37.347 DEBUG:boto:Path: /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:37.347 DEBUG:boto:Data: 2024-02-01T10:21:37.347 DEBUG:boto:Headers: {} 2024-02-01T10:21:37.347 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:21:37.347 DEBUG:boto:Port: 8000 2024-02-01T10:21:37.347 DEBUG:boto:Params: {} 2024-02-01T10:21:37.347 DEBUG:boto:Token: None 2024-02-01T10:21:37.347 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:37 GMT /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:37.347 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:4XKc+VaY6TvoI4LaJTkYhHyc/uo= 2024-02-01T10:21:37.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:4XKc+VaY6TvoI4LaJTkYhHyc/uo=', 'Content-Length': '0'} 2024-02-01T10:21:37.350 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:19:58 GMT'), ('x-amz-version-id', 'rQRpBowIFAsvxVDn92H30shYnKoGt4n'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b8a1016ac246f480-0065bb70b1-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:37.350 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:21:37.350 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:21:37.351 DEBUG:boto:path=/vclfcn-16/?prefix=dummy 2024-02-01T10:21:37.351 DEBUG:boto:auth_path=/vclfcn-16/?prefix=dummy 2024-02-01T10:21:37.351 DEBUG:boto:Method: GET 2024-02-01T10:21:37.351 DEBUG:boto:Path: /vclfcn-16/?prefix=dummy 2024-02-01T10:21:37.351 DEBUG:boto:Data: 2024-02-01T10:21:37.351 DEBUG:boto:Headers: {} 2024-02-01T10:21:37.351 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:21:37.351 DEBUG:boto:Port: 8000 2024-02-01T10:21:37.351 DEBUG:boto:Params: {} 2024-02-01T10:21:37.351 DEBUG:boto:Token: None 2024-02-01T10:21:37.351 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:37 GMT /vclfcn-16/ 2024-02-01T10:21:37.352 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mzT+qbk26MWtfo7PH5zZucVVOrY= 2024-02-01T10:21:37.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mzT+qbk26MWtfo7PH5zZucVVOrY=', 'Content-Length': '0'} 2024-02-01T10:21:37.355 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094a989de3a9119d9-0065bb70b1-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:37.396 DEBUG:boto:b'vclfcn-16dummy1000falsedummy2024-02-01T10:19:58.958Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:21:37.397 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:21:37.397 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:21:37.397 DEBUG:boto:path=/vclfcn-16/?prefix=dummy 2024-02-01T10:21:37.397 DEBUG:boto:auth_path=/vclfcn-16/?prefix=dummy 2024-02-01T10:21:37.397 DEBUG:boto:Method: GET 2024-02-01T10:21:37.398 DEBUG:boto:Path: /vclfcn-16/?prefix=dummy 2024-02-01T10:21:37.398 DEBUG:boto:Data: 2024-02-01T10:21:37.398 DEBUG:boto:Headers: {} 2024-02-01T10:21:37.398 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:21:37.398 DEBUG:boto:Port: 8000 2024-02-01T10:21:37.398 DEBUG:boto:Params: {} 2024-02-01T10:21:37.398 DEBUG:boto:Token: None 2024-02-01T10:21:37.398 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:37 GMT /vclfcn-16/ 2024-02-01T10:21:37.398 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mzT+qbk26MWtfo7PH5zZucVVOrY= 2024-02-01T10:21:37.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mzT+qbk26MWtfo7PH5zZucVVOrY=', 'Content-Length': '0'} 2024-02-01T10:21:37.402 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000022bb3533d657624e-0065bb70b1-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:37.403 DEBUG:boto:b'vclfcn-16dummy1000falsedummy2024-02-01T10:19:58.958Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:21:37.403 DEBUG:boto:path=/vclfcn-16/dummy 2024-02-01T10:21:37.403 DEBUG:boto:auth_path=/vclfcn-16/dummy 2024-02-01T10:21:37.403 DEBUG:boto:Method: HEAD 2024-02-01T10:21:37.403 DEBUG:boto:Path: /vclfcn-16/dummy 2024-02-01T10:21:37.403 DEBUG:boto:Data: 2024-02-01T10:21:37.403 DEBUG:boto:Headers: {} 2024-02-01T10:21:37.404 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:21:37.404 DEBUG:boto:Port: 8000 2024-02-01T10:21:37.404 DEBUG:boto:Params: {} 2024-02-01T10:21:37.404 DEBUG:boto:Token: None 2024-02-01T10:21:37.404 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:21:37 GMT /vclfcn-16/dummy 2024-02-01T10:21:37.404 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jfyR4DDTRZoiC/ruq81RVykTW2s= 2024-02-01T10:21:37.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jfyR4DDTRZoiC/ruq81RVykTW2s=', 'Content-Length': '0'} 2024-02-01T10:21:37.408 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:19:58 GMT'), ('x-amz-version-id', 'rQRpBowIFAsvxVDn92H30shYnKoGt4n'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002ce894beff596d92-0065bb70b1-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:37.408 DEBUG:boto:path=/vclfcn-16/dummy 2024-02-01T10:21:37.408 DEBUG:boto:auth_path=/vclfcn-16/dummy 2024-02-01T10:21:37.408 DEBUG:boto:Method: HEAD 2024-02-01T10:21:37.408 DEBUG:boto:Path: /vclfcn-16/dummy 2024-02-01T10:21:37.408 DEBUG:boto:Data: 2024-02-01T10:21:37.409 DEBUG:boto:Headers: {} 2024-02-01T10:21:37.409 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:21:37.409 DEBUG:boto:Port: 8000 2024-02-01T10:21:37.409 DEBUG:boto:Params: {} 2024-02-01T10:21:37.409 DEBUG:boto:Token: None 2024-02-01T10:21:37.409 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:21:37 GMT /vclfcn-16/dummy 2024-02-01T10:21:37.409 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jfyR4DDTRZoiC/ruq81RVykTW2s= 2024-02-01T10:21:37.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jfyR4DDTRZoiC/ruq81RVykTW2s=', 'Content-Length': '0'} 2024-02-01T10:21:37.413 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:19:58 GMT'), ('x-amz-version-id', 'rQRpBowIFAsvxVDn92H30shYnKoGt4n'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009344f5065b5389fb-0065bb70b1-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:37.413 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-02-01T10:21:37.413 DEBUG:boto:path=/vclfcn-16/dummy 2024-02-01T10:21:37.413 DEBUG:boto:auth_path=/vclfcn-16/dummy 2024-02-01T10:21:37.414 DEBUG:boto:path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:37.414 DEBUG:boto:auth_path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:37.414 DEBUG:boto:Method: GET 2024-02-01T10:21:37.414 DEBUG:boto:Path: /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:37.414 DEBUG:boto:Data: 2024-02-01T10:21:37.414 DEBUG:boto:Headers: {} 2024-02-01T10:21:37.414 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:21:37.414 DEBUG:boto:Port: 8000 2024-02-01T10:21:37.414 DEBUG:boto:Params: {} 2024-02-01T10:21:37.414 DEBUG:boto:Token: None 2024-02-01T10:21:37.414 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:37 GMT /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:37.415 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:4XKc+VaY6TvoI4LaJTkYhHyc/uo= 2024-02-01T10:21:37.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:4XKc+VaY6TvoI4LaJTkYhHyc/uo=', 'Content-Length': '0'} 2024-02-01T10:21:37.418 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:19:58 GMT'), ('x-amz-version-id', 'rQRpBowIFAsvxVDn92H30shYnKoGt4n'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000008caabfdc098ba71-0065bb70b1-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:37.418 DEBUG:boto:path=/vclfcn-16/dummy 2024-02-01T10:21:37.418 DEBUG:boto:auth_path=/vclfcn-16/dummy 2024-02-01T10:21:37.418 DEBUG:boto:path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:37.419 DEBUG:boto:auth_path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:37.419 DEBUG:boto:Method: GET 2024-02-01T10:21:37.419 DEBUG:boto:Path: /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:37.419 DEBUG:boto:Data: 2024-02-01T10:21:37.419 DEBUG:boto:Headers: {} 2024-02-01T10:21:37.419 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:21:37.419 DEBUG:boto:Port: 8000 2024-02-01T10:21:37.419 DEBUG:boto:Params: {} 2024-02-01T10:21:37.419 DEBUG:boto:Token: None 2024-02-01T10:21:37.419 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:37 GMT /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:37.420 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:4XKc+VaY6TvoI4LaJTkYhHyc/uo= 2024-02-01T10:21:37.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:4XKc+VaY6TvoI4LaJTkYhHyc/uo=', 'Content-Length': '0'} 2024-02-01T10:21:37.422 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:19:58 GMT'), ('x-amz-version-id', 'rQRpBowIFAsvxVDn92H30shYnKoGt4n'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000aa5d6a5b8c704ce3-0065bb70b1-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:37.423 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:21:37.423 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:21:37.423 DEBUG:boto:path=/vclfcn-16/?prefix=dummy 2024-02-01T10:21:37.423 DEBUG:boto:auth_path=/vclfcn-16/?prefix=dummy 2024-02-01T10:21:37.423 DEBUG:boto:Method: GET 2024-02-01T10:21:37.423 DEBUG:boto:Path: /vclfcn-16/?prefix=dummy 2024-02-01T10:21:37.423 DEBUG:boto:Data: 2024-02-01T10:21:37.423 DEBUG:boto:Headers: {} 2024-02-01T10:21:37.423 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:21:37.423 DEBUG:boto:Port: 8000 2024-02-01T10:21:37.423 DEBUG:boto:Params: {} 2024-02-01T10:21:37.424 DEBUG:boto:Token: None 2024-02-01T10:21:37.424 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:37 GMT /vclfcn-16/ 2024-02-01T10:21:37.424 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mzT+qbk26MWtfo7PH5zZucVVOrY= 2024-02-01T10:21:37.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mzT+qbk26MWtfo7PH5zZucVVOrY=', 'Content-Length': '0'} 2024-02-01T10:21:37.428 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dbbc177bc466afb9-0065bb70b1-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:37.428 DEBUG:boto:b'vclfcn-16dummy1000falsedummy2024-02-01T10:19:58.958Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:21:37.429 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:21:37.429 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:21:37.429 DEBUG:boto:path=/vclfcn-16/?prefix=dummy 2024-02-01T10:21:37.429 DEBUG:boto:auth_path=/vclfcn-16/?prefix=dummy 2024-02-01T10:21:37.429 DEBUG:boto:Method: GET 2024-02-01T10:21:37.429 DEBUG:boto:Path: /vclfcn-16/?prefix=dummy 2024-02-01T10:21:37.429 DEBUG:boto:Data: 2024-02-01T10:21:37.429 DEBUG:boto:Headers: {} 2024-02-01T10:21:37.429 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:21:37.430 DEBUG:boto:Port: 8000 2024-02-01T10:21:37.430 DEBUG:boto:Params: {} 2024-02-01T10:21:37.430 DEBUG:boto:Token: None 2024-02-01T10:21:37.430 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:37 GMT /vclfcn-16/ 2024-02-01T10:21:37.430 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mzT+qbk26MWtfo7PH5zZucVVOrY= 2024-02-01T10:21:37.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mzT+qbk26MWtfo7PH5zZucVVOrY=', 'Content-Length': '0'} 2024-02-01T10:21:37.434 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097a890a318bf9f38-0065bb70b1-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:37.434 DEBUG:boto:b'vclfcn-16dummy1000falsedummy2024-02-01T10:19:58.958Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:21:37.435 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-02-01T10:21:37.435 DEBUG:boto:path=/vclfcn-16/obj-test-zone1 2024-02-01T10:21:37.435 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1 2024-02-01T10:21:37.435 DEBUG:boto:path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:37.435 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:37.435 DEBUG:boto:Method: GET 2024-02-01T10:21:37.435 DEBUG:boto:Path: /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:37.435 DEBUG:boto:Data: 2024-02-01T10:21:37.435 DEBUG:boto:Headers: {} 2024-02-01T10:21:37.436 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:21:37.436 DEBUG:boto:Port: 8000 2024-02-01T10:21:37.436 DEBUG:boto:Params: {} 2024-02-01T10:21:37.436 DEBUG:boto:Token: None 2024-02-01T10:21:37.436 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:37 GMT /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:37.436 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vxzPXYDEoDpmELb9frXBxyUNT0E= 2024-02-01T10:21:37.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vxzPXYDEoDpmELb9frXBxyUNT0E=', 'Content-Length': '0'} 2024-02-01T10:21:37.439 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:49 GMT'), ('x-amz-version-id', 'lS9-sMOvyR-VcD.47BkPLymGmIyfWmf'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cc9c5c2e286191a9-0065bb70b1-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:37.480 DEBUG:boto:path=/vclfcn-16/obj-test-zone1 2024-02-01T10:21:37.480 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1 2024-02-01T10:21:37.480 DEBUG:boto:path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:37.481 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:37.481 DEBUG:boto:Method: GET 2024-02-01T10:21:37.481 DEBUG:boto:Path: /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:37.481 DEBUG:boto:Data: 2024-02-01T10:21:37.481 DEBUG:boto:Headers: {} 2024-02-01T10:21:37.481 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:21:37.481 DEBUG:boto:Port: 8000 2024-02-01T10:21:37.481 DEBUG:boto:Params: {} 2024-02-01T10:21:37.481 DEBUG:boto:Token: None 2024-02-01T10:21:37.481 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:37 GMT /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:37.482 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vxzPXYDEoDpmELb9frXBxyUNT0E= 2024-02-01T10:21:37.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vxzPXYDEoDpmELb9frXBxyUNT0E=', 'Content-Length': '0'} 2024-02-01T10:21:37.484 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:49 GMT'), ('x-amz-version-id', 'lS9-sMOvyR-VcD.47BkPLymGmIyfWmf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004814ea2889dfb9bd-0065bb70b1-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:37.485 DEBUG:boto:path=/vclfcn-16/obj-test-zone1 2024-02-01T10:21:37.485 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1 2024-02-01T10:21:37.485 DEBUG:boto:path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:37.485 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:37.485 DEBUG:boto:Method: HEAD 2024-02-01T10:21:37.485 DEBUG:boto:Path: /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:37.485 DEBUG:boto:Data: 2024-02-01T10:21:37.485 DEBUG:boto:Headers: {} 2024-02-01T10:21:37.486 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:21:37.486 DEBUG:boto:Port: 8000 2024-02-01T10:21:37.486 DEBUG:boto:Params: {} 2024-02-01T10:21:37.486 DEBUG:boto:Token: None 2024-02-01T10:21:37.486 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:21:37 GMT /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:37.486 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WU4U8VEm99i5xsOtNbj8OBmO35A= 2024-02-01T10:21:37.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WU4U8VEm99i5xsOtNbj8OBmO35A=', 'Content-Length': '0'} 2024-02-01T10:21:37.491 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:49 GMT'), ('x-amz-version-id', 'lS9-sMOvyR-VcD.47BkPLymGmIyfWmf'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000003877fc4db8cc191-0065bb70b1-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:37.491 DEBUG:boto:path=/vclfcn-16/obj-test-zone1 2024-02-01T10:21:37.491 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1 2024-02-01T10:21:37.492 DEBUG:boto:path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:37.492 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:37.492 DEBUG:boto:Method: HEAD 2024-02-01T10:21:37.492 DEBUG:boto:Path: /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:37.492 DEBUG:boto:Data: 2024-02-01T10:21:37.492 DEBUG:boto:Headers: {} 2024-02-01T10:21:37.492 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:21:37.492 DEBUG:boto:Port: 8000 2024-02-01T10:21:37.492 DEBUG:boto:Params: {} 2024-02-01T10:21:37.492 DEBUG:boto:Token: None 2024-02-01T10:21:37.493 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:21:37 GMT /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:37.493 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WU4U8VEm99i5xsOtNbj8OBmO35A= 2024-02-01T10:21:37.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WU4U8VEm99i5xsOtNbj8OBmO35A=', 'Content-Length': '0'} 2024-02-01T10:21:37.496 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:49 GMT'), ('x-amz-version-id', 'lS9-sMOvyR-VcD.47BkPLymGmIyfWmf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006c2c2d385feab308-0065bb70b1-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:37.496 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-02-01T10:21:37.496 DEBUG:boto:path=/vclfcn-16/obj-test-zone1 2024-02-01T10:21:37.496 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1 2024-02-01T10:21:37.496 DEBUG:boto:path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:37.496 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:37.496 DEBUG:boto:Method: GET 2024-02-01T10:21:37.497 DEBUG:boto:Path: /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:37.497 DEBUG:boto:Data: 2024-02-01T10:21:37.497 DEBUG:boto:Headers: {} 2024-02-01T10:21:37.497 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:21:37.497 DEBUG:boto:Port: 8000 2024-02-01T10:21:37.497 DEBUG:boto:Params: {} 2024-02-01T10:21:37.497 DEBUG:boto:Token: None 2024-02-01T10:21:37.497 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:37 GMT /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:37.497 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vxzPXYDEoDpmELb9frXBxyUNT0E= 2024-02-01T10:21:37.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vxzPXYDEoDpmELb9frXBxyUNT0E=', 'Content-Length': '0'} 2024-02-01T10:21:37.500 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:49 GMT'), ('x-amz-version-id', 'lS9-sMOvyR-VcD.47BkPLymGmIyfWmf'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000470d418bd260a44c-0065bb70b1-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:37.501 DEBUG:boto:path=/vclfcn-16/obj-test-zone1 2024-02-01T10:21:37.501 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1 2024-02-01T10:21:37.501 DEBUG:boto:path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:37.501 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:37.501 DEBUG:boto:Method: GET 2024-02-01T10:21:37.501 DEBUG:boto:Path: /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:37.501 DEBUG:boto:Data: 2024-02-01T10:21:37.501 DEBUG:boto:Headers: {} 2024-02-01T10:21:37.501 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:21:37.501 DEBUG:boto:Port: 8000 2024-02-01T10:21:37.501 DEBUG:boto:Params: {} 2024-02-01T10:21:37.502 DEBUG:boto:Token: None 2024-02-01T10:21:37.502 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:37 GMT /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:37.502 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vxzPXYDEoDpmELb9frXBxyUNT0E= 2024-02-01T10:21:37.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vxzPXYDEoDpmELb9frXBxyUNT0E=', 'Content-Length': '0'} 2024-02-01T10:21:37.505 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:49 GMT'), ('x-amz-version-id', 'lS9-sMOvyR-VcD.47BkPLymGmIyfWmf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000859f00887d3e558f-0065bb70b1-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:37.548 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:21:37.548 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:21:37.548 DEBUG:boto:path=/vclfcn-16/?prefix=obj-test-zone1 2024-02-01T10:21:37.549 DEBUG:boto:auth_path=/vclfcn-16/?prefix=obj-test-zone1 2024-02-01T10:21:37.549 DEBUG:boto:Method: GET 2024-02-01T10:21:37.549 DEBUG:boto:Path: /vclfcn-16/?prefix=obj-test-zone1 2024-02-01T10:21:37.549 DEBUG:boto:Data: 2024-02-01T10:21:37.549 DEBUG:boto:Headers: {} 2024-02-01T10:21:37.549 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:21:37.549 DEBUG:boto:Port: 8000 2024-02-01T10:21:37.549 DEBUG:boto:Params: {} 2024-02-01T10:21:37.549 DEBUG:boto:Token: None 2024-02-01T10:21:37.549 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:37 GMT /vclfcn-16/ 2024-02-01T10:21:37.550 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mzT+qbk26MWtfo7PH5zZucVVOrY= 2024-02-01T10:21:37.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mzT+qbk26MWtfo7PH5zZucVVOrY=', 'Content-Length': '0'} 2024-02-01T10:21:37.554 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cfe2d5e3ab08a3fc-0065bb70b1-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:37.554 DEBUG:boto:b'vclfcn-16obj-test-zone11000falseobj-test-zone12024-02-01T10:20:49.965Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:21:37.555 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:21:37.555 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:21:37.555 DEBUG:boto:path=/vclfcn-16/?prefix=obj-test-zone1 2024-02-01T10:21:37.555 DEBUG:boto:auth_path=/vclfcn-16/?prefix=obj-test-zone1 2024-02-01T10:21:37.555 DEBUG:boto:Method: GET 2024-02-01T10:21:37.555 DEBUG:boto:Path: /vclfcn-16/?prefix=obj-test-zone1 2024-02-01T10:21:37.555 DEBUG:boto:Data: 2024-02-01T10:21:37.555 DEBUG:boto:Headers: {} 2024-02-01T10:21:37.555 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:21:37.556 DEBUG:boto:Port: 8000 2024-02-01T10:21:37.556 DEBUG:boto:Params: {} 2024-02-01T10:21:37.556 DEBUG:boto:Token: None 2024-02-01T10:21:37.556 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:37 GMT /vclfcn-16/ 2024-02-01T10:21:37.556 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mzT+qbk26MWtfo7PH5zZucVVOrY= 2024-02-01T10:21:37.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mzT+qbk26MWtfo7PH5zZucVVOrY=', 'Content-Length': '0'} 2024-02-01T10:21:37.560 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000038c1980ff8c6e223-0065bb70b1-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:37.560 DEBUG:boto:b'vclfcn-16obj-test-zone11000falseobj-test-zone12024-02-01T10:20:49.965Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:21:37.561 DEBUG:boto:path=/vclfcn-16/obj-test-zone1 2024-02-01T10:21:37.561 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1 2024-02-01T10:21:37.561 DEBUG:boto:Method: HEAD 2024-02-01T10:21:37.561 DEBUG:boto:Path: /vclfcn-16/obj-test-zone1 2024-02-01T10:21:37.561 DEBUG:boto:Data: 2024-02-01T10:21:37.561 DEBUG:boto:Headers: {} 2024-02-01T10:21:37.561 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:21:37.561 DEBUG:boto:Port: 8000 2024-02-01T10:21:37.562 DEBUG:boto:Params: {} 2024-02-01T10:21:37.562 DEBUG:boto:Token: None 2024-02-01T10:21:37.562 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:21:37 GMT /vclfcn-16/obj-test-zone1 2024-02-01T10:21:37.562 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZHTQ8puzYq3/e+M+wjHThlpVS9g= 2024-02-01T10:21:37.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZHTQ8puzYq3/e+M+wjHThlpVS9g=', 'Content-Length': '0'} 2024-02-01T10:21:37.566 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:49 GMT'), ('x-amz-version-id', 'lS9-sMOvyR-VcD.47BkPLymGmIyfWmf'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b11b4d052a50954c-0065bb70b1-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:37.566 DEBUG:boto:path=/vclfcn-16/obj-test-zone1 2024-02-01T10:21:37.566 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1 2024-02-01T10:21:37.566 DEBUG:boto:Method: HEAD 2024-02-01T10:21:37.566 DEBUG:boto:Path: /vclfcn-16/obj-test-zone1 2024-02-01T10:21:37.566 DEBUG:boto:Data: 2024-02-01T10:21:37.566 DEBUG:boto:Headers: {} 2024-02-01T10:21:37.567 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:21:37.567 DEBUG:boto:Port: 8000 2024-02-01T10:21:37.567 DEBUG:boto:Params: {} 2024-02-01T10:21:37.567 DEBUG:boto:Token: None 2024-02-01T10:21:37.567 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:21:37 GMT /vclfcn-16/obj-test-zone1 2024-02-01T10:21:37.567 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZHTQ8puzYq3/e+M+wjHThlpVS9g= 2024-02-01T10:21:37.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZHTQ8puzYq3/e+M+wjHThlpVS9g=', 'Content-Length': '0'} 2024-02-01T10:21:37.571 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:49 GMT'), ('x-amz-version-id', 'lS9-sMOvyR-VcD.47BkPLymGmIyfWmf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a88660dafa6aca5b-0065bb70b1-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:37.571 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-02-01T10:21:37.571 DEBUG:boto:path=/vclfcn-16/obj-test-zone1 2024-02-01T10:21:37.571 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1 2024-02-01T10:21:37.571 DEBUG:boto:path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:37.571 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:37.571 DEBUG:boto:Method: GET 2024-02-01T10:21:37.572 DEBUG:boto:Path: /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:37.572 DEBUG:boto:Data: 2024-02-01T10:21:37.572 DEBUG:boto:Headers: {} 2024-02-01T10:21:37.572 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:21:37.572 DEBUG:boto:Port: 8000 2024-02-01T10:21:37.572 DEBUG:boto:Params: {} 2024-02-01T10:21:37.572 DEBUG:boto:Token: None 2024-02-01T10:21:37.572 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:37 GMT /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:37.572 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vxzPXYDEoDpmELb9frXBxyUNT0E= 2024-02-01T10:21:37.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vxzPXYDEoDpmELb9frXBxyUNT0E=', 'Content-Length': '0'} 2024-02-01T10:21:37.575 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:49 GMT'), ('x-amz-version-id', 'lS9-sMOvyR-VcD.47BkPLymGmIyfWmf'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003b25ec880100b747-0065bb70b1-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:37.616 DEBUG:boto:path=/vclfcn-16/obj-test-zone1 2024-02-01T10:21:37.617 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1 2024-02-01T10:21:37.617 DEBUG:boto:path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:37.617 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:37.617 DEBUG:boto:Method: GET 2024-02-01T10:21:37.617 DEBUG:boto:Path: /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:37.617 DEBUG:boto:Data: 2024-02-01T10:21:37.617 DEBUG:boto:Headers: {} 2024-02-01T10:21:37.617 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:21:37.617 DEBUG:boto:Port: 8000 2024-02-01T10:21:37.617 DEBUG:boto:Params: {} 2024-02-01T10:21:37.618 DEBUG:boto:Token: None 2024-02-01T10:21:37.618 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:37 GMT /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:37.618 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vxzPXYDEoDpmELb9frXBxyUNT0E= 2024-02-01T10:21:37.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vxzPXYDEoDpmELb9frXBxyUNT0E=', 'Content-Length': '0'} 2024-02-01T10:21:37.621 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:49 GMT'), ('x-amz-version-id', 'lS9-sMOvyR-VcD.47BkPLymGmIyfWmf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000096ef4aa99462b7db-0065bb70b1-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:37.621 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:21:37.621 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:21:37.621 DEBUG:boto:path=/vclfcn-16/?prefix=obj-test-zone1 2024-02-01T10:21:37.622 DEBUG:boto:auth_path=/vclfcn-16/?prefix=obj-test-zone1 2024-02-01T10:21:37.622 DEBUG:boto:Method: GET 2024-02-01T10:21:37.622 DEBUG:boto:Path: /vclfcn-16/?prefix=obj-test-zone1 2024-02-01T10:21:37.622 DEBUG:boto:Data: 2024-02-01T10:21:37.622 DEBUG:boto:Headers: {} 2024-02-01T10:21:37.622 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:21:37.622 DEBUG:boto:Port: 8000 2024-02-01T10:21:37.622 DEBUG:boto:Params: {} 2024-02-01T10:21:37.622 DEBUG:boto:Token: None 2024-02-01T10:21:37.623 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:37 GMT /vclfcn-16/ 2024-02-01T10:21:37.623 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mzT+qbk26MWtfo7PH5zZucVVOrY= 2024-02-01T10:21:37.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mzT+qbk26MWtfo7PH5zZucVVOrY=', 'Content-Length': '0'} 2024-02-01T10:21:37.627 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089158bf0a6d8dd2c-0065bb70b1-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:37.627 DEBUG:boto:b'vclfcn-16obj-test-zone11000falseobj-test-zone12024-02-01T10:20:49.965Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:21:37.628 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:21:37.628 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:21:37.628 DEBUG:boto:path=/vclfcn-16/?prefix=obj-test-zone1 2024-02-01T10:21:37.628 DEBUG:boto:auth_path=/vclfcn-16/?prefix=obj-test-zone1 2024-02-01T10:21:37.628 DEBUG:boto:Method: GET 2024-02-01T10:21:37.628 DEBUG:boto:Path: /vclfcn-16/?prefix=obj-test-zone1 2024-02-01T10:21:37.628 DEBUG:boto:Data: 2024-02-01T10:21:37.629 DEBUG:boto:Headers: {} 2024-02-01T10:21:37.629 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:21:37.629 DEBUG:boto:Port: 8000 2024-02-01T10:21:37.629 DEBUG:boto:Params: {} 2024-02-01T10:21:37.629 DEBUG:boto:Token: None 2024-02-01T10:21:37.629 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:37 GMT /vclfcn-16/ 2024-02-01T10:21:37.629 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mzT+qbk26MWtfo7PH5zZucVVOrY= 2024-02-01T10:21:37.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mzT+qbk26MWtfo7PH5zZucVVOrY=', 'Content-Length': '0'} 2024-02-01T10:21:37.633 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a525e73c2260cc6-0065bb70b1-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:37.676 DEBUG:boto:b'vclfcn-16obj-test-zone11000falseobj-test-zone12024-02-01T10:20:49.965Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:21:37.676 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-02-01T10:21:37.677 DEBUG:boto:path=/vclfcn-16/obj-test-zone2 2024-02-01T10:21:37.677 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2 2024-02-01T10:21:37.677 DEBUG:boto:path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:37.677 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:37.677 DEBUG:boto:Method: GET 2024-02-01T10:21:37.677 DEBUG:boto:Path: /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:37.677 DEBUG:boto:Data: 2024-02-01T10:21:37.677 DEBUG:boto:Headers: {} 2024-02-01T10:21:37.677 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:21:37.677 DEBUG:boto:Port: 8000 2024-02-01T10:21:37.678 DEBUG:boto:Params: {} 2024-02-01T10:21:37.678 DEBUG:boto:Token: None 2024-02-01T10:21:37.678 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:37 GMT /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:37.678 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:2qpAR4henzXDYPlgjohvNgGgwDQ= 2024-02-01T10:21:37.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2qpAR4henzXDYPlgjohvNgGgwDQ=', 'Content-Length': '0'} 2024-02-01T10:21:37.681 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'mE6urSxcrRA9wdyuToa41.w2aTia9p1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000000868afd00052e9-0065bb70b1-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:37.681 DEBUG:boto:path=/vclfcn-16/obj-test-zone2 2024-02-01T10:21:37.681 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2 2024-02-01T10:21:37.682 DEBUG:boto:path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:37.682 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:37.682 DEBUG:boto:Method: GET 2024-02-01T10:21:37.682 DEBUG:boto:Path: /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:37.682 DEBUG:boto:Data: 2024-02-01T10:21:37.682 DEBUG:boto:Headers: {} 2024-02-01T10:21:37.682 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:21:37.682 DEBUG:boto:Port: 8000 2024-02-01T10:21:37.682 DEBUG:boto:Params: {} 2024-02-01T10:21:37.682 DEBUG:boto:Token: None 2024-02-01T10:21:37.682 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:37 GMT /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:37.683 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:2qpAR4henzXDYPlgjohvNgGgwDQ= 2024-02-01T10:21:37.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2qpAR4henzXDYPlgjohvNgGgwDQ=', 'Content-Length': '0'} 2024-02-01T10:21:37.686 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'mE6urSxcrRA9wdyuToa41.w2aTia9p1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d058db2331d5a750-0065bb70b1-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:37.686 DEBUG:boto:path=/vclfcn-16/obj-test-zone2 2024-02-01T10:21:37.686 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2 2024-02-01T10:21:37.687 DEBUG:boto:path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:37.687 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:37.687 DEBUG:boto:Method: HEAD 2024-02-01T10:21:37.687 DEBUG:boto:Path: /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:37.687 DEBUG:boto:Data: 2024-02-01T10:21:37.687 DEBUG:boto:Headers: {} 2024-02-01T10:21:37.687 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:21:37.687 DEBUG:boto:Port: 8000 2024-02-01T10:21:37.687 DEBUG:boto:Params: {} 2024-02-01T10:21:37.687 DEBUG:boto:Token: None 2024-02-01T10:21:37.688 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:21:37 GMT /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:37.688 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:AkldWuRcL+bCwfRQjFEA1qMnaKc= 2024-02-01T10:21:37.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:AkldWuRcL+bCwfRQjFEA1qMnaKc=', 'Content-Length': '0'} 2024-02-01T10:21:37.690 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'mE6urSxcrRA9wdyuToa41.w2aTia9p1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fbf05939d1264d53-0065bb70b1-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:37.691 DEBUG:boto:path=/vclfcn-16/obj-test-zone2 2024-02-01T10:21:37.691 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2 2024-02-01T10:21:37.691 DEBUG:boto:path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:37.691 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:37.691 DEBUG:boto:Method: HEAD 2024-02-01T10:21:37.691 DEBUG:boto:Path: /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:37.691 DEBUG:boto:Data: 2024-02-01T10:21:37.691 DEBUG:boto:Headers: {} 2024-02-01T10:21:37.691 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:21:37.692 DEBUG:boto:Port: 8000 2024-02-01T10:21:37.692 DEBUG:boto:Params: {} 2024-02-01T10:21:37.692 DEBUG:boto:Token: None 2024-02-01T10:21:37.692 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:21:37 GMT /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:37.692 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:AkldWuRcL+bCwfRQjFEA1qMnaKc= 2024-02-01T10:21:37.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:AkldWuRcL+bCwfRQjFEA1qMnaKc=', 'Content-Length': '0'} 2024-02-01T10:21:37.695 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'mE6urSxcrRA9wdyuToa41.w2aTia9p1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000039755dc52ae9e9d3-0065bb70b1-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:37.696 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-02-01T10:21:37.696 DEBUG:boto:path=/vclfcn-16/obj-test-zone2 2024-02-01T10:21:37.696 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2 2024-02-01T10:21:37.696 DEBUG:boto:path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:37.696 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:37.696 DEBUG:boto:Method: GET 2024-02-01T10:21:37.696 DEBUG:boto:Path: /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:37.696 DEBUG:boto:Data: 2024-02-01T10:21:37.697 DEBUG:boto:Headers: {} 2024-02-01T10:21:37.697 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:21:37.697 DEBUG:boto:Port: 8000 2024-02-01T10:21:37.697 DEBUG:boto:Params: {} 2024-02-01T10:21:37.697 DEBUG:boto:Token: None 2024-02-01T10:21:37.697 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:37 GMT /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:37.697 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:2qpAR4henzXDYPlgjohvNgGgwDQ= 2024-02-01T10:21:37.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2qpAR4henzXDYPlgjohvNgGgwDQ=', 'Content-Length': '0'} 2024-02-01T10:21:37.700 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'mE6urSxcrRA9wdyuToa41.w2aTia9p1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bf8ab1ef5727cf64-0065bb70b1-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:37.700 DEBUG:boto:path=/vclfcn-16/obj-test-zone2 2024-02-01T10:21:37.700 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2 2024-02-01T10:21:37.700 DEBUG:boto:path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:37.701 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:37.701 DEBUG:boto:Method: GET 2024-02-01T10:21:37.701 DEBUG:boto:Path: /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:37.701 DEBUG:boto:Data: 2024-02-01T10:21:37.701 DEBUG:boto:Headers: {} 2024-02-01T10:21:37.701 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:21:37.701 DEBUG:boto:Port: 8000 2024-02-01T10:21:37.701 DEBUG:boto:Params: {} 2024-02-01T10:21:37.701 DEBUG:boto:Token: None 2024-02-01T10:21:37.701 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:37 GMT /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:37.701 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:2qpAR4henzXDYPlgjohvNgGgwDQ= 2024-02-01T10:21:37.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2qpAR4henzXDYPlgjohvNgGgwDQ=', 'Content-Length': '0'} 2024-02-01T10:21:37.704 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'mE6urSxcrRA9wdyuToa41.w2aTia9p1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000311d49882f1c261-0065bb70b1-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:37.748 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:21:37.748 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:21:37.749 DEBUG:boto:path=/vclfcn-16/?prefix=obj-test-zone2 2024-02-01T10:21:37.749 DEBUG:boto:auth_path=/vclfcn-16/?prefix=obj-test-zone2 2024-02-01T10:21:37.749 DEBUG:boto:Method: GET 2024-02-01T10:21:37.749 DEBUG:boto:Path: /vclfcn-16/?prefix=obj-test-zone2 2024-02-01T10:21:37.749 DEBUG:boto:Data: 2024-02-01T10:21:37.749 DEBUG:boto:Headers: {} 2024-02-01T10:21:37.749 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:21:37.749 DEBUG:boto:Port: 8000 2024-02-01T10:21:37.749 DEBUG:boto:Params: {} 2024-02-01T10:21:37.750 DEBUG:boto:Token: None 2024-02-01T10:21:37.750 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:37 GMT /vclfcn-16/ 2024-02-01T10:21:37.750 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mzT+qbk26MWtfo7PH5zZucVVOrY= 2024-02-01T10:21:37.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mzT+qbk26MWtfo7PH5zZucVVOrY=', 'Content-Length': '0'} 2024-02-01T10:21:37.754 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a4ed55871f7584f-0065bb70b1-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:37.755 DEBUG:boto:b'vclfcn-16obj-test-zone21000falseobj-test-zone22024-02-01T10:20:50.122Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:21:37.755 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:21:37.756 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:21:37.756 DEBUG:boto:path=/vclfcn-16/?prefix=obj-test-zone2 2024-02-01T10:21:37.756 DEBUG:boto:auth_path=/vclfcn-16/?prefix=obj-test-zone2 2024-02-01T10:21:37.756 DEBUG:boto:Method: GET 2024-02-01T10:21:37.756 DEBUG:boto:Path: /vclfcn-16/?prefix=obj-test-zone2 2024-02-01T10:21:37.756 DEBUG:boto:Data: 2024-02-01T10:21:37.756 DEBUG:boto:Headers: {} 2024-02-01T10:21:37.756 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:21:37.756 DEBUG:boto:Port: 8000 2024-02-01T10:21:37.756 DEBUG:boto:Params: {} 2024-02-01T10:21:37.757 DEBUG:boto:Token: None 2024-02-01T10:21:37.757 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:37 GMT /vclfcn-16/ 2024-02-01T10:21:37.757 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mzT+qbk26MWtfo7PH5zZucVVOrY= 2024-02-01T10:21:37.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mzT+qbk26MWtfo7PH5zZucVVOrY=', 'Content-Length': '0'} 2024-02-01T10:21:37.761 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000093912c2e7194f3d9-0065bb70b1-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:37.804 DEBUG:boto:b'vclfcn-16obj-test-zone21000falseobj-test-zone22024-02-01T10:20:50.122Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:21:37.804 DEBUG:boto:path=/vclfcn-16/obj-test-zone2 2024-02-01T10:21:37.805 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2 2024-02-01T10:21:37.805 DEBUG:boto:Method: HEAD 2024-02-01T10:21:37.805 DEBUG:boto:Path: /vclfcn-16/obj-test-zone2 2024-02-01T10:21:37.805 DEBUG:boto:Data: 2024-02-01T10:21:37.805 DEBUG:boto:Headers: {} 2024-02-01T10:21:37.805 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:21:37.805 DEBUG:boto:Port: 8000 2024-02-01T10:21:37.805 DEBUG:boto:Params: {} 2024-02-01T10:21:37.805 DEBUG:boto:Token: None 2024-02-01T10:21:37.806 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:21:37 GMT /vclfcn-16/obj-test-zone2 2024-02-01T10:21:37.806 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:qg7BfCgKLkKhnJ9VXX7pGIbhgUY= 2024-02-01T10:21:37.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:qg7BfCgKLkKhnJ9VXX7pGIbhgUY=', 'Content-Length': '0'} 2024-02-01T10:21:37.810 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'mE6urSxcrRA9wdyuToa41.w2aTia9p1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eaedb09e0a19aac8-0065bb70b1-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:37.810 DEBUG:boto:path=/vclfcn-16/obj-test-zone2 2024-02-01T10:21:37.810 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2 2024-02-01T10:21:37.810 DEBUG:boto:Method: HEAD 2024-02-01T10:21:37.810 DEBUG:boto:Path: /vclfcn-16/obj-test-zone2 2024-02-01T10:21:37.810 DEBUG:boto:Data: 2024-02-01T10:21:37.810 DEBUG:boto:Headers: {} 2024-02-01T10:21:37.810 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:21:37.811 DEBUG:boto:Port: 8000 2024-02-01T10:21:37.811 DEBUG:boto:Params: {} 2024-02-01T10:21:37.811 DEBUG:boto:Token: None 2024-02-01T10:21:37.811 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:21:37 GMT /vclfcn-16/obj-test-zone2 2024-02-01T10:21:37.811 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:qg7BfCgKLkKhnJ9VXX7pGIbhgUY= 2024-02-01T10:21:37.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:qg7BfCgKLkKhnJ9VXX7pGIbhgUY=', 'Content-Length': '0'} 2024-02-01T10:21:37.815 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'mE6urSxcrRA9wdyuToa41.w2aTia9p1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d72d38f7d73ccce4-0065bb70b1-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:37.815 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-02-01T10:21:37.815 DEBUG:boto:path=/vclfcn-16/obj-test-zone2 2024-02-01T10:21:37.815 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2 2024-02-01T10:21:37.815 DEBUG:boto:path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:37.815 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:37.815 DEBUG:boto:Method: GET 2024-02-01T10:21:37.816 DEBUG:boto:Path: /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:37.816 DEBUG:boto:Data: 2024-02-01T10:21:37.816 DEBUG:boto:Headers: {} 2024-02-01T10:21:37.816 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:21:37.816 DEBUG:boto:Port: 8000 2024-02-01T10:21:37.816 DEBUG:boto:Params: {} 2024-02-01T10:21:37.816 DEBUG:boto:Token: None 2024-02-01T10:21:37.816 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:37 GMT /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:37.817 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:2qpAR4henzXDYPlgjohvNgGgwDQ= 2024-02-01T10:21:37.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2qpAR4henzXDYPlgjohvNgGgwDQ=', 'Content-Length': '0'} 2024-02-01T10:21:37.820 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'mE6urSxcrRA9wdyuToa41.w2aTia9p1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000a54f2d3f4d0f987-0065bb70b1-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:37.860 DEBUG:boto:path=/vclfcn-16/obj-test-zone2 2024-02-01T10:21:37.860 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2 2024-02-01T10:21:37.860 DEBUG:boto:path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:37.860 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:37.861 DEBUG:boto:Method: GET 2024-02-01T10:21:37.861 DEBUG:boto:Path: /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:37.861 DEBUG:boto:Data: 2024-02-01T10:21:37.861 DEBUG:boto:Headers: {} 2024-02-01T10:21:37.861 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:21:37.861 DEBUG:boto:Port: 8000 2024-02-01T10:21:37.861 DEBUG:boto:Params: {} 2024-02-01T10:21:37.861 DEBUG:boto:Token: None 2024-02-01T10:21:37.861 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:37 GMT /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:37.862 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:2qpAR4henzXDYPlgjohvNgGgwDQ= 2024-02-01T10:21:37.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2qpAR4henzXDYPlgjohvNgGgwDQ=', 'Content-Length': '0'} 2024-02-01T10:21:37.864 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'mE6urSxcrRA9wdyuToa41.w2aTia9p1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b1db23fd0cc5a1c1-0065bb70b1-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:37.908 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:21:37.908 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:21:37.908 DEBUG:boto:path=/vclfcn-16/?prefix=obj-test-zone2 2024-02-01T10:21:37.908 DEBUG:boto:auth_path=/vclfcn-16/?prefix=obj-test-zone2 2024-02-01T10:21:37.909 DEBUG:boto:Method: GET 2024-02-01T10:21:37.909 DEBUG:boto:Path: /vclfcn-16/?prefix=obj-test-zone2 2024-02-01T10:21:37.909 DEBUG:boto:Data: 2024-02-01T10:21:37.909 DEBUG:boto:Headers: {} 2024-02-01T10:21:37.909 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:21:37.909 DEBUG:boto:Port: 8000 2024-02-01T10:21:37.909 DEBUG:boto:Params: {} 2024-02-01T10:21:37.909 DEBUG:boto:Token: None 2024-02-01T10:21:37.909 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:37 GMT /vclfcn-16/ 2024-02-01T10:21:37.910 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mzT+qbk26MWtfo7PH5zZucVVOrY= 2024-02-01T10:21:37.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mzT+qbk26MWtfo7PH5zZucVVOrY=', 'Content-Length': '0'} 2024-02-01T10:21:37.914 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc815774ad27fbf6-0065bb70b1-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:37.914 DEBUG:boto:b'vclfcn-16obj-test-zone21000falseobj-test-zone22024-02-01T10:20:50.122Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:21:37.915 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:21:37.915 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:21:37.915 DEBUG:boto:path=/vclfcn-16/?prefix=obj-test-zone2 2024-02-01T10:21:37.916 DEBUG:boto:auth_path=/vclfcn-16/?prefix=obj-test-zone2 2024-02-01T10:21:37.916 DEBUG:boto:Method: GET 2024-02-01T10:21:37.916 DEBUG:boto:Path: /vclfcn-16/?prefix=obj-test-zone2 2024-02-01T10:21:37.916 DEBUG:boto:Data: 2024-02-01T10:21:37.916 DEBUG:boto:Headers: {} 2024-02-01T10:21:37.916 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:21:37.916 DEBUG:boto:Port: 8000 2024-02-01T10:21:37.916 DEBUG:boto:Params: {} 2024-02-01T10:21:37.916 DEBUG:boto:Token: None 2024-02-01T10:21:37.917 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:37 GMT /vclfcn-16/ 2024-02-01T10:21:37.917 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mzT+qbk26MWtfo7PH5zZucVVOrY= 2024-02-01T10:21:37.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mzT+qbk26MWtfo7PH5zZucVVOrY=', 'Content-Length': '0'} 2024-02-01T10:21:37.921 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e2d7ee161166f10-0065bb70b1-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:37.964 DEBUG:boto:b'vclfcn-16obj-test-zone21000falseobj-test-zone22024-02-01T10:20:50.122Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:21:37.964 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-02-01T10:21:37.965 DEBUG:boto:path=/vclfcn-16/obj-test-zone3 2024-02-01T10:21:37.965 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3 2024-02-01T10:21:37.965 DEBUG:boto:path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:37.965 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:37.965 DEBUG:boto:Method: GET 2024-02-01T10:21:37.965 DEBUG:boto:Path: /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:37.965 DEBUG:boto:Data: 2024-02-01T10:21:37.965 DEBUG:boto:Headers: {} 2024-02-01T10:21:37.966 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:21:37.966 DEBUG:boto:Port: 8000 2024-02-01T10:21:37.966 DEBUG:boto:Params: {} 2024-02-01T10:21:37.966 DEBUG:boto:Token: None 2024-02-01T10:21:37.966 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:37 GMT /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:37.966 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7vLFfDVGI+iCPLKMoSoy+GbPubI= 2024-02-01T10:21:37.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7vLFfDVGI+iCPLKMoSoy+GbPubI=', 'Content-Length': '0'} 2024-02-01T10:21:37.969 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005de9b0efbb195c4d-0065bb70b1-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:37.970 DEBUG:boto:path=/vclfcn-16/obj-test-zone3 2024-02-01T10:21:37.970 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3 2024-02-01T10:21:37.970 DEBUG:boto:path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:37.971 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:37.971 DEBUG:boto:Method: GET 2024-02-01T10:21:37.971 DEBUG:boto:Path: /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:37.971 DEBUG:boto:Data: 2024-02-01T10:21:37.971 DEBUG:boto:Headers: {} 2024-02-01T10:21:37.971 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:21:37.971 DEBUG:boto:Port: 8000 2024-02-01T10:21:37.972 DEBUG:boto:Params: {} 2024-02-01T10:21:37.972 DEBUG:boto:Token: None 2024-02-01T10:21:37.972 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:37 GMT /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:37.972 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7vLFfDVGI+iCPLKMoSoy+GbPubI= 2024-02-01T10:21:37.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7vLFfDVGI+iCPLKMoSoy+GbPubI=', 'Content-Length': '0'} 2024-02-01T10:21:37.976 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007366bc37ee04cecd-0065bb70b1-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:37.976 DEBUG:boto:path=/vclfcn-16/obj-test-zone3 2024-02-01T10:21:37.976 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3 2024-02-01T10:21:37.977 DEBUG:boto:path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:37.977 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:37.977 DEBUG:boto:Method: HEAD 2024-02-01T10:21:37.977 DEBUG:boto:Path: /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:37.978 DEBUG:boto:Data: 2024-02-01T10:21:37.978 DEBUG:boto:Headers: {} 2024-02-01T10:21:37.978 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:21:37.978 DEBUG:boto:Port: 8000 2024-02-01T10:21:37.978 DEBUG:boto:Params: {} 2024-02-01T10:21:37.979 DEBUG:boto:Token: None 2024-02-01T10:21:37.979 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:21:37 GMT /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:37.979 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:qkW1RsVynWx5x8kWpd41IxjiQ54= 2024-02-01T10:21:37.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:qkW1RsVynWx5x8kWpd41IxjiQ54=', 'Content-Length': '0'} 2024-02-01T10:21:37.983 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000644c830ea52ffcd0-0065bb70b1-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:37.983 DEBUG:boto:path=/vclfcn-16/obj-test-zone3 2024-02-01T10:21:37.984 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3 2024-02-01T10:21:37.984 DEBUG:boto:path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:37.984 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:37.984 DEBUG:boto:Method: HEAD 2024-02-01T10:21:37.984 DEBUG:boto:Path: /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:37.984 DEBUG:boto:Data: 2024-02-01T10:21:37.985 DEBUG:boto:Headers: {} 2024-02-01T10:21:37.985 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:21:37.985 DEBUG:boto:Port: 8000 2024-02-01T10:21:37.985 DEBUG:boto:Params: {} 2024-02-01T10:21:37.985 DEBUG:boto:Token: None 2024-02-01T10:21:37.985 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:21:37 GMT /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:37.986 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:qkW1RsVynWx5x8kWpd41IxjiQ54= 2024-02-01T10:21:37.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:qkW1RsVynWx5x8kWpd41IxjiQ54=', 'Content-Length': '0'} 2024-02-01T10:21:37.989 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000049087755718490b9-0065bb70b1-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:37.989 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-02-01T10:21:37.989 DEBUG:boto:path=/vclfcn-16/obj-test-zone3 2024-02-01T10:21:37.989 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3 2024-02-01T10:21:37.989 DEBUG:boto:path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:37.989 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:37.989 DEBUG:boto:Method: GET 2024-02-01T10:21:37.990 DEBUG:boto:Path: /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:37.990 DEBUG:boto:Data: 2024-02-01T10:21:37.990 DEBUG:boto:Headers: {} 2024-02-01T10:21:37.990 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:21:37.990 DEBUG:boto:Port: 8000 2024-02-01T10:21:37.990 DEBUG:boto:Params: {} 2024-02-01T10:21:37.990 DEBUG:boto:Token: None 2024-02-01T10:21:37.990 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:37 GMT /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:37.990 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7vLFfDVGI+iCPLKMoSoy+GbPubI= 2024-02-01T10:21:37.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7vLFfDVGI+iCPLKMoSoy+GbPubI=', 'Content-Length': '0'} 2024-02-01T10:21:37.993 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000059c65dacf547fdb0-0065bb70b1-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:37.994 DEBUG:boto:path=/vclfcn-16/obj-test-zone3 2024-02-01T10:21:37.994 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3 2024-02-01T10:21:37.994 DEBUG:boto:path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:37.994 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:37.994 DEBUG:boto:Method: GET 2024-02-01T10:21:37.994 DEBUG:boto:Path: /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:37.994 DEBUG:boto:Data: 2024-02-01T10:21:37.994 DEBUG:boto:Headers: {} 2024-02-01T10:21:37.994 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:21:37.995 DEBUG:boto:Port: 8000 2024-02-01T10:21:37.995 DEBUG:boto:Params: {} 2024-02-01T10:21:37.995 DEBUG:boto:Token: None 2024-02-01T10:21:37.995 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:37 GMT /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:37.995 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7vLFfDVGI+iCPLKMoSoy+GbPubI= 2024-02-01T10:21:37.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7vLFfDVGI+iCPLKMoSoy+GbPubI=', 'Content-Length': '0'} 2024-02-01T10:21:37.998 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000095ce40a058b33519-0065bb70b2-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:38.040 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:21:38.040 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:21:38.040 DEBUG:boto:path=/vclfcn-16/?prefix=obj-test-zone3 2024-02-01T10:21:38.040 DEBUG:boto:auth_path=/vclfcn-16/?prefix=obj-test-zone3 2024-02-01T10:21:38.041 DEBUG:boto:Method: GET 2024-02-01T10:21:38.041 DEBUG:boto:Path: /vclfcn-16/?prefix=obj-test-zone3 2024-02-01T10:21:38.041 DEBUG:boto:Data: 2024-02-01T10:21:38.041 DEBUG:boto:Headers: {} 2024-02-01T10:21:38.041 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:21:38.041 DEBUG:boto:Port: 8000 2024-02-01T10:21:38.041 DEBUG:boto:Params: {} 2024-02-01T10:21:38.041 DEBUG:boto:Token: None 2024-02-01T10:21:38.041 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:38 GMT /vclfcn-16/ 2024-02-01T10:21:38.042 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lgaZ1mEIlG/hgqxRgZNZu4J9xuw= 2024-02-01T10:21:38.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lgaZ1mEIlG/hgqxRgZNZu4J9xuw=', 'Content-Length': '0'} 2024-02-01T10:21:38.045 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000197f94e3a813dd4-0065bb70b2-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:38.046 DEBUG:boto:b'vclfcn-16obj-test-zone31000falseobj-test-zone32024-02-01T10:20:50.297Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:21:38.046 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:21:38.046 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:21:38.046 DEBUG:boto:path=/vclfcn-16/?prefix=obj-test-zone3 2024-02-01T10:21:38.046 DEBUG:boto:auth_path=/vclfcn-16/?prefix=obj-test-zone3 2024-02-01T10:21:38.047 DEBUG:boto:Method: GET 2024-02-01T10:21:38.047 DEBUG:boto:Path: /vclfcn-16/?prefix=obj-test-zone3 2024-02-01T10:21:38.047 DEBUG:boto:Data: 2024-02-01T10:21:38.047 DEBUG:boto:Headers: {} 2024-02-01T10:21:38.047 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:21:38.047 DEBUG:boto:Port: 8000 2024-02-01T10:21:38.047 DEBUG:boto:Params: {} 2024-02-01T10:21:38.047 DEBUG:boto:Token: None 2024-02-01T10:21:38.047 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:38 GMT /vclfcn-16/ 2024-02-01T10:21:38.047 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lgaZ1mEIlG/hgqxRgZNZu4J9xuw= 2024-02-01T10:21:38.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lgaZ1mEIlG/hgqxRgZNZu4J9xuw=', 'Content-Length': '0'} 2024-02-01T10:21:38.052 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad260862834b92c4-0065bb70b2-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:38.096 DEBUG:boto:b'vclfcn-16obj-test-zone31000falseobj-test-zone32024-02-01T10:20:50.297Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:21:38.097 DEBUG:boto:path=/vclfcn-16/obj-test-zone3 2024-02-01T10:21:38.097 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3 2024-02-01T10:21:38.097 DEBUG:boto:Method: HEAD 2024-02-01T10:21:38.097 DEBUG:boto:Path: /vclfcn-16/obj-test-zone3 2024-02-01T10:21:38.097 DEBUG:boto:Data: 2024-02-01T10:21:38.097 DEBUG:boto:Headers: {} 2024-02-01T10:21:38.097 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:21:38.097 DEBUG:boto:Port: 8000 2024-02-01T10:21:38.097 DEBUG:boto:Params: {} 2024-02-01T10:21:38.098 DEBUG:boto:Token: None 2024-02-01T10:21:38.098 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:21:38 GMT /vclfcn-16/obj-test-zone3 2024-02-01T10:21:38.098 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5iYlLatdexg5MQ57hyap2B8ZSYQ= 2024-02-01T10:21:38.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5iYlLatdexg5MQ57hyap2B8ZSYQ=', 'Content-Length': '0'} 2024-02-01T10:21:38.102 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a905fb676b635cfb-0065bb70b2-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:38.102 DEBUG:boto:path=/vclfcn-16/obj-test-zone3 2024-02-01T10:21:38.102 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3 2024-02-01T10:21:38.102 DEBUG:boto:Method: HEAD 2024-02-01T10:21:38.102 DEBUG:boto:Path: /vclfcn-16/obj-test-zone3 2024-02-01T10:21:38.103 DEBUG:boto:Data: 2024-02-01T10:21:38.103 DEBUG:boto:Headers: {} 2024-02-01T10:21:38.103 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:21:38.103 DEBUG:boto:Port: 8000 2024-02-01T10:21:38.103 DEBUG:boto:Params: {} 2024-02-01T10:21:38.103 DEBUG:boto:Token: None 2024-02-01T10:21:38.103 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:21:38 GMT /vclfcn-16/obj-test-zone3 2024-02-01T10:21:38.103 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5iYlLatdexg5MQ57hyap2B8ZSYQ= 2024-02-01T10:21:38.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5iYlLatdexg5MQ57hyap2B8ZSYQ=', 'Content-Length': '0'} 2024-02-01T10:21:38.107 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007e9792bea7e9f24e-0065bb70b2-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:38.107 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-02-01T10:21:38.108 DEBUG:boto:path=/vclfcn-16/obj-test-zone3 2024-02-01T10:21:38.108 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3 2024-02-01T10:21:38.108 DEBUG:boto:path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:38.108 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:38.108 DEBUG:boto:Method: GET 2024-02-01T10:21:38.108 DEBUG:boto:Path: /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:38.108 DEBUG:boto:Data: 2024-02-01T10:21:38.108 DEBUG:boto:Headers: {} 2024-02-01T10:21:38.108 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:21:38.108 DEBUG:boto:Port: 8000 2024-02-01T10:21:38.108 DEBUG:boto:Params: {} 2024-02-01T10:21:38.109 DEBUG:boto:Token: None 2024-02-01T10:21:38.109 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:38 GMT /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:38.109 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fZ+q3iobJxrx7KEQ9+NHtDtcu+8= 2024-02-01T10:21:38.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fZ+q3iobJxrx7KEQ9+NHtDtcu+8=', 'Content-Length': '0'} 2024-02-01T10:21:38.112 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ace6fc102dc3cb2a-0065bb70b2-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:38.112 DEBUG:boto:path=/vclfcn-16/obj-test-zone3 2024-02-01T10:21:38.112 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3 2024-02-01T10:21:38.112 DEBUG:boto:path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:38.112 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:38.112 DEBUG:boto:Method: GET 2024-02-01T10:21:38.113 DEBUG:boto:Path: /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:38.113 DEBUG:boto:Data: 2024-02-01T10:21:38.113 DEBUG:boto:Headers: {} 2024-02-01T10:21:38.113 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:21:38.113 DEBUG:boto:Port: 8000 2024-02-01T10:21:38.113 DEBUG:boto:Params: {} 2024-02-01T10:21:38.113 DEBUG:boto:Token: None 2024-02-01T10:21:38.113 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:38 GMT /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:38.113 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fZ+q3iobJxrx7KEQ9+NHtDtcu+8= 2024-02-01T10:21:38.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fZ+q3iobJxrx7KEQ9+NHtDtcu+8=', 'Content-Length': '0'} 2024-02-01T10:21:38.116 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003fb56d1141424034-0065bb70b2-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:38.164 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:21:38.164 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:21:38.164 DEBUG:boto:path=/vclfcn-16/?prefix=obj-test-zone3 2024-02-01T10:21:38.165 DEBUG:boto:auth_path=/vclfcn-16/?prefix=obj-test-zone3 2024-02-01T10:21:38.165 DEBUG:boto:Method: GET 2024-02-01T10:21:38.165 DEBUG:boto:Path: /vclfcn-16/?prefix=obj-test-zone3 2024-02-01T10:21:38.165 DEBUG:boto:Data: 2024-02-01T10:21:38.165 DEBUG:boto:Headers: {} 2024-02-01T10:21:38.165 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:21:38.165 DEBUG:boto:Port: 8000 2024-02-01T10:21:38.165 DEBUG:boto:Params: {} 2024-02-01T10:21:38.165 DEBUG:boto:Token: None 2024-02-01T10:21:38.165 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:38 GMT /vclfcn-16/ 2024-02-01T10:21:38.166 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lgaZ1mEIlG/hgqxRgZNZu4J9xuw= 2024-02-01T10:21:38.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lgaZ1mEIlG/hgqxRgZNZu4J9xuw=', 'Content-Length': '0'} 2024-02-01T10:21:38.170 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b9bd7a65a684040-0065bb70b2-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:38.170 DEBUG:boto:b'vclfcn-16obj-test-zone31000falseobj-test-zone32024-02-01T10:20:50.297Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:21:38.171 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:21:38.171 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:21:38.171 DEBUG:boto:path=/vclfcn-16/?prefix=obj-test-zone3 2024-02-01T10:21:38.171 DEBUG:boto:auth_path=/vclfcn-16/?prefix=obj-test-zone3 2024-02-01T10:21:38.171 DEBUG:boto:Method: GET 2024-02-01T10:21:38.171 DEBUG:boto:Path: /vclfcn-16/?prefix=obj-test-zone3 2024-02-01T10:21:38.171 DEBUG:boto:Data: 2024-02-01T10:21:38.172 DEBUG:boto:Headers: {} 2024-02-01T10:21:38.172 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:21:38.172 DEBUG:boto:Port: 8000 2024-02-01T10:21:38.172 DEBUG:boto:Params: {} 2024-02-01T10:21:38.172 DEBUG:boto:Token: None 2024-02-01T10:21:38.172 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:38 GMT /vclfcn-16/ 2024-02-01T10:21:38.172 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lgaZ1mEIlG/hgqxRgZNZu4J9xuw= 2024-02-01T10:21:38.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lgaZ1mEIlG/hgqxRgZNZu4J9xuw=', 'Content-Length': '0'} 2024-02-01T10:21:38.177 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c981cc9bab3c16a1-0065bb70b2-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:38.177 DEBUG:boto:b'vclfcn-16obj-test-zone31000falseobj-test-zone32024-02-01T10:20:50.297Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:21:38.177 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-16 zones={test-zone2, test-zone1} 2024-02-01T10:21:38.178 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-16 zones={test-zone3, test-zone1} 2024-02-01T10:21:38.178 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:21:38.178 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:21:38.178 DEBUG:boto:Method: HEAD 2024-02-01T10:21:38.178 DEBUG:boto:Path: /vclfcn-16/ 2024-02-01T10:21:38.178 DEBUG:boto:Data: 2024-02-01T10:21:38.178 DEBUG:boto:Headers: {} 2024-02-01T10:21:38.178 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:21:38.178 DEBUG:boto:Port: 8001 2024-02-01T10:21:38.178 DEBUG:boto:Params: {} 2024-02-01T10:21:38.178 DEBUG:boto:Token: None 2024-02-01T10:21:38.179 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:21:38 GMT /vclfcn-16/ 2024-02-01T10:21:38.179 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:aZEHRwF1wTfeMnBVhntOocMls3w= 2024-02-01T10:21:38.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:aZEHRwF1wTfeMnBVhntOocMls3w=', 'Content-Length': '0'} 2024-02-01T10:21:38.183 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aeaf59dc39ae4df6-0065bb70b2-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:21:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:38.183 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:21:38.183 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:21:38.183 DEBUG:boto:Method: HEAD 2024-02-01T10:21:38.183 DEBUG:boto:Path: /vclfcn-16/ 2024-02-01T10:21:38.183 DEBUG:boto:Data: 2024-02-01T10:21:38.183 DEBUG:boto:Headers: {} 2024-02-01T10:21:38.184 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:21:38.184 DEBUG:boto:Port: 8000 2024-02-01T10:21:38.184 DEBUG:boto:Params: {} 2024-02-01T10:21:38.184 DEBUG:boto:Token: None 2024-02-01T10:21:38.184 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:21:38 GMT /vclfcn-16/ 2024-02-01T10:21:38.184 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:aZEHRwF1wTfeMnBVhntOocMls3w= 2024-02-01T10:21:38.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:aZEHRwF1wTfeMnBVhntOocMls3w=', 'Content-Length': '0'} 2024-02-01T10:21:38.188 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001c27bdd779a28552-0065bb70b2-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:21:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:38.188 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:21:38.188 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:21:38.188 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:21:38.188 DEBUG:boto:path=/vclfcn-16/?versions&max-keys=999 2024-02-01T10:21:38.189 DEBUG:boto:auth_path=/vclfcn-16/?versions&max-keys=999 2024-02-01T10:21:38.189 DEBUG:boto:Method: GET 2024-02-01T10:21:38.189 DEBUG:boto:Path: /vclfcn-16/?versions&max-keys=999 2024-02-01T10:21:38.189 DEBUG:boto:Data: 2024-02-01T10:21:38.189 DEBUG:boto:Headers: {} 2024-02-01T10:21:38.189 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:21:38.189 DEBUG:boto:Port: 8001 2024-02-01T10:21:38.189 DEBUG:boto:Params: {} 2024-02-01T10:21:38.189 DEBUG:boto:Token: None 2024-02-01T10:21:38.189 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:38 GMT /vclfcn-16/?versions 2024-02-01T10:21:38.190 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RWJ0Rrk7zegXpUPQM3M0YMOX7K8= 2024-02-01T10:21:38.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RWJ0Rrk7zegXpUPQM3M0YMOX7K8=', 'Content-Length': '0'} 2024-02-01T10:21:38.194 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000073bc9380f3ff653f-0065bb70b2-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:38.194 DEBUG:boto:b'vclfcn-16999falsedummyrQRpBowIFAsvxVDn92H30shYnKoGt4ntrue2024-02-01T10:19:58.958Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1lS9-sMOvyR-VcD.47BkPLymGmIyfWmftrue2024-02-01T10:20:49.965Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2mE6urSxcrRA9wdyuToa41.w2aTia9p1true2024-02-01T10:20:50.122Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjPtrue2024-02-01T10:20:50.297Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:21:38.195 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-02-01T10:21:38.195 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-02-01T10:21:38.195 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-02-01T10:21:38.195 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-02-01T10:21:38.195 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:21:38.195 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:21:38.195 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:21:38.195 DEBUG:boto:path=/vclfcn-16/?versions&max-keys=999 2024-02-01T10:21:38.195 DEBUG:boto:auth_path=/vclfcn-16/?versions&max-keys=999 2024-02-01T10:21:38.196 DEBUG:boto:Method: GET 2024-02-01T10:21:38.196 DEBUG:boto:Path: /vclfcn-16/?versions&max-keys=999 2024-02-01T10:21:38.196 DEBUG:boto:Data: 2024-02-01T10:21:38.196 DEBUG:boto:Headers: {} 2024-02-01T10:21:38.196 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:21:38.196 DEBUG:boto:Port: 8000 2024-02-01T10:21:38.196 DEBUG:boto:Params: {} 2024-02-01T10:21:38.196 DEBUG:boto:Token: None 2024-02-01T10:21:38.196 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:38 GMT /vclfcn-16/?versions 2024-02-01T10:21:38.196 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RWJ0Rrk7zegXpUPQM3M0YMOX7K8= 2024-02-01T10:21:38.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RWJ0Rrk7zegXpUPQM3M0YMOX7K8=', 'Content-Length': '0'} 2024-02-01T10:21:38.201 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc41dca29ef50501-0065bb70b2-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:38.201 DEBUG:boto:b'vclfcn-16999falsedummyrQRpBowIFAsvxVDn92H30shYnKoGt4ntrue2024-02-01T10:19:58.958Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1lS9-sMOvyR-VcD.47BkPLymGmIyfWmftrue2024-02-01T10:20:49.965Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2mE6urSxcrRA9wdyuToa41.w2aTia9p1true2024-02-01T10:20:50.122Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjPtrue2024-02-01T10:20:50.297Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:21:38.201 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-02-01T10:21:38.202 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-02-01T10:21:38.202 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-02-01T10:21:38.202 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-02-01T10:21:38.202 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:21:38.202 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:21:38.202 DEBUG:boto:path=/vclfcn-16/?versions&max-keys=999 2024-02-01T10:21:38.202 DEBUG:boto:auth_path=/vclfcn-16/?versions&max-keys=999 2024-02-01T10:21:38.202 DEBUG:boto:Method: GET 2024-02-01T10:21:38.202 DEBUG:boto:Path: /vclfcn-16/?versions&max-keys=999 2024-02-01T10:21:38.202 DEBUG:boto:Data: 2024-02-01T10:21:38.203 DEBUG:boto:Headers: {} 2024-02-01T10:21:38.203 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:21:38.203 DEBUG:boto:Port: 8001 2024-02-01T10:21:38.203 DEBUG:boto:Params: {} 2024-02-01T10:21:38.203 DEBUG:boto:Token: None 2024-02-01T10:21:38.203 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:38 GMT /vclfcn-16/?versions 2024-02-01T10:21:38.203 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RWJ0Rrk7zegXpUPQM3M0YMOX7K8= 2024-02-01T10:21:38.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RWJ0Rrk7zegXpUPQM3M0YMOX7K8=', 'Content-Length': '0'} 2024-02-01T10:21:38.207 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000079d833805adb69de-0065bb70b2-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:38.208 DEBUG:boto:b'vclfcn-16999falsedummyrQRpBowIFAsvxVDn92H30shYnKoGt4ntrue2024-02-01T10:19:58.958Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1lS9-sMOvyR-VcD.47BkPLymGmIyfWmftrue2024-02-01T10:20:49.965Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2mE6urSxcrRA9wdyuToa41.w2aTia9p1true2024-02-01T10:20:50.122Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjPtrue2024-02-01T10:20:50.297Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:21:38.208 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:21:38.208 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:21:38.208 DEBUG:boto:path=/vclfcn-16/?versions&max-keys=999 2024-02-01T10:21:38.208 DEBUG:boto:auth_path=/vclfcn-16/?versions&max-keys=999 2024-02-01T10:21:38.208 DEBUG:boto:Method: GET 2024-02-01T10:21:38.209 DEBUG:boto:Path: /vclfcn-16/?versions&max-keys=999 2024-02-01T10:21:38.209 DEBUG:boto:Data: 2024-02-01T10:21:38.209 DEBUG:boto:Headers: {} 2024-02-01T10:21:38.209 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:21:38.209 DEBUG:boto:Port: 8000 2024-02-01T10:21:38.209 DEBUG:boto:Params: {} 2024-02-01T10:21:38.209 DEBUG:boto:Token: None 2024-02-01T10:21:38.209 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:38 GMT /vclfcn-16/?versions 2024-02-01T10:21:38.209 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RWJ0Rrk7zegXpUPQM3M0YMOX7K8= 2024-02-01T10:21:38.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RWJ0Rrk7zegXpUPQM3M0YMOX7K8=', 'Content-Length': '0'} 2024-02-01T10:21:38.213 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ad809afcf6a473f-0065bb70b2-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:38.213 DEBUG:boto:b'vclfcn-16999falsedummyrQRpBowIFAsvxVDn92H30shYnKoGt4ntrue2024-02-01T10:19:58.958Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1lS9-sMOvyR-VcD.47BkPLymGmIyfWmftrue2024-02-01T10:20:49.965Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2mE6urSxcrRA9wdyuToa41.w2aTia9p1true2024-02-01T10:20:50.122Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjPtrue2024-02-01T10:20:50.297Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:21:38.214 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-02-01T10:21:38.214 DEBUG:boto:path=/vclfcn-16/dummy 2024-02-01T10:21:38.214 DEBUG:boto:auth_path=/vclfcn-16/dummy 2024-02-01T10:21:38.214 DEBUG:boto:path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:38.214 DEBUG:boto:auth_path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:38.215 DEBUG:boto:Method: GET 2024-02-01T10:21:38.215 DEBUG:boto:Path: /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:38.215 DEBUG:boto:Data: 2024-02-01T10:21:38.215 DEBUG:boto:Headers: {} 2024-02-01T10:21:38.215 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:21:38.215 DEBUG:boto:Port: 8001 2024-02-01T10:21:38.215 DEBUG:boto:Params: {} 2024-02-01T10:21:38.215 DEBUG:boto:Token: None 2024-02-01T10:21:38.215 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:38 GMT /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:38.215 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:K2q7TOfUAPR2AYG5bhKThqGyDow= 2024-02-01T10:21:38.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:K2q7TOfUAPR2AYG5bhKThqGyDow=', 'Content-Length': '0'} 2024-02-01T10:21:38.218 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:19:58 GMT'), ('x-amz-version-id', 'rQRpBowIFAsvxVDn92H30shYnKoGt4n'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000093dd6f770e0e809c-0065bb70b2-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:38.218 DEBUG:boto:path=/vclfcn-16/dummy 2024-02-01T10:21:38.218 DEBUG:boto:auth_path=/vclfcn-16/dummy 2024-02-01T10:21:38.219 DEBUG:boto:path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:38.219 DEBUG:boto:auth_path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:38.219 DEBUG:boto:Method: GET 2024-02-01T10:21:38.219 DEBUG:boto:Path: /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:38.219 DEBUG:boto:Data: 2024-02-01T10:21:38.219 DEBUG:boto:Headers: {} 2024-02-01T10:21:38.219 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:21:38.219 DEBUG:boto:Port: 8000 2024-02-01T10:21:38.219 DEBUG:boto:Params: {} 2024-02-01T10:21:38.219 DEBUG:boto:Token: None 2024-02-01T10:21:38.219 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:38 GMT /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:38.220 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:K2q7TOfUAPR2AYG5bhKThqGyDow= 2024-02-01T10:21:38.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:K2q7TOfUAPR2AYG5bhKThqGyDow=', 'Content-Length': '0'} 2024-02-01T10:21:38.222 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:19:58 GMT'), ('x-amz-version-id', 'rQRpBowIFAsvxVDn92H30shYnKoGt4n'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001849110ae8d7e8bd-0065bb70b2-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:38.223 DEBUG:boto:path=/vclfcn-16/dummy 2024-02-01T10:21:38.223 DEBUG:boto:auth_path=/vclfcn-16/dummy 2024-02-01T10:21:38.223 DEBUG:boto:path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:38.223 DEBUG:boto:auth_path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:38.223 DEBUG:boto:Method: HEAD 2024-02-01T10:21:38.223 DEBUG:boto:Path: /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:38.223 DEBUG:boto:Data: 2024-02-01T10:21:38.223 DEBUG:boto:Headers: {} 2024-02-01T10:21:38.223 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:21:38.223 DEBUG:boto:Port: 8001 2024-02-01T10:21:38.224 DEBUG:boto:Params: {} 2024-02-01T10:21:38.224 DEBUG:boto:Token: None 2024-02-01T10:21:38.224 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:21:38 GMT /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:38.224 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:I2oMKCYYgjp/zrAzYu42aqpZY6g= 2024-02-01T10:21:38.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:I2oMKCYYgjp/zrAzYu42aqpZY6g=', 'Content-Length': '0'} 2024-02-01T10:21:38.227 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:19:58 GMT'), ('x-amz-version-id', 'rQRpBowIFAsvxVDn92H30shYnKoGt4n'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f284e8d29d92cf3e-0065bb70b2-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:38.227 DEBUG:boto:path=/vclfcn-16/dummy 2024-02-01T10:21:38.227 DEBUG:boto:auth_path=/vclfcn-16/dummy 2024-02-01T10:21:38.227 DEBUG:boto:path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:38.228 DEBUG:boto:auth_path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:38.228 DEBUG:boto:Method: HEAD 2024-02-01T10:21:38.228 DEBUG:boto:Path: /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:38.228 DEBUG:boto:Data: 2024-02-01T10:21:38.228 DEBUG:boto:Headers: {} 2024-02-01T10:21:38.228 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:21:38.228 DEBUG:boto:Port: 8000 2024-02-01T10:21:38.228 DEBUG:boto:Params: {} 2024-02-01T10:21:38.228 DEBUG:boto:Token: None 2024-02-01T10:21:38.228 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:21:38 GMT /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:38.229 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:I2oMKCYYgjp/zrAzYu42aqpZY6g= 2024-02-01T10:21:38.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:I2oMKCYYgjp/zrAzYu42aqpZY6g=', 'Content-Length': '0'} 2024-02-01T10:21:38.231 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:19:58 GMT'), ('x-amz-version-id', 'rQRpBowIFAsvxVDn92H30shYnKoGt4n'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000073ea6a49762814e3-0065bb70b2-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:38.231 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-02-01T10:21:38.231 DEBUG:boto:path=/vclfcn-16/dummy 2024-02-01T10:21:38.232 DEBUG:boto:auth_path=/vclfcn-16/dummy 2024-02-01T10:21:38.232 DEBUG:boto:path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:38.232 DEBUG:boto:auth_path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:38.232 DEBUG:boto:Method: GET 2024-02-01T10:21:38.232 DEBUG:boto:Path: /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:38.232 DEBUG:boto:Data: 2024-02-01T10:21:38.232 DEBUG:boto:Headers: {} 2024-02-01T10:21:38.232 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:21:38.232 DEBUG:boto:Port: 8001 2024-02-01T10:21:38.232 DEBUG:boto:Params: {} 2024-02-01T10:21:38.233 DEBUG:boto:Token: None 2024-02-01T10:21:38.233 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:38 GMT /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:38.233 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:K2q7TOfUAPR2AYG5bhKThqGyDow= 2024-02-01T10:21:38.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:K2q7TOfUAPR2AYG5bhKThqGyDow=', 'Content-Length': '0'} 2024-02-01T10:21:38.235 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:19:58 GMT'), ('x-amz-version-id', 'rQRpBowIFAsvxVDn92H30shYnKoGt4n'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d06f4f3735b9712f-0065bb70b2-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:38.236 DEBUG:boto:path=/vclfcn-16/dummy 2024-02-01T10:21:38.236 DEBUG:boto:auth_path=/vclfcn-16/dummy 2024-02-01T10:21:38.236 DEBUG:boto:path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:38.236 DEBUG:boto:auth_path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:38.236 DEBUG:boto:Method: GET 2024-02-01T10:21:38.236 DEBUG:boto:Path: /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:38.236 DEBUG:boto:Data: 2024-02-01T10:21:38.236 DEBUG:boto:Headers: {} 2024-02-01T10:21:38.236 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:21:38.237 DEBUG:boto:Port: 8000 2024-02-01T10:21:38.237 DEBUG:boto:Params: {} 2024-02-01T10:21:38.237 DEBUG:boto:Token: None 2024-02-01T10:21:38.237 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:38 GMT /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:38.237 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:K2q7TOfUAPR2AYG5bhKThqGyDow= 2024-02-01T10:21:38.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:K2q7TOfUAPR2AYG5bhKThqGyDow=', 'Content-Length': '0'} 2024-02-01T10:21:38.240 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:19:58 GMT'), ('x-amz-version-id', 'rQRpBowIFAsvxVDn92H30shYnKoGt4n'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ed14a7be99d2bde2-0065bb70b2-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:38.240 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:21:38.240 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:21:38.240 DEBUG:boto:path=/vclfcn-16/?prefix=dummy 2024-02-01T10:21:38.240 DEBUG:boto:auth_path=/vclfcn-16/?prefix=dummy 2024-02-01T10:21:38.240 DEBUG:boto:Method: GET 2024-02-01T10:21:38.241 DEBUG:boto:Path: /vclfcn-16/?prefix=dummy 2024-02-01T10:21:38.241 DEBUG:boto:Data: 2024-02-01T10:21:38.241 DEBUG:boto:Headers: {} 2024-02-01T10:21:38.241 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:21:38.241 DEBUG:boto:Port: 8001 2024-02-01T10:21:38.241 DEBUG:boto:Params: {} 2024-02-01T10:21:38.241 DEBUG:boto:Token: None 2024-02-01T10:21:38.241 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:38 GMT /vclfcn-16/ 2024-02-01T10:21:38.241 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lgaZ1mEIlG/hgqxRgZNZu4J9xuw= 2024-02-01T10:21:38.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lgaZ1mEIlG/hgqxRgZNZu4J9xuw=', 'Content-Length': '0'} 2024-02-01T10:21:38.246 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3cec265c741291c-0065bb70b2-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:38.292 DEBUG:boto:b'vclfcn-16dummy1000falsedummy2024-02-01T10:19:58.958Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:21:38.293 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:21:38.293 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:21:38.293 DEBUG:boto:path=/vclfcn-16/?prefix=dummy 2024-02-01T10:21:38.293 DEBUG:boto:auth_path=/vclfcn-16/?prefix=dummy 2024-02-01T10:21:38.293 DEBUG:boto:Method: GET 2024-02-01T10:21:38.293 DEBUG:boto:Path: /vclfcn-16/?prefix=dummy 2024-02-01T10:21:38.293 DEBUG:boto:Data: 2024-02-01T10:21:38.293 DEBUG:boto:Headers: {} 2024-02-01T10:21:38.293 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:21:38.293 DEBUG:boto:Port: 8000 2024-02-01T10:21:38.294 DEBUG:boto:Params: {} 2024-02-01T10:21:38.294 DEBUG:boto:Token: None 2024-02-01T10:21:38.294 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:38 GMT /vclfcn-16/ 2024-02-01T10:21:38.294 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lgaZ1mEIlG/hgqxRgZNZu4J9xuw= 2024-02-01T10:21:38.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lgaZ1mEIlG/hgqxRgZNZu4J9xuw=', 'Content-Length': '0'} 2024-02-01T10:21:38.298 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b375302ed7a4e7f-0065bb70b2-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:38.299 DEBUG:boto:b'vclfcn-16dummy1000falsedummy2024-02-01T10:19:58.958Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:21:38.299 DEBUG:boto:path=/vclfcn-16/dummy 2024-02-01T10:21:38.299 DEBUG:boto:auth_path=/vclfcn-16/dummy 2024-02-01T10:21:38.299 DEBUG:boto:Method: HEAD 2024-02-01T10:21:38.299 DEBUG:boto:Path: /vclfcn-16/dummy 2024-02-01T10:21:38.300 DEBUG:boto:Data: 2024-02-01T10:21:38.300 DEBUG:boto:Headers: {} 2024-02-01T10:21:38.300 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:21:38.300 DEBUG:boto:Port: 8001 2024-02-01T10:21:38.300 DEBUG:boto:Params: {} 2024-02-01T10:21:38.300 DEBUG:boto:Token: None 2024-02-01T10:21:38.300 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:21:38 GMT /vclfcn-16/dummy 2024-02-01T10:21:38.300 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Db9Yn0RMJ3+9wtMY3LI7SOxHrDY= 2024-02-01T10:21:38.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Db9Yn0RMJ3+9wtMY3LI7SOxHrDY=', 'Content-Length': '0'} 2024-02-01T10:21:38.304 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:19:58 GMT'), ('x-amz-version-id', 'rQRpBowIFAsvxVDn92H30shYnKoGt4n'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a9b52ed8c55b0f58-0065bb70b2-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:38.304 DEBUG:boto:path=/vclfcn-16/dummy 2024-02-01T10:21:38.304 DEBUG:boto:auth_path=/vclfcn-16/dummy 2024-02-01T10:21:38.305 DEBUG:boto:Method: HEAD 2024-02-01T10:21:38.305 DEBUG:boto:Path: /vclfcn-16/dummy 2024-02-01T10:21:38.305 DEBUG:boto:Data: 2024-02-01T10:21:38.305 DEBUG:boto:Headers: {} 2024-02-01T10:21:38.305 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:21:38.305 DEBUG:boto:Port: 8000 2024-02-01T10:21:38.305 DEBUG:boto:Params: {} 2024-02-01T10:21:38.305 DEBUG:boto:Token: None 2024-02-01T10:21:38.305 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:21:38 GMT /vclfcn-16/dummy 2024-02-01T10:21:38.306 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Db9Yn0RMJ3+9wtMY3LI7SOxHrDY= 2024-02-01T10:21:38.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Db9Yn0RMJ3+9wtMY3LI7SOxHrDY=', 'Content-Length': '0'} 2024-02-01T10:21:38.309 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:19:58 GMT'), ('x-amz-version-id', 'rQRpBowIFAsvxVDn92H30shYnKoGt4n'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000057869d172f4dcfab-0065bb70b2-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:38.309 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-02-01T10:21:38.309 DEBUG:boto:path=/vclfcn-16/dummy 2024-02-01T10:21:38.309 DEBUG:boto:auth_path=/vclfcn-16/dummy 2024-02-01T10:21:38.310 DEBUG:boto:path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:38.310 DEBUG:boto:auth_path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:38.310 DEBUG:boto:Method: GET 2024-02-01T10:21:38.310 DEBUG:boto:Path: /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:38.310 DEBUG:boto:Data: 2024-02-01T10:21:38.310 DEBUG:boto:Headers: {} 2024-02-01T10:21:38.310 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:21:38.310 DEBUG:boto:Port: 8001 2024-02-01T10:21:38.310 DEBUG:boto:Params: {} 2024-02-01T10:21:38.310 DEBUG:boto:Token: None 2024-02-01T10:21:38.311 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:38 GMT /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:38.311 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:K2q7TOfUAPR2AYG5bhKThqGyDow= 2024-02-01T10:21:38.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:K2q7TOfUAPR2AYG5bhKThqGyDow=', 'Content-Length': '0'} 2024-02-01T10:21:38.313 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:19:58 GMT'), ('x-amz-version-id', 'rQRpBowIFAsvxVDn92H30shYnKoGt4n'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a118991ca7788a2b-0065bb70b2-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:38.314 DEBUG:boto:path=/vclfcn-16/dummy 2024-02-01T10:21:38.314 DEBUG:boto:auth_path=/vclfcn-16/dummy 2024-02-01T10:21:38.314 DEBUG:boto:path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:38.314 DEBUG:boto:auth_path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:38.314 DEBUG:boto:Method: GET 2024-02-01T10:21:38.314 DEBUG:boto:Path: /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:38.314 DEBUG:boto:Data: 2024-02-01T10:21:38.314 DEBUG:boto:Headers: {} 2024-02-01T10:21:38.314 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:21:38.314 DEBUG:boto:Port: 8000 2024-02-01T10:21:38.315 DEBUG:boto:Params: {} 2024-02-01T10:21:38.315 DEBUG:boto:Token: None 2024-02-01T10:21:38.315 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:38 GMT /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:38.315 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:K2q7TOfUAPR2AYG5bhKThqGyDow= 2024-02-01T10:21:38.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:K2q7TOfUAPR2AYG5bhKThqGyDow=', 'Content-Length': '0'} 2024-02-01T10:21:38.318 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:19:58 GMT'), ('x-amz-version-id', 'rQRpBowIFAsvxVDn92H30shYnKoGt4n'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000860ef636e853a5f3-0065bb70b2-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:38.318 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:21:38.318 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:21:38.318 DEBUG:boto:path=/vclfcn-16/?prefix=dummy 2024-02-01T10:21:38.318 DEBUG:boto:auth_path=/vclfcn-16/?prefix=dummy 2024-02-01T10:21:38.318 DEBUG:boto:Method: GET 2024-02-01T10:21:38.318 DEBUG:boto:Path: /vclfcn-16/?prefix=dummy 2024-02-01T10:21:38.319 DEBUG:boto:Data: 2024-02-01T10:21:38.319 DEBUG:boto:Headers: {} 2024-02-01T10:21:38.319 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:21:38.319 DEBUG:boto:Port: 8001 2024-02-01T10:21:38.319 DEBUG:boto:Params: {} 2024-02-01T10:21:38.319 DEBUG:boto:Token: None 2024-02-01T10:21:38.319 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:38 GMT /vclfcn-16/ 2024-02-01T10:21:38.319 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lgaZ1mEIlG/hgqxRgZNZu4J9xuw= 2024-02-01T10:21:38.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lgaZ1mEIlG/hgqxRgZNZu4J9xuw=', 'Content-Length': '0'} 2024-02-01T10:21:38.323 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078dd2cb51dd1912f-0065bb70b2-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:38.364 DEBUG:boto:b'vclfcn-16dummy1000falsedummy2024-02-01T10:19:58.958Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:21:38.364 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:21:38.365 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:21:38.365 DEBUG:boto:path=/vclfcn-16/?prefix=dummy 2024-02-01T10:21:38.365 DEBUG:boto:auth_path=/vclfcn-16/?prefix=dummy 2024-02-01T10:21:38.365 DEBUG:boto:Method: GET 2024-02-01T10:21:38.365 DEBUG:boto:Path: /vclfcn-16/?prefix=dummy 2024-02-01T10:21:38.365 DEBUG:boto:Data: 2024-02-01T10:21:38.365 DEBUG:boto:Headers: {} 2024-02-01T10:21:38.365 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:21:38.365 DEBUG:boto:Port: 8000 2024-02-01T10:21:38.365 DEBUG:boto:Params: {} 2024-02-01T10:21:38.366 DEBUG:boto:Token: None 2024-02-01T10:21:38.366 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:38 GMT /vclfcn-16/ 2024-02-01T10:21:38.366 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lgaZ1mEIlG/hgqxRgZNZu4J9xuw= 2024-02-01T10:21:38.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lgaZ1mEIlG/hgqxRgZNZu4J9xuw=', 'Content-Length': '0'} 2024-02-01T10:21:38.370 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000074f8e652add0cd38-0065bb70b2-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:38.370 DEBUG:boto:b'vclfcn-16dummy1000falsedummy2024-02-01T10:19:58.958Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:21:38.370 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-02-01T10:21:38.371 DEBUG:boto:path=/vclfcn-16/obj-test-zone1 2024-02-01T10:21:38.371 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1 2024-02-01T10:21:38.371 DEBUG:boto:path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:38.371 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:38.371 DEBUG:boto:Method: GET 2024-02-01T10:21:38.371 DEBUG:boto:Path: /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:38.371 DEBUG:boto:Data: 2024-02-01T10:21:38.371 DEBUG:boto:Headers: {} 2024-02-01T10:21:38.371 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:21:38.371 DEBUG:boto:Port: 8001 2024-02-01T10:21:38.372 DEBUG:boto:Params: {} 2024-02-01T10:21:38.372 DEBUG:boto:Token: None 2024-02-01T10:21:38.372 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:38 GMT /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:38.372 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:IJnK/84kNNLN34TzV78VmyOLke0= 2024-02-01T10:21:38.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:IJnK/84kNNLN34TzV78VmyOLke0=', 'Content-Length': '0'} 2024-02-01T10:21:38.375 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:49 GMT'), ('x-amz-version-id', 'lS9-sMOvyR-VcD.47BkPLymGmIyfWmf'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a37d44345c1f07ea-0065bb70b2-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:38.375 DEBUG:boto:path=/vclfcn-16/obj-test-zone1 2024-02-01T10:21:38.375 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1 2024-02-01T10:21:38.375 DEBUG:boto:path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:38.376 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:38.376 DEBUG:boto:Method: GET 2024-02-01T10:21:38.376 DEBUG:boto:Path: /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:38.376 DEBUG:boto:Data: 2024-02-01T10:21:38.376 DEBUG:boto:Headers: {} 2024-02-01T10:21:38.376 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:21:38.376 DEBUG:boto:Port: 8000 2024-02-01T10:21:38.376 DEBUG:boto:Params: {} 2024-02-01T10:21:38.376 DEBUG:boto:Token: None 2024-02-01T10:21:38.376 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:38 GMT /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:38.377 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:IJnK/84kNNLN34TzV78VmyOLke0= 2024-02-01T10:21:38.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:IJnK/84kNNLN34TzV78VmyOLke0=', 'Content-Length': '0'} 2024-02-01T10:21:38.379 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:49 GMT'), ('x-amz-version-id', 'lS9-sMOvyR-VcD.47BkPLymGmIyfWmf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007f3ab0db4f793cb2-0065bb70b2-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:38.420 DEBUG:boto:path=/vclfcn-16/obj-test-zone1 2024-02-01T10:21:38.420 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1 2024-02-01T10:21:38.420 DEBUG:boto:path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:38.421 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:38.421 DEBUG:boto:Method: HEAD 2024-02-01T10:21:38.421 DEBUG:boto:Path: /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:38.421 DEBUG:boto:Data: 2024-02-01T10:21:38.421 DEBUG:boto:Headers: {} 2024-02-01T10:21:38.421 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:21:38.421 DEBUG:boto:Port: 8001 2024-02-01T10:21:38.421 DEBUG:boto:Params: {} 2024-02-01T10:21:38.421 DEBUG:boto:Token: None 2024-02-01T10:21:38.421 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:21:38 GMT /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:38.422 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:49hiWhR6QWhVW7D3jJJ+KYt+mFM= 2024-02-01T10:21:38.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:49hiWhR6QWhVW7D3jJJ+KYt+mFM=', 'Content-Length': '0'} 2024-02-01T10:21:38.425 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:49 GMT'), ('x-amz-version-id', 'lS9-sMOvyR-VcD.47BkPLymGmIyfWmf'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000096422db54c877ca2-0065bb70b2-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:38.425 DEBUG:boto:path=/vclfcn-16/obj-test-zone1 2024-02-01T10:21:38.425 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1 2024-02-01T10:21:38.425 DEBUG:boto:path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:38.425 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:38.425 DEBUG:boto:Method: HEAD 2024-02-01T10:21:38.425 DEBUG:boto:Path: /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:38.425 DEBUG:boto:Data: 2024-02-01T10:21:38.426 DEBUG:boto:Headers: {} 2024-02-01T10:21:38.426 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:21:38.426 DEBUG:boto:Port: 8000 2024-02-01T10:21:38.426 DEBUG:boto:Params: {} 2024-02-01T10:21:38.426 DEBUG:boto:Token: None 2024-02-01T10:21:38.426 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:21:38 GMT /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:38.426 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:49hiWhR6QWhVW7D3jJJ+KYt+mFM= 2024-02-01T10:21:38.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:49hiWhR6QWhVW7D3jJJ+KYt+mFM=', 'Content-Length': '0'} 2024-02-01T10:21:38.429 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:49 GMT'), ('x-amz-version-id', 'lS9-sMOvyR-VcD.47BkPLymGmIyfWmf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001fd0c3a58676e245-0065bb70b2-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:38.429 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-02-01T10:21:38.429 DEBUG:boto:path=/vclfcn-16/obj-test-zone1 2024-02-01T10:21:38.429 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1 2024-02-01T10:21:38.430 DEBUG:boto:path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:38.430 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:38.430 DEBUG:boto:Method: GET 2024-02-01T10:21:38.430 DEBUG:boto:Path: /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:38.430 DEBUG:boto:Data: 2024-02-01T10:21:38.430 DEBUG:boto:Headers: {} 2024-02-01T10:21:38.430 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:21:38.430 DEBUG:boto:Port: 8001 2024-02-01T10:21:38.430 DEBUG:boto:Params: {} 2024-02-01T10:21:38.430 DEBUG:boto:Token: None 2024-02-01T10:21:38.430 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:38 GMT /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:38.431 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:IJnK/84kNNLN34TzV78VmyOLke0= 2024-02-01T10:21:38.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:IJnK/84kNNLN34TzV78VmyOLke0=', 'Content-Length': '0'} 2024-02-01T10:21:38.433 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:49 GMT'), ('x-amz-version-id', 'lS9-sMOvyR-VcD.47BkPLymGmIyfWmf'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005ab00ee74e862909-0065bb70b2-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:38.476 DEBUG:boto:path=/vclfcn-16/obj-test-zone1 2024-02-01T10:21:38.476 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1 2024-02-01T10:21:38.476 DEBUG:boto:path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:38.477 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:38.477 DEBUG:boto:Method: GET 2024-02-01T10:21:38.477 DEBUG:boto:Path: /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:38.477 DEBUG:boto:Data: 2024-02-01T10:21:38.477 DEBUG:boto:Headers: {} 2024-02-01T10:21:38.477 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:21:38.477 DEBUG:boto:Port: 8000 2024-02-01T10:21:38.477 DEBUG:boto:Params: {} 2024-02-01T10:21:38.477 DEBUG:boto:Token: None 2024-02-01T10:21:38.477 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:38 GMT /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:38.478 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:IJnK/84kNNLN34TzV78VmyOLke0= 2024-02-01T10:21:38.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:IJnK/84kNNLN34TzV78VmyOLke0=', 'Content-Length': '0'} 2024-02-01T10:21:38.481 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:49 GMT'), ('x-amz-version-id', 'lS9-sMOvyR-VcD.47BkPLymGmIyfWmf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e58add41e9603704-0065bb70b2-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:38.481 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:21:38.481 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:21:38.481 DEBUG:boto:path=/vclfcn-16/?prefix=obj-test-zone1 2024-02-01T10:21:38.481 DEBUG:boto:auth_path=/vclfcn-16/?prefix=obj-test-zone1 2024-02-01T10:21:38.481 DEBUG:boto:Method: GET 2024-02-01T10:21:38.481 DEBUG:boto:Path: /vclfcn-16/?prefix=obj-test-zone1 2024-02-01T10:21:38.482 DEBUG:boto:Data: 2024-02-01T10:21:38.482 DEBUG:boto:Headers: {} 2024-02-01T10:21:38.482 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:21:38.482 DEBUG:boto:Port: 8001 2024-02-01T10:21:38.482 DEBUG:boto:Params: {} 2024-02-01T10:21:38.482 DEBUG:boto:Token: None 2024-02-01T10:21:38.482 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:38 GMT /vclfcn-16/ 2024-02-01T10:21:38.482 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lgaZ1mEIlG/hgqxRgZNZu4J9xuw= 2024-02-01T10:21:38.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lgaZ1mEIlG/hgqxRgZNZu4J9xuw=', 'Content-Length': '0'} 2024-02-01T10:21:38.487 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000883a917ad962142f-0065bb70b2-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:38.528 DEBUG:boto:b'vclfcn-16obj-test-zone11000falseobj-test-zone12024-02-01T10:20:49.965Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:21:38.529 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:21:38.529 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:21:38.529 DEBUG:boto:path=/vclfcn-16/?prefix=obj-test-zone1 2024-02-01T10:21:38.529 DEBUG:boto:auth_path=/vclfcn-16/?prefix=obj-test-zone1 2024-02-01T10:21:38.529 DEBUG:boto:Method: GET 2024-02-01T10:21:38.529 DEBUG:boto:Path: /vclfcn-16/?prefix=obj-test-zone1 2024-02-01T10:21:38.529 DEBUG:boto:Data: 2024-02-01T10:21:38.529 DEBUG:boto:Headers: {} 2024-02-01T10:21:38.529 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:21:38.530 DEBUG:boto:Port: 8000 2024-02-01T10:21:38.530 DEBUG:boto:Params: {} 2024-02-01T10:21:38.530 DEBUG:boto:Token: None 2024-02-01T10:21:38.530 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:38 GMT /vclfcn-16/ 2024-02-01T10:21:38.530 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lgaZ1mEIlG/hgqxRgZNZu4J9xuw= 2024-02-01T10:21:38.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lgaZ1mEIlG/hgqxRgZNZu4J9xuw=', 'Content-Length': '0'} 2024-02-01T10:21:38.534 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da510fce9688ad9d-0065bb70b2-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:38.535 DEBUG:boto:b'vclfcn-16obj-test-zone11000falseobj-test-zone12024-02-01T10:20:49.965Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:21:38.535 DEBUG:boto:path=/vclfcn-16/obj-test-zone1 2024-02-01T10:21:38.535 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1 2024-02-01T10:21:38.535 DEBUG:boto:Method: HEAD 2024-02-01T10:21:38.535 DEBUG:boto:Path: /vclfcn-16/obj-test-zone1 2024-02-01T10:21:38.535 DEBUG:boto:Data: 2024-02-01T10:21:38.535 DEBUG:boto:Headers: {} 2024-02-01T10:21:38.535 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:21:38.535 DEBUG:boto:Port: 8001 2024-02-01T10:21:38.536 DEBUG:boto:Params: {} 2024-02-01T10:21:38.536 DEBUG:boto:Token: None 2024-02-01T10:21:38.536 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:21:38 GMT /vclfcn-16/obj-test-zone1 2024-02-01T10:21:38.536 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QNM/wnlUAztIQk+kPnkugc1oefc= 2024-02-01T10:21:38.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QNM/wnlUAztIQk+kPnkugc1oefc=', 'Content-Length': '0'} 2024-02-01T10:21:38.540 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:49 GMT'), ('x-amz-version-id', 'lS9-sMOvyR-VcD.47BkPLymGmIyfWmf'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f4f8e4bc2d026167-0065bb70b2-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:38.540 DEBUG:boto:path=/vclfcn-16/obj-test-zone1 2024-02-01T10:21:38.540 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1 2024-02-01T10:21:38.540 DEBUG:boto:Method: HEAD 2024-02-01T10:21:38.541 DEBUG:boto:Path: /vclfcn-16/obj-test-zone1 2024-02-01T10:21:38.541 DEBUG:boto:Data: 2024-02-01T10:21:38.541 DEBUG:boto:Headers: {} 2024-02-01T10:21:38.541 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:21:38.541 DEBUG:boto:Port: 8000 2024-02-01T10:21:38.541 DEBUG:boto:Params: {} 2024-02-01T10:21:38.541 DEBUG:boto:Token: None 2024-02-01T10:21:38.541 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:21:38 GMT /vclfcn-16/obj-test-zone1 2024-02-01T10:21:38.541 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QNM/wnlUAztIQk+kPnkugc1oefc= 2024-02-01T10:21:38.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QNM/wnlUAztIQk+kPnkugc1oefc=', 'Content-Length': '0'} 2024-02-01T10:21:38.545 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:49 GMT'), ('x-amz-version-id', 'lS9-sMOvyR-VcD.47BkPLymGmIyfWmf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007e3509ce1f5b4ae4-0065bb70b2-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:38.545 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-02-01T10:21:38.545 DEBUG:boto:path=/vclfcn-16/obj-test-zone1 2024-02-01T10:21:38.545 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1 2024-02-01T10:21:38.545 DEBUG:boto:path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:38.545 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:38.546 DEBUG:boto:Method: GET 2024-02-01T10:21:38.546 DEBUG:boto:Path: /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:38.546 DEBUG:boto:Data: 2024-02-01T10:21:38.546 DEBUG:boto:Headers: {} 2024-02-01T10:21:38.546 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:21:38.546 DEBUG:boto:Port: 8001 2024-02-01T10:21:38.546 DEBUG:boto:Params: {} 2024-02-01T10:21:38.546 DEBUG:boto:Token: None 2024-02-01T10:21:38.546 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:38 GMT /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:38.547 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:IJnK/84kNNLN34TzV78VmyOLke0= 2024-02-01T10:21:38.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:IJnK/84kNNLN34TzV78VmyOLke0=', 'Content-Length': '0'} 2024-02-01T10:21:38.550 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:49 GMT'), ('x-amz-version-id', 'lS9-sMOvyR-VcD.47BkPLymGmIyfWmf'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000daa25173429d16a3-0065bb70b2-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:38.592 DEBUG:boto:path=/vclfcn-16/obj-test-zone1 2024-02-01T10:21:38.592 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1 2024-02-01T10:21:38.592 DEBUG:boto:path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:38.592 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:38.593 DEBUG:boto:Method: GET 2024-02-01T10:21:38.593 DEBUG:boto:Path: /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:38.593 DEBUG:boto:Data: 2024-02-01T10:21:38.593 DEBUG:boto:Headers: {} 2024-02-01T10:21:38.593 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:21:38.593 DEBUG:boto:Port: 8000 2024-02-01T10:21:38.593 DEBUG:boto:Params: {} 2024-02-01T10:21:38.593 DEBUG:boto:Token: None 2024-02-01T10:21:38.593 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:38 GMT /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:38.594 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:IJnK/84kNNLN34TzV78VmyOLke0= 2024-02-01T10:21:38.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:IJnK/84kNNLN34TzV78VmyOLke0=', 'Content-Length': '0'} 2024-02-01T10:21:38.596 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:49 GMT'), ('x-amz-version-id', 'lS9-sMOvyR-VcD.47BkPLymGmIyfWmf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ef2e965d7a6d1700-0065bb70b2-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:38.597 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:21:38.597 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:21:38.597 DEBUG:boto:path=/vclfcn-16/?prefix=obj-test-zone1 2024-02-01T10:21:38.597 DEBUG:boto:auth_path=/vclfcn-16/?prefix=obj-test-zone1 2024-02-01T10:21:38.597 DEBUG:boto:Method: GET 2024-02-01T10:21:38.597 DEBUG:boto:Path: /vclfcn-16/?prefix=obj-test-zone1 2024-02-01T10:21:38.597 DEBUG:boto:Data: 2024-02-01T10:21:38.598 DEBUG:boto:Headers: {} 2024-02-01T10:21:38.598 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:21:38.598 DEBUG:boto:Port: 8001 2024-02-01T10:21:38.598 DEBUG:boto:Params: {} 2024-02-01T10:21:38.598 DEBUG:boto:Token: None 2024-02-01T10:21:38.598 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:38 GMT /vclfcn-16/ 2024-02-01T10:21:38.598 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lgaZ1mEIlG/hgqxRgZNZu4J9xuw= 2024-02-01T10:21:38.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lgaZ1mEIlG/hgqxRgZNZu4J9xuw=', 'Content-Length': '0'} 2024-02-01T10:21:38.603 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ea332dbbe3563cc-0065bb70b2-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:38.644 DEBUG:boto:b'vclfcn-16obj-test-zone11000falseobj-test-zone12024-02-01T10:20:49.965Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:21:38.645 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:21:38.645 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:21:38.645 DEBUG:boto:path=/vclfcn-16/?prefix=obj-test-zone1 2024-02-01T10:21:38.645 DEBUG:boto:auth_path=/vclfcn-16/?prefix=obj-test-zone1 2024-02-01T10:21:38.645 DEBUG:boto:Method: GET 2024-02-01T10:21:38.645 DEBUG:boto:Path: /vclfcn-16/?prefix=obj-test-zone1 2024-02-01T10:21:38.645 DEBUG:boto:Data: 2024-02-01T10:21:38.645 DEBUG:boto:Headers: {} 2024-02-01T10:21:38.645 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:21:38.645 DEBUG:boto:Port: 8000 2024-02-01T10:21:38.645 DEBUG:boto:Params: {} 2024-02-01T10:21:38.645 DEBUG:boto:Token: None 2024-02-01T10:21:38.646 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:38 GMT /vclfcn-16/ 2024-02-01T10:21:38.646 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lgaZ1mEIlG/hgqxRgZNZu4J9xuw= 2024-02-01T10:21:38.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lgaZ1mEIlG/hgqxRgZNZu4J9xuw=', 'Content-Length': '0'} 2024-02-01T10:21:38.650 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091b575edf9cf41cc-0065bb70b2-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:38.650 DEBUG:boto:b'vclfcn-16obj-test-zone11000falseobj-test-zone12024-02-01T10:20:49.965Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:21:38.651 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-02-01T10:21:38.651 DEBUG:boto:path=/vclfcn-16/obj-test-zone2 2024-02-01T10:21:38.651 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2 2024-02-01T10:21:38.651 DEBUG:boto:path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:38.651 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:38.651 DEBUG:boto:Method: GET 2024-02-01T10:21:38.651 DEBUG:boto:Path: /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:38.651 DEBUG:boto:Data: 2024-02-01T10:21:38.652 DEBUG:boto:Headers: {} 2024-02-01T10:21:38.652 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:21:38.652 DEBUG:boto:Port: 8001 2024-02-01T10:21:38.652 DEBUG:boto:Params: {} 2024-02-01T10:21:38.652 DEBUG:boto:Token: None 2024-02-01T10:21:38.652 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:38 GMT /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:38.652 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:EE1G8wICkFKPLdDUhPJwEEfXvzw= 2024-02-01T10:21:38.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:EE1G8wICkFKPLdDUhPJwEEfXvzw=', 'Content-Length': '0'} 2024-02-01T10:21:38.655 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'mE6urSxcrRA9wdyuToa41.w2aTia9p1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000de88359406cdef2d-0065bb70b2-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:38.696 DEBUG:boto:path=/vclfcn-16/obj-test-zone2 2024-02-01T10:21:38.696 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2 2024-02-01T10:21:38.697 DEBUG:boto:path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:38.697 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:38.697 DEBUG:boto:Method: GET 2024-02-01T10:21:38.697 DEBUG:boto:Path: /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:38.697 DEBUG:boto:Data: 2024-02-01T10:21:38.697 DEBUG:boto:Headers: {} 2024-02-01T10:21:38.697 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:21:38.697 DEBUG:boto:Port: 8000 2024-02-01T10:21:38.697 DEBUG:boto:Params: {} 2024-02-01T10:21:38.697 DEBUG:boto:Token: None 2024-02-01T10:21:38.698 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:38 GMT /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:38.698 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:EE1G8wICkFKPLdDUhPJwEEfXvzw= 2024-02-01T10:21:38.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:EE1G8wICkFKPLdDUhPJwEEfXvzw=', 'Content-Length': '0'} 2024-02-01T10:21:38.701 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'mE6urSxcrRA9wdyuToa41.w2aTia9p1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000218ff8afdfe056f3-0065bb70b2-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:38.701 DEBUG:boto:path=/vclfcn-16/obj-test-zone2 2024-02-01T10:21:38.701 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2 2024-02-01T10:21:38.702 DEBUG:boto:path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:38.702 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:38.702 DEBUG:boto:Method: HEAD 2024-02-01T10:21:38.702 DEBUG:boto:Path: /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:38.702 DEBUG:boto:Data: 2024-02-01T10:21:38.702 DEBUG:boto:Headers: {} 2024-02-01T10:21:38.702 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:21:38.702 DEBUG:boto:Port: 8001 2024-02-01T10:21:38.702 DEBUG:boto:Params: {} 2024-02-01T10:21:38.703 DEBUG:boto:Token: None 2024-02-01T10:21:38.703 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:21:38 GMT /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:38.703 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:uRhJy1S2uhmv7507UqaCEj7vgtk= 2024-02-01T10:21:38.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:uRhJy1S2uhmv7507UqaCEj7vgtk=', 'Content-Length': '0'} 2024-02-01T10:21:38.706 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'mE6urSxcrRA9wdyuToa41.w2aTia9p1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004016341de317715b-0065bb70b2-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:38.706 DEBUG:boto:path=/vclfcn-16/obj-test-zone2 2024-02-01T10:21:38.706 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2 2024-02-01T10:21:38.706 DEBUG:boto:path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:38.706 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:38.706 DEBUG:boto:Method: HEAD 2024-02-01T10:21:38.706 DEBUG:boto:Path: /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:38.707 DEBUG:boto:Data: 2024-02-01T10:21:38.707 DEBUG:boto:Headers: {} 2024-02-01T10:21:38.707 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:21:38.707 DEBUG:boto:Port: 8000 2024-02-01T10:21:38.707 DEBUG:boto:Params: {} 2024-02-01T10:21:38.707 DEBUG:boto:Token: None 2024-02-01T10:21:38.707 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:21:38 GMT /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:38.707 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:uRhJy1S2uhmv7507UqaCEj7vgtk= 2024-02-01T10:21:38.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:uRhJy1S2uhmv7507UqaCEj7vgtk=', 'Content-Length': '0'} 2024-02-01T10:21:38.710 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'mE6urSxcrRA9wdyuToa41.w2aTia9p1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007f3c1594066b052e-0065bb70b2-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:38.710 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-02-01T10:21:38.710 DEBUG:boto:path=/vclfcn-16/obj-test-zone2 2024-02-01T10:21:38.711 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2 2024-02-01T10:21:38.711 DEBUG:boto:path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:38.711 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:38.711 DEBUG:boto:Method: GET 2024-02-01T10:21:38.711 DEBUG:boto:Path: /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:38.711 DEBUG:boto:Data: 2024-02-01T10:21:38.711 DEBUG:boto:Headers: {} 2024-02-01T10:21:38.711 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:21:38.711 DEBUG:boto:Port: 8001 2024-02-01T10:21:38.711 DEBUG:boto:Params: {} 2024-02-01T10:21:38.711 DEBUG:boto:Token: None 2024-02-01T10:21:38.712 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:38 GMT /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:38.712 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:EE1G8wICkFKPLdDUhPJwEEfXvzw= 2024-02-01T10:21:38.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:EE1G8wICkFKPLdDUhPJwEEfXvzw=', 'Content-Length': '0'} 2024-02-01T10:21:38.715 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'mE6urSxcrRA9wdyuToa41.w2aTia9p1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c9412ced06d7a26e-0065bb70b2-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:38.756 DEBUG:boto:path=/vclfcn-16/obj-test-zone2 2024-02-01T10:21:38.756 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2 2024-02-01T10:21:38.756 DEBUG:boto:path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:38.757 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:38.757 DEBUG:boto:Method: GET 2024-02-01T10:21:38.757 DEBUG:boto:Path: /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:38.757 DEBUG:boto:Data: 2024-02-01T10:21:38.757 DEBUG:boto:Headers: {} 2024-02-01T10:21:38.757 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:21:38.757 DEBUG:boto:Port: 8000 2024-02-01T10:21:38.757 DEBUG:boto:Params: {} 2024-02-01T10:21:38.757 DEBUG:boto:Token: None 2024-02-01T10:21:38.758 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:38 GMT /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:38.758 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:EE1G8wICkFKPLdDUhPJwEEfXvzw= 2024-02-01T10:21:38.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:EE1G8wICkFKPLdDUhPJwEEfXvzw=', 'Content-Length': '0'} 2024-02-01T10:21:38.761 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'mE6urSxcrRA9wdyuToa41.w2aTia9p1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000308076e0c1f83516-0065bb70b2-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:38.804 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:21:38.805 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:21:38.805 DEBUG:boto:path=/vclfcn-16/?prefix=obj-test-zone2 2024-02-01T10:21:38.805 DEBUG:boto:auth_path=/vclfcn-16/?prefix=obj-test-zone2 2024-02-01T10:21:38.805 DEBUG:boto:Method: GET 2024-02-01T10:21:38.805 DEBUG:boto:Path: /vclfcn-16/?prefix=obj-test-zone2 2024-02-01T10:21:38.806 DEBUG:boto:Data: 2024-02-01T10:21:38.806 DEBUG:boto:Headers: {} 2024-02-01T10:21:38.806 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:21:38.806 DEBUG:boto:Port: 8001 2024-02-01T10:21:38.806 DEBUG:boto:Params: {} 2024-02-01T10:21:38.806 DEBUG:boto:Token: None 2024-02-01T10:21:38.806 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:38 GMT /vclfcn-16/ 2024-02-01T10:21:38.807 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lgaZ1mEIlG/hgqxRgZNZu4J9xuw= 2024-02-01T10:21:38.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lgaZ1mEIlG/hgqxRgZNZu4J9xuw=', 'Content-Length': '0'} 2024-02-01T10:21:38.811 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a7b38c135a9c93f-0065bb70b2-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:38.811 DEBUG:boto:b'vclfcn-16obj-test-zone21000falseobj-test-zone22024-02-01T10:20:50.122Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:21:38.812 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:21:38.812 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:21:38.812 DEBUG:boto:path=/vclfcn-16/?prefix=obj-test-zone2 2024-02-01T10:21:38.812 DEBUG:boto:auth_path=/vclfcn-16/?prefix=obj-test-zone2 2024-02-01T10:21:38.812 DEBUG:boto:Method: GET 2024-02-01T10:21:38.813 DEBUG:boto:Path: /vclfcn-16/?prefix=obj-test-zone2 2024-02-01T10:21:38.813 DEBUG:boto:Data: 2024-02-01T10:21:38.813 DEBUG:boto:Headers: {} 2024-02-01T10:21:38.813 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:21:38.813 DEBUG:boto:Port: 8000 2024-02-01T10:21:38.813 DEBUG:boto:Params: {} 2024-02-01T10:21:38.813 DEBUG:boto:Token: None 2024-02-01T10:21:38.813 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:38 GMT /vclfcn-16/ 2024-02-01T10:21:38.813 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lgaZ1mEIlG/hgqxRgZNZu4J9xuw= 2024-02-01T10:21:38.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lgaZ1mEIlG/hgqxRgZNZu4J9xuw=', 'Content-Length': '0'} 2024-02-01T10:21:38.817 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000613e769a92c10426-0065bb70b2-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:38.818 DEBUG:boto:b'vclfcn-16obj-test-zone21000falseobj-test-zone22024-02-01T10:20:50.122Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:21:38.818 DEBUG:boto:path=/vclfcn-16/obj-test-zone2 2024-02-01T10:21:38.818 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2 2024-02-01T10:21:38.818 DEBUG:boto:Method: HEAD 2024-02-01T10:21:38.818 DEBUG:boto:Path: /vclfcn-16/obj-test-zone2 2024-02-01T10:21:38.819 DEBUG:boto:Data: 2024-02-01T10:21:38.819 DEBUG:boto:Headers: {} 2024-02-01T10:21:38.819 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:21:38.819 DEBUG:boto:Port: 8001 2024-02-01T10:21:38.819 DEBUG:boto:Params: {} 2024-02-01T10:21:38.820 DEBUG:boto:Token: None 2024-02-01T10:21:38.820 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:21:38 GMT /vclfcn-16/obj-test-zone2 2024-02-01T10:21:38.820 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:X3XBiRwUoEhXMl6OV5gXUCgYh8E= 2024-02-01T10:21:38.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:X3XBiRwUoEhXMl6OV5gXUCgYh8E=', 'Content-Length': '0'} 2024-02-01T10:21:38.824 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'mE6urSxcrRA9wdyuToa41.w2aTia9p1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008af7bc00f0d33560-0065bb70b2-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:38.825 DEBUG:boto:path=/vclfcn-16/obj-test-zone2 2024-02-01T10:21:38.825 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2 2024-02-01T10:21:38.825 DEBUG:boto:Method: HEAD 2024-02-01T10:21:38.825 DEBUG:boto:Path: /vclfcn-16/obj-test-zone2 2024-02-01T10:21:38.825 DEBUG:boto:Data: 2024-02-01T10:21:38.825 DEBUG:boto:Headers: {} 2024-02-01T10:21:38.825 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:21:38.825 DEBUG:boto:Port: 8000 2024-02-01T10:21:38.825 DEBUG:boto:Params: {} 2024-02-01T10:21:38.826 DEBUG:boto:Token: None 2024-02-01T10:21:38.826 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:21:38 GMT /vclfcn-16/obj-test-zone2 2024-02-01T10:21:38.826 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:X3XBiRwUoEhXMl6OV5gXUCgYh8E= 2024-02-01T10:21:38.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:X3XBiRwUoEhXMl6OV5gXUCgYh8E=', 'Content-Length': '0'} 2024-02-01T10:21:38.829 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'mE6urSxcrRA9wdyuToa41.w2aTia9p1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000031e7aae2796a0375-0065bb70b2-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:38.830 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-02-01T10:21:38.830 DEBUG:boto:path=/vclfcn-16/obj-test-zone2 2024-02-01T10:21:38.830 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2 2024-02-01T10:21:38.830 DEBUG:boto:path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:38.830 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:38.830 DEBUG:boto:Method: GET 2024-02-01T10:21:38.830 DEBUG:boto:Path: /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:38.830 DEBUG:boto:Data: 2024-02-01T10:21:38.830 DEBUG:boto:Headers: {} 2024-02-01T10:21:38.831 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:21:38.831 DEBUG:boto:Port: 8001 2024-02-01T10:21:38.831 DEBUG:boto:Params: {} 2024-02-01T10:21:38.831 DEBUG:boto:Token: None 2024-02-01T10:21:38.831 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:38 GMT /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:38.831 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:EE1G8wICkFKPLdDUhPJwEEfXvzw= 2024-02-01T10:21:38.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:EE1G8wICkFKPLdDUhPJwEEfXvzw=', 'Content-Length': '0'} 2024-02-01T10:21:38.834 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'mE6urSxcrRA9wdyuToa41.w2aTia9p1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000415d1be80a3312b4-0065bb70b2-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:38.880 DEBUG:boto:path=/vclfcn-16/obj-test-zone2 2024-02-01T10:21:38.880 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2 2024-02-01T10:21:38.880 DEBUG:boto:path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:38.880 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:38.880 DEBUG:boto:Method: GET 2024-02-01T10:21:38.881 DEBUG:boto:Path: /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:38.881 DEBUG:boto:Data: 2024-02-01T10:21:38.881 DEBUG:boto:Headers: {} 2024-02-01T10:21:38.881 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:21:38.881 DEBUG:boto:Port: 8000 2024-02-01T10:21:38.881 DEBUG:boto:Params: {} 2024-02-01T10:21:38.881 DEBUG:boto:Token: None 2024-02-01T10:21:38.881 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:38 GMT /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:38.882 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:EE1G8wICkFKPLdDUhPJwEEfXvzw= 2024-02-01T10:21:38.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:EE1G8wICkFKPLdDUhPJwEEfXvzw=', 'Content-Length': '0'} 2024-02-01T10:21:38.884 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'mE6urSxcrRA9wdyuToa41.w2aTia9p1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aaa889f4135278c5-0065bb70b2-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:38.928 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:21:38.928 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:21:38.928 DEBUG:boto:path=/vclfcn-16/?prefix=obj-test-zone2 2024-02-01T10:21:38.929 DEBUG:boto:auth_path=/vclfcn-16/?prefix=obj-test-zone2 2024-02-01T10:21:38.929 DEBUG:boto:Method: GET 2024-02-01T10:21:38.929 DEBUG:boto:Path: /vclfcn-16/?prefix=obj-test-zone2 2024-02-01T10:21:38.929 DEBUG:boto:Data: 2024-02-01T10:21:38.929 DEBUG:boto:Headers: {} 2024-02-01T10:21:38.929 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:21:38.929 DEBUG:boto:Port: 8001 2024-02-01T10:21:38.929 DEBUG:boto:Params: {} 2024-02-01T10:21:38.929 DEBUG:boto:Token: None 2024-02-01T10:21:38.929 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:38 GMT /vclfcn-16/ 2024-02-01T10:21:38.930 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lgaZ1mEIlG/hgqxRgZNZu4J9xuw= 2024-02-01T10:21:38.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lgaZ1mEIlG/hgqxRgZNZu4J9xuw=', 'Content-Length': '0'} 2024-02-01T10:21:38.934 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e7adf0f4faa1bf4-0065bb70b2-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:38.976 DEBUG:boto:b'vclfcn-16obj-test-zone21000falseobj-test-zone22024-02-01T10:20:50.122Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:21:38.977 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:21:38.977 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:21:38.977 DEBUG:boto:path=/vclfcn-16/?prefix=obj-test-zone2 2024-02-01T10:21:38.977 DEBUG:boto:auth_path=/vclfcn-16/?prefix=obj-test-zone2 2024-02-01T10:21:38.977 DEBUG:boto:Method: GET 2024-02-01T10:21:38.977 DEBUG:boto:Path: /vclfcn-16/?prefix=obj-test-zone2 2024-02-01T10:21:38.977 DEBUG:boto:Data: 2024-02-01T10:21:38.977 DEBUG:boto:Headers: {} 2024-02-01T10:21:38.977 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:21:38.978 DEBUG:boto:Port: 8000 2024-02-01T10:21:38.978 DEBUG:boto:Params: {} 2024-02-01T10:21:38.978 DEBUG:boto:Token: None 2024-02-01T10:21:38.978 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:38 GMT /vclfcn-16/ 2024-02-01T10:21:38.978 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lgaZ1mEIlG/hgqxRgZNZu4J9xuw= 2024-02-01T10:21:38.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lgaZ1mEIlG/hgqxRgZNZu4J9xuw=', 'Content-Length': '0'} 2024-02-01T10:21:38.982 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e774a33c1742a2e4-0065bb70b2-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:38.983 DEBUG:boto:b'vclfcn-16obj-test-zone21000falseobj-test-zone22024-02-01T10:20:50.122Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:21:38.983 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-02-01T10:21:38.983 DEBUG:boto:path=/vclfcn-16/obj-test-zone3 2024-02-01T10:21:38.983 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3 2024-02-01T10:21:38.983 DEBUG:boto:path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:38.983 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:38.983 DEBUG:boto:Method: GET 2024-02-01T10:21:38.983 DEBUG:boto:Path: /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:38.984 DEBUG:boto:Data: 2024-02-01T10:21:38.984 DEBUG:boto:Headers: {} 2024-02-01T10:21:38.984 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:21:38.984 DEBUG:boto:Port: 8001 2024-02-01T10:21:38.984 DEBUG:boto:Params: {} 2024-02-01T10:21:38.984 DEBUG:boto:Token: None 2024-02-01T10:21:38.984 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:38 GMT /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:38.984 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fZ+q3iobJxrx7KEQ9+NHtDtcu+8= 2024-02-01T10:21:38.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fZ+q3iobJxrx7KEQ9+NHtDtcu+8=', 'Content-Length': '0'} 2024-02-01T10:21:38.987 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000069ba8fd9078da34-0065bb70b2-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:38.987 DEBUG:boto:path=/vclfcn-16/obj-test-zone3 2024-02-01T10:21:38.987 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3 2024-02-01T10:21:38.988 DEBUG:boto:path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:38.988 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:38.988 DEBUG:boto:Method: GET 2024-02-01T10:21:38.988 DEBUG:boto:Path: /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:38.988 DEBUG:boto:Data: 2024-02-01T10:21:38.988 DEBUG:boto:Headers: {} 2024-02-01T10:21:38.988 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:21:38.988 DEBUG:boto:Port: 8000 2024-02-01T10:21:38.988 DEBUG:boto:Params: {} 2024-02-01T10:21:38.988 DEBUG:boto:Token: None 2024-02-01T10:21:38.988 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:38 GMT /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:38.989 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fZ+q3iobJxrx7KEQ9+NHtDtcu+8= 2024-02-01T10:21:38.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fZ+q3iobJxrx7KEQ9+NHtDtcu+8=', 'Content-Length': '0'} 2024-02-01T10:21:38.992 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e7b5c062fc00056d-0065bb70b2-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:38.992 DEBUG:boto:path=/vclfcn-16/obj-test-zone3 2024-02-01T10:21:38.992 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3 2024-02-01T10:21:38.992 DEBUG:boto:path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:38.992 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:38.992 DEBUG:boto:Method: HEAD 2024-02-01T10:21:38.993 DEBUG:boto:Path: /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:38.993 DEBUG:boto:Data: 2024-02-01T10:21:38.993 DEBUG:boto:Headers: {} 2024-02-01T10:21:38.993 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:21:38.993 DEBUG:boto:Port: 8001 2024-02-01T10:21:38.993 DEBUG:boto:Params: {} 2024-02-01T10:21:38.993 DEBUG:boto:Token: None 2024-02-01T10:21:38.993 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:21:38 GMT /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:38.993 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:2vQhkfryA6CuQP2XB5LJ/G2EeBg= 2024-02-01T10:21:38.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2vQhkfryA6CuQP2XB5LJ/G2EeBg=', 'Content-Length': '0'} 2024-02-01T10:21:38.996 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005f4d33e50de0d931-0065bb70b2-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:38.996 DEBUG:boto:path=/vclfcn-16/obj-test-zone3 2024-02-01T10:21:38.997 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3 2024-02-01T10:21:38.997 DEBUG:boto:path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:38.997 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:38.997 DEBUG:boto:Method: HEAD 2024-02-01T10:21:38.997 DEBUG:boto:Path: /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:38.997 DEBUG:boto:Data: 2024-02-01T10:21:38.997 DEBUG:boto:Headers: {} 2024-02-01T10:21:38.997 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:21:38.997 DEBUG:boto:Port: 8000 2024-02-01T10:21:38.997 DEBUG:boto:Params: {} 2024-02-01T10:21:38.998 DEBUG:boto:Token: None 2024-02-01T10:21:38.998 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:21:38 GMT /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:38.998 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:2vQhkfryA6CuQP2XB5LJ/G2EeBg= 2024-02-01T10:21:38.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2vQhkfryA6CuQP2XB5LJ/G2EeBg=', 'Content-Length': '0'} 2024-02-01T10:21:39.000 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000066147a19fc6a160-0065bb70b3-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:39.002 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-02-01T10:21:39.002 DEBUG:boto:path=/vclfcn-16/obj-test-zone3 2024-02-01T10:21:39.002 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3 2024-02-01T10:21:39.002 DEBUG:boto:path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:39.002 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:39.002 DEBUG:boto:Method: GET 2024-02-01T10:21:39.002 DEBUG:boto:Path: /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:39.002 DEBUG:boto:Data: 2024-02-01T10:21:39.002 DEBUG:boto:Headers: {} 2024-02-01T10:21:39.002 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:21:39.003 DEBUG:boto:Port: 8001 2024-02-01T10:21:39.003 DEBUG:boto:Params: {} 2024-02-01T10:21:39.003 DEBUG:boto:Token: None 2024-02-01T10:21:39.003 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:39 GMT /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:39.003 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:hylsSlGai0izYAevG42tgR+dE7c= 2024-02-01T10:21:39.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:hylsSlGai0izYAevG42tgR+dE7c=', 'Content-Length': '0'} 2024-02-01T10:21:39.006 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007c7ac5684e8351a2-0065bb70b3-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:39.006 DEBUG:boto:path=/vclfcn-16/obj-test-zone3 2024-02-01T10:21:39.006 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3 2024-02-01T10:21:39.006 DEBUG:boto:path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:39.006 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:39.006 DEBUG:boto:Method: GET 2024-02-01T10:21:39.006 DEBUG:boto:Path: /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:39.007 DEBUG:boto:Data: 2024-02-01T10:21:39.007 DEBUG:boto:Headers: {} 2024-02-01T10:21:39.007 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:21:39.007 DEBUG:boto:Port: 8000 2024-02-01T10:21:39.007 DEBUG:boto:Params: {} 2024-02-01T10:21:39.007 DEBUG:boto:Token: None 2024-02-01T10:21:39.007 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:39 GMT /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:39.007 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:hylsSlGai0izYAevG42tgR+dE7c= 2024-02-01T10:21:39.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:hylsSlGai0izYAevG42tgR+dE7c=', 'Content-Length': '0'} 2024-02-01T10:21:39.010 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ada0a4df64d0c664-0065bb70b3-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:39.052 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:21:39.052 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:21:39.052 DEBUG:boto:path=/vclfcn-16/?prefix=obj-test-zone3 2024-02-01T10:21:39.052 DEBUG:boto:auth_path=/vclfcn-16/?prefix=obj-test-zone3 2024-02-01T10:21:39.053 DEBUG:boto:Method: GET 2024-02-01T10:21:39.053 DEBUG:boto:Path: /vclfcn-16/?prefix=obj-test-zone3 2024-02-01T10:21:39.053 DEBUG:boto:Data: 2024-02-01T10:21:39.053 DEBUG:boto:Headers: {} 2024-02-01T10:21:39.053 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:21:39.053 DEBUG:boto:Port: 8001 2024-02-01T10:21:39.053 DEBUG:boto:Params: {} 2024-02-01T10:21:39.053 DEBUG:boto:Token: None 2024-02-01T10:21:39.053 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:39 GMT /vclfcn-16/ 2024-02-01T10:21:39.054 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BJAUjtG+XjZKlBb2XHwaqyhnS3E= 2024-02-01T10:21:39.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BJAUjtG+XjZKlBb2XHwaqyhnS3E=', 'Content-Length': '0'} 2024-02-01T10:21:39.058 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e58a9f9249c2f66-0065bb70b3-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:39.058 DEBUG:boto:b'vclfcn-16obj-test-zone31000falseobj-test-zone32024-02-01T10:20:50.297Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:21:39.059 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:21:39.059 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:21:39.059 DEBUG:boto:path=/vclfcn-16/?prefix=obj-test-zone3 2024-02-01T10:21:39.059 DEBUG:boto:auth_path=/vclfcn-16/?prefix=obj-test-zone3 2024-02-01T10:21:39.059 DEBUG:boto:Method: GET 2024-02-01T10:21:39.059 DEBUG:boto:Path: /vclfcn-16/?prefix=obj-test-zone3 2024-02-01T10:21:39.059 DEBUG:boto:Data: 2024-02-01T10:21:39.059 DEBUG:boto:Headers: {} 2024-02-01T10:21:39.059 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:21:39.060 DEBUG:boto:Port: 8000 2024-02-01T10:21:39.060 DEBUG:boto:Params: {} 2024-02-01T10:21:39.060 DEBUG:boto:Token: None 2024-02-01T10:21:39.060 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:39 GMT /vclfcn-16/ 2024-02-01T10:21:39.060 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BJAUjtG+XjZKlBb2XHwaqyhnS3E= 2024-02-01T10:21:39.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BJAUjtG+XjZKlBb2XHwaqyhnS3E=', 'Content-Length': '0'} 2024-02-01T10:21:39.064 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6ab5392f94e30a5-0065bb70b3-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:39.108 DEBUG:boto:b'vclfcn-16obj-test-zone31000falseobj-test-zone32024-02-01T10:20:50.297Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:21:39.109 DEBUG:boto:path=/vclfcn-16/obj-test-zone3 2024-02-01T10:21:39.109 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3 2024-02-01T10:21:39.109 DEBUG:boto:Method: HEAD 2024-02-01T10:21:39.109 DEBUG:boto:Path: /vclfcn-16/obj-test-zone3 2024-02-01T10:21:39.109 DEBUG:boto:Data: 2024-02-01T10:21:39.109 DEBUG:boto:Headers: {} 2024-02-01T10:21:39.109 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:21:39.109 DEBUG:boto:Port: 8001 2024-02-01T10:21:39.110 DEBUG:boto:Params: {} 2024-02-01T10:21:39.110 DEBUG:boto:Token: None 2024-02-01T10:21:39.110 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:21:39 GMT /vclfcn-16/obj-test-zone3 2024-02-01T10:21:39.110 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:R5V/r3okwIab57LyW8RVEnCcG1M= 2024-02-01T10:21:39.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:R5V/r3okwIab57LyW8RVEnCcG1M=', 'Content-Length': '0'} 2024-02-01T10:21:39.114 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c1d55fbfb7b0032a-0065bb70b3-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:39.114 DEBUG:boto:path=/vclfcn-16/obj-test-zone3 2024-02-01T10:21:39.114 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3 2024-02-01T10:21:39.114 DEBUG:boto:Method: HEAD 2024-02-01T10:21:39.114 DEBUG:boto:Path: /vclfcn-16/obj-test-zone3 2024-02-01T10:21:39.114 DEBUG:boto:Data: 2024-02-01T10:21:39.115 DEBUG:boto:Headers: {} 2024-02-01T10:21:39.115 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:21:39.115 DEBUG:boto:Port: 8000 2024-02-01T10:21:39.115 DEBUG:boto:Params: {} 2024-02-01T10:21:39.115 DEBUG:boto:Token: None 2024-02-01T10:21:39.115 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:21:39 GMT /vclfcn-16/obj-test-zone3 2024-02-01T10:21:39.115 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:R5V/r3okwIab57LyW8RVEnCcG1M= 2024-02-01T10:21:39.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:R5V/r3okwIab57LyW8RVEnCcG1M=', 'Content-Length': '0'} 2024-02-01T10:21:39.119 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000002fd985622424f93-0065bb70b3-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:39.119 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-02-01T10:21:39.119 DEBUG:boto:path=/vclfcn-16/obj-test-zone3 2024-02-01T10:21:39.119 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3 2024-02-01T10:21:39.119 DEBUG:boto:path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:39.119 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:39.119 DEBUG:boto:Method: GET 2024-02-01T10:21:39.120 DEBUG:boto:Path: /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:39.120 DEBUG:boto:Data: 2024-02-01T10:21:39.120 DEBUG:boto:Headers: {} 2024-02-01T10:21:39.120 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:21:39.120 DEBUG:boto:Port: 8001 2024-02-01T10:21:39.120 DEBUG:boto:Params: {} 2024-02-01T10:21:39.120 DEBUG:boto:Token: None 2024-02-01T10:21:39.120 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:39 GMT /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:39.120 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:hylsSlGai0izYAevG42tgR+dE7c= 2024-02-01T10:21:39.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:hylsSlGai0izYAevG42tgR+dE7c=', 'Content-Length': '0'} 2024-02-01T10:21:39.123 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002a20d4cefa641f58-0065bb70b3-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:39.164 DEBUG:boto:path=/vclfcn-16/obj-test-zone3 2024-02-01T10:21:39.164 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3 2024-02-01T10:21:39.164 DEBUG:boto:path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:39.164 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:39.165 DEBUG:boto:Method: GET 2024-02-01T10:21:39.165 DEBUG:boto:Path: /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:39.165 DEBUG:boto:Data: 2024-02-01T10:21:39.165 DEBUG:boto:Headers: {} 2024-02-01T10:21:39.165 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:21:39.165 DEBUG:boto:Port: 8000 2024-02-01T10:21:39.165 DEBUG:boto:Params: {} 2024-02-01T10:21:39.165 DEBUG:boto:Token: None 2024-02-01T10:21:39.165 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:39 GMT /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:39.166 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:hylsSlGai0izYAevG42tgR+dE7c= 2024-02-01T10:21:39.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:hylsSlGai0izYAevG42tgR+dE7c=', 'Content-Length': '0'} 2024-02-01T10:21:39.172 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005f46b054b6abdfe5-0065bb70b3-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:39.212 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:21:39.212 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:21:39.213 DEBUG:boto:path=/vclfcn-16/?prefix=obj-test-zone3 2024-02-01T10:21:39.213 DEBUG:boto:auth_path=/vclfcn-16/?prefix=obj-test-zone3 2024-02-01T10:21:39.213 DEBUG:boto:Method: GET 2024-02-01T10:21:39.213 DEBUG:boto:Path: /vclfcn-16/?prefix=obj-test-zone3 2024-02-01T10:21:39.213 DEBUG:boto:Data: 2024-02-01T10:21:39.213 DEBUG:boto:Headers: {} 2024-02-01T10:21:39.213 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:21:39.214 DEBUG:boto:Port: 8001 2024-02-01T10:21:39.214 DEBUG:boto:Params: {} 2024-02-01T10:21:39.214 DEBUG:boto:Token: None 2024-02-01T10:21:39.214 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:39 GMT /vclfcn-16/ 2024-02-01T10:21:39.215 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BJAUjtG+XjZKlBb2XHwaqyhnS3E= 2024-02-01T10:21:39.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BJAUjtG+XjZKlBb2XHwaqyhnS3E=', 'Content-Length': '0'} 2024-02-01T10:21:39.219 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f34c4638dbff432c-0065bb70b3-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:39.219 DEBUG:boto:b'vclfcn-16obj-test-zone31000falseobj-test-zone32024-02-01T10:20:50.297Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:21:39.220 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:21:39.220 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:21:39.220 DEBUG:boto:path=/vclfcn-16/?prefix=obj-test-zone3 2024-02-01T10:21:39.220 DEBUG:boto:auth_path=/vclfcn-16/?prefix=obj-test-zone3 2024-02-01T10:21:39.220 DEBUG:boto:Method: GET 2024-02-01T10:21:39.220 DEBUG:boto:Path: /vclfcn-16/?prefix=obj-test-zone3 2024-02-01T10:21:39.221 DEBUG:boto:Data: 2024-02-01T10:21:39.221 DEBUG:boto:Headers: {} 2024-02-01T10:21:39.221 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:21:39.221 DEBUG:boto:Port: 8000 2024-02-01T10:21:39.221 DEBUG:boto:Params: {} 2024-02-01T10:21:39.221 DEBUG:boto:Token: None 2024-02-01T10:21:39.221 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:39 GMT /vclfcn-16/ 2024-02-01T10:21:39.221 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BJAUjtG+XjZKlBb2XHwaqyhnS3E= 2024-02-01T10:21:39.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BJAUjtG+XjZKlBb2XHwaqyhnS3E=', 'Content-Length': '0'} 2024-02-01T10:21:39.225 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000abd9907001cfcdd6-0065bb70b3-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:39.268 DEBUG:boto:b'vclfcn-16obj-test-zone31000falseobj-test-zone32024-02-01T10:20:50.297Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:21:39.269 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-16 zones={test-zone3, test-zone1} 2024-02-01T10:21:39.269 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-16 zones={test-zone3, test-zone2} 2024-02-01T10:21:39.269 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:21:39.269 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:21:39.269 DEBUG:boto:Method: HEAD 2024-02-01T10:21:39.269 DEBUG:boto:Path: /vclfcn-16/ 2024-02-01T10:21:39.269 DEBUG:boto:Data: 2024-02-01T10:21:39.269 DEBUG:boto:Headers: {} 2024-02-01T10:21:39.269 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:21:39.269 DEBUG:boto:Port: 8001 2024-02-01T10:21:39.270 DEBUG:boto:Params: {} 2024-02-01T10:21:39.270 DEBUG:boto:Token: None 2024-02-01T10:21:39.270 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:21:39 GMT /vclfcn-16/ 2024-02-01T10:21:39.270 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mAkZHvF+qf96n3DoPfuv6FvDSBY= 2024-02-01T10:21:39.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mAkZHvF+qf96n3DoPfuv6FvDSBY=', 'Content-Length': '0'} 2024-02-01T10:21:39.274 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000053e8025ffa492c69-0065bb70b3-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:21:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:39.274 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:21:39.274 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:21:39.274 DEBUG:boto:Method: HEAD 2024-02-01T10:21:39.274 DEBUG:boto:Path: /vclfcn-16/ 2024-02-01T10:21:39.274 DEBUG:boto:Data: 2024-02-01T10:21:39.275 DEBUG:boto:Headers: {} 2024-02-01T10:21:39.275 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:21:39.275 DEBUG:boto:Port: 8000 2024-02-01T10:21:39.275 DEBUG:boto:Params: {} 2024-02-01T10:21:39.275 DEBUG:boto:Token: None 2024-02-01T10:21:39.275 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:21:39 GMT /vclfcn-16/ 2024-02-01T10:21:39.275 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mAkZHvF+qf96n3DoPfuv6FvDSBY= 2024-02-01T10:21:39.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mAkZHvF+qf96n3DoPfuv6FvDSBY=', 'Content-Length': '0'} 2024-02-01T10:21:39.279 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000098e1a444a84b334e-0065bb70b3-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:21:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:39.279 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:21:39.280 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:21:39.280 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:21:39.280 DEBUG:boto:path=/vclfcn-16/?versions&max-keys=999 2024-02-01T10:21:39.280 DEBUG:boto:auth_path=/vclfcn-16/?versions&max-keys=999 2024-02-01T10:21:39.280 DEBUG:boto:Method: GET 2024-02-01T10:21:39.280 DEBUG:boto:Path: /vclfcn-16/?versions&max-keys=999 2024-02-01T10:21:39.281 DEBUG:boto:Data: 2024-02-01T10:21:39.281 DEBUG:boto:Headers: {} 2024-02-01T10:21:39.281 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:21:39.281 DEBUG:boto:Port: 8001 2024-02-01T10:21:39.281 DEBUG:boto:Params: {} 2024-02-01T10:21:39.281 DEBUG:boto:Token: None 2024-02-01T10:21:39.282 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:39 GMT /vclfcn-16/?versions 2024-02-01T10:21:39.282 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NGqKYcXb1wheQ1qduWZ1nnTmxd4= 2024-02-01T10:21:39.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NGqKYcXb1wheQ1qduWZ1nnTmxd4=', 'Content-Length': '0'} 2024-02-01T10:21:39.286 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b36bee7c183f291-0065bb70b3-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:39.287 DEBUG:boto:b'vclfcn-16999falsedummyrQRpBowIFAsvxVDn92H30shYnKoGt4ntrue2024-02-01T10:19:58.958Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1lS9-sMOvyR-VcD.47BkPLymGmIyfWmftrue2024-02-01T10:20:49.965Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2mE6urSxcrRA9wdyuToa41.w2aTia9p1true2024-02-01T10:20:50.122Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjPtrue2024-02-01T10:20:50.297Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:21:39.287 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-02-01T10:21:39.287 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-02-01T10:21:39.288 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-02-01T10:21:39.288 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-02-01T10:21:39.288 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:21:39.288 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:21:39.288 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:21:39.288 DEBUG:boto:path=/vclfcn-16/?versions&max-keys=999 2024-02-01T10:21:39.288 DEBUG:boto:auth_path=/vclfcn-16/?versions&max-keys=999 2024-02-01T10:21:39.288 DEBUG:boto:Method: GET 2024-02-01T10:21:39.288 DEBUG:boto:Path: /vclfcn-16/?versions&max-keys=999 2024-02-01T10:21:39.288 DEBUG:boto:Data: 2024-02-01T10:21:39.289 DEBUG:boto:Headers: {} 2024-02-01T10:21:39.289 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:21:39.289 DEBUG:boto:Port: 8000 2024-02-01T10:21:39.289 DEBUG:boto:Params: {} 2024-02-01T10:21:39.289 DEBUG:boto:Token: None 2024-02-01T10:21:39.289 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:39 GMT /vclfcn-16/?versions 2024-02-01T10:21:39.289 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NGqKYcXb1wheQ1qduWZ1nnTmxd4= 2024-02-01T10:21:39.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NGqKYcXb1wheQ1qduWZ1nnTmxd4=', 'Content-Length': '0'} 2024-02-01T10:21:39.293 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6fc2b1fcef84e13-0065bb70b3-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:39.293 DEBUG:boto:b'vclfcn-16999falsedummyrQRpBowIFAsvxVDn92H30shYnKoGt4ntrue2024-02-01T10:19:58.958Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1lS9-sMOvyR-VcD.47BkPLymGmIyfWmftrue2024-02-01T10:20:49.965Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2mE6urSxcrRA9wdyuToa41.w2aTia9p1true2024-02-01T10:20:50.122Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjPtrue2024-02-01T10:20:50.297Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:21:39.294 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-02-01T10:21:39.294 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-02-01T10:21:39.294 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-02-01T10:21:39.294 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-02-01T10:21:39.295 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:21:39.295 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:21:39.295 DEBUG:boto:path=/vclfcn-16/?versions&max-keys=999 2024-02-01T10:21:39.295 DEBUG:boto:auth_path=/vclfcn-16/?versions&max-keys=999 2024-02-01T10:21:39.296 DEBUG:boto:Method: GET 2024-02-01T10:21:39.296 DEBUG:boto:Path: /vclfcn-16/?versions&max-keys=999 2024-02-01T10:21:39.296 DEBUG:boto:Data: 2024-02-01T10:21:39.296 DEBUG:boto:Headers: {} 2024-02-01T10:21:39.296 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:21:39.297 DEBUG:boto:Port: 8001 2024-02-01T10:21:39.297 DEBUG:boto:Params: {} 2024-02-01T10:21:39.297 DEBUG:boto:Token: None 2024-02-01T10:21:39.297 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:39 GMT /vclfcn-16/?versions 2024-02-01T10:21:39.298 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NGqKYcXb1wheQ1qduWZ1nnTmxd4= 2024-02-01T10:21:39.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NGqKYcXb1wheQ1qduWZ1nnTmxd4=', 'Content-Length': '0'} 2024-02-01T10:21:39.303 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000677403d8458256cb-0065bb70b3-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:39.303 DEBUG:boto:b'vclfcn-16999falsedummyrQRpBowIFAsvxVDn92H30shYnKoGt4ntrue2024-02-01T10:19:58.958Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1lS9-sMOvyR-VcD.47BkPLymGmIyfWmftrue2024-02-01T10:20:49.965Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2mE6urSxcrRA9wdyuToa41.w2aTia9p1true2024-02-01T10:20:50.122Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjPtrue2024-02-01T10:20:50.297Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:21:39.303 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:21:39.304 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:21:39.304 DEBUG:boto:path=/vclfcn-16/?versions&max-keys=999 2024-02-01T10:21:39.304 DEBUG:boto:auth_path=/vclfcn-16/?versions&max-keys=999 2024-02-01T10:21:39.304 DEBUG:boto:Method: GET 2024-02-01T10:21:39.304 DEBUG:boto:Path: /vclfcn-16/?versions&max-keys=999 2024-02-01T10:21:39.304 DEBUG:boto:Data: 2024-02-01T10:21:39.304 DEBUG:boto:Headers: {} 2024-02-01T10:21:39.304 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:21:39.305 DEBUG:boto:Port: 8000 2024-02-01T10:21:39.305 DEBUG:boto:Params: {} 2024-02-01T10:21:39.305 DEBUG:boto:Token: None 2024-02-01T10:21:39.305 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:39 GMT /vclfcn-16/?versions 2024-02-01T10:21:39.305 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NGqKYcXb1wheQ1qduWZ1nnTmxd4= 2024-02-01T10:21:39.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NGqKYcXb1wheQ1qduWZ1nnTmxd4=', 'Content-Length': '0'} 2024-02-01T10:21:39.310 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005fd8139ec68171de-0065bb70b3-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:39.310 DEBUG:boto:b'vclfcn-16999falsedummyrQRpBowIFAsvxVDn92H30shYnKoGt4ntrue2024-02-01T10:19:58.958Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1lS9-sMOvyR-VcD.47BkPLymGmIyfWmftrue2024-02-01T10:20:49.965Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2mE6urSxcrRA9wdyuToa41.w2aTia9p1true2024-02-01T10:20:50.122Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjPtrue2024-02-01T10:20:50.297Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:21:39.311 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-02-01T10:21:39.311 DEBUG:boto:path=/vclfcn-16/dummy 2024-02-01T10:21:39.311 DEBUG:boto:auth_path=/vclfcn-16/dummy 2024-02-01T10:21:39.311 DEBUG:boto:path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:39.311 DEBUG:boto:auth_path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:39.311 DEBUG:boto:Method: GET 2024-02-01T10:21:39.311 DEBUG:boto:Path: /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:39.311 DEBUG:boto:Data: 2024-02-01T10:21:39.311 DEBUG:boto:Headers: {} 2024-02-01T10:21:39.311 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:21:39.311 DEBUG:boto:Port: 8001 2024-02-01T10:21:39.312 DEBUG:boto:Params: {} 2024-02-01T10:21:39.312 DEBUG:boto:Token: None 2024-02-01T10:21:39.312 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:39 GMT /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:39.312 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:eNUBM/c+BNqamXYX9K/iBIGbPF8= 2024-02-01T10:21:39.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:eNUBM/c+BNqamXYX9K/iBIGbPF8=', 'Content-Length': '0'} 2024-02-01T10:21:39.315 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:19:58 GMT'), ('x-amz-version-id', 'rQRpBowIFAsvxVDn92H30shYnKoGt4n'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b1debd4665408d84-0065bb70b3-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:39.315 DEBUG:boto:path=/vclfcn-16/dummy 2024-02-01T10:21:39.316 DEBUG:boto:auth_path=/vclfcn-16/dummy 2024-02-01T10:21:39.316 DEBUG:boto:path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:39.316 DEBUG:boto:auth_path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:39.316 DEBUG:boto:Method: GET 2024-02-01T10:21:39.317 DEBUG:boto:Path: /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:39.317 DEBUG:boto:Data: 2024-02-01T10:21:39.317 DEBUG:boto:Headers: {} 2024-02-01T10:21:39.317 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:21:39.317 DEBUG:boto:Port: 8000 2024-02-01T10:21:39.318 DEBUG:boto:Params: {} 2024-02-01T10:21:39.318 DEBUG:boto:Token: None 2024-02-01T10:21:39.318 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:39 GMT /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:39.318 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:eNUBM/c+BNqamXYX9K/iBIGbPF8= 2024-02-01T10:21:39.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:eNUBM/c+BNqamXYX9K/iBIGbPF8=', 'Content-Length': '0'} 2024-02-01T10:21:39.322 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:19:58 GMT'), ('x-amz-version-id', 'rQRpBowIFAsvxVDn92H30shYnKoGt4n'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ea44b5e6e0f58c35-0065bb70b3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:39.322 DEBUG:boto:path=/vclfcn-16/dummy 2024-02-01T10:21:39.323 DEBUG:boto:auth_path=/vclfcn-16/dummy 2024-02-01T10:21:39.323 DEBUG:boto:path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:39.323 DEBUG:boto:auth_path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:39.323 DEBUG:boto:Method: HEAD 2024-02-01T10:21:39.323 DEBUG:boto:Path: /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:39.324 DEBUG:boto:Data: 2024-02-01T10:21:39.324 DEBUG:boto:Headers: {} 2024-02-01T10:21:39.324 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:21:39.324 DEBUG:boto:Port: 8001 2024-02-01T10:21:39.324 DEBUG:boto:Params: {} 2024-02-01T10:21:39.325 DEBUG:boto:Token: None 2024-02-01T10:21:39.325 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:21:39 GMT /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:39.325 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vIG7xeq9AyFHcMbkR0v9XvF9XYA= 2024-02-01T10:21:39.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vIG7xeq9AyFHcMbkR0v9XvF9XYA=', 'Content-Length': '0'} 2024-02-01T10:21:39.328 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:19:58 GMT'), ('x-amz-version-id', 'rQRpBowIFAsvxVDn92H30shYnKoGt4n'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000150571885cac022b-0065bb70b3-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:39.328 DEBUG:boto:path=/vclfcn-16/dummy 2024-02-01T10:21:39.328 DEBUG:boto:auth_path=/vclfcn-16/dummy 2024-02-01T10:21:39.329 DEBUG:boto:path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:39.329 DEBUG:boto:auth_path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:39.329 DEBUG:boto:Method: HEAD 2024-02-01T10:21:39.329 DEBUG:boto:Path: /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:39.329 DEBUG:boto:Data: 2024-02-01T10:21:39.329 DEBUG:boto:Headers: {} 2024-02-01T10:21:39.329 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:21:39.329 DEBUG:boto:Port: 8000 2024-02-01T10:21:39.329 DEBUG:boto:Params: {} 2024-02-01T10:21:39.329 DEBUG:boto:Token: None 2024-02-01T10:21:39.330 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:21:39 GMT /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:39.330 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vIG7xeq9AyFHcMbkR0v9XvF9XYA= 2024-02-01T10:21:39.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vIG7xeq9AyFHcMbkR0v9XvF9XYA=', 'Content-Length': '0'} 2024-02-01T10:21:39.333 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:19:58 GMT'), ('x-amz-version-id', 'rQRpBowIFAsvxVDn92H30shYnKoGt4n'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000dca2c0488f7462e8-0065bb70b3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:39.333 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-02-01T10:21:39.333 DEBUG:boto:path=/vclfcn-16/dummy 2024-02-01T10:21:39.333 DEBUG:boto:auth_path=/vclfcn-16/dummy 2024-02-01T10:21:39.333 DEBUG:boto:path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:39.333 DEBUG:boto:auth_path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:39.334 DEBUG:boto:Method: GET 2024-02-01T10:21:39.334 DEBUG:boto:Path: /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:39.334 DEBUG:boto:Data: 2024-02-01T10:21:39.334 DEBUG:boto:Headers: {} 2024-02-01T10:21:39.334 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:21:39.334 DEBUG:boto:Port: 8001 2024-02-01T10:21:39.334 DEBUG:boto:Params: {} 2024-02-01T10:21:39.334 DEBUG:boto:Token: None 2024-02-01T10:21:39.334 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:39 GMT /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:39.334 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:eNUBM/c+BNqamXYX9K/iBIGbPF8= 2024-02-01T10:21:39.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:eNUBM/c+BNqamXYX9K/iBIGbPF8=', 'Content-Length': '0'} 2024-02-01T10:21:39.337 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:19:58 GMT'), ('x-amz-version-id', 'rQRpBowIFAsvxVDn92H30shYnKoGt4n'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000062e78e5494c60f9e-0065bb70b3-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:39.337 DEBUG:boto:path=/vclfcn-16/dummy 2024-02-01T10:21:39.338 DEBUG:boto:auth_path=/vclfcn-16/dummy 2024-02-01T10:21:39.338 DEBUG:boto:path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:39.338 DEBUG:boto:auth_path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:39.338 DEBUG:boto:Method: GET 2024-02-01T10:21:39.338 DEBUG:boto:Path: /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:39.338 DEBUG:boto:Data: 2024-02-01T10:21:39.338 DEBUG:boto:Headers: {} 2024-02-01T10:21:39.338 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:21:39.338 DEBUG:boto:Port: 8000 2024-02-01T10:21:39.338 DEBUG:boto:Params: {} 2024-02-01T10:21:39.338 DEBUG:boto:Token: None 2024-02-01T10:21:39.339 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:39 GMT /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:39.339 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:eNUBM/c+BNqamXYX9K/iBIGbPF8= 2024-02-01T10:21:39.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:eNUBM/c+BNqamXYX9K/iBIGbPF8=', 'Content-Length': '0'} 2024-02-01T10:21:39.342 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:19:58 GMT'), ('x-amz-version-id', 'rQRpBowIFAsvxVDn92H30shYnKoGt4n'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e8c6b6fd79185eae-0065bb70b3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:39.343 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:21:39.343 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:21:39.343 DEBUG:boto:path=/vclfcn-16/?prefix=dummy 2024-02-01T10:21:39.343 DEBUG:boto:auth_path=/vclfcn-16/?prefix=dummy 2024-02-01T10:21:39.343 DEBUG:boto:Method: GET 2024-02-01T10:21:39.343 DEBUG:boto:Path: /vclfcn-16/?prefix=dummy 2024-02-01T10:21:39.343 DEBUG:boto:Data: 2024-02-01T10:21:39.343 DEBUG:boto:Headers: {} 2024-02-01T10:21:39.343 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:21:39.343 DEBUG:boto:Port: 8001 2024-02-01T10:21:39.343 DEBUG:boto:Params: {} 2024-02-01T10:21:39.344 DEBUG:boto:Token: None 2024-02-01T10:21:39.344 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:39 GMT /vclfcn-16/ 2024-02-01T10:21:39.344 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BJAUjtG+XjZKlBb2XHwaqyhnS3E= 2024-02-01T10:21:39.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BJAUjtG+XjZKlBb2XHwaqyhnS3E=', 'Content-Length': '0'} 2024-02-01T10:21:39.348 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002dfc0ca6c5558e17-0065bb70b3-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:39.392 DEBUG:boto:b'vclfcn-16dummy1000falsedummy2024-02-01T10:19:58.958Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:21:39.393 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:21:39.393 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:21:39.393 DEBUG:boto:path=/vclfcn-16/?prefix=dummy 2024-02-01T10:21:39.393 DEBUG:boto:auth_path=/vclfcn-16/?prefix=dummy 2024-02-01T10:21:39.393 DEBUG:boto:Method: GET 2024-02-01T10:21:39.393 DEBUG:boto:Path: /vclfcn-16/?prefix=dummy 2024-02-01T10:21:39.393 DEBUG:boto:Data: 2024-02-01T10:21:39.393 DEBUG:boto:Headers: {} 2024-02-01T10:21:39.393 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:21:39.393 DEBUG:boto:Port: 8000 2024-02-01T10:21:39.393 DEBUG:boto:Params: {} 2024-02-01T10:21:39.394 DEBUG:boto:Token: None 2024-02-01T10:21:39.394 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:39 GMT /vclfcn-16/ 2024-02-01T10:21:39.394 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BJAUjtG+XjZKlBb2XHwaqyhnS3E= 2024-02-01T10:21:39.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BJAUjtG+XjZKlBb2XHwaqyhnS3E=', 'Content-Length': '0'} 2024-02-01T10:21:39.398 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025815d0b8a027197-0065bb70b3-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:39.398 DEBUG:boto:b'vclfcn-16dummy1000falsedummy2024-02-01T10:19:58.958Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:21:39.398 DEBUG:boto:path=/vclfcn-16/dummy 2024-02-01T10:21:39.398 DEBUG:boto:auth_path=/vclfcn-16/dummy 2024-02-01T10:21:39.399 DEBUG:boto:Method: HEAD 2024-02-01T10:21:39.399 DEBUG:boto:Path: /vclfcn-16/dummy 2024-02-01T10:21:39.399 DEBUG:boto:Data: 2024-02-01T10:21:39.399 DEBUG:boto:Headers: {} 2024-02-01T10:21:39.399 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:21:39.399 DEBUG:boto:Port: 8001 2024-02-01T10:21:39.399 DEBUG:boto:Params: {} 2024-02-01T10:21:39.399 DEBUG:boto:Token: None 2024-02-01T10:21:39.399 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:21:39 GMT /vclfcn-16/dummy 2024-02-01T10:21:39.399 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:OQ3W1MlqUgWUrzE/DhHmFBZcq6o= 2024-02-01T10:21:39.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:OQ3W1MlqUgWUrzE/DhHmFBZcq6o=', 'Content-Length': '0'} 2024-02-01T10:21:39.403 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:19:58 GMT'), ('x-amz-version-id', 'rQRpBowIFAsvxVDn92H30shYnKoGt4n'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e96e3a15e199cdef-0065bb70b3-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:39.403 DEBUG:boto:path=/vclfcn-16/dummy 2024-02-01T10:21:39.403 DEBUG:boto:auth_path=/vclfcn-16/dummy 2024-02-01T10:21:39.404 DEBUG:boto:Method: HEAD 2024-02-01T10:21:39.404 DEBUG:boto:Path: /vclfcn-16/dummy 2024-02-01T10:21:39.404 DEBUG:boto:Data: 2024-02-01T10:21:39.404 DEBUG:boto:Headers: {} 2024-02-01T10:21:39.404 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:21:39.404 DEBUG:boto:Port: 8000 2024-02-01T10:21:39.404 DEBUG:boto:Params: {} 2024-02-01T10:21:39.404 DEBUG:boto:Token: None 2024-02-01T10:21:39.404 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:21:39 GMT /vclfcn-16/dummy 2024-02-01T10:21:39.404 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:OQ3W1MlqUgWUrzE/DhHmFBZcq6o= 2024-02-01T10:21:39.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:OQ3W1MlqUgWUrzE/DhHmFBZcq6o=', 'Content-Length': '0'} 2024-02-01T10:21:39.408 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:19:58 GMT'), ('x-amz-version-id', 'rQRpBowIFAsvxVDn92H30shYnKoGt4n'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009ddf5bed6e9d88a1-0065bb70b3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:39.408 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-02-01T10:21:39.408 DEBUG:boto:path=/vclfcn-16/dummy 2024-02-01T10:21:39.409 DEBUG:boto:auth_path=/vclfcn-16/dummy 2024-02-01T10:21:39.409 DEBUG:boto:path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:39.409 DEBUG:boto:auth_path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:39.409 DEBUG:boto:Method: GET 2024-02-01T10:21:39.409 DEBUG:boto:Path: /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:39.409 DEBUG:boto:Data: 2024-02-01T10:21:39.409 DEBUG:boto:Headers: {} 2024-02-01T10:21:39.409 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:21:39.409 DEBUG:boto:Port: 8001 2024-02-01T10:21:39.409 DEBUG:boto:Params: {} 2024-02-01T10:21:39.409 DEBUG:boto:Token: None 2024-02-01T10:21:39.410 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:39 GMT /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:39.410 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:eNUBM/c+BNqamXYX9K/iBIGbPF8= 2024-02-01T10:21:39.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:eNUBM/c+BNqamXYX9K/iBIGbPF8=', 'Content-Length': '0'} 2024-02-01T10:21:39.413 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:19:58 GMT'), ('x-amz-version-id', 'rQRpBowIFAsvxVDn92H30shYnKoGt4n'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d9213097bf794325-0065bb70b3-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:39.413 DEBUG:boto:path=/vclfcn-16/dummy 2024-02-01T10:21:39.413 DEBUG:boto:auth_path=/vclfcn-16/dummy 2024-02-01T10:21:39.413 DEBUG:boto:path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:39.413 DEBUG:boto:auth_path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:39.413 DEBUG:boto:Method: GET 2024-02-01T10:21:39.413 DEBUG:boto:Path: /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:39.413 DEBUG:boto:Data: 2024-02-01T10:21:39.413 DEBUG:boto:Headers: {} 2024-02-01T10:21:39.414 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:21:39.414 DEBUG:boto:Port: 8000 2024-02-01T10:21:39.414 DEBUG:boto:Params: {} 2024-02-01T10:21:39.414 DEBUG:boto:Token: None 2024-02-01T10:21:39.414 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:39 GMT /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:21:39.414 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:eNUBM/c+BNqamXYX9K/iBIGbPF8= 2024-02-01T10:21:39.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:eNUBM/c+BNqamXYX9K/iBIGbPF8=', 'Content-Length': '0'} 2024-02-01T10:21:39.417 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:19:58 GMT'), ('x-amz-version-id', 'rQRpBowIFAsvxVDn92H30shYnKoGt4n'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fbcbf313a65b5432-0065bb70b3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:39.417 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:21:39.417 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:21:39.418 DEBUG:boto:path=/vclfcn-16/?prefix=dummy 2024-02-01T10:21:39.418 DEBUG:boto:auth_path=/vclfcn-16/?prefix=dummy 2024-02-01T10:21:39.418 DEBUG:boto:Method: GET 2024-02-01T10:21:39.418 DEBUG:boto:Path: /vclfcn-16/?prefix=dummy 2024-02-01T10:21:39.418 DEBUG:boto:Data: 2024-02-01T10:21:39.418 DEBUG:boto:Headers: {} 2024-02-01T10:21:39.418 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:21:39.418 DEBUG:boto:Port: 8001 2024-02-01T10:21:39.418 DEBUG:boto:Params: {} 2024-02-01T10:21:39.418 DEBUG:boto:Token: None 2024-02-01T10:21:39.418 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:39 GMT /vclfcn-16/ 2024-02-01T10:21:39.419 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BJAUjtG+XjZKlBb2XHwaqyhnS3E= 2024-02-01T10:21:39.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BJAUjtG+XjZKlBb2XHwaqyhnS3E=', 'Content-Length': '0'} 2024-02-01T10:21:39.422 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e6704f3a150bbc7-0065bb70b3-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:39.423 DEBUG:boto:b'vclfcn-16dummy1000falsedummy2024-02-01T10:19:58.958Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:21:39.423 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:21:39.423 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:21:39.423 DEBUG:boto:path=/vclfcn-16/?prefix=dummy 2024-02-01T10:21:39.423 DEBUG:boto:auth_path=/vclfcn-16/?prefix=dummy 2024-02-01T10:21:39.423 DEBUG:boto:Method: GET 2024-02-01T10:21:39.423 DEBUG:boto:Path: /vclfcn-16/?prefix=dummy 2024-02-01T10:21:39.424 DEBUG:boto:Data: 2024-02-01T10:21:39.424 DEBUG:boto:Headers: {} 2024-02-01T10:21:39.424 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:21:39.424 DEBUG:boto:Port: 8000 2024-02-01T10:21:39.424 DEBUG:boto:Params: {} 2024-02-01T10:21:39.424 DEBUG:boto:Token: None 2024-02-01T10:21:39.424 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:39 GMT /vclfcn-16/ 2024-02-01T10:21:39.424 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BJAUjtG+XjZKlBb2XHwaqyhnS3E= 2024-02-01T10:21:39.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BJAUjtG+XjZKlBb2XHwaqyhnS3E=', 'Content-Length': '0'} 2024-02-01T10:21:39.429 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007abfb1881343af76-0065bb70b3-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:39.476 DEBUG:boto:b'vclfcn-16dummy1000falsedummy2024-02-01T10:19:58.958Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:21:39.477 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-02-01T10:21:39.477 DEBUG:boto:path=/vclfcn-16/obj-test-zone1 2024-02-01T10:21:39.477 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1 2024-02-01T10:21:39.477 DEBUG:boto:path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:39.477 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:39.477 DEBUG:boto:Method: GET 2024-02-01T10:21:39.477 DEBUG:boto:Path: /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:39.477 DEBUG:boto:Data: 2024-02-01T10:21:39.477 DEBUG:boto:Headers: {} 2024-02-01T10:21:39.477 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:21:39.477 DEBUG:boto:Port: 8001 2024-02-01T10:21:39.478 DEBUG:boto:Params: {} 2024-02-01T10:21:39.478 DEBUG:boto:Token: None 2024-02-01T10:21:39.478 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:39 GMT /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:39.478 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FyrHwAP0OrcpxpIcR2lSC+xe9U4= 2024-02-01T10:21:39.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FyrHwAP0OrcpxpIcR2lSC+xe9U4=', 'Content-Length': '0'} 2024-02-01T10:21:39.481 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:49 GMT'), ('x-amz-version-id', 'lS9-sMOvyR-VcD.47BkPLymGmIyfWmf'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005b5a8ba3bda2adb5-0065bb70b3-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:39.481 DEBUG:boto:path=/vclfcn-16/obj-test-zone1 2024-02-01T10:21:39.481 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1 2024-02-01T10:21:39.482 DEBUG:boto:path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:39.482 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:39.482 DEBUG:boto:Method: GET 2024-02-01T10:21:39.482 DEBUG:boto:Path: /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:39.482 DEBUG:boto:Data: 2024-02-01T10:21:39.482 DEBUG:boto:Headers: {} 2024-02-01T10:21:39.482 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:21:39.482 DEBUG:boto:Port: 8000 2024-02-01T10:21:39.482 DEBUG:boto:Params: {} 2024-02-01T10:21:39.482 DEBUG:boto:Token: None 2024-02-01T10:21:39.482 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:39 GMT /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:39.483 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FyrHwAP0OrcpxpIcR2lSC+xe9U4= 2024-02-01T10:21:39.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FyrHwAP0OrcpxpIcR2lSC+xe9U4=', 'Content-Length': '0'} 2024-02-01T10:21:39.485 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:49 GMT'), ('x-amz-version-id', 'lS9-sMOvyR-VcD.47BkPLymGmIyfWmf'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f11962732ebb17ae-0065bb70b3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:39.486 DEBUG:boto:path=/vclfcn-16/obj-test-zone1 2024-02-01T10:21:39.486 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1 2024-02-01T10:21:39.486 DEBUG:boto:path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:39.486 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:39.486 DEBUG:boto:Method: HEAD 2024-02-01T10:21:39.486 DEBUG:boto:Path: /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:39.486 DEBUG:boto:Data: 2024-02-01T10:21:39.486 DEBUG:boto:Headers: {} 2024-02-01T10:21:39.486 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:21:39.486 DEBUG:boto:Port: 8001 2024-02-01T10:21:39.487 DEBUG:boto:Params: {} 2024-02-01T10:21:39.487 DEBUG:boto:Token: None 2024-02-01T10:21:39.487 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:21:39 GMT /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:39.487 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:q3cUNbPCEr7KkysrLDH8y1YvYrk= 2024-02-01T10:21:39.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:q3cUNbPCEr7KkysrLDH8y1YvYrk=', 'Content-Length': '0'} 2024-02-01T10:21:39.489 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:49 GMT'), ('x-amz-version-id', 'lS9-sMOvyR-VcD.47BkPLymGmIyfWmf'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000139132eaf5074ea6-0065bb70b3-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:39.490 DEBUG:boto:path=/vclfcn-16/obj-test-zone1 2024-02-01T10:21:39.490 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1 2024-02-01T10:21:39.490 DEBUG:boto:path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:39.490 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:39.490 DEBUG:boto:Method: HEAD 2024-02-01T10:21:39.490 DEBUG:boto:Path: /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:39.490 DEBUG:boto:Data: 2024-02-01T10:21:39.490 DEBUG:boto:Headers: {} 2024-02-01T10:21:39.490 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:21:39.491 DEBUG:boto:Port: 8000 2024-02-01T10:21:39.491 DEBUG:boto:Params: {} 2024-02-01T10:21:39.491 DEBUG:boto:Token: None 2024-02-01T10:21:39.491 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:21:39 GMT /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:39.491 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:q3cUNbPCEr7KkysrLDH8y1YvYrk= 2024-02-01T10:21:39.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:q3cUNbPCEr7KkysrLDH8y1YvYrk=', 'Content-Length': '0'} 2024-02-01T10:21:39.494 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:49 GMT'), ('x-amz-version-id', 'lS9-sMOvyR-VcD.47BkPLymGmIyfWmf'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009caf82ad9a9d3c8f-0065bb70b3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:39.494 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-02-01T10:21:39.494 DEBUG:boto:path=/vclfcn-16/obj-test-zone1 2024-02-01T10:21:39.494 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1 2024-02-01T10:21:39.495 DEBUG:boto:path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:39.495 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:39.495 DEBUG:boto:Method: GET 2024-02-01T10:21:39.495 DEBUG:boto:Path: /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:39.495 DEBUG:boto:Data: 2024-02-01T10:21:39.495 DEBUG:boto:Headers: {} 2024-02-01T10:21:39.495 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:21:39.495 DEBUG:boto:Port: 8001 2024-02-01T10:21:39.495 DEBUG:boto:Params: {} 2024-02-01T10:21:39.495 DEBUG:boto:Token: None 2024-02-01T10:21:39.495 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:39 GMT /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:39.496 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FyrHwAP0OrcpxpIcR2lSC+xe9U4= 2024-02-01T10:21:39.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FyrHwAP0OrcpxpIcR2lSC+xe9U4=', 'Content-Length': '0'} 2024-02-01T10:21:39.498 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:49 GMT'), ('x-amz-version-id', 'lS9-sMOvyR-VcD.47BkPLymGmIyfWmf'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000efe2eb3bf5fb7b8a-0065bb70b3-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:39.544 DEBUG:boto:path=/vclfcn-16/obj-test-zone1 2024-02-01T10:21:39.544 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1 2024-02-01T10:21:39.544 DEBUG:boto:path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:39.545 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:39.545 DEBUG:boto:Method: GET 2024-02-01T10:21:39.545 DEBUG:boto:Path: /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:39.545 DEBUG:boto:Data: 2024-02-01T10:21:39.545 DEBUG:boto:Headers: {} 2024-02-01T10:21:39.545 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:21:39.546 DEBUG:boto:Port: 8000 2024-02-01T10:21:39.546 DEBUG:boto:Params: {} 2024-02-01T10:21:39.546 DEBUG:boto:Token: None 2024-02-01T10:21:39.546 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:39 GMT /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:39.546 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FyrHwAP0OrcpxpIcR2lSC+xe9U4= 2024-02-01T10:21:39.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FyrHwAP0OrcpxpIcR2lSC+xe9U4=', 'Content-Length': '0'} 2024-02-01T10:21:39.550 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:49 GMT'), ('x-amz-version-id', 'lS9-sMOvyR-VcD.47BkPLymGmIyfWmf'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000048a1b3443d7aeec0-0065bb70b3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:39.550 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:21:39.550 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:21:39.550 DEBUG:boto:path=/vclfcn-16/?prefix=obj-test-zone1 2024-02-01T10:21:39.550 DEBUG:boto:auth_path=/vclfcn-16/?prefix=obj-test-zone1 2024-02-01T10:21:39.550 DEBUG:boto:Method: GET 2024-02-01T10:21:39.550 DEBUG:boto:Path: /vclfcn-16/?prefix=obj-test-zone1 2024-02-01T10:21:39.551 DEBUG:boto:Data: 2024-02-01T10:21:39.551 DEBUG:boto:Headers: {} 2024-02-01T10:21:39.551 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:21:39.551 DEBUG:boto:Port: 8001 2024-02-01T10:21:39.551 DEBUG:boto:Params: {} 2024-02-01T10:21:39.551 DEBUG:boto:Token: None 2024-02-01T10:21:39.551 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:39 GMT /vclfcn-16/ 2024-02-01T10:21:39.551 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BJAUjtG+XjZKlBb2XHwaqyhnS3E= 2024-02-01T10:21:39.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BJAUjtG+XjZKlBb2XHwaqyhnS3E=', 'Content-Length': '0'} 2024-02-01T10:21:39.555 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000410dc9765c14e76e-0065bb70b3-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:39.556 DEBUG:boto:b'vclfcn-16obj-test-zone11000falseobj-test-zone12024-02-01T10:20:49.965Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:21:39.556 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:21:39.556 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:21:39.556 DEBUG:boto:path=/vclfcn-16/?prefix=obj-test-zone1 2024-02-01T10:21:39.556 DEBUG:boto:auth_path=/vclfcn-16/?prefix=obj-test-zone1 2024-02-01T10:21:39.557 DEBUG:boto:Method: GET 2024-02-01T10:21:39.557 DEBUG:boto:Path: /vclfcn-16/?prefix=obj-test-zone1 2024-02-01T10:21:39.557 DEBUG:boto:Data: 2024-02-01T10:21:39.557 DEBUG:boto:Headers: {} 2024-02-01T10:21:39.557 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:21:39.557 DEBUG:boto:Port: 8000 2024-02-01T10:21:39.557 DEBUG:boto:Params: {} 2024-02-01T10:21:39.557 DEBUG:boto:Token: None 2024-02-01T10:21:39.557 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:39 GMT /vclfcn-16/ 2024-02-01T10:21:39.557 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BJAUjtG+XjZKlBb2XHwaqyhnS3E= 2024-02-01T10:21:39.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BJAUjtG+XjZKlBb2XHwaqyhnS3E=', 'Content-Length': '0'} 2024-02-01T10:21:39.562 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000329f249611ba8fe-0065bb70b3-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:39.604 DEBUG:boto:b'vclfcn-16obj-test-zone11000falseobj-test-zone12024-02-01T10:20:49.965Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:21:39.605 DEBUG:boto:path=/vclfcn-16/obj-test-zone1 2024-02-01T10:21:39.605 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1 2024-02-01T10:21:39.605 DEBUG:boto:Method: HEAD 2024-02-01T10:21:39.605 DEBUG:boto:Path: /vclfcn-16/obj-test-zone1 2024-02-01T10:21:39.605 DEBUG:boto:Data: 2024-02-01T10:21:39.605 DEBUG:boto:Headers: {} 2024-02-01T10:21:39.606 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:21:39.606 DEBUG:boto:Port: 8001 2024-02-01T10:21:39.606 DEBUG:boto:Params: {} 2024-02-01T10:21:39.606 DEBUG:boto:Token: None 2024-02-01T10:21:39.606 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:21:39 GMT /vclfcn-16/obj-test-zone1 2024-02-01T10:21:39.606 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:j0h5PgDblO8IaHlVkeclKt/bGNo= 2024-02-01T10:21:39.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:j0h5PgDblO8IaHlVkeclKt/bGNo=', 'Content-Length': '0'} 2024-02-01T10:21:39.611 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:49 GMT'), ('x-amz-version-id', 'lS9-sMOvyR-VcD.47BkPLymGmIyfWmf'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c7efa5476cdcbabd-0065bb70b3-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:39.611 DEBUG:boto:path=/vclfcn-16/obj-test-zone1 2024-02-01T10:21:39.611 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1 2024-02-01T10:21:39.611 DEBUG:boto:Method: HEAD 2024-02-01T10:21:39.611 DEBUG:boto:Path: /vclfcn-16/obj-test-zone1 2024-02-01T10:21:39.611 DEBUG:boto:Data: 2024-02-01T10:21:39.611 DEBUG:boto:Headers: {} 2024-02-01T10:21:39.612 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:21:39.612 DEBUG:boto:Port: 8000 2024-02-01T10:21:39.612 DEBUG:boto:Params: {} 2024-02-01T10:21:39.612 DEBUG:boto:Token: None 2024-02-01T10:21:39.612 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:21:39 GMT /vclfcn-16/obj-test-zone1 2024-02-01T10:21:39.612 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:j0h5PgDblO8IaHlVkeclKt/bGNo= 2024-02-01T10:21:39.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:j0h5PgDblO8IaHlVkeclKt/bGNo=', 'Content-Length': '0'} 2024-02-01T10:21:39.616 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:49 GMT'), ('x-amz-version-id', 'lS9-sMOvyR-VcD.47BkPLymGmIyfWmf'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e716ca0c75fe8c49-0065bb70b3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:39.616 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-02-01T10:21:39.616 DEBUG:boto:path=/vclfcn-16/obj-test-zone1 2024-02-01T10:21:39.617 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1 2024-02-01T10:21:39.617 DEBUG:boto:path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:39.617 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:39.617 DEBUG:boto:Method: GET 2024-02-01T10:21:39.617 DEBUG:boto:Path: /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:39.617 DEBUG:boto:Data: 2024-02-01T10:21:39.617 DEBUG:boto:Headers: {} 2024-02-01T10:21:39.617 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:21:39.617 DEBUG:boto:Port: 8001 2024-02-01T10:21:39.618 DEBUG:boto:Params: {} 2024-02-01T10:21:39.618 DEBUG:boto:Token: None 2024-02-01T10:21:39.618 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:39 GMT /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:39.618 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FyrHwAP0OrcpxpIcR2lSC+xe9U4= 2024-02-01T10:21:39.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FyrHwAP0OrcpxpIcR2lSC+xe9U4=', 'Content-Length': '0'} 2024-02-01T10:21:39.621 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:49 GMT'), ('x-amz-version-id', 'lS9-sMOvyR-VcD.47BkPLymGmIyfWmf'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006e58d359d9d5df72-0065bb70b3-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:39.668 DEBUG:boto:path=/vclfcn-16/obj-test-zone1 2024-02-01T10:21:39.668 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1 2024-02-01T10:21:39.668 DEBUG:boto:path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:39.669 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:39.669 DEBUG:boto:Method: GET 2024-02-01T10:21:39.669 DEBUG:boto:Path: /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:39.669 DEBUG:boto:Data: 2024-02-01T10:21:39.669 DEBUG:boto:Headers: {} 2024-02-01T10:21:39.669 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:21:39.669 DEBUG:boto:Port: 8000 2024-02-01T10:21:39.669 DEBUG:boto:Params: {} 2024-02-01T10:21:39.669 DEBUG:boto:Token: None 2024-02-01T10:21:39.670 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:39 GMT /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:21:39.670 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FyrHwAP0OrcpxpIcR2lSC+xe9U4= 2024-02-01T10:21:39.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FyrHwAP0OrcpxpIcR2lSC+xe9U4=', 'Content-Length': '0'} 2024-02-01T10:21:39.673 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:49 GMT'), ('x-amz-version-id', 'lS9-sMOvyR-VcD.47BkPLymGmIyfWmf'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004fe8d5423e6fc3a3-0065bb70b3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:39.673 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:21:39.673 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:21:39.673 DEBUG:boto:path=/vclfcn-16/?prefix=obj-test-zone1 2024-02-01T10:21:39.674 DEBUG:boto:auth_path=/vclfcn-16/?prefix=obj-test-zone1 2024-02-01T10:21:39.674 DEBUG:boto:Method: GET 2024-02-01T10:21:39.674 DEBUG:boto:Path: /vclfcn-16/?prefix=obj-test-zone1 2024-02-01T10:21:39.674 DEBUG:boto:Data: 2024-02-01T10:21:39.674 DEBUG:boto:Headers: {} 2024-02-01T10:21:39.674 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:21:39.674 DEBUG:boto:Port: 8001 2024-02-01T10:21:39.674 DEBUG:boto:Params: {} 2024-02-01T10:21:39.674 DEBUG:boto:Token: None 2024-02-01T10:21:39.675 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:39 GMT /vclfcn-16/ 2024-02-01T10:21:39.675 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BJAUjtG+XjZKlBb2XHwaqyhnS3E= 2024-02-01T10:21:39.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BJAUjtG+XjZKlBb2XHwaqyhnS3E=', 'Content-Length': '0'} 2024-02-01T10:21:39.679 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a78c2362a4a05b05-0065bb70b3-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:39.724 DEBUG:boto:b'vclfcn-16obj-test-zone11000falseobj-test-zone12024-02-01T10:20:49.965Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:21:39.724 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:21:39.725 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:21:39.725 DEBUG:boto:path=/vclfcn-16/?prefix=obj-test-zone1 2024-02-01T10:21:39.725 DEBUG:boto:auth_path=/vclfcn-16/?prefix=obj-test-zone1 2024-02-01T10:21:39.725 DEBUG:boto:Method: GET 2024-02-01T10:21:39.725 DEBUG:boto:Path: /vclfcn-16/?prefix=obj-test-zone1 2024-02-01T10:21:39.725 DEBUG:boto:Data: 2024-02-01T10:21:39.725 DEBUG:boto:Headers: {} 2024-02-01T10:21:39.725 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:21:39.725 DEBUG:boto:Port: 8000 2024-02-01T10:21:39.725 DEBUG:boto:Params: {} 2024-02-01T10:21:39.726 DEBUG:boto:Token: None 2024-02-01T10:21:39.726 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:39 GMT /vclfcn-16/ 2024-02-01T10:21:39.726 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BJAUjtG+XjZKlBb2XHwaqyhnS3E= 2024-02-01T10:21:39.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BJAUjtG+XjZKlBb2XHwaqyhnS3E=', 'Content-Length': '0'} 2024-02-01T10:21:39.730 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000893f1002f16e8230-0065bb70b3-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:39.731 DEBUG:boto:b'vclfcn-16obj-test-zone11000falseobj-test-zone12024-02-01T10:20:49.965Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:21:39.731 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-02-01T10:21:39.731 DEBUG:boto:path=/vclfcn-16/obj-test-zone2 2024-02-01T10:21:39.731 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2 2024-02-01T10:21:39.731 DEBUG:boto:path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:39.732 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:39.732 DEBUG:boto:Method: GET 2024-02-01T10:21:39.732 DEBUG:boto:Path: /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:39.732 DEBUG:boto:Data: 2024-02-01T10:21:39.732 DEBUG:boto:Headers: {} 2024-02-01T10:21:39.732 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:21:39.732 DEBUG:boto:Port: 8001 2024-02-01T10:21:39.732 DEBUG:boto:Params: {} 2024-02-01T10:21:39.732 DEBUG:boto:Token: None 2024-02-01T10:21:39.733 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:39 GMT /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:39.733 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cJd+bRXSJMx8pGj/1EijcfBZGR4= 2024-02-01T10:21:39.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cJd+bRXSJMx8pGj/1EijcfBZGR4=', 'Content-Length': '0'} 2024-02-01T10:21:39.735 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'mE6urSxcrRA9wdyuToa41.w2aTia9p1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002de58eb3bc61ccc2-0065bb70b3-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:39.780 DEBUG:boto:path=/vclfcn-16/obj-test-zone2 2024-02-01T10:21:39.780 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2 2024-02-01T10:21:39.781 DEBUG:boto:path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:39.781 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:39.781 DEBUG:boto:Method: GET 2024-02-01T10:21:39.781 DEBUG:boto:Path: /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:39.781 DEBUG:boto:Data: 2024-02-01T10:21:39.781 DEBUG:boto:Headers: {} 2024-02-01T10:21:39.782 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:21:39.782 DEBUG:boto:Port: 8000 2024-02-01T10:21:39.782 DEBUG:boto:Params: {} 2024-02-01T10:21:39.782 DEBUG:boto:Token: None 2024-02-01T10:21:39.782 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:39 GMT /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:39.783 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cJd+bRXSJMx8pGj/1EijcfBZGR4= 2024-02-01T10:21:39.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cJd+bRXSJMx8pGj/1EijcfBZGR4=', 'Content-Length': '0'} 2024-02-01T10:21:39.787 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'mE6urSxcrRA9wdyuToa41.w2aTia9p1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c6f887fab16fda7d-0065bb70b3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:39.787 DEBUG:boto:path=/vclfcn-16/obj-test-zone2 2024-02-01T10:21:39.788 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2 2024-02-01T10:21:39.788 DEBUG:boto:path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:39.788 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:39.788 DEBUG:boto:Method: HEAD 2024-02-01T10:21:39.788 DEBUG:boto:Path: /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:39.788 DEBUG:boto:Data: 2024-02-01T10:21:39.789 DEBUG:boto:Headers: {} 2024-02-01T10:21:39.789 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:21:39.789 DEBUG:boto:Port: 8001 2024-02-01T10:21:39.789 DEBUG:boto:Params: {} 2024-02-01T10:21:39.789 DEBUG:boto:Token: None 2024-02-01T10:21:39.789 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:21:39 GMT /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:39.790 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8yPEEOYbyn7m0hyJq3WWzPlX/SE= 2024-02-01T10:21:39.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8yPEEOYbyn7m0hyJq3WWzPlX/SE=', 'Content-Length': '0'} 2024-02-01T10:21:39.793 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'mE6urSxcrRA9wdyuToa41.w2aTia9p1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f3d07fd2abb17692-0065bb70b3-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:39.793 DEBUG:boto:path=/vclfcn-16/obj-test-zone2 2024-02-01T10:21:39.793 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2 2024-02-01T10:21:39.794 DEBUG:boto:path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:39.794 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:39.794 DEBUG:boto:Method: HEAD 2024-02-01T10:21:39.794 DEBUG:boto:Path: /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:39.794 DEBUG:boto:Data: 2024-02-01T10:21:39.794 DEBUG:boto:Headers: {} 2024-02-01T10:21:39.794 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:21:39.794 DEBUG:boto:Port: 8000 2024-02-01T10:21:39.794 DEBUG:boto:Params: {} 2024-02-01T10:21:39.794 DEBUG:boto:Token: None 2024-02-01T10:21:39.796 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:21:39 GMT /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:39.796 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8yPEEOYbyn7m0hyJq3WWzPlX/SE= 2024-02-01T10:21:39.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8yPEEOYbyn7m0hyJq3WWzPlX/SE=', 'Content-Length': '0'} 2024-02-01T10:21:39.799 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'mE6urSxcrRA9wdyuToa41.w2aTia9p1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000529451ebc4cccd85-0065bb70b3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:39.799 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-02-01T10:21:39.800 DEBUG:boto:path=/vclfcn-16/obj-test-zone2 2024-02-01T10:21:39.800 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2 2024-02-01T10:21:39.800 DEBUG:boto:path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:39.800 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:39.800 DEBUG:boto:Method: GET 2024-02-01T10:21:39.800 DEBUG:boto:Path: /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:39.800 DEBUG:boto:Data: 2024-02-01T10:21:39.800 DEBUG:boto:Headers: {} 2024-02-01T10:21:39.800 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:21:39.800 DEBUG:boto:Port: 8001 2024-02-01T10:21:39.800 DEBUG:boto:Params: {} 2024-02-01T10:21:39.801 DEBUG:boto:Token: None 2024-02-01T10:21:39.801 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:39 GMT /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:39.801 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cJd+bRXSJMx8pGj/1EijcfBZGR4= 2024-02-01T10:21:39.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cJd+bRXSJMx8pGj/1EijcfBZGR4=', 'Content-Length': '0'} 2024-02-01T10:21:39.804 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'mE6urSxcrRA9wdyuToa41.w2aTia9p1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a76a1ff23308cadd-0065bb70b3-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:39.848 DEBUG:boto:path=/vclfcn-16/obj-test-zone2 2024-02-01T10:21:39.848 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2 2024-02-01T10:21:39.848 DEBUG:boto:path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:39.848 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:39.849 DEBUG:boto:Method: GET 2024-02-01T10:21:39.849 DEBUG:boto:Path: /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:39.849 DEBUG:boto:Data: 2024-02-01T10:21:39.849 DEBUG:boto:Headers: {} 2024-02-01T10:21:39.849 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:21:39.849 DEBUG:boto:Port: 8000 2024-02-01T10:21:39.849 DEBUG:boto:Params: {} 2024-02-01T10:21:39.849 DEBUG:boto:Token: None 2024-02-01T10:21:39.849 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:39 GMT /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:39.850 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cJd+bRXSJMx8pGj/1EijcfBZGR4= 2024-02-01T10:21:39.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cJd+bRXSJMx8pGj/1EijcfBZGR4=', 'Content-Length': '0'} 2024-02-01T10:21:39.853 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'mE6urSxcrRA9wdyuToa41.w2aTia9p1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000031d6d42dd6ad95b7-0065bb70b3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:39.853 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:21:39.853 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:21:39.853 DEBUG:boto:path=/vclfcn-16/?prefix=obj-test-zone2 2024-02-01T10:21:39.854 DEBUG:boto:auth_path=/vclfcn-16/?prefix=obj-test-zone2 2024-02-01T10:21:39.854 DEBUG:boto:Method: GET 2024-02-01T10:21:39.854 DEBUG:boto:Path: /vclfcn-16/?prefix=obj-test-zone2 2024-02-01T10:21:39.854 DEBUG:boto:Data: 2024-02-01T10:21:39.854 DEBUG:boto:Headers: {} 2024-02-01T10:21:39.854 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:21:39.854 DEBUG:boto:Port: 8001 2024-02-01T10:21:39.854 DEBUG:boto:Params: {} 2024-02-01T10:21:39.854 DEBUG:boto:Token: None 2024-02-01T10:21:39.854 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:39 GMT /vclfcn-16/ 2024-02-01T10:21:39.855 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BJAUjtG+XjZKlBb2XHwaqyhnS3E= 2024-02-01T10:21:39.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BJAUjtG+XjZKlBb2XHwaqyhnS3E=', 'Content-Length': '0'} 2024-02-01T10:21:39.859 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004526e9a8e6d211bb-0065bb70b3-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:39.859 DEBUG:boto:b'vclfcn-16obj-test-zone21000falseobj-test-zone22024-02-01T10:20:50.122Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:21:39.859 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:21:39.860 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:21:39.860 DEBUG:boto:path=/vclfcn-16/?prefix=obj-test-zone2 2024-02-01T10:21:39.860 DEBUG:boto:auth_path=/vclfcn-16/?prefix=obj-test-zone2 2024-02-01T10:21:39.860 DEBUG:boto:Method: GET 2024-02-01T10:21:39.860 DEBUG:boto:Path: /vclfcn-16/?prefix=obj-test-zone2 2024-02-01T10:21:39.860 DEBUG:boto:Data: 2024-02-01T10:21:39.860 DEBUG:boto:Headers: {} 2024-02-01T10:21:39.860 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:21:39.860 DEBUG:boto:Port: 8000 2024-02-01T10:21:39.860 DEBUG:boto:Params: {} 2024-02-01T10:21:39.861 DEBUG:boto:Token: None 2024-02-01T10:21:39.861 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:39 GMT /vclfcn-16/ 2024-02-01T10:21:39.861 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BJAUjtG+XjZKlBb2XHwaqyhnS3E= 2024-02-01T10:21:39.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BJAUjtG+XjZKlBb2XHwaqyhnS3E=', 'Content-Length': '0'} 2024-02-01T10:21:39.865 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000488c777c59c04eda-0065bb70b3-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:39.908 DEBUG:boto:b'vclfcn-16obj-test-zone21000falseobj-test-zone22024-02-01T10:20:50.122Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:21:39.909 DEBUG:boto:path=/vclfcn-16/obj-test-zone2 2024-02-01T10:21:39.909 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2 2024-02-01T10:21:39.909 DEBUG:boto:Method: HEAD 2024-02-01T10:21:39.909 DEBUG:boto:Path: /vclfcn-16/obj-test-zone2 2024-02-01T10:21:39.909 DEBUG:boto:Data: 2024-02-01T10:21:39.909 DEBUG:boto:Headers: {} 2024-02-01T10:21:39.909 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:21:39.909 DEBUG:boto:Port: 8001 2024-02-01T10:21:39.909 DEBUG:boto:Params: {} 2024-02-01T10:21:39.909 DEBUG:boto:Token: None 2024-02-01T10:21:39.910 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:21:39 GMT /vclfcn-16/obj-test-zone2 2024-02-01T10:21:39.910 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:qqL1lQ2BDOv/Ofooj/0XCHLo0iU= 2024-02-01T10:21:39.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:qqL1lQ2BDOv/Ofooj/0XCHLo0iU=', 'Content-Length': '0'} 2024-02-01T10:21:39.914 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'mE6urSxcrRA9wdyuToa41.w2aTia9p1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002a176f3aa8b1c4e1-0065bb70b3-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:39.914 DEBUG:boto:path=/vclfcn-16/obj-test-zone2 2024-02-01T10:21:39.914 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2 2024-02-01T10:21:39.914 DEBUG:boto:Method: HEAD 2024-02-01T10:21:39.914 DEBUG:boto:Path: /vclfcn-16/obj-test-zone2 2024-02-01T10:21:39.914 DEBUG:boto:Data: 2024-02-01T10:21:39.914 DEBUG:boto:Headers: {} 2024-02-01T10:21:39.914 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:21:39.915 DEBUG:boto:Port: 8000 2024-02-01T10:21:39.915 DEBUG:boto:Params: {} 2024-02-01T10:21:39.915 DEBUG:boto:Token: None 2024-02-01T10:21:39.915 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:21:39 GMT /vclfcn-16/obj-test-zone2 2024-02-01T10:21:39.915 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:qqL1lQ2BDOv/Ofooj/0XCHLo0iU= 2024-02-01T10:21:39.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:qqL1lQ2BDOv/Ofooj/0XCHLo0iU=', 'Content-Length': '0'} 2024-02-01T10:21:39.919 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'mE6urSxcrRA9wdyuToa41.w2aTia9p1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d3c3dbbef013abea-0065bb70b3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:39.919 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-02-01T10:21:39.919 DEBUG:boto:path=/vclfcn-16/obj-test-zone2 2024-02-01T10:21:39.919 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2 2024-02-01T10:21:39.919 DEBUG:boto:path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:39.920 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:39.920 DEBUG:boto:Method: GET 2024-02-01T10:21:39.920 DEBUG:boto:Path: /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:39.920 DEBUG:boto:Data: 2024-02-01T10:21:39.920 DEBUG:boto:Headers: {} 2024-02-01T10:21:39.920 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:21:39.920 DEBUG:boto:Port: 8001 2024-02-01T10:21:39.920 DEBUG:boto:Params: {} 2024-02-01T10:21:39.920 DEBUG:boto:Token: None 2024-02-01T10:21:39.920 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:39 GMT /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:39.921 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cJd+bRXSJMx8pGj/1EijcfBZGR4= 2024-02-01T10:21:39.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cJd+bRXSJMx8pGj/1EijcfBZGR4=', 'Content-Length': '0'} 2024-02-01T10:21:39.923 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'mE6urSxcrRA9wdyuToa41.w2aTia9p1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000224bfb894bae7713-0065bb70b3-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:39.964 DEBUG:boto:path=/vclfcn-16/obj-test-zone2 2024-02-01T10:21:39.964 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2 2024-02-01T10:21:39.964 DEBUG:boto:path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:39.964 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:39.965 DEBUG:boto:Method: GET 2024-02-01T10:21:39.965 DEBUG:boto:Path: /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:39.965 DEBUG:boto:Data: 2024-02-01T10:21:39.965 DEBUG:boto:Headers: {} 2024-02-01T10:21:39.965 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:21:39.965 DEBUG:boto:Port: 8000 2024-02-01T10:21:39.965 DEBUG:boto:Params: {} 2024-02-01T10:21:39.965 DEBUG:boto:Token: None 2024-02-01T10:21:39.965 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:39 GMT /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:21:39.966 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cJd+bRXSJMx8pGj/1EijcfBZGR4= 2024-02-01T10:21:39.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cJd+bRXSJMx8pGj/1EijcfBZGR4=', 'Content-Length': '0'} 2024-02-01T10:21:39.968 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'mE6urSxcrRA9wdyuToa41.w2aTia9p1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f1e359c142a9c0af-0065bb70b3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:39.969 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:21:39.969 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:21:39.969 DEBUG:boto:path=/vclfcn-16/?prefix=obj-test-zone2 2024-02-01T10:21:39.969 DEBUG:boto:auth_path=/vclfcn-16/?prefix=obj-test-zone2 2024-02-01T10:21:39.969 DEBUG:boto:Method: GET 2024-02-01T10:21:39.969 DEBUG:boto:Path: /vclfcn-16/?prefix=obj-test-zone2 2024-02-01T10:21:39.970 DEBUG:boto:Data: 2024-02-01T10:21:39.970 DEBUG:boto:Headers: {} 2024-02-01T10:21:39.970 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:21:39.970 DEBUG:boto:Port: 8001 2024-02-01T10:21:39.970 DEBUG:boto:Params: {} 2024-02-01T10:21:39.970 DEBUG:boto:Token: None 2024-02-01T10:21:39.970 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:39 GMT /vclfcn-16/ 2024-02-01T10:21:39.970 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BJAUjtG+XjZKlBb2XHwaqyhnS3E= 2024-02-01T10:21:39.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BJAUjtG+XjZKlBb2XHwaqyhnS3E=', 'Content-Length': '0'} 2024-02-01T10:21:39.975 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2e971fe8cd2e35c-0065bb70b3-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:40.016 DEBUG:boto:b'vclfcn-16obj-test-zone21000falseobj-test-zone22024-02-01T10:20:50.122Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:21:40.017 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:21:40.017 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:21:40.017 DEBUG:boto:path=/vclfcn-16/?prefix=obj-test-zone2 2024-02-01T10:21:40.017 DEBUG:boto:auth_path=/vclfcn-16/?prefix=obj-test-zone2 2024-02-01T10:21:40.017 DEBUG:boto:Method: GET 2024-02-01T10:21:40.017 DEBUG:boto:Path: /vclfcn-16/?prefix=obj-test-zone2 2024-02-01T10:21:40.017 DEBUG:boto:Data: 2024-02-01T10:21:40.017 DEBUG:boto:Headers: {} 2024-02-01T10:21:40.017 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:21:40.018 DEBUG:boto:Port: 8000 2024-02-01T10:21:40.018 DEBUG:boto:Params: {} 2024-02-01T10:21:40.018 DEBUG:boto:Token: None 2024-02-01T10:21:40.018 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:40 GMT /vclfcn-16/ 2024-02-01T10:21:40.018 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:W3WlkRMDHqsxsHjO5f+qFFpnEuQ= 2024-02-01T10:21:40.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:40 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:W3WlkRMDHqsxsHjO5f+qFFpnEuQ=', 'Content-Length': '0'} 2024-02-01T10:21:40.023 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028f10717ac4dc095-0065bb70b4-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:40 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:40.024 DEBUG:boto:b'vclfcn-16obj-test-zone21000falseobj-test-zone22024-02-01T10:20:50.122Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:21:40.024 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-02-01T10:21:40.024 DEBUG:boto:path=/vclfcn-16/obj-test-zone3 2024-02-01T10:21:40.024 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3 2024-02-01T10:21:40.024 DEBUG:boto:path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:40.024 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:40.025 DEBUG:boto:Method: GET 2024-02-01T10:21:40.025 DEBUG:boto:Path: /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:40.025 DEBUG:boto:Data: 2024-02-01T10:21:40.025 DEBUG:boto:Headers: {} 2024-02-01T10:21:40.025 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:21:40.025 DEBUG:boto:Port: 8001 2024-02-01T10:21:40.025 DEBUG:boto:Params: {} 2024-02-01T10:21:40.025 DEBUG:boto:Token: None 2024-02-01T10:21:40.025 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:40 GMT /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:40.025 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:kkio/uPQcr5gxuwMxn1KcHiv78Q= 2024-02-01T10:21:40.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:40 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:kkio/uPQcr5gxuwMxn1KcHiv78Q=', 'Content-Length': '0'} 2024-02-01T10:21:40.028 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bf8e91c753d20bd8-0065bb70b4-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:40 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:40.029 DEBUG:boto:path=/vclfcn-16/obj-test-zone3 2024-02-01T10:21:40.029 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3 2024-02-01T10:21:40.029 DEBUG:boto:path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:40.029 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:40.029 DEBUG:boto:Method: GET 2024-02-01T10:21:40.029 DEBUG:boto:Path: /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:40.029 DEBUG:boto:Data: 2024-02-01T10:21:40.030 DEBUG:boto:Headers: {} 2024-02-01T10:21:40.030 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:21:40.030 DEBUG:boto:Port: 8000 2024-02-01T10:21:40.030 DEBUG:boto:Params: {} 2024-02-01T10:21:40.030 DEBUG:boto:Token: None 2024-02-01T10:21:40.030 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:40 GMT /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:40.030 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:kkio/uPQcr5gxuwMxn1KcHiv78Q= 2024-02-01T10:21:40.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:40 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:kkio/uPQcr5gxuwMxn1KcHiv78Q=', 'Content-Length': '0'} 2024-02-01T10:21:40.033 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009045b8e861c8cc3c-0065bb70b4-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:40 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:40.033 DEBUG:boto:path=/vclfcn-16/obj-test-zone3 2024-02-01T10:21:40.033 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3 2024-02-01T10:21:40.034 DEBUG:boto:path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:40.034 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:40.034 DEBUG:boto:Method: HEAD 2024-02-01T10:21:40.034 DEBUG:boto:Path: /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:40.034 DEBUG:boto:Data: 2024-02-01T10:21:40.034 DEBUG:boto:Headers: {} 2024-02-01T10:21:40.034 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:21:40.034 DEBUG:boto:Port: 8001 2024-02-01T10:21:40.034 DEBUG:boto:Params: {} 2024-02-01T10:21:40.034 DEBUG:boto:Token: None 2024-02-01T10:21:40.035 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:21:40 GMT /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:40.035 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lkm4SFQht7V5YzHJH2O81uR+rT4= 2024-02-01T10:21:40.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:40 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lkm4SFQht7V5YzHJH2O81uR+rT4=', 'Content-Length': '0'} 2024-02-01T10:21:40.037 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000566c86a604473d3a-0065bb70b4-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:40 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:40.038 DEBUG:boto:path=/vclfcn-16/obj-test-zone3 2024-02-01T10:21:40.038 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3 2024-02-01T10:21:40.038 DEBUG:boto:path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:40.038 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:40.038 DEBUG:boto:Method: HEAD 2024-02-01T10:21:40.038 DEBUG:boto:Path: /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:40.038 DEBUG:boto:Data: 2024-02-01T10:21:40.038 DEBUG:boto:Headers: {} 2024-02-01T10:21:40.039 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:21:40.039 DEBUG:boto:Port: 8000 2024-02-01T10:21:40.039 DEBUG:boto:Params: {} 2024-02-01T10:21:40.039 DEBUG:boto:Token: None 2024-02-01T10:21:40.039 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:21:40 GMT /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:40.039 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lkm4SFQht7V5YzHJH2O81uR+rT4= 2024-02-01T10:21:40.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:40 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lkm4SFQht7V5YzHJH2O81uR+rT4=', 'Content-Length': '0'} 2024-02-01T10:21:40.042 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ffb817991f4e374f-0065bb70b4-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:40 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:40.042 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-02-01T10:21:40.042 DEBUG:boto:path=/vclfcn-16/obj-test-zone3 2024-02-01T10:21:40.042 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3 2024-02-01T10:21:40.042 DEBUG:boto:path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:40.043 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:40.043 DEBUG:boto:Method: GET 2024-02-01T10:21:40.043 DEBUG:boto:Path: /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:40.043 DEBUG:boto:Data: 2024-02-01T10:21:40.043 DEBUG:boto:Headers: {} 2024-02-01T10:21:40.043 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:21:40.043 DEBUG:boto:Port: 8001 2024-02-01T10:21:40.043 DEBUG:boto:Params: {} 2024-02-01T10:21:40.043 DEBUG:boto:Token: None 2024-02-01T10:21:40.043 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:40 GMT /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:40.044 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:kkio/uPQcr5gxuwMxn1KcHiv78Q= 2024-02-01T10:21:40.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:40 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:kkio/uPQcr5gxuwMxn1KcHiv78Q=', 'Content-Length': '0'} 2024-02-01T10:21:40.046 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b8ee3abfdcf00661-0065bb70b4-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:40 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:40.088 DEBUG:boto:path=/vclfcn-16/obj-test-zone3 2024-02-01T10:21:40.088 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3 2024-02-01T10:21:40.088 DEBUG:boto:path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:40.088 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:40.089 DEBUG:boto:Method: GET 2024-02-01T10:21:40.089 DEBUG:boto:Path: /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:40.089 DEBUG:boto:Data: 2024-02-01T10:21:40.089 DEBUG:boto:Headers: {} 2024-02-01T10:21:40.089 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:21:40.089 DEBUG:boto:Port: 8000 2024-02-01T10:21:40.089 DEBUG:boto:Params: {} 2024-02-01T10:21:40.089 DEBUG:boto:Token: None 2024-02-01T10:21:40.089 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:40 GMT /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:40.090 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:kkio/uPQcr5gxuwMxn1KcHiv78Q= 2024-02-01T10:21:40.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:40 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:kkio/uPQcr5gxuwMxn1KcHiv78Q=', 'Content-Length': '0'} 2024-02-01T10:21:40.093 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b480b2120855758a-0065bb70b4-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:40 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:40.094 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:21:40.094 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:21:40.094 DEBUG:boto:path=/vclfcn-16/?prefix=obj-test-zone3 2024-02-01T10:21:40.094 DEBUG:boto:auth_path=/vclfcn-16/?prefix=obj-test-zone3 2024-02-01T10:21:40.094 DEBUG:boto:Method: GET 2024-02-01T10:21:40.094 DEBUG:boto:Path: /vclfcn-16/?prefix=obj-test-zone3 2024-02-01T10:21:40.094 DEBUG:boto:Data: 2024-02-01T10:21:40.095 DEBUG:boto:Headers: {} 2024-02-01T10:21:40.095 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:21:40.095 DEBUG:boto:Port: 8001 2024-02-01T10:21:40.095 DEBUG:boto:Params: {} 2024-02-01T10:21:40.095 DEBUG:boto:Token: None 2024-02-01T10:21:40.095 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:40 GMT /vclfcn-16/ 2024-02-01T10:21:40.096 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:W3WlkRMDHqsxsHjO5f+qFFpnEuQ= 2024-02-01T10:21:40.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:40 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:W3WlkRMDHqsxsHjO5f+qFFpnEuQ=', 'Content-Length': '0'} 2024-02-01T10:21:40.100 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000073510aa2cde03aa5-0065bb70b4-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:40 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:40.100 DEBUG:boto:b'vclfcn-16obj-test-zone31000falseobj-test-zone32024-02-01T10:20:50.297Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:21:40.101 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:21:40.101 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:21:40.101 DEBUG:boto:path=/vclfcn-16/?prefix=obj-test-zone3 2024-02-01T10:21:40.101 DEBUG:boto:auth_path=/vclfcn-16/?prefix=obj-test-zone3 2024-02-01T10:21:40.101 DEBUG:boto:Method: GET 2024-02-01T10:21:40.101 DEBUG:boto:Path: /vclfcn-16/?prefix=obj-test-zone3 2024-02-01T10:21:40.101 DEBUG:boto:Data: 2024-02-01T10:21:40.102 DEBUG:boto:Headers: {} 2024-02-01T10:21:40.102 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:21:40.102 DEBUG:boto:Port: 8000 2024-02-01T10:21:40.102 DEBUG:boto:Params: {} 2024-02-01T10:21:40.102 DEBUG:boto:Token: None 2024-02-01T10:21:40.102 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:40 GMT /vclfcn-16/ 2024-02-01T10:21:40.102 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:W3WlkRMDHqsxsHjO5f+qFFpnEuQ= 2024-02-01T10:21:40.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:40 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:W3WlkRMDHqsxsHjO5f+qFFpnEuQ=', 'Content-Length': '0'} 2024-02-01T10:21:40.107 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045d88b1fffcae57d-0065bb70b4-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:40 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:40.107 DEBUG:boto:b'vclfcn-16obj-test-zone31000falseobj-test-zone32024-02-01T10:20:50.297Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:21:40.107 DEBUG:boto:path=/vclfcn-16/obj-test-zone3 2024-02-01T10:21:40.108 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3 2024-02-01T10:21:40.108 DEBUG:boto:Method: HEAD 2024-02-01T10:21:40.108 DEBUG:boto:Path: /vclfcn-16/obj-test-zone3 2024-02-01T10:21:40.108 DEBUG:boto:Data: 2024-02-01T10:21:40.108 DEBUG:boto:Headers: {} 2024-02-01T10:21:40.108 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:21:40.108 DEBUG:boto:Port: 8001 2024-02-01T10:21:40.108 DEBUG:boto:Params: {} 2024-02-01T10:21:40.108 DEBUG:boto:Token: None 2024-02-01T10:21:40.109 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:21:40 GMT /vclfcn-16/obj-test-zone3 2024-02-01T10:21:40.109 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:O7o8P8UnK4sYCZL8Dk/0G8F/1jw= 2024-02-01T10:21:40.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:40 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:O7o8P8UnK4sYCZL8Dk/0G8F/1jw=', 'Content-Length': '0'} 2024-02-01T10:21:40.112 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a2056ee19d522510-0065bb70b4-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:40 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:40.113 DEBUG:boto:path=/vclfcn-16/obj-test-zone3 2024-02-01T10:21:40.113 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3 2024-02-01T10:21:40.113 DEBUG:boto:Method: HEAD 2024-02-01T10:21:40.113 DEBUG:boto:Path: /vclfcn-16/obj-test-zone3 2024-02-01T10:21:40.113 DEBUG:boto:Data: 2024-02-01T10:21:40.113 DEBUG:boto:Headers: {} 2024-02-01T10:21:40.114 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:21:40.114 DEBUG:boto:Port: 8000 2024-02-01T10:21:40.114 DEBUG:boto:Params: {} 2024-02-01T10:21:40.114 DEBUG:boto:Token: None 2024-02-01T10:21:40.115 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:21:40 GMT /vclfcn-16/obj-test-zone3 2024-02-01T10:21:40.115 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:O7o8P8UnK4sYCZL8Dk/0G8F/1jw= 2024-02-01T10:21:40.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:40 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:O7o8P8UnK4sYCZL8Dk/0G8F/1jw=', 'Content-Length': '0'} 2024-02-01T10:21:40.119 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003676b0e07ad2bd8e-0065bb70b4-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:40 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:40.119 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-02-01T10:21:40.119 DEBUG:boto:path=/vclfcn-16/obj-test-zone3 2024-02-01T10:21:40.119 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3 2024-02-01T10:21:40.120 DEBUG:boto:path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:40.120 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:40.120 DEBUG:boto:Method: GET 2024-02-01T10:21:40.120 DEBUG:boto:Path: /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:40.120 DEBUG:boto:Data: 2024-02-01T10:21:40.120 DEBUG:boto:Headers: {} 2024-02-01T10:21:40.121 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:21:40.122 DEBUG:boto:Port: 8001 2024-02-01T10:21:40.122 DEBUG:boto:Params: {} 2024-02-01T10:21:40.123 DEBUG:boto:Token: None 2024-02-01T10:21:40.123 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:40 GMT /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:40.123 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:kkio/uPQcr5gxuwMxn1KcHiv78Q= 2024-02-01T10:21:40.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:40 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:kkio/uPQcr5gxuwMxn1KcHiv78Q=', 'Content-Length': '0'} 2024-02-01T10:21:40.126 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000074aafd89b92ca7ce-0065bb70b4-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:40 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:40.168 DEBUG:boto:path=/vclfcn-16/obj-test-zone3 2024-02-01T10:21:40.168 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3 2024-02-01T10:21:40.168 DEBUG:boto:path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:40.168 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:40.169 DEBUG:boto:Method: GET 2024-02-01T10:21:40.169 DEBUG:boto:Path: /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:40.169 DEBUG:boto:Data: 2024-02-01T10:21:40.169 DEBUG:boto:Headers: {} 2024-02-01T10:21:40.169 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:21:40.169 DEBUG:boto:Port: 8000 2024-02-01T10:21:40.169 DEBUG:boto:Params: {} 2024-02-01T10:21:40.169 DEBUG:boto:Token: None 2024-02-01T10:21:40.170 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:40 GMT /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:21:40.170 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:kkio/uPQcr5gxuwMxn1KcHiv78Q= 2024-02-01T10:21:40.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:40 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:kkio/uPQcr5gxuwMxn1KcHiv78Q=', 'Content-Length': '0'} 2024-02-01T10:21:40.173 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ce97b9575bb6be31-0065bb70b4-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:40 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:40.173 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:21:40.173 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:21:40.173 DEBUG:boto:path=/vclfcn-16/?prefix=obj-test-zone3 2024-02-01T10:21:40.174 DEBUG:boto:auth_path=/vclfcn-16/?prefix=obj-test-zone3 2024-02-01T10:21:40.174 DEBUG:boto:Method: GET 2024-02-01T10:21:40.174 DEBUG:boto:Path: /vclfcn-16/?prefix=obj-test-zone3 2024-02-01T10:21:40.174 DEBUG:boto:Data: 2024-02-01T10:21:40.174 DEBUG:boto:Headers: {} 2024-02-01T10:21:40.174 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:21:40.174 DEBUG:boto:Port: 8001 2024-02-01T10:21:40.174 DEBUG:boto:Params: {} 2024-02-01T10:21:40.174 DEBUG:boto:Token: None 2024-02-01T10:21:40.175 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:40 GMT /vclfcn-16/ 2024-02-01T10:21:40.175 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:W3WlkRMDHqsxsHjO5f+qFFpnEuQ= 2024-02-01T10:21:40.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:40 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:W3WlkRMDHqsxsHjO5f+qFFpnEuQ=', 'Content-Length': '0'} 2024-02-01T10:21:40.179 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd41e2c1ec17a165-0065bb70b4-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:40 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:40.179 DEBUG:boto:b'vclfcn-16obj-test-zone31000falseobj-test-zone32024-02-01T10:20:50.297Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:21:40.180 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:21:40.180 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:21:40.180 DEBUG:boto:path=/vclfcn-16/?prefix=obj-test-zone3 2024-02-01T10:21:40.180 DEBUG:boto:auth_path=/vclfcn-16/?prefix=obj-test-zone3 2024-02-01T10:21:40.180 DEBUG:boto:Method: GET 2024-02-01T10:21:40.180 DEBUG:boto:Path: /vclfcn-16/?prefix=obj-test-zone3 2024-02-01T10:21:40.181 DEBUG:boto:Data: 2024-02-01T10:21:40.181 DEBUG:boto:Headers: {} 2024-02-01T10:21:40.181 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:21:40.181 DEBUG:boto:Port: 8000 2024-02-01T10:21:40.181 DEBUG:boto:Params: {} 2024-02-01T10:21:40.181 DEBUG:boto:Token: None 2024-02-01T10:21:40.181 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:21:40 GMT /vclfcn-16/ 2024-02-01T10:21:40.181 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:W3WlkRMDHqsxsHjO5f+qFFpnEuQ= 2024-02-01T10:21:40.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:21:40 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:W3WlkRMDHqsxsHjO5f+qFFpnEuQ=', 'Content-Length': '0'} 2024-02-01T10:21:40.185 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001cd64961fd7ebb8e-0065bb70b4-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:21:40 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:21:40.228 DEBUG:boto:b'vclfcn-16obj-test-zone31000falseobj-test-zone32024-02-01T10:20:50.297Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:21:40.228 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-16 zones={test-zone3, test-zone2} 2024-02-01T10:21:40.229 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=vclfcn-17 2024-02-01T10:21:40.229 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:21:40.229 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:21:40.229 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket vclfcn-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-02-01T10:21:40.280 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:21:40.281 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:21:40.345 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:21:40.345+0000 7f752fd02980 1 bucket sync caught up with source: 2024-02-01T10:21:40.345 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, 00000000010.44.6, , 00000000003.22.10, , 00000000007.72.10, , , , , ] 2024-02-01T10:21:40.345 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, 00000000010.44.6, , 00000000003.22.10, , 00000000007.72.10, , , , , ] 2024-02-01T10:21:40.345 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:21:40.345+0000 7f752fd02980 0 bucket checkpoint complete 2024-02-01T10:21:40.368 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=vclfcn-17 2024-02-01T10:21:40.368 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:21:40.369 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:21:40.369 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:21:40.419 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:21:40.419 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:21:40.478 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:21:40.481+0000 7f007f54a980 1 bucket sync caught up with source: 2024-02-01T10:21:40.478 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, 00000000010.44.6, , 00000000003.22.10, , 00000000007.72.10, , , , , ] 2024-02-01T10:21:40.478 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, 00000000010.44.6, , 00000000003.22.10, , 00000000007.72.10, , , , , ] 2024-02-01T10:21:40.479 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:21:40.481+0000 7f007f54a980 0 bucket checkpoint complete 2024-02-01T10:21:40.500 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=vclfcn-17 2024-02-01T10:21:40.501 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:21:40.501 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:21:40.501 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:21:40.542 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:21:40.542 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:21:40.603 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:21:40.605+0000 7f0d2c63f980 1 waiting for incremental sync to catch up: 2024-02-01T10:21:40.603 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, 00000000003.54.10, , 00000000004.30.12, , 00000000008.50.10, , , , , ] 2024-02-01T10:21:40.603 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, 00000000003.54.10, , 00000000010.42.6, , 00000000008.50.10, , , , , ] 2024-02-01T10:21:45.605 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:21:45.609+0000 7f0d2c63f980 1 waiting for incremental sync to catch up: 2024-02-01T10:21:45.605 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, 00000000003.54.10, , 00000000004.30.12, , 00000000008.50.10, , , , , ] 2024-02-01T10:21:45.605 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, 00000000003.54.10, , 00000000010.42.6, , 00000000008.50.10, , , , , ] 2024-02-01T10:21:47.547 INFO:tasks.rgw.c2.client.0.smithi187.stdout:2024-02-01T10:21:47.545+0000 7f7bead57640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-02-01T10:21:50.607 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:21:50.609+0000 7f0d2c63f980 1 waiting for incremental sync to catch up: 2024-02-01T10:21:50.607 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, 00000000003.54.10, , 00000000004.30.12, , 00000000008.50.10, , , , , ] 2024-02-01T10:21:50.608 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, 00000000003.54.10, , 00000000010.42.6, , 00000000008.50.10, , , , , ] 2024-02-01T10:21:55.610 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:21:55.613+0000 7f0d2c63f980 1 waiting for incremental sync to catch up: 2024-02-01T10:21:55.610 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, 00000000003.54.10, , 00000000004.30.12, , 00000000008.50.10, , , , , ] 2024-02-01T10:21:55.611 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, 00000000003.54.10, , 00000000010.42.6, , 00000000008.50.10, , , , , ] 2024-02-01T10:22:00.612 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:22:00.616+0000 7f0d2c63f980 1 waiting for incremental sync to catch up: 2024-02-01T10:22:00.612 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, 00000000003.54.10, , 00000000004.30.12, , 00000000008.50.10, , , , , ] 2024-02-01T10:22:00.613 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, 00000000003.54.10, , 00000000010.42.6, , 00000000008.50.10, , , , , ] 2024-02-01T10:22:05.615 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:22:05.616+0000 7f0d2c63f980 1 waiting for incremental sync to catch up: 2024-02-01T10:22:05.615 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, 00000000003.54.10, , 00000000004.30.12, , 00000000008.50.10, , , , , ] 2024-02-01T10:22:05.615 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, 00000000003.54.10, , 00000000010.42.6, , 00000000008.50.10, , , , , ] 2024-02-01T10:22:10.619 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:22:10.620+0000 7f0d2c63f980 1 bucket sync caught up with source: 2024-02-01T10:22:10.620 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, 00000000003.54.10, , 00000000010.42.6, , 00000000008.50.10, , , , , ] 2024-02-01T10:22:10.620 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, 00000000003.54.10, , 00000000010.42.6, , 00000000008.50.10, , , , , ] 2024-02-01T10:22:10.620 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:22:10.620+0000 7f0d2c63f980 0 bucket checkpoint complete 2024-02-01T10:22:10.654 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=vclfcn-17 2024-02-01T10:22:10.654 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:22:10.654 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:22:10.655 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:22:10.746 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:22:10.746 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:22:10.811 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:22:10.812+0000 7fa3f1f5b980 1 bucket sync caught up with source: 2024-02-01T10:22:10.811 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, 00000000003.54.10, , 00000000010.42.6, , 00000000008.50.10, , , , , ] 2024-02-01T10:22:10.811 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, 00000000003.54.10, , 00000000010.42.6, , 00000000008.50.10, , , , , ] 2024-02-01T10:22:10.812 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:22:10.812+0000 7fa3f1f5b980 0 bucket checkpoint complete 2024-02-01T10:22:10.835 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=vclfcn-17 2024-02-01T10:22:10.835 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:22:10.835 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:22:10.835 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:22:10.875 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:22:10.876 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:22:10.933 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:22:10.936+0000 7f738a786980 1 bucket sync caught up with source: 2024-02-01T10:22:10.934 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, 00000000003.139.10, , 00000000003.71.10, , 00000000013.129.6, , , , , ] 2024-02-01T10:22:10.934 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, 00000000003.139.10, , 00000000003.71.10, , 00000000013.129.6, , , , , ] 2024-02-01T10:22:10.934 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:22:10.936+0000 7f738a786980 0 bucket checkpoint complete 2024-02-01T10:22:10.955 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=vclfcn-17 2024-02-01T10:22:10.955 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:22:10.955 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:22:10.956 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket vclfcn-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-02-01T10:22:11.006 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:22:11.006 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:22:11.068 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:22:11.068+0000 7ff81493c980 1 bucket sync caught up with source: 2024-02-01T10:22:11.068 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, 00000000003.139.10, , 00000000003.71.10, , 00000000013.129.6, , , , , ] 2024-02-01T10:22:11.069 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, 00000000003.139.10, , 00000000003.71.10, , 00000000013.129.6, , , , , ] 2024-02-01T10:22:11.069 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:22:11.068+0000 7ff81493c980 0 bucket checkpoint complete 2024-02-01T10:22:11.091 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-17 zones={test-zone2, test-zone1} 2024-02-01T10:22:11.091 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:11.091 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:11.091 DEBUG:boto:Method: HEAD 2024-02-01T10:22:11.091 DEBUG:boto:Path: /vclfcn-17/ 2024-02-01T10:22:11.092 DEBUG:boto:Data: 2024-02-01T10:22:11.092 DEBUG:boto:Headers: {} 2024-02-01T10:22:11.092 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:11.092 DEBUG:boto:Port: 8000 2024-02-01T10:22:11.092 DEBUG:boto:Params: {} 2024-02-01T10:22:11.092 DEBUG:boto:Token: None 2024-02-01T10:22:11.093 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:11 GMT /vclfcn-17/ 2024-02-01T10:22:11.093 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zYCrycCn5yJUhWNStkzLAUOHI4k= 2024-02-01T10:22:11.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zYCrycCn5yJUhWNStkzLAUOHI4k=', 'Content-Length': '0'} 2024-02-01T10:22:11.096 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003c32ab93d47315d3-0065bb70d3-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:22:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:11.096 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:11.096 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:11.096 DEBUG:boto:Method: HEAD 2024-02-01T10:22:11.096 DEBUG:boto:Path: /vclfcn-17/ 2024-02-01T10:22:11.096 DEBUG:boto:Data: 2024-02-01T10:22:11.096 DEBUG:boto:Headers: {} 2024-02-01T10:22:11.097 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:11.097 DEBUG:boto:Port: 8000 2024-02-01T10:22:11.097 DEBUG:boto:Params: {} 2024-02-01T10:22:11.097 DEBUG:boto:Token: None 2024-02-01T10:22:11.097 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:11 GMT /vclfcn-17/ 2024-02-01T10:22:11.097 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zYCrycCn5yJUhWNStkzLAUOHI4k= 2024-02-01T10:22:11.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zYCrycCn5yJUhWNStkzLAUOHI4k=', 'Content-Length': '0'} 2024-02-01T10:22:11.102 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003c67a42a7ef7f50a-0065bb70d3-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:22:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:11.102 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:22:11.102 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:11.102 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:11.102 DEBUG:boto:path=/vclfcn-17/?versions&max-keys=999 2024-02-01T10:22:11.102 DEBUG:boto:auth_path=/vclfcn-17/?versions&max-keys=999 2024-02-01T10:22:11.102 DEBUG:boto:Method: GET 2024-02-01T10:22:11.102 DEBUG:boto:Path: /vclfcn-17/?versions&max-keys=999 2024-02-01T10:22:11.103 DEBUG:boto:Data: 2024-02-01T10:22:11.103 DEBUG:boto:Headers: {} 2024-02-01T10:22:11.103 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:11.103 DEBUG:boto:Port: 8000 2024-02-01T10:22:11.103 DEBUG:boto:Params: {} 2024-02-01T10:22:11.103 DEBUG:boto:Token: None 2024-02-01T10:22:11.103 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:11 GMT /vclfcn-17/?versions 2024-02-01T10:22:11.103 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Lwu6LYfB2jYbvm20s2DkV45uafY= 2024-02-01T10:22:11.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Lwu6LYfB2jYbvm20s2DkV45uafY=', 'Content-Length': '0'} 2024-02-01T10:22:11.106 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3ab0e82f7e306f1-0065bb70d3-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:11.106 DEBUG:boto:b'vclfcn-17999falsedummyhQXZXSD45t2CSRVMJOLMBXih54iMPmJtrue2024-02-01T10:20:14.297Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1xMxAGR4A-xEILepnzFPRgrWNTOLvitztrue2024-02-01T10:20:50.473Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1true2024-02-01T10:20:50.624Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3YE736OwOOXbZJHJe7d8YvcNV1LxwcQbtrue2024-02-01T10:20:50.780Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:11.107 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-02-01T10:22:11.108 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-02-01T10:22:11.108 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-02-01T10:22:11.108 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-02-01T10:22:11.108 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:22:11.108 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:11.108 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:11.108 DEBUG:boto:path=/vclfcn-17/?versions&max-keys=999 2024-02-01T10:22:11.108 DEBUG:boto:auth_path=/vclfcn-17/?versions&max-keys=999 2024-02-01T10:22:11.108 DEBUG:boto:Method: GET 2024-02-01T10:22:11.108 DEBUG:boto:Path: /vclfcn-17/?versions&max-keys=999 2024-02-01T10:22:11.109 DEBUG:boto:Data: 2024-02-01T10:22:11.109 DEBUG:boto:Headers: {} 2024-02-01T10:22:11.109 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:11.109 DEBUG:boto:Port: 8000 2024-02-01T10:22:11.109 DEBUG:boto:Params: {} 2024-02-01T10:22:11.109 DEBUG:boto:Token: None 2024-02-01T10:22:11.109 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:11 GMT /vclfcn-17/?versions 2024-02-01T10:22:11.109 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Lwu6LYfB2jYbvm20s2DkV45uafY= 2024-02-01T10:22:11.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Lwu6LYfB2jYbvm20s2DkV45uafY=', 'Content-Length': '0'} 2024-02-01T10:22:11.114 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff042d19da4d2f3b-0065bb70d3-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:11.114 DEBUG:boto:b'vclfcn-17999falsedummyhQXZXSD45t2CSRVMJOLMBXih54iMPmJtrue2024-02-01T10:20:14.297Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1xMxAGR4A-xEILepnzFPRgrWNTOLvitztrue2024-02-01T10:20:50.473Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1true2024-02-01T10:20:50.624Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3YE736OwOOXbZJHJe7d8YvcNV1LxwcQbtrue2024-02-01T10:20:50.780Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:11.115 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-02-01T10:22:11.115 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-02-01T10:22:11.115 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-02-01T10:22:11.115 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-02-01T10:22:11.115 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:11.115 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:11.115 DEBUG:boto:path=/vclfcn-17/?versions&max-keys=999 2024-02-01T10:22:11.115 DEBUG:boto:auth_path=/vclfcn-17/?versions&max-keys=999 2024-02-01T10:22:11.116 DEBUG:boto:Method: GET 2024-02-01T10:22:11.116 DEBUG:boto:Path: /vclfcn-17/?versions&max-keys=999 2024-02-01T10:22:11.116 DEBUG:boto:Data: 2024-02-01T10:22:11.116 DEBUG:boto:Headers: {} 2024-02-01T10:22:11.116 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:11.116 DEBUG:boto:Port: 8000 2024-02-01T10:22:11.116 DEBUG:boto:Params: {} 2024-02-01T10:22:11.116 DEBUG:boto:Token: None 2024-02-01T10:22:11.116 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:11 GMT /vclfcn-17/?versions 2024-02-01T10:22:11.116 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Lwu6LYfB2jYbvm20s2DkV45uafY= 2024-02-01T10:22:11.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Lwu6LYfB2jYbvm20s2DkV45uafY=', 'Content-Length': '0'} 2024-02-01T10:22:11.119 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007da214f5c195ba38-0065bb70d3-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:11.119 DEBUG:boto:b'vclfcn-17999falsedummyhQXZXSD45t2CSRVMJOLMBXih54iMPmJtrue2024-02-01T10:20:14.297Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1xMxAGR4A-xEILepnzFPRgrWNTOLvitztrue2024-02-01T10:20:50.473Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1true2024-02-01T10:20:50.624Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3YE736OwOOXbZJHJe7d8YvcNV1LxwcQbtrue2024-02-01T10:20:50.780Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:11.120 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:11.120 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:11.120 DEBUG:boto:path=/vclfcn-17/?versions&max-keys=999 2024-02-01T10:22:11.120 DEBUG:boto:auth_path=/vclfcn-17/?versions&max-keys=999 2024-02-01T10:22:11.120 DEBUG:boto:Method: GET 2024-02-01T10:22:11.120 DEBUG:boto:Path: /vclfcn-17/?versions&max-keys=999 2024-02-01T10:22:11.120 DEBUG:boto:Data: 2024-02-01T10:22:11.120 DEBUG:boto:Headers: {} 2024-02-01T10:22:11.120 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:11.120 DEBUG:boto:Port: 8000 2024-02-01T10:22:11.121 DEBUG:boto:Params: {} 2024-02-01T10:22:11.121 DEBUG:boto:Token: None 2024-02-01T10:22:11.121 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:11 GMT /vclfcn-17/?versions 2024-02-01T10:22:11.121 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Lwu6LYfB2jYbvm20s2DkV45uafY= 2024-02-01T10:22:11.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Lwu6LYfB2jYbvm20s2DkV45uafY=', 'Content-Length': '0'} 2024-02-01T10:22:11.125 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ad800b80acf1cd7-0065bb70d3-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:11.125 DEBUG:boto:b'vclfcn-17999falsedummyhQXZXSD45t2CSRVMJOLMBXih54iMPmJtrue2024-02-01T10:20:14.297Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1xMxAGR4A-xEILepnzFPRgrWNTOLvitztrue2024-02-01T10:20:50.473Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1true2024-02-01T10:20:50.624Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3YE736OwOOXbZJHJe7d8YvcNV1LxwcQbtrue2024-02-01T10:20:50.780Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:11.126 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-02-01T10:22:11.126 DEBUG:boto:path=/vclfcn-17/dummy 2024-02-01T10:22:11.126 DEBUG:boto:auth_path=/vclfcn-17/dummy 2024-02-01T10:22:11.126 DEBUG:boto:path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:11.126 DEBUG:boto:auth_path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:11.127 DEBUG:boto:Method: GET 2024-02-01T10:22:11.127 DEBUG:boto:Path: /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:11.127 DEBUG:boto:Data: 2024-02-01T10:22:11.127 DEBUG:boto:Headers: {} 2024-02-01T10:22:11.127 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:11.127 DEBUG:boto:Port: 8000 2024-02-01T10:22:11.127 DEBUG:boto:Params: {} 2024-02-01T10:22:11.127 DEBUG:boto:Token: None 2024-02-01T10:22:11.127 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:11 GMT /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:11.127 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/f6FoLgGBQDlFx9geYKheSBchRM= 2024-02-01T10:22:11.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/f6FoLgGBQDlFx9geYKheSBchRM=', 'Content-Length': '0'} 2024-02-01T10:22:11.129 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:14 GMT'), ('x-amz-version-id', 'hQXZXSD45t2CSRVMJOLMBXih54iMPmJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ed93f89de5027d80-0065bb70d3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:11.130 DEBUG:boto:path=/vclfcn-17/dummy 2024-02-01T10:22:11.130 DEBUG:boto:auth_path=/vclfcn-17/dummy 2024-02-01T10:22:11.130 DEBUG:boto:path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:11.130 DEBUG:boto:auth_path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:11.130 DEBUG:boto:Method: GET 2024-02-01T10:22:11.130 DEBUG:boto:Path: /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:11.130 DEBUG:boto:Data: 2024-02-01T10:22:11.130 DEBUG:boto:Headers: {} 2024-02-01T10:22:11.130 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:11.131 DEBUG:boto:Port: 8000 2024-02-01T10:22:11.131 DEBUG:boto:Params: {} 2024-02-01T10:22:11.131 DEBUG:boto:Token: None 2024-02-01T10:22:11.131 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:11 GMT /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:11.131 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/f6FoLgGBQDlFx9geYKheSBchRM= 2024-02-01T10:22:11.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/f6FoLgGBQDlFx9geYKheSBchRM=', 'Content-Length': '0'} 2024-02-01T10:22:11.134 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:14 GMT'), ('x-amz-version-id', 'hQXZXSD45t2CSRVMJOLMBXih54iMPmJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e1d75d6cfd13824c-0065bb70d3-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:11.134 DEBUG:boto:path=/vclfcn-17/dummy 2024-02-01T10:22:11.135 DEBUG:boto:auth_path=/vclfcn-17/dummy 2024-02-01T10:22:11.135 DEBUG:boto:path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:11.135 DEBUG:boto:auth_path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:11.135 DEBUG:boto:Method: HEAD 2024-02-01T10:22:11.135 DEBUG:boto:Path: /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:11.135 DEBUG:boto:Data: 2024-02-01T10:22:11.135 DEBUG:boto:Headers: {} 2024-02-01T10:22:11.135 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:11.135 DEBUG:boto:Port: 8000 2024-02-01T10:22:11.135 DEBUG:boto:Params: {} 2024-02-01T10:22:11.136 DEBUG:boto:Token: None 2024-02-01T10:22:11.136 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:11 GMT /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:11.136 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:kLSpsOYvlGkqRnoF7Rnw7a4iHno= 2024-02-01T10:22:11.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:kLSpsOYvlGkqRnoF7Rnw7a4iHno=', 'Content-Length': '0'} 2024-02-01T10:22:11.138 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:14 GMT'), ('x-amz-version-id', 'hQXZXSD45t2CSRVMJOLMBXih54iMPmJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000033b1a2a6d7953198-0065bb70d3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:11.138 DEBUG:boto:path=/vclfcn-17/dummy 2024-02-01T10:22:11.138 DEBUG:boto:auth_path=/vclfcn-17/dummy 2024-02-01T10:22:11.139 DEBUG:boto:path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:11.139 DEBUG:boto:auth_path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:11.139 DEBUG:boto:Method: HEAD 2024-02-01T10:22:11.139 DEBUG:boto:Path: /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:11.139 DEBUG:boto:Data: 2024-02-01T10:22:11.139 DEBUG:boto:Headers: {} 2024-02-01T10:22:11.139 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:11.139 DEBUG:boto:Port: 8000 2024-02-01T10:22:11.139 DEBUG:boto:Params: {} 2024-02-01T10:22:11.139 DEBUG:boto:Token: None 2024-02-01T10:22:11.140 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:11 GMT /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:11.140 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:kLSpsOYvlGkqRnoF7Rnw7a4iHno= 2024-02-01T10:22:11.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:kLSpsOYvlGkqRnoF7Rnw7a4iHno=', 'Content-Length': '0'} 2024-02-01T10:22:11.143 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:14 GMT'), ('x-amz-version-id', 'hQXZXSD45t2CSRVMJOLMBXih54iMPmJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002dcf518dd4a9ef55-0065bb70d3-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:11.143 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-02-01T10:22:11.143 DEBUG:boto:path=/vclfcn-17/dummy 2024-02-01T10:22:11.143 DEBUG:boto:auth_path=/vclfcn-17/dummy 2024-02-01T10:22:11.143 DEBUG:boto:path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:11.143 DEBUG:boto:auth_path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:11.143 DEBUG:boto:Method: GET 2024-02-01T10:22:11.144 DEBUG:boto:Path: /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:11.144 DEBUG:boto:Data: 2024-02-01T10:22:11.144 DEBUG:boto:Headers: {} 2024-02-01T10:22:11.144 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:11.144 DEBUG:boto:Port: 8000 2024-02-01T10:22:11.144 DEBUG:boto:Params: {} 2024-02-01T10:22:11.144 DEBUG:boto:Token: None 2024-02-01T10:22:11.144 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:11 GMT /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:11.144 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/f6FoLgGBQDlFx9geYKheSBchRM= 2024-02-01T10:22:11.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/f6FoLgGBQDlFx9geYKheSBchRM=', 'Content-Length': '0'} 2024-02-01T10:22:11.147 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:14 GMT'), ('x-amz-version-id', 'hQXZXSD45t2CSRVMJOLMBXih54iMPmJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000050998de5e0010fc-0065bb70d3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:11.147 DEBUG:boto:path=/vclfcn-17/dummy 2024-02-01T10:22:11.147 DEBUG:boto:auth_path=/vclfcn-17/dummy 2024-02-01T10:22:11.147 DEBUG:boto:path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:11.147 DEBUG:boto:auth_path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:11.147 DEBUG:boto:Method: GET 2024-02-01T10:22:11.148 DEBUG:boto:Path: /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:11.148 DEBUG:boto:Data: 2024-02-01T10:22:11.148 DEBUG:boto:Headers: {} 2024-02-01T10:22:11.148 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:11.148 DEBUG:boto:Port: 8000 2024-02-01T10:22:11.148 DEBUG:boto:Params: {} 2024-02-01T10:22:11.148 DEBUG:boto:Token: None 2024-02-01T10:22:11.148 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:11 GMT /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:11.148 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/f6FoLgGBQDlFx9geYKheSBchRM= 2024-02-01T10:22:11.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/f6FoLgGBQDlFx9geYKheSBchRM=', 'Content-Length': '0'} 2024-02-01T10:22:11.151 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:14 GMT'), ('x-amz-version-id', 'hQXZXSD45t2CSRVMJOLMBXih54iMPmJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000047c71dc8a3cd85ad-0065bb70d3-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:11.152 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:11.152 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:11.152 DEBUG:boto:path=/vclfcn-17/?prefix=dummy 2024-02-01T10:22:11.152 DEBUG:boto:auth_path=/vclfcn-17/?prefix=dummy 2024-02-01T10:22:11.152 DEBUG:boto:Method: GET 2024-02-01T10:22:11.152 DEBUG:boto:Path: /vclfcn-17/?prefix=dummy 2024-02-01T10:22:11.152 DEBUG:boto:Data: 2024-02-01T10:22:11.152 DEBUG:boto:Headers: {} 2024-02-01T10:22:11.153 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:11.153 DEBUG:boto:Port: 8000 2024-02-01T10:22:11.153 DEBUG:boto:Params: {} 2024-02-01T10:22:11.153 DEBUG:boto:Token: None 2024-02-01T10:22:11.153 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:11 GMT /vclfcn-17/ 2024-02-01T10:22:11.153 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:hkxaNl2qfovb0VWctq15J2stLQI= 2024-02-01T10:22:11.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:hkxaNl2qfovb0VWctq15J2stLQI=', 'Content-Length': '0'} 2024-02-01T10:22:11.157 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9e80e2cdc58e212-0065bb70d3-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:11.157 DEBUG:boto:b'vclfcn-17dummy1000falsedummy2024-02-01T10:20:14.297Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:11.158 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:11.158 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:11.158 DEBUG:boto:path=/vclfcn-17/?prefix=dummy 2024-02-01T10:22:11.158 DEBUG:boto:auth_path=/vclfcn-17/?prefix=dummy 2024-02-01T10:22:11.158 DEBUG:boto:Method: GET 2024-02-01T10:22:11.158 DEBUG:boto:Path: /vclfcn-17/?prefix=dummy 2024-02-01T10:22:11.158 DEBUG:boto:Data: 2024-02-01T10:22:11.159 DEBUG:boto:Headers: {} 2024-02-01T10:22:11.159 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:11.159 DEBUG:boto:Port: 8000 2024-02-01T10:22:11.159 DEBUG:boto:Params: {} 2024-02-01T10:22:11.159 DEBUG:boto:Token: None 2024-02-01T10:22:11.159 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:11 GMT /vclfcn-17/ 2024-02-01T10:22:11.159 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:hkxaNl2qfovb0VWctq15J2stLQI= 2024-02-01T10:22:11.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:hkxaNl2qfovb0VWctq15J2stLQI=', 'Content-Length': '0'} 2024-02-01T10:22:11.163 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000020f941f594ce5695-0065bb70d3-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:11.208 DEBUG:boto:b'vclfcn-17dummy1000falsedummy2024-02-01T10:20:14.297Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:11.209 DEBUG:boto:path=/vclfcn-17/dummy 2024-02-01T10:22:11.209 DEBUG:boto:auth_path=/vclfcn-17/dummy 2024-02-01T10:22:11.209 DEBUG:boto:Method: HEAD 2024-02-01T10:22:11.209 DEBUG:boto:Path: /vclfcn-17/dummy 2024-02-01T10:22:11.209 DEBUG:boto:Data: 2024-02-01T10:22:11.209 DEBUG:boto:Headers: {} 2024-02-01T10:22:11.209 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:11.209 DEBUG:boto:Port: 8000 2024-02-01T10:22:11.209 DEBUG:boto:Params: {} 2024-02-01T10:22:11.210 DEBUG:boto:Token: None 2024-02-01T10:22:11.210 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:11 GMT /vclfcn-17/dummy 2024-02-01T10:22:11.210 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YSpp98cbNd/Nok9a8lRwZWh0/3k= 2024-02-01T10:22:11.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YSpp98cbNd/Nok9a8lRwZWh0/3k=', 'Content-Length': '0'} 2024-02-01T10:22:11.214 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:14 GMT'), ('x-amz-version-id', 'hQXZXSD45t2CSRVMJOLMBXih54iMPmJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000685114f2a529d8d7-0065bb70d3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:11.214 DEBUG:boto:path=/vclfcn-17/dummy 2024-02-01T10:22:11.214 DEBUG:boto:auth_path=/vclfcn-17/dummy 2024-02-01T10:22:11.214 DEBUG:boto:Method: HEAD 2024-02-01T10:22:11.214 DEBUG:boto:Path: /vclfcn-17/dummy 2024-02-01T10:22:11.214 DEBUG:boto:Data: 2024-02-01T10:22:11.214 DEBUG:boto:Headers: {} 2024-02-01T10:22:11.214 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:11.214 DEBUG:boto:Port: 8000 2024-02-01T10:22:11.215 DEBUG:boto:Params: {} 2024-02-01T10:22:11.215 DEBUG:boto:Token: None 2024-02-01T10:22:11.215 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:11 GMT /vclfcn-17/dummy 2024-02-01T10:22:11.215 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YSpp98cbNd/Nok9a8lRwZWh0/3k= 2024-02-01T10:22:11.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YSpp98cbNd/Nok9a8lRwZWh0/3k=', 'Content-Length': '0'} 2024-02-01T10:22:11.219 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:14 GMT'), ('x-amz-version-id', 'hQXZXSD45t2CSRVMJOLMBXih54iMPmJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ad079c5269342605-0065bb70d3-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:11.219 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-02-01T10:22:11.219 DEBUG:boto:path=/vclfcn-17/dummy 2024-02-01T10:22:11.219 DEBUG:boto:auth_path=/vclfcn-17/dummy 2024-02-01T10:22:11.219 DEBUG:boto:path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:11.219 DEBUG:boto:auth_path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:11.220 DEBUG:boto:Method: GET 2024-02-01T10:22:11.220 DEBUG:boto:Path: /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:11.220 DEBUG:boto:Data: 2024-02-01T10:22:11.220 DEBUG:boto:Headers: {} 2024-02-01T10:22:11.220 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:11.220 DEBUG:boto:Port: 8000 2024-02-01T10:22:11.220 DEBUG:boto:Params: {} 2024-02-01T10:22:11.220 DEBUG:boto:Token: None 2024-02-01T10:22:11.220 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:11 GMT /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:11.221 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/f6FoLgGBQDlFx9geYKheSBchRM= 2024-02-01T10:22:11.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/f6FoLgGBQDlFx9geYKheSBchRM=', 'Content-Length': '0'} 2024-02-01T10:22:11.223 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:14 GMT'), ('x-amz-version-id', 'hQXZXSD45t2CSRVMJOLMBXih54iMPmJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000268e9ad69e753059-0065bb70d3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:11.224 DEBUG:boto:path=/vclfcn-17/dummy 2024-02-01T10:22:11.224 DEBUG:boto:auth_path=/vclfcn-17/dummy 2024-02-01T10:22:11.224 DEBUG:boto:path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:11.224 DEBUG:boto:auth_path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:11.224 DEBUG:boto:Method: GET 2024-02-01T10:22:11.224 DEBUG:boto:Path: /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:11.224 DEBUG:boto:Data: 2024-02-01T10:22:11.227 DEBUG:boto:Headers: {} 2024-02-01T10:22:11.227 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:11.227 DEBUG:boto:Port: 8000 2024-02-01T10:22:11.227 DEBUG:boto:Params: {} 2024-02-01T10:22:11.227 DEBUG:boto:Token: None 2024-02-01T10:22:11.227 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:11 GMT /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:11.227 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/f6FoLgGBQDlFx9geYKheSBchRM= 2024-02-01T10:22:11.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/f6FoLgGBQDlFx9geYKheSBchRM=', 'Content-Length': '0'} 2024-02-01T10:22:11.230 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:14 GMT'), ('x-amz-version-id', 'hQXZXSD45t2CSRVMJOLMBXih54iMPmJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000725aa8497253ff49-0065bb70d3-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:11.230 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:11.231 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:11.231 DEBUG:boto:path=/vclfcn-17/?prefix=dummy 2024-02-01T10:22:11.231 DEBUG:boto:auth_path=/vclfcn-17/?prefix=dummy 2024-02-01T10:22:11.231 DEBUG:boto:Method: GET 2024-02-01T10:22:11.231 DEBUG:boto:Path: /vclfcn-17/?prefix=dummy 2024-02-01T10:22:11.231 DEBUG:boto:Data: 2024-02-01T10:22:11.231 DEBUG:boto:Headers: {} 2024-02-01T10:22:11.231 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:11.231 DEBUG:boto:Port: 8000 2024-02-01T10:22:11.231 DEBUG:boto:Params: {} 2024-02-01T10:22:11.231 DEBUG:boto:Token: None 2024-02-01T10:22:11.231 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:11 GMT /vclfcn-17/ 2024-02-01T10:22:11.232 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:hkxaNl2qfovb0VWctq15J2stLQI= 2024-02-01T10:22:11.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:hkxaNl2qfovb0VWctq15J2stLQI=', 'Content-Length': '0'} 2024-02-01T10:22:11.236 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078dbc4b2652ff74f-0065bb70d3-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:11.236 DEBUG:boto:b'vclfcn-17dummy1000falsedummy2024-02-01T10:20:14.297Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:11.237 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:11.237 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:11.237 DEBUG:boto:path=/vclfcn-17/?prefix=dummy 2024-02-01T10:22:11.237 DEBUG:boto:auth_path=/vclfcn-17/?prefix=dummy 2024-02-01T10:22:11.237 DEBUG:boto:Method: GET 2024-02-01T10:22:11.237 DEBUG:boto:Path: /vclfcn-17/?prefix=dummy 2024-02-01T10:22:11.237 DEBUG:boto:Data: 2024-02-01T10:22:11.237 DEBUG:boto:Headers: {} 2024-02-01T10:22:11.237 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:11.237 DEBUG:boto:Port: 8000 2024-02-01T10:22:11.238 DEBUG:boto:Params: {} 2024-02-01T10:22:11.238 DEBUG:boto:Token: None 2024-02-01T10:22:11.238 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:11 GMT /vclfcn-17/ 2024-02-01T10:22:11.238 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:hkxaNl2qfovb0VWctq15J2stLQI= 2024-02-01T10:22:11.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:hkxaNl2qfovb0VWctq15J2stLQI=', 'Content-Length': '0'} 2024-02-01T10:22:11.242 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c5f7dfb31128c5d-0065bb70d3-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:11.242 DEBUG:boto:b'vclfcn-17dummy1000falsedummy2024-02-01T10:20:14.297Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:11.243 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-02-01T10:22:11.243 DEBUG:boto:path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:11.243 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:11.243 DEBUG:boto:path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:11.243 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:11.243 DEBUG:boto:Method: GET 2024-02-01T10:22:11.243 DEBUG:boto:Path: /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:11.243 DEBUG:boto:Data: 2024-02-01T10:22:11.243 DEBUG:boto:Headers: {} 2024-02-01T10:22:11.243 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:11.244 DEBUG:boto:Port: 8000 2024-02-01T10:22:11.244 DEBUG:boto:Params: {} 2024-02-01T10:22:11.244 DEBUG:boto:Token: None 2024-02-01T10:22:11.244 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:11 GMT /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:11.244 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Wfdjv7CTdabbFJvC1bUYWY4G8D0= 2024-02-01T10:22:11.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Wfdjv7CTdabbFJvC1bUYWY4G8D0=', 'Content-Length': '0'} 2024-02-01T10:22:11.247 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'xMxAGR4A-xEILepnzFPRgrWNTOLvitz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009390136f619a0fa2-0065bb70d3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:11.248 DEBUG:boto:path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:11.248 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:11.248 DEBUG:boto:path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:11.248 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:11.248 DEBUG:boto:Method: GET 2024-02-01T10:22:11.248 DEBUG:boto:Path: /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:11.248 DEBUG:boto:Data: 2024-02-01T10:22:11.248 DEBUG:boto:Headers: {} 2024-02-01T10:22:11.248 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:11.248 DEBUG:boto:Port: 8000 2024-02-01T10:22:11.249 DEBUG:boto:Params: {} 2024-02-01T10:22:11.249 DEBUG:boto:Token: None 2024-02-01T10:22:11.249 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:11 GMT /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:11.249 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Wfdjv7CTdabbFJvC1bUYWY4G8D0= 2024-02-01T10:22:11.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Wfdjv7CTdabbFJvC1bUYWY4G8D0=', 'Content-Length': '0'} 2024-02-01T10:22:11.252 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'xMxAGR4A-xEILepnzFPRgrWNTOLvitz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008339988367b37102-0065bb70d3-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:11.296 DEBUG:boto:path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:11.296 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:11.296 DEBUG:boto:path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:11.296 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:11.296 DEBUG:boto:Method: HEAD 2024-02-01T10:22:11.297 DEBUG:boto:Path: /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:11.297 DEBUG:boto:Data: 2024-02-01T10:22:11.297 DEBUG:boto:Headers: {} 2024-02-01T10:22:11.297 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:11.297 DEBUG:boto:Port: 8000 2024-02-01T10:22:11.297 DEBUG:boto:Params: {} 2024-02-01T10:22:11.297 DEBUG:boto:Token: None 2024-02-01T10:22:11.297 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:11 GMT /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:11.297 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nNW38K9HocdBZlN2pRUOZL9GQCU= 2024-02-01T10:22:11.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nNW38K9HocdBZlN2pRUOZL9GQCU=', 'Content-Length': '0'} 2024-02-01T10:22:11.301 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'xMxAGR4A-xEILepnzFPRgrWNTOLvitz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000053e4694da1ea758c-0065bb70d3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:11.301 DEBUG:boto:path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:11.301 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:11.301 DEBUG:boto:path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:11.301 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:11.301 DEBUG:boto:Method: HEAD 2024-02-01T10:22:11.302 DEBUG:boto:Path: /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:11.302 DEBUG:boto:Data: 2024-02-01T10:22:11.302 DEBUG:boto:Headers: {} 2024-02-01T10:22:11.302 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:11.302 DEBUG:boto:Port: 8000 2024-02-01T10:22:11.302 DEBUG:boto:Params: {} 2024-02-01T10:22:11.302 DEBUG:boto:Token: None 2024-02-01T10:22:11.302 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:11 GMT /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:11.302 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nNW38K9HocdBZlN2pRUOZL9GQCU= 2024-02-01T10:22:11.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nNW38K9HocdBZlN2pRUOZL9GQCU=', 'Content-Length': '0'} 2024-02-01T10:22:11.305 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'xMxAGR4A-xEILepnzFPRgrWNTOLvitz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c086354cd414e394-0065bb70d3-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:11.306 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-02-01T10:22:11.306 DEBUG:boto:path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:11.306 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:11.306 DEBUG:boto:path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:11.306 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:11.306 DEBUG:boto:Method: GET 2024-02-01T10:22:11.306 DEBUG:boto:Path: /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:11.306 DEBUG:boto:Data: 2024-02-01T10:22:11.307 DEBUG:boto:Headers: {} 2024-02-01T10:22:11.307 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:11.307 DEBUG:boto:Port: 8000 2024-02-01T10:22:11.307 DEBUG:boto:Params: {} 2024-02-01T10:22:11.307 DEBUG:boto:Token: None 2024-02-01T10:22:11.307 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:11 GMT /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:11.307 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Wfdjv7CTdabbFJvC1bUYWY4G8D0= 2024-02-01T10:22:11.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Wfdjv7CTdabbFJvC1bUYWY4G8D0=', 'Content-Length': '0'} 2024-02-01T10:22:11.310 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'xMxAGR4A-xEILepnzFPRgrWNTOLvitz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003b4ea986657d9482-0065bb70d3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:11.352 DEBUG:boto:path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:11.352 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:11.352 DEBUG:boto:path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:11.352 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:11.353 DEBUG:boto:Method: GET 2024-02-01T10:22:11.353 DEBUG:boto:Path: /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:11.353 DEBUG:boto:Data: 2024-02-01T10:22:11.353 DEBUG:boto:Headers: {} 2024-02-01T10:22:11.353 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:11.353 DEBUG:boto:Port: 8000 2024-02-01T10:22:11.353 DEBUG:boto:Params: {} 2024-02-01T10:22:11.353 DEBUG:boto:Token: None 2024-02-01T10:22:11.353 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:11 GMT /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:11.354 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Wfdjv7CTdabbFJvC1bUYWY4G8D0= 2024-02-01T10:22:11.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Wfdjv7CTdabbFJvC1bUYWY4G8D0=', 'Content-Length': '0'} 2024-02-01T10:22:11.357 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'xMxAGR4A-xEILepnzFPRgrWNTOLvitz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000067a2ccdfc3e749a6-0065bb70d3-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:11.357 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:11.357 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:11.357 DEBUG:boto:path=/vclfcn-17/?prefix=obj-test-zone1 2024-02-01T10:22:11.358 DEBUG:boto:auth_path=/vclfcn-17/?prefix=obj-test-zone1 2024-02-01T10:22:11.358 DEBUG:boto:Method: GET 2024-02-01T10:22:11.358 DEBUG:boto:Path: /vclfcn-17/?prefix=obj-test-zone1 2024-02-01T10:22:11.358 DEBUG:boto:Data: 2024-02-01T10:22:11.358 DEBUG:boto:Headers: {} 2024-02-01T10:22:11.358 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:11.358 DEBUG:boto:Port: 8000 2024-02-01T10:22:11.358 DEBUG:boto:Params: {} 2024-02-01T10:22:11.358 DEBUG:boto:Token: None 2024-02-01T10:22:11.358 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:11 GMT /vclfcn-17/ 2024-02-01T10:22:11.358 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:hkxaNl2qfovb0VWctq15J2stLQI= 2024-02-01T10:22:11.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:hkxaNl2qfovb0VWctq15J2stLQI=', 'Content-Length': '0'} 2024-02-01T10:22:11.363 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd7258e7468fff86-0065bb70d3-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:11.363 DEBUG:boto:b'vclfcn-17obj-test-zone11000falseobj-test-zone12024-02-01T10:20:50.473Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:11.363 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:11.363 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:11.363 DEBUG:boto:path=/vclfcn-17/?prefix=obj-test-zone1 2024-02-01T10:22:11.364 DEBUG:boto:auth_path=/vclfcn-17/?prefix=obj-test-zone1 2024-02-01T10:22:11.364 DEBUG:boto:Method: GET 2024-02-01T10:22:11.364 DEBUG:boto:Path: /vclfcn-17/?prefix=obj-test-zone1 2024-02-01T10:22:11.364 DEBUG:boto:Data: 2024-02-01T10:22:11.364 DEBUG:boto:Headers: {} 2024-02-01T10:22:11.364 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:11.364 DEBUG:boto:Port: 8000 2024-02-01T10:22:11.364 DEBUG:boto:Params: {} 2024-02-01T10:22:11.364 DEBUG:boto:Token: None 2024-02-01T10:22:11.365 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:11 GMT /vclfcn-17/ 2024-02-01T10:22:11.365 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:hkxaNl2qfovb0VWctq15J2stLQI= 2024-02-01T10:22:11.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:hkxaNl2qfovb0VWctq15J2stLQI=', 'Content-Length': '0'} 2024-02-01T10:22:11.369 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007edca39893805f99-0065bb70d3-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:11.369 DEBUG:boto:b'vclfcn-17obj-test-zone11000falseobj-test-zone12024-02-01T10:20:50.473Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:11.370 DEBUG:boto:path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:11.370 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:11.370 DEBUG:boto:Method: HEAD 2024-02-01T10:22:11.370 DEBUG:boto:Path: /vclfcn-17/obj-test-zone1 2024-02-01T10:22:11.370 DEBUG:boto:Data: 2024-02-01T10:22:11.370 DEBUG:boto:Headers: {} 2024-02-01T10:22:11.370 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:11.370 DEBUG:boto:Port: 8000 2024-02-01T10:22:11.370 DEBUG:boto:Params: {} 2024-02-01T10:22:11.370 DEBUG:boto:Token: None 2024-02-01T10:22:11.371 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:11 GMT /vclfcn-17/obj-test-zone1 2024-02-01T10:22:11.371 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Hrc3IN/Eu34Yz8SadfJXpWIrEDE= 2024-02-01T10:22:11.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Hrc3IN/Eu34Yz8SadfJXpWIrEDE=', 'Content-Length': '0'} 2024-02-01T10:22:11.374 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'xMxAGR4A-xEILepnzFPRgrWNTOLvitz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000167ce792ea9b8235-0065bb70d3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:11.375 DEBUG:boto:path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:11.375 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:11.375 DEBUG:boto:Method: HEAD 2024-02-01T10:22:11.375 DEBUG:boto:Path: /vclfcn-17/obj-test-zone1 2024-02-01T10:22:11.375 DEBUG:boto:Data: 2024-02-01T10:22:11.375 DEBUG:boto:Headers: {} 2024-02-01T10:22:11.375 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:11.375 DEBUG:boto:Port: 8000 2024-02-01T10:22:11.375 DEBUG:boto:Params: {} 2024-02-01T10:22:11.375 DEBUG:boto:Token: None 2024-02-01T10:22:11.375 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:11 GMT /vclfcn-17/obj-test-zone1 2024-02-01T10:22:11.376 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Hrc3IN/Eu34Yz8SadfJXpWIrEDE= 2024-02-01T10:22:11.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Hrc3IN/Eu34Yz8SadfJXpWIrEDE=', 'Content-Length': '0'} 2024-02-01T10:22:11.379 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'xMxAGR4A-xEILepnzFPRgrWNTOLvitz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009a483a630ec3f369-0065bb70d3-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:11.379 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-02-01T10:22:11.380 DEBUG:boto:path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:11.380 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:11.380 DEBUG:boto:path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:11.380 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:11.380 DEBUG:boto:Method: GET 2024-02-01T10:22:11.380 DEBUG:boto:Path: /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:11.380 DEBUG:boto:Data: 2024-02-01T10:22:11.380 DEBUG:boto:Headers: {} 2024-02-01T10:22:11.380 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:11.381 DEBUG:boto:Port: 8000 2024-02-01T10:22:11.381 DEBUG:boto:Params: {} 2024-02-01T10:22:11.381 DEBUG:boto:Token: None 2024-02-01T10:22:11.381 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:11 GMT /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:11.381 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Wfdjv7CTdabbFJvC1bUYWY4G8D0= 2024-02-01T10:22:11.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Wfdjv7CTdabbFJvC1bUYWY4G8D0=', 'Content-Length': '0'} 2024-02-01T10:22:11.384 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'xMxAGR4A-xEILepnzFPRgrWNTOLvitz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000009048c38ef11fe60-0065bb70d3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:11.428 DEBUG:boto:path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:11.428 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:11.428 DEBUG:boto:path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:11.428 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:11.429 DEBUG:boto:Method: GET 2024-02-01T10:22:11.429 DEBUG:boto:Path: /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:11.429 DEBUG:boto:Data: 2024-02-01T10:22:11.429 DEBUG:boto:Headers: {} 2024-02-01T10:22:11.429 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:11.429 DEBUG:boto:Port: 8000 2024-02-01T10:22:11.429 DEBUG:boto:Params: {} 2024-02-01T10:22:11.429 DEBUG:boto:Token: None 2024-02-01T10:22:11.429 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:11 GMT /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:11.430 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Wfdjv7CTdabbFJvC1bUYWY4G8D0= 2024-02-01T10:22:11.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Wfdjv7CTdabbFJvC1bUYWY4G8D0=', 'Content-Length': '0'} 2024-02-01T10:22:11.433 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'xMxAGR4A-xEILepnzFPRgrWNTOLvitz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007e36adb9f6b62f78-0065bb70d3-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:11.433 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:11.433 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:11.433 DEBUG:boto:path=/vclfcn-17/?prefix=obj-test-zone1 2024-02-01T10:22:11.433 DEBUG:boto:auth_path=/vclfcn-17/?prefix=obj-test-zone1 2024-02-01T10:22:11.433 DEBUG:boto:Method: GET 2024-02-01T10:22:11.433 DEBUG:boto:Path: /vclfcn-17/?prefix=obj-test-zone1 2024-02-01T10:22:11.434 DEBUG:boto:Data: 2024-02-01T10:22:11.434 DEBUG:boto:Headers: {} 2024-02-01T10:22:11.434 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:11.434 DEBUG:boto:Port: 8000 2024-02-01T10:22:11.434 DEBUG:boto:Params: {} 2024-02-01T10:22:11.434 DEBUG:boto:Token: None 2024-02-01T10:22:11.434 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:11 GMT /vclfcn-17/ 2024-02-01T10:22:11.434 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:hkxaNl2qfovb0VWctq15J2stLQI= 2024-02-01T10:22:11.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:hkxaNl2qfovb0VWctq15J2stLQI=', 'Content-Length': '0'} 2024-02-01T10:22:11.439 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005786f588dca974fa-0065bb70d3-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:11.439 DEBUG:boto:b'vclfcn-17obj-test-zone11000falseobj-test-zone12024-02-01T10:20:50.473Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:11.440 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:11.440 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:11.440 DEBUG:boto:path=/vclfcn-17/?prefix=obj-test-zone1 2024-02-01T10:22:11.440 DEBUG:boto:auth_path=/vclfcn-17/?prefix=obj-test-zone1 2024-02-01T10:22:11.440 DEBUG:boto:Method: GET 2024-02-01T10:22:11.440 DEBUG:boto:Path: /vclfcn-17/?prefix=obj-test-zone1 2024-02-01T10:22:11.440 DEBUG:boto:Data: 2024-02-01T10:22:11.440 DEBUG:boto:Headers: {} 2024-02-01T10:22:11.440 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:11.441 DEBUG:boto:Port: 8000 2024-02-01T10:22:11.441 DEBUG:boto:Params: {} 2024-02-01T10:22:11.441 DEBUG:boto:Token: None 2024-02-01T10:22:11.441 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:11 GMT /vclfcn-17/ 2024-02-01T10:22:11.441 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:hkxaNl2qfovb0VWctq15J2stLQI= 2024-02-01T10:22:11.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:hkxaNl2qfovb0VWctq15J2stLQI=', 'Content-Length': '0'} 2024-02-01T10:22:11.446 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd28b17e809adfe0-0065bb70d3-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:11.446 DEBUG:boto:b'vclfcn-17obj-test-zone11000falseobj-test-zone12024-02-01T10:20:50.473Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:11.446 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-02-01T10:22:11.446 DEBUG:boto:path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:11.447 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:11.447 DEBUG:boto:path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:11.447 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:11.447 DEBUG:boto:Method: GET 2024-02-01T10:22:11.447 DEBUG:boto:Path: /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:11.447 DEBUG:boto:Data: 2024-02-01T10:22:11.447 DEBUG:boto:Headers: {} 2024-02-01T10:22:11.447 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:11.447 DEBUG:boto:Port: 8000 2024-02-01T10:22:11.447 DEBUG:boto:Params: {} 2024-02-01T10:22:11.447 DEBUG:boto:Token: None 2024-02-01T10:22:11.448 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:11 GMT /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:11.448 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BkKyo+K5puM6Mke75EUThD1y5Gg= 2024-02-01T10:22:11.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BkKyo+K5puM6Mke75EUThD1y5Gg=', 'Content-Length': '0'} 2024-02-01T10:22:11.451 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', '-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c20de90a3648d26d-0065bb70d3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:11.451 DEBUG:boto:path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:11.451 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:11.451 DEBUG:boto:path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:11.452 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:11.452 DEBUG:boto:Method: GET 2024-02-01T10:22:11.452 DEBUG:boto:Path: /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:11.452 DEBUG:boto:Data: 2024-02-01T10:22:11.452 DEBUG:boto:Headers: {} 2024-02-01T10:22:11.452 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:11.452 DEBUG:boto:Port: 8000 2024-02-01T10:22:11.452 DEBUG:boto:Params: {} 2024-02-01T10:22:11.452 DEBUG:boto:Token: None 2024-02-01T10:22:11.452 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:11 GMT /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:11.453 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BkKyo+K5puM6Mke75EUThD1y5Gg= 2024-02-01T10:22:11.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BkKyo+K5puM6Mke75EUThD1y5Gg=', 'Content-Length': '0'} 2024-02-01T10:22:11.455 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', '-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aed8b1d86da7f29e-0065bb70d3-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:11.496 DEBUG:boto:path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:11.496 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:11.497 DEBUG:boto:path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:11.497 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:11.497 DEBUG:boto:Method: HEAD 2024-02-01T10:22:11.497 DEBUG:boto:Path: /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:11.497 DEBUG:boto:Data: 2024-02-01T10:22:11.497 DEBUG:boto:Headers: {} 2024-02-01T10:22:11.497 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:11.497 DEBUG:boto:Port: 8000 2024-02-01T10:22:11.497 DEBUG:boto:Params: {} 2024-02-01T10:22:11.498 DEBUG:boto:Token: None 2024-02-01T10:22:11.498 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:11 GMT /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:11.498 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ADIScqrSImPUGYs1NDKCUoXONpM= 2024-02-01T10:22:11.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ADIScqrSImPUGYs1NDKCUoXONpM=', 'Content-Length': '0'} 2024-02-01T10:22:11.501 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', '-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000062144eb5be274dcf-0065bb70d3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:11.502 DEBUG:boto:path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:11.502 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:11.502 DEBUG:boto:path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:11.502 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:11.502 DEBUG:boto:Method: HEAD 2024-02-01T10:22:11.502 DEBUG:boto:Path: /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:11.502 DEBUG:boto:Data: 2024-02-01T10:22:11.502 DEBUG:boto:Headers: {} 2024-02-01T10:22:11.503 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:11.503 DEBUG:boto:Port: 8000 2024-02-01T10:22:11.503 DEBUG:boto:Params: {} 2024-02-01T10:22:11.503 DEBUG:boto:Token: None 2024-02-01T10:22:11.503 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:11 GMT /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:11.503 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ADIScqrSImPUGYs1NDKCUoXONpM= 2024-02-01T10:22:11.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ADIScqrSImPUGYs1NDKCUoXONpM=', 'Content-Length': '0'} 2024-02-01T10:22:11.506 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', '-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008bc65df1e96c58de-0065bb70d3-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:11.506 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-02-01T10:22:11.506 DEBUG:boto:path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:11.507 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:11.507 DEBUG:boto:path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:11.507 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:11.507 DEBUG:boto:Method: GET 2024-02-01T10:22:11.507 DEBUG:boto:Path: /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:11.507 DEBUG:boto:Data: 2024-02-01T10:22:11.507 DEBUG:boto:Headers: {} 2024-02-01T10:22:11.507 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:11.507 DEBUG:boto:Port: 8000 2024-02-01T10:22:11.507 DEBUG:boto:Params: {} 2024-02-01T10:22:11.508 DEBUG:boto:Token: None 2024-02-01T10:22:11.508 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:11 GMT /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:11.508 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BkKyo+K5puM6Mke75EUThD1y5Gg= 2024-02-01T10:22:11.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BkKyo+K5puM6Mke75EUThD1y5Gg=', 'Content-Length': '0'} 2024-02-01T10:22:11.511 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', '-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ea104fe4819cc7c6-0065bb70d3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:11.556 DEBUG:boto:path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:11.556 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:11.556 DEBUG:boto:path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:11.556 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:11.557 DEBUG:boto:Method: GET 2024-02-01T10:22:11.557 DEBUG:boto:Path: /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:11.557 DEBUG:boto:Data: 2024-02-01T10:22:11.557 DEBUG:boto:Headers: {} 2024-02-01T10:22:11.557 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:11.557 DEBUG:boto:Port: 8000 2024-02-01T10:22:11.557 DEBUG:boto:Params: {} 2024-02-01T10:22:11.557 DEBUG:boto:Token: None 2024-02-01T10:22:11.557 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:11 GMT /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:11.558 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BkKyo+K5puM6Mke75EUThD1y5Gg= 2024-02-01T10:22:11.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BkKyo+K5puM6Mke75EUThD1y5Gg=', 'Content-Length': '0'} 2024-02-01T10:22:11.561 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', '-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005fca46ff32cead35-0065bb70d3-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:11.608 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:11.608 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:11.609 DEBUG:boto:path=/vclfcn-17/?prefix=obj-test-zone2 2024-02-01T10:22:11.609 DEBUG:boto:auth_path=/vclfcn-17/?prefix=obj-test-zone2 2024-02-01T10:22:11.609 DEBUG:boto:Method: GET 2024-02-01T10:22:11.609 DEBUG:boto:Path: /vclfcn-17/?prefix=obj-test-zone2 2024-02-01T10:22:11.609 DEBUG:boto:Data: 2024-02-01T10:22:11.609 DEBUG:boto:Headers: {} 2024-02-01T10:22:11.609 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:11.609 DEBUG:boto:Port: 8000 2024-02-01T10:22:11.609 DEBUG:boto:Params: {} 2024-02-01T10:22:11.610 DEBUG:boto:Token: None 2024-02-01T10:22:11.610 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:11 GMT /vclfcn-17/ 2024-02-01T10:22:11.610 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:hkxaNl2qfovb0VWctq15J2stLQI= 2024-02-01T10:22:11.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:hkxaNl2qfovb0VWctq15J2stLQI=', 'Content-Length': '0'} 2024-02-01T10:22:11.614 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f4af5ae6764b848-0065bb70d3-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:11.615 DEBUG:boto:b'vclfcn-17obj-test-zone21000falseobj-test-zone22024-02-01T10:20:50.624Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:11.615 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:11.615 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:11.615 DEBUG:boto:path=/vclfcn-17/?prefix=obj-test-zone2 2024-02-01T10:22:11.615 DEBUG:boto:auth_path=/vclfcn-17/?prefix=obj-test-zone2 2024-02-01T10:22:11.615 DEBUG:boto:Method: GET 2024-02-01T10:22:11.616 DEBUG:boto:Path: /vclfcn-17/?prefix=obj-test-zone2 2024-02-01T10:22:11.616 DEBUG:boto:Data: 2024-02-01T10:22:11.616 DEBUG:boto:Headers: {} 2024-02-01T10:22:11.616 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:11.616 DEBUG:boto:Port: 8000 2024-02-01T10:22:11.616 DEBUG:boto:Params: {} 2024-02-01T10:22:11.616 DEBUG:boto:Token: None 2024-02-01T10:22:11.616 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:11 GMT /vclfcn-17/ 2024-02-01T10:22:11.616 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:hkxaNl2qfovb0VWctq15J2stLQI= 2024-02-01T10:22:11.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:hkxaNl2qfovb0VWctq15J2stLQI=', 'Content-Length': '0'} 2024-02-01T10:22:11.621 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e7b3f0678e0c3438-0065bb70d3-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:11.621 DEBUG:boto:b'vclfcn-17obj-test-zone21000falseobj-test-zone22024-02-01T10:20:50.624Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:11.622 DEBUG:boto:path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:11.622 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:11.622 DEBUG:boto:Method: HEAD 2024-02-01T10:22:11.622 DEBUG:boto:Path: /vclfcn-17/obj-test-zone2 2024-02-01T10:22:11.622 DEBUG:boto:Data: 2024-02-01T10:22:11.622 DEBUG:boto:Headers: {} 2024-02-01T10:22:11.622 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:11.622 DEBUG:boto:Port: 8000 2024-02-01T10:22:11.623 DEBUG:boto:Params: {} 2024-02-01T10:22:11.623 DEBUG:boto:Token: None 2024-02-01T10:22:11.623 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:11 GMT /vclfcn-17/obj-test-zone2 2024-02-01T10:22:11.623 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:LdF2IY+ZCTcSlxG7L6T8Ym0m+GU= 2024-02-01T10:22:11.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:LdF2IY+ZCTcSlxG7L6T8Ym0m+GU=', 'Content-Length': '0'} 2024-02-01T10:22:11.627 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', '-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dfc37e55382bbb3a-0065bb70d3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:11.627 DEBUG:boto:path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:11.627 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:11.627 DEBUG:boto:Method: HEAD 2024-02-01T10:22:11.627 DEBUG:boto:Path: /vclfcn-17/obj-test-zone2 2024-02-01T10:22:11.627 DEBUG:boto:Data: 2024-02-01T10:22:11.627 DEBUG:boto:Headers: {} 2024-02-01T10:22:11.628 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:11.628 DEBUG:boto:Port: 8000 2024-02-01T10:22:11.628 DEBUG:boto:Params: {} 2024-02-01T10:22:11.628 DEBUG:boto:Token: None 2024-02-01T10:22:11.628 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:11 GMT /vclfcn-17/obj-test-zone2 2024-02-01T10:22:11.628 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:LdF2IY+ZCTcSlxG7L6T8Ym0m+GU= 2024-02-01T10:22:11.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:LdF2IY+ZCTcSlxG7L6T8Ym0m+GU=', 'Content-Length': '0'} 2024-02-01T10:22:11.632 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', '-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e19d91d3d92f1b31-0065bb70d3-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:11.632 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-02-01T10:22:11.633 DEBUG:boto:path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:11.633 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:11.633 DEBUG:boto:path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:11.633 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:11.633 DEBUG:boto:Method: GET 2024-02-01T10:22:11.633 DEBUG:boto:Path: /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:11.633 DEBUG:boto:Data: 2024-02-01T10:22:11.633 DEBUG:boto:Headers: {} 2024-02-01T10:22:11.633 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:11.633 DEBUG:boto:Port: 8000 2024-02-01T10:22:11.634 DEBUG:boto:Params: {} 2024-02-01T10:22:11.634 DEBUG:boto:Token: None 2024-02-01T10:22:11.634 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:11 GMT /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:11.634 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BkKyo+K5puM6Mke75EUThD1y5Gg= 2024-02-01T10:22:11.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BkKyo+K5puM6Mke75EUThD1y5Gg=', 'Content-Length': '0'} 2024-02-01T10:22:11.637 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', '-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a4209a5d206ceb6c-0065bb70d3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:11.637 DEBUG:boto:path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:11.637 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:11.638 DEBUG:boto:path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:11.638 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:11.638 DEBUG:boto:Method: GET 2024-02-01T10:22:11.638 DEBUG:boto:Path: /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:11.638 DEBUG:boto:Data: 2024-02-01T10:22:11.638 DEBUG:boto:Headers: {} 2024-02-01T10:22:11.638 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:11.638 DEBUG:boto:Port: 8000 2024-02-01T10:22:11.638 DEBUG:boto:Params: {} 2024-02-01T10:22:11.638 DEBUG:boto:Token: None 2024-02-01T10:22:11.639 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:11 GMT /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:11.639 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BkKyo+K5puM6Mke75EUThD1y5Gg= 2024-02-01T10:22:11.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BkKyo+K5puM6Mke75EUThD1y5Gg=', 'Content-Length': '0'} 2024-02-01T10:22:11.642 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', '-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eb6be202c1da5fd6-0065bb70d3-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:11.684 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:11.684 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:11.684 DEBUG:boto:path=/vclfcn-17/?prefix=obj-test-zone2 2024-02-01T10:22:11.685 DEBUG:boto:auth_path=/vclfcn-17/?prefix=obj-test-zone2 2024-02-01T10:22:11.685 DEBUG:boto:Method: GET 2024-02-01T10:22:11.685 DEBUG:boto:Path: /vclfcn-17/?prefix=obj-test-zone2 2024-02-01T10:22:11.685 DEBUG:boto:Data: 2024-02-01T10:22:11.685 DEBUG:boto:Headers: {} 2024-02-01T10:22:11.685 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:11.685 DEBUG:boto:Port: 8000 2024-02-01T10:22:11.685 DEBUG:boto:Params: {} 2024-02-01T10:22:11.685 DEBUG:boto:Token: None 2024-02-01T10:22:11.686 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:11 GMT /vclfcn-17/ 2024-02-01T10:22:11.686 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:hkxaNl2qfovb0VWctq15J2stLQI= 2024-02-01T10:22:11.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:hkxaNl2qfovb0VWctq15J2stLQI=', 'Content-Length': '0'} 2024-02-01T10:22:11.690 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cedb45ee802ea71c-0065bb70d3-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:11.691 DEBUG:boto:b'vclfcn-17obj-test-zone21000falseobj-test-zone22024-02-01T10:20:50.624Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:11.691 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:11.691 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:11.691 DEBUG:boto:path=/vclfcn-17/?prefix=obj-test-zone2 2024-02-01T10:22:11.691 DEBUG:boto:auth_path=/vclfcn-17/?prefix=obj-test-zone2 2024-02-01T10:22:11.691 DEBUG:boto:Method: GET 2024-02-01T10:22:11.692 DEBUG:boto:Path: /vclfcn-17/?prefix=obj-test-zone2 2024-02-01T10:22:11.692 DEBUG:boto:Data: 2024-02-01T10:22:11.692 DEBUG:boto:Headers: {} 2024-02-01T10:22:11.692 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:11.692 DEBUG:boto:Port: 8000 2024-02-01T10:22:11.692 DEBUG:boto:Params: {} 2024-02-01T10:22:11.692 DEBUG:boto:Token: None 2024-02-01T10:22:11.692 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:11 GMT /vclfcn-17/ 2024-02-01T10:22:11.692 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:hkxaNl2qfovb0VWctq15J2stLQI= 2024-02-01T10:22:11.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:hkxaNl2qfovb0VWctq15J2stLQI=', 'Content-Length': '0'} 2024-02-01T10:22:11.697 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000183d3d1bd7f9589b-0065bb70d3-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:11.697 DEBUG:boto:b'vclfcn-17obj-test-zone21000falseobj-test-zone22024-02-01T10:20:50.624Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:11.698 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-02-01T10:22:11.698 DEBUG:boto:path=/vclfcn-17/obj-test-zone3 2024-02-01T10:22:11.698 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3 2024-02-01T10:22:11.698 DEBUG:boto:path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:11.698 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:11.698 DEBUG:boto:Method: GET 2024-02-01T10:22:11.698 DEBUG:boto:Path: /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:11.698 DEBUG:boto:Data: 2024-02-01T10:22:11.698 DEBUG:boto:Headers: {} 2024-02-01T10:22:11.699 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:11.699 DEBUG:boto:Port: 8000 2024-02-01T10:22:11.699 DEBUG:boto:Params: {} 2024-02-01T10:22:11.699 DEBUG:boto:Token: None 2024-02-01T10:22:11.699 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:11 GMT /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:11.699 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BDCcdr4qxWKrQ+xkWg7wpKTtBO4= 2024-02-01T10:22:11.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BDCcdr4qxWKrQ+xkWg7wpKTtBO4=', 'Content-Length': '0'} 2024-02-01T10:22:11.702 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'YE736OwOOXbZJHJe7d8YvcNV1LxwcQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bd530b35b8c0aeff-0065bb70d3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:11.744 DEBUG:boto:path=/vclfcn-17/obj-test-zone3 2024-02-01T10:22:11.744 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3 2024-02-01T10:22:11.744 DEBUG:boto:path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:11.745 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:11.745 DEBUG:boto:Method: GET 2024-02-01T10:22:11.745 DEBUG:boto:Path: /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:11.745 DEBUG:boto:Data: 2024-02-01T10:22:11.745 DEBUG:boto:Headers: {} 2024-02-01T10:22:11.745 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:11.745 DEBUG:boto:Port: 8000 2024-02-01T10:22:11.745 DEBUG:boto:Params: {} 2024-02-01T10:22:11.745 DEBUG:boto:Token: None 2024-02-01T10:22:11.746 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:11 GMT /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:11.746 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BDCcdr4qxWKrQ+xkWg7wpKTtBO4= 2024-02-01T10:22:11.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BDCcdr4qxWKrQ+xkWg7wpKTtBO4=', 'Content-Length': '0'} 2024-02-01T10:22:11.749 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'YE736OwOOXbZJHJe7d8YvcNV1LxwcQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a5e305039f32f4db-0065bb70d3-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:11.750 DEBUG:boto:path=/vclfcn-17/obj-test-zone3 2024-02-01T10:22:11.750 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3 2024-02-01T10:22:11.750 DEBUG:boto:path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:11.750 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:11.750 DEBUG:boto:Method: HEAD 2024-02-01T10:22:11.750 DEBUG:boto:Path: /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:11.750 DEBUG:boto:Data: 2024-02-01T10:22:11.750 DEBUG:boto:Headers: {} 2024-02-01T10:22:11.751 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:11.751 DEBUG:boto:Port: 8000 2024-02-01T10:22:11.751 DEBUG:boto:Params: {} 2024-02-01T10:22:11.751 DEBUG:boto:Token: None 2024-02-01T10:22:11.751 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:11 GMT /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:11.751 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:eZD1cvkEHaWZDP/RwX+J3L7PTJQ= 2024-02-01T10:22:11.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:eZD1cvkEHaWZDP/RwX+J3L7PTJQ=', 'Content-Length': '0'} 2024-02-01T10:22:11.754 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'YE736OwOOXbZJHJe7d8YvcNV1LxwcQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000017ca626af0dd8dfb-0065bb70d3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:11.755 DEBUG:boto:path=/vclfcn-17/obj-test-zone3 2024-02-01T10:22:11.755 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3 2024-02-01T10:22:11.755 DEBUG:boto:path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:11.755 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:11.755 DEBUG:boto:Method: HEAD 2024-02-01T10:22:11.755 DEBUG:boto:Path: /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:11.755 DEBUG:boto:Data: 2024-02-01T10:22:11.755 DEBUG:boto:Headers: {} 2024-02-01T10:22:11.755 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:11.756 DEBUG:boto:Port: 8000 2024-02-01T10:22:11.756 DEBUG:boto:Params: {} 2024-02-01T10:22:11.756 DEBUG:boto:Token: None 2024-02-01T10:22:11.756 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:11 GMT /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:11.756 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:eZD1cvkEHaWZDP/RwX+J3L7PTJQ= 2024-02-01T10:22:11.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:eZD1cvkEHaWZDP/RwX+J3L7PTJQ=', 'Content-Length': '0'} 2024-02-01T10:22:11.759 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'YE736OwOOXbZJHJe7d8YvcNV1LxwcQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006641254e64ed63ac-0065bb70d3-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:11.759 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-02-01T10:22:11.759 DEBUG:boto:path=/vclfcn-17/obj-test-zone3 2024-02-01T10:22:11.760 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3 2024-02-01T10:22:11.760 DEBUG:boto:path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:11.760 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:11.760 DEBUG:boto:Method: GET 2024-02-01T10:22:11.760 DEBUG:boto:Path: /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:11.760 DEBUG:boto:Data: 2024-02-01T10:22:11.760 DEBUG:boto:Headers: {} 2024-02-01T10:22:11.760 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:11.760 DEBUG:boto:Port: 8000 2024-02-01T10:22:11.761 DEBUG:boto:Params: {} 2024-02-01T10:22:11.761 DEBUG:boto:Token: None 2024-02-01T10:22:11.761 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:11 GMT /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:11.761 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BDCcdr4qxWKrQ+xkWg7wpKTtBO4= 2024-02-01T10:22:11.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BDCcdr4qxWKrQ+xkWg7wpKTtBO4=', 'Content-Length': '0'} 2024-02-01T10:22:11.764 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'YE736OwOOXbZJHJe7d8YvcNV1LxwcQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000093cd1d072f275c2b-0065bb70d3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:11.765 DEBUG:boto:path=/vclfcn-17/obj-test-zone3 2024-02-01T10:22:11.765 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3 2024-02-01T10:22:11.765 DEBUG:boto:path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:11.765 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:11.765 DEBUG:boto:Method: GET 2024-02-01T10:22:11.765 DEBUG:boto:Path: /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:11.765 DEBUG:boto:Data: 2024-02-01T10:22:11.765 DEBUG:boto:Headers: {} 2024-02-01T10:22:11.766 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:11.766 DEBUG:boto:Port: 8000 2024-02-01T10:22:11.766 DEBUG:boto:Params: {} 2024-02-01T10:22:11.766 DEBUG:boto:Token: None 2024-02-01T10:22:11.766 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:11 GMT /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:11.766 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BDCcdr4qxWKrQ+xkWg7wpKTtBO4= 2024-02-01T10:22:11.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BDCcdr4qxWKrQ+xkWg7wpKTtBO4=', 'Content-Length': '0'} 2024-02-01T10:22:11.769 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'YE736OwOOXbZJHJe7d8YvcNV1LxwcQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e30d5292f31e90e0-0065bb70d3-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:11.812 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:11.812 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:11.812 DEBUG:boto:path=/vclfcn-17/?prefix=obj-test-zone3 2024-02-01T10:22:11.813 DEBUG:boto:auth_path=/vclfcn-17/?prefix=obj-test-zone3 2024-02-01T10:22:11.813 DEBUG:boto:Method: GET 2024-02-01T10:22:11.813 DEBUG:boto:Path: /vclfcn-17/?prefix=obj-test-zone3 2024-02-01T10:22:11.813 DEBUG:boto:Data: 2024-02-01T10:22:11.813 DEBUG:boto:Headers: {} 2024-02-01T10:22:11.813 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:11.813 DEBUG:boto:Port: 8000 2024-02-01T10:22:11.813 DEBUG:boto:Params: {} 2024-02-01T10:22:11.813 DEBUG:boto:Token: None 2024-02-01T10:22:11.814 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:11 GMT /vclfcn-17/ 2024-02-01T10:22:11.814 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:hkxaNl2qfovb0VWctq15J2stLQI= 2024-02-01T10:22:11.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:hkxaNl2qfovb0VWctq15J2stLQI=', 'Content-Length': '0'} 2024-02-01T10:22:11.819 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e7b84d304aecc18-0065bb70d3-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:11.819 DEBUG:boto:b'vclfcn-17obj-test-zone31000falseobj-test-zone32024-02-01T10:20:50.780Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:11.819 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:11.819 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:11.820 DEBUG:boto:path=/vclfcn-17/?prefix=obj-test-zone3 2024-02-01T10:22:11.820 DEBUG:boto:auth_path=/vclfcn-17/?prefix=obj-test-zone3 2024-02-01T10:22:11.820 DEBUG:boto:Method: GET 2024-02-01T10:22:11.820 DEBUG:boto:Path: /vclfcn-17/?prefix=obj-test-zone3 2024-02-01T10:22:11.820 DEBUG:boto:Data: 2024-02-01T10:22:11.820 DEBUG:boto:Headers: {} 2024-02-01T10:22:11.820 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:11.820 DEBUG:boto:Port: 8000 2024-02-01T10:22:11.820 DEBUG:boto:Params: {} 2024-02-01T10:22:11.820 DEBUG:boto:Token: None 2024-02-01T10:22:11.821 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:11 GMT /vclfcn-17/ 2024-02-01T10:22:11.821 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:hkxaNl2qfovb0VWctq15J2stLQI= 2024-02-01T10:22:11.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:hkxaNl2qfovb0VWctq15J2stLQI=', 'Content-Length': '0'} 2024-02-01T10:22:11.825 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d771426ebbbf6e90-0065bb70d3-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:11.868 DEBUG:boto:b'vclfcn-17obj-test-zone31000falseobj-test-zone32024-02-01T10:20:50.780Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:11.869 DEBUG:boto:path=/vclfcn-17/obj-test-zone3 2024-02-01T10:22:11.869 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3 2024-02-01T10:22:11.869 DEBUG:boto:Method: HEAD 2024-02-01T10:22:11.869 DEBUG:boto:Path: /vclfcn-17/obj-test-zone3 2024-02-01T10:22:11.869 DEBUG:boto:Data: 2024-02-01T10:22:11.869 DEBUG:boto:Headers: {} 2024-02-01T10:22:11.869 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:11.869 DEBUG:boto:Port: 8000 2024-02-01T10:22:11.869 DEBUG:boto:Params: {} 2024-02-01T10:22:11.870 DEBUG:boto:Token: None 2024-02-01T10:22:11.870 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:11 GMT /vclfcn-17/obj-test-zone3 2024-02-01T10:22:11.870 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mPWES5gLT/XY6IVBNTtN6kdY8T4= 2024-02-01T10:22:11.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mPWES5gLT/XY6IVBNTtN6kdY8T4=', 'Content-Length': '0'} 2024-02-01T10:22:11.874 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'YE736OwOOXbZJHJe7d8YvcNV1LxwcQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007988876a7ec46930-0065bb70d3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:11.874 DEBUG:boto:path=/vclfcn-17/obj-test-zone3 2024-02-01T10:22:11.875 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3 2024-02-01T10:22:11.875 DEBUG:boto:Method: HEAD 2024-02-01T10:22:11.875 DEBUG:boto:Path: /vclfcn-17/obj-test-zone3 2024-02-01T10:22:11.875 DEBUG:boto:Data: 2024-02-01T10:22:11.875 DEBUG:boto:Headers: {} 2024-02-01T10:22:11.875 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:11.875 DEBUG:boto:Port: 8000 2024-02-01T10:22:11.875 DEBUG:boto:Params: {} 2024-02-01T10:22:11.875 DEBUG:boto:Token: None 2024-02-01T10:22:11.875 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:11 GMT /vclfcn-17/obj-test-zone3 2024-02-01T10:22:11.876 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mPWES5gLT/XY6IVBNTtN6kdY8T4= 2024-02-01T10:22:11.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mPWES5gLT/XY6IVBNTtN6kdY8T4=', 'Content-Length': '0'} 2024-02-01T10:22:11.880 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'YE736OwOOXbZJHJe7d8YvcNV1LxwcQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e8f9d6eeb137ed8c-0065bb70d3-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:11.880 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-02-01T10:22:11.880 DEBUG:boto:path=/vclfcn-17/obj-test-zone3 2024-02-01T10:22:11.881 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3 2024-02-01T10:22:11.881 DEBUG:boto:path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:11.881 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:11.881 DEBUG:boto:Method: GET 2024-02-01T10:22:11.881 DEBUG:boto:Path: /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:11.881 DEBUG:boto:Data: 2024-02-01T10:22:11.881 DEBUG:boto:Headers: {} 2024-02-01T10:22:11.881 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:11.881 DEBUG:boto:Port: 8000 2024-02-01T10:22:11.881 DEBUG:boto:Params: {} 2024-02-01T10:22:11.882 DEBUG:boto:Token: None 2024-02-01T10:22:11.882 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:11 GMT /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:11.882 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BDCcdr4qxWKrQ+xkWg7wpKTtBO4= 2024-02-01T10:22:11.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BDCcdr4qxWKrQ+xkWg7wpKTtBO4=', 'Content-Length': '0'} 2024-02-01T10:22:11.885 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'YE736OwOOXbZJHJe7d8YvcNV1LxwcQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000079d3a8b206839ea8-0065bb70d3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:11.885 DEBUG:boto:path=/vclfcn-17/obj-test-zone3 2024-02-01T10:22:11.885 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3 2024-02-01T10:22:11.886 DEBUG:boto:path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:11.886 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:11.886 DEBUG:boto:Method: GET 2024-02-01T10:22:11.886 DEBUG:boto:Path: /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:11.886 DEBUG:boto:Data: 2024-02-01T10:22:11.886 DEBUG:boto:Headers: {} 2024-02-01T10:22:11.886 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:11.886 DEBUG:boto:Port: 8000 2024-02-01T10:22:11.886 DEBUG:boto:Params: {} 2024-02-01T10:22:11.886 DEBUG:boto:Token: None 2024-02-01T10:22:11.887 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:11 GMT /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:11.887 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BDCcdr4qxWKrQ+xkWg7wpKTtBO4= 2024-02-01T10:22:11.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BDCcdr4qxWKrQ+xkWg7wpKTtBO4=', 'Content-Length': '0'} 2024-02-01T10:22:11.890 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'YE736OwOOXbZJHJe7d8YvcNV1LxwcQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a7150e5f12737d08-0065bb70d3-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:11.932 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:11.932 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:11.933 DEBUG:boto:path=/vclfcn-17/?prefix=obj-test-zone3 2024-02-01T10:22:11.933 DEBUG:boto:auth_path=/vclfcn-17/?prefix=obj-test-zone3 2024-02-01T10:22:11.933 DEBUG:boto:Method: GET 2024-02-01T10:22:11.933 DEBUG:boto:Path: /vclfcn-17/?prefix=obj-test-zone3 2024-02-01T10:22:11.933 DEBUG:boto:Data: 2024-02-01T10:22:11.933 DEBUG:boto:Headers: {} 2024-02-01T10:22:11.933 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:11.933 DEBUG:boto:Port: 8000 2024-02-01T10:22:11.933 DEBUG:boto:Params: {} 2024-02-01T10:22:11.933 DEBUG:boto:Token: None 2024-02-01T10:22:11.934 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:11 GMT /vclfcn-17/ 2024-02-01T10:22:11.934 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:hkxaNl2qfovb0VWctq15J2stLQI= 2024-02-01T10:22:11.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:hkxaNl2qfovb0VWctq15J2stLQI=', 'Content-Length': '0'} 2024-02-01T10:22:11.938 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1d74bbbb4002e04-0065bb70d3-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:11.938 DEBUG:boto:b'vclfcn-17obj-test-zone31000falseobj-test-zone32024-02-01T10:20:50.780Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:11.939 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:11.939 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:11.939 DEBUG:boto:path=/vclfcn-17/?prefix=obj-test-zone3 2024-02-01T10:22:11.939 DEBUG:boto:auth_path=/vclfcn-17/?prefix=obj-test-zone3 2024-02-01T10:22:11.939 DEBUG:boto:Method: GET 2024-02-01T10:22:11.939 DEBUG:boto:Path: /vclfcn-17/?prefix=obj-test-zone3 2024-02-01T10:22:11.939 DEBUG:boto:Data: 2024-02-01T10:22:11.939 DEBUG:boto:Headers: {} 2024-02-01T10:22:11.940 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:11.940 DEBUG:boto:Port: 8000 2024-02-01T10:22:11.940 DEBUG:boto:Params: {} 2024-02-01T10:22:11.940 DEBUG:boto:Token: None 2024-02-01T10:22:11.940 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:11 GMT /vclfcn-17/ 2024-02-01T10:22:11.940 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:hkxaNl2qfovb0VWctq15J2stLQI= 2024-02-01T10:22:11.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:hkxaNl2qfovb0VWctq15J2stLQI=', 'Content-Length': '0'} 2024-02-01T10:22:11.945 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6bf7ae1af3da1e9-0065bb70d3-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:11.945 DEBUG:boto:b'vclfcn-17obj-test-zone31000falseobj-test-zone32024-02-01T10:20:50.780Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:11.945 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-17 zones={test-zone2, test-zone1} 2024-02-01T10:22:11.945 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-17 zones={test-zone3, test-zone1} 2024-02-01T10:22:11.946 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:11.946 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:11.946 DEBUG:boto:Method: HEAD 2024-02-01T10:22:11.946 DEBUG:boto:Path: /vclfcn-17/ 2024-02-01T10:22:11.946 DEBUG:boto:Data: 2024-02-01T10:22:11.946 DEBUG:boto:Headers: {} 2024-02-01T10:22:11.946 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:11.946 DEBUG:boto:Port: 8001 2024-02-01T10:22:11.946 DEBUG:boto:Params: {} 2024-02-01T10:22:11.946 DEBUG:boto:Token: None 2024-02-01T10:22:11.947 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:11 GMT /vclfcn-17/ 2024-02-01T10:22:11.947 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zYCrycCn5yJUhWNStkzLAUOHI4k= 2024-02-01T10:22:11.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zYCrycCn5yJUhWNStkzLAUOHI4k=', 'Content-Length': '0'} 2024-02-01T10:22:11.951 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cf871a3b9f7d6d0e-0065bb70d3-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:22:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:11.951 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:11.951 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:11.951 DEBUG:boto:Method: HEAD 2024-02-01T10:22:11.951 DEBUG:boto:Path: /vclfcn-17/ 2024-02-01T10:22:11.951 DEBUG:boto:Data: 2024-02-01T10:22:11.951 DEBUG:boto:Headers: {} 2024-02-01T10:22:11.951 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:11.952 DEBUG:boto:Port: 8000 2024-02-01T10:22:11.952 DEBUG:boto:Params: {} 2024-02-01T10:22:11.952 DEBUG:boto:Token: None 2024-02-01T10:22:11.952 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:11 GMT /vclfcn-17/ 2024-02-01T10:22:11.952 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zYCrycCn5yJUhWNStkzLAUOHI4k= 2024-02-01T10:22:11.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zYCrycCn5yJUhWNStkzLAUOHI4k=', 'Content-Length': '0'} 2024-02-01T10:22:11.956 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003f2421203d65f8f5-0065bb70d3-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:22:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:11.956 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:22:11.956 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:11.957 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:11.957 DEBUG:boto:path=/vclfcn-17/?versions&max-keys=999 2024-02-01T10:22:11.957 DEBUG:boto:auth_path=/vclfcn-17/?versions&max-keys=999 2024-02-01T10:22:11.957 DEBUG:boto:Method: GET 2024-02-01T10:22:11.957 DEBUG:boto:Path: /vclfcn-17/?versions&max-keys=999 2024-02-01T10:22:11.957 DEBUG:boto:Data: 2024-02-01T10:22:11.957 DEBUG:boto:Headers: {} 2024-02-01T10:22:11.957 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:11.957 DEBUG:boto:Port: 8001 2024-02-01T10:22:11.957 DEBUG:boto:Params: {} 2024-02-01T10:22:11.958 DEBUG:boto:Token: None 2024-02-01T10:22:11.958 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:11 GMT /vclfcn-17/?versions 2024-02-01T10:22:11.958 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Lwu6LYfB2jYbvm20s2DkV45uafY= 2024-02-01T10:22:11.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Lwu6LYfB2jYbvm20s2DkV45uafY=', 'Content-Length': '0'} 2024-02-01T10:22:11.962 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a720edc7a9348ac9-0065bb70d3-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:11.963 DEBUG:boto:b'vclfcn-17999falsedummyhQXZXSD45t2CSRVMJOLMBXih54iMPmJtrue2024-02-01T10:20:14.297Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1xMxAGR4A-xEILepnzFPRgrWNTOLvitztrue2024-02-01T10:20:50.473Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1true2024-02-01T10:20:50.624Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3YE736OwOOXbZJHJe7d8YvcNV1LxwcQbtrue2024-02-01T10:20:50.780Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:11.963 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-02-01T10:22:11.963 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-02-01T10:22:11.963 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-02-01T10:22:11.963 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-02-01T10:22:11.964 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:22:11.964 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:11.964 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:11.964 DEBUG:boto:path=/vclfcn-17/?versions&max-keys=999 2024-02-01T10:22:11.964 DEBUG:boto:auth_path=/vclfcn-17/?versions&max-keys=999 2024-02-01T10:22:11.964 DEBUG:boto:Method: GET 2024-02-01T10:22:11.964 DEBUG:boto:Path: /vclfcn-17/?versions&max-keys=999 2024-02-01T10:22:11.964 DEBUG:boto:Data: 2024-02-01T10:22:11.964 DEBUG:boto:Headers: {} 2024-02-01T10:22:11.964 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:11.964 DEBUG:boto:Port: 8000 2024-02-01T10:22:11.965 DEBUG:boto:Params: {} 2024-02-01T10:22:11.965 DEBUG:boto:Token: None 2024-02-01T10:22:11.965 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:11 GMT /vclfcn-17/?versions 2024-02-01T10:22:11.965 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Lwu6LYfB2jYbvm20s2DkV45uafY= 2024-02-01T10:22:11.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Lwu6LYfB2jYbvm20s2DkV45uafY=', 'Content-Length': '0'} 2024-02-01T10:22:11.969 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba38392e19861f25-0065bb70d3-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:11.970 DEBUG:boto:b'vclfcn-17999falsedummyhQXZXSD45t2CSRVMJOLMBXih54iMPmJtrue2024-02-01T10:20:14.297Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1xMxAGR4A-xEILepnzFPRgrWNTOLvitztrue2024-02-01T10:20:50.473Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1true2024-02-01T10:20:50.624Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3YE736OwOOXbZJHJe7d8YvcNV1LxwcQbtrue2024-02-01T10:20:50.780Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:11.970 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-02-01T10:22:11.970 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-02-01T10:22:11.970 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-02-01T10:22:11.970 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-02-01T10:22:11.970 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:11.971 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:11.971 DEBUG:boto:path=/vclfcn-17/?versions&max-keys=999 2024-02-01T10:22:11.971 DEBUG:boto:auth_path=/vclfcn-17/?versions&max-keys=999 2024-02-01T10:22:11.971 DEBUG:boto:Method: GET 2024-02-01T10:22:11.971 DEBUG:boto:Path: /vclfcn-17/?versions&max-keys=999 2024-02-01T10:22:11.971 DEBUG:boto:Data: 2024-02-01T10:22:11.971 DEBUG:boto:Headers: {} 2024-02-01T10:22:11.971 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:11.971 DEBUG:boto:Port: 8001 2024-02-01T10:22:11.971 DEBUG:boto:Params: {} 2024-02-01T10:22:11.971 DEBUG:boto:Token: None 2024-02-01T10:22:11.972 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:11 GMT /vclfcn-17/?versions 2024-02-01T10:22:11.972 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Lwu6LYfB2jYbvm20s2DkV45uafY= 2024-02-01T10:22:11.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Lwu6LYfB2jYbvm20s2DkV45uafY=', 'Content-Length': '0'} 2024-02-01T10:22:11.976 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000074b0c0172d654d3b-0065bb70d3-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:11.977 DEBUG:boto:b'vclfcn-17999falsedummyhQXZXSD45t2CSRVMJOLMBXih54iMPmJtrue2024-02-01T10:20:14.297Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1xMxAGR4A-xEILepnzFPRgrWNTOLvitztrue2024-02-01T10:20:50.473Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1true2024-02-01T10:20:50.624Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3YE736OwOOXbZJHJe7d8YvcNV1LxwcQbtrue2024-02-01T10:20:50.780Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:11.977 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:11.977 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:11.977 DEBUG:boto:path=/vclfcn-17/?versions&max-keys=999 2024-02-01T10:22:11.977 DEBUG:boto:auth_path=/vclfcn-17/?versions&max-keys=999 2024-02-01T10:22:11.977 DEBUG:boto:Method: GET 2024-02-01T10:22:11.978 DEBUG:boto:Path: /vclfcn-17/?versions&max-keys=999 2024-02-01T10:22:11.978 DEBUG:boto:Data: 2024-02-01T10:22:11.978 DEBUG:boto:Headers: {} 2024-02-01T10:22:11.978 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:11.978 DEBUG:boto:Port: 8000 2024-02-01T10:22:11.978 DEBUG:boto:Params: {} 2024-02-01T10:22:11.978 DEBUG:boto:Token: None 2024-02-01T10:22:11.978 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:11 GMT /vclfcn-17/?versions 2024-02-01T10:22:11.978 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Lwu6LYfB2jYbvm20s2DkV45uafY= 2024-02-01T10:22:11.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Lwu6LYfB2jYbvm20s2DkV45uafY=', 'Content-Length': '0'} 2024-02-01T10:22:11.983 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007cf1fb30f17e7791-0065bb70d3-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:12.024 DEBUG:boto:b'vclfcn-17999falsedummyhQXZXSD45t2CSRVMJOLMBXih54iMPmJtrue2024-02-01T10:20:14.297Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1xMxAGR4A-xEILepnzFPRgrWNTOLvitztrue2024-02-01T10:20:50.473Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1true2024-02-01T10:20:50.624Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3YE736OwOOXbZJHJe7d8YvcNV1LxwcQbtrue2024-02-01T10:20:50.780Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:12.025 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-02-01T10:22:12.025 DEBUG:boto:path=/vclfcn-17/dummy 2024-02-01T10:22:12.025 DEBUG:boto:auth_path=/vclfcn-17/dummy 2024-02-01T10:22:12.025 DEBUG:boto:path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:12.025 DEBUG:boto:auth_path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:12.025 DEBUG:boto:Method: GET 2024-02-01T10:22:12.025 DEBUG:boto:Path: /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:12.025 DEBUG:boto:Data: 2024-02-01T10:22:12.025 DEBUG:boto:Headers: {} 2024-02-01T10:22:12.026 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:12.026 DEBUG:boto:Port: 8001 2024-02-01T10:22:12.026 DEBUG:boto:Params: {} 2024-02-01T10:22:12.026 DEBUG:boto:Token: None 2024-02-01T10:22:12.026 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:12 GMT /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:12.026 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Loy4+vXg04cg/0otzvelsmuTJ3M= 2024-02-01T10:22:12.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Loy4+vXg04cg/0otzvelsmuTJ3M=', 'Content-Length': '0'} 2024-02-01T10:22:12.029 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:14 GMT'), ('x-amz-version-id', 'hQXZXSD45t2CSRVMJOLMBXih54iMPmJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e5e7ef53248664f3-0065bb70d4-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:12.029 DEBUG:boto:path=/vclfcn-17/dummy 2024-02-01T10:22:12.029 DEBUG:boto:auth_path=/vclfcn-17/dummy 2024-02-01T10:22:12.030 DEBUG:boto:path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:12.030 DEBUG:boto:auth_path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:12.030 DEBUG:boto:Method: GET 2024-02-01T10:22:12.030 DEBUG:boto:Path: /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:12.030 DEBUG:boto:Data: 2024-02-01T10:22:12.030 DEBUG:boto:Headers: {} 2024-02-01T10:22:12.030 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:12.030 DEBUG:boto:Port: 8000 2024-02-01T10:22:12.030 DEBUG:boto:Params: {} 2024-02-01T10:22:12.030 DEBUG:boto:Token: None 2024-02-01T10:22:12.030 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:12 GMT /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:12.031 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Loy4+vXg04cg/0otzvelsmuTJ3M= 2024-02-01T10:22:12.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Loy4+vXg04cg/0otzvelsmuTJ3M=', 'Content-Length': '0'} 2024-02-01T10:22:12.034 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:14 GMT'), ('x-amz-version-id', 'hQXZXSD45t2CSRVMJOLMBXih54iMPmJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004db8305737fbd476-0065bb70d4-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:12.034 DEBUG:boto:path=/vclfcn-17/dummy 2024-02-01T10:22:12.034 DEBUG:boto:auth_path=/vclfcn-17/dummy 2024-02-01T10:22:12.034 DEBUG:boto:path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:12.034 DEBUG:boto:auth_path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:12.034 DEBUG:boto:Method: HEAD 2024-02-01T10:22:12.034 DEBUG:boto:Path: /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:12.035 DEBUG:boto:Data: 2024-02-01T10:22:12.035 DEBUG:boto:Headers: {} 2024-02-01T10:22:12.035 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:12.035 DEBUG:boto:Port: 8001 2024-02-01T10:22:12.035 DEBUG:boto:Params: {} 2024-02-01T10:22:12.035 DEBUG:boto:Token: None 2024-02-01T10:22:12.035 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:12 GMT /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:12.035 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:a8srnaCydZVz5ItyPv5fA7Dt9gQ= 2024-02-01T10:22:12.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:a8srnaCydZVz5ItyPv5fA7Dt9gQ=', 'Content-Length': '0'} 2024-02-01T10:22:12.038 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:14 GMT'), ('x-amz-version-id', 'hQXZXSD45t2CSRVMJOLMBXih54iMPmJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000027223376d0392f30-0065bb70d4-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:12.038 DEBUG:boto:path=/vclfcn-17/dummy 2024-02-01T10:22:12.038 DEBUG:boto:auth_path=/vclfcn-17/dummy 2024-02-01T10:22:12.038 DEBUG:boto:path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:12.038 DEBUG:boto:auth_path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:12.039 DEBUG:boto:Method: HEAD 2024-02-01T10:22:12.039 DEBUG:boto:Path: /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:12.039 DEBUG:boto:Data: 2024-02-01T10:22:12.039 DEBUG:boto:Headers: {} 2024-02-01T10:22:12.039 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:12.039 DEBUG:boto:Port: 8000 2024-02-01T10:22:12.039 DEBUG:boto:Params: {} 2024-02-01T10:22:12.039 DEBUG:boto:Token: None 2024-02-01T10:22:12.039 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:12 GMT /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:12.039 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:a8srnaCydZVz5ItyPv5fA7Dt9gQ= 2024-02-01T10:22:12.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:a8srnaCydZVz5ItyPv5fA7Dt9gQ=', 'Content-Length': '0'} 2024-02-01T10:22:12.042 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:14 GMT'), ('x-amz-version-id', 'hQXZXSD45t2CSRVMJOLMBXih54iMPmJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bbe771473fe12319-0065bb70d4-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:12.043 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-02-01T10:22:12.043 DEBUG:boto:path=/vclfcn-17/dummy 2024-02-01T10:22:12.043 DEBUG:boto:auth_path=/vclfcn-17/dummy 2024-02-01T10:22:12.043 DEBUG:boto:path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:12.043 DEBUG:boto:auth_path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:12.043 DEBUG:boto:Method: GET 2024-02-01T10:22:12.043 DEBUG:boto:Path: /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:12.043 DEBUG:boto:Data: 2024-02-01T10:22:12.043 DEBUG:boto:Headers: {} 2024-02-01T10:22:12.043 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:12.043 DEBUG:boto:Port: 8001 2024-02-01T10:22:12.044 DEBUG:boto:Params: {} 2024-02-01T10:22:12.044 DEBUG:boto:Token: None 2024-02-01T10:22:12.044 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:12 GMT /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:12.044 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Loy4+vXg04cg/0otzvelsmuTJ3M= 2024-02-01T10:22:12.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Loy4+vXg04cg/0otzvelsmuTJ3M=', 'Content-Length': '0'} 2024-02-01T10:22:12.047 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:14 GMT'), ('x-amz-version-id', 'hQXZXSD45t2CSRVMJOLMBXih54iMPmJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ca7514ba6f76cd91-0065bb70d4-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:12.047 DEBUG:boto:path=/vclfcn-17/dummy 2024-02-01T10:22:12.047 DEBUG:boto:auth_path=/vclfcn-17/dummy 2024-02-01T10:22:12.047 DEBUG:boto:path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:12.048 DEBUG:boto:auth_path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:12.048 DEBUG:boto:Method: GET 2024-02-01T10:22:12.048 DEBUG:boto:Path: /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:12.048 DEBUG:boto:Data: 2024-02-01T10:22:12.048 DEBUG:boto:Headers: {} 2024-02-01T10:22:12.048 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:12.048 DEBUG:boto:Port: 8000 2024-02-01T10:22:12.048 DEBUG:boto:Params: {} 2024-02-01T10:22:12.048 DEBUG:boto:Token: None 2024-02-01T10:22:12.048 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:12 GMT /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:12.049 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Loy4+vXg04cg/0otzvelsmuTJ3M= 2024-02-01T10:22:12.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Loy4+vXg04cg/0otzvelsmuTJ3M=', 'Content-Length': '0'} 2024-02-01T10:22:12.051 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:14 GMT'), ('x-amz-version-id', 'hQXZXSD45t2CSRVMJOLMBXih54iMPmJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000028654429ff27dedf-0065bb70d4-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:12.052 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:12.052 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:12.052 DEBUG:boto:path=/vclfcn-17/?prefix=dummy 2024-02-01T10:22:12.052 DEBUG:boto:auth_path=/vclfcn-17/?prefix=dummy 2024-02-01T10:22:12.052 DEBUG:boto:Method: GET 2024-02-01T10:22:12.052 DEBUG:boto:Path: /vclfcn-17/?prefix=dummy 2024-02-01T10:22:12.052 DEBUG:boto:Data: 2024-02-01T10:22:12.052 DEBUG:boto:Headers: {} 2024-02-01T10:22:12.052 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:12.052 DEBUG:boto:Port: 8001 2024-02-01T10:22:12.052 DEBUG:boto:Params: {} 2024-02-01T10:22:12.053 DEBUG:boto:Token: None 2024-02-01T10:22:12.053 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:12 GMT /vclfcn-17/ 2024-02-01T10:22:12.053 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:f1HPd+2xBd4FvZlv1vmHFEHulsU= 2024-02-01T10:22:12.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:f1HPd+2xBd4FvZlv1vmHFEHulsU=', 'Content-Length': '0'} 2024-02-01T10:22:12.057 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000716f25692a4747f2-0065bb70d4-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:12.057 DEBUG:boto:b'vclfcn-17dummy1000falsedummy2024-02-01T10:20:14.297Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:12.058 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:12.058 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:12.058 DEBUG:boto:path=/vclfcn-17/?prefix=dummy 2024-02-01T10:22:12.058 DEBUG:boto:auth_path=/vclfcn-17/?prefix=dummy 2024-02-01T10:22:12.058 DEBUG:boto:Method: GET 2024-02-01T10:22:12.058 DEBUG:boto:Path: /vclfcn-17/?prefix=dummy 2024-02-01T10:22:12.058 DEBUG:boto:Data: 2024-02-01T10:22:12.058 DEBUG:boto:Headers: {} 2024-02-01T10:22:12.059 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:12.059 DEBUG:boto:Port: 8000 2024-02-01T10:22:12.059 DEBUG:boto:Params: {} 2024-02-01T10:22:12.059 DEBUG:boto:Token: None 2024-02-01T10:22:12.059 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:12 GMT /vclfcn-17/ 2024-02-01T10:22:12.059 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:f1HPd+2xBd4FvZlv1vmHFEHulsU= 2024-02-01T10:22:12.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:f1HPd+2xBd4FvZlv1vmHFEHulsU=', 'Content-Length': '0'} 2024-02-01T10:22:12.064 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000685066244b83a5a5-0065bb70d4-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:12.064 DEBUG:boto:b'vclfcn-17dummy1000falsedummy2024-02-01T10:20:14.297Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:12.065 DEBUG:boto:path=/vclfcn-17/dummy 2024-02-01T10:22:12.065 DEBUG:boto:auth_path=/vclfcn-17/dummy 2024-02-01T10:22:12.065 DEBUG:boto:Method: HEAD 2024-02-01T10:22:12.065 DEBUG:boto:Path: /vclfcn-17/dummy 2024-02-01T10:22:12.065 DEBUG:boto:Data: 2024-02-01T10:22:12.065 DEBUG:boto:Headers: {} 2024-02-01T10:22:12.065 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:12.065 DEBUG:boto:Port: 8001 2024-02-01T10:22:12.065 DEBUG:boto:Params: {} 2024-02-01T10:22:12.065 DEBUG:boto:Token: None 2024-02-01T10:22:12.066 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:12 GMT /vclfcn-17/dummy 2024-02-01T10:22:12.066 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:eU0INpgqGmTPB8IntMT9KDA5CeQ= 2024-02-01T10:22:12.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:eU0INpgqGmTPB8IntMT9KDA5CeQ=', 'Content-Length': '0'} 2024-02-01T10:22:12.069 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:14 GMT'), ('x-amz-version-id', 'hQXZXSD45t2CSRVMJOLMBXih54iMPmJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000143da72e665f755a-0065bb70d4-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:12.069 DEBUG:boto:path=/vclfcn-17/dummy 2024-02-01T10:22:12.070 DEBUG:boto:auth_path=/vclfcn-17/dummy 2024-02-01T10:22:12.070 DEBUG:boto:Method: HEAD 2024-02-01T10:22:12.070 DEBUG:boto:Path: /vclfcn-17/dummy 2024-02-01T10:22:12.070 DEBUG:boto:Data: 2024-02-01T10:22:12.070 DEBUG:boto:Headers: {} 2024-02-01T10:22:12.070 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:12.070 DEBUG:boto:Port: 8000 2024-02-01T10:22:12.070 DEBUG:boto:Params: {} 2024-02-01T10:22:12.070 DEBUG:boto:Token: None 2024-02-01T10:22:12.070 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:12 GMT /vclfcn-17/dummy 2024-02-01T10:22:12.071 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:eU0INpgqGmTPB8IntMT9KDA5CeQ= 2024-02-01T10:22:12.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:eU0INpgqGmTPB8IntMT9KDA5CeQ=', 'Content-Length': '0'} 2024-02-01T10:22:12.075 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:14 GMT'), ('x-amz-version-id', 'hQXZXSD45t2CSRVMJOLMBXih54iMPmJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008bd51bbaf8cd1007-0065bb70d4-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:12.075 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-02-01T10:22:12.075 DEBUG:boto:path=/vclfcn-17/dummy 2024-02-01T10:22:12.075 DEBUG:boto:auth_path=/vclfcn-17/dummy 2024-02-01T10:22:12.075 DEBUG:boto:path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:12.075 DEBUG:boto:auth_path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:12.076 DEBUG:boto:Method: GET 2024-02-01T10:22:12.076 DEBUG:boto:Path: /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:12.076 DEBUG:boto:Data: 2024-02-01T10:22:12.076 DEBUG:boto:Headers: {} 2024-02-01T10:22:12.076 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:12.076 DEBUG:boto:Port: 8001 2024-02-01T10:22:12.076 DEBUG:boto:Params: {} 2024-02-01T10:22:12.076 DEBUG:boto:Token: None 2024-02-01T10:22:12.076 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:12 GMT /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:12.076 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Loy4+vXg04cg/0otzvelsmuTJ3M= 2024-02-01T10:22:12.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Loy4+vXg04cg/0otzvelsmuTJ3M=', 'Content-Length': '0'} 2024-02-01T10:22:12.080 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:14 GMT'), ('x-amz-version-id', 'hQXZXSD45t2CSRVMJOLMBXih54iMPmJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000db3bb4f66cdfe41e-0065bb70d4-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:12.080 DEBUG:boto:path=/vclfcn-17/dummy 2024-02-01T10:22:12.080 DEBUG:boto:auth_path=/vclfcn-17/dummy 2024-02-01T10:22:12.080 DEBUG:boto:path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:12.080 DEBUG:boto:auth_path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:12.080 DEBUG:boto:Method: GET 2024-02-01T10:22:12.080 DEBUG:boto:Path: /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:12.081 DEBUG:boto:Data: 2024-02-01T10:22:12.081 DEBUG:boto:Headers: {} 2024-02-01T10:22:12.081 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:12.081 DEBUG:boto:Port: 8000 2024-02-01T10:22:12.081 DEBUG:boto:Params: {} 2024-02-01T10:22:12.081 DEBUG:boto:Token: None 2024-02-01T10:22:12.081 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:12 GMT /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:12.081 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Loy4+vXg04cg/0otzvelsmuTJ3M= 2024-02-01T10:22:12.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Loy4+vXg04cg/0otzvelsmuTJ3M=', 'Content-Length': '0'} 2024-02-01T10:22:12.084 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:14 GMT'), ('x-amz-version-id', 'hQXZXSD45t2CSRVMJOLMBXih54iMPmJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b64afe5e91daaafd-0065bb70d4-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:12.084 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:12.084 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:12.085 DEBUG:boto:path=/vclfcn-17/?prefix=dummy 2024-02-01T10:22:12.085 DEBUG:boto:auth_path=/vclfcn-17/?prefix=dummy 2024-02-01T10:22:12.085 DEBUG:boto:Method: GET 2024-02-01T10:22:12.085 DEBUG:boto:Path: /vclfcn-17/?prefix=dummy 2024-02-01T10:22:12.085 DEBUG:boto:Data: 2024-02-01T10:22:12.085 DEBUG:boto:Headers: {} 2024-02-01T10:22:12.085 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:12.085 DEBUG:boto:Port: 8001 2024-02-01T10:22:12.085 DEBUG:boto:Params: {} 2024-02-01T10:22:12.085 DEBUG:boto:Token: None 2024-02-01T10:22:12.086 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:12 GMT /vclfcn-17/ 2024-02-01T10:22:12.086 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:f1HPd+2xBd4FvZlv1vmHFEHulsU= 2024-02-01T10:22:12.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:f1HPd+2xBd4FvZlv1vmHFEHulsU=', 'Content-Length': '0'} 2024-02-01T10:22:12.090 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bef2e45ffd15f4cd-0065bb70d4-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:12.132 DEBUG:boto:b'vclfcn-17dummy1000falsedummy2024-02-01T10:20:14.297Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:12.133 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:12.133 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:12.133 DEBUG:boto:path=/vclfcn-17/?prefix=dummy 2024-02-01T10:22:12.133 DEBUG:boto:auth_path=/vclfcn-17/?prefix=dummy 2024-02-01T10:22:12.133 DEBUG:boto:Method: GET 2024-02-01T10:22:12.133 DEBUG:boto:Path: /vclfcn-17/?prefix=dummy 2024-02-01T10:22:12.133 DEBUG:boto:Data: 2024-02-01T10:22:12.133 DEBUG:boto:Headers: {} 2024-02-01T10:22:12.133 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:12.133 DEBUG:boto:Port: 8000 2024-02-01T10:22:12.134 DEBUG:boto:Params: {} 2024-02-01T10:22:12.134 DEBUG:boto:Token: None 2024-02-01T10:22:12.134 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:12 GMT /vclfcn-17/ 2024-02-01T10:22:12.134 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:f1HPd+2xBd4FvZlv1vmHFEHulsU= 2024-02-01T10:22:12.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:f1HPd+2xBd4FvZlv1vmHFEHulsU=', 'Content-Length': '0'} 2024-02-01T10:22:12.139 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1b123806261760a-0065bb70d4-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:12.139 DEBUG:boto:b'vclfcn-17dummy1000falsedummy2024-02-01T10:20:14.297Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:12.139 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-02-01T10:22:12.140 DEBUG:boto:path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:12.140 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:12.140 DEBUG:boto:path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:12.140 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:12.140 DEBUG:boto:Method: GET 2024-02-01T10:22:12.140 DEBUG:boto:Path: /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:12.140 DEBUG:boto:Data: 2024-02-01T10:22:12.140 DEBUG:boto:Headers: {} 2024-02-01T10:22:12.140 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:12.140 DEBUG:boto:Port: 8001 2024-02-01T10:22:12.140 DEBUG:boto:Params: {} 2024-02-01T10:22:12.141 DEBUG:boto:Token: None 2024-02-01T10:22:12.141 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:12 GMT /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:12.141 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+b2ig3m4CagEOq5/KuqSwfMXYvA= 2024-02-01T10:22:12.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+b2ig3m4CagEOq5/KuqSwfMXYvA=', 'Content-Length': '0'} 2024-02-01T10:22:12.144 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'xMxAGR4A-xEILepnzFPRgrWNTOLvitz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005d80b9da6651ef2a-0065bb70d4-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:12.184 DEBUG:boto:path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:12.184 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:12.184 DEBUG:boto:path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:12.184 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:12.185 DEBUG:boto:Method: GET 2024-02-01T10:22:12.185 DEBUG:boto:Path: /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:12.185 DEBUG:boto:Data: 2024-02-01T10:22:12.185 DEBUG:boto:Headers: {} 2024-02-01T10:22:12.185 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:12.185 DEBUG:boto:Port: 8000 2024-02-01T10:22:12.185 DEBUG:boto:Params: {} 2024-02-01T10:22:12.185 DEBUG:boto:Token: None 2024-02-01T10:22:12.185 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:12 GMT /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:12.186 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+b2ig3m4CagEOq5/KuqSwfMXYvA= 2024-02-01T10:22:12.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+b2ig3m4CagEOq5/KuqSwfMXYvA=', 'Content-Length': '0'} 2024-02-01T10:22:12.188 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'xMxAGR4A-xEILepnzFPRgrWNTOLvitz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ef13001dd4912a09-0065bb70d4-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:12.189 DEBUG:boto:path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:12.189 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:12.189 DEBUG:boto:path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:12.189 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:12.189 DEBUG:boto:Method: HEAD 2024-02-01T10:22:12.189 DEBUG:boto:Path: /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:12.189 DEBUG:boto:Data: 2024-02-01T10:22:12.189 DEBUG:boto:Headers: {} 2024-02-01T10:22:12.190 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:12.190 DEBUG:boto:Port: 8001 2024-02-01T10:22:12.190 DEBUG:boto:Params: {} 2024-02-01T10:22:12.190 DEBUG:boto:Token: None 2024-02-01T10:22:12.190 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:12 GMT /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:12.190 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:k7G5sjUUu7aBMkKEaBSoalmMG0k= 2024-02-01T10:22:12.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:k7G5sjUUu7aBMkKEaBSoalmMG0k=', 'Content-Length': '0'} 2024-02-01T10:22:12.193 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'xMxAGR4A-xEILepnzFPRgrWNTOLvitz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000293b215c2d2a662e-0065bb70d4-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:12.193 DEBUG:boto:path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:12.193 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:12.193 DEBUG:boto:path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:12.194 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:12.194 DEBUG:boto:Method: HEAD 2024-02-01T10:22:12.194 DEBUG:boto:Path: /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:12.194 DEBUG:boto:Data: 2024-02-01T10:22:12.194 DEBUG:boto:Headers: {} 2024-02-01T10:22:12.194 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:12.194 DEBUG:boto:Port: 8000 2024-02-01T10:22:12.194 DEBUG:boto:Params: {} 2024-02-01T10:22:12.194 DEBUG:boto:Token: None 2024-02-01T10:22:12.194 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:12 GMT /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:12.195 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:k7G5sjUUu7aBMkKEaBSoalmMG0k= 2024-02-01T10:22:12.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:k7G5sjUUu7aBMkKEaBSoalmMG0k=', 'Content-Length': '0'} 2024-02-01T10:22:12.197 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'xMxAGR4A-xEILepnzFPRgrWNTOLvitz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d824dba75711a31d-0065bb70d4-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:12.197 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-02-01T10:22:12.198 DEBUG:boto:path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:12.198 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:12.198 DEBUG:boto:path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:12.198 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:12.198 DEBUG:boto:Method: GET 2024-02-01T10:22:12.198 DEBUG:boto:Path: /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:12.198 DEBUG:boto:Data: 2024-02-01T10:22:12.198 DEBUG:boto:Headers: {} 2024-02-01T10:22:12.198 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:12.198 DEBUG:boto:Port: 8001 2024-02-01T10:22:12.198 DEBUG:boto:Params: {} 2024-02-01T10:22:12.198 DEBUG:boto:Token: None 2024-02-01T10:22:12.199 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:12 GMT /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:12.199 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+b2ig3m4CagEOq5/KuqSwfMXYvA= 2024-02-01T10:22:12.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+b2ig3m4CagEOq5/KuqSwfMXYvA=', 'Content-Length': '0'} 2024-02-01T10:22:12.201 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'xMxAGR4A-xEILepnzFPRgrWNTOLvitz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007738e2fe929b9b99-0065bb70d4-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:12.248 DEBUG:boto:path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:12.248 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:12.249 DEBUG:boto:path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:12.249 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:12.249 DEBUG:boto:Method: GET 2024-02-01T10:22:12.249 DEBUG:boto:Path: /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:12.249 DEBUG:boto:Data: 2024-02-01T10:22:12.249 DEBUG:boto:Headers: {} 2024-02-01T10:22:12.249 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:12.249 DEBUG:boto:Port: 8000 2024-02-01T10:22:12.249 DEBUG:boto:Params: {} 2024-02-01T10:22:12.251 DEBUG:boto:Token: None 2024-02-01T10:22:12.251 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:12 GMT /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:12.251 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+b2ig3m4CagEOq5/KuqSwfMXYvA= 2024-02-01T10:22:12.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+b2ig3m4CagEOq5/KuqSwfMXYvA=', 'Content-Length': '0'} 2024-02-01T10:22:12.254 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'xMxAGR4A-xEILepnzFPRgrWNTOLvitz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000000842febcabb9ec0-0065bb70d4-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:12.255 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:12.255 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:12.255 DEBUG:boto:path=/vclfcn-17/?prefix=obj-test-zone1 2024-02-01T10:22:12.255 DEBUG:boto:auth_path=/vclfcn-17/?prefix=obj-test-zone1 2024-02-01T10:22:12.255 DEBUG:boto:Method: GET 2024-02-01T10:22:12.255 DEBUG:boto:Path: /vclfcn-17/?prefix=obj-test-zone1 2024-02-01T10:22:12.255 DEBUG:boto:Data: 2024-02-01T10:22:12.256 DEBUG:boto:Headers: {} 2024-02-01T10:22:12.256 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:12.256 DEBUG:boto:Port: 8001 2024-02-01T10:22:12.256 DEBUG:boto:Params: {} 2024-02-01T10:22:12.256 DEBUG:boto:Token: None 2024-02-01T10:22:12.256 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:12 GMT /vclfcn-17/ 2024-02-01T10:22:12.256 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:f1HPd+2xBd4FvZlv1vmHFEHulsU= 2024-02-01T10:22:12.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:f1HPd+2xBd4FvZlv1vmHFEHulsU=', 'Content-Length': '0'} 2024-02-01T10:22:12.261 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0c8760e4e5ea047-0065bb70d4-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:12.261 DEBUG:boto:b'vclfcn-17obj-test-zone11000falseobj-test-zone12024-02-01T10:20:50.473Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:12.261 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:12.262 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:12.262 DEBUG:boto:path=/vclfcn-17/?prefix=obj-test-zone1 2024-02-01T10:22:12.262 DEBUG:boto:auth_path=/vclfcn-17/?prefix=obj-test-zone1 2024-02-01T10:22:12.262 DEBUG:boto:Method: GET 2024-02-01T10:22:12.262 DEBUG:boto:Path: /vclfcn-17/?prefix=obj-test-zone1 2024-02-01T10:22:12.262 DEBUG:boto:Data: 2024-02-01T10:22:12.262 DEBUG:boto:Headers: {} 2024-02-01T10:22:12.262 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:12.262 DEBUG:boto:Port: 8000 2024-02-01T10:22:12.262 DEBUG:boto:Params: {} 2024-02-01T10:22:12.262 DEBUG:boto:Token: None 2024-02-01T10:22:12.263 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:12 GMT /vclfcn-17/ 2024-02-01T10:22:12.263 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:f1HPd+2xBd4FvZlv1vmHFEHulsU= 2024-02-01T10:22:12.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:f1HPd+2xBd4FvZlv1vmHFEHulsU=', 'Content-Length': '0'} 2024-02-01T10:22:12.268 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000038f0278759a6815e-0065bb70d4-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:12.269 DEBUG:boto:b'vclfcn-17obj-test-zone11000falseobj-test-zone12024-02-01T10:20:50.473Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:12.269 DEBUG:boto:path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:12.269 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:12.269 DEBUG:boto:Method: HEAD 2024-02-01T10:22:12.269 DEBUG:boto:Path: /vclfcn-17/obj-test-zone1 2024-02-01T10:22:12.269 DEBUG:boto:Data: 2024-02-01T10:22:12.269 DEBUG:boto:Headers: {} 2024-02-01T10:22:12.269 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:12.270 DEBUG:boto:Port: 8001 2024-02-01T10:22:12.270 DEBUG:boto:Params: {} 2024-02-01T10:22:12.270 DEBUG:boto:Token: None 2024-02-01T10:22:12.270 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:12 GMT /vclfcn-17/obj-test-zone1 2024-02-01T10:22:12.270 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Q4Qh8qGD9l97Y0cGAHgVduI+eqs= 2024-02-01T10:22:12.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Q4Qh8qGD9l97Y0cGAHgVduI+eqs=', 'Content-Length': '0'} 2024-02-01T10:22:12.274 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'xMxAGR4A-xEILepnzFPRgrWNTOLvitz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006fc7678dd9a353f2-0065bb70d4-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:12.274 DEBUG:boto:path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:12.274 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:12.274 DEBUG:boto:Method: HEAD 2024-02-01T10:22:12.274 DEBUG:boto:Path: /vclfcn-17/obj-test-zone1 2024-02-01T10:22:12.274 DEBUG:boto:Data: 2024-02-01T10:22:12.274 DEBUG:boto:Headers: {} 2024-02-01T10:22:12.274 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:12.274 DEBUG:boto:Port: 8000 2024-02-01T10:22:12.275 DEBUG:boto:Params: {} 2024-02-01T10:22:12.275 DEBUG:boto:Token: None 2024-02-01T10:22:12.275 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:12 GMT /vclfcn-17/obj-test-zone1 2024-02-01T10:22:12.275 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Q4Qh8qGD9l97Y0cGAHgVduI+eqs= 2024-02-01T10:22:12.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Q4Qh8qGD9l97Y0cGAHgVduI+eqs=', 'Content-Length': '0'} 2024-02-01T10:22:12.279 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'xMxAGR4A-xEILepnzFPRgrWNTOLvitz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002da2133da0b2ba61-0065bb70d4-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:12.279 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-02-01T10:22:12.279 DEBUG:boto:path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:12.279 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:12.279 DEBUG:boto:path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:12.280 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:12.280 DEBUG:boto:Method: GET 2024-02-01T10:22:12.280 DEBUG:boto:Path: /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:12.280 DEBUG:boto:Data: 2024-02-01T10:22:12.280 DEBUG:boto:Headers: {} 2024-02-01T10:22:12.280 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:12.280 DEBUG:boto:Port: 8001 2024-02-01T10:22:12.280 DEBUG:boto:Params: {} 2024-02-01T10:22:12.281 DEBUG:boto:Token: None 2024-02-01T10:22:12.281 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:12 GMT /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:12.281 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+b2ig3m4CagEOq5/KuqSwfMXYvA= 2024-02-01T10:22:12.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+b2ig3m4CagEOq5/KuqSwfMXYvA=', 'Content-Length': '0'} 2024-02-01T10:22:12.283 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'xMxAGR4A-xEILepnzFPRgrWNTOLvitz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009e35dc794b665572-0065bb70d4-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:12.324 DEBUG:boto:path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:12.324 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:12.324 DEBUG:boto:path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:12.325 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:12.325 DEBUG:boto:Method: GET 2024-02-01T10:22:12.325 DEBUG:boto:Path: /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:12.325 DEBUG:boto:Data: 2024-02-01T10:22:12.325 DEBUG:boto:Headers: {} 2024-02-01T10:22:12.325 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:12.325 DEBUG:boto:Port: 8000 2024-02-01T10:22:12.325 DEBUG:boto:Params: {} 2024-02-01T10:22:12.325 DEBUG:boto:Token: None 2024-02-01T10:22:12.325 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:12 GMT /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:12.326 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+b2ig3m4CagEOq5/KuqSwfMXYvA= 2024-02-01T10:22:12.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+b2ig3m4CagEOq5/KuqSwfMXYvA=', 'Content-Length': '0'} 2024-02-01T10:22:12.329 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'xMxAGR4A-xEILepnzFPRgrWNTOLvitz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007f8784aeb613b2a4-0065bb70d4-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:12.329 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:12.329 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:12.329 DEBUG:boto:path=/vclfcn-17/?prefix=obj-test-zone1 2024-02-01T10:22:12.329 DEBUG:boto:auth_path=/vclfcn-17/?prefix=obj-test-zone1 2024-02-01T10:22:12.329 DEBUG:boto:Method: GET 2024-02-01T10:22:12.329 DEBUG:boto:Path: /vclfcn-17/?prefix=obj-test-zone1 2024-02-01T10:22:12.330 DEBUG:boto:Data: 2024-02-01T10:22:12.330 DEBUG:boto:Headers: {} 2024-02-01T10:22:12.330 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:12.330 DEBUG:boto:Port: 8001 2024-02-01T10:22:12.330 DEBUG:boto:Params: {} 2024-02-01T10:22:12.330 DEBUG:boto:Token: None 2024-02-01T10:22:12.330 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:12 GMT /vclfcn-17/ 2024-02-01T10:22:12.330 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:f1HPd+2xBd4FvZlv1vmHFEHulsU= 2024-02-01T10:22:12.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:f1HPd+2xBd4FvZlv1vmHFEHulsU=', 'Content-Length': '0'} 2024-02-01T10:22:12.335 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012b0ea4d6f09b7b9-0065bb70d4-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:12.335 DEBUG:boto:b'vclfcn-17obj-test-zone11000falseobj-test-zone12024-02-01T10:20:50.473Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:12.336 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:12.336 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:12.336 DEBUG:boto:path=/vclfcn-17/?prefix=obj-test-zone1 2024-02-01T10:22:12.336 DEBUG:boto:auth_path=/vclfcn-17/?prefix=obj-test-zone1 2024-02-01T10:22:12.336 DEBUG:boto:Method: GET 2024-02-01T10:22:12.336 DEBUG:boto:Path: /vclfcn-17/?prefix=obj-test-zone1 2024-02-01T10:22:12.336 DEBUG:boto:Data: 2024-02-01T10:22:12.336 DEBUG:boto:Headers: {} 2024-02-01T10:22:12.337 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:12.337 DEBUG:boto:Port: 8000 2024-02-01T10:22:12.337 DEBUG:boto:Params: {} 2024-02-01T10:22:12.337 DEBUG:boto:Token: None 2024-02-01T10:22:12.337 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:12 GMT /vclfcn-17/ 2024-02-01T10:22:12.337 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:f1HPd+2xBd4FvZlv1vmHFEHulsU= 2024-02-01T10:22:12.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:f1HPd+2xBd4FvZlv1vmHFEHulsU=', 'Content-Length': '0'} 2024-02-01T10:22:12.341 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000cf88fe3f17b92dc-0065bb70d4-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:12.341 DEBUG:boto:b'vclfcn-17obj-test-zone11000falseobj-test-zone12024-02-01T10:20:50.473Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:12.342 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-02-01T10:22:12.342 DEBUG:boto:path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:12.342 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:12.342 DEBUG:boto:path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:12.342 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:12.342 DEBUG:boto:Method: GET 2024-02-01T10:22:12.342 DEBUG:boto:Path: /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:12.342 DEBUG:boto:Data: 2024-02-01T10:22:12.343 DEBUG:boto:Headers: {} 2024-02-01T10:22:12.343 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:12.343 DEBUG:boto:Port: 8001 2024-02-01T10:22:12.343 DEBUG:boto:Params: {} 2024-02-01T10:22:12.343 DEBUG:boto:Token: None 2024-02-01T10:22:12.343 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:12 GMT /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:12.343 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wBR4Bgd/ZyABXyULOzfYld/Hk9E= 2024-02-01T10:22:12.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wBR4Bgd/ZyABXyULOzfYld/Hk9E=', 'Content-Length': '0'} 2024-02-01T10:22:12.346 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', '-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003709224f9533553a-0065bb70d4-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:12.388 DEBUG:boto:path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:12.388 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:12.388 DEBUG:boto:path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:12.389 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:12.389 DEBUG:boto:Method: GET 2024-02-01T10:22:12.389 DEBUG:boto:Path: /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:12.389 DEBUG:boto:Data: 2024-02-01T10:22:12.389 DEBUG:boto:Headers: {} 2024-02-01T10:22:12.389 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:12.389 DEBUG:boto:Port: 8000 2024-02-01T10:22:12.389 DEBUG:boto:Params: {} 2024-02-01T10:22:12.389 DEBUG:boto:Token: None 2024-02-01T10:22:12.389 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:12 GMT /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:12.390 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wBR4Bgd/ZyABXyULOzfYld/Hk9E= 2024-02-01T10:22:12.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wBR4Bgd/ZyABXyULOzfYld/Hk9E=', 'Content-Length': '0'} 2024-02-01T10:22:12.392 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', '-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000306d151051503d3d-0065bb70d4-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:12.436 DEBUG:boto:path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:12.436 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:12.436 DEBUG:boto:path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:12.437 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:12.437 DEBUG:boto:Method: HEAD 2024-02-01T10:22:12.437 DEBUG:boto:Path: /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:12.437 DEBUG:boto:Data: 2024-02-01T10:22:12.437 DEBUG:boto:Headers: {} 2024-02-01T10:22:12.437 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:12.437 DEBUG:boto:Port: 8001 2024-02-01T10:22:12.437 DEBUG:boto:Params: {} 2024-02-01T10:22:12.437 DEBUG:boto:Token: None 2024-02-01T10:22:12.438 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:12 GMT /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:12.438 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:OBWvZyZ+9+kqEPPvmbDdX8Ya7U8= 2024-02-01T10:22:12.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:OBWvZyZ+9+kqEPPvmbDdX8Ya7U8=', 'Content-Length': '0'} 2024-02-01T10:22:12.441 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', '-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006e18d4b30806cdea-0065bb70d4-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:12.441 DEBUG:boto:path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:12.441 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:12.441 DEBUG:boto:path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:12.441 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:12.442 DEBUG:boto:Method: HEAD 2024-02-01T10:22:12.442 DEBUG:boto:Path: /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:12.442 DEBUG:boto:Data: 2024-02-01T10:22:12.442 DEBUG:boto:Headers: {} 2024-02-01T10:22:12.442 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:12.442 DEBUG:boto:Port: 8000 2024-02-01T10:22:12.442 DEBUG:boto:Params: {} 2024-02-01T10:22:12.442 DEBUG:boto:Token: None 2024-02-01T10:22:12.442 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:12 GMT /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:12.442 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:OBWvZyZ+9+kqEPPvmbDdX8Ya7U8= 2024-02-01T10:22:12.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:OBWvZyZ+9+kqEPPvmbDdX8Ya7U8=', 'Content-Length': '0'} 2024-02-01T10:22:12.445 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', '-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f94cbf96563d36db-0065bb70d4-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:12.445 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-02-01T10:22:12.445 DEBUG:boto:path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:12.446 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:12.446 DEBUG:boto:path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:12.446 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:12.446 DEBUG:boto:Method: GET 2024-02-01T10:22:12.446 DEBUG:boto:Path: /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:12.446 DEBUG:boto:Data: 2024-02-01T10:22:12.446 DEBUG:boto:Headers: {} 2024-02-01T10:22:12.446 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:12.446 DEBUG:boto:Port: 8001 2024-02-01T10:22:12.446 DEBUG:boto:Params: {} 2024-02-01T10:22:12.446 DEBUG:boto:Token: None 2024-02-01T10:22:12.447 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:12 GMT /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:12.447 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wBR4Bgd/ZyABXyULOzfYld/Hk9E= 2024-02-01T10:22:12.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wBR4Bgd/ZyABXyULOzfYld/Hk9E=', 'Content-Length': '0'} 2024-02-01T10:22:12.450 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', '-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000006b7f6a69464c82b-0065bb70d4-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:12.492 DEBUG:boto:path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:12.492 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:12.492 DEBUG:boto:path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:12.493 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:12.493 DEBUG:boto:Method: GET 2024-02-01T10:22:12.493 DEBUG:boto:Path: /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:12.493 DEBUG:boto:Data: 2024-02-01T10:22:12.493 DEBUG:boto:Headers: {} 2024-02-01T10:22:12.493 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:12.493 DEBUG:boto:Port: 8000 2024-02-01T10:22:12.493 DEBUG:boto:Params: {} 2024-02-01T10:22:12.493 DEBUG:boto:Token: None 2024-02-01T10:22:12.493 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:12 GMT /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:12.494 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wBR4Bgd/ZyABXyULOzfYld/Hk9E= 2024-02-01T10:22:12.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wBR4Bgd/ZyABXyULOzfYld/Hk9E=', 'Content-Length': '0'} 2024-02-01T10:22:12.497 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', '-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004ee2f244ef94ae75-0065bb70d4-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:12.540 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:12.540 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:12.540 DEBUG:boto:path=/vclfcn-17/?prefix=obj-test-zone2 2024-02-01T10:22:12.540 DEBUG:boto:auth_path=/vclfcn-17/?prefix=obj-test-zone2 2024-02-01T10:22:12.541 DEBUG:boto:Method: GET 2024-02-01T10:22:12.541 DEBUG:boto:Path: /vclfcn-17/?prefix=obj-test-zone2 2024-02-01T10:22:12.541 DEBUG:boto:Data: 2024-02-01T10:22:12.541 DEBUG:boto:Headers: {} 2024-02-01T10:22:12.541 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:12.541 DEBUG:boto:Port: 8001 2024-02-01T10:22:12.541 DEBUG:boto:Params: {} 2024-02-01T10:22:12.541 DEBUG:boto:Token: None 2024-02-01T10:22:12.541 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:12 GMT /vclfcn-17/ 2024-02-01T10:22:12.542 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:f1HPd+2xBd4FvZlv1vmHFEHulsU= 2024-02-01T10:22:12.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:f1HPd+2xBd4FvZlv1vmHFEHulsU=', 'Content-Length': '0'} 2024-02-01T10:22:12.546 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001566aacfb91627ba-0065bb70d4-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:12.546 DEBUG:boto:b'vclfcn-17obj-test-zone21000falseobj-test-zone22024-02-01T10:20:50.624Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:12.547 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:12.547 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:12.547 DEBUG:boto:path=/vclfcn-17/?prefix=obj-test-zone2 2024-02-01T10:22:12.547 DEBUG:boto:auth_path=/vclfcn-17/?prefix=obj-test-zone2 2024-02-01T10:22:12.547 DEBUG:boto:Method: GET 2024-02-01T10:22:12.547 DEBUG:boto:Path: /vclfcn-17/?prefix=obj-test-zone2 2024-02-01T10:22:12.547 DEBUG:boto:Data: 2024-02-01T10:22:12.547 DEBUG:boto:Headers: {} 2024-02-01T10:22:12.547 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:12.547 DEBUG:boto:Port: 8000 2024-02-01T10:22:12.548 DEBUG:boto:Params: {} 2024-02-01T10:22:12.548 DEBUG:boto:Token: None 2024-02-01T10:22:12.548 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:12 GMT /vclfcn-17/ 2024-02-01T10:22:12.548 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:f1HPd+2xBd4FvZlv1vmHFEHulsU= 2024-02-01T10:22:12.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:f1HPd+2xBd4FvZlv1vmHFEHulsU=', 'Content-Length': '0'} 2024-02-01T10:22:12.552 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bcc119db600f9ac0-0065bb70d4-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:12.552 DEBUG:boto:b'vclfcn-17obj-test-zone21000falseobj-test-zone22024-02-01T10:20:50.624Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:12.553 DEBUG:boto:path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:12.553 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:12.553 DEBUG:boto:Method: HEAD 2024-02-01T10:22:12.553 DEBUG:boto:Path: /vclfcn-17/obj-test-zone2 2024-02-01T10:22:12.553 DEBUG:boto:Data: 2024-02-01T10:22:12.553 DEBUG:boto:Headers: {} 2024-02-01T10:22:12.553 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:12.553 DEBUG:boto:Port: 8001 2024-02-01T10:22:12.553 DEBUG:boto:Params: {} 2024-02-01T10:22:12.553 DEBUG:boto:Token: None 2024-02-01T10:22:12.554 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:12 GMT /vclfcn-17/obj-test-zone2 2024-02-01T10:22:12.554 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HBqTkAfV97GgH9QpWWc2F6z/Oac= 2024-02-01T10:22:12.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HBqTkAfV97GgH9QpWWc2F6z/Oac=', 'Content-Length': '0'} 2024-02-01T10:22:12.557 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', '-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000090e3805dcc759555-0065bb70d4-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:12.557 DEBUG:boto:path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:12.558 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:12.558 DEBUG:boto:Method: HEAD 2024-02-01T10:22:12.558 DEBUG:boto:Path: /vclfcn-17/obj-test-zone2 2024-02-01T10:22:12.558 DEBUG:boto:Data: 2024-02-01T10:22:12.558 DEBUG:boto:Headers: {} 2024-02-01T10:22:12.558 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:12.558 DEBUG:boto:Port: 8000 2024-02-01T10:22:12.558 DEBUG:boto:Params: {} 2024-02-01T10:22:12.558 DEBUG:boto:Token: None 2024-02-01T10:22:12.558 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:12 GMT /vclfcn-17/obj-test-zone2 2024-02-01T10:22:12.559 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HBqTkAfV97GgH9QpWWc2F6z/Oac= 2024-02-01T10:22:12.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HBqTkAfV97GgH9QpWWc2F6z/Oac=', 'Content-Length': '0'} 2024-02-01T10:22:12.562 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', '-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000afe56c1a3ea54f20-0065bb70d4-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:12.562 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-02-01T10:22:12.562 DEBUG:boto:path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:12.562 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:12.563 DEBUG:boto:path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:12.563 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:12.563 DEBUG:boto:Method: GET 2024-02-01T10:22:12.563 DEBUG:boto:Path: /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:12.563 DEBUG:boto:Data: 2024-02-01T10:22:12.563 DEBUG:boto:Headers: {} 2024-02-01T10:22:12.563 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:12.563 DEBUG:boto:Port: 8001 2024-02-01T10:22:12.563 DEBUG:boto:Params: {} 2024-02-01T10:22:12.563 DEBUG:boto:Token: None 2024-02-01T10:22:12.564 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:12 GMT /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:12.564 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wBR4Bgd/ZyABXyULOzfYld/Hk9E= 2024-02-01T10:22:12.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wBR4Bgd/ZyABXyULOzfYld/Hk9E=', 'Content-Length': '0'} 2024-02-01T10:22:12.566 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', '-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009ade40e37f7596a8-0065bb70d4-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:12.612 DEBUG:boto:path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:12.612 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:12.612 DEBUG:boto:path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:12.612 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:12.613 DEBUG:boto:Method: GET 2024-02-01T10:22:12.613 DEBUG:boto:Path: /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:12.613 DEBUG:boto:Data: 2024-02-01T10:22:12.613 DEBUG:boto:Headers: {} 2024-02-01T10:22:12.613 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:12.613 DEBUG:boto:Port: 8000 2024-02-01T10:22:12.613 DEBUG:boto:Params: {} 2024-02-01T10:22:12.613 DEBUG:boto:Token: None 2024-02-01T10:22:12.613 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:12 GMT /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:12.613 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wBR4Bgd/ZyABXyULOzfYld/Hk9E= 2024-02-01T10:22:12.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wBR4Bgd/ZyABXyULOzfYld/Hk9E=', 'Content-Length': '0'} 2024-02-01T10:22:12.616 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', '-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c764afe729216ac9-0065bb70d4-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:12.664 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:12.664 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:12.664 DEBUG:boto:path=/vclfcn-17/?prefix=obj-test-zone2 2024-02-01T10:22:12.665 DEBUG:boto:auth_path=/vclfcn-17/?prefix=obj-test-zone2 2024-02-01T10:22:12.665 DEBUG:boto:Method: GET 2024-02-01T10:22:12.665 DEBUG:boto:Path: /vclfcn-17/?prefix=obj-test-zone2 2024-02-01T10:22:12.665 DEBUG:boto:Data: 2024-02-01T10:22:12.665 DEBUG:boto:Headers: {} 2024-02-01T10:22:12.665 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:12.665 DEBUG:boto:Port: 8001 2024-02-01T10:22:12.665 DEBUG:boto:Params: {} 2024-02-01T10:22:12.665 DEBUG:boto:Token: None 2024-02-01T10:22:12.665 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:12 GMT /vclfcn-17/ 2024-02-01T10:22:12.666 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:f1HPd+2xBd4FvZlv1vmHFEHulsU= 2024-02-01T10:22:12.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:f1HPd+2xBd4FvZlv1vmHFEHulsU=', 'Content-Length': '0'} 2024-02-01T10:22:12.670 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003893df99862aec33-0065bb70d4-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:12.716 DEBUG:boto:b'vclfcn-17obj-test-zone21000falseobj-test-zone22024-02-01T10:20:50.624Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:12.717 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:12.717 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:12.717 DEBUG:boto:path=/vclfcn-17/?prefix=obj-test-zone2 2024-02-01T10:22:12.717 DEBUG:boto:auth_path=/vclfcn-17/?prefix=obj-test-zone2 2024-02-01T10:22:12.717 DEBUG:boto:Method: GET 2024-02-01T10:22:12.717 DEBUG:boto:Path: /vclfcn-17/?prefix=obj-test-zone2 2024-02-01T10:22:12.717 DEBUG:boto:Data: 2024-02-01T10:22:12.717 DEBUG:boto:Headers: {} 2024-02-01T10:22:12.717 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:12.718 DEBUG:boto:Port: 8000 2024-02-01T10:22:12.718 DEBUG:boto:Params: {} 2024-02-01T10:22:12.718 DEBUG:boto:Token: None 2024-02-01T10:22:12.718 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:12 GMT /vclfcn-17/ 2024-02-01T10:22:12.718 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:f1HPd+2xBd4FvZlv1vmHFEHulsU= 2024-02-01T10:22:12.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:f1HPd+2xBd4FvZlv1vmHFEHulsU=', 'Content-Length': '0'} 2024-02-01T10:22:12.723 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033cdd4f94d7bf50e-0065bb70d4-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:12.723 DEBUG:boto:b'vclfcn-17obj-test-zone21000falseobj-test-zone22024-02-01T10:20:50.624Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:12.724 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-02-01T10:22:12.724 DEBUG:boto:path=/vclfcn-17/obj-test-zone3 2024-02-01T10:22:12.724 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3 2024-02-01T10:22:12.724 DEBUG:boto:path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:12.724 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:12.724 DEBUG:boto:Method: GET 2024-02-01T10:22:12.724 DEBUG:boto:Path: /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:12.725 DEBUG:boto:Data: 2024-02-01T10:22:12.725 DEBUG:boto:Headers: {} 2024-02-01T10:22:12.725 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:12.725 DEBUG:boto:Port: 8001 2024-02-01T10:22:12.725 DEBUG:boto:Params: {} 2024-02-01T10:22:12.725 DEBUG:boto:Token: None 2024-02-01T10:22:12.726 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:12 GMT /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:12.726 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mffsrFG8zNIOB4CTggJ+0/jKlyU= 2024-02-01T10:22:12.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mffsrFG8zNIOB4CTggJ+0/jKlyU=', 'Content-Length': '0'} 2024-02-01T10:22:12.729 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'YE736OwOOXbZJHJe7d8YvcNV1LxwcQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000003e8b589b62a1be7-0065bb70d4-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:12.729 DEBUG:boto:path=/vclfcn-17/obj-test-zone3 2024-02-01T10:22:12.730 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3 2024-02-01T10:22:12.730 DEBUG:boto:path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:12.730 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:12.730 DEBUG:boto:Method: GET 2024-02-01T10:22:12.730 DEBUG:boto:Path: /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:12.730 DEBUG:boto:Data: 2024-02-01T10:22:12.730 DEBUG:boto:Headers: {} 2024-02-01T10:22:12.731 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:12.731 DEBUG:boto:Port: 8000 2024-02-01T10:22:12.731 DEBUG:boto:Params: {} 2024-02-01T10:22:12.731 DEBUG:boto:Token: None 2024-02-01T10:22:12.732 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:12 GMT /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:12.732 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mffsrFG8zNIOB4CTggJ+0/jKlyU= 2024-02-01T10:22:12.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mffsrFG8zNIOB4CTggJ+0/jKlyU=', 'Content-Length': '0'} 2024-02-01T10:22:12.735 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'YE736OwOOXbZJHJe7d8YvcNV1LxwcQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000364aab1fe727923c-0065bb70d4-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:12.735 DEBUG:boto:path=/vclfcn-17/obj-test-zone3 2024-02-01T10:22:12.735 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3 2024-02-01T10:22:12.735 DEBUG:boto:path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:12.735 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:12.736 DEBUG:boto:Method: HEAD 2024-02-01T10:22:12.736 DEBUG:boto:Path: /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:12.736 DEBUG:boto:Data: 2024-02-01T10:22:12.736 DEBUG:boto:Headers: {} 2024-02-01T10:22:12.736 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:12.736 DEBUG:boto:Port: 8001 2024-02-01T10:22:12.736 DEBUG:boto:Params: {} 2024-02-01T10:22:12.736 DEBUG:boto:Token: None 2024-02-01T10:22:12.736 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:12 GMT /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:12.737 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:duscqRtUyz+ZBDKZba99HofW1/Y= 2024-02-01T10:22:12.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:duscqRtUyz+ZBDKZba99HofW1/Y=', 'Content-Length': '0'} 2024-02-01T10:22:12.739 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'YE736OwOOXbZJHJe7d8YvcNV1LxwcQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d508480d51472768-0065bb70d4-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:12.739 DEBUG:boto:path=/vclfcn-17/obj-test-zone3 2024-02-01T10:22:12.740 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3 2024-02-01T10:22:12.740 DEBUG:boto:path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:12.740 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:12.740 DEBUG:boto:Method: HEAD 2024-02-01T10:22:12.740 DEBUG:boto:Path: /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:12.740 DEBUG:boto:Data: 2024-02-01T10:22:12.740 DEBUG:boto:Headers: {} 2024-02-01T10:22:12.740 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:12.740 DEBUG:boto:Port: 8000 2024-02-01T10:22:12.740 DEBUG:boto:Params: {} 2024-02-01T10:22:12.740 DEBUG:boto:Token: None 2024-02-01T10:22:12.741 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:12 GMT /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:12.741 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:duscqRtUyz+ZBDKZba99HofW1/Y= 2024-02-01T10:22:12.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:duscqRtUyz+ZBDKZba99HofW1/Y=', 'Content-Length': '0'} 2024-02-01T10:22:12.743 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'YE736OwOOXbZJHJe7d8YvcNV1LxwcQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000081e0233d3cd4b764-0065bb70d4-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:12.744 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-02-01T10:22:12.744 DEBUG:boto:path=/vclfcn-17/obj-test-zone3 2024-02-01T10:22:12.744 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3 2024-02-01T10:22:12.744 DEBUG:boto:path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:12.744 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:12.744 DEBUG:boto:Method: GET 2024-02-01T10:22:12.744 DEBUG:boto:Path: /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:12.744 DEBUG:boto:Data: 2024-02-01T10:22:12.744 DEBUG:boto:Headers: {} 2024-02-01T10:22:12.744 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:12.745 DEBUG:boto:Port: 8001 2024-02-01T10:22:12.745 DEBUG:boto:Params: {} 2024-02-01T10:22:12.745 DEBUG:boto:Token: None 2024-02-01T10:22:12.745 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:12 GMT /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:12.745 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mffsrFG8zNIOB4CTggJ+0/jKlyU= 2024-02-01T10:22:12.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mffsrFG8zNIOB4CTggJ+0/jKlyU=', 'Content-Length': '0'} 2024-02-01T10:22:12.748 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'YE736OwOOXbZJHJe7d8YvcNV1LxwcQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000029ee114171a62fbd-0065bb70d4-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:12.748 DEBUG:boto:path=/vclfcn-17/obj-test-zone3 2024-02-01T10:22:12.748 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3 2024-02-01T10:22:12.748 DEBUG:boto:path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:12.748 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:12.748 DEBUG:boto:Method: GET 2024-02-01T10:22:12.749 DEBUG:boto:Path: /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:12.749 DEBUG:boto:Data: 2024-02-01T10:22:12.749 DEBUG:boto:Headers: {} 2024-02-01T10:22:12.749 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:12.749 DEBUG:boto:Port: 8000 2024-02-01T10:22:12.749 DEBUG:boto:Params: {} 2024-02-01T10:22:12.749 DEBUG:boto:Token: None 2024-02-01T10:22:12.749 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:12 GMT /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:12.749 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mffsrFG8zNIOB4CTggJ+0/jKlyU= 2024-02-01T10:22:12.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mffsrFG8zNIOB4CTggJ+0/jKlyU=', 'Content-Length': '0'} 2024-02-01T10:22:12.752 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'YE736OwOOXbZJHJe7d8YvcNV1LxwcQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000084727c005e8e2b65-0065bb70d4-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:12.796 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:12.796 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:12.796 DEBUG:boto:path=/vclfcn-17/?prefix=obj-test-zone3 2024-02-01T10:22:12.796 DEBUG:boto:auth_path=/vclfcn-17/?prefix=obj-test-zone3 2024-02-01T10:22:12.797 DEBUG:boto:Method: GET 2024-02-01T10:22:12.797 DEBUG:boto:Path: /vclfcn-17/?prefix=obj-test-zone3 2024-02-01T10:22:12.797 DEBUG:boto:Data: 2024-02-01T10:22:12.797 DEBUG:boto:Headers: {} 2024-02-01T10:22:12.797 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:12.797 DEBUG:boto:Port: 8001 2024-02-01T10:22:12.797 DEBUG:boto:Params: {} 2024-02-01T10:22:12.798 DEBUG:boto:Token: None 2024-02-01T10:22:12.798 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:12 GMT /vclfcn-17/ 2024-02-01T10:22:12.798 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:f1HPd+2xBd4FvZlv1vmHFEHulsU= 2024-02-01T10:22:12.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:f1HPd+2xBd4FvZlv1vmHFEHulsU=', 'Content-Length': '0'} 2024-02-01T10:22:12.803 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007806e860baea9266-0065bb70d4-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:12.803 DEBUG:boto:b'vclfcn-17obj-test-zone31000falseobj-test-zone32024-02-01T10:20:50.780Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:12.803 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:12.803 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:12.804 DEBUG:boto:path=/vclfcn-17/?prefix=obj-test-zone3 2024-02-01T10:22:12.804 DEBUG:boto:auth_path=/vclfcn-17/?prefix=obj-test-zone3 2024-02-01T10:22:12.804 DEBUG:boto:Method: GET 2024-02-01T10:22:12.804 DEBUG:boto:Path: /vclfcn-17/?prefix=obj-test-zone3 2024-02-01T10:22:12.804 DEBUG:boto:Data: 2024-02-01T10:22:12.804 DEBUG:boto:Headers: {} 2024-02-01T10:22:12.804 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:12.804 DEBUG:boto:Port: 8000 2024-02-01T10:22:12.804 DEBUG:boto:Params: {} 2024-02-01T10:22:12.804 DEBUG:boto:Token: None 2024-02-01T10:22:12.805 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:12 GMT /vclfcn-17/ 2024-02-01T10:22:12.805 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:f1HPd+2xBd4FvZlv1vmHFEHulsU= 2024-02-01T10:22:12.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:f1HPd+2xBd4FvZlv1vmHFEHulsU=', 'Content-Length': '0'} 2024-02-01T10:22:12.809 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008710fa68bb0e31ca-0065bb70d4-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:12.809 DEBUG:boto:b'vclfcn-17obj-test-zone31000falseobj-test-zone32024-02-01T10:20:50.780Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:12.810 DEBUG:boto:path=/vclfcn-17/obj-test-zone3 2024-02-01T10:22:12.810 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3 2024-02-01T10:22:12.810 DEBUG:boto:Method: HEAD 2024-02-01T10:22:12.810 DEBUG:boto:Path: /vclfcn-17/obj-test-zone3 2024-02-01T10:22:12.810 DEBUG:boto:Data: 2024-02-01T10:22:12.810 DEBUG:boto:Headers: {} 2024-02-01T10:22:12.810 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:12.810 DEBUG:boto:Port: 8001 2024-02-01T10:22:12.811 DEBUG:boto:Params: {} 2024-02-01T10:22:12.811 DEBUG:boto:Token: None 2024-02-01T10:22:12.811 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:12 GMT /vclfcn-17/obj-test-zone3 2024-02-01T10:22:12.811 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:VJ5cWm2628W2BgWXnosDJO2BNH4= 2024-02-01T10:22:12.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:VJ5cWm2628W2BgWXnosDJO2BNH4=', 'Content-Length': '0'} 2024-02-01T10:22:12.815 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'YE736OwOOXbZJHJe7d8YvcNV1LxwcQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000db9904114753934c-0065bb70d4-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:12.815 DEBUG:boto:path=/vclfcn-17/obj-test-zone3 2024-02-01T10:22:12.815 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3 2024-02-01T10:22:12.815 DEBUG:boto:Method: HEAD 2024-02-01T10:22:12.815 DEBUG:boto:Path: /vclfcn-17/obj-test-zone3 2024-02-01T10:22:12.815 DEBUG:boto:Data: 2024-02-01T10:22:12.816 DEBUG:boto:Headers: {} 2024-02-01T10:22:12.816 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:12.816 DEBUG:boto:Port: 8000 2024-02-01T10:22:12.816 DEBUG:boto:Params: {} 2024-02-01T10:22:12.816 DEBUG:boto:Token: None 2024-02-01T10:22:12.816 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:12 GMT /vclfcn-17/obj-test-zone3 2024-02-01T10:22:12.816 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:VJ5cWm2628W2BgWXnosDJO2BNH4= 2024-02-01T10:22:12.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:VJ5cWm2628W2BgWXnosDJO2BNH4=', 'Content-Length': '0'} 2024-02-01T10:22:12.820 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'YE736OwOOXbZJHJe7d8YvcNV1LxwcQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000096f3904e45424892-0065bb70d4-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:12.820 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-02-01T10:22:12.820 DEBUG:boto:path=/vclfcn-17/obj-test-zone3 2024-02-01T10:22:12.820 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3 2024-02-01T10:22:12.820 DEBUG:boto:path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:12.820 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:12.820 DEBUG:boto:Method: GET 2024-02-01T10:22:12.821 DEBUG:boto:Path: /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:12.821 DEBUG:boto:Data: 2024-02-01T10:22:12.821 DEBUG:boto:Headers: {} 2024-02-01T10:22:12.821 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:12.821 DEBUG:boto:Port: 8001 2024-02-01T10:22:12.821 DEBUG:boto:Params: {} 2024-02-01T10:22:12.821 DEBUG:boto:Token: None 2024-02-01T10:22:12.821 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:12 GMT /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:12.821 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mffsrFG8zNIOB4CTggJ+0/jKlyU= 2024-02-01T10:22:12.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mffsrFG8zNIOB4CTggJ+0/jKlyU=', 'Content-Length': '0'} 2024-02-01T10:22:12.824 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'YE736OwOOXbZJHJe7d8YvcNV1LxwcQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005abf7eacb92e88b0-0065bb70d4-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:12.868 DEBUG:boto:path=/vclfcn-17/obj-test-zone3 2024-02-01T10:22:12.868 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3 2024-02-01T10:22:12.868 DEBUG:boto:path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:12.869 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:12.869 DEBUG:boto:Method: GET 2024-02-01T10:22:12.869 DEBUG:boto:Path: /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:12.869 DEBUG:boto:Data: 2024-02-01T10:22:12.869 DEBUG:boto:Headers: {} 2024-02-01T10:22:12.869 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:12.869 DEBUG:boto:Port: 8000 2024-02-01T10:22:12.869 DEBUG:boto:Params: {} 2024-02-01T10:22:12.869 DEBUG:boto:Token: None 2024-02-01T10:22:12.870 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:12 GMT /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:12.870 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mffsrFG8zNIOB4CTggJ+0/jKlyU= 2024-02-01T10:22:12.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mffsrFG8zNIOB4CTggJ+0/jKlyU=', 'Content-Length': '0'} 2024-02-01T10:22:12.873 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'YE736OwOOXbZJHJe7d8YvcNV1LxwcQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004cc1698fa8ee474e-0065bb70d4-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:12.916 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:12.916 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:12.916 DEBUG:boto:path=/vclfcn-17/?prefix=obj-test-zone3 2024-02-01T10:22:12.916 DEBUG:boto:auth_path=/vclfcn-17/?prefix=obj-test-zone3 2024-02-01T10:22:12.917 DEBUG:boto:Method: GET 2024-02-01T10:22:12.917 DEBUG:boto:Path: /vclfcn-17/?prefix=obj-test-zone3 2024-02-01T10:22:12.917 DEBUG:boto:Data: 2024-02-01T10:22:12.917 DEBUG:boto:Headers: {} 2024-02-01T10:22:12.917 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:12.917 DEBUG:boto:Port: 8001 2024-02-01T10:22:12.917 DEBUG:boto:Params: {} 2024-02-01T10:22:12.917 DEBUG:boto:Token: None 2024-02-01T10:22:12.917 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:12 GMT /vclfcn-17/ 2024-02-01T10:22:12.918 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:f1HPd+2xBd4FvZlv1vmHFEHulsU= 2024-02-01T10:22:12.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:f1HPd+2xBd4FvZlv1vmHFEHulsU=', 'Content-Length': '0'} 2024-02-01T10:22:12.922 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf3e2b879841feab-0065bb70d4-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:12.923 DEBUG:boto:b'vclfcn-17obj-test-zone31000falseobj-test-zone32024-02-01T10:20:50.780Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:12.923 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:12.923 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:12.924 DEBUG:boto:path=/vclfcn-17/?prefix=obj-test-zone3 2024-02-01T10:22:12.924 DEBUG:boto:auth_path=/vclfcn-17/?prefix=obj-test-zone3 2024-02-01T10:22:12.924 DEBUG:boto:Method: GET 2024-02-01T10:22:12.924 DEBUG:boto:Path: /vclfcn-17/?prefix=obj-test-zone3 2024-02-01T10:22:12.924 DEBUG:boto:Data: 2024-02-01T10:22:12.924 DEBUG:boto:Headers: {} 2024-02-01T10:22:12.924 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:12.924 DEBUG:boto:Port: 8000 2024-02-01T10:22:12.924 DEBUG:boto:Params: {} 2024-02-01T10:22:12.924 DEBUG:boto:Token: None 2024-02-01T10:22:12.925 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:12 GMT /vclfcn-17/ 2024-02-01T10:22:12.925 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:f1HPd+2xBd4FvZlv1vmHFEHulsU= 2024-02-01T10:22:12.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:f1HPd+2xBd4FvZlv1vmHFEHulsU=', 'Content-Length': '0'} 2024-02-01T10:22:12.929 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef17996030523d21-0065bb70d4-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:12.930 DEBUG:boto:b'vclfcn-17obj-test-zone31000falseobj-test-zone32024-02-01T10:20:50.780Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:12.930 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-17 zones={test-zone3, test-zone1} 2024-02-01T10:22:12.930 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-17 zones={test-zone3, test-zone2} 2024-02-01T10:22:12.930 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:12.930 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:12.930 DEBUG:boto:Method: HEAD 2024-02-01T10:22:12.931 DEBUG:boto:Path: /vclfcn-17/ 2024-02-01T10:22:12.931 DEBUG:boto:Data: 2024-02-01T10:22:12.931 DEBUG:boto:Headers: {} 2024-02-01T10:22:12.931 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:12.931 DEBUG:boto:Port: 8001 2024-02-01T10:22:12.931 DEBUG:boto:Params: {} 2024-02-01T10:22:12.931 DEBUG:boto:Token: None 2024-02-01T10:22:12.931 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:12 GMT /vclfcn-17/ 2024-02-01T10:22:12.932 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:myAP45juOCFRwU2HvMUqbBx0l8U= 2024-02-01T10:22:12.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:myAP45juOCFRwU2HvMUqbBx0l8U=', 'Content-Length': '0'} 2024-02-01T10:22:12.936 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ed4ba9934fa944ad-0065bb70d4-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:22:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:12.936 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:12.936 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:12.937 DEBUG:boto:Method: HEAD 2024-02-01T10:22:12.937 DEBUG:boto:Path: /vclfcn-17/ 2024-02-01T10:22:12.937 DEBUG:boto:Data: 2024-02-01T10:22:12.937 DEBUG:boto:Headers: {} 2024-02-01T10:22:12.937 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:12.937 DEBUG:boto:Port: 8000 2024-02-01T10:22:12.937 DEBUG:boto:Params: {} 2024-02-01T10:22:12.937 DEBUG:boto:Token: None 2024-02-01T10:22:12.937 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:12 GMT /vclfcn-17/ 2024-02-01T10:22:12.937 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:myAP45juOCFRwU2HvMUqbBx0l8U= 2024-02-01T10:22:12.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:myAP45juOCFRwU2HvMUqbBx0l8U=', 'Content-Length': '0'} 2024-02-01T10:22:12.942 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000698bb4877fc53a2c-0065bb70d4-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:22:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:12.942 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:22:12.943 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:12.943 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:12.943 DEBUG:boto:path=/vclfcn-17/?versions&max-keys=999 2024-02-01T10:22:12.943 DEBUG:boto:auth_path=/vclfcn-17/?versions&max-keys=999 2024-02-01T10:22:12.943 DEBUG:boto:Method: GET 2024-02-01T10:22:12.943 DEBUG:boto:Path: /vclfcn-17/?versions&max-keys=999 2024-02-01T10:22:12.943 DEBUG:boto:Data: 2024-02-01T10:22:12.944 DEBUG:boto:Headers: {} 2024-02-01T10:22:12.944 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:12.944 DEBUG:boto:Port: 8001 2024-02-01T10:22:12.944 DEBUG:boto:Params: {} 2024-02-01T10:22:12.944 DEBUG:boto:Token: None 2024-02-01T10:22:12.944 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:12 GMT /vclfcn-17/?versions 2024-02-01T10:22:12.944 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:IC9tBDy1JgV0D0QAOP4Q3KRzTNg= 2024-02-01T10:22:12.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:IC9tBDy1JgV0D0QAOP4Q3KRzTNg=', 'Content-Length': '0'} 2024-02-01T10:22:12.949 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f9112f27ca34c47-0065bb70d4-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:12.950 DEBUG:boto:b'vclfcn-17999falsedummyhQXZXSD45t2CSRVMJOLMBXih54iMPmJtrue2024-02-01T10:20:14.297Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1xMxAGR4A-xEILepnzFPRgrWNTOLvitztrue2024-02-01T10:20:50.473Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1true2024-02-01T10:20:50.624Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3YE736OwOOXbZJHJe7d8YvcNV1LxwcQbtrue2024-02-01T10:20:50.780Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:12.950 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-02-01T10:22:12.950 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-02-01T10:22:12.951 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-02-01T10:22:12.951 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-02-01T10:22:12.951 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:22:12.951 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:12.951 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:12.951 DEBUG:boto:path=/vclfcn-17/?versions&max-keys=999 2024-02-01T10:22:12.951 DEBUG:boto:auth_path=/vclfcn-17/?versions&max-keys=999 2024-02-01T10:22:12.951 DEBUG:boto:Method: GET 2024-02-01T10:22:12.951 DEBUG:boto:Path: /vclfcn-17/?versions&max-keys=999 2024-02-01T10:22:12.951 DEBUG:boto:Data: 2024-02-01T10:22:12.951 DEBUG:boto:Headers: {} 2024-02-01T10:22:12.952 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:12.952 DEBUG:boto:Port: 8000 2024-02-01T10:22:12.952 DEBUG:boto:Params: {} 2024-02-01T10:22:12.952 DEBUG:boto:Token: None 2024-02-01T10:22:12.952 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:12 GMT /vclfcn-17/?versions 2024-02-01T10:22:12.952 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:IC9tBDy1JgV0D0QAOP4Q3KRzTNg= 2024-02-01T10:22:12.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:IC9tBDy1JgV0D0QAOP4Q3KRzTNg=', 'Content-Length': '0'} 2024-02-01T10:22:12.957 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000308f8431fa22818f-0065bb70d4-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:12.957 DEBUG:boto:b'vclfcn-17999falsedummyhQXZXSD45t2CSRVMJOLMBXih54iMPmJtrue2024-02-01T10:20:14.297Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1xMxAGR4A-xEILepnzFPRgrWNTOLvitztrue2024-02-01T10:20:50.473Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1true2024-02-01T10:20:50.624Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3YE736OwOOXbZJHJe7d8YvcNV1LxwcQbtrue2024-02-01T10:20:50.780Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:12.958 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-02-01T10:22:12.958 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-02-01T10:22:12.958 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-02-01T10:22:12.958 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-02-01T10:22:12.958 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:12.958 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:12.958 DEBUG:boto:path=/vclfcn-17/?versions&max-keys=999 2024-02-01T10:22:12.958 DEBUG:boto:auth_path=/vclfcn-17/?versions&max-keys=999 2024-02-01T10:22:12.958 DEBUG:boto:Method: GET 2024-02-01T10:22:12.959 DEBUG:boto:Path: /vclfcn-17/?versions&max-keys=999 2024-02-01T10:22:12.959 DEBUG:boto:Data: 2024-02-01T10:22:12.959 DEBUG:boto:Headers: {} 2024-02-01T10:22:12.959 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:12.959 DEBUG:boto:Port: 8001 2024-02-01T10:22:12.959 DEBUG:boto:Params: {} 2024-02-01T10:22:12.959 DEBUG:boto:Token: None 2024-02-01T10:22:12.959 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:12 GMT /vclfcn-17/?versions 2024-02-01T10:22:12.959 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:IC9tBDy1JgV0D0QAOP4Q3KRzTNg= 2024-02-01T10:22:12.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:IC9tBDy1JgV0D0QAOP4Q3KRzTNg=', 'Content-Length': '0'} 2024-02-01T10:22:12.964 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097916719b7cc8287-0065bb70d4-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:12.964 DEBUG:boto:b'vclfcn-17999falsedummyhQXZXSD45t2CSRVMJOLMBXih54iMPmJtrue2024-02-01T10:20:14.297Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1xMxAGR4A-xEILepnzFPRgrWNTOLvitztrue2024-02-01T10:20:50.473Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1true2024-02-01T10:20:50.624Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3YE736OwOOXbZJHJe7d8YvcNV1LxwcQbtrue2024-02-01T10:20:50.780Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:12.965 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:12.965 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:12.965 DEBUG:boto:path=/vclfcn-17/?versions&max-keys=999 2024-02-01T10:22:12.965 DEBUG:boto:auth_path=/vclfcn-17/?versions&max-keys=999 2024-02-01T10:22:12.965 DEBUG:boto:Method: GET 2024-02-01T10:22:12.965 DEBUG:boto:Path: /vclfcn-17/?versions&max-keys=999 2024-02-01T10:22:12.965 DEBUG:boto:Data: 2024-02-01T10:22:12.965 DEBUG:boto:Headers: {} 2024-02-01T10:22:12.965 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:12.965 DEBUG:boto:Port: 8000 2024-02-01T10:22:12.966 DEBUG:boto:Params: {} 2024-02-01T10:22:12.966 DEBUG:boto:Token: None 2024-02-01T10:22:12.966 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:12 GMT /vclfcn-17/?versions 2024-02-01T10:22:12.966 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:IC9tBDy1JgV0D0QAOP4Q3KRzTNg= 2024-02-01T10:22:12.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:IC9tBDy1JgV0D0QAOP4Q3KRzTNg=', 'Content-Length': '0'} 2024-02-01T10:22:12.971 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001cc8cc8412b34afa-0065bb70d4-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:12.971 DEBUG:boto:b'vclfcn-17999falsedummyhQXZXSD45t2CSRVMJOLMBXih54iMPmJtrue2024-02-01T10:20:14.297Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1xMxAGR4A-xEILepnzFPRgrWNTOLvitztrue2024-02-01T10:20:50.473Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1true2024-02-01T10:20:50.624Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3YE736OwOOXbZJHJe7d8YvcNV1LxwcQbtrue2024-02-01T10:20:50.780Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:12.971 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-02-01T10:22:12.972 DEBUG:boto:path=/vclfcn-17/dummy 2024-02-01T10:22:12.972 DEBUG:boto:auth_path=/vclfcn-17/dummy 2024-02-01T10:22:12.972 DEBUG:boto:path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:12.972 DEBUG:boto:auth_path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:12.972 DEBUG:boto:Method: GET 2024-02-01T10:22:12.972 DEBUG:boto:Path: /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:12.972 DEBUG:boto:Data: 2024-02-01T10:22:12.972 DEBUG:boto:Headers: {} 2024-02-01T10:22:12.972 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:12.972 DEBUG:boto:Port: 8001 2024-02-01T10:22:12.973 DEBUG:boto:Params: {} 2024-02-01T10:22:12.973 DEBUG:boto:Token: None 2024-02-01T10:22:12.973 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:12 GMT /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:12.973 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Loy4+vXg04cg/0otzvelsmuTJ3M= 2024-02-01T10:22:12.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Loy4+vXg04cg/0otzvelsmuTJ3M=', 'Content-Length': '0'} 2024-02-01T10:22:12.977 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:14 GMT'), ('x-amz-version-id', 'hQXZXSD45t2CSRVMJOLMBXih54iMPmJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cb6cae41ea99863a-0065bb70d4-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:12.978 DEBUG:boto:path=/vclfcn-17/dummy 2024-02-01T10:22:12.978 DEBUG:boto:auth_path=/vclfcn-17/dummy 2024-02-01T10:22:12.978 DEBUG:boto:path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:12.978 DEBUG:boto:auth_path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:12.978 DEBUG:boto:Method: GET 2024-02-01T10:22:12.978 DEBUG:boto:Path: /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:12.978 DEBUG:boto:Data: 2024-02-01T10:22:12.978 DEBUG:boto:Headers: {} 2024-02-01T10:22:12.979 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:12.979 DEBUG:boto:Port: 8000 2024-02-01T10:22:12.979 DEBUG:boto:Params: {} 2024-02-01T10:22:12.979 DEBUG:boto:Token: None 2024-02-01T10:22:12.979 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:12 GMT /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:12.979 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Loy4+vXg04cg/0otzvelsmuTJ3M= 2024-02-01T10:22:12.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Loy4+vXg04cg/0otzvelsmuTJ3M=', 'Content-Length': '0'} 2024-02-01T10:22:12.982 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:14 GMT'), ('x-amz-version-id', 'hQXZXSD45t2CSRVMJOLMBXih54iMPmJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000028a965a06ff92ee6-0065bb70d4-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:12.982 DEBUG:boto:path=/vclfcn-17/dummy 2024-02-01T10:22:12.982 DEBUG:boto:auth_path=/vclfcn-17/dummy 2024-02-01T10:22:12.982 DEBUG:boto:path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:12.982 DEBUG:boto:auth_path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:12.983 DEBUG:boto:Method: HEAD 2024-02-01T10:22:12.983 DEBUG:boto:Path: /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:12.983 DEBUG:boto:Data: 2024-02-01T10:22:12.983 DEBUG:boto:Headers: {} 2024-02-01T10:22:12.983 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:12.983 DEBUG:boto:Port: 8001 2024-02-01T10:22:12.983 DEBUG:boto:Params: {} 2024-02-01T10:22:12.983 DEBUG:boto:Token: None 2024-02-01T10:22:12.984 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:12 GMT /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:12.984 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:a8srnaCydZVz5ItyPv5fA7Dt9gQ= 2024-02-01T10:22:12.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:a8srnaCydZVz5ItyPv5fA7Dt9gQ=', 'Content-Length': '0'} 2024-02-01T10:22:12.986 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:14 GMT'), ('x-amz-version-id', 'hQXZXSD45t2CSRVMJOLMBXih54iMPmJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005e5f700d9deef572-0065bb70d4-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:12.987 DEBUG:boto:path=/vclfcn-17/dummy 2024-02-01T10:22:12.987 DEBUG:boto:auth_path=/vclfcn-17/dummy 2024-02-01T10:22:12.987 DEBUG:boto:path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:12.987 DEBUG:boto:auth_path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:12.987 DEBUG:boto:Method: HEAD 2024-02-01T10:22:12.987 DEBUG:boto:Path: /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:12.987 DEBUG:boto:Data: 2024-02-01T10:22:12.987 DEBUG:boto:Headers: {} 2024-02-01T10:22:12.987 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:12.987 DEBUG:boto:Port: 8000 2024-02-01T10:22:12.988 DEBUG:boto:Params: {} 2024-02-01T10:22:12.988 DEBUG:boto:Token: None 2024-02-01T10:22:12.988 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:12 GMT /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:12.988 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:a8srnaCydZVz5ItyPv5fA7Dt9gQ= 2024-02-01T10:22:12.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:a8srnaCydZVz5ItyPv5fA7Dt9gQ=', 'Content-Length': '0'} 2024-02-01T10:22:12.991 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:14 GMT'), ('x-amz-version-id', 'hQXZXSD45t2CSRVMJOLMBXih54iMPmJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000099f026526309b4ca-0065bb70d4-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:12.991 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-02-01T10:22:12.991 DEBUG:boto:path=/vclfcn-17/dummy 2024-02-01T10:22:12.991 DEBUG:boto:auth_path=/vclfcn-17/dummy 2024-02-01T10:22:12.991 DEBUG:boto:path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:12.991 DEBUG:boto:auth_path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:12.992 DEBUG:boto:Method: GET 2024-02-01T10:22:12.992 DEBUG:boto:Path: /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:12.992 DEBUG:boto:Data: 2024-02-01T10:22:12.992 DEBUG:boto:Headers: {} 2024-02-01T10:22:12.992 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:12.992 DEBUG:boto:Port: 8001 2024-02-01T10:22:12.992 DEBUG:boto:Params: {} 2024-02-01T10:22:12.992 DEBUG:boto:Token: None 2024-02-01T10:22:12.992 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:12 GMT /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:12.992 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Loy4+vXg04cg/0otzvelsmuTJ3M= 2024-02-01T10:22:12.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Loy4+vXg04cg/0otzvelsmuTJ3M=', 'Content-Length': '0'} 2024-02-01T10:22:12.995 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:14 GMT'), ('x-amz-version-id', 'hQXZXSD45t2CSRVMJOLMBXih54iMPmJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d2ac36e205e0ebe9-0065bb70d4-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:12.996 DEBUG:boto:path=/vclfcn-17/dummy 2024-02-01T10:22:12.996 DEBUG:boto:auth_path=/vclfcn-17/dummy 2024-02-01T10:22:12.996 DEBUG:boto:path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:12.996 DEBUG:boto:auth_path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:12.996 DEBUG:boto:Method: GET 2024-02-01T10:22:12.996 DEBUG:boto:Path: /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:12.996 DEBUG:boto:Data: 2024-02-01T10:22:12.996 DEBUG:boto:Headers: {} 2024-02-01T10:22:12.996 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:12.996 DEBUG:boto:Port: 8000 2024-02-01T10:22:12.996 DEBUG:boto:Params: {} 2024-02-01T10:22:12.997 DEBUG:boto:Token: None 2024-02-01T10:22:12.997 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:12 GMT /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:12.997 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Loy4+vXg04cg/0otzvelsmuTJ3M= 2024-02-01T10:22:12.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Loy4+vXg04cg/0otzvelsmuTJ3M=', 'Content-Length': '0'} 2024-02-01T10:22:13.000 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:14 GMT'), ('x-amz-version-id', 'hQXZXSD45t2CSRVMJOLMBXih54iMPmJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000dcccab388ecb1c73-0065bb70d4-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:13.000 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:13.000 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:13.000 DEBUG:boto:path=/vclfcn-17/?prefix=dummy 2024-02-01T10:22:13.000 DEBUG:boto:auth_path=/vclfcn-17/?prefix=dummy 2024-02-01T10:22:13.000 DEBUG:boto:Method: GET 2024-02-01T10:22:13.000 DEBUG:boto:Path: /vclfcn-17/?prefix=dummy 2024-02-01T10:22:13.000 DEBUG:boto:Data: 2024-02-01T10:22:13.000 DEBUG:boto:Headers: {} 2024-02-01T10:22:13.001 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:13.001 DEBUG:boto:Port: 8001 2024-02-01T10:22:13.001 DEBUG:boto:Params: {} 2024-02-01T10:22:13.001 DEBUG:boto:Token: None 2024-02-01T10:22:13.001 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:13 GMT /vclfcn-17/ 2024-02-01T10:22:13.001 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZhPBUAcfmyHSSF5tz4MOV80Twsg= 2024-02-01T10:22:13.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZhPBUAcfmyHSSF5tz4MOV80Twsg=', 'Content-Length': '0'} 2024-02-01T10:22:13.005 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d09fe426201e2ef-0065bb70d5-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:13.048 DEBUG:boto:b'vclfcn-17dummy1000falsedummy2024-02-01T10:20:14.297Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:13.049 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:13.049 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:13.049 DEBUG:boto:path=/vclfcn-17/?prefix=dummy 2024-02-01T10:22:13.049 DEBUG:boto:auth_path=/vclfcn-17/?prefix=dummy 2024-02-01T10:22:13.049 DEBUG:boto:Method: GET 2024-02-01T10:22:13.049 DEBUG:boto:Path: /vclfcn-17/?prefix=dummy 2024-02-01T10:22:13.049 DEBUG:boto:Data: 2024-02-01T10:22:13.049 DEBUG:boto:Headers: {} 2024-02-01T10:22:13.049 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:13.049 DEBUG:boto:Port: 8000 2024-02-01T10:22:13.050 DEBUG:boto:Params: {} 2024-02-01T10:22:13.050 DEBUG:boto:Token: None 2024-02-01T10:22:13.050 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:13 GMT /vclfcn-17/ 2024-02-01T10:22:13.050 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZhPBUAcfmyHSSF5tz4MOV80Twsg= 2024-02-01T10:22:13.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZhPBUAcfmyHSSF5tz4MOV80Twsg=', 'Content-Length': '0'} 2024-02-01T10:22:13.055 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ec88fa6677c0a71-0065bb70d5-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:13.055 DEBUG:boto:b'vclfcn-17dummy1000falsedummy2024-02-01T10:20:14.297Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:13.055 DEBUG:boto:path=/vclfcn-17/dummy 2024-02-01T10:22:13.055 DEBUG:boto:auth_path=/vclfcn-17/dummy 2024-02-01T10:22:13.055 DEBUG:boto:Method: HEAD 2024-02-01T10:22:13.056 DEBUG:boto:Path: /vclfcn-17/dummy 2024-02-01T10:22:13.056 DEBUG:boto:Data: 2024-02-01T10:22:13.056 DEBUG:boto:Headers: {} 2024-02-01T10:22:13.056 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:13.056 DEBUG:boto:Port: 8001 2024-02-01T10:22:13.056 DEBUG:boto:Params: {} 2024-02-01T10:22:13.056 DEBUG:boto:Token: None 2024-02-01T10:22:13.057 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:13 GMT /vclfcn-17/dummy 2024-02-01T10:22:13.057 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:4ni6/ncZ5/iZaJG0Kanc6qeBG4I= 2024-02-01T10:22:13.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:4ni6/ncZ5/iZaJG0Kanc6qeBG4I=', 'Content-Length': '0'} 2024-02-01T10:22:13.060 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:14 GMT'), ('x-amz-version-id', 'hQXZXSD45t2CSRVMJOLMBXih54iMPmJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001e4f49c055a933e2-0065bb70d5-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:13.061 DEBUG:boto:path=/vclfcn-17/dummy 2024-02-01T10:22:13.061 DEBUG:boto:auth_path=/vclfcn-17/dummy 2024-02-01T10:22:13.061 DEBUG:boto:Method: HEAD 2024-02-01T10:22:13.061 DEBUG:boto:Path: /vclfcn-17/dummy 2024-02-01T10:22:13.061 DEBUG:boto:Data: 2024-02-01T10:22:13.061 DEBUG:boto:Headers: {} 2024-02-01T10:22:13.061 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:13.061 DEBUG:boto:Port: 8000 2024-02-01T10:22:13.061 DEBUG:boto:Params: {} 2024-02-01T10:22:13.062 DEBUG:boto:Token: None 2024-02-01T10:22:13.062 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:13 GMT /vclfcn-17/dummy 2024-02-01T10:22:13.062 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:4ni6/ncZ5/iZaJG0Kanc6qeBG4I= 2024-02-01T10:22:13.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:4ni6/ncZ5/iZaJG0Kanc6qeBG4I=', 'Content-Length': '0'} 2024-02-01T10:22:13.066 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:14 GMT'), ('x-amz-version-id', 'hQXZXSD45t2CSRVMJOLMBXih54iMPmJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f0c5d762d66bb4b9-0065bb70d5-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:13.066 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-02-01T10:22:13.067 DEBUG:boto:path=/vclfcn-17/dummy 2024-02-01T10:22:13.067 DEBUG:boto:auth_path=/vclfcn-17/dummy 2024-02-01T10:22:13.067 DEBUG:boto:path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:13.067 DEBUG:boto:auth_path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:13.067 DEBUG:boto:Method: GET 2024-02-01T10:22:13.067 DEBUG:boto:Path: /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:13.067 DEBUG:boto:Data: 2024-02-01T10:22:13.067 DEBUG:boto:Headers: {} 2024-02-01T10:22:13.067 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:13.067 DEBUG:boto:Port: 8001 2024-02-01T10:22:13.068 DEBUG:boto:Params: {} 2024-02-01T10:22:13.068 DEBUG:boto:Token: None 2024-02-01T10:22:13.068 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:13 GMT /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:13.068 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:frUOveN3JSSZEfiBUXLc09OHF8w= 2024-02-01T10:22:13.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:frUOveN3JSSZEfiBUXLc09OHF8w=', 'Content-Length': '0'} 2024-02-01T10:22:13.071 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:14 GMT'), ('x-amz-version-id', 'hQXZXSD45t2CSRVMJOLMBXih54iMPmJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000915abee8f3eecafd-0065bb70d5-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:13.071 DEBUG:boto:path=/vclfcn-17/dummy 2024-02-01T10:22:13.071 DEBUG:boto:auth_path=/vclfcn-17/dummy 2024-02-01T10:22:13.071 DEBUG:boto:path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:13.071 DEBUG:boto:auth_path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:13.071 DEBUG:boto:Method: GET 2024-02-01T10:22:13.072 DEBUG:boto:Path: /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:13.072 DEBUG:boto:Data: 2024-02-01T10:22:13.072 DEBUG:boto:Headers: {} 2024-02-01T10:22:13.072 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:13.072 DEBUG:boto:Port: 8000 2024-02-01T10:22:13.072 DEBUG:boto:Params: {} 2024-02-01T10:22:13.072 DEBUG:boto:Token: None 2024-02-01T10:22:13.072 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:13 GMT /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:13.072 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:frUOveN3JSSZEfiBUXLc09OHF8w= 2024-02-01T10:22:13.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:frUOveN3JSSZEfiBUXLc09OHF8w=', 'Content-Length': '0'} 2024-02-01T10:22:13.075 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:14 GMT'), ('x-amz-version-id', 'hQXZXSD45t2CSRVMJOLMBXih54iMPmJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008430b0f720a3df88-0065bb70d5-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:13.075 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:13.076 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:13.076 DEBUG:boto:path=/vclfcn-17/?prefix=dummy 2024-02-01T10:22:13.076 DEBUG:boto:auth_path=/vclfcn-17/?prefix=dummy 2024-02-01T10:22:13.076 DEBUG:boto:Method: GET 2024-02-01T10:22:13.076 DEBUG:boto:Path: /vclfcn-17/?prefix=dummy 2024-02-01T10:22:13.076 DEBUG:boto:Data: 2024-02-01T10:22:13.076 DEBUG:boto:Headers: {} 2024-02-01T10:22:13.076 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:13.076 DEBUG:boto:Port: 8001 2024-02-01T10:22:13.076 DEBUG:boto:Params: {} 2024-02-01T10:22:13.077 DEBUG:boto:Token: None 2024-02-01T10:22:13.077 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:13 GMT /vclfcn-17/ 2024-02-01T10:22:13.077 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZhPBUAcfmyHSSF5tz4MOV80Twsg= 2024-02-01T10:22:13.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZhPBUAcfmyHSSF5tz4MOV80Twsg=', 'Content-Length': '0'} 2024-02-01T10:22:13.081 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d599fc500aa34ab6-0065bb70d5-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:13.081 DEBUG:boto:b'vclfcn-17dummy1000falsedummy2024-02-01T10:20:14.297Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:13.082 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:13.082 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:13.082 DEBUG:boto:path=/vclfcn-17/?prefix=dummy 2024-02-01T10:22:13.082 DEBUG:boto:auth_path=/vclfcn-17/?prefix=dummy 2024-02-01T10:22:13.082 DEBUG:boto:Method: GET 2024-02-01T10:22:13.082 DEBUG:boto:Path: /vclfcn-17/?prefix=dummy 2024-02-01T10:22:13.082 DEBUG:boto:Data: 2024-02-01T10:22:13.082 DEBUG:boto:Headers: {} 2024-02-01T10:22:13.082 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:13.083 DEBUG:boto:Port: 8000 2024-02-01T10:22:13.083 DEBUG:boto:Params: {} 2024-02-01T10:22:13.083 DEBUG:boto:Token: None 2024-02-01T10:22:13.083 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:13 GMT /vclfcn-17/ 2024-02-01T10:22:13.083 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZhPBUAcfmyHSSF5tz4MOV80Twsg= 2024-02-01T10:22:13.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZhPBUAcfmyHSSF5tz4MOV80Twsg=', 'Content-Length': '0'} 2024-02-01T10:22:13.087 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e5188fd1cacc988-0065bb70d5-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:13.088 DEBUG:boto:b'vclfcn-17dummy1000falsedummy2024-02-01T10:20:14.297Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:13.088 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-02-01T10:22:13.088 DEBUG:boto:path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:13.088 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:13.088 DEBUG:boto:path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:13.088 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:13.088 DEBUG:boto:Method: GET 2024-02-01T10:22:13.088 DEBUG:boto:Path: /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:13.089 DEBUG:boto:Data: 2024-02-01T10:22:13.089 DEBUG:boto:Headers: {} 2024-02-01T10:22:13.089 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:13.089 DEBUG:boto:Port: 8001 2024-02-01T10:22:13.089 DEBUG:boto:Params: {} 2024-02-01T10:22:13.089 DEBUG:boto:Token: None 2024-02-01T10:22:13.089 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:13 GMT /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:13.089 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:hsvjy5qfn0B/PD8Lhv1V8nSoNaU= 2024-02-01T10:22:13.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:hsvjy5qfn0B/PD8Lhv1V8nSoNaU=', 'Content-Length': '0'} 2024-02-01T10:22:13.092 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'xMxAGR4A-xEILepnzFPRgrWNTOLvitz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cdc4748d02713b4f-0065bb70d5-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:13.136 DEBUG:boto:path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:13.136 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:13.136 DEBUG:boto:path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:13.137 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:13.137 DEBUG:boto:Method: GET 2024-02-01T10:22:13.137 DEBUG:boto:Path: /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:13.137 DEBUG:boto:Data: 2024-02-01T10:22:13.137 DEBUG:boto:Headers: {} 2024-02-01T10:22:13.137 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:13.137 DEBUG:boto:Port: 8000 2024-02-01T10:22:13.137 DEBUG:boto:Params: {} 2024-02-01T10:22:13.137 DEBUG:boto:Token: None 2024-02-01T10:22:13.137 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:13 GMT /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:13.138 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:hsvjy5qfn0B/PD8Lhv1V8nSoNaU= 2024-02-01T10:22:13.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:hsvjy5qfn0B/PD8Lhv1V8nSoNaU=', 'Content-Length': '0'} 2024-02-01T10:22:13.141 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'xMxAGR4A-xEILepnzFPRgrWNTOLvitz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ace885de77ba4c5d-0065bb70d5-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:13.141 DEBUG:boto:path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:13.142 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:13.142 DEBUG:boto:path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:13.142 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:13.142 DEBUG:boto:Method: HEAD 2024-02-01T10:22:13.142 DEBUG:boto:Path: /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:13.142 DEBUG:boto:Data: 2024-02-01T10:22:13.142 DEBUG:boto:Headers: {} 2024-02-01T10:22:13.142 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:13.142 DEBUG:boto:Port: 8001 2024-02-01T10:22:13.142 DEBUG:boto:Params: {} 2024-02-01T10:22:13.142 DEBUG:boto:Token: None 2024-02-01T10:22:13.143 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:13 GMT /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:13.143 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:hVdlmA4NvOHvE+pE53He741Ln6E= 2024-02-01T10:22:13.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:hVdlmA4NvOHvE+pE53He741Ln6E=', 'Content-Length': '0'} 2024-02-01T10:22:13.146 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'xMxAGR4A-xEILepnzFPRgrWNTOLvitz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001934e88f5c4d037c-0065bb70d5-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:13.147 DEBUG:boto:path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:13.147 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:13.147 DEBUG:boto:path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:13.147 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:13.147 DEBUG:boto:Method: HEAD 2024-02-01T10:22:13.147 DEBUG:boto:Path: /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:13.148 DEBUG:boto:Data: 2024-02-01T10:22:13.148 DEBUG:boto:Headers: {} 2024-02-01T10:22:13.148 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:13.148 DEBUG:boto:Port: 8000 2024-02-01T10:22:13.148 DEBUG:boto:Params: {} 2024-02-01T10:22:13.148 DEBUG:boto:Token: None 2024-02-01T10:22:13.149 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:13 GMT /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:13.149 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:hVdlmA4NvOHvE+pE53He741Ln6E= 2024-02-01T10:22:13.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:hVdlmA4NvOHvE+pE53He741Ln6E=', 'Content-Length': '0'} 2024-02-01T10:22:13.152 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'xMxAGR4A-xEILepnzFPRgrWNTOLvitz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008de0016e237e10dd-0065bb70d5-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:13.152 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-02-01T10:22:13.152 DEBUG:boto:path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:13.152 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:13.152 DEBUG:boto:path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:13.152 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:13.153 DEBUG:boto:Method: GET 2024-02-01T10:22:13.153 DEBUG:boto:Path: /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:13.153 DEBUG:boto:Data: 2024-02-01T10:22:13.153 DEBUG:boto:Headers: {} 2024-02-01T10:22:13.153 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:13.153 DEBUG:boto:Port: 8001 2024-02-01T10:22:13.153 DEBUG:boto:Params: {} 2024-02-01T10:22:13.153 DEBUG:boto:Token: None 2024-02-01T10:22:13.153 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:13 GMT /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:13.153 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:hsvjy5qfn0B/PD8Lhv1V8nSoNaU= 2024-02-01T10:22:13.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:hsvjy5qfn0B/PD8Lhv1V8nSoNaU=', 'Content-Length': '0'} 2024-02-01T10:22:13.157 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'xMxAGR4A-xEILepnzFPRgrWNTOLvitz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002080dd0c565dae1a-0065bb70d5-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:13.204 DEBUG:boto:path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:13.204 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:13.204 DEBUG:boto:path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:13.205 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:13.205 DEBUG:boto:Method: GET 2024-02-01T10:22:13.205 DEBUG:boto:Path: /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:13.205 DEBUG:boto:Data: 2024-02-01T10:22:13.205 DEBUG:boto:Headers: {} 2024-02-01T10:22:13.205 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:13.205 DEBUG:boto:Port: 8000 2024-02-01T10:22:13.205 DEBUG:boto:Params: {} 2024-02-01T10:22:13.205 DEBUG:boto:Token: None 2024-02-01T10:22:13.206 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:13 GMT /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:13.206 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:hsvjy5qfn0B/PD8Lhv1V8nSoNaU= 2024-02-01T10:22:13.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:hsvjy5qfn0B/PD8Lhv1V8nSoNaU=', 'Content-Length': '0'} 2024-02-01T10:22:13.209 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'xMxAGR4A-xEILepnzFPRgrWNTOLvitz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007f06edeac3c1d178-0065bb70d5-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:13.209 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:13.210 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:13.210 DEBUG:boto:path=/vclfcn-17/?prefix=obj-test-zone1 2024-02-01T10:22:13.210 DEBUG:boto:auth_path=/vclfcn-17/?prefix=obj-test-zone1 2024-02-01T10:22:13.210 DEBUG:boto:Method: GET 2024-02-01T10:22:13.210 DEBUG:boto:Path: /vclfcn-17/?prefix=obj-test-zone1 2024-02-01T10:22:13.210 DEBUG:boto:Data: 2024-02-01T10:22:13.210 DEBUG:boto:Headers: {} 2024-02-01T10:22:13.210 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:13.210 DEBUG:boto:Port: 8001 2024-02-01T10:22:13.210 DEBUG:boto:Params: {} 2024-02-01T10:22:13.210 DEBUG:boto:Token: None 2024-02-01T10:22:13.211 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:13 GMT /vclfcn-17/ 2024-02-01T10:22:13.211 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZhPBUAcfmyHSSF5tz4MOV80Twsg= 2024-02-01T10:22:13.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZhPBUAcfmyHSSF5tz4MOV80Twsg=', 'Content-Length': '0'} 2024-02-01T10:22:13.215 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e6eeafbce50aac6-0065bb70d5-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:13.256 DEBUG:boto:b'vclfcn-17obj-test-zone11000falseobj-test-zone12024-02-01T10:20:50.473Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:13.257 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:13.257 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:13.257 DEBUG:boto:path=/vclfcn-17/?prefix=obj-test-zone1 2024-02-01T10:22:13.258 DEBUG:boto:auth_path=/vclfcn-17/?prefix=obj-test-zone1 2024-02-01T10:22:13.258 DEBUG:boto:Method: GET 2024-02-01T10:22:13.258 DEBUG:boto:Path: /vclfcn-17/?prefix=obj-test-zone1 2024-02-01T10:22:13.258 DEBUG:boto:Data: 2024-02-01T10:22:13.258 DEBUG:boto:Headers: {} 2024-02-01T10:22:13.259 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:13.259 DEBUG:boto:Port: 8000 2024-02-01T10:22:13.259 DEBUG:boto:Params: {} 2024-02-01T10:22:13.259 DEBUG:boto:Token: None 2024-02-01T10:22:13.260 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:13 GMT /vclfcn-17/ 2024-02-01T10:22:13.260 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZhPBUAcfmyHSSF5tz4MOV80Twsg= 2024-02-01T10:22:13.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZhPBUAcfmyHSSF5tz4MOV80Twsg=', 'Content-Length': '0'} 2024-02-01T10:22:13.265 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008552101f07fada41-0065bb70d5-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:13.265 DEBUG:boto:b'vclfcn-17obj-test-zone11000falseobj-test-zone12024-02-01T10:20:50.473Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:13.266 DEBUG:boto:path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:13.266 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:13.266 DEBUG:boto:Method: HEAD 2024-02-01T10:22:13.266 DEBUG:boto:Path: /vclfcn-17/obj-test-zone1 2024-02-01T10:22:13.266 DEBUG:boto:Data: 2024-02-01T10:22:13.267 DEBUG:boto:Headers: {} 2024-02-01T10:22:13.267 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:13.267 DEBUG:boto:Port: 8001 2024-02-01T10:22:13.267 DEBUG:boto:Params: {} 2024-02-01T10:22:13.267 DEBUG:boto:Token: None 2024-02-01T10:22:13.268 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:13 GMT /vclfcn-17/obj-test-zone1 2024-02-01T10:22:13.268 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7eOg6vlo4Lp/c+4OCZkpjulngF4= 2024-02-01T10:22:13.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7eOg6vlo4Lp/c+4OCZkpjulngF4=', 'Content-Length': '0'} 2024-02-01T10:22:13.272 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'xMxAGR4A-xEILepnzFPRgrWNTOLvitz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f56eebbdb4b62ccf-0065bb70d5-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:13.273 DEBUG:boto:path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:13.273 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:13.273 DEBUG:boto:Method: HEAD 2024-02-01T10:22:13.273 DEBUG:boto:Path: /vclfcn-17/obj-test-zone1 2024-02-01T10:22:13.273 DEBUG:boto:Data: 2024-02-01T10:22:13.273 DEBUG:boto:Headers: {} 2024-02-01T10:22:13.274 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:13.274 DEBUG:boto:Port: 8000 2024-02-01T10:22:13.274 DEBUG:boto:Params: {} 2024-02-01T10:22:13.274 DEBUG:boto:Token: None 2024-02-01T10:22:13.274 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:13 GMT /vclfcn-17/obj-test-zone1 2024-02-01T10:22:13.274 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7eOg6vlo4Lp/c+4OCZkpjulngF4= 2024-02-01T10:22:13.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7eOg6vlo4Lp/c+4OCZkpjulngF4=', 'Content-Length': '0'} 2024-02-01T10:22:13.278 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'xMxAGR4A-xEILepnzFPRgrWNTOLvitz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000380a1a64de6e0cf9-0065bb70d5-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:13.278 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-02-01T10:22:13.278 DEBUG:boto:path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:13.279 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:13.279 DEBUG:boto:path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:13.279 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:13.279 DEBUG:boto:Method: GET 2024-02-01T10:22:13.279 DEBUG:boto:Path: /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:13.279 DEBUG:boto:Data: 2024-02-01T10:22:13.279 DEBUG:boto:Headers: {} 2024-02-01T10:22:13.279 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:13.279 DEBUG:boto:Port: 8001 2024-02-01T10:22:13.279 DEBUG:boto:Params: {} 2024-02-01T10:22:13.280 DEBUG:boto:Token: None 2024-02-01T10:22:13.280 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:13 GMT /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:13.280 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:hsvjy5qfn0B/PD8Lhv1V8nSoNaU= 2024-02-01T10:22:13.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:hsvjy5qfn0B/PD8Lhv1V8nSoNaU=', 'Content-Length': '0'} 2024-02-01T10:22:13.283 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'xMxAGR4A-xEILepnzFPRgrWNTOLvitz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ec66c050ea20cf09-0065bb70d5-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:13.328 DEBUG:boto:path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:13.328 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:13.328 DEBUG:boto:path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:13.328 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:13.329 DEBUG:boto:Method: GET 2024-02-01T10:22:13.329 DEBUG:boto:Path: /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:13.329 DEBUG:boto:Data: 2024-02-01T10:22:13.329 DEBUG:boto:Headers: {} 2024-02-01T10:22:13.329 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:13.329 DEBUG:boto:Port: 8000 2024-02-01T10:22:13.329 DEBUG:boto:Params: {} 2024-02-01T10:22:13.329 DEBUG:boto:Token: None 2024-02-01T10:22:13.329 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:13 GMT /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:13.330 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:hsvjy5qfn0B/PD8Lhv1V8nSoNaU= 2024-02-01T10:22:13.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:hsvjy5qfn0B/PD8Lhv1V8nSoNaU=', 'Content-Length': '0'} 2024-02-01T10:22:13.333 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'xMxAGR4A-xEILepnzFPRgrWNTOLvitz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000136f7e0d2bbb6c57-0065bb70d5-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:13.333 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:13.333 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:13.333 DEBUG:boto:path=/vclfcn-17/?prefix=obj-test-zone1 2024-02-01T10:22:13.333 DEBUG:boto:auth_path=/vclfcn-17/?prefix=obj-test-zone1 2024-02-01T10:22:13.333 DEBUG:boto:Method: GET 2024-02-01T10:22:13.333 DEBUG:boto:Path: /vclfcn-17/?prefix=obj-test-zone1 2024-02-01T10:22:13.333 DEBUG:boto:Data: 2024-02-01T10:22:13.334 DEBUG:boto:Headers: {} 2024-02-01T10:22:13.334 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:13.334 DEBUG:boto:Port: 8001 2024-02-01T10:22:13.334 DEBUG:boto:Params: {} 2024-02-01T10:22:13.334 DEBUG:boto:Token: None 2024-02-01T10:22:13.334 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:13 GMT /vclfcn-17/ 2024-02-01T10:22:13.334 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZhPBUAcfmyHSSF5tz4MOV80Twsg= 2024-02-01T10:22:13.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZhPBUAcfmyHSSF5tz4MOV80Twsg=', 'Content-Length': '0'} 2024-02-01T10:22:13.339 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006061d8ebee4ae1ad-0065bb70d5-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:13.380 DEBUG:boto:b'vclfcn-17obj-test-zone11000falseobj-test-zone12024-02-01T10:20:50.473Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:13.381 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:13.381 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:13.381 DEBUG:boto:path=/vclfcn-17/?prefix=obj-test-zone1 2024-02-01T10:22:13.381 DEBUG:boto:auth_path=/vclfcn-17/?prefix=obj-test-zone1 2024-02-01T10:22:13.381 DEBUG:boto:Method: GET 2024-02-01T10:22:13.381 DEBUG:boto:Path: /vclfcn-17/?prefix=obj-test-zone1 2024-02-01T10:22:13.381 DEBUG:boto:Data: 2024-02-01T10:22:13.381 DEBUG:boto:Headers: {} 2024-02-01T10:22:13.381 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:13.381 DEBUG:boto:Port: 8000 2024-02-01T10:22:13.381 DEBUG:boto:Params: {} 2024-02-01T10:22:13.382 DEBUG:boto:Token: None 2024-02-01T10:22:13.382 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:13 GMT /vclfcn-17/ 2024-02-01T10:22:13.382 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZhPBUAcfmyHSSF5tz4MOV80Twsg= 2024-02-01T10:22:13.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZhPBUAcfmyHSSF5tz4MOV80Twsg=', 'Content-Length': '0'} 2024-02-01T10:22:13.387 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b90d154b2fb09554-0065bb70d5-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:13.387 DEBUG:boto:b'vclfcn-17obj-test-zone11000falseobj-test-zone12024-02-01T10:20:50.473Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:13.387 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-02-01T10:22:13.387 DEBUG:boto:path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:13.387 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:13.388 DEBUG:boto:path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:13.388 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:13.388 DEBUG:boto:Method: GET 2024-02-01T10:22:13.388 DEBUG:boto:Path: /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:13.388 DEBUG:boto:Data: 2024-02-01T10:22:13.388 DEBUG:boto:Headers: {} 2024-02-01T10:22:13.388 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:13.388 DEBUG:boto:Port: 8001 2024-02-01T10:22:13.388 DEBUG:boto:Params: {} 2024-02-01T10:22:13.388 DEBUG:boto:Token: None 2024-02-01T10:22:13.389 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:13 GMT /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:13.389 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YrizJN6l0Mplh6QstnAk9e3vplc= 2024-02-01T10:22:13.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YrizJN6l0Mplh6QstnAk9e3vplc=', 'Content-Length': '0'} 2024-02-01T10:22:13.391 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', '-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bb24080d888c098c-0065bb70d5-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:13.436 DEBUG:boto:path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:13.436 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:13.436 DEBUG:boto:path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:13.437 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:13.437 DEBUG:boto:Method: GET 2024-02-01T10:22:13.437 DEBUG:boto:Path: /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:13.437 DEBUG:boto:Data: 2024-02-01T10:22:13.437 DEBUG:boto:Headers: {} 2024-02-01T10:22:13.437 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:13.437 DEBUG:boto:Port: 8000 2024-02-01T10:22:13.437 DEBUG:boto:Params: {} 2024-02-01T10:22:13.437 DEBUG:boto:Token: None 2024-02-01T10:22:13.437 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:13 GMT /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:13.438 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YrizJN6l0Mplh6QstnAk9e3vplc= 2024-02-01T10:22:13.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YrizJN6l0Mplh6QstnAk9e3vplc=', 'Content-Length': '0'} 2024-02-01T10:22:13.441 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', '-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000028ebe21808e25035-0065bb70d5-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:13.441 DEBUG:boto:path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:13.441 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:13.441 DEBUG:boto:path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:13.441 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:13.442 DEBUG:boto:Method: HEAD 2024-02-01T10:22:13.442 DEBUG:boto:Path: /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:13.442 DEBUG:boto:Data: 2024-02-01T10:22:13.442 DEBUG:boto:Headers: {} 2024-02-01T10:22:13.442 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:13.442 DEBUG:boto:Port: 8001 2024-02-01T10:22:13.442 DEBUG:boto:Params: {} 2024-02-01T10:22:13.442 DEBUG:boto:Token: None 2024-02-01T10:22:13.442 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:13 GMT /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:13.442 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:glp/mW2IqkBXhKiwu/y8qQO0Nz0= 2024-02-01T10:22:13.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:glp/mW2IqkBXhKiwu/y8qQO0Nz0=', 'Content-Length': '0'} 2024-02-01T10:22:13.445 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', '-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000184cd1fcd5a984b5-0065bb70d5-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:13.445 DEBUG:boto:path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:13.446 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:13.446 DEBUG:boto:path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:13.446 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:13.446 DEBUG:boto:Method: HEAD 2024-02-01T10:22:13.446 DEBUG:boto:Path: /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:13.446 DEBUG:boto:Data: 2024-02-01T10:22:13.446 DEBUG:boto:Headers: {} 2024-02-01T10:22:13.446 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:13.446 DEBUG:boto:Port: 8000 2024-02-01T10:22:13.446 DEBUG:boto:Params: {} 2024-02-01T10:22:13.446 DEBUG:boto:Token: None 2024-02-01T10:22:13.447 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:13 GMT /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:13.447 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:glp/mW2IqkBXhKiwu/y8qQO0Nz0= 2024-02-01T10:22:13.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:glp/mW2IqkBXhKiwu/y8qQO0Nz0=', 'Content-Length': '0'} 2024-02-01T10:22:13.450 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', '-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a38df030e8f2f478-0065bb70d5-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:13.450 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-02-01T10:22:13.450 DEBUG:boto:path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:13.450 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:13.450 DEBUG:boto:path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:13.450 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:13.450 DEBUG:boto:Method: GET 2024-02-01T10:22:13.451 DEBUG:boto:Path: /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:13.451 DEBUG:boto:Data: 2024-02-01T10:22:13.451 DEBUG:boto:Headers: {} 2024-02-01T10:22:13.451 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:13.451 DEBUG:boto:Port: 8001 2024-02-01T10:22:13.451 DEBUG:boto:Params: {} 2024-02-01T10:22:13.451 DEBUG:boto:Token: None 2024-02-01T10:22:13.451 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:13 GMT /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:13.451 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YrizJN6l0Mplh6QstnAk9e3vplc= 2024-02-01T10:22:13.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YrizJN6l0Mplh6QstnAk9e3vplc=', 'Content-Length': '0'} 2024-02-01T10:22:13.454 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', '-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ca65265443dd705f-0065bb70d5-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:13.496 DEBUG:boto:path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:13.496 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:13.497 DEBUG:boto:path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:13.497 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:13.497 DEBUG:boto:Method: GET 2024-02-01T10:22:13.497 DEBUG:boto:Path: /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:13.497 DEBUG:boto:Data: 2024-02-01T10:22:13.497 DEBUG:boto:Headers: {} 2024-02-01T10:22:13.497 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:13.497 DEBUG:boto:Port: 8000 2024-02-01T10:22:13.497 DEBUG:boto:Params: {} 2024-02-01T10:22:13.497 DEBUG:boto:Token: None 2024-02-01T10:22:13.498 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:13 GMT /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:13.498 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YrizJN6l0Mplh6QstnAk9e3vplc= 2024-02-01T10:22:13.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YrizJN6l0Mplh6QstnAk9e3vplc=', 'Content-Length': '0'} 2024-02-01T10:22:13.501 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', '-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000336ea33b7b8359c0-0065bb70d5-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:13.502 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:13.502 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:13.502 DEBUG:boto:path=/vclfcn-17/?prefix=obj-test-zone2 2024-02-01T10:22:13.502 DEBUG:boto:auth_path=/vclfcn-17/?prefix=obj-test-zone2 2024-02-01T10:22:13.502 DEBUG:boto:Method: GET 2024-02-01T10:22:13.502 DEBUG:boto:Path: /vclfcn-17/?prefix=obj-test-zone2 2024-02-01T10:22:13.502 DEBUG:boto:Data: 2024-02-01T10:22:13.502 DEBUG:boto:Headers: {} 2024-02-01T10:22:13.502 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:13.502 DEBUG:boto:Port: 8001 2024-02-01T10:22:13.502 DEBUG:boto:Params: {} 2024-02-01T10:22:13.503 DEBUG:boto:Token: None 2024-02-01T10:22:13.503 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:13 GMT /vclfcn-17/ 2024-02-01T10:22:13.503 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZhPBUAcfmyHSSF5tz4MOV80Twsg= 2024-02-01T10:22:13.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZhPBUAcfmyHSSF5tz4MOV80Twsg=', 'Content-Length': '0'} 2024-02-01T10:22:13.508 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e53ce5f06d6a1fe-0065bb70d5-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:13.508 DEBUG:boto:b'vclfcn-17obj-test-zone21000falseobj-test-zone22024-02-01T10:20:50.624Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:13.508 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:13.508 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:13.508 DEBUG:boto:path=/vclfcn-17/?prefix=obj-test-zone2 2024-02-01T10:22:13.508 DEBUG:boto:auth_path=/vclfcn-17/?prefix=obj-test-zone2 2024-02-01T10:22:13.509 DEBUG:boto:Method: GET 2024-02-01T10:22:13.509 DEBUG:boto:Path: /vclfcn-17/?prefix=obj-test-zone2 2024-02-01T10:22:13.509 DEBUG:boto:Data: 2024-02-01T10:22:13.509 DEBUG:boto:Headers: {} 2024-02-01T10:22:13.509 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:13.509 DEBUG:boto:Port: 8000 2024-02-01T10:22:13.509 DEBUG:boto:Params: {} 2024-02-01T10:22:13.509 DEBUG:boto:Token: None 2024-02-01T10:22:13.509 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:13 GMT /vclfcn-17/ 2024-02-01T10:22:13.509 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZhPBUAcfmyHSSF5tz4MOV80Twsg= 2024-02-01T10:22:13.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZhPBUAcfmyHSSF5tz4MOV80Twsg=', 'Content-Length': '0'} 2024-02-01T10:22:13.514 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c4157897ab105a7-0065bb70d5-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:13.556 DEBUG:boto:b'vclfcn-17obj-test-zone21000falseobj-test-zone22024-02-01T10:20:50.624Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:13.557 DEBUG:boto:path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:13.557 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:13.557 DEBUG:boto:Method: HEAD 2024-02-01T10:22:13.557 DEBUG:boto:Path: /vclfcn-17/obj-test-zone2 2024-02-01T10:22:13.557 DEBUG:boto:Data: 2024-02-01T10:22:13.557 DEBUG:boto:Headers: {} 2024-02-01T10:22:13.557 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:13.557 DEBUG:boto:Port: 8001 2024-02-01T10:22:13.557 DEBUG:boto:Params: {} 2024-02-01T10:22:13.557 DEBUG:boto:Token: None 2024-02-01T10:22:13.558 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:13 GMT /vclfcn-17/obj-test-zone2 2024-02-01T10:22:13.558 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:OK9N+iBbbXOnUUeg5wEXMX384sg= 2024-02-01T10:22:13.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:OK9N+iBbbXOnUUeg5wEXMX384sg=', 'Content-Length': '0'} 2024-02-01T10:22:13.561 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', '-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000030aac42e73c13628-0065bb70d5-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:13.562 DEBUG:boto:path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:13.562 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:13.562 DEBUG:boto:Method: HEAD 2024-02-01T10:22:13.562 DEBUG:boto:Path: /vclfcn-17/obj-test-zone2 2024-02-01T10:22:13.562 DEBUG:boto:Data: 2024-02-01T10:22:13.562 DEBUG:boto:Headers: {} 2024-02-01T10:22:13.562 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:13.562 DEBUG:boto:Port: 8000 2024-02-01T10:22:13.562 DEBUG:boto:Params: {} 2024-02-01T10:22:13.563 DEBUG:boto:Token: None 2024-02-01T10:22:13.563 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:13 GMT /vclfcn-17/obj-test-zone2 2024-02-01T10:22:13.563 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:OK9N+iBbbXOnUUeg5wEXMX384sg= 2024-02-01T10:22:13.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:OK9N+iBbbXOnUUeg5wEXMX384sg=', 'Content-Length': '0'} 2024-02-01T10:22:13.567 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', '-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000585ccf4188e3ceaa-0065bb70d5-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:13.567 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-02-01T10:22:13.567 DEBUG:boto:path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:13.567 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:13.567 DEBUG:boto:path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:13.567 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:13.567 DEBUG:boto:Method: GET 2024-02-01T10:22:13.567 DEBUG:boto:Path: /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:13.568 DEBUG:boto:Data: 2024-02-01T10:22:13.568 DEBUG:boto:Headers: {} 2024-02-01T10:22:13.568 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:13.568 DEBUG:boto:Port: 8001 2024-02-01T10:22:13.568 DEBUG:boto:Params: {} 2024-02-01T10:22:13.568 DEBUG:boto:Token: None 2024-02-01T10:22:13.568 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:13 GMT /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:13.568 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YrizJN6l0Mplh6QstnAk9e3vplc= 2024-02-01T10:22:13.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YrizJN6l0Mplh6QstnAk9e3vplc=', 'Content-Length': '0'} 2024-02-01T10:22:13.571 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', '-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f13508cedcda4263-0065bb70d5-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:13.612 DEBUG:boto:path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:13.612 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:13.612 DEBUG:boto:path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:13.612 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:13.613 DEBUG:boto:Method: GET 2024-02-01T10:22:13.613 DEBUG:boto:Path: /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:13.613 DEBUG:boto:Data: 2024-02-01T10:22:13.613 DEBUG:boto:Headers: {} 2024-02-01T10:22:13.613 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:13.613 DEBUG:boto:Port: 8000 2024-02-01T10:22:13.613 DEBUG:boto:Params: {} 2024-02-01T10:22:13.613 DEBUG:boto:Token: None 2024-02-01T10:22:13.613 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:13 GMT /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:13.613 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YrizJN6l0Mplh6QstnAk9e3vplc= 2024-02-01T10:22:13.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YrizJN6l0Mplh6QstnAk9e3vplc=', 'Content-Length': '0'} 2024-02-01T10:22:13.616 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', '-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004f37a1ba5dd88743-0065bb70d5-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:13.617 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:13.617 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:13.617 DEBUG:boto:path=/vclfcn-17/?prefix=obj-test-zone2 2024-02-01T10:22:13.617 DEBUG:boto:auth_path=/vclfcn-17/?prefix=obj-test-zone2 2024-02-01T10:22:13.617 DEBUG:boto:Method: GET 2024-02-01T10:22:13.617 DEBUG:boto:Path: /vclfcn-17/?prefix=obj-test-zone2 2024-02-01T10:22:13.617 DEBUG:boto:Data: 2024-02-01T10:22:13.617 DEBUG:boto:Headers: {} 2024-02-01T10:22:13.617 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:13.617 DEBUG:boto:Port: 8001 2024-02-01T10:22:13.617 DEBUG:boto:Params: {} 2024-02-01T10:22:13.618 DEBUG:boto:Token: None 2024-02-01T10:22:13.618 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:13 GMT /vclfcn-17/ 2024-02-01T10:22:13.618 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZhPBUAcfmyHSSF5tz4MOV80Twsg= 2024-02-01T10:22:13.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZhPBUAcfmyHSSF5tz4MOV80Twsg=', 'Content-Length': '0'} 2024-02-01T10:22:13.622 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009243e997082c2c38-0065bb70d5-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:13.664 DEBUG:boto:b'vclfcn-17obj-test-zone21000falseobj-test-zone22024-02-01T10:20:50.624Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:13.665 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:13.665 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:13.665 DEBUG:boto:path=/vclfcn-17/?prefix=obj-test-zone2 2024-02-01T10:22:13.665 DEBUG:boto:auth_path=/vclfcn-17/?prefix=obj-test-zone2 2024-02-01T10:22:13.665 DEBUG:boto:Method: GET 2024-02-01T10:22:13.665 DEBUG:boto:Path: /vclfcn-17/?prefix=obj-test-zone2 2024-02-01T10:22:13.665 DEBUG:boto:Data: 2024-02-01T10:22:13.665 DEBUG:boto:Headers: {} 2024-02-01T10:22:13.665 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:13.665 DEBUG:boto:Port: 8000 2024-02-01T10:22:13.665 DEBUG:boto:Params: {} 2024-02-01T10:22:13.666 DEBUG:boto:Token: None 2024-02-01T10:22:13.666 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:13 GMT /vclfcn-17/ 2024-02-01T10:22:13.666 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZhPBUAcfmyHSSF5tz4MOV80Twsg= 2024-02-01T10:22:13.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZhPBUAcfmyHSSF5tz4MOV80Twsg=', 'Content-Length': '0'} 2024-02-01T10:22:13.670 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b1bb1ac81ee0569-0065bb70d5-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:13.670 DEBUG:boto:b'vclfcn-17obj-test-zone21000falseobj-test-zone22024-02-01T10:20:50.624Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:13.671 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-02-01T10:22:13.671 DEBUG:boto:path=/vclfcn-17/obj-test-zone3 2024-02-01T10:22:13.671 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3 2024-02-01T10:22:13.671 DEBUG:boto:path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:13.671 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:13.671 DEBUG:boto:Method: GET 2024-02-01T10:22:13.671 DEBUG:boto:Path: /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:13.671 DEBUG:boto:Data: 2024-02-01T10:22:13.672 DEBUG:boto:Headers: {} 2024-02-01T10:22:13.672 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:13.672 DEBUG:boto:Port: 8001 2024-02-01T10:22:13.672 DEBUG:boto:Params: {} 2024-02-01T10:22:13.672 DEBUG:boto:Token: None 2024-02-01T10:22:13.672 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:13 GMT /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:13.672 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1BoVtwFIgoH79MSVVAO6N2yS/Jc= 2024-02-01T10:22:13.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1BoVtwFIgoH79MSVVAO6N2yS/Jc=', 'Content-Length': '0'} 2024-02-01T10:22:13.676 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'YE736OwOOXbZJHJe7d8YvcNV1LxwcQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fa26cba5a970a2b6-0065bb70d5-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:13.676 DEBUG:boto:path=/vclfcn-17/obj-test-zone3 2024-02-01T10:22:13.676 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3 2024-02-01T10:22:13.676 DEBUG:boto:path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:13.676 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:13.676 DEBUG:boto:Method: GET 2024-02-01T10:22:13.676 DEBUG:boto:Path: /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:13.676 DEBUG:boto:Data: 2024-02-01T10:22:13.677 DEBUG:boto:Headers: {} 2024-02-01T10:22:13.677 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:13.677 DEBUG:boto:Port: 8000 2024-02-01T10:22:13.677 DEBUG:boto:Params: {} 2024-02-01T10:22:13.677 DEBUG:boto:Token: None 2024-02-01T10:22:13.677 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:13 GMT /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:13.677 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1BoVtwFIgoH79MSVVAO6N2yS/Jc= 2024-02-01T10:22:13.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1BoVtwFIgoH79MSVVAO6N2yS/Jc=', 'Content-Length': '0'} 2024-02-01T10:22:13.680 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'YE736OwOOXbZJHJe7d8YvcNV1LxwcQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000844ef22aa4c0130a-0065bb70d5-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:13.680 DEBUG:boto:path=/vclfcn-17/obj-test-zone3 2024-02-01T10:22:13.680 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3 2024-02-01T10:22:13.681 DEBUG:boto:path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:13.681 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:13.681 DEBUG:boto:Method: HEAD 2024-02-01T10:22:13.681 DEBUG:boto:Path: /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:13.681 DEBUG:boto:Data: 2024-02-01T10:22:13.681 DEBUG:boto:Headers: {} 2024-02-01T10:22:13.681 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:13.681 DEBUG:boto:Port: 8001 2024-02-01T10:22:13.681 DEBUG:boto:Params: {} 2024-02-01T10:22:13.681 DEBUG:boto:Token: None 2024-02-01T10:22:13.681 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:13 GMT /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:13.681 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:J58dt7ZteiNFyZhdHtByS0SCL5o= 2024-02-01T10:22:13.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:J58dt7ZteiNFyZhdHtByS0SCL5o=', 'Content-Length': '0'} 2024-02-01T10:22:13.684 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'YE736OwOOXbZJHJe7d8YvcNV1LxwcQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000827bb1fd03d83b84-0065bb70d5-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:13.685 DEBUG:boto:path=/vclfcn-17/obj-test-zone3 2024-02-01T10:22:13.685 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3 2024-02-01T10:22:13.685 DEBUG:boto:path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:13.685 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:13.685 DEBUG:boto:Method: HEAD 2024-02-01T10:22:13.685 DEBUG:boto:Path: /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:13.685 DEBUG:boto:Data: 2024-02-01T10:22:13.685 DEBUG:boto:Headers: {} 2024-02-01T10:22:13.685 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:13.686 DEBUG:boto:Port: 8000 2024-02-01T10:22:13.686 DEBUG:boto:Params: {} 2024-02-01T10:22:13.686 DEBUG:boto:Token: None 2024-02-01T10:22:13.686 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:13 GMT /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:13.686 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:J58dt7ZteiNFyZhdHtByS0SCL5o= 2024-02-01T10:22:13.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:J58dt7ZteiNFyZhdHtByS0SCL5o=', 'Content-Length': '0'} 2024-02-01T10:22:13.691 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'YE736OwOOXbZJHJe7d8YvcNV1LxwcQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000861ebc297f2df59c-0065bb70d5-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:13.692 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-02-01T10:22:13.692 DEBUG:boto:path=/vclfcn-17/obj-test-zone3 2024-02-01T10:22:13.692 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3 2024-02-01T10:22:13.692 DEBUG:boto:path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:13.692 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:13.692 DEBUG:boto:Method: GET 2024-02-01T10:22:13.692 DEBUG:boto:Path: /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:13.692 DEBUG:boto:Data: 2024-02-01T10:22:13.692 DEBUG:boto:Headers: {} 2024-02-01T10:22:13.693 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:13.693 DEBUG:boto:Port: 8001 2024-02-01T10:22:13.693 DEBUG:boto:Params: {} 2024-02-01T10:22:13.693 DEBUG:boto:Token: None 2024-02-01T10:22:13.693 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:13 GMT /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:13.693 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1BoVtwFIgoH79MSVVAO6N2yS/Jc= 2024-02-01T10:22:13.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1BoVtwFIgoH79MSVVAO6N2yS/Jc=', 'Content-Length': '0'} 2024-02-01T10:22:13.696 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'YE736OwOOXbZJHJe7d8YvcNV1LxwcQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a21ac7b80ff396a7-0065bb70d5-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:13.696 DEBUG:boto:path=/vclfcn-17/obj-test-zone3 2024-02-01T10:22:13.697 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3 2024-02-01T10:22:13.697 DEBUG:boto:path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:13.697 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:13.697 DEBUG:boto:Method: GET 2024-02-01T10:22:13.697 DEBUG:boto:Path: /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:13.697 DEBUG:boto:Data: 2024-02-01T10:22:13.697 DEBUG:boto:Headers: {} 2024-02-01T10:22:13.697 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:13.697 DEBUG:boto:Port: 8000 2024-02-01T10:22:13.697 DEBUG:boto:Params: {} 2024-02-01T10:22:13.697 DEBUG:boto:Token: None 2024-02-01T10:22:13.698 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:13 GMT /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:13.698 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1BoVtwFIgoH79MSVVAO6N2yS/Jc= 2024-02-01T10:22:13.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1BoVtwFIgoH79MSVVAO6N2yS/Jc=', 'Content-Length': '0'} 2024-02-01T10:22:13.700 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'YE736OwOOXbZJHJe7d8YvcNV1LxwcQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ea74da4e1180c72c-0065bb70d5-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:13.744 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:13.744 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:13.744 DEBUG:boto:path=/vclfcn-17/?prefix=obj-test-zone3 2024-02-01T10:22:13.744 DEBUG:boto:auth_path=/vclfcn-17/?prefix=obj-test-zone3 2024-02-01T10:22:13.745 DEBUG:boto:Method: GET 2024-02-01T10:22:13.745 DEBUG:boto:Path: /vclfcn-17/?prefix=obj-test-zone3 2024-02-01T10:22:13.745 DEBUG:boto:Data: 2024-02-01T10:22:13.745 DEBUG:boto:Headers: {} 2024-02-01T10:22:13.745 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:13.745 DEBUG:boto:Port: 8001 2024-02-01T10:22:13.745 DEBUG:boto:Params: {} 2024-02-01T10:22:13.745 DEBUG:boto:Token: None 2024-02-01T10:22:13.745 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:13 GMT /vclfcn-17/ 2024-02-01T10:22:13.746 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZhPBUAcfmyHSSF5tz4MOV80Twsg= 2024-02-01T10:22:13.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZhPBUAcfmyHSSF5tz4MOV80Twsg=', 'Content-Length': '0'} 2024-02-01T10:22:13.752 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000081f050eb88e2bab-0065bb70d5-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:13.752 DEBUG:boto:b'vclfcn-17obj-test-zone31000falseobj-test-zone32024-02-01T10:20:50.780Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:13.753 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:13.753 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:13.753 DEBUG:boto:path=/vclfcn-17/?prefix=obj-test-zone3 2024-02-01T10:22:13.753 DEBUG:boto:auth_path=/vclfcn-17/?prefix=obj-test-zone3 2024-02-01T10:22:13.754 DEBUG:boto:Method: GET 2024-02-01T10:22:13.754 DEBUG:boto:Path: /vclfcn-17/?prefix=obj-test-zone3 2024-02-01T10:22:13.754 DEBUG:boto:Data: 2024-02-01T10:22:13.754 DEBUG:boto:Headers: {} 2024-02-01T10:22:13.754 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:13.754 DEBUG:boto:Port: 8000 2024-02-01T10:22:13.754 DEBUG:boto:Params: {} 2024-02-01T10:22:13.754 DEBUG:boto:Token: None 2024-02-01T10:22:13.754 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:13 GMT /vclfcn-17/ 2024-02-01T10:22:13.754 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZhPBUAcfmyHSSF5tz4MOV80Twsg= 2024-02-01T10:22:13.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZhPBUAcfmyHSSF5tz4MOV80Twsg=', 'Content-Length': '0'} 2024-02-01T10:22:13.759 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fcc94eb3157a6fd1-0065bb70d5-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:13.759 DEBUG:boto:b'vclfcn-17obj-test-zone31000falseobj-test-zone32024-02-01T10:20:50.780Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:13.759 DEBUG:boto:path=/vclfcn-17/obj-test-zone3 2024-02-01T10:22:13.760 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3 2024-02-01T10:22:13.760 DEBUG:boto:Method: HEAD 2024-02-01T10:22:13.760 DEBUG:boto:Path: /vclfcn-17/obj-test-zone3 2024-02-01T10:22:13.760 DEBUG:boto:Data: 2024-02-01T10:22:13.760 DEBUG:boto:Headers: {} 2024-02-01T10:22:13.760 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:13.760 DEBUG:boto:Port: 8001 2024-02-01T10:22:13.760 DEBUG:boto:Params: {} 2024-02-01T10:22:13.760 DEBUG:boto:Token: None 2024-02-01T10:22:13.761 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:13 GMT /vclfcn-17/obj-test-zone3 2024-02-01T10:22:13.761 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:E52rHC9PbWu6wSCg46JB6z91FdY= 2024-02-01T10:22:13.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:E52rHC9PbWu6wSCg46JB6z91FdY=', 'Content-Length': '0'} 2024-02-01T10:22:13.764 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'YE736OwOOXbZJHJe7d8YvcNV1LxwcQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000776c98ed97a77fe4-0065bb70d5-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:13.765 DEBUG:boto:path=/vclfcn-17/obj-test-zone3 2024-02-01T10:22:13.765 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3 2024-02-01T10:22:13.765 DEBUG:boto:Method: HEAD 2024-02-01T10:22:13.765 DEBUG:boto:Path: /vclfcn-17/obj-test-zone3 2024-02-01T10:22:13.765 DEBUG:boto:Data: 2024-02-01T10:22:13.765 DEBUG:boto:Headers: {} 2024-02-01T10:22:13.765 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:13.765 DEBUG:boto:Port: 8000 2024-02-01T10:22:13.765 DEBUG:boto:Params: {} 2024-02-01T10:22:13.765 DEBUG:boto:Token: None 2024-02-01T10:22:13.766 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:13 GMT /vclfcn-17/obj-test-zone3 2024-02-01T10:22:13.766 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:E52rHC9PbWu6wSCg46JB6z91FdY= 2024-02-01T10:22:13.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:E52rHC9PbWu6wSCg46JB6z91FdY=', 'Content-Length': '0'} 2024-02-01T10:22:13.769 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'YE736OwOOXbZJHJe7d8YvcNV1LxwcQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a6940312645dd141-0065bb70d5-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:13.770 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-02-01T10:22:13.770 DEBUG:boto:path=/vclfcn-17/obj-test-zone3 2024-02-01T10:22:13.770 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3 2024-02-01T10:22:13.770 DEBUG:boto:path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:13.770 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:13.770 DEBUG:boto:Method: GET 2024-02-01T10:22:13.770 DEBUG:boto:Path: /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:13.770 DEBUG:boto:Data: 2024-02-01T10:22:13.770 DEBUG:boto:Headers: {} 2024-02-01T10:22:13.770 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:13.771 DEBUG:boto:Port: 8001 2024-02-01T10:22:13.771 DEBUG:boto:Params: {} 2024-02-01T10:22:13.771 DEBUG:boto:Token: None 2024-02-01T10:22:13.771 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:13 GMT /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:13.771 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1BoVtwFIgoH79MSVVAO6N2yS/Jc= 2024-02-01T10:22:13.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1BoVtwFIgoH79MSVVAO6N2yS/Jc=', 'Content-Length': '0'} 2024-02-01T10:22:13.774 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'YE736OwOOXbZJHJe7d8YvcNV1LxwcQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004932e7f6b56b826e-0065bb70d5-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:13.816 DEBUG:boto:path=/vclfcn-17/obj-test-zone3 2024-02-01T10:22:13.816 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3 2024-02-01T10:22:13.816 DEBUG:boto:path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:13.816 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:13.817 DEBUG:boto:Method: GET 2024-02-01T10:22:13.817 DEBUG:boto:Path: /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:13.817 DEBUG:boto:Data: 2024-02-01T10:22:13.817 DEBUG:boto:Headers: {} 2024-02-01T10:22:13.817 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:13.817 DEBUG:boto:Port: 8000 2024-02-01T10:22:13.817 DEBUG:boto:Params: {} 2024-02-01T10:22:13.817 DEBUG:boto:Token: None 2024-02-01T10:22:13.817 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:13 GMT /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:13.818 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1BoVtwFIgoH79MSVVAO6N2yS/Jc= 2024-02-01T10:22:13.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1BoVtwFIgoH79MSVVAO6N2yS/Jc=', 'Content-Length': '0'} 2024-02-01T10:22:13.821 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'YE736OwOOXbZJHJe7d8YvcNV1LxwcQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b2abd5c914146b8f-0065bb70d5-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:13.821 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:13.821 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:13.821 DEBUG:boto:path=/vclfcn-17/?prefix=obj-test-zone3 2024-02-01T10:22:13.821 DEBUG:boto:auth_path=/vclfcn-17/?prefix=obj-test-zone3 2024-02-01T10:22:13.821 DEBUG:boto:Method: GET 2024-02-01T10:22:13.821 DEBUG:boto:Path: /vclfcn-17/?prefix=obj-test-zone3 2024-02-01T10:22:13.821 DEBUG:boto:Data: 2024-02-01T10:22:13.822 DEBUG:boto:Headers: {} 2024-02-01T10:22:13.822 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:13.822 DEBUG:boto:Port: 8001 2024-02-01T10:22:13.822 DEBUG:boto:Params: {} 2024-02-01T10:22:13.822 DEBUG:boto:Token: None 2024-02-01T10:22:13.822 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:13 GMT /vclfcn-17/ 2024-02-01T10:22:13.822 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZhPBUAcfmyHSSF5tz4MOV80Twsg= 2024-02-01T10:22:13.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZhPBUAcfmyHSSF5tz4MOV80Twsg=', 'Content-Length': '0'} 2024-02-01T10:22:13.827 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aaed7f5470d1c871-0065bb70d5-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:13.827 DEBUG:boto:b'vclfcn-17obj-test-zone31000falseobj-test-zone32024-02-01T10:20:50.780Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:13.827 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:13.828 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:13.828 DEBUG:boto:path=/vclfcn-17/?prefix=obj-test-zone3 2024-02-01T10:22:13.828 DEBUG:boto:auth_path=/vclfcn-17/?prefix=obj-test-zone3 2024-02-01T10:22:13.828 DEBUG:boto:Method: GET 2024-02-01T10:22:13.828 DEBUG:boto:Path: /vclfcn-17/?prefix=obj-test-zone3 2024-02-01T10:22:13.828 DEBUG:boto:Data: 2024-02-01T10:22:13.828 DEBUG:boto:Headers: {} 2024-02-01T10:22:13.828 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:13.828 DEBUG:boto:Port: 8000 2024-02-01T10:22:13.828 DEBUG:boto:Params: {} 2024-02-01T10:22:13.829 DEBUG:boto:Token: None 2024-02-01T10:22:13.829 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:13 GMT /vclfcn-17/ 2024-02-01T10:22:13.829 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZhPBUAcfmyHSSF5tz4MOV80Twsg= 2024-02-01T10:22:13.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZhPBUAcfmyHSSF5tz4MOV80Twsg=', 'Content-Length': '0'} 2024-02-01T10:22:13.833 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037c47a29a94f8aa4-0065bb70d5-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:13.876 DEBUG:boto:b'vclfcn-17obj-test-zone31000falseobj-test-zone32024-02-01T10:20:50.780Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:13.876 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-17 zones={test-zone3, test-zone2} 2024-02-01T10:22:13.877 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=vclfcn-18 2024-02-01T10:22:13.877 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:22:13.877 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:22:13.877 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket vclfcn-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-02-01T10:22:13.927 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:22:13.928 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:22:13.990 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:22:13.988+0000 7fc416ac4980 1 bucket sync caught up with source: 2024-02-01T10:22:13.990 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, 00000000010.64.6, , 00000000005.23.10, , 00000000008.32.10, , , , , ] 2024-02-01T10:22:13.991 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, 00000000010.64.6, , 00000000005.23.10, , 00000000008.32.10, , , , , ] 2024-02-01T10:22:13.991 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:22:13.988+0000 7fc416ac4980 0 bucket checkpoint complete 2024-02-01T10:22:14.013 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=vclfcn-18 2024-02-01T10:22:14.013 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:22:14.013 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:22:14.013 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:22:14.063 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:22:14.063 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:22:14.123 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:22:14.124+0000 7fd60d356980 1 bucket sync caught up with source: 2024-02-01T10:22:14.123 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, 00000000010.64.6, , 00000000005.23.10, , 00000000008.32.10, , , , , ] 2024-02-01T10:22:14.123 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, 00000000010.64.6, , 00000000005.23.10, , 00000000008.32.10, , , , , ] 2024-02-01T10:22:14.123 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:22:14.124+0000 7fd60d356980 0 bucket checkpoint complete 2024-02-01T10:22:14.145 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=vclfcn-18 2024-02-01T10:22:14.145 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:22:14.145 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:22:14.145 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:22:14.185 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:22:14.186 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:22:14.248 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:22:14.252+0000 7f1c3e6b0980 1 bucket sync caught up with source: 2024-02-01T10:22:14.249 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, 00000000003.52.10, , 00000000010.43.6, , 00000000008.34.10, , , , , ] 2024-02-01T10:22:14.249 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, 00000000003.52.10, , 00000000010.43.6, , 00000000008.34.10, , , , , ] 2024-02-01T10:22:14.249 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:22:14.252+0000 7f1c3e6b0980 0 bucket checkpoint complete 2024-02-01T10:22:14.271 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=vclfcn-18 2024-02-01T10:22:14.271 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:22:14.271 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:22:14.272 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:22:14.359 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:22:14.359 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:22:14.421 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:22:14.424+0000 7f5a56291980 1 bucket sync caught up with source: 2024-02-01T10:22:14.421 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, 00000000003.52.10, , 00000000010.43.6, , 00000000008.34.10, , , , , ] 2024-02-01T10:22:14.421 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, 00000000003.52.10, , 00000000010.43.6, , 00000000008.34.10, , , , , ] 2024-02-01T10:22:14.422 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:22:14.424+0000 7f5a56291980 0 bucket checkpoint complete 2024-02-01T10:22:14.443 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=vclfcn-18 2024-02-01T10:22:14.444 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:22:14.444 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:22:14.444 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:22:14.485 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:22:14.485 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:22:14.542 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:22:14.544+0000 7fa7623a1980 1 bucket sync caught up with source: 2024-02-01T10:22:14.542 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, 00000000003.140.10, , 00000000005.62.10, , 00000000015.94.10, , , , , ] 2024-02-01T10:22:14.542 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, 00000000003.140.10, , 00000000005.62.10, , 00000000015.94.10, , , , , ] 2024-02-01T10:22:14.542 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:22:14.544+0000 7fa7623a1980 0 bucket checkpoint complete 2024-02-01T10:22:14.563 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=vclfcn-18 2024-02-01T10:22:14.564 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:22:14.564 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:22:14.564 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket vclfcn-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-02-01T10:22:14.614 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:22:14.615 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:22:14.680 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:22:14.676+0000 7f777591f980 1 bucket sync caught up with source: 2024-02-01T10:22:14.680 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, 00000000003.140.10, , 00000000005.62.10, , 00000000015.94.10, , , , , ] 2024-02-01T10:22:14.680 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, 00000000003.140.10, , 00000000005.62.10, , 00000000015.94.10, , , , , ] 2024-02-01T10:22:14.680 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:22:14.676+0000 7f777591f980 0 bucket checkpoint complete 2024-02-01T10:22:14.703 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-18 zones={test-zone2, test-zone1} 2024-02-01T10:22:14.703 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:22:14.703 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:22:14.703 DEBUG:boto:Method: HEAD 2024-02-01T10:22:14.703 DEBUG:boto:Path: /vclfcn-18/ 2024-02-01T10:22:14.703 DEBUG:boto:Data: 2024-02-01T10:22:14.703 DEBUG:boto:Headers: {} 2024-02-01T10:22:14.703 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:14.703 DEBUG:boto:Port: 8000 2024-02-01T10:22:14.704 DEBUG:boto:Params: {} 2024-02-01T10:22:14.704 DEBUG:boto:Token: None 2024-02-01T10:22:14.704 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:14 GMT /vclfcn-18/ 2024-02-01T10:22:14.704 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NB3/HSblvfkKdbvjkhm4bdyDJx4= 2024-02-01T10:22:14.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NB3/HSblvfkKdbvjkhm4bdyDJx4=', 'Content-Length': '0'} 2024-02-01T10:22:14.706 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fc19de5b1c46edad-0065bb70d6-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:22:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:14.707 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:22:14.707 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:22:14.707 DEBUG:boto:Method: HEAD 2024-02-01T10:22:14.707 DEBUG:boto:Path: /vclfcn-18/ 2024-02-01T10:22:14.707 DEBUG:boto:Data: 2024-02-01T10:22:14.707 DEBUG:boto:Headers: {} 2024-02-01T10:22:14.707 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:14.707 DEBUG:boto:Port: 8000 2024-02-01T10:22:14.707 DEBUG:boto:Params: {} 2024-02-01T10:22:14.708 DEBUG:boto:Token: None 2024-02-01T10:22:14.708 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:14 GMT /vclfcn-18/ 2024-02-01T10:22:14.708 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NB3/HSblvfkKdbvjkhm4bdyDJx4= 2024-02-01T10:22:14.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NB3/HSblvfkKdbvjkhm4bdyDJx4=', 'Content-Length': '0'} 2024-02-01T10:22:14.711 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f558e1ba67377d35-0065bb70d6-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:22:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:14.712 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:22:14.712 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:22:14.712 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:22:14.712 DEBUG:boto:path=/vclfcn-18/?versions&max-keys=999 2024-02-01T10:22:14.712 DEBUG:boto:auth_path=/vclfcn-18/?versions&max-keys=999 2024-02-01T10:22:14.712 DEBUG:boto:Method: GET 2024-02-01T10:22:14.712 DEBUG:boto:Path: /vclfcn-18/?versions&max-keys=999 2024-02-01T10:22:14.712 DEBUG:boto:Data: 2024-02-01T10:22:14.712 DEBUG:boto:Headers: {} 2024-02-01T10:22:14.713 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:14.713 DEBUG:boto:Port: 8000 2024-02-01T10:22:14.713 DEBUG:boto:Params: {} 2024-02-01T10:22:14.713 DEBUG:boto:Token: None 2024-02-01T10:22:14.713 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:14 GMT /vclfcn-18/?versions 2024-02-01T10:22:14.713 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FV9mA0iqsS0HwGAUHrqGHSRssDE= 2024-02-01T10:22:14.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FV9mA0iqsS0HwGAUHrqGHSRssDE=', 'Content-Length': '0'} 2024-02-01T10:22:14.715 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000422c777be20006ab-0065bb70d6-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:14.715 DEBUG:boto:b'vclfcn-18999falsedummytxgLlJWZVbWbjqLWqkOtt87dUoD4zuntrue2024-02-01T10:20:34.645Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcKtrue2024-02-01T10:20:50.942Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutDtrue2024-02-01T10:20:51.094Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone36pAVLbw3HAwE0jxxbuxTIxwcA2p4q8Ntrue2024-02-01T10:20:51.248Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:14.716 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-02-01T10:22:14.716 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-02-01T10:22:14.716 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-02-01T10:22:14.716 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-02-01T10:22:14.716 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:22:14.717 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:22:14.717 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:22:14.717 DEBUG:boto:path=/vclfcn-18/?versions&max-keys=999 2024-02-01T10:22:14.717 DEBUG:boto:auth_path=/vclfcn-18/?versions&max-keys=999 2024-02-01T10:22:14.717 DEBUG:boto:Method: GET 2024-02-01T10:22:14.717 DEBUG:boto:Path: /vclfcn-18/?versions&max-keys=999 2024-02-01T10:22:14.717 DEBUG:boto:Data: 2024-02-01T10:22:14.717 DEBUG:boto:Headers: {} 2024-02-01T10:22:14.717 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:14.717 DEBUG:boto:Port: 8000 2024-02-01T10:22:14.717 DEBUG:boto:Params: {} 2024-02-01T10:22:14.718 DEBUG:boto:Token: None 2024-02-01T10:22:14.718 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:14 GMT /vclfcn-18/?versions 2024-02-01T10:22:14.718 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FV9mA0iqsS0HwGAUHrqGHSRssDE= 2024-02-01T10:22:14.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FV9mA0iqsS0HwGAUHrqGHSRssDE=', 'Content-Length': '0'} 2024-02-01T10:22:14.722 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad88426ae362b31e-0065bb70d6-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:14.722 DEBUG:boto:b'vclfcn-18999falsedummytxgLlJWZVbWbjqLWqkOtt87dUoD4zuntrue2024-02-01T10:20:34.645Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcKtrue2024-02-01T10:20:50.942Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutDtrue2024-02-01T10:20:51.094Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone36pAVLbw3HAwE0jxxbuxTIxwcA2p4q8Ntrue2024-02-01T10:20:51.248Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:14.723 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-02-01T10:22:14.723 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-02-01T10:22:14.723 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-02-01T10:22:14.723 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-02-01T10:22:14.723 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:22:14.723 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:22:14.723 DEBUG:boto:path=/vclfcn-18/?versions&max-keys=999 2024-02-01T10:22:14.723 DEBUG:boto:auth_path=/vclfcn-18/?versions&max-keys=999 2024-02-01T10:22:14.723 DEBUG:boto:Method: GET 2024-02-01T10:22:14.723 DEBUG:boto:Path: /vclfcn-18/?versions&max-keys=999 2024-02-01T10:22:14.723 DEBUG:boto:Data: 2024-02-01T10:22:14.724 DEBUG:boto:Headers: {} 2024-02-01T10:22:14.724 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:14.724 DEBUG:boto:Port: 8000 2024-02-01T10:22:14.724 DEBUG:boto:Params: {} 2024-02-01T10:22:14.724 DEBUG:boto:Token: None 2024-02-01T10:22:14.724 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:14 GMT /vclfcn-18/?versions 2024-02-01T10:22:14.724 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FV9mA0iqsS0HwGAUHrqGHSRssDE= 2024-02-01T10:22:14.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FV9mA0iqsS0HwGAUHrqGHSRssDE=', 'Content-Length': '0'} 2024-02-01T10:22:14.727 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f5fa4436ab43ab58-0065bb70d6-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:14.727 DEBUG:boto:b'vclfcn-18999falsedummytxgLlJWZVbWbjqLWqkOtt87dUoD4zuntrue2024-02-01T10:20:34.645Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcKtrue2024-02-01T10:20:50.942Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutDtrue2024-02-01T10:20:51.094Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone36pAVLbw3HAwE0jxxbuxTIxwcA2p4q8Ntrue2024-02-01T10:20:51.248Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:14.727 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:22:14.728 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:22:14.728 DEBUG:boto:path=/vclfcn-18/?versions&max-keys=999 2024-02-01T10:22:14.728 DEBUG:boto:auth_path=/vclfcn-18/?versions&max-keys=999 2024-02-01T10:22:14.728 DEBUG:boto:Method: GET 2024-02-01T10:22:14.728 DEBUG:boto:Path: /vclfcn-18/?versions&max-keys=999 2024-02-01T10:22:14.728 DEBUG:boto:Data: 2024-02-01T10:22:14.728 DEBUG:boto:Headers: {} 2024-02-01T10:22:14.728 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:14.728 DEBUG:boto:Port: 8000 2024-02-01T10:22:14.728 DEBUG:boto:Params: {} 2024-02-01T10:22:14.728 DEBUG:boto:Token: None 2024-02-01T10:22:14.729 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:14 GMT /vclfcn-18/?versions 2024-02-01T10:22:14.729 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FV9mA0iqsS0HwGAUHrqGHSRssDE= 2024-02-01T10:22:14.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FV9mA0iqsS0HwGAUHrqGHSRssDE=', 'Content-Length': '0'} 2024-02-01T10:22:14.733 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a538687a1fc757ed-0065bb70d6-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:14.734 DEBUG:boto:b'vclfcn-18999falsedummytxgLlJWZVbWbjqLWqkOtt87dUoD4zuntrue2024-02-01T10:20:34.645Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcKtrue2024-02-01T10:20:50.942Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutDtrue2024-02-01T10:20:51.094Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone36pAVLbw3HAwE0jxxbuxTIxwcA2p4q8Ntrue2024-02-01T10:20:51.248Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:14.734 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-02-01T10:22:14.734 DEBUG:boto:path=/vclfcn-18/dummy 2024-02-01T10:22:14.735 DEBUG:boto:auth_path=/vclfcn-18/dummy 2024-02-01T10:22:14.735 DEBUG:boto:path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:14.735 DEBUG:boto:auth_path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:14.735 DEBUG:boto:Method: GET 2024-02-01T10:22:14.735 DEBUG:boto:Path: /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:14.735 DEBUG:boto:Data: 2024-02-01T10:22:14.735 DEBUG:boto:Headers: {} 2024-02-01T10:22:14.735 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:14.735 DEBUG:boto:Port: 8000 2024-02-01T10:22:14.735 DEBUG:boto:Params: {} 2024-02-01T10:22:14.735 DEBUG:boto:Token: None 2024-02-01T10:22:14.736 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:14 GMT /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:14.736 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nx/Hk+go6jJfbtNkK5zwZ4xhMS4= 2024-02-01T10:22:14.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nx/Hk+go6jJfbtNkK5zwZ4xhMS4=', 'Content-Length': '0'} 2024-02-01T10:22:14.738 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:34 GMT'), ('x-amz-version-id', 'txgLlJWZVbWbjqLWqkOtt87dUoD4zun'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000019cefabc60234c1c-0065bb70d6-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:14.738 DEBUG:boto:path=/vclfcn-18/dummy 2024-02-01T10:22:14.738 DEBUG:boto:auth_path=/vclfcn-18/dummy 2024-02-01T10:22:14.738 DEBUG:boto:path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:14.738 DEBUG:boto:auth_path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:14.738 DEBUG:boto:Method: GET 2024-02-01T10:22:14.739 DEBUG:boto:Path: /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:14.739 DEBUG:boto:Data: 2024-02-01T10:22:14.739 DEBUG:boto:Headers: {} 2024-02-01T10:22:14.739 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:14.739 DEBUG:boto:Port: 8000 2024-02-01T10:22:14.739 DEBUG:boto:Params: {} 2024-02-01T10:22:14.739 DEBUG:boto:Token: None 2024-02-01T10:22:14.739 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:14 GMT /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:14.739 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nx/Hk+go6jJfbtNkK5zwZ4xhMS4= 2024-02-01T10:22:14.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nx/Hk+go6jJfbtNkK5zwZ4xhMS4=', 'Content-Length': '0'} 2024-02-01T10:22:14.742 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:34 GMT'), ('x-amz-version-id', 'txgLlJWZVbWbjqLWqkOtt87dUoD4zun'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000053ca430c38c823a8-0065bb70d6-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:14.742 DEBUG:boto:path=/vclfcn-18/dummy 2024-02-01T10:22:14.742 DEBUG:boto:auth_path=/vclfcn-18/dummy 2024-02-01T10:22:14.743 DEBUG:boto:path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:14.743 DEBUG:boto:auth_path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:14.743 DEBUG:boto:Method: HEAD 2024-02-01T10:22:14.743 DEBUG:boto:Path: /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:14.743 DEBUG:boto:Data: 2024-02-01T10:22:14.743 DEBUG:boto:Headers: {} 2024-02-01T10:22:14.743 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:14.743 DEBUG:boto:Port: 8000 2024-02-01T10:22:14.743 DEBUG:boto:Params: {} 2024-02-01T10:22:14.743 DEBUG:boto:Token: None 2024-02-01T10:22:14.744 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:14 GMT /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:14.744 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:15y4GZUZPW/Ep6+Bbnzj2yO68CY= 2024-02-01T10:22:14.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:15y4GZUZPW/Ep6+Bbnzj2yO68CY=', 'Content-Length': '0'} 2024-02-01T10:22:14.746 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:34 GMT'), ('x-amz-version-id', 'txgLlJWZVbWbjqLWqkOtt87dUoD4zun'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000832d7bfb319999a6-0065bb70d6-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:14.746 DEBUG:boto:path=/vclfcn-18/dummy 2024-02-01T10:22:14.746 DEBUG:boto:auth_path=/vclfcn-18/dummy 2024-02-01T10:22:14.746 DEBUG:boto:path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:14.746 DEBUG:boto:auth_path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:14.746 DEBUG:boto:Method: HEAD 2024-02-01T10:22:14.746 DEBUG:boto:Path: /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:14.746 DEBUG:boto:Data: 2024-02-01T10:22:14.747 DEBUG:boto:Headers: {} 2024-02-01T10:22:14.747 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:14.747 DEBUG:boto:Port: 8000 2024-02-01T10:22:14.747 DEBUG:boto:Params: {} 2024-02-01T10:22:14.747 DEBUG:boto:Token: None 2024-02-01T10:22:14.747 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:14 GMT /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:14.747 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:15y4GZUZPW/Ep6+Bbnzj2yO68CY= 2024-02-01T10:22:14.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:15y4GZUZPW/Ep6+Bbnzj2yO68CY=', 'Content-Length': '0'} 2024-02-01T10:22:14.750 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:34 GMT'), ('x-amz-version-id', 'txgLlJWZVbWbjqLWqkOtt87dUoD4zun'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008087f35638d3a83d-0065bb70d6-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:14.751 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-02-01T10:22:14.751 DEBUG:boto:path=/vclfcn-18/dummy 2024-02-01T10:22:14.751 DEBUG:boto:auth_path=/vclfcn-18/dummy 2024-02-01T10:22:14.751 DEBUG:boto:path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:14.751 DEBUG:boto:auth_path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:14.751 DEBUG:boto:Method: GET 2024-02-01T10:22:14.751 DEBUG:boto:Path: /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:14.752 DEBUG:boto:Data: 2024-02-01T10:22:14.752 DEBUG:boto:Headers: {} 2024-02-01T10:22:14.752 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:14.752 DEBUG:boto:Port: 8000 2024-02-01T10:22:14.752 DEBUG:boto:Params: {} 2024-02-01T10:22:14.752 DEBUG:boto:Token: None 2024-02-01T10:22:14.752 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:14 GMT /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:14.752 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nx/Hk+go6jJfbtNkK5zwZ4xhMS4= 2024-02-01T10:22:14.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nx/Hk+go6jJfbtNkK5zwZ4xhMS4=', 'Content-Length': '0'} 2024-02-01T10:22:14.755 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:34 GMT'), ('x-amz-version-id', 'txgLlJWZVbWbjqLWqkOtt87dUoD4zun'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e5ddedf582ef2762-0065bb70d6-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:14.755 DEBUG:boto:path=/vclfcn-18/dummy 2024-02-01T10:22:14.755 DEBUG:boto:auth_path=/vclfcn-18/dummy 2024-02-01T10:22:14.756 DEBUG:boto:path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:14.756 DEBUG:boto:auth_path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:14.756 DEBUG:boto:Method: GET 2024-02-01T10:22:14.756 DEBUG:boto:Path: /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:14.756 DEBUG:boto:Data: 2024-02-01T10:22:14.756 DEBUG:boto:Headers: {} 2024-02-01T10:22:14.756 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:14.756 DEBUG:boto:Port: 8000 2024-02-01T10:22:14.756 DEBUG:boto:Params: {} 2024-02-01T10:22:14.756 DEBUG:boto:Token: None 2024-02-01T10:22:14.757 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:14 GMT /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:14.757 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nx/Hk+go6jJfbtNkK5zwZ4xhMS4= 2024-02-01T10:22:14.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nx/Hk+go6jJfbtNkK5zwZ4xhMS4=', 'Content-Length': '0'} 2024-02-01T10:22:14.760 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:34 GMT'), ('x-amz-version-id', 'txgLlJWZVbWbjqLWqkOtt87dUoD4zun'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bef552c43c474b5d-0065bb70d6-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:14.760 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:22:14.760 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:22:14.760 DEBUG:boto:path=/vclfcn-18/?prefix=dummy 2024-02-01T10:22:14.761 DEBUG:boto:auth_path=/vclfcn-18/?prefix=dummy 2024-02-01T10:22:14.761 DEBUG:boto:Method: GET 2024-02-01T10:22:14.761 DEBUG:boto:Path: /vclfcn-18/?prefix=dummy 2024-02-01T10:22:14.761 DEBUG:boto:Data: 2024-02-01T10:22:14.761 DEBUG:boto:Headers: {} 2024-02-01T10:22:14.761 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:14.761 DEBUG:boto:Port: 8000 2024-02-01T10:22:14.761 DEBUG:boto:Params: {} 2024-02-01T10:22:14.761 DEBUG:boto:Token: None 2024-02-01T10:22:14.761 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:14 GMT /vclfcn-18/ 2024-02-01T10:22:14.762 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ffhIfIKCzyePBNPdPtRZmv5PgGw= 2024-02-01T10:22:14.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ffhIfIKCzyePBNPdPtRZmv5PgGw=', 'Content-Length': '0'} 2024-02-01T10:22:14.765 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000159522068d85812a-0065bb70d6-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:14.765 DEBUG:boto:b'vclfcn-18dummy1000falsedummy2024-02-01T10:20:34.645Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:14.765 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:22:14.765 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:22:14.766 DEBUG:boto:path=/vclfcn-18/?prefix=dummy 2024-02-01T10:22:14.766 DEBUG:boto:auth_path=/vclfcn-18/?prefix=dummy 2024-02-01T10:22:14.766 DEBUG:boto:Method: GET 2024-02-01T10:22:14.766 DEBUG:boto:Path: /vclfcn-18/?prefix=dummy 2024-02-01T10:22:14.766 DEBUG:boto:Data: 2024-02-01T10:22:14.766 DEBUG:boto:Headers: {} 2024-02-01T10:22:14.766 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:14.766 DEBUG:boto:Port: 8000 2024-02-01T10:22:14.766 DEBUG:boto:Params: {} 2024-02-01T10:22:14.766 DEBUG:boto:Token: None 2024-02-01T10:22:14.767 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:14 GMT /vclfcn-18/ 2024-02-01T10:22:14.767 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ffhIfIKCzyePBNPdPtRZmv5PgGw= 2024-02-01T10:22:14.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ffhIfIKCzyePBNPdPtRZmv5PgGw=', 'Content-Length': '0'} 2024-02-01T10:22:14.771 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ec9acccf9bb3843-0065bb70d6-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:14.812 DEBUG:boto:b'vclfcn-18dummy1000falsedummy2024-02-01T10:20:34.645Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:14.813 DEBUG:boto:path=/vclfcn-18/dummy 2024-02-01T10:22:14.813 DEBUG:boto:auth_path=/vclfcn-18/dummy 2024-02-01T10:22:14.813 DEBUG:boto:Method: HEAD 2024-02-01T10:22:14.813 DEBUG:boto:Path: /vclfcn-18/dummy 2024-02-01T10:22:14.813 DEBUG:boto:Data: 2024-02-01T10:22:14.813 DEBUG:boto:Headers: {} 2024-02-01T10:22:14.813 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:14.813 DEBUG:boto:Port: 8000 2024-02-01T10:22:14.813 DEBUG:boto:Params: {} 2024-02-01T10:22:14.813 DEBUG:boto:Token: None 2024-02-01T10:22:14.814 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:14 GMT /vclfcn-18/dummy 2024-02-01T10:22:14.814 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mSa4Vib4ecdg1Ym4AIz9e+DRRlA= 2024-02-01T10:22:14.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mSa4Vib4ecdg1Ym4AIz9e+DRRlA=', 'Content-Length': '0'} 2024-02-01T10:22:14.818 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:34 GMT'), ('x-amz-version-id', 'txgLlJWZVbWbjqLWqkOtt87dUoD4zun'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b90f000dab5ff90f-0065bb70d6-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:14.818 DEBUG:boto:path=/vclfcn-18/dummy 2024-02-01T10:22:14.818 DEBUG:boto:auth_path=/vclfcn-18/dummy 2024-02-01T10:22:14.818 DEBUG:boto:Method: HEAD 2024-02-01T10:22:14.818 DEBUG:boto:Path: /vclfcn-18/dummy 2024-02-01T10:22:14.819 DEBUG:boto:Data: 2024-02-01T10:22:14.819 DEBUG:boto:Headers: {} 2024-02-01T10:22:14.819 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:14.819 DEBUG:boto:Port: 8000 2024-02-01T10:22:14.819 DEBUG:boto:Params: {} 2024-02-01T10:22:14.819 DEBUG:boto:Token: None 2024-02-01T10:22:14.819 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:14 GMT /vclfcn-18/dummy 2024-02-01T10:22:14.819 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mSa4Vib4ecdg1Ym4AIz9e+DRRlA= 2024-02-01T10:22:14.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mSa4Vib4ecdg1Ym4AIz9e+DRRlA=', 'Content-Length': '0'} 2024-02-01T10:22:14.823 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:34 GMT'), ('x-amz-version-id', 'txgLlJWZVbWbjqLWqkOtt87dUoD4zun'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000061a6eaff2482171e-0065bb70d6-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:14.824 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-02-01T10:22:14.824 DEBUG:boto:path=/vclfcn-18/dummy 2024-02-01T10:22:14.824 DEBUG:boto:auth_path=/vclfcn-18/dummy 2024-02-01T10:22:14.824 DEBUG:boto:path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:14.824 DEBUG:boto:auth_path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:14.824 DEBUG:boto:Method: GET 2024-02-01T10:22:14.824 DEBUG:boto:Path: /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:14.824 DEBUG:boto:Data: 2024-02-01T10:22:14.824 DEBUG:boto:Headers: {} 2024-02-01T10:22:14.824 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:14.825 DEBUG:boto:Port: 8000 2024-02-01T10:22:14.825 DEBUG:boto:Params: {} 2024-02-01T10:22:14.825 DEBUG:boto:Token: None 2024-02-01T10:22:14.825 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:14 GMT /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:14.825 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nx/Hk+go6jJfbtNkK5zwZ4xhMS4= 2024-02-01T10:22:14.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nx/Hk+go6jJfbtNkK5zwZ4xhMS4=', 'Content-Length': '0'} 2024-02-01T10:22:14.828 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:34 GMT'), ('x-amz-version-id', 'txgLlJWZVbWbjqLWqkOtt87dUoD4zun'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ca4851d1fac97cae-0065bb70d6-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:14.828 DEBUG:boto:path=/vclfcn-18/dummy 2024-02-01T10:22:14.828 DEBUG:boto:auth_path=/vclfcn-18/dummy 2024-02-01T10:22:14.828 DEBUG:boto:path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:14.828 DEBUG:boto:auth_path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:14.828 DEBUG:boto:Method: GET 2024-02-01T10:22:14.829 DEBUG:boto:Path: /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:14.829 DEBUG:boto:Data: 2024-02-01T10:22:14.829 DEBUG:boto:Headers: {} 2024-02-01T10:22:14.829 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:14.829 DEBUG:boto:Port: 8000 2024-02-01T10:22:14.829 DEBUG:boto:Params: {} 2024-02-01T10:22:14.829 DEBUG:boto:Token: None 2024-02-01T10:22:14.829 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:14 GMT /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:14.829 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nx/Hk+go6jJfbtNkK5zwZ4xhMS4= 2024-02-01T10:22:14.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nx/Hk+go6jJfbtNkK5zwZ4xhMS4=', 'Content-Length': '0'} 2024-02-01T10:22:14.832 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:34 GMT'), ('x-amz-version-id', 'txgLlJWZVbWbjqLWqkOtt87dUoD4zun'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c0e696e141e01664-0065bb70d6-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:14.832 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:22:14.832 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:22:14.832 DEBUG:boto:path=/vclfcn-18/?prefix=dummy 2024-02-01T10:22:14.833 DEBUG:boto:auth_path=/vclfcn-18/?prefix=dummy 2024-02-01T10:22:14.833 DEBUG:boto:Method: GET 2024-02-01T10:22:14.833 DEBUG:boto:Path: /vclfcn-18/?prefix=dummy 2024-02-01T10:22:14.833 DEBUG:boto:Data: 2024-02-01T10:22:14.833 DEBUG:boto:Headers: {} 2024-02-01T10:22:14.833 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:14.833 DEBUG:boto:Port: 8000 2024-02-01T10:22:14.833 DEBUG:boto:Params: {} 2024-02-01T10:22:14.833 DEBUG:boto:Token: None 2024-02-01T10:22:14.833 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:14 GMT /vclfcn-18/ 2024-02-01T10:22:14.834 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ffhIfIKCzyePBNPdPtRZmv5PgGw= 2024-02-01T10:22:14.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ffhIfIKCzyePBNPdPtRZmv5PgGw=', 'Content-Length': '0'} 2024-02-01T10:22:14.838 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033aacb8f5589fe46-0065bb70d6-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:14.838 DEBUG:boto:b'vclfcn-18dummy1000falsedummy2024-02-01T10:20:34.645Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:14.838 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:22:14.839 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:22:14.839 DEBUG:boto:path=/vclfcn-18/?prefix=dummy 2024-02-01T10:22:14.839 DEBUG:boto:auth_path=/vclfcn-18/?prefix=dummy 2024-02-01T10:22:14.839 DEBUG:boto:Method: GET 2024-02-01T10:22:14.839 DEBUG:boto:Path: /vclfcn-18/?prefix=dummy 2024-02-01T10:22:14.839 DEBUG:boto:Data: 2024-02-01T10:22:14.839 DEBUG:boto:Headers: {} 2024-02-01T10:22:14.839 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:14.839 DEBUG:boto:Port: 8000 2024-02-01T10:22:14.839 DEBUG:boto:Params: {} 2024-02-01T10:22:14.839 DEBUG:boto:Token: None 2024-02-01T10:22:14.840 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:14 GMT /vclfcn-18/ 2024-02-01T10:22:14.840 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ffhIfIKCzyePBNPdPtRZmv5PgGw= 2024-02-01T10:22:14.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ffhIfIKCzyePBNPdPtRZmv5PgGw=', 'Content-Length': '0'} 2024-02-01T10:22:14.844 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a1b64c2107ccfb1-0065bb70d6-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:14.884 DEBUG:boto:b'vclfcn-18dummy1000falsedummy2024-02-01T10:20:34.645Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:14.884 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-02-01T10:22:14.885 DEBUG:boto:path=/vclfcn-18/obj-test-zone1 2024-02-01T10:22:14.885 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1 2024-02-01T10:22:14.885 DEBUG:boto:path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:14.885 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:14.885 DEBUG:boto:Method: GET 2024-02-01T10:22:14.885 DEBUG:boto:Path: /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:14.885 DEBUG:boto:Data: 2024-02-01T10:22:14.885 DEBUG:boto:Headers: {} 2024-02-01T10:22:14.885 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:14.885 DEBUG:boto:Port: 8000 2024-02-01T10:22:14.886 DEBUG:boto:Params: {} 2024-02-01T10:22:14.886 DEBUG:boto:Token: None 2024-02-01T10:22:14.886 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:14 GMT /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:14.886 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:CeD+UCP5LFx2r8K9UL4EUciUChU= 2024-02-01T10:22:14.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:CeD+UCP5LFx2r8K9UL4EUciUChU=', 'Content-Length': '0'} 2024-02-01T10:22:14.889 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f095a890a088742e-0065bb70d6-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:14.890 DEBUG:boto:path=/vclfcn-18/obj-test-zone1 2024-02-01T10:22:14.890 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1 2024-02-01T10:22:14.890 DEBUG:boto:path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:14.890 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:14.890 DEBUG:boto:Method: GET 2024-02-01T10:22:14.890 DEBUG:boto:Path: /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:14.890 DEBUG:boto:Data: 2024-02-01T10:22:14.890 DEBUG:boto:Headers: {} 2024-02-01T10:22:14.890 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:14.890 DEBUG:boto:Port: 8000 2024-02-01T10:22:14.890 DEBUG:boto:Params: {} 2024-02-01T10:22:14.891 DEBUG:boto:Token: None 2024-02-01T10:22:14.891 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:14 GMT /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:14.891 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:CeD+UCP5LFx2r8K9UL4EUciUChU= 2024-02-01T10:22:14.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:CeD+UCP5LFx2r8K9UL4EUciUChU=', 'Content-Length': '0'} 2024-02-01T10:22:14.894 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006bafe10e2102eebf-0065bb70d6-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:14.894 DEBUG:boto:path=/vclfcn-18/obj-test-zone1 2024-02-01T10:22:14.894 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1 2024-02-01T10:22:14.895 DEBUG:boto:path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:14.895 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:14.895 DEBUG:boto:Method: HEAD 2024-02-01T10:22:14.895 DEBUG:boto:Path: /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:14.895 DEBUG:boto:Data: 2024-02-01T10:22:14.895 DEBUG:boto:Headers: {} 2024-02-01T10:22:14.895 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:14.895 DEBUG:boto:Port: 8000 2024-02-01T10:22:14.895 DEBUG:boto:Params: {} 2024-02-01T10:22:14.895 DEBUG:boto:Token: None 2024-02-01T10:22:14.895 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:14 GMT /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:14.896 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:4k45tKOLz2eSvGB1bxFyOGlCRCE= 2024-02-01T10:22:14.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:4k45tKOLz2eSvGB1bxFyOGlCRCE=', 'Content-Length': '0'} 2024-02-01T10:22:14.899 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d1fcdf581965ff51-0065bb70d6-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:14.899 DEBUG:boto:path=/vclfcn-18/obj-test-zone1 2024-02-01T10:22:14.899 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1 2024-02-01T10:22:14.899 DEBUG:boto:path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:14.899 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:14.899 DEBUG:boto:Method: HEAD 2024-02-01T10:22:14.899 DEBUG:boto:Path: /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:14.899 DEBUG:boto:Data: 2024-02-01T10:22:14.899 DEBUG:boto:Headers: {} 2024-02-01T10:22:14.900 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:14.900 DEBUG:boto:Port: 8000 2024-02-01T10:22:14.900 DEBUG:boto:Params: {} 2024-02-01T10:22:14.900 DEBUG:boto:Token: None 2024-02-01T10:22:14.900 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:14 GMT /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:14.900 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:4k45tKOLz2eSvGB1bxFyOGlCRCE= 2024-02-01T10:22:14.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:4k45tKOLz2eSvGB1bxFyOGlCRCE=', 'Content-Length': '0'} 2024-02-01T10:22:14.903 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001c99ef7c8a0c4e79-0065bb70d6-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:14.903 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-02-01T10:22:14.904 DEBUG:boto:path=/vclfcn-18/obj-test-zone1 2024-02-01T10:22:14.904 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1 2024-02-01T10:22:14.904 DEBUG:boto:path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:14.904 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:14.904 DEBUG:boto:Method: GET 2024-02-01T10:22:14.904 DEBUG:boto:Path: /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:14.904 DEBUG:boto:Data: 2024-02-01T10:22:14.904 DEBUG:boto:Headers: {} 2024-02-01T10:22:14.904 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:14.904 DEBUG:boto:Port: 8000 2024-02-01T10:22:14.904 DEBUG:boto:Params: {} 2024-02-01T10:22:14.905 DEBUG:boto:Token: None 2024-02-01T10:22:14.905 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:14 GMT /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:14.905 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:CeD+UCP5LFx2r8K9UL4EUciUChU= 2024-02-01T10:22:14.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:CeD+UCP5LFx2r8K9UL4EUciUChU=', 'Content-Length': '0'} 2024-02-01T10:22:14.908 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000ca853ebcb9a1334-0065bb70d6-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:14.908 DEBUG:boto:path=/vclfcn-18/obj-test-zone1 2024-02-01T10:22:14.909 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1 2024-02-01T10:22:14.909 DEBUG:boto:path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:14.909 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:14.909 DEBUG:boto:Method: GET 2024-02-01T10:22:14.909 DEBUG:boto:Path: /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:14.909 DEBUG:boto:Data: 2024-02-01T10:22:14.909 DEBUG:boto:Headers: {} 2024-02-01T10:22:14.909 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:14.909 DEBUG:boto:Port: 8000 2024-02-01T10:22:14.909 DEBUG:boto:Params: {} 2024-02-01T10:22:14.909 DEBUG:boto:Token: None 2024-02-01T10:22:14.910 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:14 GMT /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:14.910 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:CeD+UCP5LFx2r8K9UL4EUciUChU= 2024-02-01T10:22:14.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:CeD+UCP5LFx2r8K9UL4EUciUChU=', 'Content-Length': '0'} 2024-02-01T10:22:14.913 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000065868623af88a124-0065bb70d6-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:14.913 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:22:14.913 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:22:14.913 DEBUG:boto:path=/vclfcn-18/?prefix=obj-test-zone1 2024-02-01T10:22:14.913 DEBUG:boto:auth_path=/vclfcn-18/?prefix=obj-test-zone1 2024-02-01T10:22:14.914 DEBUG:boto:Method: GET 2024-02-01T10:22:14.914 DEBUG:boto:Path: /vclfcn-18/?prefix=obj-test-zone1 2024-02-01T10:22:14.914 DEBUG:boto:Data: 2024-02-01T10:22:14.914 DEBUG:boto:Headers: {} 2024-02-01T10:22:14.914 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:14.914 DEBUG:boto:Port: 8000 2024-02-01T10:22:14.914 DEBUG:boto:Params: {} 2024-02-01T10:22:14.914 DEBUG:boto:Token: None 2024-02-01T10:22:14.914 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:14 GMT /vclfcn-18/ 2024-02-01T10:22:14.914 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ffhIfIKCzyePBNPdPtRZmv5PgGw= 2024-02-01T10:22:14.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ffhIfIKCzyePBNPdPtRZmv5PgGw=', 'Content-Length': '0'} 2024-02-01T10:22:14.919 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e8de3733067ecebd-0065bb70d6-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:14.960 DEBUG:boto:b'vclfcn-18obj-test-zone11000falseobj-test-zone12024-02-01T10:20:50.942Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:14.961 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:22:14.961 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:22:14.961 DEBUG:boto:path=/vclfcn-18/?prefix=obj-test-zone1 2024-02-01T10:22:14.961 DEBUG:boto:auth_path=/vclfcn-18/?prefix=obj-test-zone1 2024-02-01T10:22:14.961 DEBUG:boto:Method: GET 2024-02-01T10:22:14.961 DEBUG:boto:Path: /vclfcn-18/?prefix=obj-test-zone1 2024-02-01T10:22:14.961 DEBUG:boto:Data: 2024-02-01T10:22:14.961 DEBUG:boto:Headers: {} 2024-02-01T10:22:14.961 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:14.961 DEBUG:boto:Port: 8000 2024-02-01T10:22:14.961 DEBUG:boto:Params: {} 2024-02-01T10:22:14.962 DEBUG:boto:Token: None 2024-02-01T10:22:14.962 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:14 GMT /vclfcn-18/ 2024-02-01T10:22:14.962 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ffhIfIKCzyePBNPdPtRZmv5PgGw= 2024-02-01T10:22:14.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ffhIfIKCzyePBNPdPtRZmv5PgGw=', 'Content-Length': '0'} 2024-02-01T10:22:14.966 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007636b985acc2f43c-0065bb70d6-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:14.966 DEBUG:boto:b'vclfcn-18obj-test-zone11000falseobj-test-zone12024-02-01T10:20:50.942Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:14.967 DEBUG:boto:path=/vclfcn-18/obj-test-zone1 2024-02-01T10:22:14.967 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1 2024-02-01T10:22:14.967 DEBUG:boto:Method: HEAD 2024-02-01T10:22:14.967 DEBUG:boto:Path: /vclfcn-18/obj-test-zone1 2024-02-01T10:22:14.967 DEBUG:boto:Data: 2024-02-01T10:22:14.967 DEBUG:boto:Headers: {} 2024-02-01T10:22:14.967 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:14.967 DEBUG:boto:Port: 8000 2024-02-01T10:22:14.967 DEBUG:boto:Params: {} 2024-02-01T10:22:14.967 DEBUG:boto:Token: None 2024-02-01T10:22:14.968 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:14 GMT /vclfcn-18/obj-test-zone1 2024-02-01T10:22:14.968 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:6dGWH6DKitlZmYBJLECZWL1N5iI= 2024-02-01T10:22:14.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:6dGWH6DKitlZmYBJLECZWL1N5iI=', 'Content-Length': '0'} 2024-02-01T10:22:14.972 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ca735ea60ecbae22-0065bb70d6-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:14.972 DEBUG:boto:path=/vclfcn-18/obj-test-zone1 2024-02-01T10:22:14.972 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1 2024-02-01T10:22:14.972 DEBUG:boto:Method: HEAD 2024-02-01T10:22:14.972 DEBUG:boto:Path: /vclfcn-18/obj-test-zone1 2024-02-01T10:22:14.972 DEBUG:boto:Data: 2024-02-01T10:22:14.973 DEBUG:boto:Headers: {} 2024-02-01T10:22:14.973 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:14.973 DEBUG:boto:Port: 8000 2024-02-01T10:22:14.973 DEBUG:boto:Params: {} 2024-02-01T10:22:14.973 DEBUG:boto:Token: None 2024-02-01T10:22:14.973 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:14 GMT /vclfcn-18/obj-test-zone1 2024-02-01T10:22:14.973 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:6dGWH6DKitlZmYBJLECZWL1N5iI= 2024-02-01T10:22:14.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:6dGWH6DKitlZmYBJLECZWL1N5iI=', 'Content-Length': '0'} 2024-02-01T10:22:14.979 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000b2bd56a28e13bbf-0065bb70d6-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:14.979 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-02-01T10:22:14.980 DEBUG:boto:path=/vclfcn-18/obj-test-zone1 2024-02-01T10:22:14.980 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1 2024-02-01T10:22:14.980 DEBUG:boto:path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:14.980 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:14.980 DEBUG:boto:Method: GET 2024-02-01T10:22:14.980 DEBUG:boto:Path: /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:14.980 DEBUG:boto:Data: 2024-02-01T10:22:14.980 DEBUG:boto:Headers: {} 2024-02-01T10:22:14.980 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:14.980 DEBUG:boto:Port: 8000 2024-02-01T10:22:14.981 DEBUG:boto:Params: {} 2024-02-01T10:22:14.981 DEBUG:boto:Token: None 2024-02-01T10:22:14.981 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:14 GMT /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:14.981 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:CeD+UCP5LFx2r8K9UL4EUciUChU= 2024-02-01T10:22:14.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:CeD+UCP5LFx2r8K9UL4EUciUChU=', 'Content-Length': '0'} 2024-02-01T10:22:14.984 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eeba35e5cb6c0d7e-0065bb70d6-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:14.984 DEBUG:boto:path=/vclfcn-18/obj-test-zone1 2024-02-01T10:22:14.984 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1 2024-02-01T10:22:14.984 DEBUG:boto:path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:14.984 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:14.985 DEBUG:boto:Method: GET 2024-02-01T10:22:14.985 DEBUG:boto:Path: /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:14.985 DEBUG:boto:Data: 2024-02-01T10:22:14.985 DEBUG:boto:Headers: {} 2024-02-01T10:22:14.985 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:14.985 DEBUG:boto:Port: 8000 2024-02-01T10:22:14.985 DEBUG:boto:Params: {} 2024-02-01T10:22:14.985 DEBUG:boto:Token: None 2024-02-01T10:22:14.985 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:14 GMT /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:14.985 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:CeD+UCP5LFx2r8K9UL4EUciUChU= 2024-02-01T10:22:14.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:CeD+UCP5LFx2r8K9UL4EUciUChU=', 'Content-Length': '0'} 2024-02-01T10:22:14.989 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e1519fc0e65b6865-0065bb70d6-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:15.032 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:22:15.032 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:22:15.032 DEBUG:boto:path=/vclfcn-18/?prefix=obj-test-zone1 2024-02-01T10:22:15.033 DEBUG:boto:auth_path=/vclfcn-18/?prefix=obj-test-zone1 2024-02-01T10:22:15.033 DEBUG:boto:Method: GET 2024-02-01T10:22:15.033 DEBUG:boto:Path: /vclfcn-18/?prefix=obj-test-zone1 2024-02-01T10:22:15.033 DEBUG:boto:Data: 2024-02-01T10:22:15.033 DEBUG:boto:Headers: {} 2024-02-01T10:22:15.033 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:15.033 DEBUG:boto:Port: 8000 2024-02-01T10:22:15.033 DEBUG:boto:Params: {} 2024-02-01T10:22:15.034 DEBUG:boto:Token: None 2024-02-01T10:22:15.034 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:15 GMT /vclfcn-18/ 2024-02-01T10:22:15.034 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:3TwBtCmYeXITMvaVRg+vTOKO7IQ= 2024-02-01T10:22:15.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:3TwBtCmYeXITMvaVRg+vTOKO7IQ=', 'Content-Length': '0'} 2024-02-01T10:22:15.038 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008dac29af0d1af672-0065bb70d7-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:15.038 DEBUG:boto:b'vclfcn-18obj-test-zone11000falseobj-test-zone12024-02-01T10:20:50.942Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:15.039 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:22:15.039 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:22:15.039 DEBUG:boto:path=/vclfcn-18/?prefix=obj-test-zone1 2024-02-01T10:22:15.039 DEBUG:boto:auth_path=/vclfcn-18/?prefix=obj-test-zone1 2024-02-01T10:22:15.039 DEBUG:boto:Method: GET 2024-02-01T10:22:15.039 DEBUG:boto:Path: /vclfcn-18/?prefix=obj-test-zone1 2024-02-01T10:22:15.039 DEBUG:boto:Data: 2024-02-01T10:22:15.039 DEBUG:boto:Headers: {} 2024-02-01T10:22:15.040 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:15.040 DEBUG:boto:Port: 8000 2024-02-01T10:22:15.040 DEBUG:boto:Params: {} 2024-02-01T10:22:15.040 DEBUG:boto:Token: None 2024-02-01T10:22:15.040 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:15 GMT /vclfcn-18/ 2024-02-01T10:22:15.040 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:3TwBtCmYeXITMvaVRg+vTOKO7IQ= 2024-02-01T10:22:15.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:3TwBtCmYeXITMvaVRg+vTOKO7IQ=', 'Content-Length': '0'} 2024-02-01T10:22:15.044 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c479db9726a34d6-0065bb70d7-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:15.044 DEBUG:boto:b'vclfcn-18obj-test-zone11000falseobj-test-zone12024-02-01T10:20:50.942Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:15.045 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-02-01T10:22:15.046 DEBUG:boto:path=/vclfcn-18/obj-test-zone2 2024-02-01T10:22:15.047 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2 2024-02-01T10:22:15.047 DEBUG:boto:path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:15.047 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:15.048 DEBUG:boto:Method: GET 2024-02-01T10:22:15.048 DEBUG:boto:Path: /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:15.048 DEBUG:boto:Data: 2024-02-01T10:22:15.048 DEBUG:boto:Headers: {} 2024-02-01T10:22:15.048 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:15.048 DEBUG:boto:Port: 8000 2024-02-01T10:22:15.048 DEBUG:boto:Params: {} 2024-02-01T10:22:15.048 DEBUG:boto:Token: None 2024-02-01T10:22:15.048 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:15 GMT /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:15.049 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FfF8bGDp1pwQsXLGCxe/+eJdIjM= 2024-02-01T10:22:15.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FfF8bGDp1pwQsXLGCxe/+eJdIjM=', 'Content-Length': '0'} 2024-02-01T10:22:15.052 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', 'ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000019a8ea35daadb194-0065bb70d7-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:15.052 DEBUG:boto:path=/vclfcn-18/obj-test-zone2 2024-02-01T10:22:15.052 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2 2024-02-01T10:22:15.052 DEBUG:boto:path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:15.052 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:15.053 DEBUG:boto:Method: GET 2024-02-01T10:22:15.053 DEBUG:boto:Path: /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:15.053 DEBUG:boto:Data: 2024-02-01T10:22:15.053 DEBUG:boto:Headers: {} 2024-02-01T10:22:15.053 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:15.053 DEBUG:boto:Port: 8000 2024-02-01T10:22:15.053 DEBUG:boto:Params: {} 2024-02-01T10:22:15.053 DEBUG:boto:Token: None 2024-02-01T10:22:15.053 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:15 GMT /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:15.053 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FfF8bGDp1pwQsXLGCxe/+eJdIjM= 2024-02-01T10:22:15.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FfF8bGDp1pwQsXLGCxe/+eJdIjM=', 'Content-Length': '0'} 2024-02-01T10:22:15.057 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', 'ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b938a85608877609-0065bb70d7-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:15.104 DEBUG:boto:path=/vclfcn-18/obj-test-zone2 2024-02-01T10:22:15.104 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2 2024-02-01T10:22:15.104 DEBUG:boto:path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:15.104 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:15.105 DEBUG:boto:Method: HEAD 2024-02-01T10:22:15.105 DEBUG:boto:Path: /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:15.105 DEBUG:boto:Data: 2024-02-01T10:22:15.105 DEBUG:boto:Headers: {} 2024-02-01T10:22:15.105 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:15.105 DEBUG:boto:Port: 8000 2024-02-01T10:22:15.105 DEBUG:boto:Params: {} 2024-02-01T10:22:15.105 DEBUG:boto:Token: None 2024-02-01T10:22:15.105 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:15 GMT /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:15.106 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:eXdrvMEwLbk7PPi0PNE7sCdOReM= 2024-02-01T10:22:15.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:eXdrvMEwLbk7PPi0PNE7sCdOReM=', 'Content-Length': '0'} 2024-02-01T10:22:15.108 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', 'ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dc36b81628082c60-0065bb70d7-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:15.109 DEBUG:boto:path=/vclfcn-18/obj-test-zone2 2024-02-01T10:22:15.109 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2 2024-02-01T10:22:15.109 DEBUG:boto:path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:15.109 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:15.109 DEBUG:boto:Method: HEAD 2024-02-01T10:22:15.109 DEBUG:boto:Path: /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:15.109 DEBUG:boto:Data: 2024-02-01T10:22:15.109 DEBUG:boto:Headers: {} 2024-02-01T10:22:15.109 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:15.109 DEBUG:boto:Port: 8000 2024-02-01T10:22:15.109 DEBUG:boto:Params: {} 2024-02-01T10:22:15.110 DEBUG:boto:Token: None 2024-02-01T10:22:15.110 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:15 GMT /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:15.110 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:eXdrvMEwLbk7PPi0PNE7sCdOReM= 2024-02-01T10:22:15.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:eXdrvMEwLbk7PPi0PNE7sCdOReM=', 'Content-Length': '0'} 2024-02-01T10:22:15.113 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', 'ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000019f9243c7f8fcb47-0065bb70d7-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:15.113 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-02-01T10:22:15.113 DEBUG:boto:path=/vclfcn-18/obj-test-zone2 2024-02-01T10:22:15.113 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2 2024-02-01T10:22:15.113 DEBUG:boto:path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:15.113 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:15.113 DEBUG:boto:Method: GET 2024-02-01T10:22:15.113 DEBUG:boto:Path: /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:15.114 DEBUG:boto:Data: 2024-02-01T10:22:15.114 DEBUG:boto:Headers: {} 2024-02-01T10:22:15.114 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:15.114 DEBUG:boto:Port: 8000 2024-02-01T10:22:15.114 DEBUG:boto:Params: {} 2024-02-01T10:22:15.114 DEBUG:boto:Token: None 2024-02-01T10:22:15.114 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:15 GMT /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:15.114 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FfF8bGDp1pwQsXLGCxe/+eJdIjM= 2024-02-01T10:22:15.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FfF8bGDp1pwQsXLGCxe/+eJdIjM=', 'Content-Length': '0'} 2024-02-01T10:22:15.117 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', 'ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ee44cce994c27e88-0065bb70d7-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:15.164 DEBUG:boto:path=/vclfcn-18/obj-test-zone2 2024-02-01T10:22:15.164 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2 2024-02-01T10:22:15.164 DEBUG:boto:path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:15.164 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:15.165 DEBUG:boto:Method: GET 2024-02-01T10:22:15.165 DEBUG:boto:Path: /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:15.165 DEBUG:boto:Data: 2024-02-01T10:22:15.165 DEBUG:boto:Headers: {} 2024-02-01T10:22:15.165 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:15.165 DEBUG:boto:Port: 8000 2024-02-01T10:22:15.165 DEBUG:boto:Params: {} 2024-02-01T10:22:15.165 DEBUG:boto:Token: None 2024-02-01T10:22:15.165 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:15 GMT /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:15.166 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FfF8bGDp1pwQsXLGCxe/+eJdIjM= 2024-02-01T10:22:15.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FfF8bGDp1pwQsXLGCxe/+eJdIjM=', 'Content-Length': '0'} 2024-02-01T10:22:15.169 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', 'ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000018aae13d12a7ad95-0065bb70d7-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:15.216 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:22:15.216 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:22:15.216 DEBUG:boto:path=/vclfcn-18/?prefix=obj-test-zone2 2024-02-01T10:22:15.216 DEBUG:boto:auth_path=/vclfcn-18/?prefix=obj-test-zone2 2024-02-01T10:22:15.217 DEBUG:boto:Method: GET 2024-02-01T10:22:15.217 DEBUG:boto:Path: /vclfcn-18/?prefix=obj-test-zone2 2024-02-01T10:22:15.217 DEBUG:boto:Data: 2024-02-01T10:22:15.217 DEBUG:boto:Headers: {} 2024-02-01T10:22:15.217 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:15.217 DEBUG:boto:Port: 8000 2024-02-01T10:22:15.217 DEBUG:boto:Params: {} 2024-02-01T10:22:15.217 DEBUG:boto:Token: None 2024-02-01T10:22:15.217 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:15 GMT /vclfcn-18/ 2024-02-01T10:22:15.218 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:3TwBtCmYeXITMvaVRg+vTOKO7IQ= 2024-02-01T10:22:15.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:3TwBtCmYeXITMvaVRg+vTOKO7IQ=', 'Content-Length': '0'} 2024-02-01T10:22:15.222 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000053ca2aa0ac44a29-0065bb70d7-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:15.222 DEBUG:boto:b'vclfcn-18obj-test-zone21000falseobj-test-zone22024-02-01T10:20:51.094Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:15.222 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:22:15.222 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:22:15.222 DEBUG:boto:path=/vclfcn-18/?prefix=obj-test-zone2 2024-02-01T10:22:15.223 DEBUG:boto:auth_path=/vclfcn-18/?prefix=obj-test-zone2 2024-02-01T10:22:15.223 DEBUG:boto:Method: GET 2024-02-01T10:22:15.223 DEBUG:boto:Path: /vclfcn-18/?prefix=obj-test-zone2 2024-02-01T10:22:15.223 DEBUG:boto:Data: 2024-02-01T10:22:15.223 DEBUG:boto:Headers: {} 2024-02-01T10:22:15.223 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:15.223 DEBUG:boto:Port: 8000 2024-02-01T10:22:15.223 DEBUG:boto:Params: {} 2024-02-01T10:22:15.223 DEBUG:boto:Token: None 2024-02-01T10:22:15.223 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:15 GMT /vclfcn-18/ 2024-02-01T10:22:15.224 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:3TwBtCmYeXITMvaVRg+vTOKO7IQ= 2024-02-01T10:22:15.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:3TwBtCmYeXITMvaVRg+vTOKO7IQ=', 'Content-Length': '0'} 2024-02-01T10:22:15.227 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd0cd31155ba5a60-0065bb70d7-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:15.228 DEBUG:boto:b'vclfcn-18obj-test-zone21000falseobj-test-zone22024-02-01T10:20:51.094Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:15.228 DEBUG:boto:path=/vclfcn-18/obj-test-zone2 2024-02-01T10:22:15.228 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2 2024-02-01T10:22:15.228 DEBUG:boto:Method: HEAD 2024-02-01T10:22:15.228 DEBUG:boto:Path: /vclfcn-18/obj-test-zone2 2024-02-01T10:22:15.228 DEBUG:boto:Data: 2024-02-01T10:22:15.229 DEBUG:boto:Headers: {} 2024-02-01T10:22:15.229 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:15.229 DEBUG:boto:Port: 8000 2024-02-01T10:22:15.229 DEBUG:boto:Params: {} 2024-02-01T10:22:15.229 DEBUG:boto:Token: None 2024-02-01T10:22:15.229 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:15 GMT /vclfcn-18/obj-test-zone2 2024-02-01T10:22:15.229 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:AqNcbVqHABhTND05NysEdwIHUfM= 2024-02-01T10:22:15.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:AqNcbVqHABhTND05NysEdwIHUfM=', 'Content-Length': '0'} 2024-02-01T10:22:15.233 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', 'ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000072857c7b694bf28-0065bb70d7-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:15.234 DEBUG:boto:path=/vclfcn-18/obj-test-zone2 2024-02-01T10:22:15.234 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2 2024-02-01T10:22:15.234 DEBUG:boto:Method: HEAD 2024-02-01T10:22:15.234 DEBUG:boto:Path: /vclfcn-18/obj-test-zone2 2024-02-01T10:22:15.234 DEBUG:boto:Data: 2024-02-01T10:22:15.234 DEBUG:boto:Headers: {} 2024-02-01T10:22:15.234 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:15.234 DEBUG:boto:Port: 8000 2024-02-01T10:22:15.234 DEBUG:boto:Params: {} 2024-02-01T10:22:15.235 DEBUG:boto:Token: None 2024-02-01T10:22:15.235 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:15 GMT /vclfcn-18/obj-test-zone2 2024-02-01T10:22:15.235 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:AqNcbVqHABhTND05NysEdwIHUfM= 2024-02-01T10:22:15.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:AqNcbVqHABhTND05NysEdwIHUfM=', 'Content-Length': '0'} 2024-02-01T10:22:15.238 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', 'ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000033d33cec933d5a51-0065bb70d7-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:15.239 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-02-01T10:22:15.239 DEBUG:boto:path=/vclfcn-18/obj-test-zone2 2024-02-01T10:22:15.239 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2 2024-02-01T10:22:15.239 DEBUG:boto:path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:15.239 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:15.239 DEBUG:boto:Method: GET 2024-02-01T10:22:15.239 DEBUG:boto:Path: /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:15.239 DEBUG:boto:Data: 2024-02-01T10:22:15.239 DEBUG:boto:Headers: {} 2024-02-01T10:22:15.239 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:15.240 DEBUG:boto:Port: 8000 2024-02-01T10:22:15.240 DEBUG:boto:Params: {} 2024-02-01T10:22:15.240 DEBUG:boto:Token: None 2024-02-01T10:22:15.240 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:15 GMT /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:15.240 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FfF8bGDp1pwQsXLGCxe/+eJdIjM= 2024-02-01T10:22:15.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FfF8bGDp1pwQsXLGCxe/+eJdIjM=', 'Content-Length': '0'} 2024-02-01T10:22:15.243 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', 'ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d363303ddc5f153d-0065bb70d7-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:15.243 DEBUG:boto:path=/vclfcn-18/obj-test-zone2 2024-02-01T10:22:15.244 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2 2024-02-01T10:22:15.244 DEBUG:boto:path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:15.244 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:15.244 DEBUG:boto:Method: GET 2024-02-01T10:22:15.244 DEBUG:boto:Path: /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:15.244 DEBUG:boto:Data: 2024-02-01T10:22:15.244 DEBUG:boto:Headers: {} 2024-02-01T10:22:15.244 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:15.244 DEBUG:boto:Port: 8000 2024-02-01T10:22:15.244 DEBUG:boto:Params: {} 2024-02-01T10:22:15.244 DEBUG:boto:Token: None 2024-02-01T10:22:15.245 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:15 GMT /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:15.245 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FfF8bGDp1pwQsXLGCxe/+eJdIjM= 2024-02-01T10:22:15.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FfF8bGDp1pwQsXLGCxe/+eJdIjM=', 'Content-Length': '0'} 2024-02-01T10:22:15.248 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', 'ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001c9c9f756adf9776-0065bb70d7-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:15.292 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:22:15.293 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:22:15.293 DEBUG:boto:path=/vclfcn-18/?prefix=obj-test-zone2 2024-02-01T10:22:15.293 DEBUG:boto:auth_path=/vclfcn-18/?prefix=obj-test-zone2 2024-02-01T10:22:15.293 DEBUG:boto:Method: GET 2024-02-01T10:22:15.293 DEBUG:boto:Path: /vclfcn-18/?prefix=obj-test-zone2 2024-02-01T10:22:15.293 DEBUG:boto:Data: 2024-02-01T10:22:15.294 DEBUG:boto:Headers: {} 2024-02-01T10:22:15.294 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:15.294 DEBUG:boto:Port: 8000 2024-02-01T10:22:15.294 DEBUG:boto:Params: {} 2024-02-01T10:22:15.294 DEBUG:boto:Token: None 2024-02-01T10:22:15.294 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:15 GMT /vclfcn-18/ 2024-02-01T10:22:15.294 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:3TwBtCmYeXITMvaVRg+vTOKO7IQ= 2024-02-01T10:22:15.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:3TwBtCmYeXITMvaVRg+vTOKO7IQ=', 'Content-Length': '0'} 2024-02-01T10:22:15.298 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fda05f1ce10b2bac-0065bb70d7-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:15.299 DEBUG:boto:b'vclfcn-18obj-test-zone21000falseobj-test-zone22024-02-01T10:20:51.094Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:15.300 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:22:15.300 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:22:15.300 DEBUG:boto:path=/vclfcn-18/?prefix=obj-test-zone2 2024-02-01T10:22:15.300 DEBUG:boto:auth_path=/vclfcn-18/?prefix=obj-test-zone2 2024-02-01T10:22:15.300 DEBUG:boto:Method: GET 2024-02-01T10:22:15.300 DEBUG:boto:Path: /vclfcn-18/?prefix=obj-test-zone2 2024-02-01T10:22:15.300 DEBUG:boto:Data: 2024-02-01T10:22:15.300 DEBUG:boto:Headers: {} 2024-02-01T10:22:15.300 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:15.301 DEBUG:boto:Port: 8000 2024-02-01T10:22:15.301 DEBUG:boto:Params: {} 2024-02-01T10:22:15.301 DEBUG:boto:Token: None 2024-02-01T10:22:15.301 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:15 GMT /vclfcn-18/ 2024-02-01T10:22:15.301 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:3TwBtCmYeXITMvaVRg+vTOKO7IQ= 2024-02-01T10:22:15.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:3TwBtCmYeXITMvaVRg+vTOKO7IQ=', 'Content-Length': '0'} 2024-02-01T10:22:15.305 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d00d5690f736ede-0065bb70d7-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:15.348 DEBUG:boto:b'vclfcn-18obj-test-zone21000falseobj-test-zone22024-02-01T10:20:51.094Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:15.348 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-02-01T10:22:15.349 DEBUG:boto:path=/vclfcn-18/obj-test-zone3 2024-02-01T10:22:15.349 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3 2024-02-01T10:22:15.349 DEBUG:boto:path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:15.349 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:15.349 DEBUG:boto:Method: GET 2024-02-01T10:22:15.349 DEBUG:boto:Path: /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:15.349 DEBUG:boto:Data: 2024-02-01T10:22:15.349 DEBUG:boto:Headers: {} 2024-02-01T10:22:15.349 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:15.349 DEBUG:boto:Port: 8000 2024-02-01T10:22:15.350 DEBUG:boto:Params: {} 2024-02-01T10:22:15.350 DEBUG:boto:Token: None 2024-02-01T10:22:15.350 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:15 GMT /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:15.350 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1lL+qZo6Ajtfsro9NLdsNuHhcqI= 2024-02-01T10:22:15.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1lL+qZo6Ajtfsro9NLdsNuHhcqI=', 'Content-Length': '0'} 2024-02-01T10:22:15.353 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', '6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b245a758acc76d21-0065bb70d7-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:15.353 DEBUG:boto:path=/vclfcn-18/obj-test-zone3 2024-02-01T10:22:15.354 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3 2024-02-01T10:22:15.354 DEBUG:boto:path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:15.354 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:15.354 DEBUG:boto:Method: GET 2024-02-01T10:22:15.354 DEBUG:boto:Path: /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:15.354 DEBUG:boto:Data: 2024-02-01T10:22:15.354 DEBUG:boto:Headers: {} 2024-02-01T10:22:15.354 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:15.354 DEBUG:boto:Port: 8000 2024-02-01T10:22:15.354 DEBUG:boto:Params: {} 2024-02-01T10:22:15.354 DEBUG:boto:Token: None 2024-02-01T10:22:15.355 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:15 GMT /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:15.355 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1lL+qZo6Ajtfsro9NLdsNuHhcqI= 2024-02-01T10:22:15.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1lL+qZo6Ajtfsro9NLdsNuHhcqI=', 'Content-Length': '0'} 2024-02-01T10:22:15.357 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', '6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000e4b32f89cd235a4-0065bb70d7-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:15.358 DEBUG:boto:path=/vclfcn-18/obj-test-zone3 2024-02-01T10:22:15.358 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3 2024-02-01T10:22:15.358 DEBUG:boto:path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:15.358 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:15.358 DEBUG:boto:Method: HEAD 2024-02-01T10:22:15.358 DEBUG:boto:Path: /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:15.358 DEBUG:boto:Data: 2024-02-01T10:22:15.358 DEBUG:boto:Headers: {} 2024-02-01T10:22:15.358 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:15.358 DEBUG:boto:Port: 8000 2024-02-01T10:22:15.359 DEBUG:boto:Params: {} 2024-02-01T10:22:15.359 DEBUG:boto:Token: None 2024-02-01T10:22:15.359 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:15 GMT /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:15.359 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ym08WjZj4ZjzH5gl+na/qzs+JYk= 2024-02-01T10:22:15.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ym08WjZj4ZjzH5gl+na/qzs+JYk=', 'Content-Length': '0'} 2024-02-01T10:22:15.362 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', '6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001c18d2bed30506ae-0065bb70d7-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:15.362 DEBUG:boto:path=/vclfcn-18/obj-test-zone3 2024-02-01T10:22:15.362 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3 2024-02-01T10:22:15.362 DEBUG:boto:path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:15.362 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:15.363 DEBUG:boto:Method: HEAD 2024-02-01T10:22:15.363 DEBUG:boto:Path: /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:15.363 DEBUG:boto:Data: 2024-02-01T10:22:15.363 DEBUG:boto:Headers: {} 2024-02-01T10:22:15.363 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:15.363 DEBUG:boto:Port: 8000 2024-02-01T10:22:15.363 DEBUG:boto:Params: {} 2024-02-01T10:22:15.363 DEBUG:boto:Token: None 2024-02-01T10:22:15.363 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:15 GMT /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:15.363 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ym08WjZj4ZjzH5gl+na/qzs+JYk= 2024-02-01T10:22:15.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ym08WjZj4ZjzH5gl+na/qzs+JYk=', 'Content-Length': '0'} 2024-02-01T10:22:15.366 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', '6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000814d7afd0657544e-0065bb70d7-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:15.367 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-02-01T10:22:15.367 DEBUG:boto:path=/vclfcn-18/obj-test-zone3 2024-02-01T10:22:15.367 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3 2024-02-01T10:22:15.367 DEBUG:boto:path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:15.367 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:15.367 DEBUG:boto:Method: GET 2024-02-01T10:22:15.367 DEBUG:boto:Path: /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:15.367 DEBUG:boto:Data: 2024-02-01T10:22:15.367 DEBUG:boto:Headers: {} 2024-02-01T10:22:15.367 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:15.368 DEBUG:boto:Port: 8000 2024-02-01T10:22:15.368 DEBUG:boto:Params: {} 2024-02-01T10:22:15.368 DEBUG:boto:Token: None 2024-02-01T10:22:15.368 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:15 GMT /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:15.368 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1lL+qZo6Ajtfsro9NLdsNuHhcqI= 2024-02-01T10:22:15.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1lL+qZo6Ajtfsro9NLdsNuHhcqI=', 'Content-Length': '0'} 2024-02-01T10:22:15.371 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', '6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000660d4c18cdca3980-0065bb70d7-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:15.412 DEBUG:boto:path=/vclfcn-18/obj-test-zone3 2024-02-01T10:22:15.412 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3 2024-02-01T10:22:15.412 DEBUG:boto:path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:15.412 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:15.413 DEBUG:boto:Method: GET 2024-02-01T10:22:15.413 DEBUG:boto:Path: /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:15.413 DEBUG:boto:Data: 2024-02-01T10:22:15.413 DEBUG:boto:Headers: {} 2024-02-01T10:22:15.413 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:15.413 DEBUG:boto:Port: 8000 2024-02-01T10:22:15.413 DEBUG:boto:Params: {} 2024-02-01T10:22:15.413 DEBUG:boto:Token: None 2024-02-01T10:22:15.413 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:15 GMT /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:15.414 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1lL+qZo6Ajtfsro9NLdsNuHhcqI= 2024-02-01T10:22:15.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1lL+qZo6Ajtfsro9NLdsNuHhcqI=', 'Content-Length': '0'} 2024-02-01T10:22:15.416 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', '6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000268ad129ad295d53-0065bb70d7-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:15.464 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:22:15.464 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:22:15.464 DEBUG:boto:path=/vclfcn-18/?prefix=obj-test-zone3 2024-02-01T10:22:15.465 DEBUG:boto:auth_path=/vclfcn-18/?prefix=obj-test-zone3 2024-02-01T10:22:15.465 DEBUG:boto:Method: GET 2024-02-01T10:22:15.465 DEBUG:boto:Path: /vclfcn-18/?prefix=obj-test-zone3 2024-02-01T10:22:15.465 DEBUG:boto:Data: 2024-02-01T10:22:15.465 DEBUG:boto:Headers: {} 2024-02-01T10:22:15.465 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:15.465 DEBUG:boto:Port: 8000 2024-02-01T10:22:15.465 DEBUG:boto:Params: {} 2024-02-01T10:22:15.465 DEBUG:boto:Token: None 2024-02-01T10:22:15.465 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:15 GMT /vclfcn-18/ 2024-02-01T10:22:15.466 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:3TwBtCmYeXITMvaVRg+vTOKO7IQ= 2024-02-01T10:22:15.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:3TwBtCmYeXITMvaVRg+vTOKO7IQ=', 'Content-Length': '0'} 2024-02-01T10:22:15.470 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5eb27060af4b53c-0065bb70d7-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:15.471 DEBUG:boto:b'vclfcn-18obj-test-zone31000falseobj-test-zone32024-02-01T10:20:51.248Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:15.471 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:22:15.471 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:22:15.471 DEBUG:boto:path=/vclfcn-18/?prefix=obj-test-zone3 2024-02-01T10:22:15.471 DEBUG:boto:auth_path=/vclfcn-18/?prefix=obj-test-zone3 2024-02-01T10:22:15.472 DEBUG:boto:Method: GET 2024-02-01T10:22:15.472 DEBUG:boto:Path: /vclfcn-18/?prefix=obj-test-zone3 2024-02-01T10:22:15.472 DEBUG:boto:Data: 2024-02-01T10:22:15.472 DEBUG:boto:Headers: {} 2024-02-01T10:22:15.472 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:15.472 DEBUG:boto:Port: 8000 2024-02-01T10:22:15.472 DEBUG:boto:Params: {} 2024-02-01T10:22:15.472 DEBUG:boto:Token: None 2024-02-01T10:22:15.472 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:15 GMT /vclfcn-18/ 2024-02-01T10:22:15.472 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:3TwBtCmYeXITMvaVRg+vTOKO7IQ= 2024-02-01T10:22:15.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:3TwBtCmYeXITMvaVRg+vTOKO7IQ=', 'Content-Length': '0'} 2024-02-01T10:22:15.477 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025f259db7c09f62b-0065bb70d7-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:15.520 DEBUG:boto:b'vclfcn-18obj-test-zone31000falseobj-test-zone32024-02-01T10:20:51.248Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:15.521 DEBUG:boto:path=/vclfcn-18/obj-test-zone3 2024-02-01T10:22:15.521 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3 2024-02-01T10:22:15.521 DEBUG:boto:Method: HEAD 2024-02-01T10:22:15.521 DEBUG:boto:Path: /vclfcn-18/obj-test-zone3 2024-02-01T10:22:15.521 DEBUG:boto:Data: 2024-02-01T10:22:15.521 DEBUG:boto:Headers: {} 2024-02-01T10:22:15.521 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:15.521 DEBUG:boto:Port: 8000 2024-02-01T10:22:15.521 DEBUG:boto:Params: {} 2024-02-01T10:22:15.522 DEBUG:boto:Token: None 2024-02-01T10:22:15.522 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:15 GMT /vclfcn-18/obj-test-zone3 2024-02-01T10:22:15.522 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:iAGmybKKDPIKxjvU8J8UsOOzVwg= 2024-02-01T10:22:15.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iAGmybKKDPIKxjvU8J8UsOOzVwg=', 'Content-Length': '0'} 2024-02-01T10:22:15.526 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', '6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000043f19c55c6d83d80-0065bb70d7-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:15.526 DEBUG:boto:path=/vclfcn-18/obj-test-zone3 2024-02-01T10:22:15.526 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3 2024-02-01T10:22:15.526 DEBUG:boto:Method: HEAD 2024-02-01T10:22:15.526 DEBUG:boto:Path: /vclfcn-18/obj-test-zone3 2024-02-01T10:22:15.526 DEBUG:boto:Data: 2024-02-01T10:22:15.527 DEBUG:boto:Headers: {} 2024-02-01T10:22:15.527 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:15.527 DEBUG:boto:Port: 8000 2024-02-01T10:22:15.527 DEBUG:boto:Params: {} 2024-02-01T10:22:15.527 DEBUG:boto:Token: None 2024-02-01T10:22:15.527 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:15 GMT /vclfcn-18/obj-test-zone3 2024-02-01T10:22:15.527 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:iAGmybKKDPIKxjvU8J8UsOOzVwg= 2024-02-01T10:22:15.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iAGmybKKDPIKxjvU8J8UsOOzVwg=', 'Content-Length': '0'} 2024-02-01T10:22:15.531 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', '6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bb9330800a605297-0065bb70d7-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:15.531 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-02-01T10:22:15.531 DEBUG:boto:path=/vclfcn-18/obj-test-zone3 2024-02-01T10:22:15.532 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3 2024-02-01T10:22:15.532 DEBUG:boto:path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:15.532 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:15.532 DEBUG:boto:Method: GET 2024-02-01T10:22:15.532 DEBUG:boto:Path: /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:15.532 DEBUG:boto:Data: 2024-02-01T10:22:15.532 DEBUG:boto:Headers: {} 2024-02-01T10:22:15.532 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:15.532 DEBUG:boto:Port: 8000 2024-02-01T10:22:15.532 DEBUG:boto:Params: {} 2024-02-01T10:22:15.533 DEBUG:boto:Token: None 2024-02-01T10:22:15.533 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:15 GMT /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:15.533 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1lL+qZo6Ajtfsro9NLdsNuHhcqI= 2024-02-01T10:22:15.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1lL+qZo6Ajtfsro9NLdsNuHhcqI=', 'Content-Length': '0'} 2024-02-01T10:22:15.536 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', '6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008192b10204fd8107-0065bb70d7-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:15.536 DEBUG:boto:path=/vclfcn-18/obj-test-zone3 2024-02-01T10:22:15.536 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3 2024-02-01T10:22:15.536 DEBUG:boto:path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:15.536 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:15.537 DEBUG:boto:Method: GET 2024-02-01T10:22:15.537 DEBUG:boto:Path: /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:15.537 DEBUG:boto:Data: 2024-02-01T10:22:15.537 DEBUG:boto:Headers: {} 2024-02-01T10:22:15.537 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:15.537 DEBUG:boto:Port: 8000 2024-02-01T10:22:15.537 DEBUG:boto:Params: {} 2024-02-01T10:22:15.537 DEBUG:boto:Token: None 2024-02-01T10:22:15.537 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:15 GMT /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:15.537 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1lL+qZo6Ajtfsro9NLdsNuHhcqI= 2024-02-01T10:22:15.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1lL+qZo6Ajtfsro9NLdsNuHhcqI=', 'Content-Length': '0'} 2024-02-01T10:22:15.540 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', '6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009e859964301f32b2-0065bb70d7-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:15.588 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:22:15.588 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:22:15.588 DEBUG:boto:path=/vclfcn-18/?prefix=obj-test-zone3 2024-02-01T10:22:15.588 DEBUG:boto:auth_path=/vclfcn-18/?prefix=obj-test-zone3 2024-02-01T10:22:15.589 DEBUG:boto:Method: GET 2024-02-01T10:22:15.589 DEBUG:boto:Path: /vclfcn-18/?prefix=obj-test-zone3 2024-02-01T10:22:15.589 DEBUG:boto:Data: 2024-02-01T10:22:15.589 DEBUG:boto:Headers: {} 2024-02-01T10:22:15.589 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:15.589 DEBUG:boto:Port: 8000 2024-02-01T10:22:15.589 DEBUG:boto:Params: {} 2024-02-01T10:22:15.589 DEBUG:boto:Token: None 2024-02-01T10:22:15.589 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:15 GMT /vclfcn-18/ 2024-02-01T10:22:15.590 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:3TwBtCmYeXITMvaVRg+vTOKO7IQ= 2024-02-01T10:22:15.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:3TwBtCmYeXITMvaVRg+vTOKO7IQ=', 'Content-Length': '0'} 2024-02-01T10:22:15.594 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9fb36f92fc3111d-0065bb70d7-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:15.594 DEBUG:boto:b'vclfcn-18obj-test-zone31000falseobj-test-zone32024-02-01T10:20:51.248Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:15.595 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:22:15.595 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:22:15.595 DEBUG:boto:path=/vclfcn-18/?prefix=obj-test-zone3 2024-02-01T10:22:15.595 DEBUG:boto:auth_path=/vclfcn-18/?prefix=obj-test-zone3 2024-02-01T10:22:15.595 DEBUG:boto:Method: GET 2024-02-01T10:22:15.595 DEBUG:boto:Path: /vclfcn-18/?prefix=obj-test-zone3 2024-02-01T10:22:15.595 DEBUG:boto:Data: 2024-02-01T10:22:15.595 DEBUG:boto:Headers: {} 2024-02-01T10:22:15.595 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:15.595 DEBUG:boto:Port: 8000 2024-02-01T10:22:15.595 DEBUG:boto:Params: {} 2024-02-01T10:22:15.596 DEBUG:boto:Token: None 2024-02-01T10:22:15.596 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:15 GMT /vclfcn-18/ 2024-02-01T10:22:15.596 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:3TwBtCmYeXITMvaVRg+vTOKO7IQ= 2024-02-01T10:22:15.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:3TwBtCmYeXITMvaVRg+vTOKO7IQ=', 'Content-Length': '0'} 2024-02-01T10:22:15.600 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004af1e47ff1dae078-0065bb70d7-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:15.640 DEBUG:boto:b'vclfcn-18obj-test-zone31000falseobj-test-zone32024-02-01T10:20:51.248Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:15.705 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-18 zones={test-zone2, test-zone1} 2024-02-01T10:22:15.705 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-18 zones={test-zone3, test-zone1} 2024-02-01T10:22:15.705 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:22:15.705 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:22:15.706 DEBUG:boto:Method: HEAD 2024-02-01T10:22:15.706 DEBUG:boto:Path: /vclfcn-18/ 2024-02-01T10:22:15.706 DEBUG:boto:Data: 2024-02-01T10:22:15.706 DEBUG:boto:Headers: {} 2024-02-01T10:22:15.706 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:15.706 DEBUG:boto:Port: 8001 2024-02-01T10:22:15.706 DEBUG:boto:Params: {} 2024-02-01T10:22:15.706 DEBUG:boto:Token: None 2024-02-01T10:22:15.706 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:15 GMT /vclfcn-18/ 2024-02-01T10:22:15.706 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:VPp6d8lDQmWvVhXKHQrNUTp4Lxw= 2024-02-01T10:22:15.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:VPp6d8lDQmWvVhXKHQrNUTp4Lxw=', 'Content-Length': '0'} 2024-02-01T10:22:15.711 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000076d5c4ab3d705e36-0065bb70d7-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:22:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:15.711 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:22:15.711 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:22:15.711 DEBUG:boto:Method: HEAD 2024-02-01T10:22:15.711 DEBUG:boto:Path: /vclfcn-18/ 2024-02-01T10:22:15.711 DEBUG:boto:Data: 2024-02-01T10:22:15.711 DEBUG:boto:Headers: {} 2024-02-01T10:22:15.712 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:15.712 DEBUG:boto:Port: 8000 2024-02-01T10:22:15.712 DEBUG:boto:Params: {} 2024-02-01T10:22:15.712 DEBUG:boto:Token: None 2024-02-01T10:22:15.712 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:15 GMT /vclfcn-18/ 2024-02-01T10:22:15.712 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:VPp6d8lDQmWvVhXKHQrNUTp4Lxw= 2024-02-01T10:22:15.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:VPp6d8lDQmWvVhXKHQrNUTp4Lxw=', 'Content-Length': '0'} 2024-02-01T10:22:15.716 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fb8d46ee8226609a-0065bb70d7-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:22:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:15.716 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:22:15.717 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:22:15.717 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:22:15.717 DEBUG:boto:path=/vclfcn-18/?versions&max-keys=999 2024-02-01T10:22:15.717 DEBUG:boto:auth_path=/vclfcn-18/?versions&max-keys=999 2024-02-01T10:22:15.717 DEBUG:boto:Method: GET 2024-02-01T10:22:15.717 DEBUG:boto:Path: /vclfcn-18/?versions&max-keys=999 2024-02-01T10:22:15.717 DEBUG:boto:Data: 2024-02-01T10:22:15.717 DEBUG:boto:Headers: {} 2024-02-01T10:22:15.717 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:15.717 DEBUG:boto:Port: 8001 2024-02-01T10:22:15.717 DEBUG:boto:Params: {} 2024-02-01T10:22:15.718 DEBUG:boto:Token: None 2024-02-01T10:22:15.718 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:15 GMT /vclfcn-18/?versions 2024-02-01T10:22:15.718 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:egao6lsYAOfb6RgXIq47E79KHSc= 2024-02-01T10:22:15.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:egao6lsYAOfb6RgXIq47E79KHSc=', 'Content-Length': '0'} 2024-02-01T10:22:15.722 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b52ea19d3c35731-0065bb70d7-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:15.722 DEBUG:boto:b'vclfcn-18999falsedummytxgLlJWZVbWbjqLWqkOtt87dUoD4zuntrue2024-02-01T10:20:34.645Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcKtrue2024-02-01T10:20:50.942Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutDtrue2024-02-01T10:20:51.094Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone36pAVLbw3HAwE0jxxbuxTIxwcA2p4q8Ntrue2024-02-01T10:20:51.248Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:15.723 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-02-01T10:22:15.723 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-02-01T10:22:15.723 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-02-01T10:22:15.723 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-02-01T10:22:15.723 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:22:15.723 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:22:15.724 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:22:15.724 DEBUG:boto:path=/vclfcn-18/?versions&max-keys=999 2024-02-01T10:22:15.724 DEBUG:boto:auth_path=/vclfcn-18/?versions&max-keys=999 2024-02-01T10:22:15.724 DEBUG:boto:Method: GET 2024-02-01T10:22:15.724 DEBUG:boto:Path: /vclfcn-18/?versions&max-keys=999 2024-02-01T10:22:15.724 DEBUG:boto:Data: 2024-02-01T10:22:15.724 DEBUG:boto:Headers: {} 2024-02-01T10:22:15.724 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:15.724 DEBUG:boto:Port: 8000 2024-02-01T10:22:15.724 DEBUG:boto:Params: {} 2024-02-01T10:22:15.724 DEBUG:boto:Token: None 2024-02-01T10:22:15.725 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:15 GMT /vclfcn-18/?versions 2024-02-01T10:22:15.725 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:egao6lsYAOfb6RgXIq47E79KHSc= 2024-02-01T10:22:15.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:egao6lsYAOfb6RgXIq47E79KHSc=', 'Content-Length': '0'} 2024-02-01T10:22:15.729 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019674df9b4c3d3f9-0065bb70d7-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:15.729 DEBUG:boto:b'vclfcn-18999falsedummytxgLlJWZVbWbjqLWqkOtt87dUoD4zuntrue2024-02-01T10:20:34.645Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcKtrue2024-02-01T10:20:50.942Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutDtrue2024-02-01T10:20:51.094Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone36pAVLbw3HAwE0jxxbuxTIxwcA2p4q8Ntrue2024-02-01T10:20:51.248Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:15.730 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-02-01T10:22:15.730 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-02-01T10:22:15.730 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-02-01T10:22:15.730 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-02-01T10:22:15.730 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:22:15.730 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:22:15.731 DEBUG:boto:path=/vclfcn-18/?versions&max-keys=999 2024-02-01T10:22:15.731 DEBUG:boto:auth_path=/vclfcn-18/?versions&max-keys=999 2024-02-01T10:22:15.731 DEBUG:boto:Method: GET 2024-02-01T10:22:15.731 DEBUG:boto:Path: /vclfcn-18/?versions&max-keys=999 2024-02-01T10:22:15.731 DEBUG:boto:Data: 2024-02-01T10:22:15.731 DEBUG:boto:Headers: {} 2024-02-01T10:22:15.731 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:15.731 DEBUG:boto:Port: 8001 2024-02-01T10:22:15.731 DEBUG:boto:Params: {} 2024-02-01T10:22:15.731 DEBUG:boto:Token: None 2024-02-01T10:22:15.732 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:15 GMT /vclfcn-18/?versions 2024-02-01T10:22:15.732 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:egao6lsYAOfb6RgXIq47E79KHSc= 2024-02-01T10:22:15.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:egao6lsYAOfb6RgXIq47E79KHSc=', 'Content-Length': '0'} 2024-02-01T10:22:15.736 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001dbb0f67d0f50c35-0065bb70d7-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:15.736 DEBUG:boto:b'vclfcn-18999falsedummytxgLlJWZVbWbjqLWqkOtt87dUoD4zuntrue2024-02-01T10:20:34.645Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcKtrue2024-02-01T10:20:50.942Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutDtrue2024-02-01T10:20:51.094Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone36pAVLbw3HAwE0jxxbuxTIxwcA2p4q8Ntrue2024-02-01T10:20:51.248Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:15.737 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:22:15.737 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:22:15.737 DEBUG:boto:path=/vclfcn-18/?versions&max-keys=999 2024-02-01T10:22:15.737 DEBUG:boto:auth_path=/vclfcn-18/?versions&max-keys=999 2024-02-01T10:22:15.737 DEBUG:boto:Method: GET 2024-02-01T10:22:15.737 DEBUG:boto:Path: /vclfcn-18/?versions&max-keys=999 2024-02-01T10:22:15.737 DEBUG:boto:Data: 2024-02-01T10:22:15.737 DEBUG:boto:Headers: {} 2024-02-01T10:22:15.737 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:15.738 DEBUG:boto:Port: 8000 2024-02-01T10:22:15.738 DEBUG:boto:Params: {} 2024-02-01T10:22:15.738 DEBUG:boto:Token: None 2024-02-01T10:22:15.738 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:15 GMT /vclfcn-18/?versions 2024-02-01T10:22:15.738 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:egao6lsYAOfb6RgXIq47E79KHSc= 2024-02-01T10:22:15.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:egao6lsYAOfb6RgXIq47E79KHSc=', 'Content-Length': '0'} 2024-02-01T10:22:15.742 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc420fb1f92411de-0065bb70d7-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:15.742 DEBUG:boto:b'vclfcn-18999falsedummytxgLlJWZVbWbjqLWqkOtt87dUoD4zuntrue2024-02-01T10:20:34.645Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcKtrue2024-02-01T10:20:50.942Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutDtrue2024-02-01T10:20:51.094Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone36pAVLbw3HAwE0jxxbuxTIxwcA2p4q8Ntrue2024-02-01T10:20:51.248Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:15.743 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-02-01T10:22:15.743 DEBUG:boto:path=/vclfcn-18/dummy 2024-02-01T10:22:15.743 DEBUG:boto:auth_path=/vclfcn-18/dummy 2024-02-01T10:22:15.743 DEBUG:boto:path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:15.743 DEBUG:boto:auth_path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:15.743 DEBUG:boto:Method: GET 2024-02-01T10:22:15.744 DEBUG:boto:Path: /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:15.744 DEBUG:boto:Data: 2024-02-01T10:22:15.744 DEBUG:boto:Headers: {} 2024-02-01T10:22:15.744 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:15.744 DEBUG:boto:Port: 8001 2024-02-01T10:22:15.744 DEBUG:boto:Params: {} 2024-02-01T10:22:15.744 DEBUG:boto:Token: None 2024-02-01T10:22:15.744 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:15 GMT /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:15.744 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8cDrs/YCNEY2PXkwNSHLzmMaGOs= 2024-02-01T10:22:15.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8cDrs/YCNEY2PXkwNSHLzmMaGOs=', 'Content-Length': '0'} 2024-02-01T10:22:15.747 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:34 GMT'), ('x-amz-version-id', 'txgLlJWZVbWbjqLWqkOtt87dUoD4zun'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000faa2ed395280d258-0065bb70d7-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:15.747 DEBUG:boto:path=/vclfcn-18/dummy 2024-02-01T10:22:15.747 DEBUG:boto:auth_path=/vclfcn-18/dummy 2024-02-01T10:22:15.748 DEBUG:boto:path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:15.748 DEBUG:boto:auth_path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:15.748 DEBUG:boto:Method: GET 2024-02-01T10:22:15.748 DEBUG:boto:Path: /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:15.748 DEBUG:boto:Data: 2024-02-01T10:22:15.748 DEBUG:boto:Headers: {} 2024-02-01T10:22:15.748 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:15.748 DEBUG:boto:Port: 8000 2024-02-01T10:22:15.748 DEBUG:boto:Params: {} 2024-02-01T10:22:15.749 DEBUG:boto:Token: None 2024-02-01T10:22:15.749 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:15 GMT /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:15.749 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8cDrs/YCNEY2PXkwNSHLzmMaGOs= 2024-02-01T10:22:15.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8cDrs/YCNEY2PXkwNSHLzmMaGOs=', 'Content-Length': '0'} 2024-02-01T10:22:15.751 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:34 GMT'), ('x-amz-version-id', 'txgLlJWZVbWbjqLWqkOtt87dUoD4zun'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009b4359ca586cfed6-0065bb70d7-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:15.752 DEBUG:boto:path=/vclfcn-18/dummy 2024-02-01T10:22:15.752 DEBUG:boto:auth_path=/vclfcn-18/dummy 2024-02-01T10:22:15.752 DEBUG:boto:path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:15.752 DEBUG:boto:auth_path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:15.752 DEBUG:boto:Method: HEAD 2024-02-01T10:22:15.752 DEBUG:boto:Path: /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:15.752 DEBUG:boto:Data: 2024-02-01T10:22:15.752 DEBUG:boto:Headers: {} 2024-02-01T10:22:15.752 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:15.753 DEBUG:boto:Port: 8001 2024-02-01T10:22:15.753 DEBUG:boto:Params: {} 2024-02-01T10:22:15.753 DEBUG:boto:Token: None 2024-02-01T10:22:15.753 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:15 GMT /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:15.753 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vwkVFDKr2aoHq47qsBcIG2MnLoc= 2024-02-01T10:22:15.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vwkVFDKr2aoHq47qsBcIG2MnLoc=', 'Content-Length': '0'} 2024-02-01T10:22:15.755 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:34 GMT'), ('x-amz-version-id', 'txgLlJWZVbWbjqLWqkOtt87dUoD4zun'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000dbfd23baa0787732-0065bb70d7-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:15.756 DEBUG:boto:path=/vclfcn-18/dummy 2024-02-01T10:22:15.756 DEBUG:boto:auth_path=/vclfcn-18/dummy 2024-02-01T10:22:15.756 DEBUG:boto:path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:15.756 DEBUG:boto:auth_path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:15.756 DEBUG:boto:Method: HEAD 2024-02-01T10:22:15.756 DEBUG:boto:Path: /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:15.756 DEBUG:boto:Data: 2024-02-01T10:22:15.756 DEBUG:boto:Headers: {} 2024-02-01T10:22:15.756 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:15.757 DEBUG:boto:Port: 8000 2024-02-01T10:22:15.757 DEBUG:boto:Params: {} 2024-02-01T10:22:15.757 DEBUG:boto:Token: None 2024-02-01T10:22:15.757 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:15 GMT /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:15.757 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vwkVFDKr2aoHq47qsBcIG2MnLoc= 2024-02-01T10:22:15.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vwkVFDKr2aoHq47qsBcIG2MnLoc=', 'Content-Length': '0'} 2024-02-01T10:22:15.760 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:34 GMT'), ('x-amz-version-id', 'txgLlJWZVbWbjqLWqkOtt87dUoD4zun'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006d74c4e364830cc7-0065bb70d7-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:15.760 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-02-01T10:22:15.760 DEBUG:boto:path=/vclfcn-18/dummy 2024-02-01T10:22:15.760 DEBUG:boto:auth_path=/vclfcn-18/dummy 2024-02-01T10:22:15.760 DEBUG:boto:path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:15.760 DEBUG:boto:auth_path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:15.760 DEBUG:boto:Method: GET 2024-02-01T10:22:15.761 DEBUG:boto:Path: /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:15.761 DEBUG:boto:Data: 2024-02-01T10:22:15.761 DEBUG:boto:Headers: {} 2024-02-01T10:22:15.761 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:15.761 DEBUG:boto:Port: 8001 2024-02-01T10:22:15.761 DEBUG:boto:Params: {} 2024-02-01T10:22:15.761 DEBUG:boto:Token: None 2024-02-01T10:22:15.761 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:15 GMT /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:15.761 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8cDrs/YCNEY2PXkwNSHLzmMaGOs= 2024-02-01T10:22:15.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8cDrs/YCNEY2PXkwNSHLzmMaGOs=', 'Content-Length': '0'} 2024-02-01T10:22:15.764 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:34 GMT'), ('x-amz-version-id', 'txgLlJWZVbWbjqLWqkOtt87dUoD4zun'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004070cb2d58940ab3-0065bb70d7-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:15.764 DEBUG:boto:path=/vclfcn-18/dummy 2024-02-01T10:22:15.764 DEBUG:boto:auth_path=/vclfcn-18/dummy 2024-02-01T10:22:15.764 DEBUG:boto:path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:15.764 DEBUG:boto:auth_path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:15.765 DEBUG:boto:Method: GET 2024-02-01T10:22:15.765 DEBUG:boto:Path: /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:15.765 DEBUG:boto:Data: 2024-02-01T10:22:15.765 DEBUG:boto:Headers: {} 2024-02-01T10:22:15.765 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:15.765 DEBUG:boto:Port: 8000 2024-02-01T10:22:15.765 DEBUG:boto:Params: {} 2024-02-01T10:22:15.765 DEBUG:boto:Token: None 2024-02-01T10:22:15.765 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:15 GMT /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:15.765 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8cDrs/YCNEY2PXkwNSHLzmMaGOs= 2024-02-01T10:22:15.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8cDrs/YCNEY2PXkwNSHLzmMaGOs=', 'Content-Length': '0'} 2024-02-01T10:22:15.768 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:34 GMT'), ('x-amz-version-id', 'txgLlJWZVbWbjqLWqkOtt87dUoD4zun'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d9d4913fbdc92c6d-0065bb70d7-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:15.768 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:22:15.769 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:22:15.769 DEBUG:boto:path=/vclfcn-18/?prefix=dummy 2024-02-01T10:22:15.769 DEBUG:boto:auth_path=/vclfcn-18/?prefix=dummy 2024-02-01T10:22:15.769 DEBUG:boto:Method: GET 2024-02-01T10:22:15.769 DEBUG:boto:Path: /vclfcn-18/?prefix=dummy 2024-02-01T10:22:15.769 DEBUG:boto:Data: 2024-02-01T10:22:15.769 DEBUG:boto:Headers: {} 2024-02-01T10:22:15.769 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:15.769 DEBUG:boto:Port: 8001 2024-02-01T10:22:15.769 DEBUG:boto:Params: {} 2024-02-01T10:22:15.769 DEBUG:boto:Token: None 2024-02-01T10:22:15.770 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:15 GMT /vclfcn-18/ 2024-02-01T10:22:15.770 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:3TwBtCmYeXITMvaVRg+vTOKO7IQ= 2024-02-01T10:22:15.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:3TwBtCmYeXITMvaVRg+vTOKO7IQ=', 'Content-Length': '0'} 2024-02-01T10:22:15.774 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000423ad848c1b0e8fc-0065bb70d7-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:15.816 DEBUG:boto:b'vclfcn-18dummy1000falsedummy2024-02-01T10:20:34.645Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:15.817 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:22:15.817 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:22:15.817 DEBUG:boto:path=/vclfcn-18/?prefix=dummy 2024-02-01T10:22:15.817 DEBUG:boto:auth_path=/vclfcn-18/?prefix=dummy 2024-02-01T10:22:15.817 DEBUG:boto:Method: GET 2024-02-01T10:22:15.817 DEBUG:boto:Path: /vclfcn-18/?prefix=dummy 2024-02-01T10:22:15.817 DEBUG:boto:Data: 2024-02-01T10:22:15.817 DEBUG:boto:Headers: {} 2024-02-01T10:22:15.817 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:15.817 DEBUG:boto:Port: 8000 2024-02-01T10:22:15.817 DEBUG:boto:Params: {} 2024-02-01T10:22:15.818 DEBUG:boto:Token: None 2024-02-01T10:22:15.818 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:15 GMT /vclfcn-18/ 2024-02-01T10:22:15.818 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:3TwBtCmYeXITMvaVRg+vTOKO7IQ= 2024-02-01T10:22:15.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:3TwBtCmYeXITMvaVRg+vTOKO7IQ=', 'Content-Length': '0'} 2024-02-01T10:22:15.822 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1a8d37d9f090dc9-0065bb70d7-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:15.823 DEBUG:boto:b'vclfcn-18dummy1000falsedummy2024-02-01T10:20:34.645Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:15.823 DEBUG:boto:path=/vclfcn-18/dummy 2024-02-01T10:22:15.823 DEBUG:boto:auth_path=/vclfcn-18/dummy 2024-02-01T10:22:15.823 DEBUG:boto:Method: HEAD 2024-02-01T10:22:15.823 DEBUG:boto:Path: /vclfcn-18/dummy 2024-02-01T10:22:15.823 DEBUG:boto:Data: 2024-02-01T10:22:15.823 DEBUG:boto:Headers: {} 2024-02-01T10:22:15.824 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:15.824 DEBUG:boto:Port: 8001 2024-02-01T10:22:15.824 DEBUG:boto:Params: {} 2024-02-01T10:22:15.824 DEBUG:boto:Token: None 2024-02-01T10:22:15.824 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:15 GMT /vclfcn-18/dummy 2024-02-01T10:22:15.824 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ffeHgvvJPZoD4l/WB0G8IeDwtmo= 2024-02-01T10:22:15.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ffeHgvvJPZoD4l/WB0G8IeDwtmo=', 'Content-Length': '0'} 2024-02-01T10:22:15.828 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:34 GMT'), ('x-amz-version-id', 'txgLlJWZVbWbjqLWqkOtt87dUoD4zun'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b3a44cbd80c5ecbb-0065bb70d7-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:15.828 DEBUG:boto:path=/vclfcn-18/dummy 2024-02-01T10:22:15.828 DEBUG:boto:auth_path=/vclfcn-18/dummy 2024-02-01T10:22:15.828 DEBUG:boto:Method: HEAD 2024-02-01T10:22:15.828 DEBUG:boto:Path: /vclfcn-18/dummy 2024-02-01T10:22:15.828 DEBUG:boto:Data: 2024-02-01T10:22:15.828 DEBUG:boto:Headers: {} 2024-02-01T10:22:15.829 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:15.829 DEBUG:boto:Port: 8000 2024-02-01T10:22:15.829 DEBUG:boto:Params: {} 2024-02-01T10:22:15.829 DEBUG:boto:Token: None 2024-02-01T10:22:15.829 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:15 GMT /vclfcn-18/dummy 2024-02-01T10:22:15.829 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ffeHgvvJPZoD4l/WB0G8IeDwtmo= 2024-02-01T10:22:15.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ffeHgvvJPZoD4l/WB0G8IeDwtmo=', 'Content-Length': '0'} 2024-02-01T10:22:15.833 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:34 GMT'), ('x-amz-version-id', 'txgLlJWZVbWbjqLWqkOtt87dUoD4zun'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000129a3cedcdc3bb87-0065bb70d7-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:15.833 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-02-01T10:22:15.833 DEBUG:boto:path=/vclfcn-18/dummy 2024-02-01T10:22:15.834 DEBUG:boto:auth_path=/vclfcn-18/dummy 2024-02-01T10:22:15.834 DEBUG:boto:path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:15.834 DEBUG:boto:auth_path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:15.834 DEBUG:boto:Method: GET 2024-02-01T10:22:15.834 DEBUG:boto:Path: /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:15.834 DEBUG:boto:Data: 2024-02-01T10:22:15.834 DEBUG:boto:Headers: {} 2024-02-01T10:22:15.834 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:15.834 DEBUG:boto:Port: 8001 2024-02-01T10:22:15.834 DEBUG:boto:Params: {} 2024-02-01T10:22:15.834 DEBUG:boto:Token: None 2024-02-01T10:22:15.835 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:15 GMT /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:15.835 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8cDrs/YCNEY2PXkwNSHLzmMaGOs= 2024-02-01T10:22:15.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8cDrs/YCNEY2PXkwNSHLzmMaGOs=', 'Content-Length': '0'} 2024-02-01T10:22:15.837 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:34 GMT'), ('x-amz-version-id', 'txgLlJWZVbWbjqLWqkOtt87dUoD4zun'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000012b6887250965d71-0065bb70d7-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:15.838 DEBUG:boto:path=/vclfcn-18/dummy 2024-02-01T10:22:15.838 DEBUG:boto:auth_path=/vclfcn-18/dummy 2024-02-01T10:22:15.838 DEBUG:boto:path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:15.838 DEBUG:boto:auth_path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:15.838 DEBUG:boto:Method: GET 2024-02-01T10:22:15.838 DEBUG:boto:Path: /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:15.838 DEBUG:boto:Data: 2024-02-01T10:22:15.838 DEBUG:boto:Headers: {} 2024-02-01T10:22:15.838 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:15.839 DEBUG:boto:Port: 8000 2024-02-01T10:22:15.839 DEBUG:boto:Params: {} 2024-02-01T10:22:15.839 DEBUG:boto:Token: None 2024-02-01T10:22:15.839 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:15 GMT /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:15.839 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8cDrs/YCNEY2PXkwNSHLzmMaGOs= 2024-02-01T10:22:15.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8cDrs/YCNEY2PXkwNSHLzmMaGOs=', 'Content-Length': '0'} 2024-02-01T10:22:15.841 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:34 GMT'), ('x-amz-version-id', 'txgLlJWZVbWbjqLWqkOtt87dUoD4zun'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fa3bdc1776f1781a-0065bb70d7-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:15.842 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:22:15.842 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:22:15.842 DEBUG:boto:path=/vclfcn-18/?prefix=dummy 2024-02-01T10:22:15.842 DEBUG:boto:auth_path=/vclfcn-18/?prefix=dummy 2024-02-01T10:22:15.842 DEBUG:boto:Method: GET 2024-02-01T10:22:15.842 DEBUG:boto:Path: /vclfcn-18/?prefix=dummy 2024-02-01T10:22:15.842 DEBUG:boto:Data: 2024-02-01T10:22:15.842 DEBUG:boto:Headers: {} 2024-02-01T10:22:15.843 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:15.843 DEBUG:boto:Port: 8001 2024-02-01T10:22:15.843 DEBUG:boto:Params: {} 2024-02-01T10:22:15.843 DEBUG:boto:Token: None 2024-02-01T10:22:15.843 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:15 GMT /vclfcn-18/ 2024-02-01T10:22:15.843 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:3TwBtCmYeXITMvaVRg+vTOKO7IQ= 2024-02-01T10:22:15.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:3TwBtCmYeXITMvaVRg+vTOKO7IQ=', 'Content-Length': '0'} 2024-02-01T10:22:15.847 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000989a2cc6817cecf3-0065bb70d7-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:15.848 DEBUG:boto:b'vclfcn-18dummy1000falsedummy2024-02-01T10:20:34.645Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:15.848 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:22:15.848 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:22:15.848 DEBUG:boto:path=/vclfcn-18/?prefix=dummy 2024-02-01T10:22:15.848 DEBUG:boto:auth_path=/vclfcn-18/?prefix=dummy 2024-02-01T10:22:15.849 DEBUG:boto:Method: GET 2024-02-01T10:22:15.849 DEBUG:boto:Path: /vclfcn-18/?prefix=dummy 2024-02-01T10:22:15.849 DEBUG:boto:Data: 2024-02-01T10:22:15.849 DEBUG:boto:Headers: {} 2024-02-01T10:22:15.849 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:15.849 DEBUG:boto:Port: 8000 2024-02-01T10:22:15.849 DEBUG:boto:Params: {} 2024-02-01T10:22:15.849 DEBUG:boto:Token: None 2024-02-01T10:22:15.849 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:15 GMT /vclfcn-18/ 2024-02-01T10:22:15.849 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:3TwBtCmYeXITMvaVRg+vTOKO7IQ= 2024-02-01T10:22:15.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:3TwBtCmYeXITMvaVRg+vTOKO7IQ=', 'Content-Length': '0'} 2024-02-01T10:22:15.853 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009bf58c60bbd2a28e-0065bb70d7-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:15.896 DEBUG:boto:b'vclfcn-18dummy1000falsedummy2024-02-01T10:20:34.645Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:15.897 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-02-01T10:22:15.897 DEBUG:boto:path=/vclfcn-18/obj-test-zone1 2024-02-01T10:22:15.897 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1 2024-02-01T10:22:15.897 DEBUG:boto:path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:15.897 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:15.897 DEBUG:boto:Method: GET 2024-02-01T10:22:15.897 DEBUG:boto:Path: /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:15.898 DEBUG:boto:Data: 2024-02-01T10:22:15.898 DEBUG:boto:Headers: {} 2024-02-01T10:22:15.898 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:15.898 DEBUG:boto:Port: 8001 2024-02-01T10:22:15.898 DEBUG:boto:Params: {} 2024-02-01T10:22:15.898 DEBUG:boto:Token: None 2024-02-01T10:22:15.898 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:15 GMT /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:15.898 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:DwBhLVdtDLnT14LZTOcW0YqWOrM= 2024-02-01T10:22:15.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:DwBhLVdtDLnT14LZTOcW0YqWOrM=', 'Content-Length': '0'} 2024-02-01T10:22:15.902 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000049996cafc8f88b51-0065bb70d7-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:15.902 DEBUG:boto:path=/vclfcn-18/obj-test-zone1 2024-02-01T10:22:15.903 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1 2024-02-01T10:22:15.903 DEBUG:boto:path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:15.903 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:15.903 DEBUG:boto:Method: GET 2024-02-01T10:22:15.903 DEBUG:boto:Path: /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:15.903 DEBUG:boto:Data: 2024-02-01T10:22:15.903 DEBUG:boto:Headers: {} 2024-02-01T10:22:15.903 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:15.903 DEBUG:boto:Port: 8000 2024-02-01T10:22:15.903 DEBUG:boto:Params: {} 2024-02-01T10:22:15.904 DEBUG:boto:Token: None 2024-02-01T10:22:15.904 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:15 GMT /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:15.904 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:DwBhLVdtDLnT14LZTOcW0YqWOrM= 2024-02-01T10:22:15.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:DwBhLVdtDLnT14LZTOcW0YqWOrM=', 'Content-Length': '0'} 2024-02-01T10:22:15.907 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007f0fea075f5d473e-0065bb70d7-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:15.908 DEBUG:boto:path=/vclfcn-18/obj-test-zone1 2024-02-01T10:22:15.908 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1 2024-02-01T10:22:15.908 DEBUG:boto:path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:15.908 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:15.908 DEBUG:boto:Method: HEAD 2024-02-01T10:22:15.908 DEBUG:boto:Path: /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:15.908 DEBUG:boto:Data: 2024-02-01T10:22:15.909 DEBUG:boto:Headers: {} 2024-02-01T10:22:15.909 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:15.909 DEBUG:boto:Port: 8001 2024-02-01T10:22:15.909 DEBUG:boto:Params: {} 2024-02-01T10:22:15.909 DEBUG:boto:Token: None 2024-02-01T10:22:15.909 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:15 GMT /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:15.909 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:AfdndOOshBA7Bwrxda2GhCuVwhw= 2024-02-01T10:22:15.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:AfdndOOshBA7Bwrxda2GhCuVwhw=', 'Content-Length': '0'} 2024-02-01T10:22:15.912 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000819b575474998ac3-0065bb70d7-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:15.913 DEBUG:boto:path=/vclfcn-18/obj-test-zone1 2024-02-01T10:22:15.913 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1 2024-02-01T10:22:15.913 DEBUG:boto:path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:15.913 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:15.913 DEBUG:boto:Method: HEAD 2024-02-01T10:22:15.913 DEBUG:boto:Path: /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:15.914 DEBUG:boto:Data: 2024-02-01T10:22:15.914 DEBUG:boto:Headers: {} 2024-02-01T10:22:15.914 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:15.914 DEBUG:boto:Port: 8000 2024-02-01T10:22:15.914 DEBUG:boto:Params: {} 2024-02-01T10:22:15.914 DEBUG:boto:Token: None 2024-02-01T10:22:15.914 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:15 GMT /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:15.914 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:AfdndOOshBA7Bwrxda2GhCuVwhw= 2024-02-01T10:22:15.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:AfdndOOshBA7Bwrxda2GhCuVwhw=', 'Content-Length': '0'} 2024-02-01T10:22:15.918 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000edff397866330c97-0065bb70d7-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:15.918 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-02-01T10:22:15.918 DEBUG:boto:path=/vclfcn-18/obj-test-zone1 2024-02-01T10:22:15.918 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1 2024-02-01T10:22:15.918 DEBUG:boto:path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:15.918 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:15.918 DEBUG:boto:Method: GET 2024-02-01T10:22:15.918 DEBUG:boto:Path: /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:15.919 DEBUG:boto:Data: 2024-02-01T10:22:15.919 DEBUG:boto:Headers: {} 2024-02-01T10:22:15.919 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:15.919 DEBUG:boto:Port: 8001 2024-02-01T10:22:15.919 DEBUG:boto:Params: {} 2024-02-01T10:22:15.919 DEBUG:boto:Token: None 2024-02-01T10:22:15.919 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:15 GMT /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:15.920 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:DwBhLVdtDLnT14LZTOcW0YqWOrM= 2024-02-01T10:22:15.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:DwBhLVdtDLnT14LZTOcW0YqWOrM=', 'Content-Length': '0'} 2024-02-01T10:22:15.923 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003255e28b7fc195c0-0065bb70d7-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:15.968 DEBUG:boto:path=/vclfcn-18/obj-test-zone1 2024-02-01T10:22:15.968 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1 2024-02-01T10:22:15.968 DEBUG:boto:path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:15.969 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:15.969 DEBUG:boto:Method: GET 2024-02-01T10:22:15.969 DEBUG:boto:Path: /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:15.969 DEBUG:boto:Data: 2024-02-01T10:22:15.969 DEBUG:boto:Headers: {} 2024-02-01T10:22:15.969 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:15.969 DEBUG:boto:Port: 8000 2024-02-01T10:22:15.969 DEBUG:boto:Params: {} 2024-02-01T10:22:15.969 DEBUG:boto:Token: None 2024-02-01T10:22:15.970 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:15 GMT /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:15.970 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:DwBhLVdtDLnT14LZTOcW0YqWOrM= 2024-02-01T10:22:15.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:DwBhLVdtDLnT14LZTOcW0YqWOrM=', 'Content-Length': '0'} 2024-02-01T10:22:15.973 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007ea1c50ce29a66e4-0065bb70d7-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:15.973 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:22:15.973 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:22:15.973 DEBUG:boto:path=/vclfcn-18/?prefix=obj-test-zone1 2024-02-01T10:22:15.973 DEBUG:boto:auth_path=/vclfcn-18/?prefix=obj-test-zone1 2024-02-01T10:22:15.974 DEBUG:boto:Method: GET 2024-02-01T10:22:15.974 DEBUG:boto:Path: /vclfcn-18/?prefix=obj-test-zone1 2024-02-01T10:22:15.974 DEBUG:boto:Data: 2024-02-01T10:22:15.974 DEBUG:boto:Headers: {} 2024-02-01T10:22:15.974 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:15.974 DEBUG:boto:Port: 8001 2024-02-01T10:22:15.974 DEBUG:boto:Params: {} 2024-02-01T10:22:15.974 DEBUG:boto:Token: None 2024-02-01T10:22:15.974 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:15 GMT /vclfcn-18/ 2024-02-01T10:22:15.974 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:3TwBtCmYeXITMvaVRg+vTOKO7IQ= 2024-02-01T10:22:15.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:3TwBtCmYeXITMvaVRg+vTOKO7IQ=', 'Content-Length': '0'} 2024-02-01T10:22:15.978 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000599f81c687857967-0065bb70d7-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:15.979 DEBUG:boto:b'vclfcn-18obj-test-zone11000falseobj-test-zone12024-02-01T10:20:50.942Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:15.979 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:22:15.979 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:22:15.979 DEBUG:boto:path=/vclfcn-18/?prefix=obj-test-zone1 2024-02-01T10:22:15.979 DEBUG:boto:auth_path=/vclfcn-18/?prefix=obj-test-zone1 2024-02-01T10:22:15.979 DEBUG:boto:Method: GET 2024-02-01T10:22:15.979 DEBUG:boto:Path: /vclfcn-18/?prefix=obj-test-zone1 2024-02-01T10:22:15.980 DEBUG:boto:Data: 2024-02-01T10:22:15.980 DEBUG:boto:Headers: {} 2024-02-01T10:22:15.980 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:15.980 DEBUG:boto:Port: 8000 2024-02-01T10:22:15.980 DEBUG:boto:Params: {} 2024-02-01T10:22:15.980 DEBUG:boto:Token: None 2024-02-01T10:22:15.980 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:15 GMT /vclfcn-18/ 2024-02-01T10:22:15.980 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:3TwBtCmYeXITMvaVRg+vTOKO7IQ= 2024-02-01T10:22:15.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:3TwBtCmYeXITMvaVRg+vTOKO7IQ=', 'Content-Length': '0'} 2024-02-01T10:22:15.984 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063b5df6763e5f74a-0065bb70d7-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:15.985 DEBUG:boto:b'vclfcn-18obj-test-zone11000falseobj-test-zone12024-02-01T10:20:50.942Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:15.985 DEBUG:boto:path=/vclfcn-18/obj-test-zone1 2024-02-01T10:22:15.985 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1 2024-02-01T10:22:15.985 DEBUG:boto:Method: HEAD 2024-02-01T10:22:15.985 DEBUG:boto:Path: /vclfcn-18/obj-test-zone1 2024-02-01T10:22:15.985 DEBUG:boto:Data: 2024-02-01T10:22:15.986 DEBUG:boto:Headers: {} 2024-02-01T10:22:15.986 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:15.986 DEBUG:boto:Port: 8001 2024-02-01T10:22:15.986 DEBUG:boto:Params: {} 2024-02-01T10:22:15.986 DEBUG:boto:Token: None 2024-02-01T10:22:15.986 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:15 GMT /vclfcn-18/obj-test-zone1 2024-02-01T10:22:15.986 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:6FD+V+gf+5Ic9ZhyfkooXQr9GFo= 2024-02-01T10:22:15.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:6FD+V+gf+5Ic9ZhyfkooXQr9GFo=', 'Content-Length': '0'} 2024-02-01T10:22:15.990 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000df564c9a3828a9bc-0065bb70d7-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:15.990 DEBUG:boto:path=/vclfcn-18/obj-test-zone1 2024-02-01T10:22:15.990 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1 2024-02-01T10:22:15.990 DEBUG:boto:Method: HEAD 2024-02-01T10:22:15.990 DEBUG:boto:Path: /vclfcn-18/obj-test-zone1 2024-02-01T10:22:15.990 DEBUG:boto:Data: 2024-02-01T10:22:15.990 DEBUG:boto:Headers: {} 2024-02-01T10:22:15.990 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:15.991 DEBUG:boto:Port: 8000 2024-02-01T10:22:15.991 DEBUG:boto:Params: {} 2024-02-01T10:22:15.991 DEBUG:boto:Token: None 2024-02-01T10:22:15.991 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:15 GMT /vclfcn-18/obj-test-zone1 2024-02-01T10:22:15.991 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:6FD+V+gf+5Ic9ZhyfkooXQr9GFo= 2024-02-01T10:22:15.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:6FD+V+gf+5Ic9ZhyfkooXQr9GFo=', 'Content-Length': '0'} 2024-02-01T10:22:15.996 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a97f5029a4d5238b-0065bb70d7-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:15.997 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-02-01T10:22:15.997 DEBUG:boto:path=/vclfcn-18/obj-test-zone1 2024-02-01T10:22:15.997 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1 2024-02-01T10:22:15.997 DEBUG:boto:path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:15.997 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:15.997 DEBUG:boto:Method: GET 2024-02-01T10:22:15.998 DEBUG:boto:Path: /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:15.998 DEBUG:boto:Data: 2024-02-01T10:22:15.998 DEBUG:boto:Headers: {} 2024-02-01T10:22:15.998 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:15.998 DEBUG:boto:Port: 8001 2024-02-01T10:22:15.998 DEBUG:boto:Params: {} 2024-02-01T10:22:15.998 DEBUG:boto:Token: None 2024-02-01T10:22:15.998 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:15 GMT /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:15.998 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:DwBhLVdtDLnT14LZTOcW0YqWOrM= 2024-02-01T10:22:15.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:DwBhLVdtDLnT14LZTOcW0YqWOrM=', 'Content-Length': '0'} 2024-02-01T10:22:16.001 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b581b9fd59af4325-0065bb70d8-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:16.048 DEBUG:boto:path=/vclfcn-18/obj-test-zone1 2024-02-01T10:22:16.048 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1 2024-02-01T10:22:16.048 DEBUG:boto:path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:16.048 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:16.049 DEBUG:boto:Method: GET 2024-02-01T10:22:16.049 DEBUG:boto:Path: /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:16.049 DEBUG:boto:Data: 2024-02-01T10:22:16.049 DEBUG:boto:Headers: {} 2024-02-01T10:22:16.049 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:16.049 DEBUG:boto:Port: 8000 2024-02-01T10:22:16.049 DEBUG:boto:Params: {} 2024-02-01T10:22:16.049 DEBUG:boto:Token: None 2024-02-01T10:22:16.049 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:16 GMT /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:16.050 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ht0ASBuefeB07Ax9LQRgGl9GkNI= 2024-02-01T10:22:16.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ht0ASBuefeB07Ax9LQRgGl9GkNI=', 'Content-Length': '0'} 2024-02-01T10:22:16.053 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000692a8d7a2cb40761-0065bb70d8-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:16.053 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:22:16.053 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:22:16.053 DEBUG:boto:path=/vclfcn-18/?prefix=obj-test-zone1 2024-02-01T10:22:16.053 DEBUG:boto:auth_path=/vclfcn-18/?prefix=obj-test-zone1 2024-02-01T10:22:16.054 DEBUG:boto:Method: GET 2024-02-01T10:22:16.054 DEBUG:boto:Path: /vclfcn-18/?prefix=obj-test-zone1 2024-02-01T10:22:16.054 DEBUG:boto:Data: 2024-02-01T10:22:16.054 DEBUG:boto:Headers: {} 2024-02-01T10:22:16.054 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:16.054 DEBUG:boto:Port: 8001 2024-02-01T10:22:16.054 DEBUG:boto:Params: {} 2024-02-01T10:22:16.054 DEBUG:boto:Token: None 2024-02-01T10:22:16.054 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:16 GMT /vclfcn-18/ 2024-02-01T10:22:16.054 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ysLB0yg/KO94C8n2rwJHreWKTGw= 2024-02-01T10:22:16.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ysLB0yg/KO94C8n2rwJHreWKTGw=', 'Content-Length': '0'} 2024-02-01T10:22:16.058 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000357f60222503e9ff-0065bb70d8-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:16.059 DEBUG:boto:b'vclfcn-18obj-test-zone11000falseobj-test-zone12024-02-01T10:20:50.942Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:16.059 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:22:16.059 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:22:16.059 DEBUG:boto:path=/vclfcn-18/?prefix=obj-test-zone1 2024-02-01T10:22:16.060 DEBUG:boto:auth_path=/vclfcn-18/?prefix=obj-test-zone1 2024-02-01T10:22:16.060 DEBUG:boto:Method: GET 2024-02-01T10:22:16.060 DEBUG:boto:Path: /vclfcn-18/?prefix=obj-test-zone1 2024-02-01T10:22:16.060 DEBUG:boto:Data: 2024-02-01T10:22:16.060 DEBUG:boto:Headers: {} 2024-02-01T10:22:16.060 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:16.060 DEBUG:boto:Port: 8000 2024-02-01T10:22:16.060 DEBUG:boto:Params: {} 2024-02-01T10:22:16.060 DEBUG:boto:Token: None 2024-02-01T10:22:16.060 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:16 GMT /vclfcn-18/ 2024-02-01T10:22:16.061 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ysLB0yg/KO94C8n2rwJHreWKTGw= 2024-02-01T10:22:16.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ysLB0yg/KO94C8n2rwJHreWKTGw=', 'Content-Length': '0'} 2024-02-01T10:22:16.065 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099935da34d8624d5-0065bb70d8-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:16.065 DEBUG:boto:b'vclfcn-18obj-test-zone11000falseobj-test-zone12024-02-01T10:20:50.942Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:16.066 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-02-01T10:22:16.066 DEBUG:boto:path=/vclfcn-18/obj-test-zone2 2024-02-01T10:22:16.066 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2 2024-02-01T10:22:16.066 DEBUG:boto:path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:16.066 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:16.066 DEBUG:boto:Method: GET 2024-02-01T10:22:16.067 DEBUG:boto:Path: /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:16.067 DEBUG:boto:Data: 2024-02-01T10:22:16.067 DEBUG:boto:Headers: {} 2024-02-01T10:22:16.067 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:16.067 DEBUG:boto:Port: 8001 2024-02-01T10:22:16.067 DEBUG:boto:Params: {} 2024-02-01T10:22:16.067 DEBUG:boto:Token: None 2024-02-01T10:22:16.067 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:16 GMT /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:16.067 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:uaX8MAPCOYWWMHf1lwF4JBmTcUA= 2024-02-01T10:22:16.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:uaX8MAPCOYWWMHf1lwF4JBmTcUA=', 'Content-Length': '0'} 2024-02-01T10:22:16.070 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', 'ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002d969adfbec9d529-0065bb70d8-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:16.112 DEBUG:boto:path=/vclfcn-18/obj-test-zone2 2024-02-01T10:22:16.112 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2 2024-02-01T10:22:16.112 DEBUG:boto:path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:16.113 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:16.113 DEBUG:boto:Method: GET 2024-02-01T10:22:16.113 DEBUG:boto:Path: /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:16.113 DEBUG:boto:Data: 2024-02-01T10:22:16.113 DEBUG:boto:Headers: {} 2024-02-01T10:22:16.113 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:16.113 DEBUG:boto:Port: 8000 2024-02-01T10:22:16.113 DEBUG:boto:Params: {} 2024-02-01T10:22:16.114 DEBUG:boto:Token: None 2024-02-01T10:22:16.114 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:16 GMT /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:16.114 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:uaX8MAPCOYWWMHf1lwF4JBmTcUA= 2024-02-01T10:22:16.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:uaX8MAPCOYWWMHf1lwF4JBmTcUA=', 'Content-Length': '0'} 2024-02-01T10:22:16.117 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', 'ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000099251c5cc2403283-0065bb70d8-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:16.160 DEBUG:boto:path=/vclfcn-18/obj-test-zone2 2024-02-01T10:22:16.160 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2 2024-02-01T10:22:16.160 DEBUG:boto:path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:16.160 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:16.161 DEBUG:boto:Method: HEAD 2024-02-01T10:22:16.161 DEBUG:boto:Path: /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:16.161 DEBUG:boto:Data: 2024-02-01T10:22:16.161 DEBUG:boto:Headers: {} 2024-02-01T10:22:16.161 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:16.161 DEBUG:boto:Port: 8001 2024-02-01T10:22:16.161 DEBUG:boto:Params: {} 2024-02-01T10:22:16.161 DEBUG:boto:Token: None 2024-02-01T10:22:16.161 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:16 GMT /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:16.162 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lSAiA7ZzcAfGv2nzbkWnB5fcpnk= 2024-02-01T10:22:16.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lSAiA7ZzcAfGv2nzbkWnB5fcpnk=', 'Content-Length': '0'} 2024-02-01T10:22:16.164 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', 'ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d2dbed3a7dfaeb2f-0065bb70d8-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:16.165 DEBUG:boto:path=/vclfcn-18/obj-test-zone2 2024-02-01T10:22:16.165 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2 2024-02-01T10:22:16.165 DEBUG:boto:path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:16.165 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:16.165 DEBUG:boto:Method: HEAD 2024-02-01T10:22:16.165 DEBUG:boto:Path: /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:16.165 DEBUG:boto:Data: 2024-02-01T10:22:16.165 DEBUG:boto:Headers: {} 2024-02-01T10:22:16.165 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:16.166 DEBUG:boto:Port: 8000 2024-02-01T10:22:16.166 DEBUG:boto:Params: {} 2024-02-01T10:22:16.166 DEBUG:boto:Token: None 2024-02-01T10:22:16.166 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:16 GMT /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:16.166 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lSAiA7ZzcAfGv2nzbkWnB5fcpnk= 2024-02-01T10:22:16.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lSAiA7ZzcAfGv2nzbkWnB5fcpnk=', 'Content-Length': '0'} 2024-02-01T10:22:16.169 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', 'ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d68d4c3bd2de29e8-0065bb70d8-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:16.170 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-02-01T10:22:16.170 DEBUG:boto:path=/vclfcn-18/obj-test-zone2 2024-02-01T10:22:16.170 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2 2024-02-01T10:22:16.170 DEBUG:boto:path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:16.170 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:16.171 DEBUG:boto:Method: GET 2024-02-01T10:22:16.171 DEBUG:boto:Path: /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:16.171 DEBUG:boto:Data: 2024-02-01T10:22:16.171 DEBUG:boto:Headers: {} 2024-02-01T10:22:16.171 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:16.171 DEBUG:boto:Port: 8001 2024-02-01T10:22:16.171 DEBUG:boto:Params: {} 2024-02-01T10:22:16.171 DEBUG:boto:Token: None 2024-02-01T10:22:16.171 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:16 GMT /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:16.172 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:uaX8MAPCOYWWMHf1lwF4JBmTcUA= 2024-02-01T10:22:16.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:uaX8MAPCOYWWMHf1lwF4JBmTcUA=', 'Content-Length': '0'} 2024-02-01T10:22:16.175 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', 'ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000700ba001d63f01c0-0065bb70d8-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:16.216 DEBUG:boto:path=/vclfcn-18/obj-test-zone2 2024-02-01T10:22:16.216 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2 2024-02-01T10:22:16.216 DEBUG:boto:path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:16.216 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:16.217 DEBUG:boto:Method: GET 2024-02-01T10:22:16.217 DEBUG:boto:Path: /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:16.217 DEBUG:boto:Data: 2024-02-01T10:22:16.217 DEBUG:boto:Headers: {} 2024-02-01T10:22:16.217 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:16.217 DEBUG:boto:Port: 8000 2024-02-01T10:22:16.217 DEBUG:boto:Params: {} 2024-02-01T10:22:16.217 DEBUG:boto:Token: None 2024-02-01T10:22:16.217 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:16 GMT /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:16.218 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:uaX8MAPCOYWWMHf1lwF4JBmTcUA= 2024-02-01T10:22:16.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:uaX8MAPCOYWWMHf1lwF4JBmTcUA=', 'Content-Length': '0'} 2024-02-01T10:22:16.221 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', 'ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003bed10db2227f2b7-0065bb70d8-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:16.268 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:22:16.268 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:22:16.268 DEBUG:boto:path=/vclfcn-18/?prefix=obj-test-zone2 2024-02-01T10:22:16.269 DEBUG:boto:auth_path=/vclfcn-18/?prefix=obj-test-zone2 2024-02-01T10:22:16.269 DEBUG:boto:Method: GET 2024-02-01T10:22:16.269 DEBUG:boto:Path: /vclfcn-18/?prefix=obj-test-zone2 2024-02-01T10:22:16.269 DEBUG:boto:Data: 2024-02-01T10:22:16.269 DEBUG:boto:Headers: {} 2024-02-01T10:22:16.269 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:16.269 DEBUG:boto:Port: 8001 2024-02-01T10:22:16.269 DEBUG:boto:Params: {} 2024-02-01T10:22:16.269 DEBUG:boto:Token: None 2024-02-01T10:22:16.270 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:16 GMT /vclfcn-18/ 2024-02-01T10:22:16.270 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ysLB0yg/KO94C8n2rwJHreWKTGw= 2024-02-01T10:22:16.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ysLB0yg/KO94C8n2rwJHreWKTGw=', 'Content-Length': '0'} 2024-02-01T10:22:16.274 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000551aa24b3ac0ea37-0065bb70d8-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:16.274 DEBUG:boto:b'vclfcn-18obj-test-zone21000falseobj-test-zone22024-02-01T10:20:51.094Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:16.275 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:22:16.275 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:22:16.275 DEBUG:boto:path=/vclfcn-18/?prefix=obj-test-zone2 2024-02-01T10:22:16.275 DEBUG:boto:auth_path=/vclfcn-18/?prefix=obj-test-zone2 2024-02-01T10:22:16.275 DEBUG:boto:Method: GET 2024-02-01T10:22:16.275 DEBUG:boto:Path: /vclfcn-18/?prefix=obj-test-zone2 2024-02-01T10:22:16.275 DEBUG:boto:Data: 2024-02-01T10:22:16.276 DEBUG:boto:Headers: {} 2024-02-01T10:22:16.276 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:16.276 DEBUG:boto:Port: 8000 2024-02-01T10:22:16.276 DEBUG:boto:Params: {} 2024-02-01T10:22:16.276 DEBUG:boto:Token: None 2024-02-01T10:22:16.276 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:16 GMT /vclfcn-18/ 2024-02-01T10:22:16.276 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ysLB0yg/KO94C8n2rwJHreWKTGw= 2024-02-01T10:22:16.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ysLB0yg/KO94C8n2rwJHreWKTGw=', 'Content-Length': '0'} 2024-02-01T10:22:16.280 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b10fbe063ce03fd-0065bb70d8-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:16.280 DEBUG:boto:b'vclfcn-18obj-test-zone21000falseobj-test-zone22024-02-01T10:20:51.094Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:16.281 DEBUG:boto:path=/vclfcn-18/obj-test-zone2 2024-02-01T10:22:16.281 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2 2024-02-01T10:22:16.281 DEBUG:boto:Method: HEAD 2024-02-01T10:22:16.281 DEBUG:boto:Path: /vclfcn-18/obj-test-zone2 2024-02-01T10:22:16.281 DEBUG:boto:Data: 2024-02-01T10:22:16.281 DEBUG:boto:Headers: {} 2024-02-01T10:22:16.281 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:16.281 DEBUG:boto:Port: 8001 2024-02-01T10:22:16.281 DEBUG:boto:Params: {} 2024-02-01T10:22:16.282 DEBUG:boto:Token: None 2024-02-01T10:22:16.282 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:16 GMT /vclfcn-18/obj-test-zone2 2024-02-01T10:22:16.282 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0jzd3vK61aN0q2ny2noiXVvaazc= 2024-02-01T10:22:16.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0jzd3vK61aN0q2ny2noiXVvaazc=', 'Content-Length': '0'} 2024-02-01T10:22:16.285 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', 'ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e5d74e6d9d62720b-0065bb70d8-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:16.286 DEBUG:boto:path=/vclfcn-18/obj-test-zone2 2024-02-01T10:22:16.286 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2 2024-02-01T10:22:16.286 DEBUG:boto:Method: HEAD 2024-02-01T10:22:16.286 DEBUG:boto:Path: /vclfcn-18/obj-test-zone2 2024-02-01T10:22:16.286 DEBUG:boto:Data: 2024-02-01T10:22:16.286 DEBUG:boto:Headers: {} 2024-02-01T10:22:16.286 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:16.286 DEBUG:boto:Port: 8000 2024-02-01T10:22:16.286 DEBUG:boto:Params: {} 2024-02-01T10:22:16.286 DEBUG:boto:Token: None 2024-02-01T10:22:16.287 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:16 GMT /vclfcn-18/obj-test-zone2 2024-02-01T10:22:16.287 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0jzd3vK61aN0q2ny2noiXVvaazc= 2024-02-01T10:22:16.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0jzd3vK61aN0q2ny2noiXVvaazc=', 'Content-Length': '0'} 2024-02-01T10:22:16.290 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', 'ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ee856877b7091fd6-0065bb70d8-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:16.290 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-02-01T10:22:16.290 DEBUG:boto:path=/vclfcn-18/obj-test-zone2 2024-02-01T10:22:16.291 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2 2024-02-01T10:22:16.291 DEBUG:boto:path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:16.291 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:16.291 DEBUG:boto:Method: GET 2024-02-01T10:22:16.291 DEBUG:boto:Path: /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:16.291 DEBUG:boto:Data: 2024-02-01T10:22:16.291 DEBUG:boto:Headers: {} 2024-02-01T10:22:16.291 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:16.291 DEBUG:boto:Port: 8001 2024-02-01T10:22:16.291 DEBUG:boto:Params: {} 2024-02-01T10:22:16.291 DEBUG:boto:Token: None 2024-02-01T10:22:16.292 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:16 GMT /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:16.292 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:uaX8MAPCOYWWMHf1lwF4JBmTcUA= 2024-02-01T10:22:16.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:uaX8MAPCOYWWMHf1lwF4JBmTcUA=', 'Content-Length': '0'} 2024-02-01T10:22:16.294 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', 'ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002d1049e4dc863cb7-0065bb70d8-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:16.340 DEBUG:boto:path=/vclfcn-18/obj-test-zone2 2024-02-01T10:22:16.340 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2 2024-02-01T10:22:16.340 DEBUG:boto:path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:16.340 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:16.341 DEBUG:boto:Method: GET 2024-02-01T10:22:16.341 DEBUG:boto:Path: /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:16.341 DEBUG:boto:Data: 2024-02-01T10:22:16.341 DEBUG:boto:Headers: {} 2024-02-01T10:22:16.341 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:16.341 DEBUG:boto:Port: 8000 2024-02-01T10:22:16.341 DEBUG:boto:Params: {} 2024-02-01T10:22:16.341 DEBUG:boto:Token: None 2024-02-01T10:22:16.341 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:16 GMT /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:16.342 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:uaX8MAPCOYWWMHf1lwF4JBmTcUA= 2024-02-01T10:22:16.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:uaX8MAPCOYWWMHf1lwF4JBmTcUA=', 'Content-Length': '0'} 2024-02-01T10:22:16.345 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', 'ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000009d63d04f9277e2f-0065bb70d8-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:16.392 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:22:16.392 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:22:16.392 DEBUG:boto:path=/vclfcn-18/?prefix=obj-test-zone2 2024-02-01T10:22:16.393 DEBUG:boto:auth_path=/vclfcn-18/?prefix=obj-test-zone2 2024-02-01T10:22:16.393 DEBUG:boto:Method: GET 2024-02-01T10:22:16.393 DEBUG:boto:Path: /vclfcn-18/?prefix=obj-test-zone2 2024-02-01T10:22:16.393 DEBUG:boto:Data: 2024-02-01T10:22:16.393 DEBUG:boto:Headers: {} 2024-02-01T10:22:16.393 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:16.393 DEBUG:boto:Port: 8001 2024-02-01T10:22:16.393 DEBUG:boto:Params: {} 2024-02-01T10:22:16.393 DEBUG:boto:Token: None 2024-02-01T10:22:16.393 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:16 GMT /vclfcn-18/ 2024-02-01T10:22:16.394 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ysLB0yg/KO94C8n2rwJHreWKTGw= 2024-02-01T10:22:16.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ysLB0yg/KO94C8n2rwJHreWKTGw=', 'Content-Length': '0'} 2024-02-01T10:22:16.398 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004dc83348ee1e847b-0065bb70d8-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:16.440 DEBUG:boto:b'vclfcn-18obj-test-zone21000falseobj-test-zone22024-02-01T10:20:51.094Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:16.441 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:22:16.441 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:22:16.441 DEBUG:boto:path=/vclfcn-18/?prefix=obj-test-zone2 2024-02-01T10:22:16.441 DEBUG:boto:auth_path=/vclfcn-18/?prefix=obj-test-zone2 2024-02-01T10:22:16.441 DEBUG:boto:Method: GET 2024-02-01T10:22:16.441 DEBUG:boto:Path: /vclfcn-18/?prefix=obj-test-zone2 2024-02-01T10:22:16.441 DEBUG:boto:Data: 2024-02-01T10:22:16.441 DEBUG:boto:Headers: {} 2024-02-01T10:22:16.441 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:16.441 DEBUG:boto:Port: 8000 2024-02-01T10:22:16.441 DEBUG:boto:Params: {} 2024-02-01T10:22:16.442 DEBUG:boto:Token: None 2024-02-01T10:22:16.442 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:16 GMT /vclfcn-18/ 2024-02-01T10:22:16.442 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ysLB0yg/KO94C8n2rwJHreWKTGw= 2024-02-01T10:22:16.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ysLB0yg/KO94C8n2rwJHreWKTGw=', 'Content-Length': '0'} 2024-02-01T10:22:16.446 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc35b69f63fd36d9-0065bb70d8-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:16.446 DEBUG:boto:b'vclfcn-18obj-test-zone21000falseobj-test-zone22024-02-01T10:20:51.094Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:16.447 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-02-01T10:22:16.447 DEBUG:boto:path=/vclfcn-18/obj-test-zone3 2024-02-01T10:22:16.447 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3 2024-02-01T10:22:16.447 DEBUG:boto:path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:16.447 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:16.447 DEBUG:boto:Method: GET 2024-02-01T10:22:16.447 DEBUG:boto:Path: /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:16.447 DEBUG:boto:Data: 2024-02-01T10:22:16.447 DEBUG:boto:Headers: {} 2024-02-01T10:22:16.448 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:16.448 DEBUG:boto:Port: 8001 2024-02-01T10:22:16.448 DEBUG:boto:Params: {} 2024-02-01T10:22:16.448 DEBUG:boto:Token: None 2024-02-01T10:22:16.448 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:16 GMT /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:16.448 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:iwHVDbYda17Mt43SwJu/GcTft10= 2024-02-01T10:22:16.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iwHVDbYda17Mt43SwJu/GcTft10=', 'Content-Length': '0'} 2024-02-01T10:22:16.451 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', '6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003f9645ab4717aa98-0065bb70d8-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:16.451 DEBUG:boto:path=/vclfcn-18/obj-test-zone3 2024-02-01T10:22:16.452 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3 2024-02-01T10:22:16.452 DEBUG:boto:path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:16.452 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:16.452 DEBUG:boto:Method: GET 2024-02-01T10:22:16.452 DEBUG:boto:Path: /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:16.452 DEBUG:boto:Data: 2024-02-01T10:22:16.452 DEBUG:boto:Headers: {} 2024-02-01T10:22:16.452 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:16.452 DEBUG:boto:Port: 8000 2024-02-01T10:22:16.452 DEBUG:boto:Params: {} 2024-02-01T10:22:16.453 DEBUG:boto:Token: None 2024-02-01T10:22:16.453 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:16 GMT /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:16.453 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:iwHVDbYda17Mt43SwJu/GcTft10= 2024-02-01T10:22:16.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iwHVDbYda17Mt43SwJu/GcTft10=', 'Content-Length': '0'} 2024-02-01T10:22:16.456 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', '6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b1060898c22789ff-0065bb70d8-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:16.457 DEBUG:boto:path=/vclfcn-18/obj-test-zone3 2024-02-01T10:22:16.457 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3 2024-02-01T10:22:16.457 DEBUG:boto:path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:16.457 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:16.457 DEBUG:boto:Method: HEAD 2024-02-01T10:22:16.457 DEBUG:boto:Path: /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:16.457 DEBUG:boto:Data: 2024-02-01T10:22:16.458 DEBUG:boto:Headers: {} 2024-02-01T10:22:16.458 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:16.458 DEBUG:boto:Port: 8001 2024-02-01T10:22:16.458 DEBUG:boto:Params: {} 2024-02-01T10:22:16.458 DEBUG:boto:Token: None 2024-02-01T10:22:16.458 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:16 GMT /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:16.458 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:D4vV8aSIJdCcDICmWnEJlCx50cA= 2024-02-01T10:22:16.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:D4vV8aSIJdCcDICmWnEJlCx50cA=', 'Content-Length': '0'} 2024-02-01T10:22:16.461 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', '6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dfefbd2d691e6dd7-0065bb70d8-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:16.461 DEBUG:boto:path=/vclfcn-18/obj-test-zone3 2024-02-01T10:22:16.461 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3 2024-02-01T10:22:16.462 DEBUG:boto:path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:16.462 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:16.462 DEBUG:boto:Method: HEAD 2024-02-01T10:22:16.462 DEBUG:boto:Path: /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:16.462 DEBUG:boto:Data: 2024-02-01T10:22:16.462 DEBUG:boto:Headers: {} 2024-02-01T10:22:16.462 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:16.462 DEBUG:boto:Port: 8000 2024-02-01T10:22:16.462 DEBUG:boto:Params: {} 2024-02-01T10:22:16.462 DEBUG:boto:Token: None 2024-02-01T10:22:16.463 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:16 GMT /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:16.463 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:D4vV8aSIJdCcDICmWnEJlCx50cA= 2024-02-01T10:22:16.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:D4vV8aSIJdCcDICmWnEJlCx50cA=', 'Content-Length': '0'} 2024-02-01T10:22:16.465 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', '6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eeb8c4300dd10ce0-0065bb70d8-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:16.466 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-02-01T10:22:16.466 DEBUG:boto:path=/vclfcn-18/obj-test-zone3 2024-02-01T10:22:16.466 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3 2024-02-01T10:22:16.466 DEBUG:boto:path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:16.466 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:16.466 DEBUG:boto:Method: GET 2024-02-01T10:22:16.466 DEBUG:boto:Path: /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:16.466 DEBUG:boto:Data: 2024-02-01T10:22:16.466 DEBUG:boto:Headers: {} 2024-02-01T10:22:16.467 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:16.467 DEBUG:boto:Port: 8001 2024-02-01T10:22:16.467 DEBUG:boto:Params: {} 2024-02-01T10:22:16.467 DEBUG:boto:Token: None 2024-02-01T10:22:16.467 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:16 GMT /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:16.467 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:iwHVDbYda17Mt43SwJu/GcTft10= 2024-02-01T10:22:16.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iwHVDbYda17Mt43SwJu/GcTft10=', 'Content-Length': '0'} 2024-02-01T10:22:16.470 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', '6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d5a6505d5d367f2a-0065bb70d8-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:16.470 DEBUG:boto:path=/vclfcn-18/obj-test-zone3 2024-02-01T10:22:16.471 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3 2024-02-01T10:22:16.471 DEBUG:boto:path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:16.471 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:16.471 DEBUG:boto:Method: GET 2024-02-01T10:22:16.471 DEBUG:boto:Path: /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:16.471 DEBUG:boto:Data: 2024-02-01T10:22:16.471 DEBUG:boto:Headers: {} 2024-02-01T10:22:16.471 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:16.471 DEBUG:boto:Port: 8000 2024-02-01T10:22:16.471 DEBUG:boto:Params: {} 2024-02-01T10:22:16.472 DEBUG:boto:Token: None 2024-02-01T10:22:16.472 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:16 GMT /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:16.472 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:iwHVDbYda17Mt43SwJu/GcTft10= 2024-02-01T10:22:16.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iwHVDbYda17Mt43SwJu/GcTft10=', 'Content-Length': '0'} 2024-02-01T10:22:16.475 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', '6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000003c3d4274c2b8dc8-0065bb70d8-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:16.516 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:22:16.516 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:22:16.516 DEBUG:boto:path=/vclfcn-18/?prefix=obj-test-zone3 2024-02-01T10:22:16.517 DEBUG:boto:auth_path=/vclfcn-18/?prefix=obj-test-zone3 2024-02-01T10:22:16.517 DEBUG:boto:Method: GET 2024-02-01T10:22:16.517 DEBUG:boto:Path: /vclfcn-18/?prefix=obj-test-zone3 2024-02-01T10:22:16.517 DEBUG:boto:Data: 2024-02-01T10:22:16.517 DEBUG:boto:Headers: {} 2024-02-01T10:22:16.517 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:16.517 DEBUG:boto:Port: 8001 2024-02-01T10:22:16.517 DEBUG:boto:Params: {} 2024-02-01T10:22:16.517 DEBUG:boto:Token: None 2024-02-01T10:22:16.518 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:16 GMT /vclfcn-18/ 2024-02-01T10:22:16.518 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ysLB0yg/KO94C8n2rwJHreWKTGw= 2024-02-01T10:22:16.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ysLB0yg/KO94C8n2rwJHreWKTGw=', 'Content-Length': '0'} 2024-02-01T10:22:16.522 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043049d57bbda8d21-0065bb70d8-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:16.523 DEBUG:boto:b'vclfcn-18obj-test-zone31000falseobj-test-zone32024-02-01T10:20:51.248Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:16.523 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:22:16.523 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:22:16.523 DEBUG:boto:path=/vclfcn-18/?prefix=obj-test-zone3 2024-02-01T10:22:16.523 DEBUG:boto:auth_path=/vclfcn-18/?prefix=obj-test-zone3 2024-02-01T10:22:16.523 DEBUG:boto:Method: GET 2024-02-01T10:22:16.524 DEBUG:boto:Path: /vclfcn-18/?prefix=obj-test-zone3 2024-02-01T10:22:16.524 DEBUG:boto:Data: 2024-02-01T10:22:16.524 DEBUG:boto:Headers: {} 2024-02-01T10:22:16.524 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:16.524 DEBUG:boto:Port: 8000 2024-02-01T10:22:16.524 DEBUG:boto:Params: {} 2024-02-01T10:22:16.524 DEBUG:boto:Token: None 2024-02-01T10:22:16.524 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:16 GMT /vclfcn-18/ 2024-02-01T10:22:16.524 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ysLB0yg/KO94C8n2rwJHreWKTGw= 2024-02-01T10:22:16.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ysLB0yg/KO94C8n2rwJHreWKTGw=', 'Content-Length': '0'} 2024-02-01T10:22:16.529 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005632c172e3ab2341-0065bb70d8-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:16.529 DEBUG:boto:b'vclfcn-18obj-test-zone31000falseobj-test-zone32024-02-01T10:20:51.248Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:16.529 DEBUG:boto:path=/vclfcn-18/obj-test-zone3 2024-02-01T10:22:16.529 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3 2024-02-01T10:22:16.530 DEBUG:boto:Method: HEAD 2024-02-01T10:22:16.530 DEBUG:boto:Path: /vclfcn-18/obj-test-zone3 2024-02-01T10:22:16.530 DEBUG:boto:Data: 2024-02-01T10:22:16.530 DEBUG:boto:Headers: {} 2024-02-01T10:22:16.530 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:16.530 DEBUG:boto:Port: 8001 2024-02-01T10:22:16.530 DEBUG:boto:Params: {} 2024-02-01T10:22:16.530 DEBUG:boto:Token: None 2024-02-01T10:22:16.530 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:16 GMT /vclfcn-18/obj-test-zone3 2024-02-01T10:22:16.530 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zhYU744zTqn7JYojBWj1vO1fZPA= 2024-02-01T10:22:16.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zhYU744zTqn7JYojBWj1vO1fZPA=', 'Content-Length': '0'} 2024-02-01T10:22:16.536 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', '6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cb4f6615e6342564-0065bb70d8-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:16.536 DEBUG:boto:path=/vclfcn-18/obj-test-zone3 2024-02-01T10:22:16.536 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3 2024-02-01T10:22:16.536 DEBUG:boto:Method: HEAD 2024-02-01T10:22:16.536 DEBUG:boto:Path: /vclfcn-18/obj-test-zone3 2024-02-01T10:22:16.536 DEBUG:boto:Data: 2024-02-01T10:22:16.537 DEBUG:boto:Headers: {} 2024-02-01T10:22:16.537 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:16.537 DEBUG:boto:Port: 8000 2024-02-01T10:22:16.537 DEBUG:boto:Params: {} 2024-02-01T10:22:16.537 DEBUG:boto:Token: None 2024-02-01T10:22:16.537 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:16 GMT /vclfcn-18/obj-test-zone3 2024-02-01T10:22:16.537 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zhYU744zTqn7JYojBWj1vO1fZPA= 2024-02-01T10:22:16.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zhYU744zTqn7JYojBWj1vO1fZPA=', 'Content-Length': '0'} 2024-02-01T10:22:16.541 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', '6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d3f8f4182ae240da-0065bb70d8-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:16.541 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-02-01T10:22:16.541 DEBUG:boto:path=/vclfcn-18/obj-test-zone3 2024-02-01T10:22:16.542 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3 2024-02-01T10:22:16.542 DEBUG:boto:path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:16.542 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:16.542 DEBUG:boto:Method: GET 2024-02-01T10:22:16.542 DEBUG:boto:Path: /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:16.542 DEBUG:boto:Data: 2024-02-01T10:22:16.542 DEBUG:boto:Headers: {} 2024-02-01T10:22:16.542 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:16.542 DEBUG:boto:Port: 8001 2024-02-01T10:22:16.542 DEBUG:boto:Params: {} 2024-02-01T10:22:16.543 DEBUG:boto:Token: None 2024-02-01T10:22:16.543 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:16 GMT /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:16.543 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:iwHVDbYda17Mt43SwJu/GcTft10= 2024-02-01T10:22:16.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iwHVDbYda17Mt43SwJu/GcTft10=', 'Content-Length': '0'} 2024-02-01T10:22:16.546 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', '6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f4c74dfcbe88cf59-0065bb70d8-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:16.588 DEBUG:boto:path=/vclfcn-18/obj-test-zone3 2024-02-01T10:22:16.588 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3 2024-02-01T10:22:16.589 DEBUG:boto:path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:16.589 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:16.589 DEBUG:boto:Method: GET 2024-02-01T10:22:16.589 DEBUG:boto:Path: /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:16.589 DEBUG:boto:Data: 2024-02-01T10:22:16.589 DEBUG:boto:Headers: {} 2024-02-01T10:22:16.589 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:16.589 DEBUG:boto:Port: 8000 2024-02-01T10:22:16.589 DEBUG:boto:Params: {} 2024-02-01T10:22:16.589 DEBUG:boto:Token: None 2024-02-01T10:22:16.590 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:16 GMT /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:16.590 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:iwHVDbYda17Mt43SwJu/GcTft10= 2024-02-01T10:22:16.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iwHVDbYda17Mt43SwJu/GcTft10=', 'Content-Length': '0'} 2024-02-01T10:22:16.593 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', '6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a55453bb713bdd79-0065bb70d8-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:16.636 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:22:16.636 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:22:16.636 DEBUG:boto:path=/vclfcn-18/?prefix=obj-test-zone3 2024-02-01T10:22:16.637 DEBUG:boto:auth_path=/vclfcn-18/?prefix=obj-test-zone3 2024-02-01T10:22:16.637 DEBUG:boto:Method: GET 2024-02-01T10:22:16.637 DEBUG:boto:Path: /vclfcn-18/?prefix=obj-test-zone3 2024-02-01T10:22:16.637 DEBUG:boto:Data: 2024-02-01T10:22:16.637 DEBUG:boto:Headers: {} 2024-02-01T10:22:16.637 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:16.637 DEBUG:boto:Port: 8001 2024-02-01T10:22:16.637 DEBUG:boto:Params: {} 2024-02-01T10:22:16.637 DEBUG:boto:Token: None 2024-02-01T10:22:16.637 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:16 GMT /vclfcn-18/ 2024-02-01T10:22:16.638 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ysLB0yg/KO94C8n2rwJHreWKTGw= 2024-02-01T10:22:16.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ysLB0yg/KO94C8n2rwJHreWKTGw=', 'Content-Length': '0'} 2024-02-01T10:22:16.642 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000bd8ba7bd413a9a5-0065bb70d8-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:16.642 DEBUG:boto:b'vclfcn-18obj-test-zone31000falseobj-test-zone32024-02-01T10:20:51.248Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:16.642 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:22:16.642 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:22:16.642 DEBUG:boto:path=/vclfcn-18/?prefix=obj-test-zone3 2024-02-01T10:22:16.643 DEBUG:boto:auth_path=/vclfcn-18/?prefix=obj-test-zone3 2024-02-01T10:22:16.643 DEBUG:boto:Method: GET 2024-02-01T10:22:16.643 DEBUG:boto:Path: /vclfcn-18/?prefix=obj-test-zone3 2024-02-01T10:22:16.643 DEBUG:boto:Data: 2024-02-01T10:22:16.643 DEBUG:boto:Headers: {} 2024-02-01T10:22:16.643 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:16.643 DEBUG:boto:Port: 8000 2024-02-01T10:22:16.643 DEBUG:boto:Params: {} 2024-02-01T10:22:16.643 DEBUG:boto:Token: None 2024-02-01T10:22:16.643 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:16 GMT /vclfcn-18/ 2024-02-01T10:22:16.644 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ysLB0yg/KO94C8n2rwJHreWKTGw= 2024-02-01T10:22:16.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ysLB0yg/KO94C8n2rwJHreWKTGw=', 'Content-Length': '0'} 2024-02-01T10:22:16.647 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd8ec0a1d7da6b0c-0065bb70d8-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:16.648 DEBUG:boto:b'vclfcn-18obj-test-zone31000falseobj-test-zone32024-02-01T10:20:51.248Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:16.648 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-18 zones={test-zone3, test-zone1} 2024-02-01T10:22:16.648 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-18 zones={test-zone3, test-zone2} 2024-02-01T10:22:16.648 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:22:16.648 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:22:16.649 DEBUG:boto:Method: HEAD 2024-02-01T10:22:16.649 DEBUG:boto:Path: /vclfcn-18/ 2024-02-01T10:22:16.649 DEBUG:boto:Data: 2024-02-01T10:22:16.649 DEBUG:boto:Headers: {} 2024-02-01T10:22:16.649 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:16.649 DEBUG:boto:Port: 8001 2024-02-01T10:22:16.649 DEBUG:boto:Params: {} 2024-02-01T10:22:16.649 DEBUG:boto:Token: None 2024-02-01T10:22:16.649 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:16 GMT /vclfcn-18/ 2024-02-01T10:22:16.649 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BgQ3S4ZhfCXe0ixiFE1pN4E55Q4= 2024-02-01T10:22:16.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BgQ3S4ZhfCXe0ixiFE1pN4E55Q4=', 'Content-Length': '0'} 2024-02-01T10:22:16.653 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bd23b2e5f0e037ae-0065bb70d8-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:22:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:16.653 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:22:16.654 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:22:16.654 DEBUG:boto:Method: HEAD 2024-02-01T10:22:16.654 DEBUG:boto:Path: /vclfcn-18/ 2024-02-01T10:22:16.654 DEBUG:boto:Data: 2024-02-01T10:22:16.654 DEBUG:boto:Headers: {} 2024-02-01T10:22:16.654 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:16.654 DEBUG:boto:Port: 8000 2024-02-01T10:22:16.654 DEBUG:boto:Params: {} 2024-02-01T10:22:16.654 DEBUG:boto:Token: None 2024-02-01T10:22:16.654 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:16 GMT /vclfcn-18/ 2024-02-01T10:22:16.655 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BgQ3S4ZhfCXe0ixiFE1pN4E55Q4= 2024-02-01T10:22:16.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BgQ3S4ZhfCXe0ixiFE1pN4E55Q4=', 'Content-Length': '0'} 2024-02-01T10:22:16.659 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000046fc58942646a630-0065bb70d8-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:22:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:16.659 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:22:16.659 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:22:16.659 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:22:16.659 DEBUG:boto:path=/vclfcn-18/?versions&max-keys=999 2024-02-01T10:22:16.659 DEBUG:boto:auth_path=/vclfcn-18/?versions&max-keys=999 2024-02-01T10:22:16.659 DEBUG:boto:Method: GET 2024-02-01T10:22:16.659 DEBUG:boto:Path: /vclfcn-18/?versions&max-keys=999 2024-02-01T10:22:16.660 DEBUG:boto:Data: 2024-02-01T10:22:16.660 DEBUG:boto:Headers: {} 2024-02-01T10:22:16.660 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:16.660 DEBUG:boto:Port: 8001 2024-02-01T10:22:16.660 DEBUG:boto:Params: {} 2024-02-01T10:22:16.660 DEBUG:boto:Token: None 2024-02-01T10:22:16.660 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:16 GMT /vclfcn-18/?versions 2024-02-01T10:22:16.660 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:p2D2Te/5/UzwFEfPOCsW9+F8MBw= 2024-02-01T10:22:16.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:p2D2Te/5/UzwFEfPOCsW9+F8MBw=', 'Content-Length': '0'} 2024-02-01T10:22:16.665 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a869dd24dac653fc-0065bb70d8-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:16.666 DEBUG:boto:b'vclfcn-18999falsedummytxgLlJWZVbWbjqLWqkOtt87dUoD4zuntrue2024-02-01T10:20:34.645Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcKtrue2024-02-01T10:20:50.942Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutDtrue2024-02-01T10:20:51.094Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone36pAVLbw3HAwE0jxxbuxTIxwcA2p4q8Ntrue2024-02-01T10:20:51.248Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:16.666 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-02-01T10:22:16.666 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-02-01T10:22:16.666 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-02-01T10:22:16.666 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-02-01T10:22:16.667 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:22:16.667 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:22:16.667 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:22:16.667 DEBUG:boto:path=/vclfcn-18/?versions&max-keys=999 2024-02-01T10:22:16.667 DEBUG:boto:auth_path=/vclfcn-18/?versions&max-keys=999 2024-02-01T10:22:16.667 DEBUG:boto:Method: GET 2024-02-01T10:22:16.667 DEBUG:boto:Path: /vclfcn-18/?versions&max-keys=999 2024-02-01T10:22:16.667 DEBUG:boto:Data: 2024-02-01T10:22:16.667 DEBUG:boto:Headers: {} 2024-02-01T10:22:16.667 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:16.668 DEBUG:boto:Port: 8000 2024-02-01T10:22:16.668 DEBUG:boto:Params: {} 2024-02-01T10:22:16.668 DEBUG:boto:Token: None 2024-02-01T10:22:16.668 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:16 GMT /vclfcn-18/?versions 2024-02-01T10:22:16.668 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:p2D2Te/5/UzwFEfPOCsW9+F8MBw= 2024-02-01T10:22:16.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:p2D2Te/5/UzwFEfPOCsW9+F8MBw=', 'Content-Length': '0'} 2024-02-01T10:22:16.673 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000706ca5ef1852ffa5-0065bb70d8-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:16.673 DEBUG:boto:b'vclfcn-18999falsedummytxgLlJWZVbWbjqLWqkOtt87dUoD4zuntrue2024-02-01T10:20:34.645Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcKtrue2024-02-01T10:20:50.942Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutDtrue2024-02-01T10:20:51.094Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone36pAVLbw3HAwE0jxxbuxTIxwcA2p4q8Ntrue2024-02-01T10:20:51.248Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:16.674 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-02-01T10:22:16.674 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-02-01T10:22:16.674 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-02-01T10:22:16.674 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-02-01T10:22:16.674 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:22:16.674 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:22:16.674 DEBUG:boto:path=/vclfcn-18/?versions&max-keys=999 2024-02-01T10:22:16.674 DEBUG:boto:auth_path=/vclfcn-18/?versions&max-keys=999 2024-02-01T10:22:16.674 DEBUG:boto:Method: GET 2024-02-01T10:22:16.675 DEBUG:boto:Path: /vclfcn-18/?versions&max-keys=999 2024-02-01T10:22:16.675 DEBUG:boto:Data: 2024-02-01T10:22:16.675 DEBUG:boto:Headers: {} 2024-02-01T10:22:16.675 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:16.675 DEBUG:boto:Port: 8001 2024-02-01T10:22:16.675 DEBUG:boto:Params: {} 2024-02-01T10:22:16.675 DEBUG:boto:Token: None 2024-02-01T10:22:16.675 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:16 GMT /vclfcn-18/?versions 2024-02-01T10:22:16.675 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:p2D2Te/5/UzwFEfPOCsW9+F8MBw= 2024-02-01T10:22:16.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:p2D2Te/5/UzwFEfPOCsW9+F8MBw=', 'Content-Length': '0'} 2024-02-01T10:22:16.680 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083e035d95a5fee89-0065bb70d8-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:16.680 DEBUG:boto:b'vclfcn-18999falsedummytxgLlJWZVbWbjqLWqkOtt87dUoD4zuntrue2024-02-01T10:20:34.645Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcKtrue2024-02-01T10:20:50.942Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutDtrue2024-02-01T10:20:51.094Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone36pAVLbw3HAwE0jxxbuxTIxwcA2p4q8Ntrue2024-02-01T10:20:51.248Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:16.681 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:22:16.681 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:22:16.681 DEBUG:boto:path=/vclfcn-18/?versions&max-keys=999 2024-02-01T10:22:16.681 DEBUG:boto:auth_path=/vclfcn-18/?versions&max-keys=999 2024-02-01T10:22:16.681 DEBUG:boto:Method: GET 2024-02-01T10:22:16.681 DEBUG:boto:Path: /vclfcn-18/?versions&max-keys=999 2024-02-01T10:22:16.681 DEBUG:boto:Data: 2024-02-01T10:22:16.681 DEBUG:boto:Headers: {} 2024-02-01T10:22:16.681 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:16.681 DEBUG:boto:Port: 8000 2024-02-01T10:22:16.681 DEBUG:boto:Params: {} 2024-02-01T10:22:16.682 DEBUG:boto:Token: None 2024-02-01T10:22:16.682 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:16 GMT /vclfcn-18/?versions 2024-02-01T10:22:16.682 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:p2D2Te/5/UzwFEfPOCsW9+F8MBw= 2024-02-01T10:22:16.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:p2D2Te/5/UzwFEfPOCsW9+F8MBw=', 'Content-Length': '0'} 2024-02-01T10:22:16.686 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000053fd4f7866e4ff14-0065bb70d8-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:16.728 DEBUG:boto:b'vclfcn-18999falsedummytxgLlJWZVbWbjqLWqkOtt87dUoD4zuntrue2024-02-01T10:20:34.645Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcKtrue2024-02-01T10:20:50.942Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutDtrue2024-02-01T10:20:51.094Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone36pAVLbw3HAwE0jxxbuxTIxwcA2p4q8Ntrue2024-02-01T10:20:51.248Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:16.729 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-02-01T10:22:16.729 DEBUG:boto:path=/vclfcn-18/dummy 2024-02-01T10:22:16.729 DEBUG:boto:auth_path=/vclfcn-18/dummy 2024-02-01T10:22:16.729 DEBUG:boto:path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:16.729 DEBUG:boto:auth_path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:16.729 DEBUG:boto:Method: GET 2024-02-01T10:22:16.729 DEBUG:boto:Path: /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:16.729 DEBUG:boto:Data: 2024-02-01T10:22:16.729 DEBUG:boto:Headers: {} 2024-02-01T10:22:16.730 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:16.730 DEBUG:boto:Port: 8001 2024-02-01T10:22:16.730 DEBUG:boto:Params: {} 2024-02-01T10:22:16.730 DEBUG:boto:Token: None 2024-02-01T10:22:16.730 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:16 GMT /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:16.730 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:dEpxE5ZVUeLSFqEbP2AfnijoyWo= 2024-02-01T10:22:16.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dEpxE5ZVUeLSFqEbP2AfnijoyWo=', 'Content-Length': '0'} 2024-02-01T10:22:16.733 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:34 GMT'), ('x-amz-version-id', 'txgLlJWZVbWbjqLWqkOtt87dUoD4zun'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008e0ca84789c7986a-0065bb70d8-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:16.733 DEBUG:boto:path=/vclfcn-18/dummy 2024-02-01T10:22:16.734 DEBUG:boto:auth_path=/vclfcn-18/dummy 2024-02-01T10:22:16.734 DEBUG:boto:path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:16.734 DEBUG:boto:auth_path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:16.734 DEBUG:boto:Method: GET 2024-02-01T10:22:16.734 DEBUG:boto:Path: /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:16.734 DEBUG:boto:Data: 2024-02-01T10:22:16.734 DEBUG:boto:Headers: {} 2024-02-01T10:22:16.734 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:16.734 DEBUG:boto:Port: 8000 2024-02-01T10:22:16.734 DEBUG:boto:Params: {} 2024-02-01T10:22:16.734 DEBUG:boto:Token: None 2024-02-01T10:22:16.735 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:16 GMT /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:16.735 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:dEpxE5ZVUeLSFqEbP2AfnijoyWo= 2024-02-01T10:22:16.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dEpxE5ZVUeLSFqEbP2AfnijoyWo=', 'Content-Length': '0'} 2024-02-01T10:22:16.737 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:34 GMT'), ('x-amz-version-id', 'txgLlJWZVbWbjqLWqkOtt87dUoD4zun'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009bfb948bd3bf0002-0065bb70d8-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:16.738 DEBUG:boto:path=/vclfcn-18/dummy 2024-02-01T10:22:16.738 DEBUG:boto:auth_path=/vclfcn-18/dummy 2024-02-01T10:22:16.738 DEBUG:boto:path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:16.738 DEBUG:boto:auth_path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:16.738 DEBUG:boto:Method: HEAD 2024-02-01T10:22:16.738 DEBUG:boto:Path: /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:16.738 DEBUG:boto:Data: 2024-02-01T10:22:16.738 DEBUG:boto:Headers: {} 2024-02-01T10:22:16.739 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:16.739 DEBUG:boto:Port: 8001 2024-02-01T10:22:16.739 DEBUG:boto:Params: {} 2024-02-01T10:22:16.739 DEBUG:boto:Token: None 2024-02-01T10:22:16.739 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:16 GMT /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:16.739 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7c8O3fOOA570xRdcjwIgM2u3zps= 2024-02-01T10:22:16.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7c8O3fOOA570xRdcjwIgM2u3zps=', 'Content-Length': '0'} 2024-02-01T10:22:16.742 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:34 GMT'), ('x-amz-version-id', 'txgLlJWZVbWbjqLWqkOtt87dUoD4zun'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ebe5fe3261931634-0065bb70d8-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:16.742 DEBUG:boto:path=/vclfcn-18/dummy 2024-02-01T10:22:16.742 DEBUG:boto:auth_path=/vclfcn-18/dummy 2024-02-01T10:22:16.742 DEBUG:boto:path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:16.742 DEBUG:boto:auth_path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:16.742 DEBUG:boto:Method: HEAD 2024-02-01T10:22:16.742 DEBUG:boto:Path: /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:16.743 DEBUG:boto:Data: 2024-02-01T10:22:16.743 DEBUG:boto:Headers: {} 2024-02-01T10:22:16.743 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:16.743 DEBUG:boto:Port: 8000 2024-02-01T10:22:16.743 DEBUG:boto:Params: {} 2024-02-01T10:22:16.743 DEBUG:boto:Token: None 2024-02-01T10:22:16.743 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:16 GMT /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:16.743 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7c8O3fOOA570xRdcjwIgM2u3zps= 2024-02-01T10:22:16.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7c8O3fOOA570xRdcjwIgM2u3zps=', 'Content-Length': '0'} 2024-02-01T10:22:16.746 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:34 GMT'), ('x-amz-version-id', 'txgLlJWZVbWbjqLWqkOtt87dUoD4zun'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000010fa13e4e0798a8-0065bb70d8-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:16.747 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-02-01T10:22:16.747 DEBUG:boto:path=/vclfcn-18/dummy 2024-02-01T10:22:16.747 DEBUG:boto:auth_path=/vclfcn-18/dummy 2024-02-01T10:22:16.747 DEBUG:boto:path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:16.747 DEBUG:boto:auth_path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:16.747 DEBUG:boto:Method: GET 2024-02-01T10:22:16.747 DEBUG:boto:Path: /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:16.747 DEBUG:boto:Data: 2024-02-01T10:22:16.747 DEBUG:boto:Headers: {} 2024-02-01T10:22:16.747 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:16.747 DEBUG:boto:Port: 8001 2024-02-01T10:22:16.748 DEBUG:boto:Params: {} 2024-02-01T10:22:16.748 DEBUG:boto:Token: None 2024-02-01T10:22:16.748 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:16 GMT /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:16.748 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:dEpxE5ZVUeLSFqEbP2AfnijoyWo= 2024-02-01T10:22:16.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dEpxE5ZVUeLSFqEbP2AfnijoyWo=', 'Content-Length': '0'} 2024-02-01T10:22:16.751 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:34 GMT'), ('x-amz-version-id', 'txgLlJWZVbWbjqLWqkOtt87dUoD4zun'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a4ffc848156f733f-0065bb70d8-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:16.751 DEBUG:boto:path=/vclfcn-18/dummy 2024-02-01T10:22:16.751 DEBUG:boto:auth_path=/vclfcn-18/dummy 2024-02-01T10:22:16.751 DEBUG:boto:path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:16.751 DEBUG:boto:auth_path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:16.751 DEBUG:boto:Method: GET 2024-02-01T10:22:16.751 DEBUG:boto:Path: /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:16.751 DEBUG:boto:Data: 2024-02-01T10:22:16.751 DEBUG:boto:Headers: {} 2024-02-01T10:22:16.752 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:16.752 DEBUG:boto:Port: 8000 2024-02-01T10:22:16.752 DEBUG:boto:Params: {} 2024-02-01T10:22:16.752 DEBUG:boto:Token: None 2024-02-01T10:22:16.752 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:16 GMT /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:16.752 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:dEpxE5ZVUeLSFqEbP2AfnijoyWo= 2024-02-01T10:22:16.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dEpxE5ZVUeLSFqEbP2AfnijoyWo=', 'Content-Length': '0'} 2024-02-01T10:22:16.755 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:34 GMT'), ('x-amz-version-id', 'txgLlJWZVbWbjqLWqkOtt87dUoD4zun'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000086867ef435d9e40f-0065bb70d8-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:16.755 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:22:16.756 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:22:16.756 DEBUG:boto:path=/vclfcn-18/?prefix=dummy 2024-02-01T10:22:16.756 DEBUG:boto:auth_path=/vclfcn-18/?prefix=dummy 2024-02-01T10:22:16.756 DEBUG:boto:Method: GET 2024-02-01T10:22:16.756 DEBUG:boto:Path: /vclfcn-18/?prefix=dummy 2024-02-01T10:22:16.756 DEBUG:boto:Data: 2024-02-01T10:22:16.756 DEBUG:boto:Headers: {} 2024-02-01T10:22:16.756 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:16.756 DEBUG:boto:Port: 8001 2024-02-01T10:22:16.756 DEBUG:boto:Params: {} 2024-02-01T10:22:16.756 DEBUG:boto:Token: None 2024-02-01T10:22:16.757 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:16 GMT /vclfcn-18/ 2024-02-01T10:22:16.757 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ysLB0yg/KO94C8n2rwJHreWKTGw= 2024-02-01T10:22:16.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ysLB0yg/KO94C8n2rwJHreWKTGw=', 'Content-Length': '0'} 2024-02-01T10:22:16.761 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064d4ed619b07325a-0065bb70d8-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:16.761 DEBUG:boto:b'vclfcn-18dummy1000falsedummy2024-02-01T10:20:34.645Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:16.762 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:22:16.762 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:22:16.762 DEBUG:boto:path=/vclfcn-18/?prefix=dummy 2024-02-01T10:22:16.762 DEBUG:boto:auth_path=/vclfcn-18/?prefix=dummy 2024-02-01T10:22:16.762 DEBUG:boto:Method: GET 2024-02-01T10:22:16.762 DEBUG:boto:Path: /vclfcn-18/?prefix=dummy 2024-02-01T10:22:16.762 DEBUG:boto:Data: 2024-02-01T10:22:16.762 DEBUG:boto:Headers: {} 2024-02-01T10:22:16.762 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:16.763 DEBUG:boto:Port: 8000 2024-02-01T10:22:16.763 DEBUG:boto:Params: {} 2024-02-01T10:22:16.763 DEBUG:boto:Token: None 2024-02-01T10:22:16.763 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:16 GMT /vclfcn-18/ 2024-02-01T10:22:16.763 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ysLB0yg/KO94C8n2rwJHreWKTGw= 2024-02-01T10:22:16.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ysLB0yg/KO94C8n2rwJHreWKTGw=', 'Content-Length': '0'} 2024-02-01T10:22:16.768 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007fb9fa8016f3f5e7-0065bb70d8-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:16.768 DEBUG:boto:b'vclfcn-18dummy1000falsedummy2024-02-01T10:20:34.645Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:16.768 DEBUG:boto:path=/vclfcn-18/dummy 2024-02-01T10:22:16.768 DEBUG:boto:auth_path=/vclfcn-18/dummy 2024-02-01T10:22:16.769 DEBUG:boto:Method: HEAD 2024-02-01T10:22:16.769 DEBUG:boto:Path: /vclfcn-18/dummy 2024-02-01T10:22:16.769 DEBUG:boto:Data: 2024-02-01T10:22:16.769 DEBUG:boto:Headers: {} 2024-02-01T10:22:16.769 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:16.769 DEBUG:boto:Port: 8001 2024-02-01T10:22:16.769 DEBUG:boto:Params: {} 2024-02-01T10:22:16.770 DEBUG:boto:Token: None 2024-02-01T10:22:16.770 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:16 GMT /vclfcn-18/dummy 2024-02-01T10:22:16.770 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9ozvLr9ngry68gwD/NO6YvuTVe8= 2024-02-01T10:22:16.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9ozvLr9ngry68gwD/NO6YvuTVe8=', 'Content-Length': '0'} 2024-02-01T10:22:16.774 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:34 GMT'), ('x-amz-version-id', 'txgLlJWZVbWbjqLWqkOtt87dUoD4zun'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f1dda48c55494057-0065bb70d8-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:16.774 DEBUG:boto:path=/vclfcn-18/dummy 2024-02-01T10:22:16.774 DEBUG:boto:auth_path=/vclfcn-18/dummy 2024-02-01T10:22:16.774 DEBUG:boto:Method: HEAD 2024-02-01T10:22:16.774 DEBUG:boto:Path: /vclfcn-18/dummy 2024-02-01T10:22:16.774 DEBUG:boto:Data: 2024-02-01T10:22:16.775 DEBUG:boto:Headers: {} 2024-02-01T10:22:16.775 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:16.775 DEBUG:boto:Port: 8000 2024-02-01T10:22:16.775 DEBUG:boto:Params: {} 2024-02-01T10:22:16.775 DEBUG:boto:Token: None 2024-02-01T10:22:16.775 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:16 GMT /vclfcn-18/dummy 2024-02-01T10:22:16.776 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9ozvLr9ngry68gwD/NO6YvuTVe8= 2024-02-01T10:22:16.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9ozvLr9ngry68gwD/NO6YvuTVe8=', 'Content-Length': '0'} 2024-02-01T10:22:16.780 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:34 GMT'), ('x-amz-version-id', 'txgLlJWZVbWbjqLWqkOtt87dUoD4zun'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006cc8bf03abc7b956-0065bb70d8-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:16.780 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-02-01T10:22:16.780 DEBUG:boto:path=/vclfcn-18/dummy 2024-02-01T10:22:16.780 DEBUG:boto:auth_path=/vclfcn-18/dummy 2024-02-01T10:22:16.780 DEBUG:boto:path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:16.781 DEBUG:boto:auth_path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:16.781 DEBUG:boto:Method: GET 2024-02-01T10:22:16.781 DEBUG:boto:Path: /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:16.781 DEBUG:boto:Data: 2024-02-01T10:22:16.781 DEBUG:boto:Headers: {} 2024-02-01T10:22:16.781 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:16.781 DEBUG:boto:Port: 8001 2024-02-01T10:22:16.781 DEBUG:boto:Params: {} 2024-02-01T10:22:16.781 DEBUG:boto:Token: None 2024-02-01T10:22:16.781 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:16 GMT /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:16.782 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:dEpxE5ZVUeLSFqEbP2AfnijoyWo= 2024-02-01T10:22:16.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dEpxE5ZVUeLSFqEbP2AfnijoyWo=', 'Content-Length': '0'} 2024-02-01T10:22:16.784 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:34 GMT'), ('x-amz-version-id', 'txgLlJWZVbWbjqLWqkOtt87dUoD4zun'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007d8d48cf171b0382-0065bb70d8-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:16.784 DEBUG:boto:path=/vclfcn-18/dummy 2024-02-01T10:22:16.784 DEBUG:boto:auth_path=/vclfcn-18/dummy 2024-02-01T10:22:16.784 DEBUG:boto:path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:16.785 DEBUG:boto:auth_path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:16.785 DEBUG:boto:Method: GET 2024-02-01T10:22:16.785 DEBUG:boto:Path: /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:16.785 DEBUG:boto:Data: 2024-02-01T10:22:16.785 DEBUG:boto:Headers: {} 2024-02-01T10:22:16.785 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:16.785 DEBUG:boto:Port: 8000 2024-02-01T10:22:16.785 DEBUG:boto:Params: {} 2024-02-01T10:22:16.785 DEBUG:boto:Token: None 2024-02-01T10:22:16.785 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:16 GMT /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:22:16.786 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:dEpxE5ZVUeLSFqEbP2AfnijoyWo= 2024-02-01T10:22:16.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dEpxE5ZVUeLSFqEbP2AfnijoyWo=', 'Content-Length': '0'} 2024-02-01T10:22:16.788 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:34 GMT'), ('x-amz-version-id', 'txgLlJWZVbWbjqLWqkOtt87dUoD4zun'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000020396913315dbcd7-0065bb70d8-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:16.789 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:22:16.789 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:22:16.789 DEBUG:boto:path=/vclfcn-18/?prefix=dummy 2024-02-01T10:22:16.789 DEBUG:boto:auth_path=/vclfcn-18/?prefix=dummy 2024-02-01T10:22:16.789 DEBUG:boto:Method: GET 2024-02-01T10:22:16.789 DEBUG:boto:Path: /vclfcn-18/?prefix=dummy 2024-02-01T10:22:16.789 DEBUG:boto:Data: 2024-02-01T10:22:16.789 DEBUG:boto:Headers: {} 2024-02-01T10:22:16.789 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:16.789 DEBUG:boto:Port: 8001 2024-02-01T10:22:16.789 DEBUG:boto:Params: {} 2024-02-01T10:22:16.790 DEBUG:boto:Token: None 2024-02-01T10:22:16.790 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:16 GMT /vclfcn-18/ 2024-02-01T10:22:16.790 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ysLB0yg/KO94C8n2rwJHreWKTGw= 2024-02-01T10:22:16.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ysLB0yg/KO94C8n2rwJHreWKTGw=', 'Content-Length': '0'} 2024-02-01T10:22:16.794 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f1eb823f71c6a72-0065bb70d8-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:16.836 DEBUG:boto:b'vclfcn-18dummy1000falsedummy2024-02-01T10:20:34.645Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:16.837 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:22:16.837 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:22:16.837 DEBUG:boto:path=/vclfcn-18/?prefix=dummy 2024-02-01T10:22:16.837 DEBUG:boto:auth_path=/vclfcn-18/?prefix=dummy 2024-02-01T10:22:16.837 DEBUG:boto:Method: GET 2024-02-01T10:22:16.837 DEBUG:boto:Path: /vclfcn-18/?prefix=dummy 2024-02-01T10:22:16.837 DEBUG:boto:Data: 2024-02-01T10:22:16.837 DEBUG:boto:Headers: {} 2024-02-01T10:22:16.837 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:16.837 DEBUG:boto:Port: 8000 2024-02-01T10:22:16.837 DEBUG:boto:Params: {} 2024-02-01T10:22:16.838 DEBUG:boto:Token: None 2024-02-01T10:22:16.838 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:16 GMT /vclfcn-18/ 2024-02-01T10:22:16.838 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ysLB0yg/KO94C8n2rwJHreWKTGw= 2024-02-01T10:22:16.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ysLB0yg/KO94C8n2rwJHreWKTGw=', 'Content-Length': '0'} 2024-02-01T10:22:16.842 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023da3a1008cba0b2-0065bb70d8-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:16.842 DEBUG:boto:b'vclfcn-18dummy1000falsedummy2024-02-01T10:20:34.645Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:16.843 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-02-01T10:22:16.843 DEBUG:boto:path=/vclfcn-18/obj-test-zone1 2024-02-01T10:22:16.843 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1 2024-02-01T10:22:16.843 DEBUG:boto:path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:16.843 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:16.843 DEBUG:boto:Method: GET 2024-02-01T10:22:16.843 DEBUG:boto:Path: /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:16.843 DEBUG:boto:Data: 2024-02-01T10:22:16.844 DEBUG:boto:Headers: {} 2024-02-01T10:22:16.844 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:16.844 DEBUG:boto:Port: 8001 2024-02-01T10:22:16.844 DEBUG:boto:Params: {} 2024-02-01T10:22:16.844 DEBUG:boto:Token: None 2024-02-01T10:22:16.844 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:16 GMT /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:16.844 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ht0ASBuefeB07Ax9LQRgGl9GkNI= 2024-02-01T10:22:16.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ht0ASBuefeB07Ax9LQRgGl9GkNI=', 'Content-Length': '0'} 2024-02-01T10:22:16.847 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000821ce49d6558bd62-0065bb70d8-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:16.848 DEBUG:boto:path=/vclfcn-18/obj-test-zone1 2024-02-01T10:22:16.848 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1 2024-02-01T10:22:16.848 DEBUG:boto:path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:16.848 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:16.848 DEBUG:boto:Method: GET 2024-02-01T10:22:16.848 DEBUG:boto:Path: /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:16.848 DEBUG:boto:Data: 2024-02-01T10:22:16.848 DEBUG:boto:Headers: {} 2024-02-01T10:22:16.848 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:16.848 DEBUG:boto:Port: 8000 2024-02-01T10:22:16.848 DEBUG:boto:Params: {} 2024-02-01T10:22:16.849 DEBUG:boto:Token: None 2024-02-01T10:22:16.849 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:16 GMT /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:16.849 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ht0ASBuefeB07Ax9LQRgGl9GkNI= 2024-02-01T10:22:16.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ht0ASBuefeB07Ax9LQRgGl9GkNI=', 'Content-Length': '0'} 2024-02-01T10:22:16.852 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009952993216474743-0065bb70d8-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:16.852 DEBUG:boto:path=/vclfcn-18/obj-test-zone1 2024-02-01T10:22:16.852 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1 2024-02-01T10:22:16.852 DEBUG:boto:path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:16.853 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:16.853 DEBUG:boto:Method: HEAD 2024-02-01T10:22:16.853 DEBUG:boto:Path: /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:16.853 DEBUG:boto:Data: 2024-02-01T10:22:16.853 DEBUG:boto:Headers: {} 2024-02-01T10:22:16.853 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:16.853 DEBUG:boto:Port: 8001 2024-02-01T10:22:16.853 DEBUG:boto:Params: {} 2024-02-01T10:22:16.853 DEBUG:boto:Token: None 2024-02-01T10:22:16.853 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:16 GMT /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:16.854 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RGy5K4B0RIRzlQyBoXR8719Kf/c= 2024-02-01T10:22:16.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RGy5K4B0RIRzlQyBoXR8719Kf/c=', 'Content-Length': '0'} 2024-02-01T10:22:16.856 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a259b102d38425f0-0065bb70d8-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:16.856 DEBUG:boto:path=/vclfcn-18/obj-test-zone1 2024-02-01T10:22:16.857 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1 2024-02-01T10:22:16.857 DEBUG:boto:path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:16.857 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:16.857 DEBUG:boto:Method: HEAD 2024-02-01T10:22:16.857 DEBUG:boto:Path: /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:16.857 DEBUG:boto:Data: 2024-02-01T10:22:16.857 DEBUG:boto:Headers: {} 2024-02-01T10:22:16.857 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:16.857 DEBUG:boto:Port: 8000 2024-02-01T10:22:16.857 DEBUG:boto:Params: {} 2024-02-01T10:22:16.857 DEBUG:boto:Token: None 2024-02-01T10:22:16.858 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:16 GMT /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:16.858 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RGy5K4B0RIRzlQyBoXR8719Kf/c= 2024-02-01T10:22:16.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RGy5K4B0RIRzlQyBoXR8719Kf/c=', 'Content-Length': '0'} 2024-02-01T10:22:16.860 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ba90187f2a331fe9-0065bb70d8-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:16.861 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-02-01T10:22:16.861 DEBUG:boto:path=/vclfcn-18/obj-test-zone1 2024-02-01T10:22:16.861 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1 2024-02-01T10:22:16.861 DEBUG:boto:path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:16.861 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:16.861 DEBUG:boto:Method: GET 2024-02-01T10:22:16.861 DEBUG:boto:Path: /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:16.861 DEBUG:boto:Data: 2024-02-01T10:22:16.862 DEBUG:boto:Headers: {} 2024-02-01T10:22:16.862 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:16.862 DEBUG:boto:Port: 8001 2024-02-01T10:22:16.862 DEBUG:boto:Params: {} 2024-02-01T10:22:16.862 DEBUG:boto:Token: None 2024-02-01T10:22:16.862 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:16 GMT /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:16.862 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ht0ASBuefeB07Ax9LQRgGl9GkNI= 2024-02-01T10:22:16.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ht0ASBuefeB07Ax9LQRgGl9GkNI=', 'Content-Length': '0'} 2024-02-01T10:22:16.865 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e6173e1d67848987-0065bb70d8-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:16.908 DEBUG:boto:path=/vclfcn-18/obj-test-zone1 2024-02-01T10:22:16.908 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1 2024-02-01T10:22:16.908 DEBUG:boto:path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:16.908 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:16.909 DEBUG:boto:Method: GET 2024-02-01T10:22:16.909 DEBUG:boto:Path: /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:16.909 DEBUG:boto:Data: 2024-02-01T10:22:16.909 DEBUG:boto:Headers: {} 2024-02-01T10:22:16.909 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:16.909 DEBUG:boto:Port: 8000 2024-02-01T10:22:16.909 DEBUG:boto:Params: {} 2024-02-01T10:22:16.909 DEBUG:boto:Token: None 2024-02-01T10:22:16.909 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:16 GMT /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:16.909 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ht0ASBuefeB07Ax9LQRgGl9GkNI= 2024-02-01T10:22:16.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ht0ASBuefeB07Ax9LQRgGl9GkNI=', 'Content-Length': '0'} 2024-02-01T10:22:16.913 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e0920c4726839a02-0065bb70d8-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:16.956 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:22:16.956 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:22:16.956 DEBUG:boto:path=/vclfcn-18/?prefix=obj-test-zone1 2024-02-01T10:22:16.956 DEBUG:boto:auth_path=/vclfcn-18/?prefix=obj-test-zone1 2024-02-01T10:22:16.957 DEBUG:boto:Method: GET 2024-02-01T10:22:16.957 DEBUG:boto:Path: /vclfcn-18/?prefix=obj-test-zone1 2024-02-01T10:22:16.957 DEBUG:boto:Data: 2024-02-01T10:22:16.957 DEBUG:boto:Headers: {} 2024-02-01T10:22:16.957 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:16.957 DEBUG:boto:Port: 8001 2024-02-01T10:22:16.957 DEBUG:boto:Params: {} 2024-02-01T10:22:16.957 DEBUG:boto:Token: None 2024-02-01T10:22:16.957 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:16 GMT /vclfcn-18/ 2024-02-01T10:22:16.958 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ysLB0yg/KO94C8n2rwJHreWKTGw= 2024-02-01T10:22:16.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ysLB0yg/KO94C8n2rwJHreWKTGw=', 'Content-Length': '0'} 2024-02-01T10:22:16.964 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ecec3fc7610a6cd-0065bb70d8-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:17.008 DEBUG:boto:b'vclfcn-18obj-test-zone11000falseobj-test-zone12024-02-01T10:20:50.942Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:17.009 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:22:17.009 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:22:17.009 DEBUG:boto:path=/vclfcn-18/?prefix=obj-test-zone1 2024-02-01T10:22:17.009 DEBUG:boto:auth_path=/vclfcn-18/?prefix=obj-test-zone1 2024-02-01T10:22:17.009 DEBUG:boto:Method: GET 2024-02-01T10:22:17.009 DEBUG:boto:Path: /vclfcn-18/?prefix=obj-test-zone1 2024-02-01T10:22:17.009 DEBUG:boto:Data: 2024-02-01T10:22:17.009 DEBUG:boto:Headers: {} 2024-02-01T10:22:17.009 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:17.009 DEBUG:boto:Port: 8000 2024-02-01T10:22:17.009 DEBUG:boto:Params: {} 2024-02-01T10:22:17.010 DEBUG:boto:Token: None 2024-02-01T10:22:17.010 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:17 GMT /vclfcn-18/ 2024-02-01T10:22:17.010 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:laAyVxrDg6ToNUvy3d/KTWr89Y0= 2024-02-01T10:22:17.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:laAyVxrDg6ToNUvy3d/KTWr89Y0=', 'Content-Length': '0'} 2024-02-01T10:22:17.014 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d969f1090efab719-0065bb70d9-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:17.056 DEBUG:boto:b'vclfcn-18obj-test-zone11000falseobj-test-zone12024-02-01T10:20:50.942Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:17.057 DEBUG:boto:path=/vclfcn-18/obj-test-zone1 2024-02-01T10:22:17.057 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1 2024-02-01T10:22:17.057 DEBUG:boto:Method: HEAD 2024-02-01T10:22:17.057 DEBUG:boto:Path: /vclfcn-18/obj-test-zone1 2024-02-01T10:22:17.057 DEBUG:boto:Data: 2024-02-01T10:22:17.057 DEBUG:boto:Headers: {} 2024-02-01T10:22:17.057 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:17.057 DEBUG:boto:Port: 8001 2024-02-01T10:22:17.057 DEBUG:boto:Params: {} 2024-02-01T10:22:17.057 DEBUG:boto:Token: None 2024-02-01T10:22:17.058 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:17 GMT /vclfcn-18/obj-test-zone1 2024-02-01T10:22:17.058 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Xw9tUZ8+6wxOnxpE9Ihg0zNsFts= 2024-02-01T10:22:17.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Xw9tUZ8+6wxOnxpE9Ihg0zNsFts=', 'Content-Length': '0'} 2024-02-01T10:22:17.062 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000032d4be1b0ec77c3d-0065bb70d9-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:17.062 DEBUG:boto:path=/vclfcn-18/obj-test-zone1 2024-02-01T10:22:17.062 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1 2024-02-01T10:22:17.062 DEBUG:boto:Method: HEAD 2024-02-01T10:22:17.062 DEBUG:boto:Path: /vclfcn-18/obj-test-zone1 2024-02-01T10:22:17.063 DEBUG:boto:Data: 2024-02-01T10:22:17.063 DEBUG:boto:Headers: {} 2024-02-01T10:22:17.063 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:17.063 DEBUG:boto:Port: 8000 2024-02-01T10:22:17.063 DEBUG:boto:Params: {} 2024-02-01T10:22:17.063 DEBUG:boto:Token: None 2024-02-01T10:22:17.063 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:17 GMT /vclfcn-18/obj-test-zone1 2024-02-01T10:22:17.063 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Xw9tUZ8+6wxOnxpE9Ihg0zNsFts= 2024-02-01T10:22:17.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Xw9tUZ8+6wxOnxpE9Ihg0zNsFts=', 'Content-Length': '0'} 2024-02-01T10:22:17.067 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b61c86dc3d44993a-0065bb70d9-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:17.067 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-02-01T10:22:17.068 DEBUG:boto:path=/vclfcn-18/obj-test-zone1 2024-02-01T10:22:17.068 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1 2024-02-01T10:22:17.068 DEBUG:boto:path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:17.068 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:17.068 DEBUG:boto:Method: GET 2024-02-01T10:22:17.068 DEBUG:boto:Path: /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:17.068 DEBUG:boto:Data: 2024-02-01T10:22:17.068 DEBUG:boto:Headers: {} 2024-02-01T10:22:17.068 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:17.068 DEBUG:boto:Port: 8001 2024-02-01T10:22:17.068 DEBUG:boto:Params: {} 2024-02-01T10:22:17.069 DEBUG:boto:Token: None 2024-02-01T10:22:17.069 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:17 GMT /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:17.069 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:57xLgRyrfe907qsMkTBH82ph7sE= 2024-02-01T10:22:17.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:57xLgRyrfe907qsMkTBH82ph7sE=', 'Content-Length': '0'} 2024-02-01T10:22:17.072 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000013de7e5e98fb38d8-0065bb70d9-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:17.120 DEBUG:boto:path=/vclfcn-18/obj-test-zone1 2024-02-01T10:22:17.120 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1 2024-02-01T10:22:17.120 DEBUG:boto:path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:17.120 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:17.121 DEBUG:boto:Method: GET 2024-02-01T10:22:17.121 DEBUG:boto:Path: /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:17.121 DEBUG:boto:Data: 2024-02-01T10:22:17.121 DEBUG:boto:Headers: {} 2024-02-01T10:22:17.121 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:17.121 DEBUG:boto:Port: 8000 2024-02-01T10:22:17.121 DEBUG:boto:Params: {} 2024-02-01T10:22:17.121 DEBUG:boto:Token: None 2024-02-01T10:22:17.121 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:17 GMT /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:22:17.122 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:57xLgRyrfe907qsMkTBH82ph7sE= 2024-02-01T10:22:17.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:57xLgRyrfe907qsMkTBH82ph7sE=', 'Content-Length': '0'} 2024-02-01T10:22:17.124 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a5eae913ab2f1183-0065bb70d9-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:17.125 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:22:17.125 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:22:17.125 DEBUG:boto:path=/vclfcn-18/?prefix=obj-test-zone1 2024-02-01T10:22:17.125 DEBUG:boto:auth_path=/vclfcn-18/?prefix=obj-test-zone1 2024-02-01T10:22:17.125 DEBUG:boto:Method: GET 2024-02-01T10:22:17.125 DEBUG:boto:Path: /vclfcn-18/?prefix=obj-test-zone1 2024-02-01T10:22:17.125 DEBUG:boto:Data: 2024-02-01T10:22:17.126 DEBUG:boto:Headers: {} 2024-02-01T10:22:17.126 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:17.126 DEBUG:boto:Port: 8001 2024-02-01T10:22:17.126 DEBUG:boto:Params: {} 2024-02-01T10:22:17.126 DEBUG:boto:Token: None 2024-02-01T10:22:17.126 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:17 GMT /vclfcn-18/ 2024-02-01T10:22:17.126 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:laAyVxrDg6ToNUvy3d/KTWr89Y0= 2024-02-01T10:22:17.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:laAyVxrDg6ToNUvy3d/KTWr89Y0=', 'Content-Length': '0'} 2024-02-01T10:22:17.130 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce0c0e013aa9f320-0065bb70d9-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:17.131 DEBUG:boto:b'vclfcn-18obj-test-zone11000falseobj-test-zone12024-02-01T10:20:50.942Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:17.131 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:22:17.131 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:22:17.131 DEBUG:boto:path=/vclfcn-18/?prefix=obj-test-zone1 2024-02-01T10:22:17.131 DEBUG:boto:auth_path=/vclfcn-18/?prefix=obj-test-zone1 2024-02-01T10:22:17.131 DEBUG:boto:Method: GET 2024-02-01T10:22:17.132 DEBUG:boto:Path: /vclfcn-18/?prefix=obj-test-zone1 2024-02-01T10:22:17.132 DEBUG:boto:Data: 2024-02-01T10:22:17.132 DEBUG:boto:Headers: {} 2024-02-01T10:22:17.132 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:17.132 DEBUG:boto:Port: 8000 2024-02-01T10:22:17.132 DEBUG:boto:Params: {} 2024-02-01T10:22:17.132 DEBUG:boto:Token: None 2024-02-01T10:22:17.132 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:17 GMT /vclfcn-18/ 2024-02-01T10:22:17.132 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:laAyVxrDg6ToNUvy3d/KTWr89Y0= 2024-02-01T10:22:17.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:laAyVxrDg6ToNUvy3d/KTWr89Y0=', 'Content-Length': '0'} 2024-02-01T10:22:17.136 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df993dca8cee3386-0065bb70d9-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:17.137 DEBUG:boto:b'vclfcn-18obj-test-zone11000falseobj-test-zone12024-02-01T10:20:50.942Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:17.137 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-02-01T10:22:17.137 DEBUG:boto:path=/vclfcn-18/obj-test-zone2 2024-02-01T10:22:17.137 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2 2024-02-01T10:22:17.137 DEBUG:boto:path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:17.137 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:17.137 DEBUG:boto:Method: GET 2024-02-01T10:22:17.138 DEBUG:boto:Path: /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:17.138 DEBUG:boto:Data: 2024-02-01T10:22:17.138 DEBUG:boto:Headers: {} 2024-02-01T10:22:17.138 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:17.138 DEBUG:boto:Port: 8001 2024-02-01T10:22:17.138 DEBUG:boto:Params: {} 2024-02-01T10:22:17.138 DEBUG:boto:Token: None 2024-02-01T10:22:17.138 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:17 GMT /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:17.138 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WaU5NhpCXeEzmmtxlSVyGHthWdU= 2024-02-01T10:22:17.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WaU5NhpCXeEzmmtxlSVyGHthWdU=', 'Content-Length': '0'} 2024-02-01T10:22:17.141 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', 'ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c211d585396e1eff-0065bb70d9-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:17.184 DEBUG:boto:path=/vclfcn-18/obj-test-zone2 2024-02-01T10:22:17.184 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2 2024-02-01T10:22:17.184 DEBUG:boto:path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:17.185 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:17.185 DEBUG:boto:Method: GET 2024-02-01T10:22:17.185 DEBUG:boto:Path: /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:17.185 DEBUG:boto:Data: 2024-02-01T10:22:17.185 DEBUG:boto:Headers: {} 2024-02-01T10:22:17.185 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:17.185 DEBUG:boto:Port: 8000 2024-02-01T10:22:17.185 DEBUG:boto:Params: {} 2024-02-01T10:22:17.185 DEBUG:boto:Token: None 2024-02-01T10:22:17.186 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:17 GMT /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:17.186 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WaU5NhpCXeEzmmtxlSVyGHthWdU= 2024-02-01T10:22:17.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WaU5NhpCXeEzmmtxlSVyGHthWdU=', 'Content-Length': '0'} 2024-02-01T10:22:17.189 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', 'ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000053fd058740f94c6a-0065bb70d9-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:17.190 DEBUG:boto:path=/vclfcn-18/obj-test-zone2 2024-02-01T10:22:17.190 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2 2024-02-01T10:22:17.190 DEBUG:boto:path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:17.190 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:17.190 DEBUG:boto:Method: HEAD 2024-02-01T10:22:17.190 DEBUG:boto:Path: /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:17.191 DEBUG:boto:Data: 2024-02-01T10:22:17.191 DEBUG:boto:Headers: {} 2024-02-01T10:22:17.191 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:17.191 DEBUG:boto:Port: 8001 2024-02-01T10:22:17.191 DEBUG:boto:Params: {} 2024-02-01T10:22:17.191 DEBUG:boto:Token: None 2024-02-01T10:22:17.192 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:17 GMT /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:17.192 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:iI57+SMXSmRoQnobxkIIHZs78JY= 2024-02-01T10:22:17.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iI57+SMXSmRoQnobxkIIHZs78JY=', 'Content-Length': '0'} 2024-02-01T10:22:17.195 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', 'ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000be6cf2c2cbbac5e2-0065bb70d9-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:17.195 DEBUG:boto:path=/vclfcn-18/obj-test-zone2 2024-02-01T10:22:17.195 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2 2024-02-01T10:22:17.195 DEBUG:boto:path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:17.195 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:17.195 DEBUG:boto:Method: HEAD 2024-02-01T10:22:17.196 DEBUG:boto:Path: /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:17.196 DEBUG:boto:Data: 2024-02-01T10:22:17.196 DEBUG:boto:Headers: {} 2024-02-01T10:22:17.196 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:17.196 DEBUG:boto:Port: 8000 2024-02-01T10:22:17.196 DEBUG:boto:Params: {} 2024-02-01T10:22:17.196 DEBUG:boto:Token: None 2024-02-01T10:22:17.196 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:17 GMT /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:17.197 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:iI57+SMXSmRoQnobxkIIHZs78JY= 2024-02-01T10:22:17.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iI57+SMXSmRoQnobxkIIHZs78JY=', 'Content-Length': '0'} 2024-02-01T10:22:17.199 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', 'ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002e96e084ac8166e2-0065bb70d9-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:17.199 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-02-01T10:22:17.200 DEBUG:boto:path=/vclfcn-18/obj-test-zone2 2024-02-01T10:22:17.200 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2 2024-02-01T10:22:17.200 DEBUG:boto:path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:17.200 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:17.200 DEBUG:boto:Method: GET 2024-02-01T10:22:17.200 DEBUG:boto:Path: /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:17.200 DEBUG:boto:Data: 2024-02-01T10:22:17.200 DEBUG:boto:Headers: {} 2024-02-01T10:22:17.201 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:17.201 DEBUG:boto:Port: 8001 2024-02-01T10:22:17.201 DEBUG:boto:Params: {} 2024-02-01T10:22:17.201 DEBUG:boto:Token: None 2024-02-01T10:22:17.201 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:17 GMT /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:17.201 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WaU5NhpCXeEzmmtxlSVyGHthWdU= 2024-02-01T10:22:17.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WaU5NhpCXeEzmmtxlSVyGHthWdU=', 'Content-Length': '0'} 2024-02-01T10:22:17.204 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', 'ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004e5f8005061ee76e-0065bb70d9-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:17.248 DEBUG:boto:path=/vclfcn-18/obj-test-zone2 2024-02-01T10:22:17.248 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2 2024-02-01T10:22:17.248 DEBUG:boto:path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:17.248 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:17.248 DEBUG:boto:Method: GET 2024-02-01T10:22:17.249 DEBUG:boto:Path: /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:17.249 DEBUG:boto:Data: 2024-02-01T10:22:17.249 DEBUG:boto:Headers: {} 2024-02-01T10:22:17.249 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:17.249 DEBUG:boto:Port: 8000 2024-02-01T10:22:17.249 DEBUG:boto:Params: {} 2024-02-01T10:22:17.249 DEBUG:boto:Token: None 2024-02-01T10:22:17.249 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:17 GMT /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:17.249 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WaU5NhpCXeEzmmtxlSVyGHthWdU= 2024-02-01T10:22:17.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WaU5NhpCXeEzmmtxlSVyGHthWdU=', 'Content-Length': '0'} 2024-02-01T10:22:17.252 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', 'ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000034af4f34f036c44-0065bb70d9-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:17.253 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:22:17.253 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:22:17.253 DEBUG:boto:path=/vclfcn-18/?prefix=obj-test-zone2 2024-02-01T10:22:17.253 DEBUG:boto:auth_path=/vclfcn-18/?prefix=obj-test-zone2 2024-02-01T10:22:17.253 DEBUG:boto:Method: GET 2024-02-01T10:22:17.254 DEBUG:boto:Path: /vclfcn-18/?prefix=obj-test-zone2 2024-02-01T10:22:17.254 DEBUG:boto:Data: 2024-02-01T10:22:17.254 DEBUG:boto:Headers: {} 2024-02-01T10:22:17.254 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:17.254 DEBUG:boto:Port: 8001 2024-02-01T10:22:17.254 DEBUG:boto:Params: {} 2024-02-01T10:22:17.254 DEBUG:boto:Token: None 2024-02-01T10:22:17.254 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:17 GMT /vclfcn-18/ 2024-02-01T10:22:17.254 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:laAyVxrDg6ToNUvy3d/KTWr89Y0= 2024-02-01T10:22:17.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:laAyVxrDg6ToNUvy3d/KTWr89Y0=', 'Content-Length': '0'} 2024-02-01T10:22:17.259 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000036d084008bd3252a-0065bb70d9-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:17.259 DEBUG:boto:b'vclfcn-18obj-test-zone21000falseobj-test-zone22024-02-01T10:20:51.094Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:17.260 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:22:17.260 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:22:17.260 DEBUG:boto:path=/vclfcn-18/?prefix=obj-test-zone2 2024-02-01T10:22:17.260 DEBUG:boto:auth_path=/vclfcn-18/?prefix=obj-test-zone2 2024-02-01T10:22:17.260 DEBUG:boto:Method: GET 2024-02-01T10:22:17.260 DEBUG:boto:Path: /vclfcn-18/?prefix=obj-test-zone2 2024-02-01T10:22:17.260 DEBUG:boto:Data: 2024-02-01T10:22:17.260 DEBUG:boto:Headers: {} 2024-02-01T10:22:17.261 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:17.261 DEBUG:boto:Port: 8000 2024-02-01T10:22:17.261 DEBUG:boto:Params: {} 2024-02-01T10:22:17.261 DEBUG:boto:Token: None 2024-02-01T10:22:17.261 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:17 GMT /vclfcn-18/ 2024-02-01T10:22:17.261 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:laAyVxrDg6ToNUvy3d/KTWr89Y0= 2024-02-01T10:22:17.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:laAyVxrDg6ToNUvy3d/KTWr89Y0=', 'Content-Length': '0'} 2024-02-01T10:22:17.266 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000249c5b2c97dac936-0065bb70d9-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:17.266 DEBUG:boto:b'vclfcn-18obj-test-zone21000falseobj-test-zone22024-02-01T10:20:51.094Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:17.266 DEBUG:boto:path=/vclfcn-18/obj-test-zone2 2024-02-01T10:22:17.266 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2 2024-02-01T10:22:17.267 DEBUG:boto:Method: HEAD 2024-02-01T10:22:17.267 DEBUG:boto:Path: /vclfcn-18/obj-test-zone2 2024-02-01T10:22:17.267 DEBUG:boto:Data: 2024-02-01T10:22:17.267 DEBUG:boto:Headers: {} 2024-02-01T10:22:17.267 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:17.267 DEBUG:boto:Port: 8001 2024-02-01T10:22:17.267 DEBUG:boto:Params: {} 2024-02-01T10:22:17.267 DEBUG:boto:Token: None 2024-02-01T10:22:17.267 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:17 GMT /vclfcn-18/obj-test-zone2 2024-02-01T10:22:17.268 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ukQTH0DulYMEiwCs5UL6o3E9vcc= 2024-02-01T10:22:17.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ukQTH0DulYMEiwCs5UL6o3E9vcc=', 'Content-Length': '0'} 2024-02-01T10:22:17.271 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', 'ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cd178dfa56799cc5-0065bb70d9-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:17.272 DEBUG:boto:path=/vclfcn-18/obj-test-zone2 2024-02-01T10:22:17.272 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2 2024-02-01T10:22:17.272 DEBUG:boto:Method: HEAD 2024-02-01T10:22:17.272 DEBUG:boto:Path: /vclfcn-18/obj-test-zone2 2024-02-01T10:22:17.272 DEBUG:boto:Data: 2024-02-01T10:22:17.272 DEBUG:boto:Headers: {} 2024-02-01T10:22:17.272 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:17.273 DEBUG:boto:Port: 8000 2024-02-01T10:22:17.273 DEBUG:boto:Params: {} 2024-02-01T10:22:17.273 DEBUG:boto:Token: None 2024-02-01T10:22:17.273 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:17 GMT /vclfcn-18/obj-test-zone2 2024-02-01T10:22:17.273 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ukQTH0DulYMEiwCs5UL6o3E9vcc= 2024-02-01T10:22:17.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ukQTH0DulYMEiwCs5UL6o3E9vcc=', 'Content-Length': '0'} 2024-02-01T10:22:17.277 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', 'ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007f5e42478d2bcbb4-0065bb70d9-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:17.277 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-02-01T10:22:17.277 DEBUG:boto:path=/vclfcn-18/obj-test-zone2 2024-02-01T10:22:17.277 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2 2024-02-01T10:22:17.277 DEBUG:boto:path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:17.277 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:17.277 DEBUG:boto:Method: GET 2024-02-01T10:22:17.278 DEBUG:boto:Path: /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:17.278 DEBUG:boto:Data: 2024-02-01T10:22:17.278 DEBUG:boto:Headers: {} 2024-02-01T10:22:17.278 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:17.278 DEBUG:boto:Port: 8001 2024-02-01T10:22:17.278 DEBUG:boto:Params: {} 2024-02-01T10:22:17.278 DEBUG:boto:Token: None 2024-02-01T10:22:17.278 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:17 GMT /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:17.278 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WaU5NhpCXeEzmmtxlSVyGHthWdU= 2024-02-01T10:22:17.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WaU5NhpCXeEzmmtxlSVyGHthWdU=', 'Content-Length': '0'} 2024-02-01T10:22:17.281 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', 'ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000088dd3f48c75ac99-0065bb70d9-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:17.324 DEBUG:boto:path=/vclfcn-18/obj-test-zone2 2024-02-01T10:22:17.324 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2 2024-02-01T10:22:17.325 DEBUG:boto:path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:17.325 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:17.325 DEBUG:boto:Method: GET 2024-02-01T10:22:17.325 DEBUG:boto:Path: /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:17.325 DEBUG:boto:Data: 2024-02-01T10:22:17.325 DEBUG:boto:Headers: {} 2024-02-01T10:22:17.325 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:17.325 DEBUG:boto:Port: 8000 2024-02-01T10:22:17.325 DEBUG:boto:Params: {} 2024-02-01T10:22:17.326 DEBUG:boto:Token: None 2024-02-01T10:22:17.326 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:17 GMT /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:22:17.326 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WaU5NhpCXeEzmmtxlSVyGHthWdU= 2024-02-01T10:22:17.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WaU5NhpCXeEzmmtxlSVyGHthWdU=', 'Content-Length': '0'} 2024-02-01T10:22:17.329 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', 'ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003f79dde0b5c50a72-0065bb70d9-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:17.329 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:22:17.329 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:22:17.329 DEBUG:boto:path=/vclfcn-18/?prefix=obj-test-zone2 2024-02-01T10:22:17.329 DEBUG:boto:auth_path=/vclfcn-18/?prefix=obj-test-zone2 2024-02-01T10:22:17.329 DEBUG:boto:Method: GET 2024-02-01T10:22:17.330 DEBUG:boto:Path: /vclfcn-18/?prefix=obj-test-zone2 2024-02-01T10:22:17.330 DEBUG:boto:Data: 2024-02-01T10:22:17.330 DEBUG:boto:Headers: {} 2024-02-01T10:22:17.330 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:17.330 DEBUG:boto:Port: 8001 2024-02-01T10:22:17.330 DEBUG:boto:Params: {} 2024-02-01T10:22:17.330 DEBUG:boto:Token: None 2024-02-01T10:22:17.330 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:17 GMT /vclfcn-18/ 2024-02-01T10:22:17.330 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:laAyVxrDg6ToNUvy3d/KTWr89Y0= 2024-02-01T10:22:17.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:laAyVxrDg6ToNUvy3d/KTWr89Y0=', 'Content-Length': '0'} 2024-02-01T10:22:17.335 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d73781c79d79d2f-0065bb70d9-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:17.335 DEBUG:boto:b'vclfcn-18obj-test-zone21000falseobj-test-zone22024-02-01T10:20:51.094Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:17.335 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:22:17.335 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:22:17.336 DEBUG:boto:path=/vclfcn-18/?prefix=obj-test-zone2 2024-02-01T10:22:17.336 DEBUG:boto:auth_path=/vclfcn-18/?prefix=obj-test-zone2 2024-02-01T10:22:17.336 DEBUG:boto:Method: GET 2024-02-01T10:22:17.336 DEBUG:boto:Path: /vclfcn-18/?prefix=obj-test-zone2 2024-02-01T10:22:17.336 DEBUG:boto:Data: 2024-02-01T10:22:17.336 DEBUG:boto:Headers: {} 2024-02-01T10:22:17.336 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:17.336 DEBUG:boto:Port: 8000 2024-02-01T10:22:17.336 DEBUG:boto:Params: {} 2024-02-01T10:22:17.336 DEBUG:boto:Token: None 2024-02-01T10:22:17.337 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:17 GMT /vclfcn-18/ 2024-02-01T10:22:17.337 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:laAyVxrDg6ToNUvy3d/KTWr89Y0= 2024-02-01T10:22:17.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:laAyVxrDg6ToNUvy3d/KTWr89Y0=', 'Content-Length': '0'} 2024-02-01T10:22:17.341 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f91cc6e9bc4023f-0065bb70d9-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:17.384 DEBUG:boto:b'vclfcn-18obj-test-zone21000falseobj-test-zone22024-02-01T10:20:51.094Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:17.385 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-02-01T10:22:17.385 DEBUG:boto:path=/vclfcn-18/obj-test-zone3 2024-02-01T10:22:17.385 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3 2024-02-01T10:22:17.385 DEBUG:boto:path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:17.385 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:17.385 DEBUG:boto:Method: GET 2024-02-01T10:22:17.385 DEBUG:boto:Path: /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:17.385 DEBUG:boto:Data: 2024-02-01T10:22:17.385 DEBUG:boto:Headers: {} 2024-02-01T10:22:17.386 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:17.386 DEBUG:boto:Port: 8001 2024-02-01T10:22:17.386 DEBUG:boto:Params: {} 2024-02-01T10:22:17.386 DEBUG:boto:Token: None 2024-02-01T10:22:17.386 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:17 GMT /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:17.386 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fJPio7oY6OkZ4fWQ6X1rthe1cqk= 2024-02-01T10:22:17.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fJPio7oY6OkZ4fWQ6X1rthe1cqk=', 'Content-Length': '0'} 2024-02-01T10:22:17.389 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', '6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ca4d61e31378d49d-0065bb70d9-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:17.389 DEBUG:boto:path=/vclfcn-18/obj-test-zone3 2024-02-01T10:22:17.390 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3 2024-02-01T10:22:17.390 DEBUG:boto:path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:17.390 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:17.390 DEBUG:boto:Method: GET 2024-02-01T10:22:17.390 DEBUG:boto:Path: /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:17.390 DEBUG:boto:Data: 2024-02-01T10:22:17.390 DEBUG:boto:Headers: {} 2024-02-01T10:22:17.390 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:17.390 DEBUG:boto:Port: 8000 2024-02-01T10:22:17.390 DEBUG:boto:Params: {} 2024-02-01T10:22:17.391 DEBUG:boto:Token: None 2024-02-01T10:22:17.391 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:17 GMT /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:17.391 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fJPio7oY6OkZ4fWQ6X1rthe1cqk= 2024-02-01T10:22:17.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fJPio7oY6OkZ4fWQ6X1rthe1cqk=', 'Content-Length': '0'} 2024-02-01T10:22:17.394 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', '6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bcb326be389f7d50-0065bb70d9-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:17.394 DEBUG:boto:path=/vclfcn-18/obj-test-zone3 2024-02-01T10:22:17.394 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3 2024-02-01T10:22:17.394 DEBUG:boto:path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:17.395 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:17.395 DEBUG:boto:Method: HEAD 2024-02-01T10:22:17.395 DEBUG:boto:Path: /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:17.395 DEBUG:boto:Data: 2024-02-01T10:22:17.395 DEBUG:boto:Headers: {} 2024-02-01T10:22:17.395 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:17.395 DEBUG:boto:Port: 8001 2024-02-01T10:22:17.395 DEBUG:boto:Params: {} 2024-02-01T10:22:17.395 DEBUG:boto:Token: None 2024-02-01T10:22:17.396 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:17 GMT /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:17.396 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZUad6XG/fYtyqdr5Jyn4OZ6SiSI= 2024-02-01T10:22:17.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZUad6XG/fYtyqdr5Jyn4OZ6SiSI=', 'Content-Length': '0'} 2024-02-01T10:22:17.399 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', '6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d7281572b493796d-0065bb70d9-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:17.399 DEBUG:boto:path=/vclfcn-18/obj-test-zone3 2024-02-01T10:22:17.399 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3 2024-02-01T10:22:17.399 DEBUG:boto:path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:17.399 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:17.399 DEBUG:boto:Method: HEAD 2024-02-01T10:22:17.399 DEBUG:boto:Path: /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:17.400 DEBUG:boto:Data: 2024-02-01T10:22:17.400 DEBUG:boto:Headers: {} 2024-02-01T10:22:17.400 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:17.400 DEBUG:boto:Port: 8000 2024-02-01T10:22:17.400 DEBUG:boto:Params: {} 2024-02-01T10:22:17.400 DEBUG:boto:Token: None 2024-02-01T10:22:17.400 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:17 GMT /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:17.400 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZUad6XG/fYtyqdr5Jyn4OZ6SiSI= 2024-02-01T10:22:17.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZUad6XG/fYtyqdr5Jyn4OZ6SiSI=', 'Content-Length': '0'} 2024-02-01T10:22:17.403 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', '6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eac45e885926efe7-0065bb70d9-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:17.403 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-02-01T10:22:17.404 DEBUG:boto:path=/vclfcn-18/obj-test-zone3 2024-02-01T10:22:17.404 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3 2024-02-01T10:22:17.404 DEBUG:boto:path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:17.404 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:17.404 DEBUG:boto:Method: GET 2024-02-01T10:22:17.404 DEBUG:boto:Path: /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:17.404 DEBUG:boto:Data: 2024-02-01T10:22:17.404 DEBUG:boto:Headers: {} 2024-02-01T10:22:17.404 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:17.404 DEBUG:boto:Port: 8001 2024-02-01T10:22:17.405 DEBUG:boto:Params: {} 2024-02-01T10:22:17.405 DEBUG:boto:Token: None 2024-02-01T10:22:17.405 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:17 GMT /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:17.405 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fJPio7oY6OkZ4fWQ6X1rthe1cqk= 2024-02-01T10:22:17.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fJPio7oY6OkZ4fWQ6X1rthe1cqk=', 'Content-Length': '0'} 2024-02-01T10:22:17.408 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', '6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ca67d19e2249d1c6-0065bb70d9-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:17.408 DEBUG:boto:path=/vclfcn-18/obj-test-zone3 2024-02-01T10:22:17.408 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3 2024-02-01T10:22:17.408 DEBUG:boto:path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:17.409 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:17.409 DEBUG:boto:Method: GET 2024-02-01T10:22:17.409 DEBUG:boto:Path: /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:17.409 DEBUG:boto:Data: 2024-02-01T10:22:17.409 DEBUG:boto:Headers: {} 2024-02-01T10:22:17.409 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:17.409 DEBUG:boto:Port: 8000 2024-02-01T10:22:17.409 DEBUG:boto:Params: {} 2024-02-01T10:22:17.409 DEBUG:boto:Token: None 2024-02-01T10:22:17.409 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:17 GMT /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:17.410 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fJPio7oY6OkZ4fWQ6X1rthe1cqk= 2024-02-01T10:22:17.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fJPio7oY6OkZ4fWQ6X1rthe1cqk=', 'Content-Length': '0'} 2024-02-01T10:22:17.412 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', '6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006b5d91856a851776-0065bb70d9-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:17.413 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:22:17.413 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:22:17.413 DEBUG:boto:path=/vclfcn-18/?prefix=obj-test-zone3 2024-02-01T10:22:17.413 DEBUG:boto:auth_path=/vclfcn-18/?prefix=obj-test-zone3 2024-02-01T10:22:17.413 DEBUG:boto:Method: GET 2024-02-01T10:22:17.413 DEBUG:boto:Path: /vclfcn-18/?prefix=obj-test-zone3 2024-02-01T10:22:17.413 DEBUG:boto:Data: 2024-02-01T10:22:17.413 DEBUG:boto:Headers: {} 2024-02-01T10:22:17.413 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:17.414 DEBUG:boto:Port: 8001 2024-02-01T10:22:17.414 DEBUG:boto:Params: {} 2024-02-01T10:22:17.414 DEBUG:boto:Token: None 2024-02-01T10:22:17.414 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:17 GMT /vclfcn-18/ 2024-02-01T10:22:17.414 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:laAyVxrDg6ToNUvy3d/KTWr89Y0= 2024-02-01T10:22:17.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:laAyVxrDg6ToNUvy3d/KTWr89Y0=', 'Content-Length': '0'} 2024-02-01T10:22:17.418 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000062d68d635c60353d-0065bb70d9-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:17.460 DEBUG:boto:b'vclfcn-18obj-test-zone31000falseobj-test-zone32024-02-01T10:20:51.248Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:17.461 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:22:17.462 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:22:17.462 DEBUG:boto:path=/vclfcn-18/?prefix=obj-test-zone3 2024-02-01T10:22:17.462 DEBUG:boto:auth_path=/vclfcn-18/?prefix=obj-test-zone3 2024-02-01T10:22:17.462 DEBUG:boto:Method: GET 2024-02-01T10:22:17.462 DEBUG:boto:Path: /vclfcn-18/?prefix=obj-test-zone3 2024-02-01T10:22:17.463 DEBUG:boto:Data: 2024-02-01T10:22:17.463 DEBUG:boto:Headers: {} 2024-02-01T10:22:17.463 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:17.463 DEBUG:boto:Port: 8000 2024-02-01T10:22:17.463 DEBUG:boto:Params: {} 2024-02-01T10:22:17.464 DEBUG:boto:Token: None 2024-02-01T10:22:17.464 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:17 GMT /vclfcn-18/ 2024-02-01T10:22:17.464 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:laAyVxrDg6ToNUvy3d/KTWr89Y0= 2024-02-01T10:22:17.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:laAyVxrDg6ToNUvy3d/KTWr89Y0=', 'Content-Length': '0'} 2024-02-01T10:22:17.469 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c94dfdad995e7ba0-0065bb70d9-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:17.469 DEBUG:boto:b'vclfcn-18obj-test-zone31000falseobj-test-zone32024-02-01T10:20:51.248Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:17.469 DEBUG:boto:path=/vclfcn-18/obj-test-zone3 2024-02-01T10:22:17.469 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3 2024-02-01T10:22:17.470 DEBUG:boto:Method: HEAD 2024-02-01T10:22:17.470 DEBUG:boto:Path: /vclfcn-18/obj-test-zone3 2024-02-01T10:22:17.470 DEBUG:boto:Data: 2024-02-01T10:22:17.470 DEBUG:boto:Headers: {} 2024-02-01T10:22:17.470 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:17.470 DEBUG:boto:Port: 8001 2024-02-01T10:22:17.470 DEBUG:boto:Params: {} 2024-02-01T10:22:17.470 DEBUG:boto:Token: None 2024-02-01T10:22:17.470 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:17 GMT /vclfcn-18/obj-test-zone3 2024-02-01T10:22:17.470 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1+h0lGt6OFZUOWQ/k0xxrFhDInc= 2024-02-01T10:22:17.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1+h0lGt6OFZUOWQ/k0xxrFhDInc=', 'Content-Length': '0'} 2024-02-01T10:22:17.477 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', '6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000286966fe1e2c6c2b-0065bb70d9-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:17.478 DEBUG:boto:path=/vclfcn-18/obj-test-zone3 2024-02-01T10:22:17.478 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3 2024-02-01T10:22:17.478 DEBUG:boto:Method: HEAD 2024-02-01T10:22:17.478 DEBUG:boto:Path: /vclfcn-18/obj-test-zone3 2024-02-01T10:22:17.478 DEBUG:boto:Data: 2024-02-01T10:22:17.479 DEBUG:boto:Headers: {} 2024-02-01T10:22:17.479 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:17.479 DEBUG:boto:Port: 8000 2024-02-01T10:22:17.479 DEBUG:boto:Params: {} 2024-02-01T10:22:17.479 DEBUG:boto:Token: None 2024-02-01T10:22:17.480 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:17 GMT /vclfcn-18/obj-test-zone3 2024-02-01T10:22:17.480 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1+h0lGt6OFZUOWQ/k0xxrFhDInc= 2024-02-01T10:22:17.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1+h0lGt6OFZUOWQ/k0xxrFhDInc=', 'Content-Length': '0'} 2024-02-01T10:22:17.483 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', '6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009130809fc9d96eb8-0065bb70d9-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:17.484 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-02-01T10:22:17.484 DEBUG:boto:path=/vclfcn-18/obj-test-zone3 2024-02-01T10:22:17.484 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3 2024-02-01T10:22:17.485 DEBUG:boto:path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:17.485 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:17.485 DEBUG:boto:Method: GET 2024-02-01T10:22:17.485 DEBUG:boto:Path: /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:17.485 DEBUG:boto:Data: 2024-02-01T10:22:17.485 DEBUG:boto:Headers: {} 2024-02-01T10:22:17.486 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:17.486 DEBUG:boto:Port: 8001 2024-02-01T10:22:17.486 DEBUG:boto:Params: {} 2024-02-01T10:22:17.486 DEBUG:boto:Token: None 2024-02-01T10:22:17.486 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:17 GMT /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:17.486 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fJPio7oY6OkZ4fWQ6X1rthe1cqk= 2024-02-01T10:22:17.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fJPio7oY6OkZ4fWQ6X1rthe1cqk=', 'Content-Length': '0'} 2024-02-01T10:22:17.489 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', '6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006903f25960298e55-0065bb70d9-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:17.489 DEBUG:boto:path=/vclfcn-18/obj-test-zone3 2024-02-01T10:22:17.490 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3 2024-02-01T10:22:17.490 DEBUG:boto:path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:17.490 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:17.490 DEBUG:boto:Method: GET 2024-02-01T10:22:17.490 DEBUG:boto:Path: /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:17.490 DEBUG:boto:Data: 2024-02-01T10:22:17.490 DEBUG:boto:Headers: {} 2024-02-01T10:22:17.490 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:17.490 DEBUG:boto:Port: 8000 2024-02-01T10:22:17.491 DEBUG:boto:Params: {} 2024-02-01T10:22:17.491 DEBUG:boto:Token: None 2024-02-01T10:22:17.491 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:17 GMT /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:22:17.491 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fJPio7oY6OkZ4fWQ6X1rthe1cqk= 2024-02-01T10:22:17.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fJPio7oY6OkZ4fWQ6X1rthe1cqk=', 'Content-Length': '0'} 2024-02-01T10:22:17.494 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', '6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009d28bed94772e41e-0065bb70d9-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:17.536 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:22:17.536 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:22:17.536 DEBUG:boto:path=/vclfcn-18/?prefix=obj-test-zone3 2024-02-01T10:22:17.537 DEBUG:boto:auth_path=/vclfcn-18/?prefix=obj-test-zone3 2024-02-01T10:22:17.537 DEBUG:boto:Method: GET 2024-02-01T10:22:17.537 DEBUG:boto:Path: /vclfcn-18/?prefix=obj-test-zone3 2024-02-01T10:22:17.537 DEBUG:boto:Data: 2024-02-01T10:22:17.537 DEBUG:boto:Headers: {} 2024-02-01T10:22:17.537 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:17.537 DEBUG:boto:Port: 8001 2024-02-01T10:22:17.537 DEBUG:boto:Params: {} 2024-02-01T10:22:17.537 DEBUG:boto:Token: None 2024-02-01T10:22:17.537 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:17 GMT /vclfcn-18/ 2024-02-01T10:22:17.538 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:laAyVxrDg6ToNUvy3d/KTWr89Y0= 2024-02-01T10:22:17.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:laAyVxrDg6ToNUvy3d/KTWr89Y0=', 'Content-Length': '0'} 2024-02-01T10:22:17.541 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1c51b40dfef3963-0065bb70d9-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:17.542 DEBUG:boto:b'vclfcn-18obj-test-zone31000falseobj-test-zone32024-02-01T10:20:51.248Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:17.542 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:22:17.542 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:22:17.542 DEBUG:boto:path=/vclfcn-18/?prefix=obj-test-zone3 2024-02-01T10:22:17.542 DEBUG:boto:auth_path=/vclfcn-18/?prefix=obj-test-zone3 2024-02-01T10:22:17.543 DEBUG:boto:Method: GET 2024-02-01T10:22:17.543 DEBUG:boto:Path: /vclfcn-18/?prefix=obj-test-zone3 2024-02-01T10:22:17.543 DEBUG:boto:Data: 2024-02-01T10:22:17.543 DEBUG:boto:Headers: {} 2024-02-01T10:22:17.543 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:17.543 DEBUG:boto:Port: 8000 2024-02-01T10:22:17.543 DEBUG:boto:Params: {} 2024-02-01T10:22:17.543 DEBUG:boto:Token: None 2024-02-01T10:22:17.543 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:17 GMT /vclfcn-18/ 2024-02-01T10:22:17.543 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:laAyVxrDg6ToNUvy3d/KTWr89Y0= 2024-02-01T10:22:17.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:laAyVxrDg6ToNUvy3d/KTWr89Y0=', 'Content-Length': '0'} 2024-02-01T10:22:17.548 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ed1d4d6a892a0f2-0065bb70d9-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:17.548 DEBUG:boto:b'vclfcn-18obj-test-zone31000falseobj-test-zone32024-02-01T10:20:51.248Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:17.548 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-18 zones={test-zone3, test-zone2} 2024-02-01T10:22:17.548 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:22:17.549 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:22:17.549 DEBUG:boto:Method: HEAD 2024-02-01T10:22:17.549 DEBUG:boto:Path: /vclfcn-16/ 2024-02-01T10:22:17.549 DEBUG:boto:Data: 2024-02-01T10:22:17.549 DEBUG:boto:Headers: {} 2024-02-01T10:22:17.549 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:17.549 DEBUG:boto:Port: 8000 2024-02-01T10:22:17.549 DEBUG:boto:Params: {} 2024-02-01T10:22:17.549 DEBUG:boto:Token: None 2024-02-01T10:22:17.549 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:17 GMT /vclfcn-16/ 2024-02-01T10:22:17.549 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:siZr/g/Z8jKZvFFnIEHVX53YcSQ= 2024-02-01T10:22:17.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:siZr/g/Z8jKZvFFnIEHVX53YcSQ=', 'Content-Length': '0'} 2024-02-01T10:22:17.553 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009a7546cda9244b86-0065bb70d9-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:22:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:17.554 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:22:17.554 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:22:17.554 DEBUG:boto:path=/vclfcn-16/?versions&max-keys=999&prefix=obj-test-zone1 2024-02-01T10:22:17.554 DEBUG:boto:auth_path=/vclfcn-16/?versions&max-keys=999&prefix=obj-test-zone1 2024-02-01T10:22:17.554 DEBUG:boto:Method: GET 2024-02-01T10:22:17.554 DEBUG:boto:Path: /vclfcn-16/?versions&max-keys=999&prefix=obj-test-zone1 2024-02-01T10:22:17.554 DEBUG:boto:Data: 2024-02-01T10:22:17.554 DEBUG:boto:Headers: {} 2024-02-01T10:22:17.554 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:17.554 DEBUG:boto:Port: 8000 2024-02-01T10:22:17.554 DEBUG:boto:Params: {} 2024-02-01T10:22:17.555 DEBUG:boto:Token: None 2024-02-01T10:22:17.555 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:17 GMT /vclfcn-16/?versions 2024-02-01T10:22:17.555 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:o60OXOTenRHphy7ZzQ4fh+zc0UU= 2024-02-01T10:22:17.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:o60OXOTenRHphy7ZzQ4fh+zc0UU=', 'Content-Length': '0'} 2024-02-01T10:22:17.559 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc7746062352b1f3-0065bb70d9-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:17.559 DEBUG:boto:b'vclfcn-16obj-test-zone1999falseobj-test-zone1lS9-sMOvyR-VcD.47BkPLymGmIyfWmftrue2024-02-01T10:20:49.965Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:17.560 DEBUG:boto:path=/vclfcn-16/obj-test-zone1 2024-02-01T10:22:17.560 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1 2024-02-01T10:22:17.560 DEBUG:boto:path=/vclfcn-16/obj-test-zone1?acl 2024-02-01T10:22:17.560 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1?acl 2024-02-01T10:22:17.560 DEBUG:boto:Method: PUT 2024-02-01T10:22:17.560 DEBUG:boto:Path: /vclfcn-16/obj-test-zone1?acl 2024-02-01T10:22:17.560 DEBUG:boto:Data: 2024-02-01T10:22:17.560 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-02-01T10:22:17.561 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:17.561 DEBUG:boto:Port: 8000 2024-02-01T10:22:17.561 DEBUG:boto:Params: {} 2024-02-01T10:22:17.561 DEBUG:boto:Token: None 2024-02-01T10:22:17.561 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:22:17 GMT x-amz-acl:public-read /vclfcn-16/obj-test-zone1?acl 2024-02-01T10:22:17.561 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:uYb8hCdM6QqK/v977qBAKfnw/2E= 2024-02-01T10:22:17.561 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:uYb8hCdM6QqK/v977qBAKfnw/2E=', 'Content-Length': '0'} 2024-02-01T10:22:17.574 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000071e2bc96720d4410-0065bb70d9-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:22:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:17.574 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:22:17.574 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:22:17.574 DEBUG:boto:Method: HEAD 2024-02-01T10:22:17.574 DEBUG:boto:Path: /vclfcn-16/ 2024-02-01T10:22:17.574 DEBUG:boto:Data: 2024-02-01T10:22:17.574 DEBUG:boto:Headers: {} 2024-02-01T10:22:17.575 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:17.575 DEBUG:boto:Port: 8000 2024-02-01T10:22:17.575 DEBUG:boto:Params: {} 2024-02-01T10:22:17.575 DEBUG:boto:Token: None 2024-02-01T10:22:17.575 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:17 GMT /vclfcn-16/ 2024-02-01T10:22:17.575 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:siZr/g/Z8jKZvFFnIEHVX53YcSQ= 2024-02-01T10:22:17.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:siZr/g/Z8jKZvFFnIEHVX53YcSQ=', 'Content-Length': '0'} 2024-02-01T10:22:17.579 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d7c55eb4de876d22-0065bb70d9-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:22:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:17.579 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:22:17.579 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:22:17.579 DEBUG:boto:path=/vclfcn-16/?versions&max-keys=999&prefix=obj-test-zone2 2024-02-01T10:22:17.579 DEBUG:boto:auth_path=/vclfcn-16/?versions&max-keys=999&prefix=obj-test-zone2 2024-02-01T10:22:17.580 DEBUG:boto:Method: GET 2024-02-01T10:22:17.580 DEBUG:boto:Path: /vclfcn-16/?versions&max-keys=999&prefix=obj-test-zone2 2024-02-01T10:22:17.580 DEBUG:boto:Data: 2024-02-01T10:22:17.580 DEBUG:boto:Headers: {} 2024-02-01T10:22:17.580 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:17.580 DEBUG:boto:Port: 8000 2024-02-01T10:22:17.580 DEBUG:boto:Params: {} 2024-02-01T10:22:17.580 DEBUG:boto:Token: None 2024-02-01T10:22:17.580 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:17 GMT /vclfcn-16/?versions 2024-02-01T10:22:17.580 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:o60OXOTenRHphy7ZzQ4fh+zc0UU= 2024-02-01T10:22:17.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:o60OXOTenRHphy7ZzQ4fh+zc0UU=', 'Content-Length': '0'} 2024-02-01T10:22:17.584 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006986ee460cf066b8-0065bb70d9-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:17.632 DEBUG:boto:b'vclfcn-16obj-test-zone2999falseobj-test-zone2mE6urSxcrRA9wdyuToa41.w2aTia9p1true2024-02-01T10:20:50.122Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:17.633 DEBUG:boto:path=/vclfcn-16/obj-test-zone2 2024-02-01T10:22:17.633 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2 2024-02-01T10:22:17.633 DEBUG:boto:path=/vclfcn-16/obj-test-zone2?acl 2024-02-01T10:22:17.633 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2?acl 2024-02-01T10:22:17.633 DEBUG:boto:Method: PUT 2024-02-01T10:22:17.633 DEBUG:boto:Path: /vclfcn-16/obj-test-zone2?acl 2024-02-01T10:22:17.633 DEBUG:boto:Data: 2024-02-01T10:22:17.633 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-02-01T10:22:17.633 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:17.634 DEBUG:boto:Port: 8000 2024-02-01T10:22:17.634 DEBUG:boto:Params: {} 2024-02-01T10:22:17.634 DEBUG:boto:Token: None 2024-02-01T10:22:17.634 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:22:17 GMT x-amz-acl:public-read /vclfcn-16/obj-test-zone2?acl 2024-02-01T10:22:17.634 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:++MxmnBYj0FjaVdv1frnN332NIs= 2024-02-01T10:22:17.634 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:++MxmnBYj0FjaVdv1frnN332NIs=', 'Content-Length': '0'} 2024-02-01T10:22:17.648 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008b86060b69d7fc7b-0065bb70d9-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:22:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:17.648 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:22:17.648 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:22:17.648 DEBUG:boto:Method: HEAD 2024-02-01T10:22:17.648 DEBUG:boto:Path: /vclfcn-16/ 2024-02-01T10:22:17.648 DEBUG:boto:Data: 2024-02-01T10:22:17.648 DEBUG:boto:Headers: {} 2024-02-01T10:22:17.648 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:17.649 DEBUG:boto:Port: 8001 2024-02-01T10:22:17.649 DEBUG:boto:Params: {} 2024-02-01T10:22:17.649 DEBUG:boto:Token: None 2024-02-01T10:22:17.649 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:17 GMT /vclfcn-16/ 2024-02-01T10:22:17.649 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:siZr/g/Z8jKZvFFnIEHVX53YcSQ= 2024-02-01T10:22:17.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:siZr/g/Z8jKZvFFnIEHVX53YcSQ=', 'Content-Length': '0'} 2024-02-01T10:22:17.653 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007a9e4184ed47c576-0065bb70d9-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:22:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:17.653 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:22:17.654 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:22:17.654 DEBUG:boto:path=/vclfcn-16/?versions&max-keys=999&prefix=obj-test-zone3 2024-02-01T10:22:17.654 DEBUG:boto:auth_path=/vclfcn-16/?versions&max-keys=999&prefix=obj-test-zone3 2024-02-01T10:22:17.654 DEBUG:boto:Method: GET 2024-02-01T10:22:17.654 DEBUG:boto:Path: /vclfcn-16/?versions&max-keys=999&prefix=obj-test-zone3 2024-02-01T10:22:17.654 DEBUG:boto:Data: 2024-02-01T10:22:17.654 DEBUG:boto:Headers: {} 2024-02-01T10:22:17.654 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:17.654 DEBUG:boto:Port: 8001 2024-02-01T10:22:17.654 DEBUG:boto:Params: {} 2024-02-01T10:22:17.654 DEBUG:boto:Token: None 2024-02-01T10:22:17.655 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:17 GMT /vclfcn-16/?versions 2024-02-01T10:22:17.655 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:o60OXOTenRHphy7ZzQ4fh+zc0UU= 2024-02-01T10:22:17.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:o60OXOTenRHphy7ZzQ4fh+zc0UU=', 'Content-Length': '0'} 2024-02-01T10:22:17.658 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b79880bc9bf8757f-0065bb70d9-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:17.700 DEBUG:boto:b'vclfcn-16obj-test-zone3999falseobj-test-zone3Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjPtrue2024-02-01T10:20:50.297Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:17.700 DEBUG:boto:path=/vclfcn-16/obj-test-zone3 2024-02-01T10:22:17.701 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3 2024-02-01T10:22:17.701 DEBUG:boto:path=/vclfcn-16/obj-test-zone3?acl 2024-02-01T10:22:17.701 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3?acl 2024-02-01T10:22:17.701 DEBUG:boto:Method: PUT 2024-02-01T10:22:17.701 DEBUG:boto:Path: /vclfcn-16/obj-test-zone3?acl 2024-02-01T10:22:17.701 DEBUG:boto:Data: 2024-02-01T10:22:17.701 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-02-01T10:22:17.701 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:17.701 DEBUG:boto:Port: 8001 2024-02-01T10:22:17.701 DEBUG:boto:Params: {} 2024-02-01T10:22:17.702 DEBUG:boto:Token: None 2024-02-01T10:22:17.702 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:22:17 GMT x-amz-acl:public-read /vclfcn-16/obj-test-zone3?acl 2024-02-01T10:22:17.702 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:3Yisyekb99UjnlNoi0x8JSG4PmI= 2024-02-01T10:22:17.702 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:3Yisyekb99UjnlNoi0x8JSG4PmI=', 'Content-Length': '0'} 2024-02-01T10:22:17.719 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000043ab4a3c36817559-0065bb70d9-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:22:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:17.720 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:17.720 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:17.720 DEBUG:boto:Method: HEAD 2024-02-01T10:22:17.720 DEBUG:boto:Path: /vclfcn-17/ 2024-02-01T10:22:17.720 DEBUG:boto:Data: 2024-02-01T10:22:17.720 DEBUG:boto:Headers: {} 2024-02-01T10:22:17.721 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:17.721 DEBUG:boto:Port: 8000 2024-02-01T10:22:17.721 DEBUG:boto:Params: {} 2024-02-01T10:22:17.721 DEBUG:boto:Token: None 2024-02-01T10:22:17.721 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:17 GMT /vclfcn-17/ 2024-02-01T10:22:17.721 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FiVRU/RPLdd8789pldT3gwajYLs= 2024-02-01T10:22:17.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FiVRU/RPLdd8789pldT3gwajYLs=', 'Content-Length': '0'} 2024-02-01T10:22:17.725 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000053aee5422fe06782-0065bb70d9-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:22:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:17.726 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:17.726 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:17.726 DEBUG:boto:path=/vclfcn-17/?versions&max-keys=999&prefix=obj-test-zone1 2024-02-01T10:22:17.726 DEBUG:boto:auth_path=/vclfcn-17/?versions&max-keys=999&prefix=obj-test-zone1 2024-02-01T10:22:17.726 DEBUG:boto:Method: GET 2024-02-01T10:22:17.726 DEBUG:boto:Path: /vclfcn-17/?versions&max-keys=999&prefix=obj-test-zone1 2024-02-01T10:22:17.726 DEBUG:boto:Data: 2024-02-01T10:22:17.726 DEBUG:boto:Headers: {} 2024-02-01T10:22:17.726 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:17.726 DEBUG:boto:Port: 8000 2024-02-01T10:22:17.727 DEBUG:boto:Params: {} 2024-02-01T10:22:17.727 DEBUG:boto:Token: None 2024-02-01T10:22:17.727 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:17 GMT /vclfcn-17/?versions 2024-02-01T10:22:17.727 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:pPueLLwGGUJDYWlL36XOnXwKxAg= 2024-02-01T10:22:17.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:pPueLLwGGUJDYWlL36XOnXwKxAg=', 'Content-Length': '0'} 2024-02-01T10:22:17.734 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007372ae678e671f14-0065bb70d9-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:17.734 DEBUG:boto:b'vclfcn-17obj-test-zone1999falseobj-test-zone1xMxAGR4A-xEILepnzFPRgrWNTOLvitztrue2024-02-01T10:20:50.473Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:17.735 DEBUG:boto:path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:17.735 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:17.735 DEBUG:boto:path=/vclfcn-17/obj-test-zone1?acl 2024-02-01T10:22:17.735 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1?acl 2024-02-01T10:22:17.735 DEBUG:boto:Method: PUT 2024-02-01T10:22:17.735 DEBUG:boto:Path: /vclfcn-17/obj-test-zone1?acl 2024-02-01T10:22:17.735 DEBUG:boto:Data: 2024-02-01T10:22:17.735 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-02-01T10:22:17.735 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:17.736 DEBUG:boto:Port: 8000 2024-02-01T10:22:17.736 DEBUG:boto:Params: {} 2024-02-01T10:22:17.736 DEBUG:boto:Token: None 2024-02-01T10:22:17.736 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:22:17 GMT x-amz-acl:public-read /vclfcn-17/obj-test-zone1?acl 2024-02-01T10:22:17.736 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:hQEC/A7I96Ek61xiOxahFYg3BAY= 2024-02-01T10:22:17.736 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:hQEC/A7I96Ek61xiOxahFYg3BAY=', 'Content-Length': '0'} 2024-02-01T10:22:17.749 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000003683c2cd2ec6f81-0065bb70d9-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:22:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:17.749 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:17.749 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:17.750 DEBUG:boto:Method: HEAD 2024-02-01T10:22:17.750 DEBUG:boto:Path: /vclfcn-17/ 2024-02-01T10:22:17.750 DEBUG:boto:Data: 2024-02-01T10:22:17.750 DEBUG:boto:Headers: {} 2024-02-01T10:22:17.750 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:17.750 DEBUG:boto:Port: 8000 2024-02-01T10:22:17.750 DEBUG:boto:Params: {} 2024-02-01T10:22:17.750 DEBUG:boto:Token: None 2024-02-01T10:22:17.751 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:17 GMT /vclfcn-17/ 2024-02-01T10:22:17.751 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FiVRU/RPLdd8789pldT3gwajYLs= 2024-02-01T10:22:17.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FiVRU/RPLdd8789pldT3gwajYLs=', 'Content-Length': '0'} 2024-02-01T10:22:17.755 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000722acf5ce87e86e-0065bb70d9-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:22:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:17.755 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:17.755 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:17.755 DEBUG:boto:path=/vclfcn-17/?versions&max-keys=999&prefix=obj-test-zone2 2024-02-01T10:22:17.756 DEBUG:boto:auth_path=/vclfcn-17/?versions&max-keys=999&prefix=obj-test-zone2 2024-02-01T10:22:17.756 DEBUG:boto:Method: GET 2024-02-01T10:22:17.756 DEBUG:boto:Path: /vclfcn-17/?versions&max-keys=999&prefix=obj-test-zone2 2024-02-01T10:22:17.756 DEBUG:boto:Data: 2024-02-01T10:22:17.756 DEBUG:boto:Headers: {} 2024-02-01T10:22:17.756 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:17.756 DEBUG:boto:Port: 8000 2024-02-01T10:22:17.756 DEBUG:boto:Params: {} 2024-02-01T10:22:17.756 DEBUG:boto:Token: None 2024-02-01T10:22:17.756 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:17 GMT /vclfcn-17/?versions 2024-02-01T10:22:17.757 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:pPueLLwGGUJDYWlL36XOnXwKxAg= 2024-02-01T10:22:17.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:pPueLLwGGUJDYWlL36XOnXwKxAg=', 'Content-Length': '0'} 2024-02-01T10:22:17.761 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000931fa63c39396571-0065bb70d9-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:17.761 DEBUG:boto:b'vclfcn-17obj-test-zone2999falseobj-test-zone2-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1true2024-02-01T10:20:50.624Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:17.762 DEBUG:boto:path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:17.762 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:17.762 DEBUG:boto:path=/vclfcn-17/obj-test-zone2?acl 2024-02-01T10:22:17.762 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2?acl 2024-02-01T10:22:17.762 DEBUG:boto:Method: PUT 2024-02-01T10:22:17.762 DEBUG:boto:Path: /vclfcn-17/obj-test-zone2?acl 2024-02-01T10:22:17.762 DEBUG:boto:Data: 2024-02-01T10:22:17.762 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-02-01T10:22:17.762 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:17.763 DEBUG:boto:Port: 8000 2024-02-01T10:22:17.763 DEBUG:boto:Params: {} 2024-02-01T10:22:17.763 DEBUG:boto:Token: None 2024-02-01T10:22:17.763 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:22:17 GMT x-amz-acl:public-read /vclfcn-17/obj-test-zone2?acl 2024-02-01T10:22:17.763 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GlUktf11LHZXxLcr66NZewSa+hk= 2024-02-01T10:22:17.763 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GlUktf11LHZXxLcr66NZewSa+hk=', 'Content-Length': '0'} 2024-02-01T10:22:17.776 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c3b64f6fdd5c5df7-0065bb70d9-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:22:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:17.776 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:17.776 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:17.776 DEBUG:boto:Method: HEAD 2024-02-01T10:22:17.776 DEBUG:boto:Path: /vclfcn-17/ 2024-02-01T10:22:17.776 DEBUG:boto:Data: 2024-02-01T10:22:17.776 DEBUG:boto:Headers: {} 2024-02-01T10:22:17.776 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:17.777 DEBUG:boto:Port: 8001 2024-02-01T10:22:17.777 DEBUG:boto:Params: {} 2024-02-01T10:22:17.777 DEBUG:boto:Token: None 2024-02-01T10:22:17.780 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:17 GMT /vclfcn-17/ 2024-02-01T10:22:17.781 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FiVRU/RPLdd8789pldT3gwajYLs= 2024-02-01T10:22:17.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FiVRU/RPLdd8789pldT3gwajYLs=', 'Content-Length': '0'} 2024-02-01T10:22:17.785 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001ec4e344a5366228-0065bb70d9-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:22:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:17.785 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:17.785 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:17.785 DEBUG:boto:path=/vclfcn-17/?versions&max-keys=999&prefix=obj-test-zone3 2024-02-01T10:22:17.785 DEBUG:boto:auth_path=/vclfcn-17/?versions&max-keys=999&prefix=obj-test-zone3 2024-02-01T10:22:17.785 DEBUG:boto:Method: GET 2024-02-01T10:22:17.786 DEBUG:boto:Path: /vclfcn-17/?versions&max-keys=999&prefix=obj-test-zone3 2024-02-01T10:22:17.786 DEBUG:boto:Data: 2024-02-01T10:22:17.786 DEBUG:boto:Headers: {} 2024-02-01T10:22:17.786 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:17.786 DEBUG:boto:Port: 8001 2024-02-01T10:22:17.786 DEBUG:boto:Params: {} 2024-02-01T10:22:17.786 DEBUG:boto:Token: None 2024-02-01T10:22:17.786 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:17 GMT /vclfcn-17/?versions 2024-02-01T10:22:17.786 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:pPueLLwGGUJDYWlL36XOnXwKxAg= 2024-02-01T10:22:17.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:pPueLLwGGUJDYWlL36XOnXwKxAg=', 'Content-Length': '0'} 2024-02-01T10:22:17.791 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b1522d024ed1f33-0065bb70d9-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:17.791 DEBUG:boto:b'vclfcn-17obj-test-zone3999falseobj-test-zone3YE736OwOOXbZJHJe7d8YvcNV1LxwcQbtrue2024-02-01T10:20:50.780Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:17.791 DEBUG:boto:path=/vclfcn-17/obj-test-zone3 2024-02-01T10:22:17.791 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3 2024-02-01T10:22:17.792 DEBUG:boto:path=/vclfcn-17/obj-test-zone3?acl 2024-02-01T10:22:17.792 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3?acl 2024-02-01T10:22:17.792 DEBUG:boto:Method: PUT 2024-02-01T10:22:17.792 DEBUG:boto:Path: /vclfcn-17/obj-test-zone3?acl 2024-02-01T10:22:17.792 DEBUG:boto:Data: 2024-02-01T10:22:17.792 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-02-01T10:22:17.792 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:17.792 DEBUG:boto:Port: 8001 2024-02-01T10:22:17.792 DEBUG:boto:Params: {} 2024-02-01T10:22:17.792 DEBUG:boto:Token: None 2024-02-01T10:22:17.793 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:22:17 GMT x-amz-acl:public-read /vclfcn-17/obj-test-zone3?acl 2024-02-01T10:22:17.793 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Nveog9RCYsrzv+9eKccJT0S9K1s= 2024-02-01T10:22:17.793 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Nveog9RCYsrzv+9eKccJT0S9K1s=', 'Content-Length': '0'} 2024-02-01T10:22:17.807 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000002e161a62a1e7096-0065bb70d9-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:22:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:17.807 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:22:17.807 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:22:17.807 DEBUG:boto:Method: HEAD 2024-02-01T10:22:17.807 DEBUG:boto:Path: /vclfcn-18/ 2024-02-01T10:22:17.807 DEBUG:boto:Data: 2024-02-01T10:22:17.807 DEBUG:boto:Headers: {} 2024-02-01T10:22:17.807 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:17.808 DEBUG:boto:Port: 8000 2024-02-01T10:22:17.808 DEBUG:boto:Params: {} 2024-02-01T10:22:17.808 DEBUG:boto:Token: None 2024-02-01T10:22:17.808 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:17 GMT /vclfcn-18/ 2024-02-01T10:22:17.808 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:bEnAwtfKakUK40mVYvaLoSYpLfs= 2024-02-01T10:22:17.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:bEnAwtfKakUK40mVYvaLoSYpLfs=', 'Content-Length': '0'} 2024-02-01T10:22:17.812 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000082809ffd0b0f0789-0065bb70d9-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:22:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:17.812 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:22:17.812 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:22:17.812 DEBUG:boto:path=/vclfcn-18/?versions&max-keys=999&prefix=obj-test-zone1 2024-02-01T10:22:17.812 DEBUG:boto:auth_path=/vclfcn-18/?versions&max-keys=999&prefix=obj-test-zone1 2024-02-01T10:22:17.813 DEBUG:boto:Method: GET 2024-02-01T10:22:17.813 DEBUG:boto:Path: /vclfcn-18/?versions&max-keys=999&prefix=obj-test-zone1 2024-02-01T10:22:17.813 DEBUG:boto:Data: 2024-02-01T10:22:17.813 DEBUG:boto:Headers: {} 2024-02-01T10:22:17.813 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:17.813 DEBUG:boto:Port: 8000 2024-02-01T10:22:17.813 DEBUG:boto:Params: {} 2024-02-01T10:22:17.813 DEBUG:boto:Token: None 2024-02-01T10:22:17.813 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:17 GMT /vclfcn-18/?versions 2024-02-01T10:22:17.814 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5ll/K3LSmngii6nLaifcwhzjclw= 2024-02-01T10:22:17.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5ll/K3LSmngii6nLaifcwhzjclw=', 'Content-Length': '0'} 2024-02-01T10:22:17.818 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f3ad52742164c9b-0065bb70d9-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:17.860 DEBUG:boto:b'vclfcn-18obj-test-zone1999falseobj-test-zone1H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcKtrue2024-02-01T10:20:50.942Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:17.861 DEBUG:boto:path=/vclfcn-18/obj-test-zone1 2024-02-01T10:22:17.861 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1 2024-02-01T10:22:17.861 DEBUG:boto:path=/vclfcn-18/obj-test-zone1?acl 2024-02-01T10:22:17.861 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1?acl 2024-02-01T10:22:17.861 DEBUG:boto:Method: PUT 2024-02-01T10:22:17.861 DEBUG:boto:Path: /vclfcn-18/obj-test-zone1?acl 2024-02-01T10:22:17.861 DEBUG:boto:Data: 2024-02-01T10:22:17.861 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-02-01T10:22:17.861 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:17.861 DEBUG:boto:Port: 8000 2024-02-01T10:22:17.862 DEBUG:boto:Params: {} 2024-02-01T10:22:17.862 DEBUG:boto:Token: None 2024-02-01T10:22:17.862 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:22:17 GMT x-amz-acl:public-read /vclfcn-18/obj-test-zone1?acl 2024-02-01T10:22:17.862 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:A6tjP6LOtVIh0+hBfL96Q21C3EU= 2024-02-01T10:22:17.862 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:A6tjP6LOtVIh0+hBfL96Q21C3EU=', 'Content-Length': '0'} 2024-02-01T10:22:17.875 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c542da4cbb0b9a70-0065bb70d9-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:22:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:17.875 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:22:17.876 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:22:17.876 DEBUG:boto:Method: HEAD 2024-02-01T10:22:17.876 DEBUG:boto:Path: /vclfcn-18/ 2024-02-01T10:22:17.876 DEBUG:boto:Data: 2024-02-01T10:22:17.876 DEBUG:boto:Headers: {} 2024-02-01T10:22:17.876 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:17.876 DEBUG:boto:Port: 8000 2024-02-01T10:22:17.876 DEBUG:boto:Params: {} 2024-02-01T10:22:17.876 DEBUG:boto:Token: None 2024-02-01T10:22:17.877 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:17 GMT /vclfcn-18/ 2024-02-01T10:22:17.877 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:bEnAwtfKakUK40mVYvaLoSYpLfs= 2024-02-01T10:22:17.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:bEnAwtfKakUK40mVYvaLoSYpLfs=', 'Content-Length': '0'} 2024-02-01T10:22:17.881 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000084eab54a86a0e18-0065bb70d9-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:22:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:17.881 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:22:17.881 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:22:17.881 DEBUG:boto:path=/vclfcn-18/?versions&max-keys=999&prefix=obj-test-zone2 2024-02-01T10:22:17.881 DEBUG:boto:auth_path=/vclfcn-18/?versions&max-keys=999&prefix=obj-test-zone2 2024-02-01T10:22:17.881 DEBUG:boto:Method: GET 2024-02-01T10:22:17.881 DEBUG:boto:Path: /vclfcn-18/?versions&max-keys=999&prefix=obj-test-zone2 2024-02-01T10:22:17.881 DEBUG:boto:Data: 2024-02-01T10:22:17.882 DEBUG:boto:Headers: {} 2024-02-01T10:22:17.882 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:17.882 DEBUG:boto:Port: 8000 2024-02-01T10:22:17.882 DEBUG:boto:Params: {} 2024-02-01T10:22:17.882 DEBUG:boto:Token: None 2024-02-01T10:22:17.882 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:17 GMT /vclfcn-18/?versions 2024-02-01T10:22:17.882 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5ll/K3LSmngii6nLaifcwhzjclw= 2024-02-01T10:22:17.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5ll/K3LSmngii6nLaifcwhzjclw=', 'Content-Length': '0'} 2024-02-01T10:22:17.886 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d1d81047b8ccf48c-0065bb70d9-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:17.886 DEBUG:boto:b'vclfcn-18obj-test-zone2999falseobj-test-zone2ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutDtrue2024-02-01T10:20:51.094Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:17.887 DEBUG:boto:path=/vclfcn-18/obj-test-zone2 2024-02-01T10:22:17.887 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2 2024-02-01T10:22:17.887 DEBUG:boto:path=/vclfcn-18/obj-test-zone2?acl 2024-02-01T10:22:17.887 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2?acl 2024-02-01T10:22:17.887 DEBUG:boto:Method: PUT 2024-02-01T10:22:17.887 DEBUG:boto:Path: /vclfcn-18/obj-test-zone2?acl 2024-02-01T10:22:17.887 DEBUG:boto:Data: 2024-02-01T10:22:17.887 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-02-01T10:22:17.887 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:17.887 DEBUG:boto:Port: 8000 2024-02-01T10:22:17.888 DEBUG:boto:Params: {} 2024-02-01T10:22:17.888 DEBUG:boto:Token: None 2024-02-01T10:22:17.888 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:22:17 GMT x-amz-acl:public-read /vclfcn-18/obj-test-zone2?acl 2024-02-01T10:22:17.888 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ut6AedmkijuWB/P6myxx2txHqyQ= 2024-02-01T10:22:17.888 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ut6AedmkijuWB/P6myxx2txHqyQ=', 'Content-Length': '0'} 2024-02-01T10:22:17.902 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c42ff2623ee7900d-0065bb70d9-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:22:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:17.902 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:22:17.902 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:22:17.902 DEBUG:boto:Method: HEAD 2024-02-01T10:22:17.902 DEBUG:boto:Path: /vclfcn-18/ 2024-02-01T10:22:17.903 DEBUG:boto:Data: 2024-02-01T10:22:17.903 DEBUG:boto:Headers: {} 2024-02-01T10:22:17.903 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:17.903 DEBUG:boto:Port: 8001 2024-02-01T10:22:17.903 DEBUG:boto:Params: {} 2024-02-01T10:22:17.903 DEBUG:boto:Token: None 2024-02-01T10:22:17.903 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:17 GMT /vclfcn-18/ 2024-02-01T10:22:17.903 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:bEnAwtfKakUK40mVYvaLoSYpLfs= 2024-02-01T10:22:17.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:bEnAwtfKakUK40mVYvaLoSYpLfs=', 'Content-Length': '0'} 2024-02-01T10:22:17.907 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000898d9d6bba7fdc35-0065bb70d9-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:22:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:17.908 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:22:17.908 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:22:17.908 DEBUG:boto:path=/vclfcn-18/?versions&max-keys=999&prefix=obj-test-zone3 2024-02-01T10:22:17.908 DEBUG:boto:auth_path=/vclfcn-18/?versions&max-keys=999&prefix=obj-test-zone3 2024-02-01T10:22:17.908 DEBUG:boto:Method: GET 2024-02-01T10:22:17.908 DEBUG:boto:Path: /vclfcn-18/?versions&max-keys=999&prefix=obj-test-zone3 2024-02-01T10:22:17.908 DEBUG:boto:Data: 2024-02-01T10:22:17.909 DEBUG:boto:Headers: {} 2024-02-01T10:22:17.909 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:17.909 DEBUG:boto:Port: 8001 2024-02-01T10:22:17.909 DEBUG:boto:Params: {} 2024-02-01T10:22:17.909 DEBUG:boto:Token: None 2024-02-01T10:22:17.909 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:17 GMT /vclfcn-18/?versions 2024-02-01T10:22:17.909 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5ll/K3LSmngii6nLaifcwhzjclw= 2024-02-01T10:22:17.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5ll/K3LSmngii6nLaifcwhzjclw=', 'Content-Length': '0'} 2024-02-01T10:22:17.913 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da01429f297850b6-0065bb70d9-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:17.914 DEBUG:boto:b'vclfcn-18obj-test-zone3999falseobj-test-zone36pAVLbw3HAwE0jxxbuxTIxwcA2p4q8Ntrue2024-02-01T10:20:51.248Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:17.914 DEBUG:boto:path=/vclfcn-18/obj-test-zone3 2024-02-01T10:22:17.914 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3 2024-02-01T10:22:17.914 DEBUG:boto:path=/vclfcn-18/obj-test-zone3?acl 2024-02-01T10:22:17.914 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3?acl 2024-02-01T10:22:17.914 DEBUG:boto:Method: PUT 2024-02-01T10:22:17.915 DEBUG:boto:Path: /vclfcn-18/obj-test-zone3?acl 2024-02-01T10:22:17.915 DEBUG:boto:Data: 2024-02-01T10:22:17.915 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-02-01T10:22:17.915 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:17.915 DEBUG:boto:Port: 8001 2024-02-01T10:22:17.915 DEBUG:boto:Params: {} 2024-02-01T10:22:17.915 DEBUG:boto:Token: None 2024-02-01T10:22:17.915 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:22:17 GMT x-amz-acl:public-read /vclfcn-18/obj-test-zone3?acl 2024-02-01T10:22:17.915 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Xju0Cy/IQ1vl56DBMQDY32rcjkc= 2024-02-01T10:22:17.915 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Xju0Cy/IQ1vl56DBMQDY32rcjkc=', 'Content-Length': '0'} 2024-02-01T10:22:17.928 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007761223a3699971b-0065bb70d9-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:22:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:17.928 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=vclfcn-16 2024-02-01T10:22:17.928 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:22:17.928 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:22:17.928 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket vclfcn-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-02-01T10:22:17.978 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:22:17.978 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:22:18.040 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:22:18.040+0000 7f45d023b980 1 waiting for incremental sync to catch up: 2024-02-01T10:22:18.041 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, 00000000010.30.6, , 00000000003.20.10, , 00000000008.29.10, , , , , ] 2024-02-01T10:22:18.041 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, 00000000011.33.6, , 00000000003.20.10, , 00000000008.29.10, , , , , ] 2024-02-01T10:22:23.043 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:22:23.044+0000 7f45d023b980 1 waiting for incremental sync to catch up: 2024-02-01T10:22:23.043 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, 00000000010.30.6, , 00000000003.20.10, , 00000000008.29.10, , , , , ] 2024-02-01T10:22:23.043 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, 00000000011.33.6, , 00000000003.20.10, , 00000000008.29.10, , , , , ] 2024-02-01T10:22:28.046 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:22:28.048+0000 7f45d023b980 1 waiting for incremental sync to catch up: 2024-02-01T10:22:28.046 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, 00000000010.30.6, , 00000000003.20.10, , 00000000008.29.10, , , , , ] 2024-02-01T10:22:28.046 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, 00000000011.33.6, , 00000000003.20.10, , 00000000008.29.10, , , , , ] 2024-02-01T10:22:33.048 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:22:33.048+0000 7f45d023b980 1 bucket sync caught up with source: 2024-02-01T10:22:33.048 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, 00000000011.33.6, , 00000000003.20.10, , 00000000008.29.10, , , , , ] 2024-02-01T10:22:33.048 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, 00000000011.33.6, , 00000000003.20.10, , 00000000008.29.10, , , , , ] 2024-02-01T10:22:33.048 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:22:33.048+0000 7f45d023b980 0 bucket checkpoint complete 2024-02-01T10:22:33.084 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=vclfcn-16 2024-02-01T10:22:33.084 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:22:33.085 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:22:33.085 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:22:33.136 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:22:33.136 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:22:33.193 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:22:33.196+0000 7f2353d64980 1 waiting for incremental sync to catch up: 2024-02-01T10:22:33.193 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, 00000000010.30.6, , 00000000003.20.10, , 00000000008.29.10, , , , , ] 2024-02-01T10:22:33.194 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, 00000000011.33.6, , 00000000003.20.10, , 00000000008.29.10, , , , , ] 2024-02-01T10:22:38.196 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:22:38.196+0000 7f2353d64980 1 bucket sync caught up with source: 2024-02-01T10:22:38.196 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, 00000000011.33.6, , 00000000003.20.10, , 00000000008.29.10, , , , , ] 2024-02-01T10:22:38.196 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, 00000000011.33.6, , 00000000003.20.10, , 00000000008.29.10, , , , , ] 2024-02-01T10:22:38.196 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:22:38.196+0000 7f2353d64980 0 bucket checkpoint complete 2024-02-01T10:22:38.238 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=vclfcn-16 2024-02-01T10:22:38.238 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:22:38.238 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:22:38.238 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:22:38.328 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:22:38.328 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:22:38.390 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:22:38.392+0000 7f3d8ccf1980 1 bucket sync caught up with source: 2024-02-01T10:22:38.391 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, 00000000003.22.10, , 00000000011.34.6, , 00000000007.31.10, , , , , ] 2024-02-01T10:22:38.391 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, 00000000003.22.10, , 00000000011.34.6, , 00000000007.31.10, , , , , ] 2024-02-01T10:22:38.391 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:22:38.392+0000 7f3d8ccf1980 0 bucket checkpoint complete 2024-02-01T10:22:38.413 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=vclfcn-16 2024-02-01T10:22:38.413 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:22:38.413 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:22:38.413 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:22:38.454 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:22:38.454 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:22:38.513 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:22:38.516+0000 7ff1eda66980 1 bucket sync caught up with source: 2024-02-01T10:22:38.513 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, 00000000003.22.10, , 00000000011.34.6, , 00000000007.31.10, , , , , ] 2024-02-01T10:22:38.513 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, 00000000003.22.10, , 00000000011.34.6, , 00000000007.31.10, , , , , ] 2024-02-01T10:22:38.513 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:22:38.516+0000 7ff1eda66980 0 bucket checkpoint complete 2024-02-01T10:22:38.537 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=vclfcn-16 2024-02-01T10:22:38.537 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:22:38.537 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:22:38.537 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:22:38.625 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:22:38.625 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:22:38.686 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:22:38.688+0000 7fcdbb6f7980 1 bucket sync caught up with source: 2024-02-01T10:22:38.686 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, 00000000003.50.10, , 00000000003.28.10, , 00000000015.62.6, , , , , ] 2024-02-01T10:22:38.686 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, 00000000003.50.10, , 00000000003.28.10, , 00000000015.62.6, , , , , ] 2024-02-01T10:22:38.687 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:22:38.688+0000 7fcdbb6f7980 0 bucket checkpoint complete 2024-02-01T10:22:38.710 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=vclfcn-16 2024-02-01T10:22:38.710 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:22:38.710 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:22:38.710 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket vclfcn-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-02-01T10:22:38.761 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:22:38.761 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:22:38.826 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:22:38.828+0000 7f7328c23980 1 waiting for incremental sync to catch up: 2024-02-01T10:22:38.827 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, 00000000003.50.10, , 00000000003.28.10, , 00000000014.59.6, , , , , ] 2024-02-01T10:22:38.827 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, 00000000003.50.10, , 00000000003.28.10, , 00000000015.62.6, , , , , ] 2024-02-01T10:22:43.829 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:22:43.832+0000 7f7328c23980 1 waiting for incremental sync to catch up: 2024-02-01T10:22:43.829 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, 00000000003.50.10, , 00000000003.28.10, , 00000000014.59.6, , , , , ] 2024-02-01T10:22:43.829 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, 00000000003.50.10, , 00000000003.28.10, , 00000000015.62.6, , , , , ] 2024-02-01T10:22:48.832 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:22:48.832+0000 7f7328c23980 1 waiting for incremental sync to catch up: 2024-02-01T10:22:48.833 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, 00000000003.50.10, , 00000000003.28.10, , 00000000014.59.6, , , , , ] 2024-02-01T10:22:48.833 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, 00000000003.50.10, , 00000000003.28.10, , 00000000015.62.6, , , , , ] 2024-02-01T10:22:53.834 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:22:53.836+0000 7f7328c23980 1 bucket sync caught up with source: 2024-02-01T10:22:53.834 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, 00000000003.50.10, , 00000000003.28.10, , 00000000015.62.6, , , , , ] 2024-02-01T10:22:53.834 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, 00000000003.50.10, , 00000000003.28.10, , 00000000015.62.6, , , , , ] 2024-02-01T10:22:53.834 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:22:53.836+0000 7f7328c23980 0 bucket checkpoint complete 2024-02-01T10:22:53.871 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-16 zones={test-zone2, test-zone1} 2024-02-01T10:22:53.871 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:22:53.871 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:22:53.872 DEBUG:boto:Method: HEAD 2024-02-01T10:22:53.872 DEBUG:boto:Path: /vclfcn-16/ 2024-02-01T10:22:53.872 DEBUG:boto:Data: 2024-02-01T10:22:53.872 DEBUG:boto:Headers: {} 2024-02-01T10:22:53.872 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:53.872 DEBUG:boto:Port: 8000 2024-02-01T10:22:53.872 DEBUG:boto:Params: {} 2024-02-01T10:22:53.873 DEBUG:boto:Token: None 2024-02-01T10:22:53.873 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:53 GMT /vclfcn-16/ 2024-02-01T10:22:53.873 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1Y2hRzqPcyztD0Mah2c5752mB8k= 2024-02-01T10:22:53.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1Y2hRzqPcyztD0Mah2c5752mB8k=', 'Content-Length': '0'} 2024-02-01T10:22:53.876 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ade427fb846c00c1-0065bb70fd-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:22:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:53.877 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:22:53.877 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:22:53.877 DEBUG:boto:Method: HEAD 2024-02-01T10:22:53.877 DEBUG:boto:Path: /vclfcn-16/ 2024-02-01T10:22:53.877 DEBUG:boto:Data: 2024-02-01T10:22:53.877 DEBUG:boto:Headers: {} 2024-02-01T10:22:53.877 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:53.877 DEBUG:boto:Port: 8000 2024-02-01T10:22:53.877 DEBUG:boto:Params: {} 2024-02-01T10:22:53.878 DEBUG:boto:Token: None 2024-02-01T10:22:53.878 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:53 GMT /vclfcn-16/ 2024-02-01T10:22:53.878 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1Y2hRzqPcyztD0Mah2c5752mB8k= 2024-02-01T10:22:53.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1Y2hRzqPcyztD0Mah2c5752mB8k=', 'Content-Length': '0'} 2024-02-01T10:22:53.882 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005368f40f513bd722-0065bb70fd-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:22:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:53.882 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:22:53.882 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:22:53.882 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:22:53.883 DEBUG:boto:path=/vclfcn-16/?versions&max-keys=999 2024-02-01T10:22:53.883 DEBUG:boto:auth_path=/vclfcn-16/?versions&max-keys=999 2024-02-01T10:22:53.883 DEBUG:boto:Method: GET 2024-02-01T10:22:53.883 DEBUG:boto:Path: /vclfcn-16/?versions&max-keys=999 2024-02-01T10:22:53.883 DEBUG:boto:Data: 2024-02-01T10:22:53.883 DEBUG:boto:Headers: {} 2024-02-01T10:22:53.883 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:53.883 DEBUG:boto:Port: 8000 2024-02-01T10:22:53.883 DEBUG:boto:Params: {} 2024-02-01T10:22:53.883 DEBUG:boto:Token: None 2024-02-01T10:22:53.884 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:53 GMT /vclfcn-16/?versions 2024-02-01T10:22:53.884 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ymqfSGk7jRVPBcI++VFWNceYjBU= 2024-02-01T10:22:53.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ymqfSGk7jRVPBcI++VFWNceYjBU=', 'Content-Length': '0'} 2024-02-01T10:22:53.887 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e1a1915fe61b9dad-0065bb70fd-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:53.887 DEBUG:boto:b'vclfcn-16999falsedummyrQRpBowIFAsvxVDn92H30shYnKoGt4ntrue2024-02-01T10:19:58.958Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1lS9-sMOvyR-VcD.47BkPLymGmIyfWmftrue2024-02-01T10:20:49.965Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2mE6urSxcrRA9wdyuToa41.w2aTia9p1true2024-02-01T10:20:50.122Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjPtrue2024-02-01T10:20:50.297Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:53.888 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-02-01T10:22:53.888 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-02-01T10:22:53.888 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-02-01T10:22:53.888 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-02-01T10:22:53.888 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:22:53.888 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:22:53.889 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:22:53.889 DEBUG:boto:path=/vclfcn-16/?versions&max-keys=999 2024-02-01T10:22:53.889 DEBUG:boto:auth_path=/vclfcn-16/?versions&max-keys=999 2024-02-01T10:22:53.889 DEBUG:boto:Method: GET 2024-02-01T10:22:53.889 DEBUG:boto:Path: /vclfcn-16/?versions&max-keys=999 2024-02-01T10:22:53.889 DEBUG:boto:Data: 2024-02-01T10:22:53.889 DEBUG:boto:Headers: {} 2024-02-01T10:22:53.889 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:53.889 DEBUG:boto:Port: 8000 2024-02-01T10:22:53.889 DEBUG:boto:Params: {} 2024-02-01T10:22:53.890 DEBUG:boto:Token: None 2024-02-01T10:22:53.890 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:53 GMT /vclfcn-16/?versions 2024-02-01T10:22:53.890 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ymqfSGk7jRVPBcI++VFWNceYjBU= 2024-02-01T10:22:53.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ymqfSGk7jRVPBcI++VFWNceYjBU=', 'Content-Length': '0'} 2024-02-01T10:22:53.894 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5689990d6d512b7-0065bb70fd-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:53.894 DEBUG:boto:b'vclfcn-16999falsedummyrQRpBowIFAsvxVDn92H30shYnKoGt4ntrue2024-02-01T10:19:58.958Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1lS9-sMOvyR-VcD.47BkPLymGmIyfWmftrue2024-02-01T10:20:49.965Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2mE6urSxcrRA9wdyuToa41.w2aTia9p1true2024-02-01T10:20:50.122Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjPtrue2024-02-01T10:20:50.297Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:53.895 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-02-01T10:22:53.895 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-02-01T10:22:53.895 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-02-01T10:22:53.896 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-02-01T10:22:53.896 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:22:53.896 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:22:53.896 DEBUG:boto:path=/vclfcn-16/?versions&max-keys=999 2024-02-01T10:22:53.896 DEBUG:boto:auth_path=/vclfcn-16/?versions&max-keys=999 2024-02-01T10:22:53.897 DEBUG:boto:Method: GET 2024-02-01T10:22:53.897 DEBUG:boto:Path: /vclfcn-16/?versions&max-keys=999 2024-02-01T10:22:53.897 DEBUG:boto:Data: 2024-02-01T10:22:53.897 DEBUG:boto:Headers: {} 2024-02-01T10:22:53.897 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:53.897 DEBUG:boto:Port: 8000 2024-02-01T10:22:53.897 DEBUG:boto:Params: {} 2024-02-01T10:22:53.898 DEBUG:boto:Token: None 2024-02-01T10:22:53.898 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:53 GMT /vclfcn-16/?versions 2024-02-01T10:22:53.898 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ymqfSGk7jRVPBcI++VFWNceYjBU= 2024-02-01T10:22:53.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ymqfSGk7jRVPBcI++VFWNceYjBU=', 'Content-Length': '0'} 2024-02-01T10:22:53.902 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3d20ef77863bfae-0065bb70fd-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:53.902 DEBUG:boto:b'vclfcn-16999falsedummyrQRpBowIFAsvxVDn92H30shYnKoGt4ntrue2024-02-01T10:19:58.958Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1lS9-sMOvyR-VcD.47BkPLymGmIyfWmftrue2024-02-01T10:20:49.965Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2mE6urSxcrRA9wdyuToa41.w2aTia9p1true2024-02-01T10:20:50.122Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjPtrue2024-02-01T10:20:50.297Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:53.903 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:22:53.904 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:22:53.904 DEBUG:boto:path=/vclfcn-16/?versions&max-keys=999 2024-02-01T10:22:53.904 DEBUG:boto:auth_path=/vclfcn-16/?versions&max-keys=999 2024-02-01T10:22:53.904 DEBUG:boto:Method: GET 2024-02-01T10:22:53.904 DEBUG:boto:Path: /vclfcn-16/?versions&max-keys=999 2024-02-01T10:22:53.905 DEBUG:boto:Data: 2024-02-01T10:22:53.905 DEBUG:boto:Headers: {} 2024-02-01T10:22:53.905 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:53.905 DEBUG:boto:Port: 8000 2024-02-01T10:22:53.905 DEBUG:boto:Params: {} 2024-02-01T10:22:53.905 DEBUG:boto:Token: None 2024-02-01T10:22:53.905 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:53 GMT /vclfcn-16/?versions 2024-02-01T10:22:53.906 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ymqfSGk7jRVPBcI++VFWNceYjBU= 2024-02-01T10:22:53.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ymqfSGk7jRVPBcI++VFWNceYjBU=', 'Content-Length': '0'} 2024-02-01T10:22:53.910 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000066529679d6e15cd1-0065bb70fd-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:53.910 DEBUG:boto:b'vclfcn-16999falsedummyrQRpBowIFAsvxVDn92H30shYnKoGt4ntrue2024-02-01T10:19:58.958Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1lS9-sMOvyR-VcD.47BkPLymGmIyfWmftrue2024-02-01T10:20:49.965Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2mE6urSxcrRA9wdyuToa41.w2aTia9p1true2024-02-01T10:20:50.122Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjPtrue2024-02-01T10:20:50.297Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:53.910 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-02-01T10:22:53.911 DEBUG:boto:path=/vclfcn-16/dummy 2024-02-01T10:22:53.911 DEBUG:boto:auth_path=/vclfcn-16/dummy 2024-02-01T10:22:53.911 DEBUG:boto:path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:53.911 DEBUG:boto:auth_path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:53.911 DEBUG:boto:Method: GET 2024-02-01T10:22:53.911 DEBUG:boto:Path: /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:53.911 DEBUG:boto:Data: 2024-02-01T10:22:53.911 DEBUG:boto:Headers: {} 2024-02-01T10:22:53.911 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:53.911 DEBUG:boto:Port: 8000 2024-02-01T10:22:53.912 DEBUG:boto:Params: {} 2024-02-01T10:22:53.912 DEBUG:boto:Token: None 2024-02-01T10:22:53.912 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:53 GMT /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:53.912 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:i3cRAbmoAwS9T3ZlyzNm8+zHw20= 2024-02-01T10:22:53.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:i3cRAbmoAwS9T3ZlyzNm8+zHw20=', 'Content-Length': '0'} 2024-02-01T10:22:53.915 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:19:58 GMT'), ('x-amz-version-id', 'rQRpBowIFAsvxVDn92H30shYnKoGt4n'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b1416c1ab73d922a-0065bb70fd-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:53.915 DEBUG:boto:path=/vclfcn-16/dummy 2024-02-01T10:22:53.915 DEBUG:boto:auth_path=/vclfcn-16/dummy 2024-02-01T10:22:53.915 DEBUG:boto:path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:53.915 DEBUG:boto:auth_path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:53.916 DEBUG:boto:Method: GET 2024-02-01T10:22:53.916 DEBUG:boto:Path: /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:53.916 DEBUG:boto:Data: 2024-02-01T10:22:53.916 DEBUG:boto:Headers: {} 2024-02-01T10:22:53.916 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:53.916 DEBUG:boto:Port: 8000 2024-02-01T10:22:53.916 DEBUG:boto:Params: {} 2024-02-01T10:22:53.916 DEBUG:boto:Token: None 2024-02-01T10:22:53.916 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:53 GMT /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:53.917 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:i3cRAbmoAwS9T3ZlyzNm8+zHw20= 2024-02-01T10:22:53.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:i3cRAbmoAwS9T3ZlyzNm8+zHw20=', 'Content-Length': '0'} 2024-02-01T10:22:53.919 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:19:58 GMT'), ('x-amz-version-id', 'rQRpBowIFAsvxVDn92H30shYnKoGt4n'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000012ccf4d05085cab4-0065bb70fd-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:53.920 DEBUG:boto:path=/vclfcn-16/dummy 2024-02-01T10:22:53.920 DEBUG:boto:auth_path=/vclfcn-16/dummy 2024-02-01T10:22:53.920 DEBUG:boto:path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:53.920 DEBUG:boto:auth_path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:53.920 DEBUG:boto:Method: HEAD 2024-02-01T10:22:53.920 DEBUG:boto:Path: /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:53.920 DEBUG:boto:Data: 2024-02-01T10:22:53.920 DEBUG:boto:Headers: {} 2024-02-01T10:22:53.920 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:53.921 DEBUG:boto:Port: 8000 2024-02-01T10:22:53.921 DEBUG:boto:Params: {} 2024-02-01T10:22:53.921 DEBUG:boto:Token: None 2024-02-01T10:22:53.921 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:53 GMT /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:53.921 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/dISLxTN1GG0RA47lfFfy2Ba4yo= 2024-02-01T10:22:53.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/dISLxTN1GG0RA47lfFfy2Ba4yo=', 'Content-Length': '0'} 2024-02-01T10:22:53.924 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:19:58 GMT'), ('x-amz-version-id', 'rQRpBowIFAsvxVDn92H30shYnKoGt4n'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a990ae470d0370a4-0065bb70fd-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:53.924 DEBUG:boto:path=/vclfcn-16/dummy 2024-02-01T10:22:53.925 DEBUG:boto:auth_path=/vclfcn-16/dummy 2024-02-01T10:22:53.925 DEBUG:boto:path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:53.925 DEBUG:boto:auth_path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:53.925 DEBUG:boto:Method: HEAD 2024-02-01T10:22:53.925 DEBUG:boto:Path: /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:53.926 DEBUG:boto:Data: 2024-02-01T10:22:53.926 DEBUG:boto:Headers: {} 2024-02-01T10:22:53.926 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:53.926 DEBUG:boto:Port: 8000 2024-02-01T10:22:53.926 DEBUG:boto:Params: {} 2024-02-01T10:22:53.927 DEBUG:boto:Token: None 2024-02-01T10:22:53.927 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:53 GMT /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:53.927 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/dISLxTN1GG0RA47lfFfy2Ba4yo= 2024-02-01T10:22:53.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/dISLxTN1GG0RA47lfFfy2Ba4yo=', 'Content-Length': '0'} 2024-02-01T10:22:53.930 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:19:58 GMT'), ('x-amz-version-id', 'rQRpBowIFAsvxVDn92H30shYnKoGt4n'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a626488f1e34ddf3-0065bb70fd-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:53.931 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-02-01T10:22:53.931 DEBUG:boto:path=/vclfcn-16/dummy 2024-02-01T10:22:53.931 DEBUG:boto:auth_path=/vclfcn-16/dummy 2024-02-01T10:22:53.931 DEBUG:boto:path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:53.931 DEBUG:boto:auth_path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:53.931 DEBUG:boto:Method: GET 2024-02-01T10:22:53.931 DEBUG:boto:Path: /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:53.932 DEBUG:boto:Data: 2024-02-01T10:22:53.932 DEBUG:boto:Headers: {} 2024-02-01T10:22:53.932 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:53.932 DEBUG:boto:Port: 8000 2024-02-01T10:22:53.932 DEBUG:boto:Params: {} 2024-02-01T10:22:53.933 DEBUG:boto:Token: None 2024-02-01T10:22:53.933 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:53 GMT /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:53.933 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:i3cRAbmoAwS9T3ZlyzNm8+zHw20= 2024-02-01T10:22:53.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:i3cRAbmoAwS9T3ZlyzNm8+zHw20=', 'Content-Length': '0'} 2024-02-01T10:22:53.936 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:19:58 GMT'), ('x-amz-version-id', 'rQRpBowIFAsvxVDn92H30shYnKoGt4n'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004cbf512515d4b903-0065bb70fd-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:53.936 DEBUG:boto:path=/vclfcn-16/dummy 2024-02-01T10:22:53.936 DEBUG:boto:auth_path=/vclfcn-16/dummy 2024-02-01T10:22:53.936 DEBUG:boto:path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:53.936 DEBUG:boto:auth_path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:53.936 DEBUG:boto:Method: GET 2024-02-01T10:22:53.936 DEBUG:boto:Path: /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:53.937 DEBUG:boto:Data: 2024-02-01T10:22:53.937 DEBUG:boto:Headers: {} 2024-02-01T10:22:53.937 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:53.937 DEBUG:boto:Port: 8000 2024-02-01T10:22:53.937 DEBUG:boto:Params: {} 2024-02-01T10:22:53.937 DEBUG:boto:Token: None 2024-02-01T10:22:53.938 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:53 GMT /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:53.938 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:i3cRAbmoAwS9T3ZlyzNm8+zHw20= 2024-02-01T10:22:53.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:i3cRAbmoAwS9T3ZlyzNm8+zHw20=', 'Content-Length': '0'} 2024-02-01T10:22:53.941 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:19:58 GMT'), ('x-amz-version-id', 'rQRpBowIFAsvxVDn92H30shYnKoGt4n'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000013981b8d00117381-0065bb70fd-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:53.941 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:22:53.941 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:22:53.941 DEBUG:boto:path=/vclfcn-16/?prefix=dummy 2024-02-01T10:22:53.941 DEBUG:boto:auth_path=/vclfcn-16/?prefix=dummy 2024-02-01T10:22:53.941 DEBUG:boto:Method: GET 2024-02-01T10:22:53.942 DEBUG:boto:Path: /vclfcn-16/?prefix=dummy 2024-02-01T10:22:53.942 DEBUG:boto:Data: 2024-02-01T10:22:53.942 DEBUG:boto:Headers: {} 2024-02-01T10:22:53.942 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:53.942 DEBUG:boto:Port: 8000 2024-02-01T10:22:53.942 DEBUG:boto:Params: {} 2024-02-01T10:22:53.942 DEBUG:boto:Token: None 2024-02-01T10:22:53.942 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:53 GMT /vclfcn-16/ 2024-02-01T10:22:53.942 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:JD3p8Z4K9CkjjNBr0GZnfCJzi1g= 2024-02-01T10:22:53.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:JD3p8Z4K9CkjjNBr0GZnfCJzi1g=', 'Content-Length': '0'} 2024-02-01T10:22:53.947 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054bbcf379954aa0d-0065bb70fd-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:53.992 DEBUG:boto:b'vclfcn-16dummy1000falsedummy2024-02-01T10:19:58.958Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:53.993 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:22:53.993 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:22:53.993 DEBUG:boto:path=/vclfcn-16/?prefix=dummy 2024-02-01T10:22:53.993 DEBUG:boto:auth_path=/vclfcn-16/?prefix=dummy 2024-02-01T10:22:53.993 DEBUG:boto:Method: GET 2024-02-01T10:22:53.993 DEBUG:boto:Path: /vclfcn-16/?prefix=dummy 2024-02-01T10:22:53.993 DEBUG:boto:Data: 2024-02-01T10:22:53.993 DEBUG:boto:Headers: {} 2024-02-01T10:22:53.993 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:53.993 DEBUG:boto:Port: 8000 2024-02-01T10:22:53.994 DEBUG:boto:Params: {} 2024-02-01T10:22:53.994 DEBUG:boto:Token: None 2024-02-01T10:22:53.994 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:53 GMT /vclfcn-16/ 2024-02-01T10:22:53.994 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:JD3p8Z4K9CkjjNBr0GZnfCJzi1g= 2024-02-01T10:22:53.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:JD3p8Z4K9CkjjNBr0GZnfCJzi1g=', 'Content-Length': '0'} 2024-02-01T10:22:53.998 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b2051bca12c1794-0065bb70fe-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:53.998 DEBUG:boto:b'vclfcn-16dummy1000falsedummy2024-02-01T10:19:58.958Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:53.999 DEBUG:boto:path=/vclfcn-16/dummy 2024-02-01T10:22:53.999 DEBUG:boto:auth_path=/vclfcn-16/dummy 2024-02-01T10:22:53.999 DEBUG:boto:Method: HEAD 2024-02-01T10:22:53.999 DEBUG:boto:Path: /vclfcn-16/dummy 2024-02-01T10:22:53.999 DEBUG:boto:Data: 2024-02-01T10:22:53.999 DEBUG:boto:Headers: {} 2024-02-01T10:22:53.999 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:53.999 DEBUG:boto:Port: 8000 2024-02-01T10:22:53.999 DEBUG:boto:Params: {} 2024-02-01T10:22:54.000 DEBUG:boto:Token: None 2024-02-01T10:22:54.000 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:54 GMT /vclfcn-16/dummy 2024-02-01T10:22:54.000 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:PIjtCXZq6sqF5Atj3d51va+1+AY= 2024-02-01T10:22:54.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:PIjtCXZq6sqF5Atj3d51va+1+AY=', 'Content-Length': '0'} 2024-02-01T10:22:54.004 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:19:58 GMT'), ('x-amz-version-id', 'rQRpBowIFAsvxVDn92H30shYnKoGt4n'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cf77dbc14e1cefbb-0065bb70fe-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:54.004 DEBUG:boto:path=/vclfcn-16/dummy 2024-02-01T10:22:54.004 DEBUG:boto:auth_path=/vclfcn-16/dummy 2024-02-01T10:22:54.004 DEBUG:boto:Method: HEAD 2024-02-01T10:22:54.005 DEBUG:boto:Path: /vclfcn-16/dummy 2024-02-01T10:22:54.005 DEBUG:boto:Data: 2024-02-01T10:22:54.005 DEBUG:boto:Headers: {} 2024-02-01T10:22:54.005 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:54.005 DEBUG:boto:Port: 8000 2024-02-01T10:22:54.005 DEBUG:boto:Params: {} 2024-02-01T10:22:54.005 DEBUG:boto:Token: None 2024-02-01T10:22:54.005 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:54 GMT /vclfcn-16/dummy 2024-02-01T10:22:54.005 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:PIjtCXZq6sqF5Atj3d51va+1+AY= 2024-02-01T10:22:54.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:PIjtCXZq6sqF5Atj3d51va+1+AY=', 'Content-Length': '0'} 2024-02-01T10:22:54.009 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:19:58 GMT'), ('x-amz-version-id', 'rQRpBowIFAsvxVDn92H30shYnKoGt4n'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f238d5ca99d836a0-0065bb70fe-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:54.009 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-02-01T10:22:54.010 DEBUG:boto:path=/vclfcn-16/dummy 2024-02-01T10:22:54.010 DEBUG:boto:auth_path=/vclfcn-16/dummy 2024-02-01T10:22:54.010 DEBUG:boto:path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:54.010 DEBUG:boto:auth_path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:54.010 DEBUG:boto:Method: GET 2024-02-01T10:22:54.010 DEBUG:boto:Path: /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:54.010 DEBUG:boto:Data: 2024-02-01T10:22:54.010 DEBUG:boto:Headers: {} 2024-02-01T10:22:54.010 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:54.010 DEBUG:boto:Port: 8000 2024-02-01T10:22:54.010 DEBUG:boto:Params: {} 2024-02-01T10:22:54.011 DEBUG:boto:Token: None 2024-02-01T10:22:54.011 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:54 GMT /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:54.011 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vAGH10wmnzs67app0DcqSL6flkM= 2024-02-01T10:22:54.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vAGH10wmnzs67app0DcqSL6flkM=', 'Content-Length': '0'} 2024-02-01T10:22:54.014 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:19:58 GMT'), ('x-amz-version-id', 'rQRpBowIFAsvxVDn92H30shYnKoGt4n'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000dbc22b75e2616403-0065bb70fe-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:54.015 DEBUG:boto:path=/vclfcn-16/dummy 2024-02-01T10:22:54.015 DEBUG:boto:auth_path=/vclfcn-16/dummy 2024-02-01T10:22:54.015 DEBUG:boto:path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:54.015 DEBUG:boto:auth_path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:54.015 DEBUG:boto:Method: GET 2024-02-01T10:22:54.016 DEBUG:boto:Path: /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:54.016 DEBUG:boto:Data: 2024-02-01T10:22:54.016 DEBUG:boto:Headers: {} 2024-02-01T10:22:54.016 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:54.016 DEBUG:boto:Port: 8000 2024-02-01T10:22:54.017 DEBUG:boto:Params: {} 2024-02-01T10:22:54.017 DEBUG:boto:Token: None 2024-02-01T10:22:54.017 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:54 GMT /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:54.017 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vAGH10wmnzs67app0DcqSL6flkM= 2024-02-01T10:22:54.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vAGH10wmnzs67app0DcqSL6flkM=', 'Content-Length': '0'} 2024-02-01T10:22:54.021 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:19:58 GMT'), ('x-amz-version-id', 'rQRpBowIFAsvxVDn92H30shYnKoGt4n'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000f641362a8d95368-0065bb70fe-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:54.021 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:22:54.022 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:22:54.022 DEBUG:boto:path=/vclfcn-16/?prefix=dummy 2024-02-01T10:22:54.022 DEBUG:boto:auth_path=/vclfcn-16/?prefix=dummy 2024-02-01T10:22:54.022 DEBUG:boto:Method: GET 2024-02-01T10:22:54.022 DEBUG:boto:Path: /vclfcn-16/?prefix=dummy 2024-02-01T10:22:54.023 DEBUG:boto:Data: 2024-02-01T10:22:54.023 DEBUG:boto:Headers: {} 2024-02-01T10:22:54.023 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:54.023 DEBUG:boto:Port: 8000 2024-02-01T10:22:54.023 DEBUG:boto:Params: {} 2024-02-01T10:22:54.024 DEBUG:boto:Token: None 2024-02-01T10:22:54.024 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:54 GMT /vclfcn-16/ 2024-02-01T10:22:54.024 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:eqMQO31DWncWd0loSRCfyuBt6L4= 2024-02-01T10:22:54.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:eqMQO31DWncWd0loSRCfyuBt6L4=', 'Content-Length': '0'} 2024-02-01T10:22:54.028 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004aa6154110e3baf4-0065bb70fe-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:54.076 DEBUG:boto:b'vclfcn-16dummy1000falsedummy2024-02-01T10:19:58.958Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:54.077 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:22:54.077 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:22:54.077 DEBUG:boto:path=/vclfcn-16/?prefix=dummy 2024-02-01T10:22:54.078 DEBUG:boto:auth_path=/vclfcn-16/?prefix=dummy 2024-02-01T10:22:54.078 DEBUG:boto:Method: GET 2024-02-01T10:22:54.078 DEBUG:boto:Path: /vclfcn-16/?prefix=dummy 2024-02-01T10:22:54.078 DEBUG:boto:Data: 2024-02-01T10:22:54.078 DEBUG:boto:Headers: {} 2024-02-01T10:22:54.079 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:54.079 DEBUG:boto:Port: 8000 2024-02-01T10:22:54.079 DEBUG:boto:Params: {} 2024-02-01T10:22:54.079 DEBUG:boto:Token: None 2024-02-01T10:22:54.080 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:54 GMT /vclfcn-16/ 2024-02-01T10:22:54.080 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:eqMQO31DWncWd0loSRCfyuBt6L4= 2024-02-01T10:22:54.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:eqMQO31DWncWd0loSRCfyuBt6L4=', 'Content-Length': '0'} 2024-02-01T10:22:54.085 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000053b4807e0feefae0-0065bb70fe-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:54.085 DEBUG:boto:b'vclfcn-16dummy1000falsedummy2024-02-01T10:19:58.958Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:54.086 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-02-01T10:22:54.086 DEBUG:boto:path=/vclfcn-16/obj-test-zone1 2024-02-01T10:22:54.086 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1 2024-02-01T10:22:54.086 DEBUG:boto:path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:54.087 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:54.087 DEBUG:boto:Method: GET 2024-02-01T10:22:54.087 DEBUG:boto:Path: /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:54.087 DEBUG:boto:Data: 2024-02-01T10:22:54.087 DEBUG:boto:Headers: {} 2024-02-01T10:22:54.088 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:54.088 DEBUG:boto:Port: 8000 2024-02-01T10:22:54.088 DEBUG:boto:Params: {} 2024-02-01T10:22:54.088 DEBUG:boto:Token: None 2024-02-01T10:22:54.089 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:54 GMT /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:54.089 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:OJfbr5H5QvGd3Rpch8LF/LVD0CY= 2024-02-01T10:22:54.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:OJfbr5H5QvGd3Rpch8LF/LVD0CY=', 'Content-Length': '0'} 2024-02-01T10:22:54.092 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:49 GMT'), ('x-amz-version-id', 'lS9-sMOvyR-VcD.47BkPLymGmIyfWmf'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000021d08870fc19e9a5-0065bb70fe-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:54.093 DEBUG:boto:path=/vclfcn-16/obj-test-zone1 2024-02-01T10:22:54.093 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1 2024-02-01T10:22:54.093 DEBUG:boto:path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:54.094 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:54.094 DEBUG:boto:Method: GET 2024-02-01T10:22:54.094 DEBUG:boto:Path: /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:54.094 DEBUG:boto:Data: 2024-02-01T10:22:54.094 DEBUG:boto:Headers: {} 2024-02-01T10:22:54.094 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:54.095 DEBUG:boto:Port: 8000 2024-02-01T10:22:54.095 DEBUG:boto:Params: {} 2024-02-01T10:22:54.095 DEBUG:boto:Token: None 2024-02-01T10:22:54.095 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:54 GMT /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:54.096 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:OJfbr5H5QvGd3Rpch8LF/LVD0CY= 2024-02-01T10:22:54.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:OJfbr5H5QvGd3Rpch8LF/LVD0CY=', 'Content-Length': '0'} 2024-02-01T10:22:54.099 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:49 GMT'), ('x-amz-version-id', 'lS9-sMOvyR-VcD.47BkPLymGmIyfWmf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bd0f3db7f2ffce4d-0065bb70fe-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:54.140 DEBUG:boto:path=/vclfcn-16/obj-test-zone1 2024-02-01T10:22:54.140 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1 2024-02-01T10:22:54.141 DEBUG:boto:path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:54.141 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:54.141 DEBUG:boto:Method: HEAD 2024-02-01T10:22:54.141 DEBUG:boto:Path: /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:54.141 DEBUG:boto:Data: 2024-02-01T10:22:54.141 DEBUG:boto:Headers: {} 2024-02-01T10:22:54.141 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:54.141 DEBUG:boto:Port: 8000 2024-02-01T10:22:54.142 DEBUG:boto:Params: {} 2024-02-01T10:22:54.142 DEBUG:boto:Token: None 2024-02-01T10:22:54.142 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:54 GMT /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:54.142 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:++7jWJv8nsqDAfkF6vXGf2SnsXE= 2024-02-01T10:22:54.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:++7jWJv8nsqDAfkF6vXGf2SnsXE=', 'Content-Length': '0'} 2024-02-01T10:22:54.145 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:49 GMT'), ('x-amz-version-id', 'lS9-sMOvyR-VcD.47BkPLymGmIyfWmf'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000825d7b7085329148-0065bb70fe-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:54.145 DEBUG:boto:path=/vclfcn-16/obj-test-zone1 2024-02-01T10:22:54.146 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1 2024-02-01T10:22:54.146 DEBUG:boto:path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:54.146 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:54.146 DEBUG:boto:Method: HEAD 2024-02-01T10:22:54.146 DEBUG:boto:Path: /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:54.146 DEBUG:boto:Data: 2024-02-01T10:22:54.146 DEBUG:boto:Headers: {} 2024-02-01T10:22:54.146 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:54.146 DEBUG:boto:Port: 8000 2024-02-01T10:22:54.146 DEBUG:boto:Params: {} 2024-02-01T10:22:54.146 DEBUG:boto:Token: None 2024-02-01T10:22:54.147 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:54 GMT /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:54.147 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:++7jWJv8nsqDAfkF6vXGf2SnsXE= 2024-02-01T10:22:54.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:++7jWJv8nsqDAfkF6vXGf2SnsXE=', 'Content-Length': '0'} 2024-02-01T10:22:54.150 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:49 GMT'), ('x-amz-version-id', 'lS9-sMOvyR-VcD.47BkPLymGmIyfWmf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009a6ccd4fd085b311-0065bb70fe-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:54.150 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-02-01T10:22:54.150 DEBUG:boto:path=/vclfcn-16/obj-test-zone1 2024-02-01T10:22:54.150 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1 2024-02-01T10:22:54.150 DEBUG:boto:path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:54.150 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:54.151 DEBUG:boto:Method: GET 2024-02-01T10:22:54.151 DEBUG:boto:Path: /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:54.151 DEBUG:boto:Data: 2024-02-01T10:22:54.151 DEBUG:boto:Headers: {} 2024-02-01T10:22:54.151 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:54.151 DEBUG:boto:Port: 8000 2024-02-01T10:22:54.151 DEBUG:boto:Params: {} 2024-02-01T10:22:54.151 DEBUG:boto:Token: None 2024-02-01T10:22:54.151 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:54 GMT /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:54.152 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:OJfbr5H5QvGd3Rpch8LF/LVD0CY= 2024-02-01T10:22:54.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:OJfbr5H5QvGd3Rpch8LF/LVD0CY=', 'Content-Length': '0'} 2024-02-01T10:22:54.154 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:49 GMT'), ('x-amz-version-id', 'lS9-sMOvyR-VcD.47BkPLymGmIyfWmf'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001a990f520315c0a2-0065bb70fe-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:54.200 DEBUG:boto:path=/vclfcn-16/obj-test-zone1 2024-02-01T10:22:54.201 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1 2024-02-01T10:22:54.201 DEBUG:boto:path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:54.201 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:54.201 DEBUG:boto:Method: GET 2024-02-01T10:22:54.201 DEBUG:boto:Path: /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:54.201 DEBUG:boto:Data: 2024-02-01T10:22:54.201 DEBUG:boto:Headers: {} 2024-02-01T10:22:54.201 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:54.201 DEBUG:boto:Port: 8000 2024-02-01T10:22:54.201 DEBUG:boto:Params: {} 2024-02-01T10:22:54.202 DEBUG:boto:Token: None 2024-02-01T10:22:54.202 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:54 GMT /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:54.202 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:OJfbr5H5QvGd3Rpch8LF/LVD0CY= 2024-02-01T10:22:54.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:OJfbr5H5QvGd3Rpch8LF/LVD0CY=', 'Content-Length': '0'} 2024-02-01T10:22:54.205 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:49 GMT'), ('x-amz-version-id', 'lS9-sMOvyR-VcD.47BkPLymGmIyfWmf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006e788b7f9e3c4ae9-0065bb70fe-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:54.205 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:22:54.205 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:22:54.206 DEBUG:boto:path=/vclfcn-16/?prefix=obj-test-zone1 2024-02-01T10:22:54.206 DEBUG:boto:auth_path=/vclfcn-16/?prefix=obj-test-zone1 2024-02-01T10:22:54.206 DEBUG:boto:Method: GET 2024-02-01T10:22:54.206 DEBUG:boto:Path: /vclfcn-16/?prefix=obj-test-zone1 2024-02-01T10:22:54.206 DEBUG:boto:Data: 2024-02-01T10:22:54.206 DEBUG:boto:Headers: {} 2024-02-01T10:22:54.206 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:54.206 DEBUG:boto:Port: 8000 2024-02-01T10:22:54.206 DEBUG:boto:Params: {} 2024-02-01T10:22:54.206 DEBUG:boto:Token: None 2024-02-01T10:22:54.207 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:54 GMT /vclfcn-16/ 2024-02-01T10:22:54.207 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:eqMQO31DWncWd0loSRCfyuBt6L4= 2024-02-01T10:22:54.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:eqMQO31DWncWd0loSRCfyuBt6L4=', 'Content-Length': '0'} 2024-02-01T10:22:54.210 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c8eb13645ee36721-0065bb70fe-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:54.256 DEBUG:boto:b'vclfcn-16obj-test-zone11000falseobj-test-zone12024-02-01T10:20:49.965Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:54.257 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:22:54.257 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:22:54.257 DEBUG:boto:path=/vclfcn-16/?prefix=obj-test-zone1 2024-02-01T10:22:54.257 DEBUG:boto:auth_path=/vclfcn-16/?prefix=obj-test-zone1 2024-02-01T10:22:54.257 DEBUG:boto:Method: GET 2024-02-01T10:22:54.257 DEBUG:boto:Path: /vclfcn-16/?prefix=obj-test-zone1 2024-02-01T10:22:54.257 DEBUG:boto:Data: 2024-02-01T10:22:54.257 DEBUG:boto:Headers: {} 2024-02-01T10:22:54.257 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:54.258 DEBUG:boto:Port: 8000 2024-02-01T10:22:54.258 DEBUG:boto:Params: {} 2024-02-01T10:22:54.258 DEBUG:boto:Token: None 2024-02-01T10:22:54.258 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:54 GMT /vclfcn-16/ 2024-02-01T10:22:54.258 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:eqMQO31DWncWd0loSRCfyuBt6L4= 2024-02-01T10:22:54.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:eqMQO31DWncWd0loSRCfyuBt6L4=', 'Content-Length': '0'} 2024-02-01T10:22:54.262 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b75c89fb5b84941-0065bb70fe-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:54.262 DEBUG:boto:b'vclfcn-16obj-test-zone11000falseobj-test-zone12024-02-01T10:20:49.965Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:54.263 DEBUG:boto:path=/vclfcn-16/obj-test-zone1 2024-02-01T10:22:54.263 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1 2024-02-01T10:22:54.263 DEBUG:boto:Method: HEAD 2024-02-01T10:22:54.263 DEBUG:boto:Path: /vclfcn-16/obj-test-zone1 2024-02-01T10:22:54.263 DEBUG:boto:Data: 2024-02-01T10:22:54.263 DEBUG:boto:Headers: {} 2024-02-01T10:22:54.263 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:54.264 DEBUG:boto:Port: 8000 2024-02-01T10:22:54.264 DEBUG:boto:Params: {} 2024-02-01T10:22:54.264 DEBUG:boto:Token: None 2024-02-01T10:22:54.264 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:54 GMT /vclfcn-16/obj-test-zone1 2024-02-01T10:22:54.264 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ku6iCxj2RPAvWcFUEWyTLy6F23M= 2024-02-01T10:22:54.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ku6iCxj2RPAvWcFUEWyTLy6F23M=', 'Content-Length': '0'} 2024-02-01T10:22:54.268 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:49 GMT'), ('x-amz-version-id', 'lS9-sMOvyR-VcD.47BkPLymGmIyfWmf'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c0074afcbd7e37f2-0065bb70fe-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:54.268 DEBUG:boto:path=/vclfcn-16/obj-test-zone1 2024-02-01T10:22:54.269 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1 2024-02-01T10:22:54.269 DEBUG:boto:Method: HEAD 2024-02-01T10:22:54.269 DEBUG:boto:Path: /vclfcn-16/obj-test-zone1 2024-02-01T10:22:54.269 DEBUG:boto:Data: 2024-02-01T10:22:54.269 DEBUG:boto:Headers: {} 2024-02-01T10:22:54.269 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:54.269 DEBUG:boto:Port: 8000 2024-02-01T10:22:54.269 DEBUG:boto:Params: {} 2024-02-01T10:22:54.269 DEBUG:boto:Token: None 2024-02-01T10:22:54.269 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:54 GMT /vclfcn-16/obj-test-zone1 2024-02-01T10:22:54.270 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ku6iCxj2RPAvWcFUEWyTLy6F23M= 2024-02-01T10:22:54.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ku6iCxj2RPAvWcFUEWyTLy6F23M=', 'Content-Length': '0'} 2024-02-01T10:22:54.273 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:49 GMT'), ('x-amz-version-id', 'lS9-sMOvyR-VcD.47BkPLymGmIyfWmf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f21aaeaca31c8dd5-0065bb70fe-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:54.273 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-02-01T10:22:54.274 DEBUG:boto:path=/vclfcn-16/obj-test-zone1 2024-02-01T10:22:54.274 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1 2024-02-01T10:22:54.274 DEBUG:boto:path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:54.274 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:54.274 DEBUG:boto:Method: GET 2024-02-01T10:22:54.274 DEBUG:boto:Path: /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:54.274 DEBUG:boto:Data: 2024-02-01T10:22:54.274 DEBUG:boto:Headers: {} 2024-02-01T10:22:54.275 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:54.275 DEBUG:boto:Port: 8000 2024-02-01T10:22:54.275 DEBUG:boto:Params: {} 2024-02-01T10:22:54.275 DEBUG:boto:Token: None 2024-02-01T10:22:54.275 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:54 GMT /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:54.276 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:OJfbr5H5QvGd3Rpch8LF/LVD0CY= 2024-02-01T10:22:54.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:OJfbr5H5QvGd3Rpch8LF/LVD0CY=', 'Content-Length': '0'} 2024-02-01T10:22:54.279 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:49 GMT'), ('x-amz-version-id', 'lS9-sMOvyR-VcD.47BkPLymGmIyfWmf'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000044b1985abc4da6b7-0065bb70fe-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:54.324 DEBUG:boto:path=/vclfcn-16/obj-test-zone1 2024-02-01T10:22:54.324 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1 2024-02-01T10:22:54.324 DEBUG:boto:path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:54.325 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:54.325 DEBUG:boto:Method: GET 2024-02-01T10:22:54.325 DEBUG:boto:Path: /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:54.325 DEBUG:boto:Data: 2024-02-01T10:22:54.325 DEBUG:boto:Headers: {} 2024-02-01T10:22:54.325 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:54.325 DEBUG:boto:Port: 8000 2024-02-01T10:22:54.325 DEBUG:boto:Params: {} 2024-02-01T10:22:54.325 DEBUG:boto:Token: None 2024-02-01T10:22:54.325 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:54 GMT /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:54.326 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:OJfbr5H5QvGd3Rpch8LF/LVD0CY= 2024-02-01T10:22:54.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:OJfbr5H5QvGd3Rpch8LF/LVD0CY=', 'Content-Length': '0'} 2024-02-01T10:22:54.329 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:49 GMT'), ('x-amz-version-id', 'lS9-sMOvyR-VcD.47BkPLymGmIyfWmf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fa8acb2272153299-0065bb70fe-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:54.329 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:22:54.329 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:22:54.329 DEBUG:boto:path=/vclfcn-16/?prefix=obj-test-zone1 2024-02-01T10:22:54.329 DEBUG:boto:auth_path=/vclfcn-16/?prefix=obj-test-zone1 2024-02-01T10:22:54.329 DEBUG:boto:Method: GET 2024-02-01T10:22:54.330 DEBUG:boto:Path: /vclfcn-16/?prefix=obj-test-zone1 2024-02-01T10:22:54.330 DEBUG:boto:Data: 2024-02-01T10:22:54.330 DEBUG:boto:Headers: {} 2024-02-01T10:22:54.330 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:54.330 DEBUG:boto:Port: 8000 2024-02-01T10:22:54.330 DEBUG:boto:Params: {} 2024-02-01T10:22:54.330 DEBUG:boto:Token: None 2024-02-01T10:22:54.330 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:54 GMT /vclfcn-16/ 2024-02-01T10:22:54.330 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:eqMQO31DWncWd0loSRCfyuBt6L4= 2024-02-01T10:22:54.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:eqMQO31DWncWd0loSRCfyuBt6L4=', 'Content-Length': '0'} 2024-02-01T10:22:54.334 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047225b5b24af48dd-0065bb70fe-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:54.376 DEBUG:boto:b'vclfcn-16obj-test-zone11000falseobj-test-zone12024-02-01T10:20:49.965Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:54.377 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:22:54.377 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:22:54.377 DEBUG:boto:path=/vclfcn-16/?prefix=obj-test-zone1 2024-02-01T10:22:54.377 DEBUG:boto:auth_path=/vclfcn-16/?prefix=obj-test-zone1 2024-02-01T10:22:54.377 DEBUG:boto:Method: GET 2024-02-01T10:22:54.377 DEBUG:boto:Path: /vclfcn-16/?prefix=obj-test-zone1 2024-02-01T10:22:54.377 DEBUG:boto:Data: 2024-02-01T10:22:54.377 DEBUG:boto:Headers: {} 2024-02-01T10:22:54.377 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:54.377 DEBUG:boto:Port: 8000 2024-02-01T10:22:54.378 DEBUG:boto:Params: {} 2024-02-01T10:22:54.378 DEBUG:boto:Token: None 2024-02-01T10:22:54.378 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:54 GMT /vclfcn-16/ 2024-02-01T10:22:54.378 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:eqMQO31DWncWd0loSRCfyuBt6L4= 2024-02-01T10:22:54.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:eqMQO31DWncWd0loSRCfyuBt6L4=', 'Content-Length': '0'} 2024-02-01T10:22:54.382 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043b5f569af2f390b-0065bb70fe-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:54.382 DEBUG:boto:b'vclfcn-16obj-test-zone11000falseobj-test-zone12024-02-01T10:20:49.965Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:54.383 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-02-01T10:22:54.383 DEBUG:boto:path=/vclfcn-16/obj-test-zone2 2024-02-01T10:22:54.383 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2 2024-02-01T10:22:54.383 DEBUG:boto:path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:54.383 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:54.383 DEBUG:boto:Method: GET 2024-02-01T10:22:54.383 DEBUG:boto:Path: /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:54.384 DEBUG:boto:Data: 2024-02-01T10:22:54.384 DEBUG:boto:Headers: {} 2024-02-01T10:22:54.384 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:54.384 DEBUG:boto:Port: 8000 2024-02-01T10:22:54.384 DEBUG:boto:Params: {} 2024-02-01T10:22:54.384 DEBUG:boto:Token: None 2024-02-01T10:22:54.384 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:54 GMT /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:54.384 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:74TN0ekIBZcYrDnUcFyLC2ALPS8= 2024-02-01T10:22:54.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:74TN0ekIBZcYrDnUcFyLC2ALPS8=', 'Content-Length': '0'} 2024-02-01T10:22:54.387 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'mE6urSxcrRA9wdyuToa41.w2aTia9p1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000047ba8d2bb22b613e-0065bb70fe-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:54.388 DEBUG:boto:path=/vclfcn-16/obj-test-zone2 2024-02-01T10:22:54.388 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2 2024-02-01T10:22:54.388 DEBUG:boto:path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:54.388 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:54.388 DEBUG:boto:Method: GET 2024-02-01T10:22:54.388 DEBUG:boto:Path: /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:54.388 DEBUG:boto:Data: 2024-02-01T10:22:54.389 DEBUG:boto:Headers: {} 2024-02-01T10:22:54.389 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:54.389 DEBUG:boto:Port: 8000 2024-02-01T10:22:54.389 DEBUG:boto:Params: {} 2024-02-01T10:22:54.389 DEBUG:boto:Token: None 2024-02-01T10:22:54.389 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:54 GMT /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:54.389 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:74TN0ekIBZcYrDnUcFyLC2ALPS8= 2024-02-01T10:22:54.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:74TN0ekIBZcYrDnUcFyLC2ALPS8=', 'Content-Length': '0'} 2024-02-01T10:22:54.392 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'mE6urSxcrRA9wdyuToa41.w2aTia9p1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000605e8889bb337f56-0065bb70fe-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:54.436 DEBUG:boto:path=/vclfcn-16/obj-test-zone2 2024-02-01T10:22:54.436 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2 2024-02-01T10:22:54.436 DEBUG:boto:path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:54.436 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:54.437 DEBUG:boto:Method: HEAD 2024-02-01T10:22:54.437 DEBUG:boto:Path: /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:54.437 DEBUG:boto:Data: 2024-02-01T10:22:54.437 DEBUG:boto:Headers: {} 2024-02-01T10:22:54.437 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:54.437 DEBUG:boto:Port: 8000 2024-02-01T10:22:54.437 DEBUG:boto:Params: {} 2024-02-01T10:22:54.437 DEBUG:boto:Token: None 2024-02-01T10:22:54.437 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:54 GMT /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:54.438 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QxdqGHATOipnMTdeEHccP76sNHg= 2024-02-01T10:22:54.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QxdqGHATOipnMTdeEHccP76sNHg=', 'Content-Length': '0'} 2024-02-01T10:22:54.444 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'mE6urSxcrRA9wdyuToa41.w2aTia9p1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001f6b1df3654aa4c3-0065bb70fe-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:54.444 DEBUG:boto:path=/vclfcn-16/obj-test-zone2 2024-02-01T10:22:54.444 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2 2024-02-01T10:22:54.445 DEBUG:boto:path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:54.445 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:54.445 DEBUG:boto:Method: HEAD 2024-02-01T10:22:54.445 DEBUG:boto:Path: /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:54.445 DEBUG:boto:Data: 2024-02-01T10:22:54.445 DEBUG:boto:Headers: {} 2024-02-01T10:22:54.445 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:54.445 DEBUG:boto:Port: 8000 2024-02-01T10:22:54.445 DEBUG:boto:Params: {} 2024-02-01T10:22:54.445 DEBUG:boto:Token: None 2024-02-01T10:22:54.446 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:54 GMT /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:54.446 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QxdqGHATOipnMTdeEHccP76sNHg= 2024-02-01T10:22:54.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QxdqGHATOipnMTdeEHccP76sNHg=', 'Content-Length': '0'} 2024-02-01T10:22:54.449 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'mE6urSxcrRA9wdyuToa41.w2aTia9p1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000542970b108a68316-0065bb70fe-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:54.449 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-02-01T10:22:54.449 DEBUG:boto:path=/vclfcn-16/obj-test-zone2 2024-02-01T10:22:54.449 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2 2024-02-01T10:22:54.449 DEBUG:boto:path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:54.449 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:54.449 DEBUG:boto:Method: GET 2024-02-01T10:22:54.450 DEBUG:boto:Path: /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:54.450 DEBUG:boto:Data: 2024-02-01T10:22:54.450 DEBUG:boto:Headers: {} 2024-02-01T10:22:54.450 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:54.450 DEBUG:boto:Port: 8000 2024-02-01T10:22:54.450 DEBUG:boto:Params: {} 2024-02-01T10:22:54.450 DEBUG:boto:Token: None 2024-02-01T10:22:54.450 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:54 GMT /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:54.450 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:74TN0ekIBZcYrDnUcFyLC2ALPS8= 2024-02-01T10:22:54.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:74TN0ekIBZcYrDnUcFyLC2ALPS8=', 'Content-Length': '0'} 2024-02-01T10:22:54.453 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'mE6urSxcrRA9wdyuToa41.w2aTia9p1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001de252f773e10258-0065bb70fe-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:54.454 DEBUG:boto:path=/vclfcn-16/obj-test-zone2 2024-02-01T10:22:54.454 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2 2024-02-01T10:22:54.454 DEBUG:boto:path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:54.454 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:54.454 DEBUG:boto:Method: GET 2024-02-01T10:22:54.454 DEBUG:boto:Path: /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:54.454 DEBUG:boto:Data: 2024-02-01T10:22:54.455 DEBUG:boto:Headers: {} 2024-02-01T10:22:54.455 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:54.455 DEBUG:boto:Port: 8000 2024-02-01T10:22:54.455 DEBUG:boto:Params: {} 2024-02-01T10:22:54.455 DEBUG:boto:Token: None 2024-02-01T10:22:54.455 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:54 GMT /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:54.455 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:74TN0ekIBZcYrDnUcFyLC2ALPS8= 2024-02-01T10:22:54.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:74TN0ekIBZcYrDnUcFyLC2ALPS8=', 'Content-Length': '0'} 2024-02-01T10:22:54.458 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'mE6urSxcrRA9wdyuToa41.w2aTia9p1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005ec4d68f1eb2dbc9-0065bb70fe-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:54.504 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:22:54.504 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:22:54.504 DEBUG:boto:path=/vclfcn-16/?prefix=obj-test-zone2 2024-02-01T10:22:54.505 DEBUG:boto:auth_path=/vclfcn-16/?prefix=obj-test-zone2 2024-02-01T10:22:54.505 DEBUG:boto:Method: GET 2024-02-01T10:22:54.505 DEBUG:boto:Path: /vclfcn-16/?prefix=obj-test-zone2 2024-02-01T10:22:54.505 DEBUG:boto:Data: 2024-02-01T10:22:54.505 DEBUG:boto:Headers: {} 2024-02-01T10:22:54.505 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:54.505 DEBUG:boto:Port: 8000 2024-02-01T10:22:54.505 DEBUG:boto:Params: {} 2024-02-01T10:22:54.505 DEBUG:boto:Token: None 2024-02-01T10:22:54.505 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:54 GMT /vclfcn-16/ 2024-02-01T10:22:54.506 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:eqMQO31DWncWd0loSRCfyuBt6L4= 2024-02-01T10:22:54.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:eqMQO31DWncWd0loSRCfyuBt6L4=', 'Content-Length': '0'} 2024-02-01T10:22:54.510 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010f91349dcdec473-0065bb70fe-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:54.510 DEBUG:boto:b'vclfcn-16obj-test-zone21000falseobj-test-zone22024-02-01T10:20:50.122Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:54.510 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:22:54.510 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:22:54.511 DEBUG:boto:path=/vclfcn-16/?prefix=obj-test-zone2 2024-02-01T10:22:54.511 DEBUG:boto:auth_path=/vclfcn-16/?prefix=obj-test-zone2 2024-02-01T10:22:54.511 DEBUG:boto:Method: GET 2024-02-01T10:22:54.511 DEBUG:boto:Path: /vclfcn-16/?prefix=obj-test-zone2 2024-02-01T10:22:54.511 DEBUG:boto:Data: 2024-02-01T10:22:54.511 DEBUG:boto:Headers: {} 2024-02-01T10:22:54.511 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:54.511 DEBUG:boto:Port: 8000 2024-02-01T10:22:54.511 DEBUG:boto:Params: {} 2024-02-01T10:22:54.511 DEBUG:boto:Token: None 2024-02-01T10:22:54.512 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:54 GMT /vclfcn-16/ 2024-02-01T10:22:54.512 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:eqMQO31DWncWd0loSRCfyuBt6L4= 2024-02-01T10:22:54.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:eqMQO31DWncWd0loSRCfyuBt6L4=', 'Content-Length': '0'} 2024-02-01T10:22:54.516 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7f22d47792f66a0-0065bb70fe-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:54.560 DEBUG:boto:b'vclfcn-16obj-test-zone21000falseobj-test-zone22024-02-01T10:20:50.122Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:54.561 DEBUG:boto:path=/vclfcn-16/obj-test-zone2 2024-02-01T10:22:54.561 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2 2024-02-01T10:22:54.561 DEBUG:boto:Method: HEAD 2024-02-01T10:22:54.561 DEBUG:boto:Path: /vclfcn-16/obj-test-zone2 2024-02-01T10:22:54.561 DEBUG:boto:Data: 2024-02-01T10:22:54.561 DEBUG:boto:Headers: {} 2024-02-01T10:22:54.561 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:54.561 DEBUG:boto:Port: 8000 2024-02-01T10:22:54.561 DEBUG:boto:Params: {} 2024-02-01T10:22:54.561 DEBUG:boto:Token: None 2024-02-01T10:22:54.562 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:54 GMT /vclfcn-16/obj-test-zone2 2024-02-01T10:22:54.562 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:N367Y5DcWb6ZVNwGGyBWMxNYSz0= 2024-02-01T10:22:54.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:N367Y5DcWb6ZVNwGGyBWMxNYSz0=', 'Content-Length': '0'} 2024-02-01T10:22:54.565 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'mE6urSxcrRA9wdyuToa41.w2aTia9p1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000036ccde00acd36acc-0065bb70fe-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:54.566 DEBUG:boto:path=/vclfcn-16/obj-test-zone2 2024-02-01T10:22:54.566 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2 2024-02-01T10:22:54.566 DEBUG:boto:Method: HEAD 2024-02-01T10:22:54.566 DEBUG:boto:Path: /vclfcn-16/obj-test-zone2 2024-02-01T10:22:54.566 DEBUG:boto:Data: 2024-02-01T10:22:54.566 DEBUG:boto:Headers: {} 2024-02-01T10:22:54.566 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:54.566 DEBUG:boto:Port: 8000 2024-02-01T10:22:54.566 DEBUG:boto:Params: {} 2024-02-01T10:22:54.566 DEBUG:boto:Token: None 2024-02-01T10:22:54.567 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:54 GMT /vclfcn-16/obj-test-zone2 2024-02-01T10:22:54.567 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:N367Y5DcWb6ZVNwGGyBWMxNYSz0= 2024-02-01T10:22:54.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:N367Y5DcWb6ZVNwGGyBWMxNYSz0=', 'Content-Length': '0'} 2024-02-01T10:22:54.571 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'mE6urSxcrRA9wdyuToa41.w2aTia9p1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a385e0834698a3c9-0065bb70fe-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:54.571 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-02-01T10:22:54.571 DEBUG:boto:path=/vclfcn-16/obj-test-zone2 2024-02-01T10:22:54.571 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2 2024-02-01T10:22:54.571 DEBUG:boto:path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:54.572 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:54.572 DEBUG:boto:Method: GET 2024-02-01T10:22:54.572 DEBUG:boto:Path: /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:54.572 DEBUG:boto:Data: 2024-02-01T10:22:54.572 DEBUG:boto:Headers: {} 2024-02-01T10:22:54.572 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:54.572 DEBUG:boto:Port: 8000 2024-02-01T10:22:54.572 DEBUG:boto:Params: {} 2024-02-01T10:22:54.572 DEBUG:boto:Token: None 2024-02-01T10:22:54.572 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:54 GMT /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:54.573 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:74TN0ekIBZcYrDnUcFyLC2ALPS8= 2024-02-01T10:22:54.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:74TN0ekIBZcYrDnUcFyLC2ALPS8=', 'Content-Length': '0'} 2024-02-01T10:22:54.575 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'mE6urSxcrRA9wdyuToa41.w2aTia9p1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001c1d6a132dc902a0-0065bb70fe-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:54.616 DEBUG:boto:path=/vclfcn-16/obj-test-zone2 2024-02-01T10:22:54.616 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2 2024-02-01T10:22:54.616 DEBUG:boto:path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:54.616 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:54.617 DEBUG:boto:Method: GET 2024-02-01T10:22:54.617 DEBUG:boto:Path: /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:54.617 DEBUG:boto:Data: 2024-02-01T10:22:54.617 DEBUG:boto:Headers: {} 2024-02-01T10:22:54.617 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:54.617 DEBUG:boto:Port: 8000 2024-02-01T10:22:54.617 DEBUG:boto:Params: {} 2024-02-01T10:22:54.617 DEBUG:boto:Token: None 2024-02-01T10:22:54.617 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:54 GMT /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:54.618 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:74TN0ekIBZcYrDnUcFyLC2ALPS8= 2024-02-01T10:22:54.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:74TN0ekIBZcYrDnUcFyLC2ALPS8=', 'Content-Length': '0'} 2024-02-01T10:22:54.621 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'mE6urSxcrRA9wdyuToa41.w2aTia9p1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009bde0f4f300886e7-0065bb70fe-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:54.664 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:22:54.664 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:22:54.664 DEBUG:boto:path=/vclfcn-16/?prefix=obj-test-zone2 2024-02-01T10:22:54.665 DEBUG:boto:auth_path=/vclfcn-16/?prefix=obj-test-zone2 2024-02-01T10:22:54.665 DEBUG:boto:Method: GET 2024-02-01T10:22:54.665 DEBUG:boto:Path: /vclfcn-16/?prefix=obj-test-zone2 2024-02-01T10:22:54.665 DEBUG:boto:Data: 2024-02-01T10:22:54.665 DEBUG:boto:Headers: {} 2024-02-01T10:22:54.665 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:54.665 DEBUG:boto:Port: 8000 2024-02-01T10:22:54.665 DEBUG:boto:Params: {} 2024-02-01T10:22:54.665 DEBUG:boto:Token: None 2024-02-01T10:22:54.665 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:54 GMT /vclfcn-16/ 2024-02-01T10:22:54.666 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:eqMQO31DWncWd0loSRCfyuBt6L4= 2024-02-01T10:22:54.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:eqMQO31DWncWd0loSRCfyuBt6L4=', 'Content-Length': '0'} 2024-02-01T10:22:54.670 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000069f37f10ba69e130-0065bb70fe-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:54.670 DEBUG:boto:b'vclfcn-16obj-test-zone21000falseobj-test-zone22024-02-01T10:20:50.122Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:54.670 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:22:54.671 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:22:54.671 DEBUG:boto:path=/vclfcn-16/?prefix=obj-test-zone2 2024-02-01T10:22:54.671 DEBUG:boto:auth_path=/vclfcn-16/?prefix=obj-test-zone2 2024-02-01T10:22:54.671 DEBUG:boto:Method: GET 2024-02-01T10:22:54.671 DEBUG:boto:Path: /vclfcn-16/?prefix=obj-test-zone2 2024-02-01T10:22:54.671 DEBUG:boto:Data: 2024-02-01T10:22:54.671 DEBUG:boto:Headers: {} 2024-02-01T10:22:54.671 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:54.671 DEBUG:boto:Port: 8000 2024-02-01T10:22:54.671 DEBUG:boto:Params: {} 2024-02-01T10:22:54.672 DEBUG:boto:Token: None 2024-02-01T10:22:54.672 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:54 GMT /vclfcn-16/ 2024-02-01T10:22:54.672 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:eqMQO31DWncWd0loSRCfyuBt6L4= 2024-02-01T10:22:54.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:eqMQO31DWncWd0loSRCfyuBt6L4=', 'Content-Length': '0'} 2024-02-01T10:22:54.677 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a15f6d477a83dfe2-0065bb70fe-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:54.720 DEBUG:boto:b'vclfcn-16obj-test-zone21000falseobj-test-zone22024-02-01T10:20:50.122Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:54.721 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-02-01T10:22:54.721 DEBUG:boto:path=/vclfcn-16/obj-test-zone3 2024-02-01T10:22:54.721 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3 2024-02-01T10:22:54.721 DEBUG:boto:path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:54.721 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:54.721 DEBUG:boto:Method: GET 2024-02-01T10:22:54.721 DEBUG:boto:Path: /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:54.721 DEBUG:boto:Data: 2024-02-01T10:22:54.721 DEBUG:boto:Headers: {} 2024-02-01T10:22:54.721 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:54.722 DEBUG:boto:Port: 8000 2024-02-01T10:22:54.722 DEBUG:boto:Params: {} 2024-02-01T10:22:54.722 DEBUG:boto:Token: None 2024-02-01T10:22:54.722 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:54 GMT /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:54.722 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:IapjCcql3ZvigCIQrPXKCfYAJqE= 2024-02-01T10:22:54.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:IapjCcql3ZvigCIQrPXKCfYAJqE=', 'Content-Length': '0'} 2024-02-01T10:22:54.725 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000224cb0eb28b40b68-0065bb70fe-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:54.772 DEBUG:boto:path=/vclfcn-16/obj-test-zone3 2024-02-01T10:22:54.772 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3 2024-02-01T10:22:54.772 DEBUG:boto:path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:54.773 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:54.773 DEBUG:boto:Method: GET 2024-02-01T10:22:54.773 DEBUG:boto:Path: /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:54.773 DEBUG:boto:Data: 2024-02-01T10:22:54.773 DEBUG:boto:Headers: {} 2024-02-01T10:22:54.773 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:54.773 DEBUG:boto:Port: 8000 2024-02-01T10:22:54.773 DEBUG:boto:Params: {} 2024-02-01T10:22:54.773 DEBUG:boto:Token: None 2024-02-01T10:22:54.773 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:54 GMT /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:54.774 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:IapjCcql3ZvigCIQrPXKCfYAJqE= 2024-02-01T10:22:54.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:IapjCcql3ZvigCIQrPXKCfYAJqE=', 'Content-Length': '0'} 2024-02-01T10:22:54.777 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000056f16368a7a6d721-0065bb70fe-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:54.777 DEBUG:boto:path=/vclfcn-16/obj-test-zone3 2024-02-01T10:22:54.777 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3 2024-02-01T10:22:54.777 DEBUG:boto:path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:54.778 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:54.778 DEBUG:boto:Method: HEAD 2024-02-01T10:22:54.778 DEBUG:boto:Path: /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:54.778 DEBUG:boto:Data: 2024-02-01T10:22:54.778 DEBUG:boto:Headers: {} 2024-02-01T10:22:54.778 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:54.778 DEBUG:boto:Port: 8000 2024-02-01T10:22:54.778 DEBUG:boto:Params: {} 2024-02-01T10:22:54.778 DEBUG:boto:Token: None 2024-02-01T10:22:54.778 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:54 GMT /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:54.779 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZXPDq4e6vhxxPjla8WGR8PEVzlI= 2024-02-01T10:22:54.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZXPDq4e6vhxxPjla8WGR8PEVzlI=', 'Content-Length': '0'} 2024-02-01T10:22:54.782 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000218014683977c3b8-0065bb70fe-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:54.782 DEBUG:boto:path=/vclfcn-16/obj-test-zone3 2024-02-01T10:22:54.782 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3 2024-02-01T10:22:54.782 DEBUG:boto:path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:54.782 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:54.782 DEBUG:boto:Method: HEAD 2024-02-01T10:22:54.782 DEBUG:boto:Path: /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:54.783 DEBUG:boto:Data: 2024-02-01T10:22:54.783 DEBUG:boto:Headers: {} 2024-02-01T10:22:54.783 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:54.783 DEBUG:boto:Port: 8000 2024-02-01T10:22:54.783 DEBUG:boto:Params: {} 2024-02-01T10:22:54.783 DEBUG:boto:Token: None 2024-02-01T10:22:54.783 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:54 GMT /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:54.783 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZXPDq4e6vhxxPjla8WGR8PEVzlI= 2024-02-01T10:22:54.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZXPDq4e6vhxxPjla8WGR8PEVzlI=', 'Content-Length': '0'} 2024-02-01T10:22:54.786 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e144fccdeb51f51a-0065bb70fe-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:54.787 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-02-01T10:22:54.787 DEBUG:boto:path=/vclfcn-16/obj-test-zone3 2024-02-01T10:22:54.788 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3 2024-02-01T10:22:54.788 DEBUG:boto:path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:54.788 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:54.788 DEBUG:boto:Method: GET 2024-02-01T10:22:54.788 DEBUG:boto:Path: /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:54.788 DEBUG:boto:Data: 2024-02-01T10:22:54.789 DEBUG:boto:Headers: {} 2024-02-01T10:22:54.789 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:54.789 DEBUG:boto:Port: 8000 2024-02-01T10:22:54.789 DEBUG:boto:Params: {} 2024-02-01T10:22:54.789 DEBUG:boto:Token: None 2024-02-01T10:22:54.789 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:54 GMT /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:54.789 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:IapjCcql3ZvigCIQrPXKCfYAJqE= 2024-02-01T10:22:54.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:IapjCcql3ZvigCIQrPXKCfYAJqE=', 'Content-Length': '0'} 2024-02-01T10:22:54.792 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000836acd87a40e1202-0065bb70fe-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:54.836 DEBUG:boto:path=/vclfcn-16/obj-test-zone3 2024-02-01T10:22:54.837 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3 2024-02-01T10:22:54.837 DEBUG:boto:path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:54.837 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:54.837 DEBUG:boto:Method: GET 2024-02-01T10:22:54.837 DEBUG:boto:Path: /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:54.837 DEBUG:boto:Data: 2024-02-01T10:22:54.837 DEBUG:boto:Headers: {} 2024-02-01T10:22:54.837 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:54.837 DEBUG:boto:Port: 8000 2024-02-01T10:22:54.837 DEBUG:boto:Params: {} 2024-02-01T10:22:54.837 DEBUG:boto:Token: None 2024-02-01T10:22:54.838 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:54 GMT /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:54.838 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:IapjCcql3ZvigCIQrPXKCfYAJqE= 2024-02-01T10:22:54.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:IapjCcql3ZvigCIQrPXKCfYAJqE=', 'Content-Length': '0'} 2024-02-01T10:22:54.841 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001f9446d5811c1c41-0065bb70fe-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:54.888 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:22:54.888 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:22:54.888 DEBUG:boto:path=/vclfcn-16/?prefix=obj-test-zone3 2024-02-01T10:22:54.889 DEBUG:boto:auth_path=/vclfcn-16/?prefix=obj-test-zone3 2024-02-01T10:22:54.889 DEBUG:boto:Method: GET 2024-02-01T10:22:54.889 DEBUG:boto:Path: /vclfcn-16/?prefix=obj-test-zone3 2024-02-01T10:22:54.889 DEBUG:boto:Data: 2024-02-01T10:22:54.889 DEBUG:boto:Headers: {} 2024-02-01T10:22:54.889 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:54.889 DEBUG:boto:Port: 8000 2024-02-01T10:22:54.889 DEBUG:boto:Params: {} 2024-02-01T10:22:54.889 DEBUG:boto:Token: None 2024-02-01T10:22:54.889 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:54 GMT /vclfcn-16/ 2024-02-01T10:22:54.890 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:eqMQO31DWncWd0loSRCfyuBt6L4= 2024-02-01T10:22:54.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:eqMQO31DWncWd0loSRCfyuBt6L4=', 'Content-Length': '0'} 2024-02-01T10:22:54.894 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2a151050f1ab50b-0065bb70fe-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:54.936 DEBUG:boto:b'vclfcn-16obj-test-zone31000falseobj-test-zone32024-02-01T10:20:50.297Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:54.937 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:22:54.937 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:22:54.937 DEBUG:boto:path=/vclfcn-16/?prefix=obj-test-zone3 2024-02-01T10:22:54.937 DEBUG:boto:auth_path=/vclfcn-16/?prefix=obj-test-zone3 2024-02-01T10:22:54.937 DEBUG:boto:Method: GET 2024-02-01T10:22:54.937 DEBUG:boto:Path: /vclfcn-16/?prefix=obj-test-zone3 2024-02-01T10:22:54.937 DEBUG:boto:Data: 2024-02-01T10:22:54.937 DEBUG:boto:Headers: {} 2024-02-01T10:22:54.937 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:54.937 DEBUG:boto:Port: 8000 2024-02-01T10:22:54.938 DEBUG:boto:Params: {} 2024-02-01T10:22:54.938 DEBUG:boto:Token: None 2024-02-01T10:22:54.938 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:54 GMT /vclfcn-16/ 2024-02-01T10:22:54.938 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:eqMQO31DWncWd0loSRCfyuBt6L4= 2024-02-01T10:22:54.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:eqMQO31DWncWd0loSRCfyuBt6L4=', 'Content-Length': '0'} 2024-02-01T10:22:54.942 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009789c9e3d0266ee3-0065bb70fe-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:54.943 DEBUG:boto:b'vclfcn-16obj-test-zone31000falseobj-test-zone32024-02-01T10:20:50.297Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:54.943 DEBUG:boto:path=/vclfcn-16/obj-test-zone3 2024-02-01T10:22:54.943 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3 2024-02-01T10:22:54.943 DEBUG:boto:Method: HEAD 2024-02-01T10:22:54.943 DEBUG:boto:Path: /vclfcn-16/obj-test-zone3 2024-02-01T10:22:54.943 DEBUG:boto:Data: 2024-02-01T10:22:54.943 DEBUG:boto:Headers: {} 2024-02-01T10:22:54.943 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:54.943 DEBUG:boto:Port: 8000 2024-02-01T10:22:54.944 DEBUG:boto:Params: {} 2024-02-01T10:22:54.944 DEBUG:boto:Token: None 2024-02-01T10:22:54.944 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:54 GMT /vclfcn-16/obj-test-zone3 2024-02-01T10:22:54.944 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+UTZkK56r1FBr+4p7AXPoqzE2Hk= 2024-02-01T10:22:54.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+UTZkK56r1FBr+4p7AXPoqzE2Hk=', 'Content-Length': '0'} 2024-02-01T10:22:54.948 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000207ef8c7de02a4cb-0065bb70fe-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:54.948 DEBUG:boto:path=/vclfcn-16/obj-test-zone3 2024-02-01T10:22:54.948 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3 2024-02-01T10:22:54.948 DEBUG:boto:Method: HEAD 2024-02-01T10:22:54.948 DEBUG:boto:Path: /vclfcn-16/obj-test-zone3 2024-02-01T10:22:54.948 DEBUG:boto:Data: 2024-02-01T10:22:54.948 DEBUG:boto:Headers: {} 2024-02-01T10:22:54.949 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:54.949 DEBUG:boto:Port: 8000 2024-02-01T10:22:54.949 DEBUG:boto:Params: {} 2024-02-01T10:22:54.949 DEBUG:boto:Token: None 2024-02-01T10:22:54.949 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:54 GMT /vclfcn-16/obj-test-zone3 2024-02-01T10:22:54.949 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+UTZkK56r1FBr+4p7AXPoqzE2Hk= 2024-02-01T10:22:54.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+UTZkK56r1FBr+4p7AXPoqzE2Hk=', 'Content-Length': '0'} 2024-02-01T10:22:54.953 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005bba4a4bd7dfc2a7-0065bb70fe-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:54.953 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-02-01T10:22:54.954 DEBUG:boto:path=/vclfcn-16/obj-test-zone3 2024-02-01T10:22:54.954 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3 2024-02-01T10:22:54.954 DEBUG:boto:path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:54.954 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:54.954 DEBUG:boto:Method: GET 2024-02-01T10:22:54.954 DEBUG:boto:Path: /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:54.954 DEBUG:boto:Data: 2024-02-01T10:22:54.954 DEBUG:boto:Headers: {} 2024-02-01T10:22:54.954 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:54.954 DEBUG:boto:Port: 8000 2024-02-01T10:22:54.954 DEBUG:boto:Params: {} 2024-02-01T10:22:54.955 DEBUG:boto:Token: None 2024-02-01T10:22:54.955 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:54 GMT /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:54.955 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:IapjCcql3ZvigCIQrPXKCfYAJqE= 2024-02-01T10:22:54.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:IapjCcql3ZvigCIQrPXKCfYAJqE=', 'Content-Length': '0'} 2024-02-01T10:22:54.958 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f9f7e52b9bf19542-0065bb70fe-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:55.000 DEBUG:boto:path=/vclfcn-16/obj-test-zone3 2024-02-01T10:22:55.000 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3 2024-02-01T10:22:55.000 DEBUG:boto:path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:55.000 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:55.001 DEBUG:boto:Method: GET 2024-02-01T10:22:55.001 DEBUG:boto:Path: /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:55.001 DEBUG:boto:Data: 2024-02-01T10:22:55.001 DEBUG:boto:Headers: {} 2024-02-01T10:22:55.001 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:55.001 DEBUG:boto:Port: 8000 2024-02-01T10:22:55.001 DEBUG:boto:Params: {} 2024-02-01T10:22:55.001 DEBUG:boto:Token: None 2024-02-01T10:22:55.001 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:55 GMT /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:55.002 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:s+TPeaJNuyaRH671M8maTsqcdno= 2024-02-01T10:22:55.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:s+TPeaJNuyaRH671M8maTsqcdno=', 'Content-Length': '0'} 2024-02-01T10:22:55.005 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c9d87dab20e633f5-0065bb70ff-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:55.048 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:22:55.049 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:22:55.049 DEBUG:boto:path=/vclfcn-16/?prefix=obj-test-zone3 2024-02-01T10:22:55.049 DEBUG:boto:auth_path=/vclfcn-16/?prefix=obj-test-zone3 2024-02-01T10:22:55.049 DEBUG:boto:Method: GET 2024-02-01T10:22:55.049 DEBUG:boto:Path: /vclfcn-16/?prefix=obj-test-zone3 2024-02-01T10:22:55.050 DEBUG:boto:Data: 2024-02-01T10:22:55.050 DEBUG:boto:Headers: {} 2024-02-01T10:22:55.050 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:55.050 DEBUG:boto:Port: 8000 2024-02-01T10:22:55.050 DEBUG:boto:Params: {} 2024-02-01T10:22:55.051 DEBUG:boto:Token: None 2024-02-01T10:22:55.051 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:55 GMT /vclfcn-16/ 2024-02-01T10:22:55.051 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tdgWY91d/aSko0GhasKPmcABFIg= 2024-02-01T10:22:55.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tdgWY91d/aSko0GhasKPmcABFIg=', 'Content-Length': '0'} 2024-02-01T10:22:55.056 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000279cb3c323496b4b-0065bb70ff-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:55.056 DEBUG:boto:b'vclfcn-16obj-test-zone31000falseobj-test-zone32024-02-01T10:20:50.297Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:55.057 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:22:55.057 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:22:55.057 DEBUG:boto:path=/vclfcn-16/?prefix=obj-test-zone3 2024-02-01T10:22:55.057 DEBUG:boto:auth_path=/vclfcn-16/?prefix=obj-test-zone3 2024-02-01T10:22:55.058 DEBUG:boto:Method: GET 2024-02-01T10:22:55.058 DEBUG:boto:Path: /vclfcn-16/?prefix=obj-test-zone3 2024-02-01T10:22:55.058 DEBUG:boto:Data: 2024-02-01T10:22:55.058 DEBUG:boto:Headers: {} 2024-02-01T10:22:55.058 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:55.059 DEBUG:boto:Port: 8000 2024-02-01T10:22:55.059 DEBUG:boto:Params: {} 2024-02-01T10:22:55.059 DEBUG:boto:Token: None 2024-02-01T10:22:55.059 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:55 GMT /vclfcn-16/ 2024-02-01T10:22:55.060 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tdgWY91d/aSko0GhasKPmcABFIg= 2024-02-01T10:22:55.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tdgWY91d/aSko0GhasKPmcABFIg=', 'Content-Length': '0'} 2024-02-01T10:22:55.064 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f66d73a5bdd52ee4-0065bb70ff-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:55.064 DEBUG:boto:b'vclfcn-16obj-test-zone31000falseobj-test-zone32024-02-01T10:20:50.297Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:55.065 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-16 zones={test-zone2, test-zone1} 2024-02-01T10:22:55.065 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-16 zones={test-zone3, test-zone1} 2024-02-01T10:22:55.065 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:22:55.065 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:22:55.065 DEBUG:boto:Method: HEAD 2024-02-01T10:22:55.066 DEBUG:boto:Path: /vclfcn-16/ 2024-02-01T10:22:55.066 DEBUG:boto:Data: 2024-02-01T10:22:55.066 DEBUG:boto:Headers: {} 2024-02-01T10:22:55.066 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:55.066 DEBUG:boto:Port: 8001 2024-02-01T10:22:55.066 DEBUG:boto:Params: {} 2024-02-01T10:22:55.066 DEBUG:boto:Token: None 2024-02-01T10:22:55.067 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:55 GMT /vclfcn-16/ 2024-02-01T10:22:55.067 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RD6I6/Zd9yHI47hyc+ZjwygDmNY= 2024-02-01T10:22:55.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RD6I6/Zd9yHI47hyc+ZjwygDmNY=', 'Content-Length': '0'} 2024-02-01T10:22:55.071 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009f60f8b931514125-0065bb70ff-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:55.071 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:22:55.071 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:22:55.071 DEBUG:boto:Method: HEAD 2024-02-01T10:22:55.071 DEBUG:boto:Path: /vclfcn-16/ 2024-02-01T10:22:55.071 DEBUG:boto:Data: 2024-02-01T10:22:55.072 DEBUG:boto:Headers: {} 2024-02-01T10:22:55.072 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:55.072 DEBUG:boto:Port: 8000 2024-02-01T10:22:55.072 DEBUG:boto:Params: {} 2024-02-01T10:22:55.072 DEBUG:boto:Token: None 2024-02-01T10:22:55.072 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:55 GMT /vclfcn-16/ 2024-02-01T10:22:55.072 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RD6I6/Zd9yHI47hyc+ZjwygDmNY= 2024-02-01T10:22:55.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RD6I6/Zd9yHI47hyc+ZjwygDmNY=', 'Content-Length': '0'} 2024-02-01T10:22:55.076 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005dc84daa1ba61994-0065bb70ff-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:55.076 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:22:55.077 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:22:55.077 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:22:55.077 DEBUG:boto:path=/vclfcn-16/?versions&max-keys=999 2024-02-01T10:22:55.077 DEBUG:boto:auth_path=/vclfcn-16/?versions&max-keys=999 2024-02-01T10:22:55.077 DEBUG:boto:Method: GET 2024-02-01T10:22:55.077 DEBUG:boto:Path: /vclfcn-16/?versions&max-keys=999 2024-02-01T10:22:55.077 DEBUG:boto:Data: 2024-02-01T10:22:55.077 DEBUG:boto:Headers: {} 2024-02-01T10:22:55.077 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:55.077 DEBUG:boto:Port: 8001 2024-02-01T10:22:55.077 DEBUG:boto:Params: {} 2024-02-01T10:22:55.078 DEBUG:boto:Token: None 2024-02-01T10:22:55.078 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:55 GMT /vclfcn-16/?versions 2024-02-01T10:22:55.078 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7/FKU3+dfXkhZ2T6DOFKQnyYRfY= 2024-02-01T10:22:55.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7/FKU3+dfXkhZ2T6DOFKQnyYRfY=', 'Content-Length': '0'} 2024-02-01T10:22:55.082 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa4be72db141302c-0065bb70ff-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:55.082 DEBUG:boto:b'vclfcn-16999falsedummyrQRpBowIFAsvxVDn92H30shYnKoGt4ntrue2024-02-01T10:19:58.958Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1lS9-sMOvyR-VcD.47BkPLymGmIyfWmftrue2024-02-01T10:20:49.965Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2mE6urSxcrRA9wdyuToa41.w2aTia9p1true2024-02-01T10:20:50.122Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjPtrue2024-02-01T10:20:50.297Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:55.083 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-02-01T10:22:55.083 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-02-01T10:22:55.083 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-02-01T10:22:55.084 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-02-01T10:22:55.084 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:22:55.084 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:22:55.084 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:22:55.084 DEBUG:boto:path=/vclfcn-16/?versions&max-keys=999 2024-02-01T10:22:55.084 DEBUG:boto:auth_path=/vclfcn-16/?versions&max-keys=999 2024-02-01T10:22:55.084 DEBUG:boto:Method: GET 2024-02-01T10:22:55.084 DEBUG:boto:Path: /vclfcn-16/?versions&max-keys=999 2024-02-01T10:22:55.084 DEBUG:boto:Data: 2024-02-01T10:22:55.085 DEBUG:boto:Headers: {} 2024-02-01T10:22:55.085 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:55.086 DEBUG:boto:Port: 8000 2024-02-01T10:22:55.086 DEBUG:boto:Params: {} 2024-02-01T10:22:55.086 DEBUG:boto:Token: None 2024-02-01T10:22:55.086 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:55 GMT /vclfcn-16/?versions 2024-02-01T10:22:55.086 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7/FKU3+dfXkhZ2T6DOFKQnyYRfY= 2024-02-01T10:22:55.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7/FKU3+dfXkhZ2T6DOFKQnyYRfY=', 'Content-Length': '0'} 2024-02-01T10:22:55.091 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000001aaf7a0f822f940-0065bb70ff-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:55.132 DEBUG:boto:b'vclfcn-16999falsedummyrQRpBowIFAsvxVDn92H30shYnKoGt4ntrue2024-02-01T10:19:58.958Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1lS9-sMOvyR-VcD.47BkPLymGmIyfWmftrue2024-02-01T10:20:49.965Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2mE6urSxcrRA9wdyuToa41.w2aTia9p1true2024-02-01T10:20:50.122Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjPtrue2024-02-01T10:20:50.297Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:55.133 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-02-01T10:22:55.133 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-02-01T10:22:55.133 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-02-01T10:22:55.133 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-02-01T10:22:55.133 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:22:55.133 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:22:55.133 DEBUG:boto:path=/vclfcn-16/?versions&max-keys=999 2024-02-01T10:22:55.133 DEBUG:boto:auth_path=/vclfcn-16/?versions&max-keys=999 2024-02-01T10:22:55.133 DEBUG:boto:Method: GET 2024-02-01T10:22:55.134 DEBUG:boto:Path: /vclfcn-16/?versions&max-keys=999 2024-02-01T10:22:55.134 DEBUG:boto:Data: 2024-02-01T10:22:55.134 DEBUG:boto:Headers: {} 2024-02-01T10:22:55.134 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:55.134 DEBUG:boto:Port: 8001 2024-02-01T10:22:55.134 DEBUG:boto:Params: {} 2024-02-01T10:22:55.134 DEBUG:boto:Token: None 2024-02-01T10:22:55.134 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:55 GMT /vclfcn-16/?versions 2024-02-01T10:22:55.134 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7/FKU3+dfXkhZ2T6DOFKQnyYRfY= 2024-02-01T10:22:55.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7/FKU3+dfXkhZ2T6DOFKQnyYRfY=', 'Content-Length': '0'} 2024-02-01T10:22:55.139 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008598f83c6540c9eb-0065bb70ff-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:55.139 DEBUG:boto:b'vclfcn-16999falsedummyrQRpBowIFAsvxVDn92H30shYnKoGt4ntrue2024-02-01T10:19:58.958Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1lS9-sMOvyR-VcD.47BkPLymGmIyfWmftrue2024-02-01T10:20:49.965Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2mE6urSxcrRA9wdyuToa41.w2aTia9p1true2024-02-01T10:20:50.122Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjPtrue2024-02-01T10:20:50.297Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:55.139 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:22:55.140 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:22:55.140 DEBUG:boto:path=/vclfcn-16/?versions&max-keys=999 2024-02-01T10:22:55.140 DEBUG:boto:auth_path=/vclfcn-16/?versions&max-keys=999 2024-02-01T10:22:55.140 DEBUG:boto:Method: GET 2024-02-01T10:22:55.140 DEBUG:boto:Path: /vclfcn-16/?versions&max-keys=999 2024-02-01T10:22:55.140 DEBUG:boto:Data: 2024-02-01T10:22:55.140 DEBUG:boto:Headers: {} 2024-02-01T10:22:55.140 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:55.140 DEBUG:boto:Port: 8000 2024-02-01T10:22:55.140 DEBUG:boto:Params: {} 2024-02-01T10:22:55.140 DEBUG:boto:Token: None 2024-02-01T10:22:55.141 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:55 GMT /vclfcn-16/?versions 2024-02-01T10:22:55.141 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7/FKU3+dfXkhZ2T6DOFKQnyYRfY= 2024-02-01T10:22:55.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7/FKU3+dfXkhZ2T6DOFKQnyYRfY=', 'Content-Length': '0'} 2024-02-01T10:22:55.146 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004983b5d3a61f1ba3-0065bb70ff-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:55.146 DEBUG:boto:b'vclfcn-16999falsedummyrQRpBowIFAsvxVDn92H30shYnKoGt4ntrue2024-02-01T10:19:58.958Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1lS9-sMOvyR-VcD.47BkPLymGmIyfWmftrue2024-02-01T10:20:49.965Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2mE6urSxcrRA9wdyuToa41.w2aTia9p1true2024-02-01T10:20:50.122Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjPtrue2024-02-01T10:20:50.297Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:55.146 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-02-01T10:22:55.146 DEBUG:boto:path=/vclfcn-16/dummy 2024-02-01T10:22:55.147 DEBUG:boto:auth_path=/vclfcn-16/dummy 2024-02-01T10:22:55.147 DEBUG:boto:path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:55.147 DEBUG:boto:auth_path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:55.147 DEBUG:boto:Method: GET 2024-02-01T10:22:55.147 DEBUG:boto:Path: /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:55.147 DEBUG:boto:Data: 2024-02-01T10:22:55.147 DEBUG:boto:Headers: {} 2024-02-01T10:22:55.147 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:55.147 DEBUG:boto:Port: 8001 2024-02-01T10:22:55.147 DEBUG:boto:Params: {} 2024-02-01T10:22:55.147 DEBUG:boto:Token: None 2024-02-01T10:22:55.148 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:55 GMT /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:55.148 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:v7aqVQG6hiUnU2Gf8mlGHljKPfk= 2024-02-01T10:22:55.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:v7aqVQG6hiUnU2Gf8mlGHljKPfk=', 'Content-Length': '0'} 2024-02-01T10:22:55.151 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:19:58 GMT'), ('x-amz-version-id', 'rQRpBowIFAsvxVDn92H30shYnKoGt4n'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f875b2938c6aea20-0065bb70ff-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:55.151 DEBUG:boto:path=/vclfcn-16/dummy 2024-02-01T10:22:55.151 DEBUG:boto:auth_path=/vclfcn-16/dummy 2024-02-01T10:22:55.151 DEBUG:boto:path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:55.151 DEBUG:boto:auth_path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:55.151 DEBUG:boto:Method: GET 2024-02-01T10:22:55.152 DEBUG:boto:Path: /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:55.152 DEBUG:boto:Data: 2024-02-01T10:22:55.152 DEBUG:boto:Headers: {} 2024-02-01T10:22:55.152 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:55.152 DEBUG:boto:Port: 8000 2024-02-01T10:22:55.152 DEBUG:boto:Params: {} 2024-02-01T10:22:55.152 DEBUG:boto:Token: None 2024-02-01T10:22:55.152 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:55 GMT /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:55.152 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:v7aqVQG6hiUnU2Gf8mlGHljKPfk= 2024-02-01T10:22:55.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:v7aqVQG6hiUnU2Gf8mlGHljKPfk=', 'Content-Length': '0'} 2024-02-01T10:22:55.155 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:19:58 GMT'), ('x-amz-version-id', 'rQRpBowIFAsvxVDn92H30shYnKoGt4n'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000749cc9217040080a-0065bb70ff-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:55.155 DEBUG:boto:path=/vclfcn-16/dummy 2024-02-01T10:22:55.155 DEBUG:boto:auth_path=/vclfcn-16/dummy 2024-02-01T10:22:55.155 DEBUG:boto:path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:55.155 DEBUG:boto:auth_path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:55.156 DEBUG:boto:Method: HEAD 2024-02-01T10:22:55.156 DEBUG:boto:Path: /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:55.156 DEBUG:boto:Data: 2024-02-01T10:22:55.156 DEBUG:boto:Headers: {} 2024-02-01T10:22:55.156 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:55.156 DEBUG:boto:Port: 8001 2024-02-01T10:22:55.156 DEBUG:boto:Params: {} 2024-02-01T10:22:55.156 DEBUG:boto:Token: None 2024-02-01T10:22:55.156 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:55 GMT /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:55.156 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:t2jt5E+NorsOV2ZU4/pKdT3mJME= 2024-02-01T10:22:55.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:t2jt5E+NorsOV2ZU4/pKdT3mJME=', 'Content-Length': '0'} 2024-02-01T10:22:55.160 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:19:58 GMT'), ('x-amz-version-id', 'rQRpBowIFAsvxVDn92H30shYnKoGt4n'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000de9a65e85acae387-0065bb70ff-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:55.160 DEBUG:boto:path=/vclfcn-16/dummy 2024-02-01T10:22:55.160 DEBUG:boto:auth_path=/vclfcn-16/dummy 2024-02-01T10:22:55.160 DEBUG:boto:path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:55.161 DEBUG:boto:auth_path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:55.161 DEBUG:boto:Method: HEAD 2024-02-01T10:22:55.161 DEBUG:boto:Path: /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:55.161 DEBUG:boto:Data: 2024-02-01T10:22:55.161 DEBUG:boto:Headers: {} 2024-02-01T10:22:55.161 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:55.161 DEBUG:boto:Port: 8000 2024-02-01T10:22:55.161 DEBUG:boto:Params: {} 2024-02-01T10:22:55.161 DEBUG:boto:Token: None 2024-02-01T10:22:55.162 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:55 GMT /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:55.162 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:t2jt5E+NorsOV2ZU4/pKdT3mJME= 2024-02-01T10:22:55.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:t2jt5E+NorsOV2ZU4/pKdT3mJME=', 'Content-Length': '0'} 2024-02-01T10:22:55.165 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:19:58 GMT'), ('x-amz-version-id', 'rQRpBowIFAsvxVDn92H30shYnKoGt4n'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000056a77ad14044148a-0065bb70ff-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:55.165 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-02-01T10:22:55.165 DEBUG:boto:path=/vclfcn-16/dummy 2024-02-01T10:22:55.165 DEBUG:boto:auth_path=/vclfcn-16/dummy 2024-02-01T10:22:55.165 DEBUG:boto:path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:55.165 DEBUG:boto:auth_path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:55.165 DEBUG:boto:Method: GET 2024-02-01T10:22:55.165 DEBUG:boto:Path: /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:55.166 DEBUG:boto:Data: 2024-02-01T10:22:55.166 DEBUG:boto:Headers: {} 2024-02-01T10:22:55.166 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:55.166 DEBUG:boto:Port: 8001 2024-02-01T10:22:55.166 DEBUG:boto:Params: {} 2024-02-01T10:22:55.166 DEBUG:boto:Token: None 2024-02-01T10:22:55.166 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:55 GMT /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:55.166 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:v7aqVQG6hiUnU2Gf8mlGHljKPfk= 2024-02-01T10:22:55.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:v7aqVQG6hiUnU2Gf8mlGHljKPfk=', 'Content-Length': '0'} 2024-02-01T10:22:55.169 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:19:58 GMT'), ('x-amz-version-id', 'rQRpBowIFAsvxVDn92H30shYnKoGt4n'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004dfcdb3f6f4a3572-0065bb70ff-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:55.169 DEBUG:boto:path=/vclfcn-16/dummy 2024-02-01T10:22:55.169 DEBUG:boto:auth_path=/vclfcn-16/dummy 2024-02-01T10:22:55.169 DEBUG:boto:path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:55.170 DEBUG:boto:auth_path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:55.170 DEBUG:boto:Method: GET 2024-02-01T10:22:55.170 DEBUG:boto:Path: /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:55.170 DEBUG:boto:Data: 2024-02-01T10:22:55.170 DEBUG:boto:Headers: {} 2024-02-01T10:22:55.170 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:55.170 DEBUG:boto:Port: 8000 2024-02-01T10:22:55.170 DEBUG:boto:Params: {} 2024-02-01T10:22:55.170 DEBUG:boto:Token: None 2024-02-01T10:22:55.171 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:55 GMT /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:55.171 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:v7aqVQG6hiUnU2Gf8mlGHljKPfk= 2024-02-01T10:22:55.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:v7aqVQG6hiUnU2Gf8mlGHljKPfk=', 'Content-Length': '0'} 2024-02-01T10:22:55.173 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:19:58 GMT'), ('x-amz-version-id', 'rQRpBowIFAsvxVDn92H30shYnKoGt4n'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000372ec3c3e1b29e8e-0065bb70ff-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:55.174 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:22:55.174 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:22:55.174 DEBUG:boto:path=/vclfcn-16/?prefix=dummy 2024-02-01T10:22:55.174 DEBUG:boto:auth_path=/vclfcn-16/?prefix=dummy 2024-02-01T10:22:55.174 DEBUG:boto:Method: GET 2024-02-01T10:22:55.174 DEBUG:boto:Path: /vclfcn-16/?prefix=dummy 2024-02-01T10:22:55.174 DEBUG:boto:Data: 2024-02-01T10:22:55.174 DEBUG:boto:Headers: {} 2024-02-01T10:22:55.174 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:55.174 DEBUG:boto:Port: 8001 2024-02-01T10:22:55.175 DEBUG:boto:Params: {} 2024-02-01T10:22:55.175 DEBUG:boto:Token: None 2024-02-01T10:22:55.175 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:55 GMT /vclfcn-16/ 2024-02-01T10:22:55.175 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tdgWY91d/aSko0GhasKPmcABFIg= 2024-02-01T10:22:55.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tdgWY91d/aSko0GhasKPmcABFIg=', 'Content-Length': '0'} 2024-02-01T10:22:55.179 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000020f94d821aba92b2-0065bb70ff-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:55.220 DEBUG:boto:b'vclfcn-16dummy1000falsedummy2024-02-01T10:19:58.958Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:55.221 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:22:55.221 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:22:55.221 DEBUG:boto:path=/vclfcn-16/?prefix=dummy 2024-02-01T10:22:55.221 DEBUG:boto:auth_path=/vclfcn-16/?prefix=dummy 2024-02-01T10:22:55.221 DEBUG:boto:Method: GET 2024-02-01T10:22:55.221 DEBUG:boto:Path: /vclfcn-16/?prefix=dummy 2024-02-01T10:22:55.221 DEBUG:boto:Data: 2024-02-01T10:22:55.222 DEBUG:boto:Headers: {} 2024-02-01T10:22:55.222 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:55.222 DEBUG:boto:Port: 8000 2024-02-01T10:22:55.222 DEBUG:boto:Params: {} 2024-02-01T10:22:55.222 DEBUG:boto:Token: None 2024-02-01T10:22:55.222 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:55 GMT /vclfcn-16/ 2024-02-01T10:22:55.222 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tdgWY91d/aSko0GhasKPmcABFIg= 2024-02-01T10:22:55.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tdgWY91d/aSko0GhasKPmcABFIg=', 'Content-Length': '0'} 2024-02-01T10:22:55.226 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fec96a143784fee8-0065bb70ff-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:55.227 DEBUG:boto:b'vclfcn-16dummy1000falsedummy2024-02-01T10:19:58.958Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:55.227 DEBUG:boto:path=/vclfcn-16/dummy 2024-02-01T10:22:55.227 DEBUG:boto:auth_path=/vclfcn-16/dummy 2024-02-01T10:22:55.227 DEBUG:boto:Method: HEAD 2024-02-01T10:22:55.227 DEBUG:boto:Path: /vclfcn-16/dummy 2024-02-01T10:22:55.227 DEBUG:boto:Data: 2024-02-01T10:22:55.227 DEBUG:boto:Headers: {} 2024-02-01T10:22:55.228 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:55.228 DEBUG:boto:Port: 8001 2024-02-01T10:22:55.228 DEBUG:boto:Params: {} 2024-02-01T10:22:55.228 DEBUG:boto:Token: None 2024-02-01T10:22:55.228 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:55 GMT /vclfcn-16/dummy 2024-02-01T10:22:55.228 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:KxRBfMjL9LaT9zTZ6iJUOFGXGzM= 2024-02-01T10:22:55.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:KxRBfMjL9LaT9zTZ6iJUOFGXGzM=', 'Content-Length': '0'} 2024-02-01T10:22:55.232 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:19:58 GMT'), ('x-amz-version-id', 'rQRpBowIFAsvxVDn92H30shYnKoGt4n'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007d5e4d45fecef112-0065bb70ff-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:55.232 DEBUG:boto:path=/vclfcn-16/dummy 2024-02-01T10:22:55.232 DEBUG:boto:auth_path=/vclfcn-16/dummy 2024-02-01T10:22:55.233 DEBUG:boto:Method: HEAD 2024-02-01T10:22:55.233 DEBUG:boto:Path: /vclfcn-16/dummy 2024-02-01T10:22:55.233 DEBUG:boto:Data: 2024-02-01T10:22:55.233 DEBUG:boto:Headers: {} 2024-02-01T10:22:55.233 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:55.233 DEBUG:boto:Port: 8000 2024-02-01T10:22:55.233 DEBUG:boto:Params: {} 2024-02-01T10:22:55.233 DEBUG:boto:Token: None 2024-02-01T10:22:55.233 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:55 GMT /vclfcn-16/dummy 2024-02-01T10:22:55.234 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:KxRBfMjL9LaT9zTZ6iJUOFGXGzM= 2024-02-01T10:22:55.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:KxRBfMjL9LaT9zTZ6iJUOFGXGzM=', 'Content-Length': '0'} 2024-02-01T10:22:55.237 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:19:58 GMT'), ('x-amz-version-id', 'rQRpBowIFAsvxVDn92H30shYnKoGt4n'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000080859e7906ba0959-0065bb70ff-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:55.237 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-02-01T10:22:55.238 DEBUG:boto:path=/vclfcn-16/dummy 2024-02-01T10:22:55.238 DEBUG:boto:auth_path=/vclfcn-16/dummy 2024-02-01T10:22:55.238 DEBUG:boto:path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:55.238 DEBUG:boto:auth_path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:55.238 DEBUG:boto:Method: GET 2024-02-01T10:22:55.238 DEBUG:boto:Path: /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:55.238 DEBUG:boto:Data: 2024-02-01T10:22:55.238 DEBUG:boto:Headers: {} 2024-02-01T10:22:55.238 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:55.238 DEBUG:boto:Port: 8001 2024-02-01T10:22:55.238 DEBUG:boto:Params: {} 2024-02-01T10:22:55.239 DEBUG:boto:Token: None 2024-02-01T10:22:55.239 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:55 GMT /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:55.239 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:v7aqVQG6hiUnU2Gf8mlGHljKPfk= 2024-02-01T10:22:55.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:v7aqVQG6hiUnU2Gf8mlGHljKPfk=', 'Content-Length': '0'} 2024-02-01T10:22:55.242 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:19:58 GMT'), ('x-amz-version-id', 'rQRpBowIFAsvxVDn92H30shYnKoGt4n'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001b88a1efe05cfd44-0065bb70ff-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:55.242 DEBUG:boto:path=/vclfcn-16/dummy 2024-02-01T10:22:55.242 DEBUG:boto:auth_path=/vclfcn-16/dummy 2024-02-01T10:22:55.242 DEBUG:boto:path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:55.242 DEBUG:boto:auth_path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:55.243 DEBUG:boto:Method: GET 2024-02-01T10:22:55.243 DEBUG:boto:Path: /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:55.243 DEBUG:boto:Data: 2024-02-01T10:22:55.243 DEBUG:boto:Headers: {} 2024-02-01T10:22:55.243 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:55.243 DEBUG:boto:Port: 8000 2024-02-01T10:22:55.243 DEBUG:boto:Params: {} 2024-02-01T10:22:55.243 DEBUG:boto:Token: None 2024-02-01T10:22:55.243 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:55 GMT /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:55.243 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:v7aqVQG6hiUnU2Gf8mlGHljKPfk= 2024-02-01T10:22:55.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:v7aqVQG6hiUnU2Gf8mlGHljKPfk=', 'Content-Length': '0'} 2024-02-01T10:22:55.246 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:19:58 GMT'), ('x-amz-version-id', 'rQRpBowIFAsvxVDn92H30shYnKoGt4n'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009064e7835fa5962a-0065bb70ff-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:55.246 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:22:55.246 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:22:55.247 DEBUG:boto:path=/vclfcn-16/?prefix=dummy 2024-02-01T10:22:55.247 DEBUG:boto:auth_path=/vclfcn-16/?prefix=dummy 2024-02-01T10:22:55.247 DEBUG:boto:Method: GET 2024-02-01T10:22:55.247 DEBUG:boto:Path: /vclfcn-16/?prefix=dummy 2024-02-01T10:22:55.247 DEBUG:boto:Data: 2024-02-01T10:22:55.247 DEBUG:boto:Headers: {} 2024-02-01T10:22:55.247 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:55.247 DEBUG:boto:Port: 8001 2024-02-01T10:22:55.247 DEBUG:boto:Params: {} 2024-02-01T10:22:55.247 DEBUG:boto:Token: None 2024-02-01T10:22:55.248 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:55 GMT /vclfcn-16/ 2024-02-01T10:22:55.248 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tdgWY91d/aSko0GhasKPmcABFIg= 2024-02-01T10:22:55.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tdgWY91d/aSko0GhasKPmcABFIg=', 'Content-Length': '0'} 2024-02-01T10:22:55.252 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d1199f5b0df70846-0065bb70ff-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:55.252 DEBUG:boto:b'vclfcn-16dummy1000falsedummy2024-02-01T10:19:58.958Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:55.252 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:22:55.253 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:22:55.253 DEBUG:boto:path=/vclfcn-16/?prefix=dummy 2024-02-01T10:22:55.253 DEBUG:boto:auth_path=/vclfcn-16/?prefix=dummy 2024-02-01T10:22:55.253 DEBUG:boto:Method: GET 2024-02-01T10:22:55.253 DEBUG:boto:Path: /vclfcn-16/?prefix=dummy 2024-02-01T10:22:55.253 DEBUG:boto:Data: 2024-02-01T10:22:55.253 DEBUG:boto:Headers: {} 2024-02-01T10:22:55.253 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:55.253 DEBUG:boto:Port: 8000 2024-02-01T10:22:55.253 DEBUG:boto:Params: {} 2024-02-01T10:22:55.254 DEBUG:boto:Token: None 2024-02-01T10:22:55.254 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:55 GMT /vclfcn-16/ 2024-02-01T10:22:55.254 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tdgWY91d/aSko0GhasKPmcABFIg= 2024-02-01T10:22:55.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tdgWY91d/aSko0GhasKPmcABFIg=', 'Content-Length': '0'} 2024-02-01T10:22:55.258 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ac80c48fc1f41ff-0065bb70ff-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:55.258 DEBUG:boto:b'vclfcn-16dummy1000falsedummy2024-02-01T10:19:58.958Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:55.259 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-02-01T10:22:55.259 DEBUG:boto:path=/vclfcn-16/obj-test-zone1 2024-02-01T10:22:55.259 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1 2024-02-01T10:22:55.259 DEBUG:boto:path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:55.259 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:55.259 DEBUG:boto:Method: GET 2024-02-01T10:22:55.260 DEBUG:boto:Path: /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:55.260 DEBUG:boto:Data: 2024-02-01T10:22:55.260 DEBUG:boto:Headers: {} 2024-02-01T10:22:55.260 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:55.260 DEBUG:boto:Port: 8001 2024-02-01T10:22:55.260 DEBUG:boto:Params: {} 2024-02-01T10:22:55.260 DEBUG:boto:Token: None 2024-02-01T10:22:55.260 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:55 GMT /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:55.260 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:dzaZYpHTQ5ZzqqqEAOD2yQtyCfE= 2024-02-01T10:22:55.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dzaZYpHTQ5ZzqqqEAOD2yQtyCfE=', 'Content-Length': '0'} 2024-02-01T10:22:55.263 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:49 GMT'), ('x-amz-version-id', 'lS9-sMOvyR-VcD.47BkPLymGmIyfWmf'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000263fa3f89f4e39dc-0065bb70ff-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:55.304 DEBUG:boto:path=/vclfcn-16/obj-test-zone1 2024-02-01T10:22:55.304 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1 2024-02-01T10:22:55.305 DEBUG:boto:path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:55.305 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:55.305 DEBUG:boto:Method: GET 2024-02-01T10:22:55.305 DEBUG:boto:Path: /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:55.305 DEBUG:boto:Data: 2024-02-01T10:22:55.305 DEBUG:boto:Headers: {} 2024-02-01T10:22:55.305 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:55.305 DEBUG:boto:Port: 8000 2024-02-01T10:22:55.305 DEBUG:boto:Params: {} 2024-02-01T10:22:55.305 DEBUG:boto:Token: None 2024-02-01T10:22:55.306 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:55 GMT /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:55.306 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:dzaZYpHTQ5ZzqqqEAOD2yQtyCfE= 2024-02-01T10:22:55.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dzaZYpHTQ5ZzqqqEAOD2yQtyCfE=', 'Content-Length': '0'} 2024-02-01T10:22:55.309 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:49 GMT'), ('x-amz-version-id', 'lS9-sMOvyR-VcD.47BkPLymGmIyfWmf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000085d06f68d7450281-0065bb70ff-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:55.309 DEBUG:boto:path=/vclfcn-16/obj-test-zone1 2024-02-01T10:22:55.309 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1 2024-02-01T10:22:55.309 DEBUG:boto:path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:55.309 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:55.310 DEBUG:boto:Method: HEAD 2024-02-01T10:22:55.310 DEBUG:boto:Path: /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:55.310 DEBUG:boto:Data: 2024-02-01T10:22:55.310 DEBUG:boto:Headers: {} 2024-02-01T10:22:55.310 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:55.310 DEBUG:boto:Port: 8001 2024-02-01T10:22:55.310 DEBUG:boto:Params: {} 2024-02-01T10:22:55.310 DEBUG:boto:Token: None 2024-02-01T10:22:55.310 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:55 GMT /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:55.311 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:bur8HIKajKCRIW1FD4XcVzHb3XY= 2024-02-01T10:22:55.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:bur8HIKajKCRIW1FD4XcVzHb3XY=', 'Content-Length': '0'} 2024-02-01T10:22:55.313 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:49 GMT'), ('x-amz-version-id', 'lS9-sMOvyR-VcD.47BkPLymGmIyfWmf'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b0d2c2c5769a2027-0065bb70ff-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:55.313 DEBUG:boto:path=/vclfcn-16/obj-test-zone1 2024-02-01T10:22:55.314 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1 2024-02-01T10:22:55.314 DEBUG:boto:path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:55.314 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:55.314 DEBUG:boto:Method: HEAD 2024-02-01T10:22:55.314 DEBUG:boto:Path: /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:55.314 DEBUG:boto:Data: 2024-02-01T10:22:55.314 DEBUG:boto:Headers: {} 2024-02-01T10:22:55.314 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:55.314 DEBUG:boto:Port: 8000 2024-02-01T10:22:55.314 DEBUG:boto:Params: {} 2024-02-01T10:22:55.314 DEBUG:boto:Token: None 2024-02-01T10:22:55.315 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:55 GMT /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:55.315 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:bur8HIKajKCRIW1FD4XcVzHb3XY= 2024-02-01T10:22:55.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:bur8HIKajKCRIW1FD4XcVzHb3XY=', 'Content-Length': '0'} 2024-02-01T10:22:55.317 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:49 GMT'), ('x-amz-version-id', 'lS9-sMOvyR-VcD.47BkPLymGmIyfWmf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f2b00716ba869cb8-0065bb70ff-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:55.317 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-02-01T10:22:55.318 DEBUG:boto:path=/vclfcn-16/obj-test-zone1 2024-02-01T10:22:55.318 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1 2024-02-01T10:22:55.318 DEBUG:boto:path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:55.318 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:55.318 DEBUG:boto:Method: GET 2024-02-01T10:22:55.318 DEBUG:boto:Path: /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:55.318 DEBUG:boto:Data: 2024-02-01T10:22:55.318 DEBUG:boto:Headers: {} 2024-02-01T10:22:55.318 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:55.319 DEBUG:boto:Port: 8001 2024-02-01T10:22:55.319 DEBUG:boto:Params: {} 2024-02-01T10:22:55.319 DEBUG:boto:Token: None 2024-02-01T10:22:55.319 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:55 GMT /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:55.319 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:dzaZYpHTQ5ZzqqqEAOD2yQtyCfE= 2024-02-01T10:22:55.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dzaZYpHTQ5ZzqqqEAOD2yQtyCfE=', 'Content-Length': '0'} 2024-02-01T10:22:55.322 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:49 GMT'), ('x-amz-version-id', 'lS9-sMOvyR-VcD.47BkPLymGmIyfWmf'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000966be7223279dd4f-0065bb70ff-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:55.364 DEBUG:boto:path=/vclfcn-16/obj-test-zone1 2024-02-01T10:22:55.364 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1 2024-02-01T10:22:55.364 DEBUG:boto:path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:55.364 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:55.365 DEBUG:boto:Method: GET 2024-02-01T10:22:55.365 DEBUG:boto:Path: /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:55.365 DEBUG:boto:Data: 2024-02-01T10:22:55.365 DEBUG:boto:Headers: {} 2024-02-01T10:22:55.365 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:55.365 DEBUG:boto:Port: 8000 2024-02-01T10:22:55.365 DEBUG:boto:Params: {} 2024-02-01T10:22:55.365 DEBUG:boto:Token: None 2024-02-01T10:22:55.365 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:55 GMT /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:55.366 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:dzaZYpHTQ5ZzqqqEAOD2yQtyCfE= 2024-02-01T10:22:55.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dzaZYpHTQ5ZzqqqEAOD2yQtyCfE=', 'Content-Length': '0'} 2024-02-01T10:22:55.369 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:49 GMT'), ('x-amz-version-id', 'lS9-sMOvyR-VcD.47BkPLymGmIyfWmf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008f071c647b9800cd-0065bb70ff-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:55.369 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:22:55.369 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:22:55.369 DEBUG:boto:path=/vclfcn-16/?prefix=obj-test-zone1 2024-02-01T10:22:55.369 DEBUG:boto:auth_path=/vclfcn-16/?prefix=obj-test-zone1 2024-02-01T10:22:55.369 DEBUG:boto:Method: GET 2024-02-01T10:22:55.370 DEBUG:boto:Path: /vclfcn-16/?prefix=obj-test-zone1 2024-02-01T10:22:55.370 DEBUG:boto:Data: 2024-02-01T10:22:55.370 DEBUG:boto:Headers: {} 2024-02-01T10:22:55.370 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:55.370 DEBUG:boto:Port: 8001 2024-02-01T10:22:55.370 DEBUG:boto:Params: {} 2024-02-01T10:22:55.370 DEBUG:boto:Token: None 2024-02-01T10:22:55.370 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:55 GMT /vclfcn-16/ 2024-02-01T10:22:55.371 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tdgWY91d/aSko0GhasKPmcABFIg= 2024-02-01T10:22:55.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tdgWY91d/aSko0GhasKPmcABFIg=', 'Content-Length': '0'} 2024-02-01T10:22:55.375 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000056919b8d88b1becb-0065bb70ff-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:55.420 DEBUG:boto:b'vclfcn-16obj-test-zone11000falseobj-test-zone12024-02-01T10:20:49.965Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:55.421 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:22:55.421 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:22:55.421 DEBUG:boto:path=/vclfcn-16/?prefix=obj-test-zone1 2024-02-01T10:22:55.421 DEBUG:boto:auth_path=/vclfcn-16/?prefix=obj-test-zone1 2024-02-01T10:22:55.421 DEBUG:boto:Method: GET 2024-02-01T10:22:55.421 DEBUG:boto:Path: /vclfcn-16/?prefix=obj-test-zone1 2024-02-01T10:22:55.421 DEBUG:boto:Data: 2024-02-01T10:22:55.421 DEBUG:boto:Headers: {} 2024-02-01T10:22:55.421 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:55.422 DEBUG:boto:Port: 8000 2024-02-01T10:22:55.422 DEBUG:boto:Params: {} 2024-02-01T10:22:55.422 DEBUG:boto:Token: None 2024-02-01T10:22:55.422 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:55 GMT /vclfcn-16/ 2024-02-01T10:22:55.422 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tdgWY91d/aSko0GhasKPmcABFIg= 2024-02-01T10:22:55.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tdgWY91d/aSko0GhasKPmcABFIg=', 'Content-Length': '0'} 2024-02-01T10:22:55.426 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ab90980095af4b6-0065bb70ff-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:55.427 DEBUG:boto:b'vclfcn-16obj-test-zone11000falseobj-test-zone12024-02-01T10:20:49.965Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:55.427 DEBUG:boto:path=/vclfcn-16/obj-test-zone1 2024-02-01T10:22:55.427 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1 2024-02-01T10:22:55.427 DEBUG:boto:Method: HEAD 2024-02-01T10:22:55.427 DEBUG:boto:Path: /vclfcn-16/obj-test-zone1 2024-02-01T10:22:55.427 DEBUG:boto:Data: 2024-02-01T10:22:55.427 DEBUG:boto:Headers: {} 2024-02-01T10:22:55.428 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:55.428 DEBUG:boto:Port: 8001 2024-02-01T10:22:55.428 DEBUG:boto:Params: {} 2024-02-01T10:22:55.428 DEBUG:boto:Token: None 2024-02-01T10:22:55.428 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:55 GMT /vclfcn-16/obj-test-zone1 2024-02-01T10:22:55.428 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:s7oSz4Ge7dyRnN8jaC24P1e5ZEU= 2024-02-01T10:22:55.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:s7oSz4Ge7dyRnN8jaC24P1e5ZEU=', 'Content-Length': '0'} 2024-02-01T10:22:55.432 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:49 GMT'), ('x-amz-version-id', 'lS9-sMOvyR-VcD.47BkPLymGmIyfWmf'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000570d96d21e1c6e97-0065bb70ff-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:55.432 DEBUG:boto:path=/vclfcn-16/obj-test-zone1 2024-02-01T10:22:55.432 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1 2024-02-01T10:22:55.433 DEBUG:boto:Method: HEAD 2024-02-01T10:22:55.433 DEBUG:boto:Path: /vclfcn-16/obj-test-zone1 2024-02-01T10:22:55.433 DEBUG:boto:Data: 2024-02-01T10:22:55.433 DEBUG:boto:Headers: {} 2024-02-01T10:22:55.433 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:55.433 DEBUG:boto:Port: 8000 2024-02-01T10:22:55.433 DEBUG:boto:Params: {} 2024-02-01T10:22:55.433 DEBUG:boto:Token: None 2024-02-01T10:22:55.433 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:55 GMT /vclfcn-16/obj-test-zone1 2024-02-01T10:22:55.434 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:s7oSz4Ge7dyRnN8jaC24P1e5ZEU= 2024-02-01T10:22:55.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:s7oSz4Ge7dyRnN8jaC24P1e5ZEU=', 'Content-Length': '0'} 2024-02-01T10:22:55.437 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:49 GMT'), ('x-amz-version-id', 'lS9-sMOvyR-VcD.47BkPLymGmIyfWmf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a3bfe02a052eff4b-0065bb70ff-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:55.437 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-02-01T10:22:55.437 DEBUG:boto:path=/vclfcn-16/obj-test-zone1 2024-02-01T10:22:55.438 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1 2024-02-01T10:22:55.438 DEBUG:boto:path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:55.438 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:55.438 DEBUG:boto:Method: GET 2024-02-01T10:22:55.438 DEBUG:boto:Path: /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:55.438 DEBUG:boto:Data: 2024-02-01T10:22:55.438 DEBUG:boto:Headers: {} 2024-02-01T10:22:55.438 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:55.438 DEBUG:boto:Port: 8001 2024-02-01T10:22:55.438 DEBUG:boto:Params: {} 2024-02-01T10:22:55.439 DEBUG:boto:Token: None 2024-02-01T10:22:55.439 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:55 GMT /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:55.439 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:dzaZYpHTQ5ZzqqqEAOD2yQtyCfE= 2024-02-01T10:22:55.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dzaZYpHTQ5ZzqqqEAOD2yQtyCfE=', 'Content-Length': '0'} 2024-02-01T10:22:55.442 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:49 GMT'), ('x-amz-version-id', 'lS9-sMOvyR-VcD.47BkPLymGmIyfWmf'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b431789ee70900fb-0065bb70ff-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:55.488 DEBUG:boto:path=/vclfcn-16/obj-test-zone1 2024-02-01T10:22:55.488 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1 2024-02-01T10:22:55.488 DEBUG:boto:path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:55.489 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:55.489 DEBUG:boto:Method: GET 2024-02-01T10:22:55.489 DEBUG:boto:Path: /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:55.489 DEBUG:boto:Data: 2024-02-01T10:22:55.489 DEBUG:boto:Headers: {} 2024-02-01T10:22:55.489 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:55.489 DEBUG:boto:Port: 8000 2024-02-01T10:22:55.489 DEBUG:boto:Params: {} 2024-02-01T10:22:55.489 DEBUG:boto:Token: None 2024-02-01T10:22:55.490 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:55 GMT /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:55.490 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:dzaZYpHTQ5ZzqqqEAOD2yQtyCfE= 2024-02-01T10:22:55.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dzaZYpHTQ5ZzqqqEAOD2yQtyCfE=', 'Content-Length': '0'} 2024-02-01T10:22:55.493 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:49 GMT'), ('x-amz-version-id', 'lS9-sMOvyR-VcD.47BkPLymGmIyfWmf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001cec01b72119c4dc-0065bb70ff-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:55.493 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:22:55.493 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:22:55.493 DEBUG:boto:path=/vclfcn-16/?prefix=obj-test-zone1 2024-02-01T10:22:55.494 DEBUG:boto:auth_path=/vclfcn-16/?prefix=obj-test-zone1 2024-02-01T10:22:55.494 DEBUG:boto:Method: GET 2024-02-01T10:22:55.494 DEBUG:boto:Path: /vclfcn-16/?prefix=obj-test-zone1 2024-02-01T10:22:55.494 DEBUG:boto:Data: 2024-02-01T10:22:55.494 DEBUG:boto:Headers: {} 2024-02-01T10:22:55.494 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:55.494 DEBUG:boto:Port: 8001 2024-02-01T10:22:55.494 DEBUG:boto:Params: {} 2024-02-01T10:22:55.494 DEBUG:boto:Token: None 2024-02-01T10:22:55.494 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:55 GMT /vclfcn-16/ 2024-02-01T10:22:55.495 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tdgWY91d/aSko0GhasKPmcABFIg= 2024-02-01T10:22:55.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tdgWY91d/aSko0GhasKPmcABFIg=', 'Content-Length': '0'} 2024-02-01T10:22:55.499 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ee4e147ad716890-0065bb70ff-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:55.544 DEBUG:boto:b'vclfcn-16obj-test-zone11000falseobj-test-zone12024-02-01T10:20:49.965Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:55.545 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:22:55.545 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:22:55.545 DEBUG:boto:path=/vclfcn-16/?prefix=obj-test-zone1 2024-02-01T10:22:55.545 DEBUG:boto:auth_path=/vclfcn-16/?prefix=obj-test-zone1 2024-02-01T10:22:55.545 DEBUG:boto:Method: GET 2024-02-01T10:22:55.545 DEBUG:boto:Path: /vclfcn-16/?prefix=obj-test-zone1 2024-02-01T10:22:55.545 DEBUG:boto:Data: 2024-02-01T10:22:55.545 DEBUG:boto:Headers: {} 2024-02-01T10:22:55.545 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:55.545 DEBUG:boto:Port: 8000 2024-02-01T10:22:55.546 DEBUG:boto:Params: {} 2024-02-01T10:22:55.546 DEBUG:boto:Token: None 2024-02-01T10:22:55.546 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:55 GMT /vclfcn-16/ 2024-02-01T10:22:55.546 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tdgWY91d/aSko0GhasKPmcABFIg= 2024-02-01T10:22:55.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tdgWY91d/aSko0GhasKPmcABFIg=', 'Content-Length': '0'} 2024-02-01T10:22:55.550 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b619b5ca4b77f403-0065bb70ff-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:55.551 DEBUG:boto:b'vclfcn-16obj-test-zone11000falseobj-test-zone12024-02-01T10:20:49.965Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:55.551 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-02-01T10:22:55.551 DEBUG:boto:path=/vclfcn-16/obj-test-zone2 2024-02-01T10:22:55.551 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2 2024-02-01T10:22:55.552 DEBUG:boto:path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:55.552 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:55.552 DEBUG:boto:Method: GET 2024-02-01T10:22:55.552 DEBUG:boto:Path: /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:55.552 DEBUG:boto:Data: 2024-02-01T10:22:55.552 DEBUG:boto:Headers: {} 2024-02-01T10:22:55.552 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:55.552 DEBUG:boto:Port: 8001 2024-02-01T10:22:55.552 DEBUG:boto:Params: {} 2024-02-01T10:22:55.552 DEBUG:boto:Token: None 2024-02-01T10:22:55.553 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:55 GMT /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:55.553 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:48HpomwL9+9XSgqcUIw927j9dt8= 2024-02-01T10:22:55.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:48HpomwL9+9XSgqcUIw927j9dt8=', 'Content-Length': '0'} 2024-02-01T10:22:55.555 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'mE6urSxcrRA9wdyuToa41.w2aTia9p1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ae8f6cce5a25df6f-0065bb70ff-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:55.600 DEBUG:boto:path=/vclfcn-16/obj-test-zone2 2024-02-01T10:22:55.600 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2 2024-02-01T10:22:55.600 DEBUG:boto:path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:55.601 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:55.601 DEBUG:boto:Method: GET 2024-02-01T10:22:55.601 DEBUG:boto:Path: /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:55.601 DEBUG:boto:Data: 2024-02-01T10:22:55.601 DEBUG:boto:Headers: {} 2024-02-01T10:22:55.601 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:55.601 DEBUG:boto:Port: 8000 2024-02-01T10:22:55.601 DEBUG:boto:Params: {} 2024-02-01T10:22:55.601 DEBUG:boto:Token: None 2024-02-01T10:22:55.602 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:55 GMT /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:55.602 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:48HpomwL9+9XSgqcUIw927j9dt8= 2024-02-01T10:22:55.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:48HpomwL9+9XSgqcUIw927j9dt8=', 'Content-Length': '0'} 2024-02-01T10:22:55.605 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'mE6urSxcrRA9wdyuToa41.w2aTia9p1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000365f4d57fc211e81-0065bb70ff-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:55.605 DEBUG:boto:path=/vclfcn-16/obj-test-zone2 2024-02-01T10:22:55.605 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2 2024-02-01T10:22:55.605 DEBUG:boto:path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:55.606 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:55.606 DEBUG:boto:Method: HEAD 2024-02-01T10:22:55.606 DEBUG:boto:Path: /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:55.606 DEBUG:boto:Data: 2024-02-01T10:22:55.606 DEBUG:boto:Headers: {} 2024-02-01T10:22:55.606 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:55.606 DEBUG:boto:Port: 8001 2024-02-01T10:22:55.606 DEBUG:boto:Params: {} 2024-02-01T10:22:55.606 DEBUG:boto:Token: None 2024-02-01T10:22:55.606 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:55 GMT /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:55.606 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QqN8acuExm4bcuNlhD1rxigUvzI= 2024-02-01T10:22:55.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QqN8acuExm4bcuNlhD1rxigUvzI=', 'Content-Length': '0'} 2024-02-01T10:22:55.609 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'mE6urSxcrRA9wdyuToa41.w2aTia9p1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003669abd44d7a2409-0065bb70ff-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:55.609 DEBUG:boto:path=/vclfcn-16/obj-test-zone2 2024-02-01T10:22:55.609 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2 2024-02-01T10:22:55.610 DEBUG:boto:path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:55.610 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:55.610 DEBUG:boto:Method: HEAD 2024-02-01T10:22:55.610 DEBUG:boto:Path: /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:55.610 DEBUG:boto:Data: 2024-02-01T10:22:55.610 DEBUG:boto:Headers: {} 2024-02-01T10:22:55.610 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:55.610 DEBUG:boto:Port: 8000 2024-02-01T10:22:55.610 DEBUG:boto:Params: {} 2024-02-01T10:22:55.611 DEBUG:boto:Token: None 2024-02-01T10:22:55.611 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:55 GMT /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:55.611 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QqN8acuExm4bcuNlhD1rxigUvzI= 2024-02-01T10:22:55.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QqN8acuExm4bcuNlhD1rxigUvzI=', 'Content-Length': '0'} 2024-02-01T10:22:55.614 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'mE6urSxcrRA9wdyuToa41.w2aTia9p1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f0d09e20dd3c61b0-0065bb70ff-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:55.614 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-02-01T10:22:55.614 DEBUG:boto:path=/vclfcn-16/obj-test-zone2 2024-02-01T10:22:55.614 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2 2024-02-01T10:22:55.614 DEBUG:boto:path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:55.615 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:55.615 DEBUG:boto:Method: GET 2024-02-01T10:22:55.615 DEBUG:boto:Path: /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:55.615 DEBUG:boto:Data: 2024-02-01T10:22:55.615 DEBUG:boto:Headers: {} 2024-02-01T10:22:55.615 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:55.615 DEBUG:boto:Port: 8001 2024-02-01T10:22:55.615 DEBUG:boto:Params: {} 2024-02-01T10:22:55.615 DEBUG:boto:Token: None 2024-02-01T10:22:55.615 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:55 GMT /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:55.616 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:48HpomwL9+9XSgqcUIw927j9dt8= 2024-02-01T10:22:55.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:48HpomwL9+9XSgqcUIw927j9dt8=', 'Content-Length': '0'} 2024-02-01T10:22:55.618 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'mE6urSxcrRA9wdyuToa41.w2aTia9p1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004e15337c6b7aee2d-0065bb70ff-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:55.660 DEBUG:boto:path=/vclfcn-16/obj-test-zone2 2024-02-01T10:22:55.660 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2 2024-02-01T10:22:55.660 DEBUG:boto:path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:55.660 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:55.661 DEBUG:boto:Method: GET 2024-02-01T10:22:55.661 DEBUG:boto:Path: /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:55.661 DEBUG:boto:Data: 2024-02-01T10:22:55.661 DEBUG:boto:Headers: {} 2024-02-01T10:22:55.661 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:55.661 DEBUG:boto:Port: 8000 2024-02-01T10:22:55.661 DEBUG:boto:Params: {} 2024-02-01T10:22:55.661 DEBUG:boto:Token: None 2024-02-01T10:22:55.662 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:55 GMT /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:55.662 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:48HpomwL9+9XSgqcUIw927j9dt8= 2024-02-01T10:22:55.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:48HpomwL9+9XSgqcUIw927j9dt8=', 'Content-Length': '0'} 2024-02-01T10:22:55.666 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'mE6urSxcrRA9wdyuToa41.w2aTia9p1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a61372507520f44f-0065bb70ff-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:55.708 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:22:55.708 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:22:55.709 DEBUG:boto:path=/vclfcn-16/?prefix=obj-test-zone2 2024-02-01T10:22:55.709 DEBUG:boto:auth_path=/vclfcn-16/?prefix=obj-test-zone2 2024-02-01T10:22:55.709 DEBUG:boto:Method: GET 2024-02-01T10:22:55.709 DEBUG:boto:Path: /vclfcn-16/?prefix=obj-test-zone2 2024-02-01T10:22:55.709 DEBUG:boto:Data: 2024-02-01T10:22:55.709 DEBUG:boto:Headers: {} 2024-02-01T10:22:55.709 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:55.709 DEBUG:boto:Port: 8001 2024-02-01T10:22:55.709 DEBUG:boto:Params: {} 2024-02-01T10:22:55.710 DEBUG:boto:Token: None 2024-02-01T10:22:55.710 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:55 GMT /vclfcn-16/ 2024-02-01T10:22:55.710 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tdgWY91d/aSko0GhasKPmcABFIg= 2024-02-01T10:22:55.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tdgWY91d/aSko0GhasKPmcABFIg=', 'Content-Length': '0'} 2024-02-01T10:22:55.716 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058aa52db49a3a77e-0065bb70ff-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:55.717 DEBUG:boto:b'vclfcn-16obj-test-zone21000falseobj-test-zone22024-02-01T10:20:50.122Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:55.717 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:22:55.717 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:22:55.718 DEBUG:boto:path=/vclfcn-16/?prefix=obj-test-zone2 2024-02-01T10:22:55.718 DEBUG:boto:auth_path=/vclfcn-16/?prefix=obj-test-zone2 2024-02-01T10:22:55.718 DEBUG:boto:Method: GET 2024-02-01T10:22:55.718 DEBUG:boto:Path: /vclfcn-16/?prefix=obj-test-zone2 2024-02-01T10:22:55.718 DEBUG:boto:Data: 2024-02-01T10:22:55.718 DEBUG:boto:Headers: {} 2024-02-01T10:22:55.718 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:55.718 DEBUG:boto:Port: 8000 2024-02-01T10:22:55.718 DEBUG:boto:Params: {} 2024-02-01T10:22:55.718 DEBUG:boto:Token: None 2024-02-01T10:22:55.719 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:55 GMT /vclfcn-16/ 2024-02-01T10:22:55.719 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tdgWY91d/aSko0GhasKPmcABFIg= 2024-02-01T10:22:55.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tdgWY91d/aSko0GhasKPmcABFIg=', 'Content-Length': '0'} 2024-02-01T10:22:55.723 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9d618afa688b338-0065bb70ff-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:55.768 DEBUG:boto:b'vclfcn-16obj-test-zone21000falseobj-test-zone22024-02-01T10:20:50.122Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:55.769 DEBUG:boto:path=/vclfcn-16/obj-test-zone2 2024-02-01T10:22:55.769 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2 2024-02-01T10:22:55.769 DEBUG:boto:Method: HEAD 2024-02-01T10:22:55.769 DEBUG:boto:Path: /vclfcn-16/obj-test-zone2 2024-02-01T10:22:55.769 DEBUG:boto:Data: 2024-02-01T10:22:55.769 DEBUG:boto:Headers: {} 2024-02-01T10:22:55.769 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:55.769 DEBUG:boto:Port: 8001 2024-02-01T10:22:55.769 DEBUG:boto:Params: {} 2024-02-01T10:22:55.770 DEBUG:boto:Token: None 2024-02-01T10:22:55.770 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:55 GMT /vclfcn-16/obj-test-zone2 2024-02-01T10:22:55.770 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:b/UGPxO+7ynT79sCt4huUPdXWoo= 2024-02-01T10:22:55.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:b/UGPxO+7ynT79sCt4huUPdXWoo=', 'Content-Length': '0'} 2024-02-01T10:22:55.774 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'mE6urSxcrRA9wdyuToa41.w2aTia9p1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000019f6a820b955d8d2-0065bb70ff-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:55.774 DEBUG:boto:path=/vclfcn-16/obj-test-zone2 2024-02-01T10:22:55.774 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2 2024-02-01T10:22:55.774 DEBUG:boto:Method: HEAD 2024-02-01T10:22:55.775 DEBUG:boto:Path: /vclfcn-16/obj-test-zone2 2024-02-01T10:22:55.775 DEBUG:boto:Data: 2024-02-01T10:22:55.775 DEBUG:boto:Headers: {} 2024-02-01T10:22:55.775 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:55.775 DEBUG:boto:Port: 8000 2024-02-01T10:22:55.775 DEBUG:boto:Params: {} 2024-02-01T10:22:55.775 DEBUG:boto:Token: None 2024-02-01T10:22:55.775 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:55 GMT /vclfcn-16/obj-test-zone2 2024-02-01T10:22:55.775 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:b/UGPxO+7ynT79sCt4huUPdXWoo= 2024-02-01T10:22:55.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:b/UGPxO+7ynT79sCt4huUPdXWoo=', 'Content-Length': '0'} 2024-02-01T10:22:55.779 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'mE6urSxcrRA9wdyuToa41.w2aTia9p1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b7f2e7117877896e-0065bb70ff-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:55.779 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-02-01T10:22:55.780 DEBUG:boto:path=/vclfcn-16/obj-test-zone2 2024-02-01T10:22:55.780 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2 2024-02-01T10:22:55.780 DEBUG:boto:path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:55.780 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:55.780 DEBUG:boto:Method: GET 2024-02-01T10:22:55.780 DEBUG:boto:Path: /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:55.780 DEBUG:boto:Data: 2024-02-01T10:22:55.780 DEBUG:boto:Headers: {} 2024-02-01T10:22:55.780 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:55.781 DEBUG:boto:Port: 8001 2024-02-01T10:22:55.781 DEBUG:boto:Params: {} 2024-02-01T10:22:55.781 DEBUG:boto:Token: None 2024-02-01T10:22:55.781 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:55 GMT /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:55.781 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:48HpomwL9+9XSgqcUIw927j9dt8= 2024-02-01T10:22:55.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:48HpomwL9+9XSgqcUIw927j9dt8=', 'Content-Length': '0'} 2024-02-01T10:22:55.784 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'mE6urSxcrRA9wdyuToa41.w2aTia9p1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000006abf2286a9f01b3-0065bb70ff-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:55.824 DEBUG:boto:path=/vclfcn-16/obj-test-zone2 2024-02-01T10:22:55.824 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2 2024-02-01T10:22:55.825 DEBUG:boto:path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:55.825 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:55.825 DEBUG:boto:Method: GET 2024-02-01T10:22:55.825 DEBUG:boto:Path: /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:55.825 DEBUG:boto:Data: 2024-02-01T10:22:55.825 DEBUG:boto:Headers: {} 2024-02-01T10:22:55.825 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:55.825 DEBUG:boto:Port: 8000 2024-02-01T10:22:55.825 DEBUG:boto:Params: {} 2024-02-01T10:22:55.825 DEBUG:boto:Token: None 2024-02-01T10:22:55.826 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:55 GMT /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:55.826 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:48HpomwL9+9XSgqcUIw927j9dt8= 2024-02-01T10:22:55.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:48HpomwL9+9XSgqcUIw927j9dt8=', 'Content-Length': '0'} 2024-02-01T10:22:55.829 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'mE6urSxcrRA9wdyuToa41.w2aTia9p1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001b47bd9b98c400ba-0065bb70ff-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:55.872 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:22:55.872 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:22:55.872 DEBUG:boto:path=/vclfcn-16/?prefix=obj-test-zone2 2024-02-01T10:22:55.873 DEBUG:boto:auth_path=/vclfcn-16/?prefix=obj-test-zone2 2024-02-01T10:22:55.873 DEBUG:boto:Method: GET 2024-02-01T10:22:55.873 DEBUG:boto:Path: /vclfcn-16/?prefix=obj-test-zone2 2024-02-01T10:22:55.873 DEBUG:boto:Data: 2024-02-01T10:22:55.873 DEBUG:boto:Headers: {} 2024-02-01T10:22:55.873 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:55.873 DEBUG:boto:Port: 8001 2024-02-01T10:22:55.873 DEBUG:boto:Params: {} 2024-02-01T10:22:55.873 DEBUG:boto:Token: None 2024-02-01T10:22:55.873 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:55 GMT /vclfcn-16/ 2024-02-01T10:22:55.874 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tdgWY91d/aSko0GhasKPmcABFIg= 2024-02-01T10:22:55.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tdgWY91d/aSko0GhasKPmcABFIg=', 'Content-Length': '0'} 2024-02-01T10:22:55.878 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c50b65424a1f65ae-0065bb70ff-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:55.878 DEBUG:boto:b'vclfcn-16obj-test-zone21000falseobj-test-zone22024-02-01T10:20:50.122Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:55.878 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:22:55.879 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:22:55.879 DEBUG:boto:path=/vclfcn-16/?prefix=obj-test-zone2 2024-02-01T10:22:55.879 DEBUG:boto:auth_path=/vclfcn-16/?prefix=obj-test-zone2 2024-02-01T10:22:55.879 DEBUG:boto:Method: GET 2024-02-01T10:22:55.879 DEBUG:boto:Path: /vclfcn-16/?prefix=obj-test-zone2 2024-02-01T10:22:55.879 DEBUG:boto:Data: 2024-02-01T10:22:55.879 DEBUG:boto:Headers: {} 2024-02-01T10:22:55.879 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:55.879 DEBUG:boto:Port: 8000 2024-02-01T10:22:55.879 DEBUG:boto:Params: {} 2024-02-01T10:22:55.880 DEBUG:boto:Token: None 2024-02-01T10:22:55.880 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:55 GMT /vclfcn-16/ 2024-02-01T10:22:55.880 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tdgWY91d/aSko0GhasKPmcABFIg= 2024-02-01T10:22:55.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tdgWY91d/aSko0GhasKPmcABFIg=', 'Content-Length': '0'} 2024-02-01T10:22:55.884 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df2f649ec433e861-0065bb70ff-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:55.885 DEBUG:boto:b'vclfcn-16obj-test-zone21000falseobj-test-zone22024-02-01T10:20:50.122Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:55.885 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-02-01T10:22:55.885 DEBUG:boto:path=/vclfcn-16/obj-test-zone3 2024-02-01T10:22:55.885 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3 2024-02-01T10:22:55.885 DEBUG:boto:path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:55.886 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:55.886 DEBUG:boto:Method: GET 2024-02-01T10:22:55.886 DEBUG:boto:Path: /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:55.886 DEBUG:boto:Data: 2024-02-01T10:22:55.886 DEBUG:boto:Headers: {} 2024-02-01T10:22:55.886 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:55.886 DEBUG:boto:Port: 8001 2024-02-01T10:22:55.887 DEBUG:boto:Params: {} 2024-02-01T10:22:55.887 DEBUG:boto:Token: None 2024-02-01T10:22:55.887 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:55 GMT /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:55.887 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:s+TPeaJNuyaRH671M8maTsqcdno= 2024-02-01T10:22:55.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:s+TPeaJNuyaRH671M8maTsqcdno=', 'Content-Length': '0'} 2024-02-01T10:22:55.890 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004646768ddf09fc14-0065bb70ff-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:55.932 DEBUG:boto:path=/vclfcn-16/obj-test-zone3 2024-02-01T10:22:55.932 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3 2024-02-01T10:22:55.932 DEBUG:boto:path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:55.932 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:55.933 DEBUG:boto:Method: GET 2024-02-01T10:22:55.933 DEBUG:boto:Path: /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:55.933 DEBUG:boto:Data: 2024-02-01T10:22:55.933 DEBUG:boto:Headers: {} 2024-02-01T10:22:55.933 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:55.933 DEBUG:boto:Port: 8000 2024-02-01T10:22:55.933 DEBUG:boto:Params: {} 2024-02-01T10:22:55.933 DEBUG:boto:Token: None 2024-02-01T10:22:55.934 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:55 GMT /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:55.934 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:s+TPeaJNuyaRH671M8maTsqcdno= 2024-02-01T10:22:55.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:s+TPeaJNuyaRH671M8maTsqcdno=', 'Content-Length': '0'} 2024-02-01T10:22:55.937 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000058ea54cd9d234f59-0065bb70ff-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:55.980 DEBUG:boto:path=/vclfcn-16/obj-test-zone3 2024-02-01T10:22:55.980 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3 2024-02-01T10:22:55.980 DEBUG:boto:path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:55.981 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:55.981 DEBUG:boto:Method: HEAD 2024-02-01T10:22:55.981 DEBUG:boto:Path: /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:55.981 DEBUG:boto:Data: 2024-02-01T10:22:55.981 DEBUG:boto:Headers: {} 2024-02-01T10:22:55.981 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:55.981 DEBUG:boto:Port: 8001 2024-02-01T10:22:55.981 DEBUG:boto:Params: {} 2024-02-01T10:22:55.981 DEBUG:boto:Token: None 2024-02-01T10:22:55.981 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:55 GMT /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:55.982 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:VkdrixCy2JU2XiC1CrvaaWtaI+g= 2024-02-01T10:22:55.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:VkdrixCy2JU2XiC1CrvaaWtaI+g=', 'Content-Length': '0'} 2024-02-01T10:22:55.985 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006ff7f3c6bf6cc5f8-0065bb70ff-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:55.985 DEBUG:boto:path=/vclfcn-16/obj-test-zone3 2024-02-01T10:22:55.985 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3 2024-02-01T10:22:55.985 DEBUG:boto:path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:55.985 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:55.985 DEBUG:boto:Method: HEAD 2024-02-01T10:22:55.986 DEBUG:boto:Path: /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:55.986 DEBUG:boto:Data: 2024-02-01T10:22:55.986 DEBUG:boto:Headers: {} 2024-02-01T10:22:55.986 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:55.986 DEBUG:boto:Port: 8000 2024-02-01T10:22:55.986 DEBUG:boto:Params: {} 2024-02-01T10:22:55.986 DEBUG:boto:Token: None 2024-02-01T10:22:55.986 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:55 GMT /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:55.986 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:VkdrixCy2JU2XiC1CrvaaWtaI+g= 2024-02-01T10:22:55.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:VkdrixCy2JU2XiC1CrvaaWtaI+g=', 'Content-Length': '0'} 2024-02-01T10:22:55.989 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c4df532e3021efb9-0065bb70ff-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:55.989 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-02-01T10:22:55.989 DEBUG:boto:path=/vclfcn-16/obj-test-zone3 2024-02-01T10:22:55.989 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3 2024-02-01T10:22:55.990 DEBUG:boto:path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:55.990 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:55.990 DEBUG:boto:Method: GET 2024-02-01T10:22:55.990 DEBUG:boto:Path: /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:55.990 DEBUG:boto:Data: 2024-02-01T10:22:55.990 DEBUG:boto:Headers: {} 2024-02-01T10:22:55.990 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:55.990 DEBUG:boto:Port: 8001 2024-02-01T10:22:55.990 DEBUG:boto:Params: {} 2024-02-01T10:22:55.990 DEBUG:boto:Token: None 2024-02-01T10:22:55.991 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:55 GMT /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:55.991 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:s+TPeaJNuyaRH671M8maTsqcdno= 2024-02-01T10:22:55.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:s+TPeaJNuyaRH671M8maTsqcdno=', 'Content-Length': '0'} 2024-02-01T10:22:55.994 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003563b6d17560c71e-0065bb70ff-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:55.994 DEBUG:boto:path=/vclfcn-16/obj-test-zone3 2024-02-01T10:22:55.994 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3 2024-02-01T10:22:55.994 DEBUG:boto:path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:55.994 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:55.995 DEBUG:boto:Method: GET 2024-02-01T10:22:55.995 DEBUG:boto:Path: /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:55.995 DEBUG:boto:Data: 2024-02-01T10:22:55.995 DEBUG:boto:Headers: {} 2024-02-01T10:22:55.995 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:55.995 DEBUG:boto:Port: 8000 2024-02-01T10:22:55.995 DEBUG:boto:Params: {} 2024-02-01T10:22:55.995 DEBUG:boto:Token: None 2024-02-01T10:22:55.996 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:55 GMT /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:55.996 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:s+TPeaJNuyaRH671M8maTsqcdno= 2024-02-01T10:22:55.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:s+TPeaJNuyaRH671M8maTsqcdno=', 'Content-Length': '0'} 2024-02-01T10:22:55.999 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f8fe6d9aaba5d158-0065bb7100-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:56.040 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:22:56.040 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:22:56.041 DEBUG:boto:path=/vclfcn-16/?prefix=obj-test-zone3 2024-02-01T10:22:56.041 DEBUG:boto:auth_path=/vclfcn-16/?prefix=obj-test-zone3 2024-02-01T10:22:56.041 DEBUG:boto:Method: GET 2024-02-01T10:22:56.041 DEBUG:boto:Path: /vclfcn-16/?prefix=obj-test-zone3 2024-02-01T10:22:56.041 DEBUG:boto:Data: 2024-02-01T10:22:56.041 DEBUG:boto:Headers: {} 2024-02-01T10:22:56.041 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:56.041 DEBUG:boto:Port: 8001 2024-02-01T10:22:56.041 DEBUG:boto:Params: {} 2024-02-01T10:22:56.041 DEBUG:boto:Token: None 2024-02-01T10:22:56.042 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:56 GMT /vclfcn-16/ 2024-02-01T10:22:56.042 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sabhUtE7S7EotHtTTX812xuscbY= 2024-02-01T10:22:56.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sabhUtE7S7EotHtTTX812xuscbY=', 'Content-Length': '0'} 2024-02-01T10:22:56.046 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078df248320319cbd-0065bb7100-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:56.046 DEBUG:boto:b'vclfcn-16obj-test-zone31000falseobj-test-zone32024-02-01T10:20:50.297Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:56.047 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:22:56.047 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:22:56.047 DEBUG:boto:path=/vclfcn-16/?prefix=obj-test-zone3 2024-02-01T10:22:56.047 DEBUG:boto:auth_path=/vclfcn-16/?prefix=obj-test-zone3 2024-02-01T10:22:56.047 DEBUG:boto:Method: GET 2024-02-01T10:22:56.047 DEBUG:boto:Path: /vclfcn-16/?prefix=obj-test-zone3 2024-02-01T10:22:56.047 DEBUG:boto:Data: 2024-02-01T10:22:56.047 DEBUG:boto:Headers: {} 2024-02-01T10:22:56.047 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:56.047 DEBUG:boto:Port: 8000 2024-02-01T10:22:56.048 DEBUG:boto:Params: {} 2024-02-01T10:22:56.048 DEBUG:boto:Token: None 2024-02-01T10:22:56.048 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:56 GMT /vclfcn-16/ 2024-02-01T10:22:56.048 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sabhUtE7S7EotHtTTX812xuscbY= 2024-02-01T10:22:56.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sabhUtE7S7EotHtTTX812xuscbY=', 'Content-Length': '0'} 2024-02-01T10:22:56.052 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b95ba7f57b73f8c8-0065bb7100-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:56.100 DEBUG:boto:b'vclfcn-16obj-test-zone31000falseobj-test-zone32024-02-01T10:20:50.297Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:56.100 DEBUG:boto:path=/vclfcn-16/obj-test-zone3 2024-02-01T10:22:56.101 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3 2024-02-01T10:22:56.101 DEBUG:boto:Method: HEAD 2024-02-01T10:22:56.101 DEBUG:boto:Path: /vclfcn-16/obj-test-zone3 2024-02-01T10:22:56.101 DEBUG:boto:Data: 2024-02-01T10:22:56.101 DEBUG:boto:Headers: {} 2024-02-01T10:22:56.101 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:56.101 DEBUG:boto:Port: 8001 2024-02-01T10:22:56.101 DEBUG:boto:Params: {} 2024-02-01T10:22:56.101 DEBUG:boto:Token: None 2024-02-01T10:22:56.102 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:56 GMT /vclfcn-16/obj-test-zone3 2024-02-01T10:22:56.102 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:XltIUfWuHwAACYLk/BW5zDqb1yk= 2024-02-01T10:22:56.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:XltIUfWuHwAACYLk/BW5zDqb1yk=', 'Content-Length': '0'} 2024-02-01T10:22:56.106 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004b082328da597869-0065bb7100-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:56.107 DEBUG:boto:path=/vclfcn-16/obj-test-zone3 2024-02-01T10:22:56.107 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3 2024-02-01T10:22:56.107 DEBUG:boto:Method: HEAD 2024-02-01T10:22:56.107 DEBUG:boto:Path: /vclfcn-16/obj-test-zone3 2024-02-01T10:22:56.107 DEBUG:boto:Data: 2024-02-01T10:22:56.108 DEBUG:boto:Headers: {} 2024-02-01T10:22:56.108 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:56.108 DEBUG:boto:Port: 8000 2024-02-01T10:22:56.108 DEBUG:boto:Params: {} 2024-02-01T10:22:56.109 DEBUG:boto:Token: None 2024-02-01T10:22:56.109 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:56 GMT /vclfcn-16/obj-test-zone3 2024-02-01T10:22:56.109 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:XltIUfWuHwAACYLk/BW5zDqb1yk= 2024-02-01T10:22:56.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:XltIUfWuHwAACYLk/BW5zDqb1yk=', 'Content-Length': '0'} 2024-02-01T10:22:56.113 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bb9c8e4cd81556e3-0065bb7100-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:56.114 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-02-01T10:22:56.114 DEBUG:boto:path=/vclfcn-16/obj-test-zone3 2024-02-01T10:22:56.114 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3 2024-02-01T10:22:56.114 DEBUG:boto:path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:56.114 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:56.115 DEBUG:boto:Method: GET 2024-02-01T10:22:56.115 DEBUG:boto:Path: /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:56.115 DEBUG:boto:Data: 2024-02-01T10:22:56.115 DEBUG:boto:Headers: {} 2024-02-01T10:22:56.115 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:56.116 DEBUG:boto:Port: 8001 2024-02-01T10:22:56.116 DEBUG:boto:Params: {} 2024-02-01T10:22:56.116 DEBUG:boto:Token: None 2024-02-01T10:22:56.116 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:56 GMT /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:56.117 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1AtKpnQB1mj1n5nS+UjovLAaXQk= 2024-02-01T10:22:56.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1AtKpnQB1mj1n5nS+UjovLAaXQk=', 'Content-Length': '0'} 2024-02-01T10:22:56.120 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000049fcd93d4642551e-0065bb7100-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:56.120 DEBUG:boto:path=/vclfcn-16/obj-test-zone3 2024-02-01T10:22:56.121 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3 2024-02-01T10:22:56.121 DEBUG:boto:path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:56.121 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:56.121 DEBUG:boto:Method: GET 2024-02-01T10:22:56.121 DEBUG:boto:Path: /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:56.121 DEBUG:boto:Data: 2024-02-01T10:22:56.122 DEBUG:boto:Headers: {} 2024-02-01T10:22:56.122 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:56.122 DEBUG:boto:Port: 8000 2024-02-01T10:22:56.122 DEBUG:boto:Params: {} 2024-02-01T10:22:56.122 DEBUG:boto:Token: None 2024-02-01T10:22:56.122 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:56 GMT /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:56.123 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1AtKpnQB1mj1n5nS+UjovLAaXQk= 2024-02-01T10:22:56.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1AtKpnQB1mj1n5nS+UjovLAaXQk=', 'Content-Length': '0'} 2024-02-01T10:22:56.126 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ed75461b26c9da82-0065bb7100-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:56.172 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:22:56.172 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:22:56.172 DEBUG:boto:path=/vclfcn-16/?prefix=obj-test-zone3 2024-02-01T10:22:56.172 DEBUG:boto:auth_path=/vclfcn-16/?prefix=obj-test-zone3 2024-02-01T10:22:56.173 DEBUG:boto:Method: GET 2024-02-01T10:22:56.173 DEBUG:boto:Path: /vclfcn-16/?prefix=obj-test-zone3 2024-02-01T10:22:56.173 DEBUG:boto:Data: 2024-02-01T10:22:56.173 DEBUG:boto:Headers: {} 2024-02-01T10:22:56.173 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:56.173 DEBUG:boto:Port: 8001 2024-02-01T10:22:56.173 DEBUG:boto:Params: {} 2024-02-01T10:22:56.173 DEBUG:boto:Token: None 2024-02-01T10:22:56.173 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:56 GMT /vclfcn-16/ 2024-02-01T10:22:56.174 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sabhUtE7S7EotHtTTX812xuscbY= 2024-02-01T10:22:56.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sabhUtE7S7EotHtTTX812xuscbY=', 'Content-Length': '0'} 2024-02-01T10:22:56.178 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d842d0d534310c88-0065bb7100-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:56.178 DEBUG:boto:b'vclfcn-16obj-test-zone31000falseobj-test-zone32024-02-01T10:20:50.297Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:56.179 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:22:56.179 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:22:56.179 DEBUG:boto:path=/vclfcn-16/?prefix=obj-test-zone3 2024-02-01T10:22:56.179 DEBUG:boto:auth_path=/vclfcn-16/?prefix=obj-test-zone3 2024-02-01T10:22:56.179 DEBUG:boto:Method: GET 2024-02-01T10:22:56.179 DEBUG:boto:Path: /vclfcn-16/?prefix=obj-test-zone3 2024-02-01T10:22:56.179 DEBUG:boto:Data: 2024-02-01T10:22:56.180 DEBUG:boto:Headers: {} 2024-02-01T10:22:56.180 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:56.180 DEBUG:boto:Port: 8000 2024-02-01T10:22:56.180 DEBUG:boto:Params: {} 2024-02-01T10:22:56.180 DEBUG:boto:Token: None 2024-02-01T10:22:56.180 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:56 GMT /vclfcn-16/ 2024-02-01T10:22:56.180 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sabhUtE7S7EotHtTTX812xuscbY= 2024-02-01T10:22:56.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sabhUtE7S7EotHtTTX812xuscbY=', 'Content-Length': '0'} 2024-02-01T10:22:56.185 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000545132f56c8db88c-0065bb7100-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:56.232 DEBUG:boto:b'vclfcn-16obj-test-zone31000falseobj-test-zone32024-02-01T10:20:50.297Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:56.233 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-16 zones={test-zone3, test-zone1} 2024-02-01T10:22:56.233 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-16 zones={test-zone3, test-zone2} 2024-02-01T10:22:56.233 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:22:56.233 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:22:56.233 DEBUG:boto:Method: HEAD 2024-02-01T10:22:56.233 DEBUG:boto:Path: /vclfcn-16/ 2024-02-01T10:22:56.233 DEBUG:boto:Data: 2024-02-01T10:22:56.234 DEBUG:boto:Headers: {} 2024-02-01T10:22:56.234 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:56.234 DEBUG:boto:Port: 8001 2024-02-01T10:22:56.234 DEBUG:boto:Params: {} 2024-02-01T10:22:56.234 DEBUG:boto:Token: None 2024-02-01T10:22:56.234 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:56 GMT /vclfcn-16/ 2024-02-01T10:22:56.234 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cQpyTPI+dsx45ji3pjGKy5FLQbM= 2024-02-01T10:22:56.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cQpyTPI+dsx45ji3pjGKy5FLQbM=', 'Content-Length': '0'} 2024-02-01T10:22:56.238 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008e1e788b2c80567f-0065bb7100-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:22:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:56.238 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:22:56.239 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:22:56.239 DEBUG:boto:Method: HEAD 2024-02-01T10:22:56.239 DEBUG:boto:Path: /vclfcn-16/ 2024-02-01T10:22:56.239 DEBUG:boto:Data: 2024-02-01T10:22:56.239 DEBUG:boto:Headers: {} 2024-02-01T10:22:56.239 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:56.239 DEBUG:boto:Port: 8000 2024-02-01T10:22:56.239 DEBUG:boto:Params: {} 2024-02-01T10:22:56.239 DEBUG:boto:Token: None 2024-02-01T10:22:56.240 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:56 GMT /vclfcn-16/ 2024-02-01T10:22:56.240 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cQpyTPI+dsx45ji3pjGKy5FLQbM= 2024-02-01T10:22:56.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cQpyTPI+dsx45ji3pjGKy5FLQbM=', 'Content-Length': '0'} 2024-02-01T10:22:56.244 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f82c16444c274dff-0065bb7100-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:22:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:56.244 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:22:56.244 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:22:56.245 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:22:56.245 DEBUG:boto:path=/vclfcn-16/?versions&max-keys=999 2024-02-01T10:22:56.245 DEBUG:boto:auth_path=/vclfcn-16/?versions&max-keys=999 2024-02-01T10:22:56.245 DEBUG:boto:Method: GET 2024-02-01T10:22:56.245 DEBUG:boto:Path: /vclfcn-16/?versions&max-keys=999 2024-02-01T10:22:56.245 DEBUG:boto:Data: 2024-02-01T10:22:56.245 DEBUG:boto:Headers: {} 2024-02-01T10:22:56.245 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:56.245 DEBUG:boto:Port: 8001 2024-02-01T10:22:56.245 DEBUG:boto:Params: {} 2024-02-01T10:22:56.246 DEBUG:boto:Token: None 2024-02-01T10:22:56.246 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:56 GMT /vclfcn-16/?versions 2024-02-01T10:22:56.246 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:TGuya+2/muHKT+y5VyzBRPqIrjY= 2024-02-01T10:22:56.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:TGuya+2/muHKT+y5VyzBRPqIrjY=', 'Content-Length': '0'} 2024-02-01T10:22:56.250 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da7cb835fa545235-0065bb7100-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:56.250 DEBUG:boto:b'vclfcn-16999falsedummyrQRpBowIFAsvxVDn92H30shYnKoGt4ntrue2024-02-01T10:19:58.958Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1lS9-sMOvyR-VcD.47BkPLymGmIyfWmftrue2024-02-01T10:20:49.965Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2mE6urSxcrRA9wdyuToa41.w2aTia9p1true2024-02-01T10:20:50.122Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjPtrue2024-02-01T10:20:50.297Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:56.251 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-02-01T10:22:56.251 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-02-01T10:22:56.251 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-02-01T10:22:56.251 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-02-01T10:22:56.251 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:22:56.251 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:22:56.251 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:22:56.252 DEBUG:boto:path=/vclfcn-16/?versions&max-keys=999 2024-02-01T10:22:56.252 DEBUG:boto:auth_path=/vclfcn-16/?versions&max-keys=999 2024-02-01T10:22:56.252 DEBUG:boto:Method: GET 2024-02-01T10:22:56.252 DEBUG:boto:Path: /vclfcn-16/?versions&max-keys=999 2024-02-01T10:22:56.252 DEBUG:boto:Data: 2024-02-01T10:22:56.252 DEBUG:boto:Headers: {} 2024-02-01T10:22:56.252 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:56.252 DEBUG:boto:Port: 8000 2024-02-01T10:22:56.252 DEBUG:boto:Params: {} 2024-02-01T10:22:56.253 DEBUG:boto:Token: None 2024-02-01T10:22:56.253 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:56 GMT /vclfcn-16/?versions 2024-02-01T10:22:56.253 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:TGuya+2/muHKT+y5VyzBRPqIrjY= 2024-02-01T10:22:56.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:TGuya+2/muHKT+y5VyzBRPqIrjY=', 'Content-Length': '0'} 2024-02-01T10:22:56.258 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3c452f8ed382c8f-0065bb7100-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:56.300 DEBUG:boto:b'vclfcn-16999falsedummyrQRpBowIFAsvxVDn92H30shYnKoGt4ntrue2024-02-01T10:19:58.958Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1lS9-sMOvyR-VcD.47BkPLymGmIyfWmftrue2024-02-01T10:20:49.965Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2mE6urSxcrRA9wdyuToa41.w2aTia9p1true2024-02-01T10:20:50.122Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjPtrue2024-02-01T10:20:50.297Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:56.301 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-02-01T10:22:56.301 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-02-01T10:22:56.301 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-02-01T10:22:56.301 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-02-01T10:22:56.302 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:22:56.302 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:22:56.302 DEBUG:boto:path=/vclfcn-16/?versions&max-keys=999 2024-02-01T10:22:56.302 DEBUG:boto:auth_path=/vclfcn-16/?versions&max-keys=999 2024-02-01T10:22:56.302 DEBUG:boto:Method: GET 2024-02-01T10:22:56.302 DEBUG:boto:Path: /vclfcn-16/?versions&max-keys=999 2024-02-01T10:22:56.302 DEBUG:boto:Data: 2024-02-01T10:22:56.302 DEBUG:boto:Headers: {} 2024-02-01T10:22:56.302 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:56.302 DEBUG:boto:Port: 8001 2024-02-01T10:22:56.303 DEBUG:boto:Params: {} 2024-02-01T10:22:56.303 DEBUG:boto:Token: None 2024-02-01T10:22:56.303 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:56 GMT /vclfcn-16/?versions 2024-02-01T10:22:56.303 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:TGuya+2/muHKT+y5VyzBRPqIrjY= 2024-02-01T10:22:56.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:TGuya+2/muHKT+y5VyzBRPqIrjY=', 'Content-Length': '0'} 2024-02-01T10:22:56.308 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000402a11881326058-0065bb7100-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:56.308 DEBUG:boto:b'vclfcn-16999falsedummyrQRpBowIFAsvxVDn92H30shYnKoGt4ntrue2024-02-01T10:19:58.958Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1lS9-sMOvyR-VcD.47BkPLymGmIyfWmftrue2024-02-01T10:20:49.965Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2mE6urSxcrRA9wdyuToa41.w2aTia9p1true2024-02-01T10:20:50.122Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjPtrue2024-02-01T10:20:50.297Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:56.309 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:22:56.309 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:22:56.309 DEBUG:boto:path=/vclfcn-16/?versions&max-keys=999 2024-02-01T10:22:56.309 DEBUG:boto:auth_path=/vclfcn-16/?versions&max-keys=999 2024-02-01T10:22:56.309 DEBUG:boto:Method: GET 2024-02-01T10:22:56.310 DEBUG:boto:Path: /vclfcn-16/?versions&max-keys=999 2024-02-01T10:22:56.310 DEBUG:boto:Data: 2024-02-01T10:22:56.310 DEBUG:boto:Headers: {} 2024-02-01T10:22:56.310 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:56.310 DEBUG:boto:Port: 8000 2024-02-01T10:22:56.310 DEBUG:boto:Params: {} 2024-02-01T10:22:56.310 DEBUG:boto:Token: None 2024-02-01T10:22:56.311 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:56 GMT /vclfcn-16/?versions 2024-02-01T10:22:56.311 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:TGuya+2/muHKT+y5VyzBRPqIrjY= 2024-02-01T10:22:56.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:TGuya+2/muHKT+y5VyzBRPqIrjY=', 'Content-Length': '0'} 2024-02-01T10:22:56.315 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c543923e99619c65-0065bb7100-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:56.315 DEBUG:boto:b'vclfcn-16999falsedummyrQRpBowIFAsvxVDn92H30shYnKoGt4ntrue2024-02-01T10:19:58.958Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1lS9-sMOvyR-VcD.47BkPLymGmIyfWmftrue2024-02-01T10:20:49.965Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2mE6urSxcrRA9wdyuToa41.w2aTia9p1true2024-02-01T10:20:50.122Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjPtrue2024-02-01T10:20:50.297Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:56.316 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-02-01T10:22:56.316 DEBUG:boto:path=/vclfcn-16/dummy 2024-02-01T10:22:56.316 DEBUG:boto:auth_path=/vclfcn-16/dummy 2024-02-01T10:22:56.316 DEBUG:boto:path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:56.316 DEBUG:boto:auth_path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:56.316 DEBUG:boto:Method: GET 2024-02-01T10:22:56.316 DEBUG:boto:Path: /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:56.316 DEBUG:boto:Data: 2024-02-01T10:22:56.316 DEBUG:boto:Headers: {} 2024-02-01T10:22:56.316 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:56.317 DEBUG:boto:Port: 8001 2024-02-01T10:22:56.317 DEBUG:boto:Params: {} 2024-02-01T10:22:56.317 DEBUG:boto:Token: None 2024-02-01T10:22:56.317 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:56 GMT /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:56.317 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:O0vkAnhT165P4M9Tpyv7I/RRGxo= 2024-02-01T10:22:56.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:O0vkAnhT165P4M9Tpyv7I/RRGxo=', 'Content-Length': '0'} 2024-02-01T10:22:56.320 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:19:58 GMT'), ('x-amz-version-id', 'rQRpBowIFAsvxVDn92H30shYnKoGt4n'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000072c9404faa170376-0065bb7100-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:56.320 DEBUG:boto:path=/vclfcn-16/dummy 2024-02-01T10:22:56.320 DEBUG:boto:auth_path=/vclfcn-16/dummy 2024-02-01T10:22:56.320 DEBUG:boto:path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:56.321 DEBUG:boto:auth_path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:56.321 DEBUG:boto:Method: GET 2024-02-01T10:22:56.321 DEBUG:boto:Path: /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:56.321 DEBUG:boto:Data: 2024-02-01T10:22:56.321 DEBUG:boto:Headers: {} 2024-02-01T10:22:56.321 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:56.321 DEBUG:boto:Port: 8000 2024-02-01T10:22:56.321 DEBUG:boto:Params: {} 2024-02-01T10:22:56.321 DEBUG:boto:Token: None 2024-02-01T10:22:56.321 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:56 GMT /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:56.322 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:O0vkAnhT165P4M9Tpyv7I/RRGxo= 2024-02-01T10:22:56.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:O0vkAnhT165P4M9Tpyv7I/RRGxo=', 'Content-Length': '0'} 2024-02-01T10:22:56.324 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:19:58 GMT'), ('x-amz-version-id', 'rQRpBowIFAsvxVDn92H30shYnKoGt4n'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f61a75b42de53240-0065bb7100-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:56.325 DEBUG:boto:path=/vclfcn-16/dummy 2024-02-01T10:22:56.325 DEBUG:boto:auth_path=/vclfcn-16/dummy 2024-02-01T10:22:56.325 DEBUG:boto:path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:56.325 DEBUG:boto:auth_path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:56.325 DEBUG:boto:Method: HEAD 2024-02-01T10:22:56.326 DEBUG:boto:Path: /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:56.326 DEBUG:boto:Data: 2024-02-01T10:22:56.326 DEBUG:boto:Headers: {} 2024-02-01T10:22:56.326 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:56.326 DEBUG:boto:Port: 8001 2024-02-01T10:22:56.326 DEBUG:boto:Params: {} 2024-02-01T10:22:56.326 DEBUG:boto:Token: None 2024-02-01T10:22:56.326 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:56 GMT /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:56.327 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sDVKgPGaTUutzfCAZ/oh/brFBmg= 2024-02-01T10:22:56.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sDVKgPGaTUutzfCAZ/oh/brFBmg=', 'Content-Length': '0'} 2024-02-01T10:22:56.329 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:19:58 GMT'), ('x-amz-version-id', 'rQRpBowIFAsvxVDn92H30shYnKoGt4n'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000777efb528551a029-0065bb7100-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:56.330 DEBUG:boto:path=/vclfcn-16/dummy 2024-02-01T10:22:56.330 DEBUG:boto:auth_path=/vclfcn-16/dummy 2024-02-01T10:22:56.330 DEBUG:boto:path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:56.330 DEBUG:boto:auth_path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:56.330 DEBUG:boto:Method: HEAD 2024-02-01T10:22:56.330 DEBUG:boto:Path: /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:56.330 DEBUG:boto:Data: 2024-02-01T10:22:56.330 DEBUG:boto:Headers: {} 2024-02-01T10:22:56.330 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:56.330 DEBUG:boto:Port: 8000 2024-02-01T10:22:56.330 DEBUG:boto:Params: {} 2024-02-01T10:22:56.331 DEBUG:boto:Token: None 2024-02-01T10:22:56.331 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:56 GMT /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:56.331 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sDVKgPGaTUutzfCAZ/oh/brFBmg= 2024-02-01T10:22:56.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sDVKgPGaTUutzfCAZ/oh/brFBmg=', 'Content-Length': '0'} 2024-02-01T10:22:56.334 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:19:58 GMT'), ('x-amz-version-id', 'rQRpBowIFAsvxVDn92H30shYnKoGt4n'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007e0738852004709b-0065bb7100-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:56.334 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-02-01T10:22:56.334 DEBUG:boto:path=/vclfcn-16/dummy 2024-02-01T10:22:56.334 DEBUG:boto:auth_path=/vclfcn-16/dummy 2024-02-01T10:22:56.334 DEBUG:boto:path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:56.334 DEBUG:boto:auth_path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:56.335 DEBUG:boto:Method: GET 2024-02-01T10:22:56.335 DEBUG:boto:Path: /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:56.335 DEBUG:boto:Data: 2024-02-01T10:22:56.335 DEBUG:boto:Headers: {} 2024-02-01T10:22:56.335 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:56.335 DEBUG:boto:Port: 8001 2024-02-01T10:22:56.335 DEBUG:boto:Params: {} 2024-02-01T10:22:56.335 DEBUG:boto:Token: None 2024-02-01T10:22:56.335 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:56 GMT /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:56.336 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:O0vkAnhT165P4M9Tpyv7I/RRGxo= 2024-02-01T10:22:56.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:O0vkAnhT165P4M9Tpyv7I/RRGxo=', 'Content-Length': '0'} 2024-02-01T10:22:56.338 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:19:58 GMT'), ('x-amz-version-id', 'rQRpBowIFAsvxVDn92H30shYnKoGt4n'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fa5687d8248ae100-0065bb7100-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:56.339 DEBUG:boto:path=/vclfcn-16/dummy 2024-02-01T10:22:56.339 DEBUG:boto:auth_path=/vclfcn-16/dummy 2024-02-01T10:22:56.339 DEBUG:boto:path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:56.339 DEBUG:boto:auth_path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:56.339 DEBUG:boto:Method: GET 2024-02-01T10:22:56.339 DEBUG:boto:Path: /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:56.339 DEBUG:boto:Data: 2024-02-01T10:22:56.339 DEBUG:boto:Headers: {} 2024-02-01T10:22:56.339 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:56.339 DEBUG:boto:Port: 8000 2024-02-01T10:22:56.340 DEBUG:boto:Params: {} 2024-02-01T10:22:56.340 DEBUG:boto:Token: None 2024-02-01T10:22:56.340 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:56 GMT /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:56.340 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:O0vkAnhT165P4M9Tpyv7I/RRGxo= 2024-02-01T10:22:56.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:O0vkAnhT165P4M9Tpyv7I/RRGxo=', 'Content-Length': '0'} 2024-02-01T10:22:56.343 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:19:58 GMT'), ('x-amz-version-id', 'rQRpBowIFAsvxVDn92H30shYnKoGt4n'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000010dacd7bd6662cc8-0065bb7100-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:56.343 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:22:56.343 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:22:56.343 DEBUG:boto:path=/vclfcn-16/?prefix=dummy 2024-02-01T10:22:56.343 DEBUG:boto:auth_path=/vclfcn-16/?prefix=dummy 2024-02-01T10:22:56.343 DEBUG:boto:Method: GET 2024-02-01T10:22:56.344 DEBUG:boto:Path: /vclfcn-16/?prefix=dummy 2024-02-01T10:22:56.344 DEBUG:boto:Data: 2024-02-01T10:22:56.344 DEBUG:boto:Headers: {} 2024-02-01T10:22:56.344 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:56.344 DEBUG:boto:Port: 8001 2024-02-01T10:22:56.344 DEBUG:boto:Params: {} 2024-02-01T10:22:56.344 DEBUG:boto:Token: None 2024-02-01T10:22:56.344 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:56 GMT /vclfcn-16/ 2024-02-01T10:22:56.345 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sabhUtE7S7EotHtTTX812xuscbY= 2024-02-01T10:22:56.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sabhUtE7S7EotHtTTX812xuscbY=', 'Content-Length': '0'} 2024-02-01T10:22:56.349 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c81ec9919a2044c-0065bb7100-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:56.392 DEBUG:boto:b'vclfcn-16dummy1000falsedummy2024-02-01T10:19:58.958Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:56.393 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:22:56.393 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:22:56.393 DEBUG:boto:path=/vclfcn-16/?prefix=dummy 2024-02-01T10:22:56.393 DEBUG:boto:auth_path=/vclfcn-16/?prefix=dummy 2024-02-01T10:22:56.393 DEBUG:boto:Method: GET 2024-02-01T10:22:56.393 DEBUG:boto:Path: /vclfcn-16/?prefix=dummy 2024-02-01T10:22:56.393 DEBUG:boto:Data: 2024-02-01T10:22:56.393 DEBUG:boto:Headers: {} 2024-02-01T10:22:56.393 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:56.393 DEBUG:boto:Port: 8000 2024-02-01T10:22:56.393 DEBUG:boto:Params: {} 2024-02-01T10:22:56.394 DEBUG:boto:Token: None 2024-02-01T10:22:56.394 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:56 GMT /vclfcn-16/ 2024-02-01T10:22:56.394 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sabhUtE7S7EotHtTTX812xuscbY= 2024-02-01T10:22:56.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sabhUtE7S7EotHtTTX812xuscbY=', 'Content-Length': '0'} 2024-02-01T10:22:56.398 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003be568ecb5025c3e-0065bb7100-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:56.399 DEBUG:boto:b'vclfcn-16dummy1000falsedummy2024-02-01T10:19:58.958Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:56.399 DEBUG:boto:path=/vclfcn-16/dummy 2024-02-01T10:22:56.399 DEBUG:boto:auth_path=/vclfcn-16/dummy 2024-02-01T10:22:56.399 DEBUG:boto:Method: HEAD 2024-02-01T10:22:56.399 DEBUG:boto:Path: /vclfcn-16/dummy 2024-02-01T10:22:56.399 DEBUG:boto:Data: 2024-02-01T10:22:56.399 DEBUG:boto:Headers: {} 2024-02-01T10:22:56.400 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:56.400 DEBUG:boto:Port: 8001 2024-02-01T10:22:56.400 DEBUG:boto:Params: {} 2024-02-01T10:22:56.400 DEBUG:boto:Token: None 2024-02-01T10:22:56.400 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:56 GMT /vclfcn-16/dummy 2024-02-01T10:22:56.400 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Y4cDh+7wt17a+C3UERMhTHWE3nA= 2024-02-01T10:22:56.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Y4cDh+7wt17a+C3UERMhTHWE3nA=', 'Content-Length': '0'} 2024-02-01T10:22:56.404 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:19:58 GMT'), ('x-amz-version-id', 'rQRpBowIFAsvxVDn92H30shYnKoGt4n'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c17a52d185c07c01-0065bb7100-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:56.404 DEBUG:boto:path=/vclfcn-16/dummy 2024-02-01T10:22:56.404 DEBUG:boto:auth_path=/vclfcn-16/dummy 2024-02-01T10:22:56.404 DEBUG:boto:Method: HEAD 2024-02-01T10:22:56.404 DEBUG:boto:Path: /vclfcn-16/dummy 2024-02-01T10:22:56.404 DEBUG:boto:Data: 2024-02-01T10:22:56.405 DEBUG:boto:Headers: {} 2024-02-01T10:22:56.405 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:56.405 DEBUG:boto:Port: 8000 2024-02-01T10:22:56.405 DEBUG:boto:Params: {} 2024-02-01T10:22:56.405 DEBUG:boto:Token: None 2024-02-01T10:22:56.405 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:56 GMT /vclfcn-16/dummy 2024-02-01T10:22:56.405 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Y4cDh+7wt17a+C3UERMhTHWE3nA= 2024-02-01T10:22:56.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Y4cDh+7wt17a+C3UERMhTHWE3nA=', 'Content-Length': '0'} 2024-02-01T10:22:56.409 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:19:58 GMT'), ('x-amz-version-id', 'rQRpBowIFAsvxVDn92H30shYnKoGt4n'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001cd9d687cfbffa8e-0065bb7100-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:56.409 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-02-01T10:22:56.409 DEBUG:boto:path=/vclfcn-16/dummy 2024-02-01T10:22:56.409 DEBUG:boto:auth_path=/vclfcn-16/dummy 2024-02-01T10:22:56.409 DEBUG:boto:path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:56.409 DEBUG:boto:auth_path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:56.410 DEBUG:boto:Method: GET 2024-02-01T10:22:56.410 DEBUG:boto:Path: /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:56.410 DEBUG:boto:Data: 2024-02-01T10:22:56.410 DEBUG:boto:Headers: {} 2024-02-01T10:22:56.410 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:56.410 DEBUG:boto:Port: 8001 2024-02-01T10:22:56.410 DEBUG:boto:Params: {} 2024-02-01T10:22:56.410 DEBUG:boto:Token: None 2024-02-01T10:22:56.410 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:56 GMT /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:56.410 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:O0vkAnhT165P4M9Tpyv7I/RRGxo= 2024-02-01T10:22:56.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:O0vkAnhT165P4M9Tpyv7I/RRGxo=', 'Content-Length': '0'} 2024-02-01T10:22:56.413 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:19:58 GMT'), ('x-amz-version-id', 'rQRpBowIFAsvxVDn92H30shYnKoGt4n'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000136f675e201950c1-0065bb7100-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:56.413 DEBUG:boto:path=/vclfcn-16/dummy 2024-02-01T10:22:56.414 DEBUG:boto:auth_path=/vclfcn-16/dummy 2024-02-01T10:22:56.414 DEBUG:boto:path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:56.414 DEBUG:boto:auth_path=/vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:56.414 DEBUG:boto:Method: GET 2024-02-01T10:22:56.414 DEBUG:boto:Path: /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:56.414 DEBUG:boto:Data: 2024-02-01T10:22:56.414 DEBUG:boto:Headers: {} 2024-02-01T10:22:56.414 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:56.414 DEBUG:boto:Port: 8000 2024-02-01T10:22:56.414 DEBUG:boto:Params: {} 2024-02-01T10:22:56.414 DEBUG:boto:Token: None 2024-02-01T10:22:56.414 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:56 GMT /vclfcn-16/dummy?versionId=rQRpBowIFAsvxVDn92H30shYnKoGt4n 2024-02-01T10:22:56.415 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:O0vkAnhT165P4M9Tpyv7I/RRGxo= 2024-02-01T10:22:56.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:O0vkAnhT165P4M9Tpyv7I/RRGxo=', 'Content-Length': '0'} 2024-02-01T10:22:56.417 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:19:58 GMT'), ('x-amz-version-id', 'rQRpBowIFAsvxVDn92H30shYnKoGt4n'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009ace13e433c217e2-0065bb7100-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:56.418 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:22:56.418 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:22:56.418 DEBUG:boto:path=/vclfcn-16/?prefix=dummy 2024-02-01T10:22:56.418 DEBUG:boto:auth_path=/vclfcn-16/?prefix=dummy 2024-02-01T10:22:56.418 DEBUG:boto:Method: GET 2024-02-01T10:22:56.418 DEBUG:boto:Path: /vclfcn-16/?prefix=dummy 2024-02-01T10:22:56.418 DEBUG:boto:Data: 2024-02-01T10:22:56.418 DEBUG:boto:Headers: {} 2024-02-01T10:22:56.419 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:56.419 DEBUG:boto:Port: 8001 2024-02-01T10:22:56.419 DEBUG:boto:Params: {} 2024-02-01T10:22:56.419 DEBUG:boto:Token: None 2024-02-01T10:22:56.419 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:56 GMT /vclfcn-16/ 2024-02-01T10:22:56.419 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sabhUtE7S7EotHtTTX812xuscbY= 2024-02-01T10:22:56.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sabhUtE7S7EotHtTTX812xuscbY=', 'Content-Length': '0'} 2024-02-01T10:22:56.423 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b6ba7a043e75cbc-0065bb7100-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:56.423 DEBUG:boto:b'vclfcn-16dummy1000falsedummy2024-02-01T10:19:58.958Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:56.424 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:22:56.424 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:22:56.424 DEBUG:boto:path=/vclfcn-16/?prefix=dummy 2024-02-01T10:22:56.424 DEBUG:boto:auth_path=/vclfcn-16/?prefix=dummy 2024-02-01T10:22:56.424 DEBUG:boto:Method: GET 2024-02-01T10:22:56.424 DEBUG:boto:Path: /vclfcn-16/?prefix=dummy 2024-02-01T10:22:56.424 DEBUG:boto:Data: 2024-02-01T10:22:56.425 DEBUG:boto:Headers: {} 2024-02-01T10:22:56.425 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:56.425 DEBUG:boto:Port: 8000 2024-02-01T10:22:56.425 DEBUG:boto:Params: {} 2024-02-01T10:22:56.425 DEBUG:boto:Token: None 2024-02-01T10:22:56.425 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:56 GMT /vclfcn-16/ 2024-02-01T10:22:56.425 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sabhUtE7S7EotHtTTX812xuscbY= 2024-02-01T10:22:56.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sabhUtE7S7EotHtTTX812xuscbY=', 'Content-Length': '0'} 2024-02-01T10:22:56.429 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a5f2acdf26b7ee6-0065bb7100-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:56.476 DEBUG:boto:b'vclfcn-16dummy1000falsedummy2024-02-01T10:19:58.958Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:56.477 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-02-01T10:22:56.477 DEBUG:boto:path=/vclfcn-16/obj-test-zone1 2024-02-01T10:22:56.477 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1 2024-02-01T10:22:56.477 DEBUG:boto:path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:56.477 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:56.477 DEBUG:boto:Method: GET 2024-02-01T10:22:56.477 DEBUG:boto:Path: /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:56.477 DEBUG:boto:Data: 2024-02-01T10:22:56.477 DEBUG:boto:Headers: {} 2024-02-01T10:22:56.478 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:56.478 DEBUG:boto:Port: 8001 2024-02-01T10:22:56.478 DEBUG:boto:Params: {} 2024-02-01T10:22:56.478 DEBUG:boto:Token: None 2024-02-01T10:22:56.478 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:56 GMT /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:56.478 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:l6/DinZmTddRdxogFgIuKOTajuw= 2024-02-01T10:22:56.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:l6/DinZmTddRdxogFgIuKOTajuw=', 'Content-Length': '0'} 2024-02-01T10:22:56.481 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:49 GMT'), ('x-amz-version-id', 'lS9-sMOvyR-VcD.47BkPLymGmIyfWmf'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000028ec690fce19fa40-0065bb7100-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:56.481 DEBUG:boto:path=/vclfcn-16/obj-test-zone1 2024-02-01T10:22:56.482 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1 2024-02-01T10:22:56.482 DEBUG:boto:path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:56.482 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:56.482 DEBUG:boto:Method: GET 2024-02-01T10:22:56.482 DEBUG:boto:Path: /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:56.482 DEBUG:boto:Data: 2024-02-01T10:22:56.482 DEBUG:boto:Headers: {} 2024-02-01T10:22:56.482 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:56.482 DEBUG:boto:Port: 8000 2024-02-01T10:22:56.482 DEBUG:boto:Params: {} 2024-02-01T10:22:56.483 DEBUG:boto:Token: None 2024-02-01T10:22:56.483 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:56 GMT /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:56.483 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:l6/DinZmTddRdxogFgIuKOTajuw= 2024-02-01T10:22:56.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:l6/DinZmTddRdxogFgIuKOTajuw=', 'Content-Length': '0'} 2024-02-01T10:22:56.486 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:49 GMT'), ('x-amz-version-id', 'lS9-sMOvyR-VcD.47BkPLymGmIyfWmf'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000047cca7a6faf43b84-0065bb7100-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:56.528 DEBUG:boto:path=/vclfcn-16/obj-test-zone1 2024-02-01T10:22:56.528 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1 2024-02-01T10:22:56.528 DEBUG:boto:path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:56.529 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:56.529 DEBUG:boto:Method: HEAD 2024-02-01T10:22:56.529 DEBUG:boto:Path: /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:56.529 DEBUG:boto:Data: 2024-02-01T10:22:56.529 DEBUG:boto:Headers: {} 2024-02-01T10:22:56.529 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:56.529 DEBUG:boto:Port: 8001 2024-02-01T10:22:56.529 DEBUG:boto:Params: {} 2024-02-01T10:22:56.529 DEBUG:boto:Token: None 2024-02-01T10:22:56.530 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:56 GMT /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:56.530 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+0GcPkThCibKsPcGim8C+vql5P4= 2024-02-01T10:22:56.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+0GcPkThCibKsPcGim8C+vql5P4=', 'Content-Length': '0'} 2024-02-01T10:22:56.533 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:49 GMT'), ('x-amz-version-id', 'lS9-sMOvyR-VcD.47BkPLymGmIyfWmf'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c1f338e4866eb6fb-0065bb7100-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:56.533 DEBUG:boto:path=/vclfcn-16/obj-test-zone1 2024-02-01T10:22:56.534 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1 2024-02-01T10:22:56.534 DEBUG:boto:path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:56.534 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:56.534 DEBUG:boto:Method: HEAD 2024-02-01T10:22:56.534 DEBUG:boto:Path: /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:56.534 DEBUG:boto:Data: 2024-02-01T10:22:56.534 DEBUG:boto:Headers: {} 2024-02-01T10:22:56.534 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:56.534 DEBUG:boto:Port: 8000 2024-02-01T10:22:56.534 DEBUG:boto:Params: {} 2024-02-01T10:22:56.535 DEBUG:boto:Token: None 2024-02-01T10:22:56.535 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:56 GMT /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:56.535 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+0GcPkThCibKsPcGim8C+vql5P4= 2024-02-01T10:22:56.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+0GcPkThCibKsPcGim8C+vql5P4=', 'Content-Length': '0'} 2024-02-01T10:22:56.538 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:49 GMT'), ('x-amz-version-id', 'lS9-sMOvyR-VcD.47BkPLymGmIyfWmf'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f48c37aa710823c3-0065bb7100-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:56.538 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-02-01T10:22:56.538 DEBUG:boto:path=/vclfcn-16/obj-test-zone1 2024-02-01T10:22:56.538 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1 2024-02-01T10:22:56.538 DEBUG:boto:path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:56.538 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:56.539 DEBUG:boto:Method: GET 2024-02-01T10:22:56.539 DEBUG:boto:Path: /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:56.539 DEBUG:boto:Data: 2024-02-01T10:22:56.539 DEBUG:boto:Headers: {} 2024-02-01T10:22:56.539 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:56.539 DEBUG:boto:Port: 8001 2024-02-01T10:22:56.539 DEBUG:boto:Params: {} 2024-02-01T10:22:56.539 DEBUG:boto:Token: None 2024-02-01T10:22:56.539 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:56 GMT /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:56.539 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:l6/DinZmTddRdxogFgIuKOTajuw= 2024-02-01T10:22:56.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:l6/DinZmTddRdxogFgIuKOTajuw=', 'Content-Length': '0'} 2024-02-01T10:22:56.542 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:49 GMT'), ('x-amz-version-id', 'lS9-sMOvyR-VcD.47BkPLymGmIyfWmf'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009098d81782c4dd27-0065bb7100-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:56.584 DEBUG:boto:path=/vclfcn-16/obj-test-zone1 2024-02-01T10:22:56.584 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1 2024-02-01T10:22:56.585 DEBUG:boto:path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:56.585 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:56.585 DEBUG:boto:Method: GET 2024-02-01T10:22:56.585 DEBUG:boto:Path: /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:56.585 DEBUG:boto:Data: 2024-02-01T10:22:56.585 DEBUG:boto:Headers: {} 2024-02-01T10:22:56.585 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:56.585 DEBUG:boto:Port: 8000 2024-02-01T10:22:56.585 DEBUG:boto:Params: {} 2024-02-01T10:22:56.585 DEBUG:boto:Token: None 2024-02-01T10:22:56.585 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:56 GMT /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:56.586 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:l6/DinZmTddRdxogFgIuKOTajuw= 2024-02-01T10:22:56.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:l6/DinZmTddRdxogFgIuKOTajuw=', 'Content-Length': '0'} 2024-02-01T10:22:56.589 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:49 GMT'), ('x-amz-version-id', 'lS9-sMOvyR-VcD.47BkPLymGmIyfWmf'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003a7d7a631de70a8e-0065bb7100-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:56.636 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:22:56.636 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:22:56.636 DEBUG:boto:path=/vclfcn-16/?prefix=obj-test-zone1 2024-02-01T10:22:56.636 DEBUG:boto:auth_path=/vclfcn-16/?prefix=obj-test-zone1 2024-02-01T10:22:56.637 DEBUG:boto:Method: GET 2024-02-01T10:22:56.637 DEBUG:boto:Path: /vclfcn-16/?prefix=obj-test-zone1 2024-02-01T10:22:56.637 DEBUG:boto:Data: 2024-02-01T10:22:56.637 DEBUG:boto:Headers: {} 2024-02-01T10:22:56.637 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:56.637 DEBUG:boto:Port: 8001 2024-02-01T10:22:56.637 DEBUG:boto:Params: {} 2024-02-01T10:22:56.637 DEBUG:boto:Token: None 2024-02-01T10:22:56.637 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:56 GMT /vclfcn-16/ 2024-02-01T10:22:56.638 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sabhUtE7S7EotHtTTX812xuscbY= 2024-02-01T10:22:56.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sabhUtE7S7EotHtTTX812xuscbY=', 'Content-Length': '0'} 2024-02-01T10:22:56.641 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008cd8631f7148219e-0065bb7100-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:56.642 DEBUG:boto:b'vclfcn-16obj-test-zone11000falseobj-test-zone12024-02-01T10:20:49.965Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:56.642 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:22:56.642 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:22:56.642 DEBUG:boto:path=/vclfcn-16/?prefix=obj-test-zone1 2024-02-01T10:22:56.643 DEBUG:boto:auth_path=/vclfcn-16/?prefix=obj-test-zone1 2024-02-01T10:22:56.643 DEBUG:boto:Method: GET 2024-02-01T10:22:56.643 DEBUG:boto:Path: /vclfcn-16/?prefix=obj-test-zone1 2024-02-01T10:22:56.643 DEBUG:boto:Data: 2024-02-01T10:22:56.643 DEBUG:boto:Headers: {} 2024-02-01T10:22:56.643 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:56.643 DEBUG:boto:Port: 8000 2024-02-01T10:22:56.643 DEBUG:boto:Params: {} 2024-02-01T10:22:56.643 DEBUG:boto:Token: None 2024-02-01T10:22:56.643 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:56 GMT /vclfcn-16/ 2024-02-01T10:22:56.644 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sabhUtE7S7EotHtTTX812xuscbY= 2024-02-01T10:22:56.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sabhUtE7S7EotHtTTX812xuscbY=', 'Content-Length': '0'} 2024-02-01T10:22:56.647 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b10c13badb9927d9-0065bb7100-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:56.648 DEBUG:boto:b'vclfcn-16obj-test-zone11000falseobj-test-zone12024-02-01T10:20:49.965Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:56.648 DEBUG:boto:path=/vclfcn-16/obj-test-zone1 2024-02-01T10:22:56.648 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1 2024-02-01T10:22:56.648 DEBUG:boto:Method: HEAD 2024-02-01T10:22:56.648 DEBUG:boto:Path: /vclfcn-16/obj-test-zone1 2024-02-01T10:22:56.649 DEBUG:boto:Data: 2024-02-01T10:22:56.649 DEBUG:boto:Headers: {} 2024-02-01T10:22:56.649 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:56.649 DEBUG:boto:Port: 8001 2024-02-01T10:22:56.649 DEBUG:boto:Params: {} 2024-02-01T10:22:56.649 DEBUG:boto:Token: None 2024-02-01T10:22:56.649 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:56 GMT /vclfcn-16/obj-test-zone1 2024-02-01T10:22:56.650 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:VfvBsJoCAi8E6coLVMAVslwSa8g= 2024-02-01T10:22:56.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:VfvBsJoCAi8E6coLVMAVslwSa8g=', 'Content-Length': '0'} 2024-02-01T10:22:56.653 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:49 GMT'), ('x-amz-version-id', 'lS9-sMOvyR-VcD.47BkPLymGmIyfWmf'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b1deea85fc75205f-0065bb7100-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:56.654 DEBUG:boto:path=/vclfcn-16/obj-test-zone1 2024-02-01T10:22:56.654 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1 2024-02-01T10:22:56.655 DEBUG:boto:Method: HEAD 2024-02-01T10:22:56.655 DEBUG:boto:Path: /vclfcn-16/obj-test-zone1 2024-02-01T10:22:56.655 DEBUG:boto:Data: 2024-02-01T10:22:56.655 DEBUG:boto:Headers: {} 2024-02-01T10:22:56.655 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:56.655 DEBUG:boto:Port: 8000 2024-02-01T10:22:56.655 DEBUG:boto:Params: {} 2024-02-01T10:22:56.656 DEBUG:boto:Token: None 2024-02-01T10:22:56.656 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:56 GMT /vclfcn-16/obj-test-zone1 2024-02-01T10:22:56.656 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:VfvBsJoCAi8E6coLVMAVslwSa8g= 2024-02-01T10:22:56.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:VfvBsJoCAi8E6coLVMAVslwSa8g=', 'Content-Length': '0'} 2024-02-01T10:22:56.660 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:49 GMT'), ('x-amz-version-id', 'lS9-sMOvyR-VcD.47BkPLymGmIyfWmf'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001c38ff9d5347daa0-0065bb7100-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:56.661 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-02-01T10:22:56.661 DEBUG:boto:path=/vclfcn-16/obj-test-zone1 2024-02-01T10:22:56.661 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1 2024-02-01T10:22:56.661 DEBUG:boto:path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:56.661 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:56.661 DEBUG:boto:Method: GET 2024-02-01T10:22:56.662 DEBUG:boto:Path: /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:56.662 DEBUG:boto:Data: 2024-02-01T10:22:56.662 DEBUG:boto:Headers: {} 2024-02-01T10:22:56.662 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:56.662 DEBUG:boto:Port: 8001 2024-02-01T10:22:56.662 DEBUG:boto:Params: {} 2024-02-01T10:22:56.662 DEBUG:boto:Token: None 2024-02-01T10:22:56.662 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:56 GMT /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:56.663 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:l6/DinZmTddRdxogFgIuKOTajuw= 2024-02-01T10:22:56.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:l6/DinZmTddRdxogFgIuKOTajuw=', 'Content-Length': '0'} 2024-02-01T10:22:56.666 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:49 GMT'), ('x-amz-version-id', 'lS9-sMOvyR-VcD.47BkPLymGmIyfWmf'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009b1bcdad904ba74b-0065bb7100-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:56.712 DEBUG:boto:path=/vclfcn-16/obj-test-zone1 2024-02-01T10:22:56.712 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1 2024-02-01T10:22:56.712 DEBUG:boto:path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:56.713 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:56.713 DEBUG:boto:Method: GET 2024-02-01T10:22:56.713 DEBUG:boto:Path: /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:56.713 DEBUG:boto:Data: 2024-02-01T10:22:56.713 DEBUG:boto:Headers: {} 2024-02-01T10:22:56.713 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:56.713 DEBUG:boto:Port: 8000 2024-02-01T10:22:56.713 DEBUG:boto:Params: {} 2024-02-01T10:22:56.713 DEBUG:boto:Token: None 2024-02-01T10:22:56.714 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:56 GMT /vclfcn-16/obj-test-zone1?versionId=lS9-sMOvyR-VcD.47BkPLymGmIyfWmf 2024-02-01T10:22:56.714 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:l6/DinZmTddRdxogFgIuKOTajuw= 2024-02-01T10:22:56.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:l6/DinZmTddRdxogFgIuKOTajuw=', 'Content-Length': '0'} 2024-02-01T10:22:56.717 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:49 GMT'), ('x-amz-version-id', 'lS9-sMOvyR-VcD.47BkPLymGmIyfWmf'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000663aeb51b3746b29-0065bb7100-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:56.760 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:22:56.761 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:22:56.761 DEBUG:boto:path=/vclfcn-16/?prefix=obj-test-zone1 2024-02-01T10:22:56.761 DEBUG:boto:auth_path=/vclfcn-16/?prefix=obj-test-zone1 2024-02-01T10:22:56.761 DEBUG:boto:Method: GET 2024-02-01T10:22:56.761 DEBUG:boto:Path: /vclfcn-16/?prefix=obj-test-zone1 2024-02-01T10:22:56.762 DEBUG:boto:Data: 2024-02-01T10:22:56.762 DEBUG:boto:Headers: {} 2024-02-01T10:22:56.762 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:56.762 DEBUG:boto:Port: 8001 2024-02-01T10:22:56.762 DEBUG:boto:Params: {} 2024-02-01T10:22:56.763 DEBUG:boto:Token: None 2024-02-01T10:22:56.763 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:56 GMT /vclfcn-16/ 2024-02-01T10:22:56.763 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sabhUtE7S7EotHtTTX812xuscbY= 2024-02-01T10:22:56.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sabhUtE7S7EotHtTTX812xuscbY=', 'Content-Length': '0'} 2024-02-01T10:22:56.768 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0e648b279c82455-0065bb7100-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:56.808 DEBUG:boto:b'vclfcn-16obj-test-zone11000falseobj-test-zone12024-02-01T10:20:49.965Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:56.809 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:22:56.809 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:22:56.809 DEBUG:boto:path=/vclfcn-16/?prefix=obj-test-zone1 2024-02-01T10:22:56.809 DEBUG:boto:auth_path=/vclfcn-16/?prefix=obj-test-zone1 2024-02-01T10:22:56.809 DEBUG:boto:Method: GET 2024-02-01T10:22:56.809 DEBUG:boto:Path: /vclfcn-16/?prefix=obj-test-zone1 2024-02-01T10:22:56.809 DEBUG:boto:Data: 2024-02-01T10:22:56.809 DEBUG:boto:Headers: {} 2024-02-01T10:22:56.809 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:56.809 DEBUG:boto:Port: 8000 2024-02-01T10:22:56.810 DEBUG:boto:Params: {} 2024-02-01T10:22:56.810 DEBUG:boto:Token: None 2024-02-01T10:22:56.810 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:56 GMT /vclfcn-16/ 2024-02-01T10:22:56.810 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sabhUtE7S7EotHtTTX812xuscbY= 2024-02-01T10:22:56.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sabhUtE7S7EotHtTTX812xuscbY=', 'Content-Length': '0'} 2024-02-01T10:22:56.815 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a51cc166f14a5353-0065bb7100-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:56.856 DEBUG:boto:b'vclfcn-16obj-test-zone11000falseobj-test-zone12024-02-01T10:20:49.965Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:56.856 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-02-01T10:22:56.857 DEBUG:boto:path=/vclfcn-16/obj-test-zone2 2024-02-01T10:22:56.857 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2 2024-02-01T10:22:56.857 DEBUG:boto:path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:56.857 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:56.857 DEBUG:boto:Method: GET 2024-02-01T10:22:56.857 DEBUG:boto:Path: /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:56.857 DEBUG:boto:Data: 2024-02-01T10:22:56.857 DEBUG:boto:Headers: {} 2024-02-01T10:22:56.857 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:56.858 DEBUG:boto:Port: 8001 2024-02-01T10:22:56.858 DEBUG:boto:Params: {} 2024-02-01T10:22:56.858 DEBUG:boto:Token: None 2024-02-01T10:22:56.858 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:56 GMT /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:56.858 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:6cTOtnCclybDl/ijF2o5PeKAG4A= 2024-02-01T10:22:56.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:6cTOtnCclybDl/ijF2o5PeKAG4A=', 'Content-Length': '0'} 2024-02-01T10:22:56.861 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'mE6urSxcrRA9wdyuToa41.w2aTia9p1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e652691fbfa13ffd-0065bb7100-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:56.904 DEBUG:boto:path=/vclfcn-16/obj-test-zone2 2024-02-01T10:22:56.904 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2 2024-02-01T10:22:56.904 DEBUG:boto:path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:56.904 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:56.905 DEBUG:boto:Method: GET 2024-02-01T10:22:56.905 DEBUG:boto:Path: /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:56.905 DEBUG:boto:Data: 2024-02-01T10:22:56.905 DEBUG:boto:Headers: {} 2024-02-01T10:22:56.905 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:56.905 DEBUG:boto:Port: 8000 2024-02-01T10:22:56.905 DEBUG:boto:Params: {} 2024-02-01T10:22:56.905 DEBUG:boto:Token: None 2024-02-01T10:22:56.905 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:56 GMT /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:56.905 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:6cTOtnCclybDl/ijF2o5PeKAG4A= 2024-02-01T10:22:56.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:6cTOtnCclybDl/ijF2o5PeKAG4A=', 'Content-Length': '0'} 2024-02-01T10:22:56.909 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'mE6urSxcrRA9wdyuToa41.w2aTia9p1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c9c06c524910a522-0065bb7100-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:56.910 DEBUG:boto:path=/vclfcn-16/obj-test-zone2 2024-02-01T10:22:56.911 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2 2024-02-01T10:22:56.911 DEBUG:boto:path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:56.911 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:56.911 DEBUG:boto:Method: HEAD 2024-02-01T10:22:56.911 DEBUG:boto:Path: /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:56.911 DEBUG:boto:Data: 2024-02-01T10:22:56.911 DEBUG:boto:Headers: {} 2024-02-01T10:22:56.911 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:56.911 DEBUG:boto:Port: 8001 2024-02-01T10:22:56.912 DEBUG:boto:Params: {} 2024-02-01T10:22:56.912 DEBUG:boto:Token: None 2024-02-01T10:22:56.912 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:56 GMT /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:56.912 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:diiegCm0+36FfjbgrT3I201pQFA= 2024-02-01T10:22:56.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:diiegCm0+36FfjbgrT3I201pQFA=', 'Content-Length': '0'} 2024-02-01T10:22:56.915 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'mE6urSxcrRA9wdyuToa41.w2aTia9p1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f01c89c302c263cb-0065bb7100-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:56.915 DEBUG:boto:path=/vclfcn-16/obj-test-zone2 2024-02-01T10:22:56.915 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2 2024-02-01T10:22:56.916 DEBUG:boto:path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:56.916 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:56.916 DEBUG:boto:Method: HEAD 2024-02-01T10:22:56.916 DEBUG:boto:Path: /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:56.916 DEBUG:boto:Data: 2024-02-01T10:22:56.916 DEBUG:boto:Headers: {} 2024-02-01T10:22:56.916 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:56.916 DEBUG:boto:Port: 8000 2024-02-01T10:22:56.916 DEBUG:boto:Params: {} 2024-02-01T10:22:56.916 DEBUG:boto:Token: None 2024-02-01T10:22:56.920 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:56 GMT /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:56.920 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:diiegCm0+36FfjbgrT3I201pQFA= 2024-02-01T10:22:56.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:diiegCm0+36FfjbgrT3I201pQFA=', 'Content-Length': '0'} 2024-02-01T10:22:56.923 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'mE6urSxcrRA9wdyuToa41.w2aTia9p1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006b317178495ceae1-0065bb7100-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:56.923 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-02-01T10:22:56.923 DEBUG:boto:path=/vclfcn-16/obj-test-zone2 2024-02-01T10:22:56.923 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2 2024-02-01T10:22:56.924 DEBUG:boto:path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:56.924 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:56.924 DEBUG:boto:Method: GET 2024-02-01T10:22:56.924 DEBUG:boto:Path: /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:56.924 DEBUG:boto:Data: 2024-02-01T10:22:56.924 DEBUG:boto:Headers: {} 2024-02-01T10:22:56.924 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:56.924 DEBUG:boto:Port: 8001 2024-02-01T10:22:56.924 DEBUG:boto:Params: {} 2024-02-01T10:22:56.924 DEBUG:boto:Token: None 2024-02-01T10:22:56.925 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:56 GMT /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:56.925 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:6cTOtnCclybDl/ijF2o5PeKAG4A= 2024-02-01T10:22:56.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:6cTOtnCclybDl/ijF2o5PeKAG4A=', 'Content-Length': '0'} 2024-02-01T10:22:56.928 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'mE6urSxcrRA9wdyuToa41.w2aTia9p1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000055994d9eb10a2b5c-0065bb7100-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:56.972 DEBUG:boto:path=/vclfcn-16/obj-test-zone2 2024-02-01T10:22:56.972 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2 2024-02-01T10:22:56.972 DEBUG:boto:path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:56.972 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:56.973 DEBUG:boto:Method: GET 2024-02-01T10:22:56.973 DEBUG:boto:Path: /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:56.973 DEBUG:boto:Data: 2024-02-01T10:22:56.973 DEBUG:boto:Headers: {} 2024-02-01T10:22:56.973 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:56.973 DEBUG:boto:Port: 8000 2024-02-01T10:22:56.973 DEBUG:boto:Params: {} 2024-02-01T10:22:56.973 DEBUG:boto:Token: None 2024-02-01T10:22:56.974 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:56 GMT /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:56.974 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:6cTOtnCclybDl/ijF2o5PeKAG4A= 2024-02-01T10:22:56.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:6cTOtnCclybDl/ijF2o5PeKAG4A=', 'Content-Length': '0'} 2024-02-01T10:22:56.977 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'mE6urSxcrRA9wdyuToa41.w2aTia9p1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f48e3c0ff9dd599c-0065bb7100-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:56.977 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:22:56.977 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:22:56.978 DEBUG:boto:path=/vclfcn-16/?prefix=obj-test-zone2 2024-02-01T10:22:56.978 DEBUG:boto:auth_path=/vclfcn-16/?prefix=obj-test-zone2 2024-02-01T10:22:56.978 DEBUG:boto:Method: GET 2024-02-01T10:22:56.978 DEBUG:boto:Path: /vclfcn-16/?prefix=obj-test-zone2 2024-02-01T10:22:56.978 DEBUG:boto:Data: 2024-02-01T10:22:56.978 DEBUG:boto:Headers: {} 2024-02-01T10:22:56.978 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:56.978 DEBUG:boto:Port: 8001 2024-02-01T10:22:56.978 DEBUG:boto:Params: {} 2024-02-01T10:22:56.978 DEBUG:boto:Token: None 2024-02-01T10:22:56.978 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:56 GMT /vclfcn-16/ 2024-02-01T10:22:56.979 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sabhUtE7S7EotHtTTX812xuscbY= 2024-02-01T10:22:56.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sabhUtE7S7EotHtTTX812xuscbY=', 'Content-Length': '0'} 2024-02-01T10:22:56.983 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001fd52211440b2fdc-0065bb7100-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:56.983 DEBUG:boto:b'vclfcn-16obj-test-zone21000falseobj-test-zone22024-02-01T10:20:50.122Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:56.983 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:22:56.983 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:22:56.983 DEBUG:boto:path=/vclfcn-16/?prefix=obj-test-zone2 2024-02-01T10:22:56.984 DEBUG:boto:auth_path=/vclfcn-16/?prefix=obj-test-zone2 2024-02-01T10:22:56.984 DEBUG:boto:Method: GET 2024-02-01T10:22:56.984 DEBUG:boto:Path: /vclfcn-16/?prefix=obj-test-zone2 2024-02-01T10:22:56.984 DEBUG:boto:Data: 2024-02-01T10:22:56.984 DEBUG:boto:Headers: {} 2024-02-01T10:22:56.984 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:56.984 DEBUG:boto:Port: 8000 2024-02-01T10:22:56.984 DEBUG:boto:Params: {} 2024-02-01T10:22:56.984 DEBUG:boto:Token: None 2024-02-01T10:22:56.985 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:56 GMT /vclfcn-16/ 2024-02-01T10:22:56.985 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sabhUtE7S7EotHtTTX812xuscbY= 2024-02-01T10:22:56.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sabhUtE7S7EotHtTTX812xuscbY=', 'Content-Length': '0'} 2024-02-01T10:22:56.989 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000003a95a803843ee67-0065bb7100-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:56.989 DEBUG:boto:b'vclfcn-16obj-test-zone21000falseobj-test-zone22024-02-01T10:20:50.122Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:56.990 DEBUG:boto:path=/vclfcn-16/obj-test-zone2 2024-02-01T10:22:56.990 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2 2024-02-01T10:22:56.990 DEBUG:boto:Method: HEAD 2024-02-01T10:22:56.990 DEBUG:boto:Path: /vclfcn-16/obj-test-zone2 2024-02-01T10:22:56.990 DEBUG:boto:Data: 2024-02-01T10:22:56.991 DEBUG:boto:Headers: {} 2024-02-01T10:22:56.991 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:56.991 DEBUG:boto:Port: 8001 2024-02-01T10:22:56.991 DEBUG:boto:Params: {} 2024-02-01T10:22:56.991 DEBUG:boto:Token: None 2024-02-01T10:22:56.991 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:56 GMT /vclfcn-16/obj-test-zone2 2024-02-01T10:22:56.991 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sTwhzHsgNP0zBu0qeprwTncNQzo= 2024-02-01T10:22:56.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sTwhzHsgNP0zBu0qeprwTncNQzo=', 'Content-Length': '0'} 2024-02-01T10:22:56.995 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'mE6urSxcrRA9wdyuToa41.w2aTia9p1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000039e994444464a800-0065bb7100-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:56.996 DEBUG:boto:path=/vclfcn-16/obj-test-zone2 2024-02-01T10:22:56.996 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2 2024-02-01T10:22:56.996 DEBUG:boto:Method: HEAD 2024-02-01T10:22:56.996 DEBUG:boto:Path: /vclfcn-16/obj-test-zone2 2024-02-01T10:22:56.996 DEBUG:boto:Data: 2024-02-01T10:22:56.996 DEBUG:boto:Headers: {} 2024-02-01T10:22:56.996 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:56.996 DEBUG:boto:Port: 8000 2024-02-01T10:22:56.996 DEBUG:boto:Params: {} 2024-02-01T10:22:56.997 DEBUG:boto:Token: None 2024-02-01T10:22:56.997 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:56 GMT /vclfcn-16/obj-test-zone2 2024-02-01T10:22:56.997 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sTwhzHsgNP0zBu0qeprwTncNQzo= 2024-02-01T10:22:56.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sTwhzHsgNP0zBu0qeprwTncNQzo=', 'Content-Length': '0'} 2024-02-01T10:22:57.001 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'mE6urSxcrRA9wdyuToa41.w2aTia9p1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000017c7783b722f8dbd-0065bb7101-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:57.001 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-02-01T10:22:57.001 DEBUG:boto:path=/vclfcn-16/obj-test-zone2 2024-02-01T10:22:57.001 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2 2024-02-01T10:22:57.001 DEBUG:boto:path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:57.002 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:57.002 DEBUG:boto:Method: GET 2024-02-01T10:22:57.002 DEBUG:boto:Path: /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:57.002 DEBUG:boto:Data: 2024-02-01T10:22:57.002 DEBUG:boto:Headers: {} 2024-02-01T10:22:57.002 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:57.002 DEBUG:boto:Port: 8001 2024-02-01T10:22:57.002 DEBUG:boto:Params: {} 2024-02-01T10:22:57.002 DEBUG:boto:Token: None 2024-02-01T10:22:57.003 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:57 GMT /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:57.003 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:3INq5F1RYlqwVF99em2IFgqeq68= 2024-02-01T10:22:57.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:3INq5F1RYlqwVF99em2IFgqeq68=', 'Content-Length': '0'} 2024-02-01T10:22:57.006 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'mE6urSxcrRA9wdyuToa41.w2aTia9p1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003a721b844911810c-0065bb7101-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:57.052 DEBUG:boto:path=/vclfcn-16/obj-test-zone2 2024-02-01T10:22:57.052 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2 2024-02-01T10:22:57.052 DEBUG:boto:path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:57.053 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:57.053 DEBUG:boto:Method: GET 2024-02-01T10:22:57.053 DEBUG:boto:Path: /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:57.053 DEBUG:boto:Data: 2024-02-01T10:22:57.053 DEBUG:boto:Headers: {} 2024-02-01T10:22:57.053 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:57.053 DEBUG:boto:Port: 8000 2024-02-01T10:22:57.053 DEBUG:boto:Params: {} 2024-02-01T10:22:57.053 DEBUG:boto:Token: None 2024-02-01T10:22:57.054 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:57 GMT /vclfcn-16/obj-test-zone2?versionId=mE6urSxcrRA9wdyuToa41.w2aTia9p1 2024-02-01T10:22:57.054 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:3INq5F1RYlqwVF99em2IFgqeq68= 2024-02-01T10:22:57.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:3INq5F1RYlqwVF99em2IFgqeq68=', 'Content-Length': '0'} 2024-02-01T10:22:57.057 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'mE6urSxcrRA9wdyuToa41.w2aTia9p1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a11af0615acdd17a-0065bb7101-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:57.058 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:22:57.058 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:22:57.058 DEBUG:boto:path=/vclfcn-16/?prefix=obj-test-zone2 2024-02-01T10:22:57.058 DEBUG:boto:auth_path=/vclfcn-16/?prefix=obj-test-zone2 2024-02-01T10:22:57.058 DEBUG:boto:Method: GET 2024-02-01T10:22:57.058 DEBUG:boto:Path: /vclfcn-16/?prefix=obj-test-zone2 2024-02-01T10:22:57.058 DEBUG:boto:Data: 2024-02-01T10:22:57.059 DEBUG:boto:Headers: {} 2024-02-01T10:22:57.059 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:57.059 DEBUG:boto:Port: 8001 2024-02-01T10:22:57.059 DEBUG:boto:Params: {} 2024-02-01T10:22:57.059 DEBUG:boto:Token: None 2024-02-01T10:22:57.059 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:57 GMT /vclfcn-16/ 2024-02-01T10:22:57.059 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:OZcix+13StdoAG7BxmpKRz1uz+o= 2024-02-01T10:22:57.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:OZcix+13StdoAG7BxmpKRz1uz+o=', 'Content-Length': '0'} 2024-02-01T10:22:57.063 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063972b992503939f-0065bb7101-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:57.064 DEBUG:boto:b'vclfcn-16obj-test-zone21000falseobj-test-zone22024-02-01T10:20:50.122Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:57.064 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:22:57.064 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:22:57.064 DEBUG:boto:path=/vclfcn-16/?prefix=obj-test-zone2 2024-02-01T10:22:57.065 DEBUG:boto:auth_path=/vclfcn-16/?prefix=obj-test-zone2 2024-02-01T10:22:57.065 DEBUG:boto:Method: GET 2024-02-01T10:22:57.065 DEBUG:boto:Path: /vclfcn-16/?prefix=obj-test-zone2 2024-02-01T10:22:57.065 DEBUG:boto:Data: 2024-02-01T10:22:57.065 DEBUG:boto:Headers: {} 2024-02-01T10:22:57.065 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:57.065 DEBUG:boto:Port: 8000 2024-02-01T10:22:57.065 DEBUG:boto:Params: {} 2024-02-01T10:22:57.065 DEBUG:boto:Token: None 2024-02-01T10:22:57.065 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:57 GMT /vclfcn-16/ 2024-02-01T10:22:57.066 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:OZcix+13StdoAG7BxmpKRz1uz+o= 2024-02-01T10:22:57.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:OZcix+13StdoAG7BxmpKRz1uz+o=', 'Content-Length': '0'} 2024-02-01T10:22:57.070 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000658dd3906e43c9ae-0065bb7101-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:57.070 DEBUG:boto:b'vclfcn-16obj-test-zone21000falseobj-test-zone22024-02-01T10:20:50.122Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:57.070 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-02-01T10:22:57.071 DEBUG:boto:path=/vclfcn-16/obj-test-zone3 2024-02-01T10:22:57.071 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3 2024-02-01T10:22:57.071 DEBUG:boto:path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:57.071 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:57.071 DEBUG:boto:Method: GET 2024-02-01T10:22:57.071 DEBUG:boto:Path: /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:57.071 DEBUG:boto:Data: 2024-02-01T10:22:57.071 DEBUG:boto:Headers: {} 2024-02-01T10:22:57.071 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:57.071 DEBUG:boto:Port: 8001 2024-02-01T10:22:57.071 DEBUG:boto:Params: {} 2024-02-01T10:22:57.072 DEBUG:boto:Token: None 2024-02-01T10:22:57.072 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:57 GMT /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:57.072 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:B2VOM72P/j/9crxQHbbF9O++luw= 2024-02-01T10:22:57.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:B2VOM72P/j/9crxQHbbF9O++luw=', 'Content-Length': '0'} 2024-02-01T10:22:57.075 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005219081b8a63c541-0065bb7101-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:57.075 DEBUG:boto:path=/vclfcn-16/obj-test-zone3 2024-02-01T10:22:57.075 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3 2024-02-01T10:22:57.075 DEBUG:boto:path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:57.075 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:57.075 DEBUG:boto:Method: GET 2024-02-01T10:22:57.076 DEBUG:boto:Path: /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:57.076 DEBUG:boto:Data: 2024-02-01T10:22:57.076 DEBUG:boto:Headers: {} 2024-02-01T10:22:57.076 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:57.076 DEBUG:boto:Port: 8000 2024-02-01T10:22:57.076 DEBUG:boto:Params: {} 2024-02-01T10:22:57.076 DEBUG:boto:Token: None 2024-02-01T10:22:57.076 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:57 GMT /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:57.076 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:B2VOM72P/j/9crxQHbbF9O++luw= 2024-02-01T10:22:57.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:B2VOM72P/j/9crxQHbbF9O++luw=', 'Content-Length': '0'} 2024-02-01T10:22:57.080 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ecb4bc20e8870979-0065bb7101-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:57.128 DEBUG:boto:path=/vclfcn-16/obj-test-zone3 2024-02-01T10:22:57.128 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3 2024-02-01T10:22:57.128 DEBUG:boto:path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:57.128 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:57.128 DEBUG:boto:Method: HEAD 2024-02-01T10:22:57.128 DEBUG:boto:Path: /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:57.129 DEBUG:boto:Data: 2024-02-01T10:22:57.129 DEBUG:boto:Headers: {} 2024-02-01T10:22:57.129 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:57.129 DEBUG:boto:Port: 8001 2024-02-01T10:22:57.129 DEBUG:boto:Params: {} 2024-02-01T10:22:57.129 DEBUG:boto:Token: None 2024-02-01T10:22:57.129 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:57 GMT /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:57.129 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gMt8VHHlQHviumtxILeUuQwNiPU= 2024-02-01T10:22:57.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gMt8VHHlQHviumtxILeUuQwNiPU=', 'Content-Length': '0'} 2024-02-01T10:22:57.132 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008e4a8bff473293da-0065bb7101-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:57.133 DEBUG:boto:path=/vclfcn-16/obj-test-zone3 2024-02-01T10:22:57.133 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3 2024-02-01T10:22:57.133 DEBUG:boto:path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:57.133 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:57.133 DEBUG:boto:Method: HEAD 2024-02-01T10:22:57.133 DEBUG:boto:Path: /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:57.133 DEBUG:boto:Data: 2024-02-01T10:22:57.133 DEBUG:boto:Headers: {} 2024-02-01T10:22:57.133 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:57.133 DEBUG:boto:Port: 8000 2024-02-01T10:22:57.134 DEBUG:boto:Params: {} 2024-02-01T10:22:57.134 DEBUG:boto:Token: None 2024-02-01T10:22:57.134 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:57 GMT /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:57.134 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gMt8VHHlQHviumtxILeUuQwNiPU= 2024-02-01T10:22:57.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gMt8VHHlQHviumtxILeUuQwNiPU=', 'Content-Length': '0'} 2024-02-01T10:22:57.137 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007a291a6ad868297a-0065bb7101-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:57.137 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-02-01T10:22:57.138 DEBUG:boto:path=/vclfcn-16/obj-test-zone3 2024-02-01T10:22:57.138 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3 2024-02-01T10:22:57.138 DEBUG:boto:path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:57.138 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:57.138 DEBUG:boto:Method: GET 2024-02-01T10:22:57.138 DEBUG:boto:Path: /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:57.138 DEBUG:boto:Data: 2024-02-01T10:22:57.138 DEBUG:boto:Headers: {} 2024-02-01T10:22:57.138 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:57.138 DEBUG:boto:Port: 8001 2024-02-01T10:22:57.139 DEBUG:boto:Params: {} 2024-02-01T10:22:57.139 DEBUG:boto:Token: None 2024-02-01T10:22:57.139 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:57 GMT /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:57.139 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:B2VOM72P/j/9crxQHbbF9O++luw= 2024-02-01T10:22:57.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:B2VOM72P/j/9crxQHbbF9O++luw=', 'Content-Length': '0'} 2024-02-01T10:22:57.142 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000029883df8b918c13a-0065bb7101-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:57.142 DEBUG:boto:path=/vclfcn-16/obj-test-zone3 2024-02-01T10:22:57.142 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3 2024-02-01T10:22:57.142 DEBUG:boto:path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:57.142 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:57.143 DEBUG:boto:Method: GET 2024-02-01T10:22:57.143 DEBUG:boto:Path: /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:57.143 DEBUG:boto:Data: 2024-02-01T10:22:57.143 DEBUG:boto:Headers: {} 2024-02-01T10:22:57.143 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:57.143 DEBUG:boto:Port: 8000 2024-02-01T10:22:57.143 DEBUG:boto:Params: {} 2024-02-01T10:22:57.143 DEBUG:boto:Token: None 2024-02-01T10:22:57.143 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:57 GMT /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:57.143 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:B2VOM72P/j/9crxQHbbF9O++luw= 2024-02-01T10:22:57.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:B2VOM72P/j/9crxQHbbF9O++luw=', 'Content-Length': '0'} 2024-02-01T10:22:57.147 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001e20cd35fe7e5429-0065bb7101-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:57.192 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:22:57.192 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:22:57.193 DEBUG:boto:path=/vclfcn-16/?prefix=obj-test-zone3 2024-02-01T10:22:57.193 DEBUG:boto:auth_path=/vclfcn-16/?prefix=obj-test-zone3 2024-02-01T10:22:57.193 DEBUG:boto:Method: GET 2024-02-01T10:22:57.193 DEBUG:boto:Path: /vclfcn-16/?prefix=obj-test-zone3 2024-02-01T10:22:57.193 DEBUG:boto:Data: 2024-02-01T10:22:57.193 DEBUG:boto:Headers: {} 2024-02-01T10:22:57.193 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:57.193 DEBUG:boto:Port: 8001 2024-02-01T10:22:57.193 DEBUG:boto:Params: {} 2024-02-01T10:22:57.193 DEBUG:boto:Token: None 2024-02-01T10:22:57.194 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:57 GMT /vclfcn-16/ 2024-02-01T10:22:57.194 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:OZcix+13StdoAG7BxmpKRz1uz+o= 2024-02-01T10:22:57.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:OZcix+13StdoAG7BxmpKRz1uz+o=', 'Content-Length': '0'} 2024-02-01T10:22:57.198 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097bf1fc1aa81db46-0065bb7101-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:57.198 DEBUG:boto:b'vclfcn-16obj-test-zone31000falseobj-test-zone32024-02-01T10:20:50.297Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:57.199 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:22:57.199 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:22:57.199 DEBUG:boto:path=/vclfcn-16/?prefix=obj-test-zone3 2024-02-01T10:22:57.199 DEBUG:boto:auth_path=/vclfcn-16/?prefix=obj-test-zone3 2024-02-01T10:22:57.200 DEBUG:boto:Method: GET 2024-02-01T10:22:57.200 DEBUG:boto:Path: /vclfcn-16/?prefix=obj-test-zone3 2024-02-01T10:22:57.200 DEBUG:boto:Data: 2024-02-01T10:22:57.200 DEBUG:boto:Headers: {} 2024-02-01T10:22:57.200 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:57.200 DEBUG:boto:Port: 8000 2024-02-01T10:22:57.200 DEBUG:boto:Params: {} 2024-02-01T10:22:57.200 DEBUG:boto:Token: None 2024-02-01T10:22:57.200 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:57 GMT /vclfcn-16/ 2024-02-01T10:22:57.201 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:OZcix+13StdoAG7BxmpKRz1uz+o= 2024-02-01T10:22:57.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:OZcix+13StdoAG7BxmpKRz1uz+o=', 'Content-Length': '0'} 2024-02-01T10:22:57.204 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f5cc25ff047846f8-0065bb7101-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:57.205 DEBUG:boto:b'vclfcn-16obj-test-zone31000falseobj-test-zone32024-02-01T10:20:50.297Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:57.205 DEBUG:boto:path=/vclfcn-16/obj-test-zone3 2024-02-01T10:22:57.205 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3 2024-02-01T10:22:57.205 DEBUG:boto:Method: HEAD 2024-02-01T10:22:57.206 DEBUG:boto:Path: /vclfcn-16/obj-test-zone3 2024-02-01T10:22:57.206 DEBUG:boto:Data: 2024-02-01T10:22:57.206 DEBUG:boto:Headers: {} 2024-02-01T10:22:57.208 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:57.208 DEBUG:boto:Port: 8001 2024-02-01T10:22:57.208 DEBUG:boto:Params: {} 2024-02-01T10:22:57.208 DEBUG:boto:Token: None 2024-02-01T10:22:57.208 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:57 GMT /vclfcn-16/obj-test-zone3 2024-02-01T10:22:57.208 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:XqHNUL0yaO/mrStbxXPyBtKbhaE= 2024-02-01T10:22:57.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:XqHNUL0yaO/mrStbxXPyBtKbhaE=', 'Content-Length': '0'} 2024-02-01T10:22:57.212 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006e205d78e531a92e-0065bb7101-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:57.212 DEBUG:boto:path=/vclfcn-16/obj-test-zone3 2024-02-01T10:22:57.212 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3 2024-02-01T10:22:57.213 DEBUG:boto:Method: HEAD 2024-02-01T10:22:57.213 DEBUG:boto:Path: /vclfcn-16/obj-test-zone3 2024-02-01T10:22:57.213 DEBUG:boto:Data: 2024-02-01T10:22:57.213 DEBUG:boto:Headers: {} 2024-02-01T10:22:57.213 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:57.213 DEBUG:boto:Port: 8000 2024-02-01T10:22:57.213 DEBUG:boto:Params: {} 2024-02-01T10:22:57.213 DEBUG:boto:Token: None 2024-02-01T10:22:57.213 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:57 GMT /vclfcn-16/obj-test-zone3 2024-02-01T10:22:57.214 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:XqHNUL0yaO/mrStbxXPyBtKbhaE= 2024-02-01T10:22:57.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:XqHNUL0yaO/mrStbxXPyBtKbhaE=', 'Content-Length': '0'} 2024-02-01T10:22:57.217 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000017be527bc139a5d9-0065bb7101-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:57.218 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-02-01T10:22:57.218 DEBUG:boto:path=/vclfcn-16/obj-test-zone3 2024-02-01T10:22:57.218 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3 2024-02-01T10:22:57.218 DEBUG:boto:path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:57.218 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:57.218 DEBUG:boto:Method: GET 2024-02-01T10:22:57.218 DEBUG:boto:Path: /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:57.219 DEBUG:boto:Data: 2024-02-01T10:22:57.219 DEBUG:boto:Headers: {} 2024-02-01T10:22:57.219 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:57.219 DEBUG:boto:Port: 8001 2024-02-01T10:22:57.219 DEBUG:boto:Params: {} 2024-02-01T10:22:57.219 DEBUG:boto:Token: None 2024-02-01T10:22:57.219 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:57 GMT /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:57.219 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:B2VOM72P/j/9crxQHbbF9O++luw= 2024-02-01T10:22:57.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:B2VOM72P/j/9crxQHbbF9O++luw=', 'Content-Length': '0'} 2024-02-01T10:22:57.222 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000080b24df57171375e-0065bb7101-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:57.264 DEBUG:boto:path=/vclfcn-16/obj-test-zone3 2024-02-01T10:22:57.264 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3 2024-02-01T10:22:57.264 DEBUG:boto:path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:57.265 DEBUG:boto:auth_path=/vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:57.265 DEBUG:boto:Method: GET 2024-02-01T10:22:57.265 DEBUG:boto:Path: /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:57.265 DEBUG:boto:Data: 2024-02-01T10:22:57.265 DEBUG:boto:Headers: {} 2024-02-01T10:22:57.265 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:57.265 DEBUG:boto:Port: 8000 2024-02-01T10:22:57.266 DEBUG:boto:Params: {} 2024-02-01T10:22:57.266 DEBUG:boto:Token: None 2024-02-01T10:22:57.266 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:57 GMT /vclfcn-16/obj-test-zone3?versionId=Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP 2024-02-01T10:22:57.266 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:B2VOM72P/j/9crxQHbbF9O++luw= 2024-02-01T10:22:57.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:B2VOM72P/j/9crxQHbbF9O++luw=', 'Content-Length': '0'} 2024-02-01T10:22:57.269 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'Om9gbyJlfjr3IKmiVBlQLwMOqJ.AVjP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000253dcead64a0ee80-0065bb7101-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:57.316 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:22:57.316 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:22:57.317 DEBUG:boto:path=/vclfcn-16/?prefix=obj-test-zone3 2024-02-01T10:22:57.317 DEBUG:boto:auth_path=/vclfcn-16/?prefix=obj-test-zone3 2024-02-01T10:22:57.317 DEBUG:boto:Method: GET 2024-02-01T10:22:57.317 DEBUG:boto:Path: /vclfcn-16/?prefix=obj-test-zone3 2024-02-01T10:22:57.317 DEBUG:boto:Data: 2024-02-01T10:22:57.317 DEBUG:boto:Headers: {} 2024-02-01T10:22:57.317 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:57.317 DEBUG:boto:Port: 8001 2024-02-01T10:22:57.317 DEBUG:boto:Params: {} 2024-02-01T10:22:57.318 DEBUG:boto:Token: None 2024-02-01T10:22:57.318 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:57 GMT /vclfcn-16/ 2024-02-01T10:22:57.318 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:OZcix+13StdoAG7BxmpKRz1uz+o= 2024-02-01T10:22:57.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:OZcix+13StdoAG7BxmpKRz1uz+o=', 'Content-Length': '0'} 2024-02-01T10:22:57.322 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077c42d7981fc14b6-0065bb7101-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:57.322 DEBUG:boto:b'vclfcn-16obj-test-zone31000falseobj-test-zone32024-02-01T10:20:50.297Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:57.323 DEBUG:boto:path=/vclfcn-16/ 2024-02-01T10:22:57.323 DEBUG:boto:auth_path=/vclfcn-16/ 2024-02-01T10:22:57.323 DEBUG:boto:path=/vclfcn-16/?prefix=obj-test-zone3 2024-02-01T10:22:57.323 DEBUG:boto:auth_path=/vclfcn-16/?prefix=obj-test-zone3 2024-02-01T10:22:57.323 DEBUG:boto:Method: GET 2024-02-01T10:22:57.323 DEBUG:boto:Path: /vclfcn-16/?prefix=obj-test-zone3 2024-02-01T10:22:57.324 DEBUG:boto:Data: 2024-02-01T10:22:57.324 DEBUG:boto:Headers: {} 2024-02-01T10:22:57.324 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:57.324 DEBUG:boto:Port: 8000 2024-02-01T10:22:57.324 DEBUG:boto:Params: {} 2024-02-01T10:22:57.324 DEBUG:boto:Token: None 2024-02-01T10:22:57.324 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:57 GMT /vclfcn-16/ 2024-02-01T10:22:57.324 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:OZcix+13StdoAG7BxmpKRz1uz+o= 2024-02-01T10:22:57.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:OZcix+13StdoAG7BxmpKRz1uz+o=', 'Content-Length': '0'} 2024-02-01T10:22:57.332 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e50a16225e47f21-0065bb7101-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:57.333 DEBUG:boto:b'vclfcn-16obj-test-zone31000falseobj-test-zone32024-02-01T10:20:50.297Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:57.333 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-16 zones={test-zone3, test-zone2} 2024-02-01T10:22:57.334 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=vclfcn-17 2024-02-01T10:22:57.334 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:22:57.334 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:22:57.335 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket vclfcn-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-02-01T10:22:57.385 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:22:57.385 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:22:57.454 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:22:57.456+0000 7fa10c8aa980 1 bucket sync caught up with source: 2024-02-01T10:22:57.454 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, 00000000011.76.6, , 00000000003.22.10, , 00000000007.72.10, , , , , ] 2024-02-01T10:22:57.454 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, 00000000011.76.6, , 00000000003.22.10, , 00000000007.72.10, , , , , ] 2024-02-01T10:22:57.454 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:22:57.456+0000 7fa10c8aa980 0 bucket checkpoint complete 2024-02-01T10:22:57.475 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=vclfcn-17 2024-02-01T10:22:57.476 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:22:57.476 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:22:57.476 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:22:57.526 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:22:57.526 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:22:57.586 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:22:57.588+0000 7f5ac3fd5980 1 bucket sync caught up with source: 2024-02-01T10:22:57.586 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, 00000000011.76.6, , 00000000003.22.10, , 00000000007.72.10, , , , , ] 2024-02-01T10:22:57.586 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, 00000000011.76.6, , 00000000003.22.10, , 00000000007.72.10, , , , , ] 2024-02-01T10:22:57.587 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:22:57.588+0000 7f5ac3fd5980 0 bucket checkpoint complete 2024-02-01T10:22:57.608 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=vclfcn-17 2024-02-01T10:22:57.608 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:22:57.608 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:22:57.608 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:22:57.648 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:22:57.648 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:22:57.710 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:22:57.712+0000 7fcea1234980 1 bucket sync caught up with source: 2024-02-01T10:22:57.710 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, 00000000003.54.10, , 00000000011.45.6, , 00000000008.50.10, , , , , ] 2024-02-01T10:22:57.710 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, 00000000003.54.10, , 00000000011.45.6, , 00000000008.50.10, , , , , ] 2024-02-01T10:22:57.710 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:22:57.712+0000 7fcea1234980 0 bucket checkpoint complete 2024-02-01T10:22:57.734 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=vclfcn-17 2024-02-01T10:22:57.734 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:22:57.734 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:22:57.734 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:22:57.823 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:22:57.823 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:22:57.884 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:22:57.888+0000 7fca1823c980 1 bucket sync caught up with source: 2024-02-01T10:22:57.884 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, 00000000003.54.10, , 00000000011.45.6, , 00000000008.50.10, , , , , ] 2024-02-01T10:22:57.884 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, 00000000003.54.10, , 00000000011.45.6, , 00000000008.50.10, , , , , ] 2024-02-01T10:22:57.884 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:22:57.888+0000 7fca1823c980 0 bucket checkpoint complete 2024-02-01T10:22:57.905 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=vclfcn-17 2024-02-01T10:22:57.906 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:22:57.906 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:22:57.906 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:22:57.950 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:22:57.950 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:22:58.002 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:22:58.004+0000 7fa680c5c980 1 bucket sync caught up with source: 2024-02-01T10:22:58.002 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, 00000000003.139.10, , 00000000003.71.10, , 00000000014.144.6, , , , , ] 2024-02-01T10:22:58.002 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, 00000000003.139.10, , 00000000003.71.10, , 00000000014.144.6, , , , , ] 2024-02-01T10:22:58.002 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:22:58.004+0000 7fa680c5c980 0 bucket checkpoint complete 2024-02-01T10:22:58.024 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=vclfcn-17 2024-02-01T10:22:58.024 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:22:58.025 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:22:58.025 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket vclfcn-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-02-01T10:22:58.075 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:22:58.075 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:22:58.139 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:22:58.140+0000 7fa842764980 1 bucket sync caught up with source: 2024-02-01T10:22:58.139 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, 00000000003.139.10, , 00000000003.71.10, , 00000000014.144.6, , , , , ] 2024-02-01T10:22:58.140 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, 00000000003.139.10, , 00000000003.71.10, , 00000000014.144.6, , , , , ] 2024-02-01T10:22:58.140 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:22:58.140+0000 7fa842764980 0 bucket checkpoint complete 2024-02-01T10:22:58.162 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-17 zones={test-zone2, test-zone1} 2024-02-01T10:22:58.162 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:58.162 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:58.162 DEBUG:boto:Method: HEAD 2024-02-01T10:22:58.162 DEBUG:boto:Path: /vclfcn-17/ 2024-02-01T10:22:58.163 DEBUG:boto:Data: 2024-02-01T10:22:58.163 DEBUG:boto:Headers: {} 2024-02-01T10:22:58.163 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:58.163 DEBUG:boto:Port: 8000 2024-02-01T10:22:58.163 DEBUG:boto:Params: {} 2024-02-01T10:22:58.163 DEBUG:boto:Token: None 2024-02-01T10:22:58.163 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:58 GMT /vclfcn-17/ 2024-02-01T10:22:58.163 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:OeXcTRgP4pp8qtBO8D3pesvpD3g= 2024-02-01T10:22:58.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:OeXcTRgP4pp8qtBO8D3pesvpD3g=', 'Content-Length': '0'} 2024-02-01T10:22:58.166 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003d8840fd82a54430-0065bb7102-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:22:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:58.166 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:58.166 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:58.166 DEBUG:boto:Method: HEAD 2024-02-01T10:22:58.166 DEBUG:boto:Path: /vclfcn-17/ 2024-02-01T10:22:58.166 DEBUG:boto:Data: 2024-02-01T10:22:58.167 DEBUG:boto:Headers: {} 2024-02-01T10:22:58.167 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:58.167 DEBUG:boto:Port: 8000 2024-02-01T10:22:58.167 DEBUG:boto:Params: {} 2024-02-01T10:22:58.167 DEBUG:boto:Token: None 2024-02-01T10:22:58.167 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:58 GMT /vclfcn-17/ 2024-02-01T10:22:58.167 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:OeXcTRgP4pp8qtBO8D3pesvpD3g= 2024-02-01T10:22:58.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:OeXcTRgP4pp8qtBO8D3pesvpD3g=', 'Content-Length': '0'} 2024-02-01T10:22:58.171 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002bb729955d3a35da-0065bb7102-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:22:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:58.172 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:22:58.172 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:58.172 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:58.172 DEBUG:boto:path=/vclfcn-17/?versions&max-keys=999 2024-02-01T10:22:58.172 DEBUG:boto:auth_path=/vclfcn-17/?versions&max-keys=999 2024-02-01T10:22:58.172 DEBUG:boto:Method: GET 2024-02-01T10:22:58.172 DEBUG:boto:Path: /vclfcn-17/?versions&max-keys=999 2024-02-01T10:22:58.172 DEBUG:boto:Data: 2024-02-01T10:22:58.173 DEBUG:boto:Headers: {} 2024-02-01T10:22:58.173 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:58.173 DEBUG:boto:Port: 8000 2024-02-01T10:22:58.173 DEBUG:boto:Params: {} 2024-02-01T10:22:58.173 DEBUG:boto:Token: None 2024-02-01T10:22:58.173 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:58 GMT /vclfcn-17/?versions 2024-02-01T10:22:58.174 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Y1ILOlfPrcRTmaYrLKEyvFwul9k= 2024-02-01T10:22:58.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Y1ILOlfPrcRTmaYrLKEyvFwul9k=', 'Content-Length': '0'} 2024-02-01T10:22:58.176 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e005d18f3bee7842-0065bb7102-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:58.176 DEBUG:boto:b'vclfcn-17999falsedummyhQXZXSD45t2CSRVMJOLMBXih54iMPmJtrue2024-02-01T10:20:14.297Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1xMxAGR4A-xEILepnzFPRgrWNTOLvitztrue2024-02-01T10:20:50.473Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1true2024-02-01T10:20:50.624Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3YE736OwOOXbZJHJe7d8YvcNV1LxwcQbtrue2024-02-01T10:20:50.780Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:58.177 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-02-01T10:22:58.177 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-02-01T10:22:58.177 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-02-01T10:22:58.177 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-02-01T10:22:58.177 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:22:58.177 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:58.177 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:58.177 DEBUG:boto:path=/vclfcn-17/?versions&max-keys=999 2024-02-01T10:22:58.177 DEBUG:boto:auth_path=/vclfcn-17/?versions&max-keys=999 2024-02-01T10:22:58.177 DEBUG:boto:Method: GET 2024-02-01T10:22:58.178 DEBUG:boto:Path: /vclfcn-17/?versions&max-keys=999 2024-02-01T10:22:58.178 DEBUG:boto:Data: 2024-02-01T10:22:58.178 DEBUG:boto:Headers: {} 2024-02-01T10:22:58.178 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:58.178 DEBUG:boto:Port: 8000 2024-02-01T10:22:58.178 DEBUG:boto:Params: {} 2024-02-01T10:22:58.178 DEBUG:boto:Token: None 2024-02-01T10:22:58.178 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:58 GMT /vclfcn-17/?versions 2024-02-01T10:22:58.179 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Y1ILOlfPrcRTmaYrLKEyvFwul9k= 2024-02-01T10:22:58.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Y1ILOlfPrcRTmaYrLKEyvFwul9k=', 'Content-Length': '0'} 2024-02-01T10:22:58.183 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047a9e8fda95861e1-0065bb7102-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:58.183 DEBUG:boto:b'vclfcn-17999falsedummyhQXZXSD45t2CSRVMJOLMBXih54iMPmJtrue2024-02-01T10:20:14.297Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1xMxAGR4A-xEILepnzFPRgrWNTOLvitztrue2024-02-01T10:20:50.473Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1true2024-02-01T10:20:50.624Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3YE736OwOOXbZJHJe7d8YvcNV1LxwcQbtrue2024-02-01T10:20:50.780Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:58.184 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-02-01T10:22:58.184 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-02-01T10:22:58.184 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-02-01T10:22:58.184 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-02-01T10:22:58.184 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:58.185 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:58.185 DEBUG:boto:path=/vclfcn-17/?versions&max-keys=999 2024-02-01T10:22:58.185 DEBUG:boto:auth_path=/vclfcn-17/?versions&max-keys=999 2024-02-01T10:22:58.185 DEBUG:boto:Method: GET 2024-02-01T10:22:58.185 DEBUG:boto:Path: /vclfcn-17/?versions&max-keys=999 2024-02-01T10:22:58.185 DEBUG:boto:Data: 2024-02-01T10:22:58.185 DEBUG:boto:Headers: {} 2024-02-01T10:22:58.185 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:58.185 DEBUG:boto:Port: 8000 2024-02-01T10:22:58.185 DEBUG:boto:Params: {} 2024-02-01T10:22:58.185 DEBUG:boto:Token: None 2024-02-01T10:22:58.186 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:58 GMT /vclfcn-17/?versions 2024-02-01T10:22:58.186 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Y1ILOlfPrcRTmaYrLKEyvFwul9k= 2024-02-01T10:22:58.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Y1ILOlfPrcRTmaYrLKEyvFwul9k=', 'Content-Length': '0'} 2024-02-01T10:22:58.188 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b5910a0eb4155cc4-0065bb7102-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:58.188 DEBUG:boto:b'vclfcn-17999falsedummyhQXZXSD45t2CSRVMJOLMBXih54iMPmJtrue2024-02-01T10:20:14.297Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1xMxAGR4A-xEILepnzFPRgrWNTOLvitztrue2024-02-01T10:20:50.473Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1true2024-02-01T10:20:50.624Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3YE736OwOOXbZJHJe7d8YvcNV1LxwcQbtrue2024-02-01T10:20:50.780Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:58.189 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:58.189 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:58.189 DEBUG:boto:path=/vclfcn-17/?versions&max-keys=999 2024-02-01T10:22:58.189 DEBUG:boto:auth_path=/vclfcn-17/?versions&max-keys=999 2024-02-01T10:22:58.189 DEBUG:boto:Method: GET 2024-02-01T10:22:58.189 DEBUG:boto:Path: /vclfcn-17/?versions&max-keys=999 2024-02-01T10:22:58.189 DEBUG:boto:Data: 2024-02-01T10:22:58.189 DEBUG:boto:Headers: {} 2024-02-01T10:22:58.189 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:58.189 DEBUG:boto:Port: 8000 2024-02-01T10:22:58.190 DEBUG:boto:Params: {} 2024-02-01T10:22:58.190 DEBUG:boto:Token: None 2024-02-01T10:22:58.190 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:58 GMT /vclfcn-17/?versions 2024-02-01T10:22:58.190 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Y1ILOlfPrcRTmaYrLKEyvFwul9k= 2024-02-01T10:22:58.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Y1ILOlfPrcRTmaYrLKEyvFwul9k=', 'Content-Length': '0'} 2024-02-01T10:22:58.194 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026edfbed08b5c4bc-0065bb7102-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:58.195 DEBUG:boto:b'vclfcn-17999falsedummyhQXZXSD45t2CSRVMJOLMBXih54iMPmJtrue2024-02-01T10:20:14.297Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1xMxAGR4A-xEILepnzFPRgrWNTOLvitztrue2024-02-01T10:20:50.473Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1true2024-02-01T10:20:50.624Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3YE736OwOOXbZJHJe7d8YvcNV1LxwcQbtrue2024-02-01T10:20:50.780Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:58.195 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-02-01T10:22:58.195 DEBUG:boto:path=/vclfcn-17/dummy 2024-02-01T10:22:58.195 DEBUG:boto:auth_path=/vclfcn-17/dummy 2024-02-01T10:22:58.195 DEBUG:boto:path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:58.195 DEBUG:boto:auth_path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:58.196 DEBUG:boto:Method: GET 2024-02-01T10:22:58.196 DEBUG:boto:Path: /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:58.196 DEBUG:boto:Data: 2024-02-01T10:22:58.196 DEBUG:boto:Headers: {} 2024-02-01T10:22:58.196 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:58.196 DEBUG:boto:Port: 8000 2024-02-01T10:22:58.196 DEBUG:boto:Params: {} 2024-02-01T10:22:58.196 DEBUG:boto:Token: None 2024-02-01T10:22:58.196 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:58 GMT /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:58.196 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GQzIcX2RgAEaunm7Y+ecviMT3Pc= 2024-02-01T10:22:58.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GQzIcX2RgAEaunm7Y+ecviMT3Pc=', 'Content-Length': '0'} 2024-02-01T10:22:58.198 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:14 GMT'), ('x-amz-version-id', 'hQXZXSD45t2CSRVMJOLMBXih54iMPmJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009ec9860e05dc9a1c-0065bb7102-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:58.198 DEBUG:boto:path=/vclfcn-17/dummy 2024-02-01T10:22:58.198 DEBUG:boto:auth_path=/vclfcn-17/dummy 2024-02-01T10:22:58.199 DEBUG:boto:path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:58.199 DEBUG:boto:auth_path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:58.199 DEBUG:boto:Method: GET 2024-02-01T10:22:58.199 DEBUG:boto:Path: /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:58.199 DEBUG:boto:Data: 2024-02-01T10:22:58.199 DEBUG:boto:Headers: {} 2024-02-01T10:22:58.199 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:58.199 DEBUG:boto:Port: 8000 2024-02-01T10:22:58.199 DEBUG:boto:Params: {} 2024-02-01T10:22:58.199 DEBUG:boto:Token: None 2024-02-01T10:22:58.199 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:58 GMT /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:58.200 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GQzIcX2RgAEaunm7Y+ecviMT3Pc= 2024-02-01T10:22:58.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GQzIcX2RgAEaunm7Y+ecviMT3Pc=', 'Content-Length': '0'} 2024-02-01T10:22:58.202 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:14 GMT'), ('x-amz-version-id', 'hQXZXSD45t2CSRVMJOLMBXih54iMPmJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c3a13b000dcf6727-0065bb7102-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:58.203 DEBUG:boto:path=/vclfcn-17/dummy 2024-02-01T10:22:58.203 DEBUG:boto:auth_path=/vclfcn-17/dummy 2024-02-01T10:22:58.203 DEBUG:boto:path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:58.203 DEBUG:boto:auth_path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:58.203 DEBUG:boto:Method: HEAD 2024-02-01T10:22:58.203 DEBUG:boto:Path: /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:58.203 DEBUG:boto:Data: 2024-02-01T10:22:58.203 DEBUG:boto:Headers: {} 2024-02-01T10:22:58.203 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:58.203 DEBUG:boto:Port: 8000 2024-02-01T10:22:58.204 DEBUG:boto:Params: {} 2024-02-01T10:22:58.204 DEBUG:boto:Token: None 2024-02-01T10:22:58.204 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:58 GMT /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:58.204 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:N629AZk4qeje40W0QkIGm8K/azE= 2024-02-01T10:22:58.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:N629AZk4qeje40W0QkIGm8K/azE=', 'Content-Length': '0'} 2024-02-01T10:22:58.206 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:14 GMT'), ('x-amz-version-id', 'hQXZXSD45t2CSRVMJOLMBXih54iMPmJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006fbaa540ce85c2a3-0065bb7102-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:58.206 DEBUG:boto:path=/vclfcn-17/dummy 2024-02-01T10:22:58.206 DEBUG:boto:auth_path=/vclfcn-17/dummy 2024-02-01T10:22:58.206 DEBUG:boto:path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:58.206 DEBUG:boto:auth_path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:58.206 DEBUG:boto:Method: HEAD 2024-02-01T10:22:58.206 DEBUG:boto:Path: /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:58.207 DEBUG:boto:Data: 2024-02-01T10:22:58.207 DEBUG:boto:Headers: {} 2024-02-01T10:22:58.207 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:58.207 DEBUG:boto:Port: 8000 2024-02-01T10:22:58.207 DEBUG:boto:Params: {} 2024-02-01T10:22:58.207 DEBUG:boto:Token: None 2024-02-01T10:22:58.207 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:58 GMT /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:58.207 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:N629AZk4qeje40W0QkIGm8K/azE= 2024-02-01T10:22:58.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:N629AZk4qeje40W0QkIGm8K/azE=', 'Content-Length': '0'} 2024-02-01T10:22:58.210 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:14 GMT'), ('x-amz-version-id', 'hQXZXSD45t2CSRVMJOLMBXih54iMPmJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d26615c48ea9409a-0065bb7102-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:58.210 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-02-01T10:22:58.210 DEBUG:boto:path=/vclfcn-17/dummy 2024-02-01T10:22:58.211 DEBUG:boto:auth_path=/vclfcn-17/dummy 2024-02-01T10:22:58.211 DEBUG:boto:path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:58.211 DEBUG:boto:auth_path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:58.211 DEBUG:boto:Method: GET 2024-02-01T10:22:58.211 DEBUG:boto:Path: /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:58.211 DEBUG:boto:Data: 2024-02-01T10:22:58.211 DEBUG:boto:Headers: {} 2024-02-01T10:22:58.211 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:58.211 DEBUG:boto:Port: 8000 2024-02-01T10:22:58.211 DEBUG:boto:Params: {} 2024-02-01T10:22:58.211 DEBUG:boto:Token: None 2024-02-01T10:22:58.212 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:58 GMT /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:58.212 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GQzIcX2RgAEaunm7Y+ecviMT3Pc= 2024-02-01T10:22:58.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GQzIcX2RgAEaunm7Y+ecviMT3Pc=', 'Content-Length': '0'} 2024-02-01T10:22:58.214 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:14 GMT'), ('x-amz-version-id', 'hQXZXSD45t2CSRVMJOLMBXih54iMPmJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ee0c2cb00a66c12f-0065bb7102-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:58.214 DEBUG:boto:path=/vclfcn-17/dummy 2024-02-01T10:22:58.214 DEBUG:boto:auth_path=/vclfcn-17/dummy 2024-02-01T10:22:58.214 DEBUG:boto:path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:58.214 DEBUG:boto:auth_path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:58.215 DEBUG:boto:Method: GET 2024-02-01T10:22:58.215 DEBUG:boto:Path: /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:58.215 DEBUG:boto:Data: 2024-02-01T10:22:58.215 DEBUG:boto:Headers: {} 2024-02-01T10:22:58.215 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:58.215 DEBUG:boto:Port: 8000 2024-02-01T10:22:58.215 DEBUG:boto:Params: {} 2024-02-01T10:22:58.215 DEBUG:boto:Token: None 2024-02-01T10:22:58.215 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:58 GMT /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:58.215 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GQzIcX2RgAEaunm7Y+ecviMT3Pc= 2024-02-01T10:22:58.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GQzIcX2RgAEaunm7Y+ecviMT3Pc=', 'Content-Length': '0'} 2024-02-01T10:22:58.218 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:14 GMT'), ('x-amz-version-id', 'hQXZXSD45t2CSRVMJOLMBXih54iMPmJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c0e1496209afb2a9-0065bb7102-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:58.218 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:58.218 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:58.218 DEBUG:boto:path=/vclfcn-17/?prefix=dummy 2024-02-01T10:22:58.219 DEBUG:boto:auth_path=/vclfcn-17/?prefix=dummy 2024-02-01T10:22:58.219 DEBUG:boto:Method: GET 2024-02-01T10:22:58.219 DEBUG:boto:Path: /vclfcn-17/?prefix=dummy 2024-02-01T10:22:58.219 DEBUG:boto:Data: 2024-02-01T10:22:58.219 DEBUG:boto:Headers: {} 2024-02-01T10:22:58.219 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:58.219 DEBUG:boto:Port: 8000 2024-02-01T10:22:58.219 DEBUG:boto:Params: {} 2024-02-01T10:22:58.219 DEBUG:boto:Token: None 2024-02-01T10:22:58.219 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:58 GMT /vclfcn-17/ 2024-02-01T10:22:58.219 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:spcR2uRdujlkBoT8m7wk4oTwUj8= 2024-02-01T10:22:58.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:spcR2uRdujlkBoT8m7wk4oTwUj8=', 'Content-Length': '0'} 2024-02-01T10:22:58.222 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff3b615129faadff-0065bb7102-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:58.264 DEBUG:boto:b'vclfcn-17dummy1000falsedummy2024-02-01T10:20:14.297Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:58.265 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:58.265 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:58.265 DEBUG:boto:path=/vclfcn-17/?prefix=dummy 2024-02-01T10:22:58.265 DEBUG:boto:auth_path=/vclfcn-17/?prefix=dummy 2024-02-01T10:22:58.265 DEBUG:boto:Method: GET 2024-02-01T10:22:58.265 DEBUG:boto:Path: /vclfcn-17/?prefix=dummy 2024-02-01T10:22:58.265 DEBUG:boto:Data: 2024-02-01T10:22:58.265 DEBUG:boto:Headers: {} 2024-02-01T10:22:58.266 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:58.266 DEBUG:boto:Port: 8000 2024-02-01T10:22:58.266 DEBUG:boto:Params: {} 2024-02-01T10:22:58.266 DEBUG:boto:Token: None 2024-02-01T10:22:58.266 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:58 GMT /vclfcn-17/ 2024-02-01T10:22:58.266 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:spcR2uRdujlkBoT8m7wk4oTwUj8= 2024-02-01T10:22:58.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:spcR2uRdujlkBoT8m7wk4oTwUj8=', 'Content-Length': '0'} 2024-02-01T10:22:58.270 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d0260a77225d5164-0065bb7102-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:58.270 DEBUG:boto:b'vclfcn-17dummy1000falsedummy2024-02-01T10:20:14.297Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:58.271 DEBUG:boto:path=/vclfcn-17/dummy 2024-02-01T10:22:58.271 DEBUG:boto:auth_path=/vclfcn-17/dummy 2024-02-01T10:22:58.271 DEBUG:boto:Method: HEAD 2024-02-01T10:22:58.271 DEBUG:boto:Path: /vclfcn-17/dummy 2024-02-01T10:22:58.271 DEBUG:boto:Data: 2024-02-01T10:22:58.271 DEBUG:boto:Headers: {} 2024-02-01T10:22:58.271 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:58.272 DEBUG:boto:Port: 8000 2024-02-01T10:22:58.272 DEBUG:boto:Params: {} 2024-02-01T10:22:58.272 DEBUG:boto:Token: None 2024-02-01T10:22:58.272 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:58 GMT /vclfcn-17/dummy 2024-02-01T10:22:58.272 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:w0/s9rMtGcu1i2AKTaz5Wh4cBGE= 2024-02-01T10:22:58.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:w0/s9rMtGcu1i2AKTaz5Wh4cBGE=', 'Content-Length': '0'} 2024-02-01T10:22:58.276 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:14 GMT'), ('x-amz-version-id', 'hQXZXSD45t2CSRVMJOLMBXih54iMPmJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000f4426ba4984f2b8-0065bb7102-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:58.276 DEBUG:boto:path=/vclfcn-17/dummy 2024-02-01T10:22:58.276 DEBUG:boto:auth_path=/vclfcn-17/dummy 2024-02-01T10:22:58.276 DEBUG:boto:Method: HEAD 2024-02-01T10:22:58.276 DEBUG:boto:Path: /vclfcn-17/dummy 2024-02-01T10:22:58.276 DEBUG:boto:Data: 2024-02-01T10:22:58.276 DEBUG:boto:Headers: {} 2024-02-01T10:22:58.277 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:58.277 DEBUG:boto:Port: 8000 2024-02-01T10:22:58.277 DEBUG:boto:Params: {} 2024-02-01T10:22:58.277 DEBUG:boto:Token: None 2024-02-01T10:22:58.277 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:58 GMT /vclfcn-17/dummy 2024-02-01T10:22:58.277 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:w0/s9rMtGcu1i2AKTaz5Wh4cBGE= 2024-02-01T10:22:58.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:w0/s9rMtGcu1i2AKTaz5Wh4cBGE=', 'Content-Length': '0'} 2024-02-01T10:22:58.281 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:14 GMT'), ('x-amz-version-id', 'hQXZXSD45t2CSRVMJOLMBXih54iMPmJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000fd1eb907e395b25-0065bb7102-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:58.281 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-02-01T10:22:58.281 DEBUG:boto:path=/vclfcn-17/dummy 2024-02-01T10:22:58.282 DEBUG:boto:auth_path=/vclfcn-17/dummy 2024-02-01T10:22:58.282 DEBUG:boto:path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:58.282 DEBUG:boto:auth_path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:58.282 DEBUG:boto:Method: GET 2024-02-01T10:22:58.282 DEBUG:boto:Path: /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:58.282 DEBUG:boto:Data: 2024-02-01T10:22:58.282 DEBUG:boto:Headers: {} 2024-02-01T10:22:58.282 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:58.282 DEBUG:boto:Port: 8000 2024-02-01T10:22:58.282 DEBUG:boto:Params: {} 2024-02-01T10:22:58.282 DEBUG:boto:Token: None 2024-02-01T10:22:58.283 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:58 GMT /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:58.283 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GQzIcX2RgAEaunm7Y+ecviMT3Pc= 2024-02-01T10:22:58.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GQzIcX2RgAEaunm7Y+ecviMT3Pc=', 'Content-Length': '0'} 2024-02-01T10:22:58.286 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:14 GMT'), ('x-amz-version-id', 'hQXZXSD45t2CSRVMJOLMBXih54iMPmJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007d25bea27d780b15-0065bb7102-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:58.286 DEBUG:boto:path=/vclfcn-17/dummy 2024-02-01T10:22:58.286 DEBUG:boto:auth_path=/vclfcn-17/dummy 2024-02-01T10:22:58.286 DEBUG:boto:path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:58.286 DEBUG:boto:auth_path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:58.286 DEBUG:boto:Method: GET 2024-02-01T10:22:58.286 DEBUG:boto:Path: /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:58.287 DEBUG:boto:Data: 2024-02-01T10:22:58.287 DEBUG:boto:Headers: {} 2024-02-01T10:22:58.287 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:58.287 DEBUG:boto:Port: 8000 2024-02-01T10:22:58.287 DEBUG:boto:Params: {} 2024-02-01T10:22:58.287 DEBUG:boto:Token: None 2024-02-01T10:22:58.287 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:58 GMT /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:58.287 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GQzIcX2RgAEaunm7Y+ecviMT3Pc= 2024-02-01T10:22:58.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GQzIcX2RgAEaunm7Y+ecviMT3Pc=', 'Content-Length': '0'} 2024-02-01T10:22:58.290 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:14 GMT'), ('x-amz-version-id', 'hQXZXSD45t2CSRVMJOLMBXih54iMPmJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000024cc5bc636098246-0065bb7102-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:58.291 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:58.291 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:58.291 DEBUG:boto:path=/vclfcn-17/?prefix=dummy 2024-02-01T10:22:58.291 DEBUG:boto:auth_path=/vclfcn-17/?prefix=dummy 2024-02-01T10:22:58.291 DEBUG:boto:Method: GET 2024-02-01T10:22:58.291 DEBUG:boto:Path: /vclfcn-17/?prefix=dummy 2024-02-01T10:22:58.291 DEBUG:boto:Data: 2024-02-01T10:22:58.291 DEBUG:boto:Headers: {} 2024-02-01T10:22:58.291 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:58.292 DEBUG:boto:Port: 8000 2024-02-01T10:22:58.292 DEBUG:boto:Params: {} 2024-02-01T10:22:58.292 DEBUG:boto:Token: None 2024-02-01T10:22:58.292 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:58 GMT /vclfcn-17/ 2024-02-01T10:22:58.292 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:spcR2uRdujlkBoT8m7wk4oTwUj8= 2024-02-01T10:22:58.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:spcR2uRdujlkBoT8m7wk4oTwUj8=', 'Content-Length': '0'} 2024-02-01T10:22:58.296 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c7befaf277a1fef9-0065bb7102-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:58.296 DEBUG:boto:b'vclfcn-17dummy1000falsedummy2024-02-01T10:20:14.297Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:58.297 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:58.297 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:58.297 DEBUG:boto:path=/vclfcn-17/?prefix=dummy 2024-02-01T10:22:58.297 DEBUG:boto:auth_path=/vclfcn-17/?prefix=dummy 2024-02-01T10:22:58.297 DEBUG:boto:Method: GET 2024-02-01T10:22:58.297 DEBUG:boto:Path: /vclfcn-17/?prefix=dummy 2024-02-01T10:22:58.297 DEBUG:boto:Data: 2024-02-01T10:22:58.297 DEBUG:boto:Headers: {} 2024-02-01T10:22:58.298 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:58.298 DEBUG:boto:Port: 8000 2024-02-01T10:22:58.298 DEBUG:boto:Params: {} 2024-02-01T10:22:58.298 DEBUG:boto:Token: None 2024-02-01T10:22:58.298 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:58 GMT /vclfcn-17/ 2024-02-01T10:22:58.298 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:spcR2uRdujlkBoT8m7wk4oTwUj8= 2024-02-01T10:22:58.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:spcR2uRdujlkBoT8m7wk4oTwUj8=', 'Content-Length': '0'} 2024-02-01T10:22:58.302 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dfed82f9c4b14072-0065bb7102-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:58.303 DEBUG:boto:b'vclfcn-17dummy1000falsedummy2024-02-01T10:20:14.297Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:58.303 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-02-01T10:22:58.303 DEBUG:boto:path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:58.303 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:58.304 DEBUG:boto:path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:58.304 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:58.304 DEBUG:boto:Method: GET 2024-02-01T10:22:58.304 DEBUG:boto:Path: /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:58.304 DEBUG:boto:Data: 2024-02-01T10:22:58.304 DEBUG:boto:Headers: {} 2024-02-01T10:22:58.304 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:58.304 DEBUG:boto:Port: 8000 2024-02-01T10:22:58.304 DEBUG:boto:Params: {} 2024-02-01T10:22:58.304 DEBUG:boto:Token: None 2024-02-01T10:22:58.305 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:58 GMT /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:58.305 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:le9Y3mUQYWlRCrksGEG+ELLxScY= 2024-02-01T10:22:58.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:le9Y3mUQYWlRCrksGEG+ELLxScY=', 'Content-Length': '0'} 2024-02-01T10:22:58.308 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'xMxAGR4A-xEILepnzFPRgrWNTOLvitz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000122dd941376da5bb-0065bb7102-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:58.348 DEBUG:boto:path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:58.348 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:58.348 DEBUG:boto:path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:58.348 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:58.349 DEBUG:boto:Method: GET 2024-02-01T10:22:58.349 DEBUG:boto:Path: /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:58.349 DEBUG:boto:Data: 2024-02-01T10:22:58.349 DEBUG:boto:Headers: {} 2024-02-01T10:22:58.349 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:58.349 DEBUG:boto:Port: 8000 2024-02-01T10:22:58.349 DEBUG:boto:Params: {} 2024-02-01T10:22:58.349 DEBUG:boto:Token: None 2024-02-01T10:22:58.349 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:58 GMT /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:58.350 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:le9Y3mUQYWlRCrksGEG+ELLxScY= 2024-02-01T10:22:58.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:le9Y3mUQYWlRCrksGEG+ELLxScY=', 'Content-Length': '0'} 2024-02-01T10:22:58.353 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'xMxAGR4A-xEILepnzFPRgrWNTOLvitz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000041bd8d5d8022ff8e-0065bb7102-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:58.354 DEBUG:boto:path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:58.354 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:58.354 DEBUG:boto:path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:58.354 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:58.354 DEBUG:boto:Method: HEAD 2024-02-01T10:22:58.354 DEBUG:boto:Path: /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:58.354 DEBUG:boto:Data: 2024-02-01T10:22:58.354 DEBUG:boto:Headers: {} 2024-02-01T10:22:58.354 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:58.354 DEBUG:boto:Port: 8000 2024-02-01T10:22:58.355 DEBUG:boto:Params: {} 2024-02-01T10:22:58.355 DEBUG:boto:Token: None 2024-02-01T10:22:58.355 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:58 GMT /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:58.355 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:MKSx8BOljlbuVoyVLZJb6uZI46s= 2024-02-01T10:22:58.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:MKSx8BOljlbuVoyVLZJb6uZI46s=', 'Content-Length': '0'} 2024-02-01T10:22:58.358 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'xMxAGR4A-xEILepnzFPRgrWNTOLvitz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000016b2c82c0c1e853-0065bb7102-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:58.358 DEBUG:boto:path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:58.358 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:58.359 DEBUG:boto:path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:58.359 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:58.359 DEBUG:boto:Method: HEAD 2024-02-01T10:22:58.359 DEBUG:boto:Path: /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:58.359 DEBUG:boto:Data: 2024-02-01T10:22:58.359 DEBUG:boto:Headers: {} 2024-02-01T10:22:58.359 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:58.359 DEBUG:boto:Port: 8000 2024-02-01T10:22:58.359 DEBUG:boto:Params: {} 2024-02-01T10:22:58.359 DEBUG:boto:Token: None 2024-02-01T10:22:58.359 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:58 GMT /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:58.360 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:MKSx8BOljlbuVoyVLZJb6uZI46s= 2024-02-01T10:22:58.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:MKSx8BOljlbuVoyVLZJb6uZI46s=', 'Content-Length': '0'} 2024-02-01T10:22:58.362 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'xMxAGR4A-xEILepnzFPRgrWNTOLvitz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000939454df90514a06-0065bb7102-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:58.363 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-02-01T10:22:58.363 DEBUG:boto:path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:58.363 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:58.363 DEBUG:boto:path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:58.363 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:58.363 DEBUG:boto:Method: GET 2024-02-01T10:22:58.363 DEBUG:boto:Path: /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:58.364 DEBUG:boto:Data: 2024-02-01T10:22:58.364 DEBUG:boto:Headers: {} 2024-02-01T10:22:58.364 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:58.364 DEBUG:boto:Port: 8000 2024-02-01T10:22:58.364 DEBUG:boto:Params: {} 2024-02-01T10:22:58.364 DEBUG:boto:Token: None 2024-02-01T10:22:58.364 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:58 GMT /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:58.364 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:le9Y3mUQYWlRCrksGEG+ELLxScY= 2024-02-01T10:22:58.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:le9Y3mUQYWlRCrksGEG+ELLxScY=', 'Content-Length': '0'} 2024-02-01T10:22:58.368 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'xMxAGR4A-xEILepnzFPRgrWNTOLvitz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008fda93e2eede47b7-0065bb7102-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:58.412 DEBUG:boto:path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:58.412 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:58.412 DEBUG:boto:path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:58.412 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:58.413 DEBUG:boto:Method: GET 2024-02-01T10:22:58.413 DEBUG:boto:Path: /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:58.413 DEBUG:boto:Data: 2024-02-01T10:22:58.413 DEBUG:boto:Headers: {} 2024-02-01T10:22:58.413 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:58.413 DEBUG:boto:Port: 8000 2024-02-01T10:22:58.413 DEBUG:boto:Params: {} 2024-02-01T10:22:58.413 DEBUG:boto:Token: None 2024-02-01T10:22:58.413 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:58 GMT /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:58.414 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:le9Y3mUQYWlRCrksGEG+ELLxScY= 2024-02-01T10:22:58.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:le9Y3mUQYWlRCrksGEG+ELLxScY=', 'Content-Length': '0'} 2024-02-01T10:22:58.416 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'xMxAGR4A-xEILepnzFPRgrWNTOLvitz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fe41b41b1e2bf963-0065bb7102-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:58.417 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:58.417 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:58.417 DEBUG:boto:path=/vclfcn-17/?prefix=obj-test-zone1 2024-02-01T10:22:58.417 DEBUG:boto:auth_path=/vclfcn-17/?prefix=obj-test-zone1 2024-02-01T10:22:58.417 DEBUG:boto:Method: GET 2024-02-01T10:22:58.417 DEBUG:boto:Path: /vclfcn-17/?prefix=obj-test-zone1 2024-02-01T10:22:58.417 DEBUG:boto:Data: 2024-02-01T10:22:58.418 DEBUG:boto:Headers: {} 2024-02-01T10:22:58.418 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:58.418 DEBUG:boto:Port: 8000 2024-02-01T10:22:58.418 DEBUG:boto:Params: {} 2024-02-01T10:22:58.418 DEBUG:boto:Token: None 2024-02-01T10:22:58.418 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:58 GMT /vclfcn-17/ 2024-02-01T10:22:58.418 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:spcR2uRdujlkBoT8m7wk4oTwUj8= 2024-02-01T10:22:58.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:spcR2uRdujlkBoT8m7wk4oTwUj8=', 'Content-Length': '0'} 2024-02-01T10:22:58.424 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a7075a2d0b4d74c-0065bb7102-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:58.464 DEBUG:boto:b'vclfcn-17obj-test-zone11000falseobj-test-zone12024-02-01T10:20:50.473Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:58.465 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:58.465 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:58.465 DEBUG:boto:path=/vclfcn-17/?prefix=obj-test-zone1 2024-02-01T10:22:58.465 DEBUG:boto:auth_path=/vclfcn-17/?prefix=obj-test-zone1 2024-02-01T10:22:58.465 DEBUG:boto:Method: GET 2024-02-01T10:22:58.465 DEBUG:boto:Path: /vclfcn-17/?prefix=obj-test-zone1 2024-02-01T10:22:58.465 DEBUG:boto:Data: 2024-02-01T10:22:58.465 DEBUG:boto:Headers: {} 2024-02-01T10:22:58.465 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:58.466 DEBUG:boto:Port: 8000 2024-02-01T10:22:58.466 DEBUG:boto:Params: {} 2024-02-01T10:22:58.466 DEBUG:boto:Token: None 2024-02-01T10:22:58.466 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:58 GMT /vclfcn-17/ 2024-02-01T10:22:58.466 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:spcR2uRdujlkBoT8m7wk4oTwUj8= 2024-02-01T10:22:58.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:spcR2uRdujlkBoT8m7wk4oTwUj8=', 'Content-Length': '0'} 2024-02-01T10:22:58.471 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001be60193434b4d67-0065bb7102-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:58.471 DEBUG:boto:b'vclfcn-17obj-test-zone11000falseobj-test-zone12024-02-01T10:20:50.473Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:58.471 DEBUG:boto:path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:58.471 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:58.472 DEBUG:boto:Method: HEAD 2024-02-01T10:22:58.472 DEBUG:boto:Path: /vclfcn-17/obj-test-zone1 2024-02-01T10:22:58.472 DEBUG:boto:Data: 2024-02-01T10:22:58.472 DEBUG:boto:Headers: {} 2024-02-01T10:22:58.472 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:58.472 DEBUG:boto:Port: 8000 2024-02-01T10:22:58.472 DEBUG:boto:Params: {} 2024-02-01T10:22:58.472 DEBUG:boto:Token: None 2024-02-01T10:22:58.472 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:58 GMT /vclfcn-17/obj-test-zone1 2024-02-01T10:22:58.472 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:frIlRVSpb+X6SBPdRNAzhXM+1es= 2024-02-01T10:22:58.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:frIlRVSpb+X6SBPdRNAzhXM+1es=', 'Content-Length': '0'} 2024-02-01T10:22:58.476 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'xMxAGR4A-xEILepnzFPRgrWNTOLvitz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bc39bbab73bbd055-0065bb7102-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:58.477 DEBUG:boto:path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:58.477 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:58.477 DEBUG:boto:Method: HEAD 2024-02-01T10:22:58.477 DEBUG:boto:Path: /vclfcn-17/obj-test-zone1 2024-02-01T10:22:58.477 DEBUG:boto:Data: 2024-02-01T10:22:58.477 DEBUG:boto:Headers: {} 2024-02-01T10:22:58.477 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:58.477 DEBUG:boto:Port: 8000 2024-02-01T10:22:58.477 DEBUG:boto:Params: {} 2024-02-01T10:22:58.478 DEBUG:boto:Token: None 2024-02-01T10:22:58.478 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:58 GMT /vclfcn-17/obj-test-zone1 2024-02-01T10:22:58.478 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:frIlRVSpb+X6SBPdRNAzhXM+1es= 2024-02-01T10:22:58.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:frIlRVSpb+X6SBPdRNAzhXM+1es=', 'Content-Length': '0'} 2024-02-01T10:22:58.481 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'xMxAGR4A-xEILepnzFPRgrWNTOLvitz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000030edb56a488ca5c5-0065bb7102-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:58.482 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-02-01T10:22:58.482 DEBUG:boto:path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:58.482 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:58.482 DEBUG:boto:path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:58.482 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:58.482 DEBUG:boto:Method: GET 2024-02-01T10:22:58.482 DEBUG:boto:Path: /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:58.482 DEBUG:boto:Data: 2024-02-01T10:22:58.483 DEBUG:boto:Headers: {} 2024-02-01T10:22:58.483 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:58.483 DEBUG:boto:Port: 8000 2024-02-01T10:22:58.483 DEBUG:boto:Params: {} 2024-02-01T10:22:58.483 DEBUG:boto:Token: None 2024-02-01T10:22:58.483 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:58 GMT /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:58.483 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:le9Y3mUQYWlRCrksGEG+ELLxScY= 2024-02-01T10:22:58.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:le9Y3mUQYWlRCrksGEG+ELLxScY=', 'Content-Length': '0'} 2024-02-01T10:22:58.486 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'xMxAGR4A-xEILepnzFPRgrWNTOLvitz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000668f9c9122a5753-0065bb7102-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:58.532 DEBUG:boto:path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:58.532 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:58.532 DEBUG:boto:path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:58.532 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:58.533 DEBUG:boto:Method: GET 2024-02-01T10:22:58.533 DEBUG:boto:Path: /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:58.533 DEBUG:boto:Data: 2024-02-01T10:22:58.533 DEBUG:boto:Headers: {} 2024-02-01T10:22:58.533 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:58.533 DEBUG:boto:Port: 8000 2024-02-01T10:22:58.533 DEBUG:boto:Params: {} 2024-02-01T10:22:58.533 DEBUG:boto:Token: None 2024-02-01T10:22:58.533 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:58 GMT /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:58.534 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:le9Y3mUQYWlRCrksGEG+ELLxScY= 2024-02-01T10:22:58.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:le9Y3mUQYWlRCrksGEG+ELLxScY=', 'Content-Length': '0'} 2024-02-01T10:22:58.537 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'xMxAGR4A-xEILepnzFPRgrWNTOLvitz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a0e2115ab03d05df-0065bb7102-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:58.537 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:58.537 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:58.537 DEBUG:boto:path=/vclfcn-17/?prefix=obj-test-zone1 2024-02-01T10:22:58.538 DEBUG:boto:auth_path=/vclfcn-17/?prefix=obj-test-zone1 2024-02-01T10:22:58.538 DEBUG:boto:Method: GET 2024-02-01T10:22:58.538 DEBUG:boto:Path: /vclfcn-17/?prefix=obj-test-zone1 2024-02-01T10:22:58.538 DEBUG:boto:Data: 2024-02-01T10:22:58.538 DEBUG:boto:Headers: {} 2024-02-01T10:22:58.538 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:58.538 DEBUG:boto:Port: 8000 2024-02-01T10:22:58.538 DEBUG:boto:Params: {} 2024-02-01T10:22:58.538 DEBUG:boto:Token: None 2024-02-01T10:22:58.538 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:58 GMT /vclfcn-17/ 2024-02-01T10:22:58.539 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:spcR2uRdujlkBoT8m7wk4oTwUj8= 2024-02-01T10:22:58.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:spcR2uRdujlkBoT8m7wk4oTwUj8=', 'Content-Length': '0'} 2024-02-01T10:22:58.543 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b22ad6fbec67dcc9-0065bb7102-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:58.584 DEBUG:boto:b'vclfcn-17obj-test-zone11000falseobj-test-zone12024-02-01T10:20:50.473Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:58.585 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:58.585 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:58.585 DEBUG:boto:path=/vclfcn-17/?prefix=obj-test-zone1 2024-02-01T10:22:58.585 DEBUG:boto:auth_path=/vclfcn-17/?prefix=obj-test-zone1 2024-02-01T10:22:58.585 DEBUG:boto:Method: GET 2024-02-01T10:22:58.585 DEBUG:boto:Path: /vclfcn-17/?prefix=obj-test-zone1 2024-02-01T10:22:58.585 DEBUG:boto:Data: 2024-02-01T10:22:58.585 DEBUG:boto:Headers: {} 2024-02-01T10:22:58.585 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:58.585 DEBUG:boto:Port: 8000 2024-02-01T10:22:58.586 DEBUG:boto:Params: {} 2024-02-01T10:22:58.586 DEBUG:boto:Token: None 2024-02-01T10:22:58.586 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:58 GMT /vclfcn-17/ 2024-02-01T10:22:58.586 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:spcR2uRdujlkBoT8m7wk4oTwUj8= 2024-02-01T10:22:58.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:spcR2uRdujlkBoT8m7wk4oTwUj8=', 'Content-Length': '0'} 2024-02-01T10:22:58.590 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000620cc3515fa4953a-0065bb7102-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:58.591 DEBUG:boto:b'vclfcn-17obj-test-zone11000falseobj-test-zone12024-02-01T10:20:50.473Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:58.591 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-02-01T10:22:58.591 DEBUG:boto:path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:58.591 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:58.591 DEBUG:boto:path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:58.591 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:58.592 DEBUG:boto:Method: GET 2024-02-01T10:22:58.592 DEBUG:boto:Path: /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:58.592 DEBUG:boto:Data: 2024-02-01T10:22:58.592 DEBUG:boto:Headers: {} 2024-02-01T10:22:58.592 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:58.592 DEBUG:boto:Port: 8000 2024-02-01T10:22:58.592 DEBUG:boto:Params: {} 2024-02-01T10:22:58.593 DEBUG:boto:Token: None 2024-02-01T10:22:58.593 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:58 GMT /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:58.593 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZCE5XXQZcEAeNrQQZz0awVxUfk4= 2024-02-01T10:22:58.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZCE5XXQZcEAeNrQQZz0awVxUfk4=', 'Content-Length': '0'} 2024-02-01T10:22:58.596 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', '-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006b8c4e7d62ba9f56-0065bb7102-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:58.596 DEBUG:boto:path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:58.596 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:58.596 DEBUG:boto:path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:58.597 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:58.597 DEBUG:boto:Method: GET 2024-02-01T10:22:58.597 DEBUG:boto:Path: /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:58.597 DEBUG:boto:Data: 2024-02-01T10:22:58.597 DEBUG:boto:Headers: {} 2024-02-01T10:22:58.597 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:58.597 DEBUG:boto:Port: 8000 2024-02-01T10:22:58.597 DEBUG:boto:Params: {} 2024-02-01T10:22:58.597 DEBUG:boto:Token: None 2024-02-01T10:22:58.598 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:58 GMT /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:58.598 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZCE5XXQZcEAeNrQQZz0awVxUfk4= 2024-02-01T10:22:58.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZCE5XXQZcEAeNrQQZz0awVxUfk4=', 'Content-Length': '0'} 2024-02-01T10:22:58.601 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', '-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dadb7d151477580c-0065bb7102-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:58.644 DEBUG:boto:path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:58.644 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:58.644 DEBUG:boto:path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:58.644 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:58.645 DEBUG:boto:Method: HEAD 2024-02-01T10:22:58.645 DEBUG:boto:Path: /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:58.645 DEBUG:boto:Data: 2024-02-01T10:22:58.645 DEBUG:boto:Headers: {} 2024-02-01T10:22:58.645 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:58.645 DEBUG:boto:Port: 8000 2024-02-01T10:22:58.645 DEBUG:boto:Params: {} 2024-02-01T10:22:58.645 DEBUG:boto:Token: None 2024-02-01T10:22:58.645 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:58 GMT /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:58.646 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:h+a4ymvadvJQgEmXCwMS5nqQI2E= 2024-02-01T10:22:58.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:h+a4ymvadvJQgEmXCwMS5nqQI2E=', 'Content-Length': '0'} 2024-02-01T10:22:58.648 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', '-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000a38adad754248c9-0065bb7102-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:58.649 DEBUG:boto:path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:58.649 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:58.649 DEBUG:boto:path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:58.649 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:58.649 DEBUG:boto:Method: HEAD 2024-02-01T10:22:58.649 DEBUG:boto:Path: /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:58.649 DEBUG:boto:Data: 2024-02-01T10:22:58.649 DEBUG:boto:Headers: {} 2024-02-01T10:22:58.649 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:58.649 DEBUG:boto:Port: 8000 2024-02-01T10:22:58.649 DEBUG:boto:Params: {} 2024-02-01T10:22:58.650 DEBUG:boto:Token: None 2024-02-01T10:22:58.650 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:58 GMT /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:58.650 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:h+a4ymvadvJQgEmXCwMS5nqQI2E= 2024-02-01T10:22:58.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:h+a4ymvadvJQgEmXCwMS5nqQI2E=', 'Content-Length': '0'} 2024-02-01T10:22:58.653 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', '-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a59796b56ff32802-0065bb7102-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:58.653 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-02-01T10:22:58.653 DEBUG:boto:path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:58.653 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:58.654 DEBUG:boto:path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:58.654 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:58.654 DEBUG:boto:Method: GET 2024-02-01T10:22:58.654 DEBUG:boto:Path: /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:58.654 DEBUG:boto:Data: 2024-02-01T10:22:58.654 DEBUG:boto:Headers: {} 2024-02-01T10:22:58.654 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:58.654 DEBUG:boto:Port: 8000 2024-02-01T10:22:58.654 DEBUG:boto:Params: {} 2024-02-01T10:22:58.654 DEBUG:boto:Token: None 2024-02-01T10:22:58.655 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:58 GMT /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:58.655 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZCE5XXQZcEAeNrQQZz0awVxUfk4= 2024-02-01T10:22:58.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZCE5XXQZcEAeNrQQZz0awVxUfk4=', 'Content-Length': '0'} 2024-02-01T10:22:58.657 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', '-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fed283e8e25e33de-0065bb7102-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:58.658 DEBUG:boto:path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:58.658 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:58.658 DEBUG:boto:path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:58.658 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:58.658 DEBUG:boto:Method: GET 2024-02-01T10:22:58.658 DEBUG:boto:Path: /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:58.658 DEBUG:boto:Data: 2024-02-01T10:22:58.658 DEBUG:boto:Headers: {} 2024-02-01T10:22:58.659 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:58.659 DEBUG:boto:Port: 8000 2024-02-01T10:22:58.659 DEBUG:boto:Params: {} 2024-02-01T10:22:58.659 DEBUG:boto:Token: None 2024-02-01T10:22:58.659 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:58 GMT /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:58.659 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZCE5XXQZcEAeNrQQZz0awVxUfk4= 2024-02-01T10:22:58.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZCE5XXQZcEAeNrQQZz0awVxUfk4=', 'Content-Length': '0'} 2024-02-01T10:22:58.662 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', '-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000dd66a22b84aa550-0065bb7102-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:58.704 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:58.704 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:58.704 DEBUG:boto:path=/vclfcn-17/?prefix=obj-test-zone2 2024-02-01T10:22:58.705 DEBUG:boto:auth_path=/vclfcn-17/?prefix=obj-test-zone2 2024-02-01T10:22:58.705 DEBUG:boto:Method: GET 2024-02-01T10:22:58.705 DEBUG:boto:Path: /vclfcn-17/?prefix=obj-test-zone2 2024-02-01T10:22:58.705 DEBUG:boto:Data: 2024-02-01T10:22:58.705 DEBUG:boto:Headers: {} 2024-02-01T10:22:58.705 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:58.705 DEBUG:boto:Port: 8000 2024-02-01T10:22:58.705 DEBUG:boto:Params: {} 2024-02-01T10:22:58.705 DEBUG:boto:Token: None 2024-02-01T10:22:58.706 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:58 GMT /vclfcn-17/ 2024-02-01T10:22:58.706 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:spcR2uRdujlkBoT8m7wk4oTwUj8= 2024-02-01T10:22:58.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:spcR2uRdujlkBoT8m7wk4oTwUj8=', 'Content-Length': '0'} 2024-02-01T10:22:58.711 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063608f28a32cc962-0065bb7102-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:58.711 DEBUG:boto:b'vclfcn-17obj-test-zone21000falseobj-test-zone22024-02-01T10:20:50.624Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:58.711 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:58.711 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:58.712 DEBUG:boto:path=/vclfcn-17/?prefix=obj-test-zone2 2024-02-01T10:22:58.712 DEBUG:boto:auth_path=/vclfcn-17/?prefix=obj-test-zone2 2024-02-01T10:22:58.712 DEBUG:boto:Method: GET 2024-02-01T10:22:58.712 DEBUG:boto:Path: /vclfcn-17/?prefix=obj-test-zone2 2024-02-01T10:22:58.712 DEBUG:boto:Data: 2024-02-01T10:22:58.712 DEBUG:boto:Headers: {} 2024-02-01T10:22:58.712 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:58.712 DEBUG:boto:Port: 8000 2024-02-01T10:22:58.712 DEBUG:boto:Params: {} 2024-02-01T10:22:58.712 DEBUG:boto:Token: None 2024-02-01T10:22:58.713 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:58 GMT /vclfcn-17/ 2024-02-01T10:22:58.713 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:spcR2uRdujlkBoT8m7wk4oTwUj8= 2024-02-01T10:22:58.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:spcR2uRdujlkBoT8m7wk4oTwUj8=', 'Content-Length': '0'} 2024-02-01T10:22:58.718 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001abe792b3310a628-0065bb7102-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:58.718 DEBUG:boto:b'vclfcn-17obj-test-zone21000falseobj-test-zone22024-02-01T10:20:50.624Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:58.719 DEBUG:boto:path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:58.719 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:58.719 DEBUG:boto:Method: HEAD 2024-02-01T10:22:58.719 DEBUG:boto:Path: /vclfcn-17/obj-test-zone2 2024-02-01T10:22:58.719 DEBUG:boto:Data: 2024-02-01T10:22:58.719 DEBUG:boto:Headers: {} 2024-02-01T10:22:58.720 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:58.720 DEBUG:boto:Port: 8000 2024-02-01T10:22:58.720 DEBUG:boto:Params: {} 2024-02-01T10:22:58.720 DEBUG:boto:Token: None 2024-02-01T10:22:58.720 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:58 GMT /vclfcn-17/obj-test-zone2 2024-02-01T10:22:58.720 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jiT4NSdBoe+3aJqmMeBLjW14DFg= 2024-02-01T10:22:58.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jiT4NSdBoe+3aJqmMeBLjW14DFg=', 'Content-Length': '0'} 2024-02-01T10:22:58.724 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', '-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b644ba31db28309a-0065bb7102-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:58.724 DEBUG:boto:path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:58.724 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:58.725 DEBUG:boto:Method: HEAD 2024-02-01T10:22:58.725 DEBUG:boto:Path: /vclfcn-17/obj-test-zone2 2024-02-01T10:22:58.725 DEBUG:boto:Data: 2024-02-01T10:22:58.725 DEBUG:boto:Headers: {} 2024-02-01T10:22:58.725 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:58.725 DEBUG:boto:Port: 8000 2024-02-01T10:22:58.725 DEBUG:boto:Params: {} 2024-02-01T10:22:58.725 DEBUG:boto:Token: None 2024-02-01T10:22:58.725 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:58 GMT /vclfcn-17/obj-test-zone2 2024-02-01T10:22:58.726 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jiT4NSdBoe+3aJqmMeBLjW14DFg= 2024-02-01T10:22:58.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jiT4NSdBoe+3aJqmMeBLjW14DFg=', 'Content-Length': '0'} 2024-02-01T10:22:58.729 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', '-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000108b68cf9eeecd19-0065bb7102-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:58.729 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-02-01T10:22:58.730 DEBUG:boto:path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:58.730 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:58.730 DEBUG:boto:path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:58.730 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:58.730 DEBUG:boto:Method: GET 2024-02-01T10:22:58.730 DEBUG:boto:Path: /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:58.730 DEBUG:boto:Data: 2024-02-01T10:22:58.730 DEBUG:boto:Headers: {} 2024-02-01T10:22:58.730 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:58.730 DEBUG:boto:Port: 8000 2024-02-01T10:22:58.730 DEBUG:boto:Params: {} 2024-02-01T10:22:58.731 DEBUG:boto:Token: None 2024-02-01T10:22:58.731 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:58 GMT /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:58.731 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZCE5XXQZcEAeNrQQZz0awVxUfk4= 2024-02-01T10:22:58.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZCE5XXQZcEAeNrQQZz0awVxUfk4=', 'Content-Length': '0'} 2024-02-01T10:22:58.734 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', '-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a43f9f4088e4bfcf-0065bb7102-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:58.776 DEBUG:boto:path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:58.777 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:58.777 DEBUG:boto:path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:58.777 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:58.777 DEBUG:boto:Method: GET 2024-02-01T10:22:58.777 DEBUG:boto:Path: /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:58.777 DEBUG:boto:Data: 2024-02-01T10:22:58.777 DEBUG:boto:Headers: {} 2024-02-01T10:22:58.777 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:58.777 DEBUG:boto:Port: 8000 2024-02-01T10:22:58.777 DEBUG:boto:Params: {} 2024-02-01T10:22:58.778 DEBUG:boto:Token: None 2024-02-01T10:22:58.778 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:58 GMT /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:58.778 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZCE5XXQZcEAeNrQQZz0awVxUfk4= 2024-02-01T10:22:58.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZCE5XXQZcEAeNrQQZz0awVxUfk4=', 'Content-Length': '0'} 2024-02-01T10:22:58.781 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', '-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009b5d5d01af97537c-0065bb7102-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:58.828 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:58.828 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:58.828 DEBUG:boto:path=/vclfcn-17/?prefix=obj-test-zone2 2024-02-01T10:22:58.829 DEBUG:boto:auth_path=/vclfcn-17/?prefix=obj-test-zone2 2024-02-01T10:22:58.829 DEBUG:boto:Method: GET 2024-02-01T10:22:58.829 DEBUG:boto:Path: /vclfcn-17/?prefix=obj-test-zone2 2024-02-01T10:22:58.829 DEBUG:boto:Data: 2024-02-01T10:22:58.829 DEBUG:boto:Headers: {} 2024-02-01T10:22:58.829 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:58.829 DEBUG:boto:Port: 8000 2024-02-01T10:22:58.829 DEBUG:boto:Params: {} 2024-02-01T10:22:58.829 DEBUG:boto:Token: None 2024-02-01T10:22:58.829 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:58 GMT /vclfcn-17/ 2024-02-01T10:22:58.830 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:spcR2uRdujlkBoT8m7wk4oTwUj8= 2024-02-01T10:22:58.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:spcR2uRdujlkBoT8m7wk4oTwUj8=', 'Content-Length': '0'} 2024-02-01T10:22:58.834 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000006634ac6a7f77914-0065bb7102-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:58.834 DEBUG:boto:b'vclfcn-17obj-test-zone21000falseobj-test-zone22024-02-01T10:20:50.624Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:58.835 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:58.835 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:58.835 DEBUG:boto:path=/vclfcn-17/?prefix=obj-test-zone2 2024-02-01T10:22:58.835 DEBUG:boto:auth_path=/vclfcn-17/?prefix=obj-test-zone2 2024-02-01T10:22:58.835 DEBUG:boto:Method: GET 2024-02-01T10:22:58.835 DEBUG:boto:Path: /vclfcn-17/?prefix=obj-test-zone2 2024-02-01T10:22:58.835 DEBUG:boto:Data: 2024-02-01T10:22:58.836 DEBUG:boto:Headers: {} 2024-02-01T10:22:58.836 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:58.836 DEBUG:boto:Port: 8000 2024-02-01T10:22:58.836 DEBUG:boto:Params: {} 2024-02-01T10:22:58.836 DEBUG:boto:Token: None 2024-02-01T10:22:58.836 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:58 GMT /vclfcn-17/ 2024-02-01T10:22:58.836 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:spcR2uRdujlkBoT8m7wk4oTwUj8= 2024-02-01T10:22:58.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:spcR2uRdujlkBoT8m7wk4oTwUj8=', 'Content-Length': '0'} 2024-02-01T10:22:58.841 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a31d4b3123f7142-0065bb7102-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:58.841 DEBUG:boto:b'vclfcn-17obj-test-zone21000falseobj-test-zone22024-02-01T10:20:50.624Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:58.841 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-02-01T10:22:58.842 DEBUG:boto:path=/vclfcn-17/obj-test-zone3 2024-02-01T10:22:58.842 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3 2024-02-01T10:22:58.842 DEBUG:boto:path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:58.842 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:58.842 DEBUG:boto:Method: GET 2024-02-01T10:22:58.842 DEBUG:boto:Path: /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:58.842 DEBUG:boto:Data: 2024-02-01T10:22:58.842 DEBUG:boto:Headers: {} 2024-02-01T10:22:58.842 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:58.842 DEBUG:boto:Port: 8000 2024-02-01T10:22:58.843 DEBUG:boto:Params: {} 2024-02-01T10:22:58.843 DEBUG:boto:Token: None 2024-02-01T10:22:58.843 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:58 GMT /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:58.843 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:SmTICucsSzrI5NkosA5bIStQi70= 2024-02-01T10:22:58.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:SmTICucsSzrI5NkosA5bIStQi70=', 'Content-Length': '0'} 2024-02-01T10:22:58.846 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'YE736OwOOXbZJHJe7d8YvcNV1LxwcQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f108d18d2fcb8321-0065bb7102-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:58.888 DEBUG:boto:path=/vclfcn-17/obj-test-zone3 2024-02-01T10:22:58.888 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3 2024-02-01T10:22:58.888 DEBUG:boto:path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:58.889 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:58.889 DEBUG:boto:Method: GET 2024-02-01T10:22:58.889 DEBUG:boto:Path: /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:58.889 DEBUG:boto:Data: 2024-02-01T10:22:58.889 DEBUG:boto:Headers: {} 2024-02-01T10:22:58.889 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:58.889 DEBUG:boto:Port: 8000 2024-02-01T10:22:58.889 DEBUG:boto:Params: {} 2024-02-01T10:22:58.889 DEBUG:boto:Token: None 2024-02-01T10:22:58.890 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:58 GMT /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:58.890 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:SmTICucsSzrI5NkosA5bIStQi70= 2024-02-01T10:22:58.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:SmTICucsSzrI5NkosA5bIStQi70=', 'Content-Length': '0'} 2024-02-01T10:22:58.893 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'YE736OwOOXbZJHJe7d8YvcNV1LxwcQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000be23eb53d7d0d281-0065bb7102-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:58.893 DEBUG:boto:path=/vclfcn-17/obj-test-zone3 2024-02-01T10:22:58.894 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3 2024-02-01T10:22:58.894 DEBUG:boto:path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:58.894 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:58.894 DEBUG:boto:Method: HEAD 2024-02-01T10:22:58.894 DEBUG:boto:Path: /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:58.894 DEBUG:boto:Data: 2024-02-01T10:22:58.894 DEBUG:boto:Headers: {} 2024-02-01T10:22:58.894 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:58.894 DEBUG:boto:Port: 8000 2024-02-01T10:22:58.894 DEBUG:boto:Params: {} 2024-02-01T10:22:58.894 DEBUG:boto:Token: None 2024-02-01T10:22:58.895 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:58 GMT /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:58.895 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:L8JDb5WoF/PixufgvaG81gM6JtY= 2024-02-01T10:22:58.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:L8JDb5WoF/PixufgvaG81gM6JtY=', 'Content-Length': '0'} 2024-02-01T10:22:58.898 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'YE736OwOOXbZJHJe7d8YvcNV1LxwcQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eddd6b7b90050a52-0065bb7102-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:58.898 DEBUG:boto:path=/vclfcn-17/obj-test-zone3 2024-02-01T10:22:58.898 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3 2024-02-01T10:22:58.898 DEBUG:boto:path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:58.898 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:58.898 DEBUG:boto:Method: HEAD 2024-02-01T10:22:58.898 DEBUG:boto:Path: /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:58.899 DEBUG:boto:Data: 2024-02-01T10:22:58.899 DEBUG:boto:Headers: {} 2024-02-01T10:22:58.899 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:58.899 DEBUG:boto:Port: 8000 2024-02-01T10:22:58.899 DEBUG:boto:Params: {} 2024-02-01T10:22:58.899 DEBUG:boto:Token: None 2024-02-01T10:22:58.899 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:58 GMT /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:58.899 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:L8JDb5WoF/PixufgvaG81gM6JtY= 2024-02-01T10:22:58.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:L8JDb5WoF/PixufgvaG81gM6JtY=', 'Content-Length': '0'} 2024-02-01T10:22:58.902 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'YE736OwOOXbZJHJe7d8YvcNV1LxwcQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000087e2ef47680d1818-0065bb7102-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:58.902 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-02-01T10:22:58.902 DEBUG:boto:path=/vclfcn-17/obj-test-zone3 2024-02-01T10:22:58.903 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3 2024-02-01T10:22:58.903 DEBUG:boto:path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:58.903 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:58.903 DEBUG:boto:Method: GET 2024-02-01T10:22:58.903 DEBUG:boto:Path: /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:58.903 DEBUG:boto:Data: 2024-02-01T10:22:58.903 DEBUG:boto:Headers: {} 2024-02-01T10:22:58.903 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:58.903 DEBUG:boto:Port: 8000 2024-02-01T10:22:58.903 DEBUG:boto:Params: {} 2024-02-01T10:22:58.903 DEBUG:boto:Token: None 2024-02-01T10:22:58.904 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:58 GMT /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:58.904 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:SmTICucsSzrI5NkosA5bIStQi70= 2024-02-01T10:22:58.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:SmTICucsSzrI5NkosA5bIStQi70=', 'Content-Length': '0'} 2024-02-01T10:22:58.907 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'YE736OwOOXbZJHJe7d8YvcNV1LxwcQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f16a85de4b4f5795-0065bb7102-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:58.948 DEBUG:boto:path=/vclfcn-17/obj-test-zone3 2024-02-01T10:22:58.948 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3 2024-02-01T10:22:58.948 DEBUG:boto:path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:58.949 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:58.949 DEBUG:boto:Method: GET 2024-02-01T10:22:58.949 DEBUG:boto:Path: /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:58.949 DEBUG:boto:Data: 2024-02-01T10:22:58.949 DEBUG:boto:Headers: {} 2024-02-01T10:22:58.949 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:58.949 DEBUG:boto:Port: 8000 2024-02-01T10:22:58.949 DEBUG:boto:Params: {} 2024-02-01T10:22:58.949 DEBUG:boto:Token: None 2024-02-01T10:22:58.950 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:58 GMT /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:58.950 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:SmTICucsSzrI5NkosA5bIStQi70= 2024-02-01T10:22:58.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:SmTICucsSzrI5NkosA5bIStQi70=', 'Content-Length': '0'} 2024-02-01T10:22:58.953 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'YE736OwOOXbZJHJe7d8YvcNV1LxwcQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a1e6322198345570-0065bb7102-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:58.996 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:58.996 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:58.996 DEBUG:boto:path=/vclfcn-17/?prefix=obj-test-zone3 2024-02-01T10:22:58.996 DEBUG:boto:auth_path=/vclfcn-17/?prefix=obj-test-zone3 2024-02-01T10:22:58.997 DEBUG:boto:Method: GET 2024-02-01T10:22:58.997 DEBUG:boto:Path: /vclfcn-17/?prefix=obj-test-zone3 2024-02-01T10:22:58.997 DEBUG:boto:Data: 2024-02-01T10:22:58.997 DEBUG:boto:Headers: {} 2024-02-01T10:22:58.997 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:58.997 DEBUG:boto:Port: 8000 2024-02-01T10:22:58.997 DEBUG:boto:Params: {} 2024-02-01T10:22:58.997 DEBUG:boto:Token: None 2024-02-01T10:22:58.997 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:58 GMT /vclfcn-17/ 2024-02-01T10:22:58.998 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:spcR2uRdujlkBoT8m7wk4oTwUj8= 2024-02-01T10:22:58.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:spcR2uRdujlkBoT8m7wk4oTwUj8=', 'Content-Length': '0'} 2024-02-01T10:22:59.002 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000647232af78f42481-0065bb7103-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:59.002 DEBUG:boto:b'vclfcn-17obj-test-zone31000falseobj-test-zone32024-02-01T10:20:50.780Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:59.003 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:59.003 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:59.003 DEBUG:boto:path=/vclfcn-17/?prefix=obj-test-zone3 2024-02-01T10:22:59.003 DEBUG:boto:auth_path=/vclfcn-17/?prefix=obj-test-zone3 2024-02-01T10:22:59.003 DEBUG:boto:Method: GET 2024-02-01T10:22:59.003 DEBUG:boto:Path: /vclfcn-17/?prefix=obj-test-zone3 2024-02-01T10:22:59.004 DEBUG:boto:Data: 2024-02-01T10:22:59.004 DEBUG:boto:Headers: {} 2024-02-01T10:22:59.004 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:59.004 DEBUG:boto:Port: 8000 2024-02-01T10:22:59.004 DEBUG:boto:Params: {} 2024-02-01T10:22:59.004 DEBUG:boto:Token: None 2024-02-01T10:22:59.004 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:59 GMT /vclfcn-17/ 2024-02-01T10:22:59.004 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WmRanT6k02akI+r93G3HBmjiMxc= 2024-02-01T10:22:59.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WmRanT6k02akI+r93G3HBmjiMxc=', 'Content-Length': '0'} 2024-02-01T10:22:59.009 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e7a912ae7b747ee-0065bb7103-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:59.052 DEBUG:boto:b'vclfcn-17obj-test-zone31000falseobj-test-zone32024-02-01T10:20:50.780Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:59.053 DEBUG:boto:path=/vclfcn-17/obj-test-zone3 2024-02-01T10:22:59.053 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3 2024-02-01T10:22:59.053 DEBUG:boto:Method: HEAD 2024-02-01T10:22:59.053 DEBUG:boto:Path: /vclfcn-17/obj-test-zone3 2024-02-01T10:22:59.053 DEBUG:boto:Data: 2024-02-01T10:22:59.053 DEBUG:boto:Headers: {} 2024-02-01T10:22:59.053 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:59.053 DEBUG:boto:Port: 8000 2024-02-01T10:22:59.053 DEBUG:boto:Params: {} 2024-02-01T10:22:59.054 DEBUG:boto:Token: None 2024-02-01T10:22:59.054 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:59 GMT /vclfcn-17/obj-test-zone3 2024-02-01T10:22:59.054 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:JiK7JHDfCaJ0oTAvVkwNfwr1g5M= 2024-02-01T10:22:59.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:JiK7JHDfCaJ0oTAvVkwNfwr1g5M=', 'Content-Length': '0'} 2024-02-01T10:22:59.058 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'YE736OwOOXbZJHJe7d8YvcNV1LxwcQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002e64da3d84d54129-0065bb7103-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:59.058 DEBUG:boto:path=/vclfcn-17/obj-test-zone3 2024-02-01T10:22:59.059 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3 2024-02-01T10:22:59.059 DEBUG:boto:Method: HEAD 2024-02-01T10:22:59.059 DEBUG:boto:Path: /vclfcn-17/obj-test-zone3 2024-02-01T10:22:59.059 DEBUG:boto:Data: 2024-02-01T10:22:59.059 DEBUG:boto:Headers: {} 2024-02-01T10:22:59.059 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:59.059 DEBUG:boto:Port: 8000 2024-02-01T10:22:59.059 DEBUG:boto:Params: {} 2024-02-01T10:22:59.059 DEBUG:boto:Token: None 2024-02-01T10:22:59.059 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:59 GMT /vclfcn-17/obj-test-zone3 2024-02-01T10:22:59.060 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:JiK7JHDfCaJ0oTAvVkwNfwr1g5M= 2024-02-01T10:22:59.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:JiK7JHDfCaJ0oTAvVkwNfwr1g5M=', 'Content-Length': '0'} 2024-02-01T10:22:59.064 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'YE736OwOOXbZJHJe7d8YvcNV1LxwcQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004a700926d912ed39-0065bb7103-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:59.064 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-02-01T10:22:59.064 DEBUG:boto:path=/vclfcn-17/obj-test-zone3 2024-02-01T10:22:59.064 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3 2024-02-01T10:22:59.064 DEBUG:boto:path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:59.064 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:59.064 DEBUG:boto:Method: GET 2024-02-01T10:22:59.065 DEBUG:boto:Path: /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:59.065 DEBUG:boto:Data: 2024-02-01T10:22:59.065 DEBUG:boto:Headers: {} 2024-02-01T10:22:59.065 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:59.065 DEBUG:boto:Port: 8000 2024-02-01T10:22:59.065 DEBUG:boto:Params: {} 2024-02-01T10:22:59.065 DEBUG:boto:Token: None 2024-02-01T10:22:59.065 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:59 GMT /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:59.065 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GvoDqCAo+TlqfkFhLdPzmxwslig= 2024-02-01T10:22:59.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GvoDqCAo+TlqfkFhLdPzmxwslig=', 'Content-Length': '0'} 2024-02-01T10:22:59.068 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'YE736OwOOXbZJHJe7d8YvcNV1LxwcQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d129fe26f334d34f-0065bb7103-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:59.112 DEBUG:boto:path=/vclfcn-17/obj-test-zone3 2024-02-01T10:22:59.112 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3 2024-02-01T10:22:59.113 DEBUG:boto:path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:59.113 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:59.113 DEBUG:boto:Method: GET 2024-02-01T10:22:59.113 DEBUG:boto:Path: /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:59.113 DEBUG:boto:Data: 2024-02-01T10:22:59.113 DEBUG:boto:Headers: {} 2024-02-01T10:22:59.113 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:59.113 DEBUG:boto:Port: 8000 2024-02-01T10:22:59.113 DEBUG:boto:Params: {} 2024-02-01T10:22:59.113 DEBUG:boto:Token: None 2024-02-01T10:22:59.114 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:59 GMT /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:22:59.114 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GvoDqCAo+TlqfkFhLdPzmxwslig= 2024-02-01T10:22:59.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GvoDqCAo+TlqfkFhLdPzmxwslig=', 'Content-Length': '0'} 2024-02-01T10:22:59.117 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'YE736OwOOXbZJHJe7d8YvcNV1LxwcQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a73fc07ff7f1ddbe-0065bb7103-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:59.160 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:59.160 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:59.161 DEBUG:boto:path=/vclfcn-17/?prefix=obj-test-zone3 2024-02-01T10:22:59.161 DEBUG:boto:auth_path=/vclfcn-17/?prefix=obj-test-zone3 2024-02-01T10:22:59.161 DEBUG:boto:Method: GET 2024-02-01T10:22:59.161 DEBUG:boto:Path: /vclfcn-17/?prefix=obj-test-zone3 2024-02-01T10:22:59.161 DEBUG:boto:Data: 2024-02-01T10:22:59.161 DEBUG:boto:Headers: {} 2024-02-01T10:22:59.161 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:22:59.161 DEBUG:boto:Port: 8000 2024-02-01T10:22:59.161 DEBUG:boto:Params: {} 2024-02-01T10:22:59.161 DEBUG:boto:Token: None 2024-02-01T10:22:59.162 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:59 GMT /vclfcn-17/ 2024-02-01T10:22:59.162 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WmRanT6k02akI+r93G3HBmjiMxc= 2024-02-01T10:22:59.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WmRanT6k02akI+r93G3HBmjiMxc=', 'Content-Length': '0'} 2024-02-01T10:22:59.166 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4d110c36005fbbc-0065bb7103-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:59.166 DEBUG:boto:b'vclfcn-17obj-test-zone31000falseobj-test-zone32024-02-01T10:20:50.780Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:59.167 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:59.167 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:59.167 DEBUG:boto:path=/vclfcn-17/?prefix=obj-test-zone3 2024-02-01T10:22:59.167 DEBUG:boto:auth_path=/vclfcn-17/?prefix=obj-test-zone3 2024-02-01T10:22:59.167 DEBUG:boto:Method: GET 2024-02-01T10:22:59.167 DEBUG:boto:Path: /vclfcn-17/?prefix=obj-test-zone3 2024-02-01T10:22:59.167 DEBUG:boto:Data: 2024-02-01T10:22:59.168 DEBUG:boto:Headers: {} 2024-02-01T10:22:59.168 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:59.168 DEBUG:boto:Port: 8000 2024-02-01T10:22:59.168 DEBUG:boto:Params: {} 2024-02-01T10:22:59.168 DEBUG:boto:Token: None 2024-02-01T10:22:59.168 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:59 GMT /vclfcn-17/ 2024-02-01T10:22:59.168 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WmRanT6k02akI+r93G3HBmjiMxc= 2024-02-01T10:22:59.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WmRanT6k02akI+r93G3HBmjiMxc=', 'Content-Length': '0'} 2024-02-01T10:22:59.173 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c10179d18be3387-0065bb7103-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:59.173 DEBUG:boto:b'vclfcn-17obj-test-zone31000falseobj-test-zone32024-02-01T10:20:50.780Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:59.174 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-17 zones={test-zone2, test-zone1} 2024-02-01T10:22:59.174 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-17 zones={test-zone3, test-zone1} 2024-02-01T10:22:59.174 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:59.174 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:59.174 DEBUG:boto:Method: HEAD 2024-02-01T10:22:59.174 DEBUG:boto:Path: /vclfcn-17/ 2024-02-01T10:22:59.174 DEBUG:boto:Data: 2024-02-01T10:22:59.175 DEBUG:boto:Headers: {} 2024-02-01T10:22:59.175 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:59.175 DEBUG:boto:Port: 8001 2024-02-01T10:22:59.175 DEBUG:boto:Params: {} 2024-02-01T10:22:59.175 DEBUG:boto:Token: None 2024-02-01T10:22:59.175 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:59 GMT /vclfcn-17/ 2024-02-01T10:22:59.175 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:bv5MHHpD+/R3tCYQRNHoliDAz7g= 2024-02-01T10:22:59.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:bv5MHHpD+/R3tCYQRNHoliDAz7g=', 'Content-Length': '0'} 2024-02-01T10:22:59.180 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000abf43ccaafb7ef9a-0065bb7103-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:22:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:59.180 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:59.180 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:59.180 DEBUG:boto:Method: HEAD 2024-02-01T10:22:59.180 DEBUG:boto:Path: /vclfcn-17/ 2024-02-01T10:22:59.180 DEBUG:boto:Data: 2024-02-01T10:22:59.180 DEBUG:boto:Headers: {} 2024-02-01T10:22:59.181 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:59.181 DEBUG:boto:Port: 8000 2024-02-01T10:22:59.181 DEBUG:boto:Params: {} 2024-02-01T10:22:59.181 DEBUG:boto:Token: None 2024-02-01T10:22:59.181 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:59 GMT /vclfcn-17/ 2024-02-01T10:22:59.181 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:bv5MHHpD+/R3tCYQRNHoliDAz7g= 2024-02-01T10:22:59.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:bv5MHHpD+/R3tCYQRNHoliDAz7g=', 'Content-Length': '0'} 2024-02-01T10:22:59.185 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000002fb634e807c24e6-0065bb7103-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:22:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:59.185 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:22:59.186 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:59.186 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:59.186 DEBUG:boto:path=/vclfcn-17/?versions&max-keys=999 2024-02-01T10:22:59.186 DEBUG:boto:auth_path=/vclfcn-17/?versions&max-keys=999 2024-02-01T10:22:59.186 DEBUG:boto:Method: GET 2024-02-01T10:22:59.186 DEBUG:boto:Path: /vclfcn-17/?versions&max-keys=999 2024-02-01T10:22:59.186 DEBUG:boto:Data: 2024-02-01T10:22:59.187 DEBUG:boto:Headers: {} 2024-02-01T10:22:59.187 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:59.187 DEBUG:boto:Port: 8001 2024-02-01T10:22:59.187 DEBUG:boto:Params: {} 2024-02-01T10:22:59.187 DEBUG:boto:Token: None 2024-02-01T10:22:59.187 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:59 GMT /vclfcn-17/?versions 2024-02-01T10:22:59.187 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:dEPWOlcEL1KV9wZ9jyEzRG26zzU= 2024-02-01T10:22:59.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dEPWOlcEL1KV9wZ9jyEzRG26zzU=', 'Content-Length': '0'} 2024-02-01T10:22:59.192 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e40ef617944775d-0065bb7103-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:59.192 DEBUG:boto:b'vclfcn-17999falsedummyhQXZXSD45t2CSRVMJOLMBXih54iMPmJtrue2024-02-01T10:20:14.297Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1xMxAGR4A-xEILepnzFPRgrWNTOLvitztrue2024-02-01T10:20:50.473Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1true2024-02-01T10:20:50.624Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3YE736OwOOXbZJHJe7d8YvcNV1LxwcQbtrue2024-02-01T10:20:50.780Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:59.193 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-02-01T10:22:59.193 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-02-01T10:22:59.193 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-02-01T10:22:59.193 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-02-01T10:22:59.193 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:22:59.193 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:59.193 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:59.193 DEBUG:boto:path=/vclfcn-17/?versions&max-keys=999 2024-02-01T10:22:59.193 DEBUG:boto:auth_path=/vclfcn-17/?versions&max-keys=999 2024-02-01T10:22:59.194 DEBUG:boto:Method: GET 2024-02-01T10:22:59.194 DEBUG:boto:Path: /vclfcn-17/?versions&max-keys=999 2024-02-01T10:22:59.194 DEBUG:boto:Data: 2024-02-01T10:22:59.194 DEBUG:boto:Headers: {} 2024-02-01T10:22:59.194 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:59.194 DEBUG:boto:Port: 8000 2024-02-01T10:22:59.194 DEBUG:boto:Params: {} 2024-02-01T10:22:59.195 DEBUG:boto:Token: None 2024-02-01T10:22:59.195 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:59 GMT /vclfcn-17/?versions 2024-02-01T10:22:59.195 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:dEPWOlcEL1KV9wZ9jyEzRG26zzU= 2024-02-01T10:22:59.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dEPWOlcEL1KV9wZ9jyEzRG26zzU=', 'Content-Length': '0'} 2024-02-01T10:22:59.200 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b6ac9c4921e3e12-0065bb7103-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:59.200 DEBUG:boto:b'vclfcn-17999falsedummyhQXZXSD45t2CSRVMJOLMBXih54iMPmJtrue2024-02-01T10:20:14.297Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1xMxAGR4A-xEILepnzFPRgrWNTOLvitztrue2024-02-01T10:20:50.473Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1true2024-02-01T10:20:50.624Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3YE736OwOOXbZJHJe7d8YvcNV1LxwcQbtrue2024-02-01T10:20:50.780Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:59.201 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-02-01T10:22:59.201 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-02-01T10:22:59.201 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-02-01T10:22:59.201 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-02-01T10:22:59.201 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:59.201 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:59.201 DEBUG:boto:path=/vclfcn-17/?versions&max-keys=999 2024-02-01T10:22:59.201 DEBUG:boto:auth_path=/vclfcn-17/?versions&max-keys=999 2024-02-01T10:22:59.201 DEBUG:boto:Method: GET 2024-02-01T10:22:59.202 DEBUG:boto:Path: /vclfcn-17/?versions&max-keys=999 2024-02-01T10:22:59.202 DEBUG:boto:Data: 2024-02-01T10:22:59.202 DEBUG:boto:Headers: {} 2024-02-01T10:22:59.202 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:59.202 DEBUG:boto:Port: 8001 2024-02-01T10:22:59.202 DEBUG:boto:Params: {} 2024-02-01T10:22:59.202 DEBUG:boto:Token: None 2024-02-01T10:22:59.202 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:59 GMT /vclfcn-17/?versions 2024-02-01T10:22:59.203 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:dEPWOlcEL1KV9wZ9jyEzRG26zzU= 2024-02-01T10:22:59.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dEPWOlcEL1KV9wZ9jyEzRG26zzU=', 'Content-Length': '0'} 2024-02-01T10:22:59.207 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b965ec2ebf691e8-0065bb7103-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:59.208 DEBUG:boto:b'vclfcn-17999falsedummyhQXZXSD45t2CSRVMJOLMBXih54iMPmJtrue2024-02-01T10:20:14.297Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1xMxAGR4A-xEILepnzFPRgrWNTOLvitztrue2024-02-01T10:20:50.473Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1true2024-02-01T10:20:50.624Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3YE736OwOOXbZJHJe7d8YvcNV1LxwcQbtrue2024-02-01T10:20:50.780Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:59.208 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:59.208 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:59.208 DEBUG:boto:path=/vclfcn-17/?versions&max-keys=999 2024-02-01T10:22:59.209 DEBUG:boto:auth_path=/vclfcn-17/?versions&max-keys=999 2024-02-01T10:22:59.209 DEBUG:boto:Method: GET 2024-02-01T10:22:59.209 DEBUG:boto:Path: /vclfcn-17/?versions&max-keys=999 2024-02-01T10:22:59.209 DEBUG:boto:Data: 2024-02-01T10:22:59.209 DEBUG:boto:Headers: {} 2024-02-01T10:22:59.209 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:59.209 DEBUG:boto:Port: 8000 2024-02-01T10:22:59.209 DEBUG:boto:Params: {} 2024-02-01T10:22:59.210 DEBUG:boto:Token: None 2024-02-01T10:22:59.210 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:59 GMT /vclfcn-17/?versions 2024-02-01T10:22:59.210 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:dEPWOlcEL1KV9wZ9jyEzRG26zzU= 2024-02-01T10:22:59.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dEPWOlcEL1KV9wZ9jyEzRG26zzU=', 'Content-Length': '0'} 2024-02-01T10:22:59.215 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001dbb3f42e97ff283-0065bb7103-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:59.215 DEBUG:boto:b'vclfcn-17999falsedummyhQXZXSD45t2CSRVMJOLMBXih54iMPmJtrue2024-02-01T10:20:14.297Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1xMxAGR4A-xEILepnzFPRgrWNTOLvitztrue2024-02-01T10:20:50.473Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1true2024-02-01T10:20:50.624Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3YE736OwOOXbZJHJe7d8YvcNV1LxwcQbtrue2024-02-01T10:20:50.780Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:59.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-02-01T10:22:59.216 DEBUG:boto:path=/vclfcn-17/dummy 2024-02-01T10:22:59.216 DEBUG:boto:auth_path=/vclfcn-17/dummy 2024-02-01T10:22:59.216 DEBUG:boto:path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:59.216 DEBUG:boto:auth_path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:59.216 DEBUG:boto:Method: GET 2024-02-01T10:22:59.216 DEBUG:boto:Path: /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:59.216 DEBUG:boto:Data: 2024-02-01T10:22:59.216 DEBUG:boto:Headers: {} 2024-02-01T10:22:59.216 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:59.216 DEBUG:boto:Port: 8001 2024-02-01T10:22:59.217 DEBUG:boto:Params: {} 2024-02-01T10:22:59.217 DEBUG:boto:Token: None 2024-02-01T10:22:59.217 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:59 GMT /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:59.217 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WSDnx5IcJmE5Xn7ZWyZB1mVYxJg= 2024-02-01T10:22:59.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WSDnx5IcJmE5Xn7ZWyZB1mVYxJg=', 'Content-Length': '0'} 2024-02-01T10:22:59.220 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:14 GMT'), ('x-amz-version-id', 'hQXZXSD45t2CSRVMJOLMBXih54iMPmJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a400181df4610815-0065bb7103-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:59.220 DEBUG:boto:path=/vclfcn-17/dummy 2024-02-01T10:22:59.220 DEBUG:boto:auth_path=/vclfcn-17/dummy 2024-02-01T10:22:59.220 DEBUG:boto:path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:59.220 DEBUG:boto:auth_path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:59.220 DEBUG:boto:Method: GET 2024-02-01T10:22:59.221 DEBUG:boto:Path: /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:59.221 DEBUG:boto:Data: 2024-02-01T10:22:59.221 DEBUG:boto:Headers: {} 2024-02-01T10:22:59.221 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:59.221 DEBUG:boto:Port: 8000 2024-02-01T10:22:59.221 DEBUG:boto:Params: {} 2024-02-01T10:22:59.221 DEBUG:boto:Token: None 2024-02-01T10:22:59.221 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:59 GMT /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:59.221 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WSDnx5IcJmE5Xn7ZWyZB1mVYxJg= 2024-02-01T10:22:59.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WSDnx5IcJmE5Xn7ZWyZB1mVYxJg=', 'Content-Length': '0'} 2024-02-01T10:22:59.225 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:14 GMT'), ('x-amz-version-id', 'hQXZXSD45t2CSRVMJOLMBXih54iMPmJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000108c18cd7930a686-0065bb7103-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:59.225 DEBUG:boto:path=/vclfcn-17/dummy 2024-02-01T10:22:59.225 DEBUG:boto:auth_path=/vclfcn-17/dummy 2024-02-01T10:22:59.225 DEBUG:boto:path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:59.225 DEBUG:boto:auth_path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:59.225 DEBUG:boto:Method: HEAD 2024-02-01T10:22:59.225 DEBUG:boto:Path: /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:59.226 DEBUG:boto:Data: 2024-02-01T10:22:59.226 DEBUG:boto:Headers: {} 2024-02-01T10:22:59.226 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:59.226 DEBUG:boto:Port: 8001 2024-02-01T10:22:59.226 DEBUG:boto:Params: {} 2024-02-01T10:22:59.226 DEBUG:boto:Token: None 2024-02-01T10:22:59.226 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:59 GMT /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:59.226 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1JhPMc6X33JlovdLvZw6q9AgP8Y= 2024-02-01T10:22:59.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1JhPMc6X33JlovdLvZw6q9AgP8Y=', 'Content-Length': '0'} 2024-02-01T10:22:59.229 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:14 GMT'), ('x-amz-version-id', 'hQXZXSD45t2CSRVMJOLMBXih54iMPmJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003dd107bf5563fe8f-0065bb7103-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:59.229 DEBUG:boto:path=/vclfcn-17/dummy 2024-02-01T10:22:59.229 DEBUG:boto:auth_path=/vclfcn-17/dummy 2024-02-01T10:22:59.230 DEBUG:boto:path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:59.230 DEBUG:boto:auth_path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:59.230 DEBUG:boto:Method: HEAD 2024-02-01T10:22:59.230 DEBUG:boto:Path: /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:59.230 DEBUG:boto:Data: 2024-02-01T10:22:59.230 DEBUG:boto:Headers: {} 2024-02-01T10:22:59.230 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:59.230 DEBUG:boto:Port: 8000 2024-02-01T10:22:59.230 DEBUG:boto:Params: {} 2024-02-01T10:22:59.230 DEBUG:boto:Token: None 2024-02-01T10:22:59.231 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:59 GMT /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:59.231 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1JhPMc6X33JlovdLvZw6q9AgP8Y= 2024-02-01T10:22:59.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1JhPMc6X33JlovdLvZw6q9AgP8Y=', 'Content-Length': '0'} 2024-02-01T10:22:59.233 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:14 GMT'), ('x-amz-version-id', 'hQXZXSD45t2CSRVMJOLMBXih54iMPmJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ce11e7123a6e6510-0065bb7103-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:59.234 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-02-01T10:22:59.234 DEBUG:boto:path=/vclfcn-17/dummy 2024-02-01T10:22:59.234 DEBUG:boto:auth_path=/vclfcn-17/dummy 2024-02-01T10:22:59.234 DEBUG:boto:path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:59.234 DEBUG:boto:auth_path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:59.234 DEBUG:boto:Method: GET 2024-02-01T10:22:59.234 DEBUG:boto:Path: /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:59.234 DEBUG:boto:Data: 2024-02-01T10:22:59.234 DEBUG:boto:Headers: {} 2024-02-01T10:22:59.235 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:59.235 DEBUG:boto:Port: 8001 2024-02-01T10:22:59.235 DEBUG:boto:Params: {} 2024-02-01T10:22:59.235 DEBUG:boto:Token: None 2024-02-01T10:22:59.235 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:59 GMT /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:59.235 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WSDnx5IcJmE5Xn7ZWyZB1mVYxJg= 2024-02-01T10:22:59.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WSDnx5IcJmE5Xn7ZWyZB1mVYxJg=', 'Content-Length': '0'} 2024-02-01T10:22:59.238 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:14 GMT'), ('x-amz-version-id', 'hQXZXSD45t2CSRVMJOLMBXih54iMPmJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000878e8adfa046d9b1-0065bb7103-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:59.238 DEBUG:boto:path=/vclfcn-17/dummy 2024-02-01T10:22:59.238 DEBUG:boto:auth_path=/vclfcn-17/dummy 2024-02-01T10:22:59.239 DEBUG:boto:path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:59.239 DEBUG:boto:auth_path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:59.239 DEBUG:boto:Method: GET 2024-02-01T10:22:59.239 DEBUG:boto:Path: /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:59.239 DEBUG:boto:Data: 2024-02-01T10:22:59.239 DEBUG:boto:Headers: {} 2024-02-01T10:22:59.239 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:59.239 DEBUG:boto:Port: 8000 2024-02-01T10:22:59.239 DEBUG:boto:Params: {} 2024-02-01T10:22:59.239 DEBUG:boto:Token: None 2024-02-01T10:22:59.239 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:59 GMT /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:59.240 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WSDnx5IcJmE5Xn7ZWyZB1mVYxJg= 2024-02-01T10:22:59.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WSDnx5IcJmE5Xn7ZWyZB1mVYxJg=', 'Content-Length': '0'} 2024-02-01T10:22:59.242 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:14 GMT'), ('x-amz-version-id', 'hQXZXSD45t2CSRVMJOLMBXih54iMPmJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b839d108d6b0d1c2-0065bb7103-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:59.243 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:59.243 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:59.243 DEBUG:boto:path=/vclfcn-17/?prefix=dummy 2024-02-01T10:22:59.243 DEBUG:boto:auth_path=/vclfcn-17/?prefix=dummy 2024-02-01T10:22:59.243 DEBUG:boto:Method: GET 2024-02-01T10:22:59.243 DEBUG:boto:Path: /vclfcn-17/?prefix=dummy 2024-02-01T10:22:59.243 DEBUG:boto:Data: 2024-02-01T10:22:59.243 DEBUG:boto:Headers: {} 2024-02-01T10:22:59.243 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:59.244 DEBUG:boto:Port: 8001 2024-02-01T10:22:59.244 DEBUG:boto:Params: {} 2024-02-01T10:22:59.244 DEBUG:boto:Token: None 2024-02-01T10:22:59.244 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:59 GMT /vclfcn-17/ 2024-02-01T10:22:59.244 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WmRanT6k02akI+r93G3HBmjiMxc= 2024-02-01T10:22:59.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WmRanT6k02akI+r93G3HBmjiMxc=', 'Content-Length': '0'} 2024-02-01T10:22:59.249 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000162d9cd0c06e9fd0-0065bb7103-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:59.292 DEBUG:boto:b'vclfcn-17dummy1000falsedummy2024-02-01T10:20:14.297Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:59.293 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:59.293 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:59.293 DEBUG:boto:path=/vclfcn-17/?prefix=dummy 2024-02-01T10:22:59.293 DEBUG:boto:auth_path=/vclfcn-17/?prefix=dummy 2024-02-01T10:22:59.293 DEBUG:boto:Method: GET 2024-02-01T10:22:59.293 DEBUG:boto:Path: /vclfcn-17/?prefix=dummy 2024-02-01T10:22:59.294 DEBUG:boto:Data: 2024-02-01T10:22:59.294 DEBUG:boto:Headers: {} 2024-02-01T10:22:59.294 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:59.294 DEBUG:boto:Port: 8000 2024-02-01T10:22:59.294 DEBUG:boto:Params: {} 2024-02-01T10:22:59.294 DEBUG:boto:Token: None 2024-02-01T10:22:59.294 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:59 GMT /vclfcn-17/ 2024-02-01T10:22:59.294 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WmRanT6k02akI+r93G3HBmjiMxc= 2024-02-01T10:22:59.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WmRanT6k02akI+r93G3HBmjiMxc=', 'Content-Length': '0'} 2024-02-01T10:22:59.299 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd0ceb8d42484e82-0065bb7103-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:59.300 DEBUG:boto:b'vclfcn-17dummy1000falsedummy2024-02-01T10:20:14.297Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:59.300 DEBUG:boto:path=/vclfcn-17/dummy 2024-02-01T10:22:59.300 DEBUG:boto:auth_path=/vclfcn-17/dummy 2024-02-01T10:22:59.300 DEBUG:boto:Method: HEAD 2024-02-01T10:22:59.301 DEBUG:boto:Path: /vclfcn-17/dummy 2024-02-01T10:22:59.301 DEBUG:boto:Data: 2024-02-01T10:22:59.301 DEBUG:boto:Headers: {} 2024-02-01T10:22:59.301 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:59.301 DEBUG:boto:Port: 8001 2024-02-01T10:22:59.301 DEBUG:boto:Params: {} 2024-02-01T10:22:59.301 DEBUG:boto:Token: None 2024-02-01T10:22:59.302 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:59 GMT /vclfcn-17/dummy 2024-02-01T10:22:59.302 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QCjf/AVJSLAGccN5rEmn2hVWskQ= 2024-02-01T10:22:59.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QCjf/AVJSLAGccN5rEmn2hVWskQ=', 'Content-Length': '0'} 2024-02-01T10:22:59.305 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:14 GMT'), ('x-amz-version-id', 'hQXZXSD45t2CSRVMJOLMBXih54iMPmJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000023975122384281a1-0065bb7103-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:59.305 DEBUG:boto:path=/vclfcn-17/dummy 2024-02-01T10:22:59.306 DEBUG:boto:auth_path=/vclfcn-17/dummy 2024-02-01T10:22:59.306 DEBUG:boto:Method: HEAD 2024-02-01T10:22:59.306 DEBUG:boto:Path: /vclfcn-17/dummy 2024-02-01T10:22:59.306 DEBUG:boto:Data: 2024-02-01T10:22:59.306 DEBUG:boto:Headers: {} 2024-02-01T10:22:59.306 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:59.306 DEBUG:boto:Port: 8000 2024-02-01T10:22:59.306 DEBUG:boto:Params: {} 2024-02-01T10:22:59.307 DEBUG:boto:Token: None 2024-02-01T10:22:59.307 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:59 GMT /vclfcn-17/dummy 2024-02-01T10:22:59.307 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QCjf/AVJSLAGccN5rEmn2hVWskQ= 2024-02-01T10:22:59.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QCjf/AVJSLAGccN5rEmn2hVWskQ=', 'Content-Length': '0'} 2024-02-01T10:22:59.310 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:14 GMT'), ('x-amz-version-id', 'hQXZXSD45t2CSRVMJOLMBXih54iMPmJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000097e56fa0d0740ee3-0065bb7103-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:59.311 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-02-01T10:22:59.311 DEBUG:boto:path=/vclfcn-17/dummy 2024-02-01T10:22:59.311 DEBUG:boto:auth_path=/vclfcn-17/dummy 2024-02-01T10:22:59.311 DEBUG:boto:path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:59.311 DEBUG:boto:auth_path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:59.312 DEBUG:boto:Method: GET 2024-02-01T10:22:59.312 DEBUG:boto:Path: /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:59.312 DEBUG:boto:Data: 2024-02-01T10:22:59.312 DEBUG:boto:Headers: {} 2024-02-01T10:22:59.312 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:59.312 DEBUG:boto:Port: 8001 2024-02-01T10:22:59.312 DEBUG:boto:Params: {} 2024-02-01T10:22:59.312 DEBUG:boto:Token: None 2024-02-01T10:22:59.312 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:59 GMT /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:59.313 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WSDnx5IcJmE5Xn7ZWyZB1mVYxJg= 2024-02-01T10:22:59.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WSDnx5IcJmE5Xn7ZWyZB1mVYxJg=', 'Content-Length': '0'} 2024-02-01T10:22:59.315 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:14 GMT'), ('x-amz-version-id', 'hQXZXSD45t2CSRVMJOLMBXih54iMPmJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005cdf1c0603d69197-0065bb7103-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:59.316 DEBUG:boto:path=/vclfcn-17/dummy 2024-02-01T10:22:59.316 DEBUG:boto:auth_path=/vclfcn-17/dummy 2024-02-01T10:22:59.316 DEBUG:boto:path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:59.316 DEBUG:boto:auth_path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:59.316 DEBUG:boto:Method: GET 2024-02-01T10:22:59.316 DEBUG:boto:Path: /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:59.316 DEBUG:boto:Data: 2024-02-01T10:22:59.316 DEBUG:boto:Headers: {} 2024-02-01T10:22:59.316 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:59.317 DEBUG:boto:Port: 8000 2024-02-01T10:22:59.317 DEBUG:boto:Params: {} 2024-02-01T10:22:59.317 DEBUG:boto:Token: None 2024-02-01T10:22:59.317 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:59 GMT /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:22:59.317 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WSDnx5IcJmE5Xn7ZWyZB1mVYxJg= 2024-02-01T10:22:59.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WSDnx5IcJmE5Xn7ZWyZB1mVYxJg=', 'Content-Length': '0'} 2024-02-01T10:22:59.320 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:14 GMT'), ('x-amz-version-id', 'hQXZXSD45t2CSRVMJOLMBXih54iMPmJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000331ddd160ec4e216-0065bb7103-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:59.320 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:59.320 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:59.320 DEBUG:boto:path=/vclfcn-17/?prefix=dummy 2024-02-01T10:22:59.320 DEBUG:boto:auth_path=/vclfcn-17/?prefix=dummy 2024-02-01T10:22:59.320 DEBUG:boto:Method: GET 2024-02-01T10:22:59.321 DEBUG:boto:Path: /vclfcn-17/?prefix=dummy 2024-02-01T10:22:59.321 DEBUG:boto:Data: 2024-02-01T10:22:59.321 DEBUG:boto:Headers: {} 2024-02-01T10:22:59.321 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:59.321 DEBUG:boto:Port: 8001 2024-02-01T10:22:59.321 DEBUG:boto:Params: {} 2024-02-01T10:22:59.321 DEBUG:boto:Token: None 2024-02-01T10:22:59.321 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:59 GMT /vclfcn-17/ 2024-02-01T10:22:59.321 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WmRanT6k02akI+r93G3HBmjiMxc= 2024-02-01T10:22:59.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WmRanT6k02akI+r93G3HBmjiMxc=', 'Content-Length': '0'} 2024-02-01T10:22:59.326 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009806f928838626b0-0065bb7103-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:59.326 DEBUG:boto:b'vclfcn-17dummy1000falsedummy2024-02-01T10:20:14.297Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:59.326 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:59.326 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:59.327 DEBUG:boto:path=/vclfcn-17/?prefix=dummy 2024-02-01T10:22:59.327 DEBUG:boto:auth_path=/vclfcn-17/?prefix=dummy 2024-02-01T10:22:59.327 DEBUG:boto:Method: GET 2024-02-01T10:22:59.327 DEBUG:boto:Path: /vclfcn-17/?prefix=dummy 2024-02-01T10:22:59.327 DEBUG:boto:Data: 2024-02-01T10:22:59.327 DEBUG:boto:Headers: {} 2024-02-01T10:22:59.327 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:59.327 DEBUG:boto:Port: 8000 2024-02-01T10:22:59.327 DEBUG:boto:Params: {} 2024-02-01T10:22:59.327 DEBUG:boto:Token: None 2024-02-01T10:22:59.327 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:59 GMT /vclfcn-17/ 2024-02-01T10:22:59.328 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WmRanT6k02akI+r93G3HBmjiMxc= 2024-02-01T10:22:59.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WmRanT6k02akI+r93G3HBmjiMxc=', 'Content-Length': '0'} 2024-02-01T10:22:59.332 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000968271f242066d86-0065bb7103-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:59.332 DEBUG:boto:b'vclfcn-17dummy1000falsedummy2024-02-01T10:20:14.297Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:59.333 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-02-01T10:22:59.333 DEBUG:boto:path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:59.333 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:59.333 DEBUG:boto:path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:59.333 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:59.333 DEBUG:boto:Method: GET 2024-02-01T10:22:59.333 DEBUG:boto:Path: /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:59.333 DEBUG:boto:Data: 2024-02-01T10:22:59.334 DEBUG:boto:Headers: {} 2024-02-01T10:22:59.334 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:59.334 DEBUG:boto:Port: 8001 2024-02-01T10:22:59.334 DEBUG:boto:Params: {} 2024-02-01T10:22:59.334 DEBUG:boto:Token: None 2024-02-01T10:22:59.334 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:59 GMT /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:59.334 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:w5oxKCfN44ciR89eAONQ0qrCPW0= 2024-02-01T10:22:59.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:w5oxKCfN44ciR89eAONQ0qrCPW0=', 'Content-Length': '0'} 2024-02-01T10:22:59.338 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'xMxAGR4A-xEILepnzFPRgrWNTOLvitz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a34b08a699206850-0065bb7103-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:59.384 DEBUG:boto:path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:59.384 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:59.384 DEBUG:boto:path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:59.384 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:59.385 DEBUG:boto:Method: GET 2024-02-01T10:22:59.385 DEBUG:boto:Path: /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:59.385 DEBUG:boto:Data: 2024-02-01T10:22:59.385 DEBUG:boto:Headers: {} 2024-02-01T10:22:59.385 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:59.385 DEBUG:boto:Port: 8000 2024-02-01T10:22:59.385 DEBUG:boto:Params: {} 2024-02-01T10:22:59.385 DEBUG:boto:Token: None 2024-02-01T10:22:59.385 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:59 GMT /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:59.386 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:w5oxKCfN44ciR89eAONQ0qrCPW0= 2024-02-01T10:22:59.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:w5oxKCfN44ciR89eAONQ0qrCPW0=', 'Content-Length': '0'} 2024-02-01T10:22:59.388 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'xMxAGR4A-xEILepnzFPRgrWNTOLvitz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003bcaea5b1e955194-0065bb7103-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:59.389 DEBUG:boto:path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:59.389 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:59.389 DEBUG:boto:path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:59.389 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:59.389 DEBUG:boto:Method: HEAD 2024-02-01T10:22:59.389 DEBUG:boto:Path: /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:59.389 DEBUG:boto:Data: 2024-02-01T10:22:59.389 DEBUG:boto:Headers: {} 2024-02-01T10:22:59.389 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:59.390 DEBUG:boto:Port: 8001 2024-02-01T10:22:59.390 DEBUG:boto:Params: {} 2024-02-01T10:22:59.390 DEBUG:boto:Token: None 2024-02-01T10:22:59.390 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:59 GMT /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:59.390 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:LMYATEZZByq/PZa6ZyzmVj79oxQ= 2024-02-01T10:22:59.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:LMYATEZZByq/PZa6ZyzmVj79oxQ=', 'Content-Length': '0'} 2024-02-01T10:22:59.393 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'xMxAGR4A-xEILepnzFPRgrWNTOLvitz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008cda8118873b369a-0065bb7103-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:59.393 DEBUG:boto:path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:59.393 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:59.393 DEBUG:boto:path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:59.393 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:59.393 DEBUG:boto:Method: HEAD 2024-02-01T10:22:59.394 DEBUG:boto:Path: /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:59.394 DEBUG:boto:Data: 2024-02-01T10:22:59.394 DEBUG:boto:Headers: {} 2024-02-01T10:22:59.394 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:59.394 DEBUG:boto:Port: 8000 2024-02-01T10:22:59.394 DEBUG:boto:Params: {} 2024-02-01T10:22:59.394 DEBUG:boto:Token: None 2024-02-01T10:22:59.394 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:59 GMT /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:59.394 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:LMYATEZZByq/PZa6ZyzmVj79oxQ= 2024-02-01T10:22:59.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:LMYATEZZByq/PZa6ZyzmVj79oxQ=', 'Content-Length': '0'} 2024-02-01T10:22:59.397 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'xMxAGR4A-xEILepnzFPRgrWNTOLvitz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000011ca24981528afd5-0065bb7103-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:59.397 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-02-01T10:22:59.397 DEBUG:boto:path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:59.397 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:59.398 DEBUG:boto:path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:59.398 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:59.398 DEBUG:boto:Method: GET 2024-02-01T10:22:59.398 DEBUG:boto:Path: /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:59.398 DEBUG:boto:Data: 2024-02-01T10:22:59.398 DEBUG:boto:Headers: {} 2024-02-01T10:22:59.398 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:59.398 DEBUG:boto:Port: 8001 2024-02-01T10:22:59.398 DEBUG:boto:Params: {} 2024-02-01T10:22:59.398 DEBUG:boto:Token: None 2024-02-01T10:22:59.399 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:59 GMT /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:59.399 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:w5oxKCfN44ciR89eAONQ0qrCPW0= 2024-02-01T10:22:59.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:w5oxKCfN44ciR89eAONQ0qrCPW0=', 'Content-Length': '0'} 2024-02-01T10:22:59.401 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'xMxAGR4A-xEILepnzFPRgrWNTOLvitz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006965ab56fe12c080-0065bb7103-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:59.448 DEBUG:boto:path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:59.448 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:59.448 DEBUG:boto:path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:59.448 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:59.449 DEBUG:boto:Method: GET 2024-02-01T10:22:59.449 DEBUG:boto:Path: /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:59.449 DEBUG:boto:Data: 2024-02-01T10:22:59.449 DEBUG:boto:Headers: {} 2024-02-01T10:22:59.449 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:59.449 DEBUG:boto:Port: 8000 2024-02-01T10:22:59.449 DEBUG:boto:Params: {} 2024-02-01T10:22:59.449 DEBUG:boto:Token: None 2024-02-01T10:22:59.449 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:59 GMT /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:59.450 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:w5oxKCfN44ciR89eAONQ0qrCPW0= 2024-02-01T10:22:59.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:w5oxKCfN44ciR89eAONQ0qrCPW0=', 'Content-Length': '0'} 2024-02-01T10:22:59.452 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'xMxAGR4A-xEILepnzFPRgrWNTOLvitz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008b564cbff03ac9f7-0065bb7103-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:59.453 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:59.453 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:59.453 DEBUG:boto:path=/vclfcn-17/?prefix=obj-test-zone1 2024-02-01T10:22:59.453 DEBUG:boto:auth_path=/vclfcn-17/?prefix=obj-test-zone1 2024-02-01T10:22:59.453 DEBUG:boto:Method: GET 2024-02-01T10:22:59.453 DEBUG:boto:Path: /vclfcn-17/?prefix=obj-test-zone1 2024-02-01T10:22:59.453 DEBUG:boto:Data: 2024-02-01T10:22:59.454 DEBUG:boto:Headers: {} 2024-02-01T10:22:59.454 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:59.454 DEBUG:boto:Port: 8001 2024-02-01T10:22:59.454 DEBUG:boto:Params: {} 2024-02-01T10:22:59.454 DEBUG:boto:Token: None 2024-02-01T10:22:59.454 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:59 GMT /vclfcn-17/ 2024-02-01T10:22:59.454 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WmRanT6k02akI+r93G3HBmjiMxc= 2024-02-01T10:22:59.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WmRanT6k02akI+r93G3HBmjiMxc=', 'Content-Length': '0'} 2024-02-01T10:22:59.459 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000dc0b173148173ff-0065bb7103-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:59.504 DEBUG:boto:b'vclfcn-17obj-test-zone11000falseobj-test-zone12024-02-01T10:20:50.473Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:59.505 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:59.505 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:59.505 DEBUG:boto:path=/vclfcn-17/?prefix=obj-test-zone1 2024-02-01T10:22:59.505 DEBUG:boto:auth_path=/vclfcn-17/?prefix=obj-test-zone1 2024-02-01T10:22:59.505 DEBUG:boto:Method: GET 2024-02-01T10:22:59.505 DEBUG:boto:Path: /vclfcn-17/?prefix=obj-test-zone1 2024-02-01T10:22:59.505 DEBUG:boto:Data: 2024-02-01T10:22:59.505 DEBUG:boto:Headers: {} 2024-02-01T10:22:59.505 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:59.505 DEBUG:boto:Port: 8000 2024-02-01T10:22:59.506 DEBUG:boto:Params: {} 2024-02-01T10:22:59.506 DEBUG:boto:Token: None 2024-02-01T10:22:59.506 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:59 GMT /vclfcn-17/ 2024-02-01T10:22:59.506 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WmRanT6k02akI+r93G3HBmjiMxc= 2024-02-01T10:22:59.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WmRanT6k02akI+r93G3HBmjiMxc=', 'Content-Length': '0'} 2024-02-01T10:22:59.510 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd62c934eb87ab7b-0065bb7103-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:59.511 DEBUG:boto:b'vclfcn-17obj-test-zone11000falseobj-test-zone12024-02-01T10:20:50.473Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:59.511 DEBUG:boto:path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:59.511 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:59.511 DEBUG:boto:Method: HEAD 2024-02-01T10:22:59.511 DEBUG:boto:Path: /vclfcn-17/obj-test-zone1 2024-02-01T10:22:59.511 DEBUG:boto:Data: 2024-02-01T10:22:59.512 DEBUG:boto:Headers: {} 2024-02-01T10:22:59.512 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:59.512 DEBUG:boto:Port: 8001 2024-02-01T10:22:59.512 DEBUG:boto:Params: {} 2024-02-01T10:22:59.512 DEBUG:boto:Token: None 2024-02-01T10:22:59.512 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:59 GMT /vclfcn-17/obj-test-zone1 2024-02-01T10:22:59.512 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ijuh9RpnMs8tp0OaHsZ+cmenBQk= 2024-02-01T10:22:59.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ijuh9RpnMs8tp0OaHsZ+cmenBQk=', 'Content-Length': '0'} 2024-02-01T10:22:59.516 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'xMxAGR4A-xEILepnzFPRgrWNTOLvitz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004dccf60adf281694-0065bb7103-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:59.516 DEBUG:boto:path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:59.516 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:59.516 DEBUG:boto:Method: HEAD 2024-02-01T10:22:59.517 DEBUG:boto:Path: /vclfcn-17/obj-test-zone1 2024-02-01T10:22:59.517 DEBUG:boto:Data: 2024-02-01T10:22:59.517 DEBUG:boto:Headers: {} 2024-02-01T10:22:59.517 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:59.517 DEBUG:boto:Port: 8000 2024-02-01T10:22:59.517 DEBUG:boto:Params: {} 2024-02-01T10:22:59.517 DEBUG:boto:Token: None 2024-02-01T10:22:59.517 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:59 GMT /vclfcn-17/obj-test-zone1 2024-02-01T10:22:59.517 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ijuh9RpnMs8tp0OaHsZ+cmenBQk= 2024-02-01T10:22:59.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ijuh9RpnMs8tp0OaHsZ+cmenBQk=', 'Content-Length': '0'} 2024-02-01T10:22:59.521 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'xMxAGR4A-xEILepnzFPRgrWNTOLvitz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000de5afa2ad453c5ff-0065bb7103-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:59.521 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-02-01T10:22:59.521 DEBUG:boto:path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:59.521 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:59.522 DEBUG:boto:path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:59.522 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:59.522 DEBUG:boto:Method: GET 2024-02-01T10:22:59.522 DEBUG:boto:Path: /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:59.522 DEBUG:boto:Data: 2024-02-01T10:22:59.522 DEBUG:boto:Headers: {} 2024-02-01T10:22:59.522 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:59.522 DEBUG:boto:Port: 8001 2024-02-01T10:22:59.522 DEBUG:boto:Params: {} 2024-02-01T10:22:59.522 DEBUG:boto:Token: None 2024-02-01T10:22:59.523 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:59 GMT /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:59.523 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:w5oxKCfN44ciR89eAONQ0qrCPW0= 2024-02-01T10:22:59.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:w5oxKCfN44ciR89eAONQ0qrCPW0=', 'Content-Length': '0'} 2024-02-01T10:22:59.525 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'xMxAGR4A-xEILepnzFPRgrWNTOLvitz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ee0a9e08bdf3ce42-0065bb7103-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:59.568 DEBUG:boto:path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:59.568 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1 2024-02-01T10:22:59.569 DEBUG:boto:path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:59.569 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:59.569 DEBUG:boto:Method: GET 2024-02-01T10:22:59.569 DEBUG:boto:Path: /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:59.569 DEBUG:boto:Data: 2024-02-01T10:22:59.569 DEBUG:boto:Headers: {} 2024-02-01T10:22:59.569 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:59.570 DEBUG:boto:Port: 8000 2024-02-01T10:22:59.570 DEBUG:boto:Params: {} 2024-02-01T10:22:59.570 DEBUG:boto:Token: None 2024-02-01T10:22:59.570 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:59 GMT /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:22:59.570 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:w5oxKCfN44ciR89eAONQ0qrCPW0= 2024-02-01T10:22:59.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:w5oxKCfN44ciR89eAONQ0qrCPW0=', 'Content-Length': '0'} 2024-02-01T10:22:59.573 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'xMxAGR4A-xEILepnzFPRgrWNTOLvitz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001e515ccc6601d95c-0065bb7103-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:59.574 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:59.574 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:59.574 DEBUG:boto:path=/vclfcn-17/?prefix=obj-test-zone1 2024-02-01T10:22:59.574 DEBUG:boto:auth_path=/vclfcn-17/?prefix=obj-test-zone1 2024-02-01T10:22:59.574 DEBUG:boto:Method: GET 2024-02-01T10:22:59.574 DEBUG:boto:Path: /vclfcn-17/?prefix=obj-test-zone1 2024-02-01T10:22:59.574 DEBUG:boto:Data: 2024-02-01T10:22:59.574 DEBUG:boto:Headers: {} 2024-02-01T10:22:59.574 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:59.575 DEBUG:boto:Port: 8001 2024-02-01T10:22:59.575 DEBUG:boto:Params: {} 2024-02-01T10:22:59.575 DEBUG:boto:Token: None 2024-02-01T10:22:59.575 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:59 GMT /vclfcn-17/ 2024-02-01T10:22:59.575 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WmRanT6k02akI+r93G3HBmjiMxc= 2024-02-01T10:22:59.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WmRanT6k02akI+r93G3HBmjiMxc=', 'Content-Length': '0'} 2024-02-01T10:22:59.579 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f842a064db59dcfb-0065bb7103-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:59.624 DEBUG:boto:b'vclfcn-17obj-test-zone11000falseobj-test-zone12024-02-01T10:20:50.473Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:59.625 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:59.625 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:59.625 DEBUG:boto:path=/vclfcn-17/?prefix=obj-test-zone1 2024-02-01T10:22:59.625 DEBUG:boto:auth_path=/vclfcn-17/?prefix=obj-test-zone1 2024-02-01T10:22:59.625 DEBUG:boto:Method: GET 2024-02-01T10:22:59.625 DEBUG:boto:Path: /vclfcn-17/?prefix=obj-test-zone1 2024-02-01T10:22:59.625 DEBUG:boto:Data: 2024-02-01T10:22:59.625 DEBUG:boto:Headers: {} 2024-02-01T10:22:59.625 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:59.625 DEBUG:boto:Port: 8000 2024-02-01T10:22:59.626 DEBUG:boto:Params: {} 2024-02-01T10:22:59.626 DEBUG:boto:Token: None 2024-02-01T10:22:59.626 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:59 GMT /vclfcn-17/ 2024-02-01T10:22:59.626 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WmRanT6k02akI+r93G3HBmjiMxc= 2024-02-01T10:22:59.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WmRanT6k02akI+r93G3HBmjiMxc=', 'Content-Length': '0'} 2024-02-01T10:22:59.630 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063f735c765fc0f4d-0065bb7103-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:59.631 DEBUG:boto:b'vclfcn-17obj-test-zone11000falseobj-test-zone12024-02-01T10:20:50.473Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:59.631 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-02-01T10:22:59.631 DEBUG:boto:path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:59.631 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:59.631 DEBUG:boto:path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:59.631 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:59.632 DEBUG:boto:Method: GET 2024-02-01T10:22:59.632 DEBUG:boto:Path: /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:59.632 DEBUG:boto:Data: 2024-02-01T10:22:59.632 DEBUG:boto:Headers: {} 2024-02-01T10:22:59.632 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:59.632 DEBUG:boto:Port: 8001 2024-02-01T10:22:59.632 DEBUG:boto:Params: {} 2024-02-01T10:22:59.632 DEBUG:boto:Token: None 2024-02-01T10:22:59.632 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:59 GMT /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:59.633 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BH6rp69orEGPz6FX/NQ/Skq3R4I= 2024-02-01T10:22:59.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BH6rp69orEGPz6FX/NQ/Skq3R4I=', 'Content-Length': '0'} 2024-02-01T10:22:59.636 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', '-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000082add8f53c486767-0065bb7103-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:59.680 DEBUG:boto:path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:59.680 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:59.680 DEBUG:boto:path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:59.680 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:59.681 DEBUG:boto:Method: GET 2024-02-01T10:22:59.681 DEBUG:boto:Path: /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:59.681 DEBUG:boto:Data: 2024-02-01T10:22:59.681 DEBUG:boto:Headers: {} 2024-02-01T10:22:59.681 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:59.681 DEBUG:boto:Port: 8000 2024-02-01T10:22:59.681 DEBUG:boto:Params: {} 2024-02-01T10:22:59.681 DEBUG:boto:Token: None 2024-02-01T10:22:59.681 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:59 GMT /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:59.681 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BH6rp69orEGPz6FX/NQ/Skq3R4I= 2024-02-01T10:22:59.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BH6rp69orEGPz6FX/NQ/Skq3R4I=', 'Content-Length': '0'} 2024-02-01T10:22:59.684 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', '-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000095f2925bf905c7f1-0065bb7103-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:59.728 DEBUG:boto:path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:59.728 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:59.728 DEBUG:boto:path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:59.728 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:59.729 DEBUG:boto:Method: HEAD 2024-02-01T10:22:59.729 DEBUG:boto:Path: /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:59.729 DEBUG:boto:Data: 2024-02-01T10:22:59.729 DEBUG:boto:Headers: {} 2024-02-01T10:22:59.729 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:59.729 DEBUG:boto:Port: 8001 2024-02-01T10:22:59.729 DEBUG:boto:Params: {} 2024-02-01T10:22:59.729 DEBUG:boto:Token: None 2024-02-01T10:22:59.729 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:59 GMT /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:59.730 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:v90hwV/7U2I70PkRWfVSAIU/iiQ= 2024-02-01T10:22:59.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:v90hwV/7U2I70PkRWfVSAIU/iiQ=', 'Content-Length': '0'} 2024-02-01T10:22:59.733 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', '-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008a7cebe3f4d43580-0065bb7103-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:59.733 DEBUG:boto:path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:59.733 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:59.733 DEBUG:boto:path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:59.733 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:59.733 DEBUG:boto:Method: HEAD 2024-02-01T10:22:59.733 DEBUG:boto:Path: /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:59.733 DEBUG:boto:Data: 2024-02-01T10:22:59.734 DEBUG:boto:Headers: {} 2024-02-01T10:22:59.734 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:59.734 DEBUG:boto:Port: 8000 2024-02-01T10:22:59.734 DEBUG:boto:Params: {} 2024-02-01T10:22:59.734 DEBUG:boto:Token: None 2024-02-01T10:22:59.734 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:59 GMT /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:59.734 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:v90hwV/7U2I70PkRWfVSAIU/iiQ= 2024-02-01T10:22:59.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:v90hwV/7U2I70PkRWfVSAIU/iiQ=', 'Content-Length': '0'} 2024-02-01T10:22:59.737 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', '-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e2a6106d7e6c82c5-0065bb7103-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:59.737 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-02-01T10:22:59.737 DEBUG:boto:path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:59.738 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:59.738 DEBUG:boto:path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:59.738 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:59.738 DEBUG:boto:Method: GET 2024-02-01T10:22:59.738 DEBUG:boto:Path: /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:59.738 DEBUG:boto:Data: 2024-02-01T10:22:59.738 DEBUG:boto:Headers: {} 2024-02-01T10:22:59.738 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:59.738 DEBUG:boto:Port: 8001 2024-02-01T10:22:59.738 DEBUG:boto:Params: {} 2024-02-01T10:22:59.738 DEBUG:boto:Token: None 2024-02-01T10:22:59.739 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:59 GMT /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:59.739 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BH6rp69orEGPz6FX/NQ/Skq3R4I= 2024-02-01T10:22:59.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BH6rp69orEGPz6FX/NQ/Skq3R4I=', 'Content-Length': '0'} 2024-02-01T10:22:59.741 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', '-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005203ceb527e1ba0a-0065bb7103-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:59.788 DEBUG:boto:path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:59.788 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:59.788 DEBUG:boto:path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:59.789 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:59.789 DEBUG:boto:Method: GET 2024-02-01T10:22:59.789 DEBUG:boto:Path: /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:59.789 DEBUG:boto:Data: 2024-02-01T10:22:59.789 DEBUG:boto:Headers: {} 2024-02-01T10:22:59.789 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:59.789 DEBUG:boto:Port: 8000 2024-02-01T10:22:59.789 DEBUG:boto:Params: {} 2024-02-01T10:22:59.789 DEBUG:boto:Token: None 2024-02-01T10:22:59.789 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:59 GMT /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:59.790 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BH6rp69orEGPz6FX/NQ/Skq3R4I= 2024-02-01T10:22:59.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BH6rp69orEGPz6FX/NQ/Skq3R4I=', 'Content-Length': '0'} 2024-02-01T10:22:59.793 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', '-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000010e39bfeda9ee7af-0065bb7103-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:59.840 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:59.840 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:59.840 DEBUG:boto:path=/vclfcn-17/?prefix=obj-test-zone2 2024-02-01T10:22:59.841 DEBUG:boto:auth_path=/vclfcn-17/?prefix=obj-test-zone2 2024-02-01T10:22:59.841 DEBUG:boto:Method: GET 2024-02-01T10:22:59.841 DEBUG:boto:Path: /vclfcn-17/?prefix=obj-test-zone2 2024-02-01T10:22:59.841 DEBUG:boto:Data: 2024-02-01T10:22:59.841 DEBUG:boto:Headers: {} 2024-02-01T10:22:59.841 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:59.841 DEBUG:boto:Port: 8001 2024-02-01T10:22:59.841 DEBUG:boto:Params: {} 2024-02-01T10:22:59.841 DEBUG:boto:Token: None 2024-02-01T10:22:59.841 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:59 GMT /vclfcn-17/ 2024-02-01T10:22:59.842 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WmRanT6k02akI+r93G3HBmjiMxc= 2024-02-01T10:22:59.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WmRanT6k02akI+r93G3HBmjiMxc=', 'Content-Length': '0'} 2024-02-01T10:22:59.846 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005da5fd53829e5b4a-0065bb7103-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:59.846 DEBUG:boto:b'vclfcn-17obj-test-zone21000falseobj-test-zone22024-02-01T10:20:50.624Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:59.846 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:59.847 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:59.847 DEBUG:boto:path=/vclfcn-17/?prefix=obj-test-zone2 2024-02-01T10:22:59.847 DEBUG:boto:auth_path=/vclfcn-17/?prefix=obj-test-zone2 2024-02-01T10:22:59.847 DEBUG:boto:Method: GET 2024-02-01T10:22:59.847 DEBUG:boto:Path: /vclfcn-17/?prefix=obj-test-zone2 2024-02-01T10:22:59.847 DEBUG:boto:Data: 2024-02-01T10:22:59.847 DEBUG:boto:Headers: {} 2024-02-01T10:22:59.847 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:59.847 DEBUG:boto:Port: 8000 2024-02-01T10:22:59.847 DEBUG:boto:Params: {} 2024-02-01T10:22:59.848 DEBUG:boto:Token: None 2024-02-01T10:22:59.848 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:59 GMT /vclfcn-17/ 2024-02-01T10:22:59.848 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WmRanT6k02akI+r93G3HBmjiMxc= 2024-02-01T10:22:59.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WmRanT6k02akI+r93G3HBmjiMxc=', 'Content-Length': '0'} 2024-02-01T10:22:59.852 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bfc9dcff3c737245-0065bb7103-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:59.853 DEBUG:boto:b'vclfcn-17obj-test-zone21000falseobj-test-zone22024-02-01T10:20:50.624Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:22:59.853 DEBUG:boto:path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:59.853 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:59.853 DEBUG:boto:Method: HEAD 2024-02-01T10:22:59.853 DEBUG:boto:Path: /vclfcn-17/obj-test-zone2 2024-02-01T10:22:59.853 DEBUG:boto:Data: 2024-02-01T10:22:59.853 DEBUG:boto:Headers: {} 2024-02-01T10:22:59.853 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:59.854 DEBUG:boto:Port: 8001 2024-02-01T10:22:59.854 DEBUG:boto:Params: {} 2024-02-01T10:22:59.854 DEBUG:boto:Token: None 2024-02-01T10:22:59.854 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:59 GMT /vclfcn-17/obj-test-zone2 2024-02-01T10:22:59.854 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:MUrVkVtYMlTcI0NgxB9swEef67I= 2024-02-01T10:22:59.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:MUrVkVtYMlTcI0NgxB9swEef67I=', 'Content-Length': '0'} 2024-02-01T10:22:59.858 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', '-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000298f0010305ee28c-0065bb7103-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:59.858 DEBUG:boto:path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:59.858 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:59.858 DEBUG:boto:Method: HEAD 2024-02-01T10:22:59.858 DEBUG:boto:Path: /vclfcn-17/obj-test-zone2 2024-02-01T10:22:59.858 DEBUG:boto:Data: 2024-02-01T10:22:59.858 DEBUG:boto:Headers: {} 2024-02-01T10:22:59.859 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:59.859 DEBUG:boto:Port: 8000 2024-02-01T10:22:59.859 DEBUG:boto:Params: {} 2024-02-01T10:22:59.859 DEBUG:boto:Token: None 2024-02-01T10:22:59.859 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:22:59 GMT /vclfcn-17/obj-test-zone2 2024-02-01T10:22:59.859 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:MUrVkVtYMlTcI0NgxB9swEef67I= 2024-02-01T10:22:59.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:MUrVkVtYMlTcI0NgxB9swEef67I=', 'Content-Length': '0'} 2024-02-01T10:22:59.863 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', '-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000165c3af5a8d545b2-0065bb7103-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:59.863 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-02-01T10:22:59.863 DEBUG:boto:path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:59.863 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:59.863 DEBUG:boto:path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:59.863 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:59.864 DEBUG:boto:Method: GET 2024-02-01T10:22:59.864 DEBUG:boto:Path: /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:59.864 DEBUG:boto:Data: 2024-02-01T10:22:59.864 DEBUG:boto:Headers: {} 2024-02-01T10:22:59.864 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:59.864 DEBUG:boto:Port: 8001 2024-02-01T10:22:59.864 DEBUG:boto:Params: {} 2024-02-01T10:22:59.864 DEBUG:boto:Token: None 2024-02-01T10:22:59.865 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:59 GMT /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:59.865 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BH6rp69orEGPz6FX/NQ/Skq3R4I= 2024-02-01T10:22:59.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BH6rp69orEGPz6FX/NQ/Skq3R4I=', 'Content-Length': '0'} 2024-02-01T10:22:59.868 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', '-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d033ac3ba015a1bf-0065bb7103-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:59.908 DEBUG:boto:path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:59.908 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2 2024-02-01T10:22:59.908 DEBUG:boto:path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:59.908 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:59.909 DEBUG:boto:Method: GET 2024-02-01T10:22:59.909 DEBUG:boto:Path: /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:59.909 DEBUG:boto:Data: 2024-02-01T10:22:59.909 DEBUG:boto:Headers: {} 2024-02-01T10:22:59.909 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:22:59.909 DEBUG:boto:Port: 8000 2024-02-01T10:22:59.909 DEBUG:boto:Params: {} 2024-02-01T10:22:59.909 DEBUG:boto:Token: None 2024-02-01T10:22:59.910 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:59 GMT /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:22:59.910 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BH6rp69orEGPz6FX/NQ/Skq3R4I= 2024-02-01T10:22:59.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BH6rp69orEGPz6FX/NQ/Skq3R4I=', 'Content-Length': '0'} 2024-02-01T10:22:59.913 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', '-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ef878f4d1b879b16-0065bb7103-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:22:59.956 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:22:59.956 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:22:59.957 DEBUG:boto:path=/vclfcn-17/?prefix=obj-test-zone2 2024-02-01T10:22:59.957 DEBUG:boto:auth_path=/vclfcn-17/?prefix=obj-test-zone2 2024-02-01T10:22:59.957 DEBUG:boto:Method: GET 2024-02-01T10:22:59.957 DEBUG:boto:Path: /vclfcn-17/?prefix=obj-test-zone2 2024-02-01T10:22:59.957 DEBUG:boto:Data: 2024-02-01T10:22:59.957 DEBUG:boto:Headers: {} 2024-02-01T10:22:59.957 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:22:59.957 DEBUG:boto:Port: 8001 2024-02-01T10:22:59.957 DEBUG:boto:Params: {} 2024-02-01T10:22:59.957 DEBUG:boto:Token: None 2024-02-01T10:22:59.958 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:22:59 GMT /vclfcn-17/ 2024-02-01T10:22:59.958 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WmRanT6k02akI+r93G3HBmjiMxc= 2024-02-01T10:22:59.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:22:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WmRanT6k02akI+r93G3HBmjiMxc=', 'Content-Length': '0'} 2024-02-01T10:22:59.962 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b74d7d56a38c895c-0065bb7103-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:22:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:00.004 DEBUG:boto:b'vclfcn-17obj-test-zone21000falseobj-test-zone22024-02-01T10:20:50.624Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:00.005 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:23:00.005 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:23:00.005 DEBUG:boto:path=/vclfcn-17/?prefix=obj-test-zone2 2024-02-01T10:23:00.005 DEBUG:boto:auth_path=/vclfcn-17/?prefix=obj-test-zone2 2024-02-01T10:23:00.005 DEBUG:boto:Method: GET 2024-02-01T10:23:00.005 DEBUG:boto:Path: /vclfcn-17/?prefix=obj-test-zone2 2024-02-01T10:23:00.005 DEBUG:boto:Data: 2024-02-01T10:23:00.005 DEBUG:boto:Headers: {} 2024-02-01T10:23:00.006 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:23:00.006 DEBUG:boto:Port: 8000 2024-02-01T10:23:00.006 DEBUG:boto:Params: {} 2024-02-01T10:23:00.006 DEBUG:boto:Token: None 2024-02-01T10:23:00.006 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:00 GMT /vclfcn-17/ 2024-02-01T10:23:00.006 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:SN+NL+snQVDoyY0UVvFbtJmlCFI= 2024-02-01T10:23:00.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:SN+NL+snQVDoyY0UVvFbtJmlCFI=', 'Content-Length': '0'} 2024-02-01T10:23:00.012 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064f2feda06a8e8ce-0065bb7104-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:00.012 DEBUG:boto:b'vclfcn-17obj-test-zone21000falseobj-test-zone22024-02-01T10:20:50.624Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:00.013 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-02-01T10:23:00.013 DEBUG:boto:path=/vclfcn-17/obj-test-zone3 2024-02-01T10:23:00.014 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3 2024-02-01T10:23:00.014 DEBUG:boto:path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:23:00.014 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:23:00.014 DEBUG:boto:Method: GET 2024-02-01T10:23:00.014 DEBUG:boto:Path: /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:23:00.015 DEBUG:boto:Data: 2024-02-01T10:23:00.015 DEBUG:boto:Headers: {} 2024-02-01T10:23:00.015 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:00.015 DEBUG:boto:Port: 8001 2024-02-01T10:23:00.015 DEBUG:boto:Params: {} 2024-02-01T10:23:00.016 DEBUG:boto:Token: None 2024-02-01T10:23:00.016 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:00 GMT /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:23:00.016 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Vo5ocW0YHxLHHPfik5rf5U6gO6I= 2024-02-01T10:23:00.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Vo5ocW0YHxLHHPfik5rf5U6gO6I=', 'Content-Length': '0'} 2024-02-01T10:23:00.020 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'YE736OwOOXbZJHJe7d8YvcNV1LxwcQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001d43bb4e9e6df181-0065bb7104-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:00.021 DEBUG:boto:path=/vclfcn-17/obj-test-zone3 2024-02-01T10:23:00.021 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3 2024-02-01T10:23:00.021 DEBUG:boto:path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:23:00.021 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:23:00.022 DEBUG:boto:Method: GET 2024-02-01T10:23:00.022 DEBUG:boto:Path: /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:23:00.022 DEBUG:boto:Data: 2024-02-01T10:23:00.023 DEBUG:boto:Headers: {} 2024-02-01T10:23:00.023 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:23:00.023 DEBUG:boto:Port: 8000 2024-02-01T10:23:00.024 DEBUG:boto:Params: {} 2024-02-01T10:23:00.024 DEBUG:boto:Token: None 2024-02-01T10:23:00.024 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:00 GMT /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:23:00.025 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Vo5ocW0YHxLHHPfik5rf5U6gO6I= 2024-02-01T10:23:00.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Vo5ocW0YHxLHHPfik5rf5U6gO6I=', 'Content-Length': '0'} 2024-02-01T10:23:00.028 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'YE736OwOOXbZJHJe7d8YvcNV1LxwcQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000349edc1180894ad4-0065bb7104-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:00.029 DEBUG:boto:path=/vclfcn-17/obj-test-zone3 2024-02-01T10:23:00.029 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3 2024-02-01T10:23:00.029 DEBUG:boto:path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:23:00.029 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:23:00.029 DEBUG:boto:Method: HEAD 2024-02-01T10:23:00.029 DEBUG:boto:Path: /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:23:00.029 DEBUG:boto:Data: 2024-02-01T10:23:00.030 DEBUG:boto:Headers: {} 2024-02-01T10:23:00.030 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:00.030 DEBUG:boto:Port: 8001 2024-02-01T10:23:00.030 DEBUG:boto:Params: {} 2024-02-01T10:23:00.030 DEBUG:boto:Token: None 2024-02-01T10:23:00.030 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:23:00 GMT /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:23:00.031 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vaqG3XhdViOOe35DJ3YM2M9U/Ck= 2024-02-01T10:23:00.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vaqG3XhdViOOe35DJ3YM2M9U/Ck=', 'Content-Length': '0'} 2024-02-01T10:23:00.034 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'YE736OwOOXbZJHJe7d8YvcNV1LxwcQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d5b88642340e426f-0065bb7104-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:00.034 DEBUG:boto:path=/vclfcn-17/obj-test-zone3 2024-02-01T10:23:00.034 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3 2024-02-01T10:23:00.034 DEBUG:boto:path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:23:00.034 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:23:00.035 DEBUG:boto:Method: HEAD 2024-02-01T10:23:00.035 DEBUG:boto:Path: /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:23:00.035 DEBUG:boto:Data: 2024-02-01T10:23:00.035 DEBUG:boto:Headers: {} 2024-02-01T10:23:00.035 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:23:00.035 DEBUG:boto:Port: 8000 2024-02-01T10:23:00.035 DEBUG:boto:Params: {} 2024-02-01T10:23:00.035 DEBUG:boto:Token: None 2024-02-01T10:23:00.035 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:23:00 GMT /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:23:00.035 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vaqG3XhdViOOe35DJ3YM2M9U/Ck= 2024-02-01T10:23:00.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vaqG3XhdViOOe35DJ3YM2M9U/Ck=', 'Content-Length': '0'} 2024-02-01T10:23:00.039 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'YE736OwOOXbZJHJe7d8YvcNV1LxwcQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000065168b56c1d33a6f-0065bb7104-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:00.039 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-02-01T10:23:00.039 DEBUG:boto:path=/vclfcn-17/obj-test-zone3 2024-02-01T10:23:00.039 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3 2024-02-01T10:23:00.039 DEBUG:boto:path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:23:00.039 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:23:00.039 DEBUG:boto:Method: GET 2024-02-01T10:23:00.040 DEBUG:boto:Path: /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:23:00.040 DEBUG:boto:Data: 2024-02-01T10:23:00.040 DEBUG:boto:Headers: {} 2024-02-01T10:23:00.040 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:00.040 DEBUG:boto:Port: 8001 2024-02-01T10:23:00.040 DEBUG:boto:Params: {} 2024-02-01T10:23:00.040 DEBUG:boto:Token: None 2024-02-01T10:23:00.040 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:00 GMT /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:23:00.040 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Vo5ocW0YHxLHHPfik5rf5U6gO6I= 2024-02-01T10:23:00.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Vo5ocW0YHxLHHPfik5rf5U6gO6I=', 'Content-Length': '0'} 2024-02-01T10:23:00.043 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'YE736OwOOXbZJHJe7d8YvcNV1LxwcQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a06a6d45d4b0d852-0065bb7104-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:00.043 DEBUG:boto:path=/vclfcn-17/obj-test-zone3 2024-02-01T10:23:00.044 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3 2024-02-01T10:23:00.044 DEBUG:boto:path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:23:00.044 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:23:00.044 DEBUG:boto:Method: GET 2024-02-01T10:23:00.044 DEBUG:boto:Path: /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:23:00.044 DEBUG:boto:Data: 2024-02-01T10:23:00.044 DEBUG:boto:Headers: {} 2024-02-01T10:23:00.044 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:23:00.044 DEBUG:boto:Port: 8000 2024-02-01T10:23:00.044 DEBUG:boto:Params: {} 2024-02-01T10:23:00.045 DEBUG:boto:Token: None 2024-02-01T10:23:00.045 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:00 GMT /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:23:00.045 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Vo5ocW0YHxLHHPfik5rf5U6gO6I= 2024-02-01T10:23:00.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Vo5ocW0YHxLHHPfik5rf5U6gO6I=', 'Content-Length': '0'} 2024-02-01T10:23:00.048 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'YE736OwOOXbZJHJe7d8YvcNV1LxwcQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000048bc42a95c89110d-0065bb7104-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:00.088 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:23:00.088 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:23:00.088 DEBUG:boto:path=/vclfcn-17/?prefix=obj-test-zone3 2024-02-01T10:23:00.088 DEBUG:boto:auth_path=/vclfcn-17/?prefix=obj-test-zone3 2024-02-01T10:23:00.089 DEBUG:boto:Method: GET 2024-02-01T10:23:00.089 DEBUG:boto:Path: /vclfcn-17/?prefix=obj-test-zone3 2024-02-01T10:23:00.089 DEBUG:boto:Data: 2024-02-01T10:23:00.089 DEBUG:boto:Headers: {} 2024-02-01T10:23:00.089 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:00.089 DEBUG:boto:Port: 8001 2024-02-01T10:23:00.089 DEBUG:boto:Params: {} 2024-02-01T10:23:00.089 DEBUG:boto:Token: None 2024-02-01T10:23:00.089 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:00 GMT /vclfcn-17/ 2024-02-01T10:23:00.090 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:SN+NL+snQVDoyY0UVvFbtJmlCFI= 2024-02-01T10:23:00.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:SN+NL+snQVDoyY0UVvFbtJmlCFI=', 'Content-Length': '0'} 2024-02-01T10:23:00.094 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000014f6032072b6e00b-0065bb7104-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:00.095 DEBUG:boto:b'vclfcn-17obj-test-zone31000falseobj-test-zone32024-02-01T10:20:50.780Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:00.095 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:23:00.095 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:23:00.095 DEBUG:boto:path=/vclfcn-17/?prefix=obj-test-zone3 2024-02-01T10:23:00.096 DEBUG:boto:auth_path=/vclfcn-17/?prefix=obj-test-zone3 2024-02-01T10:23:00.096 DEBUG:boto:Method: GET 2024-02-01T10:23:00.096 DEBUG:boto:Path: /vclfcn-17/?prefix=obj-test-zone3 2024-02-01T10:23:00.096 DEBUG:boto:Data: 2024-02-01T10:23:00.096 DEBUG:boto:Headers: {} 2024-02-01T10:23:00.096 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:23:00.097 DEBUG:boto:Port: 8000 2024-02-01T10:23:00.097 DEBUG:boto:Params: {} 2024-02-01T10:23:00.097 DEBUG:boto:Token: None 2024-02-01T10:23:00.097 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:00 GMT /vclfcn-17/ 2024-02-01T10:23:00.097 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:SN+NL+snQVDoyY0UVvFbtJmlCFI= 2024-02-01T10:23:00.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:SN+NL+snQVDoyY0UVvFbtJmlCFI=', 'Content-Length': '0'} 2024-02-01T10:23:00.101 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008feab470342fd789-0065bb7104-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:00.148 DEBUG:boto:b'vclfcn-17obj-test-zone31000falseobj-test-zone32024-02-01T10:20:50.780Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:00.149 DEBUG:boto:path=/vclfcn-17/obj-test-zone3 2024-02-01T10:23:00.149 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3 2024-02-01T10:23:00.149 DEBUG:boto:Method: HEAD 2024-02-01T10:23:00.150 DEBUG:boto:Path: /vclfcn-17/obj-test-zone3 2024-02-01T10:23:00.150 DEBUG:boto:Data: 2024-02-01T10:23:00.150 DEBUG:boto:Headers: {} 2024-02-01T10:23:00.150 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:00.150 DEBUG:boto:Port: 8001 2024-02-01T10:23:00.151 DEBUG:boto:Params: {} 2024-02-01T10:23:00.151 DEBUG:boto:Token: None 2024-02-01T10:23:00.151 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:23:00 GMT /vclfcn-17/obj-test-zone3 2024-02-01T10:23:00.152 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:6l29T52atnK1+ckd2KCrTVUGB+M= 2024-02-01T10:23:00.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:6l29T52atnK1+ckd2KCrTVUGB+M=', 'Content-Length': '0'} 2024-02-01T10:23:00.156 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'YE736OwOOXbZJHJe7d8YvcNV1LxwcQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fe10e799176a559d-0065bb7104-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:00.157 DEBUG:boto:path=/vclfcn-17/obj-test-zone3 2024-02-01T10:23:00.157 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3 2024-02-01T10:23:00.157 DEBUG:boto:Method: HEAD 2024-02-01T10:23:00.157 DEBUG:boto:Path: /vclfcn-17/obj-test-zone3 2024-02-01T10:23:00.157 DEBUG:boto:Data: 2024-02-01T10:23:00.158 DEBUG:boto:Headers: {} 2024-02-01T10:23:00.158 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:23:00.158 DEBUG:boto:Port: 8000 2024-02-01T10:23:00.158 DEBUG:boto:Params: {} 2024-02-01T10:23:00.158 DEBUG:boto:Token: None 2024-02-01T10:23:00.159 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:23:00 GMT /vclfcn-17/obj-test-zone3 2024-02-01T10:23:00.159 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:6l29T52atnK1+ckd2KCrTVUGB+M= 2024-02-01T10:23:00.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:6l29T52atnK1+ckd2KCrTVUGB+M=', 'Content-Length': '0'} 2024-02-01T10:23:00.163 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'YE736OwOOXbZJHJe7d8YvcNV1LxwcQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c06cae873cbcdde3-0065bb7104-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:00.163 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-02-01T10:23:00.163 DEBUG:boto:path=/vclfcn-17/obj-test-zone3 2024-02-01T10:23:00.163 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3 2024-02-01T10:23:00.164 DEBUG:boto:path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:23:00.164 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:23:00.164 DEBUG:boto:Method: GET 2024-02-01T10:23:00.164 DEBUG:boto:Path: /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:23:00.164 DEBUG:boto:Data: 2024-02-01T10:23:00.164 DEBUG:boto:Headers: {} 2024-02-01T10:23:00.164 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:00.165 DEBUG:boto:Port: 8001 2024-02-01T10:23:00.165 DEBUG:boto:Params: {} 2024-02-01T10:23:00.165 DEBUG:boto:Token: None 2024-02-01T10:23:00.165 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:00 GMT /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:23:00.165 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Vo5ocW0YHxLHHPfik5rf5U6gO6I= 2024-02-01T10:23:00.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Vo5ocW0YHxLHHPfik5rf5U6gO6I=', 'Content-Length': '0'} 2024-02-01T10:23:00.168 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'YE736OwOOXbZJHJe7d8YvcNV1LxwcQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000db4bcdd62e41b4ef-0065bb7104-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:00.212 DEBUG:boto:path=/vclfcn-17/obj-test-zone3 2024-02-01T10:23:00.212 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3 2024-02-01T10:23:00.213 DEBUG:boto:path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:23:00.213 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:23:00.213 DEBUG:boto:Method: GET 2024-02-01T10:23:00.213 DEBUG:boto:Path: /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:23:00.213 DEBUG:boto:Data: 2024-02-01T10:23:00.213 DEBUG:boto:Headers: {} 2024-02-01T10:23:00.213 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:23:00.213 DEBUG:boto:Port: 8000 2024-02-01T10:23:00.213 DEBUG:boto:Params: {} 2024-02-01T10:23:00.214 DEBUG:boto:Token: None 2024-02-01T10:23:00.214 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:00 GMT /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:23:00.214 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Vo5ocW0YHxLHHPfik5rf5U6gO6I= 2024-02-01T10:23:00.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Vo5ocW0YHxLHHPfik5rf5U6gO6I=', 'Content-Length': '0'} 2024-02-01T10:23:00.217 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'YE736OwOOXbZJHJe7d8YvcNV1LxwcQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000167ee29c6ea3ef76-0065bb7104-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:00.260 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:23:00.260 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:23:00.260 DEBUG:boto:path=/vclfcn-17/?prefix=obj-test-zone3 2024-02-01T10:23:00.260 DEBUG:boto:auth_path=/vclfcn-17/?prefix=obj-test-zone3 2024-02-01T10:23:00.261 DEBUG:boto:Method: GET 2024-02-01T10:23:00.261 DEBUG:boto:Path: /vclfcn-17/?prefix=obj-test-zone3 2024-02-01T10:23:00.261 DEBUG:boto:Data: 2024-02-01T10:23:00.261 DEBUG:boto:Headers: {} 2024-02-01T10:23:00.261 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:00.261 DEBUG:boto:Port: 8001 2024-02-01T10:23:00.261 DEBUG:boto:Params: {} 2024-02-01T10:23:00.261 DEBUG:boto:Token: None 2024-02-01T10:23:00.261 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:00 GMT /vclfcn-17/ 2024-02-01T10:23:00.262 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:SN+NL+snQVDoyY0UVvFbtJmlCFI= 2024-02-01T10:23:00.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:SN+NL+snQVDoyY0UVvFbtJmlCFI=', 'Content-Length': '0'} 2024-02-01T10:23:00.266 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7be37085327b98f-0065bb7104-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:00.266 DEBUG:boto:b'vclfcn-17obj-test-zone31000falseobj-test-zone32024-02-01T10:20:50.780Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:00.267 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:23:00.268 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:23:00.268 DEBUG:boto:path=/vclfcn-17/?prefix=obj-test-zone3 2024-02-01T10:23:00.268 DEBUG:boto:auth_path=/vclfcn-17/?prefix=obj-test-zone3 2024-02-01T10:23:00.268 DEBUG:boto:Method: GET 2024-02-01T10:23:00.268 DEBUG:boto:Path: /vclfcn-17/?prefix=obj-test-zone3 2024-02-01T10:23:00.268 DEBUG:boto:Data: 2024-02-01T10:23:00.268 DEBUG:boto:Headers: {} 2024-02-01T10:23:00.269 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:23:00.269 DEBUG:boto:Port: 8000 2024-02-01T10:23:00.269 DEBUG:boto:Params: {} 2024-02-01T10:23:00.269 DEBUG:boto:Token: None 2024-02-01T10:23:00.269 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:00 GMT /vclfcn-17/ 2024-02-01T10:23:00.269 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:SN+NL+snQVDoyY0UVvFbtJmlCFI= 2024-02-01T10:23:00.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:SN+NL+snQVDoyY0UVvFbtJmlCFI=', 'Content-Length': '0'} 2024-02-01T10:23:00.274 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe6edfdec89faa94-0065bb7104-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:00.316 DEBUG:boto:b'vclfcn-17obj-test-zone31000falseobj-test-zone32024-02-01T10:20:50.780Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:00.317 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-17 zones={test-zone3, test-zone1} 2024-02-01T10:23:00.317 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-17 zones={test-zone3, test-zone2} 2024-02-01T10:23:00.317 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:23:00.317 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:23:00.317 DEBUG:boto:Method: HEAD 2024-02-01T10:23:00.317 DEBUG:boto:Path: /vclfcn-17/ 2024-02-01T10:23:00.317 DEBUG:boto:Data: 2024-02-01T10:23:00.317 DEBUG:boto:Headers: {} 2024-02-01T10:23:00.317 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:00.317 DEBUG:boto:Port: 8001 2024-02-01T10:23:00.318 DEBUG:boto:Params: {} 2024-02-01T10:23:00.318 DEBUG:boto:Token: None 2024-02-01T10:23:00.318 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:23:00 GMT /vclfcn-17/ 2024-02-01T10:23:00.318 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NohDPtgS/XBQEBfh9Yif6Y65Y4Q= 2024-02-01T10:23:00.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NohDPtgS/XBQEBfh9Yif6Y65Y4Q=', 'Content-Length': '0'} 2024-02-01T10:23:00.322 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cba67a758ecce7d8-0065bb7104-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:23:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:00.323 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:23:00.323 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:23:00.323 DEBUG:boto:Method: HEAD 2024-02-01T10:23:00.323 DEBUG:boto:Path: /vclfcn-17/ 2024-02-01T10:23:00.323 DEBUG:boto:Data: 2024-02-01T10:23:00.323 DEBUG:boto:Headers: {} 2024-02-01T10:23:00.323 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:00.323 DEBUG:boto:Port: 8000 2024-02-01T10:23:00.323 DEBUG:boto:Params: {} 2024-02-01T10:23:00.324 DEBUG:boto:Token: None 2024-02-01T10:23:00.324 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:23:00 GMT /vclfcn-17/ 2024-02-01T10:23:00.324 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NohDPtgS/XBQEBfh9Yif6Y65Y4Q= 2024-02-01T10:23:00.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NohDPtgS/XBQEBfh9Yif6Y65Y4Q=', 'Content-Length': '0'} 2024-02-01T10:23:00.328 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000071b8f450047d4c04-0065bb7104-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:23:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:00.328 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:23:00.328 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:23:00.329 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:23:00.329 DEBUG:boto:path=/vclfcn-17/?versions&max-keys=999 2024-02-01T10:23:00.329 DEBUG:boto:auth_path=/vclfcn-17/?versions&max-keys=999 2024-02-01T10:23:00.329 DEBUG:boto:Method: GET 2024-02-01T10:23:00.329 DEBUG:boto:Path: /vclfcn-17/?versions&max-keys=999 2024-02-01T10:23:00.329 DEBUG:boto:Data: 2024-02-01T10:23:00.329 DEBUG:boto:Headers: {} 2024-02-01T10:23:00.329 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:00.329 DEBUG:boto:Port: 8001 2024-02-01T10:23:00.329 DEBUG:boto:Params: {} 2024-02-01T10:23:00.330 DEBUG:boto:Token: None 2024-02-01T10:23:00.330 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:00 GMT /vclfcn-17/?versions 2024-02-01T10:23:00.330 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NBJahxfNAjS3g4bUruseCki0uEw= 2024-02-01T10:23:00.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NBJahxfNAjS3g4bUruseCki0uEw=', 'Content-Length': '0'} 2024-02-01T10:23:00.334 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f438a88996a8852-0065bb7104-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:00.335 DEBUG:boto:b'vclfcn-17999falsedummyhQXZXSD45t2CSRVMJOLMBXih54iMPmJtrue2024-02-01T10:20:14.297Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1xMxAGR4A-xEILepnzFPRgrWNTOLvitztrue2024-02-01T10:20:50.473Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1true2024-02-01T10:20:50.624Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3YE736OwOOXbZJHJe7d8YvcNV1LxwcQbtrue2024-02-01T10:20:50.780Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:00.335 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-02-01T10:23:00.335 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-02-01T10:23:00.336 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-02-01T10:23:00.336 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-02-01T10:23:00.336 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:23:00.336 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:23:00.336 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:23:00.336 DEBUG:boto:path=/vclfcn-17/?versions&max-keys=999 2024-02-01T10:23:00.336 DEBUG:boto:auth_path=/vclfcn-17/?versions&max-keys=999 2024-02-01T10:23:00.336 DEBUG:boto:Method: GET 2024-02-01T10:23:00.336 DEBUG:boto:Path: /vclfcn-17/?versions&max-keys=999 2024-02-01T10:23:00.336 DEBUG:boto:Data: 2024-02-01T10:23:00.337 DEBUG:boto:Headers: {} 2024-02-01T10:23:00.337 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:00.337 DEBUG:boto:Port: 8000 2024-02-01T10:23:00.337 DEBUG:boto:Params: {} 2024-02-01T10:23:00.337 DEBUG:boto:Token: None 2024-02-01T10:23:00.337 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:00 GMT /vclfcn-17/?versions 2024-02-01T10:23:00.337 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NBJahxfNAjS3g4bUruseCki0uEw= 2024-02-01T10:23:00.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NBJahxfNAjS3g4bUruseCki0uEw=', 'Content-Length': '0'} 2024-02-01T10:23:00.342 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec9a08b036fc57b3-0065bb7104-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:00.343 DEBUG:boto:b'vclfcn-17999falsedummyhQXZXSD45t2CSRVMJOLMBXih54iMPmJtrue2024-02-01T10:20:14.297Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1xMxAGR4A-xEILepnzFPRgrWNTOLvitztrue2024-02-01T10:20:50.473Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1true2024-02-01T10:20:50.624Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3YE736OwOOXbZJHJe7d8YvcNV1LxwcQbtrue2024-02-01T10:20:50.780Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:00.343 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-02-01T10:23:00.343 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-02-01T10:23:00.343 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-02-01T10:23:00.343 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-02-01T10:23:00.344 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:23:00.344 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:23:00.344 DEBUG:boto:path=/vclfcn-17/?versions&max-keys=999 2024-02-01T10:23:00.344 DEBUG:boto:auth_path=/vclfcn-17/?versions&max-keys=999 2024-02-01T10:23:00.344 DEBUG:boto:Method: GET 2024-02-01T10:23:00.344 DEBUG:boto:Path: /vclfcn-17/?versions&max-keys=999 2024-02-01T10:23:00.344 DEBUG:boto:Data: 2024-02-01T10:23:00.344 DEBUG:boto:Headers: {} 2024-02-01T10:23:00.344 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:00.344 DEBUG:boto:Port: 8001 2024-02-01T10:23:00.345 DEBUG:boto:Params: {} 2024-02-01T10:23:00.345 DEBUG:boto:Token: None 2024-02-01T10:23:00.345 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:00 GMT /vclfcn-17/?versions 2024-02-01T10:23:00.345 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NBJahxfNAjS3g4bUruseCki0uEw= 2024-02-01T10:23:00.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NBJahxfNAjS3g4bUruseCki0uEw=', 'Content-Length': '0'} 2024-02-01T10:23:00.349 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028d4dbbeb9faadbe-0065bb7104-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:00.350 DEBUG:boto:b'vclfcn-17999falsedummyhQXZXSD45t2CSRVMJOLMBXih54iMPmJtrue2024-02-01T10:20:14.297Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1xMxAGR4A-xEILepnzFPRgrWNTOLvitztrue2024-02-01T10:20:50.473Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1true2024-02-01T10:20:50.624Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3YE736OwOOXbZJHJe7d8YvcNV1LxwcQbtrue2024-02-01T10:20:50.780Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:00.351 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:23:00.351 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:23:00.351 DEBUG:boto:path=/vclfcn-17/?versions&max-keys=999 2024-02-01T10:23:00.351 DEBUG:boto:auth_path=/vclfcn-17/?versions&max-keys=999 2024-02-01T10:23:00.351 DEBUG:boto:Method: GET 2024-02-01T10:23:00.351 DEBUG:boto:Path: /vclfcn-17/?versions&max-keys=999 2024-02-01T10:23:00.352 DEBUG:boto:Data: 2024-02-01T10:23:00.352 DEBUG:boto:Headers: {} 2024-02-01T10:23:00.352 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:00.352 DEBUG:boto:Port: 8000 2024-02-01T10:23:00.352 DEBUG:boto:Params: {} 2024-02-01T10:23:00.352 DEBUG:boto:Token: None 2024-02-01T10:23:00.352 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:00 GMT /vclfcn-17/?versions 2024-02-01T10:23:00.352 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NBJahxfNAjS3g4bUruseCki0uEw= 2024-02-01T10:23:00.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NBJahxfNAjS3g4bUruseCki0uEw=', 'Content-Length': '0'} 2024-02-01T10:23:00.357 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c45f776d9dcc23b-0065bb7104-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:00.357 DEBUG:boto:b'vclfcn-17999falsedummyhQXZXSD45t2CSRVMJOLMBXih54iMPmJtrue2024-02-01T10:20:14.297Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1xMxAGR4A-xEILepnzFPRgrWNTOLvitztrue2024-02-01T10:20:50.473Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1true2024-02-01T10:20:50.624Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3YE736OwOOXbZJHJe7d8YvcNV1LxwcQbtrue2024-02-01T10:20:50.780Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:00.358 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-02-01T10:23:00.358 DEBUG:boto:path=/vclfcn-17/dummy 2024-02-01T10:23:00.358 DEBUG:boto:auth_path=/vclfcn-17/dummy 2024-02-01T10:23:00.358 DEBUG:boto:path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:23:00.358 DEBUG:boto:auth_path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:23:00.358 DEBUG:boto:Method: GET 2024-02-01T10:23:00.358 DEBUG:boto:Path: /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:23:00.358 DEBUG:boto:Data: 2024-02-01T10:23:00.358 DEBUG:boto:Headers: {} 2024-02-01T10:23:00.359 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:00.359 DEBUG:boto:Port: 8001 2024-02-01T10:23:00.359 DEBUG:boto:Params: {} 2024-02-01T10:23:00.359 DEBUG:boto:Token: None 2024-02-01T10:23:00.359 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:00 GMT /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:23:00.359 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RjYHWCPsOm+Nx2/Du3UfoiBd/Us= 2024-02-01T10:23:00.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RjYHWCPsOm+Nx2/Du3UfoiBd/Us=', 'Content-Length': '0'} 2024-02-01T10:23:00.362 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:14 GMT'), ('x-amz-version-id', 'hQXZXSD45t2CSRVMJOLMBXih54iMPmJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c373c5c2bdad0ab1-0065bb7104-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:00.362 DEBUG:boto:path=/vclfcn-17/dummy 2024-02-01T10:23:00.362 DEBUG:boto:auth_path=/vclfcn-17/dummy 2024-02-01T10:23:00.362 DEBUG:boto:path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:23:00.363 DEBUG:boto:auth_path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:23:00.363 DEBUG:boto:Method: GET 2024-02-01T10:23:00.363 DEBUG:boto:Path: /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:23:00.363 DEBUG:boto:Data: 2024-02-01T10:23:00.363 DEBUG:boto:Headers: {} 2024-02-01T10:23:00.363 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:00.363 DEBUG:boto:Port: 8000 2024-02-01T10:23:00.363 DEBUG:boto:Params: {} 2024-02-01T10:23:00.363 DEBUG:boto:Token: None 2024-02-01T10:23:00.363 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:00 GMT /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:23:00.364 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RjYHWCPsOm+Nx2/Du3UfoiBd/Us= 2024-02-01T10:23:00.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RjYHWCPsOm+Nx2/Du3UfoiBd/Us=', 'Content-Length': '0'} 2024-02-01T10:23:00.366 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:14 GMT'), ('x-amz-version-id', 'hQXZXSD45t2CSRVMJOLMBXih54iMPmJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000051cbbdefe71811df-0065bb7104-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:00.367 DEBUG:boto:path=/vclfcn-17/dummy 2024-02-01T10:23:00.367 DEBUG:boto:auth_path=/vclfcn-17/dummy 2024-02-01T10:23:00.367 DEBUG:boto:path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:23:00.367 DEBUG:boto:auth_path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:23:00.367 DEBUG:boto:Method: HEAD 2024-02-01T10:23:00.367 DEBUG:boto:Path: /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:23:00.367 DEBUG:boto:Data: 2024-02-01T10:23:00.367 DEBUG:boto:Headers: {} 2024-02-01T10:23:00.367 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:00.367 DEBUG:boto:Port: 8001 2024-02-01T10:23:00.368 DEBUG:boto:Params: {} 2024-02-01T10:23:00.368 DEBUG:boto:Token: None 2024-02-01T10:23:00.368 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:23:00 GMT /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:23:00.368 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:IQS2qqwYK1li10v3Y8MFGBuJ+CQ= 2024-02-01T10:23:00.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:IQS2qqwYK1li10v3Y8MFGBuJ+CQ=', 'Content-Length': '0'} 2024-02-01T10:23:00.371 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:14 GMT'), ('x-amz-version-id', 'hQXZXSD45t2CSRVMJOLMBXih54iMPmJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000061b4f6b75890a25c-0065bb7104-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:00.371 DEBUG:boto:path=/vclfcn-17/dummy 2024-02-01T10:23:00.371 DEBUG:boto:auth_path=/vclfcn-17/dummy 2024-02-01T10:23:00.371 DEBUG:boto:path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:23:00.371 DEBUG:boto:auth_path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:23:00.371 DEBUG:boto:Method: HEAD 2024-02-01T10:23:00.371 DEBUG:boto:Path: /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:23:00.372 DEBUG:boto:Data: 2024-02-01T10:23:00.372 DEBUG:boto:Headers: {} 2024-02-01T10:23:00.372 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:00.372 DEBUG:boto:Port: 8000 2024-02-01T10:23:00.372 DEBUG:boto:Params: {} 2024-02-01T10:23:00.372 DEBUG:boto:Token: None 2024-02-01T10:23:00.372 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:23:00 GMT /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:23:00.372 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:IQS2qqwYK1li10v3Y8MFGBuJ+CQ= 2024-02-01T10:23:00.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:IQS2qqwYK1li10v3Y8MFGBuJ+CQ=', 'Content-Length': '0'} 2024-02-01T10:23:00.375 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:14 GMT'), ('x-amz-version-id', 'hQXZXSD45t2CSRVMJOLMBXih54iMPmJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000251b13149497aac8-0065bb7104-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:00.375 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-02-01T10:23:00.376 DEBUG:boto:path=/vclfcn-17/dummy 2024-02-01T10:23:00.376 DEBUG:boto:auth_path=/vclfcn-17/dummy 2024-02-01T10:23:00.376 DEBUG:boto:path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:23:00.376 DEBUG:boto:auth_path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:23:00.376 DEBUG:boto:Method: GET 2024-02-01T10:23:00.376 DEBUG:boto:Path: /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:23:00.376 DEBUG:boto:Data: 2024-02-01T10:23:00.376 DEBUG:boto:Headers: {} 2024-02-01T10:23:00.376 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:00.377 DEBUG:boto:Port: 8001 2024-02-01T10:23:00.377 DEBUG:boto:Params: {} 2024-02-01T10:23:00.377 DEBUG:boto:Token: None 2024-02-01T10:23:00.377 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:00 GMT /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:23:00.377 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RjYHWCPsOm+Nx2/Du3UfoiBd/Us= 2024-02-01T10:23:00.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RjYHWCPsOm+Nx2/Du3UfoiBd/Us=', 'Content-Length': '0'} 2024-02-01T10:23:00.380 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:14 GMT'), ('x-amz-version-id', 'hQXZXSD45t2CSRVMJOLMBXih54iMPmJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009d24fae359155cfe-0065bb7104-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:00.380 DEBUG:boto:path=/vclfcn-17/dummy 2024-02-01T10:23:00.381 DEBUG:boto:auth_path=/vclfcn-17/dummy 2024-02-01T10:23:00.381 DEBUG:boto:path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:23:00.381 DEBUG:boto:auth_path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:23:00.381 DEBUG:boto:Method: GET 2024-02-01T10:23:00.381 DEBUG:boto:Path: /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:23:00.381 DEBUG:boto:Data: 2024-02-01T10:23:00.381 DEBUG:boto:Headers: {} 2024-02-01T10:23:00.381 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:00.381 DEBUG:boto:Port: 8000 2024-02-01T10:23:00.381 DEBUG:boto:Params: {} 2024-02-01T10:23:00.382 DEBUG:boto:Token: None 2024-02-01T10:23:00.382 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:00 GMT /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:23:00.382 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RjYHWCPsOm+Nx2/Du3UfoiBd/Us= 2024-02-01T10:23:00.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RjYHWCPsOm+Nx2/Du3UfoiBd/Us=', 'Content-Length': '0'} 2024-02-01T10:23:00.385 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:14 GMT'), ('x-amz-version-id', 'hQXZXSD45t2CSRVMJOLMBXih54iMPmJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002fe8cb83f9701498-0065bb7104-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:00.385 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:23:00.386 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:23:00.386 DEBUG:boto:path=/vclfcn-17/?prefix=dummy 2024-02-01T10:23:00.386 DEBUG:boto:auth_path=/vclfcn-17/?prefix=dummy 2024-02-01T10:23:00.386 DEBUG:boto:Method: GET 2024-02-01T10:23:00.386 DEBUG:boto:Path: /vclfcn-17/?prefix=dummy 2024-02-01T10:23:00.386 DEBUG:boto:Data: 2024-02-01T10:23:00.386 DEBUG:boto:Headers: {} 2024-02-01T10:23:00.386 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:00.386 DEBUG:boto:Port: 8001 2024-02-01T10:23:00.386 DEBUG:boto:Params: {} 2024-02-01T10:23:00.386 DEBUG:boto:Token: None 2024-02-01T10:23:00.387 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:00 GMT /vclfcn-17/ 2024-02-01T10:23:00.387 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:SN+NL+snQVDoyY0UVvFbtJmlCFI= 2024-02-01T10:23:00.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:SN+NL+snQVDoyY0UVvFbtJmlCFI=', 'Content-Length': '0'} 2024-02-01T10:23:00.391 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b815dcaa6645b8b-0065bb7104-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:00.436 DEBUG:boto:b'vclfcn-17dummy1000falsedummy2024-02-01T10:20:14.297Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:00.437 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:23:00.437 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:23:00.437 DEBUG:boto:path=/vclfcn-17/?prefix=dummy 2024-02-01T10:23:00.437 DEBUG:boto:auth_path=/vclfcn-17/?prefix=dummy 2024-02-01T10:23:00.437 DEBUG:boto:Method: GET 2024-02-01T10:23:00.437 DEBUG:boto:Path: /vclfcn-17/?prefix=dummy 2024-02-01T10:23:00.437 DEBUG:boto:Data: 2024-02-01T10:23:00.437 DEBUG:boto:Headers: {} 2024-02-01T10:23:00.437 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:00.437 DEBUG:boto:Port: 8000 2024-02-01T10:23:00.437 DEBUG:boto:Params: {} 2024-02-01T10:23:00.438 DEBUG:boto:Token: None 2024-02-01T10:23:00.438 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:00 GMT /vclfcn-17/ 2024-02-01T10:23:00.438 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:SN+NL+snQVDoyY0UVvFbtJmlCFI= 2024-02-01T10:23:00.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:SN+NL+snQVDoyY0UVvFbtJmlCFI=', 'Content-Length': '0'} 2024-02-01T10:23:00.444 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000093ac7afb72cca237-0065bb7104-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:00.444 DEBUG:boto:b'vclfcn-17dummy1000falsedummy2024-02-01T10:20:14.297Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:00.444 DEBUG:boto:path=/vclfcn-17/dummy 2024-02-01T10:23:00.444 DEBUG:boto:auth_path=/vclfcn-17/dummy 2024-02-01T10:23:00.445 DEBUG:boto:Method: HEAD 2024-02-01T10:23:00.445 DEBUG:boto:Path: /vclfcn-17/dummy 2024-02-01T10:23:00.445 DEBUG:boto:Data: 2024-02-01T10:23:00.445 DEBUG:boto:Headers: {} 2024-02-01T10:23:00.445 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:00.445 DEBUG:boto:Port: 8001 2024-02-01T10:23:00.445 DEBUG:boto:Params: {} 2024-02-01T10:23:00.445 DEBUG:boto:Token: None 2024-02-01T10:23:00.445 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:23:00 GMT /vclfcn-17/dummy 2024-02-01T10:23:00.445 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:afel3ZHJ7FJjghonTijU8VBrsYk= 2024-02-01T10:23:00.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:afel3ZHJ7FJjghonTijU8VBrsYk=', 'Content-Length': '0'} 2024-02-01T10:23:00.449 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:14 GMT'), ('x-amz-version-id', 'hQXZXSD45t2CSRVMJOLMBXih54iMPmJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004182b2ee5158cc96-0065bb7104-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:00.450 DEBUG:boto:path=/vclfcn-17/dummy 2024-02-01T10:23:00.450 DEBUG:boto:auth_path=/vclfcn-17/dummy 2024-02-01T10:23:00.450 DEBUG:boto:Method: HEAD 2024-02-01T10:23:00.450 DEBUG:boto:Path: /vclfcn-17/dummy 2024-02-01T10:23:00.450 DEBUG:boto:Data: 2024-02-01T10:23:00.450 DEBUG:boto:Headers: {} 2024-02-01T10:23:00.450 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:00.450 DEBUG:boto:Port: 8000 2024-02-01T10:23:00.450 DEBUG:boto:Params: {} 2024-02-01T10:23:00.451 DEBUG:boto:Token: None 2024-02-01T10:23:00.451 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:23:00 GMT /vclfcn-17/dummy 2024-02-01T10:23:00.451 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:afel3ZHJ7FJjghonTijU8VBrsYk= 2024-02-01T10:23:00.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:afel3ZHJ7FJjghonTijU8VBrsYk=', 'Content-Length': '0'} 2024-02-01T10:23:00.454 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:14 GMT'), ('x-amz-version-id', 'hQXZXSD45t2CSRVMJOLMBXih54iMPmJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005a88921c0e3ff38d-0065bb7104-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:00.455 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-02-01T10:23:00.455 DEBUG:boto:path=/vclfcn-17/dummy 2024-02-01T10:23:00.455 DEBUG:boto:auth_path=/vclfcn-17/dummy 2024-02-01T10:23:00.455 DEBUG:boto:path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:23:00.455 DEBUG:boto:auth_path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:23:00.455 DEBUG:boto:Method: GET 2024-02-01T10:23:00.455 DEBUG:boto:Path: /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:23:00.455 DEBUG:boto:Data: 2024-02-01T10:23:00.455 DEBUG:boto:Headers: {} 2024-02-01T10:23:00.455 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:00.456 DEBUG:boto:Port: 8001 2024-02-01T10:23:00.456 DEBUG:boto:Params: {} 2024-02-01T10:23:00.456 DEBUG:boto:Token: None 2024-02-01T10:23:00.456 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:00 GMT /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:23:00.456 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RjYHWCPsOm+Nx2/Du3UfoiBd/Us= 2024-02-01T10:23:00.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RjYHWCPsOm+Nx2/Du3UfoiBd/Us=', 'Content-Length': '0'} 2024-02-01T10:23:00.459 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:14 GMT'), ('x-amz-version-id', 'hQXZXSD45t2CSRVMJOLMBXih54iMPmJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000765baf93215c036-0065bb7104-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:00.459 DEBUG:boto:path=/vclfcn-17/dummy 2024-02-01T10:23:00.459 DEBUG:boto:auth_path=/vclfcn-17/dummy 2024-02-01T10:23:00.459 DEBUG:boto:path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:23:00.459 DEBUG:boto:auth_path=/vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:23:00.459 DEBUG:boto:Method: GET 2024-02-01T10:23:00.459 DEBUG:boto:Path: /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:23:00.460 DEBUG:boto:Data: 2024-02-01T10:23:00.460 DEBUG:boto:Headers: {} 2024-02-01T10:23:00.460 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:00.460 DEBUG:boto:Port: 8000 2024-02-01T10:23:00.460 DEBUG:boto:Params: {} 2024-02-01T10:23:00.460 DEBUG:boto:Token: None 2024-02-01T10:23:00.460 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:00 GMT /vclfcn-17/dummy?versionId=hQXZXSD45t2CSRVMJOLMBXih54iMPmJ 2024-02-01T10:23:00.460 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RjYHWCPsOm+Nx2/Du3UfoiBd/Us= 2024-02-01T10:23:00.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RjYHWCPsOm+Nx2/Du3UfoiBd/Us=', 'Content-Length': '0'} 2024-02-01T10:23:00.463 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:14 GMT'), ('x-amz-version-id', 'hQXZXSD45t2CSRVMJOLMBXih54iMPmJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000050305b80eccecf23-0065bb7104-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:00.464 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:23:00.464 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:23:00.464 DEBUG:boto:path=/vclfcn-17/?prefix=dummy 2024-02-01T10:23:00.464 DEBUG:boto:auth_path=/vclfcn-17/?prefix=dummy 2024-02-01T10:23:00.465 DEBUG:boto:Method: GET 2024-02-01T10:23:00.465 DEBUG:boto:Path: /vclfcn-17/?prefix=dummy 2024-02-01T10:23:00.465 DEBUG:boto:Data: 2024-02-01T10:23:00.465 DEBUG:boto:Headers: {} 2024-02-01T10:23:00.465 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:00.465 DEBUG:boto:Port: 8001 2024-02-01T10:23:00.465 DEBUG:boto:Params: {} 2024-02-01T10:23:00.465 DEBUG:boto:Token: None 2024-02-01T10:23:00.465 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:00 GMT /vclfcn-17/ 2024-02-01T10:23:00.466 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:SN+NL+snQVDoyY0UVvFbtJmlCFI= 2024-02-01T10:23:00.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:SN+NL+snQVDoyY0UVvFbtJmlCFI=', 'Content-Length': '0'} 2024-02-01T10:23:00.470 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000069091d7e86bbf60c-0065bb7104-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:00.470 DEBUG:boto:b'vclfcn-17dummy1000falsedummy2024-02-01T10:20:14.297Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:00.471 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:23:00.471 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:23:00.471 DEBUG:boto:path=/vclfcn-17/?prefix=dummy 2024-02-01T10:23:00.471 DEBUG:boto:auth_path=/vclfcn-17/?prefix=dummy 2024-02-01T10:23:00.471 DEBUG:boto:Method: GET 2024-02-01T10:23:00.471 DEBUG:boto:Path: /vclfcn-17/?prefix=dummy 2024-02-01T10:23:00.471 DEBUG:boto:Data: 2024-02-01T10:23:00.472 DEBUG:boto:Headers: {} 2024-02-01T10:23:00.472 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:00.472 DEBUG:boto:Port: 8000 2024-02-01T10:23:00.472 DEBUG:boto:Params: {} 2024-02-01T10:23:00.472 DEBUG:boto:Token: None 2024-02-01T10:23:00.472 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:00 GMT /vclfcn-17/ 2024-02-01T10:23:00.472 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:SN+NL+snQVDoyY0UVvFbtJmlCFI= 2024-02-01T10:23:00.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:SN+NL+snQVDoyY0UVvFbtJmlCFI=', 'Content-Length': '0'} 2024-02-01T10:23:00.477 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ab112a2b3bae446-0065bb7104-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:00.477 DEBUG:boto:b'vclfcn-17dummy1000falsedummy2024-02-01T10:20:14.297Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:00.477 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-02-01T10:23:00.478 DEBUG:boto:path=/vclfcn-17/obj-test-zone1 2024-02-01T10:23:00.478 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1 2024-02-01T10:23:00.478 DEBUG:boto:path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:23:00.478 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:23:00.478 DEBUG:boto:Method: GET 2024-02-01T10:23:00.478 DEBUG:boto:Path: /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:23:00.478 DEBUG:boto:Data: 2024-02-01T10:23:00.478 DEBUG:boto:Headers: {} 2024-02-01T10:23:00.478 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:00.478 DEBUG:boto:Port: 8001 2024-02-01T10:23:00.479 DEBUG:boto:Params: {} 2024-02-01T10:23:00.479 DEBUG:boto:Token: None 2024-02-01T10:23:00.479 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:00 GMT /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:23:00.479 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YFRezz4Vd3ZMxAFO8og/0QbFt2g= 2024-02-01T10:23:00.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YFRezz4Vd3ZMxAFO8og/0QbFt2g=', 'Content-Length': '0'} 2024-02-01T10:23:00.482 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'xMxAGR4A-xEILepnzFPRgrWNTOLvitz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000af1907b46c305878-0065bb7104-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:00.528 DEBUG:boto:path=/vclfcn-17/obj-test-zone1 2024-02-01T10:23:00.528 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1 2024-02-01T10:23:00.528 DEBUG:boto:path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:23:00.529 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:23:00.529 DEBUG:boto:Method: GET 2024-02-01T10:23:00.529 DEBUG:boto:Path: /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:23:00.529 DEBUG:boto:Data: 2024-02-01T10:23:00.529 DEBUG:boto:Headers: {} 2024-02-01T10:23:00.529 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:00.529 DEBUG:boto:Port: 8000 2024-02-01T10:23:00.529 DEBUG:boto:Params: {} 2024-02-01T10:23:00.529 DEBUG:boto:Token: None 2024-02-01T10:23:00.529 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:00 GMT /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:23:00.530 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YFRezz4Vd3ZMxAFO8og/0QbFt2g= 2024-02-01T10:23:00.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YFRezz4Vd3ZMxAFO8og/0QbFt2g=', 'Content-Length': '0'} 2024-02-01T10:23:00.533 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'xMxAGR4A-xEILepnzFPRgrWNTOLvitz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ffe7e773a47d1c29-0065bb7104-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:00.533 DEBUG:boto:path=/vclfcn-17/obj-test-zone1 2024-02-01T10:23:00.533 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1 2024-02-01T10:23:00.533 DEBUG:boto:path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:23:00.533 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:23:00.533 DEBUG:boto:Method: HEAD 2024-02-01T10:23:00.533 DEBUG:boto:Path: /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:23:00.533 DEBUG:boto:Data: 2024-02-01T10:23:00.534 DEBUG:boto:Headers: {} 2024-02-01T10:23:00.534 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:00.534 DEBUG:boto:Port: 8001 2024-02-01T10:23:00.534 DEBUG:boto:Params: {} 2024-02-01T10:23:00.534 DEBUG:boto:Token: None 2024-02-01T10:23:00.534 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:23:00 GMT /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:23:00.534 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wvFxGJQ9/leY8RHH5dXPv+xFQkE= 2024-02-01T10:23:00.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wvFxGJQ9/leY8RHH5dXPv+xFQkE=', 'Content-Length': '0'} 2024-02-01T10:23:00.537 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'xMxAGR4A-xEILepnzFPRgrWNTOLvitz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dc2ee08f2279ca52-0065bb7104-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:00.537 DEBUG:boto:path=/vclfcn-17/obj-test-zone1 2024-02-01T10:23:00.537 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1 2024-02-01T10:23:00.537 DEBUG:boto:path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:23:00.537 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:23:00.537 DEBUG:boto:Method: HEAD 2024-02-01T10:23:00.537 DEBUG:boto:Path: /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:23:00.538 DEBUG:boto:Data: 2024-02-01T10:23:00.538 DEBUG:boto:Headers: {} 2024-02-01T10:23:00.538 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:00.538 DEBUG:boto:Port: 8000 2024-02-01T10:23:00.538 DEBUG:boto:Params: {} 2024-02-01T10:23:00.538 DEBUG:boto:Token: None 2024-02-01T10:23:00.538 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:23:00 GMT /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:23:00.538 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wvFxGJQ9/leY8RHH5dXPv+xFQkE= 2024-02-01T10:23:00.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wvFxGJQ9/leY8RHH5dXPv+xFQkE=', 'Content-Length': '0'} 2024-02-01T10:23:00.541 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'xMxAGR4A-xEILepnzFPRgrWNTOLvitz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005015a695ba820074-0065bb7104-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:00.541 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-02-01T10:23:00.541 DEBUG:boto:path=/vclfcn-17/obj-test-zone1 2024-02-01T10:23:00.542 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1 2024-02-01T10:23:00.542 DEBUG:boto:path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:23:00.542 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:23:00.542 DEBUG:boto:Method: GET 2024-02-01T10:23:00.542 DEBUG:boto:Path: /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:23:00.542 DEBUG:boto:Data: 2024-02-01T10:23:00.542 DEBUG:boto:Headers: {} 2024-02-01T10:23:00.542 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:00.542 DEBUG:boto:Port: 8001 2024-02-01T10:23:00.542 DEBUG:boto:Params: {} 2024-02-01T10:23:00.542 DEBUG:boto:Token: None 2024-02-01T10:23:00.543 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:00 GMT /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:23:00.543 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YFRezz4Vd3ZMxAFO8og/0QbFt2g= 2024-02-01T10:23:00.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YFRezz4Vd3ZMxAFO8og/0QbFt2g=', 'Content-Length': '0'} 2024-02-01T10:23:00.546 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'xMxAGR4A-xEILepnzFPRgrWNTOLvitz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005d2c1d5d81c8bbb1-0065bb7104-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:00.588 DEBUG:boto:path=/vclfcn-17/obj-test-zone1 2024-02-01T10:23:00.588 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1 2024-02-01T10:23:00.588 DEBUG:boto:path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:23:00.589 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:23:00.589 DEBUG:boto:Method: GET 2024-02-01T10:23:00.589 DEBUG:boto:Path: /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:23:00.589 DEBUG:boto:Data: 2024-02-01T10:23:00.589 DEBUG:boto:Headers: {} 2024-02-01T10:23:00.589 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:00.589 DEBUG:boto:Port: 8000 2024-02-01T10:23:00.589 DEBUG:boto:Params: {} 2024-02-01T10:23:00.589 DEBUG:boto:Token: None 2024-02-01T10:23:00.589 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:00 GMT /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:23:00.590 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YFRezz4Vd3ZMxAFO8og/0QbFt2g= 2024-02-01T10:23:00.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YFRezz4Vd3ZMxAFO8og/0QbFt2g=', 'Content-Length': '0'} 2024-02-01T10:23:00.592 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'xMxAGR4A-xEILepnzFPRgrWNTOLvitz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b58c389c3b833a26-0065bb7104-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:00.640 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:23:00.640 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:23:00.641 DEBUG:boto:path=/vclfcn-17/?prefix=obj-test-zone1 2024-02-01T10:23:00.641 DEBUG:boto:auth_path=/vclfcn-17/?prefix=obj-test-zone1 2024-02-01T10:23:00.641 DEBUG:boto:Method: GET 2024-02-01T10:23:00.641 DEBUG:boto:Path: /vclfcn-17/?prefix=obj-test-zone1 2024-02-01T10:23:00.641 DEBUG:boto:Data: 2024-02-01T10:23:00.641 DEBUG:boto:Headers: {} 2024-02-01T10:23:00.641 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:00.641 DEBUG:boto:Port: 8001 2024-02-01T10:23:00.641 DEBUG:boto:Params: {} 2024-02-01T10:23:00.642 DEBUG:boto:Token: None 2024-02-01T10:23:00.642 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:00 GMT /vclfcn-17/ 2024-02-01T10:23:00.642 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:SN+NL+snQVDoyY0UVvFbtJmlCFI= 2024-02-01T10:23:00.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:SN+NL+snQVDoyY0UVvFbtJmlCFI=', 'Content-Length': '0'} 2024-02-01T10:23:00.646 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099cfb1363dba8691-0065bb7104-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:00.688 DEBUG:boto:b'vclfcn-17obj-test-zone11000falseobj-test-zone12024-02-01T10:20:50.473Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:00.689 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:23:00.689 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:23:00.689 DEBUG:boto:path=/vclfcn-17/?prefix=obj-test-zone1 2024-02-01T10:23:00.689 DEBUG:boto:auth_path=/vclfcn-17/?prefix=obj-test-zone1 2024-02-01T10:23:00.689 DEBUG:boto:Method: GET 2024-02-01T10:23:00.689 DEBUG:boto:Path: /vclfcn-17/?prefix=obj-test-zone1 2024-02-01T10:23:00.689 DEBUG:boto:Data: 2024-02-01T10:23:00.689 DEBUG:boto:Headers: {} 2024-02-01T10:23:00.689 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:00.689 DEBUG:boto:Port: 8000 2024-02-01T10:23:00.690 DEBUG:boto:Params: {} 2024-02-01T10:23:00.690 DEBUG:boto:Token: None 2024-02-01T10:23:00.690 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:00 GMT /vclfcn-17/ 2024-02-01T10:23:00.690 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:SN+NL+snQVDoyY0UVvFbtJmlCFI= 2024-02-01T10:23:00.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:SN+NL+snQVDoyY0UVvFbtJmlCFI=', 'Content-Length': '0'} 2024-02-01T10:23:00.695 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000098f5fdc597f9d407-0065bb7104-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:00.695 DEBUG:boto:b'vclfcn-17obj-test-zone11000falseobj-test-zone12024-02-01T10:20:50.473Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:00.695 DEBUG:boto:path=/vclfcn-17/obj-test-zone1 2024-02-01T10:23:00.695 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1 2024-02-01T10:23:00.695 DEBUG:boto:Method: HEAD 2024-02-01T10:23:00.696 DEBUG:boto:Path: /vclfcn-17/obj-test-zone1 2024-02-01T10:23:00.696 DEBUG:boto:Data: 2024-02-01T10:23:00.696 DEBUG:boto:Headers: {} 2024-02-01T10:23:00.696 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:00.696 DEBUG:boto:Port: 8001 2024-02-01T10:23:00.696 DEBUG:boto:Params: {} 2024-02-01T10:23:00.696 DEBUG:boto:Token: None 2024-02-01T10:23:00.696 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:23:00 GMT /vclfcn-17/obj-test-zone1 2024-02-01T10:23:00.696 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BUfG4vGs7S/EdniIQilRC8wk+do= 2024-02-01T10:23:00.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BUfG4vGs7S/EdniIQilRC8wk+do=', 'Content-Length': '0'} 2024-02-01T10:23:00.700 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'xMxAGR4A-xEILepnzFPRgrWNTOLvitz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000577cf5463ba35a37-0065bb7104-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:00.701 DEBUG:boto:path=/vclfcn-17/obj-test-zone1 2024-02-01T10:23:00.701 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1 2024-02-01T10:23:00.701 DEBUG:boto:Method: HEAD 2024-02-01T10:23:00.701 DEBUG:boto:Path: /vclfcn-17/obj-test-zone1 2024-02-01T10:23:00.701 DEBUG:boto:Data: 2024-02-01T10:23:00.701 DEBUG:boto:Headers: {} 2024-02-01T10:23:00.701 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:00.701 DEBUG:boto:Port: 8000 2024-02-01T10:23:00.701 DEBUG:boto:Params: {} 2024-02-01T10:23:00.702 DEBUG:boto:Token: None 2024-02-01T10:23:00.702 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:23:00 GMT /vclfcn-17/obj-test-zone1 2024-02-01T10:23:00.702 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BUfG4vGs7S/EdniIQilRC8wk+do= 2024-02-01T10:23:00.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BUfG4vGs7S/EdniIQilRC8wk+do=', 'Content-Length': '0'} 2024-02-01T10:23:00.705 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'xMxAGR4A-xEILepnzFPRgrWNTOLvitz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ca77d23fd73ce6ab-0065bb7104-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:00.706 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-02-01T10:23:00.706 DEBUG:boto:path=/vclfcn-17/obj-test-zone1 2024-02-01T10:23:00.706 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1 2024-02-01T10:23:00.706 DEBUG:boto:path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:23:00.706 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:23:00.706 DEBUG:boto:Method: GET 2024-02-01T10:23:00.706 DEBUG:boto:Path: /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:23:00.706 DEBUG:boto:Data: 2024-02-01T10:23:00.707 DEBUG:boto:Headers: {} 2024-02-01T10:23:00.707 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:00.707 DEBUG:boto:Port: 8001 2024-02-01T10:23:00.707 DEBUG:boto:Params: {} 2024-02-01T10:23:00.707 DEBUG:boto:Token: None 2024-02-01T10:23:00.707 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:00 GMT /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:23:00.707 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YFRezz4Vd3ZMxAFO8og/0QbFt2g= 2024-02-01T10:23:00.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YFRezz4Vd3ZMxAFO8og/0QbFt2g=', 'Content-Length': '0'} 2024-02-01T10:23:00.710 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'xMxAGR4A-xEILepnzFPRgrWNTOLvitz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c5465c5e879e69a8-0065bb7104-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:00.752 DEBUG:boto:path=/vclfcn-17/obj-test-zone1 2024-02-01T10:23:00.752 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1 2024-02-01T10:23:00.752 DEBUG:boto:path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:23:00.752 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:23:00.752 DEBUG:boto:Method: GET 2024-02-01T10:23:00.753 DEBUG:boto:Path: /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:23:00.753 DEBUG:boto:Data: 2024-02-01T10:23:00.753 DEBUG:boto:Headers: {} 2024-02-01T10:23:00.753 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:00.753 DEBUG:boto:Port: 8000 2024-02-01T10:23:00.753 DEBUG:boto:Params: {} 2024-02-01T10:23:00.753 DEBUG:boto:Token: None 2024-02-01T10:23:00.753 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:00 GMT /vclfcn-17/obj-test-zone1?versionId=xMxAGR4A-xEILepnzFPRgrWNTOLvitz 2024-02-01T10:23:00.753 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YFRezz4Vd3ZMxAFO8og/0QbFt2g= 2024-02-01T10:23:00.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YFRezz4Vd3ZMxAFO8og/0QbFt2g=', 'Content-Length': '0'} 2024-02-01T10:23:00.756 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'xMxAGR4A-xEILepnzFPRgrWNTOLvitz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c40f42327b4c53b1-0065bb7104-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:00.800 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:23:00.801 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:23:00.802 DEBUG:boto:path=/vclfcn-17/?prefix=obj-test-zone1 2024-02-01T10:23:00.802 DEBUG:boto:auth_path=/vclfcn-17/?prefix=obj-test-zone1 2024-02-01T10:23:00.802 DEBUG:boto:Method: GET 2024-02-01T10:23:00.802 DEBUG:boto:Path: /vclfcn-17/?prefix=obj-test-zone1 2024-02-01T10:23:00.802 DEBUG:boto:Data: 2024-02-01T10:23:00.802 DEBUG:boto:Headers: {} 2024-02-01T10:23:00.802 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:00.802 DEBUG:boto:Port: 8001 2024-02-01T10:23:00.802 DEBUG:boto:Params: {} 2024-02-01T10:23:00.802 DEBUG:boto:Token: None 2024-02-01T10:23:00.803 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:00 GMT /vclfcn-17/ 2024-02-01T10:23:00.803 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:SN+NL+snQVDoyY0UVvFbtJmlCFI= 2024-02-01T10:23:00.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:SN+NL+snQVDoyY0UVvFbtJmlCFI=', 'Content-Length': '0'} 2024-02-01T10:23:00.808 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a3a260e0648555c8-0065bb7104-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:00.856 DEBUG:boto:b'vclfcn-17obj-test-zone11000falseobj-test-zone12024-02-01T10:20:50.473Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:00.857 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:23:00.857 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:23:00.857 DEBUG:boto:path=/vclfcn-17/?prefix=obj-test-zone1 2024-02-01T10:23:00.857 DEBUG:boto:auth_path=/vclfcn-17/?prefix=obj-test-zone1 2024-02-01T10:23:00.857 DEBUG:boto:Method: GET 2024-02-01T10:23:00.857 DEBUG:boto:Path: /vclfcn-17/?prefix=obj-test-zone1 2024-02-01T10:23:00.857 DEBUG:boto:Data: 2024-02-01T10:23:00.857 DEBUG:boto:Headers: {} 2024-02-01T10:23:00.857 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:00.857 DEBUG:boto:Port: 8000 2024-02-01T10:23:00.857 DEBUG:boto:Params: {} 2024-02-01T10:23:00.858 DEBUG:boto:Token: None 2024-02-01T10:23:00.858 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:00 GMT /vclfcn-17/ 2024-02-01T10:23:00.858 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:SN+NL+snQVDoyY0UVvFbtJmlCFI= 2024-02-01T10:23:00.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:SN+NL+snQVDoyY0UVvFbtJmlCFI=', 'Content-Length': '0'} 2024-02-01T10:23:00.863 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028b50e04ea83497f-0065bb7104-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:00.863 DEBUG:boto:b'vclfcn-17obj-test-zone11000falseobj-test-zone12024-02-01T10:20:50.473Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:00.863 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-02-01T10:23:00.863 DEBUG:boto:path=/vclfcn-17/obj-test-zone2 2024-02-01T10:23:00.863 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2 2024-02-01T10:23:00.864 DEBUG:boto:path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:23:00.864 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:23:00.864 DEBUG:boto:Method: GET 2024-02-01T10:23:00.864 DEBUG:boto:Path: /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:23:00.864 DEBUG:boto:Data: 2024-02-01T10:23:00.864 DEBUG:boto:Headers: {} 2024-02-01T10:23:00.864 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:00.864 DEBUG:boto:Port: 8001 2024-02-01T10:23:00.864 DEBUG:boto:Params: {} 2024-02-01T10:23:00.864 DEBUG:boto:Token: None 2024-02-01T10:23:00.865 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:00 GMT /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:23:00.865 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:xYzLlr6RuBcUqiV0QKJYgkQMdRk= 2024-02-01T10:23:00.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:xYzLlr6RuBcUqiV0QKJYgkQMdRk=', 'Content-Length': '0'} 2024-02-01T10:23:00.868 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', '-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d4262358571feb2e-0065bb7104-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:00.908 DEBUG:boto:path=/vclfcn-17/obj-test-zone2 2024-02-01T10:23:00.908 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2 2024-02-01T10:23:00.908 DEBUG:boto:path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:23:00.908 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:23:00.908 DEBUG:boto:Method: GET 2024-02-01T10:23:00.909 DEBUG:boto:Path: /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:23:00.909 DEBUG:boto:Data: 2024-02-01T10:23:00.909 DEBUG:boto:Headers: {} 2024-02-01T10:23:00.909 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:00.909 DEBUG:boto:Port: 8000 2024-02-01T10:23:00.909 DEBUG:boto:Params: {} 2024-02-01T10:23:00.909 DEBUG:boto:Token: None 2024-02-01T10:23:00.909 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:00 GMT /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:23:00.909 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:xYzLlr6RuBcUqiV0QKJYgkQMdRk= 2024-02-01T10:23:00.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:xYzLlr6RuBcUqiV0QKJYgkQMdRk=', 'Content-Length': '0'} 2024-02-01T10:23:00.912 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', '-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c6f071540b821ad4-0065bb7104-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:00.913 DEBUG:boto:path=/vclfcn-17/obj-test-zone2 2024-02-01T10:23:00.913 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2 2024-02-01T10:23:00.913 DEBUG:boto:path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:23:00.913 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:23:00.913 DEBUG:boto:Method: HEAD 2024-02-01T10:23:00.913 DEBUG:boto:Path: /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:23:00.913 DEBUG:boto:Data: 2024-02-01T10:23:00.913 DEBUG:boto:Headers: {} 2024-02-01T10:23:00.913 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:00.913 DEBUG:boto:Port: 8001 2024-02-01T10:23:00.914 DEBUG:boto:Params: {} 2024-02-01T10:23:00.914 DEBUG:boto:Token: None 2024-02-01T10:23:00.914 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:23:00 GMT /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:23:00.914 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:h0iphQ8mQBGQ4B6XDsSjVOL56YM= 2024-02-01T10:23:00.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:h0iphQ8mQBGQ4B6XDsSjVOL56YM=', 'Content-Length': '0'} 2024-02-01T10:23:00.917 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', '-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009ed4b65fc2d24636-0065bb7104-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:00.917 DEBUG:boto:path=/vclfcn-17/obj-test-zone2 2024-02-01T10:23:00.917 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2 2024-02-01T10:23:00.917 DEBUG:boto:path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:23:00.917 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:23:00.917 DEBUG:boto:Method: HEAD 2024-02-01T10:23:00.917 DEBUG:boto:Path: /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:23:00.917 DEBUG:boto:Data: 2024-02-01T10:23:00.918 DEBUG:boto:Headers: {} 2024-02-01T10:23:00.918 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:00.918 DEBUG:boto:Port: 8000 2024-02-01T10:23:00.918 DEBUG:boto:Params: {} 2024-02-01T10:23:00.918 DEBUG:boto:Token: None 2024-02-01T10:23:00.918 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:23:00 GMT /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:23:00.918 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:h0iphQ8mQBGQ4B6XDsSjVOL56YM= 2024-02-01T10:23:00.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:h0iphQ8mQBGQ4B6XDsSjVOL56YM=', 'Content-Length': '0'} 2024-02-01T10:23:00.921 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', '-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002ce3622854f5ca8c-0065bb7104-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:00.921 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-02-01T10:23:00.922 DEBUG:boto:path=/vclfcn-17/obj-test-zone2 2024-02-01T10:23:00.922 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2 2024-02-01T10:23:00.922 DEBUG:boto:path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:23:00.922 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:23:00.922 DEBUG:boto:Method: GET 2024-02-01T10:23:00.922 DEBUG:boto:Path: /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:23:00.922 DEBUG:boto:Data: 2024-02-01T10:23:00.922 DEBUG:boto:Headers: {} 2024-02-01T10:23:00.922 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:00.922 DEBUG:boto:Port: 8001 2024-02-01T10:23:00.923 DEBUG:boto:Params: {} 2024-02-01T10:23:00.923 DEBUG:boto:Token: None 2024-02-01T10:23:00.923 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:00 GMT /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:23:00.923 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:xYzLlr6RuBcUqiV0QKJYgkQMdRk= 2024-02-01T10:23:00.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:xYzLlr6RuBcUqiV0QKJYgkQMdRk=', 'Content-Length': '0'} 2024-02-01T10:23:00.926 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', '-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000215843ad4e25bc81-0065bb7104-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:00.972 DEBUG:boto:path=/vclfcn-17/obj-test-zone2 2024-02-01T10:23:00.972 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2 2024-02-01T10:23:00.972 DEBUG:boto:path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:23:00.973 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:23:00.973 DEBUG:boto:Method: GET 2024-02-01T10:23:00.973 DEBUG:boto:Path: /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:23:00.973 DEBUG:boto:Data: 2024-02-01T10:23:00.973 DEBUG:boto:Headers: {} 2024-02-01T10:23:00.973 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:00.973 DEBUG:boto:Port: 8000 2024-02-01T10:23:00.973 DEBUG:boto:Params: {} 2024-02-01T10:23:00.973 DEBUG:boto:Token: None 2024-02-01T10:23:00.973 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:00 GMT /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:23:00.974 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:xYzLlr6RuBcUqiV0QKJYgkQMdRk= 2024-02-01T10:23:00.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:xYzLlr6RuBcUqiV0QKJYgkQMdRk=', 'Content-Length': '0'} 2024-02-01T10:23:00.977 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', '-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000adad4392ddac80d4-0065bb7104-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:00.977 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:23:00.977 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:23:00.977 DEBUG:boto:path=/vclfcn-17/?prefix=obj-test-zone2 2024-02-01T10:23:00.977 DEBUG:boto:auth_path=/vclfcn-17/?prefix=obj-test-zone2 2024-02-01T10:23:00.978 DEBUG:boto:Method: GET 2024-02-01T10:23:00.978 DEBUG:boto:Path: /vclfcn-17/?prefix=obj-test-zone2 2024-02-01T10:23:00.978 DEBUG:boto:Data: 2024-02-01T10:23:00.978 DEBUG:boto:Headers: {} 2024-02-01T10:23:00.978 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:00.978 DEBUG:boto:Port: 8001 2024-02-01T10:23:00.978 DEBUG:boto:Params: {} 2024-02-01T10:23:00.978 DEBUG:boto:Token: None 2024-02-01T10:23:00.978 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:00 GMT /vclfcn-17/ 2024-02-01T10:23:00.978 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:SN+NL+snQVDoyY0UVvFbtJmlCFI= 2024-02-01T10:23:00.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:SN+NL+snQVDoyY0UVvFbtJmlCFI=', 'Content-Length': '0'} 2024-02-01T10:23:00.983 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050b0d9e852a25012-0065bb7104-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:00.983 DEBUG:boto:b'vclfcn-17obj-test-zone21000falseobj-test-zone22024-02-01T10:20:50.624Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:00.984 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:23:00.984 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:23:00.984 DEBUG:boto:path=/vclfcn-17/?prefix=obj-test-zone2 2024-02-01T10:23:00.984 DEBUG:boto:auth_path=/vclfcn-17/?prefix=obj-test-zone2 2024-02-01T10:23:00.984 DEBUG:boto:Method: GET 2024-02-01T10:23:00.984 DEBUG:boto:Path: /vclfcn-17/?prefix=obj-test-zone2 2024-02-01T10:23:00.984 DEBUG:boto:Data: 2024-02-01T10:23:00.984 DEBUG:boto:Headers: {} 2024-02-01T10:23:00.984 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:00.985 DEBUG:boto:Port: 8000 2024-02-01T10:23:00.985 DEBUG:boto:Params: {} 2024-02-01T10:23:00.985 DEBUG:boto:Token: None 2024-02-01T10:23:00.985 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:00 GMT /vclfcn-17/ 2024-02-01T10:23:00.985 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:SN+NL+snQVDoyY0UVvFbtJmlCFI= 2024-02-01T10:23:00.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:SN+NL+snQVDoyY0UVvFbtJmlCFI=', 'Content-Length': '0'} 2024-02-01T10:23:00.990 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ddb2903ef571631a-0065bb7104-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:00.990 DEBUG:boto:b'vclfcn-17obj-test-zone21000falseobj-test-zone22024-02-01T10:20:50.624Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:00.990 DEBUG:boto:path=/vclfcn-17/obj-test-zone2 2024-02-01T10:23:00.991 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2 2024-02-01T10:23:00.991 DEBUG:boto:Method: HEAD 2024-02-01T10:23:00.991 DEBUG:boto:Path: /vclfcn-17/obj-test-zone2 2024-02-01T10:23:00.991 DEBUG:boto:Data: 2024-02-01T10:23:00.991 DEBUG:boto:Headers: {} 2024-02-01T10:23:00.991 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:00.991 DEBUG:boto:Port: 8001 2024-02-01T10:23:00.991 DEBUG:boto:Params: {} 2024-02-01T10:23:00.991 DEBUG:boto:Token: None 2024-02-01T10:23:00.992 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:23:00 GMT /vclfcn-17/obj-test-zone2 2024-02-01T10:23:00.992 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:CAd6/afX2XO1FEWVsoXTXyvL+2I= 2024-02-01T10:23:00.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:CAd6/afX2XO1FEWVsoXTXyvL+2I=', 'Content-Length': '0'} 2024-02-01T10:23:00.995 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', '-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007576f56ed3fc7fd1-0065bb7104-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:00.996 DEBUG:boto:path=/vclfcn-17/obj-test-zone2 2024-02-01T10:23:00.996 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2 2024-02-01T10:23:00.996 DEBUG:boto:Method: HEAD 2024-02-01T10:23:00.996 DEBUG:boto:Path: /vclfcn-17/obj-test-zone2 2024-02-01T10:23:00.996 DEBUG:boto:Data: 2024-02-01T10:23:00.996 DEBUG:boto:Headers: {} 2024-02-01T10:23:00.996 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:00.996 DEBUG:boto:Port: 8000 2024-02-01T10:23:00.996 DEBUG:boto:Params: {} 2024-02-01T10:23:00.996 DEBUG:boto:Token: None 2024-02-01T10:23:00.997 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:23:00 GMT /vclfcn-17/obj-test-zone2 2024-02-01T10:23:00.997 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:CAd6/afX2XO1FEWVsoXTXyvL+2I= 2024-02-01T10:23:00.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:CAd6/afX2XO1FEWVsoXTXyvL+2I=', 'Content-Length': '0'} 2024-02-01T10:23:01.000 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', '-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001d8ccf7698de3d5c-0065bb7105-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:01.000 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-02-01T10:23:01.001 DEBUG:boto:path=/vclfcn-17/obj-test-zone2 2024-02-01T10:23:01.001 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2 2024-02-01T10:23:01.001 DEBUG:boto:path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:23:01.001 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:23:01.001 DEBUG:boto:Method: GET 2024-02-01T10:23:01.001 DEBUG:boto:Path: /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:23:01.001 DEBUG:boto:Data: 2024-02-01T10:23:01.001 DEBUG:boto:Headers: {} 2024-02-01T10:23:01.001 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:01.001 DEBUG:boto:Port: 8001 2024-02-01T10:23:01.001 DEBUG:boto:Params: {} 2024-02-01T10:23:01.001 DEBUG:boto:Token: None 2024-02-01T10:23:01.002 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:01 GMT /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:23:01.002 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Bnh9CXTobMc/cB1ny8aqECTricc= 2024-02-01T10:23:01.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Bnh9CXTobMc/cB1ny8aqECTricc=', 'Content-Length': '0'} 2024-02-01T10:23:01.005 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', '-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005ea6286e62b9a31f-0065bb7105-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:01.048 DEBUG:boto:path=/vclfcn-17/obj-test-zone2 2024-02-01T10:23:01.049 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2 2024-02-01T10:23:01.049 DEBUG:boto:path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:23:01.049 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:23:01.049 DEBUG:boto:Method: GET 2024-02-01T10:23:01.049 DEBUG:boto:Path: /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:23:01.049 DEBUG:boto:Data: 2024-02-01T10:23:01.049 DEBUG:boto:Headers: {} 2024-02-01T10:23:01.049 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:01.049 DEBUG:boto:Port: 8000 2024-02-01T10:23:01.049 DEBUG:boto:Params: {} 2024-02-01T10:23:01.050 DEBUG:boto:Token: None 2024-02-01T10:23:01.050 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:01 GMT /vclfcn-17/obj-test-zone2?versionId=-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1 2024-02-01T10:23:01.050 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Bnh9CXTobMc/cB1ny8aqECTricc= 2024-02-01T10:23:01.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Bnh9CXTobMc/cB1ny8aqECTricc=', 'Content-Length': '0'} 2024-02-01T10:23:01.053 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', '-ZDUDYr9DGFKsQGCvFO.xXKV7f37ke1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000017381e716dcd3ac6-0065bb7105-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:01.053 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:23:01.053 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:23:01.053 DEBUG:boto:path=/vclfcn-17/?prefix=obj-test-zone2 2024-02-01T10:23:01.054 DEBUG:boto:auth_path=/vclfcn-17/?prefix=obj-test-zone2 2024-02-01T10:23:01.054 DEBUG:boto:Method: GET 2024-02-01T10:23:01.054 DEBUG:boto:Path: /vclfcn-17/?prefix=obj-test-zone2 2024-02-01T10:23:01.054 DEBUG:boto:Data: 2024-02-01T10:23:01.054 DEBUG:boto:Headers: {} 2024-02-01T10:23:01.054 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:01.054 DEBUG:boto:Port: 8001 2024-02-01T10:23:01.054 DEBUG:boto:Params: {} 2024-02-01T10:23:01.054 DEBUG:boto:Token: None 2024-02-01T10:23:01.054 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:01 GMT /vclfcn-17/ 2024-02-01T10:23:01.054 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:eD8v4pgj2hYKvk6HrG6+lE/oiYw= 2024-02-01T10:23:01.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:eD8v4pgj2hYKvk6HrG6+lE/oiYw=', 'Content-Length': '0'} 2024-02-01T10:23:01.059 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bdffeede6339b7e5-0065bb7105-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:01.100 DEBUG:boto:b'vclfcn-17obj-test-zone21000falseobj-test-zone22024-02-01T10:20:50.624Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:01.101 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:23:01.101 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:23:01.101 DEBUG:boto:path=/vclfcn-17/?prefix=obj-test-zone2 2024-02-01T10:23:01.101 DEBUG:boto:auth_path=/vclfcn-17/?prefix=obj-test-zone2 2024-02-01T10:23:01.101 DEBUG:boto:Method: GET 2024-02-01T10:23:01.101 DEBUG:boto:Path: /vclfcn-17/?prefix=obj-test-zone2 2024-02-01T10:23:01.101 DEBUG:boto:Data: 2024-02-01T10:23:01.101 DEBUG:boto:Headers: {} 2024-02-01T10:23:01.101 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:01.101 DEBUG:boto:Port: 8000 2024-02-01T10:23:01.101 DEBUG:boto:Params: {} 2024-02-01T10:23:01.102 DEBUG:boto:Token: None 2024-02-01T10:23:01.102 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:01 GMT /vclfcn-17/ 2024-02-01T10:23:01.102 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:eD8v4pgj2hYKvk6HrG6+lE/oiYw= 2024-02-01T10:23:01.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:eD8v4pgj2hYKvk6HrG6+lE/oiYw=', 'Content-Length': '0'} 2024-02-01T10:23:01.107 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005056dbf0aba8fe05-0065bb7105-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:01.107 DEBUG:boto:b'vclfcn-17obj-test-zone21000falseobj-test-zone22024-02-01T10:20:50.624Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:01.108 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-02-01T10:23:01.108 DEBUG:boto:path=/vclfcn-17/obj-test-zone3 2024-02-01T10:23:01.108 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3 2024-02-01T10:23:01.108 DEBUG:boto:path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:23:01.108 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:23:01.108 DEBUG:boto:Method: GET 2024-02-01T10:23:01.108 DEBUG:boto:Path: /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:23:01.108 DEBUG:boto:Data: 2024-02-01T10:23:01.108 DEBUG:boto:Headers: {} 2024-02-01T10:23:01.109 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:01.109 DEBUG:boto:Port: 8001 2024-02-01T10:23:01.109 DEBUG:boto:Params: {} 2024-02-01T10:23:01.109 DEBUG:boto:Token: None 2024-02-01T10:23:01.109 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:01 GMT /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:23:01.109 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:xiDH8rKHEUzQLIRgibGd7cqhqAU= 2024-02-01T10:23:01.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:xiDH8rKHEUzQLIRgibGd7cqhqAU=', 'Content-Length': '0'} 2024-02-01T10:23:01.112 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'YE736OwOOXbZJHJe7d8YvcNV1LxwcQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000425d253ed8fd0a3b-0065bb7105-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:01.112 DEBUG:boto:path=/vclfcn-17/obj-test-zone3 2024-02-01T10:23:01.112 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3 2024-02-01T10:23:01.112 DEBUG:boto:path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:23:01.113 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:23:01.113 DEBUG:boto:Method: GET 2024-02-01T10:23:01.113 DEBUG:boto:Path: /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:23:01.113 DEBUG:boto:Data: 2024-02-01T10:23:01.113 DEBUG:boto:Headers: {} 2024-02-01T10:23:01.113 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:01.113 DEBUG:boto:Port: 8000 2024-02-01T10:23:01.113 DEBUG:boto:Params: {} 2024-02-01T10:23:01.113 DEBUG:boto:Token: None 2024-02-01T10:23:01.113 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:01 GMT /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:23:01.114 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:xiDH8rKHEUzQLIRgibGd7cqhqAU= 2024-02-01T10:23:01.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:xiDH8rKHEUzQLIRgibGd7cqhqAU=', 'Content-Length': '0'} 2024-02-01T10:23:01.116 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'YE736OwOOXbZJHJe7d8YvcNV1LxwcQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007403b1acecfea240-0065bb7105-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:01.160 DEBUG:boto:path=/vclfcn-17/obj-test-zone3 2024-02-01T10:23:01.160 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3 2024-02-01T10:23:01.160 DEBUG:boto:path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:23:01.160 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:23:01.161 DEBUG:boto:Method: HEAD 2024-02-01T10:23:01.161 DEBUG:boto:Path: /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:23:01.161 DEBUG:boto:Data: 2024-02-01T10:23:01.161 DEBUG:boto:Headers: {} 2024-02-01T10:23:01.161 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:01.161 DEBUG:boto:Port: 8001 2024-02-01T10:23:01.161 DEBUG:boto:Params: {} 2024-02-01T10:23:01.161 DEBUG:boto:Token: None 2024-02-01T10:23:01.161 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:23:01 GMT /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:23:01.162 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:och9zDQ9HSLNKxs7EOwZoFJSRUs= 2024-02-01T10:23:01.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:och9zDQ9HSLNKxs7EOwZoFJSRUs=', 'Content-Length': '0'} 2024-02-01T10:23:01.165 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'YE736OwOOXbZJHJe7d8YvcNV1LxwcQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000095c0155a14759f29-0065bb7105-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:01.165 DEBUG:boto:path=/vclfcn-17/obj-test-zone3 2024-02-01T10:23:01.166 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3 2024-02-01T10:23:01.166 DEBUG:boto:path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:23:01.166 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:23:01.166 DEBUG:boto:Method: HEAD 2024-02-01T10:23:01.166 DEBUG:boto:Path: /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:23:01.166 DEBUG:boto:Data: 2024-02-01T10:23:01.166 DEBUG:boto:Headers: {} 2024-02-01T10:23:01.166 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:01.166 DEBUG:boto:Port: 8000 2024-02-01T10:23:01.166 DEBUG:boto:Params: {} 2024-02-01T10:23:01.167 DEBUG:boto:Token: None 2024-02-01T10:23:01.167 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:23:01 GMT /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:23:01.167 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:och9zDQ9HSLNKxs7EOwZoFJSRUs= 2024-02-01T10:23:01.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:och9zDQ9HSLNKxs7EOwZoFJSRUs=', 'Content-Length': '0'} 2024-02-01T10:23:01.170 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'YE736OwOOXbZJHJe7d8YvcNV1LxwcQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000721d48da78c09bf4-0065bb7105-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:01.170 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-02-01T10:23:01.170 DEBUG:boto:path=/vclfcn-17/obj-test-zone3 2024-02-01T10:23:01.170 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3 2024-02-01T10:23:01.170 DEBUG:boto:path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:23:01.170 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:23:01.170 DEBUG:boto:Method: GET 2024-02-01T10:23:01.170 DEBUG:boto:Path: /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:23:01.171 DEBUG:boto:Data: 2024-02-01T10:23:01.171 DEBUG:boto:Headers: {} 2024-02-01T10:23:01.171 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:01.171 DEBUG:boto:Port: 8001 2024-02-01T10:23:01.171 DEBUG:boto:Params: {} 2024-02-01T10:23:01.171 DEBUG:boto:Token: None 2024-02-01T10:23:01.171 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:01 GMT /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:23:01.171 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:xiDH8rKHEUzQLIRgibGd7cqhqAU= 2024-02-01T10:23:01.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:xiDH8rKHEUzQLIRgibGd7cqhqAU=', 'Content-Length': '0'} 2024-02-01T10:23:01.174 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'YE736OwOOXbZJHJe7d8YvcNV1LxwcQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000329392b622bc41c8-0065bb7105-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:01.175 DEBUG:boto:path=/vclfcn-17/obj-test-zone3 2024-02-01T10:23:01.175 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3 2024-02-01T10:23:01.175 DEBUG:boto:path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:23:01.175 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:23:01.175 DEBUG:boto:Method: GET 2024-02-01T10:23:01.175 DEBUG:boto:Path: /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:23:01.175 DEBUG:boto:Data: 2024-02-01T10:23:01.175 DEBUG:boto:Headers: {} 2024-02-01T10:23:01.175 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:01.175 DEBUG:boto:Port: 8000 2024-02-01T10:23:01.176 DEBUG:boto:Params: {} 2024-02-01T10:23:01.176 DEBUG:boto:Token: None 2024-02-01T10:23:01.176 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:01 GMT /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:23:01.176 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:xiDH8rKHEUzQLIRgibGd7cqhqAU= 2024-02-01T10:23:01.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:xiDH8rKHEUzQLIRgibGd7cqhqAU=', 'Content-Length': '0'} 2024-02-01T10:23:01.179 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'YE736OwOOXbZJHJe7d8YvcNV1LxwcQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008290962abaa57940-0065bb7105-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:01.224 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:23:01.224 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:23:01.224 DEBUG:boto:path=/vclfcn-17/?prefix=obj-test-zone3 2024-02-01T10:23:01.225 DEBUG:boto:auth_path=/vclfcn-17/?prefix=obj-test-zone3 2024-02-01T10:23:01.225 DEBUG:boto:Method: GET 2024-02-01T10:23:01.225 DEBUG:boto:Path: /vclfcn-17/?prefix=obj-test-zone3 2024-02-01T10:23:01.225 DEBUG:boto:Data: 2024-02-01T10:23:01.225 DEBUG:boto:Headers: {} 2024-02-01T10:23:01.225 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:01.225 DEBUG:boto:Port: 8001 2024-02-01T10:23:01.225 DEBUG:boto:Params: {} 2024-02-01T10:23:01.225 DEBUG:boto:Token: None 2024-02-01T10:23:01.225 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:01 GMT /vclfcn-17/ 2024-02-01T10:23:01.226 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:eD8v4pgj2hYKvk6HrG6+lE/oiYw= 2024-02-01T10:23:01.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:eD8v4pgj2hYKvk6HrG6+lE/oiYw=', 'Content-Length': '0'} 2024-02-01T10:23:01.230 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000277b93c3a6566cce-0065bb7105-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:01.230 DEBUG:boto:b'vclfcn-17obj-test-zone31000falseobj-test-zone32024-02-01T10:20:50.780Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:01.231 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:23:01.231 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:23:01.231 DEBUG:boto:path=/vclfcn-17/?prefix=obj-test-zone3 2024-02-01T10:23:01.231 DEBUG:boto:auth_path=/vclfcn-17/?prefix=obj-test-zone3 2024-02-01T10:23:01.231 DEBUG:boto:Method: GET 2024-02-01T10:23:01.232 DEBUG:boto:Path: /vclfcn-17/?prefix=obj-test-zone3 2024-02-01T10:23:01.232 DEBUG:boto:Data: 2024-02-01T10:23:01.232 DEBUG:boto:Headers: {} 2024-02-01T10:23:01.232 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:01.232 DEBUG:boto:Port: 8000 2024-02-01T10:23:01.232 DEBUG:boto:Params: {} 2024-02-01T10:23:01.232 DEBUG:boto:Token: None 2024-02-01T10:23:01.232 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:01 GMT /vclfcn-17/ 2024-02-01T10:23:01.232 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:eD8v4pgj2hYKvk6HrG6+lE/oiYw= 2024-02-01T10:23:01.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:eD8v4pgj2hYKvk6HrG6+lE/oiYw=', 'Content-Length': '0'} 2024-02-01T10:23:01.237 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6d6d02b26a0e5c0-0065bb7105-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:01.237 DEBUG:boto:b'vclfcn-17obj-test-zone31000falseobj-test-zone32024-02-01T10:20:50.780Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:01.237 DEBUG:boto:path=/vclfcn-17/obj-test-zone3 2024-02-01T10:23:01.237 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3 2024-02-01T10:23:01.238 DEBUG:boto:Method: HEAD 2024-02-01T10:23:01.238 DEBUG:boto:Path: /vclfcn-17/obj-test-zone3 2024-02-01T10:23:01.238 DEBUG:boto:Data: 2024-02-01T10:23:01.238 DEBUG:boto:Headers: {} 2024-02-01T10:23:01.238 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:01.238 DEBUG:boto:Port: 8001 2024-02-01T10:23:01.238 DEBUG:boto:Params: {} 2024-02-01T10:23:01.238 DEBUG:boto:Token: None 2024-02-01T10:23:01.238 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:23:01 GMT /vclfcn-17/obj-test-zone3 2024-02-01T10:23:01.239 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9nX1hBs7NXhyBGal0+qPrwXcMFU= 2024-02-01T10:23:01.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9nX1hBs7NXhyBGal0+qPrwXcMFU=', 'Content-Length': '0'} 2024-02-01T10:23:01.242 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'YE736OwOOXbZJHJe7d8YvcNV1LxwcQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ddeae2b257ea5337-0065bb7105-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:01.243 DEBUG:boto:path=/vclfcn-17/obj-test-zone3 2024-02-01T10:23:01.243 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3 2024-02-01T10:23:01.243 DEBUG:boto:Method: HEAD 2024-02-01T10:23:01.243 DEBUG:boto:Path: /vclfcn-17/obj-test-zone3 2024-02-01T10:23:01.243 DEBUG:boto:Data: 2024-02-01T10:23:01.243 DEBUG:boto:Headers: {} 2024-02-01T10:23:01.243 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:01.243 DEBUG:boto:Port: 8000 2024-02-01T10:23:01.243 DEBUG:boto:Params: {} 2024-02-01T10:23:01.244 DEBUG:boto:Token: None 2024-02-01T10:23:01.244 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:23:01 GMT /vclfcn-17/obj-test-zone3 2024-02-01T10:23:01.244 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9nX1hBs7NXhyBGal0+qPrwXcMFU= 2024-02-01T10:23:01.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9nX1hBs7NXhyBGal0+qPrwXcMFU=', 'Content-Length': '0'} 2024-02-01T10:23:01.248 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'YE736OwOOXbZJHJe7d8YvcNV1LxwcQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a9f9e59602badae6-0065bb7105-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:01.248 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-02-01T10:23:01.248 DEBUG:boto:path=/vclfcn-17/obj-test-zone3 2024-02-01T10:23:01.248 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3 2024-02-01T10:23:01.248 DEBUG:boto:path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:23:01.248 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:23:01.249 DEBUG:boto:Method: GET 2024-02-01T10:23:01.249 DEBUG:boto:Path: /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:23:01.249 DEBUG:boto:Data: 2024-02-01T10:23:01.249 DEBUG:boto:Headers: {} 2024-02-01T10:23:01.249 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:01.249 DEBUG:boto:Port: 8001 2024-02-01T10:23:01.249 DEBUG:boto:Params: {} 2024-02-01T10:23:01.249 DEBUG:boto:Token: None 2024-02-01T10:23:01.249 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:01 GMT /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:23:01.250 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:xiDH8rKHEUzQLIRgibGd7cqhqAU= 2024-02-01T10:23:01.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:xiDH8rKHEUzQLIRgibGd7cqhqAU=', 'Content-Length': '0'} 2024-02-01T10:23:01.253 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'YE736OwOOXbZJHJe7d8YvcNV1LxwcQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cbf93f1101554fa0-0065bb7105-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:01.301 DEBUG:boto:path=/vclfcn-17/obj-test-zone3 2024-02-01T10:23:01.301 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3 2024-02-01T10:23:01.301 DEBUG:boto:path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:23:01.301 DEBUG:boto:auth_path=/vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:23:01.301 DEBUG:boto:Method: GET 2024-02-01T10:23:01.301 DEBUG:boto:Path: /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:23:01.302 DEBUG:boto:Data: 2024-02-01T10:23:01.302 DEBUG:boto:Headers: {} 2024-02-01T10:23:01.302 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:01.302 DEBUG:boto:Port: 8000 2024-02-01T10:23:01.302 DEBUG:boto:Params: {} 2024-02-01T10:23:01.302 DEBUG:boto:Token: None 2024-02-01T10:23:01.302 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:01 GMT /vclfcn-17/obj-test-zone3?versionId=YE736OwOOXbZJHJe7d8YvcNV1LxwcQb 2024-02-01T10:23:01.302 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:xiDH8rKHEUzQLIRgibGd7cqhqAU= 2024-02-01T10:23:01.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:xiDH8rKHEUzQLIRgibGd7cqhqAU=', 'Content-Length': '0'} 2024-02-01T10:23:01.305 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'YE736OwOOXbZJHJe7d8YvcNV1LxwcQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e929ac93395edcca-0065bb7105-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:01.352 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:23:01.352 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:23:01.352 DEBUG:boto:path=/vclfcn-17/?prefix=obj-test-zone3 2024-02-01T10:23:01.353 DEBUG:boto:auth_path=/vclfcn-17/?prefix=obj-test-zone3 2024-02-01T10:23:01.353 DEBUG:boto:Method: GET 2024-02-01T10:23:01.353 DEBUG:boto:Path: /vclfcn-17/?prefix=obj-test-zone3 2024-02-01T10:23:01.353 DEBUG:boto:Data: 2024-02-01T10:23:01.353 DEBUG:boto:Headers: {} 2024-02-01T10:23:01.353 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:01.353 DEBUG:boto:Port: 8001 2024-02-01T10:23:01.353 DEBUG:boto:Params: {} 2024-02-01T10:23:01.353 DEBUG:boto:Token: None 2024-02-01T10:23:01.353 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:01 GMT /vclfcn-17/ 2024-02-01T10:23:01.354 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:eD8v4pgj2hYKvk6HrG6+lE/oiYw= 2024-02-01T10:23:01.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:eD8v4pgj2hYKvk6HrG6+lE/oiYw=', 'Content-Length': '0'} 2024-02-01T10:23:01.359 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cfb99f930853bf0d-0065bb7105-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:01.359 DEBUG:boto:b'vclfcn-17obj-test-zone31000falseobj-test-zone32024-02-01T10:20:50.780Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:01.360 DEBUG:boto:path=/vclfcn-17/ 2024-02-01T10:23:01.360 DEBUG:boto:auth_path=/vclfcn-17/ 2024-02-01T10:23:01.360 DEBUG:boto:path=/vclfcn-17/?prefix=obj-test-zone3 2024-02-01T10:23:01.360 DEBUG:boto:auth_path=/vclfcn-17/?prefix=obj-test-zone3 2024-02-01T10:23:01.360 DEBUG:boto:Method: GET 2024-02-01T10:23:01.360 DEBUG:boto:Path: /vclfcn-17/?prefix=obj-test-zone3 2024-02-01T10:23:01.360 DEBUG:boto:Data: 2024-02-01T10:23:01.360 DEBUG:boto:Headers: {} 2024-02-01T10:23:01.360 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:01.361 DEBUG:boto:Port: 8000 2024-02-01T10:23:01.361 DEBUG:boto:Params: {} 2024-02-01T10:23:01.361 DEBUG:boto:Token: None 2024-02-01T10:23:01.361 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:01 GMT /vclfcn-17/ 2024-02-01T10:23:01.361 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:eD8v4pgj2hYKvk6HrG6+lE/oiYw= 2024-02-01T10:23:01.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:eD8v4pgj2hYKvk6HrG6+lE/oiYw=', 'Content-Length': '0'} 2024-02-01T10:23:01.366 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000088f3d7245d96361-0065bb7105-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:01.408 DEBUG:boto:b'vclfcn-17obj-test-zone31000falseobj-test-zone32024-02-01T10:20:50.780Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:01.409 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-17 zones={test-zone3, test-zone2} 2024-02-01T10:23:01.409 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=vclfcn-18 2024-02-01T10:23:01.409 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:23:01.409 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:23:01.409 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket vclfcn-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-02-01T10:23:01.462 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:23:01.462 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:23:01.528 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:23:01.532+0000 7ff1e7472980 1 bucket sync caught up with source: 2024-02-01T10:23:01.529 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, 00000000011.78.6, , 00000000005.23.10, , 00000000008.32.10, , , , , ] 2024-02-01T10:23:01.529 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, 00000000011.78.6, , 00000000005.23.10, , 00000000008.32.10, , , , , ] 2024-02-01T10:23:01.529 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:23:01.532+0000 7ff1e7472980 0 bucket checkpoint complete 2024-02-01T10:23:01.552 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=vclfcn-18 2024-02-01T10:23:01.552 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:23:01.552 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:23:01.552 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:23:01.602 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:23:01.603 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:23:01.663 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:23:01.668+0000 7fefa890a980 1 bucket sync caught up with source: 2024-02-01T10:23:01.664 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, 00000000011.78.6, , 00000000005.23.10, , 00000000008.32.10, , , , , ] 2024-02-01T10:23:01.664 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, 00000000011.78.6, , 00000000005.23.10, , 00000000008.32.10, , , , , ] 2024-02-01T10:23:01.664 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:23:01.668+0000 7fefa890a980 0 bucket checkpoint complete 2024-02-01T10:23:01.689 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=vclfcn-18 2024-02-01T10:23:01.689 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:23:01.689 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:23:01.690 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:23:01.732 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:23:01.732 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:23:01.796 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:23:01.800+0000 7f4487dfc980 1 bucket sync caught up with source: 2024-02-01T10:23:01.796 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, 00000000003.52.10, , 00000000011.46.6, , 00000000008.34.10, , , , , ] 2024-02-01T10:23:01.797 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, 00000000003.52.10, , 00000000011.46.6, , 00000000008.34.10, , , , , ] 2024-02-01T10:23:01.797 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:23:01.800+0000 7f4487dfc980 0 bucket checkpoint complete 2024-02-01T10:23:01.818 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=vclfcn-18 2024-02-01T10:23:01.818 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:23:01.818 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:23:01.818 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:23:01.900 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:23:01.900 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:23:01.960 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:23:01.964+0000 7f0abad54980 1 bucket sync caught up with source: 2024-02-01T10:23:01.960 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, 00000000003.52.10, , 00000000011.46.6, , 00000000008.34.10, , , , , ] 2024-02-01T10:23:01.960 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, 00000000003.52.10, , 00000000011.46.6, , 00000000008.34.10, , , , , ] 2024-02-01T10:23:01.960 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:23:01.964+0000 7f0abad54980 0 bucket checkpoint complete 2024-02-01T10:23:01.981 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=vclfcn-18 2024-02-01T10:23:01.981 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:23:01.982 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:23:01.982 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:23:02.069 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:23:02.069 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:23:02.126 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:23:02.128+0000 7f3980b3a980 1 bucket sync caught up with source: 2024-02-01T10:23:02.126 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, 00000000003.140.10, , 00000000005.62.10, , 00000000016.98.6, , , , , ] 2024-02-01T10:23:02.127 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, 00000000003.140.10, , 00000000005.62.10, , 00000000016.98.6, , , , , ] 2024-02-01T10:23:02.127 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:23:02.128+0000 7f3980b3a980 0 bucket checkpoint complete 2024-02-01T10:23:02.149 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=vclfcn-18 2024-02-01T10:23:02.149 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:23:02.149 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:23:02.149 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket vclfcn-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-02-01T10:23:02.199 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:23:02.199 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:23:02.266 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:23:02.268+0000 7f5e86f0b980 1 bucket sync caught up with source: 2024-02-01T10:23:02.266 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, 00000000003.140.10, , 00000000005.62.10, , 00000000016.98.6, , , , , ] 2024-02-01T10:23:02.266 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, 00000000003.140.10, , 00000000005.62.10, , 00000000016.98.6, , , , , ] 2024-02-01T10:23:02.266 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:23:02.268+0000 7f5e86f0b980 0 bucket checkpoint complete 2024-02-01T10:23:02.288 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-18 zones={test-zone2, test-zone1} 2024-02-01T10:23:02.289 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:23:02.289 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:23:02.289 DEBUG:boto:Method: HEAD 2024-02-01T10:23:02.289 DEBUG:boto:Path: /vclfcn-18/ 2024-02-01T10:23:02.289 DEBUG:boto:Data: 2024-02-01T10:23:02.289 DEBUG:boto:Headers: {} 2024-02-01T10:23:02.289 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:02.289 DEBUG:boto:Port: 8000 2024-02-01T10:23:02.289 DEBUG:boto:Params: {} 2024-02-01T10:23:02.290 DEBUG:boto:Token: None 2024-02-01T10:23:02.290 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:23:02 GMT /vclfcn-18/ 2024-02-01T10:23:02.290 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:uTJbhbiGBjr6mYTy+CnzOBbF2d0= 2024-02-01T10:23:02.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:uTJbhbiGBjr6mYTy+CnzOBbF2d0=', 'Content-Length': '0'} 2024-02-01T10:23:02.292 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f38580d4b588087f-0065bb7106-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:02.292 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:23:02.292 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:23:02.293 DEBUG:boto:Method: HEAD 2024-02-01T10:23:02.293 DEBUG:boto:Path: /vclfcn-18/ 2024-02-01T10:23:02.293 DEBUG:boto:Data: 2024-02-01T10:23:02.293 DEBUG:boto:Headers: {} 2024-02-01T10:23:02.293 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:23:02.293 DEBUG:boto:Port: 8000 2024-02-01T10:23:02.293 DEBUG:boto:Params: {} 2024-02-01T10:23:02.293 DEBUG:boto:Token: None 2024-02-01T10:23:02.293 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:23:02 GMT /vclfcn-18/ 2024-02-01T10:23:02.293 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:uTJbhbiGBjr6mYTy+CnzOBbF2d0= 2024-02-01T10:23:02.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:uTJbhbiGBjr6mYTy+CnzOBbF2d0=', 'Content-Length': '0'} 2024-02-01T10:23:02.297 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d51669f210b4e103-0065bb7106-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:02.297 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:23:02.298 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:23:02.298 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:23:02.298 DEBUG:boto:path=/vclfcn-18/?versions&max-keys=999 2024-02-01T10:23:02.298 DEBUG:boto:auth_path=/vclfcn-18/?versions&max-keys=999 2024-02-01T10:23:02.298 DEBUG:boto:Method: GET 2024-02-01T10:23:02.298 DEBUG:boto:Path: /vclfcn-18/?versions&max-keys=999 2024-02-01T10:23:02.298 DEBUG:boto:Data: 2024-02-01T10:23:02.298 DEBUG:boto:Headers: {} 2024-02-01T10:23:02.298 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:02.298 DEBUG:boto:Port: 8000 2024-02-01T10:23:02.299 DEBUG:boto:Params: {} 2024-02-01T10:23:02.299 DEBUG:boto:Token: None 2024-02-01T10:23:02.299 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:02 GMT /vclfcn-18/?versions 2024-02-01T10:23:02.299 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ICDmtQ2M8jl5rz2A19k9KiHFEWM= 2024-02-01T10:23:02.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ICDmtQ2M8jl5rz2A19k9KiHFEWM=', 'Content-Length': '0'} 2024-02-01T10:23:02.301 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6f05560dd2d2078-0065bb7106-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:02.301 DEBUG:boto:b'vclfcn-18999falsedummytxgLlJWZVbWbjqLWqkOtt87dUoD4zuntrue2024-02-01T10:20:34.645Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcKtrue2024-02-01T10:20:50.942Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutDtrue2024-02-01T10:20:51.094Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone36pAVLbw3HAwE0jxxbuxTIxwcA2p4q8Ntrue2024-02-01T10:20:51.248Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:02.302 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-02-01T10:23:02.302 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-02-01T10:23:02.302 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-02-01T10:23:02.302 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-02-01T10:23:02.302 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:23:02.302 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:23:02.303 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:23:02.303 DEBUG:boto:path=/vclfcn-18/?versions&max-keys=999 2024-02-01T10:23:02.303 DEBUG:boto:auth_path=/vclfcn-18/?versions&max-keys=999 2024-02-01T10:23:02.303 DEBUG:boto:Method: GET 2024-02-01T10:23:02.303 DEBUG:boto:Path: /vclfcn-18/?versions&max-keys=999 2024-02-01T10:23:02.303 DEBUG:boto:Data: 2024-02-01T10:23:02.303 DEBUG:boto:Headers: {} 2024-02-01T10:23:02.303 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:23:02.303 DEBUG:boto:Port: 8000 2024-02-01T10:23:02.303 DEBUG:boto:Params: {} 2024-02-01T10:23:02.303 DEBUG:boto:Token: None 2024-02-01T10:23:02.304 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:02 GMT /vclfcn-18/?versions 2024-02-01T10:23:02.304 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ICDmtQ2M8jl5rz2A19k9KiHFEWM= 2024-02-01T10:23:02.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ICDmtQ2M8jl5rz2A19k9KiHFEWM=', 'Content-Length': '0'} 2024-02-01T10:23:02.308 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d18819b63682f805-0065bb7106-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:02.308 DEBUG:boto:b'vclfcn-18999falsedummytxgLlJWZVbWbjqLWqkOtt87dUoD4zuntrue2024-02-01T10:20:34.645Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcKtrue2024-02-01T10:20:50.942Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutDtrue2024-02-01T10:20:51.094Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone36pAVLbw3HAwE0jxxbuxTIxwcA2p4q8Ntrue2024-02-01T10:20:51.248Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:02.308 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-02-01T10:23:02.309 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-02-01T10:23:02.309 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-02-01T10:23:02.309 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-02-01T10:23:02.309 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:23:02.309 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:23:02.309 DEBUG:boto:path=/vclfcn-18/?versions&max-keys=999 2024-02-01T10:23:02.309 DEBUG:boto:auth_path=/vclfcn-18/?versions&max-keys=999 2024-02-01T10:23:02.309 DEBUG:boto:Method: GET 2024-02-01T10:23:02.309 DEBUG:boto:Path: /vclfcn-18/?versions&max-keys=999 2024-02-01T10:23:02.309 DEBUG:boto:Data: 2024-02-01T10:23:02.309 DEBUG:boto:Headers: {} 2024-02-01T10:23:02.310 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:02.310 DEBUG:boto:Port: 8000 2024-02-01T10:23:02.310 DEBUG:boto:Params: {} 2024-02-01T10:23:02.310 DEBUG:boto:Token: None 2024-02-01T10:23:02.310 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:02 GMT /vclfcn-18/?versions 2024-02-01T10:23:02.310 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ICDmtQ2M8jl5rz2A19k9KiHFEWM= 2024-02-01T10:23:02.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ICDmtQ2M8jl5rz2A19k9KiHFEWM=', 'Content-Length': '0'} 2024-02-01T10:23:02.312 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042cd36b252503091-0065bb7106-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:02.312 DEBUG:boto:b'vclfcn-18999falsedummytxgLlJWZVbWbjqLWqkOtt87dUoD4zuntrue2024-02-01T10:20:34.645Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcKtrue2024-02-01T10:20:50.942Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutDtrue2024-02-01T10:20:51.094Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone36pAVLbw3HAwE0jxxbuxTIxwcA2p4q8Ntrue2024-02-01T10:20:51.248Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:02.313 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:23:02.313 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:23:02.313 DEBUG:boto:path=/vclfcn-18/?versions&max-keys=999 2024-02-01T10:23:02.313 DEBUG:boto:auth_path=/vclfcn-18/?versions&max-keys=999 2024-02-01T10:23:02.313 DEBUG:boto:Method: GET 2024-02-01T10:23:02.313 DEBUG:boto:Path: /vclfcn-18/?versions&max-keys=999 2024-02-01T10:23:02.313 DEBUG:boto:Data: 2024-02-01T10:23:02.314 DEBUG:boto:Headers: {} 2024-02-01T10:23:02.314 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:23:02.314 DEBUG:boto:Port: 8000 2024-02-01T10:23:02.314 DEBUG:boto:Params: {} 2024-02-01T10:23:02.314 DEBUG:boto:Token: None 2024-02-01T10:23:02.314 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:02 GMT /vclfcn-18/?versions 2024-02-01T10:23:02.314 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ICDmtQ2M8jl5rz2A19k9KiHFEWM= 2024-02-01T10:23:02.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ICDmtQ2M8jl5rz2A19k9KiHFEWM=', 'Content-Length': '0'} 2024-02-01T10:23:02.318 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af508753628999a7-0065bb7106-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:02.319 DEBUG:boto:b'vclfcn-18999falsedummytxgLlJWZVbWbjqLWqkOtt87dUoD4zuntrue2024-02-01T10:20:34.645Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcKtrue2024-02-01T10:20:50.942Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutDtrue2024-02-01T10:20:51.094Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone36pAVLbw3HAwE0jxxbuxTIxwcA2p4q8Ntrue2024-02-01T10:20:51.248Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:02.319 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-02-01T10:23:02.319 DEBUG:boto:path=/vclfcn-18/dummy 2024-02-01T10:23:02.320 DEBUG:boto:auth_path=/vclfcn-18/dummy 2024-02-01T10:23:02.320 DEBUG:boto:path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:02.320 DEBUG:boto:auth_path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:02.320 DEBUG:boto:Method: GET 2024-02-01T10:23:02.320 DEBUG:boto:Path: /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:02.320 DEBUG:boto:Data: 2024-02-01T10:23:02.320 DEBUG:boto:Headers: {} 2024-02-01T10:23:02.320 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:02.320 DEBUG:boto:Port: 8000 2024-02-01T10:23:02.320 DEBUG:boto:Params: {} 2024-02-01T10:23:02.321 DEBUG:boto:Token: None 2024-02-01T10:23:02.321 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:02 GMT /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:02.321 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wSO9eNpMFg93RMzMk6JnIURzySI= 2024-02-01T10:23:02.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wSO9eNpMFg93RMzMk6JnIURzySI=', 'Content-Length': '0'} 2024-02-01T10:23:02.322 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:34 GMT'), ('x-amz-version-id', 'txgLlJWZVbWbjqLWqkOtt87dUoD4zun'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b3221070364516f4-0065bb7106-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:02.323 DEBUG:boto:path=/vclfcn-18/dummy 2024-02-01T10:23:02.323 DEBUG:boto:auth_path=/vclfcn-18/dummy 2024-02-01T10:23:02.323 DEBUG:boto:path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:02.323 DEBUG:boto:auth_path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:02.323 DEBUG:boto:Method: GET 2024-02-01T10:23:02.323 DEBUG:boto:Path: /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:02.323 DEBUG:boto:Data: 2024-02-01T10:23:02.323 DEBUG:boto:Headers: {} 2024-02-01T10:23:02.323 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:23:02.324 DEBUG:boto:Port: 8000 2024-02-01T10:23:02.324 DEBUG:boto:Params: {} 2024-02-01T10:23:02.324 DEBUG:boto:Token: None 2024-02-01T10:23:02.324 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:02 GMT /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:02.324 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wSO9eNpMFg93RMzMk6JnIURzySI= 2024-02-01T10:23:02.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wSO9eNpMFg93RMzMk6JnIURzySI=', 'Content-Length': '0'} 2024-02-01T10:23:02.327 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:34 GMT'), ('x-amz-version-id', 'txgLlJWZVbWbjqLWqkOtt87dUoD4zun'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fb25f3dd04c668a1-0065bb7106-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:02.327 DEBUG:boto:path=/vclfcn-18/dummy 2024-02-01T10:23:02.327 DEBUG:boto:auth_path=/vclfcn-18/dummy 2024-02-01T10:23:02.328 DEBUG:boto:path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:02.328 DEBUG:boto:auth_path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:02.328 DEBUG:boto:Method: HEAD 2024-02-01T10:23:02.328 DEBUG:boto:Path: /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:02.328 DEBUG:boto:Data: 2024-02-01T10:23:02.328 DEBUG:boto:Headers: {} 2024-02-01T10:23:02.328 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:02.328 DEBUG:boto:Port: 8000 2024-02-01T10:23:02.328 DEBUG:boto:Params: {} 2024-02-01T10:23:02.328 DEBUG:boto:Token: None 2024-02-01T10:23:02.329 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:23:02 GMT /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:02.329 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:j5vr/+MbCBo5AQmNPafLM5ZlJE8= 2024-02-01T10:23:02.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:j5vr/+MbCBo5AQmNPafLM5ZlJE8=', 'Content-Length': '0'} 2024-02-01T10:23:02.330 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:34 GMT'), ('x-amz-version-id', 'txgLlJWZVbWbjqLWqkOtt87dUoD4zun'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ef251b199d9ba94e-0065bb7106-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:02.331 DEBUG:boto:path=/vclfcn-18/dummy 2024-02-01T10:23:02.331 DEBUG:boto:auth_path=/vclfcn-18/dummy 2024-02-01T10:23:02.331 DEBUG:boto:path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:02.331 DEBUG:boto:auth_path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:02.331 DEBUG:boto:Method: HEAD 2024-02-01T10:23:02.331 DEBUG:boto:Path: /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:02.331 DEBUG:boto:Data: 2024-02-01T10:23:02.331 DEBUG:boto:Headers: {} 2024-02-01T10:23:02.331 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:23:02.331 DEBUG:boto:Port: 8000 2024-02-01T10:23:02.332 DEBUG:boto:Params: {} 2024-02-01T10:23:02.332 DEBUG:boto:Token: None 2024-02-01T10:23:02.332 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:23:02 GMT /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:02.332 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:j5vr/+MbCBo5AQmNPafLM5ZlJE8= 2024-02-01T10:23:02.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:j5vr/+MbCBo5AQmNPafLM5ZlJE8=', 'Content-Length': '0'} 2024-02-01T10:23:02.334 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:34 GMT'), ('x-amz-version-id', 'txgLlJWZVbWbjqLWqkOtt87dUoD4zun'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005bfcd762f94f5091-0065bb7106-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:02.335 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-02-01T10:23:02.335 DEBUG:boto:path=/vclfcn-18/dummy 2024-02-01T10:23:02.335 DEBUG:boto:auth_path=/vclfcn-18/dummy 2024-02-01T10:23:02.335 DEBUG:boto:path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:02.335 DEBUG:boto:auth_path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:02.335 DEBUG:boto:Method: GET 2024-02-01T10:23:02.335 DEBUG:boto:Path: /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:02.335 DEBUG:boto:Data: 2024-02-01T10:23:02.335 DEBUG:boto:Headers: {} 2024-02-01T10:23:02.336 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:02.336 DEBUG:boto:Port: 8000 2024-02-01T10:23:02.336 DEBUG:boto:Params: {} 2024-02-01T10:23:02.336 DEBUG:boto:Token: None 2024-02-01T10:23:02.336 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:02 GMT /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:02.336 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wSO9eNpMFg93RMzMk6JnIURzySI= 2024-02-01T10:23:02.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wSO9eNpMFg93RMzMk6JnIURzySI=', 'Content-Length': '0'} 2024-02-01T10:23:02.338 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:34 GMT'), ('x-amz-version-id', 'txgLlJWZVbWbjqLWqkOtt87dUoD4zun'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f4b7e55bd78a2dbf-0065bb7106-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:02.339 DEBUG:boto:path=/vclfcn-18/dummy 2024-02-01T10:23:02.339 DEBUG:boto:auth_path=/vclfcn-18/dummy 2024-02-01T10:23:02.339 DEBUG:boto:path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:02.339 DEBUG:boto:auth_path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:02.339 DEBUG:boto:Method: GET 2024-02-01T10:23:02.339 DEBUG:boto:Path: /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:02.339 DEBUG:boto:Data: 2024-02-01T10:23:02.339 DEBUG:boto:Headers: {} 2024-02-01T10:23:02.339 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:23:02.339 DEBUG:boto:Port: 8000 2024-02-01T10:23:02.340 DEBUG:boto:Params: {} 2024-02-01T10:23:02.340 DEBUG:boto:Token: None 2024-02-01T10:23:02.340 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:02 GMT /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:02.340 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wSO9eNpMFg93RMzMk6JnIURzySI= 2024-02-01T10:23:02.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wSO9eNpMFg93RMzMk6JnIURzySI=', 'Content-Length': '0'} 2024-02-01T10:23:02.343 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:34 GMT'), ('x-amz-version-id', 'txgLlJWZVbWbjqLWqkOtt87dUoD4zun'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000af476089310ac394-0065bb7106-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:02.343 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:23:02.343 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:23:02.343 DEBUG:boto:path=/vclfcn-18/?prefix=dummy 2024-02-01T10:23:02.343 DEBUG:boto:auth_path=/vclfcn-18/?prefix=dummy 2024-02-01T10:23:02.343 DEBUG:boto:Method: GET 2024-02-01T10:23:02.343 DEBUG:boto:Path: /vclfcn-18/?prefix=dummy 2024-02-01T10:23:02.343 DEBUG:boto:Data: 2024-02-01T10:23:02.344 DEBUG:boto:Headers: {} 2024-02-01T10:23:02.344 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:02.344 DEBUG:boto:Port: 8000 2024-02-01T10:23:02.344 DEBUG:boto:Params: {} 2024-02-01T10:23:02.344 DEBUG:boto:Token: None 2024-02-01T10:23:02.344 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:02 GMT /vclfcn-18/ 2024-02-01T10:23:02.344 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zWlw87JanONUh75e5vJgAWli5Qk= 2024-02-01T10:23:02.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zWlw87JanONUh75e5vJgAWli5Qk=', 'Content-Length': '0'} 2024-02-01T10:23:02.347 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099177addc4ab86bf-0065bb7106-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:02.392 DEBUG:boto:b'vclfcn-18dummy1000falsedummy2024-02-01T10:20:34.645Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:02.393 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:23:02.393 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:23:02.393 DEBUG:boto:path=/vclfcn-18/?prefix=dummy 2024-02-01T10:23:02.393 DEBUG:boto:auth_path=/vclfcn-18/?prefix=dummy 2024-02-01T10:23:02.393 DEBUG:boto:Method: GET 2024-02-01T10:23:02.393 DEBUG:boto:Path: /vclfcn-18/?prefix=dummy 2024-02-01T10:23:02.393 DEBUG:boto:Data: 2024-02-01T10:23:02.393 DEBUG:boto:Headers: {} 2024-02-01T10:23:02.393 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:23:02.393 DEBUG:boto:Port: 8000 2024-02-01T10:23:02.393 DEBUG:boto:Params: {} 2024-02-01T10:23:02.394 DEBUG:boto:Token: None 2024-02-01T10:23:02.394 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:02 GMT /vclfcn-18/ 2024-02-01T10:23:02.394 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zWlw87JanONUh75e5vJgAWli5Qk= 2024-02-01T10:23:02.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zWlw87JanONUh75e5vJgAWli5Qk=', 'Content-Length': '0'} 2024-02-01T10:23:02.398 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009549fe330479f86a-0065bb7106-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:02.398 DEBUG:boto:b'vclfcn-18dummy1000falsedummy2024-02-01T10:20:34.645Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:02.398 DEBUG:boto:path=/vclfcn-18/dummy 2024-02-01T10:23:02.398 DEBUG:boto:auth_path=/vclfcn-18/dummy 2024-02-01T10:23:02.399 DEBUG:boto:Method: HEAD 2024-02-01T10:23:02.399 DEBUG:boto:Path: /vclfcn-18/dummy 2024-02-01T10:23:02.399 DEBUG:boto:Data: 2024-02-01T10:23:02.399 DEBUG:boto:Headers: {} 2024-02-01T10:23:02.399 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:02.399 DEBUG:boto:Port: 8000 2024-02-01T10:23:02.399 DEBUG:boto:Params: {} 2024-02-01T10:23:02.399 DEBUG:boto:Token: None 2024-02-01T10:23:02.399 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:23:02 GMT /vclfcn-18/dummy 2024-02-01T10:23:02.400 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wTICgFigGPmGl0hmQZbadvLkg9Y= 2024-02-01T10:23:02.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wTICgFigGPmGl0hmQZbadvLkg9Y=', 'Content-Length': '0'} 2024-02-01T10:23:02.404 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:34 GMT'), ('x-amz-version-id', 'txgLlJWZVbWbjqLWqkOtt87dUoD4zun'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000dd0cdfb07c336dbf-0065bb7106-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:02.404 DEBUG:boto:path=/vclfcn-18/dummy 2024-02-01T10:23:02.404 DEBUG:boto:auth_path=/vclfcn-18/dummy 2024-02-01T10:23:02.404 DEBUG:boto:Method: HEAD 2024-02-01T10:23:02.405 DEBUG:boto:Path: /vclfcn-18/dummy 2024-02-01T10:23:02.405 DEBUG:boto:Data: 2024-02-01T10:23:02.405 DEBUG:boto:Headers: {} 2024-02-01T10:23:02.405 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:23:02.405 DEBUG:boto:Port: 8000 2024-02-01T10:23:02.405 DEBUG:boto:Params: {} 2024-02-01T10:23:02.405 DEBUG:boto:Token: None 2024-02-01T10:23:02.405 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:23:02 GMT /vclfcn-18/dummy 2024-02-01T10:23:02.405 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wTICgFigGPmGl0hmQZbadvLkg9Y= 2024-02-01T10:23:02.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wTICgFigGPmGl0hmQZbadvLkg9Y=', 'Content-Length': '0'} 2024-02-01T10:23:02.409 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:34 GMT'), ('x-amz-version-id', 'txgLlJWZVbWbjqLWqkOtt87dUoD4zun'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000abc277c529832b50-0065bb7106-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:02.410 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-02-01T10:23:02.410 DEBUG:boto:path=/vclfcn-18/dummy 2024-02-01T10:23:02.410 DEBUG:boto:auth_path=/vclfcn-18/dummy 2024-02-01T10:23:02.410 DEBUG:boto:path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:02.410 DEBUG:boto:auth_path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:02.410 DEBUG:boto:Method: GET 2024-02-01T10:23:02.410 DEBUG:boto:Path: /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:02.410 DEBUG:boto:Data: 2024-02-01T10:23:02.410 DEBUG:boto:Headers: {} 2024-02-01T10:23:02.410 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:02.411 DEBUG:boto:Port: 8000 2024-02-01T10:23:02.411 DEBUG:boto:Params: {} 2024-02-01T10:23:02.411 DEBUG:boto:Token: None 2024-02-01T10:23:02.411 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:02 GMT /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:02.411 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wSO9eNpMFg93RMzMk6JnIURzySI= 2024-02-01T10:23:02.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wSO9eNpMFg93RMzMk6JnIURzySI=', 'Content-Length': '0'} 2024-02-01T10:23:02.414 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:34 GMT'), ('x-amz-version-id', 'txgLlJWZVbWbjqLWqkOtt87dUoD4zun'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000489e29b76d1246a0-0065bb7106-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:02.414 DEBUG:boto:path=/vclfcn-18/dummy 2024-02-01T10:23:02.415 DEBUG:boto:auth_path=/vclfcn-18/dummy 2024-02-01T10:23:02.415 DEBUG:boto:path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:02.415 DEBUG:boto:auth_path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:02.415 DEBUG:boto:Method: GET 2024-02-01T10:23:02.415 DEBUG:boto:Path: /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:02.415 DEBUG:boto:Data: 2024-02-01T10:23:02.415 DEBUG:boto:Headers: {} 2024-02-01T10:23:02.415 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:23:02.415 DEBUG:boto:Port: 8000 2024-02-01T10:23:02.415 DEBUG:boto:Params: {} 2024-02-01T10:23:02.416 DEBUG:boto:Token: None 2024-02-01T10:23:02.416 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:02 GMT /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:02.416 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wSO9eNpMFg93RMzMk6JnIURzySI= 2024-02-01T10:23:02.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wSO9eNpMFg93RMzMk6JnIURzySI=', 'Content-Length': '0'} 2024-02-01T10:23:02.418 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:34 GMT'), ('x-amz-version-id', 'txgLlJWZVbWbjqLWqkOtt87dUoD4zun'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d567533e0f7a354e-0065bb7106-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:02.419 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:23:02.419 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:23:02.419 DEBUG:boto:path=/vclfcn-18/?prefix=dummy 2024-02-01T10:23:02.419 DEBUG:boto:auth_path=/vclfcn-18/?prefix=dummy 2024-02-01T10:23:02.419 DEBUG:boto:Method: GET 2024-02-01T10:23:02.419 DEBUG:boto:Path: /vclfcn-18/?prefix=dummy 2024-02-01T10:23:02.419 DEBUG:boto:Data: 2024-02-01T10:23:02.420 DEBUG:boto:Headers: {} 2024-02-01T10:23:02.420 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:02.420 DEBUG:boto:Port: 8000 2024-02-01T10:23:02.420 DEBUG:boto:Params: {} 2024-02-01T10:23:02.420 DEBUG:boto:Token: None 2024-02-01T10:23:02.420 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:02 GMT /vclfcn-18/ 2024-02-01T10:23:02.420 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zWlw87JanONUh75e5vJgAWli5Qk= 2024-02-01T10:23:02.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zWlw87JanONUh75e5vJgAWli5Qk=', 'Content-Length': '0'} 2024-02-01T10:23:02.424 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055a65dc06411f998-0065bb7106-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:02.424 DEBUG:boto:b'vclfcn-18dummy1000falsedummy2024-02-01T10:20:34.645Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:02.425 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:23:02.425 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:23:02.425 DEBUG:boto:path=/vclfcn-18/?prefix=dummy 2024-02-01T10:23:02.425 DEBUG:boto:auth_path=/vclfcn-18/?prefix=dummy 2024-02-01T10:23:02.425 DEBUG:boto:Method: GET 2024-02-01T10:23:02.425 DEBUG:boto:Path: /vclfcn-18/?prefix=dummy 2024-02-01T10:23:02.426 DEBUG:boto:Data: 2024-02-01T10:23:02.426 DEBUG:boto:Headers: {} 2024-02-01T10:23:02.426 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:23:02.426 DEBUG:boto:Port: 8000 2024-02-01T10:23:02.426 DEBUG:boto:Params: {} 2024-02-01T10:23:02.426 DEBUG:boto:Token: None 2024-02-01T10:23:02.426 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:02 GMT /vclfcn-18/ 2024-02-01T10:23:02.426 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zWlw87JanONUh75e5vJgAWli5Qk= 2024-02-01T10:23:02.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zWlw87JanONUh75e5vJgAWli5Qk=', 'Content-Length': '0'} 2024-02-01T10:23:02.430 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c83f83e862df7899-0065bb7106-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:02.431 DEBUG:boto:b'vclfcn-18dummy1000falsedummy2024-02-01T10:20:34.645Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:02.431 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-02-01T10:23:02.431 DEBUG:boto:path=/vclfcn-18/obj-test-zone1 2024-02-01T10:23:02.431 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1 2024-02-01T10:23:02.432 DEBUG:boto:path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:02.432 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:02.432 DEBUG:boto:Method: GET 2024-02-01T10:23:02.432 DEBUG:boto:Path: /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:02.432 DEBUG:boto:Data: 2024-02-01T10:23:02.432 DEBUG:boto:Headers: {} 2024-02-01T10:23:02.432 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:02.432 DEBUG:boto:Port: 8000 2024-02-01T10:23:02.432 DEBUG:boto:Params: {} 2024-02-01T10:23:02.432 DEBUG:boto:Token: None 2024-02-01T10:23:02.433 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:02 GMT /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:02.433 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:clncxt2mCwBcBsQeeeqnnQniqzY= 2024-02-01T10:23:02.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:clncxt2mCwBcBsQeeeqnnQniqzY=', 'Content-Length': '0'} 2024-02-01T10:23:02.436 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000030b3b7c40afd1fe8-0065bb7106-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:02.480 DEBUG:boto:path=/vclfcn-18/obj-test-zone1 2024-02-01T10:23:02.480 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1 2024-02-01T10:23:02.480 DEBUG:boto:path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:02.481 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:02.481 DEBUG:boto:Method: GET 2024-02-01T10:23:02.481 DEBUG:boto:Path: /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:02.481 DEBUG:boto:Data: 2024-02-01T10:23:02.481 DEBUG:boto:Headers: {} 2024-02-01T10:23:02.481 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:23:02.481 DEBUG:boto:Port: 8000 2024-02-01T10:23:02.481 DEBUG:boto:Params: {} 2024-02-01T10:23:02.481 DEBUG:boto:Token: None 2024-02-01T10:23:02.482 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:02 GMT /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:02.482 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:clncxt2mCwBcBsQeeeqnnQniqzY= 2024-02-01T10:23:02.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:clncxt2mCwBcBsQeeeqnnQniqzY=', 'Content-Length': '0'} 2024-02-01T10:23:02.485 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000037427cddff1e6e26-0065bb7106-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:02.485 DEBUG:boto:path=/vclfcn-18/obj-test-zone1 2024-02-01T10:23:02.485 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1 2024-02-01T10:23:02.486 DEBUG:boto:path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:02.486 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:02.486 DEBUG:boto:Method: HEAD 2024-02-01T10:23:02.486 DEBUG:boto:Path: /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:02.486 DEBUG:boto:Data: 2024-02-01T10:23:02.486 DEBUG:boto:Headers: {} 2024-02-01T10:23:02.486 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:02.486 DEBUG:boto:Port: 8000 2024-02-01T10:23:02.486 DEBUG:boto:Params: {} 2024-02-01T10:23:02.486 DEBUG:boto:Token: None 2024-02-01T10:23:02.487 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:23:02 GMT /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:02.487 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sYiLkxW5MYc3DGa+ESMDvxbtR6A= 2024-02-01T10:23:02.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sYiLkxW5MYc3DGa+ESMDvxbtR6A=', 'Content-Length': '0'} 2024-02-01T10:23:02.489 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000589110ead598865f-0065bb7106-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:02.490 DEBUG:boto:path=/vclfcn-18/obj-test-zone1 2024-02-01T10:23:02.490 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1 2024-02-01T10:23:02.490 DEBUG:boto:path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:02.490 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:02.490 DEBUG:boto:Method: HEAD 2024-02-01T10:23:02.490 DEBUG:boto:Path: /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:02.490 DEBUG:boto:Data: 2024-02-01T10:23:02.490 DEBUG:boto:Headers: {} 2024-02-01T10:23:02.490 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:23:02.491 DEBUG:boto:Port: 8000 2024-02-01T10:23:02.491 DEBUG:boto:Params: {} 2024-02-01T10:23:02.491 DEBUG:boto:Token: None 2024-02-01T10:23:02.491 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:23:02 GMT /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:02.491 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sYiLkxW5MYc3DGa+ESMDvxbtR6A= 2024-02-01T10:23:02.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sYiLkxW5MYc3DGa+ESMDvxbtR6A=', 'Content-Length': '0'} 2024-02-01T10:23:02.494 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000023c31fa1af977129-0065bb7106-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:02.494 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-02-01T10:23:02.494 DEBUG:boto:path=/vclfcn-18/obj-test-zone1 2024-02-01T10:23:02.494 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1 2024-02-01T10:23:02.494 DEBUG:boto:path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:02.494 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:02.494 DEBUG:boto:Method: GET 2024-02-01T10:23:02.494 DEBUG:boto:Path: /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:02.495 DEBUG:boto:Data: 2024-02-01T10:23:02.495 DEBUG:boto:Headers: {} 2024-02-01T10:23:02.495 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:02.495 DEBUG:boto:Port: 8000 2024-02-01T10:23:02.495 DEBUG:boto:Params: {} 2024-02-01T10:23:02.495 DEBUG:boto:Token: None 2024-02-01T10:23:02.495 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:02 GMT /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:02.495 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:clncxt2mCwBcBsQeeeqnnQniqzY= 2024-02-01T10:23:02.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:clncxt2mCwBcBsQeeeqnnQniqzY=', 'Content-Length': '0'} 2024-02-01T10:23:02.498 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000afff483a81c4571-0065bb7106-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:02.540 DEBUG:boto:path=/vclfcn-18/obj-test-zone1 2024-02-01T10:23:02.540 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1 2024-02-01T10:23:02.541 DEBUG:boto:path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:02.541 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:02.541 DEBUG:boto:Method: GET 2024-02-01T10:23:02.541 DEBUG:boto:Path: /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:02.541 DEBUG:boto:Data: 2024-02-01T10:23:02.541 DEBUG:boto:Headers: {} 2024-02-01T10:23:02.541 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:23:02.541 DEBUG:boto:Port: 8000 2024-02-01T10:23:02.541 DEBUG:boto:Params: {} 2024-02-01T10:23:02.541 DEBUG:boto:Token: None 2024-02-01T10:23:02.542 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:02 GMT /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:02.542 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:clncxt2mCwBcBsQeeeqnnQniqzY= 2024-02-01T10:23:02.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:clncxt2mCwBcBsQeeeqnnQniqzY=', 'Content-Length': '0'} 2024-02-01T10:23:02.545 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aa3ebee951d7262a-0065bb7106-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:02.545 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:23:02.545 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:23:02.545 DEBUG:boto:path=/vclfcn-18/?prefix=obj-test-zone1 2024-02-01T10:23:02.545 DEBUG:boto:auth_path=/vclfcn-18/?prefix=obj-test-zone1 2024-02-01T10:23:02.545 DEBUG:boto:Method: GET 2024-02-01T10:23:02.545 DEBUG:boto:Path: /vclfcn-18/?prefix=obj-test-zone1 2024-02-01T10:23:02.546 DEBUG:boto:Data: 2024-02-01T10:23:02.546 DEBUG:boto:Headers: {} 2024-02-01T10:23:02.546 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:02.546 DEBUG:boto:Port: 8000 2024-02-01T10:23:02.546 DEBUG:boto:Params: {} 2024-02-01T10:23:02.546 DEBUG:boto:Token: None 2024-02-01T10:23:02.546 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:02 GMT /vclfcn-18/ 2024-02-01T10:23:02.546 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zWlw87JanONUh75e5vJgAWli5Qk= 2024-02-01T10:23:02.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zWlw87JanONUh75e5vJgAWli5Qk=', 'Content-Length': '0'} 2024-02-01T10:23:02.551 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c67e6afc114cb9a-0065bb7106-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:02.592 DEBUG:boto:b'vclfcn-18obj-test-zone11000falseobj-test-zone12024-02-01T10:20:50.942Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:02.593 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:23:02.593 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:23:02.593 DEBUG:boto:path=/vclfcn-18/?prefix=obj-test-zone1 2024-02-01T10:23:02.593 DEBUG:boto:auth_path=/vclfcn-18/?prefix=obj-test-zone1 2024-02-01T10:23:02.593 DEBUG:boto:Method: GET 2024-02-01T10:23:02.593 DEBUG:boto:Path: /vclfcn-18/?prefix=obj-test-zone1 2024-02-01T10:23:02.593 DEBUG:boto:Data: 2024-02-01T10:23:02.593 DEBUG:boto:Headers: {} 2024-02-01T10:23:02.593 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:23:02.594 DEBUG:boto:Port: 8000 2024-02-01T10:23:02.594 DEBUG:boto:Params: {} 2024-02-01T10:23:02.594 DEBUG:boto:Token: None 2024-02-01T10:23:02.594 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:02 GMT /vclfcn-18/ 2024-02-01T10:23:02.594 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zWlw87JanONUh75e5vJgAWli5Qk= 2024-02-01T10:23:02.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zWlw87JanONUh75e5vJgAWli5Qk=', 'Content-Length': '0'} 2024-02-01T10:23:02.599 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bbc952264d3e4070-0065bb7106-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:02.599 DEBUG:boto:b'vclfcn-18obj-test-zone11000falseobj-test-zone12024-02-01T10:20:50.942Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:02.599 DEBUG:boto:path=/vclfcn-18/obj-test-zone1 2024-02-01T10:23:02.599 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1 2024-02-01T10:23:02.600 DEBUG:boto:Method: HEAD 2024-02-01T10:23:02.600 DEBUG:boto:Path: /vclfcn-18/obj-test-zone1 2024-02-01T10:23:02.600 DEBUG:boto:Data: 2024-02-01T10:23:02.600 DEBUG:boto:Headers: {} 2024-02-01T10:23:02.600 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:02.600 DEBUG:boto:Port: 8000 2024-02-01T10:23:02.600 DEBUG:boto:Params: {} 2024-02-01T10:23:02.600 DEBUG:boto:Token: None 2024-02-01T10:23:02.600 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:23:02 GMT /vclfcn-18/obj-test-zone1 2024-02-01T10:23:02.600 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vVUm63vReTN/bZYTeqVH1kDvStI= 2024-02-01T10:23:02.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vVUm63vReTN/bZYTeqVH1kDvStI=', 'Content-Length': '0'} 2024-02-01T10:23:02.604 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000126ac3e8f51d30e4-0065bb7106-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:02.605 DEBUG:boto:path=/vclfcn-18/obj-test-zone1 2024-02-01T10:23:02.605 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1 2024-02-01T10:23:02.605 DEBUG:boto:Method: HEAD 2024-02-01T10:23:02.605 DEBUG:boto:Path: /vclfcn-18/obj-test-zone1 2024-02-01T10:23:02.605 DEBUG:boto:Data: 2024-02-01T10:23:02.605 DEBUG:boto:Headers: {} 2024-02-01T10:23:02.605 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:23:02.605 DEBUG:boto:Port: 8000 2024-02-01T10:23:02.605 DEBUG:boto:Params: {} 2024-02-01T10:23:02.605 DEBUG:boto:Token: None 2024-02-01T10:23:02.606 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:23:02 GMT /vclfcn-18/obj-test-zone1 2024-02-01T10:23:02.606 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vVUm63vReTN/bZYTeqVH1kDvStI= 2024-02-01T10:23:02.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vVUm63vReTN/bZYTeqVH1kDvStI=', 'Content-Length': '0'} 2024-02-01T10:23:02.611 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a31afff176c944cd-0065bb7106-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:02.611 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-02-01T10:23:02.611 DEBUG:boto:path=/vclfcn-18/obj-test-zone1 2024-02-01T10:23:02.612 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1 2024-02-01T10:23:02.612 DEBUG:boto:path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:02.612 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:02.612 DEBUG:boto:Method: GET 2024-02-01T10:23:02.612 DEBUG:boto:Path: /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:02.612 DEBUG:boto:Data: 2024-02-01T10:23:02.612 DEBUG:boto:Headers: {} 2024-02-01T10:23:02.612 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:02.612 DEBUG:boto:Port: 8000 2024-02-01T10:23:02.612 DEBUG:boto:Params: {} 2024-02-01T10:23:02.613 DEBUG:boto:Token: None 2024-02-01T10:23:02.613 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:02 GMT /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:02.613 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:clncxt2mCwBcBsQeeeqnnQniqzY= 2024-02-01T10:23:02.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:clncxt2mCwBcBsQeeeqnnQniqzY=', 'Content-Length': '0'} 2024-02-01T10:23:02.616 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000088f0de1757484fa7-0065bb7106-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:02.656 DEBUG:boto:path=/vclfcn-18/obj-test-zone1 2024-02-01T10:23:02.656 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1 2024-02-01T10:23:02.656 DEBUG:boto:path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:02.657 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:02.657 DEBUG:boto:Method: GET 2024-02-01T10:23:02.657 DEBUG:boto:Path: /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:02.657 DEBUG:boto:Data: 2024-02-01T10:23:02.657 DEBUG:boto:Headers: {} 2024-02-01T10:23:02.657 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:23:02.657 DEBUG:boto:Port: 8000 2024-02-01T10:23:02.657 DEBUG:boto:Params: {} 2024-02-01T10:23:02.657 DEBUG:boto:Token: None 2024-02-01T10:23:02.658 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:02 GMT /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:02.658 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:clncxt2mCwBcBsQeeeqnnQniqzY= 2024-02-01T10:23:02.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:clncxt2mCwBcBsQeeeqnnQniqzY=', 'Content-Length': '0'} 2024-02-01T10:23:02.661 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000083133c3fbb663ad8-0065bb7106-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:02.661 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:23:02.662 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:23:02.662 DEBUG:boto:path=/vclfcn-18/?prefix=obj-test-zone1 2024-02-01T10:23:02.662 DEBUG:boto:auth_path=/vclfcn-18/?prefix=obj-test-zone1 2024-02-01T10:23:02.662 DEBUG:boto:Method: GET 2024-02-01T10:23:02.662 DEBUG:boto:Path: /vclfcn-18/?prefix=obj-test-zone1 2024-02-01T10:23:02.662 DEBUG:boto:Data: 2024-02-01T10:23:02.662 DEBUG:boto:Headers: {} 2024-02-01T10:23:02.662 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:02.662 DEBUG:boto:Port: 8000 2024-02-01T10:23:02.662 DEBUG:boto:Params: {} 2024-02-01T10:23:02.662 DEBUG:boto:Token: None 2024-02-01T10:23:02.663 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:02 GMT /vclfcn-18/ 2024-02-01T10:23:02.663 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zWlw87JanONUh75e5vJgAWli5Qk= 2024-02-01T10:23:02.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zWlw87JanONUh75e5vJgAWli5Qk=', 'Content-Length': '0'} 2024-02-01T10:23:02.666 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4d1ad5d50566050-0065bb7106-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:02.708 DEBUG:boto:b'vclfcn-18obj-test-zone11000falseobj-test-zone12024-02-01T10:20:50.942Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:02.709 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:23:02.709 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:23:02.709 DEBUG:boto:path=/vclfcn-18/?prefix=obj-test-zone1 2024-02-01T10:23:02.709 DEBUG:boto:auth_path=/vclfcn-18/?prefix=obj-test-zone1 2024-02-01T10:23:02.709 DEBUG:boto:Method: GET 2024-02-01T10:23:02.709 DEBUG:boto:Path: /vclfcn-18/?prefix=obj-test-zone1 2024-02-01T10:23:02.709 DEBUG:boto:Data: 2024-02-01T10:23:02.709 DEBUG:boto:Headers: {} 2024-02-01T10:23:02.710 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:23:02.710 DEBUG:boto:Port: 8000 2024-02-01T10:23:02.710 DEBUG:boto:Params: {} 2024-02-01T10:23:02.710 DEBUG:boto:Token: None 2024-02-01T10:23:02.710 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:02 GMT /vclfcn-18/ 2024-02-01T10:23:02.710 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zWlw87JanONUh75e5vJgAWli5Qk= 2024-02-01T10:23:02.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zWlw87JanONUh75e5vJgAWli5Qk=', 'Content-Length': '0'} 2024-02-01T10:23:02.715 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5fa7b2927704c4a-0065bb7106-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:02.716 DEBUG:boto:b'vclfcn-18obj-test-zone11000falseobj-test-zone12024-02-01T10:20:50.942Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:02.716 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-02-01T10:23:02.716 DEBUG:boto:path=/vclfcn-18/obj-test-zone2 2024-02-01T10:23:02.716 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2 2024-02-01T10:23:02.717 DEBUG:boto:path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:02.717 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:02.717 DEBUG:boto:Method: GET 2024-02-01T10:23:02.717 DEBUG:boto:Path: /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:02.717 DEBUG:boto:Data: 2024-02-01T10:23:02.717 DEBUG:boto:Headers: {} 2024-02-01T10:23:02.717 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:02.717 DEBUG:boto:Port: 8000 2024-02-01T10:23:02.717 DEBUG:boto:Params: {} 2024-02-01T10:23:02.717 DEBUG:boto:Token: None 2024-02-01T10:23:02.718 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:02 GMT /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:02.718 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cs18YrQleHKW1Zf5JBzmnIdGWzk= 2024-02-01T10:23:02.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cs18YrQleHKW1Zf5JBzmnIdGWzk=', 'Content-Length': '0'} 2024-02-01T10:23:02.720 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', 'ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000db15bfd67577a8b7-0065bb7106-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:02.721 DEBUG:boto:path=/vclfcn-18/obj-test-zone2 2024-02-01T10:23:02.721 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2 2024-02-01T10:23:02.721 DEBUG:boto:path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:02.721 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:02.721 DEBUG:boto:Method: GET 2024-02-01T10:23:02.721 DEBUG:boto:Path: /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:02.721 DEBUG:boto:Data: 2024-02-01T10:23:02.721 DEBUG:boto:Headers: {} 2024-02-01T10:23:02.722 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:23:02.722 DEBUG:boto:Port: 8000 2024-02-01T10:23:02.722 DEBUG:boto:Params: {} 2024-02-01T10:23:02.722 DEBUG:boto:Token: None 2024-02-01T10:23:02.722 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:02 GMT /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:02.722 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cs18YrQleHKW1Zf5JBzmnIdGWzk= 2024-02-01T10:23:02.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cs18YrQleHKW1Zf5JBzmnIdGWzk=', 'Content-Length': '0'} 2024-02-01T10:23:02.725 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', 'ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000a585c79b4dd33e1-0065bb7106-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:02.768 DEBUG:boto:path=/vclfcn-18/obj-test-zone2 2024-02-01T10:23:02.768 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2 2024-02-01T10:23:02.768 DEBUG:boto:path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:02.768 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:02.769 DEBUG:boto:Method: HEAD 2024-02-01T10:23:02.769 DEBUG:boto:Path: /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:02.769 DEBUG:boto:Data: 2024-02-01T10:23:02.769 DEBUG:boto:Headers: {} 2024-02-01T10:23:02.769 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:02.769 DEBUG:boto:Port: 8000 2024-02-01T10:23:02.769 DEBUG:boto:Params: {} 2024-02-01T10:23:02.769 DEBUG:boto:Token: None 2024-02-01T10:23:02.769 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:23:02 GMT /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:02.770 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wVIWEsnBSmpRC4LzUgjaYNNbkU4= 2024-02-01T10:23:02.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wVIWEsnBSmpRC4LzUgjaYNNbkU4=', 'Content-Length': '0'} 2024-02-01T10:23:02.773 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', 'ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000051a43a65a35c2c2d-0065bb7106-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:02.773 DEBUG:boto:path=/vclfcn-18/obj-test-zone2 2024-02-01T10:23:02.773 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2 2024-02-01T10:23:02.773 DEBUG:boto:path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:02.773 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:02.773 DEBUG:boto:Method: HEAD 2024-02-01T10:23:02.773 DEBUG:boto:Path: /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:02.773 DEBUG:boto:Data: 2024-02-01T10:23:02.774 DEBUG:boto:Headers: {} 2024-02-01T10:23:02.774 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:23:02.774 DEBUG:boto:Port: 8000 2024-02-01T10:23:02.774 DEBUG:boto:Params: {} 2024-02-01T10:23:02.774 DEBUG:boto:Token: None 2024-02-01T10:23:02.774 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:23:02 GMT /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:02.774 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wVIWEsnBSmpRC4LzUgjaYNNbkU4= 2024-02-01T10:23:02.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wVIWEsnBSmpRC4LzUgjaYNNbkU4=', 'Content-Length': '0'} 2024-02-01T10:23:02.777 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', 'ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000025f27b318d229b07-0065bb7106-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:02.778 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-02-01T10:23:02.778 DEBUG:boto:path=/vclfcn-18/obj-test-zone2 2024-02-01T10:23:02.778 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2 2024-02-01T10:23:02.778 DEBUG:boto:path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:02.778 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:02.778 DEBUG:boto:Method: GET 2024-02-01T10:23:02.778 DEBUG:boto:Path: /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:02.778 DEBUG:boto:Data: 2024-02-01T10:23:02.779 DEBUG:boto:Headers: {} 2024-02-01T10:23:02.779 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:02.779 DEBUG:boto:Port: 8000 2024-02-01T10:23:02.779 DEBUG:boto:Params: {} 2024-02-01T10:23:02.779 DEBUG:boto:Token: None 2024-02-01T10:23:02.779 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:02 GMT /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:02.779 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cs18YrQleHKW1Zf5JBzmnIdGWzk= 2024-02-01T10:23:02.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cs18YrQleHKW1Zf5JBzmnIdGWzk=', 'Content-Length': '0'} 2024-02-01T10:23:02.782 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', 'ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000075ea12d05d4be2f6-0065bb7106-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:02.783 DEBUG:boto:path=/vclfcn-18/obj-test-zone2 2024-02-01T10:23:02.783 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2 2024-02-01T10:23:02.783 DEBUG:boto:path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:02.783 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:02.783 DEBUG:boto:Method: GET 2024-02-01T10:23:02.783 DEBUG:boto:Path: /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:02.783 DEBUG:boto:Data: 2024-02-01T10:23:02.783 DEBUG:boto:Headers: {} 2024-02-01T10:23:02.783 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:23:02.783 DEBUG:boto:Port: 8000 2024-02-01T10:23:02.784 DEBUG:boto:Params: {} 2024-02-01T10:23:02.784 DEBUG:boto:Token: None 2024-02-01T10:23:02.784 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:02 GMT /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:02.784 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cs18YrQleHKW1Zf5JBzmnIdGWzk= 2024-02-01T10:23:02.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cs18YrQleHKW1Zf5JBzmnIdGWzk=', 'Content-Length': '0'} 2024-02-01T10:23:02.787 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', 'ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000098b00765b6adbda5-0065bb7106-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:02.828 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:23:02.828 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:23:02.828 DEBUG:boto:path=/vclfcn-18/?prefix=obj-test-zone2 2024-02-01T10:23:02.828 DEBUG:boto:auth_path=/vclfcn-18/?prefix=obj-test-zone2 2024-02-01T10:23:02.829 DEBUG:boto:Method: GET 2024-02-01T10:23:02.829 DEBUG:boto:Path: /vclfcn-18/?prefix=obj-test-zone2 2024-02-01T10:23:02.829 DEBUG:boto:Data: 2024-02-01T10:23:02.829 DEBUG:boto:Headers: {} 2024-02-01T10:23:02.829 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:02.829 DEBUG:boto:Port: 8000 2024-02-01T10:23:02.829 DEBUG:boto:Params: {} 2024-02-01T10:23:02.829 DEBUG:boto:Token: None 2024-02-01T10:23:02.829 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:02 GMT /vclfcn-18/ 2024-02-01T10:23:02.830 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zWlw87JanONUh75e5vJgAWli5Qk= 2024-02-01T10:23:02.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zWlw87JanONUh75e5vJgAWli5Qk=', 'Content-Length': '0'} 2024-02-01T10:23:02.834 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f93774236a7f32c-0065bb7106-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:02.835 DEBUG:boto:b'vclfcn-18obj-test-zone21000falseobj-test-zone22024-02-01T10:20:51.094Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:02.835 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:23:02.835 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:23:02.835 DEBUG:boto:path=/vclfcn-18/?prefix=obj-test-zone2 2024-02-01T10:23:02.835 DEBUG:boto:auth_path=/vclfcn-18/?prefix=obj-test-zone2 2024-02-01T10:23:02.835 DEBUG:boto:Method: GET 2024-02-01T10:23:02.836 DEBUG:boto:Path: /vclfcn-18/?prefix=obj-test-zone2 2024-02-01T10:23:02.836 DEBUG:boto:Data: 2024-02-01T10:23:02.836 DEBUG:boto:Headers: {} 2024-02-01T10:23:02.836 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:23:02.836 DEBUG:boto:Port: 8000 2024-02-01T10:23:02.836 DEBUG:boto:Params: {} 2024-02-01T10:23:02.836 DEBUG:boto:Token: None 2024-02-01T10:23:02.836 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:02 GMT /vclfcn-18/ 2024-02-01T10:23:02.836 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zWlw87JanONUh75e5vJgAWli5Qk= 2024-02-01T10:23:02.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zWlw87JanONUh75e5vJgAWli5Qk=', 'Content-Length': '0'} 2024-02-01T10:23:02.840 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9e45532f39c7532-0065bb7106-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:02.841 DEBUG:boto:b'vclfcn-18obj-test-zone21000falseobj-test-zone22024-02-01T10:20:51.094Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:02.841 DEBUG:boto:path=/vclfcn-18/obj-test-zone2 2024-02-01T10:23:02.841 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2 2024-02-01T10:23:02.841 DEBUG:boto:Method: HEAD 2024-02-01T10:23:02.841 DEBUG:boto:Path: /vclfcn-18/obj-test-zone2 2024-02-01T10:23:02.841 DEBUG:boto:Data: 2024-02-01T10:23:02.841 DEBUG:boto:Headers: {} 2024-02-01T10:23:02.842 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:02.842 DEBUG:boto:Port: 8000 2024-02-01T10:23:02.842 DEBUG:boto:Params: {} 2024-02-01T10:23:02.842 DEBUG:boto:Token: None 2024-02-01T10:23:02.842 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:23:02 GMT /vclfcn-18/obj-test-zone2 2024-02-01T10:23:02.842 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:t0PUDTnjxLOcnjFQYY0brJIwKkE= 2024-02-01T10:23:02.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:t0PUDTnjxLOcnjFQYY0brJIwKkE=', 'Content-Length': '0'} 2024-02-01T10:23:02.845 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', 'ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c34d57be88453162-0065bb7106-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:02.846 DEBUG:boto:path=/vclfcn-18/obj-test-zone2 2024-02-01T10:23:02.846 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2 2024-02-01T10:23:02.846 DEBUG:boto:Method: HEAD 2024-02-01T10:23:02.846 DEBUG:boto:Path: /vclfcn-18/obj-test-zone2 2024-02-01T10:23:02.846 DEBUG:boto:Data: 2024-02-01T10:23:02.846 DEBUG:boto:Headers: {} 2024-02-01T10:23:02.846 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:23:02.846 DEBUG:boto:Port: 8000 2024-02-01T10:23:02.847 DEBUG:boto:Params: {} 2024-02-01T10:23:02.847 DEBUG:boto:Token: None 2024-02-01T10:23:02.847 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:23:02 GMT /vclfcn-18/obj-test-zone2 2024-02-01T10:23:02.847 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:t0PUDTnjxLOcnjFQYY0brJIwKkE= 2024-02-01T10:23:02.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:t0PUDTnjxLOcnjFQYY0brJIwKkE=', 'Content-Length': '0'} 2024-02-01T10:23:02.851 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', 'ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c01164176b420925-0065bb7106-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:02.851 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-02-01T10:23:02.852 DEBUG:boto:path=/vclfcn-18/obj-test-zone2 2024-02-01T10:23:02.852 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2 2024-02-01T10:23:02.852 DEBUG:boto:path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:02.852 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:02.852 DEBUG:boto:Method: GET 2024-02-01T10:23:02.852 DEBUG:boto:Path: /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:02.852 DEBUG:boto:Data: 2024-02-01T10:23:02.852 DEBUG:boto:Headers: {} 2024-02-01T10:23:02.853 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:02.853 DEBUG:boto:Port: 8000 2024-02-01T10:23:02.853 DEBUG:boto:Params: {} 2024-02-01T10:23:02.853 DEBUG:boto:Token: None 2024-02-01T10:23:02.853 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:02 GMT /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:02.853 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cs18YrQleHKW1Zf5JBzmnIdGWzk= 2024-02-01T10:23:02.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cs18YrQleHKW1Zf5JBzmnIdGWzk=', 'Content-Length': '0'} 2024-02-01T10:23:02.856 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', 'ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000433ed311962c4321-0065bb7106-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:02.900 DEBUG:boto:path=/vclfcn-18/obj-test-zone2 2024-02-01T10:23:02.900 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2 2024-02-01T10:23:02.900 DEBUG:boto:path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:02.900 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:02.901 DEBUG:boto:Method: GET 2024-02-01T10:23:02.901 DEBUG:boto:Path: /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:02.901 DEBUG:boto:Data: 2024-02-01T10:23:02.901 DEBUG:boto:Headers: {} 2024-02-01T10:23:02.901 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:23:02.901 DEBUG:boto:Port: 8000 2024-02-01T10:23:02.901 DEBUG:boto:Params: {} 2024-02-01T10:23:02.901 DEBUG:boto:Token: None 2024-02-01T10:23:02.901 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:02 GMT /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:02.902 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cs18YrQleHKW1Zf5JBzmnIdGWzk= 2024-02-01T10:23:02.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cs18YrQleHKW1Zf5JBzmnIdGWzk=', 'Content-Length': '0'} 2024-02-01T10:23:02.905 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', 'ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000efd6ef17507deb71-0065bb7106-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:02.948 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:23:02.948 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:23:02.948 DEBUG:boto:path=/vclfcn-18/?prefix=obj-test-zone2 2024-02-01T10:23:02.949 DEBUG:boto:auth_path=/vclfcn-18/?prefix=obj-test-zone2 2024-02-01T10:23:02.949 DEBUG:boto:Method: GET 2024-02-01T10:23:02.949 DEBUG:boto:Path: /vclfcn-18/?prefix=obj-test-zone2 2024-02-01T10:23:02.949 DEBUG:boto:Data: 2024-02-01T10:23:02.949 DEBUG:boto:Headers: {} 2024-02-01T10:23:02.949 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:02.949 DEBUG:boto:Port: 8000 2024-02-01T10:23:02.949 DEBUG:boto:Params: {} 2024-02-01T10:23:02.949 DEBUG:boto:Token: None 2024-02-01T10:23:02.949 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:02 GMT /vclfcn-18/ 2024-02-01T10:23:02.950 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zWlw87JanONUh75e5vJgAWli5Qk= 2024-02-01T10:23:02.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zWlw87JanONUh75e5vJgAWli5Qk=', 'Content-Length': '0'} 2024-02-01T10:23:02.954 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008dd10422738b367b-0065bb7106-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:02.954 DEBUG:boto:b'vclfcn-18obj-test-zone21000falseobj-test-zone22024-02-01T10:20:51.094Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:02.954 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:23:02.955 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:23:02.955 DEBUG:boto:path=/vclfcn-18/?prefix=obj-test-zone2 2024-02-01T10:23:02.955 DEBUG:boto:auth_path=/vclfcn-18/?prefix=obj-test-zone2 2024-02-01T10:23:02.955 DEBUG:boto:Method: GET 2024-02-01T10:23:02.955 DEBUG:boto:Path: /vclfcn-18/?prefix=obj-test-zone2 2024-02-01T10:23:02.955 DEBUG:boto:Data: 2024-02-01T10:23:02.955 DEBUG:boto:Headers: {} 2024-02-01T10:23:02.956 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:23:02.956 DEBUG:boto:Port: 8000 2024-02-01T10:23:02.956 DEBUG:boto:Params: {} 2024-02-01T10:23:02.956 DEBUG:boto:Token: None 2024-02-01T10:23:02.956 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:02 GMT /vclfcn-18/ 2024-02-01T10:23:02.956 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zWlw87JanONUh75e5vJgAWli5Qk= 2024-02-01T10:23:02.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zWlw87JanONUh75e5vJgAWli5Qk=', 'Content-Length': '0'} 2024-02-01T10:23:02.960 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001bd261f932f35ccf-0065bb7106-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:02.961 DEBUG:boto:b'vclfcn-18obj-test-zone21000falseobj-test-zone22024-02-01T10:20:51.094Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:02.961 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-02-01T10:23:02.961 DEBUG:boto:path=/vclfcn-18/obj-test-zone3 2024-02-01T10:23:02.961 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3 2024-02-01T10:23:02.961 DEBUG:boto:path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:02.961 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:02.962 DEBUG:boto:Method: GET 2024-02-01T10:23:02.962 DEBUG:boto:Path: /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:02.962 DEBUG:boto:Data: 2024-02-01T10:23:02.962 DEBUG:boto:Headers: {} 2024-02-01T10:23:02.962 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:02.962 DEBUG:boto:Port: 8000 2024-02-01T10:23:02.962 DEBUG:boto:Params: {} 2024-02-01T10:23:02.962 DEBUG:boto:Token: None 2024-02-01T10:23:02.962 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:02 GMT /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:02.962 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:40dls3ChurO7xHH4V1f0D61Kj1E= 2024-02-01T10:23:02.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:40dls3ChurO7xHH4V1f0D61Kj1E=', 'Content-Length': '0'} 2024-02-01T10:23:02.965 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', '6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c3a6faa3808976f0-0065bb7106-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:03.008 DEBUG:boto:path=/vclfcn-18/obj-test-zone3 2024-02-01T10:23:03.008 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3 2024-02-01T10:23:03.008 DEBUG:boto:path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:03.008 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:03.009 DEBUG:boto:Method: GET 2024-02-01T10:23:03.009 DEBUG:boto:Path: /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:03.009 DEBUG:boto:Data: 2024-02-01T10:23:03.009 DEBUG:boto:Headers: {} 2024-02-01T10:23:03.009 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:23:03.009 DEBUG:boto:Port: 8000 2024-02-01T10:23:03.009 DEBUG:boto:Params: {} 2024-02-01T10:23:03.009 DEBUG:boto:Token: None 2024-02-01T10:23:03.009 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:03 GMT /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:03.009 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:XoPf7AGMZ5UKpo58Tbui0GlWCFY= 2024-02-01T10:23:03.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:XoPf7AGMZ5UKpo58Tbui0GlWCFY=', 'Content-Length': '0'} 2024-02-01T10:23:03.013 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', '6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ea9bbe000342aec8-0065bb7107-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:03.013 DEBUG:boto:path=/vclfcn-18/obj-test-zone3 2024-02-01T10:23:03.013 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3 2024-02-01T10:23:03.013 DEBUG:boto:path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:03.013 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:03.014 DEBUG:boto:Method: HEAD 2024-02-01T10:23:03.014 DEBUG:boto:Path: /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:03.014 DEBUG:boto:Data: 2024-02-01T10:23:03.014 DEBUG:boto:Headers: {} 2024-02-01T10:23:03.014 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:03.014 DEBUG:boto:Port: 8000 2024-02-01T10:23:03.014 DEBUG:boto:Params: {} 2024-02-01T10:23:03.014 DEBUG:boto:Token: None 2024-02-01T10:23:03.015 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:23:03 GMT /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:03.015 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:KwkW77WFGiuaQOLJ56AKzxNR8os= 2024-02-01T10:23:03.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:KwkW77WFGiuaQOLJ56AKzxNR8os=', 'Content-Length': '0'} 2024-02-01T10:23:03.018 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', '6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ae0a21f77e5237c0-0065bb7107-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:03.018 DEBUG:boto:path=/vclfcn-18/obj-test-zone3 2024-02-01T10:23:03.018 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3 2024-02-01T10:23:03.018 DEBUG:boto:path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:03.018 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:03.019 DEBUG:boto:Method: HEAD 2024-02-01T10:23:03.019 DEBUG:boto:Path: /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:03.019 DEBUG:boto:Data: 2024-02-01T10:23:03.019 DEBUG:boto:Headers: {} 2024-02-01T10:23:03.019 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:23:03.019 DEBUG:boto:Port: 8000 2024-02-01T10:23:03.019 DEBUG:boto:Params: {} 2024-02-01T10:23:03.019 DEBUG:boto:Token: None 2024-02-01T10:23:03.020 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:23:03 GMT /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:03.020 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:KwkW77WFGiuaQOLJ56AKzxNR8os= 2024-02-01T10:23:03.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:KwkW77WFGiuaQOLJ56AKzxNR8os=', 'Content-Length': '0'} 2024-02-01T10:23:03.023 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', '6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000079982985833d160e-0065bb7107-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:03.023 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-02-01T10:23:03.023 DEBUG:boto:path=/vclfcn-18/obj-test-zone3 2024-02-01T10:23:03.023 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3 2024-02-01T10:23:03.023 DEBUG:boto:path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:03.023 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:03.023 DEBUG:boto:Method: GET 2024-02-01T10:23:03.024 DEBUG:boto:Path: /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:03.024 DEBUG:boto:Data: 2024-02-01T10:23:03.024 DEBUG:boto:Headers: {} 2024-02-01T10:23:03.024 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:03.024 DEBUG:boto:Port: 8000 2024-02-01T10:23:03.024 DEBUG:boto:Params: {} 2024-02-01T10:23:03.024 DEBUG:boto:Token: None 2024-02-01T10:23:03.024 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:03 GMT /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:03.025 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:XoPf7AGMZ5UKpo58Tbui0GlWCFY= 2024-02-01T10:23:03.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:XoPf7AGMZ5UKpo58Tbui0GlWCFY=', 'Content-Length': '0'} 2024-02-01T10:23:03.028 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', '6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006ce15281465b9436-0065bb7107-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:03.068 DEBUG:boto:path=/vclfcn-18/obj-test-zone3 2024-02-01T10:23:03.068 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3 2024-02-01T10:23:03.068 DEBUG:boto:path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:03.069 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:03.069 DEBUG:boto:Method: GET 2024-02-01T10:23:03.069 DEBUG:boto:Path: /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:03.069 DEBUG:boto:Data: 2024-02-01T10:23:03.069 DEBUG:boto:Headers: {} 2024-02-01T10:23:03.069 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:23:03.069 DEBUG:boto:Port: 8000 2024-02-01T10:23:03.069 DEBUG:boto:Params: {} 2024-02-01T10:23:03.069 DEBUG:boto:Token: None 2024-02-01T10:23:03.069 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:03 GMT /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:03.070 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:XoPf7AGMZ5UKpo58Tbui0GlWCFY= 2024-02-01T10:23:03.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:XoPf7AGMZ5UKpo58Tbui0GlWCFY=', 'Content-Length': '0'} 2024-02-01T10:23:03.073 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', '6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005a871362c9f8a712-0065bb7107-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:03.116 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:23:03.116 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:23:03.116 DEBUG:boto:path=/vclfcn-18/?prefix=obj-test-zone3 2024-02-01T10:23:03.117 DEBUG:boto:auth_path=/vclfcn-18/?prefix=obj-test-zone3 2024-02-01T10:23:03.117 DEBUG:boto:Method: GET 2024-02-01T10:23:03.117 DEBUG:boto:Path: /vclfcn-18/?prefix=obj-test-zone3 2024-02-01T10:23:03.117 DEBUG:boto:Data: 2024-02-01T10:23:03.117 DEBUG:boto:Headers: {} 2024-02-01T10:23:03.117 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:03.117 DEBUG:boto:Port: 8000 2024-02-01T10:23:03.117 DEBUG:boto:Params: {} 2024-02-01T10:23:03.117 DEBUG:boto:Token: None 2024-02-01T10:23:03.117 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:03 GMT /vclfcn-18/ 2024-02-01T10:23:03.118 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HoVGoRezg9W8je4qz8QiLl1tVKo= 2024-02-01T10:23:03.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HoVGoRezg9W8je4qz8QiLl1tVKo=', 'Content-Length': '0'} 2024-02-01T10:23:03.122 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065e623a138c3604d-0065bb7107-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:03.122 DEBUG:boto:b'vclfcn-18obj-test-zone31000falseobj-test-zone32024-02-01T10:20:51.248Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:03.123 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:23:03.123 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:23:03.123 DEBUG:boto:path=/vclfcn-18/?prefix=obj-test-zone3 2024-02-01T10:23:03.123 DEBUG:boto:auth_path=/vclfcn-18/?prefix=obj-test-zone3 2024-02-01T10:23:03.123 DEBUG:boto:Method: GET 2024-02-01T10:23:03.123 DEBUG:boto:Path: /vclfcn-18/?prefix=obj-test-zone3 2024-02-01T10:23:03.123 DEBUG:boto:Data: 2024-02-01T10:23:03.123 DEBUG:boto:Headers: {} 2024-02-01T10:23:03.123 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:23:03.123 DEBUG:boto:Port: 8000 2024-02-01T10:23:03.123 DEBUG:boto:Params: {} 2024-02-01T10:23:03.123 DEBUG:boto:Token: None 2024-02-01T10:23:03.124 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:03 GMT /vclfcn-18/ 2024-02-01T10:23:03.124 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HoVGoRezg9W8je4qz8QiLl1tVKo= 2024-02-01T10:23:03.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HoVGoRezg9W8je4qz8QiLl1tVKo=', 'Content-Length': '0'} 2024-02-01T10:23:03.128 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1276f3ae3541af5-0065bb7107-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:03.176 DEBUG:boto:b'vclfcn-18obj-test-zone31000falseobj-test-zone32024-02-01T10:20:51.248Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:03.176 DEBUG:boto:path=/vclfcn-18/obj-test-zone3 2024-02-01T10:23:03.177 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3 2024-02-01T10:23:03.177 DEBUG:boto:Method: HEAD 2024-02-01T10:23:03.177 DEBUG:boto:Path: /vclfcn-18/obj-test-zone3 2024-02-01T10:23:03.177 DEBUG:boto:Data: 2024-02-01T10:23:03.177 DEBUG:boto:Headers: {} 2024-02-01T10:23:03.177 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:03.177 DEBUG:boto:Port: 8000 2024-02-01T10:23:03.177 DEBUG:boto:Params: {} 2024-02-01T10:23:03.177 DEBUG:boto:Token: None 2024-02-01T10:23:03.178 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:23:03 GMT /vclfcn-18/obj-test-zone3 2024-02-01T10:23:03.178 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QeHTRolTBlppP0ro8xeBYrmvlFA= 2024-02-01T10:23:03.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QeHTRolTBlppP0ro8xeBYrmvlFA=', 'Content-Length': '0'} 2024-02-01T10:23:03.181 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', '6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001573e9ee84370218-0065bb7107-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:03.182 DEBUG:boto:path=/vclfcn-18/obj-test-zone3 2024-02-01T10:23:03.182 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3 2024-02-01T10:23:03.182 DEBUG:boto:Method: HEAD 2024-02-01T10:23:03.182 DEBUG:boto:Path: /vclfcn-18/obj-test-zone3 2024-02-01T10:23:03.182 DEBUG:boto:Data: 2024-02-01T10:23:03.182 DEBUG:boto:Headers: {} 2024-02-01T10:23:03.182 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:23:03.182 DEBUG:boto:Port: 8000 2024-02-01T10:23:03.182 DEBUG:boto:Params: {} 2024-02-01T10:23:03.183 DEBUG:boto:Token: None 2024-02-01T10:23:03.183 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:23:03 GMT /vclfcn-18/obj-test-zone3 2024-02-01T10:23:03.183 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QeHTRolTBlppP0ro8xeBYrmvlFA= 2024-02-01T10:23:03.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QeHTRolTBlppP0ro8xeBYrmvlFA=', 'Content-Length': '0'} 2024-02-01T10:23:03.187 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', '6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b12024b2a614b5f8-0065bb7107-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:03.187 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-02-01T10:23:03.187 DEBUG:boto:path=/vclfcn-18/obj-test-zone3 2024-02-01T10:23:03.187 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3 2024-02-01T10:23:03.187 DEBUG:boto:path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:03.187 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:03.187 DEBUG:boto:Method: GET 2024-02-01T10:23:03.187 DEBUG:boto:Path: /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:03.188 DEBUG:boto:Data: 2024-02-01T10:23:03.188 DEBUG:boto:Headers: {} 2024-02-01T10:23:03.188 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:03.188 DEBUG:boto:Port: 8000 2024-02-01T10:23:03.188 DEBUG:boto:Params: {} 2024-02-01T10:23:03.188 DEBUG:boto:Token: None 2024-02-01T10:23:03.188 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:03 GMT /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:03.188 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:XoPf7AGMZ5UKpo58Tbui0GlWCFY= 2024-02-01T10:23:03.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:XoPf7AGMZ5UKpo58Tbui0GlWCFY=', 'Content-Length': '0'} 2024-02-01T10:23:03.192 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', '6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001461a6d1dc980542-0065bb7107-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:03.236 DEBUG:boto:path=/vclfcn-18/obj-test-zone3 2024-02-01T10:23:03.236 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3 2024-02-01T10:23:03.236 DEBUG:boto:path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:03.237 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:03.237 DEBUG:boto:Method: GET 2024-02-01T10:23:03.237 DEBUG:boto:Path: /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:03.237 DEBUG:boto:Data: 2024-02-01T10:23:03.237 DEBUG:boto:Headers: {} 2024-02-01T10:23:03.237 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:23:03.237 DEBUG:boto:Port: 8000 2024-02-01T10:23:03.237 DEBUG:boto:Params: {} 2024-02-01T10:23:03.237 DEBUG:boto:Token: None 2024-02-01T10:23:03.237 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:03 GMT /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:03.238 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:XoPf7AGMZ5UKpo58Tbui0GlWCFY= 2024-02-01T10:23:03.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:XoPf7AGMZ5UKpo58Tbui0GlWCFY=', 'Content-Length': '0'} 2024-02-01T10:23:03.240 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', '6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d9f38f75a2d44a29-0065bb7107-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:03.284 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:23:03.285 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:23:03.285 DEBUG:boto:path=/vclfcn-18/?prefix=obj-test-zone3 2024-02-01T10:23:03.285 DEBUG:boto:auth_path=/vclfcn-18/?prefix=obj-test-zone3 2024-02-01T10:23:03.285 DEBUG:boto:Method: GET 2024-02-01T10:23:03.285 DEBUG:boto:Path: /vclfcn-18/?prefix=obj-test-zone3 2024-02-01T10:23:03.285 DEBUG:boto:Data: 2024-02-01T10:23:03.285 DEBUG:boto:Headers: {} 2024-02-01T10:23:03.285 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:03.285 DEBUG:boto:Port: 8000 2024-02-01T10:23:03.285 DEBUG:boto:Params: {} 2024-02-01T10:23:03.286 DEBUG:boto:Token: None 2024-02-01T10:23:03.286 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:03 GMT /vclfcn-18/ 2024-02-01T10:23:03.286 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HoVGoRezg9W8je4qz8QiLl1tVKo= 2024-02-01T10:23:03.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HoVGoRezg9W8je4qz8QiLl1tVKo=', 'Content-Length': '0'} 2024-02-01T10:23:03.290 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087247cc2bc65ede2-0065bb7107-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:03.291 DEBUG:boto:b'vclfcn-18obj-test-zone31000falseobj-test-zone32024-02-01T10:20:51.248Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:03.291 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:23:03.292 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:23:03.292 DEBUG:boto:path=/vclfcn-18/?prefix=obj-test-zone3 2024-02-01T10:23:03.292 DEBUG:boto:auth_path=/vclfcn-18/?prefix=obj-test-zone3 2024-02-01T10:23:03.292 DEBUG:boto:Method: GET 2024-02-01T10:23:03.292 DEBUG:boto:Path: /vclfcn-18/?prefix=obj-test-zone3 2024-02-01T10:23:03.292 DEBUG:boto:Data: 2024-02-01T10:23:03.292 DEBUG:boto:Headers: {} 2024-02-01T10:23:03.292 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:23:03.292 DEBUG:boto:Port: 8000 2024-02-01T10:23:03.292 DEBUG:boto:Params: {} 2024-02-01T10:23:03.292 DEBUG:boto:Token: None 2024-02-01T10:23:03.293 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:03 GMT /vclfcn-18/ 2024-02-01T10:23:03.293 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HoVGoRezg9W8je4qz8QiLl1tVKo= 2024-02-01T10:23:03.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HoVGoRezg9W8je4qz8QiLl1tVKo=', 'Content-Length': '0'} 2024-02-01T10:23:03.297 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000045b616905432978-0065bb7107-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:03.344 DEBUG:boto:b'vclfcn-18obj-test-zone31000falseobj-test-zone32024-02-01T10:20:51.248Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:03.345 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-18 zones={test-zone2, test-zone1} 2024-02-01T10:23:03.345 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-18 zones={test-zone3, test-zone1} 2024-02-01T10:23:03.345 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:23:03.345 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:23:03.345 DEBUG:boto:Method: HEAD 2024-02-01T10:23:03.345 DEBUG:boto:Path: /vclfcn-18/ 2024-02-01T10:23:03.346 DEBUG:boto:Data: 2024-02-01T10:23:03.346 DEBUG:boto:Headers: {} 2024-02-01T10:23:03.346 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:03.346 DEBUG:boto:Port: 8001 2024-02-01T10:23:03.346 DEBUG:boto:Params: {} 2024-02-01T10:23:03.346 DEBUG:boto:Token: None 2024-02-01T10:23:03.346 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:23:03 GMT /vclfcn-18/ 2024-02-01T10:23:03.346 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8FJKq15FXQxw9QMIC3RNfCXzzwg= 2024-02-01T10:23:03.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8FJKq15FXQxw9QMIC3RNfCXzzwg=', 'Content-Length': '0'} 2024-02-01T10:23:03.351 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000067da429cb52f0981-0065bb7107-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:03.351 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:23:03.351 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:23:03.351 DEBUG:boto:Method: HEAD 2024-02-01T10:23:03.351 DEBUG:boto:Path: /vclfcn-18/ 2024-02-01T10:23:03.351 DEBUG:boto:Data: 2024-02-01T10:23:03.351 DEBUG:boto:Headers: {} 2024-02-01T10:23:03.351 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:23:03.352 DEBUG:boto:Port: 8000 2024-02-01T10:23:03.352 DEBUG:boto:Params: {} 2024-02-01T10:23:03.352 DEBUG:boto:Token: None 2024-02-01T10:23:03.352 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:23:03 GMT /vclfcn-18/ 2024-02-01T10:23:03.352 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8FJKq15FXQxw9QMIC3RNfCXzzwg= 2024-02-01T10:23:03.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8FJKq15FXQxw9QMIC3RNfCXzzwg=', 'Content-Length': '0'} 2024-02-01T10:23:03.356 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a1e1cba6567d8338-0065bb7107-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:03.357 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:23:03.357 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:23:03.357 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:23:03.357 DEBUG:boto:path=/vclfcn-18/?versions&max-keys=999 2024-02-01T10:23:03.357 DEBUG:boto:auth_path=/vclfcn-18/?versions&max-keys=999 2024-02-01T10:23:03.357 DEBUG:boto:Method: GET 2024-02-01T10:23:03.358 DEBUG:boto:Path: /vclfcn-18/?versions&max-keys=999 2024-02-01T10:23:03.358 DEBUG:boto:Data: 2024-02-01T10:23:03.358 DEBUG:boto:Headers: {} 2024-02-01T10:23:03.358 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:03.358 DEBUG:boto:Port: 8001 2024-02-01T10:23:03.358 DEBUG:boto:Params: {} 2024-02-01T10:23:03.358 DEBUG:boto:Token: None 2024-02-01T10:23:03.358 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:03 GMT /vclfcn-18/?versions 2024-02-01T10:23:03.358 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:l2YrQuoGa2+2DbF4ZRPYmf2jJj8= 2024-02-01T10:23:03.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:l2YrQuoGa2+2DbF4ZRPYmf2jJj8=', 'Content-Length': '0'} 2024-02-01T10:23:03.363 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000003d8abe0fc9617a2-0065bb7107-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:03.363 DEBUG:boto:b'vclfcn-18999falsedummytxgLlJWZVbWbjqLWqkOtt87dUoD4zuntrue2024-02-01T10:20:34.645Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcKtrue2024-02-01T10:20:50.942Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutDtrue2024-02-01T10:20:51.094Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone36pAVLbw3HAwE0jxxbuxTIxwcA2p4q8Ntrue2024-02-01T10:20:51.248Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:03.364 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-02-01T10:23:03.364 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-02-01T10:23:03.364 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-02-01T10:23:03.364 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-02-01T10:23:03.364 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:23:03.365 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:23:03.365 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:23:03.365 DEBUG:boto:path=/vclfcn-18/?versions&max-keys=999 2024-02-01T10:23:03.365 DEBUG:boto:auth_path=/vclfcn-18/?versions&max-keys=999 2024-02-01T10:23:03.365 DEBUG:boto:Method: GET 2024-02-01T10:23:03.365 DEBUG:boto:Path: /vclfcn-18/?versions&max-keys=999 2024-02-01T10:23:03.365 DEBUG:boto:Data: 2024-02-01T10:23:03.365 DEBUG:boto:Headers: {} 2024-02-01T10:23:03.365 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:23:03.365 DEBUG:boto:Port: 8000 2024-02-01T10:23:03.365 DEBUG:boto:Params: {} 2024-02-01T10:23:03.366 DEBUG:boto:Token: None 2024-02-01T10:23:03.366 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:03 GMT /vclfcn-18/?versions 2024-02-01T10:23:03.366 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:l2YrQuoGa2+2DbF4ZRPYmf2jJj8= 2024-02-01T10:23:03.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:l2YrQuoGa2+2DbF4ZRPYmf2jJj8=', 'Content-Length': '0'} 2024-02-01T10:23:03.371 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc78306a7b885363-0065bb7107-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:03.371 DEBUG:boto:b'vclfcn-18999falsedummytxgLlJWZVbWbjqLWqkOtt87dUoD4zuntrue2024-02-01T10:20:34.645Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcKtrue2024-02-01T10:20:50.942Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutDtrue2024-02-01T10:20:51.094Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone36pAVLbw3HAwE0jxxbuxTIxwcA2p4q8Ntrue2024-02-01T10:20:51.248Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:03.371 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-02-01T10:23:03.371 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-02-01T10:23:03.371 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-02-01T10:23:03.371 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-02-01T10:23:03.372 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:23:03.372 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:23:03.372 DEBUG:boto:path=/vclfcn-18/?versions&max-keys=999 2024-02-01T10:23:03.372 DEBUG:boto:auth_path=/vclfcn-18/?versions&max-keys=999 2024-02-01T10:23:03.372 DEBUG:boto:Method: GET 2024-02-01T10:23:03.372 DEBUG:boto:Path: /vclfcn-18/?versions&max-keys=999 2024-02-01T10:23:03.372 DEBUG:boto:Data: 2024-02-01T10:23:03.372 DEBUG:boto:Headers: {} 2024-02-01T10:23:03.372 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:03.372 DEBUG:boto:Port: 8001 2024-02-01T10:23:03.373 DEBUG:boto:Params: {} 2024-02-01T10:23:03.373 DEBUG:boto:Token: None 2024-02-01T10:23:03.373 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:03 GMT /vclfcn-18/?versions 2024-02-01T10:23:03.373 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:l2YrQuoGa2+2DbF4ZRPYmf2jJj8= 2024-02-01T10:23:03.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:l2YrQuoGa2+2DbF4ZRPYmf2jJj8=', 'Content-Length': '0'} 2024-02-01T10:23:03.377 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f7f31d29144767c-0065bb7107-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:03.377 DEBUG:boto:b'vclfcn-18999falsedummytxgLlJWZVbWbjqLWqkOtt87dUoD4zuntrue2024-02-01T10:20:34.645Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcKtrue2024-02-01T10:20:50.942Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutDtrue2024-02-01T10:20:51.094Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone36pAVLbw3HAwE0jxxbuxTIxwcA2p4q8Ntrue2024-02-01T10:20:51.248Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:03.378 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:23:03.378 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:23:03.378 DEBUG:boto:path=/vclfcn-18/?versions&max-keys=999 2024-02-01T10:23:03.378 DEBUG:boto:auth_path=/vclfcn-18/?versions&max-keys=999 2024-02-01T10:23:03.378 DEBUG:boto:Method: GET 2024-02-01T10:23:03.378 DEBUG:boto:Path: /vclfcn-18/?versions&max-keys=999 2024-02-01T10:23:03.378 DEBUG:boto:Data: 2024-02-01T10:23:03.378 DEBUG:boto:Headers: {} 2024-02-01T10:23:03.378 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:23:03.378 DEBUG:boto:Port: 8000 2024-02-01T10:23:03.378 DEBUG:boto:Params: {} 2024-02-01T10:23:03.379 DEBUG:boto:Token: None 2024-02-01T10:23:03.379 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:03 GMT /vclfcn-18/?versions 2024-02-01T10:23:03.379 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:l2YrQuoGa2+2DbF4ZRPYmf2jJj8= 2024-02-01T10:23:03.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:l2YrQuoGa2+2DbF4ZRPYmf2jJj8=', 'Content-Length': '0'} 2024-02-01T10:23:03.383 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000562a29677b614d65-0065bb7107-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:03.383 DEBUG:boto:b'vclfcn-18999falsedummytxgLlJWZVbWbjqLWqkOtt87dUoD4zuntrue2024-02-01T10:20:34.645Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcKtrue2024-02-01T10:20:50.942Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutDtrue2024-02-01T10:20:51.094Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone36pAVLbw3HAwE0jxxbuxTIxwcA2p4q8Ntrue2024-02-01T10:20:51.248Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:03.384 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-02-01T10:23:03.384 DEBUG:boto:path=/vclfcn-18/dummy 2024-02-01T10:23:03.384 DEBUG:boto:auth_path=/vclfcn-18/dummy 2024-02-01T10:23:03.384 DEBUG:boto:path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:03.384 DEBUG:boto:auth_path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:03.384 DEBUG:boto:Method: GET 2024-02-01T10:23:03.384 DEBUG:boto:Path: /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:03.384 DEBUG:boto:Data: 2024-02-01T10:23:03.385 DEBUG:boto:Headers: {} 2024-02-01T10:23:03.385 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:03.385 DEBUG:boto:Port: 8001 2024-02-01T10:23:03.385 DEBUG:boto:Params: {} 2024-02-01T10:23:03.385 DEBUG:boto:Token: None 2024-02-01T10:23:03.385 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:03 GMT /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:03.385 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ORhbqIwyV4RnBMiK9H7ER3YlA2g= 2024-02-01T10:23:03.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ORhbqIwyV4RnBMiK9H7ER3YlA2g=', 'Content-Length': '0'} 2024-02-01T10:23:03.388 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:34 GMT'), ('x-amz-version-id', 'txgLlJWZVbWbjqLWqkOtt87dUoD4zun'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f7768c70b2263ba2-0065bb7107-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:03.388 DEBUG:boto:path=/vclfcn-18/dummy 2024-02-01T10:23:03.388 DEBUG:boto:auth_path=/vclfcn-18/dummy 2024-02-01T10:23:03.388 DEBUG:boto:path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:03.388 DEBUG:boto:auth_path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:03.389 DEBUG:boto:Method: GET 2024-02-01T10:23:03.389 DEBUG:boto:Path: /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:03.389 DEBUG:boto:Data: 2024-02-01T10:23:03.389 DEBUG:boto:Headers: {} 2024-02-01T10:23:03.389 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:23:03.389 DEBUG:boto:Port: 8000 2024-02-01T10:23:03.389 DEBUG:boto:Params: {} 2024-02-01T10:23:03.390 DEBUG:boto:Token: None 2024-02-01T10:23:03.390 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:03 GMT /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:03.390 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ORhbqIwyV4RnBMiK9H7ER3YlA2g= 2024-02-01T10:23:03.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ORhbqIwyV4RnBMiK9H7ER3YlA2g=', 'Content-Length': '0'} 2024-02-01T10:23:03.392 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:34 GMT'), ('x-amz-version-id', 'txgLlJWZVbWbjqLWqkOtt87dUoD4zun'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002d6af4685bfcd197-0065bb7107-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:03.393 DEBUG:boto:path=/vclfcn-18/dummy 2024-02-01T10:23:03.393 DEBUG:boto:auth_path=/vclfcn-18/dummy 2024-02-01T10:23:03.393 DEBUG:boto:path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:03.393 DEBUG:boto:auth_path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:03.393 DEBUG:boto:Method: HEAD 2024-02-01T10:23:03.393 DEBUG:boto:Path: /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:03.393 DEBUG:boto:Data: 2024-02-01T10:23:03.393 DEBUG:boto:Headers: {} 2024-02-01T10:23:03.393 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:03.394 DEBUG:boto:Port: 8001 2024-02-01T10:23:03.394 DEBUG:boto:Params: {} 2024-02-01T10:23:03.394 DEBUG:boto:Token: None 2024-02-01T10:23:03.394 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:23:03 GMT /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:03.394 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:aTl0betK7MIc7ky5MZTxVLoZJ54= 2024-02-01T10:23:03.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:aTl0betK7MIc7ky5MZTxVLoZJ54=', 'Content-Length': '0'} 2024-02-01T10:23:03.397 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:34 GMT'), ('x-amz-version-id', 'txgLlJWZVbWbjqLWqkOtt87dUoD4zun'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000085ebc83edda23279-0065bb7107-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:03.397 DEBUG:boto:path=/vclfcn-18/dummy 2024-02-01T10:23:03.397 DEBUG:boto:auth_path=/vclfcn-18/dummy 2024-02-01T10:23:03.398 DEBUG:boto:path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:03.398 DEBUG:boto:auth_path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:03.398 DEBUG:boto:Method: HEAD 2024-02-01T10:23:03.398 DEBUG:boto:Path: /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:03.398 DEBUG:boto:Data: 2024-02-01T10:23:03.398 DEBUG:boto:Headers: {} 2024-02-01T10:23:03.398 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:23:03.398 DEBUG:boto:Port: 8000 2024-02-01T10:23:03.398 DEBUG:boto:Params: {} 2024-02-01T10:23:03.398 DEBUG:boto:Token: None 2024-02-01T10:23:03.399 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:23:03 GMT /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:03.399 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:aTl0betK7MIc7ky5MZTxVLoZJ54= 2024-02-01T10:23:03.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:aTl0betK7MIc7ky5MZTxVLoZJ54=', 'Content-Length': '0'} 2024-02-01T10:23:03.401 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:34 GMT'), ('x-amz-version-id', 'txgLlJWZVbWbjqLWqkOtt87dUoD4zun'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000018cce1c982c3bbd5-0065bb7107-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:03.402 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-02-01T10:23:03.402 DEBUG:boto:path=/vclfcn-18/dummy 2024-02-01T10:23:03.402 DEBUG:boto:auth_path=/vclfcn-18/dummy 2024-02-01T10:23:03.402 DEBUG:boto:path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:03.402 DEBUG:boto:auth_path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:03.402 DEBUG:boto:Method: GET 2024-02-01T10:23:03.402 DEBUG:boto:Path: /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:03.402 DEBUG:boto:Data: 2024-02-01T10:23:03.402 DEBUG:boto:Headers: {} 2024-02-01T10:23:03.403 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:03.403 DEBUG:boto:Port: 8001 2024-02-01T10:23:03.403 DEBUG:boto:Params: {} 2024-02-01T10:23:03.403 DEBUG:boto:Token: None 2024-02-01T10:23:03.403 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:03 GMT /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:03.403 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ORhbqIwyV4RnBMiK9H7ER3YlA2g= 2024-02-01T10:23:03.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ORhbqIwyV4RnBMiK9H7ER3YlA2g=', 'Content-Length': '0'} 2024-02-01T10:23:03.406 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:34 GMT'), ('x-amz-version-id', 'txgLlJWZVbWbjqLWqkOtt87dUoD4zun'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002f4b2746a0d321af-0065bb7107-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:03.406 DEBUG:boto:path=/vclfcn-18/dummy 2024-02-01T10:23:03.406 DEBUG:boto:auth_path=/vclfcn-18/dummy 2024-02-01T10:23:03.407 DEBUG:boto:path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:03.407 DEBUG:boto:auth_path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:03.407 DEBUG:boto:Method: GET 2024-02-01T10:23:03.407 DEBUG:boto:Path: /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:03.407 DEBUG:boto:Data: 2024-02-01T10:23:03.407 DEBUG:boto:Headers: {} 2024-02-01T10:23:03.407 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:23:03.407 DEBUG:boto:Port: 8000 2024-02-01T10:23:03.407 DEBUG:boto:Params: {} 2024-02-01T10:23:03.407 DEBUG:boto:Token: None 2024-02-01T10:23:03.407 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:03 GMT /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:03.408 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ORhbqIwyV4RnBMiK9H7ER3YlA2g= 2024-02-01T10:23:03.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ORhbqIwyV4RnBMiK9H7ER3YlA2g=', 'Content-Length': '0'} 2024-02-01T10:23:03.411 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:34 GMT'), ('x-amz-version-id', 'txgLlJWZVbWbjqLWqkOtt87dUoD4zun'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000108b320582d598b5-0065bb7107-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:03.411 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:23:03.411 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:23:03.411 DEBUG:boto:path=/vclfcn-18/?prefix=dummy 2024-02-01T10:23:03.411 DEBUG:boto:auth_path=/vclfcn-18/?prefix=dummy 2024-02-01T10:23:03.411 DEBUG:boto:Method: GET 2024-02-01T10:23:03.412 DEBUG:boto:Path: /vclfcn-18/?prefix=dummy 2024-02-01T10:23:03.412 DEBUG:boto:Data: 2024-02-01T10:23:03.412 DEBUG:boto:Headers: {} 2024-02-01T10:23:03.412 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:03.412 DEBUG:boto:Port: 8001 2024-02-01T10:23:03.412 DEBUG:boto:Params: {} 2024-02-01T10:23:03.412 DEBUG:boto:Token: None 2024-02-01T10:23:03.412 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:03 GMT /vclfcn-18/ 2024-02-01T10:23:03.412 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HoVGoRezg9W8je4qz8QiLl1tVKo= 2024-02-01T10:23:03.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HoVGoRezg9W8je4qz8QiLl1tVKo=', 'Content-Length': '0'} 2024-02-01T10:23:03.416 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007637869695edac81-0065bb7107-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:03.460 DEBUG:boto:b'vclfcn-18dummy1000falsedummy2024-02-01T10:20:34.645Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:03.460 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:23:03.461 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:23:03.461 DEBUG:boto:path=/vclfcn-18/?prefix=dummy 2024-02-01T10:23:03.461 DEBUG:boto:auth_path=/vclfcn-18/?prefix=dummy 2024-02-01T10:23:03.461 DEBUG:boto:Method: GET 2024-02-01T10:23:03.461 DEBUG:boto:Path: /vclfcn-18/?prefix=dummy 2024-02-01T10:23:03.461 DEBUG:boto:Data: 2024-02-01T10:23:03.461 DEBUG:boto:Headers: {} 2024-02-01T10:23:03.461 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:23:03.461 DEBUG:boto:Port: 8000 2024-02-01T10:23:03.461 DEBUG:boto:Params: {} 2024-02-01T10:23:03.462 DEBUG:boto:Token: None 2024-02-01T10:23:03.462 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:03 GMT /vclfcn-18/ 2024-02-01T10:23:03.462 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HoVGoRezg9W8je4qz8QiLl1tVKo= 2024-02-01T10:23:03.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HoVGoRezg9W8je4qz8QiLl1tVKo=', 'Content-Length': '0'} 2024-02-01T10:23:03.466 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000005f17a1fc94bb359-0065bb7107-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:03.466 DEBUG:boto:b'vclfcn-18dummy1000falsedummy2024-02-01T10:20:34.645Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:03.466 DEBUG:boto:path=/vclfcn-18/dummy 2024-02-01T10:23:03.467 DEBUG:boto:auth_path=/vclfcn-18/dummy 2024-02-01T10:23:03.467 DEBUG:boto:Method: HEAD 2024-02-01T10:23:03.467 DEBUG:boto:Path: /vclfcn-18/dummy 2024-02-01T10:23:03.467 DEBUG:boto:Data: 2024-02-01T10:23:03.467 DEBUG:boto:Headers: {} 2024-02-01T10:23:03.467 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:03.467 DEBUG:boto:Port: 8001 2024-02-01T10:23:03.467 DEBUG:boto:Params: {} 2024-02-01T10:23:03.467 DEBUG:boto:Token: None 2024-02-01T10:23:03.467 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:23:03 GMT /vclfcn-18/dummy 2024-02-01T10:23:03.468 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:rq4//sLj1kT7Taj1Ni/wMaeyz4k= 2024-02-01T10:23:03.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:rq4//sLj1kT7Taj1Ni/wMaeyz4k=', 'Content-Length': '0'} 2024-02-01T10:23:03.472 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:34 GMT'), ('x-amz-version-id', 'txgLlJWZVbWbjqLWqkOtt87dUoD4zun'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002b5856d5bbf909f8-0065bb7107-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:03.472 DEBUG:boto:path=/vclfcn-18/dummy 2024-02-01T10:23:03.472 DEBUG:boto:auth_path=/vclfcn-18/dummy 2024-02-01T10:23:03.472 DEBUG:boto:Method: HEAD 2024-02-01T10:23:03.472 DEBUG:boto:Path: /vclfcn-18/dummy 2024-02-01T10:23:03.472 DEBUG:boto:Data: 2024-02-01T10:23:03.472 DEBUG:boto:Headers: {} 2024-02-01T10:23:03.473 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:23:03.473 DEBUG:boto:Port: 8000 2024-02-01T10:23:03.473 DEBUG:boto:Params: {} 2024-02-01T10:23:03.473 DEBUG:boto:Token: None 2024-02-01T10:23:03.473 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:23:03 GMT /vclfcn-18/dummy 2024-02-01T10:23:03.473 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:rq4//sLj1kT7Taj1Ni/wMaeyz4k= 2024-02-01T10:23:03.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:rq4//sLj1kT7Taj1Ni/wMaeyz4k=', 'Content-Length': '0'} 2024-02-01T10:23:03.477 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:34 GMT'), ('x-amz-version-id', 'txgLlJWZVbWbjqLWqkOtt87dUoD4zun'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000423b48a3fb5035f0-0065bb7107-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:03.477 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-02-01T10:23:03.477 DEBUG:boto:path=/vclfcn-18/dummy 2024-02-01T10:23:03.477 DEBUG:boto:auth_path=/vclfcn-18/dummy 2024-02-01T10:23:03.477 DEBUG:boto:path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:03.477 DEBUG:boto:auth_path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:03.477 DEBUG:boto:Method: GET 2024-02-01T10:23:03.478 DEBUG:boto:Path: /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:03.478 DEBUG:boto:Data: 2024-02-01T10:23:03.478 DEBUG:boto:Headers: {} 2024-02-01T10:23:03.478 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:03.478 DEBUG:boto:Port: 8001 2024-02-01T10:23:03.478 DEBUG:boto:Params: {} 2024-02-01T10:23:03.478 DEBUG:boto:Token: None 2024-02-01T10:23:03.478 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:03 GMT /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:03.478 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ORhbqIwyV4RnBMiK9H7ER3YlA2g= 2024-02-01T10:23:03.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ORhbqIwyV4RnBMiK9H7ER3YlA2g=', 'Content-Length': '0'} 2024-02-01T10:23:03.481 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:34 GMT'), ('x-amz-version-id', 'txgLlJWZVbWbjqLWqkOtt87dUoD4zun'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b8aa893c25886b99-0065bb7107-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:03.481 DEBUG:boto:path=/vclfcn-18/dummy 2024-02-01T10:23:03.482 DEBUG:boto:auth_path=/vclfcn-18/dummy 2024-02-01T10:23:03.482 DEBUG:boto:path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:03.482 DEBUG:boto:auth_path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:03.482 DEBUG:boto:Method: GET 2024-02-01T10:23:03.482 DEBUG:boto:Path: /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:03.482 DEBUG:boto:Data: 2024-02-01T10:23:03.482 DEBUG:boto:Headers: {} 2024-02-01T10:23:03.482 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:23:03.482 DEBUG:boto:Port: 8000 2024-02-01T10:23:03.482 DEBUG:boto:Params: {} 2024-02-01T10:23:03.483 DEBUG:boto:Token: None 2024-02-01T10:23:03.483 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:03 GMT /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:03.483 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ORhbqIwyV4RnBMiK9H7ER3YlA2g= 2024-02-01T10:23:03.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ORhbqIwyV4RnBMiK9H7ER3YlA2g=', 'Content-Length': '0'} 2024-02-01T10:23:03.485 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:34 GMT'), ('x-amz-version-id', 'txgLlJWZVbWbjqLWqkOtt87dUoD4zun'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000098ae4a36d25e010d-0065bb7107-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:03.486 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:23:03.486 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:23:03.486 DEBUG:boto:path=/vclfcn-18/?prefix=dummy 2024-02-01T10:23:03.486 DEBUG:boto:auth_path=/vclfcn-18/?prefix=dummy 2024-02-01T10:23:03.486 DEBUG:boto:Method: GET 2024-02-01T10:23:03.486 DEBUG:boto:Path: /vclfcn-18/?prefix=dummy 2024-02-01T10:23:03.486 DEBUG:boto:Data: 2024-02-01T10:23:03.487 DEBUG:boto:Headers: {} 2024-02-01T10:23:03.487 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:03.487 DEBUG:boto:Port: 8001 2024-02-01T10:23:03.487 DEBUG:boto:Params: {} 2024-02-01T10:23:03.487 DEBUG:boto:Token: None 2024-02-01T10:23:03.487 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:03 GMT /vclfcn-18/ 2024-02-01T10:23:03.487 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HoVGoRezg9W8je4qz8QiLl1tVKo= 2024-02-01T10:23:03.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HoVGoRezg9W8je4qz8QiLl1tVKo=', 'Content-Length': '0'} 2024-02-01T10:23:03.491 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6b23343dfc1142f-0065bb7107-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:03.492 DEBUG:boto:b'vclfcn-18dummy1000falsedummy2024-02-01T10:20:34.645Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:03.492 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:23:03.492 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:23:03.492 DEBUG:boto:path=/vclfcn-18/?prefix=dummy 2024-02-01T10:23:03.492 DEBUG:boto:auth_path=/vclfcn-18/?prefix=dummy 2024-02-01T10:23:03.493 DEBUG:boto:Method: GET 2024-02-01T10:23:03.493 DEBUG:boto:Path: /vclfcn-18/?prefix=dummy 2024-02-01T10:23:03.493 DEBUG:boto:Data: 2024-02-01T10:23:03.493 DEBUG:boto:Headers: {} 2024-02-01T10:23:03.493 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:23:03.493 DEBUG:boto:Port: 8000 2024-02-01T10:23:03.493 DEBUG:boto:Params: {} 2024-02-01T10:23:03.493 DEBUG:boto:Token: None 2024-02-01T10:23:03.493 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:03 GMT /vclfcn-18/ 2024-02-01T10:23:03.493 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HoVGoRezg9W8je4qz8QiLl1tVKo= 2024-02-01T10:23:03.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HoVGoRezg9W8je4qz8QiLl1tVKo=', 'Content-Length': '0'} 2024-02-01T10:23:03.498 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000070d3ee6390418f3c-0065bb7107-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:03.544 DEBUG:boto:b'vclfcn-18dummy1000falsedummy2024-02-01T10:20:34.645Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:03.544 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-02-01T10:23:03.545 DEBUG:boto:path=/vclfcn-18/obj-test-zone1 2024-02-01T10:23:03.545 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1 2024-02-01T10:23:03.545 DEBUG:boto:path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:03.545 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:03.545 DEBUG:boto:Method: GET 2024-02-01T10:23:03.545 DEBUG:boto:Path: /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:03.545 DEBUG:boto:Data: 2024-02-01T10:23:03.545 DEBUG:boto:Headers: {} 2024-02-01T10:23:03.545 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:03.545 DEBUG:boto:Port: 8001 2024-02-01T10:23:03.545 DEBUG:boto:Params: {} 2024-02-01T10:23:03.546 DEBUG:boto:Token: None 2024-02-01T10:23:03.546 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:03 GMT /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:03.546 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ygf4cEDYPZsRnUMCENVzNyWvMc8= 2024-02-01T10:23:03.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ygf4cEDYPZsRnUMCENVzNyWvMc8=', 'Content-Length': '0'} 2024-02-01T10:23:03.549 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000025d3fb8668fd327a-0065bb7107-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:03.549 DEBUG:boto:path=/vclfcn-18/obj-test-zone1 2024-02-01T10:23:03.549 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1 2024-02-01T10:23:03.549 DEBUG:boto:path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:03.549 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:03.550 DEBUG:boto:Method: GET 2024-02-01T10:23:03.550 DEBUG:boto:Path: /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:03.550 DEBUG:boto:Data: 2024-02-01T10:23:03.550 DEBUG:boto:Headers: {} 2024-02-01T10:23:03.550 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:23:03.550 DEBUG:boto:Port: 8000 2024-02-01T10:23:03.550 DEBUG:boto:Params: {} 2024-02-01T10:23:03.550 DEBUG:boto:Token: None 2024-02-01T10:23:03.550 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:03 GMT /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:03.550 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ygf4cEDYPZsRnUMCENVzNyWvMc8= 2024-02-01T10:23:03.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ygf4cEDYPZsRnUMCENVzNyWvMc8=', 'Content-Length': '0'} 2024-02-01T10:23:03.553 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009a2711a40d546f11-0065bb7107-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:03.553 DEBUG:boto:path=/vclfcn-18/obj-test-zone1 2024-02-01T10:23:03.554 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1 2024-02-01T10:23:03.554 DEBUG:boto:path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:03.554 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:03.554 DEBUG:boto:Method: HEAD 2024-02-01T10:23:03.554 DEBUG:boto:Path: /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:03.554 DEBUG:boto:Data: 2024-02-01T10:23:03.554 DEBUG:boto:Headers: {} 2024-02-01T10:23:03.554 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:03.554 DEBUG:boto:Port: 8001 2024-02-01T10:23:03.554 DEBUG:boto:Params: {} 2024-02-01T10:23:03.554 DEBUG:boto:Token: None 2024-02-01T10:23:03.555 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:23:03 GMT /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:03.555 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7d1HNsVDExas/1DfEl6v3L+mq2w= 2024-02-01T10:23:03.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7d1HNsVDExas/1DfEl6v3L+mq2w=', 'Content-Length': '0'} 2024-02-01T10:23:03.557 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000066d992dd110d3c9f-0065bb7107-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:03.558 DEBUG:boto:path=/vclfcn-18/obj-test-zone1 2024-02-01T10:23:03.558 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1 2024-02-01T10:23:03.558 DEBUG:boto:path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:03.558 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:03.558 DEBUG:boto:Method: HEAD 2024-02-01T10:23:03.558 DEBUG:boto:Path: /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:03.558 DEBUG:boto:Data: 2024-02-01T10:23:03.558 DEBUG:boto:Headers: {} 2024-02-01T10:23:03.558 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:23:03.558 DEBUG:boto:Port: 8000 2024-02-01T10:23:03.559 DEBUG:boto:Params: {} 2024-02-01T10:23:03.559 DEBUG:boto:Token: None 2024-02-01T10:23:03.559 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:23:03 GMT /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:03.559 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7d1HNsVDExas/1DfEl6v3L+mq2w= 2024-02-01T10:23:03.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7d1HNsVDExas/1DfEl6v3L+mq2w=', 'Content-Length': '0'} 2024-02-01T10:23:03.562 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b0a732521f174417-0065bb7107-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:03.562 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-02-01T10:23:03.562 DEBUG:boto:path=/vclfcn-18/obj-test-zone1 2024-02-01T10:23:03.562 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1 2024-02-01T10:23:03.562 DEBUG:boto:path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:03.562 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:03.563 DEBUG:boto:Method: GET 2024-02-01T10:23:03.563 DEBUG:boto:Path: /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:03.563 DEBUG:boto:Data: 2024-02-01T10:23:03.563 DEBUG:boto:Headers: {} 2024-02-01T10:23:03.563 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:03.563 DEBUG:boto:Port: 8001 2024-02-01T10:23:03.563 DEBUG:boto:Params: {} 2024-02-01T10:23:03.563 DEBUG:boto:Token: None 2024-02-01T10:23:03.563 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:03 GMT /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:03.563 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ygf4cEDYPZsRnUMCENVzNyWvMc8= 2024-02-01T10:23:03.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ygf4cEDYPZsRnUMCENVzNyWvMc8=', 'Content-Length': '0'} 2024-02-01T10:23:03.566 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002b714b0d0828089d-0065bb7107-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:03.608 DEBUG:boto:path=/vclfcn-18/obj-test-zone1 2024-02-01T10:23:03.608 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1 2024-02-01T10:23:03.608 DEBUG:boto:path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:03.608 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:03.609 DEBUG:boto:Method: GET 2024-02-01T10:23:03.609 DEBUG:boto:Path: /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:03.609 DEBUG:boto:Data: 2024-02-01T10:23:03.609 DEBUG:boto:Headers: {} 2024-02-01T10:23:03.609 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:23:03.609 DEBUG:boto:Port: 8000 2024-02-01T10:23:03.609 DEBUG:boto:Params: {} 2024-02-01T10:23:03.609 DEBUG:boto:Token: None 2024-02-01T10:23:03.609 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:03 GMT /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:03.609 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ygf4cEDYPZsRnUMCENVzNyWvMc8= 2024-02-01T10:23:03.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ygf4cEDYPZsRnUMCENVzNyWvMc8=', 'Content-Length': '0'} 2024-02-01T10:23:03.612 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fb167c8c963ffb72-0065bb7107-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:03.613 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:23:03.613 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:23:03.613 DEBUG:boto:path=/vclfcn-18/?prefix=obj-test-zone1 2024-02-01T10:23:03.613 DEBUG:boto:auth_path=/vclfcn-18/?prefix=obj-test-zone1 2024-02-01T10:23:03.614 DEBUG:boto:Method: GET 2024-02-01T10:23:03.614 DEBUG:boto:Path: /vclfcn-18/?prefix=obj-test-zone1 2024-02-01T10:23:03.614 DEBUG:boto:Data: 2024-02-01T10:23:03.614 DEBUG:boto:Headers: {} 2024-02-01T10:23:03.614 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:03.614 DEBUG:boto:Port: 8001 2024-02-01T10:23:03.614 DEBUG:boto:Params: {} 2024-02-01T10:23:03.614 DEBUG:boto:Token: None 2024-02-01T10:23:03.614 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:03 GMT /vclfcn-18/ 2024-02-01T10:23:03.614 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HoVGoRezg9W8je4qz8QiLl1tVKo= 2024-02-01T10:23:03.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HoVGoRezg9W8je4qz8QiLl1tVKo=', 'Content-Length': '0'} 2024-02-01T10:23:03.619 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000485fd826f751c11b-0065bb7107-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:03.619 DEBUG:boto:b'vclfcn-18obj-test-zone11000falseobj-test-zone12024-02-01T10:20:50.942Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:03.620 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:23:03.620 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:23:03.620 DEBUG:boto:path=/vclfcn-18/?prefix=obj-test-zone1 2024-02-01T10:23:03.620 DEBUG:boto:auth_path=/vclfcn-18/?prefix=obj-test-zone1 2024-02-01T10:23:03.620 DEBUG:boto:Method: GET 2024-02-01T10:23:03.620 DEBUG:boto:Path: /vclfcn-18/?prefix=obj-test-zone1 2024-02-01T10:23:03.620 DEBUG:boto:Data: 2024-02-01T10:23:03.620 DEBUG:boto:Headers: {} 2024-02-01T10:23:03.621 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:23:03.621 DEBUG:boto:Port: 8000 2024-02-01T10:23:03.621 DEBUG:boto:Params: {} 2024-02-01T10:23:03.621 DEBUG:boto:Token: None 2024-02-01T10:23:03.621 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:03 GMT /vclfcn-18/ 2024-02-01T10:23:03.621 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HoVGoRezg9W8je4qz8QiLl1tVKo= 2024-02-01T10:23:03.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HoVGoRezg9W8je4qz8QiLl1tVKo=', 'Content-Length': '0'} 2024-02-01T10:23:03.625 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e7b6434a3f221133-0065bb7107-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:03.668 DEBUG:boto:b'vclfcn-18obj-test-zone11000falseobj-test-zone12024-02-01T10:20:50.942Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:03.668 DEBUG:boto:path=/vclfcn-18/obj-test-zone1 2024-02-01T10:23:03.669 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1 2024-02-01T10:23:03.669 DEBUG:boto:Method: HEAD 2024-02-01T10:23:03.669 DEBUG:boto:Path: /vclfcn-18/obj-test-zone1 2024-02-01T10:23:03.669 DEBUG:boto:Data: 2024-02-01T10:23:03.669 DEBUG:boto:Headers: {} 2024-02-01T10:23:03.669 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:03.669 DEBUG:boto:Port: 8001 2024-02-01T10:23:03.669 DEBUG:boto:Params: {} 2024-02-01T10:23:03.669 DEBUG:boto:Token: None 2024-02-01T10:23:03.670 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:23:03 GMT /vclfcn-18/obj-test-zone1 2024-02-01T10:23:03.670 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ddaj/snyNt+aYgLQnt400KdIQ4I= 2024-02-01T10:23:03.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ddaj/snyNt+aYgLQnt400KdIQ4I=', 'Content-Length': '0'} 2024-02-01T10:23:03.673 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000dfab3fc54928c7a-0065bb7107-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:03.674 DEBUG:boto:path=/vclfcn-18/obj-test-zone1 2024-02-01T10:23:03.674 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1 2024-02-01T10:23:03.674 DEBUG:boto:Method: HEAD 2024-02-01T10:23:03.674 DEBUG:boto:Path: /vclfcn-18/obj-test-zone1 2024-02-01T10:23:03.674 DEBUG:boto:Data: 2024-02-01T10:23:03.674 DEBUG:boto:Headers: {} 2024-02-01T10:23:03.674 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:23:03.675 DEBUG:boto:Port: 8000 2024-02-01T10:23:03.675 DEBUG:boto:Params: {} 2024-02-01T10:23:03.675 DEBUG:boto:Token: None 2024-02-01T10:23:03.675 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:23:03 GMT /vclfcn-18/obj-test-zone1 2024-02-01T10:23:03.675 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ddaj/snyNt+aYgLQnt400KdIQ4I= 2024-02-01T10:23:03.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ddaj/snyNt+aYgLQnt400KdIQ4I=', 'Content-Length': '0'} 2024-02-01T10:23:03.680 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002ccbbcb93477e9b8-0065bb7107-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:03.681 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-02-01T10:23:03.681 DEBUG:boto:path=/vclfcn-18/obj-test-zone1 2024-02-01T10:23:03.681 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1 2024-02-01T10:23:03.681 DEBUG:boto:path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:03.681 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:03.681 DEBUG:boto:Method: GET 2024-02-01T10:23:03.681 DEBUG:boto:Path: /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:03.681 DEBUG:boto:Data: 2024-02-01T10:23:03.682 DEBUG:boto:Headers: {} 2024-02-01T10:23:03.682 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:03.682 DEBUG:boto:Port: 8001 2024-02-01T10:23:03.682 DEBUG:boto:Params: {} 2024-02-01T10:23:03.682 DEBUG:boto:Token: None 2024-02-01T10:23:03.682 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:03 GMT /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:03.682 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ygf4cEDYPZsRnUMCENVzNyWvMc8= 2024-02-01T10:23:03.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ygf4cEDYPZsRnUMCENVzNyWvMc8=', 'Content-Length': '0'} 2024-02-01T10:23:03.686 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000094ebca42b4ab24aa-0065bb7107-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:03.732 DEBUG:boto:path=/vclfcn-18/obj-test-zone1 2024-02-01T10:23:03.732 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1 2024-02-01T10:23:03.732 DEBUG:boto:path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:03.732 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:03.733 DEBUG:boto:Method: GET 2024-02-01T10:23:03.733 DEBUG:boto:Path: /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:03.733 DEBUG:boto:Data: 2024-02-01T10:23:03.733 DEBUG:boto:Headers: {} 2024-02-01T10:23:03.733 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:23:03.733 DEBUG:boto:Port: 8000 2024-02-01T10:23:03.733 DEBUG:boto:Params: {} 2024-02-01T10:23:03.733 DEBUG:boto:Token: None 2024-02-01T10:23:03.733 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:03 GMT /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:03.733 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ygf4cEDYPZsRnUMCENVzNyWvMc8= 2024-02-01T10:23:03.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ygf4cEDYPZsRnUMCENVzNyWvMc8=', 'Content-Length': '0'} 2024-02-01T10:23:03.736 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aff26445e5638fe6-0065bb7107-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:03.737 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:23:03.737 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:23:03.737 DEBUG:boto:path=/vclfcn-18/?prefix=obj-test-zone1 2024-02-01T10:23:03.737 DEBUG:boto:auth_path=/vclfcn-18/?prefix=obj-test-zone1 2024-02-01T10:23:03.737 DEBUG:boto:Method: GET 2024-02-01T10:23:03.737 DEBUG:boto:Path: /vclfcn-18/?prefix=obj-test-zone1 2024-02-01T10:23:03.738 DEBUG:boto:Data: 2024-02-01T10:23:03.738 DEBUG:boto:Headers: {} 2024-02-01T10:23:03.738 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:03.738 DEBUG:boto:Port: 8001 2024-02-01T10:23:03.738 DEBUG:boto:Params: {} 2024-02-01T10:23:03.738 DEBUG:boto:Token: None 2024-02-01T10:23:03.738 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:03 GMT /vclfcn-18/ 2024-02-01T10:23:03.738 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HoVGoRezg9W8je4qz8QiLl1tVKo= 2024-02-01T10:23:03.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HoVGoRezg9W8je4qz8QiLl1tVKo=', 'Content-Length': '0'} 2024-02-01T10:23:03.743 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e13b1c0e5a74ff35-0065bb7107-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:03.784 DEBUG:boto:b'vclfcn-18obj-test-zone11000falseobj-test-zone12024-02-01T10:20:50.942Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:03.785 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:23:03.785 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:23:03.785 DEBUG:boto:path=/vclfcn-18/?prefix=obj-test-zone1 2024-02-01T10:23:03.785 DEBUG:boto:auth_path=/vclfcn-18/?prefix=obj-test-zone1 2024-02-01T10:23:03.785 DEBUG:boto:Method: GET 2024-02-01T10:23:03.785 DEBUG:boto:Path: /vclfcn-18/?prefix=obj-test-zone1 2024-02-01T10:23:03.785 DEBUG:boto:Data: 2024-02-01T10:23:03.785 DEBUG:boto:Headers: {} 2024-02-01T10:23:03.785 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:23:03.785 DEBUG:boto:Port: 8000 2024-02-01T10:23:03.786 DEBUG:boto:Params: {} 2024-02-01T10:23:03.786 DEBUG:boto:Token: None 2024-02-01T10:23:03.786 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:03 GMT /vclfcn-18/ 2024-02-01T10:23:03.786 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HoVGoRezg9W8je4qz8QiLl1tVKo= 2024-02-01T10:23:03.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HoVGoRezg9W8je4qz8QiLl1tVKo=', 'Content-Length': '0'} 2024-02-01T10:23:03.790 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e78ac375f025a1bf-0065bb7107-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:03.790 DEBUG:boto:b'vclfcn-18obj-test-zone11000falseobj-test-zone12024-02-01T10:20:50.942Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:03.791 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-02-01T10:23:03.791 DEBUG:boto:path=/vclfcn-18/obj-test-zone2 2024-02-01T10:23:03.791 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2 2024-02-01T10:23:03.791 DEBUG:boto:path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:03.791 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:03.791 DEBUG:boto:Method: GET 2024-02-01T10:23:03.792 DEBUG:boto:Path: /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:03.792 DEBUG:boto:Data: 2024-02-01T10:23:03.793 DEBUG:boto:Headers: {} 2024-02-01T10:23:03.793 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:03.793 DEBUG:boto:Port: 8001 2024-02-01T10:23:03.793 DEBUG:boto:Params: {} 2024-02-01T10:23:03.793 DEBUG:boto:Token: None 2024-02-01T10:23:03.793 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:03 GMT /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:03.793 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8T71g3iZ3svLainRre/49mdrC9Q= 2024-02-01T10:23:03.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8T71g3iZ3svLainRre/49mdrC9Q=', 'Content-Length': '0'} 2024-02-01T10:23:03.796 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', 'ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009c5781a6f8a65b2e-0065bb7107-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:03.840 DEBUG:boto:path=/vclfcn-18/obj-test-zone2 2024-02-01T10:23:03.840 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2 2024-02-01T10:23:03.840 DEBUG:boto:path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:03.840 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:03.840 DEBUG:boto:Method: GET 2024-02-01T10:23:03.841 DEBUG:boto:Path: /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:03.841 DEBUG:boto:Data: 2024-02-01T10:23:03.841 DEBUG:boto:Headers: {} 2024-02-01T10:23:03.841 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:23:03.841 DEBUG:boto:Port: 8000 2024-02-01T10:23:03.841 DEBUG:boto:Params: {} 2024-02-01T10:23:03.841 DEBUG:boto:Token: None 2024-02-01T10:23:03.841 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:03 GMT /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:03.841 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8T71g3iZ3svLainRre/49mdrC9Q= 2024-02-01T10:23:03.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8T71g3iZ3svLainRre/49mdrC9Q=', 'Content-Length': '0'} 2024-02-01T10:23:03.844 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', 'ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003aa323a96e68caf0-0065bb7107-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:03.888 DEBUG:boto:path=/vclfcn-18/obj-test-zone2 2024-02-01T10:23:03.888 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2 2024-02-01T10:23:03.888 DEBUG:boto:path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:03.888 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:03.889 DEBUG:boto:Method: HEAD 2024-02-01T10:23:03.889 DEBUG:boto:Path: /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:03.889 DEBUG:boto:Data: 2024-02-01T10:23:03.889 DEBUG:boto:Headers: {} 2024-02-01T10:23:03.889 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:03.889 DEBUG:boto:Port: 8001 2024-02-01T10:23:03.889 DEBUG:boto:Params: {} 2024-02-01T10:23:03.889 DEBUG:boto:Token: None 2024-02-01T10:23:03.889 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:23:03 GMT /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:03.889 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lArdkfjX9Ja4WYNC0XMd20Cnd1Q= 2024-02-01T10:23:03.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lArdkfjX9Ja4WYNC0XMd20Cnd1Q=', 'Content-Length': '0'} 2024-02-01T10:23:03.893 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', 'ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000077af19ec32e346df-0065bb7107-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:03.893 DEBUG:boto:path=/vclfcn-18/obj-test-zone2 2024-02-01T10:23:03.893 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2 2024-02-01T10:23:03.893 DEBUG:boto:path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:03.893 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:03.893 DEBUG:boto:Method: HEAD 2024-02-01T10:23:03.893 DEBUG:boto:Path: /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:03.893 DEBUG:boto:Data: 2024-02-01T10:23:03.894 DEBUG:boto:Headers: {} 2024-02-01T10:23:03.894 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:23:03.894 DEBUG:boto:Port: 8000 2024-02-01T10:23:03.894 DEBUG:boto:Params: {} 2024-02-01T10:23:03.894 DEBUG:boto:Token: None 2024-02-01T10:23:03.894 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:23:03 GMT /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:03.894 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lArdkfjX9Ja4WYNC0XMd20Cnd1Q= 2024-02-01T10:23:03.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lArdkfjX9Ja4WYNC0XMd20Cnd1Q=', 'Content-Length': '0'} 2024-02-01T10:23:03.897 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', 'ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d06bd6556ddd1b7d-0065bb7107-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:03.897 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-02-01T10:23:03.897 DEBUG:boto:path=/vclfcn-18/obj-test-zone2 2024-02-01T10:23:03.897 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2 2024-02-01T10:23:03.897 DEBUG:boto:path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:03.898 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:03.898 DEBUG:boto:Method: GET 2024-02-01T10:23:03.898 DEBUG:boto:Path: /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:03.898 DEBUG:boto:Data: 2024-02-01T10:23:03.898 DEBUG:boto:Headers: {} 2024-02-01T10:23:03.898 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:03.898 DEBUG:boto:Port: 8001 2024-02-01T10:23:03.898 DEBUG:boto:Params: {} 2024-02-01T10:23:03.898 DEBUG:boto:Token: None 2024-02-01T10:23:03.898 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:03 GMT /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:03.898 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8T71g3iZ3svLainRre/49mdrC9Q= 2024-02-01T10:23:03.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8T71g3iZ3svLainRre/49mdrC9Q=', 'Content-Length': '0'} 2024-02-01T10:23:03.902 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', 'ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000080d871b680c391f5-0065bb7107-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:03.948 DEBUG:boto:path=/vclfcn-18/obj-test-zone2 2024-02-01T10:23:03.948 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2 2024-02-01T10:23:03.948 DEBUG:boto:path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:03.948 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:03.949 DEBUG:boto:Method: GET 2024-02-01T10:23:03.949 DEBUG:boto:Path: /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:03.949 DEBUG:boto:Data: 2024-02-01T10:23:03.949 DEBUG:boto:Headers: {} 2024-02-01T10:23:03.949 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:23:03.949 DEBUG:boto:Port: 8000 2024-02-01T10:23:03.949 DEBUG:boto:Params: {} 2024-02-01T10:23:03.949 DEBUG:boto:Token: None 2024-02-01T10:23:03.949 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:03 GMT /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:03.950 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8T71g3iZ3svLainRre/49mdrC9Q= 2024-02-01T10:23:03.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8T71g3iZ3svLainRre/49mdrC9Q=', 'Content-Length': '0'} 2024-02-01T10:23:03.952 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', 'ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a868d4db533c523c-0065bb7107-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:04.000 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:23:04.000 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:23:04.001 DEBUG:boto:path=/vclfcn-18/?prefix=obj-test-zone2 2024-02-01T10:23:04.001 DEBUG:boto:auth_path=/vclfcn-18/?prefix=obj-test-zone2 2024-02-01T10:23:04.001 DEBUG:boto:Method: GET 2024-02-01T10:23:04.001 DEBUG:boto:Path: /vclfcn-18/?prefix=obj-test-zone2 2024-02-01T10:23:04.001 DEBUG:boto:Data: 2024-02-01T10:23:04.001 DEBUG:boto:Headers: {} 2024-02-01T10:23:04.001 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:04.001 DEBUG:boto:Port: 8001 2024-02-01T10:23:04.001 DEBUG:boto:Params: {} 2024-02-01T10:23:04.001 DEBUG:boto:Token: None 2024-02-01T10:23:04.002 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:04 GMT /vclfcn-18/ 2024-02-01T10:23:04.002 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:uScQd7oraFpm6C3eTQPDd1dHcc0= 2024-02-01T10:23:04.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:uScQd7oraFpm6C3eTQPDd1dHcc0=', 'Content-Length': '0'} 2024-02-01T10:23:04.007 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa5208ffb654143f-0065bb7108-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:04.007 DEBUG:boto:b'vclfcn-18obj-test-zone21000falseobj-test-zone22024-02-01T10:20:51.094Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:04.008 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:23:04.008 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:23:04.008 DEBUG:boto:path=/vclfcn-18/?prefix=obj-test-zone2 2024-02-01T10:23:04.008 DEBUG:boto:auth_path=/vclfcn-18/?prefix=obj-test-zone2 2024-02-01T10:23:04.008 DEBUG:boto:Method: GET 2024-02-01T10:23:04.008 DEBUG:boto:Path: /vclfcn-18/?prefix=obj-test-zone2 2024-02-01T10:23:04.008 DEBUG:boto:Data: 2024-02-01T10:23:04.008 DEBUG:boto:Headers: {} 2024-02-01T10:23:04.008 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:23:04.008 DEBUG:boto:Port: 8000 2024-02-01T10:23:04.009 DEBUG:boto:Params: {} 2024-02-01T10:23:04.009 DEBUG:boto:Token: None 2024-02-01T10:23:04.009 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:04 GMT /vclfcn-18/ 2024-02-01T10:23:04.009 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:uScQd7oraFpm6C3eTQPDd1dHcc0= 2024-02-01T10:23:04.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:uScQd7oraFpm6C3eTQPDd1dHcc0=', 'Content-Length': '0'} 2024-02-01T10:23:04.013 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000841c502aad373ed6-0065bb7108-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:04.013 DEBUG:boto:b'vclfcn-18obj-test-zone21000falseobj-test-zone22024-02-01T10:20:51.094Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:04.014 DEBUG:boto:path=/vclfcn-18/obj-test-zone2 2024-02-01T10:23:04.014 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2 2024-02-01T10:23:04.014 DEBUG:boto:Method: HEAD 2024-02-01T10:23:04.014 DEBUG:boto:Path: /vclfcn-18/obj-test-zone2 2024-02-01T10:23:04.014 DEBUG:boto:Data: 2024-02-01T10:23:04.014 DEBUG:boto:Headers: {} 2024-02-01T10:23:04.014 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:04.015 DEBUG:boto:Port: 8001 2024-02-01T10:23:04.015 DEBUG:boto:Params: {} 2024-02-01T10:23:04.015 DEBUG:boto:Token: None 2024-02-01T10:23:04.015 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:23:04 GMT /vclfcn-18/obj-test-zone2 2024-02-01T10:23:04.015 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:6F6hLVoKO4cCTbm3dr4xzHM77+o= 2024-02-01T10:23:04.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:6F6hLVoKO4cCTbm3dr4xzHM77+o=', 'Content-Length': '0'} 2024-02-01T10:23:04.019 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', 'ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c14485feee634dcb-0065bb7108-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:04.019 DEBUG:boto:path=/vclfcn-18/obj-test-zone2 2024-02-01T10:23:04.019 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2 2024-02-01T10:23:04.019 DEBUG:boto:Method: HEAD 2024-02-01T10:23:04.019 DEBUG:boto:Path: /vclfcn-18/obj-test-zone2 2024-02-01T10:23:04.019 DEBUG:boto:Data: 2024-02-01T10:23:04.020 DEBUG:boto:Headers: {} 2024-02-01T10:23:04.020 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:23:04.020 DEBUG:boto:Port: 8000 2024-02-01T10:23:04.020 DEBUG:boto:Params: {} 2024-02-01T10:23:04.020 DEBUG:boto:Token: None 2024-02-01T10:23:04.020 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:23:04 GMT /vclfcn-18/obj-test-zone2 2024-02-01T10:23:04.020 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:6F6hLVoKO4cCTbm3dr4xzHM77+o= 2024-02-01T10:23:04.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:6F6hLVoKO4cCTbm3dr4xzHM77+o=', 'Content-Length': '0'} 2024-02-01T10:23:04.024 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', 'ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006b21290c6f98f93e-0065bb7108-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:04.024 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-02-01T10:23:04.024 DEBUG:boto:path=/vclfcn-18/obj-test-zone2 2024-02-01T10:23:04.024 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2 2024-02-01T10:23:04.024 DEBUG:boto:path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:04.024 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:04.025 DEBUG:boto:Method: GET 2024-02-01T10:23:04.025 DEBUG:boto:Path: /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:04.025 DEBUG:boto:Data: 2024-02-01T10:23:04.025 DEBUG:boto:Headers: {} 2024-02-01T10:23:04.025 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:04.025 DEBUG:boto:Port: 8001 2024-02-01T10:23:04.025 DEBUG:boto:Params: {} 2024-02-01T10:23:04.025 DEBUG:boto:Token: None 2024-02-01T10:23:04.025 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:04 GMT /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:04.025 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:i9WQh/7AtDn7AxYwgSkifmuQSdM= 2024-02-01T10:23:04.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:i9WQh/7AtDn7AxYwgSkifmuQSdM=', 'Content-Length': '0'} 2024-02-01T10:23:04.028 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', 'ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004385fe3c15d57d33-0065bb7108-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:04.072 DEBUG:boto:path=/vclfcn-18/obj-test-zone2 2024-02-01T10:23:04.072 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2 2024-02-01T10:23:04.072 DEBUG:boto:path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:04.073 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:04.073 DEBUG:boto:Method: GET 2024-02-01T10:23:04.073 DEBUG:boto:Path: /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:04.073 DEBUG:boto:Data: 2024-02-01T10:23:04.073 DEBUG:boto:Headers: {} 2024-02-01T10:23:04.073 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:23:04.073 DEBUG:boto:Port: 8000 2024-02-01T10:23:04.073 DEBUG:boto:Params: {} 2024-02-01T10:23:04.073 DEBUG:boto:Token: None 2024-02-01T10:23:04.073 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:04 GMT /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:04.074 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:i9WQh/7AtDn7AxYwgSkifmuQSdM= 2024-02-01T10:23:04.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:i9WQh/7AtDn7AxYwgSkifmuQSdM=', 'Content-Length': '0'} 2024-02-01T10:23:04.076 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', 'ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a4d7c73f11578b7c-0065bb7108-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:04.120 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:23:04.120 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:23:04.120 DEBUG:boto:path=/vclfcn-18/?prefix=obj-test-zone2 2024-02-01T10:23:04.121 DEBUG:boto:auth_path=/vclfcn-18/?prefix=obj-test-zone2 2024-02-01T10:23:04.121 DEBUG:boto:Method: GET 2024-02-01T10:23:04.121 DEBUG:boto:Path: /vclfcn-18/?prefix=obj-test-zone2 2024-02-01T10:23:04.121 DEBUG:boto:Data: 2024-02-01T10:23:04.121 DEBUG:boto:Headers: {} 2024-02-01T10:23:04.121 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:04.121 DEBUG:boto:Port: 8001 2024-02-01T10:23:04.121 DEBUG:boto:Params: {} 2024-02-01T10:23:04.121 DEBUG:boto:Token: None 2024-02-01T10:23:04.121 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:04 GMT /vclfcn-18/ 2024-02-01T10:23:04.122 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:uScQd7oraFpm6C3eTQPDd1dHcc0= 2024-02-01T10:23:04.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:uScQd7oraFpm6C3eTQPDd1dHcc0=', 'Content-Length': '0'} 2024-02-01T10:23:04.126 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000007680c26e24b7fdf-0065bb7108-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:04.168 DEBUG:boto:b'vclfcn-18obj-test-zone21000falseobj-test-zone22024-02-01T10:20:51.094Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:04.169 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:23:04.169 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:23:04.169 DEBUG:boto:path=/vclfcn-18/?prefix=obj-test-zone2 2024-02-01T10:23:04.169 DEBUG:boto:auth_path=/vclfcn-18/?prefix=obj-test-zone2 2024-02-01T10:23:04.169 DEBUG:boto:Method: GET 2024-02-01T10:23:04.169 DEBUG:boto:Path: /vclfcn-18/?prefix=obj-test-zone2 2024-02-01T10:23:04.169 DEBUG:boto:Data: 2024-02-01T10:23:04.169 DEBUG:boto:Headers: {} 2024-02-01T10:23:04.169 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:23:04.169 DEBUG:boto:Port: 8000 2024-02-01T10:23:04.170 DEBUG:boto:Params: {} 2024-02-01T10:23:04.170 DEBUG:boto:Token: None 2024-02-01T10:23:04.170 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:04 GMT /vclfcn-18/ 2024-02-01T10:23:04.170 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:uScQd7oraFpm6C3eTQPDd1dHcc0= 2024-02-01T10:23:04.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:uScQd7oraFpm6C3eTQPDd1dHcc0=', 'Content-Length': '0'} 2024-02-01T10:23:04.174 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5d47cd20b4ef686-0065bb7108-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:04.216 DEBUG:boto:b'vclfcn-18obj-test-zone21000falseobj-test-zone22024-02-01T10:20:51.094Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:04.217 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-02-01T10:23:04.217 DEBUG:boto:path=/vclfcn-18/obj-test-zone3 2024-02-01T10:23:04.217 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3 2024-02-01T10:23:04.217 DEBUG:boto:path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:04.217 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:04.217 DEBUG:boto:Method: GET 2024-02-01T10:23:04.217 DEBUG:boto:Path: /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:04.217 DEBUG:boto:Data: 2024-02-01T10:23:04.218 DEBUG:boto:Headers: {} 2024-02-01T10:23:04.218 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:04.218 DEBUG:boto:Port: 8001 2024-02-01T10:23:04.218 DEBUG:boto:Params: {} 2024-02-01T10:23:04.218 DEBUG:boto:Token: None 2024-02-01T10:23:04.218 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:04 GMT /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:04.218 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:M933E7EC2Q387J/3tWzlA8pDTfs= 2024-02-01T10:23:04.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:M933E7EC2Q387J/3tWzlA8pDTfs=', 'Content-Length': '0'} 2024-02-01T10:23:04.221 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', '6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003f1fe8211303023b-0065bb7108-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:04.222 DEBUG:boto:path=/vclfcn-18/obj-test-zone3 2024-02-01T10:23:04.222 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3 2024-02-01T10:23:04.222 DEBUG:boto:path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:04.222 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:04.222 DEBUG:boto:Method: GET 2024-02-01T10:23:04.222 DEBUG:boto:Path: /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:04.223 DEBUG:boto:Data: 2024-02-01T10:23:04.223 DEBUG:boto:Headers: {} 2024-02-01T10:23:04.223 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:23:04.223 DEBUG:boto:Port: 8000 2024-02-01T10:23:04.223 DEBUG:boto:Params: {} 2024-02-01T10:23:04.223 DEBUG:boto:Token: None 2024-02-01T10:23:04.223 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:04 GMT /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:04.223 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:M933E7EC2Q387J/3tWzlA8pDTfs= 2024-02-01T10:23:04.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:M933E7EC2Q387J/3tWzlA8pDTfs=', 'Content-Length': '0'} 2024-02-01T10:23:04.226 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', '6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b1256db0dbf60b0f-0065bb7108-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:04.227 DEBUG:boto:path=/vclfcn-18/obj-test-zone3 2024-02-01T10:23:04.227 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3 2024-02-01T10:23:04.227 DEBUG:boto:path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:04.227 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:04.227 DEBUG:boto:Method: HEAD 2024-02-01T10:23:04.227 DEBUG:boto:Path: /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:04.228 DEBUG:boto:Data: 2024-02-01T10:23:04.228 DEBUG:boto:Headers: {} 2024-02-01T10:23:04.228 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:04.228 DEBUG:boto:Port: 8001 2024-02-01T10:23:04.228 DEBUG:boto:Params: {} 2024-02-01T10:23:04.228 DEBUG:boto:Token: None 2024-02-01T10:23:04.228 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:23:04 GMT /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:04.228 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nwqF/PlVk/niGRdIVn4lNxEirmg= 2024-02-01T10:23:04.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nwqF/PlVk/niGRdIVn4lNxEirmg=', 'Content-Length': '0'} 2024-02-01T10:23:04.231 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', '6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000012c7a3289633381e-0065bb7108-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:04.231 DEBUG:boto:path=/vclfcn-18/obj-test-zone3 2024-02-01T10:23:04.232 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3 2024-02-01T10:23:04.232 DEBUG:boto:path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:04.232 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:04.232 DEBUG:boto:Method: HEAD 2024-02-01T10:23:04.232 DEBUG:boto:Path: /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:04.232 DEBUG:boto:Data: 2024-02-01T10:23:04.232 DEBUG:boto:Headers: {} 2024-02-01T10:23:04.232 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:23:04.232 DEBUG:boto:Port: 8000 2024-02-01T10:23:04.233 DEBUG:boto:Params: {} 2024-02-01T10:23:04.233 DEBUG:boto:Token: None 2024-02-01T10:23:04.233 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:23:04 GMT /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:04.233 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nwqF/PlVk/niGRdIVn4lNxEirmg= 2024-02-01T10:23:04.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nwqF/PlVk/niGRdIVn4lNxEirmg=', 'Content-Length': '0'} 2024-02-01T10:23:04.236 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', '6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008341632be6a83f3f-0065bb7108-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:04.236 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-02-01T10:23:04.237 DEBUG:boto:path=/vclfcn-18/obj-test-zone3 2024-02-01T10:23:04.237 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3 2024-02-01T10:23:04.237 DEBUG:boto:path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:04.237 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:04.237 DEBUG:boto:Method: GET 2024-02-01T10:23:04.237 DEBUG:boto:Path: /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:04.237 DEBUG:boto:Data: 2024-02-01T10:23:04.237 DEBUG:boto:Headers: {} 2024-02-01T10:23:04.237 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:04.237 DEBUG:boto:Port: 8001 2024-02-01T10:23:04.238 DEBUG:boto:Params: {} 2024-02-01T10:23:04.238 DEBUG:boto:Token: None 2024-02-01T10:23:04.238 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:04 GMT /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:04.238 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:M933E7EC2Q387J/3tWzlA8pDTfs= 2024-02-01T10:23:04.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:M933E7EC2Q387J/3tWzlA8pDTfs=', 'Content-Length': '0'} 2024-02-01T10:23:04.241 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', '6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006a0376f16adfab4b-0065bb7108-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:04.241 DEBUG:boto:path=/vclfcn-18/obj-test-zone3 2024-02-01T10:23:04.241 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3 2024-02-01T10:23:04.242 DEBUG:boto:path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:04.242 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:04.242 DEBUG:boto:Method: GET 2024-02-01T10:23:04.242 DEBUG:boto:Path: /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:04.242 DEBUG:boto:Data: 2024-02-01T10:23:04.242 DEBUG:boto:Headers: {} 2024-02-01T10:23:04.242 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:23:04.242 DEBUG:boto:Port: 8000 2024-02-01T10:23:04.242 DEBUG:boto:Params: {} 2024-02-01T10:23:04.243 DEBUG:boto:Token: None 2024-02-01T10:23:04.243 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:04 GMT /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:04.243 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:M933E7EC2Q387J/3tWzlA8pDTfs= 2024-02-01T10:23:04.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:M933E7EC2Q387J/3tWzlA8pDTfs=', 'Content-Length': '0'} 2024-02-01T10:23:04.246 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', '6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fda807414e4c73c1-0065bb7108-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:04.288 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:23:04.288 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:23:04.288 DEBUG:boto:path=/vclfcn-18/?prefix=obj-test-zone3 2024-02-01T10:23:04.288 DEBUG:boto:auth_path=/vclfcn-18/?prefix=obj-test-zone3 2024-02-01T10:23:04.289 DEBUG:boto:Method: GET 2024-02-01T10:23:04.289 DEBUG:boto:Path: /vclfcn-18/?prefix=obj-test-zone3 2024-02-01T10:23:04.289 DEBUG:boto:Data: 2024-02-01T10:23:04.289 DEBUG:boto:Headers: {} 2024-02-01T10:23:04.289 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:04.289 DEBUG:boto:Port: 8001 2024-02-01T10:23:04.289 DEBUG:boto:Params: {} 2024-02-01T10:23:04.289 DEBUG:boto:Token: None 2024-02-01T10:23:04.290 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:04 GMT /vclfcn-18/ 2024-02-01T10:23:04.290 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:uScQd7oraFpm6C3eTQPDd1dHcc0= 2024-02-01T10:23:04.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:uScQd7oraFpm6C3eTQPDd1dHcc0=', 'Content-Length': '0'} 2024-02-01T10:23:04.294 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b2a091f64d1e192b-0065bb7108-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:04.295 DEBUG:boto:b'vclfcn-18obj-test-zone31000falseobj-test-zone32024-02-01T10:20:51.248Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:04.295 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:23:04.295 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:23:04.295 DEBUG:boto:path=/vclfcn-18/?prefix=obj-test-zone3 2024-02-01T10:23:04.295 DEBUG:boto:auth_path=/vclfcn-18/?prefix=obj-test-zone3 2024-02-01T10:23:04.295 DEBUG:boto:Method: GET 2024-02-01T10:23:04.295 DEBUG:boto:Path: /vclfcn-18/?prefix=obj-test-zone3 2024-02-01T10:23:04.296 DEBUG:boto:Data: 2024-02-01T10:23:04.296 DEBUG:boto:Headers: {} 2024-02-01T10:23:04.296 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:23:04.296 DEBUG:boto:Port: 8000 2024-02-01T10:23:04.296 DEBUG:boto:Params: {} 2024-02-01T10:23:04.296 DEBUG:boto:Token: None 2024-02-01T10:23:04.296 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:04 GMT /vclfcn-18/ 2024-02-01T10:23:04.296 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:uScQd7oraFpm6C3eTQPDd1dHcc0= 2024-02-01T10:23:04.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:uScQd7oraFpm6C3eTQPDd1dHcc0=', 'Content-Length': '0'} 2024-02-01T10:23:04.300 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004945ed8d76c76528-0065bb7108-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:04.348 DEBUG:boto:b'vclfcn-18obj-test-zone31000falseobj-test-zone32024-02-01T10:20:51.248Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:04.349 DEBUG:boto:path=/vclfcn-18/obj-test-zone3 2024-02-01T10:23:04.349 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3 2024-02-01T10:23:04.349 DEBUG:boto:Method: HEAD 2024-02-01T10:23:04.349 DEBUG:boto:Path: /vclfcn-18/obj-test-zone3 2024-02-01T10:23:04.349 DEBUG:boto:Data: 2024-02-01T10:23:04.349 DEBUG:boto:Headers: {} 2024-02-01T10:23:04.349 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:04.349 DEBUG:boto:Port: 8001 2024-02-01T10:23:04.349 DEBUG:boto:Params: {} 2024-02-01T10:23:04.350 DEBUG:boto:Token: None 2024-02-01T10:23:04.350 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:23:04 GMT /vclfcn-18/obj-test-zone3 2024-02-01T10:23:04.350 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:v9aAy3YVjcs85WeJztCgdhuYrdo= 2024-02-01T10:23:04.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:v9aAy3YVjcs85WeJztCgdhuYrdo=', 'Content-Length': '0'} 2024-02-01T10:23:04.356 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', '6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006d665f5202603c29-0065bb7108-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:04.356 DEBUG:boto:path=/vclfcn-18/obj-test-zone3 2024-02-01T10:23:04.356 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3 2024-02-01T10:23:04.356 DEBUG:boto:Method: HEAD 2024-02-01T10:23:04.356 DEBUG:boto:Path: /vclfcn-18/obj-test-zone3 2024-02-01T10:23:04.356 DEBUG:boto:Data: 2024-02-01T10:23:04.356 DEBUG:boto:Headers: {} 2024-02-01T10:23:04.357 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:23:04.357 DEBUG:boto:Port: 8000 2024-02-01T10:23:04.357 DEBUG:boto:Params: {} 2024-02-01T10:23:04.357 DEBUG:boto:Token: None 2024-02-01T10:23:04.357 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:23:04 GMT /vclfcn-18/obj-test-zone3 2024-02-01T10:23:04.357 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:v9aAy3YVjcs85WeJztCgdhuYrdo= 2024-02-01T10:23:04.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:v9aAy3YVjcs85WeJztCgdhuYrdo=', 'Content-Length': '0'} 2024-02-01T10:23:04.361 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', '6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007ef50c65182f504f-0065bb7108-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:04.361 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-02-01T10:23:04.361 DEBUG:boto:path=/vclfcn-18/obj-test-zone3 2024-02-01T10:23:04.361 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3 2024-02-01T10:23:04.361 DEBUG:boto:path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:04.361 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:04.361 DEBUG:boto:Method: GET 2024-02-01T10:23:04.362 DEBUG:boto:Path: /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:04.362 DEBUG:boto:Data: 2024-02-01T10:23:04.362 DEBUG:boto:Headers: {} 2024-02-01T10:23:04.362 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:04.362 DEBUG:boto:Port: 8001 2024-02-01T10:23:04.362 DEBUG:boto:Params: {} 2024-02-01T10:23:04.362 DEBUG:boto:Token: None 2024-02-01T10:23:04.362 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:04 GMT /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:04.362 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:M933E7EC2Q387J/3tWzlA8pDTfs= 2024-02-01T10:23:04.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:M933E7EC2Q387J/3tWzlA8pDTfs=', 'Content-Length': '0'} 2024-02-01T10:23:04.365 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', '6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000defb1c1d73fcae3c-0065bb7108-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:04.365 DEBUG:boto:path=/vclfcn-18/obj-test-zone3 2024-02-01T10:23:04.366 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3 2024-02-01T10:23:04.366 DEBUG:boto:path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:04.366 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:04.366 DEBUG:boto:Method: GET 2024-02-01T10:23:04.366 DEBUG:boto:Path: /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:04.366 DEBUG:boto:Data: 2024-02-01T10:23:04.367 DEBUG:boto:Headers: {} 2024-02-01T10:23:04.367 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:23:04.367 DEBUG:boto:Port: 8000 2024-02-01T10:23:04.367 DEBUG:boto:Params: {} 2024-02-01T10:23:04.367 DEBUG:boto:Token: None 2024-02-01T10:23:04.367 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:04 GMT /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:04.367 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:M933E7EC2Q387J/3tWzlA8pDTfs= 2024-02-01T10:23:04.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:M933E7EC2Q387J/3tWzlA8pDTfs=', 'Content-Length': '0'} 2024-02-01T10:23:04.370 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', '6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000db356a6a32e8b3a0-0065bb7108-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:04.412 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:23:04.412 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:23:04.412 DEBUG:boto:path=/vclfcn-18/?prefix=obj-test-zone3 2024-02-01T10:23:04.412 DEBUG:boto:auth_path=/vclfcn-18/?prefix=obj-test-zone3 2024-02-01T10:23:04.413 DEBUG:boto:Method: GET 2024-02-01T10:23:04.413 DEBUG:boto:Path: /vclfcn-18/?prefix=obj-test-zone3 2024-02-01T10:23:04.413 DEBUG:boto:Data: 2024-02-01T10:23:04.413 DEBUG:boto:Headers: {} 2024-02-01T10:23:04.413 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:04.413 DEBUG:boto:Port: 8001 2024-02-01T10:23:04.413 DEBUG:boto:Params: {} 2024-02-01T10:23:04.413 DEBUG:boto:Token: None 2024-02-01T10:23:04.413 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:04 GMT /vclfcn-18/ 2024-02-01T10:23:04.413 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:uScQd7oraFpm6C3eTQPDd1dHcc0= 2024-02-01T10:23:04.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:uScQd7oraFpm6C3eTQPDd1dHcc0=', 'Content-Length': '0'} 2024-02-01T10:23:04.417 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ecd5d60ae25b453f-0065bb7108-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:04.417 DEBUG:boto:b'vclfcn-18obj-test-zone31000falseobj-test-zone32024-02-01T10:20:51.248Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:04.418 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:23:04.418 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:23:04.418 DEBUG:boto:path=/vclfcn-18/?prefix=obj-test-zone3 2024-02-01T10:23:04.418 DEBUG:boto:auth_path=/vclfcn-18/?prefix=obj-test-zone3 2024-02-01T10:23:04.418 DEBUG:boto:Method: GET 2024-02-01T10:23:04.418 DEBUG:boto:Path: /vclfcn-18/?prefix=obj-test-zone3 2024-02-01T10:23:04.419 DEBUG:boto:Data: 2024-02-01T10:23:04.419 DEBUG:boto:Headers: {} 2024-02-01T10:23:04.419 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:23:04.419 DEBUG:boto:Port: 8000 2024-02-01T10:23:04.419 DEBUG:boto:Params: {} 2024-02-01T10:23:04.419 DEBUG:boto:Token: None 2024-02-01T10:23:04.419 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:04 GMT /vclfcn-18/ 2024-02-01T10:23:04.419 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:uScQd7oraFpm6C3eTQPDd1dHcc0= 2024-02-01T10:23:04.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:uScQd7oraFpm6C3eTQPDd1dHcc0=', 'Content-Length': '0'} 2024-02-01T10:23:04.423 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ddb76798710a6f7e-0065bb7108-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:04.464 DEBUG:boto:b'vclfcn-18obj-test-zone31000falseobj-test-zone32024-02-01T10:20:51.248Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:04.465 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-18 zones={test-zone3, test-zone1} 2024-02-01T10:23:04.465 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-18 zones={test-zone3, test-zone2} 2024-02-01T10:23:04.465 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:23:04.465 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:23:04.465 DEBUG:boto:Method: HEAD 2024-02-01T10:23:04.465 DEBUG:boto:Path: /vclfcn-18/ 2024-02-01T10:23:04.465 DEBUG:boto:Data: 2024-02-01T10:23:04.465 DEBUG:boto:Headers: {} 2024-02-01T10:23:04.465 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:04.465 DEBUG:boto:Port: 8001 2024-02-01T10:23:04.466 DEBUG:boto:Params: {} 2024-02-01T10:23:04.466 DEBUG:boto:Token: None 2024-02-01T10:23:04.466 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:23:04 GMT /vclfcn-18/ 2024-02-01T10:23:04.466 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7ErLFkCz+3QMPg28kYLwe6eHEM0= 2024-02-01T10:23:04.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7ErLFkCz+3QMPg28kYLwe6eHEM0=', 'Content-Length': '0'} 2024-02-01T10:23:04.470 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c10b4b2894bcc9b5-0065bb7108-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:23:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:04.471 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:23:04.471 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:23:04.471 DEBUG:boto:Method: HEAD 2024-02-01T10:23:04.471 DEBUG:boto:Path: /vclfcn-18/ 2024-02-01T10:23:04.471 DEBUG:boto:Data: 2024-02-01T10:23:04.471 DEBUG:boto:Headers: {} 2024-02-01T10:23:04.471 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:04.471 DEBUG:boto:Port: 8000 2024-02-01T10:23:04.471 DEBUG:boto:Params: {} 2024-02-01T10:23:04.472 DEBUG:boto:Token: None 2024-02-01T10:23:04.472 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:23:04 GMT /vclfcn-18/ 2024-02-01T10:23:04.472 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7ErLFkCz+3QMPg28kYLwe6eHEM0= 2024-02-01T10:23:04.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7ErLFkCz+3QMPg28kYLwe6eHEM0=', 'Content-Length': '0'} 2024-02-01T10:23:04.476 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e4463683ea34c8cb-0065bb7108-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:23:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:04.476 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:23:04.476 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:23:04.476 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:23:04.477 DEBUG:boto:path=/vclfcn-18/?versions&max-keys=999 2024-02-01T10:23:04.477 DEBUG:boto:auth_path=/vclfcn-18/?versions&max-keys=999 2024-02-01T10:23:04.477 DEBUG:boto:Method: GET 2024-02-01T10:23:04.477 DEBUG:boto:Path: /vclfcn-18/?versions&max-keys=999 2024-02-01T10:23:04.477 DEBUG:boto:Data: 2024-02-01T10:23:04.477 DEBUG:boto:Headers: {} 2024-02-01T10:23:04.477 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:04.477 DEBUG:boto:Port: 8001 2024-02-01T10:23:04.477 DEBUG:boto:Params: {} 2024-02-01T10:23:04.477 DEBUG:boto:Token: None 2024-02-01T10:23:04.478 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:04 GMT /vclfcn-18/?versions 2024-02-01T10:23:04.478 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:F4/ycEv7yhs16QsfLehdPcVMmfQ= 2024-02-01T10:23:04.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:F4/ycEv7yhs16QsfLehdPcVMmfQ=', 'Content-Length': '0'} 2024-02-01T10:23:04.482 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dfe6930bb5e91b9f-0065bb7108-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:04.482 DEBUG:boto:b'vclfcn-18999falsedummytxgLlJWZVbWbjqLWqkOtt87dUoD4zuntrue2024-02-01T10:20:34.645Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcKtrue2024-02-01T10:20:50.942Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutDtrue2024-02-01T10:20:51.094Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone36pAVLbw3HAwE0jxxbuxTIxwcA2p4q8Ntrue2024-02-01T10:20:51.248Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:04.483 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-02-01T10:23:04.483 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-02-01T10:23:04.483 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-02-01T10:23:04.483 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-02-01T10:23:04.483 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:23:04.483 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:23:04.483 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:23:04.483 DEBUG:boto:path=/vclfcn-18/?versions&max-keys=999 2024-02-01T10:23:04.483 DEBUG:boto:auth_path=/vclfcn-18/?versions&max-keys=999 2024-02-01T10:23:04.484 DEBUG:boto:Method: GET 2024-02-01T10:23:04.484 DEBUG:boto:Path: /vclfcn-18/?versions&max-keys=999 2024-02-01T10:23:04.484 DEBUG:boto:Data: 2024-02-01T10:23:04.484 DEBUG:boto:Headers: {} 2024-02-01T10:23:04.484 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:04.484 DEBUG:boto:Port: 8000 2024-02-01T10:23:04.484 DEBUG:boto:Params: {} 2024-02-01T10:23:04.484 DEBUG:boto:Token: None 2024-02-01T10:23:04.484 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:04 GMT /vclfcn-18/?versions 2024-02-01T10:23:04.485 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:F4/ycEv7yhs16QsfLehdPcVMmfQ= 2024-02-01T10:23:04.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:F4/ycEv7yhs16QsfLehdPcVMmfQ=', 'Content-Length': '0'} 2024-02-01T10:23:04.489 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f608629d68f9e28-0065bb7108-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:04.489 DEBUG:boto:b'vclfcn-18999falsedummytxgLlJWZVbWbjqLWqkOtt87dUoD4zuntrue2024-02-01T10:20:34.645Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcKtrue2024-02-01T10:20:50.942Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutDtrue2024-02-01T10:20:51.094Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone36pAVLbw3HAwE0jxxbuxTIxwcA2p4q8Ntrue2024-02-01T10:20:51.248Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:04.490 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-02-01T10:23:04.490 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-02-01T10:23:04.490 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-02-01T10:23:04.490 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-02-01T10:23:04.490 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:23:04.490 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:23:04.490 DEBUG:boto:path=/vclfcn-18/?versions&max-keys=999 2024-02-01T10:23:04.490 DEBUG:boto:auth_path=/vclfcn-18/?versions&max-keys=999 2024-02-01T10:23:04.490 DEBUG:boto:Method: GET 2024-02-01T10:23:04.491 DEBUG:boto:Path: /vclfcn-18/?versions&max-keys=999 2024-02-01T10:23:04.491 DEBUG:boto:Data: 2024-02-01T10:23:04.491 DEBUG:boto:Headers: {} 2024-02-01T10:23:04.491 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:04.491 DEBUG:boto:Port: 8001 2024-02-01T10:23:04.491 DEBUG:boto:Params: {} 2024-02-01T10:23:04.491 DEBUG:boto:Token: None 2024-02-01T10:23:04.491 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:04 GMT /vclfcn-18/?versions 2024-02-01T10:23:04.491 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:F4/ycEv7yhs16QsfLehdPcVMmfQ= 2024-02-01T10:23:04.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:F4/ycEv7yhs16QsfLehdPcVMmfQ=', 'Content-Length': '0'} 2024-02-01T10:23:04.496 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ca4f951d345b8cb-0065bb7108-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:04.496 DEBUG:boto:b'vclfcn-18999falsedummytxgLlJWZVbWbjqLWqkOtt87dUoD4zuntrue2024-02-01T10:20:34.645Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcKtrue2024-02-01T10:20:50.942Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutDtrue2024-02-01T10:20:51.094Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone36pAVLbw3HAwE0jxxbuxTIxwcA2p4q8Ntrue2024-02-01T10:20:51.248Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:04.497 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:23:04.497 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:23:04.497 DEBUG:boto:path=/vclfcn-18/?versions&max-keys=999 2024-02-01T10:23:04.497 DEBUG:boto:auth_path=/vclfcn-18/?versions&max-keys=999 2024-02-01T10:23:04.497 DEBUG:boto:Method: GET 2024-02-01T10:23:04.497 DEBUG:boto:Path: /vclfcn-18/?versions&max-keys=999 2024-02-01T10:23:04.497 DEBUG:boto:Data: 2024-02-01T10:23:04.497 DEBUG:boto:Headers: {} 2024-02-01T10:23:04.497 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:04.497 DEBUG:boto:Port: 8000 2024-02-01T10:23:04.498 DEBUG:boto:Params: {} 2024-02-01T10:23:04.498 DEBUG:boto:Token: None 2024-02-01T10:23:04.498 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:04 GMT /vclfcn-18/?versions 2024-02-01T10:23:04.498 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:F4/ycEv7yhs16QsfLehdPcVMmfQ= 2024-02-01T10:23:04.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:F4/ycEv7yhs16QsfLehdPcVMmfQ=', 'Content-Length': '0'} 2024-02-01T10:23:04.502 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f80fcf064b85fbf6-0065bb7108-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:04.502 DEBUG:boto:b'vclfcn-18999falsedummytxgLlJWZVbWbjqLWqkOtt87dUoD4zuntrue2024-02-01T10:20:34.645Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcKtrue2024-02-01T10:20:50.942Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutDtrue2024-02-01T10:20:51.094Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone36pAVLbw3HAwE0jxxbuxTIxwcA2p4q8Ntrue2024-02-01T10:20:51.248Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:04.503 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-02-01T10:23:04.503 DEBUG:boto:path=/vclfcn-18/dummy 2024-02-01T10:23:04.503 DEBUG:boto:auth_path=/vclfcn-18/dummy 2024-02-01T10:23:04.503 DEBUG:boto:path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:04.503 DEBUG:boto:auth_path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:04.504 DEBUG:boto:Method: GET 2024-02-01T10:23:04.504 DEBUG:boto:Path: /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:04.504 DEBUG:boto:Data: 2024-02-01T10:23:04.504 DEBUG:boto:Headers: {} 2024-02-01T10:23:04.504 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:04.504 DEBUG:boto:Port: 8001 2024-02-01T10:23:04.504 DEBUG:boto:Params: {} 2024-02-01T10:23:04.504 DEBUG:boto:Token: None 2024-02-01T10:23:04.504 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:04 GMT /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:04.504 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:n6LOQe9BObbLMqU4c0APvTo7tpc= 2024-02-01T10:23:04.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:n6LOQe9BObbLMqU4c0APvTo7tpc=', 'Content-Length': '0'} 2024-02-01T10:23:04.507 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:34 GMT'), ('x-amz-version-id', 'txgLlJWZVbWbjqLWqkOtt87dUoD4zun'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003b82f092b4f583d5-0065bb7108-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:04.507 DEBUG:boto:path=/vclfcn-18/dummy 2024-02-01T10:23:04.508 DEBUG:boto:auth_path=/vclfcn-18/dummy 2024-02-01T10:23:04.508 DEBUG:boto:path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:04.508 DEBUG:boto:auth_path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:04.508 DEBUG:boto:Method: GET 2024-02-01T10:23:04.508 DEBUG:boto:Path: /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:04.508 DEBUG:boto:Data: 2024-02-01T10:23:04.508 DEBUG:boto:Headers: {} 2024-02-01T10:23:04.508 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:04.508 DEBUG:boto:Port: 8000 2024-02-01T10:23:04.508 DEBUG:boto:Params: {} 2024-02-01T10:23:04.509 DEBUG:boto:Token: None 2024-02-01T10:23:04.509 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:04 GMT /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:04.509 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:n6LOQe9BObbLMqU4c0APvTo7tpc= 2024-02-01T10:23:04.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:n6LOQe9BObbLMqU4c0APvTo7tpc=', 'Content-Length': '0'} 2024-02-01T10:23:04.512 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:34 GMT'), ('x-amz-version-id', 'txgLlJWZVbWbjqLWqkOtt87dUoD4zun'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000093ca8bfcde315145-0065bb7108-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:04.512 DEBUG:boto:path=/vclfcn-18/dummy 2024-02-01T10:23:04.512 DEBUG:boto:auth_path=/vclfcn-18/dummy 2024-02-01T10:23:04.512 DEBUG:boto:path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:04.512 DEBUG:boto:auth_path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:04.512 DEBUG:boto:Method: HEAD 2024-02-01T10:23:04.512 DEBUG:boto:Path: /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:04.513 DEBUG:boto:Data: 2024-02-01T10:23:04.513 DEBUG:boto:Headers: {} 2024-02-01T10:23:04.513 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:04.513 DEBUG:boto:Port: 8001 2024-02-01T10:23:04.513 DEBUG:boto:Params: {} 2024-02-01T10:23:04.513 DEBUG:boto:Token: None 2024-02-01T10:23:04.513 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:23:04 GMT /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:04.513 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:dV65i8DfFlQnd0Qo7ZsWpBmJ6bc= 2024-02-01T10:23:04.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dV65i8DfFlQnd0Qo7ZsWpBmJ6bc=', 'Content-Length': '0'} 2024-02-01T10:23:04.516 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:34 GMT'), ('x-amz-version-id', 'txgLlJWZVbWbjqLWqkOtt87dUoD4zun'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000481406d7954b9152-0065bb7108-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:04.516 DEBUG:boto:path=/vclfcn-18/dummy 2024-02-01T10:23:04.516 DEBUG:boto:auth_path=/vclfcn-18/dummy 2024-02-01T10:23:04.516 DEBUG:boto:path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:04.517 DEBUG:boto:auth_path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:04.517 DEBUG:boto:Method: HEAD 2024-02-01T10:23:04.517 DEBUG:boto:Path: /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:04.517 DEBUG:boto:Data: 2024-02-01T10:23:04.517 DEBUG:boto:Headers: {} 2024-02-01T10:23:04.517 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:04.517 DEBUG:boto:Port: 8000 2024-02-01T10:23:04.517 DEBUG:boto:Params: {} 2024-02-01T10:23:04.517 DEBUG:boto:Token: None 2024-02-01T10:23:04.517 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:23:04 GMT /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:04.518 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:dV65i8DfFlQnd0Qo7ZsWpBmJ6bc= 2024-02-01T10:23:04.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dV65i8DfFlQnd0Qo7ZsWpBmJ6bc=', 'Content-Length': '0'} 2024-02-01T10:23:04.520 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:34 GMT'), ('x-amz-version-id', 'txgLlJWZVbWbjqLWqkOtt87dUoD4zun'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ad3f579ffe28f325-0065bb7108-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:04.520 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-02-01T10:23:04.521 DEBUG:boto:path=/vclfcn-18/dummy 2024-02-01T10:23:04.521 DEBUG:boto:auth_path=/vclfcn-18/dummy 2024-02-01T10:23:04.521 DEBUG:boto:path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:04.521 DEBUG:boto:auth_path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:04.521 DEBUG:boto:Method: GET 2024-02-01T10:23:04.521 DEBUG:boto:Path: /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:04.521 DEBUG:boto:Data: 2024-02-01T10:23:04.521 DEBUG:boto:Headers: {} 2024-02-01T10:23:04.521 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:04.521 DEBUG:boto:Port: 8001 2024-02-01T10:23:04.522 DEBUG:boto:Params: {} 2024-02-01T10:23:04.522 DEBUG:boto:Token: None 2024-02-01T10:23:04.522 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:04 GMT /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:04.522 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:n6LOQe9BObbLMqU4c0APvTo7tpc= 2024-02-01T10:23:04.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:n6LOQe9BObbLMqU4c0APvTo7tpc=', 'Content-Length': '0'} 2024-02-01T10:23:04.525 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:34 GMT'), ('x-amz-version-id', 'txgLlJWZVbWbjqLWqkOtt87dUoD4zun'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e0d202f4743b6f45-0065bb7108-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:04.525 DEBUG:boto:path=/vclfcn-18/dummy 2024-02-01T10:23:04.525 DEBUG:boto:auth_path=/vclfcn-18/dummy 2024-02-01T10:23:04.525 DEBUG:boto:path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:04.525 DEBUG:boto:auth_path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:04.526 DEBUG:boto:Method: GET 2024-02-01T10:23:04.526 DEBUG:boto:Path: /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:04.526 DEBUG:boto:Data: 2024-02-01T10:23:04.526 DEBUG:boto:Headers: {} 2024-02-01T10:23:04.526 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:04.526 DEBUG:boto:Port: 8000 2024-02-01T10:23:04.526 DEBUG:boto:Params: {} 2024-02-01T10:23:04.526 DEBUG:boto:Token: None 2024-02-01T10:23:04.526 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:04 GMT /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:04.526 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:n6LOQe9BObbLMqU4c0APvTo7tpc= 2024-02-01T10:23:04.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:n6LOQe9BObbLMqU4c0APvTo7tpc=', 'Content-Length': '0'} 2024-02-01T10:23:04.529 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:34 GMT'), ('x-amz-version-id', 'txgLlJWZVbWbjqLWqkOtt87dUoD4zun'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000220de7d1f5a5dd48-0065bb7108-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:04.529 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:23:04.529 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:23:04.530 DEBUG:boto:path=/vclfcn-18/?prefix=dummy 2024-02-01T10:23:04.530 DEBUG:boto:auth_path=/vclfcn-18/?prefix=dummy 2024-02-01T10:23:04.530 DEBUG:boto:Method: GET 2024-02-01T10:23:04.530 DEBUG:boto:Path: /vclfcn-18/?prefix=dummy 2024-02-01T10:23:04.530 DEBUG:boto:Data: 2024-02-01T10:23:04.530 DEBUG:boto:Headers: {} 2024-02-01T10:23:04.530 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:04.530 DEBUG:boto:Port: 8001 2024-02-01T10:23:04.530 DEBUG:boto:Params: {} 2024-02-01T10:23:04.530 DEBUG:boto:Token: None 2024-02-01T10:23:04.531 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:04 GMT /vclfcn-18/ 2024-02-01T10:23:04.531 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:uScQd7oraFpm6C3eTQPDd1dHcc0= 2024-02-01T10:23:04.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:uScQd7oraFpm6C3eTQPDd1dHcc0=', 'Content-Length': '0'} 2024-02-01T10:23:04.535 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000052a611eadf2f543-0065bb7108-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:04.580 DEBUG:boto:b'vclfcn-18dummy1000falsedummy2024-02-01T10:20:34.645Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:04.581 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:23:04.581 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:23:04.581 DEBUG:boto:path=/vclfcn-18/?prefix=dummy 2024-02-01T10:23:04.581 DEBUG:boto:auth_path=/vclfcn-18/?prefix=dummy 2024-02-01T10:23:04.581 DEBUG:boto:Method: GET 2024-02-01T10:23:04.581 DEBUG:boto:Path: /vclfcn-18/?prefix=dummy 2024-02-01T10:23:04.581 DEBUG:boto:Data: 2024-02-01T10:23:04.581 DEBUG:boto:Headers: {} 2024-02-01T10:23:04.581 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:04.581 DEBUG:boto:Port: 8000 2024-02-01T10:23:04.581 DEBUG:boto:Params: {} 2024-02-01T10:23:04.582 DEBUG:boto:Token: None 2024-02-01T10:23:04.582 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:04 GMT /vclfcn-18/ 2024-02-01T10:23:04.582 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:uScQd7oraFpm6C3eTQPDd1dHcc0= 2024-02-01T10:23:04.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:uScQd7oraFpm6C3eTQPDd1dHcc0=', 'Content-Length': '0'} 2024-02-01T10:23:04.586 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000506a544927df112e-0065bb7108-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:04.586 DEBUG:boto:b'vclfcn-18dummy1000falsedummy2024-02-01T10:20:34.645Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:04.586 DEBUG:boto:path=/vclfcn-18/dummy 2024-02-01T10:23:04.586 DEBUG:boto:auth_path=/vclfcn-18/dummy 2024-02-01T10:23:04.586 DEBUG:boto:Method: HEAD 2024-02-01T10:23:04.586 DEBUG:boto:Path: /vclfcn-18/dummy 2024-02-01T10:23:04.587 DEBUG:boto:Data: 2024-02-01T10:23:04.587 DEBUG:boto:Headers: {} 2024-02-01T10:23:04.587 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:04.587 DEBUG:boto:Port: 8001 2024-02-01T10:23:04.587 DEBUG:boto:Params: {} 2024-02-01T10:23:04.587 DEBUG:boto:Token: None 2024-02-01T10:23:04.587 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:23:04 GMT /vclfcn-18/dummy 2024-02-01T10:23:04.587 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:SCsvqAeZUCxjzaNYCZMemrmwaTU= 2024-02-01T10:23:04.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:SCsvqAeZUCxjzaNYCZMemrmwaTU=', 'Content-Length': '0'} 2024-02-01T10:23:04.591 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:34 GMT'), ('x-amz-version-id', 'txgLlJWZVbWbjqLWqkOtt87dUoD4zun'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000462346d4d3728576-0065bb7108-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:04.592 DEBUG:boto:path=/vclfcn-18/dummy 2024-02-01T10:23:04.592 DEBUG:boto:auth_path=/vclfcn-18/dummy 2024-02-01T10:23:04.592 DEBUG:boto:Method: HEAD 2024-02-01T10:23:04.592 DEBUG:boto:Path: /vclfcn-18/dummy 2024-02-01T10:23:04.592 DEBUG:boto:Data: 2024-02-01T10:23:04.592 DEBUG:boto:Headers: {} 2024-02-01T10:23:04.592 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:04.592 DEBUG:boto:Port: 8000 2024-02-01T10:23:04.592 DEBUG:boto:Params: {} 2024-02-01T10:23:04.592 DEBUG:boto:Token: None 2024-02-01T10:23:04.593 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:23:04 GMT /vclfcn-18/dummy 2024-02-01T10:23:04.593 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:SCsvqAeZUCxjzaNYCZMemrmwaTU= 2024-02-01T10:23:04.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:SCsvqAeZUCxjzaNYCZMemrmwaTU=', 'Content-Length': '0'} 2024-02-01T10:23:04.596 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:34 GMT'), ('x-amz-version-id', 'txgLlJWZVbWbjqLWqkOtt87dUoD4zun'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000031502926c63b9027-0065bb7108-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:04.596 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-02-01T10:23:04.597 DEBUG:boto:path=/vclfcn-18/dummy 2024-02-01T10:23:04.597 DEBUG:boto:auth_path=/vclfcn-18/dummy 2024-02-01T10:23:04.597 DEBUG:boto:path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:04.597 DEBUG:boto:auth_path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:04.597 DEBUG:boto:Method: GET 2024-02-01T10:23:04.597 DEBUG:boto:Path: /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:04.597 DEBUG:boto:Data: 2024-02-01T10:23:04.597 DEBUG:boto:Headers: {} 2024-02-01T10:23:04.597 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:04.597 DEBUG:boto:Port: 8001 2024-02-01T10:23:04.597 DEBUG:boto:Params: {} 2024-02-01T10:23:04.598 DEBUG:boto:Token: None 2024-02-01T10:23:04.598 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:04 GMT /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:04.598 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:n6LOQe9BObbLMqU4c0APvTo7tpc= 2024-02-01T10:23:04.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:n6LOQe9BObbLMqU4c0APvTo7tpc=', 'Content-Length': '0'} 2024-02-01T10:23:04.601 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:34 GMT'), ('x-amz-version-id', 'txgLlJWZVbWbjqLWqkOtt87dUoD4zun'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000505879b0ffda192c-0065bb7108-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:04.601 DEBUG:boto:path=/vclfcn-18/dummy 2024-02-01T10:23:04.601 DEBUG:boto:auth_path=/vclfcn-18/dummy 2024-02-01T10:23:04.601 DEBUG:boto:path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:04.601 DEBUG:boto:auth_path=/vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:04.601 DEBUG:boto:Method: GET 2024-02-01T10:23:04.602 DEBUG:boto:Path: /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:04.602 DEBUG:boto:Data: 2024-02-01T10:23:04.602 DEBUG:boto:Headers: {} 2024-02-01T10:23:04.602 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:04.602 DEBUG:boto:Port: 8000 2024-02-01T10:23:04.602 DEBUG:boto:Params: {} 2024-02-01T10:23:04.602 DEBUG:boto:Token: None 2024-02-01T10:23:04.602 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:04 GMT /vclfcn-18/dummy?versionId=txgLlJWZVbWbjqLWqkOtt87dUoD4zun 2024-02-01T10:23:04.602 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:n6LOQe9BObbLMqU4c0APvTo7tpc= 2024-02-01T10:23:04.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:n6LOQe9BObbLMqU4c0APvTo7tpc=', 'Content-Length': '0'} 2024-02-01T10:23:04.605 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:34 GMT'), ('x-amz-version-id', 'txgLlJWZVbWbjqLWqkOtt87dUoD4zun'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a57b19235374e40b-0065bb7108-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:04.605 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:23:04.606 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:23:04.606 DEBUG:boto:path=/vclfcn-18/?prefix=dummy 2024-02-01T10:23:04.606 DEBUG:boto:auth_path=/vclfcn-18/?prefix=dummy 2024-02-01T10:23:04.606 DEBUG:boto:Method: GET 2024-02-01T10:23:04.606 DEBUG:boto:Path: /vclfcn-18/?prefix=dummy 2024-02-01T10:23:04.606 DEBUG:boto:Data: 2024-02-01T10:23:04.606 DEBUG:boto:Headers: {} 2024-02-01T10:23:04.606 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:04.606 DEBUG:boto:Port: 8001 2024-02-01T10:23:04.606 DEBUG:boto:Params: {} 2024-02-01T10:23:04.607 DEBUG:boto:Token: None 2024-02-01T10:23:04.607 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:04 GMT /vclfcn-18/ 2024-02-01T10:23:04.607 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:uScQd7oraFpm6C3eTQPDd1dHcc0= 2024-02-01T10:23:04.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:uScQd7oraFpm6C3eTQPDd1dHcc0=', 'Content-Length': '0'} 2024-02-01T10:23:04.611 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d427ed52c1ff7d19-0065bb7108-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:04.652 DEBUG:boto:b'vclfcn-18dummy1000falsedummy2024-02-01T10:20:34.645Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:04.653 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:23:04.653 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:23:04.653 DEBUG:boto:path=/vclfcn-18/?prefix=dummy 2024-02-01T10:23:04.653 DEBUG:boto:auth_path=/vclfcn-18/?prefix=dummy 2024-02-01T10:23:04.653 DEBUG:boto:Method: GET 2024-02-01T10:23:04.653 DEBUG:boto:Path: /vclfcn-18/?prefix=dummy 2024-02-01T10:23:04.653 DEBUG:boto:Data: 2024-02-01T10:23:04.653 DEBUG:boto:Headers: {} 2024-02-01T10:23:04.653 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:04.654 DEBUG:boto:Port: 8000 2024-02-01T10:23:04.654 DEBUG:boto:Params: {} 2024-02-01T10:23:04.654 DEBUG:boto:Token: None 2024-02-01T10:23:04.654 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:04 GMT /vclfcn-18/ 2024-02-01T10:23:04.654 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:uScQd7oraFpm6C3eTQPDd1dHcc0= 2024-02-01T10:23:04.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:uScQd7oraFpm6C3eTQPDd1dHcc0=', 'Content-Length': '0'} 2024-02-01T10:23:04.658 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000318256b5820ca70b-0065bb7108-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:04.700 DEBUG:boto:b'vclfcn-18dummy1000falsedummy2024-02-01T10:20:34.645Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:04.700 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-02-01T10:23:04.701 DEBUG:boto:path=/vclfcn-18/obj-test-zone1 2024-02-01T10:23:04.701 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1 2024-02-01T10:23:04.701 DEBUG:boto:path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:04.701 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:04.701 DEBUG:boto:Method: GET 2024-02-01T10:23:04.701 DEBUG:boto:Path: /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:04.701 DEBUG:boto:Data: 2024-02-01T10:23:04.701 DEBUG:boto:Headers: {} 2024-02-01T10:23:04.701 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:04.701 DEBUG:boto:Port: 8001 2024-02-01T10:23:04.702 DEBUG:boto:Params: {} 2024-02-01T10:23:04.702 DEBUG:boto:Token: None 2024-02-01T10:23:04.702 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:04 GMT /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:04.702 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sioxoXm99E2YM8RkenaYe9yOlxY= 2024-02-01T10:23:04.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sioxoXm99E2YM8RkenaYe9yOlxY=', 'Content-Length': '0'} 2024-02-01T10:23:04.705 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bf85485d1091952b-0065bb7108-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:04.705 DEBUG:boto:path=/vclfcn-18/obj-test-zone1 2024-02-01T10:23:04.705 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1 2024-02-01T10:23:04.705 DEBUG:boto:path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:04.705 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:04.705 DEBUG:boto:Method: GET 2024-02-01T10:23:04.706 DEBUG:boto:Path: /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:04.706 DEBUG:boto:Data: 2024-02-01T10:23:04.706 DEBUG:boto:Headers: {} 2024-02-01T10:23:04.706 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:04.706 DEBUG:boto:Port: 8000 2024-02-01T10:23:04.706 DEBUG:boto:Params: {} 2024-02-01T10:23:04.706 DEBUG:boto:Token: None 2024-02-01T10:23:04.706 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:04 GMT /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:04.706 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sioxoXm99E2YM8RkenaYe9yOlxY= 2024-02-01T10:23:04.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sioxoXm99E2YM8RkenaYe9yOlxY=', 'Content-Length': '0'} 2024-02-01T10:23:04.709 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009619d7381c9c43b1-0065bb7108-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:04.710 DEBUG:boto:path=/vclfcn-18/obj-test-zone1 2024-02-01T10:23:04.710 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1 2024-02-01T10:23:04.710 DEBUG:boto:path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:04.710 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:04.710 DEBUG:boto:Method: HEAD 2024-02-01T10:23:04.710 DEBUG:boto:Path: /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:04.710 DEBUG:boto:Data: 2024-02-01T10:23:04.710 DEBUG:boto:Headers: {} 2024-02-01T10:23:04.711 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:04.711 DEBUG:boto:Port: 8001 2024-02-01T10:23:04.711 DEBUG:boto:Params: {} 2024-02-01T10:23:04.711 DEBUG:boto:Token: None 2024-02-01T10:23:04.711 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:23:04 GMT /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:04.711 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nHlzdUiSuucohgYjdRUD9ByQKYc= 2024-02-01T10:23:04.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nHlzdUiSuucohgYjdRUD9ByQKYc=', 'Content-Length': '0'} 2024-02-01T10:23:04.714 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006df4a66af9e294f5-0065bb7108-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:04.714 DEBUG:boto:path=/vclfcn-18/obj-test-zone1 2024-02-01T10:23:04.715 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1 2024-02-01T10:23:04.715 DEBUG:boto:path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:04.715 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:04.715 DEBUG:boto:Method: HEAD 2024-02-01T10:23:04.715 DEBUG:boto:Path: /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:04.715 DEBUG:boto:Data: 2024-02-01T10:23:04.715 DEBUG:boto:Headers: {} 2024-02-01T10:23:04.715 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:04.715 DEBUG:boto:Port: 8000 2024-02-01T10:23:04.715 DEBUG:boto:Params: {} 2024-02-01T10:23:04.716 DEBUG:boto:Token: None 2024-02-01T10:23:04.716 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:23:04 GMT /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:04.716 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nHlzdUiSuucohgYjdRUD9ByQKYc= 2024-02-01T10:23:04.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nHlzdUiSuucohgYjdRUD9ByQKYc=', 'Content-Length': '0'} 2024-02-01T10:23:04.719 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c72f921c6b882e8a-0065bb7108-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:04.719 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-02-01T10:23:04.719 DEBUG:boto:path=/vclfcn-18/obj-test-zone1 2024-02-01T10:23:04.719 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1 2024-02-01T10:23:04.719 DEBUG:boto:path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:04.719 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:04.720 DEBUG:boto:Method: GET 2024-02-01T10:23:04.720 DEBUG:boto:Path: /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:04.720 DEBUG:boto:Data: 2024-02-01T10:23:04.720 DEBUG:boto:Headers: {} 2024-02-01T10:23:04.720 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:04.720 DEBUG:boto:Port: 8001 2024-02-01T10:23:04.720 DEBUG:boto:Params: {} 2024-02-01T10:23:04.720 DEBUG:boto:Token: None 2024-02-01T10:23:04.720 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:04 GMT /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:04.720 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sioxoXm99E2YM8RkenaYe9yOlxY= 2024-02-01T10:23:04.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sioxoXm99E2YM8RkenaYe9yOlxY=', 'Content-Length': '0'} 2024-02-01T10:23:04.723 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000011cdc5ee71c4965a-0065bb7108-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:04.768 DEBUG:boto:path=/vclfcn-18/obj-test-zone1 2024-02-01T10:23:04.768 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1 2024-02-01T10:23:04.768 DEBUG:boto:path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:04.769 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:04.769 DEBUG:boto:Method: GET 2024-02-01T10:23:04.769 DEBUG:boto:Path: /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:04.769 DEBUG:boto:Data: 2024-02-01T10:23:04.769 DEBUG:boto:Headers: {} 2024-02-01T10:23:04.769 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:04.769 DEBUG:boto:Port: 8000 2024-02-01T10:23:04.769 DEBUG:boto:Params: {} 2024-02-01T10:23:04.769 DEBUG:boto:Token: None 2024-02-01T10:23:04.770 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:04 GMT /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:04.770 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sioxoXm99E2YM8RkenaYe9yOlxY= 2024-02-01T10:23:04.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sioxoXm99E2YM8RkenaYe9yOlxY=', 'Content-Length': '0'} 2024-02-01T10:23:04.772 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b5eb4cc37e4f02cd-0065bb7108-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:04.820 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:23:04.820 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:23:04.820 DEBUG:boto:path=/vclfcn-18/?prefix=obj-test-zone1 2024-02-01T10:23:04.821 DEBUG:boto:auth_path=/vclfcn-18/?prefix=obj-test-zone1 2024-02-01T10:23:04.821 DEBUG:boto:Method: GET 2024-02-01T10:23:04.821 DEBUG:boto:Path: /vclfcn-18/?prefix=obj-test-zone1 2024-02-01T10:23:04.821 DEBUG:boto:Data: 2024-02-01T10:23:04.821 DEBUG:boto:Headers: {} 2024-02-01T10:23:04.821 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:04.821 DEBUG:boto:Port: 8001 2024-02-01T10:23:04.821 DEBUG:boto:Params: {} 2024-02-01T10:23:04.821 DEBUG:boto:Token: None 2024-02-01T10:23:04.821 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:04 GMT /vclfcn-18/ 2024-02-01T10:23:04.822 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:uScQd7oraFpm6C3eTQPDd1dHcc0= 2024-02-01T10:23:04.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:uScQd7oraFpm6C3eTQPDd1dHcc0=', 'Content-Length': '0'} 2024-02-01T10:23:04.826 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084134ffa8f84a0ed-0065bb7108-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:04.868 DEBUG:boto:b'vclfcn-18obj-test-zone11000falseobj-test-zone12024-02-01T10:20:50.942Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:04.869 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:23:04.869 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:23:04.869 DEBUG:boto:path=/vclfcn-18/?prefix=obj-test-zone1 2024-02-01T10:23:04.869 DEBUG:boto:auth_path=/vclfcn-18/?prefix=obj-test-zone1 2024-02-01T10:23:04.869 DEBUG:boto:Method: GET 2024-02-01T10:23:04.869 DEBUG:boto:Path: /vclfcn-18/?prefix=obj-test-zone1 2024-02-01T10:23:04.869 DEBUG:boto:Data: 2024-02-01T10:23:04.869 DEBUG:boto:Headers: {} 2024-02-01T10:23:04.869 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:04.869 DEBUG:boto:Port: 8000 2024-02-01T10:23:04.869 DEBUG:boto:Params: {} 2024-02-01T10:23:04.870 DEBUG:boto:Token: None 2024-02-01T10:23:04.870 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:04 GMT /vclfcn-18/ 2024-02-01T10:23:04.870 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:uScQd7oraFpm6C3eTQPDd1dHcc0= 2024-02-01T10:23:04.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:uScQd7oraFpm6C3eTQPDd1dHcc0=', 'Content-Length': '0'} 2024-02-01T10:23:04.874 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1bf16f7ca00a143-0065bb7108-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:04.920 DEBUG:boto:b'vclfcn-18obj-test-zone11000falseobj-test-zone12024-02-01T10:20:50.942Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:04.921 DEBUG:boto:path=/vclfcn-18/obj-test-zone1 2024-02-01T10:23:04.921 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1 2024-02-01T10:23:04.921 DEBUG:boto:Method: HEAD 2024-02-01T10:23:04.921 DEBUG:boto:Path: /vclfcn-18/obj-test-zone1 2024-02-01T10:23:04.921 DEBUG:boto:Data: 2024-02-01T10:23:04.921 DEBUG:boto:Headers: {} 2024-02-01T10:23:04.921 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:04.921 DEBUG:boto:Port: 8001 2024-02-01T10:23:04.921 DEBUG:boto:Params: {} 2024-02-01T10:23:04.922 DEBUG:boto:Token: None 2024-02-01T10:23:04.922 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:23:04 GMT /vclfcn-18/obj-test-zone1 2024-02-01T10:23:04.922 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:OGjiJ81ynP8IeHEBXZRl5yacrDo= 2024-02-01T10:23:04.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:OGjiJ81ynP8IeHEBXZRl5yacrDo=', 'Content-Length': '0'} 2024-02-01T10:23:04.926 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b4e1df65de9e8c36-0065bb7108-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:04.927 DEBUG:boto:path=/vclfcn-18/obj-test-zone1 2024-02-01T10:23:04.927 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1 2024-02-01T10:23:04.927 DEBUG:boto:Method: HEAD 2024-02-01T10:23:04.927 DEBUG:boto:Path: /vclfcn-18/obj-test-zone1 2024-02-01T10:23:04.927 DEBUG:boto:Data: 2024-02-01T10:23:04.927 DEBUG:boto:Headers: {} 2024-02-01T10:23:04.927 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:04.927 DEBUG:boto:Port: 8000 2024-02-01T10:23:04.927 DEBUG:boto:Params: {} 2024-02-01T10:23:04.927 DEBUG:boto:Token: None 2024-02-01T10:23:04.927 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:23:04 GMT /vclfcn-18/obj-test-zone1 2024-02-01T10:23:04.928 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:OGjiJ81ynP8IeHEBXZRl5yacrDo= 2024-02-01T10:23:04.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:OGjiJ81ynP8IeHEBXZRl5yacrDo=', 'Content-Length': '0'} 2024-02-01T10:23:04.931 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000318d4fb4a2a31dcc-0065bb7108-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:04.932 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-02-01T10:23:04.932 DEBUG:boto:path=/vclfcn-18/obj-test-zone1 2024-02-01T10:23:04.932 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1 2024-02-01T10:23:04.932 DEBUG:boto:path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:04.932 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:04.932 DEBUG:boto:Method: GET 2024-02-01T10:23:04.932 DEBUG:boto:Path: /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:04.932 DEBUG:boto:Data: 2024-02-01T10:23:04.933 DEBUG:boto:Headers: {} 2024-02-01T10:23:04.933 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:04.933 DEBUG:boto:Port: 8001 2024-02-01T10:23:04.933 DEBUG:boto:Params: {} 2024-02-01T10:23:04.933 DEBUG:boto:Token: None 2024-02-01T10:23:04.933 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:04 GMT /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:04.933 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sioxoXm99E2YM8RkenaYe9yOlxY= 2024-02-01T10:23:04.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sioxoXm99E2YM8RkenaYe9yOlxY=', 'Content-Length': '0'} 2024-02-01T10:23:04.936 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fa48dbd7a192e74d-0065bb7108-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:04.980 DEBUG:boto:path=/vclfcn-18/obj-test-zone1 2024-02-01T10:23:04.980 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1 2024-02-01T10:23:04.980 DEBUG:boto:path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:04.980 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:04.981 DEBUG:boto:Method: GET 2024-02-01T10:23:04.981 DEBUG:boto:Path: /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:04.981 DEBUG:boto:Data: 2024-02-01T10:23:04.981 DEBUG:boto:Headers: {} 2024-02-01T10:23:04.981 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:04.981 DEBUG:boto:Port: 8000 2024-02-01T10:23:04.982 DEBUG:boto:Params: {} 2024-02-01T10:23:04.982 DEBUG:boto:Token: None 2024-02-01T10:23:04.982 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:04 GMT /vclfcn-18/obj-test-zone1?versionId=H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK 2024-02-01T10:23:04.982 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sioxoXm99E2YM8RkenaYe9yOlxY= 2024-02-01T10:23:04.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sioxoXm99E2YM8RkenaYe9yOlxY=', 'Content-Length': '0'} 2024-02-01T10:23:04.985 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:50 GMT'), ('x-amz-version-id', 'H.PE9rmwzEHMj5Fu1lSkwaLe73AjvcK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b7891c87373e5f1e-0065bb7108-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:05.028 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:23:05.028 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:23:05.028 DEBUG:boto:path=/vclfcn-18/?prefix=obj-test-zone1 2024-02-01T10:23:05.028 DEBUG:boto:auth_path=/vclfcn-18/?prefix=obj-test-zone1 2024-02-01T10:23:05.029 DEBUG:boto:Method: GET 2024-02-01T10:23:05.029 DEBUG:boto:Path: /vclfcn-18/?prefix=obj-test-zone1 2024-02-01T10:23:05.029 DEBUG:boto:Data: 2024-02-01T10:23:05.029 DEBUG:boto:Headers: {} 2024-02-01T10:23:05.029 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:05.029 DEBUG:boto:Port: 8001 2024-02-01T10:23:05.029 DEBUG:boto:Params: {} 2024-02-01T10:23:05.029 DEBUG:boto:Token: None 2024-02-01T10:23:05.029 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:05 GMT /vclfcn-18/ 2024-02-01T10:23:05.030 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Fu4ATxNqFxGBNvX6Sdm2xbAmts8= 2024-02-01T10:23:05.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Fu4ATxNqFxGBNvX6Sdm2xbAmts8=', 'Content-Length': '0'} 2024-02-01T10:23:05.034 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc15bc24fd921fcf-0065bb7109-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:05.034 DEBUG:boto:b'vclfcn-18obj-test-zone11000falseobj-test-zone12024-02-01T10:20:50.942Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:05.035 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:23:05.035 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:23:05.035 DEBUG:boto:path=/vclfcn-18/?prefix=obj-test-zone1 2024-02-01T10:23:05.035 DEBUG:boto:auth_path=/vclfcn-18/?prefix=obj-test-zone1 2024-02-01T10:23:05.035 DEBUG:boto:Method: GET 2024-02-01T10:23:05.035 DEBUG:boto:Path: /vclfcn-18/?prefix=obj-test-zone1 2024-02-01T10:23:05.035 DEBUG:boto:Data: 2024-02-01T10:23:05.036 DEBUG:boto:Headers: {} 2024-02-01T10:23:05.036 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:05.036 DEBUG:boto:Port: 8000 2024-02-01T10:23:05.036 DEBUG:boto:Params: {} 2024-02-01T10:23:05.036 DEBUG:boto:Token: None 2024-02-01T10:23:05.036 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:05 GMT /vclfcn-18/ 2024-02-01T10:23:05.036 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Fu4ATxNqFxGBNvX6Sdm2xbAmts8= 2024-02-01T10:23:05.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Fu4ATxNqFxGBNvX6Sdm2xbAmts8=', 'Content-Length': '0'} 2024-02-01T10:23:05.040 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a95a8f09cb61f025-0065bb7109-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:05.040 DEBUG:boto:b'vclfcn-18obj-test-zone11000falseobj-test-zone12024-02-01T10:20:50.942Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:05.041 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-02-01T10:23:05.041 DEBUG:boto:path=/vclfcn-18/obj-test-zone2 2024-02-01T10:23:05.041 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2 2024-02-01T10:23:05.041 DEBUG:boto:path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:05.041 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:05.041 DEBUG:boto:Method: GET 2024-02-01T10:23:05.041 DEBUG:boto:Path: /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:05.041 DEBUG:boto:Data: 2024-02-01T10:23:05.041 DEBUG:boto:Headers: {} 2024-02-01T10:23:05.042 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:05.042 DEBUG:boto:Port: 8001 2024-02-01T10:23:05.042 DEBUG:boto:Params: {} 2024-02-01T10:23:05.042 DEBUG:boto:Token: None 2024-02-01T10:23:05.042 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:05 GMT /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:05.042 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9C3dvKSolgY3fI+boc+PQg8AHUI= 2024-02-01T10:23:05.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9C3dvKSolgY3fI+boc+PQg8AHUI=', 'Content-Length': '0'} 2024-02-01T10:23:05.045 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', 'ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002f42c448d3becbe0-0065bb7109-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:05.092 DEBUG:boto:path=/vclfcn-18/obj-test-zone2 2024-02-01T10:23:05.092 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2 2024-02-01T10:23:05.092 DEBUG:boto:path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:05.092 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:05.093 DEBUG:boto:Method: GET 2024-02-01T10:23:05.093 DEBUG:boto:Path: /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:05.093 DEBUG:boto:Data: 2024-02-01T10:23:05.093 DEBUG:boto:Headers: {} 2024-02-01T10:23:05.093 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:05.093 DEBUG:boto:Port: 8000 2024-02-01T10:23:05.093 DEBUG:boto:Params: {} 2024-02-01T10:23:05.093 DEBUG:boto:Token: None 2024-02-01T10:23:05.093 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:05 GMT /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:05.094 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9C3dvKSolgY3fI+boc+PQg8AHUI= 2024-02-01T10:23:05.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9C3dvKSolgY3fI+boc+PQg8AHUI=', 'Content-Length': '0'} 2024-02-01T10:23:05.096 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', 'ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002ed2f146fa71c6d9-0065bb7109-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:05.144 DEBUG:boto:path=/vclfcn-18/obj-test-zone2 2024-02-01T10:23:05.144 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2 2024-02-01T10:23:05.144 DEBUG:boto:path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:05.144 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:05.145 DEBUG:boto:Method: HEAD 2024-02-01T10:23:05.145 DEBUG:boto:Path: /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:05.145 DEBUG:boto:Data: 2024-02-01T10:23:05.145 DEBUG:boto:Headers: {} 2024-02-01T10:23:05.145 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:05.145 DEBUG:boto:Port: 8001 2024-02-01T10:23:05.145 DEBUG:boto:Params: {} 2024-02-01T10:23:05.145 DEBUG:boto:Token: None 2024-02-01T10:23:05.145 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:23:05 GMT /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:05.145 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:4yWyWH7VoBJadsi5cTe2FxWzsT8= 2024-02-01T10:23:05.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:4yWyWH7VoBJadsi5cTe2FxWzsT8=', 'Content-Length': '0'} 2024-02-01T10:23:05.148 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', 'ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008cc8f1b205c64f67-0065bb7109-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:05.149 DEBUG:boto:path=/vclfcn-18/obj-test-zone2 2024-02-01T10:23:05.149 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2 2024-02-01T10:23:05.149 DEBUG:boto:path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:05.149 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:05.149 DEBUG:boto:Method: HEAD 2024-02-01T10:23:05.149 DEBUG:boto:Path: /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:05.149 DEBUG:boto:Data: 2024-02-01T10:23:05.149 DEBUG:boto:Headers: {} 2024-02-01T10:23:05.149 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:05.149 DEBUG:boto:Port: 8000 2024-02-01T10:23:05.150 DEBUG:boto:Params: {} 2024-02-01T10:23:05.150 DEBUG:boto:Token: None 2024-02-01T10:23:05.150 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:23:05 GMT /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:05.150 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:4yWyWH7VoBJadsi5cTe2FxWzsT8= 2024-02-01T10:23:05.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:4yWyWH7VoBJadsi5cTe2FxWzsT8=', 'Content-Length': '0'} 2024-02-01T10:23:05.152 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', 'ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000664d32b71349548-0065bb7109-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:05.153 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-02-01T10:23:05.153 DEBUG:boto:path=/vclfcn-18/obj-test-zone2 2024-02-01T10:23:05.153 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2 2024-02-01T10:23:05.153 DEBUG:boto:path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:05.153 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:05.153 DEBUG:boto:Method: GET 2024-02-01T10:23:05.153 DEBUG:boto:Path: /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:05.153 DEBUG:boto:Data: 2024-02-01T10:23:05.154 DEBUG:boto:Headers: {} 2024-02-01T10:23:05.154 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:05.154 DEBUG:boto:Port: 8001 2024-02-01T10:23:05.154 DEBUG:boto:Params: {} 2024-02-01T10:23:05.154 DEBUG:boto:Token: None 2024-02-01T10:23:05.154 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:05 GMT /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:05.154 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9C3dvKSolgY3fI+boc+PQg8AHUI= 2024-02-01T10:23:05.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9C3dvKSolgY3fI+boc+PQg8AHUI=', 'Content-Length': '0'} 2024-02-01T10:23:05.157 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', 'ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002b971ee86e8098b7-0065bb7109-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:05.204 DEBUG:boto:path=/vclfcn-18/obj-test-zone2 2024-02-01T10:23:05.204 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2 2024-02-01T10:23:05.204 DEBUG:boto:path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:05.204 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:05.205 DEBUG:boto:Method: GET 2024-02-01T10:23:05.205 DEBUG:boto:Path: /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:05.205 DEBUG:boto:Data: 2024-02-01T10:23:05.205 DEBUG:boto:Headers: {} 2024-02-01T10:23:05.205 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:05.205 DEBUG:boto:Port: 8000 2024-02-01T10:23:05.205 DEBUG:boto:Params: {} 2024-02-01T10:23:05.205 DEBUG:boto:Token: None 2024-02-01T10:23:05.205 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:05 GMT /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:05.206 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9C3dvKSolgY3fI+boc+PQg8AHUI= 2024-02-01T10:23:05.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9C3dvKSolgY3fI+boc+PQg8AHUI=', 'Content-Length': '0'} 2024-02-01T10:23:05.209 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', 'ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c5d643375457ee50-0065bb7109-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:05.209 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:23:05.209 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:23:05.209 DEBUG:boto:path=/vclfcn-18/?prefix=obj-test-zone2 2024-02-01T10:23:05.209 DEBUG:boto:auth_path=/vclfcn-18/?prefix=obj-test-zone2 2024-02-01T10:23:05.209 DEBUG:boto:Method: GET 2024-02-01T10:23:05.210 DEBUG:boto:Path: /vclfcn-18/?prefix=obj-test-zone2 2024-02-01T10:23:05.210 DEBUG:boto:Data: 2024-02-01T10:23:05.210 DEBUG:boto:Headers: {} 2024-02-01T10:23:05.210 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:05.210 DEBUG:boto:Port: 8001 2024-02-01T10:23:05.210 DEBUG:boto:Params: {} 2024-02-01T10:23:05.210 DEBUG:boto:Token: None 2024-02-01T10:23:05.210 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:05 GMT /vclfcn-18/ 2024-02-01T10:23:05.211 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Fu4ATxNqFxGBNvX6Sdm2xbAmts8= 2024-02-01T10:23:05.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Fu4ATxNqFxGBNvX6Sdm2xbAmts8=', 'Content-Length': '0'} 2024-02-01T10:23:05.215 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ee80362e8ce6e08-0065bb7109-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:05.215 DEBUG:boto:b'vclfcn-18obj-test-zone21000falseobj-test-zone22024-02-01T10:20:51.094Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:05.216 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:23:05.216 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:23:05.216 DEBUG:boto:path=/vclfcn-18/?prefix=obj-test-zone2 2024-02-01T10:23:05.216 DEBUG:boto:auth_path=/vclfcn-18/?prefix=obj-test-zone2 2024-02-01T10:23:05.216 DEBUG:boto:Method: GET 2024-02-01T10:23:05.216 DEBUG:boto:Path: /vclfcn-18/?prefix=obj-test-zone2 2024-02-01T10:23:05.216 DEBUG:boto:Data: 2024-02-01T10:23:05.216 DEBUG:boto:Headers: {} 2024-02-01T10:23:05.216 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:05.217 DEBUG:boto:Port: 8000 2024-02-01T10:23:05.217 DEBUG:boto:Params: {} 2024-02-01T10:23:05.217 DEBUG:boto:Token: None 2024-02-01T10:23:05.217 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:05 GMT /vclfcn-18/ 2024-02-01T10:23:05.217 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Fu4ATxNqFxGBNvX6Sdm2xbAmts8= 2024-02-01T10:23:05.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Fu4ATxNqFxGBNvX6Sdm2xbAmts8=', 'Content-Length': '0'} 2024-02-01T10:23:05.221 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060da650540d46e08-0065bb7109-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:05.221 DEBUG:boto:b'vclfcn-18obj-test-zone21000falseobj-test-zone22024-02-01T10:20:51.094Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:05.221 DEBUG:boto:path=/vclfcn-18/obj-test-zone2 2024-02-01T10:23:05.222 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2 2024-02-01T10:23:05.222 DEBUG:boto:Method: HEAD 2024-02-01T10:23:05.222 DEBUG:boto:Path: /vclfcn-18/obj-test-zone2 2024-02-01T10:23:05.222 DEBUG:boto:Data: 2024-02-01T10:23:05.222 DEBUG:boto:Headers: {} 2024-02-01T10:23:05.222 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:05.222 DEBUG:boto:Port: 8001 2024-02-01T10:23:05.222 DEBUG:boto:Params: {} 2024-02-01T10:23:05.222 DEBUG:boto:Token: None 2024-02-01T10:23:05.223 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:23:05 GMT /vclfcn-18/obj-test-zone2 2024-02-01T10:23:05.223 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:LUM1ks4Ad3lEVq6xxuEM7JPkwuk= 2024-02-01T10:23:05.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:LUM1ks4Ad3lEVq6xxuEM7JPkwuk=', 'Content-Length': '0'} 2024-02-01T10:23:05.226 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', 'ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000034322ba842d8910b-0065bb7109-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:05.226 DEBUG:boto:path=/vclfcn-18/obj-test-zone2 2024-02-01T10:23:05.226 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2 2024-02-01T10:23:05.227 DEBUG:boto:Method: HEAD 2024-02-01T10:23:05.227 DEBUG:boto:Path: /vclfcn-18/obj-test-zone2 2024-02-01T10:23:05.227 DEBUG:boto:Data: 2024-02-01T10:23:05.227 DEBUG:boto:Headers: {} 2024-02-01T10:23:05.227 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:05.227 DEBUG:boto:Port: 8000 2024-02-01T10:23:05.227 DEBUG:boto:Params: {} 2024-02-01T10:23:05.227 DEBUG:boto:Token: None 2024-02-01T10:23:05.227 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:23:05 GMT /vclfcn-18/obj-test-zone2 2024-02-01T10:23:05.227 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:LUM1ks4Ad3lEVq6xxuEM7JPkwuk= 2024-02-01T10:23:05.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:LUM1ks4Ad3lEVq6xxuEM7JPkwuk=', 'Content-Length': '0'} 2024-02-01T10:23:05.231 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', 'ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000159832d3a001436f-0065bb7109-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:05.231 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-02-01T10:23:05.231 DEBUG:boto:path=/vclfcn-18/obj-test-zone2 2024-02-01T10:23:05.232 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2 2024-02-01T10:23:05.232 DEBUG:boto:path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:05.232 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:05.232 DEBUG:boto:Method: GET 2024-02-01T10:23:05.232 DEBUG:boto:Path: /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:05.232 DEBUG:boto:Data: 2024-02-01T10:23:05.232 DEBUG:boto:Headers: {} 2024-02-01T10:23:05.232 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:05.232 DEBUG:boto:Port: 8001 2024-02-01T10:23:05.232 DEBUG:boto:Params: {} 2024-02-01T10:23:05.233 DEBUG:boto:Token: None 2024-02-01T10:23:05.233 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:05 GMT /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:05.233 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9C3dvKSolgY3fI+boc+PQg8AHUI= 2024-02-01T10:23:05.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9C3dvKSolgY3fI+boc+PQg8AHUI=', 'Content-Length': '0'} 2024-02-01T10:23:05.236 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', 'ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007ec538f9912c308b-0065bb7109-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:05.276 DEBUG:boto:path=/vclfcn-18/obj-test-zone2 2024-02-01T10:23:05.276 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2 2024-02-01T10:23:05.276 DEBUG:boto:path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:05.276 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:05.277 DEBUG:boto:Method: GET 2024-02-01T10:23:05.277 DEBUG:boto:Path: /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:05.277 DEBUG:boto:Data: 2024-02-01T10:23:05.277 DEBUG:boto:Headers: {} 2024-02-01T10:23:05.277 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:05.277 DEBUG:boto:Port: 8000 2024-02-01T10:23:05.277 DEBUG:boto:Params: {} 2024-02-01T10:23:05.277 DEBUG:boto:Token: None 2024-02-01T10:23:05.277 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:05 GMT /vclfcn-18/obj-test-zone2?versionId=ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD 2024-02-01T10:23:05.278 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9C3dvKSolgY3fI+boc+PQg8AHUI= 2024-02-01T10:23:05.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9C3dvKSolgY3fI+boc+PQg8AHUI=', 'Content-Length': '0'} 2024-02-01T10:23:05.281 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', 'ID0fLtt0vI-7pPjmoJi1jQ6ydDXQutD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a11e331b06e44942-0065bb7109-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:05.281 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:23:05.281 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:23:05.281 DEBUG:boto:path=/vclfcn-18/?prefix=obj-test-zone2 2024-02-01T10:23:05.281 DEBUG:boto:auth_path=/vclfcn-18/?prefix=obj-test-zone2 2024-02-01T10:23:05.281 DEBUG:boto:Method: GET 2024-02-01T10:23:05.282 DEBUG:boto:Path: /vclfcn-18/?prefix=obj-test-zone2 2024-02-01T10:23:05.282 DEBUG:boto:Data: 2024-02-01T10:23:05.282 DEBUG:boto:Headers: {} 2024-02-01T10:23:05.282 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:05.282 DEBUG:boto:Port: 8001 2024-02-01T10:23:05.282 DEBUG:boto:Params: {} 2024-02-01T10:23:05.282 DEBUG:boto:Token: None 2024-02-01T10:23:05.282 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:05 GMT /vclfcn-18/ 2024-02-01T10:23:05.282 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Fu4ATxNqFxGBNvX6Sdm2xbAmts8= 2024-02-01T10:23:05.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Fu4ATxNqFxGBNvX6Sdm2xbAmts8=', 'Content-Length': '0'} 2024-02-01T10:23:05.287 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067b148deb18a9e6b-0065bb7109-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:05.287 DEBUG:boto:b'vclfcn-18obj-test-zone21000falseobj-test-zone22024-02-01T10:20:51.094Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:05.287 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:23:05.287 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:23:05.288 DEBUG:boto:path=/vclfcn-18/?prefix=obj-test-zone2 2024-02-01T10:23:05.288 DEBUG:boto:auth_path=/vclfcn-18/?prefix=obj-test-zone2 2024-02-01T10:23:05.288 DEBUG:boto:Method: GET 2024-02-01T10:23:05.288 DEBUG:boto:Path: /vclfcn-18/?prefix=obj-test-zone2 2024-02-01T10:23:05.288 DEBUG:boto:Data: 2024-02-01T10:23:05.288 DEBUG:boto:Headers: {} 2024-02-01T10:23:05.288 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:05.288 DEBUG:boto:Port: 8000 2024-02-01T10:23:05.288 DEBUG:boto:Params: {} 2024-02-01T10:23:05.288 DEBUG:boto:Token: None 2024-02-01T10:23:05.288 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:05 GMT /vclfcn-18/ 2024-02-01T10:23:05.289 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Fu4ATxNqFxGBNvX6Sdm2xbAmts8= 2024-02-01T10:23:05.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Fu4ATxNqFxGBNvX6Sdm2xbAmts8=', 'Content-Length': '0'} 2024-02-01T10:23:05.293 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000048897461ce4ad9ae-0065bb7109-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:05.293 DEBUG:boto:b'vclfcn-18obj-test-zone21000falseobj-test-zone22024-02-01T10:20:51.094Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:05.294 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-02-01T10:23:05.294 DEBUG:boto:path=/vclfcn-18/obj-test-zone3 2024-02-01T10:23:05.294 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3 2024-02-01T10:23:05.294 DEBUG:boto:path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:05.294 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:05.294 DEBUG:boto:Method: GET 2024-02-01T10:23:05.294 DEBUG:boto:Path: /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:05.295 DEBUG:boto:Data: 2024-02-01T10:23:05.295 DEBUG:boto:Headers: {} 2024-02-01T10:23:05.295 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:05.295 DEBUG:boto:Port: 8001 2024-02-01T10:23:05.295 DEBUG:boto:Params: {} 2024-02-01T10:23:05.295 DEBUG:boto:Token: None 2024-02-01T10:23:05.295 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:05 GMT /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:05.295 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:OiGCnNCmqRWD1j2HL2FDy9YUgQQ= 2024-02-01T10:23:05.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:OiGCnNCmqRWD1j2HL2FDy9YUgQQ=', 'Content-Length': '0'} 2024-02-01T10:23:05.298 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', '6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000f56f8f0f0879a0c-0065bb7109-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:05.299 DEBUG:boto:path=/vclfcn-18/obj-test-zone3 2024-02-01T10:23:05.299 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3 2024-02-01T10:23:05.299 DEBUG:boto:path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:05.299 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:05.299 DEBUG:boto:Method: GET 2024-02-01T10:23:05.299 DEBUG:boto:Path: /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:05.299 DEBUG:boto:Data: 2024-02-01T10:23:05.299 DEBUG:boto:Headers: {} 2024-02-01T10:23:05.299 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:05.300 DEBUG:boto:Port: 8000 2024-02-01T10:23:05.300 DEBUG:boto:Params: {} 2024-02-01T10:23:05.300 DEBUG:boto:Token: None 2024-02-01T10:23:05.300 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:05 GMT /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:05.300 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:OiGCnNCmqRWD1j2HL2FDy9YUgQQ= 2024-02-01T10:23:05.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:OiGCnNCmqRWD1j2HL2FDy9YUgQQ=', 'Content-Length': '0'} 2024-02-01T10:23:05.303 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', '6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000747d42264510925c-0065bb7109-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:05.348 DEBUG:boto:path=/vclfcn-18/obj-test-zone3 2024-02-01T10:23:05.348 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3 2024-02-01T10:23:05.348 DEBUG:boto:path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:05.349 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:05.349 DEBUG:boto:Method: HEAD 2024-02-01T10:23:05.349 DEBUG:boto:Path: /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:05.349 DEBUG:boto:Data: 2024-02-01T10:23:05.349 DEBUG:boto:Headers: {} 2024-02-01T10:23:05.349 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:05.349 DEBUG:boto:Port: 8001 2024-02-01T10:23:05.349 DEBUG:boto:Params: {} 2024-02-01T10:23:05.349 DEBUG:boto:Token: None 2024-02-01T10:23:05.350 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:23:05 GMT /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:05.350 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Dwfx3jxz8Imc4YSmidVsM6PewMs= 2024-02-01T10:23:05.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Dwfx3jxz8Imc4YSmidVsM6PewMs=', 'Content-Length': '0'} 2024-02-01T10:23:05.352 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', '6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000098828249de67ef5e-0065bb7109-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:05.353 DEBUG:boto:path=/vclfcn-18/obj-test-zone3 2024-02-01T10:23:05.353 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3 2024-02-01T10:23:05.353 DEBUG:boto:path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:05.353 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:05.353 DEBUG:boto:Method: HEAD 2024-02-01T10:23:05.353 DEBUG:boto:Path: /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:05.353 DEBUG:boto:Data: 2024-02-01T10:23:05.353 DEBUG:boto:Headers: {} 2024-02-01T10:23:05.353 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:05.354 DEBUG:boto:Port: 8000 2024-02-01T10:23:05.354 DEBUG:boto:Params: {} 2024-02-01T10:23:05.354 DEBUG:boto:Token: None 2024-02-01T10:23:05.354 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:23:05 GMT /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:05.354 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Dwfx3jxz8Imc4YSmidVsM6PewMs= 2024-02-01T10:23:05.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Dwfx3jxz8Imc4YSmidVsM6PewMs=', 'Content-Length': '0'} 2024-02-01T10:23:05.357 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', '6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000080315a494f79c322-0065bb7109-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:05.357 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-02-01T10:23:05.357 DEBUG:boto:path=/vclfcn-18/obj-test-zone3 2024-02-01T10:23:05.358 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3 2024-02-01T10:23:05.358 DEBUG:boto:path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:05.358 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:05.358 DEBUG:boto:Method: GET 2024-02-01T10:23:05.358 DEBUG:boto:Path: /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:05.358 DEBUG:boto:Data: 2024-02-01T10:23:05.358 DEBUG:boto:Headers: {} 2024-02-01T10:23:05.358 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:05.358 DEBUG:boto:Port: 8001 2024-02-01T10:23:05.358 DEBUG:boto:Params: {} 2024-02-01T10:23:05.359 DEBUG:boto:Token: None 2024-02-01T10:23:05.359 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:05 GMT /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:05.359 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:OiGCnNCmqRWD1j2HL2FDy9YUgQQ= 2024-02-01T10:23:05.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:OiGCnNCmqRWD1j2HL2FDy9YUgQQ=', 'Content-Length': '0'} 2024-02-01T10:23:05.362 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', '6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ff2e65c781825a0d-0065bb7109-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:05.408 DEBUG:boto:path=/vclfcn-18/obj-test-zone3 2024-02-01T10:23:05.408 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3 2024-02-01T10:23:05.408 DEBUG:boto:path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:05.408 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:05.409 DEBUG:boto:Method: GET 2024-02-01T10:23:05.409 DEBUG:boto:Path: /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:05.409 DEBUG:boto:Data: 2024-02-01T10:23:05.409 DEBUG:boto:Headers: {} 2024-02-01T10:23:05.409 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:05.409 DEBUG:boto:Port: 8000 2024-02-01T10:23:05.409 DEBUG:boto:Params: {} 2024-02-01T10:23:05.409 DEBUG:boto:Token: None 2024-02-01T10:23:05.409 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:05 GMT /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:05.410 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:OiGCnNCmqRWD1j2HL2FDy9YUgQQ= 2024-02-01T10:23:05.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:OiGCnNCmqRWD1j2HL2FDy9YUgQQ=', 'Content-Length': '0'} 2024-02-01T10:23:05.412 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', '6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006dbac663cb843db4-0065bb7109-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:05.460 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:23:05.460 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:23:05.460 DEBUG:boto:path=/vclfcn-18/?prefix=obj-test-zone3 2024-02-01T10:23:05.461 DEBUG:boto:auth_path=/vclfcn-18/?prefix=obj-test-zone3 2024-02-01T10:23:05.461 DEBUG:boto:Method: GET 2024-02-01T10:23:05.461 DEBUG:boto:Path: /vclfcn-18/?prefix=obj-test-zone3 2024-02-01T10:23:05.461 DEBUG:boto:Data: 2024-02-01T10:23:05.461 DEBUG:boto:Headers: {} 2024-02-01T10:23:05.461 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:05.461 DEBUG:boto:Port: 8001 2024-02-01T10:23:05.461 DEBUG:boto:Params: {} 2024-02-01T10:23:05.461 DEBUG:boto:Token: None 2024-02-01T10:23:05.461 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:05 GMT /vclfcn-18/ 2024-02-01T10:23:05.462 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Fu4ATxNqFxGBNvX6Sdm2xbAmts8= 2024-02-01T10:23:05.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Fu4ATxNqFxGBNvX6Sdm2xbAmts8=', 'Content-Length': '0'} 2024-02-01T10:23:05.466 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa3f7593cf868b9b-0065bb7109-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:05.467 DEBUG:boto:b'vclfcn-18obj-test-zone31000falseobj-test-zone32024-02-01T10:20:51.248Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:05.467 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:23:05.467 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:23:05.468 DEBUG:boto:path=/vclfcn-18/?prefix=obj-test-zone3 2024-02-01T10:23:05.468 DEBUG:boto:auth_path=/vclfcn-18/?prefix=obj-test-zone3 2024-02-01T10:23:05.468 DEBUG:boto:Method: GET 2024-02-01T10:23:05.468 DEBUG:boto:Path: /vclfcn-18/?prefix=obj-test-zone3 2024-02-01T10:23:05.468 DEBUG:boto:Data: 2024-02-01T10:23:05.468 DEBUG:boto:Headers: {} 2024-02-01T10:23:05.468 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:05.468 DEBUG:boto:Port: 8000 2024-02-01T10:23:05.468 DEBUG:boto:Params: {} 2024-02-01T10:23:05.468 DEBUG:boto:Token: None 2024-02-01T10:23:05.469 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:05 GMT /vclfcn-18/ 2024-02-01T10:23:05.469 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Fu4ATxNqFxGBNvX6Sdm2xbAmts8= 2024-02-01T10:23:05.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Fu4ATxNqFxGBNvX6Sdm2xbAmts8=', 'Content-Length': '0'} 2024-02-01T10:23:05.473 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b346a33077efc299-0065bb7109-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:05.474 DEBUG:boto:b'vclfcn-18obj-test-zone31000falseobj-test-zone32024-02-01T10:20:51.248Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:05.474 DEBUG:boto:path=/vclfcn-18/obj-test-zone3 2024-02-01T10:23:05.474 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3 2024-02-01T10:23:05.474 DEBUG:boto:Method: HEAD 2024-02-01T10:23:05.474 DEBUG:boto:Path: /vclfcn-18/obj-test-zone3 2024-02-01T10:23:05.474 DEBUG:boto:Data: 2024-02-01T10:23:05.474 DEBUG:boto:Headers: {} 2024-02-01T10:23:05.474 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:05.475 DEBUG:boto:Port: 8001 2024-02-01T10:23:05.475 DEBUG:boto:Params: {} 2024-02-01T10:23:05.475 DEBUG:boto:Token: None 2024-02-01T10:23:05.475 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:23:05 GMT /vclfcn-18/obj-test-zone3 2024-02-01T10:23:05.475 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:breF4ZaY7gYcCR9g5xu3Ebe+Eq0= 2024-02-01T10:23:05.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:breF4ZaY7gYcCR9g5xu3Ebe+Eq0=', 'Content-Length': '0'} 2024-02-01T10:23:05.481 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', '6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004a19acc28ed4ca53-0065bb7109-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:05.482 DEBUG:boto:path=/vclfcn-18/obj-test-zone3 2024-02-01T10:23:05.482 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3 2024-02-01T10:23:05.482 DEBUG:boto:Method: HEAD 2024-02-01T10:23:05.482 DEBUG:boto:Path: /vclfcn-18/obj-test-zone3 2024-02-01T10:23:05.482 DEBUG:boto:Data: 2024-02-01T10:23:05.482 DEBUG:boto:Headers: {} 2024-02-01T10:23:05.482 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:05.482 DEBUG:boto:Port: 8000 2024-02-01T10:23:05.482 DEBUG:boto:Params: {} 2024-02-01T10:23:05.482 DEBUG:boto:Token: None 2024-02-01T10:23:05.483 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:23:05 GMT /vclfcn-18/obj-test-zone3 2024-02-01T10:23:05.483 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:breF4ZaY7gYcCR9g5xu3Ebe+Eq0= 2024-02-01T10:23:05.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:breF4ZaY7gYcCR9g5xu3Ebe+Eq0=', 'Content-Length': '0'} 2024-02-01T10:23:05.486 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', '6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007eb7003dc6079624-0065bb7109-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:05.487 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-02-01T10:23:05.487 DEBUG:boto:path=/vclfcn-18/obj-test-zone3 2024-02-01T10:23:05.487 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3 2024-02-01T10:23:05.487 DEBUG:boto:path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:05.487 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:05.487 DEBUG:boto:Method: GET 2024-02-01T10:23:05.487 DEBUG:boto:Path: /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:05.487 DEBUG:boto:Data: 2024-02-01T10:23:05.487 DEBUG:boto:Headers: {} 2024-02-01T10:23:05.487 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:05.488 DEBUG:boto:Port: 8001 2024-02-01T10:23:05.488 DEBUG:boto:Params: {} 2024-02-01T10:23:05.488 DEBUG:boto:Token: None 2024-02-01T10:23:05.488 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:05 GMT /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:05.488 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:OiGCnNCmqRWD1j2HL2FDy9YUgQQ= 2024-02-01T10:23:05.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:OiGCnNCmqRWD1j2HL2FDy9YUgQQ=', 'Content-Length': '0'} 2024-02-01T10:23:05.491 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', '6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bb87ddd2bbcbe58a-0065bb7109-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:05.492 DEBUG:boto:path=/vclfcn-18/obj-test-zone3 2024-02-01T10:23:05.492 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3 2024-02-01T10:23:05.492 DEBUG:boto:path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:05.492 DEBUG:boto:auth_path=/vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:05.492 DEBUG:boto:Method: GET 2024-02-01T10:23:05.492 DEBUG:boto:Path: /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:05.492 DEBUG:boto:Data: 2024-02-01T10:23:05.492 DEBUG:boto:Headers: {} 2024-02-01T10:23:05.492 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:05.493 DEBUG:boto:Port: 8000 2024-02-01T10:23:05.493 DEBUG:boto:Params: {} 2024-02-01T10:23:05.493 DEBUG:boto:Token: None 2024-02-01T10:23:05.493 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:05 GMT /vclfcn-18/obj-test-zone3?versionId=6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N 2024-02-01T10:23:05.493 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:OiGCnNCmqRWD1j2HL2FDy9YUgQQ= 2024-02-01T10:23:05.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:OiGCnNCmqRWD1j2HL2FDy9YUgQQ=', 'Content-Length': '0'} 2024-02-01T10:23:05.496 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:20:51 GMT'), ('x-amz-version-id', '6pAVLbw3HAwE0jxxbuxTIxwcA2p4q8N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d5f7d34daac76a83-0065bb7109-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:05.540 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:23:05.540 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:23:05.540 DEBUG:boto:path=/vclfcn-18/?prefix=obj-test-zone3 2024-02-01T10:23:05.540 DEBUG:boto:auth_path=/vclfcn-18/?prefix=obj-test-zone3 2024-02-01T10:23:05.541 DEBUG:boto:Method: GET 2024-02-01T10:23:05.541 DEBUG:boto:Path: /vclfcn-18/?prefix=obj-test-zone3 2024-02-01T10:23:05.541 DEBUG:boto:Data: 2024-02-01T10:23:05.541 DEBUG:boto:Headers: {} 2024-02-01T10:23:05.541 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:23:05.541 DEBUG:boto:Port: 8001 2024-02-01T10:23:05.541 DEBUG:boto:Params: {} 2024-02-01T10:23:05.541 DEBUG:boto:Token: None 2024-02-01T10:23:05.541 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:05 GMT /vclfcn-18/ 2024-02-01T10:23:05.541 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Fu4ATxNqFxGBNvX6Sdm2xbAmts8= 2024-02-01T10:23:05.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Fu4ATxNqFxGBNvX6Sdm2xbAmts8=', 'Content-Length': '0'} 2024-02-01T10:23:05.546 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000465d532e988c25b6-0065bb7109-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:05.546 DEBUG:boto:b'vclfcn-18obj-test-zone31000falseobj-test-zone32024-02-01T10:20:51.248Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:05.546 DEBUG:boto:path=/vclfcn-18/ 2024-02-01T10:23:05.547 DEBUG:boto:auth_path=/vclfcn-18/ 2024-02-01T10:23:05.547 DEBUG:boto:path=/vclfcn-18/?prefix=obj-test-zone3 2024-02-01T10:23:05.547 DEBUG:boto:auth_path=/vclfcn-18/?prefix=obj-test-zone3 2024-02-01T10:23:05.547 DEBUG:boto:Method: GET 2024-02-01T10:23:05.547 DEBUG:boto:Path: /vclfcn-18/?prefix=obj-test-zone3 2024-02-01T10:23:05.547 DEBUG:boto:Data: 2024-02-01T10:23:05.547 DEBUG:boto:Headers: {} 2024-02-01T10:23:05.547 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:23:05.547 DEBUG:boto:Port: 8000 2024-02-01T10:23:05.547 DEBUG:boto:Params: {} 2024-02-01T10:23:05.548 DEBUG:boto:Token: None 2024-02-01T10:23:05.548 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:23:05 GMT /vclfcn-18/ 2024-02-01T10:23:05.548 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Fu4ATxNqFxGBNvX6Sdm2xbAmts8= 2024-02-01T10:23:05.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Fu4ATxNqFxGBNvX6Sdm2xbAmts8=', 'Content-Length': '0'} 2024-02-01T10:23:05.552 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f2a94820c4e7950-0065bb7109-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:05.552 DEBUG:boto:b'vclfcn-18obj-test-zone31000falseobj-test-zone32024-02-01T10:20:51.248Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:23:05.552 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-18 zones={test-zone3, test-zone2} 2024-02-01T10:23:05.553 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_versioned_object_incremental_sync ... ok 2024-02-01T10:23:05.556 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:23:05.556 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:23:05.556 DEBUG:boto:Method: PUT 2024-02-01T10:23:05.556 DEBUG:boto:Path: /vclfcn-19/ 2024-02-01T10:23:05.556 DEBUG:boto:Data: 2024-02-01T10:23:05.556 DEBUG:boto:Headers: {} 2024-02-01T10:23:05.556 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:23:05.556 DEBUG:boto:Port: 8000 2024-02-01T10:23:05.556 DEBUG:boto:Params: {} 2024-02-01T10:23:05.557 DEBUG:boto:Token: None 2024-02-01T10:23:05.557 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:23:05 GMT /vclfcn-19/ 2024-02-01T10:23:05.557 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WCnbCVT4fz0yZV5BGORYzyAEwY0= 2024-02-01T10:23:05.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WCnbCVT4fz0yZV5BGORYzyAEwY0=', 'Content-Length': '0'} 2024-02-01T10:23:05.591 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000026162bf2dae675f5-0065bb7109-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:23:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:05.591 DEBUG:rgw_multi.tests:created bucket=vclfcn-19 2024-02-01T10:23:05.591 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:23:05.591 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:23:05.591 DEBUG:boto:path=/vclfcn-19/?versioning 2024-02-01T10:23:05.591 DEBUG:boto:auth_path=/vclfcn-19/?versioning 2024-02-01T10:23:05.592 DEBUG:boto:Method: PUT 2024-02-01T10:23:05.592 DEBUG:boto:Path: /vclfcn-19/?versioning 2024-02-01T10:23:05.592 DEBUG:boto:Data: Enabled Disabled 2024-02-01T10:23:05.592 DEBUG:boto:Headers: {} 2024-02-01T10:23:05.592 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:23:05.592 DEBUG:boto:Port: 8000 2024-02-01T10:23:05.592 DEBUG:boto:Params: {} 2024-02-01T10:23:05.592 DEBUG:boto:Token: None 2024-02-01T10:23:05.592 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:23:05 GMT /vclfcn-19/?versioning 2024-02-01T10:23:05.593 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:IWIRiaTCrpCwJZO1xaUtwfGU5wM= 2024-02-01T10:23:05.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:IWIRiaTCrpCwJZO1xaUtwfGU5wM=', 'Content-Length': '228'} 2024-02-01T10:23:05.604 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000bf218280657a4be2-0065bb7109-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:23:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:05.604 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:23:05.604 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:23:05.605 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:23:05.645 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:23:05.646 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:23:05.703 INFO:teuthology.orchestra.run.smithi148.stderr:No --period given, using current period=5dc4efbd-eccd-47a8-9471-ae4a209ec978 2024-02-01T10:23:05.734 INFO:teuthology.orchestra.run.smithi148.stdout:[{"marker":"1_1706782453.132619_30.1","last_update":"2024-02-01T10:14:13.132619Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1706782985.609410_98.1","last_update":"2024-02-01T10:23:05.609410Z"},{"marker":"1_1706782595.670952_121.1","last_update":"2024-02-01T10:16:35.670952Z"}] 2024-02-01T10:23:05.735 DEBUG:rgw_multi.tests:master meta markers={0: '1_1706782453.132619_30.1', 1: '', 2: '1_1706782985.609410_98.1', 3: '1_1706782595.670952_121.1'} 2024-02-01T10:23:05.735 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-02-01T10:23:05.735 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:23:05.735 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:23:05.735 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:23:05.781 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:23:05.781 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:23:05.834 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:23:05.836+0000 7f942f9ba980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:23:05.835 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:23:05.836+0000 7f942f9ba980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:23:05.858 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706782985.609410_98.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:23:05.609410Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:23:05Z"} 2024-02-01T10:23:05.858 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706782985.609410_98.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:23:05.609410Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:23:05Z"} 2024-02-01T10:23:05.858 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706782985.609410_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:23:05.609410Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:23:05.859 DEBUG:rgw_multi.tests:log_status={0: '1_1706782453.132619_30.1', 1: '', 2: '1_1706782985.609410_98.1', 3: '1_1706782595.670952_121.1'} 2024-02-01T10:23:05.859 DEBUG:rgw_multi.tests:sync_status={0: '1_1706782453.132619_30.1', 1: '', 2: '1_1706782985.609410_98.1', 3: '1_1706782595.670952_121.1'} 2024-02-01T10:23:05.859 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-02-01T10:23:05.859 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-02-01T10:23:05.859 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:23:05.859 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:23:05.859 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:23:05.900 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:23:05.901 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:23:05.953 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:23:05.956+0000 7f0a9cf4a980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:23:05.954 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:23:05.956+0000 7f0a9cf4a980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:23:05.976 INFO:teuthology.orchestra.run.smithi148.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706782985.577592_93.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:23:05.577592Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:23:05Z"} 2024-02-01T10:23:05.977 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706782985.577592_93.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:23:05.577592Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:23:05Z"} 2024-02-01T10:23:05.977 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706782985.577592_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:23:05.577592Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:23:05.977 DEBUG:rgw_multi.tests:log_status={0: '1_1706782453.132619_30.1', 1: '', 2: '1_1706782985.609410_98.1', 3: '1_1706782595.670952_121.1'} 2024-02-01T10:23:05.977 DEBUG:rgw_multi.tests:sync_status={0: '1_1706782453.132619_30.1', 1: '', 2: '1_1706782985.577592_93.1', 3: '1_1706782595.670952_121.1'} 2024-02-01T10:23:05.977 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1706782985.609410_98.1 target=1_1706782985.577592_93.1 2024-02-01T10:23:10.978 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:23:10.978 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:23:10.978 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:23:11.022 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:23:11.022 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:23:11.076 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:23:11.080+0000 7f2b6e12f980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:23:11.077 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:23:11.080+0000 7f2b6e12f980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:23:11.099 INFO:teuthology.orchestra.run.smithi148.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706782985.609410_98.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:23:05.609410Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:23:11Z"} 2024-02-01T10:23:11.100 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706782985.609410_98.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:23:05.609410Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:23:11Z"} 2024-02-01T10:23:11.100 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706782985.609410_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:23:05.609410Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:23:11.100 DEBUG:rgw_multi.tests:log_status={0: '1_1706782453.132619_30.1', 1: '', 2: '1_1706782985.609410_98.1', 3: '1_1706782595.670952_121.1'} 2024-02-01T10:23:11.100 DEBUG:rgw_multi.tests:sync_status={0: '1_1706782453.132619_30.1', 1: '', 2: '1_1706782985.609410_98.1', 3: '1_1706782595.670952_121.1'} 2024-02-01T10:23:11.100 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-02-01T10:23:11.100 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:23:11.101 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:23:11.101 DEBUG:boto:Method: HEAD 2024-02-01T10:23:11.101 DEBUG:boto:Path: /vclfcn-19/ 2024-02-01T10:23:11.101 DEBUG:boto:Data: 2024-02-01T10:23:11.101 DEBUG:boto:Headers: {} 2024-02-01T10:23:11.101 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:23:11.101 DEBUG:boto:Port: 8000 2024-02-01T10:23:11.101 DEBUG:boto:Params: {} 2024-02-01T10:23:11.101 DEBUG:boto:Token: None 2024-02-01T10:23:11.101 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:23:11 GMT /vclfcn-19/ 2024-02-01T10:23:11.102 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:qWPvXnR/KrCg1ZtX2cLryNp8W1Q= 2024-02-01T10:23:11.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:23:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:qWPvXnR/KrCg1ZtX2cLryNp8W1Q=', 'Content-Length': '0'} 2024-02-01T10:23:11.104 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009669aa880cd14923-0065bb710f-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:23:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:11.105 DEBUG:boto:path=/vclfcn-19/dummy 2024-02-01T10:23:11.105 DEBUG:boto:auth_path=/vclfcn-19/dummy 2024-02-01T10:23:11.105 DEBUG:boto:Method: PUT 2024-02-01T10:23:11.105 DEBUG:boto:Path: /vclfcn-19/dummy 2024-02-01T10:23:11.105 DEBUG:boto:Data: 2024-02-01T10:23:11.105 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-02-01T10:23:11.105 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:23:11.105 DEBUG:boto:Port: 8000 2024-02-01T10:23:11.105 DEBUG:boto:Params: {} 2024-02-01T10:23:11.105 DEBUG:boto:Token: None 2024-02-01T10:23:11.106 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Thu, 01 Feb 2024 10:23:11 GMT /vclfcn-19/dummy 2024-02-01T10:23:11.106 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mqapi6qa+oN8H4v8mJ/6GpO8PA0= 2024-02-01T10:23:11.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:23:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mqapi6qa+oN8H4v8mJ/6GpO8PA0='} 2024-02-01T10:23:11.117 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu'), ('x-amz-request-id', 'tx000001bad578633e57e6b-0065bb710f-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:23:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:23:11.117 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=vclfcn-19 2024-02-01T10:23:11.118 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:23:11.118 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:23:11.118 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket vclfcn-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-02-01T10:23:11.169 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:23:11.169 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:23:11.235 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:23:11.236+0000 7f7181cbc980 1 waiting to reach incremental sync.. 2024-02-01T10:23:16.235 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:23:16.236+0000 7f7181cbc980 1 waiting to reach incremental sync.. 2024-02-01T10:23:21.236 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:23:21.240+0000 7f7181cbc980 1 waiting to reach incremental sync.. 2024-02-01T10:23:26.237 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:23:26.240+0000 7f7181cbc980 1 waiting to reach incremental sync.. 2024-02-01T10:23:31.241 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:23:31.244+0000 7f7181cbc980 1 bucket sync caught up with source: 2024-02-01T10:23:31.241 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, , , , , 00000000002.13.11, , , , , ] 2024-02-01T10:23:31.241 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, , , , , 00000000002.13.11, , , , , ] 2024-02-01T10:23:31.241 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:23:31.244+0000 7f7181cbc980 0 bucket checkpoint complete 2024-02-01T10:23:31.280 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=vclfcn-19 2024-02-01T10:23:31.281 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:23:31.281 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:23:31.281 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:23:31.332 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:23:31.332 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:23:31.393 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:23:31.396+0000 7f35c6060980 1 bucket sync caught up with source: 2024-02-01T10:23:31.393 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, , , , , 00000000002.13.11, , , , , ] 2024-02-01T10:23:31.394 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, , , , , 00000000002.13.11, , , , , ] 2024-02-01T10:23:31.394 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:23:31.396+0000 7f35c6060980 0 bucket checkpoint complete 2024-02-01T10:23:31.416 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=vclfcn-19 2024-02-01T10:23:31.416 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:23:31.416 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:23:31.416 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:23:31.456 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:23:31.457 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:23:31.515 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:23:31.520+0000 7f712e81e980 1 waiting to reach incremental sync.. 2024-02-01T10:23:36.516 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:23:36.519+0000 7f712e81e980 1 waiting to reach incremental sync.. 2024-02-01T10:23:41.517 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:23:41.519+0000 7f712e81e980 1 waiting to reach incremental sync.. 2024-02-01T10:23:46.518 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:23:46.519+0000 7f712e81e980 1 waiting to reach incremental sync.. 2024-02-01T10:23:51.519 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:23:51.523+0000 7f712e81e980 1 waiting to reach incremental sync.. 2024-02-01T10:23:56.521 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:23:56.523+0000 7f712e81e980 1 waiting to reach incremental sync.. 2024-02-01T10:24:01.521 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:24:01.523+0000 7f712e81e980 1 waiting to reach incremental sync.. 2024-02-01T10:24:06.522 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:24:06.523+0000 7f712e81e980 1 waiting to reach incremental sync.. 2024-02-01T10:24:11.524 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:24:11.527+0000 7f712e81e980 1 waiting to reach incremental sync.. 2024-02-01T10:24:16.525 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:24:16.527+0000 7f712e81e980 1 waiting to reach incremental sync.. 2024-02-01T10:24:21.526 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:24:21.527+0000 7f712e81e980 1 waiting to reach incremental sync.. 2024-02-01T10:24:26.527 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:24:26.531+0000 7f712e81e980 1 waiting to reach incremental sync.. 2024-02-01T10:24:31.528 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:24:31.531+0000 7f712e81e980 1 waiting to reach incremental sync.. 2024-02-01T10:24:36.529 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:24:36.530+0000 7f712e81e980 1 waiting to reach incremental sync.. 2024-02-01T10:24:41.530 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:24:41.530+0000 7f712e81e980 1 waiting to reach incremental sync.. 2024-02-01T10:24:46.531 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:24:46.534+0000 7f712e81e980 1 waiting to reach incremental sync.. 2024-02-01T10:24:51.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:24:51.538+0000 7f712e81e980 1 waiting to reach incremental sync.. 2024-02-01T10:24:56.533 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:24:56.538+0000 7f712e81e980 1 waiting to reach incremental sync.. 2024-02-01T10:25:01.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:25:01.538+0000 7f712e81e980 1 waiting to reach incremental sync.. 2024-02-01T10:25:06.536 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:25:06.538+0000 7f712e81e980 1 waiting to reach incremental sync.. 2024-02-01T10:25:11.537 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:25:11.542+0000 7f712e81e980 1 waiting to reach incremental sync.. 2024-02-01T10:25:16.538 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:25:16.542+0000 7f712e81e980 1 waiting to reach incremental sync.. 2024-02-01T10:25:21.539 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:25:21.546+0000 7f712e81e980 1 waiting to reach incremental sync.. 2024-02-01T10:25:26.540 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:25:26.546+0000 7f712e81e980 1 waiting to reach incremental sync.. 2024-02-01T10:25:31.543 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:25:31.550+0000 7f712e81e980 1 bucket sync caught up with source: 2024-02-01T10:25:31.544 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, , , , , 00000000004.18.10, , , , , ] 2024-02-01T10:25:31.544 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, , , , , 00000000003.16.10, , , , , ] 2024-02-01T10:25:31.544 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:25:31.550+0000 7f712e81e980 0 bucket checkpoint complete 2024-02-01T10:25:31.583 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=vclfcn-19 2024-02-01T10:25:31.583 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:25:31.583 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:25:31.584 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:25:31.675 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:25:31.675 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:25:31.740 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:25:31.746+0000 7f0349821980 1 bucket sync caught up with source: 2024-02-01T10:25:31.740 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, , , , , 00000000004.18.10, , , , , ] 2024-02-01T10:25:31.740 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, , , , , 00000000004.18.10, , , , , ] 2024-02-01T10:25:31.740 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:25:31.746+0000 7f0349821980 0 bucket checkpoint complete 2024-02-01T10:25:31.765 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=vclfcn-19 2024-02-01T10:25:31.765 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:25:31.765 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:25:31.765 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:25:31.806 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:25:31.806 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:25:31.959 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:25:31.966+0000 7f63dd94a980 1 bucket sync caught up with source: 2024-02-01T10:25:31.960 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, , , , , 00000000003.24.10, , , , , ] 2024-02-01T10:25:31.960 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, , , , , 00000000003.24.10, , , , , ] 2024-02-01T10:25:31.960 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:25:31.966+0000 7f63dd94a980 0 bucket checkpoint complete 2024-02-01T10:25:31.998 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=vclfcn-19 2024-02-01T10:25:31.998 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:25:31.998 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:25:31.999 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket vclfcn-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-02-01T10:25:32.049 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:25:32.049 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:25:32.114 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:25:32.114+0000 7fba2a975980 1 bucket sync caught up with source: 2024-02-01T10:25:32.114 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, , , , , 00000000003.24.10, , , , , ] 2024-02-01T10:25:32.114 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, , , , , 00000000003.24.10, , , , , ] 2024-02-01T10:25:32.115 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:25:32.114+0000 7fba2a975980 0 bucket checkpoint complete 2024-02-01T10:25:32.137 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-19 zones={test-zone2, test-zone1} 2024-02-01T10:25:32.138 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:25:32.138 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:25:32.138 DEBUG:boto:Method: HEAD 2024-02-01T10:25:32.138 DEBUG:boto:Path: /vclfcn-19/ 2024-02-01T10:25:32.138 DEBUG:boto:Data: 2024-02-01T10:25:32.138 DEBUG:boto:Headers: {} 2024-02-01T10:25:32.139 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:25:32.139 DEBUG:boto:Port: 8000 2024-02-01T10:25:32.139 DEBUG:boto:Params: {} 2024-02-01T10:25:32.140 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T10:25:32.140 DEBUG:boto:Token: None 2024-02-01T10:25:32.140 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:25:32 GMT /vclfcn-19/ 2024-02-01T10:25:32.141 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:u/sC88oX23jSHOxuoaTPHRf0ujA= 2024-02-01T10:25:32.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:u/sC88oX23jSHOxuoaTPHRf0ujA=', 'Content-Length': '0'} 2024-02-01T10:25:32.146 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000239c4916c4944bd3-0065bb719c-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:32.146 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:25:32.146 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:25:32.146 DEBUG:boto:Method: HEAD 2024-02-01T10:25:32.147 DEBUG:boto:Path: /vclfcn-19/ 2024-02-01T10:25:32.147 DEBUG:boto:Data: 2024-02-01T10:25:32.147 DEBUG:boto:Headers: {} 2024-02-01T10:25:32.147 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:25:32.147 DEBUG:boto:Port: 8000 2024-02-01T10:25:32.147 DEBUG:boto:Params: {} 2024-02-01T10:25:32.147 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T10:25:32.147 DEBUG:boto:Token: None 2024-02-01T10:25:32.147 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:25:32 GMT /vclfcn-19/ 2024-02-01T10:25:32.148 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:u/sC88oX23jSHOxuoaTPHRf0ujA= 2024-02-01T10:25:32.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:u/sC88oX23jSHOxuoaTPHRf0ujA=', 'Content-Length': '0'} 2024-02-01T10:25:32.153 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cf77d6f6cba7edb2-0065bb719c-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:32.153 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:25:32.154 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:25:32.154 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:25:32.154 DEBUG:boto:path=/vclfcn-19/?versions&max-keys=999 2024-02-01T10:25:32.154 DEBUG:boto:auth_path=/vclfcn-19/?versions&max-keys=999 2024-02-01T10:25:32.154 DEBUG:boto:Method: GET 2024-02-01T10:25:32.154 DEBUG:boto:Path: /vclfcn-19/?versions&max-keys=999 2024-02-01T10:25:32.154 DEBUG:boto:Data: 2024-02-01T10:25:32.154 DEBUG:boto:Headers: {} 2024-02-01T10:25:32.154 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:25:32.155 DEBUG:boto:Port: 8000 2024-02-01T10:25:32.155 DEBUG:boto:Params: {} 2024-02-01T10:25:32.155 DEBUG:boto:Token: None 2024-02-01T10:25:32.155 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:25:32 GMT /vclfcn-19/?versions 2024-02-01T10:25:32.155 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:v68kRCiLHp9JI4kcOxydo6+KQy8= 2024-02-01T10:25:32.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:v68kRCiLHp9JI4kcOxydo6+KQy8=', 'Content-Length': '0'} 2024-02-01T10:25:32.157 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ba3c15fc94a63c0-0065bb719c-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:32.158 DEBUG:boto:b'vclfcn-19999falsedummyg5UScztMjPBIYZ2b5yOn-qsm.eQTcnutrue2024-02-01T10:23:11.115Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:25:32.158 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-02-01T10:25:32.159 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:25:32.159 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:25:32.159 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:25:32.159 DEBUG:boto:path=/vclfcn-19/?versions&max-keys=999 2024-02-01T10:25:32.159 DEBUG:boto:auth_path=/vclfcn-19/?versions&max-keys=999 2024-02-01T10:25:32.159 DEBUG:boto:Method: GET 2024-02-01T10:25:32.159 DEBUG:boto:Path: /vclfcn-19/?versions&max-keys=999 2024-02-01T10:25:32.159 DEBUG:boto:Data: 2024-02-01T10:25:32.159 DEBUG:boto:Headers: {} 2024-02-01T10:25:32.159 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:25:32.160 DEBUG:boto:Port: 8000 2024-02-01T10:25:32.160 DEBUG:boto:Params: {} 2024-02-01T10:25:32.160 DEBUG:boto:Token: None 2024-02-01T10:25:32.160 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:25:32 GMT /vclfcn-19/?versions 2024-02-01T10:25:32.160 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:v68kRCiLHp9JI4kcOxydo6+KQy8= 2024-02-01T10:25:32.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:v68kRCiLHp9JI4kcOxydo6+KQy8=', 'Content-Length': '0'} 2024-02-01T10:25:32.164 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001617fa2aa7f2a9be-0065bb719c-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:32.164 DEBUG:boto:b'vclfcn-19999falsedummyg5UScztMjPBIYZ2b5yOn-qsm.eQTcnutrue2024-02-01T10:23:11.115Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:25:32.165 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-02-01T10:25:32.165 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:25:32.165 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:25:32.165 DEBUG:boto:path=/vclfcn-19/?versions&max-keys=999 2024-02-01T10:25:32.165 DEBUG:boto:auth_path=/vclfcn-19/?versions&max-keys=999 2024-02-01T10:25:32.165 DEBUG:boto:Method: GET 2024-02-01T10:25:32.166 DEBUG:boto:Path: /vclfcn-19/?versions&max-keys=999 2024-02-01T10:25:32.166 DEBUG:boto:Data: 2024-02-01T10:25:32.166 DEBUG:boto:Headers: {} 2024-02-01T10:25:32.166 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:25:32.166 DEBUG:boto:Port: 8000 2024-02-01T10:25:32.166 DEBUG:boto:Params: {} 2024-02-01T10:25:32.166 DEBUG:boto:Token: None 2024-02-01T10:25:32.166 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:25:32 GMT /vclfcn-19/?versions 2024-02-01T10:25:32.166 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:v68kRCiLHp9JI4kcOxydo6+KQy8= 2024-02-01T10:25:32.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:v68kRCiLHp9JI4kcOxydo6+KQy8=', 'Content-Length': '0'} 2024-02-01T10:25:32.171 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e146d7325ea1e64-0065bb719c-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:32.172 DEBUG:boto:b'vclfcn-19999falsedummyg5UScztMjPBIYZ2b5yOn-qsm.eQTcnutrue2024-02-01T10:23:11.115Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:25:32.172 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:25:32.172 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:25:32.172 DEBUG:boto:path=/vclfcn-19/?versions&max-keys=999 2024-02-01T10:25:32.172 DEBUG:boto:auth_path=/vclfcn-19/?versions&max-keys=999 2024-02-01T10:25:32.172 DEBUG:boto:Method: GET 2024-02-01T10:25:32.172 DEBUG:boto:Path: /vclfcn-19/?versions&max-keys=999 2024-02-01T10:25:32.173 DEBUG:boto:Data: 2024-02-01T10:25:32.173 DEBUG:boto:Headers: {} 2024-02-01T10:25:32.173 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:25:32.173 DEBUG:boto:Port: 8000 2024-02-01T10:25:32.173 DEBUG:boto:Params: {} 2024-02-01T10:25:32.173 DEBUG:boto:Token: None 2024-02-01T10:25:32.174 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:25:32 GMT /vclfcn-19/?versions 2024-02-01T10:25:32.174 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:v68kRCiLHp9JI4kcOxydo6+KQy8= 2024-02-01T10:25:32.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:v68kRCiLHp9JI4kcOxydo6+KQy8=', 'Content-Length': '0'} 2024-02-01T10:25:32.179 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c9e5a0daddc4596-0065bb719c-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:32.179 DEBUG:boto:b'vclfcn-19999falsedummyg5UScztMjPBIYZ2b5yOn-qsm.eQTcnutrue2024-02-01T10:23:11.115Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:25:32.179 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-02-01T10:25:32.179 DEBUG:boto:path=/vclfcn-19/dummy 2024-02-01T10:25:32.179 DEBUG:boto:auth_path=/vclfcn-19/dummy 2024-02-01T10:25:32.179 DEBUG:boto:path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.180 DEBUG:boto:auth_path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.180 DEBUG:boto:Method: GET 2024-02-01T10:25:32.180 DEBUG:boto:Path: /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.180 DEBUG:boto:Data: 2024-02-01T10:25:32.180 DEBUG:boto:Headers: {} 2024-02-01T10:25:32.180 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:25:32.180 DEBUG:boto:Port: 8000 2024-02-01T10:25:32.180 DEBUG:boto:Params: {} 2024-02-01T10:25:32.180 DEBUG:boto:Token: None 2024-02-01T10:25:32.180 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:25:32 GMT /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.181 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BNkvIt/FGtrI4r4HrUKRhPzDHNc= 2024-02-01T10:25:32.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BNkvIt/FGtrI4r4HrUKRhPzDHNc=', 'Content-Length': '0'} 2024-02-01T10:25:32.183 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:23:11 GMT'), ('x-amz-version-id', 'g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b2bb84959f0f983c-0065bb719c-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:32.183 DEBUG:boto:path=/vclfcn-19/dummy 2024-02-01T10:25:32.183 DEBUG:boto:auth_path=/vclfcn-19/dummy 2024-02-01T10:25:32.183 DEBUG:boto:path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.184 DEBUG:boto:auth_path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.184 DEBUG:boto:Method: GET 2024-02-01T10:25:32.184 DEBUG:boto:Path: /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.184 DEBUG:boto:Data: 2024-02-01T10:25:32.184 DEBUG:boto:Headers: {} 2024-02-01T10:25:32.184 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:25:32.184 DEBUG:boto:Port: 8000 2024-02-01T10:25:32.184 DEBUG:boto:Params: {} 2024-02-01T10:25:32.184 DEBUG:boto:Token: None 2024-02-01T10:25:32.184 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:25:32 GMT /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.185 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BNkvIt/FGtrI4r4HrUKRhPzDHNc= 2024-02-01T10:25:32.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BNkvIt/FGtrI4r4HrUKRhPzDHNc=', 'Content-Length': '0'} 2024-02-01T10:25:32.187 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:23:11 GMT'), ('x-amz-version-id', 'g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d70441e9d4f938ae-0065bb719c-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:32.187 DEBUG:boto:path=/vclfcn-19/dummy 2024-02-01T10:25:32.188 DEBUG:boto:auth_path=/vclfcn-19/dummy 2024-02-01T10:25:32.188 DEBUG:boto:path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.188 DEBUG:boto:auth_path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.188 DEBUG:boto:Method: HEAD 2024-02-01T10:25:32.188 DEBUG:boto:Path: /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.188 DEBUG:boto:Data: 2024-02-01T10:25:32.188 DEBUG:boto:Headers: {} 2024-02-01T10:25:32.188 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:25:32.188 DEBUG:boto:Port: 8000 2024-02-01T10:25:32.188 DEBUG:boto:Params: {} 2024-02-01T10:25:32.188 DEBUG:boto:Token: None 2024-02-01T10:25:32.189 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:25:32 GMT /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.189 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:N/mhheEvN4guBWoVT/9oaHPWQEM= 2024-02-01T10:25:32.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:N/mhheEvN4guBWoVT/9oaHPWQEM=', 'Content-Length': '0'} 2024-02-01T10:25:32.190 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:23:11 GMT'), ('x-amz-version-id', 'g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004a30342e9ca433a2-0065bb719c-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:32.190 DEBUG:boto:path=/vclfcn-19/dummy 2024-02-01T10:25:32.191 DEBUG:boto:auth_path=/vclfcn-19/dummy 2024-02-01T10:25:32.191 DEBUG:boto:path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.191 DEBUG:boto:auth_path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.191 DEBUG:boto:Method: HEAD 2024-02-01T10:25:32.191 DEBUG:boto:Path: /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.191 DEBUG:boto:Data: 2024-02-01T10:25:32.191 DEBUG:boto:Headers: {} 2024-02-01T10:25:32.191 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:25:32.191 DEBUG:boto:Port: 8000 2024-02-01T10:25:32.191 DEBUG:boto:Params: {} 2024-02-01T10:25:32.192 DEBUG:boto:Token: None 2024-02-01T10:25:32.192 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:25:32 GMT /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.192 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:N/mhheEvN4guBWoVT/9oaHPWQEM= 2024-02-01T10:25:32.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:N/mhheEvN4guBWoVT/9oaHPWQEM=', 'Content-Length': '0'} 2024-02-01T10:25:32.194 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:23:11 GMT'), ('x-amz-version-id', 'g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d13c3b024d9ebd55-0065bb719c-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:32.195 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-02-01T10:25:32.195 DEBUG:boto:path=/vclfcn-19/dummy 2024-02-01T10:25:32.195 DEBUG:boto:auth_path=/vclfcn-19/dummy 2024-02-01T10:25:32.195 DEBUG:boto:path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.195 DEBUG:boto:auth_path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.195 DEBUG:boto:Method: GET 2024-02-01T10:25:32.195 DEBUG:boto:Path: /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.195 DEBUG:boto:Data: 2024-02-01T10:25:32.195 DEBUG:boto:Headers: {} 2024-02-01T10:25:32.196 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:25:32.196 DEBUG:boto:Port: 8000 2024-02-01T10:25:32.196 DEBUG:boto:Params: {} 2024-02-01T10:25:32.196 DEBUG:boto:Token: None 2024-02-01T10:25:32.196 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:25:32 GMT /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.196 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BNkvIt/FGtrI4r4HrUKRhPzDHNc= 2024-02-01T10:25:32.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BNkvIt/FGtrI4r4HrUKRhPzDHNc=', 'Content-Length': '0'} 2024-02-01T10:25:32.198 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:23:11 GMT'), ('x-amz-version-id', 'g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001deab4103ffcde9b-0065bb719c-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:32.199 DEBUG:boto:path=/vclfcn-19/dummy 2024-02-01T10:25:32.199 DEBUG:boto:auth_path=/vclfcn-19/dummy 2024-02-01T10:25:32.199 DEBUG:boto:path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.199 DEBUG:boto:auth_path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.199 DEBUG:boto:Method: GET 2024-02-01T10:25:32.199 DEBUG:boto:Path: /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.200 DEBUG:boto:Data: 2024-02-01T10:25:32.200 DEBUG:boto:Headers: {} 2024-02-01T10:25:32.200 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:25:32.200 DEBUG:boto:Port: 8000 2024-02-01T10:25:32.200 DEBUG:boto:Params: {} 2024-02-01T10:25:32.200 DEBUG:boto:Token: None 2024-02-01T10:25:32.200 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:25:32 GMT /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.200 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BNkvIt/FGtrI4r4HrUKRhPzDHNc= 2024-02-01T10:25:32.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BNkvIt/FGtrI4r4HrUKRhPzDHNc=', 'Content-Length': '0'} 2024-02-01T10:25:32.203 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:23:11 GMT'), ('x-amz-version-id', 'g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001c1f9fbecfe348b5-0065bb719c-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:32.204 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:25:32.204 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:25:32.204 DEBUG:boto:path=/vclfcn-19/?prefix=dummy 2024-02-01T10:25:32.204 DEBUG:boto:auth_path=/vclfcn-19/?prefix=dummy 2024-02-01T10:25:32.205 DEBUG:boto:Method: GET 2024-02-01T10:25:32.205 DEBUG:boto:Path: /vclfcn-19/?prefix=dummy 2024-02-01T10:25:32.205 DEBUG:boto:Data: 2024-02-01T10:25:32.205 DEBUG:boto:Headers: {} 2024-02-01T10:25:32.206 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:25:32.206 DEBUG:boto:Port: 8000 2024-02-01T10:25:32.206 DEBUG:boto:Params: {} 2024-02-01T10:25:32.207 DEBUG:boto:Token: None 2024-02-01T10:25:32.207 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:25:32 GMT /vclfcn-19/ 2024-02-01T10:25:32.207 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jj8mGsJ1c2d2+Omz51wa5ANBXLg= 2024-02-01T10:25:32.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jj8mGsJ1c2d2+Omz51wa5ANBXLg=', 'Content-Length': '0'} 2024-02-01T10:25:32.212 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006bbf1c40f0b5248d-0065bb719c-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:32.212 DEBUG:boto:b'vclfcn-19dummy1000falsedummy2024-02-01T10:23:11.115Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:25:32.215 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:25:32.215 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:25:32.215 DEBUG:boto:path=/vclfcn-19/?prefix=dummy 2024-02-01T10:25:32.215 DEBUG:boto:auth_path=/vclfcn-19/?prefix=dummy 2024-02-01T10:25:32.215 DEBUG:boto:Method: GET 2024-02-01T10:25:32.215 DEBUG:boto:Path: /vclfcn-19/?prefix=dummy 2024-02-01T10:25:32.215 DEBUG:boto:Data: 2024-02-01T10:25:32.215 DEBUG:boto:Headers: {} 2024-02-01T10:25:32.215 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:25:32.216 DEBUG:boto:Port: 8000 2024-02-01T10:25:32.216 DEBUG:boto:Params: {} 2024-02-01T10:25:32.216 DEBUG:boto:Token: None 2024-02-01T10:25:32.216 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:25:32 GMT /vclfcn-19/ 2024-02-01T10:25:32.216 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jj8mGsJ1c2d2+Omz51wa5ANBXLg= 2024-02-01T10:25:32.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jj8mGsJ1c2d2+Omz51wa5ANBXLg=', 'Content-Length': '0'} 2024-02-01T10:25:32.221 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f93b00213c44564-0065bb719c-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:32.268 DEBUG:boto:b'vclfcn-19dummy1000falsedummy2024-02-01T10:23:11.115Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:25:32.269 DEBUG:boto:path=/vclfcn-19/dummy 2024-02-01T10:25:32.269 DEBUG:boto:auth_path=/vclfcn-19/dummy 2024-02-01T10:25:32.269 DEBUG:boto:Method: HEAD 2024-02-01T10:25:32.270 DEBUG:boto:Path: /vclfcn-19/dummy 2024-02-01T10:25:32.270 DEBUG:boto:Data: 2024-02-01T10:25:32.270 DEBUG:boto:Headers: {} 2024-02-01T10:25:32.270 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:25:32.270 DEBUG:boto:Port: 8000 2024-02-01T10:25:32.271 DEBUG:boto:Params: {} 2024-02-01T10:25:32.271 DEBUG:boto:Token: None 2024-02-01T10:25:32.271 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:25:32 GMT /vclfcn-19/dummy 2024-02-01T10:25:32.271 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9Rrcc8ka5VLlQHUgXrrhDPHtKpc= 2024-02-01T10:25:32.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9Rrcc8ka5VLlQHUgXrrhDPHtKpc=', 'Content-Length': '0'} 2024-02-01T10:25:32.276 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:23:11 GMT'), ('x-amz-version-id', 'g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000001d9025d94f0cb12-0065bb719c-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:32.276 DEBUG:boto:path=/vclfcn-19/dummy 2024-02-01T10:25:32.276 DEBUG:boto:auth_path=/vclfcn-19/dummy 2024-02-01T10:25:32.277 DEBUG:boto:Method: HEAD 2024-02-01T10:25:32.277 DEBUG:boto:Path: /vclfcn-19/dummy 2024-02-01T10:25:32.277 DEBUG:boto:Data: 2024-02-01T10:25:32.277 DEBUG:boto:Headers: {} 2024-02-01T10:25:32.277 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:25:32.277 DEBUG:boto:Port: 8000 2024-02-01T10:25:32.277 DEBUG:boto:Params: {} 2024-02-01T10:25:32.277 DEBUG:boto:Token: None 2024-02-01T10:25:32.277 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:25:32 GMT /vclfcn-19/dummy 2024-02-01T10:25:32.278 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9Rrcc8ka5VLlQHUgXrrhDPHtKpc= 2024-02-01T10:25:32.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9Rrcc8ka5VLlQHUgXrrhDPHtKpc=', 'Content-Length': '0'} 2024-02-01T10:25:32.281 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:23:11 GMT'), ('x-amz-version-id', 'g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000319ba0ab7a34ea41-0065bb719c-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:32.281 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-02-01T10:25:32.282 DEBUG:boto:path=/vclfcn-19/dummy 2024-02-01T10:25:32.282 DEBUG:boto:auth_path=/vclfcn-19/dummy 2024-02-01T10:25:32.282 DEBUG:boto:path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.282 DEBUG:boto:auth_path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.282 DEBUG:boto:Method: GET 2024-02-01T10:25:32.282 DEBUG:boto:Path: /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.282 DEBUG:boto:Data: 2024-02-01T10:25:32.282 DEBUG:boto:Headers: {} 2024-02-01T10:25:32.282 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:25:32.282 DEBUG:boto:Port: 8000 2024-02-01T10:25:32.283 DEBUG:boto:Params: {} 2024-02-01T10:25:32.283 DEBUG:boto:Token: None 2024-02-01T10:25:32.283 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:25:32 GMT /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.283 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BNkvIt/FGtrI4r4HrUKRhPzDHNc= 2024-02-01T10:25:32.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BNkvIt/FGtrI4r4HrUKRhPzDHNc=', 'Content-Length': '0'} 2024-02-01T10:25:32.286 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:23:11 GMT'), ('x-amz-version-id', 'g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c1f5f3596ca23a2a-0065bb719c-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:32.286 DEBUG:boto:path=/vclfcn-19/dummy 2024-02-01T10:25:32.286 DEBUG:boto:auth_path=/vclfcn-19/dummy 2024-02-01T10:25:32.286 DEBUG:boto:path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.286 DEBUG:boto:auth_path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.287 DEBUG:boto:Method: GET 2024-02-01T10:25:32.287 DEBUG:boto:Path: /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.287 DEBUG:boto:Data: 2024-02-01T10:25:32.287 DEBUG:boto:Headers: {} 2024-02-01T10:25:32.287 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:25:32.287 DEBUG:boto:Port: 8000 2024-02-01T10:25:32.287 DEBUG:boto:Params: {} 2024-02-01T10:25:32.287 DEBUG:boto:Token: None 2024-02-01T10:25:32.287 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:25:32 GMT /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.287 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BNkvIt/FGtrI4r4HrUKRhPzDHNc= 2024-02-01T10:25:32.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BNkvIt/FGtrI4r4HrUKRhPzDHNc=', 'Content-Length': '0'} 2024-02-01T10:25:32.290 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:23:11 GMT'), ('x-amz-version-id', 'g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000dac59043d5f1f08c-0065bb719c-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:32.291 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:25:32.291 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:25:32.291 DEBUG:boto:path=/vclfcn-19/?prefix=dummy 2024-02-01T10:25:32.291 DEBUG:boto:auth_path=/vclfcn-19/?prefix=dummy 2024-02-01T10:25:32.291 DEBUG:boto:Method: GET 2024-02-01T10:25:32.291 DEBUG:boto:Path: /vclfcn-19/?prefix=dummy 2024-02-01T10:25:32.291 DEBUG:boto:Data: 2024-02-01T10:25:32.291 DEBUG:boto:Headers: {} 2024-02-01T10:25:32.291 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:25:32.291 DEBUG:boto:Port: 8000 2024-02-01T10:25:32.291 DEBUG:boto:Params: {} 2024-02-01T10:25:32.292 DEBUG:boto:Token: None 2024-02-01T10:25:32.292 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:25:32 GMT /vclfcn-19/ 2024-02-01T10:25:32.293 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jj8mGsJ1c2d2+Omz51wa5ANBXLg= 2024-02-01T10:25:32.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jj8mGsJ1c2d2+Omz51wa5ANBXLg=', 'Content-Length': '0'} 2024-02-01T10:25:32.297 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000085597c3f56747b88-0065bb719c-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:32.344 DEBUG:boto:b'vclfcn-19dummy1000falsedummy2024-02-01T10:23:11.115Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:25:32.344 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:25:32.345 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:25:32.345 DEBUG:boto:path=/vclfcn-19/?prefix=dummy 2024-02-01T10:25:32.345 DEBUG:boto:auth_path=/vclfcn-19/?prefix=dummy 2024-02-01T10:25:32.345 DEBUG:boto:Method: GET 2024-02-01T10:25:32.345 DEBUG:boto:Path: /vclfcn-19/?prefix=dummy 2024-02-01T10:25:32.345 DEBUG:boto:Data: 2024-02-01T10:25:32.345 DEBUG:boto:Headers: {} 2024-02-01T10:25:32.345 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:25:32.345 DEBUG:boto:Port: 8000 2024-02-01T10:25:32.345 DEBUG:boto:Params: {} 2024-02-01T10:25:32.346 DEBUG:boto:Token: None 2024-02-01T10:25:32.346 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:25:32 GMT /vclfcn-19/ 2024-02-01T10:25:32.346 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jj8mGsJ1c2d2+Omz51wa5ANBXLg= 2024-02-01T10:25:32.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jj8mGsJ1c2d2+Omz51wa5ANBXLg=', 'Content-Length': '0'} 2024-02-01T10:25:32.350 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094fb4ea7a9e6d746-0065bb719c-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:32.350 DEBUG:boto:b'vclfcn-19dummy1000falsedummy2024-02-01T10:23:11.115Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:25:32.351 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-19 zones={test-zone2, test-zone1} 2024-02-01T10:25:32.351 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-19 zones={test-zone3, test-zone1} 2024-02-01T10:25:32.352 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:25:32.352 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:25:32.352 DEBUG:boto:Method: HEAD 2024-02-01T10:25:32.352 DEBUG:boto:Path: /vclfcn-19/ 2024-02-01T10:25:32.352 DEBUG:boto:Data: 2024-02-01T10:25:32.352 DEBUG:boto:Headers: {} 2024-02-01T10:25:32.352 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:25:32.352 DEBUG:boto:Port: 8001 2024-02-01T10:25:32.352 DEBUG:boto:Params: {} 2024-02-01T10:25:32.353 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-02-01T10:25:32.353 DEBUG:boto:Token: None 2024-02-01T10:25:32.353 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:25:32 GMT /vclfcn-19/ 2024-02-01T10:25:32.353 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:u/sC88oX23jSHOxuoaTPHRf0ujA= 2024-02-01T10:25:32.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:u/sC88oX23jSHOxuoaTPHRf0ujA=', 'Content-Length': '0'} 2024-02-01T10:25:32.359 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a2db6ca1dc06bf6b-0065bb719c-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:32.359 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:25:32.360 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:25:32.360 DEBUG:boto:Method: HEAD 2024-02-01T10:25:32.360 DEBUG:boto:Path: /vclfcn-19/ 2024-02-01T10:25:32.360 DEBUG:boto:Data: 2024-02-01T10:25:32.360 DEBUG:boto:Headers: {} 2024-02-01T10:25:32.360 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:25:32.360 DEBUG:boto:Port: 8000 2024-02-01T10:25:32.360 DEBUG:boto:Params: {} 2024-02-01T10:25:32.360 DEBUG:boto:Token: None 2024-02-01T10:25:32.361 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:25:32 GMT /vclfcn-19/ 2024-02-01T10:25:32.361 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:u/sC88oX23jSHOxuoaTPHRf0ujA= 2024-02-01T10:25:32.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:u/sC88oX23jSHOxuoaTPHRf0ujA=', 'Content-Length': '0'} 2024-02-01T10:25:32.365 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000083b82460cdda380b-0065bb719c-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:32.365 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:25:32.365 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:25:32.365 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:25:32.365 DEBUG:boto:path=/vclfcn-19/?versions&max-keys=999 2024-02-01T10:25:32.365 DEBUG:boto:auth_path=/vclfcn-19/?versions&max-keys=999 2024-02-01T10:25:32.365 DEBUG:boto:Method: GET 2024-02-01T10:25:32.365 DEBUG:boto:Path: /vclfcn-19/?versions&max-keys=999 2024-02-01T10:25:32.365 DEBUG:boto:Data: 2024-02-01T10:25:32.366 DEBUG:boto:Headers: {} 2024-02-01T10:25:32.366 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:25:32.366 DEBUG:boto:Port: 8001 2024-02-01T10:25:32.366 DEBUG:boto:Params: {} 2024-02-01T10:25:32.366 DEBUG:boto:Token: None 2024-02-01T10:25:32.366 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:25:32 GMT /vclfcn-19/?versions 2024-02-01T10:25:32.366 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:v68kRCiLHp9JI4kcOxydo6+KQy8= 2024-02-01T10:25:32.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:v68kRCiLHp9JI4kcOxydo6+KQy8=', 'Content-Length': '0'} 2024-02-01T10:25:32.370 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000768d1e7cfb6688fa-0065bb719c-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:32.371 DEBUG:boto:b'vclfcn-19999falsedummyg5UScztMjPBIYZ2b5yOn-qsm.eQTcnutrue2024-02-01T10:23:11.115Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:25:32.371 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-02-01T10:25:32.371 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:25:32.371 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:25:32.371 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:25:32.371 DEBUG:boto:path=/vclfcn-19/?versions&max-keys=999 2024-02-01T10:25:32.372 DEBUG:boto:auth_path=/vclfcn-19/?versions&max-keys=999 2024-02-01T10:25:32.372 DEBUG:boto:Method: GET 2024-02-01T10:25:32.372 DEBUG:boto:Path: /vclfcn-19/?versions&max-keys=999 2024-02-01T10:25:32.372 DEBUG:boto:Data: 2024-02-01T10:25:32.372 DEBUG:boto:Headers: {} 2024-02-01T10:25:32.372 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:25:32.372 DEBUG:boto:Port: 8000 2024-02-01T10:25:32.372 DEBUG:boto:Params: {} 2024-02-01T10:25:32.372 DEBUG:boto:Token: None 2024-02-01T10:25:32.372 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:25:32 GMT /vclfcn-19/?versions 2024-02-01T10:25:32.373 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:v68kRCiLHp9JI4kcOxydo6+KQy8= 2024-02-01T10:25:32.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:v68kRCiLHp9JI4kcOxydo6+KQy8=', 'Content-Length': '0'} 2024-02-01T10:25:32.381 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000960bb3dfdc700433-0065bb719c-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:32.420 DEBUG:boto:b'vclfcn-19999falsedummyg5UScztMjPBIYZ2b5yOn-qsm.eQTcnutrue2024-02-01T10:23:11.115Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:25:32.421 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-02-01T10:25:32.421 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:25:32.421 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:25:32.422 DEBUG:boto:path=/vclfcn-19/?versions&max-keys=999 2024-02-01T10:25:32.422 DEBUG:boto:auth_path=/vclfcn-19/?versions&max-keys=999 2024-02-01T10:25:32.422 DEBUG:boto:Method: GET 2024-02-01T10:25:32.422 DEBUG:boto:Path: /vclfcn-19/?versions&max-keys=999 2024-02-01T10:25:32.422 DEBUG:boto:Data: 2024-02-01T10:25:32.423 DEBUG:boto:Headers: {} 2024-02-01T10:25:32.423 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:25:32.423 DEBUG:boto:Port: 8001 2024-02-01T10:25:32.423 DEBUG:boto:Params: {} 2024-02-01T10:25:32.424 DEBUG:boto:Token: None 2024-02-01T10:25:32.424 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:25:32 GMT /vclfcn-19/?versions 2024-02-01T10:25:32.424 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:v68kRCiLHp9JI4kcOxydo6+KQy8= 2024-02-01T10:25:32.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:v68kRCiLHp9JI4kcOxydo6+KQy8=', 'Content-Length': '0'} 2024-02-01T10:25:32.429 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b0bcc6a1b038d1a8-0065bb719c-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:32.429 DEBUG:boto:b'vclfcn-19999falsedummyg5UScztMjPBIYZ2b5yOn-qsm.eQTcnutrue2024-02-01T10:23:11.115Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:25:32.429 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:25:32.429 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:25:32.429 DEBUG:boto:path=/vclfcn-19/?versions&max-keys=999 2024-02-01T10:25:32.430 DEBUG:boto:auth_path=/vclfcn-19/?versions&max-keys=999 2024-02-01T10:25:32.430 DEBUG:boto:Method: GET 2024-02-01T10:25:32.430 DEBUG:boto:Path: /vclfcn-19/?versions&max-keys=999 2024-02-01T10:25:32.430 DEBUG:boto:Data: 2024-02-01T10:25:32.430 DEBUG:boto:Headers: {} 2024-02-01T10:25:32.430 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:25:32.430 DEBUG:boto:Port: 8000 2024-02-01T10:25:32.430 DEBUG:boto:Params: {} 2024-02-01T10:25:32.430 DEBUG:boto:Token: None 2024-02-01T10:25:32.431 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:25:32 GMT /vclfcn-19/?versions 2024-02-01T10:25:32.431 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:v68kRCiLHp9JI4kcOxydo6+KQy8= 2024-02-01T10:25:32.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:v68kRCiLHp9JI4kcOxydo6+KQy8=', 'Content-Length': '0'} 2024-02-01T10:25:32.437 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f4b7babac1b2578-0065bb719c-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:32.437 DEBUG:boto:b'vclfcn-19999falsedummyg5UScztMjPBIYZ2b5yOn-qsm.eQTcnutrue2024-02-01T10:23:11.115Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:25:32.438 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-02-01T10:25:32.438 DEBUG:boto:path=/vclfcn-19/dummy 2024-02-01T10:25:32.438 DEBUG:boto:auth_path=/vclfcn-19/dummy 2024-02-01T10:25:32.438 DEBUG:boto:path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.439 DEBUG:boto:auth_path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.439 DEBUG:boto:Method: GET 2024-02-01T10:25:32.439 DEBUG:boto:Path: /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.439 DEBUG:boto:Data: 2024-02-01T10:25:32.440 DEBUG:boto:Headers: {} 2024-02-01T10:25:32.440 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:25:32.440 DEBUG:boto:Port: 8001 2024-02-01T10:25:32.440 DEBUG:boto:Params: {} 2024-02-01T10:25:32.440 DEBUG:boto:Token: None 2024-02-01T10:25:32.441 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:25:32 GMT /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.441 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BNkvIt/FGtrI4r4HrUKRhPzDHNc= 2024-02-01T10:25:32.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BNkvIt/FGtrI4r4HrUKRhPzDHNc=', 'Content-Length': '0'} 2024-02-01T10:25:32.444 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:23:11 GMT'), ('x-amz-version-id', 'g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000b61054a6fc0f45b-0065bb719c-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:32.445 DEBUG:boto:path=/vclfcn-19/dummy 2024-02-01T10:25:32.445 DEBUG:boto:auth_path=/vclfcn-19/dummy 2024-02-01T10:25:32.445 DEBUG:boto:path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.445 DEBUG:boto:auth_path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.445 DEBUG:boto:Method: GET 2024-02-01T10:25:32.445 DEBUG:boto:Path: /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.445 DEBUG:boto:Data: 2024-02-01T10:25:32.445 DEBUG:boto:Headers: {} 2024-02-01T10:25:32.445 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:25:32.446 DEBUG:boto:Port: 8000 2024-02-01T10:25:32.446 DEBUG:boto:Params: {} 2024-02-01T10:25:32.446 DEBUG:boto:Token: None 2024-02-01T10:25:32.446 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:25:32 GMT /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.446 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BNkvIt/FGtrI4r4HrUKRhPzDHNc= 2024-02-01T10:25:32.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BNkvIt/FGtrI4r4HrUKRhPzDHNc=', 'Content-Length': '0'} 2024-02-01T10:25:32.449 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:23:11 GMT'), ('x-amz-version-id', 'g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000096776888990476d6-0065bb719c-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:32.450 DEBUG:boto:path=/vclfcn-19/dummy 2024-02-01T10:25:32.450 DEBUG:boto:auth_path=/vclfcn-19/dummy 2024-02-01T10:25:32.450 DEBUG:boto:path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.450 DEBUG:boto:auth_path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.450 DEBUG:boto:Method: HEAD 2024-02-01T10:25:32.451 DEBUG:boto:Path: /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.451 DEBUG:boto:Data: 2024-02-01T10:25:32.451 DEBUG:boto:Headers: {} 2024-02-01T10:25:32.451 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:25:32.451 DEBUG:boto:Port: 8001 2024-02-01T10:25:32.451 DEBUG:boto:Params: {} 2024-02-01T10:25:32.451 DEBUG:boto:Token: None 2024-02-01T10:25:32.451 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:25:32 GMT /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.451 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:N/mhheEvN4guBWoVT/9oaHPWQEM= 2024-02-01T10:25:32.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:N/mhheEvN4guBWoVT/9oaHPWQEM=', 'Content-Length': '0'} 2024-02-01T10:25:32.456 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:23:11 GMT'), ('x-amz-version-id', 'g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000699625285a6c2bc1-0065bb719c-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:32.456 DEBUG:boto:path=/vclfcn-19/dummy 2024-02-01T10:25:32.456 DEBUG:boto:auth_path=/vclfcn-19/dummy 2024-02-01T10:25:32.456 DEBUG:boto:path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.456 DEBUG:boto:auth_path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.456 DEBUG:boto:Method: HEAD 2024-02-01T10:25:32.457 DEBUG:boto:Path: /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.457 DEBUG:boto:Data: 2024-02-01T10:25:32.457 DEBUG:boto:Headers: {} 2024-02-01T10:25:32.457 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:25:32.457 DEBUG:boto:Port: 8000 2024-02-01T10:25:32.457 DEBUG:boto:Params: {} 2024-02-01T10:25:32.457 DEBUG:boto:Token: None 2024-02-01T10:25:32.457 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:25:32 GMT /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.457 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:N/mhheEvN4guBWoVT/9oaHPWQEM= 2024-02-01T10:25:32.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:N/mhheEvN4guBWoVT/9oaHPWQEM=', 'Content-Length': '0'} 2024-02-01T10:25:32.460 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:23:11 GMT'), ('x-amz-version-id', 'g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d9c35ae7ea006eeb-0065bb719c-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:32.460 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-02-01T10:25:32.460 DEBUG:boto:path=/vclfcn-19/dummy 2024-02-01T10:25:32.461 DEBUG:boto:auth_path=/vclfcn-19/dummy 2024-02-01T10:25:32.461 DEBUG:boto:path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.461 DEBUG:boto:auth_path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.461 DEBUG:boto:Method: GET 2024-02-01T10:25:32.461 DEBUG:boto:Path: /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.461 DEBUG:boto:Data: 2024-02-01T10:25:32.461 DEBUG:boto:Headers: {} 2024-02-01T10:25:32.461 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:25:32.461 DEBUG:boto:Port: 8001 2024-02-01T10:25:32.461 DEBUG:boto:Params: {} 2024-02-01T10:25:32.461 DEBUG:boto:Token: None 2024-02-01T10:25:32.462 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:25:32 GMT /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.462 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BNkvIt/FGtrI4r4HrUKRhPzDHNc= 2024-02-01T10:25:32.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BNkvIt/FGtrI4r4HrUKRhPzDHNc=', 'Content-Length': '0'} 2024-02-01T10:25:32.464 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:23:11 GMT'), ('x-amz-version-id', 'g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000050e14ac2aa6448d-0065bb719c-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:32.465 DEBUG:boto:path=/vclfcn-19/dummy 2024-02-01T10:25:32.465 DEBUG:boto:auth_path=/vclfcn-19/dummy 2024-02-01T10:25:32.465 DEBUG:boto:path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.465 DEBUG:boto:auth_path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.465 DEBUG:boto:Method: GET 2024-02-01T10:25:32.465 DEBUG:boto:Path: /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.465 DEBUG:boto:Data: 2024-02-01T10:25:32.465 DEBUG:boto:Headers: {} 2024-02-01T10:25:32.465 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:25:32.465 DEBUG:boto:Port: 8000 2024-02-01T10:25:32.465 DEBUG:boto:Params: {} 2024-02-01T10:25:32.466 DEBUG:boto:Token: None 2024-02-01T10:25:32.466 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:25:32 GMT /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.466 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BNkvIt/FGtrI4r4HrUKRhPzDHNc= 2024-02-01T10:25:32.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BNkvIt/FGtrI4r4HrUKRhPzDHNc=', 'Content-Length': '0'} 2024-02-01T10:25:32.468 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:23:11 GMT'), ('x-amz-version-id', 'g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fdf6b6f9a8f6a8e7-0065bb719c-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:32.469 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:25:32.469 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:25:32.469 DEBUG:boto:path=/vclfcn-19/?prefix=dummy 2024-02-01T10:25:32.469 DEBUG:boto:auth_path=/vclfcn-19/?prefix=dummy 2024-02-01T10:25:32.469 DEBUG:boto:Method: GET 2024-02-01T10:25:32.469 DEBUG:boto:Path: /vclfcn-19/?prefix=dummy 2024-02-01T10:25:32.469 DEBUG:boto:Data: 2024-02-01T10:25:32.469 DEBUG:boto:Headers: {} 2024-02-01T10:25:32.469 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:25:32.469 DEBUG:boto:Port: 8001 2024-02-01T10:25:32.470 DEBUG:boto:Params: {} 2024-02-01T10:25:32.470 DEBUG:boto:Token: None 2024-02-01T10:25:32.470 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:25:32 GMT /vclfcn-19/ 2024-02-01T10:25:32.470 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jj8mGsJ1c2d2+Omz51wa5ANBXLg= 2024-02-01T10:25:32.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jj8mGsJ1c2d2+Omz51wa5ANBXLg=', 'Content-Length': '0'} 2024-02-01T10:25:32.476 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e90389e22abe781-0065bb719c-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:32.516 DEBUG:boto:b'vclfcn-19dummy1000falsedummy2024-02-01T10:23:11.115Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:25:32.517 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:25:32.517 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:25:32.517 DEBUG:boto:path=/vclfcn-19/?prefix=dummy 2024-02-01T10:25:32.517 DEBUG:boto:auth_path=/vclfcn-19/?prefix=dummy 2024-02-01T10:25:32.517 DEBUG:boto:Method: GET 2024-02-01T10:25:32.517 DEBUG:boto:Path: /vclfcn-19/?prefix=dummy 2024-02-01T10:25:32.517 DEBUG:boto:Data: 2024-02-01T10:25:32.517 DEBUG:boto:Headers: {} 2024-02-01T10:25:32.517 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:25:32.517 DEBUG:boto:Port: 8000 2024-02-01T10:25:32.517 DEBUG:boto:Params: {} 2024-02-01T10:25:32.518 DEBUG:boto:Token: None 2024-02-01T10:25:32.518 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:25:32 GMT /vclfcn-19/ 2024-02-01T10:25:32.518 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jj8mGsJ1c2d2+Omz51wa5ANBXLg= 2024-02-01T10:25:32.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jj8mGsJ1c2d2+Omz51wa5ANBXLg=', 'Content-Length': '0'} 2024-02-01T10:25:32.522 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000007a65c58575802ee-0065bb719c-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:32.522 DEBUG:boto:b'vclfcn-19dummy1000falsedummy2024-02-01T10:23:11.115Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:25:32.523 DEBUG:boto:path=/vclfcn-19/dummy 2024-02-01T10:25:32.523 DEBUG:boto:auth_path=/vclfcn-19/dummy 2024-02-01T10:25:32.523 DEBUG:boto:Method: HEAD 2024-02-01T10:25:32.523 DEBUG:boto:Path: /vclfcn-19/dummy 2024-02-01T10:25:32.523 DEBUG:boto:Data: 2024-02-01T10:25:32.523 DEBUG:boto:Headers: {} 2024-02-01T10:25:32.523 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:25:32.524 DEBUG:boto:Port: 8001 2024-02-01T10:25:32.524 DEBUG:boto:Params: {} 2024-02-01T10:25:32.524 DEBUG:boto:Token: None 2024-02-01T10:25:32.524 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:25:32 GMT /vclfcn-19/dummy 2024-02-01T10:25:32.524 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9Rrcc8ka5VLlQHUgXrrhDPHtKpc= 2024-02-01T10:25:32.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9Rrcc8ka5VLlQHUgXrrhDPHtKpc=', 'Content-Length': '0'} 2024-02-01T10:25:32.528 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:23:11 GMT'), ('x-amz-version-id', 'g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002b179c8009e96c7f-0065bb719c-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:32.528 DEBUG:boto:path=/vclfcn-19/dummy 2024-02-01T10:25:32.528 DEBUG:boto:auth_path=/vclfcn-19/dummy 2024-02-01T10:25:32.528 DEBUG:boto:Method: HEAD 2024-02-01T10:25:32.529 DEBUG:boto:Path: /vclfcn-19/dummy 2024-02-01T10:25:32.529 DEBUG:boto:Data: 2024-02-01T10:25:32.529 DEBUG:boto:Headers: {} 2024-02-01T10:25:32.529 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:25:32.529 DEBUG:boto:Port: 8000 2024-02-01T10:25:32.529 DEBUG:boto:Params: {} 2024-02-01T10:25:32.529 DEBUG:boto:Token: None 2024-02-01T10:25:32.529 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:25:32 GMT /vclfcn-19/dummy 2024-02-01T10:25:32.529 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9Rrcc8ka5VLlQHUgXrrhDPHtKpc= 2024-02-01T10:25:32.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9Rrcc8ka5VLlQHUgXrrhDPHtKpc=', 'Content-Length': '0'} 2024-02-01T10:25:32.534 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:23:11 GMT'), ('x-amz-version-id', 'g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009eba9c796005d5fa-0065bb719c-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:32.534 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-02-01T10:25:32.534 DEBUG:boto:path=/vclfcn-19/dummy 2024-02-01T10:25:32.534 DEBUG:boto:auth_path=/vclfcn-19/dummy 2024-02-01T10:25:32.534 DEBUG:boto:path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.534 DEBUG:boto:auth_path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.535 DEBUG:boto:Method: GET 2024-02-01T10:25:32.535 DEBUG:boto:Path: /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.535 DEBUG:boto:Data: 2024-02-01T10:25:32.535 DEBUG:boto:Headers: {} 2024-02-01T10:25:32.535 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:25:32.535 DEBUG:boto:Port: 8001 2024-02-01T10:25:32.535 DEBUG:boto:Params: {} 2024-02-01T10:25:32.535 DEBUG:boto:Token: None 2024-02-01T10:25:32.535 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:25:32 GMT /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.535 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BNkvIt/FGtrI4r4HrUKRhPzDHNc= 2024-02-01T10:25:32.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BNkvIt/FGtrI4r4HrUKRhPzDHNc=', 'Content-Length': '0'} 2024-02-01T10:25:32.538 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:23:11 GMT'), ('x-amz-version-id', 'g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000003c047ed7a6398a6-0065bb719c-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:32.539 DEBUG:boto:path=/vclfcn-19/dummy 2024-02-01T10:25:32.539 DEBUG:boto:auth_path=/vclfcn-19/dummy 2024-02-01T10:25:32.539 DEBUG:boto:path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.539 DEBUG:boto:auth_path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.539 DEBUG:boto:Method: GET 2024-02-01T10:25:32.540 DEBUG:boto:Path: /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.540 DEBUG:boto:Data: 2024-02-01T10:25:32.540 DEBUG:boto:Headers: {} 2024-02-01T10:25:32.540 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:25:32.540 DEBUG:boto:Port: 8000 2024-02-01T10:25:32.540 DEBUG:boto:Params: {} 2024-02-01T10:25:32.540 DEBUG:boto:Token: None 2024-02-01T10:25:32.540 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:25:32 GMT /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.540 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BNkvIt/FGtrI4r4HrUKRhPzDHNc= 2024-02-01T10:25:32.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BNkvIt/FGtrI4r4HrUKRhPzDHNc=', 'Content-Length': '0'} 2024-02-01T10:25:32.544 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:23:11 GMT'), ('x-amz-version-id', 'g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f10ea15c2ad081fd-0065bb719c-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:32.545 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:25:32.545 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:25:32.545 DEBUG:boto:path=/vclfcn-19/?prefix=dummy 2024-02-01T10:25:32.545 DEBUG:boto:auth_path=/vclfcn-19/?prefix=dummy 2024-02-01T10:25:32.545 DEBUG:boto:Method: GET 2024-02-01T10:25:32.545 DEBUG:boto:Path: /vclfcn-19/?prefix=dummy 2024-02-01T10:25:32.545 DEBUG:boto:Data: 2024-02-01T10:25:32.545 DEBUG:boto:Headers: {} 2024-02-01T10:25:32.546 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:25:32.546 DEBUG:boto:Port: 8001 2024-02-01T10:25:32.546 DEBUG:boto:Params: {} 2024-02-01T10:25:32.546 DEBUG:boto:Token: None 2024-02-01T10:25:32.546 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:25:32 GMT /vclfcn-19/ 2024-02-01T10:25:32.546 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jj8mGsJ1c2d2+Omz51wa5ANBXLg= 2024-02-01T10:25:32.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jj8mGsJ1c2d2+Omz51wa5ANBXLg=', 'Content-Length': '0'} 2024-02-01T10:25:32.551 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d13ed5362347aaba-0065bb719c-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:32.551 DEBUG:boto:b'vclfcn-19dummy1000falsedummy2024-02-01T10:23:11.115Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:25:32.552 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:25:32.552 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:25:32.552 DEBUG:boto:path=/vclfcn-19/?prefix=dummy 2024-02-01T10:25:32.552 DEBUG:boto:auth_path=/vclfcn-19/?prefix=dummy 2024-02-01T10:25:32.552 DEBUG:boto:Method: GET 2024-02-01T10:25:32.552 DEBUG:boto:Path: /vclfcn-19/?prefix=dummy 2024-02-01T10:25:32.552 DEBUG:boto:Data: 2024-02-01T10:25:32.552 DEBUG:boto:Headers: {} 2024-02-01T10:25:32.553 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:25:32.553 DEBUG:boto:Port: 8000 2024-02-01T10:25:32.553 DEBUG:boto:Params: {} 2024-02-01T10:25:32.554 DEBUG:boto:Token: None 2024-02-01T10:25:32.554 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:25:32 GMT /vclfcn-19/ 2024-02-01T10:25:32.554 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jj8mGsJ1c2d2+Omz51wa5ANBXLg= 2024-02-01T10:25:32.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jj8mGsJ1c2d2+Omz51wa5ANBXLg=', 'Content-Length': '0'} 2024-02-01T10:25:32.558 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003fbc45d0c426e616-0065bb719c-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:32.559 DEBUG:boto:b'vclfcn-19dummy1000falsedummy2024-02-01T10:23:11.115Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:25:32.559 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-19 zones={test-zone3, test-zone1} 2024-02-01T10:25:32.559 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-19 zones={test-zone3, test-zone2} 2024-02-01T10:25:32.559 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:25:32.559 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:25:32.559 DEBUG:boto:Method: HEAD 2024-02-01T10:25:32.560 DEBUG:boto:Path: /vclfcn-19/ 2024-02-01T10:25:32.560 DEBUG:boto:Data: 2024-02-01T10:25:32.560 DEBUG:boto:Headers: {} 2024-02-01T10:25:32.560 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:25:32.560 DEBUG:boto:Port: 8001 2024-02-01T10:25:32.560 DEBUG:boto:Params: {} 2024-02-01T10:25:32.560 DEBUG:boto:Token: None 2024-02-01T10:25:32.560 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:25:32 GMT /vclfcn-19/ 2024-02-01T10:25:32.560 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:u/sC88oX23jSHOxuoaTPHRf0ujA= 2024-02-01T10:25:32.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:u/sC88oX23jSHOxuoaTPHRf0ujA=', 'Content-Length': '0'} 2024-02-01T10:25:32.565 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bb507be054aa7db8-0065bb719c-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:32.565 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:25:32.565 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:25:32.565 DEBUG:boto:Method: HEAD 2024-02-01T10:25:32.565 DEBUG:boto:Path: /vclfcn-19/ 2024-02-01T10:25:32.565 DEBUG:boto:Data: 2024-02-01T10:25:32.565 DEBUG:boto:Headers: {} 2024-02-01T10:25:32.565 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:25:32.565 DEBUG:boto:Port: 8000 2024-02-01T10:25:32.566 DEBUG:boto:Params: {} 2024-02-01T10:25:32.566 DEBUG:boto:Token: None 2024-02-01T10:25:32.566 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:25:32 GMT /vclfcn-19/ 2024-02-01T10:25:32.566 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:u/sC88oX23jSHOxuoaTPHRf0ujA= 2024-02-01T10:25:32.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:u/sC88oX23jSHOxuoaTPHRf0ujA=', 'Content-Length': '0'} 2024-02-01T10:25:32.571 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000991fde8f921c2c96-0065bb719c-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:32.571 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:25:32.571 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:25:32.571 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:25:32.571 DEBUG:boto:path=/vclfcn-19/?versions&max-keys=999 2024-02-01T10:25:32.571 DEBUG:boto:auth_path=/vclfcn-19/?versions&max-keys=999 2024-02-01T10:25:32.571 DEBUG:boto:Method: GET 2024-02-01T10:25:32.571 DEBUG:boto:Path: /vclfcn-19/?versions&max-keys=999 2024-02-01T10:25:32.572 DEBUG:boto:Data: 2024-02-01T10:25:32.572 DEBUG:boto:Headers: {} 2024-02-01T10:25:32.572 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:25:32.572 DEBUG:boto:Port: 8001 2024-02-01T10:25:32.572 DEBUG:boto:Params: {} 2024-02-01T10:25:32.572 DEBUG:boto:Token: None 2024-02-01T10:25:32.572 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:25:32 GMT /vclfcn-19/?versions 2024-02-01T10:25:32.572 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:v68kRCiLHp9JI4kcOxydo6+KQy8= 2024-02-01T10:25:32.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:v68kRCiLHp9JI4kcOxydo6+KQy8=', 'Content-Length': '0'} 2024-02-01T10:25:32.578 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083cac7ba5cd43c7d-0065bb719c-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:32.620 DEBUG:boto:b'vclfcn-19999falsedummyg5UScztMjPBIYZ2b5yOn-qsm.eQTcnutrue2024-02-01T10:23:11.115Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:25:32.621 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-02-01T10:25:32.621 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:25:32.621 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:25:32.621 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:25:32.622 DEBUG:boto:path=/vclfcn-19/?versions&max-keys=999 2024-02-01T10:25:32.622 DEBUG:boto:auth_path=/vclfcn-19/?versions&max-keys=999 2024-02-01T10:25:32.622 DEBUG:boto:Method: GET 2024-02-01T10:25:32.622 DEBUG:boto:Path: /vclfcn-19/?versions&max-keys=999 2024-02-01T10:25:32.622 DEBUG:boto:Data: 2024-02-01T10:25:32.622 DEBUG:boto:Headers: {} 2024-02-01T10:25:32.622 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:25:32.622 DEBUG:boto:Port: 8000 2024-02-01T10:25:32.622 DEBUG:boto:Params: {} 2024-02-01T10:25:32.622 DEBUG:boto:Token: None 2024-02-01T10:25:32.623 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:25:32 GMT /vclfcn-19/?versions 2024-02-01T10:25:32.623 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:v68kRCiLHp9JI4kcOxydo6+KQy8= 2024-02-01T10:25:32.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:v68kRCiLHp9JI4kcOxydo6+KQy8=', 'Content-Length': '0'} 2024-02-01T10:25:32.627 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9ffc53d85e037d7-0065bb719c-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:32.627 DEBUG:boto:b'vclfcn-19999falsedummyg5UScztMjPBIYZ2b5yOn-qsm.eQTcnutrue2024-02-01T10:23:11.115Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:25:32.628 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-02-01T10:25:32.628 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:25:32.628 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:25:32.628 DEBUG:boto:path=/vclfcn-19/?versions&max-keys=999 2024-02-01T10:25:32.628 DEBUG:boto:auth_path=/vclfcn-19/?versions&max-keys=999 2024-02-01T10:25:32.628 DEBUG:boto:Method: GET 2024-02-01T10:25:32.628 DEBUG:boto:Path: /vclfcn-19/?versions&max-keys=999 2024-02-01T10:25:32.629 DEBUG:boto:Data: 2024-02-01T10:25:32.629 DEBUG:boto:Headers: {} 2024-02-01T10:25:32.629 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:25:32.629 DEBUG:boto:Port: 8001 2024-02-01T10:25:32.629 DEBUG:boto:Params: {} 2024-02-01T10:25:32.629 DEBUG:boto:Token: None 2024-02-01T10:25:32.629 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:25:32 GMT /vclfcn-19/?versions 2024-02-01T10:25:32.629 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:v68kRCiLHp9JI4kcOxydo6+KQy8= 2024-02-01T10:25:32.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:v68kRCiLHp9JI4kcOxydo6+KQy8=', 'Content-Length': '0'} 2024-02-01T10:25:32.634 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025c58549fe4e0f6e-0065bb719c-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:32.634 DEBUG:boto:b'vclfcn-19999falsedummyg5UScztMjPBIYZ2b5yOn-qsm.eQTcnutrue2024-02-01T10:23:11.115Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:25:32.635 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:25:32.635 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:25:32.635 DEBUG:boto:path=/vclfcn-19/?versions&max-keys=999 2024-02-01T10:25:32.635 DEBUG:boto:auth_path=/vclfcn-19/?versions&max-keys=999 2024-02-01T10:25:32.635 DEBUG:boto:Method: GET 2024-02-01T10:25:32.635 DEBUG:boto:Path: /vclfcn-19/?versions&max-keys=999 2024-02-01T10:25:32.635 DEBUG:boto:Data: 2024-02-01T10:25:32.635 DEBUG:boto:Headers: {} 2024-02-01T10:25:32.635 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:25:32.635 DEBUG:boto:Port: 8000 2024-02-01T10:25:32.635 DEBUG:boto:Params: {} 2024-02-01T10:25:32.636 DEBUG:boto:Token: None 2024-02-01T10:25:32.636 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:25:32 GMT /vclfcn-19/?versions 2024-02-01T10:25:32.636 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:v68kRCiLHp9JI4kcOxydo6+KQy8= 2024-02-01T10:25:32.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:v68kRCiLHp9JI4kcOxydo6+KQy8=', 'Content-Length': '0'} 2024-02-01T10:25:32.640 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009dfbe2ba8fd7a8a6-0065bb719c-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:32.641 DEBUG:boto:b'vclfcn-19999falsedummyg5UScztMjPBIYZ2b5yOn-qsm.eQTcnutrue2024-02-01T10:23:11.115Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:25:32.641 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-02-01T10:25:32.641 DEBUG:boto:path=/vclfcn-19/dummy 2024-02-01T10:25:32.641 DEBUG:boto:auth_path=/vclfcn-19/dummy 2024-02-01T10:25:32.642 DEBUG:boto:path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.642 DEBUG:boto:auth_path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.642 DEBUG:boto:Method: GET 2024-02-01T10:25:32.642 DEBUG:boto:Path: /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.642 DEBUG:boto:Data: 2024-02-01T10:25:32.642 DEBUG:boto:Headers: {} 2024-02-01T10:25:32.642 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:25:32.642 DEBUG:boto:Port: 8001 2024-02-01T10:25:32.642 DEBUG:boto:Params: {} 2024-02-01T10:25:32.642 DEBUG:boto:Token: None 2024-02-01T10:25:32.643 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:25:32 GMT /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.643 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BNkvIt/FGtrI4r4HrUKRhPzDHNc= 2024-02-01T10:25:32.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BNkvIt/FGtrI4r4HrUKRhPzDHNc=', 'Content-Length': '0'} 2024-02-01T10:25:32.645 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:23:11 GMT'), ('x-amz-version-id', 'g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f0b8f0d338b55f0b-0065bb719c-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:32.646 DEBUG:boto:path=/vclfcn-19/dummy 2024-02-01T10:25:32.646 DEBUG:boto:auth_path=/vclfcn-19/dummy 2024-02-01T10:25:32.646 DEBUG:boto:path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.646 DEBUG:boto:auth_path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.646 DEBUG:boto:Method: GET 2024-02-01T10:25:32.646 DEBUG:boto:Path: /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.646 DEBUG:boto:Data: 2024-02-01T10:25:32.646 DEBUG:boto:Headers: {} 2024-02-01T10:25:32.646 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:25:32.647 DEBUG:boto:Port: 8000 2024-02-01T10:25:32.647 DEBUG:boto:Params: {} 2024-02-01T10:25:32.647 DEBUG:boto:Token: None 2024-02-01T10:25:32.647 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:25:32 GMT /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.647 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BNkvIt/FGtrI4r4HrUKRhPzDHNc= 2024-02-01T10:25:32.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BNkvIt/FGtrI4r4HrUKRhPzDHNc=', 'Content-Length': '0'} 2024-02-01T10:25:32.650 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:23:11 GMT'), ('x-amz-version-id', 'g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007d5d5a23821e0198-0065bb719c-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:32.650 DEBUG:boto:path=/vclfcn-19/dummy 2024-02-01T10:25:32.650 DEBUG:boto:auth_path=/vclfcn-19/dummy 2024-02-01T10:25:32.650 DEBUG:boto:path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.650 DEBUG:boto:auth_path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.651 DEBUG:boto:Method: HEAD 2024-02-01T10:25:32.651 DEBUG:boto:Path: /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.651 DEBUG:boto:Data: 2024-02-01T10:25:32.651 DEBUG:boto:Headers: {} 2024-02-01T10:25:32.651 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:25:32.651 DEBUG:boto:Port: 8001 2024-02-01T10:25:32.651 DEBUG:boto:Params: {} 2024-02-01T10:25:32.651 DEBUG:boto:Token: None 2024-02-01T10:25:32.651 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:25:32 GMT /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.651 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:N/mhheEvN4guBWoVT/9oaHPWQEM= 2024-02-01T10:25:32.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:N/mhheEvN4guBWoVT/9oaHPWQEM=', 'Content-Length': '0'} 2024-02-01T10:25:32.657 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:23:11 GMT'), ('x-amz-version-id', 'g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000787b23c2d6e6e331-0065bb719c-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:32.657 DEBUG:boto:path=/vclfcn-19/dummy 2024-02-01T10:25:32.657 DEBUG:boto:auth_path=/vclfcn-19/dummy 2024-02-01T10:25:32.657 DEBUG:boto:path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.657 DEBUG:boto:auth_path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.658 DEBUG:boto:Method: HEAD 2024-02-01T10:25:32.658 DEBUG:boto:Path: /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.658 DEBUG:boto:Data: 2024-02-01T10:25:32.658 DEBUG:boto:Headers: {} 2024-02-01T10:25:32.658 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:25:32.658 DEBUG:boto:Port: 8000 2024-02-01T10:25:32.658 DEBUG:boto:Params: {} 2024-02-01T10:25:32.658 DEBUG:boto:Token: None 2024-02-01T10:25:32.658 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:25:32 GMT /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.658 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:N/mhheEvN4guBWoVT/9oaHPWQEM= 2024-02-01T10:25:32.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:N/mhheEvN4guBWoVT/9oaHPWQEM=', 'Content-Length': '0'} 2024-02-01T10:25:32.661 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:23:11 GMT'), ('x-amz-version-id', 'g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000007591efe3a9f389b-0065bb719c-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:32.661 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-02-01T10:25:32.662 DEBUG:boto:path=/vclfcn-19/dummy 2024-02-01T10:25:32.662 DEBUG:boto:auth_path=/vclfcn-19/dummy 2024-02-01T10:25:32.662 DEBUG:boto:path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.662 DEBUG:boto:auth_path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.662 DEBUG:boto:Method: GET 2024-02-01T10:25:32.662 DEBUG:boto:Path: /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.662 DEBUG:boto:Data: 2024-02-01T10:25:32.662 DEBUG:boto:Headers: {} 2024-02-01T10:25:32.662 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:25:32.662 DEBUG:boto:Port: 8001 2024-02-01T10:25:32.662 DEBUG:boto:Params: {} 2024-02-01T10:25:32.663 DEBUG:boto:Token: None 2024-02-01T10:25:32.663 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:25:32 GMT /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.663 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BNkvIt/FGtrI4r4HrUKRhPzDHNc= 2024-02-01T10:25:32.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BNkvIt/FGtrI4r4HrUKRhPzDHNc=', 'Content-Length': '0'} 2024-02-01T10:25:32.666 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:23:11 GMT'), ('x-amz-version-id', 'g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000334affe1349f1398-0065bb719c-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:32.666 DEBUG:boto:path=/vclfcn-19/dummy 2024-02-01T10:25:32.666 DEBUG:boto:auth_path=/vclfcn-19/dummy 2024-02-01T10:25:32.666 DEBUG:boto:path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.667 DEBUG:boto:auth_path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.667 DEBUG:boto:Method: GET 2024-02-01T10:25:32.667 DEBUG:boto:Path: /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.667 DEBUG:boto:Data: 2024-02-01T10:25:32.667 DEBUG:boto:Headers: {} 2024-02-01T10:25:32.667 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:25:32.667 DEBUG:boto:Port: 8000 2024-02-01T10:25:32.667 DEBUG:boto:Params: {} 2024-02-01T10:25:32.667 DEBUG:boto:Token: None 2024-02-01T10:25:32.667 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:25:32 GMT /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.668 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BNkvIt/FGtrI4r4HrUKRhPzDHNc= 2024-02-01T10:25:32.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BNkvIt/FGtrI4r4HrUKRhPzDHNc=', 'Content-Length': '0'} 2024-02-01T10:25:32.670 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:23:11 GMT'), ('x-amz-version-id', 'g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002eba882bf59a93f2-0065bb719c-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:32.671 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:25:32.671 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:25:32.671 DEBUG:boto:path=/vclfcn-19/?prefix=dummy 2024-02-01T10:25:32.671 DEBUG:boto:auth_path=/vclfcn-19/?prefix=dummy 2024-02-01T10:25:32.671 DEBUG:boto:Method: GET 2024-02-01T10:25:32.671 DEBUG:boto:Path: /vclfcn-19/?prefix=dummy 2024-02-01T10:25:32.671 DEBUG:boto:Data: 2024-02-01T10:25:32.671 DEBUG:boto:Headers: {} 2024-02-01T10:25:32.672 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:25:32.672 DEBUG:boto:Port: 8001 2024-02-01T10:25:32.672 DEBUG:boto:Params: {} 2024-02-01T10:25:32.672 DEBUG:boto:Token: None 2024-02-01T10:25:32.672 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:25:32 GMT /vclfcn-19/ 2024-02-01T10:25:32.672 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jj8mGsJ1c2d2+Omz51wa5ANBXLg= 2024-02-01T10:25:32.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jj8mGsJ1c2d2+Omz51wa5ANBXLg=', 'Content-Length': '0'} 2024-02-01T10:25:32.677 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000419b7f1a88fe970d-0065bb719c-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:32.724 DEBUG:boto:b'vclfcn-19dummy1000falsedummy2024-02-01T10:23:11.115Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:25:32.725 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:25:32.725 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:25:32.725 DEBUG:boto:path=/vclfcn-19/?prefix=dummy 2024-02-01T10:25:32.725 DEBUG:boto:auth_path=/vclfcn-19/?prefix=dummy 2024-02-01T10:25:32.725 DEBUG:boto:Method: GET 2024-02-01T10:25:32.725 DEBUG:boto:Path: /vclfcn-19/?prefix=dummy 2024-02-01T10:25:32.725 DEBUG:boto:Data: 2024-02-01T10:25:32.725 DEBUG:boto:Headers: {} 2024-02-01T10:25:32.725 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:25:32.725 DEBUG:boto:Port: 8000 2024-02-01T10:25:32.725 DEBUG:boto:Params: {} 2024-02-01T10:25:32.726 DEBUG:boto:Token: None 2024-02-01T10:25:32.726 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:25:32 GMT /vclfcn-19/ 2024-02-01T10:25:32.726 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jj8mGsJ1c2d2+Omz51wa5ANBXLg= 2024-02-01T10:25:32.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jj8mGsJ1c2d2+Omz51wa5ANBXLg=', 'Content-Length': '0'} 2024-02-01T10:25:32.731 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e06f1c1abb836781-0065bb719c-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:32.731 DEBUG:boto:b'vclfcn-19dummy1000falsedummy2024-02-01T10:23:11.115Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:25:32.731 DEBUG:boto:path=/vclfcn-19/dummy 2024-02-01T10:25:32.731 DEBUG:boto:auth_path=/vclfcn-19/dummy 2024-02-01T10:25:32.732 DEBUG:boto:Method: HEAD 2024-02-01T10:25:32.732 DEBUG:boto:Path: /vclfcn-19/dummy 2024-02-01T10:25:32.732 DEBUG:boto:Data: 2024-02-01T10:25:32.732 DEBUG:boto:Headers: {} 2024-02-01T10:25:32.732 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:25:32.732 DEBUG:boto:Port: 8001 2024-02-01T10:25:32.732 DEBUG:boto:Params: {} 2024-02-01T10:25:32.732 DEBUG:boto:Token: None 2024-02-01T10:25:32.732 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:25:32 GMT /vclfcn-19/dummy 2024-02-01T10:25:32.733 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9Rrcc8ka5VLlQHUgXrrhDPHtKpc= 2024-02-01T10:25:32.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9Rrcc8ka5VLlQHUgXrrhDPHtKpc=', 'Content-Length': '0'} 2024-02-01T10:25:32.736 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:23:11 GMT'), ('x-amz-version-id', 'g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a24038a943918f77-0065bb719c-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:32.736 DEBUG:boto:path=/vclfcn-19/dummy 2024-02-01T10:25:32.737 DEBUG:boto:auth_path=/vclfcn-19/dummy 2024-02-01T10:25:32.737 DEBUG:boto:Method: HEAD 2024-02-01T10:25:32.737 DEBUG:boto:Path: /vclfcn-19/dummy 2024-02-01T10:25:32.737 DEBUG:boto:Data: 2024-02-01T10:25:32.737 DEBUG:boto:Headers: {} 2024-02-01T10:25:32.737 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:25:32.737 DEBUG:boto:Port: 8000 2024-02-01T10:25:32.737 DEBUG:boto:Params: {} 2024-02-01T10:25:32.737 DEBUG:boto:Token: None 2024-02-01T10:25:32.737 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:25:32 GMT /vclfcn-19/dummy 2024-02-01T10:25:32.738 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9Rrcc8ka5VLlQHUgXrrhDPHtKpc= 2024-02-01T10:25:32.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9Rrcc8ka5VLlQHUgXrrhDPHtKpc=', 'Content-Length': '0'} 2024-02-01T10:25:32.741 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:23:11 GMT'), ('x-amz-version-id', 'g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000421b5ae99b6f0e66-0065bb719c-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:32.741 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-02-01T10:25:32.742 DEBUG:boto:path=/vclfcn-19/dummy 2024-02-01T10:25:32.742 DEBUG:boto:auth_path=/vclfcn-19/dummy 2024-02-01T10:25:32.742 DEBUG:boto:path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.742 DEBUG:boto:auth_path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.742 DEBUG:boto:Method: GET 2024-02-01T10:25:32.742 DEBUG:boto:Path: /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.742 DEBUG:boto:Data: 2024-02-01T10:25:32.742 DEBUG:boto:Headers: {} 2024-02-01T10:25:32.742 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:25:32.742 DEBUG:boto:Port: 8001 2024-02-01T10:25:32.742 DEBUG:boto:Params: {} 2024-02-01T10:25:32.743 DEBUG:boto:Token: None 2024-02-01T10:25:32.743 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:25:32 GMT /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.743 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BNkvIt/FGtrI4r4HrUKRhPzDHNc= 2024-02-01T10:25:32.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BNkvIt/FGtrI4r4HrUKRhPzDHNc=', 'Content-Length': '0'} 2024-02-01T10:25:32.745 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:23:11 GMT'), ('x-amz-version-id', 'g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000af86b1bb29e4d249-0065bb719c-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:32.746 DEBUG:boto:path=/vclfcn-19/dummy 2024-02-01T10:25:32.746 DEBUG:boto:auth_path=/vclfcn-19/dummy 2024-02-01T10:25:32.746 DEBUG:boto:path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.746 DEBUG:boto:auth_path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.746 DEBUG:boto:Method: GET 2024-02-01T10:25:32.746 DEBUG:boto:Path: /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.746 DEBUG:boto:Data: 2024-02-01T10:25:32.746 DEBUG:boto:Headers: {} 2024-02-01T10:25:32.746 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:25:32.746 DEBUG:boto:Port: 8000 2024-02-01T10:25:32.746 DEBUG:boto:Params: {} 2024-02-01T10:25:32.747 DEBUG:boto:Token: None 2024-02-01T10:25:32.747 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:25:32 GMT /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:25:32.747 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BNkvIt/FGtrI4r4HrUKRhPzDHNc= 2024-02-01T10:25:32.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BNkvIt/FGtrI4r4HrUKRhPzDHNc=', 'Content-Length': '0'} 2024-02-01T10:25:32.750 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:23:11 GMT'), ('x-amz-version-id', 'g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b3b59f28e2f4c260-0065bb719c-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:32.750 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:25:32.750 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:25:32.750 DEBUG:boto:path=/vclfcn-19/?prefix=dummy 2024-02-01T10:25:32.750 DEBUG:boto:auth_path=/vclfcn-19/?prefix=dummy 2024-02-01T10:25:32.750 DEBUG:boto:Method: GET 2024-02-01T10:25:32.751 DEBUG:boto:Path: /vclfcn-19/?prefix=dummy 2024-02-01T10:25:32.751 DEBUG:boto:Data: 2024-02-01T10:25:32.751 DEBUG:boto:Headers: {} 2024-02-01T10:25:32.751 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:25:32.751 DEBUG:boto:Port: 8001 2024-02-01T10:25:32.751 DEBUG:boto:Params: {} 2024-02-01T10:25:32.751 DEBUG:boto:Token: None 2024-02-01T10:25:32.751 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:25:32 GMT /vclfcn-19/ 2024-02-01T10:25:32.751 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jj8mGsJ1c2d2+Omz51wa5ANBXLg= 2024-02-01T10:25:32.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jj8mGsJ1c2d2+Omz51wa5ANBXLg=', 'Content-Length': '0'} 2024-02-01T10:25:32.756 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024b33e68d4c02ca5-0065bb719c-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:32.800 DEBUG:boto:b'vclfcn-19dummy1000falsedummy2024-02-01T10:23:11.115Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:25:32.801 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:25:32.801 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:25:32.801 DEBUG:boto:path=/vclfcn-19/?prefix=dummy 2024-02-01T10:25:32.801 DEBUG:boto:auth_path=/vclfcn-19/?prefix=dummy 2024-02-01T10:25:32.801 DEBUG:boto:Method: GET 2024-02-01T10:25:32.801 DEBUG:boto:Path: /vclfcn-19/?prefix=dummy 2024-02-01T10:25:32.801 DEBUG:boto:Data: 2024-02-01T10:25:32.801 DEBUG:boto:Headers: {} 2024-02-01T10:25:32.801 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:25:32.801 DEBUG:boto:Port: 8000 2024-02-01T10:25:32.801 DEBUG:boto:Params: {} 2024-02-01T10:25:32.802 DEBUG:boto:Token: None 2024-02-01T10:25:32.802 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:25:32 GMT /vclfcn-19/ 2024-02-01T10:25:32.802 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jj8mGsJ1c2d2+Omz51wa5ANBXLg= 2024-02-01T10:25:32.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jj8mGsJ1c2d2+Omz51wa5ANBXLg=', 'Content-Length': '0'} 2024-02-01T10:25:32.806 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c036a6d1874939a-0065bb719c-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:32.852 DEBUG:boto:b'vclfcn-19dummy1000falsedummy2024-02-01T10:23:11.115Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:25:32.852 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-19 zones={test-zone3, test-zone2} 2024-02-01T10:25:32.853 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:25:32.853 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:25:32.853 DEBUG:boto:Method: HEAD 2024-02-01T10:25:32.853 DEBUG:boto:Path: /vclfcn-19/ 2024-02-01T10:25:32.853 DEBUG:boto:Data: 2024-02-01T10:25:32.853 DEBUG:boto:Headers: {} 2024-02-01T10:25:32.853 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:25:32.853 DEBUG:boto:Port: 8000 2024-02-01T10:25:32.854 DEBUG:boto:Params: {} 2024-02-01T10:25:32.854 DEBUG:boto:Token: None 2024-02-01T10:25:32.854 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:25:32 GMT /vclfcn-19/ 2024-02-01T10:25:32.854 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:u/sC88oX23jSHOxuoaTPHRf0ujA= 2024-02-01T10:25:32.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:u/sC88oX23jSHOxuoaTPHRf0ujA=', 'Content-Length': '0'} 2024-02-01T10:25:32.859 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004d4f5460711d1b4a-0065bb719c-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:32.860 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:25:32.860 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:25:32.860 DEBUG:boto:Method: PUT 2024-02-01T10:25:32.860 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:25:32.860 DEBUG:boto:Data: 2024-02-01T10:25:32.860 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-02-01T10:25:32.860 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:25:32.860 DEBUG:boto:Port: 8000 2024-02-01T10:25:32.860 DEBUG:boto:Params: {} 2024-02-01T10:25:32.860 DEBUG:boto:Token: None 2024-02-01T10:25:32.861 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 01 Feb 2024 10:25:32 GMT /vclfcn-19/obj 2024-02-01T10:25:32.861 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8n4NV+FC92CUyVSphS7yYOyvo/4= 2024-02-01T10:25:32.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:25:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8n4NV+FC92CUyVSphS7yYOyvo/4='} 2024-02-01T10:25:32.908 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'B5jd89.SLRVLOAPcXqackukLvQnZY1B'), ('x-amz-request-id', 'tx000008de4babb0fae0824-0065bb719c-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:32.909 DEBUG:rgw_multi.tests:zone=test-zone1 version=B5jd89.SLRVLOAPcXqackukLvQnZY1B 2024-02-01T10:25:32.909 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:25:32.909 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:25:32.910 DEBUG:boto:Method: HEAD 2024-02-01T10:25:32.910 DEBUG:boto:Path: /vclfcn-19/ 2024-02-01T10:25:32.910 DEBUG:boto:Data: 2024-02-01T10:25:32.910 DEBUG:boto:Headers: {} 2024-02-01T10:25:32.910 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:25:32.910 DEBUG:boto:Port: 8000 2024-02-01T10:25:32.911 DEBUG:boto:Params: {} 2024-02-01T10:25:32.911 DEBUG:boto:Token: None 2024-02-01T10:25:32.911 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:25:32 GMT /vclfcn-19/ 2024-02-01T10:25:32.911 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:u/sC88oX23jSHOxuoaTPHRf0ujA= 2024-02-01T10:25:32.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:u/sC88oX23jSHOxuoaTPHRf0ujA=', 'Content-Length': '0'} 2024-02-01T10:25:32.916 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006674c2bc5d682fad-0065bb719c-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:32.916 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:25:32.916 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:25:32.917 DEBUG:boto:Method: PUT 2024-02-01T10:25:32.917 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:25:32.917 DEBUG:boto:Data: 2024-02-01T10:25:32.917 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-02-01T10:25:32.917 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:25:32.918 DEBUG:boto:Port: 8000 2024-02-01T10:25:32.918 DEBUG:boto:Params: {} 2024-02-01T10:25:32.918 DEBUG:boto:Token: None 2024-02-01T10:25:32.918 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 01 Feb 2024 10:25:32 GMT /vclfcn-19/obj 2024-02-01T10:25:32.918 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8n4NV+FC92CUyVSphS7yYOyvo/4= 2024-02-01T10:25:32.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:25:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8n4NV+FC92CUyVSphS7yYOyvo/4='} 2024-02-01T10:25:32.943 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 's-qxjv7TkMdxeC61HL2kxUqODSwa-1I'), ('x-amz-request-id', 'tx00000b5cf0222fbf14ecb-0065bb719c-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:32.944 DEBUG:rgw_multi.tests:zone=test-zone2 version=s-qxjv7TkMdxeC61HL2kxUqODSwa-1I 2024-02-01T10:25:32.944 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:25:32.944 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:25:32.944 DEBUG:boto:Method: HEAD 2024-02-01T10:25:32.944 DEBUG:boto:Path: /vclfcn-19/ 2024-02-01T10:25:32.945 DEBUG:boto:Data: 2024-02-01T10:25:32.945 DEBUG:boto:Headers: {} 2024-02-01T10:25:32.945 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:25:32.945 DEBUG:boto:Port: 8001 2024-02-01T10:25:32.945 DEBUG:boto:Params: {} 2024-02-01T10:25:32.946 DEBUG:boto:Token: None 2024-02-01T10:25:32.946 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:25:32 GMT /vclfcn-19/ 2024-02-01T10:25:32.946 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:u/sC88oX23jSHOxuoaTPHRf0ujA= 2024-02-01T10:25:32.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:u/sC88oX23jSHOxuoaTPHRf0ujA=', 'Content-Length': '0'} 2024-02-01T10:25:32.951 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003fca6c0105c8c1aa-0065bb719c-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:32.952 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:25:32.952 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:25:32.952 DEBUG:boto:Method: PUT 2024-02-01T10:25:32.952 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:25:32.952 DEBUG:boto:Data: 2024-02-01T10:25:32.952 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-02-01T10:25:32.953 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:25:32.953 DEBUG:boto:Port: 8001 2024-02-01T10:25:32.953 DEBUG:boto:Params: {} 2024-02-01T10:25:32.953 DEBUG:boto:Token: None 2024-02-01T10:25:32.953 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 01 Feb 2024 10:25:32 GMT /vclfcn-19/obj 2024-02-01T10:25:32.954 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8n4NV+FC92CUyVSphS7yYOyvo/4= 2024-02-01T10:25:32.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:25:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8n4NV+FC92CUyVSphS7yYOyvo/4='} 2024-02-01T10:25:32.979 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ'), ('x-amz-request-id', 'tx000002fbd75aa3ffab567-0065bb719c-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:32.979 DEBUG:rgw_multi.tests:zone=test-zone3 version=rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ 2024-02-01T10:25:32.979 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:25:32.979 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:25:32.979 DEBUG:boto:Method: HEAD 2024-02-01T10:25:32.980 DEBUG:boto:Path: /vclfcn-19/ 2024-02-01T10:25:32.980 DEBUG:boto:Data: 2024-02-01T10:25:32.980 DEBUG:boto:Headers: {} 2024-02-01T10:25:32.980 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:25:32.980 DEBUG:boto:Port: 8000 2024-02-01T10:25:32.980 DEBUG:boto:Params: {} 2024-02-01T10:25:32.980 DEBUG:boto:Token: None 2024-02-01T10:25:32.980 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:25:32 GMT /vclfcn-19/ 2024-02-01T10:25:32.980 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:u/sC88oX23jSHOxuoaTPHRf0ujA= 2024-02-01T10:25:32.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:u/sC88oX23jSHOxuoaTPHRf0ujA=', 'Content-Length': '0'} 2024-02-01T10:25:32.985 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '8'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fd6ec9b3f341401f-0065bb719c-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:32.985 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:25:32.986 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:25:32.986 DEBUG:boto:Method: PUT 2024-02-01T10:25:32.986 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:25:32.986 DEBUG:boto:Data: 2024-02-01T10:25:32.986 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-02-01T10:25:32.986 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:25:32.986 DEBUG:boto:Port: 8000 2024-02-01T10:25:32.987 DEBUG:boto:Params: {} 2024-02-01T10:25:32.987 DEBUG:boto:Token: None 2024-02-01T10:25:32.987 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 01 Feb 2024 10:25:32 GMT /vclfcn-19/obj 2024-02-01T10:25:32.987 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8n4NV+FC92CUyVSphS7yYOyvo/4= 2024-02-01T10:25:32.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:25:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8n4NV+FC92CUyVSphS7yYOyvo/4='} 2024-02-01T10:25:33.014 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o'), ('x-amz-request-id', 'tx0000064f14793a560daeb-0065bb719c-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:33.015 DEBUG:rgw_multi.tests:zone=test-zone1 version=YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o 2024-02-01T10:25:33.015 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:25:33.015 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:25:33.015 DEBUG:boto:Method: HEAD 2024-02-01T10:25:33.015 DEBUG:boto:Path: /vclfcn-19/ 2024-02-01T10:25:33.016 DEBUG:boto:Data: 2024-02-01T10:25:33.016 DEBUG:boto:Headers: {} 2024-02-01T10:25:33.016 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:25:33.016 DEBUG:boto:Port: 8000 2024-02-01T10:25:33.016 DEBUG:boto:Params: {} 2024-02-01T10:25:33.017 DEBUG:boto:Token: None 2024-02-01T10:25:33.017 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:25:33 GMT /vclfcn-19/ 2024-02-01T10:25:33.017 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nbWj6wHMj0SXykDmBsWDO7WhJHk= 2024-02-01T10:25:33.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nbWj6wHMj0SXykDmBsWDO7WhJHk=', 'Content-Length': '0'} 2024-02-01T10:25:33.021 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '8'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000f438dc57b135ce5-0065bb719d-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:33.021 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:25:33.022 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:25:33.022 DEBUG:boto:Method: PUT 2024-02-01T10:25:33.022 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:25:33.022 DEBUG:boto:Data: 2024-02-01T10:25:33.022 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-02-01T10:25:33.022 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:25:33.022 DEBUG:boto:Port: 8000 2024-02-01T10:25:33.022 DEBUG:boto:Params: {} 2024-02-01T10:25:33.022 DEBUG:boto:Token: None 2024-02-01T10:25:33.022 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 01 Feb 2024 10:25:33 GMT /vclfcn-19/obj 2024-02-01T10:25:33.023 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jQQhAdXnykB1GGpAxsAZJJ2qrek= 2024-02-01T10:25:33.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:25:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jQQhAdXnykB1GGpAxsAZJJ2qrek='} 2024-02-01T10:25:33.044 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX'), ('x-amz-request-id', 'tx000008b265de3db252a2e-0065bb719d-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:33.044 DEBUG:rgw_multi.tests:zone=test-zone2 version=.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX 2024-02-01T10:25:33.044 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:25:33.044 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:25:33.044 DEBUG:boto:Method: HEAD 2024-02-01T10:25:33.044 DEBUG:boto:Path: /vclfcn-19/ 2024-02-01T10:25:33.044 DEBUG:boto:Data: 2024-02-01T10:25:33.044 DEBUG:boto:Headers: {} 2024-02-01T10:25:33.045 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:25:33.045 DEBUG:boto:Port: 8001 2024-02-01T10:25:33.045 DEBUG:boto:Params: {} 2024-02-01T10:25:33.045 DEBUG:boto:Token: None 2024-02-01T10:25:33.045 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:25:33 GMT /vclfcn-19/ 2024-02-01T10:25:33.045 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nbWj6wHMj0SXykDmBsWDO7WhJHk= 2024-02-01T10:25:33.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nbWj6wHMj0SXykDmBsWDO7WhJHk=', 'Content-Length': '0'} 2024-02-01T10:25:33.050 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '8'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a18d2416ec46a4e6-0065bb719d-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:33.050 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:25:33.051 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:25:33.051 DEBUG:boto:Method: PUT 2024-02-01T10:25:33.051 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:25:33.051 DEBUG:boto:Data: 2024-02-01T10:25:33.051 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-02-01T10:25:33.051 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:25:33.051 DEBUG:boto:Port: 8001 2024-02-01T10:25:33.051 DEBUG:boto:Params: {} 2024-02-01T10:25:33.051 DEBUG:boto:Token: None 2024-02-01T10:25:33.051 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 01 Feb 2024 10:25:33 GMT /vclfcn-19/obj 2024-02-01T10:25:33.052 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jQQhAdXnykB1GGpAxsAZJJ2qrek= 2024-02-01T10:25:33.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:25:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jQQhAdXnykB1GGpAxsAZJJ2qrek='} 2024-02-01T10:25:33.074 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '63CC4agggciY3jdYETzlNEHLx15uh8w'), ('x-amz-request-id', 'tx00000e4f8d29203b15134-0065bb719d-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:33.074 DEBUG:rgw_multi.tests:zone=test-zone3 version=63CC4agggciY3jdYETzlNEHLx15uh8w 2024-02-01T10:25:33.074 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:25:33.074 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:25:33.074 DEBUG:boto:Method: HEAD 2024-02-01T10:25:33.074 DEBUG:boto:Path: /vclfcn-19/ 2024-02-01T10:25:33.074 DEBUG:boto:Data: 2024-02-01T10:25:33.074 DEBUG:boto:Headers: {} 2024-02-01T10:25:33.075 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:25:33.075 DEBUG:boto:Port: 8000 2024-02-01T10:25:33.075 DEBUG:boto:Params: {} 2024-02-01T10:25:33.075 DEBUG:boto:Token: None 2024-02-01T10:25:33.075 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:25:33 GMT /vclfcn-19/ 2024-02-01T10:25:33.075 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nbWj6wHMj0SXykDmBsWDO7WhJHk= 2024-02-01T10:25:33.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nbWj6wHMj0SXykDmBsWDO7WhJHk=', 'Content-Length': '0'} 2024-02-01T10:25:33.080 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b9c0bbead1e25ea7-0065bb719d-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:33.080 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:25:33.080 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:25:33.081 DEBUG:boto:Method: PUT 2024-02-01T10:25:33.081 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:25:33.081 DEBUG:boto:Data: 2024-02-01T10:25:33.081 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-02-01T10:25:33.081 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:25:33.081 DEBUG:boto:Port: 8000 2024-02-01T10:25:33.081 DEBUG:boto:Params: {} 2024-02-01T10:25:33.081 DEBUG:boto:Token: None 2024-02-01T10:25:33.081 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 01 Feb 2024 10:25:33 GMT /vclfcn-19/obj 2024-02-01T10:25:33.081 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jQQhAdXnykB1GGpAxsAZJJ2qrek= 2024-02-01T10:25:33.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:25:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jQQhAdXnykB1GGpAxsAZJJ2qrek='} 2024-02-01T10:25:33.103 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E'), ('x-amz-request-id', 'tx0000002b4c66949c959c6-0065bb719d-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:33.104 DEBUG:rgw_multi.tests:zone=test-zone1 version=2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E 2024-02-01T10:25:33.104 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:25:33.104 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:25:33.104 DEBUG:boto:Method: HEAD 2024-02-01T10:25:33.104 DEBUG:boto:Path: /vclfcn-19/ 2024-02-01T10:25:33.104 DEBUG:boto:Data: 2024-02-01T10:25:33.104 DEBUG:boto:Headers: {} 2024-02-01T10:25:33.104 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:25:33.104 DEBUG:boto:Port: 8000 2024-02-01T10:25:33.104 DEBUG:boto:Params: {} 2024-02-01T10:25:33.105 DEBUG:boto:Token: None 2024-02-01T10:25:33.105 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:25:33 GMT /vclfcn-19/ 2024-02-01T10:25:33.105 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nbWj6wHMj0SXykDmBsWDO7WhJHk= 2024-02-01T10:25:33.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nbWj6wHMj0SXykDmBsWDO7WhJHk=', 'Content-Length': '0'} 2024-02-01T10:25:33.109 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009c80a91813bad6a8-0065bb719d-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:33.109 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:25:33.109 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:25:33.110 DEBUG:boto:Method: PUT 2024-02-01T10:25:33.110 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:25:33.110 DEBUG:boto:Data: 2024-02-01T10:25:33.110 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-02-01T10:25:33.110 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:25:33.110 DEBUG:boto:Port: 8000 2024-02-01T10:25:33.110 DEBUG:boto:Params: {} 2024-02-01T10:25:33.110 DEBUG:boto:Token: None 2024-02-01T10:25:33.110 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 01 Feb 2024 10:25:33 GMT /vclfcn-19/obj 2024-02-01T10:25:33.110 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jQQhAdXnykB1GGpAxsAZJJ2qrek= 2024-02-01T10:25:33.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:25:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jQQhAdXnykB1GGpAxsAZJJ2qrek='} 2024-02-01T10:25:33.132 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '0KR6m6ymTGitKe6vabQyVENAnBL669w'), ('x-amz-request-id', 'tx00000621e180e4303caee-0065bb719d-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:33.132 DEBUG:rgw_multi.tests:zone=test-zone2 version=0KR6m6ymTGitKe6vabQyVENAnBL669w 2024-02-01T10:25:33.132 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:25:33.132 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:25:33.132 DEBUG:boto:Method: HEAD 2024-02-01T10:25:33.133 DEBUG:boto:Path: /vclfcn-19/ 2024-02-01T10:25:33.133 DEBUG:boto:Data: 2024-02-01T10:25:33.133 DEBUG:boto:Headers: {} 2024-02-01T10:25:33.133 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:25:33.133 DEBUG:boto:Port: 8001 2024-02-01T10:25:33.133 DEBUG:boto:Params: {} 2024-02-01T10:25:33.133 DEBUG:boto:Token: None 2024-02-01T10:25:33.133 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:25:33 GMT /vclfcn-19/ 2024-02-01T10:25:33.133 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nbWj6wHMj0SXykDmBsWDO7WhJHk= 2024-02-01T10:25:33.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nbWj6wHMj0SXykDmBsWDO7WhJHk=', 'Content-Length': '0'} 2024-02-01T10:25:33.138 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009b94c44c99769063-0065bb719d-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:33.138 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:25:33.138 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:25:33.138 DEBUG:boto:Method: PUT 2024-02-01T10:25:33.138 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:25:33.138 DEBUG:boto:Data: 2024-02-01T10:25:33.139 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-02-01T10:25:33.139 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:25:33.139 DEBUG:boto:Port: 8001 2024-02-01T10:25:33.139 DEBUG:boto:Params: {} 2024-02-01T10:25:33.139 DEBUG:boto:Token: None 2024-02-01T10:25:33.139 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 01 Feb 2024 10:25:33 GMT /vclfcn-19/obj 2024-02-01T10:25:33.139 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jQQhAdXnykB1GGpAxsAZJJ2qrek= 2024-02-01T10:25:33.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:25:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jQQhAdXnykB1GGpAxsAZJJ2qrek='} 2024-02-01T10:25:33.188 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6'), ('x-amz-request-id', 'tx00000f6d0b67a25b93e37-0065bb719d-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:33.188 DEBUG:rgw_multi.tests:zone=test-zone3 version=7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6 2024-02-01T10:25:33.189 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:25:33.189 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:25:33.189 DEBUG:boto:Method: HEAD 2024-02-01T10:25:33.189 DEBUG:boto:Path: /vclfcn-19/ 2024-02-01T10:25:33.189 DEBUG:boto:Data: 2024-02-01T10:25:33.189 DEBUG:boto:Headers: {} 2024-02-01T10:25:33.189 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:25:33.189 DEBUG:boto:Port: 8000 2024-02-01T10:25:33.189 DEBUG:boto:Params: {} 2024-02-01T10:25:33.189 DEBUG:boto:Token: None 2024-02-01T10:25:33.190 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:25:33 GMT /vclfcn-19/ 2024-02-01T10:25:33.190 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nbWj6wHMj0SXykDmBsWDO7WhJHk= 2024-02-01T10:25:33.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nbWj6wHMj0SXykDmBsWDO7WhJHk=', 'Content-Length': '0'} 2024-02-01T10:25:33.194 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000008b0fe52154dc444-0065bb719d-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:33.195 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:25:33.195 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:25:33.195 DEBUG:boto:Method: PUT 2024-02-01T10:25:33.195 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:25:33.195 DEBUG:boto:Data: 2024-02-01T10:25:33.195 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-02-01T10:25:33.195 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:25:33.195 DEBUG:boto:Port: 8000 2024-02-01T10:25:33.195 DEBUG:boto:Params: {} 2024-02-01T10:25:33.196 DEBUG:boto:Token: None 2024-02-01T10:25:33.196 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 01 Feb 2024 10:25:33 GMT /vclfcn-19/obj 2024-02-01T10:25:33.196 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jQQhAdXnykB1GGpAxsAZJJ2qrek= 2024-02-01T10:25:33.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:25:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jQQhAdXnykB1GGpAxsAZJJ2qrek='} 2024-02-01T10:25:33.240 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI'), ('x-amz-request-id', 'tx00000d7f9456e7c990b76-0065bb719d-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:33.241 DEBUG:rgw_multi.tests:zone=test-zone1 version=cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI 2024-02-01T10:25:33.241 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:25:33.241 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:25:33.242 DEBUG:boto:Method: HEAD 2024-02-01T10:25:33.242 DEBUG:boto:Path: /vclfcn-19/ 2024-02-01T10:25:33.242 DEBUG:boto:Data: 2024-02-01T10:25:33.242 DEBUG:boto:Headers: {} 2024-02-01T10:25:33.242 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:25:33.242 DEBUG:boto:Port: 8000 2024-02-01T10:25:33.242 DEBUG:boto:Params: {} 2024-02-01T10:25:33.243 DEBUG:boto:Token: None 2024-02-01T10:25:33.243 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:25:33 GMT /vclfcn-19/ 2024-02-01T10:25:33.243 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nbWj6wHMj0SXykDmBsWDO7WhJHk= 2024-02-01T10:25:33.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nbWj6wHMj0SXykDmBsWDO7WhJHk=', 'Content-Length': '0'} 2024-02-01T10:25:33.247 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000017fb4533830bb5a2-0065bb719d-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:33.248 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:25:33.248 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:25:33.248 DEBUG:boto:Method: PUT 2024-02-01T10:25:33.248 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:25:33.248 DEBUG:boto:Data: 2024-02-01T10:25:33.248 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-02-01T10:25:33.248 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:25:33.249 DEBUG:boto:Port: 8000 2024-02-01T10:25:33.249 DEBUG:boto:Params: {} 2024-02-01T10:25:33.249 DEBUG:boto:Token: None 2024-02-01T10:25:33.249 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 01 Feb 2024 10:25:33 GMT /vclfcn-19/obj 2024-02-01T10:25:33.249 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jQQhAdXnykB1GGpAxsAZJJ2qrek= 2024-02-01T10:25:33.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:25:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jQQhAdXnykB1GGpAxsAZJJ2qrek='} 2024-02-01T10:25:33.272 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE'), ('x-amz-request-id', 'tx00000f8ff02e8f837e3e4-0065bb719d-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:33.272 DEBUG:rgw_multi.tests:zone=test-zone2 version=y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE 2024-02-01T10:25:33.273 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:25:33.273 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:25:33.273 DEBUG:boto:Method: HEAD 2024-02-01T10:25:33.273 DEBUG:boto:Path: /vclfcn-19/ 2024-02-01T10:25:33.273 DEBUG:boto:Data: 2024-02-01T10:25:33.273 DEBUG:boto:Headers: {} 2024-02-01T10:25:33.273 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:25:33.273 DEBUG:boto:Port: 8001 2024-02-01T10:25:33.273 DEBUG:boto:Params: {} 2024-02-01T10:25:33.274 DEBUG:boto:Token: None 2024-02-01T10:25:33.274 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:25:33 GMT /vclfcn-19/ 2024-02-01T10:25:33.274 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nbWj6wHMj0SXykDmBsWDO7WhJHk= 2024-02-01T10:25:33.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nbWj6wHMj0SXykDmBsWDO7WhJHk=', 'Content-Length': '0'} 2024-02-01T10:25:33.278 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003bd6ab6c25c6d38a-0065bb719d-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:33.279 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:25:33.279 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:25:33.279 DEBUG:boto:Method: PUT 2024-02-01T10:25:33.279 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:25:33.279 DEBUG:boto:Data: 2024-02-01T10:25:33.279 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-02-01T10:25:33.279 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:25:33.279 DEBUG:boto:Port: 8001 2024-02-01T10:25:33.279 DEBUG:boto:Params: {} 2024-02-01T10:25:33.279 DEBUG:boto:Token: None 2024-02-01T10:25:33.280 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 01 Feb 2024 10:25:33 GMT /vclfcn-19/obj 2024-02-01T10:25:33.280 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jQQhAdXnykB1GGpAxsAZJJ2qrek= 2024-02-01T10:25:33.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:25:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jQQhAdXnykB1GGpAxsAZJJ2qrek='} 2024-02-01T10:25:33.303 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'rasya8taOZsGTj9rDi7keghf.VnPFlF'), ('x-amz-request-id', 'tx00000772a14407a889e9e-0065bb719d-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:33.303 DEBUG:rgw_multi.tests:zone=test-zone3 version=rasya8taOZsGTj9rDi7keghf.VnPFlF 2024-02-01T10:25:33.303 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:25:33.303 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:25:33.304 DEBUG:boto:Method: HEAD 2024-02-01T10:25:33.304 DEBUG:boto:Path: /vclfcn-19/ 2024-02-01T10:25:33.304 DEBUG:boto:Data: 2024-02-01T10:25:33.304 DEBUG:boto:Headers: {} 2024-02-01T10:25:33.304 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:25:33.304 DEBUG:boto:Port: 8000 2024-02-01T10:25:33.304 DEBUG:boto:Params: {} 2024-02-01T10:25:33.305 DEBUG:boto:Token: None 2024-02-01T10:25:33.305 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:25:33 GMT /vclfcn-19/ 2024-02-01T10:25:33.305 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nbWj6wHMj0SXykDmBsWDO7WhJHk= 2024-02-01T10:25:33.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nbWj6wHMj0SXykDmBsWDO7WhJHk=', 'Content-Length': '0'} 2024-02-01T10:25:33.310 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '32'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009e19370b1a773989-0065bb719d-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:33.311 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:25:33.311 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:25:33.311 DEBUG:boto:Method: PUT 2024-02-01T10:25:33.311 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:25:33.311 DEBUG:boto:Data: 2024-02-01T10:25:33.311 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-02-01T10:25:33.311 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:25:33.312 DEBUG:boto:Port: 8000 2024-02-01T10:25:33.312 DEBUG:boto:Params: {} 2024-02-01T10:25:33.312 DEBUG:boto:Token: None 2024-02-01T10:25:33.312 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 01 Feb 2024 10:25:33 GMT /vclfcn-19/obj 2024-02-01T10:25:33.312 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jQQhAdXnykB1GGpAxsAZJJ2qrek= 2024-02-01T10:25:33.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:25:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jQQhAdXnykB1GGpAxsAZJJ2qrek='} 2024-02-01T10:25:33.334 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'iGO2QyMOL81IaF-H4sHDqyECPlnZBmE'), ('x-amz-request-id', 'tx00000272cd55dc4253101-0065bb719d-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:33.334 DEBUG:rgw_multi.tests:zone=test-zone1 version=iGO2QyMOL81IaF-H4sHDqyECPlnZBmE 2024-02-01T10:25:33.334 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:25:33.334 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:25:33.334 DEBUG:boto:Method: HEAD 2024-02-01T10:25:33.335 DEBUG:boto:Path: /vclfcn-19/ 2024-02-01T10:25:33.335 DEBUG:boto:Data: 2024-02-01T10:25:33.335 DEBUG:boto:Headers: {} 2024-02-01T10:25:33.335 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:25:33.335 DEBUG:boto:Port: 8000 2024-02-01T10:25:33.335 DEBUG:boto:Params: {} 2024-02-01T10:25:33.335 DEBUG:boto:Token: None 2024-02-01T10:25:33.335 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:25:33 GMT /vclfcn-19/ 2024-02-01T10:25:33.335 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nbWj6wHMj0SXykDmBsWDO7WhJHk= 2024-02-01T10:25:33.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nbWj6wHMj0SXykDmBsWDO7WhJHk=', 'Content-Length': '0'} 2024-02-01T10:25:33.340 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '32'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cb600668ed0cf673-0065bb719d-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:33.340 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:25:33.340 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:25:33.340 DEBUG:boto:Method: PUT 2024-02-01T10:25:33.340 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:25:33.341 DEBUG:boto:Data: 2024-02-01T10:25:33.341 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-02-01T10:25:33.341 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:25:33.341 DEBUG:boto:Port: 8000 2024-02-01T10:25:33.341 DEBUG:boto:Params: {} 2024-02-01T10:25:33.341 DEBUG:boto:Token: None 2024-02-01T10:25:33.341 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 01 Feb 2024 10:25:33 GMT /vclfcn-19/obj 2024-02-01T10:25:33.341 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jQQhAdXnykB1GGpAxsAZJJ2qrek= 2024-02-01T10:25:33.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:25:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jQQhAdXnykB1GGpAxsAZJJ2qrek='} 2024-02-01T10:25:33.362 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '9CJ3w1EusK8OST2k17sTdTM.cuh3E3e'), ('x-amz-request-id', 'tx0000012b796415af54848-0065bb719d-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:33.362 DEBUG:rgw_multi.tests:zone=test-zone2 version=9CJ3w1EusK8OST2k17sTdTM.cuh3E3e 2024-02-01T10:25:33.362 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:25:33.362 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:25:33.363 DEBUG:boto:Method: HEAD 2024-02-01T10:25:33.363 DEBUG:boto:Path: /vclfcn-19/ 2024-02-01T10:25:33.363 DEBUG:boto:Data: 2024-02-01T10:25:33.363 DEBUG:boto:Headers: {} 2024-02-01T10:25:33.363 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:25:33.363 DEBUG:boto:Port: 8001 2024-02-01T10:25:33.363 DEBUG:boto:Params: {} 2024-02-01T10:25:33.363 DEBUG:boto:Token: None 2024-02-01T10:25:33.363 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:25:33 GMT /vclfcn-19/ 2024-02-01T10:25:33.364 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nbWj6wHMj0SXykDmBsWDO7WhJHk= 2024-02-01T10:25:33.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nbWj6wHMj0SXykDmBsWDO7WhJHk=', 'Content-Length': '0'} 2024-02-01T10:25:33.368 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '32'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000249d874b31c52588-0065bb719d-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:33.369 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:25:33.370 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:25:33.370 DEBUG:boto:Method: PUT 2024-02-01T10:25:33.370 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:25:33.370 DEBUG:boto:Data: 2024-02-01T10:25:33.370 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-02-01T10:25:33.370 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:25:33.370 DEBUG:boto:Port: 8001 2024-02-01T10:25:33.370 DEBUG:boto:Params: {} 2024-02-01T10:25:33.371 DEBUG:boto:Token: None 2024-02-01T10:25:33.371 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 01 Feb 2024 10:25:33 GMT /vclfcn-19/obj 2024-02-01T10:25:33.371 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jQQhAdXnykB1GGpAxsAZJJ2qrek= 2024-02-01T10:25:33.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:25:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jQQhAdXnykB1GGpAxsAZJJ2qrek='} 2024-02-01T10:25:33.392 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv'), ('x-amz-request-id', 'tx00000587946c5a2472895-0065bb719d-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:33.392 DEBUG:rgw_multi.tests:zone=test-zone3 version=Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv 2024-02-01T10:25:33.392 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:25:33.392 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:25:33.393 DEBUG:boto:Method: HEAD 2024-02-01T10:25:33.393 DEBUG:boto:Path: /vclfcn-19/ 2024-02-01T10:25:33.393 DEBUG:boto:Data: 2024-02-01T10:25:33.393 DEBUG:boto:Headers: {} 2024-02-01T10:25:33.393 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:25:33.393 DEBUG:boto:Port: 8000 2024-02-01T10:25:33.393 DEBUG:boto:Params: {} 2024-02-01T10:25:33.393 DEBUG:boto:Token: None 2024-02-01T10:25:33.393 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:25:33 GMT /vclfcn-19/ 2024-02-01T10:25:33.393 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nbWj6wHMj0SXykDmBsWDO7WhJHk= 2024-02-01T10:25:33.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nbWj6wHMj0SXykDmBsWDO7WhJHk=', 'Content-Length': '0'} 2024-02-01T10:25:33.398 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '40'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000033881fd10da1dd91-0065bb719d-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:33.398 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:25:33.398 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:25:33.399 DEBUG:boto:Method: PUT 2024-02-01T10:25:33.399 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:25:33.399 DEBUG:boto:Data: 2024-02-01T10:25:33.399 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-02-01T10:25:33.399 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:25:33.399 DEBUG:boto:Port: 8000 2024-02-01T10:25:33.399 DEBUG:boto:Params: {} 2024-02-01T10:25:33.399 DEBUG:boto:Token: None 2024-02-01T10:25:33.399 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 01 Feb 2024 10:25:33 GMT /vclfcn-19/obj 2024-02-01T10:25:33.400 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jQQhAdXnykB1GGpAxsAZJJ2qrek= 2024-02-01T10:25:33.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:25:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jQQhAdXnykB1GGpAxsAZJJ2qrek='} 2024-02-01T10:25:33.421 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'K23uwsLWGmQDGYGsdu-h04Fk2wApB0O'), ('x-amz-request-id', 'tx00000edadb087158de1f4-0065bb719d-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:33.421 DEBUG:rgw_multi.tests:zone=test-zone1 version=K23uwsLWGmQDGYGsdu-h04Fk2wApB0O 2024-02-01T10:25:33.422 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:25:33.422 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:25:33.422 DEBUG:boto:Method: HEAD 2024-02-01T10:25:33.422 DEBUG:boto:Path: /vclfcn-19/ 2024-02-01T10:25:33.422 DEBUG:boto:Data: 2024-02-01T10:25:33.422 DEBUG:boto:Headers: {} 2024-02-01T10:25:33.422 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:25:33.422 DEBUG:boto:Port: 8000 2024-02-01T10:25:33.422 DEBUG:boto:Params: {} 2024-02-01T10:25:33.422 DEBUG:boto:Token: None 2024-02-01T10:25:33.423 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:25:33 GMT /vclfcn-19/ 2024-02-01T10:25:33.423 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nbWj6wHMj0SXykDmBsWDO7WhJHk= 2024-02-01T10:25:33.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nbWj6wHMj0SXykDmBsWDO7WhJHk=', 'Content-Length': '0'} 2024-02-01T10:25:33.427 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '40'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004e3f23bc6d8b20d8-0065bb719d-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:33.427 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:25:33.427 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:25:33.428 DEBUG:boto:Method: PUT 2024-02-01T10:25:33.428 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:25:33.428 DEBUG:boto:Data: 2024-02-01T10:25:33.428 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-02-01T10:25:33.428 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:25:33.428 DEBUG:boto:Port: 8000 2024-02-01T10:25:33.429 DEBUG:boto:Params: {} 2024-02-01T10:25:33.429 DEBUG:boto:Token: None 2024-02-01T10:25:33.430 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 01 Feb 2024 10:25:33 GMT /vclfcn-19/obj 2024-02-01T10:25:33.430 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jQQhAdXnykB1GGpAxsAZJJ2qrek= 2024-02-01T10:25:33.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:25:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jQQhAdXnykB1GGpAxsAZJJ2qrek='} 2024-02-01T10:25:33.476 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz'), ('x-amz-request-id', 'tx00000f6adaf27aed9e8ba-0065bb719d-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:33.476 DEBUG:rgw_multi.tests:zone=test-zone2 version=3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz 2024-02-01T10:25:33.477 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:25:33.477 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:25:33.477 DEBUG:boto:Method: HEAD 2024-02-01T10:25:33.477 DEBUG:boto:Path: /vclfcn-19/ 2024-02-01T10:25:33.477 DEBUG:boto:Data: 2024-02-01T10:25:33.477 DEBUG:boto:Headers: {} 2024-02-01T10:25:33.477 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:25:33.477 DEBUG:boto:Port: 8001 2024-02-01T10:25:33.477 DEBUG:boto:Params: {} 2024-02-01T10:25:33.477 DEBUG:boto:Token: None 2024-02-01T10:25:33.478 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:25:33 GMT /vclfcn-19/ 2024-02-01T10:25:33.478 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nbWj6wHMj0SXykDmBsWDO7WhJHk= 2024-02-01T10:25:33.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nbWj6wHMj0SXykDmBsWDO7WhJHk=', 'Content-Length': '0'} 2024-02-01T10:25:33.482 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '40'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ec8b38549e154ab1-0065bb719d-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:33.483 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:25:33.483 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:25:33.483 DEBUG:boto:Method: PUT 2024-02-01T10:25:33.483 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:25:33.483 DEBUG:boto:Data: 2024-02-01T10:25:33.483 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-02-01T10:25:33.483 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:25:33.483 DEBUG:boto:Port: 8001 2024-02-01T10:25:33.483 DEBUG:boto:Params: {} 2024-02-01T10:25:33.483 DEBUG:boto:Token: None 2024-02-01T10:25:33.484 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 01 Feb 2024 10:25:33 GMT /vclfcn-19/obj 2024-02-01T10:25:33.484 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jQQhAdXnykB1GGpAxsAZJJ2qrek= 2024-02-01T10:25:33.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:25:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jQQhAdXnykB1GGpAxsAZJJ2qrek='} 2024-02-01T10:25:33.532 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'st2TOT.K7IOe-wFfL.hOrRVfLNWblRd'), ('x-amz-request-id', 'tx0000082ec5f0b4cf86c03-0065bb719d-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:33.532 DEBUG:rgw_multi.tests:zone=test-zone3 version=st2TOT.K7IOe-wFfL.hOrRVfLNWblRd 2024-02-01T10:25:33.533 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:25:33.533 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:25:33.533 DEBUG:boto:Method: HEAD 2024-02-01T10:25:33.533 DEBUG:boto:Path: /vclfcn-19/ 2024-02-01T10:25:33.533 DEBUG:boto:Data: 2024-02-01T10:25:33.533 DEBUG:boto:Headers: {} 2024-02-01T10:25:33.533 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:25:33.533 DEBUG:boto:Port: 8000 2024-02-01T10:25:33.533 DEBUG:boto:Params: {} 2024-02-01T10:25:33.533 DEBUG:boto:Token: None 2024-02-01T10:25:33.534 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:25:33 GMT /vclfcn-19/ 2024-02-01T10:25:33.534 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nbWj6wHMj0SXykDmBsWDO7WhJHk= 2024-02-01T10:25:33.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nbWj6wHMj0SXykDmBsWDO7WhJHk=', 'Content-Length': '0'} 2024-02-01T10:25:33.538 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007d3e26b701c89e9b-0065bb719d-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:33.539 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:25:33.539 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:25:33.539 DEBUG:boto:Method: PUT 2024-02-01T10:25:33.539 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:25:33.539 DEBUG:boto:Data: 2024-02-01T10:25:33.539 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-02-01T10:25:33.539 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:25:33.540 DEBUG:boto:Port: 8000 2024-02-01T10:25:33.540 DEBUG:boto:Params: {} 2024-02-01T10:25:33.540 DEBUG:boto:Token: None 2024-02-01T10:25:33.540 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 01 Feb 2024 10:25:33 GMT /vclfcn-19/obj 2024-02-01T10:25:33.540 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jQQhAdXnykB1GGpAxsAZJJ2qrek= 2024-02-01T10:25:33.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:25:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jQQhAdXnykB1GGpAxsAZJJ2qrek='} 2024-02-01T10:25:33.584 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi'), ('x-amz-request-id', 'tx00000d0ef0e89398db566-0065bb719d-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:33.584 DEBUG:rgw_multi.tests:zone=test-zone1 version=tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi 2024-02-01T10:25:33.585 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:25:33.585 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:25:33.585 DEBUG:boto:Method: HEAD 2024-02-01T10:25:33.585 DEBUG:boto:Path: /vclfcn-19/ 2024-02-01T10:25:33.585 DEBUG:boto:Data: 2024-02-01T10:25:33.585 DEBUG:boto:Headers: {} 2024-02-01T10:25:33.585 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:25:33.585 DEBUG:boto:Port: 8000 2024-02-01T10:25:33.585 DEBUG:boto:Params: {} 2024-02-01T10:25:33.585 DEBUG:boto:Token: None 2024-02-01T10:25:33.586 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:25:33 GMT /vclfcn-19/ 2024-02-01T10:25:33.586 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nbWj6wHMj0SXykDmBsWDO7WhJHk= 2024-02-01T10:25:33.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nbWj6wHMj0SXykDmBsWDO7WhJHk=', 'Content-Length': '0'} 2024-02-01T10:25:33.590 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000007c0a8c1540bd463-0065bb719d-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:33.590 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:25:33.590 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:25:33.590 DEBUG:boto:Method: PUT 2024-02-01T10:25:33.590 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:25:33.591 DEBUG:boto:Data: 2024-02-01T10:25:33.591 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-02-01T10:25:33.591 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:25:33.591 DEBUG:boto:Port: 8000 2024-02-01T10:25:33.591 DEBUG:boto:Params: {} 2024-02-01T10:25:33.591 DEBUG:boto:Token: None 2024-02-01T10:25:33.591 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 01 Feb 2024 10:25:33 GMT /vclfcn-19/obj 2024-02-01T10:25:33.591 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jQQhAdXnykB1GGpAxsAZJJ2qrek= 2024-02-01T10:25:33.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:25:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jQQhAdXnykB1GGpAxsAZJJ2qrek='} 2024-02-01T10:25:33.613 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS'), ('x-amz-request-id', 'tx000000dfe5483b0c40883-0065bb719d-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:33.613 DEBUG:rgw_multi.tests:zone=test-zone2 version=NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS 2024-02-01T10:25:33.613 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:25:33.613 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:25:33.613 DEBUG:boto:Method: HEAD 2024-02-01T10:25:33.614 DEBUG:boto:Path: /vclfcn-19/ 2024-02-01T10:25:33.614 DEBUG:boto:Data: 2024-02-01T10:25:33.614 DEBUG:boto:Headers: {} 2024-02-01T10:25:33.614 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:25:33.614 DEBUG:boto:Port: 8001 2024-02-01T10:25:33.614 DEBUG:boto:Params: {} 2024-02-01T10:25:33.614 DEBUG:boto:Token: None 2024-02-01T10:25:33.614 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:25:33 GMT /vclfcn-19/ 2024-02-01T10:25:33.614 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nbWj6wHMj0SXykDmBsWDO7WhJHk= 2024-02-01T10:25:33.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nbWj6wHMj0SXykDmBsWDO7WhJHk=', 'Content-Length': '0'} 2024-02-01T10:25:33.618 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000078118f67b1776f77-0065bb719d-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:33.619 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:25:33.619 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:25:33.619 DEBUG:boto:Method: PUT 2024-02-01T10:25:33.619 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:25:33.619 DEBUG:boto:Data: 2024-02-01T10:25:33.619 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-02-01T10:25:33.619 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:25:33.620 DEBUG:boto:Port: 8001 2024-02-01T10:25:33.620 DEBUG:boto:Params: {} 2024-02-01T10:25:33.620 DEBUG:boto:Token: None 2024-02-01T10:25:33.620 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 01 Feb 2024 10:25:33 GMT /vclfcn-19/obj 2024-02-01T10:25:33.620 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jQQhAdXnykB1GGpAxsAZJJ2qrek= 2024-02-01T10:25:33.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:25:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jQQhAdXnykB1GGpAxsAZJJ2qrek='} 2024-02-01T10:25:33.642 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'VQT2kBCkF07dugmyrhNeRpQpR0HK7bT'), ('x-amz-request-id', 'tx0000014a7476791e4f140-0065bb719d-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:33.642 DEBUG:rgw_multi.tests:zone=test-zone3 version=VQT2kBCkF07dugmyrhNeRpQpR0HK7bT 2024-02-01T10:25:33.642 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:25:33.642 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:25:33.643 DEBUG:boto:Method: HEAD 2024-02-01T10:25:33.643 DEBUG:boto:Path: /vclfcn-19/ 2024-02-01T10:25:33.643 DEBUG:boto:Data: 2024-02-01T10:25:33.643 DEBUG:boto:Headers: {} 2024-02-01T10:25:33.643 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:25:33.643 DEBUG:boto:Port: 8000 2024-02-01T10:25:33.643 DEBUG:boto:Params: {} 2024-02-01T10:25:33.643 DEBUG:boto:Token: None 2024-02-01T10:25:33.643 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:25:33 GMT /vclfcn-19/ 2024-02-01T10:25:33.643 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nbWj6wHMj0SXykDmBsWDO7WhJHk= 2024-02-01T10:25:33.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nbWj6wHMj0SXykDmBsWDO7WhJHk=', 'Content-Length': '0'} 2024-02-01T10:25:33.648 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '56'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fab51405219a6d63-0065bb719d-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:33.649 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:25:33.649 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:25:33.649 DEBUG:boto:Method: PUT 2024-02-01T10:25:33.649 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:25:33.649 DEBUG:boto:Data: 2024-02-01T10:25:33.649 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-02-01T10:25:33.649 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:25:33.650 DEBUG:boto:Port: 8000 2024-02-01T10:25:33.650 DEBUG:boto:Params: {} 2024-02-01T10:25:33.650 DEBUG:boto:Token: None 2024-02-01T10:25:33.650 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 01 Feb 2024 10:25:33 GMT /vclfcn-19/obj 2024-02-01T10:25:33.650 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jQQhAdXnykB1GGpAxsAZJJ2qrek= 2024-02-01T10:25:33.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:25:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jQQhAdXnykB1GGpAxsAZJJ2qrek='} 2024-02-01T10:25:33.671 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9'), ('x-amz-request-id', 'tx00000cb3a85d8316b9962-0065bb719d-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:33.671 DEBUG:rgw_multi.tests:zone=test-zone1 version=qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9 2024-02-01T10:25:33.671 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:25:33.671 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:25:33.671 DEBUG:boto:Method: HEAD 2024-02-01T10:25:33.672 DEBUG:boto:Path: /vclfcn-19/ 2024-02-01T10:25:33.672 DEBUG:boto:Data: 2024-02-01T10:25:33.672 DEBUG:boto:Headers: {} 2024-02-01T10:25:33.672 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:25:33.672 DEBUG:boto:Port: 8000 2024-02-01T10:25:33.672 DEBUG:boto:Params: {} 2024-02-01T10:25:33.672 DEBUG:boto:Token: None 2024-02-01T10:25:33.672 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:25:33 GMT /vclfcn-19/ 2024-02-01T10:25:33.672 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nbWj6wHMj0SXykDmBsWDO7WhJHk= 2024-02-01T10:25:33.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nbWj6wHMj0SXykDmBsWDO7WhJHk=', 'Content-Length': '0'} 2024-02-01T10:25:33.677 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '56'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c2d0e090b34e3396-0065bb719d-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:33.677 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:25:33.677 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:25:33.677 DEBUG:boto:Method: PUT 2024-02-01T10:25:33.678 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:25:33.678 DEBUG:boto:Data: 2024-02-01T10:25:33.678 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-02-01T10:25:33.678 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:25:33.678 DEBUG:boto:Port: 8000 2024-02-01T10:25:33.678 DEBUG:boto:Params: {} 2024-02-01T10:25:33.678 DEBUG:boto:Token: None 2024-02-01T10:25:33.679 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 01 Feb 2024 10:25:33 GMT /vclfcn-19/obj 2024-02-01T10:25:33.679 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jQQhAdXnykB1GGpAxsAZJJ2qrek= 2024-02-01T10:25:33.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:25:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jQQhAdXnykB1GGpAxsAZJJ2qrek='} 2024-02-01T10:25:33.701 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r'), ('x-amz-request-id', 'tx00000a8b48a92a71bc015-0065bb719d-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:33.701 DEBUG:rgw_multi.tests:zone=test-zone2 version=oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r 2024-02-01T10:25:33.701 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:25:33.702 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:25:33.702 DEBUG:boto:Method: HEAD 2024-02-01T10:25:33.702 DEBUG:boto:Path: /vclfcn-19/ 2024-02-01T10:25:33.702 DEBUG:boto:Data: 2024-02-01T10:25:33.702 DEBUG:boto:Headers: {} 2024-02-01T10:25:33.702 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:25:33.702 DEBUG:boto:Port: 8001 2024-02-01T10:25:33.702 DEBUG:boto:Params: {} 2024-02-01T10:25:33.703 DEBUG:boto:Token: None 2024-02-01T10:25:33.703 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:25:33 GMT /vclfcn-19/ 2024-02-01T10:25:33.703 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nbWj6wHMj0SXykDmBsWDO7WhJHk= 2024-02-01T10:25:33.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nbWj6wHMj0SXykDmBsWDO7WhJHk=', 'Content-Length': '0'} 2024-02-01T10:25:33.707 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '56'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ecb292694c8dd246-0065bb719d-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:33.707 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:25:33.707 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:25:33.707 DEBUG:boto:Method: PUT 2024-02-01T10:25:33.708 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:25:33.708 DEBUG:boto:Data: 2024-02-01T10:25:33.708 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-02-01T10:25:33.708 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:25:33.708 DEBUG:boto:Port: 8001 2024-02-01T10:25:33.708 DEBUG:boto:Params: {} 2024-02-01T10:25:33.708 DEBUG:boto:Token: None 2024-02-01T10:25:33.708 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 01 Feb 2024 10:25:33 GMT /vclfcn-19/obj 2024-02-01T10:25:33.708 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jQQhAdXnykB1GGpAxsAZJJ2qrek= 2024-02-01T10:25:33.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:25:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jQQhAdXnykB1GGpAxsAZJJ2qrek='} 2024-02-01T10:25:33.729 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '2lRbqumcCYnVWCIz20mU455245nXaz8'), ('x-amz-request-id', 'tx000006b3d59ab64a06451-0065bb719d-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:33.729 DEBUG:rgw_multi.tests:zone=test-zone3 version=2lRbqumcCYnVWCIz20mU455245nXaz8 2024-02-01T10:25:33.729 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:25:33.730 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:25:33.730 DEBUG:boto:Method: HEAD 2024-02-01T10:25:33.730 DEBUG:boto:Path: /vclfcn-19/ 2024-02-01T10:25:33.730 DEBUG:boto:Data: 2024-02-01T10:25:33.730 DEBUG:boto:Headers: {} 2024-02-01T10:25:33.730 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:25:33.730 DEBUG:boto:Port: 8000 2024-02-01T10:25:33.730 DEBUG:boto:Params: {} 2024-02-01T10:25:33.730 DEBUG:boto:Token: None 2024-02-01T10:25:33.731 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:25:33 GMT /vclfcn-19/ 2024-02-01T10:25:33.731 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nbWj6wHMj0SXykDmBsWDO7WhJHk= 2024-02-01T10:25:33.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nbWj6wHMj0SXykDmBsWDO7WhJHk=', 'Content-Length': '0'} 2024-02-01T10:25:33.735 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '64'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ff31156a6229882e-0065bb719d-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:33.735 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:25:33.735 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:25:33.735 DEBUG:boto:Method: PUT 2024-02-01T10:25:33.735 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:25:33.735 DEBUG:boto:Data: 2024-02-01T10:25:33.735 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-02-01T10:25:33.735 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:25:33.736 DEBUG:boto:Port: 8000 2024-02-01T10:25:33.736 DEBUG:boto:Params: {} 2024-02-01T10:25:33.736 DEBUG:boto:Token: None 2024-02-01T10:25:33.736 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 01 Feb 2024 10:25:33 GMT /vclfcn-19/obj 2024-02-01T10:25:33.736 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jQQhAdXnykB1GGpAxsAZJJ2qrek= 2024-02-01T10:25:33.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:25:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jQQhAdXnykB1GGpAxsAZJJ2qrek='} 2024-02-01T10:25:33.755 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr.'), ('x-amz-request-id', 'tx000006b84f6341e6d7d8f-0065bb719d-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:33.755 DEBUG:rgw_multi.tests:zone=test-zone1 version=eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr. 2024-02-01T10:25:33.755 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:25:33.755 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:25:33.755 DEBUG:boto:Method: HEAD 2024-02-01T10:25:33.755 DEBUG:boto:Path: /vclfcn-19/ 2024-02-01T10:25:33.756 DEBUG:boto:Data: 2024-02-01T10:25:33.756 DEBUG:boto:Headers: {} 2024-02-01T10:25:33.756 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:25:33.756 DEBUG:boto:Port: 8000 2024-02-01T10:25:33.757 DEBUG:boto:Params: {} 2024-02-01T10:25:33.758 DEBUG:boto:Token: None 2024-02-01T10:25:33.758 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:25:33 GMT /vclfcn-19/ 2024-02-01T10:25:33.758 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nbWj6wHMj0SXykDmBsWDO7WhJHk= 2024-02-01T10:25:33.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nbWj6wHMj0SXykDmBsWDO7WhJHk=', 'Content-Length': '0'} 2024-02-01T10:25:33.763 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '64'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000001a58ee831fc4a66-0065bb719d-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:33.763 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:25:33.764 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:25:33.764 DEBUG:boto:Method: PUT 2024-02-01T10:25:33.764 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:25:33.764 DEBUG:boto:Data: 2024-02-01T10:25:33.764 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-02-01T10:25:33.765 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:25:33.765 DEBUG:boto:Port: 8000 2024-02-01T10:25:33.765 DEBUG:boto:Params: {} 2024-02-01T10:25:33.765 DEBUG:boto:Token: None 2024-02-01T10:25:33.765 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 01 Feb 2024 10:25:33 GMT /vclfcn-19/obj 2024-02-01T10:25:33.766 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jQQhAdXnykB1GGpAxsAZJJ2qrek= 2024-02-01T10:25:33.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:25:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jQQhAdXnykB1GGpAxsAZJJ2qrek='} 2024-02-01T10:25:33.808 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc'), ('x-amz-request-id', 'tx0000091dd30bad9e22837-0065bb719d-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:33.812 DEBUG:rgw_multi.tests:zone=test-zone2 version=UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc 2024-02-01T10:25:33.812 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:25:33.812 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:25:33.812 DEBUG:boto:Method: HEAD 2024-02-01T10:25:33.812 DEBUG:boto:Path: /vclfcn-19/ 2024-02-01T10:25:33.813 DEBUG:boto:Data: 2024-02-01T10:25:33.813 DEBUG:boto:Headers: {} 2024-02-01T10:25:33.813 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:25:33.813 DEBUG:boto:Port: 8001 2024-02-01T10:25:33.813 DEBUG:boto:Params: {} 2024-02-01T10:25:33.813 DEBUG:boto:Token: None 2024-02-01T10:25:33.813 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:25:33 GMT /vclfcn-19/ 2024-02-01T10:25:33.813 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nbWj6wHMj0SXykDmBsWDO7WhJHk= 2024-02-01T10:25:33.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nbWj6wHMj0SXykDmBsWDO7WhJHk=', 'Content-Length': '0'} 2024-02-01T10:25:33.818 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '64'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c0efbd78f344c4b8-0065bb719d-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:33.818 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:25:33.818 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:25:33.818 DEBUG:boto:Method: PUT 2024-02-01T10:25:33.819 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:25:33.819 DEBUG:boto:Data: 2024-02-01T10:25:33.819 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-02-01T10:25:33.819 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:25:33.819 DEBUG:boto:Port: 8001 2024-02-01T10:25:33.819 DEBUG:boto:Params: {} 2024-02-01T10:25:33.819 DEBUG:boto:Token: None 2024-02-01T10:25:33.819 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 01 Feb 2024 10:25:33 GMT /vclfcn-19/obj 2024-02-01T10:25:33.820 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jQQhAdXnykB1GGpAxsAZJJ2qrek= 2024-02-01T10:25:33.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:25:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jQQhAdXnykB1GGpAxsAZJJ2qrek='} 2024-02-01T10:25:33.868 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN'), ('x-amz-request-id', 'tx00000e22b8102239b0ab4-0065bb719d-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:33.869 DEBUG:rgw_multi.tests:zone=test-zone3 version=tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN 2024-02-01T10:25:33.869 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:25:33.869 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:25:33.869 DEBUG:boto:Method: HEAD 2024-02-01T10:25:33.869 DEBUG:boto:Path: /vclfcn-19/ 2024-02-01T10:25:33.869 DEBUG:boto:Data: 2024-02-01T10:25:33.869 DEBUG:boto:Headers: {} 2024-02-01T10:25:33.869 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:25:33.869 DEBUG:boto:Port: 8000 2024-02-01T10:25:33.870 DEBUG:boto:Params: {} 2024-02-01T10:25:33.870 DEBUG:boto:Token: None 2024-02-01T10:25:33.870 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:25:33 GMT /vclfcn-19/ 2024-02-01T10:25:33.870 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nbWj6wHMj0SXykDmBsWDO7WhJHk= 2024-02-01T10:25:33.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nbWj6wHMj0SXykDmBsWDO7WhJHk=', 'Content-Length': '0'} 2024-02-01T10:25:33.874 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000022e551956f097b28-0065bb719d-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:33.874 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:25:33.875 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:25:33.875 DEBUG:boto:Method: PUT 2024-02-01T10:25:33.875 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:25:33.875 DEBUG:boto:Data: 2024-02-01T10:25:33.875 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-02-01T10:25:33.875 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:25:33.875 DEBUG:boto:Port: 8000 2024-02-01T10:25:33.875 DEBUG:boto:Params: {} 2024-02-01T10:25:33.875 DEBUG:boto:Token: None 2024-02-01T10:25:33.875 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 01 Feb 2024 10:25:33 GMT /vclfcn-19/obj 2024-02-01T10:25:33.876 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jQQhAdXnykB1GGpAxsAZJJ2qrek= 2024-02-01T10:25:33.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:25:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jQQhAdXnykB1GGpAxsAZJJ2qrek='} 2024-02-01T10:25:33.924 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u'), ('x-amz-request-id', 'tx000002dfd837e05922ee4-0065bb719d-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:33.925 DEBUG:rgw_multi.tests:zone=test-zone1 version=fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u 2024-02-01T10:25:33.925 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:25:33.925 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:25:33.925 DEBUG:boto:Method: HEAD 2024-02-01T10:25:33.925 DEBUG:boto:Path: /vclfcn-19/ 2024-02-01T10:25:33.925 DEBUG:boto:Data: 2024-02-01T10:25:33.925 DEBUG:boto:Headers: {} 2024-02-01T10:25:33.925 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:25:33.925 DEBUG:boto:Port: 8000 2024-02-01T10:25:33.925 DEBUG:boto:Params: {} 2024-02-01T10:25:33.926 DEBUG:boto:Token: None 2024-02-01T10:25:33.926 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:25:33 GMT /vclfcn-19/ 2024-02-01T10:25:33.926 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nbWj6wHMj0SXykDmBsWDO7WhJHk= 2024-02-01T10:25:33.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nbWj6wHMj0SXykDmBsWDO7WhJHk=', 'Content-Length': '0'} 2024-02-01T10:25:33.930 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bc982bef627b105e-0065bb719d-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:33.930 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:25:33.931 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:25:33.931 DEBUG:boto:Method: PUT 2024-02-01T10:25:33.931 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:25:33.931 DEBUG:boto:Data: 2024-02-01T10:25:33.931 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-02-01T10:25:33.931 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:25:33.931 DEBUG:boto:Port: 8000 2024-02-01T10:25:33.931 DEBUG:boto:Params: {} 2024-02-01T10:25:33.931 DEBUG:boto:Token: None 2024-02-01T10:25:33.932 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 01 Feb 2024 10:25:33 GMT /vclfcn-19/obj 2024-02-01T10:25:33.932 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jQQhAdXnykB1GGpAxsAZJJ2qrek= 2024-02-01T10:25:33.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:25:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jQQhAdXnykB1GGpAxsAZJJ2qrek='} 2024-02-01T10:25:33.955 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-amz-request-id', 'tx000003aae2b72969527bb-0065bb719d-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:33.955 DEBUG:rgw_multi.tests:zone=test-zone2 version=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:25:33.955 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:25:33.955 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:25:33.955 DEBUG:boto:Method: HEAD 2024-02-01T10:25:33.955 DEBUG:boto:Path: /vclfcn-19/ 2024-02-01T10:25:33.955 DEBUG:boto:Data: 2024-02-01T10:25:33.956 DEBUG:boto:Headers: {} 2024-02-01T10:25:33.956 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:25:33.956 DEBUG:boto:Port: 8001 2024-02-01T10:25:33.956 DEBUG:boto:Params: {} 2024-02-01T10:25:33.956 DEBUG:boto:Token: None 2024-02-01T10:25:33.956 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:25:33 GMT /vclfcn-19/ 2024-02-01T10:25:33.956 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nbWj6wHMj0SXykDmBsWDO7WhJHk= 2024-02-01T10:25:33.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:25:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nbWj6wHMj0SXykDmBsWDO7WhJHk=', 'Content-Length': '0'} 2024-02-01T10:25:33.961 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a623dcda60641c9e-0065bb719d-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:33.961 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:25:33.961 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:25:33.961 DEBUG:boto:Method: PUT 2024-02-01T10:25:33.962 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:25:33.962 DEBUG:boto:Data: 2024-02-01T10:25:33.962 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-02-01T10:25:33.962 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:25:33.962 DEBUG:boto:Port: 8001 2024-02-01T10:25:33.962 DEBUG:boto:Params: {} 2024-02-01T10:25:33.962 DEBUG:boto:Token: None 2024-02-01T10:25:33.962 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 01 Feb 2024 10:25:33 GMT /vclfcn-19/obj 2024-02-01T10:25:33.962 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jQQhAdXnykB1GGpAxsAZJJ2qrek= 2024-02-01T10:25:33.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:25:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jQQhAdXnykB1GGpAxsAZJJ2qrek='} 2024-02-01T10:25:33.986 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'RrytQti15jy37pft0WxtgWdt4Zs7l4E'), ('x-amz-request-id', 'tx0000064febf310d7b0700-0065bb719d-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:25:33.986 DEBUG:rgw_multi.tests:zone=test-zone3 version=RrytQti15jy37pft0WxtgWdt4Zs7l4E 2024-02-01T10:25:33.986 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=vclfcn-19 2024-02-01T10:25:33.986 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:25:33.986 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:25:33.987 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket vclfcn-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-02-01T10:25:34.036 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:25:34.037 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:25:34.100 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:25:34.102+0000 7f9172faa980 1 waiting for incremental sync to catch up: 2024-02-01T10:25:34.100 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, , , , , 00000000004.17.10, , , , , ] 2024-02-01T10:25:34.100 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, , , , , 00000000004.17.10, , , 00000000020.56.12, , ] 2024-02-01T10:25:39.102 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:25:39.102+0000 7f9172faa980 1 waiting for incremental sync to catch up: 2024-02-01T10:25:39.102 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, , , , , 00000000004.17.10, , , , , ] 2024-02-01T10:25:39.102 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, , , , , 00000000004.17.10, , , 00000000020.56.12, , ] 2024-02-01T10:25:44.104 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:25:44.106+0000 7f9172faa980 1 waiting for incremental sync to catch up: 2024-02-01T10:25:44.104 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, , , , , 00000000004.17.10, , , , , ] 2024-02-01T10:25:44.105 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, , , , , 00000000004.17.10, , , 00000000020.56.12, , ] 2024-02-01T10:25:49.107 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:25:49.106+0000 7f9172faa980 1 waiting for incremental sync to catch up: 2024-02-01T10:25:49.107 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, , , , , 00000000004.17.10, , , 00000000005.27.6, , ] 2024-02-01T10:25:49.108 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, , , , , 00000000004.17.10, , , 00000000020.56.12, , ] 2024-02-01T10:25:54.109 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:25:54.110+0000 7f9172faa980 1 waiting for incremental sync to catch up: 2024-02-01T10:25:54.109 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, , , , , 00000000004.17.10, , , 00000000005.27.6, , ] 2024-02-01T10:25:54.109 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, , , , , 00000000004.17.10, , , 00000000020.56.12, , ] 2024-02-01T10:25:59.111 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:25:59.110+0000 7f9172faa980 1 waiting for incremental sync to catch up: 2024-02-01T10:25:59.111 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, , , , , 00000000004.17.10, , , 00000000005.27.6, , ] 2024-02-01T10:25:59.111 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, , , , , 00000000004.17.10, , , 00000000020.56.12, , ] 2024-02-01T10:26:04.113 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:26:04.114+0000 7f9172faa980 1 waiting for incremental sync to catch up: 2024-02-01T10:26:04.114 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, , , , , 00000000004.17.10, , , 00000000005.27.6, , ] 2024-02-01T10:26:04.114 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, , , , , 00000000004.17.10, , , 00000000020.56.12, , ] 2024-02-01T10:26:09.116 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:26:09.118+0000 7f9172faa980 1 waiting for incremental sync to catch up: 2024-02-01T10:26:09.116 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, , , , , 00000000004.17.10, , , 00000000007.31.6, , ] 2024-02-01T10:26:09.116 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, , , , , 00000000004.17.10, , , 00000000020.56.12, , ] 2024-02-01T10:26:14.118 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:26:14.118+0000 7f9172faa980 1 waiting for incremental sync to catch up: 2024-02-01T10:26:14.119 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, , , , , 00000000004.17.10, , , 00000000007.31.6, , ] 2024-02-01T10:26:14.119 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, , , , , 00000000004.17.10, , , 00000000020.56.12, , ] 2024-02-01T10:26:19.121 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:26:19.121+0000 7f9172faa980 1 waiting for incremental sync to catch up: 2024-02-01T10:26:19.121 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, , , , , 00000000004.17.10, , , 00000000007.31.6, , ] 2024-02-01T10:26:19.121 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, , , , , 00000000004.17.10, , , 00000000020.56.12, , ] 2024-02-01T10:26:24.123 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:26:24.121+0000 7f9172faa980 1 waiting for incremental sync to catch up: 2024-02-01T10:26:24.123 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, , , , , 00000000004.17.10, , , 00000000007.31.6, , ] 2024-02-01T10:26:24.123 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, , , , , 00000000004.17.10, , , 00000000020.56.12, , ] 2024-02-01T10:26:29.125 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:26:29.125+0000 7f9172faa980 1 waiting for incremental sync to catch up: 2024-02-01T10:26:29.126 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, , , , , 00000000004.17.10, , , 00000000009.35.6, , ] 2024-02-01T10:26:29.126 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, , , , , 00000000004.17.10, , , 00000000020.56.12, , ] 2024-02-01T10:26:34.127 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:26:34.129+0000 7f9172faa980 1 waiting for incremental sync to catch up: 2024-02-01T10:26:34.128 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, , , , , 00000000004.17.10, , , 00000000009.35.6, , ] 2024-02-01T10:26:34.128 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, , , , , 00000000004.17.10, , , 00000000020.56.12, , ] 2024-02-01T10:26:39.130 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:26:39.129+0000 7f9172faa980 1 waiting for incremental sync to catch up: 2024-02-01T10:26:39.130 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, , , , , 00000000004.17.10, , , 00000000009.35.6, , ] 2024-02-01T10:26:39.130 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, , , , , 00000000004.17.10, , , 00000000020.56.12, , ] 2024-02-01T10:26:44.132 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:26:44.133+0000 7f9172faa980 1 waiting for incremental sync to catch up: 2024-02-01T10:26:44.132 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, , , , , 00000000004.17.10, , , 00000000009.35.6, , ] 2024-02-01T10:26:44.132 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, , , , , 00000000004.17.10, , , 00000000020.56.12, , ] 2024-02-01T10:26:49.134 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:26:49.133+0000 7f9172faa980 1 bucket sync caught up with source: 2024-02-01T10:26:49.135 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, , , , , 00000000004.17.10, , , 00000000020.56.12, , ] 2024-02-01T10:26:49.135 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, , , , , 00000000004.17.10, , , 00000000020.56.12, , ] 2024-02-01T10:26:49.135 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:26:49.133+0000 7f9172faa980 0 bucket checkpoint complete 2024-02-01T10:26:49.182 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=vclfcn-19 2024-02-01T10:26:49.182 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:26:49.182 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:26:49.183 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:26:49.233 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:26:49.233 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:26:49.292 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:26:49.297+0000 7f93c831b980 1 waiting for incremental sync to catch up: 2024-02-01T10:26:49.292 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, , , , , 00000000004.17.10, , , 00000000030.86.6, , ] 2024-02-01T10:26:49.292 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, , , , , 00000000004.17.10, , , 00000000044.122.10, , ] 2024-02-01T10:26:54.294 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:26:54.300+0000 7f93c831b980 1 waiting for incremental sync to catch up: 2024-02-01T10:26:54.294 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, , , , , 00000000004.17.10, , , 00000000032.92.6, , ] 2024-02-01T10:26:54.295 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, , , , , 00000000004.17.10, , , 00000000044.122.10, , ] 2024-02-01T10:26:59.296 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:26:59.300+0000 7f93c831b980 1 waiting for incremental sync to catch up: 2024-02-01T10:26:59.297 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, , , , , 00000000004.17.10, , , 00000000032.92.6, , ] 2024-02-01T10:26:59.297 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, , , , , 00000000004.17.10, , , 00000000044.122.10, , ] 2024-02-01T10:27:04.299 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:27:04.304+0000 7f93c831b980 1 waiting for incremental sync to catch up: 2024-02-01T10:27:04.299 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, , , , , 00000000004.17.10, , , 00000000032.92.6, , ] 2024-02-01T10:27:04.299 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, , , , , 00000000004.17.10, , , 00000000044.122.10, , ] 2024-02-01T10:27:07.469 INFO:tasks.rgw.c2.client.0.smithi187.stdout:2024-02-01T10:27:07.469+0000 7f7bead57640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-02-01T10:27:09.301 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:27:09.308+0000 7f93c831b980 1 waiting for incremental sync to catch up: 2024-02-01T10:27:09.301 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, , , , , 00000000004.17.10, , , 00000000032.92.6, , ] 2024-02-01T10:27:09.301 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, , , , , 00000000004.17.10, , , 00000000044.122.10, , ] 2024-02-01T10:27:14.304 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:27:14.308+0000 7f93c831b980 1 bucket sync caught up with source: 2024-02-01T10:27:14.304 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, , , , , 00000000004.17.10, , , 00000000046.126.10, , ] 2024-02-01T10:27:14.304 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, , , , , 00000000004.17.10, , , 00000000044.122.10, , ] 2024-02-01T10:27:14.304 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:27:14.308+0000 7f93c831b980 0 bucket checkpoint complete 2024-02-01T10:27:14.340 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=vclfcn-19 2024-02-01T10:27:14.340 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:27:14.340 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:27:14.340 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:27:14.429 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:27:14.429 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:27:14.491 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:27:14.496+0000 7fee1cd7b980 1 waiting for incremental sync to catch up: 2024-02-01T10:27:14.491 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, , , , , 00000000004.18.10, , , 00000000043.116.10, , ] 2024-02-01T10:27:14.492 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, , , , , 00000000004.18.10, , , 00000000044.118.10, , ] 2024-02-01T10:27:19.494 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:27:19.500+0000 7fee1cd7b980 1 waiting for incremental sync to catch up: 2024-02-01T10:27:19.494 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, , , , , 00000000004.18.10, , , 00000000043.116.10, , ] 2024-02-01T10:27:19.494 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, , , , , 00000000004.18.10, , , 00000000044.118.10, , ] 2024-02-01T10:27:24.497 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:27:24.500+0000 7fee1cd7b980 1 waiting for incremental sync to catch up: 2024-02-01T10:27:24.497 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, , , , , 00000000004.18.10, , , 00000000043.116.10, , ] 2024-02-01T10:27:24.497 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, , , , , 00000000004.18.10, , , 00000000044.118.10, , ] 2024-02-01T10:27:29.499 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:27:29.504+0000 7fee1cd7b980 1 bucket sync caught up with source: 2024-02-01T10:27:29.499 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, , , , , 00000000004.18.10, , , 00000000044.118.10, , ] 2024-02-01T10:27:29.499 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, , , , , 00000000004.18.10, , , 00000000044.118.10, , ] 2024-02-01T10:27:29.499 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:27:29.504+0000 7fee1cd7b980 0 bucket checkpoint complete 2024-02-01T10:27:29.536 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=vclfcn-19 2024-02-01T10:27:29.536 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:27:29.536 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:27:29.536 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:27:29.583 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:27:29.583 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:27:29.645 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:27:29.652+0000 7f828f771980 1 waiting for incremental sync to catch up: 2024-02-01T10:27:29.645 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, , , , , 00000000004.18.10, , , 00000000044.118.10, , ] 2024-02-01T10:27:29.646 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, , , , , 00000000004.18.10, , , 00000000045.120.10, , ] 2024-02-01T10:27:34.647 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:27:34.652+0000 7f828f771980 1 bucket sync caught up with source: 2024-02-01T10:27:34.648 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, , , , , 00000000004.18.10, , , 00000000045.120.10, , ] 2024-02-01T10:27:34.648 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, , , , , 00000000004.18.10, , , 00000000045.120.10, , ] 2024-02-01T10:27:34.648 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:27:34.652+0000 7f828f771980 0 bucket checkpoint complete 2024-02-01T10:27:34.686 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=vclfcn-19 2024-02-01T10:27:34.686 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:27:34.686 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:27:34.686 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:27:34.737 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:27:34.737 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:27:34.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:27:34.804+0000 7f0288fb2980 1 bucket sync caught up with source: 2024-02-01T10:27:34.799 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, , , , , 00000000004.26.10, , , 00000000046.126.10, , ] 2024-02-01T10:27:34.799 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, , , , , 00000000004.26.10, , , 00000000046.126.10, , ] 2024-02-01T10:27:34.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:27:34.804+0000 7f0288fb2980 0 bucket checkpoint complete 2024-02-01T10:27:34.822 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=vclfcn-19 2024-02-01T10:27:34.822 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:27:34.822 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:27:34.823 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket vclfcn-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-02-01T10:27:34.872 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:27:34.872 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:27:34.941 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:27:34.944+0000 7fad19f5e980 1 bucket sync caught up with source: 2024-02-01T10:27:34.941 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, , , , , 00000000004.26.10, , , 00000000046.126.10, , ] 2024-02-01T10:27:34.941 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, , , , , 00000000004.26.10, , , 00000000046.126.10, , ] 2024-02-01T10:27:34.942 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:27:34.944+0000 7fad19f5e980 0 bucket checkpoint complete 2024-02-01T10:27:34.964 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-19 zones={test-zone2, test-zone1} 2024-02-01T10:27:34.964 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:34.964 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:34.965 DEBUG:boto:Method: HEAD 2024-02-01T10:27:34.965 DEBUG:boto:Path: /vclfcn-19/ 2024-02-01T10:27:34.965 DEBUG:boto:Data: 2024-02-01T10:27:34.965 DEBUG:boto:Headers: {} 2024-02-01T10:27:34.965 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:34.965 DEBUG:boto:Port: 8000 2024-02-01T10:27:34.965 DEBUG:boto:Params: {} 2024-02-01T10:27:34.966 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T10:27:34.966 DEBUG:boto:Token: None 2024-02-01T10:27:34.967 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:34 GMT /vclfcn-19/ 2024-02-01T10:27:34.967 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gh+O/+4JP2rNkyfTxD4tIU5ZJrw= 2024-02-01T10:27:34.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:34 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gh+O/+4JP2rNkyfTxD4tIU5ZJrw=', 'Content-Length': '0'} 2024-02-01T10:27:34.973 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005a6f0047f13c502b-0065bb7216-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:27:34 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:34.973 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:34.974 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:34.974 DEBUG:boto:Method: HEAD 2024-02-01T10:27:34.974 DEBUG:boto:Path: /vclfcn-19/ 2024-02-01T10:27:34.974 DEBUG:boto:Data: 2024-02-01T10:27:34.974 DEBUG:boto:Headers: {} 2024-02-01T10:27:34.974 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:34.975 DEBUG:boto:Port: 8000 2024-02-01T10:27:34.975 DEBUG:boto:Params: {} 2024-02-01T10:27:34.975 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T10:27:34.975 DEBUG:boto:Token: None 2024-02-01T10:27:34.975 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:34 GMT /vclfcn-19/ 2024-02-01T10:27:34.975 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gh+O/+4JP2rNkyfTxD4tIU5ZJrw= 2024-02-01T10:27:34.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:34 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gh+O/+4JP2rNkyfTxD4tIU5ZJrw=', 'Content-Length': '0'} 2024-02-01T10:27:34.982 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000afa45dbbe17d0db2-0065bb7216-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:27:34 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:34.983 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:27:34.983 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:34.983 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:34.983 DEBUG:boto:path=/vclfcn-19/?versions&max-keys=999 2024-02-01T10:27:34.983 DEBUG:boto:auth_path=/vclfcn-19/?versions&max-keys=999 2024-02-01T10:27:34.983 DEBUG:boto:Method: GET 2024-02-01T10:27:34.983 DEBUG:boto:Path: /vclfcn-19/?versions&max-keys=999 2024-02-01T10:27:34.983 DEBUG:boto:Data: 2024-02-01T10:27:34.983 DEBUG:boto:Headers: {} 2024-02-01T10:27:34.984 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:34.984 DEBUG:boto:Port: 8000 2024-02-01T10:27:34.984 DEBUG:boto:Params: {} 2024-02-01T10:27:34.984 DEBUG:boto:Token: None 2024-02-01T10:27:34.984 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:34 GMT /vclfcn-19/?versions 2024-02-01T10:27:34.984 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gTYEoJot4w1uTYP+tBG+F1BWxFU= 2024-02-01T10:27:34.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:34 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gTYEoJot4w1uTYP+tBG+F1BWxFU=', 'Content-Length': '0'} 2024-02-01T10:27:34.988 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000057aea348b60aeea3-0065bb7216-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:34 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:34.988 DEBUG:boto:b'vclfcn-19999falsedummyg5UScztMjPBIYZ2b5yOn-qsm.eQTcnutrue2024-02-01T10:23:11.115Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23llUchmWTvJKOzZpTkl59nzz.uMOQbtrue2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjRrytQti15jy37pft0WxtgWdt4Zs7l4Efalse2024-02-01T10:25:33.973Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjfzdlI7n.dsuwfAOSBNyk84Wd7l5p70ufalse2024-02-01T10:25:33.886Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUOa9NAxhXbH7RR3jkzlpXPh.fGAfBJcfalse2024-02-01T10:25:33.774Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjeNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr.false2024-02-01T10:25:33.746Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtGIsEYvB.VyZPdq9CgtD6kEeP4rzbDNfalse2024-02-01T10:25:33.830Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2lRbqumcCYnVWCIz20mU455245nXaz8false2024-02-01T10:25:33.719Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjoGhdnHMtrLPTgFgXlKfl.UXdmrUnL2rfalse2024-02-01T10:25:33.687Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjqex3Bh72IV4plxG2V2Lm7bvqwXqQNK9false2024-02-01T10:25:33.661Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjNCPEvRYQDJpLrTejJRqmTf2j0FWCBCSfalse2024-02-01T10:25:33.599Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjVQT2kBCkF07dugmyrhNeRpQpR0HK7bTfalse2024-02-01T10:25:33.630Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtvvjWUbgZIRYhHk9r.liT6sPCWCs2Zifalse2024-02-01T10:25:33.551Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3OSaU40E5-Hgv6pEDB8btq8uZcAFHVzfalse2024-02-01T10:25:33.438Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjK23uwsLWGmQDGYGsdu-h04Fk2wApB0Ofalse2024-02-01T10:25:33.410Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjst2TOT.K7IOe-wFfL.hOrRVfLNWblRdfalse2024-02-01T10:25:33.494Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9CJ3w1EusK8OST2k17sTdTM.cuh3E3efalse2024-02-01T10:25:33.349Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjVwfx4ETdAtTkKn2rTCQT4onqB6sYulvfalse2024-02-01T10:25:33.381Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjiGO2QyMOL81IaF-H4sHDqyECPlnZBmEfalse2024-02-01T10:25:33.323Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjcYHYWFiJtzK2FpqgaUq4bR.SX5MaCkIfalse2024-02-01T10:25:33.206Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjrasya8taOZsGTj9rDi7keghf.VnPFlFfalse2024-02-01T10:25:33.291Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjy-DPCvngcj9q7W-lUL16Jgbp.NhV7NEfalse2024-02-01T10:25:33.257Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj0KR6m6ymTGitKe6vabQyVENAnBL669wfalse2024-02-01T10:25:33.118Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2Hfm7x.sO5cadpLfjJNuhN.TAkjS54Efalse2024-02-01T10:25:33.093Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6false2024-02-01T10:25:33.150Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHXfalse2024-02-01T10:25:33.030Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj63CC4agggciY3jdYETzlNEHLx15uh8wfalse2024-02-01T10:25:33.063Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjYzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1ofalse2024-02-01T10:25:32.998Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjB5jd89.SLRVLOAPcXqackukLvQnZY1Bfalse2024-02-01T10:25:32.872Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjrcBw9NDMcjurzSdpB.bvwX4JmUDkwrQfalse2024-02-01T10:25:32.964Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjs-qxjv7TkMdxeC61HL2kxUqODSwa-1Ifalse2024-02-01T10:25:32.926Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:34.990 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-02-01T10:27:34.990 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:34.990 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:34.990 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:34.990 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:34.990 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:34.991 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:34.991 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:34.991 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:34.991 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:34.991 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:34.991 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:34.991 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:34.991 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:34.991 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:34.991 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:34.991 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:34.991 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:34.992 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:34.992 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:34.992 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:34.992 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:34.992 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:34.992 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:34.992 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:34.992 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:34.992 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:34.992 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:34.992 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:34.992 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:34.993 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:34.993 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:27:34.993 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:34.993 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:34.993 DEBUG:boto:path=/vclfcn-19/?versions&max-keys=999 2024-02-01T10:27:34.993 DEBUG:boto:auth_path=/vclfcn-19/?versions&max-keys=999 2024-02-01T10:27:34.993 DEBUG:boto:Method: GET 2024-02-01T10:27:34.993 DEBUG:boto:Path: /vclfcn-19/?versions&max-keys=999 2024-02-01T10:27:34.993 DEBUG:boto:Data: 2024-02-01T10:27:34.993 DEBUG:boto:Headers: {} 2024-02-01T10:27:34.993 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:34.994 DEBUG:boto:Port: 8000 2024-02-01T10:27:34.994 DEBUG:boto:Params: {} 2024-02-01T10:27:34.994 DEBUG:boto:Token: None 2024-02-01T10:27:34.994 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:34 GMT /vclfcn-19/?versions 2024-02-01T10:27:34.994 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gTYEoJot4w1uTYP+tBG+F1BWxFU= 2024-02-01T10:27:34.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:34 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gTYEoJot4w1uTYP+tBG+F1BWxFU=', 'Content-Length': '0'} 2024-02-01T10:27:35.001 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a622a90bc2a98c1-0065bb7217-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:35.001 DEBUG:boto:b'vclfcn-19999falsedummyg5UScztMjPBIYZ2b5yOn-qsm.eQTcnutrue2024-02-01T10:23:11.115Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23llUchmWTvJKOzZpTkl59nzz.uMOQbtrue2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjRrytQti15jy37pft0WxtgWdt4Zs7l4Efalse2024-02-01T10:25:33.973Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjfzdlI7n.dsuwfAOSBNyk84Wd7l5p70ufalse2024-02-01T10:25:33.886Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUOa9NAxhXbH7RR3jkzlpXPh.fGAfBJcfalse2024-02-01T10:25:33.774Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjeNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr.false2024-02-01T10:25:33.746Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtGIsEYvB.VyZPdq9CgtD6kEeP4rzbDNfalse2024-02-01T10:25:33.830Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2lRbqumcCYnVWCIz20mU455245nXaz8false2024-02-01T10:25:33.719Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjoGhdnHMtrLPTgFgXlKfl.UXdmrUnL2rfalse2024-02-01T10:25:33.687Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjqex3Bh72IV4plxG2V2Lm7bvqwXqQNK9false2024-02-01T10:25:33.661Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjNCPEvRYQDJpLrTejJRqmTf2j0FWCBCSfalse2024-02-01T10:25:33.599Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjVQT2kBCkF07dugmyrhNeRpQpR0HK7bTfalse2024-02-01T10:25:33.630Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtvvjWUbgZIRYhHk9r.liT6sPCWCs2Zifalse2024-02-01T10:25:33.551Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3OSaU40E5-Hgv6pEDB8btq8uZcAFHVzfalse2024-02-01T10:25:33.438Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjK23uwsLWGmQDGYGsdu-h04Fk2wApB0Ofalse2024-02-01T10:25:33.410Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjst2TOT.K7IOe-wFfL.hOrRVfLNWblRdfalse2024-02-01T10:25:33.494Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9CJ3w1EusK8OST2k17sTdTM.cuh3E3efalse2024-02-01T10:25:33.349Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjVwfx4ETdAtTkKn2rTCQT4onqB6sYulvfalse2024-02-01T10:25:33.381Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjiGO2QyMOL81IaF-H4sHDqyECPlnZBmEfalse2024-02-01T10:25:33.323Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjcYHYWFiJtzK2FpqgaUq4bR.SX5MaCkIfalse2024-02-01T10:25:33.206Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjrasya8taOZsGTj9rDi7keghf.VnPFlFfalse2024-02-01T10:25:33.291Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjy-DPCvngcj9q7W-lUL16Jgbp.NhV7NEfalse2024-02-01T10:25:33.257Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj0KR6m6ymTGitKe6vabQyVENAnBL669wfalse2024-02-01T10:25:33.118Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2Hfm7x.sO5cadpLfjJNuhN.TAkjS54Efalse2024-02-01T10:25:33.093Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6false2024-02-01T10:25:33.150Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHXfalse2024-02-01T10:25:33.030Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj63CC4agggciY3jdYETzlNEHLx15uh8wfalse2024-02-01T10:25:33.063Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjYzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1ofalse2024-02-01T10:25:32.998Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjB5jd89.SLRVLOAPcXqackukLvQnZY1Bfalse2024-02-01T10:25:32.872Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjrcBw9NDMcjurzSdpB.bvwX4JmUDkwrQfalse2024-02-01T10:25:32.964Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjs-qxjv7TkMdxeC61HL2kxUqODSwa-1Ifalse2024-02-01T10:25:32.926Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:35.003 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-02-01T10:27:35.003 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:35.003 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:35.003 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:35.003 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:35.003 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:35.003 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:35.003 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:35.003 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:35.004 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:35.004 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:35.004 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:35.004 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:35.004 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:35.004 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:35.004 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:35.004 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:35.004 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:35.004 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:35.004 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:35.004 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:35.005 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:35.005 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:35.005 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:35.005 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:35.005 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:35.005 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:35.005 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:35.005 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:35.005 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:35.005 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:35.005 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:35.005 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:35.006 DEBUG:boto:path=/vclfcn-19/?versions&max-keys=999 2024-02-01T10:27:35.006 DEBUG:boto:auth_path=/vclfcn-19/?versions&max-keys=999 2024-02-01T10:27:35.006 DEBUG:boto:Method: GET 2024-02-01T10:27:35.006 DEBUG:boto:Path: /vclfcn-19/?versions&max-keys=999 2024-02-01T10:27:35.006 DEBUG:boto:Data: 2024-02-01T10:27:35.006 DEBUG:boto:Headers: {} 2024-02-01T10:27:35.006 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:35.006 DEBUG:boto:Port: 8000 2024-02-01T10:27:35.006 DEBUG:boto:Params: {} 2024-02-01T10:27:35.006 DEBUG:boto:Token: None 2024-02-01T10:27:35.007 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:35 GMT /vclfcn-19/?versions 2024-02-01T10:27:35.007 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:aeyVUm7LE33w+5131jVIoVxe1Ww= 2024-02-01T10:27:35.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:aeyVUm7LE33w+5131jVIoVxe1Ww=', 'Content-Length': '0'} 2024-02-01T10:27:35.011 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d67cda6663dd01a0-0065bb7217-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:35.011 DEBUG:boto:b'vclfcn-19999falsedummyg5UScztMjPBIYZ2b5yOn-qsm.eQTcnutrue2024-02-01T10:23:11.115Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23llUchmWTvJKOzZpTkl59nzz.uMOQbtrue2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjRrytQti15jy37pft0WxtgWdt4Zs7l4Efalse2024-02-01T10:25:33.973Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjfzdlI7n.dsuwfAOSBNyk84Wd7l5p70ufalse2024-02-01T10:25:33.886Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUOa9NAxhXbH7RR3jkzlpXPh.fGAfBJcfalse2024-02-01T10:25:33.774Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjeNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr.false2024-02-01T10:25:33.746Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtGIsEYvB.VyZPdq9CgtD6kEeP4rzbDNfalse2024-02-01T10:25:33.830Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2lRbqumcCYnVWCIz20mU455245nXaz8false2024-02-01T10:25:33.719Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjoGhdnHMtrLPTgFgXlKfl.UXdmrUnL2rfalse2024-02-01T10:25:33.687Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjqex3Bh72IV4plxG2V2Lm7bvqwXqQNK9false2024-02-01T10:25:33.661Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjNCPEvRYQDJpLrTejJRqmTf2j0FWCBCSfalse2024-02-01T10:25:33.599Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjVQT2kBCkF07dugmyrhNeRpQpR0HK7bTfalse2024-02-01T10:25:33.630Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtvvjWUbgZIRYhHk9r.liT6sPCWCs2Zifalse2024-02-01T10:25:33.551Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3OSaU40E5-Hgv6pEDB8btq8uZcAFHVzfalse2024-02-01T10:25:33.438Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjK23uwsLWGmQDGYGsdu-h04Fk2wApB0Ofalse2024-02-01T10:25:33.410Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjst2TOT.K7IOe-wFfL.hOrRVfLNWblRdfalse2024-02-01T10:25:33.494Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9CJ3w1EusK8OST2k17sTdTM.cuh3E3efalse2024-02-01T10:25:33.349Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjVwfx4ETdAtTkKn2rTCQT4onqB6sYulvfalse2024-02-01T10:25:33.381Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjiGO2QyMOL81IaF-H4sHDqyECPlnZBmEfalse2024-02-01T10:25:33.323Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjcYHYWFiJtzK2FpqgaUq4bR.SX5MaCkIfalse2024-02-01T10:25:33.206Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjrasya8taOZsGTj9rDi7keghf.VnPFlFfalse2024-02-01T10:25:33.291Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjy-DPCvngcj9q7W-lUL16Jgbp.NhV7NEfalse2024-02-01T10:25:33.257Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj0KR6m6ymTGitKe6vabQyVENAnBL669wfalse2024-02-01T10:25:33.118Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2Hfm7x.sO5cadpLfjJNuhN.TAkjS54Efalse2024-02-01T10:25:33.093Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6false2024-02-01T10:25:33.150Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHXfalse2024-02-01T10:25:33.030Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj63CC4agggciY3jdYETzlNEHLx15uh8wfalse2024-02-01T10:25:33.063Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjYzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1ofalse2024-02-01T10:25:32.998Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjB5jd89.SLRVLOAPcXqackukLvQnZY1Bfalse2024-02-01T10:25:32.872Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjrcBw9NDMcjurzSdpB.bvwX4JmUDkwrQfalse2024-02-01T10:25:32.964Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjs-qxjv7TkMdxeC61HL2kxUqODSwa-1Ifalse2024-02-01T10:25:32.926Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:35.013 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:35.013 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:35.013 DEBUG:boto:path=/vclfcn-19/?versions&max-keys=999 2024-02-01T10:27:35.013 DEBUG:boto:auth_path=/vclfcn-19/?versions&max-keys=999 2024-02-01T10:27:35.013 DEBUG:boto:Method: GET 2024-02-01T10:27:35.014 DEBUG:boto:Path: /vclfcn-19/?versions&max-keys=999 2024-02-01T10:27:35.014 DEBUG:boto:Data: 2024-02-01T10:27:35.014 DEBUG:boto:Headers: {} 2024-02-01T10:27:35.014 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:35.014 DEBUG:boto:Port: 8000 2024-02-01T10:27:35.014 DEBUG:boto:Params: {} 2024-02-01T10:27:35.014 DEBUG:boto:Token: None 2024-02-01T10:27:35.014 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:35 GMT /vclfcn-19/?versions 2024-02-01T10:27:35.014 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:aeyVUm7LE33w+5131jVIoVxe1Ww= 2024-02-01T10:27:35.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:aeyVUm7LE33w+5131jVIoVxe1Ww=', 'Content-Length': '0'} 2024-02-01T10:27:35.021 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c293a10a4a1802f6-0065bb7217-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:35.022 DEBUG:boto:b'vclfcn-19999falsedummyg5UScztMjPBIYZ2b5yOn-qsm.eQTcnutrue2024-02-01T10:23:11.115Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23llUchmWTvJKOzZpTkl59nzz.uMOQbtrue2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjRrytQti15jy37pft0WxtgWdt4Zs7l4Efalse2024-02-01T10:25:33.973Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjfzdlI7n.dsuwfAOSBNyk84Wd7l5p70ufalse2024-02-01T10:25:33.886Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUOa9NAxhXbH7RR3jkzlpXPh.fGAfBJcfalse2024-02-01T10:25:33.774Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjeNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr.false2024-02-01T10:25:33.746Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtGIsEYvB.VyZPdq9CgtD6kEeP4rzbDNfalse2024-02-01T10:25:33.830Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2lRbqumcCYnVWCIz20mU455245nXaz8false2024-02-01T10:25:33.719Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjoGhdnHMtrLPTgFgXlKfl.UXdmrUnL2rfalse2024-02-01T10:25:33.687Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjqex3Bh72IV4plxG2V2Lm7bvqwXqQNK9false2024-02-01T10:25:33.661Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjNCPEvRYQDJpLrTejJRqmTf2j0FWCBCSfalse2024-02-01T10:25:33.599Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjVQT2kBCkF07dugmyrhNeRpQpR0HK7bTfalse2024-02-01T10:25:33.630Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtvvjWUbgZIRYhHk9r.liT6sPCWCs2Zifalse2024-02-01T10:25:33.551Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3OSaU40E5-Hgv6pEDB8btq8uZcAFHVzfalse2024-02-01T10:25:33.438Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjK23uwsLWGmQDGYGsdu-h04Fk2wApB0Ofalse2024-02-01T10:25:33.410Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjst2TOT.K7IOe-wFfL.hOrRVfLNWblRdfalse2024-02-01T10:25:33.494Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9CJ3w1EusK8OST2k17sTdTM.cuh3E3efalse2024-02-01T10:25:33.349Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjVwfx4ETdAtTkKn2rTCQT4onqB6sYulvfalse2024-02-01T10:25:33.381Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjiGO2QyMOL81IaF-H4sHDqyECPlnZBmEfalse2024-02-01T10:25:33.323Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjcYHYWFiJtzK2FpqgaUq4bR.SX5MaCkIfalse2024-02-01T10:25:33.206Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjrasya8taOZsGTj9rDi7keghf.VnPFlFfalse2024-02-01T10:25:33.291Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjy-DPCvngcj9q7W-lUL16Jgbp.NhV7NEfalse2024-02-01T10:25:33.257Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj0KR6m6ymTGitKe6vabQyVENAnBL669wfalse2024-02-01T10:25:33.118Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2Hfm7x.sO5cadpLfjJNuhN.TAkjS54Efalse2024-02-01T10:25:33.093Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6false2024-02-01T10:25:33.150Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHXfalse2024-02-01T10:25:33.030Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj63CC4agggciY3jdYETzlNEHLx15uh8wfalse2024-02-01T10:25:33.063Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjYzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1ofalse2024-02-01T10:25:32.998Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjB5jd89.SLRVLOAPcXqackukLvQnZY1Bfalse2024-02-01T10:25:32.872Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjrcBw9NDMcjurzSdpB.bvwX4JmUDkwrQfalse2024-02-01T10:25:32.964Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjs-qxjv7TkMdxeC61HL2kxUqODSwa-1Ifalse2024-02-01T10:25:32.926Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:35.023 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-02-01T10:27:35.024 DEBUG:boto:path=/vclfcn-19/dummy 2024-02-01T10:27:35.024 DEBUG:boto:auth_path=/vclfcn-19/dummy 2024-02-01T10:27:35.024 DEBUG:boto:path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:35.024 DEBUG:boto:auth_path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:35.024 DEBUG:boto:Method: GET 2024-02-01T10:27:35.024 DEBUG:boto:Path: /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:35.024 DEBUG:boto:Data: 2024-02-01T10:27:35.025 DEBUG:boto:Headers: {} 2024-02-01T10:27:35.025 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:35.025 DEBUG:boto:Port: 8000 2024-02-01T10:27:35.025 DEBUG:boto:Params: {} 2024-02-01T10:27:35.025 DEBUG:boto:Token: None 2024-02-01T10:27:35.025 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:35 GMT /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:35.025 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:r4K4guKpBm5Xx9XQE1pyRWkElhY= 2024-02-01T10:27:35.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:r4K4guKpBm5Xx9XQE1pyRWkElhY=', 'Content-Length': '0'} 2024-02-01T10:27:35.028 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:23:11 GMT'), ('x-amz-version-id', 'g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b2b70510d081ee6b-0065bb7217-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:35.028 DEBUG:boto:path=/vclfcn-19/dummy 2024-02-01T10:27:35.028 DEBUG:boto:auth_path=/vclfcn-19/dummy 2024-02-01T10:27:35.028 DEBUG:boto:path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:35.028 DEBUG:boto:auth_path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:35.028 DEBUG:boto:Method: GET 2024-02-01T10:27:35.029 DEBUG:boto:Path: /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:35.029 DEBUG:boto:Data: 2024-02-01T10:27:35.029 DEBUG:boto:Headers: {} 2024-02-01T10:27:35.029 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:35.029 DEBUG:boto:Port: 8000 2024-02-01T10:27:35.029 DEBUG:boto:Params: {} 2024-02-01T10:27:35.029 DEBUG:boto:Token: None 2024-02-01T10:27:35.029 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:35 GMT /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:35.029 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:r4K4guKpBm5Xx9XQE1pyRWkElhY= 2024-02-01T10:27:35.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:r4K4guKpBm5Xx9XQE1pyRWkElhY=', 'Content-Length': '0'} 2024-02-01T10:27:35.032 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:23:11 GMT'), ('x-amz-version-id', 'g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000013b4aa77a574c2e7-0065bb7217-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:35.032 DEBUG:boto:path=/vclfcn-19/dummy 2024-02-01T10:27:35.032 DEBUG:boto:auth_path=/vclfcn-19/dummy 2024-02-01T10:27:35.033 DEBUG:boto:path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:35.033 DEBUG:boto:auth_path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:35.033 DEBUG:boto:Method: HEAD 2024-02-01T10:27:35.033 DEBUG:boto:Path: /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:35.033 DEBUG:boto:Data: 2024-02-01T10:27:35.033 DEBUG:boto:Headers: {} 2024-02-01T10:27:35.033 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:35.033 DEBUG:boto:Port: 8000 2024-02-01T10:27:35.033 DEBUG:boto:Params: {} 2024-02-01T10:27:35.033 DEBUG:boto:Token: None 2024-02-01T10:27:35.034 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:35 GMT /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:35.034 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:MktqcSLIjweBiaizO77PldgOXPs= 2024-02-01T10:27:35.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:MktqcSLIjweBiaizO77PldgOXPs=', 'Content-Length': '0'} 2024-02-01T10:27:35.036 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:23:11 GMT'), ('x-amz-version-id', 'g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008f16ded610a64a2c-0065bb7217-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:35.036 DEBUG:boto:path=/vclfcn-19/dummy 2024-02-01T10:27:35.036 DEBUG:boto:auth_path=/vclfcn-19/dummy 2024-02-01T10:27:35.037 DEBUG:boto:path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:35.037 DEBUG:boto:auth_path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:35.037 DEBUG:boto:Method: HEAD 2024-02-01T10:27:35.037 DEBUG:boto:Path: /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:35.037 DEBUG:boto:Data: 2024-02-01T10:27:35.037 DEBUG:boto:Headers: {} 2024-02-01T10:27:35.037 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:35.037 DEBUG:boto:Port: 8000 2024-02-01T10:27:35.037 DEBUG:boto:Params: {} 2024-02-01T10:27:35.037 DEBUG:boto:Token: None 2024-02-01T10:27:35.038 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:35 GMT /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:35.038 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:MktqcSLIjweBiaizO77PldgOXPs= 2024-02-01T10:27:35.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:MktqcSLIjweBiaizO77PldgOXPs=', 'Content-Length': '0'} 2024-02-01T10:27:35.040 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:23:11 GMT'), ('x-amz-version-id', 'g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001a446db49bebb1ea-0065bb7217-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:35.041 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-02-01T10:27:35.041 DEBUG:boto:path=/vclfcn-19/dummy 2024-02-01T10:27:35.041 DEBUG:boto:auth_path=/vclfcn-19/dummy 2024-02-01T10:27:35.041 DEBUG:boto:path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:35.041 DEBUG:boto:auth_path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:35.041 DEBUG:boto:Method: GET 2024-02-01T10:27:35.041 DEBUG:boto:Path: /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:35.041 DEBUG:boto:Data: 2024-02-01T10:27:35.042 DEBUG:boto:Headers: {} 2024-02-01T10:27:35.042 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:35.042 DEBUG:boto:Port: 8000 2024-02-01T10:27:35.042 DEBUG:boto:Params: {} 2024-02-01T10:27:35.042 DEBUG:boto:Token: None 2024-02-01T10:27:35.042 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:35 GMT /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:35.042 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:r4K4guKpBm5Xx9XQE1pyRWkElhY= 2024-02-01T10:27:35.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:r4K4guKpBm5Xx9XQE1pyRWkElhY=', 'Content-Length': '0'} 2024-02-01T10:27:35.044 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:23:11 GMT'), ('x-amz-version-id', 'g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000dd49a96fa902bdd9-0065bb7217-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:35.045 DEBUG:boto:path=/vclfcn-19/dummy 2024-02-01T10:27:35.045 DEBUG:boto:auth_path=/vclfcn-19/dummy 2024-02-01T10:27:35.045 DEBUG:boto:path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:35.045 DEBUG:boto:auth_path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:35.045 DEBUG:boto:Method: GET 2024-02-01T10:27:35.045 DEBUG:boto:Path: /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:35.045 DEBUG:boto:Data: 2024-02-01T10:27:35.045 DEBUG:boto:Headers: {} 2024-02-01T10:27:35.045 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:35.045 DEBUG:boto:Port: 8000 2024-02-01T10:27:35.045 DEBUG:boto:Params: {} 2024-02-01T10:27:35.046 DEBUG:boto:Token: None 2024-02-01T10:27:35.046 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:35 GMT /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:35.046 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:r4K4guKpBm5Xx9XQE1pyRWkElhY= 2024-02-01T10:27:35.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:r4K4guKpBm5Xx9XQE1pyRWkElhY=', 'Content-Length': '0'} 2024-02-01T10:27:35.049 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:23:11 GMT'), ('x-amz-version-id', 'g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ecdf22c0a74c523f-0065bb7217-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:35.049 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:35.049 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:35.050 DEBUG:boto:path=/vclfcn-19/?prefix=dummy 2024-02-01T10:27:35.050 DEBUG:boto:auth_path=/vclfcn-19/?prefix=dummy 2024-02-01T10:27:35.050 DEBUG:boto:Method: GET 2024-02-01T10:27:35.050 DEBUG:boto:Path: /vclfcn-19/?prefix=dummy 2024-02-01T10:27:35.050 DEBUG:boto:Data: 2024-02-01T10:27:35.050 DEBUG:boto:Headers: {} 2024-02-01T10:27:35.050 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:35.050 DEBUG:boto:Port: 8000 2024-02-01T10:27:35.050 DEBUG:boto:Params: {} 2024-02-01T10:27:35.050 DEBUG:boto:Token: None 2024-02-01T10:27:35.051 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:35 GMT /vclfcn-19/ 2024-02-01T10:27:35.051 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ztaD4sEmE1szOBpCcy0qt3bA0CY= 2024-02-01T10:27:35.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ztaD4sEmE1szOBpCcy0qt3bA0CY=', 'Content-Length': '0'} 2024-02-01T10:27:35.054 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3a7a20fb665ce18-0065bb7217-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:35.096 DEBUG:boto:b'vclfcn-19dummy1000falsedummy2024-02-01T10:23:11.115Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:35.097 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:35.097 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:35.097 DEBUG:boto:path=/vclfcn-19/?prefix=dummy 2024-02-01T10:27:35.097 DEBUG:boto:auth_path=/vclfcn-19/?prefix=dummy 2024-02-01T10:27:35.097 DEBUG:boto:Method: GET 2024-02-01T10:27:35.097 DEBUG:boto:Path: /vclfcn-19/?prefix=dummy 2024-02-01T10:27:35.097 DEBUG:boto:Data: 2024-02-01T10:27:35.097 DEBUG:boto:Headers: {} 2024-02-01T10:27:35.097 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:35.097 DEBUG:boto:Port: 8000 2024-02-01T10:27:35.098 DEBUG:boto:Params: {} 2024-02-01T10:27:35.098 DEBUG:boto:Token: None 2024-02-01T10:27:35.098 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:35 GMT /vclfcn-19/ 2024-02-01T10:27:35.098 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ztaD4sEmE1szOBpCcy0qt3bA0CY= 2024-02-01T10:27:35.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ztaD4sEmE1szOBpCcy0qt3bA0CY=', 'Content-Length': '0'} 2024-02-01T10:27:35.103 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000256794b6510ac845-0065bb7217-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:35.103 DEBUG:boto:b'vclfcn-19dummy1000falsedummy2024-02-01T10:23:11.115Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:35.104 DEBUG:boto:path=/vclfcn-19/dummy 2024-02-01T10:27:35.104 DEBUG:boto:auth_path=/vclfcn-19/dummy 2024-02-01T10:27:35.104 DEBUG:boto:Method: HEAD 2024-02-01T10:27:35.104 DEBUG:boto:Path: /vclfcn-19/dummy 2024-02-01T10:27:35.104 DEBUG:boto:Data: 2024-02-01T10:27:35.104 DEBUG:boto:Headers: {} 2024-02-01T10:27:35.104 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:35.104 DEBUG:boto:Port: 8000 2024-02-01T10:27:35.104 DEBUG:boto:Params: {} 2024-02-01T10:27:35.104 DEBUG:boto:Token: None 2024-02-01T10:27:35.105 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:35 GMT /vclfcn-19/dummy 2024-02-01T10:27:35.105 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ikuZKAlmTSBeNQWqvt+YOFl4DnM= 2024-02-01T10:27:35.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ikuZKAlmTSBeNQWqvt+YOFl4DnM=', 'Content-Length': '0'} 2024-02-01T10:27:35.108 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:23:11 GMT'), ('x-amz-version-id', 'g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f7ddc9b7245905ed-0065bb7217-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:35.109 DEBUG:boto:path=/vclfcn-19/dummy 2024-02-01T10:27:35.109 DEBUG:boto:auth_path=/vclfcn-19/dummy 2024-02-01T10:27:35.109 DEBUG:boto:Method: HEAD 2024-02-01T10:27:35.109 DEBUG:boto:Path: /vclfcn-19/dummy 2024-02-01T10:27:35.109 DEBUG:boto:Data: 2024-02-01T10:27:35.109 DEBUG:boto:Headers: {} 2024-02-01T10:27:35.109 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:35.109 DEBUG:boto:Port: 8000 2024-02-01T10:27:35.109 DEBUG:boto:Params: {} 2024-02-01T10:27:35.110 DEBUG:boto:Token: None 2024-02-01T10:27:35.110 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:35 GMT /vclfcn-19/dummy 2024-02-01T10:27:35.110 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ikuZKAlmTSBeNQWqvt+YOFl4DnM= 2024-02-01T10:27:35.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ikuZKAlmTSBeNQWqvt+YOFl4DnM=', 'Content-Length': '0'} 2024-02-01T10:27:35.114 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:23:11 GMT'), ('x-amz-version-id', 'g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d4f80c43fdfdc298-0065bb7217-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:35.114 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-02-01T10:27:35.114 DEBUG:boto:path=/vclfcn-19/dummy 2024-02-01T10:27:35.114 DEBUG:boto:auth_path=/vclfcn-19/dummy 2024-02-01T10:27:35.114 DEBUG:boto:path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:35.114 DEBUG:boto:auth_path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:35.114 DEBUG:boto:Method: GET 2024-02-01T10:27:35.115 DEBUG:boto:Path: /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:35.115 DEBUG:boto:Data: 2024-02-01T10:27:35.115 DEBUG:boto:Headers: {} 2024-02-01T10:27:35.115 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:35.115 DEBUG:boto:Port: 8000 2024-02-01T10:27:35.115 DEBUG:boto:Params: {} 2024-02-01T10:27:35.115 DEBUG:boto:Token: None 2024-02-01T10:27:35.115 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:35 GMT /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:35.115 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:r4K4guKpBm5Xx9XQE1pyRWkElhY= 2024-02-01T10:27:35.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:r4K4guKpBm5Xx9XQE1pyRWkElhY=', 'Content-Length': '0'} 2024-02-01T10:27:35.118 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:23:11 GMT'), ('x-amz-version-id', 'g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c73aefb23fb4bc4c-0065bb7217-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:35.118 DEBUG:boto:path=/vclfcn-19/dummy 2024-02-01T10:27:35.118 DEBUG:boto:auth_path=/vclfcn-19/dummy 2024-02-01T10:27:35.118 DEBUG:boto:path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:35.119 DEBUG:boto:auth_path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:35.119 DEBUG:boto:Method: GET 2024-02-01T10:27:35.119 DEBUG:boto:Path: /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:35.119 DEBUG:boto:Data: 2024-02-01T10:27:35.119 DEBUG:boto:Headers: {} 2024-02-01T10:27:35.119 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:35.119 DEBUG:boto:Port: 8000 2024-02-01T10:27:35.119 DEBUG:boto:Params: {} 2024-02-01T10:27:35.119 DEBUG:boto:Token: None 2024-02-01T10:27:35.119 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:35 GMT /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:35.119 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:r4K4guKpBm5Xx9XQE1pyRWkElhY= 2024-02-01T10:27:35.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:r4K4guKpBm5Xx9XQE1pyRWkElhY=', 'Content-Length': '0'} 2024-02-01T10:27:35.122 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:23:11 GMT'), ('x-amz-version-id', 'g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e7aa1ce783fa912e-0065bb7217-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:35.123 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:35.123 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:35.123 DEBUG:boto:path=/vclfcn-19/?prefix=dummy 2024-02-01T10:27:35.123 DEBUG:boto:auth_path=/vclfcn-19/?prefix=dummy 2024-02-01T10:27:35.123 DEBUG:boto:Method: GET 2024-02-01T10:27:35.123 DEBUG:boto:Path: /vclfcn-19/?prefix=dummy 2024-02-01T10:27:35.123 DEBUG:boto:Data: 2024-02-01T10:27:35.123 DEBUG:boto:Headers: {} 2024-02-01T10:27:35.123 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:35.123 DEBUG:boto:Port: 8000 2024-02-01T10:27:35.123 DEBUG:boto:Params: {} 2024-02-01T10:27:35.124 DEBUG:boto:Token: None 2024-02-01T10:27:35.124 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:35 GMT /vclfcn-19/ 2024-02-01T10:27:35.124 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ztaD4sEmE1szOBpCcy0qt3bA0CY= 2024-02-01T10:27:35.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ztaD4sEmE1szOBpCcy0qt3bA0CY=', 'Content-Length': '0'} 2024-02-01T10:27:35.128 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dcc788186c555388-0065bb7217-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:35.128 DEBUG:boto:b'vclfcn-19dummy1000falsedummy2024-02-01T10:23:11.115Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:35.128 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:35.129 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:35.129 DEBUG:boto:path=/vclfcn-19/?prefix=dummy 2024-02-01T10:27:35.129 DEBUG:boto:auth_path=/vclfcn-19/?prefix=dummy 2024-02-01T10:27:35.129 DEBUG:boto:Method: GET 2024-02-01T10:27:35.129 DEBUG:boto:Path: /vclfcn-19/?prefix=dummy 2024-02-01T10:27:35.129 DEBUG:boto:Data: 2024-02-01T10:27:35.129 DEBUG:boto:Headers: {} 2024-02-01T10:27:35.129 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:35.129 DEBUG:boto:Port: 8000 2024-02-01T10:27:35.129 DEBUG:boto:Params: {} 2024-02-01T10:27:35.129 DEBUG:boto:Token: None 2024-02-01T10:27:35.130 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:35 GMT /vclfcn-19/ 2024-02-01T10:27:35.130 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ztaD4sEmE1szOBpCcy0qt3bA0CY= 2024-02-01T10:27:35.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ztaD4sEmE1szOBpCcy0qt3bA0CY=', 'Content-Length': '0'} 2024-02-01T10:27:35.134 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000698538732c66deb8-0065bb7217-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:35.134 DEBUG:boto:b'vclfcn-19dummy1000falsedummy2024-02-01T10:23:11.115Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:35.135 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:35.135 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:35.135 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:35.135 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:35.135 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:35.135 DEBUG:boto:Method: GET 2024-02-01T10:27:35.135 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:35.135 DEBUG:boto:Data: 2024-02-01T10:27:35.135 DEBUG:boto:Headers: {} 2024-02-01T10:27:35.135 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:35.136 DEBUG:boto:Port: 8000 2024-02-01T10:27:35.136 DEBUG:boto:Params: {} 2024-02-01T10:27:35.136 DEBUG:boto:Token: None 2024-02-01T10:27:35.136 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:35 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:35.136 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:CgiuP0q9dYuwGEuNSmNHt0623gE= 2024-02-01T10:27:35.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:CgiuP0q9dYuwGEuNSmNHt0623gE=', 'Content-Length': '0'} 2024-02-01T10:27:35.139 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000039a9775a6312dcf-0065bb7217-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:35.180 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:35.180 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:35.180 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:35.181 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:35.181 DEBUG:boto:Method: GET 2024-02-01T10:27:35.181 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:35.181 DEBUG:boto:Data: 2024-02-01T10:27:35.181 DEBUG:boto:Headers: {} 2024-02-01T10:27:35.181 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:35.181 DEBUG:boto:Port: 8000 2024-02-01T10:27:35.181 DEBUG:boto:Params: {} 2024-02-01T10:27:35.181 DEBUG:boto:Token: None 2024-02-01T10:27:35.181 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:35 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:35.182 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:CgiuP0q9dYuwGEuNSmNHt0623gE= 2024-02-01T10:27:35.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:CgiuP0q9dYuwGEuNSmNHt0623gE=', 'Content-Length': '0'} 2024-02-01T10:27:35.185 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c0f034a7b064454f-0065bb7217-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:35.185 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:35.185 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:35.185 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:35.185 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:35.185 DEBUG:boto:Method: HEAD 2024-02-01T10:27:35.185 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:35.186 DEBUG:boto:Data: 2024-02-01T10:27:35.186 DEBUG:boto:Headers: {} 2024-02-01T10:27:35.186 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:35.186 DEBUG:boto:Port: 8000 2024-02-01T10:27:35.186 DEBUG:boto:Params: {} 2024-02-01T10:27:35.186 DEBUG:boto:Token: None 2024-02-01T10:27:35.186 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:35 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:35.186 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:3B5MThoz8zlLZgn1xZulzdJMV2E= 2024-02-01T10:27:35.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:3B5MThoz8zlLZgn1xZulzdJMV2E=', 'Content-Length': '0'} 2024-02-01T10:27:35.189 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b576b07a30a07a3c-0065bb7217-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:35.189 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:35.189 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:35.190 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:35.190 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:35.190 DEBUG:boto:Method: HEAD 2024-02-01T10:27:35.190 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:35.190 DEBUG:boto:Data: 2024-02-01T10:27:35.190 DEBUG:boto:Headers: {} 2024-02-01T10:27:35.190 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:35.190 DEBUG:boto:Port: 8000 2024-02-01T10:27:35.190 DEBUG:boto:Params: {} 2024-02-01T10:27:35.190 DEBUG:boto:Token: None 2024-02-01T10:27:35.191 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:35 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:35.191 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:3B5MThoz8zlLZgn1xZulzdJMV2E= 2024-02-01T10:27:35.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:3B5MThoz8zlLZgn1xZulzdJMV2E=', 'Content-Length': '0'} 2024-02-01T10:27:35.193 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000034d3d1c82994b54b-0065bb7217-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:35.194 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:35.194 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:35.194 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:35.194 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:35.194 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:35.194 DEBUG:boto:Method: GET 2024-02-01T10:27:35.194 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:35.194 DEBUG:boto:Data: 2024-02-01T10:27:35.195 DEBUG:boto:Headers: {} 2024-02-01T10:27:35.195 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:35.195 DEBUG:boto:Port: 8000 2024-02-01T10:27:35.195 DEBUG:boto:Params: {} 2024-02-01T10:27:35.195 DEBUG:boto:Token: None 2024-02-01T10:27:35.195 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:35 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:35.195 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:CgiuP0q9dYuwGEuNSmNHt0623gE= 2024-02-01T10:27:35.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:CgiuP0q9dYuwGEuNSmNHt0623gE=', 'Content-Length': '0'} 2024-02-01T10:27:35.198 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c23565a5f6766a08-0065bb7217-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:35.199 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:35.199 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:35.199 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:35.199 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:35.199 DEBUG:boto:Method: GET 2024-02-01T10:27:35.199 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:35.199 DEBUG:boto:Data: 2024-02-01T10:27:35.199 DEBUG:boto:Headers: {} 2024-02-01T10:27:35.199 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:35.199 DEBUG:boto:Port: 8000 2024-02-01T10:27:35.200 DEBUG:boto:Params: {} 2024-02-01T10:27:35.200 DEBUG:boto:Token: None 2024-02-01T10:27:35.200 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:35 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:35.200 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:CgiuP0q9dYuwGEuNSmNHt0623gE= 2024-02-01T10:27:35.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:CgiuP0q9dYuwGEuNSmNHt0623gE=', 'Content-Length': '0'} 2024-02-01T10:27:35.203 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009d5da51fe8938835-0065bb7217-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:35.244 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:35.244 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:35.244 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:35.245 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:35.245 DEBUG:boto:Method: GET 2024-02-01T10:27:35.245 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:35.245 DEBUG:boto:Data: 2024-02-01T10:27:35.245 DEBUG:boto:Headers: {} 2024-02-01T10:27:35.245 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:35.245 DEBUG:boto:Port: 8000 2024-02-01T10:27:35.245 DEBUG:boto:Params: {} 2024-02-01T10:27:35.246 DEBUG:boto:Token: None 2024-02-01T10:27:35.246 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:35 GMT /vclfcn-19/ 2024-02-01T10:27:35.246 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ztaD4sEmE1szOBpCcy0qt3bA0CY= 2024-02-01T10:27:35.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ztaD4sEmE1szOBpCcy0qt3bA0CY=', 'Content-Length': '0'} 2024-02-01T10:27:35.252 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052be51194e84d957-0065bb7217-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:35.252 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:35.253 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:35.253 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:35.253 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:35.253 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:35.253 DEBUG:boto:Method: GET 2024-02-01T10:27:35.253 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:35.253 DEBUG:boto:Data: 2024-02-01T10:27:35.253 DEBUG:boto:Headers: {} 2024-02-01T10:27:35.253 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:35.254 DEBUG:boto:Port: 8000 2024-02-01T10:27:35.254 DEBUG:boto:Params: {} 2024-02-01T10:27:35.254 DEBUG:boto:Token: None 2024-02-01T10:27:35.254 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:35 GMT /vclfcn-19/ 2024-02-01T10:27:35.254 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ztaD4sEmE1szOBpCcy0qt3bA0CY= 2024-02-01T10:27:35.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ztaD4sEmE1szOBpCcy0qt3bA0CY=', 'Content-Length': '0'} 2024-02-01T10:27:35.259 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b62a09a42b28e6f-0065bb7217-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:35.259 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:35.260 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:35.260 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:35.260 DEBUG:boto:Method: HEAD 2024-02-01T10:27:35.260 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:35.260 DEBUG:boto:Data: 2024-02-01T10:27:35.260 DEBUG:boto:Headers: {} 2024-02-01T10:27:35.260 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:35.261 DEBUG:boto:Port: 8000 2024-02-01T10:27:35.261 DEBUG:boto:Params: {} 2024-02-01T10:27:35.261 DEBUG:boto:Token: None 2024-02-01T10:27:35.261 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:35 GMT /vclfcn-19/obj 2024-02-01T10:27:35.261 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ytaCXXHekoXVGmmpyC1lTc8j2LI= 2024-02-01T10:27:35.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ytaCXXHekoXVGmmpyC1lTc8j2LI=', 'Content-Length': '0'} 2024-02-01T10:27:35.267 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000004e4cc687ebcd660-0065bb7217-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:35.267 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:35.267 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:35.267 DEBUG:boto:Method: HEAD 2024-02-01T10:27:35.267 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:35.267 DEBUG:boto:Data: 2024-02-01T10:27:35.268 DEBUG:boto:Headers: {} 2024-02-01T10:27:35.268 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:35.268 DEBUG:boto:Port: 8000 2024-02-01T10:27:35.268 DEBUG:boto:Params: {} 2024-02-01T10:27:35.268 DEBUG:boto:Token: None 2024-02-01T10:27:35.268 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:35 GMT /vclfcn-19/obj 2024-02-01T10:27:35.268 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ytaCXXHekoXVGmmpyC1lTc8j2LI= 2024-02-01T10:27:35.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ytaCXXHekoXVGmmpyC1lTc8j2LI=', 'Content-Length': '0'} 2024-02-01T10:27:35.274 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000abaed0ff157da400-0065bb7217-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:35.275 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:35.275 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:35.275 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:35.275 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:35.275 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:35.275 DEBUG:boto:Method: GET 2024-02-01T10:27:35.275 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:35.275 DEBUG:boto:Data: 2024-02-01T10:27:35.275 DEBUG:boto:Headers: {} 2024-02-01T10:27:35.275 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:35.276 DEBUG:boto:Port: 8000 2024-02-01T10:27:35.276 DEBUG:boto:Params: {} 2024-02-01T10:27:35.276 DEBUG:boto:Token: None 2024-02-01T10:27:35.276 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:35 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:35.276 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:CgiuP0q9dYuwGEuNSmNHt0623gE= 2024-02-01T10:27:35.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:CgiuP0q9dYuwGEuNSmNHt0623gE=', 'Content-Length': '0'} 2024-02-01T10:27:35.279 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000efdc02f66b20c6ec-0065bb7217-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:35.324 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:35.324 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:35.324 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:35.324 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:35.325 DEBUG:boto:Method: GET 2024-02-01T10:27:35.325 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:35.325 DEBUG:boto:Data: 2024-02-01T10:27:35.325 DEBUG:boto:Headers: {} 2024-02-01T10:27:35.325 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:35.325 DEBUG:boto:Port: 8000 2024-02-01T10:27:35.325 DEBUG:boto:Params: {} 2024-02-01T10:27:35.325 DEBUG:boto:Token: None 2024-02-01T10:27:35.325 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:35 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:35.326 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:CgiuP0q9dYuwGEuNSmNHt0623gE= 2024-02-01T10:27:35.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:CgiuP0q9dYuwGEuNSmNHt0623gE=', 'Content-Length': '0'} 2024-02-01T10:27:35.328 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009ada636784011ccc-0065bb7217-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:35.372 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:35.372 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:35.372 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:35.372 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:35.373 DEBUG:boto:Method: GET 2024-02-01T10:27:35.373 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:35.373 DEBUG:boto:Data: 2024-02-01T10:27:35.373 DEBUG:boto:Headers: {} 2024-02-01T10:27:35.373 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:35.373 DEBUG:boto:Port: 8000 2024-02-01T10:27:35.373 DEBUG:boto:Params: {} 2024-02-01T10:27:35.373 DEBUG:boto:Token: None 2024-02-01T10:27:35.373 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:35 GMT /vclfcn-19/ 2024-02-01T10:27:35.374 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ztaD4sEmE1szOBpCcy0qt3bA0CY= 2024-02-01T10:27:35.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ztaD4sEmE1szOBpCcy0qt3bA0CY=', 'Content-Length': '0'} 2024-02-01T10:27:35.379 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e286198c99da029a-0065bb7217-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:35.379 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:35.380 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:35.380 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:35.380 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:35.380 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:35.380 DEBUG:boto:Method: GET 2024-02-01T10:27:35.380 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:35.380 DEBUG:boto:Data: 2024-02-01T10:27:35.380 DEBUG:boto:Headers: {} 2024-02-01T10:27:35.380 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:35.380 DEBUG:boto:Port: 8000 2024-02-01T10:27:35.381 DEBUG:boto:Params: {} 2024-02-01T10:27:35.381 DEBUG:boto:Token: None 2024-02-01T10:27:35.381 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:35 GMT /vclfcn-19/ 2024-02-01T10:27:35.381 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ztaD4sEmE1szOBpCcy0qt3bA0CY= 2024-02-01T10:27:35.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ztaD4sEmE1szOBpCcy0qt3bA0CY=', 'Content-Length': '0'} 2024-02-01T10:27:35.386 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000056ca3f4094d5a9c3-0065bb7217-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:35.386 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:35.387 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:35.387 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:35.387 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:35.387 DEBUG:boto:path=/vclfcn-19/obj?versionId=RrytQti15jy37pft0WxtgWdt4Zs7l4E 2024-02-01T10:27:35.387 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=RrytQti15jy37pft0WxtgWdt4Zs7l4E 2024-02-01T10:27:35.387 DEBUG:boto:Method: GET 2024-02-01T10:27:35.387 DEBUG:boto:Path: /vclfcn-19/obj?versionId=RrytQti15jy37pft0WxtgWdt4Zs7l4E 2024-02-01T10:27:35.387 DEBUG:boto:Data: 2024-02-01T10:27:35.387 DEBUG:boto:Headers: {} 2024-02-01T10:27:35.387 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:35.388 DEBUG:boto:Port: 8000 2024-02-01T10:27:35.388 DEBUG:boto:Params: {} 2024-02-01T10:27:35.388 DEBUG:boto:Token: None 2024-02-01T10:27:35.388 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:35 GMT /vclfcn-19/obj?versionId=RrytQti15jy37pft0WxtgWdt4Zs7l4E 2024-02-01T10:27:35.388 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:KXVp4R6dgDrdkt4V8ZrsmXIdT/0= 2024-02-01T10:27:35.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:KXVp4R6dgDrdkt4V8ZrsmXIdT/0=', 'Content-Length': '0'} 2024-02-01T10:27:35.391 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'RrytQti15jy37pft0WxtgWdt4Zs7l4E'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ede94875a586d8a2-0065bb7217-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:35.392 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:35.392 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:35.392 DEBUG:boto:path=/vclfcn-19/obj?versionId=RrytQti15jy37pft0WxtgWdt4Zs7l4E 2024-02-01T10:27:35.392 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=RrytQti15jy37pft0WxtgWdt4Zs7l4E 2024-02-01T10:27:35.392 DEBUG:boto:Method: GET 2024-02-01T10:27:35.392 DEBUG:boto:Path: /vclfcn-19/obj?versionId=RrytQti15jy37pft0WxtgWdt4Zs7l4E 2024-02-01T10:27:35.392 DEBUG:boto:Data: 2024-02-01T10:27:35.392 DEBUG:boto:Headers: {} 2024-02-01T10:27:35.392 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:35.392 DEBUG:boto:Port: 8000 2024-02-01T10:27:35.393 DEBUG:boto:Params: {} 2024-02-01T10:27:35.393 DEBUG:boto:Token: None 2024-02-01T10:27:35.393 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:35 GMT /vclfcn-19/obj?versionId=RrytQti15jy37pft0WxtgWdt4Zs7l4E 2024-02-01T10:27:35.393 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:KXVp4R6dgDrdkt4V8ZrsmXIdT/0= 2024-02-01T10:27:35.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:KXVp4R6dgDrdkt4V8ZrsmXIdT/0=', 'Content-Length': '0'} 2024-02-01T10:27:35.396 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'RrytQti15jy37pft0WxtgWdt4Zs7l4E'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cdf30b0d66ec7e24-0065bb7217-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:35.396 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:35.397 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:35.397 DEBUG:boto:path=/vclfcn-19/obj?versionId=RrytQti15jy37pft0WxtgWdt4Zs7l4E 2024-02-01T10:27:35.397 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=RrytQti15jy37pft0WxtgWdt4Zs7l4E 2024-02-01T10:27:35.397 DEBUG:boto:Method: HEAD 2024-02-01T10:27:35.397 DEBUG:boto:Path: /vclfcn-19/obj?versionId=RrytQti15jy37pft0WxtgWdt4Zs7l4E 2024-02-01T10:27:35.397 DEBUG:boto:Data: 2024-02-01T10:27:35.397 DEBUG:boto:Headers: {} 2024-02-01T10:27:35.397 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:35.397 DEBUG:boto:Port: 8000 2024-02-01T10:27:35.397 DEBUG:boto:Params: {} 2024-02-01T10:27:35.398 DEBUG:boto:Token: None 2024-02-01T10:27:35.398 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:35 GMT /vclfcn-19/obj?versionId=RrytQti15jy37pft0WxtgWdt4Zs7l4E 2024-02-01T10:27:35.398 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mqLqObQH3yv2YXGkBbDx41bz+WY= 2024-02-01T10:27:35.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mqLqObQH3yv2YXGkBbDx41bz+WY=', 'Content-Length': '0'} 2024-02-01T10:27:35.401 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'RrytQti15jy37pft0WxtgWdt4Zs7l4E'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008332580f2f9f7373-0065bb7217-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:35.401 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:35.401 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:35.401 DEBUG:boto:path=/vclfcn-19/obj?versionId=RrytQti15jy37pft0WxtgWdt4Zs7l4E 2024-02-01T10:27:35.401 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=RrytQti15jy37pft0WxtgWdt4Zs7l4E 2024-02-01T10:27:35.402 DEBUG:boto:Method: HEAD 2024-02-01T10:27:35.402 DEBUG:boto:Path: /vclfcn-19/obj?versionId=RrytQti15jy37pft0WxtgWdt4Zs7l4E 2024-02-01T10:27:35.402 DEBUG:boto:Data: 2024-02-01T10:27:35.402 DEBUG:boto:Headers: {} 2024-02-01T10:27:35.402 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:35.402 DEBUG:boto:Port: 8000 2024-02-01T10:27:35.402 DEBUG:boto:Params: {} 2024-02-01T10:27:35.402 DEBUG:boto:Token: None 2024-02-01T10:27:35.402 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:35 GMT /vclfcn-19/obj?versionId=RrytQti15jy37pft0WxtgWdt4Zs7l4E 2024-02-01T10:27:35.402 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mqLqObQH3yv2YXGkBbDx41bz+WY= 2024-02-01T10:27:35.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mqLqObQH3yv2YXGkBbDx41bz+WY=', 'Content-Length': '0'} 2024-02-01T10:27:35.405 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'RrytQti15jy37pft0WxtgWdt4Zs7l4E'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ddb1d74d169858cb-0065bb7217-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:35.405 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:35.405 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:35.406 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:35.406 DEBUG:boto:path=/vclfcn-19/obj?versionId=RrytQti15jy37pft0WxtgWdt4Zs7l4E 2024-02-01T10:27:35.406 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=RrytQti15jy37pft0WxtgWdt4Zs7l4E 2024-02-01T10:27:35.406 DEBUG:boto:Method: GET 2024-02-01T10:27:35.406 DEBUG:boto:Path: /vclfcn-19/obj?versionId=RrytQti15jy37pft0WxtgWdt4Zs7l4E 2024-02-01T10:27:35.406 DEBUG:boto:Data: 2024-02-01T10:27:35.406 DEBUG:boto:Headers: {} 2024-02-01T10:27:35.406 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:35.406 DEBUG:boto:Port: 8000 2024-02-01T10:27:35.406 DEBUG:boto:Params: {} 2024-02-01T10:27:35.406 DEBUG:boto:Token: None 2024-02-01T10:27:35.407 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:35 GMT /vclfcn-19/obj?versionId=RrytQti15jy37pft0WxtgWdt4Zs7l4E 2024-02-01T10:27:35.407 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:KXVp4R6dgDrdkt4V8ZrsmXIdT/0= 2024-02-01T10:27:35.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:KXVp4R6dgDrdkt4V8ZrsmXIdT/0=', 'Content-Length': '0'} 2024-02-01T10:27:35.410 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'RrytQti15jy37pft0WxtgWdt4Zs7l4E'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009b77d6f5c3a6701f-0065bb7217-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:35.410 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:35.411 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:35.411 DEBUG:boto:path=/vclfcn-19/obj?versionId=RrytQti15jy37pft0WxtgWdt4Zs7l4E 2024-02-01T10:27:35.411 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=RrytQti15jy37pft0WxtgWdt4Zs7l4E 2024-02-01T10:27:35.411 DEBUG:boto:Method: GET 2024-02-01T10:27:35.411 DEBUG:boto:Path: /vclfcn-19/obj?versionId=RrytQti15jy37pft0WxtgWdt4Zs7l4E 2024-02-01T10:27:35.411 DEBUG:boto:Data: 2024-02-01T10:27:35.411 DEBUG:boto:Headers: {} 2024-02-01T10:27:35.411 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:35.411 DEBUG:boto:Port: 8000 2024-02-01T10:27:35.411 DEBUG:boto:Params: {} 2024-02-01T10:27:35.412 DEBUG:boto:Token: None 2024-02-01T10:27:35.412 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:35 GMT /vclfcn-19/obj?versionId=RrytQti15jy37pft0WxtgWdt4Zs7l4E 2024-02-01T10:27:35.412 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:KXVp4R6dgDrdkt4V8ZrsmXIdT/0= 2024-02-01T10:27:35.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:KXVp4R6dgDrdkt4V8ZrsmXIdT/0=', 'Content-Length': '0'} 2024-02-01T10:27:35.415 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'RrytQti15jy37pft0WxtgWdt4Zs7l4E'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003655143dbdf44ef3-0065bb7217-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:35.456 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:35.456 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:35.456 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:35.457 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:35.457 DEBUG:boto:Method: GET 2024-02-01T10:27:35.457 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:35.457 DEBUG:boto:Data: 2024-02-01T10:27:35.457 DEBUG:boto:Headers: {} 2024-02-01T10:27:35.457 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:35.457 DEBUG:boto:Port: 8000 2024-02-01T10:27:35.457 DEBUG:boto:Params: {} 2024-02-01T10:27:35.457 DEBUG:boto:Token: None 2024-02-01T10:27:35.458 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:35 GMT /vclfcn-19/ 2024-02-01T10:27:35.458 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ztaD4sEmE1szOBpCcy0qt3bA0CY= 2024-02-01T10:27:35.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ztaD4sEmE1szOBpCcy0qt3bA0CY=', 'Content-Length': '0'} 2024-02-01T10:27:35.463 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000074882cdbde534b5e-0065bb7217-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:35.463 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:35.464 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:35.464 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:35.464 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:35.464 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:35.464 DEBUG:boto:Method: GET 2024-02-01T10:27:35.464 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:35.465 DEBUG:boto:Data: 2024-02-01T10:27:35.465 DEBUG:boto:Headers: {} 2024-02-01T10:27:35.465 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:35.465 DEBUG:boto:Port: 8000 2024-02-01T10:27:35.465 DEBUG:boto:Params: {} 2024-02-01T10:27:35.465 DEBUG:boto:Token: None 2024-02-01T10:27:35.465 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:35 GMT /vclfcn-19/ 2024-02-01T10:27:35.465 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ztaD4sEmE1szOBpCcy0qt3bA0CY= 2024-02-01T10:27:35.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ztaD4sEmE1szOBpCcy0qt3bA0CY=', 'Content-Length': '0'} 2024-02-01T10:27:35.470 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d5486831b4540c8-0065bb7217-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:35.471 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:35.471 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:35.471 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:35.472 DEBUG:boto:Method: HEAD 2024-02-01T10:27:35.472 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:35.472 DEBUG:boto:Data: 2024-02-01T10:27:35.472 DEBUG:boto:Headers: {} 2024-02-01T10:27:35.472 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:35.472 DEBUG:boto:Port: 8000 2024-02-01T10:27:35.472 DEBUG:boto:Params: {} 2024-02-01T10:27:35.472 DEBUG:boto:Token: None 2024-02-01T10:27:35.472 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:35 GMT /vclfcn-19/obj 2024-02-01T10:27:35.472 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ytaCXXHekoXVGmmpyC1lTc8j2LI= 2024-02-01T10:27:35.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ytaCXXHekoXVGmmpyC1lTc8j2LI=', 'Content-Length': '0'} 2024-02-01T10:27:35.478 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004e0a481451dcaa12-0065bb7217-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:35.478 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:35.478 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:35.478 DEBUG:boto:Method: HEAD 2024-02-01T10:27:35.478 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:35.478 DEBUG:boto:Data: 2024-02-01T10:27:35.479 DEBUG:boto:Headers: {} 2024-02-01T10:27:35.479 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:35.479 DEBUG:boto:Port: 8000 2024-02-01T10:27:35.479 DEBUG:boto:Params: {} 2024-02-01T10:27:35.479 DEBUG:boto:Token: None 2024-02-01T10:27:35.479 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:35 GMT /vclfcn-19/obj 2024-02-01T10:27:35.479 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ytaCXXHekoXVGmmpyC1lTc8j2LI= 2024-02-01T10:27:35.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ytaCXXHekoXVGmmpyC1lTc8j2LI=', 'Content-Length': '0'} 2024-02-01T10:27:35.486 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002f80fdffbd43de60-0065bb7217-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:35.486 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:35.486 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:35.487 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:35.487 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:35.487 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:35.487 DEBUG:boto:Method: GET 2024-02-01T10:27:35.487 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:35.487 DEBUG:boto:Data: 2024-02-01T10:27:35.487 DEBUG:boto:Headers: {} 2024-02-01T10:27:35.488 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:35.488 DEBUG:boto:Port: 8000 2024-02-01T10:27:35.488 DEBUG:boto:Params: {} 2024-02-01T10:27:35.488 DEBUG:boto:Token: None 2024-02-01T10:27:35.488 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:35 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:35.489 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:CgiuP0q9dYuwGEuNSmNHt0623gE= 2024-02-01T10:27:35.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:CgiuP0q9dYuwGEuNSmNHt0623gE=', 'Content-Length': '0'} 2024-02-01T10:27:35.492 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000da1be12450483243-0065bb7217-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:35.532 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:35.532 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:35.532 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:35.533 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:35.533 DEBUG:boto:Method: GET 2024-02-01T10:27:35.533 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:35.533 DEBUG:boto:Data: 2024-02-01T10:27:35.533 DEBUG:boto:Headers: {} 2024-02-01T10:27:35.533 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:35.533 DEBUG:boto:Port: 8000 2024-02-01T10:27:35.533 DEBUG:boto:Params: {} 2024-02-01T10:27:35.533 DEBUG:boto:Token: None 2024-02-01T10:27:35.534 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:35 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:35.534 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:CgiuP0q9dYuwGEuNSmNHt0623gE= 2024-02-01T10:27:35.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:CgiuP0q9dYuwGEuNSmNHt0623gE=', 'Content-Length': '0'} 2024-02-01T10:27:35.537 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e700e7c73ce1939b-0065bb7217-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:35.537 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:35.538 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:35.538 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:35.538 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:35.538 DEBUG:boto:Method: GET 2024-02-01T10:27:35.538 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:35.538 DEBUG:boto:Data: 2024-02-01T10:27:35.538 DEBUG:boto:Headers: {} 2024-02-01T10:27:35.538 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:35.539 DEBUG:boto:Port: 8000 2024-02-01T10:27:35.539 DEBUG:boto:Params: {} 2024-02-01T10:27:35.539 DEBUG:boto:Token: None 2024-02-01T10:27:35.539 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:35 GMT /vclfcn-19/ 2024-02-01T10:27:35.539 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ztaD4sEmE1szOBpCcy0qt3bA0CY= 2024-02-01T10:27:35.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ztaD4sEmE1szOBpCcy0qt3bA0CY=', 'Content-Length': '0'} 2024-02-01T10:27:35.544 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ed2250371afdec6-0065bb7217-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:35.545 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:35.545 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:35.545 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:35.545 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:35.545 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:35.546 DEBUG:boto:Method: GET 2024-02-01T10:27:35.546 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:35.546 DEBUG:boto:Data: 2024-02-01T10:27:35.546 DEBUG:boto:Headers: {} 2024-02-01T10:27:35.546 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:35.546 DEBUG:boto:Port: 8000 2024-02-01T10:27:35.546 DEBUG:boto:Params: {} 2024-02-01T10:27:35.546 DEBUG:boto:Token: None 2024-02-01T10:27:35.546 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:35 GMT /vclfcn-19/ 2024-02-01T10:27:35.547 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ztaD4sEmE1szOBpCcy0qt3bA0CY= 2024-02-01T10:27:35.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ztaD4sEmE1szOBpCcy0qt3bA0CY=', 'Content-Length': '0'} 2024-02-01T10:27:35.551 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006fbcb082dadfc9b9-0065bb7217-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:35.592 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:35.593 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:35.593 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:35.593 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:35.593 DEBUG:boto:path=/vclfcn-19/obj?versionId=fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u 2024-02-01T10:27:35.593 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u 2024-02-01T10:27:35.593 DEBUG:boto:Method: GET 2024-02-01T10:27:35.593 DEBUG:boto:Path: /vclfcn-19/obj?versionId=fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u 2024-02-01T10:27:35.593 DEBUG:boto:Data: 2024-02-01T10:27:35.593 DEBUG:boto:Headers: {} 2024-02-01T10:27:35.594 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:35.594 DEBUG:boto:Port: 8000 2024-02-01T10:27:35.594 DEBUG:boto:Params: {} 2024-02-01T10:27:35.594 DEBUG:boto:Token: None 2024-02-01T10:27:35.594 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:35 GMT /vclfcn-19/obj?versionId=fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u 2024-02-01T10:27:35.594 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:djfm8tHHLu/X0BCd3wVR8n4uOEM= 2024-02-01T10:27:35.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:djfm8tHHLu/X0BCd3wVR8n4uOEM=', 'Content-Length': '0'} 2024-02-01T10:27:35.597 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bb67b21abc201a39-0065bb7217-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:35.597 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:35.598 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:35.598 DEBUG:boto:path=/vclfcn-19/obj?versionId=fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u 2024-02-01T10:27:35.598 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u 2024-02-01T10:27:35.598 DEBUG:boto:Method: GET 2024-02-01T10:27:35.598 DEBUG:boto:Path: /vclfcn-19/obj?versionId=fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u 2024-02-01T10:27:35.598 DEBUG:boto:Data: 2024-02-01T10:27:35.598 DEBUG:boto:Headers: {} 2024-02-01T10:27:35.598 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:35.598 DEBUG:boto:Port: 8000 2024-02-01T10:27:35.598 DEBUG:boto:Params: {} 2024-02-01T10:27:35.599 DEBUG:boto:Token: None 2024-02-01T10:27:35.599 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:35 GMT /vclfcn-19/obj?versionId=fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u 2024-02-01T10:27:35.599 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:djfm8tHHLu/X0BCd3wVR8n4uOEM= 2024-02-01T10:27:35.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:djfm8tHHLu/X0BCd3wVR8n4uOEM=', 'Content-Length': '0'} 2024-02-01T10:27:35.602 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000069e73676edb65845-0065bb7217-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:35.603 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:35.603 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:35.603 DEBUG:boto:path=/vclfcn-19/obj?versionId=fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u 2024-02-01T10:27:35.603 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u 2024-02-01T10:27:35.603 DEBUG:boto:Method: HEAD 2024-02-01T10:27:35.603 DEBUG:boto:Path: /vclfcn-19/obj?versionId=fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u 2024-02-01T10:27:35.603 DEBUG:boto:Data: 2024-02-01T10:27:35.603 DEBUG:boto:Headers: {} 2024-02-01T10:27:35.604 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:35.604 DEBUG:boto:Port: 8000 2024-02-01T10:27:35.604 DEBUG:boto:Params: {} 2024-02-01T10:27:35.604 DEBUG:boto:Token: None 2024-02-01T10:27:35.604 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:35 GMT /vclfcn-19/obj?versionId=fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u 2024-02-01T10:27:35.604 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vNZcimrgxItRuWKGCaVJYcG7m4M= 2024-02-01T10:27:35.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vNZcimrgxItRuWKGCaVJYcG7m4M=', 'Content-Length': '0'} 2024-02-01T10:27:35.607 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e2ba866afcb756a1-0065bb7217-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:35.607 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:35.607 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:35.608 DEBUG:boto:path=/vclfcn-19/obj?versionId=fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u 2024-02-01T10:27:35.608 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u 2024-02-01T10:27:35.608 DEBUG:boto:Method: HEAD 2024-02-01T10:27:35.608 DEBUG:boto:Path: /vclfcn-19/obj?versionId=fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u 2024-02-01T10:27:35.608 DEBUG:boto:Data: 2024-02-01T10:27:35.608 DEBUG:boto:Headers: {} 2024-02-01T10:27:35.608 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:35.608 DEBUG:boto:Port: 8000 2024-02-01T10:27:35.608 DEBUG:boto:Params: {} 2024-02-01T10:27:35.609 DEBUG:boto:Token: None 2024-02-01T10:27:35.609 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:35 GMT /vclfcn-19/obj?versionId=fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u 2024-02-01T10:27:35.609 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vNZcimrgxItRuWKGCaVJYcG7m4M= 2024-02-01T10:27:35.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vNZcimrgxItRuWKGCaVJYcG7m4M=', 'Content-Length': '0'} 2024-02-01T10:27:35.612 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000007593ab8c8d7ac04-0065bb7217-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:35.612 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:35.612 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:35.612 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:35.613 DEBUG:boto:path=/vclfcn-19/obj?versionId=fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u 2024-02-01T10:27:35.613 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u 2024-02-01T10:27:35.613 DEBUG:boto:Method: GET 2024-02-01T10:27:35.613 DEBUG:boto:Path: /vclfcn-19/obj?versionId=fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u 2024-02-01T10:27:35.613 DEBUG:boto:Data: 2024-02-01T10:27:35.613 DEBUG:boto:Headers: {} 2024-02-01T10:27:35.613 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:35.613 DEBUG:boto:Port: 8000 2024-02-01T10:27:35.613 DEBUG:boto:Params: {} 2024-02-01T10:27:35.613 DEBUG:boto:Token: None 2024-02-01T10:27:35.614 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:35 GMT /vclfcn-19/obj?versionId=fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u 2024-02-01T10:27:35.614 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:djfm8tHHLu/X0BCd3wVR8n4uOEM= 2024-02-01T10:27:35.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:djfm8tHHLu/X0BCd3wVR8n4uOEM=', 'Content-Length': '0'} 2024-02-01T10:27:35.617 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fad2189e83580cba-0065bb7217-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:35.617 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:35.617 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:35.617 DEBUG:boto:path=/vclfcn-19/obj?versionId=fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u 2024-02-01T10:27:35.617 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u 2024-02-01T10:27:35.617 DEBUG:boto:Method: GET 2024-02-01T10:27:35.617 DEBUG:boto:Path: /vclfcn-19/obj?versionId=fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u 2024-02-01T10:27:35.617 DEBUG:boto:Data: 2024-02-01T10:27:35.618 DEBUG:boto:Headers: {} 2024-02-01T10:27:35.618 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:35.618 DEBUG:boto:Port: 8000 2024-02-01T10:27:35.618 DEBUG:boto:Params: {} 2024-02-01T10:27:35.618 DEBUG:boto:Token: None 2024-02-01T10:27:35.618 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:35 GMT /vclfcn-19/obj?versionId=fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u 2024-02-01T10:27:35.618 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:djfm8tHHLu/X0BCd3wVR8n4uOEM= 2024-02-01T10:27:35.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:djfm8tHHLu/X0BCd3wVR8n4uOEM=', 'Content-Length': '0'} 2024-02-01T10:27:35.621 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bc270239d96d20f3-0065bb7217-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:35.621 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:35.622 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:35.622 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:35.622 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:35.622 DEBUG:boto:Method: GET 2024-02-01T10:27:35.622 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:35.622 DEBUG:boto:Data: 2024-02-01T10:27:35.622 DEBUG:boto:Headers: {} 2024-02-01T10:27:35.622 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:35.622 DEBUG:boto:Port: 8000 2024-02-01T10:27:35.622 DEBUG:boto:Params: {} 2024-02-01T10:27:35.622 DEBUG:boto:Token: None 2024-02-01T10:27:35.623 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:35 GMT /vclfcn-19/ 2024-02-01T10:27:35.623 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ztaD4sEmE1szOBpCcy0qt3bA0CY= 2024-02-01T10:27:35.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ztaD4sEmE1szOBpCcy0qt3bA0CY=', 'Content-Length': '0'} 2024-02-01T10:27:35.628 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd99535ac3fc78f9-0065bb7217-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:35.672 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:35.673 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:35.673 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:35.673 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:35.673 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:35.673 DEBUG:boto:Method: GET 2024-02-01T10:27:35.673 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:35.673 DEBUG:boto:Data: 2024-02-01T10:27:35.673 DEBUG:boto:Headers: {} 2024-02-01T10:27:35.673 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:35.673 DEBUG:boto:Port: 8000 2024-02-01T10:27:35.673 DEBUG:boto:Params: {} 2024-02-01T10:27:35.674 DEBUG:boto:Token: None 2024-02-01T10:27:35.674 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:35 GMT /vclfcn-19/ 2024-02-01T10:27:35.674 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ztaD4sEmE1szOBpCcy0qt3bA0CY= 2024-02-01T10:27:35.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ztaD4sEmE1szOBpCcy0qt3bA0CY=', 'Content-Length': '0'} 2024-02-01T10:27:35.679 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e72ba365a73deb18-0065bb7217-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:35.680 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:35.680 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:35.681 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:35.681 DEBUG:boto:Method: HEAD 2024-02-01T10:27:35.681 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:35.681 DEBUG:boto:Data: 2024-02-01T10:27:35.681 DEBUG:boto:Headers: {} 2024-02-01T10:27:35.681 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:35.681 DEBUG:boto:Port: 8000 2024-02-01T10:27:35.681 DEBUG:boto:Params: {} 2024-02-01T10:27:35.681 DEBUG:boto:Token: None 2024-02-01T10:27:35.682 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:35 GMT /vclfcn-19/obj 2024-02-01T10:27:35.682 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ytaCXXHekoXVGmmpyC1lTc8j2LI= 2024-02-01T10:27:35.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ytaCXXHekoXVGmmpyC1lTc8j2LI=', 'Content-Length': '0'} 2024-02-01T10:27:35.687 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005ce59a19583afdc1-0065bb7217-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:35.688 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:35.688 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:35.688 DEBUG:boto:Method: HEAD 2024-02-01T10:27:35.688 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:35.688 DEBUG:boto:Data: 2024-02-01T10:27:35.688 DEBUG:boto:Headers: {} 2024-02-01T10:27:35.688 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:35.688 DEBUG:boto:Port: 8000 2024-02-01T10:27:35.688 DEBUG:boto:Params: {} 2024-02-01T10:27:35.689 DEBUG:boto:Token: None 2024-02-01T10:27:35.689 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:35 GMT /vclfcn-19/obj 2024-02-01T10:27:35.689 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ytaCXXHekoXVGmmpyC1lTc8j2LI= 2024-02-01T10:27:35.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ytaCXXHekoXVGmmpyC1lTc8j2LI=', 'Content-Length': '0'} 2024-02-01T10:27:35.695 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006099df7529d52226-0065bb7217-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:35.695 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:35.695 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:35.695 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:35.695 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:35.695 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:35.696 DEBUG:boto:Method: GET 2024-02-01T10:27:35.696 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:35.696 DEBUG:boto:Data: 2024-02-01T10:27:35.696 DEBUG:boto:Headers: {} 2024-02-01T10:27:35.696 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:35.696 DEBUG:boto:Port: 8000 2024-02-01T10:27:35.696 DEBUG:boto:Params: {} 2024-02-01T10:27:35.696 DEBUG:boto:Token: None 2024-02-01T10:27:35.696 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:35 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:35.697 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:CgiuP0q9dYuwGEuNSmNHt0623gE= 2024-02-01T10:27:35.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:CgiuP0q9dYuwGEuNSmNHt0623gE=', 'Content-Length': '0'} 2024-02-01T10:27:35.702 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f333444088c287bd-0065bb7217-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:35.702 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:35.702 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:35.702 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:35.702 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:35.702 DEBUG:boto:Method: GET 2024-02-01T10:27:35.702 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:35.703 DEBUG:boto:Data: 2024-02-01T10:27:35.703 DEBUG:boto:Headers: {} 2024-02-01T10:27:35.703 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:35.703 DEBUG:boto:Port: 8000 2024-02-01T10:27:35.703 DEBUG:boto:Params: {} 2024-02-01T10:27:35.703 DEBUG:boto:Token: None 2024-02-01T10:27:35.703 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:35 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:35.704 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:CgiuP0q9dYuwGEuNSmNHt0623gE= 2024-02-01T10:27:35.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:CgiuP0q9dYuwGEuNSmNHt0623gE=', 'Content-Length': '0'} 2024-02-01T10:27:35.707 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ccf0f2048981a581-0065bb7217-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:35.748 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:35.748 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:35.749 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:35.749 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:35.749 DEBUG:boto:Method: GET 2024-02-01T10:27:35.749 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:35.749 DEBUG:boto:Data: 2024-02-01T10:27:35.749 DEBUG:boto:Headers: {} 2024-02-01T10:27:35.749 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:35.749 DEBUG:boto:Port: 8000 2024-02-01T10:27:35.749 DEBUG:boto:Params: {} 2024-02-01T10:27:35.749 DEBUG:boto:Token: None 2024-02-01T10:27:35.750 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:35 GMT /vclfcn-19/ 2024-02-01T10:27:35.750 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ztaD4sEmE1szOBpCcy0qt3bA0CY= 2024-02-01T10:27:35.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ztaD4sEmE1szOBpCcy0qt3bA0CY=', 'Content-Length': '0'} 2024-02-01T10:27:35.755 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ca8285b954486b4-0065bb7217-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:35.755 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:35.756 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:35.756 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:35.756 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:35.756 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:35.756 DEBUG:boto:Method: GET 2024-02-01T10:27:35.756 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:35.756 DEBUG:boto:Data: 2024-02-01T10:27:35.756 DEBUG:boto:Headers: {} 2024-02-01T10:27:35.757 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:35.757 DEBUG:boto:Port: 8000 2024-02-01T10:27:35.757 DEBUG:boto:Params: {} 2024-02-01T10:27:35.757 DEBUG:boto:Token: None 2024-02-01T10:27:35.757 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:35 GMT /vclfcn-19/ 2024-02-01T10:27:35.757 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ztaD4sEmE1szOBpCcy0qt3bA0CY= 2024-02-01T10:27:35.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ztaD4sEmE1szOBpCcy0qt3bA0CY=', 'Content-Length': '0'} 2024-02-01T10:27:35.762 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5cc62d703ac54ff-0065bb7217-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:35.763 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:35.763 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:35.763 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:35.763 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:35.763 DEBUG:boto:path=/vclfcn-19/obj?versionId=UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc 2024-02-01T10:27:35.763 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc 2024-02-01T10:27:35.763 DEBUG:boto:Method: GET 2024-02-01T10:27:35.764 DEBUG:boto:Path: /vclfcn-19/obj?versionId=UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc 2024-02-01T10:27:35.764 DEBUG:boto:Data: 2024-02-01T10:27:35.764 DEBUG:boto:Headers: {} 2024-02-01T10:27:35.764 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:35.764 DEBUG:boto:Port: 8000 2024-02-01T10:27:35.764 DEBUG:boto:Params: {} 2024-02-01T10:27:35.764 DEBUG:boto:Token: None 2024-02-01T10:27:35.764 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:35 GMT /vclfcn-19/obj?versionId=UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc 2024-02-01T10:27:35.764 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:F0L4Uuq9AvM8p9C0XSAcKHal6Rw= 2024-02-01T10:27:35.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:F0L4Uuq9AvM8p9C0XSAcKHal6Rw=', 'Content-Length': '0'} 2024-02-01T10:27:35.767 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aa7c6970f1ee09ba-0065bb7217-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:35.812 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:35.812 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:35.813 DEBUG:boto:path=/vclfcn-19/obj?versionId=UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc 2024-02-01T10:27:35.813 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc 2024-02-01T10:27:35.813 DEBUG:boto:Method: GET 2024-02-01T10:27:35.813 DEBUG:boto:Path: /vclfcn-19/obj?versionId=UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc 2024-02-01T10:27:35.813 DEBUG:boto:Data: 2024-02-01T10:27:35.813 DEBUG:boto:Headers: {} 2024-02-01T10:27:35.813 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:35.813 DEBUG:boto:Port: 8000 2024-02-01T10:27:35.813 DEBUG:boto:Params: {} 2024-02-01T10:27:35.813 DEBUG:boto:Token: None 2024-02-01T10:27:35.814 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:35 GMT /vclfcn-19/obj?versionId=UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc 2024-02-01T10:27:35.814 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:F0L4Uuq9AvM8p9C0XSAcKHal6Rw= 2024-02-01T10:27:35.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:F0L4Uuq9AvM8p9C0XSAcKHal6Rw=', 'Content-Length': '0'} 2024-02-01T10:27:35.817 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001f22a2f9b9696bda-0065bb7217-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:35.817 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:35.817 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:35.817 DEBUG:boto:path=/vclfcn-19/obj?versionId=UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc 2024-02-01T10:27:35.817 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc 2024-02-01T10:27:35.818 DEBUG:boto:Method: HEAD 2024-02-01T10:27:35.818 DEBUG:boto:Path: /vclfcn-19/obj?versionId=UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc 2024-02-01T10:27:35.818 DEBUG:boto:Data: 2024-02-01T10:27:35.818 DEBUG:boto:Headers: {} 2024-02-01T10:27:35.818 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:35.818 DEBUG:boto:Port: 8000 2024-02-01T10:27:35.818 DEBUG:boto:Params: {} 2024-02-01T10:27:35.818 DEBUG:boto:Token: None 2024-02-01T10:27:35.818 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:35 GMT /vclfcn-19/obj?versionId=UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc 2024-02-01T10:27:35.819 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:yP8Qp6X/9Ugg6hjRI7lL+Na2GnU= 2024-02-01T10:27:35.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:yP8Qp6X/9Ugg6hjRI7lL+Na2GnU=', 'Content-Length': '0'} 2024-02-01T10:27:35.821 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000403cc25ca906ccb9-0065bb7217-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:35.822 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:35.822 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:35.822 DEBUG:boto:path=/vclfcn-19/obj?versionId=UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc 2024-02-01T10:27:35.822 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc 2024-02-01T10:27:35.822 DEBUG:boto:Method: HEAD 2024-02-01T10:27:35.822 DEBUG:boto:Path: /vclfcn-19/obj?versionId=UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc 2024-02-01T10:27:35.822 DEBUG:boto:Data: 2024-02-01T10:27:35.822 DEBUG:boto:Headers: {} 2024-02-01T10:27:35.822 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:35.822 DEBUG:boto:Port: 8000 2024-02-01T10:27:35.822 DEBUG:boto:Params: {} 2024-02-01T10:27:35.823 DEBUG:boto:Token: None 2024-02-01T10:27:35.823 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:35 GMT /vclfcn-19/obj?versionId=UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc 2024-02-01T10:27:35.823 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:yP8Qp6X/9Ugg6hjRI7lL+Na2GnU= 2024-02-01T10:27:35.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:yP8Qp6X/9Ugg6hjRI7lL+Na2GnU=', 'Content-Length': '0'} 2024-02-01T10:27:35.825 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c499c66204fad87a-0065bb7217-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:35.826 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:35.826 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:35.826 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:35.826 DEBUG:boto:path=/vclfcn-19/obj?versionId=UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc 2024-02-01T10:27:35.826 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc 2024-02-01T10:27:35.826 DEBUG:boto:Method: GET 2024-02-01T10:27:35.826 DEBUG:boto:Path: /vclfcn-19/obj?versionId=UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc 2024-02-01T10:27:35.827 DEBUG:boto:Data: 2024-02-01T10:27:35.827 DEBUG:boto:Headers: {} 2024-02-01T10:27:35.827 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:35.827 DEBUG:boto:Port: 8000 2024-02-01T10:27:35.827 DEBUG:boto:Params: {} 2024-02-01T10:27:35.827 DEBUG:boto:Token: None 2024-02-01T10:27:35.827 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:35 GMT /vclfcn-19/obj?versionId=UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc 2024-02-01T10:27:35.827 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:F0L4Uuq9AvM8p9C0XSAcKHal6Rw= 2024-02-01T10:27:35.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:F0L4Uuq9AvM8p9C0XSAcKHal6Rw=', 'Content-Length': '0'} 2024-02-01T10:27:35.830 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aa01bd82887d4acb-0065bb7217-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:35.830 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:35.830 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:35.831 DEBUG:boto:path=/vclfcn-19/obj?versionId=UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc 2024-02-01T10:27:35.831 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc 2024-02-01T10:27:35.831 DEBUG:boto:Method: GET 2024-02-01T10:27:35.831 DEBUG:boto:Path: /vclfcn-19/obj?versionId=UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc 2024-02-01T10:27:35.831 DEBUG:boto:Data: 2024-02-01T10:27:35.831 DEBUG:boto:Headers: {} 2024-02-01T10:27:35.831 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:35.831 DEBUG:boto:Port: 8000 2024-02-01T10:27:35.831 DEBUG:boto:Params: {} 2024-02-01T10:27:35.831 DEBUG:boto:Token: None 2024-02-01T10:27:35.832 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:35 GMT /vclfcn-19/obj?versionId=UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc 2024-02-01T10:27:35.832 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:F0L4Uuq9AvM8p9C0XSAcKHal6Rw= 2024-02-01T10:27:35.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:F0L4Uuq9AvM8p9C0XSAcKHal6Rw=', 'Content-Length': '0'} 2024-02-01T10:27:35.835 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000306a45196014bad4-0065bb7217-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:35.876 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:35.876 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:35.876 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:35.877 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:35.877 DEBUG:boto:Method: GET 2024-02-01T10:27:35.877 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:35.877 DEBUG:boto:Data: 2024-02-01T10:27:35.877 DEBUG:boto:Headers: {} 2024-02-01T10:27:35.877 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:35.877 DEBUG:boto:Port: 8000 2024-02-01T10:27:35.877 DEBUG:boto:Params: {} 2024-02-01T10:27:35.877 DEBUG:boto:Token: None 2024-02-01T10:27:35.878 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:35 GMT /vclfcn-19/ 2024-02-01T10:27:35.878 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ztaD4sEmE1szOBpCcy0qt3bA0CY= 2024-02-01T10:27:35.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ztaD4sEmE1szOBpCcy0qt3bA0CY=', 'Content-Length': '0'} 2024-02-01T10:27:35.883 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a7cfd433dbb2d3e-0065bb7217-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:35.883 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:35.883 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:35.884 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:35.884 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:35.884 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:35.884 DEBUG:boto:Method: GET 2024-02-01T10:27:35.884 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:35.884 DEBUG:boto:Data: 2024-02-01T10:27:35.884 DEBUG:boto:Headers: {} 2024-02-01T10:27:35.884 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:35.884 DEBUG:boto:Port: 8000 2024-02-01T10:27:35.884 DEBUG:boto:Params: {} 2024-02-01T10:27:35.884 DEBUG:boto:Token: None 2024-02-01T10:27:35.885 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:35 GMT /vclfcn-19/ 2024-02-01T10:27:35.885 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ztaD4sEmE1szOBpCcy0qt3bA0CY= 2024-02-01T10:27:35.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ztaD4sEmE1szOBpCcy0qt3bA0CY=', 'Content-Length': '0'} 2024-02-01T10:27:35.890 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004413b530436b3c4c-0065bb7217-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:35.890 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:35.891 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:35.891 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:35.891 DEBUG:boto:Method: HEAD 2024-02-01T10:27:35.891 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:35.891 DEBUG:boto:Data: 2024-02-01T10:27:35.891 DEBUG:boto:Headers: {} 2024-02-01T10:27:35.891 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:35.891 DEBUG:boto:Port: 8000 2024-02-01T10:27:35.891 DEBUG:boto:Params: {} 2024-02-01T10:27:35.892 DEBUG:boto:Token: None 2024-02-01T10:27:35.892 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:35 GMT /vclfcn-19/obj 2024-02-01T10:27:35.892 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ytaCXXHekoXVGmmpyC1lTc8j2LI= 2024-02-01T10:27:35.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ytaCXXHekoXVGmmpyC1lTc8j2LI=', 'Content-Length': '0'} 2024-02-01T10:27:35.897 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001914d154e7235cf2-0065bb7217-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:35.898 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:35.898 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:35.898 DEBUG:boto:Method: HEAD 2024-02-01T10:27:35.898 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:35.898 DEBUG:boto:Data: 2024-02-01T10:27:35.898 DEBUG:boto:Headers: {} 2024-02-01T10:27:35.898 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:35.898 DEBUG:boto:Port: 8000 2024-02-01T10:27:35.899 DEBUG:boto:Params: {} 2024-02-01T10:27:35.899 DEBUG:boto:Token: None 2024-02-01T10:27:35.899 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:35 GMT /vclfcn-19/obj 2024-02-01T10:27:35.899 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ytaCXXHekoXVGmmpyC1lTc8j2LI= 2024-02-01T10:27:35.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ytaCXXHekoXVGmmpyC1lTc8j2LI=', 'Content-Length': '0'} 2024-02-01T10:27:35.905 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009d42a71f90cc1c1a-0065bb7217-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:35.907 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:35.907 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:35.907 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:35.907 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:35.907 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:35.907 DEBUG:boto:Method: GET 2024-02-01T10:27:35.908 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:35.908 DEBUG:boto:Data: 2024-02-01T10:27:35.908 DEBUG:boto:Headers: {} 2024-02-01T10:27:35.908 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:35.908 DEBUG:boto:Port: 8000 2024-02-01T10:27:35.908 DEBUG:boto:Params: {} 2024-02-01T10:27:35.908 DEBUG:boto:Token: None 2024-02-01T10:27:35.909 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:35 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:35.909 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:CgiuP0q9dYuwGEuNSmNHt0623gE= 2024-02-01T10:27:35.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:CgiuP0q9dYuwGEuNSmNHt0623gE=', 'Content-Length': '0'} 2024-02-01T10:27:35.912 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002d4a27f609e320dc-0065bb7217-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:35.960 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:35.960 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:35.960 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:35.961 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:35.961 DEBUG:boto:Method: GET 2024-02-01T10:27:35.961 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:35.961 DEBUG:boto:Data: 2024-02-01T10:27:35.961 DEBUG:boto:Headers: {} 2024-02-01T10:27:35.961 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:35.961 DEBUG:boto:Port: 8000 2024-02-01T10:27:35.961 DEBUG:boto:Params: {} 2024-02-01T10:27:35.961 DEBUG:boto:Token: None 2024-02-01T10:27:35.961 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:35 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:35.962 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:CgiuP0q9dYuwGEuNSmNHt0623gE= 2024-02-01T10:27:35.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:CgiuP0q9dYuwGEuNSmNHt0623gE=', 'Content-Length': '0'} 2024-02-01T10:27:35.965 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000043945e3af51cb9f8-0065bb7217-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:35.965 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:35.966 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:35.966 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:35.966 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:35.966 DEBUG:boto:Method: GET 2024-02-01T10:27:35.966 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:35.966 DEBUG:boto:Data: 2024-02-01T10:27:35.966 DEBUG:boto:Headers: {} 2024-02-01T10:27:35.966 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:35.966 DEBUG:boto:Port: 8000 2024-02-01T10:27:35.966 DEBUG:boto:Params: {} 2024-02-01T10:27:35.967 DEBUG:boto:Token: None 2024-02-01T10:27:35.967 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:35 GMT /vclfcn-19/ 2024-02-01T10:27:35.967 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ztaD4sEmE1szOBpCcy0qt3bA0CY= 2024-02-01T10:27:35.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ztaD4sEmE1szOBpCcy0qt3bA0CY=', 'Content-Length': '0'} 2024-02-01T10:27:35.973 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cdbfbc5396ce8e2b-0065bb7217-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:35.973 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:35.974 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:35.974 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:35.974 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:35.974 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:35.974 DEBUG:boto:Method: GET 2024-02-01T10:27:35.974 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:35.974 DEBUG:boto:Data: 2024-02-01T10:27:35.975 DEBUG:boto:Headers: {} 2024-02-01T10:27:35.975 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:35.975 DEBUG:boto:Port: 8000 2024-02-01T10:27:35.975 DEBUG:boto:Params: {} 2024-02-01T10:27:35.975 DEBUG:boto:Token: None 2024-02-01T10:27:35.975 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:35 GMT /vclfcn-19/ 2024-02-01T10:27:35.975 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ztaD4sEmE1szOBpCcy0qt3bA0CY= 2024-02-01T10:27:35.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ztaD4sEmE1szOBpCcy0qt3bA0CY=', 'Content-Length': '0'} 2024-02-01T10:27:35.980 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000081900a3728df6956-0065bb7217-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:36.024 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:36.024 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:36.025 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:36.025 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:36.025 DEBUG:boto:path=/vclfcn-19/obj?versionId=eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr. 2024-02-01T10:27:36.025 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr. 2024-02-01T10:27:36.025 DEBUG:boto:Method: GET 2024-02-01T10:27:36.025 DEBUG:boto:Path: /vclfcn-19/obj?versionId=eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr. 2024-02-01T10:27:36.026 DEBUG:boto:Data: 2024-02-01T10:27:36.026 DEBUG:boto:Headers: {} 2024-02-01T10:27:36.026 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:36.026 DEBUG:boto:Port: 8000 2024-02-01T10:27:36.026 DEBUG:boto:Params: {} 2024-02-01T10:27:36.026 DEBUG:boto:Token: None 2024-02-01T10:27:36.026 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:36 GMT /vclfcn-19/obj?versionId=eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr. 2024-02-01T10:27:36.026 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:MqEZ4VrDZ/xQRdJNnU38gMwasbo= 2024-02-01T10:27:36.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:MqEZ4VrDZ/xQRdJNnU38gMwasbo=', 'Content-Length': '0'} 2024-02-01T10:27:36.031 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr.'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001710eeb3d50b7557-0065bb7218-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:36.032 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:36.032 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:36.032 DEBUG:boto:path=/vclfcn-19/obj?versionId=eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr. 2024-02-01T10:27:36.032 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr. 2024-02-01T10:27:36.032 DEBUG:boto:Method: GET 2024-02-01T10:27:36.032 DEBUG:boto:Path: /vclfcn-19/obj?versionId=eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr. 2024-02-01T10:27:36.032 DEBUG:boto:Data: 2024-02-01T10:27:36.032 DEBUG:boto:Headers: {} 2024-02-01T10:27:36.032 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:36.032 DEBUG:boto:Port: 8000 2024-02-01T10:27:36.032 DEBUG:boto:Params: {} 2024-02-01T10:27:36.033 DEBUG:boto:Token: None 2024-02-01T10:27:36.033 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:36 GMT /vclfcn-19/obj?versionId=eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr. 2024-02-01T10:27:36.033 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:MqEZ4VrDZ/xQRdJNnU38gMwasbo= 2024-02-01T10:27:36.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:MqEZ4VrDZ/xQRdJNnU38gMwasbo=', 'Content-Length': '0'} 2024-02-01T10:27:36.036 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c5adc2c10ee66a0b-0065bb7218-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:36.036 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:36.037 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:36.037 DEBUG:boto:path=/vclfcn-19/obj?versionId=eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr. 2024-02-01T10:27:36.037 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr. 2024-02-01T10:27:36.037 DEBUG:boto:Method: HEAD 2024-02-01T10:27:36.037 DEBUG:boto:Path: /vclfcn-19/obj?versionId=eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr. 2024-02-01T10:27:36.037 DEBUG:boto:Data: 2024-02-01T10:27:36.037 DEBUG:boto:Headers: {} 2024-02-01T10:27:36.037 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:36.037 DEBUG:boto:Port: 8000 2024-02-01T10:27:36.037 DEBUG:boto:Params: {} 2024-02-01T10:27:36.038 DEBUG:boto:Token: None 2024-02-01T10:27:36.038 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:36 GMT /vclfcn-19/obj?versionId=eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr. 2024-02-01T10:27:36.038 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:xwXOvDUqH8vzv33D8KU/vb1EzxY= 2024-02-01T10:27:36.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:xwXOvDUqH8vzv33D8KU/vb1EzxY=', 'Content-Length': '0'} 2024-02-01T10:27:36.041 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr.'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001c8c7622a10e6e0d-0065bb7218-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:36.041 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:36.041 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:36.041 DEBUG:boto:path=/vclfcn-19/obj?versionId=eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr. 2024-02-01T10:27:36.042 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr. 2024-02-01T10:27:36.042 DEBUG:boto:Method: HEAD 2024-02-01T10:27:36.042 DEBUG:boto:Path: /vclfcn-19/obj?versionId=eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr. 2024-02-01T10:27:36.042 DEBUG:boto:Data: 2024-02-01T10:27:36.042 DEBUG:boto:Headers: {} 2024-02-01T10:27:36.042 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:36.042 DEBUG:boto:Port: 8000 2024-02-01T10:27:36.042 DEBUG:boto:Params: {} 2024-02-01T10:27:36.042 DEBUG:boto:Token: None 2024-02-01T10:27:36.043 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:36 GMT /vclfcn-19/obj?versionId=eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr. 2024-02-01T10:27:36.043 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:xwXOvDUqH8vzv33D8KU/vb1EzxY= 2024-02-01T10:27:36.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:xwXOvDUqH8vzv33D8KU/vb1EzxY=', 'Content-Length': '0'} 2024-02-01T10:27:36.046 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008ac1df08d12e2bd3-0065bb7218-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:36.046 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:36.046 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:36.046 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:36.046 DEBUG:boto:path=/vclfcn-19/obj?versionId=eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr. 2024-02-01T10:27:36.046 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr. 2024-02-01T10:27:36.047 DEBUG:boto:Method: GET 2024-02-01T10:27:36.047 DEBUG:boto:Path: /vclfcn-19/obj?versionId=eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr. 2024-02-01T10:27:36.047 DEBUG:boto:Data: 2024-02-01T10:27:36.047 DEBUG:boto:Headers: {} 2024-02-01T10:27:36.047 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:36.047 DEBUG:boto:Port: 8000 2024-02-01T10:27:36.047 DEBUG:boto:Params: {} 2024-02-01T10:27:36.047 DEBUG:boto:Token: None 2024-02-01T10:27:36.047 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:36 GMT /vclfcn-19/obj?versionId=eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr. 2024-02-01T10:27:36.048 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:MqEZ4VrDZ/xQRdJNnU38gMwasbo= 2024-02-01T10:27:36.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:MqEZ4VrDZ/xQRdJNnU38gMwasbo=', 'Content-Length': '0'} 2024-02-01T10:27:36.050 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr.'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e120a9ebca6859a1-0065bb7218-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:36.092 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:36.093 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:36.093 DEBUG:boto:path=/vclfcn-19/obj?versionId=eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr. 2024-02-01T10:27:36.093 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr. 2024-02-01T10:27:36.093 DEBUG:boto:Method: GET 2024-02-01T10:27:36.093 DEBUG:boto:Path: /vclfcn-19/obj?versionId=eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr. 2024-02-01T10:27:36.093 DEBUG:boto:Data: 2024-02-01T10:27:36.093 DEBUG:boto:Headers: {} 2024-02-01T10:27:36.093 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:36.093 DEBUG:boto:Port: 8000 2024-02-01T10:27:36.094 DEBUG:boto:Params: {} 2024-02-01T10:27:36.094 DEBUG:boto:Token: None 2024-02-01T10:27:36.094 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:36 GMT /vclfcn-19/obj?versionId=eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr. 2024-02-01T10:27:36.094 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:MqEZ4VrDZ/xQRdJNnU38gMwasbo= 2024-02-01T10:27:36.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:MqEZ4VrDZ/xQRdJNnU38gMwasbo=', 'Content-Length': '0'} 2024-02-01T10:27:36.097 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000da2dba9477b55cb5-0065bb7218-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:36.098 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:36.098 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:36.098 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:36.098 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:36.098 DEBUG:boto:Method: GET 2024-02-01T10:27:36.098 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:36.098 DEBUG:boto:Data: 2024-02-01T10:27:36.098 DEBUG:boto:Headers: {} 2024-02-01T10:27:36.099 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:36.099 DEBUG:boto:Port: 8000 2024-02-01T10:27:36.099 DEBUG:boto:Params: {} 2024-02-01T10:27:36.099 DEBUG:boto:Token: None 2024-02-01T10:27:36.099 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:36 GMT /vclfcn-19/ 2024-02-01T10:27:36.099 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wLdNJrvhk+FzPKdT/ZcHWFu2CSY= 2024-02-01T10:27:36.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wLdNJrvhk+FzPKdT/ZcHWFu2CSY=', 'Content-Length': '0'} 2024-02-01T10:27:36.105 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e170f7cbf9787cb3-0065bb7218-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:36.105 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:36.106 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:36.106 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:36.106 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:36.106 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:36.106 DEBUG:boto:Method: GET 2024-02-01T10:27:36.106 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:36.106 DEBUG:boto:Data: 2024-02-01T10:27:36.106 DEBUG:boto:Headers: {} 2024-02-01T10:27:36.106 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:36.106 DEBUG:boto:Port: 8000 2024-02-01T10:27:36.106 DEBUG:boto:Params: {} 2024-02-01T10:27:36.107 DEBUG:boto:Token: None 2024-02-01T10:27:36.107 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:36 GMT /vclfcn-19/ 2024-02-01T10:27:36.107 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wLdNJrvhk+FzPKdT/ZcHWFu2CSY= 2024-02-01T10:27:36.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wLdNJrvhk+FzPKdT/ZcHWFu2CSY=', 'Content-Length': '0'} 2024-02-01T10:27:36.112 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023b9c55ce3b20a2d-0065bb7218-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:36.112 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:36.113 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:36.113 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:36.113 DEBUG:boto:Method: HEAD 2024-02-01T10:27:36.113 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:36.113 DEBUG:boto:Data: 2024-02-01T10:27:36.113 DEBUG:boto:Headers: {} 2024-02-01T10:27:36.113 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:36.113 DEBUG:boto:Port: 8000 2024-02-01T10:27:36.113 DEBUG:boto:Params: {} 2024-02-01T10:27:36.114 DEBUG:boto:Token: None 2024-02-01T10:27:36.114 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:36 GMT /vclfcn-19/obj 2024-02-01T10:27:36.114 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BKXlsRJGRLdYHaiqYwa5YxeBZ7A= 2024-02-01T10:27:36.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BKXlsRJGRLdYHaiqYwa5YxeBZ7A=', 'Content-Length': '0'} 2024-02-01T10:27:36.120 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000064e4ebaf4ec85d51-0065bb7218-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:36.120 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:36.120 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:36.120 DEBUG:boto:Method: HEAD 2024-02-01T10:27:36.121 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:36.121 DEBUG:boto:Data: 2024-02-01T10:27:36.121 DEBUG:boto:Headers: {} 2024-02-01T10:27:36.121 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:36.121 DEBUG:boto:Port: 8000 2024-02-01T10:27:36.121 DEBUG:boto:Params: {} 2024-02-01T10:27:36.121 DEBUG:boto:Token: None 2024-02-01T10:27:36.121 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:36 GMT /vclfcn-19/obj 2024-02-01T10:27:36.121 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BKXlsRJGRLdYHaiqYwa5YxeBZ7A= 2024-02-01T10:27:36.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BKXlsRJGRLdYHaiqYwa5YxeBZ7A=', 'Content-Length': '0'} 2024-02-01T10:27:36.127 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cd295fe17a097e9d-0065bb7218-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:36.127 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:36.127 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:36.127 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:36.127 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:36.128 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:36.128 DEBUG:boto:Method: GET 2024-02-01T10:27:36.128 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:36.128 DEBUG:boto:Data: 2024-02-01T10:27:36.128 DEBUG:boto:Headers: {} 2024-02-01T10:27:36.128 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:36.128 DEBUG:boto:Port: 8000 2024-02-01T10:27:36.128 DEBUG:boto:Params: {} 2024-02-01T10:27:36.128 DEBUG:boto:Token: None 2024-02-01T10:27:36.128 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:36 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:36.129 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:EmLUYXTFfxfeuWctLIWvjsOkHQ0= 2024-02-01T10:27:36.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:EmLUYXTFfxfeuWctLIWvjsOkHQ0=', 'Content-Length': '0'} 2024-02-01T10:27:36.132 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003986fb8233b93987-0065bb7218-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:36.173 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:36.173 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:36.173 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:36.173 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:36.173 DEBUG:boto:Method: GET 2024-02-01T10:27:36.173 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:36.174 DEBUG:boto:Data: 2024-02-01T10:27:36.174 DEBUG:boto:Headers: {} 2024-02-01T10:27:36.174 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:36.174 DEBUG:boto:Port: 8000 2024-02-01T10:27:36.174 DEBUG:boto:Params: {} 2024-02-01T10:27:36.174 DEBUG:boto:Token: None 2024-02-01T10:27:36.174 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:36 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:36.174 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:EmLUYXTFfxfeuWctLIWvjsOkHQ0= 2024-02-01T10:27:36.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:EmLUYXTFfxfeuWctLIWvjsOkHQ0=', 'Content-Length': '0'} 2024-02-01T10:27:36.179 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f840902e4c7549b8-0065bb7218-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:36.180 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:36.180 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:36.180 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:36.180 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:36.180 DEBUG:boto:Method: GET 2024-02-01T10:27:36.180 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:36.180 DEBUG:boto:Data: 2024-02-01T10:27:36.180 DEBUG:boto:Headers: {} 2024-02-01T10:27:36.181 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:36.181 DEBUG:boto:Port: 8000 2024-02-01T10:27:36.181 DEBUG:boto:Params: {} 2024-02-01T10:27:36.181 DEBUG:boto:Token: None 2024-02-01T10:27:36.181 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:36 GMT /vclfcn-19/ 2024-02-01T10:27:36.181 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wLdNJrvhk+FzPKdT/ZcHWFu2CSY= 2024-02-01T10:27:36.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wLdNJrvhk+FzPKdT/ZcHWFu2CSY=', 'Content-Length': '0'} 2024-02-01T10:27:36.186 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c14a5568e935c8b-0065bb7218-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:36.187 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:36.187 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:36.187 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:36.188 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:36.188 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:36.188 DEBUG:boto:Method: GET 2024-02-01T10:27:36.188 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:36.188 DEBUG:boto:Data: 2024-02-01T10:27:36.188 DEBUG:boto:Headers: {} 2024-02-01T10:27:36.188 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:36.188 DEBUG:boto:Port: 8000 2024-02-01T10:27:36.188 DEBUG:boto:Params: {} 2024-02-01T10:27:36.188 DEBUG:boto:Token: None 2024-02-01T10:27:36.189 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:36 GMT /vclfcn-19/ 2024-02-01T10:27:36.189 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wLdNJrvhk+FzPKdT/ZcHWFu2CSY= 2024-02-01T10:27:36.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wLdNJrvhk+FzPKdT/ZcHWFu2CSY=', 'Content-Length': '0'} 2024-02-01T10:27:36.194 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d01f271f1a0991ed-0065bb7218-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:36.195 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:36.195 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:36.195 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:36.195 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:36.195 DEBUG:boto:path=/vclfcn-19/obj?versionId=tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN 2024-02-01T10:27:36.195 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN 2024-02-01T10:27:36.196 DEBUG:boto:Method: GET 2024-02-01T10:27:36.196 DEBUG:boto:Path: /vclfcn-19/obj?versionId=tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN 2024-02-01T10:27:36.196 DEBUG:boto:Data: 2024-02-01T10:27:36.196 DEBUG:boto:Headers: {} 2024-02-01T10:27:36.196 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:36.196 DEBUG:boto:Port: 8000 2024-02-01T10:27:36.196 DEBUG:boto:Params: {} 2024-02-01T10:27:36.196 DEBUG:boto:Token: None 2024-02-01T10:27:36.196 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:36 GMT /vclfcn-19/obj?versionId=tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN 2024-02-01T10:27:36.197 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:760MmGDYK9hS3qsDJ34oCe4V8Zk= 2024-02-01T10:27:36.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:760MmGDYK9hS3qsDJ34oCe4V8Zk=', 'Content-Length': '0'} 2024-02-01T10:27:36.200 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f3f33ad27b6c84b6-0065bb7218-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:36.200 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:36.200 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:36.200 DEBUG:boto:path=/vclfcn-19/obj?versionId=tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN 2024-02-01T10:27:36.200 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN 2024-02-01T10:27:36.201 DEBUG:boto:Method: GET 2024-02-01T10:27:36.201 DEBUG:boto:Path: /vclfcn-19/obj?versionId=tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN 2024-02-01T10:27:36.201 DEBUG:boto:Data: 2024-02-01T10:27:36.201 DEBUG:boto:Headers: {} 2024-02-01T10:27:36.201 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:36.201 DEBUG:boto:Port: 8000 2024-02-01T10:27:36.201 DEBUG:boto:Params: {} 2024-02-01T10:27:36.201 DEBUG:boto:Token: None 2024-02-01T10:27:36.202 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:36 GMT /vclfcn-19/obj?versionId=tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN 2024-02-01T10:27:36.202 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:760MmGDYK9hS3qsDJ34oCe4V8Zk= 2024-02-01T10:27:36.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:760MmGDYK9hS3qsDJ34oCe4V8Zk=', 'Content-Length': '0'} 2024-02-01T10:27:36.205 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000025d8f313f51a29e6-0065bb7218-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:36.205 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:36.205 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:36.205 DEBUG:boto:path=/vclfcn-19/obj?versionId=tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN 2024-02-01T10:27:36.205 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN 2024-02-01T10:27:36.206 DEBUG:boto:Method: HEAD 2024-02-01T10:27:36.206 DEBUG:boto:Path: /vclfcn-19/obj?versionId=tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN 2024-02-01T10:27:36.206 DEBUG:boto:Data: 2024-02-01T10:27:36.206 DEBUG:boto:Headers: {} 2024-02-01T10:27:36.206 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:36.206 DEBUG:boto:Port: 8000 2024-02-01T10:27:36.206 DEBUG:boto:Params: {} 2024-02-01T10:27:36.207 DEBUG:boto:Token: None 2024-02-01T10:27:36.207 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:36 GMT /vclfcn-19/obj?versionId=tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN 2024-02-01T10:27:36.207 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Eku2/kq++4IS4dJsjs/DM5zDjIU= 2024-02-01T10:27:36.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Eku2/kq++4IS4dJsjs/DM5zDjIU=', 'Content-Length': '0'} 2024-02-01T10:27:36.209 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aad7ab24beede27a-0065bb7218-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:36.210 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:36.210 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:36.210 DEBUG:boto:path=/vclfcn-19/obj?versionId=tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN 2024-02-01T10:27:36.210 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN 2024-02-01T10:27:36.210 DEBUG:boto:Method: HEAD 2024-02-01T10:27:36.210 DEBUG:boto:Path: /vclfcn-19/obj?versionId=tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN 2024-02-01T10:27:36.210 DEBUG:boto:Data: 2024-02-01T10:27:36.210 DEBUG:boto:Headers: {} 2024-02-01T10:27:36.211 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:36.211 DEBUG:boto:Port: 8000 2024-02-01T10:27:36.211 DEBUG:boto:Params: {} 2024-02-01T10:27:36.211 DEBUG:boto:Token: None 2024-02-01T10:27:36.211 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:36 GMT /vclfcn-19/obj?versionId=tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN 2024-02-01T10:27:36.211 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Eku2/kq++4IS4dJsjs/DM5zDjIU= 2024-02-01T10:27:36.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Eku2/kq++4IS4dJsjs/DM5zDjIU=', 'Content-Length': '0'} 2024-02-01T10:27:36.214 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cabc0793523dc9ee-0065bb7218-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:36.214 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:36.214 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:36.214 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:36.215 DEBUG:boto:path=/vclfcn-19/obj?versionId=tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN 2024-02-01T10:27:36.215 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN 2024-02-01T10:27:36.215 DEBUG:boto:Method: GET 2024-02-01T10:27:36.215 DEBUG:boto:Path: /vclfcn-19/obj?versionId=tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN 2024-02-01T10:27:36.215 DEBUG:boto:Data: 2024-02-01T10:27:36.215 DEBUG:boto:Headers: {} 2024-02-01T10:27:36.215 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:36.215 DEBUG:boto:Port: 8000 2024-02-01T10:27:36.215 DEBUG:boto:Params: {} 2024-02-01T10:27:36.216 DEBUG:boto:Token: None 2024-02-01T10:27:36.216 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:36 GMT /vclfcn-19/obj?versionId=tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN 2024-02-01T10:27:36.216 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:760MmGDYK9hS3qsDJ34oCe4V8Zk= 2024-02-01T10:27:36.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:760MmGDYK9hS3qsDJ34oCe4V8Zk=', 'Content-Length': '0'} 2024-02-01T10:27:36.219 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000255b03eadf50cae5-0065bb7218-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:36.264 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:36.264 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:36.264 DEBUG:boto:path=/vclfcn-19/obj?versionId=tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN 2024-02-01T10:27:36.265 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN 2024-02-01T10:27:36.265 DEBUG:boto:Method: GET 2024-02-01T10:27:36.265 DEBUG:boto:Path: /vclfcn-19/obj?versionId=tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN 2024-02-01T10:27:36.265 DEBUG:boto:Data: 2024-02-01T10:27:36.265 DEBUG:boto:Headers: {} 2024-02-01T10:27:36.265 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:36.265 DEBUG:boto:Port: 8000 2024-02-01T10:27:36.265 DEBUG:boto:Params: {} 2024-02-01T10:27:36.265 DEBUG:boto:Token: None 2024-02-01T10:27:36.266 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:36 GMT /vclfcn-19/obj?versionId=tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN 2024-02-01T10:27:36.266 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:760MmGDYK9hS3qsDJ34oCe4V8Zk= 2024-02-01T10:27:36.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:760MmGDYK9hS3qsDJ34oCe4V8Zk=', 'Content-Length': '0'} 2024-02-01T10:27:36.269 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d863dfa5a854066f-0065bb7218-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:36.269 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:36.269 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:36.270 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:36.270 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:36.270 DEBUG:boto:Method: GET 2024-02-01T10:27:36.270 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:36.270 DEBUG:boto:Data: 2024-02-01T10:27:36.270 DEBUG:boto:Headers: {} 2024-02-01T10:27:36.270 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:36.270 DEBUG:boto:Port: 8000 2024-02-01T10:27:36.270 DEBUG:boto:Params: {} 2024-02-01T10:27:36.271 DEBUG:boto:Token: None 2024-02-01T10:27:36.271 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:36 GMT /vclfcn-19/ 2024-02-01T10:27:36.271 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wLdNJrvhk+FzPKdT/ZcHWFu2CSY= 2024-02-01T10:27:36.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wLdNJrvhk+FzPKdT/ZcHWFu2CSY=', 'Content-Length': '0'} 2024-02-01T10:27:36.276 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b23096505f12cbe-0065bb7218-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:36.276 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:36.276 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:36.277 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:36.277 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:36.277 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:36.277 DEBUG:boto:Method: GET 2024-02-01T10:27:36.277 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:36.277 DEBUG:boto:Data: 2024-02-01T10:27:36.277 DEBUG:boto:Headers: {} 2024-02-01T10:27:36.277 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:36.277 DEBUG:boto:Port: 8000 2024-02-01T10:27:36.277 DEBUG:boto:Params: {} 2024-02-01T10:27:36.278 DEBUG:boto:Token: None 2024-02-01T10:27:36.278 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:36 GMT /vclfcn-19/ 2024-02-01T10:27:36.278 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wLdNJrvhk+FzPKdT/ZcHWFu2CSY= 2024-02-01T10:27:36.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wLdNJrvhk+FzPKdT/ZcHWFu2CSY=', 'Content-Length': '0'} 2024-02-01T10:27:36.283 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed9b72f93ac86cd5-0065bb7218-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:36.284 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:36.284 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:36.284 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:36.284 DEBUG:boto:Method: HEAD 2024-02-01T10:27:36.284 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:36.284 DEBUG:boto:Data: 2024-02-01T10:27:36.285 DEBUG:boto:Headers: {} 2024-02-01T10:27:36.285 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:36.285 DEBUG:boto:Port: 8000 2024-02-01T10:27:36.285 DEBUG:boto:Params: {} 2024-02-01T10:27:36.285 DEBUG:boto:Token: None 2024-02-01T10:27:36.285 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:36 GMT /vclfcn-19/obj 2024-02-01T10:27:36.285 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BKXlsRJGRLdYHaiqYwa5YxeBZ7A= 2024-02-01T10:27:36.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BKXlsRJGRLdYHaiqYwa5YxeBZ7A=', 'Content-Length': '0'} 2024-02-01T10:27:36.291 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002f9f9062eb0319c3-0065bb7218-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:36.292 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:36.292 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:36.292 DEBUG:boto:Method: HEAD 2024-02-01T10:27:36.292 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:36.292 DEBUG:boto:Data: 2024-02-01T10:27:36.292 DEBUG:boto:Headers: {} 2024-02-01T10:27:36.292 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:36.292 DEBUG:boto:Port: 8000 2024-02-01T10:27:36.292 DEBUG:boto:Params: {} 2024-02-01T10:27:36.293 DEBUG:boto:Token: None 2024-02-01T10:27:36.293 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:36 GMT /vclfcn-19/obj 2024-02-01T10:27:36.293 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BKXlsRJGRLdYHaiqYwa5YxeBZ7A= 2024-02-01T10:27:36.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BKXlsRJGRLdYHaiqYwa5YxeBZ7A=', 'Content-Length': '0'} 2024-02-01T10:27:36.299 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009762bd96c9917979-0065bb7218-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:36.299 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:36.300 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:36.300 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:36.300 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:36.300 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:36.300 DEBUG:boto:Method: GET 2024-02-01T10:27:36.300 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:36.300 DEBUG:boto:Data: 2024-02-01T10:27:36.300 DEBUG:boto:Headers: {} 2024-02-01T10:27:36.300 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:36.300 DEBUG:boto:Port: 8000 2024-02-01T10:27:36.301 DEBUG:boto:Params: {} 2024-02-01T10:27:36.301 DEBUG:boto:Token: None 2024-02-01T10:27:36.301 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:36 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:36.301 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:EmLUYXTFfxfeuWctLIWvjsOkHQ0= 2024-02-01T10:27:36.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:EmLUYXTFfxfeuWctLIWvjsOkHQ0=', 'Content-Length': '0'} 2024-02-01T10:27:36.304 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004245dcc2e82aa24b-0065bb7218-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:36.348 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:36.348 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:36.349 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:36.349 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:36.349 DEBUG:boto:Method: GET 2024-02-01T10:27:36.349 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:36.349 DEBUG:boto:Data: 2024-02-01T10:27:36.349 DEBUG:boto:Headers: {} 2024-02-01T10:27:36.349 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:36.349 DEBUG:boto:Port: 8000 2024-02-01T10:27:36.349 DEBUG:boto:Params: {} 2024-02-01T10:27:36.349 DEBUG:boto:Token: None 2024-02-01T10:27:36.350 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:36 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:36.350 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:EmLUYXTFfxfeuWctLIWvjsOkHQ0= 2024-02-01T10:27:36.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:EmLUYXTFfxfeuWctLIWvjsOkHQ0=', 'Content-Length': '0'} 2024-02-01T10:27:36.353 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000055ed0cb92409b735-0065bb7218-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:36.353 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:36.353 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:36.353 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:36.353 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:36.353 DEBUG:boto:Method: GET 2024-02-01T10:27:36.353 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:36.354 DEBUG:boto:Data: 2024-02-01T10:27:36.354 DEBUG:boto:Headers: {} 2024-02-01T10:27:36.354 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:36.354 DEBUG:boto:Port: 8000 2024-02-01T10:27:36.354 DEBUG:boto:Params: {} 2024-02-01T10:27:36.354 DEBUG:boto:Token: None 2024-02-01T10:27:36.354 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:36 GMT /vclfcn-19/ 2024-02-01T10:27:36.354 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wLdNJrvhk+FzPKdT/ZcHWFu2CSY= 2024-02-01T10:27:36.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wLdNJrvhk+FzPKdT/ZcHWFu2CSY=', 'Content-Length': '0'} 2024-02-01T10:27:36.360 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000387f287e667b2d50-0065bb7218-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:36.360 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:36.360 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:36.361 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:36.361 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:36.361 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:36.361 DEBUG:boto:Method: GET 2024-02-01T10:27:36.361 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:36.361 DEBUG:boto:Data: 2024-02-01T10:27:36.361 DEBUG:boto:Headers: {} 2024-02-01T10:27:36.361 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:36.361 DEBUG:boto:Port: 8000 2024-02-01T10:27:36.361 DEBUG:boto:Params: {} 2024-02-01T10:27:36.362 DEBUG:boto:Token: None 2024-02-01T10:27:36.362 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:36 GMT /vclfcn-19/ 2024-02-01T10:27:36.362 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wLdNJrvhk+FzPKdT/ZcHWFu2CSY= 2024-02-01T10:27:36.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wLdNJrvhk+FzPKdT/ZcHWFu2CSY=', 'Content-Length': '0'} 2024-02-01T10:27:36.367 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3763a87602306f0-0065bb7218-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:36.408 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:36.409 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:36.409 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:36.409 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:36.409 DEBUG:boto:path=/vclfcn-19/obj?versionId=2lRbqumcCYnVWCIz20mU455245nXaz8 2024-02-01T10:27:36.409 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=2lRbqumcCYnVWCIz20mU455245nXaz8 2024-02-01T10:27:36.409 DEBUG:boto:Method: GET 2024-02-01T10:27:36.410 DEBUG:boto:Path: /vclfcn-19/obj?versionId=2lRbqumcCYnVWCIz20mU455245nXaz8 2024-02-01T10:27:36.410 DEBUG:boto:Data: 2024-02-01T10:27:36.410 DEBUG:boto:Headers: {} 2024-02-01T10:27:36.410 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:36.410 DEBUG:boto:Port: 8000 2024-02-01T10:27:36.410 DEBUG:boto:Params: {} 2024-02-01T10:27:36.411 DEBUG:boto:Token: None 2024-02-01T10:27:36.411 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:36 GMT /vclfcn-19/obj?versionId=2lRbqumcCYnVWCIz20mU455245nXaz8 2024-02-01T10:27:36.411 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:XnouYq61kVBMZkVmRumv405lVnI= 2024-02-01T10:27:36.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:XnouYq61kVBMZkVmRumv405lVnI=', 'Content-Length': '0'} 2024-02-01T10:27:36.414 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '2lRbqumcCYnVWCIz20mU455245nXaz8'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d80328248e77b61c-0065bb7218-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:36.415 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:36.415 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:36.416 DEBUG:boto:path=/vclfcn-19/obj?versionId=2lRbqumcCYnVWCIz20mU455245nXaz8 2024-02-01T10:27:36.416 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=2lRbqumcCYnVWCIz20mU455245nXaz8 2024-02-01T10:27:36.416 DEBUG:boto:Method: GET 2024-02-01T10:27:36.416 DEBUG:boto:Path: /vclfcn-19/obj?versionId=2lRbqumcCYnVWCIz20mU455245nXaz8 2024-02-01T10:27:36.416 DEBUG:boto:Data: 2024-02-01T10:27:36.416 DEBUG:boto:Headers: {} 2024-02-01T10:27:36.416 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:36.416 DEBUG:boto:Port: 8000 2024-02-01T10:27:36.416 DEBUG:boto:Params: {} 2024-02-01T10:27:36.417 DEBUG:boto:Token: None 2024-02-01T10:27:36.417 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:36 GMT /vclfcn-19/obj?versionId=2lRbqumcCYnVWCIz20mU455245nXaz8 2024-02-01T10:27:36.417 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:XnouYq61kVBMZkVmRumv405lVnI= 2024-02-01T10:27:36.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:XnouYq61kVBMZkVmRumv405lVnI=', 'Content-Length': '0'} 2024-02-01T10:27:36.420 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '2lRbqumcCYnVWCIz20mU455245nXaz8'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c23c4c50007e56d9-0065bb7218-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:36.421 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:36.421 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:36.421 DEBUG:boto:path=/vclfcn-19/obj?versionId=2lRbqumcCYnVWCIz20mU455245nXaz8 2024-02-01T10:27:36.421 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=2lRbqumcCYnVWCIz20mU455245nXaz8 2024-02-01T10:27:36.421 DEBUG:boto:Method: HEAD 2024-02-01T10:27:36.421 DEBUG:boto:Path: /vclfcn-19/obj?versionId=2lRbqumcCYnVWCIz20mU455245nXaz8 2024-02-01T10:27:36.421 DEBUG:boto:Data: 2024-02-01T10:27:36.421 DEBUG:boto:Headers: {} 2024-02-01T10:27:36.421 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:36.422 DEBUG:boto:Port: 8000 2024-02-01T10:27:36.422 DEBUG:boto:Params: {} 2024-02-01T10:27:36.422 DEBUG:boto:Token: None 2024-02-01T10:27:36.422 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:36 GMT /vclfcn-19/obj?versionId=2lRbqumcCYnVWCIz20mU455245nXaz8 2024-02-01T10:27:36.422 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:3qMeG5TlvdikEzGxIixEqG2qX1Y= 2024-02-01T10:27:36.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:3qMeG5TlvdikEzGxIixEqG2qX1Y=', 'Content-Length': '0'} 2024-02-01T10:27:36.425 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '2lRbqumcCYnVWCIz20mU455245nXaz8'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000007fe45536cfa0e7c-0065bb7218-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:36.426 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:36.426 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:36.426 DEBUG:boto:path=/vclfcn-19/obj?versionId=2lRbqumcCYnVWCIz20mU455245nXaz8 2024-02-01T10:27:36.426 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=2lRbqumcCYnVWCIz20mU455245nXaz8 2024-02-01T10:27:36.426 DEBUG:boto:Method: HEAD 2024-02-01T10:27:36.426 DEBUG:boto:Path: /vclfcn-19/obj?versionId=2lRbqumcCYnVWCIz20mU455245nXaz8 2024-02-01T10:27:36.427 DEBUG:boto:Data: 2024-02-01T10:27:36.427 DEBUG:boto:Headers: {} 2024-02-01T10:27:36.427 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:36.427 DEBUG:boto:Port: 8000 2024-02-01T10:27:36.427 DEBUG:boto:Params: {} 2024-02-01T10:27:36.427 DEBUG:boto:Token: None 2024-02-01T10:27:36.427 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:36 GMT /vclfcn-19/obj?versionId=2lRbqumcCYnVWCIz20mU455245nXaz8 2024-02-01T10:27:36.427 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:3qMeG5TlvdikEzGxIixEqG2qX1Y= 2024-02-01T10:27:36.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:3qMeG5TlvdikEzGxIixEqG2qX1Y=', 'Content-Length': '0'} 2024-02-01T10:27:36.430 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '2lRbqumcCYnVWCIz20mU455245nXaz8'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000086344d538eb269d6-0065bb7218-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:36.431 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:36.431 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:36.431 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:36.431 DEBUG:boto:path=/vclfcn-19/obj?versionId=2lRbqumcCYnVWCIz20mU455245nXaz8 2024-02-01T10:27:36.431 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=2lRbqumcCYnVWCIz20mU455245nXaz8 2024-02-01T10:27:36.431 DEBUG:boto:Method: GET 2024-02-01T10:27:36.431 DEBUG:boto:Path: /vclfcn-19/obj?versionId=2lRbqumcCYnVWCIz20mU455245nXaz8 2024-02-01T10:27:36.432 DEBUG:boto:Data: 2024-02-01T10:27:36.432 DEBUG:boto:Headers: {} 2024-02-01T10:27:36.432 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:36.432 DEBUG:boto:Port: 8000 2024-02-01T10:27:36.432 DEBUG:boto:Params: {} 2024-02-01T10:27:36.432 DEBUG:boto:Token: None 2024-02-01T10:27:36.432 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:36 GMT /vclfcn-19/obj?versionId=2lRbqumcCYnVWCIz20mU455245nXaz8 2024-02-01T10:27:36.432 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:XnouYq61kVBMZkVmRumv405lVnI= 2024-02-01T10:27:36.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:XnouYq61kVBMZkVmRumv405lVnI=', 'Content-Length': '0'} 2024-02-01T10:27:36.435 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '2lRbqumcCYnVWCIz20mU455245nXaz8'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ccb2fa8641f25f30-0065bb7218-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:36.476 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:36.476 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:36.476 DEBUG:boto:path=/vclfcn-19/obj?versionId=2lRbqumcCYnVWCIz20mU455245nXaz8 2024-02-01T10:27:36.476 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=2lRbqumcCYnVWCIz20mU455245nXaz8 2024-02-01T10:27:36.477 DEBUG:boto:Method: GET 2024-02-01T10:27:36.477 DEBUG:boto:Path: /vclfcn-19/obj?versionId=2lRbqumcCYnVWCIz20mU455245nXaz8 2024-02-01T10:27:36.477 DEBUG:boto:Data: 2024-02-01T10:27:36.477 DEBUG:boto:Headers: {} 2024-02-01T10:27:36.477 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:36.477 DEBUG:boto:Port: 8000 2024-02-01T10:27:36.477 DEBUG:boto:Params: {} 2024-02-01T10:27:36.477 DEBUG:boto:Token: None 2024-02-01T10:27:36.477 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:36 GMT /vclfcn-19/obj?versionId=2lRbqumcCYnVWCIz20mU455245nXaz8 2024-02-01T10:27:36.478 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:XnouYq61kVBMZkVmRumv405lVnI= 2024-02-01T10:27:36.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:XnouYq61kVBMZkVmRumv405lVnI=', 'Content-Length': '0'} 2024-02-01T10:27:36.481 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '2lRbqumcCYnVWCIz20mU455245nXaz8'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000efe520c79805585f-0065bb7218-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:36.481 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:36.481 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:36.481 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:36.481 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:36.481 DEBUG:boto:Method: GET 2024-02-01T10:27:36.481 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:36.481 DEBUG:boto:Data: 2024-02-01T10:27:36.482 DEBUG:boto:Headers: {} 2024-02-01T10:27:36.482 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:36.482 DEBUG:boto:Port: 8000 2024-02-01T10:27:36.482 DEBUG:boto:Params: {} 2024-02-01T10:27:36.482 DEBUG:boto:Token: None 2024-02-01T10:27:36.482 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:36 GMT /vclfcn-19/ 2024-02-01T10:27:36.482 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wLdNJrvhk+FzPKdT/ZcHWFu2CSY= 2024-02-01T10:27:36.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wLdNJrvhk+FzPKdT/ZcHWFu2CSY=', 'Content-Length': '0'} 2024-02-01T10:27:36.487 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008458ccf832907afb-0065bb7218-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:36.488 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:36.488 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:36.488 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:36.488 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:36.488 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:36.489 DEBUG:boto:Method: GET 2024-02-01T10:27:36.489 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:36.489 DEBUG:boto:Data: 2024-02-01T10:27:36.489 DEBUG:boto:Headers: {} 2024-02-01T10:27:36.489 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:36.489 DEBUG:boto:Port: 8000 2024-02-01T10:27:36.489 DEBUG:boto:Params: {} 2024-02-01T10:27:36.489 DEBUG:boto:Token: None 2024-02-01T10:27:36.489 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:36 GMT /vclfcn-19/ 2024-02-01T10:27:36.489 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wLdNJrvhk+FzPKdT/ZcHWFu2CSY= 2024-02-01T10:27:36.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wLdNJrvhk+FzPKdT/ZcHWFu2CSY=', 'Content-Length': '0'} 2024-02-01T10:27:36.495 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b76c08e9cdcd1af9-0065bb7218-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:36.536 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:36.536 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:36.537 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:36.537 DEBUG:boto:Method: HEAD 2024-02-01T10:27:36.537 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:36.537 DEBUG:boto:Data: 2024-02-01T10:27:36.537 DEBUG:boto:Headers: {} 2024-02-01T10:27:36.537 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:36.537 DEBUG:boto:Port: 8000 2024-02-01T10:27:36.537 DEBUG:boto:Params: {} 2024-02-01T10:27:36.537 DEBUG:boto:Token: None 2024-02-01T10:27:36.538 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:36 GMT /vclfcn-19/obj 2024-02-01T10:27:36.538 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BKXlsRJGRLdYHaiqYwa5YxeBZ7A= 2024-02-01T10:27:36.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BKXlsRJGRLdYHaiqYwa5YxeBZ7A=', 'Content-Length': '0'} 2024-02-01T10:27:36.543 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002929c351c505d793-0065bb7218-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:36.544 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:36.544 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:36.544 DEBUG:boto:Method: HEAD 2024-02-01T10:27:36.544 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:36.544 DEBUG:boto:Data: 2024-02-01T10:27:36.544 DEBUG:boto:Headers: {} 2024-02-01T10:27:36.544 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:36.544 DEBUG:boto:Port: 8000 2024-02-01T10:27:36.544 DEBUG:boto:Params: {} 2024-02-01T10:27:36.545 DEBUG:boto:Token: None 2024-02-01T10:27:36.545 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:36 GMT /vclfcn-19/obj 2024-02-01T10:27:36.545 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BKXlsRJGRLdYHaiqYwa5YxeBZ7A= 2024-02-01T10:27:36.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BKXlsRJGRLdYHaiqYwa5YxeBZ7A=', 'Content-Length': '0'} 2024-02-01T10:27:36.550 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000079cf439445804b9d-0065bb7218-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:36.551 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:36.551 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:36.551 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:36.551 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:36.551 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:36.551 DEBUG:boto:Method: GET 2024-02-01T10:27:36.551 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:36.551 DEBUG:boto:Data: 2024-02-01T10:27:36.552 DEBUG:boto:Headers: {} 2024-02-01T10:27:36.552 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:36.552 DEBUG:boto:Port: 8000 2024-02-01T10:27:36.552 DEBUG:boto:Params: {} 2024-02-01T10:27:36.552 DEBUG:boto:Token: None 2024-02-01T10:27:36.552 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:36 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:36.552 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:EmLUYXTFfxfeuWctLIWvjsOkHQ0= 2024-02-01T10:27:36.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:EmLUYXTFfxfeuWctLIWvjsOkHQ0=', 'Content-Length': '0'} 2024-02-01T10:27:36.555 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a9342763993e0802-0065bb7218-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:36.555 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:36.555 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:36.555 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:36.555 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:36.555 DEBUG:boto:Method: GET 2024-02-01T10:27:36.556 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:36.556 DEBUG:boto:Data: 2024-02-01T10:27:36.556 DEBUG:boto:Headers: {} 2024-02-01T10:27:36.556 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:36.556 DEBUG:boto:Port: 8000 2024-02-01T10:27:36.556 DEBUG:boto:Params: {} 2024-02-01T10:27:36.556 DEBUG:boto:Token: None 2024-02-01T10:27:36.556 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:36 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:36.556 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:EmLUYXTFfxfeuWctLIWvjsOkHQ0= 2024-02-01T10:27:36.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:EmLUYXTFfxfeuWctLIWvjsOkHQ0=', 'Content-Length': '0'} 2024-02-01T10:27:36.560 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000465b6fb8209ac1f3-0065bb7218-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:36.560 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:36.560 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:36.560 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:36.560 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:36.560 DEBUG:boto:Method: GET 2024-02-01T10:27:36.561 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:36.561 DEBUG:boto:Data: 2024-02-01T10:27:36.561 DEBUG:boto:Headers: {} 2024-02-01T10:27:36.561 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:36.561 DEBUG:boto:Port: 8000 2024-02-01T10:27:36.561 DEBUG:boto:Params: {} 2024-02-01T10:27:36.561 DEBUG:boto:Token: None 2024-02-01T10:27:36.561 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:36 GMT /vclfcn-19/ 2024-02-01T10:27:36.561 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wLdNJrvhk+FzPKdT/ZcHWFu2CSY= 2024-02-01T10:27:36.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wLdNJrvhk+FzPKdT/ZcHWFu2CSY=', 'Content-Length': '0'} 2024-02-01T10:27:36.567 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b4b149d2f42a6cd-0065bb7218-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:36.567 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:36.568 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:36.568 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:36.568 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:36.568 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:36.568 DEBUG:boto:Method: GET 2024-02-01T10:27:36.568 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:36.568 DEBUG:boto:Data: 2024-02-01T10:27:36.568 DEBUG:boto:Headers: {} 2024-02-01T10:27:36.569 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:36.569 DEBUG:boto:Port: 8000 2024-02-01T10:27:36.569 DEBUG:boto:Params: {} 2024-02-01T10:27:36.569 DEBUG:boto:Token: None 2024-02-01T10:27:36.569 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:36 GMT /vclfcn-19/ 2024-02-01T10:27:36.569 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wLdNJrvhk+FzPKdT/ZcHWFu2CSY= 2024-02-01T10:27:36.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wLdNJrvhk+FzPKdT/ZcHWFu2CSY=', 'Content-Length': '0'} 2024-02-01T10:27:36.574 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000759d523b008fb6f0-0065bb7218-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:36.575 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:36.575 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:36.575 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:36.575 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:36.575 DEBUG:boto:path=/vclfcn-19/obj?versionId=oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r 2024-02-01T10:27:36.575 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r 2024-02-01T10:27:36.575 DEBUG:boto:Method: GET 2024-02-01T10:27:36.575 DEBUG:boto:Path: /vclfcn-19/obj?versionId=oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r 2024-02-01T10:27:36.576 DEBUG:boto:Data: 2024-02-01T10:27:36.576 DEBUG:boto:Headers: {} 2024-02-01T10:27:36.576 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:36.576 DEBUG:boto:Port: 8000 2024-02-01T10:27:36.576 DEBUG:boto:Params: {} 2024-02-01T10:27:36.576 DEBUG:boto:Token: None 2024-02-01T10:27:36.576 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:36 GMT /vclfcn-19/obj?versionId=oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r 2024-02-01T10:27:36.576 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:PU1NlH5f+hNM0QIO1TeAkLNv3jI= 2024-02-01T10:27:36.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:PU1NlH5f+hNM0QIO1TeAkLNv3jI=', 'Content-Length': '0'} 2024-02-01T10:27:36.579 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000011349ae06475f9b1-0065bb7218-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:36.624 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:36.624 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:36.624 DEBUG:boto:path=/vclfcn-19/obj?versionId=oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r 2024-02-01T10:27:36.625 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r 2024-02-01T10:27:36.625 DEBUG:boto:Method: GET 2024-02-01T10:27:36.625 DEBUG:boto:Path: /vclfcn-19/obj?versionId=oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r 2024-02-01T10:27:36.625 DEBUG:boto:Data: 2024-02-01T10:27:36.625 DEBUG:boto:Headers: {} 2024-02-01T10:27:36.625 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:36.625 DEBUG:boto:Port: 8000 2024-02-01T10:27:36.625 DEBUG:boto:Params: {} 2024-02-01T10:27:36.625 DEBUG:boto:Token: None 2024-02-01T10:27:36.626 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:36 GMT /vclfcn-19/obj?versionId=oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r 2024-02-01T10:27:36.626 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:PU1NlH5f+hNM0QIO1TeAkLNv3jI= 2024-02-01T10:27:36.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:PU1NlH5f+hNM0QIO1TeAkLNv3jI=', 'Content-Length': '0'} 2024-02-01T10:27:36.629 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b1b4c5635f45bcbb-0065bb7218-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:36.629 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:36.629 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:36.629 DEBUG:boto:path=/vclfcn-19/obj?versionId=oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r 2024-02-01T10:27:36.630 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r 2024-02-01T10:27:36.630 DEBUG:boto:Method: HEAD 2024-02-01T10:27:36.630 DEBUG:boto:Path: /vclfcn-19/obj?versionId=oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r 2024-02-01T10:27:36.630 DEBUG:boto:Data: 2024-02-01T10:27:36.630 DEBUG:boto:Headers: {} 2024-02-01T10:27:36.630 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:36.630 DEBUG:boto:Port: 8000 2024-02-01T10:27:36.630 DEBUG:boto:Params: {} 2024-02-01T10:27:36.630 DEBUG:boto:Token: None 2024-02-01T10:27:36.631 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:36 GMT /vclfcn-19/obj?versionId=oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r 2024-02-01T10:27:36.631 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WKBcXvfJS2BFE1e0ck4eVL3Ekyk= 2024-02-01T10:27:36.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WKBcXvfJS2BFE1e0ck4eVL3Ekyk=', 'Content-Length': '0'} 2024-02-01T10:27:36.634 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000029ecca3e49f0de38-0065bb7218-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:36.634 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:36.634 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:36.634 DEBUG:boto:path=/vclfcn-19/obj?versionId=oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r 2024-02-01T10:27:36.634 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r 2024-02-01T10:27:36.634 DEBUG:boto:Method: HEAD 2024-02-01T10:27:36.635 DEBUG:boto:Path: /vclfcn-19/obj?versionId=oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r 2024-02-01T10:27:36.635 DEBUG:boto:Data: 2024-02-01T10:27:36.635 DEBUG:boto:Headers: {} 2024-02-01T10:27:36.635 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:36.635 DEBUG:boto:Port: 8000 2024-02-01T10:27:36.635 DEBUG:boto:Params: {} 2024-02-01T10:27:36.635 DEBUG:boto:Token: None 2024-02-01T10:27:36.635 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:36 GMT /vclfcn-19/obj?versionId=oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r 2024-02-01T10:27:36.635 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WKBcXvfJS2BFE1e0ck4eVL3Ekyk= 2024-02-01T10:27:36.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WKBcXvfJS2BFE1e0ck4eVL3Ekyk=', 'Content-Length': '0'} 2024-02-01T10:27:36.638 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aae8879e64a00bd0-0065bb7218-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:36.639 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:36.639 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:36.639 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:36.639 DEBUG:boto:path=/vclfcn-19/obj?versionId=oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r 2024-02-01T10:27:36.639 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r 2024-02-01T10:27:36.639 DEBUG:boto:Method: GET 2024-02-01T10:27:36.639 DEBUG:boto:Path: /vclfcn-19/obj?versionId=oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r 2024-02-01T10:27:36.639 DEBUG:boto:Data: 2024-02-01T10:27:36.639 DEBUG:boto:Headers: {} 2024-02-01T10:27:36.640 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:36.640 DEBUG:boto:Port: 8000 2024-02-01T10:27:36.640 DEBUG:boto:Params: {} 2024-02-01T10:27:36.640 DEBUG:boto:Token: None 2024-02-01T10:27:36.640 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:36 GMT /vclfcn-19/obj?versionId=oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r 2024-02-01T10:27:36.640 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:PU1NlH5f+hNM0QIO1TeAkLNv3jI= 2024-02-01T10:27:36.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:PU1NlH5f+hNM0QIO1TeAkLNv3jI=', 'Content-Length': '0'} 2024-02-01T10:27:36.643 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000730f082a7387da0c-0065bb7218-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:36.643 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:36.644 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:36.644 DEBUG:boto:path=/vclfcn-19/obj?versionId=oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r 2024-02-01T10:27:36.644 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r 2024-02-01T10:27:36.644 DEBUG:boto:Method: GET 2024-02-01T10:27:36.644 DEBUG:boto:Path: /vclfcn-19/obj?versionId=oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r 2024-02-01T10:27:36.644 DEBUG:boto:Data: 2024-02-01T10:27:36.644 DEBUG:boto:Headers: {} 2024-02-01T10:27:36.644 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:36.644 DEBUG:boto:Port: 8000 2024-02-01T10:27:36.644 DEBUG:boto:Params: {} 2024-02-01T10:27:36.645 DEBUG:boto:Token: None 2024-02-01T10:27:36.645 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:36 GMT /vclfcn-19/obj?versionId=oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r 2024-02-01T10:27:36.645 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:PU1NlH5f+hNM0QIO1TeAkLNv3jI= 2024-02-01T10:27:36.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:PU1NlH5f+hNM0QIO1TeAkLNv3jI=', 'Content-Length': '0'} 2024-02-01T10:27:36.648 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000728e4a64ed34edf1-0065bb7218-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:36.692 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:36.692 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:36.692 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:36.693 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:36.693 DEBUG:boto:Method: GET 2024-02-01T10:27:36.693 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:36.693 DEBUG:boto:Data: 2024-02-01T10:27:36.693 DEBUG:boto:Headers: {} 2024-02-01T10:27:36.693 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:36.693 DEBUG:boto:Port: 8000 2024-02-01T10:27:36.693 DEBUG:boto:Params: {} 2024-02-01T10:27:36.693 DEBUG:boto:Token: None 2024-02-01T10:27:36.693 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:36 GMT /vclfcn-19/ 2024-02-01T10:27:36.694 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wLdNJrvhk+FzPKdT/ZcHWFu2CSY= 2024-02-01T10:27:36.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wLdNJrvhk+FzPKdT/ZcHWFu2CSY=', 'Content-Length': '0'} 2024-02-01T10:27:36.701 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000364d17c02570d87e-0065bb7218-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:36.701 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:36.702 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:36.702 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:36.702 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:36.702 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:36.702 DEBUG:boto:Method: GET 2024-02-01T10:27:36.702 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:36.702 DEBUG:boto:Data: 2024-02-01T10:27:36.702 DEBUG:boto:Headers: {} 2024-02-01T10:27:36.703 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:36.703 DEBUG:boto:Port: 8000 2024-02-01T10:27:36.703 DEBUG:boto:Params: {} 2024-02-01T10:27:36.703 DEBUG:boto:Token: None 2024-02-01T10:27:36.703 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:36 GMT /vclfcn-19/ 2024-02-01T10:27:36.703 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wLdNJrvhk+FzPKdT/ZcHWFu2CSY= 2024-02-01T10:27:36.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wLdNJrvhk+FzPKdT/ZcHWFu2CSY=', 'Content-Length': '0'} 2024-02-01T10:27:36.711 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b72296d2232fd7bd-0065bb7218-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:36.712 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:36.712 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:36.712 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:36.713 DEBUG:boto:Method: HEAD 2024-02-01T10:27:36.713 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:36.713 DEBUG:boto:Data: 2024-02-01T10:27:36.713 DEBUG:boto:Headers: {} 2024-02-01T10:27:36.713 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:36.713 DEBUG:boto:Port: 8000 2024-02-01T10:27:36.713 DEBUG:boto:Params: {} 2024-02-01T10:27:36.713 DEBUG:boto:Token: None 2024-02-01T10:27:36.713 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:36 GMT /vclfcn-19/obj 2024-02-01T10:27:36.714 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BKXlsRJGRLdYHaiqYwa5YxeBZ7A= 2024-02-01T10:27:36.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BKXlsRJGRLdYHaiqYwa5YxeBZ7A=', 'Content-Length': '0'} 2024-02-01T10:27:36.720 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000002336b38f2bb2bbc-0065bb7218-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:36.720 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:36.720 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:36.720 DEBUG:boto:Method: HEAD 2024-02-01T10:27:36.720 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:36.721 DEBUG:boto:Data: 2024-02-01T10:27:36.721 DEBUG:boto:Headers: {} 2024-02-01T10:27:36.721 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:36.721 DEBUG:boto:Port: 8000 2024-02-01T10:27:36.721 DEBUG:boto:Params: {} 2024-02-01T10:27:36.721 DEBUG:boto:Token: None 2024-02-01T10:27:36.721 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:36 GMT /vclfcn-19/obj 2024-02-01T10:27:36.721 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BKXlsRJGRLdYHaiqYwa5YxeBZ7A= 2024-02-01T10:27:36.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BKXlsRJGRLdYHaiqYwa5YxeBZ7A=', 'Content-Length': '0'} 2024-02-01T10:27:36.727 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000297f24498e52c79b-0065bb7218-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:36.728 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:36.728 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:36.728 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:36.728 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:36.728 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:36.728 DEBUG:boto:Method: GET 2024-02-01T10:27:36.728 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:36.728 DEBUG:boto:Data: 2024-02-01T10:27:36.729 DEBUG:boto:Headers: {} 2024-02-01T10:27:36.729 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:36.729 DEBUG:boto:Port: 8000 2024-02-01T10:27:36.729 DEBUG:boto:Params: {} 2024-02-01T10:27:36.729 DEBUG:boto:Token: None 2024-02-01T10:27:36.729 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:36 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:36.729 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:EmLUYXTFfxfeuWctLIWvjsOkHQ0= 2024-02-01T10:27:36.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:EmLUYXTFfxfeuWctLIWvjsOkHQ0=', 'Content-Length': '0'} 2024-02-01T10:27:36.733 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bf5a0254df38fd09-0065bb7218-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:36.780 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:36.781 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:36.781 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:36.781 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:36.781 DEBUG:boto:Method: GET 2024-02-01T10:27:36.781 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:36.781 DEBUG:boto:Data: 2024-02-01T10:27:36.781 DEBUG:boto:Headers: {} 2024-02-01T10:27:36.781 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:36.781 DEBUG:boto:Port: 8000 2024-02-01T10:27:36.781 DEBUG:boto:Params: {} 2024-02-01T10:27:36.781 DEBUG:boto:Token: None 2024-02-01T10:27:36.782 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:36 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:36.782 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:EmLUYXTFfxfeuWctLIWvjsOkHQ0= 2024-02-01T10:27:36.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:EmLUYXTFfxfeuWctLIWvjsOkHQ0=', 'Content-Length': '0'} 2024-02-01T10:27:36.785 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000da9122be72bef73f-0065bb7218-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:36.785 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:36.785 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:36.786 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:36.786 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:36.786 DEBUG:boto:Method: GET 2024-02-01T10:27:36.786 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:36.786 DEBUG:boto:Data: 2024-02-01T10:27:36.786 DEBUG:boto:Headers: {} 2024-02-01T10:27:36.786 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:36.786 DEBUG:boto:Port: 8000 2024-02-01T10:27:36.786 DEBUG:boto:Params: {} 2024-02-01T10:27:36.786 DEBUG:boto:Token: None 2024-02-01T10:27:36.787 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:36 GMT /vclfcn-19/ 2024-02-01T10:27:36.787 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wLdNJrvhk+FzPKdT/ZcHWFu2CSY= 2024-02-01T10:27:36.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wLdNJrvhk+FzPKdT/ZcHWFu2CSY=', 'Content-Length': '0'} 2024-02-01T10:27:36.792 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a625422897a290d1-0065bb7218-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:36.792 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:36.793 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:36.793 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:36.793 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:36.793 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:36.793 DEBUG:boto:Method: GET 2024-02-01T10:27:36.793 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:36.793 DEBUG:boto:Data: 2024-02-01T10:27:36.793 DEBUG:boto:Headers: {} 2024-02-01T10:27:36.793 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:36.793 DEBUG:boto:Port: 8000 2024-02-01T10:27:36.794 DEBUG:boto:Params: {} 2024-02-01T10:27:36.794 DEBUG:boto:Token: None 2024-02-01T10:27:36.794 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:36 GMT /vclfcn-19/ 2024-02-01T10:27:36.794 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wLdNJrvhk+FzPKdT/ZcHWFu2CSY= 2024-02-01T10:27:36.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wLdNJrvhk+FzPKdT/ZcHWFu2CSY=', 'Content-Length': '0'} 2024-02-01T10:27:36.799 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000111dd76d37dd5fb6-0065bb7218-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:36.840 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:36.841 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:36.841 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:36.841 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:36.841 DEBUG:boto:path=/vclfcn-19/obj?versionId=qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9 2024-02-01T10:27:36.841 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9 2024-02-01T10:27:36.841 DEBUG:boto:Method: GET 2024-02-01T10:27:36.841 DEBUG:boto:Path: /vclfcn-19/obj?versionId=qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9 2024-02-01T10:27:36.841 DEBUG:boto:Data: 2024-02-01T10:27:36.841 DEBUG:boto:Headers: {} 2024-02-01T10:27:36.842 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:36.842 DEBUG:boto:Port: 8000 2024-02-01T10:27:36.842 DEBUG:boto:Params: {} 2024-02-01T10:27:36.842 DEBUG:boto:Token: None 2024-02-01T10:27:36.842 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:36 GMT /vclfcn-19/obj?versionId=qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9 2024-02-01T10:27:36.842 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:TLE7aCXQyhhTvJ3yBymGRxh/3Gc= 2024-02-01T10:27:36.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:TLE7aCXQyhhTvJ3yBymGRxh/3Gc=', 'Content-Length': '0'} 2024-02-01T10:27:36.845 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aff2b12f8cd547e5-0065bb7218-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:36.846 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:36.846 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:36.846 DEBUG:boto:path=/vclfcn-19/obj?versionId=qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9 2024-02-01T10:27:36.846 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9 2024-02-01T10:27:36.846 DEBUG:boto:Method: GET 2024-02-01T10:27:36.846 DEBUG:boto:Path: /vclfcn-19/obj?versionId=qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9 2024-02-01T10:27:36.846 DEBUG:boto:Data: 2024-02-01T10:27:36.846 DEBUG:boto:Headers: {} 2024-02-01T10:27:36.846 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:36.846 DEBUG:boto:Port: 8000 2024-02-01T10:27:36.846 DEBUG:boto:Params: {} 2024-02-01T10:27:36.847 DEBUG:boto:Token: None 2024-02-01T10:27:36.847 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:36 GMT /vclfcn-19/obj?versionId=qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9 2024-02-01T10:27:36.847 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:TLE7aCXQyhhTvJ3yBymGRxh/3Gc= 2024-02-01T10:27:36.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:TLE7aCXQyhhTvJ3yBymGRxh/3Gc=', 'Content-Length': '0'} 2024-02-01T10:27:36.850 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003f47fd4ba16de02d-0065bb7218-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:36.850 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:36.850 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:36.850 DEBUG:boto:path=/vclfcn-19/obj?versionId=qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9 2024-02-01T10:27:36.850 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9 2024-02-01T10:27:36.850 DEBUG:boto:Method: HEAD 2024-02-01T10:27:36.851 DEBUG:boto:Path: /vclfcn-19/obj?versionId=qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9 2024-02-01T10:27:36.851 DEBUG:boto:Data: 2024-02-01T10:27:36.851 DEBUG:boto:Headers: {} 2024-02-01T10:27:36.851 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:36.851 DEBUG:boto:Port: 8000 2024-02-01T10:27:36.851 DEBUG:boto:Params: {} 2024-02-01T10:27:36.851 DEBUG:boto:Token: None 2024-02-01T10:27:36.851 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:36 GMT /vclfcn-19/obj?versionId=qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9 2024-02-01T10:27:36.851 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:DGJmCYV9Pw6MIaJ3+VSFLPBsRFo= 2024-02-01T10:27:36.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:DGJmCYV9Pw6MIaJ3+VSFLPBsRFo=', 'Content-Length': '0'} 2024-02-01T10:27:36.854 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000889ebeb9b97e65cc-0065bb7218-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:36.854 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:36.855 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:36.855 DEBUG:boto:path=/vclfcn-19/obj?versionId=qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9 2024-02-01T10:27:36.855 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9 2024-02-01T10:27:36.855 DEBUG:boto:Method: HEAD 2024-02-01T10:27:36.855 DEBUG:boto:Path: /vclfcn-19/obj?versionId=qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9 2024-02-01T10:27:36.855 DEBUG:boto:Data: 2024-02-01T10:27:36.855 DEBUG:boto:Headers: {} 2024-02-01T10:27:36.855 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:36.855 DEBUG:boto:Port: 8000 2024-02-01T10:27:36.855 DEBUG:boto:Params: {} 2024-02-01T10:27:36.855 DEBUG:boto:Token: None 2024-02-01T10:27:36.856 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:36 GMT /vclfcn-19/obj?versionId=qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9 2024-02-01T10:27:36.856 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:DGJmCYV9Pw6MIaJ3+VSFLPBsRFo= 2024-02-01T10:27:36.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:DGJmCYV9Pw6MIaJ3+VSFLPBsRFo=', 'Content-Length': '0'} 2024-02-01T10:27:36.858 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004b1742e59bf61a75-0065bb7218-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:36.859 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:36.859 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:36.859 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:36.859 DEBUG:boto:path=/vclfcn-19/obj?versionId=qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9 2024-02-01T10:27:36.859 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9 2024-02-01T10:27:36.859 DEBUG:boto:Method: GET 2024-02-01T10:27:36.860 DEBUG:boto:Path: /vclfcn-19/obj?versionId=qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9 2024-02-01T10:27:36.860 DEBUG:boto:Data: 2024-02-01T10:27:36.860 DEBUG:boto:Headers: {} 2024-02-01T10:27:36.860 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:36.860 DEBUG:boto:Port: 8000 2024-02-01T10:27:36.860 DEBUG:boto:Params: {} 2024-02-01T10:27:36.860 DEBUG:boto:Token: None 2024-02-01T10:27:36.860 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:36 GMT /vclfcn-19/obj?versionId=qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9 2024-02-01T10:27:36.860 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:TLE7aCXQyhhTvJ3yBymGRxh/3Gc= 2024-02-01T10:27:36.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:TLE7aCXQyhhTvJ3yBymGRxh/3Gc=', 'Content-Length': '0'} 2024-02-01T10:27:36.863 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c34d75eb17beafbf-0065bb7218-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:36.864 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:36.864 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:36.864 DEBUG:boto:path=/vclfcn-19/obj?versionId=qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9 2024-02-01T10:27:36.864 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9 2024-02-01T10:27:36.864 DEBUG:boto:Method: GET 2024-02-01T10:27:36.864 DEBUG:boto:Path: /vclfcn-19/obj?versionId=qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9 2024-02-01T10:27:36.864 DEBUG:boto:Data: 2024-02-01T10:27:36.864 DEBUG:boto:Headers: {} 2024-02-01T10:27:36.864 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:36.865 DEBUG:boto:Port: 8000 2024-02-01T10:27:36.865 DEBUG:boto:Params: {} 2024-02-01T10:27:36.865 DEBUG:boto:Token: None 2024-02-01T10:27:36.865 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:36 GMT /vclfcn-19/obj?versionId=qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9 2024-02-01T10:27:36.865 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:TLE7aCXQyhhTvJ3yBymGRxh/3Gc= 2024-02-01T10:27:36.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:TLE7aCXQyhhTvJ3yBymGRxh/3Gc=', 'Content-Length': '0'} 2024-02-01T10:27:36.868 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000a80d6ce772a0f9d-0065bb7218-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:36.868 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:36.868 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:36.868 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:36.869 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:36.869 DEBUG:boto:Method: GET 2024-02-01T10:27:36.869 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:36.869 DEBUG:boto:Data: 2024-02-01T10:27:36.869 DEBUG:boto:Headers: {} 2024-02-01T10:27:36.869 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:36.869 DEBUG:boto:Port: 8000 2024-02-01T10:27:36.869 DEBUG:boto:Params: {} 2024-02-01T10:27:36.869 DEBUG:boto:Token: None 2024-02-01T10:27:36.869 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:36 GMT /vclfcn-19/ 2024-02-01T10:27:36.870 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wLdNJrvhk+FzPKdT/ZcHWFu2CSY= 2024-02-01T10:27:36.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wLdNJrvhk+FzPKdT/ZcHWFu2CSY=', 'Content-Length': '0'} 2024-02-01T10:27:36.875 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000077c4d57c5826b1e-0065bb7218-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:36.875 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:36.875 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:36.875 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:36.876 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:36.876 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:36.876 DEBUG:boto:Method: GET 2024-02-01T10:27:36.876 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:36.876 DEBUG:boto:Data: 2024-02-01T10:27:36.876 DEBUG:boto:Headers: {} 2024-02-01T10:27:36.876 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:36.876 DEBUG:boto:Port: 8000 2024-02-01T10:27:36.876 DEBUG:boto:Params: {} 2024-02-01T10:27:36.876 DEBUG:boto:Token: None 2024-02-01T10:27:36.877 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:36 GMT /vclfcn-19/ 2024-02-01T10:27:36.877 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wLdNJrvhk+FzPKdT/ZcHWFu2CSY= 2024-02-01T10:27:36.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wLdNJrvhk+FzPKdT/ZcHWFu2CSY=', 'Content-Length': '0'} 2024-02-01T10:27:36.882 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e74e94ed12f012e7-0065bb7218-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:36.924 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:36.925 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:36.925 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:36.925 DEBUG:boto:Method: HEAD 2024-02-01T10:27:36.925 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:36.925 DEBUG:boto:Data: 2024-02-01T10:27:36.925 DEBUG:boto:Headers: {} 2024-02-01T10:27:36.925 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:36.925 DEBUG:boto:Port: 8000 2024-02-01T10:27:36.925 DEBUG:boto:Params: {} 2024-02-01T10:27:36.926 DEBUG:boto:Token: None 2024-02-01T10:27:36.926 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:36 GMT /vclfcn-19/obj 2024-02-01T10:27:36.926 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BKXlsRJGRLdYHaiqYwa5YxeBZ7A= 2024-02-01T10:27:36.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BKXlsRJGRLdYHaiqYwa5YxeBZ7A=', 'Content-Length': '0'} 2024-02-01T10:27:36.932 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000083147d29dffabc7d-0065bb7218-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:36.932 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:36.932 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:36.933 DEBUG:boto:Method: HEAD 2024-02-01T10:27:36.933 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:36.933 DEBUG:boto:Data: 2024-02-01T10:27:36.933 DEBUG:boto:Headers: {} 2024-02-01T10:27:36.933 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:36.933 DEBUG:boto:Port: 8000 2024-02-01T10:27:36.933 DEBUG:boto:Params: {} 2024-02-01T10:27:36.933 DEBUG:boto:Token: None 2024-02-01T10:27:36.933 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:36 GMT /vclfcn-19/obj 2024-02-01T10:27:36.933 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BKXlsRJGRLdYHaiqYwa5YxeBZ7A= 2024-02-01T10:27:36.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BKXlsRJGRLdYHaiqYwa5YxeBZ7A=', 'Content-Length': '0'} 2024-02-01T10:27:36.940 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f4988604c0af1979-0065bb7218-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:36.940 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:36.940 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:36.941 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:36.941 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:36.941 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:36.941 DEBUG:boto:Method: GET 2024-02-01T10:27:36.941 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:36.941 DEBUG:boto:Data: 2024-02-01T10:27:36.941 DEBUG:boto:Headers: {} 2024-02-01T10:27:36.941 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:36.941 DEBUG:boto:Port: 8000 2024-02-01T10:27:36.941 DEBUG:boto:Params: {} 2024-02-01T10:27:36.941 DEBUG:boto:Token: None 2024-02-01T10:27:36.942 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:36 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:36.942 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:EmLUYXTFfxfeuWctLIWvjsOkHQ0= 2024-02-01T10:27:36.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:EmLUYXTFfxfeuWctLIWvjsOkHQ0=', 'Content-Length': '0'} 2024-02-01T10:27:36.945 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000035bdf65d3dfbc9f3-0065bb7218-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:36.988 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:36.988 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:36.988 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:36.989 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:36.989 DEBUG:boto:Method: GET 2024-02-01T10:27:36.989 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:36.989 DEBUG:boto:Data: 2024-02-01T10:27:36.989 DEBUG:boto:Headers: {} 2024-02-01T10:27:36.989 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:36.989 DEBUG:boto:Port: 8000 2024-02-01T10:27:36.989 DEBUG:boto:Params: {} 2024-02-01T10:27:36.989 DEBUG:boto:Token: None 2024-02-01T10:27:36.989 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:36 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:36.990 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:EmLUYXTFfxfeuWctLIWvjsOkHQ0= 2024-02-01T10:27:36.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:EmLUYXTFfxfeuWctLIWvjsOkHQ0=', 'Content-Length': '0'} 2024-02-01T10:27:36.993 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000072749c1f419afd40-0065bb7218-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:36.993 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:36.993 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:36.994 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:36.994 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:36.994 DEBUG:boto:Method: GET 2024-02-01T10:27:36.994 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:36.994 DEBUG:boto:Data: 2024-02-01T10:27:36.994 DEBUG:boto:Headers: {} 2024-02-01T10:27:36.994 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:36.994 DEBUG:boto:Port: 8000 2024-02-01T10:27:36.994 DEBUG:boto:Params: {} 2024-02-01T10:27:36.994 DEBUG:boto:Token: None 2024-02-01T10:27:36.994 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:36 GMT /vclfcn-19/ 2024-02-01T10:27:36.995 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wLdNJrvhk+FzPKdT/ZcHWFu2CSY= 2024-02-01T10:27:36.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wLdNJrvhk+FzPKdT/ZcHWFu2CSY=', 'Content-Length': '0'} 2024-02-01T10:27:36.999 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f502d8d447cb3861-0065bb7219-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:37.000 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:37.000 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:37.000 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:37.000 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:37.000 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:37.001 DEBUG:boto:Method: GET 2024-02-01T10:27:37.001 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:37.001 DEBUG:boto:Data: 2024-02-01T10:27:37.001 DEBUG:boto:Headers: {} 2024-02-01T10:27:37.001 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:37.001 DEBUG:boto:Port: 8000 2024-02-01T10:27:37.001 DEBUG:boto:Params: {} 2024-02-01T10:27:37.002 DEBUG:boto:Token: None 2024-02-01T10:27:37.002 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:37 GMT /vclfcn-19/ 2024-02-01T10:27:37.002 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:86djPIsytSY6O2p5N7dgCxQYYmo= 2024-02-01T10:27:37.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:86djPIsytSY6O2p5N7dgCxQYYmo=', 'Content-Length': '0'} 2024-02-01T10:27:37.007 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5d671fe8c559ae5-0065bb7219-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:37.008 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:37.008 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:37.008 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:37.008 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:37.008 DEBUG:boto:path=/vclfcn-19/obj?versionId=NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS 2024-02-01T10:27:37.008 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS 2024-02-01T10:27:37.009 DEBUG:boto:Method: GET 2024-02-01T10:27:37.009 DEBUG:boto:Path: /vclfcn-19/obj?versionId=NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS 2024-02-01T10:27:37.009 DEBUG:boto:Data: 2024-02-01T10:27:37.009 DEBUG:boto:Headers: {} 2024-02-01T10:27:37.009 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:37.009 DEBUG:boto:Port: 8000 2024-02-01T10:27:37.009 DEBUG:boto:Params: {} 2024-02-01T10:27:37.009 DEBUG:boto:Token: None 2024-02-01T10:27:37.009 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:37 GMT /vclfcn-19/obj?versionId=NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS 2024-02-01T10:27:37.009 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:EMzUo5F80zqO6oot4WfXKP/1Q5Y= 2024-02-01T10:27:37.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:EMzUo5F80zqO6oot4WfXKP/1Q5Y=', 'Content-Length': '0'} 2024-02-01T10:27:37.012 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000085800b321cc026ee-0065bb7219-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:37.012 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:37.012 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:37.013 DEBUG:boto:path=/vclfcn-19/obj?versionId=NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS 2024-02-01T10:27:37.013 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS 2024-02-01T10:27:37.013 DEBUG:boto:Method: GET 2024-02-01T10:27:37.013 DEBUG:boto:Path: /vclfcn-19/obj?versionId=NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS 2024-02-01T10:27:37.013 DEBUG:boto:Data: 2024-02-01T10:27:37.013 DEBUG:boto:Headers: {} 2024-02-01T10:27:37.013 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:37.013 DEBUG:boto:Port: 8000 2024-02-01T10:27:37.013 DEBUG:boto:Params: {} 2024-02-01T10:27:37.014 DEBUG:boto:Token: None 2024-02-01T10:27:37.014 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:37 GMT /vclfcn-19/obj?versionId=NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS 2024-02-01T10:27:37.014 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:EMzUo5F80zqO6oot4WfXKP/1Q5Y= 2024-02-01T10:27:37.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:EMzUo5F80zqO6oot4WfXKP/1Q5Y=', 'Content-Length': '0'} 2024-02-01T10:27:37.017 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f14e2211bc08d8db-0065bb7219-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:37.060 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:37.060 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:37.060 DEBUG:boto:path=/vclfcn-19/obj?versionId=NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS 2024-02-01T10:27:37.061 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS 2024-02-01T10:27:37.061 DEBUG:boto:Method: HEAD 2024-02-01T10:27:37.061 DEBUG:boto:Path: /vclfcn-19/obj?versionId=NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS 2024-02-01T10:27:37.061 DEBUG:boto:Data: 2024-02-01T10:27:37.061 DEBUG:boto:Headers: {} 2024-02-01T10:27:37.061 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:37.061 DEBUG:boto:Port: 8000 2024-02-01T10:27:37.061 DEBUG:boto:Params: {} 2024-02-01T10:27:37.061 DEBUG:boto:Token: None 2024-02-01T10:27:37.062 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:37 GMT /vclfcn-19/obj?versionId=NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS 2024-02-01T10:27:37.062 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:g2uAhK3JGS5DghiWqWyWZYEju9M= 2024-02-01T10:27:37.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:g2uAhK3JGS5DghiWqWyWZYEju9M=', 'Content-Length': '0'} 2024-02-01T10:27:37.065 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e4570fd98e4a9db9-0065bb7219-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:37.065 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:37.065 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:37.065 DEBUG:boto:path=/vclfcn-19/obj?versionId=NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS 2024-02-01T10:27:37.065 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS 2024-02-01T10:27:37.065 DEBUG:boto:Method: HEAD 2024-02-01T10:27:37.065 DEBUG:boto:Path: /vclfcn-19/obj?versionId=NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS 2024-02-01T10:27:37.065 DEBUG:boto:Data: 2024-02-01T10:27:37.066 DEBUG:boto:Headers: {} 2024-02-01T10:27:37.066 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:37.066 DEBUG:boto:Port: 8000 2024-02-01T10:27:37.066 DEBUG:boto:Params: {} 2024-02-01T10:27:37.066 DEBUG:boto:Token: None 2024-02-01T10:27:37.066 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:37 GMT /vclfcn-19/obj?versionId=NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS 2024-02-01T10:27:37.066 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:g2uAhK3JGS5DghiWqWyWZYEju9M= 2024-02-01T10:27:37.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:g2uAhK3JGS5DghiWqWyWZYEju9M=', 'Content-Length': '0'} 2024-02-01T10:27:37.069 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000060ae69c1b28a41d8-0065bb7219-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:37.069 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:37.069 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:37.070 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:37.070 DEBUG:boto:path=/vclfcn-19/obj?versionId=NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS 2024-02-01T10:27:37.070 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS 2024-02-01T10:27:37.070 DEBUG:boto:Method: GET 2024-02-01T10:27:37.070 DEBUG:boto:Path: /vclfcn-19/obj?versionId=NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS 2024-02-01T10:27:37.070 DEBUG:boto:Data: 2024-02-01T10:27:37.070 DEBUG:boto:Headers: {} 2024-02-01T10:27:37.070 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:37.070 DEBUG:boto:Port: 8000 2024-02-01T10:27:37.070 DEBUG:boto:Params: {} 2024-02-01T10:27:37.071 DEBUG:boto:Token: None 2024-02-01T10:27:37.071 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:37 GMT /vclfcn-19/obj?versionId=NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS 2024-02-01T10:27:37.071 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:EMzUo5F80zqO6oot4WfXKP/1Q5Y= 2024-02-01T10:27:37.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:EMzUo5F80zqO6oot4WfXKP/1Q5Y=', 'Content-Length': '0'} 2024-02-01T10:27:37.074 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000579569c94654112e-0065bb7219-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:37.116 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:37.116 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:37.117 DEBUG:boto:path=/vclfcn-19/obj?versionId=NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS 2024-02-01T10:27:37.117 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS 2024-02-01T10:27:37.117 DEBUG:boto:Method: GET 2024-02-01T10:27:37.117 DEBUG:boto:Path: /vclfcn-19/obj?versionId=NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS 2024-02-01T10:27:37.117 DEBUG:boto:Data: 2024-02-01T10:27:37.117 DEBUG:boto:Headers: {} 2024-02-01T10:27:37.117 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:37.117 DEBUG:boto:Port: 8000 2024-02-01T10:27:37.117 DEBUG:boto:Params: {} 2024-02-01T10:27:37.117 DEBUG:boto:Token: None 2024-02-01T10:27:37.118 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:37 GMT /vclfcn-19/obj?versionId=NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS 2024-02-01T10:27:37.118 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:EMzUo5F80zqO6oot4WfXKP/1Q5Y= 2024-02-01T10:27:37.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:EMzUo5F80zqO6oot4WfXKP/1Q5Y=', 'Content-Length': '0'} 2024-02-01T10:27:37.121 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000055d8163e63d9c837-0065bb7219-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:37.122 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:37.122 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:37.122 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:37.122 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:37.122 DEBUG:boto:Method: GET 2024-02-01T10:27:37.122 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:37.122 DEBUG:boto:Data: 2024-02-01T10:27:37.122 DEBUG:boto:Headers: {} 2024-02-01T10:27:37.122 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:37.123 DEBUG:boto:Port: 8000 2024-02-01T10:27:37.123 DEBUG:boto:Params: {} 2024-02-01T10:27:37.123 DEBUG:boto:Token: None 2024-02-01T10:27:37.123 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:37 GMT /vclfcn-19/ 2024-02-01T10:27:37.123 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:86djPIsytSY6O2p5N7dgCxQYYmo= 2024-02-01T10:27:37.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:86djPIsytSY6O2p5N7dgCxQYYmo=', 'Content-Length': '0'} 2024-02-01T10:27:37.128 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000682b131b273c09e5-0065bb7219-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:37.128 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:37.129 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:37.129 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:37.129 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:37.129 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:37.129 DEBUG:boto:Method: GET 2024-02-01T10:27:37.129 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:37.129 DEBUG:boto:Data: 2024-02-01T10:27:37.129 DEBUG:boto:Headers: {} 2024-02-01T10:27:37.129 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:37.129 DEBUG:boto:Port: 8000 2024-02-01T10:27:37.130 DEBUG:boto:Params: {} 2024-02-01T10:27:37.130 DEBUG:boto:Token: None 2024-02-01T10:27:37.130 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:37 GMT /vclfcn-19/ 2024-02-01T10:27:37.130 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:86djPIsytSY6O2p5N7dgCxQYYmo= 2024-02-01T10:27:37.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:86djPIsytSY6O2p5N7dgCxQYYmo=', 'Content-Length': '0'} 2024-02-01T10:27:37.135 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008bbd52fe4634524c-0065bb7219-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:37.135 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:37.136 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:37.136 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:37.136 DEBUG:boto:Method: HEAD 2024-02-01T10:27:37.136 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:37.136 DEBUG:boto:Data: 2024-02-01T10:27:37.136 DEBUG:boto:Headers: {} 2024-02-01T10:27:37.136 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:37.137 DEBUG:boto:Port: 8000 2024-02-01T10:27:37.137 DEBUG:boto:Params: {} 2024-02-01T10:27:37.137 DEBUG:boto:Token: None 2024-02-01T10:27:37.137 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:37 GMT /vclfcn-19/obj 2024-02-01T10:27:37.137 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wkLqM5BextiMhvsV2a/ruNi9Jqk= 2024-02-01T10:27:37.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wkLqM5BextiMhvsV2a/ruNi9Jqk=', 'Content-Length': '0'} 2024-02-01T10:27:37.143 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000850d7e85dda7371b-0065bb7219-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:37.143 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:37.143 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:37.143 DEBUG:boto:Method: HEAD 2024-02-01T10:27:37.144 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:37.144 DEBUG:boto:Data: 2024-02-01T10:27:37.144 DEBUG:boto:Headers: {} 2024-02-01T10:27:37.144 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:37.144 DEBUG:boto:Port: 8000 2024-02-01T10:27:37.144 DEBUG:boto:Params: {} 2024-02-01T10:27:37.144 DEBUG:boto:Token: None 2024-02-01T10:27:37.144 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:37 GMT /vclfcn-19/obj 2024-02-01T10:27:37.144 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wkLqM5BextiMhvsV2a/ruNi9Jqk= 2024-02-01T10:27:37.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wkLqM5BextiMhvsV2a/ruNi9Jqk=', 'Content-Length': '0'} 2024-02-01T10:27:37.150 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a99a7bc313ff4a86-0065bb7219-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:37.150 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:37.151 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:37.151 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:37.151 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:37.151 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:37.151 DEBUG:boto:Method: GET 2024-02-01T10:27:37.151 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:37.151 DEBUG:boto:Data: 2024-02-01T10:27:37.151 DEBUG:boto:Headers: {} 2024-02-01T10:27:37.151 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:37.151 DEBUG:boto:Port: 8000 2024-02-01T10:27:37.151 DEBUG:boto:Params: {} 2024-02-01T10:27:37.152 DEBUG:boto:Token: None 2024-02-01T10:27:37.152 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:37 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:37.152 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nuuTVX3bkxnNPwltDIjKuORhjRQ= 2024-02-01T10:27:37.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nuuTVX3bkxnNPwltDIjKuORhjRQ=', 'Content-Length': '0'} 2024-02-01T10:27:37.155 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000feb745076950a213-0065bb7219-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:37.200 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:37.200 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:37.201 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:37.201 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:37.201 DEBUG:boto:Method: GET 2024-02-01T10:27:37.201 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:37.201 DEBUG:boto:Data: 2024-02-01T10:27:37.201 DEBUG:boto:Headers: {} 2024-02-01T10:27:37.201 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:37.201 DEBUG:boto:Port: 8000 2024-02-01T10:27:37.201 DEBUG:boto:Params: {} 2024-02-01T10:27:37.202 DEBUG:boto:Token: None 2024-02-01T10:27:37.202 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:37 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:37.202 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nuuTVX3bkxnNPwltDIjKuORhjRQ= 2024-02-01T10:27:37.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nuuTVX3bkxnNPwltDIjKuORhjRQ=', 'Content-Length': '0'} 2024-02-01T10:27:37.205 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005eadb6e6c2d07b7c-0065bb7219-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:37.206 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:37.206 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:37.206 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:37.206 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:37.206 DEBUG:boto:Method: GET 2024-02-01T10:27:37.206 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:37.206 DEBUG:boto:Data: 2024-02-01T10:27:37.206 DEBUG:boto:Headers: {} 2024-02-01T10:27:37.206 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:37.206 DEBUG:boto:Port: 8000 2024-02-01T10:27:37.207 DEBUG:boto:Params: {} 2024-02-01T10:27:37.207 DEBUG:boto:Token: None 2024-02-01T10:27:37.207 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:37 GMT /vclfcn-19/ 2024-02-01T10:27:37.207 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:86djPIsytSY6O2p5N7dgCxQYYmo= 2024-02-01T10:27:37.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:86djPIsytSY6O2p5N7dgCxQYYmo=', 'Content-Length': '0'} 2024-02-01T10:27:37.213 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000478f2edd74603486-0065bb7219-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:37.213 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:37.214 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:37.214 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:37.214 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:37.214 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:37.214 DEBUG:boto:Method: GET 2024-02-01T10:27:37.214 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:37.214 DEBUG:boto:Data: 2024-02-01T10:27:37.214 DEBUG:boto:Headers: {} 2024-02-01T10:27:37.215 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:37.215 DEBUG:boto:Port: 8000 2024-02-01T10:27:37.215 DEBUG:boto:Params: {} 2024-02-01T10:27:37.215 DEBUG:boto:Token: None 2024-02-01T10:27:37.215 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:37 GMT /vclfcn-19/ 2024-02-01T10:27:37.215 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:86djPIsytSY6O2p5N7dgCxQYYmo= 2024-02-01T10:27:37.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:86djPIsytSY6O2p5N7dgCxQYYmo=', 'Content-Length': '0'} 2024-02-01T10:27:37.220 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9c59d5e12aa0709-0065bb7219-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:37.221 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:37.221 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:37.221 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:37.221 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:37.221 DEBUG:boto:path=/vclfcn-19/obj?versionId=VQT2kBCkF07dugmyrhNeRpQpR0HK7bT 2024-02-01T10:27:37.221 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=VQT2kBCkF07dugmyrhNeRpQpR0HK7bT 2024-02-01T10:27:37.222 DEBUG:boto:Method: GET 2024-02-01T10:27:37.222 DEBUG:boto:Path: /vclfcn-19/obj?versionId=VQT2kBCkF07dugmyrhNeRpQpR0HK7bT 2024-02-01T10:27:37.222 DEBUG:boto:Data: 2024-02-01T10:27:37.222 DEBUG:boto:Headers: {} 2024-02-01T10:27:37.222 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:37.222 DEBUG:boto:Port: 8000 2024-02-01T10:27:37.222 DEBUG:boto:Params: {} 2024-02-01T10:27:37.222 DEBUG:boto:Token: None 2024-02-01T10:27:37.222 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:37 GMT /vclfcn-19/obj?versionId=VQT2kBCkF07dugmyrhNeRpQpR0HK7bT 2024-02-01T10:27:37.222 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:b/hDQOQLh3wJUJHvvaETMLNesVE= 2024-02-01T10:27:37.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:b/hDQOQLh3wJUJHvvaETMLNesVE=', 'Content-Length': '0'} 2024-02-01T10:27:37.225 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'VQT2kBCkF07dugmyrhNeRpQpR0HK7bT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000071dfe28dfba7e418-0065bb7219-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:37.226 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:37.226 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:37.226 DEBUG:boto:path=/vclfcn-19/obj?versionId=VQT2kBCkF07dugmyrhNeRpQpR0HK7bT 2024-02-01T10:27:37.226 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=VQT2kBCkF07dugmyrhNeRpQpR0HK7bT 2024-02-01T10:27:37.226 DEBUG:boto:Method: GET 2024-02-01T10:27:37.226 DEBUG:boto:Path: /vclfcn-19/obj?versionId=VQT2kBCkF07dugmyrhNeRpQpR0HK7bT 2024-02-01T10:27:37.226 DEBUG:boto:Data: 2024-02-01T10:27:37.226 DEBUG:boto:Headers: {} 2024-02-01T10:27:37.227 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:37.227 DEBUG:boto:Port: 8000 2024-02-01T10:27:37.227 DEBUG:boto:Params: {} 2024-02-01T10:27:37.227 DEBUG:boto:Token: None 2024-02-01T10:27:37.227 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:37 GMT /vclfcn-19/obj?versionId=VQT2kBCkF07dugmyrhNeRpQpR0HK7bT 2024-02-01T10:27:37.227 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:b/hDQOQLh3wJUJHvvaETMLNesVE= 2024-02-01T10:27:37.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:b/hDQOQLh3wJUJHvvaETMLNesVE=', 'Content-Length': '0'} 2024-02-01T10:27:37.230 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'VQT2kBCkF07dugmyrhNeRpQpR0HK7bT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000be1a92220c78b5a9-0065bb7219-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:37.231 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:37.231 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:37.231 DEBUG:boto:path=/vclfcn-19/obj?versionId=VQT2kBCkF07dugmyrhNeRpQpR0HK7bT 2024-02-01T10:27:37.231 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=VQT2kBCkF07dugmyrhNeRpQpR0HK7bT 2024-02-01T10:27:37.231 DEBUG:boto:Method: HEAD 2024-02-01T10:27:37.231 DEBUG:boto:Path: /vclfcn-19/obj?versionId=VQT2kBCkF07dugmyrhNeRpQpR0HK7bT 2024-02-01T10:27:37.231 DEBUG:boto:Data: 2024-02-01T10:27:37.231 DEBUG:boto:Headers: {} 2024-02-01T10:27:37.231 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:37.232 DEBUG:boto:Port: 8000 2024-02-01T10:27:37.232 DEBUG:boto:Params: {} 2024-02-01T10:27:37.232 DEBUG:boto:Token: None 2024-02-01T10:27:37.232 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:37 GMT /vclfcn-19/obj?versionId=VQT2kBCkF07dugmyrhNeRpQpR0HK7bT 2024-02-01T10:27:37.232 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:AMp3naxHz4RFiydbXmCLtPglOiA= 2024-02-01T10:27:37.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:AMp3naxHz4RFiydbXmCLtPglOiA=', 'Content-Length': '0'} 2024-02-01T10:27:37.235 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'VQT2kBCkF07dugmyrhNeRpQpR0HK7bT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006caa8659ac0c5a13-0065bb7219-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:37.235 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:37.236 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:37.236 DEBUG:boto:path=/vclfcn-19/obj?versionId=VQT2kBCkF07dugmyrhNeRpQpR0HK7bT 2024-02-01T10:27:37.236 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=VQT2kBCkF07dugmyrhNeRpQpR0HK7bT 2024-02-01T10:27:37.236 DEBUG:boto:Method: HEAD 2024-02-01T10:27:37.236 DEBUG:boto:Path: /vclfcn-19/obj?versionId=VQT2kBCkF07dugmyrhNeRpQpR0HK7bT 2024-02-01T10:27:37.236 DEBUG:boto:Data: 2024-02-01T10:27:37.236 DEBUG:boto:Headers: {} 2024-02-01T10:27:37.236 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:37.236 DEBUG:boto:Port: 8000 2024-02-01T10:27:37.236 DEBUG:boto:Params: {} 2024-02-01T10:27:37.237 DEBUG:boto:Token: None 2024-02-01T10:27:37.237 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:37 GMT /vclfcn-19/obj?versionId=VQT2kBCkF07dugmyrhNeRpQpR0HK7bT 2024-02-01T10:27:37.237 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:AMp3naxHz4RFiydbXmCLtPglOiA= 2024-02-01T10:27:37.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:AMp3naxHz4RFiydbXmCLtPglOiA=', 'Content-Length': '0'} 2024-02-01T10:27:37.240 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'VQT2kBCkF07dugmyrhNeRpQpR0HK7bT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b1ab7652991f2ec7-0065bb7219-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:37.240 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:37.240 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:37.240 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:37.241 DEBUG:boto:path=/vclfcn-19/obj?versionId=VQT2kBCkF07dugmyrhNeRpQpR0HK7bT 2024-02-01T10:27:37.241 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=VQT2kBCkF07dugmyrhNeRpQpR0HK7bT 2024-02-01T10:27:37.241 DEBUG:boto:Method: GET 2024-02-01T10:27:37.241 DEBUG:boto:Path: /vclfcn-19/obj?versionId=VQT2kBCkF07dugmyrhNeRpQpR0HK7bT 2024-02-01T10:27:37.241 DEBUG:boto:Data: 2024-02-01T10:27:37.241 DEBUG:boto:Headers: {} 2024-02-01T10:27:37.242 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:37.242 DEBUG:boto:Port: 8000 2024-02-01T10:27:37.242 DEBUG:boto:Params: {} 2024-02-01T10:27:37.242 DEBUG:boto:Token: None 2024-02-01T10:27:37.242 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:37 GMT /vclfcn-19/obj?versionId=VQT2kBCkF07dugmyrhNeRpQpR0HK7bT 2024-02-01T10:27:37.242 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:b/hDQOQLh3wJUJHvvaETMLNesVE= 2024-02-01T10:27:37.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:b/hDQOQLh3wJUJHvvaETMLNesVE=', 'Content-Length': '0'} 2024-02-01T10:27:37.245 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'VQT2kBCkF07dugmyrhNeRpQpR0HK7bT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e2bbd2a7a71a559f-0065bb7219-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:37.288 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:37.288 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:37.288 DEBUG:boto:path=/vclfcn-19/obj?versionId=VQT2kBCkF07dugmyrhNeRpQpR0HK7bT 2024-02-01T10:27:37.288 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=VQT2kBCkF07dugmyrhNeRpQpR0HK7bT 2024-02-01T10:27:37.289 DEBUG:boto:Method: GET 2024-02-01T10:27:37.289 DEBUG:boto:Path: /vclfcn-19/obj?versionId=VQT2kBCkF07dugmyrhNeRpQpR0HK7bT 2024-02-01T10:27:37.289 DEBUG:boto:Data: 2024-02-01T10:27:37.289 DEBUG:boto:Headers: {} 2024-02-01T10:27:37.289 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:37.289 DEBUG:boto:Port: 8000 2024-02-01T10:27:37.289 DEBUG:boto:Params: {} 2024-02-01T10:27:37.289 DEBUG:boto:Token: None 2024-02-01T10:27:37.289 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:37 GMT /vclfcn-19/obj?versionId=VQT2kBCkF07dugmyrhNeRpQpR0HK7bT 2024-02-01T10:27:37.289 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:b/hDQOQLh3wJUJHvvaETMLNesVE= 2024-02-01T10:27:37.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:b/hDQOQLh3wJUJHvvaETMLNesVE=', 'Content-Length': '0'} 2024-02-01T10:27:37.292 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'VQT2kBCkF07dugmyrhNeRpQpR0HK7bT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000004a3c50e252d6674-0065bb7219-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:37.293 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:37.293 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:37.293 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:37.293 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:37.293 DEBUG:boto:Method: GET 2024-02-01T10:27:37.293 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:37.293 DEBUG:boto:Data: 2024-02-01T10:27:37.293 DEBUG:boto:Headers: {} 2024-02-01T10:27:37.294 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:37.294 DEBUG:boto:Port: 8000 2024-02-01T10:27:37.294 DEBUG:boto:Params: {} 2024-02-01T10:27:37.294 DEBUG:boto:Token: None 2024-02-01T10:27:37.294 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:37 GMT /vclfcn-19/ 2024-02-01T10:27:37.294 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:86djPIsytSY6O2p5N7dgCxQYYmo= 2024-02-01T10:27:37.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:86djPIsytSY6O2p5N7dgCxQYYmo=', 'Content-Length': '0'} 2024-02-01T10:27:37.299 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001200430bd662d995-0065bb7219-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:37.299 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:37.300 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:37.300 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:37.300 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:37.300 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:37.300 DEBUG:boto:Method: GET 2024-02-01T10:27:37.300 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:37.300 DEBUG:boto:Data: 2024-02-01T10:27:37.300 DEBUG:boto:Headers: {} 2024-02-01T10:27:37.301 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:37.301 DEBUG:boto:Port: 8000 2024-02-01T10:27:37.301 DEBUG:boto:Params: {} 2024-02-01T10:27:37.301 DEBUG:boto:Token: None 2024-02-01T10:27:37.301 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:37 GMT /vclfcn-19/ 2024-02-01T10:27:37.301 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:86djPIsytSY6O2p5N7dgCxQYYmo= 2024-02-01T10:27:37.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:86djPIsytSY6O2p5N7dgCxQYYmo=', 'Content-Length': '0'} 2024-02-01T10:27:37.306 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007217dfbe3266a431-0065bb7219-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:37.306 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:37.306 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:37.306 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:37.307 DEBUG:boto:Method: HEAD 2024-02-01T10:27:37.307 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:37.307 DEBUG:boto:Data: 2024-02-01T10:27:37.307 DEBUG:boto:Headers: {} 2024-02-01T10:27:37.307 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:37.307 DEBUG:boto:Port: 8000 2024-02-01T10:27:37.307 DEBUG:boto:Params: {} 2024-02-01T10:27:37.307 DEBUG:boto:Token: None 2024-02-01T10:27:37.307 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:37 GMT /vclfcn-19/obj 2024-02-01T10:27:37.307 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wkLqM5BextiMhvsV2a/ruNi9Jqk= 2024-02-01T10:27:37.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wkLqM5BextiMhvsV2a/ruNi9Jqk=', 'Content-Length': '0'} 2024-02-01T10:27:37.314 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005e38af23be0373a6-0065bb7219-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:37.314 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:37.314 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:37.314 DEBUG:boto:Method: HEAD 2024-02-01T10:27:37.314 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:37.315 DEBUG:boto:Data: 2024-02-01T10:27:37.315 DEBUG:boto:Headers: {} 2024-02-01T10:27:37.315 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:37.315 DEBUG:boto:Port: 8000 2024-02-01T10:27:37.315 DEBUG:boto:Params: {} 2024-02-01T10:27:37.315 DEBUG:boto:Token: None 2024-02-01T10:27:37.315 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:37 GMT /vclfcn-19/obj 2024-02-01T10:27:37.315 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wkLqM5BextiMhvsV2a/ruNi9Jqk= 2024-02-01T10:27:37.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wkLqM5BextiMhvsV2a/ruNi9Jqk=', 'Content-Length': '0'} 2024-02-01T10:27:37.321 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002c77600570a542ff-0065bb7219-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:37.321 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:37.321 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:37.322 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:37.322 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:37.322 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:37.322 DEBUG:boto:Method: GET 2024-02-01T10:27:37.322 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:37.322 DEBUG:boto:Data: 2024-02-01T10:27:37.322 DEBUG:boto:Headers: {} 2024-02-01T10:27:37.322 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:37.322 DEBUG:boto:Port: 8000 2024-02-01T10:27:37.322 DEBUG:boto:Params: {} 2024-02-01T10:27:37.323 DEBUG:boto:Token: None 2024-02-01T10:27:37.323 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:37 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:37.323 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nuuTVX3bkxnNPwltDIjKuORhjRQ= 2024-02-01T10:27:37.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nuuTVX3bkxnNPwltDIjKuORhjRQ=', 'Content-Length': '0'} 2024-02-01T10:27:37.326 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ed91ebffd7f5fdd2-0065bb7219-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:37.368 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:37.368 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:37.368 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:37.369 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:37.369 DEBUG:boto:Method: GET 2024-02-01T10:27:37.369 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:37.369 DEBUG:boto:Data: 2024-02-01T10:27:37.369 DEBUG:boto:Headers: {} 2024-02-01T10:27:37.369 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:37.369 DEBUG:boto:Port: 8000 2024-02-01T10:27:37.369 DEBUG:boto:Params: {} 2024-02-01T10:27:37.369 DEBUG:boto:Token: None 2024-02-01T10:27:37.370 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:37 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:37.370 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nuuTVX3bkxnNPwltDIjKuORhjRQ= 2024-02-01T10:27:37.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nuuTVX3bkxnNPwltDIjKuORhjRQ=', 'Content-Length': '0'} 2024-02-01T10:27:37.373 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008892d5539d9dbc44-0065bb7219-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:37.373 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:37.373 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:37.373 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:37.373 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:37.373 DEBUG:boto:Method: GET 2024-02-01T10:27:37.374 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:37.374 DEBUG:boto:Data: 2024-02-01T10:27:37.374 DEBUG:boto:Headers: {} 2024-02-01T10:27:37.374 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:37.374 DEBUG:boto:Port: 8000 2024-02-01T10:27:37.374 DEBUG:boto:Params: {} 2024-02-01T10:27:37.374 DEBUG:boto:Token: None 2024-02-01T10:27:37.374 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:37 GMT /vclfcn-19/ 2024-02-01T10:27:37.374 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:86djPIsytSY6O2p5N7dgCxQYYmo= 2024-02-01T10:27:37.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:86djPIsytSY6O2p5N7dgCxQYYmo=', 'Content-Length': '0'} 2024-02-01T10:27:37.379 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004347aaae1f0fabe7-0065bb7219-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:37.380 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:37.380 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:37.380 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:37.380 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:37.380 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:37.380 DEBUG:boto:Method: GET 2024-02-01T10:27:37.381 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:37.381 DEBUG:boto:Data: 2024-02-01T10:27:37.381 DEBUG:boto:Headers: {} 2024-02-01T10:27:37.381 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:37.381 DEBUG:boto:Port: 8000 2024-02-01T10:27:37.381 DEBUG:boto:Params: {} 2024-02-01T10:27:37.381 DEBUG:boto:Token: None 2024-02-01T10:27:37.381 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:37 GMT /vclfcn-19/ 2024-02-01T10:27:37.381 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:86djPIsytSY6O2p5N7dgCxQYYmo= 2024-02-01T10:27:37.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:86djPIsytSY6O2p5N7dgCxQYYmo=', 'Content-Length': '0'} 2024-02-01T10:27:37.387 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d4b003df8cf8445-0065bb7219-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:37.428 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:37.429 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:37.429 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:37.429 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:37.429 DEBUG:boto:path=/vclfcn-19/obj?versionId=tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi 2024-02-01T10:27:37.429 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi 2024-02-01T10:27:37.429 DEBUG:boto:Method: GET 2024-02-01T10:27:37.429 DEBUG:boto:Path: /vclfcn-19/obj?versionId=tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi 2024-02-01T10:27:37.429 DEBUG:boto:Data: 2024-02-01T10:27:37.429 DEBUG:boto:Headers: {} 2024-02-01T10:27:37.430 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:37.430 DEBUG:boto:Port: 8000 2024-02-01T10:27:37.430 DEBUG:boto:Params: {} 2024-02-01T10:27:37.430 DEBUG:boto:Token: None 2024-02-01T10:27:37.430 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:37 GMT /vclfcn-19/obj?versionId=tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi 2024-02-01T10:27:37.430 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:P5/HqCkPh2dsStKgCKTkTOtCb2U= 2024-02-01T10:27:37.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:P5/HqCkPh2dsStKgCKTkTOtCb2U=', 'Content-Length': '0'} 2024-02-01T10:27:37.433 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000618662077503974c-0065bb7219-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:37.434 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:37.434 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:37.434 DEBUG:boto:path=/vclfcn-19/obj?versionId=tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi 2024-02-01T10:27:37.434 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi 2024-02-01T10:27:37.434 DEBUG:boto:Method: GET 2024-02-01T10:27:37.434 DEBUG:boto:Path: /vclfcn-19/obj?versionId=tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi 2024-02-01T10:27:37.434 DEBUG:boto:Data: 2024-02-01T10:27:37.434 DEBUG:boto:Headers: {} 2024-02-01T10:27:37.435 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:37.435 DEBUG:boto:Port: 8000 2024-02-01T10:27:37.435 DEBUG:boto:Params: {} 2024-02-01T10:27:37.435 DEBUG:boto:Token: None 2024-02-01T10:27:37.435 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:37 GMT /vclfcn-19/obj?versionId=tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi 2024-02-01T10:27:37.435 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:P5/HqCkPh2dsStKgCKTkTOtCb2U= 2024-02-01T10:27:37.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:P5/HqCkPh2dsStKgCKTkTOtCb2U=', 'Content-Length': '0'} 2024-02-01T10:27:37.438 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006fe8018591d97c1e-0065bb7219-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:37.439 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:37.439 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:37.439 DEBUG:boto:path=/vclfcn-19/obj?versionId=tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi 2024-02-01T10:27:37.439 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi 2024-02-01T10:27:37.439 DEBUG:boto:Method: HEAD 2024-02-01T10:27:37.439 DEBUG:boto:Path: /vclfcn-19/obj?versionId=tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi 2024-02-01T10:27:37.439 DEBUG:boto:Data: 2024-02-01T10:27:37.439 DEBUG:boto:Headers: {} 2024-02-01T10:27:37.439 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:37.439 DEBUG:boto:Port: 8000 2024-02-01T10:27:37.439 DEBUG:boto:Params: {} 2024-02-01T10:27:37.440 DEBUG:boto:Token: None 2024-02-01T10:27:37.440 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:37 GMT /vclfcn-19/obj?versionId=tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi 2024-02-01T10:27:37.440 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8S4Wz4nSZLHwCU/bqo5VV1CLTP4= 2024-02-01T10:27:37.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8S4Wz4nSZLHwCU/bqo5VV1CLTP4=', 'Content-Length': '0'} 2024-02-01T10:27:37.443 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008cdc86ad90ac12ae-0065bb7219-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:37.443 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:37.443 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:37.443 DEBUG:boto:path=/vclfcn-19/obj?versionId=tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi 2024-02-01T10:27:37.443 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi 2024-02-01T10:27:37.444 DEBUG:boto:Method: HEAD 2024-02-01T10:27:37.444 DEBUG:boto:Path: /vclfcn-19/obj?versionId=tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi 2024-02-01T10:27:37.444 DEBUG:boto:Data: 2024-02-01T10:27:37.444 DEBUG:boto:Headers: {} 2024-02-01T10:27:37.444 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:37.444 DEBUG:boto:Port: 8000 2024-02-01T10:27:37.444 DEBUG:boto:Params: {} 2024-02-01T10:27:37.444 DEBUG:boto:Token: None 2024-02-01T10:27:37.444 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:37 GMT /vclfcn-19/obj?versionId=tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi 2024-02-01T10:27:37.445 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8S4Wz4nSZLHwCU/bqo5VV1CLTP4= 2024-02-01T10:27:37.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8S4Wz4nSZLHwCU/bqo5VV1CLTP4=', 'Content-Length': '0'} 2024-02-01T10:27:37.447 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f554801b27fea695-0065bb7219-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:37.447 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:37.448 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:37.448 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:37.448 DEBUG:boto:path=/vclfcn-19/obj?versionId=tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi 2024-02-01T10:27:37.448 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi 2024-02-01T10:27:37.448 DEBUG:boto:Method: GET 2024-02-01T10:27:37.448 DEBUG:boto:Path: /vclfcn-19/obj?versionId=tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi 2024-02-01T10:27:37.448 DEBUG:boto:Data: 2024-02-01T10:27:37.448 DEBUG:boto:Headers: {} 2024-02-01T10:27:37.448 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:37.448 DEBUG:boto:Port: 8000 2024-02-01T10:27:37.448 DEBUG:boto:Params: {} 2024-02-01T10:27:37.449 DEBUG:boto:Token: None 2024-02-01T10:27:37.449 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:37 GMT /vclfcn-19/obj?versionId=tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi 2024-02-01T10:27:37.449 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:P5/HqCkPh2dsStKgCKTkTOtCb2U= 2024-02-01T10:27:37.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:P5/HqCkPh2dsStKgCKTkTOtCb2U=', 'Content-Length': '0'} 2024-02-01T10:27:37.452 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a7f3054767f5d6a1-0065bb7219-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:37.452 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:37.452 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:37.452 DEBUG:boto:path=/vclfcn-19/obj?versionId=tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi 2024-02-01T10:27:37.452 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi 2024-02-01T10:27:37.452 DEBUG:boto:Method: GET 2024-02-01T10:27:37.453 DEBUG:boto:Path: /vclfcn-19/obj?versionId=tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi 2024-02-01T10:27:37.453 DEBUG:boto:Data: 2024-02-01T10:27:37.453 DEBUG:boto:Headers: {} 2024-02-01T10:27:37.453 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:37.453 DEBUG:boto:Port: 8000 2024-02-01T10:27:37.453 DEBUG:boto:Params: {} 2024-02-01T10:27:37.453 DEBUG:boto:Token: None 2024-02-01T10:27:37.453 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:37 GMT /vclfcn-19/obj?versionId=tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi 2024-02-01T10:27:37.453 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:P5/HqCkPh2dsStKgCKTkTOtCb2U= 2024-02-01T10:27:37.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:P5/HqCkPh2dsStKgCKTkTOtCb2U=', 'Content-Length': '0'} 2024-02-01T10:27:37.456 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007c1900b5aa991e25-0065bb7219-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:37.457 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:37.457 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:37.457 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:37.457 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:37.457 DEBUG:boto:Method: GET 2024-02-01T10:27:37.457 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:37.457 DEBUG:boto:Data: 2024-02-01T10:27:37.457 DEBUG:boto:Headers: {} 2024-02-01T10:27:37.458 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:37.458 DEBUG:boto:Port: 8000 2024-02-01T10:27:37.458 DEBUG:boto:Params: {} 2024-02-01T10:27:37.458 DEBUG:boto:Token: None 2024-02-01T10:27:37.458 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:37 GMT /vclfcn-19/ 2024-02-01T10:27:37.458 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:86djPIsytSY6O2p5N7dgCxQYYmo= 2024-02-01T10:27:37.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:86djPIsytSY6O2p5N7dgCxQYYmo=', 'Content-Length': '0'} 2024-02-01T10:27:37.464 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080938d1fabe03908-0065bb7219-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:37.464 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:37.465 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:37.465 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:37.465 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:37.465 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:37.465 DEBUG:boto:Method: GET 2024-02-01T10:27:37.465 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:37.465 DEBUG:boto:Data: 2024-02-01T10:27:37.466 DEBUG:boto:Headers: {} 2024-02-01T10:27:37.466 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:37.466 DEBUG:boto:Port: 8000 2024-02-01T10:27:37.466 DEBUG:boto:Params: {} 2024-02-01T10:27:37.466 DEBUG:boto:Token: None 2024-02-01T10:27:37.466 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:37 GMT /vclfcn-19/ 2024-02-01T10:27:37.466 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:86djPIsytSY6O2p5N7dgCxQYYmo= 2024-02-01T10:27:37.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:86djPIsytSY6O2p5N7dgCxQYYmo=', 'Content-Length': '0'} 2024-02-01T10:27:37.472 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ba1f6c9de1d1f92-0065bb7219-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:37.516 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:37.517 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:37.517 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:37.517 DEBUG:boto:Method: HEAD 2024-02-01T10:27:37.517 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:37.517 DEBUG:boto:Data: 2024-02-01T10:27:37.517 DEBUG:boto:Headers: {} 2024-02-01T10:27:37.517 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:37.518 DEBUG:boto:Port: 8000 2024-02-01T10:27:37.518 DEBUG:boto:Params: {} 2024-02-01T10:27:37.518 DEBUG:boto:Token: None 2024-02-01T10:27:37.518 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:37 GMT /vclfcn-19/obj 2024-02-01T10:27:37.518 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wkLqM5BextiMhvsV2a/ruNi9Jqk= 2024-02-01T10:27:37.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wkLqM5BextiMhvsV2a/ruNi9Jqk=', 'Content-Length': '0'} 2024-02-01T10:27:37.524 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f11f1f2b1ff5cce4-0065bb7219-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:37.524 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:37.525 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:37.525 DEBUG:boto:Method: HEAD 2024-02-01T10:27:37.525 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:37.525 DEBUG:boto:Data: 2024-02-01T10:27:37.525 DEBUG:boto:Headers: {} 2024-02-01T10:27:37.525 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:37.525 DEBUG:boto:Port: 8000 2024-02-01T10:27:37.525 DEBUG:boto:Params: {} 2024-02-01T10:27:37.525 DEBUG:boto:Token: None 2024-02-01T10:27:37.526 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:37 GMT /vclfcn-19/obj 2024-02-01T10:27:37.526 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wkLqM5BextiMhvsV2a/ruNi9Jqk= 2024-02-01T10:27:37.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wkLqM5BextiMhvsV2a/ruNi9Jqk=', 'Content-Length': '0'} 2024-02-01T10:27:37.532 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f33b26f5940caf17-0065bb7219-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:37.533 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:37.533 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:37.533 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:37.533 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:37.533 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:37.533 DEBUG:boto:Method: GET 2024-02-01T10:27:37.534 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:37.534 DEBUG:boto:Data: 2024-02-01T10:27:37.534 DEBUG:boto:Headers: {} 2024-02-01T10:27:37.534 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:37.534 DEBUG:boto:Port: 8000 2024-02-01T10:27:37.535 DEBUG:boto:Params: {} 2024-02-01T10:27:37.535 DEBUG:boto:Token: None 2024-02-01T10:27:37.535 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:37 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:37.535 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nuuTVX3bkxnNPwltDIjKuORhjRQ= 2024-02-01T10:27:37.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nuuTVX3bkxnNPwltDIjKuORhjRQ=', 'Content-Length': '0'} 2024-02-01T10:27:37.538 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ba6d8179eceba7ae-0065bb7219-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:37.580 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:37.580 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:37.580 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:37.581 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:37.581 DEBUG:boto:Method: GET 2024-02-01T10:27:37.581 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:37.581 DEBUG:boto:Data: 2024-02-01T10:27:37.581 DEBUG:boto:Headers: {} 2024-02-01T10:27:37.581 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:37.581 DEBUG:boto:Port: 8000 2024-02-01T10:27:37.581 DEBUG:boto:Params: {} 2024-02-01T10:27:37.581 DEBUG:boto:Token: None 2024-02-01T10:27:37.582 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:37 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:37.582 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nuuTVX3bkxnNPwltDIjKuORhjRQ= 2024-02-01T10:27:37.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nuuTVX3bkxnNPwltDIjKuORhjRQ=', 'Content-Length': '0'} 2024-02-01T10:27:37.585 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008b7fe3ec449c00d9-0065bb7219-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:37.585 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:37.585 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:37.586 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:37.586 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:37.586 DEBUG:boto:Method: GET 2024-02-01T10:27:37.586 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:37.586 DEBUG:boto:Data: 2024-02-01T10:27:37.586 DEBUG:boto:Headers: {} 2024-02-01T10:27:37.587 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:37.587 DEBUG:boto:Port: 8000 2024-02-01T10:27:37.587 DEBUG:boto:Params: {} 2024-02-01T10:27:37.587 DEBUG:boto:Token: None 2024-02-01T10:27:37.587 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:37 GMT /vclfcn-19/ 2024-02-01T10:27:37.587 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:86djPIsytSY6O2p5N7dgCxQYYmo= 2024-02-01T10:27:37.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:86djPIsytSY6O2p5N7dgCxQYYmo=', 'Content-Length': '0'} 2024-02-01T10:27:37.592 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf724496d023ea36-0065bb7219-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:37.593 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:37.593 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:37.593 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:37.593 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:37.593 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:37.593 DEBUG:boto:Method: GET 2024-02-01T10:27:37.593 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:37.594 DEBUG:boto:Data: 2024-02-01T10:27:37.594 DEBUG:boto:Headers: {} 2024-02-01T10:27:37.594 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:37.594 DEBUG:boto:Port: 8000 2024-02-01T10:27:37.594 DEBUG:boto:Params: {} 2024-02-01T10:27:37.594 DEBUG:boto:Token: None 2024-02-01T10:27:37.594 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:37 GMT /vclfcn-19/ 2024-02-01T10:27:37.594 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:86djPIsytSY6O2p5N7dgCxQYYmo= 2024-02-01T10:27:37.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:86djPIsytSY6O2p5N7dgCxQYYmo=', 'Content-Length': '0'} 2024-02-01T10:27:37.600 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009837a1a077ce4ec-0065bb7219-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:37.600 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:37.600 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:37.600 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:37.600 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:37.600 DEBUG:boto:path=/vclfcn-19/obj?versionId=3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz 2024-02-01T10:27:37.601 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz 2024-02-01T10:27:37.601 DEBUG:boto:Method: GET 2024-02-01T10:27:37.601 DEBUG:boto:Path: /vclfcn-19/obj?versionId=3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz 2024-02-01T10:27:37.601 DEBUG:boto:Data: 2024-02-01T10:27:37.601 DEBUG:boto:Headers: {} 2024-02-01T10:27:37.601 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:37.601 DEBUG:boto:Port: 8000 2024-02-01T10:27:37.601 DEBUG:boto:Params: {} 2024-02-01T10:27:37.601 DEBUG:boto:Token: None 2024-02-01T10:27:37.601 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:37 GMT /vclfcn-19/obj?versionId=3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz 2024-02-01T10:27:37.602 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tpqpn9OApRF2GKFy9Zrf6Tl9vlY= 2024-02-01T10:27:37.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tpqpn9OApRF2GKFy9Zrf6Tl9vlY=', 'Content-Length': '0'} 2024-02-01T10:27:37.604 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a9f560f7318f27eb-0065bb7219-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:37.605 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:37.605 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:37.605 DEBUG:boto:path=/vclfcn-19/obj?versionId=3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz 2024-02-01T10:27:37.605 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz 2024-02-01T10:27:37.605 DEBUG:boto:Method: GET 2024-02-01T10:27:37.605 DEBUG:boto:Path: /vclfcn-19/obj?versionId=3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz 2024-02-01T10:27:37.605 DEBUG:boto:Data: 2024-02-01T10:27:37.605 DEBUG:boto:Headers: {} 2024-02-01T10:27:37.605 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:37.605 DEBUG:boto:Port: 8000 2024-02-01T10:27:37.605 DEBUG:boto:Params: {} 2024-02-01T10:27:37.606 DEBUG:boto:Token: None 2024-02-01T10:27:37.606 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:37 GMT /vclfcn-19/obj?versionId=3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz 2024-02-01T10:27:37.606 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tpqpn9OApRF2GKFy9Zrf6Tl9vlY= 2024-02-01T10:27:37.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tpqpn9OApRF2GKFy9Zrf6Tl9vlY=', 'Content-Length': '0'} 2024-02-01T10:27:37.609 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e886aacfa0594f20-0065bb7219-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:37.656 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:37.656 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:37.656 DEBUG:boto:path=/vclfcn-19/obj?versionId=3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz 2024-02-01T10:27:37.657 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz 2024-02-01T10:27:37.657 DEBUG:boto:Method: HEAD 2024-02-01T10:27:37.657 DEBUG:boto:Path: /vclfcn-19/obj?versionId=3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz 2024-02-01T10:27:37.657 DEBUG:boto:Data: 2024-02-01T10:27:37.657 DEBUG:boto:Headers: {} 2024-02-01T10:27:37.657 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:37.657 DEBUG:boto:Port: 8000 2024-02-01T10:27:37.657 DEBUG:boto:Params: {} 2024-02-01T10:27:37.657 DEBUG:boto:Token: None 2024-02-01T10:27:37.658 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:37 GMT /vclfcn-19/obj?versionId=3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz 2024-02-01T10:27:37.658 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QeEkEscZsgntMaN3uEuM+d9eyKU= 2024-02-01T10:27:37.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QeEkEscZsgntMaN3uEuM+d9eyKU=', 'Content-Length': '0'} 2024-02-01T10:27:37.661 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000edc57458c5a1b7b-0065bb7219-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:37.661 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:37.661 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:37.661 DEBUG:boto:path=/vclfcn-19/obj?versionId=3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz 2024-02-01T10:27:37.661 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz 2024-02-01T10:27:37.661 DEBUG:boto:Method: HEAD 2024-02-01T10:27:37.661 DEBUG:boto:Path: /vclfcn-19/obj?versionId=3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz 2024-02-01T10:27:37.661 DEBUG:boto:Data: 2024-02-01T10:27:37.662 DEBUG:boto:Headers: {} 2024-02-01T10:27:37.662 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:37.662 DEBUG:boto:Port: 8000 2024-02-01T10:27:37.662 DEBUG:boto:Params: {} 2024-02-01T10:27:37.662 DEBUG:boto:Token: None 2024-02-01T10:27:37.662 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:37 GMT /vclfcn-19/obj?versionId=3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz 2024-02-01T10:27:37.662 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QeEkEscZsgntMaN3uEuM+d9eyKU= 2024-02-01T10:27:37.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QeEkEscZsgntMaN3uEuM+d9eyKU=', 'Content-Length': '0'} 2024-02-01T10:27:37.665 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007d4db492fbdf3443-0065bb7219-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:37.665 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:37.666 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:37.666 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:37.666 DEBUG:boto:path=/vclfcn-19/obj?versionId=3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz 2024-02-01T10:27:37.666 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz 2024-02-01T10:27:37.666 DEBUG:boto:Method: GET 2024-02-01T10:27:37.666 DEBUG:boto:Path: /vclfcn-19/obj?versionId=3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz 2024-02-01T10:27:37.666 DEBUG:boto:Data: 2024-02-01T10:27:37.666 DEBUG:boto:Headers: {} 2024-02-01T10:27:37.666 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:37.666 DEBUG:boto:Port: 8000 2024-02-01T10:27:37.667 DEBUG:boto:Params: {} 2024-02-01T10:27:37.667 DEBUG:boto:Token: None 2024-02-01T10:27:37.667 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:37 GMT /vclfcn-19/obj?versionId=3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz 2024-02-01T10:27:37.667 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tpqpn9OApRF2GKFy9Zrf6Tl9vlY= 2024-02-01T10:27:37.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tpqpn9OApRF2GKFy9Zrf6Tl9vlY=', 'Content-Length': '0'} 2024-02-01T10:27:37.670 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006d9ff4118a233a3f-0065bb7219-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:37.716 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:37.717 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:37.717 DEBUG:boto:path=/vclfcn-19/obj?versionId=3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz 2024-02-01T10:27:37.717 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz 2024-02-01T10:27:37.717 DEBUG:boto:Method: GET 2024-02-01T10:27:37.717 DEBUG:boto:Path: /vclfcn-19/obj?versionId=3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz 2024-02-01T10:27:37.717 DEBUG:boto:Data: 2024-02-01T10:27:37.717 DEBUG:boto:Headers: {} 2024-02-01T10:27:37.717 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:37.717 DEBUG:boto:Port: 8000 2024-02-01T10:27:37.717 DEBUG:boto:Params: {} 2024-02-01T10:27:37.718 DEBUG:boto:Token: None 2024-02-01T10:27:37.718 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:37 GMT /vclfcn-19/obj?versionId=3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz 2024-02-01T10:27:37.718 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tpqpn9OApRF2GKFy9Zrf6Tl9vlY= 2024-02-01T10:27:37.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tpqpn9OApRF2GKFy9Zrf6Tl9vlY=', 'Content-Length': '0'} 2024-02-01T10:27:37.721 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bb1afc6de4fc0ee8-0065bb7219-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:37.721 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:37.722 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:37.722 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:37.722 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:37.722 DEBUG:boto:Method: GET 2024-02-01T10:27:37.722 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:37.722 DEBUG:boto:Data: 2024-02-01T10:27:37.722 DEBUG:boto:Headers: {} 2024-02-01T10:27:37.722 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:37.723 DEBUG:boto:Port: 8000 2024-02-01T10:27:37.723 DEBUG:boto:Params: {} 2024-02-01T10:27:37.723 DEBUG:boto:Token: None 2024-02-01T10:27:37.723 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:37 GMT /vclfcn-19/ 2024-02-01T10:27:37.724 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:86djPIsytSY6O2p5N7dgCxQYYmo= 2024-02-01T10:27:37.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:86djPIsytSY6O2p5N7dgCxQYYmo=', 'Content-Length': '0'} 2024-02-01T10:27:37.729 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e80586d7659d00ec-0065bb7219-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:37.729 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:37.730 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:37.730 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:37.730 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:37.730 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:37.730 DEBUG:boto:Method: GET 2024-02-01T10:27:37.730 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:37.730 DEBUG:boto:Data: 2024-02-01T10:27:37.730 DEBUG:boto:Headers: {} 2024-02-01T10:27:37.730 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:37.730 DEBUG:boto:Port: 8000 2024-02-01T10:27:37.731 DEBUG:boto:Params: {} 2024-02-01T10:27:37.731 DEBUG:boto:Token: None 2024-02-01T10:27:37.731 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:37 GMT /vclfcn-19/ 2024-02-01T10:27:37.731 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:86djPIsytSY6O2p5N7dgCxQYYmo= 2024-02-01T10:27:37.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:86djPIsytSY6O2p5N7dgCxQYYmo=', 'Content-Length': '0'} 2024-02-01T10:27:37.738 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000137074b22a39270a-0065bb7219-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:37.738 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:37.739 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:37.739 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:37.739 DEBUG:boto:Method: HEAD 2024-02-01T10:27:37.739 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:37.739 DEBUG:boto:Data: 2024-02-01T10:27:37.739 DEBUG:boto:Headers: {} 2024-02-01T10:27:37.739 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:37.739 DEBUG:boto:Port: 8000 2024-02-01T10:27:37.740 DEBUG:boto:Params: {} 2024-02-01T10:27:37.740 DEBUG:boto:Token: None 2024-02-01T10:27:37.740 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:37 GMT /vclfcn-19/obj 2024-02-01T10:27:37.740 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wkLqM5BextiMhvsV2a/ruNi9Jqk= 2024-02-01T10:27:37.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wkLqM5BextiMhvsV2a/ruNi9Jqk=', 'Content-Length': '0'} 2024-02-01T10:27:37.746 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007acc612a94cedda2-0065bb7219-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:37.746 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:37.747 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:37.747 DEBUG:boto:Method: HEAD 2024-02-01T10:27:37.747 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:37.747 DEBUG:boto:Data: 2024-02-01T10:27:37.747 DEBUG:boto:Headers: {} 2024-02-01T10:27:37.747 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:37.747 DEBUG:boto:Port: 8000 2024-02-01T10:27:37.747 DEBUG:boto:Params: {} 2024-02-01T10:27:37.747 DEBUG:boto:Token: None 2024-02-01T10:27:37.748 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:37 GMT /vclfcn-19/obj 2024-02-01T10:27:37.748 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wkLqM5BextiMhvsV2a/ruNi9Jqk= 2024-02-01T10:27:37.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wkLqM5BextiMhvsV2a/ruNi9Jqk=', 'Content-Length': '0'} 2024-02-01T10:27:37.754 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ca8fa2b78dcc4fce-0065bb7219-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:37.754 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:37.755 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:37.755 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:37.755 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:37.755 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:37.755 DEBUG:boto:Method: GET 2024-02-01T10:27:37.755 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:37.755 DEBUG:boto:Data: 2024-02-01T10:27:37.755 DEBUG:boto:Headers: {} 2024-02-01T10:27:37.755 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:37.755 DEBUG:boto:Port: 8000 2024-02-01T10:27:37.756 DEBUG:boto:Params: {} 2024-02-01T10:27:37.756 DEBUG:boto:Token: None 2024-02-01T10:27:37.756 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:37 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:37.756 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nuuTVX3bkxnNPwltDIjKuORhjRQ= 2024-02-01T10:27:37.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nuuTVX3bkxnNPwltDIjKuORhjRQ=', 'Content-Length': '0'} 2024-02-01T10:27:37.762 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000532b103493bee894-0065bb7219-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:37.804 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:37.804 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:37.804 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:37.804 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:37.805 DEBUG:boto:Method: GET 2024-02-01T10:27:37.805 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:37.805 DEBUG:boto:Data: 2024-02-01T10:27:37.805 DEBUG:boto:Headers: {} 2024-02-01T10:27:37.805 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:37.805 DEBUG:boto:Port: 8000 2024-02-01T10:27:37.805 DEBUG:boto:Params: {} 2024-02-01T10:27:37.805 DEBUG:boto:Token: None 2024-02-01T10:27:37.805 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:37 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:37.806 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nuuTVX3bkxnNPwltDIjKuORhjRQ= 2024-02-01T10:27:37.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nuuTVX3bkxnNPwltDIjKuORhjRQ=', 'Content-Length': '0'} 2024-02-01T10:27:37.809 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000847d351a5e5ffaf4-0065bb7219-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:37.809 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:37.809 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:37.809 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:37.809 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:37.810 DEBUG:boto:Method: GET 2024-02-01T10:27:37.810 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:37.810 DEBUG:boto:Data: 2024-02-01T10:27:37.810 DEBUG:boto:Headers: {} 2024-02-01T10:27:37.810 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:37.810 DEBUG:boto:Port: 8000 2024-02-01T10:27:37.810 DEBUG:boto:Params: {} 2024-02-01T10:27:37.810 DEBUG:boto:Token: None 2024-02-01T10:27:37.810 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:37 GMT /vclfcn-19/ 2024-02-01T10:27:37.810 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:86djPIsytSY6O2p5N7dgCxQYYmo= 2024-02-01T10:27:37.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:86djPIsytSY6O2p5N7dgCxQYYmo=', 'Content-Length': '0'} 2024-02-01T10:27:37.816 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a68bf422c264f17e-0065bb7219-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:37.816 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:37.816 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:37.817 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:37.817 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:37.817 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:37.817 DEBUG:boto:Method: GET 2024-02-01T10:27:37.817 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:37.817 DEBUG:boto:Data: 2024-02-01T10:27:37.817 DEBUG:boto:Headers: {} 2024-02-01T10:27:37.817 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:37.817 DEBUG:boto:Port: 8000 2024-02-01T10:27:37.818 DEBUG:boto:Params: {} 2024-02-01T10:27:37.818 DEBUG:boto:Token: None 2024-02-01T10:27:37.818 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:37 GMT /vclfcn-19/ 2024-02-01T10:27:37.818 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:86djPIsytSY6O2p5N7dgCxQYYmo= 2024-02-01T10:27:37.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:86djPIsytSY6O2p5N7dgCxQYYmo=', 'Content-Length': '0'} 2024-02-01T10:27:37.824 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2bb5cf3eabd3396-0065bb7219-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:37.824 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:37.824 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:37.825 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:37.825 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:37.825 DEBUG:boto:path=/vclfcn-19/obj?versionId=K23uwsLWGmQDGYGsdu-h04Fk2wApB0O 2024-02-01T10:27:37.825 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=K23uwsLWGmQDGYGsdu-h04Fk2wApB0O 2024-02-01T10:27:37.825 DEBUG:boto:Method: GET 2024-02-01T10:27:37.825 DEBUG:boto:Path: /vclfcn-19/obj?versionId=K23uwsLWGmQDGYGsdu-h04Fk2wApB0O 2024-02-01T10:27:37.825 DEBUG:boto:Data: 2024-02-01T10:27:37.825 DEBUG:boto:Headers: {} 2024-02-01T10:27:37.825 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:37.825 DEBUG:boto:Port: 8000 2024-02-01T10:27:37.825 DEBUG:boto:Params: {} 2024-02-01T10:27:37.826 DEBUG:boto:Token: None 2024-02-01T10:27:37.826 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:37 GMT /vclfcn-19/obj?versionId=K23uwsLWGmQDGYGsdu-h04Fk2wApB0O 2024-02-01T10:27:37.826 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fyhLFSkOu4y3Hu1yg3jBPMdZ4N8= 2024-02-01T10:27:37.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fyhLFSkOu4y3Hu1yg3jBPMdZ4N8=', 'Content-Length': '0'} 2024-02-01T10:27:37.829 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'K23uwsLWGmQDGYGsdu-h04Fk2wApB0O'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000073d699a04ca9d97-0065bb7219-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:37.872 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:37.872 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:37.872 DEBUG:boto:path=/vclfcn-19/obj?versionId=K23uwsLWGmQDGYGsdu-h04Fk2wApB0O 2024-02-01T10:27:37.873 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=K23uwsLWGmQDGYGsdu-h04Fk2wApB0O 2024-02-01T10:27:37.873 DEBUG:boto:Method: GET 2024-02-01T10:27:37.873 DEBUG:boto:Path: /vclfcn-19/obj?versionId=K23uwsLWGmQDGYGsdu-h04Fk2wApB0O 2024-02-01T10:27:37.873 DEBUG:boto:Data: 2024-02-01T10:27:37.873 DEBUG:boto:Headers: {} 2024-02-01T10:27:37.873 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:37.873 DEBUG:boto:Port: 8000 2024-02-01T10:27:37.873 DEBUG:boto:Params: {} 2024-02-01T10:27:37.874 DEBUG:boto:Token: None 2024-02-01T10:27:37.874 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:37 GMT /vclfcn-19/obj?versionId=K23uwsLWGmQDGYGsdu-h04Fk2wApB0O 2024-02-01T10:27:37.874 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fyhLFSkOu4y3Hu1yg3jBPMdZ4N8= 2024-02-01T10:27:37.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fyhLFSkOu4y3Hu1yg3jBPMdZ4N8=', 'Content-Length': '0'} 2024-02-01T10:27:37.877 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'K23uwsLWGmQDGYGsdu-h04Fk2wApB0O'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005527e55807e5b161-0065bb7219-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:37.877 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:37.878 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:37.878 DEBUG:boto:path=/vclfcn-19/obj?versionId=K23uwsLWGmQDGYGsdu-h04Fk2wApB0O 2024-02-01T10:27:37.878 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=K23uwsLWGmQDGYGsdu-h04Fk2wApB0O 2024-02-01T10:27:37.878 DEBUG:boto:Method: HEAD 2024-02-01T10:27:37.878 DEBUG:boto:Path: /vclfcn-19/obj?versionId=K23uwsLWGmQDGYGsdu-h04Fk2wApB0O 2024-02-01T10:27:37.878 DEBUG:boto:Data: 2024-02-01T10:27:37.878 DEBUG:boto:Headers: {} 2024-02-01T10:27:37.878 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:37.878 DEBUG:boto:Port: 8000 2024-02-01T10:27:37.878 DEBUG:boto:Params: {} 2024-02-01T10:27:37.879 DEBUG:boto:Token: None 2024-02-01T10:27:37.879 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:37 GMT /vclfcn-19/obj?versionId=K23uwsLWGmQDGYGsdu-h04Fk2wApB0O 2024-02-01T10:27:37.879 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:rKtE6O/CQ7eLMivSsQC2nHY6/zM= 2024-02-01T10:27:37.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:rKtE6O/CQ7eLMivSsQC2nHY6/zM=', 'Content-Length': '0'} 2024-02-01T10:27:37.882 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'K23uwsLWGmQDGYGsdu-h04Fk2wApB0O'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000030df1060c5dd1477-0065bb7219-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:37.882 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:37.882 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:37.882 DEBUG:boto:path=/vclfcn-19/obj?versionId=K23uwsLWGmQDGYGsdu-h04Fk2wApB0O 2024-02-01T10:27:37.883 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=K23uwsLWGmQDGYGsdu-h04Fk2wApB0O 2024-02-01T10:27:37.883 DEBUG:boto:Method: HEAD 2024-02-01T10:27:37.883 DEBUG:boto:Path: /vclfcn-19/obj?versionId=K23uwsLWGmQDGYGsdu-h04Fk2wApB0O 2024-02-01T10:27:37.883 DEBUG:boto:Data: 2024-02-01T10:27:37.883 DEBUG:boto:Headers: {} 2024-02-01T10:27:37.883 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:37.883 DEBUG:boto:Port: 8000 2024-02-01T10:27:37.883 DEBUG:boto:Params: {} 2024-02-01T10:27:37.883 DEBUG:boto:Token: None 2024-02-01T10:27:37.883 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:37 GMT /vclfcn-19/obj?versionId=K23uwsLWGmQDGYGsdu-h04Fk2wApB0O 2024-02-01T10:27:37.884 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:rKtE6O/CQ7eLMivSsQC2nHY6/zM= 2024-02-01T10:27:37.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:rKtE6O/CQ7eLMivSsQC2nHY6/zM=', 'Content-Length': '0'} 2024-02-01T10:27:37.886 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'K23uwsLWGmQDGYGsdu-h04Fk2wApB0O'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002e4e054f4f45ea46-0065bb7219-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:37.887 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:37.887 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:37.887 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:37.887 DEBUG:boto:path=/vclfcn-19/obj?versionId=K23uwsLWGmQDGYGsdu-h04Fk2wApB0O 2024-02-01T10:27:37.887 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=K23uwsLWGmQDGYGsdu-h04Fk2wApB0O 2024-02-01T10:27:37.887 DEBUG:boto:Method: GET 2024-02-01T10:27:37.887 DEBUG:boto:Path: /vclfcn-19/obj?versionId=K23uwsLWGmQDGYGsdu-h04Fk2wApB0O 2024-02-01T10:27:37.888 DEBUG:boto:Data: 2024-02-01T10:27:37.888 DEBUG:boto:Headers: {} 2024-02-01T10:27:37.888 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:37.888 DEBUG:boto:Port: 8000 2024-02-01T10:27:37.888 DEBUG:boto:Params: {} 2024-02-01T10:27:37.888 DEBUG:boto:Token: None 2024-02-01T10:27:37.888 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:37 GMT /vclfcn-19/obj?versionId=K23uwsLWGmQDGYGsdu-h04Fk2wApB0O 2024-02-01T10:27:37.888 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fyhLFSkOu4y3Hu1yg3jBPMdZ4N8= 2024-02-01T10:27:37.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fyhLFSkOu4y3Hu1yg3jBPMdZ4N8=', 'Content-Length': '0'} 2024-02-01T10:27:37.891 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'K23uwsLWGmQDGYGsdu-h04Fk2wApB0O'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000057c6c9dc18c34e39-0065bb7219-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:37.932 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:37.932 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:37.932 DEBUG:boto:path=/vclfcn-19/obj?versionId=K23uwsLWGmQDGYGsdu-h04Fk2wApB0O 2024-02-01T10:27:37.933 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=K23uwsLWGmQDGYGsdu-h04Fk2wApB0O 2024-02-01T10:27:37.933 DEBUG:boto:Method: GET 2024-02-01T10:27:37.933 DEBUG:boto:Path: /vclfcn-19/obj?versionId=K23uwsLWGmQDGYGsdu-h04Fk2wApB0O 2024-02-01T10:27:37.933 DEBUG:boto:Data: 2024-02-01T10:27:37.933 DEBUG:boto:Headers: {} 2024-02-01T10:27:37.933 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:37.933 DEBUG:boto:Port: 8000 2024-02-01T10:27:37.933 DEBUG:boto:Params: {} 2024-02-01T10:27:37.933 DEBUG:boto:Token: None 2024-02-01T10:27:37.934 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:37 GMT /vclfcn-19/obj?versionId=K23uwsLWGmQDGYGsdu-h04Fk2wApB0O 2024-02-01T10:27:37.934 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fyhLFSkOu4y3Hu1yg3jBPMdZ4N8= 2024-02-01T10:27:37.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fyhLFSkOu4y3Hu1yg3jBPMdZ4N8=', 'Content-Length': '0'} 2024-02-01T10:27:37.937 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'K23uwsLWGmQDGYGsdu-h04Fk2wApB0O'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006fbee6a9fd396601-0065bb7219-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:37.937 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:37.938 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:37.938 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:37.938 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:37.938 DEBUG:boto:Method: GET 2024-02-01T10:27:37.938 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:37.938 DEBUG:boto:Data: 2024-02-01T10:27:37.938 DEBUG:boto:Headers: {} 2024-02-01T10:27:37.938 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:37.938 DEBUG:boto:Port: 8000 2024-02-01T10:27:37.938 DEBUG:boto:Params: {} 2024-02-01T10:27:37.938 DEBUG:boto:Token: None 2024-02-01T10:27:37.939 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:37 GMT /vclfcn-19/ 2024-02-01T10:27:37.939 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:86djPIsytSY6O2p5N7dgCxQYYmo= 2024-02-01T10:27:37.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:86djPIsytSY6O2p5N7dgCxQYYmo=', 'Content-Length': '0'} 2024-02-01T10:27:37.944 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e876b521710440c4-0065bb7219-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:37.944 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:37.945 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:37.945 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:37.945 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:37.945 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:37.945 DEBUG:boto:Method: GET 2024-02-01T10:27:37.945 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:37.945 DEBUG:boto:Data: 2024-02-01T10:27:37.945 DEBUG:boto:Headers: {} 2024-02-01T10:27:37.946 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:37.946 DEBUG:boto:Port: 8000 2024-02-01T10:27:37.946 DEBUG:boto:Params: {} 2024-02-01T10:27:37.946 DEBUG:boto:Token: None 2024-02-01T10:27:37.946 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:37 GMT /vclfcn-19/ 2024-02-01T10:27:37.947 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:86djPIsytSY6O2p5N7dgCxQYYmo= 2024-02-01T10:27:37.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:86djPIsytSY6O2p5N7dgCxQYYmo=', 'Content-Length': '0'} 2024-02-01T10:27:37.952 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000925fa48340d73a32-0065bb7219-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:37.992 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:37.993 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:37.993 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:37.993 DEBUG:boto:Method: HEAD 2024-02-01T10:27:37.993 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:37.993 DEBUG:boto:Data: 2024-02-01T10:27:37.993 DEBUG:boto:Headers: {} 2024-02-01T10:27:37.993 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:37.993 DEBUG:boto:Port: 8000 2024-02-01T10:27:37.993 DEBUG:boto:Params: {} 2024-02-01T10:27:37.994 DEBUG:boto:Token: None 2024-02-01T10:27:37.994 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:37 GMT /vclfcn-19/obj 2024-02-01T10:27:37.994 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wkLqM5BextiMhvsV2a/ruNi9Jqk= 2024-02-01T10:27:37.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wkLqM5BextiMhvsV2a/ruNi9Jqk=', 'Content-Length': '0'} 2024-02-01T10:27:38.000 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f108e830bb67b8e4-0065bb7219-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:38.000 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:38.000 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:38.001 DEBUG:boto:Method: HEAD 2024-02-01T10:27:38.001 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:38.001 DEBUG:boto:Data: 2024-02-01T10:27:38.001 DEBUG:boto:Headers: {} 2024-02-01T10:27:38.001 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:38.001 DEBUG:boto:Port: 8000 2024-02-01T10:27:38.001 DEBUG:boto:Params: {} 2024-02-01T10:27:38.001 DEBUG:boto:Token: None 2024-02-01T10:27:38.001 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:38 GMT /vclfcn-19/obj 2024-02-01T10:27:38.001 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:++st0oeXd3mm/tVRMqqNI504Td0= 2024-02-01T10:27:38.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:++st0oeXd3mm/tVRMqqNI504Td0=', 'Content-Length': '0'} 2024-02-01T10:27:38.007 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e1687af99f24f1f7-0065bb721a-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:38.007 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:38.007 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:38.008 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:38.008 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:38.008 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:38.008 DEBUG:boto:Method: GET 2024-02-01T10:27:38.008 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:38.008 DEBUG:boto:Data: 2024-02-01T10:27:38.008 DEBUG:boto:Headers: {} 2024-02-01T10:27:38.008 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:38.008 DEBUG:boto:Port: 8000 2024-02-01T10:27:38.008 DEBUG:boto:Params: {} 2024-02-01T10:27:38.009 DEBUG:boto:Token: None 2024-02-01T10:27:38.009 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:38 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:38.009 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fORJMwA/ifU8HAv/Tts9RTK4J7w= 2024-02-01T10:27:38.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fORJMwA/ifU8HAv/Tts9RTK4J7w=', 'Content-Length': '0'} 2024-02-01T10:27:38.012 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aab4fe2df665bf96-0065bb721a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:38.012 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:38.012 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:38.012 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:38.012 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:38.012 DEBUG:boto:Method: GET 2024-02-01T10:27:38.012 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:38.013 DEBUG:boto:Data: 2024-02-01T10:27:38.013 DEBUG:boto:Headers: {} 2024-02-01T10:27:38.013 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:38.013 DEBUG:boto:Port: 8000 2024-02-01T10:27:38.013 DEBUG:boto:Params: {} 2024-02-01T10:27:38.013 DEBUG:boto:Token: None 2024-02-01T10:27:38.013 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:38 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:38.013 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fORJMwA/ifU8HAv/Tts9RTK4J7w= 2024-02-01T10:27:38.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fORJMwA/ifU8HAv/Tts9RTK4J7w=', 'Content-Length': '0'} 2024-02-01T10:27:38.016 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000372acdaf59a0ff1d-0065bb721a-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:38.016 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:38.016 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:38.016 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:38.016 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:38.017 DEBUG:boto:Method: GET 2024-02-01T10:27:38.017 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:38.017 DEBUG:boto:Data: 2024-02-01T10:27:38.017 DEBUG:boto:Headers: {} 2024-02-01T10:27:38.017 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:38.017 DEBUG:boto:Port: 8000 2024-02-01T10:27:38.017 DEBUG:boto:Params: {} 2024-02-01T10:27:38.017 DEBUG:boto:Token: None 2024-02-01T10:27:38.017 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:38 GMT /vclfcn-19/ 2024-02-01T10:27:38.017 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5ixPGml/RgIDcptGflveRBKaBzE= 2024-02-01T10:27:38.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5ixPGml/RgIDcptGflveRBKaBzE=', 'Content-Length': '0'} 2024-02-01T10:27:38.023 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b52c863c2797695-0065bb721a-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:38.023 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:38.023 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:38.023 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:38.024 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:38.024 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:38.024 DEBUG:boto:Method: GET 2024-02-01T10:27:38.024 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:38.024 DEBUG:boto:Data: 2024-02-01T10:27:38.024 DEBUG:boto:Headers: {} 2024-02-01T10:27:38.024 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:38.024 DEBUG:boto:Port: 8000 2024-02-01T10:27:38.024 DEBUG:boto:Params: {} 2024-02-01T10:27:38.024 DEBUG:boto:Token: None 2024-02-01T10:27:38.024 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:38 GMT /vclfcn-19/ 2024-02-01T10:27:38.025 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5ixPGml/RgIDcptGflveRBKaBzE= 2024-02-01T10:27:38.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5ixPGml/RgIDcptGflveRBKaBzE=', 'Content-Length': '0'} 2024-02-01T10:27:38.030 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f2dcf69b462fd56-0065bb721a-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:38.076 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:38.076 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:38.077 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:38.077 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:38.077 DEBUG:boto:path=/vclfcn-19/obj?versionId=st2TOT.K7IOe-wFfL.hOrRVfLNWblRd 2024-02-01T10:27:38.077 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=st2TOT.K7IOe-wFfL.hOrRVfLNWblRd 2024-02-01T10:27:38.077 DEBUG:boto:Method: GET 2024-02-01T10:27:38.077 DEBUG:boto:Path: /vclfcn-19/obj?versionId=st2TOT.K7IOe-wFfL.hOrRVfLNWblRd 2024-02-01T10:27:38.077 DEBUG:boto:Data: 2024-02-01T10:27:38.077 DEBUG:boto:Headers: {} 2024-02-01T10:27:38.077 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:38.077 DEBUG:boto:Port: 8000 2024-02-01T10:27:38.078 DEBUG:boto:Params: {} 2024-02-01T10:27:38.078 DEBUG:boto:Token: None 2024-02-01T10:27:38.078 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:38 GMT /vclfcn-19/obj?versionId=st2TOT.K7IOe-wFfL.hOrRVfLNWblRd 2024-02-01T10:27:38.078 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:h4hlEp8swupeufTXEFuPs7EW1aA= 2024-02-01T10:27:38.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:h4hlEp8swupeufTXEFuPs7EW1aA=', 'Content-Length': '0'} 2024-02-01T10:27:38.081 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'st2TOT.K7IOe-wFfL.hOrRVfLNWblRd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b8a75807826e1343-0065bb721a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:38.082 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:38.082 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:38.082 DEBUG:boto:path=/vclfcn-19/obj?versionId=st2TOT.K7IOe-wFfL.hOrRVfLNWblRd 2024-02-01T10:27:38.082 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=st2TOT.K7IOe-wFfL.hOrRVfLNWblRd 2024-02-01T10:27:38.082 DEBUG:boto:Method: GET 2024-02-01T10:27:38.082 DEBUG:boto:Path: /vclfcn-19/obj?versionId=st2TOT.K7IOe-wFfL.hOrRVfLNWblRd 2024-02-01T10:27:38.082 DEBUG:boto:Data: 2024-02-01T10:27:38.082 DEBUG:boto:Headers: {} 2024-02-01T10:27:38.083 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:38.083 DEBUG:boto:Port: 8000 2024-02-01T10:27:38.083 DEBUG:boto:Params: {} 2024-02-01T10:27:38.083 DEBUG:boto:Token: None 2024-02-01T10:27:38.083 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:38 GMT /vclfcn-19/obj?versionId=st2TOT.K7IOe-wFfL.hOrRVfLNWblRd 2024-02-01T10:27:38.083 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:h4hlEp8swupeufTXEFuPs7EW1aA= 2024-02-01T10:27:38.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:h4hlEp8swupeufTXEFuPs7EW1aA=', 'Content-Length': '0'} 2024-02-01T10:27:38.086 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'st2TOT.K7IOe-wFfL.hOrRVfLNWblRd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006d5bc755938a17a1-0065bb721a-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:38.087 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:38.087 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:38.087 DEBUG:boto:path=/vclfcn-19/obj?versionId=st2TOT.K7IOe-wFfL.hOrRVfLNWblRd 2024-02-01T10:27:38.087 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=st2TOT.K7IOe-wFfL.hOrRVfLNWblRd 2024-02-01T10:27:38.087 DEBUG:boto:Method: HEAD 2024-02-01T10:27:38.087 DEBUG:boto:Path: /vclfcn-19/obj?versionId=st2TOT.K7IOe-wFfL.hOrRVfLNWblRd 2024-02-01T10:27:38.087 DEBUG:boto:Data: 2024-02-01T10:27:38.087 DEBUG:boto:Headers: {} 2024-02-01T10:27:38.087 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:38.087 DEBUG:boto:Port: 8000 2024-02-01T10:27:38.088 DEBUG:boto:Params: {} 2024-02-01T10:27:38.088 DEBUG:boto:Token: None 2024-02-01T10:27:38.088 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:38 GMT /vclfcn-19/obj?versionId=st2TOT.K7IOe-wFfL.hOrRVfLNWblRd 2024-02-01T10:27:38.088 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:CBGA0S0hj1uyXRunZSvDZ7vwm7E= 2024-02-01T10:27:38.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:CBGA0S0hj1uyXRunZSvDZ7vwm7E=', 'Content-Length': '0'} 2024-02-01T10:27:38.091 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'st2TOT.K7IOe-wFfL.hOrRVfLNWblRd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001964d8fa722f439b-0065bb721a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:38.091 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:38.091 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:38.091 DEBUG:boto:path=/vclfcn-19/obj?versionId=st2TOT.K7IOe-wFfL.hOrRVfLNWblRd 2024-02-01T10:27:38.091 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=st2TOT.K7IOe-wFfL.hOrRVfLNWblRd 2024-02-01T10:27:38.091 DEBUG:boto:Method: HEAD 2024-02-01T10:27:38.091 DEBUG:boto:Path: /vclfcn-19/obj?versionId=st2TOT.K7IOe-wFfL.hOrRVfLNWblRd 2024-02-01T10:27:38.091 DEBUG:boto:Data: 2024-02-01T10:27:38.091 DEBUG:boto:Headers: {} 2024-02-01T10:27:38.092 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:38.092 DEBUG:boto:Port: 8000 2024-02-01T10:27:38.092 DEBUG:boto:Params: {} 2024-02-01T10:27:38.092 DEBUG:boto:Token: None 2024-02-01T10:27:38.092 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:38 GMT /vclfcn-19/obj?versionId=st2TOT.K7IOe-wFfL.hOrRVfLNWblRd 2024-02-01T10:27:38.092 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:CBGA0S0hj1uyXRunZSvDZ7vwm7E= 2024-02-01T10:27:38.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:CBGA0S0hj1uyXRunZSvDZ7vwm7E=', 'Content-Length': '0'} 2024-02-01T10:27:38.095 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'st2TOT.K7IOe-wFfL.hOrRVfLNWblRd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006d71dcf0e6fae92b-0065bb721a-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:38.095 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:38.095 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:38.095 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:38.095 DEBUG:boto:path=/vclfcn-19/obj?versionId=st2TOT.K7IOe-wFfL.hOrRVfLNWblRd 2024-02-01T10:27:38.096 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=st2TOT.K7IOe-wFfL.hOrRVfLNWblRd 2024-02-01T10:27:38.096 DEBUG:boto:Method: GET 2024-02-01T10:27:38.096 DEBUG:boto:Path: /vclfcn-19/obj?versionId=st2TOT.K7IOe-wFfL.hOrRVfLNWblRd 2024-02-01T10:27:38.096 DEBUG:boto:Data: 2024-02-01T10:27:38.096 DEBUG:boto:Headers: {} 2024-02-01T10:27:38.096 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:38.096 DEBUG:boto:Port: 8000 2024-02-01T10:27:38.096 DEBUG:boto:Params: {} 2024-02-01T10:27:38.096 DEBUG:boto:Token: None 2024-02-01T10:27:38.097 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:38 GMT /vclfcn-19/obj?versionId=st2TOT.K7IOe-wFfL.hOrRVfLNWblRd 2024-02-01T10:27:38.097 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:h4hlEp8swupeufTXEFuPs7EW1aA= 2024-02-01T10:27:38.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:h4hlEp8swupeufTXEFuPs7EW1aA=', 'Content-Length': '0'} 2024-02-01T10:27:38.099 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'st2TOT.K7IOe-wFfL.hOrRVfLNWblRd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000071601f20f5b89f13-0065bb721a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:38.144 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:38.144 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:38.144 DEBUG:boto:path=/vclfcn-19/obj?versionId=st2TOT.K7IOe-wFfL.hOrRVfLNWblRd 2024-02-01T10:27:38.144 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=st2TOT.K7IOe-wFfL.hOrRVfLNWblRd 2024-02-01T10:27:38.145 DEBUG:boto:Method: GET 2024-02-01T10:27:38.145 DEBUG:boto:Path: /vclfcn-19/obj?versionId=st2TOT.K7IOe-wFfL.hOrRVfLNWblRd 2024-02-01T10:27:38.145 DEBUG:boto:Data: 2024-02-01T10:27:38.145 DEBUG:boto:Headers: {} 2024-02-01T10:27:38.145 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:38.145 DEBUG:boto:Port: 8000 2024-02-01T10:27:38.145 DEBUG:boto:Params: {} 2024-02-01T10:27:38.145 DEBUG:boto:Token: None 2024-02-01T10:27:38.145 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:38 GMT /vclfcn-19/obj?versionId=st2TOT.K7IOe-wFfL.hOrRVfLNWblRd 2024-02-01T10:27:38.145 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:h4hlEp8swupeufTXEFuPs7EW1aA= 2024-02-01T10:27:38.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:h4hlEp8swupeufTXEFuPs7EW1aA=', 'Content-Length': '0'} 2024-02-01T10:27:38.149 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'st2TOT.K7IOe-wFfL.hOrRVfLNWblRd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009940fa918d034b4c-0065bb721a-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:38.149 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:38.149 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:38.149 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:38.149 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:38.149 DEBUG:boto:Method: GET 2024-02-01T10:27:38.149 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:38.149 DEBUG:boto:Data: 2024-02-01T10:27:38.150 DEBUG:boto:Headers: {} 2024-02-01T10:27:38.150 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:38.150 DEBUG:boto:Port: 8000 2024-02-01T10:27:38.150 DEBUG:boto:Params: {} 2024-02-01T10:27:38.150 DEBUG:boto:Token: None 2024-02-01T10:27:38.150 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:38 GMT /vclfcn-19/ 2024-02-01T10:27:38.150 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5ixPGml/RgIDcptGflveRBKaBzE= 2024-02-01T10:27:38.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5ixPGml/RgIDcptGflveRBKaBzE=', 'Content-Length': '0'} 2024-02-01T10:27:38.155 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a28dca688b92afa2-0065bb721a-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:38.155 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:38.156 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:38.156 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:38.156 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:38.156 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:38.156 DEBUG:boto:Method: GET 2024-02-01T10:27:38.156 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:38.156 DEBUG:boto:Data: 2024-02-01T10:27:38.156 DEBUG:boto:Headers: {} 2024-02-01T10:27:38.157 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:38.157 DEBUG:boto:Port: 8000 2024-02-01T10:27:38.157 DEBUG:boto:Params: {} 2024-02-01T10:27:38.157 DEBUG:boto:Token: None 2024-02-01T10:27:38.157 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:38 GMT /vclfcn-19/ 2024-02-01T10:27:38.157 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5ixPGml/RgIDcptGflveRBKaBzE= 2024-02-01T10:27:38.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5ixPGml/RgIDcptGflveRBKaBzE=', 'Content-Length': '0'} 2024-02-01T10:27:38.162 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001da8abd662b0cf93-0065bb721a-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:38.204 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:38.205 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:38.205 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:38.205 DEBUG:boto:Method: HEAD 2024-02-01T10:27:38.205 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:38.205 DEBUG:boto:Data: 2024-02-01T10:27:38.205 DEBUG:boto:Headers: {} 2024-02-01T10:27:38.205 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:38.205 DEBUG:boto:Port: 8000 2024-02-01T10:27:38.205 DEBUG:boto:Params: {} 2024-02-01T10:27:38.206 DEBUG:boto:Token: None 2024-02-01T10:27:38.206 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:38 GMT /vclfcn-19/obj 2024-02-01T10:27:38.206 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:++st0oeXd3mm/tVRMqqNI504Td0= 2024-02-01T10:27:38.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:++st0oeXd3mm/tVRMqqNI504Td0=', 'Content-Length': '0'} 2024-02-01T10:27:38.212 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000048724baf6dc60023-0065bb721a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:38.212 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:38.212 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:38.212 DEBUG:boto:Method: HEAD 2024-02-01T10:27:38.212 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:38.212 DEBUG:boto:Data: 2024-02-01T10:27:38.212 DEBUG:boto:Headers: {} 2024-02-01T10:27:38.212 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:38.213 DEBUG:boto:Port: 8000 2024-02-01T10:27:38.213 DEBUG:boto:Params: {} 2024-02-01T10:27:38.213 DEBUG:boto:Token: None 2024-02-01T10:27:38.213 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:38 GMT /vclfcn-19/obj 2024-02-01T10:27:38.213 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:++st0oeXd3mm/tVRMqqNI504Td0= 2024-02-01T10:27:38.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:++st0oeXd3mm/tVRMqqNI504Td0=', 'Content-Length': '0'} 2024-02-01T10:27:38.219 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000903167a1f6523a46-0065bb721a-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:38.219 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:38.219 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:38.219 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:38.219 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:38.220 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:38.220 DEBUG:boto:Method: GET 2024-02-01T10:27:38.220 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:38.220 DEBUG:boto:Data: 2024-02-01T10:27:38.220 DEBUG:boto:Headers: {} 2024-02-01T10:27:38.220 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:38.220 DEBUG:boto:Port: 8000 2024-02-01T10:27:38.220 DEBUG:boto:Params: {} 2024-02-01T10:27:38.220 DEBUG:boto:Token: None 2024-02-01T10:27:38.221 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:38 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:38.221 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fORJMwA/ifU8HAv/Tts9RTK4J7w= 2024-02-01T10:27:38.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fORJMwA/ifU8HAv/Tts9RTK4J7w=', 'Content-Length': '0'} 2024-02-01T10:27:38.224 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000260c33563dcc832e-0065bb721a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:38.224 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:38.224 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:38.224 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:38.224 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:38.225 DEBUG:boto:Method: GET 2024-02-01T10:27:38.225 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:38.225 DEBUG:boto:Data: 2024-02-01T10:27:38.225 DEBUG:boto:Headers: {} 2024-02-01T10:27:38.225 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:38.225 DEBUG:boto:Port: 8000 2024-02-01T10:27:38.225 DEBUG:boto:Params: {} 2024-02-01T10:27:38.225 DEBUG:boto:Token: None 2024-02-01T10:27:38.225 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:38 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:38.225 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fORJMwA/ifU8HAv/Tts9RTK4J7w= 2024-02-01T10:27:38.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fORJMwA/ifU8HAv/Tts9RTK4J7w=', 'Content-Length': '0'} 2024-02-01T10:27:38.228 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000026b0bec520b69eed-0065bb721a-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:38.228 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:38.229 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:38.229 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:38.229 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:38.229 DEBUG:boto:Method: GET 2024-02-01T10:27:38.229 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:38.229 DEBUG:boto:Data: 2024-02-01T10:27:38.229 DEBUG:boto:Headers: {} 2024-02-01T10:27:38.229 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:38.229 DEBUG:boto:Port: 8000 2024-02-01T10:27:38.229 DEBUG:boto:Params: {} 2024-02-01T10:27:38.229 DEBUG:boto:Token: None 2024-02-01T10:27:38.230 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:38 GMT /vclfcn-19/ 2024-02-01T10:27:38.230 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5ixPGml/RgIDcptGflveRBKaBzE= 2024-02-01T10:27:38.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5ixPGml/RgIDcptGflveRBKaBzE=', 'Content-Length': '0'} 2024-02-01T10:27:38.235 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000561400599df74542-0065bb721a-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:38.276 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:38.277 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:38.277 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:38.277 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:38.277 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:38.277 DEBUG:boto:Method: GET 2024-02-01T10:27:38.277 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:38.277 DEBUG:boto:Data: 2024-02-01T10:27:38.277 DEBUG:boto:Headers: {} 2024-02-01T10:27:38.277 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:38.277 DEBUG:boto:Port: 8000 2024-02-01T10:27:38.277 DEBUG:boto:Params: {} 2024-02-01T10:27:38.278 DEBUG:boto:Token: None 2024-02-01T10:27:38.278 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:38 GMT /vclfcn-19/ 2024-02-01T10:27:38.278 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5ixPGml/RgIDcptGflveRBKaBzE= 2024-02-01T10:27:38.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5ixPGml/RgIDcptGflveRBKaBzE=', 'Content-Length': '0'} 2024-02-01T10:27:38.283 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3684f69dccd49eb-0065bb721a-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:38.283 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:38.283 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:38.284 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:38.284 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:38.284 DEBUG:boto:path=/vclfcn-19/obj?versionId=9CJ3w1EusK8OST2k17sTdTM.cuh3E3e 2024-02-01T10:27:38.284 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=9CJ3w1EusK8OST2k17sTdTM.cuh3E3e 2024-02-01T10:27:38.284 DEBUG:boto:Method: GET 2024-02-01T10:27:38.284 DEBUG:boto:Path: /vclfcn-19/obj?versionId=9CJ3w1EusK8OST2k17sTdTM.cuh3E3e 2024-02-01T10:27:38.285 DEBUG:boto:Data: 2024-02-01T10:27:38.285 DEBUG:boto:Headers: {} 2024-02-01T10:27:38.285 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:38.285 DEBUG:boto:Port: 8000 2024-02-01T10:27:38.285 DEBUG:boto:Params: {} 2024-02-01T10:27:38.285 DEBUG:boto:Token: None 2024-02-01T10:27:38.285 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:38 GMT /vclfcn-19/obj?versionId=9CJ3w1EusK8OST2k17sTdTM.cuh3E3e 2024-02-01T10:27:38.285 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:xKwXYnh/f+3bsbqe2h6us+9EDMY= 2024-02-01T10:27:38.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:xKwXYnh/f+3bsbqe2h6us+9EDMY=', 'Content-Length': '0'} 2024-02-01T10:27:38.288 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '9CJ3w1EusK8OST2k17sTdTM.cuh3E3e'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000064684efbe693366e-0065bb721a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:38.288 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:38.289 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:38.289 DEBUG:boto:path=/vclfcn-19/obj?versionId=9CJ3w1EusK8OST2k17sTdTM.cuh3E3e 2024-02-01T10:27:38.289 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=9CJ3w1EusK8OST2k17sTdTM.cuh3E3e 2024-02-01T10:27:38.289 DEBUG:boto:Method: GET 2024-02-01T10:27:38.289 DEBUG:boto:Path: /vclfcn-19/obj?versionId=9CJ3w1EusK8OST2k17sTdTM.cuh3E3e 2024-02-01T10:27:38.289 DEBUG:boto:Data: 2024-02-01T10:27:38.289 DEBUG:boto:Headers: {} 2024-02-01T10:27:38.289 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:38.289 DEBUG:boto:Port: 8000 2024-02-01T10:27:38.289 DEBUG:boto:Params: {} 2024-02-01T10:27:38.290 DEBUG:boto:Token: None 2024-02-01T10:27:38.290 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:38 GMT /vclfcn-19/obj?versionId=9CJ3w1EusK8OST2k17sTdTM.cuh3E3e 2024-02-01T10:27:38.290 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:xKwXYnh/f+3bsbqe2h6us+9EDMY= 2024-02-01T10:27:38.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:xKwXYnh/f+3bsbqe2h6us+9EDMY=', 'Content-Length': '0'} 2024-02-01T10:27:38.293 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '9CJ3w1EusK8OST2k17sTdTM.cuh3E3e'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f60de7efbdb8a9e6-0065bb721a-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:38.336 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:38.336 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:38.336 DEBUG:boto:path=/vclfcn-19/obj?versionId=9CJ3w1EusK8OST2k17sTdTM.cuh3E3e 2024-02-01T10:27:38.336 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=9CJ3w1EusK8OST2k17sTdTM.cuh3E3e 2024-02-01T10:27:38.337 DEBUG:boto:Method: HEAD 2024-02-01T10:27:38.337 DEBUG:boto:Path: /vclfcn-19/obj?versionId=9CJ3w1EusK8OST2k17sTdTM.cuh3E3e 2024-02-01T10:27:38.337 DEBUG:boto:Data: 2024-02-01T10:27:38.337 DEBUG:boto:Headers: {} 2024-02-01T10:27:38.337 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:38.337 DEBUG:boto:Port: 8000 2024-02-01T10:27:38.337 DEBUG:boto:Params: {} 2024-02-01T10:27:38.337 DEBUG:boto:Token: None 2024-02-01T10:27:38.337 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:38 GMT /vclfcn-19/obj?versionId=9CJ3w1EusK8OST2k17sTdTM.cuh3E3e 2024-02-01T10:27:38.338 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ioKeOo4EYOp/rP+1MldCo+LuJQo= 2024-02-01T10:27:38.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ioKeOo4EYOp/rP+1MldCo+LuJQo=', 'Content-Length': '0'} 2024-02-01T10:27:38.341 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '9CJ3w1EusK8OST2k17sTdTM.cuh3E3e'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000034d5fce0c646095-0065bb721a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:38.341 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:38.341 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:38.341 DEBUG:boto:path=/vclfcn-19/obj?versionId=9CJ3w1EusK8OST2k17sTdTM.cuh3E3e 2024-02-01T10:27:38.341 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=9CJ3w1EusK8OST2k17sTdTM.cuh3E3e 2024-02-01T10:27:38.342 DEBUG:boto:Method: HEAD 2024-02-01T10:27:38.342 DEBUG:boto:Path: /vclfcn-19/obj?versionId=9CJ3w1EusK8OST2k17sTdTM.cuh3E3e 2024-02-01T10:27:38.342 DEBUG:boto:Data: 2024-02-01T10:27:38.342 DEBUG:boto:Headers: {} 2024-02-01T10:27:38.342 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:38.342 DEBUG:boto:Port: 8000 2024-02-01T10:27:38.342 DEBUG:boto:Params: {} 2024-02-01T10:27:38.342 DEBUG:boto:Token: None 2024-02-01T10:27:38.342 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:38 GMT /vclfcn-19/obj?versionId=9CJ3w1EusK8OST2k17sTdTM.cuh3E3e 2024-02-01T10:27:38.342 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ioKeOo4EYOp/rP+1MldCo+LuJQo= 2024-02-01T10:27:38.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ioKeOo4EYOp/rP+1MldCo+LuJQo=', 'Content-Length': '0'} 2024-02-01T10:27:38.345 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '9CJ3w1EusK8OST2k17sTdTM.cuh3E3e'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000038dfa6ce6302a0d3-0065bb721a-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:38.345 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:38.346 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:38.346 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:38.346 DEBUG:boto:path=/vclfcn-19/obj?versionId=9CJ3w1EusK8OST2k17sTdTM.cuh3E3e 2024-02-01T10:27:38.346 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=9CJ3w1EusK8OST2k17sTdTM.cuh3E3e 2024-02-01T10:27:38.346 DEBUG:boto:Method: GET 2024-02-01T10:27:38.346 DEBUG:boto:Path: /vclfcn-19/obj?versionId=9CJ3w1EusK8OST2k17sTdTM.cuh3E3e 2024-02-01T10:27:38.346 DEBUG:boto:Data: 2024-02-01T10:27:38.346 DEBUG:boto:Headers: {} 2024-02-01T10:27:38.346 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:38.346 DEBUG:boto:Port: 8000 2024-02-01T10:27:38.346 DEBUG:boto:Params: {} 2024-02-01T10:27:38.347 DEBUG:boto:Token: None 2024-02-01T10:27:38.347 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:38 GMT /vclfcn-19/obj?versionId=9CJ3w1EusK8OST2k17sTdTM.cuh3E3e 2024-02-01T10:27:38.347 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:xKwXYnh/f+3bsbqe2h6us+9EDMY= 2024-02-01T10:27:38.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:xKwXYnh/f+3bsbqe2h6us+9EDMY=', 'Content-Length': '0'} 2024-02-01T10:27:38.349 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '9CJ3w1EusK8OST2k17sTdTM.cuh3E3e'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003327b9f69da59c85-0065bb721a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:38.350 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:38.350 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:38.350 DEBUG:boto:path=/vclfcn-19/obj?versionId=9CJ3w1EusK8OST2k17sTdTM.cuh3E3e 2024-02-01T10:27:38.350 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=9CJ3w1EusK8OST2k17sTdTM.cuh3E3e 2024-02-01T10:27:38.350 DEBUG:boto:Method: GET 2024-02-01T10:27:38.350 DEBUG:boto:Path: /vclfcn-19/obj?versionId=9CJ3w1EusK8OST2k17sTdTM.cuh3E3e 2024-02-01T10:27:38.350 DEBUG:boto:Data: 2024-02-01T10:27:38.350 DEBUG:boto:Headers: {} 2024-02-01T10:27:38.350 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:38.351 DEBUG:boto:Port: 8000 2024-02-01T10:27:38.351 DEBUG:boto:Params: {} 2024-02-01T10:27:38.351 DEBUG:boto:Token: None 2024-02-01T10:27:38.351 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:38 GMT /vclfcn-19/obj?versionId=9CJ3w1EusK8OST2k17sTdTM.cuh3E3e 2024-02-01T10:27:38.351 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:xKwXYnh/f+3bsbqe2h6us+9EDMY= 2024-02-01T10:27:38.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:xKwXYnh/f+3bsbqe2h6us+9EDMY=', 'Content-Length': '0'} 2024-02-01T10:27:38.354 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '9CJ3w1EusK8OST2k17sTdTM.cuh3E3e'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000070f702b30dca1cdb-0065bb721a-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:38.354 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:38.354 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:38.354 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:38.354 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:38.354 DEBUG:boto:Method: GET 2024-02-01T10:27:38.355 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:38.355 DEBUG:boto:Data: 2024-02-01T10:27:38.355 DEBUG:boto:Headers: {} 2024-02-01T10:27:38.355 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:38.355 DEBUG:boto:Port: 8000 2024-02-01T10:27:38.355 DEBUG:boto:Params: {} 2024-02-01T10:27:38.355 DEBUG:boto:Token: None 2024-02-01T10:27:38.355 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:38 GMT /vclfcn-19/ 2024-02-01T10:27:38.355 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5ixPGml/RgIDcptGflveRBKaBzE= 2024-02-01T10:27:38.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5ixPGml/RgIDcptGflveRBKaBzE=', 'Content-Length': '0'} 2024-02-01T10:27:38.361 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c3945d0b749e914-0065bb721a-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:38.408 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:38.409 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:38.409 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:38.409 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:38.409 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:38.409 DEBUG:boto:Method: GET 2024-02-01T10:27:38.409 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:38.409 DEBUG:boto:Data: 2024-02-01T10:27:38.409 DEBUG:boto:Headers: {} 2024-02-01T10:27:38.409 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:38.410 DEBUG:boto:Port: 8000 2024-02-01T10:27:38.410 DEBUG:boto:Params: {} 2024-02-01T10:27:38.410 DEBUG:boto:Token: None 2024-02-01T10:27:38.410 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:38 GMT /vclfcn-19/ 2024-02-01T10:27:38.410 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5ixPGml/RgIDcptGflveRBKaBzE= 2024-02-01T10:27:38.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5ixPGml/RgIDcptGflveRBKaBzE=', 'Content-Length': '0'} 2024-02-01T10:27:38.416 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f5b36953d4db01f-0065bb721a-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:38.416 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:38.417 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:38.417 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:38.417 DEBUG:boto:Method: HEAD 2024-02-01T10:27:38.417 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:38.417 DEBUG:boto:Data: 2024-02-01T10:27:38.417 DEBUG:boto:Headers: {} 2024-02-01T10:27:38.417 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:38.417 DEBUG:boto:Port: 8000 2024-02-01T10:27:38.417 DEBUG:boto:Params: {} 2024-02-01T10:27:38.418 DEBUG:boto:Token: None 2024-02-01T10:27:38.418 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:38 GMT /vclfcn-19/obj 2024-02-01T10:27:38.418 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:++st0oeXd3mm/tVRMqqNI504Td0= 2024-02-01T10:27:38.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:++st0oeXd3mm/tVRMqqNI504Td0=', 'Content-Length': '0'} 2024-02-01T10:27:38.424 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004cb506834eaa81ca-0065bb721a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:38.424 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:38.425 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:38.425 DEBUG:boto:Method: HEAD 2024-02-01T10:27:38.425 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:38.425 DEBUG:boto:Data: 2024-02-01T10:27:38.425 DEBUG:boto:Headers: {} 2024-02-01T10:27:38.425 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:38.425 DEBUG:boto:Port: 8000 2024-02-01T10:27:38.425 DEBUG:boto:Params: {} 2024-02-01T10:27:38.425 DEBUG:boto:Token: None 2024-02-01T10:27:38.425 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:38 GMT /vclfcn-19/obj 2024-02-01T10:27:38.426 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:++st0oeXd3mm/tVRMqqNI504Td0= 2024-02-01T10:27:38.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:++st0oeXd3mm/tVRMqqNI504Td0=', 'Content-Length': '0'} 2024-02-01T10:27:38.432 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004ac853eaa0da9496-0065bb721a-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:38.432 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:38.432 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:38.432 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:38.432 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:38.432 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:38.432 DEBUG:boto:Method: GET 2024-02-01T10:27:38.433 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:38.433 DEBUG:boto:Data: 2024-02-01T10:27:38.433 DEBUG:boto:Headers: {} 2024-02-01T10:27:38.433 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:38.433 DEBUG:boto:Port: 8000 2024-02-01T10:27:38.433 DEBUG:boto:Params: {} 2024-02-01T10:27:38.433 DEBUG:boto:Token: None 2024-02-01T10:27:38.433 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:38 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:38.433 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fORJMwA/ifU8HAv/Tts9RTK4J7w= 2024-02-01T10:27:38.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fORJMwA/ifU8HAv/Tts9RTK4J7w=', 'Content-Length': '0'} 2024-02-01T10:27:38.436 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f7ce777148d16ba1-0065bb721a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:38.436 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:38.437 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:38.437 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:38.437 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:38.437 DEBUG:boto:Method: GET 2024-02-01T10:27:38.437 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:38.437 DEBUG:boto:Data: 2024-02-01T10:27:38.437 DEBUG:boto:Headers: {} 2024-02-01T10:27:38.437 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:38.437 DEBUG:boto:Port: 8000 2024-02-01T10:27:38.437 DEBUG:boto:Params: {} 2024-02-01T10:27:38.437 DEBUG:boto:Token: None 2024-02-01T10:27:38.438 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:38 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:38.438 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fORJMwA/ifU8HAv/Tts9RTK4J7w= 2024-02-01T10:27:38.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fORJMwA/ifU8HAv/Tts9RTK4J7w=', 'Content-Length': '0'} 2024-02-01T10:27:38.440 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a492a18b1b0f892d-0065bb721a-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:38.484 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:38.484 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:38.484 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:38.485 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:38.485 DEBUG:boto:Method: GET 2024-02-01T10:27:38.485 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:38.485 DEBUG:boto:Data: 2024-02-01T10:27:38.485 DEBUG:boto:Headers: {} 2024-02-01T10:27:38.485 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:38.486 DEBUG:boto:Port: 8000 2024-02-01T10:27:38.486 DEBUG:boto:Params: {} 2024-02-01T10:27:38.486 DEBUG:boto:Token: None 2024-02-01T10:27:38.486 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:38 GMT /vclfcn-19/ 2024-02-01T10:27:38.486 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5ixPGml/RgIDcptGflveRBKaBzE= 2024-02-01T10:27:38.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5ixPGml/RgIDcptGflveRBKaBzE=', 'Content-Length': '0'} 2024-02-01T10:27:38.492 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d739e596d755b4b-0065bb721a-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:38.492 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:38.493 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:38.493 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:38.493 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:38.493 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:38.493 DEBUG:boto:Method: GET 2024-02-01T10:27:38.493 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:38.493 DEBUG:boto:Data: 2024-02-01T10:27:38.493 DEBUG:boto:Headers: {} 2024-02-01T10:27:38.494 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:38.494 DEBUG:boto:Port: 8000 2024-02-01T10:27:38.494 DEBUG:boto:Params: {} 2024-02-01T10:27:38.494 DEBUG:boto:Token: None 2024-02-01T10:27:38.494 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:38 GMT /vclfcn-19/ 2024-02-01T10:27:38.494 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5ixPGml/RgIDcptGflveRBKaBzE= 2024-02-01T10:27:38.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5ixPGml/RgIDcptGflveRBKaBzE=', 'Content-Length': '0'} 2024-02-01T10:27:38.500 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2bf3a0445fa7b76-0065bb721a-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:38.500 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:38.500 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:38.501 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:38.501 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:38.501 DEBUG:boto:path=/vclfcn-19/obj?versionId=Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv 2024-02-01T10:27:38.501 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv 2024-02-01T10:27:38.501 DEBUG:boto:Method: GET 2024-02-01T10:27:38.501 DEBUG:boto:Path: /vclfcn-19/obj?versionId=Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv 2024-02-01T10:27:38.501 DEBUG:boto:Data: 2024-02-01T10:27:38.501 DEBUG:boto:Headers: {} 2024-02-01T10:27:38.501 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:38.501 DEBUG:boto:Port: 8000 2024-02-01T10:27:38.501 DEBUG:boto:Params: {} 2024-02-01T10:27:38.502 DEBUG:boto:Token: None 2024-02-01T10:27:38.502 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:38 GMT /vclfcn-19/obj?versionId=Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv 2024-02-01T10:27:38.502 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BSIZaT1gU26PphAj5v1gYHJuO5w= 2024-02-01T10:27:38.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BSIZaT1gU26PphAj5v1gYHJuO5w=', 'Content-Length': '0'} 2024-02-01T10:27:38.505 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007d4d7e4e600a76f6-0065bb721a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:38.548 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:38.548 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:38.548 DEBUG:boto:path=/vclfcn-19/obj?versionId=Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv 2024-02-01T10:27:38.549 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv 2024-02-01T10:27:38.549 DEBUG:boto:Method: GET 2024-02-01T10:27:38.549 DEBUG:boto:Path: /vclfcn-19/obj?versionId=Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv 2024-02-01T10:27:38.549 DEBUG:boto:Data: 2024-02-01T10:27:38.549 DEBUG:boto:Headers: {} 2024-02-01T10:27:38.549 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:38.549 DEBUG:boto:Port: 8000 2024-02-01T10:27:38.549 DEBUG:boto:Params: {} 2024-02-01T10:27:38.549 DEBUG:boto:Token: None 2024-02-01T10:27:38.549 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:38 GMT /vclfcn-19/obj?versionId=Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv 2024-02-01T10:27:38.550 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BSIZaT1gU26PphAj5v1gYHJuO5w= 2024-02-01T10:27:38.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BSIZaT1gU26PphAj5v1gYHJuO5w=', 'Content-Length': '0'} 2024-02-01T10:27:38.553 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a4194ca7e18c946d-0065bb721a-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:38.553 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:38.553 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:38.553 DEBUG:boto:path=/vclfcn-19/obj?versionId=Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv 2024-02-01T10:27:38.554 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv 2024-02-01T10:27:38.554 DEBUG:boto:Method: HEAD 2024-02-01T10:27:38.554 DEBUG:boto:Path: /vclfcn-19/obj?versionId=Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv 2024-02-01T10:27:38.554 DEBUG:boto:Data: 2024-02-01T10:27:38.554 DEBUG:boto:Headers: {} 2024-02-01T10:27:38.554 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:38.554 DEBUG:boto:Port: 8000 2024-02-01T10:27:38.554 DEBUG:boto:Params: {} 2024-02-01T10:27:38.554 DEBUG:boto:Token: None 2024-02-01T10:27:38.554 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:38 GMT /vclfcn-19/obj?versionId=Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv 2024-02-01T10:27:38.555 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:UIn0fTGfLUxwSUuLvj1dH/YJ6AM= 2024-02-01T10:27:38.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:UIn0fTGfLUxwSUuLvj1dH/YJ6AM=', 'Content-Length': '0'} 2024-02-01T10:27:38.558 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007e6d256365b0832f-0065bb721a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:38.558 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:38.558 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:38.558 DEBUG:boto:path=/vclfcn-19/obj?versionId=Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv 2024-02-01T10:27:38.558 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv 2024-02-01T10:27:38.558 DEBUG:boto:Method: HEAD 2024-02-01T10:27:38.559 DEBUG:boto:Path: /vclfcn-19/obj?versionId=Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv 2024-02-01T10:27:38.559 DEBUG:boto:Data: 2024-02-01T10:27:38.559 DEBUG:boto:Headers: {} 2024-02-01T10:27:38.559 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:38.559 DEBUG:boto:Port: 8000 2024-02-01T10:27:38.559 DEBUG:boto:Params: {} 2024-02-01T10:27:38.559 DEBUG:boto:Token: None 2024-02-01T10:27:38.559 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:38 GMT /vclfcn-19/obj?versionId=Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv 2024-02-01T10:27:38.559 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:UIn0fTGfLUxwSUuLvj1dH/YJ6AM= 2024-02-01T10:27:38.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:UIn0fTGfLUxwSUuLvj1dH/YJ6AM=', 'Content-Length': '0'} 2024-02-01T10:27:38.562 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000049adf4355041087d-0065bb721a-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:38.562 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:38.562 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:38.563 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:38.563 DEBUG:boto:path=/vclfcn-19/obj?versionId=Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv 2024-02-01T10:27:38.563 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv 2024-02-01T10:27:38.563 DEBUG:boto:Method: GET 2024-02-01T10:27:38.563 DEBUG:boto:Path: /vclfcn-19/obj?versionId=Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv 2024-02-01T10:27:38.563 DEBUG:boto:Data: 2024-02-01T10:27:38.563 DEBUG:boto:Headers: {} 2024-02-01T10:27:38.563 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:38.563 DEBUG:boto:Port: 8000 2024-02-01T10:27:38.563 DEBUG:boto:Params: {} 2024-02-01T10:27:38.564 DEBUG:boto:Token: None 2024-02-01T10:27:38.564 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:38 GMT /vclfcn-19/obj?versionId=Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv 2024-02-01T10:27:38.564 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BSIZaT1gU26PphAj5v1gYHJuO5w= 2024-02-01T10:27:38.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BSIZaT1gU26PphAj5v1gYHJuO5w=', 'Content-Length': '0'} 2024-02-01T10:27:38.566 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a0ae32f6967190c2-0065bb721a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:38.608 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:38.608 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:38.608 DEBUG:boto:path=/vclfcn-19/obj?versionId=Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv 2024-02-01T10:27:38.609 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv 2024-02-01T10:27:38.609 DEBUG:boto:Method: GET 2024-02-01T10:27:38.609 DEBUG:boto:Path: /vclfcn-19/obj?versionId=Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv 2024-02-01T10:27:38.609 DEBUG:boto:Data: 2024-02-01T10:27:38.609 DEBUG:boto:Headers: {} 2024-02-01T10:27:38.609 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:38.609 DEBUG:boto:Port: 8000 2024-02-01T10:27:38.609 DEBUG:boto:Params: {} 2024-02-01T10:27:38.609 DEBUG:boto:Token: None 2024-02-01T10:27:38.610 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:38 GMT /vclfcn-19/obj?versionId=Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv 2024-02-01T10:27:38.610 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BSIZaT1gU26PphAj5v1gYHJuO5w= 2024-02-01T10:27:38.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BSIZaT1gU26PphAj5v1gYHJuO5w=', 'Content-Length': '0'} 2024-02-01T10:27:38.613 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d96eb7b9020835fe-0065bb721a-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:38.613 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:38.613 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:38.613 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:38.613 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:38.614 DEBUG:boto:Method: GET 2024-02-01T10:27:38.614 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:38.614 DEBUG:boto:Data: 2024-02-01T10:27:38.614 DEBUG:boto:Headers: {} 2024-02-01T10:27:38.614 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:38.614 DEBUG:boto:Port: 8000 2024-02-01T10:27:38.614 DEBUG:boto:Params: {} 2024-02-01T10:27:38.614 DEBUG:boto:Token: None 2024-02-01T10:27:38.614 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:38 GMT /vclfcn-19/ 2024-02-01T10:27:38.614 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5ixPGml/RgIDcptGflveRBKaBzE= 2024-02-01T10:27:38.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5ixPGml/RgIDcptGflveRBKaBzE=', 'Content-Length': '0'} 2024-02-01T10:27:38.621 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000287829903001606e-0065bb721a-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:38.664 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:38.664 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:38.665 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:38.665 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:38.665 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:38.665 DEBUG:boto:Method: GET 2024-02-01T10:27:38.665 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:38.665 DEBUG:boto:Data: 2024-02-01T10:27:38.665 DEBUG:boto:Headers: {} 2024-02-01T10:27:38.665 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:38.665 DEBUG:boto:Port: 8000 2024-02-01T10:27:38.665 DEBUG:boto:Params: {} 2024-02-01T10:27:38.665 DEBUG:boto:Token: None 2024-02-01T10:27:38.666 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:38 GMT /vclfcn-19/ 2024-02-01T10:27:38.666 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5ixPGml/RgIDcptGflveRBKaBzE= 2024-02-01T10:27:38.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5ixPGml/RgIDcptGflveRBKaBzE=', 'Content-Length': '0'} 2024-02-01T10:27:38.672 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c8d7321b2cdbab4d-0065bb721a-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:38.672 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:38.672 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:38.672 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:38.672 DEBUG:boto:Method: HEAD 2024-02-01T10:27:38.673 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:38.673 DEBUG:boto:Data: 2024-02-01T10:27:38.673 DEBUG:boto:Headers: {} 2024-02-01T10:27:38.673 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:38.673 DEBUG:boto:Port: 8000 2024-02-01T10:27:38.673 DEBUG:boto:Params: {} 2024-02-01T10:27:38.673 DEBUG:boto:Token: None 2024-02-01T10:27:38.673 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:38 GMT /vclfcn-19/obj 2024-02-01T10:27:38.673 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:++st0oeXd3mm/tVRMqqNI504Td0= 2024-02-01T10:27:38.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:++st0oeXd3mm/tVRMqqNI504Td0=', 'Content-Length': '0'} 2024-02-01T10:27:38.679 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000301f502eef15ef3e-0065bb721a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:38.680 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:38.680 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:38.680 DEBUG:boto:Method: HEAD 2024-02-01T10:27:38.680 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:38.680 DEBUG:boto:Data: 2024-02-01T10:27:38.680 DEBUG:boto:Headers: {} 2024-02-01T10:27:38.680 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:38.680 DEBUG:boto:Port: 8000 2024-02-01T10:27:38.680 DEBUG:boto:Params: {} 2024-02-01T10:27:38.681 DEBUG:boto:Token: None 2024-02-01T10:27:38.681 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:38 GMT /vclfcn-19/obj 2024-02-01T10:27:38.681 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:++st0oeXd3mm/tVRMqqNI504Td0= 2024-02-01T10:27:38.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:++st0oeXd3mm/tVRMqqNI504Td0=', 'Content-Length': '0'} 2024-02-01T10:27:38.688 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e224742405fbb881-0065bb721a-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:38.688 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:38.689 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:38.689 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:38.689 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:38.689 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:38.689 DEBUG:boto:Method: GET 2024-02-01T10:27:38.689 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:38.689 DEBUG:boto:Data: 2024-02-01T10:27:38.690 DEBUG:boto:Headers: {} 2024-02-01T10:27:38.690 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:38.690 DEBUG:boto:Port: 8000 2024-02-01T10:27:38.690 DEBUG:boto:Params: {} 2024-02-01T10:27:38.690 DEBUG:boto:Token: None 2024-02-01T10:27:38.690 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:38 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:38.690 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fORJMwA/ifU8HAv/Tts9RTK4J7w= 2024-02-01T10:27:38.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fORJMwA/ifU8HAv/Tts9RTK4J7w=', 'Content-Length': '0'} 2024-02-01T10:27:38.693 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009571151e90811b25-0065bb721a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:38.694 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:38.694 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:38.694 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:38.694 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:38.694 DEBUG:boto:Method: GET 2024-02-01T10:27:38.694 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:38.694 DEBUG:boto:Data: 2024-02-01T10:27:38.694 DEBUG:boto:Headers: {} 2024-02-01T10:27:38.695 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:38.695 DEBUG:boto:Port: 8000 2024-02-01T10:27:38.695 DEBUG:boto:Params: {} 2024-02-01T10:27:38.695 DEBUG:boto:Token: None 2024-02-01T10:27:38.695 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:38 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:38.695 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fORJMwA/ifU8HAv/Tts9RTK4J7w= 2024-02-01T10:27:38.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fORJMwA/ifU8HAv/Tts9RTK4J7w=', 'Content-Length': '0'} 2024-02-01T10:27:38.698 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007707cfd5679e168d-0065bb721a-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:38.740 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:38.740 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:38.740 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:38.741 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:38.741 DEBUG:boto:Method: GET 2024-02-01T10:27:38.741 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:38.741 DEBUG:boto:Data: 2024-02-01T10:27:38.741 DEBUG:boto:Headers: {} 2024-02-01T10:27:38.741 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:38.741 DEBUG:boto:Port: 8000 2024-02-01T10:27:38.741 DEBUG:boto:Params: {} 2024-02-01T10:27:38.741 DEBUG:boto:Token: None 2024-02-01T10:27:38.741 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:38 GMT /vclfcn-19/ 2024-02-01T10:27:38.742 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5ixPGml/RgIDcptGflveRBKaBzE= 2024-02-01T10:27:38.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5ixPGml/RgIDcptGflveRBKaBzE=', 'Content-Length': '0'} 2024-02-01T10:27:38.747 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d727c6258c024e4-0065bb721a-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:38.747 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:38.748 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:38.748 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:38.748 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:38.748 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:38.748 DEBUG:boto:Method: GET 2024-02-01T10:27:38.748 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:38.748 DEBUG:boto:Data: 2024-02-01T10:27:38.748 DEBUG:boto:Headers: {} 2024-02-01T10:27:38.748 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:38.749 DEBUG:boto:Port: 8000 2024-02-01T10:27:38.749 DEBUG:boto:Params: {} 2024-02-01T10:27:38.749 DEBUG:boto:Token: None 2024-02-01T10:27:38.749 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:38 GMT /vclfcn-19/ 2024-02-01T10:27:38.749 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5ixPGml/RgIDcptGflveRBKaBzE= 2024-02-01T10:27:38.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5ixPGml/RgIDcptGflveRBKaBzE=', 'Content-Length': '0'} 2024-02-01T10:27:38.754 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf55e63be3a8a681-0065bb721a-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:38.754 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:38.755 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:38.755 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:38.755 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:38.755 DEBUG:boto:path=/vclfcn-19/obj?versionId=iGO2QyMOL81IaF-H4sHDqyECPlnZBmE 2024-02-01T10:27:38.755 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=iGO2QyMOL81IaF-H4sHDqyECPlnZBmE 2024-02-01T10:27:38.755 DEBUG:boto:Method: GET 2024-02-01T10:27:38.755 DEBUG:boto:Path: /vclfcn-19/obj?versionId=iGO2QyMOL81IaF-H4sHDqyECPlnZBmE 2024-02-01T10:27:38.756 DEBUG:boto:Data: 2024-02-01T10:27:38.756 DEBUG:boto:Headers: {} 2024-02-01T10:27:38.756 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:38.756 DEBUG:boto:Port: 8000 2024-02-01T10:27:38.756 DEBUG:boto:Params: {} 2024-02-01T10:27:38.756 DEBUG:boto:Token: None 2024-02-01T10:27:38.756 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:38 GMT /vclfcn-19/obj?versionId=iGO2QyMOL81IaF-H4sHDqyECPlnZBmE 2024-02-01T10:27:38.756 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:eBL4MkJYstKUb6Ya6gME2AeriaY= 2024-02-01T10:27:38.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:eBL4MkJYstKUb6Ya6gME2AeriaY=', 'Content-Length': '0'} 2024-02-01T10:27:38.760 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'iGO2QyMOL81IaF-H4sHDqyECPlnZBmE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cf0000a0d655f65b-0065bb721a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:38.800 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:38.800 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:38.800 DEBUG:boto:path=/vclfcn-19/obj?versionId=iGO2QyMOL81IaF-H4sHDqyECPlnZBmE 2024-02-01T10:27:38.800 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=iGO2QyMOL81IaF-H4sHDqyECPlnZBmE 2024-02-01T10:27:38.801 DEBUG:boto:Method: GET 2024-02-01T10:27:38.801 DEBUG:boto:Path: /vclfcn-19/obj?versionId=iGO2QyMOL81IaF-H4sHDqyECPlnZBmE 2024-02-01T10:27:38.801 DEBUG:boto:Data: 2024-02-01T10:27:38.801 DEBUG:boto:Headers: {} 2024-02-01T10:27:38.801 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:38.801 DEBUG:boto:Port: 8000 2024-02-01T10:27:38.801 DEBUG:boto:Params: {} 2024-02-01T10:27:38.801 DEBUG:boto:Token: None 2024-02-01T10:27:38.801 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:38 GMT /vclfcn-19/obj?versionId=iGO2QyMOL81IaF-H4sHDqyECPlnZBmE 2024-02-01T10:27:38.802 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:eBL4MkJYstKUb6Ya6gME2AeriaY= 2024-02-01T10:27:38.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:eBL4MkJYstKUb6Ya6gME2AeriaY=', 'Content-Length': '0'} 2024-02-01T10:27:38.805 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'iGO2QyMOL81IaF-H4sHDqyECPlnZBmE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005a09d5d68f45f674-0065bb721a-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:38.805 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:38.805 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:38.805 DEBUG:boto:path=/vclfcn-19/obj?versionId=iGO2QyMOL81IaF-H4sHDqyECPlnZBmE 2024-02-01T10:27:38.805 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=iGO2QyMOL81IaF-H4sHDqyECPlnZBmE 2024-02-01T10:27:38.805 DEBUG:boto:Method: HEAD 2024-02-01T10:27:38.806 DEBUG:boto:Path: /vclfcn-19/obj?versionId=iGO2QyMOL81IaF-H4sHDqyECPlnZBmE 2024-02-01T10:27:38.806 DEBUG:boto:Data: 2024-02-01T10:27:38.806 DEBUG:boto:Headers: {} 2024-02-01T10:27:38.806 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:38.806 DEBUG:boto:Port: 8000 2024-02-01T10:27:38.806 DEBUG:boto:Params: {} 2024-02-01T10:27:38.806 DEBUG:boto:Token: None 2024-02-01T10:27:38.806 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:38 GMT /vclfcn-19/obj?versionId=iGO2QyMOL81IaF-H4sHDqyECPlnZBmE 2024-02-01T10:27:38.806 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jMpXwx3jwSx+FE7awdDnW6f1Iks= 2024-02-01T10:27:38.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jMpXwx3jwSx+FE7awdDnW6f1Iks=', 'Content-Length': '0'} 2024-02-01T10:27:38.809 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'iGO2QyMOL81IaF-H4sHDqyECPlnZBmE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007d5d39f6f19def3b-0065bb721a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:38.809 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:38.810 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:38.810 DEBUG:boto:path=/vclfcn-19/obj?versionId=iGO2QyMOL81IaF-H4sHDqyECPlnZBmE 2024-02-01T10:27:38.810 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=iGO2QyMOL81IaF-H4sHDqyECPlnZBmE 2024-02-01T10:27:38.810 DEBUG:boto:Method: HEAD 2024-02-01T10:27:38.810 DEBUG:boto:Path: /vclfcn-19/obj?versionId=iGO2QyMOL81IaF-H4sHDqyECPlnZBmE 2024-02-01T10:27:38.810 DEBUG:boto:Data: 2024-02-01T10:27:38.810 DEBUG:boto:Headers: {} 2024-02-01T10:27:38.810 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:38.810 DEBUG:boto:Port: 8000 2024-02-01T10:27:38.810 DEBUG:boto:Params: {} 2024-02-01T10:27:38.810 DEBUG:boto:Token: None 2024-02-01T10:27:38.811 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:38 GMT /vclfcn-19/obj?versionId=iGO2QyMOL81IaF-H4sHDqyECPlnZBmE 2024-02-01T10:27:38.811 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jMpXwx3jwSx+FE7awdDnW6f1Iks= 2024-02-01T10:27:38.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jMpXwx3jwSx+FE7awdDnW6f1Iks=', 'Content-Length': '0'} 2024-02-01T10:27:38.814 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'iGO2QyMOL81IaF-H4sHDqyECPlnZBmE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a7942eeb44da8a02-0065bb721a-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:38.814 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:38.814 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:38.814 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:38.814 DEBUG:boto:path=/vclfcn-19/obj?versionId=iGO2QyMOL81IaF-H4sHDqyECPlnZBmE 2024-02-01T10:27:38.814 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=iGO2QyMOL81IaF-H4sHDqyECPlnZBmE 2024-02-01T10:27:38.814 DEBUG:boto:Method: GET 2024-02-01T10:27:38.815 DEBUG:boto:Path: /vclfcn-19/obj?versionId=iGO2QyMOL81IaF-H4sHDqyECPlnZBmE 2024-02-01T10:27:38.815 DEBUG:boto:Data: 2024-02-01T10:27:38.815 DEBUG:boto:Headers: {} 2024-02-01T10:27:38.815 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:38.815 DEBUG:boto:Port: 8000 2024-02-01T10:27:38.815 DEBUG:boto:Params: {} 2024-02-01T10:27:38.815 DEBUG:boto:Token: None 2024-02-01T10:27:38.815 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:38 GMT /vclfcn-19/obj?versionId=iGO2QyMOL81IaF-H4sHDqyECPlnZBmE 2024-02-01T10:27:38.815 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:eBL4MkJYstKUb6Ya6gME2AeriaY= 2024-02-01T10:27:38.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:eBL4MkJYstKUb6Ya6gME2AeriaY=', 'Content-Length': '0'} 2024-02-01T10:27:38.818 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'iGO2QyMOL81IaF-H4sHDqyECPlnZBmE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f3ff59a332b2602d-0065bb721a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:38.860 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:38.860 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:38.860 DEBUG:boto:path=/vclfcn-19/obj?versionId=iGO2QyMOL81IaF-H4sHDqyECPlnZBmE 2024-02-01T10:27:38.860 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=iGO2QyMOL81IaF-H4sHDqyECPlnZBmE 2024-02-01T10:27:38.861 DEBUG:boto:Method: GET 2024-02-01T10:27:38.861 DEBUG:boto:Path: /vclfcn-19/obj?versionId=iGO2QyMOL81IaF-H4sHDqyECPlnZBmE 2024-02-01T10:27:38.861 DEBUG:boto:Data: 2024-02-01T10:27:38.861 DEBUG:boto:Headers: {} 2024-02-01T10:27:38.861 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:38.861 DEBUG:boto:Port: 8000 2024-02-01T10:27:38.861 DEBUG:boto:Params: {} 2024-02-01T10:27:38.861 DEBUG:boto:Token: None 2024-02-01T10:27:38.861 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:38 GMT /vclfcn-19/obj?versionId=iGO2QyMOL81IaF-H4sHDqyECPlnZBmE 2024-02-01T10:27:38.861 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:eBL4MkJYstKUb6Ya6gME2AeriaY= 2024-02-01T10:27:38.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:eBL4MkJYstKUb6Ya6gME2AeriaY=', 'Content-Length': '0'} 2024-02-01T10:27:38.865 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'iGO2QyMOL81IaF-H4sHDqyECPlnZBmE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000024eec9be5e433419-0065bb721a-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:38.865 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:38.865 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:38.865 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:38.865 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:38.866 DEBUG:boto:Method: GET 2024-02-01T10:27:38.866 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:38.866 DEBUG:boto:Data: 2024-02-01T10:27:38.866 DEBUG:boto:Headers: {} 2024-02-01T10:27:38.866 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:38.866 DEBUG:boto:Port: 8000 2024-02-01T10:27:38.866 DEBUG:boto:Params: {} 2024-02-01T10:27:38.866 DEBUG:boto:Token: None 2024-02-01T10:27:38.866 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:38 GMT /vclfcn-19/ 2024-02-01T10:27:38.866 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5ixPGml/RgIDcptGflveRBKaBzE= 2024-02-01T10:27:38.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5ixPGml/RgIDcptGflveRBKaBzE=', 'Content-Length': '0'} 2024-02-01T10:27:38.872 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a7b06987bf24b2b-0065bb721a-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:38.916 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:38.917 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:38.917 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:38.917 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:38.917 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:38.917 DEBUG:boto:Method: GET 2024-02-01T10:27:38.917 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:38.917 DEBUG:boto:Data: 2024-02-01T10:27:38.917 DEBUG:boto:Headers: {} 2024-02-01T10:27:38.917 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:38.917 DEBUG:boto:Port: 8000 2024-02-01T10:27:38.917 DEBUG:boto:Params: {} 2024-02-01T10:27:38.918 DEBUG:boto:Token: None 2024-02-01T10:27:38.918 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:38 GMT /vclfcn-19/ 2024-02-01T10:27:38.918 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5ixPGml/RgIDcptGflveRBKaBzE= 2024-02-01T10:27:38.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5ixPGml/RgIDcptGflveRBKaBzE=', 'Content-Length': '0'} 2024-02-01T10:27:38.924 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5b26e2033f1f2fe-0065bb721a-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:38.924 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:38.924 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:38.924 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:38.925 DEBUG:boto:Method: HEAD 2024-02-01T10:27:38.925 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:38.925 DEBUG:boto:Data: 2024-02-01T10:27:38.925 DEBUG:boto:Headers: {} 2024-02-01T10:27:38.925 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:38.925 DEBUG:boto:Port: 8000 2024-02-01T10:27:38.925 DEBUG:boto:Params: {} 2024-02-01T10:27:38.925 DEBUG:boto:Token: None 2024-02-01T10:27:38.925 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:38 GMT /vclfcn-19/obj 2024-02-01T10:27:38.926 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:++st0oeXd3mm/tVRMqqNI504Td0= 2024-02-01T10:27:38.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:++st0oeXd3mm/tVRMqqNI504Td0=', 'Content-Length': '0'} 2024-02-01T10:27:38.932 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000501727817fea08ae-0065bb721a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:38.932 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:38.932 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:38.932 DEBUG:boto:Method: HEAD 2024-02-01T10:27:38.932 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:38.933 DEBUG:boto:Data: 2024-02-01T10:27:38.933 DEBUG:boto:Headers: {} 2024-02-01T10:27:38.933 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:38.933 DEBUG:boto:Port: 8000 2024-02-01T10:27:38.933 DEBUG:boto:Params: {} 2024-02-01T10:27:38.933 DEBUG:boto:Token: None 2024-02-01T10:27:38.933 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:38 GMT /vclfcn-19/obj 2024-02-01T10:27:38.933 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:++st0oeXd3mm/tVRMqqNI504Td0= 2024-02-01T10:27:38.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:++st0oeXd3mm/tVRMqqNI504Td0=', 'Content-Length': '0'} 2024-02-01T10:27:38.939 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f9c7173b1473ba65-0065bb721a-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:38.940 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:38.940 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:38.940 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:38.940 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:38.940 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:38.940 DEBUG:boto:Method: GET 2024-02-01T10:27:38.940 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:38.940 DEBUG:boto:Data: 2024-02-01T10:27:38.940 DEBUG:boto:Headers: {} 2024-02-01T10:27:38.940 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:38.941 DEBUG:boto:Port: 8000 2024-02-01T10:27:38.941 DEBUG:boto:Params: {} 2024-02-01T10:27:38.941 DEBUG:boto:Token: None 2024-02-01T10:27:38.941 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:38 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:38.941 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fORJMwA/ifU8HAv/Tts9RTK4J7w= 2024-02-01T10:27:38.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fORJMwA/ifU8HAv/Tts9RTK4J7w=', 'Content-Length': '0'} 2024-02-01T10:27:38.944 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000933b0ba9c60bbfb4-0065bb721a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:38.944 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:38.944 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:38.944 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:38.945 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:38.945 DEBUG:boto:Method: GET 2024-02-01T10:27:38.945 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:38.945 DEBUG:boto:Data: 2024-02-01T10:27:38.945 DEBUG:boto:Headers: {} 2024-02-01T10:27:38.945 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:38.945 DEBUG:boto:Port: 8000 2024-02-01T10:27:38.945 DEBUG:boto:Params: {} 2024-02-01T10:27:38.945 DEBUG:boto:Token: None 2024-02-01T10:27:38.945 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:38 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:38.946 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fORJMwA/ifU8HAv/Tts9RTK4J7w= 2024-02-01T10:27:38.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fORJMwA/ifU8HAv/Tts9RTK4J7w=', 'Content-Length': '0'} 2024-02-01T10:27:38.948 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d659002100ce58e4-0065bb721a-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:38.992 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:38.992 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:38.993 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:38.993 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:38.993 DEBUG:boto:Method: GET 2024-02-01T10:27:38.993 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:38.993 DEBUG:boto:Data: 2024-02-01T10:27:38.993 DEBUG:boto:Headers: {} 2024-02-01T10:27:38.993 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:38.993 DEBUG:boto:Port: 8000 2024-02-01T10:27:38.993 DEBUG:boto:Params: {} 2024-02-01T10:27:38.993 DEBUG:boto:Token: None 2024-02-01T10:27:38.994 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:38 GMT /vclfcn-19/ 2024-02-01T10:27:38.994 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5ixPGml/RgIDcptGflveRBKaBzE= 2024-02-01T10:27:38.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5ixPGml/RgIDcptGflveRBKaBzE=', 'Content-Length': '0'} 2024-02-01T10:27:38.999 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef31f658b5844691-0065bb721a-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:38.999 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:39.000 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:39.000 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:39.000 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:39.000 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:39.000 DEBUG:boto:Method: GET 2024-02-01T10:27:39.000 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:39.000 DEBUG:boto:Data: 2024-02-01T10:27:39.001 DEBUG:boto:Headers: {} 2024-02-01T10:27:39.001 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:39.001 DEBUG:boto:Port: 8000 2024-02-01T10:27:39.001 DEBUG:boto:Params: {} 2024-02-01T10:27:39.001 DEBUG:boto:Token: None 2024-02-01T10:27:39.001 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:39 GMT /vclfcn-19/ 2024-02-01T10:27:39.001 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QTTv5SM7hBeP5CuqyFSOV1PKaxw= 2024-02-01T10:27:39.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QTTv5SM7hBeP5CuqyFSOV1PKaxw=', 'Content-Length': '0'} 2024-02-01T10:27:39.007 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006abe165ea066888c-0065bb721b-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:39.007 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:39.007 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:39.007 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:39.008 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:39.008 DEBUG:boto:path=/vclfcn-19/obj?versionId=cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI 2024-02-01T10:27:39.008 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI 2024-02-01T10:27:39.008 DEBUG:boto:Method: GET 2024-02-01T10:27:39.008 DEBUG:boto:Path: /vclfcn-19/obj?versionId=cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI 2024-02-01T10:27:39.008 DEBUG:boto:Data: 2024-02-01T10:27:39.008 DEBUG:boto:Headers: {} 2024-02-01T10:27:39.008 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:39.008 DEBUG:boto:Port: 8000 2024-02-01T10:27:39.008 DEBUG:boto:Params: {} 2024-02-01T10:27:39.009 DEBUG:boto:Token: None 2024-02-01T10:27:39.009 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:39 GMT /vclfcn-19/obj?versionId=cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI 2024-02-01T10:27:39.009 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7m8V+y3jRNj2nalN7mxgBHI5fg0= 2024-02-01T10:27:39.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7m8V+y3jRNj2nalN7mxgBHI5fg0=', 'Content-Length': '0'} 2024-02-01T10:27:39.012 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a2982f8be054d4ca-0065bb721b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:39.052 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:39.052 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:39.052 DEBUG:boto:path=/vclfcn-19/obj?versionId=cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI 2024-02-01T10:27:39.052 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI 2024-02-01T10:27:39.053 DEBUG:boto:Method: GET 2024-02-01T10:27:39.053 DEBUG:boto:Path: /vclfcn-19/obj?versionId=cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI 2024-02-01T10:27:39.053 DEBUG:boto:Data: 2024-02-01T10:27:39.053 DEBUG:boto:Headers: {} 2024-02-01T10:27:39.053 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:39.053 DEBUG:boto:Port: 8000 2024-02-01T10:27:39.053 DEBUG:boto:Params: {} 2024-02-01T10:27:39.053 DEBUG:boto:Token: None 2024-02-01T10:27:39.053 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:39 GMT /vclfcn-19/obj?versionId=cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI 2024-02-01T10:27:39.053 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7m8V+y3jRNj2nalN7mxgBHI5fg0= 2024-02-01T10:27:39.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7m8V+y3jRNj2nalN7mxgBHI5fg0=', 'Content-Length': '0'} 2024-02-01T10:27:39.056 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d0b87775ba770f0b-0065bb721b-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:39.057 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:39.057 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:39.057 DEBUG:boto:path=/vclfcn-19/obj?versionId=cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI 2024-02-01T10:27:39.057 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI 2024-02-01T10:27:39.057 DEBUG:boto:Method: HEAD 2024-02-01T10:27:39.057 DEBUG:boto:Path: /vclfcn-19/obj?versionId=cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI 2024-02-01T10:27:39.057 DEBUG:boto:Data: 2024-02-01T10:27:39.057 DEBUG:boto:Headers: {} 2024-02-01T10:27:39.058 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:39.058 DEBUG:boto:Port: 8000 2024-02-01T10:27:39.058 DEBUG:boto:Params: {} 2024-02-01T10:27:39.058 DEBUG:boto:Token: None 2024-02-01T10:27:39.058 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:39 GMT /vclfcn-19/obj?versionId=cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI 2024-02-01T10:27:39.058 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:glEEhigR4PTcFu1+LvaXvI2maZs= 2024-02-01T10:27:39.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:glEEhigR4PTcFu1+LvaXvI2maZs=', 'Content-Length': '0'} 2024-02-01T10:27:39.061 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000092bff32124e88c9b-0065bb721b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:39.061 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:39.061 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:39.061 DEBUG:boto:path=/vclfcn-19/obj?versionId=cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI 2024-02-01T10:27:39.061 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI 2024-02-01T10:27:39.062 DEBUG:boto:Method: HEAD 2024-02-01T10:27:39.062 DEBUG:boto:Path: /vclfcn-19/obj?versionId=cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI 2024-02-01T10:27:39.062 DEBUG:boto:Data: 2024-02-01T10:27:39.062 DEBUG:boto:Headers: {} 2024-02-01T10:27:39.062 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:39.062 DEBUG:boto:Port: 8000 2024-02-01T10:27:39.062 DEBUG:boto:Params: {} 2024-02-01T10:27:39.062 DEBUG:boto:Token: None 2024-02-01T10:27:39.062 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:39 GMT /vclfcn-19/obj?versionId=cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI 2024-02-01T10:27:39.062 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:glEEhigR4PTcFu1+LvaXvI2maZs= 2024-02-01T10:27:39.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:glEEhigR4PTcFu1+LvaXvI2maZs=', 'Content-Length': '0'} 2024-02-01T10:27:39.065 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d2e57cb365294f33-0065bb721b-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:39.066 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:39.066 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:39.066 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:39.066 DEBUG:boto:path=/vclfcn-19/obj?versionId=cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI 2024-02-01T10:27:39.066 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI 2024-02-01T10:27:39.066 DEBUG:boto:Method: GET 2024-02-01T10:27:39.066 DEBUG:boto:Path: /vclfcn-19/obj?versionId=cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI 2024-02-01T10:27:39.066 DEBUG:boto:Data: 2024-02-01T10:27:39.066 DEBUG:boto:Headers: {} 2024-02-01T10:27:39.066 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:39.066 DEBUG:boto:Port: 8000 2024-02-01T10:27:39.067 DEBUG:boto:Params: {} 2024-02-01T10:27:39.067 DEBUG:boto:Token: None 2024-02-01T10:27:39.067 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:39 GMT /vclfcn-19/obj?versionId=cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI 2024-02-01T10:27:39.067 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7m8V+y3jRNj2nalN7mxgBHI5fg0= 2024-02-01T10:27:39.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7m8V+y3jRNj2nalN7mxgBHI5fg0=', 'Content-Length': '0'} 2024-02-01T10:27:39.070 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009f3fd2b731493e6f-0065bb721b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:39.112 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:39.112 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:39.112 DEBUG:boto:path=/vclfcn-19/obj?versionId=cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI 2024-02-01T10:27:39.113 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI 2024-02-01T10:27:39.113 DEBUG:boto:Method: GET 2024-02-01T10:27:39.113 DEBUG:boto:Path: /vclfcn-19/obj?versionId=cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI 2024-02-01T10:27:39.113 DEBUG:boto:Data: 2024-02-01T10:27:39.113 DEBUG:boto:Headers: {} 2024-02-01T10:27:39.113 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:39.113 DEBUG:boto:Port: 8000 2024-02-01T10:27:39.113 DEBUG:boto:Params: {} 2024-02-01T10:27:39.113 DEBUG:boto:Token: None 2024-02-01T10:27:39.113 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:39 GMT /vclfcn-19/obj?versionId=cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI 2024-02-01T10:27:39.114 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7m8V+y3jRNj2nalN7mxgBHI5fg0= 2024-02-01T10:27:39.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7m8V+y3jRNj2nalN7mxgBHI5fg0=', 'Content-Length': '0'} 2024-02-01T10:27:39.117 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000027f0703b94ef8ff6-0065bb721b-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:39.117 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:39.117 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:39.117 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:39.117 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:39.117 DEBUG:boto:Method: GET 2024-02-01T10:27:39.117 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:39.118 DEBUG:boto:Data: 2024-02-01T10:27:39.118 DEBUG:boto:Headers: {} 2024-02-01T10:27:39.118 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:39.118 DEBUG:boto:Port: 8000 2024-02-01T10:27:39.118 DEBUG:boto:Params: {} 2024-02-01T10:27:39.118 DEBUG:boto:Token: None 2024-02-01T10:27:39.118 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:39 GMT /vclfcn-19/ 2024-02-01T10:27:39.118 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QTTv5SM7hBeP5CuqyFSOV1PKaxw= 2024-02-01T10:27:39.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QTTv5SM7hBeP5CuqyFSOV1PKaxw=', 'Content-Length': '0'} 2024-02-01T10:27:39.123 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003585f92d873d09d9-0065bb721b-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:39.168 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:39.169 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:39.169 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:39.169 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:39.169 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:39.169 DEBUG:boto:Method: GET 2024-02-01T10:27:39.169 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:39.169 DEBUG:boto:Data: 2024-02-01T10:27:39.169 DEBUG:boto:Headers: {} 2024-02-01T10:27:39.169 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:39.169 DEBUG:boto:Port: 8000 2024-02-01T10:27:39.169 DEBUG:boto:Params: {} 2024-02-01T10:27:39.170 DEBUG:boto:Token: None 2024-02-01T10:27:39.170 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:39 GMT /vclfcn-19/ 2024-02-01T10:27:39.170 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QTTv5SM7hBeP5CuqyFSOV1PKaxw= 2024-02-01T10:27:39.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QTTv5SM7hBeP5CuqyFSOV1PKaxw=', 'Content-Length': '0'} 2024-02-01T10:27:39.175 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001de127a15a5d797b-0065bb721b-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:39.176 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:39.176 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:39.176 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:39.177 DEBUG:boto:Method: HEAD 2024-02-01T10:27:39.177 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:39.177 DEBUG:boto:Data: 2024-02-01T10:27:39.177 DEBUG:boto:Headers: {} 2024-02-01T10:27:39.177 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:39.177 DEBUG:boto:Port: 8000 2024-02-01T10:27:39.177 DEBUG:boto:Params: {} 2024-02-01T10:27:39.177 DEBUG:boto:Token: None 2024-02-01T10:27:39.177 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:39 GMT /vclfcn-19/obj 2024-02-01T10:27:39.177 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fjU6khZbGrZBVDx5Bwe9ScugZFs= 2024-02-01T10:27:39.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fjU6khZbGrZBVDx5Bwe9ScugZFs=', 'Content-Length': '0'} 2024-02-01T10:27:39.184 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000743280386e37b943-0065bb721b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:39.184 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:39.184 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:39.184 DEBUG:boto:Method: HEAD 2024-02-01T10:27:39.185 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:39.185 DEBUG:boto:Data: 2024-02-01T10:27:39.185 DEBUG:boto:Headers: {} 2024-02-01T10:27:39.185 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:39.185 DEBUG:boto:Port: 8000 2024-02-01T10:27:39.185 DEBUG:boto:Params: {} 2024-02-01T10:27:39.185 DEBUG:boto:Token: None 2024-02-01T10:27:39.186 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:39 GMT /vclfcn-19/obj 2024-02-01T10:27:39.186 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fjU6khZbGrZBVDx5Bwe9ScugZFs= 2024-02-01T10:27:39.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fjU6khZbGrZBVDx5Bwe9ScugZFs=', 'Content-Length': '0'} 2024-02-01T10:27:39.192 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005c549c90fc94da37-0065bb721b-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:39.192 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:39.192 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:39.192 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:39.192 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:39.192 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:39.193 DEBUG:boto:Method: GET 2024-02-01T10:27:39.193 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:39.193 DEBUG:boto:Data: 2024-02-01T10:27:39.193 DEBUG:boto:Headers: {} 2024-02-01T10:27:39.193 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:39.193 DEBUG:boto:Port: 8000 2024-02-01T10:27:39.193 DEBUG:boto:Params: {} 2024-02-01T10:27:39.194 DEBUG:boto:Token: None 2024-02-01T10:27:39.194 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:39 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:39.194 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ljWgupx1GhmoY5QBmAXMjRFmWtI= 2024-02-01T10:27:39.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ljWgupx1GhmoY5QBmAXMjRFmWtI=', 'Content-Length': '0'} 2024-02-01T10:27:39.197 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f5909c0e7bac55b6-0065bb721b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:39.198 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:39.198 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:39.198 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:39.198 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:39.198 DEBUG:boto:Method: GET 2024-02-01T10:27:39.198 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:39.198 DEBUG:boto:Data: 2024-02-01T10:27:39.198 DEBUG:boto:Headers: {} 2024-02-01T10:27:39.198 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:39.198 DEBUG:boto:Port: 8000 2024-02-01T10:27:39.199 DEBUG:boto:Params: {} 2024-02-01T10:27:39.199 DEBUG:boto:Token: None 2024-02-01T10:27:39.199 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:39 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:39.199 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ljWgupx1GhmoY5QBmAXMjRFmWtI= 2024-02-01T10:27:39.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ljWgupx1GhmoY5QBmAXMjRFmWtI=', 'Content-Length': '0'} 2024-02-01T10:27:39.202 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002e76af6a2a12a598-0065bb721b-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:39.244 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:39.244 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:39.244 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:39.245 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:39.245 DEBUG:boto:Method: GET 2024-02-01T10:27:39.245 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:39.245 DEBUG:boto:Data: 2024-02-01T10:27:39.245 DEBUG:boto:Headers: {} 2024-02-01T10:27:39.245 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:39.245 DEBUG:boto:Port: 8000 2024-02-01T10:27:39.245 DEBUG:boto:Params: {} 2024-02-01T10:27:39.245 DEBUG:boto:Token: None 2024-02-01T10:27:39.245 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:39 GMT /vclfcn-19/ 2024-02-01T10:27:39.246 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QTTv5SM7hBeP5CuqyFSOV1PKaxw= 2024-02-01T10:27:39.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QTTv5SM7hBeP5CuqyFSOV1PKaxw=', 'Content-Length': '0'} 2024-02-01T10:27:39.251 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb22f8ea7d7ec06d-0065bb721b-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:39.251 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:39.251 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:39.252 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:39.252 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:39.252 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:39.252 DEBUG:boto:Method: GET 2024-02-01T10:27:39.252 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:39.252 DEBUG:boto:Data: 2024-02-01T10:27:39.252 DEBUG:boto:Headers: {} 2024-02-01T10:27:39.252 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:39.252 DEBUG:boto:Port: 8000 2024-02-01T10:27:39.252 DEBUG:boto:Params: {} 2024-02-01T10:27:39.253 DEBUG:boto:Token: None 2024-02-01T10:27:39.253 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:39 GMT /vclfcn-19/ 2024-02-01T10:27:39.253 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QTTv5SM7hBeP5CuqyFSOV1PKaxw= 2024-02-01T10:27:39.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QTTv5SM7hBeP5CuqyFSOV1PKaxw=', 'Content-Length': '0'} 2024-02-01T10:27:39.258 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a58c75093a802699-0065bb721b-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:39.258 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:39.259 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:39.259 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:39.259 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:39.259 DEBUG:boto:path=/vclfcn-19/obj?versionId=rasya8taOZsGTj9rDi7keghf.VnPFlF 2024-02-01T10:27:39.259 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=rasya8taOZsGTj9rDi7keghf.VnPFlF 2024-02-01T10:27:39.259 DEBUG:boto:Method: GET 2024-02-01T10:27:39.259 DEBUG:boto:Path: /vclfcn-19/obj?versionId=rasya8taOZsGTj9rDi7keghf.VnPFlF 2024-02-01T10:27:39.259 DEBUG:boto:Data: 2024-02-01T10:27:39.260 DEBUG:boto:Headers: {} 2024-02-01T10:27:39.260 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:39.260 DEBUG:boto:Port: 8000 2024-02-01T10:27:39.260 DEBUG:boto:Params: {} 2024-02-01T10:27:39.260 DEBUG:boto:Token: None 2024-02-01T10:27:39.260 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:39 GMT /vclfcn-19/obj?versionId=rasya8taOZsGTj9rDi7keghf.VnPFlF 2024-02-01T10:27:39.260 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1M1OE8X+Fn8vfpO8Bk18rn8boT4= 2024-02-01T10:27:39.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1M1OE8X+Fn8vfpO8Bk18rn8boT4=', 'Content-Length': '0'} 2024-02-01T10:27:39.263 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'rasya8taOZsGTj9rDi7keghf.VnPFlF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000001d82374229edf12-0065bb721b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:39.304 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:39.304 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:39.304 DEBUG:boto:path=/vclfcn-19/obj?versionId=rasya8taOZsGTj9rDi7keghf.VnPFlF 2024-02-01T10:27:39.304 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=rasya8taOZsGTj9rDi7keghf.VnPFlF 2024-02-01T10:27:39.305 DEBUG:boto:Method: GET 2024-02-01T10:27:39.305 DEBUG:boto:Path: /vclfcn-19/obj?versionId=rasya8taOZsGTj9rDi7keghf.VnPFlF 2024-02-01T10:27:39.305 DEBUG:boto:Data: 2024-02-01T10:27:39.305 DEBUG:boto:Headers: {} 2024-02-01T10:27:39.305 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:39.305 DEBUG:boto:Port: 8000 2024-02-01T10:27:39.305 DEBUG:boto:Params: {} 2024-02-01T10:27:39.305 DEBUG:boto:Token: None 2024-02-01T10:27:39.305 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:39 GMT /vclfcn-19/obj?versionId=rasya8taOZsGTj9rDi7keghf.VnPFlF 2024-02-01T10:27:39.306 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1M1OE8X+Fn8vfpO8Bk18rn8boT4= 2024-02-01T10:27:39.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1M1OE8X+Fn8vfpO8Bk18rn8boT4=', 'Content-Length': '0'} 2024-02-01T10:27:39.309 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'rasya8taOZsGTj9rDi7keghf.VnPFlF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006b3e9f278b8cee2a-0065bb721b-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:39.309 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:39.309 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:39.309 DEBUG:boto:path=/vclfcn-19/obj?versionId=rasya8taOZsGTj9rDi7keghf.VnPFlF 2024-02-01T10:27:39.309 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=rasya8taOZsGTj9rDi7keghf.VnPFlF 2024-02-01T10:27:39.309 DEBUG:boto:Method: HEAD 2024-02-01T10:27:39.309 DEBUG:boto:Path: /vclfcn-19/obj?versionId=rasya8taOZsGTj9rDi7keghf.VnPFlF 2024-02-01T10:27:39.310 DEBUG:boto:Data: 2024-02-01T10:27:39.310 DEBUG:boto:Headers: {} 2024-02-01T10:27:39.310 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:39.310 DEBUG:boto:Port: 8000 2024-02-01T10:27:39.310 DEBUG:boto:Params: {} 2024-02-01T10:27:39.310 DEBUG:boto:Token: None 2024-02-01T10:27:39.310 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:39 GMT /vclfcn-19/obj?versionId=rasya8taOZsGTj9rDi7keghf.VnPFlF 2024-02-01T10:27:39.310 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:xtDDyRu4p5ILKECwZUszHDzh8Hc= 2024-02-01T10:27:39.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:xtDDyRu4p5ILKECwZUszHDzh8Hc=', 'Content-Length': '0'} 2024-02-01T10:27:39.313 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'rasya8taOZsGTj9rDi7keghf.VnPFlF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bed853f9dd840536-0065bb721b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:39.313 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:39.313 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:39.313 DEBUG:boto:path=/vclfcn-19/obj?versionId=rasya8taOZsGTj9rDi7keghf.VnPFlF 2024-02-01T10:27:39.314 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=rasya8taOZsGTj9rDi7keghf.VnPFlF 2024-02-01T10:27:39.314 DEBUG:boto:Method: HEAD 2024-02-01T10:27:39.314 DEBUG:boto:Path: /vclfcn-19/obj?versionId=rasya8taOZsGTj9rDi7keghf.VnPFlF 2024-02-01T10:27:39.314 DEBUG:boto:Data: 2024-02-01T10:27:39.314 DEBUG:boto:Headers: {} 2024-02-01T10:27:39.314 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:39.314 DEBUG:boto:Port: 8000 2024-02-01T10:27:39.314 DEBUG:boto:Params: {} 2024-02-01T10:27:39.314 DEBUG:boto:Token: None 2024-02-01T10:27:39.314 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:39 GMT /vclfcn-19/obj?versionId=rasya8taOZsGTj9rDi7keghf.VnPFlF 2024-02-01T10:27:39.314 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:xtDDyRu4p5ILKECwZUszHDzh8Hc= 2024-02-01T10:27:39.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:xtDDyRu4p5ILKECwZUszHDzh8Hc=', 'Content-Length': '0'} 2024-02-01T10:27:39.317 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'rasya8taOZsGTj9rDi7keghf.VnPFlF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b794a9b3a3d2f387-0065bb721b-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:39.317 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:39.317 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:39.318 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:39.318 DEBUG:boto:path=/vclfcn-19/obj?versionId=rasya8taOZsGTj9rDi7keghf.VnPFlF 2024-02-01T10:27:39.318 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=rasya8taOZsGTj9rDi7keghf.VnPFlF 2024-02-01T10:27:39.318 DEBUG:boto:Method: GET 2024-02-01T10:27:39.318 DEBUG:boto:Path: /vclfcn-19/obj?versionId=rasya8taOZsGTj9rDi7keghf.VnPFlF 2024-02-01T10:27:39.318 DEBUG:boto:Data: 2024-02-01T10:27:39.318 DEBUG:boto:Headers: {} 2024-02-01T10:27:39.318 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:39.318 DEBUG:boto:Port: 8000 2024-02-01T10:27:39.318 DEBUG:boto:Params: {} 2024-02-01T10:27:39.318 DEBUG:boto:Token: None 2024-02-01T10:27:39.319 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:39 GMT /vclfcn-19/obj?versionId=rasya8taOZsGTj9rDi7keghf.VnPFlF 2024-02-01T10:27:39.319 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1M1OE8X+Fn8vfpO8Bk18rn8boT4= 2024-02-01T10:27:39.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1M1OE8X+Fn8vfpO8Bk18rn8boT4=', 'Content-Length': '0'} 2024-02-01T10:27:39.322 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'rasya8taOZsGTj9rDi7keghf.VnPFlF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f29cc3955969d4b5-0065bb721b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:39.364 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:39.364 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:39.364 DEBUG:boto:path=/vclfcn-19/obj?versionId=rasya8taOZsGTj9rDi7keghf.VnPFlF 2024-02-01T10:27:39.364 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=rasya8taOZsGTj9rDi7keghf.VnPFlF 2024-02-01T10:27:39.365 DEBUG:boto:Method: GET 2024-02-01T10:27:39.365 DEBUG:boto:Path: /vclfcn-19/obj?versionId=rasya8taOZsGTj9rDi7keghf.VnPFlF 2024-02-01T10:27:39.365 DEBUG:boto:Data: 2024-02-01T10:27:39.365 DEBUG:boto:Headers: {} 2024-02-01T10:27:39.365 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:39.365 DEBUG:boto:Port: 8000 2024-02-01T10:27:39.365 DEBUG:boto:Params: {} 2024-02-01T10:27:39.365 DEBUG:boto:Token: None 2024-02-01T10:27:39.365 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:39 GMT /vclfcn-19/obj?versionId=rasya8taOZsGTj9rDi7keghf.VnPFlF 2024-02-01T10:27:39.366 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1M1OE8X+Fn8vfpO8Bk18rn8boT4= 2024-02-01T10:27:39.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1M1OE8X+Fn8vfpO8Bk18rn8boT4=', 'Content-Length': '0'} 2024-02-01T10:27:39.368 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'rasya8taOZsGTj9rDi7keghf.VnPFlF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009a75cc866db85c5b-0065bb721b-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:39.369 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:39.369 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:39.369 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:39.369 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:39.369 DEBUG:boto:Method: GET 2024-02-01T10:27:39.369 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:39.369 DEBUG:boto:Data: 2024-02-01T10:27:39.369 DEBUG:boto:Headers: {} 2024-02-01T10:27:39.370 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:39.370 DEBUG:boto:Port: 8000 2024-02-01T10:27:39.370 DEBUG:boto:Params: {} 2024-02-01T10:27:39.370 DEBUG:boto:Token: None 2024-02-01T10:27:39.370 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:39 GMT /vclfcn-19/ 2024-02-01T10:27:39.370 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QTTv5SM7hBeP5CuqyFSOV1PKaxw= 2024-02-01T10:27:39.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QTTv5SM7hBeP5CuqyFSOV1PKaxw=', 'Content-Length': '0'} 2024-02-01T10:27:39.375 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d7b738be9d035052-0065bb721b-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:39.416 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:39.417 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:39.417 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:39.417 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:39.417 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:39.417 DEBUG:boto:Method: GET 2024-02-01T10:27:39.417 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:39.417 DEBUG:boto:Data: 2024-02-01T10:27:39.417 DEBUG:boto:Headers: {} 2024-02-01T10:27:39.417 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:39.418 DEBUG:boto:Port: 8000 2024-02-01T10:27:39.418 DEBUG:boto:Params: {} 2024-02-01T10:27:39.418 DEBUG:boto:Token: None 2024-02-01T10:27:39.418 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:39 GMT /vclfcn-19/ 2024-02-01T10:27:39.418 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QTTv5SM7hBeP5CuqyFSOV1PKaxw= 2024-02-01T10:27:39.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QTTv5SM7hBeP5CuqyFSOV1PKaxw=', 'Content-Length': '0'} 2024-02-01T10:27:39.423 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000678c868ad77975fd-0065bb721b-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:39.423 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:39.424 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:39.424 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:39.424 DEBUG:boto:Method: HEAD 2024-02-01T10:27:39.424 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:39.424 DEBUG:boto:Data: 2024-02-01T10:27:39.424 DEBUG:boto:Headers: {} 2024-02-01T10:27:39.424 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:39.424 DEBUG:boto:Port: 8000 2024-02-01T10:27:39.424 DEBUG:boto:Params: {} 2024-02-01T10:27:39.425 DEBUG:boto:Token: None 2024-02-01T10:27:39.425 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:39 GMT /vclfcn-19/obj 2024-02-01T10:27:39.425 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fjU6khZbGrZBVDx5Bwe9ScugZFs= 2024-02-01T10:27:39.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fjU6khZbGrZBVDx5Bwe9ScugZFs=', 'Content-Length': '0'} 2024-02-01T10:27:39.432 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cb14e042e2ce7d2a-0065bb721b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:39.432 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:39.432 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:39.433 DEBUG:boto:Method: HEAD 2024-02-01T10:27:39.433 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:39.433 DEBUG:boto:Data: 2024-02-01T10:27:39.433 DEBUG:boto:Headers: {} 2024-02-01T10:27:39.433 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:39.433 DEBUG:boto:Port: 8000 2024-02-01T10:27:39.433 DEBUG:boto:Params: {} 2024-02-01T10:27:39.433 DEBUG:boto:Token: None 2024-02-01T10:27:39.433 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:39 GMT /vclfcn-19/obj 2024-02-01T10:27:39.433 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fjU6khZbGrZBVDx5Bwe9ScugZFs= 2024-02-01T10:27:39.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fjU6khZbGrZBVDx5Bwe9ScugZFs=', 'Content-Length': '0'} 2024-02-01T10:27:39.439 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004bea9e4800d1ed47-0065bb721b-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:39.439 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:39.440 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:39.440 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:39.440 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:39.440 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:39.440 DEBUG:boto:Method: GET 2024-02-01T10:27:39.440 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:39.440 DEBUG:boto:Data: 2024-02-01T10:27:39.440 DEBUG:boto:Headers: {} 2024-02-01T10:27:39.440 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:39.440 DEBUG:boto:Port: 8000 2024-02-01T10:27:39.441 DEBUG:boto:Params: {} 2024-02-01T10:27:39.441 DEBUG:boto:Token: None 2024-02-01T10:27:39.441 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:39 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:39.441 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ljWgupx1GhmoY5QBmAXMjRFmWtI= 2024-02-01T10:27:39.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ljWgupx1GhmoY5QBmAXMjRFmWtI=', 'Content-Length': '0'} 2024-02-01T10:27:39.444 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003eb7cc142e903f0e-0065bb721b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:39.444 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:39.444 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:39.444 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:39.444 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:39.445 DEBUG:boto:Method: GET 2024-02-01T10:27:39.445 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:39.445 DEBUG:boto:Data: 2024-02-01T10:27:39.445 DEBUG:boto:Headers: {} 2024-02-01T10:27:39.445 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:39.445 DEBUG:boto:Port: 8000 2024-02-01T10:27:39.445 DEBUG:boto:Params: {} 2024-02-01T10:27:39.445 DEBUG:boto:Token: None 2024-02-01T10:27:39.445 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:39 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:39.445 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ljWgupx1GhmoY5QBmAXMjRFmWtI= 2024-02-01T10:27:39.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ljWgupx1GhmoY5QBmAXMjRFmWtI=', 'Content-Length': '0'} 2024-02-01T10:27:39.448 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fff79bba32474c1a-0065bb721b-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:39.492 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:39.492 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:39.493 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:39.493 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:39.493 DEBUG:boto:Method: GET 2024-02-01T10:27:39.493 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:39.493 DEBUG:boto:Data: 2024-02-01T10:27:39.493 DEBUG:boto:Headers: {} 2024-02-01T10:27:39.493 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:39.493 DEBUG:boto:Port: 8000 2024-02-01T10:27:39.493 DEBUG:boto:Params: {} 2024-02-01T10:27:39.493 DEBUG:boto:Token: None 2024-02-01T10:27:39.494 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:39 GMT /vclfcn-19/ 2024-02-01T10:27:39.494 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QTTv5SM7hBeP5CuqyFSOV1PKaxw= 2024-02-01T10:27:39.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QTTv5SM7hBeP5CuqyFSOV1PKaxw=', 'Content-Length': '0'} 2024-02-01T10:27:39.499 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095ab17f650ba75d1-0065bb721b-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:39.499 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:39.500 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:39.500 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:39.500 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:39.500 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:39.500 DEBUG:boto:Method: GET 2024-02-01T10:27:39.500 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:39.501 DEBUG:boto:Data: 2024-02-01T10:27:39.501 DEBUG:boto:Headers: {} 2024-02-01T10:27:39.501 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:39.502 DEBUG:boto:Port: 8000 2024-02-01T10:27:39.502 DEBUG:boto:Params: {} 2024-02-01T10:27:39.502 DEBUG:boto:Token: None 2024-02-01T10:27:39.502 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:39 GMT /vclfcn-19/ 2024-02-01T10:27:39.502 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QTTv5SM7hBeP5CuqyFSOV1PKaxw= 2024-02-01T10:27:39.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QTTv5SM7hBeP5CuqyFSOV1PKaxw=', 'Content-Length': '0'} 2024-02-01T10:27:39.507 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000942e1461e9c26ddd-0065bb721b-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:39.507 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:39.508 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:39.508 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:39.508 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:39.508 DEBUG:boto:path=/vclfcn-19/obj?versionId=y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE 2024-02-01T10:27:39.508 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE 2024-02-01T10:27:39.508 DEBUG:boto:Method: GET 2024-02-01T10:27:39.508 DEBUG:boto:Path: /vclfcn-19/obj?versionId=y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE 2024-02-01T10:27:39.508 DEBUG:boto:Data: 2024-02-01T10:27:39.509 DEBUG:boto:Headers: {} 2024-02-01T10:27:39.509 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:39.509 DEBUG:boto:Port: 8000 2024-02-01T10:27:39.509 DEBUG:boto:Params: {} 2024-02-01T10:27:39.509 DEBUG:boto:Token: None 2024-02-01T10:27:39.509 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:39 GMT /vclfcn-19/obj?versionId=y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE 2024-02-01T10:27:39.509 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:CTG7b/PngbTal+H515I0WMDXEqY= 2024-02-01T10:27:39.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:CTG7b/PngbTal+H515I0WMDXEqY=', 'Content-Length': '0'} 2024-02-01T10:27:39.512 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000071c849aa619947bd-0065bb721b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:39.560 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:39.560 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:39.560 DEBUG:boto:path=/vclfcn-19/obj?versionId=y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE 2024-02-01T10:27:39.561 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE 2024-02-01T10:27:39.561 DEBUG:boto:Method: GET 2024-02-01T10:27:39.561 DEBUG:boto:Path: /vclfcn-19/obj?versionId=y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE 2024-02-01T10:27:39.561 DEBUG:boto:Data: 2024-02-01T10:27:39.561 DEBUG:boto:Headers: {} 2024-02-01T10:27:39.561 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:39.561 DEBUG:boto:Port: 8000 2024-02-01T10:27:39.561 DEBUG:boto:Params: {} 2024-02-01T10:27:39.561 DEBUG:boto:Token: None 2024-02-01T10:27:39.562 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:39 GMT /vclfcn-19/obj?versionId=y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE 2024-02-01T10:27:39.562 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:CTG7b/PngbTal+H515I0WMDXEqY= 2024-02-01T10:27:39.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:CTG7b/PngbTal+H515I0WMDXEqY=', 'Content-Length': '0'} 2024-02-01T10:27:39.565 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000077c8c74f5e090e97-0065bb721b-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:39.565 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:39.565 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:39.565 DEBUG:boto:path=/vclfcn-19/obj?versionId=y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE 2024-02-01T10:27:39.565 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE 2024-02-01T10:27:39.565 DEBUG:boto:Method: HEAD 2024-02-01T10:27:39.566 DEBUG:boto:Path: /vclfcn-19/obj?versionId=y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE 2024-02-01T10:27:39.566 DEBUG:boto:Data: 2024-02-01T10:27:39.566 DEBUG:boto:Headers: {} 2024-02-01T10:27:39.566 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:39.566 DEBUG:boto:Port: 8000 2024-02-01T10:27:39.566 DEBUG:boto:Params: {} 2024-02-01T10:27:39.566 DEBUG:boto:Token: None 2024-02-01T10:27:39.566 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:39 GMT /vclfcn-19/obj?versionId=y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE 2024-02-01T10:27:39.566 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8s21vLVYUeek4nJuFhuZzOy+skM= 2024-02-01T10:27:39.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8s21vLVYUeek4nJuFhuZzOy+skM=', 'Content-Length': '0'} 2024-02-01T10:27:39.569 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000039c75cedb82f655c-0065bb721b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:39.569 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:39.569 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:39.569 DEBUG:boto:path=/vclfcn-19/obj?versionId=y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE 2024-02-01T10:27:39.570 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE 2024-02-01T10:27:39.570 DEBUG:boto:Method: HEAD 2024-02-01T10:27:39.570 DEBUG:boto:Path: /vclfcn-19/obj?versionId=y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE 2024-02-01T10:27:39.570 DEBUG:boto:Data: 2024-02-01T10:27:39.570 DEBUG:boto:Headers: {} 2024-02-01T10:27:39.570 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:39.570 DEBUG:boto:Port: 8000 2024-02-01T10:27:39.570 DEBUG:boto:Params: {} 2024-02-01T10:27:39.570 DEBUG:boto:Token: None 2024-02-01T10:27:39.570 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:39 GMT /vclfcn-19/obj?versionId=y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE 2024-02-01T10:27:39.570 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8s21vLVYUeek4nJuFhuZzOy+skM= 2024-02-01T10:27:39.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8s21vLVYUeek4nJuFhuZzOy+skM=', 'Content-Length': '0'} 2024-02-01T10:27:39.573 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003ad8faab0afeb616-0065bb721b-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:39.573 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:39.574 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:39.574 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:39.574 DEBUG:boto:path=/vclfcn-19/obj?versionId=y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE 2024-02-01T10:27:39.574 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE 2024-02-01T10:27:39.574 DEBUG:boto:Method: GET 2024-02-01T10:27:39.574 DEBUG:boto:Path: /vclfcn-19/obj?versionId=y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE 2024-02-01T10:27:39.574 DEBUG:boto:Data: 2024-02-01T10:27:39.574 DEBUG:boto:Headers: {} 2024-02-01T10:27:39.574 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:39.574 DEBUG:boto:Port: 8000 2024-02-01T10:27:39.575 DEBUG:boto:Params: {} 2024-02-01T10:27:39.575 DEBUG:boto:Token: None 2024-02-01T10:27:39.575 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:39 GMT /vclfcn-19/obj?versionId=y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE 2024-02-01T10:27:39.575 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:CTG7b/PngbTal+H515I0WMDXEqY= 2024-02-01T10:27:39.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:CTG7b/PngbTal+H515I0WMDXEqY=', 'Content-Length': '0'} 2024-02-01T10:27:39.578 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c4a4da25334b57e3-0065bb721b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:39.578 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:39.578 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:39.579 DEBUG:boto:path=/vclfcn-19/obj?versionId=y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE 2024-02-01T10:27:39.579 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE 2024-02-01T10:27:39.579 DEBUG:boto:Method: GET 2024-02-01T10:27:39.579 DEBUG:boto:Path: /vclfcn-19/obj?versionId=y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE 2024-02-01T10:27:39.579 DEBUG:boto:Data: 2024-02-01T10:27:39.579 DEBUG:boto:Headers: {} 2024-02-01T10:27:39.579 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:39.579 DEBUG:boto:Port: 8000 2024-02-01T10:27:39.579 DEBUG:boto:Params: {} 2024-02-01T10:27:39.579 DEBUG:boto:Token: None 2024-02-01T10:27:39.579 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:39 GMT /vclfcn-19/obj?versionId=y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE 2024-02-01T10:27:39.580 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:CTG7b/PngbTal+H515I0WMDXEqY= 2024-02-01T10:27:39.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:CTG7b/PngbTal+H515I0WMDXEqY=', 'Content-Length': '0'} 2024-02-01T10:27:39.583 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000088330713fb619f07-0065bb721b-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:39.624 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:39.624 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:39.624 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:39.624 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:39.625 DEBUG:boto:Method: GET 2024-02-01T10:27:39.625 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:39.625 DEBUG:boto:Data: 2024-02-01T10:27:39.625 DEBUG:boto:Headers: {} 2024-02-01T10:27:39.625 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:39.625 DEBUG:boto:Port: 8000 2024-02-01T10:27:39.625 DEBUG:boto:Params: {} 2024-02-01T10:27:39.626 DEBUG:boto:Token: None 2024-02-01T10:27:39.626 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:39 GMT /vclfcn-19/ 2024-02-01T10:27:39.626 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QTTv5SM7hBeP5CuqyFSOV1PKaxw= 2024-02-01T10:27:39.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QTTv5SM7hBeP5CuqyFSOV1PKaxw=', 'Content-Length': '0'} 2024-02-01T10:27:39.632 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035fe9d40e6127278-0065bb721b-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:39.632 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:39.633 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:39.633 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:39.633 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:39.633 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:39.633 DEBUG:boto:Method: GET 2024-02-01T10:27:39.633 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:39.633 DEBUG:boto:Data: 2024-02-01T10:27:39.633 DEBUG:boto:Headers: {} 2024-02-01T10:27:39.633 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:39.634 DEBUG:boto:Port: 8000 2024-02-01T10:27:39.634 DEBUG:boto:Params: {} 2024-02-01T10:27:39.634 DEBUG:boto:Token: None 2024-02-01T10:27:39.634 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:39 GMT /vclfcn-19/ 2024-02-01T10:27:39.634 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QTTv5SM7hBeP5CuqyFSOV1PKaxw= 2024-02-01T10:27:39.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QTTv5SM7hBeP5CuqyFSOV1PKaxw=', 'Content-Length': '0'} 2024-02-01T10:27:39.639 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000230c7e3e863720d0-0065bb721b-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:39.639 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:39.640 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:39.640 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:39.640 DEBUG:boto:Method: HEAD 2024-02-01T10:27:39.640 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:39.640 DEBUG:boto:Data: 2024-02-01T10:27:39.640 DEBUG:boto:Headers: {} 2024-02-01T10:27:39.640 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:39.640 DEBUG:boto:Port: 8000 2024-02-01T10:27:39.641 DEBUG:boto:Params: {} 2024-02-01T10:27:39.641 DEBUG:boto:Token: None 2024-02-01T10:27:39.641 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:39 GMT /vclfcn-19/obj 2024-02-01T10:27:39.641 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fjU6khZbGrZBVDx5Bwe9ScugZFs= 2024-02-01T10:27:39.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fjU6khZbGrZBVDx5Bwe9ScugZFs=', 'Content-Length': '0'} 2024-02-01T10:27:39.647 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009e21563bc4c60f16-0065bb721b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:39.647 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:39.647 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:39.647 DEBUG:boto:Method: HEAD 2024-02-01T10:27:39.647 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:39.648 DEBUG:boto:Data: 2024-02-01T10:27:39.648 DEBUG:boto:Headers: {} 2024-02-01T10:27:39.648 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:39.648 DEBUG:boto:Port: 8000 2024-02-01T10:27:39.648 DEBUG:boto:Params: {} 2024-02-01T10:27:39.648 DEBUG:boto:Token: None 2024-02-01T10:27:39.648 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:39 GMT /vclfcn-19/obj 2024-02-01T10:27:39.648 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fjU6khZbGrZBVDx5Bwe9ScugZFs= 2024-02-01T10:27:39.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fjU6khZbGrZBVDx5Bwe9ScugZFs=', 'Content-Length': '0'} 2024-02-01T10:27:39.654 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003876de11a600246b-0065bb721b-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:39.655 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:39.655 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:39.655 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:39.655 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:39.655 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:39.655 DEBUG:boto:Method: GET 2024-02-01T10:27:39.655 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:39.655 DEBUG:boto:Data: 2024-02-01T10:27:39.656 DEBUG:boto:Headers: {} 2024-02-01T10:27:39.656 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:39.656 DEBUG:boto:Port: 8000 2024-02-01T10:27:39.656 DEBUG:boto:Params: {} 2024-02-01T10:27:39.656 DEBUG:boto:Token: None 2024-02-01T10:27:39.656 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:39 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:39.656 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ljWgupx1GhmoY5QBmAXMjRFmWtI= 2024-02-01T10:27:39.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ljWgupx1GhmoY5QBmAXMjRFmWtI=', 'Content-Length': '0'} 2024-02-01T10:27:39.659 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c2b5b6adc504eedc-0065bb721b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:39.700 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:39.700 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:39.700 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:39.701 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:39.701 DEBUG:boto:Method: GET 2024-02-01T10:27:39.701 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:39.701 DEBUG:boto:Data: 2024-02-01T10:27:39.701 DEBUG:boto:Headers: {} 2024-02-01T10:27:39.701 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:39.701 DEBUG:boto:Port: 8000 2024-02-01T10:27:39.701 DEBUG:boto:Params: {} 2024-02-01T10:27:39.701 DEBUG:boto:Token: None 2024-02-01T10:27:39.701 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:39 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:39.702 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ljWgupx1GhmoY5QBmAXMjRFmWtI= 2024-02-01T10:27:39.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ljWgupx1GhmoY5QBmAXMjRFmWtI=', 'Content-Length': '0'} 2024-02-01T10:27:39.705 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006854a719f9fc4ec6-0065bb721b-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:39.705 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:39.705 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:39.705 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:39.705 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:39.706 DEBUG:boto:Method: GET 2024-02-01T10:27:39.706 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:39.706 DEBUG:boto:Data: 2024-02-01T10:27:39.706 DEBUG:boto:Headers: {} 2024-02-01T10:27:39.706 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:39.706 DEBUG:boto:Port: 8000 2024-02-01T10:27:39.706 DEBUG:boto:Params: {} 2024-02-01T10:27:39.706 DEBUG:boto:Token: None 2024-02-01T10:27:39.706 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:39 GMT /vclfcn-19/ 2024-02-01T10:27:39.706 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QTTv5SM7hBeP5CuqyFSOV1PKaxw= 2024-02-01T10:27:39.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QTTv5SM7hBeP5CuqyFSOV1PKaxw=', 'Content-Length': '0'} 2024-02-01T10:27:39.712 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b9a4cca47e5e20a9-0065bb721b-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:39.713 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:39.713 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:39.713 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:39.713 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:39.713 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:39.713 DEBUG:boto:Method: GET 2024-02-01T10:27:39.714 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:39.714 DEBUG:boto:Data: 2024-02-01T10:27:39.714 DEBUG:boto:Headers: {} 2024-02-01T10:27:39.714 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:39.714 DEBUG:boto:Port: 8000 2024-02-01T10:27:39.714 DEBUG:boto:Params: {} 2024-02-01T10:27:39.714 DEBUG:boto:Token: None 2024-02-01T10:27:39.714 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:39 GMT /vclfcn-19/ 2024-02-01T10:27:39.714 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QTTv5SM7hBeP5CuqyFSOV1PKaxw= 2024-02-01T10:27:39.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QTTv5SM7hBeP5CuqyFSOV1PKaxw=', 'Content-Length': '0'} 2024-02-01T10:27:39.720 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000898d6da0a1fd2a87-0065bb721b-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:39.768 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:39.769 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:39.769 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:39.769 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:39.769 DEBUG:boto:path=/vclfcn-19/obj?versionId=0KR6m6ymTGitKe6vabQyVENAnBL669w 2024-02-01T10:27:39.769 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=0KR6m6ymTGitKe6vabQyVENAnBL669w 2024-02-01T10:27:39.769 DEBUG:boto:Method: GET 2024-02-01T10:27:39.769 DEBUG:boto:Path: /vclfcn-19/obj?versionId=0KR6m6ymTGitKe6vabQyVENAnBL669w 2024-02-01T10:27:39.769 DEBUG:boto:Data: 2024-02-01T10:27:39.769 DEBUG:boto:Headers: {} 2024-02-01T10:27:39.770 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:39.770 DEBUG:boto:Port: 8000 2024-02-01T10:27:39.770 DEBUG:boto:Params: {} 2024-02-01T10:27:39.770 DEBUG:boto:Token: None 2024-02-01T10:27:39.770 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:39 GMT /vclfcn-19/obj?versionId=0KR6m6ymTGitKe6vabQyVENAnBL669w 2024-02-01T10:27:39.770 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:VtslKb6XBxPp7ExNYt6kFiyeZ9Y= 2024-02-01T10:27:39.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:VtslKb6XBxPp7ExNYt6kFiyeZ9Y=', 'Content-Length': '0'} 2024-02-01T10:27:39.773 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '0KR6m6ymTGitKe6vabQyVENAnBL669w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d82f7f774bf86d48-0065bb721b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:39.773 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:39.774 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:39.774 DEBUG:boto:path=/vclfcn-19/obj?versionId=0KR6m6ymTGitKe6vabQyVENAnBL669w 2024-02-01T10:27:39.774 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=0KR6m6ymTGitKe6vabQyVENAnBL669w 2024-02-01T10:27:39.774 DEBUG:boto:Method: GET 2024-02-01T10:27:39.774 DEBUG:boto:Path: /vclfcn-19/obj?versionId=0KR6m6ymTGitKe6vabQyVENAnBL669w 2024-02-01T10:27:39.775 DEBUG:boto:Data: 2024-02-01T10:27:39.775 DEBUG:boto:Headers: {} 2024-02-01T10:27:39.775 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:39.775 DEBUG:boto:Port: 8000 2024-02-01T10:27:39.775 DEBUG:boto:Params: {} 2024-02-01T10:27:39.775 DEBUG:boto:Token: None 2024-02-01T10:27:39.775 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:39 GMT /vclfcn-19/obj?versionId=0KR6m6ymTGitKe6vabQyVENAnBL669w 2024-02-01T10:27:39.775 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:VtslKb6XBxPp7ExNYt6kFiyeZ9Y= 2024-02-01T10:27:39.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:VtslKb6XBxPp7ExNYt6kFiyeZ9Y=', 'Content-Length': '0'} 2024-02-01T10:27:39.778 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '0KR6m6ymTGitKe6vabQyVENAnBL669w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007577684a8ee405c4-0065bb721b-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:39.779 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:39.779 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:39.779 DEBUG:boto:path=/vclfcn-19/obj?versionId=0KR6m6ymTGitKe6vabQyVENAnBL669w 2024-02-01T10:27:39.779 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=0KR6m6ymTGitKe6vabQyVENAnBL669w 2024-02-01T10:27:39.779 DEBUG:boto:Method: HEAD 2024-02-01T10:27:39.779 DEBUG:boto:Path: /vclfcn-19/obj?versionId=0KR6m6ymTGitKe6vabQyVENAnBL669w 2024-02-01T10:27:39.779 DEBUG:boto:Data: 2024-02-01T10:27:39.779 DEBUG:boto:Headers: {} 2024-02-01T10:27:39.779 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:39.780 DEBUG:boto:Port: 8000 2024-02-01T10:27:39.780 DEBUG:boto:Params: {} 2024-02-01T10:27:39.781 DEBUG:boto:Token: None 2024-02-01T10:27:39.781 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:39 GMT /vclfcn-19/obj?versionId=0KR6m6ymTGitKe6vabQyVENAnBL669w 2024-02-01T10:27:39.781 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QX4NEr7UKG80n07sUxj+6dpZ26o= 2024-02-01T10:27:39.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QX4NEr7UKG80n07sUxj+6dpZ26o=', 'Content-Length': '0'} 2024-02-01T10:27:39.784 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '0KR6m6ymTGitKe6vabQyVENAnBL669w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005e6c88b4d5558ac5-0065bb721b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:39.784 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:39.784 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:39.785 DEBUG:boto:path=/vclfcn-19/obj?versionId=0KR6m6ymTGitKe6vabQyVENAnBL669w 2024-02-01T10:27:39.785 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=0KR6m6ymTGitKe6vabQyVENAnBL669w 2024-02-01T10:27:39.785 DEBUG:boto:Method: HEAD 2024-02-01T10:27:39.785 DEBUG:boto:Path: /vclfcn-19/obj?versionId=0KR6m6ymTGitKe6vabQyVENAnBL669w 2024-02-01T10:27:39.785 DEBUG:boto:Data: 2024-02-01T10:27:39.785 DEBUG:boto:Headers: {} 2024-02-01T10:27:39.785 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:39.785 DEBUG:boto:Port: 8000 2024-02-01T10:27:39.785 DEBUG:boto:Params: {} 2024-02-01T10:27:39.785 DEBUG:boto:Token: None 2024-02-01T10:27:39.786 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:39 GMT /vclfcn-19/obj?versionId=0KR6m6ymTGitKe6vabQyVENAnBL669w 2024-02-01T10:27:39.786 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QX4NEr7UKG80n07sUxj+6dpZ26o= 2024-02-01T10:27:39.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QX4NEr7UKG80n07sUxj+6dpZ26o=', 'Content-Length': '0'} 2024-02-01T10:27:39.788 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '0KR6m6ymTGitKe6vabQyVENAnBL669w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000024001583f3096c61-0065bb721b-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:39.789 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:39.789 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:39.789 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:39.789 DEBUG:boto:path=/vclfcn-19/obj?versionId=0KR6m6ymTGitKe6vabQyVENAnBL669w 2024-02-01T10:27:39.789 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=0KR6m6ymTGitKe6vabQyVENAnBL669w 2024-02-01T10:27:39.789 DEBUG:boto:Method: GET 2024-02-01T10:27:39.789 DEBUG:boto:Path: /vclfcn-19/obj?versionId=0KR6m6ymTGitKe6vabQyVENAnBL669w 2024-02-01T10:27:39.789 DEBUG:boto:Data: 2024-02-01T10:27:39.789 DEBUG:boto:Headers: {} 2024-02-01T10:27:39.790 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:39.790 DEBUG:boto:Port: 8000 2024-02-01T10:27:39.790 DEBUG:boto:Params: {} 2024-02-01T10:27:39.790 DEBUG:boto:Token: None 2024-02-01T10:27:39.790 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:39 GMT /vclfcn-19/obj?versionId=0KR6m6ymTGitKe6vabQyVENAnBL669w 2024-02-01T10:27:39.790 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:VtslKb6XBxPp7ExNYt6kFiyeZ9Y= 2024-02-01T10:27:39.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:VtslKb6XBxPp7ExNYt6kFiyeZ9Y=', 'Content-Length': '0'} 2024-02-01T10:27:39.793 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '0KR6m6ymTGitKe6vabQyVENAnBL669w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b432a01ffe956f46-0065bb721b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:39.840 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:39.840 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:39.840 DEBUG:boto:path=/vclfcn-19/obj?versionId=0KR6m6ymTGitKe6vabQyVENAnBL669w 2024-02-01T10:27:39.840 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=0KR6m6ymTGitKe6vabQyVENAnBL669w 2024-02-01T10:27:39.841 DEBUG:boto:Method: GET 2024-02-01T10:27:39.841 DEBUG:boto:Path: /vclfcn-19/obj?versionId=0KR6m6ymTGitKe6vabQyVENAnBL669w 2024-02-01T10:27:39.841 DEBUG:boto:Data: 2024-02-01T10:27:39.841 DEBUG:boto:Headers: {} 2024-02-01T10:27:39.841 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:39.841 DEBUG:boto:Port: 8000 2024-02-01T10:27:39.841 DEBUG:boto:Params: {} 2024-02-01T10:27:39.841 DEBUG:boto:Token: None 2024-02-01T10:27:39.841 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:39 GMT /vclfcn-19/obj?versionId=0KR6m6ymTGitKe6vabQyVENAnBL669w 2024-02-01T10:27:39.842 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:VtslKb6XBxPp7ExNYt6kFiyeZ9Y= 2024-02-01T10:27:39.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:VtslKb6XBxPp7ExNYt6kFiyeZ9Y=', 'Content-Length': '0'} 2024-02-01T10:27:39.845 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '0KR6m6ymTGitKe6vabQyVENAnBL669w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c02c038e928e484c-0065bb721b-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:39.845 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:39.846 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:39.846 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:39.846 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:39.846 DEBUG:boto:Method: GET 2024-02-01T10:27:39.846 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:39.846 DEBUG:boto:Data: 2024-02-01T10:27:39.846 DEBUG:boto:Headers: {} 2024-02-01T10:27:39.846 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:39.846 DEBUG:boto:Port: 8000 2024-02-01T10:27:39.846 DEBUG:boto:Params: {} 2024-02-01T10:27:39.847 DEBUG:boto:Token: None 2024-02-01T10:27:39.847 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:39 GMT /vclfcn-19/ 2024-02-01T10:27:39.847 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QTTv5SM7hBeP5CuqyFSOV1PKaxw= 2024-02-01T10:27:39.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QTTv5SM7hBeP5CuqyFSOV1PKaxw=', 'Content-Length': '0'} 2024-02-01T10:27:39.853 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000053e3e85f5fcb66c0-0065bb721b-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:39.853 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:39.853 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:39.854 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:39.854 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:39.854 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:39.854 DEBUG:boto:Method: GET 2024-02-01T10:27:39.854 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:39.854 DEBUG:boto:Data: 2024-02-01T10:27:39.854 DEBUG:boto:Headers: {} 2024-02-01T10:27:39.854 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:39.854 DEBUG:boto:Port: 8000 2024-02-01T10:27:39.855 DEBUG:boto:Params: {} 2024-02-01T10:27:39.855 DEBUG:boto:Token: None 2024-02-01T10:27:39.855 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:39 GMT /vclfcn-19/ 2024-02-01T10:27:39.855 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QTTv5SM7hBeP5CuqyFSOV1PKaxw= 2024-02-01T10:27:39.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QTTv5SM7hBeP5CuqyFSOV1PKaxw=', 'Content-Length': '0'} 2024-02-01T10:27:39.860 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000992363bfe0487e1f-0065bb721b-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:39.861 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:39.861 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:39.861 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:39.861 DEBUG:boto:Method: HEAD 2024-02-01T10:27:39.861 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:39.861 DEBUG:boto:Data: 2024-02-01T10:27:39.861 DEBUG:boto:Headers: {} 2024-02-01T10:27:39.862 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:39.862 DEBUG:boto:Port: 8000 2024-02-01T10:27:39.862 DEBUG:boto:Params: {} 2024-02-01T10:27:39.862 DEBUG:boto:Token: None 2024-02-01T10:27:39.862 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:39 GMT /vclfcn-19/obj 2024-02-01T10:27:39.862 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fjU6khZbGrZBVDx5Bwe9ScugZFs= 2024-02-01T10:27:39.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fjU6khZbGrZBVDx5Bwe9ScugZFs=', 'Content-Length': '0'} 2024-02-01T10:27:39.868 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008e9ea79a69a2dffe-0065bb721b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:39.868 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:39.868 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:39.868 DEBUG:boto:Method: HEAD 2024-02-01T10:27:39.868 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:39.868 DEBUG:boto:Data: 2024-02-01T10:27:39.869 DEBUG:boto:Headers: {} 2024-02-01T10:27:39.869 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:39.869 DEBUG:boto:Port: 8000 2024-02-01T10:27:39.869 DEBUG:boto:Params: {} 2024-02-01T10:27:39.869 DEBUG:boto:Token: None 2024-02-01T10:27:39.869 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:39 GMT /vclfcn-19/obj 2024-02-01T10:27:39.869 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fjU6khZbGrZBVDx5Bwe9ScugZFs= 2024-02-01T10:27:39.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fjU6khZbGrZBVDx5Bwe9ScugZFs=', 'Content-Length': '0'} 2024-02-01T10:27:39.875 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008be2fb54963b3ef4-0065bb721b-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:39.875 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:39.876 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:39.876 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:39.876 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:39.876 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:39.876 DEBUG:boto:Method: GET 2024-02-01T10:27:39.876 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:39.876 DEBUG:boto:Data: 2024-02-01T10:27:39.876 DEBUG:boto:Headers: {} 2024-02-01T10:27:39.876 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:39.876 DEBUG:boto:Port: 8000 2024-02-01T10:27:39.876 DEBUG:boto:Params: {} 2024-02-01T10:27:39.877 DEBUG:boto:Token: None 2024-02-01T10:27:39.877 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:39 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:39.877 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ljWgupx1GhmoY5QBmAXMjRFmWtI= 2024-02-01T10:27:39.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ljWgupx1GhmoY5QBmAXMjRFmWtI=', 'Content-Length': '0'} 2024-02-01T10:27:39.880 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c51d6ed88a3e2ec2-0065bb721b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:39.928 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:39.928 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:39.928 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:39.929 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:39.929 DEBUG:boto:Method: GET 2024-02-01T10:27:39.929 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:39.929 DEBUG:boto:Data: 2024-02-01T10:27:39.929 DEBUG:boto:Headers: {} 2024-02-01T10:27:39.929 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:39.929 DEBUG:boto:Port: 8000 2024-02-01T10:27:39.929 DEBUG:boto:Params: {} 2024-02-01T10:27:39.930 DEBUG:boto:Token: None 2024-02-01T10:27:39.930 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:39 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:39.930 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ljWgupx1GhmoY5QBmAXMjRFmWtI= 2024-02-01T10:27:39.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ljWgupx1GhmoY5QBmAXMjRFmWtI=', 'Content-Length': '0'} 2024-02-01T10:27:39.933 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bab83076ba0db41d-0065bb721b-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:39.933 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:39.933 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:39.934 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:39.934 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:39.934 DEBUG:boto:Method: GET 2024-02-01T10:27:39.934 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:39.934 DEBUG:boto:Data: 2024-02-01T10:27:39.934 DEBUG:boto:Headers: {} 2024-02-01T10:27:39.934 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:39.934 DEBUG:boto:Port: 8000 2024-02-01T10:27:39.934 DEBUG:boto:Params: {} 2024-02-01T10:27:39.935 DEBUG:boto:Token: None 2024-02-01T10:27:39.935 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:39 GMT /vclfcn-19/ 2024-02-01T10:27:39.935 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QTTv5SM7hBeP5CuqyFSOV1PKaxw= 2024-02-01T10:27:39.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QTTv5SM7hBeP5CuqyFSOV1PKaxw=', 'Content-Length': '0'} 2024-02-01T10:27:39.941 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e5621b7182768c8-0065bb721b-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:39.941 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:39.942 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:39.942 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:39.942 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:39.942 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:39.942 DEBUG:boto:Method: GET 2024-02-01T10:27:39.942 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:39.942 DEBUG:boto:Data: 2024-02-01T10:27:39.942 DEBUG:boto:Headers: {} 2024-02-01T10:27:39.942 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:39.942 DEBUG:boto:Port: 8000 2024-02-01T10:27:39.943 DEBUG:boto:Params: {} 2024-02-01T10:27:39.943 DEBUG:boto:Token: None 2024-02-01T10:27:39.943 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:39 GMT /vclfcn-19/ 2024-02-01T10:27:39.943 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QTTv5SM7hBeP5CuqyFSOV1PKaxw= 2024-02-01T10:27:39.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QTTv5SM7hBeP5CuqyFSOV1PKaxw=', 'Content-Length': '0'} 2024-02-01T10:27:39.948 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058d42e614112ae65-0065bb721b-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:39.948 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:39.949 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:39.949 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:39.949 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:39.949 DEBUG:boto:path=/vclfcn-19/obj?versionId=2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E 2024-02-01T10:27:39.949 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E 2024-02-01T10:27:39.949 DEBUG:boto:Method: GET 2024-02-01T10:27:39.949 DEBUG:boto:Path: /vclfcn-19/obj?versionId=2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E 2024-02-01T10:27:39.949 DEBUG:boto:Data: 2024-02-01T10:27:39.949 DEBUG:boto:Headers: {} 2024-02-01T10:27:39.949 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:39.949 DEBUG:boto:Port: 8000 2024-02-01T10:27:39.950 DEBUG:boto:Params: {} 2024-02-01T10:27:39.950 DEBUG:boto:Token: None 2024-02-01T10:27:39.950 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:39 GMT /vclfcn-19/obj?versionId=2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E 2024-02-01T10:27:39.950 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7YI9tXB+pG4K9w6+Bv8K2o0932Q= 2024-02-01T10:27:39.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:39 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7YI9tXB+pG4K9w6+Bv8K2o0932Q=', 'Content-Length': '0'} 2024-02-01T10:27:39.953 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b8b66b8ebdc80efe-0065bb721b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:40.000 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:40.000 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:40.000 DEBUG:boto:path=/vclfcn-19/obj?versionId=2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E 2024-02-01T10:27:40.001 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E 2024-02-01T10:27:40.001 DEBUG:boto:Method: GET 2024-02-01T10:27:40.001 DEBUG:boto:Path: /vclfcn-19/obj?versionId=2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E 2024-02-01T10:27:40.001 DEBUG:boto:Data: 2024-02-01T10:27:40.001 DEBUG:boto:Headers: {} 2024-02-01T10:27:40.001 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:40.001 DEBUG:boto:Port: 8000 2024-02-01T10:27:40.001 DEBUG:boto:Params: {} 2024-02-01T10:27:40.001 DEBUG:boto:Token: None 2024-02-01T10:27:40.002 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:40 GMT /vclfcn-19/obj?versionId=2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E 2024-02-01T10:27:40.002 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:r0BfhDhn2jmS1vj0HLvBxNOvcb0= 2024-02-01T10:27:40.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:40 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:r0BfhDhn2jmS1vj0HLvBxNOvcb0=', 'Content-Length': '0'} 2024-02-01T10:27:40.005 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000045bcdff84f35f3d0-0065bb721c-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:40.005 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:40.005 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:40.005 DEBUG:boto:path=/vclfcn-19/obj?versionId=2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E 2024-02-01T10:27:40.005 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E 2024-02-01T10:27:40.005 DEBUG:boto:Method: HEAD 2024-02-01T10:27:40.006 DEBUG:boto:Path: /vclfcn-19/obj?versionId=2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E 2024-02-01T10:27:40.006 DEBUG:boto:Data: 2024-02-01T10:27:40.006 DEBUG:boto:Headers: {} 2024-02-01T10:27:40.006 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:40.006 DEBUG:boto:Port: 8000 2024-02-01T10:27:40.006 DEBUG:boto:Params: {} 2024-02-01T10:27:40.006 DEBUG:boto:Token: None 2024-02-01T10:27:40.006 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:40 GMT /vclfcn-19/obj?versionId=2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E 2024-02-01T10:27:40.006 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:aNMBTN9P30JRVu27cnIt2OK+M+Y= 2024-02-01T10:27:40.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:40 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:aNMBTN9P30JRVu27cnIt2OK+M+Y=', 'Content-Length': '0'} 2024-02-01T10:27:40.011 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000061a04790bcf54da6-0065bb721c-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:40.012 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:40.012 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:40.012 DEBUG:boto:path=/vclfcn-19/obj?versionId=2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E 2024-02-01T10:27:40.012 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E 2024-02-01T10:27:40.012 DEBUG:boto:Method: HEAD 2024-02-01T10:27:40.012 DEBUG:boto:Path: /vclfcn-19/obj?versionId=2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E 2024-02-01T10:27:40.012 DEBUG:boto:Data: 2024-02-01T10:27:40.012 DEBUG:boto:Headers: {} 2024-02-01T10:27:40.012 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:40.012 DEBUG:boto:Port: 8000 2024-02-01T10:27:40.013 DEBUG:boto:Params: {} 2024-02-01T10:27:40.013 DEBUG:boto:Token: None 2024-02-01T10:27:40.013 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:40 GMT /vclfcn-19/obj?versionId=2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E 2024-02-01T10:27:40.013 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:aNMBTN9P30JRVu27cnIt2OK+M+Y= 2024-02-01T10:27:40.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:40 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:aNMBTN9P30JRVu27cnIt2OK+M+Y=', 'Content-Length': '0'} 2024-02-01T10:27:40.016 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000068d20e5cca132fff-0065bb721c-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:40.016 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:40.016 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:40.016 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:40.016 DEBUG:boto:path=/vclfcn-19/obj?versionId=2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E 2024-02-01T10:27:40.016 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E 2024-02-01T10:27:40.016 DEBUG:boto:Method: GET 2024-02-01T10:27:40.017 DEBUG:boto:Path: /vclfcn-19/obj?versionId=2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E 2024-02-01T10:27:40.017 DEBUG:boto:Data: 2024-02-01T10:27:40.017 DEBUG:boto:Headers: {} 2024-02-01T10:27:40.017 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:40.017 DEBUG:boto:Port: 8000 2024-02-01T10:27:40.017 DEBUG:boto:Params: {} 2024-02-01T10:27:40.017 DEBUG:boto:Token: None 2024-02-01T10:27:40.017 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:40 GMT /vclfcn-19/obj?versionId=2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E 2024-02-01T10:27:40.017 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:r0BfhDhn2jmS1vj0HLvBxNOvcb0= 2024-02-01T10:27:40.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:40 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:r0BfhDhn2jmS1vj0HLvBxNOvcb0=', 'Content-Length': '0'} 2024-02-01T10:27:40.020 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f001b5efc09c2fb2-0065bb721c-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:40.064 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:40.064 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:40.065 DEBUG:boto:path=/vclfcn-19/obj?versionId=2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E 2024-02-01T10:27:40.065 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E 2024-02-01T10:27:40.065 DEBUG:boto:Method: GET 2024-02-01T10:27:40.065 DEBUG:boto:Path: /vclfcn-19/obj?versionId=2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E 2024-02-01T10:27:40.065 DEBUG:boto:Data: 2024-02-01T10:27:40.065 DEBUG:boto:Headers: {} 2024-02-01T10:27:40.065 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:40.065 DEBUG:boto:Port: 8000 2024-02-01T10:27:40.065 DEBUG:boto:Params: {} 2024-02-01T10:27:40.065 DEBUG:boto:Token: None 2024-02-01T10:27:40.066 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:40 GMT /vclfcn-19/obj?versionId=2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E 2024-02-01T10:27:40.066 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:r0BfhDhn2jmS1vj0HLvBxNOvcb0= 2024-02-01T10:27:40.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:40 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:r0BfhDhn2jmS1vj0HLvBxNOvcb0=', 'Content-Length': '0'} 2024-02-01T10:27:40.069 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bb515bb947773d73-0065bb721c-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:40.069 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:40.069 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:40.070 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:40.070 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:40.070 DEBUG:boto:Method: GET 2024-02-01T10:27:40.070 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:40.070 DEBUG:boto:Data: 2024-02-01T10:27:40.070 DEBUG:boto:Headers: {} 2024-02-01T10:27:40.070 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:40.070 DEBUG:boto:Port: 8000 2024-02-01T10:27:40.070 DEBUG:boto:Params: {} 2024-02-01T10:27:40.070 DEBUG:boto:Token: None 2024-02-01T10:27:40.070 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:40 GMT /vclfcn-19/ 2024-02-01T10:27:40.071 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:G1FlU0TeaZr50m1WkRUMinXvbA0= 2024-02-01T10:27:40.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:40 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:G1FlU0TeaZr50m1WkRUMinXvbA0=', 'Content-Length': '0'} 2024-02-01T10:27:40.076 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a86774611e7800f7-0065bb721c-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:40.076 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:40.077 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:40.077 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:40.077 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:40.077 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:40.077 DEBUG:boto:Method: GET 2024-02-01T10:27:40.077 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:40.077 DEBUG:boto:Data: 2024-02-01T10:27:40.077 DEBUG:boto:Headers: {} 2024-02-01T10:27:40.077 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:40.078 DEBUG:boto:Port: 8000 2024-02-01T10:27:40.078 DEBUG:boto:Params: {} 2024-02-01T10:27:40.078 DEBUG:boto:Token: None 2024-02-01T10:27:40.078 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:40 GMT /vclfcn-19/ 2024-02-01T10:27:40.078 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:G1FlU0TeaZr50m1WkRUMinXvbA0= 2024-02-01T10:27:40.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:40 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:G1FlU0TeaZr50m1WkRUMinXvbA0=', 'Content-Length': '0'} 2024-02-01T10:27:40.083 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c155f4e2df2c07c5-0065bb721c-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:40.124 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:40.125 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:40.125 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:40.125 DEBUG:boto:Method: HEAD 2024-02-01T10:27:40.125 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:40.125 DEBUG:boto:Data: 2024-02-01T10:27:40.125 DEBUG:boto:Headers: {} 2024-02-01T10:27:40.125 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:40.125 DEBUG:boto:Port: 8000 2024-02-01T10:27:40.125 DEBUG:boto:Params: {} 2024-02-01T10:27:40.126 DEBUG:boto:Token: None 2024-02-01T10:27:40.126 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:40 GMT /vclfcn-19/obj 2024-02-01T10:27:40.126 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fOrRaZ1X5+BTw9MNi5MBftd8KqU= 2024-02-01T10:27:40.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:40 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fOrRaZ1X5+BTw9MNi5MBftd8KqU=', 'Content-Length': '0'} 2024-02-01T10:27:40.132 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000064361567249793ce-0065bb721c-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:40.132 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:40.133 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:40.133 DEBUG:boto:Method: HEAD 2024-02-01T10:27:40.133 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:40.133 DEBUG:boto:Data: 2024-02-01T10:27:40.133 DEBUG:boto:Headers: {} 2024-02-01T10:27:40.133 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:40.133 DEBUG:boto:Port: 8000 2024-02-01T10:27:40.133 DEBUG:boto:Params: {} 2024-02-01T10:27:40.133 DEBUG:boto:Token: None 2024-02-01T10:27:40.133 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:40 GMT /vclfcn-19/obj 2024-02-01T10:27:40.133 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fOrRaZ1X5+BTw9MNi5MBftd8KqU= 2024-02-01T10:27:40.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:40 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fOrRaZ1X5+BTw9MNi5MBftd8KqU=', 'Content-Length': '0'} 2024-02-01T10:27:40.140 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000098a1a8cd581e5c01-0065bb721c-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:40.140 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:40.140 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:40.141 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:40.141 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:40.141 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:40.141 DEBUG:boto:Method: GET 2024-02-01T10:27:40.141 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:40.141 DEBUG:boto:Data: 2024-02-01T10:27:40.141 DEBUG:boto:Headers: {} 2024-02-01T10:27:40.141 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:40.141 DEBUG:boto:Port: 8000 2024-02-01T10:27:40.141 DEBUG:boto:Params: {} 2024-02-01T10:27:40.141 DEBUG:boto:Token: None 2024-02-01T10:27:40.142 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:40 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:40.142 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Bt/Gs/WYNv92el1KgJjH+bNm6Us= 2024-02-01T10:27:40.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:40 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Bt/Gs/WYNv92el1KgJjH+bNm6Us=', 'Content-Length': '0'} 2024-02-01T10:27:40.145 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bea146f3a1f02f42-0065bb721c-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:40.145 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:40.145 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:40.145 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:40.145 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:40.146 DEBUG:boto:Method: GET 2024-02-01T10:27:40.146 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:40.146 DEBUG:boto:Data: 2024-02-01T10:27:40.146 DEBUG:boto:Headers: {} 2024-02-01T10:27:40.146 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:40.146 DEBUG:boto:Port: 8000 2024-02-01T10:27:40.146 DEBUG:boto:Params: {} 2024-02-01T10:27:40.146 DEBUG:boto:Token: None 2024-02-01T10:27:40.146 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:40 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:40.146 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Bt/Gs/WYNv92el1KgJjH+bNm6Us= 2024-02-01T10:27:40.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:40 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Bt/Gs/WYNv92el1KgJjH+bNm6Us=', 'Content-Length': '0'} 2024-02-01T10:27:40.150 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000e4b520d2cc2af11-0065bb721c-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:40.150 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:40.150 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:40.150 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:40.150 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:40.150 DEBUG:boto:Method: GET 2024-02-01T10:27:40.151 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:40.151 DEBUG:boto:Data: 2024-02-01T10:27:40.151 DEBUG:boto:Headers: {} 2024-02-01T10:27:40.151 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:40.151 DEBUG:boto:Port: 8000 2024-02-01T10:27:40.151 DEBUG:boto:Params: {} 2024-02-01T10:27:40.151 DEBUG:boto:Token: None 2024-02-01T10:27:40.151 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:40 GMT /vclfcn-19/ 2024-02-01T10:27:40.151 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:G1FlU0TeaZr50m1WkRUMinXvbA0= 2024-02-01T10:27:40.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:40 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:G1FlU0TeaZr50m1WkRUMinXvbA0=', 'Content-Length': '0'} 2024-02-01T10:27:40.156 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082d9b2b9b964e7e1-0065bb721c-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:40.204 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:40.205 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:40.205 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:40.205 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:40.205 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:40.205 DEBUG:boto:Method: GET 2024-02-01T10:27:40.205 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:40.205 DEBUG:boto:Data: 2024-02-01T10:27:40.205 DEBUG:boto:Headers: {} 2024-02-01T10:27:40.205 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:40.206 DEBUG:boto:Port: 8000 2024-02-01T10:27:40.206 DEBUG:boto:Params: {} 2024-02-01T10:27:40.206 DEBUG:boto:Token: None 2024-02-01T10:27:40.206 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:40 GMT /vclfcn-19/ 2024-02-01T10:27:40.206 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:G1FlU0TeaZr50m1WkRUMinXvbA0= 2024-02-01T10:27:40.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:40 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:G1FlU0TeaZr50m1WkRUMinXvbA0=', 'Content-Length': '0'} 2024-02-01T10:27:40.211 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd367ebad49dc673-0065bb721c-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:40.212 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:40.212 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:40.215 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:40.215 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:40.215 DEBUG:boto:path=/vclfcn-19/obj?versionId=7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6 2024-02-01T10:27:40.215 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6 2024-02-01T10:27:40.215 DEBUG:boto:Method: GET 2024-02-01T10:27:40.215 DEBUG:boto:Path: /vclfcn-19/obj?versionId=7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6 2024-02-01T10:27:40.216 DEBUG:boto:Data: 2024-02-01T10:27:40.216 DEBUG:boto:Headers: {} 2024-02-01T10:27:40.216 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:40.216 DEBUG:boto:Port: 8000 2024-02-01T10:27:40.216 DEBUG:boto:Params: {} 2024-02-01T10:27:40.216 DEBUG:boto:Token: None 2024-02-01T10:27:40.216 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:40 GMT /vclfcn-19/obj?versionId=7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6 2024-02-01T10:27:40.216 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:IEPpWsvJ3OaHeQV6QdoyBEUiqgY= 2024-02-01T10:27:40.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:40 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:IEPpWsvJ3OaHeQV6QdoyBEUiqgY=', 'Content-Length': '0'} 2024-02-01T10:27:40.219 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a39e28f7bab1330f-0065bb721c-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:40.260 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:40.260 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:40.260 DEBUG:boto:path=/vclfcn-19/obj?versionId=7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6 2024-02-01T10:27:40.261 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6 2024-02-01T10:27:40.261 DEBUG:boto:Method: GET 2024-02-01T10:27:40.261 DEBUG:boto:Path: /vclfcn-19/obj?versionId=7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6 2024-02-01T10:27:40.261 DEBUG:boto:Data: 2024-02-01T10:27:40.261 DEBUG:boto:Headers: {} 2024-02-01T10:27:40.261 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:40.261 DEBUG:boto:Port: 8000 2024-02-01T10:27:40.261 DEBUG:boto:Params: {} 2024-02-01T10:27:40.261 DEBUG:boto:Token: None 2024-02-01T10:27:40.261 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:40 GMT /vclfcn-19/obj?versionId=7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6 2024-02-01T10:27:40.262 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:IEPpWsvJ3OaHeQV6QdoyBEUiqgY= 2024-02-01T10:27:40.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:40 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:IEPpWsvJ3OaHeQV6QdoyBEUiqgY=', 'Content-Length': '0'} 2024-02-01T10:27:40.265 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005471533c665b6233-0065bb721c-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:40.265 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:40.265 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:40.265 DEBUG:boto:path=/vclfcn-19/obj?versionId=7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6 2024-02-01T10:27:40.265 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6 2024-02-01T10:27:40.265 DEBUG:boto:Method: HEAD 2024-02-01T10:27:40.265 DEBUG:boto:Path: /vclfcn-19/obj?versionId=7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6 2024-02-01T10:27:40.266 DEBUG:boto:Data: 2024-02-01T10:27:40.266 DEBUG:boto:Headers: {} 2024-02-01T10:27:40.266 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:40.266 DEBUG:boto:Port: 8000 2024-02-01T10:27:40.266 DEBUG:boto:Params: {} 2024-02-01T10:27:40.266 DEBUG:boto:Token: None 2024-02-01T10:27:40.266 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:40 GMT /vclfcn-19/obj?versionId=7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6 2024-02-01T10:27:40.267 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/hUOCPAH8umVkotidPXvKzrOzfo= 2024-02-01T10:27:40.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:40 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/hUOCPAH8umVkotidPXvKzrOzfo=', 'Content-Length': '0'} 2024-02-01T10:27:40.269 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009d90e18caf4ec6ec-0065bb721c-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:40.270 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:40.270 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:40.270 DEBUG:boto:path=/vclfcn-19/obj?versionId=7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6 2024-02-01T10:27:40.270 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6 2024-02-01T10:27:40.270 DEBUG:boto:Method: HEAD 2024-02-01T10:27:40.270 DEBUG:boto:Path: /vclfcn-19/obj?versionId=7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6 2024-02-01T10:27:40.270 DEBUG:boto:Data: 2024-02-01T10:27:40.270 DEBUG:boto:Headers: {} 2024-02-01T10:27:40.270 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:40.271 DEBUG:boto:Port: 8000 2024-02-01T10:27:40.271 DEBUG:boto:Params: {} 2024-02-01T10:27:40.271 DEBUG:boto:Token: None 2024-02-01T10:27:40.271 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:40 GMT /vclfcn-19/obj?versionId=7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6 2024-02-01T10:27:40.271 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/hUOCPAH8umVkotidPXvKzrOzfo= 2024-02-01T10:27:40.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:40 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/hUOCPAH8umVkotidPXvKzrOzfo=', 'Content-Length': '0'} 2024-02-01T10:27:40.274 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bfbd8d2b6c0ae2f0-0065bb721c-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:40.274 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:40.275 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:40.275 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:40.275 DEBUG:boto:path=/vclfcn-19/obj?versionId=7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6 2024-02-01T10:27:40.276 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6 2024-02-01T10:27:40.276 DEBUG:boto:Method: GET 2024-02-01T10:27:40.276 DEBUG:boto:Path: /vclfcn-19/obj?versionId=7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6 2024-02-01T10:27:40.276 DEBUG:boto:Data: 2024-02-01T10:27:40.276 DEBUG:boto:Headers: {} 2024-02-01T10:27:40.276 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:40.276 DEBUG:boto:Port: 8000 2024-02-01T10:27:40.276 DEBUG:boto:Params: {} 2024-02-01T10:27:40.276 DEBUG:boto:Token: None 2024-02-01T10:27:40.276 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:40 GMT /vclfcn-19/obj?versionId=7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6 2024-02-01T10:27:40.277 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:IEPpWsvJ3OaHeQV6QdoyBEUiqgY= 2024-02-01T10:27:40.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:40 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:IEPpWsvJ3OaHeQV6QdoyBEUiqgY=', 'Content-Length': '0'} 2024-02-01T10:27:40.280 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007824a3cafa69fd28-0065bb721c-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:40.321 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:40.321 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:40.321 DEBUG:boto:path=/vclfcn-19/obj?versionId=7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6 2024-02-01T10:27:40.322 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6 2024-02-01T10:27:40.322 DEBUG:boto:Method: GET 2024-02-01T10:27:40.322 DEBUG:boto:Path: /vclfcn-19/obj?versionId=7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6 2024-02-01T10:27:40.322 DEBUG:boto:Data: 2024-02-01T10:27:40.322 DEBUG:boto:Headers: {} 2024-02-01T10:27:40.322 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:40.322 DEBUG:boto:Port: 8000 2024-02-01T10:27:40.322 DEBUG:boto:Params: {} 2024-02-01T10:27:40.322 DEBUG:boto:Token: None 2024-02-01T10:27:40.323 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:40 GMT /vclfcn-19/obj?versionId=7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6 2024-02-01T10:27:40.323 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:IEPpWsvJ3OaHeQV6QdoyBEUiqgY= 2024-02-01T10:27:40.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:40 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:IEPpWsvJ3OaHeQV6QdoyBEUiqgY=', 'Content-Length': '0'} 2024-02-01T10:27:40.326 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000318c5d61a65c98cd-0065bb721c-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:40.326 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:40.326 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:40.326 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:40.326 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:40.327 DEBUG:boto:Method: GET 2024-02-01T10:27:40.327 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:40.327 DEBUG:boto:Data: 2024-02-01T10:27:40.327 DEBUG:boto:Headers: {} 2024-02-01T10:27:40.327 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:40.327 DEBUG:boto:Port: 8000 2024-02-01T10:27:40.327 DEBUG:boto:Params: {} 2024-02-01T10:27:40.328 DEBUG:boto:Token: None 2024-02-01T10:27:40.328 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:40 GMT /vclfcn-19/ 2024-02-01T10:27:40.328 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:G1FlU0TeaZr50m1WkRUMinXvbA0= 2024-02-01T10:27:40.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:40 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:G1FlU0TeaZr50m1WkRUMinXvbA0=', 'Content-Length': '0'} 2024-02-01T10:27:40.333 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c685deffe3d29079-0065bb721c-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:40.333 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:40.334 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:40.334 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:40.334 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:40.334 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:40.334 DEBUG:boto:Method: GET 2024-02-01T10:27:40.334 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:40.335 DEBUG:boto:Data: 2024-02-01T10:27:40.335 DEBUG:boto:Headers: {} 2024-02-01T10:27:40.335 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:40.335 DEBUG:boto:Port: 8000 2024-02-01T10:27:40.335 DEBUG:boto:Params: {} 2024-02-01T10:27:40.335 DEBUG:boto:Token: None 2024-02-01T10:27:40.335 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:40 GMT /vclfcn-19/ 2024-02-01T10:27:40.335 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:G1FlU0TeaZr50m1WkRUMinXvbA0= 2024-02-01T10:27:40.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:40 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:G1FlU0TeaZr50m1WkRUMinXvbA0=', 'Content-Length': '0'} 2024-02-01T10:27:40.341 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007aaafd0174e6ad00-0065bb721c-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:40.341 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:40.341 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:40.341 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:40.342 DEBUG:boto:Method: HEAD 2024-02-01T10:27:40.342 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:40.342 DEBUG:boto:Data: 2024-02-01T10:27:40.342 DEBUG:boto:Headers: {} 2024-02-01T10:27:40.342 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:40.342 DEBUG:boto:Port: 8000 2024-02-01T10:27:40.342 DEBUG:boto:Params: {} 2024-02-01T10:27:40.342 DEBUG:boto:Token: None 2024-02-01T10:27:40.342 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:40 GMT /vclfcn-19/obj 2024-02-01T10:27:40.343 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fOrRaZ1X5+BTw9MNi5MBftd8KqU= 2024-02-01T10:27:40.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:40 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fOrRaZ1X5+BTw9MNi5MBftd8KqU=', 'Content-Length': '0'} 2024-02-01T10:27:40.348 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000352a57cc892bc071-0065bb721c-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:40.349 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:40.349 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:40.349 DEBUG:boto:Method: HEAD 2024-02-01T10:27:40.349 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:40.349 DEBUG:boto:Data: 2024-02-01T10:27:40.349 DEBUG:boto:Headers: {} 2024-02-01T10:27:40.349 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:40.349 DEBUG:boto:Port: 8000 2024-02-01T10:27:40.350 DEBUG:boto:Params: {} 2024-02-01T10:27:40.350 DEBUG:boto:Token: None 2024-02-01T10:27:40.350 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:40 GMT /vclfcn-19/obj 2024-02-01T10:27:40.350 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fOrRaZ1X5+BTw9MNi5MBftd8KqU= 2024-02-01T10:27:40.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:40 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fOrRaZ1X5+BTw9MNi5MBftd8KqU=', 'Content-Length': '0'} 2024-02-01T10:27:40.356 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a228e0e5cae9265c-0065bb721c-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:40.356 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:40.356 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:40.357 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:40.357 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:40.357 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:40.357 DEBUG:boto:Method: GET 2024-02-01T10:27:40.357 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:40.357 DEBUG:boto:Data: 2024-02-01T10:27:40.357 DEBUG:boto:Headers: {} 2024-02-01T10:27:40.358 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:40.358 DEBUG:boto:Port: 8000 2024-02-01T10:27:40.358 DEBUG:boto:Params: {} 2024-02-01T10:27:40.358 DEBUG:boto:Token: None 2024-02-01T10:27:40.358 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:40 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:40.358 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Bt/Gs/WYNv92el1KgJjH+bNm6Us= 2024-02-01T10:27:40.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:40 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Bt/Gs/WYNv92el1KgJjH+bNm6Us=', 'Content-Length': '0'} 2024-02-01T10:27:40.361 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000762f2114478f523-0065bb721c-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:40.404 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:40.404 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:40.405 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:40.405 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:40.405 DEBUG:boto:Method: GET 2024-02-01T10:27:40.405 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:40.405 DEBUG:boto:Data: 2024-02-01T10:27:40.405 DEBUG:boto:Headers: {} 2024-02-01T10:27:40.405 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:40.405 DEBUG:boto:Port: 8000 2024-02-01T10:27:40.405 DEBUG:boto:Params: {} 2024-02-01T10:27:40.406 DEBUG:boto:Token: None 2024-02-01T10:27:40.406 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:40 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:40.406 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Bt/Gs/WYNv92el1KgJjH+bNm6Us= 2024-02-01T10:27:40.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:40 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Bt/Gs/WYNv92el1KgJjH+bNm6Us=', 'Content-Length': '0'} 2024-02-01T10:27:40.409 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009370fe74f0e1cf99-0065bb721c-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:40.409 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:40.409 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:40.410 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:40.410 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:40.410 DEBUG:boto:Method: GET 2024-02-01T10:27:40.410 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:40.410 DEBUG:boto:Data: 2024-02-01T10:27:40.410 DEBUG:boto:Headers: {} 2024-02-01T10:27:40.410 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:40.410 DEBUG:boto:Port: 8000 2024-02-01T10:27:40.411 DEBUG:boto:Params: {} 2024-02-01T10:27:40.411 DEBUG:boto:Token: None 2024-02-01T10:27:40.411 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:40 GMT /vclfcn-19/ 2024-02-01T10:27:40.411 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:G1FlU0TeaZr50m1WkRUMinXvbA0= 2024-02-01T10:27:40.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:40 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:G1FlU0TeaZr50m1WkRUMinXvbA0=', 'Content-Length': '0'} 2024-02-01T10:27:40.419 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f02ef44b74a3ce0-0065bb721c-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:40.420 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:40.421 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:40.421 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:40.421 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:40.421 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:40.421 DEBUG:boto:Method: GET 2024-02-01T10:27:40.421 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:40.422 DEBUG:boto:Data: 2024-02-01T10:27:40.422 DEBUG:boto:Headers: {} 2024-02-01T10:27:40.422 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:40.422 DEBUG:boto:Port: 8000 2024-02-01T10:27:40.422 DEBUG:boto:Params: {} 2024-02-01T10:27:40.422 DEBUG:boto:Token: None 2024-02-01T10:27:40.422 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:40 GMT /vclfcn-19/ 2024-02-01T10:27:40.423 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:G1FlU0TeaZr50m1WkRUMinXvbA0= 2024-02-01T10:27:40.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:40 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:G1FlU0TeaZr50m1WkRUMinXvbA0=', 'Content-Length': '0'} 2024-02-01T10:27:40.428 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018714b8e0a69dbc4-0065bb721c-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:40.476 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:40.477 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:40.477 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:40.477 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:40.477 DEBUG:boto:path=/vclfcn-19/obj?versionId=.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX 2024-02-01T10:27:40.477 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX 2024-02-01T10:27:40.477 DEBUG:boto:Method: GET 2024-02-01T10:27:40.477 DEBUG:boto:Path: /vclfcn-19/obj?versionId=.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX 2024-02-01T10:27:40.477 DEBUG:boto:Data: 2024-02-01T10:27:40.477 DEBUG:boto:Headers: {} 2024-02-01T10:27:40.477 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:40.478 DEBUG:boto:Port: 8000 2024-02-01T10:27:40.478 DEBUG:boto:Params: {} 2024-02-01T10:27:40.478 DEBUG:boto:Token: None 2024-02-01T10:27:40.478 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:40 GMT /vclfcn-19/obj?versionId=.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX 2024-02-01T10:27:40.478 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Tij7NBuiFO/0grpbUKThkRR2BvE= 2024-02-01T10:27:40.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:40 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Tij7NBuiFO/0grpbUKThkRR2BvE=', 'Content-Length': '0'} 2024-02-01T10:27:40.481 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000035437a32ed0a25ac-0065bb721c-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:40.482 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:40.482 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:40.482 DEBUG:boto:path=/vclfcn-19/obj?versionId=.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX 2024-02-01T10:27:40.482 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX 2024-02-01T10:27:40.482 DEBUG:boto:Method: GET 2024-02-01T10:27:40.482 DEBUG:boto:Path: /vclfcn-19/obj?versionId=.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX 2024-02-01T10:27:40.482 DEBUG:boto:Data: 2024-02-01T10:27:40.483 DEBUG:boto:Headers: {} 2024-02-01T10:27:40.483 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:40.483 DEBUG:boto:Port: 8000 2024-02-01T10:27:40.483 DEBUG:boto:Params: {} 2024-02-01T10:27:40.483 DEBUG:boto:Token: None 2024-02-01T10:27:40.483 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:40 GMT /vclfcn-19/obj?versionId=.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX 2024-02-01T10:27:40.483 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Tij7NBuiFO/0grpbUKThkRR2BvE= 2024-02-01T10:27:40.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:40 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Tij7NBuiFO/0grpbUKThkRR2BvE=', 'Content-Length': '0'} 2024-02-01T10:27:40.486 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aca566420b26135b-0065bb721c-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:40.487 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:40.487 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:40.487 DEBUG:boto:path=/vclfcn-19/obj?versionId=.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX 2024-02-01T10:27:40.487 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX 2024-02-01T10:27:40.487 DEBUG:boto:Method: HEAD 2024-02-01T10:27:40.487 DEBUG:boto:Path: /vclfcn-19/obj?versionId=.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX 2024-02-01T10:27:40.488 DEBUG:boto:Data: 2024-02-01T10:27:40.488 DEBUG:boto:Headers: {} 2024-02-01T10:27:40.488 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:40.488 DEBUG:boto:Port: 8000 2024-02-01T10:27:40.488 DEBUG:boto:Params: {} 2024-02-01T10:27:40.488 DEBUG:boto:Token: None 2024-02-01T10:27:40.488 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:40 GMT /vclfcn-19/obj?versionId=.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX 2024-02-01T10:27:40.488 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ISbUisY7wnbV+TuTyxrfNUEDZss= 2024-02-01T10:27:40.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:40 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ISbUisY7wnbV+TuTyxrfNUEDZss=', 'Content-Length': '0'} 2024-02-01T10:27:40.491 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001c88cf9ac1c8b2b9-0065bb721c-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:40.491 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:40.492 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:40.492 DEBUG:boto:path=/vclfcn-19/obj?versionId=.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX 2024-02-01T10:27:40.492 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX 2024-02-01T10:27:40.492 DEBUG:boto:Method: HEAD 2024-02-01T10:27:40.492 DEBUG:boto:Path: /vclfcn-19/obj?versionId=.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX 2024-02-01T10:27:40.492 DEBUG:boto:Data: 2024-02-01T10:27:40.492 DEBUG:boto:Headers: {} 2024-02-01T10:27:40.492 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:40.492 DEBUG:boto:Port: 8000 2024-02-01T10:27:40.492 DEBUG:boto:Params: {} 2024-02-01T10:27:40.493 DEBUG:boto:Token: None 2024-02-01T10:27:40.493 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:40 GMT /vclfcn-19/obj?versionId=.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX 2024-02-01T10:27:40.493 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ISbUisY7wnbV+TuTyxrfNUEDZss= 2024-02-01T10:27:40.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:40 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ISbUisY7wnbV+TuTyxrfNUEDZss=', 'Content-Length': '0'} 2024-02-01T10:27:40.496 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f9dedbdd44f1b8f7-0065bb721c-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:40.496 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:40.496 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:40.496 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:40.496 DEBUG:boto:path=/vclfcn-19/obj?versionId=.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX 2024-02-01T10:27:40.496 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX 2024-02-01T10:27:40.497 DEBUG:boto:Method: GET 2024-02-01T10:27:40.497 DEBUG:boto:Path: /vclfcn-19/obj?versionId=.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX 2024-02-01T10:27:40.497 DEBUG:boto:Data: 2024-02-01T10:27:40.497 DEBUG:boto:Headers: {} 2024-02-01T10:27:40.497 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:40.497 DEBUG:boto:Port: 8000 2024-02-01T10:27:40.497 DEBUG:boto:Params: {} 2024-02-01T10:27:40.497 DEBUG:boto:Token: None 2024-02-01T10:27:40.497 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:40 GMT /vclfcn-19/obj?versionId=.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX 2024-02-01T10:27:40.498 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Tij7NBuiFO/0grpbUKThkRR2BvE= 2024-02-01T10:27:40.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:40 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Tij7NBuiFO/0grpbUKThkRR2BvE=', 'Content-Length': '0'} 2024-02-01T10:27:40.500 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000027c8623b8a3183cd-0065bb721c-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:40.501 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:40.501 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:40.501 DEBUG:boto:path=/vclfcn-19/obj?versionId=.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX 2024-02-01T10:27:40.501 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX 2024-02-01T10:27:40.501 DEBUG:boto:Method: GET 2024-02-01T10:27:40.501 DEBUG:boto:Path: /vclfcn-19/obj?versionId=.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX 2024-02-01T10:27:40.501 DEBUG:boto:Data: 2024-02-01T10:27:40.501 DEBUG:boto:Headers: {} 2024-02-01T10:27:40.501 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:40.501 DEBUG:boto:Port: 8000 2024-02-01T10:27:40.502 DEBUG:boto:Params: {} 2024-02-01T10:27:40.502 DEBUG:boto:Token: None 2024-02-01T10:27:40.502 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:40 GMT /vclfcn-19/obj?versionId=.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX 2024-02-01T10:27:40.502 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Tij7NBuiFO/0grpbUKThkRR2BvE= 2024-02-01T10:27:40.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:40 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Tij7NBuiFO/0grpbUKThkRR2BvE=', 'Content-Length': '0'} 2024-02-01T10:27:40.505 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000de38a5973921a44-0065bb721c-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:40.505 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:40.505 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:40.505 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:40.506 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:40.506 DEBUG:boto:Method: GET 2024-02-01T10:27:40.506 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:40.506 DEBUG:boto:Data: 2024-02-01T10:27:40.506 DEBUG:boto:Headers: {} 2024-02-01T10:27:40.506 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:40.506 DEBUG:boto:Port: 8000 2024-02-01T10:27:40.506 DEBUG:boto:Params: {} 2024-02-01T10:27:40.506 DEBUG:boto:Token: None 2024-02-01T10:27:40.506 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:40 GMT /vclfcn-19/ 2024-02-01T10:27:40.506 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:G1FlU0TeaZr50m1WkRUMinXvbA0= 2024-02-01T10:27:40.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:40 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:G1FlU0TeaZr50m1WkRUMinXvbA0=', 'Content-Length': '0'} 2024-02-01T10:27:40.512 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6fa845956277b5f-0065bb721c-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:40.556 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:40.557 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:40.557 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:40.557 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:40.557 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:40.557 DEBUG:boto:Method: GET 2024-02-01T10:27:40.557 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:40.557 DEBUG:boto:Data: 2024-02-01T10:27:40.557 DEBUG:boto:Headers: {} 2024-02-01T10:27:40.557 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:40.558 DEBUG:boto:Port: 8000 2024-02-01T10:27:40.558 DEBUG:boto:Params: {} 2024-02-01T10:27:40.558 DEBUG:boto:Token: None 2024-02-01T10:27:40.558 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:40 GMT /vclfcn-19/ 2024-02-01T10:27:40.558 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:G1FlU0TeaZr50m1WkRUMinXvbA0= 2024-02-01T10:27:40.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:40 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:G1FlU0TeaZr50m1WkRUMinXvbA0=', 'Content-Length': '0'} 2024-02-01T10:27:40.564 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b0fce136906f023-0065bb721c-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:40.564 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:40.565 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:40.565 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:40.565 DEBUG:boto:Method: HEAD 2024-02-01T10:27:40.565 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:40.565 DEBUG:boto:Data: 2024-02-01T10:27:40.565 DEBUG:boto:Headers: {} 2024-02-01T10:27:40.566 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:40.566 DEBUG:boto:Port: 8000 2024-02-01T10:27:40.566 DEBUG:boto:Params: {} 2024-02-01T10:27:40.566 DEBUG:boto:Token: None 2024-02-01T10:27:40.566 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:40 GMT /vclfcn-19/obj 2024-02-01T10:27:40.566 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fOrRaZ1X5+BTw9MNi5MBftd8KqU= 2024-02-01T10:27:40.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:40 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fOrRaZ1X5+BTw9MNi5MBftd8KqU=', 'Content-Length': '0'} 2024-02-01T10:27:40.572 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a5b190cfc270c1a0-0065bb721c-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:40.573 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:40.573 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:40.573 DEBUG:boto:Method: HEAD 2024-02-01T10:27:40.573 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:40.573 DEBUG:boto:Data: 2024-02-01T10:27:40.573 DEBUG:boto:Headers: {} 2024-02-01T10:27:40.573 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:40.573 DEBUG:boto:Port: 8000 2024-02-01T10:27:40.573 DEBUG:boto:Params: {} 2024-02-01T10:27:40.573 DEBUG:boto:Token: None 2024-02-01T10:27:40.574 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:40 GMT /vclfcn-19/obj 2024-02-01T10:27:40.574 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fOrRaZ1X5+BTw9MNi5MBftd8KqU= 2024-02-01T10:27:40.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:40 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fOrRaZ1X5+BTw9MNi5MBftd8KqU=', 'Content-Length': '0'} 2024-02-01T10:27:40.580 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000036b7f88e755f801d-0065bb721c-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:40.580 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:40.580 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:40.580 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:40.580 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:40.581 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:40.581 DEBUG:boto:Method: GET 2024-02-01T10:27:40.581 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:40.581 DEBUG:boto:Data: 2024-02-01T10:27:40.581 DEBUG:boto:Headers: {} 2024-02-01T10:27:40.581 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:40.581 DEBUG:boto:Port: 8000 2024-02-01T10:27:40.581 DEBUG:boto:Params: {} 2024-02-01T10:27:40.581 DEBUG:boto:Token: None 2024-02-01T10:27:40.581 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:40 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:40.582 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Bt/Gs/WYNv92el1KgJjH+bNm6Us= 2024-02-01T10:27:40.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:40 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Bt/Gs/WYNv92el1KgJjH+bNm6Us=', 'Content-Length': '0'} 2024-02-01T10:27:40.584 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009b933d7e5109ed30-0065bb721c-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:40.585 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:40.585 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:40.585 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:40.585 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:40.585 DEBUG:boto:Method: GET 2024-02-01T10:27:40.585 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:40.585 DEBUG:boto:Data: 2024-02-01T10:27:40.585 DEBUG:boto:Headers: {} 2024-02-01T10:27:40.585 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:40.585 DEBUG:boto:Port: 8000 2024-02-01T10:27:40.586 DEBUG:boto:Params: {} 2024-02-01T10:27:40.586 DEBUG:boto:Token: None 2024-02-01T10:27:40.586 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:40 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:40.586 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Bt/Gs/WYNv92el1KgJjH+bNm6Us= 2024-02-01T10:27:40.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:40 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Bt/Gs/WYNv92el1KgJjH+bNm6Us=', 'Content-Length': '0'} 2024-02-01T10:27:40.589 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000016852d4228a87579-0065bb721c-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:40.632 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:40.632 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:40.632 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:40.633 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:40.633 DEBUG:boto:Method: GET 2024-02-01T10:27:40.633 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:40.633 DEBUG:boto:Data: 2024-02-01T10:27:40.633 DEBUG:boto:Headers: {} 2024-02-01T10:27:40.633 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:40.634 DEBUG:boto:Port: 8000 2024-02-01T10:27:40.634 DEBUG:boto:Params: {} 2024-02-01T10:27:40.634 DEBUG:boto:Token: None 2024-02-01T10:27:40.634 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:40 GMT /vclfcn-19/ 2024-02-01T10:27:40.634 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:G1FlU0TeaZr50m1WkRUMinXvbA0= 2024-02-01T10:27:40.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:40 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:G1FlU0TeaZr50m1WkRUMinXvbA0=', 'Content-Length': '0'} 2024-02-01T10:27:40.640 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a0946ba65e1966b-0065bb721c-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:40.640 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:40.641 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:40.641 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:40.641 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:40.641 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:40.641 DEBUG:boto:Method: GET 2024-02-01T10:27:40.641 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:40.641 DEBUG:boto:Data: 2024-02-01T10:27:40.641 DEBUG:boto:Headers: {} 2024-02-01T10:27:40.641 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:40.642 DEBUG:boto:Port: 8000 2024-02-01T10:27:40.642 DEBUG:boto:Params: {} 2024-02-01T10:27:40.642 DEBUG:boto:Token: None 2024-02-01T10:27:40.642 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:40 GMT /vclfcn-19/ 2024-02-01T10:27:40.642 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:G1FlU0TeaZr50m1WkRUMinXvbA0= 2024-02-01T10:27:40.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:40 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:G1FlU0TeaZr50m1WkRUMinXvbA0=', 'Content-Length': '0'} 2024-02-01T10:27:40.647 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006bda13c34528363c-0065bb721c-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:40.647 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:40.648 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:40.648 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:40.648 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:40.648 DEBUG:boto:path=/vclfcn-19/obj?versionId=63CC4agggciY3jdYETzlNEHLx15uh8w 2024-02-01T10:27:40.648 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=63CC4agggciY3jdYETzlNEHLx15uh8w 2024-02-01T10:27:40.648 DEBUG:boto:Method: GET 2024-02-01T10:27:40.648 DEBUG:boto:Path: /vclfcn-19/obj?versionId=63CC4agggciY3jdYETzlNEHLx15uh8w 2024-02-01T10:27:40.649 DEBUG:boto:Data: 2024-02-01T10:27:40.649 DEBUG:boto:Headers: {} 2024-02-01T10:27:40.649 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:40.649 DEBUG:boto:Port: 8000 2024-02-01T10:27:40.649 DEBUG:boto:Params: {} 2024-02-01T10:27:40.650 DEBUG:boto:Token: None 2024-02-01T10:27:40.650 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:40 GMT /vclfcn-19/obj?versionId=63CC4agggciY3jdYETzlNEHLx15uh8w 2024-02-01T10:27:40.650 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+ox9Wdkw1lBZzIAnmnvJEVSwF7s= 2024-02-01T10:27:40.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:40 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+ox9Wdkw1lBZzIAnmnvJEVSwF7s=', 'Content-Length': '0'} 2024-02-01T10:27:40.654 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '63CC4agggciY3jdYETzlNEHLx15uh8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000020ae604dadb023b8-0065bb721c-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:40.696 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:40.696 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:40.697 DEBUG:boto:path=/vclfcn-19/obj?versionId=63CC4agggciY3jdYETzlNEHLx15uh8w 2024-02-01T10:27:40.697 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=63CC4agggciY3jdYETzlNEHLx15uh8w 2024-02-01T10:27:40.697 DEBUG:boto:Method: GET 2024-02-01T10:27:40.697 DEBUG:boto:Path: /vclfcn-19/obj?versionId=63CC4agggciY3jdYETzlNEHLx15uh8w 2024-02-01T10:27:40.698 DEBUG:boto:Data: 2024-02-01T10:27:40.698 DEBUG:boto:Headers: {} 2024-02-01T10:27:40.698 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:40.698 DEBUG:boto:Port: 8000 2024-02-01T10:27:40.698 DEBUG:boto:Params: {} 2024-02-01T10:27:40.698 DEBUG:boto:Token: None 2024-02-01T10:27:40.698 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:40 GMT /vclfcn-19/obj?versionId=63CC4agggciY3jdYETzlNEHLx15uh8w 2024-02-01T10:27:40.699 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+ox9Wdkw1lBZzIAnmnvJEVSwF7s= 2024-02-01T10:27:40.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:40 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+ox9Wdkw1lBZzIAnmnvJEVSwF7s=', 'Content-Length': '0'} 2024-02-01T10:27:40.702 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '63CC4agggciY3jdYETzlNEHLx15uh8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a3e4f30c88905c6a-0065bb721c-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:40.744 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:40.744 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:40.744 DEBUG:boto:path=/vclfcn-19/obj?versionId=63CC4agggciY3jdYETzlNEHLx15uh8w 2024-02-01T10:27:40.744 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=63CC4agggciY3jdYETzlNEHLx15uh8w 2024-02-01T10:27:40.745 DEBUG:boto:Method: HEAD 2024-02-01T10:27:40.745 DEBUG:boto:Path: /vclfcn-19/obj?versionId=63CC4agggciY3jdYETzlNEHLx15uh8w 2024-02-01T10:27:40.745 DEBUG:boto:Data: 2024-02-01T10:27:40.745 DEBUG:boto:Headers: {} 2024-02-01T10:27:40.745 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:40.745 DEBUG:boto:Port: 8000 2024-02-01T10:27:40.745 DEBUG:boto:Params: {} 2024-02-01T10:27:40.745 DEBUG:boto:Token: None 2024-02-01T10:27:40.745 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:40 GMT /vclfcn-19/obj?versionId=63CC4agggciY3jdYETzlNEHLx15uh8w 2024-02-01T10:27:40.746 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/ivzRRHFvlZRALabo9HC9zJbRpk= 2024-02-01T10:27:40.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:40 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/ivzRRHFvlZRALabo9HC9zJbRpk=', 'Content-Length': '0'} 2024-02-01T10:27:40.748 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '63CC4agggciY3jdYETzlNEHLx15uh8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004662216e8f7a843b-0065bb721c-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:40.749 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:40.749 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:40.749 DEBUG:boto:path=/vclfcn-19/obj?versionId=63CC4agggciY3jdYETzlNEHLx15uh8w 2024-02-01T10:27:40.749 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=63CC4agggciY3jdYETzlNEHLx15uh8w 2024-02-01T10:27:40.750 DEBUG:boto:Method: HEAD 2024-02-01T10:27:40.750 DEBUG:boto:Path: /vclfcn-19/obj?versionId=63CC4agggciY3jdYETzlNEHLx15uh8w 2024-02-01T10:27:40.750 DEBUG:boto:Data: 2024-02-01T10:27:40.750 DEBUG:boto:Headers: {} 2024-02-01T10:27:40.750 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:40.750 DEBUG:boto:Port: 8000 2024-02-01T10:27:40.750 DEBUG:boto:Params: {} 2024-02-01T10:27:40.750 DEBUG:boto:Token: None 2024-02-01T10:27:40.751 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:40 GMT /vclfcn-19/obj?versionId=63CC4agggciY3jdYETzlNEHLx15uh8w 2024-02-01T10:27:40.751 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/ivzRRHFvlZRALabo9HC9zJbRpk= 2024-02-01T10:27:40.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:40 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/ivzRRHFvlZRALabo9HC9zJbRpk=', 'Content-Length': '0'} 2024-02-01T10:27:40.754 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '63CC4agggciY3jdYETzlNEHLx15uh8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bc338e3c48b2e139-0065bb721c-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:40.754 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:40.754 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:40.754 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:40.754 DEBUG:boto:path=/vclfcn-19/obj?versionId=63CC4agggciY3jdYETzlNEHLx15uh8w 2024-02-01T10:27:40.754 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=63CC4agggciY3jdYETzlNEHLx15uh8w 2024-02-01T10:27:40.755 DEBUG:boto:Method: GET 2024-02-01T10:27:40.755 DEBUG:boto:Path: /vclfcn-19/obj?versionId=63CC4agggciY3jdYETzlNEHLx15uh8w 2024-02-01T10:27:40.755 DEBUG:boto:Data: 2024-02-01T10:27:40.755 DEBUG:boto:Headers: {} 2024-02-01T10:27:40.755 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:40.755 DEBUG:boto:Port: 8000 2024-02-01T10:27:40.755 DEBUG:boto:Params: {} 2024-02-01T10:27:40.755 DEBUG:boto:Token: None 2024-02-01T10:27:40.756 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:40 GMT /vclfcn-19/obj?versionId=63CC4agggciY3jdYETzlNEHLx15uh8w 2024-02-01T10:27:40.756 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+ox9Wdkw1lBZzIAnmnvJEVSwF7s= 2024-02-01T10:27:40.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:40 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+ox9Wdkw1lBZzIAnmnvJEVSwF7s=', 'Content-Length': '0'} 2024-02-01T10:27:40.759 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '63CC4agggciY3jdYETzlNEHLx15uh8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003f501453c78da085-0065bb721c-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:40.800 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:40.800 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:40.800 DEBUG:boto:path=/vclfcn-19/obj?versionId=63CC4agggciY3jdYETzlNEHLx15uh8w 2024-02-01T10:27:40.801 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=63CC4agggciY3jdYETzlNEHLx15uh8w 2024-02-01T10:27:40.801 DEBUG:boto:Method: GET 2024-02-01T10:27:40.801 DEBUG:boto:Path: /vclfcn-19/obj?versionId=63CC4agggciY3jdYETzlNEHLx15uh8w 2024-02-01T10:27:40.801 DEBUG:boto:Data: 2024-02-01T10:27:40.801 DEBUG:boto:Headers: {} 2024-02-01T10:27:40.801 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:40.801 DEBUG:boto:Port: 8000 2024-02-01T10:27:40.801 DEBUG:boto:Params: {} 2024-02-01T10:27:40.801 DEBUG:boto:Token: None 2024-02-01T10:27:40.801 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:40 GMT /vclfcn-19/obj?versionId=63CC4agggciY3jdYETzlNEHLx15uh8w 2024-02-01T10:27:40.802 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+ox9Wdkw1lBZzIAnmnvJEVSwF7s= 2024-02-01T10:27:40.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:40 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+ox9Wdkw1lBZzIAnmnvJEVSwF7s=', 'Content-Length': '0'} 2024-02-01T10:27:40.805 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '63CC4agggciY3jdYETzlNEHLx15uh8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000060cfb2ea4499212f-0065bb721c-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:40.848 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:40.848 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:40.848 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:40.849 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:40.849 DEBUG:boto:Method: GET 2024-02-01T10:27:40.849 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:40.849 DEBUG:boto:Data: 2024-02-01T10:27:40.849 DEBUG:boto:Headers: {} 2024-02-01T10:27:40.849 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:40.849 DEBUG:boto:Port: 8000 2024-02-01T10:27:40.849 DEBUG:boto:Params: {} 2024-02-01T10:27:40.849 DEBUG:boto:Token: None 2024-02-01T10:27:40.849 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:40 GMT /vclfcn-19/ 2024-02-01T10:27:40.850 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:G1FlU0TeaZr50m1WkRUMinXvbA0= 2024-02-01T10:27:40.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:40 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:G1FlU0TeaZr50m1WkRUMinXvbA0=', 'Content-Length': '0'} 2024-02-01T10:27:40.855 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ac1426120d77ef7-0065bb721c-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:40.855 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:40.855 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:40.855 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:40.856 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:40.856 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:40.856 DEBUG:boto:Method: GET 2024-02-01T10:27:40.856 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:40.856 DEBUG:boto:Data: 2024-02-01T10:27:40.856 DEBUG:boto:Headers: {} 2024-02-01T10:27:40.856 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:40.856 DEBUG:boto:Port: 8000 2024-02-01T10:27:40.856 DEBUG:boto:Params: {} 2024-02-01T10:27:40.856 DEBUG:boto:Token: None 2024-02-01T10:27:40.857 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:40 GMT /vclfcn-19/ 2024-02-01T10:27:40.857 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:G1FlU0TeaZr50m1WkRUMinXvbA0= 2024-02-01T10:27:40.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:40 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:G1FlU0TeaZr50m1WkRUMinXvbA0=', 'Content-Length': '0'} 2024-02-01T10:27:40.862 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab090e95131732b2-0065bb721c-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:40.862 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:40.862 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:40.862 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:40.862 DEBUG:boto:Method: HEAD 2024-02-01T10:27:40.863 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:40.863 DEBUG:boto:Data: 2024-02-01T10:27:40.863 DEBUG:boto:Headers: {} 2024-02-01T10:27:40.863 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:40.863 DEBUG:boto:Port: 8000 2024-02-01T10:27:40.863 DEBUG:boto:Params: {} 2024-02-01T10:27:40.863 DEBUG:boto:Token: None 2024-02-01T10:27:40.863 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:40 GMT /vclfcn-19/obj 2024-02-01T10:27:40.864 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fOrRaZ1X5+BTw9MNi5MBftd8KqU= 2024-02-01T10:27:40.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:40 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fOrRaZ1X5+BTw9MNi5MBftd8KqU=', 'Content-Length': '0'} 2024-02-01T10:27:40.869 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000091a6c11fef099431-0065bb721c-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:40.870 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:40.870 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:40.870 DEBUG:boto:Method: HEAD 2024-02-01T10:27:40.870 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:40.870 DEBUG:boto:Data: 2024-02-01T10:27:40.870 DEBUG:boto:Headers: {} 2024-02-01T10:27:40.870 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:40.870 DEBUG:boto:Port: 8000 2024-02-01T10:27:40.870 DEBUG:boto:Params: {} 2024-02-01T10:27:40.871 DEBUG:boto:Token: None 2024-02-01T10:27:40.871 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:40 GMT /vclfcn-19/obj 2024-02-01T10:27:40.871 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fOrRaZ1X5+BTw9MNi5MBftd8KqU= 2024-02-01T10:27:40.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:40 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fOrRaZ1X5+BTw9MNi5MBftd8KqU=', 'Content-Length': '0'} 2024-02-01T10:27:40.876 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000049a0f91f28043214-0065bb721c-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:40.876 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:40.877 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:40.877 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:40.877 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:40.877 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:40.877 DEBUG:boto:Method: GET 2024-02-01T10:27:40.877 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:40.877 DEBUG:boto:Data: 2024-02-01T10:27:40.877 DEBUG:boto:Headers: {} 2024-02-01T10:27:40.877 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:40.877 DEBUG:boto:Port: 8000 2024-02-01T10:27:40.877 DEBUG:boto:Params: {} 2024-02-01T10:27:40.878 DEBUG:boto:Token: None 2024-02-01T10:27:40.878 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:40 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:40.878 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Bt/Gs/WYNv92el1KgJjH+bNm6Us= 2024-02-01T10:27:40.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:40 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Bt/Gs/WYNv92el1KgJjH+bNm6Us=', 'Content-Length': '0'} 2024-02-01T10:27:40.881 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007b2d985af02661e5-0065bb721c-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:40.924 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:40.924 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:40.924 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:40.924 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:40.925 DEBUG:boto:Method: GET 2024-02-01T10:27:40.925 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:40.925 DEBUG:boto:Data: 2024-02-01T10:27:40.925 DEBUG:boto:Headers: {} 2024-02-01T10:27:40.925 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:40.925 DEBUG:boto:Port: 8000 2024-02-01T10:27:40.925 DEBUG:boto:Params: {} 2024-02-01T10:27:40.925 DEBUG:boto:Token: None 2024-02-01T10:27:40.925 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:40 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:40.926 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Bt/Gs/WYNv92el1KgJjH+bNm6Us= 2024-02-01T10:27:40.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:40 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Bt/Gs/WYNv92el1KgJjH+bNm6Us=', 'Content-Length': '0'} 2024-02-01T10:27:40.928 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c9ab8dc49525e7a6-0065bb721c-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:40.976 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:40.976 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:40.976 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:40.977 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:40.977 DEBUG:boto:Method: GET 2024-02-01T10:27:40.977 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:40.977 DEBUG:boto:Data: 2024-02-01T10:27:40.977 DEBUG:boto:Headers: {} 2024-02-01T10:27:40.977 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:40.977 DEBUG:boto:Port: 8000 2024-02-01T10:27:40.977 DEBUG:boto:Params: {} 2024-02-01T10:27:40.977 DEBUG:boto:Token: None 2024-02-01T10:27:40.977 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:40 GMT /vclfcn-19/ 2024-02-01T10:27:40.978 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:G1FlU0TeaZr50m1WkRUMinXvbA0= 2024-02-01T10:27:40.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:40 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:G1FlU0TeaZr50m1WkRUMinXvbA0=', 'Content-Length': '0'} 2024-02-01T10:27:40.983 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d70cc4ae32599304-0065bb721c-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:40.983 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:40.984 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:40.984 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:40.984 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:40.984 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:40.984 DEBUG:boto:Method: GET 2024-02-01T10:27:40.984 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:40.984 DEBUG:boto:Data: 2024-02-01T10:27:40.984 DEBUG:boto:Headers: {} 2024-02-01T10:27:40.984 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:40.984 DEBUG:boto:Port: 8000 2024-02-01T10:27:40.985 DEBUG:boto:Params: {} 2024-02-01T10:27:40.985 DEBUG:boto:Token: None 2024-02-01T10:27:40.985 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:40 GMT /vclfcn-19/ 2024-02-01T10:27:40.985 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:G1FlU0TeaZr50m1WkRUMinXvbA0= 2024-02-01T10:27:40.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:40 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:G1FlU0TeaZr50m1WkRUMinXvbA0=', 'Content-Length': '0'} 2024-02-01T10:27:40.990 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a96c3014d5e31e65-0065bb721c-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:40.990 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:40.991 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:40.991 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:40.991 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:40.991 DEBUG:boto:path=/vclfcn-19/obj?versionId=YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o 2024-02-01T10:27:40.991 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o 2024-02-01T10:27:40.991 DEBUG:boto:Method: GET 2024-02-01T10:27:40.991 DEBUG:boto:Path: /vclfcn-19/obj?versionId=YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o 2024-02-01T10:27:40.991 DEBUG:boto:Data: 2024-02-01T10:27:40.992 DEBUG:boto:Headers: {} 2024-02-01T10:27:40.992 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:40.992 DEBUG:boto:Port: 8000 2024-02-01T10:27:40.992 DEBUG:boto:Params: {} 2024-02-01T10:27:40.992 DEBUG:boto:Token: None 2024-02-01T10:27:40.992 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:40 GMT /vclfcn-19/obj?versionId=YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o 2024-02-01T10:27:40.992 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/TNdcaRm3U3bdAzH+VrzR4gFmgA= 2024-02-01T10:27:40.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:40 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/TNdcaRm3U3bdAzH+VrzR4gFmgA=', 'Content-Length': '0'} 2024-02-01T10:27:40.995 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('x-amz-version-id', 'YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003f3364e2917c16be-0065bb721c-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:41.036 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:41.036 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:41.036 DEBUG:boto:path=/vclfcn-19/obj?versionId=YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o 2024-02-01T10:27:41.036 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o 2024-02-01T10:27:41.037 DEBUG:boto:Method: GET 2024-02-01T10:27:41.037 DEBUG:boto:Path: /vclfcn-19/obj?versionId=YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o 2024-02-01T10:27:41.037 DEBUG:boto:Data: 2024-02-01T10:27:41.037 DEBUG:boto:Headers: {} 2024-02-01T10:27:41.037 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:41.037 DEBUG:boto:Port: 8000 2024-02-01T10:27:41.037 DEBUG:boto:Params: {} 2024-02-01T10:27:41.037 DEBUG:boto:Token: None 2024-02-01T10:27:41.037 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:41 GMT /vclfcn-19/obj?versionId=YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o 2024-02-01T10:27:41.037 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RXBwpDzKIS7EZB8qJ815q+kODGg= 2024-02-01T10:27:41.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:41 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RXBwpDzKIS7EZB8qJ815q+kODGg=', 'Content-Length': '0'} 2024-02-01T10:27:41.041 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('x-amz-version-id', 'YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000060369429bd99bb6d-0065bb721d-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:41.041 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:41.041 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:41.041 DEBUG:boto:path=/vclfcn-19/obj?versionId=YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o 2024-02-01T10:27:41.041 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o 2024-02-01T10:27:41.042 DEBUG:boto:Method: HEAD 2024-02-01T10:27:41.042 DEBUG:boto:Path: /vclfcn-19/obj?versionId=YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o 2024-02-01T10:27:41.042 DEBUG:boto:Data: 2024-02-01T10:27:41.042 DEBUG:boto:Headers: {} 2024-02-01T10:27:41.042 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:41.042 DEBUG:boto:Port: 8000 2024-02-01T10:27:41.042 DEBUG:boto:Params: {} 2024-02-01T10:27:41.042 DEBUG:boto:Token: None 2024-02-01T10:27:41.042 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:41 GMT /vclfcn-19/obj?versionId=YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o 2024-02-01T10:27:41.043 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HXW+PJjIBQVe1N8MfdSyPPs3knY= 2024-02-01T10:27:41.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:41 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HXW+PJjIBQVe1N8MfdSyPPs3knY=', 'Content-Length': '0'} 2024-02-01T10:27:41.045 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('x-amz-version-id', 'YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000855958f7445c70b8-0065bb721d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:41.045 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:41.045 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:41.046 DEBUG:boto:path=/vclfcn-19/obj?versionId=YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o 2024-02-01T10:27:41.046 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o 2024-02-01T10:27:41.046 DEBUG:boto:Method: HEAD 2024-02-01T10:27:41.046 DEBUG:boto:Path: /vclfcn-19/obj?versionId=YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o 2024-02-01T10:27:41.046 DEBUG:boto:Data: 2024-02-01T10:27:41.046 DEBUG:boto:Headers: {} 2024-02-01T10:27:41.046 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:41.046 DEBUG:boto:Port: 8000 2024-02-01T10:27:41.046 DEBUG:boto:Params: {} 2024-02-01T10:27:41.046 DEBUG:boto:Token: None 2024-02-01T10:27:41.046 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:41 GMT /vclfcn-19/obj?versionId=YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o 2024-02-01T10:27:41.047 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HXW+PJjIBQVe1N8MfdSyPPs3knY= 2024-02-01T10:27:41.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:41 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HXW+PJjIBQVe1N8MfdSyPPs3knY=', 'Content-Length': '0'} 2024-02-01T10:27:41.049 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('x-amz-version-id', 'YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e7c4d1bc363b9436-0065bb721d-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:41.050 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:41.050 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:41.050 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:41.050 DEBUG:boto:path=/vclfcn-19/obj?versionId=YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o 2024-02-01T10:27:41.050 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o 2024-02-01T10:27:41.050 DEBUG:boto:Method: GET 2024-02-01T10:27:41.050 DEBUG:boto:Path: /vclfcn-19/obj?versionId=YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o 2024-02-01T10:27:41.050 DEBUG:boto:Data: 2024-02-01T10:27:41.050 DEBUG:boto:Headers: {} 2024-02-01T10:27:41.050 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:41.051 DEBUG:boto:Port: 8000 2024-02-01T10:27:41.051 DEBUG:boto:Params: {} 2024-02-01T10:27:41.051 DEBUG:boto:Token: None 2024-02-01T10:27:41.051 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:41 GMT /vclfcn-19/obj?versionId=YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o 2024-02-01T10:27:41.051 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RXBwpDzKIS7EZB8qJ815q+kODGg= 2024-02-01T10:27:41.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:41 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RXBwpDzKIS7EZB8qJ815q+kODGg=', 'Content-Length': '0'} 2024-02-01T10:27:41.054 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('x-amz-version-id', 'YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000098217080963eae3b-0065bb721d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:41.100 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:41.100 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:41.100 DEBUG:boto:path=/vclfcn-19/obj?versionId=YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o 2024-02-01T10:27:41.100 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o 2024-02-01T10:27:41.101 DEBUG:boto:Method: GET 2024-02-01T10:27:41.101 DEBUG:boto:Path: /vclfcn-19/obj?versionId=YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o 2024-02-01T10:27:41.101 DEBUG:boto:Data: 2024-02-01T10:27:41.101 DEBUG:boto:Headers: {} 2024-02-01T10:27:41.101 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:41.101 DEBUG:boto:Port: 8000 2024-02-01T10:27:41.101 DEBUG:boto:Params: {} 2024-02-01T10:27:41.101 DEBUG:boto:Token: None 2024-02-01T10:27:41.101 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:41 GMT /vclfcn-19/obj?versionId=YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o 2024-02-01T10:27:41.101 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RXBwpDzKIS7EZB8qJ815q+kODGg= 2024-02-01T10:27:41.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:41 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RXBwpDzKIS7EZB8qJ815q+kODGg=', 'Content-Length': '0'} 2024-02-01T10:27:41.105 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('x-amz-version-id', 'YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c4362ed3f1a35f08-0065bb721d-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:41.105 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:41.105 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:41.105 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:41.105 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:41.106 DEBUG:boto:Method: GET 2024-02-01T10:27:41.106 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:41.106 DEBUG:boto:Data: 2024-02-01T10:27:41.106 DEBUG:boto:Headers: {} 2024-02-01T10:27:41.106 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:41.106 DEBUG:boto:Port: 8000 2024-02-01T10:27:41.106 DEBUG:boto:Params: {} 2024-02-01T10:27:41.106 DEBUG:boto:Token: None 2024-02-01T10:27:41.106 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:41 GMT /vclfcn-19/ 2024-02-01T10:27:41.106 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8ixkqvDW76nxp24RspSC+JCTgcQ= 2024-02-01T10:27:41.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:41 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8ixkqvDW76nxp24RspSC+JCTgcQ=', 'Content-Length': '0'} 2024-02-01T10:27:41.111 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf68351a474f4e65-0065bb721d-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:41.112 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:41.112 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:41.112 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:41.112 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:41.112 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:41.113 DEBUG:boto:Method: GET 2024-02-01T10:27:41.113 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:41.113 DEBUG:boto:Data: 2024-02-01T10:27:41.113 DEBUG:boto:Headers: {} 2024-02-01T10:27:41.113 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:41.113 DEBUG:boto:Port: 8000 2024-02-01T10:27:41.113 DEBUG:boto:Params: {} 2024-02-01T10:27:41.113 DEBUG:boto:Token: None 2024-02-01T10:27:41.113 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:41 GMT /vclfcn-19/ 2024-02-01T10:27:41.113 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8ixkqvDW76nxp24RspSC+JCTgcQ= 2024-02-01T10:27:41.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:41 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8ixkqvDW76nxp24RspSC+JCTgcQ=', 'Content-Length': '0'} 2024-02-01T10:27:41.119 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000423208f1bd0b74cd-0065bb721d-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:41.160 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:41.160 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:41.161 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:41.161 DEBUG:boto:Method: HEAD 2024-02-01T10:27:41.161 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:41.161 DEBUG:boto:Data: 2024-02-01T10:27:41.161 DEBUG:boto:Headers: {} 2024-02-01T10:27:41.161 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:41.161 DEBUG:boto:Port: 8000 2024-02-01T10:27:41.161 DEBUG:boto:Params: {} 2024-02-01T10:27:41.161 DEBUG:boto:Token: None 2024-02-01T10:27:41.161 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:41 GMT /vclfcn-19/obj 2024-02-01T10:27:41.161 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:qMCwb5TU1YNshEAetRY/i/FF3x4= 2024-02-01T10:27:41.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:41 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:qMCwb5TU1YNshEAetRY/i/FF3x4=', 'Content-Length': '0'} 2024-02-01T10:27:41.168 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000031683cbf1f38022f-0065bb721d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:41.169 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:41.169 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:41.169 DEBUG:boto:Method: HEAD 2024-02-01T10:27:41.169 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:41.169 DEBUG:boto:Data: 2024-02-01T10:27:41.169 DEBUG:boto:Headers: {} 2024-02-01T10:27:41.169 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:41.169 DEBUG:boto:Port: 8000 2024-02-01T10:27:41.169 DEBUG:boto:Params: {} 2024-02-01T10:27:41.169 DEBUG:boto:Token: None 2024-02-01T10:27:41.170 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:41 GMT /vclfcn-19/obj 2024-02-01T10:27:41.170 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:qMCwb5TU1YNshEAetRY/i/FF3x4= 2024-02-01T10:27:41.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:41 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:qMCwb5TU1YNshEAetRY/i/FF3x4=', 'Content-Length': '0'} 2024-02-01T10:27:41.176 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000026b79430a1e65287-0065bb721d-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:41.176 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:41.176 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:41.176 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:41.177 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:41.177 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:41.177 DEBUG:boto:Method: GET 2024-02-01T10:27:41.177 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:41.177 DEBUG:boto:Data: 2024-02-01T10:27:41.177 DEBUG:boto:Headers: {} 2024-02-01T10:27:41.177 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:41.177 DEBUG:boto:Port: 8000 2024-02-01T10:27:41.177 DEBUG:boto:Params: {} 2024-02-01T10:27:41.177 DEBUG:boto:Token: None 2024-02-01T10:27:41.177 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:41 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:41.178 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zf213Bd1qAhku1cfVZzx6dHrGsg= 2024-02-01T10:27:41.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:41 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zf213Bd1qAhku1cfVZzx6dHrGsg=', 'Content-Length': '0'} 2024-02-01T10:27:41.181 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005f152fb3538a0c83-0065bb721d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:41.181 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:41.181 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:41.181 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:41.181 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:41.181 DEBUG:boto:Method: GET 2024-02-01T10:27:41.182 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:41.182 DEBUG:boto:Data: 2024-02-01T10:27:41.182 DEBUG:boto:Headers: {} 2024-02-01T10:27:41.182 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:41.182 DEBUG:boto:Port: 8000 2024-02-01T10:27:41.182 DEBUG:boto:Params: {} 2024-02-01T10:27:41.182 DEBUG:boto:Token: None 2024-02-01T10:27:41.182 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:41 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:41.182 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zf213Bd1qAhku1cfVZzx6dHrGsg= 2024-02-01T10:27:41.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:41 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zf213Bd1qAhku1cfVZzx6dHrGsg=', 'Content-Length': '0'} 2024-02-01T10:27:41.185 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000093cb004c19d75d1b-0065bb721d-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:41.228 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:41.228 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:41.228 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:41.229 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:41.229 DEBUG:boto:Method: GET 2024-02-01T10:27:41.229 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:41.229 DEBUG:boto:Data: 2024-02-01T10:27:41.229 DEBUG:boto:Headers: {} 2024-02-01T10:27:41.229 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:41.229 DEBUG:boto:Port: 8000 2024-02-01T10:27:41.229 DEBUG:boto:Params: {} 2024-02-01T10:27:41.230 DEBUG:boto:Token: None 2024-02-01T10:27:41.230 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:41 GMT /vclfcn-19/ 2024-02-01T10:27:41.230 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8ixkqvDW76nxp24RspSC+JCTgcQ= 2024-02-01T10:27:41.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:41 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8ixkqvDW76nxp24RspSC+JCTgcQ=', 'Content-Length': '0'} 2024-02-01T10:27:41.235 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb3d2bbaed1328e5-0065bb721d-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:41.235 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:41.236 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:41.236 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:41.236 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:41.236 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:41.236 DEBUG:boto:Method: GET 2024-02-01T10:27:41.236 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:41.236 DEBUG:boto:Data: 2024-02-01T10:27:41.236 DEBUG:boto:Headers: {} 2024-02-01T10:27:41.237 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:41.237 DEBUG:boto:Port: 8000 2024-02-01T10:27:41.237 DEBUG:boto:Params: {} 2024-02-01T10:27:41.237 DEBUG:boto:Token: None 2024-02-01T10:27:41.237 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:41 GMT /vclfcn-19/ 2024-02-01T10:27:41.237 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8ixkqvDW76nxp24RspSC+JCTgcQ= 2024-02-01T10:27:41.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:41 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8ixkqvDW76nxp24RspSC+JCTgcQ=', 'Content-Length': '0'} 2024-02-01T10:27:41.244 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000631a86a9389515bf-0065bb721d-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:41.288 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:41.289 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:41.289 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:41.289 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:41.289 DEBUG:boto:path=/vclfcn-19/obj?versionId=B5jd89.SLRVLOAPcXqackukLvQnZY1B 2024-02-01T10:27:41.289 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=B5jd89.SLRVLOAPcXqackukLvQnZY1B 2024-02-01T10:27:41.289 DEBUG:boto:Method: GET 2024-02-01T10:27:41.289 DEBUG:boto:Path: /vclfcn-19/obj?versionId=B5jd89.SLRVLOAPcXqackukLvQnZY1B 2024-02-01T10:27:41.289 DEBUG:boto:Data: 2024-02-01T10:27:41.290 DEBUG:boto:Headers: {} 2024-02-01T10:27:41.290 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:41.290 DEBUG:boto:Port: 8000 2024-02-01T10:27:41.290 DEBUG:boto:Params: {} 2024-02-01T10:27:41.290 DEBUG:boto:Token: None 2024-02-01T10:27:41.290 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:41 GMT /vclfcn-19/obj?versionId=B5jd89.SLRVLOAPcXqackukLvQnZY1B 2024-02-01T10:27:41.290 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YCklKYG82GG8neVDH0w8jOMxjfg= 2024-02-01T10:27:41.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:41 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YCklKYG82GG8neVDH0w8jOMxjfg=', 'Content-Length': '0'} 2024-02-01T10:27:41.294 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('x-amz-version-id', 'B5jd89.SLRVLOAPcXqackukLvQnZY1B'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b9fdfb1bbaecb610-0065bb721d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:41.340 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:41.340 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:41.340 DEBUG:boto:path=/vclfcn-19/obj?versionId=B5jd89.SLRVLOAPcXqackukLvQnZY1B 2024-02-01T10:27:41.340 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=B5jd89.SLRVLOAPcXqackukLvQnZY1B 2024-02-01T10:27:41.341 DEBUG:boto:Method: GET 2024-02-01T10:27:41.341 DEBUG:boto:Path: /vclfcn-19/obj?versionId=B5jd89.SLRVLOAPcXqackukLvQnZY1B 2024-02-01T10:27:41.341 DEBUG:boto:Data: 2024-02-01T10:27:41.341 DEBUG:boto:Headers: {} 2024-02-01T10:27:41.341 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:41.341 DEBUG:boto:Port: 8000 2024-02-01T10:27:41.341 DEBUG:boto:Params: {} 2024-02-01T10:27:41.341 DEBUG:boto:Token: None 2024-02-01T10:27:41.341 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:41 GMT /vclfcn-19/obj?versionId=B5jd89.SLRVLOAPcXqackukLvQnZY1B 2024-02-01T10:27:41.341 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YCklKYG82GG8neVDH0w8jOMxjfg= 2024-02-01T10:27:41.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:41 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YCklKYG82GG8neVDH0w8jOMxjfg=', 'Content-Length': '0'} 2024-02-01T10:27:41.344 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('x-amz-version-id', 'B5jd89.SLRVLOAPcXqackukLvQnZY1B'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000e4e746e484283c4-0065bb721d-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:41.345 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:41.345 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:41.345 DEBUG:boto:path=/vclfcn-19/obj?versionId=B5jd89.SLRVLOAPcXqackukLvQnZY1B 2024-02-01T10:27:41.345 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=B5jd89.SLRVLOAPcXqackukLvQnZY1B 2024-02-01T10:27:41.345 DEBUG:boto:Method: HEAD 2024-02-01T10:27:41.345 DEBUG:boto:Path: /vclfcn-19/obj?versionId=B5jd89.SLRVLOAPcXqackukLvQnZY1B 2024-02-01T10:27:41.345 DEBUG:boto:Data: 2024-02-01T10:27:41.345 DEBUG:boto:Headers: {} 2024-02-01T10:27:41.345 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:41.345 DEBUG:boto:Port: 8000 2024-02-01T10:27:41.346 DEBUG:boto:Params: {} 2024-02-01T10:27:41.346 DEBUG:boto:Token: None 2024-02-01T10:27:41.346 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:41 GMT /vclfcn-19/obj?versionId=B5jd89.SLRVLOAPcXqackukLvQnZY1B 2024-02-01T10:27:41.346 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1TVkswOr6G3c4z6B3kWrUfbltEM= 2024-02-01T10:27:41.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:41 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1TVkswOr6G3c4z6B3kWrUfbltEM=', 'Content-Length': '0'} 2024-02-01T10:27:41.349 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('x-amz-version-id', 'B5jd89.SLRVLOAPcXqackukLvQnZY1B'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003995f99ee4174353-0065bb721d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:41.349 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:41.349 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:41.349 DEBUG:boto:path=/vclfcn-19/obj?versionId=B5jd89.SLRVLOAPcXqackukLvQnZY1B 2024-02-01T10:27:41.349 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=B5jd89.SLRVLOAPcXqackukLvQnZY1B 2024-02-01T10:27:41.349 DEBUG:boto:Method: HEAD 2024-02-01T10:27:41.349 DEBUG:boto:Path: /vclfcn-19/obj?versionId=B5jd89.SLRVLOAPcXqackukLvQnZY1B 2024-02-01T10:27:41.350 DEBUG:boto:Data: 2024-02-01T10:27:41.350 DEBUG:boto:Headers: {} 2024-02-01T10:27:41.350 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:41.350 DEBUG:boto:Port: 8000 2024-02-01T10:27:41.350 DEBUG:boto:Params: {} 2024-02-01T10:27:41.350 DEBUG:boto:Token: None 2024-02-01T10:27:41.350 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:41 GMT /vclfcn-19/obj?versionId=B5jd89.SLRVLOAPcXqackukLvQnZY1B 2024-02-01T10:27:41.350 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1TVkswOr6G3c4z6B3kWrUfbltEM= 2024-02-01T10:27:41.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:41 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1TVkswOr6G3c4z6B3kWrUfbltEM=', 'Content-Length': '0'} 2024-02-01T10:27:41.353 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('x-amz-version-id', 'B5jd89.SLRVLOAPcXqackukLvQnZY1B'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002296f5e3bf772509-0065bb721d-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:41.353 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:41.353 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:41.354 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:41.354 DEBUG:boto:path=/vclfcn-19/obj?versionId=B5jd89.SLRVLOAPcXqackukLvQnZY1B 2024-02-01T10:27:41.354 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=B5jd89.SLRVLOAPcXqackukLvQnZY1B 2024-02-01T10:27:41.354 DEBUG:boto:Method: GET 2024-02-01T10:27:41.354 DEBUG:boto:Path: /vclfcn-19/obj?versionId=B5jd89.SLRVLOAPcXqackukLvQnZY1B 2024-02-01T10:27:41.354 DEBUG:boto:Data: 2024-02-01T10:27:41.354 DEBUG:boto:Headers: {} 2024-02-01T10:27:41.354 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:41.354 DEBUG:boto:Port: 8000 2024-02-01T10:27:41.354 DEBUG:boto:Params: {} 2024-02-01T10:27:41.354 DEBUG:boto:Token: None 2024-02-01T10:27:41.355 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:41 GMT /vclfcn-19/obj?versionId=B5jd89.SLRVLOAPcXqackukLvQnZY1B 2024-02-01T10:27:41.355 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YCklKYG82GG8neVDH0w8jOMxjfg= 2024-02-01T10:27:41.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:41 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YCklKYG82GG8neVDH0w8jOMxjfg=', 'Content-Length': '0'} 2024-02-01T10:27:41.357 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('x-amz-version-id', 'B5jd89.SLRVLOAPcXqackukLvQnZY1B'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f7ed550b1d23069f-0065bb721d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:41.400 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:41.400 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:41.401 DEBUG:boto:path=/vclfcn-19/obj?versionId=B5jd89.SLRVLOAPcXqackukLvQnZY1B 2024-02-01T10:27:41.401 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=B5jd89.SLRVLOAPcXqackukLvQnZY1B 2024-02-01T10:27:41.401 DEBUG:boto:Method: GET 2024-02-01T10:27:41.401 DEBUG:boto:Path: /vclfcn-19/obj?versionId=B5jd89.SLRVLOAPcXqackukLvQnZY1B 2024-02-01T10:27:41.401 DEBUG:boto:Data: 2024-02-01T10:27:41.401 DEBUG:boto:Headers: {} 2024-02-01T10:27:41.402 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:41.402 DEBUG:boto:Port: 8000 2024-02-01T10:27:41.402 DEBUG:boto:Params: {} 2024-02-01T10:27:41.402 DEBUG:boto:Token: None 2024-02-01T10:27:41.402 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:41 GMT /vclfcn-19/obj?versionId=B5jd89.SLRVLOAPcXqackukLvQnZY1B 2024-02-01T10:27:41.402 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YCklKYG82GG8neVDH0w8jOMxjfg= 2024-02-01T10:27:41.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:41 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YCklKYG82GG8neVDH0w8jOMxjfg=', 'Content-Length': '0'} 2024-02-01T10:27:41.405 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('x-amz-version-id', 'B5jd89.SLRVLOAPcXqackukLvQnZY1B'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009bf39a39bc58ae46-0065bb721d-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:41.406 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:41.406 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:41.406 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:41.406 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:41.406 DEBUG:boto:Method: GET 2024-02-01T10:27:41.406 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:41.407 DEBUG:boto:Data: 2024-02-01T10:27:41.407 DEBUG:boto:Headers: {} 2024-02-01T10:27:41.407 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:41.407 DEBUG:boto:Port: 8000 2024-02-01T10:27:41.407 DEBUG:boto:Params: {} 2024-02-01T10:27:41.407 DEBUG:boto:Token: None 2024-02-01T10:27:41.408 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:41 GMT /vclfcn-19/ 2024-02-01T10:27:41.408 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8ixkqvDW76nxp24RspSC+JCTgcQ= 2024-02-01T10:27:41.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:41 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8ixkqvDW76nxp24RspSC+JCTgcQ=', 'Content-Length': '0'} 2024-02-01T10:27:41.413 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003acfe20e60c58350-0065bb721d-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:41.413 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:41.414 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:41.414 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:41.414 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:41.414 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:41.414 DEBUG:boto:Method: GET 2024-02-01T10:27:41.414 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:41.415 DEBUG:boto:Data: 2024-02-01T10:27:41.415 DEBUG:boto:Headers: {} 2024-02-01T10:27:41.415 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:41.415 DEBUG:boto:Port: 8000 2024-02-01T10:27:41.415 DEBUG:boto:Params: {} 2024-02-01T10:27:41.415 DEBUG:boto:Token: None 2024-02-01T10:27:41.415 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:41 GMT /vclfcn-19/ 2024-02-01T10:27:41.415 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8ixkqvDW76nxp24RspSC+JCTgcQ= 2024-02-01T10:27:41.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:41 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8ixkqvDW76nxp24RspSC+JCTgcQ=', 'Content-Length': '0'} 2024-02-01T10:27:41.421 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5b15229a24fb3a4-0065bb721d-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:41.421 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:41.421 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:41.422 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:41.422 DEBUG:boto:Method: HEAD 2024-02-01T10:27:41.422 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:41.422 DEBUG:boto:Data: 2024-02-01T10:27:41.422 DEBUG:boto:Headers: {} 2024-02-01T10:27:41.422 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:41.422 DEBUG:boto:Port: 8000 2024-02-01T10:27:41.422 DEBUG:boto:Params: {} 2024-02-01T10:27:41.422 DEBUG:boto:Token: None 2024-02-01T10:27:41.423 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:41 GMT /vclfcn-19/obj 2024-02-01T10:27:41.423 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:qMCwb5TU1YNshEAetRY/i/FF3x4= 2024-02-01T10:27:41.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:41 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:qMCwb5TU1YNshEAetRY/i/FF3x4=', 'Content-Length': '0'} 2024-02-01T10:27:41.429 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000096007d979c6b39b2-0065bb721d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:41.429 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:41.430 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:41.430 DEBUG:boto:Method: HEAD 2024-02-01T10:27:41.430 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:41.430 DEBUG:boto:Data: 2024-02-01T10:27:41.430 DEBUG:boto:Headers: {} 2024-02-01T10:27:41.430 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:41.430 DEBUG:boto:Port: 8000 2024-02-01T10:27:41.430 DEBUG:boto:Params: {} 2024-02-01T10:27:41.430 DEBUG:boto:Token: None 2024-02-01T10:27:41.430 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:41 GMT /vclfcn-19/obj 2024-02-01T10:27:41.431 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:qMCwb5TU1YNshEAetRY/i/FF3x4= 2024-02-01T10:27:41.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:41 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:qMCwb5TU1YNshEAetRY/i/FF3x4=', 'Content-Length': '0'} 2024-02-01T10:27:41.436 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000901b0f7b80c10b8e-0065bb721d-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:41.437 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:41.437 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:41.437 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:41.437 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:41.437 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:41.437 DEBUG:boto:Method: GET 2024-02-01T10:27:41.437 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:41.437 DEBUG:boto:Data: 2024-02-01T10:27:41.438 DEBUG:boto:Headers: {} 2024-02-01T10:27:41.438 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:41.438 DEBUG:boto:Port: 8000 2024-02-01T10:27:41.438 DEBUG:boto:Params: {} 2024-02-01T10:27:41.438 DEBUG:boto:Token: None 2024-02-01T10:27:41.438 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:41 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:41.438 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zf213Bd1qAhku1cfVZzx6dHrGsg= 2024-02-01T10:27:41.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:41 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zf213Bd1qAhku1cfVZzx6dHrGsg=', 'Content-Length': '0'} 2024-02-01T10:27:41.443 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004c4085710ef0ba79-0065bb721d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:41.488 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:41.488 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:41.488 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:41.489 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:41.489 DEBUG:boto:Method: GET 2024-02-01T10:27:41.489 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:41.489 DEBUG:boto:Data: 2024-02-01T10:27:41.489 DEBUG:boto:Headers: {} 2024-02-01T10:27:41.489 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:41.489 DEBUG:boto:Port: 8000 2024-02-01T10:27:41.489 DEBUG:boto:Params: {} 2024-02-01T10:27:41.489 DEBUG:boto:Token: None 2024-02-01T10:27:41.490 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:41 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:41.490 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zf213Bd1qAhku1cfVZzx6dHrGsg= 2024-02-01T10:27:41.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:41 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zf213Bd1qAhku1cfVZzx6dHrGsg=', 'Content-Length': '0'} 2024-02-01T10:27:41.493 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000053fd8d17867f2c0a-0065bb721d-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:41.540 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:41.540 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:41.541 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:41.541 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:41.541 DEBUG:boto:Method: GET 2024-02-01T10:27:41.541 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:41.541 DEBUG:boto:Data: 2024-02-01T10:27:41.541 DEBUG:boto:Headers: {} 2024-02-01T10:27:41.541 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:41.541 DEBUG:boto:Port: 8000 2024-02-01T10:27:41.541 DEBUG:boto:Params: {} 2024-02-01T10:27:41.542 DEBUG:boto:Token: None 2024-02-01T10:27:41.542 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:41 GMT /vclfcn-19/ 2024-02-01T10:27:41.542 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8ixkqvDW76nxp24RspSC+JCTgcQ= 2024-02-01T10:27:41.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:41 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8ixkqvDW76nxp24RspSC+JCTgcQ=', 'Content-Length': '0'} 2024-02-01T10:27:41.547 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db168240d92c5b01-0065bb721d-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:41.548 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:41.548 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:41.548 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:41.548 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:41.548 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:41.549 DEBUG:boto:Method: GET 2024-02-01T10:27:41.549 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:41.549 DEBUG:boto:Data: 2024-02-01T10:27:41.549 DEBUG:boto:Headers: {} 2024-02-01T10:27:41.549 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:41.549 DEBUG:boto:Port: 8000 2024-02-01T10:27:41.549 DEBUG:boto:Params: {} 2024-02-01T10:27:41.549 DEBUG:boto:Token: None 2024-02-01T10:27:41.549 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:41 GMT /vclfcn-19/ 2024-02-01T10:27:41.550 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8ixkqvDW76nxp24RspSC+JCTgcQ= 2024-02-01T10:27:41.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:41 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8ixkqvDW76nxp24RspSC+JCTgcQ=', 'Content-Length': '0'} 2024-02-01T10:27:41.554 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000543657b3a697915c-0065bb721d-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:41.555 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:41.555 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:41.555 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:41.555 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:41.555 DEBUG:boto:path=/vclfcn-19/obj?versionId=rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ 2024-02-01T10:27:41.555 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ 2024-02-01T10:27:41.556 DEBUG:boto:Method: GET 2024-02-01T10:27:41.556 DEBUG:boto:Path: /vclfcn-19/obj?versionId=rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ 2024-02-01T10:27:41.556 DEBUG:boto:Data: 2024-02-01T10:27:41.556 DEBUG:boto:Headers: {} 2024-02-01T10:27:41.556 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:41.556 DEBUG:boto:Port: 8000 2024-02-01T10:27:41.556 DEBUG:boto:Params: {} 2024-02-01T10:27:41.556 DEBUG:boto:Token: None 2024-02-01T10:27:41.556 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:41 GMT /vclfcn-19/obj?versionId=rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ 2024-02-01T10:27:41.557 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:pNX0yoNH/Q57c9FivhoLHxXaTt0= 2024-02-01T10:27:41.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:41 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:pNX0yoNH/Q57c9FivhoLHxXaTt0=', 'Content-Length': '0'} 2024-02-01T10:27:41.560 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('x-amz-version-id', 'rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008c8f837a11c5cafc-0065bb721d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:41.604 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:41.604 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:41.605 DEBUG:boto:path=/vclfcn-19/obj?versionId=rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ 2024-02-01T10:27:41.605 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ 2024-02-01T10:27:41.605 DEBUG:boto:Method: GET 2024-02-01T10:27:41.605 DEBUG:boto:Path: /vclfcn-19/obj?versionId=rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ 2024-02-01T10:27:41.605 DEBUG:boto:Data: 2024-02-01T10:27:41.606 DEBUG:boto:Headers: {} 2024-02-01T10:27:41.606 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:41.606 DEBUG:boto:Port: 8000 2024-02-01T10:27:41.606 DEBUG:boto:Params: {} 2024-02-01T10:27:41.606 DEBUG:boto:Token: None 2024-02-01T10:27:41.606 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:41 GMT /vclfcn-19/obj?versionId=rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ 2024-02-01T10:27:41.606 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:pNX0yoNH/Q57c9FivhoLHxXaTt0= 2024-02-01T10:27:41.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:41 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:pNX0yoNH/Q57c9FivhoLHxXaTt0=', 'Content-Length': '0'} 2024-02-01T10:27:41.610 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('x-amz-version-id', 'rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004df97da7b56c5f7e-0065bb721d-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:41.652 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:41.652 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:41.652 DEBUG:boto:path=/vclfcn-19/obj?versionId=rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ 2024-02-01T10:27:41.652 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ 2024-02-01T10:27:41.653 DEBUG:boto:Method: HEAD 2024-02-01T10:27:41.653 DEBUG:boto:Path: /vclfcn-19/obj?versionId=rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ 2024-02-01T10:27:41.653 DEBUG:boto:Data: 2024-02-01T10:27:41.653 DEBUG:boto:Headers: {} 2024-02-01T10:27:41.653 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:41.653 DEBUG:boto:Port: 8000 2024-02-01T10:27:41.653 DEBUG:boto:Params: {} 2024-02-01T10:27:41.653 DEBUG:boto:Token: None 2024-02-01T10:27:41.653 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:41 GMT /vclfcn-19/obj?versionId=rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ 2024-02-01T10:27:41.654 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+TltpG977Gxa5WHC0wi+pW1rxYM= 2024-02-01T10:27:41.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:41 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+TltpG977Gxa5WHC0wi+pW1rxYM=', 'Content-Length': '0'} 2024-02-01T10:27:41.657 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('x-amz-version-id', 'rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000af15b6d438d8160f-0065bb721d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:41.657 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:41.657 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:41.657 DEBUG:boto:path=/vclfcn-19/obj?versionId=rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ 2024-02-01T10:27:41.657 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ 2024-02-01T10:27:41.658 DEBUG:boto:Method: HEAD 2024-02-01T10:27:41.658 DEBUG:boto:Path: /vclfcn-19/obj?versionId=rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ 2024-02-01T10:27:41.658 DEBUG:boto:Data: 2024-02-01T10:27:41.658 DEBUG:boto:Headers: {} 2024-02-01T10:27:41.658 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:41.658 DEBUG:boto:Port: 8000 2024-02-01T10:27:41.658 DEBUG:boto:Params: {} 2024-02-01T10:27:41.658 DEBUG:boto:Token: None 2024-02-01T10:27:41.658 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:41 GMT /vclfcn-19/obj?versionId=rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ 2024-02-01T10:27:41.658 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+TltpG977Gxa5WHC0wi+pW1rxYM= 2024-02-01T10:27:41.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:41 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+TltpG977Gxa5WHC0wi+pW1rxYM=', 'Content-Length': '0'} 2024-02-01T10:27:41.661 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('x-amz-version-id', 'rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000208b130512efb31f-0065bb721d-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:41.662 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:41.662 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:41.662 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:41.662 DEBUG:boto:path=/vclfcn-19/obj?versionId=rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ 2024-02-01T10:27:41.662 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ 2024-02-01T10:27:41.662 DEBUG:boto:Method: GET 2024-02-01T10:27:41.662 DEBUG:boto:Path: /vclfcn-19/obj?versionId=rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ 2024-02-01T10:27:41.662 DEBUG:boto:Data: 2024-02-01T10:27:41.662 DEBUG:boto:Headers: {} 2024-02-01T10:27:41.663 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:41.663 DEBUG:boto:Port: 8000 2024-02-01T10:27:41.663 DEBUG:boto:Params: {} 2024-02-01T10:27:41.663 DEBUG:boto:Token: None 2024-02-01T10:27:41.663 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:41 GMT /vclfcn-19/obj?versionId=rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ 2024-02-01T10:27:41.663 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:pNX0yoNH/Q57c9FivhoLHxXaTt0= 2024-02-01T10:27:41.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:41 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:pNX0yoNH/Q57c9FivhoLHxXaTt0=', 'Content-Length': '0'} 2024-02-01T10:27:41.666 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('x-amz-version-id', 'rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000865971ce9b1cfa5-0065bb721d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:41.708 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:41.708 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:41.709 DEBUG:boto:path=/vclfcn-19/obj?versionId=rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ 2024-02-01T10:27:41.709 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ 2024-02-01T10:27:41.709 DEBUG:boto:Method: GET 2024-02-01T10:27:41.709 DEBUG:boto:Path: /vclfcn-19/obj?versionId=rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ 2024-02-01T10:27:41.709 DEBUG:boto:Data: 2024-02-01T10:27:41.709 DEBUG:boto:Headers: {} 2024-02-01T10:27:41.709 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:41.709 DEBUG:boto:Port: 8000 2024-02-01T10:27:41.709 DEBUG:boto:Params: {} 2024-02-01T10:27:41.710 DEBUG:boto:Token: None 2024-02-01T10:27:41.710 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:41 GMT /vclfcn-19/obj?versionId=rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ 2024-02-01T10:27:41.710 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:pNX0yoNH/Q57c9FivhoLHxXaTt0= 2024-02-01T10:27:41.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:41 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:pNX0yoNH/Q57c9FivhoLHxXaTt0=', 'Content-Length': '0'} 2024-02-01T10:27:41.713 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('x-amz-version-id', 'rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fe828d9f603965cd-0065bb721d-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:41.756 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:41.756 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:41.756 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:41.756 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:41.757 DEBUG:boto:Method: GET 2024-02-01T10:27:41.757 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:41.757 DEBUG:boto:Data: 2024-02-01T10:27:41.757 DEBUG:boto:Headers: {} 2024-02-01T10:27:41.757 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:41.757 DEBUG:boto:Port: 8000 2024-02-01T10:27:41.758 DEBUG:boto:Params: {} 2024-02-01T10:27:41.758 DEBUG:boto:Token: None 2024-02-01T10:27:41.758 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:41 GMT /vclfcn-19/ 2024-02-01T10:27:41.758 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8ixkqvDW76nxp24RspSC+JCTgcQ= 2024-02-01T10:27:41.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:41 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8ixkqvDW76nxp24RspSC+JCTgcQ=', 'Content-Length': '0'} 2024-02-01T10:27:41.763 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000825e341e63e8bbef-0065bb721d-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:41.764 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:41.765 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:41.765 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:41.765 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:41.765 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:41.765 DEBUG:boto:Method: GET 2024-02-01T10:27:41.765 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:41.765 DEBUG:boto:Data: 2024-02-01T10:27:41.765 DEBUG:boto:Headers: {} 2024-02-01T10:27:41.766 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:41.766 DEBUG:boto:Port: 8000 2024-02-01T10:27:41.766 DEBUG:boto:Params: {} 2024-02-01T10:27:41.766 DEBUG:boto:Token: None 2024-02-01T10:27:41.766 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:41 GMT /vclfcn-19/ 2024-02-01T10:27:41.766 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8ixkqvDW76nxp24RspSC+JCTgcQ= 2024-02-01T10:27:41.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:41 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8ixkqvDW76nxp24RspSC+JCTgcQ=', 'Content-Length': '0'} 2024-02-01T10:27:41.772 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b96ba0efac7a20c2-0065bb721d-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:41.772 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:41.772 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:41.772 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:41.773 DEBUG:boto:Method: HEAD 2024-02-01T10:27:41.773 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:41.773 DEBUG:boto:Data: 2024-02-01T10:27:41.773 DEBUG:boto:Headers: {} 2024-02-01T10:27:41.773 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:41.773 DEBUG:boto:Port: 8000 2024-02-01T10:27:41.773 DEBUG:boto:Params: {} 2024-02-01T10:27:41.773 DEBUG:boto:Token: None 2024-02-01T10:27:41.774 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:41 GMT /vclfcn-19/obj 2024-02-01T10:27:41.774 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:qMCwb5TU1YNshEAetRY/i/FF3x4= 2024-02-01T10:27:41.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:41 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:qMCwb5TU1YNshEAetRY/i/FF3x4=', 'Content-Length': '0'} 2024-02-01T10:27:41.780 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008c3e6c777b8edc70-0065bb721d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:41.780 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:41.780 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:41.780 DEBUG:boto:Method: HEAD 2024-02-01T10:27:41.780 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:41.780 DEBUG:boto:Data: 2024-02-01T10:27:41.780 DEBUG:boto:Headers: {} 2024-02-01T10:27:41.780 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:41.780 DEBUG:boto:Port: 8000 2024-02-01T10:27:41.781 DEBUG:boto:Params: {} 2024-02-01T10:27:41.781 DEBUG:boto:Token: None 2024-02-01T10:27:41.781 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:41 GMT /vclfcn-19/obj 2024-02-01T10:27:41.781 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:qMCwb5TU1YNshEAetRY/i/FF3x4= 2024-02-01T10:27:41.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:41 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:qMCwb5TU1YNshEAetRY/i/FF3x4=', 'Content-Length': '0'} 2024-02-01T10:27:41.787 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007f30fdc7d6087e85-0065bb721d-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:41.787 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:41.787 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:41.787 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:41.787 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:41.787 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:41.787 DEBUG:boto:Method: GET 2024-02-01T10:27:41.788 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:41.788 DEBUG:boto:Data: 2024-02-01T10:27:41.788 DEBUG:boto:Headers: {} 2024-02-01T10:27:41.788 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:41.788 DEBUG:boto:Port: 8000 2024-02-01T10:27:41.788 DEBUG:boto:Params: {} 2024-02-01T10:27:41.788 DEBUG:boto:Token: None 2024-02-01T10:27:41.788 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:41 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:41.788 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zf213Bd1qAhku1cfVZzx6dHrGsg= 2024-02-01T10:27:41.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:41 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zf213Bd1qAhku1cfVZzx6dHrGsg=', 'Content-Length': '0'} 2024-02-01T10:27:41.791 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d0bb44312a0a2fc1-0065bb721d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:41.792 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:41.792 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:41.792 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:41.792 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:41.792 DEBUG:boto:Method: GET 2024-02-01T10:27:41.792 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:41.792 DEBUG:boto:Data: 2024-02-01T10:27:41.792 DEBUG:boto:Headers: {} 2024-02-01T10:27:41.792 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:41.793 DEBUG:boto:Port: 8000 2024-02-01T10:27:41.793 DEBUG:boto:Params: {} 2024-02-01T10:27:41.793 DEBUG:boto:Token: None 2024-02-01T10:27:41.793 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:41 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:41.793 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zf213Bd1qAhku1cfVZzx6dHrGsg= 2024-02-01T10:27:41.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:41 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zf213Bd1qAhku1cfVZzx6dHrGsg=', 'Content-Length': '0'} 2024-02-01T10:27:41.796 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000083f6e417fc1889d7-0065bb721d-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:41.836 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:41.836 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:41.836 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:41.836 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:41.837 DEBUG:boto:Method: GET 2024-02-01T10:27:41.837 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:41.837 DEBUG:boto:Data: 2024-02-01T10:27:41.837 DEBUG:boto:Headers: {} 2024-02-01T10:27:41.837 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:41.837 DEBUG:boto:Port: 8000 2024-02-01T10:27:41.837 DEBUG:boto:Params: {} 2024-02-01T10:27:41.837 DEBUG:boto:Token: None 2024-02-01T10:27:41.837 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:41 GMT /vclfcn-19/ 2024-02-01T10:27:41.838 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8ixkqvDW76nxp24RspSC+JCTgcQ= 2024-02-01T10:27:41.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:41 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8ixkqvDW76nxp24RspSC+JCTgcQ=', 'Content-Length': '0'} 2024-02-01T10:27:41.843 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005010189d61410ebf-0065bb721d-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:41.844 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:41.844 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:41.844 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:41.844 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:41.845 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:41.845 DEBUG:boto:Method: GET 2024-02-01T10:27:41.845 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:41.845 DEBUG:boto:Data: 2024-02-01T10:27:41.845 DEBUG:boto:Headers: {} 2024-02-01T10:27:41.845 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:41.845 DEBUG:boto:Port: 8000 2024-02-01T10:27:41.845 DEBUG:boto:Params: {} 2024-02-01T10:27:41.845 DEBUG:boto:Token: None 2024-02-01T10:27:41.845 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:41 GMT /vclfcn-19/ 2024-02-01T10:27:41.846 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8ixkqvDW76nxp24RspSC+JCTgcQ= 2024-02-01T10:27:41.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:41 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8ixkqvDW76nxp24RspSC+JCTgcQ=', 'Content-Length': '0'} 2024-02-01T10:27:41.851 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000abb4906d081c43d2-0065bb721d-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:41.851 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:41.852 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:41.852 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:41.852 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:41.852 DEBUG:boto:path=/vclfcn-19/obj?versionId=s-qxjv7TkMdxeC61HL2kxUqODSwa-1I 2024-02-01T10:27:41.852 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=s-qxjv7TkMdxeC61HL2kxUqODSwa-1I 2024-02-01T10:27:41.852 DEBUG:boto:Method: GET 2024-02-01T10:27:41.852 DEBUG:boto:Path: /vclfcn-19/obj?versionId=s-qxjv7TkMdxeC61HL2kxUqODSwa-1I 2024-02-01T10:27:41.852 DEBUG:boto:Data: 2024-02-01T10:27:41.852 DEBUG:boto:Headers: {} 2024-02-01T10:27:41.853 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:41.853 DEBUG:boto:Port: 8000 2024-02-01T10:27:41.853 DEBUG:boto:Params: {} 2024-02-01T10:27:41.853 DEBUG:boto:Token: None 2024-02-01T10:27:41.853 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:41 GMT /vclfcn-19/obj?versionId=s-qxjv7TkMdxeC61HL2kxUqODSwa-1I 2024-02-01T10:27:41.853 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Oe7MSU2jcxn81KSqpf9z6uLEwPc= 2024-02-01T10:27:41.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:41 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Oe7MSU2jcxn81KSqpf9z6uLEwPc=', 'Content-Length': '0'} 2024-02-01T10:27:41.857 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('x-amz-version-id', 's-qxjv7TkMdxeC61HL2kxUqODSwa-1I'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fd8cf6afecff3516-0065bb721d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:41.900 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:41.900 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:41.900 DEBUG:boto:path=/vclfcn-19/obj?versionId=s-qxjv7TkMdxeC61HL2kxUqODSwa-1I 2024-02-01T10:27:41.901 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=s-qxjv7TkMdxeC61HL2kxUqODSwa-1I 2024-02-01T10:27:41.901 DEBUG:boto:Method: GET 2024-02-01T10:27:41.901 DEBUG:boto:Path: /vclfcn-19/obj?versionId=s-qxjv7TkMdxeC61HL2kxUqODSwa-1I 2024-02-01T10:27:41.901 DEBUG:boto:Data: 2024-02-01T10:27:41.901 DEBUG:boto:Headers: {} 2024-02-01T10:27:41.901 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:41.901 DEBUG:boto:Port: 8000 2024-02-01T10:27:41.901 DEBUG:boto:Params: {} 2024-02-01T10:27:41.901 DEBUG:boto:Token: None 2024-02-01T10:27:41.902 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:41 GMT /vclfcn-19/obj?versionId=s-qxjv7TkMdxeC61HL2kxUqODSwa-1I 2024-02-01T10:27:41.902 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Oe7MSU2jcxn81KSqpf9z6uLEwPc= 2024-02-01T10:27:41.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:41 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Oe7MSU2jcxn81KSqpf9z6uLEwPc=', 'Content-Length': '0'} 2024-02-01T10:27:41.905 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('x-amz-version-id', 's-qxjv7TkMdxeC61HL2kxUqODSwa-1I'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000081927cdc6a3efc0a-0065bb721d-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:41.948 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:41.948 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:41.948 DEBUG:boto:path=/vclfcn-19/obj?versionId=s-qxjv7TkMdxeC61HL2kxUqODSwa-1I 2024-02-01T10:27:41.949 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=s-qxjv7TkMdxeC61HL2kxUqODSwa-1I 2024-02-01T10:27:41.949 DEBUG:boto:Method: HEAD 2024-02-01T10:27:41.949 DEBUG:boto:Path: /vclfcn-19/obj?versionId=s-qxjv7TkMdxeC61HL2kxUqODSwa-1I 2024-02-01T10:27:41.949 DEBUG:boto:Data: 2024-02-01T10:27:41.949 DEBUG:boto:Headers: {} 2024-02-01T10:27:41.949 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:41.949 DEBUG:boto:Port: 8000 2024-02-01T10:27:41.949 DEBUG:boto:Params: {} 2024-02-01T10:27:41.949 DEBUG:boto:Token: None 2024-02-01T10:27:41.950 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:41 GMT /vclfcn-19/obj?versionId=s-qxjv7TkMdxeC61HL2kxUqODSwa-1I 2024-02-01T10:27:41.950 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:T9FAy1An9TMaNxFZRpE9UnMxzlA= 2024-02-01T10:27:41.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:41 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:T9FAy1An9TMaNxFZRpE9UnMxzlA=', 'Content-Length': '0'} 2024-02-01T10:27:41.953 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('x-amz-version-id', 's-qxjv7TkMdxeC61HL2kxUqODSwa-1I'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003e32dd9a021a9f9c-0065bb721d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:41.953 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:41.953 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:41.953 DEBUG:boto:path=/vclfcn-19/obj?versionId=s-qxjv7TkMdxeC61HL2kxUqODSwa-1I 2024-02-01T10:27:41.953 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=s-qxjv7TkMdxeC61HL2kxUqODSwa-1I 2024-02-01T10:27:41.953 DEBUG:boto:Method: HEAD 2024-02-01T10:27:41.953 DEBUG:boto:Path: /vclfcn-19/obj?versionId=s-qxjv7TkMdxeC61HL2kxUqODSwa-1I 2024-02-01T10:27:41.954 DEBUG:boto:Data: 2024-02-01T10:27:41.954 DEBUG:boto:Headers: {} 2024-02-01T10:27:41.954 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:41.954 DEBUG:boto:Port: 8000 2024-02-01T10:27:41.954 DEBUG:boto:Params: {} 2024-02-01T10:27:41.954 DEBUG:boto:Token: None 2024-02-01T10:27:41.954 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:41 GMT /vclfcn-19/obj?versionId=s-qxjv7TkMdxeC61HL2kxUqODSwa-1I 2024-02-01T10:27:41.954 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:T9FAy1An9TMaNxFZRpE9UnMxzlA= 2024-02-01T10:27:41.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:41 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:T9FAy1An9TMaNxFZRpE9UnMxzlA=', 'Content-Length': '0'} 2024-02-01T10:27:41.957 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('x-amz-version-id', 's-qxjv7TkMdxeC61HL2kxUqODSwa-1I'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002a8f760a36dfebd7-0065bb721d-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:41.957 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:41.957 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:41.958 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:41.958 DEBUG:boto:path=/vclfcn-19/obj?versionId=s-qxjv7TkMdxeC61HL2kxUqODSwa-1I 2024-02-01T10:27:41.958 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=s-qxjv7TkMdxeC61HL2kxUqODSwa-1I 2024-02-01T10:27:41.958 DEBUG:boto:Method: GET 2024-02-01T10:27:41.958 DEBUG:boto:Path: /vclfcn-19/obj?versionId=s-qxjv7TkMdxeC61HL2kxUqODSwa-1I 2024-02-01T10:27:41.958 DEBUG:boto:Data: 2024-02-01T10:27:41.958 DEBUG:boto:Headers: {} 2024-02-01T10:27:41.958 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:41.958 DEBUG:boto:Port: 8000 2024-02-01T10:27:41.958 DEBUG:boto:Params: {} 2024-02-01T10:27:41.958 DEBUG:boto:Token: None 2024-02-01T10:27:41.959 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:41 GMT /vclfcn-19/obj?versionId=s-qxjv7TkMdxeC61HL2kxUqODSwa-1I 2024-02-01T10:27:41.959 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Oe7MSU2jcxn81KSqpf9z6uLEwPc= 2024-02-01T10:27:41.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:41 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Oe7MSU2jcxn81KSqpf9z6uLEwPc=', 'Content-Length': '0'} 2024-02-01T10:27:41.962 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('x-amz-version-id', 's-qxjv7TkMdxeC61HL2kxUqODSwa-1I'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b56ac81f9548bee1-0065bb721d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:41.962 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:41.962 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:41.963 DEBUG:boto:path=/vclfcn-19/obj?versionId=s-qxjv7TkMdxeC61HL2kxUqODSwa-1I 2024-02-01T10:27:41.963 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=s-qxjv7TkMdxeC61HL2kxUqODSwa-1I 2024-02-01T10:27:41.963 DEBUG:boto:Method: GET 2024-02-01T10:27:41.963 DEBUG:boto:Path: /vclfcn-19/obj?versionId=s-qxjv7TkMdxeC61HL2kxUqODSwa-1I 2024-02-01T10:27:41.963 DEBUG:boto:Data: 2024-02-01T10:27:41.963 DEBUG:boto:Headers: {} 2024-02-01T10:27:41.963 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:41.963 DEBUG:boto:Port: 8000 2024-02-01T10:27:41.963 DEBUG:boto:Params: {} 2024-02-01T10:27:41.963 DEBUG:boto:Token: None 2024-02-01T10:27:41.963 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:41 GMT /vclfcn-19/obj?versionId=s-qxjv7TkMdxeC61HL2kxUqODSwa-1I 2024-02-01T10:27:41.964 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Oe7MSU2jcxn81KSqpf9z6uLEwPc= 2024-02-01T10:27:41.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:41 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Oe7MSU2jcxn81KSqpf9z6uLEwPc=', 'Content-Length': '0'} 2024-02-01T10:27:41.967 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('x-amz-version-id', 's-qxjv7TkMdxeC61HL2kxUqODSwa-1I'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008f5ff5ee0c983f67-0065bb721d-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:42.008 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:42.008 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:42.008 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:42.009 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:42.009 DEBUG:boto:Method: GET 2024-02-01T10:27:42.009 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:42.009 DEBUG:boto:Data: 2024-02-01T10:27:42.009 DEBUG:boto:Headers: {} 2024-02-01T10:27:42.009 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:42.009 DEBUG:boto:Port: 8000 2024-02-01T10:27:42.009 DEBUG:boto:Params: {} 2024-02-01T10:27:42.009 DEBUG:boto:Token: None 2024-02-01T10:27:42.009 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:42 GMT /vclfcn-19/ 2024-02-01T10:27:42.010 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:XISTLZllcTuoAfaN9hjt6JZUOL8= 2024-02-01T10:27:42.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:42 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:XISTLZllcTuoAfaN9hjt6JZUOL8=', 'Content-Length': '0'} 2024-02-01T10:27:42.015 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ba19cd86ffbe723-0065bb721e-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:42.015 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:42.016 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:42.016 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:42.016 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:42.016 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:42.016 DEBUG:boto:Method: GET 2024-02-01T10:27:42.016 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:42.016 DEBUG:boto:Data: 2024-02-01T10:27:42.017 DEBUG:boto:Headers: {} 2024-02-01T10:27:42.017 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:42.017 DEBUG:boto:Port: 8000 2024-02-01T10:27:42.017 DEBUG:boto:Params: {} 2024-02-01T10:27:42.017 DEBUG:boto:Token: None 2024-02-01T10:27:42.017 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:42 GMT /vclfcn-19/ 2024-02-01T10:27:42.017 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:XISTLZllcTuoAfaN9hjt6JZUOL8= 2024-02-01T10:27:42.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:42 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:XISTLZllcTuoAfaN9hjt6JZUOL8=', 'Content-Length': '0'} 2024-02-01T10:27:42.023 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f2140b843bbe2e2-0065bb721e-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:42.023 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:42.024 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:42.024 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:42.024 DEBUG:boto:Method: HEAD 2024-02-01T10:27:42.024 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:42.024 DEBUG:boto:Data: 2024-02-01T10:27:42.024 DEBUG:boto:Headers: {} 2024-02-01T10:27:42.024 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:42.024 DEBUG:boto:Port: 8000 2024-02-01T10:27:42.024 DEBUG:boto:Params: {} 2024-02-01T10:27:42.024 DEBUG:boto:Token: None 2024-02-01T10:27:42.025 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:42 GMT /vclfcn-19/obj 2024-02-01T10:27:42.025 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7B88XlrTjoEE86/ans2gBmjx5lM= 2024-02-01T10:27:42.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:42 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7B88XlrTjoEE86/ans2gBmjx5lM=', 'Content-Length': '0'} 2024-02-01T10:27:42.031 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d12b86093c2f024a-0065bb721e-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:42.032 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:42.032 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:42.032 DEBUG:boto:Method: HEAD 2024-02-01T10:27:42.032 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:42.032 DEBUG:boto:Data: 2024-02-01T10:27:42.032 DEBUG:boto:Headers: {} 2024-02-01T10:27:42.032 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:42.032 DEBUG:boto:Port: 8000 2024-02-01T10:27:42.032 DEBUG:boto:Params: {} 2024-02-01T10:27:42.033 DEBUG:boto:Token: None 2024-02-01T10:27:42.033 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:42 GMT /vclfcn-19/obj 2024-02-01T10:27:42.033 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7B88XlrTjoEE86/ans2gBmjx5lM= 2024-02-01T10:27:42.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:42 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7B88XlrTjoEE86/ans2gBmjx5lM=', 'Content-Length': '0'} 2024-02-01T10:27:42.039 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007d3d8b6bb22459ed-0065bb721e-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:42.039 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:42.039 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:42.039 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:42.039 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:42.039 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:42.039 DEBUG:boto:Method: GET 2024-02-01T10:27:42.040 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:42.040 DEBUG:boto:Data: 2024-02-01T10:27:42.040 DEBUG:boto:Headers: {} 2024-02-01T10:27:42.040 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:42.040 DEBUG:boto:Port: 8000 2024-02-01T10:27:42.040 DEBUG:boto:Params: {} 2024-02-01T10:27:42.040 DEBUG:boto:Token: None 2024-02-01T10:27:42.040 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:42 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:42.040 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9mOx2rIcNaUErPIMhkIjlSow7sw= 2024-02-01T10:27:42.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:42 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9mOx2rIcNaUErPIMhkIjlSow7sw=', 'Content-Length': '0'} 2024-02-01T10:27:42.044 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000481c24c03ef2300f-0065bb721e-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:42.084 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:42.084 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:42.084 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:42.085 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:42.085 DEBUG:boto:Method: GET 2024-02-01T10:27:42.085 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:42.085 DEBUG:boto:Data: 2024-02-01T10:27:42.085 DEBUG:boto:Headers: {} 2024-02-01T10:27:42.085 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:42.085 DEBUG:boto:Port: 8000 2024-02-01T10:27:42.085 DEBUG:boto:Params: {} 2024-02-01T10:27:42.085 DEBUG:boto:Token: None 2024-02-01T10:27:42.085 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:42 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:42.086 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9mOx2rIcNaUErPIMhkIjlSow7sw= 2024-02-01T10:27:42.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:42 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9mOx2rIcNaUErPIMhkIjlSow7sw=', 'Content-Length': '0'} 2024-02-01T10:27:42.089 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009dd4a0e46dcddcfc-0065bb721e-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:42.136 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:42.136 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:42.136 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:42.137 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:42.137 DEBUG:boto:Method: GET 2024-02-01T10:27:42.137 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:42.137 DEBUG:boto:Data: 2024-02-01T10:27:42.137 DEBUG:boto:Headers: {} 2024-02-01T10:27:42.137 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:42.137 DEBUG:boto:Port: 8000 2024-02-01T10:27:42.137 DEBUG:boto:Params: {} 2024-02-01T10:27:42.137 DEBUG:boto:Token: None 2024-02-01T10:27:42.137 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:42 GMT /vclfcn-19/ 2024-02-01T10:27:42.138 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:XISTLZllcTuoAfaN9hjt6JZUOL8= 2024-02-01T10:27:42.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:42 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:XISTLZllcTuoAfaN9hjt6JZUOL8=', 'Content-Length': '0'} 2024-02-01T10:27:42.143 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008dda133018dd78eb-0065bb721e-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:42.143 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:42.144 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:42.144 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:42.144 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:42.144 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:42.144 DEBUG:boto:Method: GET 2024-02-01T10:27:42.144 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:42.144 DEBUG:boto:Data: 2024-02-01T10:27:42.144 DEBUG:boto:Headers: {} 2024-02-01T10:27:42.144 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:42.144 DEBUG:boto:Port: 8000 2024-02-01T10:27:42.145 DEBUG:boto:Params: {} 2024-02-01T10:27:42.145 DEBUG:boto:Token: None 2024-02-01T10:27:42.145 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:42 GMT /vclfcn-19/ 2024-02-01T10:27:42.145 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:XISTLZllcTuoAfaN9hjt6JZUOL8= 2024-02-01T10:27:42.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:42 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:XISTLZllcTuoAfaN9hjt6JZUOL8=', 'Content-Length': '0'} 2024-02-01T10:27:42.150 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d167ae2f161dcdde-0065bb721e-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:42.150 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:42.151 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-19 zones={test-zone2, test-zone1} 2024-02-01T10:27:42.151 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-19 zones={test-zone3, test-zone1} 2024-02-01T10:27:42.151 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:42.151 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:42.151 DEBUG:boto:Method: HEAD 2024-02-01T10:27:42.151 DEBUG:boto:Path: /vclfcn-19/ 2024-02-01T10:27:42.152 DEBUG:boto:Data: 2024-02-01T10:27:42.152 DEBUG:boto:Headers: {} 2024-02-01T10:27:42.152 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:42.152 DEBUG:boto:Port: 8001 2024-02-01T10:27:42.152 DEBUG:boto:Params: {} 2024-02-01T10:27:42.152 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-02-01T10:27:42.152 DEBUG:boto:Token: None 2024-02-01T10:27:42.152 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:42 GMT /vclfcn-19/ 2024-02-01T10:27:42.153 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:leI0Oc9IoVlEsE85bGxLtsgpyKI= 2024-02-01T10:27:42.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:42 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:leI0Oc9IoVlEsE85bGxLtsgpyKI=', 'Content-Length': '0'} 2024-02-01T10:27:42.159 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000058cecb377dc5863b-0065bb721e-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:42.159 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:42.159 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:42.159 DEBUG:boto:Method: HEAD 2024-02-01T10:27:42.160 DEBUG:boto:Path: /vclfcn-19/ 2024-02-01T10:27:42.160 DEBUG:boto:Data: 2024-02-01T10:27:42.160 DEBUG:boto:Headers: {} 2024-02-01T10:27:42.160 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:42.160 DEBUG:boto:Port: 8000 2024-02-01T10:27:42.160 DEBUG:boto:Params: {} 2024-02-01T10:27:42.160 DEBUG:boto:Token: None 2024-02-01T10:27:42.160 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:42 GMT /vclfcn-19/ 2024-02-01T10:27:42.160 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:leI0Oc9IoVlEsE85bGxLtsgpyKI= 2024-02-01T10:27:42.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:42 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:leI0Oc9IoVlEsE85bGxLtsgpyKI=', 'Content-Length': '0'} 2024-02-01T10:27:42.165 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000017045f4764057a6d-0065bb721e-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:42.165 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:27:42.165 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:42.165 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:42.165 DEBUG:boto:path=/vclfcn-19/?versions&max-keys=999 2024-02-01T10:27:42.165 DEBUG:boto:auth_path=/vclfcn-19/?versions&max-keys=999 2024-02-01T10:27:42.165 DEBUG:boto:Method: GET 2024-02-01T10:27:42.166 DEBUG:boto:Path: /vclfcn-19/?versions&max-keys=999 2024-02-01T10:27:42.166 DEBUG:boto:Data: 2024-02-01T10:27:42.166 DEBUG:boto:Headers: {} 2024-02-01T10:27:42.166 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:42.166 DEBUG:boto:Port: 8001 2024-02-01T10:27:42.166 DEBUG:boto:Params: {} 2024-02-01T10:27:42.166 DEBUG:boto:Token: None 2024-02-01T10:27:42.166 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:42 GMT /vclfcn-19/?versions 2024-02-01T10:27:42.166 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:XqsaMhp2c2fkn/E5T5mMTPgaF+U= 2024-02-01T10:27:42.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:42 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:XqsaMhp2c2fkn/E5T5mMTPgaF+U=', 'Content-Length': '0'} 2024-02-01T10:27:42.173 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000837238d9f21e0dbd-0065bb721e-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:42.174 DEBUG:boto:b'vclfcn-19999falsedummyg5UScztMjPBIYZ2b5yOn-qsm.eQTcnutrue2024-02-01T10:23:11.115Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23llUchmWTvJKOzZpTkl59nzz.uMOQbtrue2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjRrytQti15jy37pft0WxtgWdt4Zs7l4Efalse2024-02-01T10:25:33.973Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjfzdlI7n.dsuwfAOSBNyk84Wd7l5p70ufalse2024-02-01T10:25:33.886Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUOa9NAxhXbH7RR3jkzlpXPh.fGAfBJcfalse2024-02-01T10:25:33.774Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjeNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr.false2024-02-01T10:25:33.746Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtGIsEYvB.VyZPdq9CgtD6kEeP4rzbDNfalse2024-02-01T10:25:33.830Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2lRbqumcCYnVWCIz20mU455245nXaz8false2024-02-01T10:25:33.719Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjoGhdnHMtrLPTgFgXlKfl.UXdmrUnL2rfalse2024-02-01T10:25:33.687Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjqex3Bh72IV4plxG2V2Lm7bvqwXqQNK9false2024-02-01T10:25:33.661Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjNCPEvRYQDJpLrTejJRqmTf2j0FWCBCSfalse2024-02-01T10:25:33.599Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjVQT2kBCkF07dugmyrhNeRpQpR0HK7bTfalse2024-02-01T10:25:33.630Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtvvjWUbgZIRYhHk9r.liT6sPCWCs2Zifalse2024-02-01T10:25:33.551Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3OSaU40E5-Hgv6pEDB8btq8uZcAFHVzfalse2024-02-01T10:25:33.438Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjK23uwsLWGmQDGYGsdu-h04Fk2wApB0Ofalse2024-02-01T10:25:33.410Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjst2TOT.K7IOe-wFfL.hOrRVfLNWblRdfalse2024-02-01T10:25:33.494Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9CJ3w1EusK8OST2k17sTdTM.cuh3E3efalse2024-02-01T10:25:33.349Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjVwfx4ETdAtTkKn2rTCQT4onqB6sYulvfalse2024-02-01T10:25:33.381Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjiGO2QyMOL81IaF-H4sHDqyECPlnZBmEfalse2024-02-01T10:25:33.323Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjcYHYWFiJtzK2FpqgaUq4bR.SX5MaCkIfalse2024-02-01T10:25:33.206Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjrasya8taOZsGTj9rDi7keghf.VnPFlFfalse2024-02-01T10:25:33.291Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjy-DPCvngcj9q7W-lUL16Jgbp.NhV7NEfalse2024-02-01T10:25:33.257Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj0KR6m6ymTGitKe6vabQyVENAnBL669wfalse2024-02-01T10:25:33.118Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2Hfm7x.sO5cadpLfjJNuhN.TAkjS54Efalse2024-02-01T10:25:33.093Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6false2024-02-01T10:25:33.150Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHXfalse2024-02-01T10:25:33.030Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj63CC4agggciY3jdYETzlNEHLx15uh8wfalse2024-02-01T10:25:33.063Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjYzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1ofalse2024-02-01T10:25:32.998Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjB5jd89.SLRVLOAPcXqackukLvQnZY1Bfalse2024-02-01T10:25:32.872Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjrcBw9NDMcjurzSdpB.bvwX4JmUDkwrQfalse2024-02-01T10:25:32.964Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjs-qxjv7TkMdxeC61HL2kxUqODSwa-1Ifalse2024-02-01T10:25:32.926Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:42.176 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-02-01T10:27:42.176 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:42.176 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:42.176 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:42.176 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:42.176 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:42.176 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:42.176 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:42.176 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:42.176 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:42.176 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:42.177 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:42.177 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:42.177 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:42.177 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:42.177 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:42.177 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:42.177 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:42.177 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:42.177 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:42.177 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:42.177 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:42.177 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:42.177 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:42.178 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:42.178 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:42.178 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:42.178 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:42.178 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:42.178 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:42.178 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:42.178 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:27:42.178 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:42.178 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:42.178 DEBUG:boto:path=/vclfcn-19/?versions&max-keys=999 2024-02-01T10:27:42.179 DEBUG:boto:auth_path=/vclfcn-19/?versions&max-keys=999 2024-02-01T10:27:42.179 DEBUG:boto:Method: GET 2024-02-01T10:27:42.179 DEBUG:boto:Path: /vclfcn-19/?versions&max-keys=999 2024-02-01T10:27:42.179 DEBUG:boto:Data: 2024-02-01T10:27:42.179 DEBUG:boto:Headers: {} 2024-02-01T10:27:42.179 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:42.179 DEBUG:boto:Port: 8000 2024-02-01T10:27:42.179 DEBUG:boto:Params: {} 2024-02-01T10:27:42.179 DEBUG:boto:Token: None 2024-02-01T10:27:42.179 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:42 GMT /vclfcn-19/?versions 2024-02-01T10:27:42.180 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:XqsaMhp2c2fkn/E5T5mMTPgaF+U= 2024-02-01T10:27:42.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:42 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:XqsaMhp2c2fkn/E5T5mMTPgaF+U=', 'Content-Length': '0'} 2024-02-01T10:27:42.187 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b3d0d4b7d73c780-0065bb721e-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:42.187 DEBUG:boto:b'vclfcn-19999falsedummyg5UScztMjPBIYZ2b5yOn-qsm.eQTcnutrue2024-02-01T10:23:11.115Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23llUchmWTvJKOzZpTkl59nzz.uMOQbtrue2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjRrytQti15jy37pft0WxtgWdt4Zs7l4Efalse2024-02-01T10:25:33.973Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjfzdlI7n.dsuwfAOSBNyk84Wd7l5p70ufalse2024-02-01T10:25:33.886Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUOa9NAxhXbH7RR3jkzlpXPh.fGAfBJcfalse2024-02-01T10:25:33.774Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjeNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr.false2024-02-01T10:25:33.746Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtGIsEYvB.VyZPdq9CgtD6kEeP4rzbDNfalse2024-02-01T10:25:33.830Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2lRbqumcCYnVWCIz20mU455245nXaz8false2024-02-01T10:25:33.719Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjoGhdnHMtrLPTgFgXlKfl.UXdmrUnL2rfalse2024-02-01T10:25:33.687Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjqex3Bh72IV4plxG2V2Lm7bvqwXqQNK9false2024-02-01T10:25:33.661Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjNCPEvRYQDJpLrTejJRqmTf2j0FWCBCSfalse2024-02-01T10:25:33.599Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjVQT2kBCkF07dugmyrhNeRpQpR0HK7bTfalse2024-02-01T10:25:33.630Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtvvjWUbgZIRYhHk9r.liT6sPCWCs2Zifalse2024-02-01T10:25:33.551Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3OSaU40E5-Hgv6pEDB8btq8uZcAFHVzfalse2024-02-01T10:25:33.438Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjK23uwsLWGmQDGYGsdu-h04Fk2wApB0Ofalse2024-02-01T10:25:33.410Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjst2TOT.K7IOe-wFfL.hOrRVfLNWblRdfalse2024-02-01T10:25:33.494Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9CJ3w1EusK8OST2k17sTdTM.cuh3E3efalse2024-02-01T10:25:33.349Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjVwfx4ETdAtTkKn2rTCQT4onqB6sYulvfalse2024-02-01T10:25:33.381Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjiGO2QyMOL81IaF-H4sHDqyECPlnZBmEfalse2024-02-01T10:25:33.323Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjcYHYWFiJtzK2FpqgaUq4bR.SX5MaCkIfalse2024-02-01T10:25:33.206Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjrasya8taOZsGTj9rDi7keghf.VnPFlFfalse2024-02-01T10:25:33.291Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjy-DPCvngcj9q7W-lUL16Jgbp.NhV7NEfalse2024-02-01T10:25:33.257Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj0KR6m6ymTGitKe6vabQyVENAnBL669wfalse2024-02-01T10:25:33.118Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2Hfm7x.sO5cadpLfjJNuhN.TAkjS54Efalse2024-02-01T10:25:33.093Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6false2024-02-01T10:25:33.150Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHXfalse2024-02-01T10:25:33.030Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj63CC4agggciY3jdYETzlNEHLx15uh8wfalse2024-02-01T10:25:33.063Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjYzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1ofalse2024-02-01T10:25:32.998Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjB5jd89.SLRVLOAPcXqackukLvQnZY1Bfalse2024-02-01T10:25:32.872Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjrcBw9NDMcjurzSdpB.bvwX4JmUDkwrQfalse2024-02-01T10:25:32.964Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjs-qxjv7TkMdxeC61HL2kxUqODSwa-1Ifalse2024-02-01T10:25:32.926Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:42.189 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-02-01T10:27:42.189 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:42.189 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:42.189 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:42.189 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:42.189 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:42.190 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:42.190 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:42.190 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:42.190 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:42.190 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:42.190 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:42.190 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:42.190 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:42.190 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:42.190 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:42.190 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:42.190 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:42.190 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:42.191 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:42.191 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:42.191 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:42.191 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:42.191 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:42.191 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:42.191 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:42.191 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:42.191 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:42.191 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:42.191 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:42.191 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:42.192 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:42.192 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:42.192 DEBUG:boto:path=/vclfcn-19/?versions&max-keys=999 2024-02-01T10:27:42.192 DEBUG:boto:auth_path=/vclfcn-19/?versions&max-keys=999 2024-02-01T10:27:42.192 DEBUG:boto:Method: GET 2024-02-01T10:27:42.192 DEBUG:boto:Path: /vclfcn-19/?versions&max-keys=999 2024-02-01T10:27:42.192 DEBUG:boto:Data: 2024-02-01T10:27:42.192 DEBUG:boto:Headers: {} 2024-02-01T10:27:42.192 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:42.192 DEBUG:boto:Port: 8001 2024-02-01T10:27:42.192 DEBUG:boto:Params: {} 2024-02-01T10:27:42.193 DEBUG:boto:Token: None 2024-02-01T10:27:42.193 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:42 GMT /vclfcn-19/?versions 2024-02-01T10:27:42.193 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:XqsaMhp2c2fkn/E5T5mMTPgaF+U= 2024-02-01T10:27:42.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:42 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:XqsaMhp2c2fkn/E5T5mMTPgaF+U=', 'Content-Length': '0'} 2024-02-01T10:27:42.201 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000135aa8dffc9fc640-0065bb721e-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:42.201 DEBUG:boto:b'vclfcn-19999falsedummyg5UScztMjPBIYZ2b5yOn-qsm.eQTcnutrue2024-02-01T10:23:11.115Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23llUchmWTvJKOzZpTkl59nzz.uMOQbtrue2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjRrytQti15jy37pft0WxtgWdt4Zs7l4Efalse2024-02-01T10:25:33.973Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjfzdlI7n.dsuwfAOSBNyk84Wd7l5p70ufalse2024-02-01T10:25:33.886Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUOa9NAxhXbH7RR3jkzlpXPh.fGAfBJcfalse2024-02-01T10:25:33.774Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjeNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr.false2024-02-01T10:25:33.746Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtGIsEYvB.VyZPdq9CgtD6kEeP4rzbDNfalse2024-02-01T10:25:33.830Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2lRbqumcCYnVWCIz20mU455245nXaz8false2024-02-01T10:25:33.719Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjoGhdnHMtrLPTgFgXlKfl.UXdmrUnL2rfalse2024-02-01T10:25:33.687Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjqex3Bh72IV4plxG2V2Lm7bvqwXqQNK9false2024-02-01T10:25:33.661Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjNCPEvRYQDJpLrTejJRqmTf2j0FWCBCSfalse2024-02-01T10:25:33.599Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjVQT2kBCkF07dugmyrhNeRpQpR0HK7bTfalse2024-02-01T10:25:33.630Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtvvjWUbgZIRYhHk9r.liT6sPCWCs2Zifalse2024-02-01T10:25:33.551Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3OSaU40E5-Hgv6pEDB8btq8uZcAFHVzfalse2024-02-01T10:25:33.438Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjK23uwsLWGmQDGYGsdu-h04Fk2wApB0Ofalse2024-02-01T10:25:33.410Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjst2TOT.K7IOe-wFfL.hOrRVfLNWblRdfalse2024-02-01T10:25:33.494Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9CJ3w1EusK8OST2k17sTdTM.cuh3E3efalse2024-02-01T10:25:33.349Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjVwfx4ETdAtTkKn2rTCQT4onqB6sYulvfalse2024-02-01T10:25:33.381Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjiGO2QyMOL81IaF-H4sHDqyECPlnZBmEfalse2024-02-01T10:25:33.323Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjcYHYWFiJtzK2FpqgaUq4bR.SX5MaCkIfalse2024-02-01T10:25:33.206Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjrasya8taOZsGTj9rDi7keghf.VnPFlFfalse2024-02-01T10:25:33.291Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjy-DPCvngcj9q7W-lUL16Jgbp.NhV7NEfalse2024-02-01T10:25:33.257Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj0KR6m6ymTGitKe6vabQyVENAnBL669wfalse2024-02-01T10:25:33.118Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2Hfm7x.sO5cadpLfjJNuhN.TAkjS54Efalse2024-02-01T10:25:33.093Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6false2024-02-01T10:25:33.150Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHXfalse2024-02-01T10:25:33.030Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj63CC4agggciY3jdYETzlNEHLx15uh8wfalse2024-02-01T10:25:33.063Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjYzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1ofalse2024-02-01T10:25:32.998Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjB5jd89.SLRVLOAPcXqackukLvQnZY1Bfalse2024-02-01T10:25:32.872Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjrcBw9NDMcjurzSdpB.bvwX4JmUDkwrQfalse2024-02-01T10:25:32.964Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjs-qxjv7TkMdxeC61HL2kxUqODSwa-1Ifalse2024-02-01T10:25:32.926Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:42.203 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:42.203 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:42.203 DEBUG:boto:path=/vclfcn-19/?versions&max-keys=999 2024-02-01T10:27:42.203 DEBUG:boto:auth_path=/vclfcn-19/?versions&max-keys=999 2024-02-01T10:27:42.204 DEBUG:boto:Method: GET 2024-02-01T10:27:42.204 DEBUG:boto:Path: /vclfcn-19/?versions&max-keys=999 2024-02-01T10:27:42.204 DEBUG:boto:Data: 2024-02-01T10:27:42.204 DEBUG:boto:Headers: {} 2024-02-01T10:27:42.204 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:42.204 DEBUG:boto:Port: 8000 2024-02-01T10:27:42.204 DEBUG:boto:Params: {} 2024-02-01T10:27:42.204 DEBUG:boto:Token: None 2024-02-01T10:27:42.204 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:42 GMT /vclfcn-19/?versions 2024-02-01T10:27:42.205 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:XqsaMhp2c2fkn/E5T5mMTPgaF+U= 2024-02-01T10:27:42.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:42 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:XqsaMhp2c2fkn/E5T5mMTPgaF+U=', 'Content-Length': '0'} 2024-02-01T10:27:42.212 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc09e6a42be64b3e-0065bb721e-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:42.212 DEBUG:boto:b'vclfcn-19999falsedummyg5UScztMjPBIYZ2b5yOn-qsm.eQTcnutrue2024-02-01T10:23:11.115Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23llUchmWTvJKOzZpTkl59nzz.uMOQbtrue2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjRrytQti15jy37pft0WxtgWdt4Zs7l4Efalse2024-02-01T10:25:33.973Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjfzdlI7n.dsuwfAOSBNyk84Wd7l5p70ufalse2024-02-01T10:25:33.886Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUOa9NAxhXbH7RR3jkzlpXPh.fGAfBJcfalse2024-02-01T10:25:33.774Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjeNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr.false2024-02-01T10:25:33.746Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtGIsEYvB.VyZPdq9CgtD6kEeP4rzbDNfalse2024-02-01T10:25:33.830Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2lRbqumcCYnVWCIz20mU455245nXaz8false2024-02-01T10:25:33.719Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjoGhdnHMtrLPTgFgXlKfl.UXdmrUnL2rfalse2024-02-01T10:25:33.687Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjqex3Bh72IV4plxG2V2Lm7bvqwXqQNK9false2024-02-01T10:25:33.661Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjNCPEvRYQDJpLrTejJRqmTf2j0FWCBCSfalse2024-02-01T10:25:33.599Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjVQT2kBCkF07dugmyrhNeRpQpR0HK7bTfalse2024-02-01T10:25:33.630Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtvvjWUbgZIRYhHk9r.liT6sPCWCs2Zifalse2024-02-01T10:25:33.551Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3OSaU40E5-Hgv6pEDB8btq8uZcAFHVzfalse2024-02-01T10:25:33.438Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjK23uwsLWGmQDGYGsdu-h04Fk2wApB0Ofalse2024-02-01T10:25:33.410Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjst2TOT.K7IOe-wFfL.hOrRVfLNWblRdfalse2024-02-01T10:25:33.494Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9CJ3w1EusK8OST2k17sTdTM.cuh3E3efalse2024-02-01T10:25:33.349Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjVwfx4ETdAtTkKn2rTCQT4onqB6sYulvfalse2024-02-01T10:25:33.381Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjiGO2QyMOL81IaF-H4sHDqyECPlnZBmEfalse2024-02-01T10:25:33.323Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjcYHYWFiJtzK2FpqgaUq4bR.SX5MaCkIfalse2024-02-01T10:25:33.206Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjrasya8taOZsGTj9rDi7keghf.VnPFlFfalse2024-02-01T10:25:33.291Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjy-DPCvngcj9q7W-lUL16Jgbp.NhV7NEfalse2024-02-01T10:25:33.257Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj0KR6m6ymTGitKe6vabQyVENAnBL669wfalse2024-02-01T10:25:33.118Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2Hfm7x.sO5cadpLfjJNuhN.TAkjS54Efalse2024-02-01T10:25:33.093Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6false2024-02-01T10:25:33.150Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHXfalse2024-02-01T10:25:33.030Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj63CC4agggciY3jdYETzlNEHLx15uh8wfalse2024-02-01T10:25:33.063Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjYzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1ofalse2024-02-01T10:25:32.998Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjB5jd89.SLRVLOAPcXqackukLvQnZY1Bfalse2024-02-01T10:25:32.872Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjrcBw9NDMcjurzSdpB.bvwX4JmUDkwrQfalse2024-02-01T10:25:32.964Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjs-qxjv7TkMdxeC61HL2kxUqODSwa-1Ifalse2024-02-01T10:25:32.926Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:42.214 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-02-01T10:27:42.214 DEBUG:boto:path=/vclfcn-19/dummy 2024-02-01T10:27:42.214 DEBUG:boto:auth_path=/vclfcn-19/dummy 2024-02-01T10:27:42.214 DEBUG:boto:path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:42.214 DEBUG:boto:auth_path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:42.214 DEBUG:boto:Method: GET 2024-02-01T10:27:42.215 DEBUG:boto:Path: /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:42.215 DEBUG:boto:Data: 2024-02-01T10:27:42.215 DEBUG:boto:Headers: {} 2024-02-01T10:27:42.215 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:42.215 DEBUG:boto:Port: 8001 2024-02-01T10:27:42.215 DEBUG:boto:Params: {} 2024-02-01T10:27:42.215 DEBUG:boto:Token: None 2024-02-01T10:27:42.215 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:42 GMT /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:42.215 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:aHrkx15cEEsKbI5eldi2rz/bZeE= 2024-02-01T10:27:42.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:42 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:aHrkx15cEEsKbI5eldi2rz/bZeE=', 'Content-Length': '0'} 2024-02-01T10:27:42.218 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:23:11 GMT'), ('x-amz-version-id', 'g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008bde527432778c8a-0065bb721e-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:42.218 DEBUG:boto:path=/vclfcn-19/dummy 2024-02-01T10:27:42.219 DEBUG:boto:auth_path=/vclfcn-19/dummy 2024-02-01T10:27:42.219 DEBUG:boto:path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:42.219 DEBUG:boto:auth_path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:42.219 DEBUG:boto:Method: GET 2024-02-01T10:27:42.219 DEBUG:boto:Path: /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:42.219 DEBUG:boto:Data: 2024-02-01T10:27:42.219 DEBUG:boto:Headers: {} 2024-02-01T10:27:42.219 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:42.219 DEBUG:boto:Port: 8000 2024-02-01T10:27:42.219 DEBUG:boto:Params: {} 2024-02-01T10:27:42.220 DEBUG:boto:Token: None 2024-02-01T10:27:42.220 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:42 GMT /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:42.220 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:aHrkx15cEEsKbI5eldi2rz/bZeE= 2024-02-01T10:27:42.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:42 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:aHrkx15cEEsKbI5eldi2rz/bZeE=', 'Content-Length': '0'} 2024-02-01T10:27:42.223 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:23:11 GMT'), ('x-amz-version-id', 'g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001748ce6ffcaf55e8-0065bb721e-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:42.223 DEBUG:boto:path=/vclfcn-19/dummy 2024-02-01T10:27:42.223 DEBUG:boto:auth_path=/vclfcn-19/dummy 2024-02-01T10:27:42.223 DEBUG:boto:path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:42.223 DEBUG:boto:auth_path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:42.224 DEBUG:boto:Method: HEAD 2024-02-01T10:27:42.224 DEBUG:boto:Path: /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:42.224 DEBUG:boto:Data: 2024-02-01T10:27:42.224 DEBUG:boto:Headers: {} 2024-02-01T10:27:42.224 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:42.224 DEBUG:boto:Port: 8001 2024-02-01T10:27:42.224 DEBUG:boto:Params: {} 2024-02-01T10:27:42.224 DEBUG:boto:Token: None 2024-02-01T10:27:42.224 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:42 GMT /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:42.225 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sCVT2oiE/GoGFlWp3AuFH0ZvM0U= 2024-02-01T10:27:42.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:42 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sCVT2oiE/GoGFlWp3AuFH0ZvM0U=', 'Content-Length': '0'} 2024-02-01T10:27:42.227 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:23:11 GMT'), ('x-amz-version-id', 'g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007fca2b6d64ab6179-0065bb721e-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:42.228 DEBUG:boto:path=/vclfcn-19/dummy 2024-02-01T10:27:42.228 DEBUG:boto:auth_path=/vclfcn-19/dummy 2024-02-01T10:27:42.228 DEBUG:boto:path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:42.228 DEBUG:boto:auth_path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:42.228 DEBUG:boto:Method: HEAD 2024-02-01T10:27:42.228 DEBUG:boto:Path: /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:42.228 DEBUG:boto:Data: 2024-02-01T10:27:42.228 DEBUG:boto:Headers: {} 2024-02-01T10:27:42.228 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:42.228 DEBUG:boto:Port: 8000 2024-02-01T10:27:42.228 DEBUG:boto:Params: {} 2024-02-01T10:27:42.229 DEBUG:boto:Token: None 2024-02-01T10:27:42.229 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:42 GMT /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:42.229 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sCVT2oiE/GoGFlWp3AuFH0ZvM0U= 2024-02-01T10:27:42.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:42 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sCVT2oiE/GoGFlWp3AuFH0ZvM0U=', 'Content-Length': '0'} 2024-02-01T10:27:42.232 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:23:11 GMT'), ('x-amz-version-id', 'g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000087560520237f319c-0065bb721e-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:42.232 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-02-01T10:27:42.233 DEBUG:boto:path=/vclfcn-19/dummy 2024-02-01T10:27:42.233 DEBUG:boto:auth_path=/vclfcn-19/dummy 2024-02-01T10:27:42.233 DEBUG:boto:path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:42.233 DEBUG:boto:auth_path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:42.233 DEBUG:boto:Method: GET 2024-02-01T10:27:42.234 DEBUG:boto:Path: /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:42.234 DEBUG:boto:Data: 2024-02-01T10:27:42.234 DEBUG:boto:Headers: {} 2024-02-01T10:27:42.234 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:42.234 DEBUG:boto:Port: 8001 2024-02-01T10:27:42.234 DEBUG:boto:Params: {} 2024-02-01T10:27:42.234 DEBUG:boto:Token: None 2024-02-01T10:27:42.234 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:42 GMT /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:42.235 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:aHrkx15cEEsKbI5eldi2rz/bZeE= 2024-02-01T10:27:42.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:42 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:aHrkx15cEEsKbI5eldi2rz/bZeE=', 'Content-Length': '0'} 2024-02-01T10:27:42.237 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:23:11 GMT'), ('x-amz-version-id', 'g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007c83bc0c67508913-0065bb721e-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:42.237 DEBUG:boto:path=/vclfcn-19/dummy 2024-02-01T10:27:42.238 DEBUG:boto:auth_path=/vclfcn-19/dummy 2024-02-01T10:27:42.238 DEBUG:boto:path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:42.238 DEBUG:boto:auth_path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:42.238 DEBUG:boto:Method: GET 2024-02-01T10:27:42.238 DEBUG:boto:Path: /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:42.238 DEBUG:boto:Data: 2024-02-01T10:27:42.238 DEBUG:boto:Headers: {} 2024-02-01T10:27:42.238 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:42.238 DEBUG:boto:Port: 8000 2024-02-01T10:27:42.239 DEBUG:boto:Params: {} 2024-02-01T10:27:42.239 DEBUG:boto:Token: None 2024-02-01T10:27:42.239 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:42 GMT /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:42.239 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:aHrkx15cEEsKbI5eldi2rz/bZeE= 2024-02-01T10:27:42.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:42 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:aHrkx15cEEsKbI5eldi2rz/bZeE=', 'Content-Length': '0'} 2024-02-01T10:27:42.242 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:23:11 GMT'), ('x-amz-version-id', 'g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b977af62db5b270c-0065bb721e-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:42.242 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:42.242 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:42.242 DEBUG:boto:path=/vclfcn-19/?prefix=dummy 2024-02-01T10:27:42.242 DEBUG:boto:auth_path=/vclfcn-19/?prefix=dummy 2024-02-01T10:27:42.242 DEBUG:boto:Method: GET 2024-02-01T10:27:42.242 DEBUG:boto:Path: /vclfcn-19/?prefix=dummy 2024-02-01T10:27:42.243 DEBUG:boto:Data: 2024-02-01T10:27:42.243 DEBUG:boto:Headers: {} 2024-02-01T10:27:42.243 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:42.243 DEBUG:boto:Port: 8001 2024-02-01T10:27:42.243 DEBUG:boto:Params: {} 2024-02-01T10:27:42.244 DEBUG:boto:Token: None 2024-02-01T10:27:42.244 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:42 GMT /vclfcn-19/ 2024-02-01T10:27:42.244 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:XISTLZllcTuoAfaN9hjt6JZUOL8= 2024-02-01T10:27:42.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:42 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:XISTLZllcTuoAfaN9hjt6JZUOL8=', 'Content-Length': '0'} 2024-02-01T10:27:42.249 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de8d7e773ca129d0-0065bb721e-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:42.296 DEBUG:boto:b'vclfcn-19dummy1000falsedummy2024-02-01T10:23:11.115Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:42.297 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:42.297 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:42.297 DEBUG:boto:path=/vclfcn-19/?prefix=dummy 2024-02-01T10:27:42.297 DEBUG:boto:auth_path=/vclfcn-19/?prefix=dummy 2024-02-01T10:27:42.297 DEBUG:boto:Method: GET 2024-02-01T10:27:42.297 DEBUG:boto:Path: /vclfcn-19/?prefix=dummy 2024-02-01T10:27:42.297 DEBUG:boto:Data: 2024-02-01T10:27:42.297 DEBUG:boto:Headers: {} 2024-02-01T10:27:42.297 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:42.298 DEBUG:boto:Port: 8000 2024-02-01T10:27:42.298 DEBUG:boto:Params: {} 2024-02-01T10:27:42.298 DEBUG:boto:Token: None 2024-02-01T10:27:42.298 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:42 GMT /vclfcn-19/ 2024-02-01T10:27:42.298 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:XISTLZllcTuoAfaN9hjt6JZUOL8= 2024-02-01T10:27:42.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:42 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:XISTLZllcTuoAfaN9hjt6JZUOL8=', 'Content-Length': '0'} 2024-02-01T10:27:42.303 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000000af203cd01e8956-0065bb721e-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:42.304 DEBUG:boto:b'vclfcn-19dummy1000falsedummy2024-02-01T10:23:11.115Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:42.304 DEBUG:boto:path=/vclfcn-19/dummy 2024-02-01T10:27:42.304 DEBUG:boto:auth_path=/vclfcn-19/dummy 2024-02-01T10:27:42.304 DEBUG:boto:Method: HEAD 2024-02-01T10:27:42.304 DEBUG:boto:Path: /vclfcn-19/dummy 2024-02-01T10:27:42.304 DEBUG:boto:Data: 2024-02-01T10:27:42.305 DEBUG:boto:Headers: {} 2024-02-01T10:27:42.305 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:42.305 DEBUG:boto:Port: 8001 2024-02-01T10:27:42.305 DEBUG:boto:Params: {} 2024-02-01T10:27:42.305 DEBUG:boto:Token: None 2024-02-01T10:27:42.305 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:42 GMT /vclfcn-19/dummy 2024-02-01T10:27:42.305 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ohIxawGK0+lqiPmXcG3DFW21AdA= 2024-02-01T10:27:42.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:42 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ohIxawGK0+lqiPmXcG3DFW21AdA=', 'Content-Length': '0'} 2024-02-01T10:27:42.309 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:23:11 GMT'), ('x-amz-version-id', 'g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000030388e5ae134794-0065bb721e-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:42.309 DEBUG:boto:path=/vclfcn-19/dummy 2024-02-01T10:27:42.309 DEBUG:boto:auth_path=/vclfcn-19/dummy 2024-02-01T10:27:42.310 DEBUG:boto:Method: HEAD 2024-02-01T10:27:42.310 DEBUG:boto:Path: /vclfcn-19/dummy 2024-02-01T10:27:42.310 DEBUG:boto:Data: 2024-02-01T10:27:42.310 DEBUG:boto:Headers: {} 2024-02-01T10:27:42.310 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:42.310 DEBUG:boto:Port: 8000 2024-02-01T10:27:42.310 DEBUG:boto:Params: {} 2024-02-01T10:27:42.310 DEBUG:boto:Token: None 2024-02-01T10:27:42.310 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:42 GMT /vclfcn-19/dummy 2024-02-01T10:27:42.310 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ohIxawGK0+lqiPmXcG3DFW21AdA= 2024-02-01T10:27:42.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:42 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ohIxawGK0+lqiPmXcG3DFW21AdA=', 'Content-Length': '0'} 2024-02-01T10:27:42.314 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:23:11 GMT'), ('x-amz-version-id', 'g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000017ead9b31749e0ec-0065bb721e-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:42.314 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-02-01T10:27:42.315 DEBUG:boto:path=/vclfcn-19/dummy 2024-02-01T10:27:42.315 DEBUG:boto:auth_path=/vclfcn-19/dummy 2024-02-01T10:27:42.315 DEBUG:boto:path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:42.315 DEBUG:boto:auth_path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:42.315 DEBUG:boto:Method: GET 2024-02-01T10:27:42.315 DEBUG:boto:Path: /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:42.315 DEBUG:boto:Data: 2024-02-01T10:27:42.315 DEBUG:boto:Headers: {} 2024-02-01T10:27:42.315 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:42.315 DEBUG:boto:Port: 8001 2024-02-01T10:27:42.315 DEBUG:boto:Params: {} 2024-02-01T10:27:42.316 DEBUG:boto:Token: None 2024-02-01T10:27:42.316 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:42 GMT /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:42.316 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:aHrkx15cEEsKbI5eldi2rz/bZeE= 2024-02-01T10:27:42.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:42 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:aHrkx15cEEsKbI5eldi2rz/bZeE=', 'Content-Length': '0'} 2024-02-01T10:27:42.318 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:23:11 GMT'), ('x-amz-version-id', 'g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d8fb7eadea89175a-0065bb721e-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:42.319 DEBUG:boto:path=/vclfcn-19/dummy 2024-02-01T10:27:42.319 DEBUG:boto:auth_path=/vclfcn-19/dummy 2024-02-01T10:27:42.319 DEBUG:boto:path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:42.319 DEBUG:boto:auth_path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:42.319 DEBUG:boto:Method: GET 2024-02-01T10:27:42.319 DEBUG:boto:Path: /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:42.319 DEBUG:boto:Data: 2024-02-01T10:27:42.319 DEBUG:boto:Headers: {} 2024-02-01T10:27:42.319 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:42.319 DEBUG:boto:Port: 8000 2024-02-01T10:27:42.320 DEBUG:boto:Params: {} 2024-02-01T10:27:42.320 DEBUG:boto:Token: None 2024-02-01T10:27:42.320 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:42 GMT /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:42.320 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:aHrkx15cEEsKbI5eldi2rz/bZeE= 2024-02-01T10:27:42.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:42 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:aHrkx15cEEsKbI5eldi2rz/bZeE=', 'Content-Length': '0'} 2024-02-01T10:27:42.322 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:23:11 GMT'), ('x-amz-version-id', 'g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a791b67f008476c8-0065bb721e-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:42.323 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:42.323 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:42.323 DEBUG:boto:path=/vclfcn-19/?prefix=dummy 2024-02-01T10:27:42.323 DEBUG:boto:auth_path=/vclfcn-19/?prefix=dummy 2024-02-01T10:27:42.323 DEBUG:boto:Method: GET 2024-02-01T10:27:42.323 DEBUG:boto:Path: /vclfcn-19/?prefix=dummy 2024-02-01T10:27:42.323 DEBUG:boto:Data: 2024-02-01T10:27:42.323 DEBUG:boto:Headers: {} 2024-02-01T10:27:42.324 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:42.324 DEBUG:boto:Port: 8001 2024-02-01T10:27:42.324 DEBUG:boto:Params: {} 2024-02-01T10:27:42.324 DEBUG:boto:Token: None 2024-02-01T10:27:42.324 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:42 GMT /vclfcn-19/ 2024-02-01T10:27:42.324 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:XISTLZllcTuoAfaN9hjt6JZUOL8= 2024-02-01T10:27:42.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:42 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:XISTLZllcTuoAfaN9hjt6JZUOL8=', 'Content-Length': '0'} 2024-02-01T10:27:42.328 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000036d6dc52ccdae48a-0065bb721e-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:42.329 DEBUG:boto:b'vclfcn-19dummy1000falsedummy2024-02-01T10:23:11.115Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:42.329 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:42.329 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:42.329 DEBUG:boto:path=/vclfcn-19/?prefix=dummy 2024-02-01T10:27:42.329 DEBUG:boto:auth_path=/vclfcn-19/?prefix=dummy 2024-02-01T10:27:42.330 DEBUG:boto:Method: GET 2024-02-01T10:27:42.330 DEBUG:boto:Path: /vclfcn-19/?prefix=dummy 2024-02-01T10:27:42.330 DEBUG:boto:Data: 2024-02-01T10:27:42.330 DEBUG:boto:Headers: {} 2024-02-01T10:27:42.330 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:42.330 DEBUG:boto:Port: 8000 2024-02-01T10:27:42.330 DEBUG:boto:Params: {} 2024-02-01T10:27:42.330 DEBUG:boto:Token: None 2024-02-01T10:27:42.330 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:42 GMT /vclfcn-19/ 2024-02-01T10:27:42.330 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:XISTLZllcTuoAfaN9hjt6JZUOL8= 2024-02-01T10:27:42.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:42 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:XISTLZllcTuoAfaN9hjt6JZUOL8=', 'Content-Length': '0'} 2024-02-01T10:27:42.335 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ebc9ab3c5af24cc-0065bb721e-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:42.335 DEBUG:boto:b'vclfcn-19dummy1000falsedummy2024-02-01T10:23:11.115Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:42.335 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:42.336 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:42.336 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:42.336 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:42.336 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:42.336 DEBUG:boto:Method: GET 2024-02-01T10:27:42.336 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:42.336 DEBUG:boto:Data: 2024-02-01T10:27:42.336 DEBUG:boto:Headers: {} 2024-02-01T10:27:42.336 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:42.336 DEBUG:boto:Port: 8001 2024-02-01T10:27:42.336 DEBUG:boto:Params: {} 2024-02-01T10:27:42.337 DEBUG:boto:Token: None 2024-02-01T10:27:42.337 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:42 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:42.337 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9mOx2rIcNaUErPIMhkIjlSow7sw= 2024-02-01T10:27:42.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:42 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9mOx2rIcNaUErPIMhkIjlSow7sw=', 'Content-Length': '0'} 2024-02-01T10:27:42.339 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000491bf10b6e6de169-0065bb721e-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:42.380 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:42.380 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:42.380 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:42.381 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:42.381 DEBUG:boto:Method: GET 2024-02-01T10:27:42.381 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:42.381 DEBUG:boto:Data: 2024-02-01T10:27:42.381 DEBUG:boto:Headers: {} 2024-02-01T10:27:42.381 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:42.381 DEBUG:boto:Port: 8000 2024-02-01T10:27:42.381 DEBUG:boto:Params: {} 2024-02-01T10:27:42.381 DEBUG:boto:Token: None 2024-02-01T10:27:42.381 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:42 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:42.382 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9mOx2rIcNaUErPIMhkIjlSow7sw= 2024-02-01T10:27:42.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:42 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9mOx2rIcNaUErPIMhkIjlSow7sw=', 'Content-Length': '0'} 2024-02-01T10:27:42.385 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009088e32d53d45373-0065bb721e-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:42.385 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:42.385 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:42.385 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:42.385 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:42.386 DEBUG:boto:Method: HEAD 2024-02-01T10:27:42.386 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:42.386 DEBUG:boto:Data: 2024-02-01T10:27:42.386 DEBUG:boto:Headers: {} 2024-02-01T10:27:42.386 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:42.386 DEBUG:boto:Port: 8001 2024-02-01T10:27:42.386 DEBUG:boto:Params: {} 2024-02-01T10:27:42.386 DEBUG:boto:Token: None 2024-02-01T10:27:42.386 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:42 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:42.387 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:4idCTJhSryv1JfS0Xqo0D4ORjWQ= 2024-02-01T10:27:42.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:42 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:4idCTJhSryv1JfS0Xqo0D4ORjWQ=', 'Content-Length': '0'} 2024-02-01T10:27:42.389 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fe1d0158cc16c584-0065bb721e-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:42.390 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:42.390 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:42.390 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:42.390 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:42.390 DEBUG:boto:Method: HEAD 2024-02-01T10:27:42.390 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:42.390 DEBUG:boto:Data: 2024-02-01T10:27:42.390 DEBUG:boto:Headers: {} 2024-02-01T10:27:42.390 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:42.391 DEBUG:boto:Port: 8000 2024-02-01T10:27:42.391 DEBUG:boto:Params: {} 2024-02-01T10:27:42.391 DEBUG:boto:Token: None 2024-02-01T10:27:42.391 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:42 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:42.391 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:4idCTJhSryv1JfS0Xqo0D4ORjWQ= 2024-02-01T10:27:42.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:42 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:4idCTJhSryv1JfS0Xqo0D4ORjWQ=', 'Content-Length': '0'} 2024-02-01T10:27:42.394 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fce95a060488ed4a-0065bb721e-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:42.394 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:42.394 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:42.394 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:42.395 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:42.395 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:42.395 DEBUG:boto:Method: GET 2024-02-01T10:27:42.395 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:42.395 DEBUG:boto:Data: 2024-02-01T10:27:42.395 DEBUG:boto:Headers: {} 2024-02-01T10:27:42.395 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:42.395 DEBUG:boto:Port: 8001 2024-02-01T10:27:42.395 DEBUG:boto:Params: {} 2024-02-01T10:27:42.395 DEBUG:boto:Token: None 2024-02-01T10:27:42.396 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:42 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:42.396 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9mOx2rIcNaUErPIMhkIjlSow7sw= 2024-02-01T10:27:42.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:42 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9mOx2rIcNaUErPIMhkIjlSow7sw=', 'Content-Length': '0'} 2024-02-01T10:27:42.399 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f2608c39f7ae0184-0065bb721e-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:42.399 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:42.399 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:42.399 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:42.399 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:42.400 DEBUG:boto:Method: GET 2024-02-01T10:27:42.400 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:42.400 DEBUG:boto:Data: 2024-02-01T10:27:42.400 DEBUG:boto:Headers: {} 2024-02-01T10:27:42.400 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:42.400 DEBUG:boto:Port: 8000 2024-02-01T10:27:42.400 DEBUG:boto:Params: {} 2024-02-01T10:27:42.400 DEBUG:boto:Token: None 2024-02-01T10:27:42.400 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:42 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:42.401 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9mOx2rIcNaUErPIMhkIjlSow7sw= 2024-02-01T10:27:42.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:42 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9mOx2rIcNaUErPIMhkIjlSow7sw=', 'Content-Length': '0'} 2024-02-01T10:27:42.403 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000009752dd4b2534846-0065bb721e-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:42.444 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:42.444 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:42.445 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:42.445 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:42.445 DEBUG:boto:Method: GET 2024-02-01T10:27:42.445 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:42.445 DEBUG:boto:Data: 2024-02-01T10:27:42.445 DEBUG:boto:Headers: {} 2024-02-01T10:27:42.445 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:42.445 DEBUG:boto:Port: 8001 2024-02-01T10:27:42.445 DEBUG:boto:Params: {} 2024-02-01T10:27:42.446 DEBUG:boto:Token: None 2024-02-01T10:27:42.446 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:42 GMT /vclfcn-19/ 2024-02-01T10:27:42.446 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:XISTLZllcTuoAfaN9hjt6JZUOL8= 2024-02-01T10:27:42.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:42 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:XISTLZllcTuoAfaN9hjt6JZUOL8=', 'Content-Length': '0'} 2024-02-01T10:27:42.451 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007cede5d6d9633d89-0065bb721e-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:42.451 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:42.452 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:42.452 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:42.452 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:42.452 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:42.452 DEBUG:boto:Method: GET 2024-02-01T10:27:42.452 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:42.452 DEBUG:boto:Data: 2024-02-01T10:27:42.452 DEBUG:boto:Headers: {} 2024-02-01T10:27:42.452 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:42.453 DEBUG:boto:Port: 8000 2024-02-01T10:27:42.453 DEBUG:boto:Params: {} 2024-02-01T10:27:42.453 DEBUG:boto:Token: None 2024-02-01T10:27:42.453 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:42 GMT /vclfcn-19/ 2024-02-01T10:27:42.453 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:XISTLZllcTuoAfaN9hjt6JZUOL8= 2024-02-01T10:27:42.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:42 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:XISTLZllcTuoAfaN9hjt6JZUOL8=', 'Content-Length': '0'} 2024-02-01T10:27:42.458 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008047f9d3e9a7c0c2-0065bb721e-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:42.458 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:42.459 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:42.459 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:42.459 DEBUG:boto:Method: HEAD 2024-02-01T10:27:42.459 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:42.459 DEBUG:boto:Data: 2024-02-01T10:27:42.459 DEBUG:boto:Headers: {} 2024-02-01T10:27:42.459 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:42.459 DEBUG:boto:Port: 8001 2024-02-01T10:27:42.460 DEBUG:boto:Params: {} 2024-02-01T10:27:42.460 DEBUG:boto:Token: None 2024-02-01T10:27:42.460 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:42 GMT /vclfcn-19/obj 2024-02-01T10:27:42.460 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7B88XlrTjoEE86/ans2gBmjx5lM= 2024-02-01T10:27:42.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:42 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7B88XlrTjoEE86/ans2gBmjx5lM=', 'Content-Length': '0'} 2024-02-01T10:27:42.466 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008787bd0ea2fd6840-0065bb721e-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:42.467 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:42.467 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:42.467 DEBUG:boto:Method: HEAD 2024-02-01T10:27:42.467 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:42.467 DEBUG:boto:Data: 2024-02-01T10:27:42.467 DEBUG:boto:Headers: {} 2024-02-01T10:27:42.467 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:42.467 DEBUG:boto:Port: 8000 2024-02-01T10:27:42.468 DEBUG:boto:Params: {} 2024-02-01T10:27:42.468 DEBUG:boto:Token: None 2024-02-01T10:27:42.468 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:42 GMT /vclfcn-19/obj 2024-02-01T10:27:42.468 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7B88XlrTjoEE86/ans2gBmjx5lM= 2024-02-01T10:27:42.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:42 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7B88XlrTjoEE86/ans2gBmjx5lM=', 'Content-Length': '0'} 2024-02-01T10:27:42.474 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a1246470b3ed4c7f-0065bb721e-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:42.474 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:42.475 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:42.475 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:42.475 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:42.475 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:42.475 DEBUG:boto:Method: GET 2024-02-01T10:27:42.475 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:42.476 DEBUG:boto:Data: 2024-02-01T10:27:42.476 DEBUG:boto:Headers: {} 2024-02-01T10:27:42.476 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:42.476 DEBUG:boto:Port: 8001 2024-02-01T10:27:42.476 DEBUG:boto:Params: {} 2024-02-01T10:27:42.476 DEBUG:boto:Token: None 2024-02-01T10:27:42.476 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:42 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:42.476 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9mOx2rIcNaUErPIMhkIjlSow7sw= 2024-02-01T10:27:42.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:42 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9mOx2rIcNaUErPIMhkIjlSow7sw=', 'Content-Length': '0'} 2024-02-01T10:27:42.479 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006f2c86276b86b832-0065bb721e-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:42.520 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:42.520 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:42.520 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:42.521 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:42.521 DEBUG:boto:Method: GET 2024-02-01T10:27:42.521 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:42.521 DEBUG:boto:Data: 2024-02-01T10:27:42.521 DEBUG:boto:Headers: {} 2024-02-01T10:27:42.521 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:42.521 DEBUG:boto:Port: 8000 2024-02-01T10:27:42.521 DEBUG:boto:Params: {} 2024-02-01T10:27:42.521 DEBUG:boto:Token: None 2024-02-01T10:27:42.521 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:42 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:42.522 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9mOx2rIcNaUErPIMhkIjlSow7sw= 2024-02-01T10:27:42.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:42 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9mOx2rIcNaUErPIMhkIjlSow7sw=', 'Content-Length': '0'} 2024-02-01T10:27:42.525 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000077ef170183e42e32-0065bb721e-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:42.568 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:42.568 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:42.568 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:42.569 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:42.569 DEBUG:boto:Method: GET 2024-02-01T10:27:42.569 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:42.569 DEBUG:boto:Data: 2024-02-01T10:27:42.569 DEBUG:boto:Headers: {} 2024-02-01T10:27:42.569 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:42.569 DEBUG:boto:Port: 8001 2024-02-01T10:27:42.569 DEBUG:boto:Params: {} 2024-02-01T10:27:42.569 DEBUG:boto:Token: None 2024-02-01T10:27:42.570 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:42 GMT /vclfcn-19/ 2024-02-01T10:27:42.570 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:XISTLZllcTuoAfaN9hjt6JZUOL8= 2024-02-01T10:27:42.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:42 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:XISTLZllcTuoAfaN9hjt6JZUOL8=', 'Content-Length': '0'} 2024-02-01T10:27:42.575 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000006839d3d3b81f2f6-0065bb721e-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:42.575 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:42.575 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:42.576 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:42.576 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:42.576 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:42.576 DEBUG:boto:Method: GET 2024-02-01T10:27:42.576 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:42.576 DEBUG:boto:Data: 2024-02-01T10:27:42.576 DEBUG:boto:Headers: {} 2024-02-01T10:27:42.576 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:42.576 DEBUG:boto:Port: 8000 2024-02-01T10:27:42.576 DEBUG:boto:Params: {} 2024-02-01T10:27:42.577 DEBUG:boto:Token: None 2024-02-01T10:27:42.577 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:42 GMT /vclfcn-19/ 2024-02-01T10:27:42.577 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:XISTLZllcTuoAfaN9hjt6JZUOL8= 2024-02-01T10:27:42.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:42 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:XISTLZllcTuoAfaN9hjt6JZUOL8=', 'Content-Length': '0'} 2024-02-01T10:27:42.582 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000448219d160d2b647-0065bb721e-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:42.582 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:42.583 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:42.583 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:42.583 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:42.583 DEBUG:boto:path=/vclfcn-19/obj?versionId=RrytQti15jy37pft0WxtgWdt4Zs7l4E 2024-02-01T10:27:42.583 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=RrytQti15jy37pft0WxtgWdt4Zs7l4E 2024-02-01T10:27:42.583 DEBUG:boto:Method: GET 2024-02-01T10:27:42.583 DEBUG:boto:Path: /vclfcn-19/obj?versionId=RrytQti15jy37pft0WxtgWdt4Zs7l4E 2024-02-01T10:27:42.583 DEBUG:boto:Data: 2024-02-01T10:27:42.583 DEBUG:boto:Headers: {} 2024-02-01T10:27:42.584 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:42.584 DEBUG:boto:Port: 8001 2024-02-01T10:27:42.584 DEBUG:boto:Params: {} 2024-02-01T10:27:42.584 DEBUG:boto:Token: None 2024-02-01T10:27:42.584 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:42 GMT /vclfcn-19/obj?versionId=RrytQti15jy37pft0WxtgWdt4Zs7l4E 2024-02-01T10:27:42.584 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GDiTCqTKS0+qA7Wsza+9ykbZnos= 2024-02-01T10:27:42.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:42 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GDiTCqTKS0+qA7Wsza+9ykbZnos=', 'Content-Length': '0'} 2024-02-01T10:27:42.587 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'RrytQti15jy37pft0WxtgWdt4Zs7l4E'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000091267bc7a6793e98-0065bb721e-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:42.587 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:42.588 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:42.588 DEBUG:boto:path=/vclfcn-19/obj?versionId=RrytQti15jy37pft0WxtgWdt4Zs7l4E 2024-02-01T10:27:42.588 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=RrytQti15jy37pft0WxtgWdt4Zs7l4E 2024-02-01T10:27:42.588 DEBUG:boto:Method: GET 2024-02-01T10:27:42.588 DEBUG:boto:Path: /vclfcn-19/obj?versionId=RrytQti15jy37pft0WxtgWdt4Zs7l4E 2024-02-01T10:27:42.588 DEBUG:boto:Data: 2024-02-01T10:27:42.588 DEBUG:boto:Headers: {} 2024-02-01T10:27:42.588 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:42.588 DEBUG:boto:Port: 8000 2024-02-01T10:27:42.588 DEBUG:boto:Params: {} 2024-02-01T10:27:42.588 DEBUG:boto:Token: None 2024-02-01T10:27:42.589 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:42 GMT /vclfcn-19/obj?versionId=RrytQti15jy37pft0WxtgWdt4Zs7l4E 2024-02-01T10:27:42.589 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GDiTCqTKS0+qA7Wsza+9ykbZnos= 2024-02-01T10:27:42.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:42 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GDiTCqTKS0+qA7Wsza+9ykbZnos=', 'Content-Length': '0'} 2024-02-01T10:27:42.591 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'RrytQti15jy37pft0WxtgWdt4Zs7l4E'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b79f2e3c88af1c86-0065bb721e-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:42.632 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:42.632 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:42.632 DEBUG:boto:path=/vclfcn-19/obj?versionId=RrytQti15jy37pft0WxtgWdt4Zs7l4E 2024-02-01T10:27:42.633 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=RrytQti15jy37pft0WxtgWdt4Zs7l4E 2024-02-01T10:27:42.633 DEBUG:boto:Method: HEAD 2024-02-01T10:27:42.633 DEBUG:boto:Path: /vclfcn-19/obj?versionId=RrytQti15jy37pft0WxtgWdt4Zs7l4E 2024-02-01T10:27:42.633 DEBUG:boto:Data: 2024-02-01T10:27:42.633 DEBUG:boto:Headers: {} 2024-02-01T10:27:42.633 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:42.633 DEBUG:boto:Port: 8001 2024-02-01T10:27:42.635 DEBUG:boto:Params: {} 2024-02-01T10:27:42.635 DEBUG:boto:Token: None 2024-02-01T10:27:42.635 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:42 GMT /vclfcn-19/obj?versionId=RrytQti15jy37pft0WxtgWdt4Zs7l4E 2024-02-01T10:27:42.636 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:T7JAC525dOpI0EEkT7MXY8Fyhvk= 2024-02-01T10:27:42.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:42 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:T7JAC525dOpI0EEkT7MXY8Fyhvk=', 'Content-Length': '0'} 2024-02-01T10:27:42.638 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'RrytQti15jy37pft0WxtgWdt4Zs7l4E'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000057d063310502b05-0065bb721e-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:42.639 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:42.639 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:42.639 DEBUG:boto:path=/vclfcn-19/obj?versionId=RrytQti15jy37pft0WxtgWdt4Zs7l4E 2024-02-01T10:27:42.639 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=RrytQti15jy37pft0WxtgWdt4Zs7l4E 2024-02-01T10:27:42.639 DEBUG:boto:Method: HEAD 2024-02-01T10:27:42.639 DEBUG:boto:Path: /vclfcn-19/obj?versionId=RrytQti15jy37pft0WxtgWdt4Zs7l4E 2024-02-01T10:27:42.639 DEBUG:boto:Data: 2024-02-01T10:27:42.639 DEBUG:boto:Headers: {} 2024-02-01T10:27:42.639 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:42.639 DEBUG:boto:Port: 8000 2024-02-01T10:27:42.640 DEBUG:boto:Params: {} 2024-02-01T10:27:42.640 DEBUG:boto:Token: None 2024-02-01T10:27:42.640 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:42 GMT /vclfcn-19/obj?versionId=RrytQti15jy37pft0WxtgWdt4Zs7l4E 2024-02-01T10:27:42.640 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:T7JAC525dOpI0EEkT7MXY8Fyhvk= 2024-02-01T10:27:42.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:42 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:T7JAC525dOpI0EEkT7MXY8Fyhvk=', 'Content-Length': '0'} 2024-02-01T10:27:42.643 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'RrytQti15jy37pft0WxtgWdt4Zs7l4E'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a28234059a448df8-0065bb721e-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:42.644 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:42.644 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:42.644 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:42.644 DEBUG:boto:path=/vclfcn-19/obj?versionId=RrytQti15jy37pft0WxtgWdt4Zs7l4E 2024-02-01T10:27:42.644 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=RrytQti15jy37pft0WxtgWdt4Zs7l4E 2024-02-01T10:27:42.644 DEBUG:boto:Method: GET 2024-02-01T10:27:42.644 DEBUG:boto:Path: /vclfcn-19/obj?versionId=RrytQti15jy37pft0WxtgWdt4Zs7l4E 2024-02-01T10:27:42.645 DEBUG:boto:Data: 2024-02-01T10:27:42.645 DEBUG:boto:Headers: {} 2024-02-01T10:27:42.645 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:42.645 DEBUG:boto:Port: 8001 2024-02-01T10:27:42.645 DEBUG:boto:Params: {} 2024-02-01T10:27:42.645 DEBUG:boto:Token: None 2024-02-01T10:27:42.645 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:42 GMT /vclfcn-19/obj?versionId=RrytQti15jy37pft0WxtgWdt4Zs7l4E 2024-02-01T10:27:42.646 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GDiTCqTKS0+qA7Wsza+9ykbZnos= 2024-02-01T10:27:42.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:42 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GDiTCqTKS0+qA7Wsza+9ykbZnos=', 'Content-Length': '0'} 2024-02-01T10:27:42.648 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'RrytQti15jy37pft0WxtgWdt4Zs7l4E'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000792cc430ef578db3-0065bb721e-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:42.649 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:42.649 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:42.649 DEBUG:boto:path=/vclfcn-19/obj?versionId=RrytQti15jy37pft0WxtgWdt4Zs7l4E 2024-02-01T10:27:42.649 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=RrytQti15jy37pft0WxtgWdt4Zs7l4E 2024-02-01T10:27:42.649 DEBUG:boto:Method: GET 2024-02-01T10:27:42.649 DEBUG:boto:Path: /vclfcn-19/obj?versionId=RrytQti15jy37pft0WxtgWdt4Zs7l4E 2024-02-01T10:27:42.649 DEBUG:boto:Data: 2024-02-01T10:27:42.650 DEBUG:boto:Headers: {} 2024-02-01T10:27:42.650 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:42.650 DEBUG:boto:Port: 8000 2024-02-01T10:27:42.650 DEBUG:boto:Params: {} 2024-02-01T10:27:42.650 DEBUG:boto:Token: None 2024-02-01T10:27:42.650 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:42 GMT /vclfcn-19/obj?versionId=RrytQti15jy37pft0WxtgWdt4Zs7l4E 2024-02-01T10:27:42.650 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GDiTCqTKS0+qA7Wsza+9ykbZnos= 2024-02-01T10:27:42.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:42 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GDiTCqTKS0+qA7Wsza+9ykbZnos=', 'Content-Length': '0'} 2024-02-01T10:27:42.653 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'RrytQti15jy37pft0WxtgWdt4Zs7l4E'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000339337bb3e15927-0065bb721e-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:42.700 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:42.700 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:42.700 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:42.700 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:42.701 DEBUG:boto:Method: GET 2024-02-01T10:27:42.701 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:42.701 DEBUG:boto:Data: 2024-02-01T10:27:42.701 DEBUG:boto:Headers: {} 2024-02-01T10:27:42.701 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:42.701 DEBUG:boto:Port: 8001 2024-02-01T10:27:42.701 DEBUG:boto:Params: {} 2024-02-01T10:27:42.701 DEBUG:boto:Token: None 2024-02-01T10:27:42.701 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:42 GMT /vclfcn-19/ 2024-02-01T10:27:42.701 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:XISTLZllcTuoAfaN9hjt6JZUOL8= 2024-02-01T10:27:42.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:42 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:XISTLZllcTuoAfaN9hjt6JZUOL8=', 'Content-Length': '0'} 2024-02-01T10:27:42.707 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb9f7928571c9371-0065bb721e-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:42.708 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:42.708 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:42.708 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:42.708 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:42.709 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:42.709 DEBUG:boto:Method: GET 2024-02-01T10:27:42.709 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:42.709 DEBUG:boto:Data: 2024-02-01T10:27:42.709 DEBUG:boto:Headers: {} 2024-02-01T10:27:42.709 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:42.709 DEBUG:boto:Port: 8000 2024-02-01T10:27:42.709 DEBUG:boto:Params: {} 2024-02-01T10:27:42.709 DEBUG:boto:Token: None 2024-02-01T10:27:42.709 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:42 GMT /vclfcn-19/ 2024-02-01T10:27:42.710 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:XISTLZllcTuoAfaN9hjt6JZUOL8= 2024-02-01T10:27:42.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:42 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:XISTLZllcTuoAfaN9hjt6JZUOL8=', 'Content-Length': '0'} 2024-02-01T10:27:42.716 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000341888529cda5814-0065bb721e-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:42.716 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:42.717 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:42.718 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:42.718 DEBUG:boto:Method: HEAD 2024-02-01T10:27:42.718 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:42.718 DEBUG:boto:Data: 2024-02-01T10:27:42.718 DEBUG:boto:Headers: {} 2024-02-01T10:27:42.718 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:42.718 DEBUG:boto:Port: 8001 2024-02-01T10:27:42.718 DEBUG:boto:Params: {} 2024-02-01T10:27:42.718 DEBUG:boto:Token: None 2024-02-01T10:27:42.719 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:42 GMT /vclfcn-19/obj 2024-02-01T10:27:42.719 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7B88XlrTjoEE86/ans2gBmjx5lM= 2024-02-01T10:27:42.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:42 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7B88XlrTjoEE86/ans2gBmjx5lM=', 'Content-Length': '0'} 2024-02-01T10:27:42.725 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000076defd6a78248275-0065bb721e-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:42.726 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:42.726 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:42.726 DEBUG:boto:Method: HEAD 2024-02-01T10:27:42.726 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:42.726 DEBUG:boto:Data: 2024-02-01T10:27:42.726 DEBUG:boto:Headers: {} 2024-02-01T10:27:42.726 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:42.726 DEBUG:boto:Port: 8000 2024-02-01T10:27:42.726 DEBUG:boto:Params: {} 2024-02-01T10:27:42.727 DEBUG:boto:Token: None 2024-02-01T10:27:42.727 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:42 GMT /vclfcn-19/obj 2024-02-01T10:27:42.727 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7B88XlrTjoEE86/ans2gBmjx5lM= 2024-02-01T10:27:42.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:42 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7B88XlrTjoEE86/ans2gBmjx5lM=', 'Content-Length': '0'} 2024-02-01T10:27:42.732 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bc43b84412e11f9a-0065bb721e-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:42.733 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:42.733 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:42.733 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:42.733 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:42.733 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:42.733 DEBUG:boto:Method: GET 2024-02-01T10:27:42.733 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:42.733 DEBUG:boto:Data: 2024-02-01T10:27:42.733 DEBUG:boto:Headers: {} 2024-02-01T10:27:42.734 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:42.734 DEBUG:boto:Port: 8001 2024-02-01T10:27:42.734 DEBUG:boto:Params: {} 2024-02-01T10:27:42.734 DEBUG:boto:Token: None 2024-02-01T10:27:42.734 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:42 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:42.734 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9mOx2rIcNaUErPIMhkIjlSow7sw= 2024-02-01T10:27:42.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:42 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9mOx2rIcNaUErPIMhkIjlSow7sw=', 'Content-Length': '0'} 2024-02-01T10:27:42.737 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003f30d4d3c106c565-0065bb721e-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:42.780 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:42.780 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:42.780 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:42.781 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:42.781 DEBUG:boto:Method: GET 2024-02-01T10:27:42.781 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:42.781 DEBUG:boto:Data: 2024-02-01T10:27:42.781 DEBUG:boto:Headers: {} 2024-02-01T10:27:42.781 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:42.781 DEBUG:boto:Port: 8000 2024-02-01T10:27:42.781 DEBUG:boto:Params: {} 2024-02-01T10:27:42.781 DEBUG:boto:Token: None 2024-02-01T10:27:42.781 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:42 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:42.782 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9mOx2rIcNaUErPIMhkIjlSow7sw= 2024-02-01T10:27:42.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:42 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9mOx2rIcNaUErPIMhkIjlSow7sw=', 'Content-Length': '0'} 2024-02-01T10:27:42.785 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d95c259ffbe72fb1-0065bb721e-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:42.832 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:42.832 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:42.832 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:42.832 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:42.833 DEBUG:boto:Method: GET 2024-02-01T10:27:42.833 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:42.833 DEBUG:boto:Data: 2024-02-01T10:27:42.833 DEBUG:boto:Headers: {} 2024-02-01T10:27:42.833 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:42.833 DEBUG:boto:Port: 8001 2024-02-01T10:27:42.833 DEBUG:boto:Params: {} 2024-02-01T10:27:42.833 DEBUG:boto:Token: None 2024-02-01T10:27:42.833 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:42 GMT /vclfcn-19/ 2024-02-01T10:27:42.834 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:XISTLZllcTuoAfaN9hjt6JZUOL8= 2024-02-01T10:27:42.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:42 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:XISTLZllcTuoAfaN9hjt6JZUOL8=', 'Content-Length': '0'} 2024-02-01T10:27:42.839 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000584e7726ada8c773-0065bb721e-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:42.839 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:42.840 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:42.840 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:42.840 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:42.840 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:42.840 DEBUG:boto:Method: GET 2024-02-01T10:27:42.840 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:42.840 DEBUG:boto:Data: 2024-02-01T10:27:42.840 DEBUG:boto:Headers: {} 2024-02-01T10:27:42.840 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:42.840 DEBUG:boto:Port: 8000 2024-02-01T10:27:42.840 DEBUG:boto:Params: {} 2024-02-01T10:27:42.841 DEBUG:boto:Token: None 2024-02-01T10:27:42.841 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:42 GMT /vclfcn-19/ 2024-02-01T10:27:42.841 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:XISTLZllcTuoAfaN9hjt6JZUOL8= 2024-02-01T10:27:42.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:42 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:XISTLZllcTuoAfaN9hjt6JZUOL8=', 'Content-Length': '0'} 2024-02-01T10:27:42.846 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed47a64cc0d2fd22-0065bb721e-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:42.847 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:42.847 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:42.847 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:42.847 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:42.847 DEBUG:boto:path=/vclfcn-19/obj?versionId=fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u 2024-02-01T10:27:42.848 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u 2024-02-01T10:27:42.848 DEBUG:boto:Method: GET 2024-02-01T10:27:42.848 DEBUG:boto:Path: /vclfcn-19/obj?versionId=fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u 2024-02-01T10:27:42.848 DEBUG:boto:Data: 2024-02-01T10:27:42.849 DEBUG:boto:Headers: {} 2024-02-01T10:27:42.849 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:42.849 DEBUG:boto:Port: 8001 2024-02-01T10:27:42.849 DEBUG:boto:Params: {} 2024-02-01T10:27:42.849 DEBUG:boto:Token: None 2024-02-01T10:27:42.849 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:42 GMT /vclfcn-19/obj?versionId=fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u 2024-02-01T10:27:42.849 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:aVBSiN28k1ncpx4rUbvSDfUjNqM= 2024-02-01T10:27:42.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:42 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:aVBSiN28k1ncpx4rUbvSDfUjNqM=', 'Content-Length': '0'} 2024-02-01T10:27:42.852 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a8807900f3113751-0065bb721e-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:42.852 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:42.853 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:42.853 DEBUG:boto:path=/vclfcn-19/obj?versionId=fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u 2024-02-01T10:27:42.853 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u 2024-02-01T10:27:42.853 DEBUG:boto:Method: GET 2024-02-01T10:27:42.853 DEBUG:boto:Path: /vclfcn-19/obj?versionId=fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u 2024-02-01T10:27:42.853 DEBUG:boto:Data: 2024-02-01T10:27:42.853 DEBUG:boto:Headers: {} 2024-02-01T10:27:42.853 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:42.853 DEBUG:boto:Port: 8000 2024-02-01T10:27:42.853 DEBUG:boto:Params: {} 2024-02-01T10:27:42.853 DEBUG:boto:Token: None 2024-02-01T10:27:42.854 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:42 GMT /vclfcn-19/obj?versionId=fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u 2024-02-01T10:27:42.854 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:aVBSiN28k1ncpx4rUbvSDfUjNqM= 2024-02-01T10:27:42.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:42 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:aVBSiN28k1ncpx4rUbvSDfUjNqM=', 'Content-Length': '0'} 2024-02-01T10:27:42.857 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000043c6aa6840a1c62d-0065bb721e-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:42.858 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:42.858 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:42.858 DEBUG:boto:path=/vclfcn-19/obj?versionId=fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u 2024-02-01T10:27:42.858 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u 2024-02-01T10:27:42.858 DEBUG:boto:Method: HEAD 2024-02-01T10:27:42.858 DEBUG:boto:Path: /vclfcn-19/obj?versionId=fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u 2024-02-01T10:27:42.858 DEBUG:boto:Data: 2024-02-01T10:27:42.859 DEBUG:boto:Headers: {} 2024-02-01T10:27:42.859 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:42.859 DEBUG:boto:Port: 8001 2024-02-01T10:27:42.859 DEBUG:boto:Params: {} 2024-02-01T10:27:42.859 DEBUG:boto:Token: None 2024-02-01T10:27:42.859 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:42 GMT /vclfcn-19/obj?versionId=fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u 2024-02-01T10:27:42.859 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:yKo8BimGLUOCHCyYlQxC+txSSPc= 2024-02-01T10:27:42.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:42 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:yKo8BimGLUOCHCyYlQxC+txSSPc=', 'Content-Length': '0'} 2024-02-01T10:27:42.862 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005b5dad5d58cc59d7-0065bb721e-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:42.862 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:42.863 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:42.863 DEBUG:boto:path=/vclfcn-19/obj?versionId=fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u 2024-02-01T10:27:42.863 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u 2024-02-01T10:27:42.863 DEBUG:boto:Method: HEAD 2024-02-01T10:27:42.863 DEBUG:boto:Path: /vclfcn-19/obj?versionId=fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u 2024-02-01T10:27:42.863 DEBUG:boto:Data: 2024-02-01T10:27:42.863 DEBUG:boto:Headers: {} 2024-02-01T10:27:42.863 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:42.863 DEBUG:boto:Port: 8000 2024-02-01T10:27:42.863 DEBUG:boto:Params: {} 2024-02-01T10:27:42.864 DEBUG:boto:Token: None 2024-02-01T10:27:42.864 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:42 GMT /vclfcn-19/obj?versionId=fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u 2024-02-01T10:27:42.864 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:yKo8BimGLUOCHCyYlQxC+txSSPc= 2024-02-01T10:27:42.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:42 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:yKo8BimGLUOCHCyYlQxC+txSSPc=', 'Content-Length': '0'} 2024-02-01T10:27:42.867 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f22eaf246522189a-0065bb721e-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:42.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:42.868 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:42.868 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:42.868 DEBUG:boto:path=/vclfcn-19/obj?versionId=fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u 2024-02-01T10:27:42.868 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u 2024-02-01T10:27:42.868 DEBUG:boto:Method: GET 2024-02-01T10:27:42.868 DEBUG:boto:Path: /vclfcn-19/obj?versionId=fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u 2024-02-01T10:27:42.868 DEBUG:boto:Data: 2024-02-01T10:27:42.868 DEBUG:boto:Headers: {} 2024-02-01T10:27:42.868 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:42.868 DEBUG:boto:Port: 8001 2024-02-01T10:27:42.869 DEBUG:boto:Params: {} 2024-02-01T10:27:42.869 DEBUG:boto:Token: None 2024-02-01T10:27:42.869 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:42 GMT /vclfcn-19/obj?versionId=fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u 2024-02-01T10:27:42.869 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:aVBSiN28k1ncpx4rUbvSDfUjNqM= 2024-02-01T10:27:42.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:42 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:aVBSiN28k1ncpx4rUbvSDfUjNqM=', 'Content-Length': '0'} 2024-02-01T10:27:42.872 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a177eb9e0a3ea87c-0065bb721e-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:42.872 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:42.872 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:42.872 DEBUG:boto:path=/vclfcn-19/obj?versionId=fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u 2024-02-01T10:27:42.872 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u 2024-02-01T10:27:42.873 DEBUG:boto:Method: GET 2024-02-01T10:27:42.873 DEBUG:boto:Path: /vclfcn-19/obj?versionId=fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u 2024-02-01T10:27:42.873 DEBUG:boto:Data: 2024-02-01T10:27:42.873 DEBUG:boto:Headers: {} 2024-02-01T10:27:42.873 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:42.873 DEBUG:boto:Port: 8000 2024-02-01T10:27:42.873 DEBUG:boto:Params: {} 2024-02-01T10:27:42.873 DEBUG:boto:Token: None 2024-02-01T10:27:42.873 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:42 GMT /vclfcn-19/obj?versionId=fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u 2024-02-01T10:27:42.873 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:aVBSiN28k1ncpx4rUbvSDfUjNqM= 2024-02-01T10:27:42.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:42 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:aVBSiN28k1ncpx4rUbvSDfUjNqM=', 'Content-Length': '0'} 2024-02-01T10:27:42.876 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000747683f24fca853-0065bb721e-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:42.920 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:42.920 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:42.920 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:42.921 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:42.921 DEBUG:boto:Method: GET 2024-02-01T10:27:42.921 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:42.921 DEBUG:boto:Data: 2024-02-01T10:27:42.921 DEBUG:boto:Headers: {} 2024-02-01T10:27:42.921 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:42.921 DEBUG:boto:Port: 8001 2024-02-01T10:27:42.921 DEBUG:boto:Params: {} 2024-02-01T10:27:42.921 DEBUG:boto:Token: None 2024-02-01T10:27:42.921 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:42 GMT /vclfcn-19/ 2024-02-01T10:27:42.922 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:XISTLZllcTuoAfaN9hjt6JZUOL8= 2024-02-01T10:27:42.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:42 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:XISTLZllcTuoAfaN9hjt6JZUOL8=', 'Content-Length': '0'} 2024-02-01T10:27:42.926 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7dbc8161ceee8bb-0065bb721e-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:42.926 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:42.927 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:42.927 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:42.927 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:42.927 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:42.927 DEBUG:boto:Method: GET 2024-02-01T10:27:42.927 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:42.927 DEBUG:boto:Data: 2024-02-01T10:27:42.928 DEBUG:boto:Headers: {} 2024-02-01T10:27:42.928 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:42.928 DEBUG:boto:Port: 8000 2024-02-01T10:27:42.928 DEBUG:boto:Params: {} 2024-02-01T10:27:42.928 DEBUG:boto:Token: None 2024-02-01T10:27:42.928 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:42 GMT /vclfcn-19/ 2024-02-01T10:27:42.928 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:XISTLZllcTuoAfaN9hjt6JZUOL8= 2024-02-01T10:27:42.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:42 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:XISTLZllcTuoAfaN9hjt6JZUOL8=', 'Content-Length': '0'} 2024-02-01T10:27:42.933 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000036058b8e7404fd66-0065bb721e-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:42.933 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:42.934 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:42.934 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:42.934 DEBUG:boto:Method: HEAD 2024-02-01T10:27:42.934 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:42.934 DEBUG:boto:Data: 2024-02-01T10:27:42.934 DEBUG:boto:Headers: {} 2024-02-01T10:27:42.934 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:42.934 DEBUG:boto:Port: 8001 2024-02-01T10:27:42.935 DEBUG:boto:Params: {} 2024-02-01T10:27:42.935 DEBUG:boto:Token: None 2024-02-01T10:27:42.935 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:42 GMT /vclfcn-19/obj 2024-02-01T10:27:42.935 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7B88XlrTjoEE86/ans2gBmjx5lM= 2024-02-01T10:27:42.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:42 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7B88XlrTjoEE86/ans2gBmjx5lM=', 'Content-Length': '0'} 2024-02-01T10:27:42.940 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eff4c5389db0b419-0065bb721e-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:42.941 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:42.941 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:42.941 DEBUG:boto:Method: HEAD 2024-02-01T10:27:42.941 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:42.941 DEBUG:boto:Data: 2024-02-01T10:27:42.941 DEBUG:boto:Headers: {} 2024-02-01T10:27:42.941 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:42.942 DEBUG:boto:Port: 8000 2024-02-01T10:27:42.942 DEBUG:boto:Params: {} 2024-02-01T10:27:42.942 DEBUG:boto:Token: None 2024-02-01T10:27:42.942 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:42 GMT /vclfcn-19/obj 2024-02-01T10:27:42.942 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7B88XlrTjoEE86/ans2gBmjx5lM= 2024-02-01T10:27:42.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:42 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7B88XlrTjoEE86/ans2gBmjx5lM=', 'Content-Length': '0'} 2024-02-01T10:27:42.948 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b96aec564fdb5fe2-0065bb721e-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:42.948 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:42.948 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:42.948 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:42.948 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:42.949 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:42.949 DEBUG:boto:Method: GET 2024-02-01T10:27:42.949 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:42.949 DEBUG:boto:Data: 2024-02-01T10:27:42.949 DEBUG:boto:Headers: {} 2024-02-01T10:27:42.949 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:42.949 DEBUG:boto:Port: 8001 2024-02-01T10:27:42.949 DEBUG:boto:Params: {} 2024-02-01T10:27:42.949 DEBUG:boto:Token: None 2024-02-01T10:27:42.949 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:42 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:42.950 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9mOx2rIcNaUErPIMhkIjlSow7sw= 2024-02-01T10:27:42.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:42 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9mOx2rIcNaUErPIMhkIjlSow7sw=', 'Content-Length': '0'} 2024-02-01T10:27:42.952 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f896ed8717308f25-0065bb721e-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:42.996 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:42.996 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:42.997 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:42.997 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:42.997 DEBUG:boto:Method: GET 2024-02-01T10:27:42.997 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:42.997 DEBUG:boto:Data: 2024-02-01T10:27:42.997 DEBUG:boto:Headers: {} 2024-02-01T10:27:42.997 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:42.998 DEBUG:boto:Port: 8000 2024-02-01T10:27:42.998 DEBUG:boto:Params: {} 2024-02-01T10:27:42.998 DEBUG:boto:Token: None 2024-02-01T10:27:42.998 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:42 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:42.998 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9mOx2rIcNaUErPIMhkIjlSow7sw= 2024-02-01T10:27:42.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:42 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9mOx2rIcNaUErPIMhkIjlSow7sw=', 'Content-Length': '0'} 2024-02-01T10:27:43.002 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009b0434fb76566dbf-0065bb721f-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:43 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:43.044 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:43.044 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:43.045 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:43.045 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:43.045 DEBUG:boto:Method: GET 2024-02-01T10:27:43.045 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:43.045 DEBUG:boto:Data: 2024-02-01T10:27:43.045 DEBUG:boto:Headers: {} 2024-02-01T10:27:43.045 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:43.045 DEBUG:boto:Port: 8001 2024-02-01T10:27:43.046 DEBUG:boto:Params: {} 2024-02-01T10:27:43.046 DEBUG:boto:Token: None 2024-02-01T10:27:43.046 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:43 GMT /vclfcn-19/ 2024-02-01T10:27:43.046 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:M4hBIc8Cc0p4nUgLzXYuzMfMn5o= 2024-02-01T10:27:43.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:43 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:M4hBIc8Cc0p4nUgLzXYuzMfMn5o=', 'Content-Length': '0'} 2024-02-01T10:27:43.051 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091b77585002d6874-0065bb721f-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:43 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:43.052 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:43.052 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:43.052 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:43.053 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:43.053 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:43.053 DEBUG:boto:Method: GET 2024-02-01T10:27:43.053 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:43.053 DEBUG:boto:Data: 2024-02-01T10:27:43.053 DEBUG:boto:Headers: {} 2024-02-01T10:27:43.054 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:43.054 DEBUG:boto:Port: 8000 2024-02-01T10:27:43.054 DEBUG:boto:Params: {} 2024-02-01T10:27:43.054 DEBUG:boto:Token: None 2024-02-01T10:27:43.054 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:43 GMT /vclfcn-19/ 2024-02-01T10:27:43.054 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:M4hBIc8Cc0p4nUgLzXYuzMfMn5o= 2024-02-01T10:27:43.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:43 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:M4hBIc8Cc0p4nUgLzXYuzMfMn5o=', 'Content-Length': '0'} 2024-02-01T10:27:43.060 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089b734f00f0e762d-0065bb721f-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:43 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:43.060 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:43.061 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:43.061 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:43.061 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:43.061 DEBUG:boto:path=/vclfcn-19/obj?versionId=UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc 2024-02-01T10:27:43.061 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc 2024-02-01T10:27:43.061 DEBUG:boto:Method: GET 2024-02-01T10:27:43.062 DEBUG:boto:Path: /vclfcn-19/obj?versionId=UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc 2024-02-01T10:27:43.062 DEBUG:boto:Data: 2024-02-01T10:27:43.062 DEBUG:boto:Headers: {} 2024-02-01T10:27:43.062 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:43.062 DEBUG:boto:Port: 8001 2024-02-01T10:27:43.062 DEBUG:boto:Params: {} 2024-02-01T10:27:43.062 DEBUG:boto:Token: None 2024-02-01T10:27:43.062 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:43 GMT /vclfcn-19/obj?versionId=UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc 2024-02-01T10:27:43.062 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:2zqhUNa3EAcctVE04fwmLawYwN0= 2024-02-01T10:27:43.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:43 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2zqhUNa3EAcctVE04fwmLawYwN0=', 'Content-Length': '0'} 2024-02-01T10:27:43.066 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007eca1d0c134b9569-0065bb721f-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:43 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:43.066 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:43.066 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:43.066 DEBUG:boto:path=/vclfcn-19/obj?versionId=UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc 2024-02-01T10:27:43.066 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc 2024-02-01T10:27:43.067 DEBUG:boto:Method: GET 2024-02-01T10:27:43.067 DEBUG:boto:Path: /vclfcn-19/obj?versionId=UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc 2024-02-01T10:27:43.067 DEBUG:boto:Data: 2024-02-01T10:27:43.067 DEBUG:boto:Headers: {} 2024-02-01T10:27:43.067 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:43.067 DEBUG:boto:Port: 8000 2024-02-01T10:27:43.067 DEBUG:boto:Params: {} 2024-02-01T10:27:43.067 DEBUG:boto:Token: None 2024-02-01T10:27:43.067 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:43 GMT /vclfcn-19/obj?versionId=UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc 2024-02-01T10:27:43.067 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:2zqhUNa3EAcctVE04fwmLawYwN0= 2024-02-01T10:27:43.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:43 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2zqhUNa3EAcctVE04fwmLawYwN0=', 'Content-Length': '0'} 2024-02-01T10:27:43.070 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000045dbc9e929dbe69a-0065bb721f-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:43 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:43.112 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:43.112 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:43.112 DEBUG:boto:path=/vclfcn-19/obj?versionId=UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc 2024-02-01T10:27:43.112 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc 2024-02-01T10:27:43.113 DEBUG:boto:Method: HEAD 2024-02-01T10:27:43.113 DEBUG:boto:Path: /vclfcn-19/obj?versionId=UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc 2024-02-01T10:27:43.113 DEBUG:boto:Data: 2024-02-01T10:27:43.113 DEBUG:boto:Headers: {} 2024-02-01T10:27:43.113 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:43.113 DEBUG:boto:Port: 8001 2024-02-01T10:27:43.113 DEBUG:boto:Params: {} 2024-02-01T10:27:43.113 DEBUG:boto:Token: None 2024-02-01T10:27:43.113 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:43 GMT /vclfcn-19/obj?versionId=UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc 2024-02-01T10:27:43.114 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:F4CwOH1EIgBpsoj076GTMrTrMKY= 2024-02-01T10:27:43.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:43 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:F4CwOH1EIgBpsoj076GTMrTrMKY=', 'Content-Length': '0'} 2024-02-01T10:27:43.116 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005581a3ad8e3c76f4-0065bb721f-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:43 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:43.117 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:43.117 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:43.117 DEBUG:boto:path=/vclfcn-19/obj?versionId=UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc 2024-02-01T10:27:43.117 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc 2024-02-01T10:27:43.117 DEBUG:boto:Method: HEAD 2024-02-01T10:27:43.117 DEBUG:boto:Path: /vclfcn-19/obj?versionId=UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc 2024-02-01T10:27:43.117 DEBUG:boto:Data: 2024-02-01T10:27:43.117 DEBUG:boto:Headers: {} 2024-02-01T10:27:43.118 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:43.118 DEBUG:boto:Port: 8000 2024-02-01T10:27:43.118 DEBUG:boto:Params: {} 2024-02-01T10:27:43.118 DEBUG:boto:Token: None 2024-02-01T10:27:43.118 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:43 GMT /vclfcn-19/obj?versionId=UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc 2024-02-01T10:27:43.118 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:F4CwOH1EIgBpsoj076GTMrTrMKY= 2024-02-01T10:27:43.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:43 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:F4CwOH1EIgBpsoj076GTMrTrMKY=', 'Content-Length': '0'} 2024-02-01T10:27:43.121 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000272db759b80a77da-0065bb721f-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:43 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:43.121 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:43.121 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:43.121 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:43.121 DEBUG:boto:path=/vclfcn-19/obj?versionId=UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc 2024-02-01T10:27:43.121 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc 2024-02-01T10:27:43.121 DEBUG:boto:Method: GET 2024-02-01T10:27:43.122 DEBUG:boto:Path: /vclfcn-19/obj?versionId=UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc 2024-02-01T10:27:43.122 DEBUG:boto:Data: 2024-02-01T10:27:43.122 DEBUG:boto:Headers: {} 2024-02-01T10:27:43.122 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:43.122 DEBUG:boto:Port: 8001 2024-02-01T10:27:43.122 DEBUG:boto:Params: {} 2024-02-01T10:27:43.122 DEBUG:boto:Token: None 2024-02-01T10:27:43.122 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:43 GMT /vclfcn-19/obj?versionId=UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc 2024-02-01T10:27:43.122 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:2zqhUNa3EAcctVE04fwmLawYwN0= 2024-02-01T10:27:43.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:43 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2zqhUNa3EAcctVE04fwmLawYwN0=', 'Content-Length': '0'} 2024-02-01T10:27:43.126 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000088119e635926c8ba-0065bb721f-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:43 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:43.126 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:43.126 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:43.126 DEBUG:boto:path=/vclfcn-19/obj?versionId=UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc 2024-02-01T10:27:43.126 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc 2024-02-01T10:27:43.126 DEBUG:boto:Method: GET 2024-02-01T10:27:43.127 DEBUG:boto:Path: /vclfcn-19/obj?versionId=UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc 2024-02-01T10:27:43.127 DEBUG:boto:Data: 2024-02-01T10:27:43.127 DEBUG:boto:Headers: {} 2024-02-01T10:27:43.127 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:43.127 DEBUG:boto:Port: 8000 2024-02-01T10:27:43.127 DEBUG:boto:Params: {} 2024-02-01T10:27:43.127 DEBUG:boto:Token: None 2024-02-01T10:27:43.127 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:43 GMT /vclfcn-19/obj?versionId=UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc 2024-02-01T10:27:43.127 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:2zqhUNa3EAcctVE04fwmLawYwN0= 2024-02-01T10:27:43.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:43 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2zqhUNa3EAcctVE04fwmLawYwN0=', 'Content-Length': '0'} 2024-02-01T10:27:43.130 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000056e629cc65ed1a58-0065bb721f-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:43 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:43.172 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:43.172 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:43.172 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:43.172 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:43.173 DEBUG:boto:Method: GET 2024-02-01T10:27:43.173 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:43.173 DEBUG:boto:Data: 2024-02-01T10:27:43.173 DEBUG:boto:Headers: {} 2024-02-01T10:27:43.173 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:43.173 DEBUG:boto:Port: 8001 2024-02-01T10:27:43.173 DEBUG:boto:Params: {} 2024-02-01T10:27:43.173 DEBUG:boto:Token: None 2024-02-01T10:27:43.173 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:43 GMT /vclfcn-19/ 2024-02-01T10:27:43.174 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:M4hBIc8Cc0p4nUgLzXYuzMfMn5o= 2024-02-01T10:27:43.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:43 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:M4hBIc8Cc0p4nUgLzXYuzMfMn5o=', 'Content-Length': '0'} 2024-02-01T10:27:43.179 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b280d592068a367d-0065bb721f-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:43 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:43.179 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:43.180 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:43.180 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:43.180 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:43.180 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:43.180 DEBUG:boto:Method: GET 2024-02-01T10:27:43.180 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:43.180 DEBUG:boto:Data: 2024-02-01T10:27:43.181 DEBUG:boto:Headers: {} 2024-02-01T10:27:43.181 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:43.181 DEBUG:boto:Port: 8000 2024-02-01T10:27:43.181 DEBUG:boto:Params: {} 2024-02-01T10:27:43.181 DEBUG:boto:Token: None 2024-02-01T10:27:43.181 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:43 GMT /vclfcn-19/ 2024-02-01T10:27:43.181 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:M4hBIc8Cc0p4nUgLzXYuzMfMn5o= 2024-02-01T10:27:43.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:43 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:M4hBIc8Cc0p4nUgLzXYuzMfMn5o=', 'Content-Length': '0'} 2024-02-01T10:27:43.186 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f44606c071ccd2e-0065bb721f-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:43 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:43.186 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:43.186 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:43.187 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:43.187 DEBUG:boto:Method: HEAD 2024-02-01T10:27:43.187 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:43.187 DEBUG:boto:Data: 2024-02-01T10:27:43.187 DEBUG:boto:Headers: {} 2024-02-01T10:27:43.187 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:43.187 DEBUG:boto:Port: 8001 2024-02-01T10:27:43.187 DEBUG:boto:Params: {} 2024-02-01T10:27:43.187 DEBUG:boto:Token: None 2024-02-01T10:27:43.188 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:43 GMT /vclfcn-19/obj 2024-02-01T10:27:43.188 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:PR069GC7VH4BwvHXbgqBcnYnhts= 2024-02-01T10:27:43.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:43 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:PR069GC7VH4BwvHXbgqBcnYnhts=', 'Content-Length': '0'} 2024-02-01T10:27:43.194 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008149512763baba6d-0065bb721f-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:43 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:43.194 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:43.194 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:43.194 DEBUG:boto:Method: HEAD 2024-02-01T10:27:43.194 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:43.194 DEBUG:boto:Data: 2024-02-01T10:27:43.194 DEBUG:boto:Headers: {} 2024-02-01T10:27:43.194 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:43.195 DEBUG:boto:Port: 8000 2024-02-01T10:27:43.195 DEBUG:boto:Params: {} 2024-02-01T10:27:43.195 DEBUG:boto:Token: None 2024-02-01T10:27:43.195 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:43 GMT /vclfcn-19/obj 2024-02-01T10:27:43.195 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:PR069GC7VH4BwvHXbgqBcnYnhts= 2024-02-01T10:27:43.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:43 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:PR069GC7VH4BwvHXbgqBcnYnhts=', 'Content-Length': '0'} 2024-02-01T10:27:43.201 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fd04ec55a8602887-0065bb721f-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:43 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:43.201 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:43.201 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:43.201 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:43.201 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:43.201 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:43.201 DEBUG:boto:Method: GET 2024-02-01T10:27:43.201 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:43.202 DEBUG:boto:Data: 2024-02-01T10:27:43.202 DEBUG:boto:Headers: {} 2024-02-01T10:27:43.202 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:43.202 DEBUG:boto:Port: 8001 2024-02-01T10:27:43.202 DEBUG:boto:Params: {} 2024-02-01T10:27:43.202 DEBUG:boto:Token: None 2024-02-01T10:27:43.202 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:43 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:43.202 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BNbSNftR1acbobOxe1dHzOXpIe0= 2024-02-01T10:27:43.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:43 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BNbSNftR1acbobOxe1dHzOXpIe0=', 'Content-Length': '0'} 2024-02-01T10:27:43.205 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a2073e0d8adb8bca-0065bb721f-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:43 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:43.248 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:43.248 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:43.249 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:43.249 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:43.249 DEBUG:boto:Method: GET 2024-02-01T10:27:43.249 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:43.249 DEBUG:boto:Data: 2024-02-01T10:27:43.249 DEBUG:boto:Headers: {} 2024-02-01T10:27:43.249 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:43.249 DEBUG:boto:Port: 8000 2024-02-01T10:27:43.249 DEBUG:boto:Params: {} 2024-02-01T10:27:43.249 DEBUG:boto:Token: None 2024-02-01T10:27:43.250 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:43 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:43.251 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BNbSNftR1acbobOxe1dHzOXpIe0= 2024-02-01T10:27:43.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:43 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BNbSNftR1acbobOxe1dHzOXpIe0=', 'Content-Length': '0'} 2024-02-01T10:27:43.254 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c9b183eb23d4c24d-0065bb721f-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:43 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:43.296 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:43.296 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:43.296 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:43.297 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:43.297 DEBUG:boto:Method: GET 2024-02-01T10:27:43.297 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:43.297 DEBUG:boto:Data: 2024-02-01T10:27:43.297 DEBUG:boto:Headers: {} 2024-02-01T10:27:43.297 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:43.297 DEBUG:boto:Port: 8001 2024-02-01T10:27:43.297 DEBUG:boto:Params: {} 2024-02-01T10:27:43.297 DEBUG:boto:Token: None 2024-02-01T10:27:43.298 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:43 GMT /vclfcn-19/ 2024-02-01T10:27:43.298 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:M4hBIc8Cc0p4nUgLzXYuzMfMn5o= 2024-02-01T10:27:43.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:43 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:M4hBIc8Cc0p4nUgLzXYuzMfMn5o=', 'Content-Length': '0'} 2024-02-01T10:27:43.303 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000edc1ef6077915968-0065bb721f-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:43 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:43.303 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:43.303 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:43.304 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:43.304 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:43.304 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:43.304 DEBUG:boto:Method: GET 2024-02-01T10:27:43.304 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:43.304 DEBUG:boto:Data: 2024-02-01T10:27:43.304 DEBUG:boto:Headers: {} 2024-02-01T10:27:43.304 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:43.304 DEBUG:boto:Port: 8000 2024-02-01T10:27:43.304 DEBUG:boto:Params: {} 2024-02-01T10:27:43.305 DEBUG:boto:Token: None 2024-02-01T10:27:43.305 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:43 GMT /vclfcn-19/ 2024-02-01T10:27:43.305 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:M4hBIc8Cc0p4nUgLzXYuzMfMn5o= 2024-02-01T10:27:43.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:43 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:M4hBIc8Cc0p4nUgLzXYuzMfMn5o=', 'Content-Length': '0'} 2024-02-01T10:27:43.310 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd3562038298eec7-0065bb721f-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:43 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:43.310 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:43.311 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:43.311 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:43.311 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:43.311 DEBUG:boto:path=/vclfcn-19/obj?versionId=eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr. 2024-02-01T10:27:43.311 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr. 2024-02-01T10:27:43.311 DEBUG:boto:Method: GET 2024-02-01T10:27:43.311 DEBUG:boto:Path: /vclfcn-19/obj?versionId=eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr. 2024-02-01T10:27:43.311 DEBUG:boto:Data: 2024-02-01T10:27:43.312 DEBUG:boto:Headers: {} 2024-02-01T10:27:43.312 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:43.312 DEBUG:boto:Port: 8001 2024-02-01T10:27:43.312 DEBUG:boto:Params: {} 2024-02-01T10:27:43.312 DEBUG:boto:Token: None 2024-02-01T10:27:43.312 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:43 GMT /vclfcn-19/obj?versionId=eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr. 2024-02-01T10:27:43.313 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:otPaD1gVRU45joHCxXmiDJHZUW8= 2024-02-01T10:27:43.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:43 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:otPaD1gVRU45joHCxXmiDJHZUW8=', 'Content-Length': '0'} 2024-02-01T10:27:43.316 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr.'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bf4ad54e16cba599-0065bb721f-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:43 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:43.316 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:43.316 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:43.316 DEBUG:boto:path=/vclfcn-19/obj?versionId=eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr. 2024-02-01T10:27:43.316 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr. 2024-02-01T10:27:43.316 DEBUG:boto:Method: GET 2024-02-01T10:27:43.317 DEBUG:boto:Path: /vclfcn-19/obj?versionId=eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr. 2024-02-01T10:27:43.317 DEBUG:boto:Data: 2024-02-01T10:27:43.317 DEBUG:boto:Headers: {} 2024-02-01T10:27:43.317 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:43.317 DEBUG:boto:Port: 8000 2024-02-01T10:27:43.317 DEBUG:boto:Params: {} 2024-02-01T10:27:43.317 DEBUG:boto:Token: None 2024-02-01T10:27:43.317 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:43 GMT /vclfcn-19/obj?versionId=eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr. 2024-02-01T10:27:43.317 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:otPaD1gVRU45joHCxXmiDJHZUW8= 2024-02-01T10:27:43.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:43 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:otPaD1gVRU45joHCxXmiDJHZUW8=', 'Content-Length': '0'} 2024-02-01T10:27:43.320 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000801664fbfc28321f-0065bb721f-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:43 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:43.321 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:43.321 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:43.321 DEBUG:boto:path=/vclfcn-19/obj?versionId=eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr. 2024-02-01T10:27:43.321 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr. 2024-02-01T10:27:43.321 DEBUG:boto:Method: HEAD 2024-02-01T10:27:43.321 DEBUG:boto:Path: /vclfcn-19/obj?versionId=eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr. 2024-02-01T10:27:43.321 DEBUG:boto:Data: 2024-02-01T10:27:43.321 DEBUG:boto:Headers: {} 2024-02-01T10:27:43.321 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:43.322 DEBUG:boto:Port: 8001 2024-02-01T10:27:43.322 DEBUG:boto:Params: {} 2024-02-01T10:27:43.322 DEBUG:boto:Token: None 2024-02-01T10:27:43.322 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:43 GMT /vclfcn-19/obj?versionId=eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr. 2024-02-01T10:27:43.322 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ywzqxiWLbKxgSLZS+/0rw1j0qqU= 2024-02-01T10:27:43.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:43 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ywzqxiWLbKxgSLZS+/0rw1j0qqU=', 'Content-Length': '0'} 2024-02-01T10:27:43.325 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr.'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f9348998eb9ce2fb-0065bb721f-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:43 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:43.325 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:43.325 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:43.325 DEBUG:boto:path=/vclfcn-19/obj?versionId=eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr. 2024-02-01T10:27:43.325 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr. 2024-02-01T10:27:43.326 DEBUG:boto:Method: HEAD 2024-02-01T10:27:43.326 DEBUG:boto:Path: /vclfcn-19/obj?versionId=eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr. 2024-02-01T10:27:43.326 DEBUG:boto:Data: 2024-02-01T10:27:43.326 DEBUG:boto:Headers: {} 2024-02-01T10:27:43.326 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:43.326 DEBUG:boto:Port: 8000 2024-02-01T10:27:43.326 DEBUG:boto:Params: {} 2024-02-01T10:27:43.326 DEBUG:boto:Token: None 2024-02-01T10:27:43.326 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:43 GMT /vclfcn-19/obj?versionId=eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr. 2024-02-01T10:27:43.326 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ywzqxiWLbKxgSLZS+/0rw1j0qqU= 2024-02-01T10:27:43.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:43 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ywzqxiWLbKxgSLZS+/0rw1j0qqU=', 'Content-Length': '0'} 2024-02-01T10:27:43.329 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001c089db1832fc836-0065bb721f-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:43 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:43.330 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:43.330 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:43.330 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:43.330 DEBUG:boto:path=/vclfcn-19/obj?versionId=eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr. 2024-02-01T10:27:43.330 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr. 2024-02-01T10:27:43.330 DEBUG:boto:Method: GET 2024-02-01T10:27:43.330 DEBUG:boto:Path: /vclfcn-19/obj?versionId=eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr. 2024-02-01T10:27:43.331 DEBUG:boto:Data: 2024-02-01T10:27:43.331 DEBUG:boto:Headers: {} 2024-02-01T10:27:43.331 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:43.331 DEBUG:boto:Port: 8001 2024-02-01T10:27:43.331 DEBUG:boto:Params: {} 2024-02-01T10:27:43.331 DEBUG:boto:Token: None 2024-02-01T10:27:43.331 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:43 GMT /vclfcn-19/obj?versionId=eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr. 2024-02-01T10:27:43.331 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:otPaD1gVRU45joHCxXmiDJHZUW8= 2024-02-01T10:27:43.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:43 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:otPaD1gVRU45joHCxXmiDJHZUW8=', 'Content-Length': '0'} 2024-02-01T10:27:43.334 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr.'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a92416a5dc0838ab-0065bb721f-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:43 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:43.334 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:43.334 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:43.335 DEBUG:boto:path=/vclfcn-19/obj?versionId=eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr. 2024-02-01T10:27:43.335 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr. 2024-02-01T10:27:43.335 DEBUG:boto:Method: GET 2024-02-01T10:27:43.335 DEBUG:boto:Path: /vclfcn-19/obj?versionId=eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr. 2024-02-01T10:27:43.335 DEBUG:boto:Data: 2024-02-01T10:27:43.335 DEBUG:boto:Headers: {} 2024-02-01T10:27:43.335 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:43.335 DEBUG:boto:Port: 8000 2024-02-01T10:27:43.335 DEBUG:boto:Params: {} 2024-02-01T10:27:43.335 DEBUG:boto:Token: None 2024-02-01T10:27:43.336 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:43 GMT /vclfcn-19/obj?versionId=eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr. 2024-02-01T10:27:43.336 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:otPaD1gVRU45joHCxXmiDJHZUW8= 2024-02-01T10:27:43.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:43 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:otPaD1gVRU45joHCxXmiDJHZUW8=', 'Content-Length': '0'} 2024-02-01T10:27:43.339 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a469fd2dc3342d7c-0065bb721f-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:43 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:43.380 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:43.380 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:43.380 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:43.381 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:43.381 DEBUG:boto:Method: GET 2024-02-01T10:27:43.381 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:43.381 DEBUG:boto:Data: 2024-02-01T10:27:43.381 DEBUG:boto:Headers: {} 2024-02-01T10:27:43.381 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:43.381 DEBUG:boto:Port: 8001 2024-02-01T10:27:43.381 DEBUG:boto:Params: {} 2024-02-01T10:27:43.381 DEBUG:boto:Token: None 2024-02-01T10:27:43.382 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:43 GMT /vclfcn-19/ 2024-02-01T10:27:43.382 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:M4hBIc8Cc0p4nUgLzXYuzMfMn5o= 2024-02-01T10:27:43.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:43 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:M4hBIc8Cc0p4nUgLzXYuzMfMn5o=', 'Content-Length': '0'} 2024-02-01T10:27:43.387 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e29e4b588275a54-0065bb721f-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:43 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:43.387 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:43.387 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:43.387 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:43.388 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:43.388 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:43.388 DEBUG:boto:Method: GET 2024-02-01T10:27:43.388 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:43.388 DEBUG:boto:Data: 2024-02-01T10:27:43.388 DEBUG:boto:Headers: {} 2024-02-01T10:27:43.388 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:43.388 DEBUG:boto:Port: 8000 2024-02-01T10:27:43.388 DEBUG:boto:Params: {} 2024-02-01T10:27:43.388 DEBUG:boto:Token: None 2024-02-01T10:27:43.389 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:43 GMT /vclfcn-19/ 2024-02-01T10:27:43.389 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:M4hBIc8Cc0p4nUgLzXYuzMfMn5o= 2024-02-01T10:27:43.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:43 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:M4hBIc8Cc0p4nUgLzXYuzMfMn5o=', 'Content-Length': '0'} 2024-02-01T10:27:43.394 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f53d9b8d6a8f909-0065bb721f-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:43 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:43.394 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:43.394 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:43.395 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:43.395 DEBUG:boto:Method: HEAD 2024-02-01T10:27:43.395 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:43.395 DEBUG:boto:Data: 2024-02-01T10:27:43.395 DEBUG:boto:Headers: {} 2024-02-01T10:27:43.395 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:43.395 DEBUG:boto:Port: 8001 2024-02-01T10:27:43.395 DEBUG:boto:Params: {} 2024-02-01T10:27:43.395 DEBUG:boto:Token: None 2024-02-01T10:27:43.395 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:43 GMT /vclfcn-19/obj 2024-02-01T10:27:43.396 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:PR069GC7VH4BwvHXbgqBcnYnhts= 2024-02-01T10:27:43.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:43 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:PR069GC7VH4BwvHXbgqBcnYnhts=', 'Content-Length': '0'} 2024-02-01T10:27:43.401 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009efec0d2d252249f-0065bb721f-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:43 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:43.402 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:43.402 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:43.402 DEBUG:boto:Method: HEAD 2024-02-01T10:27:43.402 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:43.402 DEBUG:boto:Data: 2024-02-01T10:27:43.402 DEBUG:boto:Headers: {} 2024-02-01T10:27:43.402 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:43.402 DEBUG:boto:Port: 8000 2024-02-01T10:27:43.402 DEBUG:boto:Params: {} 2024-02-01T10:27:43.402 DEBUG:boto:Token: None 2024-02-01T10:27:43.402 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:43 GMT /vclfcn-19/obj 2024-02-01T10:27:43.403 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:PR069GC7VH4BwvHXbgqBcnYnhts= 2024-02-01T10:27:43.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:43 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:PR069GC7VH4BwvHXbgqBcnYnhts=', 'Content-Length': '0'} 2024-02-01T10:27:43.408 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001438679635504b65-0065bb721f-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:43 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:43.408 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:43.408 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:43.409 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:43.409 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:43.409 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:43.409 DEBUG:boto:Method: GET 2024-02-01T10:27:43.409 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:43.409 DEBUG:boto:Data: 2024-02-01T10:27:43.409 DEBUG:boto:Headers: {} 2024-02-01T10:27:43.409 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:43.409 DEBUG:boto:Port: 8001 2024-02-01T10:27:43.409 DEBUG:boto:Params: {} 2024-02-01T10:27:43.409 DEBUG:boto:Token: None 2024-02-01T10:27:43.410 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:43 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:43.410 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BNbSNftR1acbobOxe1dHzOXpIe0= 2024-02-01T10:27:43.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:43 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BNbSNftR1acbobOxe1dHzOXpIe0=', 'Content-Length': '0'} 2024-02-01T10:27:43.412 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bd83b371ace80489-0065bb721f-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:43 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:43.456 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:43.456 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:43.456 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:43.456 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:43.457 DEBUG:boto:Method: GET 2024-02-01T10:27:43.457 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:43.457 DEBUG:boto:Data: 2024-02-01T10:27:43.457 DEBUG:boto:Headers: {} 2024-02-01T10:27:43.457 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:43.457 DEBUG:boto:Port: 8000 2024-02-01T10:27:43.457 DEBUG:boto:Params: {} 2024-02-01T10:27:43.457 DEBUG:boto:Token: None 2024-02-01T10:27:43.457 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:43 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:43.458 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BNbSNftR1acbobOxe1dHzOXpIe0= 2024-02-01T10:27:43.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:43 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BNbSNftR1acbobOxe1dHzOXpIe0=', 'Content-Length': '0'} 2024-02-01T10:27:43.460 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000015813fda5b099c5e-0065bb721f-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:43 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:43.504 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:43.505 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:43.505 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:43.505 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:43.505 DEBUG:boto:Method: GET 2024-02-01T10:27:43.505 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:43.505 DEBUG:boto:Data: 2024-02-01T10:27:43.505 DEBUG:boto:Headers: {} 2024-02-01T10:27:43.505 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:43.506 DEBUG:boto:Port: 8001 2024-02-01T10:27:43.506 DEBUG:boto:Params: {} 2024-02-01T10:27:43.506 DEBUG:boto:Token: None 2024-02-01T10:27:43.506 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:43 GMT /vclfcn-19/ 2024-02-01T10:27:43.506 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:M4hBIc8Cc0p4nUgLzXYuzMfMn5o= 2024-02-01T10:27:43.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:43 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:M4hBIc8Cc0p4nUgLzXYuzMfMn5o=', 'Content-Length': '0'} 2024-02-01T10:27:43.511 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b68a229973a5554c-0065bb721f-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:43 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:43.512 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:43.512 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:43.513 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:43.513 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:43.513 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:43.513 DEBUG:boto:Method: GET 2024-02-01T10:27:43.513 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:43.513 DEBUG:boto:Data: 2024-02-01T10:27:43.513 DEBUG:boto:Headers: {} 2024-02-01T10:27:43.513 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:43.513 DEBUG:boto:Port: 8000 2024-02-01T10:27:43.513 DEBUG:boto:Params: {} 2024-02-01T10:27:43.514 DEBUG:boto:Token: None 2024-02-01T10:27:43.514 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:43 GMT /vclfcn-19/ 2024-02-01T10:27:43.514 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:M4hBIc8Cc0p4nUgLzXYuzMfMn5o= 2024-02-01T10:27:43.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:43 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:M4hBIc8Cc0p4nUgLzXYuzMfMn5o=', 'Content-Length': '0'} 2024-02-01T10:27:43.519 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d66220f42f3e1091-0065bb721f-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:43 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:43.520 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:43.520 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:43.520 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:43.520 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:43.520 DEBUG:boto:path=/vclfcn-19/obj?versionId=tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN 2024-02-01T10:27:43.520 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN 2024-02-01T10:27:43.520 DEBUG:boto:Method: GET 2024-02-01T10:27:43.521 DEBUG:boto:Path: /vclfcn-19/obj?versionId=tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN 2024-02-01T10:27:43.521 DEBUG:boto:Data: 2024-02-01T10:27:43.521 DEBUG:boto:Headers: {} 2024-02-01T10:27:43.521 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:43.521 DEBUG:boto:Port: 8001 2024-02-01T10:27:43.521 DEBUG:boto:Params: {} 2024-02-01T10:27:43.521 DEBUG:boto:Token: None 2024-02-01T10:27:43.521 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:43 GMT /vclfcn-19/obj?versionId=tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN 2024-02-01T10:27:43.521 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:3Al7Ef2cYXwDgzOkRP9UJoT8oVM= 2024-02-01T10:27:43.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:43 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:3Al7Ef2cYXwDgzOkRP9UJoT8oVM=', 'Content-Length': '0'} 2024-02-01T10:27:43.525 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001e96f418168b799c-0065bb721f-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:43 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:43.525 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:43.525 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:43.525 DEBUG:boto:path=/vclfcn-19/obj?versionId=tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN 2024-02-01T10:27:43.525 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN 2024-02-01T10:27:43.525 DEBUG:boto:Method: GET 2024-02-01T10:27:43.525 DEBUG:boto:Path: /vclfcn-19/obj?versionId=tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN 2024-02-01T10:27:43.526 DEBUG:boto:Data: 2024-02-01T10:27:43.526 DEBUG:boto:Headers: {} 2024-02-01T10:27:43.526 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:43.526 DEBUG:boto:Port: 8000 2024-02-01T10:27:43.526 DEBUG:boto:Params: {} 2024-02-01T10:27:43.526 DEBUG:boto:Token: None 2024-02-01T10:27:43.526 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:43 GMT /vclfcn-19/obj?versionId=tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN 2024-02-01T10:27:43.526 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:3Al7Ef2cYXwDgzOkRP9UJoT8oVM= 2024-02-01T10:27:43.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:43 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:3Al7Ef2cYXwDgzOkRP9UJoT8oVM=', 'Content-Length': '0'} 2024-02-01T10:27:43.530 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bcd6ffae9aff447a-0065bb721f-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:43 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:43.576 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:43.576 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:43.576 DEBUG:boto:path=/vclfcn-19/obj?versionId=tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN 2024-02-01T10:27:43.576 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN 2024-02-01T10:27:43.577 DEBUG:boto:Method: HEAD 2024-02-01T10:27:43.577 DEBUG:boto:Path: /vclfcn-19/obj?versionId=tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN 2024-02-01T10:27:43.577 DEBUG:boto:Data: 2024-02-01T10:27:43.577 DEBUG:boto:Headers: {} 2024-02-01T10:27:43.577 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:43.577 DEBUG:boto:Port: 8001 2024-02-01T10:27:43.577 DEBUG:boto:Params: {} 2024-02-01T10:27:43.577 DEBUG:boto:Token: None 2024-02-01T10:27:43.577 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:43 GMT /vclfcn-19/obj?versionId=tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN 2024-02-01T10:27:43.578 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:avcgG5KSNZh3tvSHv2M76ryKfj4= 2024-02-01T10:27:43.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:43 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:avcgG5KSNZh3tvSHv2M76ryKfj4=', 'Content-Length': '0'} 2024-02-01T10:27:43.580 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008719aff3b278cd01-0065bb721f-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:43 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:43.581 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:43.581 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:43.581 DEBUG:boto:path=/vclfcn-19/obj?versionId=tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN 2024-02-01T10:27:43.581 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN 2024-02-01T10:27:43.581 DEBUG:boto:Method: HEAD 2024-02-01T10:27:43.581 DEBUG:boto:Path: /vclfcn-19/obj?versionId=tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN 2024-02-01T10:27:43.581 DEBUG:boto:Data: 2024-02-01T10:27:43.581 DEBUG:boto:Headers: {} 2024-02-01T10:27:43.581 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:43.582 DEBUG:boto:Port: 8000 2024-02-01T10:27:43.582 DEBUG:boto:Params: {} 2024-02-01T10:27:43.582 DEBUG:boto:Token: None 2024-02-01T10:27:43.582 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:43 GMT /vclfcn-19/obj?versionId=tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN 2024-02-01T10:27:43.582 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:avcgG5KSNZh3tvSHv2M76ryKfj4= 2024-02-01T10:27:43.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:43 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:avcgG5KSNZh3tvSHv2M76ryKfj4=', 'Content-Length': '0'} 2024-02-01T10:27:43.585 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000237e039e3e478d5f-0065bb721f-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:43 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:43.585 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:43.585 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:43.585 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:43.585 DEBUG:boto:path=/vclfcn-19/obj?versionId=tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN 2024-02-01T10:27:43.585 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN 2024-02-01T10:27:43.586 DEBUG:boto:Method: GET 2024-02-01T10:27:43.586 DEBUG:boto:Path: /vclfcn-19/obj?versionId=tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN 2024-02-01T10:27:43.586 DEBUG:boto:Data: 2024-02-01T10:27:43.586 DEBUG:boto:Headers: {} 2024-02-01T10:27:43.586 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:43.586 DEBUG:boto:Port: 8001 2024-02-01T10:27:43.586 DEBUG:boto:Params: {} 2024-02-01T10:27:43.586 DEBUG:boto:Token: None 2024-02-01T10:27:43.586 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:43 GMT /vclfcn-19/obj?versionId=tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN 2024-02-01T10:27:43.586 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:3Al7Ef2cYXwDgzOkRP9UJoT8oVM= 2024-02-01T10:27:43.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:43 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:3Al7Ef2cYXwDgzOkRP9UJoT8oVM=', 'Content-Length': '0'} 2024-02-01T10:27:43.589 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000065b52f981dbc3a81-0065bb721f-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:43 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:43.590 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:43.590 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:43.590 DEBUG:boto:path=/vclfcn-19/obj?versionId=tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN 2024-02-01T10:27:43.590 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN 2024-02-01T10:27:43.590 DEBUG:boto:Method: GET 2024-02-01T10:27:43.590 DEBUG:boto:Path: /vclfcn-19/obj?versionId=tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN 2024-02-01T10:27:43.590 DEBUG:boto:Data: 2024-02-01T10:27:43.590 DEBUG:boto:Headers: {} 2024-02-01T10:27:43.590 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:43.591 DEBUG:boto:Port: 8000 2024-02-01T10:27:43.591 DEBUG:boto:Params: {} 2024-02-01T10:27:43.591 DEBUG:boto:Token: None 2024-02-01T10:27:43.591 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:43 GMT /vclfcn-19/obj?versionId=tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN 2024-02-01T10:27:43.591 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:3Al7Ef2cYXwDgzOkRP9UJoT8oVM= 2024-02-01T10:27:43.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:43 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:3Al7Ef2cYXwDgzOkRP9UJoT8oVM=', 'Content-Length': '0'} 2024-02-01T10:27:43.594 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e9275a3e8891d2db-0065bb721f-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:43 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:43.640 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:43.640 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:43.640 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:43.641 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:43.641 DEBUG:boto:Method: GET 2024-02-01T10:27:43.641 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:43.641 DEBUG:boto:Data: 2024-02-01T10:27:43.641 DEBUG:boto:Headers: {} 2024-02-01T10:27:43.641 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:43.641 DEBUG:boto:Port: 8001 2024-02-01T10:27:43.641 DEBUG:boto:Params: {} 2024-02-01T10:27:43.641 DEBUG:boto:Token: None 2024-02-01T10:27:43.641 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:43 GMT /vclfcn-19/ 2024-02-01T10:27:43.642 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:M4hBIc8Cc0p4nUgLzXYuzMfMn5o= 2024-02-01T10:27:43.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:43 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:M4hBIc8Cc0p4nUgLzXYuzMfMn5o=', 'Content-Length': '0'} 2024-02-01T10:27:43.647 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ece745f2d3a95661-0065bb721f-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:43 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:43.647 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:43.648 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:43.648 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:43.648 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:43.648 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:43.648 DEBUG:boto:Method: GET 2024-02-01T10:27:43.648 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:43.648 DEBUG:boto:Data: 2024-02-01T10:27:43.648 DEBUG:boto:Headers: {} 2024-02-01T10:27:43.648 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:43.649 DEBUG:boto:Port: 8000 2024-02-01T10:27:43.649 DEBUG:boto:Params: {} 2024-02-01T10:27:43.649 DEBUG:boto:Token: None 2024-02-01T10:27:43.649 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:43 GMT /vclfcn-19/ 2024-02-01T10:27:43.649 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:M4hBIc8Cc0p4nUgLzXYuzMfMn5o= 2024-02-01T10:27:43.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:43 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:M4hBIc8Cc0p4nUgLzXYuzMfMn5o=', 'Content-Length': '0'} 2024-02-01T10:27:43.655 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a39354c3acf981a-0065bb721f-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:43 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:43.655 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:43.656 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:43.656 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:43.656 DEBUG:boto:Method: HEAD 2024-02-01T10:27:43.656 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:43.656 DEBUG:boto:Data: 2024-02-01T10:27:43.657 DEBUG:boto:Headers: {} 2024-02-01T10:27:43.657 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:43.657 DEBUG:boto:Port: 8001 2024-02-01T10:27:43.657 DEBUG:boto:Params: {} 2024-02-01T10:27:43.657 DEBUG:boto:Token: None 2024-02-01T10:27:43.657 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:43 GMT /vclfcn-19/obj 2024-02-01T10:27:43.657 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:PR069GC7VH4BwvHXbgqBcnYnhts= 2024-02-01T10:27:43.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:43 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:PR069GC7VH4BwvHXbgqBcnYnhts=', 'Content-Length': '0'} 2024-02-01T10:27:43.663 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000001a82f898af15d98-0065bb721f-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:43 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:43.663 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:43.663 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:43.664 DEBUG:boto:Method: HEAD 2024-02-01T10:27:43.664 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:43.664 DEBUG:boto:Data: 2024-02-01T10:27:43.664 DEBUG:boto:Headers: {} 2024-02-01T10:27:43.664 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:43.664 DEBUG:boto:Port: 8000 2024-02-01T10:27:43.664 DEBUG:boto:Params: {} 2024-02-01T10:27:43.664 DEBUG:boto:Token: None 2024-02-01T10:27:43.664 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:43 GMT /vclfcn-19/obj 2024-02-01T10:27:43.664 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:PR069GC7VH4BwvHXbgqBcnYnhts= 2024-02-01T10:27:43.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:43 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:PR069GC7VH4BwvHXbgqBcnYnhts=', 'Content-Length': '0'} 2024-02-01T10:27:43.670 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000064c402a96e073119-0065bb721f-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:43 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:43.671 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:43.671 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:43.671 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:43.671 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:43.671 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:43.671 DEBUG:boto:Method: GET 2024-02-01T10:27:43.671 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:43.671 DEBUG:boto:Data: 2024-02-01T10:27:43.671 DEBUG:boto:Headers: {} 2024-02-01T10:27:43.672 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:43.672 DEBUG:boto:Port: 8001 2024-02-01T10:27:43.672 DEBUG:boto:Params: {} 2024-02-01T10:27:43.672 DEBUG:boto:Token: None 2024-02-01T10:27:43.672 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:43 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:43.672 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BNbSNftR1acbobOxe1dHzOXpIe0= 2024-02-01T10:27:43.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:43 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BNbSNftR1acbobOxe1dHzOXpIe0=', 'Content-Length': '0'} 2024-02-01T10:27:43.675 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002a0db0daa9759d17-0065bb721f-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:43 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:43.720 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:43.720 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:43.720 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:43.721 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:43.721 DEBUG:boto:Method: GET 2024-02-01T10:27:43.721 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:43.721 DEBUG:boto:Data: 2024-02-01T10:27:43.721 DEBUG:boto:Headers: {} 2024-02-01T10:27:43.721 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:43.721 DEBUG:boto:Port: 8000 2024-02-01T10:27:43.721 DEBUG:boto:Params: {} 2024-02-01T10:27:43.722 DEBUG:boto:Token: None 2024-02-01T10:27:43.722 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:43 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:43.722 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BNbSNftR1acbobOxe1dHzOXpIe0= 2024-02-01T10:27:43.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:43 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BNbSNftR1acbobOxe1dHzOXpIe0=', 'Content-Length': '0'} 2024-02-01T10:27:43.726 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f826d4921a03f8e0-0065bb721f-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:43 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:43.772 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:43.772 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:43.772 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:43.772 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:43.773 DEBUG:boto:Method: GET 2024-02-01T10:27:43.773 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:43.773 DEBUG:boto:Data: 2024-02-01T10:27:43.773 DEBUG:boto:Headers: {} 2024-02-01T10:27:43.773 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:43.773 DEBUG:boto:Port: 8001 2024-02-01T10:27:43.773 DEBUG:boto:Params: {} 2024-02-01T10:27:43.773 DEBUG:boto:Token: None 2024-02-01T10:27:43.773 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:43 GMT /vclfcn-19/ 2024-02-01T10:27:43.774 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:M4hBIc8Cc0p4nUgLzXYuzMfMn5o= 2024-02-01T10:27:43.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:43 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:M4hBIc8Cc0p4nUgLzXYuzMfMn5o=', 'Content-Length': '0'} 2024-02-01T10:27:43.780 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000193c806541316960-0065bb721f-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:43 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:43.780 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:43.780 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:43.780 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:43.781 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:43.781 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:43.781 DEBUG:boto:Method: GET 2024-02-01T10:27:43.781 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:43.781 DEBUG:boto:Data: 2024-02-01T10:27:43.781 DEBUG:boto:Headers: {} 2024-02-01T10:27:43.781 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:43.781 DEBUG:boto:Port: 8000 2024-02-01T10:27:43.781 DEBUG:boto:Params: {} 2024-02-01T10:27:43.781 DEBUG:boto:Token: None 2024-02-01T10:27:43.782 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:43 GMT /vclfcn-19/ 2024-02-01T10:27:43.782 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:M4hBIc8Cc0p4nUgLzXYuzMfMn5o= 2024-02-01T10:27:43.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:43 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:M4hBIc8Cc0p4nUgLzXYuzMfMn5o=', 'Content-Length': '0'} 2024-02-01T10:27:43.787 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000579f7c6bf0c69bb1-0065bb721f-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:43 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:43.832 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:43.833 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:43.833 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:43.833 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:43.833 DEBUG:boto:path=/vclfcn-19/obj?versionId=2lRbqumcCYnVWCIz20mU455245nXaz8 2024-02-01T10:27:43.833 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=2lRbqumcCYnVWCIz20mU455245nXaz8 2024-02-01T10:27:43.833 DEBUG:boto:Method: GET 2024-02-01T10:27:43.833 DEBUG:boto:Path: /vclfcn-19/obj?versionId=2lRbqumcCYnVWCIz20mU455245nXaz8 2024-02-01T10:27:43.833 DEBUG:boto:Data: 2024-02-01T10:27:43.834 DEBUG:boto:Headers: {} 2024-02-01T10:27:43.834 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:43.834 DEBUG:boto:Port: 8001 2024-02-01T10:27:43.834 DEBUG:boto:Params: {} 2024-02-01T10:27:43.834 DEBUG:boto:Token: None 2024-02-01T10:27:43.834 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:43 GMT /vclfcn-19/obj?versionId=2lRbqumcCYnVWCIz20mU455245nXaz8 2024-02-01T10:27:43.834 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:bqZQ9BQN6NU265+JMsp2SzKdjNE= 2024-02-01T10:27:43.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:43 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:bqZQ9BQN6NU265+JMsp2SzKdjNE=', 'Content-Length': '0'} 2024-02-01T10:27:43.838 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '2lRbqumcCYnVWCIz20mU455245nXaz8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f53aebef89c78f68-0065bb721f-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:43 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:43.838 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:43.838 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:43.838 DEBUG:boto:path=/vclfcn-19/obj?versionId=2lRbqumcCYnVWCIz20mU455245nXaz8 2024-02-01T10:27:43.838 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=2lRbqumcCYnVWCIz20mU455245nXaz8 2024-02-01T10:27:43.838 DEBUG:boto:Method: GET 2024-02-01T10:27:43.838 DEBUG:boto:Path: /vclfcn-19/obj?versionId=2lRbqumcCYnVWCIz20mU455245nXaz8 2024-02-01T10:27:43.839 DEBUG:boto:Data: 2024-02-01T10:27:43.839 DEBUG:boto:Headers: {} 2024-02-01T10:27:43.839 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:43.839 DEBUG:boto:Port: 8000 2024-02-01T10:27:43.839 DEBUG:boto:Params: {} 2024-02-01T10:27:43.839 DEBUG:boto:Token: None 2024-02-01T10:27:43.839 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:43 GMT /vclfcn-19/obj?versionId=2lRbqumcCYnVWCIz20mU455245nXaz8 2024-02-01T10:27:43.839 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:bqZQ9BQN6NU265+JMsp2SzKdjNE= 2024-02-01T10:27:43.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:43 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:bqZQ9BQN6NU265+JMsp2SzKdjNE=', 'Content-Length': '0'} 2024-02-01T10:27:43.842 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '2lRbqumcCYnVWCIz20mU455245nXaz8'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eca0a75b9c17f7cd-0065bb721f-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:43 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:43.884 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:43.884 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:43.884 DEBUG:boto:path=/vclfcn-19/obj?versionId=2lRbqumcCYnVWCIz20mU455245nXaz8 2024-02-01T10:27:43.884 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=2lRbqumcCYnVWCIz20mU455245nXaz8 2024-02-01T10:27:43.885 DEBUG:boto:Method: HEAD 2024-02-01T10:27:43.885 DEBUG:boto:Path: /vclfcn-19/obj?versionId=2lRbqumcCYnVWCIz20mU455245nXaz8 2024-02-01T10:27:43.885 DEBUG:boto:Data: 2024-02-01T10:27:43.885 DEBUG:boto:Headers: {} 2024-02-01T10:27:43.885 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:43.885 DEBUG:boto:Port: 8001 2024-02-01T10:27:43.885 DEBUG:boto:Params: {} 2024-02-01T10:27:43.885 DEBUG:boto:Token: None 2024-02-01T10:27:43.885 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:43 GMT /vclfcn-19/obj?versionId=2lRbqumcCYnVWCIz20mU455245nXaz8 2024-02-01T10:27:43.886 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8hfW1p8VSV+hH57jgCBjPZQITcc= 2024-02-01T10:27:43.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:43 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8hfW1p8VSV+hH57jgCBjPZQITcc=', 'Content-Length': '0'} 2024-02-01T10:27:43.889 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '2lRbqumcCYnVWCIz20mU455245nXaz8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f18c38a39387d3a9-0065bb721f-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:43 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:43.889 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:43.889 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:43.889 DEBUG:boto:path=/vclfcn-19/obj?versionId=2lRbqumcCYnVWCIz20mU455245nXaz8 2024-02-01T10:27:43.889 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=2lRbqumcCYnVWCIz20mU455245nXaz8 2024-02-01T10:27:43.889 DEBUG:boto:Method: HEAD 2024-02-01T10:27:43.890 DEBUG:boto:Path: /vclfcn-19/obj?versionId=2lRbqumcCYnVWCIz20mU455245nXaz8 2024-02-01T10:27:43.890 DEBUG:boto:Data: 2024-02-01T10:27:43.890 DEBUG:boto:Headers: {} 2024-02-01T10:27:43.890 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:43.890 DEBUG:boto:Port: 8000 2024-02-01T10:27:43.890 DEBUG:boto:Params: {} 2024-02-01T10:27:43.890 DEBUG:boto:Token: None 2024-02-01T10:27:43.890 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:43 GMT /vclfcn-19/obj?versionId=2lRbqumcCYnVWCIz20mU455245nXaz8 2024-02-01T10:27:43.890 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8hfW1p8VSV+hH57jgCBjPZQITcc= 2024-02-01T10:27:43.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:43 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8hfW1p8VSV+hH57jgCBjPZQITcc=', 'Content-Length': '0'} 2024-02-01T10:27:43.893 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '2lRbqumcCYnVWCIz20mU455245nXaz8'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000084e60fd8249e704c-0065bb721f-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:43 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:43.894 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:43.894 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:43.894 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:43.894 DEBUG:boto:path=/vclfcn-19/obj?versionId=2lRbqumcCYnVWCIz20mU455245nXaz8 2024-02-01T10:27:43.894 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=2lRbqumcCYnVWCIz20mU455245nXaz8 2024-02-01T10:27:43.894 DEBUG:boto:Method: GET 2024-02-01T10:27:43.894 DEBUG:boto:Path: /vclfcn-19/obj?versionId=2lRbqumcCYnVWCIz20mU455245nXaz8 2024-02-01T10:27:43.894 DEBUG:boto:Data: 2024-02-01T10:27:43.895 DEBUG:boto:Headers: {} 2024-02-01T10:27:43.895 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:43.895 DEBUG:boto:Port: 8001 2024-02-01T10:27:43.895 DEBUG:boto:Params: {} 2024-02-01T10:27:43.895 DEBUG:boto:Token: None 2024-02-01T10:27:43.895 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:43 GMT /vclfcn-19/obj?versionId=2lRbqumcCYnVWCIz20mU455245nXaz8 2024-02-01T10:27:43.896 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:bqZQ9BQN6NU265+JMsp2SzKdjNE= 2024-02-01T10:27:43.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:43 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:bqZQ9BQN6NU265+JMsp2SzKdjNE=', 'Content-Length': '0'} 2024-02-01T10:27:43.899 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '2lRbqumcCYnVWCIz20mU455245nXaz8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006574d2ae9b95e240-0065bb721f-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:43 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:43.900 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:43.900 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:43.900 DEBUG:boto:path=/vclfcn-19/obj?versionId=2lRbqumcCYnVWCIz20mU455245nXaz8 2024-02-01T10:27:43.900 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=2lRbqumcCYnVWCIz20mU455245nXaz8 2024-02-01T10:27:43.900 DEBUG:boto:Method: GET 2024-02-01T10:27:43.900 DEBUG:boto:Path: /vclfcn-19/obj?versionId=2lRbqumcCYnVWCIz20mU455245nXaz8 2024-02-01T10:27:43.900 DEBUG:boto:Data: 2024-02-01T10:27:43.900 DEBUG:boto:Headers: {} 2024-02-01T10:27:43.900 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:43.900 DEBUG:boto:Port: 8000 2024-02-01T10:27:43.900 DEBUG:boto:Params: {} 2024-02-01T10:27:43.901 DEBUG:boto:Token: None 2024-02-01T10:27:43.901 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:43 GMT /vclfcn-19/obj?versionId=2lRbqumcCYnVWCIz20mU455245nXaz8 2024-02-01T10:27:43.901 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:bqZQ9BQN6NU265+JMsp2SzKdjNE= 2024-02-01T10:27:43.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:43 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:bqZQ9BQN6NU265+JMsp2SzKdjNE=', 'Content-Length': '0'} 2024-02-01T10:27:43.904 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '2lRbqumcCYnVWCIz20mU455245nXaz8'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009b9c9d6a1fd56b72-0065bb721f-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:43 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:43.948 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:43.948 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:43.948 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:43.949 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:43.949 DEBUG:boto:Method: GET 2024-02-01T10:27:43.949 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:43.949 DEBUG:boto:Data: 2024-02-01T10:27:43.949 DEBUG:boto:Headers: {} 2024-02-01T10:27:43.949 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:43.949 DEBUG:boto:Port: 8001 2024-02-01T10:27:43.949 DEBUG:boto:Params: {} 2024-02-01T10:27:43.949 DEBUG:boto:Token: None 2024-02-01T10:27:43.950 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:43 GMT /vclfcn-19/ 2024-02-01T10:27:43.950 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:M4hBIc8Cc0p4nUgLzXYuzMfMn5o= 2024-02-01T10:27:43.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:43 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:M4hBIc8Cc0p4nUgLzXYuzMfMn5o=', 'Content-Length': '0'} 2024-02-01T10:27:43.955 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a18964e200598236-0065bb721f-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:43 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:43.955 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:43.956 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:43.956 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:43.956 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:43.956 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:43.956 DEBUG:boto:Method: GET 2024-02-01T10:27:43.956 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:43.956 DEBUG:boto:Data: 2024-02-01T10:27:43.956 DEBUG:boto:Headers: {} 2024-02-01T10:27:43.957 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:43.957 DEBUG:boto:Port: 8000 2024-02-01T10:27:43.957 DEBUG:boto:Params: {} 2024-02-01T10:27:43.957 DEBUG:boto:Token: None 2024-02-01T10:27:43.957 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:43 GMT /vclfcn-19/ 2024-02-01T10:27:43.957 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:M4hBIc8Cc0p4nUgLzXYuzMfMn5o= 2024-02-01T10:27:43.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:43 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:M4hBIc8Cc0p4nUgLzXYuzMfMn5o=', 'Content-Length': '0'} 2024-02-01T10:27:43.962 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bbaf6c21850d33e3-0065bb721f-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:43 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:43.963 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:43.963 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:43.963 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:43.964 DEBUG:boto:Method: HEAD 2024-02-01T10:27:43.964 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:43.964 DEBUG:boto:Data: 2024-02-01T10:27:43.964 DEBUG:boto:Headers: {} 2024-02-01T10:27:43.964 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:43.964 DEBUG:boto:Port: 8001 2024-02-01T10:27:43.964 DEBUG:boto:Params: {} 2024-02-01T10:27:43.964 DEBUG:boto:Token: None 2024-02-01T10:27:43.964 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:43 GMT /vclfcn-19/obj 2024-02-01T10:27:43.964 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:PR069GC7VH4BwvHXbgqBcnYnhts= 2024-02-01T10:27:43.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:43 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:PR069GC7VH4BwvHXbgqBcnYnhts=', 'Content-Length': '0'} 2024-02-01T10:27:43.970 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a31b9bec46c78ab6-0065bb721f-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:43 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:43.970 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:43.971 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:43.971 DEBUG:boto:Method: HEAD 2024-02-01T10:27:43.971 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:43.971 DEBUG:boto:Data: 2024-02-01T10:27:43.971 DEBUG:boto:Headers: {} 2024-02-01T10:27:43.971 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:43.971 DEBUG:boto:Port: 8000 2024-02-01T10:27:43.971 DEBUG:boto:Params: {} 2024-02-01T10:27:43.971 DEBUG:boto:Token: None 2024-02-01T10:27:43.971 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:43 GMT /vclfcn-19/obj 2024-02-01T10:27:43.972 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:PR069GC7VH4BwvHXbgqBcnYnhts= 2024-02-01T10:27:43.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:43 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:PR069GC7VH4BwvHXbgqBcnYnhts=', 'Content-Length': '0'} 2024-02-01T10:27:43.978 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dce290dc49a2284c-0065bb721f-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:43 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:43.978 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:43.978 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:43.978 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:43.979 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:43.979 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:43.979 DEBUG:boto:Method: GET 2024-02-01T10:27:43.979 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:43.979 DEBUG:boto:Data: 2024-02-01T10:27:43.979 DEBUG:boto:Headers: {} 2024-02-01T10:27:43.979 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:43.979 DEBUG:boto:Port: 8001 2024-02-01T10:27:43.979 DEBUG:boto:Params: {} 2024-02-01T10:27:43.979 DEBUG:boto:Token: None 2024-02-01T10:27:43.980 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:43 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:43.980 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BNbSNftR1acbobOxe1dHzOXpIe0= 2024-02-01T10:27:43.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:43 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BNbSNftR1acbobOxe1dHzOXpIe0=', 'Content-Length': '0'} 2024-02-01T10:27:43.983 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005c8b29ee76b08ff6-0065bb721f-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:43 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:44.024 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:44.024 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:44.024 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:44.024 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:44.025 DEBUG:boto:Method: GET 2024-02-01T10:27:44.025 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:44.025 DEBUG:boto:Data: 2024-02-01T10:27:44.025 DEBUG:boto:Headers: {} 2024-02-01T10:27:44.025 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:44.025 DEBUG:boto:Port: 8000 2024-02-01T10:27:44.025 DEBUG:boto:Params: {} 2024-02-01T10:27:44.025 DEBUG:boto:Token: None 2024-02-01T10:27:44.026 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:44 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:44.026 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:n4zWKMInJexpRe6UprkQ+QrO+PI= 2024-02-01T10:27:44.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:44 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:n4zWKMInJexpRe6UprkQ+QrO+PI=', 'Content-Length': '0'} 2024-02-01T10:27:44.029 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000970e0d5edae9e0f2-0065bb7220-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:44 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:44.072 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:44.072 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:44.072 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:44.072 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:44.073 DEBUG:boto:Method: GET 2024-02-01T10:27:44.073 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:44.073 DEBUG:boto:Data: 2024-02-01T10:27:44.073 DEBUG:boto:Headers: {} 2024-02-01T10:27:44.073 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:44.073 DEBUG:boto:Port: 8001 2024-02-01T10:27:44.073 DEBUG:boto:Params: {} 2024-02-01T10:27:44.073 DEBUG:boto:Token: None 2024-02-01T10:27:44.073 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:44 GMT /vclfcn-19/ 2024-02-01T10:27:44.074 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:yGN7E6zLD69D8A0hKqzZH3AAuj8= 2024-02-01T10:27:44.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:44 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:yGN7E6zLD69D8A0hKqzZH3AAuj8=', 'Content-Length': '0'} 2024-02-01T10:27:44.079 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd23a44e89650d19-0065bb7220-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:44 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:44.080 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:44.080 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:44.081 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:44.081 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:44.081 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:44.081 DEBUG:boto:Method: GET 2024-02-01T10:27:44.081 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:44.081 DEBUG:boto:Data: 2024-02-01T10:27:44.081 DEBUG:boto:Headers: {} 2024-02-01T10:27:44.081 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:44.081 DEBUG:boto:Port: 8000 2024-02-01T10:27:44.081 DEBUG:boto:Params: {} 2024-02-01T10:27:44.081 DEBUG:boto:Token: None 2024-02-01T10:27:44.082 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:44 GMT /vclfcn-19/ 2024-02-01T10:27:44.082 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:yGN7E6zLD69D8A0hKqzZH3AAuj8= 2024-02-01T10:27:44.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:44 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:yGN7E6zLD69D8A0hKqzZH3AAuj8=', 'Content-Length': '0'} 2024-02-01T10:27:44.087 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d10079a9a87e9dc-0065bb7220-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:44 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:44.128 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:44.128 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:44.129 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:44.129 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:44.129 DEBUG:boto:path=/vclfcn-19/obj?versionId=oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r 2024-02-01T10:27:44.129 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r 2024-02-01T10:27:44.129 DEBUG:boto:Method: GET 2024-02-01T10:27:44.129 DEBUG:boto:Path: /vclfcn-19/obj?versionId=oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r 2024-02-01T10:27:44.129 DEBUG:boto:Data: 2024-02-01T10:27:44.129 DEBUG:boto:Headers: {} 2024-02-01T10:27:44.129 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:44.129 DEBUG:boto:Port: 8001 2024-02-01T10:27:44.130 DEBUG:boto:Params: {} 2024-02-01T10:27:44.130 DEBUG:boto:Token: None 2024-02-01T10:27:44.130 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:44 GMT /vclfcn-19/obj?versionId=oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r 2024-02-01T10:27:44.130 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:TLQSX4qNsB81q2jSmMkDMIO2peE= 2024-02-01T10:27:44.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:44 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:TLQSX4qNsB81q2jSmMkDMIO2peE=', 'Content-Length': '0'} 2024-02-01T10:27:44.133 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004cd93037da75ed9c-0065bb7220-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:44 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:44.134 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:44.134 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:44.134 DEBUG:boto:path=/vclfcn-19/obj?versionId=oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r 2024-02-01T10:27:44.134 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r 2024-02-01T10:27:44.134 DEBUG:boto:Method: GET 2024-02-01T10:27:44.134 DEBUG:boto:Path: /vclfcn-19/obj?versionId=oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r 2024-02-01T10:27:44.134 DEBUG:boto:Data: 2024-02-01T10:27:44.134 DEBUG:boto:Headers: {} 2024-02-01T10:27:44.134 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:44.134 DEBUG:boto:Port: 8000 2024-02-01T10:27:44.134 DEBUG:boto:Params: {} 2024-02-01T10:27:44.135 DEBUG:boto:Token: None 2024-02-01T10:27:44.135 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:44 GMT /vclfcn-19/obj?versionId=oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r 2024-02-01T10:27:44.135 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:TLQSX4qNsB81q2jSmMkDMIO2peE= 2024-02-01T10:27:44.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:44 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:TLQSX4qNsB81q2jSmMkDMIO2peE=', 'Content-Length': '0'} 2024-02-01T10:27:44.138 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000281f5c7b25645901-0065bb7220-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:44 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:44.184 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:44.184 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:44.184 DEBUG:boto:path=/vclfcn-19/obj?versionId=oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r 2024-02-01T10:27:44.184 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r 2024-02-01T10:27:44.185 DEBUG:boto:Method: HEAD 2024-02-01T10:27:44.185 DEBUG:boto:Path: /vclfcn-19/obj?versionId=oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r 2024-02-01T10:27:44.185 DEBUG:boto:Data: 2024-02-01T10:27:44.185 DEBUG:boto:Headers: {} 2024-02-01T10:27:44.185 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:44.185 DEBUG:boto:Port: 8001 2024-02-01T10:27:44.185 DEBUG:boto:Params: {} 2024-02-01T10:27:44.185 DEBUG:boto:Token: None 2024-02-01T10:27:44.185 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:44 GMT /vclfcn-19/obj?versionId=oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r 2024-02-01T10:27:44.186 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/elcLRq6uDJoVllmjzndS9qq9eU= 2024-02-01T10:27:44.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:44 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/elcLRq6uDJoVllmjzndS9qq9eU=', 'Content-Length': '0'} 2024-02-01T10:27:44.189 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006738ffe2816c9118-0065bb7220-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:44 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:44.189 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:44.189 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:44.189 DEBUG:boto:path=/vclfcn-19/obj?versionId=oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r 2024-02-01T10:27:44.189 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r 2024-02-01T10:27:44.189 DEBUG:boto:Method: HEAD 2024-02-01T10:27:44.189 DEBUG:boto:Path: /vclfcn-19/obj?versionId=oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r 2024-02-01T10:27:44.189 DEBUG:boto:Data: 2024-02-01T10:27:44.190 DEBUG:boto:Headers: {} 2024-02-01T10:27:44.190 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:44.190 DEBUG:boto:Port: 8000 2024-02-01T10:27:44.190 DEBUG:boto:Params: {} 2024-02-01T10:27:44.190 DEBUG:boto:Token: None 2024-02-01T10:27:44.190 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:44 GMT /vclfcn-19/obj?versionId=oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r 2024-02-01T10:27:44.190 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/elcLRq6uDJoVllmjzndS9qq9eU= 2024-02-01T10:27:44.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:44 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/elcLRq6uDJoVllmjzndS9qq9eU=', 'Content-Length': '0'} 2024-02-01T10:27:44.193 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bf8627f4833cbd77-0065bb7220-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:44 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:44.194 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:44.194 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:44.194 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:44.194 DEBUG:boto:path=/vclfcn-19/obj?versionId=oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r 2024-02-01T10:27:44.194 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r 2024-02-01T10:27:44.194 DEBUG:boto:Method: GET 2024-02-01T10:27:44.194 DEBUG:boto:Path: /vclfcn-19/obj?versionId=oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r 2024-02-01T10:27:44.194 DEBUG:boto:Data: 2024-02-01T10:27:44.194 DEBUG:boto:Headers: {} 2024-02-01T10:27:44.194 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:44.195 DEBUG:boto:Port: 8001 2024-02-01T10:27:44.195 DEBUG:boto:Params: {} 2024-02-01T10:27:44.195 DEBUG:boto:Token: None 2024-02-01T10:27:44.195 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:44 GMT /vclfcn-19/obj?versionId=oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r 2024-02-01T10:27:44.195 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:TLQSX4qNsB81q2jSmMkDMIO2peE= 2024-02-01T10:27:44.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:44 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:TLQSX4qNsB81q2jSmMkDMIO2peE=', 'Content-Length': '0'} 2024-02-01T10:27:44.198 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f8453ce832326581-0065bb7220-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:44 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:44.198 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:44.198 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:44.198 DEBUG:boto:path=/vclfcn-19/obj?versionId=oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r 2024-02-01T10:27:44.199 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r 2024-02-01T10:27:44.199 DEBUG:boto:Method: GET 2024-02-01T10:27:44.199 DEBUG:boto:Path: /vclfcn-19/obj?versionId=oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r 2024-02-01T10:27:44.199 DEBUG:boto:Data: 2024-02-01T10:27:44.199 DEBUG:boto:Headers: {} 2024-02-01T10:27:44.199 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:44.199 DEBUG:boto:Port: 8000 2024-02-01T10:27:44.199 DEBUG:boto:Params: {} 2024-02-01T10:27:44.199 DEBUG:boto:Token: None 2024-02-01T10:27:44.199 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:44 GMT /vclfcn-19/obj?versionId=oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r 2024-02-01T10:27:44.200 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:TLQSX4qNsB81q2jSmMkDMIO2peE= 2024-02-01T10:27:44.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:44 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:TLQSX4qNsB81q2jSmMkDMIO2peE=', 'Content-Length': '0'} 2024-02-01T10:27:44.203 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b5501efa6905eb14-0065bb7220-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:44 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:44.244 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:44.244 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:44.245 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:44.245 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:44.245 DEBUG:boto:Method: GET 2024-02-01T10:27:44.245 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:44.245 DEBUG:boto:Data: 2024-02-01T10:27:44.245 DEBUG:boto:Headers: {} 2024-02-01T10:27:44.245 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:44.245 DEBUG:boto:Port: 8001 2024-02-01T10:27:44.245 DEBUG:boto:Params: {} 2024-02-01T10:27:44.245 DEBUG:boto:Token: None 2024-02-01T10:27:44.246 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:44 GMT /vclfcn-19/ 2024-02-01T10:27:44.246 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:yGN7E6zLD69D8A0hKqzZH3AAuj8= 2024-02-01T10:27:44.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:44 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:yGN7E6zLD69D8A0hKqzZH3AAuj8=', 'Content-Length': '0'} 2024-02-01T10:27:44.251 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082eca3df7044cd4b-0065bb7220-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:44 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:44.251 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:44.252 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:44.252 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:44.252 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:44.252 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:44.252 DEBUG:boto:Method: GET 2024-02-01T10:27:44.252 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:44.252 DEBUG:boto:Data: 2024-02-01T10:27:44.253 DEBUG:boto:Headers: {} 2024-02-01T10:27:44.253 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:44.253 DEBUG:boto:Port: 8000 2024-02-01T10:27:44.253 DEBUG:boto:Params: {} 2024-02-01T10:27:44.253 DEBUG:boto:Token: None 2024-02-01T10:27:44.253 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:44 GMT /vclfcn-19/ 2024-02-01T10:27:44.253 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:yGN7E6zLD69D8A0hKqzZH3AAuj8= 2024-02-01T10:27:44.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:44 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:yGN7E6zLD69D8A0hKqzZH3AAuj8=', 'Content-Length': '0'} 2024-02-01T10:27:44.259 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a0a7d110d080923f-0065bb7220-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:44 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:44.259 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:44.260 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:44.260 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:44.260 DEBUG:boto:Method: HEAD 2024-02-01T10:27:44.260 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:44.260 DEBUG:boto:Data: 2024-02-01T10:27:44.260 DEBUG:boto:Headers: {} 2024-02-01T10:27:44.260 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:44.260 DEBUG:boto:Port: 8001 2024-02-01T10:27:44.260 DEBUG:boto:Params: {} 2024-02-01T10:27:44.261 DEBUG:boto:Token: None 2024-02-01T10:27:44.261 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:44 GMT /vclfcn-19/obj 2024-02-01T10:27:44.261 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0LnRLsu1pjIvQRxywmNklEozGBk= 2024-02-01T10:27:44.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:44 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0LnRLsu1pjIvQRxywmNklEozGBk=', 'Content-Length': '0'} 2024-02-01T10:27:44.267 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e464880576adde7f-0065bb7220-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:44 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:44.267 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:44.267 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:44.267 DEBUG:boto:Method: HEAD 2024-02-01T10:27:44.267 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:44.267 DEBUG:boto:Data: 2024-02-01T10:27:44.267 DEBUG:boto:Headers: {} 2024-02-01T10:27:44.267 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:44.268 DEBUG:boto:Port: 8000 2024-02-01T10:27:44.268 DEBUG:boto:Params: {} 2024-02-01T10:27:44.268 DEBUG:boto:Token: None 2024-02-01T10:27:44.268 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:44 GMT /vclfcn-19/obj 2024-02-01T10:27:44.268 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0LnRLsu1pjIvQRxywmNklEozGBk= 2024-02-01T10:27:44.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:44 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0LnRLsu1pjIvQRxywmNklEozGBk=', 'Content-Length': '0'} 2024-02-01T10:27:44.274 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000154cbc116c6ba90e-0065bb7220-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:44 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:44.274 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:44.274 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:44.274 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:44.275 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:44.275 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:44.275 DEBUG:boto:Method: GET 2024-02-01T10:27:44.275 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:44.275 DEBUG:boto:Data: 2024-02-01T10:27:44.275 DEBUG:boto:Headers: {} 2024-02-01T10:27:44.275 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:44.275 DEBUG:boto:Port: 8001 2024-02-01T10:27:44.275 DEBUG:boto:Params: {} 2024-02-01T10:27:44.275 DEBUG:boto:Token: None 2024-02-01T10:27:44.275 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:44 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:44.276 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:n4zWKMInJexpRe6UprkQ+QrO+PI= 2024-02-01T10:27:44.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:44 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:n4zWKMInJexpRe6UprkQ+QrO+PI=', 'Content-Length': '0'} 2024-02-01T10:27:44.278 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000d2a9e869d70f588-0065bb7220-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:44 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:44.320 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:44.320 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:44.320 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:44.320 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:44.321 DEBUG:boto:Method: GET 2024-02-01T10:27:44.321 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:44.321 DEBUG:boto:Data: 2024-02-01T10:27:44.321 DEBUG:boto:Headers: {} 2024-02-01T10:27:44.321 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:44.321 DEBUG:boto:Port: 8000 2024-02-01T10:27:44.321 DEBUG:boto:Params: {} 2024-02-01T10:27:44.321 DEBUG:boto:Token: None 2024-02-01T10:27:44.321 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:44 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:44.322 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:n4zWKMInJexpRe6UprkQ+QrO+PI= 2024-02-01T10:27:44.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:44 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:n4zWKMInJexpRe6UprkQ+QrO+PI=', 'Content-Length': '0'} 2024-02-01T10:27:44.324 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000807687ea49d256f3-0065bb7220-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:44 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:44.368 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:44.368 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:44.368 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:44.369 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:44.369 DEBUG:boto:Method: GET 2024-02-01T10:27:44.369 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:44.369 DEBUG:boto:Data: 2024-02-01T10:27:44.369 DEBUG:boto:Headers: {} 2024-02-01T10:27:44.369 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:44.369 DEBUG:boto:Port: 8001 2024-02-01T10:27:44.369 DEBUG:boto:Params: {} 2024-02-01T10:27:44.369 DEBUG:boto:Token: None 2024-02-01T10:27:44.369 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:44 GMT /vclfcn-19/ 2024-02-01T10:27:44.370 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:yGN7E6zLD69D8A0hKqzZH3AAuj8= 2024-02-01T10:27:44.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:44 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:yGN7E6zLD69D8A0hKqzZH3AAuj8=', 'Content-Length': '0'} 2024-02-01T10:27:44.375 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007da3f2d836aac533-0065bb7220-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:44 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:44.375 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:44.375 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:44.376 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:44.376 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:44.376 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:44.376 DEBUG:boto:Method: GET 2024-02-01T10:27:44.376 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:44.376 DEBUG:boto:Data: 2024-02-01T10:27:44.376 DEBUG:boto:Headers: {} 2024-02-01T10:27:44.376 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:44.376 DEBUG:boto:Port: 8000 2024-02-01T10:27:44.376 DEBUG:boto:Params: {} 2024-02-01T10:27:44.377 DEBUG:boto:Token: None 2024-02-01T10:27:44.377 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:44 GMT /vclfcn-19/ 2024-02-01T10:27:44.377 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:yGN7E6zLD69D8A0hKqzZH3AAuj8= 2024-02-01T10:27:44.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:44 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:yGN7E6zLD69D8A0hKqzZH3AAuj8=', 'Content-Length': '0'} 2024-02-01T10:27:44.382 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063fdb2bb39cd09dc-0065bb7220-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:44 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:44.382 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:44.383 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:44.383 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:44.383 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:44.383 DEBUG:boto:path=/vclfcn-19/obj?versionId=qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9 2024-02-01T10:27:44.383 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9 2024-02-01T10:27:44.383 DEBUG:boto:Method: GET 2024-02-01T10:27:44.383 DEBUG:boto:Path: /vclfcn-19/obj?versionId=qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9 2024-02-01T10:27:44.383 DEBUG:boto:Data: 2024-02-01T10:27:44.383 DEBUG:boto:Headers: {} 2024-02-01T10:27:44.383 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:44.383 DEBUG:boto:Port: 8001 2024-02-01T10:27:44.384 DEBUG:boto:Params: {} 2024-02-01T10:27:44.384 DEBUG:boto:Token: None 2024-02-01T10:27:44.384 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:44 GMT /vclfcn-19/obj?versionId=qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9 2024-02-01T10:27:44.384 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:MTdYghaufzX62bi4nq/i6Yuy3no= 2024-02-01T10:27:44.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:44 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:MTdYghaufzX62bi4nq/i6Yuy3no=', 'Content-Length': '0'} 2024-02-01T10:27:44.387 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000072b8759d0d83144c-0065bb7220-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:44 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:44.388 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:44.388 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:44.388 DEBUG:boto:path=/vclfcn-19/obj?versionId=qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9 2024-02-01T10:27:44.388 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9 2024-02-01T10:27:44.388 DEBUG:boto:Method: GET 2024-02-01T10:27:44.388 DEBUG:boto:Path: /vclfcn-19/obj?versionId=qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9 2024-02-01T10:27:44.388 DEBUG:boto:Data: 2024-02-01T10:27:44.388 DEBUG:boto:Headers: {} 2024-02-01T10:27:44.388 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:44.388 DEBUG:boto:Port: 8000 2024-02-01T10:27:44.388 DEBUG:boto:Params: {} 2024-02-01T10:27:44.389 DEBUG:boto:Token: None 2024-02-01T10:27:44.389 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:44 GMT /vclfcn-19/obj?versionId=qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9 2024-02-01T10:27:44.389 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:MTdYghaufzX62bi4nq/i6Yuy3no= 2024-02-01T10:27:44.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:44 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:MTdYghaufzX62bi4nq/i6Yuy3no=', 'Content-Length': '0'} 2024-02-01T10:27:44.392 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f0ae399839d936f2-0065bb7220-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:44 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:44.392 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:44.392 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:44.393 DEBUG:boto:path=/vclfcn-19/obj?versionId=qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9 2024-02-01T10:27:44.393 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9 2024-02-01T10:27:44.393 DEBUG:boto:Method: HEAD 2024-02-01T10:27:44.393 DEBUG:boto:Path: /vclfcn-19/obj?versionId=qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9 2024-02-01T10:27:44.393 DEBUG:boto:Data: 2024-02-01T10:27:44.393 DEBUG:boto:Headers: {} 2024-02-01T10:27:44.393 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:44.393 DEBUG:boto:Port: 8001 2024-02-01T10:27:44.393 DEBUG:boto:Params: {} 2024-02-01T10:27:44.393 DEBUG:boto:Token: None 2024-02-01T10:27:44.394 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:44 GMT /vclfcn-19/obj?versionId=qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9 2024-02-01T10:27:44.394 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ajxu18Egd+qf3mJHRANijxsYuO0= 2024-02-01T10:27:44.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:44 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ajxu18Egd+qf3mJHRANijxsYuO0=', 'Content-Length': '0'} 2024-02-01T10:27:44.397 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000038e0a91b677ab216-0065bb7220-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:44 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:44.397 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:44.397 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:44.397 DEBUG:boto:path=/vclfcn-19/obj?versionId=qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9 2024-02-01T10:27:44.397 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9 2024-02-01T10:27:44.397 DEBUG:boto:Method: HEAD 2024-02-01T10:27:44.397 DEBUG:boto:Path: /vclfcn-19/obj?versionId=qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9 2024-02-01T10:27:44.397 DEBUG:boto:Data: 2024-02-01T10:27:44.397 DEBUG:boto:Headers: {} 2024-02-01T10:27:44.397 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:44.398 DEBUG:boto:Port: 8000 2024-02-01T10:27:44.398 DEBUG:boto:Params: {} 2024-02-01T10:27:44.398 DEBUG:boto:Token: None 2024-02-01T10:27:44.398 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:44 GMT /vclfcn-19/obj?versionId=qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9 2024-02-01T10:27:44.398 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ajxu18Egd+qf3mJHRANijxsYuO0= 2024-02-01T10:27:44.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:44 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ajxu18Egd+qf3mJHRANijxsYuO0=', 'Content-Length': '0'} 2024-02-01T10:27:44.401 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000275e640a2e7ab491-0065bb7220-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:44 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:44.401 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:44.401 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:44.401 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:44.401 DEBUG:boto:path=/vclfcn-19/obj?versionId=qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9 2024-02-01T10:27:44.401 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9 2024-02-01T10:27:44.402 DEBUG:boto:Method: GET 2024-02-01T10:27:44.402 DEBUG:boto:Path: /vclfcn-19/obj?versionId=qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9 2024-02-01T10:27:44.402 DEBUG:boto:Data: 2024-02-01T10:27:44.402 DEBUG:boto:Headers: {} 2024-02-01T10:27:44.402 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:44.402 DEBUG:boto:Port: 8001 2024-02-01T10:27:44.402 DEBUG:boto:Params: {} 2024-02-01T10:27:44.402 DEBUG:boto:Token: None 2024-02-01T10:27:44.402 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:44 GMT /vclfcn-19/obj?versionId=qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9 2024-02-01T10:27:44.403 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:MTdYghaufzX62bi4nq/i6Yuy3no= 2024-02-01T10:27:44.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:44 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:MTdYghaufzX62bi4nq/i6Yuy3no=', 'Content-Length': '0'} 2024-02-01T10:27:44.406 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d2397bccaefad6bc-0065bb7220-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:44 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:44.406 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:44.406 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:44.406 DEBUG:boto:path=/vclfcn-19/obj?versionId=qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9 2024-02-01T10:27:44.406 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9 2024-02-01T10:27:44.406 DEBUG:boto:Method: GET 2024-02-01T10:27:44.406 DEBUG:boto:Path: /vclfcn-19/obj?versionId=qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9 2024-02-01T10:27:44.407 DEBUG:boto:Data: 2024-02-01T10:27:44.407 DEBUG:boto:Headers: {} 2024-02-01T10:27:44.407 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:44.407 DEBUG:boto:Port: 8000 2024-02-01T10:27:44.407 DEBUG:boto:Params: {} 2024-02-01T10:27:44.407 DEBUG:boto:Token: None 2024-02-01T10:27:44.407 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:44 GMT /vclfcn-19/obj?versionId=qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9 2024-02-01T10:27:44.408 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:MTdYghaufzX62bi4nq/i6Yuy3no= 2024-02-01T10:27:44.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:44 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:MTdYghaufzX62bi4nq/i6Yuy3no=', 'Content-Length': '0'} 2024-02-01T10:27:44.411 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000ba4ca94b14cda04-0065bb7220-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:44 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:44.452 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:44.452 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:44.452 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:44.453 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:44.453 DEBUG:boto:Method: GET 2024-02-01T10:27:44.453 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:44.453 DEBUG:boto:Data: 2024-02-01T10:27:44.453 DEBUG:boto:Headers: {} 2024-02-01T10:27:44.453 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:44.453 DEBUG:boto:Port: 8001 2024-02-01T10:27:44.453 DEBUG:boto:Params: {} 2024-02-01T10:27:44.453 DEBUG:boto:Token: None 2024-02-01T10:27:44.453 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:44 GMT /vclfcn-19/ 2024-02-01T10:27:44.454 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:yGN7E6zLD69D8A0hKqzZH3AAuj8= 2024-02-01T10:27:44.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:44 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:yGN7E6zLD69D8A0hKqzZH3AAuj8=', 'Content-Length': '0'} 2024-02-01T10:27:44.459 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c376323f7be45dd6-0065bb7220-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:44 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:44.459 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:44.460 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:44.460 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:44.460 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:44.460 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:44.460 DEBUG:boto:Method: GET 2024-02-01T10:27:44.460 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:44.460 DEBUG:boto:Data: 2024-02-01T10:27:44.461 DEBUG:boto:Headers: {} 2024-02-01T10:27:44.461 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:44.461 DEBUG:boto:Port: 8000 2024-02-01T10:27:44.461 DEBUG:boto:Params: {} 2024-02-01T10:27:44.461 DEBUG:boto:Token: None 2024-02-01T10:27:44.461 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:44 GMT /vclfcn-19/ 2024-02-01T10:27:44.461 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:yGN7E6zLD69D8A0hKqzZH3AAuj8= 2024-02-01T10:27:44.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:44 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:yGN7E6zLD69D8A0hKqzZH3AAuj8=', 'Content-Length': '0'} 2024-02-01T10:27:44.467 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007fcfce8948c3db97-0065bb7220-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:44 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:44.467 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:44.468 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:44.468 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:44.468 DEBUG:boto:Method: HEAD 2024-02-01T10:27:44.468 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:44.468 DEBUG:boto:Data: 2024-02-01T10:27:44.468 DEBUG:boto:Headers: {} 2024-02-01T10:27:44.468 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:44.468 DEBUG:boto:Port: 8001 2024-02-01T10:27:44.468 DEBUG:boto:Params: {} 2024-02-01T10:27:44.469 DEBUG:boto:Token: None 2024-02-01T10:27:44.469 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:44 GMT /vclfcn-19/obj 2024-02-01T10:27:44.469 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0LnRLsu1pjIvQRxywmNklEozGBk= 2024-02-01T10:27:44.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:44 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0LnRLsu1pjIvQRxywmNklEozGBk=', 'Content-Length': '0'} 2024-02-01T10:27:44.474 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d815af4c0efbfe1a-0065bb7220-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:44 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:44.474 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:44.475 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:44.475 DEBUG:boto:Method: HEAD 2024-02-01T10:27:44.475 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:44.475 DEBUG:boto:Data: 2024-02-01T10:27:44.475 DEBUG:boto:Headers: {} 2024-02-01T10:27:44.475 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:44.475 DEBUG:boto:Port: 8000 2024-02-01T10:27:44.475 DEBUG:boto:Params: {} 2024-02-01T10:27:44.475 DEBUG:boto:Token: None 2024-02-01T10:27:44.475 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:44 GMT /vclfcn-19/obj 2024-02-01T10:27:44.476 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0LnRLsu1pjIvQRxywmNklEozGBk= 2024-02-01T10:27:44.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:44 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0LnRLsu1pjIvQRxywmNklEozGBk=', 'Content-Length': '0'} 2024-02-01T10:27:44.482 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a469299556aff9e6-0065bb7220-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:44 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:44.482 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:44.482 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:44.482 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:44.483 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:44.483 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:44.483 DEBUG:boto:Method: GET 2024-02-01T10:27:44.483 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:44.483 DEBUG:boto:Data: 2024-02-01T10:27:44.483 DEBUG:boto:Headers: {} 2024-02-01T10:27:44.483 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:44.483 DEBUG:boto:Port: 8001 2024-02-01T10:27:44.483 DEBUG:boto:Params: {} 2024-02-01T10:27:44.483 DEBUG:boto:Token: None 2024-02-01T10:27:44.484 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:44 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:44.484 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:n4zWKMInJexpRe6UprkQ+QrO+PI= 2024-02-01T10:27:44.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:44 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:n4zWKMInJexpRe6UprkQ+QrO+PI=', 'Content-Length': '0'} 2024-02-01T10:27:44.487 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000789d66d36920be3-0065bb7220-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:44 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:44.528 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:44.529 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:44.529 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:44.529 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:44.529 DEBUG:boto:Method: GET 2024-02-01T10:27:44.529 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:44.529 DEBUG:boto:Data: 2024-02-01T10:27:44.529 DEBUG:boto:Headers: {} 2024-02-01T10:27:44.529 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:44.529 DEBUG:boto:Port: 8000 2024-02-01T10:27:44.529 DEBUG:boto:Params: {} 2024-02-01T10:27:44.530 DEBUG:boto:Token: None 2024-02-01T10:27:44.530 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:44 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:44.530 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:n4zWKMInJexpRe6UprkQ+QrO+PI= 2024-02-01T10:27:44.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:44 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:n4zWKMInJexpRe6UprkQ+QrO+PI=', 'Content-Length': '0'} 2024-02-01T10:27:44.533 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000093119fe307ebc303-0065bb7220-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:44 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:44.576 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:44.576 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:44.576 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:44.577 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:44.577 DEBUG:boto:Method: GET 2024-02-01T10:27:44.577 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:44.577 DEBUG:boto:Data: 2024-02-01T10:27:44.577 DEBUG:boto:Headers: {} 2024-02-01T10:27:44.577 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:44.577 DEBUG:boto:Port: 8001 2024-02-01T10:27:44.577 DEBUG:boto:Params: {} 2024-02-01T10:27:44.577 DEBUG:boto:Token: None 2024-02-01T10:27:44.578 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:44 GMT /vclfcn-19/ 2024-02-01T10:27:44.578 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:yGN7E6zLD69D8A0hKqzZH3AAuj8= 2024-02-01T10:27:44.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:44 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:yGN7E6zLD69D8A0hKqzZH3AAuj8=', 'Content-Length': '0'} 2024-02-01T10:27:44.583 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b0fd065ca367ad5-0065bb7220-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:44 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:44.583 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:44.584 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:44.584 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:44.584 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:44.584 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:44.584 DEBUG:boto:Method: GET 2024-02-01T10:27:44.584 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:44.584 DEBUG:boto:Data: 2024-02-01T10:27:44.585 DEBUG:boto:Headers: {} 2024-02-01T10:27:44.585 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:44.585 DEBUG:boto:Port: 8000 2024-02-01T10:27:44.585 DEBUG:boto:Params: {} 2024-02-01T10:27:44.585 DEBUG:boto:Token: None 2024-02-01T10:27:44.585 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:44 GMT /vclfcn-19/ 2024-02-01T10:27:44.585 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:yGN7E6zLD69D8A0hKqzZH3AAuj8= 2024-02-01T10:27:44.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:44 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:yGN7E6zLD69D8A0hKqzZH3AAuj8=', 'Content-Length': '0'} 2024-02-01T10:27:44.590 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000014a7c2ddd1364502-0065bb7220-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:44 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:44.591 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:44.591 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:44.591 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:44.591 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:44.591 DEBUG:boto:path=/vclfcn-19/obj?versionId=NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS 2024-02-01T10:27:44.591 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS 2024-02-01T10:27:44.592 DEBUG:boto:Method: GET 2024-02-01T10:27:44.592 DEBUG:boto:Path: /vclfcn-19/obj?versionId=NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS 2024-02-01T10:27:44.592 DEBUG:boto:Data: 2024-02-01T10:27:44.592 DEBUG:boto:Headers: {} 2024-02-01T10:27:44.592 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:44.592 DEBUG:boto:Port: 8001 2024-02-01T10:27:44.592 DEBUG:boto:Params: {} 2024-02-01T10:27:44.592 DEBUG:boto:Token: None 2024-02-01T10:27:44.592 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:44 GMT /vclfcn-19/obj?versionId=NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS 2024-02-01T10:27:44.593 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:H+8ScoUv/MGrK400nbUbcZwKRXA= 2024-02-01T10:27:44.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:44 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:H+8ScoUv/MGrK400nbUbcZwKRXA=', 'Content-Length': '0'} 2024-02-01T10:27:44.596 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000080566c60d2730493-0065bb7220-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:44 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:44.596 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:44.597 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:44.597 DEBUG:boto:path=/vclfcn-19/obj?versionId=NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS 2024-02-01T10:27:44.597 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS 2024-02-01T10:27:44.597 DEBUG:boto:Method: GET 2024-02-01T10:27:44.597 DEBUG:boto:Path: /vclfcn-19/obj?versionId=NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS 2024-02-01T10:27:44.597 DEBUG:boto:Data: 2024-02-01T10:27:44.598 DEBUG:boto:Headers: {} 2024-02-01T10:27:44.598 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:44.598 DEBUG:boto:Port: 8000 2024-02-01T10:27:44.598 DEBUG:boto:Params: {} 2024-02-01T10:27:44.598 DEBUG:boto:Token: None 2024-02-01T10:27:44.598 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:44 GMT /vclfcn-19/obj?versionId=NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS 2024-02-01T10:27:44.599 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:H+8ScoUv/MGrK400nbUbcZwKRXA= 2024-02-01T10:27:44.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:44 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:H+8ScoUv/MGrK400nbUbcZwKRXA=', 'Content-Length': '0'} 2024-02-01T10:27:44.602 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fdda350d8a3d5692-0065bb7220-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:44 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:44.648 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:44.648 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:44.648 DEBUG:boto:path=/vclfcn-19/obj?versionId=NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS 2024-02-01T10:27:44.649 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS 2024-02-01T10:27:44.649 DEBUG:boto:Method: HEAD 2024-02-01T10:27:44.649 DEBUG:boto:Path: /vclfcn-19/obj?versionId=NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS 2024-02-01T10:27:44.649 DEBUG:boto:Data: 2024-02-01T10:27:44.649 DEBUG:boto:Headers: {} 2024-02-01T10:27:44.649 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:44.649 DEBUG:boto:Port: 8001 2024-02-01T10:27:44.649 DEBUG:boto:Params: {} 2024-02-01T10:27:44.649 DEBUG:boto:Token: None 2024-02-01T10:27:44.650 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:44 GMT /vclfcn-19/obj?versionId=NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS 2024-02-01T10:27:44.650 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:TDJtyp5IUK2SNo5eeaIbb5rT5nM= 2024-02-01T10:27:44.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:44 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:TDJtyp5IUK2SNo5eeaIbb5rT5nM=', 'Content-Length': '0'} 2024-02-01T10:27:44.654 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000833238639f618960-0065bb7220-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:44 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:44.655 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:44.655 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:44.655 DEBUG:boto:path=/vclfcn-19/obj?versionId=NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS 2024-02-01T10:27:44.655 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS 2024-02-01T10:27:44.655 DEBUG:boto:Method: HEAD 2024-02-01T10:27:44.655 DEBUG:boto:Path: /vclfcn-19/obj?versionId=NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS 2024-02-01T10:27:44.655 DEBUG:boto:Data: 2024-02-01T10:27:44.655 DEBUG:boto:Headers: {} 2024-02-01T10:27:44.655 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:44.655 DEBUG:boto:Port: 8000 2024-02-01T10:27:44.656 DEBUG:boto:Params: {} 2024-02-01T10:27:44.656 DEBUG:boto:Token: None 2024-02-01T10:27:44.656 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:44 GMT /vclfcn-19/obj?versionId=NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS 2024-02-01T10:27:44.656 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:TDJtyp5IUK2SNo5eeaIbb5rT5nM= 2024-02-01T10:27:44.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:44 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:TDJtyp5IUK2SNo5eeaIbb5rT5nM=', 'Content-Length': '0'} 2024-02-01T10:27:44.664 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000edb3a26d220e8076-0065bb7220-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:44 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:44.665 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:44.665 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:44.665 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:44.665 DEBUG:boto:path=/vclfcn-19/obj?versionId=NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS 2024-02-01T10:27:44.665 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS 2024-02-01T10:27:44.665 DEBUG:boto:Method: GET 2024-02-01T10:27:44.665 DEBUG:boto:Path: /vclfcn-19/obj?versionId=NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS 2024-02-01T10:27:44.665 DEBUG:boto:Data: 2024-02-01T10:27:44.665 DEBUG:boto:Headers: {} 2024-02-01T10:27:44.666 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:44.666 DEBUG:boto:Port: 8001 2024-02-01T10:27:44.666 DEBUG:boto:Params: {} 2024-02-01T10:27:44.666 DEBUG:boto:Token: None 2024-02-01T10:27:44.666 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:44 GMT /vclfcn-19/obj?versionId=NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS 2024-02-01T10:27:44.666 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:H+8ScoUv/MGrK400nbUbcZwKRXA= 2024-02-01T10:27:44.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:44 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:H+8ScoUv/MGrK400nbUbcZwKRXA=', 'Content-Length': '0'} 2024-02-01T10:27:44.675 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000007ac84d27fd665de-0065bb7220-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:44 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:44.677 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:44.677 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:44.677 DEBUG:boto:path=/vclfcn-19/obj?versionId=NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS 2024-02-01T10:27:44.677 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS 2024-02-01T10:27:44.677 DEBUG:boto:Method: GET 2024-02-01T10:27:44.677 DEBUG:boto:Path: /vclfcn-19/obj?versionId=NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS 2024-02-01T10:27:44.677 DEBUG:boto:Data: 2024-02-01T10:27:44.677 DEBUG:boto:Headers: {} 2024-02-01T10:27:44.678 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:44.678 DEBUG:boto:Port: 8000 2024-02-01T10:27:44.678 DEBUG:boto:Params: {} 2024-02-01T10:27:44.678 DEBUG:boto:Token: None 2024-02-01T10:27:44.678 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:44 GMT /vclfcn-19/obj?versionId=NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS 2024-02-01T10:27:44.678 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:H+8ScoUv/MGrK400nbUbcZwKRXA= 2024-02-01T10:27:44.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:44 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:H+8ScoUv/MGrK400nbUbcZwKRXA=', 'Content-Length': '0'} 2024-02-01T10:27:44.683 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dec6f095b2a1c25c-0065bb7220-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:44 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:44.731 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:44.731 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:44.732 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:44.732 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:44.732 DEBUG:boto:Method: GET 2024-02-01T10:27:44.732 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:44.733 DEBUG:boto:Data: 2024-02-01T10:27:44.733 DEBUG:boto:Headers: {} 2024-02-01T10:27:44.733 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:44.733 DEBUG:boto:Port: 8001 2024-02-01T10:27:44.733 DEBUG:boto:Params: {} 2024-02-01T10:27:44.734 DEBUG:boto:Token: None 2024-02-01T10:27:44.734 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:44 GMT /vclfcn-19/ 2024-02-01T10:27:44.735 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:yGN7E6zLD69D8A0hKqzZH3AAuj8= 2024-02-01T10:27:44.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:44 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:yGN7E6zLD69D8A0hKqzZH3AAuj8=', 'Content-Length': '0'} 2024-02-01T10:27:44.748 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007564e3360f701044-0065bb7220-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:44 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:44.748 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:44.749 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:44.749 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:44.749 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:44.749 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:44.750 DEBUG:boto:Method: GET 2024-02-01T10:27:44.750 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:44.750 DEBUG:boto:Data: 2024-02-01T10:27:44.750 DEBUG:boto:Headers: {} 2024-02-01T10:27:44.750 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:44.750 DEBUG:boto:Port: 8000 2024-02-01T10:27:44.750 DEBUG:boto:Params: {} 2024-02-01T10:27:44.750 DEBUG:boto:Token: None 2024-02-01T10:27:44.750 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:44 GMT /vclfcn-19/ 2024-02-01T10:27:44.751 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:yGN7E6zLD69D8A0hKqzZH3AAuj8= 2024-02-01T10:27:44.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:44 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:yGN7E6zLD69D8A0hKqzZH3AAuj8=', 'Content-Length': '0'} 2024-02-01T10:27:44.756 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da045dbc413dfe1b-0065bb7220-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:44 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:44.757 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:44.758 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:44.758 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:44.758 DEBUG:boto:Method: HEAD 2024-02-01T10:27:44.759 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:44.759 DEBUG:boto:Data: 2024-02-01T10:27:44.759 DEBUG:boto:Headers: {} 2024-02-01T10:27:44.759 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:44.759 DEBUG:boto:Port: 8001 2024-02-01T10:27:44.759 DEBUG:boto:Params: {} 2024-02-01T10:27:44.760 DEBUG:boto:Token: None 2024-02-01T10:27:44.760 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:44 GMT /vclfcn-19/obj 2024-02-01T10:27:44.760 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0LnRLsu1pjIvQRxywmNklEozGBk= 2024-02-01T10:27:44.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:44 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0LnRLsu1pjIvQRxywmNklEozGBk=', 'Content-Length': '0'} 2024-02-01T10:27:44.766 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000058837c8d5ebfcaca-0065bb7220-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:44 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:44.767 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:44.767 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:44.767 DEBUG:boto:Method: HEAD 2024-02-01T10:27:44.767 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:44.767 DEBUG:boto:Data: 2024-02-01T10:27:44.767 DEBUG:boto:Headers: {} 2024-02-01T10:27:44.767 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:44.768 DEBUG:boto:Port: 8000 2024-02-01T10:27:44.768 DEBUG:boto:Params: {} 2024-02-01T10:27:44.768 DEBUG:boto:Token: None 2024-02-01T10:27:44.769 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:44 GMT /vclfcn-19/obj 2024-02-01T10:27:44.769 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0LnRLsu1pjIvQRxywmNklEozGBk= 2024-02-01T10:27:44.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:44 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0LnRLsu1pjIvQRxywmNklEozGBk=', 'Content-Length': '0'} 2024-02-01T10:27:44.774 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007ce8845a16f25a07-0065bb7220-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:44 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:44.775 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:44.775 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:44.775 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:44.775 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:44.775 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:44.775 DEBUG:boto:Method: GET 2024-02-01T10:27:44.775 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:44.775 DEBUG:boto:Data: 2024-02-01T10:27:44.776 DEBUG:boto:Headers: {} 2024-02-01T10:27:44.776 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:44.776 DEBUG:boto:Port: 8001 2024-02-01T10:27:44.776 DEBUG:boto:Params: {} 2024-02-01T10:27:44.776 DEBUG:boto:Token: None 2024-02-01T10:27:44.776 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:44 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:44.776 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:n4zWKMInJexpRe6UprkQ+QrO+PI= 2024-02-01T10:27:44.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:44 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:n4zWKMInJexpRe6UprkQ+QrO+PI=', 'Content-Length': '0'} 2024-02-01T10:27:44.782 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000056efd0ef593e5380-0065bb7220-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:44 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:44.825 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:44.825 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:44.825 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:44.825 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:44.825 DEBUG:boto:Method: GET 2024-02-01T10:27:44.826 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:44.826 DEBUG:boto:Data: 2024-02-01T10:27:44.826 DEBUG:boto:Headers: {} 2024-02-01T10:27:44.826 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:44.826 DEBUG:boto:Port: 8000 2024-02-01T10:27:44.826 DEBUG:boto:Params: {} 2024-02-01T10:27:44.826 DEBUG:boto:Token: None 2024-02-01T10:27:44.826 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:44 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:44.827 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:n4zWKMInJexpRe6UprkQ+QrO+PI= 2024-02-01T10:27:44.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:44 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:n4zWKMInJexpRe6UprkQ+QrO+PI=', 'Content-Length': '0'} 2024-02-01T10:27:44.832 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000069970bdf50be4002-0065bb7220-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:44 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:44.874 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:44.874 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:44.874 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:44.874 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:44.875 DEBUG:boto:Method: GET 2024-02-01T10:27:44.875 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:44.875 DEBUG:boto:Data: 2024-02-01T10:27:44.875 DEBUG:boto:Headers: {} 2024-02-01T10:27:44.875 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:44.875 DEBUG:boto:Port: 8001 2024-02-01T10:27:44.875 DEBUG:boto:Params: {} 2024-02-01T10:27:44.875 DEBUG:boto:Token: None 2024-02-01T10:27:44.876 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:44 GMT /vclfcn-19/ 2024-02-01T10:27:44.876 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:yGN7E6zLD69D8A0hKqzZH3AAuj8= 2024-02-01T10:27:44.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:44 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:yGN7E6zLD69D8A0hKqzZH3AAuj8=', 'Content-Length': '0'} 2024-02-01T10:27:44.884 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d00f36fded8d4553-0065bb7220-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:44 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:44.885 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:44.885 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:44.886 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:44.886 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:44.886 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:44.886 DEBUG:boto:Method: GET 2024-02-01T10:27:44.886 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:44.886 DEBUG:boto:Data: 2024-02-01T10:27:44.886 DEBUG:boto:Headers: {} 2024-02-01T10:27:44.886 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:44.886 DEBUG:boto:Port: 8000 2024-02-01T10:27:44.886 DEBUG:boto:Params: {} 2024-02-01T10:27:44.886 DEBUG:boto:Token: None 2024-02-01T10:27:44.887 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:44 GMT /vclfcn-19/ 2024-02-01T10:27:44.887 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:yGN7E6zLD69D8A0hKqzZH3AAuj8= 2024-02-01T10:27:44.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:44 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:yGN7E6zLD69D8A0hKqzZH3AAuj8=', 'Content-Length': '0'} 2024-02-01T10:27:44.892 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005cadf4514e4fb9b8-0065bb7220-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:44 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:44.892 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:44.892 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:44.893 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:44.893 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:44.893 DEBUG:boto:path=/vclfcn-19/obj?versionId=VQT2kBCkF07dugmyrhNeRpQpR0HK7bT 2024-02-01T10:27:44.893 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=VQT2kBCkF07dugmyrhNeRpQpR0HK7bT 2024-02-01T10:27:44.893 DEBUG:boto:Method: GET 2024-02-01T10:27:44.893 DEBUG:boto:Path: /vclfcn-19/obj?versionId=VQT2kBCkF07dugmyrhNeRpQpR0HK7bT 2024-02-01T10:27:44.894 DEBUG:boto:Data: 2024-02-01T10:27:44.894 DEBUG:boto:Headers: {} 2024-02-01T10:27:44.894 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:44.894 DEBUG:boto:Port: 8001 2024-02-01T10:27:44.894 DEBUG:boto:Params: {} 2024-02-01T10:27:44.894 DEBUG:boto:Token: None 2024-02-01T10:27:44.894 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:44 GMT /vclfcn-19/obj?versionId=VQT2kBCkF07dugmyrhNeRpQpR0HK7bT 2024-02-01T10:27:44.894 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YFAzLyXb0l/kWzUfMzFa8pGPTUs= 2024-02-01T10:27:44.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:44 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YFAzLyXb0l/kWzUfMzFa8pGPTUs=', 'Content-Length': '0'} 2024-02-01T10:27:44.897 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'VQT2kBCkF07dugmyrhNeRpQpR0HK7bT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008c22cc7bd0b4eb21-0065bb7220-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:44 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:44.897 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:44.898 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:44.898 DEBUG:boto:path=/vclfcn-19/obj?versionId=VQT2kBCkF07dugmyrhNeRpQpR0HK7bT 2024-02-01T10:27:44.898 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=VQT2kBCkF07dugmyrhNeRpQpR0HK7bT 2024-02-01T10:27:44.898 DEBUG:boto:Method: GET 2024-02-01T10:27:44.898 DEBUG:boto:Path: /vclfcn-19/obj?versionId=VQT2kBCkF07dugmyrhNeRpQpR0HK7bT 2024-02-01T10:27:44.898 DEBUG:boto:Data: 2024-02-01T10:27:44.898 DEBUG:boto:Headers: {} 2024-02-01T10:27:44.898 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:44.898 DEBUG:boto:Port: 8000 2024-02-01T10:27:44.898 DEBUG:boto:Params: {} 2024-02-01T10:27:44.898 DEBUG:boto:Token: None 2024-02-01T10:27:44.899 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:44 GMT /vclfcn-19/obj?versionId=VQT2kBCkF07dugmyrhNeRpQpR0HK7bT 2024-02-01T10:27:44.899 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YFAzLyXb0l/kWzUfMzFa8pGPTUs= 2024-02-01T10:27:44.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:44 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YFAzLyXb0l/kWzUfMzFa8pGPTUs=', 'Content-Length': '0'} 2024-02-01T10:27:44.902 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'VQT2kBCkF07dugmyrhNeRpQpR0HK7bT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000393ab9c6180ac33b-0065bb7220-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:44 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:44.944 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:44.944 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:44.944 DEBUG:boto:path=/vclfcn-19/obj?versionId=VQT2kBCkF07dugmyrhNeRpQpR0HK7bT 2024-02-01T10:27:44.945 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=VQT2kBCkF07dugmyrhNeRpQpR0HK7bT 2024-02-01T10:27:44.945 DEBUG:boto:Method: HEAD 2024-02-01T10:27:44.945 DEBUG:boto:Path: /vclfcn-19/obj?versionId=VQT2kBCkF07dugmyrhNeRpQpR0HK7bT 2024-02-01T10:27:44.945 DEBUG:boto:Data: 2024-02-01T10:27:44.945 DEBUG:boto:Headers: {} 2024-02-01T10:27:44.945 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:44.945 DEBUG:boto:Port: 8001 2024-02-01T10:27:44.945 DEBUG:boto:Params: {} 2024-02-01T10:27:44.945 DEBUG:boto:Token: None 2024-02-01T10:27:44.946 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:44 GMT /vclfcn-19/obj?versionId=VQT2kBCkF07dugmyrhNeRpQpR0HK7bT 2024-02-01T10:27:44.946 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:PZ6HmVNz+pWNLr02M9KDMZSXa8I= 2024-02-01T10:27:44.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:44 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:PZ6HmVNz+pWNLr02M9KDMZSXa8I=', 'Content-Length': '0'} 2024-02-01T10:27:44.949 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'VQT2kBCkF07dugmyrhNeRpQpR0HK7bT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000593ad962e8436ba-0065bb7220-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:44 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:44.949 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:44.949 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:44.949 DEBUG:boto:path=/vclfcn-19/obj?versionId=VQT2kBCkF07dugmyrhNeRpQpR0HK7bT 2024-02-01T10:27:44.949 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=VQT2kBCkF07dugmyrhNeRpQpR0HK7bT 2024-02-01T10:27:44.949 DEBUG:boto:Method: HEAD 2024-02-01T10:27:44.949 DEBUG:boto:Path: /vclfcn-19/obj?versionId=VQT2kBCkF07dugmyrhNeRpQpR0HK7bT 2024-02-01T10:27:44.950 DEBUG:boto:Data: 2024-02-01T10:27:44.950 DEBUG:boto:Headers: {} 2024-02-01T10:27:44.950 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:44.950 DEBUG:boto:Port: 8000 2024-02-01T10:27:44.950 DEBUG:boto:Params: {} 2024-02-01T10:27:44.950 DEBUG:boto:Token: None 2024-02-01T10:27:44.950 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:44 GMT /vclfcn-19/obj?versionId=VQT2kBCkF07dugmyrhNeRpQpR0HK7bT 2024-02-01T10:27:44.950 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:PZ6HmVNz+pWNLr02M9KDMZSXa8I= 2024-02-01T10:27:44.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:44 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:PZ6HmVNz+pWNLr02M9KDMZSXa8I=', 'Content-Length': '0'} 2024-02-01T10:27:44.954 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'VQT2kBCkF07dugmyrhNeRpQpR0HK7bT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006197a351ce699667-0065bb7220-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:44 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:44.954 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:44.954 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:44.954 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:44.954 DEBUG:boto:path=/vclfcn-19/obj?versionId=VQT2kBCkF07dugmyrhNeRpQpR0HK7bT 2024-02-01T10:27:44.954 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=VQT2kBCkF07dugmyrhNeRpQpR0HK7bT 2024-02-01T10:27:44.954 DEBUG:boto:Method: GET 2024-02-01T10:27:44.955 DEBUG:boto:Path: /vclfcn-19/obj?versionId=VQT2kBCkF07dugmyrhNeRpQpR0HK7bT 2024-02-01T10:27:44.955 DEBUG:boto:Data: 2024-02-01T10:27:44.955 DEBUG:boto:Headers: {} 2024-02-01T10:27:44.955 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:44.955 DEBUG:boto:Port: 8001 2024-02-01T10:27:44.955 DEBUG:boto:Params: {} 2024-02-01T10:27:44.955 DEBUG:boto:Token: None 2024-02-01T10:27:44.955 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:44 GMT /vclfcn-19/obj?versionId=VQT2kBCkF07dugmyrhNeRpQpR0HK7bT 2024-02-01T10:27:44.955 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YFAzLyXb0l/kWzUfMzFa8pGPTUs= 2024-02-01T10:27:44.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:44 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YFAzLyXb0l/kWzUfMzFa8pGPTUs=', 'Content-Length': '0'} 2024-02-01T10:27:44.958 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'VQT2kBCkF07dugmyrhNeRpQpR0HK7bT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a563111d8067c1dc-0065bb7220-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:44 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:44.959 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:44.959 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:44.959 DEBUG:boto:path=/vclfcn-19/obj?versionId=VQT2kBCkF07dugmyrhNeRpQpR0HK7bT 2024-02-01T10:27:44.959 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=VQT2kBCkF07dugmyrhNeRpQpR0HK7bT 2024-02-01T10:27:44.959 DEBUG:boto:Method: GET 2024-02-01T10:27:44.959 DEBUG:boto:Path: /vclfcn-19/obj?versionId=VQT2kBCkF07dugmyrhNeRpQpR0HK7bT 2024-02-01T10:27:44.959 DEBUG:boto:Data: 2024-02-01T10:27:44.959 DEBUG:boto:Headers: {} 2024-02-01T10:27:44.959 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:44.959 DEBUG:boto:Port: 8000 2024-02-01T10:27:44.959 DEBUG:boto:Params: {} 2024-02-01T10:27:44.960 DEBUG:boto:Token: None 2024-02-01T10:27:44.960 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:44 GMT /vclfcn-19/obj?versionId=VQT2kBCkF07dugmyrhNeRpQpR0HK7bT 2024-02-01T10:27:44.960 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YFAzLyXb0l/kWzUfMzFa8pGPTUs= 2024-02-01T10:27:44.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:44 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YFAzLyXb0l/kWzUfMzFa8pGPTUs=', 'Content-Length': '0'} 2024-02-01T10:27:44.963 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'VQT2kBCkF07dugmyrhNeRpQpR0HK7bT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000053db20f9df38d2e0-0065bb7220-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:44 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:45.004 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:45.004 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:45.005 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:45.005 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:45.005 DEBUG:boto:Method: GET 2024-02-01T10:27:45.005 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:45.005 DEBUG:boto:Data: 2024-02-01T10:27:45.005 DEBUG:boto:Headers: {} 2024-02-01T10:27:45.005 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:45.005 DEBUG:boto:Port: 8001 2024-02-01T10:27:45.005 DEBUG:boto:Params: {} 2024-02-01T10:27:45.005 DEBUG:boto:Token: None 2024-02-01T10:27:45.005 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:45 GMT /vclfcn-19/ 2024-02-01T10:27:45.006 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QWZHvqI7UqnWbywLSlKwHcznVSI= 2024-02-01T10:27:45.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:45 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QWZHvqI7UqnWbywLSlKwHcznVSI=', 'Content-Length': '0'} 2024-02-01T10:27:45.011 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000081c5ef76ccc32c68-0065bb7221-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:45 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:45.011 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:45.011 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:45.011 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:45.012 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:45.012 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:45.012 DEBUG:boto:Method: GET 2024-02-01T10:27:45.012 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:45.012 DEBUG:boto:Data: 2024-02-01T10:27:45.012 DEBUG:boto:Headers: {} 2024-02-01T10:27:45.012 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:45.012 DEBUG:boto:Port: 8000 2024-02-01T10:27:45.012 DEBUG:boto:Params: {} 2024-02-01T10:27:45.012 DEBUG:boto:Token: None 2024-02-01T10:27:45.013 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:45 GMT /vclfcn-19/ 2024-02-01T10:27:45.013 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QWZHvqI7UqnWbywLSlKwHcznVSI= 2024-02-01T10:27:45.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:45 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QWZHvqI7UqnWbywLSlKwHcznVSI=', 'Content-Length': '0'} 2024-02-01T10:27:45.019 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000da814a1f5252c7c-0065bb7221-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:45 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:45.019 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:45.020 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:45.020 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:45.020 DEBUG:boto:Method: HEAD 2024-02-01T10:27:45.020 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:45.020 DEBUG:boto:Data: 2024-02-01T10:27:45.020 DEBUG:boto:Headers: {} 2024-02-01T10:27:45.021 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:45.021 DEBUG:boto:Port: 8001 2024-02-01T10:27:45.021 DEBUG:boto:Params: {} 2024-02-01T10:27:45.021 DEBUG:boto:Token: None 2024-02-01T10:27:45.021 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:45 GMT /vclfcn-19/obj 2024-02-01T10:27:45.021 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YRAy/+3356wpirVSvtp8e3a7Xgk= 2024-02-01T10:27:45.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:45 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YRAy/+3356wpirVSvtp8e3a7Xgk=', 'Content-Length': '0'} 2024-02-01T10:27:45.028 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001353f8c88e94d8ee-0065bb7221-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:45 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:45.028 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:45.028 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:45.028 DEBUG:boto:Method: HEAD 2024-02-01T10:27:45.028 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:45.028 DEBUG:boto:Data: 2024-02-01T10:27:45.028 DEBUG:boto:Headers: {} 2024-02-01T10:27:45.029 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:45.029 DEBUG:boto:Port: 8000 2024-02-01T10:27:45.029 DEBUG:boto:Params: {} 2024-02-01T10:27:45.029 DEBUG:boto:Token: None 2024-02-01T10:27:45.029 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:45 GMT /vclfcn-19/obj 2024-02-01T10:27:45.029 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YRAy/+3356wpirVSvtp8e3a7Xgk= 2024-02-01T10:27:45.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:45 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YRAy/+3356wpirVSvtp8e3a7Xgk=', 'Content-Length': '0'} 2024-02-01T10:27:45.035 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009313fe4ebeec523e-0065bb7221-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:45 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:45.035 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:45.036 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:45.036 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:45.036 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:45.036 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:45.036 DEBUG:boto:Method: GET 2024-02-01T10:27:45.036 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:45.036 DEBUG:boto:Data: 2024-02-01T10:27:45.036 DEBUG:boto:Headers: {} 2024-02-01T10:27:45.036 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:45.036 DEBUG:boto:Port: 8001 2024-02-01T10:27:45.036 DEBUG:boto:Params: {} 2024-02-01T10:27:45.037 DEBUG:boto:Token: None 2024-02-01T10:27:45.037 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:45 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:45.037 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:XRhX7aJ1NkHdOLL4sAlngINyt/s= 2024-02-01T10:27:45.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:45 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:XRhX7aJ1NkHdOLL4sAlngINyt/s=', 'Content-Length': '0'} 2024-02-01T10:27:45.040 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000322f8283d185fcee-0065bb7221-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:45 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:45.084 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:45.084 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:45.084 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:45.084 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:45.085 DEBUG:boto:Method: GET 2024-02-01T10:27:45.085 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:45.085 DEBUG:boto:Data: 2024-02-01T10:27:45.085 DEBUG:boto:Headers: {} 2024-02-01T10:27:45.085 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:45.085 DEBUG:boto:Port: 8000 2024-02-01T10:27:45.085 DEBUG:boto:Params: {} 2024-02-01T10:27:45.085 DEBUG:boto:Token: None 2024-02-01T10:27:45.085 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:45 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:45.086 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:XRhX7aJ1NkHdOLL4sAlngINyt/s= 2024-02-01T10:27:45.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:45 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:XRhX7aJ1NkHdOLL4sAlngINyt/s=', 'Content-Length': '0'} 2024-02-01T10:27:45.089 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a4cd5282349515cf-0065bb7221-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:45 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:45.132 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:45.133 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:45.133 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:45.133 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:45.133 DEBUG:boto:Method: GET 2024-02-01T10:27:45.133 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:45.133 DEBUG:boto:Data: 2024-02-01T10:27:45.134 DEBUG:boto:Headers: {} 2024-02-01T10:27:45.134 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:45.134 DEBUG:boto:Port: 8001 2024-02-01T10:27:45.134 DEBUG:boto:Params: {} 2024-02-01T10:27:45.134 DEBUG:boto:Token: None 2024-02-01T10:27:45.134 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:45 GMT /vclfcn-19/ 2024-02-01T10:27:45.134 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QWZHvqI7UqnWbywLSlKwHcznVSI= 2024-02-01T10:27:45.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:45 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QWZHvqI7UqnWbywLSlKwHcznVSI=', 'Content-Length': '0'} 2024-02-01T10:27:45.140 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d315ffd14648550-0065bb7221-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:45 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:45.140 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:45.141 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:45.141 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:45.141 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:45.141 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:45.141 DEBUG:boto:Method: GET 2024-02-01T10:27:45.141 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:45.141 DEBUG:boto:Data: 2024-02-01T10:27:45.141 DEBUG:boto:Headers: {} 2024-02-01T10:27:45.141 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:45.142 DEBUG:boto:Port: 8000 2024-02-01T10:27:45.142 DEBUG:boto:Params: {} 2024-02-01T10:27:45.142 DEBUG:boto:Token: None 2024-02-01T10:27:45.142 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:45 GMT /vclfcn-19/ 2024-02-01T10:27:45.142 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QWZHvqI7UqnWbywLSlKwHcznVSI= 2024-02-01T10:27:45.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:45 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QWZHvqI7UqnWbywLSlKwHcznVSI=', 'Content-Length': '0'} 2024-02-01T10:27:45.147 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050c87b6cf11dcdf7-0065bb7221-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:45 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:45.148 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:45.148 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:45.148 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:45.148 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:45.148 DEBUG:boto:path=/vclfcn-19/obj?versionId=tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi 2024-02-01T10:27:45.148 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi 2024-02-01T10:27:45.148 DEBUG:boto:Method: GET 2024-02-01T10:27:45.149 DEBUG:boto:Path: /vclfcn-19/obj?versionId=tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi 2024-02-01T10:27:45.149 DEBUG:boto:Data: 2024-02-01T10:27:45.149 DEBUG:boto:Headers: {} 2024-02-01T10:27:45.149 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:45.149 DEBUG:boto:Port: 8001 2024-02-01T10:27:45.149 DEBUG:boto:Params: {} 2024-02-01T10:27:45.149 DEBUG:boto:Token: None 2024-02-01T10:27:45.149 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:45 GMT /vclfcn-19/obj?versionId=tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi 2024-02-01T10:27:45.149 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cMo/+GQaTVWjSOugoDfMklpK9IQ= 2024-02-01T10:27:45.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:45 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cMo/+GQaTVWjSOugoDfMklpK9IQ=', 'Content-Length': '0'} 2024-02-01T10:27:45.152 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000076dc7e09303d0d69-0065bb7221-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:45 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:45.152 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:45.153 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:45.153 DEBUG:boto:path=/vclfcn-19/obj?versionId=tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi 2024-02-01T10:27:45.153 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi 2024-02-01T10:27:45.153 DEBUG:boto:Method: GET 2024-02-01T10:27:45.153 DEBUG:boto:Path: /vclfcn-19/obj?versionId=tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi 2024-02-01T10:27:45.153 DEBUG:boto:Data: 2024-02-01T10:27:45.153 DEBUG:boto:Headers: {} 2024-02-01T10:27:45.153 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:45.153 DEBUG:boto:Port: 8000 2024-02-01T10:27:45.153 DEBUG:boto:Params: {} 2024-02-01T10:27:45.154 DEBUG:boto:Token: None 2024-02-01T10:27:45.154 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:45 GMT /vclfcn-19/obj?versionId=tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi 2024-02-01T10:27:45.154 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cMo/+GQaTVWjSOugoDfMklpK9IQ= 2024-02-01T10:27:45.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:45 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cMo/+GQaTVWjSOugoDfMklpK9IQ=', 'Content-Length': '0'} 2024-02-01T10:27:45.156 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007c7bc253d038f036-0065bb7221-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:45 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:45.157 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:45.157 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:45.157 DEBUG:boto:path=/vclfcn-19/obj?versionId=tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi 2024-02-01T10:27:45.157 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi 2024-02-01T10:27:45.157 DEBUG:boto:Method: HEAD 2024-02-01T10:27:45.157 DEBUG:boto:Path: /vclfcn-19/obj?versionId=tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi 2024-02-01T10:27:45.157 DEBUG:boto:Data: 2024-02-01T10:27:45.158 DEBUG:boto:Headers: {} 2024-02-01T10:27:45.158 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:45.158 DEBUG:boto:Port: 8001 2024-02-01T10:27:45.158 DEBUG:boto:Params: {} 2024-02-01T10:27:45.158 DEBUG:boto:Token: None 2024-02-01T10:27:45.158 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:45 GMT /vclfcn-19/obj?versionId=tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi 2024-02-01T10:27:45.158 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jbmujdCiKcrJhgG1InizZMoUBWs= 2024-02-01T10:27:45.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:45 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jbmujdCiKcrJhgG1InizZMoUBWs=', 'Content-Length': '0'} 2024-02-01T10:27:45.161 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001bf823df1015d9cc-0065bb7221-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:45 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:45.161 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:45.161 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:45.161 DEBUG:boto:path=/vclfcn-19/obj?versionId=tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi 2024-02-01T10:27:45.161 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi 2024-02-01T10:27:45.161 DEBUG:boto:Method: HEAD 2024-02-01T10:27:45.162 DEBUG:boto:Path: /vclfcn-19/obj?versionId=tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi 2024-02-01T10:27:45.162 DEBUG:boto:Data: 2024-02-01T10:27:45.162 DEBUG:boto:Headers: {} 2024-02-01T10:27:45.162 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:45.162 DEBUG:boto:Port: 8000 2024-02-01T10:27:45.162 DEBUG:boto:Params: {} 2024-02-01T10:27:45.162 DEBUG:boto:Token: None 2024-02-01T10:27:45.162 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:45 GMT /vclfcn-19/obj?versionId=tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi 2024-02-01T10:27:45.162 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jbmujdCiKcrJhgG1InizZMoUBWs= 2024-02-01T10:27:45.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:45 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jbmujdCiKcrJhgG1InizZMoUBWs=', 'Content-Length': '0'} 2024-02-01T10:27:45.165 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005364967094106a27-0065bb7221-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:45 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:45.165 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:45.165 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:45.165 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:45.165 DEBUG:boto:path=/vclfcn-19/obj?versionId=tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi 2024-02-01T10:27:45.166 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi 2024-02-01T10:27:45.166 DEBUG:boto:Method: GET 2024-02-01T10:27:45.166 DEBUG:boto:Path: /vclfcn-19/obj?versionId=tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi 2024-02-01T10:27:45.166 DEBUG:boto:Data: 2024-02-01T10:27:45.166 DEBUG:boto:Headers: {} 2024-02-01T10:27:45.166 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:45.166 DEBUG:boto:Port: 8001 2024-02-01T10:27:45.166 DEBUG:boto:Params: {} 2024-02-01T10:27:45.166 DEBUG:boto:Token: None 2024-02-01T10:27:45.166 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:45 GMT /vclfcn-19/obj?versionId=tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi 2024-02-01T10:27:45.167 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cMo/+GQaTVWjSOugoDfMklpK9IQ= 2024-02-01T10:27:45.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:45 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cMo/+GQaTVWjSOugoDfMklpK9IQ=', 'Content-Length': '0'} 2024-02-01T10:27:45.169 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000063cf3cffe6200ef9-0065bb7221-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:45 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:45.212 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:45.212 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:45.212 DEBUG:boto:path=/vclfcn-19/obj?versionId=tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi 2024-02-01T10:27:45.212 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi 2024-02-01T10:27:45.212 DEBUG:boto:Method: GET 2024-02-01T10:27:45.213 DEBUG:boto:Path: /vclfcn-19/obj?versionId=tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi 2024-02-01T10:27:45.213 DEBUG:boto:Data: 2024-02-01T10:27:45.213 DEBUG:boto:Headers: {} 2024-02-01T10:27:45.213 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:45.213 DEBUG:boto:Port: 8000 2024-02-01T10:27:45.213 DEBUG:boto:Params: {} 2024-02-01T10:27:45.213 DEBUG:boto:Token: None 2024-02-01T10:27:45.213 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:45 GMT /vclfcn-19/obj?versionId=tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi 2024-02-01T10:27:45.214 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cMo/+GQaTVWjSOugoDfMklpK9IQ= 2024-02-01T10:27:45.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:45 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cMo/+GQaTVWjSOugoDfMklpK9IQ=', 'Content-Length': '0'} 2024-02-01T10:27:45.217 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c915c018c1996f54-0065bb7221-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:45 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:45.217 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:45.218 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:45.218 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:45.218 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:45.218 DEBUG:boto:Method: GET 2024-02-01T10:27:45.218 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:45.218 DEBUG:boto:Data: 2024-02-01T10:27:45.218 DEBUG:boto:Headers: {} 2024-02-01T10:27:45.218 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:45.218 DEBUG:boto:Port: 8001 2024-02-01T10:27:45.219 DEBUG:boto:Params: {} 2024-02-01T10:27:45.219 DEBUG:boto:Token: None 2024-02-01T10:27:45.219 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:45 GMT /vclfcn-19/ 2024-02-01T10:27:45.219 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QWZHvqI7UqnWbywLSlKwHcznVSI= 2024-02-01T10:27:45.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:45 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QWZHvqI7UqnWbywLSlKwHcznVSI=', 'Content-Length': '0'} 2024-02-01T10:27:45.224 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e1fb2786d7ea779-0065bb7221-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:45 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:45.224 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:45.225 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:45.225 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:45.225 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:45.225 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:45.225 DEBUG:boto:Method: GET 2024-02-01T10:27:45.225 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:45.225 DEBUG:boto:Data: 2024-02-01T10:27:45.226 DEBUG:boto:Headers: {} 2024-02-01T10:27:45.226 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:45.226 DEBUG:boto:Port: 8000 2024-02-01T10:27:45.226 DEBUG:boto:Params: {} 2024-02-01T10:27:45.226 DEBUG:boto:Token: None 2024-02-01T10:27:45.226 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:45 GMT /vclfcn-19/ 2024-02-01T10:27:45.226 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QWZHvqI7UqnWbywLSlKwHcznVSI= 2024-02-01T10:27:45.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:45 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QWZHvqI7UqnWbywLSlKwHcznVSI=', 'Content-Length': '0'} 2024-02-01T10:27:45.232 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011bcacaf2ce7cb6d-0065bb7221-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:45 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:45.232 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:45.233 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:45.233 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:45.233 DEBUG:boto:Method: HEAD 2024-02-01T10:27:45.233 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:45.233 DEBUG:boto:Data: 2024-02-01T10:27:45.233 DEBUG:boto:Headers: {} 2024-02-01T10:27:45.233 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:45.233 DEBUG:boto:Port: 8001 2024-02-01T10:27:45.234 DEBUG:boto:Params: {} 2024-02-01T10:27:45.234 DEBUG:boto:Token: None 2024-02-01T10:27:45.234 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:45 GMT /vclfcn-19/obj 2024-02-01T10:27:45.234 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YRAy/+3356wpirVSvtp8e3a7Xgk= 2024-02-01T10:27:45.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:45 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YRAy/+3356wpirVSvtp8e3a7Xgk=', 'Content-Length': '0'} 2024-02-01T10:27:45.240 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006d422088ebc53aea-0065bb7221-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:45 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:45.240 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:45.240 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:45.240 DEBUG:boto:Method: HEAD 2024-02-01T10:27:45.241 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:45.241 DEBUG:boto:Data: 2024-02-01T10:27:45.241 DEBUG:boto:Headers: {} 2024-02-01T10:27:45.241 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:45.241 DEBUG:boto:Port: 8000 2024-02-01T10:27:45.241 DEBUG:boto:Params: {} 2024-02-01T10:27:45.241 DEBUG:boto:Token: None 2024-02-01T10:27:45.241 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:45 GMT /vclfcn-19/obj 2024-02-01T10:27:45.241 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YRAy/+3356wpirVSvtp8e3a7Xgk= 2024-02-01T10:27:45.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:45 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YRAy/+3356wpirVSvtp8e3a7Xgk=', 'Content-Length': '0'} 2024-02-01T10:27:45.247 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000073b329169fd222ec-0065bb7221-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:45 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:45.247 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:45.247 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:45.247 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:45.248 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:45.248 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:45.248 DEBUG:boto:Method: GET 2024-02-01T10:27:45.248 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:45.248 DEBUG:boto:Data: 2024-02-01T10:27:45.248 DEBUG:boto:Headers: {} 2024-02-01T10:27:45.248 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:45.248 DEBUG:boto:Port: 8001 2024-02-01T10:27:45.248 DEBUG:boto:Params: {} 2024-02-01T10:27:45.248 DEBUG:boto:Token: None 2024-02-01T10:27:45.249 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:45 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:45.249 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:XRhX7aJ1NkHdOLL4sAlngINyt/s= 2024-02-01T10:27:45.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:45 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:XRhX7aJ1NkHdOLL4sAlngINyt/s=', 'Content-Length': '0'} 2024-02-01T10:27:45.252 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a319160ffb8d451c-0065bb7221-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:45 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:45.292 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:45.292 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:45.292 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:45.292 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:45.293 DEBUG:boto:Method: GET 2024-02-01T10:27:45.293 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:45.293 DEBUG:boto:Data: 2024-02-01T10:27:45.293 DEBUG:boto:Headers: {} 2024-02-01T10:27:45.293 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:45.293 DEBUG:boto:Port: 8000 2024-02-01T10:27:45.293 DEBUG:boto:Params: {} 2024-02-01T10:27:45.293 DEBUG:boto:Token: None 2024-02-01T10:27:45.294 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:45 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:45.294 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:XRhX7aJ1NkHdOLL4sAlngINyt/s= 2024-02-01T10:27:45.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:45 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:XRhX7aJ1NkHdOLL4sAlngINyt/s=', 'Content-Length': '0'} 2024-02-01T10:27:45.297 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f91b1fb76315e332-0065bb7221-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:45 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:45.344 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:45.345 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:45.345 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:45.345 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:45.345 DEBUG:boto:Method: GET 2024-02-01T10:27:45.345 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:45.345 DEBUG:boto:Data: 2024-02-01T10:27:45.345 DEBUG:boto:Headers: {} 2024-02-01T10:27:45.345 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:45.345 DEBUG:boto:Port: 8001 2024-02-01T10:27:45.345 DEBUG:boto:Params: {} 2024-02-01T10:27:45.346 DEBUG:boto:Token: None 2024-02-01T10:27:45.346 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:45 GMT /vclfcn-19/ 2024-02-01T10:27:45.346 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QWZHvqI7UqnWbywLSlKwHcznVSI= 2024-02-01T10:27:45.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:45 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QWZHvqI7UqnWbywLSlKwHcznVSI=', 'Content-Length': '0'} 2024-02-01T10:27:45.351 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009a7270bd34c775f-0065bb7221-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:45 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:45.352 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:45.352 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:45.352 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:45.352 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:45.352 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:45.353 DEBUG:boto:Method: GET 2024-02-01T10:27:45.353 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:45.353 DEBUG:boto:Data: 2024-02-01T10:27:45.353 DEBUG:boto:Headers: {} 2024-02-01T10:27:45.353 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:45.353 DEBUG:boto:Port: 8000 2024-02-01T10:27:45.353 DEBUG:boto:Params: {} 2024-02-01T10:27:45.353 DEBUG:boto:Token: None 2024-02-01T10:27:45.353 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:45 GMT /vclfcn-19/ 2024-02-01T10:27:45.353 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QWZHvqI7UqnWbywLSlKwHcznVSI= 2024-02-01T10:27:45.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:45 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QWZHvqI7UqnWbywLSlKwHcznVSI=', 'Content-Length': '0'} 2024-02-01T10:27:45.358 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000caea0765e972b61c-0065bb7221-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:45 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:45.359 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:45.359 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:45.360 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:45.360 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:45.360 DEBUG:boto:path=/vclfcn-19/obj?versionId=3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz 2024-02-01T10:27:45.360 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz 2024-02-01T10:27:45.360 DEBUG:boto:Method: GET 2024-02-01T10:27:45.361 DEBUG:boto:Path: /vclfcn-19/obj?versionId=3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz 2024-02-01T10:27:45.361 DEBUG:boto:Data: 2024-02-01T10:27:45.361 DEBUG:boto:Headers: {} 2024-02-01T10:27:45.361 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:45.361 DEBUG:boto:Port: 8001 2024-02-01T10:27:45.361 DEBUG:boto:Params: {} 2024-02-01T10:27:45.361 DEBUG:boto:Token: None 2024-02-01T10:27:45.361 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:45 GMT /vclfcn-19/obj?versionId=3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz 2024-02-01T10:27:45.362 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HEP8mG75L29NzjgjWJ6zpqYAjgs= 2024-02-01T10:27:45.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:45 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HEP8mG75L29NzjgjWJ6zpqYAjgs=', 'Content-Length': '0'} 2024-02-01T10:27:45.365 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005c0d218153f258f3-0065bb7221-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:45 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:45.365 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:45.365 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:45.366 DEBUG:boto:path=/vclfcn-19/obj?versionId=3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz 2024-02-01T10:27:45.366 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz 2024-02-01T10:27:45.366 DEBUG:boto:Method: GET 2024-02-01T10:27:45.366 DEBUG:boto:Path: /vclfcn-19/obj?versionId=3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz 2024-02-01T10:27:45.366 DEBUG:boto:Data: 2024-02-01T10:27:45.366 DEBUG:boto:Headers: {} 2024-02-01T10:27:45.366 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:45.366 DEBUG:boto:Port: 8000 2024-02-01T10:27:45.366 DEBUG:boto:Params: {} 2024-02-01T10:27:45.367 DEBUG:boto:Token: None 2024-02-01T10:27:45.367 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:45 GMT /vclfcn-19/obj?versionId=3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz 2024-02-01T10:27:45.367 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HEP8mG75L29NzjgjWJ6zpqYAjgs= 2024-02-01T10:27:45.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:45 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HEP8mG75L29NzjgjWJ6zpqYAjgs=', 'Content-Length': '0'} 2024-02-01T10:27:45.370 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000adf8285cc665bfdf-0065bb7221-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:45 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:45.412 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:45.412 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:45.413 DEBUG:boto:path=/vclfcn-19/obj?versionId=3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz 2024-02-01T10:27:45.413 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz 2024-02-01T10:27:45.413 DEBUG:boto:Method: HEAD 2024-02-01T10:27:45.413 DEBUG:boto:Path: /vclfcn-19/obj?versionId=3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz 2024-02-01T10:27:45.413 DEBUG:boto:Data: 2024-02-01T10:27:45.413 DEBUG:boto:Headers: {} 2024-02-01T10:27:45.413 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:45.413 DEBUG:boto:Port: 8001 2024-02-01T10:27:45.413 DEBUG:boto:Params: {} 2024-02-01T10:27:45.414 DEBUG:boto:Token: None 2024-02-01T10:27:45.414 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:45 GMT /vclfcn-19/obj?versionId=3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz 2024-02-01T10:27:45.414 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+QAy7w7skhflm57OYmTuTojuDUI= 2024-02-01T10:27:45.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:45 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+QAy7w7skhflm57OYmTuTojuDUI=', 'Content-Length': '0'} 2024-02-01T10:27:45.417 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000987c0aca0d62e457-0065bb7221-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:45 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:45.418 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:45.418 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:45.418 DEBUG:boto:path=/vclfcn-19/obj?versionId=3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz 2024-02-01T10:27:45.418 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz 2024-02-01T10:27:45.418 DEBUG:boto:Method: HEAD 2024-02-01T10:27:45.418 DEBUG:boto:Path: /vclfcn-19/obj?versionId=3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz 2024-02-01T10:27:45.418 DEBUG:boto:Data: 2024-02-01T10:27:45.418 DEBUG:boto:Headers: {} 2024-02-01T10:27:45.418 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:45.419 DEBUG:boto:Port: 8000 2024-02-01T10:27:45.419 DEBUG:boto:Params: {} 2024-02-01T10:27:45.419 DEBUG:boto:Token: None 2024-02-01T10:27:45.419 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:45 GMT /vclfcn-19/obj?versionId=3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz 2024-02-01T10:27:45.419 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+QAy7w7skhflm57OYmTuTojuDUI= 2024-02-01T10:27:45.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:45 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+QAy7w7skhflm57OYmTuTojuDUI=', 'Content-Length': '0'} 2024-02-01T10:27:45.422 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007b15a70e28c07c30-0065bb7221-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:45 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:45.422 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:45.422 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:45.422 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:45.423 DEBUG:boto:path=/vclfcn-19/obj?versionId=3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz 2024-02-01T10:27:45.423 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz 2024-02-01T10:27:45.423 DEBUG:boto:Method: GET 2024-02-01T10:27:45.423 DEBUG:boto:Path: /vclfcn-19/obj?versionId=3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz 2024-02-01T10:27:45.423 DEBUG:boto:Data: 2024-02-01T10:27:45.423 DEBUG:boto:Headers: {} 2024-02-01T10:27:45.423 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:45.423 DEBUG:boto:Port: 8001 2024-02-01T10:27:45.423 DEBUG:boto:Params: {} 2024-02-01T10:27:45.423 DEBUG:boto:Token: None 2024-02-01T10:27:45.424 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:45 GMT /vclfcn-19/obj?versionId=3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz 2024-02-01T10:27:45.424 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HEP8mG75L29NzjgjWJ6zpqYAjgs= 2024-02-01T10:27:45.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:45 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HEP8mG75L29NzjgjWJ6zpqYAjgs=', 'Content-Length': '0'} 2024-02-01T10:27:45.426 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000635da521a4938e59-0065bb7221-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:45 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:45.472 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:45.472 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:45.472 DEBUG:boto:path=/vclfcn-19/obj?versionId=3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz 2024-02-01T10:27:45.473 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz 2024-02-01T10:27:45.473 DEBUG:boto:Method: GET 2024-02-01T10:27:45.473 DEBUG:boto:Path: /vclfcn-19/obj?versionId=3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz 2024-02-01T10:27:45.473 DEBUG:boto:Data: 2024-02-01T10:27:45.473 DEBUG:boto:Headers: {} 2024-02-01T10:27:45.473 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:45.473 DEBUG:boto:Port: 8000 2024-02-01T10:27:45.473 DEBUG:boto:Params: {} 2024-02-01T10:27:45.473 DEBUG:boto:Token: None 2024-02-01T10:27:45.474 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:45 GMT /vclfcn-19/obj?versionId=3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz 2024-02-01T10:27:45.474 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HEP8mG75L29NzjgjWJ6zpqYAjgs= 2024-02-01T10:27:45.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:45 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HEP8mG75L29NzjgjWJ6zpqYAjgs=', 'Content-Length': '0'} 2024-02-01T10:27:45.477 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006fc55ba0ccf19a6e-0065bb7221-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:45 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:45.520 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:45.520 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:45.521 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:45.521 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:45.521 DEBUG:boto:Method: GET 2024-02-01T10:27:45.521 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:45.521 DEBUG:boto:Data: 2024-02-01T10:27:45.521 DEBUG:boto:Headers: {} 2024-02-01T10:27:45.521 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:45.521 DEBUG:boto:Port: 8001 2024-02-01T10:27:45.521 DEBUG:boto:Params: {} 2024-02-01T10:27:45.521 DEBUG:boto:Token: None 2024-02-01T10:27:45.522 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:45 GMT /vclfcn-19/ 2024-02-01T10:27:45.522 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QWZHvqI7UqnWbywLSlKwHcznVSI= 2024-02-01T10:27:45.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:45 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QWZHvqI7UqnWbywLSlKwHcznVSI=', 'Content-Length': '0'} 2024-02-01T10:27:45.526 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000469d6696f5003bad-0065bb7221-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:45 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:45.572 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:45.573 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:45.573 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:45.573 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:45.573 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:45.574 DEBUG:boto:Method: GET 2024-02-01T10:27:45.574 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:45.574 DEBUG:boto:Data: 2024-02-01T10:27:45.574 DEBUG:boto:Headers: {} 2024-02-01T10:27:45.574 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:45.574 DEBUG:boto:Port: 8000 2024-02-01T10:27:45.574 DEBUG:boto:Params: {} 2024-02-01T10:27:45.574 DEBUG:boto:Token: None 2024-02-01T10:27:45.574 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:45 GMT /vclfcn-19/ 2024-02-01T10:27:45.575 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QWZHvqI7UqnWbywLSlKwHcznVSI= 2024-02-01T10:27:45.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:45 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QWZHvqI7UqnWbywLSlKwHcznVSI=', 'Content-Length': '0'} 2024-02-01T10:27:45.581 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000053a89bcf767cea5-0065bb7221-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:45 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:45.581 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:45.581 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:45.582 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:45.582 DEBUG:boto:Method: HEAD 2024-02-01T10:27:45.582 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:45.582 DEBUG:boto:Data: 2024-02-01T10:27:45.582 DEBUG:boto:Headers: {} 2024-02-01T10:27:45.582 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:45.582 DEBUG:boto:Port: 8001 2024-02-01T10:27:45.582 DEBUG:boto:Params: {} 2024-02-01T10:27:45.582 DEBUG:boto:Token: None 2024-02-01T10:27:45.582 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:45 GMT /vclfcn-19/obj 2024-02-01T10:27:45.583 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YRAy/+3356wpirVSvtp8e3a7Xgk= 2024-02-01T10:27:45.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:45 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YRAy/+3356wpirVSvtp8e3a7Xgk=', 'Content-Length': '0'} 2024-02-01T10:27:45.589 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003ac1b8111d30e920-0065bb7221-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:45 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:45.589 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:45.589 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:45.589 DEBUG:boto:Method: HEAD 2024-02-01T10:27:45.589 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:45.589 DEBUG:boto:Data: 2024-02-01T10:27:45.589 DEBUG:boto:Headers: {} 2024-02-01T10:27:45.589 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:45.590 DEBUG:boto:Port: 8000 2024-02-01T10:27:45.590 DEBUG:boto:Params: {} 2024-02-01T10:27:45.590 DEBUG:boto:Token: None 2024-02-01T10:27:45.590 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:45 GMT /vclfcn-19/obj 2024-02-01T10:27:45.590 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YRAy/+3356wpirVSvtp8e3a7Xgk= 2024-02-01T10:27:45.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:45 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YRAy/+3356wpirVSvtp8e3a7Xgk=', 'Content-Length': '0'} 2024-02-01T10:27:45.596 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cedf85f82e2e065f-0065bb7221-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:45 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:45.597 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:45.597 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:45.597 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:45.597 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:45.597 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:45.597 DEBUG:boto:Method: GET 2024-02-01T10:27:45.597 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:45.597 DEBUG:boto:Data: 2024-02-01T10:27:45.597 DEBUG:boto:Headers: {} 2024-02-01T10:27:45.597 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:45.598 DEBUG:boto:Port: 8001 2024-02-01T10:27:45.598 DEBUG:boto:Params: {} 2024-02-01T10:27:45.598 DEBUG:boto:Token: None 2024-02-01T10:27:45.598 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:45 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:45.598 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:XRhX7aJ1NkHdOLL4sAlngINyt/s= 2024-02-01T10:27:45.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:45 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:XRhX7aJ1NkHdOLL4sAlngINyt/s=', 'Content-Length': '0'} 2024-02-01T10:27:45.601 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000062f55ce1a975eb6e-0065bb7221-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:45 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:45.648 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:45.648 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:45.648 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:45.649 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:45.649 DEBUG:boto:Method: GET 2024-02-01T10:27:45.649 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:45.649 DEBUG:boto:Data: 2024-02-01T10:27:45.649 DEBUG:boto:Headers: {} 2024-02-01T10:27:45.649 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:45.649 DEBUG:boto:Port: 8000 2024-02-01T10:27:45.649 DEBUG:boto:Params: {} 2024-02-01T10:27:45.649 DEBUG:boto:Token: None 2024-02-01T10:27:45.650 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:45 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:45.650 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:XRhX7aJ1NkHdOLL4sAlngINyt/s= 2024-02-01T10:27:45.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:45 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:XRhX7aJ1NkHdOLL4sAlngINyt/s=', 'Content-Length': '0'} 2024-02-01T10:27:45.653 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004f71506ff2ebe875-0065bb7221-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:45 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:45.696 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:45.696 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:45.696 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:45.697 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:45.697 DEBUG:boto:Method: GET 2024-02-01T10:27:45.697 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:45.697 DEBUG:boto:Data: 2024-02-01T10:27:45.697 DEBUG:boto:Headers: {} 2024-02-01T10:27:45.697 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:45.697 DEBUG:boto:Port: 8001 2024-02-01T10:27:45.697 DEBUG:boto:Params: {} 2024-02-01T10:27:45.697 DEBUG:boto:Token: None 2024-02-01T10:27:45.697 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:45 GMT /vclfcn-19/ 2024-02-01T10:27:45.698 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QWZHvqI7UqnWbywLSlKwHcznVSI= 2024-02-01T10:27:45.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:45 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QWZHvqI7UqnWbywLSlKwHcznVSI=', 'Content-Length': '0'} 2024-02-01T10:27:45.704 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a96bddce2bb14284-0065bb7221-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:45 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:45.748 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:45.749 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:45.749 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:45.749 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:45.749 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:45.749 DEBUG:boto:Method: GET 2024-02-01T10:27:45.749 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:45.749 DEBUG:boto:Data: 2024-02-01T10:27:45.749 DEBUG:boto:Headers: {} 2024-02-01T10:27:45.749 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:45.749 DEBUG:boto:Port: 8000 2024-02-01T10:27:45.750 DEBUG:boto:Params: {} 2024-02-01T10:27:45.750 DEBUG:boto:Token: None 2024-02-01T10:27:45.750 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:45 GMT /vclfcn-19/ 2024-02-01T10:27:45.750 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QWZHvqI7UqnWbywLSlKwHcznVSI= 2024-02-01T10:27:45.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:45 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QWZHvqI7UqnWbywLSlKwHcznVSI=', 'Content-Length': '0'} 2024-02-01T10:27:45.756 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d7db3fe19730c08-0065bb7221-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:45 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:45.756 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:45.757 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:45.757 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:45.757 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:45.758 DEBUG:boto:path=/vclfcn-19/obj?versionId=K23uwsLWGmQDGYGsdu-h04Fk2wApB0O 2024-02-01T10:27:45.758 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=K23uwsLWGmQDGYGsdu-h04Fk2wApB0O 2024-02-01T10:27:45.758 DEBUG:boto:Method: GET 2024-02-01T10:27:45.758 DEBUG:boto:Path: /vclfcn-19/obj?versionId=K23uwsLWGmQDGYGsdu-h04Fk2wApB0O 2024-02-01T10:27:45.758 DEBUG:boto:Data: 2024-02-01T10:27:45.758 DEBUG:boto:Headers: {} 2024-02-01T10:27:45.758 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:45.758 DEBUG:boto:Port: 8001 2024-02-01T10:27:45.758 DEBUG:boto:Params: {} 2024-02-01T10:27:45.758 DEBUG:boto:Token: None 2024-02-01T10:27:45.759 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:45 GMT /vclfcn-19/obj?versionId=K23uwsLWGmQDGYGsdu-h04Fk2wApB0O 2024-02-01T10:27:45.759 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:oPxyRwY7V7bEaTWSJ/hDjIx8Kzs= 2024-02-01T10:27:45.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:45 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:oPxyRwY7V7bEaTWSJ/hDjIx8Kzs=', 'Content-Length': '0'} 2024-02-01T10:27:45.762 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'K23uwsLWGmQDGYGsdu-h04Fk2wApB0O'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b89688476342c5b2-0065bb7221-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:45 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:45.762 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:45.762 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:45.762 DEBUG:boto:path=/vclfcn-19/obj?versionId=K23uwsLWGmQDGYGsdu-h04Fk2wApB0O 2024-02-01T10:27:45.762 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=K23uwsLWGmQDGYGsdu-h04Fk2wApB0O 2024-02-01T10:27:45.763 DEBUG:boto:Method: GET 2024-02-01T10:27:45.763 DEBUG:boto:Path: /vclfcn-19/obj?versionId=K23uwsLWGmQDGYGsdu-h04Fk2wApB0O 2024-02-01T10:27:45.763 DEBUG:boto:Data: 2024-02-01T10:27:45.763 DEBUG:boto:Headers: {} 2024-02-01T10:27:45.763 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:45.763 DEBUG:boto:Port: 8000 2024-02-01T10:27:45.763 DEBUG:boto:Params: {} 2024-02-01T10:27:45.763 DEBUG:boto:Token: None 2024-02-01T10:27:45.763 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:45 GMT /vclfcn-19/obj?versionId=K23uwsLWGmQDGYGsdu-h04Fk2wApB0O 2024-02-01T10:27:45.764 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:oPxyRwY7V7bEaTWSJ/hDjIx8Kzs= 2024-02-01T10:27:45.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:45 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:oPxyRwY7V7bEaTWSJ/hDjIx8Kzs=', 'Content-Length': '0'} 2024-02-01T10:27:45.767 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'K23uwsLWGmQDGYGsdu-h04Fk2wApB0O'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000f4c5cb6ff84c33d-0065bb7221-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:45 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:45.767 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:45.767 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:45.767 DEBUG:boto:path=/vclfcn-19/obj?versionId=K23uwsLWGmQDGYGsdu-h04Fk2wApB0O 2024-02-01T10:27:45.767 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=K23uwsLWGmQDGYGsdu-h04Fk2wApB0O 2024-02-01T10:27:45.768 DEBUG:boto:Method: HEAD 2024-02-01T10:27:45.768 DEBUG:boto:Path: /vclfcn-19/obj?versionId=K23uwsLWGmQDGYGsdu-h04Fk2wApB0O 2024-02-01T10:27:45.768 DEBUG:boto:Data: 2024-02-01T10:27:45.768 DEBUG:boto:Headers: {} 2024-02-01T10:27:45.768 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:45.768 DEBUG:boto:Port: 8001 2024-02-01T10:27:45.768 DEBUG:boto:Params: {} 2024-02-01T10:27:45.768 DEBUG:boto:Token: None 2024-02-01T10:27:45.768 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:45 GMT /vclfcn-19/obj?versionId=K23uwsLWGmQDGYGsdu-h04Fk2wApB0O 2024-02-01T10:27:45.769 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:KD8trZb7buc13b1k8w5opxZYCU4= 2024-02-01T10:27:45.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:45 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:KD8trZb7buc13b1k8w5opxZYCU4=', 'Content-Length': '0'} 2024-02-01T10:27:45.771 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'K23uwsLWGmQDGYGsdu-h04Fk2wApB0O'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000073596ca59a73d6e6-0065bb7221-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:45 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:45.772 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:45.772 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:45.772 DEBUG:boto:path=/vclfcn-19/obj?versionId=K23uwsLWGmQDGYGsdu-h04Fk2wApB0O 2024-02-01T10:27:45.772 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=K23uwsLWGmQDGYGsdu-h04Fk2wApB0O 2024-02-01T10:27:45.772 DEBUG:boto:Method: HEAD 2024-02-01T10:27:45.772 DEBUG:boto:Path: /vclfcn-19/obj?versionId=K23uwsLWGmQDGYGsdu-h04Fk2wApB0O 2024-02-01T10:27:45.772 DEBUG:boto:Data: 2024-02-01T10:27:45.772 DEBUG:boto:Headers: {} 2024-02-01T10:27:45.772 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:45.772 DEBUG:boto:Port: 8000 2024-02-01T10:27:45.772 DEBUG:boto:Params: {} 2024-02-01T10:27:45.773 DEBUG:boto:Token: None 2024-02-01T10:27:45.773 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:45 GMT /vclfcn-19/obj?versionId=K23uwsLWGmQDGYGsdu-h04Fk2wApB0O 2024-02-01T10:27:45.773 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:KD8trZb7buc13b1k8w5opxZYCU4= 2024-02-01T10:27:45.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:45 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:KD8trZb7buc13b1k8w5opxZYCU4=', 'Content-Length': '0'} 2024-02-01T10:27:45.776 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'K23uwsLWGmQDGYGsdu-h04Fk2wApB0O'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000070b260715aebb126-0065bb7221-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:45 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:45.776 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:45.776 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:45.776 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:45.776 DEBUG:boto:path=/vclfcn-19/obj?versionId=K23uwsLWGmQDGYGsdu-h04Fk2wApB0O 2024-02-01T10:27:45.776 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=K23uwsLWGmQDGYGsdu-h04Fk2wApB0O 2024-02-01T10:27:45.776 DEBUG:boto:Method: GET 2024-02-01T10:27:45.777 DEBUG:boto:Path: /vclfcn-19/obj?versionId=K23uwsLWGmQDGYGsdu-h04Fk2wApB0O 2024-02-01T10:27:45.777 DEBUG:boto:Data: 2024-02-01T10:27:45.777 DEBUG:boto:Headers: {} 2024-02-01T10:27:45.777 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:45.777 DEBUG:boto:Port: 8001 2024-02-01T10:27:45.777 DEBUG:boto:Params: {} 2024-02-01T10:27:45.777 DEBUG:boto:Token: None 2024-02-01T10:27:45.777 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:45 GMT /vclfcn-19/obj?versionId=K23uwsLWGmQDGYGsdu-h04Fk2wApB0O 2024-02-01T10:27:45.777 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:oPxyRwY7V7bEaTWSJ/hDjIx8Kzs= 2024-02-01T10:27:45.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:45 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:oPxyRwY7V7bEaTWSJ/hDjIx8Kzs=', 'Content-Length': '0'} 2024-02-01T10:27:45.780 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'K23uwsLWGmQDGYGsdu-h04Fk2wApB0O'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000085c94cf9ae964164-0065bb7221-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:45 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:45.824 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:45.824 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:45.824 DEBUG:boto:path=/vclfcn-19/obj?versionId=K23uwsLWGmQDGYGsdu-h04Fk2wApB0O 2024-02-01T10:27:45.824 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=K23uwsLWGmQDGYGsdu-h04Fk2wApB0O 2024-02-01T10:27:45.825 DEBUG:boto:Method: GET 2024-02-01T10:27:45.825 DEBUG:boto:Path: /vclfcn-19/obj?versionId=K23uwsLWGmQDGYGsdu-h04Fk2wApB0O 2024-02-01T10:27:45.825 DEBUG:boto:Data: 2024-02-01T10:27:45.825 DEBUG:boto:Headers: {} 2024-02-01T10:27:45.825 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:45.825 DEBUG:boto:Port: 8000 2024-02-01T10:27:45.825 DEBUG:boto:Params: {} 2024-02-01T10:27:45.825 DEBUG:boto:Token: None 2024-02-01T10:27:45.826 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:45 GMT /vclfcn-19/obj?versionId=K23uwsLWGmQDGYGsdu-h04Fk2wApB0O 2024-02-01T10:27:45.826 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:oPxyRwY7V7bEaTWSJ/hDjIx8Kzs= 2024-02-01T10:27:45.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:45 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:oPxyRwY7V7bEaTWSJ/hDjIx8Kzs=', 'Content-Length': '0'} 2024-02-01T10:27:45.829 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'K23uwsLWGmQDGYGsdu-h04Fk2wApB0O'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ccc84740ae5dfcaf-0065bb7221-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:45 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:45.829 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:45.830 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:45.830 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:45.830 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:45.830 DEBUG:boto:Method: GET 2024-02-01T10:27:45.830 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:45.830 DEBUG:boto:Data: 2024-02-01T10:27:45.830 DEBUG:boto:Headers: {} 2024-02-01T10:27:45.830 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:45.830 DEBUG:boto:Port: 8001 2024-02-01T10:27:45.830 DEBUG:boto:Params: {} 2024-02-01T10:27:45.830 DEBUG:boto:Token: None 2024-02-01T10:27:45.831 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:45 GMT /vclfcn-19/ 2024-02-01T10:27:45.831 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QWZHvqI7UqnWbywLSlKwHcznVSI= 2024-02-01T10:27:45.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:45 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QWZHvqI7UqnWbywLSlKwHcznVSI=', 'Content-Length': '0'} 2024-02-01T10:27:45.835 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001bda0f87a2fb7388-0065bb7221-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:45 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:45.876 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:45.877 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:45.877 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:45.878 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:45.878 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:45.878 DEBUG:boto:Method: GET 2024-02-01T10:27:45.878 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:45.878 DEBUG:boto:Data: 2024-02-01T10:27:45.878 DEBUG:boto:Headers: {} 2024-02-01T10:27:45.878 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:45.879 DEBUG:boto:Port: 8000 2024-02-01T10:27:45.879 DEBUG:boto:Params: {} 2024-02-01T10:27:45.879 DEBUG:boto:Token: None 2024-02-01T10:27:45.879 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:45 GMT /vclfcn-19/ 2024-02-01T10:27:45.879 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QWZHvqI7UqnWbywLSlKwHcznVSI= 2024-02-01T10:27:45.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:45 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QWZHvqI7UqnWbywLSlKwHcznVSI=', 'Content-Length': '0'} 2024-02-01T10:27:45.885 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000223e7b765f3db887-0065bb7221-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:45 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:45.885 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:45.886 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:45.886 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:45.886 DEBUG:boto:Method: HEAD 2024-02-01T10:27:45.886 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:45.886 DEBUG:boto:Data: 2024-02-01T10:27:45.886 DEBUG:boto:Headers: {} 2024-02-01T10:27:45.886 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:45.887 DEBUG:boto:Port: 8001 2024-02-01T10:27:45.887 DEBUG:boto:Params: {} 2024-02-01T10:27:45.887 DEBUG:boto:Token: None 2024-02-01T10:27:45.888 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:45 GMT /vclfcn-19/obj 2024-02-01T10:27:45.888 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YRAy/+3356wpirVSvtp8e3a7Xgk= 2024-02-01T10:27:45.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:45 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YRAy/+3356wpirVSvtp8e3a7Xgk=', 'Content-Length': '0'} 2024-02-01T10:27:45.894 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000401335c0da105376-0065bb7221-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:45 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:45.894 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:45.894 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:45.895 DEBUG:boto:Method: HEAD 2024-02-01T10:27:45.895 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:45.895 DEBUG:boto:Data: 2024-02-01T10:27:45.895 DEBUG:boto:Headers: {} 2024-02-01T10:27:45.895 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:45.895 DEBUG:boto:Port: 8000 2024-02-01T10:27:45.895 DEBUG:boto:Params: {} 2024-02-01T10:27:45.895 DEBUG:boto:Token: None 2024-02-01T10:27:45.895 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:45 GMT /vclfcn-19/obj 2024-02-01T10:27:45.895 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YRAy/+3356wpirVSvtp8e3a7Xgk= 2024-02-01T10:27:45.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:45 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YRAy/+3356wpirVSvtp8e3a7Xgk=', 'Content-Length': '0'} 2024-02-01T10:27:45.901 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bb718445c582dbb1-0065bb7221-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:45 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:45.901 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:45.901 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:45.901 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:45.902 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:45.902 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:45.902 DEBUG:boto:Method: GET 2024-02-01T10:27:45.902 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:45.902 DEBUG:boto:Data: 2024-02-01T10:27:45.902 DEBUG:boto:Headers: {} 2024-02-01T10:27:45.902 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:45.902 DEBUG:boto:Port: 8001 2024-02-01T10:27:45.902 DEBUG:boto:Params: {} 2024-02-01T10:27:45.902 DEBUG:boto:Token: None 2024-02-01T10:27:45.903 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:45 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:45.903 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:XRhX7aJ1NkHdOLL4sAlngINyt/s= 2024-02-01T10:27:45.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:45 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:XRhX7aJ1NkHdOLL4sAlngINyt/s=', 'Content-Length': '0'} 2024-02-01T10:27:45.906 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000098a2753b40f5bdce-0065bb7221-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:45 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:45.948 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:45.948 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:45.948 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:45.949 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:45.949 DEBUG:boto:Method: GET 2024-02-01T10:27:45.949 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:45.949 DEBUG:boto:Data: 2024-02-01T10:27:45.949 DEBUG:boto:Headers: {} 2024-02-01T10:27:45.949 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:45.949 DEBUG:boto:Port: 8000 2024-02-01T10:27:45.949 DEBUG:boto:Params: {} 2024-02-01T10:27:45.949 DEBUG:boto:Token: None 2024-02-01T10:27:45.949 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:45 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:45.949 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:XRhX7aJ1NkHdOLL4sAlngINyt/s= 2024-02-01T10:27:45.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:45 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:XRhX7aJ1NkHdOLL4sAlngINyt/s=', 'Content-Length': '0'} 2024-02-01T10:27:45.953 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fd64c01dcd72b8b9-0065bb7221-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:45 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:45.996 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:45.996 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:45.996 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:45.997 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:45.997 DEBUG:boto:Method: GET 2024-02-01T10:27:45.997 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:45.997 DEBUG:boto:Data: 2024-02-01T10:27:45.997 DEBUG:boto:Headers: {} 2024-02-01T10:27:45.997 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:45.997 DEBUG:boto:Port: 8001 2024-02-01T10:27:45.997 DEBUG:boto:Params: {} 2024-02-01T10:27:45.998 DEBUG:boto:Token: None 2024-02-01T10:27:45.998 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:45 GMT /vclfcn-19/ 2024-02-01T10:27:45.998 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QWZHvqI7UqnWbywLSlKwHcznVSI= 2024-02-01T10:27:45.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:45 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QWZHvqI7UqnWbywLSlKwHcznVSI=', 'Content-Length': '0'} 2024-02-01T10:27:46.004 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0a03fd02efd3832-0065bb7222-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:46.052 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:46.053 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:46.053 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:46.053 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:46.053 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:46.054 DEBUG:boto:Method: GET 2024-02-01T10:27:46.054 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:46.054 DEBUG:boto:Data: 2024-02-01T10:27:46.054 DEBUG:boto:Headers: {} 2024-02-01T10:27:46.054 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:46.054 DEBUG:boto:Port: 8000 2024-02-01T10:27:46.054 DEBUG:boto:Params: {} 2024-02-01T10:27:46.055 DEBUG:boto:Token: None 2024-02-01T10:27:46.055 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:46 GMT /vclfcn-19/ 2024-02-01T10:27:46.055 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FQtT/TF8IcJdqG6hRlZ1dT71FEE= 2024-02-01T10:27:46.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FQtT/TF8IcJdqG6hRlZ1dT71FEE=', 'Content-Length': '0'} 2024-02-01T10:27:46.060 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c7d731cae2b2a327-0065bb7222-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:46.060 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:46.061 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:46.061 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:46.061 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:46.061 DEBUG:boto:path=/vclfcn-19/obj?versionId=st2TOT.K7IOe-wFfL.hOrRVfLNWblRd 2024-02-01T10:27:46.061 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=st2TOT.K7IOe-wFfL.hOrRVfLNWblRd 2024-02-01T10:27:46.061 DEBUG:boto:Method: GET 2024-02-01T10:27:46.062 DEBUG:boto:Path: /vclfcn-19/obj?versionId=st2TOT.K7IOe-wFfL.hOrRVfLNWblRd 2024-02-01T10:27:46.062 DEBUG:boto:Data: 2024-02-01T10:27:46.062 DEBUG:boto:Headers: {} 2024-02-01T10:27:46.062 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:46.062 DEBUG:boto:Port: 8001 2024-02-01T10:27:46.062 DEBUG:boto:Params: {} 2024-02-01T10:27:46.063 DEBUG:boto:Token: None 2024-02-01T10:27:46.063 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:46 GMT /vclfcn-19/obj?versionId=st2TOT.K7IOe-wFfL.hOrRVfLNWblRd 2024-02-01T10:27:46.063 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:loZUffg8NO14lHx+2S9RtGVhyLc= 2024-02-01T10:27:46.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:loZUffg8NO14lHx+2S9RtGVhyLc=', 'Content-Length': '0'} 2024-02-01T10:27:46.066 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'st2TOT.K7IOe-wFfL.hOrRVfLNWblRd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000732e61350727f6b5-0065bb7222-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:46.066 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:46.066 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:46.067 DEBUG:boto:path=/vclfcn-19/obj?versionId=st2TOT.K7IOe-wFfL.hOrRVfLNWblRd 2024-02-01T10:27:46.067 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=st2TOT.K7IOe-wFfL.hOrRVfLNWblRd 2024-02-01T10:27:46.067 DEBUG:boto:Method: GET 2024-02-01T10:27:46.067 DEBUG:boto:Path: /vclfcn-19/obj?versionId=st2TOT.K7IOe-wFfL.hOrRVfLNWblRd 2024-02-01T10:27:46.067 DEBUG:boto:Data: 2024-02-01T10:27:46.067 DEBUG:boto:Headers: {} 2024-02-01T10:27:46.067 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:46.067 DEBUG:boto:Port: 8000 2024-02-01T10:27:46.067 DEBUG:boto:Params: {} 2024-02-01T10:27:46.067 DEBUG:boto:Token: None 2024-02-01T10:27:46.068 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:46 GMT /vclfcn-19/obj?versionId=st2TOT.K7IOe-wFfL.hOrRVfLNWblRd 2024-02-01T10:27:46.068 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:loZUffg8NO14lHx+2S9RtGVhyLc= 2024-02-01T10:27:46.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:loZUffg8NO14lHx+2S9RtGVhyLc=', 'Content-Length': '0'} 2024-02-01T10:27:46.071 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'st2TOT.K7IOe-wFfL.hOrRVfLNWblRd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e1cd5a8f5ab71415-0065bb7222-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:46.116 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:46.117 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:46.117 DEBUG:boto:path=/vclfcn-19/obj?versionId=st2TOT.K7IOe-wFfL.hOrRVfLNWblRd 2024-02-01T10:27:46.117 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=st2TOT.K7IOe-wFfL.hOrRVfLNWblRd 2024-02-01T10:27:46.117 DEBUG:boto:Method: HEAD 2024-02-01T10:27:46.117 DEBUG:boto:Path: /vclfcn-19/obj?versionId=st2TOT.K7IOe-wFfL.hOrRVfLNWblRd 2024-02-01T10:27:46.117 DEBUG:boto:Data: 2024-02-01T10:27:46.117 DEBUG:boto:Headers: {} 2024-02-01T10:27:46.117 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:46.117 DEBUG:boto:Port: 8001 2024-02-01T10:27:46.117 DEBUG:boto:Params: {} 2024-02-01T10:27:46.118 DEBUG:boto:Token: None 2024-02-01T10:27:46.118 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:46 GMT /vclfcn-19/obj?versionId=st2TOT.K7IOe-wFfL.hOrRVfLNWblRd 2024-02-01T10:27:46.118 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WQ5dMnuuYOB2JHQLbOyYUuStAzI= 2024-02-01T10:27:46.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WQ5dMnuuYOB2JHQLbOyYUuStAzI=', 'Content-Length': '0'} 2024-02-01T10:27:46.121 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'st2TOT.K7IOe-wFfL.hOrRVfLNWblRd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000051dab6b514b8c9e0-0065bb7222-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:46.121 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:46.122 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:46.122 DEBUG:boto:path=/vclfcn-19/obj?versionId=st2TOT.K7IOe-wFfL.hOrRVfLNWblRd 2024-02-01T10:27:46.122 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=st2TOT.K7IOe-wFfL.hOrRVfLNWblRd 2024-02-01T10:27:46.122 DEBUG:boto:Method: HEAD 2024-02-01T10:27:46.122 DEBUG:boto:Path: /vclfcn-19/obj?versionId=st2TOT.K7IOe-wFfL.hOrRVfLNWblRd 2024-02-01T10:27:46.122 DEBUG:boto:Data: 2024-02-01T10:27:46.122 DEBUG:boto:Headers: {} 2024-02-01T10:27:46.122 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:46.122 DEBUG:boto:Port: 8000 2024-02-01T10:27:46.122 DEBUG:boto:Params: {} 2024-02-01T10:27:46.123 DEBUG:boto:Token: None 2024-02-01T10:27:46.123 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:46 GMT /vclfcn-19/obj?versionId=st2TOT.K7IOe-wFfL.hOrRVfLNWblRd 2024-02-01T10:27:46.123 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WQ5dMnuuYOB2JHQLbOyYUuStAzI= 2024-02-01T10:27:46.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WQ5dMnuuYOB2JHQLbOyYUuStAzI=', 'Content-Length': '0'} 2024-02-01T10:27:46.126 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'st2TOT.K7IOe-wFfL.hOrRVfLNWblRd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b33ae935f460d0a4-0065bb7222-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:46.126 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:46.126 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:46.126 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:46.126 DEBUG:boto:path=/vclfcn-19/obj?versionId=st2TOT.K7IOe-wFfL.hOrRVfLNWblRd 2024-02-01T10:27:46.127 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=st2TOT.K7IOe-wFfL.hOrRVfLNWblRd 2024-02-01T10:27:46.127 DEBUG:boto:Method: GET 2024-02-01T10:27:46.127 DEBUG:boto:Path: /vclfcn-19/obj?versionId=st2TOT.K7IOe-wFfL.hOrRVfLNWblRd 2024-02-01T10:27:46.127 DEBUG:boto:Data: 2024-02-01T10:27:46.127 DEBUG:boto:Headers: {} 2024-02-01T10:27:46.127 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:46.127 DEBUG:boto:Port: 8001 2024-02-01T10:27:46.127 DEBUG:boto:Params: {} 2024-02-01T10:27:46.127 DEBUG:boto:Token: None 2024-02-01T10:27:46.128 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:46 GMT /vclfcn-19/obj?versionId=st2TOT.K7IOe-wFfL.hOrRVfLNWblRd 2024-02-01T10:27:46.128 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:loZUffg8NO14lHx+2S9RtGVhyLc= 2024-02-01T10:27:46.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:loZUffg8NO14lHx+2S9RtGVhyLc=', 'Content-Length': '0'} 2024-02-01T10:27:46.131 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'st2TOT.K7IOe-wFfL.hOrRVfLNWblRd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b11cacd4debdb76e-0065bb7222-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:46.131 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:46.131 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:46.131 DEBUG:boto:path=/vclfcn-19/obj?versionId=st2TOT.K7IOe-wFfL.hOrRVfLNWblRd 2024-02-01T10:27:46.131 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=st2TOT.K7IOe-wFfL.hOrRVfLNWblRd 2024-02-01T10:27:46.132 DEBUG:boto:Method: GET 2024-02-01T10:27:46.132 DEBUG:boto:Path: /vclfcn-19/obj?versionId=st2TOT.K7IOe-wFfL.hOrRVfLNWblRd 2024-02-01T10:27:46.132 DEBUG:boto:Data: 2024-02-01T10:27:46.132 DEBUG:boto:Headers: {} 2024-02-01T10:27:46.132 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:46.132 DEBUG:boto:Port: 8000 2024-02-01T10:27:46.132 DEBUG:boto:Params: {} 2024-02-01T10:27:46.132 DEBUG:boto:Token: None 2024-02-01T10:27:46.133 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:46 GMT /vclfcn-19/obj?versionId=st2TOT.K7IOe-wFfL.hOrRVfLNWblRd 2024-02-01T10:27:46.133 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:loZUffg8NO14lHx+2S9RtGVhyLc= 2024-02-01T10:27:46.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:loZUffg8NO14lHx+2S9RtGVhyLc=', 'Content-Length': '0'} 2024-02-01T10:27:46.136 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'st2TOT.K7IOe-wFfL.hOrRVfLNWblRd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009bb9a20d81982bad-0065bb7222-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:46.176 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:46.176 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:46.176 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:46.177 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:46.177 DEBUG:boto:Method: GET 2024-02-01T10:27:46.177 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:46.177 DEBUG:boto:Data: 2024-02-01T10:27:46.177 DEBUG:boto:Headers: {} 2024-02-01T10:27:46.177 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:46.177 DEBUG:boto:Port: 8001 2024-02-01T10:27:46.177 DEBUG:boto:Params: {} 2024-02-01T10:27:46.177 DEBUG:boto:Token: None 2024-02-01T10:27:46.177 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:46 GMT /vclfcn-19/ 2024-02-01T10:27:46.178 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FQtT/TF8IcJdqG6hRlZ1dT71FEE= 2024-02-01T10:27:46.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FQtT/TF8IcJdqG6hRlZ1dT71FEE=', 'Content-Length': '0'} 2024-02-01T10:27:46.184 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ec94d3488b3e88e-0065bb7222-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:46.184 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:46.184 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:46.184 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:46.185 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:46.185 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:46.185 DEBUG:boto:Method: GET 2024-02-01T10:27:46.185 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:46.185 DEBUG:boto:Data: 2024-02-01T10:27:46.185 DEBUG:boto:Headers: {} 2024-02-01T10:27:46.185 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:46.185 DEBUG:boto:Port: 8000 2024-02-01T10:27:46.185 DEBUG:boto:Params: {} 2024-02-01T10:27:46.185 DEBUG:boto:Token: None 2024-02-01T10:27:46.186 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:46 GMT /vclfcn-19/ 2024-02-01T10:27:46.186 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FQtT/TF8IcJdqG6hRlZ1dT71FEE= 2024-02-01T10:27:46.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FQtT/TF8IcJdqG6hRlZ1dT71FEE=', 'Content-Length': '0'} 2024-02-01T10:27:46.191 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e980b92e5e15b96-0065bb7222-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:46.191 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:46.191 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:46.191 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:46.192 DEBUG:boto:Method: HEAD 2024-02-01T10:27:46.192 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:46.192 DEBUG:boto:Data: 2024-02-01T10:27:46.192 DEBUG:boto:Headers: {} 2024-02-01T10:27:46.192 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:46.192 DEBUG:boto:Port: 8001 2024-02-01T10:27:46.192 DEBUG:boto:Params: {} 2024-02-01T10:27:46.192 DEBUG:boto:Token: None 2024-02-01T10:27:46.192 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:46 GMT /vclfcn-19/obj 2024-02-01T10:27:46.193 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:kisoj1CchsoK0IEwzE9RgWblk7I= 2024-02-01T10:27:46.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:kisoj1CchsoK0IEwzE9RgWblk7I=', 'Content-Length': '0'} 2024-02-01T10:27:46.198 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cb9f42ac5b2aecb7-0065bb7222-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:46.199 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:46.199 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:46.199 DEBUG:boto:Method: HEAD 2024-02-01T10:27:46.199 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:46.199 DEBUG:boto:Data: 2024-02-01T10:27:46.199 DEBUG:boto:Headers: {} 2024-02-01T10:27:46.199 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:46.199 DEBUG:boto:Port: 8000 2024-02-01T10:27:46.199 DEBUG:boto:Params: {} 2024-02-01T10:27:46.199 DEBUG:boto:Token: None 2024-02-01T10:27:46.200 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:46 GMT /vclfcn-19/obj 2024-02-01T10:27:46.200 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:kisoj1CchsoK0IEwzE9RgWblk7I= 2024-02-01T10:27:46.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:kisoj1CchsoK0IEwzE9RgWblk7I=', 'Content-Length': '0'} 2024-02-01T10:27:46.206 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002be7f40ce2efe317-0065bb7222-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:46.206 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:46.206 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:46.206 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:46.206 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:46.206 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:46.206 DEBUG:boto:Method: GET 2024-02-01T10:27:46.206 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:46.207 DEBUG:boto:Data: 2024-02-01T10:27:46.207 DEBUG:boto:Headers: {} 2024-02-01T10:27:46.207 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:46.207 DEBUG:boto:Port: 8001 2024-02-01T10:27:46.207 DEBUG:boto:Params: {} 2024-02-01T10:27:46.207 DEBUG:boto:Token: None 2024-02-01T10:27:46.207 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:46 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:46.207 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:O8uzubsS9dRIYP5bRpQM0mMTkEE= 2024-02-01T10:27:46.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:O8uzubsS9dRIYP5bRpQM0mMTkEE=', 'Content-Length': '0'} 2024-02-01T10:27:46.210 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e0b1dda5f7d97af3-0065bb7222-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:46.252 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:46.252 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:46.252 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:46.252 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:46.253 DEBUG:boto:Method: GET 2024-02-01T10:27:46.253 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:46.253 DEBUG:boto:Data: 2024-02-01T10:27:46.253 DEBUG:boto:Headers: {} 2024-02-01T10:27:46.253 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:46.253 DEBUG:boto:Port: 8000 2024-02-01T10:27:46.253 DEBUG:boto:Params: {} 2024-02-01T10:27:46.253 DEBUG:boto:Token: None 2024-02-01T10:27:46.253 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:46 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:46.254 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:O8uzubsS9dRIYP5bRpQM0mMTkEE= 2024-02-01T10:27:46.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:O8uzubsS9dRIYP5bRpQM0mMTkEE=', 'Content-Length': '0'} 2024-02-01T10:27:46.256 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e696ebed625e382a-0065bb7222-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:46.300 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:46.300 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:46.301 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:46.301 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:46.301 DEBUG:boto:Method: GET 2024-02-01T10:27:46.301 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:46.301 DEBUG:boto:Data: 2024-02-01T10:27:46.301 DEBUG:boto:Headers: {} 2024-02-01T10:27:46.301 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:46.301 DEBUG:boto:Port: 8001 2024-02-01T10:27:46.301 DEBUG:boto:Params: {} 2024-02-01T10:27:46.302 DEBUG:boto:Token: None 2024-02-01T10:27:46.302 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:46 GMT /vclfcn-19/ 2024-02-01T10:27:46.302 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FQtT/TF8IcJdqG6hRlZ1dT71FEE= 2024-02-01T10:27:46.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FQtT/TF8IcJdqG6hRlZ1dT71FEE=', 'Content-Length': '0'} 2024-02-01T10:27:46.307 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000062647322d7f59ca3-0065bb7222-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:46.307 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:46.308 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:46.308 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:46.308 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:46.308 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:46.308 DEBUG:boto:Method: GET 2024-02-01T10:27:46.308 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:46.308 DEBUG:boto:Data: 2024-02-01T10:27:46.309 DEBUG:boto:Headers: {} 2024-02-01T10:27:46.309 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:46.309 DEBUG:boto:Port: 8000 2024-02-01T10:27:46.309 DEBUG:boto:Params: {} 2024-02-01T10:27:46.309 DEBUG:boto:Token: None 2024-02-01T10:27:46.309 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:46 GMT /vclfcn-19/ 2024-02-01T10:27:46.309 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FQtT/TF8IcJdqG6hRlZ1dT71FEE= 2024-02-01T10:27:46.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FQtT/TF8IcJdqG6hRlZ1dT71FEE=', 'Content-Length': '0'} 2024-02-01T10:27:46.315 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000005caa27e5db9ba02-0065bb7222-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:46.315 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:46.315 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:46.315 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:46.316 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:46.316 DEBUG:boto:path=/vclfcn-19/obj?versionId=9CJ3w1EusK8OST2k17sTdTM.cuh3E3e 2024-02-01T10:27:46.316 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=9CJ3w1EusK8OST2k17sTdTM.cuh3E3e 2024-02-01T10:27:46.316 DEBUG:boto:Method: GET 2024-02-01T10:27:46.316 DEBUG:boto:Path: /vclfcn-19/obj?versionId=9CJ3w1EusK8OST2k17sTdTM.cuh3E3e 2024-02-01T10:27:46.316 DEBUG:boto:Data: 2024-02-01T10:27:46.316 DEBUG:boto:Headers: {} 2024-02-01T10:27:46.316 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:46.316 DEBUG:boto:Port: 8001 2024-02-01T10:27:46.317 DEBUG:boto:Params: {} 2024-02-01T10:27:46.317 DEBUG:boto:Token: None 2024-02-01T10:27:46.317 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:46 GMT /vclfcn-19/obj?versionId=9CJ3w1EusK8OST2k17sTdTM.cuh3E3e 2024-02-01T10:27:46.317 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Tdhvd0J/tReF9J4KOhCRWi0KvB0= 2024-02-01T10:27:46.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Tdhvd0J/tReF9J4KOhCRWi0KvB0=', 'Content-Length': '0'} 2024-02-01T10:27:46.320 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '9CJ3w1EusK8OST2k17sTdTM.cuh3E3e'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000031bc17b9a086509-0065bb7222-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:46.321 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:46.321 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:46.321 DEBUG:boto:path=/vclfcn-19/obj?versionId=9CJ3w1EusK8OST2k17sTdTM.cuh3E3e 2024-02-01T10:27:46.321 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=9CJ3w1EusK8OST2k17sTdTM.cuh3E3e 2024-02-01T10:27:46.321 DEBUG:boto:Method: GET 2024-02-01T10:27:46.321 DEBUG:boto:Path: /vclfcn-19/obj?versionId=9CJ3w1EusK8OST2k17sTdTM.cuh3E3e 2024-02-01T10:27:46.321 DEBUG:boto:Data: 2024-02-01T10:27:46.321 DEBUG:boto:Headers: {} 2024-02-01T10:27:46.321 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:46.322 DEBUG:boto:Port: 8000 2024-02-01T10:27:46.322 DEBUG:boto:Params: {} 2024-02-01T10:27:46.322 DEBUG:boto:Token: None 2024-02-01T10:27:46.322 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:46 GMT /vclfcn-19/obj?versionId=9CJ3w1EusK8OST2k17sTdTM.cuh3E3e 2024-02-01T10:27:46.322 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Tdhvd0J/tReF9J4KOhCRWi0KvB0= 2024-02-01T10:27:46.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Tdhvd0J/tReF9J4KOhCRWi0KvB0=', 'Content-Length': '0'} 2024-02-01T10:27:46.325 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '9CJ3w1EusK8OST2k17sTdTM.cuh3E3e'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001a88bee52536fbb5-0065bb7222-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:46.368 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:46.368 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:46.369 DEBUG:boto:path=/vclfcn-19/obj?versionId=9CJ3w1EusK8OST2k17sTdTM.cuh3E3e 2024-02-01T10:27:46.369 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=9CJ3w1EusK8OST2k17sTdTM.cuh3E3e 2024-02-01T10:27:46.369 DEBUG:boto:Method: HEAD 2024-02-01T10:27:46.369 DEBUG:boto:Path: /vclfcn-19/obj?versionId=9CJ3w1EusK8OST2k17sTdTM.cuh3E3e 2024-02-01T10:27:46.369 DEBUG:boto:Data: 2024-02-01T10:27:46.369 DEBUG:boto:Headers: {} 2024-02-01T10:27:46.369 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:46.369 DEBUG:boto:Port: 8001 2024-02-01T10:27:46.369 DEBUG:boto:Params: {} 2024-02-01T10:27:46.370 DEBUG:boto:Token: None 2024-02-01T10:27:46.370 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:46 GMT /vclfcn-19/obj?versionId=9CJ3w1EusK8OST2k17sTdTM.cuh3E3e 2024-02-01T10:27:46.370 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Y7bM1G/Myv16r9fpcJct0/0rUx8= 2024-02-01T10:27:46.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Y7bM1G/Myv16r9fpcJct0/0rUx8=', 'Content-Length': '0'} 2024-02-01T10:27:46.373 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '9CJ3w1EusK8OST2k17sTdTM.cuh3E3e'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003c9b2090e6825e60-0065bb7222-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:46.374 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:46.374 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:46.374 DEBUG:boto:path=/vclfcn-19/obj?versionId=9CJ3w1EusK8OST2k17sTdTM.cuh3E3e 2024-02-01T10:27:46.374 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=9CJ3w1EusK8OST2k17sTdTM.cuh3E3e 2024-02-01T10:27:46.374 DEBUG:boto:Method: HEAD 2024-02-01T10:27:46.374 DEBUG:boto:Path: /vclfcn-19/obj?versionId=9CJ3w1EusK8OST2k17sTdTM.cuh3E3e 2024-02-01T10:27:46.374 DEBUG:boto:Data: 2024-02-01T10:27:46.374 DEBUG:boto:Headers: {} 2024-02-01T10:27:46.375 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:46.375 DEBUG:boto:Port: 8000 2024-02-01T10:27:46.375 DEBUG:boto:Params: {} 2024-02-01T10:27:46.375 DEBUG:boto:Token: None 2024-02-01T10:27:46.375 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:46 GMT /vclfcn-19/obj?versionId=9CJ3w1EusK8OST2k17sTdTM.cuh3E3e 2024-02-01T10:27:46.375 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Y7bM1G/Myv16r9fpcJct0/0rUx8= 2024-02-01T10:27:46.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Y7bM1G/Myv16r9fpcJct0/0rUx8=', 'Content-Length': '0'} 2024-02-01T10:27:46.378 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '9CJ3w1EusK8OST2k17sTdTM.cuh3E3e'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a92dc1a151521642-0065bb7222-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:46.379 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:46.379 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:46.379 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:46.379 DEBUG:boto:path=/vclfcn-19/obj?versionId=9CJ3w1EusK8OST2k17sTdTM.cuh3E3e 2024-02-01T10:27:46.379 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=9CJ3w1EusK8OST2k17sTdTM.cuh3E3e 2024-02-01T10:27:46.380 DEBUG:boto:Method: GET 2024-02-01T10:27:46.380 DEBUG:boto:Path: /vclfcn-19/obj?versionId=9CJ3w1EusK8OST2k17sTdTM.cuh3E3e 2024-02-01T10:27:46.380 DEBUG:boto:Data: 2024-02-01T10:27:46.380 DEBUG:boto:Headers: {} 2024-02-01T10:27:46.380 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:46.380 DEBUG:boto:Port: 8001 2024-02-01T10:27:46.380 DEBUG:boto:Params: {} 2024-02-01T10:27:46.380 DEBUG:boto:Token: None 2024-02-01T10:27:46.381 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:46 GMT /vclfcn-19/obj?versionId=9CJ3w1EusK8OST2k17sTdTM.cuh3E3e 2024-02-01T10:27:46.381 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Tdhvd0J/tReF9J4KOhCRWi0KvB0= 2024-02-01T10:27:46.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Tdhvd0J/tReF9J4KOhCRWi0KvB0=', 'Content-Length': '0'} 2024-02-01T10:27:46.383 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '9CJ3w1EusK8OST2k17sTdTM.cuh3E3e'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000019c7af05a24905d4-0065bb7222-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:46.424 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:46.424 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:46.424 DEBUG:boto:path=/vclfcn-19/obj?versionId=9CJ3w1EusK8OST2k17sTdTM.cuh3E3e 2024-02-01T10:27:46.424 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=9CJ3w1EusK8OST2k17sTdTM.cuh3E3e 2024-02-01T10:27:46.425 DEBUG:boto:Method: GET 2024-02-01T10:27:46.425 DEBUG:boto:Path: /vclfcn-19/obj?versionId=9CJ3w1EusK8OST2k17sTdTM.cuh3E3e 2024-02-01T10:27:46.425 DEBUG:boto:Data: 2024-02-01T10:27:46.425 DEBUG:boto:Headers: {} 2024-02-01T10:27:46.425 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:46.425 DEBUG:boto:Port: 8000 2024-02-01T10:27:46.425 DEBUG:boto:Params: {} 2024-02-01T10:27:46.425 DEBUG:boto:Token: None 2024-02-01T10:27:46.425 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:46 GMT /vclfcn-19/obj?versionId=9CJ3w1EusK8OST2k17sTdTM.cuh3E3e 2024-02-01T10:27:46.426 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Tdhvd0J/tReF9J4KOhCRWi0KvB0= 2024-02-01T10:27:46.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Tdhvd0J/tReF9J4KOhCRWi0KvB0=', 'Content-Length': '0'} 2024-02-01T10:27:46.429 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '9CJ3w1EusK8OST2k17sTdTM.cuh3E3e'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f2c5963ec903765e-0065bb7222-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:46.472 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:46.472 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:46.473 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:46.473 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:46.473 DEBUG:boto:Method: GET 2024-02-01T10:27:46.473 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:46.473 DEBUG:boto:Data: 2024-02-01T10:27:46.473 DEBUG:boto:Headers: {} 2024-02-01T10:27:46.473 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:46.473 DEBUG:boto:Port: 8001 2024-02-01T10:27:46.473 DEBUG:boto:Params: {} 2024-02-01T10:27:46.473 DEBUG:boto:Token: None 2024-02-01T10:27:46.474 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:46 GMT /vclfcn-19/ 2024-02-01T10:27:46.474 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FQtT/TF8IcJdqG6hRlZ1dT71FEE= 2024-02-01T10:27:46.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FQtT/TF8IcJdqG6hRlZ1dT71FEE=', 'Content-Length': '0'} 2024-02-01T10:27:46.479 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3f4a0ee33135638-0065bb7222-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:46.520 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:46.521 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:46.521 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:46.521 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:46.521 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:46.521 DEBUG:boto:Method: GET 2024-02-01T10:27:46.521 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:46.521 DEBUG:boto:Data: 2024-02-01T10:27:46.521 DEBUG:boto:Headers: {} 2024-02-01T10:27:46.521 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:46.521 DEBUG:boto:Port: 8000 2024-02-01T10:27:46.521 DEBUG:boto:Params: {} 2024-02-01T10:27:46.522 DEBUG:boto:Token: None 2024-02-01T10:27:46.522 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:46 GMT /vclfcn-19/ 2024-02-01T10:27:46.522 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FQtT/TF8IcJdqG6hRlZ1dT71FEE= 2024-02-01T10:27:46.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FQtT/TF8IcJdqG6hRlZ1dT71FEE=', 'Content-Length': '0'} 2024-02-01T10:27:46.527 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000044de297d1774425e-0065bb7222-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:46.527 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:46.528 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:46.528 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:46.528 DEBUG:boto:Method: HEAD 2024-02-01T10:27:46.528 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:46.528 DEBUG:boto:Data: 2024-02-01T10:27:46.528 DEBUG:boto:Headers: {} 2024-02-01T10:27:46.528 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:46.528 DEBUG:boto:Port: 8001 2024-02-01T10:27:46.529 DEBUG:boto:Params: {} 2024-02-01T10:27:46.529 DEBUG:boto:Token: None 2024-02-01T10:27:46.529 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:46 GMT /vclfcn-19/obj 2024-02-01T10:27:46.529 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:kisoj1CchsoK0IEwzE9RgWblk7I= 2024-02-01T10:27:46.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:kisoj1CchsoK0IEwzE9RgWblk7I=', 'Content-Length': '0'} 2024-02-01T10:27:46.535 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d3830418aefb2206-0065bb7222-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:46.535 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:46.535 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:46.535 DEBUG:boto:Method: HEAD 2024-02-01T10:27:46.535 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:46.535 DEBUG:boto:Data: 2024-02-01T10:27:46.535 DEBUG:boto:Headers: {} 2024-02-01T10:27:46.536 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:46.536 DEBUG:boto:Port: 8000 2024-02-01T10:27:46.536 DEBUG:boto:Params: {} 2024-02-01T10:27:46.536 DEBUG:boto:Token: None 2024-02-01T10:27:46.536 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:46 GMT /vclfcn-19/obj 2024-02-01T10:27:46.536 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:kisoj1CchsoK0IEwzE9RgWblk7I= 2024-02-01T10:27:46.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:kisoj1CchsoK0IEwzE9RgWblk7I=', 'Content-Length': '0'} 2024-02-01T10:27:46.543 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000de293894cfe20e70-0065bb7222-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:46.543 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:46.543 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:46.543 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:46.544 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:46.544 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:46.544 DEBUG:boto:Method: GET 2024-02-01T10:27:46.544 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:46.544 DEBUG:boto:Data: 2024-02-01T10:27:46.544 DEBUG:boto:Headers: {} 2024-02-01T10:27:46.544 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:46.544 DEBUG:boto:Port: 8001 2024-02-01T10:27:46.544 DEBUG:boto:Params: {} 2024-02-01T10:27:46.545 DEBUG:boto:Token: None 2024-02-01T10:27:46.545 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:46 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:46.545 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:O8uzubsS9dRIYP5bRpQM0mMTkEE= 2024-02-01T10:27:46.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:O8uzubsS9dRIYP5bRpQM0mMTkEE=', 'Content-Length': '0'} 2024-02-01T10:27:46.548 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001b5a1e8a85ae113d-0065bb7222-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:46.588 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:46.589 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:46.589 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:46.589 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:46.589 DEBUG:boto:Method: GET 2024-02-01T10:27:46.589 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:46.589 DEBUG:boto:Data: 2024-02-01T10:27:46.589 DEBUG:boto:Headers: {} 2024-02-01T10:27:46.589 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:46.589 DEBUG:boto:Port: 8000 2024-02-01T10:27:46.589 DEBUG:boto:Params: {} 2024-02-01T10:27:46.590 DEBUG:boto:Token: None 2024-02-01T10:27:46.590 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:46 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:46.590 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:O8uzubsS9dRIYP5bRpQM0mMTkEE= 2024-02-01T10:27:46.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:O8uzubsS9dRIYP5bRpQM0mMTkEE=', 'Content-Length': '0'} 2024-02-01T10:27:46.593 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006ea9aa3bcf29a319-0065bb7222-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:46.640 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:46.640 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:46.640 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:46.641 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:46.641 DEBUG:boto:Method: GET 2024-02-01T10:27:46.641 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:46.641 DEBUG:boto:Data: 2024-02-01T10:27:46.641 DEBUG:boto:Headers: {} 2024-02-01T10:27:46.641 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:46.641 DEBUG:boto:Port: 8001 2024-02-01T10:27:46.641 DEBUG:boto:Params: {} 2024-02-01T10:27:46.641 DEBUG:boto:Token: None 2024-02-01T10:27:46.642 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:46 GMT /vclfcn-19/ 2024-02-01T10:27:46.642 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FQtT/TF8IcJdqG6hRlZ1dT71FEE= 2024-02-01T10:27:46.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FQtT/TF8IcJdqG6hRlZ1dT71FEE=', 'Content-Length': '0'} 2024-02-01T10:27:46.647 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004221a08e52e25d79-0065bb7222-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:46.688 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:46.689 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:46.689 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:46.689 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:46.689 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:46.689 DEBUG:boto:Method: GET 2024-02-01T10:27:46.689 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:46.689 DEBUG:boto:Data: 2024-02-01T10:27:46.689 DEBUG:boto:Headers: {} 2024-02-01T10:27:46.689 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:46.689 DEBUG:boto:Port: 8000 2024-02-01T10:27:46.689 DEBUG:boto:Params: {} 2024-02-01T10:27:46.690 DEBUG:boto:Token: None 2024-02-01T10:27:46.690 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:46 GMT /vclfcn-19/ 2024-02-01T10:27:46.690 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FQtT/TF8IcJdqG6hRlZ1dT71FEE= 2024-02-01T10:27:46.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FQtT/TF8IcJdqG6hRlZ1dT71FEE=', 'Content-Length': '0'} 2024-02-01T10:27:46.695 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f42b4e573d9f3862-0065bb7222-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:46.696 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:46.696 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:46.696 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:46.696 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:46.696 DEBUG:boto:path=/vclfcn-19/obj?versionId=Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv 2024-02-01T10:27:46.696 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv 2024-02-01T10:27:46.696 DEBUG:boto:Method: GET 2024-02-01T10:27:46.697 DEBUG:boto:Path: /vclfcn-19/obj?versionId=Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv 2024-02-01T10:27:46.697 DEBUG:boto:Data: 2024-02-01T10:27:46.697 DEBUG:boto:Headers: {} 2024-02-01T10:27:46.697 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:46.697 DEBUG:boto:Port: 8001 2024-02-01T10:27:46.697 DEBUG:boto:Params: {} 2024-02-01T10:27:46.697 DEBUG:boto:Token: None 2024-02-01T10:27:46.697 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:46 GMT /vclfcn-19/obj?versionId=Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv 2024-02-01T10:27:46.697 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8dO14CMURxcIOg4ceUuymg/sK7U= 2024-02-01T10:27:46.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8dO14CMURxcIOg4ceUuymg/sK7U=', 'Content-Length': '0'} 2024-02-01T10:27:46.700 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000440d80fd94ea4e2d-0065bb7222-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:46.701 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:46.701 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:46.701 DEBUG:boto:path=/vclfcn-19/obj?versionId=Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv 2024-02-01T10:27:46.701 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv 2024-02-01T10:27:46.701 DEBUG:boto:Method: GET 2024-02-01T10:27:46.701 DEBUG:boto:Path: /vclfcn-19/obj?versionId=Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv 2024-02-01T10:27:46.701 DEBUG:boto:Data: 2024-02-01T10:27:46.701 DEBUG:boto:Headers: {} 2024-02-01T10:27:46.701 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:46.701 DEBUG:boto:Port: 8000 2024-02-01T10:27:46.702 DEBUG:boto:Params: {} 2024-02-01T10:27:46.702 DEBUG:boto:Token: None 2024-02-01T10:27:46.702 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:46 GMT /vclfcn-19/obj?versionId=Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv 2024-02-01T10:27:46.702 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8dO14CMURxcIOg4ceUuymg/sK7U= 2024-02-01T10:27:46.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8dO14CMURxcIOg4ceUuymg/sK7U=', 'Content-Length': '0'} 2024-02-01T10:27:46.705 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a757c37148640a80-0065bb7222-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:46.748 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:46.748 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:46.748 DEBUG:boto:path=/vclfcn-19/obj?versionId=Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv 2024-02-01T10:27:46.749 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv 2024-02-01T10:27:46.749 DEBUG:boto:Method: HEAD 2024-02-01T10:27:46.749 DEBUG:boto:Path: /vclfcn-19/obj?versionId=Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv 2024-02-01T10:27:46.749 DEBUG:boto:Data: 2024-02-01T10:27:46.749 DEBUG:boto:Headers: {} 2024-02-01T10:27:46.749 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:46.749 DEBUG:boto:Port: 8001 2024-02-01T10:27:46.749 DEBUG:boto:Params: {} 2024-02-01T10:27:46.749 DEBUG:boto:Token: None 2024-02-01T10:27:46.750 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:46 GMT /vclfcn-19/obj?versionId=Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv 2024-02-01T10:27:46.750 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:31ZDwFWOuebygGyWmqovr/vGkbI= 2024-02-01T10:27:46.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:31ZDwFWOuebygGyWmqovr/vGkbI=', 'Content-Length': '0'} 2024-02-01T10:27:46.752 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000022dd33744545e296-0065bb7222-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:46.753 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:46.753 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:46.753 DEBUG:boto:path=/vclfcn-19/obj?versionId=Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv 2024-02-01T10:27:46.753 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv 2024-02-01T10:27:46.753 DEBUG:boto:Method: HEAD 2024-02-01T10:27:46.753 DEBUG:boto:Path: /vclfcn-19/obj?versionId=Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv 2024-02-01T10:27:46.753 DEBUG:boto:Data: 2024-02-01T10:27:46.753 DEBUG:boto:Headers: {} 2024-02-01T10:27:46.753 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:46.754 DEBUG:boto:Port: 8000 2024-02-01T10:27:46.754 DEBUG:boto:Params: {} 2024-02-01T10:27:46.754 DEBUG:boto:Token: None 2024-02-01T10:27:46.754 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:46 GMT /vclfcn-19/obj?versionId=Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv 2024-02-01T10:27:46.754 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:31ZDwFWOuebygGyWmqovr/vGkbI= 2024-02-01T10:27:46.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:31ZDwFWOuebygGyWmqovr/vGkbI=', 'Content-Length': '0'} 2024-02-01T10:27:46.757 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003b02fcf2164df56d-0065bb7222-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:46.757 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:46.757 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:46.758 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:46.758 DEBUG:boto:path=/vclfcn-19/obj?versionId=Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv 2024-02-01T10:27:46.758 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv 2024-02-01T10:27:46.758 DEBUG:boto:Method: GET 2024-02-01T10:27:46.758 DEBUG:boto:Path: /vclfcn-19/obj?versionId=Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv 2024-02-01T10:27:46.758 DEBUG:boto:Data: 2024-02-01T10:27:46.758 DEBUG:boto:Headers: {} 2024-02-01T10:27:46.758 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:46.758 DEBUG:boto:Port: 8001 2024-02-01T10:27:46.758 DEBUG:boto:Params: {} 2024-02-01T10:27:46.759 DEBUG:boto:Token: None 2024-02-01T10:27:46.759 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:46 GMT /vclfcn-19/obj?versionId=Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv 2024-02-01T10:27:46.759 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8dO14CMURxcIOg4ceUuymg/sK7U= 2024-02-01T10:27:46.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8dO14CMURxcIOg4ceUuymg/sK7U=', 'Content-Length': '0'} 2024-02-01T10:27:46.763 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000020ec47b9ed014d2f-0065bb7222-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:46.764 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:46.764 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:46.764 DEBUG:boto:path=/vclfcn-19/obj?versionId=Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv 2024-02-01T10:27:46.764 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv 2024-02-01T10:27:46.764 DEBUG:boto:Method: GET 2024-02-01T10:27:46.764 DEBUG:boto:Path: /vclfcn-19/obj?versionId=Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv 2024-02-01T10:27:46.764 DEBUG:boto:Data: 2024-02-01T10:27:46.764 DEBUG:boto:Headers: {} 2024-02-01T10:27:46.764 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:46.765 DEBUG:boto:Port: 8000 2024-02-01T10:27:46.765 DEBUG:boto:Params: {} 2024-02-01T10:27:46.765 DEBUG:boto:Token: None 2024-02-01T10:27:46.765 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:46 GMT /vclfcn-19/obj?versionId=Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv 2024-02-01T10:27:46.765 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8dO14CMURxcIOg4ceUuymg/sK7U= 2024-02-01T10:27:46.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8dO14CMURxcIOg4ceUuymg/sK7U=', 'Content-Length': '0'} 2024-02-01T10:27:46.768 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a7431055aac131f8-0065bb7222-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:46.812 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:46.812 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:46.812 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:46.813 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:46.813 DEBUG:boto:Method: GET 2024-02-01T10:27:46.813 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:46.813 DEBUG:boto:Data: 2024-02-01T10:27:46.813 DEBUG:boto:Headers: {} 2024-02-01T10:27:46.813 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:46.813 DEBUG:boto:Port: 8001 2024-02-01T10:27:46.813 DEBUG:boto:Params: {} 2024-02-01T10:27:46.813 DEBUG:boto:Token: None 2024-02-01T10:27:46.814 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:46 GMT /vclfcn-19/ 2024-02-01T10:27:46.814 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FQtT/TF8IcJdqG6hRlZ1dT71FEE= 2024-02-01T10:27:46.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FQtT/TF8IcJdqG6hRlZ1dT71FEE=', 'Content-Length': '0'} 2024-02-01T10:27:46.819 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ef1d15f100c80a0-0065bb7222-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:46.819 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:46.820 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:46.820 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:46.820 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:46.820 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:46.820 DEBUG:boto:Method: GET 2024-02-01T10:27:46.820 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:46.820 DEBUG:boto:Data: 2024-02-01T10:27:46.820 DEBUG:boto:Headers: {} 2024-02-01T10:27:46.820 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:46.820 DEBUG:boto:Port: 8000 2024-02-01T10:27:46.820 DEBUG:boto:Params: {} 2024-02-01T10:27:46.821 DEBUG:boto:Token: None 2024-02-01T10:27:46.821 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:46 GMT /vclfcn-19/ 2024-02-01T10:27:46.821 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FQtT/TF8IcJdqG6hRlZ1dT71FEE= 2024-02-01T10:27:46.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FQtT/TF8IcJdqG6hRlZ1dT71FEE=', 'Content-Length': '0'} 2024-02-01T10:27:46.826 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d65219a5f6f7e9d-0065bb7222-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:46.826 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:46.826 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:46.827 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:46.827 DEBUG:boto:Method: HEAD 2024-02-01T10:27:46.828 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:46.828 DEBUG:boto:Data: 2024-02-01T10:27:46.828 DEBUG:boto:Headers: {} 2024-02-01T10:27:46.828 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:46.828 DEBUG:boto:Port: 8001 2024-02-01T10:27:46.828 DEBUG:boto:Params: {} 2024-02-01T10:27:46.828 DEBUG:boto:Token: None 2024-02-01T10:27:46.828 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:46 GMT /vclfcn-19/obj 2024-02-01T10:27:46.829 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:kisoj1CchsoK0IEwzE9RgWblk7I= 2024-02-01T10:27:46.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:kisoj1CchsoK0IEwzE9RgWblk7I=', 'Content-Length': '0'} 2024-02-01T10:27:46.834 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b3f67753a450fe6b-0065bb7222-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:46.834 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:46.835 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:46.835 DEBUG:boto:Method: HEAD 2024-02-01T10:27:46.835 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:46.835 DEBUG:boto:Data: 2024-02-01T10:27:46.835 DEBUG:boto:Headers: {} 2024-02-01T10:27:46.835 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:46.835 DEBUG:boto:Port: 8000 2024-02-01T10:27:46.835 DEBUG:boto:Params: {} 2024-02-01T10:27:46.835 DEBUG:boto:Token: None 2024-02-01T10:27:46.835 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:46 GMT /vclfcn-19/obj 2024-02-01T10:27:46.836 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:kisoj1CchsoK0IEwzE9RgWblk7I= 2024-02-01T10:27:46.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:kisoj1CchsoK0IEwzE9RgWblk7I=', 'Content-Length': '0'} 2024-02-01T10:27:46.842 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ef44dbf6e67ac1f6-0065bb7222-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:46.842 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:46.842 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:46.843 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:46.843 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:46.843 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:46.843 DEBUG:boto:Method: GET 2024-02-01T10:27:46.843 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:46.843 DEBUG:boto:Data: 2024-02-01T10:27:46.843 DEBUG:boto:Headers: {} 2024-02-01T10:27:46.843 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:46.843 DEBUG:boto:Port: 8001 2024-02-01T10:27:46.843 DEBUG:boto:Params: {} 2024-02-01T10:27:46.843 DEBUG:boto:Token: None 2024-02-01T10:27:46.844 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:46 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:46.844 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:O8uzubsS9dRIYP5bRpQM0mMTkEE= 2024-02-01T10:27:46.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:O8uzubsS9dRIYP5bRpQM0mMTkEE=', 'Content-Length': '0'} 2024-02-01T10:27:46.846 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000144981477a674ba-0065bb7222-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:46.888 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:46.888 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:46.889 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:46.889 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:46.889 DEBUG:boto:Method: GET 2024-02-01T10:27:46.889 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:46.889 DEBUG:boto:Data: 2024-02-01T10:27:46.889 DEBUG:boto:Headers: {} 2024-02-01T10:27:46.889 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:46.889 DEBUG:boto:Port: 8000 2024-02-01T10:27:46.889 DEBUG:boto:Params: {} 2024-02-01T10:27:46.889 DEBUG:boto:Token: None 2024-02-01T10:27:46.890 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:46 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:46.890 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:O8uzubsS9dRIYP5bRpQM0mMTkEE= 2024-02-01T10:27:46.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:O8uzubsS9dRIYP5bRpQM0mMTkEE=', 'Content-Length': '0'} 2024-02-01T10:27:46.893 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e5ee7d8d81337163-0065bb7222-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:46.940 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:46.940 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:46.940 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:46.940 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:46.941 DEBUG:boto:Method: GET 2024-02-01T10:27:46.941 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:46.941 DEBUG:boto:Data: 2024-02-01T10:27:46.941 DEBUG:boto:Headers: {} 2024-02-01T10:27:46.941 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:46.941 DEBUG:boto:Port: 8001 2024-02-01T10:27:46.941 DEBUG:boto:Params: {} 2024-02-01T10:27:46.941 DEBUG:boto:Token: None 2024-02-01T10:27:46.941 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:46 GMT /vclfcn-19/ 2024-02-01T10:27:46.942 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FQtT/TF8IcJdqG6hRlZ1dT71FEE= 2024-02-01T10:27:46.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FQtT/TF8IcJdqG6hRlZ1dT71FEE=', 'Content-Length': '0'} 2024-02-01T10:27:46.947 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e7ab571fa5200d5f-0065bb7222-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:46.947 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:46.948 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:46.948 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:46.948 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:46.948 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:46.948 DEBUG:boto:Method: GET 2024-02-01T10:27:46.948 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:46.948 DEBUG:boto:Data: 2024-02-01T10:27:46.949 DEBUG:boto:Headers: {} 2024-02-01T10:27:46.949 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:46.949 DEBUG:boto:Port: 8000 2024-02-01T10:27:46.949 DEBUG:boto:Params: {} 2024-02-01T10:27:46.949 DEBUG:boto:Token: None 2024-02-01T10:27:46.949 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:46 GMT /vclfcn-19/ 2024-02-01T10:27:46.949 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FQtT/TF8IcJdqG6hRlZ1dT71FEE= 2024-02-01T10:27:46.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FQtT/TF8IcJdqG6hRlZ1dT71FEE=', 'Content-Length': '0'} 2024-02-01T10:27:46.954 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d1ba568e7a2b584e-0065bb7222-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:46.955 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:46.955 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:46.955 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:46.955 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:46.955 DEBUG:boto:path=/vclfcn-19/obj?versionId=iGO2QyMOL81IaF-H4sHDqyECPlnZBmE 2024-02-01T10:27:46.955 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=iGO2QyMOL81IaF-H4sHDqyECPlnZBmE 2024-02-01T10:27:46.955 DEBUG:boto:Method: GET 2024-02-01T10:27:46.956 DEBUG:boto:Path: /vclfcn-19/obj?versionId=iGO2QyMOL81IaF-H4sHDqyECPlnZBmE 2024-02-01T10:27:46.956 DEBUG:boto:Data: 2024-02-01T10:27:46.956 DEBUG:boto:Headers: {} 2024-02-01T10:27:46.956 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:46.956 DEBUG:boto:Port: 8001 2024-02-01T10:27:46.956 DEBUG:boto:Params: {} 2024-02-01T10:27:46.956 DEBUG:boto:Token: None 2024-02-01T10:27:46.956 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:46 GMT /vclfcn-19/obj?versionId=iGO2QyMOL81IaF-H4sHDqyECPlnZBmE 2024-02-01T10:27:46.956 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:IdwEn7L/uJDajQoumiRaTAZeFFE= 2024-02-01T10:27:46.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:IdwEn7L/uJDajQoumiRaTAZeFFE=', 'Content-Length': '0'} 2024-02-01T10:27:46.959 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'iGO2QyMOL81IaF-H4sHDqyECPlnZBmE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000351d810f7392f6c2-0065bb7222-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:46.959 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:46.960 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:46.960 DEBUG:boto:path=/vclfcn-19/obj?versionId=iGO2QyMOL81IaF-H4sHDqyECPlnZBmE 2024-02-01T10:27:46.960 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=iGO2QyMOL81IaF-H4sHDqyECPlnZBmE 2024-02-01T10:27:46.960 DEBUG:boto:Method: GET 2024-02-01T10:27:46.960 DEBUG:boto:Path: /vclfcn-19/obj?versionId=iGO2QyMOL81IaF-H4sHDqyECPlnZBmE 2024-02-01T10:27:46.960 DEBUG:boto:Data: 2024-02-01T10:27:46.960 DEBUG:boto:Headers: {} 2024-02-01T10:27:46.960 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:46.960 DEBUG:boto:Port: 8000 2024-02-01T10:27:46.960 DEBUG:boto:Params: {} 2024-02-01T10:27:46.960 DEBUG:boto:Token: None 2024-02-01T10:27:46.961 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:46 GMT /vclfcn-19/obj?versionId=iGO2QyMOL81IaF-H4sHDqyECPlnZBmE 2024-02-01T10:27:46.961 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:IdwEn7L/uJDajQoumiRaTAZeFFE= 2024-02-01T10:27:46.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:IdwEn7L/uJDajQoumiRaTAZeFFE=', 'Content-Length': '0'} 2024-02-01T10:27:46.963 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'iGO2QyMOL81IaF-H4sHDqyECPlnZBmE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000046f196ac21f0e115-0065bb7222-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:46.964 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:46.964 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:46.964 DEBUG:boto:path=/vclfcn-19/obj?versionId=iGO2QyMOL81IaF-H4sHDqyECPlnZBmE 2024-02-01T10:27:46.964 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=iGO2QyMOL81IaF-H4sHDqyECPlnZBmE 2024-02-01T10:27:46.964 DEBUG:boto:Method: HEAD 2024-02-01T10:27:46.964 DEBUG:boto:Path: /vclfcn-19/obj?versionId=iGO2QyMOL81IaF-H4sHDqyECPlnZBmE 2024-02-01T10:27:46.964 DEBUG:boto:Data: 2024-02-01T10:27:46.964 DEBUG:boto:Headers: {} 2024-02-01T10:27:46.965 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:46.965 DEBUG:boto:Port: 8001 2024-02-01T10:27:46.965 DEBUG:boto:Params: {} 2024-02-01T10:27:46.967 DEBUG:boto:Token: None 2024-02-01T10:27:46.967 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:46 GMT /vclfcn-19/obj?versionId=iGO2QyMOL81IaF-H4sHDqyECPlnZBmE 2024-02-01T10:27:46.967 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:kvEQVzHwsuONyTvyCk8w3V1T5dU= 2024-02-01T10:27:46.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:kvEQVzHwsuONyTvyCk8w3V1T5dU=', 'Content-Length': '0'} 2024-02-01T10:27:46.970 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'iGO2QyMOL81IaF-H4sHDqyECPlnZBmE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b2ea3b929433f48e-0065bb7222-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:46.970 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:46.971 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:46.971 DEBUG:boto:path=/vclfcn-19/obj?versionId=iGO2QyMOL81IaF-H4sHDqyECPlnZBmE 2024-02-01T10:27:46.971 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=iGO2QyMOL81IaF-H4sHDqyECPlnZBmE 2024-02-01T10:27:46.971 DEBUG:boto:Method: HEAD 2024-02-01T10:27:46.971 DEBUG:boto:Path: /vclfcn-19/obj?versionId=iGO2QyMOL81IaF-H4sHDqyECPlnZBmE 2024-02-01T10:27:46.971 DEBUG:boto:Data: 2024-02-01T10:27:46.971 DEBUG:boto:Headers: {} 2024-02-01T10:27:46.971 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:46.971 DEBUG:boto:Port: 8000 2024-02-01T10:27:46.971 DEBUG:boto:Params: {} 2024-02-01T10:27:46.971 DEBUG:boto:Token: None 2024-02-01T10:27:46.972 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:46 GMT /vclfcn-19/obj?versionId=iGO2QyMOL81IaF-H4sHDqyECPlnZBmE 2024-02-01T10:27:46.972 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:kvEQVzHwsuONyTvyCk8w3V1T5dU= 2024-02-01T10:27:46.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:kvEQVzHwsuONyTvyCk8w3V1T5dU=', 'Content-Length': '0'} 2024-02-01T10:27:46.975 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'iGO2QyMOL81IaF-H4sHDqyECPlnZBmE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000051096e18c011f7f2-0065bb7222-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:46.975 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:46.975 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:46.975 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:46.975 DEBUG:boto:path=/vclfcn-19/obj?versionId=iGO2QyMOL81IaF-H4sHDqyECPlnZBmE 2024-02-01T10:27:46.975 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=iGO2QyMOL81IaF-H4sHDqyECPlnZBmE 2024-02-01T10:27:46.975 DEBUG:boto:Method: GET 2024-02-01T10:27:46.975 DEBUG:boto:Path: /vclfcn-19/obj?versionId=iGO2QyMOL81IaF-H4sHDqyECPlnZBmE 2024-02-01T10:27:46.976 DEBUG:boto:Data: 2024-02-01T10:27:46.976 DEBUG:boto:Headers: {} 2024-02-01T10:27:46.976 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:46.976 DEBUG:boto:Port: 8001 2024-02-01T10:27:46.976 DEBUG:boto:Params: {} 2024-02-01T10:27:46.976 DEBUG:boto:Token: None 2024-02-01T10:27:46.976 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:46 GMT /vclfcn-19/obj?versionId=iGO2QyMOL81IaF-H4sHDqyECPlnZBmE 2024-02-01T10:27:46.976 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:IdwEn7L/uJDajQoumiRaTAZeFFE= 2024-02-01T10:27:46.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:IdwEn7L/uJDajQoumiRaTAZeFFE=', 'Content-Length': '0'} 2024-02-01T10:27:46.979 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'iGO2QyMOL81IaF-H4sHDqyECPlnZBmE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000224418efc0726a2c-0065bb7222-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:47.020 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:47.020 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:47.020 DEBUG:boto:path=/vclfcn-19/obj?versionId=iGO2QyMOL81IaF-H4sHDqyECPlnZBmE 2024-02-01T10:27:47.020 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=iGO2QyMOL81IaF-H4sHDqyECPlnZBmE 2024-02-01T10:27:47.021 DEBUG:boto:Method: GET 2024-02-01T10:27:47.021 DEBUG:boto:Path: /vclfcn-19/obj?versionId=iGO2QyMOL81IaF-H4sHDqyECPlnZBmE 2024-02-01T10:27:47.021 DEBUG:boto:Data: 2024-02-01T10:27:47.021 DEBUG:boto:Headers: {} 2024-02-01T10:27:47.021 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:47.021 DEBUG:boto:Port: 8000 2024-02-01T10:27:47.021 DEBUG:boto:Params: {} 2024-02-01T10:27:47.021 DEBUG:boto:Token: None 2024-02-01T10:27:47.022 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:47 GMT /vclfcn-19/obj?versionId=iGO2QyMOL81IaF-H4sHDqyECPlnZBmE 2024-02-01T10:27:47.022 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:dIXIZcjdfCFhi36QNaJbP+GX5kI= 2024-02-01T10:27:47.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:47 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dIXIZcjdfCFhi36QNaJbP+GX5kI=', 'Content-Length': '0'} 2024-02-01T10:27:47.025 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'iGO2QyMOL81IaF-H4sHDqyECPlnZBmE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000086fc795cb717c03d-0065bb7223-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:47 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:47.026 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:47.026 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:47.026 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:47.026 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:47.026 DEBUG:boto:Method: GET 2024-02-01T10:27:47.026 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:47.026 DEBUG:boto:Data: 2024-02-01T10:27:47.026 DEBUG:boto:Headers: {} 2024-02-01T10:27:47.026 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:47.026 DEBUG:boto:Port: 8001 2024-02-01T10:27:47.027 DEBUG:boto:Params: {} 2024-02-01T10:27:47.027 DEBUG:boto:Token: None 2024-02-01T10:27:47.027 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:47 GMT /vclfcn-19/ 2024-02-01T10:27:47.027 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:KmYxnpeez+AQ0NEHLtbcT0K5zaw= 2024-02-01T10:27:47.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:47 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:KmYxnpeez+AQ0NEHLtbcT0K5zaw=', 'Content-Length': '0'} 2024-02-01T10:27:47.032 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ef096040e5112c8-0065bb7223-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:47 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:47.072 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:47.073 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:47.073 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:47.073 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:47.073 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:47.073 DEBUG:boto:Method: GET 2024-02-01T10:27:47.073 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:47.073 DEBUG:boto:Data: 2024-02-01T10:27:47.073 DEBUG:boto:Headers: {} 2024-02-01T10:27:47.073 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:47.073 DEBUG:boto:Port: 8000 2024-02-01T10:27:47.074 DEBUG:boto:Params: {} 2024-02-01T10:27:47.074 DEBUG:boto:Token: None 2024-02-01T10:27:47.074 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:47 GMT /vclfcn-19/ 2024-02-01T10:27:47.074 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:KmYxnpeez+AQ0NEHLtbcT0K5zaw= 2024-02-01T10:27:47.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:47 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:KmYxnpeez+AQ0NEHLtbcT0K5zaw=', 'Content-Length': '0'} 2024-02-01T10:27:47.080 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015dbfc23e39da67a-0065bb7223-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:47 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:47.080 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:47.080 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:47.080 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:47.081 DEBUG:boto:Method: HEAD 2024-02-01T10:27:47.081 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:47.081 DEBUG:boto:Data: 2024-02-01T10:27:47.081 DEBUG:boto:Headers: {} 2024-02-01T10:27:47.081 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:47.081 DEBUG:boto:Port: 8001 2024-02-01T10:27:47.081 DEBUG:boto:Params: {} 2024-02-01T10:27:47.081 DEBUG:boto:Token: None 2024-02-01T10:27:47.081 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:47 GMT /vclfcn-19/obj 2024-02-01T10:27:47.081 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:im9/JRJ4oRvE3tKXp3rZR6D9bxY= 2024-02-01T10:27:47.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:47 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:im9/JRJ4oRvE3tKXp3rZR6D9bxY=', 'Content-Length': '0'} 2024-02-01T10:27:47.087 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000097ebac9d0376a3c1-0065bb7223-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:47 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:47.088 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:47.088 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:47.088 DEBUG:boto:Method: HEAD 2024-02-01T10:27:47.088 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:47.088 DEBUG:boto:Data: 2024-02-01T10:27:47.088 DEBUG:boto:Headers: {} 2024-02-01T10:27:47.088 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:47.089 DEBUG:boto:Port: 8000 2024-02-01T10:27:47.089 DEBUG:boto:Params: {} 2024-02-01T10:27:47.089 DEBUG:boto:Token: None 2024-02-01T10:27:47.089 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:47 GMT /vclfcn-19/obj 2024-02-01T10:27:47.089 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:im9/JRJ4oRvE3tKXp3rZR6D9bxY= 2024-02-01T10:27:47.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:47 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:im9/JRJ4oRvE3tKXp3rZR6D9bxY=', 'Content-Length': '0'} 2024-02-01T10:27:47.095 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ada5b028fb0ca628-0065bb7223-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:47 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:47.095 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:47.095 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:47.095 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:47.095 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:47.095 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:47.096 DEBUG:boto:Method: GET 2024-02-01T10:27:47.096 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:47.096 DEBUG:boto:Data: 2024-02-01T10:27:47.096 DEBUG:boto:Headers: {} 2024-02-01T10:27:47.096 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:47.096 DEBUG:boto:Port: 8001 2024-02-01T10:27:47.096 DEBUG:boto:Params: {} 2024-02-01T10:27:47.096 DEBUG:boto:Token: None 2024-02-01T10:27:47.096 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:47 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:47.096 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:LFEJ+2e+PTkGZsFMQEqIII+OQmY= 2024-02-01T10:27:47.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:47 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:LFEJ+2e+PTkGZsFMQEqIII+OQmY=', 'Content-Length': '0'} 2024-02-01T10:27:47.099 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e43432a72d05d57c-0065bb7223-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:47 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:47.144 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:47.144 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:47.144 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:47.145 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:47.145 DEBUG:boto:Method: GET 2024-02-01T10:27:47.145 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:47.145 DEBUG:boto:Data: 2024-02-01T10:27:47.145 DEBUG:boto:Headers: {} 2024-02-01T10:27:47.145 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:47.145 DEBUG:boto:Port: 8000 2024-02-01T10:27:47.145 DEBUG:boto:Params: {} 2024-02-01T10:27:47.145 DEBUG:boto:Token: None 2024-02-01T10:27:47.146 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:47 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:47.146 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:LFEJ+2e+PTkGZsFMQEqIII+OQmY= 2024-02-01T10:27:47.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:47 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:LFEJ+2e+PTkGZsFMQEqIII+OQmY=', 'Content-Length': '0'} 2024-02-01T10:27:47.149 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000042bdfdd283d12cf-0065bb7223-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:47 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:47.192 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:47.192 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:47.193 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:47.193 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:47.193 DEBUG:boto:Method: GET 2024-02-01T10:27:47.193 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:47.193 DEBUG:boto:Data: 2024-02-01T10:27:47.193 DEBUG:boto:Headers: {} 2024-02-01T10:27:47.193 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:47.193 DEBUG:boto:Port: 8001 2024-02-01T10:27:47.193 DEBUG:boto:Params: {} 2024-02-01T10:27:47.193 DEBUG:boto:Token: None 2024-02-01T10:27:47.194 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:47 GMT /vclfcn-19/ 2024-02-01T10:27:47.194 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:KmYxnpeez+AQ0NEHLtbcT0K5zaw= 2024-02-01T10:27:47.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:47 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:KmYxnpeez+AQ0NEHLtbcT0K5zaw=', 'Content-Length': '0'} 2024-02-01T10:27:47.199 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c305261681fdde88-0065bb7223-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:47 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:47.240 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:47.241 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:47.241 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:47.241 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:47.241 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:47.241 DEBUG:boto:Method: GET 2024-02-01T10:27:47.241 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:47.241 DEBUG:boto:Data: 2024-02-01T10:27:47.241 DEBUG:boto:Headers: {} 2024-02-01T10:27:47.241 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:47.241 DEBUG:boto:Port: 8000 2024-02-01T10:27:47.242 DEBUG:boto:Params: {} 2024-02-01T10:27:47.242 DEBUG:boto:Token: None 2024-02-01T10:27:47.242 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:47 GMT /vclfcn-19/ 2024-02-01T10:27:47.242 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:KmYxnpeez+AQ0NEHLtbcT0K5zaw= 2024-02-01T10:27:47.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:47 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:KmYxnpeez+AQ0NEHLtbcT0K5zaw=', 'Content-Length': '0'} 2024-02-01T10:27:47.248 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000005d19dd8725db1b3-0065bb7223-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:47 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:47.248 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:47.248 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:47.248 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:47.249 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:47.249 DEBUG:boto:path=/vclfcn-19/obj?versionId=cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI 2024-02-01T10:27:47.249 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI 2024-02-01T10:27:47.249 DEBUG:boto:Method: GET 2024-02-01T10:27:47.249 DEBUG:boto:Path: /vclfcn-19/obj?versionId=cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI 2024-02-01T10:27:47.249 DEBUG:boto:Data: 2024-02-01T10:27:47.249 DEBUG:boto:Headers: {} 2024-02-01T10:27:47.249 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:47.249 DEBUG:boto:Port: 8001 2024-02-01T10:27:47.249 DEBUG:boto:Params: {} 2024-02-01T10:27:47.250 DEBUG:boto:Token: None 2024-02-01T10:27:47.250 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:47 GMT /vclfcn-19/obj?versionId=cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI 2024-02-01T10:27:47.250 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:j4n8ooNReKQR8/tGqMdTqOlg4EQ= 2024-02-01T10:27:47.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:47 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:j4n8ooNReKQR8/tGqMdTqOlg4EQ=', 'Content-Length': '0'} 2024-02-01T10:27:47.253 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000318860db1afcc668-0065bb7223-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:47 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:47.253 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:47.253 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:47.253 DEBUG:boto:path=/vclfcn-19/obj?versionId=cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI 2024-02-01T10:27:47.253 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI 2024-02-01T10:27:47.253 DEBUG:boto:Method: GET 2024-02-01T10:27:47.254 DEBUG:boto:Path: /vclfcn-19/obj?versionId=cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI 2024-02-01T10:27:47.254 DEBUG:boto:Data: 2024-02-01T10:27:47.254 DEBUG:boto:Headers: {} 2024-02-01T10:27:47.254 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:47.254 DEBUG:boto:Port: 8000 2024-02-01T10:27:47.254 DEBUG:boto:Params: {} 2024-02-01T10:27:47.254 DEBUG:boto:Token: None 2024-02-01T10:27:47.254 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:47 GMT /vclfcn-19/obj?versionId=cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI 2024-02-01T10:27:47.254 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:j4n8ooNReKQR8/tGqMdTqOlg4EQ= 2024-02-01T10:27:47.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:47 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:j4n8ooNReKQR8/tGqMdTqOlg4EQ=', 'Content-Length': '0'} 2024-02-01T10:27:47.257 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cd4c167cc783988e-0065bb7223-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:47 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:47.257 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:47.257 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:47.258 DEBUG:boto:path=/vclfcn-19/obj?versionId=cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI 2024-02-01T10:27:47.258 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI 2024-02-01T10:27:47.258 DEBUG:boto:Method: HEAD 2024-02-01T10:27:47.258 DEBUG:boto:Path: /vclfcn-19/obj?versionId=cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI 2024-02-01T10:27:47.258 DEBUG:boto:Data: 2024-02-01T10:27:47.258 DEBUG:boto:Headers: {} 2024-02-01T10:27:47.258 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:47.258 DEBUG:boto:Port: 8001 2024-02-01T10:27:47.258 DEBUG:boto:Params: {} 2024-02-01T10:27:47.258 DEBUG:boto:Token: None 2024-02-01T10:27:47.259 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:47 GMT /vclfcn-19/obj?versionId=cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI 2024-02-01T10:27:47.259 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WXT5kwVJCnF8JESUMVM5ns6S0Ro= 2024-02-01T10:27:47.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:47 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WXT5kwVJCnF8JESUMVM5ns6S0Ro=', 'Content-Length': '0'} 2024-02-01T10:27:47.262 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fc11a5b064e70b36-0065bb7223-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:47 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:47.262 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:47.262 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:47.262 DEBUG:boto:path=/vclfcn-19/obj?versionId=cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI 2024-02-01T10:27:47.262 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI 2024-02-01T10:27:47.262 DEBUG:boto:Method: HEAD 2024-02-01T10:27:47.262 DEBUG:boto:Path: /vclfcn-19/obj?versionId=cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI 2024-02-01T10:27:47.263 DEBUG:boto:Data: 2024-02-01T10:27:47.263 DEBUG:boto:Headers: {} 2024-02-01T10:27:47.263 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:47.263 DEBUG:boto:Port: 8000 2024-02-01T10:27:47.263 DEBUG:boto:Params: {} 2024-02-01T10:27:47.263 DEBUG:boto:Token: None 2024-02-01T10:27:47.263 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:47 GMT /vclfcn-19/obj?versionId=cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI 2024-02-01T10:27:47.263 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WXT5kwVJCnF8JESUMVM5ns6S0Ro= 2024-02-01T10:27:47.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:47 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WXT5kwVJCnF8JESUMVM5ns6S0Ro=', 'Content-Length': '0'} 2024-02-01T10:27:47.266 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003ce3a4e4972f9af3-0065bb7223-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:47 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:47.266 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:47.266 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:47.266 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:47.267 DEBUG:boto:path=/vclfcn-19/obj?versionId=cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI 2024-02-01T10:27:47.267 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI 2024-02-01T10:27:47.267 DEBUG:boto:Method: GET 2024-02-01T10:27:47.267 DEBUG:boto:Path: /vclfcn-19/obj?versionId=cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI 2024-02-01T10:27:47.267 DEBUG:boto:Data: 2024-02-01T10:27:47.267 DEBUG:boto:Headers: {} 2024-02-01T10:27:47.267 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:47.267 DEBUG:boto:Port: 8001 2024-02-01T10:27:47.267 DEBUG:boto:Params: {} 2024-02-01T10:27:47.267 DEBUG:boto:Token: None 2024-02-01T10:27:47.268 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:47 GMT /vclfcn-19/obj?versionId=cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI 2024-02-01T10:27:47.268 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:j4n8ooNReKQR8/tGqMdTqOlg4EQ= 2024-02-01T10:27:47.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:47 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:j4n8ooNReKQR8/tGqMdTqOlg4EQ=', 'Content-Length': '0'} 2024-02-01T10:27:47.271 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001f857e586c3e3110-0065bb7223-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:47 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:47.312 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:47.312 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:47.312 DEBUG:boto:path=/vclfcn-19/obj?versionId=cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI 2024-02-01T10:27:47.313 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI 2024-02-01T10:27:47.313 DEBUG:boto:Method: GET 2024-02-01T10:27:47.313 DEBUG:boto:Path: /vclfcn-19/obj?versionId=cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI 2024-02-01T10:27:47.313 DEBUG:boto:Data: 2024-02-01T10:27:47.313 DEBUG:boto:Headers: {} 2024-02-01T10:27:47.313 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:47.313 DEBUG:boto:Port: 8000 2024-02-01T10:27:47.313 DEBUG:boto:Params: {} 2024-02-01T10:27:47.313 DEBUG:boto:Token: None 2024-02-01T10:27:47.314 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:47 GMT /vclfcn-19/obj?versionId=cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI 2024-02-01T10:27:47.314 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:j4n8ooNReKQR8/tGqMdTqOlg4EQ= 2024-02-01T10:27:47.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:47 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:j4n8ooNReKQR8/tGqMdTqOlg4EQ=', 'Content-Length': '0'} 2024-02-01T10:27:47.317 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b79e194b7aa7bbdb-0065bb7223-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:47 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:47.317 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:47.317 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:47.317 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:47.318 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:47.318 DEBUG:boto:Method: GET 2024-02-01T10:27:47.318 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:47.318 DEBUG:boto:Data: 2024-02-01T10:27:47.318 DEBUG:boto:Headers: {} 2024-02-01T10:27:47.318 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:47.318 DEBUG:boto:Port: 8001 2024-02-01T10:27:47.318 DEBUG:boto:Params: {} 2024-02-01T10:27:47.318 DEBUG:boto:Token: None 2024-02-01T10:27:47.318 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:47 GMT /vclfcn-19/ 2024-02-01T10:27:47.318 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:KmYxnpeez+AQ0NEHLtbcT0K5zaw= 2024-02-01T10:27:47.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:47 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:KmYxnpeez+AQ0NEHLtbcT0K5zaw=', 'Content-Length': '0'} 2024-02-01T10:27:47.323 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ad3d001e4aceccf-0065bb7223-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:47 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:47.324 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:47.324 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:47.324 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:47.324 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:47.324 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:47.324 DEBUG:boto:Method: GET 2024-02-01T10:27:47.325 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:47.325 DEBUG:boto:Data: 2024-02-01T10:27:47.325 DEBUG:boto:Headers: {} 2024-02-01T10:27:47.325 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:47.325 DEBUG:boto:Port: 8000 2024-02-01T10:27:47.325 DEBUG:boto:Params: {} 2024-02-01T10:27:47.325 DEBUG:boto:Token: None 2024-02-01T10:27:47.325 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:47 GMT /vclfcn-19/ 2024-02-01T10:27:47.325 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:KmYxnpeez+AQ0NEHLtbcT0K5zaw= 2024-02-01T10:27:47.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:47 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:KmYxnpeez+AQ0NEHLtbcT0K5zaw=', 'Content-Length': '0'} 2024-02-01T10:27:47.330 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037d975f6ec3a3afb-0065bb7223-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:47 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:47.331 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:47.331 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:47.331 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:47.331 DEBUG:boto:Method: HEAD 2024-02-01T10:27:47.331 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:47.331 DEBUG:boto:Data: 2024-02-01T10:27:47.331 DEBUG:boto:Headers: {} 2024-02-01T10:27:47.331 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:47.332 DEBUG:boto:Port: 8001 2024-02-01T10:27:47.332 DEBUG:boto:Params: {} 2024-02-01T10:27:47.332 DEBUG:boto:Token: None 2024-02-01T10:27:47.332 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:47 GMT /vclfcn-19/obj 2024-02-01T10:27:47.332 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:im9/JRJ4oRvE3tKXp3rZR6D9bxY= 2024-02-01T10:27:47.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:47 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:im9/JRJ4oRvE3tKXp3rZR6D9bxY=', 'Content-Length': '0'} 2024-02-01T10:27:47.337 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a5aa0629644f79f7-0065bb7223-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:47 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:47.338 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:47.338 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:47.338 DEBUG:boto:Method: HEAD 2024-02-01T10:27:47.338 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:47.338 DEBUG:boto:Data: 2024-02-01T10:27:47.338 DEBUG:boto:Headers: {} 2024-02-01T10:27:47.338 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:47.338 DEBUG:boto:Port: 8000 2024-02-01T10:27:47.339 DEBUG:boto:Params: {} 2024-02-01T10:27:47.339 DEBUG:boto:Token: None 2024-02-01T10:27:47.339 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:47 GMT /vclfcn-19/obj 2024-02-01T10:27:47.339 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:im9/JRJ4oRvE3tKXp3rZR6D9bxY= 2024-02-01T10:27:47.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:47 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:im9/JRJ4oRvE3tKXp3rZR6D9bxY=', 'Content-Length': '0'} 2024-02-01T10:27:47.345 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f2e1ed2fd1b1eb25-0065bb7223-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:47 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:47.345 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:47.345 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:47.345 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:47.345 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:47.345 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:47.346 DEBUG:boto:Method: GET 2024-02-01T10:27:47.346 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:47.346 DEBUG:boto:Data: 2024-02-01T10:27:47.346 DEBUG:boto:Headers: {} 2024-02-01T10:27:47.346 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:47.346 DEBUG:boto:Port: 8001 2024-02-01T10:27:47.346 DEBUG:boto:Params: {} 2024-02-01T10:27:47.346 DEBUG:boto:Token: None 2024-02-01T10:27:47.346 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:47 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:47.346 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:LFEJ+2e+PTkGZsFMQEqIII+OQmY= 2024-02-01T10:27:47.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:47 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:LFEJ+2e+PTkGZsFMQEqIII+OQmY=', 'Content-Length': '0'} 2024-02-01T10:27:47.349 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c576c42bdaba521b-0065bb7223-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:47 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:47.392 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:47.392 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:47.392 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:47.393 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:47.393 DEBUG:boto:Method: GET 2024-02-01T10:27:47.393 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:47.393 DEBUG:boto:Data: 2024-02-01T10:27:47.393 DEBUG:boto:Headers: {} 2024-02-01T10:27:47.393 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:47.393 DEBUG:boto:Port: 8000 2024-02-01T10:27:47.393 DEBUG:boto:Params: {} 2024-02-01T10:27:47.394 DEBUG:boto:Token: None 2024-02-01T10:27:47.394 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:47 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:47.394 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:LFEJ+2e+PTkGZsFMQEqIII+OQmY= 2024-02-01T10:27:47.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:47 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:LFEJ+2e+PTkGZsFMQEqIII+OQmY=', 'Content-Length': '0'} 2024-02-01T10:27:47.397 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003e2ebb66e368a3da-0065bb7223-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:47 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:47.440 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:47.440 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:47.440 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:47.441 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:47.441 DEBUG:boto:Method: GET 2024-02-01T10:27:47.441 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:47.441 DEBUG:boto:Data: 2024-02-01T10:27:47.441 DEBUG:boto:Headers: {} 2024-02-01T10:27:47.441 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:47.441 DEBUG:boto:Port: 8001 2024-02-01T10:27:47.441 DEBUG:boto:Params: {} 2024-02-01T10:27:47.441 DEBUG:boto:Token: None 2024-02-01T10:27:47.442 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:47 GMT /vclfcn-19/ 2024-02-01T10:27:47.442 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:KmYxnpeez+AQ0NEHLtbcT0K5zaw= 2024-02-01T10:27:47.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:47 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:KmYxnpeez+AQ0NEHLtbcT0K5zaw=', 'Content-Length': '0'} 2024-02-01T10:27:47.447 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008565e9c436173307-0065bb7223-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:47 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:47.448 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:47.448 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:47.448 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:47.448 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:47.448 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:47.448 DEBUG:boto:Method: GET 2024-02-01T10:27:47.449 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:47.449 DEBUG:boto:Data: 2024-02-01T10:27:47.449 DEBUG:boto:Headers: {} 2024-02-01T10:27:47.449 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:47.449 DEBUG:boto:Port: 8000 2024-02-01T10:27:47.449 DEBUG:boto:Params: {} 2024-02-01T10:27:47.449 DEBUG:boto:Token: None 2024-02-01T10:27:47.449 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:47 GMT /vclfcn-19/ 2024-02-01T10:27:47.449 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:KmYxnpeez+AQ0NEHLtbcT0K5zaw= 2024-02-01T10:27:47.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:47 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:KmYxnpeez+AQ0NEHLtbcT0K5zaw=', 'Content-Length': '0'} 2024-02-01T10:27:47.455 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d3c9a8316966490-0065bb7223-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:47 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:47.455 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:47.455 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:47.456 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:47.456 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:47.456 DEBUG:boto:path=/vclfcn-19/obj?versionId=rasya8taOZsGTj9rDi7keghf.VnPFlF 2024-02-01T10:27:47.456 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=rasya8taOZsGTj9rDi7keghf.VnPFlF 2024-02-01T10:27:47.456 DEBUG:boto:Method: GET 2024-02-01T10:27:47.456 DEBUG:boto:Path: /vclfcn-19/obj?versionId=rasya8taOZsGTj9rDi7keghf.VnPFlF 2024-02-01T10:27:47.456 DEBUG:boto:Data: 2024-02-01T10:27:47.456 DEBUG:boto:Headers: {} 2024-02-01T10:27:47.456 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:47.457 DEBUG:boto:Port: 8001 2024-02-01T10:27:47.457 DEBUG:boto:Params: {} 2024-02-01T10:27:47.457 DEBUG:boto:Token: None 2024-02-01T10:27:47.457 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:47 GMT /vclfcn-19/obj?versionId=rasya8taOZsGTj9rDi7keghf.VnPFlF 2024-02-01T10:27:47.457 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:327I2m23xp75llxaPYCD/R2g3+s= 2024-02-01T10:27:47.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:47 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:327I2m23xp75llxaPYCD/R2g3+s=', 'Content-Length': '0'} 2024-02-01T10:27:47.460 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'rasya8taOZsGTj9rDi7keghf.VnPFlF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c77eb47c80ff3e9b-0065bb7223-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:47 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:47.460 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:47.460 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:47.461 DEBUG:boto:path=/vclfcn-19/obj?versionId=rasya8taOZsGTj9rDi7keghf.VnPFlF 2024-02-01T10:27:47.461 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=rasya8taOZsGTj9rDi7keghf.VnPFlF 2024-02-01T10:27:47.461 DEBUG:boto:Method: GET 2024-02-01T10:27:47.461 DEBUG:boto:Path: /vclfcn-19/obj?versionId=rasya8taOZsGTj9rDi7keghf.VnPFlF 2024-02-01T10:27:47.461 DEBUG:boto:Data: 2024-02-01T10:27:47.461 DEBUG:boto:Headers: {} 2024-02-01T10:27:47.461 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:47.461 DEBUG:boto:Port: 8000 2024-02-01T10:27:47.461 DEBUG:boto:Params: {} 2024-02-01T10:27:47.461 DEBUG:boto:Token: None 2024-02-01T10:27:47.462 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:47 GMT /vclfcn-19/obj?versionId=rasya8taOZsGTj9rDi7keghf.VnPFlF 2024-02-01T10:27:47.462 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:327I2m23xp75llxaPYCD/R2g3+s= 2024-02-01T10:27:47.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:47 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:327I2m23xp75llxaPYCD/R2g3+s=', 'Content-Length': '0'} 2024-02-01T10:27:47.465 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'rasya8taOZsGTj9rDi7keghf.VnPFlF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000479cf7ef06404823-0065bb7223-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:47 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:47.508 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:47.508 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:47.508 DEBUG:boto:path=/vclfcn-19/obj?versionId=rasya8taOZsGTj9rDi7keghf.VnPFlF 2024-02-01T10:27:47.509 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=rasya8taOZsGTj9rDi7keghf.VnPFlF 2024-02-01T10:27:47.509 DEBUG:boto:Method: HEAD 2024-02-01T10:27:47.509 DEBUG:boto:Path: /vclfcn-19/obj?versionId=rasya8taOZsGTj9rDi7keghf.VnPFlF 2024-02-01T10:27:47.509 DEBUG:boto:Data: 2024-02-01T10:27:47.509 DEBUG:boto:Headers: {} 2024-02-01T10:27:47.509 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:47.509 DEBUG:boto:Port: 8001 2024-02-01T10:27:47.509 DEBUG:boto:Params: {} 2024-02-01T10:27:47.510 DEBUG:boto:Token: None 2024-02-01T10:27:47.510 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:47 GMT /vclfcn-19/obj?versionId=rasya8taOZsGTj9rDi7keghf.VnPFlF 2024-02-01T10:27:47.510 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZfkCsuE7fL0C5hXrtff09i/fMZk= 2024-02-01T10:27:47.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:47 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZfkCsuE7fL0C5hXrtff09i/fMZk=', 'Content-Length': '0'} 2024-02-01T10:27:47.513 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'rasya8taOZsGTj9rDi7keghf.VnPFlF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000756a11c8b7c4bfc8-0065bb7223-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:47 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:47.513 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:47.513 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:47.513 DEBUG:boto:path=/vclfcn-19/obj?versionId=rasya8taOZsGTj9rDi7keghf.VnPFlF 2024-02-01T10:27:47.514 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=rasya8taOZsGTj9rDi7keghf.VnPFlF 2024-02-01T10:27:47.514 DEBUG:boto:Method: HEAD 2024-02-01T10:27:47.514 DEBUG:boto:Path: /vclfcn-19/obj?versionId=rasya8taOZsGTj9rDi7keghf.VnPFlF 2024-02-01T10:27:47.515 DEBUG:boto:Data: 2024-02-01T10:27:47.515 DEBUG:boto:Headers: {} 2024-02-01T10:27:47.515 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:47.516 DEBUG:boto:Port: 8000 2024-02-01T10:27:47.516 DEBUG:boto:Params: {} 2024-02-01T10:27:47.516 DEBUG:boto:Token: None 2024-02-01T10:27:47.516 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:47 GMT /vclfcn-19/obj?versionId=rasya8taOZsGTj9rDi7keghf.VnPFlF 2024-02-01T10:27:47.516 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZfkCsuE7fL0C5hXrtff09i/fMZk= 2024-02-01T10:27:47.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:47 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZfkCsuE7fL0C5hXrtff09i/fMZk=', 'Content-Length': '0'} 2024-02-01T10:27:47.519 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'rasya8taOZsGTj9rDi7keghf.VnPFlF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000082846f457f3b0c02-0065bb7223-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:47 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:47.520 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:47.520 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:47.520 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:47.520 DEBUG:boto:path=/vclfcn-19/obj?versionId=rasya8taOZsGTj9rDi7keghf.VnPFlF 2024-02-01T10:27:47.520 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=rasya8taOZsGTj9rDi7keghf.VnPFlF 2024-02-01T10:27:47.520 DEBUG:boto:Method: GET 2024-02-01T10:27:47.520 DEBUG:boto:Path: /vclfcn-19/obj?versionId=rasya8taOZsGTj9rDi7keghf.VnPFlF 2024-02-01T10:27:47.520 DEBUG:boto:Data: 2024-02-01T10:27:47.520 DEBUG:boto:Headers: {} 2024-02-01T10:27:47.520 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:47.521 DEBUG:boto:Port: 8001 2024-02-01T10:27:47.521 DEBUG:boto:Params: {} 2024-02-01T10:27:47.521 DEBUG:boto:Token: None 2024-02-01T10:27:47.521 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:47 GMT /vclfcn-19/obj?versionId=rasya8taOZsGTj9rDi7keghf.VnPFlF 2024-02-01T10:27:47.521 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:327I2m23xp75llxaPYCD/R2g3+s= 2024-02-01T10:27:47.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:47 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:327I2m23xp75llxaPYCD/R2g3+s=', 'Content-Length': '0'} 2024-02-01T10:27:47.524 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'rasya8taOZsGTj9rDi7keghf.VnPFlF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009bc738857985126b-0065bb7223-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:47 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:47.524 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:47.524 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:47.524 DEBUG:boto:path=/vclfcn-19/obj?versionId=rasya8taOZsGTj9rDi7keghf.VnPFlF 2024-02-01T10:27:47.524 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=rasya8taOZsGTj9rDi7keghf.VnPFlF 2024-02-01T10:27:47.524 DEBUG:boto:Method: GET 2024-02-01T10:27:47.524 DEBUG:boto:Path: /vclfcn-19/obj?versionId=rasya8taOZsGTj9rDi7keghf.VnPFlF 2024-02-01T10:27:47.525 DEBUG:boto:Data: 2024-02-01T10:27:47.525 DEBUG:boto:Headers: {} 2024-02-01T10:27:47.525 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:47.525 DEBUG:boto:Port: 8000 2024-02-01T10:27:47.525 DEBUG:boto:Params: {} 2024-02-01T10:27:47.525 DEBUG:boto:Token: None 2024-02-01T10:27:47.525 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:47 GMT /vclfcn-19/obj?versionId=rasya8taOZsGTj9rDi7keghf.VnPFlF 2024-02-01T10:27:47.525 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:327I2m23xp75llxaPYCD/R2g3+s= 2024-02-01T10:27:47.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:47 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:327I2m23xp75llxaPYCD/R2g3+s=', 'Content-Length': '0'} 2024-02-01T10:27:47.528 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'rasya8taOZsGTj9rDi7keghf.VnPFlF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006e452a18c48a6dee-0065bb7223-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:47 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:47.576 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:47.576 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:47.576 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:47.577 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:47.577 DEBUG:boto:Method: GET 2024-02-01T10:27:47.577 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:47.577 DEBUG:boto:Data: 2024-02-01T10:27:47.577 DEBUG:boto:Headers: {} 2024-02-01T10:27:47.577 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:47.580 DEBUG:boto:Port: 8001 2024-02-01T10:27:47.580 DEBUG:boto:Params: {} 2024-02-01T10:27:47.580 DEBUG:boto:Token: None 2024-02-01T10:27:47.580 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:47 GMT /vclfcn-19/ 2024-02-01T10:27:47.580 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:KmYxnpeez+AQ0NEHLtbcT0K5zaw= 2024-02-01T10:27:47.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:47 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:KmYxnpeez+AQ0NEHLtbcT0K5zaw=', 'Content-Length': '0'} 2024-02-01T10:27:47.586 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000611af529dd40c657-0065bb7223-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:47 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:47.586 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:47.587 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:47.587 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:47.587 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:47.587 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:47.587 DEBUG:boto:Method: GET 2024-02-01T10:27:47.587 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:47.587 DEBUG:boto:Data: 2024-02-01T10:27:47.587 DEBUG:boto:Headers: {} 2024-02-01T10:27:47.587 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:47.587 DEBUG:boto:Port: 8000 2024-02-01T10:27:47.588 DEBUG:boto:Params: {} 2024-02-01T10:27:47.588 DEBUG:boto:Token: None 2024-02-01T10:27:47.588 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:47 GMT /vclfcn-19/ 2024-02-01T10:27:47.588 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:KmYxnpeez+AQ0NEHLtbcT0K5zaw= 2024-02-01T10:27:47.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:47 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:KmYxnpeez+AQ0NEHLtbcT0K5zaw=', 'Content-Length': '0'} 2024-02-01T10:27:47.593 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a0173e542bb53c2-0065bb7223-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:47 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:47.594 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:47.594 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:47.594 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:47.594 DEBUG:boto:Method: HEAD 2024-02-01T10:27:47.594 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:47.595 DEBUG:boto:Data: 2024-02-01T10:27:47.595 DEBUG:boto:Headers: {} 2024-02-01T10:27:47.595 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:47.595 DEBUG:boto:Port: 8001 2024-02-01T10:27:47.595 DEBUG:boto:Params: {} 2024-02-01T10:27:47.595 DEBUG:boto:Token: None 2024-02-01T10:27:47.595 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:47 GMT /vclfcn-19/obj 2024-02-01T10:27:47.595 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:im9/JRJ4oRvE3tKXp3rZR6D9bxY= 2024-02-01T10:27:47.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:47 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:im9/JRJ4oRvE3tKXp3rZR6D9bxY=', 'Content-Length': '0'} 2024-02-01T10:27:47.601 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008c4878a3a2b0e532-0065bb7223-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:47 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:47.601 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:47.601 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:47.602 DEBUG:boto:Method: HEAD 2024-02-01T10:27:47.602 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:47.602 DEBUG:boto:Data: 2024-02-01T10:27:47.602 DEBUG:boto:Headers: {} 2024-02-01T10:27:47.602 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:47.602 DEBUG:boto:Port: 8000 2024-02-01T10:27:47.602 DEBUG:boto:Params: {} 2024-02-01T10:27:47.602 DEBUG:boto:Token: None 2024-02-01T10:27:47.602 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:47 GMT /vclfcn-19/obj 2024-02-01T10:27:47.602 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:im9/JRJ4oRvE3tKXp3rZR6D9bxY= 2024-02-01T10:27:47.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:47 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:im9/JRJ4oRvE3tKXp3rZR6D9bxY=', 'Content-Length': '0'} 2024-02-01T10:27:47.608 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000250a82ab95e8315d-0065bb7223-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:47 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:47.608 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:47.609 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:47.609 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:47.609 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:47.609 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:47.609 DEBUG:boto:Method: GET 2024-02-01T10:27:47.609 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:47.609 DEBUG:boto:Data: 2024-02-01T10:27:47.609 DEBUG:boto:Headers: {} 2024-02-01T10:27:47.609 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:47.609 DEBUG:boto:Port: 8001 2024-02-01T10:27:47.610 DEBUG:boto:Params: {} 2024-02-01T10:27:47.610 DEBUG:boto:Token: None 2024-02-01T10:27:47.610 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:47 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:47.610 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:LFEJ+2e+PTkGZsFMQEqIII+OQmY= 2024-02-01T10:27:47.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:47 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:LFEJ+2e+PTkGZsFMQEqIII+OQmY=', 'Content-Length': '0'} 2024-02-01T10:27:47.613 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004fd2536033779238-0065bb7223-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:47 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:47.660 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:47.660 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:47.660 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:47.661 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:47.661 DEBUG:boto:Method: GET 2024-02-01T10:27:47.661 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:47.661 DEBUG:boto:Data: 2024-02-01T10:27:47.661 DEBUG:boto:Headers: {} 2024-02-01T10:27:47.661 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:47.661 DEBUG:boto:Port: 8000 2024-02-01T10:27:47.661 DEBUG:boto:Params: {} 2024-02-01T10:27:47.661 DEBUG:boto:Token: None 2024-02-01T10:27:47.662 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:47 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:47.662 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:LFEJ+2e+PTkGZsFMQEqIII+OQmY= 2024-02-01T10:27:47.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:47 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:LFEJ+2e+PTkGZsFMQEqIII+OQmY=', 'Content-Length': '0'} 2024-02-01T10:27:47.665 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ad26b20f5a5025c7-0065bb7223-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:47 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:47.708 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:47.708 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:47.709 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:47.709 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:47.709 DEBUG:boto:Method: GET 2024-02-01T10:27:47.709 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:47.709 DEBUG:boto:Data: 2024-02-01T10:27:47.709 DEBUG:boto:Headers: {} 2024-02-01T10:27:47.709 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:47.709 DEBUG:boto:Port: 8001 2024-02-01T10:27:47.709 DEBUG:boto:Params: {} 2024-02-01T10:27:47.709 DEBUG:boto:Token: None 2024-02-01T10:27:47.710 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:47 GMT /vclfcn-19/ 2024-02-01T10:27:47.710 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:KmYxnpeez+AQ0NEHLtbcT0K5zaw= 2024-02-01T10:27:47.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:47 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:KmYxnpeez+AQ0NEHLtbcT0K5zaw=', 'Content-Length': '0'} 2024-02-01T10:27:47.716 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025a32b2fe0447421-0065bb7223-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:47 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:47.717 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:47.717 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:47.718 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:47.718 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:47.718 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:47.718 DEBUG:boto:Method: GET 2024-02-01T10:27:47.718 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:47.718 DEBUG:boto:Data: 2024-02-01T10:27:47.718 DEBUG:boto:Headers: {} 2024-02-01T10:27:47.719 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:47.719 DEBUG:boto:Port: 8000 2024-02-01T10:27:47.719 DEBUG:boto:Params: {} 2024-02-01T10:27:47.719 DEBUG:boto:Token: None 2024-02-01T10:27:47.719 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:47 GMT /vclfcn-19/ 2024-02-01T10:27:47.720 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:KmYxnpeez+AQ0NEHLtbcT0K5zaw= 2024-02-01T10:27:47.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:47 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:KmYxnpeez+AQ0NEHLtbcT0K5zaw=', 'Content-Length': '0'} 2024-02-01T10:27:47.725 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005cda75142110bb44-0065bb7223-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:47 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:47.725 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:47.726 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:47.726 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:47.726 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:47.726 DEBUG:boto:path=/vclfcn-19/obj?versionId=y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE 2024-02-01T10:27:47.726 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE 2024-02-01T10:27:47.726 DEBUG:boto:Method: GET 2024-02-01T10:27:47.727 DEBUG:boto:Path: /vclfcn-19/obj?versionId=y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE 2024-02-01T10:27:47.727 DEBUG:boto:Data: 2024-02-01T10:27:47.727 DEBUG:boto:Headers: {} 2024-02-01T10:27:47.727 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:47.727 DEBUG:boto:Port: 8001 2024-02-01T10:27:47.727 DEBUG:boto:Params: {} 2024-02-01T10:27:47.727 DEBUG:boto:Token: None 2024-02-01T10:27:47.727 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:47 GMT /vclfcn-19/obj?versionId=y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE 2024-02-01T10:27:47.727 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cOGpLITCNjTAEBkvBQlJwTqS/4Y= 2024-02-01T10:27:47.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:47 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cOGpLITCNjTAEBkvBQlJwTqS/4Y=', 'Content-Length': '0'} 2024-02-01T10:27:47.730 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ae588c89b585bc44-0065bb7223-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:47 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:47.730 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:47.731 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:47.731 DEBUG:boto:path=/vclfcn-19/obj?versionId=y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE 2024-02-01T10:27:47.731 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE 2024-02-01T10:27:47.731 DEBUG:boto:Method: GET 2024-02-01T10:27:47.731 DEBUG:boto:Path: /vclfcn-19/obj?versionId=y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE 2024-02-01T10:27:47.731 DEBUG:boto:Data: 2024-02-01T10:27:47.731 DEBUG:boto:Headers: {} 2024-02-01T10:27:47.731 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:47.731 DEBUG:boto:Port: 8000 2024-02-01T10:27:47.731 DEBUG:boto:Params: {} 2024-02-01T10:27:47.732 DEBUG:boto:Token: None 2024-02-01T10:27:47.732 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:47 GMT /vclfcn-19/obj?versionId=y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE 2024-02-01T10:27:47.732 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cOGpLITCNjTAEBkvBQlJwTqS/4Y= 2024-02-01T10:27:47.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:47 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cOGpLITCNjTAEBkvBQlJwTqS/4Y=', 'Content-Length': '0'} 2024-02-01T10:27:47.735 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004a1ff33246b03514-0065bb7223-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:47 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:47.776 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:47.776 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:47.777 DEBUG:boto:path=/vclfcn-19/obj?versionId=y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE 2024-02-01T10:27:47.777 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE 2024-02-01T10:27:47.777 DEBUG:boto:Method: HEAD 2024-02-01T10:27:47.777 DEBUG:boto:Path: /vclfcn-19/obj?versionId=y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE 2024-02-01T10:27:47.777 DEBUG:boto:Data: 2024-02-01T10:27:47.777 DEBUG:boto:Headers: {} 2024-02-01T10:27:47.777 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:47.777 DEBUG:boto:Port: 8001 2024-02-01T10:27:47.777 DEBUG:boto:Params: {} 2024-02-01T10:27:47.777 DEBUG:boto:Token: None 2024-02-01T10:27:47.778 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:47 GMT /vclfcn-19/obj?versionId=y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE 2024-02-01T10:27:47.778 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:xTLbUotC4WTTgvu7pZfFnLnLAAI= 2024-02-01T10:27:47.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:47 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:xTLbUotC4WTTgvu7pZfFnLnLAAI=', 'Content-Length': '0'} 2024-02-01T10:27:47.783 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000037fd28c9bb0de51e-0065bb7223-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:47 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:47.783 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:47.784 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:47.784 DEBUG:boto:path=/vclfcn-19/obj?versionId=y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE 2024-02-01T10:27:47.784 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE 2024-02-01T10:27:47.784 DEBUG:boto:Method: HEAD 2024-02-01T10:27:47.784 DEBUG:boto:Path: /vclfcn-19/obj?versionId=y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE 2024-02-01T10:27:47.784 DEBUG:boto:Data: 2024-02-01T10:27:47.784 DEBUG:boto:Headers: {} 2024-02-01T10:27:47.784 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:47.784 DEBUG:boto:Port: 8000 2024-02-01T10:27:47.784 DEBUG:boto:Params: {} 2024-02-01T10:27:47.785 DEBUG:boto:Token: None 2024-02-01T10:27:47.785 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:47 GMT /vclfcn-19/obj?versionId=y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE 2024-02-01T10:27:47.785 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:xTLbUotC4WTTgvu7pZfFnLnLAAI= 2024-02-01T10:27:47.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:47 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:xTLbUotC4WTTgvu7pZfFnLnLAAI=', 'Content-Length': '0'} 2024-02-01T10:27:47.788 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009e92cc63f508af1a-0065bb7223-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:47 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:47.788 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:47.789 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:47.789 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:47.789 DEBUG:boto:path=/vclfcn-19/obj?versionId=y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE 2024-02-01T10:27:47.789 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE 2024-02-01T10:27:47.789 DEBUG:boto:Method: GET 2024-02-01T10:27:47.789 DEBUG:boto:Path: /vclfcn-19/obj?versionId=y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE 2024-02-01T10:27:47.789 DEBUG:boto:Data: 2024-02-01T10:27:47.790 DEBUG:boto:Headers: {} 2024-02-01T10:27:47.790 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:47.790 DEBUG:boto:Port: 8001 2024-02-01T10:27:47.790 DEBUG:boto:Params: {} 2024-02-01T10:27:47.790 DEBUG:boto:Token: None 2024-02-01T10:27:47.790 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:47 GMT /vclfcn-19/obj?versionId=y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE 2024-02-01T10:27:47.790 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cOGpLITCNjTAEBkvBQlJwTqS/4Y= 2024-02-01T10:27:47.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:47 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cOGpLITCNjTAEBkvBQlJwTqS/4Y=', 'Content-Length': '0'} 2024-02-01T10:27:47.793 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000854a6b819c88ac32-0065bb7223-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:47 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:47.836 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:47.837 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:47.837 DEBUG:boto:path=/vclfcn-19/obj?versionId=y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE 2024-02-01T10:27:47.837 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE 2024-02-01T10:27:47.837 DEBUG:boto:Method: GET 2024-02-01T10:27:47.837 DEBUG:boto:Path: /vclfcn-19/obj?versionId=y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE 2024-02-01T10:27:47.837 DEBUG:boto:Data: 2024-02-01T10:27:47.837 DEBUG:boto:Headers: {} 2024-02-01T10:27:47.837 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:47.837 DEBUG:boto:Port: 8000 2024-02-01T10:27:47.838 DEBUG:boto:Params: {} 2024-02-01T10:27:47.838 DEBUG:boto:Token: None 2024-02-01T10:27:47.838 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:47 GMT /vclfcn-19/obj?versionId=y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE 2024-02-01T10:27:47.838 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cOGpLITCNjTAEBkvBQlJwTqS/4Y= 2024-02-01T10:27:47.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:47 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cOGpLITCNjTAEBkvBQlJwTqS/4Y=', 'Content-Length': '0'} 2024-02-01T10:27:47.841 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ed5429d9abc22af1-0065bb7223-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:47 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:47.884 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:47.884 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:47.884 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:47.885 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:47.885 DEBUG:boto:Method: GET 2024-02-01T10:27:47.885 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:47.885 DEBUG:boto:Data: 2024-02-01T10:27:47.885 DEBUG:boto:Headers: {} 2024-02-01T10:27:47.885 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:47.885 DEBUG:boto:Port: 8001 2024-02-01T10:27:47.885 DEBUG:boto:Params: {} 2024-02-01T10:27:47.885 DEBUG:boto:Token: None 2024-02-01T10:27:47.886 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:47 GMT /vclfcn-19/ 2024-02-01T10:27:47.886 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:KmYxnpeez+AQ0NEHLtbcT0K5zaw= 2024-02-01T10:27:47.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:47 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:KmYxnpeez+AQ0NEHLtbcT0K5zaw=', 'Content-Length': '0'} 2024-02-01T10:27:47.891 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f5646f5654054059-0065bb7223-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:47 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:47.932 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:47.933 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:47.933 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:47.933 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:47.933 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:47.933 DEBUG:boto:Method: GET 2024-02-01T10:27:47.933 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:47.933 DEBUG:boto:Data: 2024-02-01T10:27:47.933 DEBUG:boto:Headers: {} 2024-02-01T10:27:47.933 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:47.933 DEBUG:boto:Port: 8000 2024-02-01T10:27:47.933 DEBUG:boto:Params: {} 2024-02-01T10:27:47.934 DEBUG:boto:Token: None 2024-02-01T10:27:47.934 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:47 GMT /vclfcn-19/ 2024-02-01T10:27:47.934 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:KmYxnpeez+AQ0NEHLtbcT0K5zaw= 2024-02-01T10:27:47.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:47 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:KmYxnpeez+AQ0NEHLtbcT0K5zaw=', 'Content-Length': '0'} 2024-02-01T10:27:47.939 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc15da250cf5d267-0065bb7223-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:47 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:47.940 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:47.940 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:47.940 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:47.940 DEBUG:boto:Method: HEAD 2024-02-01T10:27:47.940 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:47.940 DEBUG:boto:Data: 2024-02-01T10:27:47.940 DEBUG:boto:Headers: {} 2024-02-01T10:27:47.941 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:47.941 DEBUG:boto:Port: 8001 2024-02-01T10:27:47.941 DEBUG:boto:Params: {} 2024-02-01T10:27:47.941 DEBUG:boto:Token: None 2024-02-01T10:27:47.941 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:47 GMT /vclfcn-19/obj 2024-02-01T10:27:47.941 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:im9/JRJ4oRvE3tKXp3rZR6D9bxY= 2024-02-01T10:27:47.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:47 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:im9/JRJ4oRvE3tKXp3rZR6D9bxY=', 'Content-Length': '0'} 2024-02-01T10:27:47.947 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fe87b6ef85db46f2-0065bb7223-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:47 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:47.947 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:47.947 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:47.947 DEBUG:boto:Method: HEAD 2024-02-01T10:27:47.948 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:47.948 DEBUG:boto:Data: 2024-02-01T10:27:47.948 DEBUG:boto:Headers: {} 2024-02-01T10:27:47.948 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:47.948 DEBUG:boto:Port: 8000 2024-02-01T10:27:47.948 DEBUG:boto:Params: {} 2024-02-01T10:27:47.948 DEBUG:boto:Token: None 2024-02-01T10:27:47.948 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:47 GMT /vclfcn-19/obj 2024-02-01T10:27:47.948 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:im9/JRJ4oRvE3tKXp3rZR6D9bxY= 2024-02-01T10:27:47.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:47 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:im9/JRJ4oRvE3tKXp3rZR6D9bxY=', 'Content-Length': '0'} 2024-02-01T10:27:47.954 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000000b7f03ad519a2cb-0065bb7223-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:47 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:47.954 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:47.955 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:47.955 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:47.955 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:47.955 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:47.955 DEBUG:boto:Method: GET 2024-02-01T10:27:47.955 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:47.955 DEBUG:boto:Data: 2024-02-01T10:27:47.955 DEBUG:boto:Headers: {} 2024-02-01T10:27:47.955 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:47.955 DEBUG:boto:Port: 8001 2024-02-01T10:27:47.955 DEBUG:boto:Params: {} 2024-02-01T10:27:47.956 DEBUG:boto:Token: None 2024-02-01T10:27:47.956 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:47 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:47.956 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:LFEJ+2e+PTkGZsFMQEqIII+OQmY= 2024-02-01T10:27:47.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:47 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:LFEJ+2e+PTkGZsFMQEqIII+OQmY=', 'Content-Length': '0'} 2024-02-01T10:27:47.959 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000055282963aacb3f34-0065bb7223-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:47 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:48.000 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:48.000 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:48.000 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:48.001 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:48.001 DEBUG:boto:Method: GET 2024-02-01T10:27:48.001 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:48.001 DEBUG:boto:Data: 2024-02-01T10:27:48.001 DEBUG:boto:Headers: {} 2024-02-01T10:27:48.001 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:48.001 DEBUG:boto:Port: 8000 2024-02-01T10:27:48.001 DEBUG:boto:Params: {} 2024-02-01T10:27:48.001 DEBUG:boto:Token: None 2024-02-01T10:27:48.001 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:48 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:48.002 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:DUyOlzSKaBK1ybAf0BkHs6Lub3A= 2024-02-01T10:27:48.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:DUyOlzSKaBK1ybAf0BkHs6Lub3A=', 'Content-Length': '0'} 2024-02-01T10:27:48.005 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000030bf20e2e5145fea-0065bb7224-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:48.048 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:48.048 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:48.048 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:48.048 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:48.049 DEBUG:boto:Method: GET 2024-02-01T10:27:48.049 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:48.049 DEBUG:boto:Data: 2024-02-01T10:27:48.049 DEBUG:boto:Headers: {} 2024-02-01T10:27:48.049 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:48.049 DEBUG:boto:Port: 8001 2024-02-01T10:27:48.049 DEBUG:boto:Params: {} 2024-02-01T10:27:48.049 DEBUG:boto:Token: None 2024-02-01T10:27:48.049 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:48 GMT /vclfcn-19/ 2024-02-01T10:27:48.050 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HRD/rg3IwtslhykDLnQayY2iTZE= 2024-02-01T10:27:48.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HRD/rg3IwtslhykDLnQayY2iTZE=', 'Content-Length': '0'} 2024-02-01T10:27:48.055 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063aff27759f0a9e4-0065bb7224-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:48.096 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:48.097 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:48.097 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:48.097 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:48.097 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:48.097 DEBUG:boto:Method: GET 2024-02-01T10:27:48.097 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:48.097 DEBUG:boto:Data: 2024-02-01T10:27:48.097 DEBUG:boto:Headers: {} 2024-02-01T10:27:48.097 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:48.097 DEBUG:boto:Port: 8000 2024-02-01T10:27:48.098 DEBUG:boto:Params: {} 2024-02-01T10:27:48.098 DEBUG:boto:Token: None 2024-02-01T10:27:48.098 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:48 GMT /vclfcn-19/ 2024-02-01T10:27:48.098 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HRD/rg3IwtslhykDLnQayY2iTZE= 2024-02-01T10:27:48.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HRD/rg3IwtslhykDLnQayY2iTZE=', 'Content-Length': '0'} 2024-02-01T10:27:48.103 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000542147a15947b613-0065bb7224-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:48.103 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:48.104 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:48.104 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:48.104 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:48.104 DEBUG:boto:path=/vclfcn-19/obj?versionId=0KR6m6ymTGitKe6vabQyVENAnBL669w 2024-02-01T10:27:48.104 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=0KR6m6ymTGitKe6vabQyVENAnBL669w 2024-02-01T10:27:48.104 DEBUG:boto:Method: GET 2024-02-01T10:27:48.105 DEBUG:boto:Path: /vclfcn-19/obj?versionId=0KR6m6ymTGitKe6vabQyVENAnBL669w 2024-02-01T10:27:48.105 DEBUG:boto:Data: 2024-02-01T10:27:48.105 DEBUG:boto:Headers: {} 2024-02-01T10:27:48.105 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:48.105 DEBUG:boto:Port: 8001 2024-02-01T10:27:48.105 DEBUG:boto:Params: {} 2024-02-01T10:27:48.105 DEBUG:boto:Token: None 2024-02-01T10:27:48.105 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:48 GMT /vclfcn-19/obj?versionId=0KR6m6ymTGitKe6vabQyVENAnBL669w 2024-02-01T10:27:48.105 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mxa6V/DARpMivdDpXaPtuhkt8P0= 2024-02-01T10:27:48.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mxa6V/DARpMivdDpXaPtuhkt8P0=', 'Content-Length': '0'} 2024-02-01T10:27:48.109 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '0KR6m6ymTGitKe6vabQyVENAnBL669w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000046db608ebdd3e25-0065bb7224-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:48.109 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:48.109 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:48.109 DEBUG:boto:path=/vclfcn-19/obj?versionId=0KR6m6ymTGitKe6vabQyVENAnBL669w 2024-02-01T10:27:48.109 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=0KR6m6ymTGitKe6vabQyVENAnBL669w 2024-02-01T10:27:48.110 DEBUG:boto:Method: GET 2024-02-01T10:27:48.110 DEBUG:boto:Path: /vclfcn-19/obj?versionId=0KR6m6ymTGitKe6vabQyVENAnBL669w 2024-02-01T10:27:48.110 DEBUG:boto:Data: 2024-02-01T10:27:48.110 DEBUG:boto:Headers: {} 2024-02-01T10:27:48.110 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:48.110 DEBUG:boto:Port: 8000 2024-02-01T10:27:48.110 DEBUG:boto:Params: {} 2024-02-01T10:27:48.110 DEBUG:boto:Token: None 2024-02-01T10:27:48.111 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:48 GMT /vclfcn-19/obj?versionId=0KR6m6ymTGitKe6vabQyVENAnBL669w 2024-02-01T10:27:48.111 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mxa6V/DARpMivdDpXaPtuhkt8P0= 2024-02-01T10:27:48.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mxa6V/DARpMivdDpXaPtuhkt8P0=', 'Content-Length': '0'} 2024-02-01T10:27:48.114 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '0KR6m6ymTGitKe6vabQyVENAnBL669w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c4cc13f1d135b2a8-0065bb7224-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:48.160 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:48.160 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:48.160 DEBUG:boto:path=/vclfcn-19/obj?versionId=0KR6m6ymTGitKe6vabQyVENAnBL669w 2024-02-01T10:27:48.161 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=0KR6m6ymTGitKe6vabQyVENAnBL669w 2024-02-01T10:27:48.161 DEBUG:boto:Method: HEAD 2024-02-01T10:27:48.161 DEBUG:boto:Path: /vclfcn-19/obj?versionId=0KR6m6ymTGitKe6vabQyVENAnBL669w 2024-02-01T10:27:48.161 DEBUG:boto:Data: 2024-02-01T10:27:48.161 DEBUG:boto:Headers: {} 2024-02-01T10:27:48.161 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:48.161 DEBUG:boto:Port: 8001 2024-02-01T10:27:48.161 DEBUG:boto:Params: {} 2024-02-01T10:27:48.161 DEBUG:boto:Token: None 2024-02-01T10:27:48.162 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:48 GMT /vclfcn-19/obj?versionId=0KR6m6ymTGitKe6vabQyVENAnBL669w 2024-02-01T10:27:48.162 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QHwxEmTj/vCQqwa78wyiSWtcdHo= 2024-02-01T10:27:48.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QHwxEmTj/vCQqwa78wyiSWtcdHo=', 'Content-Length': '0'} 2024-02-01T10:27:48.165 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '0KR6m6ymTGitKe6vabQyVENAnBL669w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000066369d80fe643b60-0065bb7224-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:48.165 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:48.165 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:48.165 DEBUG:boto:path=/vclfcn-19/obj?versionId=0KR6m6ymTGitKe6vabQyVENAnBL669w 2024-02-01T10:27:48.165 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=0KR6m6ymTGitKe6vabQyVENAnBL669w 2024-02-01T10:27:48.165 DEBUG:boto:Method: HEAD 2024-02-01T10:27:48.166 DEBUG:boto:Path: /vclfcn-19/obj?versionId=0KR6m6ymTGitKe6vabQyVENAnBL669w 2024-02-01T10:27:48.166 DEBUG:boto:Data: 2024-02-01T10:27:48.166 DEBUG:boto:Headers: {} 2024-02-01T10:27:48.166 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:48.166 DEBUG:boto:Port: 8000 2024-02-01T10:27:48.166 DEBUG:boto:Params: {} 2024-02-01T10:27:48.166 DEBUG:boto:Token: None 2024-02-01T10:27:48.166 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:48 GMT /vclfcn-19/obj?versionId=0KR6m6ymTGitKe6vabQyVENAnBL669w 2024-02-01T10:27:48.166 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QHwxEmTj/vCQqwa78wyiSWtcdHo= 2024-02-01T10:27:48.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QHwxEmTj/vCQqwa78wyiSWtcdHo=', 'Content-Length': '0'} 2024-02-01T10:27:48.170 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '0KR6m6ymTGitKe6vabQyVENAnBL669w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009fcd758dd48e1481-0065bb7224-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:48.171 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:48.171 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:48.171 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:48.171 DEBUG:boto:path=/vclfcn-19/obj?versionId=0KR6m6ymTGitKe6vabQyVENAnBL669w 2024-02-01T10:27:48.171 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=0KR6m6ymTGitKe6vabQyVENAnBL669w 2024-02-01T10:27:48.171 DEBUG:boto:Method: GET 2024-02-01T10:27:48.171 DEBUG:boto:Path: /vclfcn-19/obj?versionId=0KR6m6ymTGitKe6vabQyVENAnBL669w 2024-02-01T10:27:48.171 DEBUG:boto:Data: 2024-02-01T10:27:48.171 DEBUG:boto:Headers: {} 2024-02-01T10:27:48.172 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:48.172 DEBUG:boto:Port: 8001 2024-02-01T10:27:48.172 DEBUG:boto:Params: {} 2024-02-01T10:27:48.172 DEBUG:boto:Token: None 2024-02-01T10:27:48.172 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:48 GMT /vclfcn-19/obj?versionId=0KR6m6ymTGitKe6vabQyVENAnBL669w 2024-02-01T10:27:48.172 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mxa6V/DARpMivdDpXaPtuhkt8P0= 2024-02-01T10:27:48.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mxa6V/DARpMivdDpXaPtuhkt8P0=', 'Content-Length': '0'} 2024-02-01T10:27:48.175 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '0KR6m6ymTGitKe6vabQyVENAnBL669w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000980606d7074338c-0065bb7224-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:48.220 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:48.220 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:48.220 DEBUG:boto:path=/vclfcn-19/obj?versionId=0KR6m6ymTGitKe6vabQyVENAnBL669w 2024-02-01T10:27:48.221 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=0KR6m6ymTGitKe6vabQyVENAnBL669w 2024-02-01T10:27:48.221 DEBUG:boto:Method: GET 2024-02-01T10:27:48.221 DEBUG:boto:Path: /vclfcn-19/obj?versionId=0KR6m6ymTGitKe6vabQyVENAnBL669w 2024-02-01T10:27:48.221 DEBUG:boto:Data: 2024-02-01T10:27:48.221 DEBUG:boto:Headers: {} 2024-02-01T10:27:48.221 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:48.221 DEBUG:boto:Port: 8000 2024-02-01T10:27:48.221 DEBUG:boto:Params: {} 2024-02-01T10:27:48.221 DEBUG:boto:Token: None 2024-02-01T10:27:48.221 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:48 GMT /vclfcn-19/obj?versionId=0KR6m6ymTGitKe6vabQyVENAnBL669w 2024-02-01T10:27:48.222 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mxa6V/DARpMivdDpXaPtuhkt8P0= 2024-02-01T10:27:48.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mxa6V/DARpMivdDpXaPtuhkt8P0=', 'Content-Length': '0'} 2024-02-01T10:27:48.225 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '0KR6m6ymTGitKe6vabQyVENAnBL669w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004c9dbe015dd5061e-0065bb7224-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:48.272 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:48.272 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:48.272 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:48.272 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:48.273 DEBUG:boto:Method: GET 2024-02-01T10:27:48.273 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:48.273 DEBUG:boto:Data: 2024-02-01T10:27:48.273 DEBUG:boto:Headers: {} 2024-02-01T10:27:48.273 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:48.273 DEBUG:boto:Port: 8001 2024-02-01T10:27:48.273 DEBUG:boto:Params: {} 2024-02-01T10:27:48.273 DEBUG:boto:Token: None 2024-02-01T10:27:48.273 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:48 GMT /vclfcn-19/ 2024-02-01T10:27:48.273 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HRD/rg3IwtslhykDLnQayY2iTZE= 2024-02-01T10:27:48.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HRD/rg3IwtslhykDLnQayY2iTZE=', 'Content-Length': '0'} 2024-02-01T10:27:48.279 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cee1e28f643e50ad-0065bb7224-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:48.320 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:48.321 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:48.321 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:48.321 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:48.321 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:48.321 DEBUG:boto:Method: GET 2024-02-01T10:27:48.321 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:48.321 DEBUG:boto:Data: 2024-02-01T10:27:48.321 DEBUG:boto:Headers: {} 2024-02-01T10:27:48.321 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:48.321 DEBUG:boto:Port: 8000 2024-02-01T10:27:48.321 DEBUG:boto:Params: {} 2024-02-01T10:27:48.322 DEBUG:boto:Token: None 2024-02-01T10:27:48.322 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:48 GMT /vclfcn-19/ 2024-02-01T10:27:48.322 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HRD/rg3IwtslhykDLnQayY2iTZE= 2024-02-01T10:27:48.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HRD/rg3IwtslhykDLnQayY2iTZE=', 'Content-Length': '0'} 2024-02-01T10:27:48.327 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ebb6a7631a77cf88-0065bb7224-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:48.327 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:48.327 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:48.328 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:48.328 DEBUG:boto:Method: HEAD 2024-02-01T10:27:48.328 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:48.328 DEBUG:boto:Data: 2024-02-01T10:27:48.328 DEBUG:boto:Headers: {} 2024-02-01T10:27:48.328 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:48.328 DEBUG:boto:Port: 8001 2024-02-01T10:27:48.328 DEBUG:boto:Params: {} 2024-02-01T10:27:48.328 DEBUG:boto:Token: None 2024-02-01T10:27:48.329 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:48 GMT /vclfcn-19/obj 2024-02-01T10:27:48.329 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+s2NWXYhQkqGUB2XyAxgfiM1mk8= 2024-02-01T10:27:48.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+s2NWXYhQkqGUB2XyAxgfiM1mk8=', 'Content-Length': '0'} 2024-02-01T10:27:48.334 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f2d998338bcc249f-0065bb7224-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:48.334 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:48.335 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:48.335 DEBUG:boto:Method: HEAD 2024-02-01T10:27:48.335 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:48.335 DEBUG:boto:Data: 2024-02-01T10:27:48.335 DEBUG:boto:Headers: {} 2024-02-01T10:27:48.335 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:48.335 DEBUG:boto:Port: 8000 2024-02-01T10:27:48.335 DEBUG:boto:Params: {} 2024-02-01T10:27:48.335 DEBUG:boto:Token: None 2024-02-01T10:27:48.335 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:48 GMT /vclfcn-19/obj 2024-02-01T10:27:48.336 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+s2NWXYhQkqGUB2XyAxgfiM1mk8= 2024-02-01T10:27:48.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+s2NWXYhQkqGUB2XyAxgfiM1mk8=', 'Content-Length': '0'} 2024-02-01T10:27:48.341 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000453c84cb7b73d2f0-0065bb7224-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:48.341 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:48.341 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:48.341 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:48.342 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:48.342 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:48.342 DEBUG:boto:Method: GET 2024-02-01T10:27:48.342 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:48.342 DEBUG:boto:Data: 2024-02-01T10:27:48.342 DEBUG:boto:Headers: {} 2024-02-01T10:27:48.342 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:48.342 DEBUG:boto:Port: 8001 2024-02-01T10:27:48.342 DEBUG:boto:Params: {} 2024-02-01T10:27:48.342 DEBUG:boto:Token: None 2024-02-01T10:27:48.343 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:48 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:48.343 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:DUyOlzSKaBK1ybAf0BkHs6Lub3A= 2024-02-01T10:27:48.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:DUyOlzSKaBK1ybAf0BkHs6Lub3A=', 'Content-Length': '0'} 2024-02-01T10:27:48.346 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000003518016aa1acf0e-0065bb7224-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:48.388 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:48.388 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:48.388 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:48.388 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:48.389 DEBUG:boto:Method: GET 2024-02-01T10:27:48.389 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:48.389 DEBUG:boto:Data: 2024-02-01T10:27:48.389 DEBUG:boto:Headers: {} 2024-02-01T10:27:48.389 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:48.389 DEBUG:boto:Port: 8000 2024-02-01T10:27:48.389 DEBUG:boto:Params: {} 2024-02-01T10:27:48.389 DEBUG:boto:Token: None 2024-02-01T10:27:48.389 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:48 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:48.390 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:DUyOlzSKaBK1ybAf0BkHs6Lub3A= 2024-02-01T10:27:48.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:DUyOlzSKaBK1ybAf0BkHs6Lub3A=', 'Content-Length': '0'} 2024-02-01T10:27:48.393 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c40c93321af87c09-0065bb7224-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:48.440 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:48.440 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:48.440 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:48.440 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:48.441 DEBUG:boto:Method: GET 2024-02-01T10:27:48.441 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:48.441 DEBUG:boto:Data: 2024-02-01T10:27:48.441 DEBUG:boto:Headers: {} 2024-02-01T10:27:48.441 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:48.441 DEBUG:boto:Port: 8001 2024-02-01T10:27:48.441 DEBUG:boto:Params: {} 2024-02-01T10:27:48.441 DEBUG:boto:Token: None 2024-02-01T10:27:48.441 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:48 GMT /vclfcn-19/ 2024-02-01T10:27:48.442 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HRD/rg3IwtslhykDLnQayY2iTZE= 2024-02-01T10:27:48.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HRD/rg3IwtslhykDLnQayY2iTZE=', 'Content-Length': '0'} 2024-02-01T10:27:48.448 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008df578c3a5c3502a-0065bb7224-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:48.488 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:48.489 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:48.489 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:48.489 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:48.489 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:48.489 DEBUG:boto:Method: GET 2024-02-01T10:27:48.489 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:48.489 DEBUG:boto:Data: 2024-02-01T10:27:48.489 DEBUG:boto:Headers: {} 2024-02-01T10:27:48.489 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:48.490 DEBUG:boto:Port: 8000 2024-02-01T10:27:48.490 DEBUG:boto:Params: {} 2024-02-01T10:27:48.490 DEBUG:boto:Token: None 2024-02-01T10:27:48.490 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:48 GMT /vclfcn-19/ 2024-02-01T10:27:48.490 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HRD/rg3IwtslhykDLnQayY2iTZE= 2024-02-01T10:27:48.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HRD/rg3IwtslhykDLnQayY2iTZE=', 'Content-Length': '0'} 2024-02-01T10:27:48.495 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6ffae137bffb928-0065bb7224-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:48.536 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:48.537 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:48.537 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:48.537 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:48.537 DEBUG:boto:path=/vclfcn-19/obj?versionId=2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E 2024-02-01T10:27:48.537 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E 2024-02-01T10:27:48.537 DEBUG:boto:Method: GET 2024-02-01T10:27:48.537 DEBUG:boto:Path: /vclfcn-19/obj?versionId=2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E 2024-02-01T10:27:48.537 DEBUG:boto:Data: 2024-02-01T10:27:48.537 DEBUG:boto:Headers: {} 2024-02-01T10:27:48.537 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:48.538 DEBUG:boto:Port: 8001 2024-02-01T10:27:48.538 DEBUG:boto:Params: {} 2024-02-01T10:27:48.538 DEBUG:boto:Token: None 2024-02-01T10:27:48.538 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:48 GMT /vclfcn-19/obj?versionId=2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E 2024-02-01T10:27:48.538 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5nsJ6TkuEK8Wc0d6YTpy21Lcsf4= 2024-02-01T10:27:48.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5nsJ6TkuEK8Wc0d6YTpy21Lcsf4=', 'Content-Length': '0'} 2024-02-01T10:27:48.541 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e39ea10df84873a9-0065bb7224-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:48.542 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:48.542 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:48.542 DEBUG:boto:path=/vclfcn-19/obj?versionId=2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E 2024-02-01T10:27:48.542 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E 2024-02-01T10:27:48.542 DEBUG:boto:Method: GET 2024-02-01T10:27:48.542 DEBUG:boto:Path: /vclfcn-19/obj?versionId=2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E 2024-02-01T10:27:48.542 DEBUG:boto:Data: 2024-02-01T10:27:48.542 DEBUG:boto:Headers: {} 2024-02-01T10:27:48.543 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:48.543 DEBUG:boto:Port: 8000 2024-02-01T10:27:48.543 DEBUG:boto:Params: {} 2024-02-01T10:27:48.543 DEBUG:boto:Token: None 2024-02-01T10:27:48.543 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:48 GMT /vclfcn-19/obj?versionId=2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E 2024-02-01T10:27:48.543 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5nsJ6TkuEK8Wc0d6YTpy21Lcsf4= 2024-02-01T10:27:48.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5nsJ6TkuEK8Wc0d6YTpy21Lcsf4=', 'Content-Length': '0'} 2024-02-01T10:27:48.546 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000325e5456546d5b77-0065bb7224-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:48.546 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:48.546 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:48.547 DEBUG:boto:path=/vclfcn-19/obj?versionId=2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E 2024-02-01T10:27:48.547 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E 2024-02-01T10:27:48.547 DEBUG:boto:Method: HEAD 2024-02-01T10:27:48.547 DEBUG:boto:Path: /vclfcn-19/obj?versionId=2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E 2024-02-01T10:27:48.547 DEBUG:boto:Data: 2024-02-01T10:27:48.547 DEBUG:boto:Headers: {} 2024-02-01T10:27:48.547 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:48.547 DEBUG:boto:Port: 8001 2024-02-01T10:27:48.547 DEBUG:boto:Params: {} 2024-02-01T10:27:48.547 DEBUG:boto:Token: None 2024-02-01T10:27:48.548 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:48 GMT /vclfcn-19/obj?versionId=2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E 2024-02-01T10:27:48.548 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:uNwYod0LgXMDf380FEnYQVoCfrY= 2024-02-01T10:27:48.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:uNwYod0LgXMDf380FEnYQVoCfrY=', 'Content-Length': '0'} 2024-02-01T10:27:48.550 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ad14a644990fdc06-0065bb7224-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:48.551 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:48.551 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:48.551 DEBUG:boto:path=/vclfcn-19/obj?versionId=2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E 2024-02-01T10:27:48.551 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E 2024-02-01T10:27:48.551 DEBUG:boto:Method: HEAD 2024-02-01T10:27:48.551 DEBUG:boto:Path: /vclfcn-19/obj?versionId=2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E 2024-02-01T10:27:48.551 DEBUG:boto:Data: 2024-02-01T10:27:48.551 DEBUG:boto:Headers: {} 2024-02-01T10:27:48.551 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:48.551 DEBUG:boto:Port: 8000 2024-02-01T10:27:48.551 DEBUG:boto:Params: {} 2024-02-01T10:27:48.552 DEBUG:boto:Token: None 2024-02-01T10:27:48.552 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:48 GMT /vclfcn-19/obj?versionId=2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E 2024-02-01T10:27:48.552 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:uNwYod0LgXMDf380FEnYQVoCfrY= 2024-02-01T10:27:48.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:uNwYod0LgXMDf380FEnYQVoCfrY=', 'Content-Length': '0'} 2024-02-01T10:27:48.554 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004c32789ada4f828f-0065bb7224-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:48.555 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:48.555 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:48.555 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:48.555 DEBUG:boto:path=/vclfcn-19/obj?versionId=2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E 2024-02-01T10:27:48.555 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E 2024-02-01T10:27:48.555 DEBUG:boto:Method: GET 2024-02-01T10:27:48.555 DEBUG:boto:Path: /vclfcn-19/obj?versionId=2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E 2024-02-01T10:27:48.555 DEBUG:boto:Data: 2024-02-01T10:27:48.556 DEBUG:boto:Headers: {} 2024-02-01T10:27:48.556 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:48.556 DEBUG:boto:Port: 8001 2024-02-01T10:27:48.556 DEBUG:boto:Params: {} 2024-02-01T10:27:48.556 DEBUG:boto:Token: None 2024-02-01T10:27:48.556 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:48 GMT /vclfcn-19/obj?versionId=2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E 2024-02-01T10:27:48.556 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5nsJ6TkuEK8Wc0d6YTpy21Lcsf4= 2024-02-01T10:27:48.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5nsJ6TkuEK8Wc0d6YTpy21Lcsf4=', 'Content-Length': '0'} 2024-02-01T10:27:48.559 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005ab22cabe45e5158-0065bb7224-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:48.600 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:48.600 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:48.600 DEBUG:boto:path=/vclfcn-19/obj?versionId=2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E 2024-02-01T10:27:48.600 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E 2024-02-01T10:27:48.601 DEBUG:boto:Method: GET 2024-02-01T10:27:48.601 DEBUG:boto:Path: /vclfcn-19/obj?versionId=2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E 2024-02-01T10:27:48.601 DEBUG:boto:Data: 2024-02-01T10:27:48.601 DEBUG:boto:Headers: {} 2024-02-01T10:27:48.601 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:48.601 DEBUG:boto:Port: 8000 2024-02-01T10:27:48.601 DEBUG:boto:Params: {} 2024-02-01T10:27:48.601 DEBUG:boto:Token: None 2024-02-01T10:27:48.601 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:48 GMT /vclfcn-19/obj?versionId=2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E 2024-02-01T10:27:48.602 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5nsJ6TkuEK8Wc0d6YTpy21Lcsf4= 2024-02-01T10:27:48.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5nsJ6TkuEK8Wc0d6YTpy21Lcsf4=', 'Content-Length': '0'} 2024-02-01T10:27:48.604 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f7e1f8ebc9b6889e-0065bb7224-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:48.605 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:48.605 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:48.605 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:48.605 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:48.605 DEBUG:boto:Method: GET 2024-02-01T10:27:48.605 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:48.605 DEBUG:boto:Data: 2024-02-01T10:27:48.605 DEBUG:boto:Headers: {} 2024-02-01T10:27:48.605 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:48.606 DEBUG:boto:Port: 8001 2024-02-01T10:27:48.606 DEBUG:boto:Params: {} 2024-02-01T10:27:48.606 DEBUG:boto:Token: None 2024-02-01T10:27:48.606 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:48 GMT /vclfcn-19/ 2024-02-01T10:27:48.606 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HRD/rg3IwtslhykDLnQayY2iTZE= 2024-02-01T10:27:48.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HRD/rg3IwtslhykDLnQayY2iTZE=', 'Content-Length': '0'} 2024-02-01T10:27:48.612 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da63ca81b17c8014-0065bb7224-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:48.612 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:48.612 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:48.612 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:48.613 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:48.613 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:48.613 DEBUG:boto:Method: GET 2024-02-01T10:27:48.613 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:48.613 DEBUG:boto:Data: 2024-02-01T10:27:48.613 DEBUG:boto:Headers: {} 2024-02-01T10:27:48.613 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:48.613 DEBUG:boto:Port: 8000 2024-02-01T10:27:48.613 DEBUG:boto:Params: {} 2024-02-01T10:27:48.613 DEBUG:boto:Token: None 2024-02-01T10:27:48.614 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:48 GMT /vclfcn-19/ 2024-02-01T10:27:48.614 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HRD/rg3IwtslhykDLnQayY2iTZE= 2024-02-01T10:27:48.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HRD/rg3IwtslhykDLnQayY2iTZE=', 'Content-Length': '0'} 2024-02-01T10:27:48.619 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018625859dfa3f2ad-0065bb7224-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:48.660 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:48.660 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:48.661 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:48.661 DEBUG:boto:Method: HEAD 2024-02-01T10:27:48.661 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:48.661 DEBUG:boto:Data: 2024-02-01T10:27:48.661 DEBUG:boto:Headers: {} 2024-02-01T10:27:48.661 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:48.661 DEBUG:boto:Port: 8001 2024-02-01T10:27:48.661 DEBUG:boto:Params: {} 2024-02-01T10:27:48.661 DEBUG:boto:Token: None 2024-02-01T10:27:48.662 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:48 GMT /vclfcn-19/obj 2024-02-01T10:27:48.662 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+s2NWXYhQkqGUB2XyAxgfiM1mk8= 2024-02-01T10:27:48.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+s2NWXYhQkqGUB2XyAxgfiM1mk8=', 'Content-Length': '0'} 2024-02-01T10:27:48.670 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f8bda5acf29d3c49-0065bb7224-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:48.670 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:48.670 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:48.670 DEBUG:boto:Method: HEAD 2024-02-01T10:27:48.670 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:48.670 DEBUG:boto:Data: 2024-02-01T10:27:48.670 DEBUG:boto:Headers: {} 2024-02-01T10:27:48.670 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:48.671 DEBUG:boto:Port: 8000 2024-02-01T10:27:48.671 DEBUG:boto:Params: {} 2024-02-01T10:27:48.671 DEBUG:boto:Token: None 2024-02-01T10:27:48.671 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:48 GMT /vclfcn-19/obj 2024-02-01T10:27:48.671 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+s2NWXYhQkqGUB2XyAxgfiM1mk8= 2024-02-01T10:27:48.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+s2NWXYhQkqGUB2XyAxgfiM1mk8=', 'Content-Length': '0'} 2024-02-01T10:27:48.676 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007a107360b3cdc27d-0065bb7224-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:48.677 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:48.677 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:48.677 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:48.677 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:48.677 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:48.677 DEBUG:boto:Method: GET 2024-02-01T10:27:48.677 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:48.677 DEBUG:boto:Data: 2024-02-01T10:27:48.678 DEBUG:boto:Headers: {} 2024-02-01T10:27:48.678 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:48.678 DEBUG:boto:Port: 8001 2024-02-01T10:27:48.678 DEBUG:boto:Params: {} 2024-02-01T10:27:48.678 DEBUG:boto:Token: None 2024-02-01T10:27:48.678 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:48 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:48.678 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:DUyOlzSKaBK1ybAf0BkHs6Lub3A= 2024-02-01T10:27:48.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:DUyOlzSKaBK1ybAf0BkHs6Lub3A=', 'Content-Length': '0'} 2024-02-01T10:27:48.681 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ccc0aca7c80f016d-0065bb7224-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:48.724 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:48.725 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:48.725 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:48.725 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:48.725 DEBUG:boto:Method: GET 2024-02-01T10:27:48.725 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:48.725 DEBUG:boto:Data: 2024-02-01T10:27:48.725 DEBUG:boto:Headers: {} 2024-02-01T10:27:48.725 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:48.725 DEBUG:boto:Port: 8000 2024-02-01T10:27:48.725 DEBUG:boto:Params: {} 2024-02-01T10:27:48.726 DEBUG:boto:Token: None 2024-02-01T10:27:48.726 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:48 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:48.726 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:DUyOlzSKaBK1ybAf0BkHs6Lub3A= 2024-02-01T10:27:48.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:DUyOlzSKaBK1ybAf0BkHs6Lub3A=', 'Content-Length': '0'} 2024-02-01T10:27:48.729 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002bd4c63b0cf22e4f-0065bb7224-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:48.772 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:48.772 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:48.772 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:48.772 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:48.773 DEBUG:boto:Method: GET 2024-02-01T10:27:48.773 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:48.773 DEBUG:boto:Data: 2024-02-01T10:27:48.773 DEBUG:boto:Headers: {} 2024-02-01T10:27:48.773 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:48.773 DEBUG:boto:Port: 8001 2024-02-01T10:27:48.773 DEBUG:boto:Params: {} 2024-02-01T10:27:48.773 DEBUG:boto:Token: None 2024-02-01T10:27:48.773 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:48 GMT /vclfcn-19/ 2024-02-01T10:27:48.774 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HRD/rg3IwtslhykDLnQayY2iTZE= 2024-02-01T10:27:48.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HRD/rg3IwtslhykDLnQayY2iTZE=', 'Content-Length': '0'} 2024-02-01T10:27:48.779 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005264f7717bcff66b-0065bb7224-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:48.780 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:48.780 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:48.780 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:48.780 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:48.780 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:48.781 DEBUG:boto:Method: GET 2024-02-01T10:27:48.781 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:48.781 DEBUG:boto:Data: 2024-02-01T10:27:48.781 DEBUG:boto:Headers: {} 2024-02-01T10:27:48.781 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:48.781 DEBUG:boto:Port: 8000 2024-02-01T10:27:48.781 DEBUG:boto:Params: {} 2024-02-01T10:27:48.781 DEBUG:boto:Token: None 2024-02-01T10:27:48.781 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:48 GMT /vclfcn-19/ 2024-02-01T10:27:48.781 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HRD/rg3IwtslhykDLnQayY2iTZE= 2024-02-01T10:27:48.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HRD/rg3IwtslhykDLnQayY2iTZE=', 'Content-Length': '0'} 2024-02-01T10:27:48.786 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008425d99c914e4fd4-0065bb7224-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:48.832 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:48.833 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:48.833 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:48.833 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:48.833 DEBUG:boto:path=/vclfcn-19/obj?versionId=7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6 2024-02-01T10:27:48.833 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6 2024-02-01T10:27:48.833 DEBUG:boto:Method: GET 2024-02-01T10:27:48.833 DEBUG:boto:Path: /vclfcn-19/obj?versionId=7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6 2024-02-01T10:27:48.833 DEBUG:boto:Data: 2024-02-01T10:27:48.833 DEBUG:boto:Headers: {} 2024-02-01T10:27:48.833 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:48.834 DEBUG:boto:Port: 8001 2024-02-01T10:27:48.834 DEBUG:boto:Params: {} 2024-02-01T10:27:48.834 DEBUG:boto:Token: None 2024-02-01T10:27:48.834 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:48 GMT /vclfcn-19/obj?versionId=7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6 2024-02-01T10:27:48.834 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:EkWpbuJiuy3tECk7gRO/NMr99X8= 2024-02-01T10:27:48.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:EkWpbuJiuy3tECk7gRO/NMr99X8=', 'Content-Length': '0'} 2024-02-01T10:27:48.837 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f89ca71506cbdc37-0065bb7224-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:48.837 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:48.837 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:48.837 DEBUG:boto:path=/vclfcn-19/obj?versionId=7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6 2024-02-01T10:27:48.838 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6 2024-02-01T10:27:48.838 DEBUG:boto:Method: GET 2024-02-01T10:27:48.838 DEBUG:boto:Path: /vclfcn-19/obj?versionId=7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6 2024-02-01T10:27:48.838 DEBUG:boto:Data: 2024-02-01T10:27:48.838 DEBUG:boto:Headers: {} 2024-02-01T10:27:48.838 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:48.838 DEBUG:boto:Port: 8000 2024-02-01T10:27:48.838 DEBUG:boto:Params: {} 2024-02-01T10:27:48.838 DEBUG:boto:Token: None 2024-02-01T10:27:48.838 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:48 GMT /vclfcn-19/obj?versionId=7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6 2024-02-01T10:27:48.839 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:EkWpbuJiuy3tECk7gRO/NMr99X8= 2024-02-01T10:27:48.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:EkWpbuJiuy3tECk7gRO/NMr99X8=', 'Content-Length': '0'} 2024-02-01T10:27:48.841 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f0dce42f28a1cbc8-0065bb7224-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:48.884 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:48.884 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:48.884 DEBUG:boto:path=/vclfcn-19/obj?versionId=7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6 2024-02-01T10:27:48.884 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6 2024-02-01T10:27:48.885 DEBUG:boto:Method: HEAD 2024-02-01T10:27:48.885 DEBUG:boto:Path: /vclfcn-19/obj?versionId=7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6 2024-02-01T10:27:48.885 DEBUG:boto:Data: 2024-02-01T10:27:48.885 DEBUG:boto:Headers: {} 2024-02-01T10:27:48.885 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:48.885 DEBUG:boto:Port: 8001 2024-02-01T10:27:48.885 DEBUG:boto:Params: {} 2024-02-01T10:27:48.885 DEBUG:boto:Token: None 2024-02-01T10:27:48.885 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:48 GMT /vclfcn-19/obj?versionId=7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6 2024-02-01T10:27:48.886 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jXLxbhaNYs4Rsn9EZPrOB2ANyNo= 2024-02-01T10:27:48.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jXLxbhaNYs4Rsn9EZPrOB2ANyNo=', 'Content-Length': '0'} 2024-02-01T10:27:48.889 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f2b2c82c24202bcc-0065bb7224-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:48.889 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:48.889 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:48.889 DEBUG:boto:path=/vclfcn-19/obj?versionId=7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6 2024-02-01T10:27:48.889 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6 2024-02-01T10:27:48.889 DEBUG:boto:Method: HEAD 2024-02-01T10:27:48.889 DEBUG:boto:Path: /vclfcn-19/obj?versionId=7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6 2024-02-01T10:27:48.889 DEBUG:boto:Data: 2024-02-01T10:27:48.889 DEBUG:boto:Headers: {} 2024-02-01T10:27:48.890 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:48.890 DEBUG:boto:Port: 8000 2024-02-01T10:27:48.890 DEBUG:boto:Params: {} 2024-02-01T10:27:48.890 DEBUG:boto:Token: None 2024-02-01T10:27:48.890 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:48 GMT /vclfcn-19/obj?versionId=7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6 2024-02-01T10:27:48.890 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jXLxbhaNYs4Rsn9EZPrOB2ANyNo= 2024-02-01T10:27:48.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jXLxbhaNYs4Rsn9EZPrOB2ANyNo=', 'Content-Length': '0'} 2024-02-01T10:27:48.893 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f96f21a6785bbf9e-0065bb7224-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:48.893 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:48.893 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:48.893 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:48.893 DEBUG:boto:path=/vclfcn-19/obj?versionId=7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6 2024-02-01T10:27:48.893 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6 2024-02-01T10:27:48.894 DEBUG:boto:Method: GET 2024-02-01T10:27:48.894 DEBUG:boto:Path: /vclfcn-19/obj?versionId=7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6 2024-02-01T10:27:48.894 DEBUG:boto:Data: 2024-02-01T10:27:48.894 DEBUG:boto:Headers: {} 2024-02-01T10:27:48.894 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:48.894 DEBUG:boto:Port: 8001 2024-02-01T10:27:48.894 DEBUG:boto:Params: {} 2024-02-01T10:27:48.894 DEBUG:boto:Token: None 2024-02-01T10:27:48.894 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:48 GMT /vclfcn-19/obj?versionId=7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6 2024-02-01T10:27:48.894 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:EkWpbuJiuy3tECk7gRO/NMr99X8= 2024-02-01T10:27:48.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:EkWpbuJiuy3tECk7gRO/NMr99X8=', 'Content-Length': '0'} 2024-02-01T10:27:48.897 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bb8d289a729799ba-0065bb7224-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:48.898 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:48.898 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:48.898 DEBUG:boto:path=/vclfcn-19/obj?versionId=7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6 2024-02-01T10:27:48.898 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6 2024-02-01T10:27:48.898 DEBUG:boto:Method: GET 2024-02-01T10:27:48.898 DEBUG:boto:Path: /vclfcn-19/obj?versionId=7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6 2024-02-01T10:27:48.898 DEBUG:boto:Data: 2024-02-01T10:27:48.898 DEBUG:boto:Headers: {} 2024-02-01T10:27:48.898 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:48.898 DEBUG:boto:Port: 8000 2024-02-01T10:27:48.899 DEBUG:boto:Params: {} 2024-02-01T10:27:48.899 DEBUG:boto:Token: None 2024-02-01T10:27:48.899 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:48 GMT /vclfcn-19/obj?versionId=7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6 2024-02-01T10:27:48.899 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:EkWpbuJiuy3tECk7gRO/NMr99X8= 2024-02-01T10:27:48.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:EkWpbuJiuy3tECk7gRO/NMr99X8=', 'Content-Length': '0'} 2024-02-01T10:27:48.902 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001d5da0c6e656ae4b-0065bb7224-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:48.944 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:48.944 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:48.944 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:48.945 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:48.945 DEBUG:boto:Method: GET 2024-02-01T10:27:48.945 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:48.945 DEBUG:boto:Data: 2024-02-01T10:27:48.945 DEBUG:boto:Headers: {} 2024-02-01T10:27:48.945 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:48.945 DEBUG:boto:Port: 8001 2024-02-01T10:27:48.945 DEBUG:boto:Params: {} 2024-02-01T10:27:48.945 DEBUG:boto:Token: None 2024-02-01T10:27:48.945 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:48 GMT /vclfcn-19/ 2024-02-01T10:27:48.946 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HRD/rg3IwtslhykDLnQayY2iTZE= 2024-02-01T10:27:48.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HRD/rg3IwtslhykDLnQayY2iTZE=', 'Content-Length': '0'} 2024-02-01T10:27:48.951 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b6e6b854ef07905e-0065bb7224-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:48.951 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:48.952 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:48.952 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:48.952 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:48.953 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:48.953 DEBUG:boto:Method: GET 2024-02-01T10:27:48.953 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:48.953 DEBUG:boto:Data: 2024-02-01T10:27:48.953 DEBUG:boto:Headers: {} 2024-02-01T10:27:48.953 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:48.953 DEBUG:boto:Port: 8000 2024-02-01T10:27:48.953 DEBUG:boto:Params: {} 2024-02-01T10:27:48.953 DEBUG:boto:Token: None 2024-02-01T10:27:48.954 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:48 GMT /vclfcn-19/ 2024-02-01T10:27:48.954 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HRD/rg3IwtslhykDLnQayY2iTZE= 2024-02-01T10:27:48.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HRD/rg3IwtslhykDLnQayY2iTZE=', 'Content-Length': '0'} 2024-02-01T10:27:48.959 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c311ac36ee3f03d4-0065bb7224-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:48.960 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:48.960 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:48.960 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:48.960 DEBUG:boto:Method: HEAD 2024-02-01T10:27:48.960 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:48.961 DEBUG:boto:Data: 2024-02-01T10:27:48.961 DEBUG:boto:Headers: {} 2024-02-01T10:27:48.961 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:48.961 DEBUG:boto:Port: 8001 2024-02-01T10:27:48.961 DEBUG:boto:Params: {} 2024-02-01T10:27:48.961 DEBUG:boto:Token: None 2024-02-01T10:27:48.961 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:48 GMT /vclfcn-19/obj 2024-02-01T10:27:48.962 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+s2NWXYhQkqGUB2XyAxgfiM1mk8= 2024-02-01T10:27:48.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+s2NWXYhQkqGUB2XyAxgfiM1mk8=', 'Content-Length': '0'} 2024-02-01T10:27:48.968 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b61f0e2089377ee4-0065bb7224-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:48.969 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:48.969 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:48.969 DEBUG:boto:Method: HEAD 2024-02-01T10:27:48.969 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:48.969 DEBUG:boto:Data: 2024-02-01T10:27:48.969 DEBUG:boto:Headers: {} 2024-02-01T10:27:48.969 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:48.970 DEBUG:boto:Port: 8000 2024-02-01T10:27:48.970 DEBUG:boto:Params: {} 2024-02-01T10:27:48.970 DEBUG:boto:Token: None 2024-02-01T10:27:48.970 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:48 GMT /vclfcn-19/obj 2024-02-01T10:27:48.970 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+s2NWXYhQkqGUB2XyAxgfiM1mk8= 2024-02-01T10:27:48.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+s2NWXYhQkqGUB2XyAxgfiM1mk8=', 'Content-Length': '0'} 2024-02-01T10:27:48.976 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000000dde42fcbe886c4-0065bb7224-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:48.976 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:48.976 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:48.976 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:48.976 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:48.976 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:48.977 DEBUG:boto:Method: GET 2024-02-01T10:27:48.977 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:48.977 DEBUG:boto:Data: 2024-02-01T10:27:48.977 DEBUG:boto:Headers: {} 2024-02-01T10:27:48.977 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:48.977 DEBUG:boto:Port: 8001 2024-02-01T10:27:48.977 DEBUG:boto:Params: {} 2024-02-01T10:27:48.977 DEBUG:boto:Token: None 2024-02-01T10:27:48.977 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:48 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:48.978 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:DUyOlzSKaBK1ybAf0BkHs6Lub3A= 2024-02-01T10:27:48.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:DUyOlzSKaBK1ybAf0BkHs6Lub3A=', 'Content-Length': '0'} 2024-02-01T10:27:48.980 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d9d05d0015d551e8-0065bb7224-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:49.024 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:49.024 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:49.024 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:49.025 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:49.025 DEBUG:boto:Method: GET 2024-02-01T10:27:49.025 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:49.025 DEBUG:boto:Data: 2024-02-01T10:27:49.025 DEBUG:boto:Headers: {} 2024-02-01T10:27:49.025 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:49.025 DEBUG:boto:Port: 8000 2024-02-01T10:27:49.025 DEBUG:boto:Params: {} 2024-02-01T10:27:49.025 DEBUG:boto:Token: None 2024-02-01T10:27:49.026 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:49 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:49.026 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8Qt5QRDoKGE4iVEr73USK9IGPYs= 2024-02-01T10:27:49.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8Qt5QRDoKGE4iVEr73USK9IGPYs=', 'Content-Length': '0'} 2024-02-01T10:27:49.029 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000017ffe0a6dfecbf8a-0065bb7225-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:49.072 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:49.072 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:49.072 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:49.073 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:49.073 DEBUG:boto:Method: GET 2024-02-01T10:27:49.073 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:49.073 DEBUG:boto:Data: 2024-02-01T10:27:49.073 DEBUG:boto:Headers: {} 2024-02-01T10:27:49.073 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:49.073 DEBUG:boto:Port: 8001 2024-02-01T10:27:49.073 DEBUG:boto:Params: {} 2024-02-01T10:27:49.073 DEBUG:boto:Token: None 2024-02-01T10:27:49.074 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:49 GMT /vclfcn-19/ 2024-02-01T10:27:49.074 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ycPXi051bWgnSuGizEh7kBZvlhY= 2024-02-01T10:27:49.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ycPXi051bWgnSuGizEh7kBZvlhY=', 'Content-Length': '0'} 2024-02-01T10:27:49.079 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e614168fc95db21d-0065bb7225-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:49.079 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:49.080 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:49.080 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:49.080 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:49.080 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:49.080 DEBUG:boto:Method: GET 2024-02-01T10:27:49.080 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:49.081 DEBUG:boto:Data: 2024-02-01T10:27:49.081 DEBUG:boto:Headers: {} 2024-02-01T10:27:49.081 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:49.081 DEBUG:boto:Port: 8000 2024-02-01T10:27:49.081 DEBUG:boto:Params: {} 2024-02-01T10:27:49.081 DEBUG:boto:Token: None 2024-02-01T10:27:49.081 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:49 GMT /vclfcn-19/ 2024-02-01T10:27:49.082 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ycPXi051bWgnSuGizEh7kBZvlhY= 2024-02-01T10:27:49.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ycPXi051bWgnSuGizEh7kBZvlhY=', 'Content-Length': '0'} 2024-02-01T10:27:49.088 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f09b92fda933feb6-0065bb7225-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:49.088 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:49.088 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:49.088 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:49.089 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:49.089 DEBUG:boto:path=/vclfcn-19/obj?versionId=.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX 2024-02-01T10:27:49.089 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX 2024-02-01T10:27:49.089 DEBUG:boto:Method: GET 2024-02-01T10:27:49.089 DEBUG:boto:Path: /vclfcn-19/obj?versionId=.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX 2024-02-01T10:27:49.089 DEBUG:boto:Data: 2024-02-01T10:27:49.089 DEBUG:boto:Headers: {} 2024-02-01T10:27:49.089 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:49.089 DEBUG:boto:Port: 8001 2024-02-01T10:27:49.089 DEBUG:boto:Params: {} 2024-02-01T10:27:49.089 DEBUG:boto:Token: None 2024-02-01T10:27:49.090 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:49 GMT /vclfcn-19/obj?versionId=.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX 2024-02-01T10:27:49.090 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:m6zcNohqHn5xXYESkRjc3BGQrbI= 2024-02-01T10:27:49.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:m6zcNohqHn5xXYESkRjc3BGQrbI=', 'Content-Length': '0'} 2024-02-01T10:27:49.093 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000079dcb12a2875bb07-0065bb7225-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:49.093 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:49.093 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:49.093 DEBUG:boto:path=/vclfcn-19/obj?versionId=.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX 2024-02-01T10:27:49.093 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX 2024-02-01T10:27:49.093 DEBUG:boto:Method: GET 2024-02-01T10:27:49.094 DEBUG:boto:Path: /vclfcn-19/obj?versionId=.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX 2024-02-01T10:27:49.094 DEBUG:boto:Data: 2024-02-01T10:27:49.094 DEBUG:boto:Headers: {} 2024-02-01T10:27:49.094 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:49.094 DEBUG:boto:Port: 8000 2024-02-01T10:27:49.094 DEBUG:boto:Params: {} 2024-02-01T10:27:49.094 DEBUG:boto:Token: None 2024-02-01T10:27:49.094 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:49 GMT /vclfcn-19/obj?versionId=.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX 2024-02-01T10:27:49.094 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:m6zcNohqHn5xXYESkRjc3BGQrbI= 2024-02-01T10:27:49.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:m6zcNohqHn5xXYESkRjc3BGQrbI=', 'Content-Length': '0'} 2024-02-01T10:27:49.097 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000001860956878639b7-0065bb7225-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:49.144 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:49.144 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:49.144 DEBUG:boto:path=/vclfcn-19/obj?versionId=.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX 2024-02-01T10:27:49.144 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX 2024-02-01T10:27:49.145 DEBUG:boto:Method: HEAD 2024-02-01T10:27:49.145 DEBUG:boto:Path: /vclfcn-19/obj?versionId=.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX 2024-02-01T10:27:49.145 DEBUG:boto:Data: 2024-02-01T10:27:49.145 DEBUG:boto:Headers: {} 2024-02-01T10:27:49.145 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:49.145 DEBUG:boto:Port: 8001 2024-02-01T10:27:49.145 DEBUG:boto:Params: {} 2024-02-01T10:27:49.145 DEBUG:boto:Token: None 2024-02-01T10:27:49.146 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:49 GMT /vclfcn-19/obj?versionId=.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX 2024-02-01T10:27:49.146 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lu0wjqa+z32rVfA6uHcsyVR4kco= 2024-02-01T10:27:49.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lu0wjqa+z32rVfA6uHcsyVR4kco=', 'Content-Length': '0'} 2024-02-01T10:27:49.149 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c0ea31cc9b3160fb-0065bb7225-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:49.149 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:49.149 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:49.149 DEBUG:boto:path=/vclfcn-19/obj?versionId=.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX 2024-02-01T10:27:49.150 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX 2024-02-01T10:27:49.150 DEBUG:boto:Method: HEAD 2024-02-01T10:27:49.150 DEBUG:boto:Path: /vclfcn-19/obj?versionId=.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX 2024-02-01T10:27:49.150 DEBUG:boto:Data: 2024-02-01T10:27:49.150 DEBUG:boto:Headers: {} 2024-02-01T10:27:49.150 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:49.150 DEBUG:boto:Port: 8000 2024-02-01T10:27:49.150 DEBUG:boto:Params: {} 2024-02-01T10:27:49.150 DEBUG:boto:Token: None 2024-02-01T10:27:49.151 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:49 GMT /vclfcn-19/obj?versionId=.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX 2024-02-01T10:27:49.151 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lu0wjqa+z32rVfA6uHcsyVR4kco= 2024-02-01T10:27:49.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lu0wjqa+z32rVfA6uHcsyVR4kco=', 'Content-Length': '0'} 2024-02-01T10:27:49.153 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f88c291a9190423d-0065bb7225-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:49.154 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:49.154 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:49.154 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:49.154 DEBUG:boto:path=/vclfcn-19/obj?versionId=.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX 2024-02-01T10:27:49.154 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX 2024-02-01T10:27:49.154 DEBUG:boto:Method: GET 2024-02-01T10:27:49.154 DEBUG:boto:Path: /vclfcn-19/obj?versionId=.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX 2024-02-01T10:27:49.154 DEBUG:boto:Data: 2024-02-01T10:27:49.154 DEBUG:boto:Headers: {} 2024-02-01T10:27:49.155 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:49.155 DEBUG:boto:Port: 8001 2024-02-01T10:27:49.155 DEBUG:boto:Params: {} 2024-02-01T10:27:49.155 DEBUG:boto:Token: None 2024-02-01T10:27:49.155 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:49 GMT /vclfcn-19/obj?versionId=.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX 2024-02-01T10:27:49.155 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:m6zcNohqHn5xXYESkRjc3BGQrbI= 2024-02-01T10:27:49.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:m6zcNohqHn5xXYESkRjc3BGQrbI=', 'Content-Length': '0'} 2024-02-01T10:27:49.158 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000038dfe624037db00a-0065bb7225-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:49.200 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:49.200 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:49.201 DEBUG:boto:path=/vclfcn-19/obj?versionId=.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX 2024-02-01T10:27:49.201 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX 2024-02-01T10:27:49.201 DEBUG:boto:Method: GET 2024-02-01T10:27:49.201 DEBUG:boto:Path: /vclfcn-19/obj?versionId=.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX 2024-02-01T10:27:49.201 DEBUG:boto:Data: 2024-02-01T10:27:49.201 DEBUG:boto:Headers: {} 2024-02-01T10:27:49.201 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:49.201 DEBUG:boto:Port: 8000 2024-02-01T10:27:49.201 DEBUG:boto:Params: {} 2024-02-01T10:27:49.202 DEBUG:boto:Token: None 2024-02-01T10:27:49.202 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:49 GMT /vclfcn-19/obj?versionId=.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX 2024-02-01T10:27:49.202 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:m6zcNohqHn5xXYESkRjc3BGQrbI= 2024-02-01T10:27:49.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:m6zcNohqHn5xXYESkRjc3BGQrbI=', 'Content-Length': '0'} 2024-02-01T10:27:49.205 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a176ffc9114c7de9-0065bb7225-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:49.248 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:49.248 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:49.248 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:49.249 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:49.249 DEBUG:boto:Method: GET 2024-02-01T10:27:49.249 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:49.249 DEBUG:boto:Data: 2024-02-01T10:27:49.249 DEBUG:boto:Headers: {} 2024-02-01T10:27:49.249 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:49.249 DEBUG:boto:Port: 8001 2024-02-01T10:27:49.249 DEBUG:boto:Params: {} 2024-02-01T10:27:49.249 DEBUG:boto:Token: None 2024-02-01T10:27:49.249 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:49 GMT /vclfcn-19/ 2024-02-01T10:27:49.250 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ycPXi051bWgnSuGizEh7kBZvlhY= 2024-02-01T10:27:49.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ycPXi051bWgnSuGizEh7kBZvlhY=', 'Content-Length': '0'} 2024-02-01T10:27:49.255 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca4ef6ab63107014-0065bb7225-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:49.296 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:49.297 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:49.297 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:49.297 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:49.297 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:49.297 DEBUG:boto:Method: GET 2024-02-01T10:27:49.297 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:49.297 DEBUG:boto:Data: 2024-02-01T10:27:49.297 DEBUG:boto:Headers: {} 2024-02-01T10:27:49.297 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:49.297 DEBUG:boto:Port: 8000 2024-02-01T10:27:49.298 DEBUG:boto:Params: {} 2024-02-01T10:27:49.298 DEBUG:boto:Token: None 2024-02-01T10:27:49.298 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:49 GMT /vclfcn-19/ 2024-02-01T10:27:49.298 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ycPXi051bWgnSuGizEh7kBZvlhY= 2024-02-01T10:27:49.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ycPXi051bWgnSuGizEh7kBZvlhY=', 'Content-Length': '0'} 2024-02-01T10:27:49.304 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000074e4d0bde0f75111-0065bb7225-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:49.304 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:49.304 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:49.305 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:49.305 DEBUG:boto:Method: HEAD 2024-02-01T10:27:49.305 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:49.305 DEBUG:boto:Data: 2024-02-01T10:27:49.305 DEBUG:boto:Headers: {} 2024-02-01T10:27:49.305 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:49.305 DEBUG:boto:Port: 8001 2024-02-01T10:27:49.305 DEBUG:boto:Params: {} 2024-02-01T10:27:49.305 DEBUG:boto:Token: None 2024-02-01T10:27:49.305 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:49 GMT /vclfcn-19/obj 2024-02-01T10:27:49.306 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5vY2I3aFYDj18RtcSWgpyW3WwOE= 2024-02-01T10:27:49.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5vY2I3aFYDj18RtcSWgpyW3WwOE=', 'Content-Length': '0'} 2024-02-01T10:27:49.311 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ef9b674b48cb463c-0065bb7225-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:49.312 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:49.312 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:49.312 DEBUG:boto:Method: HEAD 2024-02-01T10:27:49.312 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:49.312 DEBUG:boto:Data: 2024-02-01T10:27:49.312 DEBUG:boto:Headers: {} 2024-02-01T10:27:49.312 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:49.312 DEBUG:boto:Port: 8000 2024-02-01T10:27:49.312 DEBUG:boto:Params: {} 2024-02-01T10:27:49.313 DEBUG:boto:Token: None 2024-02-01T10:27:49.313 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:49 GMT /vclfcn-19/obj 2024-02-01T10:27:49.313 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5vY2I3aFYDj18RtcSWgpyW3WwOE= 2024-02-01T10:27:49.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5vY2I3aFYDj18RtcSWgpyW3WwOE=', 'Content-Length': '0'} 2024-02-01T10:27:49.319 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000026c63fd0a7f6ced6-0065bb7225-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:49.320 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:49.320 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:49.320 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:49.320 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:49.320 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:49.320 DEBUG:boto:Method: GET 2024-02-01T10:27:49.320 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:49.320 DEBUG:boto:Data: 2024-02-01T10:27:49.320 DEBUG:boto:Headers: {} 2024-02-01T10:27:49.321 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:49.321 DEBUG:boto:Port: 8001 2024-02-01T10:27:49.321 DEBUG:boto:Params: {} 2024-02-01T10:27:49.321 DEBUG:boto:Token: None 2024-02-01T10:27:49.321 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:49 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:49.321 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8Qt5QRDoKGE4iVEr73USK9IGPYs= 2024-02-01T10:27:49.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8Qt5QRDoKGE4iVEr73USK9IGPYs=', 'Content-Length': '0'} 2024-02-01T10:27:49.324 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a39b1e88ab7961f6-0065bb7225-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:49.368 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:49.368 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:49.368 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:49.368 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:49.369 DEBUG:boto:Method: GET 2024-02-01T10:27:49.369 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:49.369 DEBUG:boto:Data: 2024-02-01T10:27:49.369 DEBUG:boto:Headers: {} 2024-02-01T10:27:49.369 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:49.369 DEBUG:boto:Port: 8000 2024-02-01T10:27:49.369 DEBUG:boto:Params: {} 2024-02-01T10:27:49.369 DEBUG:boto:Token: None 2024-02-01T10:27:49.369 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:49 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:49.370 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8Qt5QRDoKGE4iVEr73USK9IGPYs= 2024-02-01T10:27:49.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8Qt5QRDoKGE4iVEr73USK9IGPYs=', 'Content-Length': '0'} 2024-02-01T10:27:49.373 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f7a3ba635db89b69-0065bb7225-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:49.416 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:49.419 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:49.420 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:49.420 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:49.420 DEBUG:boto:Method: GET 2024-02-01T10:27:49.420 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:49.420 DEBUG:boto:Data: 2024-02-01T10:27:49.420 DEBUG:boto:Headers: {} 2024-02-01T10:27:49.420 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:49.420 DEBUG:boto:Port: 8001 2024-02-01T10:27:49.420 DEBUG:boto:Params: {} 2024-02-01T10:27:49.420 DEBUG:boto:Token: None 2024-02-01T10:27:49.421 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:49 GMT /vclfcn-19/ 2024-02-01T10:27:49.421 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ycPXi051bWgnSuGizEh7kBZvlhY= 2024-02-01T10:27:49.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ycPXi051bWgnSuGizEh7kBZvlhY=', 'Content-Length': '0'} 2024-02-01T10:27:49.427 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025d8a2c54bea860f-0065bb7225-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:49.468 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:49.469 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:49.469 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:49.469 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:49.469 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:49.469 DEBUG:boto:Method: GET 2024-02-01T10:27:49.469 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:49.469 DEBUG:boto:Data: 2024-02-01T10:27:49.469 DEBUG:boto:Headers: {} 2024-02-01T10:27:49.469 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:49.469 DEBUG:boto:Port: 8000 2024-02-01T10:27:49.470 DEBUG:boto:Params: {} 2024-02-01T10:27:49.470 DEBUG:boto:Token: None 2024-02-01T10:27:49.470 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:49 GMT /vclfcn-19/ 2024-02-01T10:27:49.470 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ycPXi051bWgnSuGizEh7kBZvlhY= 2024-02-01T10:27:49.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ycPXi051bWgnSuGizEh7kBZvlhY=', 'Content-Length': '0'} 2024-02-01T10:27:49.475 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000678b8983d5b5bcf6-0065bb7225-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:49.476 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:49.476 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:49.476 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:49.476 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:49.477 DEBUG:boto:path=/vclfcn-19/obj?versionId=63CC4agggciY3jdYETzlNEHLx15uh8w 2024-02-01T10:27:49.477 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=63CC4agggciY3jdYETzlNEHLx15uh8w 2024-02-01T10:27:49.477 DEBUG:boto:Method: GET 2024-02-01T10:27:49.477 DEBUG:boto:Path: /vclfcn-19/obj?versionId=63CC4agggciY3jdYETzlNEHLx15uh8w 2024-02-01T10:27:49.477 DEBUG:boto:Data: 2024-02-01T10:27:49.477 DEBUG:boto:Headers: {} 2024-02-01T10:27:49.477 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:49.477 DEBUG:boto:Port: 8001 2024-02-01T10:27:49.477 DEBUG:boto:Params: {} 2024-02-01T10:27:49.477 DEBUG:boto:Token: None 2024-02-01T10:27:49.478 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:49 GMT /vclfcn-19/obj?versionId=63CC4agggciY3jdYETzlNEHLx15uh8w 2024-02-01T10:27:49.478 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:v/JMGcVjWb0VA+UhsT6ENinPHyU= 2024-02-01T10:27:49.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:v/JMGcVjWb0VA+UhsT6ENinPHyU=', 'Content-Length': '0'} 2024-02-01T10:27:49.481 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '63CC4agggciY3jdYETzlNEHLx15uh8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a6681bf9606b38da-0065bb7225-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:49.481 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:49.481 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:49.482 DEBUG:boto:path=/vclfcn-19/obj?versionId=63CC4agggciY3jdYETzlNEHLx15uh8w 2024-02-01T10:27:49.482 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=63CC4agggciY3jdYETzlNEHLx15uh8w 2024-02-01T10:27:49.482 DEBUG:boto:Method: GET 2024-02-01T10:27:49.482 DEBUG:boto:Path: /vclfcn-19/obj?versionId=63CC4agggciY3jdYETzlNEHLx15uh8w 2024-02-01T10:27:49.482 DEBUG:boto:Data: 2024-02-01T10:27:49.482 DEBUG:boto:Headers: {} 2024-02-01T10:27:49.482 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:49.482 DEBUG:boto:Port: 8000 2024-02-01T10:27:49.482 DEBUG:boto:Params: {} 2024-02-01T10:27:49.482 DEBUG:boto:Token: None 2024-02-01T10:27:49.482 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:49 GMT /vclfcn-19/obj?versionId=63CC4agggciY3jdYETzlNEHLx15uh8w 2024-02-01T10:27:49.483 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:v/JMGcVjWb0VA+UhsT6ENinPHyU= 2024-02-01T10:27:49.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:v/JMGcVjWb0VA+UhsT6ENinPHyU=', 'Content-Length': '0'} 2024-02-01T10:27:49.485 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '63CC4agggciY3jdYETzlNEHLx15uh8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e8a2fb7125cb426c-0065bb7225-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:49.528 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:49.528 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:49.528 DEBUG:boto:path=/vclfcn-19/obj?versionId=63CC4agggciY3jdYETzlNEHLx15uh8w 2024-02-01T10:27:49.529 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=63CC4agggciY3jdYETzlNEHLx15uh8w 2024-02-01T10:27:49.529 DEBUG:boto:Method: HEAD 2024-02-01T10:27:49.529 DEBUG:boto:Path: /vclfcn-19/obj?versionId=63CC4agggciY3jdYETzlNEHLx15uh8w 2024-02-01T10:27:49.529 DEBUG:boto:Data: 2024-02-01T10:27:49.529 DEBUG:boto:Headers: {} 2024-02-01T10:27:49.529 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:49.529 DEBUG:boto:Port: 8001 2024-02-01T10:27:49.529 DEBUG:boto:Params: {} 2024-02-01T10:27:49.529 DEBUG:boto:Token: None 2024-02-01T10:27:49.529 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:49 GMT /vclfcn-19/obj?versionId=63CC4agggciY3jdYETzlNEHLx15uh8w 2024-02-01T10:27:49.530 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gc2Ktq3KxXmQId2di6gj8fLmsPU= 2024-02-01T10:27:49.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gc2Ktq3KxXmQId2di6gj8fLmsPU=', 'Content-Length': '0'} 2024-02-01T10:27:49.533 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '63CC4agggciY3jdYETzlNEHLx15uh8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000000cec5540abf48c2-0065bb7225-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:49.533 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:49.533 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:49.533 DEBUG:boto:path=/vclfcn-19/obj?versionId=63CC4agggciY3jdYETzlNEHLx15uh8w 2024-02-01T10:27:49.533 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=63CC4agggciY3jdYETzlNEHLx15uh8w 2024-02-01T10:27:49.534 DEBUG:boto:Method: HEAD 2024-02-01T10:27:49.534 DEBUG:boto:Path: /vclfcn-19/obj?versionId=63CC4agggciY3jdYETzlNEHLx15uh8w 2024-02-01T10:27:49.534 DEBUG:boto:Data: 2024-02-01T10:27:49.534 DEBUG:boto:Headers: {} 2024-02-01T10:27:49.534 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:49.534 DEBUG:boto:Port: 8000 2024-02-01T10:27:49.534 DEBUG:boto:Params: {} 2024-02-01T10:27:49.534 DEBUG:boto:Token: None 2024-02-01T10:27:49.534 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:49 GMT /vclfcn-19/obj?versionId=63CC4agggciY3jdYETzlNEHLx15uh8w 2024-02-01T10:27:49.534 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gc2Ktq3KxXmQId2di6gj8fLmsPU= 2024-02-01T10:27:49.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gc2Ktq3KxXmQId2di6gj8fLmsPU=', 'Content-Length': '0'} 2024-02-01T10:27:49.537 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '63CC4agggciY3jdYETzlNEHLx15uh8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f8b96b1bc76c1b5b-0065bb7225-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:49.537 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:49.538 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:49.538 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:49.538 DEBUG:boto:path=/vclfcn-19/obj?versionId=63CC4agggciY3jdYETzlNEHLx15uh8w 2024-02-01T10:27:49.539 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=63CC4agggciY3jdYETzlNEHLx15uh8w 2024-02-01T10:27:49.539 DEBUG:boto:Method: GET 2024-02-01T10:27:49.539 DEBUG:boto:Path: /vclfcn-19/obj?versionId=63CC4agggciY3jdYETzlNEHLx15uh8w 2024-02-01T10:27:49.539 DEBUG:boto:Data: 2024-02-01T10:27:49.539 DEBUG:boto:Headers: {} 2024-02-01T10:27:49.539 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:49.539 DEBUG:boto:Port: 8001 2024-02-01T10:27:49.539 DEBUG:boto:Params: {} 2024-02-01T10:27:49.539 DEBUG:boto:Token: None 2024-02-01T10:27:49.539 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:49 GMT /vclfcn-19/obj?versionId=63CC4agggciY3jdYETzlNEHLx15uh8w 2024-02-01T10:27:49.540 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:v/JMGcVjWb0VA+UhsT6ENinPHyU= 2024-02-01T10:27:49.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:v/JMGcVjWb0VA+UhsT6ENinPHyU=', 'Content-Length': '0'} 2024-02-01T10:27:49.542 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '63CC4agggciY3jdYETzlNEHLx15uh8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001dcef8dd072d671b-0065bb7225-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:49.543 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:49.543 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:49.543 DEBUG:boto:path=/vclfcn-19/obj?versionId=63CC4agggciY3jdYETzlNEHLx15uh8w 2024-02-01T10:27:49.543 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=63CC4agggciY3jdYETzlNEHLx15uh8w 2024-02-01T10:27:49.543 DEBUG:boto:Method: GET 2024-02-01T10:27:49.543 DEBUG:boto:Path: /vclfcn-19/obj?versionId=63CC4agggciY3jdYETzlNEHLx15uh8w 2024-02-01T10:27:49.544 DEBUG:boto:Data: 2024-02-01T10:27:49.544 DEBUG:boto:Headers: {} 2024-02-01T10:27:49.544 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:49.544 DEBUG:boto:Port: 8000 2024-02-01T10:27:49.544 DEBUG:boto:Params: {} 2024-02-01T10:27:49.544 DEBUG:boto:Token: None 2024-02-01T10:27:49.544 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:49 GMT /vclfcn-19/obj?versionId=63CC4agggciY3jdYETzlNEHLx15uh8w 2024-02-01T10:27:49.544 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:v/JMGcVjWb0VA+UhsT6ENinPHyU= 2024-02-01T10:27:49.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:v/JMGcVjWb0VA+UhsT6ENinPHyU=', 'Content-Length': '0'} 2024-02-01T10:27:49.547 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '63CC4agggciY3jdYETzlNEHLx15uh8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000256929dfa0d422c6-0065bb7225-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:49.588 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:49.588 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:49.588 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:49.589 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:49.589 DEBUG:boto:Method: GET 2024-02-01T10:27:49.589 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:49.589 DEBUG:boto:Data: 2024-02-01T10:27:49.589 DEBUG:boto:Headers: {} 2024-02-01T10:27:49.589 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:49.589 DEBUG:boto:Port: 8001 2024-02-01T10:27:49.590 DEBUG:boto:Params: {} 2024-02-01T10:27:49.590 DEBUG:boto:Token: None 2024-02-01T10:27:49.590 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:49 GMT /vclfcn-19/ 2024-02-01T10:27:49.590 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ycPXi051bWgnSuGizEh7kBZvlhY= 2024-02-01T10:27:49.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ycPXi051bWgnSuGizEh7kBZvlhY=', 'Content-Length': '0'} 2024-02-01T10:27:49.596 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4d49e4192740a1b-0065bb7225-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:49.596 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:49.597 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:49.597 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:49.597 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:49.597 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:49.597 DEBUG:boto:Method: GET 2024-02-01T10:27:49.597 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:49.597 DEBUG:boto:Data: 2024-02-01T10:27:49.597 DEBUG:boto:Headers: {} 2024-02-01T10:27:49.598 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:49.598 DEBUG:boto:Port: 8000 2024-02-01T10:27:49.598 DEBUG:boto:Params: {} 2024-02-01T10:27:49.598 DEBUG:boto:Token: None 2024-02-01T10:27:49.598 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:49 GMT /vclfcn-19/ 2024-02-01T10:27:49.598 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ycPXi051bWgnSuGizEh7kBZvlhY= 2024-02-01T10:27:49.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ycPXi051bWgnSuGizEh7kBZvlhY=', 'Content-Length': '0'} 2024-02-01T10:27:49.604 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac2cd12533fd217d-0065bb7225-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:49.604 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:49.604 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:49.604 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:49.604 DEBUG:boto:Method: HEAD 2024-02-01T10:27:49.604 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:49.605 DEBUG:boto:Data: 2024-02-01T10:27:49.605 DEBUG:boto:Headers: {} 2024-02-01T10:27:49.605 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:49.605 DEBUG:boto:Port: 8001 2024-02-01T10:27:49.605 DEBUG:boto:Params: {} 2024-02-01T10:27:49.605 DEBUG:boto:Token: None 2024-02-01T10:27:49.605 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:49 GMT /vclfcn-19/obj 2024-02-01T10:27:49.605 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5vY2I3aFYDj18RtcSWgpyW3WwOE= 2024-02-01T10:27:49.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5vY2I3aFYDj18RtcSWgpyW3WwOE=', 'Content-Length': '0'} 2024-02-01T10:27:49.612 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000925ebe16caab1954-0065bb7225-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:49.612 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:49.612 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:49.612 DEBUG:boto:Method: HEAD 2024-02-01T10:27:49.612 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:49.612 DEBUG:boto:Data: 2024-02-01T10:27:49.612 DEBUG:boto:Headers: {} 2024-02-01T10:27:49.612 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:49.613 DEBUG:boto:Port: 8000 2024-02-01T10:27:49.613 DEBUG:boto:Params: {} 2024-02-01T10:27:49.613 DEBUG:boto:Token: None 2024-02-01T10:27:49.613 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:49 GMT /vclfcn-19/obj 2024-02-01T10:27:49.613 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5vY2I3aFYDj18RtcSWgpyW3WwOE= 2024-02-01T10:27:49.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5vY2I3aFYDj18RtcSWgpyW3WwOE=', 'Content-Length': '0'} 2024-02-01T10:27:49.619 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007bf4c807971fb1b4-0065bb7225-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:49.620 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:49.620 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:49.620 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:49.620 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:49.620 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:49.620 DEBUG:boto:Method: GET 2024-02-01T10:27:49.620 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:49.620 DEBUG:boto:Data: 2024-02-01T10:27:49.620 DEBUG:boto:Headers: {} 2024-02-01T10:27:49.620 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:49.621 DEBUG:boto:Port: 8001 2024-02-01T10:27:49.621 DEBUG:boto:Params: {} 2024-02-01T10:27:49.621 DEBUG:boto:Token: None 2024-02-01T10:27:49.621 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:49 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:49.621 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8Qt5QRDoKGE4iVEr73USK9IGPYs= 2024-02-01T10:27:49.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8Qt5QRDoKGE4iVEr73USK9IGPYs=', 'Content-Length': '0'} 2024-02-01T10:27:49.624 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dde106db4991c207-0065bb7225-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:49.668 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:49.668 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:49.668 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:49.668 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:49.669 DEBUG:boto:Method: GET 2024-02-01T10:27:49.669 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:49.669 DEBUG:boto:Data: 2024-02-01T10:27:49.669 DEBUG:boto:Headers: {} 2024-02-01T10:27:49.669 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:49.669 DEBUG:boto:Port: 8000 2024-02-01T10:27:49.669 DEBUG:boto:Params: {} 2024-02-01T10:27:49.669 DEBUG:boto:Token: None 2024-02-01T10:27:49.669 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:49 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:49.670 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8Qt5QRDoKGE4iVEr73USK9IGPYs= 2024-02-01T10:27:49.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8Qt5QRDoKGE4iVEr73USK9IGPYs=', 'Content-Length': '0'} 2024-02-01T10:27:49.673 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000be8453e9c933d040-0065bb7225-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:49.716 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:49.716 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:49.717 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:49.717 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:49.717 DEBUG:boto:Method: GET 2024-02-01T10:27:49.717 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:49.717 DEBUG:boto:Data: 2024-02-01T10:27:49.717 DEBUG:boto:Headers: {} 2024-02-01T10:27:49.717 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:49.717 DEBUG:boto:Port: 8001 2024-02-01T10:27:49.717 DEBUG:boto:Params: {} 2024-02-01T10:27:49.717 DEBUG:boto:Token: None 2024-02-01T10:27:49.718 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:49 GMT /vclfcn-19/ 2024-02-01T10:27:49.718 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ycPXi051bWgnSuGizEh7kBZvlhY= 2024-02-01T10:27:49.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ycPXi051bWgnSuGizEh7kBZvlhY=', 'Content-Length': '0'} 2024-02-01T10:27:49.728 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094614d5f2eba9fd9-0065bb7225-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:49.728 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:49.729 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:49.729 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:49.729 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:49.729 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:49.729 DEBUG:boto:Method: GET 2024-02-01T10:27:49.729 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:49.729 DEBUG:boto:Data: 2024-02-01T10:27:49.729 DEBUG:boto:Headers: {} 2024-02-01T10:27:49.729 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:49.730 DEBUG:boto:Port: 8000 2024-02-01T10:27:49.730 DEBUG:boto:Params: {} 2024-02-01T10:27:49.730 DEBUG:boto:Token: None 2024-02-01T10:27:49.730 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:49 GMT /vclfcn-19/ 2024-02-01T10:27:49.730 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ycPXi051bWgnSuGizEh7kBZvlhY= 2024-02-01T10:27:49.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ycPXi051bWgnSuGizEh7kBZvlhY=', 'Content-Length': '0'} 2024-02-01T10:27:49.735 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000709de2b4d8a7e314-0065bb7225-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:49.736 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:49.736 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:49.736 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:49.736 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:49.736 DEBUG:boto:path=/vclfcn-19/obj?versionId=YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o 2024-02-01T10:27:49.737 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o 2024-02-01T10:27:49.737 DEBUG:boto:Method: GET 2024-02-01T10:27:49.737 DEBUG:boto:Path: /vclfcn-19/obj?versionId=YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o 2024-02-01T10:27:49.737 DEBUG:boto:Data: 2024-02-01T10:27:49.737 DEBUG:boto:Headers: {} 2024-02-01T10:27:49.737 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:49.737 DEBUG:boto:Port: 8001 2024-02-01T10:27:49.737 DEBUG:boto:Params: {} 2024-02-01T10:27:49.737 DEBUG:boto:Token: None 2024-02-01T10:27:49.737 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:49 GMT /vclfcn-19/obj?versionId=YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o 2024-02-01T10:27:49.738 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:3G8giVud/MtxEsWPQ5HlyvHUznw= 2024-02-01T10:27:49.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:3G8giVud/MtxEsWPQ5HlyvHUznw=', 'Content-Length': '0'} 2024-02-01T10:27:49.740 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('x-amz-version-id', 'YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009a78839dd2324fa9-0065bb7225-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:49.741 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:49.741 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:49.741 DEBUG:boto:path=/vclfcn-19/obj?versionId=YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o 2024-02-01T10:27:49.741 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o 2024-02-01T10:27:49.741 DEBUG:boto:Method: GET 2024-02-01T10:27:49.741 DEBUG:boto:Path: /vclfcn-19/obj?versionId=YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o 2024-02-01T10:27:49.741 DEBUG:boto:Data: 2024-02-01T10:27:49.741 DEBUG:boto:Headers: {} 2024-02-01T10:27:49.741 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:49.741 DEBUG:boto:Port: 8000 2024-02-01T10:27:49.742 DEBUG:boto:Params: {} 2024-02-01T10:27:49.742 DEBUG:boto:Token: None 2024-02-01T10:27:49.742 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:49 GMT /vclfcn-19/obj?versionId=YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o 2024-02-01T10:27:49.742 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:3G8giVud/MtxEsWPQ5HlyvHUznw= 2024-02-01T10:27:49.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:3G8giVud/MtxEsWPQ5HlyvHUznw=', 'Content-Length': '0'} 2024-02-01T10:27:49.745 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('x-amz-version-id', 'YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001a1c62b612b8070a-0065bb7225-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:49.745 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:49.745 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:49.745 DEBUG:boto:path=/vclfcn-19/obj?versionId=YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o 2024-02-01T10:27:49.745 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o 2024-02-01T10:27:49.745 DEBUG:boto:Method: HEAD 2024-02-01T10:27:49.746 DEBUG:boto:Path: /vclfcn-19/obj?versionId=YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o 2024-02-01T10:27:49.746 DEBUG:boto:Data: 2024-02-01T10:27:49.746 DEBUG:boto:Headers: {} 2024-02-01T10:27:49.746 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:49.746 DEBUG:boto:Port: 8001 2024-02-01T10:27:49.746 DEBUG:boto:Params: {} 2024-02-01T10:27:49.746 DEBUG:boto:Token: None 2024-02-01T10:27:49.746 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:49 GMT /vclfcn-19/obj?versionId=YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o 2024-02-01T10:27:49.746 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tfN6giT6oWgYSSn3m+Q+V1zXriM= 2024-02-01T10:27:49.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tfN6giT6oWgYSSn3m+Q+V1zXriM=', 'Content-Length': '0'} 2024-02-01T10:27:49.749 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('x-amz-version-id', 'YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cebd04c1fc160aa0-0065bb7225-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:49.749 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:49.749 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:49.749 DEBUG:boto:path=/vclfcn-19/obj?versionId=YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o 2024-02-01T10:27:49.749 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o 2024-02-01T10:27:49.750 DEBUG:boto:Method: HEAD 2024-02-01T10:27:49.750 DEBUG:boto:Path: /vclfcn-19/obj?versionId=YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o 2024-02-01T10:27:49.750 DEBUG:boto:Data: 2024-02-01T10:27:49.750 DEBUG:boto:Headers: {} 2024-02-01T10:27:49.750 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:49.750 DEBUG:boto:Port: 8000 2024-02-01T10:27:49.750 DEBUG:boto:Params: {} 2024-02-01T10:27:49.750 DEBUG:boto:Token: None 2024-02-01T10:27:49.750 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:49 GMT /vclfcn-19/obj?versionId=YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o 2024-02-01T10:27:49.750 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tfN6giT6oWgYSSn3m+Q+V1zXriM= 2024-02-01T10:27:49.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tfN6giT6oWgYSSn3m+Q+V1zXriM=', 'Content-Length': '0'} 2024-02-01T10:27:49.753 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('x-amz-version-id', 'YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003b6e17b803f6c371-0065bb7225-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:49.754 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:49.754 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:49.754 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:49.754 DEBUG:boto:path=/vclfcn-19/obj?versionId=YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o 2024-02-01T10:27:49.754 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o 2024-02-01T10:27:49.754 DEBUG:boto:Method: GET 2024-02-01T10:27:49.754 DEBUG:boto:Path: /vclfcn-19/obj?versionId=YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o 2024-02-01T10:27:49.754 DEBUG:boto:Data: 2024-02-01T10:27:49.754 DEBUG:boto:Headers: {} 2024-02-01T10:27:49.755 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:49.755 DEBUG:boto:Port: 8001 2024-02-01T10:27:49.755 DEBUG:boto:Params: {} 2024-02-01T10:27:49.755 DEBUG:boto:Token: None 2024-02-01T10:27:49.755 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:49 GMT /vclfcn-19/obj?versionId=YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o 2024-02-01T10:27:49.755 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:3G8giVud/MtxEsWPQ5HlyvHUznw= 2024-02-01T10:27:49.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:3G8giVud/MtxEsWPQ5HlyvHUznw=', 'Content-Length': '0'} 2024-02-01T10:27:49.758 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('x-amz-version-id', 'YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ba5b39138c96467a-0065bb7225-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:49.800 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:49.800 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:49.800 DEBUG:boto:path=/vclfcn-19/obj?versionId=YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o 2024-02-01T10:27:49.800 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o 2024-02-01T10:27:49.801 DEBUG:boto:Method: GET 2024-02-01T10:27:49.801 DEBUG:boto:Path: /vclfcn-19/obj?versionId=YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o 2024-02-01T10:27:49.801 DEBUG:boto:Data: 2024-02-01T10:27:49.801 DEBUG:boto:Headers: {} 2024-02-01T10:27:49.801 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:49.801 DEBUG:boto:Port: 8000 2024-02-01T10:27:49.801 DEBUG:boto:Params: {} 2024-02-01T10:27:49.801 DEBUG:boto:Token: None 2024-02-01T10:27:49.801 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:49 GMT /vclfcn-19/obj?versionId=YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o 2024-02-01T10:27:49.802 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:3G8giVud/MtxEsWPQ5HlyvHUznw= 2024-02-01T10:27:49.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:3G8giVud/MtxEsWPQ5HlyvHUznw=', 'Content-Length': '0'} 2024-02-01T10:27:49.805 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('x-amz-version-id', 'YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000af11c968353730d3-0065bb7225-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:49.805 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:49.805 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:49.805 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:49.805 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:49.806 DEBUG:boto:Method: GET 2024-02-01T10:27:49.806 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:49.806 DEBUG:boto:Data: 2024-02-01T10:27:49.806 DEBUG:boto:Headers: {} 2024-02-01T10:27:49.806 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:49.806 DEBUG:boto:Port: 8001 2024-02-01T10:27:49.806 DEBUG:boto:Params: {} 2024-02-01T10:27:49.806 DEBUG:boto:Token: None 2024-02-01T10:27:49.806 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:49 GMT /vclfcn-19/ 2024-02-01T10:27:49.806 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ycPXi051bWgnSuGizEh7kBZvlhY= 2024-02-01T10:27:49.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ycPXi051bWgnSuGizEh7kBZvlhY=', 'Content-Length': '0'} 2024-02-01T10:27:49.812 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e4dfd32ab1caa7a-0065bb7225-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:49.813 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:49.813 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:49.813 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:49.813 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:49.813 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:49.813 DEBUG:boto:Method: GET 2024-02-01T10:27:49.813 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:49.814 DEBUG:boto:Data: 2024-02-01T10:27:49.814 DEBUG:boto:Headers: {} 2024-02-01T10:27:49.814 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:49.814 DEBUG:boto:Port: 8000 2024-02-01T10:27:49.814 DEBUG:boto:Params: {} 2024-02-01T10:27:49.814 DEBUG:boto:Token: None 2024-02-01T10:27:49.814 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:49 GMT /vclfcn-19/ 2024-02-01T10:27:49.814 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ycPXi051bWgnSuGizEh7kBZvlhY= 2024-02-01T10:27:49.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ycPXi051bWgnSuGizEh7kBZvlhY=', 'Content-Length': '0'} 2024-02-01T10:27:49.819 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6b8ebc2921c0a0c-0065bb7225-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:49.819 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:49.820 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:49.820 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:49.820 DEBUG:boto:Method: HEAD 2024-02-01T10:27:49.820 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:49.820 DEBUG:boto:Data: 2024-02-01T10:27:49.820 DEBUG:boto:Headers: {} 2024-02-01T10:27:49.820 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:49.820 DEBUG:boto:Port: 8001 2024-02-01T10:27:49.820 DEBUG:boto:Params: {} 2024-02-01T10:27:49.820 DEBUG:boto:Token: None 2024-02-01T10:27:49.821 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:49 GMT /vclfcn-19/obj 2024-02-01T10:27:49.821 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5vY2I3aFYDj18RtcSWgpyW3WwOE= 2024-02-01T10:27:49.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5vY2I3aFYDj18RtcSWgpyW3WwOE=', 'Content-Length': '0'} 2024-02-01T10:27:49.826 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000012c21a7ebfaafa9a-0065bb7225-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:49.827 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:49.827 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:49.827 DEBUG:boto:Method: HEAD 2024-02-01T10:27:49.827 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:49.827 DEBUG:boto:Data: 2024-02-01T10:27:49.827 DEBUG:boto:Headers: {} 2024-02-01T10:27:49.827 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:49.827 DEBUG:boto:Port: 8000 2024-02-01T10:27:49.827 DEBUG:boto:Params: {} 2024-02-01T10:27:49.828 DEBUG:boto:Token: None 2024-02-01T10:27:49.828 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:49 GMT /vclfcn-19/obj 2024-02-01T10:27:49.828 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5vY2I3aFYDj18RtcSWgpyW3WwOE= 2024-02-01T10:27:49.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5vY2I3aFYDj18RtcSWgpyW3WwOE=', 'Content-Length': '0'} 2024-02-01T10:27:49.834 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b1c9c27d22daf2af-0065bb7225-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:49.834 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:49.834 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:49.834 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:49.834 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:49.834 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:49.834 DEBUG:boto:Method: GET 2024-02-01T10:27:49.835 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:49.835 DEBUG:boto:Data: 2024-02-01T10:27:49.835 DEBUG:boto:Headers: {} 2024-02-01T10:27:49.835 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:49.835 DEBUG:boto:Port: 8001 2024-02-01T10:27:49.835 DEBUG:boto:Params: {} 2024-02-01T10:27:49.835 DEBUG:boto:Token: None 2024-02-01T10:27:49.835 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:49 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:49.835 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8Qt5QRDoKGE4iVEr73USK9IGPYs= 2024-02-01T10:27:49.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8Qt5QRDoKGE4iVEr73USK9IGPYs=', 'Content-Length': '0'} 2024-02-01T10:27:49.838 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ea79c5851749ea17-0065bb7225-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:49.880 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:49.880 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:49.880 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:49.880 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:49.881 DEBUG:boto:Method: GET 2024-02-01T10:27:49.881 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:49.881 DEBUG:boto:Data: 2024-02-01T10:27:49.881 DEBUG:boto:Headers: {} 2024-02-01T10:27:49.881 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:49.881 DEBUG:boto:Port: 8000 2024-02-01T10:27:49.881 DEBUG:boto:Params: {} 2024-02-01T10:27:49.881 DEBUG:boto:Token: None 2024-02-01T10:27:49.881 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:49 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:49.881 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8Qt5QRDoKGE4iVEr73USK9IGPYs= 2024-02-01T10:27:49.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8Qt5QRDoKGE4iVEr73USK9IGPYs=', 'Content-Length': '0'} 2024-02-01T10:27:49.885 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000075cf89f4c19800e8-0065bb7225-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:49.928 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:49.928 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:49.928 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:49.929 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:49.929 DEBUG:boto:Method: GET 2024-02-01T10:27:49.929 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:49.929 DEBUG:boto:Data: 2024-02-01T10:27:49.929 DEBUG:boto:Headers: {} 2024-02-01T10:27:49.929 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:49.929 DEBUG:boto:Port: 8001 2024-02-01T10:27:49.929 DEBUG:boto:Params: {} 2024-02-01T10:27:49.929 DEBUG:boto:Token: None 2024-02-01T10:27:49.929 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:49 GMT /vclfcn-19/ 2024-02-01T10:27:49.930 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ycPXi051bWgnSuGizEh7kBZvlhY= 2024-02-01T10:27:49.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ycPXi051bWgnSuGizEh7kBZvlhY=', 'Content-Length': '0'} 2024-02-01T10:27:49.934 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f8a03cd1dd85870-0065bb7225-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:49.935 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:49.935 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:49.935 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:49.935 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:49.935 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:49.936 DEBUG:boto:Method: GET 2024-02-01T10:27:49.936 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:49.936 DEBUG:boto:Data: 2024-02-01T10:27:49.936 DEBUG:boto:Headers: {} 2024-02-01T10:27:49.936 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:49.936 DEBUG:boto:Port: 8000 2024-02-01T10:27:49.936 DEBUG:boto:Params: {} 2024-02-01T10:27:49.936 DEBUG:boto:Token: None 2024-02-01T10:27:49.936 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:49 GMT /vclfcn-19/ 2024-02-01T10:27:49.936 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ycPXi051bWgnSuGizEh7kBZvlhY= 2024-02-01T10:27:49.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ycPXi051bWgnSuGizEh7kBZvlhY=', 'Content-Length': '0'} 2024-02-01T10:27:49.942 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e860de1ebdd1779-0065bb7225-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:49.942 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:49.942 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:49.942 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:49.943 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:49.943 DEBUG:boto:path=/vclfcn-19/obj?versionId=B5jd89.SLRVLOAPcXqackukLvQnZY1B 2024-02-01T10:27:49.943 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=B5jd89.SLRVLOAPcXqackukLvQnZY1B 2024-02-01T10:27:49.943 DEBUG:boto:Method: GET 2024-02-01T10:27:49.943 DEBUG:boto:Path: /vclfcn-19/obj?versionId=B5jd89.SLRVLOAPcXqackukLvQnZY1B 2024-02-01T10:27:49.943 DEBUG:boto:Data: 2024-02-01T10:27:49.943 DEBUG:boto:Headers: {} 2024-02-01T10:27:49.943 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:49.943 DEBUG:boto:Port: 8001 2024-02-01T10:27:49.943 DEBUG:boto:Params: {} 2024-02-01T10:27:49.944 DEBUG:boto:Token: None 2024-02-01T10:27:49.944 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:49 GMT /vclfcn-19/obj?versionId=B5jd89.SLRVLOAPcXqackukLvQnZY1B 2024-02-01T10:27:49.944 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:e+56uC6tEuP6PzoapZWO64ED8z8= 2024-02-01T10:27:49.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:e+56uC6tEuP6PzoapZWO64ED8z8=', 'Content-Length': '0'} 2024-02-01T10:27:49.947 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('x-amz-version-id', 'B5jd89.SLRVLOAPcXqackukLvQnZY1B'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009d7f1f2a13b42ee1-0065bb7225-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:49.948 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:49.948 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:49.948 DEBUG:boto:path=/vclfcn-19/obj?versionId=B5jd89.SLRVLOAPcXqackukLvQnZY1B 2024-02-01T10:27:49.948 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=B5jd89.SLRVLOAPcXqackukLvQnZY1B 2024-02-01T10:27:49.948 DEBUG:boto:Method: GET 2024-02-01T10:27:49.948 DEBUG:boto:Path: /vclfcn-19/obj?versionId=B5jd89.SLRVLOAPcXqackukLvQnZY1B 2024-02-01T10:27:49.948 DEBUG:boto:Data: 2024-02-01T10:27:49.948 DEBUG:boto:Headers: {} 2024-02-01T10:27:49.948 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:49.949 DEBUG:boto:Port: 8000 2024-02-01T10:27:49.949 DEBUG:boto:Params: {} 2024-02-01T10:27:49.949 DEBUG:boto:Token: None 2024-02-01T10:27:49.949 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:49 GMT /vclfcn-19/obj?versionId=B5jd89.SLRVLOAPcXqackukLvQnZY1B 2024-02-01T10:27:49.949 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:e+56uC6tEuP6PzoapZWO64ED8z8= 2024-02-01T10:27:49.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:e+56uC6tEuP6PzoapZWO64ED8z8=', 'Content-Length': '0'} 2024-02-01T10:27:49.952 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('x-amz-version-id', 'B5jd89.SLRVLOAPcXqackukLvQnZY1B'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002a7f9b2315467001-0065bb7225-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:49.952 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:49.952 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:49.952 DEBUG:boto:path=/vclfcn-19/obj?versionId=B5jd89.SLRVLOAPcXqackukLvQnZY1B 2024-02-01T10:27:49.952 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=B5jd89.SLRVLOAPcXqackukLvQnZY1B 2024-02-01T10:27:49.953 DEBUG:boto:Method: HEAD 2024-02-01T10:27:49.953 DEBUG:boto:Path: /vclfcn-19/obj?versionId=B5jd89.SLRVLOAPcXqackukLvQnZY1B 2024-02-01T10:27:49.953 DEBUG:boto:Data: 2024-02-01T10:27:49.953 DEBUG:boto:Headers: {} 2024-02-01T10:27:49.953 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:49.953 DEBUG:boto:Port: 8001 2024-02-01T10:27:49.953 DEBUG:boto:Params: {} 2024-02-01T10:27:49.953 DEBUG:boto:Token: None 2024-02-01T10:27:49.953 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:49 GMT /vclfcn-19/obj?versionId=B5jd89.SLRVLOAPcXqackukLvQnZY1B 2024-02-01T10:27:49.953 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zvcpP9803gVwL3jlfDBqNf6T0no= 2024-02-01T10:27:49.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zvcpP9803gVwL3jlfDBqNf6T0no=', 'Content-Length': '0'} 2024-02-01T10:27:49.957 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('x-amz-version-id', 'B5jd89.SLRVLOAPcXqackukLvQnZY1B'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000de8630fefde01b73-0065bb7225-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:49.957 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:49.957 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:49.957 DEBUG:boto:path=/vclfcn-19/obj?versionId=B5jd89.SLRVLOAPcXqackukLvQnZY1B 2024-02-01T10:27:49.957 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=B5jd89.SLRVLOAPcXqackukLvQnZY1B 2024-02-01T10:27:49.957 DEBUG:boto:Method: HEAD 2024-02-01T10:27:49.957 DEBUG:boto:Path: /vclfcn-19/obj?versionId=B5jd89.SLRVLOAPcXqackukLvQnZY1B 2024-02-01T10:27:49.957 DEBUG:boto:Data: 2024-02-01T10:27:49.957 DEBUG:boto:Headers: {} 2024-02-01T10:27:49.958 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:49.958 DEBUG:boto:Port: 8000 2024-02-01T10:27:49.958 DEBUG:boto:Params: {} 2024-02-01T10:27:49.958 DEBUG:boto:Token: None 2024-02-01T10:27:49.958 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:49 GMT /vclfcn-19/obj?versionId=B5jd89.SLRVLOAPcXqackukLvQnZY1B 2024-02-01T10:27:49.958 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zvcpP9803gVwL3jlfDBqNf6T0no= 2024-02-01T10:27:49.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zvcpP9803gVwL3jlfDBqNf6T0no=', 'Content-Length': '0'} 2024-02-01T10:27:49.961 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('x-amz-version-id', 'B5jd89.SLRVLOAPcXqackukLvQnZY1B'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004e705ddbac83cb00-0065bb7225-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:49.961 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:49.962 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:49.962 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:49.962 DEBUG:boto:path=/vclfcn-19/obj?versionId=B5jd89.SLRVLOAPcXqackukLvQnZY1B 2024-02-01T10:27:49.962 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=B5jd89.SLRVLOAPcXqackukLvQnZY1B 2024-02-01T10:27:49.962 DEBUG:boto:Method: GET 2024-02-01T10:27:49.962 DEBUG:boto:Path: /vclfcn-19/obj?versionId=B5jd89.SLRVLOAPcXqackukLvQnZY1B 2024-02-01T10:27:49.962 DEBUG:boto:Data: 2024-02-01T10:27:49.963 DEBUG:boto:Headers: {} 2024-02-01T10:27:49.963 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:49.963 DEBUG:boto:Port: 8001 2024-02-01T10:27:49.963 DEBUG:boto:Params: {} 2024-02-01T10:27:49.963 DEBUG:boto:Token: None 2024-02-01T10:27:49.963 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:49 GMT /vclfcn-19/obj?versionId=B5jd89.SLRVLOAPcXqackukLvQnZY1B 2024-02-01T10:27:49.964 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:e+56uC6tEuP6PzoapZWO64ED8z8= 2024-02-01T10:27:49.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:e+56uC6tEuP6PzoapZWO64ED8z8=', 'Content-Length': '0'} 2024-02-01T10:27:49.967 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('x-amz-version-id', 'B5jd89.SLRVLOAPcXqackukLvQnZY1B'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000070fa4b5c3cebda5e-0065bb7225-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:50.008 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:50.008 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:50.008 DEBUG:boto:path=/vclfcn-19/obj?versionId=B5jd89.SLRVLOAPcXqackukLvQnZY1B 2024-02-01T10:27:50.008 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=B5jd89.SLRVLOAPcXqackukLvQnZY1B 2024-02-01T10:27:50.009 DEBUG:boto:Method: GET 2024-02-01T10:27:50.009 DEBUG:boto:Path: /vclfcn-19/obj?versionId=B5jd89.SLRVLOAPcXqackukLvQnZY1B 2024-02-01T10:27:50.009 DEBUG:boto:Data: 2024-02-01T10:27:50.009 DEBUG:boto:Headers: {} 2024-02-01T10:27:50.009 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:50.009 DEBUG:boto:Port: 8000 2024-02-01T10:27:50.009 DEBUG:boto:Params: {} 2024-02-01T10:27:50.009 DEBUG:boto:Token: None 2024-02-01T10:27:50.009 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:50 GMT /vclfcn-19/obj?versionId=B5jd89.SLRVLOAPcXqackukLvQnZY1B 2024-02-01T10:27:50.010 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nCxFmTJHbhJGtRkLEqbSHEC+Nbs= 2024-02-01T10:27:50.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nCxFmTJHbhJGtRkLEqbSHEC+Nbs=', 'Content-Length': '0'} 2024-02-01T10:27:50.012 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('x-amz-version-id', 'B5jd89.SLRVLOAPcXqackukLvQnZY1B'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000052c8dbfa36a2216c-0065bb7226-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:50.013 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:50.013 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:50.013 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:50.013 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:50.013 DEBUG:boto:Method: GET 2024-02-01T10:27:50.013 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:50.013 DEBUG:boto:Data: 2024-02-01T10:27:50.013 DEBUG:boto:Headers: {} 2024-02-01T10:27:50.013 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:50.014 DEBUG:boto:Port: 8001 2024-02-01T10:27:50.014 DEBUG:boto:Params: {} 2024-02-01T10:27:50.014 DEBUG:boto:Token: None 2024-02-01T10:27:50.014 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:50 GMT /vclfcn-19/ 2024-02-01T10:27:50.014 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FwK72WG2W70zMJxgzvYKnS9H2tI= 2024-02-01T10:27:50.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FwK72WG2W70zMJxgzvYKnS9H2tI=', 'Content-Length': '0'} 2024-02-01T10:27:50.019 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067274512e9b4628b-0065bb7226-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:50.064 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:50.065 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:50.065 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:50.065 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:50.065 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:50.065 DEBUG:boto:Method: GET 2024-02-01T10:27:50.065 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:50.065 DEBUG:boto:Data: 2024-02-01T10:27:50.065 DEBUG:boto:Headers: {} 2024-02-01T10:27:50.065 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:50.066 DEBUG:boto:Port: 8000 2024-02-01T10:27:50.066 DEBUG:boto:Params: {} 2024-02-01T10:27:50.066 DEBUG:boto:Token: None 2024-02-01T10:27:50.066 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:50 GMT /vclfcn-19/ 2024-02-01T10:27:50.066 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FwK72WG2W70zMJxgzvYKnS9H2tI= 2024-02-01T10:27:50.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FwK72WG2W70zMJxgzvYKnS9H2tI=', 'Content-Length': '0'} 2024-02-01T10:27:50.072 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042e597abd2fbe549-0065bb7226-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:50.072 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:50.072 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:50.072 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:50.072 DEBUG:boto:Method: HEAD 2024-02-01T10:27:50.072 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:50.073 DEBUG:boto:Data: 2024-02-01T10:27:50.073 DEBUG:boto:Headers: {} 2024-02-01T10:27:50.073 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:50.073 DEBUG:boto:Port: 8001 2024-02-01T10:27:50.073 DEBUG:boto:Params: {} 2024-02-01T10:27:50.073 DEBUG:boto:Token: None 2024-02-01T10:27:50.073 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:50 GMT /vclfcn-19/obj 2024-02-01T10:27:50.073 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:hjc0ueHw7/hSpshU8J+4Q4YhFzo= 2024-02-01T10:27:50.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:hjc0ueHw7/hSpshU8J+4Q4YhFzo=', 'Content-Length': '0'} 2024-02-01T10:27:50.079 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006ee7d47430bc7f88-0065bb7226-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:50.079 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:50.079 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:50.079 DEBUG:boto:Method: HEAD 2024-02-01T10:27:50.080 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:50.080 DEBUG:boto:Data: 2024-02-01T10:27:50.080 DEBUG:boto:Headers: {} 2024-02-01T10:27:50.080 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:50.080 DEBUG:boto:Port: 8000 2024-02-01T10:27:50.080 DEBUG:boto:Params: {} 2024-02-01T10:27:50.080 DEBUG:boto:Token: None 2024-02-01T10:27:50.080 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:50 GMT /vclfcn-19/obj 2024-02-01T10:27:50.080 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:hjc0ueHw7/hSpshU8J+4Q4YhFzo= 2024-02-01T10:27:50.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:hjc0ueHw7/hSpshU8J+4Q4YhFzo=', 'Content-Length': '0'} 2024-02-01T10:27:50.086 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000460ae8fdd4626037-0065bb7226-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:50.086 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:50.086 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:50.087 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:50.087 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:50.087 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:50.087 DEBUG:boto:Method: GET 2024-02-01T10:27:50.087 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:50.087 DEBUG:boto:Data: 2024-02-01T10:27:50.087 DEBUG:boto:Headers: {} 2024-02-01T10:27:50.087 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:50.087 DEBUG:boto:Port: 8001 2024-02-01T10:27:50.087 DEBUG:boto:Params: {} 2024-02-01T10:27:50.087 DEBUG:boto:Token: None 2024-02-01T10:27:50.088 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:50 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:50.088 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:UWEAaAH7JjdGTFY3BefzJyf4eZE= 2024-02-01T10:27:50.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:UWEAaAH7JjdGTFY3BefzJyf4eZE=', 'Content-Length': '0'} 2024-02-01T10:27:50.091 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b80b1c3962aa5f17-0065bb7226-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:50.132 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:50.132 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:50.132 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:50.133 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:50.133 DEBUG:boto:Method: GET 2024-02-01T10:27:50.133 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:50.133 DEBUG:boto:Data: 2024-02-01T10:27:50.133 DEBUG:boto:Headers: {} 2024-02-01T10:27:50.133 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:50.133 DEBUG:boto:Port: 8000 2024-02-01T10:27:50.133 DEBUG:boto:Params: {} 2024-02-01T10:27:50.133 DEBUG:boto:Token: None 2024-02-01T10:27:50.133 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:50 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:50.134 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:UWEAaAH7JjdGTFY3BefzJyf4eZE= 2024-02-01T10:27:50.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:UWEAaAH7JjdGTFY3BefzJyf4eZE=', 'Content-Length': '0'} 2024-02-01T10:27:50.137 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000434554ed6c811f1a-0065bb7226-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:50.180 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:50.180 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:50.180 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:50.181 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:50.181 DEBUG:boto:Method: GET 2024-02-01T10:27:50.181 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:50.181 DEBUG:boto:Data: 2024-02-01T10:27:50.181 DEBUG:boto:Headers: {} 2024-02-01T10:27:50.181 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:50.181 DEBUG:boto:Port: 8001 2024-02-01T10:27:50.181 DEBUG:boto:Params: {} 2024-02-01T10:27:50.181 DEBUG:boto:Token: None 2024-02-01T10:27:50.181 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:50 GMT /vclfcn-19/ 2024-02-01T10:27:50.182 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FwK72WG2W70zMJxgzvYKnS9H2tI= 2024-02-01T10:27:50.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FwK72WG2W70zMJxgzvYKnS9H2tI=', 'Content-Length': '0'} 2024-02-01T10:27:50.187 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ad2f63c74fce29f-0065bb7226-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:50.228 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:50.229 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:50.229 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:50.229 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:50.229 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:50.229 DEBUG:boto:Method: GET 2024-02-01T10:27:50.229 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:50.229 DEBUG:boto:Data: 2024-02-01T10:27:50.229 DEBUG:boto:Headers: {} 2024-02-01T10:27:50.229 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:50.229 DEBUG:boto:Port: 8000 2024-02-01T10:27:50.230 DEBUG:boto:Params: {} 2024-02-01T10:27:50.230 DEBUG:boto:Token: None 2024-02-01T10:27:50.230 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:50 GMT /vclfcn-19/ 2024-02-01T10:27:50.230 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FwK72WG2W70zMJxgzvYKnS9H2tI= 2024-02-01T10:27:50.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FwK72WG2W70zMJxgzvYKnS9H2tI=', 'Content-Length': '0'} 2024-02-01T10:27:50.236 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad8c258561b8e84b-0065bb7226-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:50.276 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:50.276 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:50.277 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:50.277 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:50.277 DEBUG:boto:path=/vclfcn-19/obj?versionId=rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ 2024-02-01T10:27:50.277 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ 2024-02-01T10:27:50.277 DEBUG:boto:Method: GET 2024-02-01T10:27:50.277 DEBUG:boto:Path: /vclfcn-19/obj?versionId=rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ 2024-02-01T10:27:50.277 DEBUG:boto:Data: 2024-02-01T10:27:50.277 DEBUG:boto:Headers: {} 2024-02-01T10:27:50.277 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:50.278 DEBUG:boto:Port: 8001 2024-02-01T10:27:50.278 DEBUG:boto:Params: {} 2024-02-01T10:27:50.278 DEBUG:boto:Token: None 2024-02-01T10:27:50.278 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:50 GMT /vclfcn-19/obj?versionId=rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ 2024-02-01T10:27:50.278 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ntAcBuHOVKdXu+P7S5sB6hMfLjg= 2024-02-01T10:27:50.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ntAcBuHOVKdXu+P7S5sB6hMfLjg=', 'Content-Length': '0'} 2024-02-01T10:27:50.281 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('x-amz-version-id', 'rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f931504b445baa11-0065bb7226-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:50.281 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:50.281 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:50.282 DEBUG:boto:path=/vclfcn-19/obj?versionId=rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ 2024-02-01T10:27:50.282 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ 2024-02-01T10:27:50.282 DEBUG:boto:Method: GET 2024-02-01T10:27:50.282 DEBUG:boto:Path: /vclfcn-19/obj?versionId=rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ 2024-02-01T10:27:50.282 DEBUG:boto:Data: 2024-02-01T10:27:50.282 DEBUG:boto:Headers: {} 2024-02-01T10:27:50.282 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:50.282 DEBUG:boto:Port: 8000 2024-02-01T10:27:50.282 DEBUG:boto:Params: {} 2024-02-01T10:27:50.282 DEBUG:boto:Token: None 2024-02-01T10:27:50.283 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:50 GMT /vclfcn-19/obj?versionId=rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ 2024-02-01T10:27:50.283 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ntAcBuHOVKdXu+P7S5sB6hMfLjg= 2024-02-01T10:27:50.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ntAcBuHOVKdXu+P7S5sB6hMfLjg=', 'Content-Length': '0'} 2024-02-01T10:27:50.285 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('x-amz-version-id', 'rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fa1835052e24f6c2-0065bb7226-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:50.328 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:50.328 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:50.328 DEBUG:boto:path=/vclfcn-19/obj?versionId=rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ 2024-02-01T10:27:50.328 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ 2024-02-01T10:27:50.328 DEBUG:boto:Method: HEAD 2024-02-01T10:27:50.329 DEBUG:boto:Path: /vclfcn-19/obj?versionId=rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ 2024-02-01T10:27:50.329 DEBUG:boto:Data: 2024-02-01T10:27:50.329 DEBUG:boto:Headers: {} 2024-02-01T10:27:50.329 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:50.329 DEBUG:boto:Port: 8001 2024-02-01T10:27:50.329 DEBUG:boto:Params: {} 2024-02-01T10:27:50.329 DEBUG:boto:Token: None 2024-02-01T10:27:50.329 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:50 GMT /vclfcn-19/obj?versionId=rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ 2024-02-01T10:27:50.329 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:p7KEBAxoAVJairrcg6CIaNXqV9s= 2024-02-01T10:27:50.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:p7KEBAxoAVJairrcg6CIaNXqV9s=', 'Content-Length': '0'} 2024-02-01T10:27:50.332 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('x-amz-version-id', 'rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008bbb0cb19b92daad-0065bb7226-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:50.332 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:50.333 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:50.333 DEBUG:boto:path=/vclfcn-19/obj?versionId=rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ 2024-02-01T10:27:50.333 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ 2024-02-01T10:27:50.333 DEBUG:boto:Method: HEAD 2024-02-01T10:27:50.333 DEBUG:boto:Path: /vclfcn-19/obj?versionId=rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ 2024-02-01T10:27:50.333 DEBUG:boto:Data: 2024-02-01T10:27:50.333 DEBUG:boto:Headers: {} 2024-02-01T10:27:50.333 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:50.333 DEBUG:boto:Port: 8000 2024-02-01T10:27:50.333 DEBUG:boto:Params: {} 2024-02-01T10:27:50.334 DEBUG:boto:Token: None 2024-02-01T10:27:50.334 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:50 GMT /vclfcn-19/obj?versionId=rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ 2024-02-01T10:27:50.334 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:p7KEBAxoAVJairrcg6CIaNXqV9s= 2024-02-01T10:27:50.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:p7KEBAxoAVJairrcg6CIaNXqV9s=', 'Content-Length': '0'} 2024-02-01T10:27:50.337 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('x-amz-version-id', 'rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000005a2da29f6b6d592-0065bb7226-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:50.337 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:50.337 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:50.337 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:50.337 DEBUG:boto:path=/vclfcn-19/obj?versionId=rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ 2024-02-01T10:27:50.338 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ 2024-02-01T10:27:50.338 DEBUG:boto:Method: GET 2024-02-01T10:27:50.338 DEBUG:boto:Path: /vclfcn-19/obj?versionId=rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ 2024-02-01T10:27:50.338 DEBUG:boto:Data: 2024-02-01T10:27:50.338 DEBUG:boto:Headers: {} 2024-02-01T10:27:50.338 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:50.338 DEBUG:boto:Port: 8001 2024-02-01T10:27:50.338 DEBUG:boto:Params: {} 2024-02-01T10:27:50.338 DEBUG:boto:Token: None 2024-02-01T10:27:50.338 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:50 GMT /vclfcn-19/obj?versionId=rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ 2024-02-01T10:27:50.339 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ntAcBuHOVKdXu+P7S5sB6hMfLjg= 2024-02-01T10:27:50.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ntAcBuHOVKdXu+P7S5sB6hMfLjg=', 'Content-Length': '0'} 2024-02-01T10:27:50.341 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('x-amz-version-id', 'rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002467885100af830d-0065bb7226-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:50.341 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:50.342 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:50.342 DEBUG:boto:path=/vclfcn-19/obj?versionId=rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ 2024-02-01T10:27:50.342 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ 2024-02-01T10:27:50.342 DEBUG:boto:Method: GET 2024-02-01T10:27:50.342 DEBUG:boto:Path: /vclfcn-19/obj?versionId=rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ 2024-02-01T10:27:50.342 DEBUG:boto:Data: 2024-02-01T10:27:50.342 DEBUG:boto:Headers: {} 2024-02-01T10:27:50.342 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:50.342 DEBUG:boto:Port: 8000 2024-02-01T10:27:50.342 DEBUG:boto:Params: {} 2024-02-01T10:27:50.343 DEBUG:boto:Token: None 2024-02-01T10:27:50.343 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:50 GMT /vclfcn-19/obj?versionId=rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ 2024-02-01T10:27:50.343 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ntAcBuHOVKdXu+P7S5sB6hMfLjg= 2024-02-01T10:27:50.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ntAcBuHOVKdXu+P7S5sB6hMfLjg=', 'Content-Length': '0'} 2024-02-01T10:27:50.345 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('x-amz-version-id', 'rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000059e8361b50d11944-0065bb7226-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:50.392 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:50.392 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:50.392 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:50.393 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:50.393 DEBUG:boto:Method: GET 2024-02-01T10:27:50.393 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:50.393 DEBUG:boto:Data: 2024-02-01T10:27:50.393 DEBUG:boto:Headers: {} 2024-02-01T10:27:50.393 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:50.393 DEBUG:boto:Port: 8001 2024-02-01T10:27:50.393 DEBUG:boto:Params: {} 2024-02-01T10:27:50.393 DEBUG:boto:Token: None 2024-02-01T10:27:50.393 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:50 GMT /vclfcn-19/ 2024-02-01T10:27:50.394 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FwK72WG2W70zMJxgzvYKnS9H2tI= 2024-02-01T10:27:50.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FwK72WG2W70zMJxgzvYKnS9H2tI=', 'Content-Length': '0'} 2024-02-01T10:27:50.398 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7252e88f7935e3e-0065bb7226-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:50.399 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:50.399 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:50.399 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:50.400 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:50.400 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:50.400 DEBUG:boto:Method: GET 2024-02-01T10:27:50.400 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:50.400 DEBUG:boto:Data: 2024-02-01T10:27:50.400 DEBUG:boto:Headers: {} 2024-02-01T10:27:50.400 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:50.400 DEBUG:boto:Port: 8000 2024-02-01T10:27:50.400 DEBUG:boto:Params: {} 2024-02-01T10:27:50.400 DEBUG:boto:Token: None 2024-02-01T10:27:50.401 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:50 GMT /vclfcn-19/ 2024-02-01T10:27:50.401 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FwK72WG2W70zMJxgzvYKnS9H2tI= 2024-02-01T10:27:50.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FwK72WG2W70zMJxgzvYKnS9H2tI=', 'Content-Length': '0'} 2024-02-01T10:27:50.406 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c344a9da94d2cb9-0065bb7226-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:50.406 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:50.406 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:50.407 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:50.407 DEBUG:boto:Method: HEAD 2024-02-01T10:27:50.407 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:50.407 DEBUG:boto:Data: 2024-02-01T10:27:50.407 DEBUG:boto:Headers: {} 2024-02-01T10:27:50.407 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:50.407 DEBUG:boto:Port: 8001 2024-02-01T10:27:50.407 DEBUG:boto:Params: {} 2024-02-01T10:27:50.407 DEBUG:boto:Token: None 2024-02-01T10:27:50.407 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:50 GMT /vclfcn-19/obj 2024-02-01T10:27:50.408 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:hjc0ueHw7/hSpshU8J+4Q4YhFzo= 2024-02-01T10:27:50.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:hjc0ueHw7/hSpshU8J+4Q4YhFzo=', 'Content-Length': '0'} 2024-02-01T10:27:50.413 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cf1432e1b1e37e94-0065bb7226-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:50.414 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:50.414 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:50.414 DEBUG:boto:Method: HEAD 2024-02-01T10:27:50.414 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:50.414 DEBUG:boto:Data: 2024-02-01T10:27:50.414 DEBUG:boto:Headers: {} 2024-02-01T10:27:50.414 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:50.414 DEBUG:boto:Port: 8000 2024-02-01T10:27:50.415 DEBUG:boto:Params: {} 2024-02-01T10:27:50.415 DEBUG:boto:Token: None 2024-02-01T10:27:50.415 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:50 GMT /vclfcn-19/obj 2024-02-01T10:27:50.415 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:hjc0ueHw7/hSpshU8J+4Q4YhFzo= 2024-02-01T10:27:50.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:hjc0ueHw7/hSpshU8J+4Q4YhFzo=', 'Content-Length': '0'} 2024-02-01T10:27:50.421 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cb1ee98a8874c25f-0065bb7226-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:50.421 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:50.421 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:50.421 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:50.421 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:50.421 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:50.421 DEBUG:boto:Method: GET 2024-02-01T10:27:50.422 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:50.422 DEBUG:boto:Data: 2024-02-01T10:27:50.422 DEBUG:boto:Headers: {} 2024-02-01T10:27:50.422 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:50.422 DEBUG:boto:Port: 8001 2024-02-01T10:27:50.422 DEBUG:boto:Params: {} 2024-02-01T10:27:50.422 DEBUG:boto:Token: None 2024-02-01T10:27:50.422 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:50 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:50.422 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:UWEAaAH7JjdGTFY3BefzJyf4eZE= 2024-02-01T10:27:50.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:UWEAaAH7JjdGTFY3BefzJyf4eZE=', 'Content-Length': '0'} 2024-02-01T10:27:50.425 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e97eff9b7b863717-0065bb7226-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:50.472 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:50.472 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:50.472 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:50.472 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:50.473 DEBUG:boto:Method: GET 2024-02-01T10:27:50.473 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:50.473 DEBUG:boto:Data: 2024-02-01T10:27:50.473 DEBUG:boto:Headers: {} 2024-02-01T10:27:50.473 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:50.473 DEBUG:boto:Port: 8000 2024-02-01T10:27:50.473 DEBUG:boto:Params: {} 2024-02-01T10:27:50.473 DEBUG:boto:Token: None 2024-02-01T10:27:50.473 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:50 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:50.474 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:UWEAaAH7JjdGTFY3BefzJyf4eZE= 2024-02-01T10:27:50.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:UWEAaAH7JjdGTFY3BefzJyf4eZE=', 'Content-Length': '0'} 2024-02-01T10:27:50.477 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003ba22a58b88c8931-0065bb7226-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:50.524 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:50.524 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:50.524 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:50.525 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:50.525 DEBUG:boto:Method: GET 2024-02-01T10:27:50.525 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:50.525 DEBUG:boto:Data: 2024-02-01T10:27:50.525 DEBUG:boto:Headers: {} 2024-02-01T10:27:50.525 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:50.525 DEBUG:boto:Port: 8001 2024-02-01T10:27:50.525 DEBUG:boto:Params: {} 2024-02-01T10:27:50.525 DEBUG:boto:Token: None 2024-02-01T10:27:50.525 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:50 GMT /vclfcn-19/ 2024-02-01T10:27:50.526 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FwK72WG2W70zMJxgzvYKnS9H2tI= 2024-02-01T10:27:50.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FwK72WG2W70zMJxgzvYKnS9H2tI=', 'Content-Length': '0'} 2024-02-01T10:27:50.531 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007dcee1873b3750f5-0065bb7226-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:50.531 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:50.531 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:50.532 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:50.532 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:50.532 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:50.532 DEBUG:boto:Method: GET 2024-02-01T10:27:50.532 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:50.532 DEBUG:boto:Data: 2024-02-01T10:27:50.532 DEBUG:boto:Headers: {} 2024-02-01T10:27:50.532 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:50.532 DEBUG:boto:Port: 8000 2024-02-01T10:27:50.533 DEBUG:boto:Params: {} 2024-02-01T10:27:50.533 DEBUG:boto:Token: None 2024-02-01T10:27:50.533 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:50 GMT /vclfcn-19/ 2024-02-01T10:27:50.533 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FwK72WG2W70zMJxgzvYKnS9H2tI= 2024-02-01T10:27:50.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FwK72WG2W70zMJxgzvYKnS9H2tI=', 'Content-Length': '0'} 2024-02-01T10:27:50.538 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db4235fac2c67cf2-0065bb7226-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:50.539 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:50.539 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:50.539 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:50.540 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:50.540 DEBUG:boto:path=/vclfcn-19/obj?versionId=s-qxjv7TkMdxeC61HL2kxUqODSwa-1I 2024-02-01T10:27:50.540 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=s-qxjv7TkMdxeC61HL2kxUqODSwa-1I 2024-02-01T10:27:50.540 DEBUG:boto:Method: GET 2024-02-01T10:27:50.541 DEBUG:boto:Path: /vclfcn-19/obj?versionId=s-qxjv7TkMdxeC61HL2kxUqODSwa-1I 2024-02-01T10:27:50.541 DEBUG:boto:Data: 2024-02-01T10:27:50.541 DEBUG:boto:Headers: {} 2024-02-01T10:27:50.541 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:50.541 DEBUG:boto:Port: 8001 2024-02-01T10:27:50.541 DEBUG:boto:Params: {} 2024-02-01T10:27:50.542 DEBUG:boto:Token: None 2024-02-01T10:27:50.542 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:50 GMT /vclfcn-19/obj?versionId=s-qxjv7TkMdxeC61HL2kxUqODSwa-1I 2024-02-01T10:27:50.542 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:EBmEu5wc5Kho93TH5/YtUVz9Sgw= 2024-02-01T10:27:50.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:EBmEu5wc5Kho93TH5/YtUVz9Sgw=', 'Content-Length': '0'} 2024-02-01T10:27:50.545 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('x-amz-version-id', 's-qxjv7TkMdxeC61HL2kxUqODSwa-1I'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eed1e1436ef46f1d-0065bb7226-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:50.545 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:50.545 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:50.545 DEBUG:boto:path=/vclfcn-19/obj?versionId=s-qxjv7TkMdxeC61HL2kxUqODSwa-1I 2024-02-01T10:27:50.546 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=s-qxjv7TkMdxeC61HL2kxUqODSwa-1I 2024-02-01T10:27:50.546 DEBUG:boto:Method: GET 2024-02-01T10:27:50.546 DEBUG:boto:Path: /vclfcn-19/obj?versionId=s-qxjv7TkMdxeC61HL2kxUqODSwa-1I 2024-02-01T10:27:50.546 DEBUG:boto:Data: 2024-02-01T10:27:50.546 DEBUG:boto:Headers: {} 2024-02-01T10:27:50.546 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:50.546 DEBUG:boto:Port: 8000 2024-02-01T10:27:50.546 DEBUG:boto:Params: {} 2024-02-01T10:27:50.546 DEBUG:boto:Token: None 2024-02-01T10:27:50.547 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:50 GMT /vclfcn-19/obj?versionId=s-qxjv7TkMdxeC61HL2kxUqODSwa-1I 2024-02-01T10:27:50.547 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:EBmEu5wc5Kho93TH5/YtUVz9Sgw= 2024-02-01T10:27:50.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:EBmEu5wc5Kho93TH5/YtUVz9Sgw=', 'Content-Length': '0'} 2024-02-01T10:27:50.550 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('x-amz-version-id', 's-qxjv7TkMdxeC61HL2kxUqODSwa-1I'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000040b8959629e84857-0065bb7226-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:50.596 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:50.596 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:50.596 DEBUG:boto:path=/vclfcn-19/obj?versionId=s-qxjv7TkMdxeC61HL2kxUqODSwa-1I 2024-02-01T10:27:50.597 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=s-qxjv7TkMdxeC61HL2kxUqODSwa-1I 2024-02-01T10:27:50.597 DEBUG:boto:Method: HEAD 2024-02-01T10:27:50.597 DEBUG:boto:Path: /vclfcn-19/obj?versionId=s-qxjv7TkMdxeC61HL2kxUqODSwa-1I 2024-02-01T10:27:50.597 DEBUG:boto:Data: 2024-02-01T10:27:50.597 DEBUG:boto:Headers: {} 2024-02-01T10:27:50.597 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:50.597 DEBUG:boto:Port: 8001 2024-02-01T10:27:50.597 DEBUG:boto:Params: {} 2024-02-01T10:27:50.597 DEBUG:boto:Token: None 2024-02-01T10:27:50.598 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:50 GMT /vclfcn-19/obj?versionId=s-qxjv7TkMdxeC61HL2kxUqODSwa-1I 2024-02-01T10:27:50.598 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:CwiSVpV9/ch9IGkVhXoOEkCazKk= 2024-02-01T10:27:50.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:CwiSVpV9/ch9IGkVhXoOEkCazKk=', 'Content-Length': '0'} 2024-02-01T10:27:50.601 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('x-amz-version-id', 's-qxjv7TkMdxeC61HL2kxUqODSwa-1I'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d808f6fe08fecd81-0065bb7226-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:50.601 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:50.601 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:50.601 DEBUG:boto:path=/vclfcn-19/obj?versionId=s-qxjv7TkMdxeC61HL2kxUqODSwa-1I 2024-02-01T10:27:50.601 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=s-qxjv7TkMdxeC61HL2kxUqODSwa-1I 2024-02-01T10:27:50.601 DEBUG:boto:Method: HEAD 2024-02-01T10:27:50.601 DEBUG:boto:Path: /vclfcn-19/obj?versionId=s-qxjv7TkMdxeC61HL2kxUqODSwa-1I 2024-02-01T10:27:50.601 DEBUG:boto:Data: 2024-02-01T10:27:50.602 DEBUG:boto:Headers: {} 2024-02-01T10:27:50.602 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:50.602 DEBUG:boto:Port: 8000 2024-02-01T10:27:50.602 DEBUG:boto:Params: {} 2024-02-01T10:27:50.602 DEBUG:boto:Token: None 2024-02-01T10:27:50.602 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:50 GMT /vclfcn-19/obj?versionId=s-qxjv7TkMdxeC61HL2kxUqODSwa-1I 2024-02-01T10:27:50.602 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:CwiSVpV9/ch9IGkVhXoOEkCazKk= 2024-02-01T10:27:50.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:CwiSVpV9/ch9IGkVhXoOEkCazKk=', 'Content-Length': '0'} 2024-02-01T10:27:50.605 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('x-amz-version-id', 's-qxjv7TkMdxeC61HL2kxUqODSwa-1I'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cdc5e6093fee7d28-0065bb7226-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:50.605 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:50.605 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:50.605 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:50.605 DEBUG:boto:path=/vclfcn-19/obj?versionId=s-qxjv7TkMdxeC61HL2kxUqODSwa-1I 2024-02-01T10:27:50.606 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=s-qxjv7TkMdxeC61HL2kxUqODSwa-1I 2024-02-01T10:27:50.606 DEBUG:boto:Method: GET 2024-02-01T10:27:50.606 DEBUG:boto:Path: /vclfcn-19/obj?versionId=s-qxjv7TkMdxeC61HL2kxUqODSwa-1I 2024-02-01T10:27:50.606 DEBUG:boto:Data: 2024-02-01T10:27:50.606 DEBUG:boto:Headers: {} 2024-02-01T10:27:50.606 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:50.606 DEBUG:boto:Port: 8001 2024-02-01T10:27:50.606 DEBUG:boto:Params: {} 2024-02-01T10:27:50.606 DEBUG:boto:Token: None 2024-02-01T10:27:50.606 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:50 GMT /vclfcn-19/obj?versionId=s-qxjv7TkMdxeC61HL2kxUqODSwa-1I 2024-02-01T10:27:50.607 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:EBmEu5wc5Kho93TH5/YtUVz9Sgw= 2024-02-01T10:27:50.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:EBmEu5wc5Kho93TH5/YtUVz9Sgw=', 'Content-Length': '0'} 2024-02-01T10:27:50.609 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('x-amz-version-id', 's-qxjv7TkMdxeC61HL2kxUqODSwa-1I'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000af3c113b64312ea2-0065bb7226-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:50.652 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:50.652 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:50.652 DEBUG:boto:path=/vclfcn-19/obj?versionId=s-qxjv7TkMdxeC61HL2kxUqODSwa-1I 2024-02-01T10:27:50.652 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=s-qxjv7TkMdxeC61HL2kxUqODSwa-1I 2024-02-01T10:27:50.653 DEBUG:boto:Method: GET 2024-02-01T10:27:50.653 DEBUG:boto:Path: /vclfcn-19/obj?versionId=s-qxjv7TkMdxeC61HL2kxUqODSwa-1I 2024-02-01T10:27:50.653 DEBUG:boto:Data: 2024-02-01T10:27:50.653 DEBUG:boto:Headers: {} 2024-02-01T10:27:50.653 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:50.653 DEBUG:boto:Port: 8000 2024-02-01T10:27:50.653 DEBUG:boto:Params: {} 2024-02-01T10:27:50.653 DEBUG:boto:Token: None 2024-02-01T10:27:50.653 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:50 GMT /vclfcn-19/obj?versionId=s-qxjv7TkMdxeC61HL2kxUqODSwa-1I 2024-02-01T10:27:50.654 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:EBmEu5wc5Kho93TH5/YtUVz9Sgw= 2024-02-01T10:27:50.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:EBmEu5wc5Kho93TH5/YtUVz9Sgw=', 'Content-Length': '0'} 2024-02-01T10:27:50.657 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('x-amz-version-id', 's-qxjv7TkMdxeC61HL2kxUqODSwa-1I'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000284061c394a3f998-0065bb7226-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:50.700 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:50.700 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:50.700 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:50.700 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:50.701 DEBUG:boto:Method: GET 2024-02-01T10:27:50.701 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:50.701 DEBUG:boto:Data: 2024-02-01T10:27:50.701 DEBUG:boto:Headers: {} 2024-02-01T10:27:50.701 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:50.701 DEBUG:boto:Port: 8001 2024-02-01T10:27:50.701 DEBUG:boto:Params: {} 2024-02-01T10:27:50.701 DEBUG:boto:Token: None 2024-02-01T10:27:50.702 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:50 GMT /vclfcn-19/ 2024-02-01T10:27:50.702 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FwK72WG2W70zMJxgzvYKnS9H2tI= 2024-02-01T10:27:50.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FwK72WG2W70zMJxgzvYKnS9H2tI=', 'Content-Length': '0'} 2024-02-01T10:27:50.707 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045ac2ba03e18ca4f-0065bb7226-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:50.748 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:50.749 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:50.749 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:50.749 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:50.749 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:50.749 DEBUG:boto:Method: GET 2024-02-01T10:27:50.749 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:50.749 DEBUG:boto:Data: 2024-02-01T10:27:50.749 DEBUG:boto:Headers: {} 2024-02-01T10:27:50.749 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:50.749 DEBUG:boto:Port: 8000 2024-02-01T10:27:50.750 DEBUG:boto:Params: {} 2024-02-01T10:27:50.750 DEBUG:boto:Token: None 2024-02-01T10:27:50.750 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:50 GMT /vclfcn-19/ 2024-02-01T10:27:50.750 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FwK72WG2W70zMJxgzvYKnS9H2tI= 2024-02-01T10:27:50.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FwK72WG2W70zMJxgzvYKnS9H2tI=', 'Content-Length': '0'} 2024-02-01T10:27:50.756 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000300d52c4b536772-0065bb7226-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:50.757 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:50.757 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:50.757 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:50.757 DEBUG:boto:Method: HEAD 2024-02-01T10:27:50.757 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:50.757 DEBUG:boto:Data: 2024-02-01T10:27:50.757 DEBUG:boto:Headers: {} 2024-02-01T10:27:50.758 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:50.758 DEBUG:boto:Port: 8001 2024-02-01T10:27:50.758 DEBUG:boto:Params: {} 2024-02-01T10:27:50.758 DEBUG:boto:Token: None 2024-02-01T10:27:50.758 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:50 GMT /vclfcn-19/obj 2024-02-01T10:27:50.758 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:hjc0ueHw7/hSpshU8J+4Q4YhFzo= 2024-02-01T10:27:50.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:hjc0ueHw7/hSpshU8J+4Q4YhFzo=', 'Content-Length': '0'} 2024-02-01T10:27:50.764 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d387a3db0e11d7c4-0065bb7226-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:50.765 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:50.765 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:50.765 DEBUG:boto:Method: HEAD 2024-02-01T10:27:50.765 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:50.765 DEBUG:boto:Data: 2024-02-01T10:27:50.765 DEBUG:boto:Headers: {} 2024-02-01T10:27:50.765 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:50.766 DEBUG:boto:Port: 8000 2024-02-01T10:27:50.766 DEBUG:boto:Params: {} 2024-02-01T10:27:50.766 DEBUG:boto:Token: None 2024-02-01T10:27:50.766 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:50 GMT /vclfcn-19/obj 2024-02-01T10:27:50.766 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:hjc0ueHw7/hSpshU8J+4Q4YhFzo= 2024-02-01T10:27:50.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:hjc0ueHw7/hSpshU8J+4Q4YhFzo=', 'Content-Length': '0'} 2024-02-01T10:27:50.772 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d7198add3f74e9a7-0065bb7226-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:50.772 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:50.772 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:50.772 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:50.772 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:50.772 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:50.772 DEBUG:boto:Method: GET 2024-02-01T10:27:50.773 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:50.773 DEBUG:boto:Data: 2024-02-01T10:27:50.773 DEBUG:boto:Headers: {} 2024-02-01T10:27:50.773 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:50.773 DEBUG:boto:Port: 8001 2024-02-01T10:27:50.773 DEBUG:boto:Params: {} 2024-02-01T10:27:50.773 DEBUG:boto:Token: None 2024-02-01T10:27:50.773 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:50 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:50.773 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:UWEAaAH7JjdGTFY3BefzJyf4eZE= 2024-02-01T10:27:50.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:UWEAaAH7JjdGTFY3BefzJyf4eZE=', 'Content-Length': '0'} 2024-02-01T10:27:50.776 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b27c3f9a2c78f4c8-0065bb7226-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:50.824 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:50.824 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:50.824 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:50.824 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:50.825 DEBUG:boto:Method: GET 2024-02-01T10:27:50.825 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:50.825 DEBUG:boto:Data: 2024-02-01T10:27:50.825 DEBUG:boto:Headers: {} 2024-02-01T10:27:50.825 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:50.825 DEBUG:boto:Port: 8000 2024-02-01T10:27:50.825 DEBUG:boto:Params: {} 2024-02-01T10:27:50.825 DEBUG:boto:Token: None 2024-02-01T10:27:50.825 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:50 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:50.826 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:UWEAaAH7JjdGTFY3BefzJyf4eZE= 2024-02-01T10:27:50.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:UWEAaAH7JjdGTFY3BefzJyf4eZE=', 'Content-Length': '0'} 2024-02-01T10:27:50.828 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000705e563841659156-0065bb7226-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:50.872 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:50.872 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:50.872 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:50.872 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:50.873 DEBUG:boto:Method: GET 2024-02-01T10:27:50.873 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:50.873 DEBUG:boto:Data: 2024-02-01T10:27:50.873 DEBUG:boto:Headers: {} 2024-02-01T10:27:50.873 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:50.873 DEBUG:boto:Port: 8001 2024-02-01T10:27:50.873 DEBUG:boto:Params: {} 2024-02-01T10:27:50.873 DEBUG:boto:Token: None 2024-02-01T10:27:50.873 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:50 GMT /vclfcn-19/ 2024-02-01T10:27:50.874 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FwK72WG2W70zMJxgzvYKnS9H2tI= 2024-02-01T10:27:50.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FwK72WG2W70zMJxgzvYKnS9H2tI=', 'Content-Length': '0'} 2024-02-01T10:27:50.878 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c8c22f37664e6af5-0065bb7226-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:50.924 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:50.925 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:50.925 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:50.925 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:50.925 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:50.925 DEBUG:boto:Method: GET 2024-02-01T10:27:50.925 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:50.925 DEBUG:boto:Data: 2024-02-01T10:27:50.925 DEBUG:boto:Headers: {} 2024-02-01T10:27:50.925 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:27:50.926 DEBUG:boto:Port: 8000 2024-02-01T10:27:50.926 DEBUG:boto:Params: {} 2024-02-01T10:27:50.926 DEBUG:boto:Token: None 2024-02-01T10:27:50.926 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:50 GMT /vclfcn-19/ 2024-02-01T10:27:50.926 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FwK72WG2W70zMJxgzvYKnS9H2tI= 2024-02-01T10:27:50.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FwK72WG2W70zMJxgzvYKnS9H2tI=', 'Content-Length': '0'} 2024-02-01T10:27:50.931 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e15d7a8e7ac011fe-0065bb7226-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:50.932 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:50.932 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-19 zones={test-zone3, test-zone1} 2024-02-01T10:27:50.932 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-19 zones={test-zone3, test-zone2} 2024-02-01T10:27:50.932 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:50.933 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:50.933 DEBUG:boto:Method: HEAD 2024-02-01T10:27:50.933 DEBUG:boto:Path: /vclfcn-19/ 2024-02-01T10:27:50.933 DEBUG:boto:Data: 2024-02-01T10:27:50.933 DEBUG:boto:Headers: {} 2024-02-01T10:27:50.933 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:50.933 DEBUG:boto:Port: 8001 2024-02-01T10:27:50.933 DEBUG:boto:Params: {} 2024-02-01T10:27:50.933 DEBUG:boto:Token: None 2024-02-01T10:27:50.933 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:50 GMT /vclfcn-19/ 2024-02-01T10:27:50.934 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:hbl5pxl2HJaJS6MyyQhgnAz1XWk= 2024-02-01T10:27:50.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:hbl5pxl2HJaJS6MyyQhgnAz1XWk=', 'Content-Length': '0'} 2024-02-01T10:27:50.938 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ea2e3d70dad25802-0065bb7226-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:50.938 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:50.938 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:50.938 DEBUG:boto:Method: HEAD 2024-02-01T10:27:50.939 DEBUG:boto:Path: /vclfcn-19/ 2024-02-01T10:27:50.939 DEBUG:boto:Data: 2024-02-01T10:27:50.939 DEBUG:boto:Headers: {} 2024-02-01T10:27:50.939 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:50.939 DEBUG:boto:Port: 8000 2024-02-01T10:27:50.939 DEBUG:boto:Params: {} 2024-02-01T10:27:50.939 DEBUG:boto:Token: None 2024-02-01T10:27:50.939 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:50 GMT /vclfcn-19/ 2024-02-01T10:27:50.939 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:hbl5pxl2HJaJS6MyyQhgnAz1XWk= 2024-02-01T10:27:50.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:hbl5pxl2HJaJS6MyyQhgnAz1XWk=', 'Content-Length': '0'} 2024-02-01T10:27:50.944 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000067b9f311a323ea32-0065bb7226-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:50.944 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:27:50.944 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:50.944 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:50.944 DEBUG:boto:path=/vclfcn-19/?versions&max-keys=999 2024-02-01T10:27:50.944 DEBUG:boto:auth_path=/vclfcn-19/?versions&max-keys=999 2024-02-01T10:27:50.944 DEBUG:boto:Method: GET 2024-02-01T10:27:50.945 DEBUG:boto:Path: /vclfcn-19/?versions&max-keys=999 2024-02-01T10:27:50.945 DEBUG:boto:Data: 2024-02-01T10:27:50.945 DEBUG:boto:Headers: {} 2024-02-01T10:27:50.945 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:50.945 DEBUG:boto:Port: 8001 2024-02-01T10:27:50.945 DEBUG:boto:Params: {} 2024-02-01T10:27:50.945 DEBUG:boto:Token: None 2024-02-01T10:27:50.945 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:50 GMT /vclfcn-19/?versions 2024-02-01T10:27:50.945 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BGD0+GArzkkf7eB+zY7Q1ryaWcU= 2024-02-01T10:27:50.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BGD0+GArzkkf7eB+zY7Q1ryaWcU=', 'Content-Length': '0'} 2024-02-01T10:27:50.954 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5f0a3e59b70883d-0065bb7226-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:50.954 DEBUG:boto:b'vclfcn-19999falsedummyg5UScztMjPBIYZ2b5yOn-qsm.eQTcnutrue2024-02-01T10:23:11.115Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23llUchmWTvJKOzZpTkl59nzz.uMOQbtrue2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjRrytQti15jy37pft0WxtgWdt4Zs7l4Efalse2024-02-01T10:25:33.973Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjfzdlI7n.dsuwfAOSBNyk84Wd7l5p70ufalse2024-02-01T10:25:33.886Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUOa9NAxhXbH7RR3jkzlpXPh.fGAfBJcfalse2024-02-01T10:25:33.774Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjeNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr.false2024-02-01T10:25:33.746Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtGIsEYvB.VyZPdq9CgtD6kEeP4rzbDNfalse2024-02-01T10:25:33.830Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2lRbqumcCYnVWCIz20mU455245nXaz8false2024-02-01T10:25:33.719Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjoGhdnHMtrLPTgFgXlKfl.UXdmrUnL2rfalse2024-02-01T10:25:33.687Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjqex3Bh72IV4plxG2V2Lm7bvqwXqQNK9false2024-02-01T10:25:33.661Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjNCPEvRYQDJpLrTejJRqmTf2j0FWCBCSfalse2024-02-01T10:25:33.599Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjVQT2kBCkF07dugmyrhNeRpQpR0HK7bTfalse2024-02-01T10:25:33.630Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtvvjWUbgZIRYhHk9r.liT6sPCWCs2Zifalse2024-02-01T10:25:33.551Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3OSaU40E5-Hgv6pEDB8btq8uZcAFHVzfalse2024-02-01T10:25:33.438Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjK23uwsLWGmQDGYGsdu-h04Fk2wApB0Ofalse2024-02-01T10:25:33.410Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjst2TOT.K7IOe-wFfL.hOrRVfLNWblRdfalse2024-02-01T10:25:33.494Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9CJ3w1EusK8OST2k17sTdTM.cuh3E3efalse2024-02-01T10:25:33.349Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjVwfx4ETdAtTkKn2rTCQT4onqB6sYulvfalse2024-02-01T10:25:33.381Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjiGO2QyMOL81IaF-H4sHDqyECPlnZBmEfalse2024-02-01T10:25:33.323Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjcYHYWFiJtzK2FpqgaUq4bR.SX5MaCkIfalse2024-02-01T10:25:33.206Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjrasya8taOZsGTj9rDi7keghf.VnPFlFfalse2024-02-01T10:25:33.291Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjy-DPCvngcj9q7W-lUL16Jgbp.NhV7NEfalse2024-02-01T10:25:33.257Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj0KR6m6ymTGitKe6vabQyVENAnBL669wfalse2024-02-01T10:25:33.118Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2Hfm7x.sO5cadpLfjJNuhN.TAkjS54Efalse2024-02-01T10:25:33.093Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6false2024-02-01T10:25:33.150Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHXfalse2024-02-01T10:25:33.030Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj63CC4agggciY3jdYETzlNEHLx15uh8wfalse2024-02-01T10:25:33.063Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjYzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1ofalse2024-02-01T10:25:32.998Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjB5jd89.SLRVLOAPcXqackukLvQnZY1Bfalse2024-02-01T10:25:32.872Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjrcBw9NDMcjurzSdpB.bvwX4JmUDkwrQfalse2024-02-01T10:25:32.964Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjs-qxjv7TkMdxeC61HL2kxUqODSwa-1Ifalse2024-02-01T10:25:32.926Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:50.956 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-02-01T10:27:50.956 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:50.956 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:50.956 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:50.956 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:50.956 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:50.956 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:50.956 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:50.957 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:50.957 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:50.957 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:50.957 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:50.957 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:50.957 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:50.957 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:50.957 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:50.957 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:50.957 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:50.957 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:50.957 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:50.958 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:50.958 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:50.958 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:50.958 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:50.958 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:50.958 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:50.958 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:50.958 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:50.958 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:50.958 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:50.958 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:50.959 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:27:50.959 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:50.959 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:50.959 DEBUG:boto:path=/vclfcn-19/?versions&max-keys=999 2024-02-01T10:27:50.959 DEBUG:boto:auth_path=/vclfcn-19/?versions&max-keys=999 2024-02-01T10:27:50.959 DEBUG:boto:Method: GET 2024-02-01T10:27:50.959 DEBUG:boto:Path: /vclfcn-19/?versions&max-keys=999 2024-02-01T10:27:50.959 DEBUG:boto:Data: 2024-02-01T10:27:50.959 DEBUG:boto:Headers: {} 2024-02-01T10:27:50.959 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:50.960 DEBUG:boto:Port: 8000 2024-02-01T10:27:50.960 DEBUG:boto:Params: {} 2024-02-01T10:27:50.960 DEBUG:boto:Token: None 2024-02-01T10:27:50.960 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:50 GMT /vclfcn-19/?versions 2024-02-01T10:27:50.960 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BGD0+GArzkkf7eB+zY7Q1ryaWcU= 2024-02-01T10:27:50.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BGD0+GArzkkf7eB+zY7Q1ryaWcU=', 'Content-Length': '0'} 2024-02-01T10:27:50.966 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bae1ea56a03b0f08-0065bb7226-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:50.967 DEBUG:boto:b'vclfcn-19999falsedummyg5UScztMjPBIYZ2b5yOn-qsm.eQTcnutrue2024-02-01T10:23:11.115Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23llUchmWTvJKOzZpTkl59nzz.uMOQbtrue2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjRrytQti15jy37pft0WxtgWdt4Zs7l4Efalse2024-02-01T10:25:33.973Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjfzdlI7n.dsuwfAOSBNyk84Wd7l5p70ufalse2024-02-01T10:25:33.886Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUOa9NAxhXbH7RR3jkzlpXPh.fGAfBJcfalse2024-02-01T10:25:33.774Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjeNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr.false2024-02-01T10:25:33.746Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtGIsEYvB.VyZPdq9CgtD6kEeP4rzbDNfalse2024-02-01T10:25:33.830Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2lRbqumcCYnVWCIz20mU455245nXaz8false2024-02-01T10:25:33.719Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjoGhdnHMtrLPTgFgXlKfl.UXdmrUnL2rfalse2024-02-01T10:25:33.687Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjqex3Bh72IV4plxG2V2Lm7bvqwXqQNK9false2024-02-01T10:25:33.661Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjNCPEvRYQDJpLrTejJRqmTf2j0FWCBCSfalse2024-02-01T10:25:33.599Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjVQT2kBCkF07dugmyrhNeRpQpR0HK7bTfalse2024-02-01T10:25:33.630Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtvvjWUbgZIRYhHk9r.liT6sPCWCs2Zifalse2024-02-01T10:25:33.551Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3OSaU40E5-Hgv6pEDB8btq8uZcAFHVzfalse2024-02-01T10:25:33.438Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjK23uwsLWGmQDGYGsdu-h04Fk2wApB0Ofalse2024-02-01T10:25:33.410Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjst2TOT.K7IOe-wFfL.hOrRVfLNWblRdfalse2024-02-01T10:25:33.494Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9CJ3w1EusK8OST2k17sTdTM.cuh3E3efalse2024-02-01T10:25:33.349Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjVwfx4ETdAtTkKn2rTCQT4onqB6sYulvfalse2024-02-01T10:25:33.381Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjiGO2QyMOL81IaF-H4sHDqyECPlnZBmEfalse2024-02-01T10:25:33.323Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjcYHYWFiJtzK2FpqgaUq4bR.SX5MaCkIfalse2024-02-01T10:25:33.206Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjrasya8taOZsGTj9rDi7keghf.VnPFlFfalse2024-02-01T10:25:33.291Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjy-DPCvngcj9q7W-lUL16Jgbp.NhV7NEfalse2024-02-01T10:25:33.257Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj0KR6m6ymTGitKe6vabQyVENAnBL669wfalse2024-02-01T10:25:33.118Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2Hfm7x.sO5cadpLfjJNuhN.TAkjS54Efalse2024-02-01T10:25:33.093Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6false2024-02-01T10:25:33.150Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHXfalse2024-02-01T10:25:33.030Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj63CC4agggciY3jdYETzlNEHLx15uh8wfalse2024-02-01T10:25:33.063Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjYzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1ofalse2024-02-01T10:25:32.998Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjB5jd89.SLRVLOAPcXqackukLvQnZY1Bfalse2024-02-01T10:25:32.872Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjrcBw9NDMcjurzSdpB.bvwX4JmUDkwrQfalse2024-02-01T10:25:32.964Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjs-qxjv7TkMdxeC61HL2kxUqODSwa-1Ifalse2024-02-01T10:25:32.926Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:50.969 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-02-01T10:27:50.969 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:50.969 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:50.969 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:50.969 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:50.969 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:50.969 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:50.969 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:50.969 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:50.969 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:50.969 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:50.970 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:50.970 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:50.970 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:50.970 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:50.970 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:50.970 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:50.970 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:50.970 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:50.970 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:50.971 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:50.971 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:50.971 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:50.971 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:50.971 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:50.971 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:50.972 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:50.972 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:50.972 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:50.972 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:50.972 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:27:50.972 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:50.972 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:50.972 DEBUG:boto:path=/vclfcn-19/?versions&max-keys=999 2024-02-01T10:27:50.972 DEBUG:boto:auth_path=/vclfcn-19/?versions&max-keys=999 2024-02-01T10:27:50.973 DEBUG:boto:Method: GET 2024-02-01T10:27:50.973 DEBUG:boto:Path: /vclfcn-19/?versions&max-keys=999 2024-02-01T10:27:50.973 DEBUG:boto:Data: 2024-02-01T10:27:50.973 DEBUG:boto:Headers: {} 2024-02-01T10:27:50.974 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:50.974 DEBUG:boto:Port: 8001 2024-02-01T10:27:50.974 DEBUG:boto:Params: {} 2024-02-01T10:27:50.974 DEBUG:boto:Token: None 2024-02-01T10:27:50.974 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:50 GMT /vclfcn-19/?versions 2024-02-01T10:27:50.975 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BGD0+GArzkkf7eB+zY7Q1ryaWcU= 2024-02-01T10:27:50.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BGD0+GArzkkf7eB+zY7Q1ryaWcU=', 'Content-Length': '0'} 2024-02-01T10:27:50.981 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000026e110f868c211f-0065bb7226-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:50.982 DEBUG:boto:b'vclfcn-19999falsedummyg5UScztMjPBIYZ2b5yOn-qsm.eQTcnutrue2024-02-01T10:23:11.115Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23llUchmWTvJKOzZpTkl59nzz.uMOQbtrue2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjRrytQti15jy37pft0WxtgWdt4Zs7l4Efalse2024-02-01T10:25:33.973Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjfzdlI7n.dsuwfAOSBNyk84Wd7l5p70ufalse2024-02-01T10:25:33.886Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUOa9NAxhXbH7RR3jkzlpXPh.fGAfBJcfalse2024-02-01T10:25:33.774Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjeNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr.false2024-02-01T10:25:33.746Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtGIsEYvB.VyZPdq9CgtD6kEeP4rzbDNfalse2024-02-01T10:25:33.830Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2lRbqumcCYnVWCIz20mU455245nXaz8false2024-02-01T10:25:33.719Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjoGhdnHMtrLPTgFgXlKfl.UXdmrUnL2rfalse2024-02-01T10:25:33.687Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjqex3Bh72IV4plxG2V2Lm7bvqwXqQNK9false2024-02-01T10:25:33.661Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjNCPEvRYQDJpLrTejJRqmTf2j0FWCBCSfalse2024-02-01T10:25:33.599Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjVQT2kBCkF07dugmyrhNeRpQpR0HK7bTfalse2024-02-01T10:25:33.630Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtvvjWUbgZIRYhHk9r.liT6sPCWCs2Zifalse2024-02-01T10:25:33.551Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3OSaU40E5-Hgv6pEDB8btq8uZcAFHVzfalse2024-02-01T10:25:33.438Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjK23uwsLWGmQDGYGsdu-h04Fk2wApB0Ofalse2024-02-01T10:25:33.410Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjst2TOT.K7IOe-wFfL.hOrRVfLNWblRdfalse2024-02-01T10:25:33.494Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9CJ3w1EusK8OST2k17sTdTM.cuh3E3efalse2024-02-01T10:25:33.349Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjVwfx4ETdAtTkKn2rTCQT4onqB6sYulvfalse2024-02-01T10:25:33.381Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjiGO2QyMOL81IaF-H4sHDqyECPlnZBmEfalse2024-02-01T10:25:33.323Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjcYHYWFiJtzK2FpqgaUq4bR.SX5MaCkIfalse2024-02-01T10:25:33.206Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjrasya8taOZsGTj9rDi7keghf.VnPFlFfalse2024-02-01T10:25:33.291Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjy-DPCvngcj9q7W-lUL16Jgbp.NhV7NEfalse2024-02-01T10:25:33.257Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj0KR6m6ymTGitKe6vabQyVENAnBL669wfalse2024-02-01T10:25:33.118Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2Hfm7x.sO5cadpLfjJNuhN.TAkjS54Efalse2024-02-01T10:25:33.093Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6false2024-02-01T10:25:33.150Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHXfalse2024-02-01T10:25:33.030Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj63CC4agggciY3jdYETzlNEHLx15uh8wfalse2024-02-01T10:25:33.063Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjYzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1ofalse2024-02-01T10:25:32.998Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjB5jd89.SLRVLOAPcXqackukLvQnZY1Bfalse2024-02-01T10:25:32.872Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjrcBw9NDMcjurzSdpB.bvwX4JmUDkwrQfalse2024-02-01T10:25:32.964Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjs-qxjv7TkMdxeC61HL2kxUqODSwa-1Ifalse2024-02-01T10:25:32.926Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:50.984 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:50.984 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:50.984 DEBUG:boto:path=/vclfcn-19/?versions&max-keys=999 2024-02-01T10:27:50.984 DEBUG:boto:auth_path=/vclfcn-19/?versions&max-keys=999 2024-02-01T10:27:50.984 DEBUG:boto:Method: GET 2024-02-01T10:27:50.984 DEBUG:boto:Path: /vclfcn-19/?versions&max-keys=999 2024-02-01T10:27:50.984 DEBUG:boto:Data: 2024-02-01T10:27:50.985 DEBUG:boto:Headers: {} 2024-02-01T10:27:50.985 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:50.985 DEBUG:boto:Port: 8000 2024-02-01T10:27:50.985 DEBUG:boto:Params: {} 2024-02-01T10:27:50.985 DEBUG:boto:Token: None 2024-02-01T10:27:50.985 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:50 GMT /vclfcn-19/?versions 2024-02-01T10:27:50.985 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BGD0+GArzkkf7eB+zY7Q1ryaWcU= 2024-02-01T10:27:50.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BGD0+GArzkkf7eB+zY7Q1ryaWcU=', 'Content-Length': '0'} 2024-02-01T10:27:50.991 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007606a8918630efdd-0065bb7226-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:50.992 DEBUG:boto:b'vclfcn-19999falsedummyg5UScztMjPBIYZ2b5yOn-qsm.eQTcnutrue2024-02-01T10:23:11.115Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23llUchmWTvJKOzZpTkl59nzz.uMOQbtrue2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjRrytQti15jy37pft0WxtgWdt4Zs7l4Efalse2024-02-01T10:25:33.973Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjfzdlI7n.dsuwfAOSBNyk84Wd7l5p70ufalse2024-02-01T10:25:33.886Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUOa9NAxhXbH7RR3jkzlpXPh.fGAfBJcfalse2024-02-01T10:25:33.774Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjeNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr.false2024-02-01T10:25:33.746Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtGIsEYvB.VyZPdq9CgtD6kEeP4rzbDNfalse2024-02-01T10:25:33.830Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2lRbqumcCYnVWCIz20mU455245nXaz8false2024-02-01T10:25:33.719Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjoGhdnHMtrLPTgFgXlKfl.UXdmrUnL2rfalse2024-02-01T10:25:33.687Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjqex3Bh72IV4plxG2V2Lm7bvqwXqQNK9false2024-02-01T10:25:33.661Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjNCPEvRYQDJpLrTejJRqmTf2j0FWCBCSfalse2024-02-01T10:25:33.599Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjVQT2kBCkF07dugmyrhNeRpQpR0HK7bTfalse2024-02-01T10:25:33.630Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtvvjWUbgZIRYhHk9r.liT6sPCWCs2Zifalse2024-02-01T10:25:33.551Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3OSaU40E5-Hgv6pEDB8btq8uZcAFHVzfalse2024-02-01T10:25:33.438Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjK23uwsLWGmQDGYGsdu-h04Fk2wApB0Ofalse2024-02-01T10:25:33.410Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjst2TOT.K7IOe-wFfL.hOrRVfLNWblRdfalse2024-02-01T10:25:33.494Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9CJ3w1EusK8OST2k17sTdTM.cuh3E3efalse2024-02-01T10:25:33.349Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjVwfx4ETdAtTkKn2rTCQT4onqB6sYulvfalse2024-02-01T10:25:33.381Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjiGO2QyMOL81IaF-H4sHDqyECPlnZBmEfalse2024-02-01T10:25:33.323Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjcYHYWFiJtzK2FpqgaUq4bR.SX5MaCkIfalse2024-02-01T10:25:33.206Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjrasya8taOZsGTj9rDi7keghf.VnPFlFfalse2024-02-01T10:25:33.291Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjy-DPCvngcj9q7W-lUL16Jgbp.NhV7NEfalse2024-02-01T10:25:33.257Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj0KR6m6ymTGitKe6vabQyVENAnBL669wfalse2024-02-01T10:25:33.118Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2Hfm7x.sO5cadpLfjJNuhN.TAkjS54Efalse2024-02-01T10:25:33.093Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6false2024-02-01T10:25:33.150Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHXfalse2024-02-01T10:25:33.030Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj63CC4agggciY3jdYETzlNEHLx15uh8wfalse2024-02-01T10:25:33.063Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjYzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1ofalse2024-02-01T10:25:32.998Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjB5jd89.SLRVLOAPcXqackukLvQnZY1Bfalse2024-02-01T10:25:32.872Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjrcBw9NDMcjurzSdpB.bvwX4JmUDkwrQfalse2024-02-01T10:25:32.964Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjs-qxjv7TkMdxeC61HL2kxUqODSwa-1Ifalse2024-02-01T10:25:32.926Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:50.993 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-02-01T10:27:50.994 DEBUG:boto:path=/vclfcn-19/dummy 2024-02-01T10:27:50.994 DEBUG:boto:auth_path=/vclfcn-19/dummy 2024-02-01T10:27:50.994 DEBUG:boto:path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:50.994 DEBUG:boto:auth_path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:50.994 DEBUG:boto:Method: GET 2024-02-01T10:27:50.994 DEBUG:boto:Path: /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:50.994 DEBUG:boto:Data: 2024-02-01T10:27:50.994 DEBUG:boto:Headers: {} 2024-02-01T10:27:50.994 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:50.994 DEBUG:boto:Port: 8001 2024-02-01T10:27:50.994 DEBUG:boto:Params: {} 2024-02-01T10:27:50.995 DEBUG:boto:Token: None 2024-02-01T10:27:50.995 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:50 GMT /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:50.995 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:g5oqfHiczHfQG79topp9dErvF7Q= 2024-02-01T10:27:50.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:g5oqfHiczHfQG79topp9dErvF7Q=', 'Content-Length': '0'} 2024-02-01T10:27:50.998 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:23:11 GMT'), ('x-amz-version-id', 'g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000741860e088a0a1b8-0065bb7227-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:50.998 DEBUG:boto:path=/vclfcn-19/dummy 2024-02-01T10:27:50.998 DEBUG:boto:auth_path=/vclfcn-19/dummy 2024-02-01T10:27:50.998 DEBUG:boto:path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:50.998 DEBUG:boto:auth_path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:50.998 DEBUG:boto:Method: GET 2024-02-01T10:27:50.998 DEBUG:boto:Path: /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:50.999 DEBUG:boto:Data: 2024-02-01T10:27:50.999 DEBUG:boto:Headers: {} 2024-02-01T10:27:50.999 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:50.999 DEBUG:boto:Port: 8000 2024-02-01T10:27:50.999 DEBUG:boto:Params: {} 2024-02-01T10:27:50.999 DEBUG:boto:Token: None 2024-02-01T10:27:50.999 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:50 GMT /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:50.999 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:g5oqfHiczHfQG79topp9dErvF7Q= 2024-02-01T10:27:50.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:g5oqfHiczHfQG79topp9dErvF7Q=', 'Content-Length': '0'} 2024-02-01T10:27:51.002 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:23:11 GMT'), ('x-amz-version-id', 'g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005e25e6339bc28e33-0065bb7227-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:51.003 DEBUG:boto:path=/vclfcn-19/dummy 2024-02-01T10:27:51.003 DEBUG:boto:auth_path=/vclfcn-19/dummy 2024-02-01T10:27:51.003 DEBUG:boto:path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:51.003 DEBUG:boto:auth_path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:51.003 DEBUG:boto:Method: HEAD 2024-02-01T10:27:51.004 DEBUG:boto:Path: /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:51.004 DEBUG:boto:Data: 2024-02-01T10:27:51.004 DEBUG:boto:Headers: {} 2024-02-01T10:27:51.004 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:51.004 DEBUG:boto:Port: 8001 2024-02-01T10:27:51.004 DEBUG:boto:Params: {} 2024-02-01T10:27:51.004 DEBUG:boto:Token: None 2024-02-01T10:27:51.004 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:51 GMT /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:51.004 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jAtedPEEkT31gItgGVMgmRDY3lU= 2024-02-01T10:27:51.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jAtedPEEkT31gItgGVMgmRDY3lU=', 'Content-Length': '0'} 2024-02-01T10:27:51.008 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:23:11 GMT'), ('x-amz-version-id', 'g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000710ed77dc69180a2-0065bb7227-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:51.008 DEBUG:boto:path=/vclfcn-19/dummy 2024-02-01T10:27:51.008 DEBUG:boto:auth_path=/vclfcn-19/dummy 2024-02-01T10:27:51.008 DEBUG:boto:path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:51.008 DEBUG:boto:auth_path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:51.009 DEBUG:boto:Method: HEAD 2024-02-01T10:27:51.009 DEBUG:boto:Path: /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:51.009 DEBUG:boto:Data: 2024-02-01T10:27:51.009 DEBUG:boto:Headers: {} 2024-02-01T10:27:51.009 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:51.009 DEBUG:boto:Port: 8000 2024-02-01T10:27:51.009 DEBUG:boto:Params: {} 2024-02-01T10:27:51.009 DEBUG:boto:Token: None 2024-02-01T10:27:51.009 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:51 GMT /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:51.009 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jAtedPEEkT31gItgGVMgmRDY3lU= 2024-02-01T10:27:51.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jAtedPEEkT31gItgGVMgmRDY3lU=', 'Content-Length': '0'} 2024-02-01T10:27:51.012 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:23:11 GMT'), ('x-amz-version-id', 'g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c13d6631470df3dd-0065bb7227-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:51.012 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-02-01T10:27:51.012 DEBUG:boto:path=/vclfcn-19/dummy 2024-02-01T10:27:51.013 DEBUG:boto:auth_path=/vclfcn-19/dummy 2024-02-01T10:27:51.013 DEBUG:boto:path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:51.013 DEBUG:boto:auth_path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:51.013 DEBUG:boto:Method: GET 2024-02-01T10:27:51.013 DEBUG:boto:Path: /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:51.013 DEBUG:boto:Data: 2024-02-01T10:27:51.013 DEBUG:boto:Headers: {} 2024-02-01T10:27:51.013 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:51.013 DEBUG:boto:Port: 8001 2024-02-01T10:27:51.013 DEBUG:boto:Params: {} 2024-02-01T10:27:51.013 DEBUG:boto:Token: None 2024-02-01T10:27:51.014 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:51 GMT /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:51.014 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:yGiC5yTX+imJEh/9sbQR8jUWIdo= 2024-02-01T10:27:51.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:yGiC5yTX+imJEh/9sbQR8jUWIdo=', 'Content-Length': '0'} 2024-02-01T10:27:51.016 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:23:11 GMT'), ('x-amz-version-id', 'g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006911e6b3083bfcc2-0065bb7227-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:51.017 DEBUG:boto:path=/vclfcn-19/dummy 2024-02-01T10:27:51.017 DEBUG:boto:auth_path=/vclfcn-19/dummy 2024-02-01T10:27:51.017 DEBUG:boto:path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:51.017 DEBUG:boto:auth_path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:51.017 DEBUG:boto:Method: GET 2024-02-01T10:27:51.017 DEBUG:boto:Path: /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:51.017 DEBUG:boto:Data: 2024-02-01T10:27:51.017 DEBUG:boto:Headers: {} 2024-02-01T10:27:51.018 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:51.018 DEBUG:boto:Port: 8000 2024-02-01T10:27:51.018 DEBUG:boto:Params: {} 2024-02-01T10:27:51.018 DEBUG:boto:Token: None 2024-02-01T10:27:51.018 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:51 GMT /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:51.018 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:yGiC5yTX+imJEh/9sbQR8jUWIdo= 2024-02-01T10:27:51.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:yGiC5yTX+imJEh/9sbQR8jUWIdo=', 'Content-Length': '0'} 2024-02-01T10:27:51.021 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:23:11 GMT'), ('x-amz-version-id', 'g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008f26587d80602b27-0065bb7227-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:51.021 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:51.022 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:51.022 DEBUG:boto:path=/vclfcn-19/?prefix=dummy 2024-02-01T10:27:51.022 DEBUG:boto:auth_path=/vclfcn-19/?prefix=dummy 2024-02-01T10:27:51.022 DEBUG:boto:Method: GET 2024-02-01T10:27:51.022 DEBUG:boto:Path: /vclfcn-19/?prefix=dummy 2024-02-01T10:27:51.022 DEBUG:boto:Data: 2024-02-01T10:27:51.022 DEBUG:boto:Headers: {} 2024-02-01T10:27:51.022 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:51.022 DEBUG:boto:Port: 8001 2024-02-01T10:27:51.022 DEBUG:boto:Params: {} 2024-02-01T10:27:51.023 DEBUG:boto:Token: None 2024-02-01T10:27:51.023 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:51 GMT /vclfcn-19/ 2024-02-01T10:27:51.023 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:K5rNl20W7rEOmAv0y5bdQs9eT0Q= 2024-02-01T10:27:51.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:K5rNl20W7rEOmAv0y5bdQs9eT0Q=', 'Content-Length': '0'} 2024-02-01T10:27:51.027 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f181ee8ccca4e59e-0065bb7227-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:51.068 DEBUG:boto:b'vclfcn-19dummy1000falsedummy2024-02-01T10:23:11.115Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:51.069 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:51.069 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:51.069 DEBUG:boto:path=/vclfcn-19/?prefix=dummy 2024-02-01T10:27:51.069 DEBUG:boto:auth_path=/vclfcn-19/?prefix=dummy 2024-02-01T10:27:51.069 DEBUG:boto:Method: GET 2024-02-01T10:27:51.069 DEBUG:boto:Path: /vclfcn-19/?prefix=dummy 2024-02-01T10:27:51.069 DEBUG:boto:Data: 2024-02-01T10:27:51.069 DEBUG:boto:Headers: {} 2024-02-01T10:27:51.069 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:51.069 DEBUG:boto:Port: 8000 2024-02-01T10:27:51.070 DEBUG:boto:Params: {} 2024-02-01T10:27:51.070 DEBUG:boto:Token: None 2024-02-01T10:27:51.070 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:51 GMT /vclfcn-19/ 2024-02-01T10:27:51.070 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:K5rNl20W7rEOmAv0y5bdQs9eT0Q= 2024-02-01T10:27:51.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:K5rNl20W7rEOmAv0y5bdQs9eT0Q=', 'Content-Length': '0'} 2024-02-01T10:27:51.075 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e8c23056a704dd9-0065bb7227-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:51.075 DEBUG:boto:b'vclfcn-19dummy1000falsedummy2024-02-01T10:23:11.115Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:51.075 DEBUG:boto:path=/vclfcn-19/dummy 2024-02-01T10:27:51.076 DEBUG:boto:auth_path=/vclfcn-19/dummy 2024-02-01T10:27:51.076 DEBUG:boto:Method: HEAD 2024-02-01T10:27:51.076 DEBUG:boto:Path: /vclfcn-19/dummy 2024-02-01T10:27:51.076 DEBUG:boto:Data: 2024-02-01T10:27:51.076 DEBUG:boto:Headers: {} 2024-02-01T10:27:51.076 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:51.076 DEBUG:boto:Port: 8001 2024-02-01T10:27:51.076 DEBUG:boto:Params: {} 2024-02-01T10:27:51.076 DEBUG:boto:Token: None 2024-02-01T10:27:51.077 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:51 GMT /vclfcn-19/dummy 2024-02-01T10:27:51.077 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BmUBIqxiQcsuiJeg0lnpd9E6SOY= 2024-02-01T10:27:51.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BmUBIqxiQcsuiJeg0lnpd9E6SOY=', 'Content-Length': '0'} 2024-02-01T10:27:51.081 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:23:11 GMT'), ('x-amz-version-id', 'g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a91655094dfaea3a-0065bb7227-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:51.081 DEBUG:boto:path=/vclfcn-19/dummy 2024-02-01T10:27:51.081 DEBUG:boto:auth_path=/vclfcn-19/dummy 2024-02-01T10:27:51.081 DEBUG:boto:Method: HEAD 2024-02-01T10:27:51.081 DEBUG:boto:Path: /vclfcn-19/dummy 2024-02-01T10:27:51.081 DEBUG:boto:Data: 2024-02-01T10:27:51.081 DEBUG:boto:Headers: {} 2024-02-01T10:27:51.081 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:51.082 DEBUG:boto:Port: 8000 2024-02-01T10:27:51.082 DEBUG:boto:Params: {} 2024-02-01T10:27:51.082 DEBUG:boto:Token: None 2024-02-01T10:27:51.082 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:51 GMT /vclfcn-19/dummy 2024-02-01T10:27:51.082 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BmUBIqxiQcsuiJeg0lnpd9E6SOY= 2024-02-01T10:27:51.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BmUBIqxiQcsuiJeg0lnpd9E6SOY=', 'Content-Length': '0'} 2024-02-01T10:27:51.086 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:23:11 GMT'), ('x-amz-version-id', 'g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e892a5fbc5576de4-0065bb7227-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:51.086 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-02-01T10:27:51.086 DEBUG:boto:path=/vclfcn-19/dummy 2024-02-01T10:27:51.086 DEBUG:boto:auth_path=/vclfcn-19/dummy 2024-02-01T10:27:51.087 DEBUG:boto:path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:51.087 DEBUG:boto:auth_path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:51.087 DEBUG:boto:Method: GET 2024-02-01T10:27:51.087 DEBUG:boto:Path: /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:51.087 DEBUG:boto:Data: 2024-02-01T10:27:51.087 DEBUG:boto:Headers: {} 2024-02-01T10:27:51.087 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:51.087 DEBUG:boto:Port: 8001 2024-02-01T10:27:51.087 DEBUG:boto:Params: {} 2024-02-01T10:27:51.087 DEBUG:boto:Token: None 2024-02-01T10:27:51.087 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:51 GMT /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:51.088 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:yGiC5yTX+imJEh/9sbQR8jUWIdo= 2024-02-01T10:27:51.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:yGiC5yTX+imJEh/9sbQR8jUWIdo=', 'Content-Length': '0'} 2024-02-01T10:27:51.090 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:23:11 GMT'), ('x-amz-version-id', 'g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e80e98198c10c409-0065bb7227-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:51.091 DEBUG:boto:path=/vclfcn-19/dummy 2024-02-01T10:27:51.091 DEBUG:boto:auth_path=/vclfcn-19/dummy 2024-02-01T10:27:51.091 DEBUG:boto:path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:51.091 DEBUG:boto:auth_path=/vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:51.091 DEBUG:boto:Method: GET 2024-02-01T10:27:51.091 DEBUG:boto:Path: /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:51.091 DEBUG:boto:Data: 2024-02-01T10:27:51.091 DEBUG:boto:Headers: {} 2024-02-01T10:27:51.091 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:51.092 DEBUG:boto:Port: 8000 2024-02-01T10:27:51.092 DEBUG:boto:Params: {} 2024-02-01T10:27:51.092 DEBUG:boto:Token: None 2024-02-01T10:27:51.092 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:51 GMT /vclfcn-19/dummy?versionId=g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu 2024-02-01T10:27:51.092 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:yGiC5yTX+imJEh/9sbQR8jUWIdo= 2024-02-01T10:27:51.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:yGiC5yTX+imJEh/9sbQR8jUWIdo=', 'Content-Length': '0'} 2024-02-01T10:27:51.095 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:23:11 GMT'), ('x-amz-version-id', 'g5UScztMjPBIYZ2b5yOn-qsm.eQTcnu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b52dac54be66169a-0065bb7227-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:51.095 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:51.095 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:51.095 DEBUG:boto:path=/vclfcn-19/?prefix=dummy 2024-02-01T10:27:51.095 DEBUG:boto:auth_path=/vclfcn-19/?prefix=dummy 2024-02-01T10:27:51.095 DEBUG:boto:Method: GET 2024-02-01T10:27:51.095 DEBUG:boto:Path: /vclfcn-19/?prefix=dummy 2024-02-01T10:27:51.096 DEBUG:boto:Data: 2024-02-01T10:27:51.096 DEBUG:boto:Headers: {} 2024-02-01T10:27:51.096 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:51.096 DEBUG:boto:Port: 8001 2024-02-01T10:27:51.096 DEBUG:boto:Params: {} 2024-02-01T10:27:51.096 DEBUG:boto:Token: None 2024-02-01T10:27:51.096 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:51 GMT /vclfcn-19/ 2024-02-01T10:27:51.096 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:K5rNl20W7rEOmAv0y5bdQs9eT0Q= 2024-02-01T10:27:51.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:K5rNl20W7rEOmAv0y5bdQs9eT0Q=', 'Content-Length': '0'} 2024-02-01T10:27:51.101 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b82104a29eed4887-0065bb7227-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:51.101 DEBUG:boto:b'vclfcn-19dummy1000falsedummy2024-02-01T10:23:11.115Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:51.101 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:51.101 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:51.101 DEBUG:boto:path=/vclfcn-19/?prefix=dummy 2024-02-01T10:27:51.101 DEBUG:boto:auth_path=/vclfcn-19/?prefix=dummy 2024-02-01T10:27:51.102 DEBUG:boto:Method: GET 2024-02-01T10:27:51.102 DEBUG:boto:Path: /vclfcn-19/?prefix=dummy 2024-02-01T10:27:51.102 DEBUG:boto:Data: 2024-02-01T10:27:51.102 DEBUG:boto:Headers: {} 2024-02-01T10:27:51.102 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:51.102 DEBUG:boto:Port: 8000 2024-02-01T10:27:51.102 DEBUG:boto:Params: {} 2024-02-01T10:27:51.102 DEBUG:boto:Token: None 2024-02-01T10:27:51.102 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:51 GMT /vclfcn-19/ 2024-02-01T10:27:51.103 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:K5rNl20W7rEOmAv0y5bdQs9eT0Q= 2024-02-01T10:27:51.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:K5rNl20W7rEOmAv0y5bdQs9eT0Q=', 'Content-Length': '0'} 2024-02-01T10:27:51.107 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a586c964d2075b86-0065bb7227-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:51.107 DEBUG:boto:b'vclfcn-19dummy1000falsedummy2024-02-01T10:23:11.115Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:51.107 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:51.107 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:51.107 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:51.108 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:51.108 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:51.108 DEBUG:boto:Method: GET 2024-02-01T10:27:51.108 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:51.108 DEBUG:boto:Data: 2024-02-01T10:27:51.108 DEBUG:boto:Headers: {} 2024-02-01T10:27:51.108 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:51.108 DEBUG:boto:Port: 8001 2024-02-01T10:27:51.108 DEBUG:boto:Params: {} 2024-02-01T10:27:51.108 DEBUG:boto:Token: None 2024-02-01T10:27:51.109 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:51 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:51.109 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Gqw4GLkEdHsQRvDi0DlwgNPDXlc= 2024-02-01T10:27:51.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Gqw4GLkEdHsQRvDi0DlwgNPDXlc=', 'Content-Length': '0'} 2024-02-01T10:27:51.111 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cea95c9167569b07-0065bb7227-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:51.152 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:51.152 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:51.152 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:51.152 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:51.153 DEBUG:boto:Method: GET 2024-02-01T10:27:51.153 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:51.153 DEBUG:boto:Data: 2024-02-01T10:27:51.153 DEBUG:boto:Headers: {} 2024-02-01T10:27:51.153 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:51.153 DEBUG:boto:Port: 8000 2024-02-01T10:27:51.153 DEBUG:boto:Params: {} 2024-02-01T10:27:51.153 DEBUG:boto:Token: None 2024-02-01T10:27:51.153 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:51 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:51.154 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Gqw4GLkEdHsQRvDi0DlwgNPDXlc= 2024-02-01T10:27:51.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Gqw4GLkEdHsQRvDi0DlwgNPDXlc=', 'Content-Length': '0'} 2024-02-01T10:27:51.157 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004f1351743e453f3c-0065bb7227-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:51.157 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:51.157 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:51.157 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:51.158 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:51.158 DEBUG:boto:Method: HEAD 2024-02-01T10:27:51.158 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:51.158 DEBUG:boto:Data: 2024-02-01T10:27:51.158 DEBUG:boto:Headers: {} 2024-02-01T10:27:51.158 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:51.158 DEBUG:boto:Port: 8001 2024-02-01T10:27:51.158 DEBUG:boto:Params: {} 2024-02-01T10:27:51.158 DEBUG:boto:Token: None 2024-02-01T10:27:51.158 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:51 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:51.159 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lE5il/b8F0X9t8PfZ4i75Bng0NY= 2024-02-01T10:27:51.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lE5il/b8F0X9t8PfZ4i75Bng0NY=', 'Content-Length': '0'} 2024-02-01T10:27:51.161 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ec7618544899c8f6-0065bb7227-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:51.162 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:51.162 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:51.162 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:51.162 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:51.162 DEBUG:boto:Method: HEAD 2024-02-01T10:27:51.162 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:51.162 DEBUG:boto:Data: 2024-02-01T10:27:51.162 DEBUG:boto:Headers: {} 2024-02-01T10:27:51.162 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:51.162 DEBUG:boto:Port: 8000 2024-02-01T10:27:51.163 DEBUG:boto:Params: {} 2024-02-01T10:27:51.163 DEBUG:boto:Token: None 2024-02-01T10:27:51.163 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:51 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:51.163 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lE5il/b8F0X9t8PfZ4i75Bng0NY= 2024-02-01T10:27:51.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lE5il/b8F0X9t8PfZ4i75Bng0NY=', 'Content-Length': '0'} 2024-02-01T10:27:51.166 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a97615cce94cef45-0065bb7227-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:51.166 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:51.166 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:51.166 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:51.166 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:51.166 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:51.166 DEBUG:boto:Method: GET 2024-02-01T10:27:51.167 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:51.167 DEBUG:boto:Data: 2024-02-01T10:27:51.167 DEBUG:boto:Headers: {} 2024-02-01T10:27:51.167 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:51.167 DEBUG:boto:Port: 8001 2024-02-01T10:27:51.167 DEBUG:boto:Params: {} 2024-02-01T10:27:51.167 DEBUG:boto:Token: None 2024-02-01T10:27:51.167 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:51 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:51.167 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Gqw4GLkEdHsQRvDi0DlwgNPDXlc= 2024-02-01T10:27:51.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Gqw4GLkEdHsQRvDi0DlwgNPDXlc=', 'Content-Length': '0'} 2024-02-01T10:27:51.171 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ff23b7e98773bf7c-0065bb7227-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:51.212 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:51.212 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:51.212 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:51.213 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:51.213 DEBUG:boto:Method: GET 2024-02-01T10:27:51.213 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:51.213 DEBUG:boto:Data: 2024-02-01T10:27:51.213 DEBUG:boto:Headers: {} 2024-02-01T10:27:51.213 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:51.213 DEBUG:boto:Port: 8000 2024-02-01T10:27:51.213 DEBUG:boto:Params: {} 2024-02-01T10:27:51.213 DEBUG:boto:Token: None 2024-02-01T10:27:51.213 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:51 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:51.214 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Gqw4GLkEdHsQRvDi0DlwgNPDXlc= 2024-02-01T10:27:51.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Gqw4GLkEdHsQRvDi0DlwgNPDXlc=', 'Content-Length': '0'} 2024-02-01T10:27:51.217 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000077a8ca29763f6f7a-0065bb7227-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:51.217 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:51.217 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:51.217 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:51.217 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:51.217 DEBUG:boto:Method: GET 2024-02-01T10:27:51.218 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:51.218 DEBUG:boto:Data: 2024-02-01T10:27:51.218 DEBUG:boto:Headers: {} 2024-02-01T10:27:51.218 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:51.218 DEBUG:boto:Port: 8001 2024-02-01T10:27:51.218 DEBUG:boto:Params: {} 2024-02-01T10:27:51.218 DEBUG:boto:Token: None 2024-02-01T10:27:51.218 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:51 GMT /vclfcn-19/ 2024-02-01T10:27:51.219 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:K5rNl20W7rEOmAv0y5bdQs9eT0Q= 2024-02-01T10:27:51.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:K5rNl20W7rEOmAv0y5bdQs9eT0Q=', 'Content-Length': '0'} 2024-02-01T10:27:51.224 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091d81a233dda19e7-0065bb7227-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:51.268 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:51.269 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:51.269 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:51.269 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:51.269 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:51.269 DEBUG:boto:Method: GET 2024-02-01T10:27:51.270 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:51.270 DEBUG:boto:Data: 2024-02-01T10:27:51.270 DEBUG:boto:Headers: {} 2024-02-01T10:27:51.270 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:51.270 DEBUG:boto:Port: 8000 2024-02-01T10:27:51.270 DEBUG:boto:Params: {} 2024-02-01T10:27:51.270 DEBUG:boto:Token: None 2024-02-01T10:27:51.271 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:51 GMT /vclfcn-19/ 2024-02-01T10:27:51.271 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:K5rNl20W7rEOmAv0y5bdQs9eT0Q= 2024-02-01T10:27:51.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:K5rNl20W7rEOmAv0y5bdQs9eT0Q=', 'Content-Length': '0'} 2024-02-01T10:27:51.276 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086c5ab6d9bdeebef-0065bb7227-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:51.276 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:51.277 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:51.277 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:51.277 DEBUG:boto:Method: HEAD 2024-02-01T10:27:51.277 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:51.277 DEBUG:boto:Data: 2024-02-01T10:27:51.277 DEBUG:boto:Headers: {} 2024-02-01T10:27:51.277 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:51.277 DEBUG:boto:Port: 8001 2024-02-01T10:27:51.277 DEBUG:boto:Params: {} 2024-02-01T10:27:51.278 DEBUG:boto:Token: None 2024-02-01T10:27:51.278 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:51 GMT /vclfcn-19/obj 2024-02-01T10:27:51.278 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:v1EHj6FlUUNuPyt018goDUboOmY= 2024-02-01T10:27:51.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:v1EHj6FlUUNuPyt018goDUboOmY=', 'Content-Length': '0'} 2024-02-01T10:27:51.284 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c85c026a7fd8904a-0065bb7227-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:51.285 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:51.285 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:51.285 DEBUG:boto:Method: HEAD 2024-02-01T10:27:51.285 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:51.285 DEBUG:boto:Data: 2024-02-01T10:27:51.285 DEBUG:boto:Headers: {} 2024-02-01T10:27:51.285 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:51.285 DEBUG:boto:Port: 8000 2024-02-01T10:27:51.285 DEBUG:boto:Params: {} 2024-02-01T10:27:51.286 DEBUG:boto:Token: None 2024-02-01T10:27:51.286 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:51 GMT /vclfcn-19/obj 2024-02-01T10:27:51.286 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:v1EHj6FlUUNuPyt018goDUboOmY= 2024-02-01T10:27:51.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:v1EHj6FlUUNuPyt018goDUboOmY=', 'Content-Length': '0'} 2024-02-01T10:27:51.292 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006c1408affe3e49fa-0065bb7227-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:51.292 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:51.293 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:51.293 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:51.293 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:51.293 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:51.293 DEBUG:boto:Method: GET 2024-02-01T10:27:51.293 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:51.293 DEBUG:boto:Data: 2024-02-01T10:27:51.293 DEBUG:boto:Headers: {} 2024-02-01T10:27:51.293 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:51.293 DEBUG:boto:Port: 8001 2024-02-01T10:27:51.293 DEBUG:boto:Params: {} 2024-02-01T10:27:51.294 DEBUG:boto:Token: None 2024-02-01T10:27:51.294 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:51 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:51.294 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Gqw4GLkEdHsQRvDi0DlwgNPDXlc= 2024-02-01T10:27:51.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Gqw4GLkEdHsQRvDi0DlwgNPDXlc=', 'Content-Length': '0'} 2024-02-01T10:27:51.297 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b0c05d49875a563c-0065bb7227-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:51.340 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:51.340 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:51.340 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:51.341 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:51.341 DEBUG:boto:Method: GET 2024-02-01T10:27:51.341 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:51.341 DEBUG:boto:Data: 2024-02-01T10:27:51.341 DEBUG:boto:Headers: {} 2024-02-01T10:27:51.341 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:51.342 DEBUG:boto:Port: 8000 2024-02-01T10:27:51.342 DEBUG:boto:Params: {} 2024-02-01T10:27:51.342 DEBUG:boto:Token: None 2024-02-01T10:27:51.342 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:51 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:51.342 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Gqw4GLkEdHsQRvDi0DlwgNPDXlc= 2024-02-01T10:27:51.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Gqw4GLkEdHsQRvDi0DlwgNPDXlc=', 'Content-Length': '0'} 2024-02-01T10:27:51.346 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008a02230b7c8f5ced-0065bb7227-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:51.346 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:51.346 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:51.347 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:51.347 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:51.347 DEBUG:boto:Method: GET 2024-02-01T10:27:51.347 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:51.347 DEBUG:boto:Data: 2024-02-01T10:27:51.347 DEBUG:boto:Headers: {} 2024-02-01T10:27:51.348 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:51.348 DEBUG:boto:Port: 8001 2024-02-01T10:27:51.348 DEBUG:boto:Params: {} 2024-02-01T10:27:51.348 DEBUG:boto:Token: None 2024-02-01T10:27:51.348 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:51 GMT /vclfcn-19/ 2024-02-01T10:27:51.349 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:K5rNl20W7rEOmAv0y5bdQs9eT0Q= 2024-02-01T10:27:51.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:K5rNl20W7rEOmAv0y5bdQs9eT0Q=', 'Content-Length': '0'} 2024-02-01T10:27:51.354 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076990968180eb1f4-0065bb7227-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:51.396 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:51.397 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:51.397 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:51.397 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:51.397 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:51.397 DEBUG:boto:Method: GET 2024-02-01T10:27:51.397 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:51.397 DEBUG:boto:Data: 2024-02-01T10:27:51.397 DEBUG:boto:Headers: {} 2024-02-01T10:27:51.398 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:51.399 DEBUG:boto:Port: 8000 2024-02-01T10:27:51.399 DEBUG:boto:Params: {} 2024-02-01T10:27:51.399 DEBUG:boto:Token: None 2024-02-01T10:27:51.399 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:51 GMT /vclfcn-19/ 2024-02-01T10:27:51.399 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:K5rNl20W7rEOmAv0y5bdQs9eT0Q= 2024-02-01T10:27:51.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:K5rNl20W7rEOmAv0y5bdQs9eT0Q=', 'Content-Length': '0'} 2024-02-01T10:27:51.405 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4eded3f0d3097fc-0065bb7227-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:51.406 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:51.406 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:51.407 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:51.407 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:51.407 DEBUG:boto:path=/vclfcn-19/obj?versionId=RrytQti15jy37pft0WxtgWdt4Zs7l4E 2024-02-01T10:27:51.407 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=RrytQti15jy37pft0WxtgWdt4Zs7l4E 2024-02-01T10:27:51.407 DEBUG:boto:Method: GET 2024-02-01T10:27:51.407 DEBUG:boto:Path: /vclfcn-19/obj?versionId=RrytQti15jy37pft0WxtgWdt4Zs7l4E 2024-02-01T10:27:51.407 DEBUG:boto:Data: 2024-02-01T10:27:51.407 DEBUG:boto:Headers: {} 2024-02-01T10:27:51.407 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:51.408 DEBUG:boto:Port: 8001 2024-02-01T10:27:51.408 DEBUG:boto:Params: {} 2024-02-01T10:27:51.408 DEBUG:boto:Token: None 2024-02-01T10:27:51.408 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:51 GMT /vclfcn-19/obj?versionId=RrytQti15jy37pft0WxtgWdt4Zs7l4E 2024-02-01T10:27:51.408 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:TaBznhFeXqPTziEoZq9nPL6cYXs= 2024-02-01T10:27:51.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:TaBznhFeXqPTziEoZq9nPL6cYXs=', 'Content-Length': '0'} 2024-02-01T10:27:51.411 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'RrytQti15jy37pft0WxtgWdt4Zs7l4E'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000479ab8c1df45bc79-0065bb7227-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:51.411 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:51.412 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:51.412 DEBUG:boto:path=/vclfcn-19/obj?versionId=RrytQti15jy37pft0WxtgWdt4Zs7l4E 2024-02-01T10:27:51.412 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=RrytQti15jy37pft0WxtgWdt4Zs7l4E 2024-02-01T10:27:51.412 DEBUG:boto:Method: GET 2024-02-01T10:27:51.412 DEBUG:boto:Path: /vclfcn-19/obj?versionId=RrytQti15jy37pft0WxtgWdt4Zs7l4E 2024-02-01T10:27:51.412 DEBUG:boto:Data: 2024-02-01T10:27:51.412 DEBUG:boto:Headers: {} 2024-02-01T10:27:51.412 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:51.413 DEBUG:boto:Port: 8000 2024-02-01T10:27:51.413 DEBUG:boto:Params: {} 2024-02-01T10:27:51.413 DEBUG:boto:Token: None 2024-02-01T10:27:51.413 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:51 GMT /vclfcn-19/obj?versionId=RrytQti15jy37pft0WxtgWdt4Zs7l4E 2024-02-01T10:27:51.413 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:TaBznhFeXqPTziEoZq9nPL6cYXs= 2024-02-01T10:27:51.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:TaBznhFeXqPTziEoZq9nPL6cYXs=', 'Content-Length': '0'} 2024-02-01T10:27:51.416 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'RrytQti15jy37pft0WxtgWdt4Zs7l4E'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005f840045284d0d29-0065bb7227-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:51.464 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:51.464 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:51.464 DEBUG:boto:path=/vclfcn-19/obj?versionId=RrytQti15jy37pft0WxtgWdt4Zs7l4E 2024-02-01T10:27:51.464 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=RrytQti15jy37pft0WxtgWdt4Zs7l4E 2024-02-01T10:27:51.465 DEBUG:boto:Method: HEAD 2024-02-01T10:27:51.465 DEBUG:boto:Path: /vclfcn-19/obj?versionId=RrytQti15jy37pft0WxtgWdt4Zs7l4E 2024-02-01T10:27:51.465 DEBUG:boto:Data: 2024-02-01T10:27:51.465 DEBUG:boto:Headers: {} 2024-02-01T10:27:51.465 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:51.465 DEBUG:boto:Port: 8001 2024-02-01T10:27:51.465 DEBUG:boto:Params: {} 2024-02-01T10:27:51.465 DEBUG:boto:Token: None 2024-02-01T10:27:51.465 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:51 GMT /vclfcn-19/obj?versionId=RrytQti15jy37pft0WxtgWdt4Zs7l4E 2024-02-01T10:27:51.465 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:VtGP865pDGkdMku8NpAs6EAqhJI= 2024-02-01T10:27:51.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:VtGP865pDGkdMku8NpAs6EAqhJI=', 'Content-Length': '0'} 2024-02-01T10:27:51.468 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'RrytQti15jy37pft0WxtgWdt4Zs7l4E'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a6af688f71386025-0065bb7227-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:51.469 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:51.469 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:51.469 DEBUG:boto:path=/vclfcn-19/obj?versionId=RrytQti15jy37pft0WxtgWdt4Zs7l4E 2024-02-01T10:27:51.469 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=RrytQti15jy37pft0WxtgWdt4Zs7l4E 2024-02-01T10:27:51.469 DEBUG:boto:Method: HEAD 2024-02-01T10:27:51.469 DEBUG:boto:Path: /vclfcn-19/obj?versionId=RrytQti15jy37pft0WxtgWdt4Zs7l4E 2024-02-01T10:27:51.469 DEBUG:boto:Data: 2024-02-01T10:27:51.470 DEBUG:boto:Headers: {} 2024-02-01T10:27:51.470 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:51.470 DEBUG:boto:Port: 8000 2024-02-01T10:27:51.470 DEBUG:boto:Params: {} 2024-02-01T10:27:51.470 DEBUG:boto:Token: None 2024-02-01T10:27:51.470 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:51 GMT /vclfcn-19/obj?versionId=RrytQti15jy37pft0WxtgWdt4Zs7l4E 2024-02-01T10:27:51.470 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:VtGP865pDGkdMku8NpAs6EAqhJI= 2024-02-01T10:27:51.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:VtGP865pDGkdMku8NpAs6EAqhJI=', 'Content-Length': '0'} 2024-02-01T10:27:51.473 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'RrytQti15jy37pft0WxtgWdt4Zs7l4E'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004a7f6f83bee7d985-0065bb7227-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:51.473 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:51.473 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:51.473 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:51.473 DEBUG:boto:path=/vclfcn-19/obj?versionId=RrytQti15jy37pft0WxtgWdt4Zs7l4E 2024-02-01T10:27:51.473 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=RrytQti15jy37pft0WxtgWdt4Zs7l4E 2024-02-01T10:27:51.474 DEBUG:boto:Method: GET 2024-02-01T10:27:51.474 DEBUG:boto:Path: /vclfcn-19/obj?versionId=RrytQti15jy37pft0WxtgWdt4Zs7l4E 2024-02-01T10:27:51.474 DEBUG:boto:Data: 2024-02-01T10:27:51.474 DEBUG:boto:Headers: {} 2024-02-01T10:27:51.474 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:51.474 DEBUG:boto:Port: 8001 2024-02-01T10:27:51.474 DEBUG:boto:Params: {} 2024-02-01T10:27:51.474 DEBUG:boto:Token: None 2024-02-01T10:27:51.474 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:51 GMT /vclfcn-19/obj?versionId=RrytQti15jy37pft0WxtgWdt4Zs7l4E 2024-02-01T10:27:51.474 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:TaBznhFeXqPTziEoZq9nPL6cYXs= 2024-02-01T10:27:51.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:TaBznhFeXqPTziEoZq9nPL6cYXs=', 'Content-Length': '0'} 2024-02-01T10:27:51.477 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'RrytQti15jy37pft0WxtgWdt4Zs7l4E'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008e8ce730cf0b44ed-0065bb7227-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:51.478 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:51.478 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:51.478 DEBUG:boto:path=/vclfcn-19/obj?versionId=RrytQti15jy37pft0WxtgWdt4Zs7l4E 2024-02-01T10:27:51.478 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=RrytQti15jy37pft0WxtgWdt4Zs7l4E 2024-02-01T10:27:51.478 DEBUG:boto:Method: GET 2024-02-01T10:27:51.478 DEBUG:boto:Path: /vclfcn-19/obj?versionId=RrytQti15jy37pft0WxtgWdt4Zs7l4E 2024-02-01T10:27:51.478 DEBUG:boto:Data: 2024-02-01T10:27:51.479 DEBUG:boto:Headers: {} 2024-02-01T10:27:51.479 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:51.479 DEBUG:boto:Port: 8000 2024-02-01T10:27:51.479 DEBUG:boto:Params: {} 2024-02-01T10:27:51.479 DEBUG:boto:Token: None 2024-02-01T10:27:51.479 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:51 GMT /vclfcn-19/obj?versionId=RrytQti15jy37pft0WxtgWdt4Zs7l4E 2024-02-01T10:27:51.479 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:TaBznhFeXqPTziEoZq9nPL6cYXs= 2024-02-01T10:27:51.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:TaBznhFeXqPTziEoZq9nPL6cYXs=', 'Content-Length': '0'} 2024-02-01T10:27:51.482 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'RrytQti15jy37pft0WxtgWdt4Zs7l4E'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000096e562eefff14c68-0065bb7227-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:51.528 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:51.528 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:51.528 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:51.529 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:51.529 DEBUG:boto:Method: GET 2024-02-01T10:27:51.529 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:51.529 DEBUG:boto:Data: 2024-02-01T10:27:51.529 DEBUG:boto:Headers: {} 2024-02-01T10:27:51.529 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:51.529 DEBUG:boto:Port: 8001 2024-02-01T10:27:51.529 DEBUG:boto:Params: {} 2024-02-01T10:27:51.529 DEBUG:boto:Token: None 2024-02-01T10:27:51.530 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:51 GMT /vclfcn-19/ 2024-02-01T10:27:51.530 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:K5rNl20W7rEOmAv0y5bdQs9eT0Q= 2024-02-01T10:27:51.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:K5rNl20W7rEOmAv0y5bdQs9eT0Q=', 'Content-Length': '0'} 2024-02-01T10:27:51.535 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d15727c01ca43fa-0065bb7227-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:51.535 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:51.536 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:51.536 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:51.536 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:51.536 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:51.536 DEBUG:boto:Method: GET 2024-02-01T10:27:51.536 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:51.536 DEBUG:boto:Data: 2024-02-01T10:27:51.536 DEBUG:boto:Headers: {} 2024-02-01T10:27:51.536 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:51.537 DEBUG:boto:Port: 8000 2024-02-01T10:27:51.537 DEBUG:boto:Params: {} 2024-02-01T10:27:51.537 DEBUG:boto:Token: None 2024-02-01T10:27:51.537 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:51 GMT /vclfcn-19/ 2024-02-01T10:27:51.537 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:K5rNl20W7rEOmAv0y5bdQs9eT0Q= 2024-02-01T10:27:51.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:K5rNl20W7rEOmAv0y5bdQs9eT0Q=', 'Content-Length': '0'} 2024-02-01T10:27:51.543 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b0072de20e90ab13-0065bb7227-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:51.543 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:51.544 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:51.544 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:51.544 DEBUG:boto:Method: HEAD 2024-02-01T10:27:51.544 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:51.544 DEBUG:boto:Data: 2024-02-01T10:27:51.544 DEBUG:boto:Headers: {} 2024-02-01T10:27:51.544 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:51.544 DEBUG:boto:Port: 8001 2024-02-01T10:27:51.544 DEBUG:boto:Params: {} 2024-02-01T10:27:51.545 DEBUG:boto:Token: None 2024-02-01T10:27:51.545 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:51 GMT /vclfcn-19/obj 2024-02-01T10:27:51.545 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:v1EHj6FlUUNuPyt018goDUboOmY= 2024-02-01T10:27:51.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:v1EHj6FlUUNuPyt018goDUboOmY=', 'Content-Length': '0'} 2024-02-01T10:27:51.551 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003685267ee530d8ee-0065bb7227-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:51.552 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:51.552 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:51.552 DEBUG:boto:Method: HEAD 2024-02-01T10:27:51.552 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:51.552 DEBUG:boto:Data: 2024-02-01T10:27:51.552 DEBUG:boto:Headers: {} 2024-02-01T10:27:51.552 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:51.552 DEBUG:boto:Port: 8000 2024-02-01T10:27:51.552 DEBUG:boto:Params: {} 2024-02-01T10:27:51.553 DEBUG:boto:Token: None 2024-02-01T10:27:51.553 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:51 GMT /vclfcn-19/obj 2024-02-01T10:27:51.553 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:v1EHj6FlUUNuPyt018goDUboOmY= 2024-02-01T10:27:51.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:v1EHj6FlUUNuPyt018goDUboOmY=', 'Content-Length': '0'} 2024-02-01T10:27:51.559 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000022947f4efc12e09a-0065bb7227-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:51.560 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:51.560 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:51.560 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:51.560 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:51.560 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:51.560 DEBUG:boto:Method: GET 2024-02-01T10:27:51.560 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:51.560 DEBUG:boto:Data: 2024-02-01T10:27:51.561 DEBUG:boto:Headers: {} 2024-02-01T10:27:51.561 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:51.561 DEBUG:boto:Port: 8001 2024-02-01T10:27:51.561 DEBUG:boto:Params: {} 2024-02-01T10:27:51.561 DEBUG:boto:Token: None 2024-02-01T10:27:51.561 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:51 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:51.561 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Gqw4GLkEdHsQRvDi0DlwgNPDXlc= 2024-02-01T10:27:51.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Gqw4GLkEdHsQRvDi0DlwgNPDXlc=', 'Content-Length': '0'} 2024-02-01T10:27:51.564 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000273bad0260c8b41d-0065bb7227-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:51.608 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:51.608 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:51.608 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:51.609 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:51.609 DEBUG:boto:Method: GET 2024-02-01T10:27:51.609 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:51.609 DEBUG:boto:Data: 2024-02-01T10:27:51.609 DEBUG:boto:Headers: {} 2024-02-01T10:27:51.609 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:51.609 DEBUG:boto:Port: 8000 2024-02-01T10:27:51.609 DEBUG:boto:Params: {} 2024-02-01T10:27:51.609 DEBUG:boto:Token: None 2024-02-01T10:27:51.610 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:51 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:51.610 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Gqw4GLkEdHsQRvDi0DlwgNPDXlc= 2024-02-01T10:27:51.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Gqw4GLkEdHsQRvDi0DlwgNPDXlc=', 'Content-Length': '0'} 2024-02-01T10:27:51.613 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f583fbbb81fb84ab-0065bb7227-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:51.613 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:51.614 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:51.614 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:51.614 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:51.614 DEBUG:boto:Method: GET 2024-02-01T10:27:51.614 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:51.614 DEBUG:boto:Data: 2024-02-01T10:27:51.614 DEBUG:boto:Headers: {} 2024-02-01T10:27:51.614 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:51.614 DEBUG:boto:Port: 8001 2024-02-01T10:27:51.614 DEBUG:boto:Params: {} 2024-02-01T10:27:51.615 DEBUG:boto:Token: None 2024-02-01T10:27:51.615 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:51 GMT /vclfcn-19/ 2024-02-01T10:27:51.615 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:K5rNl20W7rEOmAv0y5bdQs9eT0Q= 2024-02-01T10:27:51.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:K5rNl20W7rEOmAv0y5bdQs9eT0Q=', 'Content-Length': '0'} 2024-02-01T10:27:51.620 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c11741e8b7997f1-0065bb7227-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:51.620 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:51.621 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:51.621 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:51.621 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:51.621 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:51.621 DEBUG:boto:Method: GET 2024-02-01T10:27:51.621 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:51.621 DEBUG:boto:Data: 2024-02-01T10:27:51.621 DEBUG:boto:Headers: {} 2024-02-01T10:27:51.622 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:51.622 DEBUG:boto:Port: 8000 2024-02-01T10:27:51.622 DEBUG:boto:Params: {} 2024-02-01T10:27:51.622 DEBUG:boto:Token: None 2024-02-01T10:27:51.622 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:51 GMT /vclfcn-19/ 2024-02-01T10:27:51.622 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:K5rNl20W7rEOmAv0y5bdQs9eT0Q= 2024-02-01T10:27:51.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:K5rNl20W7rEOmAv0y5bdQs9eT0Q=', 'Content-Length': '0'} 2024-02-01T10:27:51.627 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000372c0bfafcdda371-0065bb7227-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:51.672 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:51.673 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:51.673 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:51.673 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:51.673 DEBUG:boto:path=/vclfcn-19/obj?versionId=fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u 2024-02-01T10:27:51.673 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u 2024-02-01T10:27:51.674 DEBUG:boto:Method: GET 2024-02-01T10:27:51.674 DEBUG:boto:Path: /vclfcn-19/obj?versionId=fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u 2024-02-01T10:27:51.674 DEBUG:boto:Data: 2024-02-01T10:27:51.674 DEBUG:boto:Headers: {} 2024-02-01T10:27:51.674 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:51.674 DEBUG:boto:Port: 8001 2024-02-01T10:27:51.674 DEBUG:boto:Params: {} 2024-02-01T10:27:51.674 DEBUG:boto:Token: None 2024-02-01T10:27:51.674 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:51 GMT /vclfcn-19/obj?versionId=fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u 2024-02-01T10:27:51.675 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:aA6PHUT7DYf6tSfwDQJGhuoFt+w= 2024-02-01T10:27:51.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:aA6PHUT7DYf6tSfwDQJGhuoFt+w=', 'Content-Length': '0'} 2024-02-01T10:27:51.679 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000023276db8568b868a-0065bb7227-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:51.680 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:51.680 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:51.680 DEBUG:boto:path=/vclfcn-19/obj?versionId=fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u 2024-02-01T10:27:51.681 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u 2024-02-01T10:27:51.681 DEBUG:boto:Method: GET 2024-02-01T10:27:51.681 DEBUG:boto:Path: /vclfcn-19/obj?versionId=fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u 2024-02-01T10:27:51.681 DEBUG:boto:Data: 2024-02-01T10:27:51.681 DEBUG:boto:Headers: {} 2024-02-01T10:27:51.682 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:51.682 DEBUG:boto:Port: 8000 2024-02-01T10:27:51.682 DEBUG:boto:Params: {} 2024-02-01T10:27:51.683 DEBUG:boto:Token: None 2024-02-01T10:27:51.683 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:51 GMT /vclfcn-19/obj?versionId=fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u 2024-02-01T10:27:51.683 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:aA6PHUT7DYf6tSfwDQJGhuoFt+w= 2024-02-01T10:27:51.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:aA6PHUT7DYf6tSfwDQJGhuoFt+w=', 'Content-Length': '0'} 2024-02-01T10:27:51.686 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000004ccbc44a785bfdf-0065bb7227-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:51.728 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:51.729 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:51.729 DEBUG:boto:path=/vclfcn-19/obj?versionId=fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u 2024-02-01T10:27:51.729 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u 2024-02-01T10:27:51.729 DEBUG:boto:Method: HEAD 2024-02-01T10:27:51.729 DEBUG:boto:Path: /vclfcn-19/obj?versionId=fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u 2024-02-01T10:27:51.730 DEBUG:boto:Data: 2024-02-01T10:27:51.730 DEBUG:boto:Headers: {} 2024-02-01T10:27:51.730 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:51.730 DEBUG:boto:Port: 8001 2024-02-01T10:27:51.730 DEBUG:boto:Params: {} 2024-02-01T10:27:51.731 DEBUG:boto:Token: None 2024-02-01T10:27:51.731 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:51 GMT /vclfcn-19/obj?versionId=fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u 2024-02-01T10:27:51.731 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:u3OTbshCtuUQ0VjqvZVReTKpc1A= 2024-02-01T10:27:51.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:u3OTbshCtuUQ0VjqvZVReTKpc1A=', 'Content-Length': '0'} 2024-02-01T10:27:51.735 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008517ea8c35f1c487-0065bb7227-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:51.735 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:51.735 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:51.736 DEBUG:boto:path=/vclfcn-19/obj?versionId=fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u 2024-02-01T10:27:51.736 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u 2024-02-01T10:27:51.736 DEBUG:boto:Method: HEAD 2024-02-01T10:27:51.736 DEBUG:boto:Path: /vclfcn-19/obj?versionId=fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u 2024-02-01T10:27:51.736 DEBUG:boto:Data: 2024-02-01T10:27:51.737 DEBUG:boto:Headers: {} 2024-02-01T10:27:51.737 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:51.737 DEBUG:boto:Port: 8000 2024-02-01T10:27:51.737 DEBUG:boto:Params: {} 2024-02-01T10:27:51.738 DEBUG:boto:Token: None 2024-02-01T10:27:51.738 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:51 GMT /vclfcn-19/obj?versionId=fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u 2024-02-01T10:27:51.738 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:u3OTbshCtuUQ0VjqvZVReTKpc1A= 2024-02-01T10:27:51.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:u3OTbshCtuUQ0VjqvZVReTKpc1A=', 'Content-Length': '0'} 2024-02-01T10:27:51.741 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007524286c5d24f543-0065bb7227-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:51.742 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:51.742 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:51.742 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:51.743 DEBUG:boto:path=/vclfcn-19/obj?versionId=fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u 2024-02-01T10:27:51.743 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u 2024-02-01T10:27:51.743 DEBUG:boto:Method: GET 2024-02-01T10:27:51.743 DEBUG:boto:Path: /vclfcn-19/obj?versionId=fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u 2024-02-01T10:27:51.744 DEBUG:boto:Data: 2024-02-01T10:27:51.744 DEBUG:boto:Headers: {} 2024-02-01T10:27:51.744 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:51.744 DEBUG:boto:Port: 8001 2024-02-01T10:27:51.744 DEBUG:boto:Params: {} 2024-02-01T10:27:51.744 DEBUG:boto:Token: None 2024-02-01T10:27:51.745 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:51 GMT /vclfcn-19/obj?versionId=fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u 2024-02-01T10:27:51.745 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:aA6PHUT7DYf6tSfwDQJGhuoFt+w= 2024-02-01T10:27:51.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:aA6PHUT7DYf6tSfwDQJGhuoFt+w=', 'Content-Length': '0'} 2024-02-01T10:27:51.748 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b41ae01ad0755ebc-0065bb7227-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:51.796 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:51.796 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:51.797 DEBUG:boto:path=/vclfcn-19/obj?versionId=fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u 2024-02-01T10:27:51.797 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u 2024-02-01T10:27:51.797 DEBUG:boto:Method: GET 2024-02-01T10:27:51.797 DEBUG:boto:Path: /vclfcn-19/obj?versionId=fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u 2024-02-01T10:27:51.797 DEBUG:boto:Data: 2024-02-01T10:27:51.797 DEBUG:boto:Headers: {} 2024-02-01T10:27:51.798 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:51.798 DEBUG:boto:Port: 8000 2024-02-01T10:27:51.798 DEBUG:boto:Params: {} 2024-02-01T10:27:51.798 DEBUG:boto:Token: None 2024-02-01T10:27:51.799 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:51 GMT /vclfcn-19/obj?versionId=fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u 2024-02-01T10:27:51.799 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:aA6PHUT7DYf6tSfwDQJGhuoFt+w= 2024-02-01T10:27:51.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:aA6PHUT7DYf6tSfwDQJGhuoFt+w=', 'Content-Length': '0'} 2024-02-01T10:27:51.802 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'fzdlI7n.dsuwfAOSBNyk84Wd7l5p70u'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bd9a21962dc088ad-0065bb7227-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:51.844 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:51.845 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:51.845 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:51.845 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:51.845 DEBUG:boto:Method: GET 2024-02-01T10:27:51.845 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:51.845 DEBUG:boto:Data: 2024-02-01T10:27:51.846 DEBUG:boto:Headers: {} 2024-02-01T10:27:51.846 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:51.846 DEBUG:boto:Port: 8001 2024-02-01T10:27:51.846 DEBUG:boto:Params: {} 2024-02-01T10:27:51.846 DEBUG:boto:Token: None 2024-02-01T10:27:51.847 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:51 GMT /vclfcn-19/ 2024-02-01T10:27:51.847 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:K5rNl20W7rEOmAv0y5bdQs9eT0Q= 2024-02-01T10:27:51.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:K5rNl20W7rEOmAv0y5bdQs9eT0Q=', 'Content-Length': '0'} 2024-02-01T10:27:51.853 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8ceadce0eb161e3-0065bb7227-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:51.853 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:51.854 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:51.854 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:51.854 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:51.854 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:51.854 DEBUG:boto:Method: GET 2024-02-01T10:27:51.854 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:51.855 DEBUG:boto:Data: 2024-02-01T10:27:51.855 DEBUG:boto:Headers: {} 2024-02-01T10:27:51.855 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:51.855 DEBUG:boto:Port: 8000 2024-02-01T10:27:51.855 DEBUG:boto:Params: {} 2024-02-01T10:27:51.855 DEBUG:boto:Token: None 2024-02-01T10:27:51.856 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:51 GMT /vclfcn-19/ 2024-02-01T10:27:51.856 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:K5rNl20W7rEOmAv0y5bdQs9eT0Q= 2024-02-01T10:27:51.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:K5rNl20W7rEOmAv0y5bdQs9eT0Q=', 'Content-Length': '0'} 2024-02-01T10:27:51.862 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000050c5bf3c8a075b6-0065bb7227-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:51.862 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:51.863 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:51.863 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:51.863 DEBUG:boto:Method: HEAD 2024-02-01T10:27:51.864 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:51.864 DEBUG:boto:Data: 2024-02-01T10:27:51.864 DEBUG:boto:Headers: {} 2024-02-01T10:27:51.864 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:51.864 DEBUG:boto:Port: 8001 2024-02-01T10:27:51.864 DEBUG:boto:Params: {} 2024-02-01T10:27:51.864 DEBUG:boto:Token: None 2024-02-01T10:27:51.867 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:51 GMT /vclfcn-19/obj 2024-02-01T10:27:51.867 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:v1EHj6FlUUNuPyt018goDUboOmY= 2024-02-01T10:27:51.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:v1EHj6FlUUNuPyt018goDUboOmY=', 'Content-Length': '0'} 2024-02-01T10:27:51.874 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000015b621b4f9e5b53d-0065bb7227-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:51.875 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:51.875 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:51.875 DEBUG:boto:Method: HEAD 2024-02-01T10:27:51.875 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:51.875 DEBUG:boto:Data: 2024-02-01T10:27:51.876 DEBUG:boto:Headers: {} 2024-02-01T10:27:51.876 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:51.876 DEBUG:boto:Port: 8000 2024-02-01T10:27:51.876 DEBUG:boto:Params: {} 2024-02-01T10:27:51.877 DEBUG:boto:Token: None 2024-02-01T10:27:51.877 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:51 GMT /vclfcn-19/obj 2024-02-01T10:27:51.877 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:v1EHj6FlUUNuPyt018goDUboOmY= 2024-02-01T10:27:51.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:v1EHj6FlUUNuPyt018goDUboOmY=', 'Content-Length': '0'} 2024-02-01T10:27:51.884 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008a831eb722a0df84-0065bb7227-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:51.884 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:51.884 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:51.885 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:51.885 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:51.885 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:51.885 DEBUG:boto:Method: GET 2024-02-01T10:27:51.885 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:51.886 DEBUG:boto:Data: 2024-02-01T10:27:51.886 DEBUG:boto:Headers: {} 2024-02-01T10:27:51.886 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:51.886 DEBUG:boto:Port: 8001 2024-02-01T10:27:51.886 DEBUG:boto:Params: {} 2024-02-01T10:27:51.887 DEBUG:boto:Token: None 2024-02-01T10:27:51.887 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:51 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:51.887 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Gqw4GLkEdHsQRvDi0DlwgNPDXlc= 2024-02-01T10:27:51.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Gqw4GLkEdHsQRvDi0DlwgNPDXlc=', 'Content-Length': '0'} 2024-02-01T10:27:51.890 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000090ef7332ed2cc5f3-0065bb7227-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:51.932 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:51.933 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:51.933 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:51.933 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:51.933 DEBUG:boto:Method: GET 2024-02-01T10:27:51.933 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:51.934 DEBUG:boto:Data: 2024-02-01T10:27:51.934 DEBUG:boto:Headers: {} 2024-02-01T10:27:51.934 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:51.934 DEBUG:boto:Port: 8000 2024-02-01T10:27:51.934 DEBUG:boto:Params: {} 2024-02-01T10:27:51.935 DEBUG:boto:Token: None 2024-02-01T10:27:51.935 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:51 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:51.935 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Gqw4GLkEdHsQRvDi0DlwgNPDXlc= 2024-02-01T10:27:51.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Gqw4GLkEdHsQRvDi0DlwgNPDXlc=', 'Content-Length': '0'} 2024-02-01T10:27:51.939 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000a7bf85d77edefee-0065bb7227-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:51.939 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:51.939 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:51.940 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:51.940 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:51.940 DEBUG:boto:Method: GET 2024-02-01T10:27:51.940 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:51.941 DEBUG:boto:Data: 2024-02-01T10:27:51.941 DEBUG:boto:Headers: {} 2024-02-01T10:27:51.941 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:51.941 DEBUG:boto:Port: 8001 2024-02-01T10:27:51.941 DEBUG:boto:Params: {} 2024-02-01T10:27:51.942 DEBUG:boto:Token: None 2024-02-01T10:27:51.942 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:51 GMT /vclfcn-19/ 2024-02-01T10:27:51.942 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:K5rNl20W7rEOmAv0y5bdQs9eT0Q= 2024-02-01T10:27:51.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:K5rNl20W7rEOmAv0y5bdQs9eT0Q=', 'Content-Length': '0'} 2024-02-01T10:27:51.948 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000646deff64993ee5d-0065bb7227-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:51.948 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:51.949 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:51.949 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:51.949 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:51.950 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:51.950 DEBUG:boto:Method: GET 2024-02-01T10:27:51.950 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:51.950 DEBUG:boto:Data: 2024-02-01T10:27:51.950 DEBUG:boto:Headers: {} 2024-02-01T10:27:51.950 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:51.951 DEBUG:boto:Port: 8000 2024-02-01T10:27:51.951 DEBUG:boto:Params: {} 2024-02-01T10:27:51.951 DEBUG:boto:Token: None 2024-02-01T10:27:51.952 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:51 GMT /vclfcn-19/ 2024-02-01T10:27:51.952 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:K5rNl20W7rEOmAv0y5bdQs9eT0Q= 2024-02-01T10:27:51.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:K5rNl20W7rEOmAv0y5bdQs9eT0Q=', 'Content-Length': '0'} 2024-02-01T10:27:51.957 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000003a2e34ee7672591-0065bb7227-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:51.958 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:51.958 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:51.959 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:51.959 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:51.959 DEBUG:boto:path=/vclfcn-19/obj?versionId=UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc 2024-02-01T10:27:51.959 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc 2024-02-01T10:27:51.959 DEBUG:boto:Method: GET 2024-02-01T10:27:51.960 DEBUG:boto:Path: /vclfcn-19/obj?versionId=UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc 2024-02-01T10:27:51.960 DEBUG:boto:Data: 2024-02-01T10:27:51.960 DEBUG:boto:Headers: {} 2024-02-01T10:27:51.960 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:51.960 DEBUG:boto:Port: 8001 2024-02-01T10:27:51.961 DEBUG:boto:Params: {} 2024-02-01T10:27:51.961 DEBUG:boto:Token: None 2024-02-01T10:27:51.961 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:51 GMT /vclfcn-19/obj?versionId=UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc 2024-02-01T10:27:51.961 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:z7HHD2prBP75rub1a+s16N82fQw= 2024-02-01T10:27:51.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:z7HHD2prBP75rub1a+s16N82fQw=', 'Content-Length': '0'} 2024-02-01T10:27:51.965 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002c683c9b2d2586a6-0065bb7227-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:51.965 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:51.966 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:51.966 DEBUG:boto:path=/vclfcn-19/obj?versionId=UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc 2024-02-01T10:27:51.966 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc 2024-02-01T10:27:51.966 DEBUG:boto:Method: GET 2024-02-01T10:27:51.966 DEBUG:boto:Path: /vclfcn-19/obj?versionId=UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc 2024-02-01T10:27:51.966 DEBUG:boto:Data: 2024-02-01T10:27:51.967 DEBUG:boto:Headers: {} 2024-02-01T10:27:51.967 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:51.967 DEBUG:boto:Port: 8000 2024-02-01T10:27:51.967 DEBUG:boto:Params: {} 2024-02-01T10:27:51.967 DEBUG:boto:Token: None 2024-02-01T10:27:51.968 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:51 GMT /vclfcn-19/obj?versionId=UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc 2024-02-01T10:27:51.968 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:z7HHD2prBP75rub1a+s16N82fQw= 2024-02-01T10:27:51.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:z7HHD2prBP75rub1a+s16N82fQw=', 'Content-Length': '0'} 2024-02-01T10:27:51.971 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004408a39016334401-0065bb7227-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:52.012 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:52.013 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:52.013 DEBUG:boto:path=/vclfcn-19/obj?versionId=UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc 2024-02-01T10:27:52.013 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc 2024-02-01T10:27:52.013 DEBUG:boto:Method: HEAD 2024-02-01T10:27:52.013 DEBUG:boto:Path: /vclfcn-19/obj?versionId=UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc 2024-02-01T10:27:52.014 DEBUG:boto:Data: 2024-02-01T10:27:52.014 DEBUG:boto:Headers: {} 2024-02-01T10:27:52.014 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:52.014 DEBUG:boto:Port: 8001 2024-02-01T10:27:52.014 DEBUG:boto:Params: {} 2024-02-01T10:27:52.015 DEBUG:boto:Token: None 2024-02-01T10:27:52.015 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:52 GMT /vclfcn-19/obj?versionId=UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc 2024-02-01T10:27:52.015 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Z0MZazdp8u2gTS5tfKd0Bvow/sU= 2024-02-01T10:27:52.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Z0MZazdp8u2gTS5tfKd0Bvow/sU=', 'Content-Length': '0'} 2024-02-01T10:27:52.019 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000498fd010db0c29f7-0065bb7228-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:52.019 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:52.020 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:52.020 DEBUG:boto:path=/vclfcn-19/obj?versionId=UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc 2024-02-01T10:27:52.020 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc 2024-02-01T10:27:52.021 DEBUG:boto:Method: HEAD 2024-02-01T10:27:52.021 DEBUG:boto:Path: /vclfcn-19/obj?versionId=UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc 2024-02-01T10:27:52.021 DEBUG:boto:Data: 2024-02-01T10:27:52.021 DEBUG:boto:Headers: {} 2024-02-01T10:27:52.021 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:52.021 DEBUG:boto:Port: 8000 2024-02-01T10:27:52.021 DEBUG:boto:Params: {} 2024-02-01T10:27:52.022 DEBUG:boto:Token: None 2024-02-01T10:27:52.022 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:52 GMT /vclfcn-19/obj?versionId=UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc 2024-02-01T10:27:52.022 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Z0MZazdp8u2gTS5tfKd0Bvow/sU= 2024-02-01T10:27:52.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Z0MZazdp8u2gTS5tfKd0Bvow/sU=', 'Content-Length': '0'} 2024-02-01T10:27:52.025 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dad454c4e8c4c0e5-0065bb7228-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:52.026 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:52.026 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:52.026 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:52.026 DEBUG:boto:path=/vclfcn-19/obj?versionId=UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc 2024-02-01T10:27:52.026 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc 2024-02-01T10:27:52.027 DEBUG:boto:Method: GET 2024-02-01T10:27:52.027 DEBUG:boto:Path: /vclfcn-19/obj?versionId=UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc 2024-02-01T10:27:52.027 DEBUG:boto:Data: 2024-02-01T10:27:52.027 DEBUG:boto:Headers: {} 2024-02-01T10:27:52.027 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:52.027 DEBUG:boto:Port: 8001 2024-02-01T10:27:52.028 DEBUG:boto:Params: {} 2024-02-01T10:27:52.028 DEBUG:boto:Token: None 2024-02-01T10:27:52.029 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:52 GMT /vclfcn-19/obj?versionId=UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc 2024-02-01T10:27:52.029 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:b8WhHLvBSAEjDI4+uJ74KuL/KXE= 2024-02-01T10:27:52.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:b8WhHLvBSAEjDI4+uJ74KuL/KXE=', 'Content-Length': '0'} 2024-02-01T10:27:52.033 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a2e15b8266faadd6-0065bb7228-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:52.076 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:52.076 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:52.077 DEBUG:boto:path=/vclfcn-19/obj?versionId=UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc 2024-02-01T10:27:52.077 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc 2024-02-01T10:27:52.077 DEBUG:boto:Method: GET 2024-02-01T10:27:52.077 DEBUG:boto:Path: /vclfcn-19/obj?versionId=UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc 2024-02-01T10:27:52.077 DEBUG:boto:Data: 2024-02-01T10:27:52.077 DEBUG:boto:Headers: {} 2024-02-01T10:27:52.078 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:52.078 DEBUG:boto:Port: 8000 2024-02-01T10:27:52.078 DEBUG:boto:Params: {} 2024-02-01T10:27:52.078 DEBUG:boto:Token: None 2024-02-01T10:27:52.078 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:52 GMT /vclfcn-19/obj?versionId=UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc 2024-02-01T10:27:52.079 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:b8WhHLvBSAEjDI4+uJ74KuL/KXE= 2024-02-01T10:27:52.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:b8WhHLvBSAEjDI4+uJ74KuL/KXE=', 'Content-Length': '0'} 2024-02-01T10:27:52.082 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'UOa9NAxhXbH7RR3jkzlpXPh.fGAfBJc'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000969b6bc511151a5-0065bb7228-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:52.082 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:52.083 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:52.083 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:52.084 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:52.084 DEBUG:boto:Method: GET 2024-02-01T10:27:52.084 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:52.084 DEBUG:boto:Data: 2024-02-01T10:27:52.084 DEBUG:boto:Headers: {} 2024-02-01T10:27:52.085 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:52.085 DEBUG:boto:Port: 8001 2024-02-01T10:27:52.085 DEBUG:boto:Params: {} 2024-02-01T10:27:52.085 DEBUG:boto:Token: None 2024-02-01T10:27:52.085 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:52 GMT /vclfcn-19/ 2024-02-01T10:27:52.086 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:PkpV0vSW9AWl7CsbDW2ArF1aM4s= 2024-02-01T10:27:52.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:PkpV0vSW9AWl7CsbDW2ArF1aM4s=', 'Content-Length': '0'} 2024-02-01T10:27:52.091 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db5788963235f07b-0065bb7228-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:52.136 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:52.137 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:52.137 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:52.137 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:52.138 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:52.138 DEBUG:boto:Method: GET 2024-02-01T10:27:52.138 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:52.138 DEBUG:boto:Data: 2024-02-01T10:27:52.138 DEBUG:boto:Headers: {} 2024-02-01T10:27:52.139 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:52.139 DEBUG:boto:Port: 8000 2024-02-01T10:27:52.139 DEBUG:boto:Params: {} 2024-02-01T10:27:52.139 DEBUG:boto:Token: None 2024-02-01T10:27:52.140 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:52 GMT /vclfcn-19/ 2024-02-01T10:27:52.140 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:PkpV0vSW9AWl7CsbDW2ArF1aM4s= 2024-02-01T10:27:52.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:PkpV0vSW9AWl7CsbDW2ArF1aM4s=', 'Content-Length': '0'} 2024-02-01T10:27:52.145 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bbdf2c7a824858ce-0065bb7228-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:52.146 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:52.146 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:52.146 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:52.147 DEBUG:boto:Method: HEAD 2024-02-01T10:27:52.147 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:52.147 DEBUG:boto:Data: 2024-02-01T10:27:52.147 DEBUG:boto:Headers: {} 2024-02-01T10:27:52.147 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:52.147 DEBUG:boto:Port: 8001 2024-02-01T10:27:52.148 DEBUG:boto:Params: {} 2024-02-01T10:27:52.148 DEBUG:boto:Token: None 2024-02-01T10:27:52.148 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:52 GMT /vclfcn-19/obj 2024-02-01T10:27:52.149 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RNzIXYrJWDY36C/3XfLkBQwUf28= 2024-02-01T10:27:52.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RNzIXYrJWDY36C/3XfLkBQwUf28=', 'Content-Length': '0'} 2024-02-01T10:27:52.155 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000062e8ddff40eb5668-0065bb7228-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:52.155 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:52.155 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:52.156 DEBUG:boto:Method: HEAD 2024-02-01T10:27:52.156 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:52.156 DEBUG:boto:Data: 2024-02-01T10:27:52.156 DEBUG:boto:Headers: {} 2024-02-01T10:27:52.156 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:52.156 DEBUG:boto:Port: 8000 2024-02-01T10:27:52.156 DEBUG:boto:Params: {} 2024-02-01T10:27:52.157 DEBUG:boto:Token: None 2024-02-01T10:27:52.157 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:52 GMT /vclfcn-19/obj 2024-02-01T10:27:52.157 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RNzIXYrJWDY36C/3XfLkBQwUf28= 2024-02-01T10:27:52.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RNzIXYrJWDY36C/3XfLkBQwUf28=', 'Content-Length': '0'} 2024-02-01T10:27:52.163 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eb3c669767f8f1b8-0065bb7228-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:52.163 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:52.163 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:52.164 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:52.164 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:52.164 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:52.164 DEBUG:boto:Method: GET 2024-02-01T10:27:52.164 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:52.164 DEBUG:boto:Data: 2024-02-01T10:27:52.164 DEBUG:boto:Headers: {} 2024-02-01T10:27:52.164 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:52.164 DEBUG:boto:Port: 8001 2024-02-01T10:27:52.164 DEBUG:boto:Params: {} 2024-02-01T10:27:52.164 DEBUG:boto:Token: None 2024-02-01T10:27:52.165 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:52 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:52.165 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:pkoNRLw4Dl5AKxWsTpH9KwNqd8M= 2024-02-01T10:27:52.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:pkoNRLw4Dl5AKxWsTpH9KwNqd8M=', 'Content-Length': '0'} 2024-02-01T10:27:52.168 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000092e81e633ea8157e-0065bb7228-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:52.208 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:52.208 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:52.208 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:52.208 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:52.209 DEBUG:boto:Method: GET 2024-02-01T10:27:52.209 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:52.209 DEBUG:boto:Data: 2024-02-01T10:27:52.209 DEBUG:boto:Headers: {} 2024-02-01T10:27:52.209 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:52.209 DEBUG:boto:Port: 8000 2024-02-01T10:27:52.209 DEBUG:boto:Params: {} 2024-02-01T10:27:52.209 DEBUG:boto:Token: None 2024-02-01T10:27:52.209 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:52 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:52.210 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:pkoNRLw4Dl5AKxWsTpH9KwNqd8M= 2024-02-01T10:27:52.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:pkoNRLw4Dl5AKxWsTpH9KwNqd8M=', 'Content-Length': '0'} 2024-02-01T10:27:52.213 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000067409133e2fa0c2-0065bb7228-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:52.214 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:52.214 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:52.214 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:52.214 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:52.214 DEBUG:boto:Method: GET 2024-02-01T10:27:52.214 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:52.214 DEBUG:boto:Data: 2024-02-01T10:27:52.215 DEBUG:boto:Headers: {} 2024-02-01T10:27:52.215 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:52.215 DEBUG:boto:Port: 8001 2024-02-01T10:27:52.215 DEBUG:boto:Params: {} 2024-02-01T10:27:52.215 DEBUG:boto:Token: None 2024-02-01T10:27:52.215 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:52 GMT /vclfcn-19/ 2024-02-01T10:27:52.215 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:PkpV0vSW9AWl7CsbDW2ArF1aM4s= 2024-02-01T10:27:52.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:PkpV0vSW9AWl7CsbDW2ArF1aM4s=', 'Content-Length': '0'} 2024-02-01T10:27:52.221 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000566c783c7ce3163b-0065bb7228-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:52.264 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:52.264 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:52.265 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:52.265 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:52.265 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:52.265 DEBUG:boto:Method: GET 2024-02-01T10:27:52.265 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:52.265 DEBUG:boto:Data: 2024-02-01T10:27:52.265 DEBUG:boto:Headers: {} 2024-02-01T10:27:52.265 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:52.266 DEBUG:boto:Port: 8000 2024-02-01T10:27:52.266 DEBUG:boto:Params: {} 2024-02-01T10:27:52.266 DEBUG:boto:Token: None 2024-02-01T10:27:52.266 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:52 GMT /vclfcn-19/ 2024-02-01T10:27:52.266 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:PkpV0vSW9AWl7CsbDW2ArF1aM4s= 2024-02-01T10:27:52.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:PkpV0vSW9AWl7CsbDW2ArF1aM4s=', 'Content-Length': '0'} 2024-02-01T10:27:52.271 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ab0a6d81c7267b4-0065bb7228-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:52.272 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:52.272 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:52.275 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:52.275 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:52.275 DEBUG:boto:path=/vclfcn-19/obj?versionId=eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr. 2024-02-01T10:27:52.275 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr. 2024-02-01T10:27:52.276 DEBUG:boto:Method: GET 2024-02-01T10:27:52.276 DEBUG:boto:Path: /vclfcn-19/obj?versionId=eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr. 2024-02-01T10:27:52.276 DEBUG:boto:Data: 2024-02-01T10:27:52.277 DEBUG:boto:Headers: {} 2024-02-01T10:27:52.277 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:52.277 DEBUG:boto:Port: 8001 2024-02-01T10:27:52.277 DEBUG:boto:Params: {} 2024-02-01T10:27:52.277 DEBUG:boto:Token: None 2024-02-01T10:27:52.277 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:52 GMT /vclfcn-19/obj?versionId=eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr. 2024-02-01T10:27:52.278 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:801QIorI1iDi1R4XH67PCWmv7rY= 2024-02-01T10:27:52.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:801QIorI1iDi1R4XH67PCWmv7rY=', 'Content-Length': '0'} 2024-02-01T10:27:52.280 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr.'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bb29f55792af7ef6-0065bb7228-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:52.281 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:52.281 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:52.281 DEBUG:boto:path=/vclfcn-19/obj?versionId=eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr. 2024-02-01T10:27:52.281 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr. 2024-02-01T10:27:52.281 DEBUG:boto:Method: GET 2024-02-01T10:27:52.281 DEBUG:boto:Path: /vclfcn-19/obj?versionId=eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr. 2024-02-01T10:27:52.282 DEBUG:boto:Data: 2024-02-01T10:27:52.282 DEBUG:boto:Headers: {} 2024-02-01T10:27:52.282 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:52.282 DEBUG:boto:Port: 8000 2024-02-01T10:27:52.282 DEBUG:boto:Params: {} 2024-02-01T10:27:52.282 DEBUG:boto:Token: None 2024-02-01T10:27:52.283 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:52 GMT /vclfcn-19/obj?versionId=eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr. 2024-02-01T10:27:52.283 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:801QIorI1iDi1R4XH67PCWmv7rY= 2024-02-01T10:27:52.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:801QIorI1iDi1R4XH67PCWmv7rY=', 'Content-Length': '0'} 2024-02-01T10:27:52.285 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr.'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ec553b1dede73539-0065bb7228-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:52.328 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:52.328 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:52.328 DEBUG:boto:path=/vclfcn-19/obj?versionId=eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr. 2024-02-01T10:27:52.328 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr. 2024-02-01T10:27:52.329 DEBUG:boto:Method: HEAD 2024-02-01T10:27:52.329 DEBUG:boto:Path: /vclfcn-19/obj?versionId=eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr. 2024-02-01T10:27:52.329 DEBUG:boto:Data: 2024-02-01T10:27:52.329 DEBUG:boto:Headers: {} 2024-02-01T10:27:52.329 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:52.329 DEBUG:boto:Port: 8001 2024-02-01T10:27:52.329 DEBUG:boto:Params: {} 2024-02-01T10:27:52.330 DEBUG:boto:Token: None 2024-02-01T10:27:52.330 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:52 GMT /vclfcn-19/obj?versionId=eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr. 2024-02-01T10:27:52.330 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8/oycom6pa7nGVONB8dofGdVC44= 2024-02-01T10:27:52.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8/oycom6pa7nGVONB8dofGdVC44=', 'Content-Length': '0'} 2024-02-01T10:27:52.333 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr.'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000071904646a7ac209b-0065bb7228-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:52.333 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:52.333 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:52.333 DEBUG:boto:path=/vclfcn-19/obj?versionId=eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr. 2024-02-01T10:27:52.334 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr. 2024-02-01T10:27:52.334 DEBUG:boto:Method: HEAD 2024-02-01T10:27:52.334 DEBUG:boto:Path: /vclfcn-19/obj?versionId=eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr. 2024-02-01T10:27:52.334 DEBUG:boto:Data: 2024-02-01T10:27:52.334 DEBUG:boto:Headers: {} 2024-02-01T10:27:52.334 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:52.334 DEBUG:boto:Port: 8000 2024-02-01T10:27:52.334 DEBUG:boto:Params: {} 2024-02-01T10:27:52.334 DEBUG:boto:Token: None 2024-02-01T10:27:52.335 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:52 GMT /vclfcn-19/obj?versionId=eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr. 2024-02-01T10:27:52.335 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8/oycom6pa7nGVONB8dofGdVC44= 2024-02-01T10:27:52.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8/oycom6pa7nGVONB8dofGdVC44=', 'Content-Length': '0'} 2024-02-01T10:27:52.337 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr.'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a2a15b7014717261-0065bb7228-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:52.338 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:52.338 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:52.338 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:52.338 DEBUG:boto:path=/vclfcn-19/obj?versionId=eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr. 2024-02-01T10:27:52.338 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr. 2024-02-01T10:27:52.338 DEBUG:boto:Method: GET 2024-02-01T10:27:52.338 DEBUG:boto:Path: /vclfcn-19/obj?versionId=eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr. 2024-02-01T10:27:52.338 DEBUG:boto:Data: 2024-02-01T10:27:52.338 DEBUG:boto:Headers: {} 2024-02-01T10:27:52.338 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:52.339 DEBUG:boto:Port: 8001 2024-02-01T10:27:52.339 DEBUG:boto:Params: {} 2024-02-01T10:27:52.339 DEBUG:boto:Token: None 2024-02-01T10:27:52.339 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:52 GMT /vclfcn-19/obj?versionId=eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr. 2024-02-01T10:27:52.339 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:801QIorI1iDi1R4XH67PCWmv7rY= 2024-02-01T10:27:52.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:801QIorI1iDi1R4XH67PCWmv7rY=', 'Content-Length': '0'} 2024-02-01T10:27:52.342 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr.'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ed78dc98fdd5d4f2-0065bb7228-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:52.384 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:52.384 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:52.384 DEBUG:boto:path=/vclfcn-19/obj?versionId=eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr. 2024-02-01T10:27:52.385 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr. 2024-02-01T10:27:52.385 DEBUG:boto:Method: GET 2024-02-01T10:27:52.385 DEBUG:boto:Path: /vclfcn-19/obj?versionId=eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr. 2024-02-01T10:27:52.385 DEBUG:boto:Data: 2024-02-01T10:27:52.385 DEBUG:boto:Headers: {} 2024-02-01T10:27:52.385 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:52.385 DEBUG:boto:Port: 8000 2024-02-01T10:27:52.385 DEBUG:boto:Params: {} 2024-02-01T10:27:52.385 DEBUG:boto:Token: None 2024-02-01T10:27:52.386 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:52 GMT /vclfcn-19/obj?versionId=eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr. 2024-02-01T10:27:52.386 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:801QIorI1iDi1R4XH67PCWmv7rY= 2024-02-01T10:27:52.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:801QIorI1iDi1R4XH67PCWmv7rY=', 'Content-Length': '0'} 2024-02-01T10:27:52.389 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'eNy4H6bqVs2Mv61rnMC-zKe0Yl8rhr.'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f0578b1cc581d60c-0065bb7228-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:52.432 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:52.432 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:52.433 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:52.433 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:52.433 DEBUG:boto:Method: GET 2024-02-01T10:27:52.433 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:52.433 DEBUG:boto:Data: 2024-02-01T10:27:52.433 DEBUG:boto:Headers: {} 2024-02-01T10:27:52.433 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:52.433 DEBUG:boto:Port: 8001 2024-02-01T10:27:52.433 DEBUG:boto:Params: {} 2024-02-01T10:27:52.433 DEBUG:boto:Token: None 2024-02-01T10:27:52.434 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:52 GMT /vclfcn-19/ 2024-02-01T10:27:52.434 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:PkpV0vSW9AWl7CsbDW2ArF1aM4s= 2024-02-01T10:27:52.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:PkpV0vSW9AWl7CsbDW2ArF1aM4s=', 'Content-Length': '0'} 2024-02-01T10:27:52.439 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000669a225c8824f5b9-0065bb7228-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:52.480 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:52.481 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:52.481 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:52.481 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:52.481 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:52.481 DEBUG:boto:Method: GET 2024-02-01T10:27:52.481 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:52.481 DEBUG:boto:Data: 2024-02-01T10:27:52.481 DEBUG:boto:Headers: {} 2024-02-01T10:27:52.482 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:52.482 DEBUG:boto:Port: 8000 2024-02-01T10:27:52.482 DEBUG:boto:Params: {} 2024-02-01T10:27:52.482 DEBUG:boto:Token: None 2024-02-01T10:27:52.482 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:52 GMT /vclfcn-19/ 2024-02-01T10:27:52.482 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:PkpV0vSW9AWl7CsbDW2ArF1aM4s= 2024-02-01T10:27:52.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:PkpV0vSW9AWl7CsbDW2ArF1aM4s=', 'Content-Length': '0'} 2024-02-01T10:27:52.488 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dacd8470ea318c96-0065bb7228-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:52.528 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:52.529 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:52.529 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:52.529 DEBUG:boto:Method: HEAD 2024-02-01T10:27:52.529 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:52.529 DEBUG:boto:Data: 2024-02-01T10:27:52.529 DEBUG:boto:Headers: {} 2024-02-01T10:27:52.529 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:52.529 DEBUG:boto:Port: 8001 2024-02-01T10:27:52.529 DEBUG:boto:Params: {} 2024-02-01T10:27:52.530 DEBUG:boto:Token: None 2024-02-01T10:27:52.530 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:52 GMT /vclfcn-19/obj 2024-02-01T10:27:52.530 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RNzIXYrJWDY36C/3XfLkBQwUf28= 2024-02-01T10:27:52.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RNzIXYrJWDY36C/3XfLkBQwUf28=', 'Content-Length': '0'} 2024-02-01T10:27:52.536 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a535c747c687e7d1-0065bb7228-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:52.536 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:52.536 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:52.536 DEBUG:boto:Method: HEAD 2024-02-01T10:27:52.536 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:52.537 DEBUG:boto:Data: 2024-02-01T10:27:52.537 DEBUG:boto:Headers: {} 2024-02-01T10:27:52.537 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:52.537 DEBUG:boto:Port: 8000 2024-02-01T10:27:52.537 DEBUG:boto:Params: {} 2024-02-01T10:27:52.537 DEBUG:boto:Token: None 2024-02-01T10:27:52.537 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:52 GMT /vclfcn-19/obj 2024-02-01T10:27:52.537 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RNzIXYrJWDY36C/3XfLkBQwUf28= 2024-02-01T10:27:52.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RNzIXYrJWDY36C/3XfLkBQwUf28=', 'Content-Length': '0'} 2024-02-01T10:27:52.543 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009fe2b0c4b7fe577d-0065bb7228-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:52.544 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:52.544 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:52.544 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:52.544 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:52.544 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:52.544 DEBUG:boto:Method: GET 2024-02-01T10:27:52.544 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:52.544 DEBUG:boto:Data: 2024-02-01T10:27:52.544 DEBUG:boto:Headers: {} 2024-02-01T10:27:52.545 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:52.545 DEBUG:boto:Port: 8001 2024-02-01T10:27:52.545 DEBUG:boto:Params: {} 2024-02-01T10:27:52.545 DEBUG:boto:Token: None 2024-02-01T10:27:52.545 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:52 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:52.545 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:pkoNRLw4Dl5AKxWsTpH9KwNqd8M= 2024-02-01T10:27:52.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:pkoNRLw4Dl5AKxWsTpH9KwNqd8M=', 'Content-Length': '0'} 2024-02-01T10:27:52.548 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d76778e936cbdd6f-0065bb7228-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:52.592 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:52.592 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:52.593 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:52.593 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:52.593 DEBUG:boto:Method: GET 2024-02-01T10:27:52.593 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:52.593 DEBUG:boto:Data: 2024-02-01T10:27:52.593 DEBUG:boto:Headers: {} 2024-02-01T10:27:52.593 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:52.593 DEBUG:boto:Port: 8000 2024-02-01T10:27:52.593 DEBUG:boto:Params: {} 2024-02-01T10:27:52.594 DEBUG:boto:Token: None 2024-02-01T10:27:52.594 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:52 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:52.594 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:pkoNRLw4Dl5AKxWsTpH9KwNqd8M= 2024-02-01T10:27:52.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:pkoNRLw4Dl5AKxWsTpH9KwNqd8M=', 'Content-Length': '0'} 2024-02-01T10:27:52.597 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007489ce1d34506ccf-0065bb7228-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:52.598 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:52.598 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:52.598 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:52.598 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:52.598 DEBUG:boto:Method: GET 2024-02-01T10:27:52.598 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:52.598 DEBUG:boto:Data: 2024-02-01T10:27:52.598 DEBUG:boto:Headers: {} 2024-02-01T10:27:52.599 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:52.599 DEBUG:boto:Port: 8001 2024-02-01T10:27:52.599 DEBUG:boto:Params: {} 2024-02-01T10:27:52.599 DEBUG:boto:Token: None 2024-02-01T10:27:52.599 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:52 GMT /vclfcn-19/ 2024-02-01T10:27:52.599 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:PkpV0vSW9AWl7CsbDW2ArF1aM4s= 2024-02-01T10:27:52.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:PkpV0vSW9AWl7CsbDW2ArF1aM4s=', 'Content-Length': '0'} 2024-02-01T10:27:52.605 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006325efe092b8e656-0065bb7228-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:52.605 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:52.606 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:52.606 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:52.606 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:52.606 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:52.607 DEBUG:boto:Method: GET 2024-02-01T10:27:52.607 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:52.607 DEBUG:boto:Data: 2024-02-01T10:27:52.607 DEBUG:boto:Headers: {} 2024-02-01T10:27:52.607 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:52.607 DEBUG:boto:Port: 8000 2024-02-01T10:27:52.607 DEBUG:boto:Params: {} 2024-02-01T10:27:52.607 DEBUG:boto:Token: None 2024-02-01T10:27:52.607 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:52 GMT /vclfcn-19/ 2024-02-01T10:27:52.608 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:PkpV0vSW9AWl7CsbDW2ArF1aM4s= 2024-02-01T10:27:52.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:PkpV0vSW9AWl7CsbDW2ArF1aM4s=', 'Content-Length': '0'} 2024-02-01T10:27:52.613 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000073b0ed757aa98360-0065bb7228-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:52.613 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:52.613 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:52.613 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:52.614 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:52.614 DEBUG:boto:path=/vclfcn-19/obj?versionId=tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN 2024-02-01T10:27:52.614 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN 2024-02-01T10:27:52.614 DEBUG:boto:Method: GET 2024-02-01T10:27:52.614 DEBUG:boto:Path: /vclfcn-19/obj?versionId=tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN 2024-02-01T10:27:52.614 DEBUG:boto:Data: 2024-02-01T10:27:52.614 DEBUG:boto:Headers: {} 2024-02-01T10:27:52.614 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:52.614 DEBUG:boto:Port: 8001 2024-02-01T10:27:52.615 DEBUG:boto:Params: {} 2024-02-01T10:27:52.615 DEBUG:boto:Token: None 2024-02-01T10:27:52.615 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:52 GMT /vclfcn-19/obj?versionId=tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN 2024-02-01T10:27:52.615 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:geCUScbSeNEPRIF355ag/hpV/ak= 2024-02-01T10:27:52.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:geCUScbSeNEPRIF355ag/hpV/ak=', 'Content-Length': '0'} 2024-02-01T10:27:52.618 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009edd8f444c5de5cc-0065bb7228-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:52.660 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:52.660 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:52.660 DEBUG:boto:path=/vclfcn-19/obj?versionId=tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN 2024-02-01T10:27:52.660 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN 2024-02-01T10:27:52.661 DEBUG:boto:Method: GET 2024-02-01T10:27:52.661 DEBUG:boto:Path: /vclfcn-19/obj?versionId=tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN 2024-02-01T10:27:52.661 DEBUG:boto:Data: 2024-02-01T10:27:52.661 DEBUG:boto:Headers: {} 2024-02-01T10:27:52.661 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:52.661 DEBUG:boto:Port: 8000 2024-02-01T10:27:52.661 DEBUG:boto:Params: {} 2024-02-01T10:27:52.662 DEBUG:boto:Token: None 2024-02-01T10:27:52.662 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:52 GMT /vclfcn-19/obj?versionId=tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN 2024-02-01T10:27:52.662 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:geCUScbSeNEPRIF355ag/hpV/ak= 2024-02-01T10:27:52.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:geCUScbSeNEPRIF355ag/hpV/ak=', 'Content-Length': '0'} 2024-02-01T10:27:52.665 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000027dbe2bf8a983609-0065bb7228-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:52.665 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:52.665 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:52.665 DEBUG:boto:path=/vclfcn-19/obj?versionId=tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN 2024-02-01T10:27:52.666 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN 2024-02-01T10:27:52.666 DEBUG:boto:Method: HEAD 2024-02-01T10:27:52.666 DEBUG:boto:Path: /vclfcn-19/obj?versionId=tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN 2024-02-01T10:27:52.666 DEBUG:boto:Data: 2024-02-01T10:27:52.666 DEBUG:boto:Headers: {} 2024-02-01T10:27:52.666 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:52.666 DEBUG:boto:Port: 8001 2024-02-01T10:27:52.666 DEBUG:boto:Params: {} 2024-02-01T10:27:52.666 DEBUG:boto:Token: None 2024-02-01T10:27:52.666 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:52 GMT /vclfcn-19/obj?versionId=tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN 2024-02-01T10:27:52.666 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:UfvoeDWQ5WuQxszecNPx9XvXpkw= 2024-02-01T10:27:52.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:UfvoeDWQ5WuQxszecNPx9XvXpkw=', 'Content-Length': '0'} 2024-02-01T10:27:52.669 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000036abcc54b7394af2-0065bb7228-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:52.670 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:52.670 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:52.670 DEBUG:boto:path=/vclfcn-19/obj?versionId=tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN 2024-02-01T10:27:52.670 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN 2024-02-01T10:27:52.670 DEBUG:boto:Method: HEAD 2024-02-01T10:27:52.670 DEBUG:boto:Path: /vclfcn-19/obj?versionId=tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN 2024-02-01T10:27:52.670 DEBUG:boto:Data: 2024-02-01T10:27:52.670 DEBUG:boto:Headers: {} 2024-02-01T10:27:52.670 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:52.670 DEBUG:boto:Port: 8000 2024-02-01T10:27:52.671 DEBUG:boto:Params: {} 2024-02-01T10:27:52.671 DEBUG:boto:Token: None 2024-02-01T10:27:52.671 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:52 GMT /vclfcn-19/obj?versionId=tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN 2024-02-01T10:27:52.671 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:UfvoeDWQ5WuQxszecNPx9XvXpkw= 2024-02-01T10:27:52.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:UfvoeDWQ5WuQxszecNPx9XvXpkw=', 'Content-Length': '0'} 2024-02-01T10:27:52.674 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000abe4a76731c4d647-0065bb7228-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:52.674 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:52.674 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:52.674 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:52.674 DEBUG:boto:path=/vclfcn-19/obj?versionId=tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN 2024-02-01T10:27:52.674 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN 2024-02-01T10:27:52.675 DEBUG:boto:Method: GET 2024-02-01T10:27:52.675 DEBUG:boto:Path: /vclfcn-19/obj?versionId=tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN 2024-02-01T10:27:52.675 DEBUG:boto:Data: 2024-02-01T10:27:52.675 DEBUG:boto:Headers: {} 2024-02-01T10:27:52.675 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:52.675 DEBUG:boto:Port: 8001 2024-02-01T10:27:52.675 DEBUG:boto:Params: {} 2024-02-01T10:27:52.675 DEBUG:boto:Token: None 2024-02-01T10:27:52.675 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:52 GMT /vclfcn-19/obj?versionId=tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN 2024-02-01T10:27:52.675 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:geCUScbSeNEPRIF355ag/hpV/ak= 2024-02-01T10:27:52.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:geCUScbSeNEPRIF355ag/hpV/ak=', 'Content-Length': '0'} 2024-02-01T10:27:52.678 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002dc0f7e5f8a30cb6-0065bb7228-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:52.679 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:52.679 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:52.679 DEBUG:boto:path=/vclfcn-19/obj?versionId=tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN 2024-02-01T10:27:52.679 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN 2024-02-01T10:27:52.679 DEBUG:boto:Method: GET 2024-02-01T10:27:52.679 DEBUG:boto:Path: /vclfcn-19/obj?versionId=tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN 2024-02-01T10:27:52.679 DEBUG:boto:Data: 2024-02-01T10:27:52.679 DEBUG:boto:Headers: {} 2024-02-01T10:27:52.679 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:52.679 DEBUG:boto:Port: 8000 2024-02-01T10:27:52.679 DEBUG:boto:Params: {} 2024-02-01T10:27:52.680 DEBUG:boto:Token: None 2024-02-01T10:27:52.680 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:52 GMT /vclfcn-19/obj?versionId=tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN 2024-02-01T10:27:52.683 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:geCUScbSeNEPRIF355ag/hpV/ak= 2024-02-01T10:27:52.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:geCUScbSeNEPRIF355ag/hpV/ak=', 'Content-Length': '0'} 2024-02-01T10:27:52.687 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'tGIsEYvB.VyZPdq9CgtD6kEeP4rzbDN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004431c73e2f599e77-0065bb7228-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:52.728 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:52.728 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:52.729 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:52.729 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:52.729 DEBUG:boto:Method: GET 2024-02-01T10:27:52.729 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:52.729 DEBUG:boto:Data: 2024-02-01T10:27:52.729 DEBUG:boto:Headers: {} 2024-02-01T10:27:52.729 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:52.729 DEBUG:boto:Port: 8001 2024-02-01T10:27:52.729 DEBUG:boto:Params: {} 2024-02-01T10:27:52.729 DEBUG:boto:Token: None 2024-02-01T10:27:52.730 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:52 GMT /vclfcn-19/ 2024-02-01T10:27:52.730 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:PkpV0vSW9AWl7CsbDW2ArF1aM4s= 2024-02-01T10:27:52.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:PkpV0vSW9AWl7CsbDW2ArF1aM4s=', 'Content-Length': '0'} 2024-02-01T10:27:52.735 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006fdc6d4021a10f7b-0065bb7228-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:52.736 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:52.736 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:52.737 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:52.737 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:52.737 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:52.737 DEBUG:boto:Method: GET 2024-02-01T10:27:52.737 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:52.737 DEBUG:boto:Data: 2024-02-01T10:27:52.737 DEBUG:boto:Headers: {} 2024-02-01T10:27:52.737 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:52.737 DEBUG:boto:Port: 8000 2024-02-01T10:27:52.737 DEBUG:boto:Params: {} 2024-02-01T10:27:52.737 DEBUG:boto:Token: None 2024-02-01T10:27:52.738 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:52 GMT /vclfcn-19/ 2024-02-01T10:27:52.738 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:PkpV0vSW9AWl7CsbDW2ArF1aM4s= 2024-02-01T10:27:52.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:PkpV0vSW9AWl7CsbDW2ArF1aM4s=', 'Content-Length': '0'} 2024-02-01T10:27:52.743 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4507cb1ef167b4f-0065bb7228-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:52.743 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:52.744 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:52.744 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:52.744 DEBUG:boto:Method: HEAD 2024-02-01T10:27:52.744 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:52.744 DEBUG:boto:Data: 2024-02-01T10:27:52.744 DEBUG:boto:Headers: {} 2024-02-01T10:27:52.744 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:52.745 DEBUG:boto:Port: 8001 2024-02-01T10:27:52.745 DEBUG:boto:Params: {} 2024-02-01T10:27:52.745 DEBUG:boto:Token: None 2024-02-01T10:27:52.745 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:52 GMT /vclfcn-19/obj 2024-02-01T10:27:52.745 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RNzIXYrJWDY36C/3XfLkBQwUf28= 2024-02-01T10:27:52.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RNzIXYrJWDY36C/3XfLkBQwUf28=', 'Content-Length': '0'} 2024-02-01T10:27:52.751 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000000e2a43c3b903b7-0065bb7228-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:52.751 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:52.751 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:52.752 DEBUG:boto:Method: HEAD 2024-02-01T10:27:52.752 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:52.752 DEBUG:boto:Data: 2024-02-01T10:27:52.752 DEBUG:boto:Headers: {} 2024-02-01T10:27:52.752 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:52.752 DEBUG:boto:Port: 8000 2024-02-01T10:27:52.752 DEBUG:boto:Params: {} 2024-02-01T10:27:52.752 DEBUG:boto:Token: None 2024-02-01T10:27:52.752 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:52 GMT /vclfcn-19/obj 2024-02-01T10:27:52.753 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RNzIXYrJWDY36C/3XfLkBQwUf28= 2024-02-01T10:27:52.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RNzIXYrJWDY36C/3XfLkBQwUf28=', 'Content-Length': '0'} 2024-02-01T10:27:52.758 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001e562d725e2b1b5d-0065bb7228-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:52.758 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:52.759 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:52.759 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:52.759 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:52.759 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:52.759 DEBUG:boto:Method: GET 2024-02-01T10:27:52.759 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:52.759 DEBUG:boto:Data: 2024-02-01T10:27:52.759 DEBUG:boto:Headers: {} 2024-02-01T10:27:52.759 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:52.759 DEBUG:boto:Port: 8001 2024-02-01T10:27:52.759 DEBUG:boto:Params: {} 2024-02-01T10:27:52.760 DEBUG:boto:Token: None 2024-02-01T10:27:52.760 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:52 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:52.760 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:pkoNRLw4Dl5AKxWsTpH9KwNqd8M= 2024-02-01T10:27:52.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:pkoNRLw4Dl5AKxWsTpH9KwNqd8M=', 'Content-Length': '0'} 2024-02-01T10:27:52.762 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fcd0fb5b050c0ec3-0065bb7228-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:52.804 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:52.804 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:52.804 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:52.804 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:52.805 DEBUG:boto:Method: GET 2024-02-01T10:27:52.805 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:52.805 DEBUG:boto:Data: 2024-02-01T10:27:52.805 DEBUG:boto:Headers: {} 2024-02-01T10:27:52.805 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:52.805 DEBUG:boto:Port: 8000 2024-02-01T10:27:52.805 DEBUG:boto:Params: {} 2024-02-01T10:27:52.805 DEBUG:boto:Token: None 2024-02-01T10:27:52.805 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:52 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:52.806 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:pkoNRLw4Dl5AKxWsTpH9KwNqd8M= 2024-02-01T10:27:52.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:pkoNRLw4Dl5AKxWsTpH9KwNqd8M=', 'Content-Length': '0'} 2024-02-01T10:27:52.809 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000805896de6ed1855c-0065bb7228-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:52.809 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:52.810 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:52.810 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:52.810 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:52.810 DEBUG:boto:Method: GET 2024-02-01T10:27:52.810 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:52.810 DEBUG:boto:Data: 2024-02-01T10:27:52.810 DEBUG:boto:Headers: {} 2024-02-01T10:27:52.810 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:52.810 DEBUG:boto:Port: 8001 2024-02-01T10:27:52.810 DEBUG:boto:Params: {} 2024-02-01T10:27:52.811 DEBUG:boto:Token: None 2024-02-01T10:27:52.811 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:52 GMT /vclfcn-19/ 2024-02-01T10:27:52.811 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:PkpV0vSW9AWl7CsbDW2ArF1aM4s= 2024-02-01T10:27:52.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:PkpV0vSW9AWl7CsbDW2ArF1aM4s=', 'Content-Length': '0'} 2024-02-01T10:27:52.816 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052595df088e89415-0065bb7228-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:52.816 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:52.816 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:52.816 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:52.817 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:52.817 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:52.817 DEBUG:boto:Method: GET 2024-02-01T10:27:52.817 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:52.817 DEBUG:boto:Data: 2024-02-01T10:27:52.817 DEBUG:boto:Headers: {} 2024-02-01T10:27:52.817 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:52.817 DEBUG:boto:Port: 8000 2024-02-01T10:27:52.817 DEBUG:boto:Params: {} 2024-02-01T10:27:52.817 DEBUG:boto:Token: None 2024-02-01T10:27:52.818 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:52 GMT /vclfcn-19/ 2024-02-01T10:27:52.818 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:PkpV0vSW9AWl7CsbDW2ArF1aM4s= 2024-02-01T10:27:52.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:PkpV0vSW9AWl7CsbDW2ArF1aM4s=', 'Content-Length': '0'} 2024-02-01T10:27:52.823 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c100bf012c045ed-0065bb7228-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:52.864 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:52.865 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:52.865 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:52.865 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:52.865 DEBUG:boto:path=/vclfcn-19/obj?versionId=2lRbqumcCYnVWCIz20mU455245nXaz8 2024-02-01T10:27:52.865 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=2lRbqumcCYnVWCIz20mU455245nXaz8 2024-02-01T10:27:52.865 DEBUG:boto:Method: GET 2024-02-01T10:27:52.865 DEBUG:boto:Path: /vclfcn-19/obj?versionId=2lRbqumcCYnVWCIz20mU455245nXaz8 2024-02-01T10:27:52.866 DEBUG:boto:Data: 2024-02-01T10:27:52.866 DEBUG:boto:Headers: {} 2024-02-01T10:27:52.866 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:52.866 DEBUG:boto:Port: 8001 2024-02-01T10:27:52.866 DEBUG:boto:Params: {} 2024-02-01T10:27:52.866 DEBUG:boto:Token: None 2024-02-01T10:27:52.866 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:52 GMT /vclfcn-19/obj?versionId=2lRbqumcCYnVWCIz20mU455245nXaz8 2024-02-01T10:27:52.866 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jZHV4unw8a/XW6R32mvyafrVoLQ= 2024-02-01T10:27:52.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jZHV4unw8a/XW6R32mvyafrVoLQ=', 'Content-Length': '0'} 2024-02-01T10:27:52.869 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '2lRbqumcCYnVWCIz20mU455245nXaz8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c9482d221df8d0c7-0065bb7228-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:52.870 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:52.870 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:52.870 DEBUG:boto:path=/vclfcn-19/obj?versionId=2lRbqumcCYnVWCIz20mU455245nXaz8 2024-02-01T10:27:52.870 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=2lRbqumcCYnVWCIz20mU455245nXaz8 2024-02-01T10:27:52.870 DEBUG:boto:Method: GET 2024-02-01T10:27:52.870 DEBUG:boto:Path: /vclfcn-19/obj?versionId=2lRbqumcCYnVWCIz20mU455245nXaz8 2024-02-01T10:27:52.870 DEBUG:boto:Data: 2024-02-01T10:27:52.870 DEBUG:boto:Headers: {} 2024-02-01T10:27:52.870 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:52.870 DEBUG:boto:Port: 8000 2024-02-01T10:27:52.871 DEBUG:boto:Params: {} 2024-02-01T10:27:52.871 DEBUG:boto:Token: None 2024-02-01T10:27:52.871 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:52 GMT /vclfcn-19/obj?versionId=2lRbqumcCYnVWCIz20mU455245nXaz8 2024-02-01T10:27:52.871 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jZHV4unw8a/XW6R32mvyafrVoLQ= 2024-02-01T10:27:52.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jZHV4unw8a/XW6R32mvyafrVoLQ=', 'Content-Length': '0'} 2024-02-01T10:27:52.874 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '2lRbqumcCYnVWCIz20mU455245nXaz8'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d8f91ccdb1c32c98-0065bb7228-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:52.920 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:52.921 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:52.921 DEBUG:boto:path=/vclfcn-19/obj?versionId=2lRbqumcCYnVWCIz20mU455245nXaz8 2024-02-01T10:27:52.921 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=2lRbqumcCYnVWCIz20mU455245nXaz8 2024-02-01T10:27:52.921 DEBUG:boto:Method: HEAD 2024-02-01T10:27:52.921 DEBUG:boto:Path: /vclfcn-19/obj?versionId=2lRbqumcCYnVWCIz20mU455245nXaz8 2024-02-01T10:27:52.921 DEBUG:boto:Data: 2024-02-01T10:27:52.921 DEBUG:boto:Headers: {} 2024-02-01T10:27:52.921 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:52.921 DEBUG:boto:Port: 8001 2024-02-01T10:27:52.921 DEBUG:boto:Params: {} 2024-02-01T10:27:52.922 DEBUG:boto:Token: None 2024-02-01T10:27:52.922 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:52 GMT /vclfcn-19/obj?versionId=2lRbqumcCYnVWCIz20mU455245nXaz8 2024-02-01T10:27:52.922 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:PlLvFVkhUAbWZ3axPgimVmefihY= 2024-02-01T10:27:52.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:PlLvFVkhUAbWZ3axPgimVmefihY=', 'Content-Length': '0'} 2024-02-01T10:27:52.925 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '2lRbqumcCYnVWCIz20mU455245nXaz8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f31d2841320ab14b-0065bb7228-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:52.926 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:52.926 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:52.926 DEBUG:boto:path=/vclfcn-19/obj?versionId=2lRbqumcCYnVWCIz20mU455245nXaz8 2024-02-01T10:27:52.926 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=2lRbqumcCYnVWCIz20mU455245nXaz8 2024-02-01T10:27:52.926 DEBUG:boto:Method: HEAD 2024-02-01T10:27:52.926 DEBUG:boto:Path: /vclfcn-19/obj?versionId=2lRbqumcCYnVWCIz20mU455245nXaz8 2024-02-01T10:27:52.926 DEBUG:boto:Data: 2024-02-01T10:27:52.927 DEBUG:boto:Headers: {} 2024-02-01T10:27:52.927 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:52.927 DEBUG:boto:Port: 8000 2024-02-01T10:27:52.927 DEBUG:boto:Params: {} 2024-02-01T10:27:52.927 DEBUG:boto:Token: None 2024-02-01T10:27:52.927 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:52 GMT /vclfcn-19/obj?versionId=2lRbqumcCYnVWCIz20mU455245nXaz8 2024-02-01T10:27:52.927 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:PlLvFVkhUAbWZ3axPgimVmefihY= 2024-02-01T10:27:52.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:PlLvFVkhUAbWZ3axPgimVmefihY=', 'Content-Length': '0'} 2024-02-01T10:27:52.930 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '2lRbqumcCYnVWCIz20mU455245nXaz8'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005ff52676af8d6c6c-0065bb7228-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:52.931 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:52.931 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:52.931 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:52.931 DEBUG:boto:path=/vclfcn-19/obj?versionId=2lRbqumcCYnVWCIz20mU455245nXaz8 2024-02-01T10:27:52.931 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=2lRbqumcCYnVWCIz20mU455245nXaz8 2024-02-01T10:27:52.931 DEBUG:boto:Method: GET 2024-02-01T10:27:52.931 DEBUG:boto:Path: /vclfcn-19/obj?versionId=2lRbqumcCYnVWCIz20mU455245nXaz8 2024-02-01T10:27:52.931 DEBUG:boto:Data: 2024-02-01T10:27:52.931 DEBUG:boto:Headers: {} 2024-02-01T10:27:52.931 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:52.932 DEBUG:boto:Port: 8001 2024-02-01T10:27:52.932 DEBUG:boto:Params: {} 2024-02-01T10:27:52.932 DEBUG:boto:Token: None 2024-02-01T10:27:52.932 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:52 GMT /vclfcn-19/obj?versionId=2lRbqumcCYnVWCIz20mU455245nXaz8 2024-02-01T10:27:52.932 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jZHV4unw8a/XW6R32mvyafrVoLQ= 2024-02-01T10:27:52.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jZHV4unw8a/XW6R32mvyafrVoLQ=', 'Content-Length': '0'} 2024-02-01T10:27:52.935 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '2lRbqumcCYnVWCIz20mU455245nXaz8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000de1bae4d5e357f37-0065bb7228-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:52.935 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:52.936 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:52.936 DEBUG:boto:path=/vclfcn-19/obj?versionId=2lRbqumcCYnVWCIz20mU455245nXaz8 2024-02-01T10:27:52.936 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=2lRbqumcCYnVWCIz20mU455245nXaz8 2024-02-01T10:27:52.936 DEBUG:boto:Method: GET 2024-02-01T10:27:52.936 DEBUG:boto:Path: /vclfcn-19/obj?versionId=2lRbqumcCYnVWCIz20mU455245nXaz8 2024-02-01T10:27:52.936 DEBUG:boto:Data: 2024-02-01T10:27:52.936 DEBUG:boto:Headers: {} 2024-02-01T10:27:52.936 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:52.936 DEBUG:boto:Port: 8000 2024-02-01T10:27:52.936 DEBUG:boto:Params: {} 2024-02-01T10:27:52.937 DEBUG:boto:Token: None 2024-02-01T10:27:52.937 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:52 GMT /vclfcn-19/obj?versionId=2lRbqumcCYnVWCIz20mU455245nXaz8 2024-02-01T10:27:52.937 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jZHV4unw8a/XW6R32mvyafrVoLQ= 2024-02-01T10:27:52.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jZHV4unw8a/XW6R32mvyafrVoLQ=', 'Content-Length': '0'} 2024-02-01T10:27:52.939 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '2lRbqumcCYnVWCIz20mU455245nXaz8'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000857d887bd322faaa-0065bb7228-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:52.980 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:52.980 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:52.980 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:52.981 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:52.981 DEBUG:boto:Method: GET 2024-02-01T10:27:52.981 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:52.981 DEBUG:boto:Data: 2024-02-01T10:27:52.981 DEBUG:boto:Headers: {} 2024-02-01T10:27:52.981 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:52.981 DEBUG:boto:Port: 8001 2024-02-01T10:27:52.981 DEBUG:boto:Params: {} 2024-02-01T10:27:52.981 DEBUG:boto:Token: None 2024-02-01T10:27:52.982 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:52 GMT /vclfcn-19/ 2024-02-01T10:27:52.982 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:PkpV0vSW9AWl7CsbDW2ArF1aM4s= 2024-02-01T10:27:52.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:PkpV0vSW9AWl7CsbDW2ArF1aM4s=', 'Content-Length': '0'} 2024-02-01T10:27:52.987 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5c2a17d42e90078-0065bb7228-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:52.987 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:52.988 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:52.988 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:52.988 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:52.988 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:52.988 DEBUG:boto:Method: GET 2024-02-01T10:27:52.988 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:52.988 DEBUG:boto:Data: 2024-02-01T10:27:52.989 DEBUG:boto:Headers: {} 2024-02-01T10:27:52.989 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:52.989 DEBUG:boto:Port: 8000 2024-02-01T10:27:52.989 DEBUG:boto:Params: {} 2024-02-01T10:27:52.989 DEBUG:boto:Token: None 2024-02-01T10:27:52.989 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:52 GMT /vclfcn-19/ 2024-02-01T10:27:52.989 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:PkpV0vSW9AWl7CsbDW2ArF1aM4s= 2024-02-01T10:27:52.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:PkpV0vSW9AWl7CsbDW2ArF1aM4s=', 'Content-Length': '0'} 2024-02-01T10:27:52.994 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e2cfd1abe876200-0065bb7228-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:53.036 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:53.037 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:53.037 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:53.037 DEBUG:boto:Method: HEAD 2024-02-01T10:27:53.037 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:53.037 DEBUG:boto:Data: 2024-02-01T10:27:53.038 DEBUG:boto:Headers: {} 2024-02-01T10:27:53.038 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:53.038 DEBUG:boto:Port: 8001 2024-02-01T10:27:53.038 DEBUG:boto:Params: {} 2024-02-01T10:27:53.038 DEBUG:boto:Token: None 2024-02-01T10:27:53.038 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:53 GMT /vclfcn-19/obj 2024-02-01T10:27:53.038 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:MjYR912/DHF5JxTn/IpWoqU1xQc= 2024-02-01T10:27:53.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:MjYR912/DHF5JxTn/IpWoqU1xQc=', 'Content-Length': '0'} 2024-02-01T10:27:53.044 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ae4b83cb0647b9a6-0065bb7229-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:53.045 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:53.045 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:53.045 DEBUG:boto:Method: HEAD 2024-02-01T10:27:53.045 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:53.045 DEBUG:boto:Data: 2024-02-01T10:27:53.046 DEBUG:boto:Headers: {} 2024-02-01T10:27:53.046 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:53.046 DEBUG:boto:Port: 8000 2024-02-01T10:27:53.046 DEBUG:boto:Params: {} 2024-02-01T10:27:53.046 DEBUG:boto:Token: None 2024-02-01T10:27:53.046 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:53 GMT /vclfcn-19/obj 2024-02-01T10:27:53.047 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:MjYR912/DHF5JxTn/IpWoqU1xQc= 2024-02-01T10:27:53.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:MjYR912/DHF5JxTn/IpWoqU1xQc=', 'Content-Length': '0'} 2024-02-01T10:27:53.053 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008b290ec53c6f6e3a-0065bb7229-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:53.053 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:53.053 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:53.054 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:53.054 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:53.054 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:53.054 DEBUG:boto:Method: GET 2024-02-01T10:27:53.054 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:53.054 DEBUG:boto:Data: 2024-02-01T10:27:53.054 DEBUG:boto:Headers: {} 2024-02-01T10:27:53.054 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:53.055 DEBUG:boto:Port: 8001 2024-02-01T10:27:53.055 DEBUG:boto:Params: {} 2024-02-01T10:27:53.055 DEBUG:boto:Token: None 2024-02-01T10:27:53.055 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:53 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:53.055 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ni7wEZeBsvLfWVhnroWf7E1p1S8= 2024-02-01T10:27:53.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ni7wEZeBsvLfWVhnroWf7E1p1S8=', 'Content-Length': '0'} 2024-02-01T10:27:53.058 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000046bab185fad031e-0065bb7229-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:53.100 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:53.100 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:53.100 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:53.101 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:53.101 DEBUG:boto:Method: GET 2024-02-01T10:27:53.101 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:53.101 DEBUG:boto:Data: 2024-02-01T10:27:53.101 DEBUG:boto:Headers: {} 2024-02-01T10:27:53.101 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:53.101 DEBUG:boto:Port: 8000 2024-02-01T10:27:53.101 DEBUG:boto:Params: {} 2024-02-01T10:27:53.101 DEBUG:boto:Token: None 2024-02-01T10:27:53.102 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:53 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:53.102 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ni7wEZeBsvLfWVhnroWf7E1p1S8= 2024-02-01T10:27:53.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ni7wEZeBsvLfWVhnroWf7E1p1S8=', 'Content-Length': '0'} 2024-02-01T10:27:53.105 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009dd807a2ac5d0b91-0065bb7229-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:53.105 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:53.105 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:53.105 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:53.105 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:53.105 DEBUG:boto:Method: GET 2024-02-01T10:27:53.106 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:53.106 DEBUG:boto:Data: 2024-02-01T10:27:53.106 DEBUG:boto:Headers: {} 2024-02-01T10:27:53.106 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:53.106 DEBUG:boto:Port: 8001 2024-02-01T10:27:53.106 DEBUG:boto:Params: {} 2024-02-01T10:27:53.106 DEBUG:boto:Token: None 2024-02-01T10:27:53.106 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:53 GMT /vclfcn-19/ 2024-02-01T10:27:53.107 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vrABj8VZn7B6S5a5z/u+OvJhviw= 2024-02-01T10:27:53.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vrABj8VZn7B6S5a5z/u+OvJhviw=', 'Content-Length': '0'} 2024-02-01T10:27:53.112 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d1c94c08a63edd7-0065bb7229-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:53.112 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:53.112 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:53.113 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:53.113 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:53.113 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:53.113 DEBUG:boto:Method: GET 2024-02-01T10:27:53.113 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:53.113 DEBUG:boto:Data: 2024-02-01T10:27:53.113 DEBUG:boto:Headers: {} 2024-02-01T10:27:53.113 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:53.113 DEBUG:boto:Port: 8000 2024-02-01T10:27:53.114 DEBUG:boto:Params: {} 2024-02-01T10:27:53.114 DEBUG:boto:Token: None 2024-02-01T10:27:53.114 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:53 GMT /vclfcn-19/ 2024-02-01T10:27:53.114 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vrABj8VZn7B6S5a5z/u+OvJhviw= 2024-02-01T10:27:53.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vrABj8VZn7B6S5a5z/u+OvJhviw=', 'Content-Length': '0'} 2024-02-01T10:27:53.119 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4bdd33c61f9cb92-0065bb7229-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:53.119 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:53.120 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:53.120 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:53.120 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:53.120 DEBUG:boto:path=/vclfcn-19/obj?versionId=oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r 2024-02-01T10:27:53.120 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r 2024-02-01T10:27:53.121 DEBUG:boto:Method: GET 2024-02-01T10:27:53.121 DEBUG:boto:Path: /vclfcn-19/obj?versionId=oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r 2024-02-01T10:27:53.121 DEBUG:boto:Data: 2024-02-01T10:27:53.121 DEBUG:boto:Headers: {} 2024-02-01T10:27:53.121 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:53.121 DEBUG:boto:Port: 8001 2024-02-01T10:27:53.121 DEBUG:boto:Params: {} 2024-02-01T10:27:53.121 DEBUG:boto:Token: None 2024-02-01T10:27:53.122 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:53 GMT /vclfcn-19/obj?versionId=oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r 2024-02-01T10:27:53.122 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:khMxJIZOm2Occ5qKRB3YajCyHOE= 2024-02-01T10:27:53.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:khMxJIZOm2Occ5qKRB3YajCyHOE=', 'Content-Length': '0'} 2024-02-01T10:27:53.125 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005f9ee605f3a4f869-0065bb7229-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:53.125 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:53.125 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:53.126 DEBUG:boto:path=/vclfcn-19/obj?versionId=oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r 2024-02-01T10:27:53.126 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r 2024-02-01T10:27:53.126 DEBUG:boto:Method: GET 2024-02-01T10:27:53.126 DEBUG:boto:Path: /vclfcn-19/obj?versionId=oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r 2024-02-01T10:27:53.126 DEBUG:boto:Data: 2024-02-01T10:27:53.126 DEBUG:boto:Headers: {} 2024-02-01T10:27:53.126 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:53.126 DEBUG:boto:Port: 8000 2024-02-01T10:27:53.126 DEBUG:boto:Params: {} 2024-02-01T10:27:53.126 DEBUG:boto:Token: None 2024-02-01T10:27:53.127 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:53 GMT /vclfcn-19/obj?versionId=oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r 2024-02-01T10:27:53.127 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:khMxJIZOm2Occ5qKRB3YajCyHOE= 2024-02-01T10:27:53.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:khMxJIZOm2Occ5qKRB3YajCyHOE=', 'Content-Length': '0'} 2024-02-01T10:27:53.129 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000026c3cb9965c911ee-0065bb7229-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:53.172 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:53.172 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:53.172 DEBUG:boto:path=/vclfcn-19/obj?versionId=oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r 2024-02-01T10:27:53.172 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r 2024-02-01T10:27:53.173 DEBUG:boto:Method: HEAD 2024-02-01T10:27:53.173 DEBUG:boto:Path: /vclfcn-19/obj?versionId=oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r 2024-02-01T10:27:53.173 DEBUG:boto:Data: 2024-02-01T10:27:53.173 DEBUG:boto:Headers: {} 2024-02-01T10:27:53.173 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:53.173 DEBUG:boto:Port: 8001 2024-02-01T10:27:53.173 DEBUG:boto:Params: {} 2024-02-01T10:27:53.174 DEBUG:boto:Token: None 2024-02-01T10:27:53.174 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:53 GMT /vclfcn-19/obj?versionId=oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r 2024-02-01T10:27:53.174 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RL3lLBnjdPqmCufiHuxZ7I6Ig8k= 2024-02-01T10:27:53.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RL3lLBnjdPqmCufiHuxZ7I6Ig8k=', 'Content-Length': '0'} 2024-02-01T10:27:53.177 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000819f20f465deb307-0065bb7229-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:53.177 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:53.177 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:53.178 DEBUG:boto:path=/vclfcn-19/obj?versionId=oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r 2024-02-01T10:27:53.178 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r 2024-02-01T10:27:53.178 DEBUG:boto:Method: HEAD 2024-02-01T10:27:53.178 DEBUG:boto:Path: /vclfcn-19/obj?versionId=oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r 2024-02-01T10:27:53.178 DEBUG:boto:Data: 2024-02-01T10:27:53.178 DEBUG:boto:Headers: {} 2024-02-01T10:27:53.178 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:53.178 DEBUG:boto:Port: 8000 2024-02-01T10:27:53.178 DEBUG:boto:Params: {} 2024-02-01T10:27:53.178 DEBUG:boto:Token: None 2024-02-01T10:27:53.178 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:53 GMT /vclfcn-19/obj?versionId=oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r 2024-02-01T10:27:53.179 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RL3lLBnjdPqmCufiHuxZ7I6Ig8k= 2024-02-01T10:27:53.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RL3lLBnjdPqmCufiHuxZ7I6Ig8k=', 'Content-Length': '0'} 2024-02-01T10:27:53.181 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ef5b62c4f3f92399-0065bb7229-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:53.182 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:53.182 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:53.182 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:53.182 DEBUG:boto:path=/vclfcn-19/obj?versionId=oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r 2024-02-01T10:27:53.182 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r 2024-02-01T10:27:53.182 DEBUG:boto:Method: GET 2024-02-01T10:27:53.182 DEBUG:boto:Path: /vclfcn-19/obj?versionId=oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r 2024-02-01T10:27:53.183 DEBUG:boto:Data: 2024-02-01T10:27:53.183 DEBUG:boto:Headers: {} 2024-02-01T10:27:53.183 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:53.183 DEBUG:boto:Port: 8001 2024-02-01T10:27:53.183 DEBUG:boto:Params: {} 2024-02-01T10:27:53.183 DEBUG:boto:Token: None 2024-02-01T10:27:53.183 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:53 GMT /vclfcn-19/obj?versionId=oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r 2024-02-01T10:27:53.183 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:khMxJIZOm2Occ5qKRB3YajCyHOE= 2024-02-01T10:27:53.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:khMxJIZOm2Occ5qKRB3YajCyHOE=', 'Content-Length': '0'} 2024-02-01T10:27:53.186 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007ca3a30a43bd48a0-0065bb7229-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:53.228 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:53.228 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:53.228 DEBUG:boto:path=/vclfcn-19/obj?versionId=oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r 2024-02-01T10:27:53.229 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r 2024-02-01T10:27:53.229 DEBUG:boto:Method: GET 2024-02-01T10:27:53.229 DEBUG:boto:Path: /vclfcn-19/obj?versionId=oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r 2024-02-01T10:27:53.229 DEBUG:boto:Data: 2024-02-01T10:27:53.229 DEBUG:boto:Headers: {} 2024-02-01T10:27:53.229 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:53.229 DEBUG:boto:Port: 8000 2024-02-01T10:27:53.229 DEBUG:boto:Params: {} 2024-02-01T10:27:53.229 DEBUG:boto:Token: None 2024-02-01T10:27:53.230 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:53 GMT /vclfcn-19/obj?versionId=oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r 2024-02-01T10:27:53.230 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:khMxJIZOm2Occ5qKRB3YajCyHOE= 2024-02-01T10:27:53.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:khMxJIZOm2Occ5qKRB3YajCyHOE=', 'Content-Length': '0'} 2024-02-01T10:27:53.233 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'oGhdnHMtrLPTgFgXlKfl.UXdmrUnL2r'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002d7b36bbb2a076a5-0065bb7229-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:53.233 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:53.233 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:53.233 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:53.233 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:53.233 DEBUG:boto:Method: GET 2024-02-01T10:27:53.233 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:53.234 DEBUG:boto:Data: 2024-02-01T10:27:53.234 DEBUG:boto:Headers: {} 2024-02-01T10:27:53.234 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:53.234 DEBUG:boto:Port: 8001 2024-02-01T10:27:53.234 DEBUG:boto:Params: {} 2024-02-01T10:27:53.234 DEBUG:boto:Token: None 2024-02-01T10:27:53.234 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:53 GMT /vclfcn-19/ 2024-02-01T10:27:53.234 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vrABj8VZn7B6S5a5z/u+OvJhviw= 2024-02-01T10:27:53.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vrABj8VZn7B6S5a5z/u+OvJhviw=', 'Content-Length': '0'} 2024-02-01T10:27:53.239 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f79f81bfb38491a2-0065bb7229-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:53.239 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:53.240 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:53.240 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:53.240 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:53.240 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:53.240 DEBUG:boto:Method: GET 2024-02-01T10:27:53.240 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:53.241 DEBUG:boto:Data: 2024-02-01T10:27:53.241 DEBUG:boto:Headers: {} 2024-02-01T10:27:53.241 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:53.241 DEBUG:boto:Port: 8000 2024-02-01T10:27:53.241 DEBUG:boto:Params: {} 2024-02-01T10:27:53.241 DEBUG:boto:Token: None 2024-02-01T10:27:53.241 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:53 GMT /vclfcn-19/ 2024-02-01T10:27:53.241 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vrABj8VZn7B6S5a5z/u+OvJhviw= 2024-02-01T10:27:53.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vrABj8VZn7B6S5a5z/u+OvJhviw=', 'Content-Length': '0'} 2024-02-01T10:27:53.246 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c8603c172a9cb66-0065bb7229-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:53.247 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:53.247 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:53.247 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:53.247 DEBUG:boto:Method: HEAD 2024-02-01T10:27:53.248 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:53.248 DEBUG:boto:Data: 2024-02-01T10:27:53.248 DEBUG:boto:Headers: {} 2024-02-01T10:27:53.248 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:53.248 DEBUG:boto:Port: 8001 2024-02-01T10:27:53.248 DEBUG:boto:Params: {} 2024-02-01T10:27:53.248 DEBUG:boto:Token: None 2024-02-01T10:27:53.248 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:53 GMT /vclfcn-19/obj 2024-02-01T10:27:53.249 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:MjYR912/DHF5JxTn/IpWoqU1xQc= 2024-02-01T10:27:53.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:MjYR912/DHF5JxTn/IpWoqU1xQc=', 'Content-Length': '0'} 2024-02-01T10:27:53.254 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008c662732db98c94c-0065bb7229-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:53.255 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:53.255 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:53.255 DEBUG:boto:Method: HEAD 2024-02-01T10:27:53.255 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:53.255 DEBUG:boto:Data: 2024-02-01T10:27:53.255 DEBUG:boto:Headers: {} 2024-02-01T10:27:53.255 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:53.255 DEBUG:boto:Port: 8000 2024-02-01T10:27:53.255 DEBUG:boto:Params: {} 2024-02-01T10:27:53.255 DEBUG:boto:Token: None 2024-02-01T10:27:53.256 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:53 GMT /vclfcn-19/obj 2024-02-01T10:27:53.256 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:MjYR912/DHF5JxTn/IpWoqU1xQc= 2024-02-01T10:27:53.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:MjYR912/DHF5JxTn/IpWoqU1xQc=', 'Content-Length': '0'} 2024-02-01T10:27:53.261 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ff03d97faf6d36b7-0065bb7229-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:53.262 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:53.262 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:53.262 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:53.262 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:53.262 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:53.262 DEBUG:boto:Method: GET 2024-02-01T10:27:53.262 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:53.262 DEBUG:boto:Data: 2024-02-01T10:27:53.262 DEBUG:boto:Headers: {} 2024-02-01T10:27:53.262 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:53.263 DEBUG:boto:Port: 8001 2024-02-01T10:27:53.263 DEBUG:boto:Params: {} 2024-02-01T10:27:53.263 DEBUG:boto:Token: None 2024-02-01T10:27:53.263 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:53 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:53.263 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ni7wEZeBsvLfWVhnroWf7E1p1S8= 2024-02-01T10:27:53.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ni7wEZeBsvLfWVhnroWf7E1p1S8=', 'Content-Length': '0'} 2024-02-01T10:27:53.266 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000cbb6253917b7558-0065bb7229-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:53.312 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:53.312 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:53.312 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:53.313 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:53.313 DEBUG:boto:Method: GET 2024-02-01T10:27:53.313 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:53.313 DEBUG:boto:Data: 2024-02-01T10:27:53.313 DEBUG:boto:Headers: {} 2024-02-01T10:27:53.313 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:53.313 DEBUG:boto:Port: 8000 2024-02-01T10:27:53.313 DEBUG:boto:Params: {} 2024-02-01T10:27:53.313 DEBUG:boto:Token: None 2024-02-01T10:27:53.314 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:53 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:53.314 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ni7wEZeBsvLfWVhnroWf7E1p1S8= 2024-02-01T10:27:53.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ni7wEZeBsvLfWVhnroWf7E1p1S8=', 'Content-Length': '0'} 2024-02-01T10:27:53.316 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000071b6036a025df785-0065bb7229-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:53.317 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:53.317 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:53.317 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:53.317 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:53.317 DEBUG:boto:Method: GET 2024-02-01T10:27:53.317 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:53.317 DEBUG:boto:Data: 2024-02-01T10:27:53.317 DEBUG:boto:Headers: {} 2024-02-01T10:27:53.317 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:53.318 DEBUG:boto:Port: 8001 2024-02-01T10:27:53.318 DEBUG:boto:Params: {} 2024-02-01T10:27:53.318 DEBUG:boto:Token: None 2024-02-01T10:27:53.318 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:53 GMT /vclfcn-19/ 2024-02-01T10:27:53.318 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vrABj8VZn7B6S5a5z/u+OvJhviw= 2024-02-01T10:27:53.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vrABj8VZn7B6S5a5z/u+OvJhviw=', 'Content-Length': '0'} 2024-02-01T10:27:53.324 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000073f351f007453e8d-0065bb7229-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:53.324 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:53.325 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:53.325 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:53.325 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:53.325 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:53.325 DEBUG:boto:Method: GET 2024-02-01T10:27:53.325 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:53.325 DEBUG:boto:Data: 2024-02-01T10:27:53.325 DEBUG:boto:Headers: {} 2024-02-01T10:27:53.325 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:53.325 DEBUG:boto:Port: 8000 2024-02-01T10:27:53.326 DEBUG:boto:Params: {} 2024-02-01T10:27:53.326 DEBUG:boto:Token: None 2024-02-01T10:27:53.326 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:53 GMT /vclfcn-19/ 2024-02-01T10:27:53.326 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vrABj8VZn7B6S5a5z/u+OvJhviw= 2024-02-01T10:27:53.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vrABj8VZn7B6S5a5z/u+OvJhviw=', 'Content-Length': '0'} 2024-02-01T10:27:53.331 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d674ea8cd1d56039-0065bb7229-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:53.331 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:53.332 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:53.332 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:53.332 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:53.332 DEBUG:boto:path=/vclfcn-19/obj?versionId=qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9 2024-02-01T10:27:53.332 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9 2024-02-01T10:27:53.332 DEBUG:boto:Method: GET 2024-02-01T10:27:53.332 DEBUG:boto:Path: /vclfcn-19/obj?versionId=qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9 2024-02-01T10:27:53.332 DEBUG:boto:Data: 2024-02-01T10:27:53.333 DEBUG:boto:Headers: {} 2024-02-01T10:27:53.333 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:53.333 DEBUG:boto:Port: 8001 2024-02-01T10:27:53.333 DEBUG:boto:Params: {} 2024-02-01T10:27:53.333 DEBUG:boto:Token: None 2024-02-01T10:27:53.333 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:53 GMT /vclfcn-19/obj?versionId=qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9 2024-02-01T10:27:53.333 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jTscny4dPgCWeYTi1oG3GV0wFdA= 2024-02-01T10:27:53.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jTscny4dPgCWeYTi1oG3GV0wFdA=', 'Content-Length': '0'} 2024-02-01T10:27:53.336 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000293b4181bda6e185-0065bb7229-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:53.336 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:53.338 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:53.338 DEBUG:boto:path=/vclfcn-19/obj?versionId=qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9 2024-02-01T10:27:53.338 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9 2024-02-01T10:27:53.338 DEBUG:boto:Method: GET 2024-02-01T10:27:53.338 DEBUG:boto:Path: /vclfcn-19/obj?versionId=qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9 2024-02-01T10:27:53.338 DEBUG:boto:Data: 2024-02-01T10:27:53.338 DEBUG:boto:Headers: {} 2024-02-01T10:27:53.338 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:53.338 DEBUG:boto:Port: 8000 2024-02-01T10:27:53.338 DEBUG:boto:Params: {} 2024-02-01T10:27:53.339 DEBUG:boto:Token: None 2024-02-01T10:27:53.339 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:53 GMT /vclfcn-19/obj?versionId=qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9 2024-02-01T10:27:53.339 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jTscny4dPgCWeYTi1oG3GV0wFdA= 2024-02-01T10:27:53.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jTscny4dPgCWeYTi1oG3GV0wFdA=', 'Content-Length': '0'} 2024-02-01T10:27:53.342 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000e73b3f6e289badf-0065bb7229-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:53.384 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:53.384 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:53.384 DEBUG:boto:path=/vclfcn-19/obj?versionId=qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9 2024-02-01T10:27:53.384 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9 2024-02-01T10:27:53.384 DEBUG:boto:Method: HEAD 2024-02-01T10:27:53.385 DEBUG:boto:Path: /vclfcn-19/obj?versionId=qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9 2024-02-01T10:27:53.385 DEBUG:boto:Data: 2024-02-01T10:27:53.385 DEBUG:boto:Headers: {} 2024-02-01T10:27:53.385 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:53.385 DEBUG:boto:Port: 8001 2024-02-01T10:27:53.385 DEBUG:boto:Params: {} 2024-02-01T10:27:53.385 DEBUG:boto:Token: None 2024-02-01T10:27:53.385 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:53 GMT /vclfcn-19/obj?versionId=qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9 2024-02-01T10:27:53.385 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/kxB5q/KW7apPCdkYLC4FQ3XX18= 2024-02-01T10:27:53.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/kxB5q/KW7apPCdkYLC4FQ3XX18=', 'Content-Length': '0'} 2024-02-01T10:27:53.388 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000019fad672de3f328a-0065bb7229-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:53.388 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:53.389 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:53.389 DEBUG:boto:path=/vclfcn-19/obj?versionId=qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9 2024-02-01T10:27:53.389 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9 2024-02-01T10:27:53.389 DEBUG:boto:Method: HEAD 2024-02-01T10:27:53.389 DEBUG:boto:Path: /vclfcn-19/obj?versionId=qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9 2024-02-01T10:27:53.389 DEBUG:boto:Data: 2024-02-01T10:27:53.389 DEBUG:boto:Headers: {} 2024-02-01T10:27:53.390 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:53.390 DEBUG:boto:Port: 8000 2024-02-01T10:27:53.390 DEBUG:boto:Params: {} 2024-02-01T10:27:53.390 DEBUG:boto:Token: None 2024-02-01T10:27:53.390 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:53 GMT /vclfcn-19/obj?versionId=qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9 2024-02-01T10:27:53.390 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/kxB5q/KW7apPCdkYLC4FQ3XX18= 2024-02-01T10:27:53.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/kxB5q/KW7apPCdkYLC4FQ3XX18=', 'Content-Length': '0'} 2024-02-01T10:27:53.393 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008bc212d4b5be89d2-0065bb7229-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:53.393 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:53.394 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:53.394 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:53.394 DEBUG:boto:path=/vclfcn-19/obj?versionId=qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9 2024-02-01T10:27:53.394 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9 2024-02-01T10:27:53.394 DEBUG:boto:Method: GET 2024-02-01T10:27:53.394 DEBUG:boto:Path: /vclfcn-19/obj?versionId=qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9 2024-02-01T10:27:53.394 DEBUG:boto:Data: 2024-02-01T10:27:53.394 DEBUG:boto:Headers: {} 2024-02-01T10:27:53.394 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:53.394 DEBUG:boto:Port: 8001 2024-02-01T10:27:53.394 DEBUG:boto:Params: {} 2024-02-01T10:27:53.395 DEBUG:boto:Token: None 2024-02-01T10:27:53.395 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:53 GMT /vclfcn-19/obj?versionId=qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9 2024-02-01T10:27:53.395 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jTscny4dPgCWeYTi1oG3GV0wFdA= 2024-02-01T10:27:53.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jTscny4dPgCWeYTi1oG3GV0wFdA=', 'Content-Length': '0'} 2024-02-01T10:27:53.398 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e13eaa6f2d57ca43-0065bb7229-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:53.440 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:53.440 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:53.440 DEBUG:boto:path=/vclfcn-19/obj?versionId=qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9 2024-02-01T10:27:53.441 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9 2024-02-01T10:27:53.441 DEBUG:boto:Method: GET 2024-02-01T10:27:53.441 DEBUG:boto:Path: /vclfcn-19/obj?versionId=qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9 2024-02-01T10:27:53.441 DEBUG:boto:Data: 2024-02-01T10:27:53.441 DEBUG:boto:Headers: {} 2024-02-01T10:27:53.441 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:53.441 DEBUG:boto:Port: 8000 2024-02-01T10:27:53.441 DEBUG:boto:Params: {} 2024-02-01T10:27:53.442 DEBUG:boto:Token: None 2024-02-01T10:27:53.442 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:53 GMT /vclfcn-19/obj?versionId=qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9 2024-02-01T10:27:53.442 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jTscny4dPgCWeYTi1oG3GV0wFdA= 2024-02-01T10:27:53.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jTscny4dPgCWeYTi1oG3GV0wFdA=', 'Content-Length': '0'} 2024-02-01T10:27:53.445 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'qex3Bh72IV4plxG2V2Lm7bvqwXqQNK9'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004620c9e80ccd6397-0065bb7229-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:53.492 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:53.492 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:53.493 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:53.493 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:53.493 DEBUG:boto:Method: GET 2024-02-01T10:27:53.493 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:53.493 DEBUG:boto:Data: 2024-02-01T10:27:53.493 DEBUG:boto:Headers: {} 2024-02-01T10:27:53.493 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:53.493 DEBUG:boto:Port: 8001 2024-02-01T10:27:53.494 DEBUG:boto:Params: {} 2024-02-01T10:27:53.494 DEBUG:boto:Token: None 2024-02-01T10:27:53.494 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:53 GMT /vclfcn-19/ 2024-02-01T10:27:53.494 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vrABj8VZn7B6S5a5z/u+OvJhviw= 2024-02-01T10:27:53.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vrABj8VZn7B6S5a5z/u+OvJhviw=', 'Content-Length': '0'} 2024-02-01T10:27:53.499 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f974db18e80fab2-0065bb7229-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:53.500 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:53.500 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:53.500 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:53.500 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:53.500 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:53.501 DEBUG:boto:Method: GET 2024-02-01T10:27:53.501 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:53.501 DEBUG:boto:Data: 2024-02-01T10:27:53.501 DEBUG:boto:Headers: {} 2024-02-01T10:27:53.501 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:53.501 DEBUG:boto:Port: 8000 2024-02-01T10:27:53.501 DEBUG:boto:Params: {} 2024-02-01T10:27:53.501 DEBUG:boto:Token: None 2024-02-01T10:27:53.501 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:53 GMT /vclfcn-19/ 2024-02-01T10:27:53.501 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vrABj8VZn7B6S5a5z/u+OvJhviw= 2024-02-01T10:27:53.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vrABj8VZn7B6S5a5z/u+OvJhviw=', 'Content-Length': '0'} 2024-02-01T10:27:53.507 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008184bba95c680a3e-0065bb7229-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:53.507 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:53.507 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:53.507 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:53.508 DEBUG:boto:Method: HEAD 2024-02-01T10:27:53.508 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:53.508 DEBUG:boto:Data: 2024-02-01T10:27:53.508 DEBUG:boto:Headers: {} 2024-02-01T10:27:53.508 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:53.508 DEBUG:boto:Port: 8001 2024-02-01T10:27:53.508 DEBUG:boto:Params: {} 2024-02-01T10:27:53.508 DEBUG:boto:Token: None 2024-02-01T10:27:53.508 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:53 GMT /vclfcn-19/obj 2024-02-01T10:27:53.509 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:MjYR912/DHF5JxTn/IpWoqU1xQc= 2024-02-01T10:27:53.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:MjYR912/DHF5JxTn/IpWoqU1xQc=', 'Content-Length': '0'} 2024-02-01T10:27:53.514 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005c48fb3005a39121-0065bb7229-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:53.515 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:53.515 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:53.515 DEBUG:boto:Method: HEAD 2024-02-01T10:27:53.515 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:53.515 DEBUG:boto:Data: 2024-02-01T10:27:53.515 DEBUG:boto:Headers: {} 2024-02-01T10:27:53.515 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:53.515 DEBUG:boto:Port: 8000 2024-02-01T10:27:53.515 DEBUG:boto:Params: {} 2024-02-01T10:27:53.515 DEBUG:boto:Token: None 2024-02-01T10:27:53.516 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:53 GMT /vclfcn-19/obj 2024-02-01T10:27:53.516 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:MjYR912/DHF5JxTn/IpWoqU1xQc= 2024-02-01T10:27:53.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:MjYR912/DHF5JxTn/IpWoqU1xQc=', 'Content-Length': '0'} 2024-02-01T10:27:53.521 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d9bc1e3eb3de6b0a-0065bb7229-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:53.521 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:53.522 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:53.522 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:53.522 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:53.522 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:53.522 DEBUG:boto:Method: GET 2024-02-01T10:27:53.522 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:53.523 DEBUG:boto:Data: 2024-02-01T10:27:53.523 DEBUG:boto:Headers: {} 2024-02-01T10:27:53.523 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:53.523 DEBUG:boto:Port: 8001 2024-02-01T10:27:53.523 DEBUG:boto:Params: {} 2024-02-01T10:27:53.523 DEBUG:boto:Token: None 2024-02-01T10:27:53.523 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:53 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:53.523 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ni7wEZeBsvLfWVhnroWf7E1p1S8= 2024-02-01T10:27:53.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ni7wEZeBsvLfWVhnroWf7E1p1S8=', 'Content-Length': '0'} 2024-02-01T10:27:53.526 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e73622f2898bf0e5-0065bb7229-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:53.572 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:53.572 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:53.572 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:53.572 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:53.573 DEBUG:boto:Method: GET 2024-02-01T10:27:53.573 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:53.573 DEBUG:boto:Data: 2024-02-01T10:27:53.573 DEBUG:boto:Headers: {} 2024-02-01T10:27:53.573 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:53.573 DEBUG:boto:Port: 8000 2024-02-01T10:27:53.573 DEBUG:boto:Params: {} 2024-02-01T10:27:53.573 DEBUG:boto:Token: None 2024-02-01T10:27:53.573 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:53 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:53.574 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ni7wEZeBsvLfWVhnroWf7E1p1S8= 2024-02-01T10:27:53.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ni7wEZeBsvLfWVhnroWf7E1p1S8=', 'Content-Length': '0'} 2024-02-01T10:27:53.577 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000e77a324e711682f-0065bb7229-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:53.577 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:53.577 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:53.577 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:53.577 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:53.578 DEBUG:boto:Method: GET 2024-02-01T10:27:53.578 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:53.578 DEBUG:boto:Data: 2024-02-01T10:27:53.578 DEBUG:boto:Headers: {} 2024-02-01T10:27:53.578 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:53.578 DEBUG:boto:Port: 8001 2024-02-01T10:27:53.578 DEBUG:boto:Params: {} 2024-02-01T10:27:53.578 DEBUG:boto:Token: None 2024-02-01T10:27:53.578 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:53 GMT /vclfcn-19/ 2024-02-01T10:27:53.578 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vrABj8VZn7B6S5a5z/u+OvJhviw= 2024-02-01T10:27:53.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vrABj8VZn7B6S5a5z/u+OvJhviw=', 'Content-Length': '0'} 2024-02-01T10:27:53.584 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b2237b05a65d530-0065bb7229-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:53.628 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:53.629 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:53.629 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:53.629 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:53.629 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:53.629 DEBUG:boto:Method: GET 2024-02-01T10:27:53.629 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:53.629 DEBUG:boto:Data: 2024-02-01T10:27:53.629 DEBUG:boto:Headers: {} 2024-02-01T10:27:53.629 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:53.629 DEBUG:boto:Port: 8000 2024-02-01T10:27:53.629 DEBUG:boto:Params: {} 2024-02-01T10:27:53.630 DEBUG:boto:Token: None 2024-02-01T10:27:53.630 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:53 GMT /vclfcn-19/ 2024-02-01T10:27:53.630 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vrABj8VZn7B6S5a5z/u+OvJhviw= 2024-02-01T10:27:53.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vrABj8VZn7B6S5a5z/u+OvJhviw=', 'Content-Length': '0'} 2024-02-01T10:27:53.636 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030b2f1c725736dc5-0065bb7229-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:53.636 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:53.636 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:53.637 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:53.637 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:53.637 DEBUG:boto:path=/vclfcn-19/obj?versionId=NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS 2024-02-01T10:27:53.637 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS 2024-02-01T10:27:53.637 DEBUG:boto:Method: GET 2024-02-01T10:27:53.637 DEBUG:boto:Path: /vclfcn-19/obj?versionId=NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS 2024-02-01T10:27:53.637 DEBUG:boto:Data: 2024-02-01T10:27:53.637 DEBUG:boto:Headers: {} 2024-02-01T10:27:53.638 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:53.638 DEBUG:boto:Port: 8001 2024-02-01T10:27:53.638 DEBUG:boto:Params: {} 2024-02-01T10:27:53.638 DEBUG:boto:Token: None 2024-02-01T10:27:53.638 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:53 GMT /vclfcn-19/obj?versionId=NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS 2024-02-01T10:27:53.638 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fIS17SBlLjnqzPun4Lz/5ITO6Ks= 2024-02-01T10:27:53.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fIS17SBlLjnqzPun4Lz/5ITO6Ks=', 'Content-Length': '0'} 2024-02-01T10:27:53.642 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006a4a782f55087fda-0065bb7229-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:53.688 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:53.688 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:53.688 DEBUG:boto:path=/vclfcn-19/obj?versionId=NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS 2024-02-01T10:27:53.689 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS 2024-02-01T10:27:53.689 DEBUG:boto:Method: GET 2024-02-01T10:27:53.689 DEBUG:boto:Path: /vclfcn-19/obj?versionId=NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS 2024-02-01T10:27:53.689 DEBUG:boto:Data: 2024-02-01T10:27:53.689 DEBUG:boto:Headers: {} 2024-02-01T10:27:53.689 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:53.689 DEBUG:boto:Port: 8000 2024-02-01T10:27:53.689 DEBUG:boto:Params: {} 2024-02-01T10:27:53.689 DEBUG:boto:Token: None 2024-02-01T10:27:53.690 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:53 GMT /vclfcn-19/obj?versionId=NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS 2024-02-01T10:27:53.690 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fIS17SBlLjnqzPun4Lz/5ITO6Ks= 2024-02-01T10:27:53.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fIS17SBlLjnqzPun4Lz/5ITO6Ks=', 'Content-Length': '0'} 2024-02-01T10:27:53.693 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c5b8159a5a4767cf-0065bb7229-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:53.693 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:53.693 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:53.693 DEBUG:boto:path=/vclfcn-19/obj?versionId=NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS 2024-02-01T10:27:53.694 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS 2024-02-01T10:27:53.694 DEBUG:boto:Method: HEAD 2024-02-01T10:27:53.694 DEBUG:boto:Path: /vclfcn-19/obj?versionId=NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS 2024-02-01T10:27:53.694 DEBUG:boto:Data: 2024-02-01T10:27:53.694 DEBUG:boto:Headers: {} 2024-02-01T10:27:53.694 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:53.694 DEBUG:boto:Port: 8001 2024-02-01T10:27:53.694 DEBUG:boto:Params: {} 2024-02-01T10:27:53.694 DEBUG:boto:Token: None 2024-02-01T10:27:53.694 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:53 GMT /vclfcn-19/obj?versionId=NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS 2024-02-01T10:27:53.695 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:dgY6iQ+JIKaoao7do3WEt7CM6R0= 2024-02-01T10:27:53.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dgY6iQ+JIKaoao7do3WEt7CM6R0=', 'Content-Length': '0'} 2024-02-01T10:27:53.697 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e2b3efb89bb26f81-0065bb7229-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:53.698 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:53.698 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:53.698 DEBUG:boto:path=/vclfcn-19/obj?versionId=NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS 2024-02-01T10:27:53.698 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS 2024-02-01T10:27:53.698 DEBUG:boto:Method: HEAD 2024-02-01T10:27:53.698 DEBUG:boto:Path: /vclfcn-19/obj?versionId=NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS 2024-02-01T10:27:53.698 DEBUG:boto:Data: 2024-02-01T10:27:53.698 DEBUG:boto:Headers: {} 2024-02-01T10:27:53.698 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:53.699 DEBUG:boto:Port: 8000 2024-02-01T10:27:53.699 DEBUG:boto:Params: {} 2024-02-01T10:27:53.699 DEBUG:boto:Token: None 2024-02-01T10:27:53.699 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:53 GMT /vclfcn-19/obj?versionId=NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS 2024-02-01T10:27:53.699 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:dgY6iQ+JIKaoao7do3WEt7CM6R0= 2024-02-01T10:27:53.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dgY6iQ+JIKaoao7do3WEt7CM6R0=', 'Content-Length': '0'} 2024-02-01T10:27:53.702 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000037c07c629d4b1874-0065bb7229-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:53.702 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:53.702 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:53.702 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:53.702 DEBUG:boto:path=/vclfcn-19/obj?versionId=NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS 2024-02-01T10:27:53.702 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS 2024-02-01T10:27:53.702 DEBUG:boto:Method: GET 2024-02-01T10:27:53.703 DEBUG:boto:Path: /vclfcn-19/obj?versionId=NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS 2024-02-01T10:27:53.703 DEBUG:boto:Data: 2024-02-01T10:27:53.703 DEBUG:boto:Headers: {} 2024-02-01T10:27:53.703 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:53.703 DEBUG:boto:Port: 8001 2024-02-01T10:27:53.703 DEBUG:boto:Params: {} 2024-02-01T10:27:53.703 DEBUG:boto:Token: None 2024-02-01T10:27:53.703 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:53 GMT /vclfcn-19/obj?versionId=NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS 2024-02-01T10:27:53.703 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fIS17SBlLjnqzPun4Lz/5ITO6Ks= 2024-02-01T10:27:53.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fIS17SBlLjnqzPun4Lz/5ITO6Ks=', 'Content-Length': '0'} 2024-02-01T10:27:53.707 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000adcb60724f27380e-0065bb7229-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:53.748 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:53.748 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:53.748 DEBUG:boto:path=/vclfcn-19/obj?versionId=NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS 2024-02-01T10:27:53.748 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS 2024-02-01T10:27:53.749 DEBUG:boto:Method: GET 2024-02-01T10:27:53.749 DEBUG:boto:Path: /vclfcn-19/obj?versionId=NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS 2024-02-01T10:27:53.749 DEBUG:boto:Data: 2024-02-01T10:27:53.749 DEBUG:boto:Headers: {} 2024-02-01T10:27:53.749 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:53.749 DEBUG:boto:Port: 8000 2024-02-01T10:27:53.749 DEBUG:boto:Params: {} 2024-02-01T10:27:53.749 DEBUG:boto:Token: None 2024-02-01T10:27:53.749 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:53 GMT /vclfcn-19/obj?versionId=NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS 2024-02-01T10:27:53.750 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fIS17SBlLjnqzPun4Lz/5ITO6Ks= 2024-02-01T10:27:53.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fIS17SBlLjnqzPun4Lz/5ITO6Ks=', 'Content-Length': '0'} 2024-02-01T10:27:53.753 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'NCPEvRYQDJpLrTejJRqmTf2j0FWCBCS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000046cdb26539c08f6a-0065bb7229-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:53.753 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:53.753 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:53.753 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:53.753 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:53.753 DEBUG:boto:Method: GET 2024-02-01T10:27:53.753 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:53.753 DEBUG:boto:Data: 2024-02-01T10:27:53.754 DEBUG:boto:Headers: {} 2024-02-01T10:27:53.754 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:53.754 DEBUG:boto:Port: 8001 2024-02-01T10:27:53.754 DEBUG:boto:Params: {} 2024-02-01T10:27:53.754 DEBUG:boto:Token: None 2024-02-01T10:27:53.754 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:53 GMT /vclfcn-19/ 2024-02-01T10:27:53.754 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vrABj8VZn7B6S5a5z/u+OvJhviw= 2024-02-01T10:27:53.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vrABj8VZn7B6S5a5z/u+OvJhviw=', 'Content-Length': '0'} 2024-02-01T10:27:53.760 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095ddfcf86aa0ae99-0065bb7229-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:53.760 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:53.760 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:53.760 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:53.761 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:53.761 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:53.761 DEBUG:boto:Method: GET 2024-02-01T10:27:53.761 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:53.761 DEBUG:boto:Data: 2024-02-01T10:27:53.761 DEBUG:boto:Headers: {} 2024-02-01T10:27:53.761 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:53.761 DEBUG:boto:Port: 8000 2024-02-01T10:27:53.761 DEBUG:boto:Params: {} 2024-02-01T10:27:53.761 DEBUG:boto:Token: None 2024-02-01T10:27:53.762 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:53 GMT /vclfcn-19/ 2024-02-01T10:27:53.762 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vrABj8VZn7B6S5a5z/u+OvJhviw= 2024-02-01T10:27:53.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vrABj8VZn7B6S5a5z/u+OvJhviw=', 'Content-Length': '0'} 2024-02-01T10:27:53.767 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d7aea4e636e10bda-0065bb7229-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:53.767 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:53.768 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:53.768 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:53.768 DEBUG:boto:Method: HEAD 2024-02-01T10:27:53.768 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:53.768 DEBUG:boto:Data: 2024-02-01T10:27:53.768 DEBUG:boto:Headers: {} 2024-02-01T10:27:53.768 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:53.768 DEBUG:boto:Port: 8001 2024-02-01T10:27:53.768 DEBUG:boto:Params: {} 2024-02-01T10:27:53.769 DEBUG:boto:Token: None 2024-02-01T10:27:53.769 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:53 GMT /vclfcn-19/obj 2024-02-01T10:27:53.769 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:MjYR912/DHF5JxTn/IpWoqU1xQc= 2024-02-01T10:27:53.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:MjYR912/DHF5JxTn/IpWoqU1xQc=', 'Content-Length': '0'} 2024-02-01T10:27:53.774 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001c2cdbfc198a2b0a-0065bb7229-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:53.775 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:53.775 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:53.775 DEBUG:boto:Method: HEAD 2024-02-01T10:27:53.775 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:53.775 DEBUG:boto:Data: 2024-02-01T10:27:53.775 DEBUG:boto:Headers: {} 2024-02-01T10:27:53.775 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:53.775 DEBUG:boto:Port: 8000 2024-02-01T10:27:53.776 DEBUG:boto:Params: {} 2024-02-01T10:27:53.776 DEBUG:boto:Token: None 2024-02-01T10:27:53.776 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:53 GMT /vclfcn-19/obj 2024-02-01T10:27:53.776 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:MjYR912/DHF5JxTn/IpWoqU1xQc= 2024-02-01T10:27:53.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:MjYR912/DHF5JxTn/IpWoqU1xQc=', 'Content-Length': '0'} 2024-02-01T10:27:53.782 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c53a75a9ed1f5bef-0065bb7229-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:53.782 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:53.782 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:53.782 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:53.783 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:53.783 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:53.783 DEBUG:boto:Method: GET 2024-02-01T10:27:53.783 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:53.783 DEBUG:boto:Data: 2024-02-01T10:27:53.783 DEBUG:boto:Headers: {} 2024-02-01T10:27:53.783 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:53.783 DEBUG:boto:Port: 8001 2024-02-01T10:27:53.783 DEBUG:boto:Params: {} 2024-02-01T10:27:53.783 DEBUG:boto:Token: None 2024-02-01T10:27:53.784 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:53 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:53.784 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ni7wEZeBsvLfWVhnroWf7E1p1S8= 2024-02-01T10:27:53.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ni7wEZeBsvLfWVhnroWf7E1p1S8=', 'Content-Length': '0'} 2024-02-01T10:27:53.786 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001418a246ea2993c9-0065bb7229-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:53.828 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:53.829 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:53.829 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:53.829 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:53.829 DEBUG:boto:Method: GET 2024-02-01T10:27:53.829 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:53.829 DEBUG:boto:Data: 2024-02-01T10:27:53.829 DEBUG:boto:Headers: {} 2024-02-01T10:27:53.829 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:53.829 DEBUG:boto:Port: 8000 2024-02-01T10:27:53.829 DEBUG:boto:Params: {} 2024-02-01T10:27:53.829 DEBUG:boto:Token: None 2024-02-01T10:27:53.830 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:53 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:53.830 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ni7wEZeBsvLfWVhnroWf7E1p1S8= 2024-02-01T10:27:53.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ni7wEZeBsvLfWVhnroWf7E1p1S8=', 'Content-Length': '0'} 2024-02-01T10:27:53.833 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e8b9ea5068a8ad9f-0065bb7229-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:53.833 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:53.833 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:53.833 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:53.833 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:53.833 DEBUG:boto:Method: GET 2024-02-01T10:27:53.834 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:53.834 DEBUG:boto:Data: 2024-02-01T10:27:53.834 DEBUG:boto:Headers: {} 2024-02-01T10:27:53.834 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:53.834 DEBUG:boto:Port: 8001 2024-02-01T10:27:53.834 DEBUG:boto:Params: {} 2024-02-01T10:27:53.834 DEBUG:boto:Token: None 2024-02-01T10:27:53.834 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:53 GMT /vclfcn-19/ 2024-02-01T10:27:53.834 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vrABj8VZn7B6S5a5z/u+OvJhviw= 2024-02-01T10:27:53.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vrABj8VZn7B6S5a5z/u+OvJhviw=', 'Content-Length': '0'} 2024-02-01T10:27:53.840 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb1d3af82ba35484-0065bb7229-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:53.840 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:53.840 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:53.840 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:53.840 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:53.841 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:53.841 DEBUG:boto:Method: GET 2024-02-01T10:27:53.841 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:53.841 DEBUG:boto:Data: 2024-02-01T10:27:53.841 DEBUG:boto:Headers: {} 2024-02-01T10:27:53.841 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:53.841 DEBUG:boto:Port: 8000 2024-02-01T10:27:53.841 DEBUG:boto:Params: {} 2024-02-01T10:27:53.841 DEBUG:boto:Token: None 2024-02-01T10:27:53.841 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:53 GMT /vclfcn-19/ 2024-02-01T10:27:53.842 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vrABj8VZn7B6S5a5z/u+OvJhviw= 2024-02-01T10:27:53.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vrABj8VZn7B6S5a5z/u+OvJhviw=', 'Content-Length': '0'} 2024-02-01T10:27:53.846 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000022208b1104928c37-0065bb7229-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:53.888 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:53.889 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:53.889 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:53.889 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:53.889 DEBUG:boto:path=/vclfcn-19/obj?versionId=VQT2kBCkF07dugmyrhNeRpQpR0HK7bT 2024-02-01T10:27:53.889 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=VQT2kBCkF07dugmyrhNeRpQpR0HK7bT 2024-02-01T10:27:53.889 DEBUG:boto:Method: GET 2024-02-01T10:27:53.889 DEBUG:boto:Path: /vclfcn-19/obj?versionId=VQT2kBCkF07dugmyrhNeRpQpR0HK7bT 2024-02-01T10:27:53.889 DEBUG:boto:Data: 2024-02-01T10:27:53.889 DEBUG:boto:Headers: {} 2024-02-01T10:27:53.889 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:53.890 DEBUG:boto:Port: 8001 2024-02-01T10:27:53.890 DEBUG:boto:Params: {} 2024-02-01T10:27:53.890 DEBUG:boto:Token: None 2024-02-01T10:27:53.890 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:53 GMT /vclfcn-19/obj?versionId=VQT2kBCkF07dugmyrhNeRpQpR0HK7bT 2024-02-01T10:27:53.890 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9htR2nu7j54z/nLGx0dra25hbz0= 2024-02-01T10:27:53.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9htR2nu7j54z/nLGx0dra25hbz0=', 'Content-Length': '0'} 2024-02-01T10:27:53.893 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'VQT2kBCkF07dugmyrhNeRpQpR0HK7bT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b09578ed88193f78-0065bb7229-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:53.893 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:53.893 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:53.894 DEBUG:boto:path=/vclfcn-19/obj?versionId=VQT2kBCkF07dugmyrhNeRpQpR0HK7bT 2024-02-01T10:27:53.894 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=VQT2kBCkF07dugmyrhNeRpQpR0HK7bT 2024-02-01T10:27:53.894 DEBUG:boto:Method: GET 2024-02-01T10:27:53.894 DEBUG:boto:Path: /vclfcn-19/obj?versionId=VQT2kBCkF07dugmyrhNeRpQpR0HK7bT 2024-02-01T10:27:53.894 DEBUG:boto:Data: 2024-02-01T10:27:53.894 DEBUG:boto:Headers: {} 2024-02-01T10:27:53.894 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:53.894 DEBUG:boto:Port: 8000 2024-02-01T10:27:53.894 DEBUG:boto:Params: {} 2024-02-01T10:27:53.894 DEBUG:boto:Token: None 2024-02-01T10:27:53.895 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:53 GMT /vclfcn-19/obj?versionId=VQT2kBCkF07dugmyrhNeRpQpR0HK7bT 2024-02-01T10:27:53.895 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9htR2nu7j54z/nLGx0dra25hbz0= 2024-02-01T10:27:53.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9htR2nu7j54z/nLGx0dra25hbz0=', 'Content-Length': '0'} 2024-02-01T10:27:53.898 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'VQT2kBCkF07dugmyrhNeRpQpR0HK7bT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000174d8bf4eb092d5f-0065bb7229-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:53.898 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:53.898 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:53.898 DEBUG:boto:path=/vclfcn-19/obj?versionId=VQT2kBCkF07dugmyrhNeRpQpR0HK7bT 2024-02-01T10:27:53.898 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=VQT2kBCkF07dugmyrhNeRpQpR0HK7bT 2024-02-01T10:27:53.898 DEBUG:boto:Method: HEAD 2024-02-01T10:27:53.898 DEBUG:boto:Path: /vclfcn-19/obj?versionId=VQT2kBCkF07dugmyrhNeRpQpR0HK7bT 2024-02-01T10:27:53.898 DEBUG:boto:Data: 2024-02-01T10:27:53.899 DEBUG:boto:Headers: {} 2024-02-01T10:27:53.899 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:53.899 DEBUG:boto:Port: 8001 2024-02-01T10:27:53.899 DEBUG:boto:Params: {} 2024-02-01T10:27:53.899 DEBUG:boto:Token: None 2024-02-01T10:27:53.899 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:53 GMT /vclfcn-19/obj?versionId=VQT2kBCkF07dugmyrhNeRpQpR0HK7bT 2024-02-01T10:27:53.899 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:uSVP579pQ65uuC/jbx/Y1rAvSPM= 2024-02-01T10:27:53.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:uSVP579pQ65uuC/jbx/Y1rAvSPM=', 'Content-Length': '0'} 2024-02-01T10:27:53.902 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'VQT2kBCkF07dugmyrhNeRpQpR0HK7bT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fbbf7d364270e99e-0065bb7229-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:53.902 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:53.902 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:53.902 DEBUG:boto:path=/vclfcn-19/obj?versionId=VQT2kBCkF07dugmyrhNeRpQpR0HK7bT 2024-02-01T10:27:53.902 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=VQT2kBCkF07dugmyrhNeRpQpR0HK7bT 2024-02-01T10:27:53.902 DEBUG:boto:Method: HEAD 2024-02-01T10:27:53.903 DEBUG:boto:Path: /vclfcn-19/obj?versionId=VQT2kBCkF07dugmyrhNeRpQpR0HK7bT 2024-02-01T10:27:53.903 DEBUG:boto:Data: 2024-02-01T10:27:53.903 DEBUG:boto:Headers: {} 2024-02-01T10:27:53.903 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:53.903 DEBUG:boto:Port: 8000 2024-02-01T10:27:53.903 DEBUG:boto:Params: {} 2024-02-01T10:27:53.903 DEBUG:boto:Token: None 2024-02-01T10:27:53.903 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:53 GMT /vclfcn-19/obj?versionId=VQT2kBCkF07dugmyrhNeRpQpR0HK7bT 2024-02-01T10:27:53.903 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:uSVP579pQ65uuC/jbx/Y1rAvSPM= 2024-02-01T10:27:53.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:uSVP579pQ65uuC/jbx/Y1rAvSPM=', 'Content-Length': '0'} 2024-02-01T10:27:53.906 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'VQT2kBCkF07dugmyrhNeRpQpR0HK7bT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000215ba5758a2999fa-0065bb7229-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:53.907 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:53.907 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:53.907 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:53.907 DEBUG:boto:path=/vclfcn-19/obj?versionId=VQT2kBCkF07dugmyrhNeRpQpR0HK7bT 2024-02-01T10:27:53.907 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=VQT2kBCkF07dugmyrhNeRpQpR0HK7bT 2024-02-01T10:27:53.907 DEBUG:boto:Method: GET 2024-02-01T10:27:53.907 DEBUG:boto:Path: /vclfcn-19/obj?versionId=VQT2kBCkF07dugmyrhNeRpQpR0HK7bT 2024-02-01T10:27:53.907 DEBUG:boto:Data: 2024-02-01T10:27:53.907 DEBUG:boto:Headers: {} 2024-02-01T10:27:53.907 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:53.908 DEBUG:boto:Port: 8001 2024-02-01T10:27:53.908 DEBUG:boto:Params: {} 2024-02-01T10:27:53.908 DEBUG:boto:Token: None 2024-02-01T10:27:53.908 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:53 GMT /vclfcn-19/obj?versionId=VQT2kBCkF07dugmyrhNeRpQpR0HK7bT 2024-02-01T10:27:53.908 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9htR2nu7j54z/nLGx0dra25hbz0= 2024-02-01T10:27:53.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9htR2nu7j54z/nLGx0dra25hbz0=', 'Content-Length': '0'} 2024-02-01T10:27:53.911 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'VQT2kBCkF07dugmyrhNeRpQpR0HK7bT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bae70501be5a457b-0065bb7229-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:53.952 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:53.952 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:53.952 DEBUG:boto:path=/vclfcn-19/obj?versionId=VQT2kBCkF07dugmyrhNeRpQpR0HK7bT 2024-02-01T10:27:53.952 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=VQT2kBCkF07dugmyrhNeRpQpR0HK7bT 2024-02-01T10:27:53.953 DEBUG:boto:Method: GET 2024-02-01T10:27:53.953 DEBUG:boto:Path: /vclfcn-19/obj?versionId=VQT2kBCkF07dugmyrhNeRpQpR0HK7bT 2024-02-01T10:27:53.953 DEBUG:boto:Data: 2024-02-01T10:27:53.953 DEBUG:boto:Headers: {} 2024-02-01T10:27:53.953 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:53.953 DEBUG:boto:Port: 8000 2024-02-01T10:27:53.953 DEBUG:boto:Params: {} 2024-02-01T10:27:53.953 DEBUG:boto:Token: None 2024-02-01T10:27:53.953 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:53 GMT /vclfcn-19/obj?versionId=VQT2kBCkF07dugmyrhNeRpQpR0HK7bT 2024-02-01T10:27:53.954 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9htR2nu7j54z/nLGx0dra25hbz0= 2024-02-01T10:27:53.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9htR2nu7j54z/nLGx0dra25hbz0=', 'Content-Length': '0'} 2024-02-01T10:27:53.956 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'VQT2kBCkF07dugmyrhNeRpQpR0HK7bT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000075fe11c69e8b7826-0065bb7229-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:54.004 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:54.004 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:54.005 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:54.005 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:54.005 DEBUG:boto:Method: GET 2024-02-01T10:27:54.005 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:54.005 DEBUG:boto:Data: 2024-02-01T10:27:54.005 DEBUG:boto:Headers: {} 2024-02-01T10:27:54.005 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:54.005 DEBUG:boto:Port: 8001 2024-02-01T10:27:54.005 DEBUG:boto:Params: {} 2024-02-01T10:27:54.005 DEBUG:boto:Token: None 2024-02-01T10:27:54.006 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:54 GMT /vclfcn-19/ 2024-02-01T10:27:54.006 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YtCERNENSXODoDJt6JkMjTFuuKs= 2024-02-01T10:27:54.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YtCERNENSXODoDJt6JkMjTFuuKs=', 'Content-Length': '0'} 2024-02-01T10:27:54.012 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b047e86a40ba704b-0065bb722a-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:54.012 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:54.012 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:54.013 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:54.013 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:54.013 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:54.013 DEBUG:boto:Method: GET 2024-02-01T10:27:54.013 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:54.013 DEBUG:boto:Data: 2024-02-01T10:27:54.013 DEBUG:boto:Headers: {} 2024-02-01T10:27:54.013 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:54.013 DEBUG:boto:Port: 8000 2024-02-01T10:27:54.013 DEBUG:boto:Params: {} 2024-02-01T10:27:54.014 DEBUG:boto:Token: None 2024-02-01T10:27:54.014 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:54 GMT /vclfcn-19/ 2024-02-01T10:27:54.014 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YtCERNENSXODoDJt6JkMjTFuuKs= 2024-02-01T10:27:54.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YtCERNENSXODoDJt6JkMjTFuuKs=', 'Content-Length': '0'} 2024-02-01T10:27:54.019 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b58ddd801375eca-0065bb722a-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:54.064 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:54.065 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:54.065 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:54.065 DEBUG:boto:Method: HEAD 2024-02-01T10:27:54.065 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:54.065 DEBUG:boto:Data: 2024-02-01T10:27:54.065 DEBUG:boto:Headers: {} 2024-02-01T10:27:54.065 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:54.065 DEBUG:boto:Port: 8001 2024-02-01T10:27:54.066 DEBUG:boto:Params: {} 2024-02-01T10:27:54.066 DEBUG:boto:Token: None 2024-02-01T10:27:54.066 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:54 GMT /vclfcn-19/obj 2024-02-01T10:27:54.066 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:MowLJxrkrKVbO3xkPdZQ9pgR6EQ= 2024-02-01T10:27:54.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:MowLJxrkrKVbO3xkPdZQ9pgR6EQ=', 'Content-Length': '0'} 2024-02-01T10:27:54.073 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a3a11388b99133fc-0065bb722a-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:54.073 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:54.073 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:54.073 DEBUG:boto:Method: HEAD 2024-02-01T10:27:54.073 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:54.073 DEBUG:boto:Data: 2024-02-01T10:27:54.073 DEBUG:boto:Headers: {} 2024-02-01T10:27:54.073 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:54.073 DEBUG:boto:Port: 8000 2024-02-01T10:27:54.074 DEBUG:boto:Params: {} 2024-02-01T10:27:54.074 DEBUG:boto:Token: None 2024-02-01T10:27:54.074 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:54 GMT /vclfcn-19/obj 2024-02-01T10:27:54.074 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:MowLJxrkrKVbO3xkPdZQ9pgR6EQ= 2024-02-01T10:27:54.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:MowLJxrkrKVbO3xkPdZQ9pgR6EQ=', 'Content-Length': '0'} 2024-02-01T10:27:54.080 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fbb00361d29ff4a1-0065bb722a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:54.080 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:54.080 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:54.080 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:54.080 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:54.080 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:54.081 DEBUG:boto:Method: GET 2024-02-01T10:27:54.081 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:54.081 DEBUG:boto:Data: 2024-02-01T10:27:54.081 DEBUG:boto:Headers: {} 2024-02-01T10:27:54.081 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:54.081 DEBUG:boto:Port: 8001 2024-02-01T10:27:54.081 DEBUG:boto:Params: {} 2024-02-01T10:27:54.082 DEBUG:boto:Token: None 2024-02-01T10:27:54.082 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:54 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:54.082 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ahb8BIqVjGk2Z7+iAAtUgLaACxY= 2024-02-01T10:27:54.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ahb8BIqVjGk2Z7+iAAtUgLaACxY=', 'Content-Length': '0'} 2024-02-01T10:27:54.085 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009d9da9083d14a32c-0065bb722a-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:54.128 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:54.128 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:54.128 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:54.128 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:54.129 DEBUG:boto:Method: GET 2024-02-01T10:27:54.129 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:54.129 DEBUG:boto:Data: 2024-02-01T10:27:54.129 DEBUG:boto:Headers: {} 2024-02-01T10:27:54.129 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:54.129 DEBUG:boto:Port: 8000 2024-02-01T10:27:54.129 DEBUG:boto:Params: {} 2024-02-01T10:27:54.130 DEBUG:boto:Token: None 2024-02-01T10:27:54.130 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:54 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:54.130 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ahb8BIqVjGk2Z7+iAAtUgLaACxY= 2024-02-01T10:27:54.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ahb8BIqVjGk2Z7+iAAtUgLaACxY=', 'Content-Length': '0'} 2024-02-01T10:27:54.134 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000063ee6d0b64b9d4ec-0065bb722a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:54.134 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:54.134 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:54.134 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:54.134 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:54.134 DEBUG:boto:Method: GET 2024-02-01T10:27:54.135 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:54.135 DEBUG:boto:Data: 2024-02-01T10:27:54.135 DEBUG:boto:Headers: {} 2024-02-01T10:27:54.135 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:54.135 DEBUG:boto:Port: 8001 2024-02-01T10:27:54.135 DEBUG:boto:Params: {} 2024-02-01T10:27:54.135 DEBUG:boto:Token: None 2024-02-01T10:27:54.135 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:54 GMT /vclfcn-19/ 2024-02-01T10:27:54.135 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YtCERNENSXODoDJt6JkMjTFuuKs= 2024-02-01T10:27:54.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YtCERNENSXODoDJt6JkMjTFuuKs=', 'Content-Length': '0'} 2024-02-01T10:27:54.141 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a318bcbacdd928d-0065bb722a-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:54.141 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:54.141 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:54.141 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:54.141 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:54.142 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:54.142 DEBUG:boto:Method: GET 2024-02-01T10:27:54.142 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:54.142 DEBUG:boto:Data: 2024-02-01T10:27:54.142 DEBUG:boto:Headers: {} 2024-02-01T10:27:54.142 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:54.142 DEBUG:boto:Port: 8000 2024-02-01T10:27:54.142 DEBUG:boto:Params: {} 2024-02-01T10:27:54.143 DEBUG:boto:Token: None 2024-02-01T10:27:54.143 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:54 GMT /vclfcn-19/ 2024-02-01T10:27:54.143 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YtCERNENSXODoDJt6JkMjTFuuKs= 2024-02-01T10:27:54.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YtCERNENSXODoDJt6JkMjTFuuKs=', 'Content-Length': '0'} 2024-02-01T10:27:54.148 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f1badc8e7eedb21-0065bb722a-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:54.149 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:54.149 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:54.149 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:54.149 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:54.150 DEBUG:boto:path=/vclfcn-19/obj?versionId=tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi 2024-02-01T10:27:54.150 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi 2024-02-01T10:27:54.150 DEBUG:boto:Method: GET 2024-02-01T10:27:54.150 DEBUG:boto:Path: /vclfcn-19/obj?versionId=tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi 2024-02-01T10:27:54.150 DEBUG:boto:Data: 2024-02-01T10:27:54.150 DEBUG:boto:Headers: {} 2024-02-01T10:27:54.150 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:54.150 DEBUG:boto:Port: 8001 2024-02-01T10:27:54.150 DEBUG:boto:Params: {} 2024-02-01T10:27:54.150 DEBUG:boto:Token: None 2024-02-01T10:27:54.151 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:54 GMT /vclfcn-19/obj?versionId=tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi 2024-02-01T10:27:54.151 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zHklQcmXNixMFDArOEtAJZW28j0= 2024-02-01T10:27:54.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zHklQcmXNixMFDArOEtAJZW28j0=', 'Content-Length': '0'} 2024-02-01T10:27:54.156 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000215318c527b29d6c-0065bb722a-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:54.196 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:54.196 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:54.197 DEBUG:boto:path=/vclfcn-19/obj?versionId=tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi 2024-02-01T10:27:54.197 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi 2024-02-01T10:27:54.197 DEBUG:boto:Method: GET 2024-02-01T10:27:54.197 DEBUG:boto:Path: /vclfcn-19/obj?versionId=tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi 2024-02-01T10:27:54.197 DEBUG:boto:Data: 2024-02-01T10:27:54.197 DEBUG:boto:Headers: {} 2024-02-01T10:27:54.197 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:54.197 DEBUG:boto:Port: 8000 2024-02-01T10:27:54.197 DEBUG:boto:Params: {} 2024-02-01T10:27:54.197 DEBUG:boto:Token: None 2024-02-01T10:27:54.198 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:54 GMT /vclfcn-19/obj?versionId=tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi 2024-02-01T10:27:54.198 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zHklQcmXNixMFDArOEtAJZW28j0= 2024-02-01T10:27:54.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zHklQcmXNixMFDArOEtAJZW28j0=', 'Content-Length': '0'} 2024-02-01T10:27:54.202 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fdb5ec11ee674994-0065bb722a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:54.244 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:54.244 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:54.244 DEBUG:boto:path=/vclfcn-19/obj?versionId=tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi 2024-02-01T10:27:54.245 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi 2024-02-01T10:27:54.245 DEBUG:boto:Method: HEAD 2024-02-01T10:27:54.245 DEBUG:boto:Path: /vclfcn-19/obj?versionId=tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi 2024-02-01T10:27:54.245 DEBUG:boto:Data: 2024-02-01T10:27:54.245 DEBUG:boto:Headers: {} 2024-02-01T10:27:54.245 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:54.245 DEBUG:boto:Port: 8001 2024-02-01T10:27:54.245 DEBUG:boto:Params: {} 2024-02-01T10:27:54.245 DEBUG:boto:Token: None 2024-02-01T10:27:54.246 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:54 GMT /vclfcn-19/obj?versionId=tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi 2024-02-01T10:27:54.246 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:dPRIX/lTJ+sHOvdOLLvbTIUKDZw= 2024-02-01T10:27:54.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dPRIX/lTJ+sHOvdOLLvbTIUKDZw=', 'Content-Length': '0'} 2024-02-01T10:27:54.249 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e57eebb0494eee69-0065bb722a-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:54.249 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:54.250 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:54.250 DEBUG:boto:path=/vclfcn-19/obj?versionId=tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi 2024-02-01T10:27:54.250 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi 2024-02-01T10:27:54.250 DEBUG:boto:Method: HEAD 2024-02-01T10:27:54.250 DEBUG:boto:Path: /vclfcn-19/obj?versionId=tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi 2024-02-01T10:27:54.250 DEBUG:boto:Data: 2024-02-01T10:27:54.250 DEBUG:boto:Headers: {} 2024-02-01T10:27:54.250 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:54.250 DEBUG:boto:Port: 8000 2024-02-01T10:27:54.250 DEBUG:boto:Params: {} 2024-02-01T10:27:54.251 DEBUG:boto:Token: None 2024-02-01T10:27:54.251 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:54 GMT /vclfcn-19/obj?versionId=tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi 2024-02-01T10:27:54.251 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:dPRIX/lTJ+sHOvdOLLvbTIUKDZw= 2024-02-01T10:27:54.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dPRIX/lTJ+sHOvdOLLvbTIUKDZw=', 'Content-Length': '0'} 2024-02-01T10:27:54.254 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eec33f30edca1e35-0065bb722a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:54.254 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:54.254 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:54.255 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:54.255 DEBUG:boto:path=/vclfcn-19/obj?versionId=tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi 2024-02-01T10:27:54.255 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi 2024-02-01T10:27:54.255 DEBUG:boto:Method: GET 2024-02-01T10:27:54.255 DEBUG:boto:Path: /vclfcn-19/obj?versionId=tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi 2024-02-01T10:27:54.255 DEBUG:boto:Data: 2024-02-01T10:27:54.255 DEBUG:boto:Headers: {} 2024-02-01T10:27:54.255 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:54.255 DEBUG:boto:Port: 8001 2024-02-01T10:27:54.255 DEBUG:boto:Params: {} 2024-02-01T10:27:54.255 DEBUG:boto:Token: None 2024-02-01T10:27:54.256 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:54 GMT /vclfcn-19/obj?versionId=tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi 2024-02-01T10:27:54.256 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zHklQcmXNixMFDArOEtAJZW28j0= 2024-02-01T10:27:54.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zHklQcmXNixMFDArOEtAJZW28j0=', 'Content-Length': '0'} 2024-02-01T10:27:54.260 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003d8e241a8790c198-0065bb722a-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:54.304 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:54.304 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:54.304 DEBUG:boto:path=/vclfcn-19/obj?versionId=tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi 2024-02-01T10:27:54.304 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi 2024-02-01T10:27:54.305 DEBUG:boto:Method: GET 2024-02-01T10:27:54.305 DEBUG:boto:Path: /vclfcn-19/obj?versionId=tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi 2024-02-01T10:27:54.305 DEBUG:boto:Data: 2024-02-01T10:27:54.305 DEBUG:boto:Headers: {} 2024-02-01T10:27:54.305 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:54.305 DEBUG:boto:Port: 8000 2024-02-01T10:27:54.305 DEBUG:boto:Params: {} 2024-02-01T10:27:54.305 DEBUG:boto:Token: None 2024-02-01T10:27:54.305 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:54 GMT /vclfcn-19/obj?versionId=tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi 2024-02-01T10:27:54.306 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zHklQcmXNixMFDArOEtAJZW28j0= 2024-02-01T10:27:54.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zHklQcmXNixMFDArOEtAJZW28j0=', 'Content-Length': '0'} 2024-02-01T10:27:54.309 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'tvvjWUbgZIRYhHk9r.liT6sPCWCs2Zi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005be0de3020b4705c-0065bb722a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:54.356 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:54.356 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:54.356 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:54.357 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:54.357 DEBUG:boto:Method: GET 2024-02-01T10:27:54.357 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:54.357 DEBUG:boto:Data: 2024-02-01T10:27:54.357 DEBUG:boto:Headers: {} 2024-02-01T10:27:54.357 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:54.357 DEBUG:boto:Port: 8001 2024-02-01T10:27:54.357 DEBUG:boto:Params: {} 2024-02-01T10:27:54.357 DEBUG:boto:Token: None 2024-02-01T10:27:54.358 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:54 GMT /vclfcn-19/ 2024-02-01T10:27:54.358 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YtCERNENSXODoDJt6JkMjTFuuKs= 2024-02-01T10:27:54.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YtCERNENSXODoDJt6JkMjTFuuKs=', 'Content-Length': '0'} 2024-02-01T10:27:54.363 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000073fa631d271a3c96-0065bb722a-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:54.363 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:54.364 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:54.364 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:54.364 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:54.364 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:54.364 DEBUG:boto:Method: GET 2024-02-01T10:27:54.364 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:54.364 DEBUG:boto:Data: 2024-02-01T10:27:54.364 DEBUG:boto:Headers: {} 2024-02-01T10:27:54.364 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:54.364 DEBUG:boto:Port: 8000 2024-02-01T10:27:54.365 DEBUG:boto:Params: {} 2024-02-01T10:27:54.365 DEBUG:boto:Token: None 2024-02-01T10:27:54.365 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:54 GMT /vclfcn-19/ 2024-02-01T10:27:54.365 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YtCERNENSXODoDJt6JkMjTFuuKs= 2024-02-01T10:27:54.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YtCERNENSXODoDJt6JkMjTFuuKs=', 'Content-Length': '0'} 2024-02-01T10:27:54.372 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011b6f4bcbe1fc974-0065bb722a-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:54.372 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:54.372 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:54.372 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:54.373 DEBUG:boto:Method: HEAD 2024-02-01T10:27:54.373 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:54.373 DEBUG:boto:Data: 2024-02-01T10:27:54.373 DEBUG:boto:Headers: {} 2024-02-01T10:27:54.373 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:54.373 DEBUG:boto:Port: 8001 2024-02-01T10:27:54.373 DEBUG:boto:Params: {} 2024-02-01T10:27:54.373 DEBUG:boto:Token: None 2024-02-01T10:27:54.373 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:54 GMT /vclfcn-19/obj 2024-02-01T10:27:54.373 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:MowLJxrkrKVbO3xkPdZQ9pgR6EQ= 2024-02-01T10:27:54.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:MowLJxrkrKVbO3xkPdZQ9pgR6EQ=', 'Content-Length': '0'} 2024-02-01T10:27:54.379 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000912fe029778e68f6-0065bb722a-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:54.379 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:54.379 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:54.380 DEBUG:boto:Method: HEAD 2024-02-01T10:27:54.380 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:54.380 DEBUG:boto:Data: 2024-02-01T10:27:54.380 DEBUG:boto:Headers: {} 2024-02-01T10:27:54.380 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:54.380 DEBUG:boto:Port: 8000 2024-02-01T10:27:54.380 DEBUG:boto:Params: {} 2024-02-01T10:27:54.380 DEBUG:boto:Token: None 2024-02-01T10:27:54.380 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:54 GMT /vclfcn-19/obj 2024-02-01T10:27:54.381 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:MowLJxrkrKVbO3xkPdZQ9pgR6EQ= 2024-02-01T10:27:54.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:MowLJxrkrKVbO3xkPdZQ9pgR6EQ=', 'Content-Length': '0'} 2024-02-01T10:27:54.386 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007942789890eeaa34-0065bb722a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:54.387 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:54.387 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:54.387 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:54.387 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:54.387 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:54.387 DEBUG:boto:Method: GET 2024-02-01T10:27:54.387 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:54.387 DEBUG:boto:Data: 2024-02-01T10:27:54.387 DEBUG:boto:Headers: {} 2024-02-01T10:27:54.388 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:54.388 DEBUG:boto:Port: 8001 2024-02-01T10:27:54.388 DEBUG:boto:Params: {} 2024-02-01T10:27:54.388 DEBUG:boto:Token: None 2024-02-01T10:27:54.388 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:54 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:54.388 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ahb8BIqVjGk2Z7+iAAtUgLaACxY= 2024-02-01T10:27:54.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ahb8BIqVjGk2Z7+iAAtUgLaACxY=', 'Content-Length': '0'} 2024-02-01T10:27:54.391 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b122554c4595907f-0065bb722a-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:54.432 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:54.432 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:54.432 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:54.432 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:54.433 DEBUG:boto:Method: GET 2024-02-01T10:27:54.433 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:54.433 DEBUG:boto:Data: 2024-02-01T10:27:54.433 DEBUG:boto:Headers: {} 2024-02-01T10:27:54.433 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:54.433 DEBUG:boto:Port: 8000 2024-02-01T10:27:54.433 DEBUG:boto:Params: {} 2024-02-01T10:27:54.433 DEBUG:boto:Token: None 2024-02-01T10:27:54.433 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:54 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:54.434 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ahb8BIqVjGk2Z7+iAAtUgLaACxY= 2024-02-01T10:27:54.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ahb8BIqVjGk2Z7+iAAtUgLaACxY=', 'Content-Length': '0'} 2024-02-01T10:27:54.436 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000038a51ab5dc659802-0065bb722a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:54.437 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:54.437 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:54.437 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:54.437 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:54.437 DEBUG:boto:Method: GET 2024-02-01T10:27:54.437 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:54.437 DEBUG:boto:Data: 2024-02-01T10:27:54.437 DEBUG:boto:Headers: {} 2024-02-01T10:27:54.438 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:54.438 DEBUG:boto:Port: 8001 2024-02-01T10:27:54.438 DEBUG:boto:Params: {} 2024-02-01T10:27:54.438 DEBUG:boto:Token: None 2024-02-01T10:27:54.438 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:54 GMT /vclfcn-19/ 2024-02-01T10:27:54.438 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YtCERNENSXODoDJt6JkMjTFuuKs= 2024-02-01T10:27:54.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YtCERNENSXODoDJt6JkMjTFuuKs=', 'Content-Length': '0'} 2024-02-01T10:27:54.444 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b0ef12177b73c2e-0065bb722a-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:54.444 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:54.444 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:54.445 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:54.445 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:54.445 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:54.445 DEBUG:boto:Method: GET 2024-02-01T10:27:54.445 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:54.445 DEBUG:boto:Data: 2024-02-01T10:27:54.445 DEBUG:boto:Headers: {} 2024-02-01T10:27:54.445 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:54.445 DEBUG:boto:Port: 8000 2024-02-01T10:27:54.445 DEBUG:boto:Params: {} 2024-02-01T10:27:54.445 DEBUG:boto:Token: None 2024-02-01T10:27:54.446 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:54 GMT /vclfcn-19/ 2024-02-01T10:27:54.446 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YtCERNENSXODoDJt6JkMjTFuuKs= 2024-02-01T10:27:54.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YtCERNENSXODoDJt6JkMjTFuuKs=', 'Content-Length': '0'} 2024-02-01T10:27:54.452 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004fce46a11fc625fe-0065bb722a-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:54.496 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:54.497 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:54.497 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:54.497 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:54.497 DEBUG:boto:path=/vclfcn-19/obj?versionId=3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz 2024-02-01T10:27:54.497 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz 2024-02-01T10:27:54.497 DEBUG:boto:Method: GET 2024-02-01T10:27:54.497 DEBUG:boto:Path: /vclfcn-19/obj?versionId=3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz 2024-02-01T10:27:54.497 DEBUG:boto:Data: 2024-02-01T10:27:54.497 DEBUG:boto:Headers: {} 2024-02-01T10:27:54.498 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:54.498 DEBUG:boto:Port: 8001 2024-02-01T10:27:54.498 DEBUG:boto:Params: {} 2024-02-01T10:27:54.498 DEBUG:boto:Token: None 2024-02-01T10:27:54.498 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:54 GMT /vclfcn-19/obj?versionId=3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz 2024-02-01T10:27:54.498 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:biVqYi4VDj+LC+4b7HeTN2wbApA= 2024-02-01T10:27:54.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:biVqYi4VDj+LC+4b7HeTN2wbApA=', 'Content-Length': '0'} 2024-02-01T10:27:54.501 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ada6f4a5a17627a6-0065bb722a-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:54.502 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:54.502 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:54.502 DEBUG:boto:path=/vclfcn-19/obj?versionId=3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz 2024-02-01T10:27:54.502 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz 2024-02-01T10:27:54.502 DEBUG:boto:Method: GET 2024-02-01T10:27:54.502 DEBUG:boto:Path: /vclfcn-19/obj?versionId=3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz 2024-02-01T10:27:54.502 DEBUG:boto:Data: 2024-02-01T10:27:54.502 DEBUG:boto:Headers: {} 2024-02-01T10:27:54.502 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:54.503 DEBUG:boto:Port: 8000 2024-02-01T10:27:54.503 DEBUG:boto:Params: {} 2024-02-01T10:27:54.503 DEBUG:boto:Token: None 2024-02-01T10:27:54.503 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:54 GMT /vclfcn-19/obj?versionId=3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz 2024-02-01T10:27:54.503 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:biVqYi4VDj+LC+4b7HeTN2wbApA= 2024-02-01T10:27:54.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:biVqYi4VDj+LC+4b7HeTN2wbApA=', 'Content-Length': '0'} 2024-02-01T10:27:54.506 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b93a2c21cf68f303-0065bb722a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:54.506 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:54.506 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:54.506 DEBUG:boto:path=/vclfcn-19/obj?versionId=3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz 2024-02-01T10:27:54.507 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz 2024-02-01T10:27:54.507 DEBUG:boto:Method: HEAD 2024-02-01T10:27:54.507 DEBUG:boto:Path: /vclfcn-19/obj?versionId=3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz 2024-02-01T10:27:54.507 DEBUG:boto:Data: 2024-02-01T10:27:54.507 DEBUG:boto:Headers: {} 2024-02-01T10:27:54.507 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:54.507 DEBUG:boto:Port: 8001 2024-02-01T10:27:54.507 DEBUG:boto:Params: {} 2024-02-01T10:27:54.507 DEBUG:boto:Token: None 2024-02-01T10:27:54.508 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:54 GMT /vclfcn-19/obj?versionId=3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz 2024-02-01T10:27:54.508 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:m/Awh6k7bcgeD6Gm2X3VmX+ufiI= 2024-02-01T10:27:54.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:m/Awh6k7bcgeD6Gm2X3VmX+ufiI=', 'Content-Length': '0'} 2024-02-01T10:27:54.510 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ad39ffb5c0a089d6-0065bb722a-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:54.511 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:54.511 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:54.511 DEBUG:boto:path=/vclfcn-19/obj?versionId=3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz 2024-02-01T10:27:54.511 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz 2024-02-01T10:27:54.511 DEBUG:boto:Method: HEAD 2024-02-01T10:27:54.511 DEBUG:boto:Path: /vclfcn-19/obj?versionId=3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz 2024-02-01T10:27:54.511 DEBUG:boto:Data: 2024-02-01T10:27:54.511 DEBUG:boto:Headers: {} 2024-02-01T10:27:54.511 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:54.511 DEBUG:boto:Port: 8000 2024-02-01T10:27:54.512 DEBUG:boto:Params: {} 2024-02-01T10:27:54.512 DEBUG:boto:Token: None 2024-02-01T10:27:54.512 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:54 GMT /vclfcn-19/obj?versionId=3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz 2024-02-01T10:27:54.512 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:m/Awh6k7bcgeD6Gm2X3VmX+ufiI= 2024-02-01T10:27:54.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:m/Awh6k7bcgeD6Gm2X3VmX+ufiI=', 'Content-Length': '0'} 2024-02-01T10:27:54.515 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cb159e094421aa66-0065bb722a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:54.515 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:54.515 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:54.515 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:54.515 DEBUG:boto:path=/vclfcn-19/obj?versionId=3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz 2024-02-01T10:27:54.515 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz 2024-02-01T10:27:54.515 DEBUG:boto:Method: GET 2024-02-01T10:27:54.515 DEBUG:boto:Path: /vclfcn-19/obj?versionId=3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz 2024-02-01T10:27:54.516 DEBUG:boto:Data: 2024-02-01T10:27:54.516 DEBUG:boto:Headers: {} 2024-02-01T10:27:54.516 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:54.516 DEBUG:boto:Port: 8001 2024-02-01T10:27:54.516 DEBUG:boto:Params: {} 2024-02-01T10:27:54.516 DEBUG:boto:Token: None 2024-02-01T10:27:54.516 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:54 GMT /vclfcn-19/obj?versionId=3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz 2024-02-01T10:27:54.516 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:biVqYi4VDj+LC+4b7HeTN2wbApA= 2024-02-01T10:27:54.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:biVqYi4VDj+LC+4b7HeTN2wbApA=', 'Content-Length': '0'} 2024-02-01T10:27:54.519 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000091d4ba8f82f7bd1b-0065bb722a-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:54.560 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:54.560 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:54.560 DEBUG:boto:path=/vclfcn-19/obj?versionId=3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz 2024-02-01T10:27:54.560 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz 2024-02-01T10:27:54.561 DEBUG:boto:Method: GET 2024-02-01T10:27:54.561 DEBUG:boto:Path: /vclfcn-19/obj?versionId=3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz 2024-02-01T10:27:54.561 DEBUG:boto:Data: 2024-02-01T10:27:54.561 DEBUG:boto:Headers: {} 2024-02-01T10:27:54.561 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:54.561 DEBUG:boto:Port: 8000 2024-02-01T10:27:54.561 DEBUG:boto:Params: {} 2024-02-01T10:27:54.561 DEBUG:boto:Token: None 2024-02-01T10:27:54.561 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:54 GMT /vclfcn-19/obj?versionId=3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz 2024-02-01T10:27:54.562 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:biVqYi4VDj+LC+4b7HeTN2wbApA= 2024-02-01T10:27:54.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:biVqYi4VDj+LC+4b7HeTN2wbApA=', 'Content-Length': '0'} 2024-02-01T10:27:54.565 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '3OSaU40E5-Hgv6pEDB8btq8uZcAFHVz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005e716778d4bc3b72-0065bb722a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:54.565 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:54.565 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:54.565 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:54.565 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:54.565 DEBUG:boto:Method: GET 2024-02-01T10:27:54.565 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:54.566 DEBUG:boto:Data: 2024-02-01T10:27:54.566 DEBUG:boto:Headers: {} 2024-02-01T10:27:54.566 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:54.566 DEBUG:boto:Port: 8001 2024-02-01T10:27:54.566 DEBUG:boto:Params: {} 2024-02-01T10:27:54.566 DEBUG:boto:Token: None 2024-02-01T10:27:54.566 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:54 GMT /vclfcn-19/ 2024-02-01T10:27:54.566 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YtCERNENSXODoDJt6JkMjTFuuKs= 2024-02-01T10:27:54.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YtCERNENSXODoDJt6JkMjTFuuKs=', 'Content-Length': '0'} 2024-02-01T10:27:54.571 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ec54f1e5c87e8b7-0065bb722a-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:54.612 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:54.612 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:54.613 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:54.613 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:54.613 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:54.613 DEBUG:boto:Method: GET 2024-02-01T10:27:54.613 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:54.613 DEBUG:boto:Data: 2024-02-01T10:27:54.613 DEBUG:boto:Headers: {} 2024-02-01T10:27:54.613 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:54.613 DEBUG:boto:Port: 8000 2024-02-01T10:27:54.613 DEBUG:boto:Params: {} 2024-02-01T10:27:54.614 DEBUG:boto:Token: None 2024-02-01T10:27:54.614 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:54 GMT /vclfcn-19/ 2024-02-01T10:27:54.614 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YtCERNENSXODoDJt6JkMjTFuuKs= 2024-02-01T10:27:54.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YtCERNENSXODoDJt6JkMjTFuuKs=', 'Content-Length': '0'} 2024-02-01T10:27:54.619 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6a772eadc37ef6a-0065bb722a-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:54.619 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:54.620 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:54.620 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:54.620 DEBUG:boto:Method: HEAD 2024-02-01T10:27:54.620 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:54.620 DEBUG:boto:Data: 2024-02-01T10:27:54.620 DEBUG:boto:Headers: {} 2024-02-01T10:27:54.621 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:54.621 DEBUG:boto:Port: 8001 2024-02-01T10:27:54.621 DEBUG:boto:Params: {} 2024-02-01T10:27:54.621 DEBUG:boto:Token: None 2024-02-01T10:27:54.621 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:54 GMT /vclfcn-19/obj 2024-02-01T10:27:54.622 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:MowLJxrkrKVbO3xkPdZQ9pgR6EQ= 2024-02-01T10:27:54.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:MowLJxrkrKVbO3xkPdZQ9pgR6EQ=', 'Content-Length': '0'} 2024-02-01T10:27:54.627 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002fd9ffa2e2d5027f-0065bb722a-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:54.628 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:54.628 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:54.628 DEBUG:boto:Method: HEAD 2024-02-01T10:27:54.628 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:54.628 DEBUG:boto:Data: 2024-02-01T10:27:54.628 DEBUG:boto:Headers: {} 2024-02-01T10:27:54.628 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:54.628 DEBUG:boto:Port: 8000 2024-02-01T10:27:54.628 DEBUG:boto:Params: {} 2024-02-01T10:27:54.629 DEBUG:boto:Token: None 2024-02-01T10:27:54.629 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:54 GMT /vclfcn-19/obj 2024-02-01T10:27:54.629 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:MowLJxrkrKVbO3xkPdZQ9pgR6EQ= 2024-02-01T10:27:54.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:MowLJxrkrKVbO3xkPdZQ9pgR6EQ=', 'Content-Length': '0'} 2024-02-01T10:27:54.634 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000363639a86b2e9f1c-0065bb722a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:54.635 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:54.635 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:54.635 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:54.635 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:54.635 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:54.635 DEBUG:boto:Method: GET 2024-02-01T10:27:54.635 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:54.636 DEBUG:boto:Data: 2024-02-01T10:27:54.636 DEBUG:boto:Headers: {} 2024-02-01T10:27:54.636 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:54.636 DEBUG:boto:Port: 8001 2024-02-01T10:27:54.636 DEBUG:boto:Params: {} 2024-02-01T10:27:54.636 DEBUG:boto:Token: None 2024-02-01T10:27:54.636 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:54 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:54.636 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ahb8BIqVjGk2Z7+iAAtUgLaACxY= 2024-02-01T10:27:54.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ahb8BIqVjGk2Z7+iAAtUgLaACxY=', 'Content-Length': '0'} 2024-02-01T10:27:54.640 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001518dd71d281b03e-0065bb722a-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:54.680 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:54.680 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:54.680 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:54.680 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:54.681 DEBUG:boto:Method: GET 2024-02-01T10:27:54.681 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:54.681 DEBUG:boto:Data: 2024-02-01T10:27:54.681 DEBUG:boto:Headers: {} 2024-02-01T10:27:54.681 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:54.681 DEBUG:boto:Port: 8000 2024-02-01T10:27:54.681 DEBUG:boto:Params: {} 2024-02-01T10:27:54.681 DEBUG:boto:Token: None 2024-02-01T10:27:54.681 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:54 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:54.682 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ahb8BIqVjGk2Z7+iAAtUgLaACxY= 2024-02-01T10:27:54.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ahb8BIqVjGk2Z7+iAAtUgLaACxY=', 'Content-Length': '0'} 2024-02-01T10:27:54.685 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000621ab72087703dc4-0065bb722a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:54.685 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:54.685 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:54.685 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:54.685 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:54.685 DEBUG:boto:Method: GET 2024-02-01T10:27:54.686 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:54.686 DEBUG:boto:Data: 2024-02-01T10:27:54.686 DEBUG:boto:Headers: {} 2024-02-01T10:27:54.686 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:54.686 DEBUG:boto:Port: 8001 2024-02-01T10:27:54.686 DEBUG:boto:Params: {} 2024-02-01T10:27:54.686 DEBUG:boto:Token: None 2024-02-01T10:27:54.686 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:54 GMT /vclfcn-19/ 2024-02-01T10:27:54.686 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YtCERNENSXODoDJt6JkMjTFuuKs= 2024-02-01T10:27:54.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YtCERNENSXODoDJt6JkMjTFuuKs=', 'Content-Length': '0'} 2024-02-01T10:27:54.691 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b47a9587e5f697a0-0065bb722a-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:54.732 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:54.733 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:54.733 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:54.733 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:54.733 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:54.733 DEBUG:boto:Method: GET 2024-02-01T10:27:54.733 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:54.734 DEBUG:boto:Data: 2024-02-01T10:27:54.734 DEBUG:boto:Headers: {} 2024-02-01T10:27:54.734 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:54.734 DEBUG:boto:Port: 8000 2024-02-01T10:27:54.734 DEBUG:boto:Params: {} 2024-02-01T10:27:54.734 DEBUG:boto:Token: None 2024-02-01T10:27:54.734 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:54 GMT /vclfcn-19/ 2024-02-01T10:27:54.734 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YtCERNENSXODoDJt6JkMjTFuuKs= 2024-02-01T10:27:54.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YtCERNENSXODoDJt6JkMjTFuuKs=', 'Content-Length': '0'} 2024-02-01T10:27:54.740 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da805d9fd90760dd-0065bb722a-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:54.740 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:54.740 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:54.741 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:54.741 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:54.741 DEBUG:boto:path=/vclfcn-19/obj?versionId=K23uwsLWGmQDGYGsdu-h04Fk2wApB0O 2024-02-01T10:27:54.741 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=K23uwsLWGmQDGYGsdu-h04Fk2wApB0O 2024-02-01T10:27:54.741 DEBUG:boto:Method: GET 2024-02-01T10:27:54.741 DEBUG:boto:Path: /vclfcn-19/obj?versionId=K23uwsLWGmQDGYGsdu-h04Fk2wApB0O 2024-02-01T10:27:54.741 DEBUG:boto:Data: 2024-02-01T10:27:54.741 DEBUG:boto:Headers: {} 2024-02-01T10:27:54.741 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:54.741 DEBUG:boto:Port: 8001 2024-02-01T10:27:54.741 DEBUG:boto:Params: {} 2024-02-01T10:27:54.741 DEBUG:boto:Token: None 2024-02-01T10:27:54.742 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:54 GMT /vclfcn-19/obj?versionId=K23uwsLWGmQDGYGsdu-h04Fk2wApB0O 2024-02-01T10:27:54.742 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YSpbsnefeZp5iQg7P7LQlrgTNE0= 2024-02-01T10:27:54.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YSpbsnefeZp5iQg7P7LQlrgTNE0=', 'Content-Length': '0'} 2024-02-01T10:27:54.744 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'K23uwsLWGmQDGYGsdu-h04Fk2wApB0O'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005699a631a876ffe1-0065bb722a-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:54.745 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:54.745 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:54.745 DEBUG:boto:path=/vclfcn-19/obj?versionId=K23uwsLWGmQDGYGsdu-h04Fk2wApB0O 2024-02-01T10:27:54.745 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=K23uwsLWGmQDGYGsdu-h04Fk2wApB0O 2024-02-01T10:27:54.745 DEBUG:boto:Method: GET 2024-02-01T10:27:54.746 DEBUG:boto:Path: /vclfcn-19/obj?versionId=K23uwsLWGmQDGYGsdu-h04Fk2wApB0O 2024-02-01T10:27:54.746 DEBUG:boto:Data: 2024-02-01T10:27:54.746 DEBUG:boto:Headers: {} 2024-02-01T10:27:54.746 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:54.746 DEBUG:boto:Port: 8000 2024-02-01T10:27:54.746 DEBUG:boto:Params: {} 2024-02-01T10:27:54.746 DEBUG:boto:Token: None 2024-02-01T10:27:54.746 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:54 GMT /vclfcn-19/obj?versionId=K23uwsLWGmQDGYGsdu-h04Fk2wApB0O 2024-02-01T10:27:54.746 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YSpbsnefeZp5iQg7P7LQlrgTNE0= 2024-02-01T10:27:54.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YSpbsnefeZp5iQg7P7LQlrgTNE0=', 'Content-Length': '0'} 2024-02-01T10:27:54.750 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'K23uwsLWGmQDGYGsdu-h04Fk2wApB0O'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f7327d1a4aa088db-0065bb722a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:54.792 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:54.792 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:54.792 DEBUG:boto:path=/vclfcn-19/obj?versionId=K23uwsLWGmQDGYGsdu-h04Fk2wApB0O 2024-02-01T10:27:54.793 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=K23uwsLWGmQDGYGsdu-h04Fk2wApB0O 2024-02-01T10:27:54.793 DEBUG:boto:Method: HEAD 2024-02-01T10:27:54.793 DEBUG:boto:Path: /vclfcn-19/obj?versionId=K23uwsLWGmQDGYGsdu-h04Fk2wApB0O 2024-02-01T10:27:54.793 DEBUG:boto:Data: 2024-02-01T10:27:54.793 DEBUG:boto:Headers: {} 2024-02-01T10:27:54.793 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:54.793 DEBUG:boto:Port: 8001 2024-02-01T10:27:54.793 DEBUG:boto:Params: {} 2024-02-01T10:27:54.793 DEBUG:boto:Token: None 2024-02-01T10:27:54.794 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:54 GMT /vclfcn-19/obj?versionId=K23uwsLWGmQDGYGsdu-h04Fk2wApB0O 2024-02-01T10:27:54.794 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gnvoLqLuFfjp+3MyEF5cnP67GFA= 2024-02-01T10:27:54.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gnvoLqLuFfjp+3MyEF5cnP67GFA=', 'Content-Length': '0'} 2024-02-01T10:27:54.797 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'K23uwsLWGmQDGYGsdu-h04Fk2wApB0O'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fffb0f267756ed55-0065bb722a-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:54.798 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:54.798 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:54.798 DEBUG:boto:path=/vclfcn-19/obj?versionId=K23uwsLWGmQDGYGsdu-h04Fk2wApB0O 2024-02-01T10:27:54.798 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=K23uwsLWGmQDGYGsdu-h04Fk2wApB0O 2024-02-01T10:27:54.798 DEBUG:boto:Method: HEAD 2024-02-01T10:27:54.798 DEBUG:boto:Path: /vclfcn-19/obj?versionId=K23uwsLWGmQDGYGsdu-h04Fk2wApB0O 2024-02-01T10:27:54.798 DEBUG:boto:Data: 2024-02-01T10:27:54.798 DEBUG:boto:Headers: {} 2024-02-01T10:27:54.798 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:54.799 DEBUG:boto:Port: 8000 2024-02-01T10:27:54.799 DEBUG:boto:Params: {} 2024-02-01T10:27:54.799 DEBUG:boto:Token: None 2024-02-01T10:27:54.799 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:54 GMT /vclfcn-19/obj?versionId=K23uwsLWGmQDGYGsdu-h04Fk2wApB0O 2024-02-01T10:27:54.799 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gnvoLqLuFfjp+3MyEF5cnP67GFA= 2024-02-01T10:27:54.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gnvoLqLuFfjp+3MyEF5cnP67GFA=', 'Content-Length': '0'} 2024-02-01T10:27:54.802 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'K23uwsLWGmQDGYGsdu-h04Fk2wApB0O'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000079798c7c50081655-0065bb722a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:54.802 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:54.802 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:54.803 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:54.803 DEBUG:boto:path=/vclfcn-19/obj?versionId=K23uwsLWGmQDGYGsdu-h04Fk2wApB0O 2024-02-01T10:27:54.803 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=K23uwsLWGmQDGYGsdu-h04Fk2wApB0O 2024-02-01T10:27:54.803 DEBUG:boto:Method: GET 2024-02-01T10:27:54.803 DEBUG:boto:Path: /vclfcn-19/obj?versionId=K23uwsLWGmQDGYGsdu-h04Fk2wApB0O 2024-02-01T10:27:54.803 DEBUG:boto:Data: 2024-02-01T10:27:54.803 DEBUG:boto:Headers: {} 2024-02-01T10:27:54.803 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:54.803 DEBUG:boto:Port: 8001 2024-02-01T10:27:54.803 DEBUG:boto:Params: {} 2024-02-01T10:27:54.803 DEBUG:boto:Token: None 2024-02-01T10:27:54.804 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:54 GMT /vclfcn-19/obj?versionId=K23uwsLWGmQDGYGsdu-h04Fk2wApB0O 2024-02-01T10:27:54.804 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YSpbsnefeZp5iQg7P7LQlrgTNE0= 2024-02-01T10:27:54.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YSpbsnefeZp5iQg7P7LQlrgTNE0=', 'Content-Length': '0'} 2024-02-01T10:27:54.806 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'K23uwsLWGmQDGYGsdu-h04Fk2wApB0O'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000064ba868e90c78041-0065bb722a-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:54.848 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:54.848 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:54.848 DEBUG:boto:path=/vclfcn-19/obj?versionId=K23uwsLWGmQDGYGsdu-h04Fk2wApB0O 2024-02-01T10:27:54.848 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=K23uwsLWGmQDGYGsdu-h04Fk2wApB0O 2024-02-01T10:27:54.849 DEBUG:boto:Method: GET 2024-02-01T10:27:54.849 DEBUG:boto:Path: /vclfcn-19/obj?versionId=K23uwsLWGmQDGYGsdu-h04Fk2wApB0O 2024-02-01T10:27:54.849 DEBUG:boto:Data: 2024-02-01T10:27:54.849 DEBUG:boto:Headers: {} 2024-02-01T10:27:54.849 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:54.849 DEBUG:boto:Port: 8000 2024-02-01T10:27:54.849 DEBUG:boto:Params: {} 2024-02-01T10:27:54.849 DEBUG:boto:Token: None 2024-02-01T10:27:54.849 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:54 GMT /vclfcn-19/obj?versionId=K23uwsLWGmQDGYGsdu-h04Fk2wApB0O 2024-02-01T10:27:54.850 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YSpbsnefeZp5iQg7P7LQlrgTNE0= 2024-02-01T10:27:54.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YSpbsnefeZp5iQg7P7LQlrgTNE0=', 'Content-Length': '0'} 2024-02-01T10:27:54.853 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'K23uwsLWGmQDGYGsdu-h04Fk2wApB0O'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b68c8e2b31642383-0065bb722a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:54.896 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:54.896 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:54.896 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:54.897 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:54.897 DEBUG:boto:Method: GET 2024-02-01T10:27:54.897 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:54.897 DEBUG:boto:Data: 2024-02-01T10:27:54.898 DEBUG:boto:Headers: {} 2024-02-01T10:27:54.898 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:54.898 DEBUG:boto:Port: 8001 2024-02-01T10:27:54.898 DEBUG:boto:Params: {} 2024-02-01T10:27:54.898 DEBUG:boto:Token: None 2024-02-01T10:27:54.898 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:54 GMT /vclfcn-19/ 2024-02-01T10:27:54.898 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YtCERNENSXODoDJt6JkMjTFuuKs= 2024-02-01T10:27:54.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YtCERNENSXODoDJt6JkMjTFuuKs=', 'Content-Length': '0'} 2024-02-01T10:27:54.903 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a97adca6dc8919d-0065bb722a-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:54.944 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:54.944 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:54.945 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:54.945 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:54.945 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:54.945 DEBUG:boto:Method: GET 2024-02-01T10:27:54.945 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:54.945 DEBUG:boto:Data: 2024-02-01T10:27:54.945 DEBUG:boto:Headers: {} 2024-02-01T10:27:54.946 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:54.946 DEBUG:boto:Port: 8000 2024-02-01T10:27:54.946 DEBUG:boto:Params: {} 2024-02-01T10:27:54.946 DEBUG:boto:Token: None 2024-02-01T10:27:54.946 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:54 GMT /vclfcn-19/ 2024-02-01T10:27:54.946 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YtCERNENSXODoDJt6JkMjTFuuKs= 2024-02-01T10:27:54.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YtCERNENSXODoDJt6JkMjTFuuKs=', 'Content-Length': '0'} 2024-02-01T10:27:54.952 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f347031ea9234dfd-0065bb722a-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:54.953 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:54.953 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:54.953 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:54.953 DEBUG:boto:Method: HEAD 2024-02-01T10:27:54.953 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:54.953 DEBUG:boto:Data: 2024-02-01T10:27:54.954 DEBUG:boto:Headers: {} 2024-02-01T10:27:54.954 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:54.954 DEBUG:boto:Port: 8001 2024-02-01T10:27:54.954 DEBUG:boto:Params: {} 2024-02-01T10:27:54.954 DEBUG:boto:Token: None 2024-02-01T10:27:54.954 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:54 GMT /vclfcn-19/obj 2024-02-01T10:27:54.954 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:MowLJxrkrKVbO3xkPdZQ9pgR6EQ= 2024-02-01T10:27:54.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:MowLJxrkrKVbO3xkPdZQ9pgR6EQ=', 'Content-Length': '0'} 2024-02-01T10:27:54.961 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009f5e6d49633aa2c2-0065bb722a-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:54.961 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:54.961 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:54.961 DEBUG:boto:Method: HEAD 2024-02-01T10:27:54.961 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:54.961 DEBUG:boto:Data: 2024-02-01T10:27:54.961 DEBUG:boto:Headers: {} 2024-02-01T10:27:54.962 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:54.962 DEBUG:boto:Port: 8000 2024-02-01T10:27:54.962 DEBUG:boto:Params: {} 2024-02-01T10:27:54.962 DEBUG:boto:Token: None 2024-02-01T10:27:54.962 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:54 GMT /vclfcn-19/obj 2024-02-01T10:27:54.962 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:MowLJxrkrKVbO3xkPdZQ9pgR6EQ= 2024-02-01T10:27:54.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:MowLJxrkrKVbO3xkPdZQ9pgR6EQ=', 'Content-Length': '0'} 2024-02-01T10:27:54.968 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009139a9197c354e60-0065bb722a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:54.968 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:54.969 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:54.969 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:54.969 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:54.969 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:54.969 DEBUG:boto:Method: GET 2024-02-01T10:27:54.969 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:54.969 DEBUG:boto:Data: 2024-02-01T10:27:54.969 DEBUG:boto:Headers: {} 2024-02-01T10:27:54.969 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:54.969 DEBUG:boto:Port: 8001 2024-02-01T10:27:54.970 DEBUG:boto:Params: {} 2024-02-01T10:27:54.970 DEBUG:boto:Token: None 2024-02-01T10:27:54.970 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:54 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:54.970 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ahb8BIqVjGk2Z7+iAAtUgLaACxY= 2024-02-01T10:27:54.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ahb8BIqVjGk2Z7+iAAtUgLaACxY=', 'Content-Length': '0'} 2024-02-01T10:27:54.973 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008ff2a23bc529f88a-0065bb722a-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:55.016 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:55.016 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:55.016 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:55.016 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:55.016 DEBUG:boto:Method: GET 2024-02-01T10:27:55.017 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:55.017 DEBUG:boto:Data: 2024-02-01T10:27:55.017 DEBUG:boto:Headers: {} 2024-02-01T10:27:55.017 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:55.017 DEBUG:boto:Port: 8000 2024-02-01T10:27:55.017 DEBUG:boto:Params: {} 2024-02-01T10:27:55.017 DEBUG:boto:Token: None 2024-02-01T10:27:55.017 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:55 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:55.017 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zYtH2v7xaLEZFLfsqScBM5GYpLM= 2024-02-01T10:27:55.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zYtH2v7xaLEZFLfsqScBM5GYpLM=', 'Content-Length': '0'} 2024-02-01T10:27:55.020 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000017b3b212ce118c74-0065bb722b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:55.021 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:55.021 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:55.021 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:55.021 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:55.021 DEBUG:boto:Method: GET 2024-02-01T10:27:55.021 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:55.021 DEBUG:boto:Data: 2024-02-01T10:27:55.021 DEBUG:boto:Headers: {} 2024-02-01T10:27:55.022 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:55.022 DEBUG:boto:Port: 8001 2024-02-01T10:27:55.022 DEBUG:boto:Params: {} 2024-02-01T10:27:55.022 DEBUG:boto:Token: None 2024-02-01T10:27:55.022 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:55 GMT /vclfcn-19/ 2024-02-01T10:27:55.022 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wki8t6MCR5BPImUDGD0NZpVUSJ4= 2024-02-01T10:27:55.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wki8t6MCR5BPImUDGD0NZpVUSJ4=', 'Content-Length': '0'} 2024-02-01T10:27:55.028 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f84746013d70d1bd-0065bb722b-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:55.072 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:55.072 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:55.073 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:55.073 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:55.073 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:55.073 DEBUG:boto:Method: GET 2024-02-01T10:27:55.073 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:55.073 DEBUG:boto:Data: 2024-02-01T10:27:55.073 DEBUG:boto:Headers: {} 2024-02-01T10:27:55.073 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:55.073 DEBUG:boto:Port: 8000 2024-02-01T10:27:55.073 DEBUG:boto:Params: {} 2024-02-01T10:27:55.074 DEBUG:boto:Token: None 2024-02-01T10:27:55.074 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:55 GMT /vclfcn-19/ 2024-02-01T10:27:55.074 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wki8t6MCR5BPImUDGD0NZpVUSJ4= 2024-02-01T10:27:55.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wki8t6MCR5BPImUDGD0NZpVUSJ4=', 'Content-Length': '0'} 2024-02-01T10:27:55.079 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003974a4f6cc22d2a1-0065bb722b-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:55.080 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:55.080 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:55.080 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:55.080 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:55.080 DEBUG:boto:path=/vclfcn-19/obj?versionId=st2TOT.K7IOe-wFfL.hOrRVfLNWblRd 2024-02-01T10:27:55.080 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=st2TOT.K7IOe-wFfL.hOrRVfLNWblRd 2024-02-01T10:27:55.081 DEBUG:boto:Method: GET 2024-02-01T10:27:55.081 DEBUG:boto:Path: /vclfcn-19/obj?versionId=st2TOT.K7IOe-wFfL.hOrRVfLNWblRd 2024-02-01T10:27:55.081 DEBUG:boto:Data: 2024-02-01T10:27:55.081 DEBUG:boto:Headers: {} 2024-02-01T10:27:55.081 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:55.081 DEBUG:boto:Port: 8001 2024-02-01T10:27:55.081 DEBUG:boto:Params: {} 2024-02-01T10:27:55.081 DEBUG:boto:Token: None 2024-02-01T10:27:55.081 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:55 GMT /vclfcn-19/obj?versionId=st2TOT.K7IOe-wFfL.hOrRVfLNWblRd 2024-02-01T10:27:55.082 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ns/Y1Tw2fMUIqj5nixegx40QHvQ= 2024-02-01T10:27:55.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ns/Y1Tw2fMUIqj5nixegx40QHvQ=', 'Content-Length': '0'} 2024-02-01T10:27:55.085 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'st2TOT.K7IOe-wFfL.hOrRVfLNWblRd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000894c34c1af089cce-0065bb722b-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:55.085 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:55.085 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:55.086 DEBUG:boto:path=/vclfcn-19/obj?versionId=st2TOT.K7IOe-wFfL.hOrRVfLNWblRd 2024-02-01T10:27:55.086 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=st2TOT.K7IOe-wFfL.hOrRVfLNWblRd 2024-02-01T10:27:55.086 DEBUG:boto:Method: GET 2024-02-01T10:27:55.086 DEBUG:boto:Path: /vclfcn-19/obj?versionId=st2TOT.K7IOe-wFfL.hOrRVfLNWblRd 2024-02-01T10:27:55.086 DEBUG:boto:Data: 2024-02-01T10:27:55.086 DEBUG:boto:Headers: {} 2024-02-01T10:27:55.086 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:55.086 DEBUG:boto:Port: 8000 2024-02-01T10:27:55.086 DEBUG:boto:Params: {} 2024-02-01T10:27:55.087 DEBUG:boto:Token: None 2024-02-01T10:27:55.087 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:55 GMT /vclfcn-19/obj?versionId=st2TOT.K7IOe-wFfL.hOrRVfLNWblRd 2024-02-01T10:27:55.087 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ns/Y1Tw2fMUIqj5nixegx40QHvQ= 2024-02-01T10:27:55.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ns/Y1Tw2fMUIqj5nixegx40QHvQ=', 'Content-Length': '0'} 2024-02-01T10:27:55.090 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'st2TOT.K7IOe-wFfL.hOrRVfLNWblRd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004cb68c8f96b36543-0065bb722b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:55.090 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:55.090 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:55.090 DEBUG:boto:path=/vclfcn-19/obj?versionId=st2TOT.K7IOe-wFfL.hOrRVfLNWblRd 2024-02-01T10:27:55.090 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=st2TOT.K7IOe-wFfL.hOrRVfLNWblRd 2024-02-01T10:27:55.090 DEBUG:boto:Method: HEAD 2024-02-01T10:27:55.091 DEBUG:boto:Path: /vclfcn-19/obj?versionId=st2TOT.K7IOe-wFfL.hOrRVfLNWblRd 2024-02-01T10:27:55.091 DEBUG:boto:Data: 2024-02-01T10:27:55.091 DEBUG:boto:Headers: {} 2024-02-01T10:27:55.091 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:55.091 DEBUG:boto:Port: 8001 2024-02-01T10:27:55.091 DEBUG:boto:Params: {} 2024-02-01T10:27:55.091 DEBUG:boto:Token: None 2024-02-01T10:27:55.091 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:55 GMT /vclfcn-19/obj?versionId=st2TOT.K7IOe-wFfL.hOrRVfLNWblRd 2024-02-01T10:27:55.091 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:SJpyvFAvVFMMgiMmxFiA3dv7lxU= 2024-02-01T10:27:55.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:SJpyvFAvVFMMgiMmxFiA3dv7lxU=', 'Content-Length': '0'} 2024-02-01T10:27:55.094 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'st2TOT.K7IOe-wFfL.hOrRVfLNWblRd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000030478b3610eb946d-0065bb722b-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:55.094 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:55.095 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:55.095 DEBUG:boto:path=/vclfcn-19/obj?versionId=st2TOT.K7IOe-wFfL.hOrRVfLNWblRd 2024-02-01T10:27:55.095 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=st2TOT.K7IOe-wFfL.hOrRVfLNWblRd 2024-02-01T10:27:55.095 DEBUG:boto:Method: HEAD 2024-02-01T10:27:55.095 DEBUG:boto:Path: /vclfcn-19/obj?versionId=st2TOT.K7IOe-wFfL.hOrRVfLNWblRd 2024-02-01T10:27:55.095 DEBUG:boto:Data: 2024-02-01T10:27:55.095 DEBUG:boto:Headers: {} 2024-02-01T10:27:55.095 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:55.095 DEBUG:boto:Port: 8000 2024-02-01T10:27:55.095 DEBUG:boto:Params: {} 2024-02-01T10:27:55.096 DEBUG:boto:Token: None 2024-02-01T10:27:55.096 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:55 GMT /vclfcn-19/obj?versionId=st2TOT.K7IOe-wFfL.hOrRVfLNWblRd 2024-02-01T10:27:55.096 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:SJpyvFAvVFMMgiMmxFiA3dv7lxU= 2024-02-01T10:27:55.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:SJpyvFAvVFMMgiMmxFiA3dv7lxU=', 'Content-Length': '0'} 2024-02-01T10:27:55.099 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'st2TOT.K7IOe-wFfL.hOrRVfLNWblRd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000013b137d260cb3032-0065bb722b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:55.099 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:55.099 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:55.099 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:55.099 DEBUG:boto:path=/vclfcn-19/obj?versionId=st2TOT.K7IOe-wFfL.hOrRVfLNWblRd 2024-02-01T10:27:55.100 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=st2TOT.K7IOe-wFfL.hOrRVfLNWblRd 2024-02-01T10:27:55.100 DEBUG:boto:Method: GET 2024-02-01T10:27:55.100 DEBUG:boto:Path: /vclfcn-19/obj?versionId=st2TOT.K7IOe-wFfL.hOrRVfLNWblRd 2024-02-01T10:27:55.100 DEBUG:boto:Data: 2024-02-01T10:27:55.100 DEBUG:boto:Headers: {} 2024-02-01T10:27:55.100 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:55.100 DEBUG:boto:Port: 8001 2024-02-01T10:27:55.100 DEBUG:boto:Params: {} 2024-02-01T10:27:55.100 DEBUG:boto:Token: None 2024-02-01T10:27:55.101 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:55 GMT /vclfcn-19/obj?versionId=st2TOT.K7IOe-wFfL.hOrRVfLNWblRd 2024-02-01T10:27:55.101 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ns/Y1Tw2fMUIqj5nixegx40QHvQ= 2024-02-01T10:27:55.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ns/Y1Tw2fMUIqj5nixegx40QHvQ=', 'Content-Length': '0'} 2024-02-01T10:27:55.104 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'st2TOT.K7IOe-wFfL.hOrRVfLNWblRd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003769df8031fa18c8-0065bb722b-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:55.104 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:55.105 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:55.105 DEBUG:boto:path=/vclfcn-19/obj?versionId=st2TOT.K7IOe-wFfL.hOrRVfLNWblRd 2024-02-01T10:27:55.105 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=st2TOT.K7IOe-wFfL.hOrRVfLNWblRd 2024-02-01T10:27:55.105 DEBUG:boto:Method: GET 2024-02-01T10:27:55.106 DEBUG:boto:Path: /vclfcn-19/obj?versionId=st2TOT.K7IOe-wFfL.hOrRVfLNWblRd 2024-02-01T10:27:55.106 DEBUG:boto:Data: 2024-02-01T10:27:55.106 DEBUG:boto:Headers: {} 2024-02-01T10:27:55.106 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:55.106 DEBUG:boto:Port: 8000 2024-02-01T10:27:55.107 DEBUG:boto:Params: {} 2024-02-01T10:27:55.107 DEBUG:boto:Token: None 2024-02-01T10:27:55.107 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:55 GMT /vclfcn-19/obj?versionId=st2TOT.K7IOe-wFfL.hOrRVfLNWblRd 2024-02-01T10:27:55.107 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ns/Y1Tw2fMUIqj5nixegx40QHvQ= 2024-02-01T10:27:55.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ns/Y1Tw2fMUIqj5nixegx40QHvQ=', 'Content-Length': '0'} 2024-02-01T10:27:55.110 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'st2TOT.K7IOe-wFfL.hOrRVfLNWblRd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004754143c24a9dd1a-0065bb722b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:55.152 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:55.152 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:55.153 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:55.153 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:55.153 DEBUG:boto:Method: GET 2024-02-01T10:27:55.154 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:55.154 DEBUG:boto:Data: 2024-02-01T10:27:55.154 DEBUG:boto:Headers: {} 2024-02-01T10:27:55.154 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:55.154 DEBUG:boto:Port: 8001 2024-02-01T10:27:55.154 DEBUG:boto:Params: {} 2024-02-01T10:27:55.154 DEBUG:boto:Token: None 2024-02-01T10:27:55.154 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:55 GMT /vclfcn-19/ 2024-02-01T10:27:55.155 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wki8t6MCR5BPImUDGD0NZpVUSJ4= 2024-02-01T10:27:55.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wki8t6MCR5BPImUDGD0NZpVUSJ4=', 'Content-Length': '0'} 2024-02-01T10:27:55.161 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f65fb24e2feb850b-0065bb722b-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:55.161 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:55.162 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:55.162 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:55.162 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:55.162 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:55.162 DEBUG:boto:Method: GET 2024-02-01T10:27:55.162 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:55.162 DEBUG:boto:Data: 2024-02-01T10:27:55.162 DEBUG:boto:Headers: {} 2024-02-01T10:27:55.162 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:55.162 DEBUG:boto:Port: 8000 2024-02-01T10:27:55.162 DEBUG:boto:Params: {} 2024-02-01T10:27:55.163 DEBUG:boto:Token: None 2024-02-01T10:27:55.163 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:55 GMT /vclfcn-19/ 2024-02-01T10:27:55.163 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wki8t6MCR5BPImUDGD0NZpVUSJ4= 2024-02-01T10:27:55.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wki8t6MCR5BPImUDGD0NZpVUSJ4=', 'Content-Length': '0'} 2024-02-01T10:27:55.168 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b912331ecf6e6efc-0065bb722b-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:55.168 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:55.169 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:55.169 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:55.169 DEBUG:boto:Method: HEAD 2024-02-01T10:27:55.169 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:55.169 DEBUG:boto:Data: 2024-02-01T10:27:55.169 DEBUG:boto:Headers: {} 2024-02-01T10:27:55.169 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:55.169 DEBUG:boto:Port: 8001 2024-02-01T10:27:55.169 DEBUG:boto:Params: {} 2024-02-01T10:27:55.170 DEBUG:boto:Token: None 2024-02-01T10:27:55.170 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:55 GMT /vclfcn-19/obj 2024-02-01T10:27:55.170 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Cyn/5a0gZPBWiGnj7AxW/A7F5hs= 2024-02-01T10:27:55.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Cyn/5a0gZPBWiGnj7AxW/A7F5hs=', 'Content-Length': '0'} 2024-02-01T10:27:55.176 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fd0f2d0a6a733a3e-0065bb722b-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:55.176 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:55.176 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:55.177 DEBUG:boto:Method: HEAD 2024-02-01T10:27:55.177 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:55.177 DEBUG:boto:Data: 2024-02-01T10:27:55.177 DEBUG:boto:Headers: {} 2024-02-01T10:27:55.177 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:55.177 DEBUG:boto:Port: 8000 2024-02-01T10:27:55.177 DEBUG:boto:Params: {} 2024-02-01T10:27:55.177 DEBUG:boto:Token: None 2024-02-01T10:27:55.177 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:55 GMT /vclfcn-19/obj 2024-02-01T10:27:55.177 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Cyn/5a0gZPBWiGnj7AxW/A7F5hs= 2024-02-01T10:27:55.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Cyn/5a0gZPBWiGnj7AxW/A7F5hs=', 'Content-Length': '0'} 2024-02-01T10:27:55.183 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e79356d0560747cb-0065bb722b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:55.183 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:55.184 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:55.184 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:55.184 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:55.184 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:55.184 DEBUG:boto:Method: GET 2024-02-01T10:27:55.184 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:55.184 DEBUG:boto:Data: 2024-02-01T10:27:55.184 DEBUG:boto:Headers: {} 2024-02-01T10:27:55.184 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:55.185 DEBUG:boto:Port: 8001 2024-02-01T10:27:55.185 DEBUG:boto:Params: {} 2024-02-01T10:27:55.185 DEBUG:boto:Token: None 2024-02-01T10:27:55.185 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:55 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:55.185 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zYtH2v7xaLEZFLfsqScBM5GYpLM= 2024-02-01T10:27:55.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zYtH2v7xaLEZFLfsqScBM5GYpLM=', 'Content-Length': '0'} 2024-02-01T10:27:55.188 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001cd6cfc8426c08b0-0065bb722b-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:55.236 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:55.236 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:55.236 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:55.237 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:55.237 DEBUG:boto:Method: GET 2024-02-01T10:27:55.237 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:55.237 DEBUG:boto:Data: 2024-02-01T10:27:55.237 DEBUG:boto:Headers: {} 2024-02-01T10:27:55.237 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:55.237 DEBUG:boto:Port: 8000 2024-02-01T10:27:55.237 DEBUG:boto:Params: {} 2024-02-01T10:27:55.237 DEBUG:boto:Token: None 2024-02-01T10:27:55.238 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:55 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:55.238 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zYtH2v7xaLEZFLfsqScBM5GYpLM= 2024-02-01T10:27:55.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zYtH2v7xaLEZFLfsqScBM5GYpLM=', 'Content-Length': '0'} 2024-02-01T10:27:55.241 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001c88ed9fd12def2b-0065bb722b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:55.241 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:55.241 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:55.241 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:55.242 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:55.242 DEBUG:boto:Method: GET 2024-02-01T10:27:55.242 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:55.242 DEBUG:boto:Data: 2024-02-01T10:27:55.242 DEBUG:boto:Headers: {} 2024-02-01T10:27:55.242 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:55.242 DEBUG:boto:Port: 8001 2024-02-01T10:27:55.242 DEBUG:boto:Params: {} 2024-02-01T10:27:55.242 DEBUG:boto:Token: None 2024-02-01T10:27:55.242 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:55 GMT /vclfcn-19/ 2024-02-01T10:27:55.243 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wki8t6MCR5BPImUDGD0NZpVUSJ4= 2024-02-01T10:27:55.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wki8t6MCR5BPImUDGD0NZpVUSJ4=', 'Content-Length': '0'} 2024-02-01T10:27:55.248 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b37c72d390be30ee-0065bb722b-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:55.248 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:55.248 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:55.248 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:55.249 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:55.249 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:55.249 DEBUG:boto:Method: GET 2024-02-01T10:27:55.249 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:55.249 DEBUG:boto:Data: 2024-02-01T10:27:55.249 DEBUG:boto:Headers: {} 2024-02-01T10:27:55.249 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:55.249 DEBUG:boto:Port: 8000 2024-02-01T10:27:55.249 DEBUG:boto:Params: {} 2024-02-01T10:27:55.249 DEBUG:boto:Token: None 2024-02-01T10:27:55.249 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:55 GMT /vclfcn-19/ 2024-02-01T10:27:55.250 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wki8t6MCR5BPImUDGD0NZpVUSJ4= 2024-02-01T10:27:55.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wki8t6MCR5BPImUDGD0NZpVUSJ4=', 'Content-Length': '0'} 2024-02-01T10:27:55.255 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3289a5790ba28c0-0065bb722b-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:55.296 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:55.296 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:55.297 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:55.297 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:55.297 DEBUG:boto:path=/vclfcn-19/obj?versionId=9CJ3w1EusK8OST2k17sTdTM.cuh3E3e 2024-02-01T10:27:55.297 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=9CJ3w1EusK8OST2k17sTdTM.cuh3E3e 2024-02-01T10:27:55.297 DEBUG:boto:Method: GET 2024-02-01T10:27:55.297 DEBUG:boto:Path: /vclfcn-19/obj?versionId=9CJ3w1EusK8OST2k17sTdTM.cuh3E3e 2024-02-01T10:27:55.297 DEBUG:boto:Data: 2024-02-01T10:27:55.297 DEBUG:boto:Headers: {} 2024-02-01T10:27:55.297 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:55.297 DEBUG:boto:Port: 8001 2024-02-01T10:27:55.298 DEBUG:boto:Params: {} 2024-02-01T10:27:55.298 DEBUG:boto:Token: None 2024-02-01T10:27:55.298 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:55 GMT /vclfcn-19/obj?versionId=9CJ3w1EusK8OST2k17sTdTM.cuh3E3e 2024-02-01T10:27:55.298 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:4go+vyySYGswET3ym1Kyh2sU068= 2024-02-01T10:27:55.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:4go+vyySYGswET3ym1Kyh2sU068=', 'Content-Length': '0'} 2024-02-01T10:27:55.301 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '9CJ3w1EusK8OST2k17sTdTM.cuh3E3e'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e688f54f39221470-0065bb722b-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:55.301 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:55.301 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:55.302 DEBUG:boto:path=/vclfcn-19/obj?versionId=9CJ3w1EusK8OST2k17sTdTM.cuh3E3e 2024-02-01T10:27:55.302 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=9CJ3w1EusK8OST2k17sTdTM.cuh3E3e 2024-02-01T10:27:55.302 DEBUG:boto:Method: GET 2024-02-01T10:27:55.302 DEBUG:boto:Path: /vclfcn-19/obj?versionId=9CJ3w1EusK8OST2k17sTdTM.cuh3E3e 2024-02-01T10:27:55.302 DEBUG:boto:Data: 2024-02-01T10:27:55.302 DEBUG:boto:Headers: {} 2024-02-01T10:27:55.302 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:55.302 DEBUG:boto:Port: 8000 2024-02-01T10:27:55.303 DEBUG:boto:Params: {} 2024-02-01T10:27:55.303 DEBUG:boto:Token: None 2024-02-01T10:27:55.303 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:55 GMT /vclfcn-19/obj?versionId=9CJ3w1EusK8OST2k17sTdTM.cuh3E3e 2024-02-01T10:27:55.303 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:4go+vyySYGswET3ym1Kyh2sU068= 2024-02-01T10:27:55.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:4go+vyySYGswET3ym1Kyh2sU068=', 'Content-Length': '0'} 2024-02-01T10:27:55.306 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '9CJ3w1EusK8OST2k17sTdTM.cuh3E3e'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001eb8a7015ed66c18-0065bb722b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:55.306 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:55.307 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:55.307 DEBUG:boto:path=/vclfcn-19/obj?versionId=9CJ3w1EusK8OST2k17sTdTM.cuh3E3e 2024-02-01T10:27:55.307 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=9CJ3w1EusK8OST2k17sTdTM.cuh3E3e 2024-02-01T10:27:55.307 DEBUG:boto:Method: HEAD 2024-02-01T10:27:55.308 DEBUG:boto:Path: /vclfcn-19/obj?versionId=9CJ3w1EusK8OST2k17sTdTM.cuh3E3e 2024-02-01T10:27:55.308 DEBUG:boto:Data: 2024-02-01T10:27:55.308 DEBUG:boto:Headers: {} 2024-02-01T10:27:55.308 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:55.308 DEBUG:boto:Port: 8001 2024-02-01T10:27:55.308 DEBUG:boto:Params: {} 2024-02-01T10:27:55.308 DEBUG:boto:Token: None 2024-02-01T10:27:55.308 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:55 GMT /vclfcn-19/obj?versionId=9CJ3w1EusK8OST2k17sTdTM.cuh3E3e 2024-02-01T10:27:55.308 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8Iq/jkcYYcuPaSKizmtdjaz38R8= 2024-02-01T10:27:55.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8Iq/jkcYYcuPaSKizmtdjaz38R8=', 'Content-Length': '0'} 2024-02-01T10:27:55.311 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '9CJ3w1EusK8OST2k17sTdTM.cuh3E3e'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009b5efc07846f1b94-0065bb722b-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:55.312 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:55.312 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:55.312 DEBUG:boto:path=/vclfcn-19/obj?versionId=9CJ3w1EusK8OST2k17sTdTM.cuh3E3e 2024-02-01T10:27:55.312 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=9CJ3w1EusK8OST2k17sTdTM.cuh3E3e 2024-02-01T10:27:55.312 DEBUG:boto:Method: HEAD 2024-02-01T10:27:55.312 DEBUG:boto:Path: /vclfcn-19/obj?versionId=9CJ3w1EusK8OST2k17sTdTM.cuh3E3e 2024-02-01T10:27:55.312 DEBUG:boto:Data: 2024-02-01T10:27:55.312 DEBUG:boto:Headers: {} 2024-02-01T10:27:55.312 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:55.312 DEBUG:boto:Port: 8000 2024-02-01T10:27:55.313 DEBUG:boto:Params: {} 2024-02-01T10:27:55.313 DEBUG:boto:Token: None 2024-02-01T10:27:55.313 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:55 GMT /vclfcn-19/obj?versionId=9CJ3w1EusK8OST2k17sTdTM.cuh3E3e 2024-02-01T10:27:55.313 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8Iq/jkcYYcuPaSKizmtdjaz38R8= 2024-02-01T10:27:55.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8Iq/jkcYYcuPaSKizmtdjaz38R8=', 'Content-Length': '0'} 2024-02-01T10:27:55.315 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '9CJ3w1EusK8OST2k17sTdTM.cuh3E3e'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ac120f14faf2d8c2-0065bb722b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:55.316 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:55.316 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:55.316 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:55.316 DEBUG:boto:path=/vclfcn-19/obj?versionId=9CJ3w1EusK8OST2k17sTdTM.cuh3E3e 2024-02-01T10:27:55.316 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=9CJ3w1EusK8OST2k17sTdTM.cuh3E3e 2024-02-01T10:27:55.316 DEBUG:boto:Method: GET 2024-02-01T10:27:55.316 DEBUG:boto:Path: /vclfcn-19/obj?versionId=9CJ3w1EusK8OST2k17sTdTM.cuh3E3e 2024-02-01T10:27:55.316 DEBUG:boto:Data: 2024-02-01T10:27:55.317 DEBUG:boto:Headers: {} 2024-02-01T10:27:55.317 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:55.317 DEBUG:boto:Port: 8001 2024-02-01T10:27:55.317 DEBUG:boto:Params: {} 2024-02-01T10:27:55.317 DEBUG:boto:Token: None 2024-02-01T10:27:55.317 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:55 GMT /vclfcn-19/obj?versionId=9CJ3w1EusK8OST2k17sTdTM.cuh3E3e 2024-02-01T10:27:55.317 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:4go+vyySYGswET3ym1Kyh2sU068= 2024-02-01T10:27:55.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:4go+vyySYGswET3ym1Kyh2sU068=', 'Content-Length': '0'} 2024-02-01T10:27:55.320 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '9CJ3w1EusK8OST2k17sTdTM.cuh3E3e'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005755a47591eb0aa4-0065bb722b-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:55.368 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:55.368 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:55.368 DEBUG:boto:path=/vclfcn-19/obj?versionId=9CJ3w1EusK8OST2k17sTdTM.cuh3E3e 2024-02-01T10:27:55.368 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=9CJ3w1EusK8OST2k17sTdTM.cuh3E3e 2024-02-01T10:27:55.369 DEBUG:boto:Method: GET 2024-02-01T10:27:55.369 DEBUG:boto:Path: /vclfcn-19/obj?versionId=9CJ3w1EusK8OST2k17sTdTM.cuh3E3e 2024-02-01T10:27:55.369 DEBUG:boto:Data: 2024-02-01T10:27:55.369 DEBUG:boto:Headers: {} 2024-02-01T10:27:55.369 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:55.369 DEBUG:boto:Port: 8000 2024-02-01T10:27:55.369 DEBUG:boto:Params: {} 2024-02-01T10:27:55.369 DEBUG:boto:Token: None 2024-02-01T10:27:55.369 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:55 GMT /vclfcn-19/obj?versionId=9CJ3w1EusK8OST2k17sTdTM.cuh3E3e 2024-02-01T10:27:55.370 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:4go+vyySYGswET3ym1Kyh2sU068= 2024-02-01T10:27:55.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:4go+vyySYGswET3ym1Kyh2sU068=', 'Content-Length': '0'} 2024-02-01T10:27:55.372 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '9CJ3w1EusK8OST2k17sTdTM.cuh3E3e'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000604d2f9655ab2e61-0065bb722b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:55.373 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:55.373 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:55.373 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:55.373 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:55.373 DEBUG:boto:Method: GET 2024-02-01T10:27:55.373 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:55.373 DEBUG:boto:Data: 2024-02-01T10:27:55.373 DEBUG:boto:Headers: {} 2024-02-01T10:27:55.373 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:55.373 DEBUG:boto:Port: 8001 2024-02-01T10:27:55.374 DEBUG:boto:Params: {} 2024-02-01T10:27:55.374 DEBUG:boto:Token: None 2024-02-01T10:27:55.374 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:55 GMT /vclfcn-19/ 2024-02-01T10:27:55.374 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wki8t6MCR5BPImUDGD0NZpVUSJ4= 2024-02-01T10:27:55.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wki8t6MCR5BPImUDGD0NZpVUSJ4=', 'Content-Length': '0'} 2024-02-01T10:27:55.379 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be2fb369629ba944-0065bb722b-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:55.420 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:55.420 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:55.421 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:55.421 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:55.421 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:55.421 DEBUG:boto:Method: GET 2024-02-01T10:27:55.421 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:55.421 DEBUG:boto:Data: 2024-02-01T10:27:55.421 DEBUG:boto:Headers: {} 2024-02-01T10:27:55.421 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:55.421 DEBUG:boto:Port: 8000 2024-02-01T10:27:55.421 DEBUG:boto:Params: {} 2024-02-01T10:27:55.422 DEBUG:boto:Token: None 2024-02-01T10:27:55.422 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:55 GMT /vclfcn-19/ 2024-02-01T10:27:55.422 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wki8t6MCR5BPImUDGD0NZpVUSJ4= 2024-02-01T10:27:55.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wki8t6MCR5BPImUDGD0NZpVUSJ4=', 'Content-Length': '0'} 2024-02-01T10:27:55.427 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006770177bc26cae8a-0065bb722b-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:55.427 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:55.428 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:55.428 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:55.428 DEBUG:boto:Method: HEAD 2024-02-01T10:27:55.428 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:55.428 DEBUG:boto:Data: 2024-02-01T10:27:55.428 DEBUG:boto:Headers: {} 2024-02-01T10:27:55.428 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:55.428 DEBUG:boto:Port: 8001 2024-02-01T10:27:55.428 DEBUG:boto:Params: {} 2024-02-01T10:27:55.429 DEBUG:boto:Token: None 2024-02-01T10:27:55.429 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:55 GMT /vclfcn-19/obj 2024-02-01T10:27:55.429 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Cyn/5a0gZPBWiGnj7AxW/A7F5hs= 2024-02-01T10:27:55.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Cyn/5a0gZPBWiGnj7AxW/A7F5hs=', 'Content-Length': '0'} 2024-02-01T10:27:55.434 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e5fbfc93ed1a7cd2-0065bb722b-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:55.435 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:55.435 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:55.435 DEBUG:boto:Method: HEAD 2024-02-01T10:27:55.435 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:55.435 DEBUG:boto:Data: 2024-02-01T10:27:55.435 DEBUG:boto:Headers: {} 2024-02-01T10:27:55.435 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:55.435 DEBUG:boto:Port: 8000 2024-02-01T10:27:55.435 DEBUG:boto:Params: {} 2024-02-01T10:27:55.436 DEBUG:boto:Token: None 2024-02-01T10:27:55.436 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:55 GMT /vclfcn-19/obj 2024-02-01T10:27:55.436 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Cyn/5a0gZPBWiGnj7AxW/A7F5hs= 2024-02-01T10:27:55.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Cyn/5a0gZPBWiGnj7AxW/A7F5hs=', 'Content-Length': '0'} 2024-02-01T10:27:55.441 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000de9e0fafbbbb0db6-0065bb722b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:55.441 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:55.442 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:55.442 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:55.442 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:55.442 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:55.442 DEBUG:boto:Method: GET 2024-02-01T10:27:55.442 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:55.442 DEBUG:boto:Data: 2024-02-01T10:27:55.442 DEBUG:boto:Headers: {} 2024-02-01T10:27:55.442 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:55.442 DEBUG:boto:Port: 8001 2024-02-01T10:27:55.442 DEBUG:boto:Params: {} 2024-02-01T10:27:55.443 DEBUG:boto:Token: None 2024-02-01T10:27:55.443 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:55 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:55.443 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zYtH2v7xaLEZFLfsqScBM5GYpLM= 2024-02-01T10:27:55.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zYtH2v7xaLEZFLfsqScBM5GYpLM=', 'Content-Length': '0'} 2024-02-01T10:27:55.446 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cb5072b8eff93583-0065bb722b-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:55.492 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:55.492 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:55.492 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:55.492 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:55.492 DEBUG:boto:Method: GET 2024-02-01T10:27:55.492 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:55.493 DEBUG:boto:Data: 2024-02-01T10:27:55.493 DEBUG:boto:Headers: {} 2024-02-01T10:27:55.493 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:55.493 DEBUG:boto:Port: 8000 2024-02-01T10:27:55.493 DEBUG:boto:Params: {} 2024-02-01T10:27:55.493 DEBUG:boto:Token: None 2024-02-01T10:27:55.493 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:55 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:55.493 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zYtH2v7xaLEZFLfsqScBM5GYpLM= 2024-02-01T10:27:55.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zYtH2v7xaLEZFLfsqScBM5GYpLM=', 'Content-Length': '0'} 2024-02-01T10:27:55.496 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a712283e95da5620-0065bb722b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:55.496 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:55.497 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:55.497 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:55.497 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:55.497 DEBUG:boto:Method: GET 2024-02-01T10:27:55.497 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:55.497 DEBUG:boto:Data: 2024-02-01T10:27:55.497 DEBUG:boto:Headers: {} 2024-02-01T10:27:55.497 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:55.497 DEBUG:boto:Port: 8001 2024-02-01T10:27:55.497 DEBUG:boto:Params: {} 2024-02-01T10:27:55.497 DEBUG:boto:Token: None 2024-02-01T10:27:55.498 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:55 GMT /vclfcn-19/ 2024-02-01T10:27:55.498 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wki8t6MCR5BPImUDGD0NZpVUSJ4= 2024-02-01T10:27:55.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wki8t6MCR5BPImUDGD0NZpVUSJ4=', 'Content-Length': '0'} 2024-02-01T10:27:55.502 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e657a2a16db2bbe-0065bb722b-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:55.544 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:55.545 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:55.545 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:55.545 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:55.545 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:55.545 DEBUG:boto:Method: GET 2024-02-01T10:27:55.545 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:55.545 DEBUG:boto:Data: 2024-02-01T10:27:55.545 DEBUG:boto:Headers: {} 2024-02-01T10:27:55.545 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:55.545 DEBUG:boto:Port: 8000 2024-02-01T10:27:55.545 DEBUG:boto:Params: {} 2024-02-01T10:27:55.546 DEBUG:boto:Token: None 2024-02-01T10:27:55.546 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:55 GMT /vclfcn-19/ 2024-02-01T10:27:55.546 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wki8t6MCR5BPImUDGD0NZpVUSJ4= 2024-02-01T10:27:55.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wki8t6MCR5BPImUDGD0NZpVUSJ4=', 'Content-Length': '0'} 2024-02-01T10:27:55.552 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b893f44690c77a65-0065bb722b-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:55.552 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:55.553 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:55.553 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:55.553 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:55.553 DEBUG:boto:path=/vclfcn-19/obj?versionId=Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv 2024-02-01T10:27:55.553 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv 2024-02-01T10:27:55.557 DEBUG:boto:Method: GET 2024-02-01T10:27:55.557 DEBUG:boto:Path: /vclfcn-19/obj?versionId=Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv 2024-02-01T10:27:55.557 DEBUG:boto:Data: 2024-02-01T10:27:55.557 DEBUG:boto:Headers: {} 2024-02-01T10:27:55.557 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:55.557 DEBUG:boto:Port: 8001 2024-02-01T10:27:55.557 DEBUG:boto:Params: {} 2024-02-01T10:27:55.558 DEBUG:boto:Token: None 2024-02-01T10:27:55.558 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:55 GMT /vclfcn-19/obj?versionId=Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv 2024-02-01T10:27:55.558 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QZ+FO5KLy1mht/JTMZP22PYiBZ0= 2024-02-01T10:27:55.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QZ+FO5KLy1mht/JTMZP22PYiBZ0=', 'Content-Length': '0'} 2024-02-01T10:27:55.561 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e4bef7c9d8037983-0065bb722b-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:55.561 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:55.561 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:55.562 DEBUG:boto:path=/vclfcn-19/obj?versionId=Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv 2024-02-01T10:27:55.562 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv 2024-02-01T10:27:55.562 DEBUG:boto:Method: GET 2024-02-01T10:27:55.562 DEBUG:boto:Path: /vclfcn-19/obj?versionId=Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv 2024-02-01T10:27:55.562 DEBUG:boto:Data: 2024-02-01T10:27:55.562 DEBUG:boto:Headers: {} 2024-02-01T10:27:55.562 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:55.562 DEBUG:boto:Port: 8000 2024-02-01T10:27:55.562 DEBUG:boto:Params: {} 2024-02-01T10:27:55.562 DEBUG:boto:Token: None 2024-02-01T10:27:55.563 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:55 GMT /vclfcn-19/obj?versionId=Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv 2024-02-01T10:27:55.563 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QZ+FO5KLy1mht/JTMZP22PYiBZ0= 2024-02-01T10:27:55.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QZ+FO5KLy1mht/JTMZP22PYiBZ0=', 'Content-Length': '0'} 2024-02-01T10:27:55.565 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000091b493d385da56a2-0065bb722b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:55.608 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:55.608 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:55.608 DEBUG:boto:path=/vclfcn-19/obj?versionId=Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv 2024-02-01T10:27:55.608 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv 2024-02-01T10:27:55.609 DEBUG:boto:Method: HEAD 2024-02-01T10:27:55.609 DEBUG:boto:Path: /vclfcn-19/obj?versionId=Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv 2024-02-01T10:27:55.609 DEBUG:boto:Data: 2024-02-01T10:27:55.609 DEBUG:boto:Headers: {} 2024-02-01T10:27:55.609 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:55.609 DEBUG:boto:Port: 8001 2024-02-01T10:27:55.609 DEBUG:boto:Params: {} 2024-02-01T10:27:55.609 DEBUG:boto:Token: None 2024-02-01T10:27:55.609 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:55 GMT /vclfcn-19/obj?versionId=Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv 2024-02-01T10:27:55.609 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:c5YW+44IwEnkjYXa4W8yWLwgGj0= 2024-02-01T10:27:55.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:c5YW+44IwEnkjYXa4W8yWLwgGj0=', 'Content-Length': '0'} 2024-02-01T10:27:55.613 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ad54f67f94b53e56-0065bb722b-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:55.613 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:55.613 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:55.613 DEBUG:boto:path=/vclfcn-19/obj?versionId=Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv 2024-02-01T10:27:55.614 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv 2024-02-01T10:27:55.614 DEBUG:boto:Method: HEAD 2024-02-01T10:27:55.614 DEBUG:boto:Path: /vclfcn-19/obj?versionId=Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv 2024-02-01T10:27:55.614 DEBUG:boto:Data: 2024-02-01T10:27:55.614 DEBUG:boto:Headers: {} 2024-02-01T10:27:55.614 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:55.614 DEBUG:boto:Port: 8000 2024-02-01T10:27:55.614 DEBUG:boto:Params: {} 2024-02-01T10:27:55.614 DEBUG:boto:Token: None 2024-02-01T10:27:55.614 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:55 GMT /vclfcn-19/obj?versionId=Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv 2024-02-01T10:27:55.615 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:c5YW+44IwEnkjYXa4W8yWLwgGj0= 2024-02-01T10:27:55.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:c5YW+44IwEnkjYXa4W8yWLwgGj0=', 'Content-Length': '0'} 2024-02-01T10:27:55.617 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000294bce2b3a6a4715-0065bb722b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:55.618 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:55.618 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:55.618 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:55.618 DEBUG:boto:path=/vclfcn-19/obj?versionId=Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv 2024-02-01T10:27:55.618 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv 2024-02-01T10:27:55.619 DEBUG:boto:Method: GET 2024-02-01T10:27:55.619 DEBUG:boto:Path: /vclfcn-19/obj?versionId=Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv 2024-02-01T10:27:55.619 DEBUG:boto:Data: 2024-02-01T10:27:55.619 DEBUG:boto:Headers: {} 2024-02-01T10:27:55.619 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:55.619 DEBUG:boto:Port: 8001 2024-02-01T10:27:55.619 DEBUG:boto:Params: {} 2024-02-01T10:27:55.619 DEBUG:boto:Token: None 2024-02-01T10:27:55.619 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:55 GMT /vclfcn-19/obj?versionId=Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv 2024-02-01T10:27:55.619 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QZ+FO5KLy1mht/JTMZP22PYiBZ0= 2024-02-01T10:27:55.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QZ+FO5KLy1mht/JTMZP22PYiBZ0=', 'Content-Length': '0'} 2024-02-01T10:27:55.622 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002e5b0d657e2d82af-0065bb722b-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:55.623 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:55.623 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:55.623 DEBUG:boto:path=/vclfcn-19/obj?versionId=Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv 2024-02-01T10:27:55.623 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv 2024-02-01T10:27:55.623 DEBUG:boto:Method: GET 2024-02-01T10:27:55.623 DEBUG:boto:Path: /vclfcn-19/obj?versionId=Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv 2024-02-01T10:27:55.623 DEBUG:boto:Data: 2024-02-01T10:27:55.623 DEBUG:boto:Headers: {} 2024-02-01T10:27:55.624 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:55.624 DEBUG:boto:Port: 8000 2024-02-01T10:27:55.624 DEBUG:boto:Params: {} 2024-02-01T10:27:55.624 DEBUG:boto:Token: None 2024-02-01T10:27:55.624 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:55 GMT /vclfcn-19/obj?versionId=Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv 2024-02-01T10:27:55.624 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QZ+FO5KLy1mht/JTMZP22PYiBZ0= 2024-02-01T10:27:55.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QZ+FO5KLy1mht/JTMZP22PYiBZ0=', 'Content-Length': '0'} 2024-02-01T10:27:55.627 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'Vwfx4ETdAtTkKn2rTCQT4onqB6sYulv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007a3ff29e8fa502f4-0065bb722b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:55.672 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:55.672 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:55.672 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:55.672 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:55.673 DEBUG:boto:Method: GET 2024-02-01T10:27:55.673 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:55.673 DEBUG:boto:Data: 2024-02-01T10:27:55.673 DEBUG:boto:Headers: {} 2024-02-01T10:27:55.673 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:55.673 DEBUG:boto:Port: 8001 2024-02-01T10:27:55.673 DEBUG:boto:Params: {} 2024-02-01T10:27:55.673 DEBUG:boto:Token: None 2024-02-01T10:27:55.673 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:55 GMT /vclfcn-19/ 2024-02-01T10:27:55.674 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wki8t6MCR5BPImUDGD0NZpVUSJ4= 2024-02-01T10:27:55.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wki8t6MCR5BPImUDGD0NZpVUSJ4=', 'Content-Length': '0'} 2024-02-01T10:27:55.680 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e995b4c720d3cb5-0065bb722b-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:55.680 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:55.680 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:55.680 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:55.681 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:55.681 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:55.681 DEBUG:boto:Method: GET 2024-02-01T10:27:55.681 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:55.681 DEBUG:boto:Data: 2024-02-01T10:27:55.681 DEBUG:boto:Headers: {} 2024-02-01T10:27:55.681 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:55.681 DEBUG:boto:Port: 8000 2024-02-01T10:27:55.681 DEBUG:boto:Params: {} 2024-02-01T10:27:55.681 DEBUG:boto:Token: None 2024-02-01T10:27:55.681 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:55 GMT /vclfcn-19/ 2024-02-01T10:27:55.682 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wki8t6MCR5BPImUDGD0NZpVUSJ4= 2024-02-01T10:27:55.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wki8t6MCR5BPImUDGD0NZpVUSJ4=', 'Content-Length': '0'} 2024-02-01T10:27:55.687 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000758481bcc6b33a1-0065bb722b-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:55.687 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:55.687 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:55.687 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:55.687 DEBUG:boto:Method: HEAD 2024-02-01T10:27:55.688 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:55.688 DEBUG:boto:Data: 2024-02-01T10:27:55.688 DEBUG:boto:Headers: {} 2024-02-01T10:27:55.688 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:55.688 DEBUG:boto:Port: 8001 2024-02-01T10:27:55.688 DEBUG:boto:Params: {} 2024-02-01T10:27:55.688 DEBUG:boto:Token: None 2024-02-01T10:27:55.688 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:55 GMT /vclfcn-19/obj 2024-02-01T10:27:55.688 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Cyn/5a0gZPBWiGnj7AxW/A7F5hs= 2024-02-01T10:27:55.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Cyn/5a0gZPBWiGnj7AxW/A7F5hs=', 'Content-Length': '0'} 2024-02-01T10:27:55.695 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002304c0608dd1a49c-0065bb722b-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:55.695 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:55.695 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:55.695 DEBUG:boto:Method: HEAD 2024-02-01T10:27:55.695 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:55.695 DEBUG:boto:Data: 2024-02-01T10:27:55.695 DEBUG:boto:Headers: {} 2024-02-01T10:27:55.695 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:55.695 DEBUG:boto:Port: 8000 2024-02-01T10:27:55.696 DEBUG:boto:Params: {} 2024-02-01T10:27:55.696 DEBUG:boto:Token: None 2024-02-01T10:27:55.696 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:55 GMT /vclfcn-19/obj 2024-02-01T10:27:55.696 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Cyn/5a0gZPBWiGnj7AxW/A7F5hs= 2024-02-01T10:27:55.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Cyn/5a0gZPBWiGnj7AxW/A7F5hs=', 'Content-Length': '0'} 2024-02-01T10:27:55.701 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e7792ccaeaf6933f-0065bb722b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:55.702 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:55.702 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:55.702 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:55.702 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:55.702 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:55.702 DEBUG:boto:Method: GET 2024-02-01T10:27:55.702 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:55.702 DEBUG:boto:Data: 2024-02-01T10:27:55.703 DEBUG:boto:Headers: {} 2024-02-01T10:27:55.703 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:55.703 DEBUG:boto:Port: 8001 2024-02-01T10:27:55.703 DEBUG:boto:Params: {} 2024-02-01T10:27:55.703 DEBUG:boto:Token: None 2024-02-01T10:27:55.703 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:55 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:55.703 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zYtH2v7xaLEZFLfsqScBM5GYpLM= 2024-02-01T10:27:55.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zYtH2v7xaLEZFLfsqScBM5GYpLM=', 'Content-Length': '0'} 2024-02-01T10:27:55.706 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000431123fa285e9fa7-0065bb722b-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:55.752 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:55.752 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:55.752 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:55.753 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:55.753 DEBUG:boto:Method: GET 2024-02-01T10:27:55.753 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:55.753 DEBUG:boto:Data: 2024-02-01T10:27:55.753 DEBUG:boto:Headers: {} 2024-02-01T10:27:55.753 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:55.753 DEBUG:boto:Port: 8000 2024-02-01T10:27:55.753 DEBUG:boto:Params: {} 2024-02-01T10:27:55.753 DEBUG:boto:Token: None 2024-02-01T10:27:55.754 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:55 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:55.754 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zYtH2v7xaLEZFLfsqScBM5GYpLM= 2024-02-01T10:27:55.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zYtH2v7xaLEZFLfsqScBM5GYpLM=', 'Content-Length': '0'} 2024-02-01T10:27:55.756 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eb7627670654dc6e-0065bb722b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:55.759 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:55.759 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:55.759 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:55.759 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:55.760 DEBUG:boto:Method: GET 2024-02-01T10:27:55.760 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:55.760 DEBUG:boto:Data: 2024-02-01T10:27:55.760 DEBUG:boto:Headers: {} 2024-02-01T10:27:55.760 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:55.760 DEBUG:boto:Port: 8001 2024-02-01T10:27:55.760 DEBUG:boto:Params: {} 2024-02-01T10:27:55.760 DEBUG:boto:Token: None 2024-02-01T10:27:55.760 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:55 GMT /vclfcn-19/ 2024-02-01T10:27:55.760 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wki8t6MCR5BPImUDGD0NZpVUSJ4= 2024-02-01T10:27:55.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wki8t6MCR5BPImUDGD0NZpVUSJ4=', 'Content-Length': '0'} 2024-02-01T10:27:55.767 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5f5a7186d873220-0065bb722b-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:55.767 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:55.768 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:55.768 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:55.768 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:55.768 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:55.768 DEBUG:boto:Method: GET 2024-02-01T10:27:55.768 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:55.768 DEBUG:boto:Data: 2024-02-01T10:27:55.768 DEBUG:boto:Headers: {} 2024-02-01T10:27:55.768 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:55.768 DEBUG:boto:Port: 8000 2024-02-01T10:27:55.769 DEBUG:boto:Params: {} 2024-02-01T10:27:55.769 DEBUG:boto:Token: None 2024-02-01T10:27:55.769 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:55 GMT /vclfcn-19/ 2024-02-01T10:27:55.769 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wki8t6MCR5BPImUDGD0NZpVUSJ4= 2024-02-01T10:27:55.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wki8t6MCR5BPImUDGD0NZpVUSJ4=', 'Content-Length': '0'} 2024-02-01T10:27:55.775 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000acca0050d09048d7-0065bb722b-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:55.816 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:55.816 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:55.817 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:55.817 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:55.817 DEBUG:boto:path=/vclfcn-19/obj?versionId=iGO2QyMOL81IaF-H4sHDqyECPlnZBmE 2024-02-01T10:27:55.817 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=iGO2QyMOL81IaF-H4sHDqyECPlnZBmE 2024-02-01T10:27:55.817 DEBUG:boto:Method: GET 2024-02-01T10:27:55.817 DEBUG:boto:Path: /vclfcn-19/obj?versionId=iGO2QyMOL81IaF-H4sHDqyECPlnZBmE 2024-02-01T10:27:55.817 DEBUG:boto:Data: 2024-02-01T10:27:55.817 DEBUG:boto:Headers: {} 2024-02-01T10:27:55.817 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:55.818 DEBUG:boto:Port: 8001 2024-02-01T10:27:55.818 DEBUG:boto:Params: {} 2024-02-01T10:27:55.818 DEBUG:boto:Token: None 2024-02-01T10:27:55.818 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:55 GMT /vclfcn-19/obj?versionId=iGO2QyMOL81IaF-H4sHDqyECPlnZBmE 2024-02-01T10:27:55.818 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/8rSWW7OPeQjwLWNWsOjuyWXLUM= 2024-02-01T10:27:55.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/8rSWW7OPeQjwLWNWsOjuyWXLUM=', 'Content-Length': '0'} 2024-02-01T10:27:55.822 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'iGO2QyMOL81IaF-H4sHDqyECPlnZBmE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000844e7d00e8a9b6e5-0065bb722b-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:55.822 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:55.823 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:55.823 DEBUG:boto:path=/vclfcn-19/obj?versionId=iGO2QyMOL81IaF-H4sHDqyECPlnZBmE 2024-02-01T10:27:55.823 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=iGO2QyMOL81IaF-H4sHDqyECPlnZBmE 2024-02-01T10:27:55.823 DEBUG:boto:Method: GET 2024-02-01T10:27:55.823 DEBUG:boto:Path: /vclfcn-19/obj?versionId=iGO2QyMOL81IaF-H4sHDqyECPlnZBmE 2024-02-01T10:27:55.823 DEBUG:boto:Data: 2024-02-01T10:27:55.823 DEBUG:boto:Headers: {} 2024-02-01T10:27:55.823 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:55.823 DEBUG:boto:Port: 8000 2024-02-01T10:27:55.823 DEBUG:boto:Params: {} 2024-02-01T10:27:55.824 DEBUG:boto:Token: None 2024-02-01T10:27:55.824 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:55 GMT /vclfcn-19/obj?versionId=iGO2QyMOL81IaF-H4sHDqyECPlnZBmE 2024-02-01T10:27:55.824 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/8rSWW7OPeQjwLWNWsOjuyWXLUM= 2024-02-01T10:27:55.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/8rSWW7OPeQjwLWNWsOjuyWXLUM=', 'Content-Length': '0'} 2024-02-01T10:27:55.827 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'iGO2QyMOL81IaF-H4sHDqyECPlnZBmE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000065749996fe4e5af5-0065bb722b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:55.872 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:55.872 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:55.872 DEBUG:boto:path=/vclfcn-19/obj?versionId=iGO2QyMOL81IaF-H4sHDqyECPlnZBmE 2024-02-01T10:27:55.873 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=iGO2QyMOL81IaF-H4sHDqyECPlnZBmE 2024-02-01T10:27:55.873 DEBUG:boto:Method: HEAD 2024-02-01T10:27:55.873 DEBUG:boto:Path: /vclfcn-19/obj?versionId=iGO2QyMOL81IaF-H4sHDqyECPlnZBmE 2024-02-01T10:27:55.873 DEBUG:boto:Data: 2024-02-01T10:27:55.873 DEBUG:boto:Headers: {} 2024-02-01T10:27:55.873 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:55.873 DEBUG:boto:Port: 8001 2024-02-01T10:27:55.873 DEBUG:boto:Params: {} 2024-02-01T10:27:55.873 DEBUG:boto:Token: None 2024-02-01T10:27:55.874 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:55 GMT /vclfcn-19/obj?versionId=iGO2QyMOL81IaF-H4sHDqyECPlnZBmE 2024-02-01T10:27:55.874 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sLk1JAybehFkt1d77NhTbA+m6ec= 2024-02-01T10:27:55.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sLk1JAybehFkt1d77NhTbA+m6ec=', 'Content-Length': '0'} 2024-02-01T10:27:55.877 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'iGO2QyMOL81IaF-H4sHDqyECPlnZBmE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005bddc7528b1c3a2c-0065bb722b-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:55.877 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:55.877 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:55.878 DEBUG:boto:path=/vclfcn-19/obj?versionId=iGO2QyMOL81IaF-H4sHDqyECPlnZBmE 2024-02-01T10:27:55.878 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=iGO2QyMOL81IaF-H4sHDqyECPlnZBmE 2024-02-01T10:27:55.878 DEBUG:boto:Method: HEAD 2024-02-01T10:27:55.878 DEBUG:boto:Path: /vclfcn-19/obj?versionId=iGO2QyMOL81IaF-H4sHDqyECPlnZBmE 2024-02-01T10:27:55.878 DEBUG:boto:Data: 2024-02-01T10:27:55.878 DEBUG:boto:Headers: {} 2024-02-01T10:27:55.878 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:55.878 DEBUG:boto:Port: 8000 2024-02-01T10:27:55.878 DEBUG:boto:Params: {} 2024-02-01T10:27:55.878 DEBUG:boto:Token: None 2024-02-01T10:27:55.878 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:55 GMT /vclfcn-19/obj?versionId=iGO2QyMOL81IaF-H4sHDqyECPlnZBmE 2024-02-01T10:27:55.879 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sLk1JAybehFkt1d77NhTbA+m6ec= 2024-02-01T10:27:55.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sLk1JAybehFkt1d77NhTbA+m6ec=', 'Content-Length': '0'} 2024-02-01T10:27:55.881 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'iGO2QyMOL81IaF-H4sHDqyECPlnZBmE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e0c9807516feb987-0065bb722b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:55.881 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:55.882 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:55.882 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:55.882 DEBUG:boto:path=/vclfcn-19/obj?versionId=iGO2QyMOL81IaF-H4sHDqyECPlnZBmE 2024-02-01T10:27:55.882 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=iGO2QyMOL81IaF-H4sHDqyECPlnZBmE 2024-02-01T10:27:55.882 DEBUG:boto:Method: GET 2024-02-01T10:27:55.882 DEBUG:boto:Path: /vclfcn-19/obj?versionId=iGO2QyMOL81IaF-H4sHDqyECPlnZBmE 2024-02-01T10:27:55.882 DEBUG:boto:Data: 2024-02-01T10:27:55.882 DEBUG:boto:Headers: {} 2024-02-01T10:27:55.882 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:55.882 DEBUG:boto:Port: 8001 2024-02-01T10:27:55.882 DEBUG:boto:Params: {} 2024-02-01T10:27:55.883 DEBUG:boto:Token: None 2024-02-01T10:27:55.883 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:55 GMT /vclfcn-19/obj?versionId=iGO2QyMOL81IaF-H4sHDqyECPlnZBmE 2024-02-01T10:27:55.883 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/8rSWW7OPeQjwLWNWsOjuyWXLUM= 2024-02-01T10:27:55.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/8rSWW7OPeQjwLWNWsOjuyWXLUM=', 'Content-Length': '0'} 2024-02-01T10:27:55.886 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'iGO2QyMOL81IaF-H4sHDqyECPlnZBmE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000496eb4dbe08e2d83-0065bb722b-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:55.928 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:55.928 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:55.928 DEBUG:boto:path=/vclfcn-19/obj?versionId=iGO2QyMOL81IaF-H4sHDqyECPlnZBmE 2024-02-01T10:27:55.929 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=iGO2QyMOL81IaF-H4sHDqyECPlnZBmE 2024-02-01T10:27:55.929 DEBUG:boto:Method: GET 2024-02-01T10:27:55.929 DEBUG:boto:Path: /vclfcn-19/obj?versionId=iGO2QyMOL81IaF-H4sHDqyECPlnZBmE 2024-02-01T10:27:55.929 DEBUG:boto:Data: 2024-02-01T10:27:55.929 DEBUG:boto:Headers: {} 2024-02-01T10:27:55.929 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:55.929 DEBUG:boto:Port: 8000 2024-02-01T10:27:55.929 DEBUG:boto:Params: {} 2024-02-01T10:27:55.929 DEBUG:boto:Token: None 2024-02-01T10:27:55.929 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:55 GMT /vclfcn-19/obj?versionId=iGO2QyMOL81IaF-H4sHDqyECPlnZBmE 2024-02-01T10:27:55.930 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/8rSWW7OPeQjwLWNWsOjuyWXLUM= 2024-02-01T10:27:55.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/8rSWW7OPeQjwLWNWsOjuyWXLUM=', 'Content-Length': '0'} 2024-02-01T10:27:55.933 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'iGO2QyMOL81IaF-H4sHDqyECPlnZBmE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000081b7f20220054b2d-0065bb722b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:55.976 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:55.976 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:55.976 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:55.977 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:55.977 DEBUG:boto:Method: GET 2024-02-01T10:27:55.977 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:55.977 DEBUG:boto:Data: 2024-02-01T10:27:55.977 DEBUG:boto:Headers: {} 2024-02-01T10:27:55.977 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:55.977 DEBUG:boto:Port: 8001 2024-02-01T10:27:55.977 DEBUG:boto:Params: {} 2024-02-01T10:27:55.977 DEBUG:boto:Token: None 2024-02-01T10:27:55.977 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:55 GMT /vclfcn-19/ 2024-02-01T10:27:55.978 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wki8t6MCR5BPImUDGD0NZpVUSJ4= 2024-02-01T10:27:55.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wki8t6MCR5BPImUDGD0NZpVUSJ4=', 'Content-Length': '0'} 2024-02-01T10:27:55.983 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb88efb7684392ce-0065bb722b-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:56.024 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:56.025 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:56.025 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:56.025 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:56.025 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:56.025 DEBUG:boto:Method: GET 2024-02-01T10:27:56.025 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:56.025 DEBUG:boto:Data: 2024-02-01T10:27:56.025 DEBUG:boto:Headers: {} 2024-02-01T10:27:56.025 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:56.025 DEBUG:boto:Port: 8000 2024-02-01T10:27:56.026 DEBUG:boto:Params: {} 2024-02-01T10:27:56.026 DEBUG:boto:Token: None 2024-02-01T10:27:56.026 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:56 GMT /vclfcn-19/ 2024-02-01T10:27:56.026 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:pyGs672SFMK75mNjBfDPLe76hag= 2024-02-01T10:27:56.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:pyGs672SFMK75mNjBfDPLe76hag=', 'Content-Length': '0'} 2024-02-01T10:27:56.032 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000826c45001a9d4352-0065bb722c-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:56.032 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:56.032 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:56.033 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:56.033 DEBUG:boto:Method: HEAD 2024-02-01T10:27:56.033 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:56.033 DEBUG:boto:Data: 2024-02-01T10:27:56.033 DEBUG:boto:Headers: {} 2024-02-01T10:27:56.033 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:56.033 DEBUG:boto:Port: 8001 2024-02-01T10:27:56.033 DEBUG:boto:Params: {} 2024-02-01T10:27:56.033 DEBUG:boto:Token: None 2024-02-01T10:27:56.033 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:56 GMT /vclfcn-19/obj 2024-02-01T10:27:56.034 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nPTPE31IXHfolj4uMohn+L+vJAM= 2024-02-01T10:27:56.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nPTPE31IXHfolj4uMohn+L+vJAM=', 'Content-Length': '0'} 2024-02-01T10:27:56.040 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006fc48ec5e3e2521f-0065bb722c-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:56.041 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:56.041 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:56.041 DEBUG:boto:Method: HEAD 2024-02-01T10:27:56.041 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:56.041 DEBUG:boto:Data: 2024-02-01T10:27:56.041 DEBUG:boto:Headers: {} 2024-02-01T10:27:56.041 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:56.041 DEBUG:boto:Port: 8000 2024-02-01T10:27:56.042 DEBUG:boto:Params: {} 2024-02-01T10:27:56.042 DEBUG:boto:Token: None 2024-02-01T10:27:56.042 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:56 GMT /vclfcn-19/obj 2024-02-01T10:27:56.042 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nPTPE31IXHfolj4uMohn+L+vJAM= 2024-02-01T10:27:56.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nPTPE31IXHfolj4uMohn+L+vJAM=', 'Content-Length': '0'} 2024-02-01T10:27:56.048 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c65cfe14ece98d07-0065bb722c-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:56.048 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:56.048 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:56.049 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:56.049 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:56.049 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:56.049 DEBUG:boto:Method: GET 2024-02-01T10:27:56.049 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:56.049 DEBUG:boto:Data: 2024-02-01T10:27:56.049 DEBUG:boto:Headers: {} 2024-02-01T10:27:56.049 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:56.049 DEBUG:boto:Port: 8001 2024-02-01T10:27:56.049 DEBUG:boto:Params: {} 2024-02-01T10:27:56.050 DEBUG:boto:Token: None 2024-02-01T10:27:56.050 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:56 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:56.050 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:2aSf7utdTDMKaQFtDMi4AnqmFQg= 2024-02-01T10:27:56.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2aSf7utdTDMKaQFtDMi4AnqmFQg=', 'Content-Length': '0'} 2024-02-01T10:27:56.053 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000249289ab8ecf36e2-0065bb722c-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:56.100 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:56.100 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:56.100 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:56.100 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:56.101 DEBUG:boto:Method: GET 2024-02-01T10:27:56.101 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:56.101 DEBUG:boto:Data: 2024-02-01T10:27:56.101 DEBUG:boto:Headers: {} 2024-02-01T10:27:56.101 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:56.101 DEBUG:boto:Port: 8000 2024-02-01T10:27:56.101 DEBUG:boto:Params: {} 2024-02-01T10:27:56.101 DEBUG:boto:Token: None 2024-02-01T10:27:56.102 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:56 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:56.102 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:2aSf7utdTDMKaQFtDMi4AnqmFQg= 2024-02-01T10:27:56.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2aSf7utdTDMKaQFtDMi4AnqmFQg=', 'Content-Length': '0'} 2024-02-01T10:27:56.105 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000694e619d0c9d42a4-0065bb722c-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:56.106 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:56.106 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:56.106 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:56.106 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:56.106 DEBUG:boto:Method: GET 2024-02-01T10:27:56.106 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:56.106 DEBUG:boto:Data: 2024-02-01T10:27:56.106 DEBUG:boto:Headers: {} 2024-02-01T10:27:56.107 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:56.107 DEBUG:boto:Port: 8001 2024-02-01T10:27:56.107 DEBUG:boto:Params: {} 2024-02-01T10:27:56.107 DEBUG:boto:Token: None 2024-02-01T10:27:56.107 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:56 GMT /vclfcn-19/ 2024-02-01T10:27:56.107 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:pyGs672SFMK75mNjBfDPLe76hag= 2024-02-01T10:27:56.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:pyGs672SFMK75mNjBfDPLe76hag=', 'Content-Length': '0'} 2024-02-01T10:27:56.113 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077e0ee8dc1196241-0065bb722c-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:56.156 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:56.157 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:56.157 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:56.157 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:56.157 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:56.157 DEBUG:boto:Method: GET 2024-02-01T10:27:56.157 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:56.157 DEBUG:boto:Data: 2024-02-01T10:27:56.157 DEBUG:boto:Headers: {} 2024-02-01T10:27:56.157 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:56.157 DEBUG:boto:Port: 8000 2024-02-01T10:27:56.158 DEBUG:boto:Params: {} 2024-02-01T10:27:56.158 DEBUG:boto:Token: None 2024-02-01T10:27:56.158 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:56 GMT /vclfcn-19/ 2024-02-01T10:27:56.158 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:pyGs672SFMK75mNjBfDPLe76hag= 2024-02-01T10:27:56.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:pyGs672SFMK75mNjBfDPLe76hag=', 'Content-Length': '0'} 2024-02-01T10:27:56.163 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006037602c2639fba7-0065bb722c-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:56.164 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:56.164 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:56.164 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:56.164 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:56.164 DEBUG:boto:path=/vclfcn-19/obj?versionId=cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI 2024-02-01T10:27:56.164 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI 2024-02-01T10:27:56.164 DEBUG:boto:Method: GET 2024-02-01T10:27:56.165 DEBUG:boto:Path: /vclfcn-19/obj?versionId=cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI 2024-02-01T10:27:56.165 DEBUG:boto:Data: 2024-02-01T10:27:56.165 DEBUG:boto:Headers: {} 2024-02-01T10:27:56.165 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:56.165 DEBUG:boto:Port: 8001 2024-02-01T10:27:56.165 DEBUG:boto:Params: {} 2024-02-01T10:27:56.165 DEBUG:boto:Token: None 2024-02-01T10:27:56.165 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:56 GMT /vclfcn-19/obj?versionId=cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI 2024-02-01T10:27:56.165 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7tcTUuvn+CyKHTxmM3gB6jGNa7k= 2024-02-01T10:27:56.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7tcTUuvn+CyKHTxmM3gB6jGNa7k=', 'Content-Length': '0'} 2024-02-01T10:27:56.168 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000768a0ae8e77e242f-0065bb722c-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:56.171 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:56.171 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:56.171 DEBUG:boto:path=/vclfcn-19/obj?versionId=cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI 2024-02-01T10:27:56.172 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI 2024-02-01T10:27:56.172 DEBUG:boto:Method: GET 2024-02-01T10:27:56.172 DEBUG:boto:Path: /vclfcn-19/obj?versionId=cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI 2024-02-01T10:27:56.172 DEBUG:boto:Data: 2024-02-01T10:27:56.172 DEBUG:boto:Headers: {} 2024-02-01T10:27:56.172 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:56.172 DEBUG:boto:Port: 8000 2024-02-01T10:27:56.172 DEBUG:boto:Params: {} 2024-02-01T10:27:56.172 DEBUG:boto:Token: None 2024-02-01T10:27:56.172 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:56 GMT /vclfcn-19/obj?versionId=cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI 2024-02-01T10:27:56.173 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7tcTUuvn+CyKHTxmM3gB6jGNa7k= 2024-02-01T10:27:56.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7tcTUuvn+CyKHTxmM3gB6jGNa7k=', 'Content-Length': '0'} 2024-02-01T10:27:56.175 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d77574de62c5114f-0065bb722c-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:56.216 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:56.216 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:56.216 DEBUG:boto:path=/vclfcn-19/obj?versionId=cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI 2024-02-01T10:27:56.216 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI 2024-02-01T10:27:56.216 DEBUG:boto:Method: HEAD 2024-02-01T10:27:56.217 DEBUG:boto:Path: /vclfcn-19/obj?versionId=cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI 2024-02-01T10:27:56.217 DEBUG:boto:Data: 2024-02-01T10:27:56.217 DEBUG:boto:Headers: {} 2024-02-01T10:27:56.217 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:56.217 DEBUG:boto:Port: 8001 2024-02-01T10:27:56.217 DEBUG:boto:Params: {} 2024-02-01T10:27:56.217 DEBUG:boto:Token: None 2024-02-01T10:27:56.217 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:56 GMT /vclfcn-19/obj?versionId=cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI 2024-02-01T10:27:56.217 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+OPMCh7qkwWFZfOSql0MljF2aEA= 2024-02-01T10:27:56.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+OPMCh7qkwWFZfOSql0MljF2aEA=', 'Content-Length': '0'} 2024-02-01T10:27:56.220 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dccc89207d4eb7dd-0065bb722c-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:56.221 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:56.221 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:56.221 DEBUG:boto:path=/vclfcn-19/obj?versionId=cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI 2024-02-01T10:27:56.221 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI 2024-02-01T10:27:56.221 DEBUG:boto:Method: HEAD 2024-02-01T10:27:56.221 DEBUG:boto:Path: /vclfcn-19/obj?versionId=cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI 2024-02-01T10:27:56.221 DEBUG:boto:Data: 2024-02-01T10:27:56.221 DEBUG:boto:Headers: {} 2024-02-01T10:27:56.222 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:56.222 DEBUG:boto:Port: 8000 2024-02-01T10:27:56.222 DEBUG:boto:Params: {} 2024-02-01T10:27:56.222 DEBUG:boto:Token: None 2024-02-01T10:27:56.222 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:56 GMT /vclfcn-19/obj?versionId=cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI 2024-02-01T10:27:56.222 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+OPMCh7qkwWFZfOSql0MljF2aEA= 2024-02-01T10:27:56.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+OPMCh7qkwWFZfOSql0MljF2aEA=', 'Content-Length': '0'} 2024-02-01T10:27:56.225 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dd6cbeb7a70abccd-0065bb722c-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:56.225 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:56.225 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:56.226 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:56.226 DEBUG:boto:path=/vclfcn-19/obj?versionId=cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI 2024-02-01T10:27:56.226 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI 2024-02-01T10:27:56.226 DEBUG:boto:Method: GET 2024-02-01T10:27:56.226 DEBUG:boto:Path: /vclfcn-19/obj?versionId=cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI 2024-02-01T10:27:56.226 DEBUG:boto:Data: 2024-02-01T10:27:56.226 DEBUG:boto:Headers: {} 2024-02-01T10:27:56.226 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:56.226 DEBUG:boto:Port: 8001 2024-02-01T10:27:56.226 DEBUG:boto:Params: {} 2024-02-01T10:27:56.227 DEBUG:boto:Token: None 2024-02-01T10:27:56.227 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:56 GMT /vclfcn-19/obj?versionId=cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI 2024-02-01T10:27:56.227 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7tcTUuvn+CyKHTxmM3gB6jGNa7k= 2024-02-01T10:27:56.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7tcTUuvn+CyKHTxmM3gB6jGNa7k=', 'Content-Length': '0'} 2024-02-01T10:27:56.230 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008ea26553d4cda421-0065bb722c-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:56.272 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:56.272 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:56.272 DEBUG:boto:path=/vclfcn-19/obj?versionId=cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI 2024-02-01T10:27:56.272 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI 2024-02-01T10:27:56.273 DEBUG:boto:Method: GET 2024-02-01T10:27:56.273 DEBUG:boto:Path: /vclfcn-19/obj?versionId=cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI 2024-02-01T10:27:56.273 DEBUG:boto:Data: 2024-02-01T10:27:56.273 DEBUG:boto:Headers: {} 2024-02-01T10:27:56.273 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:56.273 DEBUG:boto:Port: 8000 2024-02-01T10:27:56.273 DEBUG:boto:Params: {} 2024-02-01T10:27:56.273 DEBUG:boto:Token: None 2024-02-01T10:27:56.273 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:56 GMT /vclfcn-19/obj?versionId=cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI 2024-02-01T10:27:56.274 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7tcTUuvn+CyKHTxmM3gB6jGNa7k= 2024-02-01T10:27:56.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7tcTUuvn+CyKHTxmM3gB6jGNa7k=', 'Content-Length': '0'} 2024-02-01T10:27:56.277 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'cYHYWFiJtzK2FpqgaUq4bR.SX5MaCkI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c7743e9723a4b9b9-0065bb722c-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:56.320 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:56.320 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:56.320 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:56.321 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:56.321 DEBUG:boto:Method: GET 2024-02-01T10:27:56.321 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:56.321 DEBUG:boto:Data: 2024-02-01T10:27:56.321 DEBUG:boto:Headers: {} 2024-02-01T10:27:56.321 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:56.321 DEBUG:boto:Port: 8001 2024-02-01T10:27:56.321 DEBUG:boto:Params: {} 2024-02-01T10:27:56.321 DEBUG:boto:Token: None 2024-02-01T10:27:56.321 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:56 GMT /vclfcn-19/ 2024-02-01T10:27:56.322 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:pyGs672SFMK75mNjBfDPLe76hag= 2024-02-01T10:27:56.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:pyGs672SFMK75mNjBfDPLe76hag=', 'Content-Length': '0'} 2024-02-01T10:27:56.327 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f44f7837c98134ec-0065bb722c-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:56.368 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:56.368 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:56.369 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:56.369 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:56.369 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:56.369 DEBUG:boto:Method: GET 2024-02-01T10:27:56.369 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:56.369 DEBUG:boto:Data: 2024-02-01T10:27:56.369 DEBUG:boto:Headers: {} 2024-02-01T10:27:56.369 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:56.369 DEBUG:boto:Port: 8000 2024-02-01T10:27:56.369 DEBUG:boto:Params: {} 2024-02-01T10:27:56.370 DEBUG:boto:Token: None 2024-02-01T10:27:56.370 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:56 GMT /vclfcn-19/ 2024-02-01T10:27:56.370 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:pyGs672SFMK75mNjBfDPLe76hag= 2024-02-01T10:27:56.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:pyGs672SFMK75mNjBfDPLe76hag=', 'Content-Length': '0'} 2024-02-01T10:27:56.375 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2e319eecbed3eb5-0065bb722c-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:56.375 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:56.376 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:56.376 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:56.376 DEBUG:boto:Method: HEAD 2024-02-01T10:27:56.376 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:56.376 DEBUG:boto:Data: 2024-02-01T10:27:56.376 DEBUG:boto:Headers: {} 2024-02-01T10:27:56.376 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:56.376 DEBUG:boto:Port: 8001 2024-02-01T10:27:56.376 DEBUG:boto:Params: {} 2024-02-01T10:27:56.377 DEBUG:boto:Token: None 2024-02-01T10:27:56.377 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:56 GMT /vclfcn-19/obj 2024-02-01T10:27:56.377 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nPTPE31IXHfolj4uMohn+L+vJAM= 2024-02-01T10:27:56.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nPTPE31IXHfolj4uMohn+L+vJAM=', 'Content-Length': '0'} 2024-02-01T10:27:56.383 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d30462083d7218e5-0065bb722c-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:56.383 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:56.383 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:56.383 DEBUG:boto:Method: HEAD 2024-02-01T10:27:56.383 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:56.383 DEBUG:boto:Data: 2024-02-01T10:27:56.384 DEBUG:boto:Headers: {} 2024-02-01T10:27:56.384 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:56.384 DEBUG:boto:Port: 8000 2024-02-01T10:27:56.384 DEBUG:boto:Params: {} 2024-02-01T10:27:56.384 DEBUG:boto:Token: None 2024-02-01T10:27:56.384 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:56 GMT /vclfcn-19/obj 2024-02-01T10:27:56.384 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nPTPE31IXHfolj4uMohn+L+vJAM= 2024-02-01T10:27:56.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nPTPE31IXHfolj4uMohn+L+vJAM=', 'Content-Length': '0'} 2024-02-01T10:27:56.391 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007796c658391661fe-0065bb722c-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:56.392 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:56.392 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:56.392 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:56.392 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:56.392 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:56.392 DEBUG:boto:Method: GET 2024-02-01T10:27:56.392 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:56.392 DEBUG:boto:Data: 2024-02-01T10:27:56.392 DEBUG:boto:Headers: {} 2024-02-01T10:27:56.393 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:56.393 DEBUG:boto:Port: 8001 2024-02-01T10:27:56.393 DEBUG:boto:Params: {} 2024-02-01T10:27:56.393 DEBUG:boto:Token: None 2024-02-01T10:27:56.393 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:56 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:56.393 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:2aSf7utdTDMKaQFtDMi4AnqmFQg= 2024-02-01T10:27:56.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2aSf7utdTDMKaQFtDMi4AnqmFQg=', 'Content-Length': '0'} 2024-02-01T10:27:56.396 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008482afdfadc88891-0065bb722c-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:56.444 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:56.444 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:56.444 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:56.444 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:56.445 DEBUG:boto:Method: GET 2024-02-01T10:27:56.445 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:56.445 DEBUG:boto:Data: 2024-02-01T10:27:56.445 DEBUG:boto:Headers: {} 2024-02-01T10:27:56.445 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:56.445 DEBUG:boto:Port: 8000 2024-02-01T10:27:56.445 DEBUG:boto:Params: {} 2024-02-01T10:27:56.445 DEBUG:boto:Token: None 2024-02-01T10:27:56.445 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:56 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:56.446 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:2aSf7utdTDMKaQFtDMi4AnqmFQg= 2024-02-01T10:27:56.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2aSf7utdTDMKaQFtDMi4AnqmFQg=', 'Content-Length': '0'} 2024-02-01T10:27:56.449 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006f84723702bce387-0065bb722c-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:56.449 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:56.449 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:56.450 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:56.450 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:56.450 DEBUG:boto:Method: GET 2024-02-01T10:27:56.450 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:56.450 DEBUG:boto:Data: 2024-02-01T10:27:56.450 DEBUG:boto:Headers: {} 2024-02-01T10:27:56.450 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:56.450 DEBUG:boto:Port: 8001 2024-02-01T10:27:56.450 DEBUG:boto:Params: {} 2024-02-01T10:27:56.450 DEBUG:boto:Token: None 2024-02-01T10:27:56.451 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:56 GMT /vclfcn-19/ 2024-02-01T10:27:56.451 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:pyGs672SFMK75mNjBfDPLe76hag= 2024-02-01T10:27:56.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:pyGs672SFMK75mNjBfDPLe76hag=', 'Content-Length': '0'} 2024-02-01T10:27:56.456 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008783942200d88e5f-0065bb722c-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:56.504 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:56.504 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:56.505 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:56.505 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:56.505 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:56.505 DEBUG:boto:Method: GET 2024-02-01T10:27:56.505 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:56.505 DEBUG:boto:Data: 2024-02-01T10:27:56.505 DEBUG:boto:Headers: {} 2024-02-01T10:27:56.505 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:56.505 DEBUG:boto:Port: 8000 2024-02-01T10:27:56.505 DEBUG:boto:Params: {} 2024-02-01T10:27:56.506 DEBUG:boto:Token: None 2024-02-01T10:27:56.506 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:56 GMT /vclfcn-19/ 2024-02-01T10:27:56.506 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:pyGs672SFMK75mNjBfDPLe76hag= 2024-02-01T10:27:56.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:pyGs672SFMK75mNjBfDPLe76hag=', 'Content-Length': '0'} 2024-02-01T10:27:56.511 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094ad3f74218f1793-0065bb722c-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:56.512 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:56.512 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:56.512 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:56.512 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:56.513 DEBUG:boto:path=/vclfcn-19/obj?versionId=rasya8taOZsGTj9rDi7keghf.VnPFlF 2024-02-01T10:27:56.513 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=rasya8taOZsGTj9rDi7keghf.VnPFlF 2024-02-01T10:27:56.513 DEBUG:boto:Method: GET 2024-02-01T10:27:56.513 DEBUG:boto:Path: /vclfcn-19/obj?versionId=rasya8taOZsGTj9rDi7keghf.VnPFlF 2024-02-01T10:27:56.513 DEBUG:boto:Data: 2024-02-01T10:27:56.513 DEBUG:boto:Headers: {} 2024-02-01T10:27:56.513 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:56.513 DEBUG:boto:Port: 8001 2024-02-01T10:27:56.513 DEBUG:boto:Params: {} 2024-02-01T10:27:56.513 DEBUG:boto:Token: None 2024-02-01T10:27:56.513 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:56 GMT /vclfcn-19/obj?versionId=rasya8taOZsGTj9rDi7keghf.VnPFlF 2024-02-01T10:27:56.514 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7Fwsz4csjFSxQWEoy906Z44hglc= 2024-02-01T10:27:56.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7Fwsz4csjFSxQWEoy906Z44hglc=', 'Content-Length': '0'} 2024-02-01T10:27:56.516 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'rasya8taOZsGTj9rDi7keghf.VnPFlF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c3e10a43475099c6-0065bb722c-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:56.517 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:56.517 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:56.517 DEBUG:boto:path=/vclfcn-19/obj?versionId=rasya8taOZsGTj9rDi7keghf.VnPFlF 2024-02-01T10:27:56.517 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=rasya8taOZsGTj9rDi7keghf.VnPFlF 2024-02-01T10:27:56.517 DEBUG:boto:Method: GET 2024-02-01T10:27:56.517 DEBUG:boto:Path: /vclfcn-19/obj?versionId=rasya8taOZsGTj9rDi7keghf.VnPFlF 2024-02-01T10:27:56.517 DEBUG:boto:Data: 2024-02-01T10:27:56.517 DEBUG:boto:Headers: {} 2024-02-01T10:27:56.517 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:56.518 DEBUG:boto:Port: 8000 2024-02-01T10:27:56.518 DEBUG:boto:Params: {} 2024-02-01T10:27:56.518 DEBUG:boto:Token: None 2024-02-01T10:27:56.518 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:56 GMT /vclfcn-19/obj?versionId=rasya8taOZsGTj9rDi7keghf.VnPFlF 2024-02-01T10:27:56.518 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7Fwsz4csjFSxQWEoy906Z44hglc= 2024-02-01T10:27:56.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7Fwsz4csjFSxQWEoy906Z44hglc=', 'Content-Length': '0'} 2024-02-01T10:27:56.521 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'rasya8taOZsGTj9rDi7keghf.VnPFlF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000016f5128b8b551805-0065bb722c-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:56.564 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:56.565 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:56.565 DEBUG:boto:path=/vclfcn-19/obj?versionId=rasya8taOZsGTj9rDi7keghf.VnPFlF 2024-02-01T10:27:56.565 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=rasya8taOZsGTj9rDi7keghf.VnPFlF 2024-02-01T10:27:56.565 DEBUG:boto:Method: HEAD 2024-02-01T10:27:56.565 DEBUG:boto:Path: /vclfcn-19/obj?versionId=rasya8taOZsGTj9rDi7keghf.VnPFlF 2024-02-01T10:27:56.565 DEBUG:boto:Data: 2024-02-01T10:27:56.565 DEBUG:boto:Headers: {} 2024-02-01T10:27:56.565 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:56.565 DEBUG:boto:Port: 8001 2024-02-01T10:27:56.565 DEBUG:boto:Params: {} 2024-02-01T10:27:56.566 DEBUG:boto:Token: None 2024-02-01T10:27:56.566 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:56 GMT /vclfcn-19/obj?versionId=rasya8taOZsGTj9rDi7keghf.VnPFlF 2024-02-01T10:27:56.566 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:XJOljz7evcRCGMYlO6PkQ3c1KTM= 2024-02-01T10:27:56.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:XJOljz7evcRCGMYlO6PkQ3c1KTM=', 'Content-Length': '0'} 2024-02-01T10:27:56.569 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'rasya8taOZsGTj9rDi7keghf.VnPFlF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000683503a940343cfb-0065bb722c-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:56.569 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:56.569 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:56.570 DEBUG:boto:path=/vclfcn-19/obj?versionId=rasya8taOZsGTj9rDi7keghf.VnPFlF 2024-02-01T10:27:56.570 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=rasya8taOZsGTj9rDi7keghf.VnPFlF 2024-02-01T10:27:56.570 DEBUG:boto:Method: HEAD 2024-02-01T10:27:56.570 DEBUG:boto:Path: /vclfcn-19/obj?versionId=rasya8taOZsGTj9rDi7keghf.VnPFlF 2024-02-01T10:27:56.570 DEBUG:boto:Data: 2024-02-01T10:27:56.570 DEBUG:boto:Headers: {} 2024-02-01T10:27:56.570 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:56.570 DEBUG:boto:Port: 8000 2024-02-01T10:27:56.570 DEBUG:boto:Params: {} 2024-02-01T10:27:56.570 DEBUG:boto:Token: None 2024-02-01T10:27:56.571 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:56 GMT /vclfcn-19/obj?versionId=rasya8taOZsGTj9rDi7keghf.VnPFlF 2024-02-01T10:27:56.571 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:XJOljz7evcRCGMYlO6PkQ3c1KTM= 2024-02-01T10:27:56.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:XJOljz7evcRCGMYlO6PkQ3c1KTM=', 'Content-Length': '0'} 2024-02-01T10:27:56.573 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'rasya8taOZsGTj9rDi7keghf.VnPFlF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000684bb7a7e4dfc1c9-0065bb722c-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:56.573 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:56.574 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:56.574 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:56.574 DEBUG:boto:path=/vclfcn-19/obj?versionId=rasya8taOZsGTj9rDi7keghf.VnPFlF 2024-02-01T10:27:56.574 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=rasya8taOZsGTj9rDi7keghf.VnPFlF 2024-02-01T10:27:56.574 DEBUG:boto:Method: GET 2024-02-01T10:27:56.574 DEBUG:boto:Path: /vclfcn-19/obj?versionId=rasya8taOZsGTj9rDi7keghf.VnPFlF 2024-02-01T10:27:56.574 DEBUG:boto:Data: 2024-02-01T10:27:56.574 DEBUG:boto:Headers: {} 2024-02-01T10:27:56.574 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:56.574 DEBUG:boto:Port: 8001 2024-02-01T10:27:56.575 DEBUG:boto:Params: {} 2024-02-01T10:27:56.575 DEBUG:boto:Token: None 2024-02-01T10:27:56.575 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:56 GMT /vclfcn-19/obj?versionId=rasya8taOZsGTj9rDi7keghf.VnPFlF 2024-02-01T10:27:56.575 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7Fwsz4csjFSxQWEoy906Z44hglc= 2024-02-01T10:27:56.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7Fwsz4csjFSxQWEoy906Z44hglc=', 'Content-Length': '0'} 2024-02-01T10:27:56.579 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'rasya8taOZsGTj9rDi7keghf.VnPFlF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000901ad23d61589da2-0065bb722c-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:56.579 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:56.580 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:56.580 DEBUG:boto:path=/vclfcn-19/obj?versionId=rasya8taOZsGTj9rDi7keghf.VnPFlF 2024-02-01T10:27:56.580 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=rasya8taOZsGTj9rDi7keghf.VnPFlF 2024-02-01T10:27:56.580 DEBUG:boto:Method: GET 2024-02-01T10:27:56.580 DEBUG:boto:Path: /vclfcn-19/obj?versionId=rasya8taOZsGTj9rDi7keghf.VnPFlF 2024-02-01T10:27:56.580 DEBUG:boto:Data: 2024-02-01T10:27:56.580 DEBUG:boto:Headers: {} 2024-02-01T10:27:56.580 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:56.580 DEBUG:boto:Port: 8000 2024-02-01T10:27:56.580 DEBUG:boto:Params: {} 2024-02-01T10:27:56.581 DEBUG:boto:Token: None 2024-02-01T10:27:56.581 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:56 GMT /vclfcn-19/obj?versionId=rasya8taOZsGTj9rDi7keghf.VnPFlF 2024-02-01T10:27:56.581 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7Fwsz4csjFSxQWEoy906Z44hglc= 2024-02-01T10:27:56.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7Fwsz4csjFSxQWEoy906Z44hglc=', 'Content-Length': '0'} 2024-02-01T10:27:56.584 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'rasya8taOZsGTj9rDi7keghf.VnPFlF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000f88672cca3a2b97-0065bb722c-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:56.624 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:56.624 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:56.624 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:56.624 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:56.624 DEBUG:boto:Method: GET 2024-02-01T10:27:56.625 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:56.625 DEBUG:boto:Data: 2024-02-01T10:27:56.625 DEBUG:boto:Headers: {} 2024-02-01T10:27:56.625 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:56.625 DEBUG:boto:Port: 8001 2024-02-01T10:27:56.625 DEBUG:boto:Params: {} 2024-02-01T10:27:56.625 DEBUG:boto:Token: None 2024-02-01T10:27:56.625 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:56 GMT /vclfcn-19/ 2024-02-01T10:27:56.625 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:pyGs672SFMK75mNjBfDPLe76hag= 2024-02-01T10:27:56.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:pyGs672SFMK75mNjBfDPLe76hag=', 'Content-Length': '0'} 2024-02-01T10:27:56.631 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9ee884e4eb764b1-0065bb722c-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:56.631 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:56.631 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:56.632 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:56.632 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:56.632 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:56.632 DEBUG:boto:Method: GET 2024-02-01T10:27:56.632 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:56.632 DEBUG:boto:Data: 2024-02-01T10:27:56.632 DEBUG:boto:Headers: {} 2024-02-01T10:27:56.632 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:56.632 DEBUG:boto:Port: 8000 2024-02-01T10:27:56.632 DEBUG:boto:Params: {} 2024-02-01T10:27:56.633 DEBUG:boto:Token: None 2024-02-01T10:27:56.633 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:56 GMT /vclfcn-19/ 2024-02-01T10:27:56.633 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:pyGs672SFMK75mNjBfDPLe76hag= 2024-02-01T10:27:56.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:pyGs672SFMK75mNjBfDPLe76hag=', 'Content-Length': '0'} 2024-02-01T10:27:56.638 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a342b7625522dcfa-0065bb722c-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:56.680 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:56.680 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:56.681 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:56.681 DEBUG:boto:Method: HEAD 2024-02-01T10:27:56.681 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:56.681 DEBUG:boto:Data: 2024-02-01T10:27:56.681 DEBUG:boto:Headers: {} 2024-02-01T10:27:56.681 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:56.681 DEBUG:boto:Port: 8001 2024-02-01T10:27:56.681 DEBUG:boto:Params: {} 2024-02-01T10:27:56.681 DEBUG:boto:Token: None 2024-02-01T10:27:56.682 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:56 GMT /vclfcn-19/obj 2024-02-01T10:27:56.682 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nPTPE31IXHfolj4uMohn+L+vJAM= 2024-02-01T10:27:56.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nPTPE31IXHfolj4uMohn+L+vJAM=', 'Content-Length': '0'} 2024-02-01T10:27:56.688 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b16dc5e4f6022661-0065bb722c-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:56.688 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:56.688 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:56.688 DEBUG:boto:Method: HEAD 2024-02-01T10:27:56.688 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:56.688 DEBUG:boto:Data: 2024-02-01T10:27:56.688 DEBUG:boto:Headers: {} 2024-02-01T10:27:56.688 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:56.689 DEBUG:boto:Port: 8000 2024-02-01T10:27:56.689 DEBUG:boto:Params: {} 2024-02-01T10:27:56.689 DEBUG:boto:Token: None 2024-02-01T10:27:56.689 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:56 GMT /vclfcn-19/obj 2024-02-01T10:27:56.689 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nPTPE31IXHfolj4uMohn+L+vJAM= 2024-02-01T10:27:56.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nPTPE31IXHfolj4uMohn+L+vJAM=', 'Content-Length': '0'} 2024-02-01T10:27:56.695 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007c2a0830e65938c4-0065bb722c-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:56.695 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:56.695 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:56.695 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:56.696 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:56.696 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:56.696 DEBUG:boto:Method: GET 2024-02-01T10:27:56.696 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:56.696 DEBUG:boto:Data: 2024-02-01T10:27:56.696 DEBUG:boto:Headers: {} 2024-02-01T10:27:56.696 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:56.696 DEBUG:boto:Port: 8001 2024-02-01T10:27:56.696 DEBUG:boto:Params: {} 2024-02-01T10:27:56.696 DEBUG:boto:Token: None 2024-02-01T10:27:56.697 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:56 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:56.697 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:2aSf7utdTDMKaQFtDMi4AnqmFQg= 2024-02-01T10:27:56.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2aSf7utdTDMKaQFtDMi4AnqmFQg=', 'Content-Length': '0'} 2024-02-01T10:27:56.700 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f1ad2452f5a6c306-0065bb722c-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:56.740 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:56.740 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:56.740 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:56.740 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:56.740 DEBUG:boto:Method: GET 2024-02-01T10:27:56.741 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:56.741 DEBUG:boto:Data: 2024-02-01T10:27:56.741 DEBUG:boto:Headers: {} 2024-02-01T10:27:56.741 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:56.741 DEBUG:boto:Port: 8000 2024-02-01T10:27:56.741 DEBUG:boto:Params: {} 2024-02-01T10:27:56.741 DEBUG:boto:Token: None 2024-02-01T10:27:56.741 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:56 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:56.741 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:2aSf7utdTDMKaQFtDMi4AnqmFQg= 2024-02-01T10:27:56.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2aSf7utdTDMKaQFtDMi4AnqmFQg=', 'Content-Length': '0'} 2024-02-01T10:27:56.744 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d17323d576a2038b-0065bb722c-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:56.744 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:56.744 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:56.745 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:56.745 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:56.745 DEBUG:boto:Method: GET 2024-02-01T10:27:56.745 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:56.745 DEBUG:boto:Data: 2024-02-01T10:27:56.745 DEBUG:boto:Headers: {} 2024-02-01T10:27:56.745 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:56.745 DEBUG:boto:Port: 8001 2024-02-01T10:27:56.745 DEBUG:boto:Params: {} 2024-02-01T10:27:56.745 DEBUG:boto:Token: None 2024-02-01T10:27:56.745 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:56 GMT /vclfcn-19/ 2024-02-01T10:27:56.746 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:pyGs672SFMK75mNjBfDPLe76hag= 2024-02-01T10:27:56.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:pyGs672SFMK75mNjBfDPLe76hag=', 'Content-Length': '0'} 2024-02-01T10:27:56.752 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000693aedbe12b2a71a-0065bb722c-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:56.752 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:56.753 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:56.753 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:56.753 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:56.753 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:56.753 DEBUG:boto:Method: GET 2024-02-01T10:27:56.753 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:56.753 DEBUG:boto:Data: 2024-02-01T10:27:56.753 DEBUG:boto:Headers: {} 2024-02-01T10:27:56.753 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:56.753 DEBUG:boto:Port: 8000 2024-02-01T10:27:56.753 DEBUG:boto:Params: {} 2024-02-01T10:27:56.754 DEBUG:boto:Token: None 2024-02-01T10:27:56.754 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:56 GMT /vclfcn-19/ 2024-02-01T10:27:56.754 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:pyGs672SFMK75mNjBfDPLe76hag= 2024-02-01T10:27:56.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:pyGs672SFMK75mNjBfDPLe76hag=', 'Content-Length': '0'} 2024-02-01T10:27:56.759 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c5f3d324fd9dd08-0065bb722c-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:56.759 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:56.760 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:56.760 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:56.760 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:56.760 DEBUG:boto:path=/vclfcn-19/obj?versionId=y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE 2024-02-01T10:27:56.760 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE 2024-02-01T10:27:56.760 DEBUG:boto:Method: GET 2024-02-01T10:27:56.760 DEBUG:boto:Path: /vclfcn-19/obj?versionId=y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE 2024-02-01T10:27:56.760 DEBUG:boto:Data: 2024-02-01T10:27:56.761 DEBUG:boto:Headers: {} 2024-02-01T10:27:56.761 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:56.761 DEBUG:boto:Port: 8001 2024-02-01T10:27:56.761 DEBUG:boto:Params: {} 2024-02-01T10:27:56.761 DEBUG:boto:Token: None 2024-02-01T10:27:56.761 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:56 GMT /vclfcn-19/obj?versionId=y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE 2024-02-01T10:27:56.761 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:OooJl7IR9UZ3P8q1GgTJUJs1/44= 2024-02-01T10:27:56.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:OooJl7IR9UZ3P8q1GgTJUJs1/44=', 'Content-Length': '0'} 2024-02-01T10:27:56.764 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000015e7390ba707b7f0-0065bb722c-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:56.765 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:56.765 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:56.765 DEBUG:boto:path=/vclfcn-19/obj?versionId=y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE 2024-02-01T10:27:56.765 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE 2024-02-01T10:27:56.765 DEBUG:boto:Method: GET 2024-02-01T10:27:56.765 DEBUG:boto:Path: /vclfcn-19/obj?versionId=y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE 2024-02-01T10:27:56.765 DEBUG:boto:Data: 2024-02-01T10:27:56.765 DEBUG:boto:Headers: {} 2024-02-01T10:27:56.765 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:56.765 DEBUG:boto:Port: 8000 2024-02-01T10:27:56.765 DEBUG:boto:Params: {} 2024-02-01T10:27:56.766 DEBUG:boto:Token: None 2024-02-01T10:27:56.766 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:56 GMT /vclfcn-19/obj?versionId=y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE 2024-02-01T10:27:56.766 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:OooJl7IR9UZ3P8q1GgTJUJs1/44= 2024-02-01T10:27:56.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:OooJl7IR9UZ3P8q1GgTJUJs1/44=', 'Content-Length': '0'} 2024-02-01T10:27:56.769 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002b89d78d1f2625b6-0065bb722c-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:56.816 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:56.816 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:56.816 DEBUG:boto:path=/vclfcn-19/obj?versionId=y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE 2024-02-01T10:27:56.816 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE 2024-02-01T10:27:56.817 DEBUG:boto:Method: HEAD 2024-02-01T10:27:56.817 DEBUG:boto:Path: /vclfcn-19/obj?versionId=y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE 2024-02-01T10:27:56.817 DEBUG:boto:Data: 2024-02-01T10:27:56.817 DEBUG:boto:Headers: {} 2024-02-01T10:27:56.817 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:56.817 DEBUG:boto:Port: 8001 2024-02-01T10:27:56.817 DEBUG:boto:Params: {} 2024-02-01T10:27:56.817 DEBUG:boto:Token: None 2024-02-01T10:27:56.817 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:56 GMT /vclfcn-19/obj?versionId=y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE 2024-02-01T10:27:56.817 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:xvSTRXrdG3nviBBHOddFD4cbwao= 2024-02-01T10:27:56.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:xvSTRXrdG3nviBBHOddFD4cbwao=', 'Content-Length': '0'} 2024-02-01T10:27:56.820 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f036bc8ff253fa28-0065bb722c-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:56.821 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:56.821 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:56.821 DEBUG:boto:path=/vclfcn-19/obj?versionId=y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE 2024-02-01T10:27:56.821 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE 2024-02-01T10:27:56.821 DEBUG:boto:Method: HEAD 2024-02-01T10:27:56.821 DEBUG:boto:Path: /vclfcn-19/obj?versionId=y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE 2024-02-01T10:27:56.821 DEBUG:boto:Data: 2024-02-01T10:27:56.821 DEBUG:boto:Headers: {} 2024-02-01T10:27:56.821 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:56.821 DEBUG:boto:Port: 8000 2024-02-01T10:27:56.822 DEBUG:boto:Params: {} 2024-02-01T10:27:56.822 DEBUG:boto:Token: None 2024-02-01T10:27:56.822 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:56 GMT /vclfcn-19/obj?versionId=y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE 2024-02-01T10:27:56.822 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:xvSTRXrdG3nviBBHOddFD4cbwao= 2024-02-01T10:27:56.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:xvSTRXrdG3nviBBHOddFD4cbwao=', 'Content-Length': '0'} 2024-02-01T10:27:56.825 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000707069d506a85225-0065bb722c-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:56.825 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:56.825 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:56.825 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:56.825 DEBUG:boto:path=/vclfcn-19/obj?versionId=y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE 2024-02-01T10:27:56.825 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE 2024-02-01T10:27:56.825 DEBUG:boto:Method: GET 2024-02-01T10:27:56.825 DEBUG:boto:Path: /vclfcn-19/obj?versionId=y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE 2024-02-01T10:27:56.826 DEBUG:boto:Data: 2024-02-01T10:27:56.826 DEBUG:boto:Headers: {} 2024-02-01T10:27:56.826 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:56.826 DEBUG:boto:Port: 8001 2024-02-01T10:27:56.826 DEBUG:boto:Params: {} 2024-02-01T10:27:56.826 DEBUG:boto:Token: None 2024-02-01T10:27:56.826 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:56 GMT /vclfcn-19/obj?versionId=y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE 2024-02-01T10:27:56.826 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:OooJl7IR9UZ3P8q1GgTJUJs1/44= 2024-02-01T10:27:56.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:OooJl7IR9UZ3P8q1GgTJUJs1/44=', 'Content-Length': '0'} 2024-02-01T10:27:56.829 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000078a7c975bc99621d-0065bb722c-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:56.872 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:56.872 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:56.872 DEBUG:boto:path=/vclfcn-19/obj?versionId=y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE 2024-02-01T10:27:56.872 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE 2024-02-01T10:27:56.873 DEBUG:boto:Method: GET 2024-02-01T10:27:56.873 DEBUG:boto:Path: /vclfcn-19/obj?versionId=y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE 2024-02-01T10:27:56.873 DEBUG:boto:Data: 2024-02-01T10:27:56.873 DEBUG:boto:Headers: {} 2024-02-01T10:27:56.873 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:56.873 DEBUG:boto:Port: 8000 2024-02-01T10:27:56.873 DEBUG:boto:Params: {} 2024-02-01T10:27:56.873 DEBUG:boto:Token: None 2024-02-01T10:27:56.873 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:56 GMT /vclfcn-19/obj?versionId=y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE 2024-02-01T10:27:56.873 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:OooJl7IR9UZ3P8q1GgTJUJs1/44= 2024-02-01T10:27:56.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:OooJl7IR9UZ3P8q1GgTJUJs1/44=', 'Content-Length': '0'} 2024-02-01T10:27:56.876 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', 'y-DPCvngcj9q7W-lUL16Jgbp.NhV7NE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002604243b1bbd5458-0065bb722c-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:56.877 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:56.877 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:56.877 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:56.877 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:56.877 DEBUG:boto:Method: GET 2024-02-01T10:27:56.877 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:56.877 DEBUG:boto:Data: 2024-02-01T10:27:56.877 DEBUG:boto:Headers: {} 2024-02-01T10:27:56.877 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:56.878 DEBUG:boto:Port: 8001 2024-02-01T10:27:56.878 DEBUG:boto:Params: {} 2024-02-01T10:27:56.878 DEBUG:boto:Token: None 2024-02-01T10:27:56.878 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:56 GMT /vclfcn-19/ 2024-02-01T10:27:56.878 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:pyGs672SFMK75mNjBfDPLe76hag= 2024-02-01T10:27:56.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:pyGs672SFMK75mNjBfDPLe76hag=', 'Content-Length': '0'} 2024-02-01T10:27:56.883 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035a9e018f0c84c91-0065bb722c-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:56.883 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:56.884 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:56.884 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:56.884 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:56.884 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:56.884 DEBUG:boto:Method: GET 2024-02-01T10:27:56.884 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:56.884 DEBUG:boto:Data: 2024-02-01T10:27:56.885 DEBUG:boto:Headers: {} 2024-02-01T10:27:56.885 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:56.885 DEBUG:boto:Port: 8000 2024-02-01T10:27:56.885 DEBUG:boto:Params: {} 2024-02-01T10:27:56.885 DEBUG:boto:Token: None 2024-02-01T10:27:56.885 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:56 GMT /vclfcn-19/ 2024-02-01T10:27:56.885 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:pyGs672SFMK75mNjBfDPLe76hag= 2024-02-01T10:27:56.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:pyGs672SFMK75mNjBfDPLe76hag=', 'Content-Length': '0'} 2024-02-01T10:27:56.891 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e4cfb2ddec94c95c-0065bb722c-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:56.891 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:56.892 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:56.892 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:56.892 DEBUG:boto:Method: HEAD 2024-02-01T10:27:56.892 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:56.893 DEBUG:boto:Data: 2024-02-01T10:27:56.893 DEBUG:boto:Headers: {} 2024-02-01T10:27:56.893 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:56.893 DEBUG:boto:Port: 8001 2024-02-01T10:27:56.893 DEBUG:boto:Params: {} 2024-02-01T10:27:56.893 DEBUG:boto:Token: None 2024-02-01T10:27:56.893 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:56 GMT /vclfcn-19/obj 2024-02-01T10:27:56.894 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nPTPE31IXHfolj4uMohn+L+vJAM= 2024-02-01T10:27:56.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nPTPE31IXHfolj4uMohn+L+vJAM=', 'Content-Length': '0'} 2024-02-01T10:27:56.899 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aa94e503003d7807-0065bb722c-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:56.900 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:56.900 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:56.900 DEBUG:boto:Method: HEAD 2024-02-01T10:27:56.900 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:56.900 DEBUG:boto:Data: 2024-02-01T10:27:56.901 DEBUG:boto:Headers: {} 2024-02-01T10:27:56.901 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:56.901 DEBUG:boto:Port: 8000 2024-02-01T10:27:56.901 DEBUG:boto:Params: {} 2024-02-01T10:27:56.901 DEBUG:boto:Token: None 2024-02-01T10:27:56.901 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:56 GMT /vclfcn-19/obj 2024-02-01T10:27:56.901 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nPTPE31IXHfolj4uMohn+L+vJAM= 2024-02-01T10:27:56.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nPTPE31IXHfolj4uMohn+L+vJAM=', 'Content-Length': '0'} 2024-02-01T10:27:56.907 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004d9320ccb65176c7-0065bb722c-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:56.907 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:56.907 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:56.907 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:56.907 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:56.908 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:56.908 DEBUG:boto:Method: GET 2024-02-01T10:27:56.908 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:56.908 DEBUG:boto:Data: 2024-02-01T10:27:56.908 DEBUG:boto:Headers: {} 2024-02-01T10:27:56.908 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:56.908 DEBUG:boto:Port: 8001 2024-02-01T10:27:56.908 DEBUG:boto:Params: {} 2024-02-01T10:27:56.908 DEBUG:boto:Token: None 2024-02-01T10:27:56.909 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:56 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:56.909 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:2aSf7utdTDMKaQFtDMi4AnqmFQg= 2024-02-01T10:27:56.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2aSf7utdTDMKaQFtDMi4AnqmFQg=', 'Content-Length': '0'} 2024-02-01T10:27:56.911 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ff38c4b65316182e-0065bb722c-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:56.952 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:56.952 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:56.953 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:56.953 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:56.953 DEBUG:boto:Method: GET 2024-02-01T10:27:56.953 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:56.953 DEBUG:boto:Data: 2024-02-01T10:27:56.953 DEBUG:boto:Headers: {} 2024-02-01T10:27:56.954 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:56.954 DEBUG:boto:Port: 8000 2024-02-01T10:27:56.954 DEBUG:boto:Params: {} 2024-02-01T10:27:56.954 DEBUG:boto:Token: None 2024-02-01T10:27:56.954 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:56 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:56.954 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:2aSf7utdTDMKaQFtDMi4AnqmFQg= 2024-02-01T10:27:56.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2aSf7utdTDMKaQFtDMi4AnqmFQg=', 'Content-Length': '0'} 2024-02-01T10:27:56.957 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bfb8901b5d1a8c4b-0065bb722c-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:56.957 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:56.958 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:56.958 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:56.958 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:56.958 DEBUG:boto:Method: GET 2024-02-01T10:27:56.958 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:56.958 DEBUG:boto:Data: 2024-02-01T10:27:56.958 DEBUG:boto:Headers: {} 2024-02-01T10:27:56.958 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:56.958 DEBUG:boto:Port: 8001 2024-02-01T10:27:56.958 DEBUG:boto:Params: {} 2024-02-01T10:27:56.958 DEBUG:boto:Token: None 2024-02-01T10:27:56.959 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:56 GMT /vclfcn-19/ 2024-02-01T10:27:56.959 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:pyGs672SFMK75mNjBfDPLe76hag= 2024-02-01T10:27:56.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:pyGs672SFMK75mNjBfDPLe76hag=', 'Content-Length': '0'} 2024-02-01T10:27:56.964 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000022b4eb301ca632c5-0065bb722c-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:56.964 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:56.964 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:56.965 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:56.965 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:56.965 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:56.965 DEBUG:boto:Method: GET 2024-02-01T10:27:56.965 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:56.965 DEBUG:boto:Data: 2024-02-01T10:27:56.965 DEBUG:boto:Headers: {} 2024-02-01T10:27:56.965 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:56.965 DEBUG:boto:Port: 8000 2024-02-01T10:27:56.965 DEBUG:boto:Params: {} 2024-02-01T10:27:56.965 DEBUG:boto:Token: None 2024-02-01T10:27:56.966 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:56 GMT /vclfcn-19/ 2024-02-01T10:27:56.966 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:pyGs672SFMK75mNjBfDPLe76hag= 2024-02-01T10:27:56.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:pyGs672SFMK75mNjBfDPLe76hag=', 'Content-Length': '0'} 2024-02-01T10:27:56.972 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b2855b51d84a2002-0065bb722c-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:56.972 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:56.972 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:56.973 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:56.973 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:56.973 DEBUG:boto:path=/vclfcn-19/obj?versionId=0KR6m6ymTGitKe6vabQyVENAnBL669w 2024-02-01T10:27:56.973 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=0KR6m6ymTGitKe6vabQyVENAnBL669w 2024-02-01T10:27:56.973 DEBUG:boto:Method: GET 2024-02-01T10:27:56.973 DEBUG:boto:Path: /vclfcn-19/obj?versionId=0KR6m6ymTGitKe6vabQyVENAnBL669w 2024-02-01T10:27:56.973 DEBUG:boto:Data: 2024-02-01T10:27:56.973 DEBUG:boto:Headers: {} 2024-02-01T10:27:56.973 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:56.973 DEBUG:boto:Port: 8001 2024-02-01T10:27:56.974 DEBUG:boto:Params: {} 2024-02-01T10:27:56.974 DEBUG:boto:Token: None 2024-02-01T10:27:56.974 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:56 GMT /vclfcn-19/obj?versionId=0KR6m6ymTGitKe6vabQyVENAnBL669w 2024-02-01T10:27:56.974 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:xDlwCTU88unXnA7NGBPQMjvks4g= 2024-02-01T10:27:56.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:xDlwCTU88unXnA7NGBPQMjvks4g=', 'Content-Length': '0'} 2024-02-01T10:27:56.977 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '0KR6m6ymTGitKe6vabQyVENAnBL669w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000151db631ced297c3-0065bb722c-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:56.977 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:56.977 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:56.977 DEBUG:boto:path=/vclfcn-19/obj?versionId=0KR6m6ymTGitKe6vabQyVENAnBL669w 2024-02-01T10:27:56.977 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=0KR6m6ymTGitKe6vabQyVENAnBL669w 2024-02-01T10:27:56.977 DEBUG:boto:Method: GET 2024-02-01T10:27:56.978 DEBUG:boto:Path: /vclfcn-19/obj?versionId=0KR6m6ymTGitKe6vabQyVENAnBL669w 2024-02-01T10:27:56.978 DEBUG:boto:Data: 2024-02-01T10:27:56.978 DEBUG:boto:Headers: {} 2024-02-01T10:27:56.978 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:56.978 DEBUG:boto:Port: 8000 2024-02-01T10:27:56.978 DEBUG:boto:Params: {} 2024-02-01T10:27:56.978 DEBUG:boto:Token: None 2024-02-01T10:27:56.978 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:56 GMT /vclfcn-19/obj?versionId=0KR6m6ymTGitKe6vabQyVENAnBL669w 2024-02-01T10:27:56.978 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:xDlwCTU88unXnA7NGBPQMjvks4g= 2024-02-01T10:27:56.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:xDlwCTU88unXnA7NGBPQMjvks4g=', 'Content-Length': '0'} 2024-02-01T10:27:56.981 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '0KR6m6ymTGitKe6vabQyVENAnBL669w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000186d0ae8dc1abd0e-0065bb722c-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:56.981 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:56.981 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:56.982 DEBUG:boto:path=/vclfcn-19/obj?versionId=0KR6m6ymTGitKe6vabQyVENAnBL669w 2024-02-01T10:27:56.982 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=0KR6m6ymTGitKe6vabQyVENAnBL669w 2024-02-01T10:27:56.982 DEBUG:boto:Method: HEAD 2024-02-01T10:27:56.982 DEBUG:boto:Path: /vclfcn-19/obj?versionId=0KR6m6ymTGitKe6vabQyVENAnBL669w 2024-02-01T10:27:56.982 DEBUG:boto:Data: 2024-02-01T10:27:56.982 DEBUG:boto:Headers: {} 2024-02-01T10:27:56.982 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:56.982 DEBUG:boto:Port: 8001 2024-02-01T10:27:56.982 DEBUG:boto:Params: {} 2024-02-01T10:27:56.982 DEBUG:boto:Token: None 2024-02-01T10:27:56.983 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:56 GMT /vclfcn-19/obj?versionId=0KR6m6ymTGitKe6vabQyVENAnBL669w 2024-02-01T10:27:56.983 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:SO64TaDq1A5L4w9j0KWfA2Rl5lY= 2024-02-01T10:27:56.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:SO64TaDq1A5L4w9j0KWfA2Rl5lY=', 'Content-Length': '0'} 2024-02-01T10:27:56.985 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '0KR6m6ymTGitKe6vabQyVENAnBL669w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000030b6f87536086615-0065bb722c-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:56.986 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:56.986 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:56.986 DEBUG:boto:path=/vclfcn-19/obj?versionId=0KR6m6ymTGitKe6vabQyVENAnBL669w 2024-02-01T10:27:56.986 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=0KR6m6ymTGitKe6vabQyVENAnBL669w 2024-02-01T10:27:56.986 DEBUG:boto:Method: HEAD 2024-02-01T10:27:56.986 DEBUG:boto:Path: /vclfcn-19/obj?versionId=0KR6m6ymTGitKe6vabQyVENAnBL669w 2024-02-01T10:27:56.986 DEBUG:boto:Data: 2024-02-01T10:27:56.986 DEBUG:boto:Headers: {} 2024-02-01T10:27:56.986 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:56.986 DEBUG:boto:Port: 8000 2024-02-01T10:27:56.987 DEBUG:boto:Params: {} 2024-02-01T10:27:56.987 DEBUG:boto:Token: None 2024-02-01T10:27:56.987 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:56 GMT /vclfcn-19/obj?versionId=0KR6m6ymTGitKe6vabQyVENAnBL669w 2024-02-01T10:27:56.987 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:SO64TaDq1A5L4w9j0KWfA2Rl5lY= 2024-02-01T10:27:56.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:SO64TaDq1A5L4w9j0KWfA2Rl5lY=', 'Content-Length': '0'} 2024-02-01T10:27:56.990 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '0KR6m6ymTGitKe6vabQyVENAnBL669w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002bf226df1de409d5-0065bb722c-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:56.990 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:56.990 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:56.990 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:56.990 DEBUG:boto:path=/vclfcn-19/obj?versionId=0KR6m6ymTGitKe6vabQyVENAnBL669w 2024-02-01T10:27:56.990 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=0KR6m6ymTGitKe6vabQyVENAnBL669w 2024-02-01T10:27:56.990 DEBUG:boto:Method: GET 2024-02-01T10:27:56.990 DEBUG:boto:Path: /vclfcn-19/obj?versionId=0KR6m6ymTGitKe6vabQyVENAnBL669w 2024-02-01T10:27:56.990 DEBUG:boto:Data: 2024-02-01T10:27:56.991 DEBUG:boto:Headers: {} 2024-02-01T10:27:56.991 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:56.991 DEBUG:boto:Port: 8001 2024-02-01T10:27:56.991 DEBUG:boto:Params: {} 2024-02-01T10:27:56.991 DEBUG:boto:Token: None 2024-02-01T10:27:56.991 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:56 GMT /vclfcn-19/obj?versionId=0KR6m6ymTGitKe6vabQyVENAnBL669w 2024-02-01T10:27:56.991 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:xDlwCTU88unXnA7NGBPQMjvks4g= 2024-02-01T10:27:56.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:xDlwCTU88unXnA7NGBPQMjvks4g=', 'Content-Length': '0'} 2024-02-01T10:27:56.994 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '0KR6m6ymTGitKe6vabQyVENAnBL669w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bd25fd6a1dab8f7e-0065bb722d-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:57.036 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:57.036 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:57.036 DEBUG:boto:path=/vclfcn-19/obj?versionId=0KR6m6ymTGitKe6vabQyVENAnBL669w 2024-02-01T10:27:57.036 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=0KR6m6ymTGitKe6vabQyVENAnBL669w 2024-02-01T10:27:57.036 DEBUG:boto:Method: GET 2024-02-01T10:27:57.037 DEBUG:boto:Path: /vclfcn-19/obj?versionId=0KR6m6ymTGitKe6vabQyVENAnBL669w 2024-02-01T10:27:57.037 DEBUG:boto:Data: 2024-02-01T10:27:57.037 DEBUG:boto:Headers: {} 2024-02-01T10:27:57.037 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:57.037 DEBUG:boto:Port: 8000 2024-02-01T10:27:57.037 DEBUG:boto:Params: {} 2024-02-01T10:27:57.037 DEBUG:boto:Token: None 2024-02-01T10:27:57.037 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:57 GMT /vclfcn-19/obj?versionId=0KR6m6ymTGitKe6vabQyVENAnBL669w 2024-02-01T10:27:57.037 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:KbMqetWLCRUjk6Q+5cHPfx9JKf0= 2024-02-01T10:27:57.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:KbMqetWLCRUjk6Q+5cHPfx9JKf0=', 'Content-Length': '0'} 2024-02-01T10:27:57.040 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '0KR6m6ymTGitKe6vabQyVENAnBL669w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003c2f625aa507c056-0065bb722d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:57.041 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:57.041 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:57.041 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:57.041 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:57.041 DEBUG:boto:Method: GET 2024-02-01T10:27:57.041 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:57.041 DEBUG:boto:Data: 2024-02-01T10:27:57.041 DEBUG:boto:Headers: {} 2024-02-01T10:27:57.041 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:57.041 DEBUG:boto:Port: 8001 2024-02-01T10:27:57.041 DEBUG:boto:Params: {} 2024-02-01T10:27:57.042 DEBUG:boto:Token: None 2024-02-01T10:27:57.042 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:57 GMT /vclfcn-19/ 2024-02-01T10:27:57.042 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:iRx4h85JUwm0irM6Iu/3cPjaFGs= 2024-02-01T10:27:57.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iRx4h85JUwm0irM6Iu/3cPjaFGs=', 'Content-Length': '0'} 2024-02-01T10:27:57.047 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a5c464056a1ba05-0065bb722d-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:57.047 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:57.048 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:57.048 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:57.048 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:57.048 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:57.048 DEBUG:boto:Method: GET 2024-02-01T10:27:57.048 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:57.048 DEBUG:boto:Data: 2024-02-01T10:27:57.049 DEBUG:boto:Headers: {} 2024-02-01T10:27:57.049 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:57.049 DEBUG:boto:Port: 8000 2024-02-01T10:27:57.049 DEBUG:boto:Params: {} 2024-02-01T10:27:57.049 DEBUG:boto:Token: None 2024-02-01T10:27:57.049 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:57 GMT /vclfcn-19/ 2024-02-01T10:27:57.049 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:iRx4h85JUwm0irM6Iu/3cPjaFGs= 2024-02-01T10:27:57.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iRx4h85JUwm0irM6Iu/3cPjaFGs=', 'Content-Length': '0'} 2024-02-01T10:27:57.054 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c4a925d56748632-0065bb722d-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:57.054 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:57.055 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:57.055 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:57.055 DEBUG:boto:Method: HEAD 2024-02-01T10:27:57.055 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:57.055 DEBUG:boto:Data: 2024-02-01T10:27:57.055 DEBUG:boto:Headers: {} 2024-02-01T10:27:57.055 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:57.055 DEBUG:boto:Port: 8001 2024-02-01T10:27:57.055 DEBUG:boto:Params: {} 2024-02-01T10:27:57.056 DEBUG:boto:Token: None 2024-02-01T10:27:57.056 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:57 GMT /vclfcn-19/obj 2024-02-01T10:27:57.056 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:IMGogpnpS5JU7tzNsdNZn1YAlKs= 2024-02-01T10:27:57.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:IMGogpnpS5JU7tzNsdNZn1YAlKs=', 'Content-Length': '0'} 2024-02-01T10:27:57.061 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000005f46fedcdf0318-0065bb722d-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:57.062 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:57.062 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:57.062 DEBUG:boto:Method: HEAD 2024-02-01T10:27:57.062 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:57.062 DEBUG:boto:Data: 2024-02-01T10:27:57.062 DEBUG:boto:Headers: {} 2024-02-01T10:27:57.062 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:57.062 DEBUG:boto:Port: 8000 2024-02-01T10:27:57.062 DEBUG:boto:Params: {} 2024-02-01T10:27:57.063 DEBUG:boto:Token: None 2024-02-01T10:27:57.063 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:57 GMT /vclfcn-19/obj 2024-02-01T10:27:57.063 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:IMGogpnpS5JU7tzNsdNZn1YAlKs= 2024-02-01T10:27:57.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:IMGogpnpS5JU7tzNsdNZn1YAlKs=', 'Content-Length': '0'} 2024-02-01T10:27:57.068 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000079ce13c929b399ee-0065bb722d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:57.069 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:57.069 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:57.069 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:57.069 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:57.069 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:57.069 DEBUG:boto:Method: GET 2024-02-01T10:27:57.069 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:57.069 DEBUG:boto:Data: 2024-02-01T10:27:57.069 DEBUG:boto:Headers: {} 2024-02-01T10:27:57.070 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:57.070 DEBUG:boto:Port: 8001 2024-02-01T10:27:57.070 DEBUG:boto:Params: {} 2024-02-01T10:27:57.070 DEBUG:boto:Token: None 2024-02-01T10:27:57.070 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:57 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:57.070 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GJShm2mcF2m6h/GMaVxLxuweY4E= 2024-02-01T10:27:57.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GJShm2mcF2m6h/GMaVxLxuweY4E=', 'Content-Length': '0'} 2024-02-01T10:27:57.073 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ce54c3c984b418e0-0065bb722d-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:57.116 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:57.116 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:57.117 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:57.117 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:57.117 DEBUG:boto:Method: GET 2024-02-01T10:27:57.117 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:57.117 DEBUG:boto:Data: 2024-02-01T10:27:57.117 DEBUG:boto:Headers: {} 2024-02-01T10:27:57.117 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:57.117 DEBUG:boto:Port: 8000 2024-02-01T10:27:57.117 DEBUG:boto:Params: {} 2024-02-01T10:27:57.118 DEBUG:boto:Token: None 2024-02-01T10:27:57.118 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:57 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:57.118 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GJShm2mcF2m6h/GMaVxLxuweY4E= 2024-02-01T10:27:57.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GJShm2mcF2m6h/GMaVxLxuweY4E=', 'Content-Length': '0'} 2024-02-01T10:27:57.121 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002a6ece8477b62e9f-0065bb722d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:57.121 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:57.122 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:57.122 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:57.122 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:57.122 DEBUG:boto:Method: GET 2024-02-01T10:27:57.122 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:57.122 DEBUG:boto:Data: 2024-02-01T10:27:57.122 DEBUG:boto:Headers: {} 2024-02-01T10:27:57.122 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:57.122 DEBUG:boto:Port: 8001 2024-02-01T10:27:57.122 DEBUG:boto:Params: {} 2024-02-01T10:27:57.123 DEBUG:boto:Token: None 2024-02-01T10:27:57.123 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:57 GMT /vclfcn-19/ 2024-02-01T10:27:57.123 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:iRx4h85JUwm0irM6Iu/3cPjaFGs= 2024-02-01T10:27:57.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iRx4h85JUwm0irM6Iu/3cPjaFGs=', 'Content-Length': '0'} 2024-02-01T10:27:57.129 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9fa1622dba1aec8-0065bb722d-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:57.129 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:57.129 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:57.130 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:57.130 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:57.130 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:57.130 DEBUG:boto:Method: GET 2024-02-01T10:27:57.130 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:57.130 DEBUG:boto:Data: 2024-02-01T10:27:57.130 DEBUG:boto:Headers: {} 2024-02-01T10:27:57.130 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:57.131 DEBUG:boto:Port: 8000 2024-02-01T10:27:57.131 DEBUG:boto:Params: {} 2024-02-01T10:27:57.131 DEBUG:boto:Token: None 2024-02-01T10:27:57.131 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:57 GMT /vclfcn-19/ 2024-02-01T10:27:57.131 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:iRx4h85JUwm0irM6Iu/3cPjaFGs= 2024-02-01T10:27:57.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iRx4h85JUwm0irM6Iu/3cPjaFGs=', 'Content-Length': '0'} 2024-02-01T10:27:57.137 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad5c1a5abe1eacab-0065bb722d-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:57.184 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:57.185 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:57.185 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:57.185 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:57.185 DEBUG:boto:path=/vclfcn-19/obj?versionId=2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E 2024-02-01T10:27:57.185 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E 2024-02-01T10:27:57.185 DEBUG:boto:Method: GET 2024-02-01T10:27:57.185 DEBUG:boto:Path: /vclfcn-19/obj?versionId=2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E 2024-02-01T10:27:57.185 DEBUG:boto:Data: 2024-02-01T10:27:57.185 DEBUG:boto:Headers: {} 2024-02-01T10:27:57.186 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:57.186 DEBUG:boto:Port: 8001 2024-02-01T10:27:57.186 DEBUG:boto:Params: {} 2024-02-01T10:27:57.186 DEBUG:boto:Token: None 2024-02-01T10:27:57.186 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:57 GMT /vclfcn-19/obj?versionId=2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E 2024-02-01T10:27:57.186 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:L5pEo/h3Tmgcxb2vXSUCoZlIU4Q= 2024-02-01T10:27:57.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:L5pEo/h3Tmgcxb2vXSUCoZlIU4Q=', 'Content-Length': '0'} 2024-02-01T10:27:57.189 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000415abdaed34c6992-0065bb722d-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:57.190 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:57.190 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:57.190 DEBUG:boto:path=/vclfcn-19/obj?versionId=2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E 2024-02-01T10:27:57.190 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E 2024-02-01T10:27:57.190 DEBUG:boto:Method: GET 2024-02-01T10:27:57.190 DEBUG:boto:Path: /vclfcn-19/obj?versionId=2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E 2024-02-01T10:27:57.190 DEBUG:boto:Data: 2024-02-01T10:27:57.190 DEBUG:boto:Headers: {} 2024-02-01T10:27:57.190 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:57.190 DEBUG:boto:Port: 8000 2024-02-01T10:27:57.190 DEBUG:boto:Params: {} 2024-02-01T10:27:57.191 DEBUG:boto:Token: None 2024-02-01T10:27:57.191 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:57 GMT /vclfcn-19/obj?versionId=2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E 2024-02-01T10:27:57.191 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:L5pEo/h3Tmgcxb2vXSUCoZlIU4Q= 2024-02-01T10:27:57.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:L5pEo/h3Tmgcxb2vXSUCoZlIU4Q=', 'Content-Length': '0'} 2024-02-01T10:27:57.194 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007026b400394dc347-0065bb722d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:57.236 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:57.236 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:57.236 DEBUG:boto:path=/vclfcn-19/obj?versionId=2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E 2024-02-01T10:27:57.236 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E 2024-02-01T10:27:57.236 DEBUG:boto:Method: HEAD 2024-02-01T10:27:57.237 DEBUG:boto:Path: /vclfcn-19/obj?versionId=2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E 2024-02-01T10:27:57.237 DEBUG:boto:Data: 2024-02-01T10:27:57.237 DEBUG:boto:Headers: {} 2024-02-01T10:27:57.237 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:57.237 DEBUG:boto:Port: 8001 2024-02-01T10:27:57.237 DEBUG:boto:Params: {} 2024-02-01T10:27:57.237 DEBUG:boto:Token: None 2024-02-01T10:27:57.237 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:57 GMT /vclfcn-19/obj?versionId=2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E 2024-02-01T10:27:57.237 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:L97jZZtSSMcdNGBNmMpwgwQKJN4= 2024-02-01T10:27:57.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:L97jZZtSSMcdNGBNmMpwgwQKJN4=', 'Content-Length': '0'} 2024-02-01T10:27:57.240 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000995e4087104624e1-0065bb722d-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:57.240 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:57.241 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:57.241 DEBUG:boto:path=/vclfcn-19/obj?versionId=2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E 2024-02-01T10:27:57.241 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E 2024-02-01T10:27:57.241 DEBUG:boto:Method: HEAD 2024-02-01T10:27:57.241 DEBUG:boto:Path: /vclfcn-19/obj?versionId=2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E 2024-02-01T10:27:57.241 DEBUG:boto:Data: 2024-02-01T10:27:57.241 DEBUG:boto:Headers: {} 2024-02-01T10:27:57.241 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:57.241 DEBUG:boto:Port: 8000 2024-02-01T10:27:57.241 DEBUG:boto:Params: {} 2024-02-01T10:27:57.242 DEBUG:boto:Token: None 2024-02-01T10:27:57.242 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:57 GMT /vclfcn-19/obj?versionId=2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E 2024-02-01T10:27:57.242 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:L97jZZtSSMcdNGBNmMpwgwQKJN4= 2024-02-01T10:27:57.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:L97jZZtSSMcdNGBNmMpwgwQKJN4=', 'Content-Length': '0'} 2024-02-01T10:27:57.245 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a826fd460ee2c1e7-0065bb722d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:57.245 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:57.245 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:57.245 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:57.245 DEBUG:boto:path=/vclfcn-19/obj?versionId=2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E 2024-02-01T10:27:57.245 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E 2024-02-01T10:27:57.246 DEBUG:boto:Method: GET 2024-02-01T10:27:57.246 DEBUG:boto:Path: /vclfcn-19/obj?versionId=2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E 2024-02-01T10:27:57.246 DEBUG:boto:Data: 2024-02-01T10:27:57.246 DEBUG:boto:Headers: {} 2024-02-01T10:27:57.246 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:57.246 DEBUG:boto:Port: 8001 2024-02-01T10:27:57.246 DEBUG:boto:Params: {} 2024-02-01T10:27:57.246 DEBUG:boto:Token: None 2024-02-01T10:27:57.246 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:57 GMT /vclfcn-19/obj?versionId=2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E 2024-02-01T10:27:57.247 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:L5pEo/h3Tmgcxb2vXSUCoZlIU4Q= 2024-02-01T10:27:57.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:L5pEo/h3Tmgcxb2vXSUCoZlIU4Q=', 'Content-Length': '0'} 2024-02-01T10:27:57.249 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ef24994edb0b8545-0065bb722d-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:57.296 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:57.296 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:57.296 DEBUG:boto:path=/vclfcn-19/obj?versionId=2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E 2024-02-01T10:27:57.296 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E 2024-02-01T10:27:57.297 DEBUG:boto:Method: GET 2024-02-01T10:27:57.297 DEBUG:boto:Path: /vclfcn-19/obj?versionId=2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E 2024-02-01T10:27:57.297 DEBUG:boto:Data: 2024-02-01T10:27:57.297 DEBUG:boto:Headers: {} 2024-02-01T10:27:57.297 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:57.297 DEBUG:boto:Port: 8000 2024-02-01T10:27:57.297 DEBUG:boto:Params: {} 2024-02-01T10:27:57.297 DEBUG:boto:Token: None 2024-02-01T10:27:57.297 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:57 GMT /vclfcn-19/obj?versionId=2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E 2024-02-01T10:27:57.297 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:L5pEo/h3Tmgcxb2vXSUCoZlIU4Q= 2024-02-01T10:27:57.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:L5pEo/h3Tmgcxb2vXSUCoZlIU4Q=', 'Content-Length': '0'} 2024-02-01T10:27:57.300 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '2Hfm7x.sO5cadpLfjJNuhN.TAkjS54E'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e18c4c33fd905c85-0065bb722d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:57.348 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:57.348 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:57.348 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:57.348 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:57.349 DEBUG:boto:Method: GET 2024-02-01T10:27:57.349 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:57.349 DEBUG:boto:Data: 2024-02-01T10:27:57.349 DEBUG:boto:Headers: {} 2024-02-01T10:27:57.349 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:57.349 DEBUG:boto:Port: 8001 2024-02-01T10:27:57.349 DEBUG:boto:Params: {} 2024-02-01T10:27:57.349 DEBUG:boto:Token: None 2024-02-01T10:27:57.349 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:57 GMT /vclfcn-19/ 2024-02-01T10:27:57.350 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:iRx4h85JUwm0irM6Iu/3cPjaFGs= 2024-02-01T10:27:57.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iRx4h85JUwm0irM6Iu/3cPjaFGs=', 'Content-Length': '0'} 2024-02-01T10:27:57.355 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ddfc8734644832f-0065bb722d-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:57.396 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:57.397 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:57.399 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:57.399 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:57.399 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:57.399 DEBUG:boto:Method: GET 2024-02-01T10:27:57.400 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:57.400 DEBUG:boto:Data: 2024-02-01T10:27:57.400 DEBUG:boto:Headers: {} 2024-02-01T10:27:57.400 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:57.400 DEBUG:boto:Port: 8000 2024-02-01T10:27:57.400 DEBUG:boto:Params: {} 2024-02-01T10:27:57.400 DEBUG:boto:Token: None 2024-02-01T10:27:57.400 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:57 GMT /vclfcn-19/ 2024-02-01T10:27:57.401 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:iRx4h85JUwm0irM6Iu/3cPjaFGs= 2024-02-01T10:27:57.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iRx4h85JUwm0irM6Iu/3cPjaFGs=', 'Content-Length': '0'} 2024-02-01T10:27:57.406 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f5caf50be81e4b4-0065bb722d-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:57.406 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:57.407 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:57.407 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:57.407 DEBUG:boto:Method: HEAD 2024-02-01T10:27:57.407 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:57.407 DEBUG:boto:Data: 2024-02-01T10:27:57.407 DEBUG:boto:Headers: {} 2024-02-01T10:27:57.407 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:57.407 DEBUG:boto:Port: 8001 2024-02-01T10:27:57.408 DEBUG:boto:Params: {} 2024-02-01T10:27:57.408 DEBUG:boto:Token: None 2024-02-01T10:27:57.408 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:57 GMT /vclfcn-19/obj 2024-02-01T10:27:57.408 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:IMGogpnpS5JU7tzNsdNZn1YAlKs= 2024-02-01T10:27:57.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:IMGogpnpS5JU7tzNsdNZn1YAlKs=', 'Content-Length': '0'} 2024-02-01T10:27:57.414 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008985e8a746c50de0-0065bb722d-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:57.415 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:57.415 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:57.415 DEBUG:boto:Method: HEAD 2024-02-01T10:27:57.415 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:57.415 DEBUG:boto:Data: 2024-02-01T10:27:57.415 DEBUG:boto:Headers: {} 2024-02-01T10:27:57.415 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:57.415 DEBUG:boto:Port: 8000 2024-02-01T10:27:57.416 DEBUG:boto:Params: {} 2024-02-01T10:27:57.417 DEBUG:boto:Token: None 2024-02-01T10:27:57.417 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:57 GMT /vclfcn-19/obj 2024-02-01T10:27:57.417 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:IMGogpnpS5JU7tzNsdNZn1YAlKs= 2024-02-01T10:27:57.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:IMGogpnpS5JU7tzNsdNZn1YAlKs=', 'Content-Length': '0'} 2024-02-01T10:27:57.422 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a2bbb9721a2ba510-0065bb722d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:57.423 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:57.423 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:57.423 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:57.423 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:57.423 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:57.423 DEBUG:boto:Method: GET 2024-02-01T10:27:57.423 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:57.423 DEBUG:boto:Data: 2024-02-01T10:27:57.424 DEBUG:boto:Headers: {} 2024-02-01T10:27:57.424 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:57.424 DEBUG:boto:Port: 8001 2024-02-01T10:27:57.424 DEBUG:boto:Params: {} 2024-02-01T10:27:57.424 DEBUG:boto:Token: None 2024-02-01T10:27:57.424 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:57 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:57.424 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GJShm2mcF2m6h/GMaVxLxuweY4E= 2024-02-01T10:27:57.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GJShm2mcF2m6h/GMaVxLxuweY4E=', 'Content-Length': '0'} 2024-02-01T10:27:57.427 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000022cd2c370cef93e4-0065bb722d-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:57.468 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:57.468 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:57.468 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:57.468 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:57.469 DEBUG:boto:Method: GET 2024-02-01T10:27:57.469 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:57.469 DEBUG:boto:Data: 2024-02-01T10:27:57.469 DEBUG:boto:Headers: {} 2024-02-01T10:27:57.469 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:57.469 DEBUG:boto:Port: 8000 2024-02-01T10:27:57.469 DEBUG:boto:Params: {} 2024-02-01T10:27:57.469 DEBUG:boto:Token: None 2024-02-01T10:27:57.469 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:57 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:57.470 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GJShm2mcF2m6h/GMaVxLxuweY4E= 2024-02-01T10:27:57.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GJShm2mcF2m6h/GMaVxLxuweY4E=', 'Content-Length': '0'} 2024-02-01T10:27:57.473 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004ae0b79ea2ec82b7-0065bb722d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:57.473 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:57.473 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:57.473 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:57.473 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:57.473 DEBUG:boto:Method: GET 2024-02-01T10:27:57.473 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:57.474 DEBUG:boto:Data: 2024-02-01T10:27:57.474 DEBUG:boto:Headers: {} 2024-02-01T10:27:57.474 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:57.474 DEBUG:boto:Port: 8001 2024-02-01T10:27:57.474 DEBUG:boto:Params: {} 2024-02-01T10:27:57.474 DEBUG:boto:Token: None 2024-02-01T10:27:57.474 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:57 GMT /vclfcn-19/ 2024-02-01T10:27:57.474 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:iRx4h85JUwm0irM6Iu/3cPjaFGs= 2024-02-01T10:27:57.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iRx4h85JUwm0irM6Iu/3cPjaFGs=', 'Content-Length': '0'} 2024-02-01T10:27:57.479 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d2c66382724d568-0065bb722d-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:57.520 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:57.521 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:57.521 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:57.521 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:57.521 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:57.521 DEBUG:boto:Method: GET 2024-02-01T10:27:57.521 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:57.521 DEBUG:boto:Data: 2024-02-01T10:27:57.521 DEBUG:boto:Headers: {} 2024-02-01T10:27:57.521 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:57.521 DEBUG:boto:Port: 8000 2024-02-01T10:27:57.521 DEBUG:boto:Params: {} 2024-02-01T10:27:57.522 DEBUG:boto:Token: None 2024-02-01T10:27:57.522 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:57 GMT /vclfcn-19/ 2024-02-01T10:27:57.522 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:iRx4h85JUwm0irM6Iu/3cPjaFGs= 2024-02-01T10:27:57.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iRx4h85JUwm0irM6Iu/3cPjaFGs=', 'Content-Length': '0'} 2024-02-01T10:27:57.528 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000109b5106475c1210-0065bb722d-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:57.528 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:57.529 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:57.529 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:57.529 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:57.529 DEBUG:boto:path=/vclfcn-19/obj?versionId=7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6 2024-02-01T10:27:57.529 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6 2024-02-01T10:27:57.529 DEBUG:boto:Method: GET 2024-02-01T10:27:57.529 DEBUG:boto:Path: /vclfcn-19/obj?versionId=7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6 2024-02-01T10:27:57.529 DEBUG:boto:Data: 2024-02-01T10:27:57.529 DEBUG:boto:Headers: {} 2024-02-01T10:27:57.529 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:57.530 DEBUG:boto:Port: 8001 2024-02-01T10:27:57.530 DEBUG:boto:Params: {} 2024-02-01T10:27:57.530 DEBUG:boto:Token: None 2024-02-01T10:27:57.530 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:57 GMT /vclfcn-19/obj?versionId=7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6 2024-02-01T10:27:57.530 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:iMNCq97Yfwb4ra3ssfaTF8rMxLc= 2024-02-01T10:27:57.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iMNCq97Yfwb4ra3ssfaTF8rMxLc=', 'Content-Length': '0'} 2024-02-01T10:27:57.533 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000c00b7fc565bb273-0065bb722d-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:57.533 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:57.533 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:57.534 DEBUG:boto:path=/vclfcn-19/obj?versionId=7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6 2024-02-01T10:27:57.534 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6 2024-02-01T10:27:57.534 DEBUG:boto:Method: GET 2024-02-01T10:27:57.534 DEBUG:boto:Path: /vclfcn-19/obj?versionId=7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6 2024-02-01T10:27:57.534 DEBUG:boto:Data: 2024-02-01T10:27:57.534 DEBUG:boto:Headers: {} 2024-02-01T10:27:57.534 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:57.534 DEBUG:boto:Port: 8000 2024-02-01T10:27:57.534 DEBUG:boto:Params: {} 2024-02-01T10:27:57.534 DEBUG:boto:Token: None 2024-02-01T10:27:57.535 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:57 GMT /vclfcn-19/obj?versionId=7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6 2024-02-01T10:27:57.535 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:iMNCq97Yfwb4ra3ssfaTF8rMxLc= 2024-02-01T10:27:57.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iMNCq97Yfwb4ra3ssfaTF8rMxLc=', 'Content-Length': '0'} 2024-02-01T10:27:57.538 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000619d55243a0db4c6-0065bb722d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:57.580 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:57.580 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:57.580 DEBUG:boto:path=/vclfcn-19/obj?versionId=7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6 2024-02-01T10:27:57.581 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6 2024-02-01T10:27:57.581 DEBUG:boto:Method: HEAD 2024-02-01T10:27:57.581 DEBUG:boto:Path: /vclfcn-19/obj?versionId=7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6 2024-02-01T10:27:57.581 DEBUG:boto:Data: 2024-02-01T10:27:57.581 DEBUG:boto:Headers: {} 2024-02-01T10:27:57.581 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:57.581 DEBUG:boto:Port: 8001 2024-02-01T10:27:57.581 DEBUG:boto:Params: {} 2024-02-01T10:27:57.581 DEBUG:boto:Token: None 2024-02-01T10:27:57.582 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:57 GMT /vclfcn-19/obj?versionId=7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6 2024-02-01T10:27:57.582 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8n8OrayLeNuVwE7d7efdBjWhMCk= 2024-02-01T10:27:57.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8n8OrayLeNuVwE7d7efdBjWhMCk=', 'Content-Length': '0'} 2024-02-01T10:27:57.585 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004b1ac0fce67e93c4-0065bb722d-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:57.585 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:57.585 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:57.585 DEBUG:boto:path=/vclfcn-19/obj?versionId=7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6 2024-02-01T10:27:57.585 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6 2024-02-01T10:27:57.586 DEBUG:boto:Method: HEAD 2024-02-01T10:27:57.586 DEBUG:boto:Path: /vclfcn-19/obj?versionId=7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6 2024-02-01T10:27:57.586 DEBUG:boto:Data: 2024-02-01T10:27:57.586 DEBUG:boto:Headers: {} 2024-02-01T10:27:57.586 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:57.586 DEBUG:boto:Port: 8000 2024-02-01T10:27:57.586 DEBUG:boto:Params: {} 2024-02-01T10:27:57.586 DEBUG:boto:Token: None 2024-02-01T10:27:57.586 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:57 GMT /vclfcn-19/obj?versionId=7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6 2024-02-01T10:27:57.586 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8n8OrayLeNuVwE7d7efdBjWhMCk= 2024-02-01T10:27:57.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8n8OrayLeNuVwE7d7efdBjWhMCk=', 'Content-Length': '0'} 2024-02-01T10:27:57.589 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b26ac73236cf304c-0065bb722d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:57.590 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:57.590 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:57.590 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:57.590 DEBUG:boto:path=/vclfcn-19/obj?versionId=7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6 2024-02-01T10:27:57.590 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6 2024-02-01T10:27:57.590 DEBUG:boto:Method: GET 2024-02-01T10:27:57.590 DEBUG:boto:Path: /vclfcn-19/obj?versionId=7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6 2024-02-01T10:27:57.590 DEBUG:boto:Data: 2024-02-01T10:27:57.590 DEBUG:boto:Headers: {} 2024-02-01T10:27:57.590 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:57.591 DEBUG:boto:Port: 8001 2024-02-01T10:27:57.591 DEBUG:boto:Params: {} 2024-02-01T10:27:57.591 DEBUG:boto:Token: None 2024-02-01T10:27:57.591 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:57 GMT /vclfcn-19/obj?versionId=7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6 2024-02-01T10:27:57.591 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:iMNCq97Yfwb4ra3ssfaTF8rMxLc= 2024-02-01T10:27:57.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iMNCq97Yfwb4ra3ssfaTF8rMxLc=', 'Content-Length': '0'} 2024-02-01T10:27:57.594 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006fc004ea1006e482-0065bb722d-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:57.594 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:57.594 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:57.594 DEBUG:boto:path=/vclfcn-19/obj?versionId=7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6 2024-02-01T10:27:57.594 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6 2024-02-01T10:27:57.594 DEBUG:boto:Method: GET 2024-02-01T10:27:57.594 DEBUG:boto:Path: /vclfcn-19/obj?versionId=7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6 2024-02-01T10:27:57.594 DEBUG:boto:Data: 2024-02-01T10:27:57.595 DEBUG:boto:Headers: {} 2024-02-01T10:27:57.595 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:57.595 DEBUG:boto:Port: 8000 2024-02-01T10:27:57.595 DEBUG:boto:Params: {} 2024-02-01T10:27:57.595 DEBUG:boto:Token: None 2024-02-01T10:27:57.595 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:57 GMT /vclfcn-19/obj?versionId=7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6 2024-02-01T10:27:57.595 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:iMNCq97Yfwb4ra3ssfaTF8rMxLc= 2024-02-01T10:27:57.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iMNCq97Yfwb4ra3ssfaTF8rMxLc=', 'Content-Length': '0'} 2024-02-01T10:27:57.598 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '7U7FyoKYyhuM6pNG1sRPsdR7pmzebM6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004896125bc6c771ff-0065bb722d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:57.640 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:57.641 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:57.641 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:57.641 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:57.641 DEBUG:boto:Method: GET 2024-02-01T10:27:57.641 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:57.641 DEBUG:boto:Data: 2024-02-01T10:27:57.641 DEBUG:boto:Headers: {} 2024-02-01T10:27:57.641 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:57.641 DEBUG:boto:Port: 8001 2024-02-01T10:27:57.641 DEBUG:boto:Params: {} 2024-02-01T10:27:57.642 DEBUG:boto:Token: None 2024-02-01T10:27:57.642 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:57 GMT /vclfcn-19/ 2024-02-01T10:27:57.642 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:iRx4h85JUwm0irM6Iu/3cPjaFGs= 2024-02-01T10:27:57.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iRx4h85JUwm0irM6Iu/3cPjaFGs=', 'Content-Length': '0'} 2024-02-01T10:27:57.647 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000053ccf7aa1bfcf1b1-0065bb722d-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:57.647 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:57.648 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:57.648 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:57.648 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:57.648 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:57.648 DEBUG:boto:Method: GET 2024-02-01T10:27:57.648 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:57.648 DEBUG:boto:Data: 2024-02-01T10:27:57.649 DEBUG:boto:Headers: {} 2024-02-01T10:27:57.649 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:57.649 DEBUG:boto:Port: 8000 2024-02-01T10:27:57.649 DEBUG:boto:Params: {} 2024-02-01T10:27:57.649 DEBUG:boto:Token: None 2024-02-01T10:27:57.649 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:57 GMT /vclfcn-19/ 2024-02-01T10:27:57.649 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:iRx4h85JUwm0irM6Iu/3cPjaFGs= 2024-02-01T10:27:57.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iRx4h85JUwm0irM6Iu/3cPjaFGs=', 'Content-Length': '0'} 2024-02-01T10:27:57.655 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000979dcd4793ab543a-0065bb722d-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:57.655 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:57.655 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:57.655 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:57.655 DEBUG:boto:Method: HEAD 2024-02-01T10:27:57.656 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:57.656 DEBUG:boto:Data: 2024-02-01T10:27:57.656 DEBUG:boto:Headers: {} 2024-02-01T10:27:57.656 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:57.656 DEBUG:boto:Port: 8001 2024-02-01T10:27:57.656 DEBUG:boto:Params: {} 2024-02-01T10:27:57.656 DEBUG:boto:Token: None 2024-02-01T10:27:57.656 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:57 GMT /vclfcn-19/obj 2024-02-01T10:27:57.656 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:IMGogpnpS5JU7tzNsdNZn1YAlKs= 2024-02-01T10:27:57.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:IMGogpnpS5JU7tzNsdNZn1YAlKs=', 'Content-Length': '0'} 2024-02-01T10:27:57.663 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002aa87b5c8bd4ad4a-0065bb722d-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:57.663 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:57.663 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:57.663 DEBUG:boto:Method: HEAD 2024-02-01T10:27:57.663 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:57.663 DEBUG:boto:Data: 2024-02-01T10:27:57.663 DEBUG:boto:Headers: {} 2024-02-01T10:27:57.664 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:57.664 DEBUG:boto:Port: 8000 2024-02-01T10:27:57.664 DEBUG:boto:Params: {} 2024-02-01T10:27:57.664 DEBUG:boto:Token: None 2024-02-01T10:27:57.664 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:57 GMT /vclfcn-19/obj 2024-02-01T10:27:57.664 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:IMGogpnpS5JU7tzNsdNZn1YAlKs= 2024-02-01T10:27:57.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:IMGogpnpS5JU7tzNsdNZn1YAlKs=', 'Content-Length': '0'} 2024-02-01T10:27:57.670 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d43dc19773ede67c-0065bb722d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:57.671 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:57.671 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:57.671 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:57.671 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:57.671 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:57.671 DEBUG:boto:Method: GET 2024-02-01T10:27:57.671 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:57.671 DEBUG:boto:Data: 2024-02-01T10:27:57.671 DEBUG:boto:Headers: {} 2024-02-01T10:27:57.671 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:57.672 DEBUG:boto:Port: 8001 2024-02-01T10:27:57.672 DEBUG:boto:Params: {} 2024-02-01T10:27:57.672 DEBUG:boto:Token: None 2024-02-01T10:27:57.672 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:57 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:57.672 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GJShm2mcF2m6h/GMaVxLxuweY4E= 2024-02-01T10:27:57.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GJShm2mcF2m6h/GMaVxLxuweY4E=', 'Content-Length': '0'} 2024-02-01T10:27:57.675 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fd938a1ecc57a033-0065bb722d-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:57.716 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:57.716 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:57.716 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:57.717 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:57.717 DEBUG:boto:Method: GET 2024-02-01T10:27:57.717 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:57.717 DEBUG:boto:Data: 2024-02-01T10:27:57.717 DEBUG:boto:Headers: {} 2024-02-01T10:27:57.717 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:57.717 DEBUG:boto:Port: 8000 2024-02-01T10:27:57.717 DEBUG:boto:Params: {} 2024-02-01T10:27:57.717 DEBUG:boto:Token: None 2024-02-01T10:27:57.718 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:57 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:57.718 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GJShm2mcF2m6h/GMaVxLxuweY4E= 2024-02-01T10:27:57.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GJShm2mcF2m6h/GMaVxLxuweY4E=', 'Content-Length': '0'} 2024-02-01T10:27:57.721 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b0bb3600e2c2e1c2-0065bb722d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:57.722 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:57.722 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:57.722 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:57.722 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:57.722 DEBUG:boto:Method: GET 2024-02-01T10:27:57.723 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:57.723 DEBUG:boto:Data: 2024-02-01T10:27:57.723 DEBUG:boto:Headers: {} 2024-02-01T10:27:57.723 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:57.723 DEBUG:boto:Port: 8001 2024-02-01T10:27:57.723 DEBUG:boto:Params: {} 2024-02-01T10:27:57.723 DEBUG:boto:Token: None 2024-02-01T10:27:57.723 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:57 GMT /vclfcn-19/ 2024-02-01T10:27:57.723 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:iRx4h85JUwm0irM6Iu/3cPjaFGs= 2024-02-01T10:27:57.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iRx4h85JUwm0irM6Iu/3cPjaFGs=', 'Content-Length': '0'} 2024-02-01T10:27:57.731 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017482c5d2991050f-0065bb722d-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:57.731 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:57.732 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:57.732 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:57.732 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:57.732 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:57.732 DEBUG:boto:Method: GET 2024-02-01T10:27:57.732 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:57.732 DEBUG:boto:Data: 2024-02-01T10:27:57.733 DEBUG:boto:Headers: {} 2024-02-01T10:27:57.733 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:57.733 DEBUG:boto:Port: 8000 2024-02-01T10:27:57.733 DEBUG:boto:Params: {} 2024-02-01T10:27:57.734 DEBUG:boto:Token: None 2024-02-01T10:27:57.734 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:57 GMT /vclfcn-19/ 2024-02-01T10:27:57.734 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:iRx4h85JUwm0irM6Iu/3cPjaFGs= 2024-02-01T10:27:57.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iRx4h85JUwm0irM6Iu/3cPjaFGs=', 'Content-Length': '0'} 2024-02-01T10:27:57.740 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000085ac3494c2b18514-0065bb722d-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:57.780 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:57.781 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:57.781 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:57.781 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:57.781 DEBUG:boto:path=/vclfcn-19/obj?versionId=.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX 2024-02-01T10:27:57.781 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX 2024-02-01T10:27:57.781 DEBUG:boto:Method: GET 2024-02-01T10:27:57.781 DEBUG:boto:Path: /vclfcn-19/obj?versionId=.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX 2024-02-01T10:27:57.781 DEBUG:boto:Data: 2024-02-01T10:27:57.781 DEBUG:boto:Headers: {} 2024-02-01T10:27:57.782 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:57.782 DEBUG:boto:Port: 8001 2024-02-01T10:27:57.782 DEBUG:boto:Params: {} 2024-02-01T10:27:57.782 DEBUG:boto:Token: None 2024-02-01T10:27:57.782 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:57 GMT /vclfcn-19/obj?versionId=.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX 2024-02-01T10:27:57.782 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:oad6mmipgck0vWD8mjVfzTlI368= 2024-02-01T10:27:57.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:oad6mmipgck0vWD8mjVfzTlI368=', 'Content-Length': '0'} 2024-02-01T10:27:57.785 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006730b929c2d77495-0065bb722d-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:57.785 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:57.785 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:57.785 DEBUG:boto:path=/vclfcn-19/obj?versionId=.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX 2024-02-01T10:27:57.786 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX 2024-02-01T10:27:57.786 DEBUG:boto:Method: GET 2024-02-01T10:27:57.786 DEBUG:boto:Path: /vclfcn-19/obj?versionId=.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX 2024-02-01T10:27:57.786 DEBUG:boto:Data: 2024-02-01T10:27:57.786 DEBUG:boto:Headers: {} 2024-02-01T10:27:57.786 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:57.786 DEBUG:boto:Port: 8000 2024-02-01T10:27:57.786 DEBUG:boto:Params: {} 2024-02-01T10:27:57.786 DEBUG:boto:Token: None 2024-02-01T10:27:57.786 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:57 GMT /vclfcn-19/obj?versionId=.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX 2024-02-01T10:27:57.787 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:oad6mmipgck0vWD8mjVfzTlI368= 2024-02-01T10:27:57.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:oad6mmipgck0vWD8mjVfzTlI368=', 'Content-Length': '0'} 2024-02-01T10:27:57.789 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b383e0285fd64a88-0065bb722d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:57.790 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:57.790 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:57.790 DEBUG:boto:path=/vclfcn-19/obj?versionId=.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX 2024-02-01T10:27:57.790 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX 2024-02-01T10:27:57.790 DEBUG:boto:Method: HEAD 2024-02-01T10:27:57.790 DEBUG:boto:Path: /vclfcn-19/obj?versionId=.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX 2024-02-01T10:27:57.790 DEBUG:boto:Data: 2024-02-01T10:27:57.790 DEBUG:boto:Headers: {} 2024-02-01T10:27:57.790 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:57.790 DEBUG:boto:Port: 8001 2024-02-01T10:27:57.791 DEBUG:boto:Params: {} 2024-02-01T10:27:57.791 DEBUG:boto:Token: None 2024-02-01T10:27:57.791 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:57 GMT /vclfcn-19/obj?versionId=.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX 2024-02-01T10:27:57.791 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:6iPv5P3JdMsOCqWoCKhzF4uuj4Q= 2024-02-01T10:27:57.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:6iPv5P3JdMsOCqWoCKhzF4uuj4Q=', 'Content-Length': '0'} 2024-02-01T10:27:57.794 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d5fe654f096421d3-0065bb722d-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:57.794 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:57.794 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:57.794 DEBUG:boto:path=/vclfcn-19/obj?versionId=.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX 2024-02-01T10:27:57.794 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX 2024-02-01T10:27:57.795 DEBUG:boto:Method: HEAD 2024-02-01T10:27:57.795 DEBUG:boto:Path: /vclfcn-19/obj?versionId=.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX 2024-02-01T10:27:57.795 DEBUG:boto:Data: 2024-02-01T10:27:57.795 DEBUG:boto:Headers: {} 2024-02-01T10:27:57.795 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:57.795 DEBUG:boto:Port: 8000 2024-02-01T10:27:57.795 DEBUG:boto:Params: {} 2024-02-01T10:27:57.795 DEBUG:boto:Token: None 2024-02-01T10:27:57.795 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:57 GMT /vclfcn-19/obj?versionId=.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX 2024-02-01T10:27:57.795 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:6iPv5P3JdMsOCqWoCKhzF4uuj4Q= 2024-02-01T10:27:57.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:6iPv5P3JdMsOCqWoCKhzF4uuj4Q=', 'Content-Length': '0'} 2024-02-01T10:27:57.798 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000055f339d7cee99f32-0065bb722d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:57.799 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:57.799 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:57.799 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:57.799 DEBUG:boto:path=/vclfcn-19/obj?versionId=.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX 2024-02-01T10:27:57.799 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX 2024-02-01T10:27:57.799 DEBUG:boto:Method: GET 2024-02-01T10:27:57.799 DEBUG:boto:Path: /vclfcn-19/obj?versionId=.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX 2024-02-01T10:27:57.799 DEBUG:boto:Data: 2024-02-01T10:27:57.799 DEBUG:boto:Headers: {} 2024-02-01T10:27:57.799 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:57.800 DEBUG:boto:Port: 8001 2024-02-01T10:27:57.800 DEBUG:boto:Params: {} 2024-02-01T10:27:57.800 DEBUG:boto:Token: None 2024-02-01T10:27:57.800 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:57 GMT /vclfcn-19/obj?versionId=.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX 2024-02-01T10:27:57.800 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:oad6mmipgck0vWD8mjVfzTlI368= 2024-02-01T10:27:57.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:oad6mmipgck0vWD8mjVfzTlI368=', 'Content-Length': '0'} 2024-02-01T10:27:57.803 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000981315f8b9c52c82-0065bb722d-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:57.844 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:57.844 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:57.844 DEBUG:boto:path=/vclfcn-19/obj?versionId=.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX 2024-02-01T10:27:57.844 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX 2024-02-01T10:27:57.844 DEBUG:boto:Method: GET 2024-02-01T10:27:57.845 DEBUG:boto:Path: /vclfcn-19/obj?versionId=.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX 2024-02-01T10:27:57.845 DEBUG:boto:Data: 2024-02-01T10:27:57.845 DEBUG:boto:Headers: {} 2024-02-01T10:27:57.845 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:57.845 DEBUG:boto:Port: 8000 2024-02-01T10:27:57.845 DEBUG:boto:Params: {} 2024-02-01T10:27:57.845 DEBUG:boto:Token: None 2024-02-01T10:27:57.846 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:57 GMT /vclfcn-19/obj?versionId=.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX 2024-02-01T10:27:57.846 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:oad6mmipgck0vWD8mjVfzTlI368= 2024-02-01T10:27:57.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:oad6mmipgck0vWD8mjVfzTlI368=', 'Content-Length': '0'} 2024-02-01T10:27:57.849 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '.diUG6hoN7Rlb9qQ-U5iqQscXdU.pHX'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000736135e198adb646-0065bb722d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:57.849 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:57.849 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:57.849 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:57.849 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:57.849 DEBUG:boto:Method: GET 2024-02-01T10:27:57.849 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:57.849 DEBUG:boto:Data: 2024-02-01T10:27:57.850 DEBUG:boto:Headers: {} 2024-02-01T10:27:57.850 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:57.850 DEBUG:boto:Port: 8001 2024-02-01T10:27:57.850 DEBUG:boto:Params: {} 2024-02-01T10:27:57.850 DEBUG:boto:Token: None 2024-02-01T10:27:57.850 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:57 GMT /vclfcn-19/ 2024-02-01T10:27:57.850 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:iRx4h85JUwm0irM6Iu/3cPjaFGs= 2024-02-01T10:27:57.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iRx4h85JUwm0irM6Iu/3cPjaFGs=', 'Content-Length': '0'} 2024-02-01T10:27:57.855 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000880e049580682ad0-0065bb722d-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:57.856 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:57.856 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:57.856 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:57.856 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:57.856 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:57.856 DEBUG:boto:Method: GET 2024-02-01T10:27:57.857 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:57.857 DEBUG:boto:Data: 2024-02-01T10:27:57.857 DEBUG:boto:Headers: {} 2024-02-01T10:27:57.857 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:57.857 DEBUG:boto:Port: 8000 2024-02-01T10:27:57.857 DEBUG:boto:Params: {} 2024-02-01T10:27:57.857 DEBUG:boto:Token: None 2024-02-01T10:27:57.857 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:57 GMT /vclfcn-19/ 2024-02-01T10:27:57.857 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:iRx4h85JUwm0irM6Iu/3cPjaFGs= 2024-02-01T10:27:57.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iRx4h85JUwm0irM6Iu/3cPjaFGs=', 'Content-Length': '0'} 2024-02-01T10:27:57.862 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb2d31c0bb89f891-0065bb722d-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:57.908 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:57.908 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:57.908 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:57.909 DEBUG:boto:Method: HEAD 2024-02-01T10:27:57.909 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:57.909 DEBUG:boto:Data: 2024-02-01T10:27:57.909 DEBUG:boto:Headers: {} 2024-02-01T10:27:57.909 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:57.909 DEBUG:boto:Port: 8001 2024-02-01T10:27:57.909 DEBUG:boto:Params: {} 2024-02-01T10:27:57.909 DEBUG:boto:Token: None 2024-02-01T10:27:57.909 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:57 GMT /vclfcn-19/obj 2024-02-01T10:27:57.910 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:IMGogpnpS5JU7tzNsdNZn1YAlKs= 2024-02-01T10:27:57.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:IMGogpnpS5JU7tzNsdNZn1YAlKs=', 'Content-Length': '0'} 2024-02-01T10:27:57.915 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000df9bb4a4d0761c8c-0065bb722d-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:57.916 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:57.916 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:57.916 DEBUG:boto:Method: HEAD 2024-02-01T10:27:57.916 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:57.916 DEBUG:boto:Data: 2024-02-01T10:27:57.916 DEBUG:boto:Headers: {} 2024-02-01T10:27:57.916 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:57.916 DEBUG:boto:Port: 8000 2024-02-01T10:27:57.916 DEBUG:boto:Params: {} 2024-02-01T10:27:57.917 DEBUG:boto:Token: None 2024-02-01T10:27:57.917 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:57 GMT /vclfcn-19/obj 2024-02-01T10:27:57.917 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:IMGogpnpS5JU7tzNsdNZn1YAlKs= 2024-02-01T10:27:57.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:IMGogpnpS5JU7tzNsdNZn1YAlKs=', 'Content-Length': '0'} 2024-02-01T10:27:57.923 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000216f94874342a99d-0065bb722d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:57.923 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:57.923 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:57.923 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:57.923 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:57.923 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:57.923 DEBUG:boto:Method: GET 2024-02-01T10:27:57.924 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:57.924 DEBUG:boto:Data: 2024-02-01T10:27:57.924 DEBUG:boto:Headers: {} 2024-02-01T10:27:57.924 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:57.924 DEBUG:boto:Port: 8001 2024-02-01T10:27:57.924 DEBUG:boto:Params: {} 2024-02-01T10:27:57.924 DEBUG:boto:Token: None 2024-02-01T10:27:57.924 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:57 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:57.924 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GJShm2mcF2m6h/GMaVxLxuweY4E= 2024-02-01T10:27:57.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GJShm2mcF2m6h/GMaVxLxuweY4E=', 'Content-Length': '0'} 2024-02-01T10:27:57.927 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f35355657acd6b90-0065bb722d-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:57.972 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:57.972 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:57.973 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:57.973 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:57.973 DEBUG:boto:Method: GET 2024-02-01T10:27:57.973 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:57.973 DEBUG:boto:Data: 2024-02-01T10:27:57.973 DEBUG:boto:Headers: {} 2024-02-01T10:27:57.973 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:57.973 DEBUG:boto:Port: 8000 2024-02-01T10:27:57.973 DEBUG:boto:Params: {} 2024-02-01T10:27:57.973 DEBUG:boto:Token: None 2024-02-01T10:27:57.974 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:57 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:57.974 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GJShm2mcF2m6h/GMaVxLxuweY4E= 2024-02-01T10:27:57.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GJShm2mcF2m6h/GMaVxLxuweY4E=', 'Content-Length': '0'} 2024-02-01T10:27:57.976 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000009553868ce8012ed-0065bb722d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:57.977 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:57.977 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:57.977 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:57.977 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:57.977 DEBUG:boto:Method: GET 2024-02-01T10:27:57.977 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:57.977 DEBUG:boto:Data: 2024-02-01T10:27:57.977 DEBUG:boto:Headers: {} 2024-02-01T10:27:57.978 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:57.978 DEBUG:boto:Port: 8001 2024-02-01T10:27:57.978 DEBUG:boto:Params: {} 2024-02-01T10:27:57.978 DEBUG:boto:Token: None 2024-02-01T10:27:57.978 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:57 GMT /vclfcn-19/ 2024-02-01T10:27:57.978 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:iRx4h85JUwm0irM6Iu/3cPjaFGs= 2024-02-01T10:27:57.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iRx4h85JUwm0irM6Iu/3cPjaFGs=', 'Content-Length': '0'} 2024-02-01T10:27:57.983 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000676c59bc573ed6ab-0065bb722d-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:58.024 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:58.025 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:58.025 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:58.025 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:58.025 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:58.025 DEBUG:boto:Method: GET 2024-02-01T10:27:58.025 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:58.025 DEBUG:boto:Data: 2024-02-01T10:27:58.025 DEBUG:boto:Headers: {} 2024-02-01T10:27:58.025 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:58.026 DEBUG:boto:Port: 8000 2024-02-01T10:27:58.026 DEBUG:boto:Params: {} 2024-02-01T10:27:58.026 DEBUG:boto:Token: None 2024-02-01T10:27:58.026 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:58 GMT /vclfcn-19/ 2024-02-01T10:27:58.026 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cluKztGLOFRtdMpwC9EDQTr0RNQ= 2024-02-01T10:27:58.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cluKztGLOFRtdMpwC9EDQTr0RNQ=', 'Content-Length': '0'} 2024-02-01T10:27:58.032 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000036ad104a5d5d906a-0065bb722e-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:58.032 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:58.033 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:58.033 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:58.033 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:58.033 DEBUG:boto:path=/vclfcn-19/obj?versionId=63CC4agggciY3jdYETzlNEHLx15uh8w 2024-02-01T10:27:58.033 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=63CC4agggciY3jdYETzlNEHLx15uh8w 2024-02-01T10:27:58.033 DEBUG:boto:Method: GET 2024-02-01T10:27:58.033 DEBUG:boto:Path: /vclfcn-19/obj?versionId=63CC4agggciY3jdYETzlNEHLx15uh8w 2024-02-01T10:27:58.033 DEBUG:boto:Data: 2024-02-01T10:27:58.033 DEBUG:boto:Headers: {} 2024-02-01T10:27:58.033 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:58.034 DEBUG:boto:Port: 8001 2024-02-01T10:27:58.034 DEBUG:boto:Params: {} 2024-02-01T10:27:58.034 DEBUG:boto:Token: None 2024-02-01T10:27:58.034 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:58 GMT /vclfcn-19/obj?versionId=63CC4agggciY3jdYETzlNEHLx15uh8w 2024-02-01T10:27:58.034 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8KkUmsp9VylJ1N7gf9EO6634O5U= 2024-02-01T10:27:58.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8KkUmsp9VylJ1N7gf9EO6634O5U=', 'Content-Length': '0'} 2024-02-01T10:27:58.037 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '63CC4agggciY3jdYETzlNEHLx15uh8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fbcadf4498e8ad5b-0065bb722e-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:58.037 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:58.038 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:58.038 DEBUG:boto:path=/vclfcn-19/obj?versionId=63CC4agggciY3jdYETzlNEHLx15uh8w 2024-02-01T10:27:58.038 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=63CC4agggciY3jdYETzlNEHLx15uh8w 2024-02-01T10:27:58.038 DEBUG:boto:Method: GET 2024-02-01T10:27:58.038 DEBUG:boto:Path: /vclfcn-19/obj?versionId=63CC4agggciY3jdYETzlNEHLx15uh8w 2024-02-01T10:27:58.038 DEBUG:boto:Data: 2024-02-01T10:27:58.038 DEBUG:boto:Headers: {} 2024-02-01T10:27:58.038 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:58.038 DEBUG:boto:Port: 8000 2024-02-01T10:27:58.038 DEBUG:boto:Params: {} 2024-02-01T10:27:58.039 DEBUG:boto:Token: None 2024-02-01T10:27:58.039 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:58 GMT /vclfcn-19/obj?versionId=63CC4agggciY3jdYETzlNEHLx15uh8w 2024-02-01T10:27:58.039 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8KkUmsp9VylJ1N7gf9EO6634O5U= 2024-02-01T10:27:58.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8KkUmsp9VylJ1N7gf9EO6634O5U=', 'Content-Length': '0'} 2024-02-01T10:27:58.042 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '63CC4agggciY3jdYETzlNEHLx15uh8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a3dc33e58c38261b-0065bb722e-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:58.084 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:58.084 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:58.084 DEBUG:boto:path=/vclfcn-19/obj?versionId=63CC4agggciY3jdYETzlNEHLx15uh8w 2024-02-01T10:27:58.085 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=63CC4agggciY3jdYETzlNEHLx15uh8w 2024-02-01T10:27:58.085 DEBUG:boto:Method: HEAD 2024-02-01T10:27:58.085 DEBUG:boto:Path: /vclfcn-19/obj?versionId=63CC4agggciY3jdYETzlNEHLx15uh8w 2024-02-01T10:27:58.085 DEBUG:boto:Data: 2024-02-01T10:27:58.085 DEBUG:boto:Headers: {} 2024-02-01T10:27:58.085 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:58.085 DEBUG:boto:Port: 8001 2024-02-01T10:27:58.085 DEBUG:boto:Params: {} 2024-02-01T10:27:58.085 DEBUG:boto:Token: None 2024-02-01T10:27:58.085 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:58 GMT /vclfcn-19/obj?versionId=63CC4agggciY3jdYETzlNEHLx15uh8w 2024-02-01T10:27:58.086 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:JSCNTpjH1DE9g4NmReLkrHgsqfY= 2024-02-01T10:27:58.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:JSCNTpjH1DE9g4NmReLkrHgsqfY=', 'Content-Length': '0'} 2024-02-01T10:27:58.088 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '63CC4agggciY3jdYETzlNEHLx15uh8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000021717c243d54f401-0065bb722e-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:58.089 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:58.089 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:58.089 DEBUG:boto:path=/vclfcn-19/obj?versionId=63CC4agggciY3jdYETzlNEHLx15uh8w 2024-02-01T10:27:58.089 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=63CC4agggciY3jdYETzlNEHLx15uh8w 2024-02-01T10:27:58.089 DEBUG:boto:Method: HEAD 2024-02-01T10:27:58.089 DEBUG:boto:Path: /vclfcn-19/obj?versionId=63CC4agggciY3jdYETzlNEHLx15uh8w 2024-02-01T10:27:58.089 DEBUG:boto:Data: 2024-02-01T10:27:58.089 DEBUG:boto:Headers: {} 2024-02-01T10:27:58.089 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:58.089 DEBUG:boto:Port: 8000 2024-02-01T10:27:58.090 DEBUG:boto:Params: {} 2024-02-01T10:27:58.090 DEBUG:boto:Token: None 2024-02-01T10:27:58.090 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:58 GMT /vclfcn-19/obj?versionId=63CC4agggciY3jdYETzlNEHLx15uh8w 2024-02-01T10:27:58.090 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:JSCNTpjH1DE9g4NmReLkrHgsqfY= 2024-02-01T10:27:58.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:JSCNTpjH1DE9g4NmReLkrHgsqfY=', 'Content-Length': '0'} 2024-02-01T10:27:58.093 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '63CC4agggciY3jdYETzlNEHLx15uh8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000025143a8391caddd7-0065bb722e-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:58.093 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:58.093 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:58.093 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:58.093 DEBUG:boto:path=/vclfcn-19/obj?versionId=63CC4agggciY3jdYETzlNEHLx15uh8w 2024-02-01T10:27:58.093 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=63CC4agggciY3jdYETzlNEHLx15uh8w 2024-02-01T10:27:58.093 DEBUG:boto:Method: GET 2024-02-01T10:27:58.094 DEBUG:boto:Path: /vclfcn-19/obj?versionId=63CC4agggciY3jdYETzlNEHLx15uh8w 2024-02-01T10:27:58.094 DEBUG:boto:Data: 2024-02-01T10:27:58.094 DEBUG:boto:Headers: {} 2024-02-01T10:27:58.094 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:58.094 DEBUG:boto:Port: 8001 2024-02-01T10:27:58.094 DEBUG:boto:Params: {} 2024-02-01T10:27:58.094 DEBUG:boto:Token: None 2024-02-01T10:27:58.094 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:58 GMT /vclfcn-19/obj?versionId=63CC4agggciY3jdYETzlNEHLx15uh8w 2024-02-01T10:27:58.094 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8KkUmsp9VylJ1N7gf9EO6634O5U= 2024-02-01T10:27:58.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8KkUmsp9VylJ1N7gf9EO6634O5U=', 'Content-Length': '0'} 2024-02-01T10:27:58.097 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '63CC4agggciY3jdYETzlNEHLx15uh8w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002b7df17e04fbf854-0065bb722e-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:58.097 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:58.098 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:58.098 DEBUG:boto:path=/vclfcn-19/obj?versionId=63CC4agggciY3jdYETzlNEHLx15uh8w 2024-02-01T10:27:58.098 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=63CC4agggciY3jdYETzlNEHLx15uh8w 2024-02-01T10:27:58.098 DEBUG:boto:Method: GET 2024-02-01T10:27:58.098 DEBUG:boto:Path: /vclfcn-19/obj?versionId=63CC4agggciY3jdYETzlNEHLx15uh8w 2024-02-01T10:27:58.098 DEBUG:boto:Data: 2024-02-01T10:27:58.098 DEBUG:boto:Headers: {} 2024-02-01T10:27:58.098 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:58.098 DEBUG:boto:Port: 8000 2024-02-01T10:27:58.098 DEBUG:boto:Params: {} 2024-02-01T10:27:58.098 DEBUG:boto:Token: None 2024-02-01T10:27:58.099 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:58 GMT /vclfcn-19/obj?versionId=63CC4agggciY3jdYETzlNEHLx15uh8w 2024-02-01T10:27:58.099 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8KkUmsp9VylJ1N7gf9EO6634O5U= 2024-02-01T10:27:58.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8KkUmsp9VylJ1N7gf9EO6634O5U=', 'Content-Length': '0'} 2024-02-01T10:27:58.102 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '63CC4agggciY3jdYETzlNEHLx15uh8w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004c0c4cd9c6e2d7f8-0065bb722e-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:58.148 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:58.148 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:58.148 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:58.148 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:58.149 DEBUG:boto:Method: GET 2024-02-01T10:27:58.149 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:58.149 DEBUG:boto:Data: 2024-02-01T10:27:58.149 DEBUG:boto:Headers: {} 2024-02-01T10:27:58.149 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:58.149 DEBUG:boto:Port: 8001 2024-02-01T10:27:58.149 DEBUG:boto:Params: {} 2024-02-01T10:27:58.149 DEBUG:boto:Token: None 2024-02-01T10:27:58.149 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:58 GMT /vclfcn-19/ 2024-02-01T10:27:58.150 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cluKztGLOFRtdMpwC9EDQTr0RNQ= 2024-02-01T10:27:58.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cluKztGLOFRtdMpwC9EDQTr0RNQ=', 'Content-Length': '0'} 2024-02-01T10:27:58.155 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac3a53c521bcc3cb-0065bb722e-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:58.156 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:58.156 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:58.156 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:58.156 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:58.156 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:58.156 DEBUG:boto:Method: GET 2024-02-01T10:27:58.157 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:58.157 DEBUG:boto:Data: 2024-02-01T10:27:58.157 DEBUG:boto:Headers: {} 2024-02-01T10:27:58.157 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:58.157 DEBUG:boto:Port: 8000 2024-02-01T10:27:58.157 DEBUG:boto:Params: {} 2024-02-01T10:27:58.157 DEBUG:boto:Token: None 2024-02-01T10:27:58.157 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:58 GMT /vclfcn-19/ 2024-02-01T10:27:58.157 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cluKztGLOFRtdMpwC9EDQTr0RNQ= 2024-02-01T10:27:58.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cluKztGLOFRtdMpwC9EDQTr0RNQ=', 'Content-Length': '0'} 2024-02-01T10:27:58.163 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080d054fc57b7d483-0065bb722e-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:58.163 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:58.163 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:58.163 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:58.164 DEBUG:boto:Method: HEAD 2024-02-01T10:27:58.164 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:58.164 DEBUG:boto:Data: 2024-02-01T10:27:58.164 DEBUG:boto:Headers: {} 2024-02-01T10:27:58.164 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:58.164 DEBUG:boto:Port: 8001 2024-02-01T10:27:58.164 DEBUG:boto:Params: {} 2024-02-01T10:27:58.164 DEBUG:boto:Token: None 2024-02-01T10:27:58.164 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:58 GMT /vclfcn-19/obj 2024-02-01T10:27:58.165 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cgvL0IHKQduT/w9IF4MuZ0f5Zfg= 2024-02-01T10:27:58.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cgvL0IHKQduT/w9IF4MuZ0f5Zfg=', 'Content-Length': '0'} 2024-02-01T10:27:58.171 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fed8c1b715ccbd16-0065bb722e-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:58.171 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:58.171 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:58.171 DEBUG:boto:Method: HEAD 2024-02-01T10:27:58.171 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:58.171 DEBUG:boto:Data: 2024-02-01T10:27:58.171 DEBUG:boto:Headers: {} 2024-02-01T10:27:58.171 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:58.172 DEBUG:boto:Port: 8000 2024-02-01T10:27:58.172 DEBUG:boto:Params: {} 2024-02-01T10:27:58.172 DEBUG:boto:Token: None 2024-02-01T10:27:58.172 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:58 GMT /vclfcn-19/obj 2024-02-01T10:27:58.172 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cgvL0IHKQduT/w9IF4MuZ0f5Zfg= 2024-02-01T10:27:58.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cgvL0IHKQduT/w9IF4MuZ0f5Zfg=', 'Content-Length': '0'} 2024-02-01T10:27:58.177 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000240ab624e64dfc50-0065bb722e-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:58.178 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:58.178 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:58.178 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:58.178 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:58.178 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:58.178 DEBUG:boto:Method: GET 2024-02-01T10:27:58.178 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:58.178 DEBUG:boto:Data: 2024-02-01T10:27:58.179 DEBUG:boto:Headers: {} 2024-02-01T10:27:58.179 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:58.179 DEBUG:boto:Port: 8001 2024-02-01T10:27:58.179 DEBUG:boto:Params: {} 2024-02-01T10:27:58.179 DEBUG:boto:Token: None 2024-02-01T10:27:58.179 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:58 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:58.179 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:D6+oIN3Z4yOU7iQbZAYrHE3RkwM= 2024-02-01T10:27:58.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:D6+oIN3Z4yOU7iQbZAYrHE3RkwM=', 'Content-Length': '0'} 2024-02-01T10:27:58.182 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000abec32ff906239e4-0065bb722e-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:58.228 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:58.228 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:58.228 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:58.228 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:58.229 DEBUG:boto:Method: GET 2024-02-01T10:27:58.229 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:58.229 DEBUG:boto:Data: 2024-02-01T10:27:58.229 DEBUG:boto:Headers: {} 2024-02-01T10:27:58.229 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:58.229 DEBUG:boto:Port: 8000 2024-02-01T10:27:58.229 DEBUG:boto:Params: {} 2024-02-01T10:27:58.229 DEBUG:boto:Token: None 2024-02-01T10:27:58.229 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:58 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:58.230 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:D6+oIN3Z4yOU7iQbZAYrHE3RkwM= 2024-02-01T10:27:58.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:D6+oIN3Z4yOU7iQbZAYrHE3RkwM=', 'Content-Length': '0'} 2024-02-01T10:27:58.232 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000084b798595009e866-0065bb722e-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:58.233 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:58.233 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:58.233 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:58.233 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:58.233 DEBUG:boto:Method: GET 2024-02-01T10:27:58.233 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:58.233 DEBUG:boto:Data: 2024-02-01T10:27:58.234 DEBUG:boto:Headers: {} 2024-02-01T10:27:58.234 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:58.234 DEBUG:boto:Port: 8001 2024-02-01T10:27:58.234 DEBUG:boto:Params: {} 2024-02-01T10:27:58.234 DEBUG:boto:Token: None 2024-02-01T10:27:58.234 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:58 GMT /vclfcn-19/ 2024-02-01T10:27:58.234 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cluKztGLOFRtdMpwC9EDQTr0RNQ= 2024-02-01T10:27:58.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cluKztGLOFRtdMpwC9EDQTr0RNQ=', 'Content-Length': '0'} 2024-02-01T10:27:58.239 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000336d6f8cddb81dce-0065bb722e-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:58.240 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:58.240 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:58.240 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:58.241 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:58.241 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:58.241 DEBUG:boto:Method: GET 2024-02-01T10:27:58.241 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:58.241 DEBUG:boto:Data: 2024-02-01T10:27:58.241 DEBUG:boto:Headers: {} 2024-02-01T10:27:58.241 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:58.241 DEBUG:boto:Port: 8000 2024-02-01T10:27:58.241 DEBUG:boto:Params: {} 2024-02-01T10:27:58.241 DEBUG:boto:Token: None 2024-02-01T10:27:58.242 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:58 GMT /vclfcn-19/ 2024-02-01T10:27:58.242 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cluKztGLOFRtdMpwC9EDQTr0RNQ= 2024-02-01T10:27:58.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cluKztGLOFRtdMpwC9EDQTr0RNQ=', 'Content-Length': '0'} 2024-02-01T10:27:58.247 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040d3ee87b05392c2-0065bb722e-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:58.292 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:58.292 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:58.293 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:58.293 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:58.293 DEBUG:boto:path=/vclfcn-19/obj?versionId=YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o 2024-02-01T10:27:58.293 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o 2024-02-01T10:27:58.293 DEBUG:boto:Method: GET 2024-02-01T10:27:58.293 DEBUG:boto:Path: /vclfcn-19/obj?versionId=YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o 2024-02-01T10:27:58.293 DEBUG:boto:Data: 2024-02-01T10:27:58.293 DEBUG:boto:Headers: {} 2024-02-01T10:27:58.293 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:58.293 DEBUG:boto:Port: 8001 2024-02-01T10:27:58.293 DEBUG:boto:Params: {} 2024-02-01T10:27:58.294 DEBUG:boto:Token: None 2024-02-01T10:27:58.294 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:58 GMT /vclfcn-19/obj?versionId=YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o 2024-02-01T10:27:58.294 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mt7au0n6/H6jCesLNLskSvPzR2w= 2024-02-01T10:27:58.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mt7au0n6/H6jCesLNLskSvPzR2w=', 'Content-Length': '0'} 2024-02-01T10:27:58.297 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('x-amz-version-id', 'YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009cad2dd668866fc1-0065bb722e-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:58.297 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:58.297 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:58.298 DEBUG:boto:path=/vclfcn-19/obj?versionId=YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o 2024-02-01T10:27:58.298 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o 2024-02-01T10:27:58.298 DEBUG:boto:Method: GET 2024-02-01T10:27:58.298 DEBUG:boto:Path: /vclfcn-19/obj?versionId=YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o 2024-02-01T10:27:58.298 DEBUG:boto:Data: 2024-02-01T10:27:58.298 DEBUG:boto:Headers: {} 2024-02-01T10:27:58.298 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:58.298 DEBUG:boto:Port: 8000 2024-02-01T10:27:58.298 DEBUG:boto:Params: {} 2024-02-01T10:27:58.298 DEBUG:boto:Token: None 2024-02-01T10:27:58.299 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:58 GMT /vclfcn-19/obj?versionId=YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o 2024-02-01T10:27:58.299 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mt7au0n6/H6jCesLNLskSvPzR2w= 2024-02-01T10:27:58.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mt7au0n6/H6jCesLNLskSvPzR2w=', 'Content-Length': '0'} 2024-02-01T10:27:58.301 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('x-amz-version-id', 'YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004aff32f644d9f0a1-0065bb722e-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:58.344 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:58.344 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:58.344 DEBUG:boto:path=/vclfcn-19/obj?versionId=YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o 2024-02-01T10:27:58.344 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o 2024-02-01T10:27:58.344 DEBUG:boto:Method: HEAD 2024-02-01T10:27:58.345 DEBUG:boto:Path: /vclfcn-19/obj?versionId=YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o 2024-02-01T10:27:58.345 DEBUG:boto:Data: 2024-02-01T10:27:58.345 DEBUG:boto:Headers: {} 2024-02-01T10:27:58.345 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:58.345 DEBUG:boto:Port: 8001 2024-02-01T10:27:58.345 DEBUG:boto:Params: {} 2024-02-01T10:27:58.345 DEBUG:boto:Token: None 2024-02-01T10:27:58.345 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:58 GMT /vclfcn-19/obj?versionId=YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o 2024-02-01T10:27:58.345 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:xX4sQrcDE+lEOyu0nkE9YlvClrA= 2024-02-01T10:27:58.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:xX4sQrcDE+lEOyu0nkE9YlvClrA=', 'Content-Length': '0'} 2024-02-01T10:27:58.348 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('x-amz-version-id', 'YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003d0aa37916bb2f88-0065bb722e-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:58.349 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:58.349 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:58.349 DEBUG:boto:path=/vclfcn-19/obj?versionId=YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o 2024-02-01T10:27:58.349 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o 2024-02-01T10:27:58.349 DEBUG:boto:Method: HEAD 2024-02-01T10:27:58.349 DEBUG:boto:Path: /vclfcn-19/obj?versionId=YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o 2024-02-01T10:27:58.349 DEBUG:boto:Data: 2024-02-01T10:27:58.350 DEBUG:boto:Headers: {} 2024-02-01T10:27:58.350 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:58.350 DEBUG:boto:Port: 8000 2024-02-01T10:27:58.350 DEBUG:boto:Params: {} 2024-02-01T10:27:58.350 DEBUG:boto:Token: None 2024-02-01T10:27:58.350 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:58 GMT /vclfcn-19/obj?versionId=YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o 2024-02-01T10:27:58.350 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:xX4sQrcDE+lEOyu0nkE9YlvClrA= 2024-02-01T10:27:58.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:xX4sQrcDE+lEOyu0nkE9YlvClrA=', 'Content-Length': '0'} 2024-02-01T10:27:58.353 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('x-amz-version-id', 'YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e11ab4f1bc539b57-0065bb722e-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:58.354 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:58.354 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:58.354 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:58.354 DEBUG:boto:path=/vclfcn-19/obj?versionId=YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o 2024-02-01T10:27:58.354 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o 2024-02-01T10:27:58.354 DEBUG:boto:Method: GET 2024-02-01T10:27:58.354 DEBUG:boto:Path: /vclfcn-19/obj?versionId=YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o 2024-02-01T10:27:58.354 DEBUG:boto:Data: 2024-02-01T10:27:58.354 DEBUG:boto:Headers: {} 2024-02-01T10:27:58.355 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:58.355 DEBUG:boto:Port: 8001 2024-02-01T10:27:58.355 DEBUG:boto:Params: {} 2024-02-01T10:27:58.355 DEBUG:boto:Token: None 2024-02-01T10:27:58.355 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:58 GMT /vclfcn-19/obj?versionId=YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o 2024-02-01T10:27:58.355 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mt7au0n6/H6jCesLNLskSvPzR2w= 2024-02-01T10:27:58.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mt7au0n6/H6jCesLNLskSvPzR2w=', 'Content-Length': '0'} 2024-02-01T10:27:58.358 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('x-amz-version-id', 'YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000762e99e6023aeff3-0065bb722e-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:58.400 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:58.400 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:58.400 DEBUG:boto:path=/vclfcn-19/obj?versionId=YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o 2024-02-01T10:27:58.400 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o 2024-02-01T10:27:58.401 DEBUG:boto:Method: GET 2024-02-01T10:27:58.401 DEBUG:boto:Path: /vclfcn-19/obj?versionId=YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o 2024-02-01T10:27:58.401 DEBUG:boto:Data: 2024-02-01T10:27:58.401 DEBUG:boto:Headers: {} 2024-02-01T10:27:58.401 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:58.401 DEBUG:boto:Port: 8000 2024-02-01T10:27:58.401 DEBUG:boto:Params: {} 2024-02-01T10:27:58.401 DEBUG:boto:Token: None 2024-02-01T10:27:58.401 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:58 GMT /vclfcn-19/obj?versionId=YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o 2024-02-01T10:27:58.402 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mt7au0n6/H6jCesLNLskSvPzR2w= 2024-02-01T10:27:58.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mt7au0n6/H6jCesLNLskSvPzR2w=', 'Content-Length': '0'} 2024-02-01T10:27:58.405 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('x-amz-version-id', 'YzGxt4sA7Ne0IkUBiCAI2Aj4Zd7.-1o'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000041f55afe1e7d91ac-0065bb722e-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:58.452 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:58.452 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:58.453 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:58.453 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:58.453 DEBUG:boto:Method: GET 2024-02-01T10:27:58.453 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:58.453 DEBUG:boto:Data: 2024-02-01T10:27:58.453 DEBUG:boto:Headers: {} 2024-02-01T10:27:58.453 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:58.453 DEBUG:boto:Port: 8001 2024-02-01T10:27:58.453 DEBUG:boto:Params: {} 2024-02-01T10:27:58.454 DEBUG:boto:Token: None 2024-02-01T10:27:58.454 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:58 GMT /vclfcn-19/ 2024-02-01T10:27:58.454 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cluKztGLOFRtdMpwC9EDQTr0RNQ= 2024-02-01T10:27:58.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cluKztGLOFRtdMpwC9EDQTr0RNQ=', 'Content-Length': '0'} 2024-02-01T10:27:58.459 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000001a509afeda9de90-0065bb722e-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:58.504 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:58.505 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:58.505 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:58.505 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:58.505 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:58.505 DEBUG:boto:Method: GET 2024-02-01T10:27:58.505 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:58.506 DEBUG:boto:Data: 2024-02-01T10:27:58.506 DEBUG:boto:Headers: {} 2024-02-01T10:27:58.506 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:58.507 DEBUG:boto:Port: 8000 2024-02-01T10:27:58.507 DEBUG:boto:Params: {} 2024-02-01T10:27:58.507 DEBUG:boto:Token: None 2024-02-01T10:27:58.507 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:58 GMT /vclfcn-19/ 2024-02-01T10:27:58.507 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cluKztGLOFRtdMpwC9EDQTr0RNQ= 2024-02-01T10:27:58.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cluKztGLOFRtdMpwC9EDQTr0RNQ=', 'Content-Length': '0'} 2024-02-01T10:27:58.512 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6b53e7d0f61d653-0065bb722e-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:58.512 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:58.513 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:58.513 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:58.513 DEBUG:boto:Method: HEAD 2024-02-01T10:27:58.513 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:58.514 DEBUG:boto:Data: 2024-02-01T10:27:58.514 DEBUG:boto:Headers: {} 2024-02-01T10:27:58.514 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:58.514 DEBUG:boto:Port: 8001 2024-02-01T10:27:58.514 DEBUG:boto:Params: {} 2024-02-01T10:27:58.514 DEBUG:boto:Token: None 2024-02-01T10:27:58.514 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:58 GMT /vclfcn-19/obj 2024-02-01T10:27:58.515 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cgvL0IHKQduT/w9IF4MuZ0f5Zfg= 2024-02-01T10:27:58.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cgvL0IHKQduT/w9IF4MuZ0f5Zfg=', 'Content-Length': '0'} 2024-02-01T10:27:58.521 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000591baebaec6b950d-0065bb722e-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:58.521 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:58.521 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:58.521 DEBUG:boto:Method: HEAD 2024-02-01T10:27:58.521 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:58.521 DEBUG:boto:Data: 2024-02-01T10:27:58.522 DEBUG:boto:Headers: {} 2024-02-01T10:27:58.522 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:58.522 DEBUG:boto:Port: 8000 2024-02-01T10:27:58.522 DEBUG:boto:Params: {} 2024-02-01T10:27:58.522 DEBUG:boto:Token: None 2024-02-01T10:27:58.522 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:58 GMT /vclfcn-19/obj 2024-02-01T10:27:58.522 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cgvL0IHKQduT/w9IF4MuZ0f5Zfg= 2024-02-01T10:27:58.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cgvL0IHKQduT/w9IF4MuZ0f5Zfg=', 'Content-Length': '0'} 2024-02-01T10:27:58.528 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000a0193d1b3193af8-0065bb722e-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:58.528 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:58.528 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:58.528 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:58.528 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:58.528 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:58.529 DEBUG:boto:Method: GET 2024-02-01T10:27:58.529 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:58.529 DEBUG:boto:Data: 2024-02-01T10:27:58.529 DEBUG:boto:Headers: {} 2024-02-01T10:27:58.529 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:58.529 DEBUG:boto:Port: 8001 2024-02-01T10:27:58.529 DEBUG:boto:Params: {} 2024-02-01T10:27:58.529 DEBUG:boto:Token: None 2024-02-01T10:27:58.530 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:58 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:58.530 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:D6+oIN3Z4yOU7iQbZAYrHE3RkwM= 2024-02-01T10:27:58.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:D6+oIN3Z4yOU7iQbZAYrHE3RkwM=', 'Content-Length': '0'} 2024-02-01T10:27:58.532 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b429b1e2ea175a9b-0065bb722e-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:58.580 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:58.580 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:58.580 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:58.581 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:58.581 DEBUG:boto:Method: GET 2024-02-01T10:27:58.581 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:58.581 DEBUG:boto:Data: 2024-02-01T10:27:58.581 DEBUG:boto:Headers: {} 2024-02-01T10:27:58.581 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:58.581 DEBUG:boto:Port: 8000 2024-02-01T10:27:58.581 DEBUG:boto:Params: {} 2024-02-01T10:27:58.581 DEBUG:boto:Token: None 2024-02-01T10:27:58.582 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:58 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:58.582 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:D6+oIN3Z4yOU7iQbZAYrHE3RkwM= 2024-02-01T10:27:58.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:D6+oIN3Z4yOU7iQbZAYrHE3RkwM=', 'Content-Length': '0'} 2024-02-01T10:27:58.585 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d17d994c53c3dcfc-0065bb722e-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:58.585 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:58.585 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:58.585 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:58.585 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:58.586 DEBUG:boto:Method: GET 2024-02-01T10:27:58.586 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:58.586 DEBUG:boto:Data: 2024-02-01T10:27:58.586 DEBUG:boto:Headers: {} 2024-02-01T10:27:58.586 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:58.586 DEBUG:boto:Port: 8001 2024-02-01T10:27:58.586 DEBUG:boto:Params: {} 2024-02-01T10:27:58.586 DEBUG:boto:Token: None 2024-02-01T10:27:58.587 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:58 GMT /vclfcn-19/ 2024-02-01T10:27:58.587 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cluKztGLOFRtdMpwC9EDQTr0RNQ= 2024-02-01T10:27:58.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cluKztGLOFRtdMpwC9EDQTr0RNQ=', 'Content-Length': '0'} 2024-02-01T10:27:58.592 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a9a4254a08b15b2-0065bb722e-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:58.640 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:58.641 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:58.641 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:58.641 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:58.641 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:58.641 DEBUG:boto:Method: GET 2024-02-01T10:27:58.641 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:58.641 DEBUG:boto:Data: 2024-02-01T10:27:58.641 DEBUG:boto:Headers: {} 2024-02-01T10:27:58.641 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:58.642 DEBUG:boto:Port: 8000 2024-02-01T10:27:58.642 DEBUG:boto:Params: {} 2024-02-01T10:27:58.642 DEBUG:boto:Token: None 2024-02-01T10:27:58.642 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:58 GMT /vclfcn-19/ 2024-02-01T10:27:58.642 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cluKztGLOFRtdMpwC9EDQTr0RNQ= 2024-02-01T10:27:58.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cluKztGLOFRtdMpwC9EDQTr0RNQ=', 'Content-Length': '0'} 2024-02-01T10:27:58.648 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a182c291995f656a-0065bb722e-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:58.648 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:58.648 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:58.648 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:58.648 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:58.649 DEBUG:boto:path=/vclfcn-19/obj?versionId=B5jd89.SLRVLOAPcXqackukLvQnZY1B 2024-02-01T10:27:58.649 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=B5jd89.SLRVLOAPcXqackukLvQnZY1B 2024-02-01T10:27:58.649 DEBUG:boto:Method: GET 2024-02-01T10:27:58.649 DEBUG:boto:Path: /vclfcn-19/obj?versionId=B5jd89.SLRVLOAPcXqackukLvQnZY1B 2024-02-01T10:27:58.649 DEBUG:boto:Data: 2024-02-01T10:27:58.649 DEBUG:boto:Headers: {} 2024-02-01T10:27:58.649 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:58.649 DEBUG:boto:Port: 8001 2024-02-01T10:27:58.649 DEBUG:boto:Params: {} 2024-02-01T10:27:58.649 DEBUG:boto:Token: None 2024-02-01T10:27:58.649 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:58 GMT /vclfcn-19/obj?versionId=B5jd89.SLRVLOAPcXqackukLvQnZY1B 2024-02-01T10:27:58.650 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gtTTr1YoNY1g0wGuAex2+yNmNLs= 2024-02-01T10:27:58.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gtTTr1YoNY1g0wGuAex2+yNmNLs=', 'Content-Length': '0'} 2024-02-01T10:27:58.653 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('x-amz-version-id', 'B5jd89.SLRVLOAPcXqackukLvQnZY1B'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000078f6288fedeb2df1-0065bb722e-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:58.653 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:58.653 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:58.653 DEBUG:boto:path=/vclfcn-19/obj?versionId=B5jd89.SLRVLOAPcXqackukLvQnZY1B 2024-02-01T10:27:58.653 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=B5jd89.SLRVLOAPcXqackukLvQnZY1B 2024-02-01T10:27:58.653 DEBUG:boto:Method: GET 2024-02-01T10:27:58.653 DEBUG:boto:Path: /vclfcn-19/obj?versionId=B5jd89.SLRVLOAPcXqackukLvQnZY1B 2024-02-01T10:27:58.654 DEBUG:boto:Data: 2024-02-01T10:27:58.654 DEBUG:boto:Headers: {} 2024-02-01T10:27:58.654 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:58.654 DEBUG:boto:Port: 8000 2024-02-01T10:27:58.654 DEBUG:boto:Params: {} 2024-02-01T10:27:58.654 DEBUG:boto:Token: None 2024-02-01T10:27:58.654 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:58 GMT /vclfcn-19/obj?versionId=B5jd89.SLRVLOAPcXqackukLvQnZY1B 2024-02-01T10:27:58.655 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gtTTr1YoNY1g0wGuAex2+yNmNLs= 2024-02-01T10:27:58.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gtTTr1YoNY1g0wGuAex2+yNmNLs=', 'Content-Length': '0'} 2024-02-01T10:27:58.657 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('x-amz-version-id', 'B5jd89.SLRVLOAPcXqackukLvQnZY1B'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fe00489e1141effe-0065bb722e-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:58.700 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:58.700 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:58.700 DEBUG:boto:path=/vclfcn-19/obj?versionId=B5jd89.SLRVLOAPcXqackukLvQnZY1B 2024-02-01T10:27:58.701 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=B5jd89.SLRVLOAPcXqackukLvQnZY1B 2024-02-01T10:27:58.701 DEBUG:boto:Method: HEAD 2024-02-01T10:27:58.701 DEBUG:boto:Path: /vclfcn-19/obj?versionId=B5jd89.SLRVLOAPcXqackukLvQnZY1B 2024-02-01T10:27:58.701 DEBUG:boto:Data: 2024-02-01T10:27:58.701 DEBUG:boto:Headers: {} 2024-02-01T10:27:58.701 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:58.701 DEBUG:boto:Port: 8001 2024-02-01T10:27:58.701 DEBUG:boto:Params: {} 2024-02-01T10:27:58.701 DEBUG:boto:Token: None 2024-02-01T10:27:58.701 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:58 GMT /vclfcn-19/obj?versionId=B5jd89.SLRVLOAPcXqackukLvQnZY1B 2024-02-01T10:27:58.702 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:CNKw5jd/iZtWU2i1nb1m+efYKYA= 2024-02-01T10:27:58.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:CNKw5jd/iZtWU2i1nb1m+efYKYA=', 'Content-Length': '0'} 2024-02-01T10:27:58.704 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('x-amz-version-id', 'B5jd89.SLRVLOAPcXqackukLvQnZY1B'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000049854011704ecb91-0065bb722e-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:58.705 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:58.705 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:58.705 DEBUG:boto:path=/vclfcn-19/obj?versionId=B5jd89.SLRVLOAPcXqackukLvQnZY1B 2024-02-01T10:27:58.705 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=B5jd89.SLRVLOAPcXqackukLvQnZY1B 2024-02-01T10:27:58.705 DEBUG:boto:Method: HEAD 2024-02-01T10:27:58.705 DEBUG:boto:Path: /vclfcn-19/obj?versionId=B5jd89.SLRVLOAPcXqackukLvQnZY1B 2024-02-01T10:27:58.705 DEBUG:boto:Data: 2024-02-01T10:27:58.705 DEBUG:boto:Headers: {} 2024-02-01T10:27:58.705 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:58.706 DEBUG:boto:Port: 8000 2024-02-01T10:27:58.706 DEBUG:boto:Params: {} 2024-02-01T10:27:58.706 DEBUG:boto:Token: None 2024-02-01T10:27:58.706 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:58 GMT /vclfcn-19/obj?versionId=B5jd89.SLRVLOAPcXqackukLvQnZY1B 2024-02-01T10:27:58.706 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:CNKw5jd/iZtWU2i1nb1m+efYKYA= 2024-02-01T10:27:58.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:CNKw5jd/iZtWU2i1nb1m+efYKYA=', 'Content-Length': '0'} 2024-02-01T10:27:58.709 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('x-amz-version-id', 'B5jd89.SLRVLOAPcXqackukLvQnZY1B'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fa9e27e4602548c1-0065bb722e-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:58.709 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:58.709 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:58.709 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:58.709 DEBUG:boto:path=/vclfcn-19/obj?versionId=B5jd89.SLRVLOAPcXqackukLvQnZY1B 2024-02-01T10:27:58.709 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=B5jd89.SLRVLOAPcXqackukLvQnZY1B 2024-02-01T10:27:58.710 DEBUG:boto:Method: GET 2024-02-01T10:27:58.710 DEBUG:boto:Path: /vclfcn-19/obj?versionId=B5jd89.SLRVLOAPcXqackukLvQnZY1B 2024-02-01T10:27:58.710 DEBUG:boto:Data: 2024-02-01T10:27:58.710 DEBUG:boto:Headers: {} 2024-02-01T10:27:58.710 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:58.710 DEBUG:boto:Port: 8001 2024-02-01T10:27:58.710 DEBUG:boto:Params: {} 2024-02-01T10:27:58.710 DEBUG:boto:Token: None 2024-02-01T10:27:58.710 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:58 GMT /vclfcn-19/obj?versionId=B5jd89.SLRVLOAPcXqackukLvQnZY1B 2024-02-01T10:27:58.710 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gtTTr1YoNY1g0wGuAex2+yNmNLs= 2024-02-01T10:27:58.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gtTTr1YoNY1g0wGuAex2+yNmNLs=', 'Content-Length': '0'} 2024-02-01T10:27:58.713 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('x-amz-version-id', 'B5jd89.SLRVLOAPcXqackukLvQnZY1B'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004e4e4a4432a39a92-0065bb722e-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:58.760 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:58.760 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:58.760 DEBUG:boto:path=/vclfcn-19/obj?versionId=B5jd89.SLRVLOAPcXqackukLvQnZY1B 2024-02-01T10:27:58.760 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=B5jd89.SLRVLOAPcXqackukLvQnZY1B 2024-02-01T10:27:58.761 DEBUG:boto:Method: GET 2024-02-01T10:27:58.761 DEBUG:boto:Path: /vclfcn-19/obj?versionId=B5jd89.SLRVLOAPcXqackukLvQnZY1B 2024-02-01T10:27:58.761 DEBUG:boto:Data: 2024-02-01T10:27:58.761 DEBUG:boto:Headers: {} 2024-02-01T10:27:58.761 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:58.761 DEBUG:boto:Port: 8000 2024-02-01T10:27:58.761 DEBUG:boto:Params: {} 2024-02-01T10:27:58.761 DEBUG:boto:Token: None 2024-02-01T10:27:58.762 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:58 GMT /vclfcn-19/obj?versionId=B5jd89.SLRVLOAPcXqackukLvQnZY1B 2024-02-01T10:27:58.762 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gtTTr1YoNY1g0wGuAex2+yNmNLs= 2024-02-01T10:27:58.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gtTTr1YoNY1g0wGuAex2+yNmNLs=', 'Content-Length': '0'} 2024-02-01T10:27:58.765 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('x-amz-version-id', 'B5jd89.SLRVLOAPcXqackukLvQnZY1B'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e6f42848b3347d7d-0065bb722e-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:58.808 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:58.808 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:58.808 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:58.808 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:58.809 DEBUG:boto:Method: GET 2024-02-01T10:27:58.809 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:58.809 DEBUG:boto:Data: 2024-02-01T10:27:58.809 DEBUG:boto:Headers: {} 2024-02-01T10:27:58.809 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:58.809 DEBUG:boto:Port: 8001 2024-02-01T10:27:58.809 DEBUG:boto:Params: {} 2024-02-01T10:27:58.809 DEBUG:boto:Token: None 2024-02-01T10:27:58.809 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:58 GMT /vclfcn-19/ 2024-02-01T10:27:58.810 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cluKztGLOFRtdMpwC9EDQTr0RNQ= 2024-02-01T10:27:58.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cluKztGLOFRtdMpwC9EDQTr0RNQ=', 'Content-Length': '0'} 2024-02-01T10:27:58.815 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fcf98899d10c2023-0065bb722e-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:58.856 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:58.856 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:58.857 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:58.857 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:58.857 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:58.857 DEBUG:boto:Method: GET 2024-02-01T10:27:58.857 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:58.857 DEBUG:boto:Data: 2024-02-01T10:27:58.857 DEBUG:boto:Headers: {} 2024-02-01T10:27:58.857 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:58.857 DEBUG:boto:Port: 8000 2024-02-01T10:27:58.857 DEBUG:boto:Params: {} 2024-02-01T10:27:58.857 DEBUG:boto:Token: None 2024-02-01T10:27:58.858 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:58 GMT /vclfcn-19/ 2024-02-01T10:27:58.858 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cluKztGLOFRtdMpwC9EDQTr0RNQ= 2024-02-01T10:27:58.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cluKztGLOFRtdMpwC9EDQTr0RNQ=', 'Content-Length': '0'} 2024-02-01T10:27:58.863 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ffbcfca3a1eccbdd-0065bb722e-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:58.864 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:58.864 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:58.864 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:58.864 DEBUG:boto:Method: HEAD 2024-02-01T10:27:58.864 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:58.865 DEBUG:boto:Data: 2024-02-01T10:27:58.865 DEBUG:boto:Headers: {} 2024-02-01T10:27:58.865 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:58.865 DEBUG:boto:Port: 8001 2024-02-01T10:27:58.865 DEBUG:boto:Params: {} 2024-02-01T10:27:58.865 DEBUG:boto:Token: None 2024-02-01T10:27:58.866 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:58 GMT /vclfcn-19/obj 2024-02-01T10:27:58.866 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cgvL0IHKQduT/w9IF4MuZ0f5Zfg= 2024-02-01T10:27:58.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cgvL0IHKQduT/w9IF4MuZ0f5Zfg=', 'Content-Length': '0'} 2024-02-01T10:27:58.872 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aa1b0070fd0aaf56-0065bb722e-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:58.872 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:58.873 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:58.873 DEBUG:boto:Method: HEAD 2024-02-01T10:27:58.873 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:58.873 DEBUG:boto:Data: 2024-02-01T10:27:58.873 DEBUG:boto:Headers: {} 2024-02-01T10:27:58.873 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:58.873 DEBUG:boto:Port: 8000 2024-02-01T10:27:58.873 DEBUG:boto:Params: {} 2024-02-01T10:27:58.873 DEBUG:boto:Token: None 2024-02-01T10:27:58.873 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:58 GMT /vclfcn-19/obj 2024-02-01T10:27:58.874 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cgvL0IHKQduT/w9IF4MuZ0f5Zfg= 2024-02-01T10:27:58.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cgvL0IHKQduT/w9IF4MuZ0f5Zfg=', 'Content-Length': '0'} 2024-02-01T10:27:58.879 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008a6cbdc2a88bbb40-0065bb722e-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:58.880 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:58.880 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:58.881 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:58.881 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:58.881 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:58.881 DEBUG:boto:Method: GET 2024-02-01T10:27:58.881 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:58.881 DEBUG:boto:Data: 2024-02-01T10:27:58.881 DEBUG:boto:Headers: {} 2024-02-01T10:27:58.881 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:58.881 DEBUG:boto:Port: 8001 2024-02-01T10:27:58.881 DEBUG:boto:Params: {} 2024-02-01T10:27:58.883 DEBUG:boto:Token: None 2024-02-01T10:27:58.883 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:58 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:58.883 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:D6+oIN3Z4yOU7iQbZAYrHE3RkwM= 2024-02-01T10:27:58.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:D6+oIN3Z4yOU7iQbZAYrHE3RkwM=', 'Content-Length': '0'} 2024-02-01T10:27:58.886 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dc0ca02f11a20e6f-0065bb722e-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:58.928 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:58.928 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:58.928 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:58.928 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:58.928 DEBUG:boto:Method: GET 2024-02-01T10:27:58.929 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:58.929 DEBUG:boto:Data: 2024-02-01T10:27:58.929 DEBUG:boto:Headers: {} 2024-02-01T10:27:58.929 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:58.929 DEBUG:boto:Port: 8000 2024-02-01T10:27:58.929 DEBUG:boto:Params: {} 2024-02-01T10:27:58.929 DEBUG:boto:Token: None 2024-02-01T10:27:58.929 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:58 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:58.929 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:D6+oIN3Z4yOU7iQbZAYrHE3RkwM= 2024-02-01T10:27:58.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:D6+oIN3Z4yOU7iQbZAYrHE3RkwM=', 'Content-Length': '0'} 2024-02-01T10:27:58.933 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000035df4fdb586b2281-0065bb722e-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:58.933 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:58.933 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:58.933 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:58.933 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:58.933 DEBUG:boto:Method: GET 2024-02-01T10:27:58.933 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:58.934 DEBUG:boto:Data: 2024-02-01T10:27:58.934 DEBUG:boto:Headers: {} 2024-02-01T10:27:58.934 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:58.934 DEBUG:boto:Port: 8001 2024-02-01T10:27:58.934 DEBUG:boto:Params: {} 2024-02-01T10:27:58.934 DEBUG:boto:Token: None 2024-02-01T10:27:58.934 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:58 GMT /vclfcn-19/ 2024-02-01T10:27:58.934 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cluKztGLOFRtdMpwC9EDQTr0RNQ= 2024-02-01T10:27:58.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cluKztGLOFRtdMpwC9EDQTr0RNQ=', 'Content-Length': '0'} 2024-02-01T10:27:58.939 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000993eff4b08db9569-0065bb722e-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:58.980 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:58.980 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:58.981 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:58.981 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:58.981 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:58.981 DEBUG:boto:Method: GET 2024-02-01T10:27:58.981 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:58.981 DEBUG:boto:Data: 2024-02-01T10:27:58.981 DEBUG:boto:Headers: {} 2024-02-01T10:27:58.981 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:58.981 DEBUG:boto:Port: 8000 2024-02-01T10:27:58.981 DEBUG:boto:Params: {} 2024-02-01T10:27:58.982 DEBUG:boto:Token: None 2024-02-01T10:27:58.982 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:58 GMT /vclfcn-19/ 2024-02-01T10:27:58.982 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cluKztGLOFRtdMpwC9EDQTr0RNQ= 2024-02-01T10:27:58.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cluKztGLOFRtdMpwC9EDQTr0RNQ=', 'Content-Length': '0'} 2024-02-01T10:27:58.987 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008569b0f4261c56eb-0065bb722e-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:58.987 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:58.988 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:58.988 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:58.988 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:58.988 DEBUG:boto:path=/vclfcn-19/obj?versionId=rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ 2024-02-01T10:27:58.988 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ 2024-02-01T10:27:58.988 DEBUG:boto:Method: GET 2024-02-01T10:27:58.988 DEBUG:boto:Path: /vclfcn-19/obj?versionId=rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ 2024-02-01T10:27:58.988 DEBUG:boto:Data: 2024-02-01T10:27:58.989 DEBUG:boto:Headers: {} 2024-02-01T10:27:58.989 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:58.989 DEBUG:boto:Port: 8001 2024-02-01T10:27:58.989 DEBUG:boto:Params: {} 2024-02-01T10:27:58.989 DEBUG:boto:Token: None 2024-02-01T10:27:58.990 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:58 GMT /vclfcn-19/obj?versionId=rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ 2024-02-01T10:27:58.990 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wufDVNpi0lI/Bz1cglwdD4QCzjA= 2024-02-01T10:27:58.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wufDVNpi0lI/Bz1cglwdD4QCzjA=', 'Content-Length': '0'} 2024-02-01T10:27:58.992 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('x-amz-version-id', 'rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000acf0dfbfc0226a6f-0065bb722e-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:58.993 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:58.993 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:58.993 DEBUG:boto:path=/vclfcn-19/obj?versionId=rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ 2024-02-01T10:27:58.993 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ 2024-02-01T10:27:58.993 DEBUG:boto:Method: GET 2024-02-01T10:27:58.993 DEBUG:boto:Path: /vclfcn-19/obj?versionId=rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ 2024-02-01T10:27:58.994 DEBUG:boto:Data: 2024-02-01T10:27:58.994 DEBUG:boto:Headers: {} 2024-02-01T10:27:58.994 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:58.994 DEBUG:boto:Port: 8000 2024-02-01T10:27:58.994 DEBUG:boto:Params: {} 2024-02-01T10:27:58.994 DEBUG:boto:Token: None 2024-02-01T10:27:58.994 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:58 GMT /vclfcn-19/obj?versionId=rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ 2024-02-01T10:27:58.994 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wufDVNpi0lI/Bz1cglwdD4QCzjA= 2024-02-01T10:27:58.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wufDVNpi0lI/Bz1cglwdD4QCzjA=', 'Content-Length': '0'} 2024-02-01T10:27:58.997 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('x-amz-version-id', 'rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008417b76cd1b9dde5-0065bb722f-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:59.044 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:59.044 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:59.044 DEBUG:boto:path=/vclfcn-19/obj?versionId=rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ 2024-02-01T10:27:59.044 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ 2024-02-01T10:27:59.044 DEBUG:boto:Method: HEAD 2024-02-01T10:27:59.045 DEBUG:boto:Path: /vclfcn-19/obj?versionId=rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ 2024-02-01T10:27:59.045 DEBUG:boto:Data: 2024-02-01T10:27:59.045 DEBUG:boto:Headers: {} 2024-02-01T10:27:59.045 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:59.045 DEBUG:boto:Port: 8001 2024-02-01T10:27:59.045 DEBUG:boto:Params: {} 2024-02-01T10:27:59.045 DEBUG:boto:Token: None 2024-02-01T10:27:59.045 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:59 GMT /vclfcn-19/obj?versionId=rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ 2024-02-01T10:27:59.045 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:me9D1hpadSpj1nC3XOLI6RxOAXM= 2024-02-01T10:27:59.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:me9D1hpadSpj1nC3XOLI6RxOAXM=', 'Content-Length': '0'} 2024-02-01T10:27:59.048 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('x-amz-version-id', 'rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a49e88ae35c74257-0065bb722f-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:59.049 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:59.049 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:59.049 DEBUG:boto:path=/vclfcn-19/obj?versionId=rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ 2024-02-01T10:27:59.049 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ 2024-02-01T10:27:59.049 DEBUG:boto:Method: HEAD 2024-02-01T10:27:59.049 DEBUG:boto:Path: /vclfcn-19/obj?versionId=rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ 2024-02-01T10:27:59.049 DEBUG:boto:Data: 2024-02-01T10:27:59.049 DEBUG:boto:Headers: {} 2024-02-01T10:27:59.049 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:59.049 DEBUG:boto:Port: 8000 2024-02-01T10:27:59.049 DEBUG:boto:Params: {} 2024-02-01T10:27:59.050 DEBUG:boto:Token: None 2024-02-01T10:27:59.050 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:59 GMT /vclfcn-19/obj?versionId=rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ 2024-02-01T10:27:59.050 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:me9D1hpadSpj1nC3XOLI6RxOAXM= 2024-02-01T10:27:59.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:me9D1hpadSpj1nC3XOLI6RxOAXM=', 'Content-Length': '0'} 2024-02-01T10:27:59.053 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('x-amz-version-id', 'rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000beda240ccb83e3b6-0065bb722f-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:59.053 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:59.053 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:59.053 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:59.053 DEBUG:boto:path=/vclfcn-19/obj?versionId=rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ 2024-02-01T10:27:59.054 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ 2024-02-01T10:27:59.054 DEBUG:boto:Method: GET 2024-02-01T10:27:59.054 DEBUG:boto:Path: /vclfcn-19/obj?versionId=rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ 2024-02-01T10:27:59.054 DEBUG:boto:Data: 2024-02-01T10:27:59.054 DEBUG:boto:Headers: {} 2024-02-01T10:27:59.054 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:59.054 DEBUG:boto:Port: 8001 2024-02-01T10:27:59.054 DEBUG:boto:Params: {} 2024-02-01T10:27:59.054 DEBUG:boto:Token: None 2024-02-01T10:27:59.054 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:59 GMT /vclfcn-19/obj?versionId=rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ 2024-02-01T10:27:59.055 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:N/Gkdx9O5EvyJwJ4JOW2bVD2n3Q= 2024-02-01T10:27:59.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:N/Gkdx9O5EvyJwJ4JOW2bVD2n3Q=', 'Content-Length': '0'} 2024-02-01T10:27:59.057 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('x-amz-version-id', 'rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005338036eb7f4270b-0065bb722f-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:59.058 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:59.058 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:59.058 DEBUG:boto:path=/vclfcn-19/obj?versionId=rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ 2024-02-01T10:27:59.058 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ 2024-02-01T10:27:59.058 DEBUG:boto:Method: GET 2024-02-01T10:27:59.058 DEBUG:boto:Path: /vclfcn-19/obj?versionId=rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ 2024-02-01T10:27:59.058 DEBUG:boto:Data: 2024-02-01T10:27:59.058 DEBUG:boto:Headers: {} 2024-02-01T10:27:59.058 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:59.058 DEBUG:boto:Port: 8000 2024-02-01T10:27:59.058 DEBUG:boto:Params: {} 2024-02-01T10:27:59.059 DEBUG:boto:Token: None 2024-02-01T10:27:59.059 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:59 GMT /vclfcn-19/obj?versionId=rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ 2024-02-01T10:27:59.059 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:N/Gkdx9O5EvyJwJ4JOW2bVD2n3Q= 2024-02-01T10:27:59.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:N/Gkdx9O5EvyJwJ4JOW2bVD2n3Q=', 'Content-Length': '0'} 2024-02-01T10:27:59.062 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('x-amz-version-id', 'rcBw9NDMcjurzSdpB.bvwX4JmUDkwrQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a4acf4451b00a10c-0065bb722f-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:59.104 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:59.104 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:59.104 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:59.104 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:59.105 DEBUG:boto:Method: GET 2024-02-01T10:27:59.105 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:59.105 DEBUG:boto:Data: 2024-02-01T10:27:59.105 DEBUG:boto:Headers: {} 2024-02-01T10:27:59.105 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:59.105 DEBUG:boto:Port: 8001 2024-02-01T10:27:59.105 DEBUG:boto:Params: {} 2024-02-01T10:27:59.105 DEBUG:boto:Token: None 2024-02-01T10:27:59.105 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:59 GMT /vclfcn-19/ 2024-02-01T10:27:59.106 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:e66F8l/8ybxeRJYXSVu3Ne/Y9I4= 2024-02-01T10:27:59.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:e66F8l/8ybxeRJYXSVu3Ne/Y9I4=', 'Content-Length': '0'} 2024-02-01T10:27:59.111 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005884f5539ffa47ad-0065bb722f-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:59.111 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:59.111 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:59.111 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:59.111 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:59.112 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:59.112 DEBUG:boto:Method: GET 2024-02-01T10:27:59.112 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:59.112 DEBUG:boto:Data: 2024-02-01T10:27:59.112 DEBUG:boto:Headers: {} 2024-02-01T10:27:59.112 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:59.112 DEBUG:boto:Port: 8000 2024-02-01T10:27:59.112 DEBUG:boto:Params: {} 2024-02-01T10:27:59.113 DEBUG:boto:Token: None 2024-02-01T10:27:59.113 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:59 GMT /vclfcn-19/ 2024-02-01T10:27:59.113 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:e66F8l/8ybxeRJYXSVu3Ne/Y9I4= 2024-02-01T10:27:59.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:e66F8l/8ybxeRJYXSVu3Ne/Y9I4=', 'Content-Length': '0'} 2024-02-01T10:27:59.119 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3307797b93870fe-0065bb722f-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:59.164 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:59.165 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:59.165 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:59.165 DEBUG:boto:Method: HEAD 2024-02-01T10:27:59.165 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:59.166 DEBUG:boto:Data: 2024-02-01T10:27:59.166 DEBUG:boto:Headers: {} 2024-02-01T10:27:59.166 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:59.166 DEBUG:boto:Port: 8001 2024-02-01T10:27:59.166 DEBUG:boto:Params: {} 2024-02-01T10:27:59.166 DEBUG:boto:Token: None 2024-02-01T10:27:59.166 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:59 GMT /vclfcn-19/obj 2024-02-01T10:27:59.166 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HIc6WtLeqhaYJCcocvhVpa6d9/U= 2024-02-01T10:27:59.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HIc6WtLeqhaYJCcocvhVpa6d9/U=', 'Content-Length': '0'} 2024-02-01T10:27:59.173 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004601f89c3ecaff7f-0065bb722f-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:59.173 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:59.173 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:59.173 DEBUG:boto:Method: HEAD 2024-02-01T10:27:59.174 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:59.174 DEBUG:boto:Data: 2024-02-01T10:27:59.174 DEBUG:boto:Headers: {} 2024-02-01T10:27:59.174 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:59.174 DEBUG:boto:Port: 8000 2024-02-01T10:27:59.174 DEBUG:boto:Params: {} 2024-02-01T10:27:59.174 DEBUG:boto:Token: None 2024-02-01T10:27:59.174 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:59 GMT /vclfcn-19/obj 2024-02-01T10:27:59.174 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HIc6WtLeqhaYJCcocvhVpa6d9/U= 2024-02-01T10:27:59.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HIc6WtLeqhaYJCcocvhVpa6d9/U=', 'Content-Length': '0'} 2024-02-01T10:27:59.180 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000025b66b093f813594-0065bb722f-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:59.180 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:59.180 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:59.181 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:59.181 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:59.181 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:59.181 DEBUG:boto:Method: GET 2024-02-01T10:27:59.181 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:59.181 DEBUG:boto:Data: 2024-02-01T10:27:59.181 DEBUG:boto:Headers: {} 2024-02-01T10:27:59.181 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:59.181 DEBUG:boto:Port: 8001 2024-02-01T10:27:59.181 DEBUG:boto:Params: {} 2024-02-01T10:27:59.181 DEBUG:boto:Token: None 2024-02-01T10:27:59.182 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:59 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:59.182 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:qaaTCVBC1I5W/9elBzPRtmzsbxU= 2024-02-01T10:27:59.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:qaaTCVBC1I5W/9elBzPRtmzsbxU=', 'Content-Length': '0'} 2024-02-01T10:27:59.185 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000034a00c2d9c7f86ed-0065bb722f-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:59.232 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:59.235 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:59.235 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:59.235 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:59.236 DEBUG:boto:Method: GET 2024-02-01T10:27:59.236 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:59.236 DEBUG:boto:Data: 2024-02-01T10:27:59.236 DEBUG:boto:Headers: {} 2024-02-01T10:27:59.236 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:59.236 DEBUG:boto:Port: 8000 2024-02-01T10:27:59.236 DEBUG:boto:Params: {} 2024-02-01T10:27:59.236 DEBUG:boto:Token: None 2024-02-01T10:27:59.236 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:59 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:59.237 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:qaaTCVBC1I5W/9elBzPRtmzsbxU= 2024-02-01T10:27:59.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:qaaTCVBC1I5W/9elBzPRtmzsbxU=', 'Content-Length': '0'} 2024-02-01T10:27:59.239 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000041bec2859eafd51f-0065bb722f-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:59.240 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:59.240 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:59.240 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:59.240 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:59.240 DEBUG:boto:Method: GET 2024-02-01T10:27:59.240 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:59.240 DEBUG:boto:Data: 2024-02-01T10:27:59.241 DEBUG:boto:Headers: {} 2024-02-01T10:27:59.241 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:59.241 DEBUG:boto:Port: 8001 2024-02-01T10:27:59.241 DEBUG:boto:Params: {} 2024-02-01T10:27:59.241 DEBUG:boto:Token: None 2024-02-01T10:27:59.241 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:59 GMT /vclfcn-19/ 2024-02-01T10:27:59.241 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:e66F8l/8ybxeRJYXSVu3Ne/Y9I4= 2024-02-01T10:27:59.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:e66F8l/8ybxeRJYXSVu3Ne/Y9I4=', 'Content-Length': '0'} 2024-02-01T10:27:59.246 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000034a328ff352551a5-0065bb722f-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:59.247 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:59.247 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:59.247 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:59.247 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:59.247 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:59.247 DEBUG:boto:Method: GET 2024-02-01T10:27:59.248 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:59.248 DEBUG:boto:Data: 2024-02-01T10:27:59.248 DEBUG:boto:Headers: {} 2024-02-01T10:27:59.248 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:59.248 DEBUG:boto:Port: 8000 2024-02-01T10:27:59.248 DEBUG:boto:Params: {} 2024-02-01T10:27:59.248 DEBUG:boto:Token: None 2024-02-01T10:27:59.248 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:59 GMT /vclfcn-19/ 2024-02-01T10:27:59.248 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:e66F8l/8ybxeRJYXSVu3Ne/Y9I4= 2024-02-01T10:27:59.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:e66F8l/8ybxeRJYXSVu3Ne/Y9I4=', 'Content-Length': '0'} 2024-02-01T10:27:59.253 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a05ae3f2dcedad44-0065bb722f-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:59.254 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:59.254 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:59.254 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:59.254 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:59.254 DEBUG:boto:path=/vclfcn-19/obj?versionId=s-qxjv7TkMdxeC61HL2kxUqODSwa-1I 2024-02-01T10:27:59.254 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=s-qxjv7TkMdxeC61HL2kxUqODSwa-1I 2024-02-01T10:27:59.255 DEBUG:boto:Method: GET 2024-02-01T10:27:59.255 DEBUG:boto:Path: /vclfcn-19/obj?versionId=s-qxjv7TkMdxeC61HL2kxUqODSwa-1I 2024-02-01T10:27:59.255 DEBUG:boto:Data: 2024-02-01T10:27:59.255 DEBUG:boto:Headers: {} 2024-02-01T10:27:59.255 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:59.255 DEBUG:boto:Port: 8001 2024-02-01T10:27:59.255 DEBUG:boto:Params: {} 2024-02-01T10:27:59.255 DEBUG:boto:Token: None 2024-02-01T10:27:59.255 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:59 GMT /vclfcn-19/obj?versionId=s-qxjv7TkMdxeC61HL2kxUqODSwa-1I 2024-02-01T10:27:59.255 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NOKGp8wLhH2peHB0XKHrvjbOiwo= 2024-02-01T10:27:59.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NOKGp8wLhH2peHB0XKHrvjbOiwo=', 'Content-Length': '0'} 2024-02-01T10:27:59.258 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('x-amz-version-id', 's-qxjv7TkMdxeC61HL2kxUqODSwa-1I'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b40a04b32b4fba9a-0065bb722f-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:59.259 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:59.259 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:59.259 DEBUG:boto:path=/vclfcn-19/obj?versionId=s-qxjv7TkMdxeC61HL2kxUqODSwa-1I 2024-02-01T10:27:59.259 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=s-qxjv7TkMdxeC61HL2kxUqODSwa-1I 2024-02-01T10:27:59.259 DEBUG:boto:Method: GET 2024-02-01T10:27:59.259 DEBUG:boto:Path: /vclfcn-19/obj?versionId=s-qxjv7TkMdxeC61HL2kxUqODSwa-1I 2024-02-01T10:27:59.259 DEBUG:boto:Data: 2024-02-01T10:27:59.259 DEBUG:boto:Headers: {} 2024-02-01T10:27:59.259 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:59.259 DEBUG:boto:Port: 8000 2024-02-01T10:27:59.259 DEBUG:boto:Params: {} 2024-02-01T10:27:59.260 DEBUG:boto:Token: None 2024-02-01T10:27:59.260 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:59 GMT /vclfcn-19/obj?versionId=s-qxjv7TkMdxeC61HL2kxUqODSwa-1I 2024-02-01T10:27:59.260 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NOKGp8wLhH2peHB0XKHrvjbOiwo= 2024-02-01T10:27:59.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NOKGp8wLhH2peHB0XKHrvjbOiwo=', 'Content-Length': '0'} 2024-02-01T10:27:59.263 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('x-amz-version-id', 's-qxjv7TkMdxeC61HL2kxUqODSwa-1I'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b6c60a212c70dada-0065bb722f-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:59.263 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:59.263 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:59.263 DEBUG:boto:path=/vclfcn-19/obj?versionId=s-qxjv7TkMdxeC61HL2kxUqODSwa-1I 2024-02-01T10:27:59.264 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=s-qxjv7TkMdxeC61HL2kxUqODSwa-1I 2024-02-01T10:27:59.264 DEBUG:boto:Method: HEAD 2024-02-01T10:27:59.264 DEBUG:boto:Path: /vclfcn-19/obj?versionId=s-qxjv7TkMdxeC61HL2kxUqODSwa-1I 2024-02-01T10:27:59.264 DEBUG:boto:Data: 2024-02-01T10:27:59.264 DEBUG:boto:Headers: {} 2024-02-01T10:27:59.264 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:59.264 DEBUG:boto:Port: 8001 2024-02-01T10:27:59.264 DEBUG:boto:Params: {} 2024-02-01T10:27:59.264 DEBUG:boto:Token: None 2024-02-01T10:27:59.264 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:59 GMT /vclfcn-19/obj?versionId=s-qxjv7TkMdxeC61HL2kxUqODSwa-1I 2024-02-01T10:27:59.265 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:yAz1nccF6IB3HBXNEMt5cj0ZCgs= 2024-02-01T10:27:59.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:yAz1nccF6IB3HBXNEMt5cj0ZCgs=', 'Content-Length': '0'} 2024-02-01T10:27:59.267 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('x-amz-version-id', 's-qxjv7TkMdxeC61HL2kxUqODSwa-1I'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005726b4ac7fa5df5a-0065bb722f-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:59.268 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:59.268 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:59.268 DEBUG:boto:path=/vclfcn-19/obj?versionId=s-qxjv7TkMdxeC61HL2kxUqODSwa-1I 2024-02-01T10:27:59.268 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=s-qxjv7TkMdxeC61HL2kxUqODSwa-1I 2024-02-01T10:27:59.268 DEBUG:boto:Method: HEAD 2024-02-01T10:27:59.268 DEBUG:boto:Path: /vclfcn-19/obj?versionId=s-qxjv7TkMdxeC61HL2kxUqODSwa-1I 2024-02-01T10:27:59.268 DEBUG:boto:Data: 2024-02-01T10:27:59.268 DEBUG:boto:Headers: {} 2024-02-01T10:27:59.269 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:59.269 DEBUG:boto:Port: 8000 2024-02-01T10:27:59.269 DEBUG:boto:Params: {} 2024-02-01T10:27:59.269 DEBUG:boto:Token: None 2024-02-01T10:27:59.269 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:59 GMT /vclfcn-19/obj?versionId=s-qxjv7TkMdxeC61HL2kxUqODSwa-1I 2024-02-01T10:27:59.269 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:yAz1nccF6IB3HBXNEMt5cj0ZCgs= 2024-02-01T10:27:59.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:yAz1nccF6IB3HBXNEMt5cj0ZCgs=', 'Content-Length': '0'} 2024-02-01T10:27:59.272 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('x-amz-version-id', 's-qxjv7TkMdxeC61HL2kxUqODSwa-1I'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e1ed9cbd65e7bf82-0065bb722f-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:59.272 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:59.273 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:59.273 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:59.273 DEBUG:boto:path=/vclfcn-19/obj?versionId=s-qxjv7TkMdxeC61HL2kxUqODSwa-1I 2024-02-01T10:27:59.273 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=s-qxjv7TkMdxeC61HL2kxUqODSwa-1I 2024-02-01T10:27:59.273 DEBUG:boto:Method: GET 2024-02-01T10:27:59.273 DEBUG:boto:Path: /vclfcn-19/obj?versionId=s-qxjv7TkMdxeC61HL2kxUqODSwa-1I 2024-02-01T10:27:59.273 DEBUG:boto:Data: 2024-02-01T10:27:59.273 DEBUG:boto:Headers: {} 2024-02-01T10:27:59.273 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:59.273 DEBUG:boto:Port: 8001 2024-02-01T10:27:59.273 DEBUG:boto:Params: {} 2024-02-01T10:27:59.274 DEBUG:boto:Token: None 2024-02-01T10:27:59.274 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:59 GMT /vclfcn-19/obj?versionId=s-qxjv7TkMdxeC61HL2kxUqODSwa-1I 2024-02-01T10:27:59.274 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NOKGp8wLhH2peHB0XKHrvjbOiwo= 2024-02-01T10:27:59.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NOKGp8wLhH2peHB0XKHrvjbOiwo=', 'Content-Length': '0'} 2024-02-01T10:27:59.277 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('x-amz-version-id', 's-qxjv7TkMdxeC61HL2kxUqODSwa-1I'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008ee27a403208749a-0065bb722f-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:59.320 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:59.320 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:59.320 DEBUG:boto:path=/vclfcn-19/obj?versionId=s-qxjv7TkMdxeC61HL2kxUqODSwa-1I 2024-02-01T10:27:59.321 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=s-qxjv7TkMdxeC61HL2kxUqODSwa-1I 2024-02-01T10:27:59.321 DEBUG:boto:Method: GET 2024-02-01T10:27:59.321 DEBUG:boto:Path: /vclfcn-19/obj?versionId=s-qxjv7TkMdxeC61HL2kxUqODSwa-1I 2024-02-01T10:27:59.321 DEBUG:boto:Data: 2024-02-01T10:27:59.321 DEBUG:boto:Headers: {} 2024-02-01T10:27:59.321 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:59.321 DEBUG:boto:Port: 8000 2024-02-01T10:27:59.321 DEBUG:boto:Params: {} 2024-02-01T10:27:59.321 DEBUG:boto:Token: None 2024-02-01T10:27:59.321 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:59 GMT /vclfcn-19/obj?versionId=s-qxjv7TkMdxeC61HL2kxUqODSwa-1I 2024-02-01T10:27:59.322 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NOKGp8wLhH2peHB0XKHrvjbOiwo= 2024-02-01T10:27:59.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NOKGp8wLhH2peHB0XKHrvjbOiwo=', 'Content-Length': '0'} 2024-02-01T10:27:59.324 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:32 GMT'), ('x-amz-version-id', 's-qxjv7TkMdxeC61HL2kxUqODSwa-1I'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001daca3c3ebb502be-0065bb722f-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:59.325 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:59.325 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:59.325 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:59.325 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:59.325 DEBUG:boto:Method: GET 2024-02-01T10:27:59.325 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:59.325 DEBUG:boto:Data: 2024-02-01T10:27:59.325 DEBUG:boto:Headers: {} 2024-02-01T10:27:59.325 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:59.326 DEBUG:boto:Port: 8001 2024-02-01T10:27:59.326 DEBUG:boto:Params: {} 2024-02-01T10:27:59.326 DEBUG:boto:Token: None 2024-02-01T10:27:59.326 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:59 GMT /vclfcn-19/ 2024-02-01T10:27:59.326 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:e66F8l/8ybxeRJYXSVu3Ne/Y9I4= 2024-02-01T10:27:59.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:e66F8l/8ybxeRJYXSVu3Ne/Y9I4=', 'Content-Length': '0'} 2024-02-01T10:27:59.331 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa2887e5ed44b636-0065bb722f-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:59.376 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:59.377 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:59.377 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:59.377 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:59.377 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:59.377 DEBUG:boto:Method: GET 2024-02-01T10:27:59.377 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:59.377 DEBUG:boto:Data: 2024-02-01T10:27:59.377 DEBUG:boto:Headers: {} 2024-02-01T10:27:59.378 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:59.378 DEBUG:boto:Port: 8000 2024-02-01T10:27:59.378 DEBUG:boto:Params: {} 2024-02-01T10:27:59.378 DEBUG:boto:Token: None 2024-02-01T10:27:59.378 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:59 GMT /vclfcn-19/ 2024-02-01T10:27:59.378 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:e66F8l/8ybxeRJYXSVu3Ne/Y9I4= 2024-02-01T10:27:59.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:e66F8l/8ybxeRJYXSVu3Ne/Y9I4=', 'Content-Length': '0'} 2024-02-01T10:27:59.384 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b98cd3e7b327c494-0065bb722f-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:59.384 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:59.384 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:59.385 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:59.385 DEBUG:boto:Method: HEAD 2024-02-01T10:27:59.385 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:59.385 DEBUG:boto:Data: 2024-02-01T10:27:59.385 DEBUG:boto:Headers: {} 2024-02-01T10:27:59.385 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:59.385 DEBUG:boto:Port: 8001 2024-02-01T10:27:59.385 DEBUG:boto:Params: {} 2024-02-01T10:27:59.385 DEBUG:boto:Token: None 2024-02-01T10:27:59.386 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:59 GMT /vclfcn-19/obj 2024-02-01T10:27:59.386 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HIc6WtLeqhaYJCcocvhVpa6d9/U= 2024-02-01T10:27:59.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HIc6WtLeqhaYJCcocvhVpa6d9/U=', 'Content-Length': '0'} 2024-02-01T10:27:59.392 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009601f2f15be433b8-0065bb722f-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:59.392 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:59.392 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:59.392 DEBUG:boto:Method: HEAD 2024-02-01T10:27:59.392 DEBUG:boto:Path: /vclfcn-19/obj 2024-02-01T10:27:59.393 DEBUG:boto:Data: 2024-02-01T10:27:59.393 DEBUG:boto:Headers: {} 2024-02-01T10:27:59.393 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:59.393 DEBUG:boto:Port: 8000 2024-02-01T10:27:59.393 DEBUG:boto:Params: {} 2024-02-01T10:27:59.393 DEBUG:boto:Token: None 2024-02-01T10:27:59.393 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:27:59 GMT /vclfcn-19/obj 2024-02-01T10:27:59.393 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HIc6WtLeqhaYJCcocvhVpa6d9/U= 2024-02-01T10:27:59.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HIc6WtLeqhaYJCcocvhVpa6d9/U=', 'Content-Length': '0'} 2024-02-01T10:27:59.399 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d08c5fda9da3eb10-0065bb722f-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:59.400 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:27:59.400 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:59.400 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:59.400 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:59.400 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:59.400 DEBUG:boto:Method: GET 2024-02-01T10:27:59.400 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:59.400 DEBUG:boto:Data: 2024-02-01T10:27:59.400 DEBUG:boto:Headers: {} 2024-02-01T10:27:59.401 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:59.401 DEBUG:boto:Port: 8001 2024-02-01T10:27:59.401 DEBUG:boto:Params: {} 2024-02-01T10:27:59.401 DEBUG:boto:Token: None 2024-02-01T10:27:59.401 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:59 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:59.401 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:qaaTCVBC1I5W/9elBzPRtmzsbxU= 2024-02-01T10:27:59.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:qaaTCVBC1I5W/9elBzPRtmzsbxU=', 'Content-Length': '0'} 2024-02-01T10:27:59.404 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001f3d3fa05f43e796-0065bb722f-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:59.452 DEBUG:boto:path=/vclfcn-19/obj 2024-02-01T10:27:59.452 DEBUG:boto:auth_path=/vclfcn-19/obj 2024-02-01T10:27:59.452 DEBUG:boto:path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:59.453 DEBUG:boto:auth_path=/vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:59.453 DEBUG:boto:Method: GET 2024-02-01T10:27:59.453 DEBUG:boto:Path: /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:59.453 DEBUG:boto:Data: 2024-02-01T10:27:59.453 DEBUG:boto:Headers: {} 2024-02-01T10:27:59.453 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:59.453 DEBUG:boto:Port: 8000 2024-02-01T10:27:59.453 DEBUG:boto:Params: {} 2024-02-01T10:27:59.454 DEBUG:boto:Token: None 2024-02-01T10:27:59.454 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:59 GMT /vclfcn-19/obj?versionId=23llUchmWTvJKOzZpTkl59nzz.uMOQb 2024-02-01T10:27:59.454 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:qaaTCVBC1I5W/9elBzPRtmzsbxU= 2024-02-01T10:27:59.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:qaaTCVBC1I5W/9elBzPRtmzsbxU=', 'Content-Length': '0'} 2024-02-01T10:27:59.457 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:25:33 GMT'), ('x-amz-version-id', '23llUchmWTvJKOzZpTkl59nzz.uMOQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c0fa0388b8173563-0065bb722f-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:59.457 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:59.457 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:59.457 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:59.457 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:59.458 DEBUG:boto:Method: GET 2024-02-01T10:27:59.458 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:59.458 DEBUG:boto:Data: 2024-02-01T10:27:59.458 DEBUG:boto:Headers: {} 2024-02-01T10:27:59.458 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:27:59.458 DEBUG:boto:Port: 8001 2024-02-01T10:27:59.458 DEBUG:boto:Params: {} 2024-02-01T10:27:59.458 DEBUG:boto:Token: None 2024-02-01T10:27:59.458 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:59 GMT /vclfcn-19/ 2024-02-01T10:27:59.459 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:e66F8l/8ybxeRJYXSVu3Ne/Y9I4= 2024-02-01T10:27:59.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:e66F8l/8ybxeRJYXSVu3Ne/Y9I4=', 'Content-Length': '0'} 2024-02-01T10:27:59.463 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007168f4d17d9fb5a1-0065bb722f-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:59.508 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:59.509 DEBUG:boto:path=/vclfcn-19/ 2024-02-01T10:27:59.509 DEBUG:boto:auth_path=/vclfcn-19/ 2024-02-01T10:27:59.509 DEBUG:boto:path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:59.509 DEBUG:boto:auth_path=/vclfcn-19/?prefix=obj 2024-02-01T10:27:59.509 DEBUG:boto:Method: GET 2024-02-01T10:27:59.509 DEBUG:boto:Path: /vclfcn-19/?prefix=obj 2024-02-01T10:27:59.509 DEBUG:boto:Data: 2024-02-01T10:27:59.509 DEBUG:boto:Headers: {} 2024-02-01T10:27:59.509 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:27:59.509 DEBUG:boto:Port: 8000 2024-02-01T10:27:59.509 DEBUG:boto:Params: {} 2024-02-01T10:27:59.510 DEBUG:boto:Token: None 2024-02-01T10:27:59.510 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:27:59 GMT /vclfcn-19/ 2024-02-01T10:27:59.510 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:e66F8l/8ybxeRJYXSVu3Ne/Y9I4= 2024-02-01T10:27:59.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:27:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:e66F8l/8ybxeRJYXSVu3Ne/Y9I4=', 'Content-Length': '0'} 2024-02-01T10:27:59.515 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bed1290e307b1a9c-0065bb722f-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:27:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:27:59.516 DEBUG:boto:b'vclfcn-19obj1000falseobj2024-02-01T10:25:33.939Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:27:59.516 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-19 zones={test-zone3, test-zone2} 2024-02-01T10:27:59.517 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone1 target=test-zone2 2024-02-01T10:27:59.517 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:27:59.517 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:27:59.517 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:27:59.559 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:27:59.559 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:27:59.644 INFO:teuthology.orchestra.run.smithi148.stdout:[{"marker":"00000000000000000000:00000000000000002204","last_update":"2024-02-01T10:21:48.024326Z"},{"marker":"00000000000000000000:00000000000000001781","last_update":"2024-02-01T10:27:40.055105Z"},{"marker":"00000000000000000000:00000000000000005870","last_update":"2024-02-01T10:25:27.559832Z"},{"marker":"","last_update":"0.000000"}] 2024-02-01T10:27:59.644 DEBUG:rgw_multi.tests:data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000002204', 1: '00000000000000000000:00000000000000001781', 2: '00000000000000000000:00000000000000005870', 3: ''} 2024-02-01T10:27:59.644 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-02-01T10:27:59.645 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:27:59.645 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:27:59.645 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:27:59.690 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:27:59.691 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:27:59.770 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":12760245309917388665},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002204","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:21:48.024326Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000001781","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:27:40.055105Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000005870","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:25:27.559832Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-02-01T10:27:59Z"} 2024-02-01T10:27:59.771 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":12760245309917388665},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002204","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:21:48.024326Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000001781","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:27:40.055105Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000005870","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:25:27.559832Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-02-01T10:27:59Z"} 2024-02-01T10:27:59.771 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002204', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:21:48.024326Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001781', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:27:40.055105Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005870', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:25:27.559832Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2024-02-01T10:27:59.771 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002204', 1: '00000000000000000000:00000000000000001781', 2: '00000000000000000000:00000000000000005870', 3: ''} 2024-02-01T10:27:59.771 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002204', 1: '00000000000000000000:00000000000000001781', 2: '00000000000000000000:00000000000000005870', 3: ''} 2024-02-01T10:27:59.771 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-02-01T10:27:59.772 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone1 target=test-zone3 2024-02-01T10:27:59.772 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:27:59.772 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:27:59.772 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:27:59.814 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:27:59.814 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:27:59.896 INFO:teuthology.orchestra.run.smithi148.stdout:[{"marker":"00000000000000000000:00000000000000002204","last_update":"2024-02-01T10:21:48.024326Z"},{"marker":"00000000000000000000:00000000000000001781","last_update":"2024-02-01T10:27:40.055105Z"},{"marker":"00000000000000000000:00000000000000005870","last_update":"2024-02-01T10:25:27.559832Z"},{"marker":"","last_update":"0.000000"}] 2024-02-01T10:27:59.896 DEBUG:rgw_multi.tests:data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000002204', 1: '00000000000000000000:00000000000000001781', 2: '00000000000000000000:00000000000000005870', 3: ''} 2024-02-01T10:27:59.896 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2024-02-01T10:27:59.896 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:27:59.897 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:27:59.897 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:27:59.977 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:27:59.977 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:28:00.056 INFO:teuthology.orchestra.run.smithi148.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":8365149014541277257},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002204","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:21:48.024326Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000001781","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:27:40.055105Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000005870","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:25:27.559832Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-02-01T10:28:00Z"} 2024-02-01T10:28:00.057 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":8365149014541277257},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002204","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:21:48.024326Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000001781","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:27:40.055105Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000005870","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:25:27.559832Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-02-01T10:28:00Z"} 2024-02-01T10:28:00.057 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002204', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:21:48.024326Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001781', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:27:40.055105Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005870', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:25:27.559832Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2024-02-01T10:28:00.057 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002204', 1: '00000000000000000000:00000000000000001781', 2: '00000000000000000000:00000000000000005870', 3: ''} 2024-02-01T10:28:00.057 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002204', 1: '00000000000000000000:00000000000000001781', 2: '00000000000000000000:00000000000000005870', 3: ''} 2024-02-01T10:28:00.057 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2024-02-01T10:28:00.057 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone2 target=test-zone1 2024-02-01T10:28:00.057 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:28:00.057 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:28:00.058 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:28:00.098 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:28:00.098 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:28:00.183 INFO:teuthology.orchestra.run.smithi187.stdout:[{"marker":"00000000000000000000:00000000000000002486","last_update":"2024-02-01T10:22:17.904424Z"},{"marker":"00000000000000000000:00000000000000001781","last_update":"2024-02-01T10:27:40.135817Z"},{"marker":"00000000000000000000:00000000000000004601","last_update":"2024-02-01T10:24:00.115466Z"},{"marker":"","last_update":"0.000000"}] 2024-02-01T10:28:00.183 DEBUG:rgw_multi.tests:data markers for zone=test-zone2 markers={0: '00000000000000000000:00000000000000002486', 1: '00000000000000000000:00000000000000001781', 2: '00000000000000000000:00000000000000004601', 3: ''} 2024-02-01T10:28:00.184 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2024-02-01T10:28:00.184 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:28:00.184 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:28:00.184 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:28:00.227 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:28:00.227 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:28:00.307 INFO:teuthology.orchestra.run.smithi148.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":8595394196901466828},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002486","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:22:17.904424Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000001781","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:27:40.135817Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004601","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:24:00.115466Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-02-01T10:28:00Z"} 2024-02-01T10:28:00.307 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":8595394196901466828},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002486","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:22:17.904424Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000001781","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:27:40.135817Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004601","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:24:00.115466Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-02-01T10:28:00Z"} 2024-02-01T10:28:00.308 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002486', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:22:17.904424Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001781', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:27:40.135817Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004601', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:24:00.115466Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2024-02-01T10:28:00.308 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002486', 1: '00000000000000000000:00000000000000001781', 2: '00000000000000000000:00000000000000004601', 3: ''} 2024-02-01T10:28:00.308 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002486', 1: '00000000000000000000:00000000000000001781', 2: '00000000000000000000:00000000000000004601', 3: ''} 2024-02-01T10:28:00.308 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2024-02-01T10:28:00.308 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone2 target=test-zone3 2024-02-01T10:28:00.308 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:28:00.308 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:28:00.309 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:28:00.350 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:28:00.350 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:28:00.435 INFO:teuthology.orchestra.run.smithi187.stdout:[{"marker":"00000000000000000000:00000000000000002486","last_update":"2024-02-01T10:22:17.904424Z"},{"marker":"00000000000000000000:00000000000000001781","last_update":"2024-02-01T10:27:40.135817Z"},{"marker":"00000000000000000000:00000000000000004601","last_update":"2024-02-01T10:24:00.115466Z"},{"marker":"","last_update":"0.000000"}] 2024-02-01T10:28:00.435 DEBUG:rgw_multi.tests:data markers for zone=test-zone2 markers={0: '00000000000000000000:00000000000000002486', 1: '00000000000000000000:00000000000000001781', 2: '00000000000000000000:00000000000000004601', 3: ''} 2024-02-01T10:28:00.435 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2024-02-01T10:28:00.435 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:28:00.436 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:28:00.436 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:28:00.481 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:28:00.481 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:28:00.564 INFO:teuthology.orchestra.run.smithi148.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4461638765891857795},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002486","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:22:17.904424Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000001781","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:27:40.135817Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004601","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:24:00.115466Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-02-01T10:28:00Z"} 2024-02-01T10:28:00.564 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4461638765891857795},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002486","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:22:17.904424Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000001781","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:27:40.135817Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004601","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:24:00.115466Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-02-01T10:28:00Z"} 2024-02-01T10:28:00.565 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002486', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:22:17.904424Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001781', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:27:40.135817Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004601', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:24:00.115466Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2024-02-01T10:28:00.565 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002486', 1: '00000000000000000000:00000000000000001781', 2: '00000000000000000000:00000000000000004601', 3: ''} 2024-02-01T10:28:00.565 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002486', 1: '00000000000000000000:00000000000000001781', 2: '00000000000000000000:00000000000000004601', 3: ''} 2024-02-01T10:28:00.565 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2024-02-01T10:28:00.565 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone3 target=test-zone1 2024-02-01T10:28:00.565 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:28:00.565 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:28:00.565 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:28:00.598 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:28:00.598 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:28:00.679 INFO:teuthology.orchestra.run.smithi148.stdout:[{"marker":"00000000000000000000:00000000000000002909","last_update":"2024-02-01T10:21:54.106139Z"},{"marker":"00000000000000000000:00000000000000001922","last_update":"2024-02-01T10:27:24.140179Z"},{"marker":"00000000000000000000:00000000000000005588","last_update":"2024-02-01T10:25:33.664319Z"},{"marker":"","last_update":"0.000000"}] 2024-02-01T10:28:00.679 DEBUG:rgw_multi.tests:data markers for zone=test-zone3 markers={0: '00000000000000000000:00000000000000002909', 1: '00000000000000000000:00000000000000001922', 2: '00000000000000000000:00000000000000005588', 3: ''} 2024-02-01T10:28:00.679 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2024-02-01T10:28:00.680 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:28:00.680 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:28:00.680 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:28:00.759 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:28:00.759 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:28:00.834 INFO:teuthology.orchestra.run.smithi148.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":8579032341556541049},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002909","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:21:54.106139Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000001922","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:27:24.140179Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000005588","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:25:33.664319Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-02-01T10:28:00Z"} 2024-02-01T10:28:00.835 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":8579032341556541049},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002909","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:21:54.106139Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000001922","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:27:24.140179Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000005588","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:25:33.664319Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-02-01T10:28:00Z"} 2024-02-01T10:28:00.835 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002909', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:21:54.106139Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001922', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:27:24.140179Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005588', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:25:33.664319Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2024-02-01T10:28:00.835 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002909', 1: '00000000000000000000:00000000000000001922', 2: '00000000000000000000:00000000000000005588', 3: ''} 2024-02-01T10:28:00.835 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002909', 1: '00000000000000000000:00000000000000001922', 2: '00000000000000000000:00000000000000005588', 3: ''} 2024-02-01T10:28:00.835 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2024-02-01T10:28:00.835 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone3 target=test-zone2 2024-02-01T10:28:00.836 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:28:00.836 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:28:00.836 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:28:00.911 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:28:00.912 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:28:00.984 INFO:teuthology.orchestra.run.smithi148.stdout:[{"marker":"00000000000000000000:00000000000000002909","last_update":"2024-02-01T10:21:54.106139Z"},{"marker":"00000000000000000000:00000000000000001922","last_update":"2024-02-01T10:27:24.140179Z"},{"marker":"00000000000000000000:00000000000000005588","last_update":"2024-02-01T10:25:33.664319Z"},{"marker":"","last_update":"0.000000"}] 2024-02-01T10:28:00.985 DEBUG:rgw_multi.tests:data markers for zone=test-zone3 markers={0: '00000000000000000000:00000000000000002909', 1: '00000000000000000000:00000000000000001922', 2: '00000000000000000000:00000000000000005588', 3: ''} 2024-02-01T10:28:00.985 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2024-02-01T10:28:00.985 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:28:00.985 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:28:00.985 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:28:01.030 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:28:01.030 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:28:01.111 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":843725689139632598},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002909","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:21:54.106139Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000001922","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:27:24.140179Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000005588","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:25:33.664319Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-02-01T10:28:01Z"} 2024-02-01T10:28:01.112 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":843725689139632598},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002909","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:21:54.106139Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000001922","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:27:24.140179Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000005588","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:25:33.664319Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-02-01T10:28:01Z"} 2024-02-01T10:28:01.112 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002909', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:21:54.106139Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001922', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:27:24.140179Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005588', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:25:33.664319Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2024-02-01T10:28:01.112 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002909', 1: '00000000000000000000:00000000000000001922', 2: '00000000000000000000:00000000000000005588', 3: ''} 2024-02-01T10:28:01.112 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002909', 1: '00000000000000000000:00000000000000001922', 2: '00000000000000000000:00000000000000005588', 3: ''} 2024-02-01T10:28:01.112 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2024-02-01T10:28:01.113 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_concurrent_versioned_object_incremental_sync ... ok 2024-02-01T10:28:01.116 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:28:01.116 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:28:01.117 DEBUG:boto:Method: PUT 2024-02-01T10:28:01.117 DEBUG:boto:Path: /vclfcn-20/ 2024-02-01T10:28:01.117 DEBUG:boto:Data: 2024-02-01T10:28:01.117 DEBUG:boto:Headers: {} 2024-02-01T10:28:01.117 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:28:01.117 DEBUG:boto:Port: 8000 2024-02-01T10:28:01.117 DEBUG:boto:Params: {} 2024-02-01T10:28:01.117 DEBUG:boto:Token: None 2024-02-01T10:28:01.117 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:28:01 GMT /vclfcn-20/ 2024-02-01T10:28:01.117 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1Lp7sw2FkZkdDNZOVN5QznAk1NE= 2024-02-01T10:28:01.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:28:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1Lp7sw2FkZkdDNZOVN5QznAk1NE=', 'Content-Length': '0'} 2024-02-01T10:28:01.151 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001389267fd458f6b0-0065bb7231-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:28:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:28:01.152 DEBUG:rgw_multi.tests:created bucket=vclfcn-20 2024-02-01T10:28:01.153 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:28:01.153 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:28:01.153 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:28:01.191 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:28:01.191 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:28:01.241 INFO:teuthology.orchestra.run.smithi148.stderr:No --period given, using current period=5dc4efbd-eccd-47a8-9471-ae4a209ec978 2024-02-01T10:28:01.262 INFO:teuthology.orchestra.run.smithi148.stdout:[{"marker":"1_1706782453.132619_30.1","last_update":"2024-02-01T10:14:13.132619Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1706783281.156355_115.1","last_update":"2024-02-01T10:28:01.156355Z"},{"marker":"1_1706782595.670952_121.1","last_update":"2024-02-01T10:16:35.670952Z"}] 2024-02-01T10:28:01.263 DEBUG:rgw_multi.tests:master meta markers={0: '1_1706782453.132619_30.1', 1: '', 2: '1_1706783281.156355_115.1', 3: '1_1706782595.670952_121.1'} 2024-02-01T10:28:01.263 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-02-01T10:28:01.263 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:28:01.263 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:28:01.263 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:28:01.309 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:28:01.309 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:28:01.370 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:28:01.372+0000 7fb91e966980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:28:01.371 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:28:01.376+0000 7fb91e966980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:28:01.394 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783281.156355_115.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:28:01.156355Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:28:01Z"} 2024-02-01T10:28:01.395 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783281.156355_115.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:28:01.156355Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:28:01Z"} 2024-02-01T10:28:01.395 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706783281.156355_115.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:28:01.156355Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:28:01.395 DEBUG:rgw_multi.tests:log_status={0: '1_1706782453.132619_30.1', 1: '', 2: '1_1706783281.156355_115.1', 3: '1_1706782595.670952_121.1'} 2024-02-01T10:28:01.395 DEBUG:rgw_multi.tests:sync_status={0: '1_1706782453.132619_30.1', 1: '', 2: '1_1706783281.156355_115.1', 3: '1_1706782595.670952_121.1'} 2024-02-01T10:28:01.395 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-02-01T10:28:01.395 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-02-01T10:28:01.395 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:28:01.395 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:28:01.396 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:28:01.434 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:28:01.434 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:28:01.483 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:28:01.488+0000 7f97603d7980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:28:01.484 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:28:01.492+0000 7f97603d7980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:28:01.504 INFO:teuthology.orchestra.run.smithi148.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783281.156355_115.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:28:01.156355Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:28:01Z"} 2024-02-01T10:28:01.504 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783281.156355_115.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:28:01.156355Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:28:01Z"} 2024-02-01T10:28:01.505 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706783281.156355_115.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:28:01.156355Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:28:01.505 DEBUG:rgw_multi.tests:log_status={0: '1_1706782453.132619_30.1', 1: '', 2: '1_1706783281.156355_115.1', 3: '1_1706782595.670952_121.1'} 2024-02-01T10:28:01.505 DEBUG:rgw_multi.tests:sync_status={0: '1_1706782453.132619_30.1', 1: '', 2: '1_1706783281.156355_115.1', 3: '1_1706782595.670952_121.1'} 2024-02-01T10:28:01.505 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-02-01T10:28:01.505 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:28:01.505 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:28:01.505 DEBUG:boto:Method: HEAD 2024-02-01T10:28:01.505 DEBUG:boto:Path: /vclfcn-20/ 2024-02-01T10:28:01.505 DEBUG:boto:Data: 2024-02-01T10:28:01.506 DEBUG:boto:Headers: {} 2024-02-01T10:28:01.506 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:28:01.506 DEBUG:boto:Port: 8000 2024-02-01T10:28:01.506 DEBUG:boto:Params: {} 2024-02-01T10:28:01.506 DEBUG:boto:Token: None 2024-02-01T10:28:01.506 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:28:01 GMT /vclfcn-20/ 2024-02-01T10:28:01.506 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:V/QCNlcjtOwXHDPIVOUv5JPkImc= 2024-02-01T10:28:01.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:28:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:V/QCNlcjtOwXHDPIVOUv5JPkImc=', 'Content-Length': '0'} 2024-02-01T10:28:01.508 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c43d8061b27c5a76-0065bb7231-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:28:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:28:01.509 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:28:01.509 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:28:01.509 DEBUG:boto:Method: PUT 2024-02-01T10:28:01.509 DEBUG:boto:Path: /vclfcn-20/obj 2024-02-01T10:28:01.509 DEBUG:boto:Data: 2024-02-01T10:28:01.509 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-02-01T10:28:01.509 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:28:01.509 DEBUG:boto:Port: 8000 2024-02-01T10:28:01.509 DEBUG:boto:Params: {} 2024-02-01T10:28:01.510 DEBUG:boto:Token: None 2024-02-01T10:28:01.510 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Thu, 01 Feb 2024 10:28:01 GMT /vclfcn-20/obj 2024-02-01T10:28:01.510 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9+Q2C8ShSITKTlBEuLsFC6f5WJY= 2024-02-01T10:28:01.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:28:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9+Q2C8ShSITKTlBEuLsFC6f5WJY='} 2024-02-01T10:28:01.515 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f2f5a98570c61c70-0065bb7231-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:28:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:28:01.516 DEBUG:rgw_multi.tests:created initial version id=None 2024-02-01T10:28:01.516 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=vclfcn-20 2024-02-01T10:28:01.516 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:28:01.516 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:28:01.517 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket vclfcn-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-02-01T10:28:01.568 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:28:01.568 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:28:01.631 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:28:01.632+0000 7f11d3a6a980 1 waiting to reach incremental sync.. 2024-02-01T10:28:06.632 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:28:06.636+0000 7f11d3a6a980 1 waiting to reach incremental sync.. 2024-02-01T10:28:11.635 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:28:11.640+0000 7f11d3a6a980 1 bucket sync caught up with source: 2024-02-01T10:28:11.636 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, , , , , , , , 00000000001.34.6, , ] 2024-02-01T10:28:11.636 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, , , , , , , , 00000000001.34.6, , ] 2024-02-01T10:28:11.636 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:28:11.640+0000 7f11d3a6a980 0 bucket checkpoint complete 2024-02-01T10:28:11.671 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=vclfcn-20 2024-02-01T10:28:11.672 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:28:11.672 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:28:11.672 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:28:11.722 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:28:11.723 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:28:11.787 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:28:11.791+0000 7fcec3613980 1 waiting to reach incremental sync.. 2024-02-01T10:28:16.791 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:28:16.795+0000 7fcec3613980 1 bucket sync caught up with source: 2024-02-01T10:28:16.791 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, , , , , , , , 00000000001.34.6, , ] 2024-02-01T10:28:16.791 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, , , , , , , , 00000000001.34.6, , ] 2024-02-01T10:28:16.791 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:28:16.795+0000 7fcec3613980 0 bucket checkpoint complete 2024-02-01T10:28:16.827 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=vclfcn-20 2024-02-01T10:28:16.827 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:28:16.827 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:28:16.827 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:28:16.874 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:28:16.874 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:28:16.934 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:28:16.939+0000 7f2d738e7980 1 waiting to reach incremental sync.. 2024-02-01T10:28:21.935 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:28:21.939+0000 7f2d738e7980 1 waiting to reach incremental sync.. 2024-02-01T10:28:26.939 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:28:26.943+0000 7f2d738e7980 1 bucket sync caught up with source: 2024-02-01T10:28:26.939 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, , , , , , , , 00000000001.37.6, , ] 2024-02-01T10:28:26.939 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, , , , , , , , 00000000001.37.6, , ] 2024-02-01T10:28:26.939 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:28:26.943+0000 7f2d738e7980 0 bucket checkpoint complete 2024-02-01T10:28:26.984 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=vclfcn-20 2024-02-01T10:28:26.984 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:28:26.984 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:28:26.985 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:28:27.034 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:28:27.034 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:28:27.099 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:28:27.103+0000 7f9b9451e980 1 bucket sync caught up with source: 2024-02-01T10:28:27.099 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, , , , , , , , 00000000001.37.6, , ] 2024-02-01T10:28:27.099 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, , , , , , , , 00000000001.37.6, , ] 2024-02-01T10:28:27.099 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:28:27.103+0000 7f9b9451e980 0 bucket checkpoint complete 2024-02-01T10:28:27.122 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=vclfcn-20 2024-02-01T10:28:27.122 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:28:27.122 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:28:27.122 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:28:27.162 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:28:27.163 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:28:27.218 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:28:27.223+0000 7f748fbd9980 1 bucket sync caught up with source: 2024-02-01T10:28:27.219 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, , , , , , , , 00000000001.69.6, , ] 2024-02-01T10:28:27.219 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, , , , , , , , 00000000001.69.6, , ] 2024-02-01T10:28:27.219 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:28:27.223+0000 7f748fbd9980 0 bucket checkpoint complete 2024-02-01T10:28:27.241 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=vclfcn-20 2024-02-01T10:28:27.242 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:28:27.242 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:28:27.242 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket vclfcn-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-02-01T10:28:27.291 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:28:27.292 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:28:27.358 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:28:27.359+0000 7f28ee0bc980 1 waiting to reach incremental sync.. 2024-02-01T10:28:32.361 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:28:32.363+0000 7f28ee0bc980 1 bucket sync caught up with source: 2024-02-01T10:28:32.361 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, , , , , , , , 00000000001.69.6, , ] 2024-02-01T10:28:32.361 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, , , , , , , , 00000000001.69.6, , ] 2024-02-01T10:28:32.361 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:28:32.363+0000 7f28ee0bc980 0 bucket checkpoint complete 2024-02-01T10:28:32.397 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-20 zones={test-zone2, test-zone1} 2024-02-01T10:28:32.398 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:28:32.398 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:28:32.398 DEBUG:boto:Method: HEAD 2024-02-01T10:28:32.398 DEBUG:boto:Path: /vclfcn-20/ 2024-02-01T10:28:32.398 DEBUG:boto:Data: 2024-02-01T10:28:32.398 DEBUG:boto:Headers: {} 2024-02-01T10:28:32.398 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:28:32.398 DEBUG:boto:Port: 8000 2024-02-01T10:28:32.399 DEBUG:boto:Params: {} 2024-02-01T10:28:32.399 DEBUG:boto:Token: None 2024-02-01T10:28:32.399 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:28:32 GMT /vclfcn-20/ 2024-02-01T10:28:32.399 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Uwna1EOMEU3ZFTPJD6bxuWRAe1U= 2024-02-01T10:28:32.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:28:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Uwna1EOMEU3ZFTPJD6bxuWRAe1U=', 'Content-Length': '0'} 2024-02-01T10:28:32.403 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d6ce14de0c430efe-0065bb7250-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:28:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:28:32.403 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:28:32.403 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:28:32.403 DEBUG:boto:Method: HEAD 2024-02-01T10:28:32.403 DEBUG:boto:Path: /vclfcn-20/ 2024-02-01T10:28:32.403 DEBUG:boto:Data: 2024-02-01T10:28:32.403 DEBUG:boto:Headers: {} 2024-02-01T10:28:32.403 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:28:32.403 DEBUG:boto:Port: 8000 2024-02-01T10:28:32.404 DEBUG:boto:Params: {} 2024-02-01T10:28:32.404 DEBUG:boto:Token: None 2024-02-01T10:28:32.404 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:28:32 GMT /vclfcn-20/ 2024-02-01T10:28:32.404 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Uwna1EOMEU3ZFTPJD6bxuWRAe1U= 2024-02-01T10:28:32.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:28:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Uwna1EOMEU3ZFTPJD6bxuWRAe1U=', 'Content-Length': '0'} 2024-02-01T10:28:32.408 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007341fbc014e7c3f7-0065bb7250-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:28:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:28:32.408 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:28:32.409 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:28:32.409 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:28:32.409 DEBUG:boto:path=/vclfcn-20/?versions&max-keys=999 2024-02-01T10:28:32.409 DEBUG:boto:auth_path=/vclfcn-20/?versions&max-keys=999 2024-02-01T10:28:32.409 DEBUG:boto:Method: GET 2024-02-01T10:28:32.409 DEBUG:boto:Path: /vclfcn-20/?versions&max-keys=999 2024-02-01T10:28:32.409 DEBUG:boto:Data: 2024-02-01T10:28:32.409 DEBUG:boto:Headers: {} 2024-02-01T10:28:32.409 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:28:32.409 DEBUG:boto:Port: 8000 2024-02-01T10:28:32.409 DEBUG:boto:Params: {} 2024-02-01T10:28:32.410 DEBUG:boto:Token: None 2024-02-01T10:28:32.410 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:28:32 GMT /vclfcn-20/?versions 2024-02-01T10:28:32.410 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zC7B0wUvHWWBbWJNubCwj+1F5yU= 2024-02-01T10:28:32.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:28:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zC7B0wUvHWWBbWJNubCwj+1F5yU=', 'Content-Length': '0'} 2024-02-01T10:28:32.413 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a64a1b0e7d46f573-0065bb7250-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:28:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:28:32.413 DEBUG:boto:b'vclfcn-20999falseobjnulltrue2024-02-01T10:28:01.521Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:28:32.414 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:28:32.414 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:28:32.414 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:28:32.414 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:28:32.414 DEBUG:boto:path=/vclfcn-20/?versions&max-keys=999 2024-02-01T10:28:32.414 DEBUG:boto:auth_path=/vclfcn-20/?versions&max-keys=999 2024-02-01T10:28:32.414 DEBUG:boto:Method: GET 2024-02-01T10:28:32.414 DEBUG:boto:Path: /vclfcn-20/?versions&max-keys=999 2024-02-01T10:28:32.414 DEBUG:boto:Data: 2024-02-01T10:28:32.414 DEBUG:boto:Headers: {} 2024-02-01T10:28:32.415 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:28:32.415 DEBUG:boto:Port: 8000 2024-02-01T10:28:32.415 DEBUG:boto:Params: {} 2024-02-01T10:28:32.415 DEBUG:boto:Token: None 2024-02-01T10:28:32.415 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:28:32 GMT /vclfcn-20/?versions 2024-02-01T10:28:32.415 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zC7B0wUvHWWBbWJNubCwj+1F5yU= 2024-02-01T10:28:32.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:28:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zC7B0wUvHWWBbWJNubCwj+1F5yU=', 'Content-Length': '0'} 2024-02-01T10:28:32.419 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041b73816c0ab0401-0065bb7250-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:28:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:28:32.419 DEBUG:boto:b'vclfcn-20999falseobjnulltrue2024-02-01T10:28:01.521Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:28:32.420 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:28:32.420 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:28:32.420 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:28:32.420 DEBUG:boto:path=/vclfcn-20/?versions&max-keys=999 2024-02-01T10:28:32.420 DEBUG:boto:auth_path=/vclfcn-20/?versions&max-keys=999 2024-02-01T10:28:32.420 DEBUG:boto:Method: GET 2024-02-01T10:28:32.420 DEBUG:boto:Path: /vclfcn-20/?versions&max-keys=999 2024-02-01T10:28:32.421 DEBUG:boto:Data: 2024-02-01T10:28:32.421 DEBUG:boto:Headers: {} 2024-02-01T10:28:32.421 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:28:32.421 DEBUG:boto:Port: 8000 2024-02-01T10:28:32.421 DEBUG:boto:Params: {} 2024-02-01T10:28:32.421 DEBUG:boto:Token: None 2024-02-01T10:28:32.421 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:28:32 GMT /vclfcn-20/?versions 2024-02-01T10:28:32.421 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zC7B0wUvHWWBbWJNubCwj+1F5yU= 2024-02-01T10:28:32.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:28:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zC7B0wUvHWWBbWJNubCwj+1F5yU=', 'Content-Length': '0'} 2024-02-01T10:28:32.425 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6883a067915e864-0065bb7250-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:28:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:28:32.425 DEBUG:boto:b'vclfcn-20999falseobjnulltrue2024-02-01T10:28:01.521Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:28:32.425 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:28:32.426 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:28:32.426 DEBUG:boto:path=/vclfcn-20/?versions&max-keys=999 2024-02-01T10:28:32.426 DEBUG:boto:auth_path=/vclfcn-20/?versions&max-keys=999 2024-02-01T10:28:32.426 DEBUG:boto:Method: GET 2024-02-01T10:28:32.426 DEBUG:boto:Path: /vclfcn-20/?versions&max-keys=999 2024-02-01T10:28:32.426 DEBUG:boto:Data: 2024-02-01T10:28:32.426 DEBUG:boto:Headers: {} 2024-02-01T10:28:32.426 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:28:32.426 DEBUG:boto:Port: 8000 2024-02-01T10:28:32.426 DEBUG:boto:Params: {} 2024-02-01T10:28:32.426 DEBUG:boto:Token: None 2024-02-01T10:28:32.427 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:28:32 GMT /vclfcn-20/?versions 2024-02-01T10:28:32.427 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zC7B0wUvHWWBbWJNubCwj+1F5yU= 2024-02-01T10:28:32.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:28:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zC7B0wUvHWWBbWJNubCwj+1F5yU=', 'Content-Length': '0'} 2024-02-01T10:28:32.431 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000379e3fe08e6f14bd-0065bb7250-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:28:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:28:32.472 DEBUG:boto:b'vclfcn-20999falseobjnulltrue2024-02-01T10:28:01.521Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:28:32.472 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:28:32.473 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:28:32.473 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:28:32.473 DEBUG:boto:path=/vclfcn-20/obj?versionId=null 2024-02-01T10:28:32.473 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=null 2024-02-01T10:28:32.473 DEBUG:boto:Method: GET 2024-02-01T10:28:32.473 DEBUG:boto:Path: /vclfcn-20/obj?versionId=null 2024-02-01T10:28:32.473 DEBUG:boto:Data: 2024-02-01T10:28:32.473 DEBUG:boto:Headers: {} 2024-02-01T10:28:32.473 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:28:32.473 DEBUG:boto:Port: 8000 2024-02-01T10:28:32.474 DEBUG:boto:Params: {} 2024-02-01T10:28:32.474 DEBUG:boto:Token: None 2024-02-01T10:28:32.474 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:28:32 GMT /vclfcn-20/obj?versionId=null 2024-02-01T10:28:32.474 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HlGQNqZGW6oyMaE71lIuC/ZFszw= 2024-02-01T10:28:32.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:28:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HlGQNqZGW6oyMaE71lIuC/ZFszw=', 'Content-Length': '0'} 2024-02-01T10:28:32.479 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000064cd92425547884-0065bb7250-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:28:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:28:32.480 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:28:32.480 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:28:32.480 DEBUG:boto:path=/vclfcn-20/obj?versionId=null 2024-02-01T10:28:32.480 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=null 2024-02-01T10:28:32.480 DEBUG:boto:Method: GET 2024-02-01T10:28:32.480 DEBUG:boto:Path: /vclfcn-20/obj?versionId=null 2024-02-01T10:28:32.480 DEBUG:boto:Data: 2024-02-01T10:28:32.481 DEBUG:boto:Headers: {} 2024-02-01T10:28:32.481 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:28:32.481 DEBUG:boto:Port: 8000 2024-02-01T10:28:32.481 DEBUG:boto:Params: {} 2024-02-01T10:28:32.481 DEBUG:boto:Token: None 2024-02-01T10:28:32.481 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:28:32 GMT /vclfcn-20/obj?versionId=null 2024-02-01T10:28:32.481 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HlGQNqZGW6oyMaE71lIuC/ZFszw= 2024-02-01T10:28:32.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:28:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HlGQNqZGW6oyMaE71lIuC/ZFszw=', 'Content-Length': '0'} 2024-02-01T10:28:32.484 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004871e54bb9433b3a-0065bb7250-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:28:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:28:32.484 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:28:32.484 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:28:32.485 DEBUG:boto:path=/vclfcn-20/obj?versionId=null 2024-02-01T10:28:32.485 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=null 2024-02-01T10:28:32.485 DEBUG:boto:Method: HEAD 2024-02-01T10:28:32.485 DEBUG:boto:Path: /vclfcn-20/obj?versionId=null 2024-02-01T10:28:32.485 DEBUG:boto:Data: 2024-02-01T10:28:32.485 DEBUG:boto:Headers: {} 2024-02-01T10:28:32.485 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:28:32.485 DEBUG:boto:Port: 8000 2024-02-01T10:28:32.485 DEBUG:boto:Params: {} 2024-02-01T10:28:32.485 DEBUG:boto:Token: None 2024-02-01T10:28:32.486 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:28:32 GMT /vclfcn-20/obj?versionId=null 2024-02-01T10:28:32.486 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:s6X0UdFbNRuC9s68m5IE7Qqe4t4= 2024-02-01T10:28:32.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:28:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:s6X0UdFbNRuC9s68m5IE7Qqe4t4=', 'Content-Length': '0'} 2024-02-01T10:28:32.488 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cf807708b80f2749-0065bb7250-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:28:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:28:32.489 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:28:32.489 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:28:32.489 DEBUG:boto:path=/vclfcn-20/obj?versionId=null 2024-02-01T10:28:32.489 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=null 2024-02-01T10:28:32.489 DEBUG:boto:Method: HEAD 2024-02-01T10:28:32.489 DEBUG:boto:Path: /vclfcn-20/obj?versionId=null 2024-02-01T10:28:32.489 DEBUG:boto:Data: 2024-02-01T10:28:32.490 DEBUG:boto:Headers: {} 2024-02-01T10:28:32.490 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:28:32.490 DEBUG:boto:Port: 8000 2024-02-01T10:28:32.490 DEBUG:boto:Params: {} 2024-02-01T10:28:32.490 DEBUG:boto:Token: None 2024-02-01T10:28:32.490 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:28:32 GMT /vclfcn-20/obj?versionId=null 2024-02-01T10:28:32.490 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:s6X0UdFbNRuC9s68m5IE7Qqe4t4= 2024-02-01T10:28:32.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:28:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:s6X0UdFbNRuC9s68m5IE7Qqe4t4=', 'Content-Length': '0'} 2024-02-01T10:28:32.493 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004023dc21e4040e91-0065bb7250-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:28:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:28:32.494 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:28:32.494 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:28:32.494 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:28:32.494 DEBUG:boto:path=/vclfcn-20/obj?versionId=null 2024-02-01T10:28:32.494 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=null 2024-02-01T10:28:32.494 DEBUG:boto:Method: GET 2024-02-01T10:28:32.494 DEBUG:boto:Path: /vclfcn-20/obj?versionId=null 2024-02-01T10:28:32.494 DEBUG:boto:Data: 2024-02-01T10:28:32.495 DEBUG:boto:Headers: {} 2024-02-01T10:28:32.495 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:28:32.495 DEBUG:boto:Port: 8000 2024-02-01T10:28:32.495 DEBUG:boto:Params: {} 2024-02-01T10:28:32.495 DEBUG:boto:Token: None 2024-02-01T10:28:32.495 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:28:32 GMT /vclfcn-20/obj?versionId=null 2024-02-01T10:28:32.495 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HlGQNqZGW6oyMaE71lIuC/ZFszw= 2024-02-01T10:28:32.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:28:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HlGQNqZGW6oyMaE71lIuC/ZFszw=', 'Content-Length': '0'} 2024-02-01T10:28:32.498 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000006a5b42eb71a2968-0065bb7250-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:28:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:28:32.498 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:28:32.498 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:28:32.498 DEBUG:boto:path=/vclfcn-20/obj?versionId=null 2024-02-01T10:28:32.499 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=null 2024-02-01T10:28:32.499 DEBUG:boto:Method: GET 2024-02-01T10:28:32.499 DEBUG:boto:Path: /vclfcn-20/obj?versionId=null 2024-02-01T10:28:32.499 DEBUG:boto:Data: 2024-02-01T10:28:32.499 DEBUG:boto:Headers: {} 2024-02-01T10:28:32.499 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:28:32.499 DEBUG:boto:Port: 8000 2024-02-01T10:28:32.499 DEBUG:boto:Params: {} 2024-02-01T10:28:32.499 DEBUG:boto:Token: None 2024-02-01T10:28:32.499 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:28:32 GMT /vclfcn-20/obj?versionId=null 2024-02-01T10:28:32.499 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HlGQNqZGW6oyMaE71lIuC/ZFszw= 2024-02-01T10:28:32.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:28:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HlGQNqZGW6oyMaE71lIuC/ZFszw=', 'Content-Length': '0'} 2024-02-01T10:28:32.503 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e618d461b7df06d6-0065bb7250-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:28:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:28:32.503 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:28:32.503 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:28:32.503 DEBUG:boto:path=/vclfcn-20/?prefix=obj 2024-02-01T10:28:32.503 DEBUG:boto:auth_path=/vclfcn-20/?prefix=obj 2024-02-01T10:28:32.504 DEBUG:boto:Method: GET 2024-02-01T10:28:32.504 DEBUG:boto:Path: /vclfcn-20/?prefix=obj 2024-02-01T10:28:32.504 DEBUG:boto:Data: 2024-02-01T10:28:32.504 DEBUG:boto:Headers: {} 2024-02-01T10:28:32.504 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:28:32.504 DEBUG:boto:Port: 8000 2024-02-01T10:28:32.504 DEBUG:boto:Params: {} 2024-02-01T10:28:32.504 DEBUG:boto:Token: None 2024-02-01T10:28:32.504 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:28:32 GMT /vclfcn-20/ 2024-02-01T10:28:32.505 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:2iBjUBEfhbWtVzq+KPYPaZeBG+4= 2024-02-01T10:28:32.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:28:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2iBjUBEfhbWtVzq+KPYPaZeBG+4=', 'Content-Length': '0'} 2024-02-01T10:28:32.508 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d678372a02fddc9e-0065bb7250-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:28:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:28:32.552 DEBUG:boto:b'vclfcn-20obj1000falseobj2024-02-01T10:28:01.521Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:28:32.553 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:28:32.553 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:28:32.553 DEBUG:boto:path=/vclfcn-20/?prefix=obj 2024-02-01T10:28:32.553 DEBUG:boto:auth_path=/vclfcn-20/?prefix=obj 2024-02-01T10:28:32.553 DEBUG:boto:Method: GET 2024-02-01T10:28:32.553 DEBUG:boto:Path: /vclfcn-20/?prefix=obj 2024-02-01T10:28:32.553 DEBUG:boto:Data: 2024-02-01T10:28:32.553 DEBUG:boto:Headers: {} 2024-02-01T10:28:32.554 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:28:32.554 DEBUG:boto:Port: 8000 2024-02-01T10:28:32.554 DEBUG:boto:Params: {} 2024-02-01T10:28:32.554 DEBUG:boto:Token: None 2024-02-01T10:28:32.554 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:28:32 GMT /vclfcn-20/ 2024-02-01T10:28:32.554 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:2iBjUBEfhbWtVzq+KPYPaZeBG+4= 2024-02-01T10:28:32.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:28:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2iBjUBEfhbWtVzq+KPYPaZeBG+4=', 'Content-Length': '0'} 2024-02-01T10:28:32.558 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3dc5e6942060cad-0065bb7250-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:28:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:28:32.559 DEBUG:boto:b'vclfcn-20obj1000falseobj2024-02-01T10:28:01.521Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:28:32.559 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:28:32.559 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:28:32.559 DEBUG:boto:Method: HEAD 2024-02-01T10:28:32.560 DEBUG:boto:Path: /vclfcn-20/obj 2024-02-01T10:28:32.560 DEBUG:boto:Data: 2024-02-01T10:28:32.560 DEBUG:boto:Headers: {} 2024-02-01T10:28:32.560 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:28:32.560 DEBUG:boto:Port: 8000 2024-02-01T10:28:32.560 DEBUG:boto:Params: {} 2024-02-01T10:28:32.560 DEBUG:boto:Token: None 2024-02-01T10:28:32.560 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:28:32 GMT /vclfcn-20/obj 2024-02-01T10:28:32.560 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+FZ9ftT0IJFvqiNBW3YmhaIWEnw= 2024-02-01T10:28:32.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:28:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+FZ9ftT0IJFvqiNBW3YmhaIWEnw=', 'Content-Length': '0'} 2024-02-01T10:28:32.563 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005e9bbcc60a9f8c85-0065bb7250-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:28:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:28:32.563 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:28:32.564 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:28:32.564 DEBUG:boto:Method: HEAD 2024-02-01T10:28:32.564 DEBUG:boto:Path: /vclfcn-20/obj 2024-02-01T10:28:32.564 DEBUG:boto:Data: 2024-02-01T10:28:32.564 DEBUG:boto:Headers: {} 2024-02-01T10:28:32.564 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:28:32.564 DEBUG:boto:Port: 8000 2024-02-01T10:28:32.564 DEBUG:boto:Params: {} 2024-02-01T10:28:32.564 DEBUG:boto:Token: None 2024-02-01T10:28:32.565 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:28:32 GMT /vclfcn-20/obj 2024-02-01T10:28:32.565 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+FZ9ftT0IJFvqiNBW3YmhaIWEnw= 2024-02-01T10:28:32.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:28:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+FZ9ftT0IJFvqiNBW3YmhaIWEnw=', 'Content-Length': '0'} 2024-02-01T10:28:32.567 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c75c5b7b6386012e-0065bb7250-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:28:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:28:32.568 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:28:32.568 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:28:32.568 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:28:32.568 DEBUG:boto:Method: GET 2024-02-01T10:28:32.568 DEBUG:boto:Path: /vclfcn-20/obj 2024-02-01T10:28:32.568 DEBUG:boto:Data: 2024-02-01T10:28:32.568 DEBUG:boto:Headers: {} 2024-02-01T10:28:32.568 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:28:32.568 DEBUG:boto:Port: 8000 2024-02-01T10:28:32.569 DEBUG:boto:Params: {} 2024-02-01T10:28:32.569 DEBUG:boto:Token: None 2024-02-01T10:28:32.569 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:28:32 GMT /vclfcn-20/obj 2024-02-01T10:28:32.569 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WAZhAdmoH4UQlmhY5yMy/UEDyCA= 2024-02-01T10:28:32.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:28:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WAZhAdmoH4UQlmhY5yMy/UEDyCA=', 'Content-Length': '0'} 2024-02-01T10:28:32.572 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e842708b22cf698d-0065bb7250-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:28:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:28:32.572 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:28:32.572 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:28:32.572 DEBUG:boto:Method: GET 2024-02-01T10:28:32.572 DEBUG:boto:Path: /vclfcn-20/obj 2024-02-01T10:28:32.572 DEBUG:boto:Data: 2024-02-01T10:28:32.573 DEBUG:boto:Headers: {} 2024-02-01T10:28:32.573 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:28:32.573 DEBUG:boto:Port: 8000 2024-02-01T10:28:32.573 DEBUG:boto:Params: {} 2024-02-01T10:28:32.573 DEBUG:boto:Token: None 2024-02-01T10:28:32.573 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:28:32 GMT /vclfcn-20/obj 2024-02-01T10:28:32.573 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WAZhAdmoH4UQlmhY5yMy/UEDyCA= 2024-02-01T10:28:32.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:28:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WAZhAdmoH4UQlmhY5yMy/UEDyCA=', 'Content-Length': '0'} 2024-02-01T10:28:32.576 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000203dbce4e9822bea-0065bb7250-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:28:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:28:32.576 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:28:32.577 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:28:32.577 DEBUG:boto:path=/vclfcn-20/?prefix=obj 2024-02-01T10:28:32.577 DEBUG:boto:auth_path=/vclfcn-20/?prefix=obj 2024-02-01T10:28:32.577 DEBUG:boto:Method: GET 2024-02-01T10:28:32.577 DEBUG:boto:Path: /vclfcn-20/?prefix=obj 2024-02-01T10:28:32.577 DEBUG:boto:Data: 2024-02-01T10:28:32.577 DEBUG:boto:Headers: {} 2024-02-01T10:28:32.577 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:28:32.577 DEBUG:boto:Port: 8000 2024-02-01T10:28:32.577 DEBUG:boto:Params: {} 2024-02-01T10:28:32.578 DEBUG:boto:Token: None 2024-02-01T10:28:32.578 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:28:32 GMT /vclfcn-20/ 2024-02-01T10:28:32.578 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:2iBjUBEfhbWtVzq+KPYPaZeBG+4= 2024-02-01T10:28:32.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:28:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2iBjUBEfhbWtVzq+KPYPaZeBG+4=', 'Content-Length': '0'} 2024-02-01T10:28:32.582 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000782bc0e2a05bae20-0065bb7250-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:28:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:28:32.582 DEBUG:boto:b'vclfcn-20obj1000falseobj2024-02-01T10:28:01.521Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:28:32.583 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:28:32.583 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:28:32.583 DEBUG:boto:path=/vclfcn-20/?prefix=obj 2024-02-01T10:28:32.583 DEBUG:boto:auth_path=/vclfcn-20/?prefix=obj 2024-02-01T10:28:32.583 DEBUG:boto:Method: GET 2024-02-01T10:28:32.583 DEBUG:boto:Path: /vclfcn-20/?prefix=obj 2024-02-01T10:28:32.584 DEBUG:boto:Data: 2024-02-01T10:28:32.584 DEBUG:boto:Headers: {} 2024-02-01T10:28:32.584 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:28:32.584 DEBUG:boto:Port: 8000 2024-02-01T10:28:32.584 DEBUG:boto:Params: {} 2024-02-01T10:28:32.584 DEBUG:boto:Token: None 2024-02-01T10:28:32.584 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:28:32 GMT /vclfcn-20/ 2024-02-01T10:28:32.584 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:2iBjUBEfhbWtVzq+KPYPaZeBG+4= 2024-02-01T10:28:32.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:28:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2iBjUBEfhbWtVzq+KPYPaZeBG+4=', 'Content-Length': '0'} 2024-02-01T10:28:32.589 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000535b2560bc7acd23-0065bb7250-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:28:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:28:32.589 DEBUG:boto:b'vclfcn-20obj1000falseobj2024-02-01T10:28:01.521Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:28:32.589 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-20 zones={test-zone2, test-zone1} 2024-02-01T10:28:32.589 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-20 zones={test-zone3, test-zone1} 2024-02-01T10:28:32.589 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:28:32.589 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:28:32.590 DEBUG:boto:Method: HEAD 2024-02-01T10:28:32.590 DEBUG:boto:Path: /vclfcn-20/ 2024-02-01T10:28:32.590 DEBUG:boto:Data: 2024-02-01T10:28:32.590 DEBUG:boto:Headers: {} 2024-02-01T10:28:32.590 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:28:32.590 DEBUG:boto:Port: 8001 2024-02-01T10:28:32.590 DEBUG:boto:Params: {} 2024-02-01T10:28:32.590 DEBUG:boto:Token: None 2024-02-01T10:28:32.591 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:28:32 GMT /vclfcn-20/ 2024-02-01T10:28:32.591 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Uwna1EOMEU3ZFTPJD6bxuWRAe1U= 2024-02-01T10:28:32.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:28:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Uwna1EOMEU3ZFTPJD6bxuWRAe1U=', 'Content-Length': '0'} 2024-02-01T10:28:32.595 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000019ec0d85c927d6f9-0065bb7250-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:28:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:28:32.595 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:28:32.595 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:28:32.595 DEBUG:boto:Method: HEAD 2024-02-01T10:28:32.595 DEBUG:boto:Path: /vclfcn-20/ 2024-02-01T10:28:32.595 DEBUG:boto:Data: 2024-02-01T10:28:32.595 DEBUG:boto:Headers: {} 2024-02-01T10:28:32.596 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:28:32.596 DEBUG:boto:Port: 8000 2024-02-01T10:28:32.596 DEBUG:boto:Params: {} 2024-02-01T10:28:32.596 DEBUG:boto:Token: None 2024-02-01T10:28:32.596 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:28:32 GMT /vclfcn-20/ 2024-02-01T10:28:32.596 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Uwna1EOMEU3ZFTPJD6bxuWRAe1U= 2024-02-01T10:28:32.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:28:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Uwna1EOMEU3ZFTPJD6bxuWRAe1U=', 'Content-Length': '0'} 2024-02-01T10:28:32.600 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000491e4dd024953fb8-0065bb7250-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:28:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:28:32.600 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:28:32.600 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:28:32.600 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:28:32.600 DEBUG:boto:path=/vclfcn-20/?versions&max-keys=999 2024-02-01T10:28:32.600 DEBUG:boto:auth_path=/vclfcn-20/?versions&max-keys=999 2024-02-01T10:28:32.601 DEBUG:boto:Method: GET 2024-02-01T10:28:32.601 DEBUG:boto:Path: /vclfcn-20/?versions&max-keys=999 2024-02-01T10:28:32.601 DEBUG:boto:Data: 2024-02-01T10:28:32.601 DEBUG:boto:Headers: {} 2024-02-01T10:28:32.601 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:28:32.601 DEBUG:boto:Port: 8001 2024-02-01T10:28:32.601 DEBUG:boto:Params: {} 2024-02-01T10:28:32.601 DEBUG:boto:Token: None 2024-02-01T10:28:32.601 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:28:32 GMT /vclfcn-20/?versions 2024-02-01T10:28:32.602 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zC7B0wUvHWWBbWJNubCwj+1F5yU= 2024-02-01T10:28:32.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:28:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zC7B0wUvHWWBbWJNubCwj+1F5yU=', 'Content-Length': '0'} 2024-02-01T10:28:32.606 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000dafee14fbafe0c2-0065bb7250-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:28:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:28:32.607 DEBUG:boto:b'vclfcn-20999falseobjnulltrue2024-02-01T10:28:01.521Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:28:32.607 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:28:32.607 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:28:32.607 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:28:32.607 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:28:32.607 DEBUG:boto:path=/vclfcn-20/?versions&max-keys=999 2024-02-01T10:28:32.607 DEBUG:boto:auth_path=/vclfcn-20/?versions&max-keys=999 2024-02-01T10:28:32.608 DEBUG:boto:Method: GET 2024-02-01T10:28:32.608 DEBUG:boto:Path: /vclfcn-20/?versions&max-keys=999 2024-02-01T10:28:32.608 DEBUG:boto:Data: 2024-02-01T10:28:32.608 DEBUG:boto:Headers: {} 2024-02-01T10:28:32.608 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:28:32.608 DEBUG:boto:Port: 8000 2024-02-01T10:28:32.608 DEBUG:boto:Params: {} 2024-02-01T10:28:32.608 DEBUG:boto:Token: None 2024-02-01T10:28:32.608 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:28:32 GMT /vclfcn-20/?versions 2024-02-01T10:28:32.608 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zC7B0wUvHWWBbWJNubCwj+1F5yU= 2024-02-01T10:28:32.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:28:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zC7B0wUvHWWBbWJNubCwj+1F5yU=', 'Content-Length': '0'} 2024-02-01T10:28:32.612 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a277ad983478e51-0065bb7250-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:28:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:28:32.656 DEBUG:boto:b'vclfcn-20999falseobjnulltrue2024-02-01T10:28:01.521Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:28:32.656 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:28:32.657 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:28:32.657 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:28:32.657 DEBUG:boto:path=/vclfcn-20/?versions&max-keys=999 2024-02-01T10:28:32.657 DEBUG:boto:auth_path=/vclfcn-20/?versions&max-keys=999 2024-02-01T10:28:32.657 DEBUG:boto:Method: GET 2024-02-01T10:28:32.657 DEBUG:boto:Path: /vclfcn-20/?versions&max-keys=999 2024-02-01T10:28:32.657 DEBUG:boto:Data: 2024-02-01T10:28:32.657 DEBUG:boto:Headers: {} 2024-02-01T10:28:32.657 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:28:32.657 DEBUG:boto:Port: 8001 2024-02-01T10:28:32.658 DEBUG:boto:Params: {} 2024-02-01T10:28:32.658 DEBUG:boto:Token: None 2024-02-01T10:28:32.658 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:28:32 GMT /vclfcn-20/?versions 2024-02-01T10:28:32.658 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zC7B0wUvHWWBbWJNubCwj+1F5yU= 2024-02-01T10:28:32.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:28:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zC7B0wUvHWWBbWJNubCwj+1F5yU=', 'Content-Length': '0'} 2024-02-01T10:28:32.663 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000309db5904180226f-0065bb7250-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:28:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:28:32.663 DEBUG:boto:b'vclfcn-20999falseobjnulltrue2024-02-01T10:28:01.521Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:28:32.663 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:28:32.664 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:28:32.664 DEBUG:boto:path=/vclfcn-20/?versions&max-keys=999 2024-02-01T10:28:32.664 DEBUG:boto:auth_path=/vclfcn-20/?versions&max-keys=999 2024-02-01T10:28:32.664 DEBUG:boto:Method: GET 2024-02-01T10:28:32.664 DEBUG:boto:Path: /vclfcn-20/?versions&max-keys=999 2024-02-01T10:28:32.664 DEBUG:boto:Data: 2024-02-01T10:28:32.664 DEBUG:boto:Headers: {} 2024-02-01T10:28:32.664 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:28:32.664 DEBUG:boto:Port: 8000 2024-02-01T10:28:32.664 DEBUG:boto:Params: {} 2024-02-01T10:28:32.665 DEBUG:boto:Token: None 2024-02-01T10:28:32.665 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:28:32 GMT /vclfcn-20/?versions 2024-02-01T10:28:32.665 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zC7B0wUvHWWBbWJNubCwj+1F5yU= 2024-02-01T10:28:32.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:28:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zC7B0wUvHWWBbWJNubCwj+1F5yU=', 'Content-Length': '0'} 2024-02-01T10:28:32.670 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001bee79221aec09c6-0065bb7250-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:28:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:28:32.670 DEBUG:boto:b'vclfcn-20999falseobjnulltrue2024-02-01T10:28:01.521Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:28:32.670 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:28:32.670 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:28:32.670 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:28:32.670 DEBUG:boto:path=/vclfcn-20/obj?versionId=null 2024-02-01T10:28:32.671 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=null 2024-02-01T10:28:32.671 DEBUG:boto:Method: GET 2024-02-01T10:28:32.671 DEBUG:boto:Path: /vclfcn-20/obj?versionId=null 2024-02-01T10:28:32.671 DEBUG:boto:Data: 2024-02-01T10:28:32.671 DEBUG:boto:Headers: {} 2024-02-01T10:28:32.671 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:28:32.671 DEBUG:boto:Port: 8001 2024-02-01T10:28:32.671 DEBUG:boto:Params: {} 2024-02-01T10:28:32.671 DEBUG:boto:Token: None 2024-02-01T10:28:32.671 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:28:32 GMT /vclfcn-20/obj?versionId=null 2024-02-01T10:28:32.672 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HlGQNqZGW6oyMaE71lIuC/ZFszw= 2024-02-01T10:28:32.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:28:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HlGQNqZGW6oyMaE71lIuC/ZFszw=', 'Content-Length': '0'} 2024-02-01T10:28:32.675 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c5e528ff783bf3d5-0065bb7250-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:28:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:28:32.675 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:28:32.675 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:28:32.675 DEBUG:boto:path=/vclfcn-20/obj?versionId=null 2024-02-01T10:28:32.675 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=null 2024-02-01T10:28:32.676 DEBUG:boto:Method: GET 2024-02-01T10:28:32.676 DEBUG:boto:Path: /vclfcn-20/obj?versionId=null 2024-02-01T10:28:32.676 DEBUG:boto:Data: 2024-02-01T10:28:32.676 DEBUG:boto:Headers: {} 2024-02-01T10:28:32.676 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:28:32.676 DEBUG:boto:Port: 8000 2024-02-01T10:28:32.676 DEBUG:boto:Params: {} 2024-02-01T10:28:32.676 DEBUG:boto:Token: None 2024-02-01T10:28:32.676 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:28:32 GMT /vclfcn-20/obj?versionId=null 2024-02-01T10:28:32.676 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HlGQNqZGW6oyMaE71lIuC/ZFszw= 2024-02-01T10:28:32.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:28:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HlGQNqZGW6oyMaE71lIuC/ZFszw=', 'Content-Length': '0'} 2024-02-01T10:28:32.679 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006cda50b72e6ab1cb-0065bb7250-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:28:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:28:32.679 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:28:32.680 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:28:32.680 DEBUG:boto:path=/vclfcn-20/obj?versionId=null 2024-02-01T10:28:32.680 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=null 2024-02-01T10:28:32.680 DEBUG:boto:Method: HEAD 2024-02-01T10:28:32.680 DEBUG:boto:Path: /vclfcn-20/obj?versionId=null 2024-02-01T10:28:32.680 DEBUG:boto:Data: 2024-02-01T10:28:32.680 DEBUG:boto:Headers: {} 2024-02-01T10:28:32.680 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:28:32.680 DEBUG:boto:Port: 8001 2024-02-01T10:28:32.680 DEBUG:boto:Params: {} 2024-02-01T10:28:32.680 DEBUG:boto:Token: None 2024-02-01T10:28:32.681 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:28:32 GMT /vclfcn-20/obj?versionId=null 2024-02-01T10:28:32.681 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:s6X0UdFbNRuC9s68m5IE7Qqe4t4= 2024-02-01T10:28:32.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:28:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:s6X0UdFbNRuC9s68m5IE7Qqe4t4=', 'Content-Length': '0'} 2024-02-01T10:28:32.683 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ac63de3c6f99f101-0065bb7250-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:28:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:28:32.684 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:28:32.684 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:28:32.684 DEBUG:boto:path=/vclfcn-20/obj?versionId=null 2024-02-01T10:28:32.684 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=null 2024-02-01T10:28:32.684 DEBUG:boto:Method: HEAD 2024-02-01T10:28:32.684 DEBUG:boto:Path: /vclfcn-20/obj?versionId=null 2024-02-01T10:28:32.684 DEBUG:boto:Data: 2024-02-01T10:28:32.684 DEBUG:boto:Headers: {} 2024-02-01T10:28:32.685 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:28:32.685 DEBUG:boto:Port: 8000 2024-02-01T10:28:32.685 DEBUG:boto:Params: {} 2024-02-01T10:28:32.685 DEBUG:boto:Token: None 2024-02-01T10:28:32.685 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:28:32 GMT /vclfcn-20/obj?versionId=null 2024-02-01T10:28:32.685 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:s6X0UdFbNRuC9s68m5IE7Qqe4t4= 2024-02-01T10:28:32.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:28:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:s6X0UdFbNRuC9s68m5IE7Qqe4t4=', 'Content-Length': '0'} 2024-02-01T10:28:32.688 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a79f4a7f772d807b-0065bb7250-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:28:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:28:32.688 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:28:32.688 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:28:32.688 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:28:32.688 DEBUG:boto:path=/vclfcn-20/obj?versionId=null 2024-02-01T10:28:32.688 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=null 2024-02-01T10:28:32.689 DEBUG:boto:Method: GET 2024-02-01T10:28:32.689 DEBUG:boto:Path: /vclfcn-20/obj?versionId=null 2024-02-01T10:28:32.689 DEBUG:boto:Data: 2024-02-01T10:28:32.689 DEBUG:boto:Headers: {} 2024-02-01T10:28:32.689 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:28:32.689 DEBUG:boto:Port: 8001 2024-02-01T10:28:32.690 DEBUG:boto:Params: {} 2024-02-01T10:28:32.690 DEBUG:boto:Token: None 2024-02-01T10:28:32.690 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:28:32 GMT /vclfcn-20/obj?versionId=null 2024-02-01T10:28:32.690 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HlGQNqZGW6oyMaE71lIuC/ZFszw= 2024-02-01T10:28:32.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:28:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HlGQNqZGW6oyMaE71lIuC/ZFszw=', 'Content-Length': '0'} 2024-02-01T10:28:32.693 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000436580a6ec2daf5a-0065bb7250-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:28:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:28:32.694 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:28:32.694 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:28:32.694 DEBUG:boto:path=/vclfcn-20/obj?versionId=null 2024-02-01T10:28:32.694 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=null 2024-02-01T10:28:32.694 DEBUG:boto:Method: GET 2024-02-01T10:28:32.694 DEBUG:boto:Path: /vclfcn-20/obj?versionId=null 2024-02-01T10:28:32.694 DEBUG:boto:Data: 2024-02-01T10:28:32.694 DEBUG:boto:Headers: {} 2024-02-01T10:28:32.694 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:28:32.694 DEBUG:boto:Port: 8000 2024-02-01T10:28:32.694 DEBUG:boto:Params: {} 2024-02-01T10:28:32.695 DEBUG:boto:Token: None 2024-02-01T10:28:32.695 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:28:32 GMT /vclfcn-20/obj?versionId=null 2024-02-01T10:28:32.695 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HlGQNqZGW6oyMaE71lIuC/ZFszw= 2024-02-01T10:28:32.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:28:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HlGQNqZGW6oyMaE71lIuC/ZFszw=', 'Content-Length': '0'} 2024-02-01T10:28:32.697 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000aaf7f3954d5464a0-0065bb7250-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:28:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:28:32.698 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:28:32.698 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:28:32.698 DEBUG:boto:path=/vclfcn-20/?prefix=obj 2024-02-01T10:28:32.698 DEBUG:boto:auth_path=/vclfcn-20/?prefix=obj 2024-02-01T10:28:32.698 DEBUG:boto:Method: GET 2024-02-01T10:28:32.698 DEBUG:boto:Path: /vclfcn-20/?prefix=obj 2024-02-01T10:28:32.698 DEBUG:boto:Data: 2024-02-01T10:28:32.698 DEBUG:boto:Headers: {} 2024-02-01T10:28:32.698 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:28:32.698 DEBUG:boto:Port: 8001 2024-02-01T10:28:32.698 DEBUG:boto:Params: {} 2024-02-01T10:28:32.699 DEBUG:boto:Token: None 2024-02-01T10:28:32.699 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:28:32 GMT /vclfcn-20/ 2024-02-01T10:28:32.699 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:2iBjUBEfhbWtVzq+KPYPaZeBG+4= 2024-02-01T10:28:32.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:28:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2iBjUBEfhbWtVzq+KPYPaZeBG+4=', 'Content-Length': '0'} 2024-02-01T10:28:32.704 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000081b1738b4b6e9cb-0065bb7250-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:28:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:28:32.748 DEBUG:boto:b'vclfcn-20obj1000falseobj2024-02-01T10:28:01.521Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:28:32.749 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:28:32.749 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:28:32.749 DEBUG:boto:path=/vclfcn-20/?prefix=obj 2024-02-01T10:28:32.749 DEBUG:boto:auth_path=/vclfcn-20/?prefix=obj 2024-02-01T10:28:32.749 DEBUG:boto:Method: GET 2024-02-01T10:28:32.749 DEBUG:boto:Path: /vclfcn-20/?prefix=obj 2024-02-01T10:28:32.749 DEBUG:boto:Data: 2024-02-01T10:28:32.749 DEBUG:boto:Headers: {} 2024-02-01T10:28:32.750 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:28:32.750 DEBUG:boto:Port: 8000 2024-02-01T10:28:32.750 DEBUG:boto:Params: {} 2024-02-01T10:28:32.750 DEBUG:boto:Token: None 2024-02-01T10:28:32.750 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:28:32 GMT /vclfcn-20/ 2024-02-01T10:28:32.750 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:2iBjUBEfhbWtVzq+KPYPaZeBG+4= 2024-02-01T10:28:32.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:28:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2iBjUBEfhbWtVzq+KPYPaZeBG+4=', 'Content-Length': '0'} 2024-02-01T10:28:32.755 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004555594eb760084-0065bb7250-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:28:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:28:32.756 DEBUG:boto:b'vclfcn-20obj1000falseobj2024-02-01T10:28:01.521Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:28:32.756 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:28:32.756 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:28:32.757 DEBUG:boto:Method: HEAD 2024-02-01T10:28:32.757 DEBUG:boto:Path: /vclfcn-20/obj 2024-02-01T10:28:32.757 DEBUG:boto:Data: 2024-02-01T10:28:32.757 DEBUG:boto:Headers: {} 2024-02-01T10:28:32.757 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:28:32.757 DEBUG:boto:Port: 8001 2024-02-01T10:28:32.758 DEBUG:boto:Params: {} 2024-02-01T10:28:32.758 DEBUG:boto:Token: None 2024-02-01T10:28:32.758 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:28:32 GMT /vclfcn-20/obj 2024-02-01T10:28:32.758 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+FZ9ftT0IJFvqiNBW3YmhaIWEnw= 2024-02-01T10:28:32.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:28:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+FZ9ftT0IJFvqiNBW3YmhaIWEnw=', 'Content-Length': '0'} 2024-02-01T10:28:32.762 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000039f4bcad7d4c1165-0065bb7250-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:28:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:28:32.762 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:28:32.762 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:28:32.762 DEBUG:boto:Method: HEAD 2024-02-01T10:28:32.762 DEBUG:boto:Path: /vclfcn-20/obj 2024-02-01T10:28:32.763 DEBUG:boto:Data: 2024-02-01T10:28:32.763 DEBUG:boto:Headers: {} 2024-02-01T10:28:32.763 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:28:32.763 DEBUG:boto:Port: 8000 2024-02-01T10:28:32.763 DEBUG:boto:Params: {} 2024-02-01T10:28:32.763 DEBUG:boto:Token: None 2024-02-01T10:28:32.764 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:28:32 GMT /vclfcn-20/obj 2024-02-01T10:28:32.764 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+FZ9ftT0IJFvqiNBW3YmhaIWEnw= 2024-02-01T10:28:32.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:28:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+FZ9ftT0IJFvqiNBW3YmhaIWEnw=', 'Content-Length': '0'} 2024-02-01T10:28:32.767 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000427065d9389cc678-0065bb7250-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:28:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:28:32.767 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:28:32.767 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:28:32.768 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:28:32.768 DEBUG:boto:Method: GET 2024-02-01T10:28:32.768 DEBUG:boto:Path: /vclfcn-20/obj 2024-02-01T10:28:32.768 DEBUG:boto:Data: 2024-02-01T10:28:32.768 DEBUG:boto:Headers: {} 2024-02-01T10:28:32.768 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:28:32.768 DEBUG:boto:Port: 8001 2024-02-01T10:28:32.769 DEBUG:boto:Params: {} 2024-02-01T10:28:32.769 DEBUG:boto:Token: None 2024-02-01T10:28:32.769 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:28:32 GMT /vclfcn-20/obj 2024-02-01T10:28:32.769 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WAZhAdmoH4UQlmhY5yMy/UEDyCA= 2024-02-01T10:28:32.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:28:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WAZhAdmoH4UQlmhY5yMy/UEDyCA=', 'Content-Length': '0'} 2024-02-01T10:28:32.772 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f0da9e67fe44cb40-0065bb7250-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:28:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:28:32.773 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:28:32.773 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:28:32.773 DEBUG:boto:Method: GET 2024-02-01T10:28:32.773 DEBUG:boto:Path: /vclfcn-20/obj 2024-02-01T10:28:32.773 DEBUG:boto:Data: 2024-02-01T10:28:32.773 DEBUG:boto:Headers: {} 2024-02-01T10:28:32.774 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:28:32.774 DEBUG:boto:Port: 8000 2024-02-01T10:28:32.774 DEBUG:boto:Params: {} 2024-02-01T10:28:32.774 DEBUG:boto:Token: None 2024-02-01T10:28:32.774 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:28:32 GMT /vclfcn-20/obj 2024-02-01T10:28:32.775 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WAZhAdmoH4UQlmhY5yMy/UEDyCA= 2024-02-01T10:28:32.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:28:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WAZhAdmoH4UQlmhY5yMy/UEDyCA=', 'Content-Length': '0'} 2024-02-01T10:28:32.777 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000aede224b37c50b3c-0065bb7250-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:28:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:28:32.778 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:28:32.778 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:28:32.778 DEBUG:boto:path=/vclfcn-20/?prefix=obj 2024-02-01T10:28:32.778 DEBUG:boto:auth_path=/vclfcn-20/?prefix=obj 2024-02-01T10:28:32.778 DEBUG:boto:Method: GET 2024-02-01T10:28:32.778 DEBUG:boto:Path: /vclfcn-20/?prefix=obj 2024-02-01T10:28:32.778 DEBUG:boto:Data: 2024-02-01T10:28:32.778 DEBUG:boto:Headers: {} 2024-02-01T10:28:32.779 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:28:32.779 DEBUG:boto:Port: 8001 2024-02-01T10:28:32.779 DEBUG:boto:Params: {} 2024-02-01T10:28:32.779 DEBUG:boto:Token: None 2024-02-01T10:28:32.779 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:28:32 GMT /vclfcn-20/ 2024-02-01T10:28:32.779 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:2iBjUBEfhbWtVzq+KPYPaZeBG+4= 2024-02-01T10:28:32.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:28:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2iBjUBEfhbWtVzq+KPYPaZeBG+4=', 'Content-Length': '0'} 2024-02-01T10:28:32.783 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007bea84b7d3b64c89-0065bb7250-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:28:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:28:32.824 DEBUG:boto:b'vclfcn-20obj1000falseobj2024-02-01T10:28:01.521Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:28:32.825 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:28:32.825 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:28:32.825 DEBUG:boto:path=/vclfcn-20/?prefix=obj 2024-02-01T10:28:32.825 DEBUG:boto:auth_path=/vclfcn-20/?prefix=obj 2024-02-01T10:28:32.825 DEBUG:boto:Method: GET 2024-02-01T10:28:32.825 DEBUG:boto:Path: /vclfcn-20/?prefix=obj 2024-02-01T10:28:32.825 DEBUG:boto:Data: 2024-02-01T10:28:32.825 DEBUG:boto:Headers: {} 2024-02-01T10:28:32.825 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:28:32.826 DEBUG:boto:Port: 8000 2024-02-01T10:28:32.826 DEBUG:boto:Params: {} 2024-02-01T10:28:32.826 DEBUG:boto:Token: None 2024-02-01T10:28:32.826 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:28:32 GMT /vclfcn-20/ 2024-02-01T10:28:32.826 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:2iBjUBEfhbWtVzq+KPYPaZeBG+4= 2024-02-01T10:28:32.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:28:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2iBjUBEfhbWtVzq+KPYPaZeBG+4=', 'Content-Length': '0'} 2024-02-01T10:28:32.831 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084a8e1dc44673135-0065bb7250-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:28:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:28:32.831 DEBUG:boto:b'vclfcn-20obj1000falseobj2024-02-01T10:28:01.521Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:28:32.831 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-20 zones={test-zone3, test-zone1} 2024-02-01T10:28:32.832 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-20 zones={test-zone3, test-zone2} 2024-02-01T10:28:32.832 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:28:32.832 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:28:32.832 DEBUG:boto:Method: HEAD 2024-02-01T10:28:32.832 DEBUG:boto:Path: /vclfcn-20/ 2024-02-01T10:28:32.832 DEBUG:boto:Data: 2024-02-01T10:28:32.832 DEBUG:boto:Headers: {} 2024-02-01T10:28:32.832 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:28:32.832 DEBUG:boto:Port: 8001 2024-02-01T10:28:32.832 DEBUG:boto:Params: {} 2024-02-01T10:28:32.833 DEBUG:boto:Token: None 2024-02-01T10:28:32.833 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:28:32 GMT /vclfcn-20/ 2024-02-01T10:28:32.833 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Uwna1EOMEU3ZFTPJD6bxuWRAe1U= 2024-02-01T10:28:32.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:28:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Uwna1EOMEU3ZFTPJD6bxuWRAe1U=', 'Content-Length': '0'} 2024-02-01T10:28:32.839 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008385fd8015761973-0065bb7250-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:28:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:28:32.839 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:28:32.839 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:28:32.840 DEBUG:boto:Method: HEAD 2024-02-01T10:28:32.840 DEBUG:boto:Path: /vclfcn-20/ 2024-02-01T10:28:32.840 DEBUG:boto:Data: 2024-02-01T10:28:32.840 DEBUG:boto:Headers: {} 2024-02-01T10:28:32.840 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:28:32.840 DEBUG:boto:Port: 8000 2024-02-01T10:28:32.840 DEBUG:boto:Params: {} 2024-02-01T10:28:32.840 DEBUG:boto:Token: None 2024-02-01T10:28:32.840 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:28:32 GMT /vclfcn-20/ 2024-02-01T10:28:32.841 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Uwna1EOMEU3ZFTPJD6bxuWRAe1U= 2024-02-01T10:28:32.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:28:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Uwna1EOMEU3ZFTPJD6bxuWRAe1U=', 'Content-Length': '0'} 2024-02-01T10:28:32.845 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004c5979445aec7036-0065bb7250-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:28:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:28:32.845 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:28:32.845 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:28:32.845 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:28:32.845 DEBUG:boto:path=/vclfcn-20/?versions&max-keys=999 2024-02-01T10:28:32.845 DEBUG:boto:auth_path=/vclfcn-20/?versions&max-keys=999 2024-02-01T10:28:32.845 DEBUG:boto:Method: GET 2024-02-01T10:28:32.845 DEBUG:boto:Path: /vclfcn-20/?versions&max-keys=999 2024-02-01T10:28:32.845 DEBUG:boto:Data: 2024-02-01T10:28:32.845 DEBUG:boto:Headers: {} 2024-02-01T10:28:32.846 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:28:32.846 DEBUG:boto:Port: 8001 2024-02-01T10:28:32.846 DEBUG:boto:Params: {} 2024-02-01T10:28:32.846 DEBUG:boto:Token: None 2024-02-01T10:28:32.846 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:28:32 GMT /vclfcn-20/?versions 2024-02-01T10:28:32.846 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zC7B0wUvHWWBbWJNubCwj+1F5yU= 2024-02-01T10:28:32.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:28:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zC7B0wUvHWWBbWJNubCwj+1F5yU=', 'Content-Length': '0'} 2024-02-01T10:28:32.851 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a4af5eef585c74c-0065bb7250-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:28:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:28:32.851 DEBUG:boto:b'vclfcn-20999falseobjnulltrue2024-02-01T10:28:01.521Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:28:32.851 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:28:32.852 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:28:32.852 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:28:32.852 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:28:32.852 DEBUG:boto:path=/vclfcn-20/?versions&max-keys=999 2024-02-01T10:28:32.852 DEBUG:boto:auth_path=/vclfcn-20/?versions&max-keys=999 2024-02-01T10:28:32.852 DEBUG:boto:Method: GET 2024-02-01T10:28:32.852 DEBUG:boto:Path: /vclfcn-20/?versions&max-keys=999 2024-02-01T10:28:32.852 DEBUG:boto:Data: 2024-02-01T10:28:32.852 DEBUG:boto:Headers: {} 2024-02-01T10:28:32.853 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:28:32.853 DEBUG:boto:Port: 8000 2024-02-01T10:28:32.853 DEBUG:boto:Params: {} 2024-02-01T10:28:32.853 DEBUG:boto:Token: None 2024-02-01T10:28:32.853 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:28:32 GMT /vclfcn-20/?versions 2024-02-01T10:28:32.853 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zC7B0wUvHWWBbWJNubCwj+1F5yU= 2024-02-01T10:28:32.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:28:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zC7B0wUvHWWBbWJNubCwj+1F5yU=', 'Content-Length': '0'} 2024-02-01T10:28:32.858 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f31860973e7d3dc8-0065bb7250-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:28:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:28:32.858 DEBUG:boto:b'vclfcn-20999falseobjnulltrue2024-02-01T10:28:01.521Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:28:32.858 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:28:32.858 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:28:32.859 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:28:32.859 DEBUG:boto:path=/vclfcn-20/?versions&max-keys=999 2024-02-01T10:28:32.859 DEBUG:boto:auth_path=/vclfcn-20/?versions&max-keys=999 2024-02-01T10:28:32.859 DEBUG:boto:Method: GET 2024-02-01T10:28:32.859 DEBUG:boto:Path: /vclfcn-20/?versions&max-keys=999 2024-02-01T10:28:32.859 DEBUG:boto:Data: 2024-02-01T10:28:32.859 DEBUG:boto:Headers: {} 2024-02-01T10:28:32.859 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:28:32.859 DEBUG:boto:Port: 8001 2024-02-01T10:28:32.859 DEBUG:boto:Params: {} 2024-02-01T10:28:32.859 DEBUG:boto:Token: None 2024-02-01T10:28:32.860 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:28:32 GMT /vclfcn-20/?versions 2024-02-01T10:28:32.860 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zC7B0wUvHWWBbWJNubCwj+1F5yU= 2024-02-01T10:28:32.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:28:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zC7B0wUvHWWBbWJNubCwj+1F5yU=', 'Content-Length': '0'} 2024-02-01T10:28:32.864 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a1280c5e13fddcf-0065bb7250-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:28:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:28:32.908 DEBUG:boto:b'vclfcn-20999falseobjnulltrue2024-02-01T10:28:01.521Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:28:32.909 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:28:32.909 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:28:32.909 DEBUG:boto:path=/vclfcn-20/?versions&max-keys=999 2024-02-01T10:28:32.909 DEBUG:boto:auth_path=/vclfcn-20/?versions&max-keys=999 2024-02-01T10:28:32.909 DEBUG:boto:Method: GET 2024-02-01T10:28:32.909 DEBUG:boto:Path: /vclfcn-20/?versions&max-keys=999 2024-02-01T10:28:32.909 DEBUG:boto:Data: 2024-02-01T10:28:32.910 DEBUG:boto:Headers: {} 2024-02-01T10:28:32.910 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:28:32.910 DEBUG:boto:Port: 8000 2024-02-01T10:28:32.910 DEBUG:boto:Params: {} 2024-02-01T10:28:32.910 DEBUG:boto:Token: None 2024-02-01T10:28:32.910 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:28:32 GMT /vclfcn-20/?versions 2024-02-01T10:28:32.910 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zC7B0wUvHWWBbWJNubCwj+1F5yU= 2024-02-01T10:28:32.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:28:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zC7B0wUvHWWBbWJNubCwj+1F5yU=', 'Content-Length': '0'} 2024-02-01T10:28:32.915 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002515eae7ef8b4866-0065bb7250-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:28:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:28:32.915 DEBUG:boto:b'vclfcn-20999falseobjnulltrue2024-02-01T10:28:01.521Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:28:32.916 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:28:32.916 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:28:32.916 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:28:32.916 DEBUG:boto:path=/vclfcn-20/obj?versionId=null 2024-02-01T10:28:32.916 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=null 2024-02-01T10:28:32.916 DEBUG:boto:Method: GET 2024-02-01T10:28:32.917 DEBUG:boto:Path: /vclfcn-20/obj?versionId=null 2024-02-01T10:28:32.917 DEBUG:boto:Data: 2024-02-01T10:28:32.917 DEBUG:boto:Headers: {} 2024-02-01T10:28:32.917 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:28:32.917 DEBUG:boto:Port: 8001 2024-02-01T10:28:32.917 DEBUG:boto:Params: {} 2024-02-01T10:28:32.917 DEBUG:boto:Token: None 2024-02-01T10:28:32.917 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:28:32 GMT /vclfcn-20/obj?versionId=null 2024-02-01T10:28:32.917 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HlGQNqZGW6oyMaE71lIuC/ZFszw= 2024-02-01T10:28:32.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:28:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HlGQNqZGW6oyMaE71lIuC/ZFszw=', 'Content-Length': '0'} 2024-02-01T10:28:32.920 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a5a21ff357394c53-0065bb7250-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:28:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:28:32.920 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:28:32.921 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:28:32.921 DEBUG:boto:path=/vclfcn-20/obj?versionId=null 2024-02-01T10:28:32.921 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=null 2024-02-01T10:28:32.921 DEBUG:boto:Method: GET 2024-02-01T10:28:32.921 DEBUG:boto:Path: /vclfcn-20/obj?versionId=null 2024-02-01T10:28:32.921 DEBUG:boto:Data: 2024-02-01T10:28:32.921 DEBUG:boto:Headers: {} 2024-02-01T10:28:32.921 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:28:32.921 DEBUG:boto:Port: 8000 2024-02-01T10:28:32.921 DEBUG:boto:Params: {} 2024-02-01T10:28:32.921 DEBUG:boto:Token: None 2024-02-01T10:28:32.922 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:28:32 GMT /vclfcn-20/obj?versionId=null 2024-02-01T10:28:32.922 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HlGQNqZGW6oyMaE71lIuC/ZFszw= 2024-02-01T10:28:32.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:28:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HlGQNqZGW6oyMaE71lIuC/ZFszw=', 'Content-Length': '0'} 2024-02-01T10:28:32.925 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005ff0ee8705b5b2ad-0065bb7250-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:28:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:28:32.925 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:28:32.925 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:28:32.925 DEBUG:boto:path=/vclfcn-20/obj?versionId=null 2024-02-01T10:28:32.925 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=null 2024-02-01T10:28:32.925 DEBUG:boto:Method: HEAD 2024-02-01T10:28:32.925 DEBUG:boto:Path: /vclfcn-20/obj?versionId=null 2024-02-01T10:28:32.926 DEBUG:boto:Data: 2024-02-01T10:28:32.926 DEBUG:boto:Headers: {} 2024-02-01T10:28:32.926 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:28:32.926 DEBUG:boto:Port: 8001 2024-02-01T10:28:32.926 DEBUG:boto:Params: {} 2024-02-01T10:28:32.926 DEBUG:boto:Token: None 2024-02-01T10:28:32.926 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:28:32 GMT /vclfcn-20/obj?versionId=null 2024-02-01T10:28:32.926 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:s6X0UdFbNRuC9s68m5IE7Qqe4t4= 2024-02-01T10:28:32.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:28:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:s6X0UdFbNRuC9s68m5IE7Qqe4t4=', 'Content-Length': '0'} 2024-02-01T10:28:32.932 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007245a2e32a7cfc12-0065bb7250-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:28:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:28:32.932 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:28:32.932 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:28:32.932 DEBUG:boto:path=/vclfcn-20/obj?versionId=null 2024-02-01T10:28:32.932 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=null 2024-02-01T10:28:32.932 DEBUG:boto:Method: HEAD 2024-02-01T10:28:32.932 DEBUG:boto:Path: /vclfcn-20/obj?versionId=null 2024-02-01T10:28:32.933 DEBUG:boto:Data: 2024-02-01T10:28:32.933 DEBUG:boto:Headers: {} 2024-02-01T10:28:32.933 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:28:32.933 DEBUG:boto:Port: 8000 2024-02-01T10:28:32.933 DEBUG:boto:Params: {} 2024-02-01T10:28:32.933 DEBUG:boto:Token: None 2024-02-01T10:28:32.933 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:28:32 GMT /vclfcn-20/obj?versionId=null 2024-02-01T10:28:32.933 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:s6X0UdFbNRuC9s68m5IE7Qqe4t4= 2024-02-01T10:28:32.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:28:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:s6X0UdFbNRuC9s68m5IE7Qqe4t4=', 'Content-Length': '0'} 2024-02-01T10:28:32.936 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009b92e4cb238cb072-0065bb7250-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:28:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:28:32.936 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:28:32.937 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:28:32.937 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:28:32.937 DEBUG:boto:path=/vclfcn-20/obj?versionId=null 2024-02-01T10:28:32.937 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=null 2024-02-01T10:28:32.937 DEBUG:boto:Method: GET 2024-02-01T10:28:32.937 DEBUG:boto:Path: /vclfcn-20/obj?versionId=null 2024-02-01T10:28:32.937 DEBUG:boto:Data: 2024-02-01T10:28:32.937 DEBUG:boto:Headers: {} 2024-02-01T10:28:32.937 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:28:32.938 DEBUG:boto:Port: 8001 2024-02-01T10:28:32.938 DEBUG:boto:Params: {} 2024-02-01T10:28:32.938 DEBUG:boto:Token: None 2024-02-01T10:28:32.938 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:28:32 GMT /vclfcn-20/obj?versionId=null 2024-02-01T10:28:32.938 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HlGQNqZGW6oyMaE71lIuC/ZFszw= 2024-02-01T10:28:32.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:28:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HlGQNqZGW6oyMaE71lIuC/ZFszw=', 'Content-Length': '0'} 2024-02-01T10:28:32.941 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000051d386dd6529af0e-0065bb7250-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:28:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:28:32.941 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:28:32.941 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:28:32.941 DEBUG:boto:path=/vclfcn-20/obj?versionId=null 2024-02-01T10:28:32.941 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=null 2024-02-01T10:28:32.941 DEBUG:boto:Method: GET 2024-02-01T10:28:32.942 DEBUG:boto:Path: /vclfcn-20/obj?versionId=null 2024-02-01T10:28:32.942 DEBUG:boto:Data: 2024-02-01T10:28:32.942 DEBUG:boto:Headers: {} 2024-02-01T10:28:32.942 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:28:32.942 DEBUG:boto:Port: 8000 2024-02-01T10:28:32.942 DEBUG:boto:Params: {} 2024-02-01T10:28:32.942 DEBUG:boto:Token: None 2024-02-01T10:28:32.942 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:28:32 GMT /vclfcn-20/obj?versionId=null 2024-02-01T10:28:32.942 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HlGQNqZGW6oyMaE71lIuC/ZFszw= 2024-02-01T10:28:32.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:28:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HlGQNqZGW6oyMaE71lIuC/ZFszw=', 'Content-Length': '0'} 2024-02-01T10:28:32.945 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a34c6e2dd07bd657-0065bb7250-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:28:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:28:32.945 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:28:32.945 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:28:32.945 DEBUG:boto:path=/vclfcn-20/?prefix=obj 2024-02-01T10:28:32.946 DEBUG:boto:auth_path=/vclfcn-20/?prefix=obj 2024-02-01T10:28:32.946 DEBUG:boto:Method: GET 2024-02-01T10:28:32.946 DEBUG:boto:Path: /vclfcn-20/?prefix=obj 2024-02-01T10:28:32.946 DEBUG:boto:Data: 2024-02-01T10:28:32.946 DEBUG:boto:Headers: {} 2024-02-01T10:28:32.946 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:28:32.946 DEBUG:boto:Port: 8001 2024-02-01T10:28:32.946 DEBUG:boto:Params: {} 2024-02-01T10:28:32.946 DEBUG:boto:Token: None 2024-02-01T10:28:32.946 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:28:32 GMT /vclfcn-20/ 2024-02-01T10:28:32.946 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:2iBjUBEfhbWtVzq+KPYPaZeBG+4= 2024-02-01T10:28:32.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:28:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2iBjUBEfhbWtVzq+KPYPaZeBG+4=', 'Content-Length': '0'} 2024-02-01T10:28:32.952 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018670b23487cbd71-0065bb7250-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:28:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:28:32.992 DEBUG:boto:b'vclfcn-20obj1000falseobj2024-02-01T10:28:01.521Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:28:32.993 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:28:32.993 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:28:32.993 DEBUG:boto:path=/vclfcn-20/?prefix=obj 2024-02-01T10:28:32.994 DEBUG:boto:auth_path=/vclfcn-20/?prefix=obj 2024-02-01T10:28:32.994 DEBUG:boto:Method: GET 2024-02-01T10:28:32.994 DEBUG:boto:Path: /vclfcn-20/?prefix=obj 2024-02-01T10:28:32.994 DEBUG:boto:Data: 2024-02-01T10:28:32.994 DEBUG:boto:Headers: {} 2024-02-01T10:28:32.994 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:28:32.994 DEBUG:boto:Port: 8000 2024-02-01T10:28:32.995 DEBUG:boto:Params: {} 2024-02-01T10:28:32.995 DEBUG:boto:Token: None 2024-02-01T10:28:32.995 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:28:32 GMT /vclfcn-20/ 2024-02-01T10:28:32.995 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:2iBjUBEfhbWtVzq+KPYPaZeBG+4= 2024-02-01T10:28:32.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:28:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2iBjUBEfhbWtVzq+KPYPaZeBG+4=', 'Content-Length': '0'} 2024-02-01T10:28:33.000 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd586a5439fb6d75-0065bb7250-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:28:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:28:33.000 DEBUG:boto:b'vclfcn-20obj1000falseobj2024-02-01T10:28:01.521Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:28:33.001 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:28:33.001 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:28:33.001 DEBUG:boto:Method: HEAD 2024-02-01T10:28:33.001 DEBUG:boto:Path: /vclfcn-20/obj 2024-02-01T10:28:33.001 DEBUG:boto:Data: 2024-02-01T10:28:33.001 DEBUG:boto:Headers: {} 2024-02-01T10:28:33.001 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:28:33.001 DEBUG:boto:Port: 8001 2024-02-01T10:28:33.002 DEBUG:boto:Params: {} 2024-02-01T10:28:33.002 DEBUG:boto:Token: None 2024-02-01T10:28:33.002 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:28:33 GMT /vclfcn-20/obj 2024-02-01T10:28:33.002 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zVAoytbIgJ8eZrTIsFTJ+aJQx5c= 2024-02-01T10:28:33.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:28:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zVAoytbIgJ8eZrTIsFTJ+aJQx5c=', 'Content-Length': '0'} 2024-02-01T10:28:33.004 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ef8394b291241ae9-0065bb7251-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:28:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:28:33.005 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:28:33.005 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:28:33.005 DEBUG:boto:Method: HEAD 2024-02-01T10:28:33.005 DEBUG:boto:Path: /vclfcn-20/obj 2024-02-01T10:28:33.005 DEBUG:boto:Data: 2024-02-01T10:28:33.005 DEBUG:boto:Headers: {} 2024-02-01T10:28:33.005 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:28:33.005 DEBUG:boto:Port: 8000 2024-02-01T10:28:33.006 DEBUG:boto:Params: {} 2024-02-01T10:28:33.006 DEBUG:boto:Token: None 2024-02-01T10:28:33.006 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:28:33 GMT /vclfcn-20/obj 2024-02-01T10:28:33.006 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zVAoytbIgJ8eZrTIsFTJ+aJQx5c= 2024-02-01T10:28:33.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:28:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zVAoytbIgJ8eZrTIsFTJ+aJQx5c=', 'Content-Length': '0'} 2024-02-01T10:28:33.009 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000041c11b4dd2cbd6f1-0065bb7251-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:28:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:28:33.009 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:28:33.011 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:28:33.011 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:28:33.011 DEBUG:boto:Method: GET 2024-02-01T10:28:33.011 DEBUG:boto:Path: /vclfcn-20/obj 2024-02-01T10:28:33.011 DEBUG:boto:Data: 2024-02-01T10:28:33.012 DEBUG:boto:Headers: {} 2024-02-01T10:28:33.012 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:28:33.012 DEBUG:boto:Port: 8001 2024-02-01T10:28:33.012 DEBUG:boto:Params: {} 2024-02-01T10:28:33.012 DEBUG:boto:Token: None 2024-02-01T10:28:33.012 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:28:33 GMT /vclfcn-20/obj 2024-02-01T10:28:33.012 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RZ25QG97tA8G48rimNrJBYJP/1Q= 2024-02-01T10:28:33.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:28:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RZ25QG97tA8G48rimNrJBYJP/1Q=', 'Content-Length': '0'} 2024-02-01T10:28:33.015 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d73d8c9fb6c3f385-0065bb7251-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:28:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:28:33.015 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:28:33.016 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:28:33.016 DEBUG:boto:Method: GET 2024-02-01T10:28:33.016 DEBUG:boto:Path: /vclfcn-20/obj 2024-02-01T10:28:33.016 DEBUG:boto:Data: 2024-02-01T10:28:33.016 DEBUG:boto:Headers: {} 2024-02-01T10:28:33.016 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:28:33.016 DEBUG:boto:Port: 8000 2024-02-01T10:28:33.016 DEBUG:boto:Params: {} 2024-02-01T10:28:33.016 DEBUG:boto:Token: None 2024-02-01T10:28:33.017 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:28:33 GMT /vclfcn-20/obj 2024-02-01T10:28:33.017 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RZ25QG97tA8G48rimNrJBYJP/1Q= 2024-02-01T10:28:33.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:28:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RZ25QG97tA8G48rimNrJBYJP/1Q=', 'Content-Length': '0'} 2024-02-01T10:28:33.019 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d874813aafd1fa92-0065bb7251-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:28:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:28:33.020 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:28:33.020 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:28:33.020 DEBUG:boto:path=/vclfcn-20/?prefix=obj 2024-02-01T10:28:33.020 DEBUG:boto:auth_path=/vclfcn-20/?prefix=obj 2024-02-01T10:28:33.020 DEBUG:boto:Method: GET 2024-02-01T10:28:33.020 DEBUG:boto:Path: /vclfcn-20/?prefix=obj 2024-02-01T10:28:33.020 DEBUG:boto:Data: 2024-02-01T10:28:33.021 DEBUG:boto:Headers: {} 2024-02-01T10:28:33.021 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:28:33.021 DEBUG:boto:Port: 8001 2024-02-01T10:28:33.021 DEBUG:boto:Params: {} 2024-02-01T10:28:33.021 DEBUG:boto:Token: None 2024-02-01T10:28:33.021 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:28:33 GMT /vclfcn-20/ 2024-02-01T10:28:33.021 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:LTatQ2vIpRPbIBs+CDp+cc96E/A= 2024-02-01T10:28:33.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:28:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:LTatQ2vIpRPbIBs+CDp+cc96E/A=', 'Content-Length': '0'} 2024-02-01T10:28:33.026 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000decaff42475045bd-0065bb7251-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:28:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:28:33.026 DEBUG:boto:b'vclfcn-20obj1000falseobj2024-02-01T10:28:01.521Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:28:33.027 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:28:33.027 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:28:33.027 DEBUG:boto:path=/vclfcn-20/?prefix=obj 2024-02-01T10:28:33.027 DEBUG:boto:auth_path=/vclfcn-20/?prefix=obj 2024-02-01T10:28:33.027 DEBUG:boto:Method: GET 2024-02-01T10:28:33.027 DEBUG:boto:Path: /vclfcn-20/?prefix=obj 2024-02-01T10:28:33.027 DEBUG:boto:Data: 2024-02-01T10:28:33.027 DEBUG:boto:Headers: {} 2024-02-01T10:28:33.028 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:28:33.028 DEBUG:boto:Port: 8000 2024-02-01T10:28:33.028 DEBUG:boto:Params: {} 2024-02-01T10:28:33.028 DEBUG:boto:Token: None 2024-02-01T10:28:33.028 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:28:33 GMT /vclfcn-20/ 2024-02-01T10:28:33.028 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:LTatQ2vIpRPbIBs+CDp+cc96E/A= 2024-02-01T10:28:33.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:28:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:LTatQ2vIpRPbIBs+CDp+cc96E/A=', 'Content-Length': '0'} 2024-02-01T10:28:33.033 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f19f87ee7ab0206b-0065bb7251-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:28:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:28:33.033 DEBUG:boto:b'vclfcn-20obj1000falseobj2024-02-01T10:28:01.521Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:28:33.034 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-20 zones={test-zone3, test-zone2} 2024-02-01T10:28:33.034 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:28:33.034 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:28:33.034 DEBUG:boto:path=/vclfcn-20/?versioning 2024-02-01T10:28:33.034 DEBUG:boto:auth_path=/vclfcn-20/?versioning 2024-02-01T10:28:33.034 DEBUG:boto:Method: PUT 2024-02-01T10:28:33.034 DEBUG:boto:Path: /vclfcn-20/?versioning 2024-02-01T10:28:33.034 DEBUG:boto:Data: Enabled Disabled 2024-02-01T10:28:33.034 DEBUG:boto:Headers: {} 2024-02-01T10:28:33.034 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:28:33.035 DEBUG:boto:Port: 8000 2024-02-01T10:28:33.035 DEBUG:boto:Params: {} 2024-02-01T10:28:33.035 DEBUG:boto:Token: None 2024-02-01T10:28:33.035 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:28:33 GMT /vclfcn-20/?versioning 2024-02-01T10:28:33.035 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:R2srNDTJ8ppEg5bPekLU5fROmek= 2024-02-01T10:28:33.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:28:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:R2srNDTJ8ppEg5bPekLU5fROmek=', 'Content-Length': '228'} 2024-02-01T10:28:33.047 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000050fa4e277e5d7d0-0065bb7251-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:28:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:28:33.047 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:28:33.047 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:28:33.047 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:28:33.090 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:28:33.090 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:28:33.151 INFO:teuthology.orchestra.run.smithi148.stderr:No --period given, using current period=5dc4efbd-eccd-47a8-9471-ae4a209ec978 2024-02-01T10:28:33.174 INFO:teuthology.orchestra.run.smithi148.stdout:[{"marker":"1_1706782453.132619_30.1","last_update":"2024-02-01T10:14:13.132619Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1706783313.054991_118.1","last_update":"2024-02-01T10:28:33.054991Z"},{"marker":"1_1706782595.670952_121.1","last_update":"2024-02-01T10:16:35.670952Z"}] 2024-02-01T10:28:33.174 DEBUG:rgw_multi.tests:master meta markers={0: '1_1706782453.132619_30.1', 1: '', 2: '1_1706783313.054991_118.1', 3: '1_1706782595.670952_121.1'} 2024-02-01T10:28:33.174 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-02-01T10:28:33.175 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:28:33.175 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:28:33.175 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:28:33.216 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:28:33.216 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:28:33.273 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:28:33.275+0000 7f15f9ac6980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:28:33.274 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:28:33.275+0000 7f15f9ac6980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:28:33.297 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783313.054991_118.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:28:33.054991Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:28:33Z"} 2024-02-01T10:28:33.297 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783313.054991_118.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:28:33.054991Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:28:33Z"} 2024-02-01T10:28:33.298 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706783313.054991_118.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:28:33.054991Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:28:33.298 DEBUG:rgw_multi.tests:log_status={0: '1_1706782453.132619_30.1', 1: '', 2: '1_1706783313.054991_118.1', 3: '1_1706782595.670952_121.1'} 2024-02-01T10:28:33.298 DEBUG:rgw_multi.tests:sync_status={0: '1_1706782453.132619_30.1', 1: '', 2: '1_1706783313.054991_118.1', 3: '1_1706782595.670952_121.1'} 2024-02-01T10:28:33.298 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-02-01T10:28:33.298 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-02-01T10:28:33.298 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:28:33.298 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:28:33.299 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:28:33.340 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:28:33.341 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:28:33.396 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:28:33.399+0000 7f90e5273980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:28:33.397 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:28:33.403+0000 7f90e5273980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:28:33.420 INFO:teuthology.orchestra.run.smithi148.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783313.054991_118.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:28:33.054991Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:28:33Z"} 2024-02-01T10:28:33.421 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783313.054991_118.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:28:33.054991Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:28:33Z"} 2024-02-01T10:28:33.421 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706783313.054991_118.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:28:33.054991Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:28:33.421 DEBUG:rgw_multi.tests:log_status={0: '1_1706782453.132619_30.1', 1: '', 2: '1_1706783313.054991_118.1', 3: '1_1706782595.670952_121.1'} 2024-02-01T10:28:33.421 DEBUG:rgw_multi.tests:sync_status={0: '1_1706782453.132619_30.1', 1: '', 2: '1_1706783313.054991_118.1', 3: '1_1706782595.670952_121.1'} 2024-02-01T10:28:33.421 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-02-01T10:28:33.421 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:28:33.421 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:28:33.422 DEBUG:boto:Method: HEAD 2024-02-01T10:28:33.422 DEBUG:boto:Path: /vclfcn-20/ 2024-02-01T10:28:33.422 DEBUG:boto:Data: 2024-02-01T10:28:33.422 DEBUG:boto:Headers: {} 2024-02-01T10:28:33.422 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:28:33.422 DEBUG:boto:Port: 8000 2024-02-01T10:28:33.422 DEBUG:boto:Params: {} 2024-02-01T10:28:33.423 DEBUG:boto:Token: None 2024-02-01T10:28:33.423 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:28:33 GMT /vclfcn-20/ 2024-02-01T10:28:33.423 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HRfw4YDfjeQUTJyqb+vN8/jmlIE= 2024-02-01T10:28:33.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:28:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HRfw4YDfjeQUTJyqb+vN8/jmlIE=', 'Content-Length': '0'} 2024-02-01T10:28:33.425 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008c1dc2123aebaa77-0065bb7251-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:28:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:28:33.426 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:28:33.426 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:28:33.426 DEBUG:boto:Method: PUT 2024-02-01T10:28:33.426 DEBUG:boto:Path: /vclfcn-20/obj 2024-02-01T10:28:33.426 DEBUG:boto:Data: 2024-02-01T10:28:33.426 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-02-01T10:28:33.426 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:28:33.426 DEBUG:boto:Port: 8000 2024-02-01T10:28:33.427 DEBUG:boto:Params: {} 2024-02-01T10:28:33.427 DEBUG:boto:Token: None 2024-02-01T10:28:33.427 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Thu, 01 Feb 2024 10:28:33 GMT /vclfcn-20/obj 2024-02-01T10:28:33.427 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tQ09OxUDjZx939CXm/ABAk/7T0Y= 2024-02-01T10:28:33.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:28:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tQ09OxUDjZx939CXm/ABAk/7T0Y='} 2024-02-01T10:28:33.441 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6'), ('x-amz-request-id', 'tx000004cd27fbaec36f273-0065bb7251-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:28:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:28:33.441 DEBUG:rgw_multi.tests:created new version id=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:28:33.441 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=vclfcn-20 2024-02-01T10:28:33.442 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:28:33.442 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:28:33.442 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket vclfcn-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-02-01T10:28:33.492 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:28:33.492 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:28:33.556 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:28:33.559+0000 7fd554c77980 1 waiting for incremental sync to catch up: 2024-02-01T10:28:33.557 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, , , , , , , , 00000000001.34.6, , ] 2024-02-01T10:28:33.557 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, , , , , , , , 00000000003.37.13, , ] 2024-02-01T10:28:38.558 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:28:38.559+0000 7fd554c77980 1 waiting for incremental sync to catch up: 2024-02-01T10:28:38.559 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, , , , , , , , 00000000001.34.6, , ] 2024-02-01T10:28:38.559 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, , , , , , , , 00000000003.37.13, , ] 2024-02-01T10:28:43.564 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:28:43.563+0000 7fd554c77980 1 waiting for incremental sync to catch up: 2024-02-01T10:28:43.564 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, , , , , , , , 00000000001.34.6, , ] 2024-02-01T10:28:43.564 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, , , , , , , , 00000000003.37.13, , ] 2024-02-01T10:28:48.563 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:28:48.563+0000 7fd554c77980 1 waiting for incremental sync to catch up: 2024-02-01T10:28:48.563 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, , , , , , , , 00000000002.36.6, , ] 2024-02-01T10:28:48.563 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, , , , , , , , 00000000003.37.13, , ] 2024-02-01T10:28:53.565 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:28:53.567+0000 7fd554c77980 1 waiting for incremental sync to catch up: 2024-02-01T10:28:53.566 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, , , , , , , , 00000000002.36.6, , ] 2024-02-01T10:28:53.566 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, , , , , , , , 00000000003.37.13, , ] 2024-02-01T10:28:58.567 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:28:58.567+0000 7fd554c77980 1 waiting for incremental sync to catch up: 2024-02-01T10:28:58.568 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, , , , , , , , 00000000002.36.6, , ] 2024-02-01T10:28:58.568 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, , , , , , , , 00000000003.37.13, , ] 2024-02-01T10:29:03.570 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:29:03.570+0000 7fd554c77980 1 waiting for incremental sync to catch up: 2024-02-01T10:29:03.570 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, , , , , , , , 00000000002.36.6, , ] 2024-02-01T10:29:03.570 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, , , , , , , , 00000000003.37.13, , ] 2024-02-01T10:29:08.572 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:29:08.574+0000 7fd554c77980 1 waiting for incremental sync to catch up: 2024-02-01T10:29:08.572 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, , , , , , , , 00000000002.36.6, , ] 2024-02-01T10:29:08.573 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, , , , , , , , 00000000003.37.13, , ] 2024-02-01T10:29:13.574 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:29:13.574+0000 7fd554c77980 1 waiting for incremental sync to catch up: 2024-02-01T10:29:13.575 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, , , , , , , , 00000000002.36.6, , ] 2024-02-01T10:29:13.575 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, , , , , , , , 00000000003.37.13, , ] 2024-02-01T10:29:18.577 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:29:18.578+0000 7fd554c77980 1 waiting for incremental sync to catch up: 2024-02-01T10:29:18.577 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, , , , , , , , 00000000002.36.6, , ] 2024-02-01T10:29:18.577 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, , , , , , , , 00000000003.37.13, , ] 2024-02-01T10:29:23.579 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:29:23.578+0000 7fd554c77980 1 waiting for incremental sync to catch up: 2024-02-01T10:29:23.579 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, , , , , , , , 00000000002.36.6, , ] 2024-02-01T10:29:23.579 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, , , , , , , , 00000000003.37.13, , ] 2024-02-01T10:29:28.581 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:29:28.582+0000 7fd554c77980 1 waiting for incremental sync to catch up: 2024-02-01T10:29:28.581 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, , , , , , , , 00000000002.36.6, , ] 2024-02-01T10:29:28.582 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, , , , , , , , 00000000003.37.13, , ] 2024-02-01T10:29:33.584 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:29:33.582+0000 7fd554c77980 1 waiting for incremental sync to catch up: 2024-02-01T10:29:33.584 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, , , , , , , , 00000000002.36.6, , ] 2024-02-01T10:29:33.584 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, , , , , , , , 00000000003.37.13, , ] 2024-02-01T10:29:38.586 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:29:38.586+0000 7fd554c77980 1 waiting for incremental sync to catch up: 2024-02-01T10:29:38.586 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, , , , , , , , 00000000002.36.6, , ] 2024-02-01T10:29:38.586 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, , , , , , , , 00000000003.37.13, , ] 2024-02-01T10:29:43.588 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:29:43.590+0000 7fd554c77980 1 waiting for incremental sync to catch up: 2024-02-01T10:29:43.589 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, , , , , , , , 00000000002.36.6, , ] 2024-02-01T10:29:43.589 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, , , , , , , , 00000000003.37.13, , ] 2024-02-01T10:29:48.591 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:29:48.589+0000 7fd554c77980 1 waiting for incremental sync to catch up: 2024-02-01T10:29:48.591 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, , , , , , , , 00000000002.36.6, , ] 2024-02-01T10:29:48.591 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, , , , , , , , 00000000003.37.13, , ] 2024-02-01T10:29:53.593 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:29:53.593+0000 7fd554c77980 1 waiting for incremental sync to catch up: 2024-02-01T10:29:53.593 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, , , , , , , , 00000000002.36.6, , ] 2024-02-01T10:29:53.593 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, , , , , , , , 00000000003.37.13, , ] 2024-02-01T10:29:58.595 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:29:58.593+0000 7fd554c77980 1 waiting for incremental sync to catch up: 2024-02-01T10:29:58.596 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, , , , , , , , 00000000002.36.6, , ] 2024-02-01T10:29:58.596 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, , , , , , , , 00000000003.37.13, , ] 2024-02-01T10:30:03.599 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:30:03.597+0000 7fd554c77980 1 waiting for incremental sync to catch up: 2024-02-01T10:30:03.599 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, , , , , , , , 00000000002.36.6, , ] 2024-02-01T10:30:03.599 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, , , , , , , , 00000000003.37.13, , ] 2024-02-01T10:30:08.601 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:30:08.601+0000 7fd554c77980 1 waiting for incremental sync to catch up: 2024-02-01T10:30:08.601 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, , , , , , , , 00000000002.36.6, , ] 2024-02-01T10:30:08.601 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, , , , , , , , 00000000003.37.13, , ] 2024-02-01T10:30:13.603 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:30:13.601+0000 7fd554c77980 1 waiting for incremental sync to catch up: 2024-02-01T10:30:13.603 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, , , , , , , , 00000000002.36.6, , ] 2024-02-01T10:30:13.603 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, , , , , , , , 00000000003.37.13, , ] 2024-02-01T10:30:18.605 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:30:18.605+0000 7fd554c77980 1 waiting for incremental sync to catch up: 2024-02-01T10:30:18.606 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, , , , , , , , 00000000002.36.6, , ] 2024-02-01T10:30:18.606 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, , , , , , , , 00000000003.37.13, , ] 2024-02-01T10:30:23.608 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:30:23.605+0000 7fd554c77980 1 waiting for incremental sync to catch up: 2024-02-01T10:30:23.608 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, , , , , , , , 00000000002.36.6, , ] 2024-02-01T10:30:23.608 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, , , , , , , , 00000000003.37.13, , ] 2024-02-01T10:30:28.610 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:30:28.609+0000 7fd554c77980 1 waiting for incremental sync to catch up: 2024-02-01T10:30:28.610 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, , , , , , , , 00000000002.36.6, , ] 2024-02-01T10:30:28.610 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, , , , , , , , 00000000003.37.13, , ] 2024-02-01T10:30:33.612 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:30:33.609+0000 7fd554c77980 1 waiting for incremental sync to catch up: 2024-02-01T10:30:33.612 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, , , , , , , , 00000000002.36.6, , ] 2024-02-01T10:30:33.613 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, , , , , , , , 00000000003.37.13, , ] 2024-02-01T10:30:38.614 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:30:38.612+0000 7fd554c77980 1 waiting for incremental sync to catch up: 2024-02-01T10:30:38.615 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, , , , , , , , 00000000002.36.6, , ] 2024-02-01T10:30:38.615 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, , , , , , , , 00000000003.37.13, , ] 2024-02-01T10:30:43.616 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:30:43.616+0000 7fd554c77980 1 waiting for incremental sync to catch up: 2024-02-01T10:30:43.617 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, , , , , , , , 00000000002.36.6, , ] 2024-02-01T10:30:43.617 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, , , , , , , , 00000000003.37.13, , ] 2024-02-01T10:30:48.619 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:30:48.620+0000 7fd554c77980 1 waiting for incremental sync to catch up: 2024-02-01T10:30:48.619 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, , , , , , , , 00000000002.36.6, , ] 2024-02-01T10:30:48.619 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, , , , , , , , 00000000003.37.13, , ] 2024-02-01T10:30:53.621 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:30:53.624+0000 7fd554c77980 1 waiting for incremental sync to catch up: 2024-02-01T10:30:53.621 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, , , , , , , , 00000000002.36.6, , ] 2024-02-01T10:30:53.621 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, , , , , , , , 00000000003.37.13, , ] 2024-02-01T10:30:58.624 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:30:58.624+0000 7fd554c77980 1 waiting for incremental sync to catch up: 2024-02-01T10:30:58.624 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, , , , , , , , 00000000002.36.6, , ] 2024-02-01T10:30:58.624 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, , , , , , , , 00000000003.37.13, , ] 2024-02-01T10:31:03.626 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:31:03.628+0000 7fd554c77980 1 waiting for incremental sync to catch up: 2024-02-01T10:31:03.626 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, , , , , , , , 00000000002.36.6, , ] 2024-02-01T10:31:03.626 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, , , , , , , , 00000000003.37.13, , ] 2024-02-01T10:31:08.628 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:31:08.628+0000 7fd554c77980 1 waiting for incremental sync to catch up: 2024-02-01T10:31:08.628 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, , , , , , , , 00000000002.36.6, , ] 2024-02-01T10:31:08.628 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, , , , , , , , 00000000003.37.13, , ] 2024-02-01T10:31:13.630 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:31:13.632+0000 7fd554c77980 1 waiting for incremental sync to catch up: 2024-02-01T10:31:13.631 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, , , , , , , , 00000000002.36.6, , ] 2024-02-01T10:31:13.631 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, , , , , , , , 00000000003.37.13, , ] 2024-02-01T10:31:18.633 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:31:18.636+0000 7fd554c77980 1 waiting for incremental sync to catch up: 2024-02-01T10:31:18.633 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, , , , , , , , 00000000002.36.6, , ] 2024-02-01T10:31:18.633 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, , , , , , , , 00000000003.37.13, , ] 2024-02-01T10:31:23.635 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:31:23.635+0000 7fd554c77980 1 waiting for incremental sync to catch up: 2024-02-01T10:31:23.636 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, , , , , , , , 00000000002.36.6, , ] 2024-02-01T10:31:23.636 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, , , , , , , , 00000000003.37.13, , ] 2024-02-01T10:31:28.638 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:31:28.639+0000 7fd554c77980 1 bucket sync caught up with source: 2024-02-01T10:31:28.638 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, , , , , , , , 00000000003.37.13, , ] 2024-02-01T10:31:28.638 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, , , , , , , , 00000000003.37.13, , ] 2024-02-01T10:31:28.638 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:31:28.639+0000 7fd554c77980 0 bucket checkpoint complete 2024-02-01T10:31:28.677 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=vclfcn-20 2024-02-01T10:31:28.677 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:31:28.677 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:31:28.678 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:31:28.729 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:31:28.729 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:31:28.789 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:31:28.796+0000 7fe63b58e980 1 bucket sync caught up with source: 2024-02-01T10:31:28.790 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, , , , , , , , 00000000003.37.13, , ] 2024-02-01T10:31:28.790 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, , , , , , , , 00000000003.37.13, , ] 2024-02-01T10:31:28.790 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:31:28.796+0000 7fe63b58e980 0 bucket checkpoint complete 2024-02-01T10:31:28.811 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=vclfcn-20 2024-02-01T10:31:28.811 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:31:28.811 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:31:28.812 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:31:28.852 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:31:28.852 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:31:28.913 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:31:28.920+0000 7f68e7591980 1 waiting for incremental sync to catch up: 2024-02-01T10:31:28.913 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, , , , , , , , 00000000003.41.13, , ] 2024-02-01T10:31:28.914 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, , , , , , , , 00000000004.43.10, , ] 2024-02-01T10:31:33.916 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:31:33.924+0000 7f68e7591980 1 waiting for incremental sync to catch up: 2024-02-01T10:31:33.916 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, , , , , , , , 00000000003.41.13, , ] 2024-02-01T10:31:33.916 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, , , , , , , , 00000000004.43.10, , ] 2024-02-01T10:31:38.918 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:31:38.924+0000 7f68e7591980 1 waiting for incremental sync to catch up: 2024-02-01T10:31:38.918 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, , , , , , , , 00000000003.41.13, , ] 2024-02-01T10:31:38.918 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, , , , , , , , 00000000004.43.10, , ] 2024-02-01T10:31:43.920 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:31:43.927+0000 7f68e7591980 1 waiting for incremental sync to catch up: 2024-02-01T10:31:43.920 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, , , , , , , , 00000000003.41.13, , ] 2024-02-01T10:31:43.921 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, , , , , , , , 00000000004.43.10, , ] 2024-02-01T10:31:48.923 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:31:48.927+0000 7f68e7591980 1 bucket sync caught up with source: 2024-02-01T10:31:48.923 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, , , , , , , , 00000000004.43.10, , ] 2024-02-01T10:31:48.923 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, , , , , , , , 00000000004.43.10, , ] 2024-02-01T10:31:48.924 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:31:48.927+0000 7f68e7591980 0 bucket checkpoint complete 2024-02-01T10:31:48.965 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=vclfcn-20 2024-02-01T10:31:48.965 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:31:48.966 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:31:48.966 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:31:49.055 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:31:49.056 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:31:49.120 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:31:49.127+0000 7fd2c934b980 1 bucket sync caught up with source: 2024-02-01T10:31:49.120 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, , , , , , , , 00000000004.43.10, , ] 2024-02-01T10:31:49.120 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, , , , , , , , 00000000004.43.10, , ] 2024-02-01T10:31:49.120 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:31:49.127+0000 7fd2c934b980 0 bucket checkpoint complete 2024-02-01T10:31:49.142 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=vclfcn-20 2024-02-01T10:31:49.142 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:31:49.142 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:31:49.143 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:31:49.183 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:31:49.184 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:31:49.239 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:31:49.247+0000 7fe96ae58980 1 bucket sync caught up with source: 2024-02-01T10:31:49.240 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, , , , , , , , 00000000004.75.10, , ] 2024-02-01T10:31:49.240 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, , , , , , , , 00000000004.75.10, , ] 2024-02-01T10:31:49.240 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:31:49.247+0000 7fe96ae58980 0 bucket checkpoint complete 2024-02-01T10:31:49.263 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=vclfcn-20 2024-02-01T10:31:49.264 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:31:49.264 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:31:49.264 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket vclfcn-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-02-01T10:31:49.313 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:31:49.313 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:31:49.379 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:31:49.379+0000 7ff68e0b6980 1 bucket sync caught up with source: 2024-02-01T10:31:49.379 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, , , , , , , , 00000000004.75.10, , ] 2024-02-01T10:31:49.379 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, , , , , , , , 00000000004.75.10, , ] 2024-02-01T10:31:49.379 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:31:49.379+0000 7ff68e0b6980 0 bucket checkpoint complete 2024-02-01T10:31:49.401 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-20 zones={test-zone2, test-zone1} 2024-02-01T10:31:49.402 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:31:49.402 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:31:49.402 DEBUG:boto:Method: HEAD 2024-02-01T10:31:49.402 DEBUG:boto:Path: /vclfcn-20/ 2024-02-01T10:31:49.402 DEBUG:boto:Data: 2024-02-01T10:31:49.402 DEBUG:boto:Headers: {} 2024-02-01T10:31:49.402 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:31:49.402 DEBUG:boto:Port: 8000 2024-02-01T10:31:49.402 DEBUG:boto:Params: {} 2024-02-01T10:31:49.403 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T10:31:49.403 DEBUG:boto:Token: None 2024-02-01T10:31:49.404 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:31:49 GMT /vclfcn-20/ 2024-02-01T10:31:49.404 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:3Wc1rL3mvAJwE1QxaAVz7r4N7zk= 2024-02-01T10:31:49.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:3Wc1rL3mvAJwE1QxaAVz7r4N7zk=', 'Content-Length': '0'} 2024-02-01T10:31:49.410 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000007bf095b7f02d08b-0065bb7315-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:31:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:49.410 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:31:49.410 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:31:49.411 DEBUG:boto:Method: HEAD 2024-02-01T10:31:49.411 DEBUG:boto:Path: /vclfcn-20/ 2024-02-01T10:31:49.411 DEBUG:boto:Data: 2024-02-01T10:31:49.411 DEBUG:boto:Headers: {} 2024-02-01T10:31:49.411 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:31:49.411 DEBUG:boto:Port: 8000 2024-02-01T10:31:49.411 DEBUG:boto:Params: {} 2024-02-01T10:31:49.411 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T10:31:49.411 DEBUG:boto:Token: None 2024-02-01T10:31:49.411 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:31:49 GMT /vclfcn-20/ 2024-02-01T10:31:49.412 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:3Wc1rL3mvAJwE1QxaAVz7r4N7zk= 2024-02-01T10:31:49.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:3Wc1rL3mvAJwE1QxaAVz7r4N7zk=', 'Content-Length': '0'} 2024-02-01T10:31:49.420 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a8bedcecf23bddc9-0065bb7315-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:31:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:49.420 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:31:49.421 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:31:49.421 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:31:49.421 DEBUG:boto:path=/vclfcn-20/?versions&max-keys=999 2024-02-01T10:31:49.421 DEBUG:boto:auth_path=/vclfcn-20/?versions&max-keys=999 2024-02-01T10:31:49.421 DEBUG:boto:Method: GET 2024-02-01T10:31:49.421 DEBUG:boto:Path: /vclfcn-20/?versions&max-keys=999 2024-02-01T10:31:49.421 DEBUG:boto:Data: 2024-02-01T10:31:49.421 DEBUG:boto:Headers: {} 2024-02-01T10:31:49.421 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:31:49.422 DEBUG:boto:Port: 8000 2024-02-01T10:31:49.422 DEBUG:boto:Params: {} 2024-02-01T10:31:49.422 DEBUG:boto:Token: None 2024-02-01T10:31:49.422 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:31:49 GMT /vclfcn-20/?versions 2024-02-01T10:31:49.422 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9vimAxEjZmQnwBZsg52UKTmNUQs= 2024-02-01T10:31:49.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9vimAxEjZmQnwBZsg52UKTmNUQs=', 'Content-Length': '0'} 2024-02-01T10:31:49.425 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a5411e9c1795f4d-0065bb7315-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:49.425 DEBUG:boto:b'vclfcn-20999falseobjrSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6true2024-02-01T10:28:33.438Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2024-02-01T10:28:01.521Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:31:49.426 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:31:49.426 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:31:49.426 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:31:49.426 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:31:49.426 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:31:49.426 DEBUG:boto:path=/vclfcn-20/?versions&max-keys=999 2024-02-01T10:31:49.426 DEBUG:boto:auth_path=/vclfcn-20/?versions&max-keys=999 2024-02-01T10:31:49.426 DEBUG:boto:Method: GET 2024-02-01T10:31:49.426 DEBUG:boto:Path: /vclfcn-20/?versions&max-keys=999 2024-02-01T10:31:49.427 DEBUG:boto:Data: 2024-02-01T10:31:49.427 DEBUG:boto:Headers: {} 2024-02-01T10:31:49.427 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:31:49.427 DEBUG:boto:Port: 8000 2024-02-01T10:31:49.427 DEBUG:boto:Params: {} 2024-02-01T10:31:49.427 DEBUG:boto:Token: None 2024-02-01T10:31:49.427 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:31:49 GMT /vclfcn-20/?versions 2024-02-01T10:31:49.427 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9vimAxEjZmQnwBZsg52UKTmNUQs= 2024-02-01T10:31:49.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9vimAxEjZmQnwBZsg52UKTmNUQs=', 'Content-Length': '0'} 2024-02-01T10:31:49.432 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000056e8b0f031cf96c3-0065bb7315-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:49.432 DEBUG:boto:b'vclfcn-20999falseobjrSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6true2024-02-01T10:28:33.438Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2024-02-01T10:28:01.521Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:31:49.433 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:31:49.433 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:31:49.433 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:31:49.433 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:31:49.433 DEBUG:boto:path=/vclfcn-20/?versions&max-keys=999 2024-02-01T10:31:49.433 DEBUG:boto:auth_path=/vclfcn-20/?versions&max-keys=999 2024-02-01T10:31:49.433 DEBUG:boto:Method: GET 2024-02-01T10:31:49.433 DEBUG:boto:Path: /vclfcn-20/?versions&max-keys=999 2024-02-01T10:31:49.433 DEBUG:boto:Data: 2024-02-01T10:31:49.434 DEBUG:boto:Headers: {} 2024-02-01T10:31:49.434 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:31:49.434 DEBUG:boto:Port: 8000 2024-02-01T10:31:49.434 DEBUG:boto:Params: {} 2024-02-01T10:31:49.434 DEBUG:boto:Token: None 2024-02-01T10:31:49.434 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:31:49 GMT /vclfcn-20/?versions 2024-02-01T10:31:49.434 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9vimAxEjZmQnwBZsg52UKTmNUQs= 2024-02-01T10:31:49.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9vimAxEjZmQnwBZsg52UKTmNUQs=', 'Content-Length': '0'} 2024-02-01T10:31:49.437 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000074081e1631bff544-0065bb7315-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:49.437 DEBUG:boto:b'vclfcn-20999falseobjrSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6true2024-02-01T10:28:33.438Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2024-02-01T10:28:01.521Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:31:49.437 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:31:49.437 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:31:49.437 DEBUG:boto:path=/vclfcn-20/?versions&max-keys=999 2024-02-01T10:31:49.437 DEBUG:boto:auth_path=/vclfcn-20/?versions&max-keys=999 2024-02-01T10:31:49.438 DEBUG:boto:Method: GET 2024-02-01T10:31:49.438 DEBUG:boto:Path: /vclfcn-20/?versions&max-keys=999 2024-02-01T10:31:49.438 DEBUG:boto:Data: 2024-02-01T10:31:49.438 DEBUG:boto:Headers: {} 2024-02-01T10:31:49.438 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:31:49.438 DEBUG:boto:Port: 8000 2024-02-01T10:31:49.438 DEBUG:boto:Params: {} 2024-02-01T10:31:49.438 DEBUG:boto:Token: None 2024-02-01T10:31:49.438 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:31:49 GMT /vclfcn-20/?versions 2024-02-01T10:31:49.438 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9vimAxEjZmQnwBZsg52UKTmNUQs= 2024-02-01T10:31:49.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9vimAxEjZmQnwBZsg52UKTmNUQs=', 'Content-Length': '0'} 2024-02-01T10:31:49.442 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6ba5ce4cf6b23e1-0065bb7315-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:49.443 DEBUG:boto:b'vclfcn-20999falseobjrSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6true2024-02-01T10:28:33.438Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2024-02-01T10:28:01.521Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:31:49.443 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:31:49.443 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:31:49.443 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:31:49.443 DEBUG:boto:path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:49.443 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:49.443 DEBUG:boto:Method: GET 2024-02-01T10:31:49.444 DEBUG:boto:Path: /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:49.444 DEBUG:boto:Data: 2024-02-01T10:31:49.444 DEBUG:boto:Headers: {} 2024-02-01T10:31:49.444 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:31:49.444 DEBUG:boto:Port: 8000 2024-02-01T10:31:49.444 DEBUG:boto:Params: {} 2024-02-01T10:31:49.444 DEBUG:boto:Token: None 2024-02-01T10:31:49.444 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:31:49 GMT /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:49.444 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:uOcyU8SC6g5VJOjqxXFkEiCuTW4= 2024-02-01T10:31:49.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:uOcyU8SC6g5VJOjqxXFkEiCuTW4=', 'Content-Length': '0'} 2024-02-01T10:31:49.446 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:33 GMT'), ('x-amz-version-id', 'rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d5dbf1545e156bc7-0065bb7315-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:49.447 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:31:49.447 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:31:49.447 DEBUG:boto:path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:49.447 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:49.447 DEBUG:boto:Method: GET 2024-02-01T10:31:49.447 DEBUG:boto:Path: /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:49.447 DEBUG:boto:Data: 2024-02-01T10:31:49.447 DEBUG:boto:Headers: {} 2024-02-01T10:31:49.447 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:31:49.448 DEBUG:boto:Port: 8000 2024-02-01T10:31:49.448 DEBUG:boto:Params: {} 2024-02-01T10:31:49.448 DEBUG:boto:Token: None 2024-02-01T10:31:49.448 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:31:49 GMT /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:49.448 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:uOcyU8SC6g5VJOjqxXFkEiCuTW4= 2024-02-01T10:31:49.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:uOcyU8SC6g5VJOjqxXFkEiCuTW4=', 'Content-Length': '0'} 2024-02-01T10:31:49.452 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:33 GMT'), ('x-amz-version-id', 'rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000af61473a7994477e-0065bb7315-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:49.452 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:31:49.452 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:31:49.452 DEBUG:boto:path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:49.452 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:49.452 DEBUG:boto:Method: HEAD 2024-02-01T10:31:49.452 DEBUG:boto:Path: /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:49.452 DEBUG:boto:Data: 2024-02-01T10:31:49.453 DEBUG:boto:Headers: {} 2024-02-01T10:31:49.453 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:31:49.453 DEBUG:boto:Port: 8000 2024-02-01T10:31:49.453 DEBUG:boto:Params: {} 2024-02-01T10:31:49.453 DEBUG:boto:Token: None 2024-02-01T10:31:49.453 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:31:49 GMT /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:49.453 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NKi2D68fViA+/b+natPKHKoQxJk= 2024-02-01T10:31:49.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NKi2D68fViA+/b+natPKHKoQxJk=', 'Content-Length': '0'} 2024-02-01T10:31:49.455 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:33 GMT'), ('x-amz-version-id', 'rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000653fabbdd94b4d5f-0065bb7315-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:49.455 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:31:49.455 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:31:49.456 DEBUG:boto:path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:49.456 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:49.456 DEBUG:boto:Method: HEAD 2024-02-01T10:31:49.456 DEBUG:boto:Path: /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:49.456 DEBUG:boto:Data: 2024-02-01T10:31:49.456 DEBUG:boto:Headers: {} 2024-02-01T10:31:49.456 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:31:49.456 DEBUG:boto:Port: 8000 2024-02-01T10:31:49.456 DEBUG:boto:Params: {} 2024-02-01T10:31:49.456 DEBUG:boto:Token: None 2024-02-01T10:31:49.457 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:31:49 GMT /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:49.457 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NKi2D68fViA+/b+natPKHKoQxJk= 2024-02-01T10:31:49.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NKi2D68fViA+/b+natPKHKoQxJk=', 'Content-Length': '0'} 2024-02-01T10:31:49.459 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:33 GMT'), ('x-amz-version-id', 'rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e9959ad4137cff03-0065bb7315-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:49.460 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:31:49.460 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:31:49.460 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:31:49.460 DEBUG:boto:path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:49.460 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:49.460 DEBUG:boto:Method: GET 2024-02-01T10:31:49.460 DEBUG:boto:Path: /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:49.460 DEBUG:boto:Data: 2024-02-01T10:31:49.460 DEBUG:boto:Headers: {} 2024-02-01T10:31:49.460 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:31:49.460 DEBUG:boto:Port: 8000 2024-02-01T10:31:49.461 DEBUG:boto:Params: {} 2024-02-01T10:31:49.461 DEBUG:boto:Token: None 2024-02-01T10:31:49.461 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:31:49 GMT /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:49.461 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:uOcyU8SC6g5VJOjqxXFkEiCuTW4= 2024-02-01T10:31:49.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:uOcyU8SC6g5VJOjqxXFkEiCuTW4=', 'Content-Length': '0'} 2024-02-01T10:31:49.463 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:33 GMT'), ('x-amz-version-id', 'rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001ec0347bbe6b0109-0065bb7315-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:49.464 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:31:49.464 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:31:49.464 DEBUG:boto:path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:49.464 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:49.464 DEBUG:boto:Method: GET 2024-02-01T10:31:49.464 DEBUG:boto:Path: /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:49.464 DEBUG:boto:Data: 2024-02-01T10:31:49.464 DEBUG:boto:Headers: {} 2024-02-01T10:31:49.464 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:31:49.464 DEBUG:boto:Port: 8000 2024-02-01T10:31:49.464 DEBUG:boto:Params: {} 2024-02-01T10:31:49.465 DEBUG:boto:Token: None 2024-02-01T10:31:49.465 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:31:49 GMT /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:49.465 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:uOcyU8SC6g5VJOjqxXFkEiCuTW4= 2024-02-01T10:31:49.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:uOcyU8SC6g5VJOjqxXFkEiCuTW4=', 'Content-Length': '0'} 2024-02-01T10:31:49.467 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:33 GMT'), ('x-amz-version-id', 'rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009947cfbb8e116777-0065bb7315-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:49.468 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:31:49.468 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:31:49.468 DEBUG:boto:path=/vclfcn-20/?prefix=obj 2024-02-01T10:31:49.468 DEBUG:boto:auth_path=/vclfcn-20/?prefix=obj 2024-02-01T10:31:49.468 DEBUG:boto:Method: GET 2024-02-01T10:31:49.468 DEBUG:boto:Path: /vclfcn-20/?prefix=obj 2024-02-01T10:31:49.468 DEBUG:boto:Data: 2024-02-01T10:31:49.468 DEBUG:boto:Headers: {} 2024-02-01T10:31:49.468 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:31:49.469 DEBUG:boto:Port: 8000 2024-02-01T10:31:49.469 DEBUG:boto:Params: {} 2024-02-01T10:31:49.469 DEBUG:boto:Token: None 2024-02-01T10:31:49.469 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:31:49 GMT /vclfcn-20/ 2024-02-01T10:31:49.469 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9mX1KS2dWfNIEIhGw3onxrrn8KM= 2024-02-01T10:31:49.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9mX1KS2dWfNIEIhGw3onxrrn8KM=', 'Content-Length': '0'} 2024-02-01T10:31:49.471 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019cdadebb5746656-0065bb7315-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:49.512 DEBUG:boto:b'vclfcn-20obj1000falseobj2024-02-01T10:28:33.438Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:31:49.513 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:31:49.513 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:31:49.513 DEBUG:boto:path=/vclfcn-20/?prefix=obj 2024-02-01T10:31:49.513 DEBUG:boto:auth_path=/vclfcn-20/?prefix=obj 2024-02-01T10:31:49.513 DEBUG:boto:Method: GET 2024-02-01T10:31:49.513 DEBUG:boto:Path: /vclfcn-20/?prefix=obj 2024-02-01T10:31:49.513 DEBUG:boto:Data: 2024-02-01T10:31:49.513 DEBUG:boto:Headers: {} 2024-02-01T10:31:49.513 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:31:49.513 DEBUG:boto:Port: 8000 2024-02-01T10:31:49.513 DEBUG:boto:Params: {} 2024-02-01T10:31:49.514 DEBUG:boto:Token: None 2024-02-01T10:31:49.514 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:31:49 GMT /vclfcn-20/ 2024-02-01T10:31:49.514 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9mX1KS2dWfNIEIhGw3onxrrn8KM= 2024-02-01T10:31:49.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9mX1KS2dWfNIEIhGw3onxrrn8KM=', 'Content-Length': '0'} 2024-02-01T10:31:49.518 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000738e9a1160113c55-0065bb7315-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:49.519 DEBUG:boto:b'vclfcn-20obj1000falseobj2024-02-01T10:28:33.438Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:31:49.519 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:31:49.519 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:31:49.519 DEBUG:boto:Method: HEAD 2024-02-01T10:31:49.519 DEBUG:boto:Path: /vclfcn-20/obj 2024-02-01T10:31:49.520 DEBUG:boto:Data: 2024-02-01T10:31:49.520 DEBUG:boto:Headers: {} 2024-02-01T10:31:49.520 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:31:49.520 DEBUG:boto:Port: 8000 2024-02-01T10:31:49.520 DEBUG:boto:Params: {} 2024-02-01T10:31:49.520 DEBUG:boto:Token: None 2024-02-01T10:31:49.520 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:31:49 GMT /vclfcn-20/obj 2024-02-01T10:31:49.520 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1LuBuFPpmGlGLTJVFACAiUm7qdY= 2024-02-01T10:31:49.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1LuBuFPpmGlGLTJVFACAiUm7qdY=', 'Content-Length': '0'} 2024-02-01T10:31:49.524 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:33 GMT'), ('x-amz-version-id', 'rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c13516ab83f5a6be-0065bb7315-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:49.524 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:31:49.524 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:31:49.525 DEBUG:boto:Method: HEAD 2024-02-01T10:31:49.525 DEBUG:boto:Path: /vclfcn-20/obj 2024-02-01T10:31:49.525 DEBUG:boto:Data: 2024-02-01T10:31:49.525 DEBUG:boto:Headers: {} 2024-02-01T10:31:49.525 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:31:49.525 DEBUG:boto:Port: 8000 2024-02-01T10:31:49.525 DEBUG:boto:Params: {} 2024-02-01T10:31:49.525 DEBUG:boto:Token: None 2024-02-01T10:31:49.525 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:31:49 GMT /vclfcn-20/obj 2024-02-01T10:31:49.525 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1LuBuFPpmGlGLTJVFACAiUm7qdY= 2024-02-01T10:31:49.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1LuBuFPpmGlGLTJVFACAiUm7qdY=', 'Content-Length': '0'} 2024-02-01T10:31:49.529 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:33 GMT'), ('x-amz-version-id', 'rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a2e580e1874d59f6-0065bb7315-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:49.529 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:31:49.530 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:31:49.530 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:31:49.530 DEBUG:boto:path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:49.530 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:49.530 DEBUG:boto:Method: GET 2024-02-01T10:31:49.530 DEBUG:boto:Path: /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:49.530 DEBUG:boto:Data: 2024-02-01T10:31:49.530 DEBUG:boto:Headers: {} 2024-02-01T10:31:49.530 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:31:49.530 DEBUG:boto:Port: 8000 2024-02-01T10:31:49.530 DEBUG:boto:Params: {} 2024-02-01T10:31:49.531 DEBUG:boto:Token: None 2024-02-01T10:31:49.531 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:31:49 GMT /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:49.531 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:uOcyU8SC6g5VJOjqxXFkEiCuTW4= 2024-02-01T10:31:49.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:uOcyU8SC6g5VJOjqxXFkEiCuTW4=', 'Content-Length': '0'} 2024-02-01T10:31:49.533 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:33 GMT'), ('x-amz-version-id', 'rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000018ca5647b4a423e4-0065bb7315-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:49.534 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:31:49.534 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:31:49.534 DEBUG:boto:path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:49.534 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:49.534 DEBUG:boto:Method: GET 2024-02-01T10:31:49.534 DEBUG:boto:Path: /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:49.534 DEBUG:boto:Data: 2024-02-01T10:31:49.534 DEBUG:boto:Headers: {} 2024-02-01T10:31:49.534 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:31:49.535 DEBUG:boto:Port: 8000 2024-02-01T10:31:49.535 DEBUG:boto:Params: {} 2024-02-01T10:31:49.535 DEBUG:boto:Token: None 2024-02-01T10:31:49.535 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:31:49 GMT /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:49.535 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:uOcyU8SC6g5VJOjqxXFkEiCuTW4= 2024-02-01T10:31:49.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:uOcyU8SC6g5VJOjqxXFkEiCuTW4=', 'Content-Length': '0'} 2024-02-01T10:31:49.538 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:33 GMT'), ('x-amz-version-id', 'rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008c421e3be097ccc1-0065bb7315-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:49.538 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:31:49.538 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:31:49.538 DEBUG:boto:path=/vclfcn-20/?prefix=obj 2024-02-01T10:31:49.538 DEBUG:boto:auth_path=/vclfcn-20/?prefix=obj 2024-02-01T10:31:49.538 DEBUG:boto:Method: GET 2024-02-01T10:31:49.538 DEBUG:boto:Path: /vclfcn-20/?prefix=obj 2024-02-01T10:31:49.539 DEBUG:boto:Data: 2024-02-01T10:31:49.539 DEBUG:boto:Headers: {} 2024-02-01T10:31:49.539 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:31:49.539 DEBUG:boto:Port: 8000 2024-02-01T10:31:49.539 DEBUG:boto:Params: {} 2024-02-01T10:31:49.539 DEBUG:boto:Token: None 2024-02-01T10:31:49.539 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:31:49 GMT /vclfcn-20/ 2024-02-01T10:31:49.539 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9mX1KS2dWfNIEIhGw3onxrrn8KM= 2024-02-01T10:31:49.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9mX1KS2dWfNIEIhGw3onxrrn8KM=', 'Content-Length': '0'} 2024-02-01T10:31:49.543 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc6dfb303f573bfd-0065bb7315-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:49.584 DEBUG:boto:b'vclfcn-20obj1000falseobj2024-02-01T10:28:33.438Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:31:49.585 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:31:49.585 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:31:49.585 DEBUG:boto:path=/vclfcn-20/?prefix=obj 2024-02-01T10:31:49.585 DEBUG:boto:auth_path=/vclfcn-20/?prefix=obj 2024-02-01T10:31:49.585 DEBUG:boto:Method: GET 2024-02-01T10:31:49.585 DEBUG:boto:Path: /vclfcn-20/?prefix=obj 2024-02-01T10:31:49.585 DEBUG:boto:Data: 2024-02-01T10:31:49.585 DEBUG:boto:Headers: {} 2024-02-01T10:31:49.585 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:31:49.585 DEBUG:boto:Port: 8000 2024-02-01T10:31:49.586 DEBUG:boto:Params: {} 2024-02-01T10:31:49.586 DEBUG:boto:Token: None 2024-02-01T10:31:49.586 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:31:49 GMT /vclfcn-20/ 2024-02-01T10:31:49.586 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9mX1KS2dWfNIEIhGw3onxrrn8KM= 2024-02-01T10:31:49.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9mX1KS2dWfNIEIhGw3onxrrn8KM=', 'Content-Length': '0'} 2024-02-01T10:31:49.590 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5dd2e4560a00a63-0065bb7315-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:49.590 DEBUG:boto:b'vclfcn-20obj1000falseobj2024-02-01T10:28:33.438Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:31:49.591 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:31:49.591 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:31:49.591 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:31:49.591 DEBUG:boto:path=/vclfcn-20/obj?versionId=null 2024-02-01T10:31:49.591 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=null 2024-02-01T10:31:49.591 DEBUG:boto:Method: GET 2024-02-01T10:31:49.591 DEBUG:boto:Path: /vclfcn-20/obj?versionId=null 2024-02-01T10:31:49.592 DEBUG:boto:Data: 2024-02-01T10:31:49.592 DEBUG:boto:Headers: {} 2024-02-01T10:31:49.592 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:31:49.592 DEBUG:boto:Port: 8000 2024-02-01T10:31:49.592 DEBUG:boto:Params: {} 2024-02-01T10:31:49.592 DEBUG:boto:Token: None 2024-02-01T10:31:49.592 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:31:49 GMT /vclfcn-20/obj?versionId=null 2024-02-01T10:31:49.592 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YYnPDEgZKFy0QWkVF+HQnlK6oAA= 2024-02-01T10:31:49.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YYnPDEgZKFy0QWkVF+HQnlK6oAA=', 'Content-Length': '0'} 2024-02-01T10:31:49.595 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f95b6aa0dc010773-0065bb7315-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:49.595 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:31:49.595 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:31:49.595 DEBUG:boto:path=/vclfcn-20/obj?versionId=null 2024-02-01T10:31:49.596 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=null 2024-02-01T10:31:49.596 DEBUG:boto:Method: GET 2024-02-01T10:31:49.596 DEBUG:boto:Path: /vclfcn-20/obj?versionId=null 2024-02-01T10:31:49.596 DEBUG:boto:Data: 2024-02-01T10:31:49.596 DEBUG:boto:Headers: {} 2024-02-01T10:31:49.596 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:31:49.596 DEBUG:boto:Port: 8000 2024-02-01T10:31:49.596 DEBUG:boto:Params: {} 2024-02-01T10:31:49.596 DEBUG:boto:Token: None 2024-02-01T10:31:49.596 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:31:49 GMT /vclfcn-20/obj?versionId=null 2024-02-01T10:31:49.597 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YYnPDEgZKFy0QWkVF+HQnlK6oAA= 2024-02-01T10:31:49.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YYnPDEgZKFy0QWkVF+HQnlK6oAA=', 'Content-Length': '0'} 2024-02-01T10:31:49.600 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ffde1d0a1f149d20-0065bb7315-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:49.600 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:31:49.600 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:31:49.600 DEBUG:boto:path=/vclfcn-20/obj?versionId=null 2024-02-01T10:31:49.600 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=null 2024-02-01T10:31:49.600 DEBUG:boto:Method: HEAD 2024-02-01T10:31:49.601 DEBUG:boto:Path: /vclfcn-20/obj?versionId=null 2024-02-01T10:31:49.601 DEBUG:boto:Data: 2024-02-01T10:31:49.601 DEBUG:boto:Headers: {} 2024-02-01T10:31:49.601 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:31:49.601 DEBUG:boto:Port: 8000 2024-02-01T10:31:49.601 DEBUG:boto:Params: {} 2024-02-01T10:31:49.601 DEBUG:boto:Token: None 2024-02-01T10:31:49.601 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:31:49 GMT /vclfcn-20/obj?versionId=null 2024-02-01T10:31:49.602 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:blD1l+cS4TFMKkSa2rvXbp8Yai4= 2024-02-01T10:31:49.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:blD1l+cS4TFMKkSa2rvXbp8Yai4=', 'Content-Length': '0'} 2024-02-01T10:31:49.604 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ce8aea5d4a18fbe0-0065bb7315-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:49.604 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:31:49.605 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:31:49.605 DEBUG:boto:path=/vclfcn-20/obj?versionId=null 2024-02-01T10:31:49.605 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=null 2024-02-01T10:31:49.605 DEBUG:boto:Method: HEAD 2024-02-01T10:31:49.605 DEBUG:boto:Path: /vclfcn-20/obj?versionId=null 2024-02-01T10:31:49.605 DEBUG:boto:Data: 2024-02-01T10:31:49.605 DEBUG:boto:Headers: {} 2024-02-01T10:31:49.605 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:31:49.605 DEBUG:boto:Port: 8000 2024-02-01T10:31:49.605 DEBUG:boto:Params: {} 2024-02-01T10:31:49.606 DEBUG:boto:Token: None 2024-02-01T10:31:49.606 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:31:49 GMT /vclfcn-20/obj?versionId=null 2024-02-01T10:31:49.606 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:blD1l+cS4TFMKkSa2rvXbp8Yai4= 2024-02-01T10:31:49.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:blD1l+cS4TFMKkSa2rvXbp8Yai4=', 'Content-Length': '0'} 2024-02-01T10:31:49.608 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ac1a27eaa1ff4701-0065bb7315-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:49.609 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:31:49.609 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:31:49.609 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:31:49.609 DEBUG:boto:path=/vclfcn-20/obj?versionId=null 2024-02-01T10:31:49.609 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=null 2024-02-01T10:31:49.609 DEBUG:boto:Method: GET 2024-02-01T10:31:49.609 DEBUG:boto:Path: /vclfcn-20/obj?versionId=null 2024-02-01T10:31:49.610 DEBUG:boto:Data: 2024-02-01T10:31:49.610 DEBUG:boto:Headers: {} 2024-02-01T10:31:49.610 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:31:49.610 DEBUG:boto:Port: 8000 2024-02-01T10:31:49.610 DEBUG:boto:Params: {} 2024-02-01T10:31:49.610 DEBUG:boto:Token: None 2024-02-01T10:31:49.610 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:31:49 GMT /vclfcn-20/obj?versionId=null 2024-02-01T10:31:49.610 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YYnPDEgZKFy0QWkVF+HQnlK6oAA= 2024-02-01T10:31:49.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YYnPDEgZKFy0QWkVF+HQnlK6oAA=', 'Content-Length': '0'} 2024-02-01T10:31:49.613 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003bc190abd2fc964f-0065bb7315-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:49.613 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:31:49.613 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:31:49.614 DEBUG:boto:path=/vclfcn-20/obj?versionId=null 2024-02-01T10:31:49.614 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=null 2024-02-01T10:31:49.614 DEBUG:boto:Method: GET 2024-02-01T10:31:49.614 DEBUG:boto:Path: /vclfcn-20/obj?versionId=null 2024-02-01T10:31:49.614 DEBUG:boto:Data: 2024-02-01T10:31:49.614 DEBUG:boto:Headers: {} 2024-02-01T10:31:49.614 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:31:49.614 DEBUG:boto:Port: 8000 2024-02-01T10:31:49.615 DEBUG:boto:Params: {} 2024-02-01T10:31:49.615 DEBUG:boto:Token: None 2024-02-01T10:31:49.615 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:31:49 GMT /vclfcn-20/obj?versionId=null 2024-02-01T10:31:49.615 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YYnPDEgZKFy0QWkVF+HQnlK6oAA= 2024-02-01T10:31:49.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YYnPDEgZKFy0QWkVF+HQnlK6oAA=', 'Content-Length': '0'} 2024-02-01T10:31:49.618 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002ff361dce76f155d-0065bb7315-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:49.618 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:31:49.618 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:31:49.618 DEBUG:boto:path=/vclfcn-20/?prefix=obj 2024-02-01T10:31:49.618 DEBUG:boto:auth_path=/vclfcn-20/?prefix=obj 2024-02-01T10:31:49.618 DEBUG:boto:Method: GET 2024-02-01T10:31:49.619 DEBUG:boto:Path: /vclfcn-20/?prefix=obj 2024-02-01T10:31:49.619 DEBUG:boto:Data: 2024-02-01T10:31:49.619 DEBUG:boto:Headers: {} 2024-02-01T10:31:49.619 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:31:49.619 DEBUG:boto:Port: 8000 2024-02-01T10:31:49.619 DEBUG:boto:Params: {} 2024-02-01T10:31:49.619 DEBUG:boto:Token: None 2024-02-01T10:31:49.619 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:31:49 GMT /vclfcn-20/ 2024-02-01T10:31:49.620 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9mX1KS2dWfNIEIhGw3onxrrn8KM= 2024-02-01T10:31:49.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9mX1KS2dWfNIEIhGw3onxrrn8KM=', 'Content-Length': '0'} 2024-02-01T10:31:49.624 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000022163a42ba3d84fb-0065bb7315-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:49.668 DEBUG:boto:b'vclfcn-20obj1000falseobj2024-02-01T10:28:33.438Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:31:49.668 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:31:49.668 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:31:49.669 DEBUG:boto:path=/vclfcn-20/?prefix=obj 2024-02-01T10:31:49.669 DEBUG:boto:auth_path=/vclfcn-20/?prefix=obj 2024-02-01T10:31:49.669 DEBUG:boto:Method: GET 2024-02-01T10:31:49.669 DEBUG:boto:Path: /vclfcn-20/?prefix=obj 2024-02-01T10:31:49.669 DEBUG:boto:Data: 2024-02-01T10:31:49.669 DEBUG:boto:Headers: {} 2024-02-01T10:31:49.669 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:31:49.669 DEBUG:boto:Port: 8000 2024-02-01T10:31:49.669 DEBUG:boto:Params: {} 2024-02-01T10:31:49.669 DEBUG:boto:Token: None 2024-02-01T10:31:49.670 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:31:49 GMT /vclfcn-20/ 2024-02-01T10:31:49.670 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9mX1KS2dWfNIEIhGw3onxrrn8KM= 2024-02-01T10:31:49.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9mX1KS2dWfNIEIhGw3onxrrn8KM=', 'Content-Length': '0'} 2024-02-01T10:31:49.674 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2e274549700a932-0065bb7315-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:49.674 DEBUG:boto:b'vclfcn-20obj1000falseobj2024-02-01T10:28:33.438Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:31:49.675 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:31:49.675 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:31:49.675 DEBUG:boto:Method: HEAD 2024-02-01T10:31:49.675 DEBUG:boto:Path: /vclfcn-20/obj 2024-02-01T10:31:49.675 DEBUG:boto:Data: 2024-02-01T10:31:49.675 DEBUG:boto:Headers: {} 2024-02-01T10:31:49.675 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:31:49.675 DEBUG:boto:Port: 8000 2024-02-01T10:31:49.676 DEBUG:boto:Params: {} 2024-02-01T10:31:49.676 DEBUG:boto:Token: None 2024-02-01T10:31:49.676 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:31:49 GMT /vclfcn-20/obj 2024-02-01T10:31:49.676 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1LuBuFPpmGlGLTJVFACAiUm7qdY= 2024-02-01T10:31:49.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1LuBuFPpmGlGLTJVFACAiUm7qdY=', 'Content-Length': '0'} 2024-02-01T10:31:49.680 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:33 GMT'), ('x-amz-version-id', 'rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004e57a90200d43788-0065bb7315-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:49.680 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:31:49.680 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:31:49.680 DEBUG:boto:Method: HEAD 2024-02-01T10:31:49.680 DEBUG:boto:Path: /vclfcn-20/obj 2024-02-01T10:31:49.681 DEBUG:boto:Data: 2024-02-01T10:31:49.681 DEBUG:boto:Headers: {} 2024-02-01T10:31:49.681 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:31:49.681 DEBUG:boto:Port: 8000 2024-02-01T10:31:49.681 DEBUG:boto:Params: {} 2024-02-01T10:31:49.681 DEBUG:boto:Token: None 2024-02-01T10:31:49.681 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:31:49 GMT /vclfcn-20/obj 2024-02-01T10:31:49.681 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1LuBuFPpmGlGLTJVFACAiUm7qdY= 2024-02-01T10:31:49.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1LuBuFPpmGlGLTJVFACAiUm7qdY=', 'Content-Length': '0'} 2024-02-01T10:31:49.686 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:33 GMT'), ('x-amz-version-id', 'rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001bcccbeed58e9e40-0065bb7315-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:49.687 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:31:49.687 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:31:49.687 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:31:49.687 DEBUG:boto:path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:49.687 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:49.687 DEBUG:boto:Method: GET 2024-02-01T10:31:49.687 DEBUG:boto:Path: /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:49.687 DEBUG:boto:Data: 2024-02-01T10:31:49.688 DEBUG:boto:Headers: {} 2024-02-01T10:31:49.688 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:31:49.688 DEBUG:boto:Port: 8000 2024-02-01T10:31:49.688 DEBUG:boto:Params: {} 2024-02-01T10:31:49.688 DEBUG:boto:Token: None 2024-02-01T10:31:49.688 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:31:49 GMT /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:49.688 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:uOcyU8SC6g5VJOjqxXFkEiCuTW4= 2024-02-01T10:31:49.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:uOcyU8SC6g5VJOjqxXFkEiCuTW4=', 'Content-Length': '0'} 2024-02-01T10:31:49.691 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:33 GMT'), ('x-amz-version-id', 'rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000281d363b0b328a7f-0065bb7315-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:49.691 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:31:49.691 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:31:49.692 DEBUG:boto:path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:49.692 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:49.692 DEBUG:boto:Method: GET 2024-02-01T10:31:49.692 DEBUG:boto:Path: /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:49.692 DEBUG:boto:Data: 2024-02-01T10:31:49.692 DEBUG:boto:Headers: {} 2024-02-01T10:31:49.692 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:31:49.692 DEBUG:boto:Port: 8000 2024-02-01T10:31:49.692 DEBUG:boto:Params: {} 2024-02-01T10:31:49.692 DEBUG:boto:Token: None 2024-02-01T10:31:49.692 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:31:49 GMT /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:49.693 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:uOcyU8SC6g5VJOjqxXFkEiCuTW4= 2024-02-01T10:31:49.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:uOcyU8SC6g5VJOjqxXFkEiCuTW4=', 'Content-Length': '0'} 2024-02-01T10:31:49.696 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:33 GMT'), ('x-amz-version-id', 'rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000184b1b1bf49584a9-0065bb7315-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:49.696 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:31:49.696 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:31:49.696 DEBUG:boto:path=/vclfcn-20/?prefix=obj 2024-02-01T10:31:49.696 DEBUG:boto:auth_path=/vclfcn-20/?prefix=obj 2024-02-01T10:31:49.696 DEBUG:boto:Method: GET 2024-02-01T10:31:49.696 DEBUG:boto:Path: /vclfcn-20/?prefix=obj 2024-02-01T10:31:49.697 DEBUG:boto:Data: 2024-02-01T10:31:49.697 DEBUG:boto:Headers: {} 2024-02-01T10:31:49.697 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:31:49.697 DEBUG:boto:Port: 8000 2024-02-01T10:31:49.697 DEBUG:boto:Params: {} 2024-02-01T10:31:49.697 DEBUG:boto:Token: None 2024-02-01T10:31:49.697 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:31:49 GMT /vclfcn-20/ 2024-02-01T10:31:49.697 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9mX1KS2dWfNIEIhGw3onxrrn8KM= 2024-02-01T10:31:49.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9mX1KS2dWfNIEIhGw3onxrrn8KM=', 'Content-Length': '0'} 2024-02-01T10:31:49.702 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007083d88ffdf00547-0065bb7315-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:49.702 DEBUG:boto:b'vclfcn-20obj1000falseobj2024-02-01T10:28:33.438Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:31:49.702 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:31:49.702 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:31:49.702 DEBUG:boto:path=/vclfcn-20/?prefix=obj 2024-02-01T10:31:49.702 DEBUG:boto:auth_path=/vclfcn-20/?prefix=obj 2024-02-01T10:31:49.703 DEBUG:boto:Method: GET 2024-02-01T10:31:49.703 DEBUG:boto:Path: /vclfcn-20/?prefix=obj 2024-02-01T10:31:49.703 DEBUG:boto:Data: 2024-02-01T10:31:49.703 DEBUG:boto:Headers: {} 2024-02-01T10:31:49.703 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:31:49.703 DEBUG:boto:Port: 8000 2024-02-01T10:31:49.703 DEBUG:boto:Params: {} 2024-02-01T10:31:49.703 DEBUG:boto:Token: None 2024-02-01T10:31:49.703 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:31:49 GMT /vclfcn-20/ 2024-02-01T10:31:49.704 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9mX1KS2dWfNIEIhGw3onxrrn8KM= 2024-02-01T10:31:49.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9mX1KS2dWfNIEIhGw3onxrrn8KM=', 'Content-Length': '0'} 2024-02-01T10:31:49.708 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea147eec26e33675-0065bb7315-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:49.708 DEBUG:boto:b'vclfcn-20obj1000falseobj2024-02-01T10:28:33.438Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:31:49.709 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-20 zones={test-zone2, test-zone1} 2024-02-01T10:31:49.709 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-20 zones={test-zone3, test-zone1} 2024-02-01T10:31:49.709 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:31:49.709 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:31:49.709 DEBUG:boto:Method: HEAD 2024-02-01T10:31:49.709 DEBUG:boto:Path: /vclfcn-20/ 2024-02-01T10:31:49.709 DEBUG:boto:Data: 2024-02-01T10:31:49.709 DEBUG:boto:Headers: {} 2024-02-01T10:31:49.709 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:31:49.709 DEBUG:boto:Port: 8001 2024-02-01T10:31:49.710 DEBUG:boto:Params: {} 2024-02-01T10:31:49.710 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-02-01T10:31:49.710 DEBUG:boto:Token: None 2024-02-01T10:31:49.710 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:31:49 GMT /vclfcn-20/ 2024-02-01T10:31:49.710 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:3Wc1rL3mvAJwE1QxaAVz7r4N7zk= 2024-02-01T10:31:49.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:3Wc1rL3mvAJwE1QxaAVz7r4N7zk=', 'Content-Length': '0'} 2024-02-01T10:31:49.720 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a6d094d4f476bcde-0065bb7315-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:31:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:49.720 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:31:49.721 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:31:49.721 DEBUG:boto:Method: HEAD 2024-02-01T10:31:49.721 DEBUG:boto:Path: /vclfcn-20/ 2024-02-01T10:31:49.721 DEBUG:boto:Data: 2024-02-01T10:31:49.721 DEBUG:boto:Headers: {} 2024-02-01T10:31:49.721 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:31:49.721 DEBUG:boto:Port: 8000 2024-02-01T10:31:49.721 DEBUG:boto:Params: {} 2024-02-01T10:31:49.721 DEBUG:boto:Token: None 2024-02-01T10:31:49.721 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:31:49 GMT /vclfcn-20/ 2024-02-01T10:31:49.722 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:3Wc1rL3mvAJwE1QxaAVz7r4N7zk= 2024-02-01T10:31:49.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:3Wc1rL3mvAJwE1QxaAVz7r4N7zk=', 'Content-Length': '0'} 2024-02-01T10:31:49.726 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ea69febe8002e126-0065bb7315-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:31:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:49.726 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:31:49.726 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:31:49.726 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:31:49.726 DEBUG:boto:path=/vclfcn-20/?versions&max-keys=999 2024-02-01T10:31:49.726 DEBUG:boto:auth_path=/vclfcn-20/?versions&max-keys=999 2024-02-01T10:31:49.727 DEBUG:boto:Method: GET 2024-02-01T10:31:49.727 DEBUG:boto:Path: /vclfcn-20/?versions&max-keys=999 2024-02-01T10:31:49.727 DEBUG:boto:Data: 2024-02-01T10:31:49.727 DEBUG:boto:Headers: {} 2024-02-01T10:31:49.727 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:31:49.727 DEBUG:boto:Port: 8001 2024-02-01T10:31:49.727 DEBUG:boto:Params: {} 2024-02-01T10:31:49.727 DEBUG:boto:Token: None 2024-02-01T10:31:49.727 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:31:49 GMT /vclfcn-20/?versions 2024-02-01T10:31:49.727 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9vimAxEjZmQnwBZsg52UKTmNUQs= 2024-02-01T10:31:49.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9vimAxEjZmQnwBZsg52UKTmNUQs=', 'Content-Length': '0'} 2024-02-01T10:31:49.732 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0a5e1eff294e85a-0065bb7315-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:49.732 DEBUG:boto:b'vclfcn-20999falseobjrSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6true2024-02-01T10:28:33.438Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2024-02-01T10:28:01.521Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:31:49.732 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:31:49.733 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:31:49.733 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:31:49.733 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:31:49.733 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:31:49.733 DEBUG:boto:path=/vclfcn-20/?versions&max-keys=999 2024-02-01T10:31:49.733 DEBUG:boto:auth_path=/vclfcn-20/?versions&max-keys=999 2024-02-01T10:31:49.733 DEBUG:boto:Method: GET 2024-02-01T10:31:49.733 DEBUG:boto:Path: /vclfcn-20/?versions&max-keys=999 2024-02-01T10:31:49.733 DEBUG:boto:Data: 2024-02-01T10:31:49.733 DEBUG:boto:Headers: {} 2024-02-01T10:31:49.733 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:31:49.733 DEBUG:boto:Port: 8000 2024-02-01T10:31:49.734 DEBUG:boto:Params: {} 2024-02-01T10:31:49.734 DEBUG:boto:Token: None 2024-02-01T10:31:49.734 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:31:49 GMT /vclfcn-20/?versions 2024-02-01T10:31:49.734 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9vimAxEjZmQnwBZsg52UKTmNUQs= 2024-02-01T10:31:49.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9vimAxEjZmQnwBZsg52UKTmNUQs=', 'Content-Length': '0'} 2024-02-01T10:31:49.740 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000302ed964e6ca3aea-0065bb7315-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:49.784 DEBUG:boto:b'vclfcn-20999falseobjrSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6true2024-02-01T10:28:33.438Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2024-02-01T10:28:01.521Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:31:49.784 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:31:49.784 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:31:49.785 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:31:49.785 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:31:49.785 DEBUG:boto:path=/vclfcn-20/?versions&max-keys=999 2024-02-01T10:31:49.785 DEBUG:boto:auth_path=/vclfcn-20/?versions&max-keys=999 2024-02-01T10:31:49.785 DEBUG:boto:Method: GET 2024-02-01T10:31:49.785 DEBUG:boto:Path: /vclfcn-20/?versions&max-keys=999 2024-02-01T10:31:49.785 DEBUG:boto:Data: 2024-02-01T10:31:49.785 DEBUG:boto:Headers: {} 2024-02-01T10:31:49.785 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:31:49.785 DEBUG:boto:Port: 8001 2024-02-01T10:31:49.785 DEBUG:boto:Params: {} 2024-02-01T10:31:49.786 DEBUG:boto:Token: None 2024-02-01T10:31:49.786 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:31:49 GMT /vclfcn-20/?versions 2024-02-01T10:31:49.786 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9vimAxEjZmQnwBZsg52UKTmNUQs= 2024-02-01T10:31:49.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9vimAxEjZmQnwBZsg52UKTmNUQs=', 'Content-Length': '0'} 2024-02-01T10:31:49.792 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000326c0aa3dcaf405-0065bb7315-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:49.792 DEBUG:boto:b'vclfcn-20999falseobjrSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6true2024-02-01T10:28:33.438Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2024-02-01T10:28:01.521Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:31:49.792 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:31:49.793 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:31:49.793 DEBUG:boto:path=/vclfcn-20/?versions&max-keys=999 2024-02-01T10:31:49.793 DEBUG:boto:auth_path=/vclfcn-20/?versions&max-keys=999 2024-02-01T10:31:49.793 DEBUG:boto:Method: GET 2024-02-01T10:31:49.793 DEBUG:boto:Path: /vclfcn-20/?versions&max-keys=999 2024-02-01T10:31:49.793 DEBUG:boto:Data: 2024-02-01T10:31:49.793 DEBUG:boto:Headers: {} 2024-02-01T10:31:49.793 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:31:49.793 DEBUG:boto:Port: 8000 2024-02-01T10:31:49.793 DEBUG:boto:Params: {} 2024-02-01T10:31:49.793 DEBUG:boto:Token: None 2024-02-01T10:31:49.794 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:31:49 GMT /vclfcn-20/?versions 2024-02-01T10:31:49.794 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9vimAxEjZmQnwBZsg52UKTmNUQs= 2024-02-01T10:31:49.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9vimAxEjZmQnwBZsg52UKTmNUQs=', 'Content-Length': '0'} 2024-02-01T10:31:49.799 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0ae0d9823487ef2-0065bb7315-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:49.799 DEBUG:boto:b'vclfcn-20999falseobjrSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6true2024-02-01T10:28:33.438Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2024-02-01T10:28:01.521Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:31:49.799 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:31:49.799 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:31:49.799 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:31:49.799 DEBUG:boto:path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:49.800 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:49.800 DEBUG:boto:Method: GET 2024-02-01T10:31:49.800 DEBUG:boto:Path: /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:49.800 DEBUG:boto:Data: 2024-02-01T10:31:49.800 DEBUG:boto:Headers: {} 2024-02-01T10:31:49.800 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:31:49.800 DEBUG:boto:Port: 8001 2024-02-01T10:31:49.800 DEBUG:boto:Params: {} 2024-02-01T10:31:49.800 DEBUG:boto:Token: None 2024-02-01T10:31:49.803 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:31:49 GMT /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:49.803 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:uOcyU8SC6g5VJOjqxXFkEiCuTW4= 2024-02-01T10:31:49.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:uOcyU8SC6g5VJOjqxXFkEiCuTW4=', 'Content-Length': '0'} 2024-02-01T10:31:49.807 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:33 GMT'), ('x-amz-version-id', 'rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000aa2cff8ab7c87947-0065bb7315-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:49.807 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:31:49.807 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:31:49.808 DEBUG:boto:path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:49.808 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:49.808 DEBUG:boto:Method: GET 2024-02-01T10:31:49.808 DEBUG:boto:Path: /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:49.808 DEBUG:boto:Data: 2024-02-01T10:31:49.808 DEBUG:boto:Headers: {} 2024-02-01T10:31:49.808 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:31:49.808 DEBUG:boto:Port: 8000 2024-02-01T10:31:49.808 DEBUG:boto:Params: {} 2024-02-01T10:31:49.808 DEBUG:boto:Token: None 2024-02-01T10:31:49.809 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:31:49 GMT /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:49.809 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:uOcyU8SC6g5VJOjqxXFkEiCuTW4= 2024-02-01T10:31:49.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:uOcyU8SC6g5VJOjqxXFkEiCuTW4=', 'Content-Length': '0'} 2024-02-01T10:31:49.811 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:33 GMT'), ('x-amz-version-id', 'rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000de093df769650448-0065bb7315-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:49.811 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:31:49.812 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:31:49.812 DEBUG:boto:path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:49.812 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:49.812 DEBUG:boto:Method: HEAD 2024-02-01T10:31:49.812 DEBUG:boto:Path: /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:49.812 DEBUG:boto:Data: 2024-02-01T10:31:49.812 DEBUG:boto:Headers: {} 2024-02-01T10:31:49.812 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:31:49.812 DEBUG:boto:Port: 8001 2024-02-01T10:31:49.812 DEBUG:boto:Params: {} 2024-02-01T10:31:49.812 DEBUG:boto:Token: None 2024-02-01T10:31:49.813 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:31:49 GMT /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:49.813 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NKi2D68fViA+/b+natPKHKoQxJk= 2024-02-01T10:31:49.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NKi2D68fViA+/b+natPKHKoQxJk=', 'Content-Length': '0'} 2024-02-01T10:31:49.816 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:33 GMT'), ('x-amz-version-id', 'rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000000688300193f2011-0065bb7315-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:49.816 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:31:49.816 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:31:49.816 DEBUG:boto:path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:49.816 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:49.816 DEBUG:boto:Method: HEAD 2024-02-01T10:31:49.816 DEBUG:boto:Path: /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:49.816 DEBUG:boto:Data: 2024-02-01T10:31:49.816 DEBUG:boto:Headers: {} 2024-02-01T10:31:49.817 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:31:49.817 DEBUG:boto:Port: 8000 2024-02-01T10:31:49.817 DEBUG:boto:Params: {} 2024-02-01T10:31:49.817 DEBUG:boto:Token: None 2024-02-01T10:31:49.817 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:31:49 GMT /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:49.817 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NKi2D68fViA+/b+natPKHKoQxJk= 2024-02-01T10:31:49.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NKi2D68fViA+/b+natPKHKoQxJk=', 'Content-Length': '0'} 2024-02-01T10:31:49.820 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:33 GMT'), ('x-amz-version-id', 'rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000001d124870c25e789-0065bb7315-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:49.820 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:31:49.820 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:31:49.820 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:31:49.820 DEBUG:boto:path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:49.820 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:49.821 DEBUG:boto:Method: GET 2024-02-01T10:31:49.821 DEBUG:boto:Path: /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:49.821 DEBUG:boto:Data: 2024-02-01T10:31:49.821 DEBUG:boto:Headers: {} 2024-02-01T10:31:49.821 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:31:49.821 DEBUG:boto:Port: 8001 2024-02-01T10:31:49.821 DEBUG:boto:Params: {} 2024-02-01T10:31:49.821 DEBUG:boto:Token: None 2024-02-01T10:31:49.821 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:31:49 GMT /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:49.821 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:uOcyU8SC6g5VJOjqxXFkEiCuTW4= 2024-02-01T10:31:49.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:uOcyU8SC6g5VJOjqxXFkEiCuTW4=', 'Content-Length': '0'} 2024-02-01T10:31:49.824 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:33 GMT'), ('x-amz-version-id', 'rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ac466035cf962f82-0065bb7315-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:49.824 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:31:49.825 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:31:49.825 DEBUG:boto:path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:49.825 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:49.825 DEBUG:boto:Method: GET 2024-02-01T10:31:49.825 DEBUG:boto:Path: /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:49.825 DEBUG:boto:Data: 2024-02-01T10:31:49.825 DEBUG:boto:Headers: {} 2024-02-01T10:31:49.825 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:31:49.825 DEBUG:boto:Port: 8000 2024-02-01T10:31:49.825 DEBUG:boto:Params: {} 2024-02-01T10:31:49.825 DEBUG:boto:Token: None 2024-02-01T10:31:49.826 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:31:49 GMT /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:49.826 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:uOcyU8SC6g5VJOjqxXFkEiCuTW4= 2024-02-01T10:31:49.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:uOcyU8SC6g5VJOjqxXFkEiCuTW4=', 'Content-Length': '0'} 2024-02-01T10:31:49.829 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:33 GMT'), ('x-amz-version-id', 'rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a0029fae187ae39d-0065bb7315-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:49.829 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:31:49.829 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:31:49.829 DEBUG:boto:path=/vclfcn-20/?prefix=obj 2024-02-01T10:31:49.829 DEBUG:boto:auth_path=/vclfcn-20/?prefix=obj 2024-02-01T10:31:49.829 DEBUG:boto:Method: GET 2024-02-01T10:31:49.830 DEBUG:boto:Path: /vclfcn-20/?prefix=obj 2024-02-01T10:31:49.830 DEBUG:boto:Data: 2024-02-01T10:31:49.830 DEBUG:boto:Headers: {} 2024-02-01T10:31:49.830 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:31:49.830 DEBUG:boto:Port: 8001 2024-02-01T10:31:49.830 DEBUG:boto:Params: {} 2024-02-01T10:31:49.830 DEBUG:boto:Token: None 2024-02-01T10:31:49.830 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:31:49 GMT /vclfcn-20/ 2024-02-01T10:31:49.830 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9mX1KS2dWfNIEIhGw3onxrrn8KM= 2024-02-01T10:31:49.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9mX1KS2dWfNIEIhGw3onxrrn8KM=', 'Content-Length': '0'} 2024-02-01T10:31:49.835 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003465e48fc96a7aab-0065bb7315-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:49.876 DEBUG:boto:b'vclfcn-20obj1000falseobj2024-02-01T10:28:33.438Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:31:49.877 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:31:49.877 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:31:49.877 DEBUG:boto:path=/vclfcn-20/?prefix=obj 2024-02-01T10:31:49.877 DEBUG:boto:auth_path=/vclfcn-20/?prefix=obj 2024-02-01T10:31:49.877 DEBUG:boto:Method: GET 2024-02-01T10:31:49.877 DEBUG:boto:Path: /vclfcn-20/?prefix=obj 2024-02-01T10:31:49.877 DEBUG:boto:Data: 2024-02-01T10:31:49.877 DEBUG:boto:Headers: {} 2024-02-01T10:31:49.877 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:31:49.877 DEBUG:boto:Port: 8000 2024-02-01T10:31:49.878 DEBUG:boto:Params: {} 2024-02-01T10:31:49.878 DEBUG:boto:Token: None 2024-02-01T10:31:49.878 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:31:49 GMT /vclfcn-20/ 2024-02-01T10:31:49.878 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9mX1KS2dWfNIEIhGw3onxrrn8KM= 2024-02-01T10:31:49.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9mX1KS2dWfNIEIhGw3onxrrn8KM=', 'Content-Length': '0'} 2024-02-01T10:31:49.883 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e3f912c25a9cafb-0065bb7315-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:49.883 DEBUG:boto:b'vclfcn-20obj1000falseobj2024-02-01T10:28:33.438Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:31:49.884 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:31:49.884 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:31:49.884 DEBUG:boto:Method: HEAD 2024-02-01T10:31:49.884 DEBUG:boto:Path: /vclfcn-20/obj 2024-02-01T10:31:49.884 DEBUG:boto:Data: 2024-02-01T10:31:49.884 DEBUG:boto:Headers: {} 2024-02-01T10:31:49.884 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:31:49.884 DEBUG:boto:Port: 8001 2024-02-01T10:31:49.884 DEBUG:boto:Params: {} 2024-02-01T10:31:49.885 DEBUG:boto:Token: None 2024-02-01T10:31:49.885 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:31:49 GMT /vclfcn-20/obj 2024-02-01T10:31:49.885 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1LuBuFPpmGlGLTJVFACAiUm7qdY= 2024-02-01T10:31:49.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1LuBuFPpmGlGLTJVFACAiUm7qdY=', 'Content-Length': '0'} 2024-02-01T10:31:49.888 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:33 GMT'), ('x-amz-version-id', 'rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d58c9e39afa0d64d-0065bb7315-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:49.888 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:31:49.889 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:31:49.889 DEBUG:boto:Method: HEAD 2024-02-01T10:31:49.889 DEBUG:boto:Path: /vclfcn-20/obj 2024-02-01T10:31:49.889 DEBUG:boto:Data: 2024-02-01T10:31:49.889 DEBUG:boto:Headers: {} 2024-02-01T10:31:49.889 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:31:49.889 DEBUG:boto:Port: 8000 2024-02-01T10:31:49.889 DEBUG:boto:Params: {} 2024-02-01T10:31:49.889 DEBUG:boto:Token: None 2024-02-01T10:31:49.889 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:31:49 GMT /vclfcn-20/obj 2024-02-01T10:31:49.890 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1LuBuFPpmGlGLTJVFACAiUm7qdY= 2024-02-01T10:31:49.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1LuBuFPpmGlGLTJVFACAiUm7qdY=', 'Content-Length': '0'} 2024-02-01T10:31:49.893 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:33 GMT'), ('x-amz-version-id', 'rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009e5edc65523613d4-0065bb7315-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:49.893 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:31:49.893 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:31:49.894 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:31:49.894 DEBUG:boto:path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:49.894 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:49.894 DEBUG:boto:Method: GET 2024-02-01T10:31:49.894 DEBUG:boto:Path: /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:49.894 DEBUG:boto:Data: 2024-02-01T10:31:49.894 DEBUG:boto:Headers: {} 2024-02-01T10:31:49.894 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:31:49.894 DEBUG:boto:Port: 8001 2024-02-01T10:31:49.894 DEBUG:boto:Params: {} 2024-02-01T10:31:49.894 DEBUG:boto:Token: None 2024-02-01T10:31:49.895 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:31:49 GMT /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:49.895 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:uOcyU8SC6g5VJOjqxXFkEiCuTW4= 2024-02-01T10:31:49.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:uOcyU8SC6g5VJOjqxXFkEiCuTW4=', 'Content-Length': '0'} 2024-02-01T10:31:49.898 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:33 GMT'), ('x-amz-version-id', 'rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000049db9990b91379c-0065bb7315-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:49.898 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:31:49.898 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:31:49.898 DEBUG:boto:path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:49.898 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:49.898 DEBUG:boto:Method: GET 2024-02-01T10:31:49.899 DEBUG:boto:Path: /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:49.899 DEBUG:boto:Data: 2024-02-01T10:31:49.899 DEBUG:boto:Headers: {} 2024-02-01T10:31:49.899 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:31:49.899 DEBUG:boto:Port: 8000 2024-02-01T10:31:49.899 DEBUG:boto:Params: {} 2024-02-01T10:31:49.899 DEBUG:boto:Token: None 2024-02-01T10:31:49.899 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:31:49 GMT /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:49.899 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:uOcyU8SC6g5VJOjqxXFkEiCuTW4= 2024-02-01T10:31:49.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:uOcyU8SC6g5VJOjqxXFkEiCuTW4=', 'Content-Length': '0'} 2024-02-01T10:31:49.902 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:33 GMT'), ('x-amz-version-id', 'rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009b6cb1670c3567bd-0065bb7315-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:49.903 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:31:49.903 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:31:49.903 DEBUG:boto:path=/vclfcn-20/?prefix=obj 2024-02-01T10:31:49.903 DEBUG:boto:auth_path=/vclfcn-20/?prefix=obj 2024-02-01T10:31:49.903 DEBUG:boto:Method: GET 2024-02-01T10:31:49.903 DEBUG:boto:Path: /vclfcn-20/?prefix=obj 2024-02-01T10:31:49.903 DEBUG:boto:Data: 2024-02-01T10:31:49.903 DEBUG:boto:Headers: {} 2024-02-01T10:31:49.903 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:31:49.903 DEBUG:boto:Port: 8001 2024-02-01T10:31:49.903 DEBUG:boto:Params: {} 2024-02-01T10:31:49.904 DEBUG:boto:Token: None 2024-02-01T10:31:49.904 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:31:49 GMT /vclfcn-20/ 2024-02-01T10:31:49.904 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9mX1KS2dWfNIEIhGw3onxrrn8KM= 2024-02-01T10:31:49.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9mX1KS2dWfNIEIhGw3onxrrn8KM=', 'Content-Length': '0'} 2024-02-01T10:31:49.908 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000242403a360461e7b-0065bb7315-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:49.952 DEBUG:boto:b'vclfcn-20obj1000falseobj2024-02-01T10:28:33.438Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:31:49.953 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:31:49.953 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:31:49.953 DEBUG:boto:path=/vclfcn-20/?prefix=obj 2024-02-01T10:31:49.953 DEBUG:boto:auth_path=/vclfcn-20/?prefix=obj 2024-02-01T10:31:49.953 DEBUG:boto:Method: GET 2024-02-01T10:31:49.953 DEBUG:boto:Path: /vclfcn-20/?prefix=obj 2024-02-01T10:31:49.953 DEBUG:boto:Data: 2024-02-01T10:31:49.953 DEBUG:boto:Headers: {} 2024-02-01T10:31:49.953 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:31:49.953 DEBUG:boto:Port: 8000 2024-02-01T10:31:49.954 DEBUG:boto:Params: {} 2024-02-01T10:31:49.954 DEBUG:boto:Token: None 2024-02-01T10:31:49.954 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:31:49 GMT /vclfcn-20/ 2024-02-01T10:31:49.954 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9mX1KS2dWfNIEIhGw3onxrrn8KM= 2024-02-01T10:31:49.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9mX1KS2dWfNIEIhGw3onxrrn8KM=', 'Content-Length': '0'} 2024-02-01T10:31:49.959 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec3341168123186b-0065bb7315-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:49.959 DEBUG:boto:b'vclfcn-20obj1000falseobj2024-02-01T10:28:33.438Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:31:49.960 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:31:49.960 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:31:49.960 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:31:49.960 DEBUG:boto:path=/vclfcn-20/obj?versionId=null 2024-02-01T10:31:49.960 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=null 2024-02-01T10:31:49.960 DEBUG:boto:Method: GET 2024-02-01T10:31:49.960 DEBUG:boto:Path: /vclfcn-20/obj?versionId=null 2024-02-01T10:31:49.960 DEBUG:boto:Data: 2024-02-01T10:31:49.960 DEBUG:boto:Headers: {} 2024-02-01T10:31:49.960 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:31:49.960 DEBUG:boto:Port: 8001 2024-02-01T10:31:49.961 DEBUG:boto:Params: {} 2024-02-01T10:31:49.961 DEBUG:boto:Token: None 2024-02-01T10:31:49.961 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:31:49 GMT /vclfcn-20/obj?versionId=null 2024-02-01T10:31:49.961 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YYnPDEgZKFy0QWkVF+HQnlK6oAA= 2024-02-01T10:31:49.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YYnPDEgZKFy0QWkVF+HQnlK6oAA=', 'Content-Length': '0'} 2024-02-01T10:31:49.963 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c86b8b0df741ba1d-0065bb7315-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:49.964 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:31:49.964 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:31:49.964 DEBUG:boto:path=/vclfcn-20/obj?versionId=null 2024-02-01T10:31:49.964 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=null 2024-02-01T10:31:49.964 DEBUG:boto:Method: GET 2024-02-01T10:31:49.964 DEBUG:boto:Path: /vclfcn-20/obj?versionId=null 2024-02-01T10:31:49.964 DEBUG:boto:Data: 2024-02-01T10:31:49.964 DEBUG:boto:Headers: {} 2024-02-01T10:31:49.965 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:31:49.965 DEBUG:boto:Port: 8000 2024-02-01T10:31:49.965 DEBUG:boto:Params: {} 2024-02-01T10:31:49.965 DEBUG:boto:Token: None 2024-02-01T10:31:49.965 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:31:49 GMT /vclfcn-20/obj?versionId=null 2024-02-01T10:31:49.965 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YYnPDEgZKFy0QWkVF+HQnlK6oAA= 2024-02-01T10:31:49.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YYnPDEgZKFy0QWkVF+HQnlK6oAA=', 'Content-Length': '0'} 2024-02-01T10:31:49.968 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006ac8b37e11df1232-0065bb7315-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:49.968 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:31:49.968 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:31:49.968 DEBUG:boto:path=/vclfcn-20/obj?versionId=null 2024-02-01T10:31:49.968 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=null 2024-02-01T10:31:49.968 DEBUG:boto:Method: HEAD 2024-02-01T10:31:49.969 DEBUG:boto:Path: /vclfcn-20/obj?versionId=null 2024-02-01T10:31:49.969 DEBUG:boto:Data: 2024-02-01T10:31:49.969 DEBUG:boto:Headers: {} 2024-02-01T10:31:49.969 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:31:49.969 DEBUG:boto:Port: 8001 2024-02-01T10:31:49.969 DEBUG:boto:Params: {} 2024-02-01T10:31:49.969 DEBUG:boto:Token: None 2024-02-01T10:31:49.969 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:31:49 GMT /vclfcn-20/obj?versionId=null 2024-02-01T10:31:49.969 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:blD1l+cS4TFMKkSa2rvXbp8Yai4= 2024-02-01T10:31:49.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:blD1l+cS4TFMKkSa2rvXbp8Yai4=', 'Content-Length': '0'} 2024-02-01T10:31:49.972 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000056388995dac0e665-0065bb7315-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:49.972 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:31:49.972 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:31:49.972 DEBUG:boto:path=/vclfcn-20/obj?versionId=null 2024-02-01T10:31:49.972 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=null 2024-02-01T10:31:49.973 DEBUG:boto:Method: HEAD 2024-02-01T10:31:49.973 DEBUG:boto:Path: /vclfcn-20/obj?versionId=null 2024-02-01T10:31:49.973 DEBUG:boto:Data: 2024-02-01T10:31:49.973 DEBUG:boto:Headers: {} 2024-02-01T10:31:49.973 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:31:49.973 DEBUG:boto:Port: 8000 2024-02-01T10:31:49.973 DEBUG:boto:Params: {} 2024-02-01T10:31:49.973 DEBUG:boto:Token: None 2024-02-01T10:31:49.973 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:31:49 GMT /vclfcn-20/obj?versionId=null 2024-02-01T10:31:49.974 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:blD1l+cS4TFMKkSa2rvXbp8Yai4= 2024-02-01T10:31:49.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:blD1l+cS4TFMKkSa2rvXbp8Yai4=', 'Content-Length': '0'} 2024-02-01T10:31:49.976 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d904562d735da0be-0065bb7315-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:49.976 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:31:49.977 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:31:49.977 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:31:49.977 DEBUG:boto:path=/vclfcn-20/obj?versionId=null 2024-02-01T10:31:49.977 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=null 2024-02-01T10:31:49.977 DEBUG:boto:Method: GET 2024-02-01T10:31:49.977 DEBUG:boto:Path: /vclfcn-20/obj?versionId=null 2024-02-01T10:31:49.977 DEBUG:boto:Data: 2024-02-01T10:31:49.977 DEBUG:boto:Headers: {} 2024-02-01T10:31:49.977 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:31:49.977 DEBUG:boto:Port: 8001 2024-02-01T10:31:49.977 DEBUG:boto:Params: {} 2024-02-01T10:31:49.978 DEBUG:boto:Token: None 2024-02-01T10:31:49.978 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:31:49 GMT /vclfcn-20/obj?versionId=null 2024-02-01T10:31:49.978 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YYnPDEgZKFy0QWkVF+HQnlK6oAA= 2024-02-01T10:31:49.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YYnPDEgZKFy0QWkVF+HQnlK6oAA=', 'Content-Length': '0'} 2024-02-01T10:31:49.981 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008edddcf8d437a910-0065bb7315-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:49.981 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:31:49.981 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:31:49.981 DEBUG:boto:path=/vclfcn-20/obj?versionId=null 2024-02-01T10:31:49.981 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=null 2024-02-01T10:31:49.981 DEBUG:boto:Method: GET 2024-02-01T10:31:49.981 DEBUG:boto:Path: /vclfcn-20/obj?versionId=null 2024-02-01T10:31:49.981 DEBUG:boto:Data: 2024-02-01T10:31:49.981 DEBUG:boto:Headers: {} 2024-02-01T10:31:49.982 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:31:49.982 DEBUG:boto:Port: 8000 2024-02-01T10:31:49.982 DEBUG:boto:Params: {} 2024-02-01T10:31:49.982 DEBUG:boto:Token: None 2024-02-01T10:31:49.982 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:31:49 GMT /vclfcn-20/obj?versionId=null 2024-02-01T10:31:49.982 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YYnPDEgZKFy0QWkVF+HQnlK6oAA= 2024-02-01T10:31:49.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YYnPDEgZKFy0QWkVF+HQnlK6oAA=', 'Content-Length': '0'} 2024-02-01T10:31:49.985 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000633680b2119df776-0065bb7315-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:49.985 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:31:49.985 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:31:49.985 DEBUG:boto:path=/vclfcn-20/?prefix=obj 2024-02-01T10:31:49.985 DEBUG:boto:auth_path=/vclfcn-20/?prefix=obj 2024-02-01T10:31:49.985 DEBUG:boto:Method: GET 2024-02-01T10:31:49.985 DEBUG:boto:Path: /vclfcn-20/?prefix=obj 2024-02-01T10:31:49.985 DEBUG:boto:Data: 2024-02-01T10:31:49.986 DEBUG:boto:Headers: {} 2024-02-01T10:31:49.986 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:31:49.986 DEBUG:boto:Port: 8001 2024-02-01T10:31:49.986 DEBUG:boto:Params: {} 2024-02-01T10:31:49.986 DEBUG:boto:Token: None 2024-02-01T10:31:49.986 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:31:49 GMT /vclfcn-20/ 2024-02-01T10:31:49.986 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9mX1KS2dWfNIEIhGw3onxrrn8KM= 2024-02-01T10:31:49.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9mX1KS2dWfNIEIhGw3onxrrn8KM=', 'Content-Length': '0'} 2024-02-01T10:31:49.991 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f74d7ada2262d47-0065bb7315-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:50.032 DEBUG:boto:b'vclfcn-20obj1000falseobj2024-02-01T10:28:33.438Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:31:50.033 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:31:50.033 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:31:50.033 DEBUG:boto:path=/vclfcn-20/?prefix=obj 2024-02-01T10:31:50.034 DEBUG:boto:auth_path=/vclfcn-20/?prefix=obj 2024-02-01T10:31:50.034 DEBUG:boto:Method: GET 2024-02-01T10:31:50.034 DEBUG:boto:Path: /vclfcn-20/?prefix=obj 2024-02-01T10:31:50.034 DEBUG:boto:Data: 2024-02-01T10:31:50.034 DEBUG:boto:Headers: {} 2024-02-01T10:31:50.034 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:31:50.034 DEBUG:boto:Port: 8000 2024-02-01T10:31:50.034 DEBUG:boto:Params: {} 2024-02-01T10:31:50.034 DEBUG:boto:Token: None 2024-02-01T10:31:50.035 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:31:50 GMT /vclfcn-20/ 2024-02-01T10:31:50.035 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:AwovwTRPxx0Vc6/GGzl9hWNtDS0= 2024-02-01T10:31:50.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:AwovwTRPxx0Vc6/GGzl9hWNtDS0=', 'Content-Length': '0'} 2024-02-01T10:31:50.040 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000400d328444ab6e73-0065bb7316-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:50.040 DEBUG:boto:b'vclfcn-20obj1000falseobj2024-02-01T10:28:33.438Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:31:50.040 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:31:50.040 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:31:50.040 DEBUG:boto:Method: HEAD 2024-02-01T10:31:50.041 DEBUG:boto:Path: /vclfcn-20/obj 2024-02-01T10:31:50.041 DEBUG:boto:Data: 2024-02-01T10:31:50.041 DEBUG:boto:Headers: {} 2024-02-01T10:31:50.041 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:31:50.041 DEBUG:boto:Port: 8001 2024-02-01T10:31:50.041 DEBUG:boto:Params: {} 2024-02-01T10:31:50.041 DEBUG:boto:Token: None 2024-02-01T10:31:50.041 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:31:50 GMT /vclfcn-20/obj 2024-02-01T10:31:50.041 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:VgWorJrYumhgyWC/F1h615nKGSk= 2024-02-01T10:31:50.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:VgWorJrYumhgyWC/F1h615nKGSk=', 'Content-Length': '0'} 2024-02-01T10:31:50.045 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:33 GMT'), ('x-amz-version-id', 'rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000120f6c2c1cd01ce4-0065bb7316-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:50.045 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:31:50.045 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:31:50.045 DEBUG:boto:Method: HEAD 2024-02-01T10:31:50.045 DEBUG:boto:Path: /vclfcn-20/obj 2024-02-01T10:31:50.046 DEBUG:boto:Data: 2024-02-01T10:31:50.046 DEBUG:boto:Headers: {} 2024-02-01T10:31:50.046 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:31:50.046 DEBUG:boto:Port: 8000 2024-02-01T10:31:50.046 DEBUG:boto:Params: {} 2024-02-01T10:31:50.046 DEBUG:boto:Token: None 2024-02-01T10:31:50.046 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:31:50 GMT /vclfcn-20/obj 2024-02-01T10:31:50.046 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:VgWorJrYumhgyWC/F1h615nKGSk= 2024-02-01T10:31:50.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:VgWorJrYumhgyWC/F1h615nKGSk=', 'Content-Length': '0'} 2024-02-01T10:31:50.050 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:33 GMT'), ('x-amz-version-id', 'rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000056f84c1f32bb7f69-0065bb7316-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:50.050 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:31:50.050 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:31:50.050 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:31:50.050 DEBUG:boto:path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:50.050 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:50.050 DEBUG:boto:Method: GET 2024-02-01T10:31:50.050 DEBUG:boto:Path: /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:50.051 DEBUG:boto:Data: 2024-02-01T10:31:50.051 DEBUG:boto:Headers: {} 2024-02-01T10:31:50.051 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:31:50.051 DEBUG:boto:Port: 8001 2024-02-01T10:31:50.051 DEBUG:boto:Params: {} 2024-02-01T10:31:50.051 DEBUG:boto:Token: None 2024-02-01T10:31:50.051 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:31:50 GMT /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:50.051 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GURvpYML+oEUrQuvwLV39+zEgjE= 2024-02-01T10:31:50.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GURvpYML+oEUrQuvwLV39+zEgjE=', 'Content-Length': '0'} 2024-02-01T10:31:50.054 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:33 GMT'), ('x-amz-version-id', 'rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bc1e7a55796beb5d-0065bb7316-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:50.055 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:31:50.055 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:31:50.055 DEBUG:boto:path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:50.055 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:50.055 DEBUG:boto:Method: GET 2024-02-01T10:31:50.055 DEBUG:boto:Path: /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:50.055 DEBUG:boto:Data: 2024-02-01T10:31:50.055 DEBUG:boto:Headers: {} 2024-02-01T10:31:50.055 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:31:50.055 DEBUG:boto:Port: 8000 2024-02-01T10:31:50.055 DEBUG:boto:Params: {} 2024-02-01T10:31:50.056 DEBUG:boto:Token: None 2024-02-01T10:31:50.056 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:31:50 GMT /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:50.056 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GURvpYML+oEUrQuvwLV39+zEgjE= 2024-02-01T10:31:50.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GURvpYML+oEUrQuvwLV39+zEgjE=', 'Content-Length': '0'} 2024-02-01T10:31:50.058 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:33 GMT'), ('x-amz-version-id', 'rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000040248051b8381472-0065bb7316-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:50.059 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:31:50.059 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:31:50.059 DEBUG:boto:path=/vclfcn-20/?prefix=obj 2024-02-01T10:31:50.059 DEBUG:boto:auth_path=/vclfcn-20/?prefix=obj 2024-02-01T10:31:50.059 DEBUG:boto:Method: GET 2024-02-01T10:31:50.059 DEBUG:boto:Path: /vclfcn-20/?prefix=obj 2024-02-01T10:31:50.059 DEBUG:boto:Data: 2024-02-01T10:31:50.059 DEBUG:boto:Headers: {} 2024-02-01T10:31:50.059 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:31:50.060 DEBUG:boto:Port: 8001 2024-02-01T10:31:50.060 DEBUG:boto:Params: {} 2024-02-01T10:31:50.060 DEBUG:boto:Token: None 2024-02-01T10:31:50.060 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:31:50 GMT /vclfcn-20/ 2024-02-01T10:31:50.060 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:AwovwTRPxx0Vc6/GGzl9hWNtDS0= 2024-02-01T10:31:50.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:AwovwTRPxx0Vc6/GGzl9hWNtDS0=', 'Content-Length': '0'} 2024-02-01T10:31:50.064 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000803ea63da12fd55d-0065bb7316-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:50.108 DEBUG:boto:b'vclfcn-20obj1000falseobj2024-02-01T10:28:33.438Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:31:50.108 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:31:50.109 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:31:50.109 DEBUG:boto:path=/vclfcn-20/?prefix=obj 2024-02-01T10:31:50.109 DEBUG:boto:auth_path=/vclfcn-20/?prefix=obj 2024-02-01T10:31:50.109 DEBUG:boto:Method: GET 2024-02-01T10:31:50.109 DEBUG:boto:Path: /vclfcn-20/?prefix=obj 2024-02-01T10:31:50.109 DEBUG:boto:Data: 2024-02-01T10:31:50.109 DEBUG:boto:Headers: {} 2024-02-01T10:31:50.109 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:31:50.109 DEBUG:boto:Port: 8000 2024-02-01T10:31:50.109 DEBUG:boto:Params: {} 2024-02-01T10:31:50.110 DEBUG:boto:Token: None 2024-02-01T10:31:50.110 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:31:50 GMT /vclfcn-20/ 2024-02-01T10:31:50.110 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:AwovwTRPxx0Vc6/GGzl9hWNtDS0= 2024-02-01T10:31:50.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:AwovwTRPxx0Vc6/GGzl9hWNtDS0=', 'Content-Length': '0'} 2024-02-01T10:31:50.115 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3b6c2e0f8914429-0065bb7316-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:50.115 DEBUG:boto:b'vclfcn-20obj1000falseobj2024-02-01T10:28:33.438Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:31:50.115 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-20 zones={test-zone3, test-zone1} 2024-02-01T10:31:50.115 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-20 zones={test-zone3, test-zone2} 2024-02-01T10:31:50.116 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:31:50.116 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:31:50.116 DEBUG:boto:Method: HEAD 2024-02-01T10:31:50.116 DEBUG:boto:Path: /vclfcn-20/ 2024-02-01T10:31:50.116 DEBUG:boto:Data: 2024-02-01T10:31:50.116 DEBUG:boto:Headers: {} 2024-02-01T10:31:50.116 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:31:50.116 DEBUG:boto:Port: 8001 2024-02-01T10:31:50.116 DEBUG:boto:Params: {} 2024-02-01T10:31:50.116 DEBUG:boto:Token: None 2024-02-01T10:31:50.117 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:31:50 GMT /vclfcn-20/ 2024-02-01T10:31:50.117 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:D4mmgj6QoztqbgvqYfRhUltDUBM= 2024-02-01T10:31:50.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:D4mmgj6QoztqbgvqYfRhUltDUBM=', 'Content-Length': '0'} 2024-02-01T10:31:50.121 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b7c520df7ed0a8cb-0065bb7316-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:50.121 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:31:50.121 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:31:50.121 DEBUG:boto:Method: HEAD 2024-02-01T10:31:50.121 DEBUG:boto:Path: /vclfcn-20/ 2024-02-01T10:31:50.121 DEBUG:boto:Data: 2024-02-01T10:31:50.121 DEBUG:boto:Headers: {} 2024-02-01T10:31:50.121 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:31:50.122 DEBUG:boto:Port: 8000 2024-02-01T10:31:50.122 DEBUG:boto:Params: {} 2024-02-01T10:31:50.122 DEBUG:boto:Token: None 2024-02-01T10:31:50.122 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:31:50 GMT /vclfcn-20/ 2024-02-01T10:31:50.122 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:D4mmgj6QoztqbgvqYfRhUltDUBM= 2024-02-01T10:31:50.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:D4mmgj6QoztqbgvqYfRhUltDUBM=', 'Content-Length': '0'} 2024-02-01T10:31:50.126 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a2cdf578742810de-0065bb7316-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:50.126 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:31:50.126 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:31:50.126 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:31:50.126 DEBUG:boto:path=/vclfcn-20/?versions&max-keys=999 2024-02-01T10:31:50.126 DEBUG:boto:auth_path=/vclfcn-20/?versions&max-keys=999 2024-02-01T10:31:50.127 DEBUG:boto:Method: GET 2024-02-01T10:31:50.127 DEBUG:boto:Path: /vclfcn-20/?versions&max-keys=999 2024-02-01T10:31:50.127 DEBUG:boto:Data: 2024-02-01T10:31:50.127 DEBUG:boto:Headers: {} 2024-02-01T10:31:50.127 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:31:50.127 DEBUG:boto:Port: 8001 2024-02-01T10:31:50.127 DEBUG:boto:Params: {} 2024-02-01T10:31:50.127 DEBUG:boto:Token: None 2024-02-01T10:31:50.127 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:31:50 GMT /vclfcn-20/?versions 2024-02-01T10:31:50.127 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cqOxxdej9RhofCLKESTC/HuafSg= 2024-02-01T10:31:50.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cqOxxdej9RhofCLKESTC/HuafSg=', 'Content-Length': '0'} 2024-02-01T10:31:50.132 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d99b75a6f427e6f8-0065bb7316-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:50.132 DEBUG:boto:b'vclfcn-20999falseobjrSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6true2024-02-01T10:28:33.438Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2024-02-01T10:28:01.521Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:31:50.132 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:31:50.132 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:31:50.133 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:31:50.133 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:31:50.133 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:31:50.133 DEBUG:boto:path=/vclfcn-20/?versions&max-keys=999 2024-02-01T10:31:50.133 DEBUG:boto:auth_path=/vclfcn-20/?versions&max-keys=999 2024-02-01T10:31:50.133 DEBUG:boto:Method: GET 2024-02-01T10:31:50.133 DEBUG:boto:Path: /vclfcn-20/?versions&max-keys=999 2024-02-01T10:31:50.133 DEBUG:boto:Data: 2024-02-01T10:31:50.133 DEBUG:boto:Headers: {} 2024-02-01T10:31:50.133 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:31:50.134 DEBUG:boto:Port: 8000 2024-02-01T10:31:50.134 DEBUG:boto:Params: {} 2024-02-01T10:31:50.134 DEBUG:boto:Token: None 2024-02-01T10:31:50.134 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:31:50 GMT /vclfcn-20/?versions 2024-02-01T10:31:50.134 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cqOxxdej9RhofCLKESTC/HuafSg= 2024-02-01T10:31:50.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cqOxxdej9RhofCLKESTC/HuafSg=', 'Content-Length': '0'} 2024-02-01T10:31:50.138 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b19111100face1f3-0065bb7316-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:50.139 DEBUG:boto:b'vclfcn-20999falseobjrSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6true2024-02-01T10:28:33.438Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2024-02-01T10:28:01.521Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:31:50.139 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:31:50.139 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:31:50.139 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:31:50.139 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:31:50.139 DEBUG:boto:path=/vclfcn-20/?versions&max-keys=999 2024-02-01T10:31:50.140 DEBUG:boto:auth_path=/vclfcn-20/?versions&max-keys=999 2024-02-01T10:31:50.140 DEBUG:boto:Method: GET 2024-02-01T10:31:50.140 DEBUG:boto:Path: /vclfcn-20/?versions&max-keys=999 2024-02-01T10:31:50.140 DEBUG:boto:Data: 2024-02-01T10:31:50.140 DEBUG:boto:Headers: {} 2024-02-01T10:31:50.140 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:31:50.140 DEBUG:boto:Port: 8001 2024-02-01T10:31:50.140 DEBUG:boto:Params: {} 2024-02-01T10:31:50.140 DEBUG:boto:Token: None 2024-02-01T10:31:50.140 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:31:50 GMT /vclfcn-20/?versions 2024-02-01T10:31:50.141 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cqOxxdej9RhofCLKESTC/HuafSg= 2024-02-01T10:31:50.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cqOxxdej9RhofCLKESTC/HuafSg=', 'Content-Length': '0'} 2024-02-01T10:31:50.145 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000442aab4409fa9a2c-0065bb7316-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:50.188 DEBUG:boto:b'vclfcn-20999falseobjrSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6true2024-02-01T10:28:33.438Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2024-02-01T10:28:01.521Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:31:50.189 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:31:50.189 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:31:50.189 DEBUG:boto:path=/vclfcn-20/?versions&max-keys=999 2024-02-01T10:31:50.189 DEBUG:boto:auth_path=/vclfcn-20/?versions&max-keys=999 2024-02-01T10:31:50.189 DEBUG:boto:Method: GET 2024-02-01T10:31:50.189 DEBUG:boto:Path: /vclfcn-20/?versions&max-keys=999 2024-02-01T10:31:50.189 DEBUG:boto:Data: 2024-02-01T10:31:50.189 DEBUG:boto:Headers: {} 2024-02-01T10:31:50.189 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:31:50.189 DEBUG:boto:Port: 8000 2024-02-01T10:31:50.189 DEBUG:boto:Params: {} 2024-02-01T10:31:50.190 DEBUG:boto:Token: None 2024-02-01T10:31:50.190 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:31:50 GMT /vclfcn-20/?versions 2024-02-01T10:31:50.190 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cqOxxdej9RhofCLKESTC/HuafSg= 2024-02-01T10:31:50.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cqOxxdej9RhofCLKESTC/HuafSg=', 'Content-Length': '0'} 2024-02-01T10:31:50.194 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a905d5ded836a40-0065bb7316-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:50.194 DEBUG:boto:b'vclfcn-20999falseobjrSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6true2024-02-01T10:28:33.438Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2024-02-01T10:28:01.521Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:31:50.195 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:31:50.195 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:31:50.195 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:31:50.195 DEBUG:boto:path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:50.195 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:50.195 DEBUG:boto:Method: GET 2024-02-01T10:31:50.195 DEBUG:boto:Path: /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:50.196 DEBUG:boto:Data: 2024-02-01T10:31:50.196 DEBUG:boto:Headers: {} 2024-02-01T10:31:50.196 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:31:50.196 DEBUG:boto:Port: 8001 2024-02-01T10:31:50.196 DEBUG:boto:Params: {} 2024-02-01T10:31:50.196 DEBUG:boto:Token: None 2024-02-01T10:31:50.196 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:31:50 GMT /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:50.196 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GURvpYML+oEUrQuvwLV39+zEgjE= 2024-02-01T10:31:50.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GURvpYML+oEUrQuvwLV39+zEgjE=', 'Content-Length': '0'} 2024-02-01T10:31:50.199 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:33 GMT'), ('x-amz-version-id', 'rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006d0a1fec3344b64e-0065bb7316-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:50.199 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:31:50.199 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:31:50.200 DEBUG:boto:path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:50.200 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:50.200 DEBUG:boto:Method: GET 2024-02-01T10:31:50.200 DEBUG:boto:Path: /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:50.200 DEBUG:boto:Data: 2024-02-01T10:31:50.200 DEBUG:boto:Headers: {} 2024-02-01T10:31:50.200 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:31:50.200 DEBUG:boto:Port: 8000 2024-02-01T10:31:50.200 DEBUG:boto:Params: {} 2024-02-01T10:31:50.200 DEBUG:boto:Token: None 2024-02-01T10:31:50.200 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:31:50 GMT /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:50.201 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GURvpYML+oEUrQuvwLV39+zEgjE= 2024-02-01T10:31:50.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GURvpYML+oEUrQuvwLV39+zEgjE=', 'Content-Length': '0'} 2024-02-01T10:31:50.203 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:33 GMT'), ('x-amz-version-id', 'rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e88de86b56f7b09a-0065bb7316-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:50.204 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:31:50.204 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:31:50.204 DEBUG:boto:path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:50.204 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:50.204 DEBUG:boto:Method: HEAD 2024-02-01T10:31:50.204 DEBUG:boto:Path: /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:50.204 DEBUG:boto:Data: 2024-02-01T10:31:50.204 DEBUG:boto:Headers: {} 2024-02-01T10:31:50.204 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:31:50.204 DEBUG:boto:Port: 8001 2024-02-01T10:31:50.205 DEBUG:boto:Params: {} 2024-02-01T10:31:50.205 DEBUG:boto:Token: None 2024-02-01T10:31:50.205 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:31:50 GMT /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:50.205 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vUbAmK6R/bPHGiKf4Dkc3KOA8Y0= 2024-02-01T10:31:50.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vUbAmK6R/bPHGiKf4Dkc3KOA8Y0=', 'Content-Length': '0'} 2024-02-01T10:31:50.207 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:33 GMT'), ('x-amz-version-id', 'rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008e7fe5c8a390632a-0065bb7316-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:50.208 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:31:50.208 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:31:50.208 DEBUG:boto:path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:50.208 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:50.211 DEBUG:boto:Method: HEAD 2024-02-01T10:31:50.211 DEBUG:boto:Path: /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:50.211 DEBUG:boto:Data: 2024-02-01T10:31:50.211 DEBUG:boto:Headers: {} 2024-02-01T10:31:50.211 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:31:50.211 DEBUG:boto:Port: 8000 2024-02-01T10:31:50.211 DEBUG:boto:Params: {} 2024-02-01T10:31:50.212 DEBUG:boto:Token: None 2024-02-01T10:31:50.212 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:31:50 GMT /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:50.212 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vUbAmK6R/bPHGiKf4Dkc3KOA8Y0= 2024-02-01T10:31:50.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vUbAmK6R/bPHGiKf4Dkc3KOA8Y0=', 'Content-Length': '0'} 2024-02-01T10:31:50.215 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:33 GMT'), ('x-amz-version-id', 'rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000977e16c0d019bdb9-0065bb7316-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:50.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:31:50.215 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:31:50.215 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:31:50.215 DEBUG:boto:path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:50.215 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:50.215 DEBUG:boto:Method: GET 2024-02-01T10:31:50.216 DEBUG:boto:Path: /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:50.216 DEBUG:boto:Data: 2024-02-01T10:31:50.216 DEBUG:boto:Headers: {} 2024-02-01T10:31:50.216 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:31:50.216 DEBUG:boto:Port: 8001 2024-02-01T10:31:50.216 DEBUG:boto:Params: {} 2024-02-01T10:31:50.216 DEBUG:boto:Token: None 2024-02-01T10:31:50.216 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:31:50 GMT /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:50.216 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GURvpYML+oEUrQuvwLV39+zEgjE= 2024-02-01T10:31:50.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GURvpYML+oEUrQuvwLV39+zEgjE=', 'Content-Length': '0'} 2024-02-01T10:31:50.219 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:33 GMT'), ('x-amz-version-id', 'rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f184be623dcf39d8-0065bb7316-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:50.220 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:31:50.220 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:31:50.220 DEBUG:boto:path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:50.220 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:50.220 DEBUG:boto:Method: GET 2024-02-01T10:31:50.220 DEBUG:boto:Path: /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:50.220 DEBUG:boto:Data: 2024-02-01T10:31:50.220 DEBUG:boto:Headers: {} 2024-02-01T10:31:50.220 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:31:50.220 DEBUG:boto:Port: 8000 2024-02-01T10:31:50.220 DEBUG:boto:Params: {} 2024-02-01T10:31:50.221 DEBUG:boto:Token: None 2024-02-01T10:31:50.221 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:31:50 GMT /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:50.221 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GURvpYML+oEUrQuvwLV39+zEgjE= 2024-02-01T10:31:50.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GURvpYML+oEUrQuvwLV39+zEgjE=', 'Content-Length': '0'} 2024-02-01T10:31:50.224 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:33 GMT'), ('x-amz-version-id', 'rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c751432432dc3157-0065bb7316-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:50.224 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:31:50.224 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:31:50.224 DEBUG:boto:path=/vclfcn-20/?prefix=obj 2024-02-01T10:31:50.224 DEBUG:boto:auth_path=/vclfcn-20/?prefix=obj 2024-02-01T10:31:50.224 DEBUG:boto:Method: GET 2024-02-01T10:31:50.224 DEBUG:boto:Path: /vclfcn-20/?prefix=obj 2024-02-01T10:31:50.225 DEBUG:boto:Data: 2024-02-01T10:31:50.225 DEBUG:boto:Headers: {} 2024-02-01T10:31:50.225 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:31:50.225 DEBUG:boto:Port: 8001 2024-02-01T10:31:50.225 DEBUG:boto:Params: {} 2024-02-01T10:31:50.225 DEBUG:boto:Token: None 2024-02-01T10:31:50.225 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:31:50 GMT /vclfcn-20/ 2024-02-01T10:31:50.225 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:AwovwTRPxx0Vc6/GGzl9hWNtDS0= 2024-02-01T10:31:50.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:AwovwTRPxx0Vc6/GGzl9hWNtDS0=', 'Content-Length': '0'} 2024-02-01T10:31:50.229 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ed298ca4d1183f4-0065bb7316-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:50.230 DEBUG:boto:b'vclfcn-20obj1000falseobj2024-02-01T10:28:33.438Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:31:50.230 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:31:50.230 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:31:50.230 DEBUG:boto:path=/vclfcn-20/?prefix=obj 2024-02-01T10:31:50.231 DEBUG:boto:auth_path=/vclfcn-20/?prefix=obj 2024-02-01T10:31:50.231 DEBUG:boto:Method: GET 2024-02-01T10:31:50.231 DEBUG:boto:Path: /vclfcn-20/?prefix=obj 2024-02-01T10:31:50.231 DEBUG:boto:Data: 2024-02-01T10:31:50.231 DEBUG:boto:Headers: {} 2024-02-01T10:31:50.231 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:31:50.231 DEBUG:boto:Port: 8000 2024-02-01T10:31:50.231 DEBUG:boto:Params: {} 2024-02-01T10:31:50.232 DEBUG:boto:Token: None 2024-02-01T10:31:50.232 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:31:50 GMT /vclfcn-20/ 2024-02-01T10:31:50.232 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:AwovwTRPxx0Vc6/GGzl9hWNtDS0= 2024-02-01T10:31:50.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:AwovwTRPxx0Vc6/GGzl9hWNtDS0=', 'Content-Length': '0'} 2024-02-01T10:31:50.236 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7f91805854e7b93-0065bb7316-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:50.280 DEBUG:boto:b'vclfcn-20obj1000falseobj2024-02-01T10:28:33.438Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:31:50.280 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:31:50.280 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:31:50.280 DEBUG:boto:Method: HEAD 2024-02-01T10:31:50.281 DEBUG:boto:Path: /vclfcn-20/obj 2024-02-01T10:31:50.281 DEBUG:boto:Data: 2024-02-01T10:31:50.281 DEBUG:boto:Headers: {} 2024-02-01T10:31:50.281 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:31:50.281 DEBUG:boto:Port: 8001 2024-02-01T10:31:50.281 DEBUG:boto:Params: {} 2024-02-01T10:31:50.281 DEBUG:boto:Token: None 2024-02-01T10:31:50.281 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:31:50 GMT /vclfcn-20/obj 2024-02-01T10:31:50.281 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:VgWorJrYumhgyWC/F1h615nKGSk= 2024-02-01T10:31:50.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:VgWorJrYumhgyWC/F1h615nKGSk=', 'Content-Length': '0'} 2024-02-01T10:31:50.285 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:33 GMT'), ('x-amz-version-id', 'rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000009152507720fd5a2-0065bb7316-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:50.286 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:31:50.286 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:31:50.286 DEBUG:boto:Method: HEAD 2024-02-01T10:31:50.286 DEBUG:boto:Path: /vclfcn-20/obj 2024-02-01T10:31:50.286 DEBUG:boto:Data: 2024-02-01T10:31:50.286 DEBUG:boto:Headers: {} 2024-02-01T10:31:50.286 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:31:50.286 DEBUG:boto:Port: 8000 2024-02-01T10:31:50.286 DEBUG:boto:Params: {} 2024-02-01T10:31:50.287 DEBUG:boto:Token: None 2024-02-01T10:31:50.287 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:31:50 GMT /vclfcn-20/obj 2024-02-01T10:31:50.287 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:VgWorJrYumhgyWC/F1h615nKGSk= 2024-02-01T10:31:50.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:VgWorJrYumhgyWC/F1h615nKGSk=', 'Content-Length': '0'} 2024-02-01T10:31:50.291 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:33 GMT'), ('x-amz-version-id', 'rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000737aa221226dce1-0065bb7316-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:50.291 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:31:50.291 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:31:50.291 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:31:50.291 DEBUG:boto:path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:50.291 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:50.291 DEBUG:boto:Method: GET 2024-02-01T10:31:50.291 DEBUG:boto:Path: /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:50.292 DEBUG:boto:Data: 2024-02-01T10:31:50.292 DEBUG:boto:Headers: {} 2024-02-01T10:31:50.292 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:31:50.292 DEBUG:boto:Port: 8001 2024-02-01T10:31:50.292 DEBUG:boto:Params: {} 2024-02-01T10:31:50.292 DEBUG:boto:Token: None 2024-02-01T10:31:50.292 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:31:50 GMT /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:50.292 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GURvpYML+oEUrQuvwLV39+zEgjE= 2024-02-01T10:31:50.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GURvpYML+oEUrQuvwLV39+zEgjE=', 'Content-Length': '0'} 2024-02-01T10:31:50.295 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:33 GMT'), ('x-amz-version-id', 'rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000faf9cf066129fef3-0065bb7316-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:50.296 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:31:50.296 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:31:50.296 DEBUG:boto:path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:50.296 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:50.296 DEBUG:boto:Method: GET 2024-02-01T10:31:50.296 DEBUG:boto:Path: /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:50.296 DEBUG:boto:Data: 2024-02-01T10:31:50.296 DEBUG:boto:Headers: {} 2024-02-01T10:31:50.296 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:31:50.296 DEBUG:boto:Port: 8000 2024-02-01T10:31:50.296 DEBUG:boto:Params: {} 2024-02-01T10:31:50.297 DEBUG:boto:Token: None 2024-02-01T10:31:50.297 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:31:50 GMT /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:50.297 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GURvpYML+oEUrQuvwLV39+zEgjE= 2024-02-01T10:31:50.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GURvpYML+oEUrQuvwLV39+zEgjE=', 'Content-Length': '0'} 2024-02-01T10:31:50.300 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:33 GMT'), ('x-amz-version-id', 'rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009ef64683f9485cc0-0065bb7316-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:50.300 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:31:50.300 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:31:50.300 DEBUG:boto:path=/vclfcn-20/?prefix=obj 2024-02-01T10:31:50.300 DEBUG:boto:auth_path=/vclfcn-20/?prefix=obj 2024-02-01T10:31:50.300 DEBUG:boto:Method: GET 2024-02-01T10:31:50.300 DEBUG:boto:Path: /vclfcn-20/?prefix=obj 2024-02-01T10:31:50.300 DEBUG:boto:Data: 2024-02-01T10:31:50.300 DEBUG:boto:Headers: {} 2024-02-01T10:31:50.301 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:31:50.301 DEBUG:boto:Port: 8001 2024-02-01T10:31:50.301 DEBUG:boto:Params: {} 2024-02-01T10:31:50.301 DEBUG:boto:Token: None 2024-02-01T10:31:50.301 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:31:50 GMT /vclfcn-20/ 2024-02-01T10:31:50.301 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:AwovwTRPxx0Vc6/GGzl9hWNtDS0= 2024-02-01T10:31:50.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:AwovwTRPxx0Vc6/GGzl9hWNtDS0=', 'Content-Length': '0'} 2024-02-01T10:31:50.305 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b84ffdfa36196428-0065bb7316-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:50.305 DEBUG:boto:b'vclfcn-20obj1000falseobj2024-02-01T10:28:33.438Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:31:50.306 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:31:50.306 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:31:50.306 DEBUG:boto:path=/vclfcn-20/?prefix=obj 2024-02-01T10:31:50.306 DEBUG:boto:auth_path=/vclfcn-20/?prefix=obj 2024-02-01T10:31:50.306 DEBUG:boto:Method: GET 2024-02-01T10:31:50.306 DEBUG:boto:Path: /vclfcn-20/?prefix=obj 2024-02-01T10:31:50.306 DEBUG:boto:Data: 2024-02-01T10:31:50.306 DEBUG:boto:Headers: {} 2024-02-01T10:31:50.306 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:31:50.307 DEBUG:boto:Port: 8000 2024-02-01T10:31:50.307 DEBUG:boto:Params: {} 2024-02-01T10:31:50.307 DEBUG:boto:Token: None 2024-02-01T10:31:50.307 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:31:50 GMT /vclfcn-20/ 2024-02-01T10:31:50.307 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:AwovwTRPxx0Vc6/GGzl9hWNtDS0= 2024-02-01T10:31:50.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:AwovwTRPxx0Vc6/GGzl9hWNtDS0=', 'Content-Length': '0'} 2024-02-01T10:31:50.311 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b78c45a2563111ef-0065bb7316-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:50.311 DEBUG:boto:b'vclfcn-20obj1000falseobj2024-02-01T10:28:33.438Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:31:50.312 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:31:50.312 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:31:50.312 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:31:50.312 DEBUG:boto:path=/vclfcn-20/obj?versionId=null 2024-02-01T10:31:50.312 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=null 2024-02-01T10:31:50.312 DEBUG:boto:Method: GET 2024-02-01T10:31:50.313 DEBUG:boto:Path: /vclfcn-20/obj?versionId=null 2024-02-01T10:31:50.313 DEBUG:boto:Data: 2024-02-01T10:31:50.313 DEBUG:boto:Headers: {} 2024-02-01T10:31:50.313 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:31:50.313 DEBUG:boto:Port: 8001 2024-02-01T10:31:50.313 DEBUG:boto:Params: {} 2024-02-01T10:31:50.313 DEBUG:boto:Token: None 2024-02-01T10:31:50.313 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:31:50 GMT /vclfcn-20/obj?versionId=null 2024-02-01T10:31:50.313 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:yTowTSmKHIJVn7dNXDm/JEROTCg= 2024-02-01T10:31:50.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:yTowTSmKHIJVn7dNXDm/JEROTCg=', 'Content-Length': '0'} 2024-02-01T10:31:50.316 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000009d01b18d405aa4b-0065bb7316-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:50.316 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:31:50.316 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:31:50.317 DEBUG:boto:path=/vclfcn-20/obj?versionId=null 2024-02-01T10:31:50.317 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=null 2024-02-01T10:31:50.317 DEBUG:boto:Method: GET 2024-02-01T10:31:50.317 DEBUG:boto:Path: /vclfcn-20/obj?versionId=null 2024-02-01T10:31:50.317 DEBUG:boto:Data: 2024-02-01T10:31:50.317 DEBUG:boto:Headers: {} 2024-02-01T10:31:50.317 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:31:50.317 DEBUG:boto:Port: 8000 2024-02-01T10:31:50.317 DEBUG:boto:Params: {} 2024-02-01T10:31:50.317 DEBUG:boto:Token: None 2024-02-01T10:31:50.318 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:31:50 GMT /vclfcn-20/obj?versionId=null 2024-02-01T10:31:50.318 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:yTowTSmKHIJVn7dNXDm/JEROTCg= 2024-02-01T10:31:50.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:yTowTSmKHIJVn7dNXDm/JEROTCg=', 'Content-Length': '0'} 2024-02-01T10:31:50.320 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005f0d69f0d7185ac2-0065bb7316-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:50.321 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:31:50.321 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:31:50.321 DEBUG:boto:path=/vclfcn-20/obj?versionId=null 2024-02-01T10:31:50.321 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=null 2024-02-01T10:31:50.321 DEBUG:boto:Method: HEAD 2024-02-01T10:31:50.321 DEBUG:boto:Path: /vclfcn-20/obj?versionId=null 2024-02-01T10:31:50.321 DEBUG:boto:Data: 2024-02-01T10:31:50.321 DEBUG:boto:Headers: {} 2024-02-01T10:31:50.321 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:31:50.321 DEBUG:boto:Port: 8001 2024-02-01T10:31:50.322 DEBUG:boto:Params: {} 2024-02-01T10:31:50.322 DEBUG:boto:Token: None 2024-02-01T10:31:50.322 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:31:50 GMT /vclfcn-20/obj?versionId=null 2024-02-01T10:31:50.322 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:XJF4II9w8GuB12M+6kuwO7kjr9U= 2024-02-01T10:31:50.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:XJF4II9w8GuB12M+6kuwO7kjr9U=', 'Content-Length': '0'} 2024-02-01T10:31:50.325 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a344fbadfebd7ef5-0065bb7316-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:50.325 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:31:50.325 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:31:50.325 DEBUG:boto:path=/vclfcn-20/obj?versionId=null 2024-02-01T10:31:50.325 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=null 2024-02-01T10:31:50.325 DEBUG:boto:Method: HEAD 2024-02-01T10:31:50.325 DEBUG:boto:Path: /vclfcn-20/obj?versionId=null 2024-02-01T10:31:50.325 DEBUG:boto:Data: 2024-02-01T10:31:50.326 DEBUG:boto:Headers: {} 2024-02-01T10:31:50.326 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:31:50.326 DEBUG:boto:Port: 8000 2024-02-01T10:31:50.326 DEBUG:boto:Params: {} 2024-02-01T10:31:50.326 DEBUG:boto:Token: None 2024-02-01T10:31:50.326 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:31:50 GMT /vclfcn-20/obj?versionId=null 2024-02-01T10:31:50.326 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:XJF4II9w8GuB12M+6kuwO7kjr9U= 2024-02-01T10:31:50.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:XJF4II9w8GuB12M+6kuwO7kjr9U=', 'Content-Length': '0'} 2024-02-01T10:31:50.329 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000072be415f8674820b-0065bb7316-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:50.329 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:31:50.329 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:31:50.329 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:31:50.329 DEBUG:boto:path=/vclfcn-20/obj?versionId=null 2024-02-01T10:31:50.330 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=null 2024-02-01T10:31:50.330 DEBUG:boto:Method: GET 2024-02-01T10:31:50.330 DEBUG:boto:Path: /vclfcn-20/obj?versionId=null 2024-02-01T10:31:50.330 DEBUG:boto:Data: 2024-02-01T10:31:50.330 DEBUG:boto:Headers: {} 2024-02-01T10:31:50.330 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:31:50.330 DEBUG:boto:Port: 8001 2024-02-01T10:31:50.330 DEBUG:boto:Params: {} 2024-02-01T10:31:50.330 DEBUG:boto:Token: None 2024-02-01T10:31:50.330 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:31:50 GMT /vclfcn-20/obj?versionId=null 2024-02-01T10:31:50.330 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:yTowTSmKHIJVn7dNXDm/JEROTCg= 2024-02-01T10:31:50.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:yTowTSmKHIJVn7dNXDm/JEROTCg=', 'Content-Length': '0'} 2024-02-01T10:31:50.333 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000081519b7524261f02-0065bb7316-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:50.333 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:31:50.334 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:31:50.334 DEBUG:boto:path=/vclfcn-20/obj?versionId=null 2024-02-01T10:31:50.334 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=null 2024-02-01T10:31:50.334 DEBUG:boto:Method: GET 2024-02-01T10:31:50.334 DEBUG:boto:Path: /vclfcn-20/obj?versionId=null 2024-02-01T10:31:50.334 DEBUG:boto:Data: 2024-02-01T10:31:50.334 DEBUG:boto:Headers: {} 2024-02-01T10:31:50.334 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:31:50.334 DEBUG:boto:Port: 8000 2024-02-01T10:31:50.334 DEBUG:boto:Params: {} 2024-02-01T10:31:50.334 DEBUG:boto:Token: None 2024-02-01T10:31:50.335 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:31:50 GMT /vclfcn-20/obj?versionId=null 2024-02-01T10:31:50.335 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:yTowTSmKHIJVn7dNXDm/JEROTCg= 2024-02-01T10:31:50.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:yTowTSmKHIJVn7dNXDm/JEROTCg=', 'Content-Length': '0'} 2024-02-01T10:31:50.337 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000f3881c850bf6eb8-0065bb7316-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:50.338 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:31:50.338 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:31:50.338 DEBUG:boto:path=/vclfcn-20/?prefix=obj 2024-02-01T10:31:50.338 DEBUG:boto:auth_path=/vclfcn-20/?prefix=obj 2024-02-01T10:31:50.338 DEBUG:boto:Method: GET 2024-02-01T10:31:50.338 DEBUG:boto:Path: /vclfcn-20/?prefix=obj 2024-02-01T10:31:50.338 DEBUG:boto:Data: 2024-02-01T10:31:50.338 DEBUG:boto:Headers: {} 2024-02-01T10:31:50.338 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:31:50.338 DEBUG:boto:Port: 8001 2024-02-01T10:31:50.338 DEBUG:boto:Params: {} 2024-02-01T10:31:50.339 DEBUG:boto:Token: None 2024-02-01T10:31:50.339 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:31:50 GMT /vclfcn-20/ 2024-02-01T10:31:50.339 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:AwovwTRPxx0Vc6/GGzl9hWNtDS0= 2024-02-01T10:31:50.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:AwovwTRPxx0Vc6/GGzl9hWNtDS0=', 'Content-Length': '0'} 2024-02-01T10:31:50.344 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007289f304333d08e2-0065bb7316-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:50.384 DEBUG:boto:b'vclfcn-20obj1000falseobj2024-02-01T10:28:33.438Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:31:50.384 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:31:50.385 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:31:50.385 DEBUG:boto:path=/vclfcn-20/?prefix=obj 2024-02-01T10:31:50.385 DEBUG:boto:auth_path=/vclfcn-20/?prefix=obj 2024-02-01T10:31:50.385 DEBUG:boto:Method: GET 2024-02-01T10:31:50.385 DEBUG:boto:Path: /vclfcn-20/?prefix=obj 2024-02-01T10:31:50.385 DEBUG:boto:Data: 2024-02-01T10:31:50.385 DEBUG:boto:Headers: {} 2024-02-01T10:31:50.385 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:31:50.385 DEBUG:boto:Port: 8000 2024-02-01T10:31:50.385 DEBUG:boto:Params: {} 2024-02-01T10:31:50.385 DEBUG:boto:Token: None 2024-02-01T10:31:50.386 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:31:50 GMT /vclfcn-20/ 2024-02-01T10:31:50.386 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:AwovwTRPxx0Vc6/GGzl9hWNtDS0= 2024-02-01T10:31:50.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:AwovwTRPxx0Vc6/GGzl9hWNtDS0=', 'Content-Length': '0'} 2024-02-01T10:31:50.390 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042041c96afd29150-0065bb7316-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:50.391 DEBUG:boto:b'vclfcn-20obj1000falseobj2024-02-01T10:28:33.438Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:31:50.391 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:31:50.391 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:31:50.391 DEBUG:boto:Method: HEAD 2024-02-01T10:31:50.391 DEBUG:boto:Path: /vclfcn-20/obj 2024-02-01T10:31:50.392 DEBUG:boto:Data: 2024-02-01T10:31:50.392 DEBUG:boto:Headers: {} 2024-02-01T10:31:50.392 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:31:50.392 DEBUG:boto:Port: 8001 2024-02-01T10:31:50.392 DEBUG:boto:Params: {} 2024-02-01T10:31:50.392 DEBUG:boto:Token: None 2024-02-01T10:31:50.392 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:31:50 GMT /vclfcn-20/obj 2024-02-01T10:31:50.392 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:VgWorJrYumhgyWC/F1h615nKGSk= 2024-02-01T10:31:50.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:VgWorJrYumhgyWC/F1h615nKGSk=', 'Content-Length': '0'} 2024-02-01T10:31:50.396 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:33 GMT'), ('x-amz-version-id', 'rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001e4a32a1b95be98c-0065bb7316-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:50.396 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:31:50.396 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:31:50.396 DEBUG:boto:Method: HEAD 2024-02-01T10:31:50.396 DEBUG:boto:Path: /vclfcn-20/obj 2024-02-01T10:31:50.397 DEBUG:boto:Data: 2024-02-01T10:31:50.397 DEBUG:boto:Headers: {} 2024-02-01T10:31:50.397 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:31:50.397 DEBUG:boto:Port: 8000 2024-02-01T10:31:50.397 DEBUG:boto:Params: {} 2024-02-01T10:31:50.397 DEBUG:boto:Token: None 2024-02-01T10:31:50.397 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:31:50 GMT /vclfcn-20/obj 2024-02-01T10:31:50.397 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:VgWorJrYumhgyWC/F1h615nKGSk= 2024-02-01T10:31:50.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:VgWorJrYumhgyWC/F1h615nKGSk=', 'Content-Length': '0'} 2024-02-01T10:31:50.401 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:33 GMT'), ('x-amz-version-id', 'rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008070f097ebc56194-0065bb7316-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:50.402 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:31:50.402 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:31:50.402 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:31:50.402 DEBUG:boto:path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:50.402 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:50.402 DEBUG:boto:Method: GET 2024-02-01T10:31:50.402 DEBUG:boto:Path: /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:50.402 DEBUG:boto:Data: 2024-02-01T10:31:50.402 DEBUG:boto:Headers: {} 2024-02-01T10:31:50.403 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:31:50.403 DEBUG:boto:Port: 8001 2024-02-01T10:31:50.403 DEBUG:boto:Params: {} 2024-02-01T10:31:50.403 DEBUG:boto:Token: None 2024-02-01T10:31:50.403 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:31:50 GMT /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:50.403 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GURvpYML+oEUrQuvwLV39+zEgjE= 2024-02-01T10:31:50.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GURvpYML+oEUrQuvwLV39+zEgjE=', 'Content-Length': '0'} 2024-02-01T10:31:50.406 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:33 GMT'), ('x-amz-version-id', 'rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e7945e545d2bfff4-0065bb7316-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:50.406 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:31:50.406 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:31:50.406 DEBUG:boto:path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:50.406 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:50.406 DEBUG:boto:Method: GET 2024-02-01T10:31:50.406 DEBUG:boto:Path: /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:50.407 DEBUG:boto:Data: 2024-02-01T10:31:50.407 DEBUG:boto:Headers: {} 2024-02-01T10:31:50.407 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:31:50.407 DEBUG:boto:Port: 8000 2024-02-01T10:31:50.407 DEBUG:boto:Params: {} 2024-02-01T10:31:50.407 DEBUG:boto:Token: None 2024-02-01T10:31:50.407 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:31:50 GMT /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:31:50.407 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GURvpYML+oEUrQuvwLV39+zEgjE= 2024-02-01T10:31:50.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GURvpYML+oEUrQuvwLV39+zEgjE=', 'Content-Length': '0'} 2024-02-01T10:31:50.410 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:33 GMT'), ('x-amz-version-id', 'rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f433a62884f88dd4-0065bb7316-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:50.410 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:31:50.410 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:31:50.410 DEBUG:boto:path=/vclfcn-20/?prefix=obj 2024-02-01T10:31:50.411 DEBUG:boto:auth_path=/vclfcn-20/?prefix=obj 2024-02-01T10:31:50.411 DEBUG:boto:Method: GET 2024-02-01T10:31:50.411 DEBUG:boto:Path: /vclfcn-20/?prefix=obj 2024-02-01T10:31:50.411 DEBUG:boto:Data: 2024-02-01T10:31:50.411 DEBUG:boto:Headers: {} 2024-02-01T10:31:50.411 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:31:50.411 DEBUG:boto:Port: 8001 2024-02-01T10:31:50.411 DEBUG:boto:Params: {} 2024-02-01T10:31:50.411 DEBUG:boto:Token: None 2024-02-01T10:31:50.411 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:31:50 GMT /vclfcn-20/ 2024-02-01T10:31:50.411 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:AwovwTRPxx0Vc6/GGzl9hWNtDS0= 2024-02-01T10:31:50.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:AwovwTRPxx0Vc6/GGzl9hWNtDS0=', 'Content-Length': '0'} 2024-02-01T10:31:50.416 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb8413ca4a512591-0065bb7316-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:50.416 DEBUG:boto:b'vclfcn-20obj1000falseobj2024-02-01T10:28:33.438Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:31:50.417 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:31:50.417 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:31:50.417 DEBUG:boto:path=/vclfcn-20/?prefix=obj 2024-02-01T10:31:50.417 DEBUG:boto:auth_path=/vclfcn-20/?prefix=obj 2024-02-01T10:31:50.417 DEBUG:boto:Method: GET 2024-02-01T10:31:50.417 DEBUG:boto:Path: /vclfcn-20/?prefix=obj 2024-02-01T10:31:50.417 DEBUG:boto:Data: 2024-02-01T10:31:50.417 DEBUG:boto:Headers: {} 2024-02-01T10:31:50.417 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:31:50.418 DEBUG:boto:Port: 8000 2024-02-01T10:31:50.418 DEBUG:boto:Params: {} 2024-02-01T10:31:50.418 DEBUG:boto:Token: None 2024-02-01T10:31:50.418 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:31:50 GMT /vclfcn-20/ 2024-02-01T10:31:50.418 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:AwovwTRPxx0Vc6/GGzl9hWNtDS0= 2024-02-01T10:31:50.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:AwovwTRPxx0Vc6/GGzl9hWNtDS0=', 'Content-Length': '0'} 2024-02-01T10:31:50.422 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007346d72b9297338f-0065bb7316-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:50.464 DEBUG:boto:b'vclfcn-20obj1000falseobj2024-02-01T10:28:33.438Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:31:50.464 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-20 zones={test-zone3, test-zone2} 2024-02-01T10:31:50.465 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:31:50.465 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:31:50.465 DEBUG:boto:path=/vclfcn-20/?versioning 2024-02-01T10:31:50.465 DEBUG:boto:auth_path=/vclfcn-20/?versioning 2024-02-01T10:31:50.465 DEBUG:boto:Method: PUT 2024-02-01T10:31:50.465 DEBUG:boto:Path: /vclfcn-20/?versioning 2024-02-01T10:31:50.465 DEBUG:boto:Data: Suspended Disabled 2024-02-01T10:31:50.465 DEBUG:boto:Headers: {} 2024-02-01T10:31:50.465 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:31:50.466 DEBUG:boto:Port: 8000 2024-02-01T10:31:50.466 DEBUG:boto:Params: {} 2024-02-01T10:31:50.466 DEBUG:boto:Token: None 2024-02-01T10:31:50.466 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:31:50 GMT /vclfcn-20/?versioning 2024-02-01T10:31:50.466 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Pk2Z87AFNieo916GgdLczSu5Kyk= 2024-02-01T10:31:50.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Pk2Z87AFNieo916GgdLczSu5Kyk=', 'Content-Length': '230'} 2024-02-01T10:31:50.478 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e0b75cf049936602-0065bb7316-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:50.478 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:31:50.478 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:31:50.478 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:31:50.518 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:31:50.519 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:31:50.574 INFO:teuthology.orchestra.run.smithi148.stderr:No --period given, using current period=5dc4efbd-eccd-47a8-9471-ae4a209ec978 2024-02-01T10:31:50.598 INFO:teuthology.orchestra.run.smithi148.stdout:[{"marker":"1_1706782453.132619_30.1","last_update":"2024-02-01T10:14:13.132619Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1706783510.486411_123.1","last_update":"2024-02-01T10:31:50.486411Z"},{"marker":"1_1706782595.670952_121.1","last_update":"2024-02-01T10:16:35.670952Z"}] 2024-02-01T10:31:50.599 DEBUG:rgw_multi.tests:master meta markers={0: '1_1706782453.132619_30.1', 1: '', 2: '1_1706783510.486411_123.1', 3: '1_1706782595.670952_121.1'} 2024-02-01T10:31:50.599 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-02-01T10:31:50.599 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:31:50.599 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:31:50.599 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:31:50.646 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:31:50.646 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:31:50.701 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:31:50.703+0000 7fde84318980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:31:50.701 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:31:50.703+0000 7fde84318980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:31:50.725 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783510.486411_123.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:31:50.486411Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:31:50Z"} 2024-02-01T10:31:50.725 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783510.486411_123.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:31:50.486411Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:31:50Z"} 2024-02-01T10:31:50.725 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706783510.486411_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:31:50.486411Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:31:50.726 DEBUG:rgw_multi.tests:log_status={0: '1_1706782453.132619_30.1', 1: '', 2: '1_1706783510.486411_123.1', 3: '1_1706782595.670952_121.1'} 2024-02-01T10:31:50.726 DEBUG:rgw_multi.tests:sync_status={0: '1_1706782453.132619_30.1', 1: '', 2: '1_1706783510.486411_123.1', 3: '1_1706782595.670952_121.1'} 2024-02-01T10:31:50.726 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-02-01T10:31:50.726 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-02-01T10:31:50.726 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:31:50.726 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:31:50.726 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:31:50.766 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:31:50.767 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:31:50.824 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:31:50.827+0000 7fdd8a93c980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:31:50.824 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:31:50.827+0000 7fdd8a93c980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:31:50.844 INFO:teuthology.orchestra.run.smithi148.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783510.486411_123.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:31:50.486411Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:31:50Z"} 2024-02-01T10:31:50.845 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783510.486411_123.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:31:50.486411Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:31:50Z"} 2024-02-01T10:31:50.845 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706783510.486411_123.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:31:50.486411Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:31:50.845 DEBUG:rgw_multi.tests:log_status={0: '1_1706782453.132619_30.1', 1: '', 2: '1_1706783510.486411_123.1', 3: '1_1706782595.670952_121.1'} 2024-02-01T10:31:50.845 DEBUG:rgw_multi.tests:sync_status={0: '1_1706782453.132619_30.1', 1: '', 2: '1_1706783510.486411_123.1', 3: '1_1706782595.670952_121.1'} 2024-02-01T10:31:50.845 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-02-01T10:31:50.845 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:31:50.845 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:31:50.845 DEBUG:boto:Method: HEAD 2024-02-01T10:31:50.846 DEBUG:boto:Path: /vclfcn-20/ 2024-02-01T10:31:50.846 DEBUG:boto:Data: 2024-02-01T10:31:50.846 DEBUG:boto:Headers: {} 2024-02-01T10:31:50.846 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:31:50.846 DEBUG:boto:Port: 8000 2024-02-01T10:31:50.846 DEBUG:boto:Params: {} 2024-02-01T10:31:50.846 DEBUG:boto:Token: None 2024-02-01T10:31:50.846 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:31:50 GMT /vclfcn-20/ 2024-02-01T10:31:50.846 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:D4mmgj6QoztqbgvqYfRhUltDUBM= 2024-02-01T10:31:50.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:31:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:D4mmgj6QoztqbgvqYfRhUltDUBM=', 'Content-Length': '0'} 2024-02-01T10:31:50.849 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008baaceb7c72a0258-0065bb7316-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:50.849 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:31:50.849 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:31:50.849 DEBUG:boto:Method: PUT 2024-02-01T10:31:50.850 DEBUG:boto:Path: /vclfcn-20/obj 2024-02-01T10:31:50.850 DEBUG:boto:Data: 2024-02-01T10:31:50.850 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-02-01T10:31:50.850 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:31:50.850 DEBUG:boto:Port: 8000 2024-02-01T10:31:50.850 DEBUG:boto:Params: {} 2024-02-01T10:31:50.850 DEBUG:boto:Token: None 2024-02-01T10:31:50.850 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Thu, 01 Feb 2024 10:31:50 GMT /vclfcn-20/obj 2024-02-01T10:31:50.850 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5C4z3vYStcG/Q8JIxEkx+t8NHCE= 2024-02-01T10:31:50.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:31:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5C4z3vYStcG/Q8JIxEkx+t8NHCE='} 2024-02-01T10:31:50.866 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b4c00324011f1aff-0065bb7316-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:31:50.866 DEBUG:rgw_multi.tests:created null version id=None 2024-02-01T10:31:50.866 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=vclfcn-20 2024-02-01T10:31:50.866 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:31:50.867 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:31:50.867 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket vclfcn-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-02-01T10:31:50.919 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:31:50.919 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:31:50.984 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:31:50.983+0000 7f694fe42980 1 waiting for incremental sync to catch up: 2024-02-01T10:31:50.984 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, , , , , , , , 00000000003.37.13, , ] 2024-02-01T10:31:50.984 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, , , , , , , , 00000000005.41.18, , ] 2024-02-01T10:31:55.986 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:31:55.987+0000 7f694fe42980 1 waiting for incremental sync to catch up: 2024-02-01T10:31:55.986 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, , , , , , , , 00000000003.37.13, , ] 2024-02-01T10:31:55.987 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, , , , , , , , 00000000005.41.18, , ] 2024-02-01T10:32:00.989 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:32:00.991+0000 7f694fe42980 1 waiting for incremental sync to catch up: 2024-02-01T10:32:00.989 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, , , , , , , , 00000000003.37.13, , ] 2024-02-01T10:32:00.989 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, , , , , , , , 00000000005.41.18, , ] 2024-02-01T10:32:05.991 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:32:05.990+0000 7f694fe42980 1 waiting for incremental sync to catch up: 2024-02-01T10:32:05.992 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, , , , , , , , 00000000003.37.13, , ] 2024-02-01T10:32:05.992 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, , , , , , , , 00000000005.41.18, , ] 2024-02-01T10:32:10.994 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:32:10.994+0000 7f694fe42980 1 bucket sync caught up with source: 2024-02-01T10:32:10.994 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, , , , , , , , 00000000005.41.18, , ] 2024-02-01T10:32:10.994 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, , , , , , , , 00000000005.41.18, , ] 2024-02-01T10:32:10.994 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:32:10.994+0000 7f694fe42980 0 bucket checkpoint complete 2024-02-01T10:32:11.037 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=vclfcn-20 2024-02-01T10:32:11.038 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:32:11.038 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:32:11.038 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:32:11.088 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:32:11.089 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:32:11.150 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:32:11.154+0000 7f277dff8980 1 bucket sync caught up with source: 2024-02-01T10:32:11.150 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, , , , , , , , 00000000005.41.18, , ] 2024-02-01T10:32:11.150 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, , , , , , , , 00000000005.41.18, , ] 2024-02-01T10:32:11.150 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:32:11.154+0000 7f277dff8980 0 bucket checkpoint complete 2024-02-01T10:32:11.172 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=vclfcn-20 2024-02-01T10:32:11.172 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:32:11.172 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:32:11.172 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:32:11.213 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:32:11.213 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:32:11.273 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:32:11.278+0000 7f8fccb61980 1 waiting for incremental sync to catch up: 2024-02-01T10:32:11.273 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, , , , , , , , 00000000004.43.10, , ] 2024-02-01T10:32:11.274 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, , , , , , , , 00000000006.48.18, , ] 2024-02-01T10:32:16.276 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:32:16.282+0000 7f8fccb61980 1 waiting for incremental sync to catch up: 2024-02-01T10:32:16.276 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, , , , , , , , 00000000004.43.10, , ] 2024-02-01T10:32:16.276 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, , , , , , , , 00000000006.48.18, , ] 2024-02-01T10:32:21.278 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:32:21.286+0000 7f8fccb61980 1 waiting for incremental sync to catch up: 2024-02-01T10:32:21.279 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, , , , , , , , 00000000004.43.10, , ] 2024-02-01T10:32:21.279 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, , , , , , , , 00000000006.48.18, , ] 2024-02-01T10:32:26.281 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:32:26.290+0000 7f8fccb61980 1 waiting for incremental sync to catch up: 2024-02-01T10:32:26.281 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, , , , , , , , 00000000004.43.10, , ] 2024-02-01T10:32:26.281 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, , , , , , , , 00000000006.48.18, , ] 2024-02-01T10:32:27.489 INFO:tasks.rgw.c2.client.0.smithi187.stdout:2024-02-01T10:32:27.490+0000 7f7bead57640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-02-01T10:32:31.283 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:32:31.290+0000 7f8fccb61980 1 bucket sync caught up with source: 2024-02-01T10:32:31.284 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, , , , , , , , 00000000006.48.18, , ] 2024-02-01T10:32:31.284 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, , , , , , , , 00000000006.48.18, , ] 2024-02-01T10:32:31.284 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:32:31.290+0000 7f8fccb61980 0 bucket checkpoint complete 2024-02-01T10:32:31.323 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=vclfcn-20 2024-02-01T10:32:31.323 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:32:31.323 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:32:31.323 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:32:31.418 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:32:31.418 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:32:31.482 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:32:31.490+0000 7f35ebbf6980 1 waiting for incremental sync to catch up: 2024-02-01T10:32:31.482 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, , , , , , , , 00000000006.48.18, , ] 2024-02-01T10:32:31.482 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, , , , , , , , 00000000007.50.15, , ] 2024-02-01T10:32:36.485 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:32:36.493+0000 7f35ebbf6980 1 bucket sync caught up with source: 2024-02-01T10:32:36.485 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, , , , , , , , 00000000007.50.15, , ] 2024-02-01T10:32:36.485 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, , , , , , , , 00000000007.50.15, , ] 2024-02-01T10:32:36.485 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:32:36.493+0000 7f35ebbf6980 0 bucket checkpoint complete 2024-02-01T10:32:36.529 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=vclfcn-20 2024-02-01T10:32:36.529 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:32:36.529 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:32:36.529 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:32:36.578 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:32:36.578 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:32:36.640 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:32:36.649+0000 7ff9d3b17980 1 bucket sync caught up with source: 2024-02-01T10:32:36.640 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, , , , , , , , 00000000007.82.15, , ] 2024-02-01T10:32:36.640 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, , , , , , , , 00000000007.82.15, , ] 2024-02-01T10:32:36.640 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:32:36.649+0000 7ff9d3b17980 0 bucket checkpoint complete 2024-02-01T10:32:36.663 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=vclfcn-20 2024-02-01T10:32:36.663 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:32:36.663 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:32:36.663 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket vclfcn-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-02-01T10:32:36.714 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:32:36.714 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:32:36.777 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:32:36.774+0000 7f346f000980 1 bucket sync caught up with source: 2024-02-01T10:32:36.777 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, , , , , , , , 00000000007.82.15, , ] 2024-02-01T10:32:36.777 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, , , , , , , , 00000000007.82.15, , ] 2024-02-01T10:32:36.777 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:32:36.774+0000 7f346f000980 0 bucket checkpoint complete 2024-02-01T10:32:36.801 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-20 zones={test-zone2, test-zone1} 2024-02-01T10:32:36.801 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:32:36.801 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:32:36.802 DEBUG:boto:Method: HEAD 2024-02-01T10:32:36.802 DEBUG:boto:Path: /vclfcn-20/ 2024-02-01T10:32:36.802 DEBUG:boto:Data: 2024-02-01T10:32:36.802 DEBUG:boto:Headers: {} 2024-02-01T10:32:36.802 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:32:36.802 DEBUG:boto:Port: 8000 2024-02-01T10:32:36.802 DEBUG:boto:Params: {} 2024-02-01T10:32:36.803 DEBUG:boto:Token: None 2024-02-01T10:32:36.803 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:32:36 GMT /vclfcn-20/ 2024-02-01T10:32:36.803 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9v7MN7ABvwrbLitJ3slNhkIDJ9Q= 2024-02-01T10:32:36.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9v7MN7ABvwrbLitJ3slNhkIDJ9Q=', 'Content-Length': '0'} 2024-02-01T10:32:36.806 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007afd281998b4abea-0065bb7344-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:32:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:36.806 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:32:36.806 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:32:36.806 DEBUG:boto:Method: HEAD 2024-02-01T10:32:36.806 DEBUG:boto:Path: /vclfcn-20/ 2024-02-01T10:32:36.806 DEBUG:boto:Data: 2024-02-01T10:32:36.807 DEBUG:boto:Headers: {} 2024-02-01T10:32:36.807 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:32:36.807 DEBUG:boto:Port: 8000 2024-02-01T10:32:36.807 DEBUG:boto:Params: {} 2024-02-01T10:32:36.807 DEBUG:boto:Token: None 2024-02-01T10:32:36.807 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:32:36 GMT /vclfcn-20/ 2024-02-01T10:32:36.807 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9v7MN7ABvwrbLitJ3slNhkIDJ9Q= 2024-02-01T10:32:36.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9v7MN7ABvwrbLitJ3slNhkIDJ9Q=', 'Content-Length': '0'} 2024-02-01T10:32:36.811 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005f912020e3659529-0065bb7344-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:32:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:36.811 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:32:36.812 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:32:36.812 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:32:36.812 DEBUG:boto:path=/vclfcn-20/?versions&max-keys=999 2024-02-01T10:32:36.812 DEBUG:boto:auth_path=/vclfcn-20/?versions&max-keys=999 2024-02-01T10:32:36.812 DEBUG:boto:Method: GET 2024-02-01T10:32:36.812 DEBUG:boto:Path: /vclfcn-20/?versions&max-keys=999 2024-02-01T10:32:36.812 DEBUG:boto:Data: 2024-02-01T10:32:36.812 DEBUG:boto:Headers: {} 2024-02-01T10:32:36.812 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:32:36.812 DEBUG:boto:Port: 8000 2024-02-01T10:32:36.813 DEBUG:boto:Params: {} 2024-02-01T10:32:36.813 DEBUG:boto:Token: None 2024-02-01T10:32:36.813 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:32:36 GMT /vclfcn-20/?versions 2024-02-01T10:32:36.813 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Q/S5CMOwgd2Ip4DhTY7z431aUOs= 2024-02-01T10:32:36.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Q/S5CMOwgd2Ip4DhTY7z431aUOs=', 'Content-Length': '0'} 2024-02-01T10:32:36.815 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc89e0208d8f4011-0065bb7344-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:36.816 DEBUG:boto:b'vclfcn-20999falseobjnulltrue2024-02-01T10:31:50.863Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjrSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6false2024-02-01T10:28:33.438Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:32:36.816 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:32:36.816 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:32:36.816 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:32:36.817 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:32:36.817 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:32:36.817 DEBUG:boto:path=/vclfcn-20/?versions&max-keys=999 2024-02-01T10:32:36.817 DEBUG:boto:auth_path=/vclfcn-20/?versions&max-keys=999 2024-02-01T10:32:36.817 DEBUG:boto:Method: GET 2024-02-01T10:32:36.817 DEBUG:boto:Path: /vclfcn-20/?versions&max-keys=999 2024-02-01T10:32:36.817 DEBUG:boto:Data: 2024-02-01T10:32:36.817 DEBUG:boto:Headers: {} 2024-02-01T10:32:36.817 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:32:36.817 DEBUG:boto:Port: 8000 2024-02-01T10:32:36.817 DEBUG:boto:Params: {} 2024-02-01T10:32:36.818 DEBUG:boto:Token: None 2024-02-01T10:32:36.818 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:32:36 GMT /vclfcn-20/?versions 2024-02-01T10:32:36.818 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Q/S5CMOwgd2Ip4DhTY7z431aUOs= 2024-02-01T10:32:36.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Q/S5CMOwgd2Ip4DhTY7z431aUOs=', 'Content-Length': '0'} 2024-02-01T10:32:36.822 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000062522ce378bb115b-0065bb7344-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:36.822 DEBUG:boto:b'vclfcn-20999falseobjnulltrue2024-02-01T10:31:50.863Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjrSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6false2024-02-01T10:28:33.438Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:32:36.823 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:32:36.823 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:32:36.823 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:32:36.823 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:32:36.823 DEBUG:boto:path=/vclfcn-20/?versions&max-keys=999 2024-02-01T10:32:36.823 DEBUG:boto:auth_path=/vclfcn-20/?versions&max-keys=999 2024-02-01T10:32:36.823 DEBUG:boto:Method: GET 2024-02-01T10:32:36.823 DEBUG:boto:Path: /vclfcn-20/?versions&max-keys=999 2024-02-01T10:32:36.823 DEBUG:boto:Data: 2024-02-01T10:32:36.824 DEBUG:boto:Headers: {} 2024-02-01T10:32:36.824 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:32:36.824 DEBUG:boto:Port: 8000 2024-02-01T10:32:36.824 DEBUG:boto:Params: {} 2024-02-01T10:32:36.824 DEBUG:boto:Token: None 2024-02-01T10:32:36.824 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:32:36 GMT /vclfcn-20/?versions 2024-02-01T10:32:36.824 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Q/S5CMOwgd2Ip4DhTY7z431aUOs= 2024-02-01T10:32:36.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Q/S5CMOwgd2Ip4DhTY7z431aUOs=', 'Content-Length': '0'} 2024-02-01T10:32:36.826 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a96df3603d5d697-0065bb7344-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:36.827 DEBUG:boto:b'vclfcn-20999falseobjnulltrue2024-02-01T10:31:50.863Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjrSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6false2024-02-01T10:28:33.438Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:32:36.827 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:32:36.827 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:32:36.827 DEBUG:boto:path=/vclfcn-20/?versions&max-keys=999 2024-02-01T10:32:36.827 DEBUG:boto:auth_path=/vclfcn-20/?versions&max-keys=999 2024-02-01T10:32:36.827 DEBUG:boto:Method: GET 2024-02-01T10:32:36.827 DEBUG:boto:Path: /vclfcn-20/?versions&max-keys=999 2024-02-01T10:32:36.827 DEBUG:boto:Data: 2024-02-01T10:32:36.828 DEBUG:boto:Headers: {} 2024-02-01T10:32:36.828 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:32:36.828 DEBUG:boto:Port: 8000 2024-02-01T10:32:36.828 DEBUG:boto:Params: {} 2024-02-01T10:32:36.828 DEBUG:boto:Token: None 2024-02-01T10:32:36.828 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:32:36 GMT /vclfcn-20/?versions 2024-02-01T10:32:36.828 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Q/S5CMOwgd2Ip4DhTY7z431aUOs= 2024-02-01T10:32:36.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Q/S5CMOwgd2Ip4DhTY7z431aUOs=', 'Content-Length': '0'} 2024-02-01T10:32:36.833 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000318e24720fbd9514-0065bb7344-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:36.876 DEBUG:boto:b'vclfcn-20999falseobjnulltrue2024-02-01T10:31:50.863Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjrSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6false2024-02-01T10:28:33.438Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:32:36.876 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:32:36.877 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:32:36.877 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:32:36.877 DEBUG:boto:path=/vclfcn-20/obj?versionId=null 2024-02-01T10:32:36.877 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=null 2024-02-01T10:32:36.877 DEBUG:boto:Method: GET 2024-02-01T10:32:36.877 DEBUG:boto:Path: /vclfcn-20/obj?versionId=null 2024-02-01T10:32:36.877 DEBUG:boto:Data: 2024-02-01T10:32:36.877 DEBUG:boto:Headers: {} 2024-02-01T10:32:36.877 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:32:36.877 DEBUG:boto:Port: 8000 2024-02-01T10:32:36.878 DEBUG:boto:Params: {} 2024-02-01T10:32:36.878 DEBUG:boto:Token: None 2024-02-01T10:32:36.878 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:32:36 GMT /vclfcn-20/obj?versionId=null 2024-02-01T10:32:36.878 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:CNV3TY/Ka770nHL3cz/sojn/lkk= 2024-02-01T10:32:36.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:CNV3TY/Ka770nHL3cz/sojn/lkk=', 'Content-Length': '0'} 2024-02-01T10:32:36.881 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000dc986597400d570e-0065bb7344-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:36.881 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:32:36.881 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:32:36.881 DEBUG:boto:path=/vclfcn-20/obj?versionId=null 2024-02-01T10:32:36.882 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=null 2024-02-01T10:32:36.882 DEBUG:boto:Method: GET 2024-02-01T10:32:36.882 DEBUG:boto:Path: /vclfcn-20/obj?versionId=null 2024-02-01T10:32:36.882 DEBUG:boto:Data: 2024-02-01T10:32:36.882 DEBUG:boto:Headers: {} 2024-02-01T10:32:36.882 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:32:36.882 DEBUG:boto:Port: 8000 2024-02-01T10:32:36.882 DEBUG:boto:Params: {} 2024-02-01T10:32:36.882 DEBUG:boto:Token: None 2024-02-01T10:32:36.882 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:32:36 GMT /vclfcn-20/obj?versionId=null 2024-02-01T10:32:36.883 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:CNV3TY/Ka770nHL3cz/sojn/lkk= 2024-02-01T10:32:36.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:CNV3TY/Ka770nHL3cz/sojn/lkk=', 'Content-Length': '0'} 2024-02-01T10:32:36.885 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a74bb8999b5ca1a9-0065bb7344-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:36.886 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:32:36.886 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:32:36.886 DEBUG:boto:path=/vclfcn-20/obj?versionId=null 2024-02-01T10:32:36.886 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=null 2024-02-01T10:32:36.886 DEBUG:boto:Method: HEAD 2024-02-01T10:32:36.886 DEBUG:boto:Path: /vclfcn-20/obj?versionId=null 2024-02-01T10:32:36.886 DEBUG:boto:Data: 2024-02-01T10:32:36.886 DEBUG:boto:Headers: {} 2024-02-01T10:32:36.886 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:32:36.886 DEBUG:boto:Port: 8000 2024-02-01T10:32:36.886 DEBUG:boto:Params: {} 2024-02-01T10:32:36.887 DEBUG:boto:Token: None 2024-02-01T10:32:36.887 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:32:36 GMT /vclfcn-20/obj?versionId=null 2024-02-01T10:32:36.887 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:iqge9vaBszAdLIdRvZVLS8PY5b8= 2024-02-01T10:32:36.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iqge9vaBszAdLIdRvZVLS8PY5b8=', 'Content-Length': '0'} 2024-02-01T10:32:36.889 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000099409d185d46a5-0065bb7344-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:36.890 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:32:36.890 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:32:36.890 DEBUG:boto:path=/vclfcn-20/obj?versionId=null 2024-02-01T10:32:36.890 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=null 2024-02-01T10:32:36.890 DEBUG:boto:Method: HEAD 2024-02-01T10:32:36.890 DEBUG:boto:Path: /vclfcn-20/obj?versionId=null 2024-02-01T10:32:36.890 DEBUG:boto:Data: 2024-02-01T10:32:36.890 DEBUG:boto:Headers: {} 2024-02-01T10:32:36.890 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:32:36.890 DEBUG:boto:Port: 8000 2024-02-01T10:32:36.890 DEBUG:boto:Params: {} 2024-02-01T10:32:36.891 DEBUG:boto:Token: None 2024-02-01T10:32:36.891 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:32:36 GMT /vclfcn-20/obj?versionId=null 2024-02-01T10:32:36.891 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:iqge9vaBszAdLIdRvZVLS8PY5b8= 2024-02-01T10:32:36.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iqge9vaBszAdLIdRvZVLS8PY5b8=', 'Content-Length': '0'} 2024-02-01T10:32:36.893 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003c63545248dd5896-0065bb7344-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:36.894 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:32:36.894 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:32:36.894 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:32:36.894 DEBUG:boto:path=/vclfcn-20/obj?versionId=null 2024-02-01T10:32:36.894 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=null 2024-02-01T10:32:36.894 DEBUG:boto:Method: GET 2024-02-01T10:32:36.894 DEBUG:boto:Path: /vclfcn-20/obj?versionId=null 2024-02-01T10:32:36.894 DEBUG:boto:Data: 2024-02-01T10:32:36.894 DEBUG:boto:Headers: {} 2024-02-01T10:32:36.894 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:32:36.895 DEBUG:boto:Port: 8000 2024-02-01T10:32:36.895 DEBUG:boto:Params: {} 2024-02-01T10:32:36.895 DEBUG:boto:Token: None 2024-02-01T10:32:36.895 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:32:36 GMT /vclfcn-20/obj?versionId=null 2024-02-01T10:32:36.895 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:CNV3TY/Ka770nHL3cz/sojn/lkk= 2024-02-01T10:32:36.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:CNV3TY/Ka770nHL3cz/sojn/lkk=', 'Content-Length': '0'} 2024-02-01T10:32:36.898 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b8b035850e555213-0065bb7344-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:36.898 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:32:36.898 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:32:36.898 DEBUG:boto:path=/vclfcn-20/obj?versionId=null 2024-02-01T10:32:36.898 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=null 2024-02-01T10:32:36.898 DEBUG:boto:Method: GET 2024-02-01T10:32:36.898 DEBUG:boto:Path: /vclfcn-20/obj?versionId=null 2024-02-01T10:32:36.899 DEBUG:boto:Data: 2024-02-01T10:32:36.899 DEBUG:boto:Headers: {} 2024-02-01T10:32:36.899 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:32:36.899 DEBUG:boto:Port: 8000 2024-02-01T10:32:36.899 DEBUG:boto:Params: {} 2024-02-01T10:32:36.899 DEBUG:boto:Token: None 2024-02-01T10:32:36.899 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:32:36 GMT /vclfcn-20/obj?versionId=null 2024-02-01T10:32:36.899 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:CNV3TY/Ka770nHL3cz/sojn/lkk= 2024-02-01T10:32:36.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:CNV3TY/Ka770nHL3cz/sojn/lkk=', 'Content-Length': '0'} 2024-02-01T10:32:36.902 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b855ee42c1c5f2a3-0065bb7344-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:36.902 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:32:36.902 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:32:36.902 DEBUG:boto:path=/vclfcn-20/?prefix=obj 2024-02-01T10:32:36.903 DEBUG:boto:auth_path=/vclfcn-20/?prefix=obj 2024-02-01T10:32:36.903 DEBUG:boto:Method: GET 2024-02-01T10:32:36.903 DEBUG:boto:Path: /vclfcn-20/?prefix=obj 2024-02-01T10:32:36.903 DEBUG:boto:Data: 2024-02-01T10:32:36.903 DEBUG:boto:Headers: {} 2024-02-01T10:32:36.903 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:32:36.903 DEBUG:boto:Port: 8000 2024-02-01T10:32:36.903 DEBUG:boto:Params: {} 2024-02-01T10:32:36.903 DEBUG:boto:Token: None 2024-02-01T10:32:36.903 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:32:36 GMT /vclfcn-20/ 2024-02-01T10:32:36.903 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:IyQUTnTgZf/3IGqZBaTriRoHCWo= 2024-02-01T10:32:36.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:IyQUTnTgZf/3IGqZBaTriRoHCWo=', 'Content-Length': '0'} 2024-02-01T10:32:36.907 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009a1f34ea851cbb8-0065bb7344-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:36.948 DEBUG:boto:b'vclfcn-20obj1000falseobj2024-02-01T10:31:50.863Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:32:36.948 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:32:36.949 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:32:36.949 DEBUG:boto:path=/vclfcn-20/?prefix=obj 2024-02-01T10:32:36.949 DEBUG:boto:auth_path=/vclfcn-20/?prefix=obj 2024-02-01T10:32:36.949 DEBUG:boto:Method: GET 2024-02-01T10:32:36.949 DEBUG:boto:Path: /vclfcn-20/?prefix=obj 2024-02-01T10:32:36.949 DEBUG:boto:Data: 2024-02-01T10:32:36.949 DEBUG:boto:Headers: {} 2024-02-01T10:32:36.949 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:32:36.949 DEBUG:boto:Port: 8000 2024-02-01T10:32:36.949 DEBUG:boto:Params: {} 2024-02-01T10:32:36.950 DEBUG:boto:Token: None 2024-02-01T10:32:36.950 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:32:36 GMT /vclfcn-20/ 2024-02-01T10:32:36.950 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:IyQUTnTgZf/3IGqZBaTriRoHCWo= 2024-02-01T10:32:36.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:IyQUTnTgZf/3IGqZBaTriRoHCWo=', 'Content-Length': '0'} 2024-02-01T10:32:36.954 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002347e09dfab86f28-0065bb7344-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:36.955 DEBUG:boto:b'vclfcn-20obj1000falseobj2024-02-01T10:31:50.863Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:32:36.955 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:32:36.955 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:32:36.955 DEBUG:boto:Method: HEAD 2024-02-01T10:32:36.955 DEBUG:boto:Path: /vclfcn-20/obj 2024-02-01T10:32:36.955 DEBUG:boto:Data: 2024-02-01T10:32:36.955 DEBUG:boto:Headers: {} 2024-02-01T10:32:36.955 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:32:36.956 DEBUG:boto:Port: 8000 2024-02-01T10:32:36.956 DEBUG:boto:Params: {} 2024-02-01T10:32:36.956 DEBUG:boto:Token: None 2024-02-01T10:32:36.956 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:32:36 GMT /vclfcn-20/obj 2024-02-01T10:32:36.956 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:3i831yOWDFlI00g+X32BgSwqTDk= 2024-02-01T10:32:36.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:3i831yOWDFlI00g+X32BgSwqTDk=', 'Content-Length': '0'} 2024-02-01T10:32:36.959 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000088deae63039d698d-0065bb7344-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:36.959 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:32:36.959 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:32:36.959 DEBUG:boto:Method: HEAD 2024-02-01T10:32:36.959 DEBUG:boto:Path: /vclfcn-20/obj 2024-02-01T10:32:36.959 DEBUG:boto:Data: 2024-02-01T10:32:36.959 DEBUG:boto:Headers: {} 2024-02-01T10:32:36.959 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:32:36.960 DEBUG:boto:Port: 8000 2024-02-01T10:32:36.960 DEBUG:boto:Params: {} 2024-02-01T10:32:36.960 DEBUG:boto:Token: None 2024-02-01T10:32:36.960 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:32:36 GMT /vclfcn-20/obj 2024-02-01T10:32:36.960 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:3i831yOWDFlI00g+X32BgSwqTDk= 2024-02-01T10:32:36.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:3i831yOWDFlI00g+X32BgSwqTDk=', 'Content-Length': '0'} 2024-02-01T10:32:36.963 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007047241e5c0dc7ba-0065bb7344-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:36.963 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:32:36.963 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:32:36.963 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:32:36.963 DEBUG:boto:Method: GET 2024-02-01T10:32:36.963 DEBUG:boto:Path: /vclfcn-20/obj 2024-02-01T10:32:36.964 DEBUG:boto:Data: 2024-02-01T10:32:36.964 DEBUG:boto:Headers: {} 2024-02-01T10:32:36.964 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:32:36.964 DEBUG:boto:Port: 8000 2024-02-01T10:32:36.964 DEBUG:boto:Params: {} 2024-02-01T10:32:36.964 DEBUG:boto:Token: None 2024-02-01T10:32:36.964 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:32:36 GMT /vclfcn-20/obj 2024-02-01T10:32:36.964 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YOFpC0BY7chMkeofymC6vX9R7AQ= 2024-02-01T10:32:36.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YOFpC0BY7chMkeofymC6vX9R7AQ=', 'Content-Length': '0'} 2024-02-01T10:32:36.967 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000c4a2b8f81bb543c-0065bb7344-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:36.967 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:32:36.968 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:32:36.968 DEBUG:boto:Method: GET 2024-02-01T10:32:36.968 DEBUG:boto:Path: /vclfcn-20/obj 2024-02-01T10:32:36.968 DEBUG:boto:Data: 2024-02-01T10:32:36.968 DEBUG:boto:Headers: {} 2024-02-01T10:32:36.968 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:32:36.968 DEBUG:boto:Port: 8000 2024-02-01T10:32:36.968 DEBUG:boto:Params: {} 2024-02-01T10:32:36.968 DEBUG:boto:Token: None 2024-02-01T10:32:36.968 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:32:36 GMT /vclfcn-20/obj 2024-02-01T10:32:36.968 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YOFpC0BY7chMkeofymC6vX9R7AQ= 2024-02-01T10:32:36.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YOFpC0BY7chMkeofymC6vX9R7AQ=', 'Content-Length': '0'} 2024-02-01T10:32:36.971 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000708fec002f5ff3b3-0065bb7344-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:36.971 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:32:36.971 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:32:36.972 DEBUG:boto:path=/vclfcn-20/?prefix=obj 2024-02-01T10:32:36.972 DEBUG:boto:auth_path=/vclfcn-20/?prefix=obj 2024-02-01T10:32:36.972 DEBUG:boto:Method: GET 2024-02-01T10:32:36.972 DEBUG:boto:Path: /vclfcn-20/?prefix=obj 2024-02-01T10:32:36.972 DEBUG:boto:Data: 2024-02-01T10:32:36.972 DEBUG:boto:Headers: {} 2024-02-01T10:32:36.972 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:32:36.972 DEBUG:boto:Port: 8000 2024-02-01T10:32:36.972 DEBUG:boto:Params: {} 2024-02-01T10:32:36.972 DEBUG:boto:Token: None 2024-02-01T10:32:36.972 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:32:36 GMT /vclfcn-20/ 2024-02-01T10:32:36.973 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:IyQUTnTgZf/3IGqZBaTriRoHCWo= 2024-02-01T10:32:36.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:IyQUTnTgZf/3IGqZBaTriRoHCWo=', 'Content-Length': '0'} 2024-02-01T10:32:36.977 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000098357913dc6ca373-0065bb7344-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:36.977 DEBUG:boto:b'vclfcn-20obj1000falseobj2024-02-01T10:31:50.863Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:32:36.977 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:32:36.978 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:32:36.978 DEBUG:boto:path=/vclfcn-20/?prefix=obj 2024-02-01T10:32:36.978 DEBUG:boto:auth_path=/vclfcn-20/?prefix=obj 2024-02-01T10:32:36.978 DEBUG:boto:Method: GET 2024-02-01T10:32:36.978 DEBUG:boto:Path: /vclfcn-20/?prefix=obj 2024-02-01T10:32:36.978 DEBUG:boto:Data: 2024-02-01T10:32:36.978 DEBUG:boto:Headers: {} 2024-02-01T10:32:36.978 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:32:36.978 DEBUG:boto:Port: 8000 2024-02-01T10:32:36.978 DEBUG:boto:Params: {} 2024-02-01T10:32:36.978 DEBUG:boto:Token: None 2024-02-01T10:32:36.979 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:32:36 GMT /vclfcn-20/ 2024-02-01T10:32:36.979 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:IyQUTnTgZf/3IGqZBaTriRoHCWo= 2024-02-01T10:32:36.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:IyQUTnTgZf/3IGqZBaTriRoHCWo=', 'Content-Length': '0'} 2024-02-01T10:32:36.983 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed70d866615ec6d1-0065bb7344-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:36.983 DEBUG:boto:b'vclfcn-20obj1000falseobj2024-02-01T10:31:50.863Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:32:36.983 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:32:36.984 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:32:36.984 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:32:36.984 DEBUG:boto:path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:32:36.984 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:32:36.984 DEBUG:boto:Method: GET 2024-02-01T10:32:36.984 DEBUG:boto:Path: /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:32:36.984 DEBUG:boto:Data: 2024-02-01T10:32:36.984 DEBUG:boto:Headers: {} 2024-02-01T10:32:36.984 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:32:36.984 DEBUG:boto:Port: 8000 2024-02-01T10:32:36.984 DEBUG:boto:Params: {} 2024-02-01T10:32:36.985 DEBUG:boto:Token: None 2024-02-01T10:32:36.985 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:32:36 GMT /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:32:36.985 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sBImqMmbB9bJaXwEGyxhDO/7lTc= 2024-02-01T10:32:36.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sBImqMmbB9bJaXwEGyxhDO/7lTc=', 'Content-Length': '0'} 2024-02-01T10:32:36.988 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:33 GMT'), ('x-amz-version-id', 'rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c4efc44a1ec3f5e2-0065bb7344-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:36.988 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:32:36.988 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:32:36.988 DEBUG:boto:path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:32:36.988 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:32:36.988 DEBUG:boto:Method: GET 2024-02-01T10:32:36.988 DEBUG:boto:Path: /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:32:36.988 DEBUG:boto:Data: 2024-02-01T10:32:36.989 DEBUG:boto:Headers: {} 2024-02-01T10:32:36.989 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:32:36.989 DEBUG:boto:Port: 8000 2024-02-01T10:32:36.989 DEBUG:boto:Params: {} 2024-02-01T10:32:36.989 DEBUG:boto:Token: None 2024-02-01T10:32:36.989 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:32:36 GMT /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:32:36.989 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sBImqMmbB9bJaXwEGyxhDO/7lTc= 2024-02-01T10:32:36.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sBImqMmbB9bJaXwEGyxhDO/7lTc=', 'Content-Length': '0'} 2024-02-01T10:32:36.992 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:33 GMT'), ('x-amz-version-id', 'rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000eba17751bcbef5e3-0065bb7344-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:36.993 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:32:36.993 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:32:36.993 DEBUG:boto:path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:32:36.993 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:32:36.993 DEBUG:boto:Method: HEAD 2024-02-01T10:32:36.993 DEBUG:boto:Path: /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:32:36.993 DEBUG:boto:Data: 2024-02-01T10:32:36.993 DEBUG:boto:Headers: {} 2024-02-01T10:32:36.993 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:32:36.993 DEBUG:boto:Port: 8000 2024-02-01T10:32:36.994 DEBUG:boto:Params: {} 2024-02-01T10:32:36.994 DEBUG:boto:Token: None 2024-02-01T10:32:36.994 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:32:36 GMT /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:32:36.994 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YuFCU7vAhhaN7AQytVjHKfFr+Z4= 2024-02-01T10:32:36.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YuFCU7vAhhaN7AQytVjHKfFr+Z4=', 'Content-Length': '0'} 2024-02-01T10:32:36.997 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:33 GMT'), ('x-amz-version-id', 'rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000086c9bf35b32bfbc3-0065bb7344-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:36 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:36.997 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:32:36.997 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:32:36.997 DEBUG:boto:path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:32:36.997 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:32:36.997 DEBUG:boto:Method: HEAD 2024-02-01T10:32:36.997 DEBUG:boto:Path: /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:32:36.998 DEBUG:boto:Data: 2024-02-01T10:32:36.998 DEBUG:boto:Headers: {} 2024-02-01T10:32:36.998 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:32:36.998 DEBUG:boto:Port: 8000 2024-02-01T10:32:36.998 DEBUG:boto:Params: {} 2024-02-01T10:32:36.998 DEBUG:boto:Token: None 2024-02-01T10:32:36.998 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:32:36 GMT /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:32:36.998 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YuFCU7vAhhaN7AQytVjHKfFr+Z4= 2024-02-01T10:32:36.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:36 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YuFCU7vAhhaN7AQytVjHKfFr+Z4=', 'Content-Length': '0'} 2024-02-01T10:32:37.001 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:33 GMT'), ('x-amz-version-id', 'rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009596c857661d14c6-0065bb7345-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:37.002 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:32:37.002 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:32:37.002 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:32:37.002 DEBUG:boto:path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:32:37.002 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:32:37.003 DEBUG:boto:Method: GET 2024-02-01T10:32:37.003 DEBUG:boto:Path: /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:32:37.003 DEBUG:boto:Data: 2024-02-01T10:32:37.003 DEBUG:boto:Headers: {} 2024-02-01T10:32:37.003 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:32:37.003 DEBUG:boto:Port: 8000 2024-02-01T10:32:37.003 DEBUG:boto:Params: {} 2024-02-01T10:32:37.003 DEBUG:boto:Token: None 2024-02-01T10:32:37.003 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:32:37 GMT /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:32:37.004 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HXVe9Xw+b4qPDb2xxEi1E5KIOgA= 2024-02-01T10:32:37.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HXVe9Xw+b4qPDb2xxEi1E5KIOgA=', 'Content-Length': '0'} 2024-02-01T10:32:37.007 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:33 GMT'), ('x-amz-version-id', 'rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004e8558fa6f153f0c-0065bb7345-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:37.007 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:32:37.007 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:32:37.007 DEBUG:boto:path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:32:37.007 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:32:37.007 DEBUG:boto:Method: GET 2024-02-01T10:32:37.007 DEBUG:boto:Path: /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:32:37.007 DEBUG:boto:Data: 2024-02-01T10:32:37.007 DEBUG:boto:Headers: {} 2024-02-01T10:32:37.008 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:32:37.008 DEBUG:boto:Port: 8000 2024-02-01T10:32:37.008 DEBUG:boto:Params: {} 2024-02-01T10:32:37.008 DEBUG:boto:Token: None 2024-02-01T10:32:37.008 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:32:37 GMT /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:32:37.008 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HXVe9Xw+b4qPDb2xxEi1E5KIOgA= 2024-02-01T10:32:37.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HXVe9Xw+b4qPDb2xxEi1E5KIOgA=', 'Content-Length': '0'} 2024-02-01T10:32:37.011 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:33 GMT'), ('x-amz-version-id', 'rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fca727c791b0ad1b-0065bb7345-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:37.011 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:32:37.011 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:32:37.011 DEBUG:boto:path=/vclfcn-20/?prefix=obj 2024-02-01T10:32:37.011 DEBUG:boto:auth_path=/vclfcn-20/?prefix=obj 2024-02-01T10:32:37.011 DEBUG:boto:Method: GET 2024-02-01T10:32:37.012 DEBUG:boto:Path: /vclfcn-20/?prefix=obj 2024-02-01T10:32:37.012 DEBUG:boto:Data: 2024-02-01T10:32:37.012 DEBUG:boto:Headers: {} 2024-02-01T10:32:37.012 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:32:37.012 DEBUG:boto:Port: 8000 2024-02-01T10:32:37.012 DEBUG:boto:Params: {} 2024-02-01T10:32:37.012 DEBUG:boto:Token: None 2024-02-01T10:32:37.012 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:32:37 GMT /vclfcn-20/ 2024-02-01T10:32:37.012 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gpzla5sjZ2Cosw4UWO75lal0U6c= 2024-02-01T10:32:37.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gpzla5sjZ2Cosw4UWO75lal0U6c=', 'Content-Length': '0'} 2024-02-01T10:32:37.017 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017a618cdd4b4ef74-0065bb7345-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:37.060 DEBUG:boto:b'vclfcn-20obj1000falseobj2024-02-01T10:31:50.863Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:32:37.061 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:32:37.061 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:32:37.061 DEBUG:boto:path=/vclfcn-20/?prefix=obj 2024-02-01T10:32:37.061 DEBUG:boto:auth_path=/vclfcn-20/?prefix=obj 2024-02-01T10:32:37.061 DEBUG:boto:Method: GET 2024-02-01T10:32:37.061 DEBUG:boto:Path: /vclfcn-20/?prefix=obj 2024-02-01T10:32:37.061 DEBUG:boto:Data: 2024-02-01T10:32:37.061 DEBUG:boto:Headers: {} 2024-02-01T10:32:37.061 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:32:37.061 DEBUG:boto:Port: 8000 2024-02-01T10:32:37.061 DEBUG:boto:Params: {} 2024-02-01T10:32:37.062 DEBUG:boto:Token: None 2024-02-01T10:32:37.062 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:32:37 GMT /vclfcn-20/ 2024-02-01T10:32:37.062 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gpzla5sjZ2Cosw4UWO75lal0U6c= 2024-02-01T10:32:37.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gpzla5sjZ2Cosw4UWO75lal0U6c=', 'Content-Length': '0'} 2024-02-01T10:32:37.066 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001021799b4ae80ba5-0065bb7345-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:37.067 DEBUG:boto:b'vclfcn-20obj1000falseobj2024-02-01T10:31:50.863Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:32:37.067 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:32:37.067 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:32:37.067 DEBUG:boto:Method: HEAD 2024-02-01T10:32:37.067 DEBUG:boto:Path: /vclfcn-20/obj 2024-02-01T10:32:37.067 DEBUG:boto:Data: 2024-02-01T10:32:37.067 DEBUG:boto:Headers: {} 2024-02-01T10:32:37.067 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:32:37.067 DEBUG:boto:Port: 8000 2024-02-01T10:32:37.068 DEBUG:boto:Params: {} 2024-02-01T10:32:37.068 DEBUG:boto:Token: None 2024-02-01T10:32:37.068 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:32:37 GMT /vclfcn-20/obj 2024-02-01T10:32:37.068 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ncizQ1dZ2N6DbjZiRa6Buohj94I= 2024-02-01T10:32:37.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ncizQ1dZ2N6DbjZiRa6Buohj94I=', 'Content-Length': '0'} 2024-02-01T10:32:37.071 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000824f595845aa2ca4-0065bb7345-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:37.071 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:32:37.071 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:32:37.071 DEBUG:boto:Method: HEAD 2024-02-01T10:32:37.071 DEBUG:boto:Path: /vclfcn-20/obj 2024-02-01T10:32:37.071 DEBUG:boto:Data: 2024-02-01T10:32:37.071 DEBUG:boto:Headers: {} 2024-02-01T10:32:37.071 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:32:37.072 DEBUG:boto:Port: 8000 2024-02-01T10:32:37.072 DEBUG:boto:Params: {} 2024-02-01T10:32:37.072 DEBUG:boto:Token: None 2024-02-01T10:32:37.072 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:32:37 GMT /vclfcn-20/obj 2024-02-01T10:32:37.072 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ncizQ1dZ2N6DbjZiRa6Buohj94I= 2024-02-01T10:32:37.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ncizQ1dZ2N6DbjZiRa6Buohj94I=', 'Content-Length': '0'} 2024-02-01T10:32:37.075 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008dd6cb31ea13af7c-0065bb7345-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:37.075 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:32:37.075 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:32:37.076 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:32:37.076 DEBUG:boto:Method: GET 2024-02-01T10:32:37.076 DEBUG:boto:Path: /vclfcn-20/obj 2024-02-01T10:32:37.076 DEBUG:boto:Data: 2024-02-01T10:32:37.076 DEBUG:boto:Headers: {} 2024-02-01T10:32:37.076 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:32:37.076 DEBUG:boto:Port: 8000 2024-02-01T10:32:37.076 DEBUG:boto:Params: {} 2024-02-01T10:32:37.076 DEBUG:boto:Token: None 2024-02-01T10:32:37.076 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:32:37 GMT /vclfcn-20/obj 2024-02-01T10:32:37.077 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QGRpIV1VkSouoJ3RcWzpWTlQtzE= 2024-02-01T10:32:37.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QGRpIV1VkSouoJ3RcWzpWTlQtzE=', 'Content-Length': '0'} 2024-02-01T10:32:37.079 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000165d9976dc9699fd-0065bb7345-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:37.080 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:32:37.080 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:32:37.080 DEBUG:boto:Method: GET 2024-02-01T10:32:37.080 DEBUG:boto:Path: /vclfcn-20/obj 2024-02-01T10:32:37.080 DEBUG:boto:Data: 2024-02-01T10:32:37.080 DEBUG:boto:Headers: {} 2024-02-01T10:32:37.080 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:32:37.080 DEBUG:boto:Port: 8000 2024-02-01T10:32:37.081 DEBUG:boto:Params: {} 2024-02-01T10:32:37.081 DEBUG:boto:Token: None 2024-02-01T10:32:37.081 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:32:37 GMT /vclfcn-20/obj 2024-02-01T10:32:37.081 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QGRpIV1VkSouoJ3RcWzpWTlQtzE= 2024-02-01T10:32:37.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QGRpIV1VkSouoJ3RcWzpWTlQtzE=', 'Content-Length': '0'} 2024-02-01T10:32:37.084 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002a376b6d0ffe831b-0065bb7345-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:37.084 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:32:37.084 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:32:37.084 DEBUG:boto:path=/vclfcn-20/?prefix=obj 2024-02-01T10:32:37.084 DEBUG:boto:auth_path=/vclfcn-20/?prefix=obj 2024-02-01T10:32:37.085 DEBUG:boto:Method: GET 2024-02-01T10:32:37.085 DEBUG:boto:Path: /vclfcn-20/?prefix=obj 2024-02-01T10:32:37.085 DEBUG:boto:Data: 2024-02-01T10:32:37.085 DEBUG:boto:Headers: {} 2024-02-01T10:32:37.085 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:32:37.085 DEBUG:boto:Port: 8000 2024-02-01T10:32:37.085 DEBUG:boto:Params: {} 2024-02-01T10:32:37.085 DEBUG:boto:Token: None 2024-02-01T10:32:37.085 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:32:37 GMT /vclfcn-20/ 2024-02-01T10:32:37.085 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gpzla5sjZ2Cosw4UWO75lal0U6c= 2024-02-01T10:32:37.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gpzla5sjZ2Cosw4UWO75lal0U6c=', 'Content-Length': '0'} 2024-02-01T10:32:37.090 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cfe596ba14822580-0065bb7345-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:37.090 DEBUG:boto:b'vclfcn-20obj1000falseobj2024-02-01T10:31:50.863Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:32:37.091 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:32:37.091 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:32:37.091 DEBUG:boto:path=/vclfcn-20/?prefix=obj 2024-02-01T10:32:37.091 DEBUG:boto:auth_path=/vclfcn-20/?prefix=obj 2024-02-01T10:32:37.091 DEBUG:boto:Method: GET 2024-02-01T10:32:37.091 DEBUG:boto:Path: /vclfcn-20/?prefix=obj 2024-02-01T10:32:37.091 DEBUG:boto:Data: 2024-02-01T10:32:37.091 DEBUG:boto:Headers: {} 2024-02-01T10:32:37.091 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:32:37.091 DEBUG:boto:Port: 8000 2024-02-01T10:32:37.091 DEBUG:boto:Params: {} 2024-02-01T10:32:37.092 DEBUG:boto:Token: None 2024-02-01T10:32:37.092 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:32:37 GMT /vclfcn-20/ 2024-02-01T10:32:37.092 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gpzla5sjZ2Cosw4UWO75lal0U6c= 2024-02-01T10:32:37.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gpzla5sjZ2Cosw4UWO75lal0U6c=', 'Content-Length': '0'} 2024-02-01T10:32:37.097 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d95c3b55870956a8-0065bb7345-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:37.097 DEBUG:boto:b'vclfcn-20obj1000falseobj2024-02-01T10:31:50.863Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:32:37.097 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-20 zones={test-zone2, test-zone1} 2024-02-01T10:32:37.097 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-20 zones={test-zone3, test-zone1} 2024-02-01T10:32:37.097 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:32:37.097 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:32:37.098 DEBUG:boto:Method: HEAD 2024-02-01T10:32:37.098 DEBUG:boto:Path: /vclfcn-20/ 2024-02-01T10:32:37.098 DEBUG:boto:Data: 2024-02-01T10:32:37.098 DEBUG:boto:Headers: {} 2024-02-01T10:32:37.098 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:32:37.098 DEBUG:boto:Port: 8001 2024-02-01T10:32:37.098 DEBUG:boto:Params: {} 2024-02-01T10:32:37.098 DEBUG:boto:Token: None 2024-02-01T10:32:37.098 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:32:37 GMT /vclfcn-20/ 2024-02-01T10:32:37.098 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8F/AECtVgAUuf0GAEf9TGuzODe4= 2024-02-01T10:32:37.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8F/AECtVgAUuf0GAEf9TGuzODe4=', 'Content-Length': '0'} 2024-02-01T10:32:37.103 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ae769b751fd88ef6-0065bb7345-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:32:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:37.103 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:32:37.103 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:32:37.103 DEBUG:boto:Method: HEAD 2024-02-01T10:32:37.103 DEBUG:boto:Path: /vclfcn-20/ 2024-02-01T10:32:37.103 DEBUG:boto:Data: 2024-02-01T10:32:37.103 DEBUG:boto:Headers: {} 2024-02-01T10:32:37.103 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:32:37.103 DEBUG:boto:Port: 8000 2024-02-01T10:32:37.104 DEBUG:boto:Params: {} 2024-02-01T10:32:37.104 DEBUG:boto:Token: None 2024-02-01T10:32:37.104 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:32:37 GMT /vclfcn-20/ 2024-02-01T10:32:37.104 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8F/AECtVgAUuf0GAEf9TGuzODe4= 2024-02-01T10:32:37.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8F/AECtVgAUuf0GAEf9TGuzODe4=', 'Content-Length': '0'} 2024-02-01T10:32:37.108 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001361bd6f9e312642-0065bb7345-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:32:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:37.108 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:32:37.108 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:32:37.109 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:32:37.109 DEBUG:boto:path=/vclfcn-20/?versions&max-keys=999 2024-02-01T10:32:37.109 DEBUG:boto:auth_path=/vclfcn-20/?versions&max-keys=999 2024-02-01T10:32:37.109 DEBUG:boto:Method: GET 2024-02-01T10:32:37.109 DEBUG:boto:Path: /vclfcn-20/?versions&max-keys=999 2024-02-01T10:32:37.109 DEBUG:boto:Data: 2024-02-01T10:32:37.109 DEBUG:boto:Headers: {} 2024-02-01T10:32:37.109 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:32:37.109 DEBUG:boto:Port: 8001 2024-02-01T10:32:37.109 DEBUG:boto:Params: {} 2024-02-01T10:32:37.110 DEBUG:boto:Token: None 2024-02-01T10:32:37.110 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:32:37 GMT /vclfcn-20/?versions 2024-02-01T10:32:37.110 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:y/bVfHK1R8VHMlDmgrlDf0H2tHo= 2024-02-01T10:32:37.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:y/bVfHK1R8VHMlDmgrlDf0H2tHo=', 'Content-Length': '0'} 2024-02-01T10:32:37.115 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a6f71868b50fdeb-0065bb7345-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:37.160 DEBUG:boto:b'vclfcn-20999falseobjnulltrue2024-02-01T10:31:50.863Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjrSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6false2024-02-01T10:28:33.438Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:32:37.161 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:32:37.161 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:32:37.161 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:32:37.161 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:32:37.161 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:32:37.161 DEBUG:boto:path=/vclfcn-20/?versions&max-keys=999 2024-02-01T10:32:37.161 DEBUG:boto:auth_path=/vclfcn-20/?versions&max-keys=999 2024-02-01T10:32:37.161 DEBUG:boto:Method: GET 2024-02-01T10:32:37.161 DEBUG:boto:Path: /vclfcn-20/?versions&max-keys=999 2024-02-01T10:32:37.162 DEBUG:boto:Data: 2024-02-01T10:32:37.162 DEBUG:boto:Headers: {} 2024-02-01T10:32:37.162 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:32:37.162 DEBUG:boto:Port: 8000 2024-02-01T10:32:37.162 DEBUG:boto:Params: {} 2024-02-01T10:32:37.162 DEBUG:boto:Token: None 2024-02-01T10:32:37.162 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:32:37 GMT /vclfcn-20/?versions 2024-02-01T10:32:37.162 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:y/bVfHK1R8VHMlDmgrlDf0H2tHo= 2024-02-01T10:32:37.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:y/bVfHK1R8VHMlDmgrlDf0H2tHo=', 'Content-Length': '0'} 2024-02-01T10:32:37.167 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad620fe2cc7c7fa7-0065bb7345-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:37.167 DEBUG:boto:b'vclfcn-20999falseobjnulltrue2024-02-01T10:31:50.863Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjrSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6false2024-02-01T10:28:33.438Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:32:37.167 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:32:37.167 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:32:37.168 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:32:37.168 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:32:37.168 DEBUG:boto:path=/vclfcn-20/?versions&max-keys=999 2024-02-01T10:32:37.168 DEBUG:boto:auth_path=/vclfcn-20/?versions&max-keys=999 2024-02-01T10:32:37.168 DEBUG:boto:Method: GET 2024-02-01T10:32:37.168 DEBUG:boto:Path: /vclfcn-20/?versions&max-keys=999 2024-02-01T10:32:37.168 DEBUG:boto:Data: 2024-02-01T10:32:37.168 DEBUG:boto:Headers: {} 2024-02-01T10:32:37.168 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:32:37.169 DEBUG:boto:Port: 8001 2024-02-01T10:32:37.169 DEBUG:boto:Params: {} 2024-02-01T10:32:37.169 DEBUG:boto:Token: None 2024-02-01T10:32:37.169 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:32:37 GMT /vclfcn-20/?versions 2024-02-01T10:32:37.169 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:y/bVfHK1R8VHMlDmgrlDf0H2tHo= 2024-02-01T10:32:37.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:y/bVfHK1R8VHMlDmgrlDf0H2tHo=', 'Content-Length': '0'} 2024-02-01T10:32:37.174 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009eaa65cf8f5d4c6b-0065bb7345-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:37.174 DEBUG:boto:b'vclfcn-20999falseobjnulltrue2024-02-01T10:31:50.863Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjrSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6false2024-02-01T10:28:33.438Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:32:37.174 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:32:37.174 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:32:37.174 DEBUG:boto:path=/vclfcn-20/?versions&max-keys=999 2024-02-01T10:32:37.175 DEBUG:boto:auth_path=/vclfcn-20/?versions&max-keys=999 2024-02-01T10:32:37.175 DEBUG:boto:Method: GET 2024-02-01T10:32:37.175 DEBUG:boto:Path: /vclfcn-20/?versions&max-keys=999 2024-02-01T10:32:37.175 DEBUG:boto:Data: 2024-02-01T10:32:37.175 DEBUG:boto:Headers: {} 2024-02-01T10:32:37.175 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:32:37.175 DEBUG:boto:Port: 8000 2024-02-01T10:32:37.175 DEBUG:boto:Params: {} 2024-02-01T10:32:37.175 DEBUG:boto:Token: None 2024-02-01T10:32:37.175 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:32:37 GMT /vclfcn-20/?versions 2024-02-01T10:32:37.176 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:y/bVfHK1R8VHMlDmgrlDf0H2tHo= 2024-02-01T10:32:37.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:y/bVfHK1R8VHMlDmgrlDf0H2tHo=', 'Content-Length': '0'} 2024-02-01T10:32:37.180 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000049d9f725627af3b0-0065bb7345-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:37.228 DEBUG:boto:b'vclfcn-20999falseobjnulltrue2024-02-01T10:31:50.863Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjrSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6false2024-02-01T10:28:33.438Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:32:37.229 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:32:37.229 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:32:37.229 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:32:37.229 DEBUG:boto:path=/vclfcn-20/obj?versionId=null 2024-02-01T10:32:37.229 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=null 2024-02-01T10:32:37.229 DEBUG:boto:Method: GET 2024-02-01T10:32:37.229 DEBUG:boto:Path: /vclfcn-20/obj?versionId=null 2024-02-01T10:32:37.229 DEBUG:boto:Data: 2024-02-01T10:32:37.229 DEBUG:boto:Headers: {} 2024-02-01T10:32:37.230 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:32:37.230 DEBUG:boto:Port: 8001 2024-02-01T10:32:37.230 DEBUG:boto:Params: {} 2024-02-01T10:32:37.230 DEBUG:boto:Token: None 2024-02-01T10:32:37.230 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:32:37 GMT /vclfcn-20/obj?versionId=null 2024-02-01T10:32:37.230 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vB14Fl2xTgJ8qGpb5alw2hrdD2w= 2024-02-01T10:32:37.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vB14Fl2xTgJ8qGpb5alw2hrdD2w=', 'Content-Length': '0'} 2024-02-01T10:32:37.233 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003fa24c9c721eff4e-0065bb7345-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:37.233 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:32:37.234 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:32:37.234 DEBUG:boto:path=/vclfcn-20/obj?versionId=null 2024-02-01T10:32:37.234 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=null 2024-02-01T10:32:37.234 DEBUG:boto:Method: GET 2024-02-01T10:32:37.234 DEBUG:boto:Path: /vclfcn-20/obj?versionId=null 2024-02-01T10:32:37.234 DEBUG:boto:Data: 2024-02-01T10:32:37.234 DEBUG:boto:Headers: {} 2024-02-01T10:32:37.234 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:32:37.234 DEBUG:boto:Port: 8000 2024-02-01T10:32:37.234 DEBUG:boto:Params: {} 2024-02-01T10:32:37.234 DEBUG:boto:Token: None 2024-02-01T10:32:37.235 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:32:37 GMT /vclfcn-20/obj?versionId=null 2024-02-01T10:32:37.235 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vB14Fl2xTgJ8qGpb5alw2hrdD2w= 2024-02-01T10:32:37.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vB14Fl2xTgJ8qGpb5alw2hrdD2w=', 'Content-Length': '0'} 2024-02-01T10:32:37.238 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c2a17d29e5bdc29f-0065bb7345-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:37.238 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:32:37.238 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:32:37.239 DEBUG:boto:path=/vclfcn-20/obj?versionId=null 2024-02-01T10:32:37.239 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=null 2024-02-01T10:32:37.239 DEBUG:boto:Method: HEAD 2024-02-01T10:32:37.239 DEBUG:boto:Path: /vclfcn-20/obj?versionId=null 2024-02-01T10:32:37.239 DEBUG:boto:Data: 2024-02-01T10:32:37.239 DEBUG:boto:Headers: {} 2024-02-01T10:32:37.239 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:32:37.239 DEBUG:boto:Port: 8001 2024-02-01T10:32:37.239 DEBUG:boto:Params: {} 2024-02-01T10:32:37.239 DEBUG:boto:Token: None 2024-02-01T10:32:37.240 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:32:37 GMT /vclfcn-20/obj?versionId=null 2024-02-01T10:32:37.240 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:yf5HYTZGPYih//5iQ/cx5mO5w1s= 2024-02-01T10:32:37.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:yf5HYTZGPYih//5iQ/cx5mO5w1s=', 'Content-Length': '0'} 2024-02-01T10:32:37.243 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000599a77d371f94024-0065bb7345-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:37.243 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:32:37.243 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:32:37.244 DEBUG:boto:path=/vclfcn-20/obj?versionId=null 2024-02-01T10:32:37.244 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=null 2024-02-01T10:32:37.244 DEBUG:boto:Method: HEAD 2024-02-01T10:32:37.244 DEBUG:boto:Path: /vclfcn-20/obj?versionId=null 2024-02-01T10:32:37.244 DEBUG:boto:Data: 2024-02-01T10:32:37.244 DEBUG:boto:Headers: {} 2024-02-01T10:32:37.244 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:32:37.244 DEBUG:boto:Port: 8000 2024-02-01T10:32:37.244 DEBUG:boto:Params: {} 2024-02-01T10:32:37.244 DEBUG:boto:Token: None 2024-02-01T10:32:37.245 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:32:37 GMT /vclfcn-20/obj?versionId=null 2024-02-01T10:32:37.245 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:yf5HYTZGPYih//5iQ/cx5mO5w1s= 2024-02-01T10:32:37.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:yf5HYTZGPYih//5iQ/cx5mO5w1s=', 'Content-Length': '0'} 2024-02-01T10:32:37.247 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f125a64678fbed5e-0065bb7345-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:37.248 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:32:37.248 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:32:37.248 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:32:37.248 DEBUG:boto:path=/vclfcn-20/obj?versionId=null 2024-02-01T10:32:37.248 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=null 2024-02-01T10:32:37.248 DEBUG:boto:Method: GET 2024-02-01T10:32:37.248 DEBUG:boto:Path: /vclfcn-20/obj?versionId=null 2024-02-01T10:32:37.248 DEBUG:boto:Data: 2024-02-01T10:32:37.249 DEBUG:boto:Headers: {} 2024-02-01T10:32:37.249 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:32:37.249 DEBUG:boto:Port: 8001 2024-02-01T10:32:37.249 DEBUG:boto:Params: {} 2024-02-01T10:32:37.249 DEBUG:boto:Token: None 2024-02-01T10:32:37.249 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:32:37 GMT /vclfcn-20/obj?versionId=null 2024-02-01T10:32:37.249 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vB14Fl2xTgJ8qGpb5alw2hrdD2w= 2024-02-01T10:32:37.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vB14Fl2xTgJ8qGpb5alw2hrdD2w=', 'Content-Length': '0'} 2024-02-01T10:32:37.252 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b3a755ac0f388d2f-0065bb7345-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:37.252 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:32:37.252 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:32:37.253 DEBUG:boto:path=/vclfcn-20/obj?versionId=null 2024-02-01T10:32:37.253 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=null 2024-02-01T10:32:37.253 DEBUG:boto:Method: GET 2024-02-01T10:32:37.253 DEBUG:boto:Path: /vclfcn-20/obj?versionId=null 2024-02-01T10:32:37.253 DEBUG:boto:Data: 2024-02-01T10:32:37.253 DEBUG:boto:Headers: {} 2024-02-01T10:32:37.253 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:32:37.253 DEBUG:boto:Port: 8000 2024-02-01T10:32:37.253 DEBUG:boto:Params: {} 2024-02-01T10:32:37.253 DEBUG:boto:Token: None 2024-02-01T10:32:37.254 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:32:37 GMT /vclfcn-20/obj?versionId=null 2024-02-01T10:32:37.254 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vB14Fl2xTgJ8qGpb5alw2hrdD2w= 2024-02-01T10:32:37.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vB14Fl2xTgJ8qGpb5alw2hrdD2w=', 'Content-Length': '0'} 2024-02-01T10:32:37.256 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000080e8dd8e6cb204d0-0065bb7345-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:37.257 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:32:37.257 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:32:37.257 DEBUG:boto:path=/vclfcn-20/?prefix=obj 2024-02-01T10:32:37.257 DEBUG:boto:auth_path=/vclfcn-20/?prefix=obj 2024-02-01T10:32:37.257 DEBUG:boto:Method: GET 2024-02-01T10:32:37.257 DEBUG:boto:Path: /vclfcn-20/?prefix=obj 2024-02-01T10:32:37.257 DEBUG:boto:Data: 2024-02-01T10:32:37.257 DEBUG:boto:Headers: {} 2024-02-01T10:32:37.257 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:32:37.258 DEBUG:boto:Port: 8001 2024-02-01T10:32:37.258 DEBUG:boto:Params: {} 2024-02-01T10:32:37.258 DEBUG:boto:Token: None 2024-02-01T10:32:37.258 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:32:37 GMT /vclfcn-20/ 2024-02-01T10:32:37.258 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gpzla5sjZ2Cosw4UWO75lal0U6c= 2024-02-01T10:32:37.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gpzla5sjZ2Cosw4UWO75lal0U6c=', 'Content-Length': '0'} 2024-02-01T10:32:37.262 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab2ddb9cf2a58f28-0065bb7345-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:37.304 DEBUG:boto:b'vclfcn-20obj1000falseobj2024-02-01T10:31:50.863Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:32:37.305 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:32:37.305 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:32:37.305 DEBUG:boto:path=/vclfcn-20/?prefix=obj 2024-02-01T10:32:37.305 DEBUG:boto:auth_path=/vclfcn-20/?prefix=obj 2024-02-01T10:32:37.306 DEBUG:boto:Method: GET 2024-02-01T10:32:37.306 DEBUG:boto:Path: /vclfcn-20/?prefix=obj 2024-02-01T10:32:37.306 DEBUG:boto:Data: 2024-02-01T10:32:37.306 DEBUG:boto:Headers: {} 2024-02-01T10:32:37.306 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:32:37.306 DEBUG:boto:Port: 8000 2024-02-01T10:32:37.306 DEBUG:boto:Params: {} 2024-02-01T10:32:37.306 DEBUG:boto:Token: None 2024-02-01T10:32:37.306 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:32:37 GMT /vclfcn-20/ 2024-02-01T10:32:37.307 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gpzla5sjZ2Cosw4UWO75lal0U6c= 2024-02-01T10:32:37.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gpzla5sjZ2Cosw4UWO75lal0U6c=', 'Content-Length': '0'} 2024-02-01T10:32:37.311 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d380e2bb974ae622-0065bb7345-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:37.312 DEBUG:boto:b'vclfcn-20obj1000falseobj2024-02-01T10:31:50.863Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:32:37.312 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:32:37.312 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:32:37.312 DEBUG:boto:Method: HEAD 2024-02-01T10:32:37.312 DEBUG:boto:Path: /vclfcn-20/obj 2024-02-01T10:32:37.312 DEBUG:boto:Data: 2024-02-01T10:32:37.313 DEBUG:boto:Headers: {} 2024-02-01T10:32:37.313 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:32:37.313 DEBUG:boto:Port: 8001 2024-02-01T10:32:37.313 DEBUG:boto:Params: {} 2024-02-01T10:32:37.313 DEBUG:boto:Token: None 2024-02-01T10:32:37.313 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:32:37 GMT /vclfcn-20/obj 2024-02-01T10:32:37.313 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ncizQ1dZ2N6DbjZiRa6Buohj94I= 2024-02-01T10:32:37.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ncizQ1dZ2N6DbjZiRa6Buohj94I=', 'Content-Length': '0'} 2024-02-01T10:32:37.316 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005ae6244c0b1ac5cd-0065bb7345-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:37.316 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:32:37.316 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:32:37.316 DEBUG:boto:Method: HEAD 2024-02-01T10:32:37.316 DEBUG:boto:Path: /vclfcn-20/obj 2024-02-01T10:32:37.317 DEBUG:boto:Data: 2024-02-01T10:32:37.317 DEBUG:boto:Headers: {} 2024-02-01T10:32:37.317 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:32:37.317 DEBUG:boto:Port: 8000 2024-02-01T10:32:37.317 DEBUG:boto:Params: {} 2024-02-01T10:32:37.317 DEBUG:boto:Token: None 2024-02-01T10:32:37.317 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:32:37 GMT /vclfcn-20/obj 2024-02-01T10:32:37.317 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ncizQ1dZ2N6DbjZiRa6Buohj94I= 2024-02-01T10:32:37.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ncizQ1dZ2N6DbjZiRa6Buohj94I=', 'Content-Length': '0'} 2024-02-01T10:32:37.320 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005efd9019360a1be7-0065bb7345-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:37.320 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:32:37.320 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:32:37.320 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:32:37.320 DEBUG:boto:Method: GET 2024-02-01T10:32:37.321 DEBUG:boto:Path: /vclfcn-20/obj 2024-02-01T10:32:37.321 DEBUG:boto:Data: 2024-02-01T10:32:37.321 DEBUG:boto:Headers: {} 2024-02-01T10:32:37.321 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:32:37.321 DEBUG:boto:Port: 8001 2024-02-01T10:32:37.321 DEBUG:boto:Params: {} 2024-02-01T10:32:37.321 DEBUG:boto:Token: None 2024-02-01T10:32:37.321 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:32:37 GMT /vclfcn-20/obj 2024-02-01T10:32:37.321 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QGRpIV1VkSouoJ3RcWzpWTlQtzE= 2024-02-01T10:32:37.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QGRpIV1VkSouoJ3RcWzpWTlQtzE=', 'Content-Length': '0'} 2024-02-01T10:32:37.324 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a9df361653359fca-0065bb7345-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:37.325 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:32:37.325 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:32:37.325 DEBUG:boto:Method: GET 2024-02-01T10:32:37.325 DEBUG:boto:Path: /vclfcn-20/obj 2024-02-01T10:32:37.325 DEBUG:boto:Data: 2024-02-01T10:32:37.325 DEBUG:boto:Headers: {} 2024-02-01T10:32:37.325 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:32:37.325 DEBUG:boto:Port: 8000 2024-02-01T10:32:37.326 DEBUG:boto:Params: {} 2024-02-01T10:32:37.326 DEBUG:boto:Token: None 2024-02-01T10:32:37.326 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:32:37 GMT /vclfcn-20/obj 2024-02-01T10:32:37.326 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QGRpIV1VkSouoJ3RcWzpWTlQtzE= 2024-02-01T10:32:37.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QGRpIV1VkSouoJ3RcWzpWTlQtzE=', 'Content-Length': '0'} 2024-02-01T10:32:37.329 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000c5d528954368694-0065bb7345-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:37.329 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:32:37.329 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:32:37.329 DEBUG:boto:path=/vclfcn-20/?prefix=obj 2024-02-01T10:32:37.330 DEBUG:boto:auth_path=/vclfcn-20/?prefix=obj 2024-02-01T10:32:37.330 DEBUG:boto:Method: GET 2024-02-01T10:32:37.330 DEBUG:boto:Path: /vclfcn-20/?prefix=obj 2024-02-01T10:32:37.330 DEBUG:boto:Data: 2024-02-01T10:32:37.330 DEBUG:boto:Headers: {} 2024-02-01T10:32:37.330 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:32:37.330 DEBUG:boto:Port: 8001 2024-02-01T10:32:37.330 DEBUG:boto:Params: {} 2024-02-01T10:32:37.330 DEBUG:boto:Token: None 2024-02-01T10:32:37.330 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:32:37 GMT /vclfcn-20/ 2024-02-01T10:32:37.331 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gpzla5sjZ2Cosw4UWO75lal0U6c= 2024-02-01T10:32:37.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gpzla5sjZ2Cosw4UWO75lal0U6c=', 'Content-Length': '0'} 2024-02-01T10:32:37.335 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5ca07d1ae6a247c-0065bb7345-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:37.335 DEBUG:boto:b'vclfcn-20obj1000falseobj2024-02-01T10:31:50.863Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:32:37.335 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:32:37.335 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:32:37.335 DEBUG:boto:path=/vclfcn-20/?prefix=obj 2024-02-01T10:32:37.335 DEBUG:boto:auth_path=/vclfcn-20/?prefix=obj 2024-02-01T10:32:37.336 DEBUG:boto:Method: GET 2024-02-01T10:32:37.336 DEBUG:boto:Path: /vclfcn-20/?prefix=obj 2024-02-01T10:32:37.336 DEBUG:boto:Data: 2024-02-01T10:32:37.336 DEBUG:boto:Headers: {} 2024-02-01T10:32:37.336 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:32:37.336 DEBUG:boto:Port: 8000 2024-02-01T10:32:37.336 DEBUG:boto:Params: {} 2024-02-01T10:32:37.336 DEBUG:boto:Token: None 2024-02-01T10:32:37.336 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:32:37 GMT /vclfcn-20/ 2024-02-01T10:32:37.337 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gpzla5sjZ2Cosw4UWO75lal0U6c= 2024-02-01T10:32:37.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gpzla5sjZ2Cosw4UWO75lal0U6c=', 'Content-Length': '0'} 2024-02-01T10:32:37.341 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028ab58e9f0081ea7-0065bb7345-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:37.384 DEBUG:boto:b'vclfcn-20obj1000falseobj2024-02-01T10:31:50.863Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:32:37.384 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:32:37.385 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:32:37.385 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:32:37.385 DEBUG:boto:path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:32:37.385 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:32:37.385 DEBUG:boto:Method: GET 2024-02-01T10:32:37.385 DEBUG:boto:Path: /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:32:37.385 DEBUG:boto:Data: 2024-02-01T10:32:37.385 DEBUG:boto:Headers: {} 2024-02-01T10:32:37.386 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:32:37.386 DEBUG:boto:Port: 8001 2024-02-01T10:32:37.386 DEBUG:boto:Params: {} 2024-02-01T10:32:37.386 DEBUG:boto:Token: None 2024-02-01T10:32:37.386 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:32:37 GMT /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:32:37.386 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HXVe9Xw+b4qPDb2xxEi1E5KIOgA= 2024-02-01T10:32:37.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HXVe9Xw+b4qPDb2xxEi1E5KIOgA=', 'Content-Length': '0'} 2024-02-01T10:32:37.389 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:33 GMT'), ('x-amz-version-id', 'rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000068476b99b0b65a13-0065bb7345-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:37.390 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:32:37.390 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:32:37.390 DEBUG:boto:path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:32:37.390 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:32:37.390 DEBUG:boto:Method: GET 2024-02-01T10:32:37.390 DEBUG:boto:Path: /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:32:37.390 DEBUG:boto:Data: 2024-02-01T10:32:37.390 DEBUG:boto:Headers: {} 2024-02-01T10:32:37.390 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:32:37.390 DEBUG:boto:Port: 8000 2024-02-01T10:32:37.391 DEBUG:boto:Params: {} 2024-02-01T10:32:37.391 DEBUG:boto:Token: None 2024-02-01T10:32:37.391 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:32:37 GMT /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:32:37.391 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HXVe9Xw+b4qPDb2xxEi1E5KIOgA= 2024-02-01T10:32:37.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HXVe9Xw+b4qPDb2xxEi1E5KIOgA=', 'Content-Length': '0'} 2024-02-01T10:32:37.394 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:33 GMT'), ('x-amz-version-id', 'rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005db140a8c954928d-0065bb7345-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:37.394 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:32:37.394 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:32:37.394 DEBUG:boto:path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:32:37.394 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:32:37.394 DEBUG:boto:Method: HEAD 2024-02-01T10:32:37.395 DEBUG:boto:Path: /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:32:37.395 DEBUG:boto:Data: 2024-02-01T10:32:37.395 DEBUG:boto:Headers: {} 2024-02-01T10:32:37.395 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:32:37.395 DEBUG:boto:Port: 8001 2024-02-01T10:32:37.395 DEBUG:boto:Params: {} 2024-02-01T10:32:37.395 DEBUG:boto:Token: None 2024-02-01T10:32:37.395 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:32:37 GMT /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:32:37.395 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GvhXjAtxW7kerCSj23FNi63ciQo= 2024-02-01T10:32:37.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GvhXjAtxW7kerCSj23FNi63ciQo=', 'Content-Length': '0'} 2024-02-01T10:32:37.398 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:33 GMT'), ('x-amz-version-id', 'rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000004637a7a2c6e1b6a-0065bb7345-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:37.399 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:32:37.399 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:32:37.399 DEBUG:boto:path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:32:37.399 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:32:37.399 DEBUG:boto:Method: HEAD 2024-02-01T10:32:37.399 DEBUG:boto:Path: /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:32:37.399 DEBUG:boto:Data: 2024-02-01T10:32:37.399 DEBUG:boto:Headers: {} 2024-02-01T10:32:37.400 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:32:37.400 DEBUG:boto:Port: 8000 2024-02-01T10:32:37.400 DEBUG:boto:Params: {} 2024-02-01T10:32:37.400 DEBUG:boto:Token: None 2024-02-01T10:32:37.400 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:32:37 GMT /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:32:37.400 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GvhXjAtxW7kerCSj23FNi63ciQo= 2024-02-01T10:32:37.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GvhXjAtxW7kerCSj23FNi63ciQo=', 'Content-Length': '0'} 2024-02-01T10:32:37.403 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:33 GMT'), ('x-amz-version-id', 'rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e42e3aed1d880bb2-0065bb7345-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:37.403 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:32:37.403 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:32:37.404 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:32:37.404 DEBUG:boto:path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:32:37.404 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:32:37.404 DEBUG:boto:Method: GET 2024-02-01T10:32:37.404 DEBUG:boto:Path: /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:32:37.404 DEBUG:boto:Data: 2024-02-01T10:32:37.404 DEBUG:boto:Headers: {} 2024-02-01T10:32:37.404 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:32:37.404 DEBUG:boto:Port: 8001 2024-02-01T10:32:37.405 DEBUG:boto:Params: {} 2024-02-01T10:32:37.405 DEBUG:boto:Token: None 2024-02-01T10:32:37.405 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:32:37 GMT /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:32:37.405 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HXVe9Xw+b4qPDb2xxEi1E5KIOgA= 2024-02-01T10:32:37.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HXVe9Xw+b4qPDb2xxEi1E5KIOgA=', 'Content-Length': '0'} 2024-02-01T10:32:37.408 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:33 GMT'), ('x-amz-version-id', 'rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000626574ba5c1098ef-0065bb7345-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:37.409 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:32:37.409 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:32:37.409 DEBUG:boto:path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:32:37.409 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:32:37.409 DEBUG:boto:Method: GET 2024-02-01T10:32:37.409 DEBUG:boto:Path: /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:32:37.409 DEBUG:boto:Data: 2024-02-01T10:32:37.409 DEBUG:boto:Headers: {} 2024-02-01T10:32:37.410 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:32:37.410 DEBUG:boto:Port: 8000 2024-02-01T10:32:37.410 DEBUG:boto:Params: {} 2024-02-01T10:32:37.410 DEBUG:boto:Token: None 2024-02-01T10:32:37.410 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:32:37 GMT /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:32:37.410 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HXVe9Xw+b4qPDb2xxEi1E5KIOgA= 2024-02-01T10:32:37.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HXVe9Xw+b4qPDb2xxEi1E5KIOgA=', 'Content-Length': '0'} 2024-02-01T10:32:37.413 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:33 GMT'), ('x-amz-version-id', 'rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c634654e1aff8e57-0065bb7345-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:37.413 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:32:37.414 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:32:37.414 DEBUG:boto:path=/vclfcn-20/?prefix=obj 2024-02-01T10:32:37.414 DEBUG:boto:auth_path=/vclfcn-20/?prefix=obj 2024-02-01T10:32:37.414 DEBUG:boto:Method: GET 2024-02-01T10:32:37.414 DEBUG:boto:Path: /vclfcn-20/?prefix=obj 2024-02-01T10:32:37.414 DEBUG:boto:Data: 2024-02-01T10:32:37.414 DEBUG:boto:Headers: {} 2024-02-01T10:32:37.414 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:32:37.414 DEBUG:boto:Port: 8001 2024-02-01T10:32:37.414 DEBUG:boto:Params: {} 2024-02-01T10:32:37.415 DEBUG:boto:Token: None 2024-02-01T10:32:37.415 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:32:37 GMT /vclfcn-20/ 2024-02-01T10:32:37.415 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gpzla5sjZ2Cosw4UWO75lal0U6c= 2024-02-01T10:32:37.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gpzla5sjZ2Cosw4UWO75lal0U6c=', 'Content-Length': '0'} 2024-02-01T10:32:37.419 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d06ad70aeb11037-0065bb7345-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:37.464 DEBUG:boto:b'vclfcn-20obj1000falseobj2024-02-01T10:31:50.863Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:32:37.465 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:32:37.465 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:32:37.465 DEBUG:boto:path=/vclfcn-20/?prefix=obj 2024-02-01T10:32:37.465 DEBUG:boto:auth_path=/vclfcn-20/?prefix=obj 2024-02-01T10:32:37.465 DEBUG:boto:Method: GET 2024-02-01T10:32:37.465 DEBUG:boto:Path: /vclfcn-20/?prefix=obj 2024-02-01T10:32:37.465 DEBUG:boto:Data: 2024-02-01T10:32:37.466 DEBUG:boto:Headers: {} 2024-02-01T10:32:37.466 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:32:37.466 DEBUG:boto:Port: 8000 2024-02-01T10:32:37.466 DEBUG:boto:Params: {} 2024-02-01T10:32:37.466 DEBUG:boto:Token: None 2024-02-01T10:32:37.466 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:32:37 GMT /vclfcn-20/ 2024-02-01T10:32:37.467 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gpzla5sjZ2Cosw4UWO75lal0U6c= 2024-02-01T10:32:37.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gpzla5sjZ2Cosw4UWO75lal0U6c=', 'Content-Length': '0'} 2024-02-01T10:32:37.472 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000926b9ec1655b4a2a-0065bb7345-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:37.472 DEBUG:boto:b'vclfcn-20obj1000falseobj2024-02-01T10:31:50.863Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:32:37.472 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:32:37.472 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:32:37.473 DEBUG:boto:Method: HEAD 2024-02-01T10:32:37.473 DEBUG:boto:Path: /vclfcn-20/obj 2024-02-01T10:32:37.473 DEBUG:boto:Data: 2024-02-01T10:32:37.473 DEBUG:boto:Headers: {} 2024-02-01T10:32:37.473 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:32:37.473 DEBUG:boto:Port: 8001 2024-02-01T10:32:37.473 DEBUG:boto:Params: {} 2024-02-01T10:32:37.473 DEBUG:boto:Token: None 2024-02-01T10:32:37.473 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:32:37 GMT /vclfcn-20/obj 2024-02-01T10:32:37.473 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ncizQ1dZ2N6DbjZiRa6Buohj94I= 2024-02-01T10:32:37.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ncizQ1dZ2N6DbjZiRa6Buohj94I=', 'Content-Length': '0'} 2024-02-01T10:32:37.476 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006f988ffbd3c94d8e-0065bb7345-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:37.476 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:32:37.477 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:32:37.477 DEBUG:boto:Method: HEAD 2024-02-01T10:32:37.477 DEBUG:boto:Path: /vclfcn-20/obj 2024-02-01T10:32:37.477 DEBUG:boto:Data: 2024-02-01T10:32:37.477 DEBUG:boto:Headers: {} 2024-02-01T10:32:37.477 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:32:37.477 DEBUG:boto:Port: 8000 2024-02-01T10:32:37.477 DEBUG:boto:Params: {} 2024-02-01T10:32:37.478 DEBUG:boto:Token: None 2024-02-01T10:32:37.478 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:32:37 GMT /vclfcn-20/obj 2024-02-01T10:32:37.478 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ncizQ1dZ2N6DbjZiRa6Buohj94I= 2024-02-01T10:32:37.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ncizQ1dZ2N6DbjZiRa6Buohj94I=', 'Content-Length': '0'} 2024-02-01T10:32:37.481 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000067fc8c9bcaf5ea5a-0065bb7345-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:37.481 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:32:37.481 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:32:37.481 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:32:37.481 DEBUG:boto:Method: GET 2024-02-01T10:32:37.481 DEBUG:boto:Path: /vclfcn-20/obj 2024-02-01T10:32:37.482 DEBUG:boto:Data: 2024-02-01T10:32:37.482 DEBUG:boto:Headers: {} 2024-02-01T10:32:37.482 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:32:37.482 DEBUG:boto:Port: 8001 2024-02-01T10:32:37.482 DEBUG:boto:Params: {} 2024-02-01T10:32:37.482 DEBUG:boto:Token: None 2024-02-01T10:32:37.482 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:32:37 GMT /vclfcn-20/obj 2024-02-01T10:32:37.482 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QGRpIV1VkSouoJ3RcWzpWTlQtzE= 2024-02-01T10:32:37.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QGRpIV1VkSouoJ3RcWzpWTlQtzE=', 'Content-Length': '0'} 2024-02-01T10:32:37.485 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003c44c8299e501b42-0065bb7345-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:37.485 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:32:37.486 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:32:37.486 DEBUG:boto:Method: GET 2024-02-01T10:32:37.486 DEBUG:boto:Path: /vclfcn-20/obj 2024-02-01T10:32:37.486 DEBUG:boto:Data: 2024-02-01T10:32:37.486 DEBUG:boto:Headers: {} 2024-02-01T10:32:37.486 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:32:37.486 DEBUG:boto:Port: 8000 2024-02-01T10:32:37.486 DEBUG:boto:Params: {} 2024-02-01T10:32:37.487 DEBUG:boto:Token: None 2024-02-01T10:32:37.487 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:32:37 GMT /vclfcn-20/obj 2024-02-01T10:32:37.487 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QGRpIV1VkSouoJ3RcWzpWTlQtzE= 2024-02-01T10:32:37.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QGRpIV1VkSouoJ3RcWzpWTlQtzE=', 'Content-Length': '0'} 2024-02-01T10:32:37.490 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a8f4da66a28c4f9f-0065bb7345-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:37.490 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:32:37.490 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:32:37.490 DEBUG:boto:path=/vclfcn-20/?prefix=obj 2024-02-01T10:32:37.491 DEBUG:boto:auth_path=/vclfcn-20/?prefix=obj 2024-02-01T10:32:37.491 DEBUG:boto:Method: GET 2024-02-01T10:32:37.491 DEBUG:boto:Path: /vclfcn-20/?prefix=obj 2024-02-01T10:32:37.491 DEBUG:boto:Data: 2024-02-01T10:32:37.491 DEBUG:boto:Headers: {} 2024-02-01T10:32:37.491 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:32:37.491 DEBUG:boto:Port: 8001 2024-02-01T10:32:37.491 DEBUG:boto:Params: {} 2024-02-01T10:32:37.492 DEBUG:boto:Token: None 2024-02-01T10:32:37.492 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:32:37 GMT /vclfcn-20/ 2024-02-01T10:32:37.492 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gpzla5sjZ2Cosw4UWO75lal0U6c= 2024-02-01T10:32:37.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gpzla5sjZ2Cosw4UWO75lal0U6c=', 'Content-Length': '0'} 2024-02-01T10:32:37.496 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f08ba2edfcc0abab-0065bb7345-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:37.545 DEBUG:boto:b'vclfcn-20obj1000falseobj2024-02-01T10:31:50.863Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:32:37.545 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:32:37.545 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:32:37.545 DEBUG:boto:path=/vclfcn-20/?prefix=obj 2024-02-01T10:32:37.546 DEBUG:boto:auth_path=/vclfcn-20/?prefix=obj 2024-02-01T10:32:37.546 DEBUG:boto:Method: GET 2024-02-01T10:32:37.546 DEBUG:boto:Path: /vclfcn-20/?prefix=obj 2024-02-01T10:32:37.546 DEBUG:boto:Data: 2024-02-01T10:32:37.546 DEBUG:boto:Headers: {} 2024-02-01T10:32:37.546 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:32:37.546 DEBUG:boto:Port: 8000 2024-02-01T10:32:37.546 DEBUG:boto:Params: {} 2024-02-01T10:32:37.547 DEBUG:boto:Token: None 2024-02-01T10:32:37.547 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:32:37 GMT /vclfcn-20/ 2024-02-01T10:32:37.547 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gpzla5sjZ2Cosw4UWO75lal0U6c= 2024-02-01T10:32:37.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gpzla5sjZ2Cosw4UWO75lal0U6c=', 'Content-Length': '0'} 2024-02-01T10:32:37.552 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000316519ed837ebd1d-0065bb7345-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:37.552 DEBUG:boto:b'vclfcn-20obj1000falseobj2024-02-01T10:31:50.863Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:32:37.552 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-20 zones={test-zone3, test-zone1} 2024-02-01T10:32:37.552 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-20 zones={test-zone3, test-zone2} 2024-02-01T10:32:37.553 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:32:37.553 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:32:37.553 DEBUG:boto:Method: HEAD 2024-02-01T10:32:37.553 DEBUG:boto:Path: /vclfcn-20/ 2024-02-01T10:32:37.553 DEBUG:boto:Data: 2024-02-01T10:32:37.553 DEBUG:boto:Headers: {} 2024-02-01T10:32:37.553 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:32:37.553 DEBUG:boto:Port: 8001 2024-02-01T10:32:37.553 DEBUG:boto:Params: {} 2024-02-01T10:32:37.554 DEBUG:boto:Token: None 2024-02-01T10:32:37.554 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:32:37 GMT /vclfcn-20/ 2024-02-01T10:32:37.554 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8F/AECtVgAUuf0GAEf9TGuzODe4= 2024-02-01T10:32:37.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8F/AECtVgAUuf0GAEf9TGuzODe4=', 'Content-Length': '0'} 2024-02-01T10:32:37.558 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009c31c82c9fd54fc4-0065bb7345-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:32:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:37.558 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:32:37.558 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:32:37.559 DEBUG:boto:Method: HEAD 2024-02-01T10:32:37.559 DEBUG:boto:Path: /vclfcn-20/ 2024-02-01T10:32:37.559 DEBUG:boto:Data: 2024-02-01T10:32:37.559 DEBUG:boto:Headers: {} 2024-02-01T10:32:37.559 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:32:37.559 DEBUG:boto:Port: 8000 2024-02-01T10:32:37.559 DEBUG:boto:Params: {} 2024-02-01T10:32:37.559 DEBUG:boto:Token: None 2024-02-01T10:32:37.559 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:32:37 GMT /vclfcn-20/ 2024-02-01T10:32:37.560 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8F/AECtVgAUuf0GAEf9TGuzODe4= 2024-02-01T10:32:37.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8F/AECtVgAUuf0GAEf9TGuzODe4=', 'Content-Length': '0'} 2024-02-01T10:32:37.564 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000001a593c9b47d1815-0065bb7345-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:32:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:37.564 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:32:37.564 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:32:37.564 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:32:37.564 DEBUG:boto:path=/vclfcn-20/?versions&max-keys=999 2024-02-01T10:32:37.564 DEBUG:boto:auth_path=/vclfcn-20/?versions&max-keys=999 2024-02-01T10:32:37.564 DEBUG:boto:Method: GET 2024-02-01T10:32:37.565 DEBUG:boto:Path: /vclfcn-20/?versions&max-keys=999 2024-02-01T10:32:37.565 DEBUG:boto:Data: 2024-02-01T10:32:37.565 DEBUG:boto:Headers: {} 2024-02-01T10:32:37.565 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:32:37.565 DEBUG:boto:Port: 8001 2024-02-01T10:32:37.565 DEBUG:boto:Params: {} 2024-02-01T10:32:37.565 DEBUG:boto:Token: None 2024-02-01T10:32:37.565 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:32:37 GMT /vclfcn-20/?versions 2024-02-01T10:32:37.565 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:y/bVfHK1R8VHMlDmgrlDf0H2tHo= 2024-02-01T10:32:37.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:y/bVfHK1R8VHMlDmgrlDf0H2tHo=', 'Content-Length': '0'} 2024-02-01T10:32:37.570 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080e9919fc90b0660-0065bb7345-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:37.570 DEBUG:boto:b'vclfcn-20999falseobjnulltrue2024-02-01T10:31:50.863Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjrSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6false2024-02-01T10:28:33.438Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:32:37.571 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:32:37.571 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:32:37.571 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:32:37.571 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:32:37.571 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:32:37.571 DEBUG:boto:path=/vclfcn-20/?versions&max-keys=999 2024-02-01T10:32:37.571 DEBUG:boto:auth_path=/vclfcn-20/?versions&max-keys=999 2024-02-01T10:32:37.571 DEBUG:boto:Method: GET 2024-02-01T10:32:37.571 DEBUG:boto:Path: /vclfcn-20/?versions&max-keys=999 2024-02-01T10:32:37.571 DEBUG:boto:Data: 2024-02-01T10:32:37.572 DEBUG:boto:Headers: {} 2024-02-01T10:32:37.572 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:32:37.572 DEBUG:boto:Port: 8000 2024-02-01T10:32:37.572 DEBUG:boto:Params: {} 2024-02-01T10:32:37.572 DEBUG:boto:Token: None 2024-02-01T10:32:37.572 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:32:37 GMT /vclfcn-20/?versions 2024-02-01T10:32:37.572 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:y/bVfHK1R8VHMlDmgrlDf0H2tHo= 2024-02-01T10:32:37.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:y/bVfHK1R8VHMlDmgrlDf0H2tHo=', 'Content-Length': '0'} 2024-02-01T10:32:37.577 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000735f717cb0ab4212-0065bb7345-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:37.577 DEBUG:boto:b'vclfcn-20999falseobjnulltrue2024-02-01T10:31:50.863Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjrSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6false2024-02-01T10:28:33.438Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:32:37.577 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:32:37.578 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:32:37.578 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:32:37.578 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:32:37.578 DEBUG:boto:path=/vclfcn-20/?versions&max-keys=999 2024-02-01T10:32:37.578 DEBUG:boto:auth_path=/vclfcn-20/?versions&max-keys=999 2024-02-01T10:32:37.578 DEBUG:boto:Method: GET 2024-02-01T10:32:37.578 DEBUG:boto:Path: /vclfcn-20/?versions&max-keys=999 2024-02-01T10:32:37.578 DEBUG:boto:Data: 2024-02-01T10:32:37.578 DEBUG:boto:Headers: {} 2024-02-01T10:32:37.578 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:32:37.579 DEBUG:boto:Port: 8001 2024-02-01T10:32:37.579 DEBUG:boto:Params: {} 2024-02-01T10:32:37.579 DEBUG:boto:Token: None 2024-02-01T10:32:37.579 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:32:37 GMT /vclfcn-20/?versions 2024-02-01T10:32:37.579 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:y/bVfHK1R8VHMlDmgrlDf0H2tHo= 2024-02-01T10:32:37.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:y/bVfHK1R8VHMlDmgrlDf0H2tHo=', 'Content-Length': '0'} 2024-02-01T10:32:37.583 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000afab9a7697a80b22-0065bb7345-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:37.624 DEBUG:boto:b'vclfcn-20999falseobjnulltrue2024-02-01T10:31:50.863Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjrSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6false2024-02-01T10:28:33.438Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:32:37.625 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:32:37.625 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:32:37.625 DEBUG:boto:path=/vclfcn-20/?versions&max-keys=999 2024-02-01T10:32:37.625 DEBUG:boto:auth_path=/vclfcn-20/?versions&max-keys=999 2024-02-01T10:32:37.625 DEBUG:boto:Method: GET 2024-02-01T10:32:37.625 DEBUG:boto:Path: /vclfcn-20/?versions&max-keys=999 2024-02-01T10:32:37.625 DEBUG:boto:Data: 2024-02-01T10:32:37.626 DEBUG:boto:Headers: {} 2024-02-01T10:32:37.626 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:32:37.626 DEBUG:boto:Port: 8000 2024-02-01T10:32:37.626 DEBUG:boto:Params: {} 2024-02-01T10:32:37.626 DEBUG:boto:Token: None 2024-02-01T10:32:37.626 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:32:37 GMT /vclfcn-20/?versions 2024-02-01T10:32:37.627 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:y/bVfHK1R8VHMlDmgrlDf0H2tHo= 2024-02-01T10:32:37.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:y/bVfHK1R8VHMlDmgrlDf0H2tHo=', 'Content-Length': '0'} 2024-02-01T10:32:37.631 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090fa447579624ab7-0065bb7345-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:37.631 DEBUG:boto:b'vclfcn-20999falseobjnulltrue2024-02-01T10:31:50.863Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjrSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6false2024-02-01T10:28:33.438Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:32:37.632 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:32:37.632 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:32:37.632 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:32:37.632 DEBUG:boto:path=/vclfcn-20/obj?versionId=null 2024-02-01T10:32:37.632 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=null 2024-02-01T10:32:37.632 DEBUG:boto:Method: GET 2024-02-01T10:32:37.632 DEBUG:boto:Path: /vclfcn-20/obj?versionId=null 2024-02-01T10:32:37.632 DEBUG:boto:Data: 2024-02-01T10:32:37.632 DEBUG:boto:Headers: {} 2024-02-01T10:32:37.632 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:32:37.633 DEBUG:boto:Port: 8001 2024-02-01T10:32:37.633 DEBUG:boto:Params: {} 2024-02-01T10:32:37.633 DEBUG:boto:Token: None 2024-02-01T10:32:37.633 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:32:37 GMT /vclfcn-20/obj?versionId=null 2024-02-01T10:32:37.633 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vB14Fl2xTgJ8qGpb5alw2hrdD2w= 2024-02-01T10:32:37.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vB14Fl2xTgJ8qGpb5alw2hrdD2w=', 'Content-Length': '0'} 2024-02-01T10:32:37.636 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ea956ab944d1473b-0065bb7345-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:37.636 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:32:37.637 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:32:37.637 DEBUG:boto:path=/vclfcn-20/obj?versionId=null 2024-02-01T10:32:37.637 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=null 2024-02-01T10:32:37.637 DEBUG:boto:Method: GET 2024-02-01T10:32:37.637 DEBUG:boto:Path: /vclfcn-20/obj?versionId=null 2024-02-01T10:32:37.637 DEBUG:boto:Data: 2024-02-01T10:32:37.637 DEBUG:boto:Headers: {} 2024-02-01T10:32:37.637 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:32:37.637 DEBUG:boto:Port: 8000 2024-02-01T10:32:37.638 DEBUG:boto:Params: {} 2024-02-01T10:32:37.638 DEBUG:boto:Token: None 2024-02-01T10:32:37.638 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:32:37 GMT /vclfcn-20/obj?versionId=null 2024-02-01T10:32:37.638 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vB14Fl2xTgJ8qGpb5alw2hrdD2w= 2024-02-01T10:32:37.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vB14Fl2xTgJ8qGpb5alw2hrdD2w=', 'Content-Length': '0'} 2024-02-01T10:32:37.641 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a4b069e9e133d01c-0065bb7345-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:37.641 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:32:37.641 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:32:37.642 DEBUG:boto:path=/vclfcn-20/obj?versionId=null 2024-02-01T10:32:37.642 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=null 2024-02-01T10:32:37.642 DEBUG:boto:Method: HEAD 2024-02-01T10:32:37.642 DEBUG:boto:Path: /vclfcn-20/obj?versionId=null 2024-02-01T10:32:37.642 DEBUG:boto:Data: 2024-02-01T10:32:37.642 DEBUG:boto:Headers: {} 2024-02-01T10:32:37.642 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:32:37.642 DEBUG:boto:Port: 8001 2024-02-01T10:32:37.642 DEBUG:boto:Params: {} 2024-02-01T10:32:37.642 DEBUG:boto:Token: None 2024-02-01T10:32:37.643 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:32:37 GMT /vclfcn-20/obj?versionId=null 2024-02-01T10:32:37.643 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:yf5HYTZGPYih//5iQ/cx5mO5w1s= 2024-02-01T10:32:37.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:yf5HYTZGPYih//5iQ/cx5mO5w1s=', 'Content-Length': '0'} 2024-02-01T10:32:37.646 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ca37b93bf7e89f41-0065bb7345-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:37.646 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:32:37.646 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:32:37.646 DEBUG:boto:path=/vclfcn-20/obj?versionId=null 2024-02-01T10:32:37.646 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=null 2024-02-01T10:32:37.646 DEBUG:boto:Method: HEAD 2024-02-01T10:32:37.647 DEBUG:boto:Path: /vclfcn-20/obj?versionId=null 2024-02-01T10:32:37.647 DEBUG:boto:Data: 2024-02-01T10:32:37.647 DEBUG:boto:Headers: {} 2024-02-01T10:32:37.647 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:32:37.647 DEBUG:boto:Port: 8000 2024-02-01T10:32:37.647 DEBUG:boto:Params: {} 2024-02-01T10:32:37.647 DEBUG:boto:Token: None 2024-02-01T10:32:37.647 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:32:37 GMT /vclfcn-20/obj?versionId=null 2024-02-01T10:32:37.647 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:yf5HYTZGPYih//5iQ/cx5mO5w1s= 2024-02-01T10:32:37.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:yf5HYTZGPYih//5iQ/cx5mO5w1s=', 'Content-Length': '0'} 2024-02-01T10:32:37.650 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009c949f635088696a-0065bb7345-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:37.651 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:32:37.651 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:32:37.651 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:32:37.651 DEBUG:boto:path=/vclfcn-20/obj?versionId=null 2024-02-01T10:32:37.651 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=null 2024-02-01T10:32:37.651 DEBUG:boto:Method: GET 2024-02-01T10:32:37.651 DEBUG:boto:Path: /vclfcn-20/obj?versionId=null 2024-02-01T10:32:37.652 DEBUG:boto:Data: 2024-02-01T10:32:37.652 DEBUG:boto:Headers: {} 2024-02-01T10:32:37.652 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:32:37.652 DEBUG:boto:Port: 8001 2024-02-01T10:32:37.652 DEBUG:boto:Params: {} 2024-02-01T10:32:37.652 DEBUG:boto:Token: None 2024-02-01T10:32:37.652 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:32:37 GMT /vclfcn-20/obj?versionId=null 2024-02-01T10:32:37.652 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vB14Fl2xTgJ8qGpb5alw2hrdD2w= 2024-02-01T10:32:37.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vB14Fl2xTgJ8qGpb5alw2hrdD2w=', 'Content-Length': '0'} 2024-02-01T10:32:37.655 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000016912af14ef0571b-0065bb7345-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:37.656 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:32:37.656 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:32:37.656 DEBUG:boto:path=/vclfcn-20/obj?versionId=null 2024-02-01T10:32:37.656 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=null 2024-02-01T10:32:37.656 DEBUG:boto:Method: GET 2024-02-01T10:32:37.656 DEBUG:boto:Path: /vclfcn-20/obj?versionId=null 2024-02-01T10:32:37.656 DEBUG:boto:Data: 2024-02-01T10:32:37.656 DEBUG:boto:Headers: {} 2024-02-01T10:32:37.656 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:32:37.656 DEBUG:boto:Port: 8000 2024-02-01T10:32:37.657 DEBUG:boto:Params: {} 2024-02-01T10:32:37.657 DEBUG:boto:Token: None 2024-02-01T10:32:37.657 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:32:37 GMT /vclfcn-20/obj?versionId=null 2024-02-01T10:32:37.657 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vB14Fl2xTgJ8qGpb5alw2hrdD2w= 2024-02-01T10:32:37.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vB14Fl2xTgJ8qGpb5alw2hrdD2w=', 'Content-Length': '0'} 2024-02-01T10:32:37.660 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000da0f2e194cfdd008-0065bb7345-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:37.660 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:32:37.660 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:32:37.660 DEBUG:boto:path=/vclfcn-20/?prefix=obj 2024-02-01T10:32:37.660 DEBUG:boto:auth_path=/vclfcn-20/?prefix=obj 2024-02-01T10:32:37.661 DEBUG:boto:Method: GET 2024-02-01T10:32:37.661 DEBUG:boto:Path: /vclfcn-20/?prefix=obj 2024-02-01T10:32:37.661 DEBUG:boto:Data: 2024-02-01T10:32:37.661 DEBUG:boto:Headers: {} 2024-02-01T10:32:37.661 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:32:37.661 DEBUG:boto:Port: 8001 2024-02-01T10:32:37.661 DEBUG:boto:Params: {} 2024-02-01T10:32:37.661 DEBUG:boto:Token: None 2024-02-01T10:32:37.661 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:32:37 GMT /vclfcn-20/ 2024-02-01T10:32:37.662 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gpzla5sjZ2Cosw4UWO75lal0U6c= 2024-02-01T10:32:37.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gpzla5sjZ2Cosw4UWO75lal0U6c=', 'Content-Length': '0'} 2024-02-01T10:32:37.666 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052d304196c19976c-0065bb7345-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:37.708 DEBUG:boto:b'vclfcn-20obj1000falseobj2024-02-01T10:31:50.863Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:32:37.709 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:32:37.709 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:32:37.709 DEBUG:boto:path=/vclfcn-20/?prefix=obj 2024-02-01T10:32:37.709 DEBUG:boto:auth_path=/vclfcn-20/?prefix=obj 2024-02-01T10:32:37.709 DEBUG:boto:Method: GET 2024-02-01T10:32:37.709 DEBUG:boto:Path: /vclfcn-20/?prefix=obj 2024-02-01T10:32:37.709 DEBUG:boto:Data: 2024-02-01T10:32:37.710 DEBUG:boto:Headers: {} 2024-02-01T10:32:37.710 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:32:37.710 DEBUG:boto:Port: 8000 2024-02-01T10:32:37.710 DEBUG:boto:Params: {} 2024-02-01T10:32:37.710 DEBUG:boto:Token: None 2024-02-01T10:32:37.710 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:32:37 GMT /vclfcn-20/ 2024-02-01T10:32:37.710 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gpzla5sjZ2Cosw4UWO75lal0U6c= 2024-02-01T10:32:37.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gpzla5sjZ2Cosw4UWO75lal0U6c=', 'Content-Length': '0'} 2024-02-01T10:32:37.715 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000544cab6e19e434dc-0065bb7345-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:37.715 DEBUG:boto:b'vclfcn-20obj1000falseobj2024-02-01T10:31:50.863Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:32:37.715 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:32:37.715 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:32:37.716 DEBUG:boto:Method: HEAD 2024-02-01T10:32:37.716 DEBUG:boto:Path: /vclfcn-20/obj 2024-02-01T10:32:37.716 DEBUG:boto:Data: 2024-02-01T10:32:37.716 DEBUG:boto:Headers: {} 2024-02-01T10:32:37.716 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:32:37.716 DEBUG:boto:Port: 8001 2024-02-01T10:32:37.716 DEBUG:boto:Params: {} 2024-02-01T10:32:37.716 DEBUG:boto:Token: None 2024-02-01T10:32:37.716 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:32:37 GMT /vclfcn-20/obj 2024-02-01T10:32:37.717 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ncizQ1dZ2N6DbjZiRa6Buohj94I= 2024-02-01T10:32:37.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ncizQ1dZ2N6DbjZiRa6Buohj94I=', 'Content-Length': '0'} 2024-02-01T10:32:37.719 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002bd994b683ff1087-0065bb7345-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:37.720 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:32:37.720 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:32:37.720 DEBUG:boto:Method: HEAD 2024-02-01T10:32:37.720 DEBUG:boto:Path: /vclfcn-20/obj 2024-02-01T10:32:37.720 DEBUG:boto:Data: 2024-02-01T10:32:37.720 DEBUG:boto:Headers: {} 2024-02-01T10:32:37.720 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:32:37.720 DEBUG:boto:Port: 8000 2024-02-01T10:32:37.721 DEBUG:boto:Params: {} 2024-02-01T10:32:37.721 DEBUG:boto:Token: None 2024-02-01T10:32:37.721 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:32:37 GMT /vclfcn-20/obj 2024-02-01T10:32:37.721 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ncizQ1dZ2N6DbjZiRa6Buohj94I= 2024-02-01T10:32:37.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ncizQ1dZ2N6DbjZiRa6Buohj94I=', 'Content-Length': '0'} 2024-02-01T10:32:37.724 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000eb62ce091a01ed2e-0065bb7345-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:37.724 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:32:37.724 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:32:37.724 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:32:37.724 DEBUG:boto:Method: GET 2024-02-01T10:32:37.725 DEBUG:boto:Path: /vclfcn-20/obj 2024-02-01T10:32:37.725 DEBUG:boto:Data: 2024-02-01T10:32:37.725 DEBUG:boto:Headers: {} 2024-02-01T10:32:37.725 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:32:37.725 DEBUG:boto:Port: 8001 2024-02-01T10:32:37.726 DEBUG:boto:Params: {} 2024-02-01T10:32:37.726 DEBUG:boto:Token: None 2024-02-01T10:32:37.726 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:32:37 GMT /vclfcn-20/obj 2024-02-01T10:32:37.726 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QGRpIV1VkSouoJ3RcWzpWTlQtzE= 2024-02-01T10:32:37.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QGRpIV1VkSouoJ3RcWzpWTlQtzE=', 'Content-Length': '0'} 2024-02-01T10:32:37.729 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000818ed2cd892a6856-0065bb7345-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:37.729 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:32:37.729 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:32:37.729 DEBUG:boto:Method: GET 2024-02-01T10:32:37.729 DEBUG:boto:Path: /vclfcn-20/obj 2024-02-01T10:32:37.729 DEBUG:boto:Data: 2024-02-01T10:32:37.729 DEBUG:boto:Headers: {} 2024-02-01T10:32:37.729 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:32:37.730 DEBUG:boto:Port: 8000 2024-02-01T10:32:37.730 DEBUG:boto:Params: {} 2024-02-01T10:32:37.730 DEBUG:boto:Token: None 2024-02-01T10:32:37.730 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:32:37 GMT /vclfcn-20/obj 2024-02-01T10:32:37.730 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QGRpIV1VkSouoJ3RcWzpWTlQtzE= 2024-02-01T10:32:37.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QGRpIV1VkSouoJ3RcWzpWTlQtzE=', 'Content-Length': '0'} 2024-02-01T10:32:37.733 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d1deede03e2e0c28-0065bb7345-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:37.733 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:32:37.733 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:32:37.733 DEBUG:boto:path=/vclfcn-20/?prefix=obj 2024-02-01T10:32:37.733 DEBUG:boto:auth_path=/vclfcn-20/?prefix=obj 2024-02-01T10:32:37.734 DEBUG:boto:Method: GET 2024-02-01T10:32:37.734 DEBUG:boto:Path: /vclfcn-20/?prefix=obj 2024-02-01T10:32:37.734 DEBUG:boto:Data: 2024-02-01T10:32:37.734 DEBUG:boto:Headers: {} 2024-02-01T10:32:37.734 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:32:37.734 DEBUG:boto:Port: 8001 2024-02-01T10:32:37.734 DEBUG:boto:Params: {} 2024-02-01T10:32:37.734 DEBUG:boto:Token: None 2024-02-01T10:32:37.734 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:32:37 GMT /vclfcn-20/ 2024-02-01T10:32:37.734 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gpzla5sjZ2Cosw4UWO75lal0U6c= 2024-02-01T10:32:37.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gpzla5sjZ2Cosw4UWO75lal0U6c=', 'Content-Length': '0'} 2024-02-01T10:32:37.741 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000662a8de3ceea1f4c-0065bb7345-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:37.784 DEBUG:boto:b'vclfcn-20obj1000falseobj2024-02-01T10:31:50.863Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:32:37.785 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:32:37.785 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:32:37.785 DEBUG:boto:path=/vclfcn-20/?prefix=obj 2024-02-01T10:32:37.785 DEBUG:boto:auth_path=/vclfcn-20/?prefix=obj 2024-02-01T10:32:37.785 DEBUG:boto:Method: GET 2024-02-01T10:32:37.785 DEBUG:boto:Path: /vclfcn-20/?prefix=obj 2024-02-01T10:32:37.785 DEBUG:boto:Data: 2024-02-01T10:32:37.785 DEBUG:boto:Headers: {} 2024-02-01T10:32:37.786 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:32:37.786 DEBUG:boto:Port: 8000 2024-02-01T10:32:37.786 DEBUG:boto:Params: {} 2024-02-01T10:32:37.786 DEBUG:boto:Token: None 2024-02-01T10:32:37.786 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:32:37 GMT /vclfcn-20/ 2024-02-01T10:32:37.787 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gpzla5sjZ2Cosw4UWO75lal0U6c= 2024-02-01T10:32:37.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gpzla5sjZ2Cosw4UWO75lal0U6c=', 'Content-Length': '0'} 2024-02-01T10:32:37.791 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f48ed27ec703a2b3-0065bb7345-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:37.791 DEBUG:boto:b'vclfcn-20obj1000falseobj2024-02-01T10:31:50.863Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:32:37.792 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:32:37.792 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:32:37.792 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:32:37.792 DEBUG:boto:path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:32:37.792 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:32:37.792 DEBUG:boto:Method: GET 2024-02-01T10:32:37.792 DEBUG:boto:Path: /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:32:37.792 DEBUG:boto:Data: 2024-02-01T10:32:37.793 DEBUG:boto:Headers: {} 2024-02-01T10:32:37.793 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:32:37.793 DEBUG:boto:Port: 8001 2024-02-01T10:32:37.793 DEBUG:boto:Params: {} 2024-02-01T10:32:37.793 DEBUG:boto:Token: None 2024-02-01T10:32:37.793 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:32:37 GMT /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:32:37.793 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HXVe9Xw+b4qPDb2xxEi1E5KIOgA= 2024-02-01T10:32:37.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HXVe9Xw+b4qPDb2xxEi1E5KIOgA=', 'Content-Length': '0'} 2024-02-01T10:32:37.796 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:33 GMT'), ('x-amz-version-id', 'rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000af49f7c1052c21c7-0065bb7345-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:37.796 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:32:37.796 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:32:37.796 DEBUG:boto:path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:32:37.796 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:32:37.796 DEBUG:boto:Method: GET 2024-02-01T10:32:37.797 DEBUG:boto:Path: /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:32:37.797 DEBUG:boto:Data: 2024-02-01T10:32:37.797 DEBUG:boto:Headers: {} 2024-02-01T10:32:37.797 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:32:37.797 DEBUG:boto:Port: 8000 2024-02-01T10:32:37.797 DEBUG:boto:Params: {} 2024-02-01T10:32:37.797 DEBUG:boto:Token: None 2024-02-01T10:32:37.797 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:32:37 GMT /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:32:37.797 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HXVe9Xw+b4qPDb2xxEi1E5KIOgA= 2024-02-01T10:32:37.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HXVe9Xw+b4qPDb2xxEi1E5KIOgA=', 'Content-Length': '0'} 2024-02-01T10:32:37.800 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:33 GMT'), ('x-amz-version-id', 'rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ba84a18dc905298e-0065bb7345-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:37.800 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:32:37.800 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:32:37.800 DEBUG:boto:path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:32:37.801 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:32:37.801 DEBUG:boto:Method: HEAD 2024-02-01T10:32:37.801 DEBUG:boto:Path: /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:32:37.801 DEBUG:boto:Data: 2024-02-01T10:32:37.801 DEBUG:boto:Headers: {} 2024-02-01T10:32:37.801 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:32:37.801 DEBUG:boto:Port: 8001 2024-02-01T10:32:37.801 DEBUG:boto:Params: {} 2024-02-01T10:32:37.801 DEBUG:boto:Token: None 2024-02-01T10:32:37.801 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:32:37 GMT /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:32:37.802 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GvhXjAtxW7kerCSj23FNi63ciQo= 2024-02-01T10:32:37.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GvhXjAtxW7kerCSj23FNi63ciQo=', 'Content-Length': '0'} 2024-02-01T10:32:37.805 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:33 GMT'), ('x-amz-version-id', 'rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007bcbfc702b26513f-0065bb7345-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:37.805 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:32:37.806 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:32:37.806 DEBUG:boto:path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:32:37.806 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:32:37.806 DEBUG:boto:Method: HEAD 2024-02-01T10:32:37.806 DEBUG:boto:Path: /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:32:37.806 DEBUG:boto:Data: 2024-02-01T10:32:37.806 DEBUG:boto:Headers: {} 2024-02-01T10:32:37.806 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:32:37.806 DEBUG:boto:Port: 8000 2024-02-01T10:32:37.806 DEBUG:boto:Params: {} 2024-02-01T10:32:37.807 DEBUG:boto:Token: None 2024-02-01T10:32:37.807 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:32:37 GMT /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:32:37.807 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GvhXjAtxW7kerCSj23FNi63ciQo= 2024-02-01T10:32:37.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GvhXjAtxW7kerCSj23FNi63ciQo=', 'Content-Length': '0'} 2024-02-01T10:32:37.809 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:33 GMT'), ('x-amz-version-id', 'rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000004a0b6ef9411b620-0065bb7345-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:37.810 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:32:37.810 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:32:37.810 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:32:37.810 DEBUG:boto:path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:32:37.810 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:32:37.810 DEBUG:boto:Method: GET 2024-02-01T10:32:37.811 DEBUG:boto:Path: /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:32:37.811 DEBUG:boto:Data: 2024-02-01T10:32:37.811 DEBUG:boto:Headers: {} 2024-02-01T10:32:37.811 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:32:37.811 DEBUG:boto:Port: 8001 2024-02-01T10:32:37.811 DEBUG:boto:Params: {} 2024-02-01T10:32:37.811 DEBUG:boto:Token: None 2024-02-01T10:32:37.811 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:32:37 GMT /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:32:37.811 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HXVe9Xw+b4qPDb2xxEi1E5KIOgA= 2024-02-01T10:32:37.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HXVe9Xw+b4qPDb2xxEi1E5KIOgA=', 'Content-Length': '0'} 2024-02-01T10:32:37.814 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:33 GMT'), ('x-amz-version-id', 'rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ab63b8f5927bd474-0065bb7345-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:37.814 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:32:37.814 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:32:37.814 DEBUG:boto:path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:32:37.815 DEBUG:boto:auth_path=/vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:32:37.815 DEBUG:boto:Method: GET 2024-02-01T10:32:37.815 DEBUG:boto:Path: /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:32:37.815 DEBUG:boto:Data: 2024-02-01T10:32:37.815 DEBUG:boto:Headers: {} 2024-02-01T10:32:37.815 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:32:37.815 DEBUG:boto:Port: 8000 2024-02-01T10:32:37.815 DEBUG:boto:Params: {} 2024-02-01T10:32:37.815 DEBUG:boto:Token: None 2024-02-01T10:32:37.815 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:32:37 GMT /vclfcn-20/obj?versionId=rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6 2024-02-01T10:32:37.816 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HXVe9Xw+b4qPDb2xxEi1E5KIOgA= 2024-02-01T10:32:37.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HXVe9Xw+b4qPDb2xxEi1E5KIOgA=', 'Content-Length': '0'} 2024-02-01T10:32:37.818 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:28:33 GMT'), ('x-amz-version-id', 'rSQHB.0vjmR-TG8O3yMrgIES-Mwsr.6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007cad6e334987fd84-0065bb7345-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:37.818 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:32:37.819 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:32:37.819 DEBUG:boto:path=/vclfcn-20/?prefix=obj 2024-02-01T10:32:37.819 DEBUG:boto:auth_path=/vclfcn-20/?prefix=obj 2024-02-01T10:32:37.819 DEBUG:boto:Method: GET 2024-02-01T10:32:37.819 DEBUG:boto:Path: /vclfcn-20/?prefix=obj 2024-02-01T10:32:37.819 DEBUG:boto:Data: 2024-02-01T10:32:37.819 DEBUG:boto:Headers: {} 2024-02-01T10:32:37.819 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:32:37.819 DEBUG:boto:Port: 8001 2024-02-01T10:32:37.819 DEBUG:boto:Params: {} 2024-02-01T10:32:37.820 DEBUG:boto:Token: None 2024-02-01T10:32:37.820 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:32:37 GMT /vclfcn-20/ 2024-02-01T10:32:37.820 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gpzla5sjZ2Cosw4UWO75lal0U6c= 2024-02-01T10:32:37.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gpzla5sjZ2Cosw4UWO75lal0U6c=', 'Content-Length': '0'} 2024-02-01T10:32:37.824 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3530af634a0c86b-0065bb7345-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:37.868 DEBUG:boto:b'vclfcn-20obj1000falseobj2024-02-01T10:31:50.863Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:32:37.868 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:32:37.869 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:32:37.869 DEBUG:boto:path=/vclfcn-20/?prefix=obj 2024-02-01T10:32:37.869 DEBUG:boto:auth_path=/vclfcn-20/?prefix=obj 2024-02-01T10:32:37.869 DEBUG:boto:Method: GET 2024-02-01T10:32:37.869 DEBUG:boto:Path: /vclfcn-20/?prefix=obj 2024-02-01T10:32:37.869 DEBUG:boto:Data: 2024-02-01T10:32:37.869 DEBUG:boto:Headers: {} 2024-02-01T10:32:37.869 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:32:37.869 DEBUG:boto:Port: 8000 2024-02-01T10:32:37.869 DEBUG:boto:Params: {} 2024-02-01T10:32:37.870 DEBUG:boto:Token: None 2024-02-01T10:32:37.870 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:32:37 GMT /vclfcn-20/ 2024-02-01T10:32:37.870 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gpzla5sjZ2Cosw4UWO75lal0U6c= 2024-02-01T10:32:37.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gpzla5sjZ2Cosw4UWO75lal0U6c=', 'Content-Length': '0'} 2024-02-01T10:32:37.874 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000497062623cd5e840-0065bb7345-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:37.875 DEBUG:boto:b'vclfcn-20obj1000falseobj2024-02-01T10:31:50.863Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:32:37.875 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:32:37.875 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:32:37.875 DEBUG:boto:Method: HEAD 2024-02-01T10:32:37.875 DEBUG:boto:Path: /vclfcn-20/obj 2024-02-01T10:32:37.875 DEBUG:boto:Data: 2024-02-01T10:32:37.875 DEBUG:boto:Headers: {} 2024-02-01T10:32:37.875 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:32:37.876 DEBUG:boto:Port: 8001 2024-02-01T10:32:37.876 DEBUG:boto:Params: {} 2024-02-01T10:32:37.876 DEBUG:boto:Token: None 2024-02-01T10:32:37.876 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:32:37 GMT /vclfcn-20/obj 2024-02-01T10:32:37.876 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ncizQ1dZ2N6DbjZiRa6Buohj94I= 2024-02-01T10:32:37.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ncizQ1dZ2N6DbjZiRa6Buohj94I=', 'Content-Length': '0'} 2024-02-01T10:32:37.879 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007470149882f1bd42-0065bb7345-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:37.879 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:32:37.879 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:32:37.879 DEBUG:boto:Method: HEAD 2024-02-01T10:32:37.880 DEBUG:boto:Path: /vclfcn-20/obj 2024-02-01T10:32:37.880 DEBUG:boto:Data: 2024-02-01T10:32:37.880 DEBUG:boto:Headers: {} 2024-02-01T10:32:37.880 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:32:37.880 DEBUG:boto:Port: 8000 2024-02-01T10:32:37.880 DEBUG:boto:Params: {} 2024-02-01T10:32:37.880 DEBUG:boto:Token: None 2024-02-01T10:32:37.880 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:32:37 GMT /vclfcn-20/obj 2024-02-01T10:32:37.880 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ncizQ1dZ2N6DbjZiRa6Buohj94I= 2024-02-01T10:32:37.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ncizQ1dZ2N6DbjZiRa6Buohj94I=', 'Content-Length': '0'} 2024-02-01T10:32:37.883 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007309adbfb4b737be-0065bb7345-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:37.884 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:32:37.884 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:32:37.884 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:32:37.884 DEBUG:boto:Method: GET 2024-02-01T10:32:37.884 DEBUG:boto:Path: /vclfcn-20/obj 2024-02-01T10:32:37.884 DEBUG:boto:Data: 2024-02-01T10:32:37.884 DEBUG:boto:Headers: {} 2024-02-01T10:32:37.884 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:32:37.884 DEBUG:boto:Port: 8001 2024-02-01T10:32:37.885 DEBUG:boto:Params: {} 2024-02-01T10:32:37.885 DEBUG:boto:Token: None 2024-02-01T10:32:37.885 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:32:37 GMT /vclfcn-20/obj 2024-02-01T10:32:37.885 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QGRpIV1VkSouoJ3RcWzpWTlQtzE= 2024-02-01T10:32:37.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QGRpIV1VkSouoJ3RcWzpWTlQtzE=', 'Content-Length': '0'} 2024-02-01T10:32:37.888 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e8a1b8e5006061f3-0065bb7345-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:37.888 DEBUG:boto:path=/vclfcn-20/obj 2024-02-01T10:32:37.888 DEBUG:boto:auth_path=/vclfcn-20/obj 2024-02-01T10:32:37.888 DEBUG:boto:Method: GET 2024-02-01T10:32:37.888 DEBUG:boto:Path: /vclfcn-20/obj 2024-02-01T10:32:37.888 DEBUG:boto:Data: 2024-02-01T10:32:37.889 DEBUG:boto:Headers: {} 2024-02-01T10:32:37.889 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:32:37.889 DEBUG:boto:Port: 8000 2024-02-01T10:32:37.889 DEBUG:boto:Params: {} 2024-02-01T10:32:37.889 DEBUG:boto:Token: None 2024-02-01T10:32:37.889 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:32:37 GMT /vclfcn-20/obj 2024-02-01T10:32:37.889 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QGRpIV1VkSouoJ3RcWzpWTlQtzE= 2024-02-01T10:32:37.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QGRpIV1VkSouoJ3RcWzpWTlQtzE=', 'Content-Length': '0'} 2024-02-01T10:32:37.892 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:31:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e44d31117e6fdfdf-0065bb7345-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:37.892 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:32:37.892 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:32:37.892 DEBUG:boto:path=/vclfcn-20/?prefix=obj 2024-02-01T10:32:37.892 DEBUG:boto:auth_path=/vclfcn-20/?prefix=obj 2024-02-01T10:32:37.893 DEBUG:boto:Method: GET 2024-02-01T10:32:37.893 DEBUG:boto:Path: /vclfcn-20/?prefix=obj 2024-02-01T10:32:37.893 DEBUG:boto:Data: 2024-02-01T10:32:37.893 DEBUG:boto:Headers: {} 2024-02-01T10:32:37.893 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:32:37.893 DEBUG:boto:Port: 8001 2024-02-01T10:32:37.893 DEBUG:boto:Params: {} 2024-02-01T10:32:37.893 DEBUG:boto:Token: None 2024-02-01T10:32:37.893 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:32:37 GMT /vclfcn-20/ 2024-02-01T10:32:37.893 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gpzla5sjZ2Cosw4UWO75lal0U6c= 2024-02-01T10:32:37.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gpzla5sjZ2Cosw4UWO75lal0U6c=', 'Content-Length': '0'} 2024-02-01T10:32:37.898 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af3a7b4113fca4b7-0065bb7345-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:37.898 DEBUG:boto:b'vclfcn-20obj1000falseobj2024-02-01T10:31:50.863Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:32:37.899 DEBUG:boto:path=/vclfcn-20/ 2024-02-01T10:32:37.899 DEBUG:boto:auth_path=/vclfcn-20/ 2024-02-01T10:32:37.899 DEBUG:boto:path=/vclfcn-20/?prefix=obj 2024-02-01T10:32:37.899 DEBUG:boto:auth_path=/vclfcn-20/?prefix=obj 2024-02-01T10:32:37.899 DEBUG:boto:Method: GET 2024-02-01T10:32:37.899 DEBUG:boto:Path: /vclfcn-20/?prefix=obj 2024-02-01T10:32:37.899 DEBUG:boto:Data: 2024-02-01T10:32:37.899 DEBUG:boto:Headers: {} 2024-02-01T10:32:37.899 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:32:37.900 DEBUG:boto:Port: 8000 2024-02-01T10:32:37.900 DEBUG:boto:Params: {} 2024-02-01T10:32:37.900 DEBUG:boto:Token: None 2024-02-01T10:32:37.900 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:32:37 GMT /vclfcn-20/ 2024-02-01T10:32:37.900 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gpzla5sjZ2Cosw4UWO75lal0U6c= 2024-02-01T10:32:37.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gpzla5sjZ2Cosw4UWO75lal0U6c=', 'Content-Length': '0'} 2024-02-01T10:32:37.904 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000aef758fb260b6d8-0065bb7345-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:37.904 DEBUG:boto:b'vclfcn-20obj1000falseobj2024-02-01T10:31:50.863Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:32:37.904 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-20 zones={test-zone3, test-zone2} 2024-02-01T10:32:37.905 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_version_suspended_incremental_sync ... ok 2024-02-01T10:32:37.906 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=vclfcn-21 2024-02-01T10:32:37.907 DEBUG:boto:path=/vclfcn-21/ 2024-02-01T10:32:37.907 DEBUG:boto:auth_path=/vclfcn-21/ 2024-02-01T10:32:37.907 DEBUG:boto:Method: PUT 2024-02-01T10:32:37.907 DEBUG:boto:Path: /vclfcn-21/ 2024-02-01T10:32:37.907 DEBUG:boto:Data: 2024-02-01T10:32:37.907 DEBUG:boto:Headers: {} 2024-02-01T10:32:37.907 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:32:37.907 DEBUG:boto:Port: 8000 2024-02-01T10:32:37.907 DEBUG:boto:Params: {} 2024-02-01T10:32:37.907 DEBUG:boto:Token: None 2024-02-01T10:32:37.908 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:32:37 GMT /vclfcn-21/ 2024-02-01T10:32:37.908 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Hrx2q7n2hWLEt5gr63JltnJu5/8= 2024-02-01T10:32:37.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Hrx2q7n2hWLEt5gr63JltnJu5/8=', 'Content-Length': '0'} 2024-02-01T10:32:37.947 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000207a90401b3f171c-0065bb7345-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:32:37 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:37.947 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=vclfcn-22 2024-02-01T10:32:37.948 DEBUG:boto:path=/vclfcn-22/ 2024-02-01T10:32:37.948 DEBUG:boto:auth_path=/vclfcn-22/ 2024-02-01T10:32:37.948 DEBUG:boto:Method: PUT 2024-02-01T10:32:37.948 DEBUG:boto:Path: /vclfcn-22/ 2024-02-01T10:32:37.948 DEBUG:boto:Data: 2024-02-01T10:32:37.948 DEBUG:boto:Headers: {} 2024-02-01T10:32:37.948 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:32:37.948 DEBUG:boto:Port: 8000 2024-02-01T10:32:37.948 DEBUG:boto:Params: {} 2024-02-01T10:32:37.949 DEBUG:boto:Token: None 2024-02-01T10:32:37.949 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:32:37 GMT /vclfcn-22/ 2024-02-01T10:32:37.949 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:KQRQV1rCWVm50T2fA5tYnjrfaUM= 2024-02-01T10:32:37.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:KQRQV1rCWVm50T2fA5tYnjrfaUM=', 'Content-Length': '0'} 2024-02-01T10:32:38.010 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000110bfc288f7ff057-0065bb7345-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:32:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:38.010 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=vclfcn-23 2024-02-01T10:32:38.010 DEBUG:boto:path=/vclfcn-23/ 2024-02-01T10:32:38.010 DEBUG:boto:auth_path=/vclfcn-23/ 2024-02-01T10:32:38.010 DEBUG:boto:Method: PUT 2024-02-01T10:32:38.010 DEBUG:boto:Path: /vclfcn-23/ 2024-02-01T10:32:38.010 DEBUG:boto:Data: 2024-02-01T10:32:38.011 DEBUG:boto:Headers: {} 2024-02-01T10:32:38.011 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:32:38.011 DEBUG:boto:Port: 8001 2024-02-01T10:32:38.011 DEBUG:boto:Params: {} 2024-02-01T10:32:38.011 DEBUG:boto:Token: None 2024-02-01T10:32:38.011 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:32:38 GMT /vclfcn-23/ 2024-02-01T10:32:38.011 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ExDjP6CrSxaqcD+NqeVTjm/nbdc= 2024-02-01T10:32:38.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ExDjP6CrSxaqcD+NqeVTjm/nbdc=', 'Content-Length': '0'} 2024-02-01T10:32:38.067 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fc5ed63a7b0db169-0065bb7346-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:32:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:38.067 DEBUG:boto:path=/vclfcn-21/ 2024-02-01T10:32:38.067 DEBUG:boto:auth_path=/vclfcn-21/ 2024-02-01T10:32:38.068 DEBUG:boto:path=/vclfcn-21/?versioning 2024-02-01T10:32:38.068 DEBUG:boto:auth_path=/vclfcn-21/?versioning 2024-02-01T10:32:38.068 DEBUG:boto:Method: PUT 2024-02-01T10:32:38.068 DEBUG:boto:Path: /vclfcn-21/?versioning 2024-02-01T10:32:38.068 DEBUG:boto:Data: Enabled Disabled 2024-02-01T10:32:38.068 DEBUG:boto:Headers: {} 2024-02-01T10:32:38.068 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:32:38.068 DEBUG:boto:Port: 8000 2024-02-01T10:32:38.068 DEBUG:boto:Params: {} 2024-02-01T10:32:38.069 DEBUG:boto:Token: None 2024-02-01T10:32:38.069 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:32:38 GMT /vclfcn-21/?versioning 2024-02-01T10:32:38.069 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5Z0I+O97X210l1OOsV7b9MNPcvk= 2024-02-01T10:32:38.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5Z0I+O97X210l1OOsV7b9MNPcvk=', 'Content-Length': '228'} 2024-02-01T10:32:38.080 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004603c810ac7994ae-0065bb7346-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:32:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:38.081 DEBUG:boto:path=/vclfcn-22/ 2024-02-01T10:32:38.081 DEBUG:boto:auth_path=/vclfcn-22/ 2024-02-01T10:32:38.081 DEBUG:boto:path=/vclfcn-22/?versioning 2024-02-01T10:32:38.081 DEBUG:boto:auth_path=/vclfcn-22/?versioning 2024-02-01T10:32:38.081 DEBUG:boto:Method: PUT 2024-02-01T10:32:38.081 DEBUG:boto:Path: /vclfcn-22/?versioning 2024-02-01T10:32:38.081 DEBUG:boto:Data: Enabled Disabled 2024-02-01T10:32:38.081 DEBUG:boto:Headers: {} 2024-02-01T10:32:38.081 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:32:38.081 DEBUG:boto:Port: 8000 2024-02-01T10:32:38.082 DEBUG:boto:Params: {} 2024-02-01T10:32:38.082 DEBUG:boto:Token: None 2024-02-01T10:32:38.082 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:32:38 GMT /vclfcn-22/?versioning 2024-02-01T10:32:38.082 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vLgm9dX2mCuIeC8RADr9nFpKqrs= 2024-02-01T10:32:38.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vLgm9dX2mCuIeC8RADr9nFpKqrs=', 'Content-Length': '228'} 2024-02-01T10:32:38.095 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000dba57e4f2d7ca345-0065bb7346-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:32:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:38.095 DEBUG:boto:path=/vclfcn-23/ 2024-02-01T10:32:38.095 DEBUG:boto:auth_path=/vclfcn-23/ 2024-02-01T10:32:38.096 DEBUG:boto:path=/vclfcn-23/?versioning 2024-02-01T10:32:38.096 DEBUG:boto:auth_path=/vclfcn-23/?versioning 2024-02-01T10:32:38.096 DEBUG:boto:Method: PUT 2024-02-01T10:32:38.096 DEBUG:boto:Path: /vclfcn-23/?versioning 2024-02-01T10:32:38.096 DEBUG:boto:Data: Enabled Disabled 2024-02-01T10:32:38.096 DEBUG:boto:Headers: {} 2024-02-01T10:32:38.096 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:32:38.096 DEBUG:boto:Port: 8001 2024-02-01T10:32:38.096 DEBUG:boto:Params: {} 2024-02-01T10:32:38.096 DEBUG:boto:Token: None 2024-02-01T10:32:38.097 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:32:38 GMT /vclfcn-23/?versioning 2024-02-01T10:32:38.097 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:kdf6/aSy7dehVOBFDnoCGsxUcew= 2024-02-01T10:32:38.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:kdf6/aSy7dehVOBFDnoCGsxUcew=', 'Content-Length': '228'} 2024-02-01T10:32:38.111 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002fb0bba8b91e2c6b-0065bb7346-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:32:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:38.111 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:32:38.111 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:32:38.112 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:32:38.151 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:32:38.151 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:32:38.206 INFO:teuthology.orchestra.run.smithi148.stderr:No --period given, using current period=5dc4efbd-eccd-47a8-9471-ae4a209ec978 2024-02-01T10:32:38.235 INFO:teuthology.orchestra.run.smithi148.stdout:[{"marker":"1_1706782453.132619_30.1","last_update":"2024-02-01T10:14:13.132619Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1706783558.115340_142.1","last_update":"2024-02-01T10:32:38.115340Z"},{"marker":"1_1706782595.670952_121.1","last_update":"2024-02-01T10:16:35.670952Z"}] 2024-02-01T10:32:38.236 DEBUG:rgw_multi.tests:master meta markers={0: '1_1706782453.132619_30.1', 1: '', 2: '1_1706783558.115340_142.1', 3: '1_1706782595.670952_121.1'} 2024-02-01T10:32:38.236 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-02-01T10:32:38.236 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:32:38.236 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:32:38.236 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:32:38.287 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:32:38.287 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:32:38.341 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:32:38.342+0000 7fb3da4ae980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:32:38.342 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:32:38.342+0000 7fb3da4ae980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:32:38.365 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783557.936493_125.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:32:37.936493Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:32:38Z"} 2024-02-01T10:32:38.365 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783557.936493_125.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:32:37.936493Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:32:38Z"} 2024-02-01T10:32:38.365 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706783557.936493_125.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:32:37.936493Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:32:38.365 DEBUG:rgw_multi.tests:log_status={0: '1_1706782453.132619_30.1', 1: '', 2: '1_1706783558.115340_142.1', 3: '1_1706782595.670952_121.1'} 2024-02-01T10:32:38.365 DEBUG:rgw_multi.tests:sync_status={0: '1_1706782453.132619_30.1', 1: '', 2: '1_1706783557.936493_125.1', 3: '1_1706782595.670952_121.1'} 2024-02-01T10:32:38.366 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1706783558.115340_142.1 target=1_1706783557.936493_125.1 2024-02-01T10:32:43.366 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:32:43.367 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:32:43.367 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:32:43.410 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:32:43.410 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:32:43.464 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:32:43.462+0000 7f5066959980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:32:43.465 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:32:43.466+0000 7f5066959980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:32:43.487 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783558.115340_142.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:32:38.115340Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:32:43Z"} 2024-02-01T10:32:43.488 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783558.115340_142.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:32:38.115340Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:32:43Z"} 2024-02-01T10:32:43.488 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706783558.115340_142.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:32:38.115340Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:32:43.488 DEBUG:rgw_multi.tests:log_status={0: '1_1706782453.132619_30.1', 1: '', 2: '1_1706783558.115340_142.1', 3: '1_1706782595.670952_121.1'} 2024-02-01T10:32:43.488 DEBUG:rgw_multi.tests:sync_status={0: '1_1706782453.132619_30.1', 1: '', 2: '1_1706783558.115340_142.1', 3: '1_1706782595.670952_121.1'} 2024-02-01T10:32:43.489 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-02-01T10:32:43.489 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-02-01T10:32:43.489 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:32:43.489 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:32:43.490 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:32:43.532 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:32:43.533 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:32:43.587 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:32:43.593+0000 7f972608f980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:32:43.589 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:32:43.597+0000 7f972608f980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:32:43.612 INFO:teuthology.orchestra.run.smithi148.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783558.115340_142.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:32:38.115340Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:32:43Z"} 2024-02-01T10:32:43.612 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783558.115340_142.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:32:38.115340Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:32:43Z"} 2024-02-01T10:32:43.612 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706783558.115340_142.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:32:38.115340Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:32:43.613 DEBUG:rgw_multi.tests:log_status={0: '1_1706782453.132619_30.1', 1: '', 2: '1_1706783558.115340_142.1', 3: '1_1706782595.670952_121.1'} 2024-02-01T10:32:43.613 DEBUG:rgw_multi.tests:sync_status={0: '1_1706782453.132619_30.1', 1: '', 2: '1_1706783558.115340_142.1', 3: '1_1706782595.670952_121.1'} 2024-02-01T10:32:43.613 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-02-01T10:32:43.614 DEBUG:boto:path=/vclfcn-21/ 2024-02-01T10:32:43.614 DEBUG:boto:auth_path=/vclfcn-21/ 2024-02-01T10:32:43.614 DEBUG:boto:Method: HEAD 2024-02-01T10:32:43.615 DEBUG:boto:Path: /vclfcn-21/ 2024-02-01T10:32:43.615 DEBUG:boto:Data: 2024-02-01T10:32:43.615 DEBUG:boto:Headers: {} 2024-02-01T10:32:43.615 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:32:43.615 DEBUG:boto:Port: 8000 2024-02-01T10:32:43.615 DEBUG:boto:Params: {} 2024-02-01T10:32:43.616 DEBUG:boto:Token: None 2024-02-01T10:32:43.617 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:32:43 GMT /vclfcn-21/ 2024-02-01T10:32:43.617 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:dEA/BQ1DdOYX6/ZSV8b3Ycn6wUg= 2024-02-01T10:32:43.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:43 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dEA/BQ1DdOYX6/ZSV8b3Ycn6wUg=', 'Content-Length': '0'} 2024-02-01T10:32:43.620 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b631622bef85acca-0065bb734b-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:32:43 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:43.621 DEBUG:boto:path=/vclfcn-21/obj 2024-02-01T10:32:43.622 DEBUG:boto:auth_path=/vclfcn-21/obj 2024-02-01T10:32:43.622 DEBUG:boto:Method: PUT 2024-02-01T10:32:43.623 DEBUG:boto:Path: /vclfcn-21/obj 2024-02-01T10:32:43.623 DEBUG:boto:Data: 2024-02-01T10:32:43.623 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-02-01T10:32:43.623 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:32:43.623 DEBUG:boto:Port: 8000 2024-02-01T10:32:43.624 DEBUG:boto:Params: {} 2024-02-01T10:32:43.624 DEBUG:boto:Token: None 2024-02-01T10:32:43.624 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Thu, 01 Feb 2024 10:32:43 GMT /vclfcn-21/obj 2024-02-01T10:32:43.626 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:bFGTJodeQgdU2YTpNEQFZd9OQv4= 2024-02-01T10:32:43.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:32:43 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:bFGTJodeQgdU2YTpNEQFZd9OQv4='} 2024-02-01T10:32:43.638 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY'), ('x-amz-request-id', 'tx00000941cd890132aad17-0065bb734b-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:43 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:43.639 DEBUG:boto:path=/vclfcn-21/ 2024-02-01T10:32:43.639 DEBUG:boto:auth_path=/vclfcn-21/ 2024-02-01T10:32:43.639 DEBUG:boto:Method: HEAD 2024-02-01T10:32:43.639 DEBUG:boto:Path: /vclfcn-21/ 2024-02-01T10:32:43.639 DEBUG:boto:Data: 2024-02-01T10:32:43.639 DEBUG:boto:Headers: {} 2024-02-01T10:32:43.639 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:32:43.639 DEBUG:boto:Port: 8000 2024-02-01T10:32:43.639 DEBUG:boto:Params: {} 2024-02-01T10:32:43.640 DEBUG:boto:Token: None 2024-02-01T10:32:43.640 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:32:43 GMT /vclfcn-21/ 2024-02-01T10:32:43.640 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:dEA/BQ1DdOYX6/ZSV8b3Ycn6wUg= 2024-02-01T10:32:43.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:43 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dEA/BQ1DdOYX6/ZSV8b3Ycn6wUg=', 'Content-Length': '0'} 2024-02-01T10:32:43.642 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005ccde3a92f04f149-0065bb734b-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:32:43 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:43.643 DEBUG:boto:path=/vclfcn-21/obj 2024-02-01T10:32:43.643 DEBUG:boto:auth_path=/vclfcn-21/obj 2024-02-01T10:32:43.643 DEBUG:boto:Method: DELETE 2024-02-01T10:32:43.643 DEBUG:boto:Path: /vclfcn-21/obj 2024-02-01T10:32:43.643 DEBUG:boto:Data: 2024-02-01T10:32:43.643 DEBUG:boto:Headers: {} 2024-02-01T10:32:43.643 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:32:43.643 DEBUG:boto:Port: 8000 2024-02-01T10:32:43.643 DEBUG:boto:Params: {} 2024-02-01T10:32:43.644 DEBUG:boto:Token: None 2024-02-01T10:32:43.644 DEBUG:boto:StringToSign: DELETE Thu, 01 Feb 2024 10:32:43 GMT /vclfcn-21/obj 2024-02-01T10:32:43.644 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:c6pqtFL4ie2v9Qh1A9Z/DRhen78= 2024-02-01T10:32:43.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:43 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:c6pqtFL4ie2v9Qh1A9Z/DRhen78=', 'Content-Length': '0'} 2024-02-01T10:32:43.652 DEBUG:boto:Response headers: [('x-amz-version-id', 'iTuO-63D.1f6Od9rzTk7NLj1aX9iPCo'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx0000090e14ec7ac733c51-0065bb734b-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:43 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:43.652 DEBUG:boto:path=/vclfcn-22/ 2024-02-01T10:32:43.652 DEBUG:boto:auth_path=/vclfcn-22/ 2024-02-01T10:32:43.652 DEBUG:boto:Method: HEAD 2024-02-01T10:32:43.653 DEBUG:boto:Path: /vclfcn-22/ 2024-02-01T10:32:43.653 DEBUG:boto:Data: 2024-02-01T10:32:43.653 DEBUG:boto:Headers: {} 2024-02-01T10:32:43.653 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:32:43.653 DEBUG:boto:Port: 8000 2024-02-01T10:32:43.653 DEBUG:boto:Params: {} 2024-02-01T10:32:43.653 DEBUG:boto:Token: None 2024-02-01T10:32:43.653 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:32:43 GMT /vclfcn-22/ 2024-02-01T10:32:43.653 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:OVLMEjVr2jxZCtAXSfv5ViYNdp4= 2024-02-01T10:32:43.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:43 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:OVLMEjVr2jxZCtAXSfv5ViYNdp4=', 'Content-Length': '0'} 2024-02-01T10:32:43.657 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b39526873bd73f10-0065bb734b-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:32:43 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:43.658 DEBUG:boto:path=/vclfcn-22/obj 2024-02-01T10:32:43.658 DEBUG:boto:auth_path=/vclfcn-22/obj 2024-02-01T10:32:43.658 DEBUG:boto:Method: PUT 2024-02-01T10:32:43.658 DEBUG:boto:Path: /vclfcn-22/obj 2024-02-01T10:32:43.658 DEBUG:boto:Data: 2024-02-01T10:32:43.658 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-02-01T10:32:43.658 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:32:43.659 DEBUG:boto:Port: 8000 2024-02-01T10:32:43.659 DEBUG:boto:Params: {} 2024-02-01T10:32:43.659 DEBUG:boto:Token: None 2024-02-01T10:32:43.659 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Thu, 01 Feb 2024 10:32:43 GMT /vclfcn-22/obj 2024-02-01T10:32:43.659 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cYgWlVyoFjvFy+T5aPphV0Uyidc= 2024-02-01T10:32:43.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:32:43 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cYgWlVyoFjvFy+T5aPphV0Uyidc='} 2024-02-01T10:32:43.687 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB'), ('x-amz-request-id', 'tx0000081b4ecc913799db0-0065bb734b-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:43 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:43.687 DEBUG:boto:path=/vclfcn-22/ 2024-02-01T10:32:43.687 DEBUG:boto:auth_path=/vclfcn-22/ 2024-02-01T10:32:43.688 DEBUG:boto:Method: HEAD 2024-02-01T10:32:43.688 DEBUG:boto:Path: /vclfcn-22/ 2024-02-01T10:32:43.688 DEBUG:boto:Data: 2024-02-01T10:32:43.688 DEBUG:boto:Headers: {} 2024-02-01T10:32:43.688 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:32:43.688 DEBUG:boto:Port: 8000 2024-02-01T10:32:43.688 DEBUG:boto:Params: {} 2024-02-01T10:32:43.689 DEBUG:boto:Token: None 2024-02-01T10:32:43.689 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:32:43 GMT /vclfcn-22/ 2024-02-01T10:32:43.689 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:OVLMEjVr2jxZCtAXSfv5ViYNdp4= 2024-02-01T10:32:43.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:43 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:OVLMEjVr2jxZCtAXSfv5ViYNdp4=', 'Content-Length': '0'} 2024-02-01T10:32:43.693 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f6fdefd65725e6fd-0065bb734b-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:32:43 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:43.693 DEBUG:boto:path=/vclfcn-22/obj 2024-02-01T10:32:43.693 DEBUG:boto:auth_path=/vclfcn-22/obj 2024-02-01T10:32:43.694 DEBUG:boto:Method: DELETE 2024-02-01T10:32:43.694 DEBUG:boto:Path: /vclfcn-22/obj 2024-02-01T10:32:43.694 DEBUG:boto:Data: 2024-02-01T10:32:43.694 DEBUG:boto:Headers: {} 2024-02-01T10:32:43.694 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:32:43.694 DEBUG:boto:Port: 8000 2024-02-01T10:32:43.694 DEBUG:boto:Params: {} 2024-02-01T10:32:43.694 DEBUG:boto:Token: None 2024-02-01T10:32:43.694 DEBUG:boto:StringToSign: DELETE Thu, 01 Feb 2024 10:32:43 GMT /vclfcn-22/obj 2024-02-01T10:32:43.695 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:4y0LKdPxgI3nRal5j403tpZM4j8= 2024-02-01T10:32:43.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:43 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:4y0LKdPxgI3nRal5j403tpZM4j8=', 'Content-Length': '0'} 2024-02-01T10:32:43.711 DEBUG:boto:Response headers: [('x-amz-version-id', 'B4BhmM3thsoWYrVmsTC9qpVYsD42.EG'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx000003e59fd1792873431-0065bb734b-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:43 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:43.712 DEBUG:boto:path=/vclfcn-23/ 2024-02-01T10:32:43.712 DEBUG:boto:auth_path=/vclfcn-23/ 2024-02-01T10:32:43.712 DEBUG:boto:Method: HEAD 2024-02-01T10:32:43.712 DEBUG:boto:Path: /vclfcn-23/ 2024-02-01T10:32:43.712 DEBUG:boto:Data: 2024-02-01T10:32:43.712 DEBUG:boto:Headers: {} 2024-02-01T10:32:43.712 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:32:43.712 DEBUG:boto:Port: 8001 2024-02-01T10:32:43.712 DEBUG:boto:Params: {} 2024-02-01T10:32:43.713 DEBUG:boto:Token: None 2024-02-01T10:32:43.713 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:32:43 GMT /vclfcn-23/ 2024-02-01T10:32:43.713 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:F4qUxBJgGhML/dqffVmumq/sdFY= 2024-02-01T10:32:43.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:43 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:F4qUxBJgGhML/dqffVmumq/sdFY=', 'Content-Length': '0'} 2024-02-01T10:32:43.717 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d3522c7fa289849a-0065bb734b-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:32:43 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:43.718 DEBUG:boto:path=/vclfcn-23/obj 2024-02-01T10:32:43.718 DEBUG:boto:auth_path=/vclfcn-23/obj 2024-02-01T10:32:43.718 DEBUG:boto:Method: PUT 2024-02-01T10:32:43.718 DEBUG:boto:Path: /vclfcn-23/obj 2024-02-01T10:32:43.718 DEBUG:boto:Data: 2024-02-01T10:32:43.718 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-02-01T10:32:43.718 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:32:43.718 DEBUG:boto:Port: 8001 2024-02-01T10:32:43.719 DEBUG:boto:Params: {} 2024-02-01T10:32:43.719 DEBUG:boto:Token: None 2024-02-01T10:32:43.719 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Thu, 01 Feb 2024 10:32:43 GMT /vclfcn-23/obj 2024-02-01T10:32:43.719 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gq08TgN1iTUNFYn6n1OKbY+Byyw= 2024-02-01T10:32:43.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:32:43 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gq08TgN1iTUNFYn6n1OKbY+Byyw='} 2024-02-01T10:32:43.764 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'FL110tVslNs52SVUbHZqPNK10BnpUdZ'), ('x-amz-request-id', 'tx00000fc03b5aebf0ee639-0065bb734b-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:43 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:43.764 DEBUG:boto:path=/vclfcn-23/ 2024-02-01T10:32:43.765 DEBUG:boto:auth_path=/vclfcn-23/ 2024-02-01T10:32:43.765 DEBUG:boto:Method: HEAD 2024-02-01T10:32:43.765 DEBUG:boto:Path: /vclfcn-23/ 2024-02-01T10:32:43.765 DEBUG:boto:Data: 2024-02-01T10:32:43.765 DEBUG:boto:Headers: {} 2024-02-01T10:32:43.765 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:32:43.765 DEBUG:boto:Port: 8001 2024-02-01T10:32:43.765 DEBUG:boto:Params: {} 2024-02-01T10:32:43.765 DEBUG:boto:Token: None 2024-02-01T10:32:43.765 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:32:43 GMT /vclfcn-23/ 2024-02-01T10:32:43.766 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:F4qUxBJgGhML/dqffVmumq/sdFY= 2024-02-01T10:32:43.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:43 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:F4qUxBJgGhML/dqffVmumq/sdFY=', 'Content-Length': '0'} 2024-02-01T10:32:43.770 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003b14fcfb51869319-0065bb734b-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:32:43 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:43.770 DEBUG:boto:path=/vclfcn-23/obj 2024-02-01T10:32:43.770 DEBUG:boto:auth_path=/vclfcn-23/obj 2024-02-01T10:32:43.770 DEBUG:boto:Method: DELETE 2024-02-01T10:32:43.770 DEBUG:boto:Path: /vclfcn-23/obj 2024-02-01T10:32:43.770 DEBUG:boto:Data: 2024-02-01T10:32:43.770 DEBUG:boto:Headers: {} 2024-02-01T10:32:43.771 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:32:43.771 DEBUG:boto:Port: 8001 2024-02-01T10:32:43.771 DEBUG:boto:Params: {} 2024-02-01T10:32:43.771 DEBUG:boto:Token: None 2024-02-01T10:32:43.771 DEBUG:boto:StringToSign: DELETE Thu, 01 Feb 2024 10:32:43 GMT /vclfcn-23/obj 2024-02-01T10:32:43.771 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:U1UNkFgt2/tlApPrXjVxZN3PsN0= 2024-02-01T10:32:43.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:32:43 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:U1UNkFgt2/tlApPrXjVxZN3PsN0=', 'Content-Length': '0'} 2024-02-01T10:32:43.790 DEBUG:boto:Response headers: [('x-amz-version-id', 'cfAEDiw8G8l6M7HFA62MzKKoAdV-ERA'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx00000a7c0602eeed693d8-0065bb734b-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:32:43 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:32:43.790 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=vclfcn-21 2024-02-01T10:32:43.790 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:32:43.790 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:32:43.790 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket vclfcn-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-02-01T10:32:43.840 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:32:43.840 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:32:43.903 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:32:43.902+0000 7f889a1c2980 1 waiting to reach incremental sync.. 2024-02-01T10:32:48.907 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:32:48.906+0000 7f889a1c2980 1 bucket sync caught up with source: 2024-02-01T10:32:48.907 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, , , , , , , , 00000000003.32.12, , ] 2024-02-01T10:32:48.907 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, , , , , , , , 00000000003.32.12, , ] 2024-02-01T10:32:48.907 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:32:48.906+0000 7f889a1c2980 0 bucket checkpoint complete 2024-02-01T10:32:48.955 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=vclfcn-21 2024-02-01T10:32:48.955 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:32:48.955 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:32:48.956 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:32:49.009 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:32:49.009 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:32:49.068 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:32:49.073+0000 7f443e3c9980 1 waiting to reach incremental sync.. 2024-02-01T10:32:54.075 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:32:54.077+0000 7f443e3c9980 1 bucket sync caught up with source: 2024-02-01T10:32:54.075 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, , , , , , , , 00000000003.32.12, , ] 2024-02-01T10:32:54.075 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, , , , , , , , 00000000003.32.12, , ] 2024-02-01T10:32:54.076 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:32:54.077+0000 7f443e3c9980 0 bucket checkpoint complete 2024-02-01T10:32:54.106 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=vclfcn-21 2024-02-01T10:32:54.106 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:32:54.106 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:32:54.106 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-21 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:32:54.154 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:32:54.154 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:32:54.214 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:32:54.221+0000 7f22fdcde980 1 waiting to reach incremental sync.. 2024-02-01T10:32:59.216 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:32:59.220+0000 7f22fdcde980 1 waiting to reach incremental sync.. 2024-02-01T10:33:04.217 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:33:04.220+0000 7f22fdcde980 1 waiting to reach incremental sync.. 2024-02-01T10:33:09.220 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:33:09.224+0000 7f22fdcde980 1 bucket sync caught up with source: 2024-02-01T10:33:09.220 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, , , , , , , , 00000000002.88.6, , ] 2024-02-01T10:33:09.221 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, , , , , , , , 00000000002.88.6, , ] 2024-02-01T10:33:09.221 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:33:09.224+0000 7f22fdcde980 0 bucket checkpoint complete 2024-02-01T10:33:09.261 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=vclfcn-21 2024-02-01T10:33:09.261 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:33:09.261 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:33:09.262 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-21 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:33:09.309 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:33:09.309 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:33:09.370 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:33:09.376+0000 7fc6e43bc980 1 bucket sync caught up with source: 2024-02-01T10:33:09.370 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, , , , , , , , 00000000002.88.6, , ] 2024-02-01T10:33:09.370 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, , , , , , , , 00000000002.88.6, , ] 2024-02-01T10:33:09.371 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:33:09.376+0000 7fc6e43bc980 0 bucket checkpoint complete 2024-02-01T10:33:09.393 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=vclfcn-21 2024-02-01T10:33:09.393 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:33:09.393 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:33:09.393 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-21 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:33:09.434 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:33:09.434 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:33:09.492 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:33:09.496+0000 7f406045c980 1 bucket sync caught up with source: 2024-02-01T10:33:09.492 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, , , , , , , , 00000000002.100.6, , ] 2024-02-01T10:33:09.492 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, , , , , , , , 00000000002.100.6, , ] 2024-02-01T10:33:09.492 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:33:09.496+0000 7f406045c980 0 bucket checkpoint complete 2024-02-01T10:33:09.515 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=vclfcn-21 2024-02-01T10:33:09.515 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:33:09.516 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:33:09.516 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket vclfcn-21 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-02-01T10:33:09.566 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:33:09.567 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:33:09.629 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:33:09.629+0000 7f30e31ec980 1 bucket sync caught up with source: 2024-02-01T10:33:09.630 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, , , , , , , , 00000000002.100.6, , ] 2024-02-01T10:33:09.630 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, , , , , , , , 00000000002.100.6, , ] 2024-02-01T10:33:09.630 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:33:09.629+0000 7f30e31ec980 0 bucket checkpoint complete 2024-02-01T10:33:09.652 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-21 zones={test-zone2, test-zone1} 2024-02-01T10:33:09.816 DEBUG:boto:path=/vclfcn-21/ 2024-02-01T10:33:09.816 DEBUG:boto:auth_path=/vclfcn-21/ 2024-02-01T10:33:09.816 DEBUG:boto:Method: HEAD 2024-02-01T10:33:09.816 DEBUG:boto:Path: /vclfcn-21/ 2024-02-01T10:33:09.816 DEBUG:boto:Data: 2024-02-01T10:33:09.817 DEBUG:boto:Headers: {} 2024-02-01T10:33:09.817 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:33:09.817 DEBUG:boto:Port: 8000 2024-02-01T10:33:09.817 DEBUG:boto:Params: {} 2024-02-01T10:33:09.817 DEBUG:boto:Token: None 2024-02-01T10:33:09.817 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:33:09 GMT /vclfcn-21/ 2024-02-01T10:33:09.817 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:V3No3QkqgEM0QYAOyvhMSyGbDfE= 2024-02-01T10:33:09.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:V3No3QkqgEM0QYAOyvhMSyGbDfE=', 'Content-Length': '0'} 2024-02-01T10:33:09.824 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e85db7f6a03e113c-0065bb7365-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:33:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:09.824 DEBUG:boto:path=/vclfcn-21/ 2024-02-01T10:33:09.824 DEBUG:boto:auth_path=/vclfcn-21/ 2024-02-01T10:33:09.824 DEBUG:boto:Method: HEAD 2024-02-01T10:33:09.824 DEBUG:boto:Path: /vclfcn-21/ 2024-02-01T10:33:09.824 DEBUG:boto:Data: 2024-02-01T10:33:09.824 DEBUG:boto:Headers: {} 2024-02-01T10:33:09.824 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:33:09.824 DEBUG:boto:Port: 8000 2024-02-01T10:33:09.825 DEBUG:boto:Params: {} 2024-02-01T10:33:09.825 DEBUG:boto:Token: None 2024-02-01T10:33:09.825 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:33:09 GMT /vclfcn-21/ 2024-02-01T10:33:09.825 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:V3No3QkqgEM0QYAOyvhMSyGbDfE= 2024-02-01T10:33:09.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:V3No3QkqgEM0QYAOyvhMSyGbDfE=', 'Content-Length': '0'} 2024-02-01T10:33:09.829 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000071816a1e88b804fc-0065bb7365-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:33:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:09.829 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:33:09.830 DEBUG:boto:path=/vclfcn-21/ 2024-02-01T10:33:09.830 DEBUG:boto:auth_path=/vclfcn-21/ 2024-02-01T10:33:09.830 DEBUG:boto:path=/vclfcn-21/?versions&max-keys=999 2024-02-01T10:33:09.830 DEBUG:boto:auth_path=/vclfcn-21/?versions&max-keys=999 2024-02-01T10:33:09.830 DEBUG:boto:Method: GET 2024-02-01T10:33:09.830 DEBUG:boto:Path: /vclfcn-21/?versions&max-keys=999 2024-02-01T10:33:09.830 DEBUG:boto:Data: 2024-02-01T10:33:09.830 DEBUG:boto:Headers: {} 2024-02-01T10:33:09.830 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:33:09.830 DEBUG:boto:Port: 8000 2024-02-01T10:33:09.830 DEBUG:boto:Params: {} 2024-02-01T10:33:09.831 DEBUG:boto:Token: None 2024-02-01T10:33:09.831 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:09 GMT /vclfcn-21/?versions 2024-02-01T10:33:09.831 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/HITi1o1v8xNVkIRp2zysDUe8sU= 2024-02-01T10:33:09.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/HITi1o1v8xNVkIRp2zysDUe8sU=', 'Content-Length': '0'} 2024-02-01T10:33:09.835 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad745272a9d32bb4-0065bb7365-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:09.835 DEBUG:boto:b'vclfcn-21999falseobjiTuO-63D.1f6Od9rzTk7NLj1aX9iPCotrue2024-02-01T10:32:43.656Zrgw-multisite-test-userMultisite Test UserNormalobj6pTCwvsER73lrwuF6KIU7A5eJ1YCuvYfalse2024-02-01T10:32:43.638Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:33:09.836 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:33:09.836 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:33:09.836 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:33:09.836 DEBUG:boto:path=/vclfcn-21/ 2024-02-01T10:33:09.836 DEBUG:boto:auth_path=/vclfcn-21/ 2024-02-01T10:33:09.836 DEBUG:boto:path=/vclfcn-21/?versions&max-keys=999 2024-02-01T10:33:09.836 DEBUG:boto:auth_path=/vclfcn-21/?versions&max-keys=999 2024-02-01T10:33:09.836 DEBUG:boto:Method: GET 2024-02-01T10:33:09.837 DEBUG:boto:Path: /vclfcn-21/?versions&max-keys=999 2024-02-01T10:33:09.837 DEBUG:boto:Data: 2024-02-01T10:33:09.837 DEBUG:boto:Headers: {} 2024-02-01T10:33:09.837 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:33:09.837 DEBUG:boto:Port: 8000 2024-02-01T10:33:09.837 DEBUG:boto:Params: {} 2024-02-01T10:33:09.837 DEBUG:boto:Token: None 2024-02-01T10:33:09.837 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:09 GMT /vclfcn-21/?versions 2024-02-01T10:33:09.838 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/HITi1o1v8xNVkIRp2zysDUe8sU= 2024-02-01T10:33:09.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/HITi1o1v8xNVkIRp2zysDUe8sU=', 'Content-Length': '0'} 2024-02-01T10:33:09.842 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ecdc69efc1eff2d2-0065bb7365-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:09.884 DEBUG:boto:b'vclfcn-21999falseobjiTuO-63D.1f6Od9rzTk7NLj1aX9iPCotrue2024-02-01T10:32:43.656Zrgw-multisite-test-userMultisite Test UserNormalobj6pTCwvsER73lrwuF6KIU7A5eJ1YCuvYfalse2024-02-01T10:32:43.638Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:33:09.885 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:33:09.885 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:33:09.885 DEBUG:boto:path=/vclfcn-21/ 2024-02-01T10:33:09.885 DEBUG:boto:auth_path=/vclfcn-21/ 2024-02-01T10:33:09.885 DEBUG:boto:path=/vclfcn-21/?versions&max-keys=999 2024-02-01T10:33:09.885 DEBUG:boto:auth_path=/vclfcn-21/?versions&max-keys=999 2024-02-01T10:33:09.885 DEBUG:boto:Method: GET 2024-02-01T10:33:09.885 DEBUG:boto:Path: /vclfcn-21/?versions&max-keys=999 2024-02-01T10:33:09.885 DEBUG:boto:Data: 2024-02-01T10:33:09.886 DEBUG:boto:Headers: {} 2024-02-01T10:33:09.886 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:33:09.886 DEBUG:boto:Port: 8000 2024-02-01T10:33:09.886 DEBUG:boto:Params: {} 2024-02-01T10:33:09.886 DEBUG:boto:Token: None 2024-02-01T10:33:09.886 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:09 GMT /vclfcn-21/?versions 2024-02-01T10:33:09.886 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/HITi1o1v8xNVkIRp2zysDUe8sU= 2024-02-01T10:33:09.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/HITi1o1v8xNVkIRp2zysDUe8sU=', 'Content-Length': '0'} 2024-02-01T10:33:09.890 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e72276f6f2da879e-0065bb7365-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:09.891 DEBUG:boto:b'vclfcn-21999falseobjiTuO-63D.1f6Od9rzTk7NLj1aX9iPCotrue2024-02-01T10:32:43.656Zrgw-multisite-test-userMultisite Test UserNormalobj6pTCwvsER73lrwuF6KIU7A5eJ1YCuvYfalse2024-02-01T10:32:43.638Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:33:09.891 DEBUG:boto:path=/vclfcn-21/ 2024-02-01T10:33:09.891 DEBUG:boto:auth_path=/vclfcn-21/ 2024-02-01T10:33:09.891 DEBUG:boto:path=/vclfcn-21/?versions&max-keys=999 2024-02-01T10:33:09.891 DEBUG:boto:auth_path=/vclfcn-21/?versions&max-keys=999 2024-02-01T10:33:09.891 DEBUG:boto:Method: GET 2024-02-01T10:33:09.892 DEBUG:boto:Path: /vclfcn-21/?versions&max-keys=999 2024-02-01T10:33:09.892 DEBUG:boto:Data: 2024-02-01T10:33:09.892 DEBUG:boto:Headers: {} 2024-02-01T10:33:09.892 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:33:09.892 DEBUG:boto:Port: 8000 2024-02-01T10:33:09.892 DEBUG:boto:Params: {} 2024-02-01T10:33:09.892 DEBUG:boto:Token: None 2024-02-01T10:33:09.892 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:09 GMT /vclfcn-21/?versions 2024-02-01T10:33:09.892 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/HITi1o1v8xNVkIRp2zysDUe8sU= 2024-02-01T10:33:09.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/HITi1o1v8xNVkIRp2zysDUe8sU=', 'Content-Length': '0'} 2024-02-01T10:33:09.897 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008bc5b84e9479cd1a-0065bb7365-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:09.897 DEBUG:boto:b'vclfcn-21999falseobjiTuO-63D.1f6Od9rzTk7NLj1aX9iPCotrue2024-02-01T10:32:43.656Zrgw-multisite-test-userMultisite Test UserNormalobj6pTCwvsER73lrwuF6KIU7A5eJ1YCuvYfalse2024-02-01T10:32:43.638Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:33:09.897 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:33:09.897 DEBUG:boto:path=/vclfcn-21/obj 2024-02-01T10:33:09.897 DEBUG:boto:auth_path=/vclfcn-21/obj 2024-02-01T10:33:09.898 DEBUG:boto:Method: HEAD 2024-02-01T10:33:09.898 DEBUG:boto:Path: /vclfcn-21/obj 2024-02-01T10:33:09.898 DEBUG:boto:Data: 2024-02-01T10:33:09.898 DEBUG:boto:Headers: {} 2024-02-01T10:33:09.898 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:33:09.898 DEBUG:boto:Port: 8000 2024-02-01T10:33:09.898 DEBUG:boto:Params: {} 2024-02-01T10:33:09.898 DEBUG:boto:Token: None 2024-02-01T10:33:09.898 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:33:09 GMT /vclfcn-21/obj 2024-02-01T10:33:09.898 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:oOuc9cbiVWoJb1SabvszPD2U2CY= 2024-02-01T10:33:09.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:oOuc9cbiVWoJb1SabvszPD2U2CY=', 'Content-Length': '0'} 2024-02-01T10:33:09.902 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000003e07078786e34696-0065bb7365-4307-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:09.902 DEBUG:boto:path=/vclfcn-21/obj 2024-02-01T10:33:09.902 DEBUG:boto:auth_path=/vclfcn-21/obj 2024-02-01T10:33:09.902 DEBUG:boto:Method: HEAD 2024-02-01T10:33:09.902 DEBUG:boto:Path: /vclfcn-21/obj 2024-02-01T10:33:09.902 DEBUG:boto:Data: 2024-02-01T10:33:09.902 DEBUG:boto:Headers: {} 2024-02-01T10:33:09.903 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:33:09.903 DEBUG:boto:Port: 8000 2024-02-01T10:33:09.903 DEBUG:boto:Params: {} 2024-02-01T10:33:09.903 DEBUG:boto:Token: None 2024-02-01T10:33:09.903 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:33:09 GMT /vclfcn-21/obj 2024-02-01T10:33:09.903 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:oOuc9cbiVWoJb1SabvszPD2U2CY= 2024-02-01T10:33:09.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:oOuc9cbiVWoJb1SabvszPD2U2CY=', 'Content-Length': '0'} 2024-02-01T10:33:09.906 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000007b8ab9a5331b214e-0065bb7365-4351-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:09.906 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:33:09.907 DEBUG:boto:path=/vclfcn-21/obj 2024-02-01T10:33:09.907 DEBUG:boto:auth_path=/vclfcn-21/obj 2024-02-01T10:33:09.907 DEBUG:boto:path=/vclfcn-21/obj?versionId=6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY 2024-02-01T10:33:09.907 DEBUG:boto:auth_path=/vclfcn-21/obj?versionId=6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY 2024-02-01T10:33:09.907 DEBUG:boto:Method: GET 2024-02-01T10:33:09.907 DEBUG:boto:Path: /vclfcn-21/obj?versionId=6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY 2024-02-01T10:33:09.907 DEBUG:boto:Data: 2024-02-01T10:33:09.907 DEBUG:boto:Headers: {} 2024-02-01T10:33:09.907 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:33:09.907 DEBUG:boto:Port: 8000 2024-02-01T10:33:09.907 DEBUG:boto:Params: {} 2024-02-01T10:33:09.908 DEBUG:boto:Token: None 2024-02-01T10:33:09.908 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:09 GMT /vclfcn-21/obj?versionId=6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY 2024-02-01T10:33:09.908 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RqFjkGy47A4TILqlNIkvSHkQi8Y= 2024-02-01T10:33:09.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RqFjkGy47A4TILqlNIkvSHkQi8Y=', 'Content-Length': '0'} 2024-02-01T10:33:09.911 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:32:43 GMT'), ('x-amz-version-id', '6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002c3474e61efc2d1f-0065bb7365-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:09.911 DEBUG:boto:path=/vclfcn-21/obj 2024-02-01T10:33:09.911 DEBUG:boto:auth_path=/vclfcn-21/obj 2024-02-01T10:33:09.911 DEBUG:boto:path=/vclfcn-21/obj?versionId=6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY 2024-02-01T10:33:09.911 DEBUG:boto:auth_path=/vclfcn-21/obj?versionId=6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY 2024-02-01T10:33:09.912 DEBUG:boto:Method: GET 2024-02-01T10:33:09.912 DEBUG:boto:Path: /vclfcn-21/obj?versionId=6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY 2024-02-01T10:33:09.912 DEBUG:boto:Data: 2024-02-01T10:33:09.912 DEBUG:boto:Headers: {} 2024-02-01T10:33:09.912 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:33:09.912 DEBUG:boto:Port: 8000 2024-02-01T10:33:09.912 DEBUG:boto:Params: {} 2024-02-01T10:33:09.912 DEBUG:boto:Token: None 2024-02-01T10:33:09.912 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:09 GMT /vclfcn-21/obj?versionId=6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY 2024-02-01T10:33:09.913 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RqFjkGy47A4TILqlNIkvSHkQi8Y= 2024-02-01T10:33:09.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RqFjkGy47A4TILqlNIkvSHkQi8Y=', 'Content-Length': '0'} 2024-02-01T10:33:09.915 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:32:43 GMT'), ('x-amz-version-id', '6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003e1ce3403ca47527-0065bb7365-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:09.915 DEBUG:boto:path=/vclfcn-21/obj 2024-02-01T10:33:09.915 DEBUG:boto:auth_path=/vclfcn-21/obj 2024-02-01T10:33:09.915 DEBUG:boto:path=/vclfcn-21/obj?versionId=6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY 2024-02-01T10:33:09.916 DEBUG:boto:auth_path=/vclfcn-21/obj?versionId=6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY 2024-02-01T10:33:09.916 DEBUG:boto:Method: HEAD 2024-02-01T10:33:09.916 DEBUG:boto:Path: /vclfcn-21/obj?versionId=6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY 2024-02-01T10:33:09.916 DEBUG:boto:Data: 2024-02-01T10:33:09.916 DEBUG:boto:Headers: {} 2024-02-01T10:33:09.916 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:33:09.916 DEBUG:boto:Port: 8000 2024-02-01T10:33:09.916 DEBUG:boto:Params: {} 2024-02-01T10:33:09.916 DEBUG:boto:Token: None 2024-02-01T10:33:09.916 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:33:09 GMT /vclfcn-21/obj?versionId=6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY 2024-02-01T10:33:09.917 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+iDcbxbttdlQ+7nSKyWCu2+wF5c= 2024-02-01T10:33:09.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+iDcbxbttdlQ+7nSKyWCu2+wF5c=', 'Content-Length': '0'} 2024-02-01T10:33:09.919 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:32:43 GMT'), ('x-amz-version-id', '6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002c71a77e0190954e-0065bb7365-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:09.920 DEBUG:boto:path=/vclfcn-21/obj 2024-02-01T10:33:09.920 DEBUG:boto:auth_path=/vclfcn-21/obj 2024-02-01T10:33:09.920 DEBUG:boto:path=/vclfcn-21/obj?versionId=6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY 2024-02-01T10:33:09.920 DEBUG:boto:auth_path=/vclfcn-21/obj?versionId=6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY 2024-02-01T10:33:09.920 DEBUG:boto:Method: HEAD 2024-02-01T10:33:09.920 DEBUG:boto:Path: /vclfcn-21/obj?versionId=6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY 2024-02-01T10:33:09.920 DEBUG:boto:Data: 2024-02-01T10:33:09.920 DEBUG:boto:Headers: {} 2024-02-01T10:33:09.921 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:33:09.921 DEBUG:boto:Port: 8000 2024-02-01T10:33:09.921 DEBUG:boto:Params: {} 2024-02-01T10:33:09.921 DEBUG:boto:Token: None 2024-02-01T10:33:09.921 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:33:09 GMT /vclfcn-21/obj?versionId=6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY 2024-02-01T10:33:09.921 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+iDcbxbttdlQ+7nSKyWCu2+wF5c= 2024-02-01T10:33:09.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+iDcbxbttdlQ+7nSKyWCu2+wF5c=', 'Content-Length': '0'} 2024-02-01T10:33:09.924 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:32:43 GMT'), ('x-amz-version-id', '6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a9154349419688b9-0065bb7365-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:09.924 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:33:09.924 DEBUG:boto:path=/vclfcn-21/obj 2024-02-01T10:33:09.924 DEBUG:boto:auth_path=/vclfcn-21/obj 2024-02-01T10:33:09.924 DEBUG:boto:path=/vclfcn-21/obj?versionId=6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY 2024-02-01T10:33:09.924 DEBUG:boto:auth_path=/vclfcn-21/obj?versionId=6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY 2024-02-01T10:33:09.925 DEBUG:boto:Method: GET 2024-02-01T10:33:09.925 DEBUG:boto:Path: /vclfcn-21/obj?versionId=6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY 2024-02-01T10:33:09.925 DEBUG:boto:Data: 2024-02-01T10:33:09.925 DEBUG:boto:Headers: {} 2024-02-01T10:33:09.925 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:33:09.925 DEBUG:boto:Port: 8000 2024-02-01T10:33:09.925 DEBUG:boto:Params: {} 2024-02-01T10:33:09.925 DEBUG:boto:Token: None 2024-02-01T10:33:09.925 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:09 GMT /vclfcn-21/obj?versionId=6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY 2024-02-01T10:33:09.925 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RqFjkGy47A4TILqlNIkvSHkQi8Y= 2024-02-01T10:33:09.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RqFjkGy47A4TILqlNIkvSHkQi8Y=', 'Content-Length': '0'} 2024-02-01T10:33:09.929 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:32:43 GMT'), ('x-amz-version-id', '6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007c021d05ed783d2d-0065bb7365-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:09.929 DEBUG:boto:path=/vclfcn-21/obj 2024-02-01T10:33:09.930 DEBUG:boto:auth_path=/vclfcn-21/obj 2024-02-01T10:33:09.930 DEBUG:boto:path=/vclfcn-21/obj?versionId=6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY 2024-02-01T10:33:09.930 DEBUG:boto:auth_path=/vclfcn-21/obj?versionId=6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY 2024-02-01T10:33:09.930 DEBUG:boto:Method: GET 2024-02-01T10:33:09.930 DEBUG:boto:Path: /vclfcn-21/obj?versionId=6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY 2024-02-01T10:33:09.930 DEBUG:boto:Data: 2024-02-01T10:33:09.930 DEBUG:boto:Headers: {} 2024-02-01T10:33:09.930 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:33:09.930 DEBUG:boto:Port: 8000 2024-02-01T10:33:09.930 DEBUG:boto:Params: {} 2024-02-01T10:33:09.930 DEBUG:boto:Token: None 2024-02-01T10:33:09.931 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:09 GMT /vclfcn-21/obj?versionId=6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY 2024-02-01T10:33:09.931 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RqFjkGy47A4TILqlNIkvSHkQi8Y= 2024-02-01T10:33:09.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RqFjkGy47A4TILqlNIkvSHkQi8Y=', 'Content-Length': '0'} 2024-02-01T10:33:09.934 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:32:43 GMT'), ('x-amz-version-id', '6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000015a0e0f4139e3a5d-0065bb7365-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:09.934 DEBUG:boto:path=/vclfcn-21/ 2024-02-01T10:33:09.934 DEBUG:boto:auth_path=/vclfcn-21/ 2024-02-01T10:33:09.934 DEBUG:boto:path=/vclfcn-21/?prefix=obj 2024-02-01T10:33:09.934 DEBUG:boto:auth_path=/vclfcn-21/?prefix=obj 2024-02-01T10:33:09.934 DEBUG:boto:Method: GET 2024-02-01T10:33:09.934 DEBUG:boto:Path: /vclfcn-21/?prefix=obj 2024-02-01T10:33:09.934 DEBUG:boto:Data: 2024-02-01T10:33:09.935 DEBUG:boto:Headers: {} 2024-02-01T10:33:09.935 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:33:09.935 DEBUG:boto:Port: 8000 2024-02-01T10:33:09.935 DEBUG:boto:Params: {} 2024-02-01T10:33:09.935 DEBUG:boto:Token: None 2024-02-01T10:33:09.935 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:09 GMT /vclfcn-21/ 2024-02-01T10:33:09.935 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:TX9/XUYORwDpK1LYS0c2XINOI2E= 2024-02-01T10:33:09.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:TX9/XUYORwDpK1LYS0c2XINOI2E=', 'Content-Length': '0'} 2024-02-01T10:33:09.939 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8c5c4a57cce42a5-0065bb7365-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:09.980 DEBUG:boto:b'vclfcn-21obj1000false' 2024-02-01T10:33:09.981 DEBUG:boto:path=/vclfcn-21/ 2024-02-01T10:33:09.981 DEBUG:boto:auth_path=/vclfcn-21/ 2024-02-01T10:33:09.981 DEBUG:boto:path=/vclfcn-21/?prefix=obj 2024-02-01T10:33:09.981 DEBUG:boto:auth_path=/vclfcn-21/?prefix=obj 2024-02-01T10:33:09.981 DEBUG:boto:Method: GET 2024-02-01T10:33:09.981 DEBUG:boto:Path: /vclfcn-21/?prefix=obj 2024-02-01T10:33:09.981 DEBUG:boto:Data: 2024-02-01T10:33:09.981 DEBUG:boto:Headers: {} 2024-02-01T10:33:09.981 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:33:09.981 DEBUG:boto:Port: 8000 2024-02-01T10:33:09.981 DEBUG:boto:Params: {} 2024-02-01T10:33:09.982 DEBUG:boto:Token: None 2024-02-01T10:33:09.982 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:09 GMT /vclfcn-21/ 2024-02-01T10:33:09.982 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:TX9/XUYORwDpK1LYS0c2XINOI2E= 2024-02-01T10:33:09.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:TX9/XUYORwDpK1LYS0c2XINOI2E=', 'Content-Length': '0'} 2024-02-01T10:33:09.986 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e983bd17e9dbb299-0065bb7365-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:09.986 DEBUG:boto:b'vclfcn-21obj1000false' 2024-02-01T10:33:09.986 DEBUG:boto:path=/vclfcn-21/obj 2024-02-01T10:33:09.987 DEBUG:boto:auth_path=/vclfcn-21/obj 2024-02-01T10:33:09.987 DEBUG:boto:Method: HEAD 2024-02-01T10:33:09.987 DEBUG:boto:Path: /vclfcn-21/obj 2024-02-01T10:33:09.987 DEBUG:boto:Data: 2024-02-01T10:33:09.987 DEBUG:boto:Headers: {} 2024-02-01T10:33:09.987 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:33:09.987 DEBUG:boto:Port: 8000 2024-02-01T10:33:09.987 DEBUG:boto:Params: {} 2024-02-01T10:33:09.987 DEBUG:boto:Token: None 2024-02-01T10:33:09.987 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:33:09 GMT /vclfcn-21/obj 2024-02-01T10:33:09.988 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:oOuc9cbiVWoJb1SabvszPD2U2CY= 2024-02-01T10:33:09.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:oOuc9cbiVWoJb1SabvszPD2U2CY=', 'Content-Length': '0'} 2024-02-01T10:33:09.991 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000004e0f6ab0f40ec0d4-0065bb7365-4307-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:09.991 DEBUG:boto:path=/vclfcn-21/obj 2024-02-01T10:33:09.991 DEBUG:boto:auth_path=/vclfcn-21/obj 2024-02-01T10:33:09.992 DEBUG:boto:Method: HEAD 2024-02-01T10:33:09.992 DEBUG:boto:Path: /vclfcn-21/obj 2024-02-01T10:33:09.992 DEBUG:boto:Data: 2024-02-01T10:33:09.992 DEBUG:boto:Headers: {} 2024-02-01T10:33:09.992 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:33:09.992 DEBUG:boto:Port: 8000 2024-02-01T10:33:09.992 DEBUG:boto:Params: {} 2024-02-01T10:33:09.992 DEBUG:boto:Token: None 2024-02-01T10:33:09.992 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:33:09 GMT /vclfcn-21/obj 2024-02-01T10:33:09.992 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:oOuc9cbiVWoJb1SabvszPD2U2CY= 2024-02-01T10:33:09.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:oOuc9cbiVWoJb1SabvszPD2U2CY=', 'Content-Length': '0'} 2024-02-01T10:33:09.996 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000e9197ec5ed41b878-0065bb7366-4351-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:09.996 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-21 zones={test-zone2, test-zone1} 2024-02-01T10:33:09.996 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-21 zones={test-zone3, test-zone1} 2024-02-01T10:33:09.996 DEBUG:boto:path=/vclfcn-21/ 2024-02-01T10:33:09.996 DEBUG:boto:auth_path=/vclfcn-21/ 2024-02-01T10:33:09.996 DEBUG:boto:Method: HEAD 2024-02-01T10:33:09.996 DEBUG:boto:Path: /vclfcn-21/ 2024-02-01T10:33:09.996 DEBUG:boto:Data: 2024-02-01T10:33:09.997 DEBUG:boto:Headers: {} 2024-02-01T10:33:09.997 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:33:09.997 DEBUG:boto:Port: 8001 2024-02-01T10:33:09.997 DEBUG:boto:Params: {} 2024-02-01T10:33:09.997 DEBUG:boto:Token: None 2024-02-01T10:33:09.997 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:33:09 GMT /vclfcn-21/ 2024-02-01T10:33:09.997 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:V3No3QkqgEM0QYAOyvhMSyGbDfE= 2024-02-01T10:33:09.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:V3No3QkqgEM0QYAOyvhMSyGbDfE=', 'Content-Length': '0'} 2024-02-01T10:33:10.001 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b6c25eb64aa98ce8-0065bb7366-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:33:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:10.001 DEBUG:boto:path=/vclfcn-21/ 2024-02-01T10:33:10.001 DEBUG:boto:auth_path=/vclfcn-21/ 2024-02-01T10:33:10.001 DEBUG:boto:Method: HEAD 2024-02-01T10:33:10.001 DEBUG:boto:Path: /vclfcn-21/ 2024-02-01T10:33:10.001 DEBUG:boto:Data: 2024-02-01T10:33:10.002 DEBUG:boto:Headers: {} 2024-02-01T10:33:10.002 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:33:10.002 DEBUG:boto:Port: 8000 2024-02-01T10:33:10.002 DEBUG:boto:Params: {} 2024-02-01T10:33:10.002 DEBUG:boto:Token: None 2024-02-01T10:33:10.002 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:33:10 GMT /vclfcn-21/ 2024-02-01T10:33:10.002 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:DysXSP4lMA2gmpOBcAWW7at4ZNA= 2024-02-01T10:33:10.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:DysXSP4lMA2gmpOBcAWW7at4ZNA=', 'Content-Length': '0'} 2024-02-01T10:33:10.005 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008c0ff605278407a7-0065bb7366-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:33:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:10.006 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:33:10.006 DEBUG:boto:path=/vclfcn-21/ 2024-02-01T10:33:10.006 DEBUG:boto:auth_path=/vclfcn-21/ 2024-02-01T10:33:10.006 DEBUG:boto:path=/vclfcn-21/?versions&max-keys=999 2024-02-01T10:33:10.006 DEBUG:boto:auth_path=/vclfcn-21/?versions&max-keys=999 2024-02-01T10:33:10.006 DEBUG:boto:Method: GET 2024-02-01T10:33:10.006 DEBUG:boto:Path: /vclfcn-21/?versions&max-keys=999 2024-02-01T10:33:10.006 DEBUG:boto:Data: 2024-02-01T10:33:10.006 DEBUG:boto:Headers: {} 2024-02-01T10:33:10.007 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:33:10.007 DEBUG:boto:Port: 8001 2024-02-01T10:33:10.007 DEBUG:boto:Params: {} 2024-02-01T10:33:10.007 DEBUG:boto:Token: None 2024-02-01T10:33:10.007 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:10 GMT /vclfcn-21/?versions 2024-02-01T10:33:10.007 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tp2xoatwOxfDWEaUbuz+7jzyPfU= 2024-02-01T10:33:10.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tp2xoatwOxfDWEaUbuz+7jzyPfU=', 'Content-Length': '0'} 2024-02-01T10:33:10.010 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d421252d479e71bb-0065bb7366-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:10.056 DEBUG:boto:b'vclfcn-21999falseobjiTuO-63D.1f6Od9rzTk7NLj1aX9iPCotrue2024-02-01T10:32:43.656Zrgw-multisite-test-userMultisite Test UserNormalobj6pTCwvsER73lrwuF6KIU7A5eJ1YCuvYfalse2024-02-01T10:32:43.638Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:33:10.057 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:33:10.057 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:33:10.057 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:33:10.057 DEBUG:boto:path=/vclfcn-21/ 2024-02-01T10:33:10.057 DEBUG:boto:auth_path=/vclfcn-21/ 2024-02-01T10:33:10.057 DEBUG:boto:path=/vclfcn-21/?versions&max-keys=999 2024-02-01T10:33:10.057 DEBUG:boto:auth_path=/vclfcn-21/?versions&max-keys=999 2024-02-01T10:33:10.057 DEBUG:boto:Method: GET 2024-02-01T10:33:10.058 DEBUG:boto:Path: /vclfcn-21/?versions&max-keys=999 2024-02-01T10:33:10.058 DEBUG:boto:Data: 2024-02-01T10:33:10.058 DEBUG:boto:Headers: {} 2024-02-01T10:33:10.058 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:33:10.058 DEBUG:boto:Port: 8000 2024-02-01T10:33:10.058 DEBUG:boto:Params: {} 2024-02-01T10:33:10.058 DEBUG:boto:Token: None 2024-02-01T10:33:10.058 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:10 GMT /vclfcn-21/?versions 2024-02-01T10:33:10.058 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tp2xoatwOxfDWEaUbuz+7jzyPfU= 2024-02-01T10:33:10.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tp2xoatwOxfDWEaUbuz+7jzyPfU=', 'Content-Length': '0'} 2024-02-01T10:33:10.064 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092806bed105bdb52-0065bb7366-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:10.064 DEBUG:boto:b'vclfcn-21999falseobjiTuO-63D.1f6Od9rzTk7NLj1aX9iPCotrue2024-02-01T10:32:43.656Zrgw-multisite-test-userMultisite Test UserNormalobj6pTCwvsER73lrwuF6KIU7A5eJ1YCuvYfalse2024-02-01T10:32:43.638Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:33:10.064 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:33:10.065 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:33:10.065 DEBUG:boto:path=/vclfcn-21/ 2024-02-01T10:33:10.065 DEBUG:boto:auth_path=/vclfcn-21/ 2024-02-01T10:33:10.065 DEBUG:boto:path=/vclfcn-21/?versions&max-keys=999 2024-02-01T10:33:10.065 DEBUG:boto:auth_path=/vclfcn-21/?versions&max-keys=999 2024-02-01T10:33:10.065 DEBUG:boto:Method: GET 2024-02-01T10:33:10.065 DEBUG:boto:Path: /vclfcn-21/?versions&max-keys=999 2024-02-01T10:33:10.065 DEBUG:boto:Data: 2024-02-01T10:33:10.065 DEBUG:boto:Headers: {} 2024-02-01T10:33:10.065 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:33:10.066 DEBUG:boto:Port: 8001 2024-02-01T10:33:10.066 DEBUG:boto:Params: {} 2024-02-01T10:33:10.066 DEBUG:boto:Token: None 2024-02-01T10:33:10.066 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:10 GMT /vclfcn-21/?versions 2024-02-01T10:33:10.066 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tp2xoatwOxfDWEaUbuz+7jzyPfU= 2024-02-01T10:33:10.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tp2xoatwOxfDWEaUbuz+7jzyPfU=', 'Content-Length': '0'} 2024-02-01T10:33:10.070 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000afae35e1edf0fab8-0065bb7366-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:10.070 DEBUG:boto:b'vclfcn-21999falseobjiTuO-63D.1f6Od9rzTk7NLj1aX9iPCotrue2024-02-01T10:32:43.656Zrgw-multisite-test-userMultisite Test UserNormalobj6pTCwvsER73lrwuF6KIU7A5eJ1YCuvYfalse2024-02-01T10:32:43.638Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:33:10.071 DEBUG:boto:path=/vclfcn-21/ 2024-02-01T10:33:10.071 DEBUG:boto:auth_path=/vclfcn-21/ 2024-02-01T10:33:10.071 DEBUG:boto:path=/vclfcn-21/?versions&max-keys=999 2024-02-01T10:33:10.071 DEBUG:boto:auth_path=/vclfcn-21/?versions&max-keys=999 2024-02-01T10:33:10.071 DEBUG:boto:Method: GET 2024-02-01T10:33:10.071 DEBUG:boto:Path: /vclfcn-21/?versions&max-keys=999 2024-02-01T10:33:10.071 DEBUG:boto:Data: 2024-02-01T10:33:10.071 DEBUG:boto:Headers: {} 2024-02-01T10:33:10.071 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:33:10.071 DEBUG:boto:Port: 8000 2024-02-01T10:33:10.072 DEBUG:boto:Params: {} 2024-02-01T10:33:10.072 DEBUG:boto:Token: None 2024-02-01T10:33:10.072 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:10 GMT /vclfcn-21/?versions 2024-02-01T10:33:10.072 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tp2xoatwOxfDWEaUbuz+7jzyPfU= 2024-02-01T10:33:10.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tp2xoatwOxfDWEaUbuz+7jzyPfU=', 'Content-Length': '0'} 2024-02-01T10:33:10.076 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028d96b9590e746e9-0065bb7366-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:10.077 DEBUG:boto:b'vclfcn-21999falseobjiTuO-63D.1f6Od9rzTk7NLj1aX9iPCotrue2024-02-01T10:32:43.656Zrgw-multisite-test-userMultisite Test UserNormalobj6pTCwvsER73lrwuF6KIU7A5eJ1YCuvYfalse2024-02-01T10:32:43.638Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:33:10.077 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:33:10.077 DEBUG:boto:path=/vclfcn-21/obj 2024-02-01T10:33:10.077 DEBUG:boto:auth_path=/vclfcn-21/obj 2024-02-01T10:33:10.077 DEBUG:boto:Method: HEAD 2024-02-01T10:33:10.077 DEBUG:boto:Path: /vclfcn-21/obj 2024-02-01T10:33:10.078 DEBUG:boto:Data: 2024-02-01T10:33:10.078 DEBUG:boto:Headers: {} 2024-02-01T10:33:10.078 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:33:10.078 DEBUG:boto:Port: 8001 2024-02-01T10:33:10.078 DEBUG:boto:Params: {} 2024-02-01T10:33:10.078 DEBUG:boto:Token: None 2024-02-01T10:33:10.078 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:33:10 GMT /vclfcn-21/obj 2024-02-01T10:33:10.078 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:v8WUker6efdQsW45SZV2rgrb5zo= 2024-02-01T10:33:10.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:v8WUker6efdQsW45SZV2rgrb5zo=', 'Content-Length': '0'} 2024-02-01T10:33:10.082 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000000c5c00b626f5a4ef-0065bb7366-4358-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:10.082 DEBUG:boto:path=/vclfcn-21/obj 2024-02-01T10:33:10.082 DEBUG:boto:auth_path=/vclfcn-21/obj 2024-02-01T10:33:10.082 DEBUG:boto:Method: HEAD 2024-02-01T10:33:10.082 DEBUG:boto:Path: /vclfcn-21/obj 2024-02-01T10:33:10.082 DEBUG:boto:Data: 2024-02-01T10:33:10.082 DEBUG:boto:Headers: {} 2024-02-01T10:33:10.082 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:33:10.082 DEBUG:boto:Port: 8000 2024-02-01T10:33:10.082 DEBUG:boto:Params: {} 2024-02-01T10:33:10.083 DEBUG:boto:Token: None 2024-02-01T10:33:10.083 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:33:10 GMT /vclfcn-21/obj 2024-02-01T10:33:10.083 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:v8WUker6efdQsW45SZV2rgrb5zo= 2024-02-01T10:33:10.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:v8WUker6efdQsW45SZV2rgrb5zo=', 'Content-Length': '0'} 2024-02-01T10:33:10.086 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx0000022d6ebd20ccd47bc-0065bb7366-4351-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:10.086 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:33:10.086 DEBUG:boto:path=/vclfcn-21/obj 2024-02-01T10:33:10.086 DEBUG:boto:auth_path=/vclfcn-21/obj 2024-02-01T10:33:10.086 DEBUG:boto:path=/vclfcn-21/obj?versionId=6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY 2024-02-01T10:33:10.087 DEBUG:boto:auth_path=/vclfcn-21/obj?versionId=6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY 2024-02-01T10:33:10.087 DEBUG:boto:Method: GET 2024-02-01T10:33:10.087 DEBUG:boto:Path: /vclfcn-21/obj?versionId=6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY 2024-02-01T10:33:10.087 DEBUG:boto:Data: 2024-02-01T10:33:10.087 DEBUG:boto:Headers: {} 2024-02-01T10:33:10.087 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:33:10.087 DEBUG:boto:Port: 8001 2024-02-01T10:33:10.087 DEBUG:boto:Params: {} 2024-02-01T10:33:10.087 DEBUG:boto:Token: None 2024-02-01T10:33:10.087 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:10 GMT /vclfcn-21/obj?versionId=6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY 2024-02-01T10:33:10.087 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YHZchquBlmzjCDgp77xdWQa8vhw= 2024-02-01T10:33:10.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YHZchquBlmzjCDgp77xdWQa8vhw=', 'Content-Length': '0'} 2024-02-01T10:33:10.090 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:32:43 GMT'), ('x-amz-version-id', '6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fa35c158557dffde-0065bb7366-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:10.091 DEBUG:boto:path=/vclfcn-21/obj 2024-02-01T10:33:10.091 DEBUG:boto:auth_path=/vclfcn-21/obj 2024-02-01T10:33:10.091 DEBUG:boto:path=/vclfcn-21/obj?versionId=6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY 2024-02-01T10:33:10.091 DEBUG:boto:auth_path=/vclfcn-21/obj?versionId=6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY 2024-02-01T10:33:10.091 DEBUG:boto:Method: GET 2024-02-01T10:33:10.091 DEBUG:boto:Path: /vclfcn-21/obj?versionId=6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY 2024-02-01T10:33:10.091 DEBUG:boto:Data: 2024-02-01T10:33:10.091 DEBUG:boto:Headers: {} 2024-02-01T10:33:10.092 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:33:10.092 DEBUG:boto:Port: 8000 2024-02-01T10:33:10.092 DEBUG:boto:Params: {} 2024-02-01T10:33:10.092 DEBUG:boto:Token: None 2024-02-01T10:33:10.092 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:10 GMT /vclfcn-21/obj?versionId=6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY 2024-02-01T10:33:10.092 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YHZchquBlmzjCDgp77xdWQa8vhw= 2024-02-01T10:33:10.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YHZchquBlmzjCDgp77xdWQa8vhw=', 'Content-Length': '0'} 2024-02-01T10:33:10.095 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:32:43 GMT'), ('x-amz-version-id', '6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001b196dddd892b99e-0065bb7366-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:10.095 DEBUG:boto:path=/vclfcn-21/obj 2024-02-01T10:33:10.095 DEBUG:boto:auth_path=/vclfcn-21/obj 2024-02-01T10:33:10.095 DEBUG:boto:path=/vclfcn-21/obj?versionId=6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY 2024-02-01T10:33:10.095 DEBUG:boto:auth_path=/vclfcn-21/obj?versionId=6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY 2024-02-01T10:33:10.095 DEBUG:boto:Method: HEAD 2024-02-01T10:33:10.095 DEBUG:boto:Path: /vclfcn-21/obj?versionId=6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY 2024-02-01T10:33:10.096 DEBUG:boto:Data: 2024-02-01T10:33:10.096 DEBUG:boto:Headers: {} 2024-02-01T10:33:10.096 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:33:10.096 DEBUG:boto:Port: 8001 2024-02-01T10:33:10.096 DEBUG:boto:Params: {} 2024-02-01T10:33:10.096 DEBUG:boto:Token: None 2024-02-01T10:33:10.096 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:33:10 GMT /vclfcn-21/obj?versionId=6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY 2024-02-01T10:33:10.096 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:s4EEeZaoxOIKgOJ9CVG4sWLAB34= 2024-02-01T10:33:10.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:s4EEeZaoxOIKgOJ9CVG4sWLAB34=', 'Content-Length': '0'} 2024-02-01T10:33:10.099 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:32:43 GMT'), ('x-amz-version-id', '6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000015807876f4c80f1a-0065bb7366-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:10.099 DEBUG:boto:path=/vclfcn-21/obj 2024-02-01T10:33:10.100 DEBUG:boto:auth_path=/vclfcn-21/obj 2024-02-01T10:33:10.100 DEBUG:boto:path=/vclfcn-21/obj?versionId=6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY 2024-02-01T10:33:10.100 DEBUG:boto:auth_path=/vclfcn-21/obj?versionId=6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY 2024-02-01T10:33:10.100 DEBUG:boto:Method: HEAD 2024-02-01T10:33:10.100 DEBUG:boto:Path: /vclfcn-21/obj?versionId=6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY 2024-02-01T10:33:10.100 DEBUG:boto:Data: 2024-02-01T10:33:10.100 DEBUG:boto:Headers: {} 2024-02-01T10:33:10.100 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:33:10.100 DEBUG:boto:Port: 8000 2024-02-01T10:33:10.100 DEBUG:boto:Params: {} 2024-02-01T10:33:10.101 DEBUG:boto:Token: None 2024-02-01T10:33:10.101 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:33:10 GMT /vclfcn-21/obj?versionId=6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY 2024-02-01T10:33:10.101 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:s4EEeZaoxOIKgOJ9CVG4sWLAB34= 2024-02-01T10:33:10.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:s4EEeZaoxOIKgOJ9CVG4sWLAB34=', 'Content-Length': '0'} 2024-02-01T10:33:10.103 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:32:43 GMT'), ('x-amz-version-id', '6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a35c5a507aec0186-0065bb7366-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:10.104 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:33:10.104 DEBUG:boto:path=/vclfcn-21/obj 2024-02-01T10:33:10.104 DEBUG:boto:auth_path=/vclfcn-21/obj 2024-02-01T10:33:10.104 DEBUG:boto:path=/vclfcn-21/obj?versionId=6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY 2024-02-01T10:33:10.104 DEBUG:boto:auth_path=/vclfcn-21/obj?versionId=6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY 2024-02-01T10:33:10.104 DEBUG:boto:Method: GET 2024-02-01T10:33:10.104 DEBUG:boto:Path: /vclfcn-21/obj?versionId=6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY 2024-02-01T10:33:10.104 DEBUG:boto:Data: 2024-02-01T10:33:10.105 DEBUG:boto:Headers: {} 2024-02-01T10:33:10.105 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:33:10.105 DEBUG:boto:Port: 8001 2024-02-01T10:33:10.105 DEBUG:boto:Params: {} 2024-02-01T10:33:10.105 DEBUG:boto:Token: None 2024-02-01T10:33:10.105 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:10 GMT /vclfcn-21/obj?versionId=6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY 2024-02-01T10:33:10.106 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YHZchquBlmzjCDgp77xdWQa8vhw= 2024-02-01T10:33:10.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YHZchquBlmzjCDgp77xdWQa8vhw=', 'Content-Length': '0'} 2024-02-01T10:33:10.108 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:32:43 GMT'), ('x-amz-version-id', '6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c2f0e46e9e89ebe3-0065bb7366-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:10.108 DEBUG:boto:path=/vclfcn-21/obj 2024-02-01T10:33:10.109 DEBUG:boto:auth_path=/vclfcn-21/obj 2024-02-01T10:33:10.109 DEBUG:boto:path=/vclfcn-21/obj?versionId=6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY 2024-02-01T10:33:10.109 DEBUG:boto:auth_path=/vclfcn-21/obj?versionId=6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY 2024-02-01T10:33:10.109 DEBUG:boto:Method: GET 2024-02-01T10:33:10.109 DEBUG:boto:Path: /vclfcn-21/obj?versionId=6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY 2024-02-01T10:33:10.109 DEBUG:boto:Data: 2024-02-01T10:33:10.109 DEBUG:boto:Headers: {} 2024-02-01T10:33:10.109 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:33:10.109 DEBUG:boto:Port: 8000 2024-02-01T10:33:10.109 DEBUG:boto:Params: {} 2024-02-01T10:33:10.110 DEBUG:boto:Token: None 2024-02-01T10:33:10.110 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:10 GMT /vclfcn-21/obj?versionId=6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY 2024-02-01T10:33:10.110 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YHZchquBlmzjCDgp77xdWQa8vhw= 2024-02-01T10:33:10.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YHZchquBlmzjCDgp77xdWQa8vhw=', 'Content-Length': '0'} 2024-02-01T10:33:10.112 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:32:43 GMT'), ('x-amz-version-id', '6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ce2bfea41c08eb5f-0065bb7366-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:10.113 DEBUG:boto:path=/vclfcn-21/ 2024-02-01T10:33:10.113 DEBUG:boto:auth_path=/vclfcn-21/ 2024-02-01T10:33:10.113 DEBUG:boto:path=/vclfcn-21/?prefix=obj 2024-02-01T10:33:10.113 DEBUG:boto:auth_path=/vclfcn-21/?prefix=obj 2024-02-01T10:33:10.113 DEBUG:boto:Method: GET 2024-02-01T10:33:10.113 DEBUG:boto:Path: /vclfcn-21/?prefix=obj 2024-02-01T10:33:10.113 DEBUG:boto:Data: 2024-02-01T10:33:10.113 DEBUG:boto:Headers: {} 2024-02-01T10:33:10.113 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:33:10.113 DEBUG:boto:Port: 8001 2024-02-01T10:33:10.114 DEBUG:boto:Params: {} 2024-02-01T10:33:10.114 DEBUG:boto:Token: None 2024-02-01T10:33:10.114 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:10 GMT /vclfcn-21/ 2024-02-01T10:33:10.114 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vM9/6+JWJMgU3blVEnfTayztGBw= 2024-02-01T10:33:10.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vM9/6+JWJMgU3blVEnfTayztGBw=', 'Content-Length': '0'} 2024-02-01T10:33:10.118 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f714e2f91c4925f-0065bb7366-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:10.160 DEBUG:boto:b'vclfcn-21obj1000false' 2024-02-01T10:33:10.160 DEBUG:boto:path=/vclfcn-21/ 2024-02-01T10:33:10.161 DEBUG:boto:auth_path=/vclfcn-21/ 2024-02-01T10:33:10.161 DEBUG:boto:path=/vclfcn-21/?prefix=obj 2024-02-01T10:33:10.161 DEBUG:boto:auth_path=/vclfcn-21/?prefix=obj 2024-02-01T10:33:10.161 DEBUG:boto:Method: GET 2024-02-01T10:33:10.161 DEBUG:boto:Path: /vclfcn-21/?prefix=obj 2024-02-01T10:33:10.161 DEBUG:boto:Data: 2024-02-01T10:33:10.161 DEBUG:boto:Headers: {} 2024-02-01T10:33:10.161 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:33:10.161 DEBUG:boto:Port: 8000 2024-02-01T10:33:10.161 DEBUG:boto:Params: {} 2024-02-01T10:33:10.161 DEBUG:boto:Token: None 2024-02-01T10:33:10.162 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:10 GMT /vclfcn-21/ 2024-02-01T10:33:10.162 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vM9/6+JWJMgU3blVEnfTayztGBw= 2024-02-01T10:33:10.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vM9/6+JWJMgU3blVEnfTayztGBw=', 'Content-Length': '0'} 2024-02-01T10:33:10.166 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc3e2316b5eebe85-0065bb7366-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:10.166 DEBUG:boto:b'vclfcn-21obj1000false' 2024-02-01T10:33:10.166 DEBUG:boto:path=/vclfcn-21/obj 2024-02-01T10:33:10.166 DEBUG:boto:auth_path=/vclfcn-21/obj 2024-02-01T10:33:10.166 DEBUG:boto:Method: HEAD 2024-02-01T10:33:10.167 DEBUG:boto:Path: /vclfcn-21/obj 2024-02-01T10:33:10.167 DEBUG:boto:Data: 2024-02-01T10:33:10.167 DEBUG:boto:Headers: {} 2024-02-01T10:33:10.167 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:33:10.167 DEBUG:boto:Port: 8001 2024-02-01T10:33:10.167 DEBUG:boto:Params: {} 2024-02-01T10:33:10.167 DEBUG:boto:Token: None 2024-02-01T10:33:10.167 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:33:10 GMT /vclfcn-21/obj 2024-02-01T10:33:10.167 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:v8WUker6efdQsW45SZV2rgrb5zo= 2024-02-01T10:33:10.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:v8WUker6efdQsW45SZV2rgrb5zo=', 'Content-Length': '0'} 2024-02-01T10:33:10.171 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000001c68d21359cd12ed-0065bb7366-4358-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:10.171 DEBUG:boto:path=/vclfcn-21/obj 2024-02-01T10:33:10.171 DEBUG:boto:auth_path=/vclfcn-21/obj 2024-02-01T10:33:10.171 DEBUG:boto:Method: HEAD 2024-02-01T10:33:10.171 DEBUG:boto:Path: /vclfcn-21/obj 2024-02-01T10:33:10.171 DEBUG:boto:Data: 2024-02-01T10:33:10.171 DEBUG:boto:Headers: {} 2024-02-01T10:33:10.172 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:33:10.172 DEBUG:boto:Port: 8000 2024-02-01T10:33:10.172 DEBUG:boto:Params: {} 2024-02-01T10:33:10.172 DEBUG:boto:Token: None 2024-02-01T10:33:10.172 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:33:10 GMT /vclfcn-21/obj 2024-02-01T10:33:10.172 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:v8WUker6efdQsW45SZV2rgrb5zo= 2024-02-01T10:33:10.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:v8WUker6efdQsW45SZV2rgrb5zo=', 'Content-Length': '0'} 2024-02-01T10:33:10.175 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000dbfca481df3985b7-0065bb7366-4351-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:10.176 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-21 zones={test-zone3, test-zone1} 2024-02-01T10:33:10.176 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-21 zones={test-zone3, test-zone2} 2024-02-01T10:33:10.176 DEBUG:boto:path=/vclfcn-21/ 2024-02-01T10:33:10.176 DEBUG:boto:auth_path=/vclfcn-21/ 2024-02-01T10:33:10.176 DEBUG:boto:Method: HEAD 2024-02-01T10:33:10.176 DEBUG:boto:Path: /vclfcn-21/ 2024-02-01T10:33:10.176 DEBUG:boto:Data: 2024-02-01T10:33:10.176 DEBUG:boto:Headers: {} 2024-02-01T10:33:10.176 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:33:10.176 DEBUG:boto:Port: 8001 2024-02-01T10:33:10.177 DEBUG:boto:Params: {} 2024-02-01T10:33:10.177 DEBUG:boto:Token: None 2024-02-01T10:33:10.177 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:33:10 GMT /vclfcn-21/ 2024-02-01T10:33:10.177 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:DysXSP4lMA2gmpOBcAWW7at4ZNA= 2024-02-01T10:33:10.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:DysXSP4lMA2gmpOBcAWW7at4ZNA=', 'Content-Length': '0'} 2024-02-01T10:33:10.181 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000096a63d7d9d8764d8-0065bb7366-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:33:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:10.182 DEBUG:boto:path=/vclfcn-21/ 2024-02-01T10:33:10.182 DEBUG:boto:auth_path=/vclfcn-21/ 2024-02-01T10:33:10.182 DEBUG:boto:Method: HEAD 2024-02-01T10:33:10.182 DEBUG:boto:Path: /vclfcn-21/ 2024-02-01T10:33:10.182 DEBUG:boto:Data: 2024-02-01T10:33:10.182 DEBUG:boto:Headers: {} 2024-02-01T10:33:10.182 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:33:10.183 DEBUG:boto:Port: 8000 2024-02-01T10:33:10.183 DEBUG:boto:Params: {} 2024-02-01T10:33:10.183 DEBUG:boto:Token: None 2024-02-01T10:33:10.183 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:33:10 GMT /vclfcn-21/ 2024-02-01T10:33:10.183 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:DysXSP4lMA2gmpOBcAWW7at4ZNA= 2024-02-01T10:33:10.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:DysXSP4lMA2gmpOBcAWW7at4ZNA=', 'Content-Length': '0'} 2024-02-01T10:33:10.187 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007aaf9a50d2bbd480-0065bb7366-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:33:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:10.188 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:33:10.188 DEBUG:boto:path=/vclfcn-21/ 2024-02-01T10:33:10.188 DEBUG:boto:auth_path=/vclfcn-21/ 2024-02-01T10:33:10.188 DEBUG:boto:path=/vclfcn-21/?versions&max-keys=999 2024-02-01T10:33:10.188 DEBUG:boto:auth_path=/vclfcn-21/?versions&max-keys=999 2024-02-01T10:33:10.188 DEBUG:boto:Method: GET 2024-02-01T10:33:10.188 DEBUG:boto:Path: /vclfcn-21/?versions&max-keys=999 2024-02-01T10:33:10.188 DEBUG:boto:Data: 2024-02-01T10:33:10.189 DEBUG:boto:Headers: {} 2024-02-01T10:33:10.189 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:33:10.189 DEBUG:boto:Port: 8001 2024-02-01T10:33:10.189 DEBUG:boto:Params: {} 2024-02-01T10:33:10.189 DEBUG:boto:Token: None 2024-02-01T10:33:10.189 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:10 GMT /vclfcn-21/?versions 2024-02-01T10:33:10.189 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tp2xoatwOxfDWEaUbuz+7jzyPfU= 2024-02-01T10:33:10.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tp2xoatwOxfDWEaUbuz+7jzyPfU=', 'Content-Length': '0'} 2024-02-01T10:33:10.193 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000647a388e1646b25a-0065bb7366-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:10.194 DEBUG:boto:b'vclfcn-21999falseobjiTuO-63D.1f6Od9rzTk7NLj1aX9iPCotrue2024-02-01T10:32:43.656Zrgw-multisite-test-userMultisite Test UserNormalobj6pTCwvsER73lrwuF6KIU7A5eJ1YCuvYfalse2024-02-01T10:32:43.638Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:33:10.194 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:33:10.194 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:33:10.194 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:33:10.194 DEBUG:boto:path=/vclfcn-21/ 2024-02-01T10:33:10.194 DEBUG:boto:auth_path=/vclfcn-21/ 2024-02-01T10:33:10.195 DEBUG:boto:path=/vclfcn-21/?versions&max-keys=999 2024-02-01T10:33:10.195 DEBUG:boto:auth_path=/vclfcn-21/?versions&max-keys=999 2024-02-01T10:33:10.195 DEBUG:boto:Method: GET 2024-02-01T10:33:10.195 DEBUG:boto:Path: /vclfcn-21/?versions&max-keys=999 2024-02-01T10:33:10.195 DEBUG:boto:Data: 2024-02-01T10:33:10.195 DEBUG:boto:Headers: {} 2024-02-01T10:33:10.195 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:33:10.195 DEBUG:boto:Port: 8000 2024-02-01T10:33:10.195 DEBUG:boto:Params: {} 2024-02-01T10:33:10.195 DEBUG:boto:Token: None 2024-02-01T10:33:10.196 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:10 GMT /vclfcn-21/?versions 2024-02-01T10:33:10.196 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tp2xoatwOxfDWEaUbuz+7jzyPfU= 2024-02-01T10:33:10.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tp2xoatwOxfDWEaUbuz+7jzyPfU=', 'Content-Length': '0'} 2024-02-01T10:33:10.200 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a08915b28dd9c5e-0065bb7366-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:10.244 DEBUG:boto:b'vclfcn-21999falseobjiTuO-63D.1f6Od9rzTk7NLj1aX9iPCotrue2024-02-01T10:32:43.656Zrgw-multisite-test-userMultisite Test UserNormalobj6pTCwvsER73lrwuF6KIU7A5eJ1YCuvYfalse2024-02-01T10:32:43.638Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:33:10.245 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:33:10.245 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:33:10.246 DEBUG:boto:path=/vclfcn-21/ 2024-02-01T10:33:10.246 DEBUG:boto:auth_path=/vclfcn-21/ 2024-02-01T10:33:10.246 DEBUG:boto:path=/vclfcn-21/?versions&max-keys=999 2024-02-01T10:33:10.246 DEBUG:boto:auth_path=/vclfcn-21/?versions&max-keys=999 2024-02-01T10:33:10.246 DEBUG:boto:Method: GET 2024-02-01T10:33:10.247 DEBUG:boto:Path: /vclfcn-21/?versions&max-keys=999 2024-02-01T10:33:10.247 DEBUG:boto:Data: 2024-02-01T10:33:10.247 DEBUG:boto:Headers: {} 2024-02-01T10:33:10.247 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:33:10.247 DEBUG:boto:Port: 8001 2024-02-01T10:33:10.248 DEBUG:boto:Params: {} 2024-02-01T10:33:10.248 DEBUG:boto:Token: None 2024-02-01T10:33:10.248 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:10 GMT /vclfcn-21/?versions 2024-02-01T10:33:10.249 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tp2xoatwOxfDWEaUbuz+7jzyPfU= 2024-02-01T10:33:10.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tp2xoatwOxfDWEaUbuz+7jzyPfU=', 'Content-Length': '0'} 2024-02-01T10:33:10.253 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000436b2fb7d667fd4a-0065bb7366-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:10.296 DEBUG:boto:b'vclfcn-21999falseobjiTuO-63D.1f6Od9rzTk7NLj1aX9iPCotrue2024-02-01T10:32:43.656Zrgw-multisite-test-userMultisite Test UserNormalobj6pTCwvsER73lrwuF6KIU7A5eJ1YCuvYfalse2024-02-01T10:32:43.638Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:33:10.297 DEBUG:boto:path=/vclfcn-21/ 2024-02-01T10:33:10.297 DEBUG:boto:auth_path=/vclfcn-21/ 2024-02-01T10:33:10.297 DEBUG:boto:path=/vclfcn-21/?versions&max-keys=999 2024-02-01T10:33:10.297 DEBUG:boto:auth_path=/vclfcn-21/?versions&max-keys=999 2024-02-01T10:33:10.297 DEBUG:boto:Method: GET 2024-02-01T10:33:10.297 DEBUG:boto:Path: /vclfcn-21/?versions&max-keys=999 2024-02-01T10:33:10.297 DEBUG:boto:Data: 2024-02-01T10:33:10.297 DEBUG:boto:Headers: {} 2024-02-01T10:33:10.297 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:33:10.297 DEBUG:boto:Port: 8000 2024-02-01T10:33:10.298 DEBUG:boto:Params: {} 2024-02-01T10:33:10.298 DEBUG:boto:Token: None 2024-02-01T10:33:10.298 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:10 GMT /vclfcn-21/?versions 2024-02-01T10:33:10.298 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tp2xoatwOxfDWEaUbuz+7jzyPfU= 2024-02-01T10:33:10.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tp2xoatwOxfDWEaUbuz+7jzyPfU=', 'Content-Length': '0'} 2024-02-01T10:33:10.302 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001968fec712b9d63c-0065bb7366-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:10.344 DEBUG:boto:b'vclfcn-21999falseobjiTuO-63D.1f6Od9rzTk7NLj1aX9iPCotrue2024-02-01T10:32:43.656Zrgw-multisite-test-userMultisite Test UserNormalobj6pTCwvsER73lrwuF6KIU7A5eJ1YCuvYfalse2024-02-01T10:32:43.638Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:33:10.385 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:33:10.385 DEBUG:boto:path=/vclfcn-21/obj 2024-02-01T10:33:10.385 DEBUG:boto:auth_path=/vclfcn-21/obj 2024-02-01T10:33:10.385 DEBUG:boto:Method: HEAD 2024-02-01T10:33:10.385 DEBUG:boto:Path: /vclfcn-21/obj 2024-02-01T10:33:10.385 DEBUG:boto:Data: 2024-02-01T10:33:10.385 DEBUG:boto:Headers: {} 2024-02-01T10:33:10.385 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:33:10.386 DEBUG:boto:Port: 8001 2024-02-01T10:33:10.386 DEBUG:boto:Params: {} 2024-02-01T10:33:10.386 DEBUG:boto:Token: None 2024-02-01T10:33:10.386 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:33:10 GMT /vclfcn-21/obj 2024-02-01T10:33:10.386 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:v8WUker6efdQsW45SZV2rgrb5zo= 2024-02-01T10:33:10.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:v8WUker6efdQsW45SZV2rgrb5zo=', 'Content-Length': '0'} 2024-02-01T10:33:10.390 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000e64ade49e1679c19-0065bb7366-4358-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:10.390 DEBUG:boto:path=/vclfcn-21/obj 2024-02-01T10:33:10.390 DEBUG:boto:auth_path=/vclfcn-21/obj 2024-02-01T10:33:10.390 DEBUG:boto:Method: HEAD 2024-02-01T10:33:10.390 DEBUG:boto:Path: /vclfcn-21/obj 2024-02-01T10:33:10.390 DEBUG:boto:Data: 2024-02-01T10:33:10.390 DEBUG:boto:Headers: {} 2024-02-01T10:33:10.390 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:33:10.390 DEBUG:boto:Port: 8000 2024-02-01T10:33:10.391 DEBUG:boto:Params: {} 2024-02-01T10:33:10.391 DEBUG:boto:Token: None 2024-02-01T10:33:10.391 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:33:10 GMT /vclfcn-21/obj 2024-02-01T10:33:10.391 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:v8WUker6efdQsW45SZV2rgrb5zo= 2024-02-01T10:33:10.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:v8WUker6efdQsW45SZV2rgrb5zo=', 'Content-Length': '0'} 2024-02-01T10:33:10.394 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx0000051eb1ba1fd1b3d73-0065bb7366-4307-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:10.394 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:33:10.395 DEBUG:boto:path=/vclfcn-21/obj 2024-02-01T10:33:10.395 DEBUG:boto:auth_path=/vclfcn-21/obj 2024-02-01T10:33:10.395 DEBUG:boto:path=/vclfcn-21/obj?versionId=6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY 2024-02-01T10:33:10.395 DEBUG:boto:auth_path=/vclfcn-21/obj?versionId=6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY 2024-02-01T10:33:10.395 DEBUG:boto:Method: GET 2024-02-01T10:33:10.395 DEBUG:boto:Path: /vclfcn-21/obj?versionId=6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY 2024-02-01T10:33:10.395 DEBUG:boto:Data: 2024-02-01T10:33:10.395 DEBUG:boto:Headers: {} 2024-02-01T10:33:10.395 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:33:10.395 DEBUG:boto:Port: 8001 2024-02-01T10:33:10.395 DEBUG:boto:Params: {} 2024-02-01T10:33:10.396 DEBUG:boto:Token: None 2024-02-01T10:33:10.396 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:10 GMT /vclfcn-21/obj?versionId=6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY 2024-02-01T10:33:10.396 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YHZchquBlmzjCDgp77xdWQa8vhw= 2024-02-01T10:33:10.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YHZchquBlmzjCDgp77xdWQa8vhw=', 'Content-Length': '0'} 2024-02-01T10:33:10.399 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:32:43 GMT'), ('x-amz-version-id', '6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003e652e0a4fcf71d5-0065bb7366-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:10.399 DEBUG:boto:path=/vclfcn-21/obj 2024-02-01T10:33:10.399 DEBUG:boto:auth_path=/vclfcn-21/obj 2024-02-01T10:33:10.400 DEBUG:boto:path=/vclfcn-21/obj?versionId=6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY 2024-02-01T10:33:10.400 DEBUG:boto:auth_path=/vclfcn-21/obj?versionId=6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY 2024-02-01T10:33:10.400 DEBUG:boto:Method: GET 2024-02-01T10:33:10.400 DEBUG:boto:Path: /vclfcn-21/obj?versionId=6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY 2024-02-01T10:33:10.400 DEBUG:boto:Data: 2024-02-01T10:33:10.400 DEBUG:boto:Headers: {} 2024-02-01T10:33:10.400 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:33:10.400 DEBUG:boto:Port: 8000 2024-02-01T10:33:10.400 DEBUG:boto:Params: {} 2024-02-01T10:33:10.400 DEBUG:boto:Token: None 2024-02-01T10:33:10.401 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:10 GMT /vclfcn-21/obj?versionId=6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY 2024-02-01T10:33:10.401 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YHZchquBlmzjCDgp77xdWQa8vhw= 2024-02-01T10:33:10.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YHZchquBlmzjCDgp77xdWQa8vhw=', 'Content-Length': '0'} 2024-02-01T10:33:10.405 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:32:43 GMT'), ('x-amz-version-id', '6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003c417b476d03fa45-0065bb7366-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:10.405 DEBUG:boto:path=/vclfcn-21/obj 2024-02-01T10:33:10.405 DEBUG:boto:auth_path=/vclfcn-21/obj 2024-02-01T10:33:10.405 DEBUG:boto:path=/vclfcn-21/obj?versionId=6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY 2024-02-01T10:33:10.405 DEBUG:boto:auth_path=/vclfcn-21/obj?versionId=6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY 2024-02-01T10:33:10.405 DEBUG:boto:Method: HEAD 2024-02-01T10:33:10.406 DEBUG:boto:Path: /vclfcn-21/obj?versionId=6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY 2024-02-01T10:33:10.406 DEBUG:boto:Data: 2024-02-01T10:33:10.406 DEBUG:boto:Headers: {} 2024-02-01T10:33:10.406 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:33:10.406 DEBUG:boto:Port: 8001 2024-02-01T10:33:10.406 DEBUG:boto:Params: {} 2024-02-01T10:33:10.406 DEBUG:boto:Token: None 2024-02-01T10:33:10.406 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:33:10 GMT /vclfcn-21/obj?versionId=6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY 2024-02-01T10:33:10.406 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:s4EEeZaoxOIKgOJ9CVG4sWLAB34= 2024-02-01T10:33:10.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:s4EEeZaoxOIKgOJ9CVG4sWLAB34=', 'Content-Length': '0'} 2024-02-01T10:33:10.409 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:32:43 GMT'), ('x-amz-version-id', '6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b26090fdfa1ea794-0065bb7366-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:10.410 DEBUG:boto:path=/vclfcn-21/obj 2024-02-01T10:33:10.410 DEBUG:boto:auth_path=/vclfcn-21/obj 2024-02-01T10:33:10.410 DEBUG:boto:path=/vclfcn-21/obj?versionId=6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY 2024-02-01T10:33:10.410 DEBUG:boto:auth_path=/vclfcn-21/obj?versionId=6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY 2024-02-01T10:33:10.410 DEBUG:boto:Method: HEAD 2024-02-01T10:33:10.410 DEBUG:boto:Path: /vclfcn-21/obj?versionId=6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY 2024-02-01T10:33:10.410 DEBUG:boto:Data: 2024-02-01T10:33:10.410 DEBUG:boto:Headers: {} 2024-02-01T10:33:10.410 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:33:10.410 DEBUG:boto:Port: 8000 2024-02-01T10:33:10.410 DEBUG:boto:Params: {} 2024-02-01T10:33:10.411 DEBUG:boto:Token: None 2024-02-01T10:33:10.411 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:33:10 GMT /vclfcn-21/obj?versionId=6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY 2024-02-01T10:33:10.411 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:s4EEeZaoxOIKgOJ9CVG4sWLAB34= 2024-02-01T10:33:10.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:s4EEeZaoxOIKgOJ9CVG4sWLAB34=', 'Content-Length': '0'} 2024-02-01T10:33:10.414 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:32:43 GMT'), ('x-amz-version-id', '6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e344a7cf53bb2a16-0065bb7366-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:10.414 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:33:10.414 DEBUG:boto:path=/vclfcn-21/obj 2024-02-01T10:33:10.414 DEBUG:boto:auth_path=/vclfcn-21/obj 2024-02-01T10:33:10.414 DEBUG:boto:path=/vclfcn-21/obj?versionId=6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY 2024-02-01T10:33:10.414 DEBUG:boto:auth_path=/vclfcn-21/obj?versionId=6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY 2024-02-01T10:33:10.414 DEBUG:boto:Method: GET 2024-02-01T10:33:10.414 DEBUG:boto:Path: /vclfcn-21/obj?versionId=6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY 2024-02-01T10:33:10.415 DEBUG:boto:Data: 2024-02-01T10:33:10.415 DEBUG:boto:Headers: {} 2024-02-01T10:33:10.415 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:33:10.415 DEBUG:boto:Port: 8001 2024-02-01T10:33:10.415 DEBUG:boto:Params: {} 2024-02-01T10:33:10.415 DEBUG:boto:Token: None 2024-02-01T10:33:10.415 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:10 GMT /vclfcn-21/obj?versionId=6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY 2024-02-01T10:33:10.415 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YHZchquBlmzjCDgp77xdWQa8vhw= 2024-02-01T10:33:10.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YHZchquBlmzjCDgp77xdWQa8vhw=', 'Content-Length': '0'} 2024-02-01T10:33:10.418 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:32:43 GMT'), ('x-amz-version-id', '6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bae9b5049f32af5c-0065bb7366-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:10.418 DEBUG:boto:path=/vclfcn-21/obj 2024-02-01T10:33:10.419 DEBUG:boto:auth_path=/vclfcn-21/obj 2024-02-01T10:33:10.419 DEBUG:boto:path=/vclfcn-21/obj?versionId=6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY 2024-02-01T10:33:10.419 DEBUG:boto:auth_path=/vclfcn-21/obj?versionId=6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY 2024-02-01T10:33:10.419 DEBUG:boto:Method: GET 2024-02-01T10:33:10.419 DEBUG:boto:Path: /vclfcn-21/obj?versionId=6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY 2024-02-01T10:33:10.419 DEBUG:boto:Data: 2024-02-01T10:33:10.419 DEBUG:boto:Headers: {} 2024-02-01T10:33:10.419 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:33:10.419 DEBUG:boto:Port: 8000 2024-02-01T10:33:10.419 DEBUG:boto:Params: {} 2024-02-01T10:33:10.420 DEBUG:boto:Token: None 2024-02-01T10:33:10.420 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:10 GMT /vclfcn-21/obj?versionId=6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY 2024-02-01T10:33:10.420 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YHZchquBlmzjCDgp77xdWQa8vhw= 2024-02-01T10:33:10.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YHZchquBlmzjCDgp77xdWQa8vhw=', 'Content-Length': '0'} 2024-02-01T10:33:10.422 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:32:43 GMT'), ('x-amz-version-id', '6pTCwvsER73lrwuF6KIU7A5eJ1YCuvY'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000982b38f99aa88c3f-0065bb7366-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:10.423 DEBUG:boto:path=/vclfcn-21/ 2024-02-01T10:33:10.423 DEBUG:boto:auth_path=/vclfcn-21/ 2024-02-01T10:33:10.423 DEBUG:boto:path=/vclfcn-21/?prefix=obj 2024-02-01T10:33:10.423 DEBUG:boto:auth_path=/vclfcn-21/?prefix=obj 2024-02-01T10:33:10.423 DEBUG:boto:Method: GET 2024-02-01T10:33:10.423 DEBUG:boto:Path: /vclfcn-21/?prefix=obj 2024-02-01T10:33:10.423 DEBUG:boto:Data: 2024-02-01T10:33:10.423 DEBUG:boto:Headers: {} 2024-02-01T10:33:10.423 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:33:10.424 DEBUG:boto:Port: 8001 2024-02-01T10:33:10.424 DEBUG:boto:Params: {} 2024-02-01T10:33:10.424 DEBUG:boto:Token: None 2024-02-01T10:33:10.424 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:10 GMT /vclfcn-21/ 2024-02-01T10:33:10.424 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vM9/6+JWJMgU3blVEnfTayztGBw= 2024-02-01T10:33:10.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vM9/6+JWJMgU3blVEnfTayztGBw=', 'Content-Length': '0'} 2024-02-01T10:33:10.428 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e4dbb28293f4af2-0065bb7366-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:10.472 DEBUG:boto:b'vclfcn-21obj1000false' 2024-02-01T10:33:10.473 DEBUG:boto:path=/vclfcn-21/ 2024-02-01T10:33:10.473 DEBUG:boto:auth_path=/vclfcn-21/ 2024-02-01T10:33:10.473 DEBUG:boto:path=/vclfcn-21/?prefix=obj 2024-02-01T10:33:10.473 DEBUG:boto:auth_path=/vclfcn-21/?prefix=obj 2024-02-01T10:33:10.473 DEBUG:boto:Method: GET 2024-02-01T10:33:10.473 DEBUG:boto:Path: /vclfcn-21/?prefix=obj 2024-02-01T10:33:10.473 DEBUG:boto:Data: 2024-02-01T10:33:10.473 DEBUG:boto:Headers: {} 2024-02-01T10:33:10.473 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:33:10.473 DEBUG:boto:Port: 8000 2024-02-01T10:33:10.473 DEBUG:boto:Params: {} 2024-02-01T10:33:10.474 DEBUG:boto:Token: None 2024-02-01T10:33:10.474 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:10 GMT /vclfcn-21/ 2024-02-01T10:33:10.474 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vM9/6+JWJMgU3blVEnfTayztGBw= 2024-02-01T10:33:10.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vM9/6+JWJMgU3blVEnfTayztGBw=', 'Content-Length': '0'} 2024-02-01T10:33:10.478 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000572bbcd214456ab6-0065bb7366-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:10.478 DEBUG:boto:b'vclfcn-21obj1000false' 2024-02-01T10:33:10.479 DEBUG:boto:path=/vclfcn-21/obj 2024-02-01T10:33:10.479 DEBUG:boto:auth_path=/vclfcn-21/obj 2024-02-01T10:33:10.479 DEBUG:boto:Method: HEAD 2024-02-01T10:33:10.479 DEBUG:boto:Path: /vclfcn-21/obj 2024-02-01T10:33:10.479 DEBUG:boto:Data: 2024-02-01T10:33:10.479 DEBUG:boto:Headers: {} 2024-02-01T10:33:10.479 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:33:10.479 DEBUG:boto:Port: 8001 2024-02-01T10:33:10.479 DEBUG:boto:Params: {} 2024-02-01T10:33:10.480 DEBUG:boto:Token: None 2024-02-01T10:33:10.480 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:33:10 GMT /vclfcn-21/obj 2024-02-01T10:33:10.480 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:v8WUker6efdQsW45SZV2rgrb5zo= 2024-02-01T10:33:10.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:v8WUker6efdQsW45SZV2rgrb5zo=', 'Content-Length': '0'} 2024-02-01T10:33:10.483 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000e082bb1acb8c5d95-0065bb7366-4358-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:10.483 DEBUG:boto:path=/vclfcn-21/obj 2024-02-01T10:33:10.483 DEBUG:boto:auth_path=/vclfcn-21/obj 2024-02-01T10:33:10.484 DEBUG:boto:Method: HEAD 2024-02-01T10:33:10.484 DEBUG:boto:Path: /vclfcn-21/obj 2024-02-01T10:33:10.484 DEBUG:boto:Data: 2024-02-01T10:33:10.484 DEBUG:boto:Headers: {} 2024-02-01T10:33:10.484 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:33:10.484 DEBUG:boto:Port: 8000 2024-02-01T10:33:10.484 DEBUG:boto:Params: {} 2024-02-01T10:33:10.484 DEBUG:boto:Token: None 2024-02-01T10:33:10.484 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:33:10 GMT /vclfcn-21/obj 2024-02-01T10:33:10.484 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:v8WUker6efdQsW45SZV2rgrb5zo= 2024-02-01T10:33:10.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:v8WUker6efdQsW45SZV2rgrb5zo=', 'Content-Length': '0'} 2024-02-01T10:33:10.488 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000de8b65ce29678df4-0065bb7366-4307-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:10.488 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-21 zones={test-zone3, test-zone2} 2024-02-01T10:33:10.488 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=vclfcn-22 2024-02-01T10:33:10.488 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:33:10.489 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:33:10.489 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket vclfcn-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-02-01T10:33:10.539 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:33:10.539 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:33:10.604 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:33:10.601+0000 7f82fe626980 1 waiting to reach incremental sync.. 2024-02-01T10:33:15.605 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:33:15.605+0000 7f82fe626980 1 waiting to reach incremental sync.. 2024-02-01T10:33:20.607 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:33:20.605+0000 7f82fe626980 1 waiting to reach incremental sync.. 2024-02-01T10:33:25.608 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:33:25.605+0000 7f82fe626980 1 waiting to reach incremental sync.. 2024-02-01T10:33:30.611 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:33:30.609+0000 7f82fe626980 1 bucket sync caught up with source: 2024-02-01T10:33:30.611 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, , , , , , , , 00000000002.19.6, , ] 2024-02-01T10:33:30.611 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, , , , , , , , 00000000002.19.6, , ] 2024-02-01T10:33:30.611 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:33:30.609+0000 7f82fe626980 0 bucket checkpoint complete 2024-02-01T10:33:30.654 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=vclfcn-22 2024-02-01T10:33:30.654 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:33:30.654 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:33:30.655 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:33:30.706 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:33:30.706 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:33:30.768 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:33:30.771+0000 7f745103a980 1 bucket sync caught up with source: 2024-02-01T10:33:30.768 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, , , , , , , , 00000000002.19.6, , ] 2024-02-01T10:33:30.768 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, , , , , , , , 00000000002.19.6, , ] 2024-02-01T10:33:30.769 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:33:30.771+0000 7f745103a980 0 bucket checkpoint complete 2024-02-01T10:33:30.791 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=vclfcn-22 2024-02-01T10:33:30.791 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:33:30.791 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:33:30.791 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-22 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:33:30.831 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:33:30.832 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:33:30.893 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:33:30.899+0000 7fd347b52980 1 bucket sync caught up with source: 2024-02-01T10:33:30.893 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, , , , , , , , 00000000003.27.12, , ] 2024-02-01T10:33:30.894 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, , , , , , , , 00000000003.27.12, , ] 2024-02-01T10:33:30.894 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:33:30.899+0000 7fd347b52980 0 bucket checkpoint complete 2024-02-01T10:33:30.915 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=vclfcn-22 2024-02-01T10:33:30.915 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:33:30.915 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:33:30.915 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-22 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:33:31.003 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:33:31.003 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:33:31.063 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:33:31.067+0000 7f3be936e980 1 bucket sync caught up with source: 2024-02-01T10:33:31.063 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, , , , , , , , 00000000003.27.12, , ] 2024-02-01T10:33:31.063 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, , , , , , , , 00000000003.27.12, , ] 2024-02-01T10:33:31.063 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:33:31.067+0000 7f3be936e980 0 bucket checkpoint complete 2024-02-01T10:33:31.084 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=vclfcn-22 2024-02-01T10:33:31.085 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:33:31.085 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:33:31.085 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-22 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:33:31.126 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:33:31.126 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:33:31.181 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:33:31.183+0000 7f3024599980 1 bucket sync caught up with source: 2024-02-01T10:33:31.181 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, , , , , , , , 00000000003.40.6, , ] 2024-02-01T10:33:31.181 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, , , , , , , , 00000000003.40.6, , ] 2024-02-01T10:33:31.181 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:33:31.183+0000 7f3024599980 0 bucket checkpoint complete 2024-02-01T10:33:31.200 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=vclfcn-22 2024-02-01T10:33:31.200 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:33:31.200 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:33:31.201 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket vclfcn-22 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-02-01T10:33:31.251 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:33:31.251 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:33:31.314 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:33:31.313+0000 7fe746351980 1 bucket sync caught up with source: 2024-02-01T10:33:31.314 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, , , , , , , , 00000000003.40.6, , ] 2024-02-01T10:33:31.314 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, , , , , , , , 00000000003.40.6, , ] 2024-02-01T10:33:31.315 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:33:31.313+0000 7fe746351980 0 bucket checkpoint complete 2024-02-01T10:33:31.337 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-22 zones={test-zone2, test-zone1} 2024-02-01T10:33:31.337 DEBUG:boto:path=/vclfcn-22/ 2024-02-01T10:33:31.337 DEBUG:boto:auth_path=/vclfcn-22/ 2024-02-01T10:33:31.337 DEBUG:boto:Method: HEAD 2024-02-01T10:33:31.338 DEBUG:boto:Path: /vclfcn-22/ 2024-02-01T10:33:31.338 DEBUG:boto:Data: 2024-02-01T10:33:31.338 DEBUG:boto:Headers: {} 2024-02-01T10:33:31.338 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:33:31.338 DEBUG:boto:Port: 8000 2024-02-01T10:33:31.338 DEBUG:boto:Params: {} 2024-02-01T10:33:31.338 DEBUG:boto:Token: None 2024-02-01T10:33:31.338 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:33:31 GMT /vclfcn-22/ 2024-02-01T10:33:31.339 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7IrmJjdenew8rKYsEPD5NvrbPbg= 2024-02-01T10:33:31.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:31 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7IrmJjdenew8rKYsEPD5NvrbPbg=', 'Content-Length': '0'} 2024-02-01T10:33:31.341 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000065c8ff8346dcddc-0065bb737b-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:33:31 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:31.341 DEBUG:boto:path=/vclfcn-22/ 2024-02-01T10:33:31.341 DEBUG:boto:auth_path=/vclfcn-22/ 2024-02-01T10:33:31.342 DEBUG:boto:Method: HEAD 2024-02-01T10:33:31.342 DEBUG:boto:Path: /vclfcn-22/ 2024-02-01T10:33:31.342 DEBUG:boto:Data: 2024-02-01T10:33:31.342 DEBUG:boto:Headers: {} 2024-02-01T10:33:31.342 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:33:31.342 DEBUG:boto:Port: 8000 2024-02-01T10:33:31.343 DEBUG:boto:Params: {} 2024-02-01T10:33:31.343 DEBUG:boto:Token: None 2024-02-01T10:33:31.343 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:33:31 GMT /vclfcn-22/ 2024-02-01T10:33:31.343 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7IrmJjdenew8rKYsEPD5NvrbPbg= 2024-02-01T10:33:31.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:31 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7IrmJjdenew8rKYsEPD5NvrbPbg=', 'Content-Length': '0'} 2024-02-01T10:33:31.347 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000033141aaa0d77fb5-0065bb737b-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:33:31 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:31.347 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:33:31.347 DEBUG:boto:path=/vclfcn-22/ 2024-02-01T10:33:31.348 DEBUG:boto:auth_path=/vclfcn-22/ 2024-02-01T10:33:31.348 DEBUG:boto:path=/vclfcn-22/?versions&max-keys=999 2024-02-01T10:33:31.348 DEBUG:boto:auth_path=/vclfcn-22/?versions&max-keys=999 2024-02-01T10:33:31.348 DEBUG:boto:Method: GET 2024-02-01T10:33:31.348 DEBUG:boto:Path: /vclfcn-22/?versions&max-keys=999 2024-02-01T10:33:31.348 DEBUG:boto:Data: 2024-02-01T10:33:31.348 DEBUG:boto:Headers: {} 2024-02-01T10:33:31.348 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:33:31.348 DEBUG:boto:Port: 8000 2024-02-01T10:33:31.348 DEBUG:boto:Params: {} 2024-02-01T10:33:31.349 DEBUG:boto:Token: None 2024-02-01T10:33:31.349 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:31 GMT /vclfcn-22/?versions 2024-02-01T10:33:31.349 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:eM2liZWjXHGbRNz1UOHqi92IEEg= 2024-02-01T10:33:31.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:31 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:eM2liZWjXHGbRNz1UOHqi92IEEg=', 'Content-Length': '0'} 2024-02-01T10:33:31.351 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fdb6496a9e46d04f-0065bb737b-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:31 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:31.351 DEBUG:boto:b'vclfcn-22999falseobjB4BhmM3thsoWYrVmsTC9qpVYsD42.EGtrue2024-02-01T10:32:43.702Zrgw-multisite-test-userMultisite Test UserNormalobjgjHQyIwf4nDIizFkOIFbxEy7p.MOtRBfalse2024-02-01T10:32:43.666Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:33:31.353 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:33:31.353 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:33:31.353 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:33:31.353 DEBUG:boto:path=/vclfcn-22/ 2024-02-01T10:33:31.353 DEBUG:boto:auth_path=/vclfcn-22/ 2024-02-01T10:33:31.353 DEBUG:boto:path=/vclfcn-22/?versions&max-keys=999 2024-02-01T10:33:31.354 DEBUG:boto:auth_path=/vclfcn-22/?versions&max-keys=999 2024-02-01T10:33:31.354 DEBUG:boto:Method: GET 2024-02-01T10:33:31.354 DEBUG:boto:Path: /vclfcn-22/?versions&max-keys=999 2024-02-01T10:33:31.354 DEBUG:boto:Data: 2024-02-01T10:33:31.354 DEBUG:boto:Headers: {} 2024-02-01T10:33:31.354 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:33:31.354 DEBUG:boto:Port: 8000 2024-02-01T10:33:31.354 DEBUG:boto:Params: {} 2024-02-01T10:33:31.354 DEBUG:boto:Token: None 2024-02-01T10:33:31.354 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:31 GMT /vclfcn-22/?versions 2024-02-01T10:33:31.355 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:eM2liZWjXHGbRNz1UOHqi92IEEg= 2024-02-01T10:33:31.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:31 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:eM2liZWjXHGbRNz1UOHqi92IEEg=', 'Content-Length': '0'} 2024-02-01T10:33:31.358 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e0794594f1c77e3-0065bb737b-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:31 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:31.358 DEBUG:boto:b'vclfcn-22999falseobjB4BhmM3thsoWYrVmsTC9qpVYsD42.EGtrue2024-02-01T10:32:43.702Zrgw-multisite-test-userMultisite Test UserNormalobjgjHQyIwf4nDIizFkOIFbxEy7p.MOtRBfalse2024-02-01T10:32:43.666Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:33:31.359 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:33:31.359 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:33:31.359 DEBUG:boto:path=/vclfcn-22/ 2024-02-01T10:33:31.359 DEBUG:boto:auth_path=/vclfcn-22/ 2024-02-01T10:33:31.359 DEBUG:boto:path=/vclfcn-22/?versions&max-keys=999 2024-02-01T10:33:31.359 DEBUG:boto:auth_path=/vclfcn-22/?versions&max-keys=999 2024-02-01T10:33:31.359 DEBUG:boto:Method: GET 2024-02-01T10:33:31.360 DEBUG:boto:Path: /vclfcn-22/?versions&max-keys=999 2024-02-01T10:33:31.360 DEBUG:boto:Data: 2024-02-01T10:33:31.360 DEBUG:boto:Headers: {} 2024-02-01T10:33:31.360 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:33:31.360 DEBUG:boto:Port: 8000 2024-02-01T10:33:31.360 DEBUG:boto:Params: {} 2024-02-01T10:33:31.360 DEBUG:boto:Token: None 2024-02-01T10:33:31.360 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:31 GMT /vclfcn-22/?versions 2024-02-01T10:33:31.360 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:eM2liZWjXHGbRNz1UOHqi92IEEg= 2024-02-01T10:33:31.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:31 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:eM2liZWjXHGbRNz1UOHqi92IEEg=', 'Content-Length': '0'} 2024-02-01T10:33:31.363 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094b37ec755c5203d-0065bb737b-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:31 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:31.363 DEBUG:boto:b'vclfcn-22999falseobjB4BhmM3thsoWYrVmsTC9qpVYsD42.EGtrue2024-02-01T10:32:43.702Zrgw-multisite-test-userMultisite Test UserNormalobjgjHQyIwf4nDIizFkOIFbxEy7p.MOtRBfalse2024-02-01T10:32:43.666Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:33:31.363 DEBUG:boto:path=/vclfcn-22/ 2024-02-01T10:33:31.363 DEBUG:boto:auth_path=/vclfcn-22/ 2024-02-01T10:33:31.363 DEBUG:boto:path=/vclfcn-22/?versions&max-keys=999 2024-02-01T10:33:31.363 DEBUG:boto:auth_path=/vclfcn-22/?versions&max-keys=999 2024-02-01T10:33:31.364 DEBUG:boto:Method: GET 2024-02-01T10:33:31.364 DEBUG:boto:Path: /vclfcn-22/?versions&max-keys=999 2024-02-01T10:33:31.364 DEBUG:boto:Data: 2024-02-01T10:33:31.364 DEBUG:boto:Headers: {} 2024-02-01T10:33:31.364 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:33:31.364 DEBUG:boto:Port: 8000 2024-02-01T10:33:31.364 DEBUG:boto:Params: {} 2024-02-01T10:33:31.364 DEBUG:boto:Token: None 2024-02-01T10:33:31.364 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:31 GMT /vclfcn-22/?versions 2024-02-01T10:33:31.365 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:eM2liZWjXHGbRNz1UOHqi92IEEg= 2024-02-01T10:33:31.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:31 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:eM2liZWjXHGbRNz1UOHqi92IEEg=', 'Content-Length': '0'} 2024-02-01T10:33:31.369 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d58cf454dbed71e-0065bb737b-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:31 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:31.412 DEBUG:boto:b'vclfcn-22999falseobjB4BhmM3thsoWYrVmsTC9qpVYsD42.EGtrue2024-02-01T10:32:43.702Zrgw-multisite-test-userMultisite Test UserNormalobjgjHQyIwf4nDIizFkOIFbxEy7p.MOtRBfalse2024-02-01T10:32:43.666Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:33:31.412 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:33:31.413 DEBUG:boto:path=/vclfcn-22/obj 2024-02-01T10:33:31.413 DEBUG:boto:auth_path=/vclfcn-22/obj 2024-02-01T10:33:31.413 DEBUG:boto:Method: HEAD 2024-02-01T10:33:31.413 DEBUG:boto:Path: /vclfcn-22/obj 2024-02-01T10:33:31.413 DEBUG:boto:Data: 2024-02-01T10:33:31.413 DEBUG:boto:Headers: {} 2024-02-01T10:33:31.413 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:33:31.413 DEBUG:boto:Port: 8000 2024-02-01T10:33:31.414 DEBUG:boto:Params: {} 2024-02-01T10:33:31.414 DEBUG:boto:Token: None 2024-02-01T10:33:31.414 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:33:31 GMT /vclfcn-22/obj 2024-02-01T10:33:31.414 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ae65M6bYbcjYZbdghu8cIFNw8cQ= 2024-02-01T10:33:31.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:31 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ae65M6bYbcjYZbdghu8cIFNw8cQ=', 'Content-Length': '0'} 2024-02-01T10:33:31.418 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000aec4be02321cb308-0065bb737b-4307-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:31 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:31.418 DEBUG:boto:path=/vclfcn-22/obj 2024-02-01T10:33:31.418 DEBUG:boto:auth_path=/vclfcn-22/obj 2024-02-01T10:33:31.418 DEBUG:boto:Method: HEAD 2024-02-01T10:33:31.418 DEBUG:boto:Path: /vclfcn-22/obj 2024-02-01T10:33:31.419 DEBUG:boto:Data: 2024-02-01T10:33:31.419 DEBUG:boto:Headers: {} 2024-02-01T10:33:31.419 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:33:31.419 DEBUG:boto:Port: 8000 2024-02-01T10:33:31.419 DEBUG:boto:Params: {} 2024-02-01T10:33:31.419 DEBUG:boto:Token: None 2024-02-01T10:33:31.419 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:33:31 GMT /vclfcn-22/obj 2024-02-01T10:33:31.419 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ae65M6bYbcjYZbdghu8cIFNw8cQ= 2024-02-01T10:33:31.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:31 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ae65M6bYbcjYZbdghu8cIFNw8cQ=', 'Content-Length': '0'} 2024-02-01T10:33:31.423 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000a6932169f0a39eb1-0065bb737b-4351-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:31 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:31.423 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:33:31.423 DEBUG:boto:path=/vclfcn-22/obj 2024-02-01T10:33:31.423 DEBUG:boto:auth_path=/vclfcn-22/obj 2024-02-01T10:33:31.424 DEBUG:boto:path=/vclfcn-22/obj?versionId=gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB 2024-02-01T10:33:31.424 DEBUG:boto:auth_path=/vclfcn-22/obj?versionId=gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB 2024-02-01T10:33:31.424 DEBUG:boto:Method: GET 2024-02-01T10:33:31.424 DEBUG:boto:Path: /vclfcn-22/obj?versionId=gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB 2024-02-01T10:33:31.424 DEBUG:boto:Data: 2024-02-01T10:33:31.424 DEBUG:boto:Headers: {} 2024-02-01T10:33:31.424 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:33:31.425 DEBUG:boto:Port: 8000 2024-02-01T10:33:31.425 DEBUG:boto:Params: {} 2024-02-01T10:33:31.425 DEBUG:boto:Token: None 2024-02-01T10:33:31.426 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:31 GMT /vclfcn-22/obj?versionId=gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB 2024-02-01T10:33:31.426 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gZU+fS847i6HX3/6bRAuMAr/2Mw= 2024-02-01T10:33:31.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:31 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gZU+fS847i6HX3/6bRAuMAr/2Mw=', 'Content-Length': '0'} 2024-02-01T10:33:31.429 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:32:43 GMT'), ('x-amz-version-id', 'gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007475e6be7f29ee44-0065bb737b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:31 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:31.429 DEBUG:boto:path=/vclfcn-22/obj 2024-02-01T10:33:31.429 DEBUG:boto:auth_path=/vclfcn-22/obj 2024-02-01T10:33:31.429 DEBUG:boto:path=/vclfcn-22/obj?versionId=gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB 2024-02-01T10:33:31.429 DEBUG:boto:auth_path=/vclfcn-22/obj?versionId=gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB 2024-02-01T10:33:31.429 DEBUG:boto:Method: GET 2024-02-01T10:33:31.429 DEBUG:boto:Path: /vclfcn-22/obj?versionId=gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB 2024-02-01T10:33:31.429 DEBUG:boto:Data: 2024-02-01T10:33:31.430 DEBUG:boto:Headers: {} 2024-02-01T10:33:31.430 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:33:31.430 DEBUG:boto:Port: 8000 2024-02-01T10:33:31.430 DEBUG:boto:Params: {} 2024-02-01T10:33:31.430 DEBUG:boto:Token: None 2024-02-01T10:33:31.430 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:31 GMT /vclfcn-22/obj?versionId=gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB 2024-02-01T10:33:31.430 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gZU+fS847i6HX3/6bRAuMAr/2Mw= 2024-02-01T10:33:31.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:31 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gZU+fS847i6HX3/6bRAuMAr/2Mw=', 'Content-Length': '0'} 2024-02-01T10:33:31.433 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:32:43 GMT'), ('x-amz-version-id', 'gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bb30de531de7cbd6-0065bb737b-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:31 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:31.433 DEBUG:boto:path=/vclfcn-22/obj 2024-02-01T10:33:31.433 DEBUG:boto:auth_path=/vclfcn-22/obj 2024-02-01T10:33:31.433 DEBUG:boto:path=/vclfcn-22/obj?versionId=gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB 2024-02-01T10:33:31.433 DEBUG:boto:auth_path=/vclfcn-22/obj?versionId=gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB 2024-02-01T10:33:31.434 DEBUG:boto:Method: HEAD 2024-02-01T10:33:31.434 DEBUG:boto:Path: /vclfcn-22/obj?versionId=gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB 2024-02-01T10:33:31.434 DEBUG:boto:Data: 2024-02-01T10:33:31.434 DEBUG:boto:Headers: {} 2024-02-01T10:33:31.434 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:33:31.434 DEBUG:boto:Port: 8000 2024-02-01T10:33:31.434 DEBUG:boto:Params: {} 2024-02-01T10:33:31.434 DEBUG:boto:Token: None 2024-02-01T10:33:31.434 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:33:31 GMT /vclfcn-22/obj?versionId=gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB 2024-02-01T10:33:31.435 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:biVHi4JnRc/71OtPxxgR9MCvFlA= 2024-02-01T10:33:31.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:31 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:biVHi4JnRc/71OtPxxgR9MCvFlA=', 'Content-Length': '0'} 2024-02-01T10:33:31.437 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:32:43 GMT'), ('x-amz-version-id', 'gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000821d0c3879a8c2d6-0065bb737b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:31 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:31.438 DEBUG:boto:path=/vclfcn-22/obj 2024-02-01T10:33:31.438 DEBUG:boto:auth_path=/vclfcn-22/obj 2024-02-01T10:33:31.438 DEBUG:boto:path=/vclfcn-22/obj?versionId=gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB 2024-02-01T10:33:31.438 DEBUG:boto:auth_path=/vclfcn-22/obj?versionId=gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB 2024-02-01T10:33:31.438 DEBUG:boto:Method: HEAD 2024-02-01T10:33:31.438 DEBUG:boto:Path: /vclfcn-22/obj?versionId=gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB 2024-02-01T10:33:31.438 DEBUG:boto:Data: 2024-02-01T10:33:31.438 DEBUG:boto:Headers: {} 2024-02-01T10:33:31.438 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:33:31.438 DEBUG:boto:Port: 8000 2024-02-01T10:33:31.439 DEBUG:boto:Params: {} 2024-02-01T10:33:31.439 DEBUG:boto:Token: None 2024-02-01T10:33:31.439 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:33:31 GMT /vclfcn-22/obj?versionId=gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB 2024-02-01T10:33:31.439 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:biVHi4JnRc/71OtPxxgR9MCvFlA= 2024-02-01T10:33:31.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:31 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:biVHi4JnRc/71OtPxxgR9MCvFlA=', 'Content-Length': '0'} 2024-02-01T10:33:31.441 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:32:43 GMT'), ('x-amz-version-id', 'gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005d62327abe38eef0-0065bb737b-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:31 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:31.442 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:33:31.442 DEBUG:boto:path=/vclfcn-22/obj 2024-02-01T10:33:31.442 DEBUG:boto:auth_path=/vclfcn-22/obj 2024-02-01T10:33:31.442 DEBUG:boto:path=/vclfcn-22/obj?versionId=gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB 2024-02-01T10:33:31.442 DEBUG:boto:auth_path=/vclfcn-22/obj?versionId=gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB 2024-02-01T10:33:31.442 DEBUG:boto:Method: GET 2024-02-01T10:33:31.442 DEBUG:boto:Path: /vclfcn-22/obj?versionId=gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB 2024-02-01T10:33:31.443 DEBUG:boto:Data: 2024-02-01T10:33:31.443 DEBUG:boto:Headers: {} 2024-02-01T10:33:31.443 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:33:31.443 DEBUG:boto:Port: 8000 2024-02-01T10:33:31.443 DEBUG:boto:Params: {} 2024-02-01T10:33:31.443 DEBUG:boto:Token: None 2024-02-01T10:33:31.443 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:31 GMT /vclfcn-22/obj?versionId=gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB 2024-02-01T10:33:31.443 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gZU+fS847i6HX3/6bRAuMAr/2Mw= 2024-02-01T10:33:31.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:31 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gZU+fS847i6HX3/6bRAuMAr/2Mw=', 'Content-Length': '0'} 2024-02-01T10:33:31.446 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:32:43 GMT'), ('x-amz-version-id', 'gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000597041abcc0f620e-0065bb737b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:31 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:31.446 DEBUG:boto:path=/vclfcn-22/obj 2024-02-01T10:33:31.447 DEBUG:boto:auth_path=/vclfcn-22/obj 2024-02-01T10:33:31.447 DEBUG:boto:path=/vclfcn-22/obj?versionId=gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB 2024-02-01T10:33:31.447 DEBUG:boto:auth_path=/vclfcn-22/obj?versionId=gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB 2024-02-01T10:33:31.447 DEBUG:boto:Method: GET 2024-02-01T10:33:31.447 DEBUG:boto:Path: /vclfcn-22/obj?versionId=gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB 2024-02-01T10:33:31.447 DEBUG:boto:Data: 2024-02-01T10:33:31.447 DEBUG:boto:Headers: {} 2024-02-01T10:33:31.447 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:33:31.447 DEBUG:boto:Port: 8000 2024-02-01T10:33:31.448 DEBUG:boto:Params: {} 2024-02-01T10:33:31.448 DEBUG:boto:Token: None 2024-02-01T10:33:31.448 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:31 GMT /vclfcn-22/obj?versionId=gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB 2024-02-01T10:33:31.448 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gZU+fS847i6HX3/6bRAuMAr/2Mw= 2024-02-01T10:33:31.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:31 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gZU+fS847i6HX3/6bRAuMAr/2Mw=', 'Content-Length': '0'} 2024-02-01T10:33:31.451 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:32:43 GMT'), ('x-amz-version-id', 'gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000007d0327e4c055c15-0065bb737b-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:31 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:31.451 DEBUG:boto:path=/vclfcn-22/ 2024-02-01T10:33:31.451 DEBUG:boto:auth_path=/vclfcn-22/ 2024-02-01T10:33:31.451 DEBUG:boto:path=/vclfcn-22/?prefix=obj 2024-02-01T10:33:31.451 DEBUG:boto:auth_path=/vclfcn-22/?prefix=obj 2024-02-01T10:33:31.451 DEBUG:boto:Method: GET 2024-02-01T10:33:31.452 DEBUG:boto:Path: /vclfcn-22/?prefix=obj 2024-02-01T10:33:31.452 DEBUG:boto:Data: 2024-02-01T10:33:31.452 DEBUG:boto:Headers: {} 2024-02-01T10:33:31.452 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:33:31.452 DEBUG:boto:Port: 8000 2024-02-01T10:33:31.452 DEBUG:boto:Params: {} 2024-02-01T10:33:31.452 DEBUG:boto:Token: None 2024-02-01T10:33:31.452 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:31 GMT /vclfcn-22/ 2024-02-01T10:33:31.453 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Y4tHVN1dtmccOCe5DH8fLdhC18g= 2024-02-01T10:33:31.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:31 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Y4tHVN1dtmccOCe5DH8fLdhC18g=', 'Content-Length': '0'} 2024-02-01T10:33:31.456 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b2a0222693c6570a-0065bb737b-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:31 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:31.500 DEBUG:boto:b'vclfcn-22obj1000false' 2024-02-01T10:33:31.501 DEBUG:boto:path=/vclfcn-22/ 2024-02-01T10:33:31.501 DEBUG:boto:auth_path=/vclfcn-22/ 2024-02-01T10:33:31.501 DEBUG:boto:path=/vclfcn-22/?prefix=obj 2024-02-01T10:33:31.501 DEBUG:boto:auth_path=/vclfcn-22/?prefix=obj 2024-02-01T10:33:31.501 DEBUG:boto:Method: GET 2024-02-01T10:33:31.501 DEBUG:boto:Path: /vclfcn-22/?prefix=obj 2024-02-01T10:33:31.501 DEBUG:boto:Data: 2024-02-01T10:33:31.501 DEBUG:boto:Headers: {} 2024-02-01T10:33:31.501 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:33:31.501 DEBUG:boto:Port: 8000 2024-02-01T10:33:31.501 DEBUG:boto:Params: {} 2024-02-01T10:33:31.502 DEBUG:boto:Token: None 2024-02-01T10:33:31.502 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:31 GMT /vclfcn-22/ 2024-02-01T10:33:31.502 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Y4tHVN1dtmccOCe5DH8fLdhC18g= 2024-02-01T10:33:31.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:31 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Y4tHVN1dtmccOCe5DH8fLdhC18g=', 'Content-Length': '0'} 2024-02-01T10:33:31.506 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc4d3055ee0ec932-0065bb737b-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:31 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:31.506 DEBUG:boto:b'vclfcn-22obj1000false' 2024-02-01T10:33:31.507 DEBUG:boto:path=/vclfcn-22/obj 2024-02-01T10:33:31.507 DEBUG:boto:auth_path=/vclfcn-22/obj 2024-02-01T10:33:31.507 DEBUG:boto:Method: HEAD 2024-02-01T10:33:31.507 DEBUG:boto:Path: /vclfcn-22/obj 2024-02-01T10:33:31.507 DEBUG:boto:Data: 2024-02-01T10:33:31.507 DEBUG:boto:Headers: {} 2024-02-01T10:33:31.507 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:33:31.508 DEBUG:boto:Port: 8000 2024-02-01T10:33:31.508 DEBUG:boto:Params: {} 2024-02-01T10:33:31.508 DEBUG:boto:Token: None 2024-02-01T10:33:31.508 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:33:31 GMT /vclfcn-22/obj 2024-02-01T10:33:31.508 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ae65M6bYbcjYZbdghu8cIFNw8cQ= 2024-02-01T10:33:31.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:31 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ae65M6bYbcjYZbdghu8cIFNw8cQ=', 'Content-Length': '0'} 2024-02-01T10:33:31.512 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000c8ade94200a59169-0065bb737b-4307-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:31 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:31.512 DEBUG:boto:path=/vclfcn-22/obj 2024-02-01T10:33:31.512 DEBUG:boto:auth_path=/vclfcn-22/obj 2024-02-01T10:33:31.512 DEBUG:boto:Method: HEAD 2024-02-01T10:33:31.513 DEBUG:boto:Path: /vclfcn-22/obj 2024-02-01T10:33:31.513 DEBUG:boto:Data: 2024-02-01T10:33:31.513 DEBUG:boto:Headers: {} 2024-02-01T10:33:31.513 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:33:31.513 DEBUG:boto:Port: 8000 2024-02-01T10:33:31.513 DEBUG:boto:Params: {} 2024-02-01T10:33:31.513 DEBUG:boto:Token: None 2024-02-01T10:33:31.514 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:33:31 GMT /vclfcn-22/obj 2024-02-01T10:33:31.514 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ae65M6bYbcjYZbdghu8cIFNw8cQ= 2024-02-01T10:33:31.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:31 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ae65M6bYbcjYZbdghu8cIFNw8cQ=', 'Content-Length': '0'} 2024-02-01T10:33:31.517 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000001ccc59c5e502a46c-0065bb737b-4351-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:31 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:31.518 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-22 zones={test-zone2, test-zone1} 2024-02-01T10:33:31.518 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-22 zones={test-zone3, test-zone1} 2024-02-01T10:33:31.518 DEBUG:boto:path=/vclfcn-22/ 2024-02-01T10:33:31.518 DEBUG:boto:auth_path=/vclfcn-22/ 2024-02-01T10:33:31.518 DEBUG:boto:Method: HEAD 2024-02-01T10:33:31.518 DEBUG:boto:Path: /vclfcn-22/ 2024-02-01T10:33:31.518 DEBUG:boto:Data: 2024-02-01T10:33:31.518 DEBUG:boto:Headers: {} 2024-02-01T10:33:31.518 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:33:31.519 DEBUG:boto:Port: 8001 2024-02-01T10:33:31.519 DEBUG:boto:Params: {} 2024-02-01T10:33:31.519 DEBUG:boto:Token: None 2024-02-01T10:33:31.519 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:33:31 GMT /vclfcn-22/ 2024-02-01T10:33:31.519 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7IrmJjdenew8rKYsEPD5NvrbPbg= 2024-02-01T10:33:31.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:31 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7IrmJjdenew8rKYsEPD5NvrbPbg=', 'Content-Length': '0'} 2024-02-01T10:33:31.523 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000059b369c1bcfc3aaf-0065bb737b-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:33:31 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:31.524 DEBUG:boto:path=/vclfcn-22/ 2024-02-01T10:33:31.524 DEBUG:boto:auth_path=/vclfcn-22/ 2024-02-01T10:33:31.524 DEBUG:boto:Method: HEAD 2024-02-01T10:33:31.524 DEBUG:boto:Path: /vclfcn-22/ 2024-02-01T10:33:31.524 DEBUG:boto:Data: 2024-02-01T10:33:31.524 DEBUG:boto:Headers: {} 2024-02-01T10:33:31.524 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:33:31.524 DEBUG:boto:Port: 8000 2024-02-01T10:33:31.524 DEBUG:boto:Params: {} 2024-02-01T10:33:31.525 DEBUG:boto:Token: None 2024-02-01T10:33:31.525 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:33:31 GMT /vclfcn-22/ 2024-02-01T10:33:31.525 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7IrmJjdenew8rKYsEPD5NvrbPbg= 2024-02-01T10:33:31.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:31 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7IrmJjdenew8rKYsEPD5NvrbPbg=', 'Content-Length': '0'} 2024-02-01T10:33:31.529 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c11d425f4b6f2308-0065bb737b-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:33:31 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:31.529 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:33:31.529 DEBUG:boto:path=/vclfcn-22/ 2024-02-01T10:33:31.529 DEBUG:boto:auth_path=/vclfcn-22/ 2024-02-01T10:33:31.529 DEBUG:boto:path=/vclfcn-22/?versions&max-keys=999 2024-02-01T10:33:31.529 DEBUG:boto:auth_path=/vclfcn-22/?versions&max-keys=999 2024-02-01T10:33:31.529 DEBUG:boto:Method: GET 2024-02-01T10:33:31.530 DEBUG:boto:Path: /vclfcn-22/?versions&max-keys=999 2024-02-01T10:33:31.530 DEBUG:boto:Data: 2024-02-01T10:33:31.530 DEBUG:boto:Headers: {} 2024-02-01T10:33:31.530 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:33:31.530 DEBUG:boto:Port: 8001 2024-02-01T10:33:31.530 DEBUG:boto:Params: {} 2024-02-01T10:33:31.530 DEBUG:boto:Token: None 2024-02-01T10:33:31.530 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:31 GMT /vclfcn-22/?versions 2024-02-01T10:33:31.530 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:eM2liZWjXHGbRNz1UOHqi92IEEg= 2024-02-01T10:33:31.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:31 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:eM2liZWjXHGbRNz1UOHqi92IEEg=', 'Content-Length': '0'} 2024-02-01T10:33:31.534 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ebb6b8b0a357786f-0065bb737b-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:31 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:31.535 DEBUG:boto:b'vclfcn-22999falseobjB4BhmM3thsoWYrVmsTC9qpVYsD42.EGtrue2024-02-01T10:32:43.702Zrgw-multisite-test-userMultisite Test UserNormalobjgjHQyIwf4nDIizFkOIFbxEy7p.MOtRBfalse2024-02-01T10:32:43.666Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:33:31.535 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:33:31.535 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:33:31.535 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:33:31.535 DEBUG:boto:path=/vclfcn-22/ 2024-02-01T10:33:31.536 DEBUG:boto:auth_path=/vclfcn-22/ 2024-02-01T10:33:31.536 DEBUG:boto:path=/vclfcn-22/?versions&max-keys=999 2024-02-01T10:33:31.536 DEBUG:boto:auth_path=/vclfcn-22/?versions&max-keys=999 2024-02-01T10:33:31.536 DEBUG:boto:Method: GET 2024-02-01T10:33:31.536 DEBUG:boto:Path: /vclfcn-22/?versions&max-keys=999 2024-02-01T10:33:31.536 DEBUG:boto:Data: 2024-02-01T10:33:31.536 DEBUG:boto:Headers: {} 2024-02-01T10:33:31.536 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:33:31.536 DEBUG:boto:Port: 8000 2024-02-01T10:33:31.536 DEBUG:boto:Params: {} 2024-02-01T10:33:31.537 DEBUG:boto:Token: None 2024-02-01T10:33:31.537 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:31 GMT /vclfcn-22/?versions 2024-02-01T10:33:31.537 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:eM2liZWjXHGbRNz1UOHqi92IEEg= 2024-02-01T10:33:31.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:31 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:eM2liZWjXHGbRNz1UOHqi92IEEg=', 'Content-Length': '0'} 2024-02-01T10:33:31.540 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000002056b215b6673b-0065bb737b-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:31 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:31.584 DEBUG:boto:b'vclfcn-22999falseobjB4BhmM3thsoWYrVmsTC9qpVYsD42.EGtrue2024-02-01T10:32:43.702Zrgw-multisite-test-userMultisite Test UserNormalobjgjHQyIwf4nDIizFkOIFbxEy7p.MOtRBfalse2024-02-01T10:32:43.666Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:33:31.585 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:33:31.585 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:33:31.585 DEBUG:boto:path=/vclfcn-22/ 2024-02-01T10:33:31.585 DEBUG:boto:auth_path=/vclfcn-22/ 2024-02-01T10:33:31.585 DEBUG:boto:path=/vclfcn-22/?versions&max-keys=999 2024-02-01T10:33:31.585 DEBUG:boto:auth_path=/vclfcn-22/?versions&max-keys=999 2024-02-01T10:33:31.585 DEBUG:boto:Method: GET 2024-02-01T10:33:31.585 DEBUG:boto:Path: /vclfcn-22/?versions&max-keys=999 2024-02-01T10:33:31.586 DEBUG:boto:Data: 2024-02-01T10:33:31.586 DEBUG:boto:Headers: {} 2024-02-01T10:33:31.586 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:33:31.586 DEBUG:boto:Port: 8001 2024-02-01T10:33:31.586 DEBUG:boto:Params: {} 2024-02-01T10:33:31.586 DEBUG:boto:Token: None 2024-02-01T10:33:31.586 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:31 GMT /vclfcn-22/?versions 2024-02-01T10:33:31.586 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:eM2liZWjXHGbRNz1UOHqi92IEEg= 2024-02-01T10:33:31.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:31 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:eM2liZWjXHGbRNz1UOHqi92IEEg=', 'Content-Length': '0'} 2024-02-01T10:33:31.591 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003cd9e07d94dc2023-0065bb737b-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:31 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:31.591 DEBUG:boto:b'vclfcn-22999falseobjB4BhmM3thsoWYrVmsTC9qpVYsD42.EGtrue2024-02-01T10:32:43.702Zrgw-multisite-test-userMultisite Test UserNormalobjgjHQyIwf4nDIizFkOIFbxEy7p.MOtRBfalse2024-02-01T10:32:43.666Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:33:31.591 DEBUG:boto:path=/vclfcn-22/ 2024-02-01T10:33:31.591 DEBUG:boto:auth_path=/vclfcn-22/ 2024-02-01T10:33:31.591 DEBUG:boto:path=/vclfcn-22/?versions&max-keys=999 2024-02-01T10:33:31.592 DEBUG:boto:auth_path=/vclfcn-22/?versions&max-keys=999 2024-02-01T10:33:31.592 DEBUG:boto:Method: GET 2024-02-01T10:33:31.592 DEBUG:boto:Path: /vclfcn-22/?versions&max-keys=999 2024-02-01T10:33:31.592 DEBUG:boto:Data: 2024-02-01T10:33:31.592 DEBUG:boto:Headers: {} 2024-02-01T10:33:31.592 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:33:31.592 DEBUG:boto:Port: 8000 2024-02-01T10:33:31.592 DEBUG:boto:Params: {} 2024-02-01T10:33:31.592 DEBUG:boto:Token: None 2024-02-01T10:33:31.592 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:31 GMT /vclfcn-22/?versions 2024-02-01T10:33:31.593 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:eM2liZWjXHGbRNz1UOHqi92IEEg= 2024-02-01T10:33:31.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:31 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:eM2liZWjXHGbRNz1UOHqi92IEEg=', 'Content-Length': '0'} 2024-02-01T10:33:31.596 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006615fdcbc33c3047-0065bb737b-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:31 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:31.597 DEBUG:boto:b'vclfcn-22999falseobjB4BhmM3thsoWYrVmsTC9qpVYsD42.EGtrue2024-02-01T10:32:43.702Zrgw-multisite-test-userMultisite Test UserNormalobjgjHQyIwf4nDIizFkOIFbxEy7p.MOtRBfalse2024-02-01T10:32:43.666Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:33:31.597 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:33:31.597 DEBUG:boto:path=/vclfcn-22/obj 2024-02-01T10:33:31.597 DEBUG:boto:auth_path=/vclfcn-22/obj 2024-02-01T10:33:31.598 DEBUG:boto:Method: HEAD 2024-02-01T10:33:31.598 DEBUG:boto:Path: /vclfcn-22/obj 2024-02-01T10:33:31.598 DEBUG:boto:Data: 2024-02-01T10:33:31.598 DEBUG:boto:Headers: {} 2024-02-01T10:33:31.598 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:33:31.598 DEBUG:boto:Port: 8001 2024-02-01T10:33:31.598 DEBUG:boto:Params: {} 2024-02-01T10:33:31.598 DEBUG:boto:Token: None 2024-02-01T10:33:31.598 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:33:31 GMT /vclfcn-22/obj 2024-02-01T10:33:31.598 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ae65M6bYbcjYZbdghu8cIFNw8cQ= 2024-02-01T10:33:31.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:31 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ae65M6bYbcjYZbdghu8cIFNw8cQ=', 'Content-Length': '0'} 2024-02-01T10:33:31.602 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000aafdcfad4ea64b96-0065bb737b-4358-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:31 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:31.602 DEBUG:boto:path=/vclfcn-22/obj 2024-02-01T10:33:31.602 DEBUG:boto:auth_path=/vclfcn-22/obj 2024-02-01T10:33:31.602 DEBUG:boto:Method: HEAD 2024-02-01T10:33:31.602 DEBUG:boto:Path: /vclfcn-22/obj 2024-02-01T10:33:31.602 DEBUG:boto:Data: 2024-02-01T10:33:31.602 DEBUG:boto:Headers: {} 2024-02-01T10:33:31.603 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:33:31.603 DEBUG:boto:Port: 8000 2024-02-01T10:33:31.603 DEBUG:boto:Params: {} 2024-02-01T10:33:31.603 DEBUG:boto:Token: None 2024-02-01T10:33:31.603 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:33:31 GMT /vclfcn-22/obj 2024-02-01T10:33:31.603 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ae65M6bYbcjYZbdghu8cIFNw8cQ= 2024-02-01T10:33:31.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:31 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ae65M6bYbcjYZbdghu8cIFNw8cQ=', 'Content-Length': '0'} 2024-02-01T10:33:31.607 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000009a7333dd18d9c07f-0065bb737b-4351-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:31 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:31.607 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:33:31.607 DEBUG:boto:path=/vclfcn-22/obj 2024-02-01T10:33:31.607 DEBUG:boto:auth_path=/vclfcn-22/obj 2024-02-01T10:33:31.607 DEBUG:boto:path=/vclfcn-22/obj?versionId=gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB 2024-02-01T10:33:31.607 DEBUG:boto:auth_path=/vclfcn-22/obj?versionId=gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB 2024-02-01T10:33:31.607 DEBUG:boto:Method: GET 2024-02-01T10:33:31.607 DEBUG:boto:Path: /vclfcn-22/obj?versionId=gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB 2024-02-01T10:33:31.607 DEBUG:boto:Data: 2024-02-01T10:33:31.608 DEBUG:boto:Headers: {} 2024-02-01T10:33:31.608 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:33:31.608 DEBUG:boto:Port: 8001 2024-02-01T10:33:31.608 DEBUG:boto:Params: {} 2024-02-01T10:33:31.608 DEBUG:boto:Token: None 2024-02-01T10:33:31.608 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:31 GMT /vclfcn-22/obj?versionId=gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB 2024-02-01T10:33:31.608 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gZU+fS847i6HX3/6bRAuMAr/2Mw= 2024-02-01T10:33:31.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:31 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gZU+fS847i6HX3/6bRAuMAr/2Mw=', 'Content-Length': '0'} 2024-02-01T10:33:31.611 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:32:43 GMT'), ('x-amz-version-id', 'gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c57de9b004fdf7f0-0065bb737b-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:31 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:31.611 DEBUG:boto:path=/vclfcn-22/obj 2024-02-01T10:33:31.611 DEBUG:boto:auth_path=/vclfcn-22/obj 2024-02-01T10:33:31.612 DEBUG:boto:path=/vclfcn-22/obj?versionId=gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB 2024-02-01T10:33:31.612 DEBUG:boto:auth_path=/vclfcn-22/obj?versionId=gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB 2024-02-01T10:33:31.612 DEBUG:boto:Method: GET 2024-02-01T10:33:31.612 DEBUG:boto:Path: /vclfcn-22/obj?versionId=gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB 2024-02-01T10:33:31.612 DEBUG:boto:Data: 2024-02-01T10:33:31.612 DEBUG:boto:Headers: {} 2024-02-01T10:33:31.612 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:33:31.612 DEBUG:boto:Port: 8000 2024-02-01T10:33:31.613 DEBUG:boto:Params: {} 2024-02-01T10:33:31.613 DEBUG:boto:Token: None 2024-02-01T10:33:31.613 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:31 GMT /vclfcn-22/obj?versionId=gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB 2024-02-01T10:33:31.613 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gZU+fS847i6HX3/6bRAuMAr/2Mw= 2024-02-01T10:33:31.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:31 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gZU+fS847i6HX3/6bRAuMAr/2Mw=', 'Content-Length': '0'} 2024-02-01T10:33:31.615 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:32:43 GMT'), ('x-amz-version-id', 'gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000f322b6790b6d3a7-0065bb737b-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:31 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:31.616 DEBUG:boto:path=/vclfcn-22/obj 2024-02-01T10:33:31.616 DEBUG:boto:auth_path=/vclfcn-22/obj 2024-02-01T10:33:31.616 DEBUG:boto:path=/vclfcn-22/obj?versionId=gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB 2024-02-01T10:33:31.616 DEBUG:boto:auth_path=/vclfcn-22/obj?versionId=gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB 2024-02-01T10:33:31.616 DEBUG:boto:Method: HEAD 2024-02-01T10:33:31.616 DEBUG:boto:Path: /vclfcn-22/obj?versionId=gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB 2024-02-01T10:33:31.616 DEBUG:boto:Data: 2024-02-01T10:33:31.616 DEBUG:boto:Headers: {} 2024-02-01T10:33:31.617 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:33:31.617 DEBUG:boto:Port: 8001 2024-02-01T10:33:31.617 DEBUG:boto:Params: {} 2024-02-01T10:33:31.617 DEBUG:boto:Token: None 2024-02-01T10:33:31.617 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:33:31 GMT /vclfcn-22/obj?versionId=gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB 2024-02-01T10:33:31.617 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:biVHi4JnRc/71OtPxxgR9MCvFlA= 2024-02-01T10:33:31.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:31 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:biVHi4JnRc/71OtPxxgR9MCvFlA=', 'Content-Length': '0'} 2024-02-01T10:33:31.620 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:32:43 GMT'), ('x-amz-version-id', 'gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004f1c3790ce989aa8-0065bb737b-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:31 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:31.620 DEBUG:boto:path=/vclfcn-22/obj 2024-02-01T10:33:31.620 DEBUG:boto:auth_path=/vclfcn-22/obj 2024-02-01T10:33:31.620 DEBUG:boto:path=/vclfcn-22/obj?versionId=gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB 2024-02-01T10:33:31.620 DEBUG:boto:auth_path=/vclfcn-22/obj?versionId=gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB 2024-02-01T10:33:31.621 DEBUG:boto:Method: HEAD 2024-02-01T10:33:31.621 DEBUG:boto:Path: /vclfcn-22/obj?versionId=gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB 2024-02-01T10:33:31.621 DEBUG:boto:Data: 2024-02-01T10:33:31.621 DEBUG:boto:Headers: {} 2024-02-01T10:33:31.621 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:33:31.621 DEBUG:boto:Port: 8000 2024-02-01T10:33:31.621 DEBUG:boto:Params: {} 2024-02-01T10:33:31.621 DEBUG:boto:Token: None 2024-02-01T10:33:31.621 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:33:31 GMT /vclfcn-22/obj?versionId=gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB 2024-02-01T10:33:31.621 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:biVHi4JnRc/71OtPxxgR9MCvFlA= 2024-02-01T10:33:31.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:31 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:biVHi4JnRc/71OtPxxgR9MCvFlA=', 'Content-Length': '0'} 2024-02-01T10:33:31.624 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:32:43 GMT'), ('x-amz-version-id', 'gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000625c58c530202514-0065bb737b-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:31 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:31.624 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:33:31.625 DEBUG:boto:path=/vclfcn-22/obj 2024-02-01T10:33:31.625 DEBUG:boto:auth_path=/vclfcn-22/obj 2024-02-01T10:33:31.625 DEBUG:boto:path=/vclfcn-22/obj?versionId=gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB 2024-02-01T10:33:31.625 DEBUG:boto:auth_path=/vclfcn-22/obj?versionId=gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB 2024-02-01T10:33:31.625 DEBUG:boto:Method: GET 2024-02-01T10:33:31.625 DEBUG:boto:Path: /vclfcn-22/obj?versionId=gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB 2024-02-01T10:33:31.625 DEBUG:boto:Data: 2024-02-01T10:33:31.625 DEBUG:boto:Headers: {} 2024-02-01T10:33:31.625 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:33:31.625 DEBUG:boto:Port: 8001 2024-02-01T10:33:31.625 DEBUG:boto:Params: {} 2024-02-01T10:33:31.626 DEBUG:boto:Token: None 2024-02-01T10:33:31.626 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:31 GMT /vclfcn-22/obj?versionId=gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB 2024-02-01T10:33:31.626 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gZU+fS847i6HX3/6bRAuMAr/2Mw= 2024-02-01T10:33:31.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:31 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gZU+fS847i6HX3/6bRAuMAr/2Mw=', 'Content-Length': '0'} 2024-02-01T10:33:31.628 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:32:43 GMT'), ('x-amz-version-id', 'gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d3bc653a9c459c15-0065bb737b-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:31 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:31.629 DEBUG:boto:path=/vclfcn-22/obj 2024-02-01T10:33:31.629 DEBUG:boto:auth_path=/vclfcn-22/obj 2024-02-01T10:33:31.629 DEBUG:boto:path=/vclfcn-22/obj?versionId=gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB 2024-02-01T10:33:31.629 DEBUG:boto:auth_path=/vclfcn-22/obj?versionId=gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB 2024-02-01T10:33:31.629 DEBUG:boto:Method: GET 2024-02-01T10:33:31.629 DEBUG:boto:Path: /vclfcn-22/obj?versionId=gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB 2024-02-01T10:33:31.629 DEBUG:boto:Data: 2024-02-01T10:33:31.629 DEBUG:boto:Headers: {} 2024-02-01T10:33:31.630 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:33:31.630 DEBUG:boto:Port: 8000 2024-02-01T10:33:31.630 DEBUG:boto:Params: {} 2024-02-01T10:33:31.630 DEBUG:boto:Token: None 2024-02-01T10:33:31.630 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:31 GMT /vclfcn-22/obj?versionId=gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB 2024-02-01T10:33:31.630 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gZU+fS847i6HX3/6bRAuMAr/2Mw= 2024-02-01T10:33:31.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:31 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gZU+fS847i6HX3/6bRAuMAr/2Mw=', 'Content-Length': '0'} 2024-02-01T10:33:31.633 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:32:43 GMT'), ('x-amz-version-id', 'gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000055354ccb67b5b352-0065bb737b-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:31 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:31.633 DEBUG:boto:path=/vclfcn-22/ 2024-02-01T10:33:31.633 DEBUG:boto:auth_path=/vclfcn-22/ 2024-02-01T10:33:31.633 DEBUG:boto:path=/vclfcn-22/?prefix=obj 2024-02-01T10:33:31.633 DEBUG:boto:auth_path=/vclfcn-22/?prefix=obj 2024-02-01T10:33:31.634 DEBUG:boto:Method: GET 2024-02-01T10:33:31.634 DEBUG:boto:Path: /vclfcn-22/?prefix=obj 2024-02-01T10:33:31.634 DEBUG:boto:Data: 2024-02-01T10:33:31.634 DEBUG:boto:Headers: {} 2024-02-01T10:33:31.634 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:33:31.634 DEBUG:boto:Port: 8001 2024-02-01T10:33:31.634 DEBUG:boto:Params: {} 2024-02-01T10:33:31.634 DEBUG:boto:Token: None 2024-02-01T10:33:31.634 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:31 GMT /vclfcn-22/ 2024-02-01T10:33:31.634 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Y4tHVN1dtmccOCe5DH8fLdhC18g= 2024-02-01T10:33:31.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:31 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Y4tHVN1dtmccOCe5DH8fLdhC18g=', 'Content-Length': '0'} 2024-02-01T10:33:31.639 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5d1d19d4ceca5a8-0065bb737b-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:31 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:31.680 DEBUG:boto:b'vclfcn-22obj1000false' 2024-02-01T10:33:31.681 DEBUG:boto:path=/vclfcn-22/ 2024-02-01T10:33:31.681 DEBUG:boto:auth_path=/vclfcn-22/ 2024-02-01T10:33:31.681 DEBUG:boto:path=/vclfcn-22/?prefix=obj 2024-02-01T10:33:31.681 DEBUG:boto:auth_path=/vclfcn-22/?prefix=obj 2024-02-01T10:33:31.681 DEBUG:boto:Method: GET 2024-02-01T10:33:31.681 DEBUG:boto:Path: /vclfcn-22/?prefix=obj 2024-02-01T10:33:31.681 DEBUG:boto:Data: 2024-02-01T10:33:31.681 DEBUG:boto:Headers: {} 2024-02-01T10:33:31.681 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:33:31.681 DEBUG:boto:Port: 8000 2024-02-01T10:33:31.682 DEBUG:boto:Params: {} 2024-02-01T10:33:31.682 DEBUG:boto:Token: None 2024-02-01T10:33:31.682 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:31 GMT /vclfcn-22/ 2024-02-01T10:33:31.682 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Y4tHVN1dtmccOCe5DH8fLdhC18g= 2024-02-01T10:33:31.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:31 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Y4tHVN1dtmccOCe5DH8fLdhC18g=', 'Content-Length': '0'} 2024-02-01T10:33:31.686 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc9555096db13fdf-0065bb737b-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:31 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:31.687 DEBUG:boto:b'vclfcn-22obj1000false' 2024-02-01T10:33:31.687 DEBUG:boto:path=/vclfcn-22/obj 2024-02-01T10:33:31.687 DEBUG:boto:auth_path=/vclfcn-22/obj 2024-02-01T10:33:31.687 DEBUG:boto:Method: HEAD 2024-02-01T10:33:31.687 DEBUG:boto:Path: /vclfcn-22/obj 2024-02-01T10:33:31.687 DEBUG:boto:Data: 2024-02-01T10:33:31.687 DEBUG:boto:Headers: {} 2024-02-01T10:33:31.688 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:33:31.688 DEBUG:boto:Port: 8001 2024-02-01T10:33:31.688 DEBUG:boto:Params: {} 2024-02-01T10:33:31.688 DEBUG:boto:Token: None 2024-02-01T10:33:31.688 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:33:31 GMT /vclfcn-22/obj 2024-02-01T10:33:31.688 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ae65M6bYbcjYZbdghu8cIFNw8cQ= 2024-02-01T10:33:31.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:31 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ae65M6bYbcjYZbdghu8cIFNw8cQ=', 'Content-Length': '0'} 2024-02-01T10:33:31.692 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx0000005fdd6089fe67fc0-0065bb737b-4358-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:31 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:31.692 DEBUG:boto:path=/vclfcn-22/obj 2024-02-01T10:33:31.692 DEBUG:boto:auth_path=/vclfcn-22/obj 2024-02-01T10:33:31.692 DEBUG:boto:Method: HEAD 2024-02-01T10:33:31.692 DEBUG:boto:Path: /vclfcn-22/obj 2024-02-01T10:33:31.692 DEBUG:boto:Data: 2024-02-01T10:33:31.692 DEBUG:boto:Headers: {} 2024-02-01T10:33:31.692 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:33:31.692 DEBUG:boto:Port: 8000 2024-02-01T10:33:31.692 DEBUG:boto:Params: {} 2024-02-01T10:33:31.693 DEBUG:boto:Token: None 2024-02-01T10:33:31.693 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:33:31 GMT /vclfcn-22/obj 2024-02-01T10:33:31.693 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ae65M6bYbcjYZbdghu8cIFNw8cQ= 2024-02-01T10:33:31.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:31 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ae65M6bYbcjYZbdghu8cIFNw8cQ=', 'Content-Length': '0'} 2024-02-01T10:33:31.697 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000f760d7059009c7f8-0065bb737b-4351-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:31 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:31.697 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-22 zones={test-zone3, test-zone1} 2024-02-01T10:33:31.697 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-22 zones={test-zone3, test-zone2} 2024-02-01T10:33:31.697 DEBUG:boto:path=/vclfcn-22/ 2024-02-01T10:33:31.697 DEBUG:boto:auth_path=/vclfcn-22/ 2024-02-01T10:33:31.697 DEBUG:boto:Method: HEAD 2024-02-01T10:33:31.697 DEBUG:boto:Path: /vclfcn-22/ 2024-02-01T10:33:31.697 DEBUG:boto:Data: 2024-02-01T10:33:31.697 DEBUG:boto:Headers: {} 2024-02-01T10:33:31.698 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:33:31.698 DEBUG:boto:Port: 8001 2024-02-01T10:33:31.698 DEBUG:boto:Params: {} 2024-02-01T10:33:31.698 DEBUG:boto:Token: None 2024-02-01T10:33:31.698 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:33:31 GMT /vclfcn-22/ 2024-02-01T10:33:31.699 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7IrmJjdenew8rKYsEPD5NvrbPbg= 2024-02-01T10:33:31.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:31 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7IrmJjdenew8rKYsEPD5NvrbPbg=', 'Content-Length': '0'} 2024-02-01T10:33:31.703 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000099c776bb056c6c55-0065bb737b-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:33:31 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:31.703 DEBUG:boto:path=/vclfcn-22/ 2024-02-01T10:33:31.703 DEBUG:boto:auth_path=/vclfcn-22/ 2024-02-01T10:33:31.703 DEBUG:boto:Method: HEAD 2024-02-01T10:33:31.703 DEBUG:boto:Path: /vclfcn-22/ 2024-02-01T10:33:31.703 DEBUG:boto:Data: 2024-02-01T10:33:31.703 DEBUG:boto:Headers: {} 2024-02-01T10:33:31.703 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:33:31.703 DEBUG:boto:Port: 8000 2024-02-01T10:33:31.704 DEBUG:boto:Params: {} 2024-02-01T10:33:31.704 DEBUG:boto:Token: None 2024-02-01T10:33:31.704 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:33:31 GMT /vclfcn-22/ 2024-02-01T10:33:31.704 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7IrmJjdenew8rKYsEPD5NvrbPbg= 2024-02-01T10:33:31.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:31 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7IrmJjdenew8rKYsEPD5NvrbPbg=', 'Content-Length': '0'} 2024-02-01T10:33:31.708 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000638e610e22d3e668-0065bb737b-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:33:31 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:31.708 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:33:31.708 DEBUG:boto:path=/vclfcn-22/ 2024-02-01T10:33:31.708 DEBUG:boto:auth_path=/vclfcn-22/ 2024-02-01T10:33:31.708 DEBUG:boto:path=/vclfcn-22/?versions&max-keys=999 2024-02-01T10:33:31.709 DEBUG:boto:auth_path=/vclfcn-22/?versions&max-keys=999 2024-02-01T10:33:31.709 DEBUG:boto:Method: GET 2024-02-01T10:33:31.709 DEBUG:boto:Path: /vclfcn-22/?versions&max-keys=999 2024-02-01T10:33:31.709 DEBUG:boto:Data: 2024-02-01T10:33:31.709 DEBUG:boto:Headers: {} 2024-02-01T10:33:31.709 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:33:31.709 DEBUG:boto:Port: 8001 2024-02-01T10:33:31.709 DEBUG:boto:Params: {} 2024-02-01T10:33:31.709 DEBUG:boto:Token: None 2024-02-01T10:33:31.709 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:31 GMT /vclfcn-22/?versions 2024-02-01T10:33:31.710 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:eM2liZWjXHGbRNz1UOHqi92IEEg= 2024-02-01T10:33:31.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:31 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:eM2liZWjXHGbRNz1UOHqi92IEEg=', 'Content-Length': '0'} 2024-02-01T10:33:31.714 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c87293ca92dea9c-0065bb737b-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:31 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:31.872 DEBUG:boto:b'vclfcn-22999falseobjB4BhmM3thsoWYrVmsTC9qpVYsD42.EGtrue2024-02-01T10:32:43.702Zrgw-multisite-test-userMultisite Test UserNormalobjgjHQyIwf4nDIizFkOIFbxEy7p.MOtRBfalse2024-02-01T10:32:43.666Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:33:31.872 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:33:31.872 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:33:31.872 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:33:31.873 DEBUG:boto:path=/vclfcn-22/ 2024-02-01T10:33:31.873 DEBUG:boto:auth_path=/vclfcn-22/ 2024-02-01T10:33:31.873 DEBUG:boto:path=/vclfcn-22/?versions&max-keys=999 2024-02-01T10:33:31.873 DEBUG:boto:auth_path=/vclfcn-22/?versions&max-keys=999 2024-02-01T10:33:31.873 DEBUG:boto:Method: GET 2024-02-01T10:33:31.873 DEBUG:boto:Path: /vclfcn-22/?versions&max-keys=999 2024-02-01T10:33:31.873 DEBUG:boto:Data: 2024-02-01T10:33:31.873 DEBUG:boto:Headers: {} 2024-02-01T10:33:31.873 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:33:31.873 DEBUG:boto:Port: 8000 2024-02-01T10:33:31.873 DEBUG:boto:Params: {} 2024-02-01T10:33:31.874 DEBUG:boto:Token: None 2024-02-01T10:33:31.874 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:31 GMT /vclfcn-22/?versions 2024-02-01T10:33:31.874 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:eM2liZWjXHGbRNz1UOHqi92IEEg= 2024-02-01T10:33:31.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:31 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:eM2liZWjXHGbRNz1UOHqi92IEEg=', 'Content-Length': '0'} 2024-02-01T10:33:31.878 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d804bf46df440d97-0065bb737b-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:31 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:31.879 DEBUG:boto:b'vclfcn-22999falseobjB4BhmM3thsoWYrVmsTC9qpVYsD42.EGtrue2024-02-01T10:32:43.702Zrgw-multisite-test-userMultisite Test UserNormalobjgjHQyIwf4nDIizFkOIFbxEy7p.MOtRBfalse2024-02-01T10:32:43.666Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:33:31.879 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:33:31.879 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:33:31.879 DEBUG:boto:path=/vclfcn-22/ 2024-02-01T10:33:31.879 DEBUG:boto:auth_path=/vclfcn-22/ 2024-02-01T10:33:31.880 DEBUG:boto:path=/vclfcn-22/?versions&max-keys=999 2024-02-01T10:33:31.880 DEBUG:boto:auth_path=/vclfcn-22/?versions&max-keys=999 2024-02-01T10:33:31.880 DEBUG:boto:Method: GET 2024-02-01T10:33:31.880 DEBUG:boto:Path: /vclfcn-22/?versions&max-keys=999 2024-02-01T10:33:31.880 DEBUG:boto:Data: 2024-02-01T10:33:31.880 DEBUG:boto:Headers: {} 2024-02-01T10:33:31.880 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:33:31.880 DEBUG:boto:Port: 8001 2024-02-01T10:33:31.880 DEBUG:boto:Params: {} 2024-02-01T10:33:31.880 DEBUG:boto:Token: None 2024-02-01T10:33:31.880 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:31 GMT /vclfcn-22/?versions 2024-02-01T10:33:31.881 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:eM2liZWjXHGbRNz1UOHqi92IEEg= 2024-02-01T10:33:31.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:31 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:eM2liZWjXHGbRNz1UOHqi92IEEg=', 'Content-Length': '0'} 2024-02-01T10:33:31.886 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca5f3ba8bb11d846-0065bb737b-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:31 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:31.886 DEBUG:boto:b'vclfcn-22999falseobjB4BhmM3thsoWYrVmsTC9qpVYsD42.EGtrue2024-02-01T10:32:43.702Zrgw-multisite-test-userMultisite Test UserNormalobjgjHQyIwf4nDIizFkOIFbxEy7p.MOtRBfalse2024-02-01T10:32:43.666Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:33:31.887 DEBUG:boto:path=/vclfcn-22/ 2024-02-01T10:33:31.887 DEBUG:boto:auth_path=/vclfcn-22/ 2024-02-01T10:33:31.887 DEBUG:boto:path=/vclfcn-22/?versions&max-keys=999 2024-02-01T10:33:31.887 DEBUG:boto:auth_path=/vclfcn-22/?versions&max-keys=999 2024-02-01T10:33:31.887 DEBUG:boto:Method: GET 2024-02-01T10:33:31.887 DEBUG:boto:Path: /vclfcn-22/?versions&max-keys=999 2024-02-01T10:33:31.888 DEBUG:boto:Data: 2024-02-01T10:33:31.888 DEBUG:boto:Headers: {} 2024-02-01T10:33:31.888 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:33:31.888 DEBUG:boto:Port: 8000 2024-02-01T10:33:31.888 DEBUG:boto:Params: {} 2024-02-01T10:33:31.888 DEBUG:boto:Token: None 2024-02-01T10:33:31.888 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:31 GMT /vclfcn-22/?versions 2024-02-01T10:33:31.888 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:eM2liZWjXHGbRNz1UOHqi92IEEg= 2024-02-01T10:33:31.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:31 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:eM2liZWjXHGbRNz1UOHqi92IEEg=', 'Content-Length': '0'} 2024-02-01T10:33:31.893 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000000f4cd198bb79880-0065bb737b-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:31 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:31.893 DEBUG:boto:b'vclfcn-22999falseobjB4BhmM3thsoWYrVmsTC9qpVYsD42.EGtrue2024-02-01T10:32:43.702Zrgw-multisite-test-userMultisite Test UserNormalobjgjHQyIwf4nDIizFkOIFbxEy7p.MOtRBfalse2024-02-01T10:32:43.666Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:33:31.894 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:33:31.894 DEBUG:boto:path=/vclfcn-22/obj 2024-02-01T10:33:31.894 DEBUG:boto:auth_path=/vclfcn-22/obj 2024-02-01T10:33:31.894 DEBUG:boto:Method: HEAD 2024-02-01T10:33:31.894 DEBUG:boto:Path: /vclfcn-22/obj 2024-02-01T10:33:31.894 DEBUG:boto:Data: 2024-02-01T10:33:31.894 DEBUG:boto:Headers: {} 2024-02-01T10:33:31.894 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:33:31.894 DEBUG:boto:Port: 8001 2024-02-01T10:33:31.895 DEBUG:boto:Params: {} 2024-02-01T10:33:31.895 DEBUG:boto:Token: None 2024-02-01T10:33:31.895 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:33:31 GMT /vclfcn-22/obj 2024-02-01T10:33:31.895 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ae65M6bYbcjYZbdghu8cIFNw8cQ= 2024-02-01T10:33:31.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:31 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ae65M6bYbcjYZbdghu8cIFNw8cQ=', 'Content-Length': '0'} 2024-02-01T10:33:31.900 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000069e9e7414ef71f8-0065bb737b-4358-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:31 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:31.900 DEBUG:boto:path=/vclfcn-22/obj 2024-02-01T10:33:31.901 DEBUG:boto:auth_path=/vclfcn-22/obj 2024-02-01T10:33:31.901 DEBUG:boto:Method: HEAD 2024-02-01T10:33:31.901 DEBUG:boto:Path: /vclfcn-22/obj 2024-02-01T10:33:31.901 DEBUG:boto:Data: 2024-02-01T10:33:31.901 DEBUG:boto:Headers: {} 2024-02-01T10:33:31.901 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:33:31.901 DEBUG:boto:Port: 8000 2024-02-01T10:33:31.901 DEBUG:boto:Params: {} 2024-02-01T10:33:31.901 DEBUG:boto:Token: None 2024-02-01T10:33:31.901 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:33:31 GMT /vclfcn-22/obj 2024-02-01T10:33:31.902 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ae65M6bYbcjYZbdghu8cIFNw8cQ= 2024-02-01T10:33:31.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:31 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ae65M6bYbcjYZbdghu8cIFNw8cQ=', 'Content-Length': '0'} 2024-02-01T10:33:31.905 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000abdab37a97124e60-0065bb737b-4307-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:31 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:31.905 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:33:31.905 DEBUG:boto:path=/vclfcn-22/obj 2024-02-01T10:33:31.905 DEBUG:boto:auth_path=/vclfcn-22/obj 2024-02-01T10:33:31.906 DEBUG:boto:path=/vclfcn-22/obj?versionId=gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB 2024-02-01T10:33:31.906 DEBUG:boto:auth_path=/vclfcn-22/obj?versionId=gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB 2024-02-01T10:33:31.906 DEBUG:boto:Method: GET 2024-02-01T10:33:31.906 DEBUG:boto:Path: /vclfcn-22/obj?versionId=gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB 2024-02-01T10:33:31.906 DEBUG:boto:Data: 2024-02-01T10:33:31.906 DEBUG:boto:Headers: {} 2024-02-01T10:33:31.906 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:33:31.906 DEBUG:boto:Port: 8001 2024-02-01T10:33:31.906 DEBUG:boto:Params: {} 2024-02-01T10:33:31.906 DEBUG:boto:Token: None 2024-02-01T10:33:31.907 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:31 GMT /vclfcn-22/obj?versionId=gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB 2024-02-01T10:33:31.907 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gZU+fS847i6HX3/6bRAuMAr/2Mw= 2024-02-01T10:33:31.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:31 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gZU+fS847i6HX3/6bRAuMAr/2Mw=', 'Content-Length': '0'} 2024-02-01T10:33:31.909 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:32:43 GMT'), ('x-amz-version-id', 'gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d61f2826d690fc65-0065bb737b-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:31 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:31.910 DEBUG:boto:path=/vclfcn-22/obj 2024-02-01T10:33:31.910 DEBUG:boto:auth_path=/vclfcn-22/obj 2024-02-01T10:33:31.910 DEBUG:boto:path=/vclfcn-22/obj?versionId=gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB 2024-02-01T10:33:31.910 DEBUG:boto:auth_path=/vclfcn-22/obj?versionId=gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB 2024-02-01T10:33:31.910 DEBUG:boto:Method: GET 2024-02-01T10:33:31.910 DEBUG:boto:Path: /vclfcn-22/obj?versionId=gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB 2024-02-01T10:33:31.910 DEBUG:boto:Data: 2024-02-01T10:33:31.910 DEBUG:boto:Headers: {} 2024-02-01T10:33:31.911 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:33:31.911 DEBUG:boto:Port: 8000 2024-02-01T10:33:31.911 DEBUG:boto:Params: {} 2024-02-01T10:33:31.911 DEBUG:boto:Token: None 2024-02-01T10:33:31.911 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:31 GMT /vclfcn-22/obj?versionId=gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB 2024-02-01T10:33:31.911 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gZU+fS847i6HX3/6bRAuMAr/2Mw= 2024-02-01T10:33:31.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:31 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gZU+fS847i6HX3/6bRAuMAr/2Mw=', 'Content-Length': '0'} 2024-02-01T10:33:31.914 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:32:43 GMT'), ('x-amz-version-id', 'gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000082075b91d9fe3d79-0065bb737b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:31 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:31.915 DEBUG:boto:path=/vclfcn-22/obj 2024-02-01T10:33:31.915 DEBUG:boto:auth_path=/vclfcn-22/obj 2024-02-01T10:33:31.915 DEBUG:boto:path=/vclfcn-22/obj?versionId=gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB 2024-02-01T10:33:31.915 DEBUG:boto:auth_path=/vclfcn-22/obj?versionId=gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB 2024-02-01T10:33:31.915 DEBUG:boto:Method: HEAD 2024-02-01T10:33:31.915 DEBUG:boto:Path: /vclfcn-22/obj?versionId=gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB 2024-02-01T10:33:31.915 DEBUG:boto:Data: 2024-02-01T10:33:31.915 DEBUG:boto:Headers: {} 2024-02-01T10:33:31.915 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:33:31.916 DEBUG:boto:Port: 8001 2024-02-01T10:33:31.916 DEBUG:boto:Params: {} 2024-02-01T10:33:31.916 DEBUG:boto:Token: None 2024-02-01T10:33:31.916 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:33:31 GMT /vclfcn-22/obj?versionId=gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB 2024-02-01T10:33:31.916 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:biVHi4JnRc/71OtPxxgR9MCvFlA= 2024-02-01T10:33:31.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:31 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:biVHi4JnRc/71OtPxxgR9MCvFlA=', 'Content-Length': '0'} 2024-02-01T10:33:31.920 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:32:43 GMT'), ('x-amz-version-id', 'gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d9b3905950c8f62a-0065bb737b-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:31 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:31.920 DEBUG:boto:path=/vclfcn-22/obj 2024-02-01T10:33:31.920 DEBUG:boto:auth_path=/vclfcn-22/obj 2024-02-01T10:33:31.920 DEBUG:boto:path=/vclfcn-22/obj?versionId=gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB 2024-02-01T10:33:31.920 DEBUG:boto:auth_path=/vclfcn-22/obj?versionId=gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB 2024-02-01T10:33:31.920 DEBUG:boto:Method: HEAD 2024-02-01T10:33:31.921 DEBUG:boto:Path: /vclfcn-22/obj?versionId=gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB 2024-02-01T10:33:31.921 DEBUG:boto:Data: 2024-02-01T10:33:31.921 DEBUG:boto:Headers: {} 2024-02-01T10:33:31.921 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:33:31.921 DEBUG:boto:Port: 8000 2024-02-01T10:33:31.921 DEBUG:boto:Params: {} 2024-02-01T10:33:31.921 DEBUG:boto:Token: None 2024-02-01T10:33:31.921 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:33:31 GMT /vclfcn-22/obj?versionId=gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB 2024-02-01T10:33:31.921 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:biVHi4JnRc/71OtPxxgR9MCvFlA= 2024-02-01T10:33:31.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:31 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:biVHi4JnRc/71OtPxxgR9MCvFlA=', 'Content-Length': '0'} 2024-02-01T10:33:31.924 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:32:43 GMT'), ('x-amz-version-id', 'gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008cfbdeb5a3c79d1e-0065bb737b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:31 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:31.925 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:33:31.925 DEBUG:boto:path=/vclfcn-22/obj 2024-02-01T10:33:31.925 DEBUG:boto:auth_path=/vclfcn-22/obj 2024-02-01T10:33:31.925 DEBUG:boto:path=/vclfcn-22/obj?versionId=gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB 2024-02-01T10:33:31.925 DEBUG:boto:auth_path=/vclfcn-22/obj?versionId=gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB 2024-02-01T10:33:31.925 DEBUG:boto:Method: GET 2024-02-01T10:33:31.925 DEBUG:boto:Path: /vclfcn-22/obj?versionId=gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB 2024-02-01T10:33:31.925 DEBUG:boto:Data: 2024-02-01T10:33:31.925 DEBUG:boto:Headers: {} 2024-02-01T10:33:31.925 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:33:31.926 DEBUG:boto:Port: 8001 2024-02-01T10:33:31.926 DEBUG:boto:Params: {} 2024-02-01T10:33:31.926 DEBUG:boto:Token: None 2024-02-01T10:33:31.926 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:31 GMT /vclfcn-22/obj?versionId=gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB 2024-02-01T10:33:31.926 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gZU+fS847i6HX3/6bRAuMAr/2Mw= 2024-02-01T10:33:31.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:31 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gZU+fS847i6HX3/6bRAuMAr/2Mw=', 'Content-Length': '0'} 2024-02-01T10:33:31.929 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:32:43 GMT'), ('x-amz-version-id', 'gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a0c85b7680bac5dd-0065bb737b-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:31 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:31.929 DEBUG:boto:path=/vclfcn-22/obj 2024-02-01T10:33:31.929 DEBUG:boto:auth_path=/vclfcn-22/obj 2024-02-01T10:33:31.929 DEBUG:boto:path=/vclfcn-22/obj?versionId=gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB 2024-02-01T10:33:31.929 DEBUG:boto:auth_path=/vclfcn-22/obj?versionId=gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB 2024-02-01T10:33:31.930 DEBUG:boto:Method: GET 2024-02-01T10:33:31.930 DEBUG:boto:Path: /vclfcn-22/obj?versionId=gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB 2024-02-01T10:33:31.930 DEBUG:boto:Data: 2024-02-01T10:33:31.930 DEBUG:boto:Headers: {} 2024-02-01T10:33:31.930 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:33:31.930 DEBUG:boto:Port: 8000 2024-02-01T10:33:31.930 DEBUG:boto:Params: {} 2024-02-01T10:33:31.930 DEBUG:boto:Token: None 2024-02-01T10:33:31.930 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:31 GMT /vclfcn-22/obj?versionId=gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB 2024-02-01T10:33:31.930 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gZU+fS847i6HX3/6bRAuMAr/2Mw= 2024-02-01T10:33:31.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:31 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gZU+fS847i6HX3/6bRAuMAr/2Mw=', 'Content-Length': '0'} 2024-02-01T10:33:31.933 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:32:43 GMT'), ('x-amz-version-id', 'gjHQyIwf4nDIizFkOIFbxEy7p.MOtRB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d728b27e851dc140-0065bb737b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:31 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:31.933 DEBUG:boto:path=/vclfcn-22/ 2024-02-01T10:33:31.933 DEBUG:boto:auth_path=/vclfcn-22/ 2024-02-01T10:33:31.934 DEBUG:boto:path=/vclfcn-22/?prefix=obj 2024-02-01T10:33:31.934 DEBUG:boto:auth_path=/vclfcn-22/?prefix=obj 2024-02-01T10:33:31.934 DEBUG:boto:Method: GET 2024-02-01T10:33:31.934 DEBUG:boto:Path: /vclfcn-22/?prefix=obj 2024-02-01T10:33:31.934 DEBUG:boto:Data: 2024-02-01T10:33:31.934 DEBUG:boto:Headers: {} 2024-02-01T10:33:31.934 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:33:31.934 DEBUG:boto:Port: 8001 2024-02-01T10:33:31.934 DEBUG:boto:Params: {} 2024-02-01T10:33:31.934 DEBUG:boto:Token: None 2024-02-01T10:33:31.935 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:31 GMT /vclfcn-22/ 2024-02-01T10:33:31.935 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Y4tHVN1dtmccOCe5DH8fLdhC18g= 2024-02-01T10:33:31.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:31 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Y4tHVN1dtmccOCe5DH8fLdhC18g=', 'Content-Length': '0'} 2024-02-01T10:33:31.939 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cfdbcd748faf66df-0065bb737b-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:31 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:31.984 DEBUG:boto:b'vclfcn-22obj1000false' 2024-02-01T10:33:31.984 DEBUG:boto:path=/vclfcn-22/ 2024-02-01T10:33:31.985 DEBUG:boto:auth_path=/vclfcn-22/ 2024-02-01T10:33:31.985 DEBUG:boto:path=/vclfcn-22/?prefix=obj 2024-02-01T10:33:31.985 DEBUG:boto:auth_path=/vclfcn-22/?prefix=obj 2024-02-01T10:33:31.985 DEBUG:boto:Method: GET 2024-02-01T10:33:31.985 DEBUG:boto:Path: /vclfcn-22/?prefix=obj 2024-02-01T10:33:31.985 DEBUG:boto:Data: 2024-02-01T10:33:31.985 DEBUG:boto:Headers: {} 2024-02-01T10:33:31.985 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:33:31.985 DEBUG:boto:Port: 8000 2024-02-01T10:33:31.985 DEBUG:boto:Params: {} 2024-02-01T10:33:31.986 DEBUG:boto:Token: None 2024-02-01T10:33:31.986 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:31 GMT /vclfcn-22/ 2024-02-01T10:33:31.986 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Y4tHVN1dtmccOCe5DH8fLdhC18g= 2024-02-01T10:33:31.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:31 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Y4tHVN1dtmccOCe5DH8fLdhC18g=', 'Content-Length': '0'} 2024-02-01T10:33:31.990 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000257bb7b524c56310-0065bb737b-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:31 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:31.990 DEBUG:boto:b'vclfcn-22obj1000false' 2024-02-01T10:33:31.991 DEBUG:boto:path=/vclfcn-22/obj 2024-02-01T10:33:31.991 DEBUG:boto:auth_path=/vclfcn-22/obj 2024-02-01T10:33:31.991 DEBUG:boto:Method: HEAD 2024-02-01T10:33:31.991 DEBUG:boto:Path: /vclfcn-22/obj 2024-02-01T10:33:31.991 DEBUG:boto:Data: 2024-02-01T10:33:31.991 DEBUG:boto:Headers: {} 2024-02-01T10:33:31.991 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:33:31.991 DEBUG:boto:Port: 8001 2024-02-01T10:33:31.991 DEBUG:boto:Params: {} 2024-02-01T10:33:31.991 DEBUG:boto:Token: None 2024-02-01T10:33:31.992 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:33:31 GMT /vclfcn-22/obj 2024-02-01T10:33:31.992 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ae65M6bYbcjYZbdghu8cIFNw8cQ= 2024-02-01T10:33:31.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:31 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ae65M6bYbcjYZbdghu8cIFNw8cQ=', 'Content-Length': '0'} 2024-02-01T10:33:31.995 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx0000077779d01bded2b5d-0065bb737b-4358-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:31 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:31.995 DEBUG:boto:path=/vclfcn-22/obj 2024-02-01T10:33:31.996 DEBUG:boto:auth_path=/vclfcn-22/obj 2024-02-01T10:33:31.996 DEBUG:boto:Method: HEAD 2024-02-01T10:33:31.996 DEBUG:boto:Path: /vclfcn-22/obj 2024-02-01T10:33:31.996 DEBUG:boto:Data: 2024-02-01T10:33:31.996 DEBUG:boto:Headers: {} 2024-02-01T10:33:31.996 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:33:31.996 DEBUG:boto:Port: 8000 2024-02-01T10:33:31.996 DEBUG:boto:Params: {} 2024-02-01T10:33:31.996 DEBUG:boto:Token: None 2024-02-01T10:33:31.996 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:33:31 GMT /vclfcn-22/obj 2024-02-01T10:33:31.997 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ae65M6bYbcjYZbdghu8cIFNw8cQ= 2024-02-01T10:33:31.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:31 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ae65M6bYbcjYZbdghu8cIFNw8cQ=', 'Content-Length': '0'} 2024-02-01T10:33:32.000 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000d0c01ea7a73bf987-0065bb737b-4307-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:31 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:32.000 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-22 zones={test-zone3, test-zone2} 2024-02-01T10:33:32.000 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=vclfcn-23 2024-02-01T10:33:32.000 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:33:32.001 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:33:32.001 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket vclfcn-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-02-01T10:33:32.051 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:33:32.051 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:33:32.117 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:33:32.117+0000 7f798b927980 1 bucket sync caught up with source: 2024-02-01T10:33:32.118 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, , , , , , , , 00000000002.36.6, , ] 2024-02-01T10:33:32.118 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, , , , , , , , 00000000002.36.6, , ] 2024-02-01T10:33:32.118 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:33:32.117+0000 7f798b927980 0 bucket checkpoint complete 2024-02-01T10:33:32.141 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=vclfcn-23 2024-02-01T10:33:32.141 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:33:32.141 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:33:32.141 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:33:32.192 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:33:32.192 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:33:32.252 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:33:32.255+0000 7f9e0273d980 1 bucket sync caught up with source: 2024-02-01T10:33:32.252 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, , , , , , , , 00000000002.36.6, , ] 2024-02-01T10:33:32.252 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, , , , , , , , 00000000002.36.6, , ] 2024-02-01T10:33:32.252 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:33:32.255+0000 7f9e0273d980 0 bucket checkpoint complete 2024-02-01T10:33:32.275 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=vclfcn-23 2024-02-01T10:33:32.275 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:33:32.275 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:33:32.275 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-23 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:33:32.316 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:33:32.316 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:33:32.376 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:33:32.379+0000 7f3a6ec4c980 1 bucket sync caught up with source: 2024-02-01T10:33:32.376 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, , , , , , , , 00000000002.83.6, , ] 2024-02-01T10:33:32.376 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, , , , , , , , 00000000002.83.6, , ] 2024-02-01T10:33:32.376 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:33:32.379+0000 7f3a6ec4c980 0 bucket checkpoint complete 2024-02-01T10:33:32.398 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=vclfcn-23 2024-02-01T10:33:32.398 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:33:32.398 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:33:32.399 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-23 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:33:32.486 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:33:32.486 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:33:32.547 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:33:32.551+0000 7f9f1ebd8980 1 bucket sync caught up with source: 2024-02-01T10:33:32.547 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, , , , , , , , 00000000002.83.6, , ] 2024-02-01T10:33:32.547 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, , , , , , , , 00000000002.83.6, , ] 2024-02-01T10:33:32.547 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:33:32.551+0000 7f9f1ebd8980 0 bucket checkpoint complete 2024-02-01T10:33:32.568 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=vclfcn-23 2024-02-01T10:33:32.569 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:33:32.569 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:33:32.569 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-23 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:33:32.609 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:33:32.609 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:33:32.665 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:33:32.667+0000 7f5487e37980 1 bucket sync caught up with source: 2024-02-01T10:33:32.665 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, , , , , , , , 00000000003.95.12, , ] 2024-02-01T10:33:32.665 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, , , , , , , , 00000000003.95.12, , ] 2024-02-01T10:33:32.665 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:33:32.667+0000 7f5487e37980 0 bucket checkpoint complete 2024-02-01T10:33:32.691 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=vclfcn-23 2024-02-01T10:33:32.691 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:33:32.691 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:33:32.691 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket vclfcn-23 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-02-01T10:33:32.741 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:33:32.741 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:33:32.805 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:33:32.805+0000 7fada1206980 1 bucket sync caught up with source: 2024-02-01T10:33:32.806 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, , , , , , , , 00000000003.95.12, , ] 2024-02-01T10:33:32.806 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, , , , , , , , 00000000003.95.12, , ] 2024-02-01T10:33:32.806 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:33:32.805+0000 7fada1206980 0 bucket checkpoint complete 2024-02-01T10:33:32.829 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-23 zones={test-zone2, test-zone1} 2024-02-01T10:33:32.829 DEBUG:boto:path=/vclfcn-23/ 2024-02-01T10:33:32.830 DEBUG:boto:auth_path=/vclfcn-23/ 2024-02-01T10:33:32.830 DEBUG:boto:Method: HEAD 2024-02-01T10:33:32.830 DEBUG:boto:Path: /vclfcn-23/ 2024-02-01T10:33:32.830 DEBUG:boto:Data: 2024-02-01T10:33:32.830 DEBUG:boto:Headers: {} 2024-02-01T10:33:32.830 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:33:32.830 DEBUG:boto:Port: 8000 2024-02-01T10:33:32.830 DEBUG:boto:Params: {} 2024-02-01T10:33:32.830 DEBUG:boto:Token: None 2024-02-01T10:33:32.831 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:33:32 GMT /vclfcn-23/ 2024-02-01T10:33:32.831 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:S4O2qTqr+qIL6cSWipO9P/pcazc= 2024-02-01T10:33:32.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:S4O2qTqr+qIL6cSWipO9P/pcazc=', 'Content-Length': '0'} 2024-02-01T10:33:32.833 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000078ee2b3dd4fafe35-0065bb737c-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:33:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:32.833 DEBUG:boto:path=/vclfcn-23/ 2024-02-01T10:33:32.833 DEBUG:boto:auth_path=/vclfcn-23/ 2024-02-01T10:33:32.833 DEBUG:boto:Method: HEAD 2024-02-01T10:33:32.834 DEBUG:boto:Path: /vclfcn-23/ 2024-02-01T10:33:32.834 DEBUG:boto:Data: 2024-02-01T10:33:32.834 DEBUG:boto:Headers: {} 2024-02-01T10:33:32.834 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:33:32.834 DEBUG:boto:Port: 8000 2024-02-01T10:33:32.834 DEBUG:boto:Params: {} 2024-02-01T10:33:32.834 DEBUG:boto:Token: None 2024-02-01T10:33:32.834 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:33:32 GMT /vclfcn-23/ 2024-02-01T10:33:32.834 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:S4O2qTqr+qIL6cSWipO9P/pcazc= 2024-02-01T10:33:32.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:S4O2qTqr+qIL6cSWipO9P/pcazc=', 'Content-Length': '0'} 2024-02-01T10:33:32.838 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fc4513c8c51b6800-0065bb737c-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:33:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:32.839 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:33:32.839 DEBUG:boto:path=/vclfcn-23/ 2024-02-01T10:33:32.839 DEBUG:boto:auth_path=/vclfcn-23/ 2024-02-01T10:33:32.839 DEBUG:boto:path=/vclfcn-23/?versions&max-keys=999 2024-02-01T10:33:32.839 DEBUG:boto:auth_path=/vclfcn-23/?versions&max-keys=999 2024-02-01T10:33:32.839 DEBUG:boto:Method: GET 2024-02-01T10:33:32.839 DEBUG:boto:Path: /vclfcn-23/?versions&max-keys=999 2024-02-01T10:33:32.839 DEBUG:boto:Data: 2024-02-01T10:33:32.839 DEBUG:boto:Headers: {} 2024-02-01T10:33:32.839 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:33:32.840 DEBUG:boto:Port: 8000 2024-02-01T10:33:32.840 DEBUG:boto:Params: {} 2024-02-01T10:33:32.840 DEBUG:boto:Token: None 2024-02-01T10:33:32.840 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:32 GMT /vclfcn-23/?versions 2024-02-01T10:33:32.840 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:i7QV033izMXxP0ZGWw1cvIzgL7c= 2024-02-01T10:33:32.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:i7QV033izMXxP0ZGWw1cvIzgL7c=', 'Content-Length': '0'} 2024-02-01T10:33:32.842 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ddb3f2192aea7bc5-0065bb737c-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:32.843 DEBUG:boto:b'vclfcn-23999falseobjcfAEDiw8G8l6M7HFA62MzKKoAdV-ERAtrue2024-02-01T10:32:43.786Zrgw-multisite-test-userMultisite Test UserNormalobjFL110tVslNs52SVUbHZqPNK10BnpUdZfalse2024-02-01T10:32:43.732Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:33:32.843 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:33:32.843 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:33:32.843 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:33:32.843 DEBUG:boto:path=/vclfcn-23/ 2024-02-01T10:33:32.843 DEBUG:boto:auth_path=/vclfcn-23/ 2024-02-01T10:33:32.844 DEBUG:boto:path=/vclfcn-23/?versions&max-keys=999 2024-02-01T10:33:32.844 DEBUG:boto:auth_path=/vclfcn-23/?versions&max-keys=999 2024-02-01T10:33:32.844 DEBUG:boto:Method: GET 2024-02-01T10:33:32.844 DEBUG:boto:Path: /vclfcn-23/?versions&max-keys=999 2024-02-01T10:33:32.844 DEBUG:boto:Data: 2024-02-01T10:33:32.844 DEBUG:boto:Headers: {} 2024-02-01T10:33:32.844 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:33:32.844 DEBUG:boto:Port: 8000 2024-02-01T10:33:32.844 DEBUG:boto:Params: {} 2024-02-01T10:33:32.844 DEBUG:boto:Token: None 2024-02-01T10:33:32.845 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:32 GMT /vclfcn-23/?versions 2024-02-01T10:33:32.845 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:i7QV033izMXxP0ZGWw1cvIzgL7c= 2024-02-01T10:33:32.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:i7QV033izMXxP0ZGWw1cvIzgL7c=', 'Content-Length': '0'} 2024-02-01T10:33:32.849 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f9ed4236a77ded8-0065bb737c-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:32.849 DEBUG:boto:b'vclfcn-23999falseobjcfAEDiw8G8l6M7HFA62MzKKoAdV-ERAtrue2024-02-01T10:32:43.786Zrgw-multisite-test-userMultisite Test UserNormalobjFL110tVslNs52SVUbHZqPNK10BnpUdZfalse2024-02-01T10:32:43.732Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:33:32.850 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:33:32.850 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:33:32.850 DEBUG:boto:path=/vclfcn-23/ 2024-02-01T10:33:32.850 DEBUG:boto:auth_path=/vclfcn-23/ 2024-02-01T10:33:32.850 DEBUG:boto:path=/vclfcn-23/?versions&max-keys=999 2024-02-01T10:33:32.850 DEBUG:boto:auth_path=/vclfcn-23/?versions&max-keys=999 2024-02-01T10:33:32.851 DEBUG:boto:Method: GET 2024-02-01T10:33:32.851 DEBUG:boto:Path: /vclfcn-23/?versions&max-keys=999 2024-02-01T10:33:32.851 DEBUG:boto:Data: 2024-02-01T10:33:32.851 DEBUG:boto:Headers: {} 2024-02-01T10:33:32.851 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:33:32.851 DEBUG:boto:Port: 8000 2024-02-01T10:33:32.851 DEBUG:boto:Params: {} 2024-02-01T10:33:32.851 DEBUG:boto:Token: None 2024-02-01T10:33:32.851 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:32 GMT /vclfcn-23/?versions 2024-02-01T10:33:32.851 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:i7QV033izMXxP0ZGWw1cvIzgL7c= 2024-02-01T10:33:32.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:i7QV033izMXxP0ZGWw1cvIzgL7c=', 'Content-Length': '0'} 2024-02-01T10:33:32.854 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e3de5d14a810081-0065bb737c-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:32.854 DEBUG:boto:b'vclfcn-23999falseobjcfAEDiw8G8l6M7HFA62MzKKoAdV-ERAtrue2024-02-01T10:32:43.786Zrgw-multisite-test-userMultisite Test UserNormalobjFL110tVslNs52SVUbHZqPNK10BnpUdZfalse2024-02-01T10:32:43.732Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:33:32.854 DEBUG:boto:path=/vclfcn-23/ 2024-02-01T10:33:32.854 DEBUG:boto:auth_path=/vclfcn-23/ 2024-02-01T10:33:32.855 DEBUG:boto:path=/vclfcn-23/?versions&max-keys=999 2024-02-01T10:33:32.855 DEBUG:boto:auth_path=/vclfcn-23/?versions&max-keys=999 2024-02-01T10:33:32.855 DEBUG:boto:Method: GET 2024-02-01T10:33:32.855 DEBUG:boto:Path: /vclfcn-23/?versions&max-keys=999 2024-02-01T10:33:32.855 DEBUG:boto:Data: 2024-02-01T10:33:32.855 DEBUG:boto:Headers: {} 2024-02-01T10:33:32.855 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:33:32.855 DEBUG:boto:Port: 8000 2024-02-01T10:33:32.855 DEBUG:boto:Params: {} 2024-02-01T10:33:32.855 DEBUG:boto:Token: None 2024-02-01T10:33:32.855 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:32 GMT /vclfcn-23/?versions 2024-02-01T10:33:32.856 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:i7QV033izMXxP0ZGWw1cvIzgL7c= 2024-02-01T10:33:32.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:i7QV033izMXxP0ZGWw1cvIzgL7c=', 'Content-Length': '0'} 2024-02-01T10:33:32.860 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b6750d14a38eed1-0065bb737c-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:32.900 DEBUG:boto:b'vclfcn-23999falseobjcfAEDiw8G8l6M7HFA62MzKKoAdV-ERAtrue2024-02-01T10:32:43.786Zrgw-multisite-test-userMultisite Test UserNormalobjFL110tVslNs52SVUbHZqPNK10BnpUdZfalse2024-02-01T10:32:43.732Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:33:32.901 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:33:32.901 DEBUG:boto:path=/vclfcn-23/obj 2024-02-01T10:33:32.901 DEBUG:boto:auth_path=/vclfcn-23/obj 2024-02-01T10:33:32.901 DEBUG:boto:Method: HEAD 2024-02-01T10:33:32.901 DEBUG:boto:Path: /vclfcn-23/obj 2024-02-01T10:33:32.901 DEBUG:boto:Data: 2024-02-01T10:33:32.901 DEBUG:boto:Headers: {} 2024-02-01T10:33:32.901 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:33:32.902 DEBUG:boto:Port: 8000 2024-02-01T10:33:32.902 DEBUG:boto:Params: {} 2024-02-01T10:33:32.902 DEBUG:boto:Token: None 2024-02-01T10:33:32.902 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:33:32 GMT /vclfcn-23/obj 2024-02-01T10:33:32.902 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:CjU9N+TpazktAqgL44qx8Wakq8Q= 2024-02-01T10:33:32.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:CjU9N+TpazktAqgL44qx8Wakq8Q=', 'Content-Length': '0'} 2024-02-01T10:33:32.904 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000330fcd18d885cbe6-0065bb737c-4307-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:32.905 DEBUG:boto:path=/vclfcn-23/obj 2024-02-01T10:33:32.905 DEBUG:boto:auth_path=/vclfcn-23/obj 2024-02-01T10:33:32.905 DEBUG:boto:Method: HEAD 2024-02-01T10:33:32.905 DEBUG:boto:Path: /vclfcn-23/obj 2024-02-01T10:33:32.905 DEBUG:boto:Data: 2024-02-01T10:33:32.905 DEBUG:boto:Headers: {} 2024-02-01T10:33:32.905 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:33:32.905 DEBUG:boto:Port: 8000 2024-02-01T10:33:32.905 DEBUG:boto:Params: {} 2024-02-01T10:33:32.905 DEBUG:boto:Token: None 2024-02-01T10:33:32.906 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:33:32 GMT /vclfcn-23/obj 2024-02-01T10:33:32.906 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:CjU9N+TpazktAqgL44qx8Wakq8Q= 2024-02-01T10:33:32.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:CjU9N+TpazktAqgL44qx8Wakq8Q=', 'Content-Length': '0'} 2024-02-01T10:33:32.909 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000beff1f3b42bd4acd-0065bb737c-4351-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:32.909 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:33:32.910 DEBUG:boto:path=/vclfcn-23/obj 2024-02-01T10:33:32.910 DEBUG:boto:auth_path=/vclfcn-23/obj 2024-02-01T10:33:32.910 DEBUG:boto:path=/vclfcn-23/obj?versionId=FL110tVslNs52SVUbHZqPNK10BnpUdZ 2024-02-01T10:33:32.910 DEBUG:boto:auth_path=/vclfcn-23/obj?versionId=FL110tVslNs52SVUbHZqPNK10BnpUdZ 2024-02-01T10:33:32.910 DEBUG:boto:Method: GET 2024-02-01T10:33:32.910 DEBUG:boto:Path: /vclfcn-23/obj?versionId=FL110tVslNs52SVUbHZqPNK10BnpUdZ 2024-02-01T10:33:32.910 DEBUG:boto:Data: 2024-02-01T10:33:32.910 DEBUG:boto:Headers: {} 2024-02-01T10:33:32.910 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:33:32.910 DEBUG:boto:Port: 8000 2024-02-01T10:33:32.910 DEBUG:boto:Params: {} 2024-02-01T10:33:32.911 DEBUG:boto:Token: None 2024-02-01T10:33:32.911 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:32 GMT /vclfcn-23/obj?versionId=FL110tVslNs52SVUbHZqPNK10BnpUdZ 2024-02-01T10:33:32.911 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:soKVdOfPCXCSQjKtnMBGPqlq/8I= 2024-02-01T10:33:32.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:soKVdOfPCXCSQjKtnMBGPqlq/8I=', 'Content-Length': '0'} 2024-02-01T10:33:32.913 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:32:43 GMT'), ('x-amz-version-id', 'FL110tVslNs52SVUbHZqPNK10BnpUdZ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004e81c8621b0f6973-0065bb737c-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:32.913 DEBUG:boto:path=/vclfcn-23/obj 2024-02-01T10:33:32.913 DEBUG:boto:auth_path=/vclfcn-23/obj 2024-02-01T10:33:32.914 DEBUG:boto:path=/vclfcn-23/obj?versionId=FL110tVslNs52SVUbHZqPNK10BnpUdZ 2024-02-01T10:33:32.914 DEBUG:boto:auth_path=/vclfcn-23/obj?versionId=FL110tVslNs52SVUbHZqPNK10BnpUdZ 2024-02-01T10:33:32.914 DEBUG:boto:Method: GET 2024-02-01T10:33:32.914 DEBUG:boto:Path: /vclfcn-23/obj?versionId=FL110tVslNs52SVUbHZqPNK10BnpUdZ 2024-02-01T10:33:32.914 DEBUG:boto:Data: 2024-02-01T10:33:32.914 DEBUG:boto:Headers: {} 2024-02-01T10:33:32.914 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:33:32.914 DEBUG:boto:Port: 8000 2024-02-01T10:33:32.914 DEBUG:boto:Params: {} 2024-02-01T10:33:32.914 DEBUG:boto:Token: None 2024-02-01T10:33:32.915 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:32 GMT /vclfcn-23/obj?versionId=FL110tVslNs52SVUbHZqPNK10BnpUdZ 2024-02-01T10:33:32.915 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:soKVdOfPCXCSQjKtnMBGPqlq/8I= 2024-02-01T10:33:32.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:soKVdOfPCXCSQjKtnMBGPqlq/8I=', 'Content-Length': '0'} 2024-02-01T10:33:32.918 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:32:43 GMT'), ('x-amz-version-id', 'FL110tVslNs52SVUbHZqPNK10BnpUdZ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008722e3e26ab51d21-0065bb737c-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:32.918 DEBUG:boto:path=/vclfcn-23/obj 2024-02-01T10:33:32.918 DEBUG:boto:auth_path=/vclfcn-23/obj 2024-02-01T10:33:32.918 DEBUG:boto:path=/vclfcn-23/obj?versionId=FL110tVslNs52SVUbHZqPNK10BnpUdZ 2024-02-01T10:33:32.918 DEBUG:boto:auth_path=/vclfcn-23/obj?versionId=FL110tVslNs52SVUbHZqPNK10BnpUdZ 2024-02-01T10:33:32.919 DEBUG:boto:Method: HEAD 2024-02-01T10:33:32.919 DEBUG:boto:Path: /vclfcn-23/obj?versionId=FL110tVslNs52SVUbHZqPNK10BnpUdZ 2024-02-01T10:33:32.919 DEBUG:boto:Data: 2024-02-01T10:33:32.919 DEBUG:boto:Headers: {} 2024-02-01T10:33:32.919 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:33:32.919 DEBUG:boto:Port: 8000 2024-02-01T10:33:32.919 DEBUG:boto:Params: {} 2024-02-01T10:33:32.919 DEBUG:boto:Token: None 2024-02-01T10:33:32.919 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:33:32 GMT /vclfcn-23/obj?versionId=FL110tVslNs52SVUbHZqPNK10BnpUdZ 2024-02-01T10:33:32.919 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QrDH1OhhVDLGRsTH38VBwnK72c0= 2024-02-01T10:33:32.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QrDH1OhhVDLGRsTH38VBwnK72c0=', 'Content-Length': '0'} 2024-02-01T10:33:32.922 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:32:43 GMT'), ('x-amz-version-id', 'FL110tVslNs52SVUbHZqPNK10BnpUdZ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000550bdbe926f72a2e-0065bb737c-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:32.922 DEBUG:boto:path=/vclfcn-23/obj 2024-02-01T10:33:32.922 DEBUG:boto:auth_path=/vclfcn-23/obj 2024-02-01T10:33:32.922 DEBUG:boto:path=/vclfcn-23/obj?versionId=FL110tVslNs52SVUbHZqPNK10BnpUdZ 2024-02-01T10:33:32.922 DEBUG:boto:auth_path=/vclfcn-23/obj?versionId=FL110tVslNs52SVUbHZqPNK10BnpUdZ 2024-02-01T10:33:32.922 DEBUG:boto:Method: HEAD 2024-02-01T10:33:32.922 DEBUG:boto:Path: /vclfcn-23/obj?versionId=FL110tVslNs52SVUbHZqPNK10BnpUdZ 2024-02-01T10:33:32.923 DEBUG:boto:Data: 2024-02-01T10:33:32.923 DEBUG:boto:Headers: {} 2024-02-01T10:33:32.923 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:33:32.923 DEBUG:boto:Port: 8000 2024-02-01T10:33:32.923 DEBUG:boto:Params: {} 2024-02-01T10:33:32.923 DEBUG:boto:Token: None 2024-02-01T10:33:32.923 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:33:32 GMT /vclfcn-23/obj?versionId=FL110tVslNs52SVUbHZqPNK10BnpUdZ 2024-02-01T10:33:32.923 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QrDH1OhhVDLGRsTH38VBwnK72c0= 2024-02-01T10:33:32.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QrDH1OhhVDLGRsTH38VBwnK72c0=', 'Content-Length': '0'} 2024-02-01T10:33:32.926 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:32:43 GMT'), ('x-amz-version-id', 'FL110tVslNs52SVUbHZqPNK10BnpUdZ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ef17dd8cd348f924-0065bb737c-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:32.926 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:33:32.926 DEBUG:boto:path=/vclfcn-23/obj 2024-02-01T10:33:32.927 DEBUG:boto:auth_path=/vclfcn-23/obj 2024-02-01T10:33:32.927 DEBUG:boto:path=/vclfcn-23/obj?versionId=FL110tVslNs52SVUbHZqPNK10BnpUdZ 2024-02-01T10:33:32.927 DEBUG:boto:auth_path=/vclfcn-23/obj?versionId=FL110tVslNs52SVUbHZqPNK10BnpUdZ 2024-02-01T10:33:32.927 DEBUG:boto:Method: GET 2024-02-01T10:33:32.927 DEBUG:boto:Path: /vclfcn-23/obj?versionId=FL110tVslNs52SVUbHZqPNK10BnpUdZ 2024-02-01T10:33:32.927 DEBUG:boto:Data: 2024-02-01T10:33:32.927 DEBUG:boto:Headers: {} 2024-02-01T10:33:32.927 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:33:32.927 DEBUG:boto:Port: 8000 2024-02-01T10:33:32.927 DEBUG:boto:Params: {} 2024-02-01T10:33:32.927 DEBUG:boto:Token: None 2024-02-01T10:33:32.928 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:32 GMT /vclfcn-23/obj?versionId=FL110tVslNs52SVUbHZqPNK10BnpUdZ 2024-02-01T10:33:32.928 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:soKVdOfPCXCSQjKtnMBGPqlq/8I= 2024-02-01T10:33:32.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:soKVdOfPCXCSQjKtnMBGPqlq/8I=', 'Content-Length': '0'} 2024-02-01T10:33:32.931 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:32:43 GMT'), ('x-amz-version-id', 'FL110tVslNs52SVUbHZqPNK10BnpUdZ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000da96260804e08703-0065bb737c-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:32.931 DEBUG:boto:path=/vclfcn-23/obj 2024-02-01T10:33:32.931 DEBUG:boto:auth_path=/vclfcn-23/obj 2024-02-01T10:33:32.931 DEBUG:boto:path=/vclfcn-23/obj?versionId=FL110tVslNs52SVUbHZqPNK10BnpUdZ 2024-02-01T10:33:32.931 DEBUG:boto:auth_path=/vclfcn-23/obj?versionId=FL110tVslNs52SVUbHZqPNK10BnpUdZ 2024-02-01T10:33:32.931 DEBUG:boto:Method: GET 2024-02-01T10:33:32.931 DEBUG:boto:Path: /vclfcn-23/obj?versionId=FL110tVslNs52SVUbHZqPNK10BnpUdZ 2024-02-01T10:33:32.932 DEBUG:boto:Data: 2024-02-01T10:33:32.932 DEBUG:boto:Headers: {} 2024-02-01T10:33:32.932 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:33:32.932 DEBUG:boto:Port: 8000 2024-02-01T10:33:32.932 DEBUG:boto:Params: {} 2024-02-01T10:33:32.932 DEBUG:boto:Token: None 2024-02-01T10:33:32.932 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:32 GMT /vclfcn-23/obj?versionId=FL110tVslNs52SVUbHZqPNK10BnpUdZ 2024-02-01T10:33:32.932 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:soKVdOfPCXCSQjKtnMBGPqlq/8I= 2024-02-01T10:33:32.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:soKVdOfPCXCSQjKtnMBGPqlq/8I=', 'Content-Length': '0'} 2024-02-01T10:33:32.936 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:32:43 GMT'), ('x-amz-version-id', 'FL110tVslNs52SVUbHZqPNK10BnpUdZ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000af95a2e1539d3afa-0065bb737c-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:32.936 DEBUG:boto:path=/vclfcn-23/ 2024-02-01T10:33:32.936 DEBUG:boto:auth_path=/vclfcn-23/ 2024-02-01T10:33:32.936 DEBUG:boto:path=/vclfcn-23/?prefix=obj 2024-02-01T10:33:32.936 DEBUG:boto:auth_path=/vclfcn-23/?prefix=obj 2024-02-01T10:33:32.936 DEBUG:boto:Method: GET 2024-02-01T10:33:32.937 DEBUG:boto:Path: /vclfcn-23/?prefix=obj 2024-02-01T10:33:32.937 DEBUG:boto:Data: 2024-02-01T10:33:32.937 DEBUG:boto:Headers: {} 2024-02-01T10:33:32.937 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:33:32.937 DEBUG:boto:Port: 8000 2024-02-01T10:33:32.937 DEBUG:boto:Params: {} 2024-02-01T10:33:32.937 DEBUG:boto:Token: None 2024-02-01T10:33:32.937 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:32 GMT /vclfcn-23/ 2024-02-01T10:33:32.937 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:KgbkH/v0ahv5yW+MnVGHR+021W0= 2024-02-01T10:33:32.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:KgbkH/v0ahv5yW+MnVGHR+021W0=', 'Content-Length': '0'} 2024-02-01T10:33:32.942 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ee6f35a7d4aff30-0065bb737c-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:32.984 DEBUG:boto:b'vclfcn-23obj1000false' 2024-02-01T10:33:32.984 DEBUG:boto:path=/vclfcn-23/ 2024-02-01T10:33:32.985 DEBUG:boto:auth_path=/vclfcn-23/ 2024-02-01T10:33:32.985 DEBUG:boto:path=/vclfcn-23/?prefix=obj 2024-02-01T10:33:32.985 DEBUG:boto:auth_path=/vclfcn-23/?prefix=obj 2024-02-01T10:33:32.985 DEBUG:boto:Method: GET 2024-02-01T10:33:32.985 DEBUG:boto:Path: /vclfcn-23/?prefix=obj 2024-02-01T10:33:32.985 DEBUG:boto:Data: 2024-02-01T10:33:32.985 DEBUG:boto:Headers: {} 2024-02-01T10:33:32.985 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:33:32.985 DEBUG:boto:Port: 8000 2024-02-01T10:33:32.985 DEBUG:boto:Params: {} 2024-02-01T10:33:32.986 DEBUG:boto:Token: None 2024-02-01T10:33:32.986 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:32 GMT /vclfcn-23/ 2024-02-01T10:33:32.986 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:KgbkH/v0ahv5yW+MnVGHR+021W0= 2024-02-01T10:33:32.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:KgbkH/v0ahv5yW+MnVGHR+021W0=', 'Content-Length': '0'} 2024-02-01T10:33:32.990 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b4ae92eb8fbd8d3-0065bb737c-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:32.991 DEBUG:boto:b'vclfcn-23obj1000false' 2024-02-01T10:33:32.991 DEBUG:boto:path=/vclfcn-23/obj 2024-02-01T10:33:32.991 DEBUG:boto:auth_path=/vclfcn-23/obj 2024-02-01T10:33:32.991 DEBUG:boto:Method: HEAD 2024-02-01T10:33:32.991 DEBUG:boto:Path: /vclfcn-23/obj 2024-02-01T10:33:32.991 DEBUG:boto:Data: 2024-02-01T10:33:32.991 DEBUG:boto:Headers: {} 2024-02-01T10:33:32.991 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:33:32.992 DEBUG:boto:Port: 8000 2024-02-01T10:33:32.992 DEBUG:boto:Params: {} 2024-02-01T10:33:32.992 DEBUG:boto:Token: None 2024-02-01T10:33:32.992 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:33:32 GMT /vclfcn-23/obj 2024-02-01T10:33:32.992 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:CjU9N+TpazktAqgL44qx8Wakq8Q= 2024-02-01T10:33:32.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:CjU9N+TpazktAqgL44qx8Wakq8Q=', 'Content-Length': '0'} 2024-02-01T10:33:32.996 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000449c7bb16888a122-0065bb737c-4307-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:32.996 DEBUG:boto:path=/vclfcn-23/obj 2024-02-01T10:33:32.996 DEBUG:boto:auth_path=/vclfcn-23/obj 2024-02-01T10:33:32.996 DEBUG:boto:Method: HEAD 2024-02-01T10:33:32.996 DEBUG:boto:Path: /vclfcn-23/obj 2024-02-01T10:33:32.996 DEBUG:boto:Data: 2024-02-01T10:33:32.996 DEBUG:boto:Headers: {} 2024-02-01T10:33:32.996 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:33:32.997 DEBUG:boto:Port: 8000 2024-02-01T10:33:32.997 DEBUG:boto:Params: {} 2024-02-01T10:33:32.997 DEBUG:boto:Token: None 2024-02-01T10:33:32.997 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:33:32 GMT /vclfcn-23/obj 2024-02-01T10:33:32.997 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:CjU9N+TpazktAqgL44qx8Wakq8Q= 2024-02-01T10:33:32.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:CjU9N+TpazktAqgL44qx8Wakq8Q=', 'Content-Length': '0'} 2024-02-01T10:33:33.001 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000a8b4b9f6bf699efd-0065bb737d-4351-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:33.001 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-23 zones={test-zone2, test-zone1} 2024-02-01T10:33:33.001 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-23 zones={test-zone3, test-zone1} 2024-02-01T10:33:33.001 DEBUG:boto:path=/vclfcn-23/ 2024-02-01T10:33:33.001 DEBUG:boto:auth_path=/vclfcn-23/ 2024-02-01T10:33:33.001 DEBUG:boto:Method: HEAD 2024-02-01T10:33:33.002 DEBUG:boto:Path: /vclfcn-23/ 2024-02-01T10:33:33.002 DEBUG:boto:Data: 2024-02-01T10:33:33.002 DEBUG:boto:Headers: {} 2024-02-01T10:33:33.002 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:33:33.002 DEBUG:boto:Port: 8001 2024-02-01T10:33:33.002 DEBUG:boto:Params: {} 2024-02-01T10:33:33.002 DEBUG:boto:Token: None 2024-02-01T10:33:33.002 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:33:33 GMT /vclfcn-23/ 2024-02-01T10:33:33.002 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:l6ZeWW45N/XduizpjyUL0CmoOaA= 2024-02-01T10:33:33.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:l6ZeWW45N/XduizpjyUL0CmoOaA=', 'Content-Length': '0'} 2024-02-01T10:33:33.007 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001c8389df0d93565b-0065bb737d-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:33:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:33.007 DEBUG:boto:path=/vclfcn-23/ 2024-02-01T10:33:33.007 DEBUG:boto:auth_path=/vclfcn-23/ 2024-02-01T10:33:33.007 DEBUG:boto:Method: HEAD 2024-02-01T10:33:33.007 DEBUG:boto:Path: /vclfcn-23/ 2024-02-01T10:33:33.008 DEBUG:boto:Data: 2024-02-01T10:33:33.008 DEBUG:boto:Headers: {} 2024-02-01T10:33:33.008 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:33:33.008 DEBUG:boto:Port: 8000 2024-02-01T10:33:33.008 DEBUG:boto:Params: {} 2024-02-01T10:33:33.008 DEBUG:boto:Token: None 2024-02-01T10:33:33.008 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:33:33 GMT /vclfcn-23/ 2024-02-01T10:33:33.008 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:l6ZeWW45N/XduizpjyUL0CmoOaA= 2024-02-01T10:33:33.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:l6ZeWW45N/XduizpjyUL0CmoOaA=', 'Content-Length': '0'} 2024-02-01T10:33:33.012 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d180885ed4f36fd8-0065bb737d-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:33:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:33.012 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:33:33.013 DEBUG:boto:path=/vclfcn-23/ 2024-02-01T10:33:33.013 DEBUG:boto:auth_path=/vclfcn-23/ 2024-02-01T10:33:33.013 DEBUG:boto:path=/vclfcn-23/?versions&max-keys=999 2024-02-01T10:33:33.013 DEBUG:boto:auth_path=/vclfcn-23/?versions&max-keys=999 2024-02-01T10:33:33.013 DEBUG:boto:Method: GET 2024-02-01T10:33:33.013 DEBUG:boto:Path: /vclfcn-23/?versions&max-keys=999 2024-02-01T10:33:33.013 DEBUG:boto:Data: 2024-02-01T10:33:33.013 DEBUG:boto:Headers: {} 2024-02-01T10:33:33.013 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:33:33.013 DEBUG:boto:Port: 8001 2024-02-01T10:33:33.014 DEBUG:boto:Params: {} 2024-02-01T10:33:33.014 DEBUG:boto:Token: None 2024-02-01T10:33:33.014 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:33 GMT /vclfcn-23/?versions 2024-02-01T10:33:33.014 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:m1YLTB8Jwrtit7aJ/IKpGNWnauM= 2024-02-01T10:33:33.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:m1YLTB8Jwrtit7aJ/IKpGNWnauM=', 'Content-Length': '0'} 2024-02-01T10:33:33.018 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000835a62acb8a11da3-0065bb737d-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:33.019 DEBUG:boto:b'vclfcn-23999falseobjcfAEDiw8G8l6M7HFA62MzKKoAdV-ERAtrue2024-02-01T10:32:43.786Zrgw-multisite-test-userMultisite Test UserNormalobjFL110tVslNs52SVUbHZqPNK10BnpUdZfalse2024-02-01T10:32:43.732Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:33:33.019 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:33:33.019 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:33:33.019 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:33:33.019 DEBUG:boto:path=/vclfcn-23/ 2024-02-01T10:33:33.019 DEBUG:boto:auth_path=/vclfcn-23/ 2024-02-01T10:33:33.020 DEBUG:boto:path=/vclfcn-23/?versions&max-keys=999 2024-02-01T10:33:33.020 DEBUG:boto:auth_path=/vclfcn-23/?versions&max-keys=999 2024-02-01T10:33:33.020 DEBUG:boto:Method: GET 2024-02-01T10:33:33.020 DEBUG:boto:Path: /vclfcn-23/?versions&max-keys=999 2024-02-01T10:33:33.020 DEBUG:boto:Data: 2024-02-01T10:33:33.020 DEBUG:boto:Headers: {} 2024-02-01T10:33:33.020 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:33:33.020 DEBUG:boto:Port: 8000 2024-02-01T10:33:33.020 DEBUG:boto:Params: {} 2024-02-01T10:33:33.020 DEBUG:boto:Token: None 2024-02-01T10:33:33.020 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:33 GMT /vclfcn-23/?versions 2024-02-01T10:33:33.021 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:m1YLTB8Jwrtit7aJ/IKpGNWnauM= 2024-02-01T10:33:33.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:m1YLTB8Jwrtit7aJ/IKpGNWnauM=', 'Content-Length': '0'} 2024-02-01T10:33:33.026 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c66ab173f3a9038b-0065bb737d-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:33.068 DEBUG:boto:b'vclfcn-23999falseobjcfAEDiw8G8l6M7HFA62MzKKoAdV-ERAtrue2024-02-01T10:32:43.786Zrgw-multisite-test-userMultisite Test UserNormalobjFL110tVslNs52SVUbHZqPNK10BnpUdZfalse2024-02-01T10:32:43.732Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:33:33.069 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:33:33.069 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:33:33.069 DEBUG:boto:path=/vclfcn-23/ 2024-02-01T10:33:33.070 DEBUG:boto:auth_path=/vclfcn-23/ 2024-02-01T10:33:33.070 DEBUG:boto:path=/vclfcn-23/?versions&max-keys=999 2024-02-01T10:33:33.070 DEBUG:boto:auth_path=/vclfcn-23/?versions&max-keys=999 2024-02-01T10:33:33.070 DEBUG:boto:Method: GET 2024-02-01T10:33:33.070 DEBUG:boto:Path: /vclfcn-23/?versions&max-keys=999 2024-02-01T10:33:33.070 DEBUG:boto:Data: 2024-02-01T10:33:33.070 DEBUG:boto:Headers: {} 2024-02-01T10:33:33.070 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:33:33.070 DEBUG:boto:Port: 8001 2024-02-01T10:33:33.071 DEBUG:boto:Params: {} 2024-02-01T10:33:33.071 DEBUG:boto:Token: None 2024-02-01T10:33:33.071 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:33 GMT /vclfcn-23/?versions 2024-02-01T10:33:33.071 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:m1YLTB8Jwrtit7aJ/IKpGNWnauM= 2024-02-01T10:33:33.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:m1YLTB8Jwrtit7aJ/IKpGNWnauM=', 'Content-Length': '0'} 2024-02-01T10:33:33.076 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef7387d455eac8ad-0065bb737d-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:33.076 DEBUG:boto:b'vclfcn-23999falseobjcfAEDiw8G8l6M7HFA62MzKKoAdV-ERAtrue2024-02-01T10:32:43.786Zrgw-multisite-test-userMultisite Test UserNormalobjFL110tVslNs52SVUbHZqPNK10BnpUdZfalse2024-02-01T10:32:43.732Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:33:33.076 DEBUG:boto:path=/vclfcn-23/ 2024-02-01T10:33:33.077 DEBUG:boto:auth_path=/vclfcn-23/ 2024-02-01T10:33:33.077 DEBUG:boto:path=/vclfcn-23/?versions&max-keys=999 2024-02-01T10:33:33.077 DEBUG:boto:auth_path=/vclfcn-23/?versions&max-keys=999 2024-02-01T10:33:33.077 DEBUG:boto:Method: GET 2024-02-01T10:33:33.077 DEBUG:boto:Path: /vclfcn-23/?versions&max-keys=999 2024-02-01T10:33:33.077 DEBUG:boto:Data: 2024-02-01T10:33:33.077 DEBUG:boto:Headers: {} 2024-02-01T10:33:33.077 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:33:33.077 DEBUG:boto:Port: 8000 2024-02-01T10:33:33.077 DEBUG:boto:Params: {} 2024-02-01T10:33:33.077 DEBUG:boto:Token: None 2024-02-01T10:33:33.078 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:33 GMT /vclfcn-23/?versions 2024-02-01T10:33:33.078 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:m1YLTB8Jwrtit7aJ/IKpGNWnauM= 2024-02-01T10:33:33.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:m1YLTB8Jwrtit7aJ/IKpGNWnauM=', 'Content-Length': '0'} 2024-02-01T10:33:33.082 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f297eeaddf2ab32-0065bb737d-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:33.082 DEBUG:boto:b'vclfcn-23999falseobjcfAEDiw8G8l6M7HFA62MzKKoAdV-ERAtrue2024-02-01T10:32:43.786Zrgw-multisite-test-userMultisite Test UserNormalobjFL110tVslNs52SVUbHZqPNK10BnpUdZfalse2024-02-01T10:32:43.732Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:33:33.083 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:33:33.083 DEBUG:boto:path=/vclfcn-23/obj 2024-02-01T10:33:33.083 DEBUG:boto:auth_path=/vclfcn-23/obj 2024-02-01T10:33:33.083 DEBUG:boto:Method: HEAD 2024-02-01T10:33:33.083 DEBUG:boto:Path: /vclfcn-23/obj 2024-02-01T10:33:33.083 DEBUG:boto:Data: 2024-02-01T10:33:33.083 DEBUG:boto:Headers: {} 2024-02-01T10:33:33.083 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:33:33.084 DEBUG:boto:Port: 8001 2024-02-01T10:33:33.084 DEBUG:boto:Params: {} 2024-02-01T10:33:33.084 DEBUG:boto:Token: None 2024-02-01T10:33:33.084 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:33:33 GMT /vclfcn-23/obj 2024-02-01T10:33:33.084 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:qFSRYVpVpYh1oINPFs7EMzTdO74= 2024-02-01T10:33:33.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:qFSRYVpVpYh1oINPFs7EMzTdO74=', 'Content-Length': '0'} 2024-02-01T10:33:33.087 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000d303fa7408532a77-0065bb737d-4358-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:33.088 DEBUG:boto:path=/vclfcn-23/obj 2024-02-01T10:33:33.088 DEBUG:boto:auth_path=/vclfcn-23/obj 2024-02-01T10:33:33.088 DEBUG:boto:Method: HEAD 2024-02-01T10:33:33.088 DEBUG:boto:Path: /vclfcn-23/obj 2024-02-01T10:33:33.088 DEBUG:boto:Data: 2024-02-01T10:33:33.088 DEBUG:boto:Headers: {} 2024-02-01T10:33:33.088 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:33:33.088 DEBUG:boto:Port: 8000 2024-02-01T10:33:33.089 DEBUG:boto:Params: {} 2024-02-01T10:33:33.089 DEBUG:boto:Token: None 2024-02-01T10:33:33.089 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:33:33 GMT /vclfcn-23/obj 2024-02-01T10:33:33.089 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:qFSRYVpVpYh1oINPFs7EMzTdO74= 2024-02-01T10:33:33.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:qFSRYVpVpYh1oINPFs7EMzTdO74=', 'Content-Length': '0'} 2024-02-01T10:33:33.092 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000002b034ecaa0471573-0065bb737d-4351-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:33.093 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:33:33.093 DEBUG:boto:path=/vclfcn-23/obj 2024-02-01T10:33:33.093 DEBUG:boto:auth_path=/vclfcn-23/obj 2024-02-01T10:33:33.093 DEBUG:boto:path=/vclfcn-23/obj?versionId=FL110tVslNs52SVUbHZqPNK10BnpUdZ 2024-02-01T10:33:33.093 DEBUG:boto:auth_path=/vclfcn-23/obj?versionId=FL110tVslNs52SVUbHZqPNK10BnpUdZ 2024-02-01T10:33:33.093 DEBUG:boto:Method: GET 2024-02-01T10:33:33.093 DEBUG:boto:Path: /vclfcn-23/obj?versionId=FL110tVslNs52SVUbHZqPNK10BnpUdZ 2024-02-01T10:33:33.093 DEBUG:boto:Data: 2024-02-01T10:33:33.094 DEBUG:boto:Headers: {} 2024-02-01T10:33:33.094 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:33:33.094 DEBUG:boto:Port: 8001 2024-02-01T10:33:33.094 DEBUG:boto:Params: {} 2024-02-01T10:33:33.094 DEBUG:boto:Token: None 2024-02-01T10:33:33.094 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:33 GMT /vclfcn-23/obj?versionId=FL110tVslNs52SVUbHZqPNK10BnpUdZ 2024-02-01T10:33:33.094 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ilxHl1l+LlP/FTXVxsi8YevuLl8= 2024-02-01T10:33:33.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ilxHl1l+LlP/FTXVxsi8YevuLl8=', 'Content-Length': '0'} 2024-02-01T10:33:33.097 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:32:43 GMT'), ('x-amz-version-id', 'FL110tVslNs52SVUbHZqPNK10BnpUdZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002f0e2817589c50db-0065bb737d-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:33.097 DEBUG:boto:path=/vclfcn-23/obj 2024-02-01T10:33:33.098 DEBUG:boto:auth_path=/vclfcn-23/obj 2024-02-01T10:33:33.098 DEBUG:boto:path=/vclfcn-23/obj?versionId=FL110tVslNs52SVUbHZqPNK10BnpUdZ 2024-02-01T10:33:33.098 DEBUG:boto:auth_path=/vclfcn-23/obj?versionId=FL110tVslNs52SVUbHZqPNK10BnpUdZ 2024-02-01T10:33:33.098 DEBUG:boto:Method: GET 2024-02-01T10:33:33.098 DEBUG:boto:Path: /vclfcn-23/obj?versionId=FL110tVslNs52SVUbHZqPNK10BnpUdZ 2024-02-01T10:33:33.098 DEBUG:boto:Data: 2024-02-01T10:33:33.098 DEBUG:boto:Headers: {} 2024-02-01T10:33:33.098 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:33:33.098 DEBUG:boto:Port: 8000 2024-02-01T10:33:33.098 DEBUG:boto:Params: {} 2024-02-01T10:33:33.098 DEBUG:boto:Token: None 2024-02-01T10:33:33.099 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:33 GMT /vclfcn-23/obj?versionId=FL110tVslNs52SVUbHZqPNK10BnpUdZ 2024-02-01T10:33:33.099 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ilxHl1l+LlP/FTXVxsi8YevuLl8= 2024-02-01T10:33:33.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ilxHl1l+LlP/FTXVxsi8YevuLl8=', 'Content-Length': '0'} 2024-02-01T10:33:33.102 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:32:43 GMT'), ('x-amz-version-id', 'FL110tVslNs52SVUbHZqPNK10BnpUdZ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e808e804a1511ef1-0065bb737d-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:33.102 DEBUG:boto:path=/vclfcn-23/obj 2024-02-01T10:33:33.102 DEBUG:boto:auth_path=/vclfcn-23/obj 2024-02-01T10:33:33.102 DEBUG:boto:path=/vclfcn-23/obj?versionId=FL110tVslNs52SVUbHZqPNK10BnpUdZ 2024-02-01T10:33:33.102 DEBUG:boto:auth_path=/vclfcn-23/obj?versionId=FL110tVslNs52SVUbHZqPNK10BnpUdZ 2024-02-01T10:33:33.102 DEBUG:boto:Method: HEAD 2024-02-01T10:33:33.102 DEBUG:boto:Path: /vclfcn-23/obj?versionId=FL110tVslNs52SVUbHZqPNK10BnpUdZ 2024-02-01T10:33:33.102 DEBUG:boto:Data: 2024-02-01T10:33:33.102 DEBUG:boto:Headers: {} 2024-02-01T10:33:33.103 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:33:33.103 DEBUG:boto:Port: 8001 2024-02-01T10:33:33.103 DEBUG:boto:Params: {} 2024-02-01T10:33:33.103 DEBUG:boto:Token: None 2024-02-01T10:33:33.103 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:33:33 GMT /vclfcn-23/obj?versionId=FL110tVslNs52SVUbHZqPNK10BnpUdZ 2024-02-01T10:33:33.103 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:y5qNTlfNlX/AEbsDRa+C64GsQ+4= 2024-02-01T10:33:33.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:y5qNTlfNlX/AEbsDRa+C64GsQ+4=', 'Content-Length': '0'} 2024-02-01T10:33:33.106 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:32:43 GMT'), ('x-amz-version-id', 'FL110tVslNs52SVUbHZqPNK10BnpUdZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007570cfe42cffc642-0065bb737d-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:33.106 DEBUG:boto:path=/vclfcn-23/obj 2024-02-01T10:33:33.106 DEBUG:boto:auth_path=/vclfcn-23/obj 2024-02-01T10:33:33.107 DEBUG:boto:path=/vclfcn-23/obj?versionId=FL110tVslNs52SVUbHZqPNK10BnpUdZ 2024-02-01T10:33:33.107 DEBUG:boto:auth_path=/vclfcn-23/obj?versionId=FL110tVslNs52SVUbHZqPNK10BnpUdZ 2024-02-01T10:33:33.107 DEBUG:boto:Method: HEAD 2024-02-01T10:33:33.107 DEBUG:boto:Path: /vclfcn-23/obj?versionId=FL110tVslNs52SVUbHZqPNK10BnpUdZ 2024-02-01T10:33:33.107 DEBUG:boto:Data: 2024-02-01T10:33:33.107 DEBUG:boto:Headers: {} 2024-02-01T10:33:33.107 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:33:33.107 DEBUG:boto:Port: 8000 2024-02-01T10:33:33.107 DEBUG:boto:Params: {} 2024-02-01T10:33:33.107 DEBUG:boto:Token: None 2024-02-01T10:33:33.108 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:33:33 GMT /vclfcn-23/obj?versionId=FL110tVslNs52SVUbHZqPNK10BnpUdZ 2024-02-01T10:33:33.108 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:y5qNTlfNlX/AEbsDRa+C64GsQ+4= 2024-02-01T10:33:33.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:y5qNTlfNlX/AEbsDRa+C64GsQ+4=', 'Content-Length': '0'} 2024-02-01T10:33:33.110 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:32:43 GMT'), ('x-amz-version-id', 'FL110tVslNs52SVUbHZqPNK10BnpUdZ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007d4fa6177b05d212-0065bb737d-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:33.111 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:33:33.111 DEBUG:boto:path=/vclfcn-23/obj 2024-02-01T10:33:33.111 DEBUG:boto:auth_path=/vclfcn-23/obj 2024-02-01T10:33:33.111 DEBUG:boto:path=/vclfcn-23/obj?versionId=FL110tVslNs52SVUbHZqPNK10BnpUdZ 2024-02-01T10:33:33.111 DEBUG:boto:auth_path=/vclfcn-23/obj?versionId=FL110tVslNs52SVUbHZqPNK10BnpUdZ 2024-02-01T10:33:33.111 DEBUG:boto:Method: GET 2024-02-01T10:33:33.111 DEBUG:boto:Path: /vclfcn-23/obj?versionId=FL110tVslNs52SVUbHZqPNK10BnpUdZ 2024-02-01T10:33:33.111 DEBUG:boto:Data: 2024-02-01T10:33:33.111 DEBUG:boto:Headers: {} 2024-02-01T10:33:33.112 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:33:33.112 DEBUG:boto:Port: 8001 2024-02-01T10:33:33.112 DEBUG:boto:Params: {} 2024-02-01T10:33:33.112 DEBUG:boto:Token: None 2024-02-01T10:33:33.112 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:33 GMT /vclfcn-23/obj?versionId=FL110tVslNs52SVUbHZqPNK10BnpUdZ 2024-02-01T10:33:33.113 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ilxHl1l+LlP/FTXVxsi8YevuLl8= 2024-02-01T10:33:33.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ilxHl1l+LlP/FTXVxsi8YevuLl8=', 'Content-Length': '0'} 2024-02-01T10:33:33.115 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:32:43 GMT'), ('x-amz-version-id', 'FL110tVslNs52SVUbHZqPNK10BnpUdZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ebca04f488f2502b-0065bb737d-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:33.115 DEBUG:boto:path=/vclfcn-23/obj 2024-02-01T10:33:33.115 DEBUG:boto:auth_path=/vclfcn-23/obj 2024-02-01T10:33:33.116 DEBUG:boto:path=/vclfcn-23/obj?versionId=FL110tVslNs52SVUbHZqPNK10BnpUdZ 2024-02-01T10:33:33.116 DEBUG:boto:auth_path=/vclfcn-23/obj?versionId=FL110tVslNs52SVUbHZqPNK10BnpUdZ 2024-02-01T10:33:33.116 DEBUG:boto:Method: GET 2024-02-01T10:33:33.116 DEBUG:boto:Path: /vclfcn-23/obj?versionId=FL110tVslNs52SVUbHZqPNK10BnpUdZ 2024-02-01T10:33:33.116 DEBUG:boto:Data: 2024-02-01T10:33:33.116 DEBUG:boto:Headers: {} 2024-02-01T10:33:33.116 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:33:33.116 DEBUG:boto:Port: 8000 2024-02-01T10:33:33.116 DEBUG:boto:Params: {} 2024-02-01T10:33:33.116 DEBUG:boto:Token: None 2024-02-01T10:33:33.117 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:33 GMT /vclfcn-23/obj?versionId=FL110tVslNs52SVUbHZqPNK10BnpUdZ 2024-02-01T10:33:33.117 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ilxHl1l+LlP/FTXVxsi8YevuLl8= 2024-02-01T10:33:33.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ilxHl1l+LlP/FTXVxsi8YevuLl8=', 'Content-Length': '0'} 2024-02-01T10:33:33.119 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:32:43 GMT'), ('x-amz-version-id', 'FL110tVslNs52SVUbHZqPNK10BnpUdZ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003bb665f598e71ba8-0065bb737d-4351-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:33.120 DEBUG:boto:path=/vclfcn-23/ 2024-02-01T10:33:33.120 DEBUG:boto:auth_path=/vclfcn-23/ 2024-02-01T10:33:33.120 DEBUG:boto:path=/vclfcn-23/?prefix=obj 2024-02-01T10:33:33.120 DEBUG:boto:auth_path=/vclfcn-23/?prefix=obj 2024-02-01T10:33:33.120 DEBUG:boto:Method: GET 2024-02-01T10:33:33.120 DEBUG:boto:Path: /vclfcn-23/?prefix=obj 2024-02-01T10:33:33.120 DEBUG:boto:Data: 2024-02-01T10:33:33.120 DEBUG:boto:Headers: {} 2024-02-01T10:33:33.121 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:33:33.121 DEBUG:boto:Port: 8001 2024-02-01T10:33:33.121 DEBUG:boto:Params: {} 2024-02-01T10:33:33.121 DEBUG:boto:Token: None 2024-02-01T10:33:33.121 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:33 GMT /vclfcn-23/ 2024-02-01T10:33:33.121 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RsRRNS/95ZoZdfc/GxHV0/ZbOX8= 2024-02-01T10:33:33.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RsRRNS/95ZoZdfc/GxHV0/ZbOX8=', 'Content-Length': '0'} 2024-02-01T10:33:33.125 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095a47d3feda65989-0065bb737d-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:33.168 DEBUG:boto:b'vclfcn-23obj1000false' 2024-02-01T10:33:33.169 DEBUG:boto:path=/vclfcn-23/ 2024-02-01T10:33:33.169 DEBUG:boto:auth_path=/vclfcn-23/ 2024-02-01T10:33:33.169 DEBUG:boto:path=/vclfcn-23/?prefix=obj 2024-02-01T10:33:33.169 DEBUG:boto:auth_path=/vclfcn-23/?prefix=obj 2024-02-01T10:33:33.169 DEBUG:boto:Method: GET 2024-02-01T10:33:33.169 DEBUG:boto:Path: /vclfcn-23/?prefix=obj 2024-02-01T10:33:33.169 DEBUG:boto:Data: 2024-02-01T10:33:33.169 DEBUG:boto:Headers: {} 2024-02-01T10:33:33.169 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:33:33.169 DEBUG:boto:Port: 8000 2024-02-01T10:33:33.169 DEBUG:boto:Params: {} 2024-02-01T10:33:33.170 DEBUG:boto:Token: None 2024-02-01T10:33:33.170 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:33 GMT /vclfcn-23/ 2024-02-01T10:33:33.170 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RsRRNS/95ZoZdfc/GxHV0/ZbOX8= 2024-02-01T10:33:33.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RsRRNS/95ZoZdfc/GxHV0/ZbOX8=', 'Content-Length': '0'} 2024-02-01T10:33:33.175 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000add454e5e6d4fbfa-0065bb737d-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:33.175 DEBUG:boto:b'vclfcn-23obj1000false' 2024-02-01T10:33:33.175 DEBUG:boto:path=/vclfcn-23/obj 2024-02-01T10:33:33.175 DEBUG:boto:auth_path=/vclfcn-23/obj 2024-02-01T10:33:33.176 DEBUG:boto:Method: HEAD 2024-02-01T10:33:33.176 DEBUG:boto:Path: /vclfcn-23/obj 2024-02-01T10:33:33.176 DEBUG:boto:Data: 2024-02-01T10:33:33.176 DEBUG:boto:Headers: {} 2024-02-01T10:33:33.176 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:33:33.176 DEBUG:boto:Port: 8001 2024-02-01T10:33:33.176 DEBUG:boto:Params: {} 2024-02-01T10:33:33.176 DEBUG:boto:Token: None 2024-02-01T10:33:33.176 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:33:33 GMT /vclfcn-23/obj 2024-02-01T10:33:33.177 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:qFSRYVpVpYh1oINPFs7EMzTdO74= 2024-02-01T10:33:33.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:qFSRYVpVpYh1oINPFs7EMzTdO74=', 'Content-Length': '0'} 2024-02-01T10:33:33.180 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000e187e051b9144587-0065bb737d-4358-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:33.180 DEBUG:boto:path=/vclfcn-23/obj 2024-02-01T10:33:33.180 DEBUG:boto:auth_path=/vclfcn-23/obj 2024-02-01T10:33:33.180 DEBUG:boto:Method: HEAD 2024-02-01T10:33:33.180 DEBUG:boto:Path: /vclfcn-23/obj 2024-02-01T10:33:33.180 DEBUG:boto:Data: 2024-02-01T10:33:33.180 DEBUG:boto:Headers: {} 2024-02-01T10:33:33.180 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:33:33.181 DEBUG:boto:Port: 8000 2024-02-01T10:33:33.181 DEBUG:boto:Params: {} 2024-02-01T10:33:33.181 DEBUG:boto:Token: None 2024-02-01T10:33:33.181 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:33:33 GMT /vclfcn-23/obj 2024-02-01T10:33:33.181 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:qFSRYVpVpYh1oINPFs7EMzTdO74= 2024-02-01T10:33:33.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:qFSRYVpVpYh1oINPFs7EMzTdO74=', 'Content-Length': '0'} 2024-02-01T10:33:33.184 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000001ef8dcc3cce87e13-0065bb737d-4351-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:33.185 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-23 zones={test-zone3, test-zone1} 2024-02-01T10:33:33.185 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-23 zones={test-zone3, test-zone2} 2024-02-01T10:33:33.185 DEBUG:boto:path=/vclfcn-23/ 2024-02-01T10:33:33.185 DEBUG:boto:auth_path=/vclfcn-23/ 2024-02-01T10:33:33.185 DEBUG:boto:Method: HEAD 2024-02-01T10:33:33.185 DEBUG:boto:Path: /vclfcn-23/ 2024-02-01T10:33:33.186 DEBUG:boto:Data: 2024-02-01T10:33:33.186 DEBUG:boto:Headers: {} 2024-02-01T10:33:33.186 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:33:33.186 DEBUG:boto:Port: 8001 2024-02-01T10:33:33.186 DEBUG:boto:Params: {} 2024-02-01T10:33:33.186 DEBUG:boto:Token: None 2024-02-01T10:33:33.186 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:33:33 GMT /vclfcn-23/ 2024-02-01T10:33:33.186 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:l6ZeWW45N/XduizpjyUL0CmoOaA= 2024-02-01T10:33:33.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:l6ZeWW45N/XduizpjyUL0CmoOaA=', 'Content-Length': '0'} 2024-02-01T10:33:33.191 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000026d30b5737cf7744-0065bb737d-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:33:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:33.191 DEBUG:boto:path=/vclfcn-23/ 2024-02-01T10:33:33.191 DEBUG:boto:auth_path=/vclfcn-23/ 2024-02-01T10:33:33.191 DEBUG:boto:Method: HEAD 2024-02-01T10:33:33.191 DEBUG:boto:Path: /vclfcn-23/ 2024-02-01T10:33:33.191 DEBUG:boto:Data: 2024-02-01T10:33:33.191 DEBUG:boto:Headers: {} 2024-02-01T10:33:33.191 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:33:33.191 DEBUG:boto:Port: 8000 2024-02-01T10:33:33.192 DEBUG:boto:Params: {} 2024-02-01T10:33:33.192 DEBUG:boto:Token: None 2024-02-01T10:33:33.192 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:33:33 GMT /vclfcn-23/ 2024-02-01T10:33:33.192 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:l6ZeWW45N/XduizpjyUL0CmoOaA= 2024-02-01T10:33:33.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:l6ZeWW45N/XduizpjyUL0CmoOaA=', 'Content-Length': '0'} 2024-02-01T10:33:33.196 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000051527b3cb6e9b3f-0065bb737d-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:33:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:33.196 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:33:33.196 DEBUG:boto:path=/vclfcn-23/ 2024-02-01T10:33:33.196 DEBUG:boto:auth_path=/vclfcn-23/ 2024-02-01T10:33:33.197 DEBUG:boto:path=/vclfcn-23/?versions&max-keys=999 2024-02-01T10:33:33.197 DEBUG:boto:auth_path=/vclfcn-23/?versions&max-keys=999 2024-02-01T10:33:33.197 DEBUG:boto:Method: GET 2024-02-01T10:33:33.197 DEBUG:boto:Path: /vclfcn-23/?versions&max-keys=999 2024-02-01T10:33:33.197 DEBUG:boto:Data: 2024-02-01T10:33:33.197 DEBUG:boto:Headers: {} 2024-02-01T10:33:33.197 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:33:33.197 DEBUG:boto:Port: 8001 2024-02-01T10:33:33.197 DEBUG:boto:Params: {} 2024-02-01T10:33:33.197 DEBUG:boto:Token: None 2024-02-01T10:33:33.198 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:33 GMT /vclfcn-23/?versions 2024-02-01T10:33:33.198 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:m1YLTB8Jwrtit7aJ/IKpGNWnauM= 2024-02-01T10:33:33.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:m1YLTB8Jwrtit7aJ/IKpGNWnauM=', 'Content-Length': '0'} 2024-02-01T10:33:33.202 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000450ca96ff05e97cc-0065bb737d-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:33.202 DEBUG:boto:b'vclfcn-23999falseobjcfAEDiw8G8l6M7HFA62MzKKoAdV-ERAtrue2024-02-01T10:32:43.786Zrgw-multisite-test-userMultisite Test UserNormalobjFL110tVslNs52SVUbHZqPNK10BnpUdZfalse2024-02-01T10:32:43.732Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:33:33.203 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:33:33.203 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:33:33.203 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:33:33.203 DEBUG:boto:path=/vclfcn-23/ 2024-02-01T10:33:33.203 DEBUG:boto:auth_path=/vclfcn-23/ 2024-02-01T10:33:33.203 DEBUG:boto:path=/vclfcn-23/?versions&max-keys=999 2024-02-01T10:33:33.203 DEBUG:boto:auth_path=/vclfcn-23/?versions&max-keys=999 2024-02-01T10:33:33.203 DEBUG:boto:Method: GET 2024-02-01T10:33:33.203 DEBUG:boto:Path: /vclfcn-23/?versions&max-keys=999 2024-02-01T10:33:33.204 DEBUG:boto:Data: 2024-02-01T10:33:33.204 DEBUG:boto:Headers: {} 2024-02-01T10:33:33.204 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:33:33.204 DEBUG:boto:Port: 8000 2024-02-01T10:33:33.204 DEBUG:boto:Params: {} 2024-02-01T10:33:33.204 DEBUG:boto:Token: None 2024-02-01T10:33:33.204 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:33 GMT /vclfcn-23/?versions 2024-02-01T10:33:33.204 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:m1YLTB8Jwrtit7aJ/IKpGNWnauM= 2024-02-01T10:33:33.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:m1YLTB8Jwrtit7aJ/IKpGNWnauM=', 'Content-Length': '0'} 2024-02-01T10:33:33.209 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ce405ec88f321bd-0065bb737d-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:33.256 DEBUG:boto:b'vclfcn-23999falseobjcfAEDiw8G8l6M7HFA62MzKKoAdV-ERAtrue2024-02-01T10:32:43.786Zrgw-multisite-test-userMultisite Test UserNormalobjFL110tVslNs52SVUbHZqPNK10BnpUdZfalse2024-02-01T10:32:43.732Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:33:33.257 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:33:33.257 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:33:33.257 DEBUG:boto:path=/vclfcn-23/ 2024-02-01T10:33:33.257 DEBUG:boto:auth_path=/vclfcn-23/ 2024-02-01T10:33:33.257 DEBUG:boto:path=/vclfcn-23/?versions&max-keys=999 2024-02-01T10:33:33.257 DEBUG:boto:auth_path=/vclfcn-23/?versions&max-keys=999 2024-02-01T10:33:33.257 DEBUG:boto:Method: GET 2024-02-01T10:33:33.257 DEBUG:boto:Path: /vclfcn-23/?versions&max-keys=999 2024-02-01T10:33:33.257 DEBUG:boto:Data: 2024-02-01T10:33:33.258 DEBUG:boto:Headers: {} 2024-02-01T10:33:33.258 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:33:33.258 DEBUG:boto:Port: 8001 2024-02-01T10:33:33.258 DEBUG:boto:Params: {} 2024-02-01T10:33:33.258 DEBUG:boto:Token: None 2024-02-01T10:33:33.258 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:33 GMT /vclfcn-23/?versions 2024-02-01T10:33:33.259 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:m1YLTB8Jwrtit7aJ/IKpGNWnauM= 2024-02-01T10:33:33.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:m1YLTB8Jwrtit7aJ/IKpGNWnauM=', 'Content-Length': '0'} 2024-02-01T10:33:33.264 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040b84ced04a4cc7d-0065bb737d-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:33.308 DEBUG:boto:b'vclfcn-23999falseobjcfAEDiw8G8l6M7HFA62MzKKoAdV-ERAtrue2024-02-01T10:32:43.786Zrgw-multisite-test-userMultisite Test UserNormalobjFL110tVslNs52SVUbHZqPNK10BnpUdZfalse2024-02-01T10:32:43.732Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:33:33.309 DEBUG:boto:path=/vclfcn-23/ 2024-02-01T10:33:33.309 DEBUG:boto:auth_path=/vclfcn-23/ 2024-02-01T10:33:33.309 DEBUG:boto:path=/vclfcn-23/?versions&max-keys=999 2024-02-01T10:33:33.309 DEBUG:boto:auth_path=/vclfcn-23/?versions&max-keys=999 2024-02-01T10:33:33.309 DEBUG:boto:Method: GET 2024-02-01T10:33:33.309 DEBUG:boto:Path: /vclfcn-23/?versions&max-keys=999 2024-02-01T10:33:33.309 DEBUG:boto:Data: 2024-02-01T10:33:33.309 DEBUG:boto:Headers: {} 2024-02-01T10:33:33.309 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:33:33.309 DEBUG:boto:Port: 8000 2024-02-01T10:33:33.309 DEBUG:boto:Params: {} 2024-02-01T10:33:33.310 DEBUG:boto:Token: None 2024-02-01T10:33:33.310 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:33 GMT /vclfcn-23/?versions 2024-02-01T10:33:33.310 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:m1YLTB8Jwrtit7aJ/IKpGNWnauM= 2024-02-01T10:33:33.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:m1YLTB8Jwrtit7aJ/IKpGNWnauM=', 'Content-Length': '0'} 2024-02-01T10:33:33.315 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b5a77777b48db835-0065bb737d-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:33.356 DEBUG:boto:b'vclfcn-23999falseobjcfAEDiw8G8l6M7HFA62MzKKoAdV-ERAtrue2024-02-01T10:32:43.786Zrgw-multisite-test-userMultisite Test UserNormalobjFL110tVslNs52SVUbHZqPNK10BnpUdZfalse2024-02-01T10:32:43.732Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:33:33.357 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:33:33.357 DEBUG:boto:path=/vclfcn-23/obj 2024-02-01T10:33:33.357 DEBUG:boto:auth_path=/vclfcn-23/obj 2024-02-01T10:33:33.357 DEBUG:boto:Method: HEAD 2024-02-01T10:33:33.357 DEBUG:boto:Path: /vclfcn-23/obj 2024-02-01T10:33:33.357 DEBUG:boto:Data: 2024-02-01T10:33:33.357 DEBUG:boto:Headers: {} 2024-02-01T10:33:33.358 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:33:33.358 DEBUG:boto:Port: 8001 2024-02-01T10:33:33.358 DEBUG:boto:Params: {} 2024-02-01T10:33:33.358 DEBUG:boto:Token: None 2024-02-01T10:33:33.358 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:33:33 GMT /vclfcn-23/obj 2024-02-01T10:33:33.358 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:qFSRYVpVpYh1oINPFs7EMzTdO74= 2024-02-01T10:33:33.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:qFSRYVpVpYh1oINPFs7EMzTdO74=', 'Content-Length': '0'} 2024-02-01T10:33:33.362 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000af9937af29cb66e4-0065bb737d-4358-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:33.362 DEBUG:boto:path=/vclfcn-23/obj 2024-02-01T10:33:33.362 DEBUG:boto:auth_path=/vclfcn-23/obj 2024-02-01T10:33:33.362 DEBUG:boto:Method: HEAD 2024-02-01T10:33:33.362 DEBUG:boto:Path: /vclfcn-23/obj 2024-02-01T10:33:33.362 DEBUG:boto:Data: 2024-02-01T10:33:33.363 DEBUG:boto:Headers: {} 2024-02-01T10:33:33.363 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:33:33.363 DEBUG:boto:Port: 8000 2024-02-01T10:33:33.363 DEBUG:boto:Params: {} 2024-02-01T10:33:33.363 DEBUG:boto:Token: None 2024-02-01T10:33:33.363 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:33:33 GMT /vclfcn-23/obj 2024-02-01T10:33:33.363 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:qFSRYVpVpYh1oINPFs7EMzTdO74= 2024-02-01T10:33:33.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:qFSRYVpVpYh1oINPFs7EMzTdO74=', 'Content-Length': '0'} 2024-02-01T10:33:33.367 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000f4c7bf47f8a2f007-0065bb737d-4307-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:33.367 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:33:33.367 DEBUG:boto:path=/vclfcn-23/obj 2024-02-01T10:33:33.367 DEBUG:boto:auth_path=/vclfcn-23/obj 2024-02-01T10:33:33.368 DEBUG:boto:path=/vclfcn-23/obj?versionId=FL110tVslNs52SVUbHZqPNK10BnpUdZ 2024-02-01T10:33:33.368 DEBUG:boto:auth_path=/vclfcn-23/obj?versionId=FL110tVslNs52SVUbHZqPNK10BnpUdZ 2024-02-01T10:33:33.368 DEBUG:boto:Method: GET 2024-02-01T10:33:33.368 DEBUG:boto:Path: /vclfcn-23/obj?versionId=FL110tVslNs52SVUbHZqPNK10BnpUdZ 2024-02-01T10:33:33.368 DEBUG:boto:Data: 2024-02-01T10:33:33.368 DEBUG:boto:Headers: {} 2024-02-01T10:33:33.368 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:33:33.368 DEBUG:boto:Port: 8001 2024-02-01T10:33:33.368 DEBUG:boto:Params: {} 2024-02-01T10:33:33.368 DEBUG:boto:Token: None 2024-02-01T10:33:33.369 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:33 GMT /vclfcn-23/obj?versionId=FL110tVslNs52SVUbHZqPNK10BnpUdZ 2024-02-01T10:33:33.369 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ilxHl1l+LlP/FTXVxsi8YevuLl8= 2024-02-01T10:33:33.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ilxHl1l+LlP/FTXVxsi8YevuLl8=', 'Content-Length': '0'} 2024-02-01T10:33:33.371 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:32:43 GMT'), ('x-amz-version-id', 'FL110tVslNs52SVUbHZqPNK10BnpUdZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000017a647a7ecbc0878-0065bb737d-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:33.372 DEBUG:boto:path=/vclfcn-23/obj 2024-02-01T10:33:33.372 DEBUG:boto:auth_path=/vclfcn-23/obj 2024-02-01T10:33:33.372 DEBUG:boto:path=/vclfcn-23/obj?versionId=FL110tVslNs52SVUbHZqPNK10BnpUdZ 2024-02-01T10:33:33.372 DEBUG:boto:auth_path=/vclfcn-23/obj?versionId=FL110tVslNs52SVUbHZqPNK10BnpUdZ 2024-02-01T10:33:33.372 DEBUG:boto:Method: GET 2024-02-01T10:33:33.372 DEBUG:boto:Path: /vclfcn-23/obj?versionId=FL110tVslNs52SVUbHZqPNK10BnpUdZ 2024-02-01T10:33:33.372 DEBUG:boto:Data: 2024-02-01T10:33:33.372 DEBUG:boto:Headers: {} 2024-02-01T10:33:33.372 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:33:33.373 DEBUG:boto:Port: 8000 2024-02-01T10:33:33.373 DEBUG:boto:Params: {} 2024-02-01T10:33:33.373 DEBUG:boto:Token: None 2024-02-01T10:33:33.373 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:33 GMT /vclfcn-23/obj?versionId=FL110tVslNs52SVUbHZqPNK10BnpUdZ 2024-02-01T10:33:33.373 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ilxHl1l+LlP/FTXVxsi8YevuLl8= 2024-02-01T10:33:33.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ilxHl1l+LlP/FTXVxsi8YevuLl8=', 'Content-Length': '0'} 2024-02-01T10:33:33.376 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:32:43 GMT'), ('x-amz-version-id', 'FL110tVslNs52SVUbHZqPNK10BnpUdZ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ee6d5e0d62c1df1a-0065bb737d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:33.376 DEBUG:boto:path=/vclfcn-23/obj 2024-02-01T10:33:33.376 DEBUG:boto:auth_path=/vclfcn-23/obj 2024-02-01T10:33:33.376 DEBUG:boto:path=/vclfcn-23/obj?versionId=FL110tVslNs52SVUbHZqPNK10BnpUdZ 2024-02-01T10:33:33.376 DEBUG:boto:auth_path=/vclfcn-23/obj?versionId=FL110tVslNs52SVUbHZqPNK10BnpUdZ 2024-02-01T10:33:33.376 DEBUG:boto:Method: HEAD 2024-02-01T10:33:33.377 DEBUG:boto:Path: /vclfcn-23/obj?versionId=FL110tVslNs52SVUbHZqPNK10BnpUdZ 2024-02-01T10:33:33.377 DEBUG:boto:Data: 2024-02-01T10:33:33.377 DEBUG:boto:Headers: {} 2024-02-01T10:33:33.377 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:33:33.377 DEBUG:boto:Port: 8001 2024-02-01T10:33:33.377 DEBUG:boto:Params: {} 2024-02-01T10:33:33.377 DEBUG:boto:Token: None 2024-02-01T10:33:33.377 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:33:33 GMT /vclfcn-23/obj?versionId=FL110tVslNs52SVUbHZqPNK10BnpUdZ 2024-02-01T10:33:33.377 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:y5qNTlfNlX/AEbsDRa+C64GsQ+4= 2024-02-01T10:33:33.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:y5qNTlfNlX/AEbsDRa+C64GsQ+4=', 'Content-Length': '0'} 2024-02-01T10:33:33.381 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:32:43 GMT'), ('x-amz-version-id', 'FL110tVslNs52SVUbHZqPNK10BnpUdZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002c85c1bd50ceaa3e-0065bb737d-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:33.381 DEBUG:boto:path=/vclfcn-23/obj 2024-02-01T10:33:33.381 DEBUG:boto:auth_path=/vclfcn-23/obj 2024-02-01T10:33:33.381 DEBUG:boto:path=/vclfcn-23/obj?versionId=FL110tVslNs52SVUbHZqPNK10BnpUdZ 2024-02-01T10:33:33.382 DEBUG:boto:auth_path=/vclfcn-23/obj?versionId=FL110tVslNs52SVUbHZqPNK10BnpUdZ 2024-02-01T10:33:33.382 DEBUG:boto:Method: HEAD 2024-02-01T10:33:33.382 DEBUG:boto:Path: /vclfcn-23/obj?versionId=FL110tVslNs52SVUbHZqPNK10BnpUdZ 2024-02-01T10:33:33.382 DEBUG:boto:Data: 2024-02-01T10:33:33.382 DEBUG:boto:Headers: {} 2024-02-01T10:33:33.382 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:33:33.382 DEBUG:boto:Port: 8000 2024-02-01T10:33:33.382 DEBUG:boto:Params: {} 2024-02-01T10:33:33.382 DEBUG:boto:Token: None 2024-02-01T10:33:33.382 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:33:33 GMT /vclfcn-23/obj?versionId=FL110tVslNs52SVUbHZqPNK10BnpUdZ 2024-02-01T10:33:33.383 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:y5qNTlfNlX/AEbsDRa+C64GsQ+4= 2024-02-01T10:33:33.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:y5qNTlfNlX/AEbsDRa+C64GsQ+4=', 'Content-Length': '0'} 2024-02-01T10:33:33.386 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:32:43 GMT'), ('x-amz-version-id', 'FL110tVslNs52SVUbHZqPNK10BnpUdZ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002d196f3d8d4751d5-0065bb737d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:33.386 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:33:33.386 DEBUG:boto:path=/vclfcn-23/obj 2024-02-01T10:33:33.386 DEBUG:boto:auth_path=/vclfcn-23/obj 2024-02-01T10:33:33.386 DEBUG:boto:path=/vclfcn-23/obj?versionId=FL110tVslNs52SVUbHZqPNK10BnpUdZ 2024-02-01T10:33:33.386 DEBUG:boto:auth_path=/vclfcn-23/obj?versionId=FL110tVslNs52SVUbHZqPNK10BnpUdZ 2024-02-01T10:33:33.386 DEBUG:boto:Method: GET 2024-02-01T10:33:33.387 DEBUG:boto:Path: /vclfcn-23/obj?versionId=FL110tVslNs52SVUbHZqPNK10BnpUdZ 2024-02-01T10:33:33.387 DEBUG:boto:Data: 2024-02-01T10:33:33.387 DEBUG:boto:Headers: {} 2024-02-01T10:33:33.387 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:33:33.387 DEBUG:boto:Port: 8001 2024-02-01T10:33:33.387 DEBUG:boto:Params: {} 2024-02-01T10:33:33.387 DEBUG:boto:Token: None 2024-02-01T10:33:33.387 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:33 GMT /vclfcn-23/obj?versionId=FL110tVslNs52SVUbHZqPNK10BnpUdZ 2024-02-01T10:33:33.387 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ilxHl1l+LlP/FTXVxsi8YevuLl8= 2024-02-01T10:33:33.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ilxHl1l+LlP/FTXVxsi8YevuLl8=', 'Content-Length': '0'} 2024-02-01T10:33:33.390 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:32:43 GMT'), ('x-amz-version-id', 'FL110tVslNs52SVUbHZqPNK10BnpUdZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a1bc009ef4abfc66-0065bb737d-4358-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:33.390 DEBUG:boto:path=/vclfcn-23/obj 2024-02-01T10:33:33.390 DEBUG:boto:auth_path=/vclfcn-23/obj 2024-02-01T10:33:33.390 DEBUG:boto:path=/vclfcn-23/obj?versionId=FL110tVslNs52SVUbHZqPNK10BnpUdZ 2024-02-01T10:33:33.390 DEBUG:boto:auth_path=/vclfcn-23/obj?versionId=FL110tVslNs52SVUbHZqPNK10BnpUdZ 2024-02-01T10:33:33.390 DEBUG:boto:Method: GET 2024-02-01T10:33:33.391 DEBUG:boto:Path: /vclfcn-23/obj?versionId=FL110tVslNs52SVUbHZqPNK10BnpUdZ 2024-02-01T10:33:33.391 DEBUG:boto:Data: 2024-02-01T10:33:33.391 DEBUG:boto:Headers: {} 2024-02-01T10:33:33.391 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:33:33.391 DEBUG:boto:Port: 8000 2024-02-01T10:33:33.391 DEBUG:boto:Params: {} 2024-02-01T10:33:33.391 DEBUG:boto:Token: None 2024-02-01T10:33:33.391 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:33 GMT /vclfcn-23/obj?versionId=FL110tVslNs52SVUbHZqPNK10BnpUdZ 2024-02-01T10:33:33.391 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ilxHl1l+LlP/FTXVxsi8YevuLl8= 2024-02-01T10:33:33.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ilxHl1l+LlP/FTXVxsi8YevuLl8=', 'Content-Length': '0'} 2024-02-01T10:33:33.394 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:32:43 GMT'), ('x-amz-version-id', 'FL110tVslNs52SVUbHZqPNK10BnpUdZ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b79859c987feda41-0065bb737d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:33.394 DEBUG:boto:path=/vclfcn-23/ 2024-02-01T10:33:33.394 DEBUG:boto:auth_path=/vclfcn-23/ 2024-02-01T10:33:33.395 DEBUG:boto:path=/vclfcn-23/?prefix=obj 2024-02-01T10:33:33.395 DEBUG:boto:auth_path=/vclfcn-23/?prefix=obj 2024-02-01T10:33:33.395 DEBUG:boto:Method: GET 2024-02-01T10:33:33.395 DEBUG:boto:Path: /vclfcn-23/?prefix=obj 2024-02-01T10:33:33.395 DEBUG:boto:Data: 2024-02-01T10:33:33.395 DEBUG:boto:Headers: {} 2024-02-01T10:33:33.395 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:33:33.395 DEBUG:boto:Port: 8001 2024-02-01T10:33:33.395 DEBUG:boto:Params: {} 2024-02-01T10:33:33.395 DEBUG:boto:Token: None 2024-02-01T10:33:33.395 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:33 GMT /vclfcn-23/ 2024-02-01T10:33:33.395 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RsRRNS/95ZoZdfc/GxHV0/ZbOX8= 2024-02-01T10:33:33.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RsRRNS/95ZoZdfc/GxHV0/ZbOX8=', 'Content-Length': '0'} 2024-02-01T10:33:33.400 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000713230b9e773001e-0065bb737d-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:33.444 DEBUG:boto:b'vclfcn-23obj1000false' 2024-02-01T10:33:33.444 DEBUG:boto:path=/vclfcn-23/ 2024-02-01T10:33:33.445 DEBUG:boto:auth_path=/vclfcn-23/ 2024-02-01T10:33:33.445 DEBUG:boto:path=/vclfcn-23/?prefix=obj 2024-02-01T10:33:33.445 DEBUG:boto:auth_path=/vclfcn-23/?prefix=obj 2024-02-01T10:33:33.445 DEBUG:boto:Method: GET 2024-02-01T10:33:33.445 DEBUG:boto:Path: /vclfcn-23/?prefix=obj 2024-02-01T10:33:33.445 DEBUG:boto:Data: 2024-02-01T10:33:33.445 DEBUG:boto:Headers: {} 2024-02-01T10:33:33.445 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:33:33.445 DEBUG:boto:Port: 8000 2024-02-01T10:33:33.445 DEBUG:boto:Params: {} 2024-02-01T10:33:33.446 DEBUG:boto:Token: None 2024-02-01T10:33:33.446 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:33:33 GMT /vclfcn-23/ 2024-02-01T10:33:33.446 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RsRRNS/95ZoZdfc/GxHV0/ZbOX8= 2024-02-01T10:33:33.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RsRRNS/95ZoZdfc/GxHV0/ZbOX8=', 'Content-Length': '0'} 2024-02-01T10:33:33.451 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e8beb7cddf334c6b-0065bb737d-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:33.451 DEBUG:boto:b'vclfcn-23obj1000false' 2024-02-01T10:33:33.451 DEBUG:boto:path=/vclfcn-23/obj 2024-02-01T10:33:33.452 DEBUG:boto:auth_path=/vclfcn-23/obj 2024-02-01T10:33:33.452 DEBUG:boto:Method: HEAD 2024-02-01T10:33:33.452 DEBUG:boto:Path: /vclfcn-23/obj 2024-02-01T10:33:33.452 DEBUG:boto:Data: 2024-02-01T10:33:33.452 DEBUG:boto:Headers: {} 2024-02-01T10:33:33.452 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:33:33.452 DEBUG:boto:Port: 8001 2024-02-01T10:33:33.452 DEBUG:boto:Params: {} 2024-02-01T10:33:33.452 DEBUG:boto:Token: None 2024-02-01T10:33:33.452 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:33:33 GMT /vclfcn-23/obj 2024-02-01T10:33:33.453 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:qFSRYVpVpYh1oINPFs7EMzTdO74= 2024-02-01T10:33:33.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:qFSRYVpVpYh1oINPFs7EMzTdO74=', 'Content-Length': '0'} 2024-02-01T10:33:33.456 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000005e4fbc58106f226b-0065bb737d-4358-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:33.456 DEBUG:boto:path=/vclfcn-23/obj 2024-02-01T10:33:33.457 DEBUG:boto:auth_path=/vclfcn-23/obj 2024-02-01T10:33:33.457 DEBUG:boto:Method: HEAD 2024-02-01T10:33:33.457 DEBUG:boto:Path: /vclfcn-23/obj 2024-02-01T10:33:33.457 DEBUG:boto:Data: 2024-02-01T10:33:33.457 DEBUG:boto:Headers: {} 2024-02-01T10:33:33.457 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:33:33.457 DEBUG:boto:Port: 8000 2024-02-01T10:33:33.457 DEBUG:boto:Params: {} 2024-02-01T10:33:33.457 DEBUG:boto:Token: None 2024-02-01T10:33:33.457 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:33:33 GMT /vclfcn-23/obj 2024-02-01T10:33:33.458 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:qFSRYVpVpYh1oINPFs7EMzTdO74= 2024-02-01T10:33:33.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:qFSRYVpVpYh1oINPFs7EMzTdO74=', 'Content-Length': '0'} 2024-02-01T10:33:33.461 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000ff6928cb2449c572-0065bb737d-4307-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:33.461 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-23 zones={test-zone3, test-zone2} 2024-02-01T10:33:33.461 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_delete_marker_full_sync ... ok 2024-02-01T10:33:33.463 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=vclfcn-24 2024-02-01T10:33:33.463 DEBUG:boto:path=/vclfcn-24/ 2024-02-01T10:33:33.463 DEBUG:boto:auth_path=/vclfcn-24/ 2024-02-01T10:33:33.463 DEBUG:boto:Method: PUT 2024-02-01T10:33:33.463 DEBUG:boto:Path: /vclfcn-24/ 2024-02-01T10:33:33.463 DEBUG:boto:Data: 2024-02-01T10:33:33.463 DEBUG:boto:Headers: {} 2024-02-01T10:33:33.463 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:33:33.463 DEBUG:boto:Port: 8000 2024-02-01T10:33:33.463 DEBUG:boto:Params: {} 2024-02-01T10:33:33.464 DEBUG:boto:Token: None 2024-02-01T10:33:33.464 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:33:33 GMT /vclfcn-24/ 2024-02-01T10:33:33.464 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:emM0OS/tUIrO26J/YC9R8SNJkAo= 2024-02-01T10:33:33.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:emM0OS/tUIrO26J/YC9R8SNJkAo=', 'Content-Length': '0'} 2024-02-01T10:33:33.497 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000deefed7db7e59596-0065bb737d-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:33:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:33.497 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=vclfcn-25 2024-02-01T10:33:33.497 DEBUG:boto:path=/vclfcn-25/ 2024-02-01T10:33:33.498 DEBUG:boto:auth_path=/vclfcn-25/ 2024-02-01T10:33:33.498 DEBUG:boto:Method: PUT 2024-02-01T10:33:33.498 DEBUG:boto:Path: /vclfcn-25/ 2024-02-01T10:33:33.498 DEBUG:boto:Data: 2024-02-01T10:33:33.498 DEBUG:boto:Headers: {} 2024-02-01T10:33:33.498 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:33:33.498 DEBUG:boto:Port: 8000 2024-02-01T10:33:33.498 DEBUG:boto:Params: {} 2024-02-01T10:33:33.498 DEBUG:boto:Token: None 2024-02-01T10:33:33.498 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:33:33 GMT /vclfcn-25/ 2024-02-01T10:33:33.499 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:XiLo2+VBo99dGVyB9tY4lxDb0DA= 2024-02-01T10:33:33.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:XiLo2+VBo99dGVyB9tY4lxDb0DA=', 'Content-Length': '0'} 2024-02-01T10:33:33.557 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009c754c2b3fe4b31a-0065bb737d-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:33:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:33.557 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=vclfcn-26 2024-02-01T10:33:33.557 DEBUG:boto:path=/vclfcn-26/ 2024-02-01T10:33:33.557 DEBUG:boto:auth_path=/vclfcn-26/ 2024-02-01T10:33:33.557 DEBUG:boto:Method: PUT 2024-02-01T10:33:33.557 DEBUG:boto:Path: /vclfcn-26/ 2024-02-01T10:33:33.557 DEBUG:boto:Data: 2024-02-01T10:33:33.558 DEBUG:boto:Headers: {} 2024-02-01T10:33:33.558 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:33:33.558 DEBUG:boto:Port: 8001 2024-02-01T10:33:33.558 DEBUG:boto:Params: {} 2024-02-01T10:33:33.558 DEBUG:boto:Token: None 2024-02-01T10:33:33.558 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:33:33 GMT /vclfcn-26/ 2024-02-01T10:33:33.558 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vACZuMMWmpGkbTNUhUa6sjwrkms= 2024-02-01T10:33:33.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vACZuMMWmpGkbTNUhUa6sjwrkms=', 'Content-Length': '0'} 2024-02-01T10:33:33.620 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003310637825a3e05c-0065bb737d-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:33:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:33.620 DEBUG:boto:path=/vclfcn-24/ 2024-02-01T10:33:33.620 DEBUG:boto:auth_path=/vclfcn-24/ 2024-02-01T10:33:33.620 DEBUG:boto:path=/vclfcn-24/?versioning 2024-02-01T10:33:33.620 DEBUG:boto:auth_path=/vclfcn-24/?versioning 2024-02-01T10:33:33.620 DEBUG:boto:Method: PUT 2024-02-01T10:33:33.620 DEBUG:boto:Path: /vclfcn-24/?versioning 2024-02-01T10:33:33.621 DEBUG:boto:Data: Enabled Disabled 2024-02-01T10:33:33.621 DEBUG:boto:Headers: {} 2024-02-01T10:33:33.621 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:33:33.621 DEBUG:boto:Port: 8000 2024-02-01T10:33:33.621 DEBUG:boto:Params: {} 2024-02-01T10:33:33.621 DEBUG:boto:Token: None 2024-02-01T10:33:33.621 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:33:33 GMT /vclfcn-24/?versioning 2024-02-01T10:33:33.621 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:dkQIhAct+PYdX+BgO65gtvdfMwA= 2024-02-01T10:33:33.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dkQIhAct+PYdX+BgO65gtvdfMwA=', 'Content-Length': '228'} 2024-02-01T10:33:33.632 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009ef5bf1ab3d67d66-0065bb737d-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:33:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:33.633 DEBUG:boto:path=/vclfcn-24/ 2024-02-01T10:33:33.633 DEBUG:boto:auth_path=/vclfcn-24/ 2024-02-01T10:33:33.633 DEBUG:boto:path=/vclfcn-24/?versioning 2024-02-01T10:33:33.633 DEBUG:boto:auth_path=/vclfcn-24/?versioning 2024-02-01T10:33:33.633 DEBUG:boto:Method: PUT 2024-02-01T10:33:33.633 DEBUG:boto:Path: /vclfcn-24/?versioning 2024-02-01T10:33:33.633 DEBUG:boto:Data: Suspended Disabled 2024-02-01T10:33:33.633 DEBUG:boto:Headers: {} 2024-02-01T10:33:33.633 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:33:33.633 DEBUG:boto:Port: 8000 2024-02-01T10:33:33.633 DEBUG:boto:Params: {} 2024-02-01T10:33:33.634 DEBUG:boto:Token: None 2024-02-01T10:33:33.634 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:33:33 GMT /vclfcn-24/?versioning 2024-02-01T10:33:33.634 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:dkQIhAct+PYdX+BgO65gtvdfMwA= 2024-02-01T10:33:33.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dkQIhAct+PYdX+BgO65gtvdfMwA=', 'Content-Length': '230'} 2024-02-01T10:33:33.647 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008a543165919f9145-0065bb737d-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:33:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:33.647 DEBUG:boto:path=/vclfcn-25/ 2024-02-01T10:33:33.647 DEBUG:boto:auth_path=/vclfcn-25/ 2024-02-01T10:33:33.647 DEBUG:boto:path=/vclfcn-25/?versioning 2024-02-01T10:33:33.647 DEBUG:boto:auth_path=/vclfcn-25/?versioning 2024-02-01T10:33:33.647 DEBUG:boto:Method: PUT 2024-02-01T10:33:33.647 DEBUG:boto:Path: /vclfcn-25/?versioning 2024-02-01T10:33:33.647 DEBUG:boto:Data: Enabled Disabled 2024-02-01T10:33:33.647 DEBUG:boto:Headers: {} 2024-02-01T10:33:33.648 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:33:33.648 DEBUG:boto:Port: 8000 2024-02-01T10:33:33.648 DEBUG:boto:Params: {} 2024-02-01T10:33:33.648 DEBUG:boto:Token: None 2024-02-01T10:33:33.648 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:33:33 GMT /vclfcn-25/?versioning 2024-02-01T10:33:33.648 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GhjMPtUtRUItg2BRAyuYu2xugs4= 2024-02-01T10:33:33.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GhjMPtUtRUItg2BRAyuYu2xugs4=', 'Content-Length': '228'} 2024-02-01T10:33:33.683 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004354ec376b6db88e-0065bb737d-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:33:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:33.684 DEBUG:boto:path=/vclfcn-25/ 2024-02-01T10:33:33.684 DEBUG:boto:auth_path=/vclfcn-25/ 2024-02-01T10:33:33.684 DEBUG:boto:path=/vclfcn-25/?versioning 2024-02-01T10:33:33.684 DEBUG:boto:auth_path=/vclfcn-25/?versioning 2024-02-01T10:33:33.684 DEBUG:boto:Method: PUT 2024-02-01T10:33:33.684 DEBUG:boto:Path: /vclfcn-25/?versioning 2024-02-01T10:33:33.684 DEBUG:boto:Data: Suspended Disabled 2024-02-01T10:33:33.684 DEBUG:boto:Headers: {} 2024-02-01T10:33:33.684 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:33:33.685 DEBUG:boto:Port: 8000 2024-02-01T10:33:33.685 DEBUG:boto:Params: {} 2024-02-01T10:33:33.685 DEBUG:boto:Token: None 2024-02-01T10:33:33.685 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:33:33 GMT /vclfcn-25/?versioning 2024-02-01T10:33:33.685 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GhjMPtUtRUItg2BRAyuYu2xugs4= 2024-02-01T10:33:33.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GhjMPtUtRUItg2BRAyuYu2xugs4=', 'Content-Length': '230'} 2024-02-01T10:33:33.714 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007e7744f116c34c46-0065bb737d-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:33:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:33.714 DEBUG:boto:path=/vclfcn-26/ 2024-02-01T10:33:33.714 DEBUG:boto:auth_path=/vclfcn-26/ 2024-02-01T10:33:33.714 DEBUG:boto:path=/vclfcn-26/?versioning 2024-02-01T10:33:33.715 DEBUG:boto:auth_path=/vclfcn-26/?versioning 2024-02-01T10:33:33.715 DEBUG:boto:Method: PUT 2024-02-01T10:33:33.715 DEBUG:boto:Path: /vclfcn-26/?versioning 2024-02-01T10:33:33.715 DEBUG:boto:Data: Enabled Disabled 2024-02-01T10:33:33.715 DEBUG:boto:Headers: {} 2024-02-01T10:33:33.715 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:33:33.715 DEBUG:boto:Port: 8001 2024-02-01T10:33:33.715 DEBUG:boto:Params: {} 2024-02-01T10:33:33.715 DEBUG:boto:Token: None 2024-02-01T10:33:33.715 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:33:33 GMT /vclfcn-26/?versioning 2024-02-01T10:33:33.716 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8sGPEUHAMREkq08nR2myWRHbvR4= 2024-02-01T10:33:33.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8sGPEUHAMREkq08nR2myWRHbvR4=', 'Content-Length': '228'} 2024-02-01T10:33:33.734 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b111a24735260056-0065bb737d-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:33:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:33.734 DEBUG:boto:path=/vclfcn-26/ 2024-02-01T10:33:33.734 DEBUG:boto:auth_path=/vclfcn-26/ 2024-02-01T10:33:33.734 DEBUG:boto:path=/vclfcn-26/?versioning 2024-02-01T10:33:33.734 DEBUG:boto:auth_path=/vclfcn-26/?versioning 2024-02-01T10:33:33.734 DEBUG:boto:Method: PUT 2024-02-01T10:33:33.734 DEBUG:boto:Path: /vclfcn-26/?versioning 2024-02-01T10:33:33.735 DEBUG:boto:Data: Suspended Disabled 2024-02-01T10:33:33.735 DEBUG:boto:Headers: {} 2024-02-01T10:33:33.735 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:33:33.735 DEBUG:boto:Port: 8001 2024-02-01T10:33:33.735 DEBUG:boto:Params: {} 2024-02-01T10:33:33.735 DEBUG:boto:Token: None 2024-02-01T10:33:33.735 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:33:33 GMT /vclfcn-26/?versioning 2024-02-01T10:33:33.735 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8sGPEUHAMREkq08nR2myWRHbvR4= 2024-02-01T10:33:33.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8sGPEUHAMREkq08nR2myWRHbvR4=', 'Content-Length': '230'} 2024-02-01T10:33:33.752 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000073bd5ec25c3caaaf-0065bb737d-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:33:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:33.752 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:33:33.752 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:33:33.752 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:33:33.794 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:33:33.794 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:33:33.857 INFO:teuthology.orchestra.run.smithi148.stderr:No --period given, using current period=5dc4efbd-eccd-47a8-9471-ae4a209ec978 2024-02-01T10:33:33.881 INFO:teuthology.orchestra.run.smithi148.stdout:[{"marker":"1_1706782453.132619_30.1","last_update":"2024-02-01T10:14:13.132619Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1706783613.755304_167.1","last_update":"2024-02-01T10:33:33.755304Z"},{"marker":"1_1706782595.670952_121.1","last_update":"2024-02-01T10:16:35.670952Z"}] 2024-02-01T10:33:33.881 DEBUG:rgw_multi.tests:master meta markers={0: '1_1706782453.132619_30.1', 1: '', 2: '1_1706783613.755304_167.1', 3: '1_1706782595.670952_121.1'} 2024-02-01T10:33:33.881 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-02-01T10:33:33.881 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:33:33.881 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:33:33.882 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:33:33.928 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:33:33.928 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:33:33.985 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:33:33.981+0000 7fdbe90d1980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:33:33.986 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:33:33.985+0000 7fdbe90d1980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:33:34.009 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783613.755304_167.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:33:33.755304Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:33:33Z"} 2024-02-01T10:33:34.009 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783613.755304_167.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:33:33.755304Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:33:33Z"} 2024-02-01T10:33:34.010 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706783613.755304_167.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:33:33.755304Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:33:34.010 DEBUG:rgw_multi.tests:log_status={0: '1_1706782453.132619_30.1', 1: '', 2: '1_1706783613.755304_167.1', 3: '1_1706782595.670952_121.1'} 2024-02-01T10:33:34.010 DEBUG:rgw_multi.tests:sync_status={0: '1_1706782453.132619_30.1', 1: '', 2: '1_1706783613.755304_167.1', 3: '1_1706782595.670952_121.1'} 2024-02-01T10:33:34.010 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-02-01T10:33:34.010 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-02-01T10:33:34.010 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:33:34.010 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:33:34.010 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:33:34.052 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:33:34.052 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:33:34.110 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:33:34.115+0000 7fd90a7ec980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:33:34.111 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:33:34.115+0000 7fd90a7ec980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:33:34.133 INFO:teuthology.orchestra.run.smithi148.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783558.115340_142.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:32:38.115340Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:33:34Z"} 2024-02-01T10:33:34.134 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783558.115340_142.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:32:38.115340Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:33:34Z"} 2024-02-01T10:33:34.134 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706783558.115340_142.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:32:38.115340Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:33:34.134 DEBUG:rgw_multi.tests:log_status={0: '1_1706782453.132619_30.1', 1: '', 2: '1_1706783613.755304_167.1', 3: '1_1706782595.670952_121.1'} 2024-02-01T10:33:34.134 DEBUG:rgw_multi.tests:sync_status={0: '1_1706782453.132619_30.1', 1: '', 2: '1_1706783558.115340_142.1', 3: '1_1706782595.670952_121.1'} 2024-02-01T10:33:34.134 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1706783613.755304_167.1 target=1_1706783558.115340_142.1 2024-02-01T10:33:39.135 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:33:39.135 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:33:39.135 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:33:39.189 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:33:39.189 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:33:39.256 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:33:39.259+0000 7f06227c7980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:33:39.256 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:33:39.259+0000 7f06227c7980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:33:39.279 INFO:teuthology.orchestra.run.smithi148.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783613.632812_156.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:33:33.632812Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:33:39Z"} 2024-02-01T10:33:39.279 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783613.632812_156.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:33:33.632812Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:33:39Z"} 2024-02-01T10:33:39.279 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706783613.632812_156.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:33:33.632812Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:33:39.280 DEBUG:rgw_multi.tests:log_status={0: '1_1706782453.132619_30.1', 1: '', 2: '1_1706783613.755304_167.1', 3: '1_1706782595.670952_121.1'} 2024-02-01T10:33:39.280 DEBUG:rgw_multi.tests:sync_status={0: '1_1706782453.132619_30.1', 1: '', 2: '1_1706783613.632812_156.1', 3: '1_1706782595.670952_121.1'} 2024-02-01T10:33:39.280 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1706783613.755304_167.1 target=1_1706783613.632812_156.1 2024-02-01T10:33:44.281 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:33:44.281 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:33:44.281 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:33:44.324 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:33:44.324 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:33:44.377 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:33:44.383+0000 7f1cb78b3980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:33:44.379 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:33:44.383+0000 7f1cb78b3980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:33:44.402 INFO:teuthology.orchestra.run.smithi148.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783613.755304_167.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:33:33.755304Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:33:44Z"} 2024-02-01T10:33:44.402 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783613.755304_167.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:33:33.755304Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:33:44Z"} 2024-02-01T10:33:44.403 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706783613.755304_167.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:33:33.755304Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:33:44.403 DEBUG:rgw_multi.tests:log_status={0: '1_1706782453.132619_30.1', 1: '', 2: '1_1706783613.755304_167.1', 3: '1_1706782595.670952_121.1'} 2024-02-01T10:33:44.403 DEBUG:rgw_multi.tests:sync_status={0: '1_1706782453.132619_30.1', 1: '', 2: '1_1706783613.755304_167.1', 3: '1_1706782595.670952_121.1'} 2024-02-01T10:33:44.403 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-02-01T10:33:44.403 DEBUG:boto:path=/vclfcn-24/ 2024-02-01T10:33:44.403 DEBUG:boto:auth_path=/vclfcn-24/ 2024-02-01T10:33:44.403 DEBUG:boto:Method: HEAD 2024-02-01T10:33:44.403 DEBUG:boto:Path: /vclfcn-24/ 2024-02-01T10:33:44.403 DEBUG:boto:Data: 2024-02-01T10:33:44.404 DEBUG:boto:Headers: {} 2024-02-01T10:33:44.404 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:33:44.404 DEBUG:boto:Port: 8000 2024-02-01T10:33:44.404 DEBUG:boto:Params: {} 2024-02-01T10:33:44.404 DEBUG:boto:Token: None 2024-02-01T10:33:44.404 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:33:44 GMT /vclfcn-24/ 2024-02-01T10:33:44.404 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:XzHcx0d5JqACiCfG2DMstBLx3sg= 2024-02-01T10:33:44.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:44 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:XzHcx0d5JqACiCfG2DMstBLx3sg=', 'Content-Length': '0'} 2024-02-01T10:33:44.408 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000beb04d6fd6998329-0065bb7388-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:44.408 DEBUG:boto:path=/vclfcn-24/obj 2024-02-01T10:33:44.408 DEBUG:boto:auth_path=/vclfcn-24/obj 2024-02-01T10:33:44.408 DEBUG:boto:Method: PUT 2024-02-01T10:33:44.408 DEBUG:boto:Path: /vclfcn-24/obj 2024-02-01T10:33:44.409 DEBUG:boto:Data: 2024-02-01T10:33:44.409 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-02-01T10:33:44.409 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:33:44.409 DEBUG:boto:Port: 8000 2024-02-01T10:33:44.409 DEBUG:boto:Params: {} 2024-02-01T10:33:44.409 DEBUG:boto:Token: None 2024-02-01T10:33:44.409 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Thu, 01 Feb 2024 10:33:44 GMT /vclfcn-24/obj 2024-02-01T10:33:44.409 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/AHwLMVIq5eHrdHW7bUNystKfrA= 2024-02-01T10:33:44.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:33:44 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/AHwLMVIq5eHrdHW7bUNystKfrA='} 2024-02-01T10:33:44.416 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000645df79118da721b-0065bb7388-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:44.416 DEBUG:boto:path=/vclfcn-24/ 2024-02-01T10:33:44.416 DEBUG:boto:auth_path=/vclfcn-24/ 2024-02-01T10:33:44.416 DEBUG:boto:Method: HEAD 2024-02-01T10:33:44.416 DEBUG:boto:Path: /vclfcn-24/ 2024-02-01T10:33:44.416 DEBUG:boto:Data: 2024-02-01T10:33:44.416 DEBUG:boto:Headers: {} 2024-02-01T10:33:44.416 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:33:44.417 DEBUG:boto:Port: 8000 2024-02-01T10:33:44.417 DEBUG:boto:Params: {} 2024-02-01T10:33:44.417 DEBUG:boto:Token: None 2024-02-01T10:33:44.417 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:33:44 GMT /vclfcn-24/ 2024-02-01T10:33:44.417 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:XzHcx0d5JqACiCfG2DMstBLx3sg= 2024-02-01T10:33:44.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:44 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:XzHcx0d5JqACiCfG2DMstBLx3sg=', 'Content-Length': '0'} 2024-02-01T10:33:44.419 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f9babddabf25c499-0065bb7388-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:44.419 DEBUG:boto:path=/vclfcn-24/obj 2024-02-01T10:33:44.420 DEBUG:boto:auth_path=/vclfcn-24/obj 2024-02-01T10:33:44.420 DEBUG:boto:Method: DELETE 2024-02-01T10:33:44.420 DEBUG:boto:Path: /vclfcn-24/obj 2024-02-01T10:33:44.420 DEBUG:boto:Data: 2024-02-01T10:33:44.420 DEBUG:boto:Headers: {} 2024-02-01T10:33:44.420 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:33:44.420 DEBUG:boto:Port: 8000 2024-02-01T10:33:44.420 DEBUG:boto:Params: {} 2024-02-01T10:33:44.420 DEBUG:boto:Token: None 2024-02-01T10:33:44.421 DEBUG:boto:StringToSign: DELETE Thu, 01 Feb 2024 10:33:44 GMT /vclfcn-24/obj 2024-02-01T10:33:44.421 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:g9DutnzpnSEf4PiqS44Lb0xjrG0= 2024-02-01T10:33:44.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:44 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:g9DutnzpnSEf4PiqS44Lb0xjrG0=', 'Content-Length': '0'} 2024-02-01T10:33:44.431 DEBUG:boto:Response headers: [('x-amz-version-id', 'null'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx00000fde98290e353c77f-0065bb7388-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:44.431 DEBUG:boto:path=/vclfcn-25/ 2024-02-01T10:33:44.431 DEBUG:boto:auth_path=/vclfcn-25/ 2024-02-01T10:33:44.431 DEBUG:boto:Method: HEAD 2024-02-01T10:33:44.431 DEBUG:boto:Path: /vclfcn-25/ 2024-02-01T10:33:44.431 DEBUG:boto:Data: 2024-02-01T10:33:44.431 DEBUG:boto:Headers: {} 2024-02-01T10:33:44.431 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:33:44.431 DEBUG:boto:Port: 8000 2024-02-01T10:33:44.432 DEBUG:boto:Params: {} 2024-02-01T10:33:44.432 DEBUG:boto:Token: None 2024-02-01T10:33:44.432 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:33:44 GMT /vclfcn-25/ 2024-02-01T10:33:44.432 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:B0rsZVr2yajtpv+8MrWksARx1qg= 2024-02-01T10:33:44.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:44 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:B0rsZVr2yajtpv+8MrWksARx1qg=', 'Content-Length': '0'} 2024-02-01T10:33:44.436 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f7fc1ea2b292149f-0065bb7388-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:44.436 DEBUG:boto:path=/vclfcn-25/obj 2024-02-01T10:33:44.436 DEBUG:boto:auth_path=/vclfcn-25/obj 2024-02-01T10:33:44.436 DEBUG:boto:Method: PUT 2024-02-01T10:33:44.437 DEBUG:boto:Path: /vclfcn-25/obj 2024-02-01T10:33:44.437 DEBUG:boto:Data: 2024-02-01T10:33:44.437 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-02-01T10:33:44.437 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:33:44.437 DEBUG:boto:Port: 8000 2024-02-01T10:33:44.437 DEBUG:boto:Params: {} 2024-02-01T10:33:44.437 DEBUG:boto:Token: None 2024-02-01T10:33:44.437 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Thu, 01 Feb 2024 10:33:44 GMT /vclfcn-25/obj 2024-02-01T10:33:44.437 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/U2MpBnP/Mgg9is6cwE1HtRkWmU= 2024-02-01T10:33:44.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:33:44 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/U2MpBnP/Mgg9is6cwE1HtRkWmU='} 2024-02-01T10:33:44.451 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000037d66d36dddf7378-0065bb7388-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:44.452 DEBUG:boto:path=/vclfcn-25/ 2024-02-01T10:33:44.452 DEBUG:boto:auth_path=/vclfcn-25/ 2024-02-01T10:33:44.452 DEBUG:boto:Method: HEAD 2024-02-01T10:33:44.452 DEBUG:boto:Path: /vclfcn-25/ 2024-02-01T10:33:44.452 DEBUG:boto:Data: 2024-02-01T10:33:44.452 DEBUG:boto:Headers: {} 2024-02-01T10:33:44.452 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:33:44.452 DEBUG:boto:Port: 8000 2024-02-01T10:33:44.452 DEBUG:boto:Params: {} 2024-02-01T10:33:44.452 DEBUG:boto:Token: None 2024-02-01T10:33:44.453 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:33:44 GMT /vclfcn-25/ 2024-02-01T10:33:44.453 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:B0rsZVr2yajtpv+8MrWksARx1qg= 2024-02-01T10:33:44.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:44 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:B0rsZVr2yajtpv+8MrWksARx1qg=', 'Content-Length': '0'} 2024-02-01T10:33:44.458 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000622fee1a68b3ce78-0065bb7388-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:44.458 DEBUG:boto:path=/vclfcn-25/obj 2024-02-01T10:33:44.458 DEBUG:boto:auth_path=/vclfcn-25/obj 2024-02-01T10:33:44.458 DEBUG:boto:Method: DELETE 2024-02-01T10:33:44.458 DEBUG:boto:Path: /vclfcn-25/obj 2024-02-01T10:33:44.458 DEBUG:boto:Data: 2024-02-01T10:33:44.459 DEBUG:boto:Headers: {} 2024-02-01T10:33:44.459 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:33:44.459 DEBUG:boto:Port: 8000 2024-02-01T10:33:44.459 DEBUG:boto:Params: {} 2024-02-01T10:33:44.459 DEBUG:boto:Token: None 2024-02-01T10:33:44.459 DEBUG:boto:StringToSign: DELETE Thu, 01 Feb 2024 10:33:44 GMT /vclfcn-25/obj 2024-02-01T10:33:44.459 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:rxVd7W2SniRqSasiyj8HEk6u3I4= 2024-02-01T10:33:44.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:44 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:rxVd7W2SniRqSasiyj8HEk6u3I4=', 'Content-Length': '0'} 2024-02-01T10:33:44.482 DEBUG:boto:Response headers: [('x-amz-version-id', 'null'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx00000d3555c1544cf07fa-0065bb7388-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:44.482 DEBUG:boto:path=/vclfcn-26/ 2024-02-01T10:33:44.482 DEBUG:boto:auth_path=/vclfcn-26/ 2024-02-01T10:33:44.482 DEBUG:boto:Method: HEAD 2024-02-01T10:33:44.483 DEBUG:boto:Path: /vclfcn-26/ 2024-02-01T10:33:44.483 DEBUG:boto:Data: 2024-02-01T10:33:44.483 DEBUG:boto:Headers: {} 2024-02-01T10:33:44.483 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:33:44.483 DEBUG:boto:Port: 8001 2024-02-01T10:33:44.483 DEBUG:boto:Params: {} 2024-02-01T10:33:44.483 DEBUG:boto:Token: None 2024-02-01T10:33:44.483 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:33:44 GMT /vclfcn-26/ 2024-02-01T10:33:44.483 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZWjy7HbaHQfeCw1LIk53euSeh00= 2024-02-01T10:33:44.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:44 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZWjy7HbaHQfeCw1LIk53euSeh00=', 'Content-Length': '0'} 2024-02-01T10:33:44.487 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000587bd6e072c25fc0-0065bb7388-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:44.488 DEBUG:boto:path=/vclfcn-26/obj 2024-02-01T10:33:44.488 DEBUG:boto:auth_path=/vclfcn-26/obj 2024-02-01T10:33:44.488 DEBUG:boto:Method: PUT 2024-02-01T10:33:44.488 DEBUG:boto:Path: /vclfcn-26/obj 2024-02-01T10:33:44.488 DEBUG:boto:Data: 2024-02-01T10:33:44.488 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-02-01T10:33:44.488 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:33:44.489 DEBUG:boto:Port: 8001 2024-02-01T10:33:44.489 DEBUG:boto:Params: {} 2024-02-01T10:33:44.489 DEBUG:boto:Token: None 2024-02-01T10:33:44.490 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Thu, 01 Feb 2024 10:33:44 GMT /vclfcn-26/obj 2024-02-01T10:33:44.490 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:g1CxDnW2Q2cBlqnuOtNbSdEW/xw= 2024-02-01T10:33:44.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:33:44 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:g1CxDnW2Q2cBlqnuOtNbSdEW/xw='} 2024-02-01T10:33:44.504 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b806dec535fe3f8b-0065bb7388-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:44.504 DEBUG:boto:path=/vclfcn-26/ 2024-02-01T10:33:44.504 DEBUG:boto:auth_path=/vclfcn-26/ 2024-02-01T10:33:44.504 DEBUG:boto:Method: HEAD 2024-02-01T10:33:44.504 DEBUG:boto:Path: /vclfcn-26/ 2024-02-01T10:33:44.504 DEBUG:boto:Data: 2024-02-01T10:33:44.504 DEBUG:boto:Headers: {} 2024-02-01T10:33:44.504 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:33:44.504 DEBUG:boto:Port: 8001 2024-02-01T10:33:44.505 DEBUG:boto:Params: {} 2024-02-01T10:33:44.505 DEBUG:boto:Token: None 2024-02-01T10:33:44.505 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:33:44 GMT /vclfcn-26/ 2024-02-01T10:33:44.505 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZWjy7HbaHQfeCw1LIk53euSeh00= 2024-02-01T10:33:44.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:44 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZWjy7HbaHQfeCw1LIk53euSeh00=', 'Content-Length': '0'} 2024-02-01T10:33:44.509 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d068ebf5ef167d5f-0065bb7388-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:44.509 DEBUG:boto:path=/vclfcn-26/obj 2024-02-01T10:33:44.509 DEBUG:boto:auth_path=/vclfcn-26/obj 2024-02-01T10:33:44.509 DEBUG:boto:Method: DELETE 2024-02-01T10:33:44.509 DEBUG:boto:Path: /vclfcn-26/obj 2024-02-01T10:33:44.510 DEBUG:boto:Data: 2024-02-01T10:33:44.510 DEBUG:boto:Headers: {} 2024-02-01T10:33:44.510 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:33:44.510 DEBUG:boto:Port: 8001 2024-02-01T10:33:44.510 DEBUG:boto:Params: {} 2024-02-01T10:33:44.510 DEBUG:boto:Token: None 2024-02-01T10:33:44.510 DEBUG:boto:StringToSign: DELETE Thu, 01 Feb 2024 10:33:44 GMT /vclfcn-26/obj 2024-02-01T10:33:44.510 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nmuLyyiEQZ1dmHYUwgq7Uu+Cngw= 2024-02-01T10:33:44.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:33:44 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nmuLyyiEQZ1dmHYUwgq7Uu+Cngw=', 'Content-Length': '0'} 2024-02-01T10:33:44.531 DEBUG:boto:Response headers: [('x-amz-version-id', 'null'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx00000968e883df4ed81e5-0065bb7388-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:33:44.532 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=vclfcn-24 2024-02-01T10:33:44.532 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:33:44.532 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:33:44.532 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket vclfcn-24 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-02-01T10:33:44.582 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:33:44.582 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:33:44.648 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:33:44.645+0000 7f85689e2980 1 waiting to reach incremental sync.. 2024-02-01T10:33:49.651 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:33:49.648+0000 7f85689e2980 1 bucket sync caught up with source: 2024-02-01T10:33:49.651 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, , , , , , , , 00000000003.23.7, , ] 2024-02-01T10:33:49.651 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, , , , , , , , 00000000003.23.7, , ] 2024-02-01T10:33:49.651 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:33:49.648+0000 7f85689e2980 0 bucket checkpoint complete 2024-02-01T10:33:49.689 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=vclfcn-24 2024-02-01T10:33:49.690 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:33:49.690 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:33:49.690 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-24 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:33:49.745 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:33:49.745 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:33:49.808 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:33:49.811+0000 7f116418c980 1 waiting to reach incremental sync.. 2024-02-01T10:33:54.811 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:33:54.814+0000 7f116418c980 1 bucket sync caught up with source: 2024-02-01T10:33:54.811 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, , , , , , , , 00000000003.23.7, , ] 2024-02-01T10:33:54.811 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, , , , , , , , 00000000003.23.7, , ] 2024-02-01T10:33:54.811 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:33:54.814+0000 7f116418c980 0 bucket checkpoint complete 2024-02-01T10:33:54.850 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=vclfcn-24 2024-02-01T10:33:54.851 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:33:54.851 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:33:54.851 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-24 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:33:54.899 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:33:54.900 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:33:54.961 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:33:54.962+0000 7fd0dcf21980 1 waiting to reach incremental sync.. 2024-02-01T10:33:59.962 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:33:59.966+0000 7fd0dcf21980 1 waiting to reach incremental sync.. 2024-02-01T10:34:04.963 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:34:04.966+0000 7fd0dcf21980 1 waiting to reach incremental sync.. 2024-02-01T10:34:09.966 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:34:09.970+0000 7fd0dcf21980 1 bucket sync caught up with source: 2024-02-01T10:34:09.966 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, , , , , , , , 00000000001.36.14, , ] 2024-02-01T10:34:09.966 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, , , , , , , , 00000000001.36.14, , ] 2024-02-01T10:34:09.966 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:34:09.970+0000 7fd0dcf21980 0 bucket checkpoint complete 2024-02-01T10:34:10.008 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=vclfcn-24 2024-02-01T10:34:10.008 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:34:10.009 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:34:10.009 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-24 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:34:10.097 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:34:10.098 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:34:10.163 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:34:10.166+0000 7f728aec5980 1 bucket sync caught up with source: 2024-02-01T10:34:10.164 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, , , , , , , , 00000000001.36.14, , ] 2024-02-01T10:34:10.164 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, , , , , , , , 00000000001.36.14, , ] 2024-02-01T10:34:10.164 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:34:10.166+0000 7f728aec5980 0 bucket checkpoint complete 2024-02-01T10:34:10.186 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=vclfcn-24 2024-02-01T10:34:10.187 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:34:10.187 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:34:10.187 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-24 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:34:10.227 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:34:10.228 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:34:10.284 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:34:10.286+0000 7f5152982980 1 bucket sync caught up with source: 2024-02-01T10:34:10.285 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, , , , , , , , 00000000001.76.14, , ] 2024-02-01T10:34:10.285 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, , , , , , , , 00000000001.76.14, , ] 2024-02-01T10:34:10.285 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:34:10.286+0000 7f5152982980 0 bucket checkpoint complete 2024-02-01T10:34:10.306 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=vclfcn-24 2024-02-01T10:34:10.307 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:34:10.307 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:34:10.307 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket vclfcn-24 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-02-01T10:34:10.358 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:34:10.358 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:34:10.427 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:34:10.428+0000 7ff817ad3980 1 bucket sync caught up with source: 2024-02-01T10:34:10.427 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, , , , , , , , 00000000001.76.14, , ] 2024-02-01T10:34:10.428 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, , , , , , , , 00000000001.76.14, , ] 2024-02-01T10:34:10.428 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:34:10.428+0000 7ff817ad3980 0 bucket checkpoint complete 2024-02-01T10:34:10.452 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-24 zones={test-zone2, test-zone1} 2024-02-01T10:34:10.452 DEBUG:boto:path=/vclfcn-24/ 2024-02-01T10:34:10.453 DEBUG:boto:auth_path=/vclfcn-24/ 2024-02-01T10:34:10.453 DEBUG:boto:Method: HEAD 2024-02-01T10:34:10.453 DEBUG:boto:Path: /vclfcn-24/ 2024-02-01T10:34:10.453 DEBUG:boto:Data: 2024-02-01T10:34:10.453 DEBUG:boto:Headers: {} 2024-02-01T10:34:10.453 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:34:10.453 DEBUG:boto:Port: 8000 2024-02-01T10:34:10.453 DEBUG:boto:Params: {} 2024-02-01T10:34:10.453 DEBUG:boto:Token: None 2024-02-01T10:34:10.454 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:34:10 GMT /vclfcn-24/ 2024-02-01T10:34:10.454 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:kHkr3SRmqHUqxnU1RjTsDs1rCVo= 2024-02-01T10:34:10.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:34:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:kHkr3SRmqHUqxnU1RjTsDs1rCVo=', 'Content-Length': '0'} 2024-02-01T10:34:10.456 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000584219ba56474089-0065bb73a2-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:34:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:34:10.457 DEBUG:boto:path=/vclfcn-24/ 2024-02-01T10:34:10.457 DEBUG:boto:auth_path=/vclfcn-24/ 2024-02-01T10:34:10.457 DEBUG:boto:Method: HEAD 2024-02-01T10:34:10.457 DEBUG:boto:Path: /vclfcn-24/ 2024-02-01T10:34:10.457 DEBUG:boto:Data: 2024-02-01T10:34:10.457 DEBUG:boto:Headers: {} 2024-02-01T10:34:10.457 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:34:10.457 DEBUG:boto:Port: 8000 2024-02-01T10:34:10.457 DEBUG:boto:Params: {} 2024-02-01T10:34:10.458 DEBUG:boto:Token: None 2024-02-01T10:34:10.458 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:34:10 GMT /vclfcn-24/ 2024-02-01T10:34:10.458 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:kHkr3SRmqHUqxnU1RjTsDs1rCVo= 2024-02-01T10:34:10.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:34:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:kHkr3SRmqHUqxnU1RjTsDs1rCVo=', 'Content-Length': '0'} 2024-02-01T10:34:10.462 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000030471e3a5909b01-0065bb73a2-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:34:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:34:10.462 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:34:10.462 DEBUG:boto:path=/vclfcn-24/ 2024-02-01T10:34:10.462 DEBUG:boto:auth_path=/vclfcn-24/ 2024-02-01T10:34:10.462 DEBUG:boto:path=/vclfcn-24/?versions&max-keys=999 2024-02-01T10:34:10.462 DEBUG:boto:auth_path=/vclfcn-24/?versions&max-keys=999 2024-02-01T10:34:10.462 DEBUG:boto:Method: GET 2024-02-01T10:34:10.463 DEBUG:boto:Path: /vclfcn-24/?versions&max-keys=999 2024-02-01T10:34:10.463 DEBUG:boto:Data: 2024-02-01T10:34:10.463 DEBUG:boto:Headers: {} 2024-02-01T10:34:10.463 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:34:10.463 DEBUG:boto:Port: 8000 2024-02-01T10:34:10.463 DEBUG:boto:Params: {} 2024-02-01T10:34:10.463 DEBUG:boto:Token: None 2024-02-01T10:34:10.463 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:34:10 GMT /vclfcn-24/?versions 2024-02-01T10:34:10.463 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:buPW2SQ0OGzWO0Fj8LwH1clSbw8= 2024-02-01T10:34:10.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:34:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:buPW2SQ0OGzWO0Fj8LwH1clSbw8=', 'Content-Length': '0'} 2024-02-01T10:34:10.465 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000075f4650228573b75-0065bb73a2-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:34:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:34:10.466 DEBUG:boto:b'vclfcn-24999falseobjnulltrue2024-02-01T10:33:44.431Zrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:34:10.466 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:34:10.466 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:34:10.467 DEBUG:boto:path=/vclfcn-24/ 2024-02-01T10:34:10.467 DEBUG:boto:auth_path=/vclfcn-24/ 2024-02-01T10:34:10.467 DEBUG:boto:path=/vclfcn-24/?versions&max-keys=999 2024-02-01T10:34:10.467 DEBUG:boto:auth_path=/vclfcn-24/?versions&max-keys=999 2024-02-01T10:34:10.467 DEBUG:boto:Method: GET 2024-02-01T10:34:10.467 DEBUG:boto:Path: /vclfcn-24/?versions&max-keys=999 2024-02-01T10:34:10.467 DEBUG:boto:Data: 2024-02-01T10:34:10.467 DEBUG:boto:Headers: {} 2024-02-01T10:34:10.467 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:34:10.467 DEBUG:boto:Port: 8000 2024-02-01T10:34:10.467 DEBUG:boto:Params: {} 2024-02-01T10:34:10.468 DEBUG:boto:Token: None 2024-02-01T10:34:10.468 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:34:10 GMT /vclfcn-24/?versions 2024-02-01T10:34:10.468 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:buPW2SQ0OGzWO0Fj8LwH1clSbw8= 2024-02-01T10:34:10.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:34:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:buPW2SQ0OGzWO0Fj8LwH1clSbw8=', 'Content-Length': '0'} 2024-02-01T10:34:10.472 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094aaec01702f60f8-0065bb73a2-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:34:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:34:10.473 DEBUG:boto:b'vclfcn-24999falseobjnulltrue2024-02-01T10:33:44.431Zrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:34:10.473 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:34:10.473 DEBUG:boto:path=/vclfcn-24/ 2024-02-01T10:34:10.473 DEBUG:boto:auth_path=/vclfcn-24/ 2024-02-01T10:34:10.473 DEBUG:boto:path=/vclfcn-24/?versions&max-keys=999 2024-02-01T10:34:10.473 DEBUG:boto:auth_path=/vclfcn-24/?versions&max-keys=999 2024-02-01T10:34:10.473 DEBUG:boto:Method: GET 2024-02-01T10:34:10.473 DEBUG:boto:Path: /vclfcn-24/?versions&max-keys=999 2024-02-01T10:34:10.474 DEBUG:boto:Data: 2024-02-01T10:34:10.474 DEBUG:boto:Headers: {} 2024-02-01T10:34:10.474 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:34:10.474 DEBUG:boto:Port: 8000 2024-02-01T10:34:10.474 DEBUG:boto:Params: {} 2024-02-01T10:34:10.474 DEBUG:boto:Token: None 2024-02-01T10:34:10.474 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:34:10 GMT /vclfcn-24/?versions 2024-02-01T10:34:10.474 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:buPW2SQ0OGzWO0Fj8LwH1clSbw8= 2024-02-01T10:34:10.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:34:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:buPW2SQ0OGzWO0Fj8LwH1clSbw8=', 'Content-Length': '0'} 2024-02-01T10:34:10.477 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b22435b505f5e93c-0065bb73a2-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:34:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:34:10.477 DEBUG:boto:b'vclfcn-24999falseobjnulltrue2024-02-01T10:33:44.431Zrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:34:10.477 DEBUG:boto:path=/vclfcn-24/ 2024-02-01T10:34:10.478 DEBUG:boto:auth_path=/vclfcn-24/ 2024-02-01T10:34:10.478 DEBUG:boto:path=/vclfcn-24/?versions&max-keys=999 2024-02-01T10:34:10.478 DEBUG:boto:auth_path=/vclfcn-24/?versions&max-keys=999 2024-02-01T10:34:10.478 DEBUG:boto:Method: GET 2024-02-01T10:34:10.478 DEBUG:boto:Path: /vclfcn-24/?versions&max-keys=999 2024-02-01T10:34:10.478 DEBUG:boto:Data: 2024-02-01T10:34:10.478 DEBUG:boto:Headers: {} 2024-02-01T10:34:10.478 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:34:10.478 DEBUG:boto:Port: 8000 2024-02-01T10:34:10.478 DEBUG:boto:Params: {} 2024-02-01T10:34:10.478 DEBUG:boto:Token: None 2024-02-01T10:34:10.479 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:34:10 GMT /vclfcn-24/?versions 2024-02-01T10:34:10.479 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:buPW2SQ0OGzWO0Fj8LwH1clSbw8= 2024-02-01T10:34:10.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:34:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:buPW2SQ0OGzWO0Fj8LwH1clSbw8=', 'Content-Length': '0'} 2024-02-01T10:34:10.483 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000070d616a82b7c408b-0065bb73a2-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:34:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:34:10.483 DEBUG:boto:b'vclfcn-24999falseobjnulltrue2024-02-01T10:33:44.431Zrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:34:10.483 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:34:10.484 DEBUG:boto:path=/vclfcn-24/obj 2024-02-01T10:34:10.484 DEBUG:boto:auth_path=/vclfcn-24/obj 2024-02-01T10:34:10.484 DEBUG:boto:Method: HEAD 2024-02-01T10:34:10.484 DEBUG:boto:Path: /vclfcn-24/obj 2024-02-01T10:34:10.484 DEBUG:boto:Data: 2024-02-01T10:34:10.484 DEBUG:boto:Headers: {} 2024-02-01T10:34:10.484 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:34:10.484 DEBUG:boto:Port: 8000 2024-02-01T10:34:10.484 DEBUG:boto:Params: {} 2024-02-01T10:34:10.485 DEBUG:boto:Token: None 2024-02-01T10:34:10.485 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:34:10 GMT /vclfcn-24/obj 2024-02-01T10:34:10.485 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NIWixTEwLLtcOd7z+2r5YWYjZ48= 2024-02-01T10:34:10.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:34:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NIWixTEwLLtcOd7z+2r5YWYjZ48=', 'Content-Length': '0'} 2024-02-01T10:34:10.487 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000104f4fe8ccd1a6bb-0065bb73a2-4307-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:34:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:34:10.487 DEBUG:boto:path=/vclfcn-24/obj 2024-02-01T10:34:10.487 DEBUG:boto:auth_path=/vclfcn-24/obj 2024-02-01T10:34:10.487 DEBUG:boto:Method: HEAD 2024-02-01T10:34:10.487 DEBUG:boto:Path: /vclfcn-24/obj 2024-02-01T10:34:10.487 DEBUG:boto:Data: 2024-02-01T10:34:10.487 DEBUG:boto:Headers: {} 2024-02-01T10:34:10.487 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:34:10.487 DEBUG:boto:Port: 8000 2024-02-01T10:34:10.488 DEBUG:boto:Params: {} 2024-02-01T10:34:10.488 DEBUG:boto:Token: None 2024-02-01T10:34:10.488 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:34:10 GMT /vclfcn-24/obj 2024-02-01T10:34:10.488 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NIWixTEwLLtcOd7z+2r5YWYjZ48= 2024-02-01T10:34:10.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:34:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NIWixTEwLLtcOd7z+2r5YWYjZ48=', 'Content-Length': '0'} 2024-02-01T10:34:10.490 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000004b632e380b53581c-0065bb73a2-4351-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:34:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:34:10.490 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-24 zones={test-zone2, test-zone1} 2024-02-01T10:34:10.491 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-24 zones={test-zone3, test-zone1} 2024-02-01T10:34:10.491 DEBUG:boto:path=/vclfcn-24/ 2024-02-01T10:34:10.491 DEBUG:boto:auth_path=/vclfcn-24/ 2024-02-01T10:34:10.491 DEBUG:boto:Method: HEAD 2024-02-01T10:34:10.491 DEBUG:boto:Path: /vclfcn-24/ 2024-02-01T10:34:10.491 DEBUG:boto:Data: 2024-02-01T10:34:10.491 DEBUG:boto:Headers: {} 2024-02-01T10:34:10.491 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:34:10.491 DEBUG:boto:Port: 8001 2024-02-01T10:34:10.491 DEBUG:boto:Params: {} 2024-02-01T10:34:10.492 DEBUG:boto:Token: None 2024-02-01T10:34:10.492 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:34:10 GMT /vclfcn-24/ 2024-02-01T10:34:10.492 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:kHkr3SRmqHUqxnU1RjTsDs1rCVo= 2024-02-01T10:34:10.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:34:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:kHkr3SRmqHUqxnU1RjTsDs1rCVo=', 'Content-Length': '0'} 2024-02-01T10:34:10.496 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a5461e6f27da4fcd-0065bb73a2-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:34:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:34:10.496 DEBUG:boto:path=/vclfcn-24/ 2024-02-01T10:34:10.496 DEBUG:boto:auth_path=/vclfcn-24/ 2024-02-01T10:34:10.496 DEBUG:boto:Method: HEAD 2024-02-01T10:34:10.496 DEBUG:boto:Path: /vclfcn-24/ 2024-02-01T10:34:10.496 DEBUG:boto:Data: 2024-02-01T10:34:10.496 DEBUG:boto:Headers: {} 2024-02-01T10:34:10.496 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:34:10.496 DEBUG:boto:Port: 8000 2024-02-01T10:34:10.497 DEBUG:boto:Params: {} 2024-02-01T10:34:10.497 DEBUG:boto:Token: None 2024-02-01T10:34:10.497 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:34:10 GMT /vclfcn-24/ 2024-02-01T10:34:10.497 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:kHkr3SRmqHUqxnU1RjTsDs1rCVo= 2024-02-01T10:34:10.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:34:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:kHkr3SRmqHUqxnU1RjTsDs1rCVo=', 'Content-Length': '0'} 2024-02-01T10:34:10.501 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ecfe97fe4962aea6-0065bb73a2-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:34:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:34:10.501 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:34:10.501 DEBUG:boto:path=/vclfcn-24/ 2024-02-01T10:34:10.501 DEBUG:boto:auth_path=/vclfcn-24/ 2024-02-01T10:34:10.501 DEBUG:boto:path=/vclfcn-24/?versions&max-keys=999 2024-02-01T10:34:10.501 DEBUG:boto:auth_path=/vclfcn-24/?versions&max-keys=999 2024-02-01T10:34:10.502 DEBUG:boto:Method: GET 2024-02-01T10:34:10.502 DEBUG:boto:Path: /vclfcn-24/?versions&max-keys=999 2024-02-01T10:34:10.502 DEBUG:boto:Data: 2024-02-01T10:34:10.502 DEBUG:boto:Headers: {} 2024-02-01T10:34:10.502 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:34:10.502 DEBUG:boto:Port: 8001 2024-02-01T10:34:10.502 DEBUG:boto:Params: {} 2024-02-01T10:34:10.502 DEBUG:boto:Token: None 2024-02-01T10:34:10.502 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:34:10 GMT /vclfcn-24/?versions 2024-02-01T10:34:10.502 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:buPW2SQ0OGzWO0Fj8LwH1clSbw8= 2024-02-01T10:34:10.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:34:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:buPW2SQ0OGzWO0Fj8LwH1clSbw8=', 'Content-Length': '0'} 2024-02-01T10:34:10.507 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006fd981672218d7be-0065bb73a2-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:34:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:34:10.507 DEBUG:boto:b'vclfcn-24999falseobjnulltrue2024-02-01T10:33:44.431Zrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:34:10.507 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:34:10.507 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:34:10.507 DEBUG:boto:path=/vclfcn-24/ 2024-02-01T10:34:10.507 DEBUG:boto:auth_path=/vclfcn-24/ 2024-02-01T10:34:10.508 DEBUG:boto:path=/vclfcn-24/?versions&max-keys=999 2024-02-01T10:34:10.508 DEBUG:boto:auth_path=/vclfcn-24/?versions&max-keys=999 2024-02-01T10:34:10.508 DEBUG:boto:Method: GET 2024-02-01T10:34:10.508 DEBUG:boto:Path: /vclfcn-24/?versions&max-keys=999 2024-02-01T10:34:10.508 DEBUG:boto:Data: 2024-02-01T10:34:10.508 DEBUG:boto:Headers: {} 2024-02-01T10:34:10.508 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:34:10.508 DEBUG:boto:Port: 8000 2024-02-01T10:34:10.508 DEBUG:boto:Params: {} 2024-02-01T10:34:10.508 DEBUG:boto:Token: None 2024-02-01T10:34:10.509 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:34:10 GMT /vclfcn-24/?versions 2024-02-01T10:34:10.509 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:buPW2SQ0OGzWO0Fj8LwH1clSbw8= 2024-02-01T10:34:10.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:34:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:buPW2SQ0OGzWO0Fj8LwH1clSbw8=', 'Content-Length': '0'} 2024-02-01T10:34:10.513 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005cad15cd5c31db8f-0065bb73a2-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:34:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:34:10.556 DEBUG:boto:b'vclfcn-24999falseobjnulltrue2024-02-01T10:33:44.431Zrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:34:10.557 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:34:10.557 DEBUG:boto:path=/vclfcn-24/ 2024-02-01T10:34:10.557 DEBUG:boto:auth_path=/vclfcn-24/ 2024-02-01T10:34:10.557 DEBUG:boto:path=/vclfcn-24/?versions&max-keys=999 2024-02-01T10:34:10.557 DEBUG:boto:auth_path=/vclfcn-24/?versions&max-keys=999 2024-02-01T10:34:10.557 DEBUG:boto:Method: GET 2024-02-01T10:34:10.557 DEBUG:boto:Path: /vclfcn-24/?versions&max-keys=999 2024-02-01T10:34:10.557 DEBUG:boto:Data: 2024-02-01T10:34:10.557 DEBUG:boto:Headers: {} 2024-02-01T10:34:10.557 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:34:10.557 DEBUG:boto:Port: 8001 2024-02-01T10:34:10.558 DEBUG:boto:Params: {} 2024-02-01T10:34:10.558 DEBUG:boto:Token: None 2024-02-01T10:34:10.558 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:34:10 GMT /vclfcn-24/?versions 2024-02-01T10:34:10.558 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:buPW2SQ0OGzWO0Fj8LwH1clSbw8= 2024-02-01T10:34:10.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:34:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:buPW2SQ0OGzWO0Fj8LwH1clSbw8=', 'Content-Length': '0'} 2024-02-01T10:34:10.562 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000523b6fba2f5bd4b5-0065bb73a2-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:34:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:34:10.562 DEBUG:boto:b'vclfcn-24999falseobjnulltrue2024-02-01T10:33:44.431Zrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:34:10.563 DEBUG:boto:path=/vclfcn-24/ 2024-02-01T10:34:10.563 DEBUG:boto:auth_path=/vclfcn-24/ 2024-02-01T10:34:10.563 DEBUG:boto:path=/vclfcn-24/?versions&max-keys=999 2024-02-01T10:34:10.563 DEBUG:boto:auth_path=/vclfcn-24/?versions&max-keys=999 2024-02-01T10:34:10.563 DEBUG:boto:Method: GET 2024-02-01T10:34:10.563 DEBUG:boto:Path: /vclfcn-24/?versions&max-keys=999 2024-02-01T10:34:10.563 DEBUG:boto:Data: 2024-02-01T10:34:10.563 DEBUG:boto:Headers: {} 2024-02-01T10:34:10.563 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:34:10.563 DEBUG:boto:Port: 8000 2024-02-01T10:34:10.563 DEBUG:boto:Params: {} 2024-02-01T10:34:10.564 DEBUG:boto:Token: None 2024-02-01T10:34:10.564 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:34:10 GMT /vclfcn-24/?versions 2024-02-01T10:34:10.564 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:buPW2SQ0OGzWO0Fj8LwH1clSbw8= 2024-02-01T10:34:10.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:34:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:buPW2SQ0OGzWO0Fj8LwH1clSbw8=', 'Content-Length': '0'} 2024-02-01T10:34:10.568 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026847525ea3cf9ca-0065bb73a2-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:34:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:34:10.568 DEBUG:boto:b'vclfcn-24999falseobjnulltrue2024-02-01T10:33:44.431Zrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:34:10.569 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:34:10.569 DEBUG:boto:path=/vclfcn-24/obj 2024-02-01T10:34:10.569 DEBUG:boto:auth_path=/vclfcn-24/obj 2024-02-01T10:34:10.569 DEBUG:boto:Method: HEAD 2024-02-01T10:34:10.569 DEBUG:boto:Path: /vclfcn-24/obj 2024-02-01T10:34:10.569 DEBUG:boto:Data: 2024-02-01T10:34:10.569 DEBUG:boto:Headers: {} 2024-02-01T10:34:10.569 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:34:10.569 DEBUG:boto:Port: 8001 2024-02-01T10:34:10.569 DEBUG:boto:Params: {} 2024-02-01T10:34:10.570 DEBUG:boto:Token: None 2024-02-01T10:34:10.570 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:34:10 GMT /vclfcn-24/obj 2024-02-01T10:34:10.570 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NIWixTEwLLtcOd7z+2r5YWYjZ48= 2024-02-01T10:34:10.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:34:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NIWixTEwLLtcOd7z+2r5YWYjZ48=', 'Content-Length': '0'} 2024-02-01T10:34:10.572 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx0000008743e02ed55c030-0065bb73a2-4358-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:34:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:34:10.573 DEBUG:boto:path=/vclfcn-24/obj 2024-02-01T10:34:10.573 DEBUG:boto:auth_path=/vclfcn-24/obj 2024-02-01T10:34:10.573 DEBUG:boto:Method: HEAD 2024-02-01T10:34:10.573 DEBUG:boto:Path: /vclfcn-24/obj 2024-02-01T10:34:10.573 DEBUG:boto:Data: 2024-02-01T10:34:10.573 DEBUG:boto:Headers: {} 2024-02-01T10:34:10.573 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:34:10.573 DEBUG:boto:Port: 8000 2024-02-01T10:34:10.573 DEBUG:boto:Params: {} 2024-02-01T10:34:10.573 DEBUG:boto:Token: None 2024-02-01T10:34:10.574 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:34:10 GMT /vclfcn-24/obj 2024-02-01T10:34:10.574 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NIWixTEwLLtcOd7z+2r5YWYjZ48= 2024-02-01T10:34:10.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:34:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NIWixTEwLLtcOd7z+2r5YWYjZ48=', 'Content-Length': '0'} 2024-02-01T10:34:10.576 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000b78c1a73400b8959-0065bb73a2-4351-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:34:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:34:10.576 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-24 zones={test-zone3, test-zone1} 2024-02-01T10:34:10.576 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-24 zones={test-zone3, test-zone2} 2024-02-01T10:34:10.576 DEBUG:boto:path=/vclfcn-24/ 2024-02-01T10:34:10.576 DEBUG:boto:auth_path=/vclfcn-24/ 2024-02-01T10:34:10.577 DEBUG:boto:Method: HEAD 2024-02-01T10:34:10.577 DEBUG:boto:Path: /vclfcn-24/ 2024-02-01T10:34:10.577 DEBUG:boto:Data: 2024-02-01T10:34:10.577 DEBUG:boto:Headers: {} 2024-02-01T10:34:10.577 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:34:10.577 DEBUG:boto:Port: 8001 2024-02-01T10:34:10.577 DEBUG:boto:Params: {} 2024-02-01T10:34:10.577 DEBUG:boto:Token: None 2024-02-01T10:34:10.577 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:34:10 GMT /vclfcn-24/ 2024-02-01T10:34:10.577 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:kHkr3SRmqHUqxnU1RjTsDs1rCVo= 2024-02-01T10:34:10.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:34:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:kHkr3SRmqHUqxnU1RjTsDs1rCVo=', 'Content-Length': '0'} 2024-02-01T10:34:10.582 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000ff8bec06783f72d-0065bb73a2-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:34:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:34:10.582 DEBUG:boto:path=/vclfcn-24/ 2024-02-01T10:34:10.582 DEBUG:boto:auth_path=/vclfcn-24/ 2024-02-01T10:34:10.582 DEBUG:boto:Method: HEAD 2024-02-01T10:34:10.582 DEBUG:boto:Path: /vclfcn-24/ 2024-02-01T10:34:10.582 DEBUG:boto:Data: 2024-02-01T10:34:10.582 DEBUG:boto:Headers: {} 2024-02-01T10:34:10.582 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:34:10.582 DEBUG:boto:Port: 8000 2024-02-01T10:34:10.582 DEBUG:boto:Params: {} 2024-02-01T10:34:10.583 DEBUG:boto:Token: None 2024-02-01T10:34:10.583 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:34:10 GMT /vclfcn-24/ 2024-02-01T10:34:10.583 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:kHkr3SRmqHUqxnU1RjTsDs1rCVo= 2024-02-01T10:34:10.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:34:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:kHkr3SRmqHUqxnU1RjTsDs1rCVo=', 'Content-Length': '0'} 2024-02-01T10:34:10.587 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d894da30c6b193cf-0065bb73a2-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:34:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:34:10.587 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:34:10.588 DEBUG:boto:path=/vclfcn-24/ 2024-02-01T10:34:10.588 DEBUG:boto:auth_path=/vclfcn-24/ 2024-02-01T10:34:10.588 DEBUG:boto:path=/vclfcn-24/?versions&max-keys=999 2024-02-01T10:34:10.588 DEBUG:boto:auth_path=/vclfcn-24/?versions&max-keys=999 2024-02-01T10:34:10.588 DEBUG:boto:Method: GET 2024-02-01T10:34:10.588 DEBUG:boto:Path: /vclfcn-24/?versions&max-keys=999 2024-02-01T10:34:10.588 DEBUG:boto:Data: 2024-02-01T10:34:10.588 DEBUG:boto:Headers: {} 2024-02-01T10:34:10.588 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:34:10.588 DEBUG:boto:Port: 8001 2024-02-01T10:34:10.589 DEBUG:boto:Params: {} 2024-02-01T10:34:10.589 DEBUG:boto:Token: None 2024-02-01T10:34:10.589 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:34:10 GMT /vclfcn-24/?versions 2024-02-01T10:34:10.589 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:buPW2SQ0OGzWO0Fj8LwH1clSbw8= 2024-02-01T10:34:10.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:34:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:buPW2SQ0OGzWO0Fj8LwH1clSbw8=', 'Content-Length': '0'} 2024-02-01T10:34:10.593 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f15cd0ad43a0e45d-0065bb73a2-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:34:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:34:10.636 DEBUG:boto:b'vclfcn-24999falseobjnulltrue2024-02-01T10:33:44.431Zrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:34:10.637 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:34:10.637 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:34:10.637 DEBUG:boto:path=/vclfcn-24/ 2024-02-01T10:34:10.637 DEBUG:boto:auth_path=/vclfcn-24/ 2024-02-01T10:34:10.637 DEBUG:boto:path=/vclfcn-24/?versions&max-keys=999 2024-02-01T10:34:10.637 DEBUG:boto:auth_path=/vclfcn-24/?versions&max-keys=999 2024-02-01T10:34:10.637 DEBUG:boto:Method: GET 2024-02-01T10:34:10.637 DEBUG:boto:Path: /vclfcn-24/?versions&max-keys=999 2024-02-01T10:34:10.638 DEBUG:boto:Data: 2024-02-01T10:34:10.638 DEBUG:boto:Headers: {} 2024-02-01T10:34:10.638 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:34:10.638 DEBUG:boto:Port: 8000 2024-02-01T10:34:10.638 DEBUG:boto:Params: {} 2024-02-01T10:34:10.638 DEBUG:boto:Token: None 2024-02-01T10:34:10.638 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:34:10 GMT /vclfcn-24/?versions 2024-02-01T10:34:10.638 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:buPW2SQ0OGzWO0Fj8LwH1clSbw8= 2024-02-01T10:34:10.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:34:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:buPW2SQ0OGzWO0Fj8LwH1clSbw8=', 'Content-Length': '0'} 2024-02-01T10:34:10.642 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ffb3ed1f7c5d8206-0065bb73a2-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:34:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:34:10.642 DEBUG:boto:b'vclfcn-24999falseobjnulltrue2024-02-01T10:33:44.431Zrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:34:10.643 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:34:10.643 DEBUG:boto:path=/vclfcn-24/ 2024-02-01T10:34:10.643 DEBUG:boto:auth_path=/vclfcn-24/ 2024-02-01T10:34:10.643 DEBUG:boto:path=/vclfcn-24/?versions&max-keys=999 2024-02-01T10:34:10.643 DEBUG:boto:auth_path=/vclfcn-24/?versions&max-keys=999 2024-02-01T10:34:10.643 DEBUG:boto:Method: GET 2024-02-01T10:34:10.643 DEBUG:boto:Path: /vclfcn-24/?versions&max-keys=999 2024-02-01T10:34:10.643 DEBUG:boto:Data: 2024-02-01T10:34:10.644 DEBUG:boto:Headers: {} 2024-02-01T10:34:10.644 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:34:10.644 DEBUG:boto:Port: 8001 2024-02-01T10:34:10.644 DEBUG:boto:Params: {} 2024-02-01T10:34:10.644 DEBUG:boto:Token: None 2024-02-01T10:34:10.644 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:34:10 GMT /vclfcn-24/?versions 2024-02-01T10:34:10.644 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:buPW2SQ0OGzWO0Fj8LwH1clSbw8= 2024-02-01T10:34:10.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:34:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:buPW2SQ0OGzWO0Fj8LwH1clSbw8=', 'Content-Length': '0'} 2024-02-01T10:34:10.649 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f7284cd51103fde-0065bb73a2-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:34:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:34:10.649 DEBUG:boto:b'vclfcn-24999falseobjnulltrue2024-02-01T10:33:44.431Zrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:34:10.649 DEBUG:boto:path=/vclfcn-24/ 2024-02-01T10:34:10.650 DEBUG:boto:auth_path=/vclfcn-24/ 2024-02-01T10:34:10.650 DEBUG:boto:path=/vclfcn-24/?versions&max-keys=999 2024-02-01T10:34:10.650 DEBUG:boto:auth_path=/vclfcn-24/?versions&max-keys=999 2024-02-01T10:34:10.650 DEBUG:boto:Method: GET 2024-02-01T10:34:10.650 DEBUG:boto:Path: /vclfcn-24/?versions&max-keys=999 2024-02-01T10:34:10.650 DEBUG:boto:Data: 2024-02-01T10:34:10.650 DEBUG:boto:Headers: {} 2024-02-01T10:34:10.650 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:34:10.650 DEBUG:boto:Port: 8000 2024-02-01T10:34:10.650 DEBUG:boto:Params: {} 2024-02-01T10:34:10.650 DEBUG:boto:Token: None 2024-02-01T10:34:10.651 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:34:10 GMT /vclfcn-24/?versions 2024-02-01T10:34:10.651 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:buPW2SQ0OGzWO0Fj8LwH1clSbw8= 2024-02-01T10:34:10.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:34:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:buPW2SQ0OGzWO0Fj8LwH1clSbw8=', 'Content-Length': '0'} 2024-02-01T10:34:10.655 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000005fca58e03bfccd6-0065bb73a2-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:34:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:34:10.700 DEBUG:boto:b'vclfcn-24999falseobjnulltrue2024-02-01T10:33:44.431Zrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:34:10.701 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:34:10.701 DEBUG:boto:path=/vclfcn-24/obj 2024-02-01T10:34:10.701 DEBUG:boto:auth_path=/vclfcn-24/obj 2024-02-01T10:34:10.701 DEBUG:boto:Method: HEAD 2024-02-01T10:34:10.701 DEBUG:boto:Path: /vclfcn-24/obj 2024-02-01T10:34:10.701 DEBUG:boto:Data: 2024-02-01T10:34:10.701 DEBUG:boto:Headers: {} 2024-02-01T10:34:10.701 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:34:10.701 DEBUG:boto:Port: 8001 2024-02-01T10:34:10.701 DEBUG:boto:Params: {} 2024-02-01T10:34:10.702 DEBUG:boto:Token: None 2024-02-01T10:34:10.702 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:34:10 GMT /vclfcn-24/obj 2024-02-01T10:34:10.702 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NIWixTEwLLtcOd7z+2r5YWYjZ48= 2024-02-01T10:34:10.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:34:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NIWixTEwLLtcOd7z+2r5YWYjZ48=', 'Content-Length': '0'} 2024-02-01T10:34:10.705 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000004c5880ed5b90220f-0065bb73a2-4358-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:34:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:34:10.705 DEBUG:boto:path=/vclfcn-24/obj 2024-02-01T10:34:10.705 DEBUG:boto:auth_path=/vclfcn-24/obj 2024-02-01T10:34:10.705 DEBUG:boto:Method: HEAD 2024-02-01T10:34:10.705 DEBUG:boto:Path: /vclfcn-24/obj 2024-02-01T10:34:10.705 DEBUG:boto:Data: 2024-02-01T10:34:10.705 DEBUG:boto:Headers: {} 2024-02-01T10:34:10.705 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:34:10.706 DEBUG:boto:Port: 8000 2024-02-01T10:34:10.706 DEBUG:boto:Params: {} 2024-02-01T10:34:10.706 DEBUG:boto:Token: None 2024-02-01T10:34:10.706 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:34:10 GMT /vclfcn-24/obj 2024-02-01T10:34:10.706 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NIWixTEwLLtcOd7z+2r5YWYjZ48= 2024-02-01T10:34:10.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:34:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NIWixTEwLLtcOd7z+2r5YWYjZ48=', 'Content-Length': '0'} 2024-02-01T10:34:10.709 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000f3e0c8798593ebe9-0065bb73a2-4307-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:34:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:34:10.709 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-24 zones={test-zone3, test-zone2} 2024-02-01T10:34:10.709 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=vclfcn-25 2024-02-01T10:34:10.709 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:34:10.709 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:34:10.710 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket vclfcn-25 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-02-01T10:34:10.760 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:34:10.760 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:34:10.823 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:34:10.824+0000 7fb82e1c0980 1 bucket sync caught up with source: 2024-02-01T10:34:10.823 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, , , , , , , , 00000000001.47.14, , ] 2024-02-01T10:34:10.823 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, , , , , , , , 00000000001.47.14, , ] 2024-02-01T10:34:10.823 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:34:10.824+0000 7fb82e1c0980 0 bucket checkpoint complete 2024-02-01T10:34:10.847 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=vclfcn-25 2024-02-01T10:34:10.847 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:34:10.847 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:34:10.847 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-25 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:34:10.897 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:34:10.898 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:34:10.958 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:34:10.962+0000 7f7f4e5b0980 1 waiting to reach incremental sync.. 2024-02-01T10:34:15.962 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:34:15.970+0000 7f7f4e5b0980 1 bucket sync caught up with source: 2024-02-01T10:34:15.962 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, , , , , , , , 00000000001.47.14, , ] 2024-02-01T10:34:15.962 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, , , , , , , , 00000000001.47.14, , ] 2024-02-01T10:34:15.962 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:34:15.970+0000 7f7f4e5b0980 0 bucket checkpoint complete 2024-02-01T10:34:16.003 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=vclfcn-25 2024-02-01T10:34:16.004 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:34:16.004 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:34:16.005 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-25 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:34:16.053 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:34:16.053 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:34:16.117 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:34:16.122+0000 7f5c697f4980 1 bucket sync caught up with source: 2024-02-01T10:34:16.117 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, , , , , , , , 00000000003.91.7, , ] 2024-02-01T10:34:16.117 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, , , , , , , , 00000000003.91.7, , ] 2024-02-01T10:34:16.117 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:34:16.122+0000 7f5c697f4980 0 bucket checkpoint complete 2024-02-01T10:34:16.139 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=vclfcn-25 2024-02-01T10:34:16.139 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:34:16.140 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:34:16.140 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-25 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:34:16.181 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:34:16.181 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:34:16.241 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:34:16.246+0000 7fa0a5df0980 1 bucket sync caught up with source: 2024-02-01T10:34:16.241 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, , , , , , , , 00000000003.91.7, , ] 2024-02-01T10:34:16.241 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, , , , , , , , 00000000003.91.7, , ] 2024-02-01T10:34:16.241 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:34:16.246+0000 7fa0a5df0980 0 bucket checkpoint complete 2024-02-01T10:34:16.263 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=vclfcn-25 2024-02-01T10:34:16.263 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:34:16.263 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:34:16.264 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-25 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:34:16.352 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:34:16.352 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:34:16.411 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:34:16.418+0000 7fa6dc398980 1 bucket sync caught up with source: 2024-02-01T10:34:16.411 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, , , , , , , , 00000000001.110.14, , ] 2024-02-01T10:34:16.411 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, , , , , , , , 00000000001.110.14, , ] 2024-02-01T10:34:16.411 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:34:16.418+0000 7fa6dc398980 0 bucket checkpoint complete 2024-02-01T10:34:16.432 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=vclfcn-25 2024-02-01T10:34:16.432 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:34:16.433 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:34:16.433 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket vclfcn-25 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-02-01T10:34:16.484 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:34:16.484 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:34:16.545 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:34:16.548+0000 7f0ae9ce3980 1 waiting to reach incremental sync.. 2024-02-01T10:34:21.546 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:34:21.548+0000 7f0ae9ce3980 1 waiting to reach incremental sync.. 2024-02-01T10:34:26.547 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:34:26.548+0000 7f0ae9ce3980 1 waiting to reach incremental sync.. 2024-02-01T10:34:31.548 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:34:31.548+0000 7f0ae9ce3980 1 waiting to reach incremental sync.. 2024-02-01T10:34:36.550 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:34:36.552+0000 7f0ae9ce3980 1 waiting to reach incremental sync.. 2024-02-01T10:34:41.551 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:34:41.552+0000 7f0ae9ce3980 1 waiting to reach incremental sync.. 2024-02-01T10:34:46.552 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:34:46.551+0000 7f0ae9ce3980 1 waiting to reach incremental sync.. 2024-02-01T10:34:51.553 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:34:51.551+0000 7f0ae9ce3980 1 waiting to reach incremental sync.. 2024-02-01T10:34:56.554 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:34:56.555+0000 7f0ae9ce3980 1 waiting to reach incremental sync.. 2024-02-01T10:35:01.555 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:35:01.555+0000 7f0ae9ce3980 1 waiting to reach incremental sync.. 2024-02-01T10:35:06.556 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:35:06.555+0000 7f0ae9ce3980 1 waiting to reach incremental sync.. 2024-02-01T10:35:11.557 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:35:11.555+0000 7f0ae9ce3980 1 waiting to reach incremental sync.. 2024-02-01T10:35:16.558 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:35:16.559+0000 7f0ae9ce3980 1 waiting to reach incremental sync.. 2024-02-01T10:35:21.559 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:35:21.559+0000 7f0ae9ce3980 1 waiting to reach incremental sync.. 2024-02-01T10:35:26.561 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:35:26.559+0000 7f0ae9ce3980 1 waiting to reach incremental sync.. 2024-02-01T10:35:31.564 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:35:31.563+0000 7f0ae9ce3980 1 bucket sync caught up with source: 2024-02-01T10:35:31.564 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, , , , , , , , 00000000001.110.14, , ] 2024-02-01T10:35:31.565 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, , , , , , , , 00000000001.110.14, , ] 2024-02-01T10:35:31.565 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:35:31.563+0000 7f0ae9ce3980 0 bucket checkpoint complete 2024-02-01T10:35:31.608 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-25 zones={test-zone2, test-zone1} 2024-02-01T10:35:31.608 DEBUG:boto:path=/vclfcn-25/ 2024-02-01T10:35:31.608 DEBUG:boto:auth_path=/vclfcn-25/ 2024-02-01T10:35:31.608 DEBUG:boto:Method: HEAD 2024-02-01T10:35:31.609 DEBUG:boto:Path: /vclfcn-25/ 2024-02-01T10:35:31.609 DEBUG:boto:Data: 2024-02-01T10:35:31.609 DEBUG:boto:Headers: {} 2024-02-01T10:35:31.609 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:35:31.609 DEBUG:boto:Port: 8000 2024-02-01T10:35:31.609 DEBUG:boto:Params: {} 2024-02-01T10:35:31.609 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T10:35:31.610 DEBUG:boto:Token: None 2024-02-01T10:35:31.610 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:35:31 GMT /vclfcn-25/ 2024-02-01T10:35:31.610 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:joybZXTu5mtS9W3m6+iSnw9mUxc= 2024-02-01T10:35:31.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:31 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:joybZXTu5mtS9W3m6+iSnw9mUxc=', 'Content-Length': '0'} 2024-02-01T10:35:31.615 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a07ec6f1c475c81f-0065bb73f3-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:35:31 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:31.616 DEBUG:boto:path=/vclfcn-25/ 2024-02-01T10:35:31.616 DEBUG:boto:auth_path=/vclfcn-25/ 2024-02-01T10:35:31.616 DEBUG:boto:Method: HEAD 2024-02-01T10:35:31.616 DEBUG:boto:Path: /vclfcn-25/ 2024-02-01T10:35:31.616 DEBUG:boto:Data: 2024-02-01T10:35:31.616 DEBUG:boto:Headers: {} 2024-02-01T10:35:31.616 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:35:31.616 DEBUG:boto:Port: 8000 2024-02-01T10:35:31.616 DEBUG:boto:Params: {} 2024-02-01T10:35:31.617 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T10:35:31.617 DEBUG:boto:Token: None 2024-02-01T10:35:31.617 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:35:31 GMT /vclfcn-25/ 2024-02-01T10:35:31.617 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:joybZXTu5mtS9W3m6+iSnw9mUxc= 2024-02-01T10:35:31.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:31 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:joybZXTu5mtS9W3m6+iSnw9mUxc=', 'Content-Length': '0'} 2024-02-01T10:35:31.623 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008b895f4d217c4ded-0065bb73f3-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:35:31 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:31.623 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:35:31.623 DEBUG:boto:path=/vclfcn-25/ 2024-02-01T10:35:31.623 DEBUG:boto:auth_path=/vclfcn-25/ 2024-02-01T10:35:31.623 DEBUG:boto:path=/vclfcn-25/?versions&max-keys=999 2024-02-01T10:35:31.623 DEBUG:boto:auth_path=/vclfcn-25/?versions&max-keys=999 2024-02-01T10:35:31.623 DEBUG:boto:Method: GET 2024-02-01T10:35:31.624 DEBUG:boto:Path: /vclfcn-25/?versions&max-keys=999 2024-02-01T10:35:31.624 DEBUG:boto:Data: 2024-02-01T10:35:31.624 DEBUG:boto:Headers: {} 2024-02-01T10:35:31.624 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:35:31.624 DEBUG:boto:Port: 8000 2024-02-01T10:35:31.624 DEBUG:boto:Params: {} 2024-02-01T10:35:31.624 DEBUG:boto:Token: None 2024-02-01T10:35:31.624 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:35:31 GMT /vclfcn-25/?versions 2024-02-01T10:35:31.624 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:B2cwUhGKSS1a/WLjXNX2BD0cBM4= 2024-02-01T10:35:31.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:31 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:B2cwUhGKSS1a/WLjXNX2BD0cBM4=', 'Content-Length': '0'} 2024-02-01T10:35:31.627 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001595f05777549795-0065bb73f3-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:35:31 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:31.628 DEBUG:boto:b'vclfcn-25999falseobjnulltrue2024-02-01T10:33:44.465Zrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:35:31.628 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:35:31.628 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:35:31.628 DEBUG:boto:path=/vclfcn-25/ 2024-02-01T10:35:31.629 DEBUG:boto:auth_path=/vclfcn-25/ 2024-02-01T10:35:31.629 DEBUG:boto:path=/vclfcn-25/?versions&max-keys=999 2024-02-01T10:35:31.629 DEBUG:boto:auth_path=/vclfcn-25/?versions&max-keys=999 2024-02-01T10:35:31.629 DEBUG:boto:Method: GET 2024-02-01T10:35:31.629 DEBUG:boto:Path: /vclfcn-25/?versions&max-keys=999 2024-02-01T10:35:31.629 DEBUG:boto:Data: 2024-02-01T10:35:31.629 DEBUG:boto:Headers: {} 2024-02-01T10:35:31.629 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:35:31.629 DEBUG:boto:Port: 8000 2024-02-01T10:35:31.629 DEBUG:boto:Params: {} 2024-02-01T10:35:31.629 DEBUG:boto:Token: None 2024-02-01T10:35:31.630 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:35:31 GMT /vclfcn-25/?versions 2024-02-01T10:35:31.630 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:B2cwUhGKSS1a/WLjXNX2BD0cBM4= 2024-02-01T10:35:31.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:31 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:B2cwUhGKSS1a/WLjXNX2BD0cBM4=', 'Content-Length': '0'} 2024-02-01T10:35:31.634 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a181b1ef8d8db095-0065bb73f3-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:35:31 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:31.634 DEBUG:boto:b'vclfcn-25999falseobjnulltrue2024-02-01T10:33:44.465Zrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:35:31.634 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:35:31.634 DEBUG:boto:path=/vclfcn-25/ 2024-02-01T10:35:31.635 DEBUG:boto:auth_path=/vclfcn-25/ 2024-02-01T10:35:31.635 DEBUG:boto:path=/vclfcn-25/?versions&max-keys=999 2024-02-01T10:35:31.635 DEBUG:boto:auth_path=/vclfcn-25/?versions&max-keys=999 2024-02-01T10:35:31.635 DEBUG:boto:Method: GET 2024-02-01T10:35:31.635 DEBUG:boto:Path: /vclfcn-25/?versions&max-keys=999 2024-02-01T10:35:31.635 DEBUG:boto:Data: 2024-02-01T10:35:31.635 DEBUG:boto:Headers: {} 2024-02-01T10:35:31.635 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:35:31.635 DEBUG:boto:Port: 8000 2024-02-01T10:35:31.636 DEBUG:boto:Params: {} 2024-02-01T10:35:31.636 DEBUG:boto:Token: None 2024-02-01T10:35:31.636 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:35:31 GMT /vclfcn-25/?versions 2024-02-01T10:35:31.636 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:B2cwUhGKSS1a/WLjXNX2BD0cBM4= 2024-02-01T10:35:31.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:31 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:B2cwUhGKSS1a/WLjXNX2BD0cBM4=', 'Content-Length': '0'} 2024-02-01T10:35:31.639 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a6ea3f181e1be22-0065bb73f3-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:35:31 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:31.639 DEBUG:boto:b'vclfcn-25999falseobjnulltrue2024-02-01T10:33:44.465Zrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:35:31.640 DEBUG:boto:path=/vclfcn-25/ 2024-02-01T10:35:31.640 DEBUG:boto:auth_path=/vclfcn-25/ 2024-02-01T10:35:31.640 DEBUG:boto:path=/vclfcn-25/?versions&max-keys=999 2024-02-01T10:35:31.640 DEBUG:boto:auth_path=/vclfcn-25/?versions&max-keys=999 2024-02-01T10:35:31.640 DEBUG:boto:Method: GET 2024-02-01T10:35:31.640 DEBUG:boto:Path: /vclfcn-25/?versions&max-keys=999 2024-02-01T10:35:31.640 DEBUG:boto:Data: 2024-02-01T10:35:31.641 DEBUG:boto:Headers: {} 2024-02-01T10:35:31.641 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:35:31.641 DEBUG:boto:Port: 8000 2024-02-01T10:35:31.641 DEBUG:boto:Params: {} 2024-02-01T10:35:31.641 DEBUG:boto:Token: None 2024-02-01T10:35:31.641 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:35:31 GMT /vclfcn-25/?versions 2024-02-01T10:35:31.641 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:B2cwUhGKSS1a/WLjXNX2BD0cBM4= 2024-02-01T10:35:31.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:31 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:B2cwUhGKSS1a/WLjXNX2BD0cBM4=', 'Content-Length': '0'} 2024-02-01T10:35:31.645 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d5c4e6d18d1c874-0065bb73f3-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:35:31 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:31.646 DEBUG:boto:b'vclfcn-25999falseobjnulltrue2024-02-01T10:33:44.465Zrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:35:31.646 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:35:31.646 DEBUG:boto:path=/vclfcn-25/obj 2024-02-01T10:35:31.646 DEBUG:boto:auth_path=/vclfcn-25/obj 2024-02-01T10:35:31.646 DEBUG:boto:Method: HEAD 2024-02-01T10:35:31.646 DEBUG:boto:Path: /vclfcn-25/obj 2024-02-01T10:35:31.646 DEBUG:boto:Data: 2024-02-01T10:35:31.647 DEBUG:boto:Headers: {} 2024-02-01T10:35:31.647 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:35:31.647 DEBUG:boto:Port: 8000 2024-02-01T10:35:31.647 DEBUG:boto:Params: {} 2024-02-01T10:35:31.647 DEBUG:boto:Token: None 2024-02-01T10:35:31.647 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:35:31 GMT /vclfcn-25/obj 2024-02-01T10:35:31.647 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:K/tttt+bGlWA8kJTSbN3DiEYmxU= 2024-02-01T10:35:31.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:31 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:K/tttt+bGlWA8kJTSbN3DiEYmxU=', 'Content-Length': '0'} 2024-02-01T10:35:31.649 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000008f584263197e3831-0065bb73f3-4307-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:35:31 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:31.649 DEBUG:boto:path=/vclfcn-25/obj 2024-02-01T10:35:31.649 DEBUG:boto:auth_path=/vclfcn-25/obj 2024-02-01T10:35:31.649 DEBUG:boto:Method: HEAD 2024-02-01T10:35:31.649 DEBUG:boto:Path: /vclfcn-25/obj 2024-02-01T10:35:31.649 DEBUG:boto:Data: 2024-02-01T10:35:31.649 DEBUG:boto:Headers: {} 2024-02-01T10:35:31.650 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:35:31.650 DEBUG:boto:Port: 8000 2024-02-01T10:35:31.650 DEBUG:boto:Params: {} 2024-02-01T10:35:31.650 DEBUG:boto:Token: None 2024-02-01T10:35:31.650 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:35:31 GMT /vclfcn-25/obj 2024-02-01T10:35:31.650 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:K/tttt+bGlWA8kJTSbN3DiEYmxU= 2024-02-01T10:35:31.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:31 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:K/tttt+bGlWA8kJTSbN3DiEYmxU=', 'Content-Length': '0'} 2024-02-01T10:35:31.652 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000ebf6a5b2ec1a42ea-0065bb73f3-4351-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:35:31 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:31.653 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-25 zones={test-zone2, test-zone1} 2024-02-01T10:35:31.653 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-25 zones={test-zone3, test-zone1} 2024-02-01T10:35:31.653 DEBUG:boto:path=/vclfcn-25/ 2024-02-01T10:35:31.653 DEBUG:boto:auth_path=/vclfcn-25/ 2024-02-01T10:35:31.653 DEBUG:boto:Method: HEAD 2024-02-01T10:35:31.653 DEBUG:boto:Path: /vclfcn-25/ 2024-02-01T10:35:31.653 DEBUG:boto:Data: 2024-02-01T10:35:31.653 DEBUG:boto:Headers: {} 2024-02-01T10:35:31.653 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:35:31.654 DEBUG:boto:Port: 8001 2024-02-01T10:35:31.654 DEBUG:boto:Params: {} 2024-02-01T10:35:31.654 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-02-01T10:35:31.654 DEBUG:boto:Token: None 2024-02-01T10:35:31.654 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:35:31 GMT /vclfcn-25/ 2024-02-01T10:35:31.654 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:joybZXTu5mtS9W3m6+iSnw9mUxc= 2024-02-01T10:35:31.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:31 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:joybZXTu5mtS9W3m6+iSnw9mUxc=', 'Content-Length': '0'} 2024-02-01T10:35:31.660 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000342b72b1504386c-0065bb73f3-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:35:31 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:31.660 DEBUG:boto:path=/vclfcn-25/ 2024-02-01T10:35:31.660 DEBUG:boto:auth_path=/vclfcn-25/ 2024-02-01T10:35:31.661 DEBUG:boto:Method: HEAD 2024-02-01T10:35:31.661 DEBUG:boto:Path: /vclfcn-25/ 2024-02-01T10:35:31.661 DEBUG:boto:Data: 2024-02-01T10:35:31.661 DEBUG:boto:Headers: {} 2024-02-01T10:35:31.661 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:35:31.661 DEBUG:boto:Port: 8000 2024-02-01T10:35:31.661 DEBUG:boto:Params: {} 2024-02-01T10:35:31.661 DEBUG:boto:Token: None 2024-02-01T10:35:31.661 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:35:31 GMT /vclfcn-25/ 2024-02-01T10:35:31.662 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:joybZXTu5mtS9W3m6+iSnw9mUxc= 2024-02-01T10:35:31.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:31 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:joybZXTu5mtS9W3m6+iSnw9mUxc=', 'Content-Length': '0'} 2024-02-01T10:35:31.665 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ed5ed3caa3cd9da6-0065bb73f3-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:35:31 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:31.665 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:35:31.665 DEBUG:boto:path=/vclfcn-25/ 2024-02-01T10:35:31.666 DEBUG:boto:auth_path=/vclfcn-25/ 2024-02-01T10:35:31.666 DEBUG:boto:path=/vclfcn-25/?versions&max-keys=999 2024-02-01T10:35:31.666 DEBUG:boto:auth_path=/vclfcn-25/?versions&max-keys=999 2024-02-01T10:35:31.666 DEBUG:boto:Method: GET 2024-02-01T10:35:31.666 DEBUG:boto:Path: /vclfcn-25/?versions&max-keys=999 2024-02-01T10:35:31.666 DEBUG:boto:Data: 2024-02-01T10:35:31.666 DEBUG:boto:Headers: {} 2024-02-01T10:35:31.666 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:35:31.666 DEBUG:boto:Port: 8001 2024-02-01T10:35:31.666 DEBUG:boto:Params: {} 2024-02-01T10:35:31.666 DEBUG:boto:Token: None 2024-02-01T10:35:31.667 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:35:31 GMT /vclfcn-25/?versions 2024-02-01T10:35:31.667 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:B2cwUhGKSS1a/WLjXNX2BD0cBM4= 2024-02-01T10:35:31.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:31 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:B2cwUhGKSS1a/WLjXNX2BD0cBM4=', 'Content-Length': '0'} 2024-02-01T10:35:31.671 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002fa42336c3780710-0065bb73f3-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:35:31 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:31.671 DEBUG:boto:b'vclfcn-25999falseobjnulltrue2024-02-01T10:33:44.465Zrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:35:31.671 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:35:31.671 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:35:31.672 DEBUG:boto:path=/vclfcn-25/ 2024-02-01T10:35:31.672 DEBUG:boto:auth_path=/vclfcn-25/ 2024-02-01T10:35:31.673 DEBUG:boto:path=/vclfcn-25/?versions&max-keys=999 2024-02-01T10:35:31.673 DEBUG:boto:auth_path=/vclfcn-25/?versions&max-keys=999 2024-02-01T10:35:31.673 DEBUG:boto:Method: GET 2024-02-01T10:35:31.673 DEBUG:boto:Path: /vclfcn-25/?versions&max-keys=999 2024-02-01T10:35:31.673 DEBUG:boto:Data: 2024-02-01T10:35:31.673 DEBUG:boto:Headers: {} 2024-02-01T10:35:31.673 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:35:31.673 DEBUG:boto:Port: 8000 2024-02-01T10:35:31.673 DEBUG:boto:Params: {} 2024-02-01T10:35:31.673 DEBUG:boto:Token: None 2024-02-01T10:35:31.674 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:35:31 GMT /vclfcn-25/?versions 2024-02-01T10:35:31.674 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:B2cwUhGKSS1a/WLjXNX2BD0cBM4= 2024-02-01T10:35:31.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:31 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:B2cwUhGKSS1a/WLjXNX2BD0cBM4=', 'Content-Length': '0'} 2024-02-01T10:35:31.678 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc3639f22757ac70-0065bb73f3-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:35:31 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:31.724 DEBUG:boto:b'vclfcn-25999falseobjnulltrue2024-02-01T10:33:44.465Zrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:35:31.725 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:35:31.725 DEBUG:boto:path=/vclfcn-25/ 2024-02-01T10:35:31.725 DEBUG:boto:auth_path=/vclfcn-25/ 2024-02-01T10:35:31.725 DEBUG:boto:path=/vclfcn-25/?versions&max-keys=999 2024-02-01T10:35:31.725 DEBUG:boto:auth_path=/vclfcn-25/?versions&max-keys=999 2024-02-01T10:35:31.725 DEBUG:boto:Method: GET 2024-02-01T10:35:31.725 DEBUG:boto:Path: /vclfcn-25/?versions&max-keys=999 2024-02-01T10:35:31.725 DEBUG:boto:Data: 2024-02-01T10:35:31.725 DEBUG:boto:Headers: {} 2024-02-01T10:35:31.725 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:35:31.726 DEBUG:boto:Port: 8001 2024-02-01T10:35:31.726 DEBUG:boto:Params: {} 2024-02-01T10:35:31.726 DEBUG:boto:Token: None 2024-02-01T10:35:31.726 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:35:31 GMT /vclfcn-25/?versions 2024-02-01T10:35:31.726 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:B2cwUhGKSS1a/WLjXNX2BD0cBM4= 2024-02-01T10:35:31.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:31 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:B2cwUhGKSS1a/WLjXNX2BD0cBM4=', 'Content-Length': '0'} 2024-02-01T10:35:31.731 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000277c050be06f039e-0065bb73f3-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:35:31 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:31.731 DEBUG:boto:b'vclfcn-25999falseobjnulltrue2024-02-01T10:33:44.465Zrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:35:31.732 DEBUG:boto:path=/vclfcn-25/ 2024-02-01T10:35:31.732 DEBUG:boto:auth_path=/vclfcn-25/ 2024-02-01T10:35:31.732 DEBUG:boto:path=/vclfcn-25/?versions&max-keys=999 2024-02-01T10:35:31.732 DEBUG:boto:auth_path=/vclfcn-25/?versions&max-keys=999 2024-02-01T10:35:31.732 DEBUG:boto:Method: GET 2024-02-01T10:35:31.732 DEBUG:boto:Path: /vclfcn-25/?versions&max-keys=999 2024-02-01T10:35:31.732 DEBUG:boto:Data: 2024-02-01T10:35:31.732 DEBUG:boto:Headers: {} 2024-02-01T10:35:31.732 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:35:31.732 DEBUG:boto:Port: 8000 2024-02-01T10:35:31.733 DEBUG:boto:Params: {} 2024-02-01T10:35:31.733 DEBUG:boto:Token: None 2024-02-01T10:35:31.733 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:35:31 GMT /vclfcn-25/?versions 2024-02-01T10:35:31.733 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:B2cwUhGKSS1a/WLjXNX2BD0cBM4= 2024-02-01T10:35:31.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:31 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:B2cwUhGKSS1a/WLjXNX2BD0cBM4=', 'Content-Length': '0'} 2024-02-01T10:35:31.738 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e06efaeeaa804cc9-0065bb73f3-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:35:31 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:31.738 DEBUG:boto:b'vclfcn-25999falseobjnulltrue2024-02-01T10:33:44.465Zrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:35:31.739 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:35:31.739 DEBUG:boto:path=/vclfcn-25/obj 2024-02-01T10:35:31.739 DEBUG:boto:auth_path=/vclfcn-25/obj 2024-02-01T10:35:31.739 DEBUG:boto:Method: HEAD 2024-02-01T10:35:31.739 DEBUG:boto:Path: /vclfcn-25/obj 2024-02-01T10:35:31.739 DEBUG:boto:Data: 2024-02-01T10:35:31.740 DEBUG:boto:Headers: {} 2024-02-01T10:35:31.740 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:35:31.740 DEBUG:boto:Port: 8001 2024-02-01T10:35:31.740 DEBUG:boto:Params: {} 2024-02-01T10:35:31.740 DEBUG:boto:Token: None 2024-02-01T10:35:31.740 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:35:31 GMT /vclfcn-25/obj 2024-02-01T10:35:31.740 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:K/tttt+bGlWA8kJTSbN3DiEYmxU= 2024-02-01T10:35:31.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:31 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:K/tttt+bGlWA8kJTSbN3DiEYmxU=', 'Content-Length': '0'} 2024-02-01T10:35:31.743 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx0000073079e16a61af43c-0065bb73f3-4358-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:35:31 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:31.743 DEBUG:boto:path=/vclfcn-25/obj 2024-02-01T10:35:31.743 DEBUG:boto:auth_path=/vclfcn-25/obj 2024-02-01T10:35:31.743 DEBUG:boto:Method: HEAD 2024-02-01T10:35:31.743 DEBUG:boto:Path: /vclfcn-25/obj 2024-02-01T10:35:31.743 DEBUG:boto:Data: 2024-02-01T10:35:31.743 DEBUG:boto:Headers: {} 2024-02-01T10:35:31.744 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:35:31.744 DEBUG:boto:Port: 8000 2024-02-01T10:35:31.744 DEBUG:boto:Params: {} 2024-02-01T10:35:31.744 DEBUG:boto:Token: None 2024-02-01T10:35:31.744 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:35:31 GMT /vclfcn-25/obj 2024-02-01T10:35:31.744 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:K/tttt+bGlWA8kJTSbN3DiEYmxU= 2024-02-01T10:35:31.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:31 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:K/tttt+bGlWA8kJTSbN3DiEYmxU=', 'Content-Length': '0'} 2024-02-01T10:35:31.746 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000aadc586124edeb7c-0065bb73f3-4351-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:35:31 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:31.747 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-25 zones={test-zone3, test-zone1} 2024-02-01T10:35:31.747 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-25 zones={test-zone3, test-zone2} 2024-02-01T10:35:31.747 DEBUG:boto:path=/vclfcn-25/ 2024-02-01T10:35:31.747 DEBUG:boto:auth_path=/vclfcn-25/ 2024-02-01T10:35:31.747 DEBUG:boto:Method: HEAD 2024-02-01T10:35:31.747 DEBUG:boto:Path: /vclfcn-25/ 2024-02-01T10:35:31.747 DEBUG:boto:Data: 2024-02-01T10:35:31.747 DEBUG:boto:Headers: {} 2024-02-01T10:35:31.748 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:35:31.748 DEBUG:boto:Port: 8001 2024-02-01T10:35:31.748 DEBUG:boto:Params: {} 2024-02-01T10:35:31.748 DEBUG:boto:Token: None 2024-02-01T10:35:31.748 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:35:31 GMT /vclfcn-25/ 2024-02-01T10:35:31.748 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:joybZXTu5mtS9W3m6+iSnw9mUxc= 2024-02-01T10:35:31.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:31 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:joybZXTu5mtS9W3m6+iSnw9mUxc=', 'Content-Length': '0'} 2024-02-01T10:35:31.752 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000046db1a2b99bb2259-0065bb73f3-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:35:31 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:31.752 DEBUG:boto:path=/vclfcn-25/ 2024-02-01T10:35:31.752 DEBUG:boto:auth_path=/vclfcn-25/ 2024-02-01T10:35:31.753 DEBUG:boto:Method: HEAD 2024-02-01T10:35:31.753 DEBUG:boto:Path: /vclfcn-25/ 2024-02-01T10:35:31.753 DEBUG:boto:Data: 2024-02-01T10:35:31.753 DEBUG:boto:Headers: {} 2024-02-01T10:35:31.753 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:35:31.753 DEBUG:boto:Port: 8000 2024-02-01T10:35:31.753 DEBUG:boto:Params: {} 2024-02-01T10:35:31.753 DEBUG:boto:Token: None 2024-02-01T10:35:31.753 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:35:31 GMT /vclfcn-25/ 2024-02-01T10:35:31.754 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:joybZXTu5mtS9W3m6+iSnw9mUxc= 2024-02-01T10:35:31.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:31 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:joybZXTu5mtS9W3m6+iSnw9mUxc=', 'Content-Length': '0'} 2024-02-01T10:35:31.757 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a4129afaf2b1f30a-0065bb73f3-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:35:31 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:31.758 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:35:31.758 DEBUG:boto:path=/vclfcn-25/ 2024-02-01T10:35:31.758 DEBUG:boto:auth_path=/vclfcn-25/ 2024-02-01T10:35:31.758 DEBUG:boto:path=/vclfcn-25/?versions&max-keys=999 2024-02-01T10:35:31.758 DEBUG:boto:auth_path=/vclfcn-25/?versions&max-keys=999 2024-02-01T10:35:31.758 DEBUG:boto:Method: GET 2024-02-01T10:35:31.758 DEBUG:boto:Path: /vclfcn-25/?versions&max-keys=999 2024-02-01T10:35:31.758 DEBUG:boto:Data: 2024-02-01T10:35:31.758 DEBUG:boto:Headers: {} 2024-02-01T10:35:31.759 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:35:31.759 DEBUG:boto:Port: 8001 2024-02-01T10:35:31.759 DEBUG:boto:Params: {} 2024-02-01T10:35:31.759 DEBUG:boto:Token: None 2024-02-01T10:35:31.760 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:35:31 GMT /vclfcn-25/?versions 2024-02-01T10:35:31.760 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:B2cwUhGKSS1a/WLjXNX2BD0cBM4= 2024-02-01T10:35:31.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:31 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:B2cwUhGKSS1a/WLjXNX2BD0cBM4=', 'Content-Length': '0'} 2024-02-01T10:35:31.764 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab6b359d696fd50c-0065bb73f3-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:35:31 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:31.808 DEBUG:boto:b'vclfcn-25999falseobjnulltrue2024-02-01T10:33:44.465Zrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:35:31.809 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:35:31.809 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:35:31.809 DEBUG:boto:path=/vclfcn-25/ 2024-02-01T10:35:31.809 DEBUG:boto:auth_path=/vclfcn-25/ 2024-02-01T10:35:31.809 DEBUG:boto:path=/vclfcn-25/?versions&max-keys=999 2024-02-01T10:35:31.809 DEBUG:boto:auth_path=/vclfcn-25/?versions&max-keys=999 2024-02-01T10:35:31.809 DEBUG:boto:Method: GET 2024-02-01T10:35:31.810 DEBUG:boto:Path: /vclfcn-25/?versions&max-keys=999 2024-02-01T10:35:31.810 DEBUG:boto:Data: 2024-02-01T10:35:31.810 DEBUG:boto:Headers: {} 2024-02-01T10:35:31.810 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:35:31.810 DEBUG:boto:Port: 8000 2024-02-01T10:35:31.810 DEBUG:boto:Params: {} 2024-02-01T10:35:31.811 DEBUG:boto:Token: None 2024-02-01T10:35:31.811 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:35:31 GMT /vclfcn-25/?versions 2024-02-01T10:35:31.811 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:B2cwUhGKSS1a/WLjXNX2BD0cBM4= 2024-02-01T10:35:31.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:31 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:B2cwUhGKSS1a/WLjXNX2BD0cBM4=', 'Content-Length': '0'} 2024-02-01T10:35:31.816 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054c8774316fb1fed-0065bb73f3-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:35:31 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:31.816 DEBUG:boto:b'vclfcn-25999falseobjnulltrue2024-02-01T10:33:44.465Zrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:35:31.816 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:35:31.816 DEBUG:boto:path=/vclfcn-25/ 2024-02-01T10:35:31.816 DEBUG:boto:auth_path=/vclfcn-25/ 2024-02-01T10:35:31.817 DEBUG:boto:path=/vclfcn-25/?versions&max-keys=999 2024-02-01T10:35:31.817 DEBUG:boto:auth_path=/vclfcn-25/?versions&max-keys=999 2024-02-01T10:35:31.817 DEBUG:boto:Method: GET 2024-02-01T10:35:31.817 DEBUG:boto:Path: /vclfcn-25/?versions&max-keys=999 2024-02-01T10:35:31.817 DEBUG:boto:Data: 2024-02-01T10:35:31.817 DEBUG:boto:Headers: {} 2024-02-01T10:35:31.817 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:35:31.817 DEBUG:boto:Port: 8001 2024-02-01T10:35:31.817 DEBUG:boto:Params: {} 2024-02-01T10:35:31.818 DEBUG:boto:Token: None 2024-02-01T10:35:31.818 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:35:31 GMT /vclfcn-25/?versions 2024-02-01T10:35:31.818 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:B2cwUhGKSS1a/WLjXNX2BD0cBM4= 2024-02-01T10:35:31.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:31 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:B2cwUhGKSS1a/WLjXNX2BD0cBM4=', 'Content-Length': '0'} 2024-02-01T10:35:31.822 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d43f3203b0f4f28-0065bb73f3-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:35:31 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:31.822 DEBUG:boto:b'vclfcn-25999falseobjnulltrue2024-02-01T10:33:44.465Zrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:35:31.823 DEBUG:boto:path=/vclfcn-25/ 2024-02-01T10:35:31.823 DEBUG:boto:auth_path=/vclfcn-25/ 2024-02-01T10:35:31.823 DEBUG:boto:path=/vclfcn-25/?versions&max-keys=999 2024-02-01T10:35:31.823 DEBUG:boto:auth_path=/vclfcn-25/?versions&max-keys=999 2024-02-01T10:35:31.823 DEBUG:boto:Method: GET 2024-02-01T10:35:31.823 DEBUG:boto:Path: /vclfcn-25/?versions&max-keys=999 2024-02-01T10:35:31.823 DEBUG:boto:Data: 2024-02-01T10:35:31.823 DEBUG:boto:Headers: {} 2024-02-01T10:35:31.823 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:35:31.824 DEBUG:boto:Port: 8000 2024-02-01T10:35:31.824 DEBUG:boto:Params: {} 2024-02-01T10:35:31.824 DEBUG:boto:Token: None 2024-02-01T10:35:31.824 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:35:31 GMT /vclfcn-25/?versions 2024-02-01T10:35:31.824 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:B2cwUhGKSS1a/WLjXNX2BD0cBM4= 2024-02-01T10:35:31.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:31 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:B2cwUhGKSS1a/WLjXNX2BD0cBM4=', 'Content-Length': '0'} 2024-02-01T10:35:31.828 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042dbedb35e3699ea-0065bb73f3-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:35:31 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:31.828 DEBUG:boto:b'vclfcn-25999falseobjnulltrue2024-02-01T10:33:44.465Zrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:35:31.829 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:35:31.829 DEBUG:boto:path=/vclfcn-25/obj 2024-02-01T10:35:31.829 DEBUG:boto:auth_path=/vclfcn-25/obj 2024-02-01T10:35:31.829 DEBUG:boto:Method: HEAD 2024-02-01T10:35:31.829 DEBUG:boto:Path: /vclfcn-25/obj 2024-02-01T10:35:31.829 DEBUG:boto:Data: 2024-02-01T10:35:31.830 DEBUG:boto:Headers: {} 2024-02-01T10:35:31.830 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:35:31.830 DEBUG:boto:Port: 8001 2024-02-01T10:35:31.830 DEBUG:boto:Params: {} 2024-02-01T10:35:31.830 DEBUG:boto:Token: None 2024-02-01T10:35:31.830 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:35:31 GMT /vclfcn-25/obj 2024-02-01T10:35:31.830 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:K/tttt+bGlWA8kJTSbN3DiEYmxU= 2024-02-01T10:35:31.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:31 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:K/tttt+bGlWA8kJTSbN3DiEYmxU=', 'Content-Length': '0'} 2024-02-01T10:35:31.833 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000bfb34992a87c4f51-0065bb73f3-4358-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:35:31 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:31.833 DEBUG:boto:path=/vclfcn-25/obj 2024-02-01T10:35:31.833 DEBUG:boto:auth_path=/vclfcn-25/obj 2024-02-01T10:35:31.833 DEBUG:boto:Method: HEAD 2024-02-01T10:35:31.833 DEBUG:boto:Path: /vclfcn-25/obj 2024-02-01T10:35:31.833 DEBUG:boto:Data: 2024-02-01T10:35:31.834 DEBUG:boto:Headers: {} 2024-02-01T10:35:31.834 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:35:31.834 DEBUG:boto:Port: 8000 2024-02-01T10:35:31.834 DEBUG:boto:Params: {} 2024-02-01T10:35:31.834 DEBUG:boto:Token: None 2024-02-01T10:35:31.834 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:35:31 GMT /vclfcn-25/obj 2024-02-01T10:35:31.834 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:K/tttt+bGlWA8kJTSbN3DiEYmxU= 2024-02-01T10:35:31.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:31 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:K/tttt+bGlWA8kJTSbN3DiEYmxU=', 'Content-Length': '0'} 2024-02-01T10:35:31.837 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000e26b5ec7648cf4e7-0065bb73f3-4307-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:35:31 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:31.837 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-25 zones={test-zone3, test-zone2} 2024-02-01T10:35:31.837 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=vclfcn-26 2024-02-01T10:35:31.837 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:35:31.838 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:35:31.838 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket vclfcn-26 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-02-01T10:35:31.888 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:35:31.888 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:35:31.951 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:35:31.951+0000 7fc05abb5980 1 bucket sync caught up with source: 2024-02-01T10:35:31.951 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, , , , , , , , 00000000001.15.14, , ] 2024-02-01T10:35:31.951 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, , , , , , , , 00000000001.15.14, , ] 2024-02-01T10:35:31.952 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:35:31.951+0000 7fc05abb5980 0 bucket checkpoint complete 2024-02-01T10:35:31.973 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=vclfcn-26 2024-02-01T10:35:31.974 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:35:31.974 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:35:31.974 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-26 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:35:32.023 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:35:32.024 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:35:32.085 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:35:32.088+0000 7f123893b980 1 bucket sync caught up with source: 2024-02-01T10:35:32.085 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, , , , , , , , 00000000001.15.14, , ] 2024-02-01T10:35:32.085 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, , , , , , , , 00000000001.15.14, , ] 2024-02-01T10:35:32.085 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:35:32.088+0000 7f123893b980 0 bucket checkpoint complete 2024-02-01T10:35:32.107 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=vclfcn-26 2024-02-01T10:35:32.107 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:35:32.107 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:35:32.107 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-26 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:35:32.148 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:35:32.148 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:35:32.208 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:35:32.212+0000 7f165f505980 1 bucket sync caught up with source: 2024-02-01T10:35:32.209 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, , , , , , , , 00000000001.51.14, , ] 2024-02-01T10:35:32.209 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, , , , , , , , 00000000001.51.14, , ] 2024-02-01T10:35:32.209 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:35:32.212+0000 7f165f505980 0 bucket checkpoint complete 2024-02-01T10:35:32.230 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=vclfcn-26 2024-02-01T10:35:32.230 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:35:32.231 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:35:32.231 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-26 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:35:32.320 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:35:32.320 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:35:32.379 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:35:32.384+0000 7f8825405980 1 bucket sync caught up with source: 2024-02-01T10:35:32.380 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, , , , , , , , 00000000001.51.14, , ] 2024-02-01T10:35:32.380 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, , , , , , , , 00000000001.51.14, , ] 2024-02-01T10:35:32.380 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:35:32.384+0000 7f8825405980 0 bucket checkpoint complete 2024-02-01T10:35:32.402 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=vclfcn-26 2024-02-01T10:35:32.402 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:35:32.402 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:35:32.403 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-26 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:35:32.443 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:35:32.443 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:35:32.500 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:35:32.504+0000 7f5e5ed54980 1 bucket sync caught up with source: 2024-02-01T10:35:32.501 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [, , , , , , , , 00000000003.81.7, , ] 2024-02-01T10:35:32.501 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [, , , , , , , , 00000000003.81.7, , ] 2024-02-01T10:35:32.501 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:35:32.504+0000 7f5e5ed54980 0 bucket checkpoint complete 2024-02-01T10:35:32.522 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=vclfcn-26 2024-02-01T10:35:32.523 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:35:32.523 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:35:32.523 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket vclfcn-26 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-02-01T10:35:32.573 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:35:32.573 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:35:32.637 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:35:32.635+0000 7f8a25ee8980 1 bucket sync caught up with source: 2024-02-01T10:35:32.637 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [, , , , , , , , 00000000003.81.7, , ] 2024-02-01T10:35:32.637 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [, , , , , , , , 00000000003.81.7, , ] 2024-02-01T10:35:32.637 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:35:32.635+0000 7f8a25ee8980 0 bucket checkpoint complete 2024-02-01T10:35:32.664 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-26 zones={test-zone2, test-zone1} 2024-02-01T10:35:32.664 DEBUG:boto:path=/vclfcn-26/ 2024-02-01T10:35:32.664 DEBUG:boto:auth_path=/vclfcn-26/ 2024-02-01T10:35:32.665 DEBUG:boto:Method: HEAD 2024-02-01T10:35:32.665 DEBUG:boto:Path: /vclfcn-26/ 2024-02-01T10:35:32.665 DEBUG:boto:Data: 2024-02-01T10:35:32.665 DEBUG:boto:Headers: {} 2024-02-01T10:35:32.665 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:35:32.665 DEBUG:boto:Port: 8000 2024-02-01T10:35:32.665 DEBUG:boto:Params: {} 2024-02-01T10:35:32.665 DEBUG:boto:Token: None 2024-02-01T10:35:32.665 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:35:32 GMT /vclfcn-26/ 2024-02-01T10:35:32.665 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cJtK3ODPFpOlhCgcd/G+Rtwewlg= 2024-02-01T10:35:32.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cJtK3ODPFpOlhCgcd/G+Rtwewlg=', 'Content-Length': '0'} 2024-02-01T10:35:32.668 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002eec10a7dcc37bfd-0065bb73f4-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:35:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:32.668 DEBUG:boto:path=/vclfcn-26/ 2024-02-01T10:35:32.668 DEBUG:boto:auth_path=/vclfcn-26/ 2024-02-01T10:35:32.669 DEBUG:boto:Method: HEAD 2024-02-01T10:35:32.669 DEBUG:boto:Path: /vclfcn-26/ 2024-02-01T10:35:32.669 DEBUG:boto:Data: 2024-02-01T10:35:32.669 DEBUG:boto:Headers: {} 2024-02-01T10:35:32.669 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:35:32.669 DEBUG:boto:Port: 8000 2024-02-01T10:35:32.669 DEBUG:boto:Params: {} 2024-02-01T10:35:32.669 DEBUG:boto:Token: None 2024-02-01T10:35:32.669 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:35:32 GMT /vclfcn-26/ 2024-02-01T10:35:32.669 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cJtK3ODPFpOlhCgcd/G+Rtwewlg= 2024-02-01T10:35:32.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cJtK3ODPFpOlhCgcd/G+Rtwewlg=', 'Content-Length': '0'} 2024-02-01T10:35:32.673 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000009c7f335a540f9a5-0065bb73f4-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:35:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:32.673 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:35:32.673 DEBUG:boto:path=/vclfcn-26/ 2024-02-01T10:35:32.674 DEBUG:boto:auth_path=/vclfcn-26/ 2024-02-01T10:35:32.674 DEBUG:boto:path=/vclfcn-26/?versions&max-keys=999 2024-02-01T10:35:32.674 DEBUG:boto:auth_path=/vclfcn-26/?versions&max-keys=999 2024-02-01T10:35:32.674 DEBUG:boto:Method: GET 2024-02-01T10:35:32.674 DEBUG:boto:Path: /vclfcn-26/?versions&max-keys=999 2024-02-01T10:35:32.674 DEBUG:boto:Data: 2024-02-01T10:35:32.674 DEBUG:boto:Headers: {} 2024-02-01T10:35:32.674 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:35:32.674 DEBUG:boto:Port: 8000 2024-02-01T10:35:32.674 DEBUG:boto:Params: {} 2024-02-01T10:35:32.674 DEBUG:boto:Token: None 2024-02-01T10:35:32.675 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:35:32 GMT /vclfcn-26/?versions 2024-02-01T10:35:32.675 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:osgOZaGKvpwLJJosu41ZdhmrKp8= 2024-02-01T10:35:32.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:osgOZaGKvpwLJJosu41ZdhmrKp8=', 'Content-Length': '0'} 2024-02-01T10:35:32.677 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b5f3bea2531b59b-0065bb73f4-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:35:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:32.677 DEBUG:boto:b'vclfcn-26999falseobjnulltrue2024-02-01T10:33:44.522Zrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:35:32.677 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:35:32.677 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:35:32.677 DEBUG:boto:path=/vclfcn-26/ 2024-02-01T10:35:32.678 DEBUG:boto:auth_path=/vclfcn-26/ 2024-02-01T10:35:32.678 DEBUG:boto:path=/vclfcn-26/?versions&max-keys=999 2024-02-01T10:35:32.678 DEBUG:boto:auth_path=/vclfcn-26/?versions&max-keys=999 2024-02-01T10:35:32.678 DEBUG:boto:Method: GET 2024-02-01T10:35:32.678 DEBUG:boto:Path: /vclfcn-26/?versions&max-keys=999 2024-02-01T10:35:32.678 DEBUG:boto:Data: 2024-02-01T10:35:32.678 DEBUG:boto:Headers: {} 2024-02-01T10:35:32.678 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:35:32.678 DEBUG:boto:Port: 8000 2024-02-01T10:35:32.678 DEBUG:boto:Params: {} 2024-02-01T10:35:32.678 DEBUG:boto:Token: None 2024-02-01T10:35:32.679 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:35:32 GMT /vclfcn-26/?versions 2024-02-01T10:35:32.679 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:osgOZaGKvpwLJJosu41ZdhmrKp8= 2024-02-01T10:35:32.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:osgOZaGKvpwLJJosu41ZdhmrKp8=', 'Content-Length': '0'} 2024-02-01T10:35:32.683 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010e889ba61b29267-0065bb73f4-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:35:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:32.724 DEBUG:boto:b'vclfcn-26999falseobjnulltrue2024-02-01T10:33:44.522Zrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:35:32.725 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:35:32.725 DEBUG:boto:path=/vclfcn-26/ 2024-02-01T10:35:32.725 DEBUG:boto:auth_path=/vclfcn-26/ 2024-02-01T10:35:32.725 DEBUG:boto:path=/vclfcn-26/?versions&max-keys=999 2024-02-01T10:35:32.725 DEBUG:boto:auth_path=/vclfcn-26/?versions&max-keys=999 2024-02-01T10:35:32.725 DEBUG:boto:Method: GET 2024-02-01T10:35:32.725 DEBUG:boto:Path: /vclfcn-26/?versions&max-keys=999 2024-02-01T10:35:32.725 DEBUG:boto:Data: 2024-02-01T10:35:32.725 DEBUG:boto:Headers: {} 2024-02-01T10:35:32.726 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:35:32.726 DEBUG:boto:Port: 8000 2024-02-01T10:35:32.726 DEBUG:boto:Params: {} 2024-02-01T10:35:32.726 DEBUG:boto:Token: None 2024-02-01T10:35:32.726 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:35:32 GMT /vclfcn-26/?versions 2024-02-01T10:35:32.726 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:osgOZaGKvpwLJJosu41ZdhmrKp8= 2024-02-01T10:35:32.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:osgOZaGKvpwLJJosu41ZdhmrKp8=', 'Content-Length': '0'} 2024-02-01T10:35:32.730 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083ab6665c4f3dc70-0065bb73f4-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:35:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:32.731 DEBUG:boto:b'vclfcn-26999falseobjnulltrue2024-02-01T10:33:44.522Zrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:35:32.731 DEBUG:boto:path=/vclfcn-26/ 2024-02-01T10:35:32.731 DEBUG:boto:auth_path=/vclfcn-26/ 2024-02-01T10:35:32.731 DEBUG:boto:path=/vclfcn-26/?versions&max-keys=999 2024-02-01T10:35:32.731 DEBUG:boto:auth_path=/vclfcn-26/?versions&max-keys=999 2024-02-01T10:35:32.731 DEBUG:boto:Method: GET 2024-02-01T10:35:32.731 DEBUG:boto:Path: /vclfcn-26/?versions&max-keys=999 2024-02-01T10:35:32.731 DEBUG:boto:Data: 2024-02-01T10:35:32.732 DEBUG:boto:Headers: {} 2024-02-01T10:35:32.732 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:35:32.732 DEBUG:boto:Port: 8000 2024-02-01T10:35:32.732 DEBUG:boto:Params: {} 2024-02-01T10:35:32.732 DEBUG:boto:Token: None 2024-02-01T10:35:32.732 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:35:32 GMT /vclfcn-26/?versions 2024-02-01T10:35:32.732 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:osgOZaGKvpwLJJosu41ZdhmrKp8= 2024-02-01T10:35:32.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:osgOZaGKvpwLJJosu41ZdhmrKp8=', 'Content-Length': '0'} 2024-02-01T10:35:32.736 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5e7e72500542b98-0065bb73f4-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:35:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:32.737 DEBUG:boto:b'vclfcn-26999falseobjnulltrue2024-02-01T10:33:44.522Zrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:35:32.737 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:35:32.737 DEBUG:boto:path=/vclfcn-26/obj 2024-02-01T10:35:32.737 DEBUG:boto:auth_path=/vclfcn-26/obj 2024-02-01T10:35:32.737 DEBUG:boto:Method: HEAD 2024-02-01T10:35:32.737 DEBUG:boto:Path: /vclfcn-26/obj 2024-02-01T10:35:32.737 DEBUG:boto:Data: 2024-02-01T10:35:32.738 DEBUG:boto:Headers: {} 2024-02-01T10:35:32.738 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:35:32.738 DEBUG:boto:Port: 8000 2024-02-01T10:35:32.738 DEBUG:boto:Params: {} 2024-02-01T10:35:32.738 DEBUG:boto:Token: None 2024-02-01T10:35:32.738 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:35:32 GMT /vclfcn-26/obj 2024-02-01T10:35:32.738 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RUK3wPxvl0ngu+W88g0gqsgtEPI= 2024-02-01T10:35:32.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RUK3wPxvl0ngu+W88g0gqsgtEPI=', 'Content-Length': '0'} 2024-02-01T10:35:32.741 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000c9b4e62760141d4e-0065bb73f4-4307-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:35:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:32.741 DEBUG:boto:path=/vclfcn-26/obj 2024-02-01T10:35:32.741 DEBUG:boto:auth_path=/vclfcn-26/obj 2024-02-01T10:35:32.741 DEBUG:boto:Method: HEAD 2024-02-01T10:35:32.741 DEBUG:boto:Path: /vclfcn-26/obj 2024-02-01T10:35:32.742 DEBUG:boto:Data: 2024-02-01T10:35:32.742 DEBUG:boto:Headers: {} 2024-02-01T10:35:32.742 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:35:32.742 DEBUG:boto:Port: 8000 2024-02-01T10:35:32.742 DEBUG:boto:Params: {} 2024-02-01T10:35:32.742 DEBUG:boto:Token: None 2024-02-01T10:35:32.742 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:35:32 GMT /vclfcn-26/obj 2024-02-01T10:35:32.742 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RUK3wPxvl0ngu+W88g0gqsgtEPI= 2024-02-01T10:35:32.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RUK3wPxvl0ngu+W88g0gqsgtEPI=', 'Content-Length': '0'} 2024-02-01T10:35:32.745 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000a35df25bcc2f95b5-0065bb73f4-4351-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:35:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:32.745 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-26 zones={test-zone2, test-zone1} 2024-02-01T10:35:32.745 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-26 zones={test-zone3, test-zone1} 2024-02-01T10:35:32.745 DEBUG:boto:path=/vclfcn-26/ 2024-02-01T10:35:32.745 DEBUG:boto:auth_path=/vclfcn-26/ 2024-02-01T10:35:32.745 DEBUG:boto:Method: HEAD 2024-02-01T10:35:32.745 DEBUG:boto:Path: /vclfcn-26/ 2024-02-01T10:35:32.745 DEBUG:boto:Data: 2024-02-01T10:35:32.745 DEBUG:boto:Headers: {} 2024-02-01T10:35:32.746 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:35:32.746 DEBUG:boto:Port: 8001 2024-02-01T10:35:32.746 DEBUG:boto:Params: {} 2024-02-01T10:35:32.746 DEBUG:boto:Token: None 2024-02-01T10:35:32.746 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:35:32 GMT /vclfcn-26/ 2024-02-01T10:35:32.746 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cJtK3ODPFpOlhCgcd/G+Rtwewlg= 2024-02-01T10:35:32.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cJtK3ODPFpOlhCgcd/G+Rtwewlg=', 'Content-Length': '0'} 2024-02-01T10:35:32.750 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000005edb8925c2d2a1-0065bb73f4-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:35:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:32.750 DEBUG:boto:path=/vclfcn-26/ 2024-02-01T10:35:32.750 DEBUG:boto:auth_path=/vclfcn-26/ 2024-02-01T10:35:32.750 DEBUG:boto:Method: HEAD 2024-02-01T10:35:32.750 DEBUG:boto:Path: /vclfcn-26/ 2024-02-01T10:35:32.750 DEBUG:boto:Data: 2024-02-01T10:35:32.751 DEBUG:boto:Headers: {} 2024-02-01T10:35:32.751 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:35:32.751 DEBUG:boto:Port: 8000 2024-02-01T10:35:32.751 DEBUG:boto:Params: {} 2024-02-01T10:35:32.751 DEBUG:boto:Token: None 2024-02-01T10:35:32.751 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:35:32 GMT /vclfcn-26/ 2024-02-01T10:35:32.751 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cJtK3ODPFpOlhCgcd/G+Rtwewlg= 2024-02-01T10:35:32.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cJtK3ODPFpOlhCgcd/G+Rtwewlg=', 'Content-Length': '0'} 2024-02-01T10:35:32.755 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f669d15fe0be74a4-0065bb73f4-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:35:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:32.755 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:35:32.755 DEBUG:boto:path=/vclfcn-26/ 2024-02-01T10:35:32.755 DEBUG:boto:auth_path=/vclfcn-26/ 2024-02-01T10:35:32.756 DEBUG:boto:path=/vclfcn-26/?versions&max-keys=999 2024-02-01T10:35:32.756 DEBUG:boto:auth_path=/vclfcn-26/?versions&max-keys=999 2024-02-01T10:35:32.756 DEBUG:boto:Method: GET 2024-02-01T10:35:32.756 DEBUG:boto:Path: /vclfcn-26/?versions&max-keys=999 2024-02-01T10:35:32.756 DEBUG:boto:Data: 2024-02-01T10:35:32.756 DEBUG:boto:Headers: {} 2024-02-01T10:35:32.756 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:35:32.756 DEBUG:boto:Port: 8001 2024-02-01T10:35:32.756 DEBUG:boto:Params: {} 2024-02-01T10:35:32.756 DEBUG:boto:Token: None 2024-02-01T10:35:32.757 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:35:32 GMT /vclfcn-26/?versions 2024-02-01T10:35:32.757 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:osgOZaGKvpwLJJosu41ZdhmrKp8= 2024-02-01T10:35:32.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:osgOZaGKvpwLJJosu41ZdhmrKp8=', 'Content-Length': '0'} 2024-02-01T10:35:32.761 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ee24dc43b3876d1-0065bb73f4-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:35:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:32.804 DEBUG:boto:b'vclfcn-26999falseobjnulltrue2024-02-01T10:33:44.522Zrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:35:32.804 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:35:32.805 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:35:32.805 DEBUG:boto:path=/vclfcn-26/ 2024-02-01T10:35:32.805 DEBUG:boto:auth_path=/vclfcn-26/ 2024-02-01T10:35:32.805 DEBUG:boto:path=/vclfcn-26/?versions&max-keys=999 2024-02-01T10:35:32.805 DEBUG:boto:auth_path=/vclfcn-26/?versions&max-keys=999 2024-02-01T10:35:32.805 DEBUG:boto:Method: GET 2024-02-01T10:35:32.805 DEBUG:boto:Path: /vclfcn-26/?versions&max-keys=999 2024-02-01T10:35:32.805 DEBUG:boto:Data: 2024-02-01T10:35:32.805 DEBUG:boto:Headers: {} 2024-02-01T10:35:32.805 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:35:32.806 DEBUG:boto:Port: 8000 2024-02-01T10:35:32.806 DEBUG:boto:Params: {} 2024-02-01T10:35:32.806 DEBUG:boto:Token: None 2024-02-01T10:35:32.806 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:35:32 GMT /vclfcn-26/?versions 2024-02-01T10:35:32.806 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:osgOZaGKvpwLJJosu41ZdhmrKp8= 2024-02-01T10:35:32.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:osgOZaGKvpwLJJosu41ZdhmrKp8=', 'Content-Length': '0'} 2024-02-01T10:35:32.810 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001375c1df623fbeb7-0065bb73f4-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:35:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:32.810 DEBUG:boto:b'vclfcn-26999falseobjnulltrue2024-02-01T10:33:44.522Zrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:35:32.811 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:35:32.811 DEBUG:boto:path=/vclfcn-26/ 2024-02-01T10:35:32.811 DEBUG:boto:auth_path=/vclfcn-26/ 2024-02-01T10:35:32.811 DEBUG:boto:path=/vclfcn-26/?versions&max-keys=999 2024-02-01T10:35:32.811 DEBUG:boto:auth_path=/vclfcn-26/?versions&max-keys=999 2024-02-01T10:35:32.811 DEBUG:boto:Method: GET 2024-02-01T10:35:32.811 DEBUG:boto:Path: /vclfcn-26/?versions&max-keys=999 2024-02-01T10:35:32.812 DEBUG:boto:Data: 2024-02-01T10:35:32.812 DEBUG:boto:Headers: {} 2024-02-01T10:35:32.812 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:35:32.812 DEBUG:boto:Port: 8001 2024-02-01T10:35:32.812 DEBUG:boto:Params: {} 2024-02-01T10:35:32.813 DEBUG:boto:Token: None 2024-02-01T10:35:32.813 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:35:32 GMT /vclfcn-26/?versions 2024-02-01T10:35:32.813 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:osgOZaGKvpwLJJosu41ZdhmrKp8= 2024-02-01T10:35:32.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:osgOZaGKvpwLJJosu41ZdhmrKp8=', 'Content-Length': '0'} 2024-02-01T10:35:32.817 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9ca7facd3e2415d-0065bb73f4-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:35:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:32.817 DEBUG:boto:b'vclfcn-26999falseobjnulltrue2024-02-01T10:33:44.522Zrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:35:32.818 DEBUG:boto:path=/vclfcn-26/ 2024-02-01T10:35:32.818 DEBUG:boto:auth_path=/vclfcn-26/ 2024-02-01T10:35:32.818 DEBUG:boto:path=/vclfcn-26/?versions&max-keys=999 2024-02-01T10:35:32.818 DEBUG:boto:auth_path=/vclfcn-26/?versions&max-keys=999 2024-02-01T10:35:32.818 DEBUG:boto:Method: GET 2024-02-01T10:35:32.818 DEBUG:boto:Path: /vclfcn-26/?versions&max-keys=999 2024-02-01T10:35:32.818 DEBUG:boto:Data: 2024-02-01T10:35:32.818 DEBUG:boto:Headers: {} 2024-02-01T10:35:32.818 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:35:32.818 DEBUG:boto:Port: 8000 2024-02-01T10:35:32.818 DEBUG:boto:Params: {} 2024-02-01T10:35:32.819 DEBUG:boto:Token: None 2024-02-01T10:35:32.819 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:35:32 GMT /vclfcn-26/?versions 2024-02-01T10:35:32.819 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:osgOZaGKvpwLJJosu41ZdhmrKp8= 2024-02-01T10:35:32.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:osgOZaGKvpwLJJosu41ZdhmrKp8=', 'Content-Length': '0'} 2024-02-01T10:35:32.823 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7ae3d9b41e831a4-0065bb73f4-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:35:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:32.823 DEBUG:boto:b'vclfcn-26999falseobjnulltrue2024-02-01T10:33:44.522Zrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:35:32.823 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:35:32.823 DEBUG:boto:path=/vclfcn-26/obj 2024-02-01T10:35:32.823 DEBUG:boto:auth_path=/vclfcn-26/obj 2024-02-01T10:35:32.824 DEBUG:boto:Method: HEAD 2024-02-01T10:35:32.824 DEBUG:boto:Path: /vclfcn-26/obj 2024-02-01T10:35:32.824 DEBUG:boto:Data: 2024-02-01T10:35:32.824 DEBUG:boto:Headers: {} 2024-02-01T10:35:32.824 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:35:32.824 DEBUG:boto:Port: 8001 2024-02-01T10:35:32.824 DEBUG:boto:Params: {} 2024-02-01T10:35:32.824 DEBUG:boto:Token: None 2024-02-01T10:35:32.824 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:35:32 GMT /vclfcn-26/obj 2024-02-01T10:35:32.824 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RUK3wPxvl0ngu+W88g0gqsgtEPI= 2024-02-01T10:35:32.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RUK3wPxvl0ngu+W88g0gqsgtEPI=', 'Content-Length': '0'} 2024-02-01T10:35:32.827 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000003e2d2135910afeaf-0065bb73f4-4358-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:35:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:32.827 DEBUG:boto:path=/vclfcn-26/obj 2024-02-01T10:35:32.827 DEBUG:boto:auth_path=/vclfcn-26/obj 2024-02-01T10:35:32.827 DEBUG:boto:Method: HEAD 2024-02-01T10:35:32.828 DEBUG:boto:Path: /vclfcn-26/obj 2024-02-01T10:35:32.828 DEBUG:boto:Data: 2024-02-01T10:35:32.828 DEBUG:boto:Headers: {} 2024-02-01T10:35:32.828 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:35:32.828 DEBUG:boto:Port: 8000 2024-02-01T10:35:32.828 DEBUG:boto:Params: {} 2024-02-01T10:35:32.828 DEBUG:boto:Token: None 2024-02-01T10:35:32.828 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:35:32 GMT /vclfcn-26/obj 2024-02-01T10:35:32.828 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RUK3wPxvl0ngu+W88g0gqsgtEPI= 2024-02-01T10:35:32.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RUK3wPxvl0ngu+W88g0gqsgtEPI=', 'Content-Length': '0'} 2024-02-01T10:35:32.831 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000007f2b90890d8feeef-0065bb73f4-4351-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:35:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:32.831 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-26 zones={test-zone3, test-zone1} 2024-02-01T10:35:32.831 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-26 zones={test-zone3, test-zone2} 2024-02-01T10:35:32.831 DEBUG:boto:path=/vclfcn-26/ 2024-02-01T10:35:32.831 DEBUG:boto:auth_path=/vclfcn-26/ 2024-02-01T10:35:32.831 DEBUG:boto:Method: HEAD 2024-02-01T10:35:32.831 DEBUG:boto:Path: /vclfcn-26/ 2024-02-01T10:35:32.832 DEBUG:boto:Data: 2024-02-01T10:35:32.832 DEBUG:boto:Headers: {} 2024-02-01T10:35:32.832 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:35:32.832 DEBUG:boto:Port: 8001 2024-02-01T10:35:32.832 DEBUG:boto:Params: {} 2024-02-01T10:35:32.832 DEBUG:boto:Token: None 2024-02-01T10:35:32.832 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:35:32 GMT /vclfcn-26/ 2024-02-01T10:35:32.832 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cJtK3ODPFpOlhCgcd/G+Rtwewlg= 2024-02-01T10:35:32.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cJtK3ODPFpOlhCgcd/G+Rtwewlg=', 'Content-Length': '0'} 2024-02-01T10:35:32.836 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e084fefcf4f0a2eb-0065bb73f4-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:35:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:32.836 DEBUG:boto:path=/vclfcn-26/ 2024-02-01T10:35:32.837 DEBUG:boto:auth_path=/vclfcn-26/ 2024-02-01T10:35:32.837 DEBUG:boto:Method: HEAD 2024-02-01T10:35:32.837 DEBUG:boto:Path: /vclfcn-26/ 2024-02-01T10:35:32.837 DEBUG:boto:Data: 2024-02-01T10:35:32.837 DEBUG:boto:Headers: {} 2024-02-01T10:35:32.837 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:35:32.837 DEBUG:boto:Port: 8000 2024-02-01T10:35:32.837 DEBUG:boto:Params: {} 2024-02-01T10:35:32.837 DEBUG:boto:Token: None 2024-02-01T10:35:32.837 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:35:32 GMT /vclfcn-26/ 2024-02-01T10:35:32.838 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cJtK3ODPFpOlhCgcd/G+Rtwewlg= 2024-02-01T10:35:32.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cJtK3ODPFpOlhCgcd/G+Rtwewlg=', 'Content-Length': '0'} 2024-02-01T10:35:32.842 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008d60f6617c1d793e-0065bb73f4-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:35:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:32.842 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:35:32.842 DEBUG:boto:path=/vclfcn-26/ 2024-02-01T10:35:32.842 DEBUG:boto:auth_path=/vclfcn-26/ 2024-02-01T10:35:32.842 DEBUG:boto:path=/vclfcn-26/?versions&max-keys=999 2024-02-01T10:35:32.842 DEBUG:boto:auth_path=/vclfcn-26/?versions&max-keys=999 2024-02-01T10:35:32.842 DEBUG:boto:Method: GET 2024-02-01T10:35:32.842 DEBUG:boto:Path: /vclfcn-26/?versions&max-keys=999 2024-02-01T10:35:32.842 DEBUG:boto:Data: 2024-02-01T10:35:32.843 DEBUG:boto:Headers: {} 2024-02-01T10:35:32.843 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:35:32.843 DEBUG:boto:Port: 8001 2024-02-01T10:35:32.843 DEBUG:boto:Params: {} 2024-02-01T10:35:32.843 DEBUG:boto:Token: None 2024-02-01T10:35:32.843 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:35:32 GMT /vclfcn-26/?versions 2024-02-01T10:35:32.843 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:osgOZaGKvpwLJJosu41ZdhmrKp8= 2024-02-01T10:35:32.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:osgOZaGKvpwLJJosu41ZdhmrKp8=', 'Content-Length': '0'} 2024-02-01T10:35:32.848 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d8c27a3c2cc7b9d1-0065bb73f4-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:35:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:32.892 DEBUG:boto:b'vclfcn-26999falseobjnulltrue2024-02-01T10:33:44.522Zrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:35:32.893 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:35:32.893 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:35:32.893 DEBUG:boto:path=/vclfcn-26/ 2024-02-01T10:35:32.893 DEBUG:boto:auth_path=/vclfcn-26/ 2024-02-01T10:35:32.893 DEBUG:boto:path=/vclfcn-26/?versions&max-keys=999 2024-02-01T10:35:32.893 DEBUG:boto:auth_path=/vclfcn-26/?versions&max-keys=999 2024-02-01T10:35:32.893 DEBUG:boto:Method: GET 2024-02-01T10:35:32.893 DEBUG:boto:Path: /vclfcn-26/?versions&max-keys=999 2024-02-01T10:35:32.894 DEBUG:boto:Data: 2024-02-01T10:35:32.894 DEBUG:boto:Headers: {} 2024-02-01T10:35:32.894 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:35:32.894 DEBUG:boto:Port: 8000 2024-02-01T10:35:32.894 DEBUG:boto:Params: {} 2024-02-01T10:35:32.894 DEBUG:boto:Token: None 2024-02-01T10:35:32.894 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:35:32 GMT /vclfcn-26/?versions 2024-02-01T10:35:32.894 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:osgOZaGKvpwLJJosu41ZdhmrKp8= 2024-02-01T10:35:32.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:osgOZaGKvpwLJJosu41ZdhmrKp8=', 'Content-Length': '0'} 2024-02-01T10:35:32.898 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004306512a1b5d3f7c-0065bb73f4-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:35:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:32.898 DEBUG:boto:b'vclfcn-26999falseobjnulltrue2024-02-01T10:33:44.522Zrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:35:32.899 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-02-01T10:35:32.899 DEBUG:boto:path=/vclfcn-26/ 2024-02-01T10:35:32.899 DEBUG:boto:auth_path=/vclfcn-26/ 2024-02-01T10:35:32.899 DEBUG:boto:path=/vclfcn-26/?versions&max-keys=999 2024-02-01T10:35:32.899 DEBUG:boto:auth_path=/vclfcn-26/?versions&max-keys=999 2024-02-01T10:35:32.899 DEBUG:boto:Method: GET 2024-02-01T10:35:32.899 DEBUG:boto:Path: /vclfcn-26/?versions&max-keys=999 2024-02-01T10:35:32.900 DEBUG:boto:Data: 2024-02-01T10:35:32.900 DEBUG:boto:Headers: {} 2024-02-01T10:35:32.900 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:35:32.900 DEBUG:boto:Port: 8001 2024-02-01T10:35:32.900 DEBUG:boto:Params: {} 2024-02-01T10:35:32.900 DEBUG:boto:Token: None 2024-02-01T10:35:32.900 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:35:32 GMT /vclfcn-26/?versions 2024-02-01T10:35:32.900 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:osgOZaGKvpwLJJosu41ZdhmrKp8= 2024-02-01T10:35:32.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:osgOZaGKvpwLJJosu41ZdhmrKp8=', 'Content-Length': '0'} 2024-02-01T10:35:32.904 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b90c26f6b57903e-0065bb73f4-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:35:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:32.905 DEBUG:boto:b'vclfcn-26999falseobjnulltrue2024-02-01T10:33:44.522Zrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:35:32.905 DEBUG:boto:path=/vclfcn-26/ 2024-02-01T10:35:32.905 DEBUG:boto:auth_path=/vclfcn-26/ 2024-02-01T10:35:32.905 DEBUG:boto:path=/vclfcn-26/?versions&max-keys=999 2024-02-01T10:35:32.905 DEBUG:boto:auth_path=/vclfcn-26/?versions&max-keys=999 2024-02-01T10:35:32.905 DEBUG:boto:Method: GET 2024-02-01T10:35:32.906 DEBUG:boto:Path: /vclfcn-26/?versions&max-keys=999 2024-02-01T10:35:32.906 DEBUG:boto:Data: 2024-02-01T10:35:32.906 DEBUG:boto:Headers: {} 2024-02-01T10:35:32.906 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:35:32.906 DEBUG:boto:Port: 8000 2024-02-01T10:35:32.906 DEBUG:boto:Params: {} 2024-02-01T10:35:32.906 DEBUG:boto:Token: None 2024-02-01T10:35:32.906 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:35:32 GMT /vclfcn-26/?versions 2024-02-01T10:35:32.906 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:osgOZaGKvpwLJJosu41ZdhmrKp8= 2024-02-01T10:35:32.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:osgOZaGKvpwLJJosu41ZdhmrKp8=', 'Content-Length': '0'} 2024-02-01T10:35:32.910 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d92125a4b9dfda4-0065bb73f4-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:35:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:32.956 DEBUG:boto:b'vclfcn-26999falseobjnulltrue2024-02-01T10:33:44.522Zrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:35:32.957 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-02-01T10:35:32.957 DEBUG:boto:path=/vclfcn-26/obj 2024-02-01T10:35:32.957 DEBUG:boto:auth_path=/vclfcn-26/obj 2024-02-01T10:35:32.957 DEBUG:boto:Method: HEAD 2024-02-01T10:35:32.957 DEBUG:boto:Path: /vclfcn-26/obj 2024-02-01T10:35:32.957 DEBUG:boto:Data: 2024-02-01T10:35:32.957 DEBUG:boto:Headers: {} 2024-02-01T10:35:32.957 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:35:32.957 DEBUG:boto:Port: 8001 2024-02-01T10:35:32.957 DEBUG:boto:Params: {} 2024-02-01T10:35:32.958 DEBUG:boto:Token: None 2024-02-01T10:35:32.958 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:35:32 GMT /vclfcn-26/obj 2024-02-01T10:35:32.958 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RUK3wPxvl0ngu+W88g0gqsgtEPI= 2024-02-01T10:35:32.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RUK3wPxvl0ngu+W88g0gqsgtEPI=', 'Content-Length': '0'} 2024-02-01T10:35:32.961 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000008d18ff7b93f86878-0065bb73f4-4358-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:35:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:32.961 DEBUG:boto:path=/vclfcn-26/obj 2024-02-01T10:35:32.961 DEBUG:boto:auth_path=/vclfcn-26/obj 2024-02-01T10:35:32.962 DEBUG:boto:Method: HEAD 2024-02-01T10:35:32.962 DEBUG:boto:Path: /vclfcn-26/obj 2024-02-01T10:35:32.962 DEBUG:boto:Data: 2024-02-01T10:35:32.962 DEBUG:boto:Headers: {} 2024-02-01T10:35:32.962 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:35:32.962 DEBUG:boto:Port: 8000 2024-02-01T10:35:32.962 DEBUG:boto:Params: {} 2024-02-01T10:35:32.962 DEBUG:boto:Token: None 2024-02-01T10:35:32.962 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:35:32 GMT /vclfcn-26/obj 2024-02-01T10:35:32.963 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RUK3wPxvl0ngu+W88g0gqsgtEPI= 2024-02-01T10:35:32.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RUK3wPxvl0ngu+W88g0gqsgtEPI=', 'Content-Length': '0'} 2024-02-01T10:35:32.965 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000011096f54c0b448f-0065bb73f4-4307-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:35:32 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:32.965 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-26 zones={test-zone3, test-zone2} 2024-02-01T10:35:32.966 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_suspended_delete_marker_full_sync ... ok 2024-02-01T10:35:32.967 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=vclfcn-27 2024-02-01T10:35:32.967 DEBUG:boto:path=/vclfcn-27/ 2024-02-01T10:35:32.968 DEBUG:boto:auth_path=/vclfcn-27/ 2024-02-01T10:35:32.968 DEBUG:boto:Method: PUT 2024-02-01T10:35:32.968 DEBUG:boto:Path: /vclfcn-27/ 2024-02-01T10:35:32.968 DEBUG:boto:Data: 2024-02-01T10:35:32.968 DEBUG:boto:Headers: {} 2024-02-01T10:35:32.968 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:35:32.968 DEBUG:boto:Port: 8000 2024-02-01T10:35:32.968 DEBUG:boto:Params: {} 2024-02-01T10:35:32.968 DEBUG:boto:Token: None 2024-02-01T10:35:32.969 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:35:32 GMT /vclfcn-27/ 2024-02-01T10:35:32.969 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7jVIXRIMlOvgnuV3DTgOauPpQ5Q= 2024-02-01T10:35:32.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7jVIXRIMlOvgnuV3DTgOauPpQ5Q=', 'Content-Length': '0'} 2024-02-01T10:35:33.002 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000dac63594aaba2cf7-0065bb73f4-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:35:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:33.003 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=vclfcn-28 2024-02-01T10:35:33.003 DEBUG:boto:path=/vclfcn-28/ 2024-02-01T10:35:33.003 DEBUG:boto:auth_path=/vclfcn-28/ 2024-02-01T10:35:33.003 DEBUG:boto:Method: PUT 2024-02-01T10:35:33.003 DEBUG:boto:Path: /vclfcn-28/ 2024-02-01T10:35:33.003 DEBUG:boto:Data: 2024-02-01T10:35:33.003 DEBUG:boto:Headers: {} 2024-02-01T10:35:33.003 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:35:33.003 DEBUG:boto:Port: 8000 2024-02-01T10:35:33.004 DEBUG:boto:Params: {} 2024-02-01T10:35:33.004 DEBUG:boto:Token: None 2024-02-01T10:35:33.004 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:35:33 GMT /vclfcn-28/ 2024-02-01T10:35:33.004 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:I7/6cuVK3eVOUPP5mE/MDUpQ3ko= 2024-02-01T10:35:33.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:I7/6cuVK3eVOUPP5mE/MDUpQ3ko=', 'Content-Length': '0'} 2024-02-01T10:35:33.068 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ac330535e26ce16e-0065bb73f5-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:35:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:33.068 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=vclfcn-29 2024-02-01T10:35:33.069 DEBUG:boto:path=/vclfcn-29/ 2024-02-01T10:35:33.069 DEBUG:boto:auth_path=/vclfcn-29/ 2024-02-01T10:35:33.069 DEBUG:boto:Method: PUT 2024-02-01T10:35:33.069 DEBUG:boto:Path: /vclfcn-29/ 2024-02-01T10:35:33.069 DEBUG:boto:Data: 2024-02-01T10:35:33.069 DEBUG:boto:Headers: {} 2024-02-01T10:35:33.069 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:35:33.069 DEBUG:boto:Port: 8001 2024-02-01T10:35:33.069 DEBUG:boto:Params: {} 2024-02-01T10:35:33.069 DEBUG:boto:Token: None 2024-02-01T10:35:33.070 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:35:33 GMT /vclfcn-29/ 2024-02-01T10:35:33.070 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:AT6Fe81L47WWC/NmluOnUBcATV8= 2024-02-01T10:35:33.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:AT6Fe81L47WWC/NmluOnUBcATV8=', 'Content-Length': '0'} 2024-02-01T10:35:33.125 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fbc001df8679bd78-0065bb73f5-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:35:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:33.125 DEBUG:boto:path=/vclfcn-27/ 2024-02-01T10:35:33.125 DEBUG:boto:auth_path=/vclfcn-27/ 2024-02-01T10:35:33.125 DEBUG:boto:path=/vclfcn-27/?versioning 2024-02-01T10:35:33.125 DEBUG:boto:auth_path=/vclfcn-27/?versioning 2024-02-01T10:35:33.126 DEBUG:boto:Method: PUT 2024-02-01T10:35:33.126 DEBUG:boto:Path: /vclfcn-27/?versioning 2024-02-01T10:35:33.126 DEBUG:boto:Data: Enabled Disabled 2024-02-01T10:35:33.126 DEBUG:boto:Headers: {} 2024-02-01T10:35:33.126 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:35:33.126 DEBUG:boto:Port: 8000 2024-02-01T10:35:33.126 DEBUG:boto:Params: {} 2024-02-01T10:35:33.126 DEBUG:boto:Token: None 2024-02-01T10:35:33.126 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:35:33 GMT /vclfcn-27/?versioning 2024-02-01T10:35:33.127 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+H9s+SA8TYMOyXXy+tBqwmwbFho= 2024-02-01T10:35:33.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+H9s+SA8TYMOyXXy+tBqwmwbFho=', 'Content-Length': '228'} 2024-02-01T10:35:33.142 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000021fecb2db59fa9d1-0065bb73f5-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:35:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:33.142 DEBUG:boto:path=/vclfcn-27/ 2024-02-01T10:35:33.142 DEBUG:boto:auth_path=/vclfcn-27/ 2024-02-01T10:35:33.142 DEBUG:boto:path=/vclfcn-27/?versioning 2024-02-01T10:35:33.142 DEBUG:boto:auth_path=/vclfcn-27/?versioning 2024-02-01T10:35:33.143 DEBUG:boto:Method: GET 2024-02-01T10:35:33.143 DEBUG:boto:Path: /vclfcn-27/?versioning 2024-02-01T10:35:33.143 DEBUG:boto:Data: 2024-02-01T10:35:33.143 DEBUG:boto:Headers: {} 2024-02-01T10:35:33.143 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:35:33.143 DEBUG:boto:Port: 8000 2024-02-01T10:35:33.143 DEBUG:boto:Params: {} 2024-02-01T10:35:33.143 DEBUG:boto:Token: None 2024-02-01T10:35:33.143 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:35:33 GMT /vclfcn-27/?versioning 2024-02-01T10:35:33.144 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:LGkTiEv6quymCKAWAy16DDtJLHU= 2024-02-01T10:35:33.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:LGkTiEv6quymCKAWAy16DDtJLHU=', 'Content-Length': '0'} 2024-02-01T10:35:33.145 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000b516141ea4b3056-0065bb73f5-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '192'), ('Date', 'Thu, 01 Feb 2024 10:35:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:33.188 DEBUG:boto:EnabledDisabled 2024-02-01T10:35:33.189 DEBUG:boto:path=/vclfcn-28/ 2024-02-01T10:35:33.189 DEBUG:boto:auth_path=/vclfcn-28/ 2024-02-01T10:35:33.189 DEBUG:boto:path=/vclfcn-28/?versioning 2024-02-01T10:35:33.189 DEBUG:boto:auth_path=/vclfcn-28/?versioning 2024-02-01T10:35:33.189 DEBUG:boto:Method: PUT 2024-02-01T10:35:33.189 DEBUG:boto:Path: /vclfcn-28/?versioning 2024-02-01T10:35:33.189 DEBUG:boto:Data: Enabled Disabled 2024-02-01T10:35:33.189 DEBUG:boto:Headers: {} 2024-02-01T10:35:33.189 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:35:33.189 DEBUG:boto:Port: 8000 2024-02-01T10:35:33.189 DEBUG:boto:Params: {} 2024-02-01T10:35:33.190 DEBUG:boto:Token: None 2024-02-01T10:35:33.190 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:35:33 GMT /vclfcn-28/?versioning 2024-02-01T10:35:33.190 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:qgwAsHFEFxP0f7CRQyVig5W2/P0= 2024-02-01T10:35:33.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:qgwAsHFEFxP0f7CRQyVig5W2/P0=', 'Content-Length': '228'} 2024-02-01T10:35:33.206 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000085eb6f71b017a037-0065bb73f5-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:35:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:33.206 DEBUG:boto:path=/vclfcn-28/ 2024-02-01T10:35:33.206 DEBUG:boto:auth_path=/vclfcn-28/ 2024-02-01T10:35:33.206 DEBUG:boto:path=/vclfcn-28/?versioning 2024-02-01T10:35:33.206 DEBUG:boto:auth_path=/vclfcn-28/?versioning 2024-02-01T10:35:33.206 DEBUG:boto:Method: GET 2024-02-01T10:35:33.207 DEBUG:boto:Path: /vclfcn-28/?versioning 2024-02-01T10:35:33.207 DEBUG:boto:Data: 2024-02-01T10:35:33.207 DEBUG:boto:Headers: {} 2024-02-01T10:35:33.207 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:35:33.207 DEBUG:boto:Port: 8000 2024-02-01T10:35:33.207 DEBUG:boto:Params: {} 2024-02-01T10:35:33.207 DEBUG:boto:Token: None 2024-02-01T10:35:33.207 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:35:33 GMT /vclfcn-28/?versioning 2024-02-01T10:35:33.207 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NeC5c7LoFpP+n+l1cYc0Qc8Nrio= 2024-02-01T10:35:33.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NeC5c7LoFpP+n+l1cYc0Qc8Nrio=', 'Content-Length': '0'} 2024-02-01T10:35:33.209 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000db732ac7de322766-0065bb73f5-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '192'), ('Date', 'Thu, 01 Feb 2024 10:35:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:33.209 DEBUG:boto:EnabledDisabled 2024-02-01T10:35:33.209 DEBUG:boto:path=/vclfcn-29/ 2024-02-01T10:35:33.210 DEBUG:boto:auth_path=/vclfcn-29/ 2024-02-01T10:35:33.210 DEBUG:boto:path=/vclfcn-29/?versioning 2024-02-01T10:35:33.210 DEBUG:boto:auth_path=/vclfcn-29/?versioning 2024-02-01T10:35:33.210 DEBUG:boto:Method: PUT 2024-02-01T10:35:33.210 DEBUG:boto:Path: /vclfcn-29/?versioning 2024-02-01T10:35:33.210 DEBUG:boto:Data: Enabled Disabled 2024-02-01T10:35:33.210 DEBUG:boto:Headers: {} 2024-02-01T10:35:33.210 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:35:33.210 DEBUG:boto:Port: 8001 2024-02-01T10:35:33.210 DEBUG:boto:Params: {} 2024-02-01T10:35:33.210 DEBUG:boto:Token: None 2024-02-01T10:35:33.210 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:35:33 GMT /vclfcn-29/?versioning 2024-02-01T10:35:33.211 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:hA85XlhSvAF/CVB2lwewu7BDHU0= 2024-02-01T10:35:33.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:hA85XlhSvAF/CVB2lwewu7BDHU0=', 'Content-Length': '228'} 2024-02-01T10:35:33.224 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006e635a103fcff218-0065bb73f5-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:35:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:33.224 DEBUG:boto:path=/vclfcn-29/ 2024-02-01T10:35:33.224 DEBUG:boto:auth_path=/vclfcn-29/ 2024-02-01T10:35:33.224 DEBUG:boto:path=/vclfcn-29/?versioning 2024-02-01T10:35:33.225 DEBUG:boto:auth_path=/vclfcn-29/?versioning 2024-02-01T10:35:33.225 DEBUG:boto:Method: GET 2024-02-01T10:35:33.225 DEBUG:boto:Path: /vclfcn-29/?versioning 2024-02-01T10:35:33.225 DEBUG:boto:Data: 2024-02-01T10:35:33.225 DEBUG:boto:Headers: {} 2024-02-01T10:35:33.225 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:35:33.225 DEBUG:boto:Port: 8001 2024-02-01T10:35:33.225 DEBUG:boto:Params: {} 2024-02-01T10:35:33.225 DEBUG:boto:Token: None 2024-02-01T10:35:33.225 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:35:33 GMT /vclfcn-29/?versioning 2024-02-01T10:35:33.225 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/CHPbWm++J65etgICd8qeyhjCFE= 2024-02-01T10:35:33.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/CHPbWm++J65etgICd8qeyhjCFE=', 'Content-Length': '0'} 2024-02-01T10:35:33.227 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d5e1e2226a067159-0065bb73f5-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '192'), ('Date', 'Thu, 01 Feb 2024 10:35:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:33.268 DEBUG:boto:EnabledDisabled 2024-02-01T10:35:33.268 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_versioning ... ok 2024-02-01T10:35:33.269 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=vclfcn-30 2024-02-01T10:35:33.269 DEBUG:boto:path=/vclfcn-30/ 2024-02-01T10:35:33.269 DEBUG:boto:auth_path=/vclfcn-30/ 2024-02-01T10:35:33.269 DEBUG:boto:Method: PUT 2024-02-01T10:35:33.269 DEBUG:boto:Path: /vclfcn-30/ 2024-02-01T10:35:33.269 DEBUG:boto:Data: 2024-02-01T10:35:33.270 DEBUG:boto:Headers: {} 2024-02-01T10:35:33.270 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:35:33.270 DEBUG:boto:Port: 8000 2024-02-01T10:35:33.270 DEBUG:boto:Params: {} 2024-02-01T10:35:33.270 DEBUG:boto:Token: None 2024-02-01T10:35:33.270 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:35:33 GMT /vclfcn-30/ 2024-02-01T10:35:33.270 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:G20HYHFcUV2orcusft2C3zrv7so= 2024-02-01T10:35:33.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:G20HYHFcUV2orcusft2C3zrv7so=', 'Content-Length': '0'} 2024-02-01T10:35:33.304 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b1a00f71c02e9057-0065bb73f5-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:35:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:33.304 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=vclfcn-31 2024-02-01T10:35:33.304 DEBUG:boto:path=/vclfcn-31/ 2024-02-01T10:35:33.304 DEBUG:boto:auth_path=/vclfcn-31/ 2024-02-01T10:35:33.304 DEBUG:boto:Method: PUT 2024-02-01T10:35:33.304 DEBUG:boto:Path: /vclfcn-31/ 2024-02-01T10:35:33.304 DEBUG:boto:Data: 2024-02-01T10:35:33.304 DEBUG:boto:Headers: {} 2024-02-01T10:35:33.304 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:35:33.304 DEBUG:boto:Port: 8000 2024-02-01T10:35:33.305 DEBUG:boto:Params: {} 2024-02-01T10:35:33.305 DEBUG:boto:Token: None 2024-02-01T10:35:33.305 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:35:33 GMT /vclfcn-31/ 2024-02-01T10:35:33.305 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:63u9UptE2e29CS6k/MwOMJ7S2Bg= 2024-02-01T10:35:33.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:63u9UptE2e29CS6k/MwOMJ7S2Bg=', 'Content-Length': '0'} 2024-02-01T10:35:33.367 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002d5cf05dabde055a-0065bb73f5-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:35:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:33.368 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=vclfcn-32 2024-02-01T10:35:33.368 DEBUG:boto:path=/vclfcn-32/ 2024-02-01T10:35:33.368 DEBUG:boto:auth_path=/vclfcn-32/ 2024-02-01T10:35:33.368 DEBUG:boto:Method: PUT 2024-02-01T10:35:33.368 DEBUG:boto:Path: /vclfcn-32/ 2024-02-01T10:35:33.368 DEBUG:boto:Data: 2024-02-01T10:35:33.368 DEBUG:boto:Headers: {} 2024-02-01T10:35:33.368 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:35:33.369 DEBUG:boto:Port: 8001 2024-02-01T10:35:33.369 DEBUG:boto:Params: {} 2024-02-01T10:35:33.369 DEBUG:boto:Token: None 2024-02-01T10:35:33.369 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:35:33 GMT /vclfcn-32/ 2024-02-01T10:35:33.369 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:IeK/v3YD/5e+aLhYBWh0zE2wRkg= 2024-02-01T10:35:33.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:IeK/v3YD/5e+aLhYBWh0zE2wRkg=', 'Content-Length': '0'} 2024-02-01T10:35:33.420 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000013ecb26fe1bb3a76-0065bb73f5-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:35:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:33.420 DEBUG:boto:path=/vclfcn-30/ 2024-02-01T10:35:33.420 DEBUG:boto:auth_path=/vclfcn-30/ 2024-02-01T10:35:33.421 DEBUG:boto:path=/vclfcn-30/?acl 2024-02-01T10:35:33.421 DEBUG:boto:auth_path=/vclfcn-30/?acl 2024-02-01T10:35:33.421 DEBUG:boto:Method: GET 2024-02-01T10:35:33.421 DEBUG:boto:Path: /vclfcn-30/?acl 2024-02-01T10:35:33.421 DEBUG:boto:Data: 2024-02-01T10:35:33.421 DEBUG:boto:Headers: {} 2024-02-01T10:35:33.421 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:35:33.421 DEBUG:boto:Port: 8000 2024-02-01T10:35:33.421 DEBUG:boto:Params: {} 2024-02-01T10:35:33.421 DEBUG:boto:Token: None 2024-02-01T10:35:33.422 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:35:33 GMT /vclfcn-30/?acl 2024-02-01T10:35:33.422 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HQFOqT2irvOSP/I4zJUeySiXIiw= 2024-02-01T10:35:33.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HQFOqT2irvOSP/I4zJUeySiXIiw=', 'Content-Length': '0'} 2024-02-01T10:35:33.423 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007cb4d0b11310eff8-0065bb73f5-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '489'), ('Date', 'Thu, 01 Feb 2024 10:35:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:33.424 DEBUG:boto:path=/vclfcn-30/ 2024-02-01T10:35:33.424 DEBUG:boto:auth_path=/vclfcn-30/ 2024-02-01T10:35:33.424 DEBUG:boto:path=/vclfcn-30/?acl 2024-02-01T10:35:33.424 DEBUG:boto:auth_path=/vclfcn-30/?acl 2024-02-01T10:35:33.424 DEBUG:boto:Method: PUT 2024-02-01T10:35:33.424 DEBUG:boto:Path: /vclfcn-30/?acl 2024-02-01T10:35:33.424 DEBUG:boto:Data: 2024-02-01T10:35:33.424 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-02-01T10:35:33.424 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:35:33.424 DEBUG:boto:Port: 8000 2024-02-01T10:35:33.425 DEBUG:boto:Params: {} 2024-02-01T10:35:33.425 DEBUG:boto:Token: None 2024-02-01T10:35:33.425 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:35:33 GMT x-amz-acl:public-read /vclfcn-30/?acl 2024-02-01T10:35:33.425 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:s7CUs7Mk3Ep8AvHbb4v8ilDQnzg= 2024-02-01T10:35:33.425 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:s7CUs7Mk3Ep8AvHbb4v8ilDQnzg=', 'Content-Length': '0'} 2024-02-01T10:35:33.435 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002450c2e7d31575f6-0065bb73f5-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:35:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:33.435 DEBUG:boto:path=/vclfcn-30/ 2024-02-01T10:35:33.436 DEBUG:boto:auth_path=/vclfcn-30/ 2024-02-01T10:35:33.436 DEBUG:boto:path=/vclfcn-30/?acl 2024-02-01T10:35:33.436 DEBUG:boto:auth_path=/vclfcn-30/?acl 2024-02-01T10:35:33.436 DEBUG:boto:Method: GET 2024-02-01T10:35:33.436 DEBUG:boto:Path: /vclfcn-30/?acl 2024-02-01T10:35:33.436 DEBUG:boto:Data: 2024-02-01T10:35:33.436 DEBUG:boto:Headers: {} 2024-02-01T10:35:33.436 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:35:33.436 DEBUG:boto:Port: 8000 2024-02-01T10:35:33.436 DEBUG:boto:Params: {} 2024-02-01T10:35:33.437 DEBUG:boto:Token: None 2024-02-01T10:35:33.437 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:35:33 GMT /vclfcn-30/?acl 2024-02-01T10:35:33.437 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HQFOqT2irvOSP/I4zJUeySiXIiw= 2024-02-01T10:35:33.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HQFOqT2irvOSP/I4zJUeySiXIiw=', 'Content-Length': '0'} 2024-02-01T10:35:33.438 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001acd87e2ebf2cdc2-0065bb73f5-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '681'), ('Date', 'Thu, 01 Feb 2024 10:35:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:33.439 DEBUG:boto:path=/vclfcn-31/ 2024-02-01T10:35:33.439 DEBUG:boto:auth_path=/vclfcn-31/ 2024-02-01T10:35:33.439 DEBUG:boto:path=/vclfcn-31/?acl 2024-02-01T10:35:33.439 DEBUG:boto:auth_path=/vclfcn-31/?acl 2024-02-01T10:35:33.439 DEBUG:boto:Method: GET 2024-02-01T10:35:33.439 DEBUG:boto:Path: /vclfcn-31/?acl 2024-02-01T10:35:33.439 DEBUG:boto:Data: 2024-02-01T10:35:33.440 DEBUG:boto:Headers: {} 2024-02-01T10:35:33.440 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:35:33.440 DEBUG:boto:Port: 8000 2024-02-01T10:35:33.440 DEBUG:boto:Params: {} 2024-02-01T10:35:33.440 DEBUG:boto:Token: None 2024-02-01T10:35:33.440 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:35:33 GMT /vclfcn-31/?acl 2024-02-01T10:35:33.440 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ql2CTcDkVmJy7ihIbAm0/1YpcJ4= 2024-02-01T10:35:33.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ql2CTcDkVmJy7ihIbAm0/1YpcJ4=', 'Content-Length': '0'} 2024-02-01T10:35:33.442 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c57e73b4cbc62abd-0065bb73f5-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '489'), ('Date', 'Thu, 01 Feb 2024 10:35:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:33.442 DEBUG:boto:path=/vclfcn-31/ 2024-02-01T10:35:33.442 DEBUG:boto:auth_path=/vclfcn-31/ 2024-02-01T10:35:33.442 DEBUG:boto:path=/vclfcn-31/?acl 2024-02-01T10:35:33.442 DEBUG:boto:auth_path=/vclfcn-31/?acl 2024-02-01T10:35:33.443 DEBUG:boto:Method: PUT 2024-02-01T10:35:33.443 DEBUG:boto:Path: /vclfcn-31/?acl 2024-02-01T10:35:33.443 DEBUG:boto:Data: 2024-02-01T10:35:33.443 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-02-01T10:35:33.443 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:35:33.443 DEBUG:boto:Port: 8000 2024-02-01T10:35:33.443 DEBUG:boto:Params: {} 2024-02-01T10:35:33.443 DEBUG:boto:Token: None 2024-02-01T10:35:33.443 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:35:33 GMT x-amz-acl:public-read /vclfcn-31/?acl 2024-02-01T10:35:33.443 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:q8WunZSDaFNRSI6M9D2YdY7kSTY= 2024-02-01T10:35:33.444 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:q8WunZSDaFNRSI6M9D2YdY7kSTY=', 'Content-Length': '0'} 2024-02-01T10:35:33.460 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fed53897c59bfb77-0065bb73f5-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:35:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:33.460 DEBUG:boto:path=/vclfcn-31/ 2024-02-01T10:35:33.460 DEBUG:boto:auth_path=/vclfcn-31/ 2024-02-01T10:35:33.460 DEBUG:boto:path=/vclfcn-31/?acl 2024-02-01T10:35:33.460 DEBUG:boto:auth_path=/vclfcn-31/?acl 2024-02-01T10:35:33.460 DEBUG:boto:Method: GET 2024-02-01T10:35:33.460 DEBUG:boto:Path: /vclfcn-31/?acl 2024-02-01T10:35:33.460 DEBUG:boto:Data: 2024-02-01T10:35:33.460 DEBUG:boto:Headers: {} 2024-02-01T10:35:33.461 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:35:33.461 DEBUG:boto:Port: 8000 2024-02-01T10:35:33.461 DEBUG:boto:Params: {} 2024-02-01T10:35:33.461 DEBUG:boto:Token: None 2024-02-01T10:35:33.461 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:35:33 GMT /vclfcn-31/?acl 2024-02-01T10:35:33.461 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ql2CTcDkVmJy7ihIbAm0/1YpcJ4= 2024-02-01T10:35:33.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ql2CTcDkVmJy7ihIbAm0/1YpcJ4=', 'Content-Length': '0'} 2024-02-01T10:35:33.463 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000066ad3cfdd44b0b56-0065bb73f5-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '681'), ('Date', 'Thu, 01 Feb 2024 10:35:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:33.504 DEBUG:boto:path=/vclfcn-32/ 2024-02-01T10:35:33.504 DEBUG:boto:auth_path=/vclfcn-32/ 2024-02-01T10:35:33.505 DEBUG:boto:path=/vclfcn-32/?acl 2024-02-01T10:35:33.505 DEBUG:boto:auth_path=/vclfcn-32/?acl 2024-02-01T10:35:33.505 DEBUG:boto:Method: GET 2024-02-01T10:35:33.505 DEBUG:boto:Path: /vclfcn-32/?acl 2024-02-01T10:35:33.505 DEBUG:boto:Data: 2024-02-01T10:35:33.505 DEBUG:boto:Headers: {} 2024-02-01T10:35:33.505 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:35:33.505 DEBUG:boto:Port: 8001 2024-02-01T10:35:33.505 DEBUG:boto:Params: {} 2024-02-01T10:35:33.505 DEBUG:boto:Token: None 2024-02-01T10:35:33.506 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:35:33 GMT /vclfcn-32/?acl 2024-02-01T10:35:33.506 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1ipcv0/hUTP37T10Se5ZU9UHDAE= 2024-02-01T10:35:33.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1ipcv0/hUTP37T10Se5ZU9UHDAE=', 'Content-Length': '0'} 2024-02-01T10:35:33.507 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008559cbb6ff35dc39-0065bb73f5-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '489'), ('Date', 'Thu, 01 Feb 2024 10:35:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:33.508 DEBUG:boto:path=/vclfcn-32/ 2024-02-01T10:35:33.508 DEBUG:boto:auth_path=/vclfcn-32/ 2024-02-01T10:35:33.508 DEBUG:boto:path=/vclfcn-32/?acl 2024-02-01T10:35:33.508 DEBUG:boto:auth_path=/vclfcn-32/?acl 2024-02-01T10:35:33.508 DEBUG:boto:Method: PUT 2024-02-01T10:35:33.508 DEBUG:boto:Path: /vclfcn-32/?acl 2024-02-01T10:35:33.508 DEBUG:boto:Data: 2024-02-01T10:35:33.509 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-02-01T10:35:33.509 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:35:33.509 DEBUG:boto:Port: 8001 2024-02-01T10:35:33.509 DEBUG:boto:Params: {} 2024-02-01T10:35:33.509 DEBUG:boto:Token: None 2024-02-01T10:35:33.509 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:35:33 GMT x-amz-acl:public-read /vclfcn-32/?acl 2024-02-01T10:35:33.509 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:PKw1ekQh7pTcrXf4rDcbTQYM8ek= 2024-02-01T10:35:33.509 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:PKw1ekQh7pTcrXf4rDcbTQYM8ek=', 'Content-Length': '0'} 2024-02-01T10:35:33.563 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000830fe63f056d1d30-0065bb73f5-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:35:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:33.563 DEBUG:boto:path=/vclfcn-32/ 2024-02-01T10:35:33.563 DEBUG:boto:auth_path=/vclfcn-32/ 2024-02-01T10:35:33.563 DEBUG:boto:path=/vclfcn-32/?acl 2024-02-01T10:35:33.563 DEBUG:boto:auth_path=/vclfcn-32/?acl 2024-02-01T10:35:33.563 DEBUG:boto:Method: GET 2024-02-01T10:35:33.564 DEBUG:boto:Path: /vclfcn-32/?acl 2024-02-01T10:35:33.564 DEBUG:boto:Data: 2024-02-01T10:35:33.564 DEBUG:boto:Headers: {} 2024-02-01T10:35:33.564 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:35:33.564 DEBUG:boto:Port: 8001 2024-02-01T10:35:33.564 DEBUG:boto:Params: {} 2024-02-01T10:35:33.564 DEBUG:boto:Token: None 2024-02-01T10:35:33.564 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:35:33 GMT /vclfcn-32/?acl 2024-02-01T10:35:33.564 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1ipcv0/hUTP37T10Se5ZU9UHDAE= 2024-02-01T10:35:33.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1ipcv0/hUTP37T10Se5ZU9UHDAE=', 'Content-Length': '0'} 2024-02-01T10:35:33.566 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000157e7fcd71f77199-0065bb73f5-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '681'), ('Date', 'Thu, 01 Feb 2024 10:35:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:33.567 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_acl ... ok 2024-02-01T10:35:33.568 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=vclfcn-33 2024-02-01T10:35:33.568 DEBUG:boto:path=/vclfcn-33/ 2024-02-01T10:35:33.568 DEBUG:boto:auth_path=/vclfcn-33/ 2024-02-01T10:35:33.568 DEBUG:boto:Method: PUT 2024-02-01T10:35:33.568 DEBUG:boto:Path: /vclfcn-33/ 2024-02-01T10:35:33.568 DEBUG:boto:Data: 2024-02-01T10:35:33.568 DEBUG:boto:Headers: {} 2024-02-01T10:35:33.568 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:35:33.568 DEBUG:boto:Port: 8000 2024-02-01T10:35:33.569 DEBUG:boto:Params: {} 2024-02-01T10:35:33.569 DEBUG:boto:Token: None 2024-02-01T10:35:33.569 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:35:33 GMT /vclfcn-33/ 2024-02-01T10:35:33.569 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:SlQMSEy896SnmYnJVA4y5Og87F8= 2024-02-01T10:35:33.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:SlQMSEy896SnmYnJVA4y5Og87F8=', 'Content-Length': '0'} 2024-02-01T10:35:33.604 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000746e50b49b317158-0065bb73f5-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:35:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:33.604 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=vclfcn-34 2024-02-01T10:35:33.604 DEBUG:boto:path=/vclfcn-34/ 2024-02-01T10:35:33.605 DEBUG:boto:auth_path=/vclfcn-34/ 2024-02-01T10:35:33.605 DEBUG:boto:Method: PUT 2024-02-01T10:35:33.605 DEBUG:boto:Path: /vclfcn-34/ 2024-02-01T10:35:33.605 DEBUG:boto:Data: 2024-02-01T10:35:33.605 DEBUG:boto:Headers: {} 2024-02-01T10:35:33.605 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:35:33.605 DEBUG:boto:Port: 8000 2024-02-01T10:35:33.605 DEBUG:boto:Params: {} 2024-02-01T10:35:33.605 DEBUG:boto:Token: None 2024-02-01T10:35:33.606 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:35:33 GMT /vclfcn-34/ 2024-02-01T10:35:33.606 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:p9YcqGYnqDx0nBMdHI+Mr2neDeA= 2024-02-01T10:35:33.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:p9YcqGYnqDx0nBMdHI+Mr2neDeA=', 'Content-Length': '0'} 2024-02-01T10:35:33.652 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008ec5d66a01ea96a1-0065bb73f5-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:35:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:33.652 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=vclfcn-35 2024-02-01T10:35:33.652 DEBUG:boto:path=/vclfcn-35/ 2024-02-01T10:35:33.652 DEBUG:boto:auth_path=/vclfcn-35/ 2024-02-01T10:35:33.652 DEBUG:boto:Method: PUT 2024-02-01T10:35:33.652 DEBUG:boto:Path: /vclfcn-35/ 2024-02-01T10:35:33.653 DEBUG:boto:Data: 2024-02-01T10:35:33.653 DEBUG:boto:Headers: {} 2024-02-01T10:35:33.653 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:35:33.653 DEBUG:boto:Port: 8001 2024-02-01T10:35:33.653 DEBUG:boto:Params: {} 2024-02-01T10:35:33.653 DEBUG:boto:Token: None 2024-02-01T10:35:33.653 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:35:33 GMT /vclfcn-35/ 2024-02-01T10:35:33.654 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:XdZ8ELl9ZMDxGwlf8kieQtUr1jk= 2024-02-01T10:35:33.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:XdZ8ELl9ZMDxGwlf8kieQtUr1jk=', 'Content-Length': '0'} 2024-02-01T10:35:33.710 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008da9683ba350f08c-0065bb73f5-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:35:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:33.710 DEBUG:boto:path=/vclfcn-33/ 2024-02-01T10:35:33.710 DEBUG:boto:auth_path=/vclfcn-33/ 2024-02-01T10:35:33.711 DEBUG:boto:path=/vclfcn-33/?cors 2024-02-01T10:35:33.711 DEBUG:boto:auth_path=/vclfcn-33/?cors 2024-02-01T10:35:33.711 DEBUG:boto:Method: PUT 2024-02-01T10:35:33.711 DEBUG:boto:Path: /vclfcn-33/?cors 2024-02-01T10:35:33.711 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2024-02-01T10:35:33.711 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2024-02-01T10:35:33.711 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:35:33.711 DEBUG:boto:Port: 8000 2024-02-01T10:35:33.711 DEBUG:boto:Params: {} 2024-02-01T10:35:33.711 DEBUG:boto:Token: None 2024-02-01T10:35:33.712 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Thu, 01 Feb 2024 10:35:33 GMT /vclfcn-33/?cors 2024-02-01T10:35:33.712 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7JuUMFzGsP2SWKEov8x9pbFCyHE= 2024-02-01T10:35:33.712 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7JuUMFzGsP2SWKEov8x9pbFCyHE=', 'Content-Length': '218'} 2024-02-01T10:35:33.722 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000084eac6c68f975cc2-0065bb73f5-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:35:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:33.722 DEBUG:boto:path=/vclfcn-33/ 2024-02-01T10:35:33.722 DEBUG:boto:auth_path=/vclfcn-33/ 2024-02-01T10:35:33.722 DEBUG:boto:path=/vclfcn-33/?cors 2024-02-01T10:35:33.722 DEBUG:boto:auth_path=/vclfcn-33/?cors 2024-02-01T10:35:33.722 DEBUG:boto:Method: GET 2024-02-01T10:35:33.722 DEBUG:boto:Path: /vclfcn-33/?cors 2024-02-01T10:35:33.722 DEBUG:boto:Data: 2024-02-01T10:35:33.723 DEBUG:boto:Headers: {} 2024-02-01T10:35:33.723 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:35:33.723 DEBUG:boto:Port: 8000 2024-02-01T10:35:33.723 DEBUG:boto:Params: {} 2024-02-01T10:35:33.723 DEBUG:boto:Token: None 2024-02-01T10:35:33.723 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:35:33 GMT /vclfcn-33/?cors 2024-02-01T10:35:33.723 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:JohMOawssMtivSmAZ2rDg/cxbU4= 2024-02-01T10:35:33.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:JohMOawssMtivSmAZ2rDg/cxbU4=', 'Content-Length': '0'} 2024-02-01T10:35:33.724 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d83c0bec3b78e627-0065bb73f5-4351-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '266'), ('Date', 'Thu, 01 Feb 2024 10:35:33 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:33.725 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2024-02-01T10:35:33.725 DEBUG:boto:path=/vclfcn-34/ 2024-02-01T10:35:33.725 DEBUG:boto:auth_path=/vclfcn-34/ 2024-02-01T10:35:33.725 DEBUG:boto:path=/vclfcn-34/?cors 2024-02-01T10:35:33.725 DEBUG:boto:auth_path=/vclfcn-34/?cors 2024-02-01T10:35:33.726 DEBUG:boto:Method: PUT 2024-02-01T10:35:33.726 DEBUG:boto:Path: /vclfcn-34/?cors 2024-02-01T10:35:33.726 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2024-02-01T10:35:33.726 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2024-02-01T10:35:33.726 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:35:33.726 DEBUG:boto:Port: 8000 2024-02-01T10:35:33.726 DEBUG:boto:Params: {} 2024-02-01T10:35:33.726 DEBUG:boto:Token: None 2024-02-01T10:35:33.726 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Thu, 01 Feb 2024 10:35:33 GMT /vclfcn-34/?cors 2024-02-01T10:35:33.726 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Vg1/rJSuIPKpSTTOXyY0biwcSzc= 2024-02-01T10:35:33.727 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:33 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Vg1/rJSuIPKpSTTOXyY0biwcSzc=', 'Content-Length': '218'} 2024-02-01T10:35:34.782 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fb529793e97460b5-0065bb73f5-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:35:34 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:34.783 DEBUG:boto:path=/vclfcn-34/ 2024-02-01T10:35:34.783 DEBUG:boto:auth_path=/vclfcn-34/ 2024-02-01T10:35:34.783 DEBUG:boto:path=/vclfcn-34/?cors 2024-02-01T10:35:34.783 DEBUG:boto:auth_path=/vclfcn-34/?cors 2024-02-01T10:35:34.783 DEBUG:boto:Method: GET 2024-02-01T10:35:34.783 DEBUG:boto:Path: /vclfcn-34/?cors 2024-02-01T10:35:34.783 DEBUG:boto:Data: 2024-02-01T10:35:34.783 DEBUG:boto:Headers: {} 2024-02-01T10:35:34.783 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:35:34.784 DEBUG:boto:Port: 8000 2024-02-01T10:35:34.784 DEBUG:boto:Params: {} 2024-02-01T10:35:34.784 DEBUG:boto:Token: None 2024-02-01T10:35:34.784 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:35:34 GMT /vclfcn-34/?cors 2024-02-01T10:35:34.784 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:USlifySRF0nehrNUK3dWy1B2ciE= 2024-02-01T10:35:34.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:34 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:USlifySRF0nehrNUK3dWy1B2ciE=', 'Content-Length': '0'} 2024-02-01T10:35:34.786 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000cd40717eceb82ebc-0065bb73f6-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '266'), ('Date', 'Thu, 01 Feb 2024 10:35:34 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:34.786 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2024-02-01T10:35:34.786 DEBUG:boto:path=/vclfcn-35/ 2024-02-01T10:35:34.787 DEBUG:boto:auth_path=/vclfcn-35/ 2024-02-01T10:35:34.787 DEBUG:boto:path=/vclfcn-35/?cors 2024-02-01T10:35:34.787 DEBUG:boto:auth_path=/vclfcn-35/?cors 2024-02-01T10:35:34.787 DEBUG:boto:Method: PUT 2024-02-01T10:35:34.787 DEBUG:boto:Path: /vclfcn-35/?cors 2024-02-01T10:35:34.787 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2024-02-01T10:35:34.787 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2024-02-01T10:35:34.787 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:35:34.787 DEBUG:boto:Port: 8001 2024-02-01T10:35:34.787 DEBUG:boto:Params: {} 2024-02-01T10:35:34.788 DEBUG:boto:Token: None 2024-02-01T10:35:34.788 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Thu, 01 Feb 2024 10:35:34 GMT /vclfcn-35/?cors 2024-02-01T10:35:34.788 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5hWT8SSVawz3KYqTsiUXL1HQx0U= 2024-02-01T10:35:34.788 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:34 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5hWT8SSVawz3KYqTsiUXL1HQx0U=', 'Content-Length': '218'} 2024-02-01T10:35:35.807 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000418c57ddc3894a0-0065bb73f6-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:35:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:35.808 DEBUG:boto:path=/vclfcn-35/ 2024-02-01T10:35:35.808 DEBUG:boto:auth_path=/vclfcn-35/ 2024-02-01T10:35:35.808 DEBUG:boto:path=/vclfcn-35/?cors 2024-02-01T10:35:35.808 DEBUG:boto:auth_path=/vclfcn-35/?cors 2024-02-01T10:35:35.808 DEBUG:boto:Method: GET 2024-02-01T10:35:35.808 DEBUG:boto:Path: /vclfcn-35/?cors 2024-02-01T10:35:35.809 DEBUG:boto:Data: 2024-02-01T10:35:35.809 DEBUG:boto:Headers: {} 2024-02-01T10:35:35.809 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:35:35.809 DEBUG:boto:Port: 8001 2024-02-01T10:35:35.809 DEBUG:boto:Params: {} 2024-02-01T10:35:35.809 DEBUG:boto:Token: None 2024-02-01T10:35:35.809 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:35:35 GMT /vclfcn-35/?cors 2024-02-01T10:35:35.810 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fjovUmme6O67RBjEwk1OfA+iW2I= 2024-02-01T10:35:35.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fjovUmme6O67RBjEwk1OfA+iW2I=', 'Content-Length': '0'} 2024-02-01T10:35:35.811 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006dc7eadb88f9eecc-0065bb73f7-4358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '266'), ('Date', 'Thu, 01 Feb 2024 10:35:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:35.811 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2024-02-01T10:35:35.812 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_cors ... ok 2024-02-01T10:35:35.813 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=vclfcn-36 2024-02-01T10:35:35.813 DEBUG:boto:path=/vclfcn-36/ 2024-02-01T10:35:35.813 DEBUG:boto:auth_path=/vclfcn-36/ 2024-02-01T10:35:35.813 DEBUG:boto:Method: PUT 2024-02-01T10:35:35.813 DEBUG:boto:Path: /vclfcn-36/ 2024-02-01T10:35:35.813 DEBUG:boto:Data: 2024-02-01T10:35:35.813 DEBUG:boto:Headers: {} 2024-02-01T10:35:35.813 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:35:35.813 DEBUG:boto:Port: 8000 2024-02-01T10:35:35.813 DEBUG:boto:Params: {} 2024-02-01T10:35:35.814 DEBUG:boto:Token: None 2024-02-01T10:35:35.814 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:35:35 GMT /vclfcn-36/ 2024-02-01T10:35:35.814 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:eTPq2uDjAwf7zyuR6zW5kmOe5WM= 2024-02-01T10:35:35.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:eTPq2uDjAwf7zyuR6zW5kmOe5WM=', 'Content-Length': '0'} 2024-02-01T10:35:35.850 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000084971e31a824625a-0065bb73f7-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:35:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:35.850 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=vclfcn-37 2024-02-01T10:35:35.850 DEBUG:boto:path=/vclfcn-37/ 2024-02-01T10:35:35.850 DEBUG:boto:auth_path=/vclfcn-37/ 2024-02-01T10:35:35.850 DEBUG:boto:Method: PUT 2024-02-01T10:35:35.850 DEBUG:boto:Path: /vclfcn-37/ 2024-02-01T10:35:35.851 DEBUG:boto:Data: 2024-02-01T10:35:35.851 DEBUG:boto:Headers: {} 2024-02-01T10:35:35.851 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:35:35.851 DEBUG:boto:Port: 8000 2024-02-01T10:35:35.851 DEBUG:boto:Params: {} 2024-02-01T10:35:35.851 DEBUG:boto:Token: None 2024-02-01T10:35:35.851 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:35:35 GMT /vclfcn-37/ 2024-02-01T10:35:35.851 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:a06G24tNAFSgvbrDYxCTaAIL1UE= 2024-02-01T10:35:35.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:a06G24tNAFSgvbrDYxCTaAIL1UE=', 'Content-Length': '0'} 2024-02-01T10:35:35.928 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000038c451d35e9f9f29-0065bb73f7-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:35:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:35.929 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=vclfcn-38 2024-02-01T10:35:35.929 DEBUG:boto:path=/vclfcn-38/ 2024-02-01T10:35:35.929 DEBUG:boto:auth_path=/vclfcn-38/ 2024-02-01T10:35:35.929 DEBUG:boto:Method: PUT 2024-02-01T10:35:35.929 DEBUG:boto:Path: /vclfcn-38/ 2024-02-01T10:35:35.929 DEBUG:boto:Data: 2024-02-01T10:35:35.929 DEBUG:boto:Headers: {} 2024-02-01T10:35:35.929 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:35:35.929 DEBUG:boto:Port: 8001 2024-02-01T10:35:35.930 DEBUG:boto:Params: {} 2024-02-01T10:35:35.930 DEBUG:boto:Token: None 2024-02-01T10:35:35.930 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:35:35 GMT /vclfcn-38/ 2024-02-01T10:35:35.930 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:i5U3nUKqhedBwEHK2XQGbzCwfQo= 2024-02-01T10:35:35.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:35 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:i5U3nUKqhedBwEHK2XQGbzCwfQo=', 'Content-Length': '0'} 2024-02-01T10:35:35.980 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e2a20d966fcf69f5-0065bb73f7-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:35:35 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:35.980 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:35:35.980 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:35:35.981 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:35:36.018 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:35:36.018 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:35:36.072 INFO:teuthology.orchestra.run.smithi148.stderr:No --period given, using current period=5dc4efbd-eccd-47a8-9471-ae4a209ec978 2024-02-01T10:35:36.096 INFO:teuthology.orchestra.run.smithi148.stdout:[{"marker":"1_1706782453.132619_30.1","last_update":"2024-02-01T10:14:13.132619Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1706783735.974063_243.1","last_update":"2024-02-01T10:35:35.974063Z"},{"marker":"1_1706782595.670952_121.1","last_update":"2024-02-01T10:16:35.670952Z"}] 2024-02-01T10:35:36.096 DEBUG:rgw_multi.tests:master meta markers={0: '1_1706782453.132619_30.1', 1: '', 2: '1_1706783735.974063_243.1', 3: '1_1706782595.670952_121.1'} 2024-02-01T10:35:36.097 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-02-01T10:35:36.097 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:35:36.097 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:35:36.097 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:35:36.137 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:35:36.138 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:35:36.193 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:35:36.191+0000 7f19c90c3980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:35:36.193 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:35:36.195+0000 7f19c90c3980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:35:36.220 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783733.293745_196.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:35:33.293745Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:35:36Z"} 2024-02-01T10:35:36.220 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783733.293745_196.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:35:33.293745Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:35:36Z"} 2024-02-01T10:35:36.220 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706783733.293745_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:35:33.293745Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:35:36.220 DEBUG:rgw_multi.tests:log_status={0: '1_1706782453.132619_30.1', 1: '', 2: '1_1706783735.974063_243.1', 3: '1_1706782595.670952_121.1'} 2024-02-01T10:35:36.221 DEBUG:rgw_multi.tests:sync_status={0: '1_1706782453.132619_30.1', 1: '', 2: '1_1706783733.293745_196.1', 3: '1_1706782595.670952_121.1'} 2024-02-01T10:35:36.221 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1706783735.974063_243.1 target=1_1706783733.293745_196.1 2024-02-01T10:35:41.222 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:35:41.222 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:35:41.222 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:35:41.268 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:35:41.269 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:35:41.322 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:35:41.323+0000 7f93eee4f980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:35:41.323 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:35:41.323+0000 7f93eee4f980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:35:41.346 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783733.307848_199.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:35:33.307848Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:35:41Z"} 2024-02-01T10:35:41.346 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783733.307848_199.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:35:33.307848Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:35:41Z"} 2024-02-01T10:35:41.347 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706783733.307848_199.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:35:33.307848Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:35:41.347 DEBUG:rgw_multi.tests:log_status={0: '1_1706782453.132619_30.1', 1: '', 2: '1_1706783735.974063_243.1', 3: '1_1706782595.670952_121.1'} 2024-02-01T10:35:41.347 DEBUG:rgw_multi.tests:sync_status={0: '1_1706782453.132619_30.1', 1: '', 2: '1_1706783733.307848_199.1', 3: '1_1706782595.670952_121.1'} 2024-02-01T10:35:41.347 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1706783735.974063_243.1 target=1_1706783733.307848_199.1 2024-02-01T10:35:46.348 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:35:46.348 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:35:46.349 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:35:46.392 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:35:46.392 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:35:46.448 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:35:46.447+0000 7ff20c2e8980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:35:46.450 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:35:46.451+0000 7ff20c2e8980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:35:46.475 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783735.974063_243.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:35:35.974063Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:35:46Z"} 2024-02-01T10:35:46.475 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783735.974063_243.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:35:35.974063Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:35:46Z"} 2024-02-01T10:35:46.475 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706783735.974063_243.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:35:35.974063Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:35:46.476 DEBUG:rgw_multi.tests:log_status={0: '1_1706782453.132619_30.1', 1: '', 2: '1_1706783735.974063_243.1', 3: '1_1706782595.670952_121.1'} 2024-02-01T10:35:46.476 DEBUG:rgw_multi.tests:sync_status={0: '1_1706782453.132619_30.1', 1: '', 2: '1_1706783735.974063_243.1', 3: '1_1706782595.670952_121.1'} 2024-02-01T10:35:46.476 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-02-01T10:35:46.476 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-02-01T10:35:46.476 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:35:46.476 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:35:46.476 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:35:46.518 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:35:46.519 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:35:46.573 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:35:46.575+0000 7f5d63f76980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:35:46.574 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:35:46.575+0000 7f5d63f76980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:35:46.596 INFO:teuthology.orchestra.run.smithi148.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783735.974063_243.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:35:35.974063Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:35:46Z"} 2024-02-01T10:35:46.597 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783735.974063_243.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:35:35.974063Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:35:46Z"} 2024-02-01T10:35:46.597 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706783735.974063_243.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:35:35.974063Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:35:46.597 DEBUG:rgw_multi.tests:log_status={0: '1_1706782453.132619_30.1', 1: '', 2: '1_1706783735.974063_243.1', 3: '1_1706782595.670952_121.1'} 2024-02-01T10:35:46.597 DEBUG:rgw_multi.tests:sync_status={0: '1_1706782453.132619_30.1', 1: '', 2: '1_1706783735.974063_243.1', 3: '1_1706782595.670952_121.1'} 2024-02-01T10:35:46.597 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-02-01T10:35:46.597 DEBUG:boto:path=/vclfcn-36/ 2024-02-01T10:35:46.597 DEBUG:boto:auth_path=/vclfcn-36/ 2024-02-01T10:35:46.598 DEBUG:boto:Method: HEAD 2024-02-01T10:35:46.598 DEBUG:boto:Path: /vclfcn-36/ 2024-02-01T10:35:46.598 DEBUG:boto:Data: 2024-02-01T10:35:46.598 DEBUG:boto:Headers: {} 2024-02-01T10:35:46.598 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:35:46.598 DEBUG:boto:Port: 8000 2024-02-01T10:35:46.598 DEBUG:boto:Params: {} 2024-02-01T10:35:46.598 DEBUG:boto:Token: None 2024-02-01T10:35:46.598 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:35:46 GMT /vclfcn-36/ 2024-02-01T10:35:46.599 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8fm7umiAoB6QVZ6Mt3juFtWEd1U= 2024-02-01T10:35:46.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8fm7umiAoB6QVZ6Mt3juFtWEd1U=', 'Content-Length': '0'} 2024-02-01T10:35:46.601 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000083dd7a57f3a2b83d-0065bb7402-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:35:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:46.602 DEBUG:boto:path=/vclfcn-36/foo 2024-02-01T10:35:46.602 DEBUG:boto:auth_path=/vclfcn-36/foo 2024-02-01T10:35:46.602 DEBUG:boto:Method: PUT 2024-02-01T10:35:46.602 DEBUG:boto:Path: /vclfcn-36/foo 2024-02-01T10:35:46.602 DEBUG:boto:Data: 2024-02-01T10:35:46.602 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'N7UdGUp1E+RbVvZSTy1R8g==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-02-01T10:35:46.602 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:35:46.603 DEBUG:boto:Port: 8000 2024-02-01T10:35:46.603 DEBUG:boto:Params: {} 2024-02-01T10:35:46.603 DEBUG:boto:Token: None 2024-02-01T10:35:46.603 DEBUG:boto:StringToSign: PUT N7UdGUp1E+RbVvZSTy1R8g== application/octet-stream Thu, 01 Feb 2024 10:35:46 GMT /vclfcn-36/foo 2024-02-01T10:35:46.603 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:PgEk7MUZ8hAyJGp8WS+EhDiPSm8= 2024-02-01T10:35:46.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'N7UdGUp1E+RbVvZSTy1R8g==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:35:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:PgEk7MUZ8hAyJGp8WS+EhDiPSm8='} 2024-02-01T10:35:46.648 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"37b51d194a7513e45b56f6524f2d51f2"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b22686b336134caa-0065bb7402-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:35:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:46.649 DEBUG:boto:path=/vclfcn-36/ 2024-02-01T10:35:46.649 DEBUG:boto:auth_path=/vclfcn-36/ 2024-02-01T10:35:46.649 DEBUG:boto:Method: DELETE 2024-02-01T10:35:46.649 DEBUG:boto:Path: /vclfcn-36/ 2024-02-01T10:35:46.649 DEBUG:boto:Data: 2024-02-01T10:35:46.649 DEBUG:boto:Headers: {} 2024-02-01T10:35:46.649 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:35:46.650 DEBUG:boto:Port: 8000 2024-02-01T10:35:46.650 DEBUG:boto:Params: {} 2024-02-01T10:35:46.650 DEBUG:boto:Token: None 2024-02-01T10:35:46.650 DEBUG:boto:StringToSign: DELETE Thu, 01 Feb 2024 10:35:46 GMT /vclfcn-36/ 2024-02-01T10:35:46.650 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:o9ruvh1uNObpkymm7y/dKLotUzU= 2024-02-01T10:35:46.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:o9ruvh1uNObpkymm7y/dKLotUzU=', 'Content-Length': '0'} 2024-02-01T10:35:46.662 DEBUG:boto:Response headers: [('Content-Length', '253'), ('x-amz-request-id', 'tx000000ec878b05bb58555-0065bb7402-4351-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:35:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:46.663 DEBUG:boto:path=/vclfcn-37/ 2024-02-01T10:35:46.663 DEBUG:boto:auth_path=/vclfcn-37/ 2024-02-01T10:35:46.663 DEBUG:boto:Method: HEAD 2024-02-01T10:35:46.663 DEBUG:boto:Path: /vclfcn-37/ 2024-02-01T10:35:46.663 DEBUG:boto:Data: 2024-02-01T10:35:46.663 DEBUG:boto:Headers: {} 2024-02-01T10:35:46.664 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:35:46.664 DEBUG:boto:Port: 8000 2024-02-01T10:35:46.664 DEBUG:boto:Params: {} 2024-02-01T10:35:46.664 DEBUG:boto:Token: None 2024-02-01T10:35:46.664 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:35:46 GMT /vclfcn-37/ 2024-02-01T10:35:46.664 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7rhByFVNrreNszB2Zp14S6ggjHI= 2024-02-01T10:35:46.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7rhByFVNrreNszB2Zp14S6ggjHI=', 'Content-Length': '0'} 2024-02-01T10:35:46.668 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c66faffc035a2ee0-0065bb7402-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:35:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:46.669 DEBUG:boto:path=/vclfcn-37/foo 2024-02-01T10:35:46.669 DEBUG:boto:auth_path=/vclfcn-37/foo 2024-02-01T10:35:46.669 DEBUG:boto:Method: PUT 2024-02-01T10:35:46.669 DEBUG:boto:Path: /vclfcn-37/foo 2024-02-01T10:35:46.669 DEBUG:boto:Data: 2024-02-01T10:35:46.669 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'N7UdGUp1E+RbVvZSTy1R8g==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-02-01T10:35:46.669 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:35:46.669 DEBUG:boto:Port: 8000 2024-02-01T10:35:46.670 DEBUG:boto:Params: {} 2024-02-01T10:35:46.670 DEBUG:boto:Token: None 2024-02-01T10:35:46.670 DEBUG:boto:StringToSign: PUT N7UdGUp1E+RbVvZSTy1R8g== application/octet-stream Thu, 01 Feb 2024 10:35:46 GMT /vclfcn-37/foo 2024-02-01T10:35:46.670 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:TvHROqRjgE7sutnmkOtVQ+XaYkU= 2024-02-01T10:35:46.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'N7UdGUp1E+RbVvZSTy1R8g==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:35:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:TvHROqRjgE7sutnmkOtVQ+XaYkU='} 2024-02-01T10:35:46.716 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"37b51d194a7513e45b56f6524f2d51f2"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000029674c0cd3c70ee0-0065bb7402-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:35:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:46.717 DEBUG:boto:path=/vclfcn-37/ 2024-02-01T10:35:46.717 DEBUG:boto:auth_path=/vclfcn-37/ 2024-02-01T10:35:46.717 DEBUG:boto:Method: DELETE 2024-02-01T10:35:46.717 DEBUG:boto:Path: /vclfcn-37/ 2024-02-01T10:35:46.717 DEBUG:boto:Data: 2024-02-01T10:35:46.717 DEBUG:boto:Headers: {} 2024-02-01T10:35:46.717 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:35:46.717 DEBUG:boto:Port: 8000 2024-02-01T10:35:46.717 DEBUG:boto:Params: {} 2024-02-01T10:35:46.718 DEBUG:boto:Token: None 2024-02-01T10:35:46.718 DEBUG:boto:StringToSign: DELETE Thu, 01 Feb 2024 10:35:46 GMT /vclfcn-37/ 2024-02-01T10:35:46.718 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:UELX+GYyfi/3Gh+tAxG95Gihx7w= 2024-02-01T10:35:46.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:UELX+GYyfi/3Gh+tAxG95Gihx7w=', 'Content-Length': '0'} 2024-02-01T10:35:46.734 DEBUG:boto:Response headers: [('Content-Length', '253'), ('x-amz-request-id', 'tx00000405f4122d9dec418-0065bb7402-4307-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:35:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:46.735 DEBUG:boto:path=/vclfcn-38/ 2024-02-01T10:35:46.735 DEBUG:boto:auth_path=/vclfcn-38/ 2024-02-01T10:35:46.735 DEBUG:boto:Method: HEAD 2024-02-01T10:35:46.735 DEBUG:boto:Path: /vclfcn-38/ 2024-02-01T10:35:46.735 DEBUG:boto:Data: 2024-02-01T10:35:46.735 DEBUG:boto:Headers: {} 2024-02-01T10:35:46.735 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:35:46.736 DEBUG:boto:Port: 8001 2024-02-01T10:35:46.736 DEBUG:boto:Params: {} 2024-02-01T10:35:46.736 DEBUG:boto:Token: None 2024-02-01T10:35:46.736 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:35:46 GMT /vclfcn-38/ 2024-02-01T10:35:46.736 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:PyRTwrqSM0v5ial6LaD+qUz+ZzA= 2024-02-01T10:35:46.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:PyRTwrqSM0v5ial6LaD+qUz+ZzA=', 'Content-Length': '0'} 2024-02-01T10:35:46.740 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000988c6683710d5487-0065bb7402-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:35:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:46.740 DEBUG:boto:path=/vclfcn-38/foo 2024-02-01T10:35:46.740 DEBUG:boto:auth_path=/vclfcn-38/foo 2024-02-01T10:35:46.740 DEBUG:boto:Method: PUT 2024-02-01T10:35:46.740 DEBUG:boto:Path: /vclfcn-38/foo 2024-02-01T10:35:46.740 DEBUG:boto:Data: 2024-02-01T10:35:46.740 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'N7UdGUp1E+RbVvZSTy1R8g==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-02-01T10:35:46.741 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:35:46.741 DEBUG:boto:Port: 8001 2024-02-01T10:35:46.741 DEBUG:boto:Params: {} 2024-02-01T10:35:46.741 DEBUG:boto:Token: None 2024-02-01T10:35:46.741 DEBUG:boto:StringToSign: PUT N7UdGUp1E+RbVvZSTy1R8g== application/octet-stream Thu, 01 Feb 2024 10:35:46 GMT /vclfcn-38/foo 2024-02-01T10:35:46.741 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:dX/N6YDXw1zT0K5mBk1AuruNZiM= 2024-02-01T10:35:46.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'N7UdGUp1E+RbVvZSTy1R8g==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:35:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dX/N6YDXw1zT0K5mBk1AuruNZiM='} 2024-02-01T10:35:46.764 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"37b51d194a7513e45b56f6524f2d51f2"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000050eaded393ebfd3b-0065bb7402-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:35:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:46.764 DEBUG:boto:path=/vclfcn-38/ 2024-02-01T10:35:46.764 DEBUG:boto:auth_path=/vclfcn-38/ 2024-02-01T10:35:46.765 DEBUG:boto:Method: DELETE 2024-02-01T10:35:46.765 DEBUG:boto:Path: /vclfcn-38/ 2024-02-01T10:35:46.765 DEBUG:boto:Data: 2024-02-01T10:35:46.765 DEBUG:boto:Headers: {} 2024-02-01T10:35:46.765 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:35:46.765 DEBUG:boto:Port: 8001 2024-02-01T10:35:46.765 DEBUG:boto:Params: {} 2024-02-01T10:35:46.765 DEBUG:boto:Token: None 2024-02-01T10:35:46.765 DEBUG:boto:StringToSign: DELETE Thu, 01 Feb 2024 10:35:46 GMT /vclfcn-38/ 2024-02-01T10:35:46.766 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:JJkIg1wcyQ3R06jgL4dF1UwuHQg= 2024-02-01T10:35:46.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:JJkIg1wcyQ3R06jgL4dF1UwuHQg=', 'Content-Length': '0'} 2024-02-01T10:35:46.783 DEBUG:boto:Response headers: [('Content-Length', '253'), ('x-amz-request-id', 'tx00000b2679c4416ecbbd7-0065bb7402-4358-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:35:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:46.828 DEBUG:boto:path=/vclfcn-36/ 2024-02-01T10:35:46.828 DEBUG:boto:auth_path=/vclfcn-36/ 2024-02-01T10:35:46.829 DEBUG:boto:Method: HEAD 2024-02-01T10:35:46.829 DEBUG:boto:Path: /vclfcn-36/ 2024-02-01T10:35:46.829 DEBUG:boto:Data: 2024-02-01T10:35:46.829 DEBUG:boto:Headers: {} 2024-02-01T10:35:46.829 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:35:46.829 DEBUG:boto:Port: 8000 2024-02-01T10:35:46.829 DEBUG:boto:Params: {} 2024-02-01T10:35:46.829 DEBUG:boto:Token: None 2024-02-01T10:35:46.829 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:35:46 GMT /vclfcn-36/ 2024-02-01T10:35:46.830 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8fm7umiAoB6QVZ6Mt3juFtWEd1U= 2024-02-01T10:35:46.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8fm7umiAoB6QVZ6Mt3juFtWEd1U=', 'Content-Length': '0'} 2024-02-01T10:35:46.833 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b1a80273d3d4d4f3-0065bb7402-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:35:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:46.834 DEBUG:boto:path=/vclfcn-37/ 2024-02-01T10:35:46.834 DEBUG:boto:auth_path=/vclfcn-37/ 2024-02-01T10:35:46.834 DEBUG:boto:Method: HEAD 2024-02-01T10:35:46.834 DEBUG:boto:Path: /vclfcn-37/ 2024-02-01T10:35:46.834 DEBUG:boto:Data: 2024-02-01T10:35:46.834 DEBUG:boto:Headers: {} 2024-02-01T10:35:46.834 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:35:46.834 DEBUG:boto:Port: 8000 2024-02-01T10:35:46.834 DEBUG:boto:Params: {} 2024-02-01T10:35:46.834 DEBUG:boto:Token: None 2024-02-01T10:35:46.834 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:35:46 GMT /vclfcn-37/ 2024-02-01T10:35:46.835 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7rhByFVNrreNszB2Zp14S6ggjHI= 2024-02-01T10:35:46.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7rhByFVNrreNszB2Zp14S6ggjHI=', 'Content-Length': '0'} 2024-02-01T10:35:46.838 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f7cf5181b8a6e5e7-0065bb7402-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:35:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:46.839 DEBUG:boto:path=/vclfcn-38/ 2024-02-01T10:35:46.839 DEBUG:boto:auth_path=/vclfcn-38/ 2024-02-01T10:35:46.839 DEBUG:boto:Method: HEAD 2024-02-01T10:35:46.839 DEBUG:boto:Path: /vclfcn-38/ 2024-02-01T10:35:46.839 DEBUG:boto:Data: 2024-02-01T10:35:46.839 DEBUG:boto:Headers: {} 2024-02-01T10:35:46.839 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:35:46.839 DEBUG:boto:Port: 8000 2024-02-01T10:35:46.839 DEBUG:boto:Params: {} 2024-02-01T10:35:46.840 DEBUG:boto:Token: None 2024-02-01T10:35:46.840 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:35:46 GMT /vclfcn-38/ 2024-02-01T10:35:46.840 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:PyRTwrqSM0v5ial6LaD+qUz+ZzA= 2024-02-01T10:35:46.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:PyRTwrqSM0v5ial6LaD+qUz+ZzA=', 'Content-Length': '0'} 2024-02-01T10:35:46.844 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d2385d1b1bf322ec-0065bb7402-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:35:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:46.844 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_delete_notempty ... ok 2024-02-01T10:35:46.845 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=vclfcn-39 2024-02-01T10:35:46.845 DEBUG:boto:path=/vclfcn-39/ 2024-02-01T10:35:46.845 DEBUG:boto:auth_path=/vclfcn-39/ 2024-02-01T10:35:46.845 DEBUG:boto:Method: PUT 2024-02-01T10:35:46.845 DEBUG:boto:Path: /vclfcn-39/ 2024-02-01T10:35:46.845 DEBUG:boto:Data: 2024-02-01T10:35:46.845 DEBUG:boto:Headers: {} 2024-02-01T10:35:46.845 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:35:46.845 DEBUG:boto:Port: 8000 2024-02-01T10:35:46.845 DEBUG:boto:Params: {} 2024-02-01T10:35:46.845 DEBUG:boto:Token: None 2024-02-01T10:35:46.846 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:35:46 GMT /vclfcn-39/ 2024-02-01T10:35:46.846 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1/uZ3r9hRIWdWRVZETYqaw6vAK0= 2024-02-01T10:35:46.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1/uZ3r9hRIWdWRVZETYqaw6vAK0=', 'Content-Length': '0'} 2024-02-01T10:35:46.881 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000046165ce3b40092bf-0065bb7402-4351-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:35:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:46.882 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=vclfcn-40 2024-02-01T10:35:46.882 DEBUG:boto:path=/vclfcn-40/ 2024-02-01T10:35:46.882 DEBUG:boto:auth_path=/vclfcn-40/ 2024-02-01T10:35:46.882 DEBUG:boto:Method: PUT 2024-02-01T10:35:46.882 DEBUG:boto:Path: /vclfcn-40/ 2024-02-01T10:35:46.882 DEBUG:boto:Data: 2024-02-01T10:35:46.882 DEBUG:boto:Headers: {} 2024-02-01T10:35:46.882 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:35:46.882 DEBUG:boto:Port: 8000 2024-02-01T10:35:46.882 DEBUG:boto:Params: {} 2024-02-01T10:35:46.883 DEBUG:boto:Token: None 2024-02-01T10:35:46.883 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:35:46 GMT /vclfcn-40/ 2024-02-01T10:35:46.883 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BhIv7m9h1zLr3uNB33uOKdY/fKA= 2024-02-01T10:35:46.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BhIv7m9h1zLr3uNB33uOKdY/fKA=', 'Content-Length': '0'} 2024-02-01T10:35:46.954 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000069bfa36e22083c82-0065bb7402-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:35:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:46.954 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=vclfcn-41 2024-02-01T10:35:46.954 DEBUG:boto:path=/vclfcn-41/ 2024-02-01T10:35:46.954 DEBUG:boto:auth_path=/vclfcn-41/ 2024-02-01T10:35:46.954 DEBUG:boto:Method: PUT 2024-02-01T10:35:46.954 DEBUG:boto:Path: /vclfcn-41/ 2024-02-01T10:35:46.954 DEBUG:boto:Data: 2024-02-01T10:35:46.955 DEBUG:boto:Headers: {} 2024-02-01T10:35:46.955 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:35:46.955 DEBUG:boto:Port: 8001 2024-02-01T10:35:46.955 DEBUG:boto:Params: {} 2024-02-01T10:35:46.955 DEBUG:boto:Token: None 2024-02-01T10:35:46.955 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:35:46 GMT /vclfcn-41/ 2024-02-01T10:35:46.955 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:DmbW5O5w5sWY/eOUbTrph6NtDUI= 2024-02-01T10:35:46.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:35:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:DmbW5O5w5sWY/eOUbTrph6NtDUI=', 'Content-Length': '0'} 2024-02-01T10:35:47.021 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fb3105b6b21c1fb1-0065bb7402-4358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:35:47 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:35:47.021 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:35:47.021 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:35:47.022 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:35:47.059 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:35:47.060 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:35:47.120 INFO:teuthology.orchestra.run.smithi148.stderr:No --period given, using current period=5dc4efbd-eccd-47a8-9471-ae4a209ec978 2024-02-01T10:35:47.144 INFO:teuthology.orchestra.run.smithi148.stdout:[{"marker":"1_1706782453.132619_30.1","last_update":"2024-02-01T10:14:13.132619Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1706783747.007639_255.1","last_update":"2024-02-01T10:35:47.007639Z"},{"marker":"1_1706782595.670952_121.1","last_update":"2024-02-01T10:16:35.670952Z"}] 2024-02-01T10:35:47.145 DEBUG:rgw_multi.tests:master meta markers={0: '1_1706782453.132619_30.1', 1: '', 2: '1_1706783747.007639_255.1', 3: '1_1706782595.670952_121.1'} 2024-02-01T10:35:47.145 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-02-01T10:35:47.145 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:35:47.145 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:35:47.145 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:35:47.190 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:35:47.190 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:35:47.244 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:35:47.243+0000 7feb2666d980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:35:47.244 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:35:47.243+0000 7feb2666d980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:35:47.268 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783747.007639_255.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:35:47.007639Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:35:47Z"} 2024-02-01T10:35:47.269 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783747.007639_255.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:35:47.007639Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:35:47Z"} 2024-02-01T10:35:47.269 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706783747.007639_255.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:35:47.007639Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:35:47.269 DEBUG:rgw_multi.tests:log_status={0: '1_1706782453.132619_30.1', 1: '', 2: '1_1706783747.007639_255.1', 3: '1_1706782595.670952_121.1'} 2024-02-01T10:35:47.269 DEBUG:rgw_multi.tests:sync_status={0: '1_1706782453.132619_30.1', 1: '', 2: '1_1706783747.007639_255.1', 3: '1_1706782595.670952_121.1'} 2024-02-01T10:35:47.269 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-02-01T10:35:47.269 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-02-01T10:35:47.269 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:35:47.270 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:35:47.270 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:35:47.313 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:35:47.313 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:35:47.367 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:35:47.371+0000 7f72c46b8980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:35:47.368 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:35:47.371+0000 7f72c46b8980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:35:47.392 INFO:teuthology.orchestra.run.smithi148.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783747.007639_255.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:35:47.007639Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:35:47Z"} 2024-02-01T10:35:47.392 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783747.007639_255.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:35:47.007639Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:35:47Z"} 2024-02-01T10:35:47.392 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706783747.007639_255.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:35:47.007639Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:35:47.393 DEBUG:rgw_multi.tests:log_status={0: '1_1706782453.132619_30.1', 1: '', 2: '1_1706783747.007639_255.1', 3: '1_1706782595.670952_121.1'} 2024-02-01T10:35:47.393 DEBUG:rgw_multi.tests:sync_status={0: '1_1706782453.132619_30.1', 1: '', 2: '1_1706783747.007639_255.1', 3: '1_1706782595.670952_121.1'} 2024-02-01T10:35:47.393 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-02-01T10:35:47.393 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2024-02-01T10:35:47.393 INFO:teuthology.orchestra.run:waiting for 300 2024-02-01T10:35:47.434 INFO:tasks.rgw.c1.client.1.smithi148.stdout:2024-02-01T10:35:47.439+0000 7fd4bbf9f640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 36686) UID: 0 2024-02-01T10:35:47.435 INFO:tasks.rgw.c1.client.1.smithi148.stdout:2024-02-01T10:35:47.439+0000 7fd4bf7e7a40 -1 shutting down 2024-02-01T10:35:51.300 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2024-02-01T10:35:53.496 INFO:tasks.rgw.c1.client.1:Stopped 2024-02-01T10:35:53.496 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:35:53.497 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:35:53.497 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --master --cluster c2 --debug-rgw 0 --debug-ms 0 2024-02-01T10:35:53.540 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:35:53.540 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:35:53.578 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:35:53.578+0000 7f92d4f91980 0 NOTICE: overriding master zone: ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01 2024-02-01T10:35:53.592 INFO:teuthology.orchestra.run.smithi187.stdout:{"id":"01fb6d04-f0ef-49b2-9ebc-c1b530925485","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"1C56UIO2VKZWOSJZ","secret_key":"4X9hEalkm5sn0VTq71G3qlDBF1FRjy9C"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"c7bb4449-4a82-4742-a9c6-c7422f73e18f","notif_pool":"test-zone2.rgw.log:notif"} 2024-02-01T10:35:53.592 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:35:53.592 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:35:53.593 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2024-02-01T10:35:53.675 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:35:53.675 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:35:53.748 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:35:53.746+0000 7f4714989980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:35:53.749 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:35:53.746+0000 7f4714989980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:35:53.793 INFO:teuthology.orchestra.run.smithi187.stdout:{"id":"976daaff-eac6-4c84-bd63-38e72aa959ff","epoch":1,"predecessor_uuid":"5dc4efbd-eccd-47a8-9471-ae4a209ec978","sync_status":["1_1706782453.132619_30.1","","1_1706783747.007639_255.1","1_1706782595.670952_121.1"],"period_map":{"id":"976daaff-eac6-4c84-bd63-38e72aa959ff","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":true,"endpoints":["http://smithi148.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"01fb6d04-f0ef-49b2-9ebc-c1b530925485","zones":[{"id":"01fb6d04-f0ef-49b2-9ebc-c1b530925485","name":"test-zone2","endpoints":["http://smithi187.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]},{"id":"2ea581c3-3f48-49d7-a989-68024c5b8eac","name":"test-zone3","endpoints":["http://smithi148.front.sepia.ceph.com:8001"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]},{"id":"ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01","name":"test-zone1","endpoints":["http://smithi148.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"c7bb4449-4a82-4742-a9c6-c7422f73e18f","sync_policy":{"groups":[]},"enabled_features":["resharding"]}],"short_zone_ids":[{"key":"01fb6d04-f0ef-49b2-9ebc-c1b530925485","val":1553472663},{"key":"2ea581c3-3f48-49d7-a989-68024c5b8eac","val":1173357353},{"key":"ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01","val":3362809119}]},"master_zonegroup":"test-zonegroup","master_zone":"01fb6d04-f0ef-49b2-9ebc-c1b530925485","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"bucket_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"anonymous_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false}},"realm_id":"c7bb4449-4a82-4742-a9c6-c7422f73e18f","realm_epoch":3} 2024-02-01T10:35:53.793 INFO:rgw_multi.tests:Set master zone=test-zone2, waiting 60s for reconfiguration.. 2024-02-01T10:35:56.907 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2024-02-01T10:36:53.795 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=vclfcn-42 2024-02-01T10:36:53.795 DEBUG:boto:path=/vclfcn-42/ 2024-02-01T10:36:53.796 DEBUG:boto:auth_path=/vclfcn-42/ 2024-02-01T10:36:53.796 DEBUG:boto:Method: PUT 2024-02-01T10:36:53.796 DEBUG:boto:Path: /vclfcn-42/ 2024-02-01T10:36:53.796 DEBUG:boto:Data: 2024-02-01T10:36:53.796 DEBUG:boto:Headers: {} 2024-02-01T10:36:53.796 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:36:53.796 DEBUG:boto:Port: 8000 2024-02-01T10:36:53.796 DEBUG:boto:Params: {} 2024-02-01T10:36:53.797 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T10:36:53.797 DEBUG:boto:Token: None 2024-02-01T10:36:53.797 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:36:53 GMT /vclfcn-42/ 2024-02-01T10:36:53.797 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GklNI+aRyc5Cnhw6sdyuPGWiYDM= 2024-02-01T10:36:53.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:36:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GklNI+aRyc5Cnhw6sdyuPGWiYDM=', 'Content-Length': '0'} 2024-02-01T10:36:53.864 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000067a4fac762095a8-0065bb7445-5293-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:36:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:36:53.864 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=vclfcn-43 2024-02-01T10:36:53.865 DEBUG:boto:path=/vclfcn-43/ 2024-02-01T10:36:53.865 DEBUG:boto:auth_path=/vclfcn-43/ 2024-02-01T10:36:53.865 DEBUG:boto:Method: PUT 2024-02-01T10:36:53.865 DEBUG:boto:Path: /vclfcn-43/ 2024-02-01T10:36:53.865 DEBUG:boto:Data: 2024-02-01T10:36:53.865 DEBUG:boto:Headers: {} 2024-02-01T10:36:53.865 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:36:53.865 DEBUG:boto:Port: 8000 2024-02-01T10:36:53.865 DEBUG:boto:Params: {} 2024-02-01T10:36:53.866 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T10:36:53.866 DEBUG:boto:Token: None 2024-02-01T10:36:53.866 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:36:53 GMT /vclfcn-43/ 2024-02-01T10:36:53.866 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:IEpsbnk5LcQMkph6kdqMWmn3Hrs= 2024-02-01T10:36:53.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:36:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:IEpsbnk5LcQMkph6kdqMWmn3Hrs=', 'Content-Length': '0'} 2024-02-01T10:36:53.901 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e2ca4d361400ae5f-0065bb7445-4994-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:36:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:36:53.902 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:36:53.902 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:36:53.902 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:36:53.939 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:36:53.939 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:36:54.002 INFO:teuthology.orchestra.run.smithi187.stderr:No --period given, using current period=976daaff-eac6-4c84-bd63-38e72aa959ff 2024-02-01T10:36:54.025 INFO:teuthology.orchestra.run.smithi187.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1706783813.899233_136.1","last_update":"2024-02-01T10:36:53.899233Z"},{"marker":"","last_update":"0.000000"}] 2024-02-01T10:36:54.026 DEBUG:rgw_multi.tests:master meta markers={0: '', 1: '', 2: '1_1706783813.899233_136.1', 3: ''} 2024-02-01T10:36:54.026 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone1 2024-02-01T10:36:54.026 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:36:54.026 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:36:54.026 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:36:54.067 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:36:54.067 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:36:54.123 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:36:54.129+0000 7f1fe3be0980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:36:54.124 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:36:54.129+0000 7f1fe3be0980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:36:54.147 INFO:teuthology.orchestra.run.smithi148.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"976daaff-eac6-4c84-bd63-38e72aa959ff","realm_epoch":3},"markers":[{"key":0,"val":{"state":0,"marker":"bucket.instance:vclfcn-2:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.2","next_step_marker":"","total_entries":80,"pos":22,"timestamp":"0.000000","realm_epoch":0}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":3}},{"key":2,"val":{"state":1,"marker":"1_1706783813.899233_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:36:53.899233Z","realm_epoch":3}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":3}}]},"full_sync":{"total":80,"complete":22},"current_time":"2024-02-01T10:36:54Z"} 2024-02-01T10:36:54.147 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"976daaff-eac6-4c84-bd63-38e72aa959ff","realm_epoch":3},"markers":[{"key":0,"val":{"state":0,"marker":"bucket.instance:vclfcn-2:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.2","next_step_marker":"","total_entries":80,"pos":22,"timestamp":"0.000000","realm_epoch":0}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":3}},{"key":2,"val":{"state":1,"marker":"1_1706783813.899233_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:36:53.899233Z","realm_epoch":3}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":3}}]},"full_sync":{"total":80,"complete":22},"current_time":"2024-02-01T10:36:54Z"} 2024-02-01T10:36:54.148 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 0, 'marker': 'bucket.instance:vclfcn-2:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.2', 'next_step_marker': '', 'total_entries': 80, 'pos': 22, 'timestamp': '0.000000', 'realm_epoch': 0}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 3}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706783813.899233_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:36:53.899233Z', 'realm_epoch': 3}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 3}}] 2024-02-01T10:36:54.148 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1706783813.899233_136.1', 3: ''} 2024-02-01T10:36:54.148 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1706783813.899233_136.1', 3: ''} 2024-02-01T10:36:54.148 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone1 2024-02-01T10:36:54.148 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:36:54.148 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'modify', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:36:54.148 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone modify --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --master --cluster c1 --debug-rgw 0 --debug-ms 0 2024-02-01T10:36:54.224 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:36:54.224 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:36:54.262 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:36:54.265+0000 7f91e031e980 0 NOTICE: overriding master zone: 01fb6d04-f0ef-49b2-9ebc-c1b530925485 2024-02-01T10:36:54.275 INFO:teuthology.orchestra.run.smithi148.stdout:{"id":"ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01","name":"test-zone1","domain_root":"test-zone1.rgw.meta:root","control_pool":"test-zone1.rgw.control","gc_pool":"test-zone1.rgw.log:gc","lc_pool":"test-zone1.rgw.log:lc","log_pool":"test-zone1.rgw.log","intent_log_pool":"test-zone1.rgw.log:intent","usage_log_pool":"test-zone1.rgw.log:usage","roles_pool":"test-zone1.rgw.meta:roles","reshard_pool":"test-zone1.rgw.log:reshard","user_keys_pool":"test-zone1.rgw.meta:users.keys","user_email_pool":"test-zone1.rgw.meta:users.email","user_swift_pool":"test-zone1.rgw.meta:users.swift","user_uid_pool":"test-zone1.rgw.meta:users.uid","otp_pool":"test-zone1.rgw.otp","system_key":{"access_key":"1C56UIO2VKZWOSJZ","secret_key":"4X9hEalkm5sn0VTq71G3qlDBF1FRjy9C"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone1.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone1.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone1.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"c7bb4449-4a82-4742-a9c6-c7422f73e18f","notif_pool":"test-zone1.rgw.log:notif"} 2024-02-01T10:36:54.275 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:36:54.276 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:36:54.276 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-02-01T10:36:54.352 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:36:54.352 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:36:54.417 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:36:54.421+0000 7f8a1faae980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:36:54.418 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:36:54.421+0000 7f8a1faae980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:36:54.451 INFO:teuthology.orchestra.run.smithi148.stdout:{"id":"f2e6c290-e7aa-4279-b77e-15403b18d61f","epoch":1,"predecessor_uuid":"976daaff-eac6-4c84-bd63-38e72aa959ff","sync_status":["","","1_1706783813.899233_136.1",""],"period_map":{"id":"f2e6c290-e7aa-4279-b77e-15403b18d61f","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":true,"endpoints":["http://smithi148.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01","zones":[{"id":"01fb6d04-f0ef-49b2-9ebc-c1b530925485","name":"test-zone2","endpoints":["http://smithi187.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]},{"id":"2ea581c3-3f48-49d7-a989-68024c5b8eac","name":"test-zone3","endpoints":["http://smithi148.front.sepia.ceph.com:8001"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]},{"id":"ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01","name":"test-zone1","endpoints":["http://smithi148.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"c7bb4449-4a82-4742-a9c6-c7422f73e18f","sync_policy":{"groups":[]},"enabled_features":["resharding"]}],"short_zone_ids":[{"key":"01fb6d04-f0ef-49b2-9ebc-c1b530925485","val":1553472663},{"key":"2ea581c3-3f48-49d7-a989-68024c5b8eac","val":1173357353},{"key":"ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01","val":3362809119}]},"master_zonegroup":"test-zonegroup","master_zone":"ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"bucket_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"anonymous_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false}},"realm_id":"c7bb4449-4a82-4742-a9c6-c7422f73e18f","realm_epoch":4} 2024-02-01T10:36:54.452 INFO:rgw_multi.tests:Set master zone=test-zone1, waiting 60s for reconfiguration.. 2024-02-01T10:37:54.453 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=vclfcn-44 2024-02-01T10:37:54.454 DEBUG:boto:path=/vclfcn-44/ 2024-02-01T10:37:54.454 DEBUG:boto:auth_path=/vclfcn-44/ 2024-02-01T10:37:54.454 DEBUG:boto:Method: PUT 2024-02-01T10:37:54.454 DEBUG:boto:Path: /vclfcn-44/ 2024-02-01T10:37:54.455 DEBUG:boto:Data: 2024-02-01T10:37:54.455 DEBUG:boto:Headers: {} 2024-02-01T10:37:54.455 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:37:54.455 DEBUG:boto:Port: 8000 2024-02-01T10:37:54.455 DEBUG:boto:Params: {} 2024-02-01T10:37:54.455 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T10:37:54.456 DEBUG:boto:Token: None 2024-02-01T10:37:54.456 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:37:54 GMT /vclfcn-44/ 2024-02-01T10:37:54.456 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8J8wCGKc5xMlOnYYqygDQ4v0ExQ= 2024-02-01T10:37:54.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:37:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8J8wCGKc5xMlOnYYqygDQ4v0ExQ=', 'Content-Length': '0'} 2024-02-01T10:37:54.498 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000820da4b2933b440a-0065bb7482-5317-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:37:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:37:54.498 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=vclfcn-45 2024-02-01T10:37:54.498 DEBUG:boto:path=/vclfcn-45/ 2024-02-01T10:37:54.498 DEBUG:boto:auth_path=/vclfcn-45/ 2024-02-01T10:37:54.498 DEBUG:boto:Method: PUT 2024-02-01T10:37:54.498 DEBUG:boto:Path: /vclfcn-45/ 2024-02-01T10:37:54.499 DEBUG:boto:Data: 2024-02-01T10:37:54.499 DEBUG:boto:Headers: {} 2024-02-01T10:37:54.499 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:37:54.499 DEBUG:boto:Port: 8000 2024-02-01T10:37:54.499 DEBUG:boto:Params: {} 2024-02-01T10:37:54.499 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T10:37:54.499 DEBUG:boto:Token: None 2024-02-01T10:37:54.499 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:37:54 GMT /vclfcn-45/ 2024-02-01T10:37:54.500 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:4RP64OdMM+PcrcowavQ4tiu2NIU= 2024-02-01T10:37:54.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:37:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:4RP64OdMM+PcrcowavQ4tiu2NIU=', 'Content-Length': '0'} 2024-02-01T10:37:54.559 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005b2ba8cc3200d4ba-0065bb7482-5004-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:37:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:37:54.559 INFO:tasks.rgw.c1.client.1:Restarting daemon 2024-02-01T10:37:54.560 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2024-02-01T10:37:54.563 INFO:tasks.rgw.c1.client.1:Started 2024-02-01T10:37:54.564 DEBUG:teuthology.orchestra.run.smithi148:> curl http://smithi148.front.sepia.ceph.com:8001 2024-02-01T10:37:54.576 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-02-01T10:37:54.576 INFO:teuthology.orchestra.run.smithi148.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-02-01T10:37:54.576 INFO:teuthology.orchestra.run.smithi148.stderr: Dload Upload Total Spent Left Speed 2024-02-01T10:37:54.577 INFO:teuthology.orchestra.run.smithi148.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2024-02-01T10:37:54.577 INFO:teuthology.orchestra.run.smithi148.stderr:curl: (7) Failed to connect to smithi148.front.sepia.ceph.com port 8001 after 0 ms: Connection refused 2024-02-01T10:37:55.578 DEBUG:teuthology.orchestra.run.smithi148:> curl http://smithi148.front.sepia.ceph.com:8001 2024-02-01T10:37:55.590 INFO:teuthology.orchestra.run.smithi148.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-02-01T10:37:55.590 INFO:teuthology.orchestra.run.smithi148.stderr: Dload Upload Total Spent Left Speed 2024-02-01T10:37:55.593 INFO:teuthology.orchestra.run.smithi148.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 62155 0 --:--:-- --:--:-- --:--:-- 71333 2024-02-01T10:37:55.594 INFO:teuthology.orchestra.run.smithi148.stdout:anonymous 2024-02-01T10:37:55.594 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:37:55.595 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:37:55.595 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:37:55.678 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:37:55.679 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:37:55.735 INFO:teuthology.orchestra.run.smithi148.stderr:No --period given, using current period=f2e6c290-e7aa-4279-b77e-15403b18d61f 2024-02-01T10:37:55.759 INFO:teuthology.orchestra.run.smithi148.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1706783874.544771_104.1","last_update":"2024-02-01T10:37:54.544771Z"},{"marker":"","last_update":"0.000000"}] 2024-02-01T10:37:55.759 DEBUG:rgw_multi.tests:master meta markers={0: '', 1: '', 2: '1_1706783874.544771_104.1', 3: ''} 2024-02-01T10:37:55.759 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-02-01T10:37:55.759 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:37:55.759 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:37:55.760 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:37:55.805 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:37:55.805 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:37:55.860 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:37:55.857+0000 7f484cfd8980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:37:55.860 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:37:55.857+0000 7f484cfd8980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:37:55.883 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"976daaff-eac6-4c84-bd63-38e72aa959ff","realm_epoch":3},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783813.827558_129.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:36:53.827558Z","realm_epoch":3}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:37:55Z"} 2024-02-01T10:37:55.883 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"976daaff-eac6-4c84-bd63-38e72aa959ff","realm_epoch":3},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783813.827558_129.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:36:53.827558Z","realm_epoch":3}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:37:55Z"} 2024-02-01T10:37:55.884 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706783813.827558_129.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:36:53.827558Z', 'realm_epoch': 3}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:37:55.884 WARNING:rgw_multi.tests:zone test-zone2 is syncing realm epoch=3, behind current realm epoch=4 2024-02-01T10:38:00.885 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:00.885 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:00.885 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:38:00.930 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:38:00.930 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:38:00.985 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:38:00.981+0000 7f5787c97980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:38:00.986 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:38:00.985+0000 7f5787c97980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:38:01.010 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783874.544771_104.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:37:54.544771Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:38:00Z"} 2024-02-01T10:38:01.011 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783874.544771_104.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:37:54.544771Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:38:00Z"} 2024-02-01T10:38:01.011 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706783874.544771_104.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:37:54.544771Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:38:01.011 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1706783874.544771_104.1', 3: ''} 2024-02-01T10:38:01.011 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1706783874.544771_104.1', 3: ''} 2024-02-01T10:38:01.011 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-02-01T10:38:01.011 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-02-01T10:38:01.012 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:01.012 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:01.012 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:38:01.057 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:38:01.057 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:38:01.111 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:38:01.116+0000 7f84d18f7980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:38:01.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:38:01.116+0000 7f84d18f7980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:38:01.136 INFO:teuthology.orchestra.run.smithi148.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783874.544771_104.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:37:54.544771Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:38:01Z"} 2024-02-01T10:38:01.136 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783874.544771_104.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:37:54.544771Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:38:01Z"} 2024-02-01T10:38:01.137 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706783874.544771_104.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:37:54.544771Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:38:01.137 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1706783874.544771_104.1', 3: ''} 2024-02-01T10:38:01.137 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1706783874.544771_104.1', 3: ''} 2024-02-01T10:38:01.137 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-02-01T10:38:01.138 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-39 zones={test-zone2, test-zone1} 2024-02-01T10:38:01.138 DEBUG:boto:path=/vclfcn-39/ 2024-02-01T10:38:01.138 DEBUG:boto:auth_path=/vclfcn-39/ 2024-02-01T10:38:01.138 DEBUG:boto:Method: HEAD 2024-02-01T10:38:01.138 DEBUG:boto:Path: /vclfcn-39/ 2024-02-01T10:38:01.138 DEBUG:boto:Data: 2024-02-01T10:38:01.139 DEBUG:boto:Headers: {} 2024-02-01T10:38:01.139 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:01.139 DEBUG:boto:Port: 8000 2024-02-01T10:38:01.139 DEBUG:boto:Params: {} 2024-02-01T10:38:01.139 DEBUG:boto:Token: None 2024-02-01T10:38:01.139 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:01 GMT /vclfcn-39/ 2024-02-01T10:38:01.140 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:UkFOn2bXjiKxX/VqEEoV4Bdq2Ms= 2024-02-01T10:38:01.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:UkFOn2bXjiKxX/VqEEoV4Bdq2Ms=', 'Content-Length': '0'} 2024-02-01T10:38:01.145 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cc039ef6d367ba27-0065bb7489-5004-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:01.146 DEBUG:boto:path=/vclfcn-39/ 2024-02-01T10:38:01.146 DEBUG:boto:auth_path=/vclfcn-39/ 2024-02-01T10:38:01.146 DEBUG:boto:Method: HEAD 2024-02-01T10:38:01.146 DEBUG:boto:Path: /vclfcn-39/ 2024-02-01T10:38:01.146 DEBUG:boto:Data: 2024-02-01T10:38:01.146 DEBUG:boto:Headers: {} 2024-02-01T10:38:01.146 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:01.146 DEBUG:boto:Port: 8000 2024-02-01T10:38:01.147 DEBUG:boto:Params: {} 2024-02-01T10:38:01.147 DEBUG:boto:Token: None 2024-02-01T10:38:01.147 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:01 GMT /vclfcn-39/ 2024-02-01T10:38:01.147 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:UkFOn2bXjiKxX/VqEEoV4Bdq2Ms= 2024-02-01T10:38:01.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:UkFOn2bXjiKxX/VqEEoV4Bdq2Ms=', 'Content-Length': '0'} 2024-02-01T10:38:01.150 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fedfa7dd9f90f7be-0065bb7489-5317-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:01.151 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:38:01.151 DEBUG:boto:path=/vclfcn-39/ 2024-02-01T10:38:01.151 DEBUG:boto:auth_path=/vclfcn-39/ 2024-02-01T10:38:01.151 DEBUG:boto:path=/vclfcn-39/?versions&max-keys=999 2024-02-01T10:38:01.151 DEBUG:boto:auth_path=/vclfcn-39/?versions&max-keys=999 2024-02-01T10:38:01.151 DEBUG:boto:Method: GET 2024-02-01T10:38:01.151 DEBUG:boto:Path: /vclfcn-39/?versions&max-keys=999 2024-02-01T10:38:01.152 DEBUG:boto:Data: 2024-02-01T10:38:01.152 DEBUG:boto:Headers: {} 2024-02-01T10:38:01.152 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:01.152 DEBUG:boto:Port: 8000 2024-02-01T10:38:01.152 DEBUG:boto:Params: {} 2024-02-01T10:38:01.152 DEBUG:boto:Token: None 2024-02-01T10:38:01.155 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:01 GMT /vclfcn-39/?versions 2024-02-01T10:38:01.155 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:4XtsDvhmjfEuS0/9g9LWpAoj4iw= 2024-02-01T10:38:01.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:4XtsDvhmjfEuS0/9g9LWpAoj4iw=', 'Content-Length': '0'} 2024-02-01T10:38:01.159 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf17c7ec0830f321-0065bb7489-5004-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:01.160 DEBUG:boto:b'vclfcn-39999false' 2024-02-01T10:38:01.160 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:38:01.160 DEBUG:boto:path=/vclfcn-39/ 2024-02-01T10:38:01.160 DEBUG:boto:auth_path=/vclfcn-39/ 2024-02-01T10:38:01.161 DEBUG:boto:path=/vclfcn-39/?versions&max-keys=999 2024-02-01T10:38:01.161 DEBUG:boto:auth_path=/vclfcn-39/?versions&max-keys=999 2024-02-01T10:38:01.161 DEBUG:boto:Method: GET 2024-02-01T10:38:01.161 DEBUG:boto:Path: /vclfcn-39/?versions&max-keys=999 2024-02-01T10:38:01.161 DEBUG:boto:Data: 2024-02-01T10:38:01.161 DEBUG:boto:Headers: {} 2024-02-01T10:38:01.162 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:01.162 DEBUG:boto:Port: 8000 2024-02-01T10:38:01.162 DEBUG:boto:Params: {} 2024-02-01T10:38:01.162 DEBUG:boto:Token: None 2024-02-01T10:38:01.162 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:01 GMT /vclfcn-39/?versions 2024-02-01T10:38:01.162 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:4XtsDvhmjfEuS0/9g9LWpAoj4iw= 2024-02-01T10:38:01.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:4XtsDvhmjfEuS0/9g9LWpAoj4iw=', 'Content-Length': '0'} 2024-02-01T10:38:01.166 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ab7ae37f740ca4e-0065bb7489-5317-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:01.166 DEBUG:boto:b'vclfcn-39999false' 2024-02-01T10:38:01.166 DEBUG:boto:path=/vclfcn-39/ 2024-02-01T10:38:01.167 DEBUG:boto:auth_path=/vclfcn-39/ 2024-02-01T10:38:01.167 DEBUG:boto:path=/vclfcn-39/?versions&max-keys=999 2024-02-01T10:38:01.167 DEBUG:boto:auth_path=/vclfcn-39/?versions&max-keys=999 2024-02-01T10:38:01.167 DEBUG:boto:Method: GET 2024-02-01T10:38:01.167 DEBUG:boto:Path: /vclfcn-39/?versions&max-keys=999 2024-02-01T10:38:01.167 DEBUG:boto:Data: 2024-02-01T10:38:01.167 DEBUG:boto:Headers: {} 2024-02-01T10:38:01.167 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:01.167 DEBUG:boto:Port: 8000 2024-02-01T10:38:01.167 DEBUG:boto:Params: {} 2024-02-01T10:38:01.168 DEBUG:boto:Token: None 2024-02-01T10:38:01.168 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:01 GMT /vclfcn-39/?versions 2024-02-01T10:38:01.168 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:4XtsDvhmjfEuS0/9g9LWpAoj4iw= 2024-02-01T10:38:01.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:4XtsDvhmjfEuS0/9g9LWpAoj4iw=', 'Content-Length': '0'} 2024-02-01T10:38:01.172 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000040b3b106a78886b-0065bb7489-5004-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:01.172 DEBUG:boto:b'vclfcn-39999false' 2024-02-01T10:38:01.173 DEBUG:boto:path=/vclfcn-39/ 2024-02-01T10:38:01.173 DEBUG:boto:auth_path=/vclfcn-39/ 2024-02-01T10:38:01.173 DEBUG:boto:path=/vclfcn-39/?versions&max-keys=999 2024-02-01T10:38:01.173 DEBUG:boto:auth_path=/vclfcn-39/?versions&max-keys=999 2024-02-01T10:38:01.173 DEBUG:boto:Method: GET 2024-02-01T10:38:01.173 DEBUG:boto:Path: /vclfcn-39/?versions&max-keys=999 2024-02-01T10:38:01.173 DEBUG:boto:Data: 2024-02-01T10:38:01.173 DEBUG:boto:Headers: {} 2024-02-01T10:38:01.173 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:01.173 DEBUG:boto:Port: 8000 2024-02-01T10:38:01.174 DEBUG:boto:Params: {} 2024-02-01T10:38:01.174 DEBUG:boto:Token: None 2024-02-01T10:38:01.174 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:01 GMT /vclfcn-39/?versions 2024-02-01T10:38:01.174 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:4XtsDvhmjfEuS0/9g9LWpAoj4iw= 2024-02-01T10:38:01.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:4XtsDvhmjfEuS0/9g9LWpAoj4iw=', 'Content-Length': '0'} 2024-02-01T10:38:01.177 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e18a9a631f14c17-0065bb7489-5317-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:01.177 DEBUG:boto:b'vclfcn-39999false' 2024-02-01T10:38:01.178 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-39 zones={test-zone2, test-zone1} 2024-02-01T10:38:01.178 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-39 zones={test-zone3, test-zone1} 2024-02-01T10:38:01.178 DEBUG:boto:path=/vclfcn-39/ 2024-02-01T10:38:01.178 DEBUG:boto:auth_path=/vclfcn-39/ 2024-02-01T10:38:01.178 DEBUG:boto:Method: HEAD 2024-02-01T10:38:01.178 DEBUG:boto:Path: /vclfcn-39/ 2024-02-01T10:38:01.178 DEBUG:boto:Data: 2024-02-01T10:38:01.178 DEBUG:boto:Headers: {} 2024-02-01T10:38:01.179 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:01.179 DEBUG:boto:Port: 8001 2024-02-01T10:38:01.179 DEBUG:boto:Params: {} 2024-02-01T10:38:01.179 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-02-01T10:38:01.179 DEBUG:boto:Token: None 2024-02-01T10:38:01.180 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:01 GMT /vclfcn-39/ 2024-02-01T10:38:01.180 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:UkFOn2bXjiKxX/VqEEoV4Bdq2Ms= 2024-02-01T10:38:01.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:UkFOn2bXjiKxX/VqEEoV4Bdq2Ms=', 'Content-Length': '0'} 2024-02-01T10:38:01.189 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ca5953433370ba69-0065bb7489-5324-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:01.189 DEBUG:boto:path=/vclfcn-39/ 2024-02-01T10:38:01.190 DEBUG:boto:auth_path=/vclfcn-39/ 2024-02-01T10:38:01.190 DEBUG:boto:Method: HEAD 2024-02-01T10:38:01.190 DEBUG:boto:Path: /vclfcn-39/ 2024-02-01T10:38:01.190 DEBUG:boto:Data: 2024-02-01T10:38:01.190 DEBUG:boto:Headers: {} 2024-02-01T10:38:01.190 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:01.190 DEBUG:boto:Port: 8000 2024-02-01T10:38:01.190 DEBUG:boto:Params: {} 2024-02-01T10:38:01.190 DEBUG:boto:Token: None 2024-02-01T10:38:01.191 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:01 GMT /vclfcn-39/ 2024-02-01T10:38:01.191 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:UkFOn2bXjiKxX/VqEEoV4Bdq2Ms= 2024-02-01T10:38:01.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:UkFOn2bXjiKxX/VqEEoV4Bdq2Ms=', 'Content-Length': '0'} 2024-02-01T10:38:01.193 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000adae638c715c6b74-0065bb7489-5317-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:01.193 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:38:01.193 DEBUG:boto:path=/vclfcn-39/ 2024-02-01T10:38:01.194 DEBUG:boto:auth_path=/vclfcn-39/ 2024-02-01T10:38:01.194 DEBUG:boto:path=/vclfcn-39/?versions&max-keys=999 2024-02-01T10:38:01.194 DEBUG:boto:auth_path=/vclfcn-39/?versions&max-keys=999 2024-02-01T10:38:01.194 DEBUG:boto:Method: GET 2024-02-01T10:38:01.194 DEBUG:boto:Path: /vclfcn-39/?versions&max-keys=999 2024-02-01T10:38:01.194 DEBUG:boto:Data: 2024-02-01T10:38:01.194 DEBUG:boto:Headers: {} 2024-02-01T10:38:01.194 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:01.194 DEBUG:boto:Port: 8001 2024-02-01T10:38:01.194 DEBUG:boto:Params: {} 2024-02-01T10:38:01.195 DEBUG:boto:Token: None 2024-02-01T10:38:01.195 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:01 GMT /vclfcn-39/?versions 2024-02-01T10:38:01.195 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:4XtsDvhmjfEuS0/9g9LWpAoj4iw= 2024-02-01T10:38:01.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:4XtsDvhmjfEuS0/9g9LWpAoj4iw=', 'Content-Length': '0'} 2024-02-01T10:38:01.199 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff855ea89c01b016-0065bb7489-5324-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:01.199 DEBUG:boto:b'vclfcn-39999false' 2024-02-01T10:38:01.199 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:38:01.199 DEBUG:boto:path=/vclfcn-39/ 2024-02-01T10:38:01.199 DEBUG:boto:auth_path=/vclfcn-39/ 2024-02-01T10:38:01.199 DEBUG:boto:path=/vclfcn-39/?versions&max-keys=999 2024-02-01T10:38:01.200 DEBUG:boto:auth_path=/vclfcn-39/?versions&max-keys=999 2024-02-01T10:38:01.200 DEBUG:boto:Method: GET 2024-02-01T10:38:01.200 DEBUG:boto:Path: /vclfcn-39/?versions&max-keys=999 2024-02-01T10:38:01.200 DEBUG:boto:Data: 2024-02-01T10:38:01.200 DEBUG:boto:Headers: {} 2024-02-01T10:38:01.200 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:01.200 DEBUG:boto:Port: 8000 2024-02-01T10:38:01.201 DEBUG:boto:Params: {} 2024-02-01T10:38:01.201 DEBUG:boto:Token: None 2024-02-01T10:38:01.201 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:01 GMT /vclfcn-39/?versions 2024-02-01T10:38:01.201 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:4XtsDvhmjfEuS0/9g9LWpAoj4iw= 2024-02-01T10:38:01.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:4XtsDvhmjfEuS0/9g9LWpAoj4iw=', 'Content-Length': '0'} 2024-02-01T10:38:01.204 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040b98c88288d804c-0065bb7489-5317-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:01.244 DEBUG:boto:b'vclfcn-39999false' 2024-02-01T10:38:01.245 DEBUG:boto:path=/vclfcn-39/ 2024-02-01T10:38:01.245 DEBUG:boto:auth_path=/vclfcn-39/ 2024-02-01T10:38:01.245 DEBUG:boto:path=/vclfcn-39/?versions&max-keys=999 2024-02-01T10:38:01.245 DEBUG:boto:auth_path=/vclfcn-39/?versions&max-keys=999 2024-02-01T10:38:01.245 DEBUG:boto:Method: GET 2024-02-01T10:38:01.245 DEBUG:boto:Path: /vclfcn-39/?versions&max-keys=999 2024-02-01T10:38:01.245 DEBUG:boto:Data: 2024-02-01T10:38:01.245 DEBUG:boto:Headers: {} 2024-02-01T10:38:01.245 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:01.245 DEBUG:boto:Port: 8001 2024-02-01T10:38:01.246 DEBUG:boto:Params: {} 2024-02-01T10:38:01.246 DEBUG:boto:Token: None 2024-02-01T10:38:01.246 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:01 GMT /vclfcn-39/?versions 2024-02-01T10:38:01.246 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:4XtsDvhmjfEuS0/9g9LWpAoj4iw= 2024-02-01T10:38:01.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:4XtsDvhmjfEuS0/9g9LWpAoj4iw=', 'Content-Length': '0'} 2024-02-01T10:38:01.250 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c2c7ef8a9d6dfff-0065bb7489-5324-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:01.251 DEBUG:boto:b'vclfcn-39999false' 2024-02-01T10:38:01.251 DEBUG:boto:path=/vclfcn-39/ 2024-02-01T10:38:01.251 DEBUG:boto:auth_path=/vclfcn-39/ 2024-02-01T10:38:01.251 DEBUG:boto:path=/vclfcn-39/?versions&max-keys=999 2024-02-01T10:38:01.251 DEBUG:boto:auth_path=/vclfcn-39/?versions&max-keys=999 2024-02-01T10:38:01.251 DEBUG:boto:Method: GET 2024-02-01T10:38:01.252 DEBUG:boto:Path: /vclfcn-39/?versions&max-keys=999 2024-02-01T10:38:01.252 DEBUG:boto:Data: 2024-02-01T10:38:01.252 DEBUG:boto:Headers: {} 2024-02-01T10:38:01.252 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:01.252 DEBUG:boto:Port: 8000 2024-02-01T10:38:01.252 DEBUG:boto:Params: {} 2024-02-01T10:38:01.252 DEBUG:boto:Token: None 2024-02-01T10:38:01.253 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:01 GMT /vclfcn-39/?versions 2024-02-01T10:38:01.253 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:4XtsDvhmjfEuS0/9g9LWpAoj4iw= 2024-02-01T10:38:01.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:4XtsDvhmjfEuS0/9g9LWpAoj4iw=', 'Content-Length': '0'} 2024-02-01T10:38:01.257 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000901c9ad84b85f8a-0065bb7489-5317-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:01.257 DEBUG:boto:b'vclfcn-39999false' 2024-02-01T10:38:01.257 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-39 zones={test-zone3, test-zone1} 2024-02-01T10:38:01.257 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-39 zones={test-zone1, test-zone2} 2024-02-01T10:38:01.257 DEBUG:boto:path=/vclfcn-39/ 2024-02-01T10:38:01.257 DEBUG:boto:auth_path=/vclfcn-39/ 2024-02-01T10:38:01.258 DEBUG:boto:Method: HEAD 2024-02-01T10:38:01.258 DEBUG:boto:Path: /vclfcn-39/ 2024-02-01T10:38:01.258 DEBUG:boto:Data: 2024-02-01T10:38:01.258 DEBUG:boto:Headers: {} 2024-02-01T10:38:01.258 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:01.258 DEBUG:boto:Port: 8000 2024-02-01T10:38:01.258 DEBUG:boto:Params: {} 2024-02-01T10:38:01.258 DEBUG:boto:Token: None 2024-02-01T10:38:01.258 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:01 GMT /vclfcn-39/ 2024-02-01T10:38:01.259 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:UkFOn2bXjiKxX/VqEEoV4Bdq2Ms= 2024-02-01T10:38:01.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:UkFOn2bXjiKxX/VqEEoV4Bdq2Ms=', 'Content-Length': '0'} 2024-02-01T10:38:01.262 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000da71b1b9d2d2fe52-0065bb7489-5317-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:01.263 DEBUG:boto:path=/vclfcn-39/ 2024-02-01T10:38:01.263 DEBUG:boto:auth_path=/vclfcn-39/ 2024-02-01T10:38:01.263 DEBUG:boto:Method: HEAD 2024-02-01T10:38:01.263 DEBUG:boto:Path: /vclfcn-39/ 2024-02-01T10:38:01.263 DEBUG:boto:Data: 2024-02-01T10:38:01.263 DEBUG:boto:Headers: {} 2024-02-01T10:38:01.263 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:01.263 DEBUG:boto:Port: 8000 2024-02-01T10:38:01.263 DEBUG:boto:Params: {} 2024-02-01T10:38:01.264 DEBUG:boto:Token: None 2024-02-01T10:38:01.264 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:01 GMT /vclfcn-39/ 2024-02-01T10:38:01.264 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:UkFOn2bXjiKxX/VqEEoV4Bdq2Ms= 2024-02-01T10:38:01.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:UkFOn2bXjiKxX/VqEEoV4Bdq2Ms=', 'Content-Length': '0'} 2024-02-01T10:38:01.268 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008603cff8b0889c56-0065bb7489-5004-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:01.268 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:38:01.268 DEBUG:boto:path=/vclfcn-39/ 2024-02-01T10:38:01.268 DEBUG:boto:auth_path=/vclfcn-39/ 2024-02-01T10:38:01.268 DEBUG:boto:path=/vclfcn-39/?versions&max-keys=999 2024-02-01T10:38:01.268 DEBUG:boto:auth_path=/vclfcn-39/?versions&max-keys=999 2024-02-01T10:38:01.269 DEBUG:boto:Method: GET 2024-02-01T10:38:01.269 DEBUG:boto:Path: /vclfcn-39/?versions&max-keys=999 2024-02-01T10:38:01.269 DEBUG:boto:Data: 2024-02-01T10:38:01.269 DEBUG:boto:Headers: {} 2024-02-01T10:38:01.269 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:01.269 DEBUG:boto:Port: 8000 2024-02-01T10:38:01.269 DEBUG:boto:Params: {} 2024-02-01T10:38:01.269 DEBUG:boto:Token: None 2024-02-01T10:38:01.269 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:01 GMT /vclfcn-39/?versions 2024-02-01T10:38:01.270 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:4XtsDvhmjfEuS0/9g9LWpAoj4iw= 2024-02-01T10:38:01.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:4XtsDvhmjfEuS0/9g9LWpAoj4iw=', 'Content-Length': '0'} 2024-02-01T10:38:01.273 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f07f32f91bddefcc-0065bb7489-5317-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:01.316 DEBUG:boto:b'vclfcn-39999false' 2024-02-01T10:38:01.317 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:38:01.318 DEBUG:boto:path=/vclfcn-39/ 2024-02-01T10:38:01.318 DEBUG:boto:auth_path=/vclfcn-39/ 2024-02-01T10:38:01.318 DEBUG:boto:path=/vclfcn-39/?versions&max-keys=999 2024-02-01T10:38:01.318 DEBUG:boto:auth_path=/vclfcn-39/?versions&max-keys=999 2024-02-01T10:38:01.318 DEBUG:boto:Method: GET 2024-02-01T10:38:01.318 DEBUG:boto:Path: /vclfcn-39/?versions&max-keys=999 2024-02-01T10:38:01.318 DEBUG:boto:Data: 2024-02-01T10:38:01.318 DEBUG:boto:Headers: {} 2024-02-01T10:38:01.319 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:01.319 DEBUG:boto:Port: 8000 2024-02-01T10:38:01.319 DEBUG:boto:Params: {} 2024-02-01T10:38:01.319 DEBUG:boto:Token: None 2024-02-01T10:38:01.319 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:01 GMT /vclfcn-39/?versions 2024-02-01T10:38:01.319 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:4XtsDvhmjfEuS0/9g9LWpAoj4iw= 2024-02-01T10:38:01.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:4XtsDvhmjfEuS0/9g9LWpAoj4iw=', 'Content-Length': '0'} 2024-02-01T10:38:01.323 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb4d01efec4422de-0065bb7489-5004-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:01.324 DEBUG:boto:b'vclfcn-39999false' 2024-02-01T10:38:01.324 DEBUG:boto:path=/vclfcn-39/ 2024-02-01T10:38:01.325 DEBUG:boto:auth_path=/vclfcn-39/ 2024-02-01T10:38:01.325 DEBUG:boto:path=/vclfcn-39/?versions&max-keys=999 2024-02-01T10:38:01.325 DEBUG:boto:auth_path=/vclfcn-39/?versions&max-keys=999 2024-02-01T10:38:01.325 DEBUG:boto:Method: GET 2024-02-01T10:38:01.325 DEBUG:boto:Path: /vclfcn-39/?versions&max-keys=999 2024-02-01T10:38:01.325 DEBUG:boto:Data: 2024-02-01T10:38:01.325 DEBUG:boto:Headers: {} 2024-02-01T10:38:01.325 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:01.325 DEBUG:boto:Port: 8000 2024-02-01T10:38:01.325 DEBUG:boto:Params: {} 2024-02-01T10:38:01.326 DEBUG:boto:Token: None 2024-02-01T10:38:01.326 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:01 GMT /vclfcn-39/?versions 2024-02-01T10:38:01.326 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:4XtsDvhmjfEuS0/9g9LWpAoj4iw= 2024-02-01T10:38:01.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:4XtsDvhmjfEuS0/9g9LWpAoj4iw=', 'Content-Length': '0'} 2024-02-01T10:38:01.330 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0d77a5a01e8c24f-0065bb7489-5317-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:01.330 DEBUG:boto:b'vclfcn-39999false' 2024-02-01T10:38:01.331 DEBUG:boto:path=/vclfcn-39/ 2024-02-01T10:38:01.331 DEBUG:boto:auth_path=/vclfcn-39/ 2024-02-01T10:38:01.331 DEBUG:boto:path=/vclfcn-39/?versions&max-keys=999 2024-02-01T10:38:01.331 DEBUG:boto:auth_path=/vclfcn-39/?versions&max-keys=999 2024-02-01T10:38:01.331 DEBUG:boto:Method: GET 2024-02-01T10:38:01.331 DEBUG:boto:Path: /vclfcn-39/?versions&max-keys=999 2024-02-01T10:38:01.331 DEBUG:boto:Data: 2024-02-01T10:38:01.332 DEBUG:boto:Headers: {} 2024-02-01T10:38:01.332 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:01.332 DEBUG:boto:Port: 8000 2024-02-01T10:38:01.332 DEBUG:boto:Params: {} 2024-02-01T10:38:01.332 DEBUG:boto:Token: None 2024-02-01T10:38:01.333 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:01 GMT /vclfcn-39/?versions 2024-02-01T10:38:01.333 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:4XtsDvhmjfEuS0/9g9LWpAoj4iw= 2024-02-01T10:38:01.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:4XtsDvhmjfEuS0/9g9LWpAoj4iw=', 'Content-Length': '0'} 2024-02-01T10:38:01.338 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae30aab9c17105fb-0065bb7489-5004-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:01.338 DEBUG:boto:b'vclfcn-39999false' 2024-02-01T10:38:01.339 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-39 zones={test-zone1, test-zone2} 2024-02-01T10:38:01.339 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-39 zones={test-zone3, test-zone2} 2024-02-01T10:38:01.339 DEBUG:boto:path=/vclfcn-39/ 2024-02-01T10:38:01.339 DEBUG:boto:auth_path=/vclfcn-39/ 2024-02-01T10:38:01.339 DEBUG:boto:Method: HEAD 2024-02-01T10:38:01.339 DEBUG:boto:Path: /vclfcn-39/ 2024-02-01T10:38:01.339 DEBUG:boto:Data: 2024-02-01T10:38:01.339 DEBUG:boto:Headers: {} 2024-02-01T10:38:01.339 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:01.339 DEBUG:boto:Port: 8001 2024-02-01T10:38:01.339 DEBUG:boto:Params: {} 2024-02-01T10:38:01.340 DEBUG:boto:Token: None 2024-02-01T10:38:01.340 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:01 GMT /vclfcn-39/ 2024-02-01T10:38:01.340 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:UkFOn2bXjiKxX/VqEEoV4Bdq2Ms= 2024-02-01T10:38:01.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:UkFOn2bXjiKxX/VqEEoV4Bdq2Ms=', 'Content-Length': '0'} 2024-02-01T10:38:01.344 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007572fcf04aadefbc-0065bb7489-5324-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:01.345 DEBUG:boto:path=/vclfcn-39/ 2024-02-01T10:38:01.345 DEBUG:boto:auth_path=/vclfcn-39/ 2024-02-01T10:38:01.345 DEBUG:boto:Method: HEAD 2024-02-01T10:38:01.345 DEBUG:boto:Path: /vclfcn-39/ 2024-02-01T10:38:01.345 DEBUG:boto:Data: 2024-02-01T10:38:01.345 DEBUG:boto:Headers: {} 2024-02-01T10:38:01.345 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:01.345 DEBUG:boto:Port: 8000 2024-02-01T10:38:01.346 DEBUG:boto:Params: {} 2024-02-01T10:38:01.346 DEBUG:boto:Token: None 2024-02-01T10:38:01.346 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:01 GMT /vclfcn-39/ 2024-02-01T10:38:01.346 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:UkFOn2bXjiKxX/VqEEoV4Bdq2Ms= 2024-02-01T10:38:01.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:UkFOn2bXjiKxX/VqEEoV4Bdq2Ms=', 'Content-Length': '0'} 2024-02-01T10:38:01.350 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000873dbefeb097c754-0065bb7489-5004-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:01.351 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:38:01.351 DEBUG:boto:path=/vclfcn-39/ 2024-02-01T10:38:01.351 DEBUG:boto:auth_path=/vclfcn-39/ 2024-02-01T10:38:01.351 DEBUG:boto:path=/vclfcn-39/?versions&max-keys=999 2024-02-01T10:38:01.351 DEBUG:boto:auth_path=/vclfcn-39/?versions&max-keys=999 2024-02-01T10:38:01.351 DEBUG:boto:Method: GET 2024-02-01T10:38:01.351 DEBUG:boto:Path: /vclfcn-39/?versions&max-keys=999 2024-02-01T10:38:01.352 DEBUG:boto:Data: 2024-02-01T10:38:01.352 DEBUG:boto:Headers: {} 2024-02-01T10:38:01.352 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:01.352 DEBUG:boto:Port: 8001 2024-02-01T10:38:01.352 DEBUG:boto:Params: {} 2024-02-01T10:38:01.352 DEBUG:boto:Token: None 2024-02-01T10:38:01.352 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:01 GMT /vclfcn-39/?versions 2024-02-01T10:38:01.352 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:4XtsDvhmjfEuS0/9g9LWpAoj4iw= 2024-02-01T10:38:01.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:4XtsDvhmjfEuS0/9g9LWpAoj4iw=', 'Content-Length': '0'} 2024-02-01T10:38:01.360 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041c8d32c481dd8ca-0065bb7489-5324-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:01.360 DEBUG:boto:b'vclfcn-39999false' 2024-02-01T10:38:01.361 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:38:01.361 DEBUG:boto:path=/vclfcn-39/ 2024-02-01T10:38:01.362 DEBUG:boto:auth_path=/vclfcn-39/ 2024-02-01T10:38:01.362 DEBUG:boto:path=/vclfcn-39/?versions&max-keys=999 2024-02-01T10:38:01.362 DEBUG:boto:auth_path=/vclfcn-39/?versions&max-keys=999 2024-02-01T10:38:01.363 DEBUG:boto:Method: GET 2024-02-01T10:38:01.363 DEBUG:boto:Path: /vclfcn-39/?versions&max-keys=999 2024-02-01T10:38:01.363 DEBUG:boto:Data: 2024-02-01T10:38:01.363 DEBUG:boto:Headers: {} 2024-02-01T10:38:01.364 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:01.364 DEBUG:boto:Port: 8000 2024-02-01T10:38:01.364 DEBUG:boto:Params: {} 2024-02-01T10:38:01.364 DEBUG:boto:Token: None 2024-02-01T10:38:01.365 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:01 GMT /vclfcn-39/?versions 2024-02-01T10:38:01.365 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:4XtsDvhmjfEuS0/9g9LWpAoj4iw= 2024-02-01T10:38:01.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:4XtsDvhmjfEuS0/9g9LWpAoj4iw=', 'Content-Length': '0'} 2024-02-01T10:38:01.372 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4a3b5371d10e116-0065bb7489-5004-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:01.412 DEBUG:boto:b'vclfcn-39999false' 2024-02-01T10:38:01.413 DEBUG:boto:path=/vclfcn-39/ 2024-02-01T10:38:01.413 DEBUG:boto:auth_path=/vclfcn-39/ 2024-02-01T10:38:01.413 DEBUG:boto:path=/vclfcn-39/?versions&max-keys=999 2024-02-01T10:38:01.413 DEBUG:boto:auth_path=/vclfcn-39/?versions&max-keys=999 2024-02-01T10:38:01.413 DEBUG:boto:Method: GET 2024-02-01T10:38:01.413 DEBUG:boto:Path: /vclfcn-39/?versions&max-keys=999 2024-02-01T10:38:01.414 DEBUG:boto:Data: 2024-02-01T10:38:01.414 DEBUG:boto:Headers: {} 2024-02-01T10:38:01.414 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:01.414 DEBUG:boto:Port: 8001 2024-02-01T10:38:01.414 DEBUG:boto:Params: {} 2024-02-01T10:38:01.414 DEBUG:boto:Token: None 2024-02-01T10:38:01.414 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:01 GMT /vclfcn-39/?versions 2024-02-01T10:38:01.414 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:4XtsDvhmjfEuS0/9g9LWpAoj4iw= 2024-02-01T10:38:01.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:4XtsDvhmjfEuS0/9g9LWpAoj4iw=', 'Content-Length': '0'} 2024-02-01T10:38:01.419 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed0d63c65ff1302b-0065bb7489-5324-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:01.419 DEBUG:boto:b'vclfcn-39999false' 2024-02-01T10:38:01.419 DEBUG:boto:path=/vclfcn-39/ 2024-02-01T10:38:01.419 DEBUG:boto:auth_path=/vclfcn-39/ 2024-02-01T10:38:01.419 DEBUG:boto:path=/vclfcn-39/?versions&max-keys=999 2024-02-01T10:38:01.419 DEBUG:boto:auth_path=/vclfcn-39/?versions&max-keys=999 2024-02-01T10:38:01.420 DEBUG:boto:Method: GET 2024-02-01T10:38:01.420 DEBUG:boto:Path: /vclfcn-39/?versions&max-keys=999 2024-02-01T10:38:01.420 DEBUG:boto:Data: 2024-02-01T10:38:01.420 DEBUG:boto:Headers: {} 2024-02-01T10:38:01.420 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:01.420 DEBUG:boto:Port: 8000 2024-02-01T10:38:01.420 DEBUG:boto:Params: {} 2024-02-01T10:38:01.420 DEBUG:boto:Token: None 2024-02-01T10:38:01.420 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:01 GMT /vclfcn-39/?versions 2024-02-01T10:38:01.421 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:4XtsDvhmjfEuS0/9g9LWpAoj4iw= 2024-02-01T10:38:01.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:4XtsDvhmjfEuS0/9g9LWpAoj4iw=', 'Content-Length': '0'} 2024-02-01T10:38:01.424 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009606afc3c9e28016-0065bb7489-5004-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:01.425 DEBUG:boto:b'vclfcn-39999false' 2024-02-01T10:38:01.425 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-39 zones={test-zone3, test-zone2} 2024-02-01T10:38:01.425 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-39 zones={test-zone1, test-zone3} 2024-02-01T10:38:01.426 DEBUG:boto:path=/vclfcn-39/ 2024-02-01T10:38:01.426 DEBUG:boto:auth_path=/vclfcn-39/ 2024-02-01T10:38:01.426 DEBUG:boto:Method: HEAD 2024-02-01T10:38:01.426 DEBUG:boto:Path: /vclfcn-39/ 2024-02-01T10:38:01.426 DEBUG:boto:Data: 2024-02-01T10:38:01.426 DEBUG:boto:Headers: {} 2024-02-01T10:38:01.426 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:01.426 DEBUG:boto:Port: 8000 2024-02-01T10:38:01.427 DEBUG:boto:Params: {} 2024-02-01T10:38:01.427 DEBUG:boto:Token: None 2024-02-01T10:38:01.427 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:01 GMT /vclfcn-39/ 2024-02-01T10:38:01.427 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:UkFOn2bXjiKxX/VqEEoV4Bdq2Ms= 2024-02-01T10:38:01.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:UkFOn2bXjiKxX/VqEEoV4Bdq2Ms=', 'Content-Length': '0'} 2024-02-01T10:38:01.431 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000351e4689d8d9bbac-0065bb7489-5317-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:01.431 DEBUG:boto:path=/vclfcn-39/ 2024-02-01T10:38:01.431 DEBUG:boto:auth_path=/vclfcn-39/ 2024-02-01T10:38:01.431 DEBUG:boto:Method: HEAD 2024-02-01T10:38:01.432 DEBUG:boto:Path: /vclfcn-39/ 2024-02-01T10:38:01.432 DEBUG:boto:Data: 2024-02-01T10:38:01.432 DEBUG:boto:Headers: {} 2024-02-01T10:38:01.432 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:01.432 DEBUG:boto:Port: 8001 2024-02-01T10:38:01.432 DEBUG:boto:Params: {} 2024-02-01T10:38:01.432 DEBUG:boto:Token: None 2024-02-01T10:38:01.432 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:01 GMT /vclfcn-39/ 2024-02-01T10:38:01.432 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:UkFOn2bXjiKxX/VqEEoV4Bdq2Ms= 2024-02-01T10:38:01.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:UkFOn2bXjiKxX/VqEEoV4Bdq2Ms=', 'Content-Length': '0'} 2024-02-01T10:38:01.436 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002f0840cb518b90dd-0065bb7489-5324-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:01.437 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:38:01.437 DEBUG:boto:path=/vclfcn-39/ 2024-02-01T10:38:01.437 DEBUG:boto:auth_path=/vclfcn-39/ 2024-02-01T10:38:01.437 DEBUG:boto:path=/vclfcn-39/?versions&max-keys=999 2024-02-01T10:38:01.437 DEBUG:boto:auth_path=/vclfcn-39/?versions&max-keys=999 2024-02-01T10:38:01.437 DEBUG:boto:Method: GET 2024-02-01T10:38:01.437 DEBUG:boto:Path: /vclfcn-39/?versions&max-keys=999 2024-02-01T10:38:01.438 DEBUG:boto:Data: 2024-02-01T10:38:01.438 DEBUG:boto:Headers: {} 2024-02-01T10:38:01.438 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:01.438 DEBUG:boto:Port: 8000 2024-02-01T10:38:01.438 DEBUG:boto:Params: {} 2024-02-01T10:38:01.438 DEBUG:boto:Token: None 2024-02-01T10:38:01.439 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:01 GMT /vclfcn-39/?versions 2024-02-01T10:38:01.439 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:4XtsDvhmjfEuS0/9g9LWpAoj4iw= 2024-02-01T10:38:01.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:4XtsDvhmjfEuS0/9g9LWpAoj4iw=', 'Content-Length': '0'} 2024-02-01T10:38:01.444 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b0765837917fca9-0065bb7489-5317-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:01.445 DEBUG:boto:b'vclfcn-39999false' 2024-02-01T10:38:01.445 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:38:01.445 DEBUG:boto:path=/vclfcn-39/ 2024-02-01T10:38:01.446 DEBUG:boto:auth_path=/vclfcn-39/ 2024-02-01T10:38:01.446 DEBUG:boto:path=/vclfcn-39/?versions&max-keys=999 2024-02-01T10:38:01.446 DEBUG:boto:auth_path=/vclfcn-39/?versions&max-keys=999 2024-02-01T10:38:01.446 DEBUG:boto:Method: GET 2024-02-01T10:38:01.446 DEBUG:boto:Path: /vclfcn-39/?versions&max-keys=999 2024-02-01T10:38:01.446 DEBUG:boto:Data: 2024-02-01T10:38:01.446 DEBUG:boto:Headers: {} 2024-02-01T10:38:01.446 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:01.447 DEBUG:boto:Port: 8001 2024-02-01T10:38:01.447 DEBUG:boto:Params: {} 2024-02-01T10:38:01.447 DEBUG:boto:Token: None 2024-02-01T10:38:01.447 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:01 GMT /vclfcn-39/?versions 2024-02-01T10:38:01.447 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:4XtsDvhmjfEuS0/9g9LWpAoj4iw= 2024-02-01T10:38:01.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:4XtsDvhmjfEuS0/9g9LWpAoj4iw=', 'Content-Length': '0'} 2024-02-01T10:38:01.452 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb975b122d5d9875-0065bb7489-5324-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:01.492 DEBUG:boto:b'vclfcn-39999false' 2024-02-01T10:38:01.493 DEBUG:boto:path=/vclfcn-39/ 2024-02-01T10:38:01.493 DEBUG:boto:auth_path=/vclfcn-39/ 2024-02-01T10:38:01.493 DEBUG:boto:path=/vclfcn-39/?versions&max-keys=999 2024-02-01T10:38:01.493 DEBUG:boto:auth_path=/vclfcn-39/?versions&max-keys=999 2024-02-01T10:38:01.494 DEBUG:boto:Method: GET 2024-02-01T10:38:01.494 DEBUG:boto:Path: /vclfcn-39/?versions&max-keys=999 2024-02-01T10:38:01.494 DEBUG:boto:Data: 2024-02-01T10:38:01.494 DEBUG:boto:Headers: {} 2024-02-01T10:38:01.494 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:01.494 DEBUG:boto:Port: 8000 2024-02-01T10:38:01.494 DEBUG:boto:Params: {} 2024-02-01T10:38:01.494 DEBUG:boto:Token: None 2024-02-01T10:38:01.494 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:01 GMT /vclfcn-39/?versions 2024-02-01T10:38:01.495 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:4XtsDvhmjfEuS0/9g9LWpAoj4iw= 2024-02-01T10:38:01.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:4XtsDvhmjfEuS0/9g9LWpAoj4iw=', 'Content-Length': '0'} 2024-02-01T10:38:01.500 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9cbb38b51c5c920-0065bb7489-5317-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:01.501 DEBUG:boto:b'vclfcn-39999false' 2024-02-01T10:38:01.502 DEBUG:boto:path=/vclfcn-39/ 2024-02-01T10:38:01.502 DEBUG:boto:auth_path=/vclfcn-39/ 2024-02-01T10:38:01.502 DEBUG:boto:path=/vclfcn-39/?versions&max-keys=999 2024-02-01T10:38:01.502 DEBUG:boto:auth_path=/vclfcn-39/?versions&max-keys=999 2024-02-01T10:38:01.502 DEBUG:boto:Method: GET 2024-02-01T10:38:01.502 DEBUG:boto:Path: /vclfcn-39/?versions&max-keys=999 2024-02-01T10:38:01.502 DEBUG:boto:Data: 2024-02-01T10:38:01.502 DEBUG:boto:Headers: {} 2024-02-01T10:38:01.502 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:01.502 DEBUG:boto:Port: 8001 2024-02-01T10:38:01.502 DEBUG:boto:Params: {} 2024-02-01T10:38:01.503 DEBUG:boto:Token: None 2024-02-01T10:38:01.503 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:01 GMT /vclfcn-39/?versions 2024-02-01T10:38:01.503 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:4XtsDvhmjfEuS0/9g9LWpAoj4iw= 2024-02-01T10:38:01.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:4XtsDvhmjfEuS0/9g9LWpAoj4iw=', 'Content-Length': '0'} 2024-02-01T10:38:01.507 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f4ec91bcdaf9a4ff-0065bb7489-5324-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:01.508 DEBUG:boto:b'vclfcn-39999false' 2024-02-01T10:38:01.508 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-39 zones={test-zone1, test-zone3} 2024-02-01T10:38:01.509 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-39 zones={test-zone2, test-zone3} 2024-02-01T10:38:01.509 DEBUG:boto:path=/vclfcn-39/ 2024-02-01T10:38:01.509 DEBUG:boto:auth_path=/vclfcn-39/ 2024-02-01T10:38:01.509 DEBUG:boto:Method: HEAD 2024-02-01T10:38:01.509 DEBUG:boto:Path: /vclfcn-39/ 2024-02-01T10:38:01.509 DEBUG:boto:Data: 2024-02-01T10:38:01.509 DEBUG:boto:Headers: {} 2024-02-01T10:38:01.509 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:01.510 DEBUG:boto:Port: 8000 2024-02-01T10:38:01.510 DEBUG:boto:Params: {} 2024-02-01T10:38:01.510 DEBUG:boto:Token: None 2024-02-01T10:38:01.510 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:01 GMT /vclfcn-39/ 2024-02-01T10:38:01.510 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:UkFOn2bXjiKxX/VqEEoV4Bdq2Ms= 2024-02-01T10:38:01.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:UkFOn2bXjiKxX/VqEEoV4Bdq2Ms=', 'Content-Length': '0'} 2024-02-01T10:38:01.514 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000d149e58d4fd9fcb-0065bb7489-5004-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:01.515 DEBUG:boto:path=/vclfcn-39/ 2024-02-01T10:38:01.515 DEBUG:boto:auth_path=/vclfcn-39/ 2024-02-01T10:38:01.515 DEBUG:boto:Method: HEAD 2024-02-01T10:38:01.515 DEBUG:boto:Path: /vclfcn-39/ 2024-02-01T10:38:01.515 DEBUG:boto:Data: 2024-02-01T10:38:01.515 DEBUG:boto:Headers: {} 2024-02-01T10:38:01.515 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:01.515 DEBUG:boto:Port: 8001 2024-02-01T10:38:01.516 DEBUG:boto:Params: {} 2024-02-01T10:38:01.516 DEBUG:boto:Token: None 2024-02-01T10:38:01.516 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:01 GMT /vclfcn-39/ 2024-02-01T10:38:01.516 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:UkFOn2bXjiKxX/VqEEoV4Bdq2Ms= 2024-02-01T10:38:01.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:UkFOn2bXjiKxX/VqEEoV4Bdq2Ms=', 'Content-Length': '0'} 2024-02-01T10:38:01.521 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a4df092fca2eac88-0065bb7489-5324-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:01.521 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:38:01.521 DEBUG:boto:path=/vclfcn-39/ 2024-02-01T10:38:01.521 DEBUG:boto:auth_path=/vclfcn-39/ 2024-02-01T10:38:01.521 DEBUG:boto:path=/vclfcn-39/?versions&max-keys=999 2024-02-01T10:38:01.522 DEBUG:boto:auth_path=/vclfcn-39/?versions&max-keys=999 2024-02-01T10:38:01.522 DEBUG:boto:Method: GET 2024-02-01T10:38:01.522 DEBUG:boto:Path: /vclfcn-39/?versions&max-keys=999 2024-02-01T10:38:01.522 DEBUG:boto:Data: 2024-02-01T10:38:01.522 DEBUG:boto:Headers: {} 2024-02-01T10:38:01.522 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:01.522 DEBUG:boto:Port: 8000 2024-02-01T10:38:01.522 DEBUG:boto:Params: {} 2024-02-01T10:38:01.522 DEBUG:boto:Token: None 2024-02-01T10:38:01.523 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:01 GMT /vclfcn-39/?versions 2024-02-01T10:38:01.523 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:4XtsDvhmjfEuS0/9g9LWpAoj4iw= 2024-02-01T10:38:01.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:4XtsDvhmjfEuS0/9g9LWpAoj4iw=', 'Content-Length': '0'} 2024-02-01T10:38:01.527 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000007a7617ddd45b046-0065bb7489-5004-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:01.527 DEBUG:boto:b'vclfcn-39999false' 2024-02-01T10:38:01.528 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:38:01.528 DEBUG:boto:path=/vclfcn-39/ 2024-02-01T10:38:01.528 DEBUG:boto:auth_path=/vclfcn-39/ 2024-02-01T10:38:01.528 DEBUG:boto:path=/vclfcn-39/?versions&max-keys=999 2024-02-01T10:38:01.528 DEBUG:boto:auth_path=/vclfcn-39/?versions&max-keys=999 2024-02-01T10:38:01.528 DEBUG:boto:Method: GET 2024-02-01T10:38:01.528 DEBUG:boto:Path: /vclfcn-39/?versions&max-keys=999 2024-02-01T10:38:01.529 DEBUG:boto:Data: 2024-02-01T10:38:01.529 DEBUG:boto:Headers: {} 2024-02-01T10:38:01.529 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:01.529 DEBUG:boto:Port: 8001 2024-02-01T10:38:01.529 DEBUG:boto:Params: {} 2024-02-01T10:38:01.529 DEBUG:boto:Token: None 2024-02-01T10:38:01.529 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:01 GMT /vclfcn-39/?versions 2024-02-01T10:38:01.529 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:4XtsDvhmjfEuS0/9g9LWpAoj4iw= 2024-02-01T10:38:01.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:4XtsDvhmjfEuS0/9g9LWpAoj4iw=', 'Content-Length': '0'} 2024-02-01T10:38:01.535 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000061a1e2540f9e14ab-0065bb7489-5324-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:01.576 DEBUG:boto:b'vclfcn-39999false' 2024-02-01T10:38:01.577 DEBUG:boto:path=/vclfcn-39/ 2024-02-01T10:38:01.577 DEBUG:boto:auth_path=/vclfcn-39/ 2024-02-01T10:38:01.577 DEBUG:boto:path=/vclfcn-39/?versions&max-keys=999 2024-02-01T10:38:01.577 DEBUG:boto:auth_path=/vclfcn-39/?versions&max-keys=999 2024-02-01T10:38:01.577 DEBUG:boto:Method: GET 2024-02-01T10:38:01.577 DEBUG:boto:Path: /vclfcn-39/?versions&max-keys=999 2024-02-01T10:38:01.578 DEBUG:boto:Data: 2024-02-01T10:38:01.578 DEBUG:boto:Headers: {} 2024-02-01T10:38:01.578 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:01.578 DEBUG:boto:Port: 8000 2024-02-01T10:38:01.578 DEBUG:boto:Params: {} 2024-02-01T10:38:01.578 DEBUG:boto:Token: None 2024-02-01T10:38:01.579 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:01 GMT /vclfcn-39/?versions 2024-02-01T10:38:01.579 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:4XtsDvhmjfEuS0/9g9LWpAoj4iw= 2024-02-01T10:38:01.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:4XtsDvhmjfEuS0/9g9LWpAoj4iw=', 'Content-Length': '0'} 2024-02-01T10:38:01.583 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000062a7072dfcf64abb-0065bb7489-5004-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:01.583 DEBUG:boto:b'vclfcn-39999false' 2024-02-01T10:38:01.584 DEBUG:boto:path=/vclfcn-39/ 2024-02-01T10:38:01.584 DEBUG:boto:auth_path=/vclfcn-39/ 2024-02-01T10:38:01.584 DEBUG:boto:path=/vclfcn-39/?versions&max-keys=999 2024-02-01T10:38:01.584 DEBUG:boto:auth_path=/vclfcn-39/?versions&max-keys=999 2024-02-01T10:38:01.584 DEBUG:boto:Method: GET 2024-02-01T10:38:01.584 DEBUG:boto:Path: /vclfcn-39/?versions&max-keys=999 2024-02-01T10:38:01.585 DEBUG:boto:Data: 2024-02-01T10:38:01.585 DEBUG:boto:Headers: {} 2024-02-01T10:38:01.585 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:01.585 DEBUG:boto:Port: 8001 2024-02-01T10:38:01.585 DEBUG:boto:Params: {} 2024-02-01T10:38:01.585 DEBUG:boto:Token: None 2024-02-01T10:38:01.585 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:01 GMT /vclfcn-39/?versions 2024-02-01T10:38:01.585 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:4XtsDvhmjfEuS0/9g9LWpAoj4iw= 2024-02-01T10:38:01.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:4XtsDvhmjfEuS0/9g9LWpAoj4iw=', 'Content-Length': '0'} 2024-02-01T10:38:01.590 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa9a708ad83e655d-0065bb7489-5324-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:01.590 DEBUG:boto:b'vclfcn-39999false' 2024-02-01T10:38:01.590 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-39 zones={test-zone2, test-zone3} 2024-02-01T10:38:01.590 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-40 zones={test-zone2, test-zone1} 2024-02-01T10:38:01.590 DEBUG:boto:path=/vclfcn-40/ 2024-02-01T10:38:01.590 DEBUG:boto:auth_path=/vclfcn-40/ 2024-02-01T10:38:01.591 DEBUG:boto:Method: HEAD 2024-02-01T10:38:01.591 DEBUG:boto:Path: /vclfcn-40/ 2024-02-01T10:38:01.591 DEBUG:boto:Data: 2024-02-01T10:38:01.591 DEBUG:boto:Headers: {} 2024-02-01T10:38:01.591 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:01.591 DEBUG:boto:Port: 8000 2024-02-01T10:38:01.591 DEBUG:boto:Params: {} 2024-02-01T10:38:01.591 DEBUG:boto:Token: None 2024-02-01T10:38:01.591 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:01 GMT /vclfcn-40/ 2024-02-01T10:38:01.592 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lTVb65/SuUzc1XXVDeGpKc1//FA= 2024-02-01T10:38:01.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lTVb65/SuUzc1XXVDeGpKc1//FA=', 'Content-Length': '0'} 2024-02-01T10:38:01.597 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000189c002bf12eeb9c-0065bb7489-5004-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:01.597 DEBUG:boto:path=/vclfcn-40/ 2024-02-01T10:38:01.598 DEBUG:boto:auth_path=/vclfcn-40/ 2024-02-01T10:38:01.598 DEBUG:boto:Method: HEAD 2024-02-01T10:38:01.598 DEBUG:boto:Path: /vclfcn-40/ 2024-02-01T10:38:01.598 DEBUG:boto:Data: 2024-02-01T10:38:01.598 DEBUG:boto:Headers: {} 2024-02-01T10:38:01.598 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:01.598 DEBUG:boto:Port: 8000 2024-02-01T10:38:01.598 DEBUG:boto:Params: {} 2024-02-01T10:38:01.598 DEBUG:boto:Token: None 2024-02-01T10:38:01.599 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:01 GMT /vclfcn-40/ 2024-02-01T10:38:01.599 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lTVb65/SuUzc1XXVDeGpKc1//FA= 2024-02-01T10:38:01.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lTVb65/SuUzc1XXVDeGpKc1//FA=', 'Content-Length': '0'} 2024-02-01T10:38:01.605 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000085a9e963958db5e8-0065bb7489-5317-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:01.605 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:38:01.606 DEBUG:boto:path=/vclfcn-40/ 2024-02-01T10:38:01.606 DEBUG:boto:auth_path=/vclfcn-40/ 2024-02-01T10:38:01.606 DEBUG:boto:path=/vclfcn-40/?versions&max-keys=999 2024-02-01T10:38:01.606 DEBUG:boto:auth_path=/vclfcn-40/?versions&max-keys=999 2024-02-01T10:38:01.606 DEBUG:boto:Method: GET 2024-02-01T10:38:01.606 DEBUG:boto:Path: /vclfcn-40/?versions&max-keys=999 2024-02-01T10:38:01.606 DEBUG:boto:Data: 2024-02-01T10:38:01.606 DEBUG:boto:Headers: {} 2024-02-01T10:38:01.606 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:01.606 DEBUG:boto:Port: 8000 2024-02-01T10:38:01.606 DEBUG:boto:Params: {} 2024-02-01T10:38:01.606 DEBUG:boto:Token: None 2024-02-01T10:38:01.607 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:01 GMT /vclfcn-40/?versions 2024-02-01T10:38:01.607 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:kuEVduto497AKRfimgwFV+ZqznQ= 2024-02-01T10:38:01.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:kuEVduto497AKRfimgwFV+ZqznQ=', 'Content-Length': '0'} 2024-02-01T10:38:01.611 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e76e02803129aa4-0065bb7489-5004-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:01.652 DEBUG:boto:b'vclfcn-40999false' 2024-02-01T10:38:01.653 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:38:01.653 DEBUG:boto:path=/vclfcn-40/ 2024-02-01T10:38:01.653 DEBUG:boto:auth_path=/vclfcn-40/ 2024-02-01T10:38:01.653 DEBUG:boto:path=/vclfcn-40/?versions&max-keys=999 2024-02-01T10:38:01.653 DEBUG:boto:auth_path=/vclfcn-40/?versions&max-keys=999 2024-02-01T10:38:01.653 DEBUG:boto:Method: GET 2024-02-01T10:38:01.654 DEBUG:boto:Path: /vclfcn-40/?versions&max-keys=999 2024-02-01T10:38:01.654 DEBUG:boto:Data: 2024-02-01T10:38:01.654 DEBUG:boto:Headers: {} 2024-02-01T10:38:01.654 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:01.654 DEBUG:boto:Port: 8000 2024-02-01T10:38:01.654 DEBUG:boto:Params: {} 2024-02-01T10:38:01.654 DEBUG:boto:Token: None 2024-02-01T10:38:01.654 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:01 GMT /vclfcn-40/?versions 2024-02-01T10:38:01.655 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:kuEVduto497AKRfimgwFV+ZqznQ= 2024-02-01T10:38:01.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:kuEVduto497AKRfimgwFV+ZqznQ=', 'Content-Length': '0'} 2024-02-01T10:38:01.659 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ff2347f2ce2daf6-0065bb7489-5317-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:01.659 DEBUG:boto:b'vclfcn-40999false' 2024-02-01T10:38:01.660 DEBUG:boto:path=/vclfcn-40/ 2024-02-01T10:38:01.660 DEBUG:boto:auth_path=/vclfcn-40/ 2024-02-01T10:38:01.660 DEBUG:boto:path=/vclfcn-40/?versions&max-keys=999 2024-02-01T10:38:01.660 DEBUG:boto:auth_path=/vclfcn-40/?versions&max-keys=999 2024-02-01T10:38:01.660 DEBUG:boto:Method: GET 2024-02-01T10:38:01.660 DEBUG:boto:Path: /vclfcn-40/?versions&max-keys=999 2024-02-01T10:38:01.660 DEBUG:boto:Data: 2024-02-01T10:38:01.661 DEBUG:boto:Headers: {} 2024-02-01T10:38:01.661 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:01.661 DEBUG:boto:Port: 8000 2024-02-01T10:38:01.661 DEBUG:boto:Params: {} 2024-02-01T10:38:01.661 DEBUG:boto:Token: None 2024-02-01T10:38:01.661 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:01 GMT /vclfcn-40/?versions 2024-02-01T10:38:01.661 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:kuEVduto497AKRfimgwFV+ZqznQ= 2024-02-01T10:38:01.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:kuEVduto497AKRfimgwFV+ZqznQ=', 'Content-Length': '0'} 2024-02-01T10:38:01.666 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b43773d6f8514731-0065bb7489-5004-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:01.667 DEBUG:boto:b'vclfcn-40999false' 2024-02-01T10:38:01.667 DEBUG:boto:path=/vclfcn-40/ 2024-02-01T10:38:01.667 DEBUG:boto:auth_path=/vclfcn-40/ 2024-02-01T10:38:01.667 DEBUG:boto:path=/vclfcn-40/?versions&max-keys=999 2024-02-01T10:38:01.668 DEBUG:boto:auth_path=/vclfcn-40/?versions&max-keys=999 2024-02-01T10:38:01.668 DEBUG:boto:Method: GET 2024-02-01T10:38:01.668 DEBUG:boto:Path: /vclfcn-40/?versions&max-keys=999 2024-02-01T10:38:01.668 DEBUG:boto:Data: 2024-02-01T10:38:01.668 DEBUG:boto:Headers: {} 2024-02-01T10:38:01.668 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:01.668 DEBUG:boto:Port: 8000 2024-02-01T10:38:01.668 DEBUG:boto:Params: {} 2024-02-01T10:38:01.668 DEBUG:boto:Token: None 2024-02-01T10:38:01.669 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:01 GMT /vclfcn-40/?versions 2024-02-01T10:38:01.669 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:kuEVduto497AKRfimgwFV+ZqznQ= 2024-02-01T10:38:01.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:kuEVduto497AKRfimgwFV+ZqznQ=', 'Content-Length': '0'} 2024-02-01T10:38:01.674 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b6a00a62af967222-0065bb7489-5317-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:01.716 DEBUG:boto:b'vclfcn-40999false' 2024-02-01T10:38:01.716 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-40 zones={test-zone2, test-zone1} 2024-02-01T10:38:01.717 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-40 zones={test-zone3, test-zone1} 2024-02-01T10:38:01.717 DEBUG:boto:path=/vclfcn-40/ 2024-02-01T10:38:01.717 DEBUG:boto:auth_path=/vclfcn-40/ 2024-02-01T10:38:01.717 DEBUG:boto:Method: HEAD 2024-02-01T10:38:01.717 DEBUG:boto:Path: /vclfcn-40/ 2024-02-01T10:38:01.717 DEBUG:boto:Data: 2024-02-01T10:38:01.717 DEBUG:boto:Headers: {} 2024-02-01T10:38:01.717 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:01.717 DEBUG:boto:Port: 8001 2024-02-01T10:38:01.717 DEBUG:boto:Params: {} 2024-02-01T10:38:01.718 DEBUG:boto:Token: None 2024-02-01T10:38:01.718 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:01 GMT /vclfcn-40/ 2024-02-01T10:38:01.718 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lTVb65/SuUzc1XXVDeGpKc1//FA= 2024-02-01T10:38:01.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lTVb65/SuUzc1XXVDeGpKc1//FA=', 'Content-Length': '0'} 2024-02-01T10:38:01.724 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003006c4ab589950a2-0065bb7489-5324-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:01.724 DEBUG:boto:path=/vclfcn-40/ 2024-02-01T10:38:01.725 DEBUG:boto:auth_path=/vclfcn-40/ 2024-02-01T10:38:01.725 DEBUG:boto:Method: HEAD 2024-02-01T10:38:01.725 DEBUG:boto:Path: /vclfcn-40/ 2024-02-01T10:38:01.725 DEBUG:boto:Data: 2024-02-01T10:38:01.725 DEBUG:boto:Headers: {} 2024-02-01T10:38:01.725 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:01.725 DEBUG:boto:Port: 8000 2024-02-01T10:38:01.725 DEBUG:boto:Params: {} 2024-02-01T10:38:01.725 DEBUG:boto:Token: None 2024-02-01T10:38:01.725 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:01 GMT /vclfcn-40/ 2024-02-01T10:38:01.726 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lTVb65/SuUzc1XXVDeGpKc1//FA= 2024-02-01T10:38:01.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lTVb65/SuUzc1XXVDeGpKc1//FA=', 'Content-Length': '0'} 2024-02-01T10:38:01.733 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000008b34b43c3579695-0065bb7489-5317-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:01.733 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:38:01.733 DEBUG:boto:path=/vclfcn-40/ 2024-02-01T10:38:01.733 DEBUG:boto:auth_path=/vclfcn-40/ 2024-02-01T10:38:01.733 DEBUG:boto:path=/vclfcn-40/?versions&max-keys=999 2024-02-01T10:38:01.733 DEBUG:boto:auth_path=/vclfcn-40/?versions&max-keys=999 2024-02-01T10:38:01.733 DEBUG:boto:Method: GET 2024-02-01T10:38:01.733 DEBUG:boto:Path: /vclfcn-40/?versions&max-keys=999 2024-02-01T10:38:01.734 DEBUG:boto:Data: 2024-02-01T10:38:01.734 DEBUG:boto:Headers: {} 2024-02-01T10:38:01.734 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:01.734 DEBUG:boto:Port: 8001 2024-02-01T10:38:01.734 DEBUG:boto:Params: {} 2024-02-01T10:38:01.734 DEBUG:boto:Token: None 2024-02-01T10:38:01.734 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:01 GMT /vclfcn-40/?versions 2024-02-01T10:38:01.734 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:kuEVduto497AKRfimgwFV+ZqznQ= 2024-02-01T10:38:01.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:kuEVduto497AKRfimgwFV+ZqznQ=', 'Content-Length': '0'} 2024-02-01T10:38:01.739 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca4dc1df44b28775-0065bb7489-5324-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:01.739 DEBUG:boto:b'vclfcn-40999false' 2024-02-01T10:38:01.740 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:38:01.740 DEBUG:boto:path=/vclfcn-40/ 2024-02-01T10:38:01.741 DEBUG:boto:auth_path=/vclfcn-40/ 2024-02-01T10:38:01.741 DEBUG:boto:path=/vclfcn-40/?versions&max-keys=999 2024-02-01T10:38:01.741 DEBUG:boto:auth_path=/vclfcn-40/?versions&max-keys=999 2024-02-01T10:38:01.741 DEBUG:boto:Method: GET 2024-02-01T10:38:01.741 DEBUG:boto:Path: /vclfcn-40/?versions&max-keys=999 2024-02-01T10:38:01.741 DEBUG:boto:Data: 2024-02-01T10:38:01.741 DEBUG:boto:Headers: {} 2024-02-01T10:38:01.741 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:01.741 DEBUG:boto:Port: 8000 2024-02-01T10:38:01.741 DEBUG:boto:Params: {} 2024-02-01T10:38:01.741 DEBUG:boto:Token: None 2024-02-01T10:38:01.742 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:01 GMT /vclfcn-40/?versions 2024-02-01T10:38:01.742 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:kuEVduto497AKRfimgwFV+ZqznQ= 2024-02-01T10:38:01.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:kuEVduto497AKRfimgwFV+ZqznQ=', 'Content-Length': '0'} 2024-02-01T10:38:01.746 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e13620604c3a34e4-0065bb7489-5317-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:01.746 DEBUG:boto:b'vclfcn-40999false' 2024-02-01T10:38:01.747 DEBUG:boto:path=/vclfcn-40/ 2024-02-01T10:38:01.747 DEBUG:boto:auth_path=/vclfcn-40/ 2024-02-01T10:38:01.747 DEBUG:boto:path=/vclfcn-40/?versions&max-keys=999 2024-02-01T10:38:01.747 DEBUG:boto:auth_path=/vclfcn-40/?versions&max-keys=999 2024-02-01T10:38:01.747 DEBUG:boto:Method: GET 2024-02-01T10:38:01.747 DEBUG:boto:Path: /vclfcn-40/?versions&max-keys=999 2024-02-01T10:38:01.747 DEBUG:boto:Data: 2024-02-01T10:38:01.747 DEBUG:boto:Headers: {} 2024-02-01T10:38:01.747 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:01.748 DEBUG:boto:Port: 8001 2024-02-01T10:38:01.748 DEBUG:boto:Params: {} 2024-02-01T10:38:01.748 DEBUG:boto:Token: None 2024-02-01T10:38:01.748 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:01 GMT /vclfcn-40/?versions 2024-02-01T10:38:01.748 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:kuEVduto497AKRfimgwFV+ZqznQ= 2024-02-01T10:38:01.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:kuEVduto497AKRfimgwFV+ZqznQ=', 'Content-Length': '0'} 2024-02-01T10:38:01.756 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e808512d53ff3e6-0065bb7489-5324-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:01.796 DEBUG:boto:b'vclfcn-40999false' 2024-02-01T10:38:01.796 DEBUG:boto:path=/vclfcn-40/ 2024-02-01T10:38:01.797 DEBUG:boto:auth_path=/vclfcn-40/ 2024-02-01T10:38:01.797 DEBUG:boto:path=/vclfcn-40/?versions&max-keys=999 2024-02-01T10:38:01.797 DEBUG:boto:auth_path=/vclfcn-40/?versions&max-keys=999 2024-02-01T10:38:01.797 DEBUG:boto:Method: GET 2024-02-01T10:38:01.797 DEBUG:boto:Path: /vclfcn-40/?versions&max-keys=999 2024-02-01T10:38:01.797 DEBUG:boto:Data: 2024-02-01T10:38:01.797 DEBUG:boto:Headers: {} 2024-02-01T10:38:01.797 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:01.797 DEBUG:boto:Port: 8000 2024-02-01T10:38:01.797 DEBUG:boto:Params: {} 2024-02-01T10:38:01.798 DEBUG:boto:Token: None 2024-02-01T10:38:01.798 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:01 GMT /vclfcn-40/?versions 2024-02-01T10:38:01.798 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:kuEVduto497AKRfimgwFV+ZqznQ= 2024-02-01T10:38:01.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:kuEVduto497AKRfimgwFV+ZqznQ=', 'Content-Length': '0'} 2024-02-01T10:38:01.803 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f87c91d692a6bc85-0065bb7489-5317-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:01.803 DEBUG:boto:b'vclfcn-40999false' 2024-02-01T10:38:01.803 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-40 zones={test-zone3, test-zone1} 2024-02-01T10:38:01.803 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-40 zones={test-zone1, test-zone2} 2024-02-01T10:38:01.803 DEBUG:boto:path=/vclfcn-40/ 2024-02-01T10:38:01.803 DEBUG:boto:auth_path=/vclfcn-40/ 2024-02-01T10:38:01.803 DEBUG:boto:Method: HEAD 2024-02-01T10:38:01.804 DEBUG:boto:Path: /vclfcn-40/ 2024-02-01T10:38:01.804 DEBUG:boto:Data: 2024-02-01T10:38:01.804 DEBUG:boto:Headers: {} 2024-02-01T10:38:01.804 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:01.804 DEBUG:boto:Port: 8000 2024-02-01T10:38:01.804 DEBUG:boto:Params: {} 2024-02-01T10:38:01.804 DEBUG:boto:Token: None 2024-02-01T10:38:01.804 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:01 GMT /vclfcn-40/ 2024-02-01T10:38:01.804 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lTVb65/SuUzc1XXVDeGpKc1//FA= 2024-02-01T10:38:01.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lTVb65/SuUzc1XXVDeGpKc1//FA=', 'Content-Length': '0'} 2024-02-01T10:38:01.809 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004af6560e393f5d88-0065bb7489-5317-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:01.809 DEBUG:boto:path=/vclfcn-40/ 2024-02-01T10:38:01.809 DEBUG:boto:auth_path=/vclfcn-40/ 2024-02-01T10:38:01.809 DEBUG:boto:Method: HEAD 2024-02-01T10:38:01.810 DEBUG:boto:Path: /vclfcn-40/ 2024-02-01T10:38:01.810 DEBUG:boto:Data: 2024-02-01T10:38:01.810 DEBUG:boto:Headers: {} 2024-02-01T10:38:01.810 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:01.810 DEBUG:boto:Port: 8000 2024-02-01T10:38:01.810 DEBUG:boto:Params: {} 2024-02-01T10:38:01.810 DEBUG:boto:Token: None 2024-02-01T10:38:01.810 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:01 GMT /vclfcn-40/ 2024-02-01T10:38:01.810 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lTVb65/SuUzc1XXVDeGpKc1//FA= 2024-02-01T10:38:01.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lTVb65/SuUzc1XXVDeGpKc1//FA=', 'Content-Length': '0'} 2024-02-01T10:38:01.815 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000c1fc045c09172ac-0065bb7489-5004-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:01.815 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:38:01.815 DEBUG:boto:path=/vclfcn-40/ 2024-02-01T10:38:01.815 DEBUG:boto:auth_path=/vclfcn-40/ 2024-02-01T10:38:01.815 DEBUG:boto:path=/vclfcn-40/?versions&max-keys=999 2024-02-01T10:38:01.815 DEBUG:boto:auth_path=/vclfcn-40/?versions&max-keys=999 2024-02-01T10:38:01.815 DEBUG:boto:Method: GET 2024-02-01T10:38:01.815 DEBUG:boto:Path: /vclfcn-40/?versions&max-keys=999 2024-02-01T10:38:01.815 DEBUG:boto:Data: 2024-02-01T10:38:01.815 DEBUG:boto:Headers: {} 2024-02-01T10:38:01.816 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:01.816 DEBUG:boto:Port: 8000 2024-02-01T10:38:01.816 DEBUG:boto:Params: {} 2024-02-01T10:38:01.816 DEBUG:boto:Token: None 2024-02-01T10:38:01.816 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:01 GMT /vclfcn-40/?versions 2024-02-01T10:38:01.816 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:kuEVduto497AKRfimgwFV+ZqznQ= 2024-02-01T10:38:01.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:kuEVduto497AKRfimgwFV+ZqznQ=', 'Content-Length': '0'} 2024-02-01T10:38:01.820 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7ef47f01cfe6bd3-0065bb7489-5317-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:01.864 DEBUG:boto:b'vclfcn-40999false' 2024-02-01T10:38:01.864 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:38:01.865 DEBUG:boto:path=/vclfcn-40/ 2024-02-01T10:38:01.865 DEBUG:boto:auth_path=/vclfcn-40/ 2024-02-01T10:38:01.865 DEBUG:boto:path=/vclfcn-40/?versions&max-keys=999 2024-02-01T10:38:01.865 DEBUG:boto:auth_path=/vclfcn-40/?versions&max-keys=999 2024-02-01T10:38:01.865 DEBUG:boto:Method: GET 2024-02-01T10:38:01.865 DEBUG:boto:Path: /vclfcn-40/?versions&max-keys=999 2024-02-01T10:38:01.865 DEBUG:boto:Data: 2024-02-01T10:38:01.865 DEBUG:boto:Headers: {} 2024-02-01T10:38:01.865 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:01.865 DEBUG:boto:Port: 8000 2024-02-01T10:38:01.865 DEBUG:boto:Params: {} 2024-02-01T10:38:01.866 DEBUG:boto:Token: None 2024-02-01T10:38:01.866 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:01 GMT /vclfcn-40/?versions 2024-02-01T10:38:01.866 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:kuEVduto497AKRfimgwFV+ZqznQ= 2024-02-01T10:38:01.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:kuEVduto497AKRfimgwFV+ZqznQ=', 'Content-Length': '0'} 2024-02-01T10:38:01.870 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a1587f6cece77de-0065bb7489-5004-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:01.871 DEBUG:boto:b'vclfcn-40999false' 2024-02-01T10:38:01.871 DEBUG:boto:path=/vclfcn-40/ 2024-02-01T10:38:01.871 DEBUG:boto:auth_path=/vclfcn-40/ 2024-02-01T10:38:01.871 DEBUG:boto:path=/vclfcn-40/?versions&max-keys=999 2024-02-01T10:38:01.871 DEBUG:boto:auth_path=/vclfcn-40/?versions&max-keys=999 2024-02-01T10:38:01.871 DEBUG:boto:Method: GET 2024-02-01T10:38:01.871 DEBUG:boto:Path: /vclfcn-40/?versions&max-keys=999 2024-02-01T10:38:01.871 DEBUG:boto:Data: 2024-02-01T10:38:01.872 DEBUG:boto:Headers: {} 2024-02-01T10:38:01.872 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:01.872 DEBUG:boto:Port: 8000 2024-02-01T10:38:01.872 DEBUG:boto:Params: {} 2024-02-01T10:38:01.872 DEBUG:boto:Token: None 2024-02-01T10:38:01.872 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:01 GMT /vclfcn-40/?versions 2024-02-01T10:38:01.872 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:kuEVduto497AKRfimgwFV+ZqznQ= 2024-02-01T10:38:01.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:kuEVduto497AKRfimgwFV+ZqznQ=', 'Content-Length': '0'} 2024-02-01T10:38:01.877 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051cbaa905d2899c4-0065bb7489-5317-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:01.877 DEBUG:boto:b'vclfcn-40999false' 2024-02-01T10:38:01.877 DEBUG:boto:path=/vclfcn-40/ 2024-02-01T10:38:01.878 DEBUG:boto:auth_path=/vclfcn-40/ 2024-02-01T10:38:01.878 DEBUG:boto:path=/vclfcn-40/?versions&max-keys=999 2024-02-01T10:38:01.878 DEBUG:boto:auth_path=/vclfcn-40/?versions&max-keys=999 2024-02-01T10:38:01.878 DEBUG:boto:Method: GET 2024-02-01T10:38:01.878 DEBUG:boto:Path: /vclfcn-40/?versions&max-keys=999 2024-02-01T10:38:01.878 DEBUG:boto:Data: 2024-02-01T10:38:01.878 DEBUG:boto:Headers: {} 2024-02-01T10:38:01.878 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:01.878 DEBUG:boto:Port: 8000 2024-02-01T10:38:01.878 DEBUG:boto:Params: {} 2024-02-01T10:38:01.878 DEBUG:boto:Token: None 2024-02-01T10:38:01.879 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:01 GMT /vclfcn-40/?versions 2024-02-01T10:38:01.879 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:kuEVduto497AKRfimgwFV+ZqznQ= 2024-02-01T10:38:01.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:kuEVduto497AKRfimgwFV+ZqznQ=', 'Content-Length': '0'} 2024-02-01T10:38:01.883 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f900572826b6bcc1-0065bb7489-5004-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:01.883 DEBUG:boto:b'vclfcn-40999false' 2024-02-01T10:38:01.883 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-40 zones={test-zone1, test-zone2} 2024-02-01T10:38:01.883 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-40 zones={test-zone3, test-zone2} 2024-02-01T10:38:01.884 DEBUG:boto:path=/vclfcn-40/ 2024-02-01T10:38:01.884 DEBUG:boto:auth_path=/vclfcn-40/ 2024-02-01T10:38:01.884 DEBUG:boto:Method: HEAD 2024-02-01T10:38:01.884 DEBUG:boto:Path: /vclfcn-40/ 2024-02-01T10:38:01.884 DEBUG:boto:Data: 2024-02-01T10:38:01.884 DEBUG:boto:Headers: {} 2024-02-01T10:38:01.884 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:01.884 DEBUG:boto:Port: 8001 2024-02-01T10:38:01.885 DEBUG:boto:Params: {} 2024-02-01T10:38:01.885 DEBUG:boto:Token: None 2024-02-01T10:38:01.885 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:01 GMT /vclfcn-40/ 2024-02-01T10:38:01.885 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lTVb65/SuUzc1XXVDeGpKc1//FA= 2024-02-01T10:38:01.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lTVb65/SuUzc1XXVDeGpKc1//FA=', 'Content-Length': '0'} 2024-02-01T10:38:01.889 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f7f34075a650d494-0065bb7489-5324-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:01.889 DEBUG:boto:path=/vclfcn-40/ 2024-02-01T10:38:01.889 DEBUG:boto:auth_path=/vclfcn-40/ 2024-02-01T10:38:01.889 DEBUG:boto:Method: HEAD 2024-02-01T10:38:01.890 DEBUG:boto:Path: /vclfcn-40/ 2024-02-01T10:38:01.890 DEBUG:boto:Data: 2024-02-01T10:38:01.890 DEBUG:boto:Headers: {} 2024-02-01T10:38:01.890 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:01.890 DEBUG:boto:Port: 8000 2024-02-01T10:38:01.890 DEBUG:boto:Params: {} 2024-02-01T10:38:01.890 DEBUG:boto:Token: None 2024-02-01T10:38:01.890 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:01 GMT /vclfcn-40/ 2024-02-01T10:38:01.890 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lTVb65/SuUzc1XXVDeGpKc1//FA= 2024-02-01T10:38:01.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lTVb65/SuUzc1XXVDeGpKc1//FA=', 'Content-Length': '0'} 2024-02-01T10:38:01.894 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cda8b5e8c7ddf225-0065bb7489-5004-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:01.895 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:38:01.895 DEBUG:boto:path=/vclfcn-40/ 2024-02-01T10:38:01.895 DEBUG:boto:auth_path=/vclfcn-40/ 2024-02-01T10:38:01.895 DEBUG:boto:path=/vclfcn-40/?versions&max-keys=999 2024-02-01T10:38:01.895 DEBUG:boto:auth_path=/vclfcn-40/?versions&max-keys=999 2024-02-01T10:38:01.895 DEBUG:boto:Method: GET 2024-02-01T10:38:01.895 DEBUG:boto:Path: /vclfcn-40/?versions&max-keys=999 2024-02-01T10:38:01.895 DEBUG:boto:Data: 2024-02-01T10:38:01.895 DEBUG:boto:Headers: {} 2024-02-01T10:38:01.895 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:01.896 DEBUG:boto:Port: 8001 2024-02-01T10:38:01.896 DEBUG:boto:Params: {} 2024-02-01T10:38:01.896 DEBUG:boto:Token: None 2024-02-01T10:38:01.896 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:01 GMT /vclfcn-40/?versions 2024-02-01T10:38:01.896 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:kuEVduto497AKRfimgwFV+ZqznQ= 2024-02-01T10:38:01.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:kuEVduto497AKRfimgwFV+ZqznQ=', 'Content-Length': '0'} 2024-02-01T10:38:01.900 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000049a76638f3b4099f-0065bb7489-5324-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:01.901 DEBUG:boto:b'vclfcn-40999false' 2024-02-01T10:38:01.901 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:38:01.901 DEBUG:boto:path=/vclfcn-40/ 2024-02-01T10:38:01.901 DEBUG:boto:auth_path=/vclfcn-40/ 2024-02-01T10:38:01.901 DEBUG:boto:path=/vclfcn-40/?versions&max-keys=999 2024-02-01T10:38:01.902 DEBUG:boto:auth_path=/vclfcn-40/?versions&max-keys=999 2024-02-01T10:38:01.902 DEBUG:boto:Method: GET 2024-02-01T10:38:01.902 DEBUG:boto:Path: /vclfcn-40/?versions&max-keys=999 2024-02-01T10:38:01.902 DEBUG:boto:Data: 2024-02-01T10:38:01.902 DEBUG:boto:Headers: {} 2024-02-01T10:38:01.902 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:01.902 DEBUG:boto:Port: 8000 2024-02-01T10:38:01.902 DEBUG:boto:Params: {} 2024-02-01T10:38:01.903 DEBUG:boto:Token: None 2024-02-01T10:38:01.903 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:01 GMT /vclfcn-40/?versions 2024-02-01T10:38:01.903 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:kuEVduto497AKRfimgwFV+ZqznQ= 2024-02-01T10:38:01.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:kuEVduto497AKRfimgwFV+ZqznQ=', 'Content-Length': '0'} 2024-02-01T10:38:01.908 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006343386097495b74-0065bb7489-5004-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:01.953 DEBUG:boto:b'vclfcn-40999false' 2024-02-01T10:38:01.953 DEBUG:boto:path=/vclfcn-40/ 2024-02-01T10:38:01.954 DEBUG:boto:auth_path=/vclfcn-40/ 2024-02-01T10:38:01.954 DEBUG:boto:path=/vclfcn-40/?versions&max-keys=999 2024-02-01T10:38:01.954 DEBUG:boto:auth_path=/vclfcn-40/?versions&max-keys=999 2024-02-01T10:38:01.954 DEBUG:boto:Method: GET 2024-02-01T10:38:01.954 DEBUG:boto:Path: /vclfcn-40/?versions&max-keys=999 2024-02-01T10:38:01.954 DEBUG:boto:Data: 2024-02-01T10:38:01.954 DEBUG:boto:Headers: {} 2024-02-01T10:38:01.954 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:01.954 DEBUG:boto:Port: 8001 2024-02-01T10:38:01.954 DEBUG:boto:Params: {} 2024-02-01T10:38:01.955 DEBUG:boto:Token: None 2024-02-01T10:38:01.955 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:01 GMT /vclfcn-40/?versions 2024-02-01T10:38:01.955 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:kuEVduto497AKRfimgwFV+ZqznQ= 2024-02-01T10:38:01.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:kuEVduto497AKRfimgwFV+ZqznQ=', 'Content-Length': '0'} 2024-02-01T10:38:01.959 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a15a3cba512b6e75-0065bb7489-5324-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:01.959 DEBUG:boto:b'vclfcn-40999false' 2024-02-01T10:38:01.960 DEBUG:boto:path=/vclfcn-40/ 2024-02-01T10:38:01.960 DEBUG:boto:auth_path=/vclfcn-40/ 2024-02-01T10:38:01.960 DEBUG:boto:path=/vclfcn-40/?versions&max-keys=999 2024-02-01T10:38:01.961 DEBUG:boto:auth_path=/vclfcn-40/?versions&max-keys=999 2024-02-01T10:38:01.961 DEBUG:boto:Method: GET 2024-02-01T10:38:01.961 DEBUG:boto:Path: /vclfcn-40/?versions&max-keys=999 2024-02-01T10:38:01.961 DEBUG:boto:Data: 2024-02-01T10:38:01.961 DEBUG:boto:Headers: {} 2024-02-01T10:38:01.961 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:01.961 DEBUG:boto:Port: 8000 2024-02-01T10:38:01.961 DEBUG:boto:Params: {} 2024-02-01T10:38:01.961 DEBUG:boto:Token: None 2024-02-01T10:38:01.961 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:01 GMT /vclfcn-40/?versions 2024-02-01T10:38:01.962 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:kuEVduto497AKRfimgwFV+ZqznQ= 2024-02-01T10:38:01.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:kuEVduto497AKRfimgwFV+ZqznQ=', 'Content-Length': '0'} 2024-02-01T10:38:01.967 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d75fdb264edd7a09-0065bb7489-5004-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:01.967 DEBUG:boto:b'vclfcn-40999false' 2024-02-01T10:38:01.967 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-40 zones={test-zone3, test-zone2} 2024-02-01T10:38:01.967 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-40 zones={test-zone1, test-zone3} 2024-02-01T10:38:01.967 DEBUG:boto:path=/vclfcn-40/ 2024-02-01T10:38:01.967 DEBUG:boto:auth_path=/vclfcn-40/ 2024-02-01T10:38:01.967 DEBUG:boto:Method: HEAD 2024-02-01T10:38:01.968 DEBUG:boto:Path: /vclfcn-40/ 2024-02-01T10:38:01.968 DEBUG:boto:Data: 2024-02-01T10:38:01.968 DEBUG:boto:Headers: {} 2024-02-01T10:38:01.968 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:01.968 DEBUG:boto:Port: 8000 2024-02-01T10:38:01.968 DEBUG:boto:Params: {} 2024-02-01T10:38:01.968 DEBUG:boto:Token: None 2024-02-01T10:38:01.968 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:01 GMT /vclfcn-40/ 2024-02-01T10:38:01.968 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lTVb65/SuUzc1XXVDeGpKc1//FA= 2024-02-01T10:38:01.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lTVb65/SuUzc1XXVDeGpKc1//FA=', 'Content-Length': '0'} 2024-02-01T10:38:01.974 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000856dd96f4806fb7d-0065bb7489-5317-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:01.974 DEBUG:boto:path=/vclfcn-40/ 2024-02-01T10:38:01.974 DEBUG:boto:auth_path=/vclfcn-40/ 2024-02-01T10:38:01.974 DEBUG:boto:Method: HEAD 2024-02-01T10:38:01.975 DEBUG:boto:Path: /vclfcn-40/ 2024-02-01T10:38:01.975 DEBUG:boto:Data: 2024-02-01T10:38:01.975 DEBUG:boto:Headers: {} 2024-02-01T10:38:01.975 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:01.975 DEBUG:boto:Port: 8001 2024-02-01T10:38:01.975 DEBUG:boto:Params: {} 2024-02-01T10:38:01.975 DEBUG:boto:Token: None 2024-02-01T10:38:01.975 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:01 GMT /vclfcn-40/ 2024-02-01T10:38:01.975 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lTVb65/SuUzc1XXVDeGpKc1//FA= 2024-02-01T10:38:01.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lTVb65/SuUzc1XXVDeGpKc1//FA=', 'Content-Length': '0'} 2024-02-01T10:38:01.980 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fe2d11ee893852f1-0065bb7489-5324-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:01.980 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:38:01.980 DEBUG:boto:path=/vclfcn-40/ 2024-02-01T10:38:01.980 DEBUG:boto:auth_path=/vclfcn-40/ 2024-02-01T10:38:01.981 DEBUG:boto:path=/vclfcn-40/?versions&max-keys=999 2024-02-01T10:38:01.981 DEBUG:boto:auth_path=/vclfcn-40/?versions&max-keys=999 2024-02-01T10:38:01.981 DEBUG:boto:Method: GET 2024-02-01T10:38:01.981 DEBUG:boto:Path: /vclfcn-40/?versions&max-keys=999 2024-02-01T10:38:01.981 DEBUG:boto:Data: 2024-02-01T10:38:01.981 DEBUG:boto:Headers: {} 2024-02-01T10:38:01.981 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:01.981 DEBUG:boto:Port: 8000 2024-02-01T10:38:01.981 DEBUG:boto:Params: {} 2024-02-01T10:38:01.981 DEBUG:boto:Token: None 2024-02-01T10:38:01.982 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:01 GMT /vclfcn-40/?versions 2024-02-01T10:38:01.982 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:kuEVduto497AKRfimgwFV+ZqznQ= 2024-02-01T10:38:01.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:kuEVduto497AKRfimgwFV+ZqznQ=', 'Content-Length': '0'} 2024-02-01T10:38:01.986 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8a7e250495a6df6-0065bb7489-5317-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:01.986 DEBUG:boto:b'vclfcn-40999false' 2024-02-01T10:38:01.986 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:38:01.986 DEBUG:boto:path=/vclfcn-40/ 2024-02-01T10:38:01.987 DEBUG:boto:auth_path=/vclfcn-40/ 2024-02-01T10:38:01.987 DEBUG:boto:path=/vclfcn-40/?versions&max-keys=999 2024-02-01T10:38:01.987 DEBUG:boto:auth_path=/vclfcn-40/?versions&max-keys=999 2024-02-01T10:38:01.987 DEBUG:boto:Method: GET 2024-02-01T10:38:01.987 DEBUG:boto:Path: /vclfcn-40/?versions&max-keys=999 2024-02-01T10:38:01.987 DEBUG:boto:Data: 2024-02-01T10:38:01.987 DEBUG:boto:Headers: {} 2024-02-01T10:38:01.987 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:01.987 DEBUG:boto:Port: 8001 2024-02-01T10:38:01.987 DEBUG:boto:Params: {} 2024-02-01T10:38:01.987 DEBUG:boto:Token: None 2024-02-01T10:38:01.988 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:01 GMT /vclfcn-40/?versions 2024-02-01T10:38:01.988 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:kuEVduto497AKRfimgwFV+ZqznQ= 2024-02-01T10:38:01.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:kuEVduto497AKRfimgwFV+ZqznQ=', 'Content-Length': '0'} 2024-02-01T10:38:01.992 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000038c20629b13e03a8-0065bb7489-5324-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:02.040 DEBUG:boto:b'vclfcn-40999false' 2024-02-01T10:38:02.040 DEBUG:boto:path=/vclfcn-40/ 2024-02-01T10:38:02.041 DEBUG:boto:auth_path=/vclfcn-40/ 2024-02-01T10:38:02.041 DEBUG:boto:path=/vclfcn-40/?versions&max-keys=999 2024-02-01T10:38:02.041 DEBUG:boto:auth_path=/vclfcn-40/?versions&max-keys=999 2024-02-01T10:38:02.041 DEBUG:boto:Method: GET 2024-02-01T10:38:02.041 DEBUG:boto:Path: /vclfcn-40/?versions&max-keys=999 2024-02-01T10:38:02.041 DEBUG:boto:Data: 2024-02-01T10:38:02.041 DEBUG:boto:Headers: {} 2024-02-01T10:38:02.041 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:02.041 DEBUG:boto:Port: 8000 2024-02-01T10:38:02.041 DEBUG:boto:Params: {} 2024-02-01T10:38:02.042 DEBUG:boto:Token: None 2024-02-01T10:38:02.042 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:02 GMT /vclfcn-40/?versions 2024-02-01T10:38:02.042 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/+oFab3yc91i4++Be3USVrERgE8= 2024-02-01T10:38:02.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/+oFab3yc91i4++Be3USVrERgE8=', 'Content-Length': '0'} 2024-02-01T10:38:02.046 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000501be9adf0a6ef10-0065bb748a-5317-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:02.046 DEBUG:boto:b'vclfcn-40999false' 2024-02-01T10:38:02.047 DEBUG:boto:path=/vclfcn-40/ 2024-02-01T10:38:02.047 DEBUG:boto:auth_path=/vclfcn-40/ 2024-02-01T10:38:02.047 DEBUG:boto:path=/vclfcn-40/?versions&max-keys=999 2024-02-01T10:38:02.047 DEBUG:boto:auth_path=/vclfcn-40/?versions&max-keys=999 2024-02-01T10:38:02.047 DEBUG:boto:Method: GET 2024-02-01T10:38:02.047 DEBUG:boto:Path: /vclfcn-40/?versions&max-keys=999 2024-02-01T10:38:02.047 DEBUG:boto:Data: 2024-02-01T10:38:02.048 DEBUG:boto:Headers: {} 2024-02-01T10:38:02.048 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:02.048 DEBUG:boto:Port: 8001 2024-02-01T10:38:02.048 DEBUG:boto:Params: {} 2024-02-01T10:38:02.048 DEBUG:boto:Token: None 2024-02-01T10:38:02.048 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:02 GMT /vclfcn-40/?versions 2024-02-01T10:38:02.048 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/+oFab3yc91i4++Be3USVrERgE8= 2024-02-01T10:38:02.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/+oFab3yc91i4++Be3USVrERgE8=', 'Content-Length': '0'} 2024-02-01T10:38:02.053 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc7fb02df413d98c-0065bb748a-5324-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:02.053 DEBUG:boto:b'vclfcn-40999false' 2024-02-01T10:38:02.054 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-40 zones={test-zone1, test-zone3} 2024-02-01T10:38:02.054 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-40 zones={test-zone2, test-zone3} 2024-02-01T10:38:02.054 DEBUG:boto:path=/vclfcn-40/ 2024-02-01T10:38:02.054 DEBUG:boto:auth_path=/vclfcn-40/ 2024-02-01T10:38:02.054 DEBUG:boto:Method: HEAD 2024-02-01T10:38:02.054 DEBUG:boto:Path: /vclfcn-40/ 2024-02-01T10:38:02.054 DEBUG:boto:Data: 2024-02-01T10:38:02.055 DEBUG:boto:Headers: {} 2024-02-01T10:38:02.055 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:02.055 DEBUG:boto:Port: 8000 2024-02-01T10:38:02.055 DEBUG:boto:Params: {} 2024-02-01T10:38:02.055 DEBUG:boto:Token: None 2024-02-01T10:38:02.055 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:02 GMT /vclfcn-40/ 2024-02-01T10:38:02.055 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZouuSKoGCsp/GCX+Zuj7NNK7qDs= 2024-02-01T10:38:02.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZouuSKoGCsp/GCX+Zuj7NNK7qDs=', 'Content-Length': '0'} 2024-02-01T10:38:02.060 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008cb5bc11abc7ad6e-0065bb748a-5004-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:02.060 DEBUG:boto:path=/vclfcn-40/ 2024-02-01T10:38:02.060 DEBUG:boto:auth_path=/vclfcn-40/ 2024-02-01T10:38:02.060 DEBUG:boto:Method: HEAD 2024-02-01T10:38:02.061 DEBUG:boto:Path: /vclfcn-40/ 2024-02-01T10:38:02.061 DEBUG:boto:Data: 2024-02-01T10:38:02.061 DEBUG:boto:Headers: {} 2024-02-01T10:38:02.061 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:02.061 DEBUG:boto:Port: 8001 2024-02-01T10:38:02.061 DEBUG:boto:Params: {} 2024-02-01T10:38:02.061 DEBUG:boto:Token: None 2024-02-01T10:38:02.061 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:02 GMT /vclfcn-40/ 2024-02-01T10:38:02.061 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZouuSKoGCsp/GCX+Zuj7NNK7qDs= 2024-02-01T10:38:02.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZouuSKoGCsp/GCX+Zuj7NNK7qDs=', 'Content-Length': '0'} 2024-02-01T10:38:02.065 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003de8979dead27078-0065bb748a-5324-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:02.066 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:38:02.066 DEBUG:boto:path=/vclfcn-40/ 2024-02-01T10:38:02.066 DEBUG:boto:auth_path=/vclfcn-40/ 2024-02-01T10:38:02.066 DEBUG:boto:path=/vclfcn-40/?versions&max-keys=999 2024-02-01T10:38:02.066 DEBUG:boto:auth_path=/vclfcn-40/?versions&max-keys=999 2024-02-01T10:38:02.066 DEBUG:boto:Method: GET 2024-02-01T10:38:02.066 DEBUG:boto:Path: /vclfcn-40/?versions&max-keys=999 2024-02-01T10:38:02.066 DEBUG:boto:Data: 2024-02-01T10:38:02.066 DEBUG:boto:Headers: {} 2024-02-01T10:38:02.067 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:02.067 DEBUG:boto:Port: 8000 2024-02-01T10:38:02.067 DEBUG:boto:Params: {} 2024-02-01T10:38:02.067 DEBUG:boto:Token: None 2024-02-01T10:38:02.067 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:02 GMT /vclfcn-40/?versions 2024-02-01T10:38:02.067 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/+oFab3yc91i4++Be3USVrERgE8= 2024-02-01T10:38:02.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/+oFab3yc91i4++Be3USVrERgE8=', 'Content-Length': '0'} 2024-02-01T10:38:02.071 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000096487c2d88819358-0065bb748a-5004-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:02.071 DEBUG:boto:b'vclfcn-40999false' 2024-02-01T10:38:02.072 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:38:02.072 DEBUG:boto:path=/vclfcn-40/ 2024-02-01T10:38:02.072 DEBUG:boto:auth_path=/vclfcn-40/ 2024-02-01T10:38:02.072 DEBUG:boto:path=/vclfcn-40/?versions&max-keys=999 2024-02-01T10:38:02.072 DEBUG:boto:auth_path=/vclfcn-40/?versions&max-keys=999 2024-02-01T10:38:02.072 DEBUG:boto:Method: GET 2024-02-01T10:38:02.073 DEBUG:boto:Path: /vclfcn-40/?versions&max-keys=999 2024-02-01T10:38:02.073 DEBUG:boto:Data: 2024-02-01T10:38:02.073 DEBUG:boto:Headers: {} 2024-02-01T10:38:02.073 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:02.073 DEBUG:boto:Port: 8001 2024-02-01T10:38:02.073 DEBUG:boto:Params: {} 2024-02-01T10:38:02.073 DEBUG:boto:Token: None 2024-02-01T10:38:02.073 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:02 GMT /vclfcn-40/?versions 2024-02-01T10:38:02.073 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/+oFab3yc91i4++Be3USVrERgE8= 2024-02-01T10:38:02.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/+oFab3yc91i4++Be3USVrERgE8=', 'Content-Length': '0'} 2024-02-01T10:38:02.078 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d0832745630550f5-0065bb748a-5324-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:02.120 DEBUG:boto:b'vclfcn-40999false' 2024-02-01T10:38:02.121 DEBUG:boto:path=/vclfcn-40/ 2024-02-01T10:38:02.121 DEBUG:boto:auth_path=/vclfcn-40/ 2024-02-01T10:38:02.121 DEBUG:boto:path=/vclfcn-40/?versions&max-keys=999 2024-02-01T10:38:02.121 DEBUG:boto:auth_path=/vclfcn-40/?versions&max-keys=999 2024-02-01T10:38:02.121 DEBUG:boto:Method: GET 2024-02-01T10:38:02.121 DEBUG:boto:Path: /vclfcn-40/?versions&max-keys=999 2024-02-01T10:38:02.121 DEBUG:boto:Data: 2024-02-01T10:38:02.121 DEBUG:boto:Headers: {} 2024-02-01T10:38:02.121 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:02.121 DEBUG:boto:Port: 8000 2024-02-01T10:38:02.122 DEBUG:boto:Params: {} 2024-02-01T10:38:02.122 DEBUG:boto:Token: None 2024-02-01T10:38:02.122 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:02 GMT /vclfcn-40/?versions 2024-02-01T10:38:02.122 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/+oFab3yc91i4++Be3USVrERgE8= 2024-02-01T10:38:02.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/+oFab3yc91i4++Be3USVrERgE8=', 'Content-Length': '0'} 2024-02-01T10:38:02.127 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6e2c327476c16a4-0065bb748a-5004-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:02.127 DEBUG:boto:b'vclfcn-40999false' 2024-02-01T10:38:02.127 DEBUG:boto:path=/vclfcn-40/ 2024-02-01T10:38:02.127 DEBUG:boto:auth_path=/vclfcn-40/ 2024-02-01T10:38:02.127 DEBUG:boto:path=/vclfcn-40/?versions&max-keys=999 2024-02-01T10:38:02.128 DEBUG:boto:auth_path=/vclfcn-40/?versions&max-keys=999 2024-02-01T10:38:02.128 DEBUG:boto:Method: GET 2024-02-01T10:38:02.128 DEBUG:boto:Path: /vclfcn-40/?versions&max-keys=999 2024-02-01T10:38:02.128 DEBUG:boto:Data: 2024-02-01T10:38:02.128 DEBUG:boto:Headers: {} 2024-02-01T10:38:02.128 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:02.128 DEBUG:boto:Port: 8001 2024-02-01T10:38:02.128 DEBUG:boto:Params: {} 2024-02-01T10:38:02.128 DEBUG:boto:Token: None 2024-02-01T10:38:02.129 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:02 GMT /vclfcn-40/?versions 2024-02-01T10:38:02.129 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/+oFab3yc91i4++Be3USVrERgE8= 2024-02-01T10:38:02.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/+oFab3yc91i4++Be3USVrERgE8=', 'Content-Length': '0'} 2024-02-01T10:38:02.133 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001128d05ed86d2bb6-0065bb748a-5324-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:02.133 DEBUG:boto:b'vclfcn-40999false' 2024-02-01T10:38:02.134 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-40 zones={test-zone2, test-zone3} 2024-02-01T10:38:02.134 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-41 zones={test-zone2, test-zone1} 2024-02-01T10:38:02.134 DEBUG:boto:path=/vclfcn-41/ 2024-02-01T10:38:02.134 DEBUG:boto:auth_path=/vclfcn-41/ 2024-02-01T10:38:02.134 DEBUG:boto:Method: HEAD 2024-02-01T10:38:02.134 DEBUG:boto:Path: /vclfcn-41/ 2024-02-01T10:38:02.134 DEBUG:boto:Data: 2024-02-01T10:38:02.134 DEBUG:boto:Headers: {} 2024-02-01T10:38:02.134 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:02.134 DEBUG:boto:Port: 8000 2024-02-01T10:38:02.134 DEBUG:boto:Params: {} 2024-02-01T10:38:02.135 DEBUG:boto:Token: None 2024-02-01T10:38:02.135 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:02 GMT /vclfcn-41/ 2024-02-01T10:38:02.135 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:OhN34pQF0m2790gYTBsTmQCOj70= 2024-02-01T10:38:02.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:OhN34pQF0m2790gYTBsTmQCOj70=', 'Content-Length': '0'} 2024-02-01T10:38:02.140 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e926fee1d9b6d2f8-0065bb748a-5004-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:02.140 DEBUG:boto:path=/vclfcn-41/ 2024-02-01T10:38:02.140 DEBUG:boto:auth_path=/vclfcn-41/ 2024-02-01T10:38:02.140 DEBUG:boto:Method: HEAD 2024-02-01T10:38:02.140 DEBUG:boto:Path: /vclfcn-41/ 2024-02-01T10:38:02.141 DEBUG:boto:Data: 2024-02-01T10:38:02.141 DEBUG:boto:Headers: {} 2024-02-01T10:38:02.141 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:02.141 DEBUG:boto:Port: 8000 2024-02-01T10:38:02.141 DEBUG:boto:Params: {} 2024-02-01T10:38:02.141 DEBUG:boto:Token: None 2024-02-01T10:38:02.141 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:02 GMT /vclfcn-41/ 2024-02-01T10:38:02.141 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:OhN34pQF0m2790gYTBsTmQCOj70= 2024-02-01T10:38:02.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:OhN34pQF0m2790gYTBsTmQCOj70=', 'Content-Length': '0'} 2024-02-01T10:38:02.147 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008694a875792f280d-0065bb748a-5317-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:02.147 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:38:02.148 DEBUG:boto:path=/vclfcn-41/ 2024-02-01T10:38:02.148 DEBUG:boto:auth_path=/vclfcn-41/ 2024-02-01T10:38:02.148 DEBUG:boto:path=/vclfcn-41/?versions&max-keys=999 2024-02-01T10:38:02.148 DEBUG:boto:auth_path=/vclfcn-41/?versions&max-keys=999 2024-02-01T10:38:02.148 DEBUG:boto:Method: GET 2024-02-01T10:38:02.148 DEBUG:boto:Path: /vclfcn-41/?versions&max-keys=999 2024-02-01T10:38:02.148 DEBUG:boto:Data: 2024-02-01T10:38:02.148 DEBUG:boto:Headers: {} 2024-02-01T10:38:02.148 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:02.148 DEBUG:boto:Port: 8000 2024-02-01T10:38:02.148 DEBUG:boto:Params: {} 2024-02-01T10:38:02.149 DEBUG:boto:Token: None 2024-02-01T10:38:02.149 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:02 GMT /vclfcn-41/?versions 2024-02-01T10:38:02.149 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:P+pwBlP9jbghJJMkDbb9VcBmkqE= 2024-02-01T10:38:02.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:P+pwBlP9jbghJJMkDbb9VcBmkqE=', 'Content-Length': '0'} 2024-02-01T10:38:02.154 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000446a6d873cd10acb-0065bb748a-5004-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:02.196 DEBUG:boto:b'vclfcn-41999false' 2024-02-01T10:38:02.196 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:38:02.197 DEBUG:boto:path=/vclfcn-41/ 2024-02-01T10:38:02.197 DEBUG:boto:auth_path=/vclfcn-41/ 2024-02-01T10:38:02.197 DEBUG:boto:path=/vclfcn-41/?versions&max-keys=999 2024-02-01T10:38:02.197 DEBUG:boto:auth_path=/vclfcn-41/?versions&max-keys=999 2024-02-01T10:38:02.197 DEBUG:boto:Method: GET 2024-02-01T10:38:02.197 DEBUG:boto:Path: /vclfcn-41/?versions&max-keys=999 2024-02-01T10:38:02.197 DEBUG:boto:Data: 2024-02-01T10:38:02.197 DEBUG:boto:Headers: {} 2024-02-01T10:38:02.197 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:02.197 DEBUG:boto:Port: 8000 2024-02-01T10:38:02.198 DEBUG:boto:Params: {} 2024-02-01T10:38:02.198 DEBUG:boto:Token: None 2024-02-01T10:38:02.198 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:02 GMT /vclfcn-41/?versions 2024-02-01T10:38:02.198 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:P+pwBlP9jbghJJMkDbb9VcBmkqE= 2024-02-01T10:38:02.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:P+pwBlP9jbghJJMkDbb9VcBmkqE=', 'Content-Length': '0'} 2024-02-01T10:38:02.203 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045b46240746cb63d-0065bb748a-5317-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:02.203 DEBUG:boto:b'vclfcn-41999false' 2024-02-01T10:38:02.203 DEBUG:boto:path=/vclfcn-41/ 2024-02-01T10:38:02.203 DEBUG:boto:auth_path=/vclfcn-41/ 2024-02-01T10:38:02.203 DEBUG:boto:path=/vclfcn-41/?versions&max-keys=999 2024-02-01T10:38:02.204 DEBUG:boto:auth_path=/vclfcn-41/?versions&max-keys=999 2024-02-01T10:38:02.204 DEBUG:boto:Method: GET 2024-02-01T10:38:02.204 DEBUG:boto:Path: /vclfcn-41/?versions&max-keys=999 2024-02-01T10:38:02.204 DEBUG:boto:Data: 2024-02-01T10:38:02.204 DEBUG:boto:Headers: {} 2024-02-01T10:38:02.204 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:02.204 DEBUG:boto:Port: 8000 2024-02-01T10:38:02.204 DEBUG:boto:Params: {} 2024-02-01T10:38:02.204 DEBUG:boto:Token: None 2024-02-01T10:38:02.204 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:02 GMT /vclfcn-41/?versions 2024-02-01T10:38:02.205 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:P+pwBlP9jbghJJMkDbb9VcBmkqE= 2024-02-01T10:38:02.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:P+pwBlP9jbghJJMkDbb9VcBmkqE=', 'Content-Length': '0'} 2024-02-01T10:38:02.209 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024f3150d120a65f4-0065bb748a-5004-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:02.209 DEBUG:boto:b'vclfcn-41999false' 2024-02-01T10:38:02.209 DEBUG:boto:path=/vclfcn-41/ 2024-02-01T10:38:02.210 DEBUG:boto:auth_path=/vclfcn-41/ 2024-02-01T10:38:02.210 DEBUG:boto:path=/vclfcn-41/?versions&max-keys=999 2024-02-01T10:38:02.210 DEBUG:boto:auth_path=/vclfcn-41/?versions&max-keys=999 2024-02-01T10:38:02.210 DEBUG:boto:Method: GET 2024-02-01T10:38:02.210 DEBUG:boto:Path: /vclfcn-41/?versions&max-keys=999 2024-02-01T10:38:02.210 DEBUG:boto:Data: 2024-02-01T10:38:02.210 DEBUG:boto:Headers: {} 2024-02-01T10:38:02.210 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:02.210 DEBUG:boto:Port: 8000 2024-02-01T10:38:02.210 DEBUG:boto:Params: {} 2024-02-01T10:38:02.210 DEBUG:boto:Token: None 2024-02-01T10:38:02.211 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:02 GMT /vclfcn-41/?versions 2024-02-01T10:38:02.211 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:P+pwBlP9jbghJJMkDbb9VcBmkqE= 2024-02-01T10:38:02.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:P+pwBlP9jbghJJMkDbb9VcBmkqE=', 'Content-Length': '0'} 2024-02-01T10:38:02.215 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ba420a9da29c353-0065bb748a-5317-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:02.256 DEBUG:boto:b'vclfcn-41999false' 2024-02-01T10:38:02.256 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-41 zones={test-zone2, test-zone1} 2024-02-01T10:38:02.257 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-41 zones={test-zone3, test-zone1} 2024-02-01T10:38:02.257 DEBUG:boto:path=/vclfcn-41/ 2024-02-01T10:38:02.257 DEBUG:boto:auth_path=/vclfcn-41/ 2024-02-01T10:38:02.257 DEBUG:boto:Method: HEAD 2024-02-01T10:38:02.257 DEBUG:boto:Path: /vclfcn-41/ 2024-02-01T10:38:02.257 DEBUG:boto:Data: 2024-02-01T10:38:02.257 DEBUG:boto:Headers: {} 2024-02-01T10:38:02.257 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:02.257 DEBUG:boto:Port: 8001 2024-02-01T10:38:02.258 DEBUG:boto:Params: {} 2024-02-01T10:38:02.258 DEBUG:boto:Token: None 2024-02-01T10:38:02.258 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:02 GMT /vclfcn-41/ 2024-02-01T10:38:02.258 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:OhN34pQF0m2790gYTBsTmQCOj70= 2024-02-01T10:38:02.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:OhN34pQF0m2790gYTBsTmQCOj70=', 'Content-Length': '0'} 2024-02-01T10:38:02.264 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ceb8ad3742089b6d-0065bb748a-5324-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:02.264 DEBUG:boto:path=/vclfcn-41/ 2024-02-01T10:38:02.265 DEBUG:boto:auth_path=/vclfcn-41/ 2024-02-01T10:38:02.265 DEBUG:boto:Method: HEAD 2024-02-01T10:38:02.265 DEBUG:boto:Path: /vclfcn-41/ 2024-02-01T10:38:02.265 DEBUG:boto:Data: 2024-02-01T10:38:02.265 DEBUG:boto:Headers: {} 2024-02-01T10:38:02.265 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:02.265 DEBUG:boto:Port: 8000 2024-02-01T10:38:02.266 DEBUG:boto:Params: {} 2024-02-01T10:38:02.266 DEBUG:boto:Token: None 2024-02-01T10:38:02.266 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:02 GMT /vclfcn-41/ 2024-02-01T10:38:02.266 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:OhN34pQF0m2790gYTBsTmQCOj70= 2024-02-01T10:38:02.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:OhN34pQF0m2790gYTBsTmQCOj70=', 'Content-Length': '0'} 2024-02-01T10:38:02.271 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000068cb3eef4bf631d1-0065bb748a-5317-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:02.271 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:38:02.271 DEBUG:boto:path=/vclfcn-41/ 2024-02-01T10:38:02.271 DEBUG:boto:auth_path=/vclfcn-41/ 2024-02-01T10:38:02.271 DEBUG:boto:path=/vclfcn-41/?versions&max-keys=999 2024-02-01T10:38:02.271 DEBUG:boto:auth_path=/vclfcn-41/?versions&max-keys=999 2024-02-01T10:38:02.272 DEBUG:boto:Method: GET 2024-02-01T10:38:02.272 DEBUG:boto:Path: /vclfcn-41/?versions&max-keys=999 2024-02-01T10:38:02.272 DEBUG:boto:Data: 2024-02-01T10:38:02.272 DEBUG:boto:Headers: {} 2024-02-01T10:38:02.272 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:02.272 DEBUG:boto:Port: 8001 2024-02-01T10:38:02.272 DEBUG:boto:Params: {} 2024-02-01T10:38:02.272 DEBUG:boto:Token: None 2024-02-01T10:38:02.272 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:02 GMT /vclfcn-41/?versions 2024-02-01T10:38:02.273 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:P+pwBlP9jbghJJMkDbb9VcBmkqE= 2024-02-01T10:38:02.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:P+pwBlP9jbghJJMkDbb9VcBmkqE=', 'Content-Length': '0'} 2024-02-01T10:38:02.277 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d235234d74704502-0065bb748a-5324-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:02.277 DEBUG:boto:b'vclfcn-41999false' 2024-02-01T10:38:02.277 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:38:02.278 DEBUG:boto:path=/vclfcn-41/ 2024-02-01T10:38:02.278 DEBUG:boto:auth_path=/vclfcn-41/ 2024-02-01T10:38:02.278 DEBUG:boto:path=/vclfcn-41/?versions&max-keys=999 2024-02-01T10:38:02.278 DEBUG:boto:auth_path=/vclfcn-41/?versions&max-keys=999 2024-02-01T10:38:02.278 DEBUG:boto:Method: GET 2024-02-01T10:38:02.278 DEBUG:boto:Path: /vclfcn-41/?versions&max-keys=999 2024-02-01T10:38:02.278 DEBUG:boto:Data: 2024-02-01T10:38:02.278 DEBUG:boto:Headers: {} 2024-02-01T10:38:02.278 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:02.279 DEBUG:boto:Port: 8000 2024-02-01T10:38:02.279 DEBUG:boto:Params: {} 2024-02-01T10:38:02.279 DEBUG:boto:Token: None 2024-02-01T10:38:02.279 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:02 GMT /vclfcn-41/?versions 2024-02-01T10:38:02.279 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:P+pwBlP9jbghJJMkDbb9VcBmkqE= 2024-02-01T10:38:02.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:P+pwBlP9jbghJJMkDbb9VcBmkqE=', 'Content-Length': '0'} 2024-02-01T10:38:02.284 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043123aea376c1108-0065bb748a-5317-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:02.284 DEBUG:boto:b'vclfcn-41999false' 2024-02-01T10:38:02.284 DEBUG:boto:path=/vclfcn-41/ 2024-02-01T10:38:02.284 DEBUG:boto:auth_path=/vclfcn-41/ 2024-02-01T10:38:02.285 DEBUG:boto:path=/vclfcn-41/?versions&max-keys=999 2024-02-01T10:38:02.285 DEBUG:boto:auth_path=/vclfcn-41/?versions&max-keys=999 2024-02-01T10:38:02.285 DEBUG:boto:Method: GET 2024-02-01T10:38:02.285 DEBUG:boto:Path: /vclfcn-41/?versions&max-keys=999 2024-02-01T10:38:02.285 DEBUG:boto:Data: 2024-02-01T10:38:02.285 DEBUG:boto:Headers: {} 2024-02-01T10:38:02.285 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:02.285 DEBUG:boto:Port: 8001 2024-02-01T10:38:02.285 DEBUG:boto:Params: {} 2024-02-01T10:38:02.286 DEBUG:boto:Token: None 2024-02-01T10:38:02.286 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:02 GMT /vclfcn-41/?versions 2024-02-01T10:38:02.286 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:P+pwBlP9jbghJJMkDbb9VcBmkqE= 2024-02-01T10:38:02.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:P+pwBlP9jbghJJMkDbb9VcBmkqE=', 'Content-Length': '0'} 2024-02-01T10:38:02.291 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4c78511dac773d3-0065bb748a-5324-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:02.336 DEBUG:boto:b'vclfcn-41999false' 2024-02-01T10:38:02.337 DEBUG:boto:path=/vclfcn-41/ 2024-02-01T10:38:02.337 DEBUG:boto:auth_path=/vclfcn-41/ 2024-02-01T10:38:02.337 DEBUG:boto:path=/vclfcn-41/?versions&max-keys=999 2024-02-01T10:38:02.337 DEBUG:boto:auth_path=/vclfcn-41/?versions&max-keys=999 2024-02-01T10:38:02.337 DEBUG:boto:Method: GET 2024-02-01T10:38:02.337 DEBUG:boto:Path: /vclfcn-41/?versions&max-keys=999 2024-02-01T10:38:02.337 DEBUG:boto:Data: 2024-02-01T10:38:02.337 DEBUG:boto:Headers: {} 2024-02-01T10:38:02.337 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:02.338 DEBUG:boto:Port: 8000 2024-02-01T10:38:02.338 DEBUG:boto:Params: {} 2024-02-01T10:38:02.338 DEBUG:boto:Token: None 2024-02-01T10:38:02.338 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:02 GMT /vclfcn-41/?versions 2024-02-01T10:38:02.338 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:P+pwBlP9jbghJJMkDbb9VcBmkqE= 2024-02-01T10:38:02.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:P+pwBlP9jbghJJMkDbb9VcBmkqE=', 'Content-Length': '0'} 2024-02-01T10:38:02.343 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000853071b64a3354c4-0065bb748a-5317-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:02.343 DEBUG:boto:b'vclfcn-41999false' 2024-02-01T10:38:02.344 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-41 zones={test-zone3, test-zone1} 2024-02-01T10:38:02.344 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-41 zones={test-zone1, test-zone2} 2024-02-01T10:38:02.345 DEBUG:boto:path=/vclfcn-41/ 2024-02-01T10:38:02.345 DEBUG:boto:auth_path=/vclfcn-41/ 2024-02-01T10:38:02.345 DEBUG:boto:Method: HEAD 2024-02-01T10:38:02.345 DEBUG:boto:Path: /vclfcn-41/ 2024-02-01T10:38:02.345 DEBUG:boto:Data: 2024-02-01T10:38:02.345 DEBUG:boto:Headers: {} 2024-02-01T10:38:02.345 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:02.345 DEBUG:boto:Port: 8000 2024-02-01T10:38:02.345 DEBUG:boto:Params: {} 2024-02-01T10:38:02.346 DEBUG:boto:Token: None 2024-02-01T10:38:02.346 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:02 GMT /vclfcn-41/ 2024-02-01T10:38:02.348 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:OhN34pQF0m2790gYTBsTmQCOj70= 2024-02-01T10:38:02.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:OhN34pQF0m2790gYTBsTmQCOj70=', 'Content-Length': '0'} 2024-02-01T10:38:02.353 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ee45718d693201f1-0065bb748a-5317-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:02.353 DEBUG:boto:path=/vclfcn-41/ 2024-02-01T10:38:02.353 DEBUG:boto:auth_path=/vclfcn-41/ 2024-02-01T10:38:02.353 DEBUG:boto:Method: HEAD 2024-02-01T10:38:02.354 DEBUG:boto:Path: /vclfcn-41/ 2024-02-01T10:38:02.354 DEBUG:boto:Data: 2024-02-01T10:38:02.354 DEBUG:boto:Headers: {} 2024-02-01T10:38:02.354 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:02.354 DEBUG:boto:Port: 8000 2024-02-01T10:38:02.354 DEBUG:boto:Params: {} 2024-02-01T10:38:02.354 DEBUG:boto:Token: None 2024-02-01T10:38:02.354 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:02 GMT /vclfcn-41/ 2024-02-01T10:38:02.354 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:OhN34pQF0m2790gYTBsTmQCOj70= 2024-02-01T10:38:02.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:OhN34pQF0m2790gYTBsTmQCOj70=', 'Content-Length': '0'} 2024-02-01T10:38:02.359 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006157b2090375bdb1-0065bb748a-5004-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:02.359 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:38:02.359 DEBUG:boto:path=/vclfcn-41/ 2024-02-01T10:38:02.360 DEBUG:boto:auth_path=/vclfcn-41/ 2024-02-01T10:38:02.360 DEBUG:boto:path=/vclfcn-41/?versions&max-keys=999 2024-02-01T10:38:02.360 DEBUG:boto:auth_path=/vclfcn-41/?versions&max-keys=999 2024-02-01T10:38:02.360 DEBUG:boto:Method: GET 2024-02-01T10:38:02.360 DEBUG:boto:Path: /vclfcn-41/?versions&max-keys=999 2024-02-01T10:38:02.360 DEBUG:boto:Data: 2024-02-01T10:38:02.360 DEBUG:boto:Headers: {} 2024-02-01T10:38:02.360 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:02.360 DEBUG:boto:Port: 8000 2024-02-01T10:38:02.360 DEBUG:boto:Params: {} 2024-02-01T10:38:02.361 DEBUG:boto:Token: None 2024-02-01T10:38:02.361 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:02 GMT /vclfcn-41/?versions 2024-02-01T10:38:02.361 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:P+pwBlP9jbghJJMkDbb9VcBmkqE= 2024-02-01T10:38:02.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:P+pwBlP9jbghJJMkDbb9VcBmkqE=', 'Content-Length': '0'} 2024-02-01T10:38:02.365 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d7c06a53d42654c-0065bb748a-5317-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:02.408 DEBUG:boto:b'vclfcn-41999false' 2024-02-01T10:38:02.409 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:38:02.409 DEBUG:boto:path=/vclfcn-41/ 2024-02-01T10:38:02.409 DEBUG:boto:auth_path=/vclfcn-41/ 2024-02-01T10:38:02.409 DEBUG:boto:path=/vclfcn-41/?versions&max-keys=999 2024-02-01T10:38:02.409 DEBUG:boto:auth_path=/vclfcn-41/?versions&max-keys=999 2024-02-01T10:38:02.409 DEBUG:boto:Method: GET 2024-02-01T10:38:02.409 DEBUG:boto:Path: /vclfcn-41/?versions&max-keys=999 2024-02-01T10:38:02.409 DEBUG:boto:Data: 2024-02-01T10:38:02.410 DEBUG:boto:Headers: {} 2024-02-01T10:38:02.410 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:02.410 DEBUG:boto:Port: 8000 2024-02-01T10:38:02.410 DEBUG:boto:Params: {} 2024-02-01T10:38:02.410 DEBUG:boto:Token: None 2024-02-01T10:38:02.410 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:02 GMT /vclfcn-41/?versions 2024-02-01T10:38:02.411 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:P+pwBlP9jbghJJMkDbb9VcBmkqE= 2024-02-01T10:38:02.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:P+pwBlP9jbghJJMkDbb9VcBmkqE=', 'Content-Length': '0'} 2024-02-01T10:38:02.416 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000573746f17f49765d-0065bb748a-5004-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:02.416 DEBUG:boto:b'vclfcn-41999false' 2024-02-01T10:38:02.417 DEBUG:boto:path=/vclfcn-41/ 2024-02-01T10:38:02.417 DEBUG:boto:auth_path=/vclfcn-41/ 2024-02-01T10:38:02.417 DEBUG:boto:path=/vclfcn-41/?versions&max-keys=999 2024-02-01T10:38:02.417 DEBUG:boto:auth_path=/vclfcn-41/?versions&max-keys=999 2024-02-01T10:38:02.417 DEBUG:boto:Method: GET 2024-02-01T10:38:02.417 DEBUG:boto:Path: /vclfcn-41/?versions&max-keys=999 2024-02-01T10:38:02.417 DEBUG:boto:Data: 2024-02-01T10:38:02.417 DEBUG:boto:Headers: {} 2024-02-01T10:38:02.418 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:02.418 DEBUG:boto:Port: 8000 2024-02-01T10:38:02.418 DEBUG:boto:Params: {} 2024-02-01T10:38:02.418 DEBUG:boto:Token: None 2024-02-01T10:38:02.418 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:02 GMT /vclfcn-41/?versions 2024-02-01T10:38:02.418 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:P+pwBlP9jbghJJMkDbb9VcBmkqE= 2024-02-01T10:38:02.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:P+pwBlP9jbghJJMkDbb9VcBmkqE=', 'Content-Length': '0'} 2024-02-01T10:38:02.424 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d57f129a3d43dba4-0065bb748a-5317-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:02.424 DEBUG:boto:b'vclfcn-41999false' 2024-02-01T10:38:02.424 DEBUG:boto:path=/vclfcn-41/ 2024-02-01T10:38:02.424 DEBUG:boto:auth_path=/vclfcn-41/ 2024-02-01T10:38:02.424 DEBUG:boto:path=/vclfcn-41/?versions&max-keys=999 2024-02-01T10:38:02.424 DEBUG:boto:auth_path=/vclfcn-41/?versions&max-keys=999 2024-02-01T10:38:02.425 DEBUG:boto:Method: GET 2024-02-01T10:38:02.425 DEBUG:boto:Path: /vclfcn-41/?versions&max-keys=999 2024-02-01T10:38:02.425 DEBUG:boto:Data: 2024-02-01T10:38:02.425 DEBUG:boto:Headers: {} 2024-02-01T10:38:02.425 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:02.425 DEBUG:boto:Port: 8000 2024-02-01T10:38:02.425 DEBUG:boto:Params: {} 2024-02-01T10:38:02.425 DEBUG:boto:Token: None 2024-02-01T10:38:02.425 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:02 GMT /vclfcn-41/?versions 2024-02-01T10:38:02.425 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:P+pwBlP9jbghJJMkDbb9VcBmkqE= 2024-02-01T10:38:02.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:P+pwBlP9jbghJJMkDbb9VcBmkqE=', 'Content-Length': '0'} 2024-02-01T10:38:02.430 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c18332d692f3cc7c-0065bb748a-5004-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:02.430 DEBUG:boto:b'vclfcn-41999false' 2024-02-01T10:38:02.430 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-41 zones={test-zone1, test-zone2} 2024-02-01T10:38:02.431 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-41 zones={test-zone3, test-zone2} 2024-02-01T10:38:02.431 DEBUG:boto:path=/vclfcn-41/ 2024-02-01T10:38:02.431 DEBUG:boto:auth_path=/vclfcn-41/ 2024-02-01T10:38:02.431 DEBUG:boto:Method: HEAD 2024-02-01T10:38:02.431 DEBUG:boto:Path: /vclfcn-41/ 2024-02-01T10:38:02.431 DEBUG:boto:Data: 2024-02-01T10:38:02.431 DEBUG:boto:Headers: {} 2024-02-01T10:38:02.431 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:02.431 DEBUG:boto:Port: 8001 2024-02-01T10:38:02.431 DEBUG:boto:Params: {} 2024-02-01T10:38:02.432 DEBUG:boto:Token: None 2024-02-01T10:38:02.432 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:02 GMT /vclfcn-41/ 2024-02-01T10:38:02.432 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:OhN34pQF0m2790gYTBsTmQCOj70= 2024-02-01T10:38:02.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:OhN34pQF0m2790gYTBsTmQCOj70=', 'Content-Length': '0'} 2024-02-01T10:38:02.436 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008e1ab0563bf55829-0065bb748a-5324-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:02.436 DEBUG:boto:path=/vclfcn-41/ 2024-02-01T10:38:02.437 DEBUG:boto:auth_path=/vclfcn-41/ 2024-02-01T10:38:02.437 DEBUG:boto:Method: HEAD 2024-02-01T10:38:02.437 DEBUG:boto:Path: /vclfcn-41/ 2024-02-01T10:38:02.437 DEBUG:boto:Data: 2024-02-01T10:38:02.437 DEBUG:boto:Headers: {} 2024-02-01T10:38:02.437 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:02.437 DEBUG:boto:Port: 8000 2024-02-01T10:38:02.438 DEBUG:boto:Params: {} 2024-02-01T10:38:02.438 DEBUG:boto:Token: None 2024-02-01T10:38:02.438 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:02 GMT /vclfcn-41/ 2024-02-01T10:38:02.438 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:OhN34pQF0m2790gYTBsTmQCOj70= 2024-02-01T10:38:02.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:OhN34pQF0m2790gYTBsTmQCOj70=', 'Content-Length': '0'} 2024-02-01T10:38:02.443 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004faeeee78cd2ddbf-0065bb748a-5004-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:02.443 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:38:02.443 DEBUG:boto:path=/vclfcn-41/ 2024-02-01T10:38:02.443 DEBUG:boto:auth_path=/vclfcn-41/ 2024-02-01T10:38:02.443 DEBUG:boto:path=/vclfcn-41/?versions&max-keys=999 2024-02-01T10:38:02.443 DEBUG:boto:auth_path=/vclfcn-41/?versions&max-keys=999 2024-02-01T10:38:02.443 DEBUG:boto:Method: GET 2024-02-01T10:38:02.443 DEBUG:boto:Path: /vclfcn-41/?versions&max-keys=999 2024-02-01T10:38:02.443 DEBUG:boto:Data: 2024-02-01T10:38:02.444 DEBUG:boto:Headers: {} 2024-02-01T10:38:02.444 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:02.444 DEBUG:boto:Port: 8001 2024-02-01T10:38:02.444 DEBUG:boto:Params: {} 2024-02-01T10:38:02.444 DEBUG:boto:Token: None 2024-02-01T10:38:02.444 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:02 GMT /vclfcn-41/?versions 2024-02-01T10:38:02.444 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:P+pwBlP9jbghJJMkDbb9VcBmkqE= 2024-02-01T10:38:02.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:P+pwBlP9jbghJJMkDbb9VcBmkqE=', 'Content-Length': '0'} 2024-02-01T10:38:02.449 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f4ee69b6cd1a3ce-0065bb748a-5324-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:02.450 DEBUG:boto:b'vclfcn-41999false' 2024-02-01T10:38:02.450 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:38:02.450 DEBUG:boto:path=/vclfcn-41/ 2024-02-01T10:38:02.450 DEBUG:boto:auth_path=/vclfcn-41/ 2024-02-01T10:38:02.450 DEBUG:boto:path=/vclfcn-41/?versions&max-keys=999 2024-02-01T10:38:02.451 DEBUG:boto:auth_path=/vclfcn-41/?versions&max-keys=999 2024-02-01T10:38:02.451 DEBUG:boto:Method: GET 2024-02-01T10:38:02.451 DEBUG:boto:Path: /vclfcn-41/?versions&max-keys=999 2024-02-01T10:38:02.451 DEBUG:boto:Data: 2024-02-01T10:38:02.451 DEBUG:boto:Headers: {} 2024-02-01T10:38:02.451 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:02.451 DEBUG:boto:Port: 8000 2024-02-01T10:38:02.451 DEBUG:boto:Params: {} 2024-02-01T10:38:02.451 DEBUG:boto:Token: None 2024-02-01T10:38:02.451 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:02 GMT /vclfcn-41/?versions 2024-02-01T10:38:02.452 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:P+pwBlP9jbghJJMkDbb9VcBmkqE= 2024-02-01T10:38:02.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:P+pwBlP9jbghJJMkDbb9VcBmkqE=', 'Content-Length': '0'} 2024-02-01T10:38:02.460 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c59460788a44e77c-0065bb748a-5004-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:02.500 DEBUG:boto:b'vclfcn-41999false' 2024-02-01T10:38:02.500 DEBUG:boto:path=/vclfcn-41/ 2024-02-01T10:38:02.501 DEBUG:boto:auth_path=/vclfcn-41/ 2024-02-01T10:38:02.501 DEBUG:boto:path=/vclfcn-41/?versions&max-keys=999 2024-02-01T10:38:02.501 DEBUG:boto:auth_path=/vclfcn-41/?versions&max-keys=999 2024-02-01T10:38:02.501 DEBUG:boto:Method: GET 2024-02-01T10:38:02.501 DEBUG:boto:Path: /vclfcn-41/?versions&max-keys=999 2024-02-01T10:38:02.501 DEBUG:boto:Data: 2024-02-01T10:38:02.501 DEBUG:boto:Headers: {} 2024-02-01T10:38:02.501 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:02.501 DEBUG:boto:Port: 8001 2024-02-01T10:38:02.501 DEBUG:boto:Params: {} 2024-02-01T10:38:02.502 DEBUG:boto:Token: None 2024-02-01T10:38:02.502 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:02 GMT /vclfcn-41/?versions 2024-02-01T10:38:02.502 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:P+pwBlP9jbghJJMkDbb9VcBmkqE= 2024-02-01T10:38:02.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:P+pwBlP9jbghJJMkDbb9VcBmkqE=', 'Content-Length': '0'} 2024-02-01T10:38:02.507 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e353671763e04cf5-0065bb748a-5324-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:02.507 DEBUG:boto:b'vclfcn-41999false' 2024-02-01T10:38:02.508 DEBUG:boto:path=/vclfcn-41/ 2024-02-01T10:38:02.508 DEBUG:boto:auth_path=/vclfcn-41/ 2024-02-01T10:38:02.508 DEBUG:boto:path=/vclfcn-41/?versions&max-keys=999 2024-02-01T10:38:02.508 DEBUG:boto:auth_path=/vclfcn-41/?versions&max-keys=999 2024-02-01T10:38:02.508 DEBUG:boto:Method: GET 2024-02-01T10:38:02.508 DEBUG:boto:Path: /vclfcn-41/?versions&max-keys=999 2024-02-01T10:38:02.508 DEBUG:boto:Data: 2024-02-01T10:38:02.508 DEBUG:boto:Headers: {} 2024-02-01T10:38:02.508 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:02.509 DEBUG:boto:Port: 8000 2024-02-01T10:38:02.509 DEBUG:boto:Params: {} 2024-02-01T10:38:02.509 DEBUG:boto:Token: None 2024-02-01T10:38:02.509 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:02 GMT /vclfcn-41/?versions 2024-02-01T10:38:02.509 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:P+pwBlP9jbghJJMkDbb9VcBmkqE= 2024-02-01T10:38:02.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:P+pwBlP9jbghJJMkDbb9VcBmkqE=', 'Content-Length': '0'} 2024-02-01T10:38:02.513 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000093d5b8a399456cf1-0065bb748a-5004-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:02.513 DEBUG:boto:b'vclfcn-41999false' 2024-02-01T10:38:02.513 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-41 zones={test-zone3, test-zone2} 2024-02-01T10:38:02.513 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-41 zones={test-zone1, test-zone3} 2024-02-01T10:38:02.514 DEBUG:boto:path=/vclfcn-41/ 2024-02-01T10:38:02.514 DEBUG:boto:auth_path=/vclfcn-41/ 2024-02-01T10:38:02.514 DEBUG:boto:Method: HEAD 2024-02-01T10:38:02.514 DEBUG:boto:Path: /vclfcn-41/ 2024-02-01T10:38:02.514 DEBUG:boto:Data: 2024-02-01T10:38:02.514 DEBUG:boto:Headers: {} 2024-02-01T10:38:02.514 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:02.514 DEBUG:boto:Port: 8000 2024-02-01T10:38:02.514 DEBUG:boto:Params: {} 2024-02-01T10:38:02.514 DEBUG:boto:Token: None 2024-02-01T10:38:02.514 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:02 GMT /vclfcn-41/ 2024-02-01T10:38:02.515 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:OhN34pQF0m2790gYTBsTmQCOj70= 2024-02-01T10:38:02.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:OhN34pQF0m2790gYTBsTmQCOj70=', 'Content-Length': '0'} 2024-02-01T10:38:02.518 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ae2d580a0d2b50bd-0065bb748a-5317-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:02.519 DEBUG:boto:path=/vclfcn-41/ 2024-02-01T10:38:02.519 DEBUG:boto:auth_path=/vclfcn-41/ 2024-02-01T10:38:02.519 DEBUG:boto:Method: HEAD 2024-02-01T10:38:02.519 DEBUG:boto:Path: /vclfcn-41/ 2024-02-01T10:38:02.519 DEBUG:boto:Data: 2024-02-01T10:38:02.519 DEBUG:boto:Headers: {} 2024-02-01T10:38:02.519 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:02.519 DEBUG:boto:Port: 8001 2024-02-01T10:38:02.519 DEBUG:boto:Params: {} 2024-02-01T10:38:02.519 DEBUG:boto:Token: None 2024-02-01T10:38:02.520 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:02 GMT /vclfcn-41/ 2024-02-01T10:38:02.520 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:OhN34pQF0m2790gYTBsTmQCOj70= 2024-02-01T10:38:02.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:OhN34pQF0m2790gYTBsTmQCOj70=', 'Content-Length': '0'} 2024-02-01T10:38:02.524 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003afd6f373d2c4c59-0065bb748a-5324-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:02.524 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:38:02.525 DEBUG:boto:path=/vclfcn-41/ 2024-02-01T10:38:02.525 DEBUG:boto:auth_path=/vclfcn-41/ 2024-02-01T10:38:02.525 DEBUG:boto:path=/vclfcn-41/?versions&max-keys=999 2024-02-01T10:38:02.525 DEBUG:boto:auth_path=/vclfcn-41/?versions&max-keys=999 2024-02-01T10:38:02.525 DEBUG:boto:Method: GET 2024-02-01T10:38:02.525 DEBUG:boto:Path: /vclfcn-41/?versions&max-keys=999 2024-02-01T10:38:02.525 DEBUG:boto:Data: 2024-02-01T10:38:02.525 DEBUG:boto:Headers: {} 2024-02-01T10:38:02.525 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:02.525 DEBUG:boto:Port: 8000 2024-02-01T10:38:02.526 DEBUG:boto:Params: {} 2024-02-01T10:38:02.526 DEBUG:boto:Token: None 2024-02-01T10:38:02.526 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:02 GMT /vclfcn-41/?versions 2024-02-01T10:38:02.526 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:P+pwBlP9jbghJJMkDbb9VcBmkqE= 2024-02-01T10:38:02.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:P+pwBlP9jbghJJMkDbb9VcBmkqE=', 'Content-Length': '0'} 2024-02-01T10:38:02.531 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009365b626a79ebd5c-0065bb748a-5317-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:02.531 DEBUG:boto:b'vclfcn-41999false' 2024-02-01T10:38:02.532 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:38:02.532 DEBUG:boto:path=/vclfcn-41/ 2024-02-01T10:38:02.532 DEBUG:boto:auth_path=/vclfcn-41/ 2024-02-01T10:38:02.532 DEBUG:boto:path=/vclfcn-41/?versions&max-keys=999 2024-02-01T10:38:02.532 DEBUG:boto:auth_path=/vclfcn-41/?versions&max-keys=999 2024-02-01T10:38:02.532 DEBUG:boto:Method: GET 2024-02-01T10:38:02.532 DEBUG:boto:Path: /vclfcn-41/?versions&max-keys=999 2024-02-01T10:38:02.532 DEBUG:boto:Data: 2024-02-01T10:38:02.532 DEBUG:boto:Headers: {} 2024-02-01T10:38:02.533 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:02.533 DEBUG:boto:Port: 8001 2024-02-01T10:38:02.533 DEBUG:boto:Params: {} 2024-02-01T10:38:02.533 DEBUG:boto:Token: None 2024-02-01T10:38:02.533 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:02 GMT /vclfcn-41/?versions 2024-02-01T10:38:02.533 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:P+pwBlP9jbghJJMkDbb9VcBmkqE= 2024-02-01T10:38:02.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:P+pwBlP9jbghJJMkDbb9VcBmkqE=', 'Content-Length': '0'} 2024-02-01T10:38:02.538 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab02f926f45db6e0-0065bb748a-5324-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:02.580 DEBUG:boto:b'vclfcn-41999false' 2024-02-01T10:38:02.580 DEBUG:boto:path=/vclfcn-41/ 2024-02-01T10:38:02.584 DEBUG:boto:auth_path=/vclfcn-41/ 2024-02-01T10:38:02.584 DEBUG:boto:path=/vclfcn-41/?versions&max-keys=999 2024-02-01T10:38:02.584 DEBUG:boto:auth_path=/vclfcn-41/?versions&max-keys=999 2024-02-01T10:38:02.584 DEBUG:boto:Method: GET 2024-02-01T10:38:02.584 DEBUG:boto:Path: /vclfcn-41/?versions&max-keys=999 2024-02-01T10:38:02.584 DEBUG:boto:Data: 2024-02-01T10:38:02.584 DEBUG:boto:Headers: {} 2024-02-01T10:38:02.584 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:02.584 DEBUG:boto:Port: 8000 2024-02-01T10:38:02.584 DEBUG:boto:Params: {} 2024-02-01T10:38:02.585 DEBUG:boto:Token: None 2024-02-01T10:38:02.585 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:02 GMT /vclfcn-41/?versions 2024-02-01T10:38:02.585 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:P+pwBlP9jbghJJMkDbb9VcBmkqE= 2024-02-01T10:38:02.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:P+pwBlP9jbghJJMkDbb9VcBmkqE=', 'Content-Length': '0'} 2024-02-01T10:38:02.589 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c7dfbf7945bd119-0065bb748a-5317-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:02.590 DEBUG:boto:b'vclfcn-41999false' 2024-02-01T10:38:02.590 DEBUG:boto:path=/vclfcn-41/ 2024-02-01T10:38:02.590 DEBUG:boto:auth_path=/vclfcn-41/ 2024-02-01T10:38:02.590 DEBUG:boto:path=/vclfcn-41/?versions&max-keys=999 2024-02-01T10:38:02.590 DEBUG:boto:auth_path=/vclfcn-41/?versions&max-keys=999 2024-02-01T10:38:02.590 DEBUG:boto:Method: GET 2024-02-01T10:38:02.590 DEBUG:boto:Path: /vclfcn-41/?versions&max-keys=999 2024-02-01T10:38:02.591 DEBUG:boto:Data: 2024-02-01T10:38:02.591 DEBUG:boto:Headers: {} 2024-02-01T10:38:02.591 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:02.591 DEBUG:boto:Port: 8001 2024-02-01T10:38:02.591 DEBUG:boto:Params: {} 2024-02-01T10:38:02.591 DEBUG:boto:Token: None 2024-02-01T10:38:02.591 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:02 GMT /vclfcn-41/?versions 2024-02-01T10:38:02.591 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:P+pwBlP9jbghJJMkDbb9VcBmkqE= 2024-02-01T10:38:02.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:P+pwBlP9jbghJJMkDbb9VcBmkqE=', 'Content-Length': '0'} 2024-02-01T10:38:02.596 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000094feebc1845818b-0065bb748a-5324-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:02.596 DEBUG:boto:b'vclfcn-41999false' 2024-02-01T10:38:02.596 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-41 zones={test-zone1, test-zone3} 2024-02-01T10:38:02.596 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-41 zones={test-zone2, test-zone3} 2024-02-01T10:38:02.597 DEBUG:boto:path=/vclfcn-41/ 2024-02-01T10:38:02.597 DEBUG:boto:auth_path=/vclfcn-41/ 2024-02-01T10:38:02.597 DEBUG:boto:Method: HEAD 2024-02-01T10:38:02.597 DEBUG:boto:Path: /vclfcn-41/ 2024-02-01T10:38:02.597 DEBUG:boto:Data: 2024-02-01T10:38:02.597 DEBUG:boto:Headers: {} 2024-02-01T10:38:02.597 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:02.597 DEBUG:boto:Port: 8000 2024-02-01T10:38:02.598 DEBUG:boto:Params: {} 2024-02-01T10:38:02.598 DEBUG:boto:Token: None 2024-02-01T10:38:02.598 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:02 GMT /vclfcn-41/ 2024-02-01T10:38:02.598 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:OhN34pQF0m2790gYTBsTmQCOj70= 2024-02-01T10:38:02.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:OhN34pQF0m2790gYTBsTmQCOj70=', 'Content-Length': '0'} 2024-02-01T10:38:02.602 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007f2767e16b8e365a-0065bb748a-5004-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:02.603 DEBUG:boto:path=/vclfcn-41/ 2024-02-01T10:38:02.603 DEBUG:boto:auth_path=/vclfcn-41/ 2024-02-01T10:38:02.603 DEBUG:boto:Method: HEAD 2024-02-01T10:38:02.603 DEBUG:boto:Path: /vclfcn-41/ 2024-02-01T10:38:02.603 DEBUG:boto:Data: 2024-02-01T10:38:02.603 DEBUG:boto:Headers: {} 2024-02-01T10:38:02.603 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:02.603 DEBUG:boto:Port: 8001 2024-02-01T10:38:02.603 DEBUG:boto:Params: {} 2024-02-01T10:38:02.603 DEBUG:boto:Token: None 2024-02-01T10:38:02.604 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:02 GMT /vclfcn-41/ 2024-02-01T10:38:02.604 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:OhN34pQF0m2790gYTBsTmQCOj70= 2024-02-01T10:38:02.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:OhN34pQF0m2790gYTBsTmQCOj70=', 'Content-Length': '0'} 2024-02-01T10:38:02.608 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e96e226f557882a8-0065bb748a-5324-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:02.609 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:38:02.609 DEBUG:boto:path=/vclfcn-41/ 2024-02-01T10:38:02.609 DEBUG:boto:auth_path=/vclfcn-41/ 2024-02-01T10:38:02.609 DEBUG:boto:path=/vclfcn-41/?versions&max-keys=999 2024-02-01T10:38:02.610 DEBUG:boto:auth_path=/vclfcn-41/?versions&max-keys=999 2024-02-01T10:38:02.610 DEBUG:boto:Method: GET 2024-02-01T10:38:02.611 DEBUG:boto:Path: /vclfcn-41/?versions&max-keys=999 2024-02-01T10:38:02.611 DEBUG:boto:Data: 2024-02-01T10:38:02.611 DEBUG:boto:Headers: {} 2024-02-01T10:38:02.611 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:02.611 DEBUG:boto:Port: 8000 2024-02-01T10:38:02.611 DEBUG:boto:Params: {} 2024-02-01T10:38:02.611 DEBUG:boto:Token: None 2024-02-01T10:38:02.612 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:02 GMT /vclfcn-41/?versions 2024-02-01T10:38:02.612 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:P+pwBlP9jbghJJMkDbb9VcBmkqE= 2024-02-01T10:38:02.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:P+pwBlP9jbghJJMkDbb9VcBmkqE=', 'Content-Length': '0'} 2024-02-01T10:38:02.617 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008705d7f08532d30f-0065bb748a-5004-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:02.617 DEBUG:boto:b'vclfcn-41999false' 2024-02-01T10:38:02.617 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:38:02.617 DEBUG:boto:path=/vclfcn-41/ 2024-02-01T10:38:02.618 DEBUG:boto:auth_path=/vclfcn-41/ 2024-02-01T10:38:02.618 DEBUG:boto:path=/vclfcn-41/?versions&max-keys=999 2024-02-01T10:38:02.618 DEBUG:boto:auth_path=/vclfcn-41/?versions&max-keys=999 2024-02-01T10:38:02.618 DEBUG:boto:Method: GET 2024-02-01T10:38:02.618 DEBUG:boto:Path: /vclfcn-41/?versions&max-keys=999 2024-02-01T10:38:02.618 DEBUG:boto:Data: 2024-02-01T10:38:02.618 DEBUG:boto:Headers: {} 2024-02-01T10:38:02.618 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:02.618 DEBUG:boto:Port: 8001 2024-02-01T10:38:02.618 DEBUG:boto:Params: {} 2024-02-01T10:38:02.618 DEBUG:boto:Token: None 2024-02-01T10:38:02.619 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:02 GMT /vclfcn-41/?versions 2024-02-01T10:38:02.619 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:P+pwBlP9jbghJJMkDbb9VcBmkqE= 2024-02-01T10:38:02.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:P+pwBlP9jbghJJMkDbb9VcBmkqE=', 'Content-Length': '0'} 2024-02-01T10:38:02.623 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000007e9d2e3f081ebac-0065bb748a-5324-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:02.664 DEBUG:boto:b'vclfcn-41999false' 2024-02-01T10:38:02.664 DEBUG:boto:path=/vclfcn-41/ 2024-02-01T10:38:02.665 DEBUG:boto:auth_path=/vclfcn-41/ 2024-02-01T10:38:02.665 DEBUG:boto:path=/vclfcn-41/?versions&max-keys=999 2024-02-01T10:38:02.665 DEBUG:boto:auth_path=/vclfcn-41/?versions&max-keys=999 2024-02-01T10:38:02.665 DEBUG:boto:Method: GET 2024-02-01T10:38:02.665 DEBUG:boto:Path: /vclfcn-41/?versions&max-keys=999 2024-02-01T10:38:02.665 DEBUG:boto:Data: 2024-02-01T10:38:02.665 DEBUG:boto:Headers: {} 2024-02-01T10:38:02.665 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:02.665 DEBUG:boto:Port: 8000 2024-02-01T10:38:02.665 DEBUG:boto:Params: {} 2024-02-01T10:38:02.666 DEBUG:boto:Token: None 2024-02-01T10:38:02.666 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:02 GMT /vclfcn-41/?versions 2024-02-01T10:38:02.666 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:P+pwBlP9jbghJJMkDbb9VcBmkqE= 2024-02-01T10:38:02.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:P+pwBlP9jbghJJMkDbb9VcBmkqE=', 'Content-Length': '0'} 2024-02-01T10:38:02.671 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002eab079b8255789d-0065bb748a-5004-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:02.671 DEBUG:boto:b'vclfcn-41999false' 2024-02-01T10:38:02.671 DEBUG:boto:path=/vclfcn-41/ 2024-02-01T10:38:02.671 DEBUG:boto:auth_path=/vclfcn-41/ 2024-02-01T10:38:02.671 DEBUG:boto:path=/vclfcn-41/?versions&max-keys=999 2024-02-01T10:38:02.671 DEBUG:boto:auth_path=/vclfcn-41/?versions&max-keys=999 2024-02-01T10:38:02.672 DEBUG:boto:Method: GET 2024-02-01T10:38:02.672 DEBUG:boto:Path: /vclfcn-41/?versions&max-keys=999 2024-02-01T10:38:02.672 DEBUG:boto:Data: 2024-02-01T10:38:02.672 DEBUG:boto:Headers: {} 2024-02-01T10:38:02.672 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:02.672 DEBUG:boto:Port: 8001 2024-02-01T10:38:02.672 DEBUG:boto:Params: {} 2024-02-01T10:38:02.672 DEBUG:boto:Token: None 2024-02-01T10:38:02.672 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:02 GMT /vclfcn-41/?versions 2024-02-01T10:38:02.672 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:P+pwBlP9jbghJJMkDbb9VcBmkqE= 2024-02-01T10:38:02.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:P+pwBlP9jbghJJMkDbb9VcBmkqE=', 'Content-Length': '0'} 2024-02-01T10:38:02.677 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7ba902293b321b6-0065bb748a-5324-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:02.677 DEBUG:boto:b'vclfcn-41999false' 2024-02-01T10:38:02.678 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-41 zones={test-zone2, test-zone3} 2024-02-01T10:38:02.678 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-42 zones={test-zone2, test-zone1} 2024-02-01T10:38:02.678 DEBUG:boto:path=/vclfcn-42/ 2024-02-01T10:38:02.678 DEBUG:boto:auth_path=/vclfcn-42/ 2024-02-01T10:38:02.678 DEBUG:boto:Method: HEAD 2024-02-01T10:38:02.678 DEBUG:boto:Path: /vclfcn-42/ 2024-02-01T10:38:02.678 DEBUG:boto:Data: 2024-02-01T10:38:02.678 DEBUG:boto:Headers: {} 2024-02-01T10:38:02.679 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:02.679 DEBUG:boto:Port: 8000 2024-02-01T10:38:02.679 DEBUG:boto:Params: {} 2024-02-01T10:38:02.679 DEBUG:boto:Token: None 2024-02-01T10:38:02.679 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:02 GMT /vclfcn-42/ 2024-02-01T10:38:02.679 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WFVYtZwQLeTky+knwVtuwGaKWI8= 2024-02-01T10:38:02.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WFVYtZwQLeTky+knwVtuwGaKWI8=', 'Content-Length': '0'} 2024-02-01T10:38:02.683 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ef8c5e1712bcbf7f-0065bb748a-5004-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:02.683 DEBUG:boto:path=/vclfcn-42/ 2024-02-01T10:38:02.683 DEBUG:boto:auth_path=/vclfcn-42/ 2024-02-01T10:38:02.684 DEBUG:boto:Method: HEAD 2024-02-01T10:38:02.684 DEBUG:boto:Path: /vclfcn-42/ 2024-02-01T10:38:02.684 DEBUG:boto:Data: 2024-02-01T10:38:02.684 DEBUG:boto:Headers: {} 2024-02-01T10:38:02.684 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:02.684 DEBUG:boto:Port: 8000 2024-02-01T10:38:02.684 DEBUG:boto:Params: {} 2024-02-01T10:38:02.684 DEBUG:boto:Token: None 2024-02-01T10:38:02.684 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:02 GMT /vclfcn-42/ 2024-02-01T10:38:02.685 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WFVYtZwQLeTky+knwVtuwGaKWI8= 2024-02-01T10:38:02.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WFVYtZwQLeTky+knwVtuwGaKWI8=', 'Content-Length': '0'} 2024-02-01T10:38:02.689 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007d82995d1c3cf683-0065bb748a-5317-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:02.689 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:38:02.690 DEBUG:boto:path=/vclfcn-42/ 2024-02-01T10:38:02.690 DEBUG:boto:auth_path=/vclfcn-42/ 2024-02-01T10:38:02.690 DEBUG:boto:path=/vclfcn-42/?versions&max-keys=999 2024-02-01T10:38:02.690 DEBUG:boto:auth_path=/vclfcn-42/?versions&max-keys=999 2024-02-01T10:38:02.690 DEBUG:boto:Method: GET 2024-02-01T10:38:02.690 DEBUG:boto:Path: /vclfcn-42/?versions&max-keys=999 2024-02-01T10:38:02.690 DEBUG:boto:Data: 2024-02-01T10:38:02.690 DEBUG:boto:Headers: {} 2024-02-01T10:38:02.690 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:02.690 DEBUG:boto:Port: 8000 2024-02-01T10:38:02.691 DEBUG:boto:Params: {} 2024-02-01T10:38:02.691 DEBUG:boto:Token: None 2024-02-01T10:38:02.691 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:02 GMT /vclfcn-42/?versions 2024-02-01T10:38:02.691 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sGbZ7J6PR0MaMfv0L646ZUCeIZQ= 2024-02-01T10:38:02.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sGbZ7J6PR0MaMfv0L646ZUCeIZQ=', 'Content-Length': '0'} 2024-02-01T10:38:02.695 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050c897efffbd14d5-0065bb748a-5004-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:02.736 DEBUG:boto:b'vclfcn-42999false' 2024-02-01T10:38:02.736 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:38:02.737 DEBUG:boto:path=/vclfcn-42/ 2024-02-01T10:38:02.737 DEBUG:boto:auth_path=/vclfcn-42/ 2024-02-01T10:38:02.737 DEBUG:boto:path=/vclfcn-42/?versions&max-keys=999 2024-02-01T10:38:02.737 DEBUG:boto:auth_path=/vclfcn-42/?versions&max-keys=999 2024-02-01T10:38:02.737 DEBUG:boto:Method: GET 2024-02-01T10:38:02.737 DEBUG:boto:Path: /vclfcn-42/?versions&max-keys=999 2024-02-01T10:38:02.737 DEBUG:boto:Data: 2024-02-01T10:38:02.737 DEBUG:boto:Headers: {} 2024-02-01T10:38:02.737 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:02.737 DEBUG:boto:Port: 8000 2024-02-01T10:38:02.738 DEBUG:boto:Params: {} 2024-02-01T10:38:02.738 DEBUG:boto:Token: None 2024-02-01T10:38:02.738 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:02 GMT /vclfcn-42/?versions 2024-02-01T10:38:02.738 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sGbZ7J6PR0MaMfv0L646ZUCeIZQ= 2024-02-01T10:38:02.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sGbZ7J6PR0MaMfv0L646ZUCeIZQ=', 'Content-Length': '0'} 2024-02-01T10:38:02.743 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000930c30e0be8ba66a-0065bb748a-5317-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:02.743 DEBUG:boto:b'vclfcn-42999false' 2024-02-01T10:38:02.743 DEBUG:boto:path=/vclfcn-42/ 2024-02-01T10:38:02.743 DEBUG:boto:auth_path=/vclfcn-42/ 2024-02-01T10:38:02.743 DEBUG:boto:path=/vclfcn-42/?versions&max-keys=999 2024-02-01T10:38:02.743 DEBUG:boto:auth_path=/vclfcn-42/?versions&max-keys=999 2024-02-01T10:38:02.744 DEBUG:boto:Method: GET 2024-02-01T10:38:02.744 DEBUG:boto:Path: /vclfcn-42/?versions&max-keys=999 2024-02-01T10:38:02.744 DEBUG:boto:Data: 2024-02-01T10:38:02.744 DEBUG:boto:Headers: {} 2024-02-01T10:38:02.744 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:02.744 DEBUG:boto:Port: 8000 2024-02-01T10:38:02.744 DEBUG:boto:Params: {} 2024-02-01T10:38:02.744 DEBUG:boto:Token: None 2024-02-01T10:38:02.744 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:02 GMT /vclfcn-42/?versions 2024-02-01T10:38:02.744 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sGbZ7J6PR0MaMfv0L646ZUCeIZQ= 2024-02-01T10:38:02.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sGbZ7J6PR0MaMfv0L646ZUCeIZQ=', 'Content-Length': '0'} 2024-02-01T10:38:02.748 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076dae4895e3a2e67-0065bb748a-5004-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:02.749 DEBUG:boto:b'vclfcn-42999false' 2024-02-01T10:38:02.749 DEBUG:boto:path=/vclfcn-42/ 2024-02-01T10:38:02.749 DEBUG:boto:auth_path=/vclfcn-42/ 2024-02-01T10:38:02.749 DEBUG:boto:path=/vclfcn-42/?versions&max-keys=999 2024-02-01T10:38:02.750 DEBUG:boto:auth_path=/vclfcn-42/?versions&max-keys=999 2024-02-01T10:38:02.750 DEBUG:boto:Method: GET 2024-02-01T10:38:02.750 DEBUG:boto:Path: /vclfcn-42/?versions&max-keys=999 2024-02-01T10:38:02.750 DEBUG:boto:Data: 2024-02-01T10:38:02.750 DEBUG:boto:Headers: {} 2024-02-01T10:38:02.750 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:02.750 DEBUG:boto:Port: 8000 2024-02-01T10:38:02.750 DEBUG:boto:Params: {} 2024-02-01T10:38:02.750 DEBUG:boto:Token: None 2024-02-01T10:38:02.750 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:02 GMT /vclfcn-42/?versions 2024-02-01T10:38:02.751 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sGbZ7J6PR0MaMfv0L646ZUCeIZQ= 2024-02-01T10:38:02.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sGbZ7J6PR0MaMfv0L646ZUCeIZQ=', 'Content-Length': '0'} 2024-02-01T10:38:02.755 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a882e961b3640ef-0065bb748a-5317-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:02.796 DEBUG:boto:b'vclfcn-42999false' 2024-02-01T10:38:02.796 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-42 zones={test-zone2, test-zone1} 2024-02-01T10:38:02.797 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-42 zones={test-zone3, test-zone1} 2024-02-01T10:38:02.797 DEBUG:boto:path=/vclfcn-42/ 2024-02-01T10:38:02.797 DEBUG:boto:auth_path=/vclfcn-42/ 2024-02-01T10:38:02.797 DEBUG:boto:Method: HEAD 2024-02-01T10:38:02.797 DEBUG:boto:Path: /vclfcn-42/ 2024-02-01T10:38:02.797 DEBUG:boto:Data: 2024-02-01T10:38:02.797 DEBUG:boto:Headers: {} 2024-02-01T10:38:02.797 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:02.797 DEBUG:boto:Port: 8001 2024-02-01T10:38:02.798 DEBUG:boto:Params: {} 2024-02-01T10:38:02.798 DEBUG:boto:Token: None 2024-02-01T10:38:02.798 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:02 GMT /vclfcn-42/ 2024-02-01T10:38:02.798 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WFVYtZwQLeTky+knwVtuwGaKWI8= 2024-02-01T10:38:02.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WFVYtZwQLeTky+knwVtuwGaKWI8=', 'Content-Length': '0'} 2024-02-01T10:38:02.802 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000038ab00a862c527a1-0065bb748a-5324-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:02.802 DEBUG:boto:path=/vclfcn-42/ 2024-02-01T10:38:02.802 DEBUG:boto:auth_path=/vclfcn-42/ 2024-02-01T10:38:02.802 DEBUG:boto:Method: HEAD 2024-02-01T10:38:02.802 DEBUG:boto:Path: /vclfcn-42/ 2024-02-01T10:38:02.803 DEBUG:boto:Data: 2024-02-01T10:38:02.803 DEBUG:boto:Headers: {} 2024-02-01T10:38:02.803 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:02.803 DEBUG:boto:Port: 8000 2024-02-01T10:38:02.803 DEBUG:boto:Params: {} 2024-02-01T10:38:02.803 DEBUG:boto:Token: None 2024-02-01T10:38:02.803 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:02 GMT /vclfcn-42/ 2024-02-01T10:38:02.803 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WFVYtZwQLeTky+knwVtuwGaKWI8= 2024-02-01T10:38:02.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WFVYtZwQLeTky+knwVtuwGaKWI8=', 'Content-Length': '0'} 2024-02-01T10:38:02.807 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000313a054d6b7ce6b1-0065bb748a-5317-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:02.807 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:38:02.807 DEBUG:boto:path=/vclfcn-42/ 2024-02-01T10:38:02.808 DEBUG:boto:auth_path=/vclfcn-42/ 2024-02-01T10:38:02.808 DEBUG:boto:path=/vclfcn-42/?versions&max-keys=999 2024-02-01T10:38:02.808 DEBUG:boto:auth_path=/vclfcn-42/?versions&max-keys=999 2024-02-01T10:38:02.808 DEBUG:boto:Method: GET 2024-02-01T10:38:02.808 DEBUG:boto:Path: /vclfcn-42/?versions&max-keys=999 2024-02-01T10:38:02.808 DEBUG:boto:Data: 2024-02-01T10:38:02.808 DEBUG:boto:Headers: {} 2024-02-01T10:38:02.808 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:02.808 DEBUG:boto:Port: 8001 2024-02-01T10:38:02.808 DEBUG:boto:Params: {} 2024-02-01T10:38:02.808 DEBUG:boto:Token: None 2024-02-01T10:38:02.809 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:02 GMT /vclfcn-42/?versions 2024-02-01T10:38:02.809 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sGbZ7J6PR0MaMfv0L646ZUCeIZQ= 2024-02-01T10:38:02.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sGbZ7J6PR0MaMfv0L646ZUCeIZQ=', 'Content-Length': '0'} 2024-02-01T10:38:02.813 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013f6558dd2fcbb2c-0065bb748a-5324-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:02.813 DEBUG:boto:b'vclfcn-42999false' 2024-02-01T10:38:02.813 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:38:02.814 DEBUG:boto:path=/vclfcn-42/ 2024-02-01T10:38:02.814 DEBUG:boto:auth_path=/vclfcn-42/ 2024-02-01T10:38:02.814 DEBUG:boto:path=/vclfcn-42/?versions&max-keys=999 2024-02-01T10:38:02.814 DEBUG:boto:auth_path=/vclfcn-42/?versions&max-keys=999 2024-02-01T10:38:02.814 DEBUG:boto:Method: GET 2024-02-01T10:38:02.814 DEBUG:boto:Path: /vclfcn-42/?versions&max-keys=999 2024-02-01T10:38:02.814 DEBUG:boto:Data: 2024-02-01T10:38:02.814 DEBUG:boto:Headers: {} 2024-02-01T10:38:02.814 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:02.814 DEBUG:boto:Port: 8000 2024-02-01T10:38:02.815 DEBUG:boto:Params: {} 2024-02-01T10:38:02.815 DEBUG:boto:Token: None 2024-02-01T10:38:02.815 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:02 GMT /vclfcn-42/?versions 2024-02-01T10:38:02.815 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sGbZ7J6PR0MaMfv0L646ZUCeIZQ= 2024-02-01T10:38:02.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sGbZ7J6PR0MaMfv0L646ZUCeIZQ=', 'Content-Length': '0'} 2024-02-01T10:38:02.819 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb9e44d6b8fcb8c9-0065bb748a-5317-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:02.819 DEBUG:boto:b'vclfcn-42999false' 2024-02-01T10:38:02.820 DEBUG:boto:path=/vclfcn-42/ 2024-02-01T10:38:02.820 DEBUG:boto:auth_path=/vclfcn-42/ 2024-02-01T10:38:02.820 DEBUG:boto:path=/vclfcn-42/?versions&max-keys=999 2024-02-01T10:38:02.821 DEBUG:boto:auth_path=/vclfcn-42/?versions&max-keys=999 2024-02-01T10:38:02.821 DEBUG:boto:Method: GET 2024-02-01T10:38:02.821 DEBUG:boto:Path: /vclfcn-42/?versions&max-keys=999 2024-02-01T10:38:02.821 DEBUG:boto:Data: 2024-02-01T10:38:02.821 DEBUG:boto:Headers: {} 2024-02-01T10:38:02.821 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:02.821 DEBUG:boto:Port: 8001 2024-02-01T10:38:02.821 DEBUG:boto:Params: {} 2024-02-01T10:38:02.822 DEBUG:boto:Token: None 2024-02-01T10:38:02.822 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:02 GMT /vclfcn-42/?versions 2024-02-01T10:38:02.822 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sGbZ7J6PR0MaMfv0L646ZUCeIZQ= 2024-02-01T10:38:02.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sGbZ7J6PR0MaMfv0L646ZUCeIZQ=', 'Content-Length': '0'} 2024-02-01T10:38:02.827 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002aa7eee843cc761c-0065bb748a-5324-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:02.868 DEBUG:boto:b'vclfcn-42999false' 2024-02-01T10:38:02.868 DEBUG:boto:path=/vclfcn-42/ 2024-02-01T10:38:02.869 DEBUG:boto:auth_path=/vclfcn-42/ 2024-02-01T10:38:02.869 DEBUG:boto:path=/vclfcn-42/?versions&max-keys=999 2024-02-01T10:38:02.869 DEBUG:boto:auth_path=/vclfcn-42/?versions&max-keys=999 2024-02-01T10:38:02.869 DEBUG:boto:Method: GET 2024-02-01T10:38:02.869 DEBUG:boto:Path: /vclfcn-42/?versions&max-keys=999 2024-02-01T10:38:02.869 DEBUG:boto:Data: 2024-02-01T10:38:02.869 DEBUG:boto:Headers: {} 2024-02-01T10:38:02.869 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:02.869 DEBUG:boto:Port: 8000 2024-02-01T10:38:02.869 DEBUG:boto:Params: {} 2024-02-01T10:38:02.870 DEBUG:boto:Token: None 2024-02-01T10:38:02.870 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:02 GMT /vclfcn-42/?versions 2024-02-01T10:38:02.870 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sGbZ7J6PR0MaMfv0L646ZUCeIZQ= 2024-02-01T10:38:02.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sGbZ7J6PR0MaMfv0L646ZUCeIZQ=', 'Content-Length': '0'} 2024-02-01T10:38:02.874 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000902c0d07b768fa17-0065bb748a-5317-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:02.874 DEBUG:boto:b'vclfcn-42999false' 2024-02-01T10:38:02.874 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-42 zones={test-zone3, test-zone1} 2024-02-01T10:38:02.875 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-42 zones={test-zone1, test-zone2} 2024-02-01T10:38:02.875 DEBUG:boto:path=/vclfcn-42/ 2024-02-01T10:38:02.875 DEBUG:boto:auth_path=/vclfcn-42/ 2024-02-01T10:38:02.875 DEBUG:boto:Method: HEAD 2024-02-01T10:38:02.875 DEBUG:boto:Path: /vclfcn-42/ 2024-02-01T10:38:02.875 DEBUG:boto:Data: 2024-02-01T10:38:02.875 DEBUG:boto:Headers: {} 2024-02-01T10:38:02.875 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:02.875 DEBUG:boto:Port: 8000 2024-02-01T10:38:02.875 DEBUG:boto:Params: {} 2024-02-01T10:38:02.876 DEBUG:boto:Token: None 2024-02-01T10:38:02.876 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:02 GMT /vclfcn-42/ 2024-02-01T10:38:02.876 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WFVYtZwQLeTky+knwVtuwGaKWI8= 2024-02-01T10:38:02.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WFVYtZwQLeTky+knwVtuwGaKWI8=', 'Content-Length': '0'} 2024-02-01T10:38:02.880 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000075f2f37aead413c8-0065bb748a-5317-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:02.880 DEBUG:boto:path=/vclfcn-42/ 2024-02-01T10:38:02.880 DEBUG:boto:auth_path=/vclfcn-42/ 2024-02-01T10:38:02.881 DEBUG:boto:Method: HEAD 2024-02-01T10:38:02.881 DEBUG:boto:Path: /vclfcn-42/ 2024-02-01T10:38:02.881 DEBUG:boto:Data: 2024-02-01T10:38:02.881 DEBUG:boto:Headers: {} 2024-02-01T10:38:02.881 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:02.881 DEBUG:boto:Port: 8000 2024-02-01T10:38:02.881 DEBUG:boto:Params: {} 2024-02-01T10:38:02.881 DEBUG:boto:Token: None 2024-02-01T10:38:02.881 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:02 GMT /vclfcn-42/ 2024-02-01T10:38:02.882 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WFVYtZwQLeTky+knwVtuwGaKWI8= 2024-02-01T10:38:02.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WFVYtZwQLeTky+knwVtuwGaKWI8=', 'Content-Length': '0'} 2024-02-01T10:38:02.885 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000030e35aabf91eef29-0065bb748a-5004-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:02.886 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:38:02.886 DEBUG:boto:path=/vclfcn-42/ 2024-02-01T10:38:02.886 DEBUG:boto:auth_path=/vclfcn-42/ 2024-02-01T10:38:02.886 DEBUG:boto:path=/vclfcn-42/?versions&max-keys=999 2024-02-01T10:38:02.886 DEBUG:boto:auth_path=/vclfcn-42/?versions&max-keys=999 2024-02-01T10:38:02.886 DEBUG:boto:Method: GET 2024-02-01T10:38:02.886 DEBUG:boto:Path: /vclfcn-42/?versions&max-keys=999 2024-02-01T10:38:02.886 DEBUG:boto:Data: 2024-02-01T10:38:02.886 DEBUG:boto:Headers: {} 2024-02-01T10:38:02.886 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:02.887 DEBUG:boto:Port: 8000 2024-02-01T10:38:02.887 DEBUG:boto:Params: {} 2024-02-01T10:38:02.887 DEBUG:boto:Token: None 2024-02-01T10:38:02.887 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:02 GMT /vclfcn-42/?versions 2024-02-01T10:38:02.887 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sGbZ7J6PR0MaMfv0L646ZUCeIZQ= 2024-02-01T10:38:02.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sGbZ7J6PR0MaMfv0L646ZUCeIZQ=', 'Content-Length': '0'} 2024-02-01T10:38:02.892 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000034441684733508b5-0065bb748a-5317-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:02.932 DEBUG:boto:b'vclfcn-42999false' 2024-02-01T10:38:02.932 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:38:02.933 DEBUG:boto:path=/vclfcn-42/ 2024-02-01T10:38:02.933 DEBUG:boto:auth_path=/vclfcn-42/ 2024-02-01T10:38:02.933 DEBUG:boto:path=/vclfcn-42/?versions&max-keys=999 2024-02-01T10:38:02.933 DEBUG:boto:auth_path=/vclfcn-42/?versions&max-keys=999 2024-02-01T10:38:02.933 DEBUG:boto:Method: GET 2024-02-01T10:38:02.933 DEBUG:boto:Path: /vclfcn-42/?versions&max-keys=999 2024-02-01T10:38:02.933 DEBUG:boto:Data: 2024-02-01T10:38:02.933 DEBUG:boto:Headers: {} 2024-02-01T10:38:02.933 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:02.933 DEBUG:boto:Port: 8000 2024-02-01T10:38:02.933 DEBUG:boto:Params: {} 2024-02-01T10:38:02.934 DEBUG:boto:Token: None 2024-02-01T10:38:02.934 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:02 GMT /vclfcn-42/?versions 2024-02-01T10:38:02.934 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sGbZ7J6PR0MaMfv0L646ZUCeIZQ= 2024-02-01T10:38:02.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sGbZ7J6PR0MaMfv0L646ZUCeIZQ=', 'Content-Length': '0'} 2024-02-01T10:38:02.938 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d4204b83d812d16-0065bb748a-5004-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:02.939 DEBUG:boto:b'vclfcn-42999false' 2024-02-01T10:38:02.939 DEBUG:boto:path=/vclfcn-42/ 2024-02-01T10:38:02.939 DEBUG:boto:auth_path=/vclfcn-42/ 2024-02-01T10:38:02.939 DEBUG:boto:path=/vclfcn-42/?versions&max-keys=999 2024-02-01T10:38:02.939 DEBUG:boto:auth_path=/vclfcn-42/?versions&max-keys=999 2024-02-01T10:38:02.939 DEBUG:boto:Method: GET 2024-02-01T10:38:02.939 DEBUG:boto:Path: /vclfcn-42/?versions&max-keys=999 2024-02-01T10:38:02.940 DEBUG:boto:Data: 2024-02-01T10:38:02.940 DEBUG:boto:Headers: {} 2024-02-01T10:38:02.940 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:02.940 DEBUG:boto:Port: 8000 2024-02-01T10:38:02.940 DEBUG:boto:Params: {} 2024-02-01T10:38:02.940 DEBUG:boto:Token: None 2024-02-01T10:38:02.940 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:02 GMT /vclfcn-42/?versions 2024-02-01T10:38:02.940 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sGbZ7J6PR0MaMfv0L646ZUCeIZQ= 2024-02-01T10:38:02.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sGbZ7J6PR0MaMfv0L646ZUCeIZQ=', 'Content-Length': '0'} 2024-02-01T10:38:02.944 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f40910152ddd3185-0065bb748a-5317-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:02.944 DEBUG:boto:b'vclfcn-42999false' 2024-02-01T10:38:02.945 DEBUG:boto:path=/vclfcn-42/ 2024-02-01T10:38:02.945 DEBUG:boto:auth_path=/vclfcn-42/ 2024-02-01T10:38:02.945 DEBUG:boto:path=/vclfcn-42/?versions&max-keys=999 2024-02-01T10:38:02.945 DEBUG:boto:auth_path=/vclfcn-42/?versions&max-keys=999 2024-02-01T10:38:02.945 DEBUG:boto:Method: GET 2024-02-01T10:38:02.945 DEBUG:boto:Path: /vclfcn-42/?versions&max-keys=999 2024-02-01T10:38:02.945 DEBUG:boto:Data: 2024-02-01T10:38:02.945 DEBUG:boto:Headers: {} 2024-02-01T10:38:02.946 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:02.946 DEBUG:boto:Port: 8000 2024-02-01T10:38:02.946 DEBUG:boto:Params: {} 2024-02-01T10:38:02.946 DEBUG:boto:Token: None 2024-02-01T10:38:02.947 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:02 GMT /vclfcn-42/?versions 2024-02-01T10:38:02.947 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sGbZ7J6PR0MaMfv0L646ZUCeIZQ= 2024-02-01T10:38:02.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sGbZ7J6PR0MaMfv0L646ZUCeIZQ=', 'Content-Length': '0'} 2024-02-01T10:38:02.951 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000079be78ad28d16fc6-0065bb748a-5004-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:02.951 DEBUG:boto:b'vclfcn-42999false' 2024-02-01T10:38:02.951 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-42 zones={test-zone1, test-zone2} 2024-02-01T10:38:02.952 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-42 zones={test-zone3, test-zone2} 2024-02-01T10:38:02.952 DEBUG:boto:path=/vclfcn-42/ 2024-02-01T10:38:02.952 DEBUG:boto:auth_path=/vclfcn-42/ 2024-02-01T10:38:02.952 DEBUG:boto:Method: HEAD 2024-02-01T10:38:02.952 DEBUG:boto:Path: /vclfcn-42/ 2024-02-01T10:38:02.952 DEBUG:boto:Data: 2024-02-01T10:38:02.952 DEBUG:boto:Headers: {} 2024-02-01T10:38:02.952 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:02.952 DEBUG:boto:Port: 8001 2024-02-01T10:38:02.952 DEBUG:boto:Params: {} 2024-02-01T10:38:02.953 DEBUG:boto:Token: None 2024-02-01T10:38:02.953 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:02 GMT /vclfcn-42/ 2024-02-01T10:38:02.953 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WFVYtZwQLeTky+knwVtuwGaKWI8= 2024-02-01T10:38:02.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WFVYtZwQLeTky+knwVtuwGaKWI8=', 'Content-Length': '0'} 2024-02-01T10:38:02.957 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005c75bcea79b600b8-0065bb748a-5324-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:02.958 DEBUG:boto:path=/vclfcn-42/ 2024-02-01T10:38:02.958 DEBUG:boto:auth_path=/vclfcn-42/ 2024-02-01T10:38:02.958 DEBUG:boto:Method: HEAD 2024-02-01T10:38:02.958 DEBUG:boto:Path: /vclfcn-42/ 2024-02-01T10:38:02.958 DEBUG:boto:Data: 2024-02-01T10:38:02.958 DEBUG:boto:Headers: {} 2024-02-01T10:38:02.958 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:02.959 DEBUG:boto:Port: 8000 2024-02-01T10:38:02.959 DEBUG:boto:Params: {} 2024-02-01T10:38:02.959 DEBUG:boto:Token: None 2024-02-01T10:38:02.959 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:02 GMT /vclfcn-42/ 2024-02-01T10:38:02.959 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WFVYtZwQLeTky+knwVtuwGaKWI8= 2024-02-01T10:38:02.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WFVYtZwQLeTky+knwVtuwGaKWI8=', 'Content-Length': '0'} 2024-02-01T10:38:02.963 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b00c2d3f9eaf549f-0065bb748a-5004-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:02.963 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:38:02.963 DEBUG:boto:path=/vclfcn-42/ 2024-02-01T10:38:02.963 DEBUG:boto:auth_path=/vclfcn-42/ 2024-02-01T10:38:02.963 DEBUG:boto:path=/vclfcn-42/?versions&max-keys=999 2024-02-01T10:38:02.964 DEBUG:boto:auth_path=/vclfcn-42/?versions&max-keys=999 2024-02-01T10:38:02.964 DEBUG:boto:Method: GET 2024-02-01T10:38:02.964 DEBUG:boto:Path: /vclfcn-42/?versions&max-keys=999 2024-02-01T10:38:02.964 DEBUG:boto:Data: 2024-02-01T10:38:02.964 DEBUG:boto:Headers: {} 2024-02-01T10:38:02.964 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:02.964 DEBUG:boto:Port: 8001 2024-02-01T10:38:02.964 DEBUG:boto:Params: {} 2024-02-01T10:38:02.964 DEBUG:boto:Token: None 2024-02-01T10:38:02.964 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:02 GMT /vclfcn-42/?versions 2024-02-01T10:38:02.965 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sGbZ7J6PR0MaMfv0L646ZUCeIZQ= 2024-02-01T10:38:02.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sGbZ7J6PR0MaMfv0L646ZUCeIZQ=', 'Content-Length': '0'} 2024-02-01T10:38:02.969 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004207187db7969719-0065bb748a-5324-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:02.969 DEBUG:boto:b'vclfcn-42999false' 2024-02-01T10:38:02.969 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:38:02.969 DEBUG:boto:path=/vclfcn-42/ 2024-02-01T10:38:02.970 DEBUG:boto:auth_path=/vclfcn-42/ 2024-02-01T10:38:02.970 DEBUG:boto:path=/vclfcn-42/?versions&max-keys=999 2024-02-01T10:38:02.970 DEBUG:boto:auth_path=/vclfcn-42/?versions&max-keys=999 2024-02-01T10:38:02.970 DEBUG:boto:Method: GET 2024-02-01T10:38:02.970 DEBUG:boto:Path: /vclfcn-42/?versions&max-keys=999 2024-02-01T10:38:02.970 DEBUG:boto:Data: 2024-02-01T10:38:02.970 DEBUG:boto:Headers: {} 2024-02-01T10:38:02.970 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:02.970 DEBUG:boto:Port: 8000 2024-02-01T10:38:02.970 DEBUG:boto:Params: {} 2024-02-01T10:38:02.970 DEBUG:boto:Token: None 2024-02-01T10:38:02.971 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:02 GMT /vclfcn-42/?versions 2024-02-01T10:38:02.971 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sGbZ7J6PR0MaMfv0L646ZUCeIZQ= 2024-02-01T10:38:02.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sGbZ7J6PR0MaMfv0L646ZUCeIZQ=', 'Content-Length': '0'} 2024-02-01T10:38:02.975 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c4eb39b14a8ec8c-0065bb748a-5004-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:03.016 DEBUG:boto:b'vclfcn-42999false' 2024-02-01T10:38:03.017 DEBUG:boto:path=/vclfcn-42/ 2024-02-01T10:38:03.017 DEBUG:boto:auth_path=/vclfcn-42/ 2024-02-01T10:38:03.017 DEBUG:boto:path=/vclfcn-42/?versions&max-keys=999 2024-02-01T10:38:03.017 DEBUG:boto:auth_path=/vclfcn-42/?versions&max-keys=999 2024-02-01T10:38:03.017 DEBUG:boto:Method: GET 2024-02-01T10:38:03.017 DEBUG:boto:Path: /vclfcn-42/?versions&max-keys=999 2024-02-01T10:38:03.017 DEBUG:boto:Data: 2024-02-01T10:38:03.017 DEBUG:boto:Headers: {} 2024-02-01T10:38:03.017 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:03.018 DEBUG:boto:Port: 8001 2024-02-01T10:38:03.018 DEBUG:boto:Params: {} 2024-02-01T10:38:03.018 DEBUG:boto:Token: None 2024-02-01T10:38:03.018 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:03 GMT /vclfcn-42/?versions 2024-02-01T10:38:03.018 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:6X+8QuBbLpvNDkPsHdboZc9yzqo= 2024-02-01T10:38:03.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:6X+8QuBbLpvNDkPsHdboZc9yzqo=', 'Content-Length': '0'} 2024-02-01T10:38:03.022 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae5964e9a47d6805-0065bb748b-5324-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:03.023 DEBUG:boto:b'vclfcn-42999false' 2024-02-01T10:38:03.023 DEBUG:boto:path=/vclfcn-42/ 2024-02-01T10:38:03.023 DEBUG:boto:auth_path=/vclfcn-42/ 2024-02-01T10:38:03.023 DEBUG:boto:path=/vclfcn-42/?versions&max-keys=999 2024-02-01T10:38:03.023 DEBUG:boto:auth_path=/vclfcn-42/?versions&max-keys=999 2024-02-01T10:38:03.023 DEBUG:boto:Method: GET 2024-02-01T10:38:03.023 DEBUG:boto:Path: /vclfcn-42/?versions&max-keys=999 2024-02-01T10:38:03.024 DEBUG:boto:Data: 2024-02-01T10:38:03.024 DEBUG:boto:Headers: {} 2024-02-01T10:38:03.024 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:03.024 DEBUG:boto:Port: 8000 2024-02-01T10:38:03.024 DEBUG:boto:Params: {} 2024-02-01T10:38:03.024 DEBUG:boto:Token: None 2024-02-01T10:38:03.024 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:03 GMT /vclfcn-42/?versions 2024-02-01T10:38:03.024 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:6X+8QuBbLpvNDkPsHdboZc9yzqo= 2024-02-01T10:38:03.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:6X+8QuBbLpvNDkPsHdboZc9yzqo=', 'Content-Length': '0'} 2024-02-01T10:38:03.028 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e3ed60b45b2d708-0065bb748b-5004-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:03.028 DEBUG:boto:b'vclfcn-42999false' 2024-02-01T10:38:03.029 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-42 zones={test-zone3, test-zone2} 2024-02-01T10:38:03.029 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-42 zones={test-zone1, test-zone3} 2024-02-01T10:38:03.029 DEBUG:boto:path=/vclfcn-42/ 2024-02-01T10:38:03.029 DEBUG:boto:auth_path=/vclfcn-42/ 2024-02-01T10:38:03.029 DEBUG:boto:Method: HEAD 2024-02-01T10:38:03.029 DEBUG:boto:Path: /vclfcn-42/ 2024-02-01T10:38:03.029 DEBUG:boto:Data: 2024-02-01T10:38:03.029 DEBUG:boto:Headers: {} 2024-02-01T10:38:03.029 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:03.030 DEBUG:boto:Port: 8000 2024-02-01T10:38:03.030 DEBUG:boto:Params: {} 2024-02-01T10:38:03.030 DEBUG:boto:Token: None 2024-02-01T10:38:03.030 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:03 GMT /vclfcn-42/ 2024-02-01T10:38:03.030 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7iX2t4JmpqBZFfeOgLl4EF//xpg= 2024-02-01T10:38:03.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7iX2t4JmpqBZFfeOgLl4EF//xpg=', 'Content-Length': '0'} 2024-02-01T10:38:03.034 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000056b5cb1a977ca4b2-0065bb748b-5317-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:03.034 DEBUG:boto:path=/vclfcn-42/ 2024-02-01T10:38:03.034 DEBUG:boto:auth_path=/vclfcn-42/ 2024-02-01T10:38:03.034 DEBUG:boto:Method: HEAD 2024-02-01T10:38:03.034 DEBUG:boto:Path: /vclfcn-42/ 2024-02-01T10:38:03.034 DEBUG:boto:Data: 2024-02-01T10:38:03.034 DEBUG:boto:Headers: {} 2024-02-01T10:38:03.034 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:03.035 DEBUG:boto:Port: 8001 2024-02-01T10:38:03.035 DEBUG:boto:Params: {} 2024-02-01T10:38:03.035 DEBUG:boto:Token: None 2024-02-01T10:38:03.035 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:03 GMT /vclfcn-42/ 2024-02-01T10:38:03.035 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7iX2t4JmpqBZFfeOgLl4EF//xpg= 2024-02-01T10:38:03.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7iX2t4JmpqBZFfeOgLl4EF//xpg=', 'Content-Length': '0'} 2024-02-01T10:38:03.039 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003b56e889201e5469-0065bb748b-5324-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:03.039 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:38:03.039 DEBUG:boto:path=/vclfcn-42/ 2024-02-01T10:38:03.039 DEBUG:boto:auth_path=/vclfcn-42/ 2024-02-01T10:38:03.039 DEBUG:boto:path=/vclfcn-42/?versions&max-keys=999 2024-02-01T10:38:03.040 DEBUG:boto:auth_path=/vclfcn-42/?versions&max-keys=999 2024-02-01T10:38:03.040 DEBUG:boto:Method: GET 2024-02-01T10:38:03.040 DEBUG:boto:Path: /vclfcn-42/?versions&max-keys=999 2024-02-01T10:38:03.040 DEBUG:boto:Data: 2024-02-01T10:38:03.040 DEBUG:boto:Headers: {} 2024-02-01T10:38:03.040 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:03.040 DEBUG:boto:Port: 8000 2024-02-01T10:38:03.040 DEBUG:boto:Params: {} 2024-02-01T10:38:03.040 DEBUG:boto:Token: None 2024-02-01T10:38:03.040 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:03 GMT /vclfcn-42/?versions 2024-02-01T10:38:03.041 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:6X+8QuBbLpvNDkPsHdboZc9yzqo= 2024-02-01T10:38:03.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:6X+8QuBbLpvNDkPsHdboZc9yzqo=', 'Content-Length': '0'} 2024-02-01T10:38:03.045 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000946f61b3e733bdd4-0065bb748b-5317-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:03.045 DEBUG:boto:b'vclfcn-42999false' 2024-02-01T10:38:03.045 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:38:03.045 DEBUG:boto:path=/vclfcn-42/ 2024-02-01T10:38:03.045 DEBUG:boto:auth_path=/vclfcn-42/ 2024-02-01T10:38:03.046 DEBUG:boto:path=/vclfcn-42/?versions&max-keys=999 2024-02-01T10:38:03.046 DEBUG:boto:auth_path=/vclfcn-42/?versions&max-keys=999 2024-02-01T10:38:03.046 DEBUG:boto:Method: GET 2024-02-01T10:38:03.046 DEBUG:boto:Path: /vclfcn-42/?versions&max-keys=999 2024-02-01T10:38:03.046 DEBUG:boto:Data: 2024-02-01T10:38:03.046 DEBUG:boto:Headers: {} 2024-02-01T10:38:03.046 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:03.046 DEBUG:boto:Port: 8001 2024-02-01T10:38:03.046 DEBUG:boto:Params: {} 2024-02-01T10:38:03.046 DEBUG:boto:Token: None 2024-02-01T10:38:03.047 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:03 GMT /vclfcn-42/?versions 2024-02-01T10:38:03.047 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:6X+8QuBbLpvNDkPsHdboZc9yzqo= 2024-02-01T10:38:03.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:6X+8QuBbLpvNDkPsHdboZc9yzqo=', 'Content-Length': '0'} 2024-02-01T10:38:03.051 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab2f74934461a845-0065bb748b-5324-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:03.092 DEBUG:boto:b'vclfcn-42999false' 2024-02-01T10:38:03.093 DEBUG:boto:path=/vclfcn-42/ 2024-02-01T10:38:03.093 DEBUG:boto:auth_path=/vclfcn-42/ 2024-02-01T10:38:03.093 DEBUG:boto:path=/vclfcn-42/?versions&max-keys=999 2024-02-01T10:38:03.093 DEBUG:boto:auth_path=/vclfcn-42/?versions&max-keys=999 2024-02-01T10:38:03.093 DEBUG:boto:Method: GET 2024-02-01T10:38:03.093 DEBUG:boto:Path: /vclfcn-42/?versions&max-keys=999 2024-02-01T10:38:03.093 DEBUG:boto:Data: 2024-02-01T10:38:03.093 DEBUG:boto:Headers: {} 2024-02-01T10:38:03.093 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:03.093 DEBUG:boto:Port: 8000 2024-02-01T10:38:03.093 DEBUG:boto:Params: {} 2024-02-01T10:38:03.094 DEBUG:boto:Token: None 2024-02-01T10:38:03.094 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:03 GMT /vclfcn-42/?versions 2024-02-01T10:38:03.094 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:6X+8QuBbLpvNDkPsHdboZc9yzqo= 2024-02-01T10:38:03.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:6X+8QuBbLpvNDkPsHdboZc9yzqo=', 'Content-Length': '0'} 2024-02-01T10:38:03.098 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000022ee634715b6aea6-0065bb748b-5317-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:03.098 DEBUG:boto:b'vclfcn-42999false' 2024-02-01T10:38:03.099 DEBUG:boto:path=/vclfcn-42/ 2024-02-01T10:38:03.099 DEBUG:boto:auth_path=/vclfcn-42/ 2024-02-01T10:38:03.099 DEBUG:boto:path=/vclfcn-42/?versions&max-keys=999 2024-02-01T10:38:03.099 DEBUG:boto:auth_path=/vclfcn-42/?versions&max-keys=999 2024-02-01T10:38:03.099 DEBUG:boto:Method: GET 2024-02-01T10:38:03.099 DEBUG:boto:Path: /vclfcn-42/?versions&max-keys=999 2024-02-01T10:38:03.099 DEBUG:boto:Data: 2024-02-01T10:38:03.099 DEBUG:boto:Headers: {} 2024-02-01T10:38:03.099 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:03.100 DEBUG:boto:Port: 8001 2024-02-01T10:38:03.100 DEBUG:boto:Params: {} 2024-02-01T10:38:03.100 DEBUG:boto:Token: None 2024-02-01T10:38:03.100 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:03 GMT /vclfcn-42/?versions 2024-02-01T10:38:03.100 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:6X+8QuBbLpvNDkPsHdboZc9yzqo= 2024-02-01T10:38:03.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:6X+8QuBbLpvNDkPsHdboZc9yzqo=', 'Content-Length': '0'} 2024-02-01T10:38:03.104 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007111aaa639700234-0065bb748b-5324-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:03.105 DEBUG:boto:b'vclfcn-42999false' 2024-02-01T10:38:03.105 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-42 zones={test-zone1, test-zone3} 2024-02-01T10:38:03.105 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-42 zones={test-zone2, test-zone3} 2024-02-01T10:38:03.105 DEBUG:boto:path=/vclfcn-42/ 2024-02-01T10:38:03.105 DEBUG:boto:auth_path=/vclfcn-42/ 2024-02-01T10:38:03.105 DEBUG:boto:Method: HEAD 2024-02-01T10:38:03.106 DEBUG:boto:Path: /vclfcn-42/ 2024-02-01T10:38:03.106 DEBUG:boto:Data: 2024-02-01T10:38:03.106 DEBUG:boto:Headers: {} 2024-02-01T10:38:03.106 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:03.106 DEBUG:boto:Port: 8000 2024-02-01T10:38:03.106 DEBUG:boto:Params: {} 2024-02-01T10:38:03.106 DEBUG:boto:Token: None 2024-02-01T10:38:03.106 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:03 GMT /vclfcn-42/ 2024-02-01T10:38:03.106 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7iX2t4JmpqBZFfeOgLl4EF//xpg= 2024-02-01T10:38:03.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7iX2t4JmpqBZFfeOgLl4EF//xpg=', 'Content-Length': '0'} 2024-02-01T10:38:03.110 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000023dda6f54a8249e7-0065bb748b-5004-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:03.110 DEBUG:boto:path=/vclfcn-42/ 2024-02-01T10:38:03.111 DEBUG:boto:auth_path=/vclfcn-42/ 2024-02-01T10:38:03.111 DEBUG:boto:Method: HEAD 2024-02-01T10:38:03.111 DEBUG:boto:Path: /vclfcn-42/ 2024-02-01T10:38:03.111 DEBUG:boto:Data: 2024-02-01T10:38:03.111 DEBUG:boto:Headers: {} 2024-02-01T10:38:03.111 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:03.112 DEBUG:boto:Port: 8001 2024-02-01T10:38:03.112 DEBUG:boto:Params: {} 2024-02-01T10:38:03.112 DEBUG:boto:Token: None 2024-02-01T10:38:03.112 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:03 GMT /vclfcn-42/ 2024-02-01T10:38:03.112 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7iX2t4JmpqBZFfeOgLl4EF//xpg= 2024-02-01T10:38:03.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7iX2t4JmpqBZFfeOgLl4EF//xpg=', 'Content-Length': '0'} 2024-02-01T10:38:03.116 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a13b167ad797e17a-0065bb748b-5324-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:03.116 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:38:03.116 DEBUG:boto:path=/vclfcn-42/ 2024-02-01T10:38:03.116 DEBUG:boto:auth_path=/vclfcn-42/ 2024-02-01T10:38:03.116 DEBUG:boto:path=/vclfcn-42/?versions&max-keys=999 2024-02-01T10:38:03.116 DEBUG:boto:auth_path=/vclfcn-42/?versions&max-keys=999 2024-02-01T10:38:03.117 DEBUG:boto:Method: GET 2024-02-01T10:38:03.117 DEBUG:boto:Path: /vclfcn-42/?versions&max-keys=999 2024-02-01T10:38:03.117 DEBUG:boto:Data: 2024-02-01T10:38:03.117 DEBUG:boto:Headers: {} 2024-02-01T10:38:03.117 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:03.117 DEBUG:boto:Port: 8000 2024-02-01T10:38:03.117 DEBUG:boto:Params: {} 2024-02-01T10:38:03.117 DEBUG:boto:Token: None 2024-02-01T10:38:03.117 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:03 GMT /vclfcn-42/?versions 2024-02-01T10:38:03.117 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:6X+8QuBbLpvNDkPsHdboZc9yzqo= 2024-02-01T10:38:03.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:6X+8QuBbLpvNDkPsHdboZc9yzqo=', 'Content-Length': '0'} 2024-02-01T10:38:03.121 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e44e34bf039cb887-0065bb748b-5004-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:03.122 DEBUG:boto:b'vclfcn-42999false' 2024-02-01T10:38:03.122 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:38:03.122 DEBUG:boto:path=/vclfcn-42/ 2024-02-01T10:38:03.122 DEBUG:boto:auth_path=/vclfcn-42/ 2024-02-01T10:38:03.122 DEBUG:boto:path=/vclfcn-42/?versions&max-keys=999 2024-02-01T10:38:03.122 DEBUG:boto:auth_path=/vclfcn-42/?versions&max-keys=999 2024-02-01T10:38:03.123 DEBUG:boto:Method: GET 2024-02-01T10:38:03.123 DEBUG:boto:Path: /vclfcn-42/?versions&max-keys=999 2024-02-01T10:38:03.123 DEBUG:boto:Data: 2024-02-01T10:38:03.123 DEBUG:boto:Headers: {} 2024-02-01T10:38:03.123 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:03.123 DEBUG:boto:Port: 8001 2024-02-01T10:38:03.123 DEBUG:boto:Params: {} 2024-02-01T10:38:03.124 DEBUG:boto:Token: None 2024-02-01T10:38:03.124 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:03 GMT /vclfcn-42/?versions 2024-02-01T10:38:03.124 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:6X+8QuBbLpvNDkPsHdboZc9yzqo= 2024-02-01T10:38:03.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:6X+8QuBbLpvNDkPsHdboZc9yzqo=', 'Content-Length': '0'} 2024-02-01T10:38:03.128 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd089694a98ce739-0065bb748b-5324-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:03.128 DEBUG:boto:b'vclfcn-42999false' 2024-02-01T10:38:03.129 DEBUG:boto:path=/vclfcn-42/ 2024-02-01T10:38:03.129 DEBUG:boto:auth_path=/vclfcn-42/ 2024-02-01T10:38:03.129 DEBUG:boto:path=/vclfcn-42/?versions&max-keys=999 2024-02-01T10:38:03.129 DEBUG:boto:auth_path=/vclfcn-42/?versions&max-keys=999 2024-02-01T10:38:03.129 DEBUG:boto:Method: GET 2024-02-01T10:38:03.129 DEBUG:boto:Path: /vclfcn-42/?versions&max-keys=999 2024-02-01T10:38:03.129 DEBUG:boto:Data: 2024-02-01T10:38:03.129 DEBUG:boto:Headers: {} 2024-02-01T10:38:03.129 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:03.129 DEBUG:boto:Port: 8000 2024-02-01T10:38:03.129 DEBUG:boto:Params: {} 2024-02-01T10:38:03.130 DEBUG:boto:Token: None 2024-02-01T10:38:03.130 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:03 GMT /vclfcn-42/?versions 2024-02-01T10:38:03.131 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:6X+8QuBbLpvNDkPsHdboZc9yzqo= 2024-02-01T10:38:03.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:6X+8QuBbLpvNDkPsHdboZc9yzqo=', 'Content-Length': '0'} 2024-02-01T10:38:03.135 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a84e31ecaa3f7d1-0065bb748b-5004-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:03.176 DEBUG:boto:b'vclfcn-42999false' 2024-02-01T10:38:03.176 DEBUG:boto:path=/vclfcn-42/ 2024-02-01T10:38:03.177 DEBUG:boto:auth_path=/vclfcn-42/ 2024-02-01T10:38:03.177 DEBUG:boto:path=/vclfcn-42/?versions&max-keys=999 2024-02-01T10:38:03.177 DEBUG:boto:auth_path=/vclfcn-42/?versions&max-keys=999 2024-02-01T10:38:03.177 DEBUG:boto:Method: GET 2024-02-01T10:38:03.177 DEBUG:boto:Path: /vclfcn-42/?versions&max-keys=999 2024-02-01T10:38:03.177 DEBUG:boto:Data: 2024-02-01T10:38:03.177 DEBUG:boto:Headers: {} 2024-02-01T10:38:03.177 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:03.177 DEBUG:boto:Port: 8001 2024-02-01T10:38:03.177 DEBUG:boto:Params: {} 2024-02-01T10:38:03.178 DEBUG:boto:Token: None 2024-02-01T10:38:03.178 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:03 GMT /vclfcn-42/?versions 2024-02-01T10:38:03.178 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:6X+8QuBbLpvNDkPsHdboZc9yzqo= 2024-02-01T10:38:03.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:6X+8QuBbLpvNDkPsHdboZc9yzqo=', 'Content-Length': '0'} 2024-02-01T10:38:03.183 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c7453e3488dc3590-0065bb748b-5324-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:03.183 DEBUG:boto:b'vclfcn-42999false' 2024-02-01T10:38:03.183 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-42 zones={test-zone2, test-zone3} 2024-02-01T10:38:03.183 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-43 zones={test-zone2, test-zone1} 2024-02-01T10:38:03.184 DEBUG:boto:path=/vclfcn-43/ 2024-02-01T10:38:03.184 DEBUG:boto:auth_path=/vclfcn-43/ 2024-02-01T10:38:03.184 DEBUG:boto:Method: HEAD 2024-02-01T10:38:03.184 DEBUG:boto:Path: /vclfcn-43/ 2024-02-01T10:38:03.184 DEBUG:boto:Data: 2024-02-01T10:38:03.184 DEBUG:boto:Headers: {} 2024-02-01T10:38:03.184 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:03.184 DEBUG:boto:Port: 8000 2024-02-01T10:38:03.184 DEBUG:boto:Params: {} 2024-02-01T10:38:03.185 DEBUG:boto:Token: None 2024-02-01T10:38:03.185 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:03 GMT /vclfcn-43/ 2024-02-01T10:38:03.185 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:G/S3LB1/g8Dl05kcrYrvpt+ZYao= 2024-02-01T10:38:03.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:G/S3LB1/g8Dl05kcrYrvpt+ZYao=', 'Content-Length': '0'} 2024-02-01T10:38:03.189 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007f685717b8bc3fde-0065bb748b-5004-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:03.189 DEBUG:boto:path=/vclfcn-43/ 2024-02-01T10:38:03.189 DEBUG:boto:auth_path=/vclfcn-43/ 2024-02-01T10:38:03.189 DEBUG:boto:Method: HEAD 2024-02-01T10:38:03.189 DEBUG:boto:Path: /vclfcn-43/ 2024-02-01T10:38:03.190 DEBUG:boto:Data: 2024-02-01T10:38:03.190 DEBUG:boto:Headers: {} 2024-02-01T10:38:03.190 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:03.190 DEBUG:boto:Port: 8000 2024-02-01T10:38:03.190 DEBUG:boto:Params: {} 2024-02-01T10:38:03.190 DEBUG:boto:Token: None 2024-02-01T10:38:03.190 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:03 GMT /vclfcn-43/ 2024-02-01T10:38:03.190 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:G/S3LB1/g8Dl05kcrYrvpt+ZYao= 2024-02-01T10:38:03.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:G/S3LB1/g8Dl05kcrYrvpt+ZYao=', 'Content-Length': '0'} 2024-02-01T10:38:03.195 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002263b9bd78e04fd9-0065bb748b-5317-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:03.196 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:38:03.196 DEBUG:boto:path=/vclfcn-43/ 2024-02-01T10:38:03.196 DEBUG:boto:auth_path=/vclfcn-43/ 2024-02-01T10:38:03.196 DEBUG:boto:path=/vclfcn-43/?versions&max-keys=999 2024-02-01T10:38:03.196 DEBUG:boto:auth_path=/vclfcn-43/?versions&max-keys=999 2024-02-01T10:38:03.196 DEBUG:boto:Method: GET 2024-02-01T10:38:03.196 DEBUG:boto:Path: /vclfcn-43/?versions&max-keys=999 2024-02-01T10:38:03.196 DEBUG:boto:Data: 2024-02-01T10:38:03.196 DEBUG:boto:Headers: {} 2024-02-01T10:38:03.196 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:03.197 DEBUG:boto:Port: 8000 2024-02-01T10:38:03.197 DEBUG:boto:Params: {} 2024-02-01T10:38:03.197 DEBUG:boto:Token: None 2024-02-01T10:38:03.197 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:03 GMT /vclfcn-43/?versions 2024-02-01T10:38:03.197 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:bVQdn/0ZyWvhL+a20WcfIOIwNZI= 2024-02-01T10:38:03.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:bVQdn/0ZyWvhL+a20WcfIOIwNZI=', 'Content-Length': '0'} 2024-02-01T10:38:03.201 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f01e35dcde62acf-0065bb748b-5004-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:03.202 DEBUG:boto:b'vclfcn-43999false' 2024-02-01T10:38:03.202 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:38:03.202 DEBUG:boto:path=/vclfcn-43/ 2024-02-01T10:38:03.202 DEBUG:boto:auth_path=/vclfcn-43/ 2024-02-01T10:38:03.202 DEBUG:boto:path=/vclfcn-43/?versions&max-keys=999 2024-02-01T10:38:03.202 DEBUG:boto:auth_path=/vclfcn-43/?versions&max-keys=999 2024-02-01T10:38:03.202 DEBUG:boto:Method: GET 2024-02-01T10:38:03.203 DEBUG:boto:Path: /vclfcn-43/?versions&max-keys=999 2024-02-01T10:38:03.203 DEBUG:boto:Data: 2024-02-01T10:38:03.203 DEBUG:boto:Headers: {} 2024-02-01T10:38:03.203 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:03.203 DEBUG:boto:Port: 8000 2024-02-01T10:38:03.203 DEBUG:boto:Params: {} 2024-02-01T10:38:03.203 DEBUG:boto:Token: None 2024-02-01T10:38:03.203 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:03 GMT /vclfcn-43/?versions 2024-02-01T10:38:03.203 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:bVQdn/0ZyWvhL+a20WcfIOIwNZI= 2024-02-01T10:38:03.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:bVQdn/0ZyWvhL+a20WcfIOIwNZI=', 'Content-Length': '0'} 2024-02-01T10:38:03.208 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000767c720aee5659f5-0065bb748b-5317-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:03.252 DEBUG:boto:b'vclfcn-43999false' 2024-02-01T10:38:03.253 DEBUG:boto:path=/vclfcn-43/ 2024-02-01T10:38:03.253 DEBUG:boto:auth_path=/vclfcn-43/ 2024-02-01T10:38:03.253 DEBUG:boto:path=/vclfcn-43/?versions&max-keys=999 2024-02-01T10:38:03.253 DEBUG:boto:auth_path=/vclfcn-43/?versions&max-keys=999 2024-02-01T10:38:03.253 DEBUG:boto:Method: GET 2024-02-01T10:38:03.253 DEBUG:boto:Path: /vclfcn-43/?versions&max-keys=999 2024-02-01T10:38:03.253 DEBUG:boto:Data: 2024-02-01T10:38:03.253 DEBUG:boto:Headers: {} 2024-02-01T10:38:03.254 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:03.254 DEBUG:boto:Port: 8000 2024-02-01T10:38:03.254 DEBUG:boto:Params: {} 2024-02-01T10:38:03.254 DEBUG:boto:Token: None 2024-02-01T10:38:03.254 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:03 GMT /vclfcn-43/?versions 2024-02-01T10:38:03.255 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:bVQdn/0ZyWvhL+a20WcfIOIwNZI= 2024-02-01T10:38:03.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:bVQdn/0ZyWvhL+a20WcfIOIwNZI=', 'Content-Length': '0'} 2024-02-01T10:38:03.259 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009cb9fa44f8d6133f-0065bb748b-5004-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:03.259 DEBUG:boto:b'vclfcn-43999false' 2024-02-01T10:38:03.259 DEBUG:boto:path=/vclfcn-43/ 2024-02-01T10:38:03.259 DEBUG:boto:auth_path=/vclfcn-43/ 2024-02-01T10:38:03.260 DEBUG:boto:path=/vclfcn-43/?versions&max-keys=999 2024-02-01T10:38:03.260 DEBUG:boto:auth_path=/vclfcn-43/?versions&max-keys=999 2024-02-01T10:38:03.260 DEBUG:boto:Method: GET 2024-02-01T10:38:03.260 DEBUG:boto:Path: /vclfcn-43/?versions&max-keys=999 2024-02-01T10:38:03.260 DEBUG:boto:Data: 2024-02-01T10:38:03.260 DEBUG:boto:Headers: {} 2024-02-01T10:38:03.260 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:03.260 DEBUG:boto:Port: 8000 2024-02-01T10:38:03.260 DEBUG:boto:Params: {} 2024-02-01T10:38:03.260 DEBUG:boto:Token: None 2024-02-01T10:38:03.260 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:03 GMT /vclfcn-43/?versions 2024-02-01T10:38:03.261 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:bVQdn/0ZyWvhL+a20WcfIOIwNZI= 2024-02-01T10:38:03.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:bVQdn/0ZyWvhL+a20WcfIOIwNZI=', 'Content-Length': '0'} 2024-02-01T10:38:03.265 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007580d23f62d505a0-0065bb748b-5317-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:03.265 DEBUG:boto:b'vclfcn-43999false' 2024-02-01T10:38:03.266 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-43 zones={test-zone2, test-zone1} 2024-02-01T10:38:03.266 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-43 zones={test-zone3, test-zone1} 2024-02-01T10:38:03.266 DEBUG:boto:path=/vclfcn-43/ 2024-02-01T10:38:03.266 DEBUG:boto:auth_path=/vclfcn-43/ 2024-02-01T10:38:03.266 DEBUG:boto:Method: HEAD 2024-02-01T10:38:03.266 DEBUG:boto:Path: /vclfcn-43/ 2024-02-01T10:38:03.266 DEBUG:boto:Data: 2024-02-01T10:38:03.266 DEBUG:boto:Headers: {} 2024-02-01T10:38:03.266 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:03.266 DEBUG:boto:Port: 8001 2024-02-01T10:38:03.266 DEBUG:boto:Params: {} 2024-02-01T10:38:03.267 DEBUG:boto:Token: None 2024-02-01T10:38:03.267 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:03 GMT /vclfcn-43/ 2024-02-01T10:38:03.267 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:G/S3LB1/g8Dl05kcrYrvpt+ZYao= 2024-02-01T10:38:03.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:G/S3LB1/g8Dl05kcrYrvpt+ZYao=', 'Content-Length': '0'} 2024-02-01T10:38:03.270 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000ce97236b0ed3625-0065bb748b-5324-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:03.271 DEBUG:boto:path=/vclfcn-43/ 2024-02-01T10:38:03.271 DEBUG:boto:auth_path=/vclfcn-43/ 2024-02-01T10:38:03.271 DEBUG:boto:Method: HEAD 2024-02-01T10:38:03.271 DEBUG:boto:Path: /vclfcn-43/ 2024-02-01T10:38:03.271 DEBUG:boto:Data: 2024-02-01T10:38:03.271 DEBUG:boto:Headers: {} 2024-02-01T10:38:03.271 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:03.271 DEBUG:boto:Port: 8000 2024-02-01T10:38:03.271 DEBUG:boto:Params: {} 2024-02-01T10:38:03.271 DEBUG:boto:Token: None 2024-02-01T10:38:03.272 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:03 GMT /vclfcn-43/ 2024-02-01T10:38:03.272 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:G/S3LB1/g8Dl05kcrYrvpt+ZYao= 2024-02-01T10:38:03.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:G/S3LB1/g8Dl05kcrYrvpt+ZYao=', 'Content-Length': '0'} 2024-02-01T10:38:03.276 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000035574e611859050c-0065bb748b-5317-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:03.276 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:38:03.276 DEBUG:boto:path=/vclfcn-43/ 2024-02-01T10:38:03.276 DEBUG:boto:auth_path=/vclfcn-43/ 2024-02-01T10:38:03.276 DEBUG:boto:path=/vclfcn-43/?versions&max-keys=999 2024-02-01T10:38:03.276 DEBUG:boto:auth_path=/vclfcn-43/?versions&max-keys=999 2024-02-01T10:38:03.277 DEBUG:boto:Method: GET 2024-02-01T10:38:03.277 DEBUG:boto:Path: /vclfcn-43/?versions&max-keys=999 2024-02-01T10:38:03.277 DEBUG:boto:Data: 2024-02-01T10:38:03.277 DEBUG:boto:Headers: {} 2024-02-01T10:38:03.277 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:03.277 DEBUG:boto:Port: 8001 2024-02-01T10:38:03.277 DEBUG:boto:Params: {} 2024-02-01T10:38:03.277 DEBUG:boto:Token: None 2024-02-01T10:38:03.277 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:03 GMT /vclfcn-43/?versions 2024-02-01T10:38:03.277 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:bVQdn/0ZyWvhL+a20WcfIOIwNZI= 2024-02-01T10:38:03.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:bVQdn/0ZyWvhL+a20WcfIOIwNZI=', 'Content-Length': '0'} 2024-02-01T10:38:03.281 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dcbdf76c04893c21-0065bb748b-5324-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:03.324 DEBUG:boto:b'vclfcn-43999false' 2024-02-01T10:38:03.325 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:38:03.325 DEBUG:boto:path=/vclfcn-43/ 2024-02-01T10:38:03.325 DEBUG:boto:auth_path=/vclfcn-43/ 2024-02-01T10:38:03.325 DEBUG:boto:path=/vclfcn-43/?versions&max-keys=999 2024-02-01T10:38:03.325 DEBUG:boto:auth_path=/vclfcn-43/?versions&max-keys=999 2024-02-01T10:38:03.325 DEBUG:boto:Method: GET 2024-02-01T10:38:03.325 DEBUG:boto:Path: /vclfcn-43/?versions&max-keys=999 2024-02-01T10:38:03.325 DEBUG:boto:Data: 2024-02-01T10:38:03.326 DEBUG:boto:Headers: {} 2024-02-01T10:38:03.326 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:03.326 DEBUG:boto:Port: 8000 2024-02-01T10:38:03.326 DEBUG:boto:Params: {} 2024-02-01T10:38:03.326 DEBUG:boto:Token: None 2024-02-01T10:38:03.326 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:03 GMT /vclfcn-43/?versions 2024-02-01T10:38:03.326 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:bVQdn/0ZyWvhL+a20WcfIOIwNZI= 2024-02-01T10:38:03.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:bVQdn/0ZyWvhL+a20WcfIOIwNZI=', 'Content-Length': '0'} 2024-02-01T10:38:03.331 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092f5f11555781300-0065bb748b-5317-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:03.332 DEBUG:boto:b'vclfcn-43999false' 2024-02-01T10:38:03.332 DEBUG:boto:path=/vclfcn-43/ 2024-02-01T10:38:03.332 DEBUG:boto:auth_path=/vclfcn-43/ 2024-02-01T10:38:03.332 DEBUG:boto:path=/vclfcn-43/?versions&max-keys=999 2024-02-01T10:38:03.332 DEBUG:boto:auth_path=/vclfcn-43/?versions&max-keys=999 2024-02-01T10:38:03.332 DEBUG:boto:Method: GET 2024-02-01T10:38:03.333 DEBUG:boto:Path: /vclfcn-43/?versions&max-keys=999 2024-02-01T10:38:03.333 DEBUG:boto:Data: 2024-02-01T10:38:03.333 DEBUG:boto:Headers: {} 2024-02-01T10:38:03.333 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:03.333 DEBUG:boto:Port: 8001 2024-02-01T10:38:03.333 DEBUG:boto:Params: {} 2024-02-01T10:38:03.333 DEBUG:boto:Token: None 2024-02-01T10:38:03.334 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:03 GMT /vclfcn-43/?versions 2024-02-01T10:38:03.334 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:bVQdn/0ZyWvhL+a20WcfIOIwNZI= 2024-02-01T10:38:03.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:bVQdn/0ZyWvhL+a20WcfIOIwNZI=', 'Content-Length': '0'} 2024-02-01T10:38:03.338 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018b1932681a1979b-0065bb748b-5324-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:03.339 DEBUG:boto:b'vclfcn-43999false' 2024-02-01T10:38:03.339 DEBUG:boto:path=/vclfcn-43/ 2024-02-01T10:38:03.339 DEBUG:boto:auth_path=/vclfcn-43/ 2024-02-01T10:38:03.339 DEBUG:boto:path=/vclfcn-43/?versions&max-keys=999 2024-02-01T10:38:03.339 DEBUG:boto:auth_path=/vclfcn-43/?versions&max-keys=999 2024-02-01T10:38:03.339 DEBUG:boto:Method: GET 2024-02-01T10:38:03.339 DEBUG:boto:Path: /vclfcn-43/?versions&max-keys=999 2024-02-01T10:38:03.340 DEBUG:boto:Data: 2024-02-01T10:38:03.340 DEBUG:boto:Headers: {} 2024-02-01T10:38:03.340 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:03.340 DEBUG:boto:Port: 8000 2024-02-01T10:38:03.340 DEBUG:boto:Params: {} 2024-02-01T10:38:03.340 DEBUG:boto:Token: None 2024-02-01T10:38:03.340 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:03 GMT /vclfcn-43/?versions 2024-02-01T10:38:03.340 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:bVQdn/0ZyWvhL+a20WcfIOIwNZI= 2024-02-01T10:38:03.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:bVQdn/0ZyWvhL+a20WcfIOIwNZI=', 'Content-Length': '0'} 2024-02-01T10:38:03.344 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000293c449e42895d43-0065bb748b-5317-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:03.393 DEBUG:boto:b'vclfcn-43999false' 2024-02-01T10:38:03.393 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-43 zones={test-zone3, test-zone1} 2024-02-01T10:38:03.393 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-43 zones={test-zone1, test-zone2} 2024-02-01T10:38:03.393 DEBUG:boto:path=/vclfcn-43/ 2024-02-01T10:38:03.393 DEBUG:boto:auth_path=/vclfcn-43/ 2024-02-01T10:38:03.394 DEBUG:boto:Method: HEAD 2024-02-01T10:38:03.394 DEBUG:boto:Path: /vclfcn-43/ 2024-02-01T10:38:03.394 DEBUG:boto:Data: 2024-02-01T10:38:03.394 DEBUG:boto:Headers: {} 2024-02-01T10:38:03.394 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:03.394 DEBUG:boto:Port: 8000 2024-02-01T10:38:03.394 DEBUG:boto:Params: {} 2024-02-01T10:38:03.394 DEBUG:boto:Token: None 2024-02-01T10:38:03.395 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:03 GMT /vclfcn-43/ 2024-02-01T10:38:03.395 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:G/S3LB1/g8Dl05kcrYrvpt+ZYao= 2024-02-01T10:38:03.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:G/S3LB1/g8Dl05kcrYrvpt+ZYao=', 'Content-Length': '0'} 2024-02-01T10:38:03.400 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b216b2ac9800af39-0065bb748b-5317-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:03.400 DEBUG:boto:path=/vclfcn-43/ 2024-02-01T10:38:03.401 DEBUG:boto:auth_path=/vclfcn-43/ 2024-02-01T10:38:03.401 DEBUG:boto:Method: HEAD 2024-02-01T10:38:03.401 DEBUG:boto:Path: /vclfcn-43/ 2024-02-01T10:38:03.401 DEBUG:boto:Data: 2024-02-01T10:38:03.401 DEBUG:boto:Headers: {} 2024-02-01T10:38:03.401 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:03.401 DEBUG:boto:Port: 8000 2024-02-01T10:38:03.401 DEBUG:boto:Params: {} 2024-02-01T10:38:03.401 DEBUG:boto:Token: None 2024-02-01T10:38:03.402 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:03 GMT /vclfcn-43/ 2024-02-01T10:38:03.402 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:G/S3LB1/g8Dl05kcrYrvpt+ZYao= 2024-02-01T10:38:03.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:G/S3LB1/g8Dl05kcrYrvpt+ZYao=', 'Content-Length': '0'} 2024-02-01T10:38:03.406 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000970125e3bcbb0657-0065bb748b-5004-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:03.406 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:38:03.406 DEBUG:boto:path=/vclfcn-43/ 2024-02-01T10:38:03.406 DEBUG:boto:auth_path=/vclfcn-43/ 2024-02-01T10:38:03.406 DEBUG:boto:path=/vclfcn-43/?versions&max-keys=999 2024-02-01T10:38:03.407 DEBUG:boto:auth_path=/vclfcn-43/?versions&max-keys=999 2024-02-01T10:38:03.407 DEBUG:boto:Method: GET 2024-02-01T10:38:03.407 DEBUG:boto:Path: /vclfcn-43/?versions&max-keys=999 2024-02-01T10:38:03.407 DEBUG:boto:Data: 2024-02-01T10:38:03.407 DEBUG:boto:Headers: {} 2024-02-01T10:38:03.407 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:03.407 DEBUG:boto:Port: 8000 2024-02-01T10:38:03.407 DEBUG:boto:Params: {} 2024-02-01T10:38:03.407 DEBUG:boto:Token: None 2024-02-01T10:38:03.408 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:03 GMT /vclfcn-43/?versions 2024-02-01T10:38:03.408 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:bVQdn/0ZyWvhL+a20WcfIOIwNZI= 2024-02-01T10:38:03.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:bVQdn/0ZyWvhL+a20WcfIOIwNZI=', 'Content-Length': '0'} 2024-02-01T10:38:03.412 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a978c7084ac55b55-0065bb748b-5317-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:03.412 DEBUG:boto:b'vclfcn-43999false' 2024-02-01T10:38:03.413 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:38:03.413 DEBUG:boto:path=/vclfcn-43/ 2024-02-01T10:38:03.413 DEBUG:boto:auth_path=/vclfcn-43/ 2024-02-01T10:38:03.413 DEBUG:boto:path=/vclfcn-43/?versions&max-keys=999 2024-02-01T10:38:03.413 DEBUG:boto:auth_path=/vclfcn-43/?versions&max-keys=999 2024-02-01T10:38:03.414 DEBUG:boto:Method: GET 2024-02-01T10:38:03.414 DEBUG:boto:Path: /vclfcn-43/?versions&max-keys=999 2024-02-01T10:38:03.414 DEBUG:boto:Data: 2024-02-01T10:38:03.414 DEBUG:boto:Headers: {} 2024-02-01T10:38:03.414 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:03.414 DEBUG:boto:Port: 8000 2024-02-01T10:38:03.414 DEBUG:boto:Params: {} 2024-02-01T10:38:03.415 DEBUG:boto:Token: None 2024-02-01T10:38:03.415 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:03 GMT /vclfcn-43/?versions 2024-02-01T10:38:03.415 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:bVQdn/0ZyWvhL+a20WcfIOIwNZI= 2024-02-01T10:38:03.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:bVQdn/0ZyWvhL+a20WcfIOIwNZI=', 'Content-Length': '0'} 2024-02-01T10:38:03.419 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003bc570f0cfdeb945-0065bb748b-5004-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:03.460 DEBUG:boto:b'vclfcn-43999false' 2024-02-01T10:38:03.460 DEBUG:boto:path=/vclfcn-43/ 2024-02-01T10:38:03.461 DEBUG:boto:auth_path=/vclfcn-43/ 2024-02-01T10:38:03.461 DEBUG:boto:path=/vclfcn-43/?versions&max-keys=999 2024-02-01T10:38:03.461 DEBUG:boto:auth_path=/vclfcn-43/?versions&max-keys=999 2024-02-01T10:38:03.461 DEBUG:boto:Method: GET 2024-02-01T10:38:03.461 DEBUG:boto:Path: /vclfcn-43/?versions&max-keys=999 2024-02-01T10:38:03.461 DEBUG:boto:Data: 2024-02-01T10:38:03.461 DEBUG:boto:Headers: {} 2024-02-01T10:38:03.461 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:03.461 DEBUG:boto:Port: 8000 2024-02-01T10:38:03.461 DEBUG:boto:Params: {} 2024-02-01T10:38:03.462 DEBUG:boto:Token: None 2024-02-01T10:38:03.462 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:03 GMT /vclfcn-43/?versions 2024-02-01T10:38:03.462 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:bVQdn/0ZyWvhL+a20WcfIOIwNZI= 2024-02-01T10:38:03.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:bVQdn/0ZyWvhL+a20WcfIOIwNZI=', 'Content-Length': '0'} 2024-02-01T10:38:03.466 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6832153fbf8fa64-0065bb748b-5317-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:03.467 DEBUG:boto:b'vclfcn-43999false' 2024-02-01T10:38:03.467 DEBUG:boto:path=/vclfcn-43/ 2024-02-01T10:38:03.467 DEBUG:boto:auth_path=/vclfcn-43/ 2024-02-01T10:38:03.467 DEBUG:boto:path=/vclfcn-43/?versions&max-keys=999 2024-02-01T10:38:03.467 DEBUG:boto:auth_path=/vclfcn-43/?versions&max-keys=999 2024-02-01T10:38:03.467 DEBUG:boto:Method: GET 2024-02-01T10:38:03.468 DEBUG:boto:Path: /vclfcn-43/?versions&max-keys=999 2024-02-01T10:38:03.468 DEBUG:boto:Data: 2024-02-01T10:38:03.468 DEBUG:boto:Headers: {} 2024-02-01T10:38:03.468 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:03.468 DEBUG:boto:Port: 8000 2024-02-01T10:38:03.468 DEBUG:boto:Params: {} 2024-02-01T10:38:03.468 DEBUG:boto:Token: None 2024-02-01T10:38:03.469 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:03 GMT /vclfcn-43/?versions 2024-02-01T10:38:03.469 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:bVQdn/0ZyWvhL+a20WcfIOIwNZI= 2024-02-01T10:38:03.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:bVQdn/0ZyWvhL+a20WcfIOIwNZI=', 'Content-Length': '0'} 2024-02-01T10:38:03.473 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076a00381ac0e4b80-0065bb748b-5004-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:03.473 DEBUG:boto:b'vclfcn-43999false' 2024-02-01T10:38:03.474 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-43 zones={test-zone1, test-zone2} 2024-02-01T10:38:03.474 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-43 zones={test-zone3, test-zone2} 2024-02-01T10:38:03.474 DEBUG:boto:path=/vclfcn-43/ 2024-02-01T10:38:03.474 DEBUG:boto:auth_path=/vclfcn-43/ 2024-02-01T10:38:03.474 DEBUG:boto:Method: HEAD 2024-02-01T10:38:03.474 DEBUG:boto:Path: /vclfcn-43/ 2024-02-01T10:38:03.474 DEBUG:boto:Data: 2024-02-01T10:38:03.475 DEBUG:boto:Headers: {} 2024-02-01T10:38:03.475 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:03.475 DEBUG:boto:Port: 8001 2024-02-01T10:38:03.475 DEBUG:boto:Params: {} 2024-02-01T10:38:03.475 DEBUG:boto:Token: None 2024-02-01T10:38:03.475 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:03 GMT /vclfcn-43/ 2024-02-01T10:38:03.475 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:G/S3LB1/g8Dl05kcrYrvpt+ZYao= 2024-02-01T10:38:03.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:G/S3LB1/g8Dl05kcrYrvpt+ZYao=', 'Content-Length': '0'} 2024-02-01T10:38:03.479 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f48997fd18903319-0065bb748b-5324-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:03.479 DEBUG:boto:path=/vclfcn-43/ 2024-02-01T10:38:03.480 DEBUG:boto:auth_path=/vclfcn-43/ 2024-02-01T10:38:03.480 DEBUG:boto:Method: HEAD 2024-02-01T10:38:03.480 DEBUG:boto:Path: /vclfcn-43/ 2024-02-01T10:38:03.480 DEBUG:boto:Data: 2024-02-01T10:38:03.480 DEBUG:boto:Headers: {} 2024-02-01T10:38:03.480 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:03.480 DEBUG:boto:Port: 8000 2024-02-01T10:38:03.480 DEBUG:boto:Params: {} 2024-02-01T10:38:03.481 DEBUG:boto:Token: None 2024-02-01T10:38:03.481 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:03 GMT /vclfcn-43/ 2024-02-01T10:38:03.481 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:G/S3LB1/g8Dl05kcrYrvpt+ZYao= 2024-02-01T10:38:03.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:G/S3LB1/g8Dl05kcrYrvpt+ZYao=', 'Content-Length': '0'} 2024-02-01T10:38:03.485 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000103bce1b4bc1f8c0-0065bb748b-5004-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:03.485 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:38:03.485 DEBUG:boto:path=/vclfcn-43/ 2024-02-01T10:38:03.485 DEBUG:boto:auth_path=/vclfcn-43/ 2024-02-01T10:38:03.486 DEBUG:boto:path=/vclfcn-43/?versions&max-keys=999 2024-02-01T10:38:03.486 DEBUG:boto:auth_path=/vclfcn-43/?versions&max-keys=999 2024-02-01T10:38:03.486 DEBUG:boto:Method: GET 2024-02-01T10:38:03.486 DEBUG:boto:Path: /vclfcn-43/?versions&max-keys=999 2024-02-01T10:38:03.486 DEBUG:boto:Data: 2024-02-01T10:38:03.486 DEBUG:boto:Headers: {} 2024-02-01T10:38:03.487 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:03.487 DEBUG:boto:Port: 8001 2024-02-01T10:38:03.487 DEBUG:boto:Params: {} 2024-02-01T10:38:03.487 DEBUG:boto:Token: None 2024-02-01T10:38:03.487 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:03 GMT /vclfcn-43/?versions 2024-02-01T10:38:03.487 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:bVQdn/0ZyWvhL+a20WcfIOIwNZI= 2024-02-01T10:38:03.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:bVQdn/0ZyWvhL+a20WcfIOIwNZI=', 'Content-Length': '0'} 2024-02-01T10:38:03.492 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000006b44814b0ff9bf3-0065bb748b-5324-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:03.493 DEBUG:boto:b'vclfcn-43999false' 2024-02-01T10:38:03.494 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:38:03.494 DEBUG:boto:path=/vclfcn-43/ 2024-02-01T10:38:03.494 DEBUG:boto:auth_path=/vclfcn-43/ 2024-02-01T10:38:03.495 DEBUG:boto:path=/vclfcn-43/?versions&max-keys=999 2024-02-01T10:38:03.495 DEBUG:boto:auth_path=/vclfcn-43/?versions&max-keys=999 2024-02-01T10:38:03.495 DEBUG:boto:Method: GET 2024-02-01T10:38:03.495 DEBUG:boto:Path: /vclfcn-43/?versions&max-keys=999 2024-02-01T10:38:03.495 DEBUG:boto:Data: 2024-02-01T10:38:03.495 DEBUG:boto:Headers: {} 2024-02-01T10:38:03.495 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:03.495 DEBUG:boto:Port: 8000 2024-02-01T10:38:03.495 DEBUG:boto:Params: {} 2024-02-01T10:38:03.495 DEBUG:boto:Token: None 2024-02-01T10:38:03.495 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:03 GMT /vclfcn-43/?versions 2024-02-01T10:38:03.496 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:bVQdn/0ZyWvhL+a20WcfIOIwNZI= 2024-02-01T10:38:03.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:bVQdn/0ZyWvhL+a20WcfIOIwNZI=', 'Content-Length': '0'} 2024-02-01T10:38:03.500 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c89f1bdd6958070-0065bb748b-5004-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:03.540 DEBUG:boto:b'vclfcn-43999false' 2024-02-01T10:38:03.541 DEBUG:boto:path=/vclfcn-43/ 2024-02-01T10:38:03.541 DEBUG:boto:auth_path=/vclfcn-43/ 2024-02-01T10:38:03.541 DEBUG:boto:path=/vclfcn-43/?versions&max-keys=999 2024-02-01T10:38:03.541 DEBUG:boto:auth_path=/vclfcn-43/?versions&max-keys=999 2024-02-01T10:38:03.541 DEBUG:boto:Method: GET 2024-02-01T10:38:03.541 DEBUG:boto:Path: /vclfcn-43/?versions&max-keys=999 2024-02-01T10:38:03.541 DEBUG:boto:Data: 2024-02-01T10:38:03.541 DEBUG:boto:Headers: {} 2024-02-01T10:38:03.542 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:03.542 DEBUG:boto:Port: 8001 2024-02-01T10:38:03.542 DEBUG:boto:Params: {} 2024-02-01T10:38:03.542 DEBUG:boto:Token: None 2024-02-01T10:38:03.542 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:03 GMT /vclfcn-43/?versions 2024-02-01T10:38:03.542 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:bVQdn/0ZyWvhL+a20WcfIOIwNZI= 2024-02-01T10:38:03.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:bVQdn/0ZyWvhL+a20WcfIOIwNZI=', 'Content-Length': '0'} 2024-02-01T10:38:03.547 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000003565f6ea9cd6683-0065bb748b-5324-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:03.547 DEBUG:boto:b'vclfcn-43999false' 2024-02-01T10:38:03.547 DEBUG:boto:path=/vclfcn-43/ 2024-02-01T10:38:03.547 DEBUG:boto:auth_path=/vclfcn-43/ 2024-02-01T10:38:03.547 DEBUG:boto:path=/vclfcn-43/?versions&max-keys=999 2024-02-01T10:38:03.547 DEBUG:boto:auth_path=/vclfcn-43/?versions&max-keys=999 2024-02-01T10:38:03.548 DEBUG:boto:Method: GET 2024-02-01T10:38:03.548 DEBUG:boto:Path: /vclfcn-43/?versions&max-keys=999 2024-02-01T10:38:03.548 DEBUG:boto:Data: 2024-02-01T10:38:03.548 DEBUG:boto:Headers: {} 2024-02-01T10:38:03.548 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:03.548 DEBUG:boto:Port: 8000 2024-02-01T10:38:03.548 DEBUG:boto:Params: {} 2024-02-01T10:38:03.548 DEBUG:boto:Token: None 2024-02-01T10:38:03.548 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:03 GMT /vclfcn-43/?versions 2024-02-01T10:38:03.549 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:bVQdn/0ZyWvhL+a20WcfIOIwNZI= 2024-02-01T10:38:03.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:bVQdn/0ZyWvhL+a20WcfIOIwNZI=', 'Content-Length': '0'} 2024-02-01T10:38:03.553 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000466d413843f5d105-0065bb748b-5004-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:03.553 DEBUG:boto:b'vclfcn-43999false' 2024-02-01T10:38:03.553 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-43 zones={test-zone3, test-zone2} 2024-02-01T10:38:03.553 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-43 zones={test-zone1, test-zone3} 2024-02-01T10:38:03.553 DEBUG:boto:path=/vclfcn-43/ 2024-02-01T10:38:03.554 DEBUG:boto:auth_path=/vclfcn-43/ 2024-02-01T10:38:03.554 DEBUG:boto:Method: HEAD 2024-02-01T10:38:03.554 DEBUG:boto:Path: /vclfcn-43/ 2024-02-01T10:38:03.554 DEBUG:boto:Data: 2024-02-01T10:38:03.554 DEBUG:boto:Headers: {} 2024-02-01T10:38:03.554 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:03.554 DEBUG:boto:Port: 8000 2024-02-01T10:38:03.554 DEBUG:boto:Params: {} 2024-02-01T10:38:03.555 DEBUG:boto:Token: None 2024-02-01T10:38:03.555 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:03 GMT /vclfcn-43/ 2024-02-01T10:38:03.555 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:G/S3LB1/g8Dl05kcrYrvpt+ZYao= 2024-02-01T10:38:03.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:G/S3LB1/g8Dl05kcrYrvpt+ZYao=', 'Content-Length': '0'} 2024-02-01T10:38:03.559 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000098a75b73c42f33a8-0065bb748b-5317-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:03.559 DEBUG:boto:path=/vclfcn-43/ 2024-02-01T10:38:03.560 DEBUG:boto:auth_path=/vclfcn-43/ 2024-02-01T10:38:03.560 DEBUG:boto:Method: HEAD 2024-02-01T10:38:03.560 DEBUG:boto:Path: /vclfcn-43/ 2024-02-01T10:38:03.560 DEBUG:boto:Data: 2024-02-01T10:38:03.560 DEBUG:boto:Headers: {} 2024-02-01T10:38:03.560 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:03.560 DEBUG:boto:Port: 8001 2024-02-01T10:38:03.560 DEBUG:boto:Params: {} 2024-02-01T10:38:03.560 DEBUG:boto:Token: None 2024-02-01T10:38:03.560 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:03 GMT /vclfcn-43/ 2024-02-01T10:38:03.561 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:G/S3LB1/g8Dl05kcrYrvpt+ZYao= 2024-02-01T10:38:03.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:G/S3LB1/g8Dl05kcrYrvpt+ZYao=', 'Content-Length': '0'} 2024-02-01T10:38:03.564 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000841f0cd2148b7dc4-0065bb748b-5324-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:03.565 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:38:03.565 DEBUG:boto:path=/vclfcn-43/ 2024-02-01T10:38:03.565 DEBUG:boto:auth_path=/vclfcn-43/ 2024-02-01T10:38:03.565 DEBUG:boto:path=/vclfcn-43/?versions&max-keys=999 2024-02-01T10:38:03.565 DEBUG:boto:auth_path=/vclfcn-43/?versions&max-keys=999 2024-02-01T10:38:03.565 DEBUG:boto:Method: GET 2024-02-01T10:38:03.565 DEBUG:boto:Path: /vclfcn-43/?versions&max-keys=999 2024-02-01T10:38:03.565 DEBUG:boto:Data: 2024-02-01T10:38:03.565 DEBUG:boto:Headers: {} 2024-02-01T10:38:03.566 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:03.566 DEBUG:boto:Port: 8000 2024-02-01T10:38:03.566 DEBUG:boto:Params: {} 2024-02-01T10:38:03.566 DEBUG:boto:Token: None 2024-02-01T10:38:03.566 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:03 GMT /vclfcn-43/?versions 2024-02-01T10:38:03.566 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:bVQdn/0ZyWvhL+a20WcfIOIwNZI= 2024-02-01T10:38:03.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:bVQdn/0ZyWvhL+a20WcfIOIwNZI=', 'Content-Length': '0'} 2024-02-01T10:38:03.570 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000034390c8bc9e9fd55-0065bb748b-5317-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:03.616 DEBUG:boto:b'vclfcn-43999false' 2024-02-01T10:38:03.616 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:38:03.617 DEBUG:boto:path=/vclfcn-43/ 2024-02-01T10:38:03.617 DEBUG:boto:auth_path=/vclfcn-43/ 2024-02-01T10:38:03.617 DEBUG:boto:path=/vclfcn-43/?versions&max-keys=999 2024-02-01T10:38:03.617 DEBUG:boto:auth_path=/vclfcn-43/?versions&max-keys=999 2024-02-01T10:38:03.617 DEBUG:boto:Method: GET 2024-02-01T10:38:03.617 DEBUG:boto:Path: /vclfcn-43/?versions&max-keys=999 2024-02-01T10:38:03.617 DEBUG:boto:Data: 2024-02-01T10:38:03.617 DEBUG:boto:Headers: {} 2024-02-01T10:38:03.617 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:03.617 DEBUG:boto:Port: 8001 2024-02-01T10:38:03.618 DEBUG:boto:Params: {} 2024-02-01T10:38:03.618 DEBUG:boto:Token: None 2024-02-01T10:38:03.618 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:03 GMT /vclfcn-43/?versions 2024-02-01T10:38:03.618 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:bVQdn/0ZyWvhL+a20WcfIOIwNZI= 2024-02-01T10:38:03.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:bVQdn/0ZyWvhL+a20WcfIOIwNZI=', 'Content-Length': '0'} 2024-02-01T10:38:03.623 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bfaba1f23b7eb964-0065bb748b-5324-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:03.623 DEBUG:boto:b'vclfcn-43999false' 2024-02-01T10:38:03.623 DEBUG:boto:path=/vclfcn-43/ 2024-02-01T10:38:03.623 DEBUG:boto:auth_path=/vclfcn-43/ 2024-02-01T10:38:03.623 DEBUG:boto:path=/vclfcn-43/?versions&max-keys=999 2024-02-01T10:38:03.623 DEBUG:boto:auth_path=/vclfcn-43/?versions&max-keys=999 2024-02-01T10:38:03.623 DEBUG:boto:Method: GET 2024-02-01T10:38:03.624 DEBUG:boto:Path: /vclfcn-43/?versions&max-keys=999 2024-02-01T10:38:03.624 DEBUG:boto:Data: 2024-02-01T10:38:03.624 DEBUG:boto:Headers: {} 2024-02-01T10:38:03.624 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:03.624 DEBUG:boto:Port: 8000 2024-02-01T10:38:03.624 DEBUG:boto:Params: {} 2024-02-01T10:38:03.624 DEBUG:boto:Token: None 2024-02-01T10:38:03.624 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:03 GMT /vclfcn-43/?versions 2024-02-01T10:38:03.624 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:bVQdn/0ZyWvhL+a20WcfIOIwNZI= 2024-02-01T10:38:03.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:bVQdn/0ZyWvhL+a20WcfIOIwNZI=', 'Content-Length': '0'} 2024-02-01T10:38:03.628 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004aa0b425cea4d82-0065bb748b-5317-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:03.629 DEBUG:boto:b'vclfcn-43999false' 2024-02-01T10:38:03.629 DEBUG:boto:path=/vclfcn-43/ 2024-02-01T10:38:03.629 DEBUG:boto:auth_path=/vclfcn-43/ 2024-02-01T10:38:03.629 DEBUG:boto:path=/vclfcn-43/?versions&max-keys=999 2024-02-01T10:38:03.629 DEBUG:boto:auth_path=/vclfcn-43/?versions&max-keys=999 2024-02-01T10:38:03.629 DEBUG:boto:Method: GET 2024-02-01T10:38:03.629 DEBUG:boto:Path: /vclfcn-43/?versions&max-keys=999 2024-02-01T10:38:03.629 DEBUG:boto:Data: 2024-02-01T10:38:03.630 DEBUG:boto:Headers: {} 2024-02-01T10:38:03.630 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:03.630 DEBUG:boto:Port: 8001 2024-02-01T10:38:03.630 DEBUG:boto:Params: {} 2024-02-01T10:38:03.630 DEBUG:boto:Token: None 2024-02-01T10:38:03.630 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:03 GMT /vclfcn-43/?versions 2024-02-01T10:38:03.630 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:bVQdn/0ZyWvhL+a20WcfIOIwNZI= 2024-02-01T10:38:03.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:bVQdn/0ZyWvhL+a20WcfIOIwNZI=', 'Content-Length': '0'} 2024-02-01T10:38:03.634 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de93e90e07073e7d-0065bb748b-5324-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:03.676 DEBUG:boto:b'vclfcn-43999false' 2024-02-01T10:38:03.676 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-43 zones={test-zone1, test-zone3} 2024-02-01T10:38:03.676 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-43 zones={test-zone2, test-zone3} 2024-02-01T10:38:03.677 DEBUG:boto:path=/vclfcn-43/ 2024-02-01T10:38:03.677 DEBUG:boto:auth_path=/vclfcn-43/ 2024-02-01T10:38:03.677 DEBUG:boto:Method: HEAD 2024-02-01T10:38:03.677 DEBUG:boto:Path: /vclfcn-43/ 2024-02-01T10:38:03.677 DEBUG:boto:Data: 2024-02-01T10:38:03.678 DEBUG:boto:Headers: {} 2024-02-01T10:38:03.678 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:03.678 DEBUG:boto:Port: 8000 2024-02-01T10:38:03.678 DEBUG:boto:Params: {} 2024-02-01T10:38:03.678 DEBUG:boto:Token: None 2024-02-01T10:38:03.678 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:03 GMT /vclfcn-43/ 2024-02-01T10:38:03.678 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:G/S3LB1/g8Dl05kcrYrvpt+ZYao= 2024-02-01T10:38:03.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:G/S3LB1/g8Dl05kcrYrvpt+ZYao=', 'Content-Length': '0'} 2024-02-01T10:38:03.682 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000410eeb08f0969348-0065bb748b-5004-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:03.682 DEBUG:boto:path=/vclfcn-43/ 2024-02-01T10:38:03.682 DEBUG:boto:auth_path=/vclfcn-43/ 2024-02-01T10:38:03.683 DEBUG:boto:Method: HEAD 2024-02-01T10:38:03.683 DEBUG:boto:Path: /vclfcn-43/ 2024-02-01T10:38:03.683 DEBUG:boto:Data: 2024-02-01T10:38:03.683 DEBUG:boto:Headers: {} 2024-02-01T10:38:03.683 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:03.683 DEBUG:boto:Port: 8001 2024-02-01T10:38:03.683 DEBUG:boto:Params: {} 2024-02-01T10:38:03.683 DEBUG:boto:Token: None 2024-02-01T10:38:03.683 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:03 GMT /vclfcn-43/ 2024-02-01T10:38:03.684 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:G/S3LB1/g8Dl05kcrYrvpt+ZYao= 2024-02-01T10:38:03.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:G/S3LB1/g8Dl05kcrYrvpt+ZYao=', 'Content-Length': '0'} 2024-02-01T10:38:03.688 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b35ad60dd3cdcf35-0065bb748b-5324-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:03.688 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:38:03.689 DEBUG:boto:path=/vclfcn-43/ 2024-02-01T10:38:03.689 DEBUG:boto:auth_path=/vclfcn-43/ 2024-02-01T10:38:03.689 DEBUG:boto:path=/vclfcn-43/?versions&max-keys=999 2024-02-01T10:38:03.689 DEBUG:boto:auth_path=/vclfcn-43/?versions&max-keys=999 2024-02-01T10:38:03.689 DEBUG:boto:Method: GET 2024-02-01T10:38:03.689 DEBUG:boto:Path: /vclfcn-43/?versions&max-keys=999 2024-02-01T10:38:03.689 DEBUG:boto:Data: 2024-02-01T10:38:03.689 DEBUG:boto:Headers: {} 2024-02-01T10:38:03.689 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:03.690 DEBUG:boto:Port: 8000 2024-02-01T10:38:03.690 DEBUG:boto:Params: {} 2024-02-01T10:38:03.690 DEBUG:boto:Token: None 2024-02-01T10:38:03.690 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:03 GMT /vclfcn-43/?versions 2024-02-01T10:38:03.690 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:bVQdn/0ZyWvhL+a20WcfIOIwNZI= 2024-02-01T10:38:03.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:bVQdn/0ZyWvhL+a20WcfIOIwNZI=', 'Content-Length': '0'} 2024-02-01T10:38:03.694 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000deae6ee3a12dab8-0065bb748b-5004-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:03.694 DEBUG:boto:b'vclfcn-43999false' 2024-02-01T10:38:03.695 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:38:03.695 DEBUG:boto:path=/vclfcn-43/ 2024-02-01T10:38:03.695 DEBUG:boto:auth_path=/vclfcn-43/ 2024-02-01T10:38:03.695 DEBUG:boto:path=/vclfcn-43/?versions&max-keys=999 2024-02-01T10:38:03.695 DEBUG:boto:auth_path=/vclfcn-43/?versions&max-keys=999 2024-02-01T10:38:03.695 DEBUG:boto:Method: GET 2024-02-01T10:38:03.695 DEBUG:boto:Path: /vclfcn-43/?versions&max-keys=999 2024-02-01T10:38:03.695 DEBUG:boto:Data: 2024-02-01T10:38:03.695 DEBUG:boto:Headers: {} 2024-02-01T10:38:03.695 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:03.696 DEBUG:boto:Port: 8001 2024-02-01T10:38:03.696 DEBUG:boto:Params: {} 2024-02-01T10:38:03.696 DEBUG:boto:Token: None 2024-02-01T10:38:03.696 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:03 GMT /vclfcn-43/?versions 2024-02-01T10:38:03.696 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:bVQdn/0ZyWvhL+a20WcfIOIwNZI= 2024-02-01T10:38:03.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:bVQdn/0ZyWvhL+a20WcfIOIwNZI=', 'Content-Length': '0'} 2024-02-01T10:38:03.700 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008452f39eb2a6da96-0065bb748b-5324-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:03.701 DEBUG:boto:b'vclfcn-43999false' 2024-02-01T10:38:03.701 DEBUG:boto:path=/vclfcn-43/ 2024-02-01T10:38:03.701 DEBUG:boto:auth_path=/vclfcn-43/ 2024-02-01T10:38:03.701 DEBUG:boto:path=/vclfcn-43/?versions&max-keys=999 2024-02-01T10:38:03.702 DEBUG:boto:auth_path=/vclfcn-43/?versions&max-keys=999 2024-02-01T10:38:03.702 DEBUG:boto:Method: GET 2024-02-01T10:38:03.702 DEBUG:boto:Path: /vclfcn-43/?versions&max-keys=999 2024-02-01T10:38:03.702 DEBUG:boto:Data: 2024-02-01T10:38:03.702 DEBUG:boto:Headers: {} 2024-02-01T10:38:03.702 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:03.702 DEBUG:boto:Port: 8000 2024-02-01T10:38:03.702 DEBUG:boto:Params: {} 2024-02-01T10:38:03.702 DEBUG:boto:Token: None 2024-02-01T10:38:03.702 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:03 GMT /vclfcn-43/?versions 2024-02-01T10:38:03.703 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:bVQdn/0ZyWvhL+a20WcfIOIwNZI= 2024-02-01T10:38:03.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:bVQdn/0ZyWvhL+a20WcfIOIwNZI=', 'Content-Length': '0'} 2024-02-01T10:38:03.707 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a669edab4f3fec50-0065bb748b-5004-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:03.752 DEBUG:boto:b'vclfcn-43999false' 2024-02-01T10:38:03.752 DEBUG:boto:path=/vclfcn-43/ 2024-02-01T10:38:03.753 DEBUG:boto:auth_path=/vclfcn-43/ 2024-02-01T10:38:03.753 DEBUG:boto:path=/vclfcn-43/?versions&max-keys=999 2024-02-01T10:38:03.753 DEBUG:boto:auth_path=/vclfcn-43/?versions&max-keys=999 2024-02-01T10:38:03.753 DEBUG:boto:Method: GET 2024-02-01T10:38:03.753 DEBUG:boto:Path: /vclfcn-43/?versions&max-keys=999 2024-02-01T10:38:03.753 DEBUG:boto:Data: 2024-02-01T10:38:03.753 DEBUG:boto:Headers: {} 2024-02-01T10:38:03.753 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:03.754 DEBUG:boto:Port: 8001 2024-02-01T10:38:03.754 DEBUG:boto:Params: {} 2024-02-01T10:38:03.754 DEBUG:boto:Token: None 2024-02-01T10:38:03.754 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:03 GMT /vclfcn-43/?versions 2024-02-01T10:38:03.754 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:bVQdn/0ZyWvhL+a20WcfIOIwNZI= 2024-02-01T10:38:03.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:bVQdn/0ZyWvhL+a20WcfIOIwNZI=', 'Content-Length': '0'} 2024-02-01T10:38:03.759 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2ee90ab2d010d56-0065bb748b-5324-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:03.759 DEBUG:boto:b'vclfcn-43999false' 2024-02-01T10:38:03.759 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-43 zones={test-zone2, test-zone3} 2024-02-01T10:38:03.759 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-44 zones={test-zone2, test-zone1} 2024-02-01T10:38:03.760 DEBUG:boto:path=/vclfcn-44/ 2024-02-01T10:38:03.760 DEBUG:boto:auth_path=/vclfcn-44/ 2024-02-01T10:38:03.760 DEBUG:boto:Method: HEAD 2024-02-01T10:38:03.760 DEBUG:boto:Path: /vclfcn-44/ 2024-02-01T10:38:03.760 DEBUG:boto:Data: 2024-02-01T10:38:03.760 DEBUG:boto:Headers: {} 2024-02-01T10:38:03.760 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:03.760 DEBUG:boto:Port: 8000 2024-02-01T10:38:03.760 DEBUG:boto:Params: {} 2024-02-01T10:38:03.761 DEBUG:boto:Token: None 2024-02-01T10:38:03.761 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:03 GMT /vclfcn-44/ 2024-02-01T10:38:03.761 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ROZNmp++BWYyJU20yz2go5Hk5hA= 2024-02-01T10:38:03.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ROZNmp++BWYyJU20yz2go5Hk5hA=', 'Content-Length': '0'} 2024-02-01T10:38:03.766 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a00820a4eb104e63-0065bb748b-5004-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:03.766 DEBUG:boto:path=/vclfcn-44/ 2024-02-01T10:38:03.766 DEBUG:boto:auth_path=/vclfcn-44/ 2024-02-01T10:38:03.766 DEBUG:boto:Method: HEAD 2024-02-01T10:38:03.766 DEBUG:boto:Path: /vclfcn-44/ 2024-02-01T10:38:03.767 DEBUG:boto:Data: 2024-02-01T10:38:03.767 DEBUG:boto:Headers: {} 2024-02-01T10:38:03.767 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:03.767 DEBUG:boto:Port: 8000 2024-02-01T10:38:03.767 DEBUG:boto:Params: {} 2024-02-01T10:38:03.768 DEBUG:boto:Token: None 2024-02-01T10:38:03.768 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:03 GMT /vclfcn-44/ 2024-02-01T10:38:03.768 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ROZNmp++BWYyJU20yz2go5Hk5hA= 2024-02-01T10:38:03.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ROZNmp++BWYyJU20yz2go5Hk5hA=', 'Content-Length': '0'} 2024-02-01T10:38:03.772 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c8c5a3b8abec1b42-0065bb748b-5317-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:03.772 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:38:03.773 DEBUG:boto:path=/vclfcn-44/ 2024-02-01T10:38:03.773 DEBUG:boto:auth_path=/vclfcn-44/ 2024-02-01T10:38:03.773 DEBUG:boto:path=/vclfcn-44/?versions&max-keys=999 2024-02-01T10:38:03.773 DEBUG:boto:auth_path=/vclfcn-44/?versions&max-keys=999 2024-02-01T10:38:03.773 DEBUG:boto:Method: GET 2024-02-01T10:38:03.773 DEBUG:boto:Path: /vclfcn-44/?versions&max-keys=999 2024-02-01T10:38:03.773 DEBUG:boto:Data: 2024-02-01T10:38:03.774 DEBUG:boto:Headers: {} 2024-02-01T10:38:03.774 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:03.774 DEBUG:boto:Port: 8000 2024-02-01T10:38:03.774 DEBUG:boto:Params: {} 2024-02-01T10:38:03.774 DEBUG:boto:Token: None 2024-02-01T10:38:03.774 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:03 GMT /vclfcn-44/?versions 2024-02-01T10:38:03.774 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1L0TucnkCB31BtgZkDAjVl0YdKo= 2024-02-01T10:38:03.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1L0TucnkCB31BtgZkDAjVl0YdKo=', 'Content-Length': '0'} 2024-02-01T10:38:03.778 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000475922bb3c8b0ad5-0065bb748b-5004-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:03.778 DEBUG:boto:b'vclfcn-44999false' 2024-02-01T10:38:03.779 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:38:03.779 DEBUG:boto:path=/vclfcn-44/ 2024-02-01T10:38:03.779 DEBUG:boto:auth_path=/vclfcn-44/ 2024-02-01T10:38:03.779 DEBUG:boto:path=/vclfcn-44/?versions&max-keys=999 2024-02-01T10:38:03.779 DEBUG:boto:auth_path=/vclfcn-44/?versions&max-keys=999 2024-02-01T10:38:03.779 DEBUG:boto:Method: GET 2024-02-01T10:38:03.779 DEBUG:boto:Path: /vclfcn-44/?versions&max-keys=999 2024-02-01T10:38:03.779 DEBUG:boto:Data: 2024-02-01T10:38:03.779 DEBUG:boto:Headers: {} 2024-02-01T10:38:03.779 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:03.780 DEBUG:boto:Port: 8000 2024-02-01T10:38:03.780 DEBUG:boto:Params: {} 2024-02-01T10:38:03.780 DEBUG:boto:Token: None 2024-02-01T10:38:03.780 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:03 GMT /vclfcn-44/?versions 2024-02-01T10:38:03.780 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1L0TucnkCB31BtgZkDAjVl0YdKo= 2024-02-01T10:38:03.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1L0TucnkCB31BtgZkDAjVl0YdKo=', 'Content-Length': '0'} 2024-02-01T10:38:03.784 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071aead00bf074224-0065bb748b-5317-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:03.784 DEBUG:boto:b'vclfcn-44999false' 2024-02-01T10:38:03.785 DEBUG:boto:path=/vclfcn-44/ 2024-02-01T10:38:03.785 DEBUG:boto:auth_path=/vclfcn-44/ 2024-02-01T10:38:03.785 DEBUG:boto:path=/vclfcn-44/?versions&max-keys=999 2024-02-01T10:38:03.785 DEBUG:boto:auth_path=/vclfcn-44/?versions&max-keys=999 2024-02-01T10:38:03.785 DEBUG:boto:Method: GET 2024-02-01T10:38:03.785 DEBUG:boto:Path: /vclfcn-44/?versions&max-keys=999 2024-02-01T10:38:03.785 DEBUG:boto:Data: 2024-02-01T10:38:03.785 DEBUG:boto:Headers: {} 2024-02-01T10:38:03.785 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:03.785 DEBUG:boto:Port: 8000 2024-02-01T10:38:03.786 DEBUG:boto:Params: {} 2024-02-01T10:38:03.786 DEBUG:boto:Token: None 2024-02-01T10:38:03.786 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:03 GMT /vclfcn-44/?versions 2024-02-01T10:38:03.786 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1L0TucnkCB31BtgZkDAjVl0YdKo= 2024-02-01T10:38:03.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1L0TucnkCB31BtgZkDAjVl0YdKo=', 'Content-Length': '0'} 2024-02-01T10:38:03.790 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000390042cb9c43faa-0065bb748b-5004-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:03.832 DEBUG:boto:b'vclfcn-44999false' 2024-02-01T10:38:03.833 DEBUG:boto:path=/vclfcn-44/ 2024-02-01T10:38:03.833 DEBUG:boto:auth_path=/vclfcn-44/ 2024-02-01T10:38:03.833 DEBUG:boto:path=/vclfcn-44/?versions&max-keys=999 2024-02-01T10:38:03.833 DEBUG:boto:auth_path=/vclfcn-44/?versions&max-keys=999 2024-02-01T10:38:03.833 DEBUG:boto:Method: GET 2024-02-01T10:38:03.833 DEBUG:boto:Path: /vclfcn-44/?versions&max-keys=999 2024-02-01T10:38:03.833 DEBUG:boto:Data: 2024-02-01T10:38:03.833 DEBUG:boto:Headers: {} 2024-02-01T10:38:03.833 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:03.834 DEBUG:boto:Port: 8000 2024-02-01T10:38:03.834 DEBUG:boto:Params: {} 2024-02-01T10:38:03.834 DEBUG:boto:Token: None 2024-02-01T10:38:03.834 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:03 GMT /vclfcn-44/?versions 2024-02-01T10:38:03.834 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1L0TucnkCB31BtgZkDAjVl0YdKo= 2024-02-01T10:38:03.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1L0TucnkCB31BtgZkDAjVl0YdKo=', 'Content-Length': '0'} 2024-02-01T10:38:03.839 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cdb9eaca1ae69fe6-0065bb748b-5317-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:03.839 DEBUG:boto:b'vclfcn-44999false' 2024-02-01T10:38:03.839 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-44 zones={test-zone2, test-zone1} 2024-02-01T10:38:03.839 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-44 zones={test-zone3, test-zone1} 2024-02-01T10:38:03.839 DEBUG:boto:path=/vclfcn-44/ 2024-02-01T10:38:03.840 DEBUG:boto:auth_path=/vclfcn-44/ 2024-02-01T10:38:03.840 DEBUG:boto:Method: HEAD 2024-02-01T10:38:03.840 DEBUG:boto:Path: /vclfcn-44/ 2024-02-01T10:38:03.840 DEBUG:boto:Data: 2024-02-01T10:38:03.840 DEBUG:boto:Headers: {} 2024-02-01T10:38:03.840 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:03.840 DEBUG:boto:Port: 8001 2024-02-01T10:38:03.840 DEBUG:boto:Params: {} 2024-02-01T10:38:03.840 DEBUG:boto:Token: None 2024-02-01T10:38:03.840 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:03 GMT /vclfcn-44/ 2024-02-01T10:38:03.841 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ROZNmp++BWYyJU20yz2go5Hk5hA= 2024-02-01T10:38:03.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ROZNmp++BWYyJU20yz2go5Hk5hA=', 'Content-Length': '0'} 2024-02-01T10:38:03.844 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000874607466ff9842d-0065bb748b-5324-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:03.845 DEBUG:boto:path=/vclfcn-44/ 2024-02-01T10:38:03.845 DEBUG:boto:auth_path=/vclfcn-44/ 2024-02-01T10:38:03.845 DEBUG:boto:Method: HEAD 2024-02-01T10:38:03.845 DEBUG:boto:Path: /vclfcn-44/ 2024-02-01T10:38:03.845 DEBUG:boto:Data: 2024-02-01T10:38:03.845 DEBUG:boto:Headers: {} 2024-02-01T10:38:03.846 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:03.846 DEBUG:boto:Port: 8000 2024-02-01T10:38:03.846 DEBUG:boto:Params: {} 2024-02-01T10:38:03.846 DEBUG:boto:Token: None 2024-02-01T10:38:03.846 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:03 GMT /vclfcn-44/ 2024-02-01T10:38:03.846 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ROZNmp++BWYyJU20yz2go5Hk5hA= 2024-02-01T10:38:03.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ROZNmp++BWYyJU20yz2go5Hk5hA=', 'Content-Length': '0'} 2024-02-01T10:38:03.850 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b0a16f03aa7ba3e0-0065bb748b-5317-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:03.850 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:38:03.851 DEBUG:boto:path=/vclfcn-44/ 2024-02-01T10:38:03.851 DEBUG:boto:auth_path=/vclfcn-44/ 2024-02-01T10:38:03.851 DEBUG:boto:path=/vclfcn-44/?versions&max-keys=999 2024-02-01T10:38:03.851 DEBUG:boto:auth_path=/vclfcn-44/?versions&max-keys=999 2024-02-01T10:38:03.851 DEBUG:boto:Method: GET 2024-02-01T10:38:03.852 DEBUG:boto:Path: /vclfcn-44/?versions&max-keys=999 2024-02-01T10:38:03.852 DEBUG:boto:Data: 2024-02-01T10:38:03.852 DEBUG:boto:Headers: {} 2024-02-01T10:38:03.852 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:03.852 DEBUG:boto:Port: 8001 2024-02-01T10:38:03.852 DEBUG:boto:Params: {} 2024-02-01T10:38:03.852 DEBUG:boto:Token: None 2024-02-01T10:38:03.852 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:03 GMT /vclfcn-44/?versions 2024-02-01T10:38:03.853 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1L0TucnkCB31BtgZkDAjVl0YdKo= 2024-02-01T10:38:03.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1L0TucnkCB31BtgZkDAjVl0YdKo=', 'Content-Length': '0'} 2024-02-01T10:38:03.857 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000675fb673dc0a90c-0065bb748b-5324-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:03.904 DEBUG:boto:b'vclfcn-44999false' 2024-02-01T10:38:03.905 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:38:03.905 DEBUG:boto:path=/vclfcn-44/ 2024-02-01T10:38:03.905 DEBUG:boto:auth_path=/vclfcn-44/ 2024-02-01T10:38:03.905 DEBUG:boto:path=/vclfcn-44/?versions&max-keys=999 2024-02-01T10:38:03.905 DEBUG:boto:auth_path=/vclfcn-44/?versions&max-keys=999 2024-02-01T10:38:03.905 DEBUG:boto:Method: GET 2024-02-01T10:38:03.905 DEBUG:boto:Path: /vclfcn-44/?versions&max-keys=999 2024-02-01T10:38:03.905 DEBUG:boto:Data: 2024-02-01T10:38:03.906 DEBUG:boto:Headers: {} 2024-02-01T10:38:03.906 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:03.906 DEBUG:boto:Port: 8000 2024-02-01T10:38:03.906 DEBUG:boto:Params: {} 2024-02-01T10:38:03.906 DEBUG:boto:Token: None 2024-02-01T10:38:03.906 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:03 GMT /vclfcn-44/?versions 2024-02-01T10:38:03.906 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1L0TucnkCB31BtgZkDAjVl0YdKo= 2024-02-01T10:38:03.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1L0TucnkCB31BtgZkDAjVl0YdKo=', 'Content-Length': '0'} 2024-02-01T10:38:03.911 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bdd11eef614a4f9f-0065bb748b-5317-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:03.911 DEBUG:boto:b'vclfcn-44999false' 2024-02-01T10:38:03.911 DEBUG:boto:path=/vclfcn-44/ 2024-02-01T10:38:03.911 DEBUG:boto:auth_path=/vclfcn-44/ 2024-02-01T10:38:03.911 DEBUG:boto:path=/vclfcn-44/?versions&max-keys=999 2024-02-01T10:38:03.912 DEBUG:boto:auth_path=/vclfcn-44/?versions&max-keys=999 2024-02-01T10:38:03.912 DEBUG:boto:Method: GET 2024-02-01T10:38:03.912 DEBUG:boto:Path: /vclfcn-44/?versions&max-keys=999 2024-02-01T10:38:03.912 DEBUG:boto:Data: 2024-02-01T10:38:03.912 DEBUG:boto:Headers: {} 2024-02-01T10:38:03.912 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:03.912 DEBUG:boto:Port: 8001 2024-02-01T10:38:03.912 DEBUG:boto:Params: {} 2024-02-01T10:38:03.912 DEBUG:boto:Token: None 2024-02-01T10:38:03.912 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:03 GMT /vclfcn-44/?versions 2024-02-01T10:38:03.913 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1L0TucnkCB31BtgZkDAjVl0YdKo= 2024-02-01T10:38:03.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1L0TucnkCB31BtgZkDAjVl0YdKo=', 'Content-Length': '0'} 2024-02-01T10:38:03.917 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000624e42fb76e7bc47-0065bb748b-5324-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:03.918 DEBUG:boto:b'vclfcn-44999false' 2024-02-01T10:38:03.918 DEBUG:boto:path=/vclfcn-44/ 2024-02-01T10:38:03.918 DEBUG:boto:auth_path=/vclfcn-44/ 2024-02-01T10:38:03.918 DEBUG:boto:path=/vclfcn-44/?versions&max-keys=999 2024-02-01T10:38:03.919 DEBUG:boto:auth_path=/vclfcn-44/?versions&max-keys=999 2024-02-01T10:38:03.919 DEBUG:boto:Method: GET 2024-02-01T10:38:03.919 DEBUG:boto:Path: /vclfcn-44/?versions&max-keys=999 2024-02-01T10:38:03.919 DEBUG:boto:Data: 2024-02-01T10:38:03.919 DEBUG:boto:Headers: {} 2024-02-01T10:38:03.919 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:03.919 DEBUG:boto:Port: 8000 2024-02-01T10:38:03.919 DEBUG:boto:Params: {} 2024-02-01T10:38:03.919 DEBUG:boto:Token: None 2024-02-01T10:38:03.920 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:03 GMT /vclfcn-44/?versions 2024-02-01T10:38:03.920 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1L0TucnkCB31BtgZkDAjVl0YdKo= 2024-02-01T10:38:03.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1L0TucnkCB31BtgZkDAjVl0YdKo=', 'Content-Length': '0'} 2024-02-01T10:38:03.924 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006af0050ebbf9979f-0065bb748b-5317-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:03.925 DEBUG:boto:b'vclfcn-44999false' 2024-02-01T10:38:03.927 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-44 zones={test-zone3, test-zone1} 2024-02-01T10:38:03.927 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-44 zones={test-zone1, test-zone2} 2024-02-01T10:38:03.927 DEBUG:boto:path=/vclfcn-44/ 2024-02-01T10:38:03.927 DEBUG:boto:auth_path=/vclfcn-44/ 2024-02-01T10:38:03.927 DEBUG:boto:Method: HEAD 2024-02-01T10:38:03.927 DEBUG:boto:Path: /vclfcn-44/ 2024-02-01T10:38:03.927 DEBUG:boto:Data: 2024-02-01T10:38:03.928 DEBUG:boto:Headers: {} 2024-02-01T10:38:03.928 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:03.928 DEBUG:boto:Port: 8000 2024-02-01T10:38:03.928 DEBUG:boto:Params: {} 2024-02-01T10:38:03.928 DEBUG:boto:Token: None 2024-02-01T10:38:03.928 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:03 GMT /vclfcn-44/ 2024-02-01T10:38:03.928 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ROZNmp++BWYyJU20yz2go5Hk5hA= 2024-02-01T10:38:03.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ROZNmp++BWYyJU20yz2go5Hk5hA=', 'Content-Length': '0'} 2024-02-01T10:38:03.933 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006fabff404b5784ef-0065bb748b-5317-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:03.933 DEBUG:boto:path=/vclfcn-44/ 2024-02-01T10:38:03.933 DEBUG:boto:auth_path=/vclfcn-44/ 2024-02-01T10:38:03.933 DEBUG:boto:Method: HEAD 2024-02-01T10:38:03.934 DEBUG:boto:Path: /vclfcn-44/ 2024-02-01T10:38:03.934 DEBUG:boto:Data: 2024-02-01T10:38:03.934 DEBUG:boto:Headers: {} 2024-02-01T10:38:03.934 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:03.934 DEBUG:boto:Port: 8000 2024-02-01T10:38:03.934 DEBUG:boto:Params: {} 2024-02-01T10:38:03.934 DEBUG:boto:Token: None 2024-02-01T10:38:03.934 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:03 GMT /vclfcn-44/ 2024-02-01T10:38:03.934 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ROZNmp++BWYyJU20yz2go5Hk5hA= 2024-02-01T10:38:03.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ROZNmp++BWYyJU20yz2go5Hk5hA=', 'Content-Length': '0'} 2024-02-01T10:38:03.939 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000337eb8463aee0fff-0065bb748b-5004-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:03.939 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:38:03.939 DEBUG:boto:path=/vclfcn-44/ 2024-02-01T10:38:03.939 DEBUG:boto:auth_path=/vclfcn-44/ 2024-02-01T10:38:03.939 DEBUG:boto:path=/vclfcn-44/?versions&max-keys=999 2024-02-01T10:38:03.940 DEBUG:boto:auth_path=/vclfcn-44/?versions&max-keys=999 2024-02-01T10:38:03.940 DEBUG:boto:Method: GET 2024-02-01T10:38:03.940 DEBUG:boto:Path: /vclfcn-44/?versions&max-keys=999 2024-02-01T10:38:03.940 DEBUG:boto:Data: 2024-02-01T10:38:03.940 DEBUG:boto:Headers: {} 2024-02-01T10:38:03.940 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:03.940 DEBUG:boto:Port: 8000 2024-02-01T10:38:03.940 DEBUG:boto:Params: {} 2024-02-01T10:38:03.941 DEBUG:boto:Token: None 2024-02-01T10:38:03.941 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:03 GMT /vclfcn-44/?versions 2024-02-01T10:38:03.941 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1L0TucnkCB31BtgZkDAjVl0YdKo= 2024-02-01T10:38:03.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1L0TucnkCB31BtgZkDAjVl0YdKo=', 'Content-Length': '0'} 2024-02-01T10:38:03.945 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb047ad07a279280-0065bb748b-5317-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:03.993 DEBUG:boto:b'vclfcn-44999false' 2024-02-01T10:38:03.994 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:38:03.994 DEBUG:boto:path=/vclfcn-44/ 2024-02-01T10:38:03.994 DEBUG:boto:auth_path=/vclfcn-44/ 2024-02-01T10:38:03.994 DEBUG:boto:path=/vclfcn-44/?versions&max-keys=999 2024-02-01T10:38:03.995 DEBUG:boto:auth_path=/vclfcn-44/?versions&max-keys=999 2024-02-01T10:38:03.995 DEBUG:boto:Method: GET 2024-02-01T10:38:03.995 DEBUG:boto:Path: /vclfcn-44/?versions&max-keys=999 2024-02-01T10:38:03.995 DEBUG:boto:Data: 2024-02-01T10:38:03.995 DEBUG:boto:Headers: {} 2024-02-01T10:38:03.995 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:03.995 DEBUG:boto:Port: 8000 2024-02-01T10:38:03.995 DEBUG:boto:Params: {} 2024-02-01T10:38:03.996 DEBUG:boto:Token: None 2024-02-01T10:38:03.996 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:03 GMT /vclfcn-44/?versions 2024-02-01T10:38:03.996 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1L0TucnkCB31BtgZkDAjVl0YdKo= 2024-02-01T10:38:03.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1L0TucnkCB31BtgZkDAjVl0YdKo=', 'Content-Length': '0'} 2024-02-01T10:38:04.002 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fcf36685abd60a55-0065bb748b-5004-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:04.003 DEBUG:boto:b'vclfcn-44999false' 2024-02-01T10:38:04.003 DEBUG:boto:path=/vclfcn-44/ 2024-02-01T10:38:04.003 DEBUG:boto:auth_path=/vclfcn-44/ 2024-02-01T10:38:04.003 DEBUG:boto:path=/vclfcn-44/?versions&max-keys=999 2024-02-01T10:38:04.003 DEBUG:boto:auth_path=/vclfcn-44/?versions&max-keys=999 2024-02-01T10:38:04.003 DEBUG:boto:Method: GET 2024-02-01T10:38:04.004 DEBUG:boto:Path: /vclfcn-44/?versions&max-keys=999 2024-02-01T10:38:04.004 DEBUG:boto:Data: 2024-02-01T10:38:04.004 DEBUG:boto:Headers: {} 2024-02-01T10:38:04.004 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:04.004 DEBUG:boto:Port: 8000 2024-02-01T10:38:04.004 DEBUG:boto:Params: {} 2024-02-01T10:38:04.005 DEBUG:boto:Token: None 2024-02-01T10:38:04.005 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:04 GMT /vclfcn-44/?versions 2024-02-01T10:38:04.005 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7nWrm5tTclA1gOH0X7HaX95X9JI= 2024-02-01T10:38:04.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7nWrm5tTclA1gOH0X7HaX95X9JI=', 'Content-Length': '0'} 2024-02-01T10:38:04.012 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086d3b1504cef86f6-0065bb748c-5317-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:04.012 DEBUG:boto:b'vclfcn-44999false' 2024-02-01T10:38:04.012 DEBUG:boto:path=/vclfcn-44/ 2024-02-01T10:38:04.013 DEBUG:boto:auth_path=/vclfcn-44/ 2024-02-01T10:38:04.013 DEBUG:boto:path=/vclfcn-44/?versions&max-keys=999 2024-02-01T10:38:04.013 DEBUG:boto:auth_path=/vclfcn-44/?versions&max-keys=999 2024-02-01T10:38:04.013 DEBUG:boto:Method: GET 2024-02-01T10:38:04.013 DEBUG:boto:Path: /vclfcn-44/?versions&max-keys=999 2024-02-01T10:38:04.013 DEBUG:boto:Data: 2024-02-01T10:38:04.013 DEBUG:boto:Headers: {} 2024-02-01T10:38:04.013 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:04.013 DEBUG:boto:Port: 8000 2024-02-01T10:38:04.013 DEBUG:boto:Params: {} 2024-02-01T10:38:04.013 DEBUG:boto:Token: None 2024-02-01T10:38:04.014 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:04 GMT /vclfcn-44/?versions 2024-02-01T10:38:04.014 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7nWrm5tTclA1gOH0X7HaX95X9JI= 2024-02-01T10:38:04.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7nWrm5tTclA1gOH0X7HaX95X9JI=', 'Content-Length': '0'} 2024-02-01T10:38:04.018 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4fbef1bdc318afd-0065bb748c-5004-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:04.018 DEBUG:boto:b'vclfcn-44999false' 2024-02-01T10:38:04.018 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-44 zones={test-zone1, test-zone2} 2024-02-01T10:38:04.019 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-44 zones={test-zone3, test-zone2} 2024-02-01T10:38:04.019 DEBUG:boto:path=/vclfcn-44/ 2024-02-01T10:38:04.019 DEBUG:boto:auth_path=/vclfcn-44/ 2024-02-01T10:38:04.020 DEBUG:boto:Method: HEAD 2024-02-01T10:38:04.020 DEBUG:boto:Path: /vclfcn-44/ 2024-02-01T10:38:04.020 DEBUG:boto:Data: 2024-02-01T10:38:04.020 DEBUG:boto:Headers: {} 2024-02-01T10:38:04.020 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:04.020 DEBUG:boto:Port: 8001 2024-02-01T10:38:04.020 DEBUG:boto:Params: {} 2024-02-01T10:38:04.020 DEBUG:boto:Token: None 2024-02-01T10:38:04.021 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:04 GMT /vclfcn-44/ 2024-02-01T10:38:04.021 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:K8ywy1pu4jlJIoqkqLlvqKRaAM0= 2024-02-01T10:38:04.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:K8ywy1pu4jlJIoqkqLlvqKRaAM0=', 'Content-Length': '0'} 2024-02-01T10:38:04.025 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c87c9a2636e95f1e-0065bb748c-5324-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:04.026 DEBUG:boto:path=/vclfcn-44/ 2024-02-01T10:38:04.026 DEBUG:boto:auth_path=/vclfcn-44/ 2024-02-01T10:38:04.026 DEBUG:boto:Method: HEAD 2024-02-01T10:38:04.026 DEBUG:boto:Path: /vclfcn-44/ 2024-02-01T10:38:04.026 DEBUG:boto:Data: 2024-02-01T10:38:04.026 DEBUG:boto:Headers: {} 2024-02-01T10:38:04.026 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:04.027 DEBUG:boto:Port: 8000 2024-02-01T10:38:04.027 DEBUG:boto:Params: {} 2024-02-01T10:38:04.027 DEBUG:boto:Token: None 2024-02-01T10:38:04.027 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:04 GMT /vclfcn-44/ 2024-02-01T10:38:04.027 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:K8ywy1pu4jlJIoqkqLlvqKRaAM0= 2024-02-01T10:38:04.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:K8ywy1pu4jlJIoqkqLlvqKRaAM0=', 'Content-Length': '0'} 2024-02-01T10:38:04.031 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000021b5fba0de5e83c4-0065bb748c-5004-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:04.031 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:38:04.031 DEBUG:boto:path=/vclfcn-44/ 2024-02-01T10:38:04.032 DEBUG:boto:auth_path=/vclfcn-44/ 2024-02-01T10:38:04.032 DEBUG:boto:path=/vclfcn-44/?versions&max-keys=999 2024-02-01T10:38:04.032 DEBUG:boto:auth_path=/vclfcn-44/?versions&max-keys=999 2024-02-01T10:38:04.032 DEBUG:boto:Method: GET 2024-02-01T10:38:04.033 DEBUG:boto:Path: /vclfcn-44/?versions&max-keys=999 2024-02-01T10:38:04.033 DEBUG:boto:Data: 2024-02-01T10:38:04.033 DEBUG:boto:Headers: {} 2024-02-01T10:38:04.033 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:04.033 DEBUG:boto:Port: 8001 2024-02-01T10:38:04.033 DEBUG:boto:Params: {} 2024-02-01T10:38:04.033 DEBUG:boto:Token: None 2024-02-01T10:38:04.033 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:04 GMT /vclfcn-44/?versions 2024-02-01T10:38:04.034 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7nWrm5tTclA1gOH0X7HaX95X9JI= 2024-02-01T10:38:04.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7nWrm5tTclA1gOH0X7HaX95X9JI=', 'Content-Length': '0'} 2024-02-01T10:38:04.038 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f40a68f9dee4485-0065bb748c-5324-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:04.038 DEBUG:boto:b'vclfcn-44999false' 2024-02-01T10:38:04.039 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:38:04.039 DEBUG:boto:path=/vclfcn-44/ 2024-02-01T10:38:04.039 DEBUG:boto:auth_path=/vclfcn-44/ 2024-02-01T10:38:04.040 DEBUG:boto:path=/vclfcn-44/?versions&max-keys=999 2024-02-01T10:38:04.040 DEBUG:boto:auth_path=/vclfcn-44/?versions&max-keys=999 2024-02-01T10:38:04.040 DEBUG:boto:Method: GET 2024-02-01T10:38:04.040 DEBUG:boto:Path: /vclfcn-44/?versions&max-keys=999 2024-02-01T10:38:04.040 DEBUG:boto:Data: 2024-02-01T10:38:04.041 DEBUG:boto:Headers: {} 2024-02-01T10:38:04.041 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:04.041 DEBUG:boto:Port: 8000 2024-02-01T10:38:04.041 DEBUG:boto:Params: {} 2024-02-01T10:38:04.041 DEBUG:boto:Token: None 2024-02-01T10:38:04.042 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:04 GMT /vclfcn-44/?versions 2024-02-01T10:38:04.042 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7nWrm5tTclA1gOH0X7HaX95X9JI= 2024-02-01T10:38:04.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7nWrm5tTclA1gOH0X7HaX95X9JI=', 'Content-Length': '0'} 2024-02-01T10:38:04.047 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d1f961174eeb1db-0065bb748c-5004-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:04.088 DEBUG:boto:b'vclfcn-44999false' 2024-02-01T10:38:04.088 DEBUG:boto:path=/vclfcn-44/ 2024-02-01T10:38:04.089 DEBUG:boto:auth_path=/vclfcn-44/ 2024-02-01T10:38:04.089 DEBUG:boto:path=/vclfcn-44/?versions&max-keys=999 2024-02-01T10:38:04.089 DEBUG:boto:auth_path=/vclfcn-44/?versions&max-keys=999 2024-02-01T10:38:04.089 DEBUG:boto:Method: GET 2024-02-01T10:38:04.089 DEBUG:boto:Path: /vclfcn-44/?versions&max-keys=999 2024-02-01T10:38:04.089 DEBUG:boto:Data: 2024-02-01T10:38:04.089 DEBUG:boto:Headers: {} 2024-02-01T10:38:04.089 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:04.089 DEBUG:boto:Port: 8001 2024-02-01T10:38:04.090 DEBUG:boto:Params: {} 2024-02-01T10:38:04.090 DEBUG:boto:Token: None 2024-02-01T10:38:04.090 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:04 GMT /vclfcn-44/?versions 2024-02-01T10:38:04.090 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7nWrm5tTclA1gOH0X7HaX95X9JI= 2024-02-01T10:38:04.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7nWrm5tTclA1gOH0X7HaX95X9JI=', 'Content-Length': '0'} 2024-02-01T10:38:04.094 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f7f386494a312b0-0065bb748c-5324-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:04.094 DEBUG:boto:b'vclfcn-44999false' 2024-02-01T10:38:04.095 DEBUG:boto:path=/vclfcn-44/ 2024-02-01T10:38:04.095 DEBUG:boto:auth_path=/vclfcn-44/ 2024-02-01T10:38:04.095 DEBUG:boto:path=/vclfcn-44/?versions&max-keys=999 2024-02-01T10:38:04.095 DEBUG:boto:auth_path=/vclfcn-44/?versions&max-keys=999 2024-02-01T10:38:04.095 DEBUG:boto:Method: GET 2024-02-01T10:38:04.095 DEBUG:boto:Path: /vclfcn-44/?versions&max-keys=999 2024-02-01T10:38:04.095 DEBUG:boto:Data: 2024-02-01T10:38:04.095 DEBUG:boto:Headers: {} 2024-02-01T10:38:04.095 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:04.096 DEBUG:boto:Port: 8000 2024-02-01T10:38:04.096 DEBUG:boto:Params: {} 2024-02-01T10:38:04.096 DEBUG:boto:Token: None 2024-02-01T10:38:04.096 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:04 GMT /vclfcn-44/?versions 2024-02-01T10:38:04.096 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7nWrm5tTclA1gOH0X7HaX95X9JI= 2024-02-01T10:38:04.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7nWrm5tTclA1gOH0X7HaX95X9JI=', 'Content-Length': '0'} 2024-02-01T10:38:04.100 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a29274d9f023c750-0065bb748c-5004-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:04.100 DEBUG:boto:b'vclfcn-44999false' 2024-02-01T10:38:04.101 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-44 zones={test-zone3, test-zone2} 2024-02-01T10:38:04.101 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-44 zones={test-zone1, test-zone3} 2024-02-01T10:38:04.101 DEBUG:boto:path=/vclfcn-44/ 2024-02-01T10:38:04.101 DEBUG:boto:auth_path=/vclfcn-44/ 2024-02-01T10:38:04.102 DEBUG:boto:Method: HEAD 2024-02-01T10:38:04.102 DEBUG:boto:Path: /vclfcn-44/ 2024-02-01T10:38:04.102 DEBUG:boto:Data: 2024-02-01T10:38:04.102 DEBUG:boto:Headers: {} 2024-02-01T10:38:04.102 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:04.102 DEBUG:boto:Port: 8000 2024-02-01T10:38:04.102 DEBUG:boto:Params: {} 2024-02-01T10:38:04.102 DEBUG:boto:Token: None 2024-02-01T10:38:04.103 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:04 GMT /vclfcn-44/ 2024-02-01T10:38:04.103 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:K8ywy1pu4jlJIoqkqLlvqKRaAM0= 2024-02-01T10:38:04.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:K8ywy1pu4jlJIoqkqLlvqKRaAM0=', 'Content-Length': '0'} 2024-02-01T10:38:04.107 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000066266815ae5ec7bc-0065bb748c-5317-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:04.108 DEBUG:boto:path=/vclfcn-44/ 2024-02-01T10:38:04.108 DEBUG:boto:auth_path=/vclfcn-44/ 2024-02-01T10:38:04.108 DEBUG:boto:Method: HEAD 2024-02-01T10:38:04.109 DEBUG:boto:Path: /vclfcn-44/ 2024-02-01T10:38:04.109 DEBUG:boto:Data: 2024-02-01T10:38:04.109 DEBUG:boto:Headers: {} 2024-02-01T10:38:04.109 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:04.109 DEBUG:boto:Port: 8001 2024-02-01T10:38:04.110 DEBUG:boto:Params: {} 2024-02-01T10:38:04.110 DEBUG:boto:Token: None 2024-02-01T10:38:04.110 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:04 GMT /vclfcn-44/ 2024-02-01T10:38:04.110 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:K8ywy1pu4jlJIoqkqLlvqKRaAM0= 2024-02-01T10:38:04.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:K8ywy1pu4jlJIoqkqLlvqKRaAM0=', 'Content-Length': '0'} 2024-02-01T10:38:04.115 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b746af6ba635c267-0065bb748c-5324-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:04.115 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:38:04.115 DEBUG:boto:path=/vclfcn-44/ 2024-02-01T10:38:04.116 DEBUG:boto:auth_path=/vclfcn-44/ 2024-02-01T10:38:04.116 DEBUG:boto:path=/vclfcn-44/?versions&max-keys=999 2024-02-01T10:38:04.117 DEBUG:boto:auth_path=/vclfcn-44/?versions&max-keys=999 2024-02-01T10:38:04.117 DEBUG:boto:Method: GET 2024-02-01T10:38:04.118 DEBUG:boto:Path: /vclfcn-44/?versions&max-keys=999 2024-02-01T10:38:04.118 DEBUG:boto:Data: 2024-02-01T10:38:04.118 DEBUG:boto:Headers: {} 2024-02-01T10:38:04.118 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:04.118 DEBUG:boto:Port: 8000 2024-02-01T10:38:04.119 DEBUG:boto:Params: {} 2024-02-01T10:38:04.119 DEBUG:boto:Token: None 2024-02-01T10:38:04.119 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:04 GMT /vclfcn-44/?versions 2024-02-01T10:38:04.120 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7nWrm5tTclA1gOH0X7HaX95X9JI= 2024-02-01T10:38:04.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7nWrm5tTclA1gOH0X7HaX95X9JI=', 'Content-Length': '0'} 2024-02-01T10:38:04.124 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078141060fa302273-0065bb748c-5317-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:04.124 DEBUG:boto:b'vclfcn-44999false' 2024-02-01T10:38:04.125 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:38:04.125 DEBUG:boto:path=/vclfcn-44/ 2024-02-01T10:38:04.125 DEBUG:boto:auth_path=/vclfcn-44/ 2024-02-01T10:38:04.125 DEBUG:boto:path=/vclfcn-44/?versions&max-keys=999 2024-02-01T10:38:04.126 DEBUG:boto:auth_path=/vclfcn-44/?versions&max-keys=999 2024-02-01T10:38:04.126 DEBUG:boto:Method: GET 2024-02-01T10:38:04.126 DEBUG:boto:Path: /vclfcn-44/?versions&max-keys=999 2024-02-01T10:38:04.126 DEBUG:boto:Data: 2024-02-01T10:38:04.126 DEBUG:boto:Headers: {} 2024-02-01T10:38:04.127 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:04.127 DEBUG:boto:Port: 8001 2024-02-01T10:38:04.127 DEBUG:boto:Params: {} 2024-02-01T10:38:04.128 DEBUG:boto:Token: None 2024-02-01T10:38:04.128 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:04 GMT /vclfcn-44/?versions 2024-02-01T10:38:04.128 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7nWrm5tTclA1gOH0X7HaX95X9JI= 2024-02-01T10:38:04.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7nWrm5tTclA1gOH0X7HaX95X9JI=', 'Content-Length': '0'} 2024-02-01T10:38:04.133 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041b6aa3777e8a2e5-0065bb748c-5324-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:04.176 DEBUG:boto:b'vclfcn-44999false' 2024-02-01T10:38:04.177 DEBUG:boto:path=/vclfcn-44/ 2024-02-01T10:38:04.177 DEBUG:boto:auth_path=/vclfcn-44/ 2024-02-01T10:38:04.177 DEBUG:boto:path=/vclfcn-44/?versions&max-keys=999 2024-02-01T10:38:04.178 DEBUG:boto:auth_path=/vclfcn-44/?versions&max-keys=999 2024-02-01T10:38:04.178 DEBUG:boto:Method: GET 2024-02-01T10:38:04.178 DEBUG:boto:Path: /vclfcn-44/?versions&max-keys=999 2024-02-01T10:38:04.178 DEBUG:boto:Data: 2024-02-01T10:38:04.178 DEBUG:boto:Headers: {} 2024-02-01T10:38:04.179 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:04.179 DEBUG:boto:Port: 8000 2024-02-01T10:38:04.180 DEBUG:boto:Params: {} 2024-02-01T10:38:04.180 DEBUG:boto:Token: None 2024-02-01T10:38:04.180 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:04 GMT /vclfcn-44/?versions 2024-02-01T10:38:04.180 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7nWrm5tTclA1gOH0X7HaX95X9JI= 2024-02-01T10:38:04.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7nWrm5tTclA1gOH0X7HaX95X9JI=', 'Content-Length': '0'} 2024-02-01T10:38:04.185 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed53fd4114e33d5c-0065bb748c-5317-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:04.185 DEBUG:boto:b'vclfcn-44999false' 2024-02-01T10:38:04.185 DEBUG:boto:path=/vclfcn-44/ 2024-02-01T10:38:04.185 DEBUG:boto:auth_path=/vclfcn-44/ 2024-02-01T10:38:04.186 DEBUG:boto:path=/vclfcn-44/?versions&max-keys=999 2024-02-01T10:38:04.186 DEBUG:boto:auth_path=/vclfcn-44/?versions&max-keys=999 2024-02-01T10:38:04.186 DEBUG:boto:Method: GET 2024-02-01T10:38:04.186 DEBUG:boto:Path: /vclfcn-44/?versions&max-keys=999 2024-02-01T10:38:04.186 DEBUG:boto:Data: 2024-02-01T10:38:04.186 DEBUG:boto:Headers: {} 2024-02-01T10:38:04.186 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:04.186 DEBUG:boto:Port: 8001 2024-02-01T10:38:04.186 DEBUG:boto:Params: {} 2024-02-01T10:38:04.186 DEBUG:boto:Token: None 2024-02-01T10:38:04.187 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:04 GMT /vclfcn-44/?versions 2024-02-01T10:38:04.187 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7nWrm5tTclA1gOH0X7HaX95X9JI= 2024-02-01T10:38:04.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7nWrm5tTclA1gOH0X7HaX95X9JI=', 'Content-Length': '0'} 2024-02-01T10:38:04.191 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000059bed34817484d76-0065bb748c-5324-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:04.191 DEBUG:boto:b'vclfcn-44999false' 2024-02-01T10:38:04.191 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-44 zones={test-zone1, test-zone3} 2024-02-01T10:38:04.192 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-44 zones={test-zone2, test-zone3} 2024-02-01T10:38:04.192 DEBUG:boto:path=/vclfcn-44/ 2024-02-01T10:38:04.192 DEBUG:boto:auth_path=/vclfcn-44/ 2024-02-01T10:38:04.192 DEBUG:boto:Method: HEAD 2024-02-01T10:38:04.192 DEBUG:boto:Path: /vclfcn-44/ 2024-02-01T10:38:04.192 DEBUG:boto:Data: 2024-02-01T10:38:04.192 DEBUG:boto:Headers: {} 2024-02-01T10:38:04.192 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:04.192 DEBUG:boto:Port: 8000 2024-02-01T10:38:04.192 DEBUG:boto:Params: {} 2024-02-01T10:38:04.193 DEBUG:boto:Token: None 2024-02-01T10:38:04.193 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:04 GMT /vclfcn-44/ 2024-02-01T10:38:04.193 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:K8ywy1pu4jlJIoqkqLlvqKRaAM0= 2024-02-01T10:38:04.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:K8ywy1pu4jlJIoqkqLlvqKRaAM0=', 'Content-Length': '0'} 2024-02-01T10:38:04.197 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000de2323eab18d1746-0065bb748c-5004-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:04.197 DEBUG:boto:path=/vclfcn-44/ 2024-02-01T10:38:04.197 DEBUG:boto:auth_path=/vclfcn-44/ 2024-02-01T10:38:04.197 DEBUG:boto:Method: HEAD 2024-02-01T10:38:04.197 DEBUG:boto:Path: /vclfcn-44/ 2024-02-01T10:38:04.197 DEBUG:boto:Data: 2024-02-01T10:38:04.197 DEBUG:boto:Headers: {} 2024-02-01T10:38:04.198 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:04.198 DEBUG:boto:Port: 8001 2024-02-01T10:38:04.198 DEBUG:boto:Params: {} 2024-02-01T10:38:04.198 DEBUG:boto:Token: None 2024-02-01T10:38:04.198 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:04 GMT /vclfcn-44/ 2024-02-01T10:38:04.198 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:K8ywy1pu4jlJIoqkqLlvqKRaAM0= 2024-02-01T10:38:04.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:K8ywy1pu4jlJIoqkqLlvqKRaAM0=', 'Content-Length': '0'} 2024-02-01T10:38:04.202 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ddf5cdad7150b0d0-0065bb748c-5324-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:04.202 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:38:04.202 DEBUG:boto:path=/vclfcn-44/ 2024-02-01T10:38:04.203 DEBUG:boto:auth_path=/vclfcn-44/ 2024-02-01T10:38:04.203 DEBUG:boto:path=/vclfcn-44/?versions&max-keys=999 2024-02-01T10:38:04.203 DEBUG:boto:auth_path=/vclfcn-44/?versions&max-keys=999 2024-02-01T10:38:04.203 DEBUG:boto:Method: GET 2024-02-01T10:38:04.203 DEBUG:boto:Path: /vclfcn-44/?versions&max-keys=999 2024-02-01T10:38:04.203 DEBUG:boto:Data: 2024-02-01T10:38:04.203 DEBUG:boto:Headers: {} 2024-02-01T10:38:04.203 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:04.203 DEBUG:boto:Port: 8000 2024-02-01T10:38:04.203 DEBUG:boto:Params: {} 2024-02-01T10:38:04.203 DEBUG:boto:Token: None 2024-02-01T10:38:04.204 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:04 GMT /vclfcn-44/?versions 2024-02-01T10:38:04.204 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7nWrm5tTclA1gOH0X7HaX95X9JI= 2024-02-01T10:38:04.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7nWrm5tTclA1gOH0X7HaX95X9JI=', 'Content-Length': '0'} 2024-02-01T10:38:04.208 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ba66f7a6929affc-0065bb748c-5004-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:04.208 DEBUG:boto:b'vclfcn-44999false' 2024-02-01T10:38:04.209 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:38:04.209 DEBUG:boto:path=/vclfcn-44/ 2024-02-01T10:38:04.209 DEBUG:boto:auth_path=/vclfcn-44/ 2024-02-01T10:38:04.209 DEBUG:boto:path=/vclfcn-44/?versions&max-keys=999 2024-02-01T10:38:04.209 DEBUG:boto:auth_path=/vclfcn-44/?versions&max-keys=999 2024-02-01T10:38:04.210 DEBUG:boto:Method: GET 2024-02-01T10:38:04.210 DEBUG:boto:Path: /vclfcn-44/?versions&max-keys=999 2024-02-01T10:38:04.210 DEBUG:boto:Data: 2024-02-01T10:38:04.211 DEBUG:boto:Headers: {} 2024-02-01T10:38:04.211 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:04.211 DEBUG:boto:Port: 8001 2024-02-01T10:38:04.212 DEBUG:boto:Params: {} 2024-02-01T10:38:04.212 DEBUG:boto:Token: None 2024-02-01T10:38:04.213 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:04 GMT /vclfcn-44/?versions 2024-02-01T10:38:04.213 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7nWrm5tTclA1gOH0X7HaX95X9JI= 2024-02-01T10:38:04.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7nWrm5tTclA1gOH0X7HaX95X9JI=', 'Content-Length': '0'} 2024-02-01T10:38:04.217 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000785743287098e6e3-0065bb748c-5324-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:04.260 DEBUG:boto:b'vclfcn-44999false' 2024-02-01T10:38:04.261 DEBUG:boto:path=/vclfcn-44/ 2024-02-01T10:38:04.261 DEBUG:boto:auth_path=/vclfcn-44/ 2024-02-01T10:38:04.261 DEBUG:boto:path=/vclfcn-44/?versions&max-keys=999 2024-02-01T10:38:04.261 DEBUG:boto:auth_path=/vclfcn-44/?versions&max-keys=999 2024-02-01T10:38:04.261 DEBUG:boto:Method: GET 2024-02-01T10:38:04.261 DEBUG:boto:Path: /vclfcn-44/?versions&max-keys=999 2024-02-01T10:38:04.261 DEBUG:boto:Data: 2024-02-01T10:38:04.262 DEBUG:boto:Headers: {} 2024-02-01T10:38:04.262 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:04.262 DEBUG:boto:Port: 8000 2024-02-01T10:38:04.262 DEBUG:boto:Params: {} 2024-02-01T10:38:04.262 DEBUG:boto:Token: None 2024-02-01T10:38:04.262 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:04 GMT /vclfcn-44/?versions 2024-02-01T10:38:04.262 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7nWrm5tTclA1gOH0X7HaX95X9JI= 2024-02-01T10:38:04.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7nWrm5tTclA1gOH0X7HaX95X9JI=', 'Content-Length': '0'} 2024-02-01T10:38:04.267 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a4d0453437a05346-0065bb748c-5004-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:04.267 DEBUG:boto:b'vclfcn-44999false' 2024-02-01T10:38:04.268 DEBUG:boto:path=/vclfcn-44/ 2024-02-01T10:38:04.268 DEBUG:boto:auth_path=/vclfcn-44/ 2024-02-01T10:38:04.268 DEBUG:boto:path=/vclfcn-44/?versions&max-keys=999 2024-02-01T10:38:04.268 DEBUG:boto:auth_path=/vclfcn-44/?versions&max-keys=999 2024-02-01T10:38:04.268 DEBUG:boto:Method: GET 2024-02-01T10:38:04.268 DEBUG:boto:Path: /vclfcn-44/?versions&max-keys=999 2024-02-01T10:38:04.269 DEBUG:boto:Data: 2024-02-01T10:38:04.269 DEBUG:boto:Headers: {} 2024-02-01T10:38:04.269 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:04.269 DEBUG:boto:Port: 8001 2024-02-01T10:38:04.269 DEBUG:boto:Params: {} 2024-02-01T10:38:04.269 DEBUG:boto:Token: None 2024-02-01T10:38:04.269 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:04 GMT /vclfcn-44/?versions 2024-02-01T10:38:04.269 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7nWrm5tTclA1gOH0X7HaX95X9JI= 2024-02-01T10:38:04.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7nWrm5tTclA1gOH0X7HaX95X9JI=', 'Content-Length': '0'} 2024-02-01T10:38:04.274 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040e65ec3a516b82d-0065bb748c-5324-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:04.274 DEBUG:boto:b'vclfcn-44999false' 2024-02-01T10:38:04.274 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-44 zones={test-zone2, test-zone3} 2024-02-01T10:38:04.274 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-45 zones={test-zone2, test-zone1} 2024-02-01T10:38:04.275 DEBUG:boto:path=/vclfcn-45/ 2024-02-01T10:38:04.275 DEBUG:boto:auth_path=/vclfcn-45/ 2024-02-01T10:38:04.275 DEBUG:boto:Method: HEAD 2024-02-01T10:38:04.275 DEBUG:boto:Path: /vclfcn-45/ 2024-02-01T10:38:04.275 DEBUG:boto:Data: 2024-02-01T10:38:04.275 DEBUG:boto:Headers: {} 2024-02-01T10:38:04.275 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:04.275 DEBUG:boto:Port: 8000 2024-02-01T10:38:04.275 DEBUG:boto:Params: {} 2024-02-01T10:38:04.275 DEBUG:boto:Token: None 2024-02-01T10:38:04.276 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:04 GMT /vclfcn-45/ 2024-02-01T10:38:04.276 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0NbZWhxm6aLIufEnHM6l03UCdU4= 2024-02-01T10:38:04.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0NbZWhxm6aLIufEnHM6l03UCdU4=', 'Content-Length': '0'} 2024-02-01T10:38:04.280 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000051faa31773e985fa-0065bb748c-5004-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:04.280 DEBUG:boto:path=/vclfcn-45/ 2024-02-01T10:38:04.280 DEBUG:boto:auth_path=/vclfcn-45/ 2024-02-01T10:38:04.281 DEBUG:boto:Method: HEAD 2024-02-01T10:38:04.281 DEBUG:boto:Path: /vclfcn-45/ 2024-02-01T10:38:04.281 DEBUG:boto:Data: 2024-02-01T10:38:04.281 DEBUG:boto:Headers: {} 2024-02-01T10:38:04.281 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:04.281 DEBUG:boto:Port: 8000 2024-02-01T10:38:04.281 DEBUG:boto:Params: {} 2024-02-01T10:38:04.281 DEBUG:boto:Token: None 2024-02-01T10:38:04.281 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:04 GMT /vclfcn-45/ 2024-02-01T10:38:04.282 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0NbZWhxm6aLIufEnHM6l03UCdU4= 2024-02-01T10:38:04.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0NbZWhxm6aLIufEnHM6l03UCdU4=', 'Content-Length': '0'} 2024-02-01T10:38:04.286 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006f0b06a7bd995d0f-0065bb748c-5317-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:04.286 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:38:04.286 DEBUG:boto:path=/vclfcn-45/ 2024-02-01T10:38:04.286 DEBUG:boto:auth_path=/vclfcn-45/ 2024-02-01T10:38:04.286 DEBUG:boto:path=/vclfcn-45/?versions&max-keys=999 2024-02-01T10:38:04.286 DEBUG:boto:auth_path=/vclfcn-45/?versions&max-keys=999 2024-02-01T10:38:04.287 DEBUG:boto:Method: GET 2024-02-01T10:38:04.287 DEBUG:boto:Path: /vclfcn-45/?versions&max-keys=999 2024-02-01T10:38:04.287 DEBUG:boto:Data: 2024-02-01T10:38:04.287 DEBUG:boto:Headers: {} 2024-02-01T10:38:04.287 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:04.287 DEBUG:boto:Port: 8000 2024-02-01T10:38:04.287 DEBUG:boto:Params: {} 2024-02-01T10:38:04.287 DEBUG:boto:Token: None 2024-02-01T10:38:04.288 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:04 GMT /vclfcn-45/?versions 2024-02-01T10:38:04.288 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:a9t1YSSWOhHSzmOYoZoPdWCSDII= 2024-02-01T10:38:04.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:a9t1YSSWOhHSzmOYoZoPdWCSDII=', 'Content-Length': '0'} 2024-02-01T10:38:04.292 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b401a051dd851388-0065bb748c-5004-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:04.332 DEBUG:boto:b'vclfcn-45999false' 2024-02-01T10:38:04.332 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:38:04.333 DEBUG:boto:path=/vclfcn-45/ 2024-02-01T10:38:04.333 DEBUG:boto:auth_path=/vclfcn-45/ 2024-02-01T10:38:04.333 DEBUG:boto:path=/vclfcn-45/?versions&max-keys=999 2024-02-01T10:38:04.333 DEBUG:boto:auth_path=/vclfcn-45/?versions&max-keys=999 2024-02-01T10:38:04.333 DEBUG:boto:Method: GET 2024-02-01T10:38:04.333 DEBUG:boto:Path: /vclfcn-45/?versions&max-keys=999 2024-02-01T10:38:04.333 DEBUG:boto:Data: 2024-02-01T10:38:04.333 DEBUG:boto:Headers: {} 2024-02-01T10:38:04.333 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:04.333 DEBUG:boto:Port: 8000 2024-02-01T10:38:04.334 DEBUG:boto:Params: {} 2024-02-01T10:38:04.334 DEBUG:boto:Token: None 2024-02-01T10:38:04.334 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:04 GMT /vclfcn-45/?versions 2024-02-01T10:38:04.334 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:a9t1YSSWOhHSzmOYoZoPdWCSDII= 2024-02-01T10:38:04.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:a9t1YSSWOhHSzmOYoZoPdWCSDII=', 'Content-Length': '0'} 2024-02-01T10:38:04.339 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000016b729dbcb5e068b-0065bb748c-5317-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:04.339 DEBUG:boto:b'vclfcn-45999false' 2024-02-01T10:38:04.340 DEBUG:boto:path=/vclfcn-45/ 2024-02-01T10:38:04.340 DEBUG:boto:auth_path=/vclfcn-45/ 2024-02-01T10:38:04.340 DEBUG:boto:path=/vclfcn-45/?versions&max-keys=999 2024-02-01T10:38:04.340 DEBUG:boto:auth_path=/vclfcn-45/?versions&max-keys=999 2024-02-01T10:38:04.341 DEBUG:boto:Method: GET 2024-02-01T10:38:04.341 DEBUG:boto:Path: /vclfcn-45/?versions&max-keys=999 2024-02-01T10:38:04.341 DEBUG:boto:Data: 2024-02-01T10:38:04.341 DEBUG:boto:Headers: {} 2024-02-01T10:38:04.341 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:04.341 DEBUG:boto:Port: 8000 2024-02-01T10:38:04.341 DEBUG:boto:Params: {} 2024-02-01T10:38:04.341 DEBUG:boto:Token: None 2024-02-01T10:38:04.341 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:04 GMT /vclfcn-45/?versions 2024-02-01T10:38:04.341 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:a9t1YSSWOhHSzmOYoZoPdWCSDII= 2024-02-01T10:38:04.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:a9t1YSSWOhHSzmOYoZoPdWCSDII=', 'Content-Length': '0'} 2024-02-01T10:38:04.346 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000049e68b0b55cc2493-0065bb748c-5004-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:04.346 DEBUG:boto:b'vclfcn-45999false' 2024-02-01T10:38:04.346 DEBUG:boto:path=/vclfcn-45/ 2024-02-01T10:38:04.346 DEBUG:boto:auth_path=/vclfcn-45/ 2024-02-01T10:38:04.347 DEBUG:boto:path=/vclfcn-45/?versions&max-keys=999 2024-02-01T10:38:04.347 DEBUG:boto:auth_path=/vclfcn-45/?versions&max-keys=999 2024-02-01T10:38:04.347 DEBUG:boto:Method: GET 2024-02-01T10:38:04.347 DEBUG:boto:Path: /vclfcn-45/?versions&max-keys=999 2024-02-01T10:38:04.347 DEBUG:boto:Data: 2024-02-01T10:38:04.347 DEBUG:boto:Headers: {} 2024-02-01T10:38:04.347 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:04.347 DEBUG:boto:Port: 8000 2024-02-01T10:38:04.347 DEBUG:boto:Params: {} 2024-02-01T10:38:04.347 DEBUG:boto:Token: None 2024-02-01T10:38:04.348 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:04 GMT /vclfcn-45/?versions 2024-02-01T10:38:04.348 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:a9t1YSSWOhHSzmOYoZoPdWCSDII= 2024-02-01T10:38:04.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:a9t1YSSWOhHSzmOYoZoPdWCSDII=', 'Content-Length': '0'} 2024-02-01T10:38:04.352 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4ea201e5a204f33-0065bb748c-5317-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:04.392 DEBUG:boto:b'vclfcn-45999false' 2024-02-01T10:38:04.392 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-45 zones={test-zone2, test-zone1} 2024-02-01T10:38:04.393 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-45 zones={test-zone3, test-zone1} 2024-02-01T10:38:04.393 DEBUG:boto:path=/vclfcn-45/ 2024-02-01T10:38:04.393 DEBUG:boto:auth_path=/vclfcn-45/ 2024-02-01T10:38:04.393 DEBUG:boto:Method: HEAD 2024-02-01T10:38:04.393 DEBUG:boto:Path: /vclfcn-45/ 2024-02-01T10:38:04.393 DEBUG:boto:Data: 2024-02-01T10:38:04.393 DEBUG:boto:Headers: {} 2024-02-01T10:38:04.393 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:04.393 DEBUG:boto:Port: 8001 2024-02-01T10:38:04.393 DEBUG:boto:Params: {} 2024-02-01T10:38:04.394 DEBUG:boto:Token: None 2024-02-01T10:38:04.394 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:04 GMT /vclfcn-45/ 2024-02-01T10:38:04.394 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0NbZWhxm6aLIufEnHM6l03UCdU4= 2024-02-01T10:38:04.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0NbZWhxm6aLIufEnHM6l03UCdU4=', 'Content-Length': '0'} 2024-02-01T10:38:04.398 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ea3f75172f46600f-0065bb748c-5324-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:04.399 DEBUG:boto:path=/vclfcn-45/ 2024-02-01T10:38:04.399 DEBUG:boto:auth_path=/vclfcn-45/ 2024-02-01T10:38:04.399 DEBUG:boto:Method: HEAD 2024-02-01T10:38:04.399 DEBUG:boto:Path: /vclfcn-45/ 2024-02-01T10:38:04.399 DEBUG:boto:Data: 2024-02-01T10:38:04.399 DEBUG:boto:Headers: {} 2024-02-01T10:38:04.399 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:04.400 DEBUG:boto:Port: 8000 2024-02-01T10:38:04.400 DEBUG:boto:Params: {} 2024-02-01T10:38:04.400 DEBUG:boto:Token: None 2024-02-01T10:38:04.400 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:04 GMT /vclfcn-45/ 2024-02-01T10:38:04.401 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0NbZWhxm6aLIufEnHM6l03UCdU4= 2024-02-01T10:38:04.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0NbZWhxm6aLIufEnHM6l03UCdU4=', 'Content-Length': '0'} 2024-02-01T10:38:04.405 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000961ad92520e31934-0065bb748c-5317-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:04.405 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:38:04.405 DEBUG:boto:path=/vclfcn-45/ 2024-02-01T10:38:04.406 DEBUG:boto:auth_path=/vclfcn-45/ 2024-02-01T10:38:04.406 DEBUG:boto:path=/vclfcn-45/?versions&max-keys=999 2024-02-01T10:38:04.406 DEBUG:boto:auth_path=/vclfcn-45/?versions&max-keys=999 2024-02-01T10:38:04.406 DEBUG:boto:Method: GET 2024-02-01T10:38:04.406 DEBUG:boto:Path: /vclfcn-45/?versions&max-keys=999 2024-02-01T10:38:04.406 DEBUG:boto:Data: 2024-02-01T10:38:04.406 DEBUG:boto:Headers: {} 2024-02-01T10:38:04.406 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:04.406 DEBUG:boto:Port: 8001 2024-02-01T10:38:04.406 DEBUG:boto:Params: {} 2024-02-01T10:38:04.407 DEBUG:boto:Token: None 2024-02-01T10:38:04.407 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:04 GMT /vclfcn-45/?versions 2024-02-01T10:38:04.407 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:a9t1YSSWOhHSzmOYoZoPdWCSDII= 2024-02-01T10:38:04.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:a9t1YSSWOhHSzmOYoZoPdWCSDII=', 'Content-Length': '0'} 2024-02-01T10:38:04.411 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac539487952d039e-0065bb748c-5324-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:04.411 DEBUG:boto:b'vclfcn-45999false' 2024-02-01T10:38:04.412 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:38:04.412 DEBUG:boto:path=/vclfcn-45/ 2024-02-01T10:38:04.412 DEBUG:boto:auth_path=/vclfcn-45/ 2024-02-01T10:38:04.412 DEBUG:boto:path=/vclfcn-45/?versions&max-keys=999 2024-02-01T10:38:04.412 DEBUG:boto:auth_path=/vclfcn-45/?versions&max-keys=999 2024-02-01T10:38:04.412 DEBUG:boto:Method: GET 2024-02-01T10:38:04.412 DEBUG:boto:Path: /vclfcn-45/?versions&max-keys=999 2024-02-01T10:38:04.412 DEBUG:boto:Data: 2024-02-01T10:38:04.413 DEBUG:boto:Headers: {} 2024-02-01T10:38:04.413 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:04.413 DEBUG:boto:Port: 8000 2024-02-01T10:38:04.413 DEBUG:boto:Params: {} 2024-02-01T10:38:04.413 DEBUG:boto:Token: None 2024-02-01T10:38:04.413 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:04 GMT /vclfcn-45/?versions 2024-02-01T10:38:04.414 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:a9t1YSSWOhHSzmOYoZoPdWCSDII= 2024-02-01T10:38:04.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:a9t1YSSWOhHSzmOYoZoPdWCSDII=', 'Content-Length': '0'} 2024-02-01T10:38:04.419 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001fe7bab94d89487f-0065bb748c-5317-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:04.420 DEBUG:boto:b'vclfcn-45999false' 2024-02-01T10:38:04.420 DEBUG:boto:path=/vclfcn-45/ 2024-02-01T10:38:04.420 DEBUG:boto:auth_path=/vclfcn-45/ 2024-02-01T10:38:04.420 DEBUG:boto:path=/vclfcn-45/?versions&max-keys=999 2024-02-01T10:38:04.420 DEBUG:boto:auth_path=/vclfcn-45/?versions&max-keys=999 2024-02-01T10:38:04.420 DEBUG:boto:Method: GET 2024-02-01T10:38:04.421 DEBUG:boto:Path: /vclfcn-45/?versions&max-keys=999 2024-02-01T10:38:04.421 DEBUG:boto:Data: 2024-02-01T10:38:04.421 DEBUG:boto:Headers: {} 2024-02-01T10:38:04.421 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:04.421 DEBUG:boto:Port: 8001 2024-02-01T10:38:04.421 DEBUG:boto:Params: {} 2024-02-01T10:38:04.421 DEBUG:boto:Token: None 2024-02-01T10:38:04.421 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:04 GMT /vclfcn-45/?versions 2024-02-01T10:38:04.421 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:a9t1YSSWOhHSzmOYoZoPdWCSDII= 2024-02-01T10:38:04.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:a9t1YSSWOhHSzmOYoZoPdWCSDII=', 'Content-Length': '0'} 2024-02-01T10:38:04.426 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0c80b7b17b698c9-0065bb748c-5324-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:04.468 DEBUG:boto:b'vclfcn-45999false' 2024-02-01T10:38:04.468 DEBUG:boto:path=/vclfcn-45/ 2024-02-01T10:38:04.469 DEBUG:boto:auth_path=/vclfcn-45/ 2024-02-01T10:38:04.469 DEBUG:boto:path=/vclfcn-45/?versions&max-keys=999 2024-02-01T10:38:04.469 DEBUG:boto:auth_path=/vclfcn-45/?versions&max-keys=999 2024-02-01T10:38:04.469 DEBUG:boto:Method: GET 2024-02-01T10:38:04.469 DEBUG:boto:Path: /vclfcn-45/?versions&max-keys=999 2024-02-01T10:38:04.469 DEBUG:boto:Data: 2024-02-01T10:38:04.469 DEBUG:boto:Headers: {} 2024-02-01T10:38:04.469 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:04.469 DEBUG:boto:Port: 8000 2024-02-01T10:38:04.469 DEBUG:boto:Params: {} 2024-02-01T10:38:04.470 DEBUG:boto:Token: None 2024-02-01T10:38:04.470 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:04 GMT /vclfcn-45/?versions 2024-02-01T10:38:04.470 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:a9t1YSSWOhHSzmOYoZoPdWCSDII= 2024-02-01T10:38:04.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:a9t1YSSWOhHSzmOYoZoPdWCSDII=', 'Content-Length': '0'} 2024-02-01T10:38:04.474 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000656dc584fdcfd7ba-0065bb748c-5317-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:04.474 DEBUG:boto:b'vclfcn-45999false' 2024-02-01T10:38:04.475 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-45 zones={test-zone3, test-zone1} 2024-02-01T10:38:04.475 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-45 zones={test-zone1, test-zone2} 2024-02-01T10:38:04.475 DEBUG:boto:path=/vclfcn-45/ 2024-02-01T10:38:04.475 DEBUG:boto:auth_path=/vclfcn-45/ 2024-02-01T10:38:04.475 DEBUG:boto:Method: HEAD 2024-02-01T10:38:04.475 DEBUG:boto:Path: /vclfcn-45/ 2024-02-01T10:38:04.475 DEBUG:boto:Data: 2024-02-01T10:38:04.475 DEBUG:boto:Headers: {} 2024-02-01T10:38:04.476 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:04.476 DEBUG:boto:Port: 8000 2024-02-01T10:38:04.476 DEBUG:boto:Params: {} 2024-02-01T10:38:04.476 DEBUG:boto:Token: None 2024-02-01T10:38:04.476 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:04 GMT /vclfcn-45/ 2024-02-01T10:38:04.476 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0NbZWhxm6aLIufEnHM6l03UCdU4= 2024-02-01T10:38:04.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0NbZWhxm6aLIufEnHM6l03UCdU4=', 'Content-Length': '0'} 2024-02-01T10:38:04.480 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006b16aed110053180-0065bb748c-5317-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:04.480 DEBUG:boto:path=/vclfcn-45/ 2024-02-01T10:38:04.480 DEBUG:boto:auth_path=/vclfcn-45/ 2024-02-01T10:38:04.481 DEBUG:boto:Method: HEAD 2024-02-01T10:38:04.481 DEBUG:boto:Path: /vclfcn-45/ 2024-02-01T10:38:04.481 DEBUG:boto:Data: 2024-02-01T10:38:04.481 DEBUG:boto:Headers: {} 2024-02-01T10:38:04.481 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:04.481 DEBUG:boto:Port: 8000 2024-02-01T10:38:04.481 DEBUG:boto:Params: {} 2024-02-01T10:38:04.481 DEBUG:boto:Token: None 2024-02-01T10:38:04.482 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:04 GMT /vclfcn-45/ 2024-02-01T10:38:04.483 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0NbZWhxm6aLIufEnHM6l03UCdU4= 2024-02-01T10:38:04.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0NbZWhxm6aLIufEnHM6l03UCdU4=', 'Content-Length': '0'} 2024-02-01T10:38:04.488 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000047e075b8a320c127-0065bb748c-5004-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:04.488 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:38:04.489 DEBUG:boto:path=/vclfcn-45/ 2024-02-01T10:38:04.489 DEBUG:boto:auth_path=/vclfcn-45/ 2024-02-01T10:38:04.489 DEBUG:boto:path=/vclfcn-45/?versions&max-keys=999 2024-02-01T10:38:04.489 DEBUG:boto:auth_path=/vclfcn-45/?versions&max-keys=999 2024-02-01T10:38:04.489 DEBUG:boto:Method: GET 2024-02-01T10:38:04.489 DEBUG:boto:Path: /vclfcn-45/?versions&max-keys=999 2024-02-01T10:38:04.489 DEBUG:boto:Data: 2024-02-01T10:38:04.489 DEBUG:boto:Headers: {} 2024-02-01T10:38:04.489 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:04.489 DEBUG:boto:Port: 8000 2024-02-01T10:38:04.489 DEBUG:boto:Params: {} 2024-02-01T10:38:04.490 DEBUG:boto:Token: None 2024-02-01T10:38:04.490 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:04 GMT /vclfcn-45/?versions 2024-02-01T10:38:04.490 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:a9t1YSSWOhHSzmOYoZoPdWCSDII= 2024-02-01T10:38:04.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:a9t1YSSWOhHSzmOYoZoPdWCSDII=', 'Content-Length': '0'} 2024-02-01T10:38:04.494 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c812b98f6a79cc4-0065bb748c-5317-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:04.536 DEBUG:boto:b'vclfcn-45999false' 2024-02-01T10:38:04.537 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:38:04.537 DEBUG:boto:path=/vclfcn-45/ 2024-02-01T10:38:04.537 DEBUG:boto:auth_path=/vclfcn-45/ 2024-02-01T10:38:04.537 DEBUG:boto:path=/vclfcn-45/?versions&max-keys=999 2024-02-01T10:38:04.537 DEBUG:boto:auth_path=/vclfcn-45/?versions&max-keys=999 2024-02-01T10:38:04.537 DEBUG:boto:Method: GET 2024-02-01T10:38:04.538 DEBUG:boto:Path: /vclfcn-45/?versions&max-keys=999 2024-02-01T10:38:04.538 DEBUG:boto:Data: 2024-02-01T10:38:04.538 DEBUG:boto:Headers: {} 2024-02-01T10:38:04.538 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:04.538 DEBUG:boto:Port: 8000 2024-02-01T10:38:04.538 DEBUG:boto:Params: {} 2024-02-01T10:38:04.538 DEBUG:boto:Token: None 2024-02-01T10:38:04.538 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:04 GMT /vclfcn-45/?versions 2024-02-01T10:38:04.538 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:a9t1YSSWOhHSzmOYoZoPdWCSDII= 2024-02-01T10:38:04.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:a9t1YSSWOhHSzmOYoZoPdWCSDII=', 'Content-Length': '0'} 2024-02-01T10:38:04.543 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000009ab689d8216283-0065bb748c-5004-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:04.544 DEBUG:boto:b'vclfcn-45999false' 2024-02-01T10:38:04.544 DEBUG:boto:path=/vclfcn-45/ 2024-02-01T10:38:04.544 DEBUG:boto:auth_path=/vclfcn-45/ 2024-02-01T10:38:04.544 DEBUG:boto:path=/vclfcn-45/?versions&max-keys=999 2024-02-01T10:38:04.544 DEBUG:boto:auth_path=/vclfcn-45/?versions&max-keys=999 2024-02-01T10:38:04.544 DEBUG:boto:Method: GET 2024-02-01T10:38:04.544 DEBUG:boto:Path: /vclfcn-45/?versions&max-keys=999 2024-02-01T10:38:04.544 DEBUG:boto:Data: 2024-02-01T10:38:04.545 DEBUG:boto:Headers: {} 2024-02-01T10:38:04.545 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:04.545 DEBUG:boto:Port: 8000 2024-02-01T10:38:04.545 DEBUG:boto:Params: {} 2024-02-01T10:38:04.545 DEBUG:boto:Token: None 2024-02-01T10:38:04.545 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:04 GMT /vclfcn-45/?versions 2024-02-01T10:38:04.545 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:a9t1YSSWOhHSzmOYoZoPdWCSDII= 2024-02-01T10:38:04.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:a9t1YSSWOhHSzmOYoZoPdWCSDII=', 'Content-Length': '0'} 2024-02-01T10:38:04.550 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058fe00d333d9d984-0065bb748c-5317-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:04.550 DEBUG:boto:b'vclfcn-45999false' 2024-02-01T10:38:04.550 DEBUG:boto:path=/vclfcn-45/ 2024-02-01T10:38:04.550 DEBUG:boto:auth_path=/vclfcn-45/ 2024-02-01T10:38:04.551 DEBUG:boto:path=/vclfcn-45/?versions&max-keys=999 2024-02-01T10:38:04.551 DEBUG:boto:auth_path=/vclfcn-45/?versions&max-keys=999 2024-02-01T10:38:04.551 DEBUG:boto:Method: GET 2024-02-01T10:38:04.551 DEBUG:boto:Path: /vclfcn-45/?versions&max-keys=999 2024-02-01T10:38:04.552 DEBUG:boto:Data: 2024-02-01T10:38:04.552 DEBUG:boto:Headers: {} 2024-02-01T10:38:04.552 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:04.552 DEBUG:boto:Port: 8000 2024-02-01T10:38:04.552 DEBUG:boto:Params: {} 2024-02-01T10:38:04.552 DEBUG:boto:Token: None 2024-02-01T10:38:04.552 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:04 GMT /vclfcn-45/?versions 2024-02-01T10:38:04.552 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:a9t1YSSWOhHSzmOYoZoPdWCSDII= 2024-02-01T10:38:04.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:a9t1YSSWOhHSzmOYoZoPdWCSDII=', 'Content-Length': '0'} 2024-02-01T10:38:04.556 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c2b1a8e730d59b4-0065bb748c-5004-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:04.557 DEBUG:boto:b'vclfcn-45999false' 2024-02-01T10:38:04.557 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-45 zones={test-zone1, test-zone2} 2024-02-01T10:38:04.557 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-45 zones={test-zone3, test-zone2} 2024-02-01T10:38:04.557 DEBUG:boto:path=/vclfcn-45/ 2024-02-01T10:38:04.557 DEBUG:boto:auth_path=/vclfcn-45/ 2024-02-01T10:38:04.557 DEBUG:boto:Method: HEAD 2024-02-01T10:38:04.557 DEBUG:boto:Path: /vclfcn-45/ 2024-02-01T10:38:04.558 DEBUG:boto:Data: 2024-02-01T10:38:04.558 DEBUG:boto:Headers: {} 2024-02-01T10:38:04.558 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:04.558 DEBUG:boto:Port: 8001 2024-02-01T10:38:04.558 DEBUG:boto:Params: {} 2024-02-01T10:38:04.558 DEBUG:boto:Token: None 2024-02-01T10:38:04.558 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:04 GMT /vclfcn-45/ 2024-02-01T10:38:04.558 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0NbZWhxm6aLIufEnHM6l03UCdU4= 2024-02-01T10:38:04.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0NbZWhxm6aLIufEnHM6l03UCdU4=', 'Content-Length': '0'} 2024-02-01T10:38:04.563 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ebcbf460070587c0-0065bb748c-5324-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:04.563 DEBUG:boto:path=/vclfcn-45/ 2024-02-01T10:38:04.563 DEBUG:boto:auth_path=/vclfcn-45/ 2024-02-01T10:38:04.564 DEBUG:boto:Method: HEAD 2024-02-01T10:38:04.564 DEBUG:boto:Path: /vclfcn-45/ 2024-02-01T10:38:04.564 DEBUG:boto:Data: 2024-02-01T10:38:04.564 DEBUG:boto:Headers: {} 2024-02-01T10:38:04.564 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:04.564 DEBUG:boto:Port: 8000 2024-02-01T10:38:04.564 DEBUG:boto:Params: {} 2024-02-01T10:38:04.564 DEBUG:boto:Token: None 2024-02-01T10:38:04.564 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:04 GMT /vclfcn-45/ 2024-02-01T10:38:04.565 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0NbZWhxm6aLIufEnHM6l03UCdU4= 2024-02-01T10:38:04.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0NbZWhxm6aLIufEnHM6l03UCdU4=', 'Content-Length': '0'} 2024-02-01T10:38:04.568 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000049f7cee73083563d-0065bb748c-5004-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:04.569 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:38:04.569 DEBUG:boto:path=/vclfcn-45/ 2024-02-01T10:38:04.569 DEBUG:boto:auth_path=/vclfcn-45/ 2024-02-01T10:38:04.569 DEBUG:boto:path=/vclfcn-45/?versions&max-keys=999 2024-02-01T10:38:04.569 DEBUG:boto:auth_path=/vclfcn-45/?versions&max-keys=999 2024-02-01T10:38:04.569 DEBUG:boto:Method: GET 2024-02-01T10:38:04.569 DEBUG:boto:Path: /vclfcn-45/?versions&max-keys=999 2024-02-01T10:38:04.569 DEBUG:boto:Data: 2024-02-01T10:38:04.569 DEBUG:boto:Headers: {} 2024-02-01T10:38:04.570 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:04.570 DEBUG:boto:Port: 8001 2024-02-01T10:38:04.570 DEBUG:boto:Params: {} 2024-02-01T10:38:04.570 DEBUG:boto:Token: None 2024-02-01T10:38:04.570 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:04 GMT /vclfcn-45/?versions 2024-02-01T10:38:04.570 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:a9t1YSSWOhHSzmOYoZoPdWCSDII= 2024-02-01T10:38:04.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:a9t1YSSWOhHSzmOYoZoPdWCSDII=', 'Content-Length': '0'} 2024-02-01T10:38:04.575 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cdc26940f6121355-0065bb748c-5324-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:04.575 DEBUG:boto:b'vclfcn-45999false' 2024-02-01T10:38:04.576 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:38:04.576 DEBUG:boto:path=/vclfcn-45/ 2024-02-01T10:38:04.576 DEBUG:boto:auth_path=/vclfcn-45/ 2024-02-01T10:38:04.576 DEBUG:boto:path=/vclfcn-45/?versions&max-keys=999 2024-02-01T10:38:04.576 DEBUG:boto:auth_path=/vclfcn-45/?versions&max-keys=999 2024-02-01T10:38:04.576 DEBUG:boto:Method: GET 2024-02-01T10:38:04.576 DEBUG:boto:Path: /vclfcn-45/?versions&max-keys=999 2024-02-01T10:38:04.576 DEBUG:boto:Data: 2024-02-01T10:38:04.577 DEBUG:boto:Headers: {} 2024-02-01T10:38:04.577 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:04.577 DEBUG:boto:Port: 8000 2024-02-01T10:38:04.577 DEBUG:boto:Params: {} 2024-02-01T10:38:04.577 DEBUG:boto:Token: None 2024-02-01T10:38:04.577 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:04 GMT /vclfcn-45/?versions 2024-02-01T10:38:04.577 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:a9t1YSSWOhHSzmOYoZoPdWCSDII= 2024-02-01T10:38:04.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:a9t1YSSWOhHSzmOYoZoPdWCSDII=', 'Content-Length': '0'} 2024-02-01T10:38:04.581 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9b49881b841b951-0065bb748c-5004-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:04.624 DEBUG:boto:b'vclfcn-45999false' 2024-02-01T10:38:04.625 DEBUG:boto:path=/vclfcn-45/ 2024-02-01T10:38:04.625 DEBUG:boto:auth_path=/vclfcn-45/ 2024-02-01T10:38:04.625 DEBUG:boto:path=/vclfcn-45/?versions&max-keys=999 2024-02-01T10:38:04.625 DEBUG:boto:auth_path=/vclfcn-45/?versions&max-keys=999 2024-02-01T10:38:04.625 DEBUG:boto:Method: GET 2024-02-01T10:38:04.625 DEBUG:boto:Path: /vclfcn-45/?versions&max-keys=999 2024-02-01T10:38:04.625 DEBUG:boto:Data: 2024-02-01T10:38:04.625 DEBUG:boto:Headers: {} 2024-02-01T10:38:04.625 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:04.625 DEBUG:boto:Port: 8001 2024-02-01T10:38:04.626 DEBUG:boto:Params: {} 2024-02-01T10:38:04.626 DEBUG:boto:Token: None 2024-02-01T10:38:04.626 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:04 GMT /vclfcn-45/?versions 2024-02-01T10:38:04.626 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:a9t1YSSWOhHSzmOYoZoPdWCSDII= 2024-02-01T10:38:04.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:a9t1YSSWOhHSzmOYoZoPdWCSDII=', 'Content-Length': '0'} 2024-02-01T10:38:04.630 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a69f3b2ba57229d-0065bb748c-5324-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:04.631 DEBUG:boto:b'vclfcn-45999false' 2024-02-01T10:38:04.631 DEBUG:boto:path=/vclfcn-45/ 2024-02-01T10:38:04.631 DEBUG:boto:auth_path=/vclfcn-45/ 2024-02-01T10:38:04.631 DEBUG:boto:path=/vclfcn-45/?versions&max-keys=999 2024-02-01T10:38:04.631 DEBUG:boto:auth_path=/vclfcn-45/?versions&max-keys=999 2024-02-01T10:38:04.631 DEBUG:boto:Method: GET 2024-02-01T10:38:04.631 DEBUG:boto:Path: /vclfcn-45/?versions&max-keys=999 2024-02-01T10:38:04.631 DEBUG:boto:Data: 2024-02-01T10:38:04.632 DEBUG:boto:Headers: {} 2024-02-01T10:38:04.632 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:04.632 DEBUG:boto:Port: 8000 2024-02-01T10:38:04.632 DEBUG:boto:Params: {} 2024-02-01T10:38:04.632 DEBUG:boto:Token: None 2024-02-01T10:38:04.632 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:04 GMT /vclfcn-45/?versions 2024-02-01T10:38:04.632 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:a9t1YSSWOhHSzmOYoZoPdWCSDII= 2024-02-01T10:38:04.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:a9t1YSSWOhHSzmOYoZoPdWCSDII=', 'Content-Length': '0'} 2024-02-01T10:38:04.636 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005982a7dfbbad8929-0065bb748c-5004-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:04.637 DEBUG:boto:b'vclfcn-45999false' 2024-02-01T10:38:04.637 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-45 zones={test-zone3, test-zone2} 2024-02-01T10:38:04.637 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-45 zones={test-zone1, test-zone3} 2024-02-01T10:38:04.637 DEBUG:boto:path=/vclfcn-45/ 2024-02-01T10:38:04.637 DEBUG:boto:auth_path=/vclfcn-45/ 2024-02-01T10:38:04.637 DEBUG:boto:Method: HEAD 2024-02-01T10:38:04.637 DEBUG:boto:Path: /vclfcn-45/ 2024-02-01T10:38:04.638 DEBUG:boto:Data: 2024-02-01T10:38:04.638 DEBUG:boto:Headers: {} 2024-02-01T10:38:04.638 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:04.638 DEBUG:boto:Port: 8000 2024-02-01T10:38:04.638 DEBUG:boto:Params: {} 2024-02-01T10:38:04.638 DEBUG:boto:Token: None 2024-02-01T10:38:04.638 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:04 GMT /vclfcn-45/ 2024-02-01T10:38:04.638 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0NbZWhxm6aLIufEnHM6l03UCdU4= 2024-02-01T10:38:04.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0NbZWhxm6aLIufEnHM6l03UCdU4=', 'Content-Length': '0'} 2024-02-01T10:38:04.642 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003dcdc3ed28bdedd8-0065bb748c-5317-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:04.643 DEBUG:boto:path=/vclfcn-45/ 2024-02-01T10:38:04.643 DEBUG:boto:auth_path=/vclfcn-45/ 2024-02-01T10:38:04.643 DEBUG:boto:Method: HEAD 2024-02-01T10:38:04.643 DEBUG:boto:Path: /vclfcn-45/ 2024-02-01T10:38:04.643 DEBUG:boto:Data: 2024-02-01T10:38:04.643 DEBUG:boto:Headers: {} 2024-02-01T10:38:04.643 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:04.643 DEBUG:boto:Port: 8001 2024-02-01T10:38:04.643 DEBUG:boto:Params: {} 2024-02-01T10:38:04.644 DEBUG:boto:Token: None 2024-02-01T10:38:04.644 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:04 GMT /vclfcn-45/ 2024-02-01T10:38:04.644 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0NbZWhxm6aLIufEnHM6l03UCdU4= 2024-02-01T10:38:04.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0NbZWhxm6aLIufEnHM6l03UCdU4=', 'Content-Length': '0'} 2024-02-01T10:38:04.648 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001586f4e6c8555781-0065bb748c-5324-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:04.648 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:38:04.648 DEBUG:boto:path=/vclfcn-45/ 2024-02-01T10:38:04.648 DEBUG:boto:auth_path=/vclfcn-45/ 2024-02-01T10:38:04.648 DEBUG:boto:path=/vclfcn-45/?versions&max-keys=999 2024-02-01T10:38:04.649 DEBUG:boto:auth_path=/vclfcn-45/?versions&max-keys=999 2024-02-01T10:38:04.649 DEBUG:boto:Method: GET 2024-02-01T10:38:04.649 DEBUG:boto:Path: /vclfcn-45/?versions&max-keys=999 2024-02-01T10:38:04.649 DEBUG:boto:Data: 2024-02-01T10:38:04.649 DEBUG:boto:Headers: {} 2024-02-01T10:38:04.649 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:04.649 DEBUG:boto:Port: 8000 2024-02-01T10:38:04.649 DEBUG:boto:Params: {} 2024-02-01T10:38:04.649 DEBUG:boto:Token: None 2024-02-01T10:38:04.649 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:04 GMT /vclfcn-45/?versions 2024-02-01T10:38:04.649 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:a9t1YSSWOhHSzmOYoZoPdWCSDII= 2024-02-01T10:38:04.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:a9t1YSSWOhHSzmOYoZoPdWCSDII=', 'Content-Length': '0'} 2024-02-01T10:38:04.653 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000054587c09a910633-0065bb748c-5317-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:04.654 DEBUG:boto:b'vclfcn-45999false' 2024-02-01T10:38:04.654 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:38:04.654 DEBUG:boto:path=/vclfcn-45/ 2024-02-01T10:38:04.654 DEBUG:boto:auth_path=/vclfcn-45/ 2024-02-01T10:38:04.655 DEBUG:boto:path=/vclfcn-45/?versions&max-keys=999 2024-02-01T10:38:04.655 DEBUG:boto:auth_path=/vclfcn-45/?versions&max-keys=999 2024-02-01T10:38:04.655 DEBUG:boto:Method: GET 2024-02-01T10:38:04.655 DEBUG:boto:Path: /vclfcn-45/?versions&max-keys=999 2024-02-01T10:38:04.655 DEBUG:boto:Data: 2024-02-01T10:38:04.655 DEBUG:boto:Headers: {} 2024-02-01T10:38:04.655 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:04.655 DEBUG:boto:Port: 8001 2024-02-01T10:38:04.655 DEBUG:boto:Params: {} 2024-02-01T10:38:04.655 DEBUG:boto:Token: None 2024-02-01T10:38:04.656 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:04 GMT /vclfcn-45/?versions 2024-02-01T10:38:04.656 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:a9t1YSSWOhHSzmOYoZoPdWCSDII= 2024-02-01T10:38:04.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:a9t1YSSWOhHSzmOYoZoPdWCSDII=', 'Content-Length': '0'} 2024-02-01T10:38:04.659 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b15086a77633b4d-0065bb748c-5324-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:04.704 DEBUG:boto:b'vclfcn-45999false' 2024-02-01T10:38:04.704 DEBUG:boto:path=/vclfcn-45/ 2024-02-01T10:38:04.705 DEBUG:boto:auth_path=/vclfcn-45/ 2024-02-01T10:38:04.705 DEBUG:boto:path=/vclfcn-45/?versions&max-keys=999 2024-02-01T10:38:04.705 DEBUG:boto:auth_path=/vclfcn-45/?versions&max-keys=999 2024-02-01T10:38:04.705 DEBUG:boto:Method: GET 2024-02-01T10:38:04.705 DEBUG:boto:Path: /vclfcn-45/?versions&max-keys=999 2024-02-01T10:38:04.705 DEBUG:boto:Data: 2024-02-01T10:38:04.705 DEBUG:boto:Headers: {} 2024-02-01T10:38:04.705 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:04.705 DEBUG:boto:Port: 8000 2024-02-01T10:38:04.705 DEBUG:boto:Params: {} 2024-02-01T10:38:04.706 DEBUG:boto:Token: None 2024-02-01T10:38:04.706 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:04 GMT /vclfcn-45/?versions 2024-02-01T10:38:04.706 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:a9t1YSSWOhHSzmOYoZoPdWCSDII= 2024-02-01T10:38:04.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:a9t1YSSWOhHSzmOYoZoPdWCSDII=', 'Content-Length': '0'} 2024-02-01T10:38:04.711 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e0ea8ee40aefb2bf-0065bb748c-5317-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:04.713 DEBUG:boto:b'vclfcn-45999false' 2024-02-01T10:38:04.714 DEBUG:boto:path=/vclfcn-45/ 2024-02-01T10:38:04.714 DEBUG:boto:auth_path=/vclfcn-45/ 2024-02-01T10:38:04.714 DEBUG:boto:path=/vclfcn-45/?versions&max-keys=999 2024-02-01T10:38:04.714 DEBUG:boto:auth_path=/vclfcn-45/?versions&max-keys=999 2024-02-01T10:38:04.715 DEBUG:boto:Method: GET 2024-02-01T10:38:04.715 DEBUG:boto:Path: /vclfcn-45/?versions&max-keys=999 2024-02-01T10:38:04.715 DEBUG:boto:Data: 2024-02-01T10:38:04.715 DEBUG:boto:Headers: {} 2024-02-01T10:38:04.715 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:04.715 DEBUG:boto:Port: 8001 2024-02-01T10:38:04.715 DEBUG:boto:Params: {} 2024-02-01T10:38:04.715 DEBUG:boto:Token: None 2024-02-01T10:38:04.715 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:04 GMT /vclfcn-45/?versions 2024-02-01T10:38:04.716 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:a9t1YSSWOhHSzmOYoZoPdWCSDII= 2024-02-01T10:38:04.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:a9t1YSSWOhHSzmOYoZoPdWCSDII=', 'Content-Length': '0'} 2024-02-01T10:38:04.720 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000063782d6b3ffdd87-0065bb748c-5324-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:04.720 DEBUG:boto:b'vclfcn-45999false' 2024-02-01T10:38:04.720 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-45 zones={test-zone1, test-zone3} 2024-02-01T10:38:04.720 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-45 zones={test-zone2, test-zone3} 2024-02-01T10:38:04.721 DEBUG:boto:path=/vclfcn-45/ 2024-02-01T10:38:04.721 DEBUG:boto:auth_path=/vclfcn-45/ 2024-02-01T10:38:04.721 DEBUG:boto:Method: HEAD 2024-02-01T10:38:04.721 DEBUG:boto:Path: /vclfcn-45/ 2024-02-01T10:38:04.721 DEBUG:boto:Data: 2024-02-01T10:38:04.721 DEBUG:boto:Headers: {} 2024-02-01T10:38:04.721 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:04.721 DEBUG:boto:Port: 8000 2024-02-01T10:38:04.721 DEBUG:boto:Params: {} 2024-02-01T10:38:04.722 DEBUG:boto:Token: None 2024-02-01T10:38:04.722 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:04 GMT /vclfcn-45/ 2024-02-01T10:38:04.722 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0NbZWhxm6aLIufEnHM6l03UCdU4= 2024-02-01T10:38:04.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0NbZWhxm6aLIufEnHM6l03UCdU4=', 'Content-Length': '0'} 2024-02-01T10:38:04.726 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000009b855c8232e6f6e-0065bb748c-5004-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:04.726 DEBUG:boto:path=/vclfcn-45/ 2024-02-01T10:38:04.726 DEBUG:boto:auth_path=/vclfcn-45/ 2024-02-01T10:38:04.726 DEBUG:boto:Method: HEAD 2024-02-01T10:38:04.726 DEBUG:boto:Path: /vclfcn-45/ 2024-02-01T10:38:04.726 DEBUG:boto:Data: 2024-02-01T10:38:04.726 DEBUG:boto:Headers: {} 2024-02-01T10:38:04.726 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:04.726 DEBUG:boto:Port: 8001 2024-02-01T10:38:04.727 DEBUG:boto:Params: {} 2024-02-01T10:38:04.727 DEBUG:boto:Token: None 2024-02-01T10:38:04.727 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:04 GMT /vclfcn-45/ 2024-02-01T10:38:04.727 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0NbZWhxm6aLIufEnHM6l03UCdU4= 2024-02-01T10:38:04.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0NbZWhxm6aLIufEnHM6l03UCdU4=', 'Content-Length': '0'} 2024-02-01T10:38:04.731 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000088cecc145e37d34c-0065bb748c-5324-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:04.731 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:38:04.731 DEBUG:boto:path=/vclfcn-45/ 2024-02-01T10:38:04.731 DEBUG:boto:auth_path=/vclfcn-45/ 2024-02-01T10:38:04.731 DEBUG:boto:path=/vclfcn-45/?versions&max-keys=999 2024-02-01T10:38:04.731 DEBUG:boto:auth_path=/vclfcn-45/?versions&max-keys=999 2024-02-01T10:38:04.731 DEBUG:boto:Method: GET 2024-02-01T10:38:04.732 DEBUG:boto:Path: /vclfcn-45/?versions&max-keys=999 2024-02-01T10:38:04.732 DEBUG:boto:Data: 2024-02-01T10:38:04.732 DEBUG:boto:Headers: {} 2024-02-01T10:38:04.732 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:04.732 DEBUG:boto:Port: 8000 2024-02-01T10:38:04.732 DEBUG:boto:Params: {} 2024-02-01T10:38:04.732 DEBUG:boto:Token: None 2024-02-01T10:38:04.732 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:04 GMT /vclfcn-45/?versions 2024-02-01T10:38:04.733 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:a9t1YSSWOhHSzmOYoZoPdWCSDII= 2024-02-01T10:38:04.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:a9t1YSSWOhHSzmOYoZoPdWCSDII=', 'Content-Length': '0'} 2024-02-01T10:38:04.736 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000005ccbb3570fac590-0065bb748c-5004-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:04.736 DEBUG:boto:b'vclfcn-45999false' 2024-02-01T10:38:04.737 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:38:04.737 DEBUG:boto:path=/vclfcn-45/ 2024-02-01T10:38:04.737 DEBUG:boto:auth_path=/vclfcn-45/ 2024-02-01T10:38:04.737 DEBUG:boto:path=/vclfcn-45/?versions&max-keys=999 2024-02-01T10:38:04.737 DEBUG:boto:auth_path=/vclfcn-45/?versions&max-keys=999 2024-02-01T10:38:04.737 DEBUG:boto:Method: GET 2024-02-01T10:38:04.738 DEBUG:boto:Path: /vclfcn-45/?versions&max-keys=999 2024-02-01T10:38:04.738 DEBUG:boto:Data: 2024-02-01T10:38:04.738 DEBUG:boto:Headers: {} 2024-02-01T10:38:04.738 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:04.738 DEBUG:boto:Port: 8001 2024-02-01T10:38:04.738 DEBUG:boto:Params: {} 2024-02-01T10:38:04.738 DEBUG:boto:Token: None 2024-02-01T10:38:04.738 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:04 GMT /vclfcn-45/?versions 2024-02-01T10:38:04.739 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:a9t1YSSWOhHSzmOYoZoPdWCSDII= 2024-02-01T10:38:04.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:a9t1YSSWOhHSzmOYoZoPdWCSDII=', 'Content-Length': '0'} 2024-02-01T10:38:04.743 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a7a319f8eb74300-0065bb748c-5324-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:04.788 DEBUG:boto:b'vclfcn-45999false' 2024-02-01T10:38:04.788 DEBUG:boto:path=/vclfcn-45/ 2024-02-01T10:38:04.789 DEBUG:boto:auth_path=/vclfcn-45/ 2024-02-01T10:38:04.789 DEBUG:boto:path=/vclfcn-45/?versions&max-keys=999 2024-02-01T10:38:04.789 DEBUG:boto:auth_path=/vclfcn-45/?versions&max-keys=999 2024-02-01T10:38:04.789 DEBUG:boto:Method: GET 2024-02-01T10:38:04.789 DEBUG:boto:Path: /vclfcn-45/?versions&max-keys=999 2024-02-01T10:38:04.789 DEBUG:boto:Data: 2024-02-01T10:38:04.789 DEBUG:boto:Headers: {} 2024-02-01T10:38:04.789 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:04.789 DEBUG:boto:Port: 8000 2024-02-01T10:38:04.789 DEBUG:boto:Params: {} 2024-02-01T10:38:04.790 DEBUG:boto:Token: None 2024-02-01T10:38:04.790 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:04 GMT /vclfcn-45/?versions 2024-02-01T10:38:04.790 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:a9t1YSSWOhHSzmOYoZoPdWCSDII= 2024-02-01T10:38:04.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:a9t1YSSWOhHSzmOYoZoPdWCSDII=', 'Content-Length': '0'} 2024-02-01T10:38:04.794 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d7c7ec9c2de1e890-0065bb748c-5004-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:04.794 DEBUG:boto:b'vclfcn-45999false' 2024-02-01T10:38:04.795 DEBUG:boto:path=/vclfcn-45/ 2024-02-01T10:38:04.795 DEBUG:boto:auth_path=/vclfcn-45/ 2024-02-01T10:38:04.795 DEBUG:boto:path=/vclfcn-45/?versions&max-keys=999 2024-02-01T10:38:04.795 DEBUG:boto:auth_path=/vclfcn-45/?versions&max-keys=999 2024-02-01T10:38:04.795 DEBUG:boto:Method: GET 2024-02-01T10:38:04.795 DEBUG:boto:Path: /vclfcn-45/?versions&max-keys=999 2024-02-01T10:38:04.795 DEBUG:boto:Data: 2024-02-01T10:38:04.795 DEBUG:boto:Headers: {} 2024-02-01T10:38:04.796 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:04.796 DEBUG:boto:Port: 8001 2024-02-01T10:38:04.796 DEBUG:boto:Params: {} 2024-02-01T10:38:04.796 DEBUG:boto:Token: None 2024-02-01T10:38:04.796 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:38:04 GMT /vclfcn-45/?versions 2024-02-01T10:38:04.796 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:a9t1YSSWOhHSzmOYoZoPdWCSDII= 2024-02-01T10:38:04.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:a9t1YSSWOhHSzmOYoZoPdWCSDII=', 'Content-Length': '0'} 2024-02-01T10:38:04.801 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a69073ea1114f7ae-0065bb748c-5324-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:04.802 DEBUG:boto:b'vclfcn-45999false' 2024-02-01T10:38:04.802 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-45 zones={test-zone2, test-zone3} 2024-02-01T10:38:04.802 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '5dc4efbd-eccd-47a8-9471-ae4a209ec978', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:04.802 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '5dc4efbd-eccd-47a8-9471-ae4a209ec978', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:04.802 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 5dc4efbd-eccd-47a8-9471-ae4a209ec978 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:38:04.845 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:38:04.845 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:38:04.927 INFO:teuthology.orchestra.run.smithi148.stdout:[{"id":"1_1706782453.129936_29.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2024-02-01T10:14:13.129936524Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RY7up0tQcOf9ElajXOF5d6i","ver":1},"status":{"status":"write"}}},{"id":"1_1706782453.132619_30.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2024-02-01T10:14:13.132619785Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RY7up0tQcOf9ElajXOF5d6i","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782596.137698_33.1","section":"bucket.instance","name":"vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19","timestamp":"2024-02-01T10:16:36.137698180Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gSnH8BiIXGWkG2sGorT95Ca","ver":1},"status":{"status":"write"}}},{"id":"1_1706782596.141485_34.1","section":"bucket.instance","name":"vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19","timestamp":"2024-02-01T10:16:36.141485128Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gSnH8BiIXGWkG2sGorT95Ca","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782596.142883_35.1","section":"bucket","name":"vclfcn-10","timestamp":"2024-02-01T10:16:36.142883452Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_C-y9oz7otdaRTy0zc_QeA8H","ver":1},"status":{"status":"write"}}},{"id":"1_1706782596.147037_36.1","section":"bucket","name":"vclfcn-10","timestamp":"2024-02-01T10:16:36.147037349Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_C-y9oz7otdaRTy0zc_QeA8H","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782596.165713_37.1","section":"bucket.instance","name":"vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20","timestamp":"2024-02-01T10:16:36.165713463Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ugvw9Z3SeaHLB2NBY8cOvIs","ver":1},"status":{"status":"write"}}},{"id":"1_1706782596.170196_38.1","section":"bucket.instance","name":"vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20","timestamp":"2024-02-01T10:16:36.170196200Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ugvw9Z3SeaHLB2NBY8cOvIs","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782596.171951_39.1","section":"bucket","name":"vclfcn-11","timestamp":"2024-02-01T10:16:36.171951352Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_clR9bGsKa2Bt45u61U1vnaU","ver":1},"status":{"status":"write"}}},{"id":"1_1706782596.176142_40.1","section":"bucket","name":"vclfcn-11","timestamp":"2024-02-01T10:16:36.176142514Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_clR9bGsKa2Bt45u61U1vnaU","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782596.215001_41.1","section":"bucket.instance","name":"vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21","timestamp":"2024-02-01T10:16:36.215001761Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0aJjbbhzkBKwKh67jUyjFXU","ver":1},"status":{"status":"write"}}},{"id":"1_1706782596.221792_42.1","section":"bucket.instance","name":"vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21","timestamp":"2024-02-01T10:16:36.221792015Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0aJjbbhzkBKwKh67jUyjFXU","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782596.224377_43.1","section":"bucket","name":"vclfcn-12","timestamp":"2024-02-01T10:16:36.224377534Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YzQLQ2ZvUn8a9M8WMLPPlj-","ver":1},"status":{"status":"write"}}},{"id":"1_1706782596.230502_44.1","section":"bucket","name":"vclfcn-12","timestamp":"2024-02-01T10:16:36.230502418Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YzQLQ2ZvUn8a9M8WMLPPlj-","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782693.582816_53.1","section":"bucket.instance","name":"vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22","timestamp":"2024-02-01T10:18:13.582816318Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_IyMDVQZq1Dm4Ik60abXwswh","ver":1},"status":{"status":"write"}}},{"id":"1_1706782693.589569_54.1","section":"bucket.instance","name":"vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22","timestamp":"2024-02-01T10:18:13.589569915Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_IyMDVQZq1Dm4Ik60abXwswh","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782693.592623_55.1","section":"bucket","name":"vclfcn-13","timestamp":"2024-02-01T10:18:13.592623458Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SBtUJfjUPgWxvP4marStPj8","ver":1},"status":{"status":"write"}}},{"id":"1_1706782693.599222_56.1","section":"bucket","name":"vclfcn-13","timestamp":"2024-02-01T10:18:13.599222251Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SBtUJfjUPgWxvP4marStPj8","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782693.622788_57.1","section":"bucket.instance","name":"vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23","timestamp":"2024-02-01T10:18:13.622788043Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OLXZUmEaWvH6RgSaGLsC5gp","ver":1},"status":{"status":"write"}}},{"id":"1_1706782693.629469_58.1","section":"bucket.instance","name":"vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23","timestamp":"2024-02-01T10:18:13.629469201Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OLXZUmEaWvH6RgSaGLsC5gp","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782693.632089_59.1","section":"bucket","name":"vclfcn-14","timestamp":"2024-02-01T10:18:13.632089534Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6H3UQNuqVeMSp5sl1iVxf3G","ver":1},"status":{"status":"write"}}},{"id":"1_1706782693.638718_60.1","section":"bucket","name":"vclfcn-14","timestamp":"2024-02-01T10:18:13.638718897Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6H3UQNuqVeMSp5sl1iVxf3G","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782693.686988_61.1","section":"bucket.instance","name":"vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24","timestamp":"2024-02-01T10:18:13.686988320Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GhbTNO6qlpBiEMg1HJVBNAz","ver":1},"status":{"status":"write"}}},{"id":"1_1706782693.694347_62.1","section":"bucket.instance","name":"vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24","timestamp":"2024-02-01T10:18:13.694347551Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GhbTNO6qlpBiEMg1HJVBNAz","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782693.696886_63.1","section":"bucket","name":"vclfcn-15","timestamp":"2024-02-01T10:18:13.696886663Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PC87lFbJZuz6KYHILchNgZJ","ver":1},"status":{"status":"write"}}},{"id":"1_1706782693.703781_64.1","section":"bucket","name":"vclfcn-15","timestamp":"2024-02-01T10:18:13.703781181Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PC87lFbJZuz6KYHILchNgZJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782788.125323_68.1","section":"bucket.instance","name":"vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25","timestamp":"2024-02-01T10:19:48.125323856Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GJQLwPJ67-aampE80YQ6oYN","ver":1},"status":{"status":"write"}}},{"id":"1_1706782788.132848_69.1","section":"bucket.instance","name":"vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25","timestamp":"2024-02-01T10:19:48.132848685Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GJQLwPJ67-aampE80YQ6oYN","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782788.136003_70.1","section":"bucket","name":"vclfcn-16","timestamp":"2024-02-01T10:19:48.136003157Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yZQS2Ooyuo-DWcyRYk0NZG5","ver":1},"status":{"status":"write"}}},{"id":"1_1706782788.142977_71.1","section":"bucket","name":"vclfcn-16","timestamp":"2024-02-01T10:19:48.142977363Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yZQS2Ooyuo-DWcyRYk0NZG5","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782788.168776_72.1","section":"bucket.instance","name":"vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26","timestamp":"2024-02-01T10:19:48.168776620Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ds-zBeTuxJy65JHqqd4lb5Z","ver":1},"status":{"status":"write"}}},{"id":"1_1706782788.174951_73.1","section":"bucket.instance","name":"vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26","timestamp":"2024-02-01T10:19:48.174951550Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ds-zBeTuxJy65JHqqd4lb5Z","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782788.178001_74.1","section":"bucket","name":"vclfcn-17","timestamp":"2024-02-01T10:19:48.178001249Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZIbkkBCCgIhVLT8B2r1w5K_","ver":1},"status":{"status":"write"}}},{"id":"1_1706782788.184593_75.1","section":"bucket","name":"vclfcn-17","timestamp":"2024-02-01T10:19:48.184593776Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZIbkkBCCgIhVLT8B2r1w5K_","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782788.225397_76.1","section":"bucket.instance","name":"vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27","timestamp":"2024-02-01T10:19:48.225397011Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gkEVh6TTRHTmUu-UImISUWd","ver":1},"status":{"status":"write"}}},{"id":"1_1706782788.230219_77.1","section":"bucket.instance","name":"vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27","timestamp":"2024-02-01T10:19:48.230219600Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gkEVh6TTRHTmUu-UImISUWd","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782788.232911_78.1","section":"bucket","name":"vclfcn-18","timestamp":"2024-02-01T10:19:48.232911150Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HlQ9gchNi7E4cFgOusBiYBD","ver":1},"status":{"status":"write"}}},{"id":"1_1706782788.241244_79.1","section":"bucket","name":"vclfcn-18","timestamp":"2024-02-01T10:19:48.241244766Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HlQ9gchNi7E4cFgOusBiYBD","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782788.264154_80.1","section":"bucket.instance","name":"vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25","timestamp":"2024-02-01T10:19:48.264154269Z","data":{"read_version":{"tag":"_GJQLwPJ67-aampE80YQ6oYN","ver":1},"write_version":{"tag":"_GJQLwPJ67-aampE80YQ6oYN","ver":2},"status":{"status":"write"}}},{"id":"1_1706782788.268634_81.1","section":"bucket.instance","name":"vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25","timestamp":"2024-02-01T10:19:48.268634480Z","data":{"read_version":{"tag":"_GJQLwPJ67-aampE80YQ6oYN","ver":1},"write_version":{"tag":"_GJQLwPJ67-aampE80YQ6oYN","ver":2},"status":{"status":"complete"}}},{"id":"1_1706782788.275810_82.1","section":"bucket.instance","name":"vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26","timestamp":"2024-02-01T10:19:48.275810646Z","data":{"read_version":{"tag":"_ds-zBeTuxJy65JHqqd4lb5Z","ver":1},"write_version":{"tag":"_ds-zBeTuxJy65JHqqd4lb5Z","ver":2},"status":{"status":"write"}}},{"id":"1_1706782788.281245_83.1","section":"bucket.instance","name":"vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26","timestamp":"2024-02-01T10:19:48.281245680Z","data":{"read_version":{"tag":"_ds-zBeTuxJy65JHqqd4lb5Z","ver":1},"write_version":{"tag":"_ds-zBeTuxJy65JHqqd4lb5Z","ver":2},"status":{"status":"complete"}}},{"id":"1_1706782788.295656_84.1","section":"bucket.instance","name":"vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27","timestamp":"2024-02-01T10:19:48.295656474Z","data":{"read_version":{"tag":"_gkEVh6TTRHTmUu-UImISUWd","ver":1},"write_version":{"tag":"_gkEVh6TTRHTmUu-UImISUWd","ver":2},"status":{"status":"write"}}},{"id":"1_1706782788.299861_85.1","section":"bucket.instance","name":"vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27","timestamp":"2024-02-01T10:19:48.299861431Z","data":{"read_version":{"tag":"_gkEVh6TTRHTmUu-UImISUWd","ver":1},"write_version":{"tag":"_gkEVh6TTRHTmUu-UImISUWd","ver":2},"status":{"status":"complete"}}},{"id":"1_1706782985.577592_93.1","section":"bucket.instance","name":"vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28","timestamp":"2024-02-01T10:23:05.577592857Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Tkzhdhq2YxvoQoXNUGw-VlA","ver":1},"status":{"status":"write"}}},{"id":"1_1706782985.584276_94.1","section":"bucket.instance","name":"vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28","timestamp":"2024-02-01T10:23:05.584276295Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Tkzhdhq2YxvoQoXNUGw-VlA","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782985.586795_95.1","section":"bucket","name":"vclfcn-19","timestamp":"2024-02-01T10:23:05.586795365Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uFY0fAasS29yh69Gp_uVUE2","ver":1},"status":{"status":"write"}}},{"id":"1_1706782985.593052_96.1","section":"bucket","name":"vclfcn-19","timestamp":"2024-02-01T10:23:05.593052024Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uFY0fAasS29yh69Gp_uVUE2","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782985.602177_97.1","section":"bucket.instance","name":"vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28","timestamp":"2024-02-01T10:23:05.602177901Z","data":{"read_version":{"tag":"_Tkzhdhq2YxvoQoXNUGw-VlA","ver":1},"write_version":{"tag":"_Tkzhdhq2YxvoQoXNUGw-VlA","ver":2},"status":{"status":"write"}}},{"id":"1_1706782985.609410_98.1","section":"bucket.instance","name":"vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28","timestamp":"2024-02-01T10:23:05.609410401Z","data":{"read_version":{"tag":"_Tkzhdhq2YxvoQoXNUGw-VlA","ver":1},"write_version":{"tag":"_Tkzhdhq2YxvoQoXNUGw-VlA","ver":2},"status":{"status":"complete"}}},{"id":"1_1706783281.140663_112.1","section":"bucket.instance","name":"vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29","timestamp":"2024-02-01T10:28:01.140663757Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RY5J0cHq0mZCxKxUWbHq1nu","ver":1},"status":{"status":"write"}}},{"id":"1_1706783281.146879_113.1","section":"bucket.instance","name":"vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29","timestamp":"2024-02-01T10:28:01.146879624Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RY5J0cHq0mZCxKxUWbHq1nu","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783281.149599_114.1","section":"bucket","name":"vclfcn-20","timestamp":"2024-02-01T10:28:01.149599676Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LBpQRODD-fP4UvS_L3UYNrJ","ver":1},"status":{"status":"write"}}},{"id":"1_1706783281.156355_115.1","section":"bucket","name":"vclfcn-20","timestamp":"2024-02-01T10:28:01.156355553Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LBpQRODD-fP4UvS_L3UYNrJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783313.047690_117.1","section":"bucket.instance","name":"vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29","timestamp":"2024-02-01T10:28:33.047690221Z","data":{"read_version":{"tag":"_RY5J0cHq0mZCxKxUWbHq1nu","ver":1},"write_version":{"tag":"_RY5J0cHq0mZCxKxUWbHq1nu","ver":2},"status":{"status":"write"}}},{"id":"1_1706783313.054991_118.1","section":"bucket.instance","name":"vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29","timestamp":"2024-02-01T10:28:33.054991128Z","data":{"read_version":{"tag":"_RY5J0cHq0mZCxKxUWbHq1nu","ver":1},"write_version":{"tag":"_RY5J0cHq0mZCxKxUWbHq1nu","ver":2},"status":{"status":"complete"}}},{"id":"1_1706783510.479238_122.1","section":"bucket.instance","name":"vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29","timestamp":"2024-02-01T10:31:50.479238314Z","data":{"read_version":{"tag":"_RY5J0cHq0mZCxKxUWbHq1nu","ver":2},"write_version":{"tag":"_RY5J0cHq0mZCxKxUWbHq1nu","ver":3},"status":{"status":"write"}}},{"id":"1_1706783510.486411_123.1","section":"bucket.instance","name":"vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29","timestamp":"2024-02-01T10:31:50.486411437Z","data":{"read_version":{"tag":"_RY5J0cHq0mZCxKxUWbHq1nu","ver":2},"write_version":{"tag":"_RY5J0cHq0mZCxKxUWbHq1nu","ver":3},"status":{"status":"complete"}}},{"id":"1_1706783557.936493_125.1","section":"bucket.instance","name":"vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30","timestamp":"2024-02-01T10:32:37.936493302Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SBaAf2hqSgxj2ifisrCDUQ-","ver":1},"status":{"status":"write"}}},{"id":"1_1706783557.943553_126.1","section":"bucket.instance","name":"vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30","timestamp":"2024-02-01T10:32:37.943553989Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SBaAf2hqSgxj2ifisrCDUQ-","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783557.946282_127.1","section":"bucket","name":"vclfcn-21","timestamp":"2024-02-01T10:32:37.946282859Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_aS4q16MZBGYstqXzbnAETLq","ver":1},"status":{"status":"write"}}},{"id":"1_1706783557.952481_128.1","section":"bucket","name":"vclfcn-21","timestamp":"2024-02-01T10:32:37.952481787Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_aS4q16MZBGYstqXzbnAETLq","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783557.977604_129.1","section":"bucket.instance","name":"vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31","timestamp":"2024-02-01T10:32:37.977604938Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AivkNk9zM8C_ACc4CMMCA9j","ver":1},"status":{"status":"write"}}},{"id":"1_1706783557.984354_130.1","section":"bucket.instance","name":"vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31","timestamp":"2024-02-01T10:32:37.984354568Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AivkNk9zM8C_ACc4CMMCA9j","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783557.986976_131.1","section":"bucket","name":"vclfcn-22","timestamp":"2024-02-01T10:32:37.986976409Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OC84jTcY1mo5V4XtRR3FK81","ver":1},"status":{"status":"write"}}},{"id":"1_1706783557.993660_132.1","section":"bucket","name":"vclfcn-22","timestamp":"2024-02-01T10:32:37.993660224Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OC84jTcY1mo5V4XtRR3FK81","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783558.040696_133.1","section":"bucket.instance","name":"vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32","timestamp":"2024-02-01T10:32:38.040696413Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_y0EPEo2ASjenysOQ86zdnP6","ver":1},"status":{"status":"write"}}},{"id":"1_1706783558.047555_134.1","section":"bucket.instance","name":"vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32","timestamp":"2024-02-01T10:32:38.047555506Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_y0EPEo2ASjenysOQ86zdnP6","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783558.049874_135.1","section":"bucket","name":"vclfcn-23","timestamp":"2024-02-01T10:32:38.049874456Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pctO5Jr19wCuuci8zhYun2q","ver":1},"status":{"status":"write"}}},{"id":"1_1706783558.055502_136.1","section":"bucket","name":"vclfcn-23","timestamp":"2024-02-01T10:32:38.055502136Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pctO5Jr19wCuuci8zhYun2q","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783558.081356_137.1","section":"bucket.instance","name":"vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30","timestamp":"2024-02-01T10:32:38.081356052Z","data":{"read_version":{"tag":"_SBaAf2hqSgxj2ifisrCDUQ-","ver":1},"write_version":{"tag":"_SBaAf2hqSgxj2ifisrCDUQ-","ver":2},"status":{"status":"write"}}},{"id":"1_1706783558.088380_138.1","section":"bucket.instance","name":"vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30","timestamp":"2024-02-01T10:32:38.088380225Z","data":{"read_version":{"tag":"_SBaAf2hqSgxj2ifisrCDUQ-","ver":1},"write_version":{"tag":"_SBaAf2hqSgxj2ifisrCDUQ-","ver":2},"status":{"status":"complete"}}},{"id":"1_1706783558.095619_139.1","section":"bucket.instance","name":"vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31","timestamp":"2024-02-01T10:32:38.095619390Z","data":{"read_version":{"tag":"_AivkNk9zM8C_ACc4CMMCA9j","ver":1},"write_version":{"tag":"_AivkNk9zM8C_ACc4CMMCA9j","ver":2},"status":{"status":"write"}}},{"id":"1_1706783558.100472_140.1","section":"bucket.instance","name":"vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31","timestamp":"2024-02-01T10:32:38.100472386Z","data":{"read_version":{"tag":"_AivkNk9zM8C_ACc4CMMCA9j","ver":1},"write_version":{"tag":"_AivkNk9zM8C_ACc4CMMCA9j","ver":2},"status":{"status":"complete"}}},{"id":"1_1706783558.110239_141.1","section":"bucket.instance","name":"vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32","timestamp":"2024-02-01T10:32:38.110239478Z","data":{"read_version":{"tag":"_y0EPEo2ASjenysOQ86zdnP6","ver":1},"write_version":{"tag":"_y0EPEo2ASjenysOQ86zdnP6","ver":2},"status":{"status":"write"}}},{"id":"1_1706783558.115340_142.1","section":"bucket.instance","name":"vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32","timestamp":"2024-02-01T10:32:38.115340472Z","data":{"read_version":{"tag":"_y0EPEo2ASjenysOQ86zdnP6","ver":1},"write_version":{"tag":"_y0EPEo2ASjenysOQ86zdnP6","ver":2},"status":{"status":"complete"}}},{"id":"1_1706783613.485844_144.1","section":"bucket.instance","name":"vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33","timestamp":"2024-02-01T10:33:33.485844007Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KQaPa_IyHr1JRoHAufGblmN","ver":1},"status":{"status":"write"}}},{"id":"1_1706783613.493076_145.1","section":"bucket.instance","name":"vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33","timestamp":"2024-02-01T10:33:33.493076639Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KQaPa_IyHr1JRoHAufGblmN","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783613.495609_146.1","section":"bucket","name":"vclfcn-24","timestamp":"2024-02-01T10:33:33.495609093Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wdRz7umYZCWDz61bnbDJ5DR","ver":1},"status":{"status":"write"}}},{"id":"1_1706783613.501842_147.1","section":"bucket","name":"vclfcn-24","timestamp":"2024-02-01T10:33:33.501842098Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wdRz7umYZCWDz61bnbDJ5DR","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783613.526071_148.1","section":"bucket.instance","name":"vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34","timestamp":"2024-02-01T10:33:33.526071464Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TJp0m8FTngleqfn93NSNEFQ","ver":1},"status":{"status":"write"}}},{"id":"1_1706783613.533006_149.1","section":"bucket.instance","name":"vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34","timestamp":"2024-02-01T10:33:33.533006576Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TJp0m8FTngleqfn93NSNEFQ","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783613.535429_150.1","section":"bucket","name":"vclfcn-25","timestamp":"2024-02-01T10:33:33.535429751Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XUEOg4Lx9rBHbAJwKq1lwrJ","ver":1},"status":{"status":"write"}}},{"id":"1_1706783613.541933_151.1","section":"bucket","name":"vclfcn-25","timestamp":"2024-02-01T10:33:33.541933857Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XUEOg4Lx9rBHbAJwKq1lwrJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783613.587133_152.1","section":"bucket.instance","name":"vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35","timestamp":"2024-02-01T10:33:33.587133123Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_djT8hpdivEzfKbjzZM1tGYc","ver":1},"status":{"status":"write"}}},{"id":"1_1706783613.594437_153.1","section":"bucket.instance","name":"vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35","timestamp":"2024-02-01T10:33:33.594437539Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_djT8hpdivEzfKbjzZM1tGYc","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783613.597205_154.1","section":"bucket","name":"vclfcn-26","timestamp":"2024-02-01T10:33:33.597205142Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3fH7UbMU-OxdukYrx1ZeZDR","ver":1},"status":{"status":"write"}}},{"id":"1_1706783613.604282_155.1","section":"bucket","name":"vclfcn-26","timestamp":"2024-02-01T10:33:33.604282859Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3fH7UbMU-OxdukYrx1ZeZDR","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783613.632812_156.1","section":"bucket.instance","name":"vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33","timestamp":"2024-02-01T10:33:33.632812026Z","data":{"read_version":{"tag":"_KQaPa_IyHr1JRoHAufGblmN","ver":1},"write_version":{"tag":"_KQaPa_IyHr1JRoHAufGblmN","ver":2},"status":{"status":"write"}}},{"id":"1_1706783613.639457_157.1","section":"bucket.instance","name":"vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33","timestamp":"2024-02-01T10:33:33.639457857Z","data":{"read_version":{"tag":"_KQaPa_IyHr1JRoHAufGblmN","ver":1},"write_version":{"tag":"_KQaPa_IyHr1JRoHAufGblmN","ver":2},"status":{"status":"complete"}}},{"id":"1_1706783613.645309_158.1","section":"bucket.instance","name":"vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33","timestamp":"2024-02-01T10:33:33.645309049Z","data":{"read_version":{"tag":"_KQaPa_IyHr1JRoHAufGblmN","ver":2},"write_version":{"tag":"_KQaPa_IyHr1JRoHAufGblmN","ver":3},"status":{"status":"write"}}},{"id":"1_1706783613.652911_159.1","section":"bucket.instance","name":"vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33","timestamp":"2024-02-01T10:33:33.652911988Z","data":{"read_version":{"tag":"_KQaPa_IyHr1JRoHAufGblmN","ver":2},"write_version":{"tag":"_KQaPa_IyHr1JRoHAufGblmN","ver":3},"status":{"status":"complete"}}},{"id":"1_1706783613.662523_160.1","section":"bucket.instance","name":"vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34","timestamp":"2024-02-01T10:33:33.662523190Z","data":{"read_version":{"tag":"_TJp0m8FTngleqfn93NSNEFQ","ver":1},"write_version":{"tag":"_TJp0m8FTngleqfn93NSNEFQ","ver":2},"status":{"status":"write"}}},{"id":"1_1706783613.674476_161.1","section":"bucket.instance","name":"vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34","timestamp":"2024-02-01T10:33:33.674476638Z","data":{"read_version":{"tag":"_TJp0m8FTngleqfn93NSNEFQ","ver":1},"write_version":{"tag":"_TJp0m8FTngleqfn93NSNEFQ","ver":2},"status":{"status":"complete"}}},{"id":"1_1706783613.698284_162.1","section":"bucket.instance","name":"vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34","timestamp":"2024-02-01T10:33:33.698284010Z","data":{"read_version":{"tag":"_TJp0m8FTngleqfn93NSNEFQ","ver":2},"write_version":{"tag":"_TJp0m8FTngleqfn93NSNEFQ","ver":3},"status":{"status":"write"}}},{"id":"1_1706783613.707513_163.1","section":"bucket.instance","name":"vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34","timestamp":"2024-02-01T10:33:33.707513991Z","data":{"read_version":{"tag":"_TJp0m8FTngleqfn93NSNEFQ","ver":2},"write_version":{"tag":"_TJp0m8FTngleqfn93NSNEFQ","ver":3},"status":{"status":"complete"}}},{"id":"1_1706783613.728949_164.1","section":"bucket.instance","name":"vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35","timestamp":"2024-02-01T10:33:33.728949508Z","data":{"read_version":{"tag":"_djT8hpdivEzfKbjzZM1tGYc","ver":1},"write_version":{"tag":"_djT8hpdivEzfKbjzZM1tGYc","ver":2},"status":{"status":"write"}}},{"id":"1_1706783613.736733_165.1","section":"bucket.instance","name":"vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35","timestamp":"2024-02-01T10:33:33.736733042Z","data":{"read_version":{"tag":"_djT8hpdivEzfKbjzZM1tGYc","ver":1},"write_version":{"tag":"_djT8hpdivEzfKbjzZM1tGYc","ver":2},"status":{"status":"complete"}}},{"id":"1_1706783613.748133_166.1","section":"bucket.instance","name":"vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35","timestamp":"2024-02-01T10:33:33.748133962Z","data":{"read_version":{"tag":"_djT8hpdivEzfKbjzZM1tGYc","ver":2},"write_version":{"tag":"_djT8hpdivEzfKbjzZM1tGYc","ver":3},"status":{"status":"write"}}},{"id":"1_1706783613.755304_167.1","section":"bucket.instance","name":"vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35","timestamp":"2024-02-01T10:33:33.755304841Z","data":{"read_version":{"tag":"_djT8hpdivEzfKbjzZM1tGYc","ver":2},"write_version":{"tag":"_djT8hpdivEzfKbjzZM1tGYc","ver":3},"status":{"status":"complete"}}},{"id":"1_1706783732.990670_178.1","section":"bucket.instance","name":"vclfcn-27:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.36","timestamp":"2024-02-01T10:35:32.990670009Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rYzBOFkXYI9xqwLbwgqkEVW","ver":1},"status":{"status":"write"}}},{"id":"1_1706783732.996955_179.1","section":"bucket.instance","name":"vclfcn-27:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.36","timestamp":"2024-02-01T10:35:32.996955046Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rYzBOFkXYI9xqwLbwgqkEVW","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783732.999583_180.1","section":"bucket","name":"vclfcn-27","timestamp":"2024-02-01T10:35:32.999583128Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9cHBpx7BkJp-jUw4o1CRrfh","ver":1},"status":{"status":"write"}}},{"id":"1_1706783733.006036_181.1","section":"bucket","name":"vclfcn-27","timestamp":"2024-02-01T10:35:33.006036701Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9cHBpx7BkJp-jUw4o1CRrfh","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783733.034354_182.1","section":"bucket.instance","name":"vclfcn-28:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.37","timestamp":"2024-02-01T10:35:33.034354608Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ppjqf2iiQQLeeW9nmcMFwt7","ver":1},"status":{"status":"write"}}},{"id":"1_1706783733.041168_183.1","section":"bucket.instance","name":"vclfcn-28:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.37","timestamp":"2024-02-01T10:35:33.041168668Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ppjqf2iiQQLeeW9nmcMFwt7","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783733.043826_184.1","section":"bucket","name":"vclfcn-28","timestamp":"2024-02-01T10:35:33.043826329Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_O0D5bB8GV3hYxs2j9GMFn0q","ver":1},"status":{"status":"write"}}},{"id":"1_1706783733.050085_185.1","section":"bucket","name":"vclfcn-28","timestamp":"2024-02-01T10:35:33.050085404Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_O0D5bB8GV3hYxs2j9GMFn0q","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783733.095185_186.1","section":"bucket.instance","name":"vclfcn-29:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.38","timestamp":"2024-02-01T10:35:33.095185012Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_b7qqNp_v7vEEA2IVwp_dZhU","ver":1},"status":{"status":"write"}}},{"id":"1_1706783733.100262_187.1","section":"bucket.instance","name":"vclfcn-29:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.38","timestamp":"2024-02-01T10:35:33.100262420Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_b7qqNp_v7vEEA2IVwp_dZhU","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783733.102142_188.1","section":"bucket","name":"vclfcn-29","timestamp":"2024-02-01T10:35:33.102142458Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_oH5-kep0dGu6Rb8cPoOFEHk","ver":1},"status":{"status":"write"}}},{"id":"1_1706783733.107054_189.1","section":"bucket","name":"vclfcn-29","timestamp":"2024-02-01T10:35:33.107054109Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_oH5-kep0dGu6Rb8cPoOFEHk","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783733.136865_190.1","section":"bucket.instance","name":"vclfcn-27:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.36","timestamp":"2024-02-01T10:35:33.136865088Z","data":{"read_version":{"tag":"_rYzBOFkXYI9xqwLbwgqkEVW","ver":1},"write_version":{"tag":"_rYzBOFkXYI9xqwLbwgqkEVW","ver":2},"status":{"status":"write"}}},{"id":"1_1706783733.148545_191.1","section":"bucket.instance","name":"vclfcn-27:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.36","timestamp":"2024-02-01T10:35:33.148545378Z","data":{"read_version":{"tag":"_rYzBOFkXYI9xqwLbwgqkEVW","ver":1},"write_version":{"tag":"_rYzBOFkXYI9xqwLbwgqkEVW","ver":2},"status":{"status":"complete"}}},{"id":"1_1706783733.202069_192.1","section":"bucket.instance","name":"vclfcn-28:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.37","timestamp":"2024-02-01T10:35:33.202069645Z","data":{"read_version":{"tag":"_Ppjqf2iiQQLeeW9nmcMFwt7","ver":1},"write_version":{"tag":"_Ppjqf2iiQQLeeW9nmcMFwt7","ver":2},"status":{"status":"write"}}},{"id":"1_1706783733.207436_193.1","section":"bucket.instance","name":"vclfcn-28:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.37","timestamp":"2024-02-01T10:35:33.207436436Z","data":{"read_version":{"tag":"_Ppjqf2iiQQLeeW9nmcMFwt7","ver":1},"write_version":{"tag":"_Ppjqf2iiQQLeeW9nmcMFwt7","ver":2},"status":{"status":"complete"}}},{"id":"1_1706783733.222466_194.1","section":"bucket.instance","name":"vclfcn-29:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.38","timestamp":"2024-02-01T10:35:33.222466987Z","data":{"read_version":{"tag":"_b7qqNp_v7vEEA2IVwp_dZhU","ver":1},"write_version":{"tag":"_b7qqNp_v7vEEA2IVwp_dZhU","ver":2},"status":{"status":"write"}}},{"id":"1_1706783733.227246_195.1","section":"bucket.instance","name":"vclfcn-29:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.38","timestamp":"2024-02-01T10:35:33.227246173Z","data":{"read_version":{"tag":"_b7qqNp_v7vEEA2IVwp_dZhU","ver":1},"write_version":{"tag":"_b7qqNp_v7vEEA2IVwp_dZhU","ver":2},"status":{"status":"complete"}}},{"id":"1_1706783733.293745_196.1","section":"bucket.instance","name":"vclfcn-30:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.39","timestamp":"2024-02-01T10:35:33.293745968Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HhnF-4-cQEYFZsmhVgAo_3W","ver":1},"status":{"status":"write"}}},{"id":"1_1706783733.299312_197.1","section":"bucket.instance","name":"vclfcn-30:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.39","timestamp":"2024-02-01T10:35:33.299312767Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HhnF-4-cQEYFZsmhVgAo_3W","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783733.301968_198.1","section":"bucket","name":"vclfcn-30","timestamp":"2024-02-01T10:35:33.301968348Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Yct8ulBblkK4SDKF5OWcOcv","ver":1},"status":{"status":"write"}}},{"id":"1_1706783733.307848_199.1","section":"bucket","name":"vclfcn-30","timestamp":"2024-02-01T10:35:33.307848983Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Yct8ulBblkK4SDKF5OWcOcv","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783733.330768_200.1","section":"bucket.instance","name":"vclfcn-31:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.40","timestamp":"2024-02-01T10:35:33.330768162Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OsUE1ilsRE6kOECGtCDtxHd","ver":1},"status":{"status":"write"}}},{"id":"1_1706783733.336478_201.1","section":"bucket.instance","name":"vclfcn-31:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.40","timestamp":"2024-02-01T10:35:33.336478192Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OsUE1ilsRE6kOECGtCDtxHd","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783733.338988_202.1","section":"bucket","name":"vclfcn-31","timestamp":"2024-02-01T10:35:33.338988237Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5E7wmMDv9o08f28guv8WeNq","ver":1},"status":{"status":"write"}}},{"id":"1_1706783733.345150_203.1","section":"bucket","name":"vclfcn-31","timestamp":"2024-02-01T10:35:33.345150939Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5E7wmMDv9o08f28guv8WeNq","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783733.391532_204.1","section":"bucket.instance","name":"vclfcn-32:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.41","timestamp":"2024-02-01T10:35:33.391532601Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PICgXYT5_p4s6Npm6_d7L86","ver":1},"status":{"status":"write"}}},{"id":"1_1706783733.396882_205.1","section":"bucket.instance","name":"vclfcn-32:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.41","timestamp":"2024-02-01T10:35:33.396882656Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PICgXYT5_p4s6Npm6_d7L86","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783733.399225_206.1","section":"bucket","name":"vclfcn-32","timestamp":"2024-02-01T10:35:33.399225264Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jXGNidsm4ya5U_EGuN9pxg2","ver":1},"status":{"status":"write"}}},{"id":"1_1706783733.404709_207.1","section":"bucket","name":"vclfcn-32","timestamp":"2024-02-01T10:35:33.404709608Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jXGNidsm4ya5U_EGuN9pxg2","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783733.435292_208.1","section":"bucket.instance","name":"vclfcn-30:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.39","timestamp":"2024-02-01T10:35:33.435292002Z","data":{"read_version":{"tag":"_HhnF-4-cQEYFZsmhVgAo_3W","ver":1},"write_version":{"tag":"_HhnF-4-cQEYFZsmhVgAo_3W","ver":2},"status":{"status":"write"}}},{"id":"1_1706783733.441533_209.1","section":"bucket.instance","name":"vclfcn-30:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.39","timestamp":"2024-02-01T10:35:33.441533749Z","data":{"read_version":{"tag":"_HhnF-4-cQEYFZsmhVgAo_3W","ver":1},"write_version":{"tag":"_HhnF-4-cQEYFZsmhVgAo_3W","ver":2},"status":{"status":"complete"}}},{"id":"1_1706783733.454934_210.1","section":"bucket.instance","name":"vclfcn-31:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.40","timestamp":"2024-02-01T10:35:33.454934992Z","data":{"read_version":{"tag":"_OsUE1ilsRE6kOECGtCDtxHd","ver":1},"write_version":{"tag":"_OsUE1ilsRE6kOECGtCDtxHd","ver":2},"status":{"status":"write"}}},{"id":"1_1706783733.460796_211.1","section":"bucket.instance","name":"vclfcn-31:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.40","timestamp":"2024-02-01T10:35:33.460796234Z","data":{"read_version":{"tag":"_OsUE1ilsRE6kOECGtCDtxHd","ver":1},"write_version":{"tag":"_OsUE1ilsRE6kOECGtCDtxHd","ver":2},"status":{"status":"complete"}}},{"id":"1_1706783733.521076_212.1","section":"bucket.instance","name":"vclfcn-32:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.41","timestamp":"2024-02-01T10:35:33.521076539Z","data":{"read_version":{"tag":"_PICgXYT5_p4s6Npm6_d7L86","ver":1},"write_version":{"tag":"_PICgXYT5_p4s6Npm6_d7L86","ver":2},"status":{"status":"write"}}},{"id":"1_1706783733.537903_213.1","section":"bucket.instance","name":"vclfcn-32:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.41","timestamp":"2024-02-01T10:35:33.537903176Z","data":{"read_version":{"tag":"_PICgXYT5_p4s6Npm6_d7L86","ver":1},"write_version":{"tag":"_PICgXYT5_p4s6Npm6_d7L86","ver":2},"status":{"status":"complete"}}},{"id":"1_1706783733.593359_214.1","section":"bucket.instance","name":"vclfcn-33:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.42","timestamp":"2024-02-01T10:35:33.593359133Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MHNCkb6d03Mpbe8ExDNyOIe","ver":1},"status":{"status":"write"}}},{"id":"1_1706783733.603919_215.1","section":"bucket.instance","name":"vclfcn-33:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.42","timestamp":"2024-02-01T10:35:33.603919491Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MHNCkb6d03Mpbe8ExDNyOIe","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783733.605766_216.1","section":"bucket","name":"vclfcn-33","timestamp":"2024-02-01T10:35:33.605766755Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_t7Tsi3a8X7oyOSKwXbGKwOO","ver":1},"status":{"status":"write"}}},{"id":"1_1706783733.609698_217.1","section":"bucket","name":"vclfcn-33","timestamp":"2024-02-01T10:35:33.609698777Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_t7Tsi3a8X7oyOSKwXbGKwOO","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783733.628224_218.1","section":"bucket.instance","name":"vclfcn-34:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.43","timestamp":"2024-02-01T10:35:33.628224591Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_V7tzE1wcwyuANojoierxhFz","ver":1},"status":{"status":"write"}}},{"id":"1_1706783733.632156_219.1","section":"bucket.instance","name":"vclfcn-34:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.43","timestamp":"2024-02-01T10:35:33.632156395Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_V7tzE1wcwyuANojoierxhFz","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783733.633573_220.1","section":"bucket","name":"vclfcn-34","timestamp":"2024-02-01T10:35:33.633573750Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Z5byZtsMURp7i2bsj6dBpHv","ver":1},"status":{"status":"write"}}},{"id":"1_1706783733.637172_221.1","section":"bucket","name":"vclfcn-34","timestamp":"2024-02-01T10:35:33.637172728Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Z5byZtsMURp7i2bsj6dBpHv","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783733.677953_222.1","section":"bucket.instance","name":"vclfcn-35:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.44","timestamp":"2024-02-01T10:35:33.677953821Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_C_0FSFOkL0ACO1-gSETb17o","ver":1},"status":{"status":"write"}}},{"id":"1_1706783733.683763_223.1","section":"bucket.instance","name":"vclfcn-35:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.44","timestamp":"2024-02-01T10:35:33.683763713Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_C_0FSFOkL0ACO1-gSETb17o","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783733.686120_224.1","section":"bucket","name":"vclfcn-35","timestamp":"2024-02-01T10:35:33.686120912Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CzHizcA9wKsnJKxAmY_lmbG","ver":1},"status":{"status":"write"}}},{"id":"1_1706783733.691403_225.1","section":"bucket","name":"vclfcn-35","timestamp":"2024-02-01T10:35:33.691403393Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CzHizcA9wKsnJKxAmY_lmbG","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783733.722103_226.1","section":"bucket.instance","name":"vclfcn-33:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.42","timestamp":"2024-02-01T10:35:33.722103624Z","data":{"read_version":{"tag":"_MHNCkb6d03Mpbe8ExDNyOIe","ver":1},"write_version":{"tag":"_MHNCkb6d03Mpbe8ExDNyOIe","ver":2},"status":{"status":"write"}}},{"id":"1_1706783733.727948_227.1","section":"bucket.instance","name":"vclfcn-33:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.42","timestamp":"2024-02-01T10:35:33.727948851Z","data":{"read_version":{"tag":"_MHNCkb6d03Mpbe8ExDNyOIe","ver":1},"write_version":{"tag":"_MHNCkb6d03Mpbe8ExDNyOIe","ver":2},"status":{"status":"complete"}}},{"id":"1_1706783734.739760_228.1","section":"bucket.instance","name":"vclfcn-34:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.43","timestamp":"2024-02-01T10:35:34.739760076Z","data":{"read_version":{"tag":"_V7tzE1wcwyuANojoierxhFz","ver":1},"write_version":{"tag":"_V7tzE1wcwyuANojoierxhFz","ver":2},"status":{"status":"write"}}},{"id":"1_1706783734.747549_229.1","section":"bucket.instance","name":"vclfcn-34:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.43","timestamp":"2024-02-01T10:35:34.747549598Z","data":{"read_version":{"tag":"_V7tzE1wcwyuANojoierxhFz","ver":1},"write_version":{"tag":"_V7tzE1wcwyuANojoierxhFz","ver":2},"status":{"status":"complete"}}},{"id":"1_1706783735.800850_230.1","section":"bucket.instance","name":"vclfcn-35:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.44","timestamp":"2024-02-01T10:35:35.800850730Z","data":{"read_version":{"tag":"_C_0FSFOkL0ACO1-gSETb17o","ver":1},"write_version":{"tag":"_C_0FSFOkL0ACO1-gSETb17o","ver":2},"status":{"status":"write"}}},{"id":"1_1706783735.808526_231.1","section":"bucket.instance","name":"vclfcn-35:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.44","timestamp":"2024-02-01T10:35:35.808526056Z","data":{"read_version":{"tag":"_C_0FSFOkL0ACO1-gSETb17o","ver":1},"write_version":{"tag":"_C_0FSFOkL0ACO1-gSETb17o","ver":2},"status":{"status":"complete"}}},{"id":"1_1706783735.837558_232.1","section":"bucket.instance","name":"vclfcn-36:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.45","timestamp":"2024-02-01T10:35:35.837558347Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9X_9QezGXcrZpslOj2s6DRz","ver":1},"status":{"status":"write"}}},{"id":"1_1706783735.844352_233.1","section":"bucket.instance","name":"vclfcn-36:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.45","timestamp":"2024-02-01T10:35:35.844352448Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9X_9QezGXcrZpslOj2s6DRz","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783735.846902_234.1","section":"bucket","name":"vclfcn-36","timestamp":"2024-02-01T10:35:35.846902742Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_I6xC7jnhAySZruK6KFfJcFj","ver":1},"status":{"status":"write"}}},{"id":"1_1706783735.853365_235.1","section":"bucket","name":"vclfcn-36","timestamp":"2024-02-01T10:35:35.853365976Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_I6xC7jnhAySZruK6KFfJcFj","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783735.880850_236.1","section":"bucket.instance","name":"vclfcn-37:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.46","timestamp":"2024-02-01T10:35:35.880850867Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_S59hS1hjhhTMEjgkmPP0l40","ver":1},"status":{"status":"write"}}},{"id":"1_1706783735.888598_237.1","section":"bucket.instance","name":"vclfcn-37:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.46","timestamp":"2024-02-01T10:35:35.888598691Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_S59hS1hjhhTMEjgkmPP0l40","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783735.891588_238.1","section":"bucket","name":"vclfcn-37","timestamp":"2024-02-01T10:35:35.891588303Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_u9b9H7g55Gxvm7UwsBsXRkc","ver":1},"status":{"status":"write"}}},{"id":"1_1706783735.903784_239.1","section":"bucket","name":"vclfcn-37","timestamp":"2024-02-01T10:35:35.903784912Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_u9b9H7g55Gxvm7UwsBsXRkc","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783735.956302_240.1","section":"bucket.instance","name":"vclfcn-38:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.47","timestamp":"2024-02-01T10:35:35.956302005Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_f3LeuHUV7bKmB4iX_c1NfhG","ver":1},"status":{"status":"write"}}},{"id":"1_1706783735.967905_241.1","section":"bucket.instance","name":"vclfcn-38:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.47","timestamp":"2024-02-01T10:35:35.967905199Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_f3LeuHUV7bKmB4iX_c1NfhG","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783735.969939_242.1","section":"bucket","name":"vclfcn-38","timestamp":"2024-02-01T10:35:35.969939474Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kCOsVG7gpmdckTvQ0-OqkTT","ver":1},"status":{"status":"write"}}},{"id":"1_1706783735.974063_243.1","section":"bucket","name":"vclfcn-38","timestamp":"2024-02-01T10:35:35.974063216Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kCOsVG7gpmdckTvQ0-OqkTT","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783746.868400_244.1","section":"bucket.instance","name":"vclfcn-39:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.48","timestamp":"2024-02-01T10:35:46.868400756Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2ismKVnh0upW2_YAiZ3dCm0","ver":1},"status":{"status":"write"}}},{"id":"1_1706783746.874991_245.1","section":"bucket.instance","name":"vclfcn-39:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.48","timestamp":"2024-02-01T10:35:46.874991156Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2ismKVnh0upW2_YAiZ3dCm0","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783746.877780_246.1","section":"bucket","name":"vclfcn-39","timestamp":"2024-02-01T10:35:46.877780982Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Mtq3wgmpQ2UVbTzYfiOTADK","ver":1},"status":{"status":"write"}}},{"id":"1_1706783746.884330_247.1","section":"bucket","name":"vclfcn-39","timestamp":"2024-02-01T10:35:46.884330514Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Mtq3wgmpQ2UVbTzYfiOTADK","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783746.913795_248.1","section":"bucket.instance","name":"vclfcn-40:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.49","timestamp":"2024-02-01T10:35:46.913795216Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wB4GCZVccemaw1QY5tXPf2_","ver":1},"status":{"status":"write"}}},{"id":"1_1706783746.921924_249.1","section":"bucket.instance","name":"vclfcn-40:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.49","timestamp":"2024-02-01T10:35:46.921924008Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wB4GCZVccemaw1QY5tXPf2_","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783746.925325_250.1","section":"bucket","name":"vclfcn-40","timestamp":"2024-02-01T10:35:46.925325542Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_DzmwhWoct-KkCaXHuhCBDRu","ver":1},"status":{"status":"write"}}},{"id":"1_1706783746.934867_251.1","section":"bucket","name":"vclfcn-40","timestamp":"2024-02-01T10:35:46.934867482Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_DzmwhWoct-KkCaXHuhCBDRu","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783746.990514_252.1","section":"bucket.instance","name":"vclfcn-41:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.50","timestamp":"2024-02-01T10:35:46.990514760Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TYFOExYyROxXd60biRdH4WR","ver":1},"status":{"status":"write"}}},{"id":"1_1706783746.999155_253.1","section":"bucket.instance","name":"vclfcn-41:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.50","timestamp":"2024-02-01T10:35:46.999155781Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TYFOExYyROxXd60biRdH4WR","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783747.001868_254.1","section":"bucket","name":"vclfcn-41","timestamp":"2024-02-01T10:35:47.001868659Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-h9FvYOZI-YjsdziH0V4lBk","ver":1},"status":{"status":"write"}}},{"id":"1_1706783747.007639_255.1","section":"bucket","name":"vclfcn-41","timestamp":"2024-02-01T10:35:47.007639832Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-h9FvYOZI-YjsdziH0V4lBk","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782574.724766_44.1","section":"bucket.instance","name":"vclfcn-1:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.1","timestamp":"2024-02-01T10:16:14.724766052Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ibyZ4ny-dO1eyNWrgR0xDdL","ver":1},"status":{"status":"write"}}},{"id":"1_1706782574.747809_45.1","section":"bucket.instance","name":"vclfcn-1:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.1","timestamp":"2024-02-01T10:16:14.747809032Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ibyZ4ny-dO1eyNWrgR0xDdL","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782574.750145_46.1","section":"bucket","name":"vclfcn-1","timestamp":"2024-02-01T10:16:14.750145401Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_N_oVpqg2avT9RyIn3n93zXI","ver":1},"status":{"status":"write"}}},{"id":"1_1706782574.755240_47.1","section":"bucket","name":"vclfcn-1","timestamp":"2024-02-01T10:16:14.755240100Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_N_oVpqg2avT9RyIn3n93zXI","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782574.779443_48.1","section":"bucket.instance","name":"vclfcn-2:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.2","timestamp":"2024-02-01T10:16:14.779443186Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_50A5h7IO153g3aLFcIk9L8F","ver":1},"status":{"status":"write"}}},{"id":"1_1706782574.784819_49.1","section":"bucket.instance","name":"vclfcn-2:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.2","timestamp":"2024-02-01T10:16:14.784819298Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_50A5h7IO153g3aLFcIk9L8F","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782574.786565_50.1","section":"bucket","name":"vclfcn-2","timestamp":"2024-02-01T10:16:14.786565198Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_t8Dr2abL8IoMUXCcMicPxaz","ver":1},"status":{"status":"write"}}},{"id":"1_1706782574.791675_51.1","section":"bucket","name":"vclfcn-2","timestamp":"2024-02-01T10:16:14.791675561Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_t8Dr2abL8IoMUXCcMicPxaz","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782577.457619_52.1","section":"bucket.instance","name":"vclfcn-3:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.3","timestamp":"2024-02-01T10:16:17.457619330Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AARS1itaxavcuaxye1dSdHX","ver":1},"status":{"status":"write"}}},{"id":"1_1706782577.467024_53.1","section":"bucket.instance","name":"vclfcn-3:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.3","timestamp":"2024-02-01T10:16:17.467024067Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AARS1itaxavcuaxye1dSdHX","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782577.469499_54.1","section":"bucket","name":"vclfcn-3","timestamp":"2024-02-01T10:16:17.469499300Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dY9nFqtzaiF6gs-LUJpzNgy","ver":1},"status":{"status":"write"}}},{"id":"1_1706782577.476062_55.1","section":"bucket","name":"vclfcn-3","timestamp":"2024-02-01T10:16:17.476062234Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dY9nFqtzaiF6gs-LUJpzNgy","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782583.048966_56.1","section":"bucket.instance","name":"vclfcn-4:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.4","timestamp":"2024-02-01T10:16:23.048966924Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jZZOT-7qYVNFmmz--YeJW8c","ver":1},"status":{"status":"write"}}},{"id":"1_1706782583.053776_57.1","section":"bucket.instance","name":"vclfcn-4:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.4","timestamp":"2024-02-01T10:16:23.053776598Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jZZOT-7qYVNFmmz--YeJW8c","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782583.055771_58.1","section":"bucket","name":"vclfcn-4","timestamp":"2024-02-01T10:16:23.055771006Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7FYomkbpFjXsMEVUf--3FIN","ver":1},"status":{"status":"write"}}},{"id":"1_1706782583.060768_59.1","section":"bucket","name":"vclfcn-4","timestamp":"2024-02-01T10:16:23.060768290Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7FYomkbpFjXsMEVUf--3FIN","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782583.080541_60.1","section":"bucket.instance","name":"vclfcn-5:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.5","timestamp":"2024-02-01T10:16:23.080541747Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GuZ_ydrh7uZExo0k-XCf-mR","ver":1},"status":{"status":"write"}}},{"id":"1_1706782583.085084_61.1","section":"bucket.instance","name":"vclfcn-5:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.5","timestamp":"2024-02-01T10:16:23.085084437Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GuZ_ydrh7uZExo0k-XCf-mR","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782583.087346_62.1","section":"bucket","name":"vclfcn-5","timestamp":"2024-02-01T10:16:23.087346765Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KFexYQNYfYmKGSCAKJDSs_v","ver":1},"status":{"status":"write"}}},{"id":"1_1706782583.091891_63.1","section":"bucket","name":"vclfcn-5","timestamp":"2024-02-01T10:16:23.091891354Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KFexYQNYfYmKGSCAKJDSs_v","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782583.135000_64.1","section":"bucket.instance","name":"vclfcn-6:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.6","timestamp":"2024-02-01T10:16:23.135000549Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nUYxPNk4mpHlEoPbPAjZI6V","ver":1},"status":{"status":"write"}}},{"id":"1_1706782583.140997_65.1","section":"bucket.instance","name":"vclfcn-6:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.6","timestamp":"2024-02-01T10:16:23.140997195Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nUYxPNk4mpHlEoPbPAjZI6V","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782583.143037_66.1","section":"bucket","name":"vclfcn-6","timestamp":"2024-02-01T10:16:23.143037696Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4xyZKpWpwMtUf5XA3nUVwvT","ver":1},"status":{"status":"write"}}},{"id":"1_1706782583.148165_67.1","section":"bucket","name":"vclfcn-6","timestamp":"2024-02-01T10:16:23.148165375Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4xyZKpWpwMtUf5XA3nUVwvT","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782583.582106_68.1","section":"bucket.instance","name":"vclfcn-4:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.7","timestamp":"2024-02-01T10:16:23.582106079Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_K-GLBBcTrWLHEjhiQTh8AAO","ver":1},"status":{"status":"write"}}},{"id":"1_1706782583.585480_69.1","section":"bucket.instance","name":"vclfcn-4:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.7","timestamp":"2024-02-01T10:16:23.585480929Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_K-GLBBcTrWLHEjhiQTh8AAO","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782583.586646_70.1","section":"bucket","name":"vclfcn-4","timestamp":"2024-02-01T10:16:23.586646340Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7FYomkbpFjXsMEVUf--3FIN","ver":1},"status":{"status":"write"}}},{"id":"1_1706782583.588852_71.1","section":"bucket","name":"vclfcn-4","timestamp":"2024-02-01T10:16:23.588852889Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7FYomkbpFjXsMEVUf--3FIN","ver":1},"status":{"status":"abort"}}},{"id":"1_1706782583.604965_72.1","section":"bucket.instance","name":"vclfcn-5:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.8","timestamp":"2024-02-01T10:16:23.604965201Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-38jr9mPIsX5Hbh-yF_INzw","ver":1},"status":{"status":"write"}}},{"id":"1_1706782583.608205_73.1","section":"bucket.instance","name":"vclfcn-5:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.8","timestamp":"2024-02-01T10:16:23.608205537Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-38jr9mPIsX5Hbh-yF_INzw","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782583.609727_74.1","section":"bucket","name":"vclfcn-5","timestamp":"2024-02-01T10:16:23.609727931Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KFexYQNYfYmKGSCAKJDSs_v","ver":1},"status":{"status":"write"}}},{"id":"1_1706782583.612105_75.1","section":"bucket","name":"vclfcn-5","timestamp":"2024-02-01T10:16:23.612105467Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KFexYQNYfYmKGSCAKJDSs_v","ver":1},"status":{"status":"abort"}}},{"id":"1_1706782583.632804_76.1","section":"bucket.instance","name":"vclfcn-6:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.9","timestamp":"2024-02-01T10:16:23.632804818Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xPuFAl83iV88j7bpn1hxTNX","ver":1},"status":{"status":"write"}}},{"id":"1_1706782583.638590_77.1","section":"bucket.instance","name":"vclfcn-6:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.9","timestamp":"2024-02-01T10:16:23.638590664Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xPuFAl83iV88j7bpn1hxTNX","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782583.640189_78.1","section":"bucket","name":"vclfcn-6","timestamp":"2024-02-01T10:16:23.640189514Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4xyZKpWpwMtUf5XA3nUVwvT","ver":1},"status":{"status":"write"}}},{"id":"1_1706782583.642926_79.1","section":"bucket","name":"vclfcn-6","timestamp":"2024-02-01T10:16:23.642926395Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4xyZKpWpwMtUf5XA3nUVwvT","ver":1},"status":{"status":"abort"}}},{"id":"1_1706782583.671310_80.1","section":"bucket.instance","name":"vclfcn-4:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.10","timestamp":"2024-02-01T10:16:23.671310356Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1U8H0rwTUD_-21XS12rwqpQ","ver":1},"status":{"status":"write"}}},{"id":"1_1706782583.677081_81.1","section":"bucket.instance","name":"vclfcn-4:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.10","timestamp":"2024-02-01T10:16:23.677081964Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1U8H0rwTUD_-21XS12rwqpQ","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782583.678607_82.1","section":"bucket","name":"vclfcn-4","timestamp":"2024-02-01T10:16:23.678607917Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7FYomkbpFjXsMEVUf--3FIN","ver":1},"status":{"status":"write"}}},{"id":"1_1706782583.681355_83.1","section":"bucket","name":"vclfcn-4","timestamp":"2024-02-01T10:16:23.681355024Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7FYomkbpFjXsMEVUf--3FIN","ver":1},"status":{"status":"abort"}}},{"id":"1_1706782583.724585_84.1","section":"bucket.instance","name":"vclfcn-5:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.11","timestamp":"2024-02-01T10:16:23.724585309Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1uvujq5ws5ZqUnKk2XeudY4","ver":1},"status":{"status":"write"}}},{"id":"1_1706782583.728526_85.1","section":"bucket.instance","name":"vclfcn-5:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.11","timestamp":"2024-02-01T10:16:23.728526702Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1uvujq5ws5ZqUnKk2XeudY4","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782583.729969_86.1","section":"bucket","name":"vclfcn-5","timestamp":"2024-02-01T10:16:23.729969063Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KFexYQNYfYmKGSCAKJDSs_v","ver":1},"status":{"status":"write"}}},{"id":"1_1706782583.733921_87.1","section":"bucket","name":"vclfcn-5","timestamp":"2024-02-01T10:16:23.733921102Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KFexYQNYfYmKGSCAKJDSs_v","ver":1},"status":{"status":"abort"}}},{"id":"1_1706782583.821698_88.1","section":"bucket.instance","name":"vclfcn-6:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.12","timestamp":"2024-02-01T10:16:23.821698722Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_InLI6Y5Soa-cFmcsAza4-1x","ver":1},"status":{"status":"write"}}},{"id":"1_1706782583.828621_89.1","section":"bucket.instance","name":"vclfcn-6:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.12","timestamp":"2024-02-01T10:16:23.828621437Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_InLI6Y5Soa-cFmcsAza4-1x","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782583.831465_90.1","section":"bucket","name":"vclfcn-6","timestamp":"2024-02-01T10:16:23.831465748Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4xyZKpWpwMtUf5XA3nUVwvT","ver":1},"status":{"status":"write"}}},{"id":"1_1706782583.837030_91.1","section":"bucket","name":"vclfcn-6","timestamp":"2024-02-01T10:16:23.837030233Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4xyZKpWpwMtUf5XA3nUVwvT","ver":1},"status":{"status":"abort"}}},{"id":"1_1706782583.931558_92.1","section":"bucket.instance","name":"vclfcn-4:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.13","timestamp":"2024-02-01T10:16:23.931558665Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_mh6kxfzVQf2159X6vdQEtr9","ver":1},"status":{"status":"write"}}},{"id":"1_1706782583.939774_93.1","section":"bucket.instance","name":"vclfcn-4:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.13","timestamp":"2024-02-01T10:16:23.939774137Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_mh6kxfzVQf2159X6vdQEtr9","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782583.942334_94.1","section":"bucket","name":"vclfcn-4","timestamp":"2024-02-01T10:16:23.942334764Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7FYomkbpFjXsMEVUf--3FIN","ver":1},"status":{"status":"write"}}},{"id":"1_1706782583.947201_95.1","section":"bucket","name":"vclfcn-4","timestamp":"2024-02-01T10:16:23.947201866Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7FYomkbpFjXsMEVUf--3FIN","ver":1},"status":{"status":"abort"}}},{"id":"1_1706782583.997720_96.1","section":"bucket.instance","name":"vclfcn-5:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.14","timestamp":"2024-02-01T10:16:23.997720926Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_u7t7dVEXnS4njZhJkFAad6L","ver":1},"status":{"status":"write"}}},{"id":"1_1706782584.004511_97.1","section":"bucket.instance","name":"vclfcn-5:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.14","timestamp":"2024-02-01T10:16:24.004511156Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_u7t7dVEXnS4njZhJkFAad6L","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782584.007009_98.1","section":"bucket","name":"vclfcn-5","timestamp":"2024-02-01T10:16:24.007009106Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KFexYQNYfYmKGSCAKJDSs_v","ver":1},"status":{"status":"write"}}},{"id":"1_1706782584.011561_99.1","section":"bucket","name":"vclfcn-5","timestamp":"2024-02-01T10:16:24.011561820Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KFexYQNYfYmKGSCAKJDSs_v","ver":1},"status":{"status":"abort"}}},{"id":"1_1706782584.107340_100.1","section":"bucket.instance","name":"vclfcn-6:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.15","timestamp":"2024-02-01T10:16:24.107340431Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SwtrgKLrPfbA6nAuIOAjPY5","ver":1},"status":{"status":"write"}}},{"id":"1_1706782584.114866_101.1","section":"bucket.instance","name":"vclfcn-6:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.15","timestamp":"2024-02-01T10:16:24.114866482Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SwtrgKLrPfbA6nAuIOAjPY5","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782584.117713_102.1","section":"bucket","name":"vclfcn-6","timestamp":"2024-02-01T10:16:24.117713159Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4xyZKpWpwMtUf5XA3nUVwvT","ver":1},"status":{"status":"write"}}},{"id":"1_1706782584.123490_103.1","section":"bucket","name":"vclfcn-6","timestamp":"2024-02-01T10:16:24.123490508Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4xyZKpWpwMtUf5XA3nUVwvT","ver":1},"status":{"status":"abort"}}},{"id":"1_1706782589.804979_104.1","section":"bucket.instance","name":"vclfcn-7:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.16","timestamp":"2024-02-01T10:16:29.804979118Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_z1mGy633o2dYXHeltm-sLIr","ver":1},"status":{"status":"write"}}},{"id":"1_1706782589.808233_105.1","section":"bucket.instance","name":"vclfcn-7:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.16","timestamp":"2024-02-01T10:16:29.808233327Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_z1mGy633o2dYXHeltm-sLIr","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782589.809506_106.1","section":"bucket","name":"vclfcn-7","timestamp":"2024-02-01T10:16:29.809506716Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_aQqC4o_dOalWjSRbtEyqQ9y","ver":1},"status":{"status":"write"}}},{"id":"1_1706782589.812385_107.1","section":"bucket","name":"vclfcn-7","timestamp":"2024-02-01T10:16:29.812385583Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_aQqC4o_dOalWjSRbtEyqQ9y","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782589.827390_108.1","section":"bucket.instance","name":"vclfcn-8:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.17","timestamp":"2024-02-01T10:16:29.827390273Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fg266bvdezCNOcfE9SKi_vH","ver":1},"status":{"status":"write"}}},{"id":"1_1706782589.832050_109.1","section":"bucket.instance","name":"vclfcn-8:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.17","timestamp":"2024-02-01T10:16:29.832050187Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fg266bvdezCNOcfE9SKi_vH","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782589.833185_110.1","section":"bucket","name":"vclfcn-8","timestamp":"2024-02-01T10:16:29.833185878Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VxqI9_Wg5TOlSHC_n8MLWjO","ver":1},"status":{"status":"write"}}},{"id":"1_1706782589.836457_111.1","section":"bucket","name":"vclfcn-8","timestamp":"2024-02-01T10:16:29.836457633Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VxqI9_Wg5TOlSHC_n8MLWjO","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782589.877746_112.1","section":"bucket.instance","name":"vclfcn-9:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.18","timestamp":"2024-02-01T10:16:29.877746869Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jIv31DiHuYuZmjj2jtUcl5K","ver":1},"status":{"status":"write"}}},{"id":"1_1706782589.884299_113.1","section":"bucket.instance","name":"vclfcn-9:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.18","timestamp":"2024-02-01T10:16:29.884299625Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jIv31DiHuYuZmjj2jtUcl5K","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782589.886780_114.1","section":"bucket","name":"vclfcn-9","timestamp":"2024-02-01T10:16:29.886780733Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fop2OKI1GWzp_3SQxaIH8uR","ver":1},"status":{"status":"write"}}},{"id":"1_1706782589.893748_115.1","section":"bucket","name":"vclfcn-9","timestamp":"2024-02-01T10:16:29.893748078Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fop2OKI1GWzp_3SQxaIH8uR","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782595.478858_116.1","section":"bucket","name":"vclfcn-7","timestamp":"2024-02-01T10:16:35.478858938Z","data":{"read_version":{"tag":"_aQqC4o_dOalWjSRbtEyqQ9y","ver":1},"write_version":{"tag":"_aQqC4o_dOalWjSRbtEyqQ9y","ver":2},"status":{"status":"remove"}}},{"id":"1_1706782595.482342_117.1","section":"bucket","name":"vclfcn-7","timestamp":"2024-02-01T10:16:35.482342431Z","data":{"read_version":{"tag":"_aQqC4o_dOalWjSRbtEyqQ9y","ver":1},"write_version":{"tag":"_aQqC4o_dOalWjSRbtEyqQ9y","ver":2},"status":{"status":"complete"}}},{"id":"1_1706782595.541467_118.1","section":"bucket","name":"vclfcn-8","timestamp":"2024-02-01T10:16:35.541467766Z","data":{"read_version":{"tag":"_VxqI9_Wg5TOlSHC_n8MLWjO","ver":1},"write_version":{"tag":"_VxqI9_Wg5TOlSHC_n8MLWjO","ver":2},"status":{"status":"remove"}}},{"id":"1_1706782595.545966_119.1","section":"bucket","name":"vclfcn-8","timestamp":"2024-02-01T10:16:35.545966065Z","data":{"read_version":{"tag":"_VxqI9_Wg5TOlSHC_n8MLWjO","ver":1},"write_version":{"tag":"_VxqI9_Wg5TOlSHC_n8MLWjO","ver":2},"status":{"status":"complete"}}},{"id":"1_1706782595.664631_120.1","section":"bucket","name":"vclfcn-9","timestamp":"2024-02-01T10:16:35.664631193Z","data":{"read_version":{"tag":"_fop2OKI1GWzp_3SQxaIH8uR","ver":1},"write_version":{"tag":"_fop2OKI1GWzp_3SQxaIH8uR","ver":2},"status":{"status":"remove"}}},{"id":"1_1706782595.670952_121.1","section":"bucket","name":"vclfcn-9","timestamp":"2024-02-01T10:16:35.670952808Z","data":{"read_version":{"tag":"_fop2OKI1GWzp_3SQxaIH8uR","ver":1},"write_version":{"tag":"_fop2OKI1GWzp_3SQxaIH8uR","ver":2},"status":{"status":"complete"}}}] 2024-02-01T10:38:04.929 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '5dc4efbd-eccd-47a8-9471-ae4a209ec978', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:04.929 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '5dc4efbd-eccd-47a8-9471-ae4a209ec978', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:04.929 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 5dc4efbd-eccd-47a8-9471-ae4a209ec978 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:38:04.969 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:38:04.970 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:38:05.053 INFO:teuthology.orchestra.run.smithi187.stdout:[{"id":"1_1706782453.129936_29.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2024-02-01T10:14:13.129936524Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RY7up0tQcOf9ElajXOF5d6i","ver":1},"status":{"status":"write"}}},{"id":"1_1706782453.132619_30.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2024-02-01T10:14:13.132619785Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RY7up0tQcOf9ElajXOF5d6i","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782596.137698_33.1","section":"bucket.instance","name":"vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19","timestamp":"2024-02-01T10:16:36.137698180Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gSnH8BiIXGWkG2sGorT95Ca","ver":1},"status":{"status":"write"}}},{"id":"1_1706782596.141485_34.1","section":"bucket.instance","name":"vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19","timestamp":"2024-02-01T10:16:36.141485128Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gSnH8BiIXGWkG2sGorT95Ca","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782596.142883_35.1","section":"bucket","name":"vclfcn-10","timestamp":"2024-02-01T10:16:36.142883452Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_C-y9oz7otdaRTy0zc_QeA8H","ver":1},"status":{"status":"write"}}},{"id":"1_1706782596.147037_36.1","section":"bucket","name":"vclfcn-10","timestamp":"2024-02-01T10:16:36.147037349Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_C-y9oz7otdaRTy0zc_QeA8H","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782596.165713_37.1","section":"bucket.instance","name":"vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20","timestamp":"2024-02-01T10:16:36.165713463Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ugvw9Z3SeaHLB2NBY8cOvIs","ver":1},"status":{"status":"write"}}},{"id":"1_1706782596.170196_38.1","section":"bucket.instance","name":"vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20","timestamp":"2024-02-01T10:16:36.170196200Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ugvw9Z3SeaHLB2NBY8cOvIs","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782596.171951_39.1","section":"bucket","name":"vclfcn-11","timestamp":"2024-02-01T10:16:36.171951352Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_clR9bGsKa2Bt45u61U1vnaU","ver":1},"status":{"status":"write"}}},{"id":"1_1706782596.176142_40.1","section":"bucket","name":"vclfcn-11","timestamp":"2024-02-01T10:16:36.176142514Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_clR9bGsKa2Bt45u61U1vnaU","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782596.215001_41.1","section":"bucket.instance","name":"vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21","timestamp":"2024-02-01T10:16:36.215001761Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0aJjbbhzkBKwKh67jUyjFXU","ver":1},"status":{"status":"write"}}},{"id":"1_1706782596.221792_42.1","section":"bucket.instance","name":"vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21","timestamp":"2024-02-01T10:16:36.221792015Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0aJjbbhzkBKwKh67jUyjFXU","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782596.224377_43.1","section":"bucket","name":"vclfcn-12","timestamp":"2024-02-01T10:16:36.224377534Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YzQLQ2ZvUn8a9M8WMLPPlj-","ver":1},"status":{"status":"write"}}},{"id":"1_1706782596.230502_44.1","section":"bucket","name":"vclfcn-12","timestamp":"2024-02-01T10:16:36.230502418Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YzQLQ2ZvUn8a9M8WMLPPlj-","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782693.582816_53.1","section":"bucket.instance","name":"vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22","timestamp":"2024-02-01T10:18:13.582816318Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_IyMDVQZq1Dm4Ik60abXwswh","ver":1},"status":{"status":"write"}}},{"id":"1_1706782693.589569_54.1","section":"bucket.instance","name":"vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22","timestamp":"2024-02-01T10:18:13.589569915Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_IyMDVQZq1Dm4Ik60abXwswh","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782693.592623_55.1","section":"bucket","name":"vclfcn-13","timestamp":"2024-02-01T10:18:13.592623458Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SBtUJfjUPgWxvP4marStPj8","ver":1},"status":{"status":"write"}}},{"id":"1_1706782693.599222_56.1","section":"bucket","name":"vclfcn-13","timestamp":"2024-02-01T10:18:13.599222251Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SBtUJfjUPgWxvP4marStPj8","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782693.622788_57.1","section":"bucket.instance","name":"vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23","timestamp":"2024-02-01T10:18:13.622788043Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OLXZUmEaWvH6RgSaGLsC5gp","ver":1},"status":{"status":"write"}}},{"id":"1_1706782693.629469_58.1","section":"bucket.instance","name":"vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23","timestamp":"2024-02-01T10:18:13.629469201Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OLXZUmEaWvH6RgSaGLsC5gp","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782693.632089_59.1","section":"bucket","name":"vclfcn-14","timestamp":"2024-02-01T10:18:13.632089534Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6H3UQNuqVeMSp5sl1iVxf3G","ver":1},"status":{"status":"write"}}},{"id":"1_1706782693.638718_60.1","section":"bucket","name":"vclfcn-14","timestamp":"2024-02-01T10:18:13.638718897Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6H3UQNuqVeMSp5sl1iVxf3G","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782693.686988_61.1","section":"bucket.instance","name":"vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24","timestamp":"2024-02-01T10:18:13.686988320Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GhbTNO6qlpBiEMg1HJVBNAz","ver":1},"status":{"status":"write"}}},{"id":"1_1706782693.694347_62.1","section":"bucket.instance","name":"vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24","timestamp":"2024-02-01T10:18:13.694347551Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GhbTNO6qlpBiEMg1HJVBNAz","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782693.696886_63.1","section":"bucket","name":"vclfcn-15","timestamp":"2024-02-01T10:18:13.696886663Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PC87lFbJZuz6KYHILchNgZJ","ver":1},"status":{"status":"write"}}},{"id":"1_1706782693.703781_64.1","section":"bucket","name":"vclfcn-15","timestamp":"2024-02-01T10:18:13.703781181Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PC87lFbJZuz6KYHILchNgZJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782788.125323_68.1","section":"bucket.instance","name":"vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25","timestamp":"2024-02-01T10:19:48.125323856Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GJQLwPJ67-aampE80YQ6oYN","ver":1},"status":{"status":"write"}}},{"id":"1_1706782788.132848_69.1","section":"bucket.instance","name":"vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25","timestamp":"2024-02-01T10:19:48.132848685Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GJQLwPJ67-aampE80YQ6oYN","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782788.136003_70.1","section":"bucket","name":"vclfcn-16","timestamp":"2024-02-01T10:19:48.136003157Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yZQS2Ooyuo-DWcyRYk0NZG5","ver":1},"status":{"status":"write"}}},{"id":"1_1706782788.142977_71.1","section":"bucket","name":"vclfcn-16","timestamp":"2024-02-01T10:19:48.142977363Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yZQS2Ooyuo-DWcyRYk0NZG5","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782788.168776_72.1","section":"bucket.instance","name":"vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26","timestamp":"2024-02-01T10:19:48.168776620Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ds-zBeTuxJy65JHqqd4lb5Z","ver":1},"status":{"status":"write"}}},{"id":"1_1706782788.174951_73.1","section":"bucket.instance","name":"vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26","timestamp":"2024-02-01T10:19:48.174951550Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ds-zBeTuxJy65JHqqd4lb5Z","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782788.178001_74.1","section":"bucket","name":"vclfcn-17","timestamp":"2024-02-01T10:19:48.178001249Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZIbkkBCCgIhVLT8B2r1w5K_","ver":1},"status":{"status":"write"}}},{"id":"1_1706782788.184593_75.1","section":"bucket","name":"vclfcn-17","timestamp":"2024-02-01T10:19:48.184593776Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZIbkkBCCgIhVLT8B2r1w5K_","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782788.225397_76.1","section":"bucket.instance","name":"vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27","timestamp":"2024-02-01T10:19:48.225397011Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gkEVh6TTRHTmUu-UImISUWd","ver":1},"status":{"status":"write"}}},{"id":"1_1706782788.230219_77.1","section":"bucket.instance","name":"vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27","timestamp":"2024-02-01T10:19:48.230219600Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gkEVh6TTRHTmUu-UImISUWd","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782788.232911_78.1","section":"bucket","name":"vclfcn-18","timestamp":"2024-02-01T10:19:48.232911150Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HlQ9gchNi7E4cFgOusBiYBD","ver":1},"status":{"status":"write"}}},{"id":"1_1706782788.241244_79.1","section":"bucket","name":"vclfcn-18","timestamp":"2024-02-01T10:19:48.241244766Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HlQ9gchNi7E4cFgOusBiYBD","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782788.264154_80.1","section":"bucket.instance","name":"vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25","timestamp":"2024-02-01T10:19:48.264154269Z","data":{"read_version":{"tag":"_GJQLwPJ67-aampE80YQ6oYN","ver":1},"write_version":{"tag":"_GJQLwPJ67-aampE80YQ6oYN","ver":2},"status":{"status":"write"}}},{"id":"1_1706782788.268634_81.1","section":"bucket.instance","name":"vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25","timestamp":"2024-02-01T10:19:48.268634480Z","data":{"read_version":{"tag":"_GJQLwPJ67-aampE80YQ6oYN","ver":1},"write_version":{"tag":"_GJQLwPJ67-aampE80YQ6oYN","ver":2},"status":{"status":"complete"}}},{"id":"1_1706782788.275810_82.1","section":"bucket.instance","name":"vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26","timestamp":"2024-02-01T10:19:48.275810646Z","data":{"read_version":{"tag":"_ds-zBeTuxJy65JHqqd4lb5Z","ver":1},"write_version":{"tag":"_ds-zBeTuxJy65JHqqd4lb5Z","ver":2},"status":{"status":"write"}}},{"id":"1_1706782788.281245_83.1","section":"bucket.instance","name":"vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26","timestamp":"2024-02-01T10:19:48.281245680Z","data":{"read_version":{"tag":"_ds-zBeTuxJy65JHqqd4lb5Z","ver":1},"write_version":{"tag":"_ds-zBeTuxJy65JHqqd4lb5Z","ver":2},"status":{"status":"complete"}}},{"id":"1_1706782788.295656_84.1","section":"bucket.instance","name":"vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27","timestamp":"2024-02-01T10:19:48.295656474Z","data":{"read_version":{"tag":"_gkEVh6TTRHTmUu-UImISUWd","ver":1},"write_version":{"tag":"_gkEVh6TTRHTmUu-UImISUWd","ver":2},"status":{"status":"write"}}},{"id":"1_1706782788.299861_85.1","section":"bucket.instance","name":"vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27","timestamp":"2024-02-01T10:19:48.299861431Z","data":{"read_version":{"tag":"_gkEVh6TTRHTmUu-UImISUWd","ver":1},"write_version":{"tag":"_gkEVh6TTRHTmUu-UImISUWd","ver":2},"status":{"status":"complete"}}},{"id":"1_1706782985.577592_93.1","section":"bucket.instance","name":"vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28","timestamp":"2024-02-01T10:23:05.577592857Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Tkzhdhq2YxvoQoXNUGw-VlA","ver":1},"status":{"status":"write"}}},{"id":"1_1706782985.584276_94.1","section":"bucket.instance","name":"vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28","timestamp":"2024-02-01T10:23:05.584276295Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Tkzhdhq2YxvoQoXNUGw-VlA","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782985.586795_95.1","section":"bucket","name":"vclfcn-19","timestamp":"2024-02-01T10:23:05.586795365Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uFY0fAasS29yh69Gp_uVUE2","ver":1},"status":{"status":"write"}}},{"id":"1_1706782985.593052_96.1","section":"bucket","name":"vclfcn-19","timestamp":"2024-02-01T10:23:05.593052024Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uFY0fAasS29yh69Gp_uVUE2","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782985.602177_97.1","section":"bucket.instance","name":"vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28","timestamp":"2024-02-01T10:23:05.602177901Z","data":{"read_version":{"tag":"_Tkzhdhq2YxvoQoXNUGw-VlA","ver":1},"write_version":{"tag":"_Tkzhdhq2YxvoQoXNUGw-VlA","ver":2},"status":{"status":"write"}}},{"id":"1_1706782985.609410_98.1","section":"bucket.instance","name":"vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28","timestamp":"2024-02-01T10:23:05.609410401Z","data":{"read_version":{"tag":"_Tkzhdhq2YxvoQoXNUGw-VlA","ver":1},"write_version":{"tag":"_Tkzhdhq2YxvoQoXNUGw-VlA","ver":2},"status":{"status":"complete"}}},{"id":"1_1706783281.140663_112.1","section":"bucket.instance","name":"vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29","timestamp":"2024-02-01T10:28:01.140663757Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RY5J0cHq0mZCxKxUWbHq1nu","ver":1},"status":{"status":"write"}}},{"id":"1_1706783281.146879_113.1","section":"bucket.instance","name":"vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29","timestamp":"2024-02-01T10:28:01.146879624Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RY5J0cHq0mZCxKxUWbHq1nu","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783281.149599_114.1","section":"bucket","name":"vclfcn-20","timestamp":"2024-02-01T10:28:01.149599676Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LBpQRODD-fP4UvS_L3UYNrJ","ver":1},"status":{"status":"write"}}},{"id":"1_1706783281.156355_115.1","section":"bucket","name":"vclfcn-20","timestamp":"2024-02-01T10:28:01.156355553Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LBpQRODD-fP4UvS_L3UYNrJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783313.047690_117.1","section":"bucket.instance","name":"vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29","timestamp":"2024-02-01T10:28:33.047690221Z","data":{"read_version":{"tag":"_RY5J0cHq0mZCxKxUWbHq1nu","ver":1},"write_version":{"tag":"_RY5J0cHq0mZCxKxUWbHq1nu","ver":2},"status":{"status":"write"}}},{"id":"1_1706783313.054991_118.1","section":"bucket.instance","name":"vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29","timestamp":"2024-02-01T10:28:33.054991128Z","data":{"read_version":{"tag":"_RY5J0cHq0mZCxKxUWbHq1nu","ver":1},"write_version":{"tag":"_RY5J0cHq0mZCxKxUWbHq1nu","ver":2},"status":{"status":"complete"}}},{"id":"1_1706783510.479238_122.1","section":"bucket.instance","name":"vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29","timestamp":"2024-02-01T10:31:50.479238314Z","data":{"read_version":{"tag":"_RY5J0cHq0mZCxKxUWbHq1nu","ver":2},"write_version":{"tag":"_RY5J0cHq0mZCxKxUWbHq1nu","ver":3},"status":{"status":"write"}}},{"id":"1_1706783510.486411_123.1","section":"bucket.instance","name":"vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29","timestamp":"2024-02-01T10:31:50.486411437Z","data":{"read_version":{"tag":"_RY5J0cHq0mZCxKxUWbHq1nu","ver":2},"write_version":{"tag":"_RY5J0cHq0mZCxKxUWbHq1nu","ver":3},"status":{"status":"complete"}}},{"id":"1_1706783557.936493_125.1","section":"bucket.instance","name":"vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30","timestamp":"2024-02-01T10:32:37.936493302Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SBaAf2hqSgxj2ifisrCDUQ-","ver":1},"status":{"status":"write"}}},{"id":"1_1706783557.943553_126.1","section":"bucket.instance","name":"vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30","timestamp":"2024-02-01T10:32:37.943553989Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SBaAf2hqSgxj2ifisrCDUQ-","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783557.946282_127.1","section":"bucket","name":"vclfcn-21","timestamp":"2024-02-01T10:32:37.946282859Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_aS4q16MZBGYstqXzbnAETLq","ver":1},"status":{"status":"write"}}},{"id":"1_1706783557.952481_128.1","section":"bucket","name":"vclfcn-21","timestamp":"2024-02-01T10:32:37.952481787Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_aS4q16MZBGYstqXzbnAETLq","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783557.977604_129.1","section":"bucket.instance","name":"vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31","timestamp":"2024-02-01T10:32:37.977604938Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AivkNk9zM8C_ACc4CMMCA9j","ver":1},"status":{"status":"write"}}},{"id":"1_1706783557.984354_130.1","section":"bucket.instance","name":"vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31","timestamp":"2024-02-01T10:32:37.984354568Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AivkNk9zM8C_ACc4CMMCA9j","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783557.986976_131.1","section":"bucket","name":"vclfcn-22","timestamp":"2024-02-01T10:32:37.986976409Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OC84jTcY1mo5V4XtRR3FK81","ver":1},"status":{"status":"write"}}},{"id":"1_1706783557.993660_132.1","section":"bucket","name":"vclfcn-22","timestamp":"2024-02-01T10:32:37.993660224Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OC84jTcY1mo5V4XtRR3FK81","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783558.040696_133.1","section":"bucket.instance","name":"vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32","timestamp":"2024-02-01T10:32:38.040696413Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_y0EPEo2ASjenysOQ86zdnP6","ver":1},"status":{"status":"write"}}},{"id":"1_1706783558.047555_134.1","section":"bucket.instance","name":"vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32","timestamp":"2024-02-01T10:32:38.047555506Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_y0EPEo2ASjenysOQ86zdnP6","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783558.049874_135.1","section":"bucket","name":"vclfcn-23","timestamp":"2024-02-01T10:32:38.049874456Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pctO5Jr19wCuuci8zhYun2q","ver":1},"status":{"status":"write"}}},{"id":"1_1706783558.055502_136.1","section":"bucket","name":"vclfcn-23","timestamp":"2024-02-01T10:32:38.055502136Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pctO5Jr19wCuuci8zhYun2q","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783558.081356_137.1","section":"bucket.instance","name":"vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30","timestamp":"2024-02-01T10:32:38.081356052Z","data":{"read_version":{"tag":"_SBaAf2hqSgxj2ifisrCDUQ-","ver":1},"write_version":{"tag":"_SBaAf2hqSgxj2ifisrCDUQ-","ver":2},"status":{"status":"write"}}},{"id":"1_1706783558.088380_138.1","section":"bucket.instance","name":"vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30","timestamp":"2024-02-01T10:32:38.088380225Z","data":{"read_version":{"tag":"_SBaAf2hqSgxj2ifisrCDUQ-","ver":1},"write_version":{"tag":"_SBaAf2hqSgxj2ifisrCDUQ-","ver":2},"status":{"status":"complete"}}},{"id":"1_1706783558.095619_139.1","section":"bucket.instance","name":"vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31","timestamp":"2024-02-01T10:32:38.095619390Z","data":{"read_version":{"tag":"_AivkNk9zM8C_ACc4CMMCA9j","ver":1},"write_version":{"tag":"_AivkNk9zM8C_ACc4CMMCA9j","ver":2},"status":{"status":"write"}}},{"id":"1_1706783558.100472_140.1","section":"bucket.instance","name":"vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31","timestamp":"2024-02-01T10:32:38.100472386Z","data":{"read_version":{"tag":"_AivkNk9zM8C_ACc4CMMCA9j","ver":1},"write_version":{"tag":"_AivkNk9zM8C_ACc4CMMCA9j","ver":2},"status":{"status":"complete"}}},{"id":"1_1706783558.110239_141.1","section":"bucket.instance","name":"vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32","timestamp":"2024-02-01T10:32:38.110239478Z","data":{"read_version":{"tag":"_y0EPEo2ASjenysOQ86zdnP6","ver":1},"write_version":{"tag":"_y0EPEo2ASjenysOQ86zdnP6","ver":2},"status":{"status":"write"}}},{"id":"1_1706783558.115340_142.1","section":"bucket.instance","name":"vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32","timestamp":"2024-02-01T10:32:38.115340472Z","data":{"read_version":{"tag":"_y0EPEo2ASjenysOQ86zdnP6","ver":1},"write_version":{"tag":"_y0EPEo2ASjenysOQ86zdnP6","ver":2},"status":{"status":"complete"}}},{"id":"1_1706783613.485844_144.1","section":"bucket.instance","name":"vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33","timestamp":"2024-02-01T10:33:33.485844007Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KQaPa_IyHr1JRoHAufGblmN","ver":1},"status":{"status":"write"}}},{"id":"1_1706783613.493076_145.1","section":"bucket.instance","name":"vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33","timestamp":"2024-02-01T10:33:33.493076639Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KQaPa_IyHr1JRoHAufGblmN","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783613.495609_146.1","section":"bucket","name":"vclfcn-24","timestamp":"2024-02-01T10:33:33.495609093Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wdRz7umYZCWDz61bnbDJ5DR","ver":1},"status":{"status":"write"}}},{"id":"1_1706783613.501842_147.1","section":"bucket","name":"vclfcn-24","timestamp":"2024-02-01T10:33:33.501842098Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wdRz7umYZCWDz61bnbDJ5DR","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783613.526071_148.1","section":"bucket.instance","name":"vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34","timestamp":"2024-02-01T10:33:33.526071464Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TJp0m8FTngleqfn93NSNEFQ","ver":1},"status":{"status":"write"}}},{"id":"1_1706783613.533006_149.1","section":"bucket.instance","name":"vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34","timestamp":"2024-02-01T10:33:33.533006576Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TJp0m8FTngleqfn93NSNEFQ","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783613.535429_150.1","section":"bucket","name":"vclfcn-25","timestamp":"2024-02-01T10:33:33.535429751Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XUEOg4Lx9rBHbAJwKq1lwrJ","ver":1},"status":{"status":"write"}}},{"id":"1_1706783613.541933_151.1","section":"bucket","name":"vclfcn-25","timestamp":"2024-02-01T10:33:33.541933857Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XUEOg4Lx9rBHbAJwKq1lwrJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783613.587133_152.1","section":"bucket.instance","name":"vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35","timestamp":"2024-02-01T10:33:33.587133123Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_djT8hpdivEzfKbjzZM1tGYc","ver":1},"status":{"status":"write"}}},{"id":"1_1706783613.594437_153.1","section":"bucket.instance","name":"vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35","timestamp":"2024-02-01T10:33:33.594437539Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_djT8hpdivEzfKbjzZM1tGYc","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783613.597205_154.1","section":"bucket","name":"vclfcn-26","timestamp":"2024-02-01T10:33:33.597205142Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3fH7UbMU-OxdukYrx1ZeZDR","ver":1},"status":{"status":"write"}}},{"id":"1_1706783613.604282_155.1","section":"bucket","name":"vclfcn-26","timestamp":"2024-02-01T10:33:33.604282859Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3fH7UbMU-OxdukYrx1ZeZDR","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783613.632812_156.1","section":"bucket.instance","name":"vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33","timestamp":"2024-02-01T10:33:33.632812026Z","data":{"read_version":{"tag":"_KQaPa_IyHr1JRoHAufGblmN","ver":1},"write_version":{"tag":"_KQaPa_IyHr1JRoHAufGblmN","ver":2},"status":{"status":"write"}}},{"id":"1_1706783613.639457_157.1","section":"bucket.instance","name":"vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33","timestamp":"2024-02-01T10:33:33.639457857Z","data":{"read_version":{"tag":"_KQaPa_IyHr1JRoHAufGblmN","ver":1},"write_version":{"tag":"_KQaPa_IyHr1JRoHAufGblmN","ver":2},"status":{"status":"complete"}}},{"id":"1_1706783613.645309_158.1","section":"bucket.instance","name":"vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33","timestamp":"2024-02-01T10:33:33.645309049Z","data":{"read_version":{"tag":"_KQaPa_IyHr1JRoHAufGblmN","ver":2},"write_version":{"tag":"_KQaPa_IyHr1JRoHAufGblmN","ver":3},"status":{"status":"write"}}},{"id":"1_1706783613.652911_159.1","section":"bucket.instance","name":"vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33","timestamp":"2024-02-01T10:33:33.652911988Z","data":{"read_version":{"tag":"_KQaPa_IyHr1JRoHAufGblmN","ver":2},"write_version":{"tag":"_KQaPa_IyHr1JRoHAufGblmN","ver":3},"status":{"status":"complete"}}},{"id":"1_1706783613.662523_160.1","section":"bucket.instance","name":"vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34","timestamp":"2024-02-01T10:33:33.662523190Z","data":{"read_version":{"tag":"_TJp0m8FTngleqfn93NSNEFQ","ver":1},"write_version":{"tag":"_TJp0m8FTngleqfn93NSNEFQ","ver":2},"status":{"status":"write"}}},{"id":"1_1706783613.674476_161.1","section":"bucket.instance","name":"vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34","timestamp":"2024-02-01T10:33:33.674476638Z","data":{"read_version":{"tag":"_TJp0m8FTngleqfn93NSNEFQ","ver":1},"write_version":{"tag":"_TJp0m8FTngleqfn93NSNEFQ","ver":2},"status":{"status":"complete"}}},{"id":"1_1706783613.698284_162.1","section":"bucket.instance","name":"vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34","timestamp":"2024-02-01T10:33:33.698284010Z","data":{"read_version":{"tag":"_TJp0m8FTngleqfn93NSNEFQ","ver":2},"write_version":{"tag":"_TJp0m8FTngleqfn93NSNEFQ","ver":3},"status":{"status":"write"}}},{"id":"1_1706783613.707513_163.1","section":"bucket.instance","name":"vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34","timestamp":"2024-02-01T10:33:33.707513991Z","data":{"read_version":{"tag":"_TJp0m8FTngleqfn93NSNEFQ","ver":2},"write_version":{"tag":"_TJp0m8FTngleqfn93NSNEFQ","ver":3},"status":{"status":"complete"}}},{"id":"1_1706783613.728949_164.1","section":"bucket.instance","name":"vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35","timestamp":"2024-02-01T10:33:33.728949508Z","data":{"read_version":{"tag":"_djT8hpdivEzfKbjzZM1tGYc","ver":1},"write_version":{"tag":"_djT8hpdivEzfKbjzZM1tGYc","ver":2},"status":{"status":"write"}}},{"id":"1_1706783613.736733_165.1","section":"bucket.instance","name":"vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35","timestamp":"2024-02-01T10:33:33.736733042Z","data":{"read_version":{"tag":"_djT8hpdivEzfKbjzZM1tGYc","ver":1},"write_version":{"tag":"_djT8hpdivEzfKbjzZM1tGYc","ver":2},"status":{"status":"complete"}}},{"id":"1_1706783613.748133_166.1","section":"bucket.instance","name":"vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35","timestamp":"2024-02-01T10:33:33.748133962Z","data":{"read_version":{"tag":"_djT8hpdivEzfKbjzZM1tGYc","ver":2},"write_version":{"tag":"_djT8hpdivEzfKbjzZM1tGYc","ver":3},"status":{"status":"write"}}},{"id":"1_1706783613.755304_167.1","section":"bucket.instance","name":"vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35","timestamp":"2024-02-01T10:33:33.755304841Z","data":{"read_version":{"tag":"_djT8hpdivEzfKbjzZM1tGYc","ver":2},"write_version":{"tag":"_djT8hpdivEzfKbjzZM1tGYc","ver":3},"status":{"status":"complete"}}},{"id":"1_1706783732.990670_178.1","section":"bucket.instance","name":"vclfcn-27:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.36","timestamp":"2024-02-01T10:35:32.990670009Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rYzBOFkXYI9xqwLbwgqkEVW","ver":1},"status":{"status":"write"}}},{"id":"1_1706783732.996955_179.1","section":"bucket.instance","name":"vclfcn-27:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.36","timestamp":"2024-02-01T10:35:32.996955046Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rYzBOFkXYI9xqwLbwgqkEVW","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783732.999583_180.1","section":"bucket","name":"vclfcn-27","timestamp":"2024-02-01T10:35:32.999583128Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9cHBpx7BkJp-jUw4o1CRrfh","ver":1},"status":{"status":"write"}}},{"id":"1_1706783733.006036_181.1","section":"bucket","name":"vclfcn-27","timestamp":"2024-02-01T10:35:33.006036701Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9cHBpx7BkJp-jUw4o1CRrfh","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783733.034354_182.1","section":"bucket.instance","name":"vclfcn-28:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.37","timestamp":"2024-02-01T10:35:33.034354608Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ppjqf2iiQQLeeW9nmcMFwt7","ver":1},"status":{"status":"write"}}},{"id":"1_1706783733.041168_183.1","section":"bucket.instance","name":"vclfcn-28:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.37","timestamp":"2024-02-01T10:35:33.041168668Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ppjqf2iiQQLeeW9nmcMFwt7","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783733.043826_184.1","section":"bucket","name":"vclfcn-28","timestamp":"2024-02-01T10:35:33.043826329Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_O0D5bB8GV3hYxs2j9GMFn0q","ver":1},"status":{"status":"write"}}},{"id":"1_1706783733.050085_185.1","section":"bucket","name":"vclfcn-28","timestamp":"2024-02-01T10:35:33.050085404Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_O0D5bB8GV3hYxs2j9GMFn0q","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783733.095185_186.1","section":"bucket.instance","name":"vclfcn-29:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.38","timestamp":"2024-02-01T10:35:33.095185012Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_b7qqNp_v7vEEA2IVwp_dZhU","ver":1},"status":{"status":"write"}}},{"id":"1_1706783733.100262_187.1","section":"bucket.instance","name":"vclfcn-29:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.38","timestamp":"2024-02-01T10:35:33.100262420Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_b7qqNp_v7vEEA2IVwp_dZhU","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783733.102142_188.1","section":"bucket","name":"vclfcn-29","timestamp":"2024-02-01T10:35:33.102142458Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_oH5-kep0dGu6Rb8cPoOFEHk","ver":1},"status":{"status":"write"}}},{"id":"1_1706783733.107054_189.1","section":"bucket","name":"vclfcn-29","timestamp":"2024-02-01T10:35:33.107054109Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_oH5-kep0dGu6Rb8cPoOFEHk","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783733.136865_190.1","section":"bucket.instance","name":"vclfcn-27:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.36","timestamp":"2024-02-01T10:35:33.136865088Z","data":{"read_version":{"tag":"_rYzBOFkXYI9xqwLbwgqkEVW","ver":1},"write_version":{"tag":"_rYzBOFkXYI9xqwLbwgqkEVW","ver":2},"status":{"status":"write"}}},{"id":"1_1706783733.148545_191.1","section":"bucket.instance","name":"vclfcn-27:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.36","timestamp":"2024-02-01T10:35:33.148545378Z","data":{"read_version":{"tag":"_rYzBOFkXYI9xqwLbwgqkEVW","ver":1},"write_version":{"tag":"_rYzBOFkXYI9xqwLbwgqkEVW","ver":2},"status":{"status":"complete"}}},{"id":"1_1706783733.202069_192.1","section":"bucket.instance","name":"vclfcn-28:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.37","timestamp":"2024-02-01T10:35:33.202069645Z","data":{"read_version":{"tag":"_Ppjqf2iiQQLeeW9nmcMFwt7","ver":1},"write_version":{"tag":"_Ppjqf2iiQQLeeW9nmcMFwt7","ver":2},"status":{"status":"write"}}},{"id":"1_1706783733.207436_193.1","section":"bucket.instance","name":"vclfcn-28:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.37","timestamp":"2024-02-01T10:35:33.207436436Z","data":{"read_version":{"tag":"_Ppjqf2iiQQLeeW9nmcMFwt7","ver":1},"write_version":{"tag":"_Ppjqf2iiQQLeeW9nmcMFwt7","ver":2},"status":{"status":"complete"}}},{"id":"1_1706783733.222466_194.1","section":"bucket.instance","name":"vclfcn-29:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.38","timestamp":"2024-02-01T10:35:33.222466987Z","data":{"read_version":{"tag":"_b7qqNp_v7vEEA2IVwp_dZhU","ver":1},"write_version":{"tag":"_b7qqNp_v7vEEA2IVwp_dZhU","ver":2},"status":{"status":"write"}}},{"id":"1_1706783733.227246_195.1","section":"bucket.instance","name":"vclfcn-29:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.38","timestamp":"2024-02-01T10:35:33.227246173Z","data":{"read_version":{"tag":"_b7qqNp_v7vEEA2IVwp_dZhU","ver":1},"write_version":{"tag":"_b7qqNp_v7vEEA2IVwp_dZhU","ver":2},"status":{"status":"complete"}}},{"id":"1_1706783733.293745_196.1","section":"bucket.instance","name":"vclfcn-30:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.39","timestamp":"2024-02-01T10:35:33.293745968Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HhnF-4-cQEYFZsmhVgAo_3W","ver":1},"status":{"status":"write"}}},{"id":"1_1706783733.299312_197.1","section":"bucket.instance","name":"vclfcn-30:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.39","timestamp":"2024-02-01T10:35:33.299312767Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HhnF-4-cQEYFZsmhVgAo_3W","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783733.301968_198.1","section":"bucket","name":"vclfcn-30","timestamp":"2024-02-01T10:35:33.301968348Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Yct8ulBblkK4SDKF5OWcOcv","ver":1},"status":{"status":"write"}}},{"id":"1_1706783733.307848_199.1","section":"bucket","name":"vclfcn-30","timestamp":"2024-02-01T10:35:33.307848983Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Yct8ulBblkK4SDKF5OWcOcv","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783733.330768_200.1","section":"bucket.instance","name":"vclfcn-31:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.40","timestamp":"2024-02-01T10:35:33.330768162Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OsUE1ilsRE6kOECGtCDtxHd","ver":1},"status":{"status":"write"}}},{"id":"1_1706783733.336478_201.1","section":"bucket.instance","name":"vclfcn-31:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.40","timestamp":"2024-02-01T10:35:33.336478192Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OsUE1ilsRE6kOECGtCDtxHd","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783733.338988_202.1","section":"bucket","name":"vclfcn-31","timestamp":"2024-02-01T10:35:33.338988237Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5E7wmMDv9o08f28guv8WeNq","ver":1},"status":{"status":"write"}}},{"id":"1_1706783733.345150_203.1","section":"bucket","name":"vclfcn-31","timestamp":"2024-02-01T10:35:33.345150939Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5E7wmMDv9o08f28guv8WeNq","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783733.391532_204.1","section":"bucket.instance","name":"vclfcn-32:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.41","timestamp":"2024-02-01T10:35:33.391532601Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PICgXYT5_p4s6Npm6_d7L86","ver":1},"status":{"status":"write"}}},{"id":"1_1706783733.396882_205.1","section":"bucket.instance","name":"vclfcn-32:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.41","timestamp":"2024-02-01T10:35:33.396882656Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PICgXYT5_p4s6Npm6_d7L86","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783733.399225_206.1","section":"bucket","name":"vclfcn-32","timestamp":"2024-02-01T10:35:33.399225264Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jXGNidsm4ya5U_EGuN9pxg2","ver":1},"status":{"status":"write"}}},{"id":"1_1706783733.404709_207.1","section":"bucket","name":"vclfcn-32","timestamp":"2024-02-01T10:35:33.404709608Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jXGNidsm4ya5U_EGuN9pxg2","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783733.435292_208.1","section":"bucket.instance","name":"vclfcn-30:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.39","timestamp":"2024-02-01T10:35:33.435292002Z","data":{"read_version":{"tag":"_HhnF-4-cQEYFZsmhVgAo_3W","ver":1},"write_version":{"tag":"_HhnF-4-cQEYFZsmhVgAo_3W","ver":2},"status":{"status":"write"}}},{"id":"1_1706783733.441533_209.1","section":"bucket.instance","name":"vclfcn-30:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.39","timestamp":"2024-02-01T10:35:33.441533749Z","data":{"read_version":{"tag":"_HhnF-4-cQEYFZsmhVgAo_3W","ver":1},"write_version":{"tag":"_HhnF-4-cQEYFZsmhVgAo_3W","ver":2},"status":{"status":"complete"}}},{"id":"1_1706783733.454934_210.1","section":"bucket.instance","name":"vclfcn-31:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.40","timestamp":"2024-02-01T10:35:33.454934992Z","data":{"read_version":{"tag":"_OsUE1ilsRE6kOECGtCDtxHd","ver":1},"write_version":{"tag":"_OsUE1ilsRE6kOECGtCDtxHd","ver":2},"status":{"status":"write"}}},{"id":"1_1706783733.460796_211.1","section":"bucket.instance","name":"vclfcn-31:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.40","timestamp":"2024-02-01T10:35:33.460796234Z","data":{"read_version":{"tag":"_OsUE1ilsRE6kOECGtCDtxHd","ver":1},"write_version":{"tag":"_OsUE1ilsRE6kOECGtCDtxHd","ver":2},"status":{"status":"complete"}}},{"id":"1_1706783733.521076_212.1","section":"bucket.instance","name":"vclfcn-32:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.41","timestamp":"2024-02-01T10:35:33.521076539Z","data":{"read_version":{"tag":"_PICgXYT5_p4s6Npm6_d7L86","ver":1},"write_version":{"tag":"_PICgXYT5_p4s6Npm6_d7L86","ver":2},"status":{"status":"write"}}},{"id":"1_1706783733.537903_213.1","section":"bucket.instance","name":"vclfcn-32:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.41","timestamp":"2024-02-01T10:35:33.537903176Z","data":{"read_version":{"tag":"_PICgXYT5_p4s6Npm6_d7L86","ver":1},"write_version":{"tag":"_PICgXYT5_p4s6Npm6_d7L86","ver":2},"status":{"status":"complete"}}},{"id":"1_1706783733.593359_214.1","section":"bucket.instance","name":"vclfcn-33:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.42","timestamp":"2024-02-01T10:35:33.593359133Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MHNCkb6d03Mpbe8ExDNyOIe","ver":1},"status":{"status":"write"}}},{"id":"1_1706783733.603919_215.1","section":"bucket.instance","name":"vclfcn-33:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.42","timestamp":"2024-02-01T10:35:33.603919491Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MHNCkb6d03Mpbe8ExDNyOIe","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783733.605766_216.1","section":"bucket","name":"vclfcn-33","timestamp":"2024-02-01T10:35:33.605766755Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_t7Tsi3a8X7oyOSKwXbGKwOO","ver":1},"status":{"status":"write"}}},{"id":"1_1706783733.609698_217.1","section":"bucket","name":"vclfcn-33","timestamp":"2024-02-01T10:35:33.609698777Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_t7Tsi3a8X7oyOSKwXbGKwOO","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783733.628224_218.1","section":"bucket.instance","name":"vclfcn-34:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.43","timestamp":"2024-02-01T10:35:33.628224591Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_V7tzE1wcwyuANojoierxhFz","ver":1},"status":{"status":"write"}}},{"id":"1_1706783733.632156_219.1","section":"bucket.instance","name":"vclfcn-34:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.43","timestamp":"2024-02-01T10:35:33.632156395Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_V7tzE1wcwyuANojoierxhFz","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783733.633573_220.1","section":"bucket","name":"vclfcn-34","timestamp":"2024-02-01T10:35:33.633573750Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Z5byZtsMURp7i2bsj6dBpHv","ver":1},"status":{"status":"write"}}},{"id":"1_1706783733.637172_221.1","section":"bucket","name":"vclfcn-34","timestamp":"2024-02-01T10:35:33.637172728Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Z5byZtsMURp7i2bsj6dBpHv","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783733.677953_222.1","section":"bucket.instance","name":"vclfcn-35:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.44","timestamp":"2024-02-01T10:35:33.677953821Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_C_0FSFOkL0ACO1-gSETb17o","ver":1},"status":{"status":"write"}}},{"id":"1_1706783733.683763_223.1","section":"bucket.instance","name":"vclfcn-35:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.44","timestamp":"2024-02-01T10:35:33.683763713Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_C_0FSFOkL0ACO1-gSETb17o","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783733.686120_224.1","section":"bucket","name":"vclfcn-35","timestamp":"2024-02-01T10:35:33.686120912Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CzHizcA9wKsnJKxAmY_lmbG","ver":1},"status":{"status":"write"}}},{"id":"1_1706783733.691403_225.1","section":"bucket","name":"vclfcn-35","timestamp":"2024-02-01T10:35:33.691403393Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CzHizcA9wKsnJKxAmY_lmbG","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783733.722103_226.1","section":"bucket.instance","name":"vclfcn-33:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.42","timestamp":"2024-02-01T10:35:33.722103624Z","data":{"read_version":{"tag":"_MHNCkb6d03Mpbe8ExDNyOIe","ver":1},"write_version":{"tag":"_MHNCkb6d03Mpbe8ExDNyOIe","ver":2},"status":{"status":"write"}}},{"id":"1_1706783733.727948_227.1","section":"bucket.instance","name":"vclfcn-33:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.42","timestamp":"2024-02-01T10:35:33.727948851Z","data":{"read_version":{"tag":"_MHNCkb6d03Mpbe8ExDNyOIe","ver":1},"write_version":{"tag":"_MHNCkb6d03Mpbe8ExDNyOIe","ver":2},"status":{"status":"complete"}}},{"id":"1_1706783734.739760_228.1","section":"bucket.instance","name":"vclfcn-34:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.43","timestamp":"2024-02-01T10:35:34.739760076Z","data":{"read_version":{"tag":"_V7tzE1wcwyuANojoierxhFz","ver":1},"write_version":{"tag":"_V7tzE1wcwyuANojoierxhFz","ver":2},"status":{"status":"write"}}},{"id":"1_1706783734.747549_229.1","section":"bucket.instance","name":"vclfcn-34:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.43","timestamp":"2024-02-01T10:35:34.747549598Z","data":{"read_version":{"tag":"_V7tzE1wcwyuANojoierxhFz","ver":1},"write_version":{"tag":"_V7tzE1wcwyuANojoierxhFz","ver":2},"status":{"status":"complete"}}},{"id":"1_1706783735.800850_230.1","section":"bucket.instance","name":"vclfcn-35:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.44","timestamp":"2024-02-01T10:35:35.800850730Z","data":{"read_version":{"tag":"_C_0FSFOkL0ACO1-gSETb17o","ver":1},"write_version":{"tag":"_C_0FSFOkL0ACO1-gSETb17o","ver":2},"status":{"status":"write"}}},{"id":"1_1706783735.808526_231.1","section":"bucket.instance","name":"vclfcn-35:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.44","timestamp":"2024-02-01T10:35:35.808526056Z","data":{"read_version":{"tag":"_C_0FSFOkL0ACO1-gSETb17o","ver":1},"write_version":{"tag":"_C_0FSFOkL0ACO1-gSETb17o","ver":2},"status":{"status":"complete"}}},{"id":"1_1706783735.837558_232.1","section":"bucket.instance","name":"vclfcn-36:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.45","timestamp":"2024-02-01T10:35:35.837558347Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9X_9QezGXcrZpslOj2s6DRz","ver":1},"status":{"status":"write"}}},{"id":"1_1706783735.844352_233.1","section":"bucket.instance","name":"vclfcn-36:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.45","timestamp":"2024-02-01T10:35:35.844352448Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9X_9QezGXcrZpslOj2s6DRz","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783735.846902_234.1","section":"bucket","name":"vclfcn-36","timestamp":"2024-02-01T10:35:35.846902742Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_I6xC7jnhAySZruK6KFfJcFj","ver":1},"status":{"status":"write"}}},{"id":"1_1706783735.853365_235.1","section":"bucket","name":"vclfcn-36","timestamp":"2024-02-01T10:35:35.853365976Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_I6xC7jnhAySZruK6KFfJcFj","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783735.880850_236.1","section":"bucket.instance","name":"vclfcn-37:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.46","timestamp":"2024-02-01T10:35:35.880850867Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_S59hS1hjhhTMEjgkmPP0l40","ver":1},"status":{"status":"write"}}},{"id":"1_1706783735.888598_237.1","section":"bucket.instance","name":"vclfcn-37:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.46","timestamp":"2024-02-01T10:35:35.888598691Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_S59hS1hjhhTMEjgkmPP0l40","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783735.891588_238.1","section":"bucket","name":"vclfcn-37","timestamp":"2024-02-01T10:35:35.891588303Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_u9b9H7g55Gxvm7UwsBsXRkc","ver":1},"status":{"status":"write"}}},{"id":"1_1706783735.903784_239.1","section":"bucket","name":"vclfcn-37","timestamp":"2024-02-01T10:35:35.903784912Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_u9b9H7g55Gxvm7UwsBsXRkc","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783735.956302_240.1","section":"bucket.instance","name":"vclfcn-38:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.47","timestamp":"2024-02-01T10:35:35.956302005Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_f3LeuHUV7bKmB4iX_c1NfhG","ver":1},"status":{"status":"write"}}},{"id":"1_1706783735.967905_241.1","section":"bucket.instance","name":"vclfcn-38:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.47","timestamp":"2024-02-01T10:35:35.967905199Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_f3LeuHUV7bKmB4iX_c1NfhG","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783735.969939_242.1","section":"bucket","name":"vclfcn-38","timestamp":"2024-02-01T10:35:35.969939474Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kCOsVG7gpmdckTvQ0-OqkTT","ver":1},"status":{"status":"write"}}},{"id":"1_1706783735.974063_243.1","section":"bucket","name":"vclfcn-38","timestamp":"2024-02-01T10:35:35.974063216Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kCOsVG7gpmdckTvQ0-OqkTT","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783746.868400_244.1","section":"bucket.instance","name":"vclfcn-39:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.48","timestamp":"2024-02-01T10:35:46.868400756Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2ismKVnh0upW2_YAiZ3dCm0","ver":1},"status":{"status":"write"}}},{"id":"1_1706783746.874991_245.1","section":"bucket.instance","name":"vclfcn-39:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.48","timestamp":"2024-02-01T10:35:46.874991156Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2ismKVnh0upW2_YAiZ3dCm0","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783746.877780_246.1","section":"bucket","name":"vclfcn-39","timestamp":"2024-02-01T10:35:46.877780982Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Mtq3wgmpQ2UVbTzYfiOTADK","ver":1},"status":{"status":"write"}}},{"id":"1_1706783746.884330_247.1","section":"bucket","name":"vclfcn-39","timestamp":"2024-02-01T10:35:46.884330514Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Mtq3wgmpQ2UVbTzYfiOTADK","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783746.913795_248.1","section":"bucket.instance","name":"vclfcn-40:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.49","timestamp":"2024-02-01T10:35:46.913795216Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wB4GCZVccemaw1QY5tXPf2_","ver":1},"status":{"status":"write"}}},{"id":"1_1706783746.921924_249.1","section":"bucket.instance","name":"vclfcn-40:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.49","timestamp":"2024-02-01T10:35:46.921924008Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wB4GCZVccemaw1QY5tXPf2_","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783746.925325_250.1","section":"bucket","name":"vclfcn-40","timestamp":"2024-02-01T10:35:46.925325542Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_DzmwhWoct-KkCaXHuhCBDRu","ver":1},"status":{"status":"write"}}},{"id":"1_1706783746.934867_251.1","section":"bucket","name":"vclfcn-40","timestamp":"2024-02-01T10:35:46.934867482Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_DzmwhWoct-KkCaXHuhCBDRu","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783746.990514_252.1","section":"bucket.instance","name":"vclfcn-41:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.50","timestamp":"2024-02-01T10:35:46.990514760Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TYFOExYyROxXd60biRdH4WR","ver":1},"status":{"status":"write"}}},{"id":"1_1706783746.999155_253.1","section":"bucket.instance","name":"vclfcn-41:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.50","timestamp":"2024-02-01T10:35:46.999155781Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TYFOExYyROxXd60biRdH4WR","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783747.001868_254.1","section":"bucket","name":"vclfcn-41","timestamp":"2024-02-01T10:35:47.001868659Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-h9FvYOZI-YjsdziH0V4lBk","ver":1},"status":{"status":"write"}}},{"id":"1_1706783747.007639_255.1","section":"bucket","name":"vclfcn-41","timestamp":"2024-02-01T10:35:47.007639832Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-h9FvYOZI-YjsdziH0V4lBk","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782574.724766_44.1","section":"bucket.instance","name":"vclfcn-1:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.1","timestamp":"2024-02-01T10:16:14.724766052Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ibyZ4ny-dO1eyNWrgR0xDdL","ver":1},"status":{"status":"write"}}},{"id":"1_1706782574.747809_45.1","section":"bucket.instance","name":"vclfcn-1:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.1","timestamp":"2024-02-01T10:16:14.747809032Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ibyZ4ny-dO1eyNWrgR0xDdL","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782574.750145_46.1","section":"bucket","name":"vclfcn-1","timestamp":"2024-02-01T10:16:14.750145401Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_N_oVpqg2avT9RyIn3n93zXI","ver":1},"status":{"status":"write"}}},{"id":"1_1706782574.755240_47.1","section":"bucket","name":"vclfcn-1","timestamp":"2024-02-01T10:16:14.755240100Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_N_oVpqg2avT9RyIn3n93zXI","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782574.779443_48.1","section":"bucket.instance","name":"vclfcn-2:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.2","timestamp":"2024-02-01T10:16:14.779443186Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_50A5h7IO153g3aLFcIk9L8F","ver":1},"status":{"status":"write"}}},{"id":"1_1706782574.784819_49.1","section":"bucket.instance","name":"vclfcn-2:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.2","timestamp":"2024-02-01T10:16:14.784819298Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_50A5h7IO153g3aLFcIk9L8F","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782574.786565_50.1","section":"bucket","name":"vclfcn-2","timestamp":"2024-02-01T10:16:14.786565198Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_t8Dr2abL8IoMUXCcMicPxaz","ver":1},"status":{"status":"write"}}},{"id":"1_1706782574.791675_51.1","section":"bucket","name":"vclfcn-2","timestamp":"2024-02-01T10:16:14.791675561Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_t8Dr2abL8IoMUXCcMicPxaz","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782577.457619_52.1","section":"bucket.instance","name":"vclfcn-3:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.3","timestamp":"2024-02-01T10:16:17.457619330Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AARS1itaxavcuaxye1dSdHX","ver":1},"status":{"status":"write"}}},{"id":"1_1706782577.467024_53.1","section":"bucket.instance","name":"vclfcn-3:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.3","timestamp":"2024-02-01T10:16:17.467024067Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AARS1itaxavcuaxye1dSdHX","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782577.469499_54.1","section":"bucket","name":"vclfcn-3","timestamp":"2024-02-01T10:16:17.469499300Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dY9nFqtzaiF6gs-LUJpzNgy","ver":1},"status":{"status":"write"}}},{"id":"1_1706782577.476062_55.1","section":"bucket","name":"vclfcn-3","timestamp":"2024-02-01T10:16:17.476062234Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dY9nFqtzaiF6gs-LUJpzNgy","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782583.048966_56.1","section":"bucket.instance","name":"vclfcn-4:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.4","timestamp":"2024-02-01T10:16:23.048966924Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jZZOT-7qYVNFmmz--YeJW8c","ver":1},"status":{"status":"write"}}},{"id":"1_1706782583.053776_57.1","section":"bucket.instance","name":"vclfcn-4:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.4","timestamp":"2024-02-01T10:16:23.053776598Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jZZOT-7qYVNFmmz--YeJW8c","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782583.055771_58.1","section":"bucket","name":"vclfcn-4","timestamp":"2024-02-01T10:16:23.055771006Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7FYomkbpFjXsMEVUf--3FIN","ver":1},"status":{"status":"write"}}},{"id":"1_1706782583.060768_59.1","section":"bucket","name":"vclfcn-4","timestamp":"2024-02-01T10:16:23.060768290Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7FYomkbpFjXsMEVUf--3FIN","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782583.080541_60.1","section":"bucket.instance","name":"vclfcn-5:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.5","timestamp":"2024-02-01T10:16:23.080541747Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GuZ_ydrh7uZExo0k-XCf-mR","ver":1},"status":{"status":"write"}}},{"id":"1_1706782583.085084_61.1","section":"bucket.instance","name":"vclfcn-5:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.5","timestamp":"2024-02-01T10:16:23.085084437Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GuZ_ydrh7uZExo0k-XCf-mR","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782583.087346_62.1","section":"bucket","name":"vclfcn-5","timestamp":"2024-02-01T10:16:23.087346765Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KFexYQNYfYmKGSCAKJDSs_v","ver":1},"status":{"status":"write"}}},{"id":"1_1706782583.091891_63.1","section":"bucket","name":"vclfcn-5","timestamp":"2024-02-01T10:16:23.091891354Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KFexYQNYfYmKGSCAKJDSs_v","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782583.135000_64.1","section":"bucket.instance","name":"vclfcn-6:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.6","timestamp":"2024-02-01T10:16:23.135000549Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nUYxPNk4mpHlEoPbPAjZI6V","ver":1},"status":{"status":"write"}}},{"id":"1_1706782583.140997_65.1","section":"bucket.instance","name":"vclfcn-6:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.6","timestamp":"2024-02-01T10:16:23.140997195Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nUYxPNk4mpHlEoPbPAjZI6V","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782583.143037_66.1","section":"bucket","name":"vclfcn-6","timestamp":"2024-02-01T10:16:23.143037696Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4xyZKpWpwMtUf5XA3nUVwvT","ver":1},"status":{"status":"write"}}},{"id":"1_1706782583.148165_67.1","section":"bucket","name":"vclfcn-6","timestamp":"2024-02-01T10:16:23.148165375Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4xyZKpWpwMtUf5XA3nUVwvT","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782583.582106_68.1","section":"bucket.instance","name":"vclfcn-4:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.7","timestamp":"2024-02-01T10:16:23.582106079Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_K-GLBBcTrWLHEjhiQTh8AAO","ver":1},"status":{"status":"write"}}},{"id":"1_1706782583.585480_69.1","section":"bucket.instance","name":"vclfcn-4:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.7","timestamp":"2024-02-01T10:16:23.585480929Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_K-GLBBcTrWLHEjhiQTh8AAO","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782583.586646_70.1","section":"bucket","name":"vclfcn-4","timestamp":"2024-02-01T10:16:23.586646340Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7FYomkbpFjXsMEVUf--3FIN","ver":1},"status":{"status":"write"}}},{"id":"1_1706782583.588852_71.1","section":"bucket","name":"vclfcn-4","timestamp":"2024-02-01T10:16:23.588852889Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7FYomkbpFjXsMEVUf--3FIN","ver":1},"status":{"status":"abort"}}},{"id":"1_1706782583.604965_72.1","section":"bucket.instance","name":"vclfcn-5:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.8","timestamp":"2024-02-01T10:16:23.604965201Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-38jr9mPIsX5Hbh-yF_INzw","ver":1},"status":{"status":"write"}}},{"id":"1_1706782583.608205_73.1","section":"bucket.instance","name":"vclfcn-5:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.8","timestamp":"2024-02-01T10:16:23.608205537Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-38jr9mPIsX5Hbh-yF_INzw","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782583.609727_74.1","section":"bucket","name":"vclfcn-5","timestamp":"2024-02-01T10:16:23.609727931Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KFexYQNYfYmKGSCAKJDSs_v","ver":1},"status":{"status":"write"}}},{"id":"1_1706782583.612105_75.1","section":"bucket","name":"vclfcn-5","timestamp":"2024-02-01T10:16:23.612105467Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KFexYQNYfYmKGSCAKJDSs_v","ver":1},"status":{"status":"abort"}}},{"id":"1_1706782583.632804_76.1","section":"bucket.instance","name":"vclfcn-6:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.9","timestamp":"2024-02-01T10:16:23.632804818Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xPuFAl83iV88j7bpn1hxTNX","ver":1},"status":{"status":"write"}}},{"id":"1_1706782583.638590_77.1","section":"bucket.instance","name":"vclfcn-6:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.9","timestamp":"2024-02-01T10:16:23.638590664Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xPuFAl83iV88j7bpn1hxTNX","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782583.640189_78.1","section":"bucket","name":"vclfcn-6","timestamp":"2024-02-01T10:16:23.640189514Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4xyZKpWpwMtUf5XA3nUVwvT","ver":1},"status":{"status":"write"}}},{"id":"1_1706782583.642926_79.1","section":"bucket","name":"vclfcn-6","timestamp":"2024-02-01T10:16:23.642926395Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4xyZKpWpwMtUf5XA3nUVwvT","ver":1},"status":{"status":"abort"}}},{"id":"1_1706782583.671310_80.1","section":"bucket.instance","name":"vclfcn-4:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.10","timestamp":"2024-02-01T10:16:23.671310356Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1U8H0rwTUD_-21XS12rwqpQ","ver":1},"status":{"status":"write"}}},{"id":"1_1706782583.677081_81.1","section":"bucket.instance","name":"vclfcn-4:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.10","timestamp":"2024-02-01T10:16:23.677081964Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1U8H0rwTUD_-21XS12rwqpQ","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782583.678607_82.1","section":"bucket","name":"vclfcn-4","timestamp":"2024-02-01T10:16:23.678607917Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7FYomkbpFjXsMEVUf--3FIN","ver":1},"status":{"status":"write"}}},{"id":"1_1706782583.681355_83.1","section":"bucket","name":"vclfcn-4","timestamp":"2024-02-01T10:16:23.681355024Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7FYomkbpFjXsMEVUf--3FIN","ver":1},"status":{"status":"abort"}}},{"id":"1_1706782583.724585_84.1","section":"bucket.instance","name":"vclfcn-5:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.11","timestamp":"2024-02-01T10:16:23.724585309Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1uvujq5ws5ZqUnKk2XeudY4","ver":1},"status":{"status":"write"}}},{"id":"1_1706782583.728526_85.1","section":"bucket.instance","name":"vclfcn-5:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.11","timestamp":"2024-02-01T10:16:23.728526702Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1uvujq5ws5ZqUnKk2XeudY4","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782583.729969_86.1","section":"bucket","name":"vclfcn-5","timestamp":"2024-02-01T10:16:23.729969063Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KFexYQNYfYmKGSCAKJDSs_v","ver":1},"status":{"status":"write"}}},{"id":"1_1706782583.733921_87.1","section":"bucket","name":"vclfcn-5","timestamp":"2024-02-01T10:16:23.733921102Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KFexYQNYfYmKGSCAKJDSs_v","ver":1},"status":{"status":"abort"}}},{"id":"1_1706782583.821698_88.1","section":"bucket.instance","name":"vclfcn-6:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.12","timestamp":"2024-02-01T10:16:23.821698722Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_InLI6Y5Soa-cFmcsAza4-1x","ver":1},"status":{"status":"write"}}},{"id":"1_1706782583.828621_89.1","section":"bucket.instance","name":"vclfcn-6:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.12","timestamp":"2024-02-01T10:16:23.828621437Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_InLI6Y5Soa-cFmcsAza4-1x","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782583.831465_90.1","section":"bucket","name":"vclfcn-6","timestamp":"2024-02-01T10:16:23.831465748Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4xyZKpWpwMtUf5XA3nUVwvT","ver":1},"status":{"status":"write"}}},{"id":"1_1706782583.837030_91.1","section":"bucket","name":"vclfcn-6","timestamp":"2024-02-01T10:16:23.837030233Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4xyZKpWpwMtUf5XA3nUVwvT","ver":1},"status":{"status":"abort"}}},{"id":"1_1706782583.931558_92.1","section":"bucket.instance","name":"vclfcn-4:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.13","timestamp":"2024-02-01T10:16:23.931558665Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_mh6kxfzVQf2159X6vdQEtr9","ver":1},"status":{"status":"write"}}},{"id":"1_1706782583.939774_93.1","section":"bucket.instance","name":"vclfcn-4:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.13","timestamp":"2024-02-01T10:16:23.939774137Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_mh6kxfzVQf2159X6vdQEtr9","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782583.942334_94.1","section":"bucket","name":"vclfcn-4","timestamp":"2024-02-01T10:16:23.942334764Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7FYomkbpFjXsMEVUf--3FIN","ver":1},"status":{"status":"write"}}},{"id":"1_1706782583.947201_95.1","section":"bucket","name":"vclfcn-4","timestamp":"2024-02-01T10:16:23.947201866Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7FYomkbpFjXsMEVUf--3FIN","ver":1},"status":{"status":"abort"}}},{"id":"1_1706782583.997720_96.1","section":"bucket.instance","name":"vclfcn-5:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.14","timestamp":"2024-02-01T10:16:23.997720926Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_u7t7dVEXnS4njZhJkFAad6L","ver":1},"status":{"status":"write"}}},{"id":"1_1706782584.004511_97.1","section":"bucket.instance","name":"vclfcn-5:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.14","timestamp":"2024-02-01T10:16:24.004511156Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_u7t7dVEXnS4njZhJkFAad6L","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782584.007009_98.1","section":"bucket","name":"vclfcn-5","timestamp":"2024-02-01T10:16:24.007009106Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KFexYQNYfYmKGSCAKJDSs_v","ver":1},"status":{"status":"write"}}},{"id":"1_1706782584.011561_99.1","section":"bucket","name":"vclfcn-5","timestamp":"2024-02-01T10:16:24.011561820Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KFexYQNYfYmKGSCAKJDSs_v","ver":1},"status":{"status":"abort"}}},{"id":"1_1706782584.107340_100.1","section":"bucket.instance","name":"vclfcn-6:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.15","timestamp":"2024-02-01T10:16:24.107340431Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SwtrgKLrPfbA6nAuIOAjPY5","ver":1},"status":{"status":"write"}}},{"id":"1_1706782584.114866_101.1","section":"bucket.instance","name":"vclfcn-6:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.15","timestamp":"2024-02-01T10:16:24.114866482Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SwtrgKLrPfbA6nAuIOAjPY5","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782584.117713_102.1","section":"bucket","name":"vclfcn-6","timestamp":"2024-02-01T10:16:24.117713159Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4xyZKpWpwMtUf5XA3nUVwvT","ver":1},"status":{"status":"write"}}},{"id":"1_1706782584.123490_103.1","section":"bucket","name":"vclfcn-6","timestamp":"2024-02-01T10:16:24.123490508Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4xyZKpWpwMtUf5XA3nUVwvT","ver":1},"status":{"status":"abort"}}},{"id":"1_1706782589.804979_104.1","section":"bucket.instance","name":"vclfcn-7:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.16","timestamp":"2024-02-01T10:16:29.804979118Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_z1mGy633o2dYXHeltm-sLIr","ver":1},"status":{"status":"write"}}},{"id":"1_1706782589.808233_105.1","section":"bucket.instance","name":"vclfcn-7:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.16","timestamp":"2024-02-01T10:16:29.808233327Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_z1mGy633o2dYXHeltm-sLIr","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782589.809506_106.1","section":"bucket","name":"vclfcn-7","timestamp":"2024-02-01T10:16:29.809506716Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_aQqC4o_dOalWjSRbtEyqQ9y","ver":1},"status":{"status":"write"}}},{"id":"1_1706782589.812385_107.1","section":"bucket","name":"vclfcn-7","timestamp":"2024-02-01T10:16:29.812385583Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_aQqC4o_dOalWjSRbtEyqQ9y","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782589.827390_108.1","section":"bucket.instance","name":"vclfcn-8:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.17","timestamp":"2024-02-01T10:16:29.827390273Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fg266bvdezCNOcfE9SKi_vH","ver":1},"status":{"status":"write"}}},{"id":"1_1706782589.832050_109.1","section":"bucket.instance","name":"vclfcn-8:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.17","timestamp":"2024-02-01T10:16:29.832050187Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fg266bvdezCNOcfE9SKi_vH","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782589.833185_110.1","section":"bucket","name":"vclfcn-8","timestamp":"2024-02-01T10:16:29.833185878Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VxqI9_Wg5TOlSHC_n8MLWjO","ver":1},"status":{"status":"write"}}},{"id":"1_1706782589.836457_111.1","section":"bucket","name":"vclfcn-8","timestamp":"2024-02-01T10:16:29.836457633Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VxqI9_Wg5TOlSHC_n8MLWjO","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782589.877746_112.1","section":"bucket.instance","name":"vclfcn-9:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.18","timestamp":"2024-02-01T10:16:29.877746869Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jIv31DiHuYuZmjj2jtUcl5K","ver":1},"status":{"status":"write"}}},{"id":"1_1706782589.884299_113.1","section":"bucket.instance","name":"vclfcn-9:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.18","timestamp":"2024-02-01T10:16:29.884299625Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jIv31DiHuYuZmjj2jtUcl5K","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782589.886780_114.1","section":"bucket","name":"vclfcn-9","timestamp":"2024-02-01T10:16:29.886780733Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fop2OKI1GWzp_3SQxaIH8uR","ver":1},"status":{"status":"write"}}},{"id":"1_1706782589.893748_115.1","section":"bucket","name":"vclfcn-9","timestamp":"2024-02-01T10:16:29.893748078Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fop2OKI1GWzp_3SQxaIH8uR","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782595.478858_116.1","section":"bucket","name":"vclfcn-7","timestamp":"2024-02-01T10:16:35.478858938Z","data":{"read_version":{"tag":"_aQqC4o_dOalWjSRbtEyqQ9y","ver":1},"write_version":{"tag":"_aQqC4o_dOalWjSRbtEyqQ9y","ver":2},"status":{"status":"remove"}}},{"id":"1_1706782595.482342_117.1","section":"bucket","name":"vclfcn-7","timestamp":"2024-02-01T10:16:35.482342431Z","data":{"read_version":{"tag":"_aQqC4o_dOalWjSRbtEyqQ9y","ver":1},"write_version":{"tag":"_aQqC4o_dOalWjSRbtEyqQ9y","ver":2},"status":{"status":"complete"}}},{"id":"1_1706782595.541467_118.1","section":"bucket","name":"vclfcn-8","timestamp":"2024-02-01T10:16:35.541467766Z","data":{"read_version":{"tag":"_VxqI9_Wg5TOlSHC_n8MLWjO","ver":1},"write_version":{"tag":"_VxqI9_Wg5TOlSHC_n8MLWjO","ver":2},"status":{"status":"remove"}}},{"id":"1_1706782595.545966_119.1","section":"bucket","name":"vclfcn-8","timestamp":"2024-02-01T10:16:35.545966065Z","data":{"read_version":{"tag":"_VxqI9_Wg5TOlSHC_n8MLWjO","ver":1},"write_version":{"tag":"_VxqI9_Wg5TOlSHC_n8MLWjO","ver":2},"status":{"status":"complete"}}},{"id":"1_1706782595.664631_120.1","section":"bucket","name":"vclfcn-9","timestamp":"2024-02-01T10:16:35.664631193Z","data":{"read_version":{"tag":"_fop2OKI1GWzp_3SQxaIH8uR","ver":1},"write_version":{"tag":"_fop2OKI1GWzp_3SQxaIH8uR","ver":2},"status":{"status":"remove"}}},{"id":"1_1706782595.670952_121.1","section":"bucket","name":"vclfcn-9","timestamp":"2024-02-01T10:16:35.670952808Z","data":{"read_version":{"tag":"_fop2OKI1GWzp_3SQxaIH8uR","ver":1},"write_version":{"tag":"_fop2OKI1GWzp_3SQxaIH8uR","ver":2},"status":{"status":"complete"}}}] 2024-02-01T10:38:05.054 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '5dc4efbd-eccd-47a8-9471-ae4a209ec978', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:05.054 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '5dc4efbd-eccd-47a8-9471-ae4a209ec978', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:05.055 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 5dc4efbd-eccd-47a8-9471-ae4a209ec978 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:38:05.095 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:38:05.095 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:38:05.180 INFO:teuthology.orchestra.run.smithi148.stdout:[{"id":"1_1706782453.129936_29.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2024-02-01T10:14:13.129936524Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RY7up0tQcOf9ElajXOF5d6i","ver":1},"status":{"status":"write"}}},{"id":"1_1706782453.132619_30.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2024-02-01T10:14:13.132619785Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RY7up0tQcOf9ElajXOF5d6i","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782596.137698_33.1","section":"bucket.instance","name":"vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19","timestamp":"2024-02-01T10:16:36.137698180Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gSnH8BiIXGWkG2sGorT95Ca","ver":1},"status":{"status":"write"}}},{"id":"1_1706782596.141485_34.1","section":"bucket.instance","name":"vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19","timestamp":"2024-02-01T10:16:36.141485128Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gSnH8BiIXGWkG2sGorT95Ca","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782596.142883_35.1","section":"bucket","name":"vclfcn-10","timestamp":"2024-02-01T10:16:36.142883452Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_C-y9oz7otdaRTy0zc_QeA8H","ver":1},"status":{"status":"write"}}},{"id":"1_1706782596.147037_36.1","section":"bucket","name":"vclfcn-10","timestamp":"2024-02-01T10:16:36.147037349Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_C-y9oz7otdaRTy0zc_QeA8H","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782596.165713_37.1","section":"bucket.instance","name":"vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20","timestamp":"2024-02-01T10:16:36.165713463Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ugvw9Z3SeaHLB2NBY8cOvIs","ver":1},"status":{"status":"write"}}},{"id":"1_1706782596.170196_38.1","section":"bucket.instance","name":"vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20","timestamp":"2024-02-01T10:16:36.170196200Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ugvw9Z3SeaHLB2NBY8cOvIs","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782596.171951_39.1","section":"bucket","name":"vclfcn-11","timestamp":"2024-02-01T10:16:36.171951352Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_clR9bGsKa2Bt45u61U1vnaU","ver":1},"status":{"status":"write"}}},{"id":"1_1706782596.176142_40.1","section":"bucket","name":"vclfcn-11","timestamp":"2024-02-01T10:16:36.176142514Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_clR9bGsKa2Bt45u61U1vnaU","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782596.215001_41.1","section":"bucket.instance","name":"vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21","timestamp":"2024-02-01T10:16:36.215001761Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0aJjbbhzkBKwKh67jUyjFXU","ver":1},"status":{"status":"write"}}},{"id":"1_1706782596.221792_42.1","section":"bucket.instance","name":"vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21","timestamp":"2024-02-01T10:16:36.221792015Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0aJjbbhzkBKwKh67jUyjFXU","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782596.224377_43.1","section":"bucket","name":"vclfcn-12","timestamp":"2024-02-01T10:16:36.224377534Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YzQLQ2ZvUn8a9M8WMLPPlj-","ver":1},"status":{"status":"write"}}},{"id":"1_1706782596.230502_44.1","section":"bucket","name":"vclfcn-12","timestamp":"2024-02-01T10:16:36.230502418Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YzQLQ2ZvUn8a9M8WMLPPlj-","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782693.582816_53.1","section":"bucket.instance","name":"vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22","timestamp":"2024-02-01T10:18:13.582816318Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_IyMDVQZq1Dm4Ik60abXwswh","ver":1},"status":{"status":"write"}}},{"id":"1_1706782693.589569_54.1","section":"bucket.instance","name":"vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22","timestamp":"2024-02-01T10:18:13.589569915Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_IyMDVQZq1Dm4Ik60abXwswh","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782693.592623_55.1","section":"bucket","name":"vclfcn-13","timestamp":"2024-02-01T10:18:13.592623458Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SBtUJfjUPgWxvP4marStPj8","ver":1},"status":{"status":"write"}}},{"id":"1_1706782693.599222_56.1","section":"bucket","name":"vclfcn-13","timestamp":"2024-02-01T10:18:13.599222251Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SBtUJfjUPgWxvP4marStPj8","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782693.622788_57.1","section":"bucket.instance","name":"vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23","timestamp":"2024-02-01T10:18:13.622788043Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OLXZUmEaWvH6RgSaGLsC5gp","ver":1},"status":{"status":"write"}}},{"id":"1_1706782693.629469_58.1","section":"bucket.instance","name":"vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23","timestamp":"2024-02-01T10:18:13.629469201Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OLXZUmEaWvH6RgSaGLsC5gp","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782693.632089_59.1","section":"bucket","name":"vclfcn-14","timestamp":"2024-02-01T10:18:13.632089534Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6H3UQNuqVeMSp5sl1iVxf3G","ver":1},"status":{"status":"write"}}},{"id":"1_1706782693.638718_60.1","section":"bucket","name":"vclfcn-14","timestamp":"2024-02-01T10:18:13.638718897Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6H3UQNuqVeMSp5sl1iVxf3G","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782693.686988_61.1","section":"bucket.instance","name":"vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24","timestamp":"2024-02-01T10:18:13.686988320Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GhbTNO6qlpBiEMg1HJVBNAz","ver":1},"status":{"status":"write"}}},{"id":"1_1706782693.694347_62.1","section":"bucket.instance","name":"vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24","timestamp":"2024-02-01T10:18:13.694347551Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GhbTNO6qlpBiEMg1HJVBNAz","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782693.696886_63.1","section":"bucket","name":"vclfcn-15","timestamp":"2024-02-01T10:18:13.696886663Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PC87lFbJZuz6KYHILchNgZJ","ver":1},"status":{"status":"write"}}},{"id":"1_1706782693.703781_64.1","section":"bucket","name":"vclfcn-15","timestamp":"2024-02-01T10:18:13.703781181Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PC87lFbJZuz6KYHILchNgZJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782788.125323_68.1","section":"bucket.instance","name":"vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25","timestamp":"2024-02-01T10:19:48.125323856Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GJQLwPJ67-aampE80YQ6oYN","ver":1},"status":{"status":"write"}}},{"id":"1_1706782788.132848_69.1","section":"bucket.instance","name":"vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25","timestamp":"2024-02-01T10:19:48.132848685Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GJQLwPJ67-aampE80YQ6oYN","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782788.136003_70.1","section":"bucket","name":"vclfcn-16","timestamp":"2024-02-01T10:19:48.136003157Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yZQS2Ooyuo-DWcyRYk0NZG5","ver":1},"status":{"status":"write"}}},{"id":"1_1706782788.142977_71.1","section":"bucket","name":"vclfcn-16","timestamp":"2024-02-01T10:19:48.142977363Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yZQS2Ooyuo-DWcyRYk0NZG5","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782788.168776_72.1","section":"bucket.instance","name":"vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26","timestamp":"2024-02-01T10:19:48.168776620Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ds-zBeTuxJy65JHqqd4lb5Z","ver":1},"status":{"status":"write"}}},{"id":"1_1706782788.174951_73.1","section":"bucket.instance","name":"vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26","timestamp":"2024-02-01T10:19:48.174951550Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ds-zBeTuxJy65JHqqd4lb5Z","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782788.178001_74.1","section":"bucket","name":"vclfcn-17","timestamp":"2024-02-01T10:19:48.178001249Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZIbkkBCCgIhVLT8B2r1w5K_","ver":1},"status":{"status":"write"}}},{"id":"1_1706782788.184593_75.1","section":"bucket","name":"vclfcn-17","timestamp":"2024-02-01T10:19:48.184593776Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZIbkkBCCgIhVLT8B2r1w5K_","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782788.225397_76.1","section":"bucket.instance","name":"vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27","timestamp":"2024-02-01T10:19:48.225397011Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gkEVh6TTRHTmUu-UImISUWd","ver":1},"status":{"status":"write"}}},{"id":"1_1706782788.230219_77.1","section":"bucket.instance","name":"vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27","timestamp":"2024-02-01T10:19:48.230219600Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gkEVh6TTRHTmUu-UImISUWd","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782788.232911_78.1","section":"bucket","name":"vclfcn-18","timestamp":"2024-02-01T10:19:48.232911150Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HlQ9gchNi7E4cFgOusBiYBD","ver":1},"status":{"status":"write"}}},{"id":"1_1706782788.241244_79.1","section":"bucket","name":"vclfcn-18","timestamp":"2024-02-01T10:19:48.241244766Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HlQ9gchNi7E4cFgOusBiYBD","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782788.264154_80.1","section":"bucket.instance","name":"vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25","timestamp":"2024-02-01T10:19:48.264154269Z","data":{"read_version":{"tag":"_GJQLwPJ67-aampE80YQ6oYN","ver":1},"write_version":{"tag":"_GJQLwPJ67-aampE80YQ6oYN","ver":2},"status":{"status":"write"}}},{"id":"1_1706782788.268634_81.1","section":"bucket.instance","name":"vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25","timestamp":"2024-02-01T10:19:48.268634480Z","data":{"read_version":{"tag":"_GJQLwPJ67-aampE80YQ6oYN","ver":1},"write_version":{"tag":"_GJQLwPJ67-aampE80YQ6oYN","ver":2},"status":{"status":"complete"}}},{"id":"1_1706782788.275810_82.1","section":"bucket.instance","name":"vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26","timestamp":"2024-02-01T10:19:48.275810646Z","data":{"read_version":{"tag":"_ds-zBeTuxJy65JHqqd4lb5Z","ver":1},"write_version":{"tag":"_ds-zBeTuxJy65JHqqd4lb5Z","ver":2},"status":{"status":"write"}}},{"id":"1_1706782788.281245_83.1","section":"bucket.instance","name":"vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26","timestamp":"2024-02-01T10:19:48.281245680Z","data":{"read_version":{"tag":"_ds-zBeTuxJy65JHqqd4lb5Z","ver":1},"write_version":{"tag":"_ds-zBeTuxJy65JHqqd4lb5Z","ver":2},"status":{"status":"complete"}}},{"id":"1_1706782788.295656_84.1","section":"bucket.instance","name":"vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27","timestamp":"2024-02-01T10:19:48.295656474Z","data":{"read_version":{"tag":"_gkEVh6TTRHTmUu-UImISUWd","ver":1},"write_version":{"tag":"_gkEVh6TTRHTmUu-UImISUWd","ver":2},"status":{"status":"write"}}},{"id":"1_1706782788.299861_85.1","section":"bucket.instance","name":"vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27","timestamp":"2024-02-01T10:19:48.299861431Z","data":{"read_version":{"tag":"_gkEVh6TTRHTmUu-UImISUWd","ver":1},"write_version":{"tag":"_gkEVh6TTRHTmUu-UImISUWd","ver":2},"status":{"status":"complete"}}},{"id":"1_1706782985.577592_93.1","section":"bucket.instance","name":"vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28","timestamp":"2024-02-01T10:23:05.577592857Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Tkzhdhq2YxvoQoXNUGw-VlA","ver":1},"status":{"status":"write"}}},{"id":"1_1706782985.584276_94.1","section":"bucket.instance","name":"vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28","timestamp":"2024-02-01T10:23:05.584276295Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Tkzhdhq2YxvoQoXNUGw-VlA","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782985.586795_95.1","section":"bucket","name":"vclfcn-19","timestamp":"2024-02-01T10:23:05.586795365Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uFY0fAasS29yh69Gp_uVUE2","ver":1},"status":{"status":"write"}}},{"id":"1_1706782985.593052_96.1","section":"bucket","name":"vclfcn-19","timestamp":"2024-02-01T10:23:05.593052024Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uFY0fAasS29yh69Gp_uVUE2","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782985.602177_97.1","section":"bucket.instance","name":"vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28","timestamp":"2024-02-01T10:23:05.602177901Z","data":{"read_version":{"tag":"_Tkzhdhq2YxvoQoXNUGw-VlA","ver":1},"write_version":{"tag":"_Tkzhdhq2YxvoQoXNUGw-VlA","ver":2},"status":{"status":"write"}}},{"id":"1_1706782985.609410_98.1","section":"bucket.instance","name":"vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28","timestamp":"2024-02-01T10:23:05.609410401Z","data":{"read_version":{"tag":"_Tkzhdhq2YxvoQoXNUGw-VlA","ver":1},"write_version":{"tag":"_Tkzhdhq2YxvoQoXNUGw-VlA","ver":2},"status":{"status":"complete"}}},{"id":"1_1706783281.140663_112.1","section":"bucket.instance","name":"vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29","timestamp":"2024-02-01T10:28:01.140663757Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RY5J0cHq0mZCxKxUWbHq1nu","ver":1},"status":{"status":"write"}}},{"id":"1_1706783281.146879_113.1","section":"bucket.instance","name":"vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29","timestamp":"2024-02-01T10:28:01.146879624Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RY5J0cHq0mZCxKxUWbHq1nu","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783281.149599_114.1","section":"bucket","name":"vclfcn-20","timestamp":"2024-02-01T10:28:01.149599676Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LBpQRODD-fP4UvS_L3UYNrJ","ver":1},"status":{"status":"write"}}},{"id":"1_1706783281.156355_115.1","section":"bucket","name":"vclfcn-20","timestamp":"2024-02-01T10:28:01.156355553Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LBpQRODD-fP4UvS_L3UYNrJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783313.047690_117.1","section":"bucket.instance","name":"vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29","timestamp":"2024-02-01T10:28:33.047690221Z","data":{"read_version":{"tag":"_RY5J0cHq0mZCxKxUWbHq1nu","ver":1},"write_version":{"tag":"_RY5J0cHq0mZCxKxUWbHq1nu","ver":2},"status":{"status":"write"}}},{"id":"1_1706783313.054991_118.1","section":"bucket.instance","name":"vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29","timestamp":"2024-02-01T10:28:33.054991128Z","data":{"read_version":{"tag":"_RY5J0cHq0mZCxKxUWbHq1nu","ver":1},"write_version":{"tag":"_RY5J0cHq0mZCxKxUWbHq1nu","ver":2},"status":{"status":"complete"}}},{"id":"1_1706783510.479238_122.1","section":"bucket.instance","name":"vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29","timestamp":"2024-02-01T10:31:50.479238314Z","data":{"read_version":{"tag":"_RY5J0cHq0mZCxKxUWbHq1nu","ver":2},"write_version":{"tag":"_RY5J0cHq0mZCxKxUWbHq1nu","ver":3},"status":{"status":"write"}}},{"id":"1_1706783510.486411_123.1","section":"bucket.instance","name":"vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29","timestamp":"2024-02-01T10:31:50.486411437Z","data":{"read_version":{"tag":"_RY5J0cHq0mZCxKxUWbHq1nu","ver":2},"write_version":{"tag":"_RY5J0cHq0mZCxKxUWbHq1nu","ver":3},"status":{"status":"complete"}}},{"id":"1_1706783557.936493_125.1","section":"bucket.instance","name":"vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30","timestamp":"2024-02-01T10:32:37.936493302Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SBaAf2hqSgxj2ifisrCDUQ-","ver":1},"status":{"status":"write"}}},{"id":"1_1706783557.943553_126.1","section":"bucket.instance","name":"vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30","timestamp":"2024-02-01T10:32:37.943553989Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SBaAf2hqSgxj2ifisrCDUQ-","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783557.946282_127.1","section":"bucket","name":"vclfcn-21","timestamp":"2024-02-01T10:32:37.946282859Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_aS4q16MZBGYstqXzbnAETLq","ver":1},"status":{"status":"write"}}},{"id":"1_1706783557.952481_128.1","section":"bucket","name":"vclfcn-21","timestamp":"2024-02-01T10:32:37.952481787Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_aS4q16MZBGYstqXzbnAETLq","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783557.977604_129.1","section":"bucket.instance","name":"vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31","timestamp":"2024-02-01T10:32:37.977604938Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AivkNk9zM8C_ACc4CMMCA9j","ver":1},"status":{"status":"write"}}},{"id":"1_1706783557.984354_130.1","section":"bucket.instance","name":"vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31","timestamp":"2024-02-01T10:32:37.984354568Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AivkNk9zM8C_ACc4CMMCA9j","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783557.986976_131.1","section":"bucket","name":"vclfcn-22","timestamp":"2024-02-01T10:32:37.986976409Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OC84jTcY1mo5V4XtRR3FK81","ver":1},"status":{"status":"write"}}},{"id":"1_1706783557.993660_132.1","section":"bucket","name":"vclfcn-22","timestamp":"2024-02-01T10:32:37.993660224Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OC84jTcY1mo5V4XtRR3FK81","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783558.040696_133.1","section":"bucket.instance","name":"vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32","timestamp":"2024-02-01T10:32:38.040696413Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_y0EPEo2ASjenysOQ86zdnP6","ver":1},"status":{"status":"write"}}},{"id":"1_1706783558.047555_134.1","section":"bucket.instance","name":"vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32","timestamp":"2024-02-01T10:32:38.047555506Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_y0EPEo2ASjenysOQ86zdnP6","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783558.049874_135.1","section":"bucket","name":"vclfcn-23","timestamp":"2024-02-01T10:32:38.049874456Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pctO5Jr19wCuuci8zhYun2q","ver":1},"status":{"status":"write"}}},{"id":"1_1706783558.055502_136.1","section":"bucket","name":"vclfcn-23","timestamp":"2024-02-01T10:32:38.055502136Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pctO5Jr19wCuuci8zhYun2q","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783558.081356_137.1","section":"bucket.instance","name":"vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30","timestamp":"2024-02-01T10:32:38.081356052Z","data":{"read_version":{"tag":"_SBaAf2hqSgxj2ifisrCDUQ-","ver":1},"write_version":{"tag":"_SBaAf2hqSgxj2ifisrCDUQ-","ver":2},"status":{"status":"write"}}},{"id":"1_1706783558.088380_138.1","section":"bucket.instance","name":"vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30","timestamp":"2024-02-01T10:32:38.088380225Z","data":{"read_version":{"tag":"_SBaAf2hqSgxj2ifisrCDUQ-","ver":1},"write_version":{"tag":"_SBaAf2hqSgxj2ifisrCDUQ-","ver":2},"status":{"status":"complete"}}},{"id":"1_1706783558.095619_139.1","section":"bucket.instance","name":"vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31","timestamp":"2024-02-01T10:32:38.095619390Z","data":{"read_version":{"tag":"_AivkNk9zM8C_ACc4CMMCA9j","ver":1},"write_version":{"tag":"_AivkNk9zM8C_ACc4CMMCA9j","ver":2},"status":{"status":"write"}}},{"id":"1_1706783558.100472_140.1","section":"bucket.instance","name":"vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31","timestamp":"2024-02-01T10:32:38.100472386Z","data":{"read_version":{"tag":"_AivkNk9zM8C_ACc4CMMCA9j","ver":1},"write_version":{"tag":"_AivkNk9zM8C_ACc4CMMCA9j","ver":2},"status":{"status":"complete"}}},{"id":"1_1706783558.110239_141.1","section":"bucket.instance","name":"vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32","timestamp":"2024-02-01T10:32:38.110239478Z","data":{"read_version":{"tag":"_y0EPEo2ASjenysOQ86zdnP6","ver":1},"write_version":{"tag":"_y0EPEo2ASjenysOQ86zdnP6","ver":2},"status":{"status":"write"}}},{"id":"1_1706783558.115340_142.1","section":"bucket.instance","name":"vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32","timestamp":"2024-02-01T10:32:38.115340472Z","data":{"read_version":{"tag":"_y0EPEo2ASjenysOQ86zdnP6","ver":1},"write_version":{"tag":"_y0EPEo2ASjenysOQ86zdnP6","ver":2},"status":{"status":"complete"}}},{"id":"1_1706783613.485844_144.1","section":"bucket.instance","name":"vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33","timestamp":"2024-02-01T10:33:33.485844007Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KQaPa_IyHr1JRoHAufGblmN","ver":1},"status":{"status":"write"}}},{"id":"1_1706783613.493076_145.1","section":"bucket.instance","name":"vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33","timestamp":"2024-02-01T10:33:33.493076639Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KQaPa_IyHr1JRoHAufGblmN","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783613.495609_146.1","section":"bucket","name":"vclfcn-24","timestamp":"2024-02-01T10:33:33.495609093Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wdRz7umYZCWDz61bnbDJ5DR","ver":1},"status":{"status":"write"}}},{"id":"1_1706783613.501842_147.1","section":"bucket","name":"vclfcn-24","timestamp":"2024-02-01T10:33:33.501842098Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wdRz7umYZCWDz61bnbDJ5DR","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783613.526071_148.1","section":"bucket.instance","name":"vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34","timestamp":"2024-02-01T10:33:33.526071464Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TJp0m8FTngleqfn93NSNEFQ","ver":1},"status":{"status":"write"}}},{"id":"1_1706783613.533006_149.1","section":"bucket.instance","name":"vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34","timestamp":"2024-02-01T10:33:33.533006576Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TJp0m8FTngleqfn93NSNEFQ","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783613.535429_150.1","section":"bucket","name":"vclfcn-25","timestamp":"2024-02-01T10:33:33.535429751Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XUEOg4Lx9rBHbAJwKq1lwrJ","ver":1},"status":{"status":"write"}}},{"id":"1_1706783613.541933_151.1","section":"bucket","name":"vclfcn-25","timestamp":"2024-02-01T10:33:33.541933857Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XUEOg4Lx9rBHbAJwKq1lwrJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783613.587133_152.1","section":"bucket.instance","name":"vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35","timestamp":"2024-02-01T10:33:33.587133123Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_djT8hpdivEzfKbjzZM1tGYc","ver":1},"status":{"status":"write"}}},{"id":"1_1706783613.594437_153.1","section":"bucket.instance","name":"vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35","timestamp":"2024-02-01T10:33:33.594437539Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_djT8hpdivEzfKbjzZM1tGYc","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783613.597205_154.1","section":"bucket","name":"vclfcn-26","timestamp":"2024-02-01T10:33:33.597205142Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3fH7UbMU-OxdukYrx1ZeZDR","ver":1},"status":{"status":"write"}}},{"id":"1_1706783613.604282_155.1","section":"bucket","name":"vclfcn-26","timestamp":"2024-02-01T10:33:33.604282859Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3fH7UbMU-OxdukYrx1ZeZDR","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783613.632812_156.1","section":"bucket.instance","name":"vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33","timestamp":"2024-02-01T10:33:33.632812026Z","data":{"read_version":{"tag":"_KQaPa_IyHr1JRoHAufGblmN","ver":1},"write_version":{"tag":"_KQaPa_IyHr1JRoHAufGblmN","ver":2},"status":{"status":"write"}}},{"id":"1_1706783613.639457_157.1","section":"bucket.instance","name":"vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33","timestamp":"2024-02-01T10:33:33.639457857Z","data":{"read_version":{"tag":"_KQaPa_IyHr1JRoHAufGblmN","ver":1},"write_version":{"tag":"_KQaPa_IyHr1JRoHAufGblmN","ver":2},"status":{"status":"complete"}}},{"id":"1_1706783613.645309_158.1","section":"bucket.instance","name":"vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33","timestamp":"2024-02-01T10:33:33.645309049Z","data":{"read_version":{"tag":"_KQaPa_IyHr1JRoHAufGblmN","ver":2},"write_version":{"tag":"_KQaPa_IyHr1JRoHAufGblmN","ver":3},"status":{"status":"write"}}},{"id":"1_1706783613.652911_159.1","section":"bucket.instance","name":"vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33","timestamp":"2024-02-01T10:33:33.652911988Z","data":{"read_version":{"tag":"_KQaPa_IyHr1JRoHAufGblmN","ver":2},"write_version":{"tag":"_KQaPa_IyHr1JRoHAufGblmN","ver":3},"status":{"status":"complete"}}},{"id":"1_1706783613.662523_160.1","section":"bucket.instance","name":"vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34","timestamp":"2024-02-01T10:33:33.662523190Z","data":{"read_version":{"tag":"_TJp0m8FTngleqfn93NSNEFQ","ver":1},"write_version":{"tag":"_TJp0m8FTngleqfn93NSNEFQ","ver":2},"status":{"status":"write"}}},{"id":"1_1706783613.674476_161.1","section":"bucket.instance","name":"vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34","timestamp":"2024-02-01T10:33:33.674476638Z","data":{"read_version":{"tag":"_TJp0m8FTngleqfn93NSNEFQ","ver":1},"write_version":{"tag":"_TJp0m8FTngleqfn93NSNEFQ","ver":2},"status":{"status":"complete"}}},{"id":"1_1706783613.698284_162.1","section":"bucket.instance","name":"vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34","timestamp":"2024-02-01T10:33:33.698284010Z","data":{"read_version":{"tag":"_TJp0m8FTngleqfn93NSNEFQ","ver":2},"write_version":{"tag":"_TJp0m8FTngleqfn93NSNEFQ","ver":3},"status":{"status":"write"}}},{"id":"1_1706783613.707513_163.1","section":"bucket.instance","name":"vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34","timestamp":"2024-02-01T10:33:33.707513991Z","data":{"read_version":{"tag":"_TJp0m8FTngleqfn93NSNEFQ","ver":2},"write_version":{"tag":"_TJp0m8FTngleqfn93NSNEFQ","ver":3},"status":{"status":"complete"}}},{"id":"1_1706783613.728949_164.1","section":"bucket.instance","name":"vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35","timestamp":"2024-02-01T10:33:33.728949508Z","data":{"read_version":{"tag":"_djT8hpdivEzfKbjzZM1tGYc","ver":1},"write_version":{"tag":"_djT8hpdivEzfKbjzZM1tGYc","ver":2},"status":{"status":"write"}}},{"id":"1_1706783613.736733_165.1","section":"bucket.instance","name":"vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35","timestamp":"2024-02-01T10:33:33.736733042Z","data":{"read_version":{"tag":"_djT8hpdivEzfKbjzZM1tGYc","ver":1},"write_version":{"tag":"_djT8hpdivEzfKbjzZM1tGYc","ver":2},"status":{"status":"complete"}}},{"id":"1_1706783613.748133_166.1","section":"bucket.instance","name":"vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35","timestamp":"2024-02-01T10:33:33.748133962Z","data":{"read_version":{"tag":"_djT8hpdivEzfKbjzZM1tGYc","ver":2},"write_version":{"tag":"_djT8hpdivEzfKbjzZM1tGYc","ver":3},"status":{"status":"write"}}},{"id":"1_1706783613.755304_167.1","section":"bucket.instance","name":"vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35","timestamp":"2024-02-01T10:33:33.755304841Z","data":{"read_version":{"tag":"_djT8hpdivEzfKbjzZM1tGYc","ver":2},"write_version":{"tag":"_djT8hpdivEzfKbjzZM1tGYc","ver":3},"status":{"status":"complete"}}},{"id":"1_1706783732.990670_178.1","section":"bucket.instance","name":"vclfcn-27:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.36","timestamp":"2024-02-01T10:35:32.990670009Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rYzBOFkXYI9xqwLbwgqkEVW","ver":1},"status":{"status":"write"}}},{"id":"1_1706783732.996955_179.1","section":"bucket.instance","name":"vclfcn-27:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.36","timestamp":"2024-02-01T10:35:32.996955046Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rYzBOFkXYI9xqwLbwgqkEVW","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783732.999583_180.1","section":"bucket","name":"vclfcn-27","timestamp":"2024-02-01T10:35:32.999583128Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9cHBpx7BkJp-jUw4o1CRrfh","ver":1},"status":{"status":"write"}}},{"id":"1_1706783733.006036_181.1","section":"bucket","name":"vclfcn-27","timestamp":"2024-02-01T10:35:33.006036701Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9cHBpx7BkJp-jUw4o1CRrfh","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783733.034354_182.1","section":"bucket.instance","name":"vclfcn-28:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.37","timestamp":"2024-02-01T10:35:33.034354608Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ppjqf2iiQQLeeW9nmcMFwt7","ver":1},"status":{"status":"write"}}},{"id":"1_1706783733.041168_183.1","section":"bucket.instance","name":"vclfcn-28:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.37","timestamp":"2024-02-01T10:35:33.041168668Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ppjqf2iiQQLeeW9nmcMFwt7","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783733.043826_184.1","section":"bucket","name":"vclfcn-28","timestamp":"2024-02-01T10:35:33.043826329Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_O0D5bB8GV3hYxs2j9GMFn0q","ver":1},"status":{"status":"write"}}},{"id":"1_1706783733.050085_185.1","section":"bucket","name":"vclfcn-28","timestamp":"2024-02-01T10:35:33.050085404Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_O0D5bB8GV3hYxs2j9GMFn0q","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783733.095185_186.1","section":"bucket.instance","name":"vclfcn-29:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.38","timestamp":"2024-02-01T10:35:33.095185012Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_b7qqNp_v7vEEA2IVwp_dZhU","ver":1},"status":{"status":"write"}}},{"id":"1_1706783733.100262_187.1","section":"bucket.instance","name":"vclfcn-29:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.38","timestamp":"2024-02-01T10:35:33.100262420Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_b7qqNp_v7vEEA2IVwp_dZhU","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783733.102142_188.1","section":"bucket","name":"vclfcn-29","timestamp":"2024-02-01T10:35:33.102142458Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_oH5-kep0dGu6Rb8cPoOFEHk","ver":1},"status":{"status":"write"}}},{"id":"1_1706783733.107054_189.1","section":"bucket","name":"vclfcn-29","timestamp":"2024-02-01T10:35:33.107054109Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_oH5-kep0dGu6Rb8cPoOFEHk","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783733.136865_190.1","section":"bucket.instance","name":"vclfcn-27:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.36","timestamp":"2024-02-01T10:35:33.136865088Z","data":{"read_version":{"tag":"_rYzBOFkXYI9xqwLbwgqkEVW","ver":1},"write_version":{"tag":"_rYzBOFkXYI9xqwLbwgqkEVW","ver":2},"status":{"status":"write"}}},{"id":"1_1706783733.148545_191.1","section":"bucket.instance","name":"vclfcn-27:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.36","timestamp":"2024-02-01T10:35:33.148545378Z","data":{"read_version":{"tag":"_rYzBOFkXYI9xqwLbwgqkEVW","ver":1},"write_version":{"tag":"_rYzBOFkXYI9xqwLbwgqkEVW","ver":2},"status":{"status":"complete"}}},{"id":"1_1706783733.202069_192.1","section":"bucket.instance","name":"vclfcn-28:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.37","timestamp":"2024-02-01T10:35:33.202069645Z","data":{"read_version":{"tag":"_Ppjqf2iiQQLeeW9nmcMFwt7","ver":1},"write_version":{"tag":"_Ppjqf2iiQQLeeW9nmcMFwt7","ver":2},"status":{"status":"write"}}},{"id":"1_1706783733.207436_193.1","section":"bucket.instance","name":"vclfcn-28:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.37","timestamp":"2024-02-01T10:35:33.207436436Z","data":{"read_version":{"tag":"_Ppjqf2iiQQLeeW9nmcMFwt7","ver":1},"write_version":{"tag":"_Ppjqf2iiQQLeeW9nmcMFwt7","ver":2},"status":{"status":"complete"}}},{"id":"1_1706783733.222466_194.1","section":"bucket.instance","name":"vclfcn-29:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.38","timestamp":"2024-02-01T10:35:33.222466987Z","data":{"read_version":{"tag":"_b7qqNp_v7vEEA2IVwp_dZhU","ver":1},"write_version":{"tag":"_b7qqNp_v7vEEA2IVwp_dZhU","ver":2},"status":{"status":"write"}}},{"id":"1_1706783733.227246_195.1","section":"bucket.instance","name":"vclfcn-29:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.38","timestamp":"2024-02-01T10:35:33.227246173Z","data":{"read_version":{"tag":"_b7qqNp_v7vEEA2IVwp_dZhU","ver":1},"write_version":{"tag":"_b7qqNp_v7vEEA2IVwp_dZhU","ver":2},"status":{"status":"complete"}}},{"id":"1_1706783733.293745_196.1","section":"bucket.instance","name":"vclfcn-30:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.39","timestamp":"2024-02-01T10:35:33.293745968Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HhnF-4-cQEYFZsmhVgAo_3W","ver":1},"status":{"status":"write"}}},{"id":"1_1706783733.299312_197.1","section":"bucket.instance","name":"vclfcn-30:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.39","timestamp":"2024-02-01T10:35:33.299312767Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HhnF-4-cQEYFZsmhVgAo_3W","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783733.301968_198.1","section":"bucket","name":"vclfcn-30","timestamp":"2024-02-01T10:35:33.301968348Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Yct8ulBblkK4SDKF5OWcOcv","ver":1},"status":{"status":"write"}}},{"id":"1_1706783733.307848_199.1","section":"bucket","name":"vclfcn-30","timestamp":"2024-02-01T10:35:33.307848983Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Yct8ulBblkK4SDKF5OWcOcv","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783733.330768_200.1","section":"bucket.instance","name":"vclfcn-31:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.40","timestamp":"2024-02-01T10:35:33.330768162Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OsUE1ilsRE6kOECGtCDtxHd","ver":1},"status":{"status":"write"}}},{"id":"1_1706783733.336478_201.1","section":"bucket.instance","name":"vclfcn-31:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.40","timestamp":"2024-02-01T10:35:33.336478192Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OsUE1ilsRE6kOECGtCDtxHd","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783733.338988_202.1","section":"bucket","name":"vclfcn-31","timestamp":"2024-02-01T10:35:33.338988237Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5E7wmMDv9o08f28guv8WeNq","ver":1},"status":{"status":"write"}}},{"id":"1_1706783733.345150_203.1","section":"bucket","name":"vclfcn-31","timestamp":"2024-02-01T10:35:33.345150939Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5E7wmMDv9o08f28guv8WeNq","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783733.391532_204.1","section":"bucket.instance","name":"vclfcn-32:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.41","timestamp":"2024-02-01T10:35:33.391532601Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PICgXYT5_p4s6Npm6_d7L86","ver":1},"status":{"status":"write"}}},{"id":"1_1706783733.396882_205.1","section":"bucket.instance","name":"vclfcn-32:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.41","timestamp":"2024-02-01T10:35:33.396882656Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PICgXYT5_p4s6Npm6_d7L86","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783733.399225_206.1","section":"bucket","name":"vclfcn-32","timestamp":"2024-02-01T10:35:33.399225264Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jXGNidsm4ya5U_EGuN9pxg2","ver":1},"status":{"status":"write"}}},{"id":"1_1706783733.404709_207.1","section":"bucket","name":"vclfcn-32","timestamp":"2024-02-01T10:35:33.404709608Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jXGNidsm4ya5U_EGuN9pxg2","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783733.435292_208.1","section":"bucket.instance","name":"vclfcn-30:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.39","timestamp":"2024-02-01T10:35:33.435292002Z","data":{"read_version":{"tag":"_HhnF-4-cQEYFZsmhVgAo_3W","ver":1},"write_version":{"tag":"_HhnF-4-cQEYFZsmhVgAo_3W","ver":2},"status":{"status":"write"}}},{"id":"1_1706783733.441533_209.1","section":"bucket.instance","name":"vclfcn-30:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.39","timestamp":"2024-02-01T10:35:33.441533749Z","data":{"read_version":{"tag":"_HhnF-4-cQEYFZsmhVgAo_3W","ver":1},"write_version":{"tag":"_HhnF-4-cQEYFZsmhVgAo_3W","ver":2},"status":{"status":"complete"}}},{"id":"1_1706783733.454934_210.1","section":"bucket.instance","name":"vclfcn-31:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.40","timestamp":"2024-02-01T10:35:33.454934992Z","data":{"read_version":{"tag":"_OsUE1ilsRE6kOECGtCDtxHd","ver":1},"write_version":{"tag":"_OsUE1ilsRE6kOECGtCDtxHd","ver":2},"status":{"status":"write"}}},{"id":"1_1706783733.460796_211.1","section":"bucket.instance","name":"vclfcn-31:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.40","timestamp":"2024-02-01T10:35:33.460796234Z","data":{"read_version":{"tag":"_OsUE1ilsRE6kOECGtCDtxHd","ver":1},"write_version":{"tag":"_OsUE1ilsRE6kOECGtCDtxHd","ver":2},"status":{"status":"complete"}}},{"id":"1_1706783733.521076_212.1","section":"bucket.instance","name":"vclfcn-32:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.41","timestamp":"2024-02-01T10:35:33.521076539Z","data":{"read_version":{"tag":"_PICgXYT5_p4s6Npm6_d7L86","ver":1},"write_version":{"tag":"_PICgXYT5_p4s6Npm6_d7L86","ver":2},"status":{"status":"write"}}},{"id":"1_1706783733.537903_213.1","section":"bucket.instance","name":"vclfcn-32:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.41","timestamp":"2024-02-01T10:35:33.537903176Z","data":{"read_version":{"tag":"_PICgXYT5_p4s6Npm6_d7L86","ver":1},"write_version":{"tag":"_PICgXYT5_p4s6Npm6_d7L86","ver":2},"status":{"status":"complete"}}},{"id":"1_1706783733.593359_214.1","section":"bucket.instance","name":"vclfcn-33:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.42","timestamp":"2024-02-01T10:35:33.593359133Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MHNCkb6d03Mpbe8ExDNyOIe","ver":1},"status":{"status":"write"}}},{"id":"1_1706783733.603919_215.1","section":"bucket.instance","name":"vclfcn-33:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.42","timestamp":"2024-02-01T10:35:33.603919491Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MHNCkb6d03Mpbe8ExDNyOIe","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783733.605766_216.1","section":"bucket","name":"vclfcn-33","timestamp":"2024-02-01T10:35:33.605766755Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_t7Tsi3a8X7oyOSKwXbGKwOO","ver":1},"status":{"status":"write"}}},{"id":"1_1706783733.609698_217.1","section":"bucket","name":"vclfcn-33","timestamp":"2024-02-01T10:35:33.609698777Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_t7Tsi3a8X7oyOSKwXbGKwOO","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783733.628224_218.1","section":"bucket.instance","name":"vclfcn-34:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.43","timestamp":"2024-02-01T10:35:33.628224591Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_V7tzE1wcwyuANojoierxhFz","ver":1},"status":{"status":"write"}}},{"id":"1_1706783733.632156_219.1","section":"bucket.instance","name":"vclfcn-34:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.43","timestamp":"2024-02-01T10:35:33.632156395Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_V7tzE1wcwyuANojoierxhFz","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783733.633573_220.1","section":"bucket","name":"vclfcn-34","timestamp":"2024-02-01T10:35:33.633573750Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Z5byZtsMURp7i2bsj6dBpHv","ver":1},"status":{"status":"write"}}},{"id":"1_1706783733.637172_221.1","section":"bucket","name":"vclfcn-34","timestamp":"2024-02-01T10:35:33.637172728Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Z5byZtsMURp7i2bsj6dBpHv","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783733.677953_222.1","section":"bucket.instance","name":"vclfcn-35:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.44","timestamp":"2024-02-01T10:35:33.677953821Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_C_0FSFOkL0ACO1-gSETb17o","ver":1},"status":{"status":"write"}}},{"id":"1_1706783733.683763_223.1","section":"bucket.instance","name":"vclfcn-35:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.44","timestamp":"2024-02-01T10:35:33.683763713Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_C_0FSFOkL0ACO1-gSETb17o","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783733.686120_224.1","section":"bucket","name":"vclfcn-35","timestamp":"2024-02-01T10:35:33.686120912Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CzHizcA9wKsnJKxAmY_lmbG","ver":1},"status":{"status":"write"}}},{"id":"1_1706783733.691403_225.1","section":"bucket","name":"vclfcn-35","timestamp":"2024-02-01T10:35:33.691403393Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CzHizcA9wKsnJKxAmY_lmbG","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783733.722103_226.1","section":"bucket.instance","name":"vclfcn-33:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.42","timestamp":"2024-02-01T10:35:33.722103624Z","data":{"read_version":{"tag":"_MHNCkb6d03Mpbe8ExDNyOIe","ver":1},"write_version":{"tag":"_MHNCkb6d03Mpbe8ExDNyOIe","ver":2},"status":{"status":"write"}}},{"id":"1_1706783733.727948_227.1","section":"bucket.instance","name":"vclfcn-33:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.42","timestamp":"2024-02-01T10:35:33.727948851Z","data":{"read_version":{"tag":"_MHNCkb6d03Mpbe8ExDNyOIe","ver":1},"write_version":{"tag":"_MHNCkb6d03Mpbe8ExDNyOIe","ver":2},"status":{"status":"complete"}}},{"id":"1_1706783734.739760_228.1","section":"bucket.instance","name":"vclfcn-34:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.43","timestamp":"2024-02-01T10:35:34.739760076Z","data":{"read_version":{"tag":"_V7tzE1wcwyuANojoierxhFz","ver":1},"write_version":{"tag":"_V7tzE1wcwyuANojoierxhFz","ver":2},"status":{"status":"write"}}},{"id":"1_1706783734.747549_229.1","section":"bucket.instance","name":"vclfcn-34:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.43","timestamp":"2024-02-01T10:35:34.747549598Z","data":{"read_version":{"tag":"_V7tzE1wcwyuANojoierxhFz","ver":1},"write_version":{"tag":"_V7tzE1wcwyuANojoierxhFz","ver":2},"status":{"status":"complete"}}},{"id":"1_1706783735.800850_230.1","section":"bucket.instance","name":"vclfcn-35:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.44","timestamp":"2024-02-01T10:35:35.800850730Z","data":{"read_version":{"tag":"_C_0FSFOkL0ACO1-gSETb17o","ver":1},"write_version":{"tag":"_C_0FSFOkL0ACO1-gSETb17o","ver":2},"status":{"status":"write"}}},{"id":"1_1706783735.808526_231.1","section":"bucket.instance","name":"vclfcn-35:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.44","timestamp":"2024-02-01T10:35:35.808526056Z","data":{"read_version":{"tag":"_C_0FSFOkL0ACO1-gSETb17o","ver":1},"write_version":{"tag":"_C_0FSFOkL0ACO1-gSETb17o","ver":2},"status":{"status":"complete"}}},{"id":"1_1706783735.837558_232.1","section":"bucket.instance","name":"vclfcn-36:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.45","timestamp":"2024-02-01T10:35:35.837558347Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9X_9QezGXcrZpslOj2s6DRz","ver":1},"status":{"status":"write"}}},{"id":"1_1706783735.844352_233.1","section":"bucket.instance","name":"vclfcn-36:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.45","timestamp":"2024-02-01T10:35:35.844352448Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9X_9QezGXcrZpslOj2s6DRz","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783735.846902_234.1","section":"bucket","name":"vclfcn-36","timestamp":"2024-02-01T10:35:35.846902742Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_I6xC7jnhAySZruK6KFfJcFj","ver":1},"status":{"status":"write"}}},{"id":"1_1706783735.853365_235.1","section":"bucket","name":"vclfcn-36","timestamp":"2024-02-01T10:35:35.853365976Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_I6xC7jnhAySZruK6KFfJcFj","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783735.880850_236.1","section":"bucket.instance","name":"vclfcn-37:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.46","timestamp":"2024-02-01T10:35:35.880850867Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_S59hS1hjhhTMEjgkmPP0l40","ver":1},"status":{"status":"write"}}},{"id":"1_1706783735.888598_237.1","section":"bucket.instance","name":"vclfcn-37:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.46","timestamp":"2024-02-01T10:35:35.888598691Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_S59hS1hjhhTMEjgkmPP0l40","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783735.891588_238.1","section":"bucket","name":"vclfcn-37","timestamp":"2024-02-01T10:35:35.891588303Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_u9b9H7g55Gxvm7UwsBsXRkc","ver":1},"status":{"status":"write"}}},{"id":"1_1706783735.903784_239.1","section":"bucket","name":"vclfcn-37","timestamp":"2024-02-01T10:35:35.903784912Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_u9b9H7g55Gxvm7UwsBsXRkc","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783735.956302_240.1","section":"bucket.instance","name":"vclfcn-38:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.47","timestamp":"2024-02-01T10:35:35.956302005Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_f3LeuHUV7bKmB4iX_c1NfhG","ver":1},"status":{"status":"write"}}},{"id":"1_1706783735.967905_241.1","section":"bucket.instance","name":"vclfcn-38:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.47","timestamp":"2024-02-01T10:35:35.967905199Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_f3LeuHUV7bKmB4iX_c1NfhG","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783735.969939_242.1","section":"bucket","name":"vclfcn-38","timestamp":"2024-02-01T10:35:35.969939474Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kCOsVG7gpmdckTvQ0-OqkTT","ver":1},"status":{"status":"write"}}},{"id":"1_1706783735.974063_243.1","section":"bucket","name":"vclfcn-38","timestamp":"2024-02-01T10:35:35.974063216Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kCOsVG7gpmdckTvQ0-OqkTT","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783746.868400_244.1","section":"bucket.instance","name":"vclfcn-39:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.48","timestamp":"2024-02-01T10:35:46.868400756Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2ismKVnh0upW2_YAiZ3dCm0","ver":1},"status":{"status":"write"}}},{"id":"1_1706783746.874991_245.1","section":"bucket.instance","name":"vclfcn-39:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.48","timestamp":"2024-02-01T10:35:46.874991156Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2ismKVnh0upW2_YAiZ3dCm0","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783746.877780_246.1","section":"bucket","name":"vclfcn-39","timestamp":"2024-02-01T10:35:46.877780982Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Mtq3wgmpQ2UVbTzYfiOTADK","ver":1},"status":{"status":"write"}}},{"id":"1_1706783746.884330_247.1","section":"bucket","name":"vclfcn-39","timestamp":"2024-02-01T10:35:46.884330514Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Mtq3wgmpQ2UVbTzYfiOTADK","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783746.913795_248.1","section":"bucket.instance","name":"vclfcn-40:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.49","timestamp":"2024-02-01T10:35:46.913795216Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wB4GCZVccemaw1QY5tXPf2_","ver":1},"status":{"status":"write"}}},{"id":"1_1706783746.921924_249.1","section":"bucket.instance","name":"vclfcn-40:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.49","timestamp":"2024-02-01T10:35:46.921924008Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wB4GCZVccemaw1QY5tXPf2_","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783746.925325_250.1","section":"bucket","name":"vclfcn-40","timestamp":"2024-02-01T10:35:46.925325542Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_DzmwhWoct-KkCaXHuhCBDRu","ver":1},"status":{"status":"write"}}},{"id":"1_1706783746.934867_251.1","section":"bucket","name":"vclfcn-40","timestamp":"2024-02-01T10:35:46.934867482Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_DzmwhWoct-KkCaXHuhCBDRu","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783746.990514_252.1","section":"bucket.instance","name":"vclfcn-41:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.50","timestamp":"2024-02-01T10:35:46.990514760Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TYFOExYyROxXd60biRdH4WR","ver":1},"status":{"status":"write"}}},{"id":"1_1706783746.999155_253.1","section":"bucket.instance","name":"vclfcn-41:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.50","timestamp":"2024-02-01T10:35:46.999155781Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TYFOExYyROxXd60biRdH4WR","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783747.001868_254.1","section":"bucket","name":"vclfcn-41","timestamp":"2024-02-01T10:35:47.001868659Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-h9FvYOZI-YjsdziH0V4lBk","ver":1},"status":{"status":"write"}}},{"id":"1_1706783747.007639_255.1","section":"bucket","name":"vclfcn-41","timestamp":"2024-02-01T10:35:47.007639832Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-h9FvYOZI-YjsdziH0V4lBk","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782574.724766_44.1","section":"bucket.instance","name":"vclfcn-1:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.1","timestamp":"2024-02-01T10:16:14.724766052Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ibyZ4ny-dO1eyNWrgR0xDdL","ver":1},"status":{"status":"write"}}},{"id":"1_1706782574.747809_45.1","section":"bucket.instance","name":"vclfcn-1:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.1","timestamp":"2024-02-01T10:16:14.747809032Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ibyZ4ny-dO1eyNWrgR0xDdL","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782574.750145_46.1","section":"bucket","name":"vclfcn-1","timestamp":"2024-02-01T10:16:14.750145401Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_N_oVpqg2avT9RyIn3n93zXI","ver":1},"status":{"status":"write"}}},{"id":"1_1706782574.755240_47.1","section":"bucket","name":"vclfcn-1","timestamp":"2024-02-01T10:16:14.755240100Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_N_oVpqg2avT9RyIn3n93zXI","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782574.779443_48.1","section":"bucket.instance","name":"vclfcn-2:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.2","timestamp":"2024-02-01T10:16:14.779443186Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_50A5h7IO153g3aLFcIk9L8F","ver":1},"status":{"status":"write"}}},{"id":"1_1706782574.784819_49.1","section":"bucket.instance","name":"vclfcn-2:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.2","timestamp":"2024-02-01T10:16:14.784819298Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_50A5h7IO153g3aLFcIk9L8F","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782574.786565_50.1","section":"bucket","name":"vclfcn-2","timestamp":"2024-02-01T10:16:14.786565198Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_t8Dr2abL8IoMUXCcMicPxaz","ver":1},"status":{"status":"write"}}},{"id":"1_1706782574.791675_51.1","section":"bucket","name":"vclfcn-2","timestamp":"2024-02-01T10:16:14.791675561Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_t8Dr2abL8IoMUXCcMicPxaz","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782577.457619_52.1","section":"bucket.instance","name":"vclfcn-3:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.3","timestamp":"2024-02-01T10:16:17.457619330Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AARS1itaxavcuaxye1dSdHX","ver":1},"status":{"status":"write"}}},{"id":"1_1706782577.467024_53.1","section":"bucket.instance","name":"vclfcn-3:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.3","timestamp":"2024-02-01T10:16:17.467024067Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AARS1itaxavcuaxye1dSdHX","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782577.469499_54.1","section":"bucket","name":"vclfcn-3","timestamp":"2024-02-01T10:16:17.469499300Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dY9nFqtzaiF6gs-LUJpzNgy","ver":1},"status":{"status":"write"}}},{"id":"1_1706782577.476062_55.1","section":"bucket","name":"vclfcn-3","timestamp":"2024-02-01T10:16:17.476062234Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dY9nFqtzaiF6gs-LUJpzNgy","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782583.048966_56.1","section":"bucket.instance","name":"vclfcn-4:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.4","timestamp":"2024-02-01T10:16:23.048966924Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jZZOT-7qYVNFmmz--YeJW8c","ver":1},"status":{"status":"write"}}},{"id":"1_1706782583.053776_57.1","section":"bucket.instance","name":"vclfcn-4:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.4","timestamp":"2024-02-01T10:16:23.053776598Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jZZOT-7qYVNFmmz--YeJW8c","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782583.055771_58.1","section":"bucket","name":"vclfcn-4","timestamp":"2024-02-01T10:16:23.055771006Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7FYomkbpFjXsMEVUf--3FIN","ver":1},"status":{"status":"write"}}},{"id":"1_1706782583.060768_59.1","section":"bucket","name":"vclfcn-4","timestamp":"2024-02-01T10:16:23.060768290Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7FYomkbpFjXsMEVUf--3FIN","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782583.080541_60.1","section":"bucket.instance","name":"vclfcn-5:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.5","timestamp":"2024-02-01T10:16:23.080541747Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GuZ_ydrh7uZExo0k-XCf-mR","ver":1},"status":{"status":"write"}}},{"id":"1_1706782583.085084_61.1","section":"bucket.instance","name":"vclfcn-5:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.5","timestamp":"2024-02-01T10:16:23.085084437Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GuZ_ydrh7uZExo0k-XCf-mR","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782583.087346_62.1","section":"bucket","name":"vclfcn-5","timestamp":"2024-02-01T10:16:23.087346765Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KFexYQNYfYmKGSCAKJDSs_v","ver":1},"status":{"status":"write"}}},{"id":"1_1706782583.091891_63.1","section":"bucket","name":"vclfcn-5","timestamp":"2024-02-01T10:16:23.091891354Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KFexYQNYfYmKGSCAKJDSs_v","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782583.135000_64.1","section":"bucket.instance","name":"vclfcn-6:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.6","timestamp":"2024-02-01T10:16:23.135000549Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nUYxPNk4mpHlEoPbPAjZI6V","ver":1},"status":{"status":"write"}}},{"id":"1_1706782583.140997_65.1","section":"bucket.instance","name":"vclfcn-6:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.6","timestamp":"2024-02-01T10:16:23.140997195Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nUYxPNk4mpHlEoPbPAjZI6V","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782583.143037_66.1","section":"bucket","name":"vclfcn-6","timestamp":"2024-02-01T10:16:23.143037696Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4xyZKpWpwMtUf5XA3nUVwvT","ver":1},"status":{"status":"write"}}},{"id":"1_1706782583.148165_67.1","section":"bucket","name":"vclfcn-6","timestamp":"2024-02-01T10:16:23.148165375Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4xyZKpWpwMtUf5XA3nUVwvT","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782583.582106_68.1","section":"bucket.instance","name":"vclfcn-4:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.7","timestamp":"2024-02-01T10:16:23.582106079Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_K-GLBBcTrWLHEjhiQTh8AAO","ver":1},"status":{"status":"write"}}},{"id":"1_1706782583.585480_69.1","section":"bucket.instance","name":"vclfcn-4:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.7","timestamp":"2024-02-01T10:16:23.585480929Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_K-GLBBcTrWLHEjhiQTh8AAO","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782583.586646_70.1","section":"bucket","name":"vclfcn-4","timestamp":"2024-02-01T10:16:23.586646340Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7FYomkbpFjXsMEVUf--3FIN","ver":1},"status":{"status":"write"}}},{"id":"1_1706782583.588852_71.1","section":"bucket","name":"vclfcn-4","timestamp":"2024-02-01T10:16:23.588852889Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7FYomkbpFjXsMEVUf--3FIN","ver":1},"status":{"status":"abort"}}},{"id":"1_1706782583.604965_72.1","section":"bucket.instance","name":"vclfcn-5:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.8","timestamp":"2024-02-01T10:16:23.604965201Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-38jr9mPIsX5Hbh-yF_INzw","ver":1},"status":{"status":"write"}}},{"id":"1_1706782583.608205_73.1","section":"bucket.instance","name":"vclfcn-5:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.8","timestamp":"2024-02-01T10:16:23.608205537Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-38jr9mPIsX5Hbh-yF_INzw","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782583.609727_74.1","section":"bucket","name":"vclfcn-5","timestamp":"2024-02-01T10:16:23.609727931Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KFexYQNYfYmKGSCAKJDSs_v","ver":1},"status":{"status":"write"}}},{"id":"1_1706782583.612105_75.1","section":"bucket","name":"vclfcn-5","timestamp":"2024-02-01T10:16:23.612105467Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KFexYQNYfYmKGSCAKJDSs_v","ver":1},"status":{"status":"abort"}}},{"id":"1_1706782583.632804_76.1","section":"bucket.instance","name":"vclfcn-6:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.9","timestamp":"2024-02-01T10:16:23.632804818Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xPuFAl83iV88j7bpn1hxTNX","ver":1},"status":{"status":"write"}}},{"id":"1_1706782583.638590_77.1","section":"bucket.instance","name":"vclfcn-6:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.9","timestamp":"2024-02-01T10:16:23.638590664Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xPuFAl83iV88j7bpn1hxTNX","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782583.640189_78.1","section":"bucket","name":"vclfcn-6","timestamp":"2024-02-01T10:16:23.640189514Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4xyZKpWpwMtUf5XA3nUVwvT","ver":1},"status":{"status":"write"}}},{"id":"1_1706782583.642926_79.1","section":"bucket","name":"vclfcn-6","timestamp":"2024-02-01T10:16:23.642926395Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4xyZKpWpwMtUf5XA3nUVwvT","ver":1},"status":{"status":"abort"}}},{"id":"1_1706782583.671310_80.1","section":"bucket.instance","name":"vclfcn-4:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.10","timestamp":"2024-02-01T10:16:23.671310356Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1U8H0rwTUD_-21XS12rwqpQ","ver":1},"status":{"status":"write"}}},{"id":"1_1706782583.677081_81.1","section":"bucket.instance","name":"vclfcn-4:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.10","timestamp":"2024-02-01T10:16:23.677081964Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1U8H0rwTUD_-21XS12rwqpQ","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782583.678607_82.1","section":"bucket","name":"vclfcn-4","timestamp":"2024-02-01T10:16:23.678607917Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7FYomkbpFjXsMEVUf--3FIN","ver":1},"status":{"status":"write"}}},{"id":"1_1706782583.681355_83.1","section":"bucket","name":"vclfcn-4","timestamp":"2024-02-01T10:16:23.681355024Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7FYomkbpFjXsMEVUf--3FIN","ver":1},"status":{"status":"abort"}}},{"id":"1_1706782583.724585_84.1","section":"bucket.instance","name":"vclfcn-5:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.11","timestamp":"2024-02-01T10:16:23.724585309Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1uvujq5ws5ZqUnKk2XeudY4","ver":1},"status":{"status":"write"}}},{"id":"1_1706782583.728526_85.1","section":"bucket.instance","name":"vclfcn-5:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.11","timestamp":"2024-02-01T10:16:23.728526702Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1uvujq5ws5ZqUnKk2XeudY4","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782583.729969_86.1","section":"bucket","name":"vclfcn-5","timestamp":"2024-02-01T10:16:23.729969063Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KFexYQNYfYmKGSCAKJDSs_v","ver":1},"status":{"status":"write"}}},{"id":"1_1706782583.733921_87.1","section":"bucket","name":"vclfcn-5","timestamp":"2024-02-01T10:16:23.733921102Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KFexYQNYfYmKGSCAKJDSs_v","ver":1},"status":{"status":"abort"}}},{"id":"1_1706782583.821698_88.1","section":"bucket.instance","name":"vclfcn-6:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.12","timestamp":"2024-02-01T10:16:23.821698722Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_InLI6Y5Soa-cFmcsAza4-1x","ver":1},"status":{"status":"write"}}},{"id":"1_1706782583.828621_89.1","section":"bucket.instance","name":"vclfcn-6:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.12","timestamp":"2024-02-01T10:16:23.828621437Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_InLI6Y5Soa-cFmcsAza4-1x","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782583.831465_90.1","section":"bucket","name":"vclfcn-6","timestamp":"2024-02-01T10:16:23.831465748Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4xyZKpWpwMtUf5XA3nUVwvT","ver":1},"status":{"status":"write"}}},{"id":"1_1706782583.837030_91.1","section":"bucket","name":"vclfcn-6","timestamp":"2024-02-01T10:16:23.837030233Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4xyZKpWpwMtUf5XA3nUVwvT","ver":1},"status":{"status":"abort"}}},{"id":"1_1706782583.931558_92.1","section":"bucket.instance","name":"vclfcn-4:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.13","timestamp":"2024-02-01T10:16:23.931558665Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_mh6kxfzVQf2159X6vdQEtr9","ver":1},"status":{"status":"write"}}},{"id":"1_1706782583.939774_93.1","section":"bucket.instance","name":"vclfcn-4:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.13","timestamp":"2024-02-01T10:16:23.939774137Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_mh6kxfzVQf2159X6vdQEtr9","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782583.942334_94.1","section":"bucket","name":"vclfcn-4","timestamp":"2024-02-01T10:16:23.942334764Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7FYomkbpFjXsMEVUf--3FIN","ver":1},"status":{"status":"write"}}},{"id":"1_1706782583.947201_95.1","section":"bucket","name":"vclfcn-4","timestamp":"2024-02-01T10:16:23.947201866Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7FYomkbpFjXsMEVUf--3FIN","ver":1},"status":{"status":"abort"}}},{"id":"1_1706782583.997720_96.1","section":"bucket.instance","name":"vclfcn-5:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.14","timestamp":"2024-02-01T10:16:23.997720926Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_u7t7dVEXnS4njZhJkFAad6L","ver":1},"status":{"status":"write"}}},{"id":"1_1706782584.004511_97.1","section":"bucket.instance","name":"vclfcn-5:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.14","timestamp":"2024-02-01T10:16:24.004511156Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_u7t7dVEXnS4njZhJkFAad6L","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782584.007009_98.1","section":"bucket","name":"vclfcn-5","timestamp":"2024-02-01T10:16:24.007009106Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KFexYQNYfYmKGSCAKJDSs_v","ver":1},"status":{"status":"write"}}},{"id":"1_1706782584.011561_99.1","section":"bucket","name":"vclfcn-5","timestamp":"2024-02-01T10:16:24.011561820Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KFexYQNYfYmKGSCAKJDSs_v","ver":1},"status":{"status":"abort"}}},{"id":"1_1706782584.107340_100.1","section":"bucket.instance","name":"vclfcn-6:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.15","timestamp":"2024-02-01T10:16:24.107340431Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SwtrgKLrPfbA6nAuIOAjPY5","ver":1},"status":{"status":"write"}}},{"id":"1_1706782584.114866_101.1","section":"bucket.instance","name":"vclfcn-6:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.15","timestamp":"2024-02-01T10:16:24.114866482Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SwtrgKLrPfbA6nAuIOAjPY5","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782584.117713_102.1","section":"bucket","name":"vclfcn-6","timestamp":"2024-02-01T10:16:24.117713159Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4xyZKpWpwMtUf5XA3nUVwvT","ver":1},"status":{"status":"write"}}},{"id":"1_1706782584.123490_103.1","section":"bucket","name":"vclfcn-6","timestamp":"2024-02-01T10:16:24.123490508Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4xyZKpWpwMtUf5XA3nUVwvT","ver":1},"status":{"status":"abort"}}},{"id":"1_1706782589.804979_104.1","section":"bucket.instance","name":"vclfcn-7:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.16","timestamp":"2024-02-01T10:16:29.804979118Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_z1mGy633o2dYXHeltm-sLIr","ver":1},"status":{"status":"write"}}},{"id":"1_1706782589.808233_105.1","section":"bucket.instance","name":"vclfcn-7:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.16","timestamp":"2024-02-01T10:16:29.808233327Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_z1mGy633o2dYXHeltm-sLIr","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782589.809506_106.1","section":"bucket","name":"vclfcn-7","timestamp":"2024-02-01T10:16:29.809506716Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_aQqC4o_dOalWjSRbtEyqQ9y","ver":1},"status":{"status":"write"}}},{"id":"1_1706782589.812385_107.1","section":"bucket","name":"vclfcn-7","timestamp":"2024-02-01T10:16:29.812385583Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_aQqC4o_dOalWjSRbtEyqQ9y","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782589.827390_108.1","section":"bucket.instance","name":"vclfcn-8:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.17","timestamp":"2024-02-01T10:16:29.827390273Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fg266bvdezCNOcfE9SKi_vH","ver":1},"status":{"status":"write"}}},{"id":"1_1706782589.832050_109.1","section":"bucket.instance","name":"vclfcn-8:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.17","timestamp":"2024-02-01T10:16:29.832050187Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fg266bvdezCNOcfE9SKi_vH","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782589.833185_110.1","section":"bucket","name":"vclfcn-8","timestamp":"2024-02-01T10:16:29.833185878Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VxqI9_Wg5TOlSHC_n8MLWjO","ver":1},"status":{"status":"write"}}},{"id":"1_1706782589.836457_111.1","section":"bucket","name":"vclfcn-8","timestamp":"2024-02-01T10:16:29.836457633Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VxqI9_Wg5TOlSHC_n8MLWjO","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782589.877746_112.1","section":"bucket.instance","name":"vclfcn-9:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.18","timestamp":"2024-02-01T10:16:29.877746869Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jIv31DiHuYuZmjj2jtUcl5K","ver":1},"status":{"status":"write"}}},{"id":"1_1706782589.884299_113.1","section":"bucket.instance","name":"vclfcn-9:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.18","timestamp":"2024-02-01T10:16:29.884299625Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jIv31DiHuYuZmjj2jtUcl5K","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782589.886780_114.1","section":"bucket","name":"vclfcn-9","timestamp":"2024-02-01T10:16:29.886780733Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fop2OKI1GWzp_3SQxaIH8uR","ver":1},"status":{"status":"write"}}},{"id":"1_1706782589.893748_115.1","section":"bucket","name":"vclfcn-9","timestamp":"2024-02-01T10:16:29.893748078Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fop2OKI1GWzp_3SQxaIH8uR","ver":1},"status":{"status":"complete"}}},{"id":"1_1706782595.478858_116.1","section":"bucket","name":"vclfcn-7","timestamp":"2024-02-01T10:16:35.478858938Z","data":{"read_version":{"tag":"_aQqC4o_dOalWjSRbtEyqQ9y","ver":1},"write_version":{"tag":"_aQqC4o_dOalWjSRbtEyqQ9y","ver":2},"status":{"status":"remove"}}},{"id":"1_1706782595.482342_117.1","section":"bucket","name":"vclfcn-7","timestamp":"2024-02-01T10:16:35.482342431Z","data":{"read_version":{"tag":"_aQqC4o_dOalWjSRbtEyqQ9y","ver":1},"write_version":{"tag":"_aQqC4o_dOalWjSRbtEyqQ9y","ver":2},"status":{"status":"complete"}}},{"id":"1_1706782595.541467_118.1","section":"bucket","name":"vclfcn-8","timestamp":"2024-02-01T10:16:35.541467766Z","data":{"read_version":{"tag":"_VxqI9_Wg5TOlSHC_n8MLWjO","ver":1},"write_version":{"tag":"_VxqI9_Wg5TOlSHC_n8MLWjO","ver":2},"status":{"status":"remove"}}},{"id":"1_1706782595.545966_119.1","section":"bucket","name":"vclfcn-8","timestamp":"2024-02-01T10:16:35.545966065Z","data":{"read_version":{"tag":"_VxqI9_Wg5TOlSHC_n8MLWjO","ver":1},"write_version":{"tag":"_VxqI9_Wg5TOlSHC_n8MLWjO","ver":2},"status":{"status":"complete"}}},{"id":"1_1706782595.664631_120.1","section":"bucket","name":"vclfcn-9","timestamp":"2024-02-01T10:16:35.664631193Z","data":{"read_version":{"tag":"_fop2OKI1GWzp_3SQxaIH8uR","ver":1},"write_version":{"tag":"_fop2OKI1GWzp_3SQxaIH8uR","ver":2},"status":{"status":"remove"}}},{"id":"1_1706782595.670952_121.1","section":"bucket","name":"vclfcn-9","timestamp":"2024-02-01T10:16:35.670952808Z","data":{"read_version":{"tag":"_fop2OKI1GWzp_3SQxaIH8uR","ver":1},"write_version":{"tag":"_fop2OKI1GWzp_3SQxaIH8uR","ver":2},"status":{"status":"complete"}}}] 2024-02-01T10:38:05.182 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '976daaff-eac6-4c84-bd63-38e72aa959ff', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:05.182 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '976daaff-eac6-4c84-bd63-38e72aa959ff', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:05.183 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 976daaff-eac6-4c84-bd63-38e72aa959ff --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:38:05.254 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:38:05.254 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:38:05.330 INFO:teuthology.orchestra.run.smithi148.stdout:[{"id":"1_1706783813.827558_129.1","section":"bucket.instance","name":"vclfcn-42:01fb6d04-f0ef-49b2-9ebc-c1b530925485.4994.1","timestamp":"2024-02-01T10:36:53.827558969Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2DGa-fzD1lcgw59GJScJXSx","ver":1},"status":{"status":"write"}}},{"id":"1_1706783813.834582_130.1","section":"bucket.instance","name":"vclfcn-42:01fb6d04-f0ef-49b2-9ebc-c1b530925485.4994.1","timestamp":"2024-02-01T10:36:53.834582927Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2DGa-fzD1lcgw59GJScJXSx","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783813.836909_131.1","section":"bucket","name":"vclfcn-42","timestamp":"2024-02-01T10:36:53.836909804Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tEk6DgJXPgCB6nXN6D_nhFN","ver":1},"status":{"status":"write"}}},{"id":"1_1706783813.843251_132.1","section":"bucket","name":"vclfcn-42","timestamp":"2024-02-01T10:36:53.843251202Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tEk6DgJXPgCB6nXN6D_nhFN","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783813.885410_133.1","section":"bucket.instance","name":"vclfcn-43:01fb6d04-f0ef-49b2-9ebc-c1b530925485.4994.2","timestamp":"2024-02-01T10:36:53.885410755Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_12S7aGzI2fwivcem-mJY_Zc","ver":1},"status":{"status":"write"}}},{"id":"1_1706783813.891268_134.1","section":"bucket.instance","name":"vclfcn-43:01fb6d04-f0ef-49b2-9ebc-c1b530925485.4994.2","timestamp":"2024-02-01T10:36:53.891268660Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_12S7aGzI2fwivcem-mJY_Zc","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783813.893469_135.1","section":"bucket","name":"vclfcn-43","timestamp":"2024-02-01T10:36:53.893469630Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_No8wJYnT5bTKPR3WSbsGIv3","ver":1},"status":{"status":"write"}}},{"id":"1_1706783813.899233_136.1","section":"bucket","name":"vclfcn-43","timestamp":"2024-02-01T10:36:53.899233806Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_No8wJYnT5bTKPR3WSbsGIv3","ver":1},"status":{"status":"complete"}}}] 2024-02-01T10:38:05.331 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '976daaff-eac6-4c84-bd63-38e72aa959ff', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:05.331 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '976daaff-eac6-4c84-bd63-38e72aa959ff', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:05.331 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 976daaff-eac6-4c84-bd63-38e72aa959ff --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:38:05.371 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:38:05.371 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:38:05.449 INFO:teuthology.orchestra.run.smithi187.stdout:[{"id":"1_1706783813.827558_129.1","section":"bucket.instance","name":"vclfcn-42:01fb6d04-f0ef-49b2-9ebc-c1b530925485.4994.1","timestamp":"2024-02-01T10:36:53.827558969Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2DGa-fzD1lcgw59GJScJXSx","ver":1},"status":{"status":"write"}}},{"id":"1_1706783813.834582_130.1","section":"bucket.instance","name":"vclfcn-42:01fb6d04-f0ef-49b2-9ebc-c1b530925485.4994.1","timestamp":"2024-02-01T10:36:53.834582927Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2DGa-fzD1lcgw59GJScJXSx","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783813.836909_131.1","section":"bucket","name":"vclfcn-42","timestamp":"2024-02-01T10:36:53.836909804Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tEk6DgJXPgCB6nXN6D_nhFN","ver":1},"status":{"status":"write"}}},{"id":"1_1706783813.843251_132.1","section":"bucket","name":"vclfcn-42","timestamp":"2024-02-01T10:36:53.843251202Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tEk6DgJXPgCB6nXN6D_nhFN","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783813.885410_133.1","section":"bucket.instance","name":"vclfcn-43:01fb6d04-f0ef-49b2-9ebc-c1b530925485.4994.2","timestamp":"2024-02-01T10:36:53.885410755Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_12S7aGzI2fwivcem-mJY_Zc","ver":1},"status":{"status":"write"}}},{"id":"1_1706783813.891268_134.1","section":"bucket.instance","name":"vclfcn-43:01fb6d04-f0ef-49b2-9ebc-c1b530925485.4994.2","timestamp":"2024-02-01T10:36:53.891268660Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_12S7aGzI2fwivcem-mJY_Zc","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783813.893469_135.1","section":"bucket","name":"vclfcn-43","timestamp":"2024-02-01T10:36:53.893469630Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_No8wJYnT5bTKPR3WSbsGIv3","ver":1},"status":{"status":"write"}}},{"id":"1_1706783813.899233_136.1","section":"bucket","name":"vclfcn-43","timestamp":"2024-02-01T10:36:53.899233806Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_No8wJYnT5bTKPR3WSbsGIv3","ver":1},"status":{"status":"complete"}}}] 2024-02-01T10:38:05.450 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '976daaff-eac6-4c84-bd63-38e72aa959ff', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:05.450 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '976daaff-eac6-4c84-bd63-38e72aa959ff', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:05.450 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 976daaff-eac6-4c84-bd63-38e72aa959ff --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:38:05.489 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:38:05.489 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:38:05.567 INFO:teuthology.orchestra.run.smithi148.stdout:[{"id":"1_1706783813.827558_129.1","section":"bucket.instance","name":"vclfcn-42:01fb6d04-f0ef-49b2-9ebc-c1b530925485.4994.1","timestamp":"2024-02-01T10:36:53.827558969Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2DGa-fzD1lcgw59GJScJXSx","ver":1},"status":{"status":"write"}}},{"id":"1_1706783813.834582_130.1","section":"bucket.instance","name":"vclfcn-42:01fb6d04-f0ef-49b2-9ebc-c1b530925485.4994.1","timestamp":"2024-02-01T10:36:53.834582927Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2DGa-fzD1lcgw59GJScJXSx","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783813.836909_131.1","section":"bucket","name":"vclfcn-42","timestamp":"2024-02-01T10:36:53.836909804Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tEk6DgJXPgCB6nXN6D_nhFN","ver":1},"status":{"status":"write"}}},{"id":"1_1706783813.843251_132.1","section":"bucket","name":"vclfcn-42","timestamp":"2024-02-01T10:36:53.843251202Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tEk6DgJXPgCB6nXN6D_nhFN","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783813.885410_133.1","section":"bucket.instance","name":"vclfcn-43:01fb6d04-f0ef-49b2-9ebc-c1b530925485.4994.2","timestamp":"2024-02-01T10:36:53.885410755Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_12S7aGzI2fwivcem-mJY_Zc","ver":1},"status":{"status":"write"}}},{"id":"1_1706783813.891268_134.1","section":"bucket.instance","name":"vclfcn-43:01fb6d04-f0ef-49b2-9ebc-c1b530925485.4994.2","timestamp":"2024-02-01T10:36:53.891268660Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_12S7aGzI2fwivcem-mJY_Zc","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783813.893469_135.1","section":"bucket","name":"vclfcn-43","timestamp":"2024-02-01T10:36:53.893469630Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_No8wJYnT5bTKPR3WSbsGIv3","ver":1},"status":{"status":"write"}}},{"id":"1_1706783813.899233_136.1","section":"bucket","name":"vclfcn-43","timestamp":"2024-02-01T10:36:53.899233806Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_No8wJYnT5bTKPR3WSbsGIv3","ver":1},"status":{"status":"complete"}}}] 2024-02-01T10:38:05.568 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'f2e6c290-e7aa-4279-b77e-15403b18d61f', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:05.568 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'f2e6c290-e7aa-4279-b77e-15403b18d61f', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:05.568 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period f2e6c290-e7aa-4279-b77e-15403b18d61f --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:38:05.642 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:38:05.643 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:38:05.726 INFO:teuthology.orchestra.run.smithi148.stdout:[{"id":"1_1706783874.485999_97.1","section":"bucket.instance","name":"vclfcn-44:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.1","timestamp":"2024-02-01T10:37:54.485999388Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RpLmglqhnFTDNkz7x5FxvWC","ver":1},"status":{"status":"write"}}},{"id":"1_1706783874.492836_98.1","section":"bucket.instance","name":"vclfcn-44:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.1","timestamp":"2024-02-01T10:37:54.492836549Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RpLmglqhnFTDNkz7x5FxvWC","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783874.495361_99.1","section":"bucket","name":"vclfcn-44","timestamp":"2024-02-01T10:37:54.495361334Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Xe-yVkV5xQAkEcvw76GN33r","ver":1},"status":{"status":"write"}}},{"id":"1_1706783874.501218_100.1","section":"bucket","name":"vclfcn-44","timestamp":"2024-02-01T10:37:54.501218679Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Xe-yVkV5xQAkEcvw76GN33r","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783874.533837_101.1","section":"bucket.instance","name":"vclfcn-45:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.2","timestamp":"2024-02-01T10:37:54.533837509Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_unGR34TqPxeG931GH5eQTS5","ver":1},"status":{"status":"write"}}},{"id":"1_1706783874.539239_102.1","section":"bucket.instance","name":"vclfcn-45:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.2","timestamp":"2024-02-01T10:37:54.539239570Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_unGR34TqPxeG931GH5eQTS5","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783874.540800_103.1","section":"bucket","name":"vclfcn-45","timestamp":"2024-02-01T10:37:54.540800598Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_W8MRvBrFGAA73hVyHDDxPfM","ver":1},"status":{"status":"write"}}},{"id":"1_1706783874.544771_104.1","section":"bucket","name":"vclfcn-45","timestamp":"2024-02-01T10:37:54.544771711Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_W8MRvBrFGAA73hVyHDDxPfM","ver":1},"status":{"status":"complete"}}}] 2024-02-01T10:38:05.726 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'f2e6c290-e7aa-4279-b77e-15403b18d61f', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:05.726 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'f2e6c290-e7aa-4279-b77e-15403b18d61f', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:05.727 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period f2e6c290-e7aa-4279-b77e-15403b18d61f --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:38:05.767 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:38:05.768 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:38:05.846 INFO:teuthology.orchestra.run.smithi187.stdout:[{"id":"1_1706783874.485999_97.1","section":"bucket.instance","name":"vclfcn-44:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.1","timestamp":"2024-02-01T10:37:54.485999388Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RpLmglqhnFTDNkz7x5FxvWC","ver":1},"status":{"status":"write"}}},{"id":"1_1706783874.492836_98.1","section":"bucket.instance","name":"vclfcn-44:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.1","timestamp":"2024-02-01T10:37:54.492836549Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RpLmglqhnFTDNkz7x5FxvWC","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783874.495361_99.1","section":"bucket","name":"vclfcn-44","timestamp":"2024-02-01T10:37:54.495361334Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Xe-yVkV5xQAkEcvw76GN33r","ver":1},"status":{"status":"write"}}},{"id":"1_1706783874.501218_100.1","section":"bucket","name":"vclfcn-44","timestamp":"2024-02-01T10:37:54.501218679Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Xe-yVkV5xQAkEcvw76GN33r","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783874.533837_101.1","section":"bucket.instance","name":"vclfcn-45:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.2","timestamp":"2024-02-01T10:37:54.533837509Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_unGR34TqPxeG931GH5eQTS5","ver":1},"status":{"status":"write"}}},{"id":"1_1706783874.539239_102.1","section":"bucket.instance","name":"vclfcn-45:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.2","timestamp":"2024-02-01T10:37:54.539239570Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_unGR34TqPxeG931GH5eQTS5","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783874.540800_103.1","section":"bucket","name":"vclfcn-45","timestamp":"2024-02-01T10:37:54.540800598Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_W8MRvBrFGAA73hVyHDDxPfM","ver":1},"status":{"status":"write"}}},{"id":"1_1706783874.544771_104.1","section":"bucket","name":"vclfcn-45","timestamp":"2024-02-01T10:37:54.544771711Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_W8MRvBrFGAA73hVyHDDxPfM","ver":1},"status":{"status":"complete"}}}] 2024-02-01T10:38:05.846 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'f2e6c290-e7aa-4279-b77e-15403b18d61f', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:05.846 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'f2e6c290-e7aa-4279-b77e-15403b18d61f', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:05.846 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period f2e6c290-e7aa-4279-b77e-15403b18d61f --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:38:05.886 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:38:05.886 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:38:05.963 INFO:teuthology.orchestra.run.smithi148.stdout:[{"id":"1_1706783874.485999_97.1","section":"bucket.instance","name":"vclfcn-44:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.1","timestamp":"2024-02-01T10:37:54.485999388Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RpLmglqhnFTDNkz7x5FxvWC","ver":1},"status":{"status":"write"}}},{"id":"1_1706783874.492836_98.1","section":"bucket.instance","name":"vclfcn-44:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.1","timestamp":"2024-02-01T10:37:54.492836549Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RpLmglqhnFTDNkz7x5FxvWC","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783874.495361_99.1","section":"bucket","name":"vclfcn-44","timestamp":"2024-02-01T10:37:54.495361334Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Xe-yVkV5xQAkEcvw76GN33r","ver":1},"status":{"status":"write"}}},{"id":"1_1706783874.501218_100.1","section":"bucket","name":"vclfcn-44","timestamp":"2024-02-01T10:37:54.501218679Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Xe-yVkV5xQAkEcvw76GN33r","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783874.533837_101.1","section":"bucket.instance","name":"vclfcn-45:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.2","timestamp":"2024-02-01T10:37:54.533837509Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_unGR34TqPxeG931GH5eQTS5","ver":1},"status":{"status":"write"}}},{"id":"1_1706783874.539239_102.1","section":"bucket.instance","name":"vclfcn-45:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.2","timestamp":"2024-02-01T10:37:54.539239570Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_unGR34TqPxeG931GH5eQTS5","ver":1},"status":{"status":"complete"}}},{"id":"1_1706783874.540800_103.1","section":"bucket","name":"vclfcn-45","timestamp":"2024-02-01T10:37:54.540800598Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_W8MRvBrFGAA73hVyHDDxPfM","ver":1},"status":{"status":"write"}}},{"id":"1_1706783874.544771_104.1","section":"bucket","name":"vclfcn-45","timestamp":"2024-02-01T10:37:54.544771711Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_W8MRvBrFGAA73hVyHDDxPfM","ver":1},"status":{"status":"complete"}}}] 2024-02-01T10:38:05.964 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:38:05.964 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:38:05.964 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2024-02-01T10:38:06.036 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:38:06.036 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:38:06.134 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:38:06.134 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:38:06.134 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog autotrim --cluster c2 --debug-rgw 0 --debug-ms 0 2024-02-01T10:38:06.171 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:38:06.172 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:38:06.387 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:38:06.387 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:38:06.387 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2024-02-01T10:38:06.424 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:38:06.425 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:38:06.509 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '5dc4efbd-eccd-47a8-9471-ae4a209ec978', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:06.509 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '5dc4efbd-eccd-47a8-9471-ae4a209ec978', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:06.510 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 5dc4efbd-eccd-47a8-9471-ae4a209ec978 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:38:06.581 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:38:06.582 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:38:06.657 INFO:teuthology.orchestra.run.smithi148.stdout:[] 2024-02-01T10:38:06.657 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '5dc4efbd-eccd-47a8-9471-ae4a209ec978', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:06.657 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '5dc4efbd-eccd-47a8-9471-ae4a209ec978', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:06.658 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 5dc4efbd-eccd-47a8-9471-ae4a209ec978 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:38:06.698 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:38:06.698 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:38:06.778 INFO:teuthology.orchestra.run.smithi187.stdout:[] 2024-02-01T10:38:06.778 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '5dc4efbd-eccd-47a8-9471-ae4a209ec978', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:06.779 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '5dc4efbd-eccd-47a8-9471-ae4a209ec978', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:06.779 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 5dc4efbd-eccd-47a8-9471-ae4a209ec978 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:38:06.817 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:38:06.818 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:38:06.895 INFO:teuthology.orchestra.run.smithi148.stdout:[] 2024-02-01T10:38:06.896 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '976daaff-eac6-4c84-bd63-38e72aa959ff', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:06.896 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '976daaff-eac6-4c84-bd63-38e72aa959ff', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:06.896 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 976daaff-eac6-4c84-bd63-38e72aa959ff --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:38:06.972 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:38:06.972 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:38:07.048 INFO:teuthology.orchestra.run.smithi148.stdout:[] 2024-02-01T10:38:07.048 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '976daaff-eac6-4c84-bd63-38e72aa959ff', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:07.049 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '976daaff-eac6-4c84-bd63-38e72aa959ff', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:07.049 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 976daaff-eac6-4c84-bd63-38e72aa959ff --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:38:07.089 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:38:07.089 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:38:07.167 INFO:teuthology.orchestra.run.smithi187.stdout:[] 2024-02-01T10:38:07.168 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '976daaff-eac6-4c84-bd63-38e72aa959ff', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:07.168 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '976daaff-eac6-4c84-bd63-38e72aa959ff', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:07.168 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 976daaff-eac6-4c84-bd63-38e72aa959ff --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:38:07.208 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:38:07.208 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:38:07.289 INFO:teuthology.orchestra.run.smithi148.stdout:[] 2024-02-01T10:38:07.290 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'f2e6c290-e7aa-4279-b77e-15403b18d61f', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:07.290 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'f2e6c290-e7aa-4279-b77e-15403b18d61f', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:07.290 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period f2e6c290-e7aa-4279-b77e-15403b18d61f --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:38:07.372 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:38:07.372 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:38:07.449 INFO:teuthology.orchestra.run.smithi148.stdout:[] 2024-02-01T10:38:07.450 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'f2e6c290-e7aa-4279-b77e-15403b18d61f', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:07.450 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'f2e6c290-e7aa-4279-b77e-15403b18d61f', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:07.450 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period f2e6c290-e7aa-4279-b77e-15403b18d61f --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:38:07.491 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:38:07.491 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:38:07.572 INFO:teuthology.orchestra.run.smithi187.stdout:[] 2024-02-01T10:38:07.573 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'f2e6c290-e7aa-4279-b77e-15403b18d61f', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:07.573 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'f2e6c290-e7aa-4279-b77e-15403b18d61f', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:07.573 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period f2e6c290-e7aa-4279-b77e-15403b18d61f --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:38:07.614 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:38:07.614 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:38:07.696 INFO:teuthology.orchestra.run.smithi148.stdout:[] 2024-02-01T10:38:07.697 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_period_incremental_sync ... ok 2024-02-01T10:38:07.698 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=vclfcn-46 2024-02-01T10:38:07.698 DEBUG:boto:path=/vclfcn-46/ 2024-02-01T10:38:07.698 DEBUG:boto:auth_path=/vclfcn-46/ 2024-02-01T10:38:07.698 DEBUG:boto:Method: PUT 2024-02-01T10:38:07.699 DEBUG:boto:Path: /vclfcn-46/ 2024-02-01T10:38:07.699 DEBUG:boto:Data: 2024-02-01T10:38:07.699 DEBUG:boto:Headers: {} 2024-02-01T10:38:07.699 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:07.699 DEBUG:boto:Port: 8000 2024-02-01T10:38:07.699 DEBUG:boto:Params: {} 2024-02-01T10:38:07.699 DEBUG:boto:Token: None 2024-02-01T10:38:07.699 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:38:07 GMT /vclfcn-46/ 2024-02-01T10:38:07.700 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vaayT4g8/ccfS/ImtKKYGH6d6jE= 2024-02-01T10:38:07.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vaayT4g8/ccfS/ImtKKYGH6d6jE=', 'Content-Length': '0'} 2024-02-01T10:38:07.716 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001e35f50057f87448-0065bb748f-5317-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:07.717 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=vclfcn-47 2024-02-01T10:38:07.717 DEBUG:boto:path=/vclfcn-47/ 2024-02-01T10:38:07.717 DEBUG:boto:auth_path=/vclfcn-47/ 2024-02-01T10:38:07.717 DEBUG:boto:Method: PUT 2024-02-01T10:38:07.717 DEBUG:boto:Path: /vclfcn-47/ 2024-02-01T10:38:07.717 DEBUG:boto:Data: 2024-02-01T10:38:07.717 DEBUG:boto:Headers: {} 2024-02-01T10:38:07.718 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:07.718 DEBUG:boto:Port: 8000 2024-02-01T10:38:07.718 DEBUG:boto:Params: {} 2024-02-01T10:38:07.718 DEBUG:boto:Token: None 2024-02-01T10:38:07.718 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:38:07 GMT /vclfcn-47/ 2024-02-01T10:38:07.718 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Hb4xPWuwDdPx6avr1tTfJutR62c= 2024-02-01T10:38:07.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Hb4xPWuwDdPx6avr1tTfJutR62c=', 'Content-Length': '0'} 2024-02-01T10:38:07.764 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000008de903f74267b9a-0065bb748f-5004-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:07.764 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=vclfcn-48 2024-02-01T10:38:07.764 DEBUG:boto:path=/vclfcn-48/ 2024-02-01T10:38:07.764 DEBUG:boto:auth_path=/vclfcn-48/ 2024-02-01T10:38:07.764 DEBUG:boto:Method: PUT 2024-02-01T10:38:07.764 DEBUG:boto:Path: /vclfcn-48/ 2024-02-01T10:38:07.765 DEBUG:boto:Data: 2024-02-01T10:38:07.765 DEBUG:boto:Headers: {} 2024-02-01T10:38:07.765 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:07.765 DEBUG:boto:Port: 8001 2024-02-01T10:38:07.765 DEBUG:boto:Params: {} 2024-02-01T10:38:07.765 DEBUG:boto:Token: None 2024-02-01T10:38:07.765 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:38:07 GMT /vclfcn-48/ 2024-02-01T10:38:07.765 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:g98gOe1w0fc6JE/sO3mNfXP6534= 2024-02-01T10:38:07.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:g98gOe1w0fc6JE/sO3mNfXP6534=', 'Content-Length': '0'} 2024-02-01T10:38:07.805 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005c6c8760c0a72a70-0065bb748f-5324-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:07.806 DEBUG:boto:path=/vclfcn-46/ 2024-02-01T10:38:07.806 DEBUG:boto:auth_path=/vclfcn-46/ 2024-02-01T10:38:07.806 DEBUG:boto:Method: HEAD 2024-02-01T10:38:07.806 DEBUG:boto:Path: /vclfcn-46/ 2024-02-01T10:38:07.806 DEBUG:boto:Data: 2024-02-01T10:38:07.806 DEBUG:boto:Headers: {} 2024-02-01T10:38:07.807 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:07.807 DEBUG:boto:Port: 8000 2024-02-01T10:38:07.807 DEBUG:boto:Params: {} 2024-02-01T10:38:07.807 DEBUG:boto:Token: None 2024-02-01T10:38:07.807 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:07 GMT /vclfcn-46/ 2024-02-01T10:38:07.807 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HIYYb64vuP40iQhTJxiUn/x61KE= 2024-02-01T10:38:07.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HIYYb64vuP40iQhTJxiUn/x61KE=', 'Content-Length': '0'} 2024-02-01T10:38:07.810 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000faf6dd5f5ea7afaa-0065bb748f-5317-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:07.810 DEBUG:boto:path=/vclfcn-46/key 2024-02-01T10:38:07.810 DEBUG:boto:auth_path=/vclfcn-46/key 2024-02-01T10:38:07.810 DEBUG:boto:Method: PUT 2024-02-01T10:38:07.811 DEBUG:boto:Path: /vclfcn-46/key 2024-02-01T10:38:07.811 DEBUG:boto:Data: 2024-02-01T10:38:07.811 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue'} 2024-02-01T10:38:07.811 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:38:07.811 DEBUG:boto:Port: 8000 2024-02-01T10:38:07.811 DEBUG:boto:Params: {} 2024-02-01T10:38:07.811 DEBUG:boto:Token: None 2024-02-01T10:38:07.811 DEBUG:boto:StringToSign: PUT hBotaJrYa9FhFEdFPCLG/A== application/octet-stream Thu, 01 Feb 2024 10:38:07 GMT /vclfcn-46/key 2024-02-01T10:38:07.811 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fyBCw7cSwX43eO+NXt1awWOXDLM= 2024-02-01T10:38:07.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:38:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fyBCw7cSwX43eO+NXt1awWOXDLM='} 2024-02-01T10:38:07.860 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"841a2d689ad86bd1611447453c22c6fc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a23c8b6ae0da7fe2-0065bb748f-5317-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:07.860 DEBUG:boto:path=/vclfcn-47/ 2024-02-01T10:38:07.861 DEBUG:boto:auth_path=/vclfcn-47/ 2024-02-01T10:38:07.861 DEBUG:boto:Method: HEAD 2024-02-01T10:38:07.861 DEBUG:boto:Path: /vclfcn-47/ 2024-02-01T10:38:07.861 DEBUG:boto:Data: 2024-02-01T10:38:07.861 DEBUG:boto:Headers: {} 2024-02-01T10:38:07.861 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:07.861 DEBUG:boto:Port: 8000 2024-02-01T10:38:07.861 DEBUG:boto:Params: {} 2024-02-01T10:38:07.861 DEBUG:boto:Token: None 2024-02-01T10:38:07.862 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:07 GMT /vclfcn-47/ 2024-02-01T10:38:07.862 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:pWpghQALtkoCa0KP5d0rxHCYwOM= 2024-02-01T10:38:07.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:pWpghQALtkoCa0KP5d0rxHCYwOM=', 'Content-Length': '0'} 2024-02-01T10:38:07.866 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004129003f50fb98a4-0065bb748f-5004-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:07.867 DEBUG:boto:path=/vclfcn-47/key 2024-02-01T10:38:07.867 DEBUG:boto:auth_path=/vclfcn-47/key 2024-02-01T10:38:07.867 DEBUG:boto:Method: PUT 2024-02-01T10:38:07.867 DEBUG:boto:Path: /vclfcn-47/key 2024-02-01T10:38:07.867 DEBUG:boto:Data: 2024-02-01T10:38:07.867 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue'} 2024-02-01T10:38:07.867 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:38:07.867 DEBUG:boto:Port: 8000 2024-02-01T10:38:07.867 DEBUG:boto:Params: {} 2024-02-01T10:38:07.868 DEBUG:boto:Token: None 2024-02-01T10:38:07.868 DEBUG:boto:StringToSign: PUT hBotaJrYa9FhFEdFPCLG/A== application/octet-stream Thu, 01 Feb 2024 10:38:07 GMT /vclfcn-47/key 2024-02-01T10:38:07.868 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:LkLDbedCF5HHvgJ5YV4MIq35Nzc= 2024-02-01T10:38:07.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:38:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:LkLDbedCF5HHvgJ5YV4MIq35Nzc='} 2024-02-01T10:38:07.912 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"841a2d689ad86bd1611447453c22c6fc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000058332d0faedb97b1-0065bb748f-5004-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:07.913 DEBUG:boto:path=/vclfcn-48/ 2024-02-01T10:38:07.913 DEBUG:boto:auth_path=/vclfcn-48/ 2024-02-01T10:38:07.913 DEBUG:boto:Method: HEAD 2024-02-01T10:38:07.913 DEBUG:boto:Path: /vclfcn-48/ 2024-02-01T10:38:07.913 DEBUG:boto:Data: 2024-02-01T10:38:07.913 DEBUG:boto:Headers: {} 2024-02-01T10:38:07.913 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:07.913 DEBUG:boto:Port: 8001 2024-02-01T10:38:07.913 DEBUG:boto:Params: {} 2024-02-01T10:38:07.915 DEBUG:boto:Token: None 2024-02-01T10:38:07.915 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:38:07 GMT /vclfcn-48/ 2024-02-01T10:38:07.915 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:pvQFg3SHCY35hQPgLVX0lUEfjsY= 2024-02-01T10:38:07.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:38:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:pvQFg3SHCY35hQPgLVX0lUEfjsY=', 'Content-Length': '0'} 2024-02-01T10:38:07.920 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c48190c61a8c8549-0065bb748f-5324-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:38:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:07.920 DEBUG:boto:path=/vclfcn-48/key 2024-02-01T10:38:07.921 DEBUG:boto:auth_path=/vclfcn-48/key 2024-02-01T10:38:07.921 DEBUG:boto:Method: PUT 2024-02-01T10:38:07.921 DEBUG:boto:Path: /vclfcn-48/key 2024-02-01T10:38:07.921 DEBUG:boto:Data: 2024-02-01T10:38:07.921 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue'} 2024-02-01T10:38:07.921 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:38:07.921 DEBUG:boto:Port: 8001 2024-02-01T10:38:07.921 DEBUG:boto:Params: {} 2024-02-01T10:38:07.921 DEBUG:boto:Token: None 2024-02-01T10:38:07.922 DEBUG:boto:StringToSign: PUT hBotaJrYa9FhFEdFPCLG/A== application/octet-stream Thu, 01 Feb 2024 10:38:07 GMT /vclfcn-48/key 2024-02-01T10:38:07.922 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:KhbVLZO5PcvBcNUfbk8PFKYSRiA= 2024-02-01T10:38:07.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:38:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:KhbVLZO5PcvBcNUfbk8PFKYSRiA='} 2024-02-01T10:38:07.940 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"841a2d689ad86bd1611447453c22c6fc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d87acf136b7a44ad-0065bb748f-5324-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:38:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:38:07.941 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:07.941 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:07.941 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:38:07.984 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:38:07.985 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:38:08.040 INFO:teuthology.orchestra.run.smithi148.stderr:No --period given, using current period=f2e6c290-e7aa-4279-b77e-15403b18d61f 2024-02-01T10:38:08.062 INFO:teuthology.orchestra.run.smithi148.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1706783887.801303_121.1","last_update":"2024-02-01T10:38:07.801303Z"},{"marker":"","last_update":"0.000000"}] 2024-02-01T10:38:08.062 DEBUG:rgw_multi.tests:master meta markers={0: '', 1: '', 2: '1_1706783887.801303_121.1', 3: ''} 2024-02-01T10:38:08.062 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-02-01T10:38:08.062 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:08.063 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:08.063 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:38:08.104 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:38:08.104 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:38:08.158 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:38:08.157+0000 7f39bebc6980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:38:08.158 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:38:08.157+0000 7f39bebc6980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:38:08.182 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783887.715895_108.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:38:07.715895Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:38:08Z"} 2024-02-01T10:38:08.183 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783887.715895_108.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:38:07.715895Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:38:08Z"} 2024-02-01T10:38:08.183 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706783887.715895_108.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:38:07.715895Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:38:08.183 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1706783887.801303_121.1', 3: ''} 2024-02-01T10:38:08.183 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1706783887.715895_108.1', 3: ''} 2024-02-01T10:38:08.183 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1706783887.801303_121.1 target=1_1706783887.715895_108.1 2024-02-01T10:38:09.746 INFO:tasks.rgw.c2.client.0.smithi187.stdout:2024-02-01T10:38:09.744+0000 7f7be9d55640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-02-01T10:38:13.185 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:13.185 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:13.185 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:38:13.228 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:38:13.229 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:38:13.285 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:38:13.280+0000 7f124cc37980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:38:13.286 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:38:13.284+0000 7f124cc37980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:38:13.309 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783887.801303_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:38:07.801303Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:38:13Z"} 2024-02-01T10:38:13.310 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783887.801303_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:38:07.801303Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:38:13Z"} 2024-02-01T10:38:13.310 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706783887.801303_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:38:07.801303Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:38:13.310 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1706783887.801303_121.1', 3: ''} 2024-02-01T10:38:13.310 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1706783887.801303_121.1', 3: ''} 2024-02-01T10:38:13.310 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-02-01T10:38:13.311 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-02-01T10:38:13.311 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:13.311 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:13.311 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:38:13.353 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:38:13.354 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:38:13.408 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:38:13.412+0000 7f5ba5dcd980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:38:13.409 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:38:13.412+0000 7f5ba5dcd980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:38:13.432 INFO:teuthology.orchestra.run.smithi148.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783887.801303_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:38:07.801303Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:38:13Z"} 2024-02-01T10:38:13.433 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783887.801303_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:38:07.801303Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:38:13Z"} 2024-02-01T10:38:13.433 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706783887.801303_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:38:07.801303Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:38:13.433 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1706783887.801303_121.1', 3: ''} 2024-02-01T10:38:13.433 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1706783887.801303_121.1', 3: ''} 2024-02-01T10:38:13.433 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-02-01T10:38:13.433 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone1 target=test-zone2 2024-02-01T10:38:13.434 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:13.434 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:13.434 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:38:13.507 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:38:13.507 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:38:13.590 INFO:teuthology.orchestra.run.smithi148.stdout:[{"marker":"00000000000000000000:00000000000000002204","last_update":"2024-02-01T10:21:48.024326Z"},{"marker":"00000000000000000000:00000000000000004601","last_update":"2024-02-01T10:38:07.826686Z"},{"marker":"00000000000000000000:00000000000000005870","last_update":"2024-02-01T10:25:27.559832Z"},{"marker":"00000000000000000000:00000000000000000512","last_update":"2024-02-01T10:35:46.620905Z"}] 2024-02-01T10:38:13.591 DEBUG:rgw_multi.tests:data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000002204', 1: '00000000000000000000:00000000000000004601', 2: '00000000000000000000:00000000000000005870', 3: '00000000000000000000:00000000000000000512'} 2024-02-01T10:38:13.591 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-02-01T10:38:13.591 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:13.591 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:13.591 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:38:13.636 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:38:13.636 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:38:13.717 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":12760245309917388665},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002204","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:21:48.024326Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004601","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:38:07.826686Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000005870","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:25:27.559832Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000512","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:35:46.620905Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-02-01T10:38:13Z"} 2024-02-01T10:38:13.717 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":12760245309917388665},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002204","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:21:48.024326Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004601","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:38:07.826686Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000005870","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:25:27.559832Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000512","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:35:46.620905Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-02-01T10:38:13Z"} 2024-02-01T10:38:13.717 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002204', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:21:48.024326Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004601', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:38:07.826686Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005870', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:25:27.559832Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000512', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:35:46.620905Z'}}] 2024-02-01T10:38:13.717 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002204', 1: '00000000000000000000:00000000000000004601', 2: '00000000000000000000:00000000000000005870', 3: '00000000000000000000:00000000000000000512'} 2024-02-01T10:38:13.717 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002204', 1: '00000000000000000000:00000000000000004601', 2: '00000000000000000000:00000000000000005870', 3: '00000000000000000000:00000000000000000512'} 2024-02-01T10:38:13.717 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-02-01T10:38:13.718 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone1 target=test-zone3 2024-02-01T10:38:13.718 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:13.718 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:13.718 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:38:13.760 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:38:13.760 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:38:13.852 INFO:teuthology.orchestra.run.smithi148.stdout:[{"marker":"00000000000000000000:00000000000000002204","last_update":"2024-02-01T10:21:48.024326Z"},{"marker":"00000000000000000000:00000000000000004601","last_update":"2024-02-01T10:38:07.826686Z"},{"marker":"00000000000000000000:00000000000000005870","last_update":"2024-02-01T10:25:27.559832Z"},{"marker":"00000000000000000000:00000000000000000512","last_update":"2024-02-01T10:35:46.620905Z"}] 2024-02-01T10:38:13.852 DEBUG:rgw_multi.tests:data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000002204', 1: '00000000000000000000:00000000000000004601', 2: '00000000000000000000:00000000000000005870', 3: '00000000000000000000:00000000000000000512'} 2024-02-01T10:38:13.852 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2024-02-01T10:38:13.853 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:13.853 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:13.853 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:38:13.929 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:38:13.929 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:38:14.003 INFO:teuthology.orchestra.run.smithi148.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":8365149014541277257},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002204","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:21:48.024326Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004460","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:33:54.075193Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000005870","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:25:27.559832Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000512","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:35:46.620905Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-02-01T10:38:13Z"} 2024-02-01T10:38:14.003 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":8365149014541277257},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002204","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:21:48.024326Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004460","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:33:54.075193Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000005870","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:25:27.559832Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000512","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:35:46.620905Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-02-01T10:38:13Z"} 2024-02-01T10:38:14.003 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002204', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:21:48.024326Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004460', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:33:54.075193Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005870', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:25:27.559832Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000512', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:35:46.620905Z'}}] 2024-02-01T10:38:14.003 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002204', 1: '00000000000000000000:00000000000000004601', 2: '00000000000000000000:00000000000000005870', 3: '00000000000000000000:00000000000000000512'} 2024-02-01T10:38:14.003 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002204', 1: '00000000000000000000:00000000000000004460', 2: '00000000000000000000:00000000000000005870', 3: '00000000000000000000:00000000000000000512'} 2024-02-01T10:38:14.003 WARNING:rgw_multi.tests:data of zone test-zone3 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000004601 target=00000000000000000000:00000000000000004460 2024-02-01T10:38:14.688 INFO:tasks.rgw.c2.client.0.smithi187.stdout:2024-02-01T10:38:14.684+0000 7f7bead57640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-02-01T10:38:19.004 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:19.004 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:19.005 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:38:19.050 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:38:19.050 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:38:19.129 INFO:teuthology.orchestra.run.smithi148.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":8365149014541277257},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002204","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:21:48.024326Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004741","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:38:14.727082Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000005870","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:25:27.559832Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000796","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:38:14.730286Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-02-01T10:38:19Z"} 2024-02-01T10:38:19.129 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":8365149014541277257},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002204","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:21:48.024326Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004741","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:38:14.727082Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000005870","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:25:27.559832Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000796","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:38:14.730286Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-02-01T10:38:19Z"} 2024-02-01T10:38:19.129 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002204', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:21:48.024326Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004741', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:38:14.727082Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005870', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:25:27.559832Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000796', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:38:14.730286Z'}}] 2024-02-01T10:38:19.130 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002204', 1: '00000000000000000000:00000000000000004601', 2: '00000000000000000000:00000000000000005870', 3: '00000000000000000000:00000000000000000512'} 2024-02-01T10:38:19.130 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002204', 1: '00000000000000000000:00000000000000004741', 2: '00000000000000000000:00000000000000005870', 3: '00000000000000000000:00000000000000000796'} 2024-02-01T10:38:19.130 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2024-02-01T10:38:19.130 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone2 target=test-zone1 2024-02-01T10:38:19.130 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:19.130 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:19.130 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:38:19.172 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:38:19.172 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:38:19.258 INFO:teuthology.orchestra.run.smithi187.stdout:[{"marker":"00000000000000000000:00000000000000002486","last_update":"2024-02-01T10:22:17.904424Z"},{"marker":"00000000000000000000:00000000000000005164","last_update":"2024-02-01T10:38:09.883042Z"},{"marker":"00000000000000000000:00000000000000004601","last_update":"2024-02-01T10:24:00.115466Z"},{"marker":"00000000000000000000:00000000000000000796","last_update":"2024-02-01T10:38:14.839079Z"}] 2024-02-01T10:38:19.258 DEBUG:rgw_multi.tests:data markers for zone=test-zone2 markers={0: '00000000000000000000:00000000000000002486', 1: '00000000000000000000:00000000000000005164', 2: '00000000000000000000:00000000000000004601', 3: '00000000000000000000:00000000000000000796'} 2024-02-01T10:38:19.259 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2024-02-01T10:38:19.259 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:19.259 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:19.259 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:38:19.304 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:38:19.304 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:38:19.384 INFO:teuthology.orchestra.run.smithi148.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":8595394196901466828},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002486","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:22:17.904424Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000005164","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:38:09.883042Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004601","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:24:00.115466Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000654","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:35:47.376932Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-02-01T10:38:19Z"} 2024-02-01T10:38:19.384 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":8595394196901466828},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002486","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:22:17.904424Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000005164","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:38:09.883042Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004601","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:24:00.115466Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000654","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:35:47.376932Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-02-01T10:38:19Z"} 2024-02-01T10:38:19.385 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002486', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:22:17.904424Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005164', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:38:09.883042Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004601', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:24:00.115466Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000654', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:35:47.376932Z'}}] 2024-02-01T10:38:19.385 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002486', 1: '00000000000000000000:00000000000000005164', 2: '00000000000000000000:00000000000000004601', 3: '00000000000000000000:00000000000000000796'} 2024-02-01T10:38:19.386 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002486', 1: '00000000000000000000:00000000000000005164', 2: '00000000000000000000:00000000000000004601', 3: '00000000000000000000:00000000000000000654'} 2024-02-01T10:38:19.386 WARNING:rgw_multi.tests:data of zone test-zone1 behind zone test-zone2: shard=3 master=00000000000000000000:00000000000000000796 target=00000000000000000000:00000000000000000654 2024-02-01T10:38:24.387 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:24.387 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:24.387 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:38:24.432 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:38:24.432 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:38:24.512 INFO:teuthology.orchestra.run.smithi148.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":8595394196901466828},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002486","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:22:17.904424Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000005164","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:38:09.883042Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004601","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:24:00.115466Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000654","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:35:47.376932Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-02-01T10:38:24Z"} 2024-02-01T10:38:24.513 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":8595394196901466828},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002486","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:22:17.904424Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000005164","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:38:09.883042Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004601","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:24:00.115466Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000654","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:35:47.376932Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-02-01T10:38:24Z"} 2024-02-01T10:38:24.513 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002486', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:22:17.904424Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005164', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:38:09.883042Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004601', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:24:00.115466Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000654', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:35:47.376932Z'}}] 2024-02-01T10:38:24.513 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002486', 1: '00000000000000000000:00000000000000005164', 2: '00000000000000000000:00000000000000004601', 3: '00000000000000000000:00000000000000000796'} 2024-02-01T10:38:24.513 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002486', 1: '00000000000000000000:00000000000000005164', 2: '00000000000000000000:00000000000000004601', 3: '00000000000000000000:00000000000000000654'} 2024-02-01T10:38:24.513 WARNING:rgw_multi.tests:data of zone test-zone1 behind zone test-zone2: shard=3 master=00000000000000000000:00000000000000000796 target=00000000000000000000:00000000000000000654 2024-02-01T10:38:29.515 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:29.515 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:29.515 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:38:29.561 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:38:29.562 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:38:29.640 INFO:teuthology.orchestra.run.smithi148.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":8595394196901466828},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002486","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:22:17.904424Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000005164","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:38:09.883042Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004601","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:24:00.115466Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000654","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:35:47.376932Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-02-01T10:38:29Z"} 2024-02-01T10:38:29.641 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":8595394196901466828},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002486","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:22:17.904424Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000005164","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:38:09.883042Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004601","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:24:00.115466Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000654","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:35:47.376932Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-02-01T10:38:29Z"} 2024-02-01T10:38:29.641 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002486', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:22:17.904424Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005164', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:38:09.883042Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004601', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:24:00.115466Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000654', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:35:47.376932Z'}}] 2024-02-01T10:38:29.641 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002486', 1: '00000000000000000000:00000000000000005164', 2: '00000000000000000000:00000000000000004601', 3: '00000000000000000000:00000000000000000796'} 2024-02-01T10:38:29.641 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002486', 1: '00000000000000000000:00000000000000005164', 2: '00000000000000000000:00000000000000004601', 3: '00000000000000000000:00000000000000000654'} 2024-02-01T10:38:29.641 WARNING:rgw_multi.tests:data of zone test-zone1 behind zone test-zone2: shard=3 master=00000000000000000000:00000000000000000796 target=00000000000000000000:00000000000000000654 2024-02-01T10:38:34.643 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:34.643 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:34.643 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:38:34.688 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:38:34.688 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:38:34.775 INFO:teuthology.orchestra.run.smithi148.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":8595394196901466828},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002486","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:22:17.904424Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000005164","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:38:09.883042Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004601","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:24:00.115466Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000796","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:38:14.839079Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-02-01T10:38:34Z"} 2024-02-01T10:38:34.775 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":8595394196901466828},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002486","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:22:17.904424Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000005164","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:38:09.883042Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004601","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:24:00.115466Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000796","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:38:14.839079Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-02-01T10:38:34Z"} 2024-02-01T10:38:34.775 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002486', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:22:17.904424Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005164', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:38:09.883042Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004601', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:24:00.115466Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000796', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:38:14.839079Z'}}] 2024-02-01T10:38:34.775 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002486', 1: '00000000000000000000:00000000000000005164', 2: '00000000000000000000:00000000000000004601', 3: '00000000000000000000:00000000000000000796'} 2024-02-01T10:38:34.775 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002486', 1: '00000000000000000000:00000000000000005164', 2: '00000000000000000000:00000000000000004601', 3: '00000000000000000000:00000000000000000796'} 2024-02-01T10:38:34.776 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2024-02-01T10:38:34.776 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone2 target=test-zone3 2024-02-01T10:38:34.776 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:34.776 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:34.776 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:38:34.814 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:38:34.815 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:38:34.905 INFO:teuthology.orchestra.run.smithi187.stdout:[{"marker":"00000000000000000000:00000000000000002486","last_update":"2024-02-01T10:22:17.904424Z"},{"marker":"00000000000000000000:00000000000000005304","last_update":"2024-02-01T10:38:29.902463Z"},{"marker":"00000000000000000000:00000000000000004601","last_update":"2024-02-01T10:24:00.115466Z"},{"marker":"00000000000000000000:00000000000000000796","last_update":"2024-02-01T10:38:14.839079Z"}] 2024-02-01T10:38:34.906 DEBUG:rgw_multi.tests:data markers for zone=test-zone2 markers={0: '00000000000000000000:00000000000000002486', 1: '00000000000000000000:00000000000000005304', 2: '00000000000000000000:00000000000000004601', 3: '00000000000000000000:00000000000000000796'} 2024-02-01T10:38:34.906 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2024-02-01T10:38:34.906 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:34.906 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:34.906 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:38:34.950 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:38:34.950 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:38:35.029 INFO:teuthology.orchestra.run.smithi148.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4461638765891857795},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002486","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:22:17.904424Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000005304","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:38:29.902463Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004601","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:24:00.115466Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000796","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:38:14.839079Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-02-01T10:38:35Z"} 2024-02-01T10:38:35.029 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4461638765891857795},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002486","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:22:17.904424Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000005304","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:38:29.902463Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004601","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:24:00.115466Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000796","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:38:14.839079Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-02-01T10:38:35Z"} 2024-02-01T10:38:35.029 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002486', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:22:17.904424Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005304', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:38:29.902463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004601', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:24:00.115466Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000796', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:38:14.839079Z'}}] 2024-02-01T10:38:35.030 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002486', 1: '00000000000000000000:00000000000000005304', 2: '00000000000000000000:00000000000000004601', 3: '00000000000000000000:00000000000000000796'} 2024-02-01T10:38:35.030 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002486', 1: '00000000000000000000:00000000000000005304', 2: '00000000000000000000:00000000000000004601', 3: '00000000000000000000:00000000000000000796'} 2024-02-01T10:38:35.030 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2024-02-01T10:38:35.030 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone3 target=test-zone1 2024-02-01T10:38:35.030 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:35.030 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:35.030 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:38:35.062 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:38:35.062 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:38:35.142 INFO:teuthology.orchestra.run.smithi148.stdout:[{"marker":"00000000000000000000:00000000000000002909","last_update":"2024-02-01T10:21:54.106139Z"},{"marker":"00000000000000000000:00000000000000004881","last_update":"2024-02-01T10:38:14.946782Z"},{"marker":"00000000000000000000:00000000000000005588","last_update":"2024-02-01T10:25:33.664319Z"},{"marker":"00000000000000000000:00000000000000000796","last_update":"2024-02-01T10:37:54.901571Z"}] 2024-02-01T10:38:35.143 DEBUG:rgw_multi.tests:data markers for zone=test-zone3 markers={0: '00000000000000000000:00000000000000002909', 1: '00000000000000000000:00000000000000004881', 2: '00000000000000000000:00000000000000005588', 3: '00000000000000000000:00000000000000000796'} 2024-02-01T10:38:35.143 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2024-02-01T10:38:35.143 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:35.143 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:35.143 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:38:35.231 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:38:35.231 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:38:35.309 INFO:teuthology.orchestra.run.smithi148.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":8579032341556541049},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002909","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:21:54.106139Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004881","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:38:14.946782Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000005588","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:25:33.664319Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000796","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:37:54.901571Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-02-01T10:38:35Z"} 2024-02-01T10:38:35.309 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":8579032341556541049},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002909","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:21:54.106139Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004881","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:38:14.946782Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000005588","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:25:33.664319Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000796","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:37:54.901571Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-02-01T10:38:35Z"} 2024-02-01T10:38:35.310 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002909', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:21:54.106139Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004881', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:38:14.946782Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005588', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:25:33.664319Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000796', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:37:54.901571Z'}}] 2024-02-01T10:38:35.310 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002909', 1: '00000000000000000000:00000000000000004881', 2: '00000000000000000000:00000000000000005588', 3: '00000000000000000000:00000000000000000796'} 2024-02-01T10:38:35.310 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002909', 1: '00000000000000000000:00000000000000004881', 2: '00000000000000000000:00000000000000005588', 3: '00000000000000000000:00000000000000000796'} 2024-02-01T10:38:35.310 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2024-02-01T10:38:35.310 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone3 target=test-zone2 2024-02-01T10:38:35.310 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:35.310 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:35.310 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:38:35.384 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:38:35.384 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:38:35.464 INFO:teuthology.orchestra.run.smithi148.stdout:[{"marker":"00000000000000000000:00000000000000002909","last_update":"2024-02-01T10:21:54.106139Z"},{"marker":"00000000000000000000:00000000000000004881","last_update":"2024-02-01T10:38:14.946782Z"},{"marker":"00000000000000000000:00000000000000005588","last_update":"2024-02-01T10:25:33.664319Z"},{"marker":"00000000000000000000:00000000000000000796","last_update":"2024-02-01T10:37:54.901571Z"}] 2024-02-01T10:38:35.465 DEBUG:rgw_multi.tests:data markers for zone=test-zone3 markers={0: '00000000000000000000:00000000000000002909', 1: '00000000000000000000:00000000000000004881', 2: '00000000000000000000:00000000000000005588', 3: '00000000000000000000:00000000000000000796'} 2024-02-01T10:38:35.465 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2024-02-01T10:38:35.465 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:35.465 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:35.465 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:38:35.510 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:38:35.510 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:38:35.591 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":843725689139632598},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002909","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:21:54.106139Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004881","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:38:14.946782Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000005588","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:25:33.664319Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000796","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:37:54.901571Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-02-01T10:38:35Z"} 2024-02-01T10:38:35.591 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":843725689139632598},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002909","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:21:54.106139Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004881","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:38:14.946782Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000005588","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:25:33.664319Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000796","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:37:54.901571Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-02-01T10:38:35Z"} 2024-02-01T10:38:35.591 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002909', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:21:54.106139Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004881', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:38:14.946782Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005588', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:25:33.664319Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000796', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:37:54.901571Z'}}] 2024-02-01T10:38:35.591 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002909', 1: '00000000000000000000:00000000000000004881', 2: '00000000000000000000:00000000000000005588', 3: '00000000000000000000:00000000000000000796'} 2024-02-01T10:38:35.591 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002909', 1: '00000000000000000000:00000000000000004881', 2: '00000000000000000000:00000000000000005588', 3: '00000000000000000000:00000000000000000796'} 2024-02-01T10:38:35.591 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2024-02-01T10:38:35.592 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:35.592 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:35.592 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:38:35.629 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:38:35.630 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:38:35.714 INFO:teuthology.orchestra.run.smithi148.stdout:[{"marker":"00000000000000000000:00000000000000002204","last_update":"2024-02-01T10:21:48.024326Z"},{"marker":"00000000000000000000:00000000000000004881","last_update":"2024-02-01T10:38:15.877435Z"},{"marker":"00000000000000000000:00000000000000005870","last_update":"2024-02-01T10:25:27.559832Z"},{"marker":"00000000000000000000:00000000000000000796","last_update":"2024-02-01T10:38:14.730286Z"}] 2024-02-01T10:38:35.715 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:38:35.715 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:38:35.715 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2024-02-01T10:38:35.791 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:38:35.791 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:38:35.886 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '0', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:35.887 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '0', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:35.887 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 0 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:38:35.963 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:38:35.963 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:38:36.035 INFO:teuthology.orchestra.run.smithi148.stdout:[] 2024-02-01T10:38:36.036 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '1', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:36.036 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '1', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:36.036 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 1 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:38:36.114 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:38:36.115 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:38:36.183 INFO:teuthology.orchestra.run.smithi148.stdout:[] 2024-02-01T10:38:36.183 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '2', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:36.183 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '2', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:36.184 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 2 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:38:36.261 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:38:36.262 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:38:36.329 INFO:teuthology.orchestra.run.smithi148.stdout:[] 2024-02-01T10:38:36.330 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '3', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:36.330 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '3', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:36.330 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 3 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:38:36.401 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:38:36.402 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:38:36.469 INFO:teuthology.orchestra.run.smithi148.stdout:[] 2024-02-01T10:38:36.469 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:36.469 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:36.469 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:38:36.508 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:38:36.508 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:38:36.592 INFO:teuthology.orchestra.run.smithi187.stdout:[{"marker":"00000000000000000000:00000000000000002486","last_update":"2024-02-01T10:22:17.904424Z"},{"marker":"00000000000000000000:00000000000000005304","last_update":"2024-02-01T10:38:29.902463Z"},{"marker":"00000000000000000000:00000000000000004601","last_update":"2024-02-01T10:24:00.115466Z"},{"marker":"00000000000000000000:00000000000000000796","last_update":"2024-02-01T10:38:14.839079Z"}] 2024-02-01T10:38:36.593 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:38:36.593 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:38:36.593 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog autotrim --cluster c2 --debug-rgw 0 --debug-ms 0 2024-02-01T10:38:36.662 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:38:36.663 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:38:36.756 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'list', '--shard-id', '0', '--max-entries', '1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:36.756 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'list', '--shard-id', '0', '--max-entries', '1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:36.756 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog list --shard-id 0 --max-entries 1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:38:36.831 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:38:36.833 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:38:36.908 INFO:teuthology.orchestra.run.smithi187.stdout:[] 2024-02-01T10:38:36.909 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'list', '--shard-id', '1', '--max-entries', '1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:36.909 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'list', '--shard-id', '1', '--max-entries', '1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:36.909 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog list --shard-id 1 --max-entries 1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:38:36.983 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:38:36.983 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:38:37.059 INFO:teuthology.orchestra.run.smithi187.stdout:[] 2024-02-01T10:38:37.060 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'list', '--shard-id', '2', '--max-entries', '1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:37.060 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'list', '--shard-id', '2', '--max-entries', '1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:37.060 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog list --shard-id 2 --max-entries 1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:38:37.138 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:38:37.138 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:38:37.271 INFO:teuthology.orchestra.run.smithi187.stdout:[] 2024-02-01T10:38:37.271 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'list', '--shard-id', '3', '--max-entries', '1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:37.271 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'list', '--shard-id', '3', '--max-entries', '1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:37.272 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog list --shard-id 3 --max-entries 1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:38:37.352 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:38:37.352 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:38:37.456 INFO:teuthology.orchestra.run.smithi187.stdout:[] 2024-02-01T10:38:37.457 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:37.457 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:37.457 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:38:37.494 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:38:37.494 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:38:37.577 INFO:teuthology.orchestra.run.smithi148.stdout:[{"marker":"00000000000000000000:00000000000000002204","last_update":"2024-02-01T10:21:48.024326Z"},{"marker":"00000000000000000000:00000000000000004881","last_update":"2024-02-01T10:38:15.877435Z"},{"marker":"00000000000000000000:00000000000000005870","last_update":"2024-02-01T10:25:27.559832Z"},{"marker":"00000000000000000000:00000000000000000796","last_update":"2024-02-01T10:38:14.730286Z"}] 2024-02-01T10:38:37.578 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:38:37.578 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:38:37.578 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2024-02-01T10:38:37.649 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:38:37.649 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:38:37.734 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '0', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:37.734 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '0', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:37.734 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 0 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:38:37.812 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:38:37.812 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:38:37.881 INFO:teuthology.orchestra.run.smithi148.stdout:[] 2024-02-01T10:38:37.881 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '1', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:37.881 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '1', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:37.881 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 1 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:38:37.958 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:38:37.959 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:38:38.027 INFO:teuthology.orchestra.run.smithi148.stdout:[] 2024-02-01T10:38:38.028 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '2', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:38.028 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '2', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:38.028 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 2 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:38:38.107 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:38:38.107 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:38:38.176 INFO:teuthology.orchestra.run.smithi148.stdout:[] 2024-02-01T10:38:38.176 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '3', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:38.176 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '3', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:38:38.177 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 3 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:38:38.250 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:38:38.250 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:38:38.319 INFO:teuthology.orchestra.run.smithi148.stdout:[] 2024-02-01T10:38:38.320 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_datalog_autotrim ... ok 2024-02-01T10:38:38.321 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:38:38.322 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:38:38.322 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --sync-from-all=false --cluster c2 --debug-rgw 0 --debug-ms 0 2024-02-01T10:38:38.364 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:38:38.364 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:38:38.414 INFO:teuthology.orchestra.run.smithi187.stdout:{"id":"01fb6d04-f0ef-49b2-9ebc-c1b530925485","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"1C56UIO2VKZWOSJZ","secret_key":"4X9hEalkm5sn0VTq71G3qlDBF1FRjy9C"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"c7bb4449-4a82-4742-a9c6-c7422f73e18f","notif_pool":"test-zone2.rgw.log:notif"} 2024-02-01T10:38:38.414 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:38:38.414 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:38:38.415 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2024-02-01T10:38:38.490 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:38:38.490 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:38:38.555 INFO:teuthology.orchestra.run.smithi187.stderr:Sending period to new master zone ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01 2024-02-01T10:38:38.628 INFO:teuthology.orchestra.run.smithi187.stdout:{"id":"f2e6c290-e7aa-4279-b77e-15403b18d61f","epoch":2,"predecessor_uuid":"976daaff-eac6-4c84-bd63-38e72aa959ff","sync_status":["","","1_1706783813.899233_136.1",""],"period_map":{"id":"f2e6c290-e7aa-4279-b77e-15403b18d61f","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":true,"endpoints":["http://smithi148.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01","zones":[{"id":"01fb6d04-f0ef-49b2-9ebc-c1b530925485","name":"test-zone2","endpoints":["http://smithi187.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":false,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]},{"id":"2ea581c3-3f48-49d7-a989-68024c5b8eac","name":"test-zone3","endpoints":["http://smithi148.front.sepia.ceph.com:8001"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]},{"id":"ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01","name":"test-zone1","endpoints":["http://smithi148.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"c7bb4449-4a82-4742-a9c6-c7422f73e18f","sync_policy":{"groups":[]},"enabled_features":["resharding"]}],"short_zone_ids":[{"key":"01fb6d04-f0ef-49b2-9ebc-c1b530925485","val":1553472663},{"key":"2ea581c3-3f48-49d7-a989-68024c5b8eac","val":1173357353},{"key":"ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01","val":3362809119}]},"master_zonegroup":"test-zonegroup","master_zone":"ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"bucket_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"anonymous_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false}},"realm_id":"c7bb4449-4a82-4742-a9c6-c7422f73e18f","realm_epoch":4} 2024-02-01T10:38:38.629 INFO:rgw_multi.tests:Set sync_from_all flag on zone test-zone2 to false 2024-02-01T10:39:38.629 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=vclfcn-49 2024-02-01T10:39:38.630 DEBUG:boto:path=/vclfcn-49/ 2024-02-01T10:39:38.630 DEBUG:boto:auth_path=/vclfcn-49/ 2024-02-01T10:39:38.631 DEBUG:boto:Method: PUT 2024-02-01T10:39:38.631 DEBUG:boto:Path: /vclfcn-49/ 2024-02-01T10:39:38.631 DEBUG:boto:Data: 2024-02-01T10:39:38.631 DEBUG:boto:Headers: {} 2024-02-01T10:39:38.631 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:39:38.631 DEBUG:boto:Port: 8000 2024-02-01T10:39:38.631 DEBUG:boto:Params: {} 2024-02-01T10:39:38.632 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T10:39:38.632 DEBUG:boto:Token: None 2024-02-01T10:39:38.633 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:39:38 GMT /vclfcn-49/ 2024-02-01T10:39:38.633 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:bU5V5RTI8S0AsHH6bqkfEUUAK2g= 2024-02-01T10:39:38.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:39:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:bU5V5RTI8S0AsHH6bqkfEUUAK2g=', 'Content-Length': '0'} 2024-02-01T10:39:38.678 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b81ca28a5f76a593-0065bb74ea-5625-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:39:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:39:38.679 DEBUG:boto:path=/vclfcn-49/testredirect 2024-02-01T10:39:38.679 DEBUG:boto:auth_path=/vclfcn-49/testredirect 2024-02-01T10:39:38.679 DEBUG:boto:Method: PUT 2024-02-01T10:39:38.679 DEBUG:boto:Path: /vclfcn-49/testredirect 2024-02-01T10:39:38.680 DEBUG:boto:Data: 2024-02-01T10:39:38.680 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2024-02-01T10:39:38.680 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:39:38.680 DEBUG:boto:Port: 8000 2024-02-01T10:39:38.680 DEBUG:boto:Params: {} 2024-02-01T10:39:38.680 DEBUG:boto:Token: None 2024-02-01T10:39:38.680 DEBUG:boto:StringToSign: PUT 3FCGuEcom6i4veFJuDiBdQ== application/octet-stream Thu, 01 Feb 2024 10:39:38 GMT /vclfcn-49/testredirect 2024-02-01T10:39:38.680 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:uTBHkR0AQ4NjCNvcyiF0zQcYA0Y= 2024-02-01T10:39:38.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:39:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:uTBHkR0AQ4NjCNvcyiF0zQcYA0Y='} 2024-02-01T10:39:38.698 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000eebe062fc01bdf40-0065bb74ea-5625-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:39:38 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:39:38.698 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:39:38.698 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:39:38.699 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:39:38.741 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:39:38.741 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:39:38.799 INFO:teuthology.orchestra.run.smithi148.stderr:No --period given, using current period=f2e6c290-e7aa-4279-b77e-15403b18d61f 2024-02-01T10:39:38.822 INFO:teuthology.orchestra.run.smithi148.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1706783978.681171_139.1","last_update":"2024-02-01T10:39:38.681171Z"},{"marker":"","last_update":"0.000000"}] 2024-02-01T10:39:38.823 DEBUG:rgw_multi.tests:master meta markers={0: '', 1: '', 2: '1_1706783978.681171_139.1', 3: ''} 2024-02-01T10:39:38.823 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-02-01T10:39:38.823 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:39:38.823 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:39:38.823 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:39:38.865 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:39:38.865 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:39:38.921 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:39:38.919+0000 7f14cf12c980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:39:38.921 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:39:38.923+0000 7f14cf12c980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:39:38.944 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783887.801303_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:38:07.801303Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:39:38Z"} 2024-02-01T10:39:38.945 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783887.801303_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:38:07.801303Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:39:38Z"} 2024-02-01T10:39:38.945 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706783887.801303_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:38:07.801303Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:39:38.945 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1706783978.681171_139.1', 3: ''} 2024-02-01T10:39:38.945 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1706783887.801303_121.1', 3: ''} 2024-02-01T10:39:38.945 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1706783978.681171_139.1 target=1_1706783887.801303_121.1 2024-02-01T10:39:43.946 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:39:43.946 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:39:43.946 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:39:43.990 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:39:43.990 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:39:44.045 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:39:44.043+0000 7f5aa4f16980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:39:44.045 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:39:44.047+0000 7f5aa4f16980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:39:44.069 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783887.801303_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:38:07.801303Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:39:44Z"} 2024-02-01T10:39:44.069 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783887.801303_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:38:07.801303Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:39:44Z"} 2024-02-01T10:39:44.069 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706783887.801303_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:38:07.801303Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:39:44.070 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1706783978.681171_139.1', 3: ''} 2024-02-01T10:39:44.070 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1706783887.801303_121.1', 3: ''} 2024-02-01T10:39:44.070 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1706783978.681171_139.1 target=1_1706783887.801303_121.1 2024-02-01T10:39:49.071 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:39:49.071 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:39:49.072 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:39:49.114 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:39:49.114 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:39:49.170 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:39:49.170+0000 7f35acd60980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:39:49.172 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:39:49.170+0000 7f35acd60980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:39:49.195 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783887.801303_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:38:07.801303Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:39:49Z"} 2024-02-01T10:39:49.196 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783887.801303_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:38:07.801303Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:39:49Z"} 2024-02-01T10:39:49.196 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706783887.801303_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:38:07.801303Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:39:49.196 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1706783978.681171_139.1', 3: ''} 2024-02-01T10:39:49.196 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1706783887.801303_121.1', 3: ''} 2024-02-01T10:39:49.196 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1706783978.681171_139.1 target=1_1706783887.801303_121.1 2024-02-01T10:39:54.198 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:39:54.198 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:39:54.198 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:39:54.242 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:39:54.242 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:39:54.296 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:39:54.294+0000 7f1c3e34a980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:39:54.297 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:39:54.298+0000 7f1c3e34a980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:39:54.321 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783887.801303_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:38:07.801303Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:39:54Z"} 2024-02-01T10:39:54.321 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783887.801303_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:38:07.801303Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:39:54Z"} 2024-02-01T10:39:54.321 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706783887.801303_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:38:07.801303Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:39:54.321 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1706783978.681171_139.1', 3: ''} 2024-02-01T10:39:54.322 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1706783887.801303_121.1', 3: ''} 2024-02-01T10:39:54.322 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1706783978.681171_139.1 target=1_1706783887.801303_121.1 2024-02-01T10:39:59.322 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:39:59.323 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:39:59.323 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:39:59.365 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:39:59.366 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:39:59.420 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:39:59.418+0000 7fdd55e94980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:39:59.421 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:39:59.418+0000 7fdd55e94980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:39:59.445 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783887.801303_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:38:07.801303Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:39:59Z"} 2024-02-01T10:39:59.445 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783887.801303_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:38:07.801303Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:39:59Z"} 2024-02-01T10:39:59.445 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706783887.801303_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:38:07.801303Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:39:59.445 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1706783978.681171_139.1', 3: ''} 2024-02-01T10:39:59.445 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1706783887.801303_121.1', 3: ''} 2024-02-01T10:39:59.445 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1706783978.681171_139.1 target=1_1706783887.801303_121.1 2024-02-01T10:40:04.447 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:40:04.447 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:40:04.447 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:40:04.490 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:40:04.491 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:40:04.546 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:40:04.546+0000 7f282c679980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:40:04.547 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:40:04.546+0000 7f282c679980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:40:04.570 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783887.801303_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:38:07.801303Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:40:04Z"} 2024-02-01T10:40:04.571 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783887.801303_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:38:07.801303Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:40:04Z"} 2024-02-01T10:40:04.571 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706783887.801303_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:38:07.801303Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:40:04.571 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1706783978.681171_139.1', 3: ''} 2024-02-01T10:40:04.571 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1706783887.801303_121.1', 3: ''} 2024-02-01T10:40:04.571 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1706783978.681171_139.1 target=1_1706783887.801303_121.1 2024-02-01T10:40:09.572 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:40:09.572 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:40:09.573 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:40:09.617 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:40:09.617 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:40:09.671 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:40:09.670+0000 7fab5e602980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:40:09.672 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:40:09.670+0000 7fab5e602980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:40:09.695 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783887.801303_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:38:07.801303Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:40:09Z"} 2024-02-01T10:40:09.696 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783887.801303_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:38:07.801303Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:40:09Z"} 2024-02-01T10:40:09.696 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706783887.801303_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:38:07.801303Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:40:09.696 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1706783978.681171_139.1', 3: ''} 2024-02-01T10:40:09.696 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1706783887.801303_121.1', 3: ''} 2024-02-01T10:40:09.696 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1706783978.681171_139.1 target=1_1706783887.801303_121.1 2024-02-01T10:40:09.943 INFO:tasks.rgw.c1.client.0.smithi148.stdout:2024-02-01T10:40:09.949+0000 7f5f4234d640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-02-01T10:40:09.943 INFO:tasks.rgw.c1.client.1.smithi148.stdout:2024-02-01T10:40:09.949+0000 7f8357755640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-02-01T10:40:14.697 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:40:14.697 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:40:14.698 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:40:14.740 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:40:14.741 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:40:14.794 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:40:14.794+0000 7fb2c89d2980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:40:14.795 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:40:14.794+0000 7fb2c89d2980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:40:14.819 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783887.801303_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:38:07.801303Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:40:14Z"} 2024-02-01T10:40:14.819 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783887.801303_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:38:07.801303Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:40:14Z"} 2024-02-01T10:40:14.819 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706783887.801303_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:38:07.801303Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:40:14.820 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1706783978.681171_139.1', 3: ''} 2024-02-01T10:40:14.820 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1706783887.801303_121.1', 3: ''} 2024-02-01T10:40:14.820 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1706783978.681171_139.1 target=1_1706783887.801303_121.1 2024-02-01T10:40:19.821 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:40:19.822 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:40:19.822 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:40:19.866 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:40:19.866 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:40:19.920 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:40:19.918+0000 7fcb9664c980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:40:19.921 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:40:19.922+0000 7fcb9664c980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:40:19.945 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783887.801303_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:38:07.801303Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:40:19Z"} 2024-02-01T10:40:19.945 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783887.801303_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:38:07.801303Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:40:19Z"} 2024-02-01T10:40:19.945 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706783887.801303_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:38:07.801303Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:40:19.945 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1706783978.681171_139.1', 3: ''} 2024-02-01T10:40:19.945 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1706783887.801303_121.1', 3: ''} 2024-02-01T10:40:19.945 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1706783978.681171_139.1 target=1_1706783887.801303_121.1 2024-02-01T10:40:24.946 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:40:24.947 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:40:24.947 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:40:24.993 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:40:24.994 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:40:25.049 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:40:25.046+0000 7f5241b9b980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:40:25.050 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:40:25.050+0000 7f5241b9b980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:40:25.073 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783887.801303_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:38:07.801303Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:40:25Z"} 2024-02-01T10:40:25.073 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783887.801303_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:38:07.801303Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:40:25Z"} 2024-02-01T10:40:25.073 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706783887.801303_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:38:07.801303Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:40:25.074 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1706783978.681171_139.1', 3: ''} 2024-02-01T10:40:25.074 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1706783887.801303_121.1', 3: ''} 2024-02-01T10:40:25.074 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1706783978.681171_139.1 target=1_1706783887.801303_121.1 2024-02-01T10:40:30.074 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:40:30.075 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:40:30.075 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:40:30.117 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:40:30.117 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:40:30.172 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:40:30.170+0000 7fc7ff6b8980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:40:30.173 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:40:30.170+0000 7fc7ff6b8980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:40:30.196 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783887.801303_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:38:07.801303Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:40:30Z"} 2024-02-01T10:40:30.196 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783887.801303_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:38:07.801303Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:40:30Z"} 2024-02-01T10:40:30.197 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706783887.801303_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:38:07.801303Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:40:30.197 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1706783978.681171_139.1', 3: ''} 2024-02-01T10:40:30.197 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1706783887.801303_121.1', 3: ''} 2024-02-01T10:40:30.197 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1706783978.681171_139.1 target=1_1706783887.801303_121.1 2024-02-01T10:40:35.198 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:40:35.199 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:40:35.199 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:40:35.241 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:40:35.242 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:40:35.299 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:40:35.298+0000 7f4d66058980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:40:35.300 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:40:35.298+0000 7f4d66058980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:40:35.323 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783887.801303_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:38:07.801303Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:40:35Z"} 2024-02-01T10:40:35.324 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783887.801303_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:38:07.801303Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:40:35Z"} 2024-02-01T10:40:35.324 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706783887.801303_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:38:07.801303Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:40:35.324 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1706783978.681171_139.1', 3: ''} 2024-02-01T10:40:35.324 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1706783887.801303_121.1', 3: ''} 2024-02-01T10:40:35.324 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1706783978.681171_139.1 target=1_1706783887.801303_121.1 2024-02-01T10:40:39.949 INFO:tasks.rgw.c1.client.0.smithi148.stdout:2024-02-01T10:40:39.953+0000 7f5f40b4a640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-02-01T10:40:40.326 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:40:40.326 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:40:40.326 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:40:40.370 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:40:40.370 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:40:40.424 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:40:40.421+0000 7f06a788d980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:40:40.425 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:40:40.421+0000 7f06a788d980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:40:40.450 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783978.681171_139.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:39:38.681171Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:40:40Z"} 2024-02-01T10:40:40.450 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783978.681171_139.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:39:38.681171Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:40:40Z"} 2024-02-01T10:40:40.450 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706783978.681171_139.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:39:38.681171Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:40:40.450 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1706783978.681171_139.1', 3: ''} 2024-02-01T10:40:40.450 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1706783978.681171_139.1', 3: ''} 2024-02-01T10:40:40.451 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-02-01T10:40:40.451 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-02-01T10:40:40.451 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:40:40.451 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:40:40.451 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:40:40.494 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:40:40.494 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:40:40.548 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:40:40.553+0000 7f56cedd5980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:40:40.549 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:40:40.553+0000 7f56cedd5980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:40:40.572 INFO:teuthology.orchestra.run.smithi148.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783978.681171_139.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:39:38.681171Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:40:40Z"} 2024-02-01T10:40:40.572 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783978.681171_139.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:39:38.681171Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:40:40Z"} 2024-02-01T10:40:40.572 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706783978.681171_139.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:39:38.681171Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:40:40.573 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1706783978.681171_139.1', 3: ''} 2024-02-01T10:40:40.573 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1706783978.681171_139.1', 3: ''} 2024-02-01T10:40:40.573 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-02-01T10:40:40.573 DEBUG:boto:path=/vclfcn-49/ 2024-02-01T10:40:40.573 DEBUG:boto:auth_path=/vclfcn-49/ 2024-02-01T10:40:40.574 DEBUG:boto:Method: HEAD 2024-02-01T10:40:40.574 DEBUG:boto:Path: /vclfcn-49/ 2024-02-01T10:40:40.574 DEBUG:boto:Data: 2024-02-01T10:40:40.574 DEBUG:boto:Headers: {} 2024-02-01T10:40:40.574 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:40:40.574 DEBUG:boto:Port: 8000 2024-02-01T10:40:40.574 DEBUG:boto:Params: {} 2024-02-01T10:40:40.574 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T10:40:40.575 DEBUG:boto:Token: None 2024-02-01T10:40:40.575 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:40:40 GMT /vclfcn-49/ 2024-02-01T10:40:40.575 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5GvejGXzNJbC4NxueJJWZxjD+h8= 2024-02-01T10:40:40.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:40:40 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5GvejGXzNJbC4NxueJJWZxjD+h8=', 'Content-Length': '0'} 2024-02-01T10:40:40.584 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d388921908406fb1-0065bb7528-5168-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:40:40 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:40:40.584 DEBUG:boto:path=/vclfcn-49/testredirect 2024-02-01T10:40:40.585 DEBUG:boto:auth_path=/vclfcn-49/testredirect 2024-02-01T10:40:40.585 DEBUG:boto:Method: HEAD 2024-02-01T10:40:40.585 DEBUG:boto:Path: /vclfcn-49/testredirect 2024-02-01T10:40:40.585 DEBUG:boto:Data: 2024-02-01T10:40:40.585 DEBUG:boto:Headers: {} 2024-02-01T10:40:40.585 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:40:40.585 DEBUG:boto:Port: 8000 2024-02-01T10:40:40.585 DEBUG:boto:Params: {} 2024-02-01T10:40:40.585 DEBUG:boto:Token: None 2024-02-01T10:40:40.585 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:40:40 GMT /vclfcn-49/testredirect 2024-02-01T10:40:40.586 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:UaP9LFE5Pi1LwM3+Gxv3NKZDe80= 2024-02-01T10:40:40.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:40:40 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:UaP9LFE5Pi1LwM3+Gxv3NKZDe80=', 'Content-Length': '0'} 2024-02-01T10:40:40.588 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000009862e92a29ea8da3-0065bb7528-5168-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:40:40 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:40:40.588 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:40:40.588 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:40:40.589 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --redirect-zone=ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-02-01T10:40:40.629 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:40:40.629 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:40:40.682 INFO:teuthology.orchestra.run.smithi187.stdout:{"id":"01fb6d04-f0ef-49b2-9ebc-c1b530925485","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"1C56UIO2VKZWOSJZ","secret_key":"4X9hEalkm5sn0VTq71G3qlDBF1FRjy9C"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"c7bb4449-4a82-4742-a9c6-c7422f73e18f","notif_pool":"test-zone2.rgw.log:notif"} 2024-02-01T10:40:40.682 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:40:40.682 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:40:40.683 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2024-02-01T10:40:40.767 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:40:40.767 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:40:40.832 INFO:teuthology.orchestra.run.smithi187.stderr:Sending period to new master zone ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01 2024-02-01T10:40:40.905 INFO:teuthology.orchestra.run.smithi187.stdout:{"id":"f2e6c290-e7aa-4279-b77e-15403b18d61f","epoch":3,"predecessor_uuid":"976daaff-eac6-4c84-bd63-38e72aa959ff","sync_status":["","","1_1706783813.899233_136.1",""],"period_map":{"id":"f2e6c290-e7aa-4279-b77e-15403b18d61f","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":true,"endpoints":["http://smithi148.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01","zones":[{"id":"01fb6d04-f0ef-49b2-9ebc-c1b530925485","name":"test-zone2","endpoints":["http://smithi187.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":false,"sync_from":[],"redirect_zone":"ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01","supported_features":["compress-encrypted","resharding"]},{"id":"2ea581c3-3f48-49d7-a989-68024c5b8eac","name":"test-zone3","endpoints":["http://smithi148.front.sepia.ceph.com:8001"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]},{"id":"ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01","name":"test-zone1","endpoints":["http://smithi148.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"c7bb4449-4a82-4742-a9c6-c7422f73e18f","sync_policy":{"groups":[]},"enabled_features":["resharding"]}],"short_zone_ids":[{"key":"01fb6d04-f0ef-49b2-9ebc-c1b530925485","val":1553472663},{"key":"2ea581c3-3f48-49d7-a989-68024c5b8eac","val":1173357353},{"key":"ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01","val":3362809119}]},"master_zonegroup":"test-zonegroup","master_zone":"ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"bucket_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"anonymous_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false}},"realm_id":"c7bb4449-4a82-4742-a9c6-c7422f73e18f","realm_epoch":4} 2024-02-01T10:40:40.906 INFO:rgw_multi.tests:Set redirect_zone zone test-zone2 to "ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01" 2024-02-01T10:41:40.907 DEBUG:boto:path=/vclfcn-49/testredirect 2024-02-01T10:41:40.907 DEBUG:boto:auth_path=/vclfcn-49/testredirect 2024-02-01T10:41:40.908 DEBUG:boto:Method: HEAD 2024-02-01T10:41:40.908 DEBUG:boto:Path: /vclfcn-49/testredirect 2024-02-01T10:41:40.908 DEBUG:boto:Data: 2024-02-01T10:41:40.908 DEBUG:boto:Headers: {} 2024-02-01T10:41:40.908 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:41:40.908 DEBUG:boto:Port: 8000 2024-02-01T10:41:40.908 DEBUG:boto:Params: {} 2024-02-01T10:41:40.909 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T10:41:40.909 DEBUG:boto:Token: None 2024-02-01T10:41:40.909 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:41:40 GMT /vclfcn-49/testredirect 2024-02-01T10:41:40.909 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Qy7TY2x+WNLC4LlRmXaCS8cHI6Y= 2024-02-01T10:41:40.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:41:40 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Qy7TY2x+WNLC4LlRmXaCS8cHI6Y=', 'Content-Length': '0'} 2024-02-01T10:41:40.918 DEBUG:boto:Response headers: [('Location', 'http://smithi148.front.sepia.ceph.com:8000/vclfcn-49/testredirect?'), ('x-amz-request-id', 'tx000009335ba190bc4f8c4-0065bb7564-5276-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:41:40 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:41:40.919 DEBUG:boto:Redirecting: http://smithi148.front.sepia.ceph.com/vclfcn-49/testredirect 2024-02-01T10:41:40.919 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': '8000'} 2024-02-01T10:41:40.919 DEBUG:boto:Token: None 2024-02-01T10:41:40.919 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:41:40 GMT /vclfcn-49/testredirect 2024-02-01T10:41:40.919 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Qy7TY2x+WNLC4LlRmXaCS8cHI6Y= 2024-02-01T10:41:40.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:41:40 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Qy7TY2x+WNLC4LlRmXaCS8cHI6Y=', 'Content-Length': '0'} 2024-02-01T10:41:40.928 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:39:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx00000255a99c8a8e2b711-0065bb7564-5645-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:41:40 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:41:40.929 DEBUG:boto:path=/vclfcn-49/testredirect 2024-02-01T10:41:40.929 DEBUG:boto:auth_path=/vclfcn-49/testredirect 2024-02-01T10:41:40.929 DEBUG:boto:Method: GET 2024-02-01T10:41:40.930 DEBUG:boto:Path: /vclfcn-49/testredirect 2024-02-01T10:41:40.930 DEBUG:boto:Data: 2024-02-01T10:41:40.930 DEBUG:boto:Headers: {} 2024-02-01T10:41:40.930 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:41:40.930 DEBUG:boto:Port: 8000 2024-02-01T10:41:40.930 DEBUG:boto:Params: {} 2024-02-01T10:41:40.930 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T10:41:40.931 DEBUG:boto:Token: None 2024-02-01T10:41:40.931 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:41:40 GMT /vclfcn-49/testredirect 2024-02-01T10:41:40.931 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FxDV3Zytsgt5UtY/JxDCCguiNR8= 2024-02-01T10:41:40.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:41:40 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FxDV3Zytsgt5UtY/JxDCCguiNR8=', 'Content-Length': '0'} 2024-02-01T10:41:40.938 DEBUG:boto:Response headers: [('Location', 'http://smithi148.front.sepia.ceph.com:8000/vclfcn-49/testredirect?'), ('x-amz-request-id', 'tx000000a65da8f2048a6a6-0065bb7564-5276-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:41:40 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:41:40.939 DEBUG:boto:Redirecting: http://smithi148.front.sepia.ceph.com/vclfcn-49/testredirect 2024-02-01T10:41:40.939 DEBUG:boto:Token: None 2024-02-01T10:41:40.939 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:41:40 GMT /vclfcn-49/testredirect 2024-02-01T10:41:40.939 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FxDV3Zytsgt5UtY/JxDCCguiNR8= 2024-02-01T10:41:40.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:41:40 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FxDV3Zytsgt5UtY/JxDCCguiNR8=', 'Content-Length': '0'} 2024-02-01T10:41:40.942 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:39:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx00000f3a5135ae6a2f55b-0065bb7564-5645-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:41:40 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:41:40.943 DEBUG:boto:path=/vclfcn-49/testredirect 2024-02-01T10:41:40.944 DEBUG:boto:auth_path=/vclfcn-49/testredirect 2024-02-01T10:41:40.944 DEBUG:boto:Method: PUT 2024-02-01T10:41:40.944 DEBUG:boto:Path: /vclfcn-49/testredirect 2024-02-01T10:41:40.944 DEBUG:boto:Data: 2024-02-01T10:41:40.944 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'VpBzljOcorCZvRIkX5Nt3A==', 'Content-Length': '512', 'Expect': '100-Continue'} 2024-02-01T10:41:40.944 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:41:40.944 DEBUG:boto:Port: 8000 2024-02-01T10:41:40.945 DEBUG:boto:Params: {} 2024-02-01T10:41:40.945 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T10:41:40.945 DEBUG:boto:Token: None 2024-02-01T10:41:40.945 DEBUG:boto:StringToSign: PUT VpBzljOcorCZvRIkX5Nt3A== application/octet-stream Thu, 01 Feb 2024 10:41:40 GMT /vclfcn-49/testredirect 2024-02-01T10:41:40.947 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:U4qZGLrX2OlME4EDQBalnDhM6Bo= 2024-02-01T10:41:40.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'VpBzljOcorCZvRIkX5Nt3A==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:41:40 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:U4qZGLrX2OlME4EDQBalnDhM6Bo='} 2024-02-01T10:41:40.972 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"56907396339ca2b099bd12245f936ddc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000074053a88b678c5da-0065bb7564-5645-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:41:40 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:41:40.972 DEBUG:boto:path=/vclfcn-49/testredirect 2024-02-01T10:41:40.973 DEBUG:boto:auth_path=/vclfcn-49/testredirect 2024-02-01T10:41:40.973 DEBUG:boto:Method: GET 2024-02-01T10:41:40.973 DEBUG:boto:Path: /vclfcn-49/testredirect 2024-02-01T10:41:40.973 DEBUG:boto:Data: 2024-02-01T10:41:40.973 DEBUG:boto:Headers: {} 2024-02-01T10:41:40.973 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:41:40.973 DEBUG:boto:Port: 8000 2024-02-01T10:41:40.974 DEBUG:boto:Params: {} 2024-02-01T10:41:40.974 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T10:41:40.974 DEBUG:boto:Token: None 2024-02-01T10:41:40.974 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:41:40 GMT /vclfcn-49/testredirect 2024-02-01T10:41:40.974 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FxDV3Zytsgt5UtY/JxDCCguiNR8= 2024-02-01T10:41:40.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:41:40 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FxDV3Zytsgt5UtY/JxDCCguiNR8=', 'Content-Length': '0'} 2024-02-01T10:41:40.979 DEBUG:boto:Response headers: [('Location', 'http://smithi148.front.sepia.ceph.com:8000/vclfcn-49/testredirect?'), ('x-amz-request-id', 'tx000005bb696c571f00a42-0065bb7564-5276-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:41:40 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:41:40.979 DEBUG:boto:Redirecting: http://smithi148.front.sepia.ceph.com/vclfcn-49/testredirect 2024-02-01T10:41:40.979 DEBUG:boto:Token: None 2024-02-01T10:41:40.979 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:41:40 GMT /vclfcn-49/testredirect 2024-02-01T10:41:40.979 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FxDV3Zytsgt5UtY/JxDCCguiNR8= 2024-02-01T10:41:40.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:41:40 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FxDV3Zytsgt5UtY/JxDCCguiNR8=', 'Content-Length': '0'} 2024-02-01T10:41:40.983 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:41:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"56907396339ca2b099bd12245f936ddc"'), ('x-amz-request-id', 'tx00000adb5f219517dc73e-0065bb7564-5645-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:41:40 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:41:40.983 DEBUG:boto:path=/vclfcn-49/testredirect 2024-02-01T10:41:40.983 DEBUG:boto:auth_path=/vclfcn-49/testredirect 2024-02-01T10:41:40.983 DEBUG:boto:Method: PUT 2024-02-01T10:41:40.984 DEBUG:boto:Path: /vclfcn-49/testredirect 2024-02-01T10:41:40.984 DEBUG:boto:Data: 2024-02-01T10:41:40.984 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'uk9S5NXZfBvPq4jGr+LM5g==', 'Content-Length': '512', 'Expect': '100-Continue'} 2024-02-01T10:41:40.984 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:41:40.984 DEBUG:boto:Port: 8000 2024-02-01T10:41:40.984 DEBUG:boto:Params: {} 2024-02-01T10:41:40.984 DEBUG:boto:Token: None 2024-02-01T10:41:40.984 DEBUG:boto:StringToSign: PUT uk9S5NXZfBvPq4jGr+LM5g== application/octet-stream Thu, 01 Feb 2024 10:41:40 GMT /vclfcn-49/testredirect 2024-02-01T10:41:40.984 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:2CkLhYMU/nyPV1OmZa4cSfXcaGo= 2024-02-01T10:41:40.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'uk9S5NXZfBvPq4jGr+LM5g==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:41:40 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2CkLhYMU/nyPV1OmZa4cSfXcaGo='} 2024-02-01T10:41:40.996 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"ba4f52e4d5d97c1bcfab88c6afe2cce6"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000818a0a5d1536e64c-0065bb7564-5645-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:41:40 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:41:40.996 DEBUG:boto:path=/vclfcn-49/testredirect 2024-02-01T10:41:40.996 DEBUG:boto:auth_path=/vclfcn-49/testredirect 2024-02-01T10:41:40.996 DEBUG:boto:Method: GET 2024-02-01T10:41:40.997 DEBUG:boto:Path: /vclfcn-49/testredirect 2024-02-01T10:41:40.997 DEBUG:boto:Data: 2024-02-01T10:41:40.997 DEBUG:boto:Headers: {} 2024-02-01T10:41:40.997 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:41:40.997 DEBUG:boto:Port: 8000 2024-02-01T10:41:40.997 DEBUG:boto:Params: {} 2024-02-01T10:41:40.997 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T10:41:40.997 DEBUG:boto:Token: None 2024-02-01T10:41:40.997 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:41:40 GMT /vclfcn-49/testredirect 2024-02-01T10:41:40.998 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FxDV3Zytsgt5UtY/JxDCCguiNR8= 2024-02-01T10:41:40.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:41:40 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FxDV3Zytsgt5UtY/JxDCCguiNR8=', 'Content-Length': '0'} 2024-02-01T10:41:41.002 DEBUG:boto:Response headers: [('Location', 'http://smithi148.front.sepia.ceph.com:8000/vclfcn-49/testredirect?'), ('x-amz-request-id', 'tx00000455737742c97b140-0065bb7564-5276-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:41:41 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:41:41.002 DEBUG:boto:Redirecting: http://smithi148.front.sepia.ceph.com/vclfcn-49/testredirect 2024-02-01T10:41:41.003 DEBUG:boto:Token: None 2024-02-01T10:41:41.003 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:41:40 GMT /vclfcn-49/testredirect 2024-02-01T10:41:41.003 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FxDV3Zytsgt5UtY/JxDCCguiNR8= 2024-02-01T10:41:41.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:41:40 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FxDV3Zytsgt5UtY/JxDCCguiNR8=', 'Content-Length': '0'} 2024-02-01T10:41:41.006 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:41:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"ba4f52e4d5d97c1bcfab88c6afe2cce6"'), ('x-amz-request-id', 'tx000002b791e298b72a462-0065bb7565-5645-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:41:41 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:41:41.006 DEBUG:boto:path=/vclfcn-49/testredirect 2024-02-01T10:41:41.006 DEBUG:boto:auth_path=/vclfcn-49/testredirect 2024-02-01T10:41:41.007 DEBUG:boto:Method: PUT 2024-02-01T10:41:41.007 DEBUG:boto:Path: /vclfcn-49/testredirect 2024-02-01T10:41:41.007 DEBUG:boto:Data: 2024-02-01T10:41:41.007 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '6aKmvaJMgaK7Z2rfDZvVeg==', 'Content-Length': '512', 'Expect': '100-Continue'} 2024-02-01T10:41:41.007 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:41:41.007 DEBUG:boto:Port: 8000 2024-02-01T10:41:41.007 DEBUG:boto:Params: {} 2024-02-01T10:41:41.007 DEBUG:boto:Token: None 2024-02-01T10:41:41.007 DEBUG:boto:StringToSign: PUT 6aKmvaJMgaK7Z2rfDZvVeg== application/octet-stream Thu, 01 Feb 2024 10:41:41 GMT /vclfcn-49/testredirect 2024-02-01T10:41:41.008 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+MkubP8qyupHcpoI6i5ZU6LnM3M= 2024-02-01T10:41:41.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '6aKmvaJMgaK7Z2rfDZvVeg==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:41:41 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+MkubP8qyupHcpoI6i5ZU6LnM3M='} 2024-02-01T10:41:41.020 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"e9a2a6bda24c81a2bb676adf0d9bd57a"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000974ad9fd0931e0dc-0065bb7565-5645-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:41:41 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:41:41.020 DEBUG:boto:path=/vclfcn-49/testredirect 2024-02-01T10:41:41.020 DEBUG:boto:auth_path=/vclfcn-49/testredirect 2024-02-01T10:41:41.020 DEBUG:boto:Method: GET 2024-02-01T10:41:41.020 DEBUG:boto:Path: /vclfcn-49/testredirect 2024-02-01T10:41:41.021 DEBUG:boto:Data: 2024-02-01T10:41:41.021 DEBUG:boto:Headers: {} 2024-02-01T10:41:41.021 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:41:41.021 DEBUG:boto:Port: 8000 2024-02-01T10:41:41.021 DEBUG:boto:Params: {} 2024-02-01T10:41:41.021 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T10:41:41.021 DEBUG:boto:Token: None 2024-02-01T10:41:41.022 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:41:41 GMT /vclfcn-49/testredirect 2024-02-01T10:41:41.022 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:aQf9P9fg/qSjz61Vy057NG4Paik= 2024-02-01T10:41:41.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:41:41 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:aQf9P9fg/qSjz61Vy057NG4Paik=', 'Content-Length': '0'} 2024-02-01T10:41:41.026 DEBUG:boto:Response headers: [('Location', 'http://smithi148.front.sepia.ceph.com:8000/vclfcn-49/testredirect?'), ('x-amz-request-id', 'tx000000eebce76fad0e45a-0065bb7565-5276-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:41:41 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:41:41.027 DEBUG:boto:Redirecting: http://smithi148.front.sepia.ceph.com/vclfcn-49/testredirect 2024-02-01T10:41:41.027 DEBUG:boto:Token: None 2024-02-01T10:41:41.027 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:41:41 GMT /vclfcn-49/testredirect 2024-02-01T10:41:41.027 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:aQf9P9fg/qSjz61Vy057NG4Paik= 2024-02-01T10:41:41.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:41:41 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:aQf9P9fg/qSjz61Vy057NG4Paik=', 'Content-Length': '0'} 2024-02-01T10:41:41.030 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:41:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"e9a2a6bda24c81a2bb676adf0d9bd57a"'), ('x-amz-request-id', 'tx0000041a4aa69bcce88ec-0065bb7565-5645-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:41:41 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:41:41.031 DEBUG:boto:path=/vclfcn-49/testredirect 2024-02-01T10:41:41.031 DEBUG:boto:auth_path=/vclfcn-49/testredirect 2024-02-01T10:41:41.031 DEBUG:boto:Method: PUT 2024-02-01T10:41:41.031 DEBUG:boto:Path: /vclfcn-49/testredirect 2024-02-01T10:41:41.031 DEBUG:boto:Data: 2024-02-01T10:41:41.031 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'GbNgNQ5MAukCtgY+rsT5Xw==', 'Content-Length': '512', 'Expect': '100-Continue'} 2024-02-01T10:41:41.032 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:41:41.032 DEBUG:boto:Port: 8000 2024-02-01T10:41:41.032 DEBUG:boto:Params: {} 2024-02-01T10:41:41.032 DEBUG:boto:Token: None 2024-02-01T10:41:41.032 DEBUG:boto:StringToSign: PUT GbNgNQ5MAukCtgY+rsT5Xw== application/octet-stream Thu, 01 Feb 2024 10:41:41 GMT /vclfcn-49/testredirect 2024-02-01T10:41:41.032 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/WQ/plr9pPk2HA8FAovoVW4T0BE= 2024-02-01T10:41:41.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'GbNgNQ5MAukCtgY+rsT5Xw==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:41:41 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/WQ/plr9pPk2HA8FAovoVW4T0BE='} 2024-02-01T10:41:41.076 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"19b360350e4c02e902b6063eaec4f95f"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c3a6e06af316f91b-0065bb7565-5645-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:41:41 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:41:41.076 DEBUG:boto:path=/vclfcn-49/testredirect 2024-02-01T10:41:41.077 DEBUG:boto:auth_path=/vclfcn-49/testredirect 2024-02-01T10:41:41.077 DEBUG:boto:Method: GET 2024-02-01T10:41:41.077 DEBUG:boto:Path: /vclfcn-49/testredirect 2024-02-01T10:41:41.077 DEBUG:boto:Data: 2024-02-01T10:41:41.077 DEBUG:boto:Headers: {} 2024-02-01T10:41:41.077 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:41:41.077 DEBUG:boto:Port: 8000 2024-02-01T10:41:41.077 DEBUG:boto:Params: {} 2024-02-01T10:41:41.078 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T10:41:41.078 DEBUG:boto:Token: None 2024-02-01T10:41:41.078 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:41:41 GMT /vclfcn-49/testredirect 2024-02-01T10:41:41.078 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:aQf9P9fg/qSjz61Vy057NG4Paik= 2024-02-01T10:41:41.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:41:41 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:aQf9P9fg/qSjz61Vy057NG4Paik=', 'Content-Length': '0'} 2024-02-01T10:41:41.082 DEBUG:boto:Response headers: [('Location', 'http://smithi148.front.sepia.ceph.com:8000/vclfcn-49/testredirect?'), ('x-amz-request-id', 'tx000007cb8b8de940155a3-0065bb7565-5276-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:41:41 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:41:41.082 DEBUG:boto:Redirecting: http://smithi148.front.sepia.ceph.com/vclfcn-49/testredirect 2024-02-01T10:41:41.082 DEBUG:boto:Token: None 2024-02-01T10:41:41.082 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:41:41 GMT /vclfcn-49/testredirect 2024-02-01T10:41:41.082 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:aQf9P9fg/qSjz61Vy057NG4Paik= 2024-02-01T10:41:41.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:41:41 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:aQf9P9fg/qSjz61Vy057NG4Paik=', 'Content-Length': '0'} 2024-02-01T10:41:41.085 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:41:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"19b360350e4c02e902b6063eaec4f95f"'), ('x-amz-request-id', 'tx000006b6c6cdc890e03ab-0065bb7565-5645-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:41:41 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:41:41.086 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=true', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:41:41.086 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=true', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:41:41.086 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --sync-from-all=true --cluster c2 --debug-rgw 0 --debug-ms 0 2024-02-01T10:41:41.130 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:41:41.130 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:41:41.181 INFO:teuthology.orchestra.run.smithi187.stdout:{"id":"01fb6d04-f0ef-49b2-9ebc-c1b530925485","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"1C56UIO2VKZWOSJZ","secret_key":"4X9hEalkm5sn0VTq71G3qlDBF1FRjy9C"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"c7bb4449-4a82-4742-a9c6-c7422f73e18f","notif_pool":"test-zone2.rgw.log:notif"} 2024-02-01T10:41:41.182 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:41:41.182 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:41:41.182 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2024-02-01T10:41:41.264 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:41:41.265 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:41:41.330 INFO:teuthology.orchestra.run.smithi187.stderr:Sending period to new master zone ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01 2024-02-01T10:41:41.407 INFO:teuthology.orchestra.run.smithi187.stdout:{"id":"f2e6c290-e7aa-4279-b77e-15403b18d61f","epoch":4,"predecessor_uuid":"976daaff-eac6-4c84-bd63-38e72aa959ff","sync_status":["","","1_1706783813.899233_136.1",""],"period_map":{"id":"f2e6c290-e7aa-4279-b77e-15403b18d61f","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":true,"endpoints":["http://smithi148.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01","zones":[{"id":"01fb6d04-f0ef-49b2-9ebc-c1b530925485","name":"test-zone2","endpoints":["http://smithi187.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01","supported_features":["compress-encrypted","resharding"]},{"id":"2ea581c3-3f48-49d7-a989-68024c5b8eac","name":"test-zone3","endpoints":["http://smithi148.front.sepia.ceph.com:8001"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]},{"id":"ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01","name":"test-zone1","endpoints":["http://smithi148.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"c7bb4449-4a82-4742-a9c6-c7422f73e18f","sync_policy":{"groups":[]},"enabled_features":["resharding"]}],"short_zone_ids":[{"key":"01fb6d04-f0ef-49b2-9ebc-c1b530925485","val":1553472663},{"key":"2ea581c3-3f48-49d7-a989-68024c5b8eac","val":1173357353},{"key":"ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01","val":3362809119}]},"master_zonegroup":"test-zonegroup","master_zone":"ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"bucket_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"anonymous_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false}},"realm_id":"c7bb4449-4a82-4742-a9c6-c7422f73e18f","realm_epoch":4} 2024-02-01T10:41:41.407 INFO:rgw_multi.tests:Set sync_from_all flag on zone test-zone2 to true 2024-02-01T10:41:41.636 INFO:tasks.rgw.c2.client.0.smithi187.stdout:2024-02-01T10:41:41.632+0000 7f7bead57640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-02-01T10:41:41.648 INFO:tasks.rgw.c2.client.0.smithi187.stdout:2024-02-01T10:41:41.644+0000 7f7be9554640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-02-01T10:42:12.641 INFO:tasks.rgw.c1.client.0.smithi148.stdout:2024-02-01T10:42:12.645+0000 7f5f4234d640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-02-01T10:42:41.408 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:42:41.409 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:42:41.409 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --redirect-zone= --cluster c2 --debug-rgw 0 --debug-ms 0 2024-02-01T10:42:41.452 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:42:41.453 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:42:41.508 INFO:teuthology.orchestra.run.smithi187.stdout:{"id":"01fb6d04-f0ef-49b2-9ebc-c1b530925485","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"1C56UIO2VKZWOSJZ","secret_key":"4X9hEalkm5sn0VTq71G3qlDBF1FRjy9C"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"c7bb4449-4a82-4742-a9c6-c7422f73e18f","notif_pool":"test-zone2.rgw.log:notif"} 2024-02-01T10:42:41.509 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:42:41.509 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:42:41.509 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2024-02-01T10:42:41.584 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:42:41.584 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:42:41.652 INFO:teuthology.orchestra.run.smithi187.stderr:Sending period to new master zone ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01 2024-02-01T10:42:41.723 INFO:teuthology.orchestra.run.smithi187.stdout:{"id":"f2e6c290-e7aa-4279-b77e-15403b18d61f","epoch":5,"predecessor_uuid":"976daaff-eac6-4c84-bd63-38e72aa959ff","sync_status":["","","1_1706783813.899233_136.1",""],"period_map":{"id":"f2e6c290-e7aa-4279-b77e-15403b18d61f","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":true,"endpoints":["http://smithi148.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01","zones":[{"id":"01fb6d04-f0ef-49b2-9ebc-c1b530925485","name":"test-zone2","endpoints":["http://smithi187.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]},{"id":"2ea581c3-3f48-49d7-a989-68024c5b8eac","name":"test-zone3","endpoints":["http://smithi148.front.sepia.ceph.com:8001"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]},{"id":"ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01","name":"test-zone1","endpoints":["http://smithi148.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"c7bb4449-4a82-4742-a9c6-c7422f73e18f","sync_policy":{"groups":[]},"enabled_features":["resharding"]}],"short_zone_ids":[{"key":"01fb6d04-f0ef-49b2-9ebc-c1b530925485","val":1553472663},{"key":"2ea581c3-3f48-49d7-a989-68024c5b8eac","val":1173357353},{"key":"ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01","val":3362809119}]},"master_zonegroup":"test-zonegroup","master_zone":"ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"bucket_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"anonymous_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false}},"realm_id":"c7bb4449-4a82-4742-a9c6-c7422f73e18f","realm_epoch":4} 2024-02-01T10:42:41.723 INFO:rgw_multi.tests:Set redirect_zone zone test-zone2 to "" 2024-02-01T10:42:41.993 INFO:tasks.rgw.c1.client.0.smithi148.stdout:2024-02-01T10:42:41.996+0000 7f5f40b4a640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-02-01T10:43:41.724 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_zone_redirect ... ok 2024-02-01T10:43:41.726 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:43:41.726 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:43:41.726 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:43:41.772 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:43:41.820 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:43:41.831 INFO:teuthology.orchestra.run.smithi148.stderr:No --period given, using current period=f2e6c290-e7aa-4279-b77e-15403b18d61f 2024-02-01T10:43:41.856 INFO:teuthology.orchestra.run.smithi148.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1706783978.681171_139.1","last_update":"2024-02-01T10:39:38.681171Z"},{"marker":"","last_update":"0.000000"}] 2024-02-01T10:43:41.862 DEBUG:rgw_multi.tests:master meta markers={0: '', 1: '', 2: '1_1706783978.681171_139.1', 3: ''} 2024-02-01T10:43:41.862 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-02-01T10:43:41.862 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:43:41.862 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:43:41.862 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:43:41.905 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:43:41.920 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:43:41.961 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:43:41.958+0000 7fe74f170980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:43:41.962 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:43:41.962+0000 7fe74f170980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:43:41.986 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783978.681171_139.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:39:38.681171Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:43:41Z"} 2024-02-01T10:43:41.986 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783978.681171_139.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:39:38.681171Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:43:41Z"} 2024-02-01T10:43:41.986 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706783978.681171_139.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:39:38.681171Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:43:41.986 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1706783978.681171_139.1', 3: ''} 2024-02-01T10:43:41.986 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1706783978.681171_139.1', 3: ''} 2024-02-01T10:43:41.987 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-02-01T10:43:41.987 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-02-01T10:43:41.987 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:43:41.987 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:43:41.987 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:43:42.030 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:43:42.045 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:43:42.084 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:43:42.085+0000 7fea5af0f980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:43:42.085 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:43:42.089+0000 7fea5af0f980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:43:42.113 INFO:teuthology.orchestra.run.smithi148.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783978.681171_139.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:39:38.681171Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:43:42Z"} 2024-02-01T10:43:42.114 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783978.681171_139.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:39:38.681171Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:43:42Z"} 2024-02-01T10:43:42.114 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706783978.681171_139.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:39:38.681171Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:43:42.114 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1706783978.681171_139.1', 3: ''} 2024-02-01T10:43:42.114 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1706783978.681171_139.1', 3: ''} 2024-02-01T10:43:42.114 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-02-01T10:43:42.114 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', '1C56UIO2VKZWOSJZ', '--secret', '4X9hEalkm5sn0VTq71G3qlDBF1FRjy9C', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:43:42.114 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', '1C56UIO2VKZWOSJZ', '--secret', '4X9hEalkm5sn0VTq71G3qlDBF1FRjy9C', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:43:42.115 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key 1C56UIO2VKZWOSJZ --secret 4X9hEalkm5sn0VTq71G3qlDBF1FRjy9C --cluster c2 --debug-rgw 0 --debug-ms 0 2024-02-01T10:43:42.160 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:43:42.160 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:43:42.196 INFO:teuthology.orchestra.run.smithi187.stderr:NOTICE: set zone's realm_id=c7bb4449-4a82-4742-a9c6-c7422f73e18f 2024-02-01T10:43:42.221 INFO:teuthology.orchestra.run.smithi187.stdout:{"id":"65ab88ec-44f2-44b9-be4b-6061213b790e","name":"remove","domain_root":"remove.rgw.meta:root","control_pool":"remove.rgw.control","gc_pool":"remove.rgw.log:gc","lc_pool":"remove.rgw.log:lc","log_pool":"remove.rgw.log","intent_log_pool":"remove.rgw.log:intent","usage_log_pool":"remove.rgw.log:usage","roles_pool":"remove.rgw.meta:roles","reshard_pool":"remove.rgw.log:reshard","user_keys_pool":"remove.rgw.meta:users.keys","user_email_pool":"remove.rgw.meta:users.email","user_swift_pool":"remove.rgw.meta:users.swift","user_uid_pool":"remove.rgw.meta:users.uid","otp_pool":"remove.rgw.otp","system_key":{"access_key":"1C56UIO2VKZWOSJZ","secret_key":"4X9hEalkm5sn0VTq71G3qlDBF1FRjy9C"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"remove.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"remove.rgw.buckets.data"}},"data_extra_pool":"remove.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"c7bb4449-4a82-4742-a9c6-c7422f73e18f","notif_pool":"remove.rgw.log:notif"} 2024-02-01T10:43:42.222 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:43:42.222 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:43:42.222 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2024-02-01T10:43:42.300 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:43:42.320 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:43:42.338 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:43:42.338+0000 7fbf1692a980 0 period (f2e6c290-e7aa-4279-b77e-15403b18d61f does not have zone 65ab88ec-44f2-44b9-be4b-6061213b790e configured 2024-02-01T10:43:48.882 INFO:teuthology.orchestra.run.smithi187.stderr:Sending period to new master zone ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01 2024-02-01T10:43:48.970 INFO:teuthology.orchestra.run.smithi187.stdout:{"id":"f2e6c290-e7aa-4279-b77e-15403b18d61f","epoch":6,"predecessor_uuid":"976daaff-eac6-4c84-bd63-38e72aa959ff","sync_status":["","","1_1706783813.899233_136.1",""],"period_map":{"id":"f2e6c290-e7aa-4279-b77e-15403b18d61f","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":true,"endpoints":["http://smithi148.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01","zones":[{"id":"01fb6d04-f0ef-49b2-9ebc-c1b530925485","name":"test-zone2","endpoints":["http://smithi187.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]},{"id":"2ea581c3-3f48-49d7-a989-68024c5b8eac","name":"test-zone3","endpoints":["http://smithi148.front.sepia.ceph.com:8001"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]},{"id":"65ab88ec-44f2-44b9-be4b-6061213b790e","name":"remove","endpoints":[],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]},{"id":"ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01","name":"test-zone1","endpoints":["http://smithi148.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"c7bb4449-4a82-4742-a9c6-c7422f73e18f","sync_policy":{"groups":[]},"enabled_features":["resharding"]}],"short_zone_ids":[{"key":"01fb6d04-f0ef-49b2-9ebc-c1b530925485","val":1553472663},{"key":"2ea581c3-3f48-49d7-a989-68024c5b8eac","val":1173357353},{"key":"65ab88ec-44f2-44b9-be4b-6061213b790e","val":2336022615},{"key":"ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01","val":3362809119}]},"master_zonegroup":"test-zonegroup","master_zone":"ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"bucket_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"anonymous_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false}},"realm_id":"c7bb4449-4a82-4742-a9c6-c7422f73e18f","realm_epoch":4} 2024-02-01T10:43:48.971 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:43:48.971 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:43:48.971 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2024-02-01T10:43:49.015 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:43:49.015 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:43:49.081 INFO:teuthology.orchestra.run.smithi148.stdout:{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":true,"endpoints":["http://smithi148.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01","zones":[{"id":"01fb6d04-f0ef-49b2-9ebc-c1b530925485","name":"test-zone2","endpoints":["http://smithi187.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]},{"id":"2ea581c3-3f48-49d7-a989-68024c5b8eac","name":"test-zone3","endpoints":["http://smithi148.front.sepia.ceph.com:8001"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]},{"id":"ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01","name":"test-zone1","endpoints":["http://smithi148.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"c7bb4449-4a82-4742-a9c6-c7422f73e18f","sync_policy":{"groups":[]},"enabled_features":["resharding"]} 2024-02-01T10:43:49.082 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:43:49.082 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:43:49.082 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2024-02-01T10:43:49.170 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:43:49.170 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:43:49.232 INFO:teuthology.orchestra.run.smithi148.stderr:zone name remove not found in zonegroup test-zonegroup 2024-02-01T10:43:49.253 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-02-01T10:43:49.254 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:43:49.254 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'delete', '--rgw-zone', 'remove', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:43:49.255 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone delete --rgw-zone remove --cluster c2 --debug-rgw 0 --debug-ms 0 2024-02-01T10:43:49.293 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:43:49.293 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:43:49.336 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:43:49.334+0000 7f3c3ffff640 0 ms_deliver_dispatch: unhandled message 0x7f3de0023e50 osd_map(52..52 src has 1..52) v4 from mon.0 v2:172.21.15.187:3300/0 2024-02-01T10:43:49.354 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:43:49.350+0000 7f3df95f2980 0 Removed zone from zonegroup test-zonegroup 2024-02-01T10:43:49.371 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:43:49.371 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:43:49.372 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-02-01T10:43:49.413 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:43:49.478 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:43:49.479 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:43:49.473+0000 7fe8f9323980 0 WARNING: can't generate connection for zone 65ab88ec-44f2-44b9-be4b-6061213b790e id remove: no endpoints defined 2024-02-01T10:43:49.550 INFO:teuthology.orchestra.run.smithi148.stdout:{"id":"f2e6c290-e7aa-4279-b77e-15403b18d61f","epoch":7,"predecessor_uuid":"976daaff-eac6-4c84-bd63-38e72aa959ff","sync_status":["","","1_1706783813.899233_136.1",""],"period_map":{"id":"f2e6c290-e7aa-4279-b77e-15403b18d61f","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":true,"endpoints":["http://smithi148.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01","zones":[{"id":"01fb6d04-f0ef-49b2-9ebc-c1b530925485","name":"test-zone2","endpoints":["http://smithi187.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]},{"id":"2ea581c3-3f48-49d7-a989-68024c5b8eac","name":"test-zone3","endpoints":["http://smithi148.front.sepia.ceph.com:8001"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]},{"id":"ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01","name":"test-zone1","endpoints":["http://smithi148.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"c7bb4449-4a82-4742-a9c6-c7422f73e18f","sync_policy":{"groups":[]},"enabled_features":["resharding"]}],"short_zone_ids":[{"key":"01fb6d04-f0ef-49b2-9ebc-c1b530925485","val":1553472663},{"key":"2ea581c3-3f48-49d7-a989-68024c5b8eac","val":1173357353},{"key":"ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01","val":3362809119}]},"master_zonegroup":"test-zonegroup","master_zone":"ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"bucket_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"anonymous_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false}},"realm_id":"c7bb4449-4a82-4742-a9c6-c7422f73e18f","realm_epoch":4} 2024-02-01T10:43:49.550 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_zonegroup_remove ... ok 2024-02-01T10:43:49.551 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:43:49.551 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:43:49.551 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup create --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2024-02-01T10:43:49.619 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:43:49.678 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:43:49.679 INFO:teuthology.orchestra.run.smithi148.stdout:{"id":"4339cf37-4249-496b-8613-540ab92271a1","name":"remove_zg","api_name":"remove_zg","is_master":false,"endpoints":[],"hostnames":[],"hostnames_s3website":[],"master_zone":"","zones":[],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":[]}],"default_placement":"default-placement","realm_id":"c7bb4449-4a82-4742-a9c6-c7422f73e18f","sync_policy":{"groups":[]},"enabled_features":["resharding"]} 2024-02-01T10:43:49.679 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:43:49.680 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:43:49.680 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2024-02-01T10:43:49.761 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:43:49.761 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:43:49.869 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:43:49.869+0000 7fad0df1c980 0 NOTICE: promoted remove as new master_zone of zonegroup remove_zg 2024-02-01T10:43:49.889 INFO:teuthology.orchestra.run.smithi148.stdout:{"id":"3834783a-ea9a-46ff-93ef-95140f757d93","name":"remove","domain_root":"remove.rgw.meta:root","control_pool":"remove.rgw.control","gc_pool":"remove.rgw.log:gc","lc_pool":"remove.rgw.log:lc","log_pool":"remove.rgw.log","intent_log_pool":"remove.rgw.log:intent","usage_log_pool":"remove.rgw.log:usage","roles_pool":"remove.rgw.meta:roles","reshard_pool":"remove.rgw.log:reshard","user_keys_pool":"remove.rgw.meta:users.keys","user_email_pool":"remove.rgw.meta:users.email","user_swift_pool":"remove.rgw.meta:users.swift","user_uid_pool":"remove.rgw.meta:users.uid","otp_pool":"remove.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"remove.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"remove.rgw.buckets.data"}},"data_extra_pool":"remove.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"c7bb4449-4a82-4742-a9c6-c7422f73e18f","notif_pool":"remove.rgw.log:notif"} 2024-02-01T10:43:49.889 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:43:49.889 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:43:49.889 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-02-01T10:43:49.976 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:43:49.977 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:43:50.119 INFO:teuthology.orchestra.run.smithi148.stdout:{"id":"f2e6c290-e7aa-4279-b77e-15403b18d61f","epoch":8,"predecessor_uuid":"976daaff-eac6-4c84-bd63-38e72aa959ff","sync_status":["","","1_1706783813.899233_136.1",""],"period_map":{"id":"f2e6c290-e7aa-4279-b77e-15403b18d61f","zonegroups":[{"id":"4339cf37-4249-496b-8613-540ab92271a1","name":"remove_zg","api_name":"remove_zg","is_master":false,"endpoints":[],"hostnames":[],"hostnames_s3website":[],"master_zone":"3834783a-ea9a-46ff-93ef-95140f757d93","zones":[{"id":"3834783a-ea9a-46ff-93ef-95140f757d93","name":"remove","endpoints":[],"log_meta":false,"log_data":false,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"c7bb4449-4a82-4742-a9c6-c7422f73e18f","sync_policy":{"groups":[]},"enabled_features":["resharding"]},{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":true,"endpoints":["http://smithi148.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01","zones":[{"id":"01fb6d04-f0ef-49b2-9ebc-c1b530925485","name":"test-zone2","endpoints":["http://smithi187.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]},{"id":"2ea581c3-3f48-49d7-a989-68024c5b8eac","name":"test-zone3","endpoints":["http://smithi148.front.sepia.ceph.com:8001"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]},{"id":"ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01","name":"test-zone1","endpoints":["http://smithi148.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"c7bb4449-4a82-4742-a9c6-c7422f73e18f","sync_policy":{"groups":[]},"enabled_features":["resharding"]}],"short_zone_ids":[{"key":"01fb6d04-f0ef-49b2-9ebc-c1b530925485","val":1553472663},{"key":"2ea581c3-3f48-49d7-a989-68024c5b8eac","val":1173357353},{"key":"3834783a-ea9a-46ff-93ef-95140f757d93","val":846690924},{"key":"ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01","val":3362809119}]},"master_zonegroup":"test-zonegroup","master_zone":"ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"bucket_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"anonymous_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false}},"realm_id":"c7bb4449-4a82-4742-a9c6-c7422f73e18f","realm_epoch":4} 2024-02-01T10:43:50.120 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:43:50.120 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:43:50.120 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone delete --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2024-02-01T10:43:50.201 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:43:50.201 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:43:50.258 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:43:50.261+0000 7ff81bed5980 0 NOTICE: removed master_zone of zonegroup remove_zg 2024-02-01T10:43:50.259 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:43:50.261+0000 7ff81bed5980 0 Removed zone from zonegroup remove_zg 2024-02-01T10:43:50.284 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:43:50.284 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:43:50.284 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-02-01T10:43:50.371 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:43:50.372 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:43:50.436 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:43:50.437+0000 7f125a7e4640 0 ms_deliver_dispatch: unhandled message 0x7f1388043b50 osd_map(56..56 src has 1..56) v4 from mon.0 v2:172.21.15.148:3300/0 2024-02-01T10:43:50.458 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:43:50.461+0000 7f1397d28980 0 ERROR: zonegroup remove_zg has a non existent master zone 2024-02-01T10:43:50.480 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-02-01T10:43:50.481 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:43:50.481 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:43:50.481 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup delete --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2024-02-01T10:43:50.520 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:43:50.521 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:43:50.589 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:43:50.589 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:43:50.589 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-02-01T10:43:50.671 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:43:50.672 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:43:50.805 INFO:teuthology.orchestra.run.smithi148.stdout:{"id":"f2e6c290-e7aa-4279-b77e-15403b18d61f","epoch":9,"predecessor_uuid":"976daaff-eac6-4c84-bd63-38e72aa959ff","sync_status":["","","1_1706783813.899233_136.1",""],"period_map":{"id":"f2e6c290-e7aa-4279-b77e-15403b18d61f","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":true,"endpoints":["http://smithi148.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01","zones":[{"id":"01fb6d04-f0ef-49b2-9ebc-c1b530925485","name":"test-zone2","endpoints":["http://smithi187.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]},{"id":"2ea581c3-3f48-49d7-a989-68024c5b8eac","name":"test-zone3","endpoints":["http://smithi148.front.sepia.ceph.com:8001"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]},{"id":"ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01","name":"test-zone1","endpoints":["http://smithi148.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"c7bb4449-4a82-4742-a9c6-c7422f73e18f","sync_policy":{"groups":[]},"enabled_features":["resharding"]}],"short_zone_ids":[{"key":"01fb6d04-f0ef-49b2-9ebc-c1b530925485","val":1553472663},{"key":"2ea581c3-3f48-49d7-a989-68024c5b8eac","val":1173357353},{"key":"ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01","val":3362809119}]},"master_zonegroup":"test-zonegroup","master_zone":"ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"bucket_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"anonymous_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false}},"realm_id":"c7bb4449-4a82-4742-a9c6-c7422f73e18f","realm_epoch":4} 2024-02-01T10:43:50.805 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_zg_master_zone_delete ... ok 2024-02-01T10:43:50.806 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=vclfcn-50 2024-02-01T10:43:50.806 DEBUG:boto:path=/vclfcn-50/ 2024-02-01T10:43:50.807 DEBUG:boto:auth_path=/vclfcn-50/ 2024-02-01T10:43:50.807 DEBUG:boto:Method: PUT 2024-02-01T10:43:50.807 DEBUG:boto:Path: /vclfcn-50/ 2024-02-01T10:43:50.807 DEBUG:boto:Data: 2024-02-01T10:43:50.807 DEBUG:boto:Headers: {} 2024-02-01T10:43:50.807 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:43:50.807 DEBUG:boto:Port: 8000 2024-02-01T10:43:50.807 DEBUG:boto:Params: {} 2024-02-01T10:43:50.808 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T10:43:50.808 DEBUG:boto:Token: None 2024-02-01T10:43:50.808 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:43:50 GMT /vclfcn-50/ 2024-02-01T10:43:50.808 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ff7u6KPWLW9hruQkWLjhqIEZuXA= 2024-02-01T10:43:50.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:43:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ff7u6KPWLW9hruQkWLjhqIEZuXA=', 'Content-Length': '0'} 2024-02-01T10:43:51.119 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000102feee4b51a4dcd-0065bb75e7-5765-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:43:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:43:51.119 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=vclfcn-51 2024-02-01T10:43:51.119 DEBUG:boto:path=/vclfcn-51/ 2024-02-01T10:43:51.119 DEBUG:boto:auth_path=/vclfcn-51/ 2024-02-01T10:43:51.119 DEBUG:boto:Method: PUT 2024-02-01T10:43:51.119 DEBUG:boto:Path: /vclfcn-51/ 2024-02-01T10:43:51.119 DEBUG:boto:Data: 2024-02-01T10:43:51.120 DEBUG:boto:Headers: {} 2024-02-01T10:43:51.120 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:43:51.120 DEBUG:boto:Port: 8000 2024-02-01T10:43:51.120 DEBUG:boto:Params: {} 2024-02-01T10:43:51.120 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T10:43:51.120 DEBUG:boto:Token: None 2024-02-01T10:43:51.121 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:43:51 GMT /vclfcn-51/ 2024-02-01T10:43:51.121 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:3EJKfTtGSAi4b4+55AlFK2oTsh0= 2024-02-01T10:43:51.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:43:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:3EJKfTtGSAi4b4+55AlFK2oTsh0=', 'Content-Length': '0'} 2024-02-01T10:43:51.165 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004785fcef3d2aba0b-0065bb75e7-5350-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:43:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:43:51.165 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=vclfcn-52 2024-02-01T10:43:51.165 DEBUG:boto:path=/vclfcn-52/ 2024-02-01T10:43:51.166 DEBUG:boto:auth_path=/vclfcn-52/ 2024-02-01T10:43:51.166 DEBUG:boto:Method: PUT 2024-02-01T10:43:51.166 DEBUG:boto:Path: /vclfcn-52/ 2024-02-01T10:43:51.166 DEBUG:boto:Data: 2024-02-01T10:43:51.166 DEBUG:boto:Headers: {} 2024-02-01T10:43:51.166 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:43:51.166 DEBUG:boto:Port: 8001 2024-02-01T10:43:51.166 DEBUG:boto:Params: {} 2024-02-01T10:43:51.167 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-02-01T10:43:51.167 DEBUG:boto:Token: None 2024-02-01T10:43:51.167 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:43:51 GMT /vclfcn-52/ 2024-02-01T10:43:51.167 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:PCWlbcEP7axiKyxjpi9BYS7lHgU= 2024-02-01T10:43:51.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:43:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:PCWlbcEP7axiKyxjpi9BYS7lHgU=', 'Content-Length': '0'} 2024-02-01T10:43:51.209 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006a3668d50b6101c2-0065bb75e7-5762-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:43:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:43:51.209 DEBUG:boto:path=/vclfcn-50/ 2024-02-01T10:43:51.209 DEBUG:boto:auth_path=/vclfcn-50/ 2024-02-01T10:43:51.209 DEBUG:boto:path=/vclfcn-50/?website 2024-02-01T10:43:51.210 DEBUG:boto:auth_path=/vclfcn-50/?website 2024-02-01T10:43:51.210 DEBUG:boto:Method: PUT 2024-02-01T10:43:51.210 DEBUG:boto:Path: /vclfcn-50/?website 2024-02-01T10:43:51.210 DEBUG:boto:Data: index.htmlerror.html 2024-02-01T10:43:51.210 DEBUG:boto:Headers: {} 2024-02-01T10:43:51.210 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:43:51.210 DEBUG:boto:Port: 8000 2024-02-01T10:43:51.210 DEBUG:boto:Params: {} 2024-02-01T10:43:51.210 DEBUG:boto:Token: None 2024-02-01T10:43:51.211 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:43:51 GMT /vclfcn-50/?website 2024-02-01T10:43:51.211 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zxRhLbKCD6bsrbkLcK6Z5hINqtI= 2024-02-01T10:43:51.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:43:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zxRhLbKCD6bsrbkLcK6Z5hINqtI=', 'Content-Length': '241'} 2024-02-01T10:43:51.212 DEBUG:boto:Response headers: [('Content-Length', '221'), ('x-amz-request-id', 'tx000006ae4805c53bde47a-0065bb75e7-5765-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:43:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:43:51.214 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_set_bucket_website ... SKIP: test_set_bucket_website skipped. Requires rgw_enable_static_website = 1. 2024-02-01T10:43:51.214 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=vclfcn-53 2024-02-01T10:43:51.215 DEBUG:boto:path=/vclfcn-53/ 2024-02-01T10:43:51.215 DEBUG:boto:auth_path=/vclfcn-53/ 2024-02-01T10:43:51.215 DEBUG:boto:Method: PUT 2024-02-01T10:43:51.215 DEBUG:boto:Path: /vclfcn-53/ 2024-02-01T10:43:51.216 DEBUG:boto:Data: 2024-02-01T10:43:51.216 DEBUG:boto:Headers: {} 2024-02-01T10:43:51.216 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:43:51.216 DEBUG:boto:Port: 8000 2024-02-01T10:43:51.216 DEBUG:boto:Params: {} 2024-02-01T10:43:51.216 DEBUG:boto:Token: None 2024-02-01T10:43:51.217 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:43:51 GMT /vclfcn-53/ 2024-02-01T10:43:51.217 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jk2iUNss/xctm8ZjKhFQFQVCOJk= 2024-02-01T10:43:51.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:43:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jk2iUNss/xctm8ZjKhFQFQVCOJk=', 'Content-Length': '0'} 2024-02-01T10:43:51.240 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d2c08622704dd623-0065bb75e7-5765-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:43:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:43:51.240 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=vclfcn-54 2024-02-01T10:43:51.240 DEBUG:boto:path=/vclfcn-54/ 2024-02-01T10:43:51.240 DEBUG:boto:auth_path=/vclfcn-54/ 2024-02-01T10:43:51.240 DEBUG:boto:Method: PUT 2024-02-01T10:43:51.240 DEBUG:boto:Path: /vclfcn-54/ 2024-02-01T10:43:51.241 DEBUG:boto:Data: 2024-02-01T10:43:51.241 DEBUG:boto:Headers: {} 2024-02-01T10:43:51.241 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:43:51.241 DEBUG:boto:Port: 8000 2024-02-01T10:43:51.241 DEBUG:boto:Params: {} 2024-02-01T10:43:51.241 DEBUG:boto:Token: None 2024-02-01T10:43:51.241 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:43:51 GMT /vclfcn-54/ 2024-02-01T10:43:51.241 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:4iSEYBGo837bjZi+M4BsHqIlbEY= 2024-02-01T10:43:51.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:43:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:4iSEYBGo837bjZi+M4BsHqIlbEY=', 'Content-Length': '0'} 2024-02-01T10:43:51.275 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000093788356cc0ddef5-0065bb75e7-5350-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:43:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:43:51.275 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=vclfcn-55 2024-02-01T10:43:51.275 DEBUG:boto:path=/vclfcn-55/ 2024-02-01T10:43:51.275 DEBUG:boto:auth_path=/vclfcn-55/ 2024-02-01T10:43:51.275 DEBUG:boto:Method: PUT 2024-02-01T10:43:51.275 DEBUG:boto:Path: /vclfcn-55/ 2024-02-01T10:43:51.276 DEBUG:boto:Data: 2024-02-01T10:43:51.276 DEBUG:boto:Headers: {} 2024-02-01T10:43:51.276 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:43:51.276 DEBUG:boto:Port: 8001 2024-02-01T10:43:51.276 DEBUG:boto:Params: {} 2024-02-01T10:43:51.276 DEBUG:boto:Token: None 2024-02-01T10:43:51.276 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:43:51 GMT /vclfcn-55/ 2024-02-01T10:43:51.276 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:uX8YGiVSIGm0RFjekkFJEmVA1Kg= 2024-02-01T10:43:51.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:43:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:uX8YGiVSIGm0RFjekkFJEmVA1Kg=', 'Content-Length': '0'} 2024-02-01T10:43:51.308 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001de38fe9166c764e-0065bb75e7-5762-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:43:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:43:51.308 DEBUG:boto:path=/vclfcn-53/ 2024-02-01T10:43:51.308 DEBUG:boto:auth_path=/vclfcn-53/ 2024-02-01T10:43:51.308 DEBUG:boto:path=/vclfcn-53/?policy 2024-02-01T10:43:51.308 DEBUG:boto:auth_path=/vclfcn-53/?policy 2024-02-01T10:43:51.308 DEBUG:boto:Method: PUT 2024-02-01T10:43:51.308 DEBUG:boto:Path: /vclfcn-53/?policy 2024-02-01T10:43:51.309 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2024-02-01T10:43:51.309 DEBUG:boto:Headers: {} 2024-02-01T10:43:51.309 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:43:51.309 DEBUG:boto:Port: 8000 2024-02-01T10:43:51.309 DEBUG:boto:Params: {} 2024-02-01T10:43:51.309 DEBUG:boto:Token: None 2024-02-01T10:43:51.309 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:43:51 GMT /vclfcn-53/?policy 2024-02-01T10:43:51.309 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:4XLpRz+K78sBCN/0JXnkWRoxzxY= 2024-02-01T10:43:51.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:43:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:4XLpRz+K78sBCN/0JXnkWRoxzxY=', 'Content-Length': '97'} 2024-02-01T10:43:51.316 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000011eb8f4c386be5a5-0065bb75e7-5765-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:43:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:43:51.316 DEBUG:boto:path=/vclfcn-53/ 2024-02-01T10:43:51.316 DEBUG:boto:auth_path=/vclfcn-53/ 2024-02-01T10:43:51.316 DEBUG:boto:path=/vclfcn-53/?policy 2024-02-01T10:43:51.316 DEBUG:boto:auth_path=/vclfcn-53/?policy 2024-02-01T10:43:51.316 DEBUG:boto:Method: GET 2024-02-01T10:43:51.316 DEBUG:boto:Path: /vclfcn-53/?policy 2024-02-01T10:43:51.316 DEBUG:boto:Data: 2024-02-01T10:43:51.316 DEBUG:boto:Headers: {} 2024-02-01T10:43:51.317 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:43:51.317 DEBUG:boto:Port: 8000 2024-02-01T10:43:51.317 DEBUG:boto:Params: {} 2024-02-01T10:43:51.317 DEBUG:boto:Token: None 2024-02-01T10:43:51.317 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:43:51 GMT /vclfcn-53/?policy 2024-02-01T10:43:51.317 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:q98H1whoJsKXLn5FJTN22m5Zmaw= 2024-02-01T10:43:51.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:43:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:q98H1whoJsKXLn5FJTN22m5Zmaw=', 'Content-Length': '0'} 2024-02-01T10:43:51.318 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008af22b20961472f5-0065bb75e7-5765-test-zone1'), ('Content-Type', 'application/json'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '97'), ('Date', 'Thu, 01 Feb 2024 10:43:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:43:51.319 DEBUG:boto:path=/vclfcn-54/ 2024-02-01T10:43:51.319 DEBUG:boto:auth_path=/vclfcn-54/ 2024-02-01T10:43:51.319 DEBUG:boto:path=/vclfcn-54/?policy 2024-02-01T10:43:51.319 DEBUG:boto:auth_path=/vclfcn-54/?policy 2024-02-01T10:43:51.319 DEBUG:boto:Method: PUT 2024-02-01T10:43:51.319 DEBUG:boto:Path: /vclfcn-54/?policy 2024-02-01T10:43:51.319 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2024-02-01T10:43:51.319 DEBUG:boto:Headers: {} 2024-02-01T10:43:51.319 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:43:51.319 DEBUG:boto:Port: 8000 2024-02-01T10:43:51.319 DEBUG:boto:Params: {} 2024-02-01T10:43:51.320 DEBUG:boto:Token: None 2024-02-01T10:43:51.320 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:43:51 GMT /vclfcn-54/?policy 2024-02-01T10:43:51.320 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cTwv+UFaEBaYgw32yBpQkLfC87I= 2024-02-01T10:43:51.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:43:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cTwv+UFaEBaYgw32yBpQkLfC87I=', 'Content-Length': '97'} 2024-02-01T10:43:52.338 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000038f570208f53c89d-0065bb75e7-5350-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:43:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:43:52.339 DEBUG:boto:path=/vclfcn-54/ 2024-02-01T10:43:52.339 DEBUG:boto:auth_path=/vclfcn-54/ 2024-02-01T10:43:52.339 DEBUG:boto:path=/vclfcn-54/?policy 2024-02-01T10:43:52.339 DEBUG:boto:auth_path=/vclfcn-54/?policy 2024-02-01T10:43:52.339 DEBUG:boto:Method: GET 2024-02-01T10:43:52.339 DEBUG:boto:Path: /vclfcn-54/?policy 2024-02-01T10:43:52.339 DEBUG:boto:Data: 2024-02-01T10:43:52.339 DEBUG:boto:Headers: {} 2024-02-01T10:43:52.340 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:43:52.340 DEBUG:boto:Port: 8000 2024-02-01T10:43:52.340 DEBUG:boto:Params: {} 2024-02-01T10:43:52.340 DEBUG:boto:Token: None 2024-02-01T10:43:52.340 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:43:52 GMT /vclfcn-54/?policy 2024-02-01T10:43:52.341 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:H5Gjtj5v6h+hFkdNK+lgZDXetDs= 2024-02-01T10:43:52.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:43:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:H5Gjtj5v6h+hFkdNK+lgZDXetDs=', 'Content-Length': '0'} 2024-02-01T10:43:52.343 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a0bc3d1cc9b1f1cc-0065bb75e8-5350-test-zone2'), ('Content-Type', 'application/json'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '97'), ('Date', 'Thu, 01 Feb 2024 10:43:52 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:43:52.343 DEBUG:boto:path=/vclfcn-55/ 2024-02-01T10:43:52.343 DEBUG:boto:auth_path=/vclfcn-55/ 2024-02-01T10:43:52.343 DEBUG:boto:path=/vclfcn-55/?policy 2024-02-01T10:43:52.343 DEBUG:boto:auth_path=/vclfcn-55/?policy 2024-02-01T10:43:52.343 DEBUG:boto:Method: PUT 2024-02-01T10:43:52.343 DEBUG:boto:Path: /vclfcn-55/?policy 2024-02-01T10:43:52.343 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2024-02-01T10:43:52.344 DEBUG:boto:Headers: {} 2024-02-01T10:43:52.344 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:43:52.344 DEBUG:boto:Port: 8001 2024-02-01T10:43:52.344 DEBUG:boto:Params: {} 2024-02-01T10:43:52.344 DEBUG:boto:Token: None 2024-02-01T10:43:52.344 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:43:52 GMT /vclfcn-55/?policy 2024-02-01T10:43:52.344 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:VkOC+5roGL1ggJqJD8OpfuSR8xo= 2024-02-01T10:43:52.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:43:52 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:VkOC+5roGL1ggJqJD8OpfuSR8xo=', 'Content-Length': '97'} 2024-02-01T10:43:53.364 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000023439675c78845ee-0065bb75e8-5762-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:43:53.365 DEBUG:boto:path=/vclfcn-55/ 2024-02-01T10:43:53.365 DEBUG:boto:auth_path=/vclfcn-55/ 2024-02-01T10:43:53.365 DEBUG:boto:path=/vclfcn-55/?policy 2024-02-01T10:43:53.365 DEBUG:boto:auth_path=/vclfcn-55/?policy 2024-02-01T10:43:53.365 DEBUG:boto:Method: GET 2024-02-01T10:43:53.365 DEBUG:boto:Path: /vclfcn-55/?policy 2024-02-01T10:43:53.365 DEBUG:boto:Data: 2024-02-01T10:43:53.365 DEBUG:boto:Headers: {} 2024-02-01T10:43:53.365 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:43:53.366 DEBUG:boto:Port: 8001 2024-02-01T10:43:53.366 DEBUG:boto:Params: {} 2024-02-01T10:43:53.366 DEBUG:boto:Token: None 2024-02-01T10:43:53.366 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:43:53 GMT /vclfcn-55/?policy 2024-02-01T10:43:53.366 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:MskcIIf53lNlKsVH1ijTl5Ns2E0= 2024-02-01T10:43:53.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:43:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:MskcIIf53lNlKsVH1ijTl5Ns2E0=', 'Content-Length': '0'} 2024-02-01T10:43:53.368 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ecbf6b24e32cacac-0065bb75e9-5762-test-zone3'), ('Content-Type', 'application/json'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '97'), ('Date', 'Thu, 01 Feb 2024 10:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:43:53.368 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_set_bucket_policy ... ok 2024-02-01T10:43:53.369 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=vclfcn-56 2024-02-01T10:43:53.369 DEBUG:boto:path=/vclfcn-56/ 2024-02-01T10:43:53.369 DEBUG:boto:auth_path=/vclfcn-56/ 2024-02-01T10:43:53.369 DEBUG:boto:Method: PUT 2024-02-01T10:43:53.369 DEBUG:boto:Path: /vclfcn-56/ 2024-02-01T10:43:53.369 DEBUG:boto:Data: 2024-02-01T10:43:53.370 DEBUG:boto:Headers: {} 2024-02-01T10:43:53.370 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:43:53.370 DEBUG:boto:Port: 8000 2024-02-01T10:43:53.370 DEBUG:boto:Params: {} 2024-02-01T10:43:53.370 DEBUG:boto:Token: None 2024-02-01T10:43:53.370 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:43:53 GMT /vclfcn-56/ 2024-02-01T10:43:53.370 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:xrYvyd+PkT2r2Bnroj4gw/eEbLc= 2024-02-01T10:43:53.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:43:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:xrYvyd+PkT2r2Bnroj4gw/eEbLc=', 'Content-Length': '0'} 2024-02-01T10:43:53.407 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a8d13218591d7d49-0065bb75e9-5765-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:43:53.407 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=vclfcn-57 2024-02-01T10:43:53.407 DEBUG:boto:path=/vclfcn-57/ 2024-02-01T10:43:53.407 DEBUG:boto:auth_path=/vclfcn-57/ 2024-02-01T10:43:53.407 DEBUG:boto:Method: PUT 2024-02-01T10:43:53.408 DEBUG:boto:Path: /vclfcn-57/ 2024-02-01T10:43:53.408 DEBUG:boto:Data: 2024-02-01T10:43:53.408 DEBUG:boto:Headers: {} 2024-02-01T10:43:53.408 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:43:53.408 DEBUG:boto:Port: 8000 2024-02-01T10:43:53.408 DEBUG:boto:Params: {} 2024-02-01T10:43:53.408 DEBUG:boto:Token: None 2024-02-01T10:43:53.408 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:43:53 GMT /vclfcn-57/ 2024-02-01T10:43:53.409 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:4P+PHxbJd7pefd7KYam8xA4zmlk= 2024-02-01T10:43:53.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:43:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:4P+PHxbJd7pefd7KYam8xA4zmlk=', 'Content-Length': '0'} 2024-02-01T10:43:53.479 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ebd5a6d10022ef74-0065bb75e9-5350-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:43:53.479 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=vclfcn-58 2024-02-01T10:43:53.479 DEBUG:boto:path=/vclfcn-58/ 2024-02-01T10:43:53.479 DEBUG:boto:auth_path=/vclfcn-58/ 2024-02-01T10:43:53.479 DEBUG:boto:Method: PUT 2024-02-01T10:43:53.479 DEBUG:boto:Path: /vclfcn-58/ 2024-02-01T10:43:53.479 DEBUG:boto:Data: 2024-02-01T10:43:53.480 DEBUG:boto:Headers: {} 2024-02-01T10:43:53.480 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:43:53.480 DEBUG:boto:Port: 8001 2024-02-01T10:43:53.480 DEBUG:boto:Params: {} 2024-02-01T10:43:53.480 DEBUG:boto:Token: None 2024-02-01T10:43:53.480 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:43:53 GMT /vclfcn-58/ 2024-02-01T10:43:53.480 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gnLYd96wCt9cfHMe0Ubs0b3ltNs= 2024-02-01T10:43:53.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:43:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gnLYd96wCt9cfHMe0Ubs0b3ltNs=', 'Content-Length': '0'} 2024-02-01T10:43:53.544 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000be4dd24ed520083a-0065bb75e9-5762-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:43:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:43:53.544 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:43:53.544 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:43:53.545 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:43:53.584 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:43:53.584 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:43:53.640 INFO:teuthology.orchestra.run.smithi148.stderr:No --period given, using current period=f2e6c290-e7aa-4279-b77e-15403b18d61f 2024-02-01T10:43:53.665 INFO:teuthology.orchestra.run.smithi148.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1706784233.523070_244.1","last_update":"2024-02-01T10:43:53.523070Z"},{"marker":"","last_update":"0.000000"}] 2024-02-01T10:43:53.665 DEBUG:rgw_multi.tests:master meta markers={0: '', 1: '', 2: '1_1706784233.523070_244.1', 3: ''} 2024-02-01T10:43:53.665 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-02-01T10:43:53.665 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:43:53.666 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:43:53.666 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:43:53.708 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:43:53.708 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:43:53.764 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:43:53.762+0000 7fd2d07aa980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:43:53.764 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:43:53.762+0000 7fd2d07aa980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:43:53.788 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783978.681171_139.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:39:38.681171Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:43:53Z"} 2024-02-01T10:43:53.789 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783978.681171_139.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:39:38.681171Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:43:53Z"} 2024-02-01T10:43:53.789 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706783978.681171_139.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:39:38.681171Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:43:53.789 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1706784233.523070_244.1', 3: ''} 2024-02-01T10:43:53.789 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1706783978.681171_139.1', 3: ''} 2024-02-01T10:43:53.789 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1706784233.523070_244.1 target=1_1706783978.681171_139.1 2024-02-01T10:43:58.790 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:43:58.791 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:43:58.791 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:43:58.834 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:43:58.834 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:43:58.889 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:43:58.886+0000 7faf03629980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:43:58.890 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:43:58.890+0000 7faf03629980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:43:58.914 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783978.681171_139.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:39:38.681171Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:43:58Z"} 2024-02-01T10:43:58.914 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783978.681171_139.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:39:38.681171Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:43:58Z"} 2024-02-01T10:43:58.914 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706783978.681171_139.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:39:38.681171Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:43:58.914 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1706784233.523070_244.1', 3: ''} 2024-02-01T10:43:58.914 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1706783978.681171_139.1', 3: ''} 2024-02-01T10:43:58.915 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1706784233.523070_244.1 target=1_1706783978.681171_139.1 2024-02-01T10:44:03.915 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:44:03.916 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:44:03.916 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:44:03.958 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:44:03.959 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:44:04.013 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:44:04.010+0000 7f9c9822e980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:44:04.014 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:44:04.014+0000 7f9c9822e980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:44:04.038 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783978.681171_139.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:39:38.681171Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:44:04Z"} 2024-02-01T10:44:04.038 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783978.681171_139.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:39:38.681171Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:44:04Z"} 2024-02-01T10:44:04.038 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706783978.681171_139.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:39:38.681171Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:44:04.038 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1706784233.523070_244.1', 3: ''} 2024-02-01T10:44:04.038 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1706783978.681171_139.1', 3: ''} 2024-02-01T10:44:04.039 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1706784233.523070_244.1 target=1_1706783978.681171_139.1 2024-02-01T10:44:09.040 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:44:09.040 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:44:09.040 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:44:09.083 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:44:09.085 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:44:09.140 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:44:09.134+0000 7f96aa465980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:44:09.143 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:44:09.138+0000 7f96aa465980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:44:09.162 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783978.681171_139.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:39:38.681171Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:44:09Z"} 2024-02-01T10:44:09.162 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783978.681171_139.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:39:38.681171Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:44:09Z"} 2024-02-01T10:44:09.162 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706783978.681171_139.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:39:38.681171Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:44:09.163 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1706784233.523070_244.1', 3: ''} 2024-02-01T10:44:09.163 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1706783978.681171_139.1', 3: ''} 2024-02-01T10:44:09.163 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1706784233.523070_244.1 target=1_1706783978.681171_139.1 2024-02-01T10:44:14.165 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:44:14.165 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:44:14.165 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:44:14.208 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:44:14.209 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:44:14.263 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:44:14.262+0000 7f209b7cc980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:44:14.264 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:44:14.262+0000 7f209b7cc980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:44:14.288 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783978.681171_139.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:39:38.681171Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:44:14Z"} 2024-02-01T10:44:14.288 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783978.681171_139.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:39:38.681171Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:44:14Z"} 2024-02-01T10:44:14.288 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706783978.681171_139.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:39:38.681171Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:44:14.288 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1706784233.523070_244.1', 3: ''} 2024-02-01T10:44:14.289 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1706783978.681171_139.1', 3: ''} 2024-02-01T10:44:14.289 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1706784233.523070_244.1 target=1_1706783978.681171_139.1 2024-02-01T10:44:19.289 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:44:19.290 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:44:19.290 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:44:19.334 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:44:19.334 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:44:19.389 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:44:19.386+0000 7f77f7f68980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:44:19.389 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:44:19.386+0000 7f77f7f68980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:44:19.414 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783978.681171_139.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:39:38.681171Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:44:19Z"} 2024-02-01T10:44:19.414 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783978.681171_139.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:39:38.681171Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:44:19Z"} 2024-02-01T10:44:19.414 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706783978.681171_139.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:39:38.681171Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:44:19.414 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1706784233.523070_244.1', 3: ''} 2024-02-01T10:44:19.415 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1706783978.681171_139.1', 3: ''} 2024-02-01T10:44:19.415 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1706784233.523070_244.1 target=1_1706783978.681171_139.1 2024-02-01T10:44:22.127 INFO:tasks.rgw.c1.client.1.smithi148.stdout:2024-02-01T10:44:22.128+0000 7f8357755640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-02-01T10:44:24.416 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:44:24.416 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:44:24.416 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:44:24.460 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:44:24.461 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:44:24.520 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:44:24.518+0000 7f4ab6f02980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:44:24.521 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:44:24.518+0000 7f4ab6f02980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:44:24.545 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783978.681171_139.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:39:38.681171Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:44:24Z"} 2024-02-01T10:44:24.545 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783978.681171_139.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:39:38.681171Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:44:24Z"} 2024-02-01T10:44:24.545 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706783978.681171_139.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:39:38.681171Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:44:24.546 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1706784233.523070_244.1', 3: ''} 2024-02-01T10:44:24.546 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1706783978.681171_139.1', 3: ''} 2024-02-01T10:44:24.546 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1706784233.523070_244.1 target=1_1706783978.681171_139.1 2024-02-01T10:44:29.547 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:44:29.547 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:44:29.547 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:44:29.591 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:44:29.591 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:44:29.646 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:44:29.646+0000 7eff12158980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:44:29.647 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:44:29.646+0000 7eff12158980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:44:29.671 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783978.681171_139.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:39:38.681171Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:44:29Z"} 2024-02-01T10:44:29.672 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783978.681171_139.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:39:38.681171Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:44:29Z"} 2024-02-01T10:44:29.672 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706783978.681171_139.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:39:38.681171Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:44:29.672 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1706784233.523070_244.1', 3: ''} 2024-02-01T10:44:29.673 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1706783978.681171_139.1', 3: ''} 2024-02-01T10:44:29.673 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1706784233.523070_244.1 target=1_1706783978.681171_139.1 2024-02-01T10:44:34.674 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:44:34.674 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:44:34.674 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:44:34.717 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:44:34.717 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:44:34.772 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:44:34.770+0000 7efc19bb5980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:44:34.772 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:44:34.770+0000 7efc19bb5980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:44:34.796 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783978.681171_139.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:39:38.681171Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:44:34Z"} 2024-02-01T10:44:34.797 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783978.681171_139.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:39:38.681171Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:44:34Z"} 2024-02-01T10:44:34.797 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706783978.681171_139.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:39:38.681171Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:44:34.797 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1706784233.523070_244.1', 3: ''} 2024-02-01T10:44:34.797 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1706783978.681171_139.1', 3: ''} 2024-02-01T10:44:34.797 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1706784233.523070_244.1 target=1_1706783978.681171_139.1 2024-02-01T10:44:39.798 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:44:39.798 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:44:39.798 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:44:39.842 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:44:39.843 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:44:39.900 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:44:39.897+0000 7f235d35e980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:44:39.900 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:44:39.897+0000 7f235d35e980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:44:39.925 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783978.681171_139.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:39:38.681171Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:44:39Z"} 2024-02-01T10:44:39.925 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783978.681171_139.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:39:38.681171Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:44:39Z"} 2024-02-01T10:44:39.925 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706783978.681171_139.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:39:38.681171Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:44:39.925 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1706784233.523070_244.1', 3: ''} 2024-02-01T10:44:39.926 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1706783978.681171_139.1', 3: ''} 2024-02-01T10:44:39.926 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1706784233.523070_244.1 target=1_1706783978.681171_139.1 2024-02-01T10:44:44.927 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:44:44.927 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:44:44.928 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:44:44.972 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:44:44.975 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:44:45.027 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:44:45.025+0000 7f906dc81980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:44:45.028 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:44:45.025+0000 7f906dc81980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:44:45.052 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783978.681171_139.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:39:38.681171Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:44:45Z"} 2024-02-01T10:44:45.052 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783978.681171_139.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:39:38.681171Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:44:45Z"} 2024-02-01T10:44:45.052 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706783978.681171_139.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:39:38.681171Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:44:45.052 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1706784233.523070_244.1', 3: ''} 2024-02-01T10:44:45.053 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1706783978.681171_139.1', 3: ''} 2024-02-01T10:44:45.053 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1706784233.523070_244.1 target=1_1706783978.681171_139.1 2024-02-01T10:44:50.053 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:44:50.054 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:44:50.054 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:44:50.096 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:44:50.097 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:44:50.151 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:44:50.149+0000 7f7725afe980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:44:50.152 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:44:50.149+0000 7f7725afe980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:44:50.176 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783978.681171_139.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:39:38.681171Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:44:50Z"} 2024-02-01T10:44:50.176 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783978.681171_139.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:39:38.681171Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:44:50Z"} 2024-02-01T10:44:50.176 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706783978.681171_139.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:39:38.681171Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:44:50.176 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1706784233.523070_244.1', 3: ''} 2024-02-01T10:44:50.176 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1706783978.681171_139.1', 3: ''} 2024-02-01T10:44:50.177 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1706784233.523070_244.1 target=1_1706783978.681171_139.1 2024-02-01T10:44:55.178 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:44:55.178 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:44:55.179 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:44:55.221 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:44:55.221 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:44:55.275 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:44:55.273+0000 7fdbbe2eb980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:44:55.276 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:44:55.273+0000 7fdbbe2eb980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:44:55.300 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783978.681171_139.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:39:38.681171Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:44:55Z"} 2024-02-01T10:44:55.300 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783978.681171_139.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:39:38.681171Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:44:55Z"} 2024-02-01T10:44:55.301 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706783978.681171_139.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:39:38.681171Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:44:55.301 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1706784233.523070_244.1', 3: ''} 2024-02-01T10:44:55.301 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1706783978.681171_139.1', 3: ''} 2024-02-01T10:44:55.301 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1706784233.523070_244.1 target=1_1706783978.681171_139.1 2024-02-01T10:45:00.302 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:45:00.302 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:45:00.303 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:45:00.346 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:45:00.346 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:45:00.400 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:45:00.397+0000 7fc1984fc980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:45:00.401 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:45:00.397+0000 7fc1984fc980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:45:00.425 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783978.681171_139.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:39:38.681171Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:45:00Z"} 2024-02-01T10:45:00.426 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783978.681171_139.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:39:38.681171Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:45:00Z"} 2024-02-01T10:45:00.426 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706783978.681171_139.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:39:38.681171Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:45:00.426 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1706784233.523070_244.1', 3: ''} 2024-02-01T10:45:00.426 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1706783978.681171_139.1', 3: ''} 2024-02-01T10:45:00.426 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1706784233.523070_244.1 target=1_1706783978.681171_139.1 2024-02-01T10:45:05.428 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:45:05.428 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:45:05.428 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:45:05.472 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:45:05.472 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:45:05.530 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:45:05.529+0000 7f87ee01e980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:45:05.531 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:45:05.529+0000 7f87ee01e980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:45:05.555 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783978.681171_139.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:39:38.681171Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:45:05Z"} 2024-02-01T10:45:05.555 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783978.681171_139.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:39:38.681171Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:45:05Z"} 2024-02-01T10:45:05.555 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706783978.681171_139.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:39:38.681171Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:45:05.555 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1706784233.523070_244.1', 3: ''} 2024-02-01T10:45:05.556 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1706783978.681171_139.1', 3: ''} 2024-02-01T10:45:05.556 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1706784233.523070_244.1 target=1_1706783978.681171_139.1 2024-02-01T10:45:10.557 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:45:10.557 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:45:10.557 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:45:10.600 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:45:10.600 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:45:10.657 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:45:10.653+0000 7f35914b5980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:45:10.658 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:45:10.657+0000 7f35914b5980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:45:10.682 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783978.681171_139.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:39:38.681171Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:45:10Z"} 2024-02-01T10:45:10.682 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783978.681171_139.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:39:38.681171Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:45:10Z"} 2024-02-01T10:45:10.682 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706783978.681171_139.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:39:38.681171Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:45:10.682 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1706784233.523070_244.1', 3: ''} 2024-02-01T10:45:10.682 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1706783978.681171_139.1', 3: ''} 2024-02-01T10:45:10.683 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1706784233.523070_244.1 target=1_1706783978.681171_139.1 2024-02-01T10:45:15.684 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:45:15.685 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:45:15.685 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:45:15.727 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:45:15.727 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:45:15.783 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:45:15.781+0000 7fe3546da980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:45:15.784 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:45:15.781+0000 7fe3546da980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:45:15.808 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783978.681171_139.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:39:38.681171Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:45:15Z"} 2024-02-01T10:45:15.809 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783978.681171_139.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:39:38.681171Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:45:15Z"} 2024-02-01T10:45:15.809 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706783978.681171_139.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:39:38.681171Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:45:15.809 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1706784233.523070_244.1', 3: ''} 2024-02-01T10:45:15.809 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1706783978.681171_139.1', 3: ''} 2024-02-01T10:45:15.809 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1706784233.523070_244.1 target=1_1706783978.681171_139.1 2024-02-01T10:45:20.811 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:45:20.811 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:45:20.811 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:45:20.854 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:45:20.854 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:45:20.908 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:45:20.905+0000 7fe5f75d3980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:45:20.909 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:45:20.905+0000 7fe5f75d3980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:45:20.934 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783978.681171_139.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:39:38.681171Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:45:20Z"} 2024-02-01T10:45:20.934 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783978.681171_139.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:39:38.681171Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:45:20Z"} 2024-02-01T10:45:20.934 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706783978.681171_139.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:39:38.681171Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:45:20.934 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1706784233.523070_244.1', 3: ''} 2024-02-01T10:45:20.934 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1706783978.681171_139.1', 3: ''} 2024-02-01T10:45:20.934 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1706784233.523070_244.1 target=1_1706783978.681171_139.1 2024-02-01T10:45:25.935 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:45:25.935 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:45:25.936 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:45:25.978 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:45:25.978 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:45:26.032 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:45:26.029+0000 7f8f5f662980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:45:26.033 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:45:26.029+0000 7f8f5f662980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:45:26.057 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783978.681171_139.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:39:38.681171Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:45:26Z"} 2024-02-01T10:45:26.057 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783978.681171_139.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:39:38.681171Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:45:26Z"} 2024-02-01T10:45:26.057 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706783978.681171_139.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:39:38.681171Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:45:26.058 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1706784233.523070_244.1', 3: ''} 2024-02-01T10:45:26.058 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1706783978.681171_139.1', 3: ''} 2024-02-01T10:45:26.058 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1706784233.523070_244.1 target=1_1706783978.681171_139.1 2024-02-01T10:45:31.059 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:45:31.059 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:45:31.060 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:45:31.102 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:45:31.102 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:45:31.159 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:45:31.157+0000 7fa749568980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:45:31.159 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:45:31.157+0000 7fa749568980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:45:31.184 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783978.681171_139.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:39:38.681171Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:45:31Z"} 2024-02-01T10:45:31.184 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783978.681171_139.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:39:38.681171Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:45:31Z"} 2024-02-01T10:45:31.185 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706783978.681171_139.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:39:38.681171Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:45:31.185 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1706784233.523070_244.1', 3: ''} 2024-02-01T10:45:31.185 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1706783978.681171_139.1', 3: ''} 2024-02-01T10:45:31.185 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1706784233.523070_244.1 target=1_1706783978.681171_139.1 2024-02-01T10:45:36.186 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:45:36.186 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:45:36.186 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:45:36.230 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:45:36.230 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:45:36.288 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:45:36.284+0000 7f19ff5ad980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:45:36.288 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:45:36.284+0000 7f19ff5ad980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:45:36.312 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783978.681171_139.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:39:38.681171Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:45:36Z"} 2024-02-01T10:45:36.312 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783978.681171_139.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:39:38.681171Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:45:36Z"} 2024-02-01T10:45:36.312 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706783978.681171_139.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:39:38.681171Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:45:36.313 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1706784233.523070_244.1', 3: ''} 2024-02-01T10:45:36.313 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1706783978.681171_139.1', 3: ''} 2024-02-01T10:45:36.313 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1706784233.523070_244.1 target=1_1706783978.681171_139.1 2024-02-01T10:45:41.314 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:45:41.314 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:45:41.315 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:45:41.357 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:45:41.358 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:45:41.413 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:45:41.408+0000 7fb7123dc980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:45:41.415 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:45:41.412+0000 7fb7123dc980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:45:41.438 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783978.681171_139.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:39:38.681171Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:45:41Z"} 2024-02-01T10:45:41.438 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783978.681171_139.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:39:38.681171Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:45:41Z"} 2024-02-01T10:45:41.438 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706783978.681171_139.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:39:38.681171Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:45:41.439 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1706784233.523070_244.1', 3: ''} 2024-02-01T10:45:41.439 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1706783978.681171_139.1', 3: ''} 2024-02-01T10:45:41.439 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1706784233.523070_244.1 target=1_1706783978.681171_139.1 2024-02-01T10:45:46.439 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:45:46.440 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:45:46.440 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:45:46.485 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:45:46.485 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:45:46.542 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:45:46.544+0000 7f5d181ef980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:45:46.543 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:45:46.544+0000 7f5d181ef980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:45:46.568 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783978.681171_139.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:39:38.681171Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:45:46Z"} 2024-02-01T10:45:46.569 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783978.681171_139.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:39:38.681171Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:45:46Z"} 2024-02-01T10:45:46.569 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706783978.681171_139.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:39:38.681171Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:45:46.569 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1706784233.523070_244.1', 3: ''} 2024-02-01T10:45:46.569 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1706783978.681171_139.1', 3: ''} 2024-02-01T10:45:46.569 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1706784233.523070_244.1 target=1_1706783978.681171_139.1 2024-02-01T10:45:51.571 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:45:51.571 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:45:51.571 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:45:51.613 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:45:51.614 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:45:51.670 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:45:51.672+0000 7f86a249f980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:45:51.672 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:45:51.672+0000 7f86a249f980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:45:51.696 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783978.681171_139.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:39:38.681171Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:45:51Z"} 2024-02-01T10:45:51.696 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706783978.681171_139.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:39:38.681171Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:45:51Z"} 2024-02-01T10:45:51.696 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706783978.681171_139.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:39:38.681171Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:45:51.696 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1706784233.523070_244.1', 3: ''} 2024-02-01T10:45:51.696 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1706783978.681171_139.1', 3: ''} 2024-02-01T10:45:51.697 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1706784233.523070_244.1 target=1_1706783978.681171_139.1 2024-02-01T10:45:56.697 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:45:56.697 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:45:56.698 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:45:56.748 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:45:56.748 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:45:56.804 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:45:56.804+0000 7fc3de6fe980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:45:56.805 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:45:56.804+0000 7fc3de6fe980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:45:56.829 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706784231.187391_209.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:43:51.187391Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:45:56Z"} 2024-02-01T10:45:56.829 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706784231.187391_209.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:43:51.187391Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:45:56Z"} 2024-02-01T10:45:56.829 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706784231.187391_209.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:43:51.187391Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:45:56.830 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1706784233.523070_244.1', 3: ''} 2024-02-01T10:45:56.830 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1706784231.187391_209.1', 3: ''} 2024-02-01T10:45:56.830 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1706784233.523070_244.1 target=1_1706784231.187391_209.1 2024-02-01T10:46:01.831 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:46:01.831 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:46:01.831 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:46:01.875 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:46:01.875 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:46:01.944 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:46:01.944+0000 7f9643019980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:46:01.944 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:46:01.944+0000 7f9643019980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:46:01.968 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706784231.198935_212.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:43:51.198935Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:46:01Z"} 2024-02-01T10:46:01.969 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706784231.198935_212.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:43:51.198935Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:46:01Z"} 2024-02-01T10:46:01.969 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706784231.198935_212.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:43:51.198935Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:46:01.969 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1706784233.523070_244.1', 3: ''} 2024-02-01T10:46:01.969 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1706784231.198935_212.1', 3: ''} 2024-02-01T10:46:01.969 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1706784233.523070_244.1 target=1_1706784231.198935_212.1 2024-02-01T10:46:06.970 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:46:06.971 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:46:06.971 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:46:07.014 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:46:07.015 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:46:07.070 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:46:07.072+0000 7f5345bc5980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:46:07.071 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:46:07.072+0000 7f5345bc5980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:46:07.095 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706784231.231911_213.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:43:51.231911Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:46:07Z"} 2024-02-01T10:46:07.095 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706784231.231911_213.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:43:51.231911Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:46:07Z"} 2024-02-01T10:46:07.096 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706784231.231911_213.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:43:51.231911Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:46:07.096 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1706784233.523070_244.1', 3: ''} 2024-02-01T10:46:07.096 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1706784231.231911_213.1', 3: ''} 2024-02-01T10:46:07.099 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1706784233.523070_244.1 target=1_1706784231.231911_213.1 2024-02-01T10:46:12.100 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:46:12.101 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:46:12.101 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:46:12.144 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:46:12.144 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:46:12.205 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:46:12.204+0000 7f9ad245f980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:46:12.205 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:46:12.204+0000 7f9ad245f980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:46:12.228 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706784233.523070_244.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:43:53.523070Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:46:12Z"} 2024-02-01T10:46:12.228 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706784233.523070_244.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:43:53.523070Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:46:12Z"} 2024-02-01T10:46:12.228 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706784233.523070_244.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:43:53.523070Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:46:12.228 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1706784233.523070_244.1', 3: ''} 2024-02-01T10:46:12.228 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1706784233.523070_244.1', 3: ''} 2024-02-01T10:46:12.228 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-02-01T10:46:12.229 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-02-01T10:46:12.229 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:46:12.229 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:46:12.229 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:46:12.271 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:46:12.271 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:46:12.326 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:46:12.324+0000 7f8242c0a980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:46:12.327 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:46:12.328+0000 7f8242c0a980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:46:12.350 INFO:teuthology.orchestra.run.smithi148.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706784233.523070_244.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:43:53.523070Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:46:12Z"} 2024-02-01T10:46:12.350 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706784233.523070_244.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:43:53.523070Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:46:12Z"} 2024-02-01T10:46:12.350 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706784233.523070_244.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:43:53.523070Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:46:12.350 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1706784233.523070_244.1', 3: ''} 2024-02-01T10:46:12.351 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1706784233.523070_244.1', 3: ''} 2024-02-01T10:46:12.351 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-02-01T10:46:12.351 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'vclfcn-56', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:46:12.351 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'vclfcn-56', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:46:12.351 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket vclfcn-56 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-02-01T10:46:12.426 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:46:12.426 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:46:12.531 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'vclfcn-57', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:46:12.531 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'vclfcn-57', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:46:12.531 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket vclfcn-57 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-02-01T10:46:12.613 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:46:12.635 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:46:12.723 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'vclfcn-58', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:46:12.723 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'vclfcn-58', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:46:12.723 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket vclfcn-58 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-02-01T10:46:12.807 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:46:12.807 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:46:12.916 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['log', 'list', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:46:12.916 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'log', 'list', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:46:12.916 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 log list --rgw-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:46:12.995 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:46:12.996 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:46:13.098 INFO:teuthology.orchestra.run.smithi148.stdout:["obj_delete_at_hint.0000000078","meta.history","bucket.full-sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31:6","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-47:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.4:3","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19:8","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34:1","sync.error-log.23","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20:7","obj_delete_at_hint.0000000070","obj_delete_at_hint.0000000104","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35:9","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-49:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5625.1:4","obj_delete_at_hint.0000000026","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34:5","bucket.full-sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-49:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5625.1","datalog.sync-status.shard.01fb6d04-f0ef-49b2-9ebc-c1b530925485.0.retry","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-46:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.3:1","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32:8","sync.error-log.8","obj_delete_at_hint.0000000028","bucket.full-sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34:1","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-36:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.45:4","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-47:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.4:10","bucket.full-sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20:6","data_log.2.wf3XSZiHoip.0","bucket.full-sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26:2","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29:5","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23:10","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31:8","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19:1","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-36:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.45:3","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-38:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.47:5","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28:10","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-46:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.3:5","obj_delete_at_hint.0000000040","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33:4","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-48:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.5:2","datalog.sync-status.shard.01fb6d04-f0ef-49b2-9ebc-c1b530925485.1","sync.error-log.6","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-48:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.5:5","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-38:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.47:9","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22:3","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32:9","bucket.full-sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25:8","obj_delete_at_hint.0000000015","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23:5","bucket.full-sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19","obj_delete_at_hint.0000000069","obj_delete_at_hint.0000000095","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-46:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.3:10","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19:2","obj_delete_at_hint.0000000003","obj_delete_at_hint.0000000047","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19:0","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30:6","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28:4","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25:2","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32:1","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32:8","data_log.1.dfx1qhLTgoD.0","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-46:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.3:10","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23:8","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23:1","obj_delete_at_hint.0000000035","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27:9","obj_delete_at_hint.0000000037","bucket.full-sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-47:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.4","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30:7","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33:8","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31:3","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29:9","bucket.full-sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20:10","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31:1","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29:4","obj_delete_at_hint.0000000024","obj_delete_at_hint.0000000000","obj_delete_at_hint.0000000031","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25:7","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26:9","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19:1","bucket.full-sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33","data_log.3","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32:9","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34:5","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29:0","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35:1","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19:7","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-46:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.3:3","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20:4","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22:1","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19:4","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-48:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.5:0","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27:9","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28:6","bucket.full-sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28:8","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35:1","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23:10","data_loggenerations_metadata","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32:2","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-46:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.3:6","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-47:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.4:8","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30:0","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22:0","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19:4","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-36:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.45:8","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-36:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.45:1","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24:0","obj_delete_at_hint.0000000076","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28:5","obj_delete_at_hint.0000000113","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34:3","obj_delete_at_hint.0000000005","obj_delete_at_hint.0000000011","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28:0","obj_delete_at_hint.0000000058","obj_delete_at_hint.0000000052","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-48:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.5:8","sync.error-log.21","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-47:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.4:7","bucket.full-sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31","bucket.full-sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23:3","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27:10","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-37:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.46:9","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24:1","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24:8","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28:10","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-38:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.47:7","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25:6","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34:7","obj_delete_at_hint.0000000088","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-48:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.5:3","bucket.full-sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23","obj_delete_at_hint.0000000080","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-49:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5625.1:1","obj_delete_at_hint.0000000090","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25:0","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-49:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5625.1:5","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21:0","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22:10","obj_delete_at_hint.0000000110","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21:4","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34:0","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20:2","obj_delete_at_hint.0000000096","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34:8","obj_delete_at_hint.0000000087","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23:4","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32:3","obj_delete_at_hint.0000000008","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35:3","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28:6","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21:10","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-46:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.3:6","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20:3","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33:10","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35:4","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34:9","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23:9","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-47:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.4:6","obj_delete_at_hint.0000000006","obj_delete_at_hint.0000000029","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25:4","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22:7","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34:2","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20:4","obj_delete_at_hint.0000000089","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33:9","obj_delete_at_hint.0000000022","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35:8","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-46:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.3:8","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33:10","bucket.full-sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-48:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.5:4","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33:1","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31:10","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26:0","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27:8","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-46:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.3:2","bilog.trim","obj_delete_at_hint.0000000016","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19:6","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27:1","obj_delete_at_hint.0000000041","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24:3","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-38:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.47:2","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26:2","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-37:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.46:10","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33:6","obj_delete_at_hint.0000000018","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32:4","obj_delete_at_hint.0000000092","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-48:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.5:3","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21:1","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27:7","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20:5","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-36:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.45:9","sync.error-log.4","datalog.sync-status.shard.2ea581c3-3f48-49d7-a989-68024c5b8eac.3","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-37:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.46:1","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34:4","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35:7","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35:9","obj_delete_at_hint.0000000014","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27:8","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-49:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5625.1:8","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-38:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.47:3","obj_delete_at_hint.0000000112","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-49:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5625.1:9","obj_delete_at_hint.0000000007","obj_delete_at_hint.0000000021","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25:9","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32:10","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20:1","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24:5","sync.error-log.16","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20:2","obj_delete_at_hint.0000000064","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19:5","datalog.sync-status.shard.01fb6d04-f0ef-49b2-9ebc-c1b530925485.3","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25:10","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33:7","obj_delete_at_hint.0000000071","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22:0","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-47:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.4:5","obj_delete_at_hint.0000000074","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33:2","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27:10","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-49:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5625.1:10","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21:9","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26:9","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-38:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.47:6","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26:8","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20:6","obj_delete_at_hint.0000000081","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28:4","obj_delete_at_hint.0000000009","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-38:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.47:0","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24:0","obj_delete_at_hint.0000000121","data_log.1","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20:8","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21:8","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-37:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.46:4","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-38:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.47:9","sync.error-log.17","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20:9","obj_delete_at_hint.0000000125","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30:6","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22:10","obj_delete_at_hint.0000000082","bucket.full-sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-37:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.46","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29:5","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32:6","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22:8","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27:0","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32:0","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26:10","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25:6","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31:7","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-38:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.47:2","obj_delete_at_hint.0000000105","obj_delete_at_hint.0000000059","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-36:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.45:6","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23:1","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26:10","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-37:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.46:6","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-47:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.4:2","obj_delete_at_hint.0000000077","bucket.full-sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-47:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.4","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35:2","obj_delete_at_hint.0000000032","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24:10","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31:8","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-38:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.47:0","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-36:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.45:2","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24:7","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20:9","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33:5","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21:7","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-48:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.5:10","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27:6","sync.error-log.24","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19:9","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-38:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.47:4","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-38:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.47:5","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31:2","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19:6","bucket.full-sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32:10","obj_delete_at_hint.0000000053","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-36:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.45:6","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31:1","obj_delete_at_hint.0000000091","bucket.full-sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-48:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.5","bucket.full-sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-38:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.47","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35:0","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-37:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.46:4","obj_delete_at_hint.0000000065","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-46:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.3:0","obj_delete_at_hint.0000000083","obj_delete_at_hint.0000000010","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-38:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.47:1","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31:10","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-49:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5625.1:3","bucket.full-sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-47:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.4:2","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22:4","sync.error-log.7","sync.error-log.3","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-47:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.4:3","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34:3","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23:6","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-46:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.3:7","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29:8","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30:7","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-46:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.3:5","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20:8","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31:5","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22:7","mdlog.sync-status.shard.2","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-48:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.5:0","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33:3","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35:3","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35:8","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23:4","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22:2","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-38:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.47:6","meta.log.f2e6c290-e7aa-4279-b77e-15403b18d61f.2","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-36:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.45:10","obj_delete_at_hint.0000000045","obj_delete_at_hint.0000000002","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22:4","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20:1","sync.error-log.12","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23:6","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34:2","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26:1","obj_delete_at_hint.0000000116","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-49:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5625.1:3","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-37:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.46:5","sync.error-log.25","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32:4","obj_delete_at_hint.0000000034","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-48:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.5:7","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26:1","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-47:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.4:8","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30:10","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-49:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5625.1:0","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-46:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.3:2","obj_delete_at_hint.0000000101","sync.error-log.13","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25:3","mdlog.sync-status.shard.1","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34:4","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31:7","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29:6","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-49:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5625.1:2","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24:9","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21:7","obj_delete_at_hint.0000000079","obj_delete_at_hint.0000000049","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28:1","obj_delete_at_hint.0000000117","obj_delete_at_hint.0000000044","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-46:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.3:0","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21:5","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29:4","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32:1","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34:9","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32:7","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30:1","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24:6","sync.error-log.14","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-38:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.47:1","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33:5","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25:8","obj_delete_at_hint.0000000066","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31:4","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29:0","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35:2","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24:3","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-37:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.46:9","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28:9","obj_delete_at_hint.0000000068","obj_delete_at_hint.0000000085","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29:2","bucket.full-sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27:3","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23:7","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19:2","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21:6","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30:3","obj_delete_at_hint.0000000073","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24:10","bucket.full-sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20","obj_delete_at_hint.0000000038","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21:5","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27:7","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28:3","bucket.full-sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22","sync.error-log.10","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35:5","obj_delete_at_hint.0000000118","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-38:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.47:8","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27:0","obj_delete_at_hint.0000000036","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-36:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.45:0","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22:9","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32:6","obj_delete_at_hint.0000000103","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26:4","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23:3","mdlog.sync-status.shard.3","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35:7","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-49:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5625.1:7","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35:10","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31:9","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25:4","obj_delete_at_hint.0000000119","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31:4","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-36:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.45:9","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34:6","obj_delete_at_hint.0000000098","datalog.sync-status.shard.01fb6d04-f0ef-49b2-9ebc-c1b530925485.2.retry","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-37:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.46:2","datalog.sync-status.shard.01fb6d04-f0ef-49b2-9ebc-c1b530925485.3.retry","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25:10","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30:10","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30:9","mdlog.sync-status.shard.0","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32:5","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-46:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.3:4","obj_delete_at_hint.0000000027","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20:7","mdlog.sync-status","obj_delete_at_hint.0000000019","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-47:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.4:1","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-36:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.45:7","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19:10","obj_delete_at_hint.0000000039","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31:6","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25:7","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-47:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.4:6","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31:2","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-49:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5625.1:5","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25:2","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27:3","obj_delete_at_hint.0000000100","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26:5","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30:5","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-48:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.5:2","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33:4","obj_delete_at_hint.0000000093","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34:0","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20:10","bucket.full-sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27","bucket.full-sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-37:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.46","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30:1","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27:4","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-47:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.4:5","obj_delete_at_hint.0000000004","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-47:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.4:0","obj_delete_at_hint.0000000063","obj_delete_at_hint.0000000122","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-49:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5625.1:6","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29:10","obj_delete_at_hint.0000000057","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26:6","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22:3","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22:9","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-48:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.5:9","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-46:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.3:9","datalog.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485","obj_delete_at_hint.0000000054","obj_delete_at_hint.0000000114","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26:3","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22:2","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28:7","sync.error-log.2","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-47:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.4:9","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29:7","data_log.0","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25:9","bucket.full-sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-38:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.47:4","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23:0","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26:6","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27:6","bucket.full-sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-36:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.45","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24:7","obj_delete_at_hint.0000000012","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32:2","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24:6","bucket.full-sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33:0","obj_delete_at_hint.0000000084","obj_delete_at_hint.0000000043","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27:5","bucket.full-sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-46:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.3","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26:7","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24:8","bucket.full-sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27","sync.error-log.11","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-49:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5625.1:8","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20:0","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-48:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.5:6","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24:9","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28:2","bucket.full-sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-36:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.45","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-48:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.5:1","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-38:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.47:3","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-38:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.47:10","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23:8","obj_delete_at_hint.0000000111","obj_delete_at_hint.0000000048","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28:2","bucket.full-sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-36:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.45:5","obj_delete_at_hint.0000000020","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-48:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.5:9","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-37:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.46:3","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25:0","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27:1","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21:10","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21:3","datalog.sync-status.shard.2ea581c3-3f48-49d7-a989-68024c5b8eac.0","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30:2","obj_delete_at_hint.0000000099","obj_delete_at_hint.0000000056","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-46:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.3:1","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29:9","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34:10","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24:4","sync.error-log.19","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-46:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.3:7","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-47:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.4:9","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35:10","bucket.full-sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23","obj_delete_at_hint.0000000072","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24:1","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34:10","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-49:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5625.1:9","obj_delete_at_hint.0000000062","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19:3","obj_delete_at_hint.0000000115","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19:7","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22:5","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32:3","sync.error-log.1","obj_delete_at_hint.0000000033","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19:5","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-36:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.45:0","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-37:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.46:7","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25:1","datalog.sync-status.shard.01fb6d04-f0ef-49b2-9ebc-c1b530925485.1.retry","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20:0","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21:2","bucket.full-sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21:8","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29:7","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-36:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.45:4","obj_delete_at_hint.0000000025","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33:2","obj_delete_at_hint.0000000123","datalog.sync-status.shard.2ea581c3-3f48-49d7-a989-68024c5b8eac.1.retry","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25:5","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-38:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.47:10","sync.error-log.18","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-49:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5625.1:1","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29:2","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20:3","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30:4","obj_delete_at_hint.0000000108","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20:5","obj_delete_at_hint.0000000094","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-48:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.5:10","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33:8","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27:5","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-36:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.45:7","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21:3","obj_delete_at_hint.0000000060","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27:4","obj_delete_at_hint.0000000109","sync.error-log.15","bucket.full-sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28:9","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-48:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.5:1","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32:7","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25:1","obj_delete_at_hint.0000000013","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28:5","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-37:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.46:6","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22:1","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24:5","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32:0","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24:2","bucket.full-sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-36:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.45:3","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31:9","datalog.sync-status.shard.01fb6d04-f0ef-49b2-9ebc-c1b530925485.0","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-36:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.45:8","data_log.3.ebudAIa5lyB.0","obj_delete_at_hint.0000000051","obj_delete_at_hint.0000000106","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21:2","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29:6","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30:4","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21:6","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22:5","obj_delete_at_hint.0000000017","bucket.full-sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35:0","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-47:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.4:7","bucket.full-sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-48:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.5","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23:2","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-37:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.46:8","datalog.sync-status.shard.2ea581c3-3f48-49d7-a989-68024c5b8eac.0.retry","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30:3","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-47:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.4:4","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29:1","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30:8","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35:4","obj_delete_at_hint.0000000023","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26:3","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-49:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5625.1:0","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-49:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5625.1:4","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24:2","datalog.sync-status.shard.01fb6d04-f0ef-49b2-9ebc-c1b530925485.2","obj_delete_at_hint.0000000107","bucket.full-sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21:4","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32:5","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31:3","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27:2","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30:9","obj_delete_at_hint.0000000046","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26:4","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-37:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.46:0","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19:3","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-46:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.3:9","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-37:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.46:2","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25:5","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-38:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.47:7","bucket.full-sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22","sync.error-log.22","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26:8","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22:6","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30:2","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-49:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5625.1:2","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29:8","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28:3","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30:0","data_log.2","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21:1","obj_delete_at_hint.0000000067","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19:0","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25:3","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-46:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.3:4","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28:7","bucket.full-sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-49:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5625.1","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-37:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.46:0","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31:5","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23:9","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35:5","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-48:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.5:8","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-46:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.3:8","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23:5","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33:9","sync.error-log.9","obj_delete_at_hint.0000000086","bucket.full-sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-46:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.3","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35:6","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-47:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.4:4","bucket.full-sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21","sync.error-log.5","data_log.0.ULXCf85TqY6.0","bucket.full-sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35:6","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23:0","datalog.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-36:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.45:1","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29:10","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21:9","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30:5","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-47:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.4:10","bucket.full-sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-38:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.47","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29:1","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-47:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.4:1","sync.error-log.20","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19:8","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31:0","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-48:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.5:4","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-37:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.46:3","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33:7","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-49:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5625.1:6","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34:7","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30:8","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29:3","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23:2","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-37:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.46:5","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19:9","obj_delete_at_hint.0000000050","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-48:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.5:7","obj_delete_at_hint.0000000102","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-49:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5625.1:7","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-37:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.46:10","datalog.sync-status.shard.2ea581c3-3f48-49d7-a989-68024c5b8eac.2","bucket.full-sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28:8","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31:0","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-36:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.45:10","obj_delete_at_hint.0000000124","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34:8","obj_delete_at_hint.0000000001","bucket.full-sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28:0","obj_delete_at_hint.0000000126","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27:2","obj_delete_at_hint.0000000055","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33:1","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33:3","obj_delete_at_hint.0000000061","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-46:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.3:3","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26:7","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29:3","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21:0","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-37:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.46:8","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-37:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.46:1","obj_delete_at_hint.0000000097","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28:1","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26:5","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34:6","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33:0","obj_delete_at_hint.0000000042","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-49:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5625.1:10","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26:0","datalog.sync-status.shard.2ea581c3-3f48-49d7-a989-68024c5b8eac.2.retry","datalog.sync-status.shard.2ea581c3-3f48-49d7-a989-68024c5b8eac.1","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-36:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.45:5","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33:6","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24:4","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22:8","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-48:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.5:5","obj_delete_at_hint.0000000075","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23:7","obj_delete_at_hint.0000000120","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-47:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.4:0","obj_delete_at_hint.0000000030","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19:10","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22:6","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-48:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.5:6","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-37:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.46:7","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-38:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.47:8","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-36:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.45:2"] 2024-02-01T10:46:13.123 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:46:13.123 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:46:13.123 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:46:13.163 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:46:13.163 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:46:13.265 INFO:teuthology.orchestra.run.smithi187.stdout:["bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-48:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.5:2","obj_delete_at_hint.0000000078","meta.history","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29:7","bucket.full-sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-47:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.4:3","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31:9","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20:7","obj_delete_at_hint.0000000070","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28:7","obj_delete_at_hint.0000000104","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35:9","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-47:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.4:4","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-49:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5625.1:4","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34:1","obj_delete_at_hint.0000000026","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34:9","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-48:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.5:10","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34:5","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32:5","sync.error-log.8","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-49:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5625.1:6","obj_delete_at_hint.0000000028","bucket.full-sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34:1","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21:8","bucket.full-sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32:7","bucket.full-sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26:2","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23:10","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-47:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.4:0","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22:10","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-38:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.47:5","obj_delete_at_hint.0000000040","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33:4","sync.error-log.6","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-48:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.5:5","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-38:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.47:9","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27:3","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25:8","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34:5","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35:1","obj_delete_at_hint.0000000015","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24:9","bucket.full-sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19","obj_delete_at_hint.0000000069","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24:4","obj_delete_at_hint.0000000095","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19:2","obj_delete_at_hint.0000000003","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-48:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.5:3","obj_delete_at_hint.0000000047","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19:0","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30:6","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25:2","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32:8","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24:0","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-46:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.3:10","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23:8","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23:1","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35:5","obj_delete_at_hint.0000000035","obj_delete_at_hint.0000000037","bucket.full-sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-47:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.4","bucket.full-sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32","bucket.full-sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30:7","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31:3","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-38:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.47:5","bucket.full-sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23:6","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32:1","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-49:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5625.1:0","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29:4","obj_delete_at_hint.0000000024","obj_delete_at_hint.0000000000","obj_delete_at_hint.0000000031","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19:9","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25:7","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33:3","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19:1","data_log.3","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32:9","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29:0","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19:7","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-37:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.46:7","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-46:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.3:3","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22:1","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29:0","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27:6","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27:7","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23:1","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35:8","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33:4","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27:9","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28:6","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-49:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5625.1:2","bucket.full-sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34:3","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35:1","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23:10","data_loggenerations_metadata","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-49:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5625.1:10","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32:2","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-47:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.4:8","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30:0","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22:0","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19:4","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29:10","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-36:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.45:1","obj_delete_at_hint.0000000076","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28:5","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22:4","obj_delete_at_hint.0000000113","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34:3","obj_delete_at_hint.0000000005","obj_delete_at_hint.0000000011","obj_delete_at_hint.0000000058","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27:4","obj_delete_at_hint.0000000052","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-49:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5625.1:1","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20:10","sync.error-log.21","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20:1","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-36:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.45:4","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27:10","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-48:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.5:1","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24:1","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24:8","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28:10","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-38:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.47:7","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25:5","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19:1","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34:2","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31:3","obj_delete_at_hint.0000000088","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-48:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.5:3","obj_delete_at_hint.0000000080","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-49:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5625.1:1","obj_delete_at_hint.0000000090","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-49:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5625.1:5","bucket.full-sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-37:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.46","obj_delete_at_hint.0000000110","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21:4","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34:0","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-49:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5625.1:5","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-36:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.45:2","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20:2","obj_delete_at_hint.0000000096","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34:8","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20:6","obj_delete_at_hint.0000000087","obj_delete_at_hint.0000000008","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35:3","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-46:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.3:6","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20:3","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29:3","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35:4","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20:7","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34:9","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23:9","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22:5","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-49:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5625.1:9","bucket.full-sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27","datalog.sync-status.shard.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.2","obj_delete_at_hint.0000000006","obj_delete_at_hint.0000000029","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23:0","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22:7","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20:4","obj_delete_at_hint.0000000089","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33:9","obj_delete_at_hint.0000000022","bucket.full-sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21","bucket.full-sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33:10","bucket.full-sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32:3","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33:1","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31:10","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26:0","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27:8","bilog.trim","obj_delete_at_hint.0000000016","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31:4","bucket.full-sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-49:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5625.1:4","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26:10","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19:6","obj_delete_at_hint.0000000041","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24:3","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-38:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.47:2","bucket.full-sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20:9","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26:5","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33:6","obj_delete_at_hint.0000000018","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32:4","obj_delete_at_hint.0000000092","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27:7","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32:0","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20:5","data_log.1.iE2ha9apB3A.0","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28:2","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-46:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.3:9","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26:4","sync.error-log.4","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-38:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.47:0","datalog.sync-status.shard.2ea581c3-3f48-49d7-a989-68024c5b8eac.3","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-46:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.3:4","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33:5","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-37:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.46:1","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23:8","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31:5","datalog.sync-status.shard.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.0.retry","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34:4","bucket.full-sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-38:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.47","obj_delete_at_hint.0000000014","bucket.full-sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-49:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5625.1","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20:2","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-38:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.47:3","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-38:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.47:4","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30:3","bucket.full-sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22","obj_delete_at_hint.0000000112","obj_delete_at_hint.0000000007","obj_delete_at_hint.0000000021","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35:0","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25:9","data_log.0.FDQM4G83HfQ.0","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20:1","sync.error-log.16","obj_delete_at_hint.0000000064","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31:8","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33:6","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33:7","obj_delete_at_hint.0000000071","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-47:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.4:5","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-46:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.3:3","obj_delete_at_hint.0000000074","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33:2","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25:9","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25:8","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-36:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.45:3","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21:9","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26:9","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19:6","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20:6","obj_delete_at_hint.0000000081","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27:10","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28:4","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33:7","obj_delete_at_hint.0000000009","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24:0","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19:8","obj_delete_at_hint.0000000121","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27:1","data_log.1","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-36:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.45:5","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21:8","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-46:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.3:6","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-37:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.46:4","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31:0","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20:5","sync.error-log.17","obj_delete_at_hint.0000000125","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22:10","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21:2","obj_delete_at_hint.0000000082","bucket.full-sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-37:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.46","bucket.full-sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-48:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.5","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28:10","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29:5","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32:0","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26:10","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25:6","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27:5","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31:7","obj_delete_at_hint.0000000105","obj_delete_at_hint.0000000059","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-36:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.45:6","bucket.full-sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-47:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.4","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32:10","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24:10","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-37:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.46:5","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32:6","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-37:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.46:6","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-47:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.4:2","obj_delete_at_hint.0000000077","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34:10","obj_delete_at_hint.0000000032","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29:6","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31:8","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23:2","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-38:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.47:0","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24:7","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20:9","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21:7","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-37:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.46:0","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-37:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.46:1","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-48:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.5:10","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-46:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.3:7","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22:2","bucket.full-sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24","data_log.3.TFchYbACxBD.0","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32:10","obj_delete_at_hint.0000000053","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26:2","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31:1","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28:6","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24:7","obj_delete_at_hint.0000000091","datalog.sync-status.shard.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.3","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20:4","bucket.full-sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-38:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.47","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35:0","obj_delete_at_hint.0000000065","bucket.full-sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31","obj_delete_at_hint.0000000083","obj_delete_at_hint.0000000010","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-47:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.4:1","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-38:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.47:1","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-36:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.45:1","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19:10","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-48:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.5:6","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24:6","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22:4","sync.error-log.7","sync.error-log.3","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23:6","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-46:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.3:7","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29:8","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-46:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.3:5","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20:8","mdlog.sync-status.shard.2","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-48:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.5:0","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33:3","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19:3","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35:8","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-48:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.5:4","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34:4","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23:4","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30:10","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22:2","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-38:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.47:6","meta.log.f2e6c290-e7aa-4279-b77e-15403b18d61f.2","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21:0","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22:0","obj_delete_at_hint.0000000045","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25:3","obj_delete_at_hint.0000000002","sync.error-log.12","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34:2","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27:2","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26:1","obj_delete_at_hint.0000000116","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-49:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5625.1:3","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-37:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.46:5","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-37:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.46:4","obj_delete_at_hint.0000000034","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-38:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.47:6","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-37:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.46:3","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-49:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5625.1:0","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-46:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.3:2","obj_delete_at_hint.0000000101","sync.error-log.13","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25:3","mdlog.sync-status.shard.1","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19:2","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-47:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.4:9","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-48:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.5:9","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29:6","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-49:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5625.1:2","obj_delete_at_hint.0000000079","obj_delete_at_hint.0000000049","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28:1","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26:0","obj_delete_at_hint.0000000117","obj_delete_at_hint.0000000044","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-46:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.3:0","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21:5","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32:1","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32:7","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35:3","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24:6","sync.error-log.14","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22:6","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33:5","obj_delete_at_hint.0000000066","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31:4","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35:2","datalog.sync-status.shard.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.1.retry","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-47:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.4:3","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-36:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.45:10","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-37:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.46:9","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-38:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.47:9","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28:9","obj_delete_at_hint.0000000068","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25:6","obj_delete_at_hint.0000000085","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29:2","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32:4","bucket.full-sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29:9","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33:9","bucket.full-sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24","obj_delete_at_hint.0000000073","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24:10","obj_delete_at_hint.0000000038","bucket.full-sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33:0","sync.error-log.10","obj_delete_at_hint.0000000118","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-37:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.46:6","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-38:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.47:8","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19:0","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35:4","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27:0","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-38:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.47:8","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35:10","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30:7","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21:6","obj_delete_at_hint.0000000036","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-36:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.45:0","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32:6","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19:5","obj_delete_at_hint.0000000103","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21:10","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26:4","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23:3","mdlog.sync-status.shard.3","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35:7","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-36:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.45:6","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31:9","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25:4","obj_delete_at_hint.0000000119","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-36:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.45:9","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34:6","obj_delete_at_hint.0000000098","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24:3","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27:9","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33:2","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-46:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.3:5","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25:4","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25:10","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30:10","mdlog.sync-status.shard.0","obj_delete_at_hint.0000000027","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29:1","mdlog.sync-status","obj_delete_at_hint.0000000019","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-36:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.45:7","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19:10","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23:3","obj_delete_at_hint.0000000039","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31:6","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-37:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.46:10","bucket.full-sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31:6","bucket.full-sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-46:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.3","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-47:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.4:6","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31:2","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-37:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.46:9","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27:3","obj_delete_at_hint.0000000100","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26:9","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-48:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.5:2","obj_delete_at_hint.0000000093","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20:10","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30:1","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-47:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.4:6","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35:2","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-49:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5625.1:3","obj_delete_at_hint.0000000004","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-47:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.4:0","obj_delete_at_hint.0000000063","obj_delete_at_hint.0000000122","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-49:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5625.1:6","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29:10","obj_delete_at_hint.0000000057","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22:3","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20:3","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22:9","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-48:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.5:9","obj_delete_at_hint.0000000054","obj_delete_at_hint.0000000114","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32:2","sync.error-log.2","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-47:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.4:9","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29:7","data_log.0","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23:9","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-38:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.47:4","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23:0","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26:6","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27:6","bucket.full-sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-36:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.45","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28:3","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-48:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.5:8","obj_delete_at_hint.0000000012","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24:8","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-38:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.47:10","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22:8","bucket.full-sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28:5","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30:5","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-47:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.4:2","obj_delete_at_hint.0000000084","obj_delete_at_hint.0000000043","bucket.full-sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-46:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.3","bucket.full-sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27","sync.error-log.11","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-36:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.45:0","datalog.sync-status.shard.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.1","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-47:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.4:10","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-49:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5625.1:8","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24:9","obj_delete_at_hint.0000000111","obj_delete_at_hint.0000000048","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-46:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.3:10","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21:7","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28:2","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-36:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.45:5","obj_delete_at_hint.0000000020","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24:5","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30:8","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25:0","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33:10","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27:1","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21:10","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21:3","datalog.sync-status.shard.2ea581c3-3f48-49d7-a989-68024c5b8eac.0","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21:9","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30:2","obj_delete_at_hint.0000000099","obj_delete_at_hint.0000000056","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22:1","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-46:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.3:1","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29:9","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28:9","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34:10","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-36:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.45:7","sync.error-log.19","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35:10","bucket.full-sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23","obj_delete_at_hint.0000000072","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22:7","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-49:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5625.1:9","obj_delete_at_hint.0000000062","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32:8","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19:3","obj_delete_at_hint.0000000115","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22:5","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32:3","sync.error-log.1","obj_delete_at_hint.0000000033","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19:5","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24:1","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23:5","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-37:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.46:7","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34:6","bucket.full-sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25:1","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25:0","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20:0","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20:0","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21:2","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28:0","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-36:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.45:4","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21:5","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-47:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.4:7","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-46:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.3:8","obj_delete_at_hint.0000000025","obj_delete_at_hint.0000000123","datalog.sync-status.shard.2ea581c3-3f48-49d7-a989-68024c5b8eac.1.retry","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-36:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.45:9","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-49:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5625.1:8","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-38:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.47:10","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26:6","sync.error-log.18","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30:4","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34:8","obj_delete_at_hint.0000000108","obj_delete_at_hint.0000000094","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33:8","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27:5","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-36:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.45:8","obj_delete_at_hint.0000000060","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27:4","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-38:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.47:3","obj_delete_at_hint.0000000109","sync.error-log.15","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21:3","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-48:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.5:1","obj_delete_at_hint.0000000013","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20:8","bucket.full-sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24:5","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25:10","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21:1","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22:9","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27:0","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26:7","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-47:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.4:8","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-36:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.45:3","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24:2","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22:3","data_log.2./70W66wz+ZV.0","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-36:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.45:8","obj_delete_at_hint.0000000051","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26:1","obj_delete_at_hint.0000000106","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19:7","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-49:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5625.1:7","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30:4","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29:2","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21:6","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30:0","obj_delete_at_hint.0000000017","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28:4","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33:1","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-47:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.4:7","bucket.full-sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-48:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.5","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-37:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.46:8","datalog.sync-status.shard.2ea581c3-3f48-49d7-a989-68024c5b8eac.0.retry","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30:3","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-47:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.4:4","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35:7","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29:5","obj_delete_at_hint.0000000023","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26:3","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24:2","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31:7","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-48:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.5:7","obj_delete_at_hint.0000000107","bucket.full-sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-47:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.4:5","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32:5","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27:2","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30:9","obj_delete_at_hint.0000000046","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34:7","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-46:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.3:9","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35:6","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-37:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.46:2","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25:5","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35:9","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25:7","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34:0","bucket.full-sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22","sync.error-log.22","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-37:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.46:8","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26:8","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22:6","datalog.sync-status.shard.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.2.retry","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21:4","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29:8","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28:3","data_log.2","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30:6","datalog.sync-status.shard.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.3.retry","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25:2","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21:1","obj_delete_at_hint.0000000067","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-46:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.3:4","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28:7","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-48:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.5:5","bucket.full-sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-49:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5625.1","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-37:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.46:0","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31:5","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35:5","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-48:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.5:8","bucket.full-sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-36:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.45","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-46:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.3:8","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23:5","sync.error-log.9","obj_delete_at_hint.0000000086","sync.error-log.5","datalog.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28:8","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30:9","bucket.full-sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28:1","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35:6","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-46:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.3:2","datalog.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23:4","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30:5","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-47:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.4:10","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29:1","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-47:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.4:1","sync.error-log.20","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19:8","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-48:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.5:4","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-37:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.46:3","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-46:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.3:1","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34:7","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30:8","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23:2","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27:8","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30:1","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19:9","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19:4","obj_delete_at_hint.0000000050","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-48:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.5:7","obj_delete_at_hint.0000000102","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31:10","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-49:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5625.1:7","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-37:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.46:10","datalog.sync-status.shard.2ea581c3-3f48-49d7-a989-68024c5b8eac.2","bucket.full-sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-38:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.47:1","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28:8","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31:0","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-36:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.45:10","obj_delete_at_hint.0000000124","datalog.sync-status.shard.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.0","obj_delete_at_hint.0000000001","bucket.full-sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28:0","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-46:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.3:0","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-48:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.5:0","obj_delete_at_hint.0000000126","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31:1","obj_delete_at_hint.0000000055","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26:3","obj_delete_at_hint.0000000061","bucket.full-sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23:7","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26:7","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29:3","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21:0","obj_delete_at_hint.0000000097","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26:5","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33:0","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33:8","obj_delete_at_hint.0000000042","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29:4","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31:2","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-49:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5625.1:10","datalog.sync-status.shard.2ea581c3-3f48-49d7-a989-68024c5b8eac.2.retry","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-38:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.47:2","datalog.sync-status.shard.2ea581c3-3f48-49d7-a989-68024c5b8eac.1","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24:4","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22:8","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-38:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.47:7","obj_delete_at_hint.0000000075","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23:7","obj_delete_at_hint.0000000120","obj_delete_at_hint.0000000030","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-48:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.5:6","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32:9","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30:2","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26:8","bucket.sync-status.2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-36:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.45:2","bucket.full-sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25:1","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-37:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.46:2"] 2024-02-01T10:46:13.290 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['log', 'list', '--rgw-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:46:13.291 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'log', 'list', '--rgw-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:46:13.291 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 log list --rgw-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:46:13.331 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:46:13.331 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:46:13.434 INFO:teuthology.orchestra.run.smithi148.stdout:["bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-48:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.5:2","obj_delete_at_hint.0000000078","meta.history","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29:7","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31:6","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19:8","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31:9","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34:1","obj_delete_at_hint.0000000070","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28:7","obj_delete_at_hint.0000000104","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-47:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.4:4","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34:1","obj_delete_at_hint.0000000026","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34:9","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-48:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.5:10","bucket.full-sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-49:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5625.1","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32:5","datalog.sync-status.shard.01fb6d04-f0ef-49b2-9ebc-c1b530925485.0.retry","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-46:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.3:1","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32:8","sync.error-log.8","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-49:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5625.1:6","obj_delete_at_hint.0000000028","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-36:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.45:4","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21:8","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-47:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.4:10","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20:6","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32:7","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29:5","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23:10","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23:10","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-47:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.4:0","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31:8","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19:1","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22:10","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-36:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.45:3","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28:10","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-46:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.3:5","obj_delete_at_hint.0000000040","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-48:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.5:2","datalog.sync-status.shard.01fb6d04-f0ef-49b2-9ebc-c1b530925485.1","sync.error-log.6","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27:3","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22:3","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32:9","bucket.full-sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34:5","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35:1","obj_delete_at_hint.0000000015","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23:5","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24:9","obj_delete_at_hint.0000000069","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24:4","obj_delete_at_hint.0000000095","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-46:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.3:10","obj_delete_at_hint.0000000003","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-48:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.5:3","obj_delete_at_hint.0000000047","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28:4","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32:1","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24:0","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35:5","obj_delete_at_hint.0000000035","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27:9","obj_delete_at_hint.0000000037","bucket.full-sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32","bucket.full-sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33:8","data_log.2.IYNcSXkOGPI.0","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-38:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.47:5","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29:9","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23:6","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20:10","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31:1","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32:1","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-49:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5625.1:0","obj_delete_at_hint.0000000024","obj_delete_at_hint.0000000000","obj_delete_at_hint.0000000031","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19:9","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26:9","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33:3","bucket.full-sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33","data_log.3","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34:5","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35:1","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-37:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.46:7","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20:4","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29:0","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19:4","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-48:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.5:0","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27:6","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27:7","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23:1","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35:8","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33:4","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-49:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5625.1:2","bucket.full-sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34:3","bucket.full-sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28:8","data_loggenerations_metadata","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-49:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5625.1:10","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-46:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.3:6","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-36:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.45:8","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29:10","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24:0","obj_delete_at_hint.0000000076","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22:4","obj_delete_at_hint.0000000113","obj_delete_at_hint.0000000005","obj_delete_at_hint.0000000011","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28:0","obj_delete_at_hint.0000000058","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27:4","obj_delete_at_hint.0000000052","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-49:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5625.1:1","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20:10","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-48:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.5:8","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-47:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.4:7","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20:1","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-36:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.45:4","bucket.full-sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31","bucket.full-sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23:3","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-48:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.5:1","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-37:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.46:9","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25:5","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19:1","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25:6","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34:2","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34:7","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31:3","obj_delete_at_hint.0000000088","bucket.full-sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23","obj_delete_at_hint.0000000080","obj_delete_at_hint.0000000090","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25:0","bucket.full-sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-37:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.46","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21:0","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22:10","obj_delete_at_hint.0000000110","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-49:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5625.1:5","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-36:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.45:2","obj_delete_at_hint.0000000096","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20:6","obj_delete_at_hint.0000000087","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23:4","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32:3","obj_delete_at_hint.0000000008","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28:6","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21:10","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29:3","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33:10","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20:7","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22:5","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-49:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5625.1:9","bucket.full-sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-47:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.4:6","datalog.sync-status.shard.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.2","obj_delete_at_hint.0000000006","obj_delete_at_hint.0000000029","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23:0","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25:4","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34:2","obj_delete_at_hint.0000000089","obj_delete_at_hint.0000000022","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35:8","bucket.full-sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21","bucket.full-sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-46:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.3:8","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32:3","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-48:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.5:4","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-46:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.3:2","bilog.trim","obj_delete_at_hint.0000000016","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31:4","bucket.full-sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-49:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5625.1:4","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26:10","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27:1","obj_delete_at_hint.0000000041","bucket.full-sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26:2","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20:9","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26:5","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-37:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.46:10","obj_delete_at_hint.0000000018","obj_delete_at_hint.0000000092","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-48:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.5:3","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21:1","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32:0","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28:2","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-46:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.3:9","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26:4","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-36:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.45:9","sync.error-log.4","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-38:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.47:0","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-46:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.3:4","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33:5","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23:8","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31:5","datalog.sync-status.shard.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.0.retry","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35:7","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35:9","bucket.full-sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-38:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.47","obj_delete_at_hint.0000000014","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27:8","bucket.full-sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-49:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5625.1","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20:2","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-49:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5625.1:8","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-38:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.47:4","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30:3","bucket.full-sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22","obj_delete_at_hint.0000000112","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-49:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5625.1:9","obj_delete_at_hint.0000000007","obj_delete_at_hint.0000000021","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35:0","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32:10","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24:5","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20:2","obj_delete_at_hint.0000000064","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19:5","datalog.sync-status.shard.01fb6d04-f0ef-49b2-9ebc-c1b530925485.3","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31:8","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33:6","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25:10","obj_delete_at_hint.0000000071","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22:0","meta.log.5dc4efbd-eccd-47a8-9471-ae4a209ec978.2","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-46:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.3:3","obj_delete_at_hint.0000000074","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25:9","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27:10","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25:8","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-49:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5625.1:10","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-36:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.45:3","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-38:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.47:6","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19:6","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26:8","obj_delete_at_hint.0000000081","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27:10","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33:7","obj_delete_at_hint.0000000009","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-38:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.47:0","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19:8","obj_delete_at_hint.0000000121","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27:1","data_log.1","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20:8","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-36:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.45:5","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-46:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.3:6","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-38:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.47:9","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31:0","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20:5","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20:9","obj_delete_at_hint.0000000125","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30:6","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21:2","obj_delete_at_hint.0000000082","bucket.full-sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-48:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.5","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28:10","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32:6","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22:8","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27:0","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27:5","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-38:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.47:2","obj_delete_at_hint.0000000105","obj_delete_at_hint.0000000059","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23:1","bucket.full-sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-47:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.4","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26:10","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32:10","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24:10","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-37:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.46:5","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32:6","obj_delete_at_hint.0000000077","bucket.full-sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-47:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.4","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35:2","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34:10","obj_delete_at_hint.0000000032","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29:6","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24:10","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23:2","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-36:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.45:2","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33:5","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-37:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.46:0","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-37:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.46:1","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27:6","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19:9","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-46:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.3:7","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-38:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.47:4","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-38:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.47:5","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22:2","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31:2","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19:6","obj_delete_at_hint.0000000053","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26:2","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-36:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.45:6","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28:6","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24:7","obj_delete_at_hint.0000000091","datalog.sync-status.shard.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.3","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20:4","bucket.full-sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-48:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.5","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-37:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.46:4","obj_delete_at_hint.0000000065","bucket.full-sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-46:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.3:0","obj_delete_at_hint.0000000083","obj_delete_at_hint.0000000010","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-47:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.4:1","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-36:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.45:1","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31:10","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-49:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5625.1:3","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19:10","bucket.full-sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-48:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.5:6","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24:6","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-47:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.4:2","sync.error-log.7","sync.error-log.3","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-47:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.4:3","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34:3","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29:8","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29:8","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30:7","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31:5","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22:7","meta.log.5dc4efbd-eccd-47a8-9471-ae4a209ec978.0","mdlog.sync-status.shard.2","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19:3","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35:3","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-48:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.5:4","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34:4","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30:10","meta.log.f2e6c290-e7aa-4279-b77e-15403b18d61f.2","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-36:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.45:10","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21:0","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22:0","obj_delete_at_hint.0000000045","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25:3","data_log.3.rpcXzJCYrFO.0","obj_delete_at_hint.0000000002","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22:4","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20:1","sync.error-log.12","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23:6","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27:2","obj_delete_at_hint.0000000116","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32:4","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-37:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.46:4","obj_delete_at_hint.0000000034","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-48:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.5:7","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26:1","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-38:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.47:6","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-37:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.46:3","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-47:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.4:8","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30:10","obj_delete_at_hint.0000000101","mdlog.sync-status.shard.1","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19:2","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-47:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.4:9","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34:4","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-48:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.5:9","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31:7","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24:9","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21:7","obj_delete_at_hint.0000000079","obj_delete_at_hint.0000000049","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26:0","obj_delete_at_hint.0000000117","obj_delete_at_hint.0000000044","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29:4","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34:9","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30:1","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35:3","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-38:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.47:1","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22:6","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25:8","obj_delete_at_hint.0000000066","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29:0","datalog.sync-status.shard.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.1.retry","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-47:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.4:3","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24:3","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-36:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.45:10","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-38:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.47:9","obj_delete_at_hint.0000000068","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25:6","obj_delete_at_hint.0000000085","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32:4","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29:9","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27:3","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23:7","meta.log.5dc4efbd-eccd-47a8-9471-ae4a209ec978.3","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19:2","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33:9","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21:6","bucket.full-sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30:3","obj_delete_at_hint.0000000073","bucket.full-sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20","obj_delete_at_hint.0000000038","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21:5","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27:7","bucket.full-sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28:3","bucket.full-sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33:0","sync.error-log.10","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35:5","obj_delete_at_hint.0000000118","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-37:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.46:6","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19:0","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35:4","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-38:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.47:8","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35:10","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30:7","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21:6","obj_delete_at_hint.0000000036","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22:9","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19:5","obj_delete_at_hint.0000000103","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21:10","mdlog.sync-status.shard.3","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-49:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5625.1:7","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35:10","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-36:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.45:6","obj_delete_at_hint.0000000119","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31:4","obj_delete_at_hint.0000000098","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24:3","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27:9","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33:2","datalog.sync-status.shard.01fb6d04-f0ef-49b2-9ebc-c1b530925485.2.retry","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-37:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.46:2","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-46:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.3:5","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25:4","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30:9","mdlog.sync-status.shard.0","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32:5","data_log.0.VgdX4BCwZRQ.0","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-46:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.3:4","obj_delete_at_hint.0000000027","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20:7","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29:1","mdlog.sync-status","obj_delete_at_hint.0000000019","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-47:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.4:1","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23:3","obj_delete_at_hint.0000000039","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31:6","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-37:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.46:10","bucket.full-sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28","bucket.full-sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-46:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.3","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25:7","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-37:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.46:9","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-49:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5625.1:5","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25:2","obj_delete_at_hint.0000000100","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26:9","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26:5","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30:5","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33:4","obj_delete_at_hint.0000000093","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34:0","bucket.full-sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27","bucket.full-sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-37:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.46","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-47:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.4:6","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35:2","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27:4","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-49:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5625.1:3","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-47:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.4:5","obj_delete_at_hint.0000000004","obj_delete_at_hint.0000000063","obj_delete_at_hint.0000000122","obj_delete_at_hint.0000000057","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26:6","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20:3","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-46:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.3:9","datalog.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485","obj_delete_at_hint.0000000054","obj_delete_at_hint.0000000114","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32:2","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26:3","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22:2","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28:7","sync.error-log.2","data_log.0","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23:9","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25:9","bucket.full-sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28:3","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-48:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.5:8","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24:7","obj_delete_at_hint.0000000012","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32:2","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24:8","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-38:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.47:10","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22:8","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24:6","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28:5","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30:5","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-47:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.4:2","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33:0","obj_delete_at_hint.0000000084","obj_delete_at_hint.0000000043","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27:5","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26:7","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24:8","sync.error-log.11","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-36:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.45:0","datalog.sync-status.shard.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.1","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-47:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.4:10","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20:0","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-48:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.5:6","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28:2","bucket.full-sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-36:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.45","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-48:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.5:1","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-38:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.47:3","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-38:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.47:10","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23:8","obj_delete_at_hint.0000000111","obj_delete_at_hint.0000000048","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-46:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.3:10","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21:7","bucket.full-sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26","obj_delete_at_hint.0000000020","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24:5","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-48:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.5:9","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30:8","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-37:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.46:3","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33:10","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21:9","obj_delete_at_hint.0000000099","obj_delete_at_hint.0000000056","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22:1","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28:9","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-36:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.45:7","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24:4","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-46:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.3:7","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-47:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.4:9","obj_delete_at_hint.0000000072","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22:7","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24:1","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34:10","obj_delete_at_hint.0000000062","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32:8","obj_delete_at_hint.0000000115","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19:7","sync.error-log.1","obj_delete_at_hint.0000000033","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-36:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.45:0","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24:1","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23:5","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34:6","bucket.full-sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30","datalog.sync-status.shard.01fb6d04-f0ef-49b2-9ebc-c1b530925485.1.retry","meta.log.976daaff-eac6-4c84-bd63-38e72aa959ff.2","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25:0","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20:0","bucket.full-sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28:0","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21:8","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29:7","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21:5","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-47:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.4:7","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-46:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.3:8","obj_delete_at_hint.0000000025","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33:2","obj_delete_at_hint.0000000123","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-36:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.45:9","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25:5","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-49:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5625.1:8","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26:6","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30:4","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-49:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5625.1:1","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29:2","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34:8","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20:3","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30:4","obj_delete_at_hint.0000000108","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20:5","obj_delete_at_hint.0000000094","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-48:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.5:10","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-36:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.45:7","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-36:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.45:8","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21:3","obj_delete_at_hint.0000000060","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-38:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.47:3","obj_delete_at_hint.0000000109","bucket.full-sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28:9","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21:3","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32:7","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25:1","obj_delete_at_hint.0000000013","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28:5","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-37:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.46:6","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22:1","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-11:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.20:8","bucket.full-sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32:0","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25:10","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21:1","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22:9","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27:0","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26:7","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24:2","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-47:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.4:8","bucket.full-sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31:9","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-15:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.24:2","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22:3","datalog.sync-status.shard.01fb6d04-f0ef-49b2-9ebc-c1b530925485.0","obj_delete_at_hint.0000000051","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26:1","obj_delete_at_hint.0000000106","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21:2","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19:7","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-49:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5625.1:7","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29:6","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29:2","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22:5","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30:0","obj_delete_at_hint.0000000017","bucket.full-sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28:4","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33:1","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35:0","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23:2","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29:1","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35:7","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30:8","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29:5","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35:4","obj_delete_at_hint.0000000023","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-49:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5625.1:0","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-49:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5625.1:4","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31:7","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-48:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.5:7","datalog.sync-status.shard.01fb6d04-f0ef-49b2-9ebc-c1b530925485.2","obj_delete_at_hint.0000000107","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21:4","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-47:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.4:5","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31:3","obj_delete_at_hint.0000000046","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26:4","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34:7","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-37:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.46:0","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19:3","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35:6","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35:9","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25:7","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34:0","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-38:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.47:7","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-37:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.46:8","datalog.sync-status.shard.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.2.retry","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21:4","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30:2","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-49:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5625.1:2","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30:0","data_log.2","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30:6","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25:2","obj_delete_at_hint.0000000067","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19:0","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25:3","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-48:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.5:5","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23:9","bucket.full-sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-36:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.45","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33:9","sync.error-log.9","obj_delete_at_hint.0000000086","bucket.full-sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-46:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.3","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-26:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.35:6","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-47:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.4:4","bucket.full-sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21","sync.error-log.5","datalog.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28:8","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30:9","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28:1","data_log.1.tsZBUQY8c9n.0","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23:0","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-46:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.3:2","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-36:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.45:1","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23:4","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29:10","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-12:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.21:9","bucket.full-sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-38:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.47","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31:0","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33:7","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-49:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5625.1:6","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-46:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.3:1","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29:3","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27:8","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-37:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.46:5","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30:1","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19:4","obj_delete_at_hint.0000000050","obj_delete_at_hint.0000000102","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31:10","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-38:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.47:1","obj_delete_at_hint.0000000124","datalog.sync-status.shard.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.0","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34:8","obj_delete_at_hint.0000000001","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-46:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.3:0","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-48:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.5:0","obj_delete_at_hint.0000000126","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-18:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.27:2","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31:1","obj_delete_at_hint.0000000055","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26:3","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33:1","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33:3","obj_delete_at_hint.0000000061","bucket.full-sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-46:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.3:3","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-14:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.23:7","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-37:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.46:8","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-37:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.46:1","obj_delete_at_hint.0000000097","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-19:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.28:1","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-25:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.34:6","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33:8","obj_delete_at_hint.0000000042","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-20:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.29:4","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-22:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.31:2","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26:0","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-38:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.47:2","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-36:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.45:5","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-24:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.33:6","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-38:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.47:7","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-48:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.5:5","obj_delete_at_hint.0000000075","obj_delete_at_hint.0000000120","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-47:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5317.4:0","obj_delete_at_hint.0000000030","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19:10","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-13:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.22:6","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-23:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.32:9","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-37:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.46:7","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-21:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.30:2","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-17:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.26:8","bucket.sync-status.01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-38:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.47:8","bucket.full-sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-10:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.19","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-16:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.25:1","bucket.sync-status.ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-37:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.4351.46:2"] 2024-02-01T10:46:13.454 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone1 target=test-zone2 2024-02-01T10:46:13.455 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:46:13.455 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:46:13.455 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:46:13.539 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:46:13.539 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:46:13.625 INFO:teuthology.orchestra.run.smithi148.stdout:[{"marker":"00000000000000000000:00000000000000003045","last_update":"2024-02-01T10:46:12.887277Z"},{"marker":"00000000000000000000:00000000000000006141","last_update":"2024-02-01T10:46:12.888515Z"},{"marker":"00000000000000000000:00000000000000007551","last_update":"2024-02-01T10:46:12.889691Z"},{"marker":"00000000000000000000:00000000000000002060","last_update":"2024-02-01T10:46:12.890801Z"}] 2024-02-01T10:46:13.625 DEBUG:rgw_multi.tests:data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000003045', 1: '00000000000000000000:00000000000000006141', 2: '00000000000000000000:00000000000000007551', 3: '00000000000000000000:00000000000000002060'} 2024-02-01T10:46:13.625 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-02-01T10:46:13.625 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:46:13.625 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:46:13.626 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:46:13.670 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:46:13.670 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:46:13.754 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":12760245309917388665},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002204","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:21:48.024326Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004881","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:38:15.877435Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000006291","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:41:41.505224Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000796","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:38:14.730286Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-02-01T10:46:13Z"} 2024-02-01T10:46:13.755 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":12760245309917388665},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002204","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:21:48.024326Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004881","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:38:15.877435Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000006291","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:41:41.505224Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000796","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:38:14.730286Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-02-01T10:46:13Z"} 2024-02-01T10:46:13.755 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002204', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:21:48.024326Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004881', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:38:15.877435Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006291', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:41:41.505224Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000796', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:38:14.730286Z'}}] 2024-02-01T10:46:13.755 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003045', 1: '00000000000000000000:00000000000000006141', 2: '00000000000000000000:00000000000000007551', 3: '00000000000000000000:00000000000000002060'} 2024-02-01T10:46:13.756 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002204', 1: '00000000000000000000:00000000000000004881', 2: '00000000000000000000:00000000000000006291', 3: '00000000000000000000:00000000000000000796'} 2024-02-01T10:46:13.756 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000003045 target=00000000000000000000:00000000000000002204, shard=1 master=00000000000000000000:00000000000000006141 target=00000000000000000000:00000000000000004881, shard=2 master=00000000000000000000:00000000000000007551 target=00000000000000000000:00000000000000006291, shard=3 master=00000000000000000000:00000000000000002060 target=00000000000000000000:00000000000000000796 2024-02-01T10:46:18.757 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:46:18.757 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:46:18.757 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:46:18.802 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:46:18.802 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:46:18.885 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":12760245309917388665},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002204","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:21:48.024326Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004881","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:38:15.877435Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000006291","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:41:41.505224Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000796","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:38:14.730286Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-02-01T10:46:18Z"} 2024-02-01T10:46:18.886 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":12760245309917388665},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002204","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:21:48.024326Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004881","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:38:15.877435Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000006291","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:41:41.505224Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000796","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:38:14.730286Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-02-01T10:46:18Z"} 2024-02-01T10:46:18.886 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002204', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:21:48.024326Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004881', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:38:15.877435Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006291', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:41:41.505224Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000796', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:38:14.730286Z'}}] 2024-02-01T10:46:18.886 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003045', 1: '00000000000000000000:00000000000000006141', 2: '00000000000000000000:00000000000000007551', 3: '00000000000000000000:00000000000000002060'} 2024-02-01T10:46:18.886 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002204', 1: '00000000000000000000:00000000000000004881', 2: '00000000000000000000:00000000000000006291', 3: '00000000000000000000:00000000000000000796'} 2024-02-01T10:46:18.886 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000003045 target=00000000000000000000:00000000000000002204, shard=1 master=00000000000000000000:00000000000000006141 target=00000000000000000000:00000000000000004881, shard=2 master=00000000000000000000:00000000000000007551 target=00000000000000000000:00000000000000006291, shard=3 master=00000000000000000000:00000000000000002060 target=00000000000000000000:00000000000000000796 2024-02-01T10:46:23.887 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:46:23.887 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:46:23.887 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:46:23.932 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:46:23.932 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:46:24.015 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":12760245309917388665},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002204","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:21:48.024326Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004881","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:38:15.877435Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000006291","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:41:41.505224Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000796","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:38:14.730286Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-02-01T10:46:23Z"} 2024-02-01T10:46:24.016 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":12760245309917388665},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002204","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:21:48.024326Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004881","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:38:15.877435Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000006291","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:41:41.505224Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000796","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:38:14.730286Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-02-01T10:46:23Z"} 2024-02-01T10:46:24.016 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002204', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:21:48.024326Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004881', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:38:15.877435Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006291', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:41:41.505224Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000796', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:38:14.730286Z'}}] 2024-02-01T10:46:24.016 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003045', 1: '00000000000000000000:00000000000000006141', 2: '00000000000000000000:00000000000000007551', 3: '00000000000000000000:00000000000000002060'} 2024-02-01T10:46:24.016 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002204', 1: '00000000000000000000:00000000000000004881', 2: '00000000000000000000:00000000000000006291', 3: '00000000000000000000:00000000000000000796'} 2024-02-01T10:46:24.016 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000003045 target=00000000000000000000:00000000000000002204, shard=1 master=00000000000000000000:00000000000000006141 target=00000000000000000000:00000000000000004881, shard=2 master=00000000000000000000:00000000000000007551 target=00000000000000000000:00000000000000006291, shard=3 master=00000000000000000000:00000000000000002060 target=00000000000000000000:00000000000000000796 2024-02-01T10:46:29.018 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:46:29.018 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:46:29.018 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:46:29.058 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:46:29.059 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:46:29.142 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":12760245309917388665},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002204","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:21:48.024326Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004881","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:38:15.877435Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000006291","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:41:41.505224Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000796","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:38:14.730286Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-02-01T10:46:29Z"} 2024-02-01T10:46:29.143 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":12760245309917388665},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002204","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:21:48.024326Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004881","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:38:15.877435Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000006291","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:41:41.505224Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000796","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:38:14.730286Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-02-01T10:46:29Z"} 2024-02-01T10:46:29.143 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002204', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:21:48.024326Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004881', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:38:15.877435Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006291', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:41:41.505224Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000796', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:38:14.730286Z'}}] 2024-02-01T10:46:29.143 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003045', 1: '00000000000000000000:00000000000000006141', 2: '00000000000000000000:00000000000000007551', 3: '00000000000000000000:00000000000000002060'} 2024-02-01T10:46:29.143 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002204', 1: '00000000000000000000:00000000000000004881', 2: '00000000000000000000:00000000000000006291', 3: '00000000000000000000:00000000000000000796'} 2024-02-01T10:46:29.143 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000003045 target=00000000000000000000:00000000000000002204, shard=1 master=00000000000000000000:00000000000000006141 target=00000000000000000000:00000000000000004881, shard=2 master=00000000000000000000:00000000000000007551 target=00000000000000000000:00000000000000006291, shard=3 master=00000000000000000000:00000000000000002060 target=00000000000000000000:00000000000000000796 2024-02-01T10:46:32.119 INFO:tasks.rgw.c1.client.0.smithi148.stdout:2024-02-01T10:46:32.115+0000 7f5f40b4a640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-02-01T10:46:32.147 INFO:tasks.rgw.c2.client.0.smithi187.stdout:2024-02-01T10:46:32.147+0000 7f7bead57640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-02-01T10:46:32.191 INFO:tasks.rgw.c2.client.0.smithi187.stdout:2024-02-01T10:46:32.191+0000 7f7be9d55640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-02-01T10:46:34.144 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:46:34.145 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:46:34.145 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:46:34.191 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:46:34.191 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:46:34.272 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":12760245309917388665},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003045","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:46:12.887277Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000006141","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:46:12.888515Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000007551","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:46:12.889691Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002060","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:46:12.890801Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-02-01T10:46:34Z"} 2024-02-01T10:46:34.273 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":12760245309917388665},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003045","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:46:12.887277Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000006141","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:46:12.888515Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000007551","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:46:12.889691Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002060","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:46:12.890801Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-02-01T10:46:34Z"} 2024-02-01T10:46:34.273 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003045', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:46:12.887277Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006141', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:46:12.888515Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007551', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:46:12.889691Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002060', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:46:12.890801Z'}}] 2024-02-01T10:46:34.273 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003045', 1: '00000000000000000000:00000000000000006141', 2: '00000000000000000000:00000000000000007551', 3: '00000000000000000000:00000000000000002060'} 2024-02-01T10:46:34.274 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003045', 1: '00000000000000000000:00000000000000006141', 2: '00000000000000000000:00000000000000007551', 3: '00000000000000000000:00000000000000002060'} 2024-02-01T10:46:34.274 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-02-01T10:46:34.274 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone1 target=test-zone3 2024-02-01T10:46:34.274 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:46:34.274 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:46:34.275 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:46:34.316 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:46:34.316 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:46:34.404 INFO:teuthology.orchestra.run.smithi148.stdout:[{"marker":"00000000000000000000:00000000000000003045","last_update":"2024-02-01T10:46:12.887277Z"},{"marker":"00000000000000000000:00000000000000006141","last_update":"2024-02-01T10:46:12.888515Z"},{"marker":"00000000000000000000:00000000000000007551","last_update":"2024-02-01T10:46:12.889691Z"},{"marker":"00000000000000000000:00000000000000002060","last_update":"2024-02-01T10:46:12.890801Z"}] 2024-02-01T10:46:34.405 DEBUG:rgw_multi.tests:data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000003045', 1: '00000000000000000000:00000000000000006141', 2: '00000000000000000000:00000000000000007551', 3: '00000000000000000000:00000000000000002060'} 2024-02-01T10:46:34.405 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2024-02-01T10:46:34.405 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:46:34.405 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:46:34.405 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:46:34.487 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:46:34.487 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:46:34.562 INFO:teuthology.orchestra.run.smithi148.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":8365149014541277257},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003045","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:46:12.887277Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000006141","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:46:12.888515Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000007551","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:46:12.889691Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002060","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:46:12.890801Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-02-01T10:46:34Z"} 2024-02-01T10:46:34.562 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":8365149014541277257},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003045","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:46:12.887277Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000006141","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:46:12.888515Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000007551","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:46:12.889691Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002060","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:46:12.890801Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-02-01T10:46:34Z"} 2024-02-01T10:46:34.563 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003045', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:46:12.887277Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006141', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:46:12.888515Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007551', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:46:12.889691Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002060', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:46:12.890801Z'}}] 2024-02-01T10:46:34.563 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003045', 1: '00000000000000000000:00000000000000006141', 2: '00000000000000000000:00000000000000007551', 3: '00000000000000000000:00000000000000002060'} 2024-02-01T10:46:34.563 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003045', 1: '00000000000000000000:00000000000000006141', 2: '00000000000000000000:00000000000000007551', 3: '00000000000000000000:00000000000000002060'} 2024-02-01T10:46:34.563 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2024-02-01T10:46:34.563 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone2 target=test-zone1 2024-02-01T10:46:34.563 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:46:34.563 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:46:34.564 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:46:34.605 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:46:34.606 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:46:34.691 INFO:teuthology.orchestra.run.smithi187.stdout:[{"marker":"00000000000000000000:00000000000000003327","last_update":"2024-02-01T10:46:28.576252Z"},{"marker":"00000000000000000000:00000000000000006564","last_update":"2024-02-01T10:46:28.579770Z"},{"marker":"00000000000000000000:00000000000000006002","last_update":"2024-02-01T10:46:28.582580Z"},{"marker":"00000000000000000000:00000000000000002060","last_update":"2024-02-01T10:46:28.585339Z"}] 2024-02-01T10:46:34.692 DEBUG:rgw_multi.tests:data markers for zone=test-zone2 markers={0: '00000000000000000000:00000000000000003327', 1: '00000000000000000000:00000000000000006564', 2: '00000000000000000000:00000000000000006002', 3: '00000000000000000000:00000000000000002060'} 2024-02-01T10:46:34.692 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2024-02-01T10:46:34.692 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:46:34.692 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:46:34.692 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:46:34.737 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:46:34.737 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:46:34.820 INFO:teuthology.orchestra.run.smithi148.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":8595394196901466828},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002486","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:22:17.904424Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000005304","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:38:29.902463Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004742","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:41:41.720312Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000796","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:38:14.839079Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-02-01T10:46:34Z"} 2024-02-01T10:46:34.820 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":8595394196901466828},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002486","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:22:17.904424Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000005304","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:38:29.902463Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004742","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:41:41.720312Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000796","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:38:14.839079Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-02-01T10:46:34Z"} 2024-02-01T10:46:34.820 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002486', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:22:17.904424Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005304', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:38:29.902463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004742', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:41:41.720312Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000796', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:38:14.839079Z'}}] 2024-02-01T10:46:34.821 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003327', 1: '00000000000000000000:00000000000000006564', 2: '00000000000000000000:00000000000000006002', 3: '00000000000000000000:00000000000000002060'} 2024-02-01T10:46:34.821 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002486', 1: '00000000000000000000:00000000000000005304', 2: '00000000000000000000:00000000000000004742', 3: '00000000000000000000:00000000000000000796'} 2024-02-01T10:46:34.821 WARNING:rgw_multi.tests:data of zone test-zone1 behind zone test-zone2: shard=0 master=00000000000000000000:00000000000000003327 target=00000000000000000000:00000000000000002486, shard=1 master=00000000000000000000:00000000000000006564 target=00000000000000000000:00000000000000005304, shard=2 master=00000000000000000000:00000000000000006002 target=00000000000000000000:00000000000000004742, shard=3 master=00000000000000000000:00000000000000002060 target=00000000000000000000:00000000000000000796 2024-02-01T10:46:39.823 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:46:39.823 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:46:39.823 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:46:39.868 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:46:39.869 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:46:39.952 INFO:teuthology.orchestra.run.smithi148.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":8595394196901466828},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002486","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:22:17.904424Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000005304","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:38:29.902463Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004742","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:41:41.720312Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000796","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:38:14.839079Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-02-01T10:46:39Z"} 2024-02-01T10:46:39.953 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":8595394196901466828},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002486","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:22:17.904424Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000005304","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:38:29.902463Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004742","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:41:41.720312Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000796","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:38:14.839079Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-02-01T10:46:39Z"} 2024-02-01T10:46:39.953 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002486', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:22:17.904424Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005304', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:38:29.902463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004742', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:41:41.720312Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000796', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:38:14.839079Z'}}] 2024-02-01T10:46:39.953 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003327', 1: '00000000000000000000:00000000000000006564', 2: '00000000000000000000:00000000000000006002', 3: '00000000000000000000:00000000000000002060'} 2024-02-01T10:46:39.953 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002486', 1: '00000000000000000000:00000000000000005304', 2: '00000000000000000000:00000000000000004742', 3: '00000000000000000000:00000000000000000796'} 2024-02-01T10:46:39.953 WARNING:rgw_multi.tests:data of zone test-zone1 behind zone test-zone2: shard=0 master=00000000000000000000:00000000000000003327 target=00000000000000000000:00000000000000002486, shard=1 master=00000000000000000000:00000000000000006564 target=00000000000000000000:00000000000000005304, shard=2 master=00000000000000000000:00000000000000006002 target=00000000000000000000:00000000000000004742, shard=3 master=00000000000000000000:00000000000000002060 target=00000000000000000000:00000000000000000796 2024-02-01T10:46:42.151 INFO:tasks.rgw.c1.client.1.smithi148.stdout:2024-02-01T10:46:42.147+0000 7f8358f58640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-02-01T10:46:42.155 INFO:tasks.rgw.c1.client.0.smithi148.stdout:2024-02-01T10:46:42.155+0000 7f5f4234d640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-02-01T10:46:44.954 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:46:44.955 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:46:44.955 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:46:44.999 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:46:44.999 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:46:45.081 INFO:teuthology.orchestra.run.smithi148.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":8595394196901466828},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003327","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:46:28.576252Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000006564","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:46:28.579770Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000006002","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:46:28.582580Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002060","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:46:28.585339Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-02-01T10:46:45Z"} 2024-02-01T10:46:45.081 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":8595394196901466828},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003327","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:46:28.576252Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000006564","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:46:28.579770Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000006002","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:46:28.582580Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002060","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:46:28.585339Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-02-01T10:46:45Z"} 2024-02-01T10:46:45.082 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003327', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:46:28.576252Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006564', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:46:28.579770Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006002', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:46:28.582580Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002060', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:46:28.585339Z'}}] 2024-02-01T10:46:45.082 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003327', 1: '00000000000000000000:00000000000000006564', 2: '00000000000000000000:00000000000000006002', 3: '00000000000000000000:00000000000000002060'} 2024-02-01T10:46:45.082 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003327', 1: '00000000000000000000:00000000000000006564', 2: '00000000000000000000:00000000000000006002', 3: '00000000000000000000:00000000000000002060'} 2024-02-01T10:46:45.082 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2024-02-01T10:46:45.082 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone2 target=test-zone3 2024-02-01T10:46:45.082 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:46:45.082 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:46:45.082 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:46:45.124 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:46:45.125 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:46:45.212 INFO:teuthology.orchestra.run.smithi187.stdout:[{"marker":"00000000000000000000:00000000000000003327","last_update":"2024-02-01T10:46:28.576252Z"},{"marker":"00000000000000000000:00000000000000006564","last_update":"2024-02-01T10:46:28.579770Z"},{"marker":"00000000000000000000:00000000000000006002","last_update":"2024-02-01T10:46:28.582580Z"},{"marker":"00000000000000000000:00000000000000002060","last_update":"2024-02-01T10:46:28.585339Z"}] 2024-02-01T10:46:45.213 DEBUG:rgw_multi.tests:data markers for zone=test-zone2 markers={0: '00000000000000000000:00000000000000003327', 1: '00000000000000000000:00000000000000006564', 2: '00000000000000000000:00000000000000006002', 3: '00000000000000000000:00000000000000002060'} 2024-02-01T10:46:45.213 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2024-02-01T10:46:45.213 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:46:45.213 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:46:45.214 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:46:45.259 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:46:45.260 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:46:45.341 INFO:teuthology.orchestra.run.smithi148.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4461638765891857795},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003327","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:46:28.576252Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000006564","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:46:28.579770Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000006002","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:46:28.582580Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002060","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:46:28.585339Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-02-01T10:46:45Z"} 2024-02-01T10:46:45.341 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4461638765891857795},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003327","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:46:28.576252Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000006564","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:46:28.579770Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000006002","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:46:28.582580Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002060","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:46:28.585339Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-02-01T10:46:45Z"} 2024-02-01T10:46:45.342 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003327', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:46:28.576252Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006564', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:46:28.579770Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006002', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:46:28.582580Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002060', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:46:28.585339Z'}}] 2024-02-01T10:46:45.342 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003327', 1: '00000000000000000000:00000000000000006564', 2: '00000000000000000000:00000000000000006002', 3: '00000000000000000000:00000000000000002060'} 2024-02-01T10:46:45.342 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003327', 1: '00000000000000000000:00000000000000006564', 2: '00000000000000000000:00000000000000006002', 3: '00000000000000000000:00000000000000002060'} 2024-02-01T10:46:45.342 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2024-02-01T10:46:45.343 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone3 target=test-zone1 2024-02-01T10:46:45.343 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:46:45.343 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:46:45.343 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:46:45.376 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:46:45.376 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:46:45.456 INFO:teuthology.orchestra.run.smithi148.stdout:[{"marker":"00000000000000000000:00000000000000003750","last_update":"2024-02-01T10:46:18.584528Z"},{"marker":"00000000000000000000:00000000000000006141","last_update":"2024-02-01T10:46:18.587000Z"},{"marker":"00000000000000000000:00000000000000007129","last_update":"2024-02-01T10:46:18.589183Z"},{"marker":"00000000000000000000:00000000000000002060","last_update":"2024-02-01T10:46:18.591871Z"}] 2024-02-01T10:46:45.457 DEBUG:rgw_multi.tests:data markers for zone=test-zone3 markers={0: '00000000000000000000:00000000000000003750', 1: '00000000000000000000:00000000000000006141', 2: '00000000000000000000:00000000000000007129', 3: '00000000000000000000:00000000000000002060'} 2024-02-01T10:46:45.457 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2024-02-01T10:46:45.457 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:46:45.457 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:46:45.458 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:46:45.536 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:46:45.536 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:46:45.614 INFO:teuthology.orchestra.run.smithi148.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":8579032341556541049},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003750","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:46:18.584528Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000006141","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:46:18.587000Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000007129","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:46:18.589183Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002060","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:46:18.591871Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-02-01T10:46:45Z"} 2024-02-01T10:46:45.614 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":8579032341556541049},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003750","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:46:18.584528Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000006141","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:46:18.587000Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000007129","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:46:18.589183Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002060","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:46:18.591871Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-02-01T10:46:45Z"} 2024-02-01T10:46:45.614 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003750', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:46:18.584528Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006141', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:46:18.587000Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007129', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:46:18.589183Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002060', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:46:18.591871Z'}}] 2024-02-01T10:46:45.615 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003750', 1: '00000000000000000000:00000000000000006141', 2: '00000000000000000000:00000000000000007129', 3: '00000000000000000000:00000000000000002060'} 2024-02-01T10:46:45.615 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003750', 1: '00000000000000000000:00000000000000006141', 2: '00000000000000000000:00000000000000007129', 3: '00000000000000000000:00000000000000002060'} 2024-02-01T10:46:45.615 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2024-02-01T10:46:45.615 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone3 target=test-zone2 2024-02-01T10:46:45.615 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:46:45.615 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:46:45.615 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:46:45.693 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:46:45.693 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:46:45.779 INFO:teuthology.orchestra.run.smithi148.stdout:[{"marker":"00000000000000000000:00000000000000003750","last_update":"2024-02-01T10:46:18.584528Z"},{"marker":"00000000000000000000:00000000000000006141","last_update":"2024-02-01T10:46:18.587000Z"},{"marker":"00000000000000000000:00000000000000007129","last_update":"2024-02-01T10:46:18.589183Z"},{"marker":"00000000000000000000:00000000000000002060","last_update":"2024-02-01T10:46:18.591871Z"}] 2024-02-01T10:46:45.779 DEBUG:rgw_multi.tests:data markers for zone=test-zone3 markers={0: '00000000000000000000:00000000000000003750', 1: '00000000000000000000:00000000000000006141', 2: '00000000000000000000:00000000000000007129', 3: '00000000000000000000:00000000000000002060'} 2024-02-01T10:46:45.780 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2024-02-01T10:46:45.780 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:46:45.780 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:46:45.780 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:46:45.825 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:46:45.826 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:46:45.912 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":843725689139632598},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003750","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:46:18.584528Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000006141","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:46:18.587000Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000007129","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:46:18.589183Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002060","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:46:18.591871Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-02-01T10:46:45Z"} 2024-02-01T10:46:45.912 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":843725689139632598},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003750","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:46:18.584528Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000006141","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:46:18.587000Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000007129","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:46:18.589183Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002060","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:46:18.591871Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-02-01T10:46:45Z"} 2024-02-01T10:46:45.912 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003750', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:46:18.584528Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006141', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:46:18.587000Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007129', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:46:18.589183Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002060', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:46:18.591871Z'}}] 2024-02-01T10:46:45.912 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003750', 1: '00000000000000000000:00000000000000006141', 2: '00000000000000000000:00000000000000007129', 3: '00000000000000000000:00000000000000002060'} 2024-02-01T10:46:45.912 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003750', 1: '00000000000000000000:00000000000000006141', 2: '00000000000000000000:00000000000000007129', 3: '00000000000000000000:00000000000000002060'} 2024-02-01T10:46:45.913 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2024-02-01T10:46:45.913 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_disable ... ok 2024-02-01T10:46:45.914 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=vclfcn-59 2024-02-01T10:46:45.914 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:46:45.914 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:46:45.915 DEBUG:boto:Method: PUT 2024-02-01T10:46:45.915 DEBUG:boto:Path: /vclfcn-59/ 2024-02-01T10:46:45.915 DEBUG:boto:Data: 2024-02-01T10:46:45.915 DEBUG:boto:Headers: {} 2024-02-01T10:46:45.915 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:46:45.915 DEBUG:boto:Port: 8000 2024-02-01T10:46:45.915 DEBUG:boto:Params: {} 2024-02-01T10:46:45.916 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T10:46:45.916 DEBUG:boto:Token: None 2024-02-01T10:46:45.916 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:46:45 GMT /vclfcn-59/ 2024-02-01T10:46:45.916 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:rxO+33fWIz/gPPFYPYLGBkAOJ1Y= 2024-02-01T10:46:45.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:46:45 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:rxO+33fWIz/gPPFYPYLGBkAOJ1Y=', 'Content-Length': '0'} 2024-02-01T10:46:45.955 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000037e4eb2a4c85c54-0065bb7695-5765-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:46:45 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:46:45.955 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=vclfcn-60 2024-02-01T10:46:45.955 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:46:45.955 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:46:45.956 DEBUG:boto:Method: PUT 2024-02-01T10:46:45.956 DEBUG:boto:Path: /vclfcn-60/ 2024-02-01T10:46:45.956 DEBUG:boto:Data: 2024-02-01T10:46:45.956 DEBUG:boto:Headers: {} 2024-02-01T10:46:45.956 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:46:45.956 DEBUG:boto:Port: 8000 2024-02-01T10:46:45.956 DEBUG:boto:Params: {} 2024-02-01T10:46:45.956 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T10:46:45.957 DEBUG:boto:Token: None 2024-02-01T10:46:45.957 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:46:45 GMT /vclfcn-60/ 2024-02-01T10:46:45.957 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mUmTHlrLc1y8ygnx6rPJgT7R/6w= 2024-02-01T10:46:45.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:46:45 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mUmTHlrLc1y8ygnx6rPJgT7R/6w=', 'Content-Length': '0'} 2024-02-01T10:46:46.020 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d04aa0d4e5e2e2b6-0065bb7695-5350-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:46:46.020 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=vclfcn-61 2024-02-01T10:46:46.020 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:46:46.021 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:46:46.021 DEBUG:boto:Method: PUT 2024-02-01T10:46:46.021 DEBUG:boto:Path: /vclfcn-61/ 2024-02-01T10:46:46.021 DEBUG:boto:Data: 2024-02-01T10:46:46.021 DEBUG:boto:Headers: {} 2024-02-01T10:46:46.021 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:46:46.021 DEBUG:boto:Port: 8001 2024-02-01T10:46:46.021 DEBUG:boto:Params: {} 2024-02-01T10:46:46.021 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-02-01T10:46:46.022 DEBUG:boto:Token: None 2024-02-01T10:46:46.022 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:46:46 GMT /vclfcn-61/ 2024-02-01T10:46:46.022 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ohLQmYfTKUl8kPGfV+o5P3moa0k= 2024-02-01T10:46:46.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:46:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ohLQmYfTKUl8kPGfV+o5P3moa0k=', 'Content-Length': '0'} 2024-02-01T10:46:46.079 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fc07e849e62ad80b-0065bb7696-5762-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:46:46.080 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:46:46.080 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:46:46.080 DEBUG:boto:Method: HEAD 2024-02-01T10:46:46.080 DEBUG:boto:Path: /vclfcn-59/ 2024-02-01T10:46:46.080 DEBUG:boto:Data: 2024-02-01T10:46:46.080 DEBUG:boto:Headers: {} 2024-02-01T10:46:46.080 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:46:46.080 DEBUG:boto:Port: 8000 2024-02-01T10:46:46.081 DEBUG:boto:Params: {} 2024-02-01T10:46:46.081 DEBUG:boto:Token: None 2024-02-01T10:46:46.081 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:46:46 GMT /vclfcn-59/ 2024-02-01T10:46:46.081 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9uA89Z2nh6RzYy1dmQsOsevh65E= 2024-02-01T10:46:46.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:46:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9uA89Z2nh6RzYy1dmQsOsevh65E=', 'Content-Length': '0'} 2024-02-01T10:46:46.085 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000eb028d8cbd2f934e-0065bb7696-5765-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:46:46.085 DEBUG:boto:path=/vclfcn-59/obj1 2024-02-01T10:46:46.085 DEBUG:boto:auth_path=/vclfcn-59/obj1 2024-02-01T10:46:46.085 DEBUG:boto:Method: PUT 2024-02-01T10:46:46.085 DEBUG:boto:Path: /vclfcn-59/obj1 2024-02-01T10:46:46.086 DEBUG:boto:Data: 2024-02-01T10:46:46.086 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-02-01T10:46:46.086 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:46:46.086 DEBUG:boto:Port: 8000 2024-02-01T10:46:46.086 DEBUG:boto:Params: {} 2024-02-01T10:46:46.086 DEBUG:boto:Token: None 2024-02-01T10:46:46.086 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 01 Feb 2024 10:46:46 GMT /vclfcn-59/obj1 2024-02-01T10:46:46.086 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:pUTGL4IITk7qf699Ens21Q3af2I= 2024-02-01T10:46:46.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:46:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:pUTGL4IITk7qf699Ens21Q3af2I='} 2024-02-01T10:46:46.099 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000db34e0d5ec59f132-0065bb7696-5765-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:46:46.100 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:46:46.100 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:46:46.100 DEBUG:boto:Method: HEAD 2024-02-01T10:46:46.100 DEBUG:boto:Path: /vclfcn-59/ 2024-02-01T10:46:46.100 DEBUG:boto:Data: 2024-02-01T10:46:46.100 DEBUG:boto:Headers: {} 2024-02-01T10:46:46.100 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:46:46.100 DEBUG:boto:Port: 8000 2024-02-01T10:46:46.100 DEBUG:boto:Params: {} 2024-02-01T10:46:46.101 DEBUG:boto:Token: None 2024-02-01T10:46:46.101 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:46:46 GMT /vclfcn-59/ 2024-02-01T10:46:46.101 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9uA89Z2nh6RzYy1dmQsOsevh65E= 2024-02-01T10:46:46.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:46:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9uA89Z2nh6RzYy1dmQsOsevh65E=', 'Content-Length': '0'} 2024-02-01T10:46:46.104 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003cce3ca81ca883e0-0065bb7696-5765-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:46:46.105 DEBUG:boto:path=/vclfcn-59/obj2 2024-02-01T10:46:46.105 DEBUG:boto:auth_path=/vclfcn-59/obj2 2024-02-01T10:46:46.105 DEBUG:boto:Method: PUT 2024-02-01T10:46:46.105 DEBUG:boto:Path: /vclfcn-59/obj2 2024-02-01T10:46:46.105 DEBUG:boto:Data: 2024-02-01T10:46:46.105 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-02-01T10:46:46.105 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:46:46.105 DEBUG:boto:Port: 8000 2024-02-01T10:46:46.105 DEBUG:boto:Params: {} 2024-02-01T10:46:46.106 DEBUG:boto:Token: None 2024-02-01T10:46:46.106 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 01 Feb 2024 10:46:46 GMT /vclfcn-59/obj2 2024-02-01T10:46:46.106 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tvUNf28GU/YCK0fF3cIqh0tyMIY= 2024-02-01T10:46:46.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:46:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tvUNf28GU/YCK0fF3cIqh0tyMIY='} 2024-02-01T10:46:46.148 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002151ea080c7063c9-0065bb7696-5765-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:46:46.148 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:46:46.149 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:46:46.149 DEBUG:boto:Method: HEAD 2024-02-01T10:46:46.149 DEBUG:boto:Path: /vclfcn-59/ 2024-02-01T10:46:46.149 DEBUG:boto:Data: 2024-02-01T10:46:46.149 DEBUG:boto:Headers: {} 2024-02-01T10:46:46.149 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:46:46.149 DEBUG:boto:Port: 8000 2024-02-01T10:46:46.150 DEBUG:boto:Params: {} 2024-02-01T10:46:46.150 DEBUG:boto:Token: None 2024-02-01T10:46:46.150 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:46:46 GMT /vclfcn-59/ 2024-02-01T10:46:46.150 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9uA89Z2nh6RzYy1dmQsOsevh65E= 2024-02-01T10:46:46.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:46:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9uA89Z2nh6RzYy1dmQsOsevh65E=', 'Content-Length': '0'} 2024-02-01T10:46:46.154 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a00537706ce307b3-0065bb7696-5765-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:46:46.155 DEBUG:boto:path=/vclfcn-59/obj3 2024-02-01T10:46:46.155 DEBUG:boto:auth_path=/vclfcn-59/obj3 2024-02-01T10:46:46.155 DEBUG:boto:Method: PUT 2024-02-01T10:46:46.155 DEBUG:boto:Path: /vclfcn-59/obj3 2024-02-01T10:46:46.155 DEBUG:boto:Data: 2024-02-01T10:46:46.155 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-02-01T10:46:46.155 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:46:46.155 DEBUG:boto:Port: 8000 2024-02-01T10:46:46.155 DEBUG:boto:Params: {} 2024-02-01T10:46:46.155 DEBUG:boto:Token: None 2024-02-01T10:46:46.156 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 01 Feb 2024 10:46:46 GMT /vclfcn-59/obj3 2024-02-01T10:46:46.156 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lIImDgEfczAlItsemcHJ0eiciOc= 2024-02-01T10:46:46.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:46:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lIImDgEfczAlItsemcHJ0eiciOc='} 2024-02-01T10:46:46.171 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008a2a6cb625e6e7df-0065bb7696-5765-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:46:46.171 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:46:46.171 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:46:46.171 DEBUG:boto:Method: HEAD 2024-02-01T10:46:46.171 DEBUG:boto:Path: /vclfcn-59/ 2024-02-01T10:46:46.172 DEBUG:boto:Data: 2024-02-01T10:46:46.172 DEBUG:boto:Headers: {} 2024-02-01T10:46:46.172 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:46:46.172 DEBUG:boto:Port: 8000 2024-02-01T10:46:46.172 DEBUG:boto:Params: {} 2024-02-01T10:46:46.172 DEBUG:boto:Token: None 2024-02-01T10:46:46.172 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:46:46 GMT /vclfcn-59/ 2024-02-01T10:46:46.172 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9uA89Z2nh6RzYy1dmQsOsevh65E= 2024-02-01T10:46:46.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:46:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9uA89Z2nh6RzYy1dmQsOsevh65E=', 'Content-Length': '0'} 2024-02-01T10:46:46.180 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005308420aba3013f2-0065bb7696-5765-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:46:46.183 DEBUG:boto:path=/vclfcn-59/obj4 2024-02-01T10:46:46.183 DEBUG:boto:auth_path=/vclfcn-59/obj4 2024-02-01T10:46:46.183 DEBUG:boto:Method: PUT 2024-02-01T10:46:46.183 DEBUG:boto:Path: /vclfcn-59/obj4 2024-02-01T10:46:46.183 DEBUG:boto:Data: 2024-02-01T10:46:46.183 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-02-01T10:46:46.183 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:46:46.184 DEBUG:boto:Port: 8000 2024-02-01T10:46:46.184 DEBUG:boto:Params: {} 2024-02-01T10:46:46.184 DEBUG:boto:Token: None 2024-02-01T10:46:46.184 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 01 Feb 2024 10:46:46 GMT /vclfcn-59/obj4 2024-02-01T10:46:46.184 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/2M0zXPB3ZG6t7cCa5ZL2G39o40= 2024-02-01T10:46:46.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:46:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/2M0zXPB3ZG6t7cCa5ZL2G39o40='} 2024-02-01T10:46:46.228 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007ceb61ab0f06c674-0065bb7696-5765-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:46:46.229 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:46:46.229 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:46:46.229 DEBUG:boto:Method: HEAD 2024-02-01T10:46:46.229 DEBUG:boto:Path: /vclfcn-60/ 2024-02-01T10:46:46.229 DEBUG:boto:Data: 2024-02-01T10:46:46.229 DEBUG:boto:Headers: {} 2024-02-01T10:46:46.229 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:46:46.229 DEBUG:boto:Port: 8000 2024-02-01T10:46:46.229 DEBUG:boto:Params: {} 2024-02-01T10:46:46.230 DEBUG:boto:Token: None 2024-02-01T10:46:46.230 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:46:46 GMT /vclfcn-60/ 2024-02-01T10:46:46.230 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:x7lKiW1MMwfdEgBKYF/n8Qwcu7A= 2024-02-01T10:46:46.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:46:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:x7lKiW1MMwfdEgBKYF/n8Qwcu7A=', 'Content-Length': '0'} 2024-02-01T10:46:46.235 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f3f704926cc2163a-0065bb7696-5350-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:46:46.235 DEBUG:boto:path=/vclfcn-60/obj1 2024-02-01T10:46:46.235 DEBUG:boto:auth_path=/vclfcn-60/obj1 2024-02-01T10:46:46.235 DEBUG:boto:Method: PUT 2024-02-01T10:46:46.236 DEBUG:boto:Path: /vclfcn-60/obj1 2024-02-01T10:46:46.236 DEBUG:boto:Data: 2024-02-01T10:46:46.236 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-02-01T10:46:46.236 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:46:46.236 DEBUG:boto:Port: 8000 2024-02-01T10:46:46.237 DEBUG:boto:Params: {} 2024-02-01T10:46:46.237 DEBUG:boto:Token: None 2024-02-01T10:46:46.237 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 01 Feb 2024 10:46:46 GMT /vclfcn-60/obj1 2024-02-01T10:46:46.237 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:VHiswJhQdx+p75rrN2w6qaPdmKg= 2024-02-01T10:46:46.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:46:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:VHiswJhQdx+p75rrN2w6qaPdmKg='} 2024-02-01T10:46:46.253 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006783880fbaa6f566-0065bb7696-5350-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:46:46.253 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:46:46.253 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:46:46.253 DEBUG:boto:Method: HEAD 2024-02-01T10:46:46.253 DEBUG:boto:Path: /vclfcn-60/ 2024-02-01T10:46:46.254 DEBUG:boto:Data: 2024-02-01T10:46:46.254 DEBUG:boto:Headers: {} 2024-02-01T10:46:46.254 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:46:46.254 DEBUG:boto:Port: 8000 2024-02-01T10:46:46.254 DEBUG:boto:Params: {} 2024-02-01T10:46:46.254 DEBUG:boto:Token: None 2024-02-01T10:46:46.255 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:46:46 GMT /vclfcn-60/ 2024-02-01T10:46:46.255 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:x7lKiW1MMwfdEgBKYF/n8Qwcu7A= 2024-02-01T10:46:46.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:46:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:x7lKiW1MMwfdEgBKYF/n8Qwcu7A=', 'Content-Length': '0'} 2024-02-01T10:46:46.260 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fd2871f06002d6d4-0065bb7696-5350-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:46:46.260 DEBUG:boto:path=/vclfcn-60/obj2 2024-02-01T10:46:46.260 DEBUG:boto:auth_path=/vclfcn-60/obj2 2024-02-01T10:46:46.260 DEBUG:boto:Method: PUT 2024-02-01T10:46:46.260 DEBUG:boto:Path: /vclfcn-60/obj2 2024-02-01T10:46:46.260 DEBUG:boto:Data: 2024-02-01T10:46:46.260 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-02-01T10:46:46.260 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:46:46.261 DEBUG:boto:Port: 8000 2024-02-01T10:46:46.261 DEBUG:boto:Params: {} 2024-02-01T10:46:46.261 DEBUG:boto:Token: None 2024-02-01T10:46:46.261 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 01 Feb 2024 10:46:46 GMT /vclfcn-60/obj2 2024-02-01T10:46:46.261 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:pdAJeWtlhAeHEN2P4vvmEZvActA= 2024-02-01T10:46:46.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:46:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:pdAJeWtlhAeHEN2P4vvmEZvActA='} 2024-02-01T10:46:46.272 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000910e88a8de245c2f-0065bb7696-5350-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:46:46.273 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:46:46.273 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:46:46.273 DEBUG:boto:Method: HEAD 2024-02-01T10:46:46.273 DEBUG:boto:Path: /vclfcn-60/ 2024-02-01T10:46:46.273 DEBUG:boto:Data: 2024-02-01T10:46:46.273 DEBUG:boto:Headers: {} 2024-02-01T10:46:46.273 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:46:46.273 DEBUG:boto:Port: 8000 2024-02-01T10:46:46.273 DEBUG:boto:Params: {} 2024-02-01T10:46:46.274 DEBUG:boto:Token: None 2024-02-01T10:46:46.274 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:46:46 GMT /vclfcn-60/ 2024-02-01T10:46:46.274 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:x7lKiW1MMwfdEgBKYF/n8Qwcu7A= 2024-02-01T10:46:46.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:46:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:x7lKiW1MMwfdEgBKYF/n8Qwcu7A=', 'Content-Length': '0'} 2024-02-01T10:46:46.279 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007cda8a2e6be38540-0065bb7696-5350-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:46:46.279 DEBUG:boto:path=/vclfcn-60/obj3 2024-02-01T10:46:46.279 DEBUG:boto:auth_path=/vclfcn-60/obj3 2024-02-01T10:46:46.280 DEBUG:boto:Method: PUT 2024-02-01T10:46:46.280 DEBUG:boto:Path: /vclfcn-60/obj3 2024-02-01T10:46:46.280 DEBUG:boto:Data: 2024-02-01T10:46:46.280 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-02-01T10:46:46.280 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:46:46.280 DEBUG:boto:Port: 8000 2024-02-01T10:46:46.280 DEBUG:boto:Params: {} 2024-02-01T10:46:46.280 DEBUG:boto:Token: None 2024-02-01T10:46:46.280 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 01 Feb 2024 10:46:46 GMT /vclfcn-60/obj3 2024-02-01T10:46:46.281 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:AM/TXHdEqMTUSa7i978syxwsECE= 2024-02-01T10:46:46.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:46:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:AM/TXHdEqMTUSa7i978syxwsECE='} 2024-02-01T10:46:46.324 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001121923421853240-0065bb7696-5350-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:46:46.325 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:46:46.325 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:46:46.325 DEBUG:boto:Method: HEAD 2024-02-01T10:46:46.325 DEBUG:boto:Path: /vclfcn-60/ 2024-02-01T10:46:46.325 DEBUG:boto:Data: 2024-02-01T10:46:46.325 DEBUG:boto:Headers: {} 2024-02-01T10:46:46.325 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:46:46.325 DEBUG:boto:Port: 8000 2024-02-01T10:46:46.325 DEBUG:boto:Params: {} 2024-02-01T10:46:46.326 DEBUG:boto:Token: None 2024-02-01T10:46:46.326 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:46:46 GMT /vclfcn-60/ 2024-02-01T10:46:46.326 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:x7lKiW1MMwfdEgBKYF/n8Qwcu7A= 2024-02-01T10:46:46.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:46:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:x7lKiW1MMwfdEgBKYF/n8Qwcu7A=', 'Content-Length': '0'} 2024-02-01T10:46:46.330 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000447b9e7f0ddbb1a-0065bb7696-5350-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:46:46.330 DEBUG:boto:path=/vclfcn-60/obj4 2024-02-01T10:46:46.330 DEBUG:boto:auth_path=/vclfcn-60/obj4 2024-02-01T10:46:46.331 DEBUG:boto:Method: PUT 2024-02-01T10:46:46.331 DEBUG:boto:Path: /vclfcn-60/obj4 2024-02-01T10:46:46.331 DEBUG:boto:Data: 2024-02-01T10:46:46.331 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-02-01T10:46:46.331 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:46:46.331 DEBUG:boto:Port: 8000 2024-02-01T10:46:46.331 DEBUG:boto:Params: {} 2024-02-01T10:46:46.331 DEBUG:boto:Token: None 2024-02-01T10:46:46.331 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 01 Feb 2024 10:46:46 GMT /vclfcn-60/obj4 2024-02-01T10:46:46.331 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Piu41vDMAUh9SZwsw+8Tza8c6nY= 2024-02-01T10:46:46.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:46:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Piu41vDMAUh9SZwsw+8Tza8c6nY='} 2024-02-01T10:46:46.344 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bf8752129c204424-0065bb7696-5350-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:46:46.344 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:46:46.344 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:46:46.344 DEBUG:boto:Method: HEAD 2024-02-01T10:46:46.344 DEBUG:boto:Path: /vclfcn-61/ 2024-02-01T10:46:46.344 DEBUG:boto:Data: 2024-02-01T10:46:46.344 DEBUG:boto:Headers: {} 2024-02-01T10:46:46.344 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:46:46.344 DEBUG:boto:Port: 8001 2024-02-01T10:46:46.345 DEBUG:boto:Params: {} 2024-02-01T10:46:46.345 DEBUG:boto:Token: None 2024-02-01T10:46:46.345 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:46:46 GMT /vclfcn-61/ 2024-02-01T10:46:46.345 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:UxejqhAH9kImSMZ1+JOZZFwjb6M= 2024-02-01T10:46:46.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:46:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:UxejqhAH9kImSMZ1+JOZZFwjb6M=', 'Content-Length': '0'} 2024-02-01T10:46:46.350 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b648a55adda02578-0065bb7696-5762-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:46:46.350 DEBUG:boto:path=/vclfcn-61/obj1 2024-02-01T10:46:46.350 DEBUG:boto:auth_path=/vclfcn-61/obj1 2024-02-01T10:46:46.350 DEBUG:boto:Method: PUT 2024-02-01T10:46:46.351 DEBUG:boto:Path: /vclfcn-61/obj1 2024-02-01T10:46:46.351 DEBUG:boto:Data: 2024-02-01T10:46:46.351 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-02-01T10:46:46.351 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:46:46.351 DEBUG:boto:Port: 8001 2024-02-01T10:46:46.351 DEBUG:boto:Params: {} 2024-02-01T10:46:46.352 DEBUG:boto:Token: None 2024-02-01T10:46:46.352 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 01 Feb 2024 10:46:46 GMT /vclfcn-61/obj1 2024-02-01T10:46:46.352 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:eN9YxcuJowMMNeqUfHV0gH02tW4= 2024-02-01T10:46:46.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:46:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:eN9YxcuJowMMNeqUfHV0gH02tW4='} 2024-02-01T10:46:46.367 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a93f5dbad94a4233-0065bb7696-5762-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:46:46.367 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:46:46.368 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:46:46.368 DEBUG:boto:Method: HEAD 2024-02-01T10:46:46.368 DEBUG:boto:Path: /vclfcn-61/ 2024-02-01T10:46:46.368 DEBUG:boto:Data: 2024-02-01T10:46:46.368 DEBUG:boto:Headers: {} 2024-02-01T10:46:46.368 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:46:46.369 DEBUG:boto:Port: 8001 2024-02-01T10:46:46.369 DEBUG:boto:Params: {} 2024-02-01T10:46:46.369 DEBUG:boto:Token: None 2024-02-01T10:46:46.369 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:46:46 GMT /vclfcn-61/ 2024-02-01T10:46:46.369 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:UxejqhAH9kImSMZ1+JOZZFwjb6M= 2024-02-01T10:46:46.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:46:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:UxejqhAH9kImSMZ1+JOZZFwjb6M=', 'Content-Length': '0'} 2024-02-01T10:46:46.373 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004d0994b2abdda8b6-0065bb7696-5762-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:46:46.373 DEBUG:boto:path=/vclfcn-61/obj2 2024-02-01T10:46:46.374 DEBUG:boto:auth_path=/vclfcn-61/obj2 2024-02-01T10:46:46.374 DEBUG:boto:Method: PUT 2024-02-01T10:46:46.374 DEBUG:boto:Path: /vclfcn-61/obj2 2024-02-01T10:46:46.374 DEBUG:boto:Data: 2024-02-01T10:46:46.374 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-02-01T10:46:46.374 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:46:46.374 DEBUG:boto:Port: 8001 2024-02-01T10:46:46.374 DEBUG:boto:Params: {} 2024-02-01T10:46:46.374 DEBUG:boto:Token: None 2024-02-01T10:46:46.375 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 01 Feb 2024 10:46:46 GMT /vclfcn-61/obj2 2024-02-01T10:46:46.375 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:TbAQ5A8Y05ai5JDP+EisySsdeV4= 2024-02-01T10:46:46.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:46:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:TbAQ5A8Y05ai5JDP+EisySsdeV4='} 2024-02-01T10:46:46.420 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002fd1aed11f122fc2-0065bb7696-5762-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:46:46.420 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:46:46.421 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:46:46.421 DEBUG:boto:Method: HEAD 2024-02-01T10:46:46.421 DEBUG:boto:Path: /vclfcn-61/ 2024-02-01T10:46:46.421 DEBUG:boto:Data: 2024-02-01T10:46:46.421 DEBUG:boto:Headers: {} 2024-02-01T10:46:46.421 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:46:46.421 DEBUG:boto:Port: 8001 2024-02-01T10:46:46.421 DEBUG:boto:Params: {} 2024-02-01T10:46:46.421 DEBUG:boto:Token: None 2024-02-01T10:46:46.422 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:46:46 GMT /vclfcn-61/ 2024-02-01T10:46:46.422 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:UxejqhAH9kImSMZ1+JOZZFwjb6M= 2024-02-01T10:46:46.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:46:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:UxejqhAH9kImSMZ1+JOZZFwjb6M=', 'Content-Length': '0'} 2024-02-01T10:46:46.427 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000050b6bca5575c91ba-0065bb7696-5762-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:46:46.427 DEBUG:boto:path=/vclfcn-61/obj3 2024-02-01T10:46:46.427 DEBUG:boto:auth_path=/vclfcn-61/obj3 2024-02-01T10:46:46.427 DEBUG:boto:Method: PUT 2024-02-01T10:46:46.427 DEBUG:boto:Path: /vclfcn-61/obj3 2024-02-01T10:46:46.427 DEBUG:boto:Data: 2024-02-01T10:46:46.428 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-02-01T10:46:46.428 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:46:46.428 DEBUG:boto:Port: 8001 2024-02-01T10:46:46.428 DEBUG:boto:Params: {} 2024-02-01T10:46:46.428 DEBUG:boto:Token: None 2024-02-01T10:46:46.428 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 01 Feb 2024 10:46:46 GMT /vclfcn-61/obj3 2024-02-01T10:46:46.428 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:O02cN23rl+goRkbUoE5Flod+H7U= 2024-02-01T10:46:46.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:46:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:O02cN23rl+goRkbUoE5Flod+H7U='} 2024-02-01T10:46:46.440 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c152f668d6160a7e-0065bb7696-5762-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:46:46.440 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:46:46.440 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:46:46.440 DEBUG:boto:Method: HEAD 2024-02-01T10:46:46.440 DEBUG:boto:Path: /vclfcn-61/ 2024-02-01T10:46:46.440 DEBUG:boto:Data: 2024-02-01T10:46:46.441 DEBUG:boto:Headers: {} 2024-02-01T10:46:46.441 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:46:46.441 DEBUG:boto:Port: 8001 2024-02-01T10:46:46.441 DEBUG:boto:Params: {} 2024-02-01T10:46:46.441 DEBUG:boto:Token: None 2024-02-01T10:46:46.441 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:46:46 GMT /vclfcn-61/ 2024-02-01T10:46:46.441 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:UxejqhAH9kImSMZ1+JOZZFwjb6M= 2024-02-01T10:46:46.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:46:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:UxejqhAH9kImSMZ1+JOZZFwjb6M=', 'Content-Length': '0'} 2024-02-01T10:46:46.446 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b5dda4d0fd6b5f26-0065bb7696-5762-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:46:46.446 DEBUG:boto:path=/vclfcn-61/obj4 2024-02-01T10:46:46.446 DEBUG:boto:auth_path=/vclfcn-61/obj4 2024-02-01T10:46:46.446 DEBUG:boto:Method: PUT 2024-02-01T10:46:46.446 DEBUG:boto:Path: /vclfcn-61/obj4 2024-02-01T10:46:46.446 DEBUG:boto:Data: 2024-02-01T10:46:46.446 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-02-01T10:46:46.447 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:46:46.447 DEBUG:boto:Port: 8001 2024-02-01T10:46:46.447 DEBUG:boto:Params: {} 2024-02-01T10:46:46.447 DEBUG:boto:Token: None 2024-02-01T10:46:46.447 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 01 Feb 2024 10:46:46 GMT /vclfcn-61/obj4 2024-02-01T10:46:46.447 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:y5Xk0M4ginGDsaiacrmk9rhu+p8= 2024-02-01T10:46:46.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:46:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:y5Xk0M4ginGDsaiacrmk9rhu+p8='} 2024-02-01T10:46:46.492 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cad8a6047f97ab5b-0065bb7696-5762-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:46:46.493 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:46:46.493 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:46:46.493 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:46:46.533 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:46:46.533 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:46:46.590 INFO:teuthology.orchestra.run.smithi148.stderr:No --period given, using current period=f2e6c290-e7aa-4279-b77e-15403b18d61f 2024-02-01T10:46:46.613 INFO:teuthology.orchestra.run.smithi148.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1706784406.060246_262.1","last_update":"2024-02-01T10:46:46.060246Z"},{"marker":"","last_update":"0.000000"}] 2024-02-01T10:46:46.614 DEBUG:rgw_multi.tests:master meta markers={0: '', 1: '', 2: '1_1706784406.060246_262.1', 3: ''} 2024-02-01T10:46:46.614 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-02-01T10:46:46.614 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:46:46.614 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:46:46.614 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:46:46.656 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:46:46.657 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:46:46.711 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:46:46.711+0000 7faec9ea1980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:46:46.712 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:46:46.711+0000 7faec9ea1980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:46:46.736 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706784406.054828_261.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:46:46.054828Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:46:46Z"} 2024-02-01T10:46:46.736 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706784406.054828_261.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:46:46.054828Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:46:46Z"} 2024-02-01T10:46:46.737 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706784406.054828_261.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:46:46.054828Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:46:46.737 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1706784406.060246_262.1', 3: ''} 2024-02-01T10:46:46.737 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1706784406.054828_261.1', 3: ''} 2024-02-01T10:46:46.737 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1706784406.060246_262.1 target=1_1706784406.054828_261.1 2024-02-01T10:46:51.738 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:46:51.738 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:46:51.739 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:46:51.782 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:46:51.782 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:46:51.838 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:46:51.835+0000 7f732ad99980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:46:51.839 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:46:51.839+0000 7f732ad99980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:46:51.865 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706784406.060246_262.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:46:46.060246Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:46:51Z"} 2024-02-01T10:46:51.865 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706784406.060246_262.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:46:46.060246Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:46:51Z"} 2024-02-01T10:46:51.865 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706784406.060246_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:46:46.060246Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:46:51.865 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1706784406.060246_262.1', 3: ''} 2024-02-01T10:46:51.866 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1706784406.060246_262.1', 3: ''} 2024-02-01T10:46:51.866 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-02-01T10:46:51.866 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-02-01T10:46:51.866 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:46:51.866 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:46:51.866 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:46:51.908 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:46:51.908 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:46:51.963 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:46:51.963+0000 7fef42564980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:46:51.964 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:46:51.963+0000 7fef42564980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:46:51.987 INFO:teuthology.orchestra.run.smithi148.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706784406.060246_262.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:46:46.060246Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:46:51Z"} 2024-02-01T10:46:51.987 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706784406.060246_262.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:46:46.060246Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:46:51Z"} 2024-02-01T10:46:51.988 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706784406.060246_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:46:46.060246Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:46:51.988 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1706784406.060246_262.1', 3: ''} 2024-02-01T10:46:51.988 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1706784406.060246_262.1', 3: ''} 2024-02-01T10:46:51.988 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-02-01T10:46:51.988 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=vclfcn-59 2024-02-01T10:46:51.988 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:46:51.988 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:46:51.989 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket vclfcn-59 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-02-01T10:46:52.038 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:46:52.038 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:46:52.101 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:46:52.099+0000 7fded28fb980 1 waiting to reach incremental sync.. 2024-02-01T10:46:57.105 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:46:57.103+0000 7fded28fb980 1 bucket sync caught up with source: 2024-02-01T10:46:57.105 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [00000000001.40.6, , 00000000001.40.6, , 00000000001.38.6, , , 00000000001.52.6, , , ] 2024-02-01T10:46:57.106 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [00000000001.40.6, , 00000000001.40.6, , 00000000001.38.6, , , 00000000001.52.6, , , ] 2024-02-01T10:46:57.106 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:46:57.103+0000 7fded28fb980 0 bucket checkpoint complete 2024-02-01T10:46:57.145 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=vclfcn-59 2024-02-01T10:46:57.145 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:46:57.146 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:46:57.146 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-59 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:46:57.195 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:46:57.196 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:46:57.257 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:46:57.254+0000 7feea6d54980 1 waiting to reach incremental sync.. 2024-02-01T10:47:02.258 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:47:02.254+0000 7feea6d54980 1 waiting to reach incremental sync.. 2024-02-01T10:47:07.262 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:47:07.258+0000 7feea6d54980 1 bucket sync caught up with source: 2024-02-01T10:47:07.262 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [00000000001.40.6, , 00000000001.40.6, , 00000000001.38.6, , , 00000000001.52.6, , , ] 2024-02-01T10:47:07.262 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [00000000001.40.6, , 00000000001.40.6, , 00000000001.38.6, , , 00000000001.52.6, , , ] 2024-02-01T10:47:07.262 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:47:07.258+0000 7feea6d54980 0 bucket checkpoint complete 2024-02-01T10:47:07.302 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=vclfcn-59 2024-02-01T10:47:07.303 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-59', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:47:07.303 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-59', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:47:07.303 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-59 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:47:07.351 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:47:07.351 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:47:07.415 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:47:07.414+0000 7fd1120b5980 1 bucket sync caught up with source: 2024-02-01T10:47:07.415 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [00000000001.229.6, , 00000000001.192.6, , 00000000001.227.6, , , 00000000001.264.6, , , ] 2024-02-01T10:47:07.415 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [00000000001.229.6, , 00000000001.192.6, , 00000000001.227.6, , , 00000000001.264.6, , , ] 2024-02-01T10:47:07.416 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:47:07.414+0000 7fd1120b5980 0 bucket checkpoint complete 2024-02-01T10:47:07.439 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=vclfcn-59 2024-02-01T10:47:07.439 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-59', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:47:07.439 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-59', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:47:07.440 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-59 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:47:07.480 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:47:07.480 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:47:07.539 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:47:07.534+0000 7f847ed17980 1 bucket sync caught up with source: 2024-02-01T10:47:07.540 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [00000000001.229.6, , 00000000001.192.6, , 00000000001.227.6, , , 00000000001.264.6, , , ] 2024-02-01T10:47:07.540 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [00000000001.229.6, , 00000000001.192.6, , 00000000001.227.6, , , 00000000001.264.6, , , ] 2024-02-01T10:47:07.540 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:47:07.538+0000 7f847ed17980 0 bucket checkpoint complete 2024-02-01T10:47:07.561 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=vclfcn-59 2024-02-01T10:47:07.562 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-59', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:47:07.562 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-59', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:47:07.562 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-59 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:47:07.644 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:47:07.645 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:47:07.701 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:47:07.698+0000 7f2432c84980 1 waiting to reach incremental sync.. 2024-02-01T10:47:12.702 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:47:12.698+0000 7f2432c84980 1 waiting to reach incremental sync.. 2024-02-01T10:47:17.705 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:47:17.706+0000 7f2432c84980 1 bucket sync caught up with source: 2024-02-01T10:47:17.705 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [00000000001.175.6, , 00000000002.77.6, , 00000000002.176.6, , , 00000000001.173.6, , , ] 2024-02-01T10:47:17.705 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [00000000001.175.6, , 00000000002.77.6, , 00000000002.176.6, , , 00000000001.173.6, , , ] 2024-02-01T10:47:17.705 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:47:17.706+0000 7f2432c84980 0 bucket checkpoint complete 2024-02-01T10:47:17.753 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=vclfcn-59 2024-02-01T10:47:17.754 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-59', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:47:17.754 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-59', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:47:17.754 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket vclfcn-59 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-02-01T10:47:17.804 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:47:17.804 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:47:17.871 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:47:17.870+0000 7f837a875980 1 bucket sync caught up with source: 2024-02-01T10:47:17.871 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [00000000001.175.6, , 00000000002.77.6, , 00000000002.176.6, , , 00000000001.173.6, , , ] 2024-02-01T10:47:17.871 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [00000000001.175.6, , 00000000002.77.6, , 00000000002.176.6, , , 00000000001.173.6, , , ] 2024-02-01T10:47:17.871 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:47:17.870+0000 7f837a875980 0 bucket checkpoint complete 2024-02-01T10:47:17.894 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-59 zones={test-zone2, test-zone1} 2024-02-01T10:47:17.895 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:47:17.895 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:47:17.895 DEBUG:boto:Method: HEAD 2024-02-01T10:47:17.895 DEBUG:boto:Path: /vclfcn-59/ 2024-02-01T10:47:17.895 DEBUG:boto:Data: 2024-02-01T10:47:17.895 DEBUG:boto:Headers: {} 2024-02-01T10:47:17.895 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:17.895 DEBUG:boto:Port: 8000 2024-02-01T10:47:17.896 DEBUG:boto:Params: {} 2024-02-01T10:47:17.896 DEBUG:boto:Token: None 2024-02-01T10:47:17.896 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:17 GMT /vclfcn-59/ 2024-02-01T10:47:17.896 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vSDQyTSLn/L3miMAiZ5pvhY2Sh4= 2024-02-01T10:47:17.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vSDQyTSLn/L3miMAiZ5pvhY2Sh4=', 'Content-Length': '0'} 2024-02-01T10:47:17.899 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000004af38f404cb98fc-0065bb76b5-5350-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:17.899 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:47:17.899 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:47:17.899 DEBUG:boto:Method: HEAD 2024-02-01T10:47:17.900 DEBUG:boto:Path: /vclfcn-59/ 2024-02-01T10:47:17.900 DEBUG:boto:Data: 2024-02-01T10:47:17.900 DEBUG:boto:Headers: {} 2024-02-01T10:47:17.900 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:17.900 DEBUG:boto:Port: 8000 2024-02-01T10:47:17.900 DEBUG:boto:Params: {} 2024-02-01T10:47:17.900 DEBUG:boto:Token: None 2024-02-01T10:47:17.900 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:17 GMT /vclfcn-59/ 2024-02-01T10:47:17.901 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vSDQyTSLn/L3miMAiZ5pvhY2Sh4= 2024-02-01T10:47:17.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vSDQyTSLn/L3miMAiZ5pvhY2Sh4=', 'Content-Length': '0'} 2024-02-01T10:47:17.905 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007839448c8ea379ac-0065bb76b5-5765-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:17.905 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:47:17.905 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:47:17.905 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:47:17.906 DEBUG:boto:path=/vclfcn-59/?versions&max-keys=999 2024-02-01T10:47:17.906 DEBUG:boto:auth_path=/vclfcn-59/?versions&max-keys=999 2024-02-01T10:47:17.906 DEBUG:boto:Method: GET 2024-02-01T10:47:17.906 DEBUG:boto:Path: /vclfcn-59/?versions&max-keys=999 2024-02-01T10:47:17.906 DEBUG:boto:Data: 2024-02-01T10:47:17.906 DEBUG:boto:Headers: {} 2024-02-01T10:47:17.906 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:17.906 DEBUG:boto:Port: 8000 2024-02-01T10:47:17.906 DEBUG:boto:Params: {} 2024-02-01T10:47:17.907 DEBUG:boto:Token: None 2024-02-01T10:47:17.907 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:17 GMT /vclfcn-59/?versions 2024-02-01T10:47:17.907 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:4ZD0r74I0ymNSrM3hRXUywIiTFo= 2024-02-01T10:47:17.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:4ZD0r74I0ymNSrM3hRXUywIiTFo=', 'Content-Length': '0'} 2024-02-01T10:47:17.909 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9e79bd6e1b570bd-0065bb76b5-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:17.910 DEBUG:boto:b'vclfcn-59999falseobj1nulltrue2024-02-01T10:46:46.093Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:46:46.111Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:46:46.161Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:46:46.190Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:17.910 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-02-01T10:47:17.911 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-02-01T10:47:17.911 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-02-01T10:47:17.911 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-02-01T10:47:17.911 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:47:17.911 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:47:17.911 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:47:17.911 DEBUG:boto:path=/vclfcn-59/?versions&max-keys=999 2024-02-01T10:47:17.911 DEBUG:boto:auth_path=/vclfcn-59/?versions&max-keys=999 2024-02-01T10:47:17.911 DEBUG:boto:Method: GET 2024-02-01T10:47:17.912 DEBUG:boto:Path: /vclfcn-59/?versions&max-keys=999 2024-02-01T10:47:17.912 DEBUG:boto:Data: 2024-02-01T10:47:17.912 DEBUG:boto:Headers: {} 2024-02-01T10:47:17.912 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:17.912 DEBUG:boto:Port: 8000 2024-02-01T10:47:17.912 DEBUG:boto:Params: {} 2024-02-01T10:47:17.912 DEBUG:boto:Token: None 2024-02-01T10:47:17.912 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:17 GMT /vclfcn-59/?versions 2024-02-01T10:47:17.912 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:4ZD0r74I0ymNSrM3hRXUywIiTFo= 2024-02-01T10:47:17.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:4ZD0r74I0ymNSrM3hRXUywIiTFo=', 'Content-Length': '0'} 2024-02-01T10:47:17.917 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001457ec075f32672f-0065bb76b5-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:17.917 DEBUG:boto:b'vclfcn-59999falseobj1nulltrue2024-02-01T10:46:46.093Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:46:46.111Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:46:46.161Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:46:46.190Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:17.918 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-02-01T10:47:17.918 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-02-01T10:47:17.918 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-02-01T10:47:17.918 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-02-01T10:47:17.918 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:47:17.918 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:47:17.918 DEBUG:boto:path=/vclfcn-59/?versions&max-keys=999 2024-02-01T10:47:17.918 DEBUG:boto:auth_path=/vclfcn-59/?versions&max-keys=999 2024-02-01T10:47:17.918 DEBUG:boto:Method: GET 2024-02-01T10:47:17.919 DEBUG:boto:Path: /vclfcn-59/?versions&max-keys=999 2024-02-01T10:47:17.919 DEBUG:boto:Data: 2024-02-01T10:47:17.919 DEBUG:boto:Headers: {} 2024-02-01T10:47:17.919 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:17.919 DEBUG:boto:Port: 8000 2024-02-01T10:47:17.919 DEBUG:boto:Params: {} 2024-02-01T10:47:17.919 DEBUG:boto:Token: None 2024-02-01T10:47:17.919 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:17 GMT /vclfcn-59/?versions 2024-02-01T10:47:17.919 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:4ZD0r74I0ymNSrM3hRXUywIiTFo= 2024-02-01T10:47:17.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:4ZD0r74I0ymNSrM3hRXUywIiTFo=', 'Content-Length': '0'} 2024-02-01T10:47:17.922 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000385c739e19e402c7-0065bb76b5-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:17.922 DEBUG:boto:b'vclfcn-59999falseobj1nulltrue2024-02-01T10:46:46.093Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:46:46.111Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:46:46.161Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:46:46.190Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:17.923 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:47:17.923 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:47:17.923 DEBUG:boto:path=/vclfcn-59/?versions&max-keys=999 2024-02-01T10:47:17.923 DEBUG:boto:auth_path=/vclfcn-59/?versions&max-keys=999 2024-02-01T10:47:17.923 DEBUG:boto:Method: GET 2024-02-01T10:47:17.923 DEBUG:boto:Path: /vclfcn-59/?versions&max-keys=999 2024-02-01T10:47:17.923 DEBUG:boto:Data: 2024-02-01T10:47:17.923 DEBUG:boto:Headers: {} 2024-02-01T10:47:17.923 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:17.924 DEBUG:boto:Port: 8000 2024-02-01T10:47:17.924 DEBUG:boto:Params: {} 2024-02-01T10:47:17.924 DEBUG:boto:Token: None 2024-02-01T10:47:17.924 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:17 GMT /vclfcn-59/?versions 2024-02-01T10:47:17.924 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:4ZD0r74I0ymNSrM3hRXUywIiTFo= 2024-02-01T10:47:17.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:4ZD0r74I0ymNSrM3hRXUywIiTFo=', 'Content-Length': '0'} 2024-02-01T10:47:17.929 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000904b11e11fdbad96-0065bb76b5-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:17.929 DEBUG:boto:b'vclfcn-59999falseobj1nulltrue2024-02-01T10:46:46.093Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:46:46.111Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:46:46.161Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:46:46.190Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:17.929 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:47:17.930 DEBUG:boto:path=/vclfcn-59/obj1 2024-02-01T10:47:17.930 DEBUG:boto:auth_path=/vclfcn-59/obj1 2024-02-01T10:47:17.930 DEBUG:boto:path=/vclfcn-59/obj1?versionId=null 2024-02-01T10:47:17.930 DEBUG:boto:auth_path=/vclfcn-59/obj1?versionId=null 2024-02-01T10:47:17.930 DEBUG:boto:Method: GET 2024-02-01T10:47:17.930 DEBUG:boto:Path: /vclfcn-59/obj1?versionId=null 2024-02-01T10:47:17.930 DEBUG:boto:Data: 2024-02-01T10:47:17.930 DEBUG:boto:Headers: {} 2024-02-01T10:47:17.930 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:17.931 DEBUG:boto:Port: 8000 2024-02-01T10:47:17.931 DEBUG:boto:Params: {} 2024-02-01T10:47:17.931 DEBUG:boto:Token: None 2024-02-01T10:47:17.931 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:17 GMT /vclfcn-59/obj1?versionId=null 2024-02-01T10:47:17.931 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:f5rxvp06sjbfKvOYtfyFgHf1pK8= 2024-02-01T10:47:17.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:f5rxvp06sjbfKvOYtfyFgHf1pK8=', 'Content-Length': '0'} 2024-02-01T10:47:17.933 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fa49242846a90012-0065bb76b5-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:17.976 DEBUG:boto:path=/vclfcn-59/obj1 2024-02-01T10:47:17.976 DEBUG:boto:auth_path=/vclfcn-59/obj1 2024-02-01T10:47:17.976 DEBUG:boto:path=/vclfcn-59/obj1?versionId=null 2024-02-01T10:47:17.976 DEBUG:boto:auth_path=/vclfcn-59/obj1?versionId=null 2024-02-01T10:47:17.976 DEBUG:boto:Method: GET 2024-02-01T10:47:17.977 DEBUG:boto:Path: /vclfcn-59/obj1?versionId=null 2024-02-01T10:47:17.977 DEBUG:boto:Data: 2024-02-01T10:47:17.977 DEBUG:boto:Headers: {} 2024-02-01T10:47:17.977 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:17.977 DEBUG:boto:Port: 8000 2024-02-01T10:47:17.977 DEBUG:boto:Params: {} 2024-02-01T10:47:17.977 DEBUG:boto:Token: None 2024-02-01T10:47:17.977 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:17 GMT /vclfcn-59/obj1?versionId=null 2024-02-01T10:47:17.977 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:f5rxvp06sjbfKvOYtfyFgHf1pK8= 2024-02-01T10:47:17.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:f5rxvp06sjbfKvOYtfyFgHf1pK8=', 'Content-Length': '0'} 2024-02-01T10:47:17.980 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008469107909cf0156-0065bb76b5-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:17.981 DEBUG:boto:path=/vclfcn-59/obj1 2024-02-01T10:47:17.981 DEBUG:boto:auth_path=/vclfcn-59/obj1 2024-02-01T10:47:17.981 DEBUG:boto:path=/vclfcn-59/obj1?versionId=null 2024-02-01T10:47:17.981 DEBUG:boto:auth_path=/vclfcn-59/obj1?versionId=null 2024-02-01T10:47:17.982 DEBUG:boto:Method: HEAD 2024-02-01T10:47:17.982 DEBUG:boto:Path: /vclfcn-59/obj1?versionId=null 2024-02-01T10:47:17.982 DEBUG:boto:Data: 2024-02-01T10:47:17.982 DEBUG:boto:Headers: {} 2024-02-01T10:47:17.982 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:17.982 DEBUG:boto:Port: 8000 2024-02-01T10:47:17.982 DEBUG:boto:Params: {} 2024-02-01T10:47:17.982 DEBUG:boto:Token: None 2024-02-01T10:47:17.983 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:17 GMT /vclfcn-59/obj1?versionId=null 2024-02-01T10:47:17.983 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:XhnFypolLNoAb0qulMeSeUXP4k8= 2024-02-01T10:47:17.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:XhnFypolLNoAb0qulMeSeUXP4k8=', 'Content-Length': '0'} 2024-02-01T10:47:17.985 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002e47a3fd89a0d28b-0065bb76b5-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:17.985 DEBUG:boto:path=/vclfcn-59/obj1 2024-02-01T10:47:17.986 DEBUG:boto:auth_path=/vclfcn-59/obj1 2024-02-01T10:47:17.986 DEBUG:boto:path=/vclfcn-59/obj1?versionId=null 2024-02-01T10:47:17.986 DEBUG:boto:auth_path=/vclfcn-59/obj1?versionId=null 2024-02-01T10:47:17.986 DEBUG:boto:Method: HEAD 2024-02-01T10:47:17.986 DEBUG:boto:Path: /vclfcn-59/obj1?versionId=null 2024-02-01T10:47:17.986 DEBUG:boto:Data: 2024-02-01T10:47:17.986 DEBUG:boto:Headers: {} 2024-02-01T10:47:17.986 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:17.986 DEBUG:boto:Port: 8000 2024-02-01T10:47:17.986 DEBUG:boto:Params: {} 2024-02-01T10:47:17.987 DEBUG:boto:Token: None 2024-02-01T10:47:17.987 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:17 GMT /vclfcn-59/obj1?versionId=null 2024-02-01T10:47:17.987 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:XhnFypolLNoAb0qulMeSeUXP4k8= 2024-02-01T10:47:17.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:XhnFypolLNoAb0qulMeSeUXP4k8=', 'Content-Length': '0'} 2024-02-01T10:47:17.990 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ff3d8d4547193e89-0065bb76b5-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:17.990 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:47:17.991 DEBUG:boto:path=/vclfcn-59/obj1 2024-02-01T10:47:17.991 DEBUG:boto:auth_path=/vclfcn-59/obj1 2024-02-01T10:47:17.991 DEBUG:boto:path=/vclfcn-59/obj1?versionId=null 2024-02-01T10:47:17.991 DEBUG:boto:auth_path=/vclfcn-59/obj1?versionId=null 2024-02-01T10:47:17.991 DEBUG:boto:Method: GET 2024-02-01T10:47:17.991 DEBUG:boto:Path: /vclfcn-59/obj1?versionId=null 2024-02-01T10:47:17.991 DEBUG:boto:Data: 2024-02-01T10:47:17.991 DEBUG:boto:Headers: {} 2024-02-01T10:47:17.991 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:17.991 DEBUG:boto:Port: 8000 2024-02-01T10:47:17.992 DEBUG:boto:Params: {} 2024-02-01T10:47:17.992 DEBUG:boto:Token: None 2024-02-01T10:47:17.992 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:17 GMT /vclfcn-59/obj1?versionId=null 2024-02-01T10:47:17.992 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:f5rxvp06sjbfKvOYtfyFgHf1pK8= 2024-02-01T10:47:17.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:f5rxvp06sjbfKvOYtfyFgHf1pK8=', 'Content-Length': '0'} 2024-02-01T10:47:17.994 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fb4c04c20a6c79fa-0065bb76b5-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:17.995 DEBUG:boto:path=/vclfcn-59/obj1 2024-02-01T10:47:17.995 DEBUG:boto:auth_path=/vclfcn-59/obj1 2024-02-01T10:47:17.995 DEBUG:boto:path=/vclfcn-59/obj1?versionId=null 2024-02-01T10:47:17.995 DEBUG:boto:auth_path=/vclfcn-59/obj1?versionId=null 2024-02-01T10:47:17.995 DEBUG:boto:Method: GET 2024-02-01T10:47:17.995 DEBUG:boto:Path: /vclfcn-59/obj1?versionId=null 2024-02-01T10:47:17.995 DEBUG:boto:Data: 2024-02-01T10:47:17.996 DEBUG:boto:Headers: {} 2024-02-01T10:47:17.996 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:17.996 DEBUG:boto:Port: 8000 2024-02-01T10:47:17.996 DEBUG:boto:Params: {} 2024-02-01T10:47:17.996 DEBUG:boto:Token: None 2024-02-01T10:47:17.996 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:17 GMT /vclfcn-59/obj1?versionId=null 2024-02-01T10:47:17.996 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:f5rxvp06sjbfKvOYtfyFgHf1pK8= 2024-02-01T10:47:17.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:f5rxvp06sjbfKvOYtfyFgHf1pK8=', 'Content-Length': '0'} 2024-02-01T10:47:17.999 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e2494f31ecbaf30d-0065bb76b5-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:18.044 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:47:18.044 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:47:18.044 DEBUG:boto:path=/vclfcn-59/?prefix=obj1 2024-02-01T10:47:18.044 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj1 2024-02-01T10:47:18.045 DEBUG:boto:Method: GET 2024-02-01T10:47:18.045 DEBUG:boto:Path: /vclfcn-59/?prefix=obj1 2024-02-01T10:47:18.045 DEBUG:boto:Data: 2024-02-01T10:47:18.045 DEBUG:boto:Headers: {} 2024-02-01T10:47:18.045 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:18.045 DEBUG:boto:Port: 8000 2024-02-01T10:47:18.045 DEBUG:boto:Params: {} 2024-02-01T10:47:18.045 DEBUG:boto:Token: None 2024-02-01T10:47:18.045 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:18 GMT /vclfcn-59/ 2024-02-01T10:47:18.046 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:g5MYBh+0JvaTqP0VigvFjp1Sa00= 2024-02-01T10:47:18.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:g5MYBh+0JvaTqP0VigvFjp1Sa00=', 'Content-Length': '0'} 2024-02-01T10:47:18.050 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ae7d3f1687a42be-0065bb76b6-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:18.050 DEBUG:boto:b'vclfcn-59obj11000falseobj12024-02-01T10:46:46.093Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:18.051 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:47:18.051 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:47:18.051 DEBUG:boto:path=/vclfcn-59/?prefix=obj1 2024-02-01T10:47:18.051 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj1 2024-02-01T10:47:18.051 DEBUG:boto:Method: GET 2024-02-01T10:47:18.051 DEBUG:boto:Path: /vclfcn-59/?prefix=obj1 2024-02-01T10:47:18.051 DEBUG:boto:Data: 2024-02-01T10:47:18.052 DEBUG:boto:Headers: {} 2024-02-01T10:47:18.052 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:18.052 DEBUG:boto:Port: 8000 2024-02-01T10:47:18.052 DEBUG:boto:Params: {} 2024-02-01T10:47:18.052 DEBUG:boto:Token: None 2024-02-01T10:47:18.052 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:18 GMT /vclfcn-59/ 2024-02-01T10:47:18.052 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:g5MYBh+0JvaTqP0VigvFjp1Sa00= 2024-02-01T10:47:18.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:g5MYBh+0JvaTqP0VigvFjp1Sa00=', 'Content-Length': '0'} 2024-02-01T10:47:18.057 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005de47a7f8ebc4fe8-0065bb76b6-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:18.057 DEBUG:boto:b'vclfcn-59obj11000falseobj12024-02-01T10:46:46.093Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:18.057 DEBUG:boto:path=/vclfcn-59/obj1 2024-02-01T10:47:18.057 DEBUG:boto:auth_path=/vclfcn-59/obj1 2024-02-01T10:47:18.058 DEBUG:boto:Method: HEAD 2024-02-01T10:47:18.058 DEBUG:boto:Path: /vclfcn-59/obj1 2024-02-01T10:47:18.058 DEBUG:boto:Data: 2024-02-01T10:47:18.058 DEBUG:boto:Headers: {} 2024-02-01T10:47:18.058 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:18.058 DEBUG:boto:Port: 8000 2024-02-01T10:47:18.058 DEBUG:boto:Params: {} 2024-02-01T10:47:18.058 DEBUG:boto:Token: None 2024-02-01T10:47:18.058 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:18 GMT /vclfcn-59/obj1 2024-02-01T10:47:18.059 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Oz7IszNVrYC7zVBniesNJz8M/+E= 2024-02-01T10:47:18.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Oz7IszNVrYC7zVBniesNJz8M/+E=', 'Content-Length': '0'} 2024-02-01T10:47:18.061 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000061ebd1cd7a887a96-0065bb76b6-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:18.062 DEBUG:boto:path=/vclfcn-59/obj1 2024-02-01T10:47:18.062 DEBUG:boto:auth_path=/vclfcn-59/obj1 2024-02-01T10:47:18.062 DEBUG:boto:Method: HEAD 2024-02-01T10:47:18.062 DEBUG:boto:Path: /vclfcn-59/obj1 2024-02-01T10:47:18.062 DEBUG:boto:Data: 2024-02-01T10:47:18.062 DEBUG:boto:Headers: {} 2024-02-01T10:47:18.062 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:18.062 DEBUG:boto:Port: 8000 2024-02-01T10:47:18.062 DEBUG:boto:Params: {} 2024-02-01T10:47:18.063 DEBUG:boto:Token: None 2024-02-01T10:47:18.063 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:18 GMT /vclfcn-59/obj1 2024-02-01T10:47:18.063 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Oz7IszNVrYC7zVBniesNJz8M/+E= 2024-02-01T10:47:18.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Oz7IszNVrYC7zVBniesNJz8M/+E=', 'Content-Length': '0'} 2024-02-01T10:47:18.066 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002e20dc9ea66593ed-0065bb76b6-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:18.066 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:47:18.066 DEBUG:boto:path=/vclfcn-59/obj1 2024-02-01T10:47:18.066 DEBUG:boto:auth_path=/vclfcn-59/obj1 2024-02-01T10:47:18.066 DEBUG:boto:Method: GET 2024-02-01T10:47:18.067 DEBUG:boto:Path: /vclfcn-59/obj1 2024-02-01T10:47:18.067 DEBUG:boto:Data: 2024-02-01T10:47:18.067 DEBUG:boto:Headers: {} 2024-02-01T10:47:18.067 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:18.067 DEBUG:boto:Port: 8000 2024-02-01T10:47:18.067 DEBUG:boto:Params: {} 2024-02-01T10:47:18.067 DEBUG:boto:Token: None 2024-02-01T10:47:18.067 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:18 GMT /vclfcn-59/obj1 2024-02-01T10:47:18.067 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:OQn5N4oQucQn0+txfoMn/zKOp7c= 2024-02-01T10:47:18.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:OQn5N4oQucQn0+txfoMn/zKOp7c=', 'Content-Length': '0'} 2024-02-01T10:47:18.071 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e21c720dea999a77-0065bb76b6-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:18.112 DEBUG:boto:path=/vclfcn-59/obj1 2024-02-01T10:47:18.112 DEBUG:boto:auth_path=/vclfcn-59/obj1 2024-02-01T10:47:18.112 DEBUG:boto:Method: GET 2024-02-01T10:47:18.112 DEBUG:boto:Path: /vclfcn-59/obj1 2024-02-01T10:47:18.112 DEBUG:boto:Data: 2024-02-01T10:47:18.112 DEBUG:boto:Headers: {} 2024-02-01T10:47:18.113 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:18.113 DEBUG:boto:Port: 8000 2024-02-01T10:47:18.113 DEBUG:boto:Params: {} 2024-02-01T10:47:18.113 DEBUG:boto:Token: None 2024-02-01T10:47:18.113 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:18 GMT /vclfcn-59/obj1 2024-02-01T10:47:18.113 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:OQn5N4oQucQn0+txfoMn/zKOp7c= 2024-02-01T10:47:18.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:OQn5N4oQucQn0+txfoMn/zKOp7c=', 'Content-Length': '0'} 2024-02-01T10:47:18.116 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001fe1f4ead7a56f48-0065bb76b6-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:18.116 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:47:18.116 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:47:18.117 DEBUG:boto:path=/vclfcn-59/?prefix=obj1 2024-02-01T10:47:18.117 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj1 2024-02-01T10:47:18.117 DEBUG:boto:Method: GET 2024-02-01T10:47:18.117 DEBUG:boto:Path: /vclfcn-59/?prefix=obj1 2024-02-01T10:47:18.117 DEBUG:boto:Data: 2024-02-01T10:47:18.117 DEBUG:boto:Headers: {} 2024-02-01T10:47:18.117 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:18.117 DEBUG:boto:Port: 8000 2024-02-01T10:47:18.117 DEBUG:boto:Params: {} 2024-02-01T10:47:18.117 DEBUG:boto:Token: None 2024-02-01T10:47:18.118 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:18 GMT /vclfcn-59/ 2024-02-01T10:47:18.118 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:g5MYBh+0JvaTqP0VigvFjp1Sa00= 2024-02-01T10:47:18.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:g5MYBh+0JvaTqP0VigvFjp1Sa00=', 'Content-Length': '0'} 2024-02-01T10:47:18.122 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000519050b36d0488a5-0065bb76b6-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:18.122 DEBUG:boto:b'vclfcn-59obj11000falseobj12024-02-01T10:46:46.093Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:18.122 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:47:18.122 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:47:18.122 DEBUG:boto:path=/vclfcn-59/?prefix=obj1 2024-02-01T10:47:18.122 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj1 2024-02-01T10:47:18.123 DEBUG:boto:Method: GET 2024-02-01T10:47:18.123 DEBUG:boto:Path: /vclfcn-59/?prefix=obj1 2024-02-01T10:47:18.123 DEBUG:boto:Data: 2024-02-01T10:47:18.123 DEBUG:boto:Headers: {} 2024-02-01T10:47:18.123 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:18.123 DEBUG:boto:Port: 8000 2024-02-01T10:47:18.123 DEBUG:boto:Params: {} 2024-02-01T10:47:18.123 DEBUG:boto:Token: None 2024-02-01T10:47:18.123 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:18 GMT /vclfcn-59/ 2024-02-01T10:47:18.123 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:g5MYBh+0JvaTqP0VigvFjp1Sa00= 2024-02-01T10:47:18.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:g5MYBh+0JvaTqP0VigvFjp1Sa00=', 'Content-Length': '0'} 2024-02-01T10:47:18.128 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cbf9613112d58ce0-0065bb76b6-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:18.172 DEBUG:boto:b'vclfcn-59obj11000falseobj12024-02-01T10:46:46.093Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:18.172 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:47:18.172 DEBUG:boto:path=/vclfcn-59/obj2 2024-02-01T10:47:18.173 DEBUG:boto:auth_path=/vclfcn-59/obj2 2024-02-01T10:47:18.173 DEBUG:boto:path=/vclfcn-59/obj2?versionId=null 2024-02-01T10:47:18.173 DEBUG:boto:auth_path=/vclfcn-59/obj2?versionId=null 2024-02-01T10:47:18.173 DEBUG:boto:Method: GET 2024-02-01T10:47:18.173 DEBUG:boto:Path: /vclfcn-59/obj2?versionId=null 2024-02-01T10:47:18.173 DEBUG:boto:Data: 2024-02-01T10:47:18.173 DEBUG:boto:Headers: {} 2024-02-01T10:47:18.173 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:18.173 DEBUG:boto:Port: 8000 2024-02-01T10:47:18.173 DEBUG:boto:Params: {} 2024-02-01T10:47:18.173 DEBUG:boto:Token: None 2024-02-01T10:47:18.174 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:18 GMT /vclfcn-59/obj2?versionId=null 2024-02-01T10:47:18.174 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:M7HzWXwuXvbfd6C4xNF6CZn3JAI= 2024-02-01T10:47:18.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:M7HzWXwuXvbfd6C4xNF6CZn3JAI=', 'Content-Length': '0'} 2024-02-01T10:47:18.177 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fbbf428fa45cc1fd-0065bb76b6-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:18.177 DEBUG:boto:path=/vclfcn-59/obj2 2024-02-01T10:47:18.177 DEBUG:boto:auth_path=/vclfcn-59/obj2 2024-02-01T10:47:18.177 DEBUG:boto:path=/vclfcn-59/obj2?versionId=null 2024-02-01T10:47:18.177 DEBUG:boto:auth_path=/vclfcn-59/obj2?versionId=null 2024-02-01T10:47:18.177 DEBUG:boto:Method: GET 2024-02-01T10:47:18.178 DEBUG:boto:Path: /vclfcn-59/obj2?versionId=null 2024-02-01T10:47:18.178 DEBUG:boto:Data: 2024-02-01T10:47:18.178 DEBUG:boto:Headers: {} 2024-02-01T10:47:18.178 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:18.178 DEBUG:boto:Port: 8000 2024-02-01T10:47:18.178 DEBUG:boto:Params: {} 2024-02-01T10:47:18.178 DEBUG:boto:Token: None 2024-02-01T10:47:18.178 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:18 GMT /vclfcn-59/obj2?versionId=null 2024-02-01T10:47:18.178 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:M7HzWXwuXvbfd6C4xNF6CZn3JAI= 2024-02-01T10:47:18.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:M7HzWXwuXvbfd6C4xNF6CZn3JAI=', 'Content-Length': '0'} 2024-02-01T10:47:18.181 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a69c16fe88e59927-0065bb76b6-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:18.182 DEBUG:boto:path=/vclfcn-59/obj2 2024-02-01T10:47:18.182 DEBUG:boto:auth_path=/vclfcn-59/obj2 2024-02-01T10:47:18.182 DEBUG:boto:path=/vclfcn-59/obj2?versionId=null 2024-02-01T10:47:18.182 DEBUG:boto:auth_path=/vclfcn-59/obj2?versionId=null 2024-02-01T10:47:18.182 DEBUG:boto:Method: HEAD 2024-02-01T10:47:18.182 DEBUG:boto:Path: /vclfcn-59/obj2?versionId=null 2024-02-01T10:47:18.182 DEBUG:boto:Data: 2024-02-01T10:47:18.182 DEBUG:boto:Headers: {} 2024-02-01T10:47:18.182 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:18.182 DEBUG:boto:Port: 8000 2024-02-01T10:47:18.183 DEBUG:boto:Params: {} 2024-02-01T10:47:18.183 DEBUG:boto:Token: None 2024-02-01T10:47:18.183 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:18 GMT /vclfcn-59/obj2?versionId=null 2024-02-01T10:47:18.183 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:UJ+s5BaQ2AthCybUZwWRHg19nZA= 2024-02-01T10:47:18.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:UJ+s5BaQ2AthCybUZwWRHg19nZA=', 'Content-Length': '0'} 2024-02-01T10:47:18.186 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000007b3f3eccfd6d169-0065bb76b6-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:18.186 DEBUG:boto:path=/vclfcn-59/obj2 2024-02-01T10:47:18.186 DEBUG:boto:auth_path=/vclfcn-59/obj2 2024-02-01T10:47:18.186 DEBUG:boto:path=/vclfcn-59/obj2?versionId=null 2024-02-01T10:47:18.187 DEBUG:boto:auth_path=/vclfcn-59/obj2?versionId=null 2024-02-01T10:47:18.187 DEBUG:boto:Method: HEAD 2024-02-01T10:47:18.187 DEBUG:boto:Path: /vclfcn-59/obj2?versionId=null 2024-02-01T10:47:18.187 DEBUG:boto:Data: 2024-02-01T10:47:18.187 DEBUG:boto:Headers: {} 2024-02-01T10:47:18.187 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:18.187 DEBUG:boto:Port: 8000 2024-02-01T10:47:18.187 DEBUG:boto:Params: {} 2024-02-01T10:47:18.187 DEBUG:boto:Token: None 2024-02-01T10:47:18.187 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:18 GMT /vclfcn-59/obj2?versionId=null 2024-02-01T10:47:18.188 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:UJ+s5BaQ2AthCybUZwWRHg19nZA= 2024-02-01T10:47:18.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:UJ+s5BaQ2AthCybUZwWRHg19nZA=', 'Content-Length': '0'} 2024-02-01T10:47:18.190 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000099c70abe4e0affd8-0065bb76b6-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:18.190 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:47:18.191 DEBUG:boto:path=/vclfcn-59/obj2 2024-02-01T10:47:18.191 DEBUG:boto:auth_path=/vclfcn-59/obj2 2024-02-01T10:47:18.191 DEBUG:boto:path=/vclfcn-59/obj2?versionId=null 2024-02-01T10:47:18.191 DEBUG:boto:auth_path=/vclfcn-59/obj2?versionId=null 2024-02-01T10:47:18.191 DEBUG:boto:Method: GET 2024-02-01T10:47:18.191 DEBUG:boto:Path: /vclfcn-59/obj2?versionId=null 2024-02-01T10:47:18.191 DEBUG:boto:Data: 2024-02-01T10:47:18.191 DEBUG:boto:Headers: {} 2024-02-01T10:47:18.191 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:18.191 DEBUG:boto:Port: 8000 2024-02-01T10:47:18.191 DEBUG:boto:Params: {} 2024-02-01T10:47:18.192 DEBUG:boto:Token: None 2024-02-01T10:47:18.192 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:18 GMT /vclfcn-59/obj2?versionId=null 2024-02-01T10:47:18.192 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:M7HzWXwuXvbfd6C4xNF6CZn3JAI= 2024-02-01T10:47:18.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:M7HzWXwuXvbfd6C4xNF6CZn3JAI=', 'Content-Length': '0'} 2024-02-01T10:47:18.195 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000069d314a61a969ec6-0065bb76b6-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:18.236 DEBUG:boto:path=/vclfcn-59/obj2 2024-02-01T10:47:18.236 DEBUG:boto:auth_path=/vclfcn-59/obj2 2024-02-01T10:47:18.236 DEBUG:boto:path=/vclfcn-59/obj2?versionId=null 2024-02-01T10:47:18.236 DEBUG:boto:auth_path=/vclfcn-59/obj2?versionId=null 2024-02-01T10:47:18.236 DEBUG:boto:Method: GET 2024-02-01T10:47:18.237 DEBUG:boto:Path: /vclfcn-59/obj2?versionId=null 2024-02-01T10:47:18.237 DEBUG:boto:Data: 2024-02-01T10:47:18.237 DEBUG:boto:Headers: {} 2024-02-01T10:47:18.237 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:18.237 DEBUG:boto:Port: 8000 2024-02-01T10:47:18.237 DEBUG:boto:Params: {} 2024-02-01T10:47:18.237 DEBUG:boto:Token: None 2024-02-01T10:47:18.237 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:18 GMT /vclfcn-59/obj2?versionId=null 2024-02-01T10:47:18.237 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:M7HzWXwuXvbfd6C4xNF6CZn3JAI= 2024-02-01T10:47:18.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:M7HzWXwuXvbfd6C4xNF6CZn3JAI=', 'Content-Length': '0'} 2024-02-01T10:47:18.241 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000efb2091be423be18-0065bb76b6-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:18.241 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:47:18.241 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:47:18.241 DEBUG:boto:path=/vclfcn-59/?prefix=obj2 2024-02-01T10:47:18.242 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj2 2024-02-01T10:47:18.242 DEBUG:boto:Method: GET 2024-02-01T10:47:18.242 DEBUG:boto:Path: /vclfcn-59/?prefix=obj2 2024-02-01T10:47:18.242 DEBUG:boto:Data: 2024-02-01T10:47:18.242 DEBUG:boto:Headers: {} 2024-02-01T10:47:18.242 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:18.242 DEBUG:boto:Port: 8000 2024-02-01T10:47:18.242 DEBUG:boto:Params: {} 2024-02-01T10:47:18.242 DEBUG:boto:Token: None 2024-02-01T10:47:18.242 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:18 GMT /vclfcn-59/ 2024-02-01T10:47:18.243 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:g5MYBh+0JvaTqP0VigvFjp1Sa00= 2024-02-01T10:47:18.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:g5MYBh+0JvaTqP0VigvFjp1Sa00=', 'Content-Length': '0'} 2024-02-01T10:47:18.247 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e08991f54d784845-0065bb76b6-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:18.247 DEBUG:boto:b'vclfcn-59obj21000falseobj22024-02-01T10:46:46.111Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:18.248 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:47:18.248 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:47:18.248 DEBUG:boto:path=/vclfcn-59/?prefix=obj2 2024-02-01T10:47:18.248 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj2 2024-02-01T10:47:18.248 DEBUG:boto:Method: GET 2024-02-01T10:47:18.248 DEBUG:boto:Path: /vclfcn-59/?prefix=obj2 2024-02-01T10:47:18.248 DEBUG:boto:Data: 2024-02-01T10:47:18.248 DEBUG:boto:Headers: {} 2024-02-01T10:47:18.248 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:18.248 DEBUG:boto:Port: 8000 2024-02-01T10:47:18.249 DEBUG:boto:Params: {} 2024-02-01T10:47:18.249 DEBUG:boto:Token: None 2024-02-01T10:47:18.249 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:18 GMT /vclfcn-59/ 2024-02-01T10:47:18.249 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:g5MYBh+0JvaTqP0VigvFjp1Sa00= 2024-02-01T10:47:18.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:g5MYBh+0JvaTqP0VigvFjp1Sa00=', 'Content-Length': '0'} 2024-02-01T10:47:18.253 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c376f4bedfa3938c-0065bb76b6-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:18.296 DEBUG:boto:b'vclfcn-59obj21000falseobj22024-02-01T10:46:46.111Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:18.296 DEBUG:boto:path=/vclfcn-59/obj2 2024-02-01T10:47:18.296 DEBUG:boto:auth_path=/vclfcn-59/obj2 2024-02-01T10:47:18.297 DEBUG:boto:Method: HEAD 2024-02-01T10:47:18.297 DEBUG:boto:Path: /vclfcn-59/obj2 2024-02-01T10:47:18.297 DEBUG:boto:Data: 2024-02-01T10:47:18.297 DEBUG:boto:Headers: {} 2024-02-01T10:47:18.297 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:18.297 DEBUG:boto:Port: 8000 2024-02-01T10:47:18.297 DEBUG:boto:Params: {} 2024-02-01T10:47:18.297 DEBUG:boto:Token: None 2024-02-01T10:47:18.297 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:18 GMT /vclfcn-59/obj2 2024-02-01T10:47:18.298 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NFRG+8bjmlvPoCV+568xC/eqQ9c= 2024-02-01T10:47:18.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NFRG+8bjmlvPoCV+568xC/eqQ9c=', 'Content-Length': '0'} 2024-02-01T10:47:18.300 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000093eb6016b437eeeb-0065bb76b6-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:18.301 DEBUG:boto:path=/vclfcn-59/obj2 2024-02-01T10:47:18.301 DEBUG:boto:auth_path=/vclfcn-59/obj2 2024-02-01T10:47:18.301 DEBUG:boto:Method: HEAD 2024-02-01T10:47:18.301 DEBUG:boto:Path: /vclfcn-59/obj2 2024-02-01T10:47:18.301 DEBUG:boto:Data: 2024-02-01T10:47:18.301 DEBUG:boto:Headers: {} 2024-02-01T10:47:18.301 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:18.301 DEBUG:boto:Port: 8000 2024-02-01T10:47:18.301 DEBUG:boto:Params: {} 2024-02-01T10:47:18.301 DEBUG:boto:Token: None 2024-02-01T10:47:18.302 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:18 GMT /vclfcn-59/obj2 2024-02-01T10:47:18.302 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NFRG+8bjmlvPoCV+568xC/eqQ9c= 2024-02-01T10:47:18.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NFRG+8bjmlvPoCV+568xC/eqQ9c=', 'Content-Length': '0'} 2024-02-01T10:47:18.304 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d01d8e87487defa1-0065bb76b6-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:18.305 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:47:18.305 DEBUG:boto:path=/vclfcn-59/obj2 2024-02-01T10:47:18.305 DEBUG:boto:auth_path=/vclfcn-59/obj2 2024-02-01T10:47:18.305 DEBUG:boto:Method: GET 2024-02-01T10:47:18.305 DEBUG:boto:Path: /vclfcn-59/obj2 2024-02-01T10:47:18.305 DEBUG:boto:Data: 2024-02-01T10:47:18.305 DEBUG:boto:Headers: {} 2024-02-01T10:47:18.305 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:18.305 DEBUG:boto:Port: 8000 2024-02-01T10:47:18.305 DEBUG:boto:Params: {} 2024-02-01T10:47:18.306 DEBUG:boto:Token: None 2024-02-01T10:47:18.306 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:18 GMT /vclfcn-59/obj2 2024-02-01T10:47:18.306 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:uxOdN/BlVAM2EY4XxpsK9h20xU4= 2024-02-01T10:47:18.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:uxOdN/BlVAM2EY4XxpsK9h20xU4=', 'Content-Length': '0'} 2024-02-01T10:47:18.309 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000083a660a7b18ff97f-0065bb76b6-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:18.352 DEBUG:boto:path=/vclfcn-59/obj2 2024-02-01T10:47:18.352 DEBUG:boto:auth_path=/vclfcn-59/obj2 2024-02-01T10:47:18.352 DEBUG:boto:Method: GET 2024-02-01T10:47:18.352 DEBUG:boto:Path: /vclfcn-59/obj2 2024-02-01T10:47:18.353 DEBUG:boto:Data: 2024-02-01T10:47:18.353 DEBUG:boto:Headers: {} 2024-02-01T10:47:18.353 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:18.353 DEBUG:boto:Port: 8000 2024-02-01T10:47:18.353 DEBUG:boto:Params: {} 2024-02-01T10:47:18.353 DEBUG:boto:Token: None 2024-02-01T10:47:18.353 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:18 GMT /vclfcn-59/obj2 2024-02-01T10:47:18.353 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:uxOdN/BlVAM2EY4XxpsK9h20xU4= 2024-02-01T10:47:18.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:uxOdN/BlVAM2EY4XxpsK9h20xU4=', 'Content-Length': '0'} 2024-02-01T10:47:18.357 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000966bd64ea9d8c5ac-0065bb76b6-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:18.357 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:47:18.357 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:47:18.358 DEBUG:boto:path=/vclfcn-59/?prefix=obj2 2024-02-01T10:47:18.358 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj2 2024-02-01T10:47:18.358 DEBUG:boto:Method: GET 2024-02-01T10:47:18.358 DEBUG:boto:Path: /vclfcn-59/?prefix=obj2 2024-02-01T10:47:18.358 DEBUG:boto:Data: 2024-02-01T10:47:18.358 DEBUG:boto:Headers: {} 2024-02-01T10:47:18.358 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:18.358 DEBUG:boto:Port: 8000 2024-02-01T10:47:18.358 DEBUG:boto:Params: {} 2024-02-01T10:47:18.358 DEBUG:boto:Token: None 2024-02-01T10:47:18.359 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:18 GMT /vclfcn-59/ 2024-02-01T10:47:18.359 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:g5MYBh+0JvaTqP0VigvFjp1Sa00= 2024-02-01T10:47:18.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:g5MYBh+0JvaTqP0VigvFjp1Sa00=', 'Content-Length': '0'} 2024-02-01T10:47:18.363 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002458e598ff3051ce-0065bb76b6-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:18.363 DEBUG:boto:b'vclfcn-59obj21000falseobj22024-02-01T10:46:46.111Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:18.363 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:47:18.363 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:47:18.364 DEBUG:boto:path=/vclfcn-59/?prefix=obj2 2024-02-01T10:47:18.364 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj2 2024-02-01T10:47:18.364 DEBUG:boto:Method: GET 2024-02-01T10:47:18.364 DEBUG:boto:Path: /vclfcn-59/?prefix=obj2 2024-02-01T10:47:18.364 DEBUG:boto:Data: 2024-02-01T10:47:18.364 DEBUG:boto:Headers: {} 2024-02-01T10:47:18.364 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:18.364 DEBUG:boto:Port: 8000 2024-02-01T10:47:18.364 DEBUG:boto:Params: {} 2024-02-01T10:47:18.364 DEBUG:boto:Token: None 2024-02-01T10:47:18.365 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:18 GMT /vclfcn-59/ 2024-02-01T10:47:18.365 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:g5MYBh+0JvaTqP0VigvFjp1Sa00= 2024-02-01T10:47:18.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:g5MYBh+0JvaTqP0VigvFjp1Sa00=', 'Content-Length': '0'} 2024-02-01T10:47:18.369 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000badb6bfc4bb23958-0065bb76b6-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:18.369 DEBUG:boto:b'vclfcn-59obj21000falseobj22024-02-01T10:46:46.111Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:18.369 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:47:18.369 DEBUG:boto:path=/vclfcn-59/obj3 2024-02-01T10:47:18.369 DEBUG:boto:auth_path=/vclfcn-59/obj3 2024-02-01T10:47:18.370 DEBUG:boto:path=/vclfcn-59/obj3?versionId=null 2024-02-01T10:47:18.370 DEBUG:boto:auth_path=/vclfcn-59/obj3?versionId=null 2024-02-01T10:47:18.370 DEBUG:boto:Method: GET 2024-02-01T10:47:18.370 DEBUG:boto:Path: /vclfcn-59/obj3?versionId=null 2024-02-01T10:47:18.370 DEBUG:boto:Data: 2024-02-01T10:47:18.370 DEBUG:boto:Headers: {} 2024-02-01T10:47:18.370 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:18.370 DEBUG:boto:Port: 8000 2024-02-01T10:47:18.370 DEBUG:boto:Params: {} 2024-02-01T10:47:18.370 DEBUG:boto:Token: None 2024-02-01T10:47:18.371 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:18 GMT /vclfcn-59/obj3?versionId=null 2024-02-01T10:47:18.371 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nnt7LVPJ0Z3QgK3efQxb+Kopdjk= 2024-02-01T10:47:18.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nnt7LVPJ0Z3QgK3efQxb+Kopdjk=', 'Content-Length': '0'} 2024-02-01T10:47:18.373 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c408099d474177f2-0065bb76b6-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:18.374 DEBUG:boto:path=/vclfcn-59/obj3 2024-02-01T10:47:18.374 DEBUG:boto:auth_path=/vclfcn-59/obj3 2024-02-01T10:47:18.374 DEBUG:boto:path=/vclfcn-59/obj3?versionId=null 2024-02-01T10:47:18.374 DEBUG:boto:auth_path=/vclfcn-59/obj3?versionId=null 2024-02-01T10:47:18.374 DEBUG:boto:Method: GET 2024-02-01T10:47:18.374 DEBUG:boto:Path: /vclfcn-59/obj3?versionId=null 2024-02-01T10:47:18.374 DEBUG:boto:Data: 2024-02-01T10:47:18.374 DEBUG:boto:Headers: {} 2024-02-01T10:47:18.374 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:18.375 DEBUG:boto:Port: 8000 2024-02-01T10:47:18.375 DEBUG:boto:Params: {} 2024-02-01T10:47:18.375 DEBUG:boto:Token: None 2024-02-01T10:47:18.375 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:18 GMT /vclfcn-59/obj3?versionId=null 2024-02-01T10:47:18.375 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nnt7LVPJ0Z3QgK3efQxb+Kopdjk= 2024-02-01T10:47:18.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nnt7LVPJ0Z3QgK3efQxb+Kopdjk=', 'Content-Length': '0'} 2024-02-01T10:47:18.378 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000753a1a761e57c898-0065bb76b6-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:18.420 DEBUG:boto:path=/vclfcn-59/obj3 2024-02-01T10:47:18.420 DEBUG:boto:auth_path=/vclfcn-59/obj3 2024-02-01T10:47:18.420 DEBUG:boto:path=/vclfcn-59/obj3?versionId=null 2024-02-01T10:47:18.420 DEBUG:boto:auth_path=/vclfcn-59/obj3?versionId=null 2024-02-01T10:47:18.420 DEBUG:boto:Method: HEAD 2024-02-01T10:47:18.420 DEBUG:boto:Path: /vclfcn-59/obj3?versionId=null 2024-02-01T10:47:18.421 DEBUG:boto:Data: 2024-02-01T10:47:18.421 DEBUG:boto:Headers: {} 2024-02-01T10:47:18.421 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:18.421 DEBUG:boto:Port: 8000 2024-02-01T10:47:18.421 DEBUG:boto:Params: {} 2024-02-01T10:47:18.421 DEBUG:boto:Token: None 2024-02-01T10:47:18.421 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:18 GMT /vclfcn-59/obj3?versionId=null 2024-02-01T10:47:18.421 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vbFQZFmVVFuFcIdmGviZgFX33i4= 2024-02-01T10:47:18.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vbFQZFmVVFuFcIdmGviZgFX33i4=', 'Content-Length': '0'} 2024-02-01T10:47:18.424 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000da438c65a86237db-0065bb76b6-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:18.424 DEBUG:boto:path=/vclfcn-59/obj3 2024-02-01T10:47:18.425 DEBUG:boto:auth_path=/vclfcn-59/obj3 2024-02-01T10:47:18.425 DEBUG:boto:path=/vclfcn-59/obj3?versionId=null 2024-02-01T10:47:18.425 DEBUG:boto:auth_path=/vclfcn-59/obj3?versionId=null 2024-02-01T10:47:18.425 DEBUG:boto:Method: HEAD 2024-02-01T10:47:18.425 DEBUG:boto:Path: /vclfcn-59/obj3?versionId=null 2024-02-01T10:47:18.425 DEBUG:boto:Data: 2024-02-01T10:47:18.425 DEBUG:boto:Headers: {} 2024-02-01T10:47:18.425 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:18.425 DEBUG:boto:Port: 8000 2024-02-01T10:47:18.425 DEBUG:boto:Params: {} 2024-02-01T10:47:18.425 DEBUG:boto:Token: None 2024-02-01T10:47:18.426 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:18 GMT /vclfcn-59/obj3?versionId=null 2024-02-01T10:47:18.426 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vbFQZFmVVFuFcIdmGviZgFX33i4= 2024-02-01T10:47:18.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vbFQZFmVVFuFcIdmGviZgFX33i4=', 'Content-Length': '0'} 2024-02-01T10:47:18.429 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eb8139e6b3ff73d4-0065bb76b6-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:18.429 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:47:18.429 DEBUG:boto:path=/vclfcn-59/obj3 2024-02-01T10:47:18.429 DEBUG:boto:auth_path=/vclfcn-59/obj3 2024-02-01T10:47:18.429 DEBUG:boto:path=/vclfcn-59/obj3?versionId=null 2024-02-01T10:47:18.429 DEBUG:boto:auth_path=/vclfcn-59/obj3?versionId=null 2024-02-01T10:47:18.429 DEBUG:boto:Method: GET 2024-02-01T10:47:18.430 DEBUG:boto:Path: /vclfcn-59/obj3?versionId=null 2024-02-01T10:47:18.430 DEBUG:boto:Data: 2024-02-01T10:47:18.430 DEBUG:boto:Headers: {} 2024-02-01T10:47:18.430 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:18.430 DEBUG:boto:Port: 8000 2024-02-01T10:47:18.430 DEBUG:boto:Params: {} 2024-02-01T10:47:18.430 DEBUG:boto:Token: None 2024-02-01T10:47:18.430 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:18 GMT /vclfcn-59/obj3?versionId=null 2024-02-01T10:47:18.430 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nnt7LVPJ0Z3QgK3efQxb+Kopdjk= 2024-02-01T10:47:18.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nnt7LVPJ0Z3QgK3efQxb+Kopdjk=', 'Content-Length': '0'} 2024-02-01T10:47:18.433 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f871b3a6ffd0343f-0065bb76b6-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:18.476 DEBUG:boto:path=/vclfcn-59/obj3 2024-02-01T10:47:18.476 DEBUG:boto:auth_path=/vclfcn-59/obj3 2024-02-01T10:47:18.476 DEBUG:boto:path=/vclfcn-59/obj3?versionId=null 2024-02-01T10:47:18.476 DEBUG:boto:auth_path=/vclfcn-59/obj3?versionId=null 2024-02-01T10:47:18.476 DEBUG:boto:Method: GET 2024-02-01T10:47:18.477 DEBUG:boto:Path: /vclfcn-59/obj3?versionId=null 2024-02-01T10:47:18.477 DEBUG:boto:Data: 2024-02-01T10:47:18.477 DEBUG:boto:Headers: {} 2024-02-01T10:47:18.477 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:18.477 DEBUG:boto:Port: 8000 2024-02-01T10:47:18.477 DEBUG:boto:Params: {} 2024-02-01T10:47:18.477 DEBUG:boto:Token: None 2024-02-01T10:47:18.477 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:18 GMT /vclfcn-59/obj3?versionId=null 2024-02-01T10:47:18.477 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nnt7LVPJ0Z3QgK3efQxb+Kopdjk= 2024-02-01T10:47:18.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nnt7LVPJ0Z3QgK3efQxb+Kopdjk=', 'Content-Length': '0'} 2024-02-01T10:47:18.483 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005d7fc667e977f890-0065bb76b6-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:18.484 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:47:18.484 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:47:18.484 DEBUG:boto:path=/vclfcn-59/?prefix=obj3 2024-02-01T10:47:18.484 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj3 2024-02-01T10:47:18.484 DEBUG:boto:Method: GET 2024-02-01T10:47:18.484 DEBUG:boto:Path: /vclfcn-59/?prefix=obj3 2024-02-01T10:47:18.484 DEBUG:boto:Data: 2024-02-01T10:47:18.484 DEBUG:boto:Headers: {} 2024-02-01T10:47:18.484 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:18.485 DEBUG:boto:Port: 8000 2024-02-01T10:47:18.485 DEBUG:boto:Params: {} 2024-02-01T10:47:18.485 DEBUG:boto:Token: None 2024-02-01T10:47:18.485 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:18 GMT /vclfcn-59/ 2024-02-01T10:47:18.485 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:g5MYBh+0JvaTqP0VigvFjp1Sa00= 2024-02-01T10:47:18.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:g5MYBh+0JvaTqP0VigvFjp1Sa00=', 'Content-Length': '0'} 2024-02-01T10:47:18.489 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb95ac40506ca057-0065bb76b6-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:18.490 DEBUG:boto:b'vclfcn-59obj31000falseobj32024-02-01T10:46:46.161Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:18.490 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:47:18.490 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:47:18.490 DEBUG:boto:path=/vclfcn-59/?prefix=obj3 2024-02-01T10:47:18.490 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj3 2024-02-01T10:47:18.491 DEBUG:boto:Method: GET 2024-02-01T10:47:18.491 DEBUG:boto:Path: /vclfcn-59/?prefix=obj3 2024-02-01T10:47:18.491 DEBUG:boto:Data: 2024-02-01T10:47:18.491 DEBUG:boto:Headers: {} 2024-02-01T10:47:18.491 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:18.491 DEBUG:boto:Port: 8000 2024-02-01T10:47:18.491 DEBUG:boto:Params: {} 2024-02-01T10:47:18.491 DEBUG:boto:Token: None 2024-02-01T10:47:18.491 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:18 GMT /vclfcn-59/ 2024-02-01T10:47:18.491 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:g5MYBh+0JvaTqP0VigvFjp1Sa00= 2024-02-01T10:47:18.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:g5MYBh+0JvaTqP0VigvFjp1Sa00=', 'Content-Length': '0'} 2024-02-01T10:47:18.495 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084b8753cc1522caa-0065bb76b6-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:18.496 DEBUG:boto:b'vclfcn-59obj31000falseobj32024-02-01T10:46:46.161Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:18.496 DEBUG:boto:path=/vclfcn-59/obj3 2024-02-01T10:47:18.496 DEBUG:boto:auth_path=/vclfcn-59/obj3 2024-02-01T10:47:18.496 DEBUG:boto:Method: HEAD 2024-02-01T10:47:18.496 DEBUG:boto:Path: /vclfcn-59/obj3 2024-02-01T10:47:18.496 DEBUG:boto:Data: 2024-02-01T10:47:18.497 DEBUG:boto:Headers: {} 2024-02-01T10:47:18.497 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:18.497 DEBUG:boto:Port: 8000 2024-02-01T10:47:18.497 DEBUG:boto:Params: {} 2024-02-01T10:47:18.497 DEBUG:boto:Token: None 2024-02-01T10:47:18.497 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:18 GMT /vclfcn-59/obj3 2024-02-01T10:47:18.497 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:XiffkKmrfgVLPObfjugxfNt8qIk= 2024-02-01T10:47:18.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:XiffkKmrfgVLPObfjugxfNt8qIk=', 'Content-Length': '0'} 2024-02-01T10:47:18.500 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000beef359a9996a0d8-0065bb76b6-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:18.500 DEBUG:boto:path=/vclfcn-59/obj3 2024-02-01T10:47:18.500 DEBUG:boto:auth_path=/vclfcn-59/obj3 2024-02-01T10:47:18.500 DEBUG:boto:Method: HEAD 2024-02-01T10:47:18.500 DEBUG:boto:Path: /vclfcn-59/obj3 2024-02-01T10:47:18.501 DEBUG:boto:Data: 2024-02-01T10:47:18.501 DEBUG:boto:Headers: {} 2024-02-01T10:47:18.501 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:18.501 DEBUG:boto:Port: 8000 2024-02-01T10:47:18.501 DEBUG:boto:Params: {} 2024-02-01T10:47:18.501 DEBUG:boto:Token: None 2024-02-01T10:47:18.501 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:18 GMT /vclfcn-59/obj3 2024-02-01T10:47:18.501 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:XiffkKmrfgVLPObfjugxfNt8qIk= 2024-02-01T10:47:18.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:XiffkKmrfgVLPObfjugxfNt8qIk=', 'Content-Length': '0'} 2024-02-01T10:47:18.504 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001f3f383de20e5d9f-0065bb76b6-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:18.504 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:47:18.504 DEBUG:boto:path=/vclfcn-59/obj3 2024-02-01T10:47:18.504 DEBUG:boto:auth_path=/vclfcn-59/obj3 2024-02-01T10:47:18.505 DEBUG:boto:Method: GET 2024-02-01T10:47:18.505 DEBUG:boto:Path: /vclfcn-59/obj3 2024-02-01T10:47:18.505 DEBUG:boto:Data: 2024-02-01T10:47:18.505 DEBUG:boto:Headers: {} 2024-02-01T10:47:18.505 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:18.505 DEBUG:boto:Port: 8000 2024-02-01T10:47:18.505 DEBUG:boto:Params: {} 2024-02-01T10:47:18.505 DEBUG:boto:Token: None 2024-02-01T10:47:18.505 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:18 GMT /vclfcn-59/obj3 2024-02-01T10:47:18.506 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YpTeKIU2VuEtcSql0YlppNqun20= 2024-02-01T10:47:18.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YpTeKIU2VuEtcSql0YlppNqun20=', 'Content-Length': '0'} 2024-02-01T10:47:18.508 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d52e970876c46d1d-0065bb76b6-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:18.552 DEBUG:boto:path=/vclfcn-59/obj3 2024-02-01T10:47:18.552 DEBUG:boto:auth_path=/vclfcn-59/obj3 2024-02-01T10:47:18.552 DEBUG:boto:Method: GET 2024-02-01T10:47:18.552 DEBUG:boto:Path: /vclfcn-59/obj3 2024-02-01T10:47:18.552 DEBUG:boto:Data: 2024-02-01T10:47:18.552 DEBUG:boto:Headers: {} 2024-02-01T10:47:18.553 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:18.553 DEBUG:boto:Port: 8000 2024-02-01T10:47:18.553 DEBUG:boto:Params: {} 2024-02-01T10:47:18.553 DEBUG:boto:Token: None 2024-02-01T10:47:18.553 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:18 GMT /vclfcn-59/obj3 2024-02-01T10:47:18.553 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YpTeKIU2VuEtcSql0YlppNqun20= 2024-02-01T10:47:18.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YpTeKIU2VuEtcSql0YlppNqun20=', 'Content-Length': '0'} 2024-02-01T10:47:18.556 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008dea668df83eac28-0065bb76b6-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:18.557 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:47:18.557 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:47:18.557 DEBUG:boto:path=/vclfcn-59/?prefix=obj3 2024-02-01T10:47:18.557 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj3 2024-02-01T10:47:18.557 DEBUG:boto:Method: GET 2024-02-01T10:47:18.557 DEBUG:boto:Path: /vclfcn-59/?prefix=obj3 2024-02-01T10:47:18.557 DEBUG:boto:Data: 2024-02-01T10:47:18.557 DEBUG:boto:Headers: {} 2024-02-01T10:47:18.557 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:18.557 DEBUG:boto:Port: 8000 2024-02-01T10:47:18.557 DEBUG:boto:Params: {} 2024-02-01T10:47:18.558 DEBUG:boto:Token: None 2024-02-01T10:47:18.558 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:18 GMT /vclfcn-59/ 2024-02-01T10:47:18.558 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:g5MYBh+0JvaTqP0VigvFjp1Sa00= 2024-02-01T10:47:18.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:g5MYBh+0JvaTqP0VigvFjp1Sa00=', 'Content-Length': '0'} 2024-02-01T10:47:18.562 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f506e11e59337bc9-0065bb76b6-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:18.604 DEBUG:boto:b'vclfcn-59obj31000falseobj32024-02-01T10:46:46.161Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:18.605 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:47:18.605 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:47:18.605 DEBUG:boto:path=/vclfcn-59/?prefix=obj3 2024-02-01T10:47:18.605 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj3 2024-02-01T10:47:18.605 DEBUG:boto:Method: GET 2024-02-01T10:47:18.605 DEBUG:boto:Path: /vclfcn-59/?prefix=obj3 2024-02-01T10:47:18.605 DEBUG:boto:Data: 2024-02-01T10:47:18.605 DEBUG:boto:Headers: {} 2024-02-01T10:47:18.606 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:18.606 DEBUG:boto:Port: 8000 2024-02-01T10:47:18.606 DEBUG:boto:Params: {} 2024-02-01T10:47:18.606 DEBUG:boto:Token: None 2024-02-01T10:47:18.606 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:18 GMT /vclfcn-59/ 2024-02-01T10:47:18.606 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:g5MYBh+0JvaTqP0VigvFjp1Sa00= 2024-02-01T10:47:18.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:g5MYBh+0JvaTqP0VigvFjp1Sa00=', 'Content-Length': '0'} 2024-02-01T10:47:18.610 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037c8af4651b47a65-0065bb76b6-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:18.610 DEBUG:boto:b'vclfcn-59obj31000falseobj32024-02-01T10:46:46.161Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:18.611 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:47:18.611 DEBUG:boto:path=/vclfcn-59/obj4 2024-02-01T10:47:18.611 DEBUG:boto:auth_path=/vclfcn-59/obj4 2024-02-01T10:47:18.611 DEBUG:boto:path=/vclfcn-59/obj4?versionId=null 2024-02-01T10:47:18.611 DEBUG:boto:auth_path=/vclfcn-59/obj4?versionId=null 2024-02-01T10:47:18.611 DEBUG:boto:Method: GET 2024-02-01T10:47:18.611 DEBUG:boto:Path: /vclfcn-59/obj4?versionId=null 2024-02-01T10:47:18.611 DEBUG:boto:Data: 2024-02-01T10:47:18.611 DEBUG:boto:Headers: {} 2024-02-01T10:47:18.612 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:18.612 DEBUG:boto:Port: 8000 2024-02-01T10:47:18.612 DEBUG:boto:Params: {} 2024-02-01T10:47:18.612 DEBUG:boto:Token: None 2024-02-01T10:47:18.612 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:18 GMT /vclfcn-59/obj4?versionId=null 2024-02-01T10:47:18.612 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5nbKn9EFjhNc/+Y2/eTRmalQBmY= 2024-02-01T10:47:18.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5nbKn9EFjhNc/+Y2/eTRmalQBmY=', 'Content-Length': '0'} 2024-02-01T10:47:18.615 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fc178252667fb810-0065bb76b6-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:18.615 DEBUG:boto:path=/vclfcn-59/obj4 2024-02-01T10:47:18.616 DEBUG:boto:auth_path=/vclfcn-59/obj4 2024-02-01T10:47:18.616 DEBUG:boto:path=/vclfcn-59/obj4?versionId=null 2024-02-01T10:47:18.616 DEBUG:boto:auth_path=/vclfcn-59/obj4?versionId=null 2024-02-01T10:47:18.616 DEBUG:boto:Method: GET 2024-02-01T10:47:18.616 DEBUG:boto:Path: /vclfcn-59/obj4?versionId=null 2024-02-01T10:47:18.616 DEBUG:boto:Data: 2024-02-01T10:47:18.616 DEBUG:boto:Headers: {} 2024-02-01T10:47:18.616 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:18.616 DEBUG:boto:Port: 8000 2024-02-01T10:47:18.616 DEBUG:boto:Params: {} 2024-02-01T10:47:18.617 DEBUG:boto:Token: None 2024-02-01T10:47:18.617 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:18 GMT /vclfcn-59/obj4?versionId=null 2024-02-01T10:47:18.617 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5nbKn9EFjhNc/+Y2/eTRmalQBmY= 2024-02-01T10:47:18.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5nbKn9EFjhNc/+Y2/eTRmalQBmY=', 'Content-Length': '0'} 2024-02-01T10:47:18.620 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000052ae572c9d0253c4-0065bb76b6-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:18.664 DEBUG:boto:path=/vclfcn-59/obj4 2024-02-01T10:47:18.664 DEBUG:boto:auth_path=/vclfcn-59/obj4 2024-02-01T10:47:18.664 DEBUG:boto:path=/vclfcn-59/obj4?versionId=null 2024-02-01T10:47:18.664 DEBUG:boto:auth_path=/vclfcn-59/obj4?versionId=null 2024-02-01T10:47:18.664 DEBUG:boto:Method: HEAD 2024-02-01T10:47:18.665 DEBUG:boto:Path: /vclfcn-59/obj4?versionId=null 2024-02-01T10:47:18.665 DEBUG:boto:Data: 2024-02-01T10:47:18.665 DEBUG:boto:Headers: {} 2024-02-01T10:47:18.665 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:18.665 DEBUG:boto:Port: 8000 2024-02-01T10:47:18.665 DEBUG:boto:Params: {} 2024-02-01T10:47:18.665 DEBUG:boto:Token: None 2024-02-01T10:47:18.665 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:18 GMT /vclfcn-59/obj4?versionId=null 2024-02-01T10:47:18.665 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:AwFdUZi1Sd+OBKJEgdn1dPr55Wo= 2024-02-01T10:47:18.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:AwFdUZi1Sd+OBKJEgdn1dPr55Wo=', 'Content-Length': '0'} 2024-02-01T10:47:18.668 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000081e6ecc237ea06dd-0065bb76b6-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:18.669 DEBUG:boto:path=/vclfcn-59/obj4 2024-02-01T10:47:18.669 DEBUG:boto:auth_path=/vclfcn-59/obj4 2024-02-01T10:47:18.669 DEBUG:boto:path=/vclfcn-59/obj4?versionId=null 2024-02-01T10:47:18.669 DEBUG:boto:auth_path=/vclfcn-59/obj4?versionId=null 2024-02-01T10:47:18.669 DEBUG:boto:Method: HEAD 2024-02-01T10:47:18.669 DEBUG:boto:Path: /vclfcn-59/obj4?versionId=null 2024-02-01T10:47:18.669 DEBUG:boto:Data: 2024-02-01T10:47:18.669 DEBUG:boto:Headers: {} 2024-02-01T10:47:18.670 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:18.670 DEBUG:boto:Port: 8000 2024-02-01T10:47:18.670 DEBUG:boto:Params: {} 2024-02-01T10:47:18.670 DEBUG:boto:Token: None 2024-02-01T10:47:18.670 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:18 GMT /vclfcn-59/obj4?versionId=null 2024-02-01T10:47:18.670 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:AwFdUZi1Sd+OBKJEgdn1dPr55Wo= 2024-02-01T10:47:18.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:AwFdUZi1Sd+OBKJEgdn1dPr55Wo=', 'Content-Length': '0'} 2024-02-01T10:47:18.673 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000004872b03fa15c173-0065bb76b6-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:18.673 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:47:18.673 DEBUG:boto:path=/vclfcn-59/obj4 2024-02-01T10:47:18.673 DEBUG:boto:auth_path=/vclfcn-59/obj4 2024-02-01T10:47:18.674 DEBUG:boto:path=/vclfcn-59/obj4?versionId=null 2024-02-01T10:47:18.674 DEBUG:boto:auth_path=/vclfcn-59/obj4?versionId=null 2024-02-01T10:47:18.674 DEBUG:boto:Method: GET 2024-02-01T10:47:18.674 DEBUG:boto:Path: /vclfcn-59/obj4?versionId=null 2024-02-01T10:47:18.674 DEBUG:boto:Data: 2024-02-01T10:47:18.674 DEBUG:boto:Headers: {} 2024-02-01T10:47:18.674 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:18.674 DEBUG:boto:Port: 8000 2024-02-01T10:47:18.674 DEBUG:boto:Params: {} 2024-02-01T10:47:18.674 DEBUG:boto:Token: None 2024-02-01T10:47:18.674 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:18 GMT /vclfcn-59/obj4?versionId=null 2024-02-01T10:47:18.675 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5nbKn9EFjhNc/+Y2/eTRmalQBmY= 2024-02-01T10:47:18.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5nbKn9EFjhNc/+Y2/eTRmalQBmY=', 'Content-Length': '0'} 2024-02-01T10:47:18.678 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006ecec809d824aacb-0065bb76b6-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:18.720 DEBUG:boto:path=/vclfcn-59/obj4 2024-02-01T10:47:18.720 DEBUG:boto:auth_path=/vclfcn-59/obj4 2024-02-01T10:47:18.720 DEBUG:boto:path=/vclfcn-59/obj4?versionId=null 2024-02-01T10:47:18.720 DEBUG:boto:auth_path=/vclfcn-59/obj4?versionId=null 2024-02-01T10:47:18.721 DEBUG:boto:Method: GET 2024-02-01T10:47:18.721 DEBUG:boto:Path: /vclfcn-59/obj4?versionId=null 2024-02-01T10:47:18.721 DEBUG:boto:Data: 2024-02-01T10:47:18.721 DEBUG:boto:Headers: {} 2024-02-01T10:47:18.721 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:18.721 DEBUG:boto:Port: 8000 2024-02-01T10:47:18.721 DEBUG:boto:Params: {} 2024-02-01T10:47:18.721 DEBUG:boto:Token: None 2024-02-01T10:47:18.722 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:18 GMT /vclfcn-59/obj4?versionId=null 2024-02-01T10:47:18.722 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5nbKn9EFjhNc/+Y2/eTRmalQBmY= 2024-02-01T10:47:18.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5nbKn9EFjhNc/+Y2/eTRmalQBmY=', 'Content-Length': '0'} 2024-02-01T10:47:18.725 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000039209cae339b53d6-0065bb76b6-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:18.725 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:47:18.725 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:47:18.725 DEBUG:boto:path=/vclfcn-59/?prefix=obj4 2024-02-01T10:47:18.725 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj4 2024-02-01T10:47:18.725 DEBUG:boto:Method: GET 2024-02-01T10:47:18.726 DEBUG:boto:Path: /vclfcn-59/?prefix=obj4 2024-02-01T10:47:18.726 DEBUG:boto:Data: 2024-02-01T10:47:18.726 DEBUG:boto:Headers: {} 2024-02-01T10:47:18.726 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:18.726 DEBUG:boto:Port: 8000 2024-02-01T10:47:18.726 DEBUG:boto:Params: {} 2024-02-01T10:47:18.726 DEBUG:boto:Token: None 2024-02-01T10:47:18.726 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:18 GMT /vclfcn-59/ 2024-02-01T10:47:18.726 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:g5MYBh+0JvaTqP0VigvFjp1Sa00= 2024-02-01T10:47:18.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:g5MYBh+0JvaTqP0VigvFjp1Sa00=', 'Content-Length': '0'} 2024-02-01T10:47:18.731 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb034697dc1590aa-0065bb76b6-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:18.772 DEBUG:boto:b'vclfcn-59obj41000falseobj42024-02-01T10:46:46.190Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:18.772 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:47:18.772 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:47:18.773 DEBUG:boto:path=/vclfcn-59/?prefix=obj4 2024-02-01T10:47:18.773 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj4 2024-02-01T10:47:18.773 DEBUG:boto:Method: GET 2024-02-01T10:47:18.773 DEBUG:boto:Path: /vclfcn-59/?prefix=obj4 2024-02-01T10:47:18.773 DEBUG:boto:Data: 2024-02-01T10:47:18.773 DEBUG:boto:Headers: {} 2024-02-01T10:47:18.773 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:18.773 DEBUG:boto:Port: 8000 2024-02-01T10:47:18.773 DEBUG:boto:Params: {} 2024-02-01T10:47:18.773 DEBUG:boto:Token: None 2024-02-01T10:47:18.774 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:18 GMT /vclfcn-59/ 2024-02-01T10:47:18.774 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:g5MYBh+0JvaTqP0VigvFjp1Sa00= 2024-02-01T10:47:18.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:g5MYBh+0JvaTqP0VigvFjp1Sa00=', 'Content-Length': '0'} 2024-02-01T10:47:18.778 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002a48e27784f9247-0065bb76b6-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:18.778 DEBUG:boto:b'vclfcn-59obj41000falseobj42024-02-01T10:46:46.190Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:18.779 DEBUG:boto:path=/vclfcn-59/obj4 2024-02-01T10:47:18.779 DEBUG:boto:auth_path=/vclfcn-59/obj4 2024-02-01T10:47:18.779 DEBUG:boto:Method: HEAD 2024-02-01T10:47:18.779 DEBUG:boto:Path: /vclfcn-59/obj4 2024-02-01T10:47:18.779 DEBUG:boto:Data: 2024-02-01T10:47:18.779 DEBUG:boto:Headers: {} 2024-02-01T10:47:18.779 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:18.779 DEBUG:boto:Port: 8000 2024-02-01T10:47:18.779 DEBUG:boto:Params: {} 2024-02-01T10:47:18.780 DEBUG:boto:Token: None 2024-02-01T10:47:18.780 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:18 GMT /vclfcn-59/obj4 2024-02-01T10:47:18.780 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:TwwRHvakBdUOdeuPIlpNBFzPW5w= 2024-02-01T10:47:18.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:TwwRHvakBdUOdeuPIlpNBFzPW5w=', 'Content-Length': '0'} 2024-02-01T10:47:18.782 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001b4dec51447461b7-0065bb76b6-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:18.783 DEBUG:boto:path=/vclfcn-59/obj4 2024-02-01T10:47:18.783 DEBUG:boto:auth_path=/vclfcn-59/obj4 2024-02-01T10:47:18.783 DEBUG:boto:Method: HEAD 2024-02-01T10:47:18.783 DEBUG:boto:Path: /vclfcn-59/obj4 2024-02-01T10:47:18.783 DEBUG:boto:Data: 2024-02-01T10:47:18.783 DEBUG:boto:Headers: {} 2024-02-01T10:47:18.783 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:18.783 DEBUG:boto:Port: 8000 2024-02-01T10:47:18.784 DEBUG:boto:Params: {} 2024-02-01T10:47:18.784 DEBUG:boto:Token: None 2024-02-01T10:47:18.784 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:18 GMT /vclfcn-59/obj4 2024-02-01T10:47:18.784 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:TwwRHvakBdUOdeuPIlpNBFzPW5w= 2024-02-01T10:47:18.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:TwwRHvakBdUOdeuPIlpNBFzPW5w=', 'Content-Length': '0'} 2024-02-01T10:47:18.787 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c706c438fe61401d-0065bb76b6-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:18.787 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:47:18.788 DEBUG:boto:path=/vclfcn-59/obj4 2024-02-01T10:47:18.788 DEBUG:boto:auth_path=/vclfcn-59/obj4 2024-02-01T10:47:18.788 DEBUG:boto:Method: GET 2024-02-01T10:47:18.788 DEBUG:boto:Path: /vclfcn-59/obj4 2024-02-01T10:47:18.788 DEBUG:boto:Data: 2024-02-01T10:47:18.788 DEBUG:boto:Headers: {} 2024-02-01T10:47:18.788 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:18.788 DEBUG:boto:Port: 8000 2024-02-01T10:47:18.788 DEBUG:boto:Params: {} 2024-02-01T10:47:18.789 DEBUG:boto:Token: None 2024-02-01T10:47:18.789 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:18 GMT /vclfcn-59/obj4 2024-02-01T10:47:18.789 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:860Jo6L8wwkCLh2/SvDkxLVxOJg= 2024-02-01T10:47:18.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:860Jo6L8wwkCLh2/SvDkxLVxOJg=', 'Content-Length': '0'} 2024-02-01T10:47:18.792 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000017e10b730fe98262-0065bb76b6-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:18.832 DEBUG:boto:path=/vclfcn-59/obj4 2024-02-01T10:47:18.832 DEBUG:boto:auth_path=/vclfcn-59/obj4 2024-02-01T10:47:18.832 DEBUG:boto:Method: GET 2024-02-01T10:47:18.832 DEBUG:boto:Path: /vclfcn-59/obj4 2024-02-01T10:47:18.833 DEBUG:boto:Data: 2024-02-01T10:47:18.833 DEBUG:boto:Headers: {} 2024-02-01T10:47:18.833 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:18.833 DEBUG:boto:Port: 8000 2024-02-01T10:47:18.833 DEBUG:boto:Params: {} 2024-02-01T10:47:18.833 DEBUG:boto:Token: None 2024-02-01T10:47:18.833 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:18 GMT /vclfcn-59/obj4 2024-02-01T10:47:18.833 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:860Jo6L8wwkCLh2/SvDkxLVxOJg= 2024-02-01T10:47:18.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:860Jo6L8wwkCLh2/SvDkxLVxOJg=', 'Content-Length': '0'} 2024-02-01T10:47:18.836 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000017821e62e129cb61-0065bb76b6-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:18.836 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:47:18.837 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:47:18.837 DEBUG:boto:path=/vclfcn-59/?prefix=obj4 2024-02-01T10:47:18.837 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj4 2024-02-01T10:47:18.837 DEBUG:boto:Method: GET 2024-02-01T10:47:18.837 DEBUG:boto:Path: /vclfcn-59/?prefix=obj4 2024-02-01T10:47:18.837 DEBUG:boto:Data: 2024-02-01T10:47:18.837 DEBUG:boto:Headers: {} 2024-02-01T10:47:18.837 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:18.837 DEBUG:boto:Port: 8000 2024-02-01T10:47:18.837 DEBUG:boto:Params: {} 2024-02-01T10:47:18.837 DEBUG:boto:Token: None 2024-02-01T10:47:18.838 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:18 GMT /vclfcn-59/ 2024-02-01T10:47:18.838 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:g5MYBh+0JvaTqP0VigvFjp1Sa00= 2024-02-01T10:47:18.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:g5MYBh+0JvaTqP0VigvFjp1Sa00=', 'Content-Length': '0'} 2024-02-01T10:47:18.843 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc932752cfacac34-0065bb76b6-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:18.884 DEBUG:boto:b'vclfcn-59obj41000falseobj42024-02-01T10:46:46.190Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:18.885 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:47:18.885 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:47:18.885 DEBUG:boto:path=/vclfcn-59/?prefix=obj4 2024-02-01T10:47:18.885 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj4 2024-02-01T10:47:18.885 DEBUG:boto:Method: GET 2024-02-01T10:47:18.885 DEBUG:boto:Path: /vclfcn-59/?prefix=obj4 2024-02-01T10:47:18.885 DEBUG:boto:Data: 2024-02-01T10:47:18.885 DEBUG:boto:Headers: {} 2024-02-01T10:47:18.885 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:18.885 DEBUG:boto:Port: 8000 2024-02-01T10:47:18.886 DEBUG:boto:Params: {} 2024-02-01T10:47:18.886 DEBUG:boto:Token: None 2024-02-01T10:47:18.886 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:18 GMT /vclfcn-59/ 2024-02-01T10:47:18.886 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:g5MYBh+0JvaTqP0VigvFjp1Sa00= 2024-02-01T10:47:18.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:g5MYBh+0JvaTqP0VigvFjp1Sa00=', 'Content-Length': '0'} 2024-02-01T10:47:18.890 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e0668e03d8548fd2-0065bb76b6-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:18.890 DEBUG:boto:b'vclfcn-59obj41000falseobj42024-02-01T10:46:46.190Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:18.890 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-59 zones={test-zone2, test-zone1} 2024-02-01T10:47:18.891 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-59 zones={test-zone3, test-zone1} 2024-02-01T10:47:18.891 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:47:18.891 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:47:18.891 DEBUG:boto:Method: HEAD 2024-02-01T10:47:18.891 DEBUG:boto:Path: /vclfcn-59/ 2024-02-01T10:47:18.891 DEBUG:boto:Data: 2024-02-01T10:47:18.891 DEBUG:boto:Headers: {} 2024-02-01T10:47:18.891 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:18.891 DEBUG:boto:Port: 8001 2024-02-01T10:47:18.891 DEBUG:boto:Params: {} 2024-02-01T10:47:18.892 DEBUG:boto:Token: None 2024-02-01T10:47:18.892 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:18 GMT /vclfcn-59/ 2024-02-01T10:47:18.892 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:EnG46xU0IUMC02wqB+VdsgHVZSc= 2024-02-01T10:47:18.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:EnG46xU0IUMC02wqB+VdsgHVZSc=', 'Content-Length': '0'} 2024-02-01T10:47:18.896 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a63a034b3c770c5f-0065bb76b6-5762-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:18.897 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:47:18.897 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:47:18.897 DEBUG:boto:Method: HEAD 2024-02-01T10:47:18.897 DEBUG:boto:Path: /vclfcn-59/ 2024-02-01T10:47:18.897 DEBUG:boto:Data: 2024-02-01T10:47:18.897 DEBUG:boto:Headers: {} 2024-02-01T10:47:18.897 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:18.897 DEBUG:boto:Port: 8000 2024-02-01T10:47:18.897 DEBUG:boto:Params: {} 2024-02-01T10:47:18.897 DEBUG:boto:Token: None 2024-02-01T10:47:18.898 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:18 GMT /vclfcn-59/ 2024-02-01T10:47:18.898 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:EnG46xU0IUMC02wqB+VdsgHVZSc= 2024-02-01T10:47:18.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:EnG46xU0IUMC02wqB+VdsgHVZSc=', 'Content-Length': '0'} 2024-02-01T10:47:18.901 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000781b1f9f19074466-0065bb76b6-5765-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:18.902 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:47:18.902 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:47:18.902 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:47:18.902 DEBUG:boto:path=/vclfcn-59/?versions&max-keys=999 2024-02-01T10:47:18.902 DEBUG:boto:auth_path=/vclfcn-59/?versions&max-keys=999 2024-02-01T10:47:18.902 DEBUG:boto:Method: GET 2024-02-01T10:47:18.902 DEBUG:boto:Path: /vclfcn-59/?versions&max-keys=999 2024-02-01T10:47:18.902 DEBUG:boto:Data: 2024-02-01T10:47:18.902 DEBUG:boto:Headers: {} 2024-02-01T10:47:18.902 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:18.903 DEBUG:boto:Port: 8001 2024-02-01T10:47:18.903 DEBUG:boto:Params: {} 2024-02-01T10:47:18.903 DEBUG:boto:Token: None 2024-02-01T10:47:18.903 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:18 GMT /vclfcn-59/?versions 2024-02-01T10:47:18.903 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ynzvhs00eITKwaGLrLXr9R+WZjE= 2024-02-01T10:47:18.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ynzvhs00eITKwaGLrLXr9R+WZjE=', 'Content-Length': '0'} 2024-02-01T10:47:18.908 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3007f7d9f656b49-0065bb76b6-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:18.908 DEBUG:boto:b'vclfcn-59999falseobj1nulltrue2024-02-01T10:46:46.093Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:46:46.111Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:46:46.161Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:46:46.190Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:18.908 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-02-01T10:47:18.909 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-02-01T10:47:18.909 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-02-01T10:47:18.909 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-02-01T10:47:18.909 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:47:18.909 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:47:18.909 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:47:18.909 DEBUG:boto:path=/vclfcn-59/?versions&max-keys=999 2024-02-01T10:47:18.909 DEBUG:boto:auth_path=/vclfcn-59/?versions&max-keys=999 2024-02-01T10:47:18.909 DEBUG:boto:Method: GET 2024-02-01T10:47:18.910 DEBUG:boto:Path: /vclfcn-59/?versions&max-keys=999 2024-02-01T10:47:18.910 DEBUG:boto:Data: 2024-02-01T10:47:18.910 DEBUG:boto:Headers: {} 2024-02-01T10:47:18.910 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:18.910 DEBUG:boto:Port: 8000 2024-02-01T10:47:18.910 DEBUG:boto:Params: {} 2024-02-01T10:47:18.910 DEBUG:boto:Token: None 2024-02-01T10:47:18.910 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:18 GMT /vclfcn-59/?versions 2024-02-01T10:47:18.910 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ynzvhs00eITKwaGLrLXr9R+WZjE= 2024-02-01T10:47:18.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ynzvhs00eITKwaGLrLXr9R+WZjE=', 'Content-Length': '0'} 2024-02-01T10:47:18.915 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b19f958d81ab66c9-0065bb76b6-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:18.915 DEBUG:boto:b'vclfcn-59999falseobj1nulltrue2024-02-01T10:46:46.093Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:46:46.111Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:46:46.161Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:46:46.190Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:18.915 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-02-01T10:47:18.915 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-02-01T10:47:18.915 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-02-01T10:47:18.916 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-02-01T10:47:18.916 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:47:18.916 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:47:18.916 DEBUG:boto:path=/vclfcn-59/?versions&max-keys=999 2024-02-01T10:47:18.916 DEBUG:boto:auth_path=/vclfcn-59/?versions&max-keys=999 2024-02-01T10:47:18.916 DEBUG:boto:Method: GET 2024-02-01T10:47:18.916 DEBUG:boto:Path: /vclfcn-59/?versions&max-keys=999 2024-02-01T10:47:18.916 DEBUG:boto:Data: 2024-02-01T10:47:18.916 DEBUG:boto:Headers: {} 2024-02-01T10:47:18.916 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:18.917 DEBUG:boto:Port: 8001 2024-02-01T10:47:18.917 DEBUG:boto:Params: {} 2024-02-01T10:47:18.917 DEBUG:boto:Token: None 2024-02-01T10:47:18.917 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:18 GMT /vclfcn-59/?versions 2024-02-01T10:47:18.917 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ynzvhs00eITKwaGLrLXr9R+WZjE= 2024-02-01T10:47:18.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ynzvhs00eITKwaGLrLXr9R+WZjE=', 'Content-Length': '0'} 2024-02-01T10:47:18.921 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060eb1d9ad41e49d0-0065bb76b6-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:18.922 DEBUG:boto:b'vclfcn-59999falseobj1nulltrue2024-02-01T10:46:46.093Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:46:46.111Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:46:46.161Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:46:46.190Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:18.922 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:47:18.922 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:47:18.922 DEBUG:boto:path=/vclfcn-59/?versions&max-keys=999 2024-02-01T10:47:18.922 DEBUG:boto:auth_path=/vclfcn-59/?versions&max-keys=999 2024-02-01T10:47:18.923 DEBUG:boto:Method: GET 2024-02-01T10:47:18.923 DEBUG:boto:Path: /vclfcn-59/?versions&max-keys=999 2024-02-01T10:47:18.923 DEBUG:boto:Data: 2024-02-01T10:47:18.923 DEBUG:boto:Headers: {} 2024-02-01T10:47:18.923 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:18.923 DEBUG:boto:Port: 8000 2024-02-01T10:47:18.923 DEBUG:boto:Params: {} 2024-02-01T10:47:18.923 DEBUG:boto:Token: None 2024-02-01T10:47:18.923 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:18 GMT /vclfcn-59/?versions 2024-02-01T10:47:18.923 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ynzvhs00eITKwaGLrLXr9R+WZjE= 2024-02-01T10:47:18.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ynzvhs00eITKwaGLrLXr9R+WZjE=', 'Content-Length': '0'} 2024-02-01T10:47:18.928 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000006d58f7442241c30-0065bb76b6-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:18.928 DEBUG:boto:b'vclfcn-59999falseobj1nulltrue2024-02-01T10:46:46.093Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:46:46.111Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:46:46.161Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:46:46.190Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:18.928 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:47:18.929 DEBUG:boto:path=/vclfcn-59/obj1 2024-02-01T10:47:18.929 DEBUG:boto:auth_path=/vclfcn-59/obj1 2024-02-01T10:47:18.929 DEBUG:boto:path=/vclfcn-59/obj1?versionId=null 2024-02-01T10:47:18.929 DEBUG:boto:auth_path=/vclfcn-59/obj1?versionId=null 2024-02-01T10:47:18.929 DEBUG:boto:Method: GET 2024-02-01T10:47:18.929 DEBUG:boto:Path: /vclfcn-59/obj1?versionId=null 2024-02-01T10:47:18.929 DEBUG:boto:Data: 2024-02-01T10:47:18.929 DEBUG:boto:Headers: {} 2024-02-01T10:47:18.929 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:18.929 DEBUG:boto:Port: 8001 2024-02-01T10:47:18.929 DEBUG:boto:Params: {} 2024-02-01T10:47:18.930 DEBUG:boto:Token: None 2024-02-01T10:47:18.930 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:18 GMT /vclfcn-59/obj1?versionId=null 2024-02-01T10:47:18.930 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nPebUgLIoFopDGyHDUoYRYSknRk= 2024-02-01T10:47:18.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nPebUgLIoFopDGyHDUoYRYSknRk=', 'Content-Length': '0'} 2024-02-01T10:47:18.932 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001ff259a33fecf62e-0065bb76b6-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:18.976 DEBUG:boto:path=/vclfcn-59/obj1 2024-02-01T10:47:18.976 DEBUG:boto:auth_path=/vclfcn-59/obj1 2024-02-01T10:47:18.976 DEBUG:boto:path=/vclfcn-59/obj1?versionId=null 2024-02-01T10:47:18.976 DEBUG:boto:auth_path=/vclfcn-59/obj1?versionId=null 2024-02-01T10:47:18.977 DEBUG:boto:Method: GET 2024-02-01T10:47:18.977 DEBUG:boto:Path: /vclfcn-59/obj1?versionId=null 2024-02-01T10:47:18.977 DEBUG:boto:Data: 2024-02-01T10:47:18.977 DEBUG:boto:Headers: {} 2024-02-01T10:47:18.977 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:18.977 DEBUG:boto:Port: 8000 2024-02-01T10:47:18.977 DEBUG:boto:Params: {} 2024-02-01T10:47:18.977 DEBUG:boto:Token: None 2024-02-01T10:47:18.977 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:18 GMT /vclfcn-59/obj1?versionId=null 2024-02-01T10:47:18.978 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nPebUgLIoFopDGyHDUoYRYSknRk= 2024-02-01T10:47:18.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nPebUgLIoFopDGyHDUoYRYSknRk=', 'Content-Length': '0'} 2024-02-01T10:47:18.981 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000060338ab5aee2a3e2-0065bb76b6-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:18.981 DEBUG:boto:path=/vclfcn-59/obj1 2024-02-01T10:47:18.981 DEBUG:boto:auth_path=/vclfcn-59/obj1 2024-02-01T10:47:18.981 DEBUG:boto:path=/vclfcn-59/obj1?versionId=null 2024-02-01T10:47:18.981 DEBUG:boto:auth_path=/vclfcn-59/obj1?versionId=null 2024-02-01T10:47:18.981 DEBUG:boto:Method: HEAD 2024-02-01T10:47:18.981 DEBUG:boto:Path: /vclfcn-59/obj1?versionId=null 2024-02-01T10:47:18.981 DEBUG:boto:Data: 2024-02-01T10:47:18.982 DEBUG:boto:Headers: {} 2024-02-01T10:47:18.982 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:18.982 DEBUG:boto:Port: 8001 2024-02-01T10:47:18.982 DEBUG:boto:Params: {} 2024-02-01T10:47:18.982 DEBUG:boto:Token: None 2024-02-01T10:47:18.982 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:18 GMT /vclfcn-59/obj1?versionId=null 2024-02-01T10:47:18.982 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8h+7zcsX+YEIucVqr9F0TFBmPAY= 2024-02-01T10:47:18.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8h+7zcsX+YEIucVqr9F0TFBmPAY=', 'Content-Length': '0'} 2024-02-01T10:47:18.985 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008f5e07268ed9233a-0065bb76b6-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:18.985 DEBUG:boto:path=/vclfcn-59/obj1 2024-02-01T10:47:18.986 DEBUG:boto:auth_path=/vclfcn-59/obj1 2024-02-01T10:47:18.986 DEBUG:boto:path=/vclfcn-59/obj1?versionId=null 2024-02-01T10:47:18.986 DEBUG:boto:auth_path=/vclfcn-59/obj1?versionId=null 2024-02-01T10:47:18.986 DEBUG:boto:Method: HEAD 2024-02-01T10:47:18.986 DEBUG:boto:Path: /vclfcn-59/obj1?versionId=null 2024-02-01T10:47:18.986 DEBUG:boto:Data: 2024-02-01T10:47:18.986 DEBUG:boto:Headers: {} 2024-02-01T10:47:18.986 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:18.986 DEBUG:boto:Port: 8000 2024-02-01T10:47:18.986 DEBUG:boto:Params: {} 2024-02-01T10:47:18.987 DEBUG:boto:Token: None 2024-02-01T10:47:18.987 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:18 GMT /vclfcn-59/obj1?versionId=null 2024-02-01T10:47:18.987 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8h+7zcsX+YEIucVqr9F0TFBmPAY= 2024-02-01T10:47:18.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8h+7zcsX+YEIucVqr9F0TFBmPAY=', 'Content-Length': '0'} 2024-02-01T10:47:18.990 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000af3c5e8f0bd94345-0065bb76b6-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:18.990 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:47:18.990 DEBUG:boto:path=/vclfcn-59/obj1 2024-02-01T10:47:18.990 DEBUG:boto:auth_path=/vclfcn-59/obj1 2024-02-01T10:47:18.990 DEBUG:boto:path=/vclfcn-59/obj1?versionId=null 2024-02-01T10:47:18.990 DEBUG:boto:auth_path=/vclfcn-59/obj1?versionId=null 2024-02-01T10:47:18.990 DEBUG:boto:Method: GET 2024-02-01T10:47:18.991 DEBUG:boto:Path: /vclfcn-59/obj1?versionId=null 2024-02-01T10:47:18.991 DEBUG:boto:Data: 2024-02-01T10:47:18.991 DEBUG:boto:Headers: {} 2024-02-01T10:47:18.991 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:18.991 DEBUG:boto:Port: 8001 2024-02-01T10:47:18.991 DEBUG:boto:Params: {} 2024-02-01T10:47:18.991 DEBUG:boto:Token: None 2024-02-01T10:47:18.991 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:18 GMT /vclfcn-59/obj1?versionId=null 2024-02-01T10:47:18.991 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nPebUgLIoFopDGyHDUoYRYSknRk= 2024-02-01T10:47:18.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nPebUgLIoFopDGyHDUoYRYSknRk=', 'Content-Length': '0'} 2024-02-01T10:47:18.994 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000097d1df9fc81beb85-0065bb76b6-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:19.036 DEBUG:boto:path=/vclfcn-59/obj1 2024-02-01T10:47:19.036 DEBUG:boto:auth_path=/vclfcn-59/obj1 2024-02-01T10:47:19.036 DEBUG:boto:path=/vclfcn-59/obj1?versionId=null 2024-02-01T10:47:19.037 DEBUG:boto:auth_path=/vclfcn-59/obj1?versionId=null 2024-02-01T10:47:19.037 DEBUG:boto:Method: GET 2024-02-01T10:47:19.037 DEBUG:boto:Path: /vclfcn-59/obj1?versionId=null 2024-02-01T10:47:19.037 DEBUG:boto:Data: 2024-02-01T10:47:19.037 DEBUG:boto:Headers: {} 2024-02-01T10:47:19.037 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:19.037 DEBUG:boto:Port: 8000 2024-02-01T10:47:19.037 DEBUG:boto:Params: {} 2024-02-01T10:47:19.037 DEBUG:boto:Token: None 2024-02-01T10:47:19.037 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:19 GMT /vclfcn-59/obj1?versionId=null 2024-02-01T10:47:19.038 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:si7xzTjOV181avV0UJ/V7cIhMbc= 2024-02-01T10:47:19.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:si7xzTjOV181avV0UJ/V7cIhMbc=', 'Content-Length': '0'} 2024-02-01T10:47:19.041 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000053d1f61c2ba2d4aa-0065bb76b7-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:19.041 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:47:19.042 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:47:19.042 DEBUG:boto:path=/vclfcn-59/?prefix=obj1 2024-02-01T10:47:19.042 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj1 2024-02-01T10:47:19.042 DEBUG:boto:Method: GET 2024-02-01T10:47:19.042 DEBUG:boto:Path: /vclfcn-59/?prefix=obj1 2024-02-01T10:47:19.042 DEBUG:boto:Data: 2024-02-01T10:47:19.042 DEBUG:boto:Headers: {} 2024-02-01T10:47:19.042 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:19.042 DEBUG:boto:Port: 8001 2024-02-01T10:47:19.042 DEBUG:boto:Params: {} 2024-02-01T10:47:19.043 DEBUG:boto:Token: None 2024-02-01T10:47:19.043 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:19 GMT /vclfcn-59/ 2024-02-01T10:47:19.043 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8lQZdEka3uCMRAGzvQOFxqE0ZoI= 2024-02-01T10:47:19.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8lQZdEka3uCMRAGzvQOFxqE0ZoI=', 'Content-Length': '0'} 2024-02-01T10:47:19.047 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4cc09dfac0affbe-0065bb76b7-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:19.088 DEBUG:boto:b'vclfcn-59obj11000falseobj12024-02-01T10:46:46.093Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:19.088 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:47:19.088 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:47:19.089 DEBUG:boto:path=/vclfcn-59/?prefix=obj1 2024-02-01T10:47:19.089 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj1 2024-02-01T10:47:19.089 DEBUG:boto:Method: GET 2024-02-01T10:47:19.089 DEBUG:boto:Path: /vclfcn-59/?prefix=obj1 2024-02-01T10:47:19.089 DEBUG:boto:Data: 2024-02-01T10:47:19.089 DEBUG:boto:Headers: {} 2024-02-01T10:47:19.089 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:19.089 DEBUG:boto:Port: 8000 2024-02-01T10:47:19.089 DEBUG:boto:Params: {} 2024-02-01T10:47:19.089 DEBUG:boto:Token: None 2024-02-01T10:47:19.090 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:19 GMT /vclfcn-59/ 2024-02-01T10:47:19.090 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8lQZdEka3uCMRAGzvQOFxqE0ZoI= 2024-02-01T10:47:19.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8lQZdEka3uCMRAGzvQOFxqE0ZoI=', 'Content-Length': '0'} 2024-02-01T10:47:19.095 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055418624ac9713be-0065bb76b7-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:19.096 DEBUG:boto:b'vclfcn-59obj11000falseobj12024-02-01T10:46:46.093Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:19.096 DEBUG:boto:path=/vclfcn-59/obj1 2024-02-01T10:47:19.096 DEBUG:boto:auth_path=/vclfcn-59/obj1 2024-02-01T10:47:19.097 DEBUG:boto:Method: HEAD 2024-02-01T10:47:19.097 DEBUG:boto:Path: /vclfcn-59/obj1 2024-02-01T10:47:19.097 DEBUG:boto:Data: 2024-02-01T10:47:19.097 DEBUG:boto:Headers: {} 2024-02-01T10:47:19.097 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:19.097 DEBUG:boto:Port: 8001 2024-02-01T10:47:19.097 DEBUG:boto:Params: {} 2024-02-01T10:47:19.097 DEBUG:boto:Token: None 2024-02-01T10:47:19.097 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:19 GMT /vclfcn-59/obj1 2024-02-01T10:47:19.098 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:UUzA7dNpc+THCxPTcBNyIEAgdvI= 2024-02-01T10:47:19.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:UUzA7dNpc+THCxPTcBNyIEAgdvI=', 'Content-Length': '0'} 2024-02-01T10:47:19.100 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000005101568319aaee7-0065bb76b7-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:19.100 DEBUG:boto:path=/vclfcn-59/obj1 2024-02-01T10:47:19.101 DEBUG:boto:auth_path=/vclfcn-59/obj1 2024-02-01T10:47:19.101 DEBUG:boto:Method: HEAD 2024-02-01T10:47:19.101 DEBUG:boto:Path: /vclfcn-59/obj1 2024-02-01T10:47:19.101 DEBUG:boto:Data: 2024-02-01T10:47:19.101 DEBUG:boto:Headers: {} 2024-02-01T10:47:19.101 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:19.101 DEBUG:boto:Port: 8000 2024-02-01T10:47:19.101 DEBUG:boto:Params: {} 2024-02-01T10:47:19.101 DEBUG:boto:Token: None 2024-02-01T10:47:19.101 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:19 GMT /vclfcn-59/obj1 2024-02-01T10:47:19.102 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:UUzA7dNpc+THCxPTcBNyIEAgdvI= 2024-02-01T10:47:19.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:UUzA7dNpc+THCxPTcBNyIEAgdvI=', 'Content-Length': '0'} 2024-02-01T10:47:19.104 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007a4b5965e888cf2b-0065bb76b7-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:19.104 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:47:19.104 DEBUG:boto:path=/vclfcn-59/obj1 2024-02-01T10:47:19.105 DEBUG:boto:auth_path=/vclfcn-59/obj1 2024-02-01T10:47:19.105 DEBUG:boto:Method: GET 2024-02-01T10:47:19.105 DEBUG:boto:Path: /vclfcn-59/obj1 2024-02-01T10:47:19.105 DEBUG:boto:Data: 2024-02-01T10:47:19.105 DEBUG:boto:Headers: {} 2024-02-01T10:47:19.105 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:19.105 DEBUG:boto:Port: 8001 2024-02-01T10:47:19.105 DEBUG:boto:Params: {} 2024-02-01T10:47:19.105 DEBUG:boto:Token: None 2024-02-01T10:47:19.105 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:19 GMT /vclfcn-59/obj1 2024-02-01T10:47:19.106 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:LWgSpyniuA/FNnMbrwvHSRawFAU= 2024-02-01T10:47:19.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:LWgSpyniuA/FNnMbrwvHSRawFAU=', 'Content-Length': '0'} 2024-02-01T10:47:19.108 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d4a0704a8fd9ff9f-0065bb76b7-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:19.152 DEBUG:boto:path=/vclfcn-59/obj1 2024-02-01T10:47:19.152 DEBUG:boto:auth_path=/vclfcn-59/obj1 2024-02-01T10:47:19.152 DEBUG:boto:Method: GET 2024-02-01T10:47:19.152 DEBUG:boto:Path: /vclfcn-59/obj1 2024-02-01T10:47:19.152 DEBUG:boto:Data: 2024-02-01T10:47:19.153 DEBUG:boto:Headers: {} 2024-02-01T10:47:19.153 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:19.153 DEBUG:boto:Port: 8000 2024-02-01T10:47:19.153 DEBUG:boto:Params: {} 2024-02-01T10:47:19.153 DEBUG:boto:Token: None 2024-02-01T10:47:19.153 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:19 GMT /vclfcn-59/obj1 2024-02-01T10:47:19.153 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:LWgSpyniuA/FNnMbrwvHSRawFAU= 2024-02-01T10:47:19.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:LWgSpyniuA/FNnMbrwvHSRawFAU=', 'Content-Length': '0'} 2024-02-01T10:47:19.156 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000064c033c4fa0e7588-0065bb76b7-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:19.156 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:47:19.157 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:47:19.157 DEBUG:boto:path=/vclfcn-59/?prefix=obj1 2024-02-01T10:47:19.157 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj1 2024-02-01T10:47:19.157 DEBUG:boto:Method: GET 2024-02-01T10:47:19.157 DEBUG:boto:Path: /vclfcn-59/?prefix=obj1 2024-02-01T10:47:19.157 DEBUG:boto:Data: 2024-02-01T10:47:19.157 DEBUG:boto:Headers: {} 2024-02-01T10:47:19.157 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:19.157 DEBUG:boto:Port: 8001 2024-02-01T10:47:19.157 DEBUG:boto:Params: {} 2024-02-01T10:47:19.157 DEBUG:boto:Token: None 2024-02-01T10:47:19.158 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:19 GMT /vclfcn-59/ 2024-02-01T10:47:19.158 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8lQZdEka3uCMRAGzvQOFxqE0ZoI= 2024-02-01T10:47:19.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8lQZdEka3uCMRAGzvQOFxqE0ZoI=', 'Content-Length': '0'} 2024-02-01T10:47:19.162 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000186e28a26ea3412e-0065bb76b7-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:19.204 DEBUG:boto:b'vclfcn-59obj11000falseobj12024-02-01T10:46:46.093Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:19.205 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:47:19.205 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:47:19.205 DEBUG:boto:path=/vclfcn-59/?prefix=obj1 2024-02-01T10:47:19.205 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj1 2024-02-01T10:47:19.205 DEBUG:boto:Method: GET 2024-02-01T10:47:19.205 DEBUG:boto:Path: /vclfcn-59/?prefix=obj1 2024-02-01T10:47:19.205 DEBUG:boto:Data: 2024-02-01T10:47:19.205 DEBUG:boto:Headers: {} 2024-02-01T10:47:19.205 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:19.205 DEBUG:boto:Port: 8000 2024-02-01T10:47:19.206 DEBUG:boto:Params: {} 2024-02-01T10:47:19.206 DEBUG:boto:Token: None 2024-02-01T10:47:19.206 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:19 GMT /vclfcn-59/ 2024-02-01T10:47:19.206 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8lQZdEka3uCMRAGzvQOFxqE0ZoI= 2024-02-01T10:47:19.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8lQZdEka3uCMRAGzvQOFxqE0ZoI=', 'Content-Length': '0'} 2024-02-01T10:47:19.210 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d78f364e9836a13-0065bb76b7-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:19.210 DEBUG:boto:b'vclfcn-59obj11000falseobj12024-02-01T10:46:46.093Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:19.211 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:47:19.211 DEBUG:boto:path=/vclfcn-59/obj2 2024-02-01T10:47:19.211 DEBUG:boto:auth_path=/vclfcn-59/obj2 2024-02-01T10:47:19.211 DEBUG:boto:path=/vclfcn-59/obj2?versionId=null 2024-02-01T10:47:19.211 DEBUG:boto:auth_path=/vclfcn-59/obj2?versionId=null 2024-02-01T10:47:19.211 DEBUG:boto:Method: GET 2024-02-01T10:47:19.211 DEBUG:boto:Path: /vclfcn-59/obj2?versionId=null 2024-02-01T10:47:19.212 DEBUG:boto:Data: 2024-02-01T10:47:19.212 DEBUG:boto:Headers: {} 2024-02-01T10:47:19.212 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:19.212 DEBUG:boto:Port: 8001 2024-02-01T10:47:19.212 DEBUG:boto:Params: {} 2024-02-01T10:47:19.212 DEBUG:boto:Token: None 2024-02-01T10:47:19.212 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:19 GMT /vclfcn-59/obj2?versionId=null 2024-02-01T10:47:19.212 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:kKHMZFKtBpYU4jPcQtJCJ0s0K8A= 2024-02-01T10:47:19.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:kKHMZFKtBpYU4jPcQtJCJ0s0K8A=', 'Content-Length': '0'} 2024-02-01T10:47:19.215 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f67dbb9497590565-0065bb76b7-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:19.216 DEBUG:boto:path=/vclfcn-59/obj2 2024-02-01T10:47:19.216 DEBUG:boto:auth_path=/vclfcn-59/obj2 2024-02-01T10:47:19.216 DEBUG:boto:path=/vclfcn-59/obj2?versionId=null 2024-02-01T10:47:19.216 DEBUG:boto:auth_path=/vclfcn-59/obj2?versionId=null 2024-02-01T10:47:19.216 DEBUG:boto:Method: GET 2024-02-01T10:47:19.216 DEBUG:boto:Path: /vclfcn-59/obj2?versionId=null 2024-02-01T10:47:19.216 DEBUG:boto:Data: 2024-02-01T10:47:19.216 DEBUG:boto:Headers: {} 2024-02-01T10:47:19.216 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:19.216 DEBUG:boto:Port: 8000 2024-02-01T10:47:19.217 DEBUG:boto:Params: {} 2024-02-01T10:47:19.217 DEBUG:boto:Token: None 2024-02-01T10:47:19.217 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:19 GMT /vclfcn-59/obj2?versionId=null 2024-02-01T10:47:19.217 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:kKHMZFKtBpYU4jPcQtJCJ0s0K8A= 2024-02-01T10:47:19.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:kKHMZFKtBpYU4jPcQtJCJ0s0K8A=', 'Content-Length': '0'} 2024-02-01T10:47:19.220 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c5c586643ee16041-0065bb76b7-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:19.264 DEBUG:boto:path=/vclfcn-59/obj2 2024-02-01T10:47:19.264 DEBUG:boto:auth_path=/vclfcn-59/obj2 2024-02-01T10:47:19.264 DEBUG:boto:path=/vclfcn-59/obj2?versionId=null 2024-02-01T10:47:19.264 DEBUG:boto:auth_path=/vclfcn-59/obj2?versionId=null 2024-02-01T10:47:19.265 DEBUG:boto:Method: HEAD 2024-02-01T10:47:19.265 DEBUG:boto:Path: /vclfcn-59/obj2?versionId=null 2024-02-01T10:47:19.265 DEBUG:boto:Data: 2024-02-01T10:47:19.265 DEBUG:boto:Headers: {} 2024-02-01T10:47:19.265 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:19.265 DEBUG:boto:Port: 8001 2024-02-01T10:47:19.265 DEBUG:boto:Params: {} 2024-02-01T10:47:19.265 DEBUG:boto:Token: None 2024-02-01T10:47:19.265 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:19 GMT /vclfcn-59/obj2?versionId=null 2024-02-01T10:47:19.266 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fCIEr/3T2uscvWtvaJLDyd9kPkw= 2024-02-01T10:47:19.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fCIEr/3T2uscvWtvaJLDyd9kPkw=', 'Content-Length': '0'} 2024-02-01T10:47:19.268 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000139e91b05063c44b-0065bb76b7-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:19.269 DEBUG:boto:path=/vclfcn-59/obj2 2024-02-01T10:47:19.269 DEBUG:boto:auth_path=/vclfcn-59/obj2 2024-02-01T10:47:19.269 DEBUG:boto:path=/vclfcn-59/obj2?versionId=null 2024-02-01T10:47:19.269 DEBUG:boto:auth_path=/vclfcn-59/obj2?versionId=null 2024-02-01T10:47:19.269 DEBUG:boto:Method: HEAD 2024-02-01T10:47:19.269 DEBUG:boto:Path: /vclfcn-59/obj2?versionId=null 2024-02-01T10:47:19.269 DEBUG:boto:Data: 2024-02-01T10:47:19.269 DEBUG:boto:Headers: {} 2024-02-01T10:47:19.270 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:19.270 DEBUG:boto:Port: 8000 2024-02-01T10:47:19.270 DEBUG:boto:Params: {} 2024-02-01T10:47:19.270 DEBUG:boto:Token: None 2024-02-01T10:47:19.270 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:19 GMT /vclfcn-59/obj2?versionId=null 2024-02-01T10:47:19.270 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fCIEr/3T2uscvWtvaJLDyd9kPkw= 2024-02-01T10:47:19.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fCIEr/3T2uscvWtvaJLDyd9kPkw=', 'Content-Length': '0'} 2024-02-01T10:47:19.273 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000005ab6434d7d21c23-0065bb76b7-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:19.273 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:47:19.273 DEBUG:boto:path=/vclfcn-59/obj2 2024-02-01T10:47:19.273 DEBUG:boto:auth_path=/vclfcn-59/obj2 2024-02-01T10:47:19.273 DEBUG:boto:path=/vclfcn-59/obj2?versionId=null 2024-02-01T10:47:19.273 DEBUG:boto:auth_path=/vclfcn-59/obj2?versionId=null 2024-02-01T10:47:19.274 DEBUG:boto:Method: GET 2024-02-01T10:47:19.274 DEBUG:boto:Path: /vclfcn-59/obj2?versionId=null 2024-02-01T10:47:19.274 DEBUG:boto:Data: 2024-02-01T10:47:19.274 DEBUG:boto:Headers: {} 2024-02-01T10:47:19.274 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:19.274 DEBUG:boto:Port: 8001 2024-02-01T10:47:19.274 DEBUG:boto:Params: {} 2024-02-01T10:47:19.274 DEBUG:boto:Token: None 2024-02-01T10:47:19.274 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:19 GMT /vclfcn-59/obj2?versionId=null 2024-02-01T10:47:19.274 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:kKHMZFKtBpYU4jPcQtJCJ0s0K8A= 2024-02-01T10:47:19.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:kKHMZFKtBpYU4jPcQtJCJ0s0K8A=', 'Content-Length': '0'} 2024-02-01T10:47:19.277 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f32f679c8dbcd451-0065bb76b7-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:19.320 DEBUG:boto:path=/vclfcn-59/obj2 2024-02-01T10:47:19.320 DEBUG:boto:auth_path=/vclfcn-59/obj2 2024-02-01T10:47:19.320 DEBUG:boto:path=/vclfcn-59/obj2?versionId=null 2024-02-01T10:47:19.320 DEBUG:boto:auth_path=/vclfcn-59/obj2?versionId=null 2024-02-01T10:47:19.321 DEBUG:boto:Method: GET 2024-02-01T10:47:19.321 DEBUG:boto:Path: /vclfcn-59/obj2?versionId=null 2024-02-01T10:47:19.321 DEBUG:boto:Data: 2024-02-01T10:47:19.321 DEBUG:boto:Headers: {} 2024-02-01T10:47:19.321 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:19.321 DEBUG:boto:Port: 8000 2024-02-01T10:47:19.321 DEBUG:boto:Params: {} 2024-02-01T10:47:19.321 DEBUG:boto:Token: None 2024-02-01T10:47:19.321 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:19 GMT /vclfcn-59/obj2?versionId=null 2024-02-01T10:47:19.322 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:kKHMZFKtBpYU4jPcQtJCJ0s0K8A= 2024-02-01T10:47:19.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:kKHMZFKtBpYU4jPcQtJCJ0s0K8A=', 'Content-Length': '0'} 2024-02-01T10:47:19.325 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004164ab8beae3427e-0065bb76b7-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:19.325 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:47:19.325 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:47:19.325 DEBUG:boto:path=/vclfcn-59/?prefix=obj2 2024-02-01T10:47:19.326 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj2 2024-02-01T10:47:19.326 DEBUG:boto:Method: GET 2024-02-01T10:47:19.326 DEBUG:boto:Path: /vclfcn-59/?prefix=obj2 2024-02-01T10:47:19.326 DEBUG:boto:Data: 2024-02-01T10:47:19.326 DEBUG:boto:Headers: {} 2024-02-01T10:47:19.326 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:19.326 DEBUG:boto:Port: 8001 2024-02-01T10:47:19.326 DEBUG:boto:Params: {} 2024-02-01T10:47:19.326 DEBUG:boto:Token: None 2024-02-01T10:47:19.326 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:19 GMT /vclfcn-59/ 2024-02-01T10:47:19.327 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8lQZdEka3uCMRAGzvQOFxqE0ZoI= 2024-02-01T10:47:19.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8lQZdEka3uCMRAGzvQOFxqE0ZoI=', 'Content-Length': '0'} 2024-02-01T10:47:19.331 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001bffce17fe68394e-0065bb76b7-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:19.372 DEBUG:boto:b'vclfcn-59obj21000falseobj22024-02-01T10:46:46.111Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:19.373 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:47:19.373 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:47:19.373 DEBUG:boto:path=/vclfcn-59/?prefix=obj2 2024-02-01T10:47:19.373 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj2 2024-02-01T10:47:19.373 DEBUG:boto:Method: GET 2024-02-01T10:47:19.373 DEBUG:boto:Path: /vclfcn-59/?prefix=obj2 2024-02-01T10:47:19.373 DEBUG:boto:Data: 2024-02-01T10:47:19.374 DEBUG:boto:Headers: {} 2024-02-01T10:47:19.374 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:19.374 DEBUG:boto:Port: 8000 2024-02-01T10:47:19.374 DEBUG:boto:Params: {} 2024-02-01T10:47:19.374 DEBUG:boto:Token: None 2024-02-01T10:47:19.374 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:19 GMT /vclfcn-59/ 2024-02-01T10:47:19.374 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8lQZdEka3uCMRAGzvQOFxqE0ZoI= 2024-02-01T10:47:19.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8lQZdEka3uCMRAGzvQOFxqE0ZoI=', 'Content-Length': '0'} 2024-02-01T10:47:19.378 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000742598204fb85a46-0065bb76b7-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:19.379 DEBUG:boto:b'vclfcn-59obj21000falseobj22024-02-01T10:46:46.111Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:19.379 DEBUG:boto:path=/vclfcn-59/obj2 2024-02-01T10:47:19.379 DEBUG:boto:auth_path=/vclfcn-59/obj2 2024-02-01T10:47:19.379 DEBUG:boto:Method: HEAD 2024-02-01T10:47:19.379 DEBUG:boto:Path: /vclfcn-59/obj2 2024-02-01T10:47:19.380 DEBUG:boto:Data: 2024-02-01T10:47:19.380 DEBUG:boto:Headers: {} 2024-02-01T10:47:19.380 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:19.380 DEBUG:boto:Port: 8001 2024-02-01T10:47:19.380 DEBUG:boto:Params: {} 2024-02-01T10:47:19.380 DEBUG:boto:Token: None 2024-02-01T10:47:19.380 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:19 GMT /vclfcn-59/obj2 2024-02-01T10:47:19.380 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5oidn+gm9t0vJ89AaA33JUjrkys= 2024-02-01T10:47:19.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5oidn+gm9t0vJ89AaA33JUjrkys=', 'Content-Length': '0'} 2024-02-01T10:47:19.383 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b9efbf346e27ee96-0065bb76b7-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:19.384 DEBUG:boto:path=/vclfcn-59/obj2 2024-02-01T10:47:19.384 DEBUG:boto:auth_path=/vclfcn-59/obj2 2024-02-01T10:47:19.384 DEBUG:boto:Method: HEAD 2024-02-01T10:47:19.384 DEBUG:boto:Path: /vclfcn-59/obj2 2024-02-01T10:47:19.384 DEBUG:boto:Data: 2024-02-01T10:47:19.384 DEBUG:boto:Headers: {} 2024-02-01T10:47:19.384 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:19.384 DEBUG:boto:Port: 8000 2024-02-01T10:47:19.384 DEBUG:boto:Params: {} 2024-02-01T10:47:19.385 DEBUG:boto:Token: None 2024-02-01T10:47:19.385 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:19 GMT /vclfcn-59/obj2 2024-02-01T10:47:19.385 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5oidn+gm9t0vJ89AaA33JUjrkys= 2024-02-01T10:47:19.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5oidn+gm9t0vJ89AaA33JUjrkys=', 'Content-Length': '0'} 2024-02-01T10:47:19.388 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f7069fb34ba6d3e8-0065bb76b7-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:19.388 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:47:19.388 DEBUG:boto:path=/vclfcn-59/obj2 2024-02-01T10:47:19.388 DEBUG:boto:auth_path=/vclfcn-59/obj2 2024-02-01T10:47:19.388 DEBUG:boto:Method: GET 2024-02-01T10:47:19.388 DEBUG:boto:Path: /vclfcn-59/obj2 2024-02-01T10:47:19.388 DEBUG:boto:Data: 2024-02-01T10:47:19.389 DEBUG:boto:Headers: {} 2024-02-01T10:47:19.389 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:19.389 DEBUG:boto:Port: 8001 2024-02-01T10:47:19.389 DEBUG:boto:Params: {} 2024-02-01T10:47:19.389 DEBUG:boto:Token: None 2024-02-01T10:47:19.389 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:19 GMT /vclfcn-59/obj2 2024-02-01T10:47:19.389 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9Sha2g4yINAU+i0bnZOs96Wkors= 2024-02-01T10:47:19.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9Sha2g4yINAU+i0bnZOs96Wkors=', 'Content-Length': '0'} 2024-02-01T10:47:19.392 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004409d3f0b24ca167-0065bb76b7-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:19.436 DEBUG:boto:path=/vclfcn-59/obj2 2024-02-01T10:47:19.436 DEBUG:boto:auth_path=/vclfcn-59/obj2 2024-02-01T10:47:19.437 DEBUG:boto:Method: GET 2024-02-01T10:47:19.437 DEBUG:boto:Path: /vclfcn-59/obj2 2024-02-01T10:47:19.437 DEBUG:boto:Data: 2024-02-01T10:47:19.437 DEBUG:boto:Headers: {} 2024-02-01T10:47:19.437 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:19.437 DEBUG:boto:Port: 8000 2024-02-01T10:47:19.437 DEBUG:boto:Params: {} 2024-02-01T10:47:19.437 DEBUG:boto:Token: None 2024-02-01T10:47:19.437 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:19 GMT /vclfcn-59/obj2 2024-02-01T10:47:19.438 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9Sha2g4yINAU+i0bnZOs96Wkors= 2024-02-01T10:47:19.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9Sha2g4yINAU+i0bnZOs96Wkors=', 'Content-Length': '0'} 2024-02-01T10:47:19.441 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000083949d4115700a4c-0065bb76b7-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:19.441 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:47:19.442 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:47:19.442 DEBUG:boto:path=/vclfcn-59/?prefix=obj2 2024-02-01T10:47:19.442 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj2 2024-02-01T10:47:19.442 DEBUG:boto:Method: GET 2024-02-01T10:47:19.442 DEBUG:boto:Path: /vclfcn-59/?prefix=obj2 2024-02-01T10:47:19.442 DEBUG:boto:Data: 2024-02-01T10:47:19.442 DEBUG:boto:Headers: {} 2024-02-01T10:47:19.442 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:19.442 DEBUG:boto:Port: 8001 2024-02-01T10:47:19.442 DEBUG:boto:Params: {} 2024-02-01T10:47:19.442 DEBUG:boto:Token: None 2024-02-01T10:47:19.443 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:19 GMT /vclfcn-59/ 2024-02-01T10:47:19.443 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8lQZdEka3uCMRAGzvQOFxqE0ZoI= 2024-02-01T10:47:19.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8lQZdEka3uCMRAGzvQOFxqE0ZoI=', 'Content-Length': '0'} 2024-02-01T10:47:19.447 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000177b953607141d58-0065bb76b7-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:19.488 DEBUG:boto:b'vclfcn-59obj21000falseobj22024-02-01T10:46:46.111Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:19.489 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:47:19.489 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:47:19.489 DEBUG:boto:path=/vclfcn-59/?prefix=obj2 2024-02-01T10:47:19.489 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj2 2024-02-01T10:47:19.489 DEBUG:boto:Method: GET 2024-02-01T10:47:19.489 DEBUG:boto:Path: /vclfcn-59/?prefix=obj2 2024-02-01T10:47:19.489 DEBUG:boto:Data: 2024-02-01T10:47:19.489 DEBUG:boto:Headers: {} 2024-02-01T10:47:19.489 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:19.489 DEBUG:boto:Port: 8000 2024-02-01T10:47:19.490 DEBUG:boto:Params: {} 2024-02-01T10:47:19.490 DEBUG:boto:Token: None 2024-02-01T10:47:19.490 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:19 GMT /vclfcn-59/ 2024-02-01T10:47:19.490 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8lQZdEka3uCMRAGzvQOFxqE0ZoI= 2024-02-01T10:47:19.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8lQZdEka3uCMRAGzvQOFxqE0ZoI=', 'Content-Length': '0'} 2024-02-01T10:47:19.494 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007cbf0536cee46f19-0065bb76b7-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:19.494 DEBUG:boto:b'vclfcn-59obj21000falseobj22024-02-01T10:46:46.111Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:19.494 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:47:19.495 DEBUG:boto:path=/vclfcn-59/obj3 2024-02-01T10:47:19.495 DEBUG:boto:auth_path=/vclfcn-59/obj3 2024-02-01T10:47:19.495 DEBUG:boto:path=/vclfcn-59/obj3?versionId=null 2024-02-01T10:47:19.495 DEBUG:boto:auth_path=/vclfcn-59/obj3?versionId=null 2024-02-01T10:47:19.495 DEBUG:boto:Method: GET 2024-02-01T10:47:19.495 DEBUG:boto:Path: /vclfcn-59/obj3?versionId=null 2024-02-01T10:47:19.495 DEBUG:boto:Data: 2024-02-01T10:47:19.495 DEBUG:boto:Headers: {} 2024-02-01T10:47:19.495 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:19.495 DEBUG:boto:Port: 8001 2024-02-01T10:47:19.496 DEBUG:boto:Params: {} 2024-02-01T10:47:19.496 DEBUG:boto:Token: None 2024-02-01T10:47:19.496 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:19 GMT /vclfcn-59/obj3?versionId=null 2024-02-01T10:47:19.496 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:IhaahNw2ICMgmbZ8t/vpJPJg1kk= 2024-02-01T10:47:19.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:IhaahNw2ICMgmbZ8t/vpJPJg1kk=', 'Content-Length': '0'} 2024-02-01T10:47:19.499 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000436e240eda4b2373-0065bb76b7-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:19.540 DEBUG:boto:path=/vclfcn-59/obj3 2024-02-01T10:47:19.540 DEBUG:boto:auth_path=/vclfcn-59/obj3 2024-02-01T10:47:19.540 DEBUG:boto:path=/vclfcn-59/obj3?versionId=null 2024-02-01T10:47:19.540 DEBUG:boto:auth_path=/vclfcn-59/obj3?versionId=null 2024-02-01T10:47:19.540 DEBUG:boto:Method: GET 2024-02-01T10:47:19.541 DEBUG:boto:Path: /vclfcn-59/obj3?versionId=null 2024-02-01T10:47:19.541 DEBUG:boto:Data: 2024-02-01T10:47:19.541 DEBUG:boto:Headers: {} 2024-02-01T10:47:19.541 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:19.541 DEBUG:boto:Port: 8000 2024-02-01T10:47:19.541 DEBUG:boto:Params: {} 2024-02-01T10:47:19.541 DEBUG:boto:Token: None 2024-02-01T10:47:19.541 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:19 GMT /vclfcn-59/obj3?versionId=null 2024-02-01T10:47:19.541 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:IhaahNw2ICMgmbZ8t/vpJPJg1kk= 2024-02-01T10:47:19.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:IhaahNw2ICMgmbZ8t/vpJPJg1kk=', 'Content-Length': '0'} 2024-02-01T10:47:19.545 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ae80fd26f2d6e275-0065bb76b7-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:19.545 DEBUG:boto:path=/vclfcn-59/obj3 2024-02-01T10:47:19.545 DEBUG:boto:auth_path=/vclfcn-59/obj3 2024-02-01T10:47:19.545 DEBUG:boto:path=/vclfcn-59/obj3?versionId=null 2024-02-01T10:47:19.545 DEBUG:boto:auth_path=/vclfcn-59/obj3?versionId=null 2024-02-01T10:47:19.546 DEBUG:boto:Method: HEAD 2024-02-01T10:47:19.546 DEBUG:boto:Path: /vclfcn-59/obj3?versionId=null 2024-02-01T10:47:19.546 DEBUG:boto:Data: 2024-02-01T10:47:19.546 DEBUG:boto:Headers: {} 2024-02-01T10:47:19.546 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:19.546 DEBUG:boto:Port: 8001 2024-02-01T10:47:19.546 DEBUG:boto:Params: {} 2024-02-01T10:47:19.546 DEBUG:boto:Token: None 2024-02-01T10:47:19.546 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:19 GMT /vclfcn-59/obj3?versionId=null 2024-02-01T10:47:19.547 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:JpAOU5V3rlGlUfeKF8rjuMl/5lM= 2024-02-01T10:47:19.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:JpAOU5V3rlGlUfeKF8rjuMl/5lM=', 'Content-Length': '0'} 2024-02-01T10:47:19.550 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f61f39e89fac854c-0065bb76b7-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:19.550 DEBUG:boto:path=/vclfcn-59/obj3 2024-02-01T10:47:19.550 DEBUG:boto:auth_path=/vclfcn-59/obj3 2024-02-01T10:47:19.550 DEBUG:boto:path=/vclfcn-59/obj3?versionId=null 2024-02-01T10:47:19.550 DEBUG:boto:auth_path=/vclfcn-59/obj3?versionId=null 2024-02-01T10:47:19.551 DEBUG:boto:Method: HEAD 2024-02-01T10:47:19.551 DEBUG:boto:Path: /vclfcn-59/obj3?versionId=null 2024-02-01T10:47:19.551 DEBUG:boto:Data: 2024-02-01T10:47:19.551 DEBUG:boto:Headers: {} 2024-02-01T10:47:19.551 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:19.551 DEBUG:boto:Port: 8000 2024-02-01T10:47:19.551 DEBUG:boto:Params: {} 2024-02-01T10:47:19.551 DEBUG:boto:Token: None 2024-02-01T10:47:19.551 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:19 GMT /vclfcn-59/obj3?versionId=null 2024-02-01T10:47:19.552 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:JpAOU5V3rlGlUfeKF8rjuMl/5lM= 2024-02-01T10:47:19.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:JpAOU5V3rlGlUfeKF8rjuMl/5lM=', 'Content-Length': '0'} 2024-02-01T10:47:19.555 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d0135e8f8f0029e6-0065bb76b7-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:19.555 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:47:19.555 DEBUG:boto:path=/vclfcn-59/obj3 2024-02-01T10:47:19.555 DEBUG:boto:auth_path=/vclfcn-59/obj3 2024-02-01T10:47:19.555 DEBUG:boto:path=/vclfcn-59/obj3?versionId=null 2024-02-01T10:47:19.555 DEBUG:boto:auth_path=/vclfcn-59/obj3?versionId=null 2024-02-01T10:47:19.555 DEBUG:boto:Method: GET 2024-02-01T10:47:19.555 DEBUG:boto:Path: /vclfcn-59/obj3?versionId=null 2024-02-01T10:47:19.556 DEBUG:boto:Data: 2024-02-01T10:47:19.556 DEBUG:boto:Headers: {} 2024-02-01T10:47:19.556 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:19.556 DEBUG:boto:Port: 8001 2024-02-01T10:47:19.556 DEBUG:boto:Params: {} 2024-02-01T10:47:19.556 DEBUG:boto:Token: None 2024-02-01T10:47:19.556 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:19 GMT /vclfcn-59/obj3?versionId=null 2024-02-01T10:47:19.556 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:IhaahNw2ICMgmbZ8t/vpJPJg1kk= 2024-02-01T10:47:19.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:IhaahNw2ICMgmbZ8t/vpJPJg1kk=', 'Content-Length': '0'} 2024-02-01T10:47:19.559 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000adbed90a623ddbf1-0065bb76b7-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:19.600 DEBUG:boto:path=/vclfcn-59/obj3 2024-02-01T10:47:19.600 DEBUG:boto:auth_path=/vclfcn-59/obj3 2024-02-01T10:47:19.600 DEBUG:boto:path=/vclfcn-59/obj3?versionId=null 2024-02-01T10:47:19.601 DEBUG:boto:auth_path=/vclfcn-59/obj3?versionId=null 2024-02-01T10:47:19.601 DEBUG:boto:Method: GET 2024-02-01T10:47:19.601 DEBUG:boto:Path: /vclfcn-59/obj3?versionId=null 2024-02-01T10:47:19.601 DEBUG:boto:Data: 2024-02-01T10:47:19.601 DEBUG:boto:Headers: {} 2024-02-01T10:47:19.601 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:19.601 DEBUG:boto:Port: 8000 2024-02-01T10:47:19.602 DEBUG:boto:Params: {} 2024-02-01T10:47:19.602 DEBUG:boto:Token: None 2024-02-01T10:47:19.602 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:19 GMT /vclfcn-59/obj3?versionId=null 2024-02-01T10:47:19.603 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:IhaahNw2ICMgmbZ8t/vpJPJg1kk= 2024-02-01T10:47:19.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:IhaahNw2ICMgmbZ8t/vpJPJg1kk=', 'Content-Length': '0'} 2024-02-01T10:47:19.606 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000036b9796ebac1a284-0065bb76b7-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:19.606 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:47:19.606 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:47:19.606 DEBUG:boto:path=/vclfcn-59/?prefix=obj3 2024-02-01T10:47:19.607 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj3 2024-02-01T10:47:19.607 DEBUG:boto:Method: GET 2024-02-01T10:47:19.607 DEBUG:boto:Path: /vclfcn-59/?prefix=obj3 2024-02-01T10:47:19.607 DEBUG:boto:Data: 2024-02-01T10:47:19.607 DEBUG:boto:Headers: {} 2024-02-01T10:47:19.607 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:19.607 DEBUG:boto:Port: 8001 2024-02-01T10:47:19.607 DEBUG:boto:Params: {} 2024-02-01T10:47:19.608 DEBUG:boto:Token: None 2024-02-01T10:47:19.608 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:19 GMT /vclfcn-59/ 2024-02-01T10:47:19.608 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8lQZdEka3uCMRAGzvQOFxqE0ZoI= 2024-02-01T10:47:19.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8lQZdEka3uCMRAGzvQOFxqE0ZoI=', 'Content-Length': '0'} 2024-02-01T10:47:19.612 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000038325169fe358394-0065bb76b7-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:19.656 DEBUG:boto:b'vclfcn-59obj31000falseobj32024-02-01T10:46:46.161Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:19.657 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:47:19.657 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:47:19.657 DEBUG:boto:path=/vclfcn-59/?prefix=obj3 2024-02-01T10:47:19.657 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj3 2024-02-01T10:47:19.657 DEBUG:boto:Method: GET 2024-02-01T10:47:19.657 DEBUG:boto:Path: /vclfcn-59/?prefix=obj3 2024-02-01T10:47:19.657 DEBUG:boto:Data: 2024-02-01T10:47:19.657 DEBUG:boto:Headers: {} 2024-02-01T10:47:19.657 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:19.658 DEBUG:boto:Port: 8000 2024-02-01T10:47:19.658 DEBUG:boto:Params: {} 2024-02-01T10:47:19.658 DEBUG:boto:Token: None 2024-02-01T10:47:19.658 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:19 GMT /vclfcn-59/ 2024-02-01T10:47:19.658 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8lQZdEka3uCMRAGzvQOFxqE0ZoI= 2024-02-01T10:47:19.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8lQZdEka3uCMRAGzvQOFxqE0ZoI=', 'Content-Length': '0'} 2024-02-01T10:47:19.662 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a3eec8f293dce2f0-0065bb76b7-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:19.663 DEBUG:boto:b'vclfcn-59obj31000falseobj32024-02-01T10:46:46.161Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:19.663 DEBUG:boto:path=/vclfcn-59/obj3 2024-02-01T10:47:19.663 DEBUG:boto:auth_path=/vclfcn-59/obj3 2024-02-01T10:47:19.663 DEBUG:boto:Method: HEAD 2024-02-01T10:47:19.663 DEBUG:boto:Path: /vclfcn-59/obj3 2024-02-01T10:47:19.664 DEBUG:boto:Data: 2024-02-01T10:47:19.664 DEBUG:boto:Headers: {} 2024-02-01T10:47:19.664 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:19.664 DEBUG:boto:Port: 8001 2024-02-01T10:47:19.664 DEBUG:boto:Params: {} 2024-02-01T10:47:19.664 DEBUG:boto:Token: None 2024-02-01T10:47:19.664 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:19 GMT /vclfcn-59/obj3 2024-02-01T10:47:19.664 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:SJCvzqZoeobwrVbzZT5taGa5EgE= 2024-02-01T10:47:19.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:SJCvzqZoeobwrVbzZT5taGa5EgE=', 'Content-Length': '0'} 2024-02-01T10:47:19.667 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a3e905c11f50221b-0065bb76b7-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:19.667 DEBUG:boto:path=/vclfcn-59/obj3 2024-02-01T10:47:19.668 DEBUG:boto:auth_path=/vclfcn-59/obj3 2024-02-01T10:47:19.668 DEBUG:boto:Method: HEAD 2024-02-01T10:47:19.668 DEBUG:boto:Path: /vclfcn-59/obj3 2024-02-01T10:47:19.668 DEBUG:boto:Data: 2024-02-01T10:47:19.668 DEBUG:boto:Headers: {} 2024-02-01T10:47:19.668 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:19.668 DEBUG:boto:Port: 8000 2024-02-01T10:47:19.668 DEBUG:boto:Params: {} 2024-02-01T10:47:19.668 DEBUG:boto:Token: None 2024-02-01T10:47:19.668 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:19 GMT /vclfcn-59/obj3 2024-02-01T10:47:19.669 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:SJCvzqZoeobwrVbzZT5taGa5EgE= 2024-02-01T10:47:19.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:SJCvzqZoeobwrVbzZT5taGa5EgE=', 'Content-Length': '0'} 2024-02-01T10:47:19.671 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000f86ef51897833a2-0065bb76b7-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:19.672 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:47:19.672 DEBUG:boto:path=/vclfcn-59/obj3 2024-02-01T10:47:19.672 DEBUG:boto:auth_path=/vclfcn-59/obj3 2024-02-01T10:47:19.672 DEBUG:boto:Method: GET 2024-02-01T10:47:19.672 DEBUG:boto:Path: /vclfcn-59/obj3 2024-02-01T10:47:19.672 DEBUG:boto:Data: 2024-02-01T10:47:19.672 DEBUG:boto:Headers: {} 2024-02-01T10:47:19.672 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:19.673 DEBUG:boto:Port: 8001 2024-02-01T10:47:19.673 DEBUG:boto:Params: {} 2024-02-01T10:47:19.673 DEBUG:boto:Token: None 2024-02-01T10:47:19.673 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:19 GMT /vclfcn-59/obj3 2024-02-01T10:47:19.673 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:i+wwYjeWeGQYqyPRE7MsHzehCdo= 2024-02-01T10:47:19.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:i+wwYjeWeGQYqyPRE7MsHzehCdo=', 'Content-Length': '0'} 2024-02-01T10:47:19.676 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009c9f5973d0d8a3ba-0065bb76b7-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:19.716 DEBUG:boto:path=/vclfcn-59/obj3 2024-02-01T10:47:19.716 DEBUG:boto:auth_path=/vclfcn-59/obj3 2024-02-01T10:47:19.717 DEBUG:boto:Method: GET 2024-02-01T10:47:19.717 DEBUG:boto:Path: /vclfcn-59/obj3 2024-02-01T10:47:19.717 DEBUG:boto:Data: 2024-02-01T10:47:19.717 DEBUG:boto:Headers: {} 2024-02-01T10:47:19.717 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:19.717 DEBUG:boto:Port: 8000 2024-02-01T10:47:19.717 DEBUG:boto:Params: {} 2024-02-01T10:47:19.717 DEBUG:boto:Token: None 2024-02-01T10:47:19.717 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:19 GMT /vclfcn-59/obj3 2024-02-01T10:47:19.718 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:i+wwYjeWeGQYqyPRE7MsHzehCdo= 2024-02-01T10:47:19.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:i+wwYjeWeGQYqyPRE7MsHzehCdo=', 'Content-Length': '0'} 2024-02-01T10:47:19.721 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000079d268ba626b7a9c-0065bb76b7-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:19.721 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:47:19.721 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:47:19.721 DEBUG:boto:path=/vclfcn-59/?prefix=obj3 2024-02-01T10:47:19.721 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj3 2024-02-01T10:47:19.722 DEBUG:boto:Method: GET 2024-02-01T10:47:19.722 DEBUG:boto:Path: /vclfcn-59/?prefix=obj3 2024-02-01T10:47:19.722 DEBUG:boto:Data: 2024-02-01T10:47:19.722 DEBUG:boto:Headers: {} 2024-02-01T10:47:19.722 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:19.722 DEBUG:boto:Port: 8001 2024-02-01T10:47:19.722 DEBUG:boto:Params: {} 2024-02-01T10:47:19.722 DEBUG:boto:Token: None 2024-02-01T10:47:19.722 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:19 GMT /vclfcn-59/ 2024-02-01T10:47:19.723 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8lQZdEka3uCMRAGzvQOFxqE0ZoI= 2024-02-01T10:47:19.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8lQZdEka3uCMRAGzvQOFxqE0ZoI=', 'Content-Length': '0'} 2024-02-01T10:47:19.727 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000566dff4dc57215cc-0065bb76b7-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:19.727 DEBUG:boto:b'vclfcn-59obj31000falseobj32024-02-01T10:46:46.161Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:19.727 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:47:19.728 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:47:19.728 DEBUG:boto:path=/vclfcn-59/?prefix=obj3 2024-02-01T10:47:19.728 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj3 2024-02-01T10:47:19.728 DEBUG:boto:Method: GET 2024-02-01T10:47:19.728 DEBUG:boto:Path: /vclfcn-59/?prefix=obj3 2024-02-01T10:47:19.728 DEBUG:boto:Data: 2024-02-01T10:47:19.728 DEBUG:boto:Headers: {} 2024-02-01T10:47:19.728 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:19.728 DEBUG:boto:Port: 8000 2024-02-01T10:47:19.728 DEBUG:boto:Params: {} 2024-02-01T10:47:19.729 DEBUG:boto:Token: None 2024-02-01T10:47:19.729 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:19 GMT /vclfcn-59/ 2024-02-01T10:47:19.729 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8lQZdEka3uCMRAGzvQOFxqE0ZoI= 2024-02-01T10:47:19.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8lQZdEka3uCMRAGzvQOFxqE0ZoI=', 'Content-Length': '0'} 2024-02-01T10:47:19.735 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000304bd6c8c4d5a5f0-0065bb76b7-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:19.735 DEBUG:boto:b'vclfcn-59obj31000falseobj32024-02-01T10:46:46.161Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:19.735 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:47:19.735 DEBUG:boto:path=/vclfcn-59/obj4 2024-02-01T10:47:19.736 DEBUG:boto:auth_path=/vclfcn-59/obj4 2024-02-01T10:47:19.736 DEBUG:boto:path=/vclfcn-59/obj4?versionId=null 2024-02-01T10:47:19.736 DEBUG:boto:auth_path=/vclfcn-59/obj4?versionId=null 2024-02-01T10:47:19.736 DEBUG:boto:Method: GET 2024-02-01T10:47:19.736 DEBUG:boto:Path: /vclfcn-59/obj4?versionId=null 2024-02-01T10:47:19.736 DEBUG:boto:Data: 2024-02-01T10:47:19.736 DEBUG:boto:Headers: {} 2024-02-01T10:47:19.736 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:19.736 DEBUG:boto:Port: 8001 2024-02-01T10:47:19.736 DEBUG:boto:Params: {} 2024-02-01T10:47:19.737 DEBUG:boto:Token: None 2024-02-01T10:47:19.737 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:19 GMT /vclfcn-59/obj4?versionId=null 2024-02-01T10:47:19.737 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:VBIOdbXB48yHt7QFXOn3Bj/LWkU= 2024-02-01T10:47:19.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:VBIOdbXB48yHt7QFXOn3Bj/LWkU=', 'Content-Length': '0'} 2024-02-01T10:47:19.740 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000854fa1c4c8cea495-0065bb76b7-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:19.780 DEBUG:boto:path=/vclfcn-59/obj4 2024-02-01T10:47:19.780 DEBUG:boto:auth_path=/vclfcn-59/obj4 2024-02-01T10:47:19.780 DEBUG:boto:path=/vclfcn-59/obj4?versionId=null 2024-02-01T10:47:19.780 DEBUG:boto:auth_path=/vclfcn-59/obj4?versionId=null 2024-02-01T10:47:19.781 DEBUG:boto:Method: GET 2024-02-01T10:47:19.781 DEBUG:boto:Path: /vclfcn-59/obj4?versionId=null 2024-02-01T10:47:19.781 DEBUG:boto:Data: 2024-02-01T10:47:19.781 DEBUG:boto:Headers: {} 2024-02-01T10:47:19.781 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:19.781 DEBUG:boto:Port: 8000 2024-02-01T10:47:19.781 DEBUG:boto:Params: {} 2024-02-01T10:47:19.782 DEBUG:boto:Token: None 2024-02-01T10:47:19.782 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:19 GMT /vclfcn-59/obj4?versionId=null 2024-02-01T10:47:19.782 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:VBIOdbXB48yHt7QFXOn3Bj/LWkU= 2024-02-01T10:47:19.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:VBIOdbXB48yHt7QFXOn3Bj/LWkU=', 'Content-Length': '0'} 2024-02-01T10:47:19.785 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004efec36833a40baf-0065bb76b7-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:19.786 DEBUG:boto:path=/vclfcn-59/obj4 2024-02-01T10:47:19.786 DEBUG:boto:auth_path=/vclfcn-59/obj4 2024-02-01T10:47:19.786 DEBUG:boto:path=/vclfcn-59/obj4?versionId=null 2024-02-01T10:47:19.786 DEBUG:boto:auth_path=/vclfcn-59/obj4?versionId=null 2024-02-01T10:47:19.786 DEBUG:boto:Method: HEAD 2024-02-01T10:47:19.786 DEBUG:boto:Path: /vclfcn-59/obj4?versionId=null 2024-02-01T10:47:19.786 DEBUG:boto:Data: 2024-02-01T10:47:19.786 DEBUG:boto:Headers: {} 2024-02-01T10:47:19.786 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:19.786 DEBUG:boto:Port: 8001 2024-02-01T10:47:19.787 DEBUG:boto:Params: {} 2024-02-01T10:47:19.787 DEBUG:boto:Token: None 2024-02-01T10:47:19.787 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:19 GMT /vclfcn-59/obj4?versionId=null 2024-02-01T10:47:19.787 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gJLXWEubhdgeW649DNEL/vd2wgY= 2024-02-01T10:47:19.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gJLXWEubhdgeW649DNEL/vd2wgY=', 'Content-Length': '0'} 2024-02-01T10:47:19.790 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bb1c62c41cc1e76b-0065bb76b7-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:19.790 DEBUG:boto:path=/vclfcn-59/obj4 2024-02-01T10:47:19.790 DEBUG:boto:auth_path=/vclfcn-59/obj4 2024-02-01T10:47:19.790 DEBUG:boto:path=/vclfcn-59/obj4?versionId=null 2024-02-01T10:47:19.790 DEBUG:boto:auth_path=/vclfcn-59/obj4?versionId=null 2024-02-01T10:47:19.790 DEBUG:boto:Method: HEAD 2024-02-01T10:47:19.790 DEBUG:boto:Path: /vclfcn-59/obj4?versionId=null 2024-02-01T10:47:19.791 DEBUG:boto:Data: 2024-02-01T10:47:19.791 DEBUG:boto:Headers: {} 2024-02-01T10:47:19.791 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:19.791 DEBUG:boto:Port: 8000 2024-02-01T10:47:19.791 DEBUG:boto:Params: {} 2024-02-01T10:47:19.791 DEBUG:boto:Token: None 2024-02-01T10:47:19.791 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:19 GMT /vclfcn-59/obj4?versionId=null 2024-02-01T10:47:19.791 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gJLXWEubhdgeW649DNEL/vd2wgY= 2024-02-01T10:47:19.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gJLXWEubhdgeW649DNEL/vd2wgY=', 'Content-Length': '0'} 2024-02-01T10:47:19.794 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f3a314696099af0a-0065bb76b7-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:19.794 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:47:19.794 DEBUG:boto:path=/vclfcn-59/obj4 2024-02-01T10:47:19.794 DEBUG:boto:auth_path=/vclfcn-59/obj4 2024-02-01T10:47:19.794 DEBUG:boto:path=/vclfcn-59/obj4?versionId=null 2024-02-01T10:47:19.795 DEBUG:boto:auth_path=/vclfcn-59/obj4?versionId=null 2024-02-01T10:47:19.795 DEBUG:boto:Method: GET 2024-02-01T10:47:19.795 DEBUG:boto:Path: /vclfcn-59/obj4?versionId=null 2024-02-01T10:47:19.795 DEBUG:boto:Data: 2024-02-01T10:47:19.795 DEBUG:boto:Headers: {} 2024-02-01T10:47:19.795 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:19.795 DEBUG:boto:Port: 8001 2024-02-01T10:47:19.795 DEBUG:boto:Params: {} 2024-02-01T10:47:19.795 DEBUG:boto:Token: None 2024-02-01T10:47:19.795 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:19 GMT /vclfcn-59/obj4?versionId=null 2024-02-01T10:47:19.796 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:VBIOdbXB48yHt7QFXOn3Bj/LWkU= 2024-02-01T10:47:19.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:VBIOdbXB48yHt7QFXOn3Bj/LWkU=', 'Content-Length': '0'} 2024-02-01T10:47:19.798 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cabfdc9e495e0020-0065bb76b7-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:19.840 DEBUG:boto:path=/vclfcn-59/obj4 2024-02-01T10:47:19.840 DEBUG:boto:auth_path=/vclfcn-59/obj4 2024-02-01T10:47:19.840 DEBUG:boto:path=/vclfcn-59/obj4?versionId=null 2024-02-01T10:47:19.841 DEBUG:boto:auth_path=/vclfcn-59/obj4?versionId=null 2024-02-01T10:47:19.841 DEBUG:boto:Method: GET 2024-02-01T10:47:19.841 DEBUG:boto:Path: /vclfcn-59/obj4?versionId=null 2024-02-01T10:47:19.841 DEBUG:boto:Data: 2024-02-01T10:47:19.841 DEBUG:boto:Headers: {} 2024-02-01T10:47:19.841 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:19.841 DEBUG:boto:Port: 8000 2024-02-01T10:47:19.841 DEBUG:boto:Params: {} 2024-02-01T10:47:19.841 DEBUG:boto:Token: None 2024-02-01T10:47:19.841 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:19 GMT /vclfcn-59/obj4?versionId=null 2024-02-01T10:47:19.842 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:VBIOdbXB48yHt7QFXOn3Bj/LWkU= 2024-02-01T10:47:19.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:VBIOdbXB48yHt7QFXOn3Bj/LWkU=', 'Content-Length': '0'} 2024-02-01T10:47:19.845 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003ef6f06d906a8dae-0065bb76b7-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:19.845 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:47:19.845 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:47:19.845 DEBUG:boto:path=/vclfcn-59/?prefix=obj4 2024-02-01T10:47:19.845 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj4 2024-02-01T10:47:19.845 DEBUG:boto:Method: GET 2024-02-01T10:47:19.845 DEBUG:boto:Path: /vclfcn-59/?prefix=obj4 2024-02-01T10:47:19.846 DEBUG:boto:Data: 2024-02-01T10:47:19.846 DEBUG:boto:Headers: {} 2024-02-01T10:47:19.846 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:19.846 DEBUG:boto:Port: 8001 2024-02-01T10:47:19.846 DEBUG:boto:Params: {} 2024-02-01T10:47:19.846 DEBUG:boto:Token: None 2024-02-01T10:47:19.846 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:19 GMT /vclfcn-59/ 2024-02-01T10:47:19.846 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8lQZdEka3uCMRAGzvQOFxqE0ZoI= 2024-02-01T10:47:19.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8lQZdEka3uCMRAGzvQOFxqE0ZoI=', 'Content-Length': '0'} 2024-02-01T10:47:19.850 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000634c3cffb7420e73-0065bb76b7-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:19.892 DEBUG:boto:b'vclfcn-59obj41000falseobj42024-02-01T10:46:46.190Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:19.893 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:47:19.893 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:47:19.893 DEBUG:boto:path=/vclfcn-59/?prefix=obj4 2024-02-01T10:47:19.893 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj4 2024-02-01T10:47:19.893 DEBUG:boto:Method: GET 2024-02-01T10:47:19.893 DEBUG:boto:Path: /vclfcn-59/?prefix=obj4 2024-02-01T10:47:19.894 DEBUG:boto:Data: 2024-02-01T10:47:19.894 DEBUG:boto:Headers: {} 2024-02-01T10:47:19.894 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:19.894 DEBUG:boto:Port: 8000 2024-02-01T10:47:19.894 DEBUG:boto:Params: {} 2024-02-01T10:47:19.894 DEBUG:boto:Token: None 2024-02-01T10:47:19.894 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:19 GMT /vclfcn-59/ 2024-02-01T10:47:19.894 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8lQZdEka3uCMRAGzvQOFxqE0ZoI= 2024-02-01T10:47:19.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8lQZdEka3uCMRAGzvQOFxqE0ZoI=', 'Content-Length': '0'} 2024-02-01T10:47:19.899 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000021b157577cf9846-0065bb76b7-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:19.899 DEBUG:boto:b'vclfcn-59obj41000falseobj42024-02-01T10:46:46.190Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:19.900 DEBUG:boto:path=/vclfcn-59/obj4 2024-02-01T10:47:19.900 DEBUG:boto:auth_path=/vclfcn-59/obj4 2024-02-01T10:47:19.900 DEBUG:boto:Method: HEAD 2024-02-01T10:47:19.900 DEBUG:boto:Path: /vclfcn-59/obj4 2024-02-01T10:47:19.900 DEBUG:boto:Data: 2024-02-01T10:47:19.900 DEBUG:boto:Headers: {} 2024-02-01T10:47:19.900 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:19.900 DEBUG:boto:Port: 8001 2024-02-01T10:47:19.900 DEBUG:boto:Params: {} 2024-02-01T10:47:19.901 DEBUG:boto:Token: None 2024-02-01T10:47:19.901 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:19 GMT /vclfcn-59/obj4 2024-02-01T10:47:19.901 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WYHiaV/Hg26EfYwAILCBsaNeCVQ= 2024-02-01T10:47:19.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WYHiaV/Hg26EfYwAILCBsaNeCVQ=', 'Content-Length': '0'} 2024-02-01T10:47:19.904 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000834fd31fce21085-0065bb76b7-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:19.904 DEBUG:boto:path=/vclfcn-59/obj4 2024-02-01T10:47:19.904 DEBUG:boto:auth_path=/vclfcn-59/obj4 2024-02-01T10:47:19.904 DEBUG:boto:Method: HEAD 2024-02-01T10:47:19.905 DEBUG:boto:Path: /vclfcn-59/obj4 2024-02-01T10:47:19.905 DEBUG:boto:Data: 2024-02-01T10:47:19.905 DEBUG:boto:Headers: {} 2024-02-01T10:47:19.905 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:19.905 DEBUG:boto:Port: 8000 2024-02-01T10:47:19.905 DEBUG:boto:Params: {} 2024-02-01T10:47:19.905 DEBUG:boto:Token: None 2024-02-01T10:47:19.905 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:19 GMT /vclfcn-59/obj4 2024-02-01T10:47:19.905 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WYHiaV/Hg26EfYwAILCBsaNeCVQ= 2024-02-01T10:47:19.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WYHiaV/Hg26EfYwAILCBsaNeCVQ=', 'Content-Length': '0'} 2024-02-01T10:47:19.908 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000051170c13bbd9138e-0065bb76b7-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:19.908 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:47:19.908 DEBUG:boto:path=/vclfcn-59/obj4 2024-02-01T10:47:19.908 DEBUG:boto:auth_path=/vclfcn-59/obj4 2024-02-01T10:47:19.909 DEBUG:boto:Method: GET 2024-02-01T10:47:19.909 DEBUG:boto:Path: /vclfcn-59/obj4 2024-02-01T10:47:19.909 DEBUG:boto:Data: 2024-02-01T10:47:19.909 DEBUG:boto:Headers: {} 2024-02-01T10:47:19.909 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:19.909 DEBUG:boto:Port: 8001 2024-02-01T10:47:19.909 DEBUG:boto:Params: {} 2024-02-01T10:47:19.909 DEBUG:boto:Token: None 2024-02-01T10:47:19.909 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:19 GMT /vclfcn-59/obj4 2024-02-01T10:47:19.910 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QFazy8Sq30+xwoCTS8avp4aFSoE= 2024-02-01T10:47:19.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QFazy8Sq30+xwoCTS8avp4aFSoE=', 'Content-Length': '0'} 2024-02-01T10:47:19.912 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000fdec3bd148e9d8f-0065bb76b7-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:19.956 DEBUG:boto:path=/vclfcn-59/obj4 2024-02-01T10:47:19.956 DEBUG:boto:auth_path=/vclfcn-59/obj4 2024-02-01T10:47:19.956 DEBUG:boto:Method: GET 2024-02-01T10:47:19.957 DEBUG:boto:Path: /vclfcn-59/obj4 2024-02-01T10:47:19.957 DEBUG:boto:Data: 2024-02-01T10:47:19.957 DEBUG:boto:Headers: {} 2024-02-01T10:47:19.957 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:19.957 DEBUG:boto:Port: 8000 2024-02-01T10:47:19.957 DEBUG:boto:Params: {} 2024-02-01T10:47:19.957 DEBUG:boto:Token: None 2024-02-01T10:47:19.957 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:19 GMT /vclfcn-59/obj4 2024-02-01T10:47:19.957 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QFazy8Sq30+xwoCTS8avp4aFSoE= 2024-02-01T10:47:19.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QFazy8Sq30+xwoCTS8avp4aFSoE=', 'Content-Length': '0'} 2024-02-01T10:47:19.960 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002802d92b85f4b569-0065bb76b7-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:19.961 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:47:19.961 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:47:19.961 DEBUG:boto:path=/vclfcn-59/?prefix=obj4 2024-02-01T10:47:19.961 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj4 2024-02-01T10:47:19.961 DEBUG:boto:Method: GET 2024-02-01T10:47:19.961 DEBUG:boto:Path: /vclfcn-59/?prefix=obj4 2024-02-01T10:47:19.961 DEBUG:boto:Data: 2024-02-01T10:47:19.961 DEBUG:boto:Headers: {} 2024-02-01T10:47:19.961 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:19.962 DEBUG:boto:Port: 8001 2024-02-01T10:47:19.962 DEBUG:boto:Params: {} 2024-02-01T10:47:19.962 DEBUG:boto:Token: None 2024-02-01T10:47:19.962 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:19 GMT /vclfcn-59/ 2024-02-01T10:47:19.962 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8lQZdEka3uCMRAGzvQOFxqE0ZoI= 2024-02-01T10:47:19.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8lQZdEka3uCMRAGzvQOFxqE0ZoI=', 'Content-Length': '0'} 2024-02-01T10:47:19.966 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b2b6f78749b3ca0-0065bb76b7-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:19.966 DEBUG:boto:b'vclfcn-59obj41000falseobj42024-02-01T10:46:46.190Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:19.967 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:47:19.967 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:47:19.967 DEBUG:boto:path=/vclfcn-59/?prefix=obj4 2024-02-01T10:47:19.967 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj4 2024-02-01T10:47:19.967 DEBUG:boto:Method: GET 2024-02-01T10:47:19.967 DEBUG:boto:Path: /vclfcn-59/?prefix=obj4 2024-02-01T10:47:19.968 DEBUG:boto:Data: 2024-02-01T10:47:19.968 DEBUG:boto:Headers: {} 2024-02-01T10:47:19.968 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:19.968 DEBUG:boto:Port: 8000 2024-02-01T10:47:19.968 DEBUG:boto:Params: {} 2024-02-01T10:47:19.968 DEBUG:boto:Token: None 2024-02-01T10:47:19.968 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:19 GMT /vclfcn-59/ 2024-02-01T10:47:19.968 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8lQZdEka3uCMRAGzvQOFxqE0ZoI= 2024-02-01T10:47:19.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8lQZdEka3uCMRAGzvQOFxqE0ZoI=', 'Content-Length': '0'} 2024-02-01T10:47:19.973 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7f39cfd35c39198-0065bb76b7-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:19.973 DEBUG:boto:b'vclfcn-59obj41000falseobj42024-02-01T10:46:46.190Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:19.973 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-59 zones={test-zone3, test-zone1} 2024-02-01T10:47:19.973 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-59 zones={test-zone3, test-zone2} 2024-02-01T10:47:19.974 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:47:19.974 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:47:19.974 DEBUG:boto:Method: HEAD 2024-02-01T10:47:19.974 DEBUG:boto:Path: /vclfcn-59/ 2024-02-01T10:47:19.974 DEBUG:boto:Data: 2024-02-01T10:47:19.974 DEBUG:boto:Headers: {} 2024-02-01T10:47:19.974 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:19.974 DEBUG:boto:Port: 8001 2024-02-01T10:47:19.974 DEBUG:boto:Params: {} 2024-02-01T10:47:19.974 DEBUG:boto:Token: None 2024-02-01T10:47:19.975 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:19 GMT /vclfcn-59/ 2024-02-01T10:47:19.975 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Tjh8aRM4YQkbPqIdosWZzYIM6sI= 2024-02-01T10:47:19.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Tjh8aRM4YQkbPqIdosWZzYIM6sI=', 'Content-Length': '0'} 2024-02-01T10:47:19.979 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009f09918b5e40e37f-0065bb76b7-5762-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:19.979 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:47:19.979 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:47:19.979 DEBUG:boto:Method: HEAD 2024-02-01T10:47:19.979 DEBUG:boto:Path: /vclfcn-59/ 2024-02-01T10:47:19.980 DEBUG:boto:Data: 2024-02-01T10:47:19.980 DEBUG:boto:Headers: {} 2024-02-01T10:47:19.980 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:19.980 DEBUG:boto:Port: 8000 2024-02-01T10:47:19.980 DEBUG:boto:Params: {} 2024-02-01T10:47:19.980 DEBUG:boto:Token: None 2024-02-01T10:47:19.980 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:19 GMT /vclfcn-59/ 2024-02-01T10:47:19.980 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Tjh8aRM4YQkbPqIdosWZzYIM6sI= 2024-02-01T10:47:19.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Tjh8aRM4YQkbPqIdosWZzYIM6sI=', 'Content-Length': '0'} 2024-02-01T10:47:19.985 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008adce88f2edec050-0065bb76b7-5350-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:19.985 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:47:19.986 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:47:19.986 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:47:19.986 DEBUG:boto:path=/vclfcn-59/?versions&max-keys=999 2024-02-01T10:47:19.986 DEBUG:boto:auth_path=/vclfcn-59/?versions&max-keys=999 2024-02-01T10:47:19.986 DEBUG:boto:Method: GET 2024-02-01T10:47:19.986 DEBUG:boto:Path: /vclfcn-59/?versions&max-keys=999 2024-02-01T10:47:19.986 DEBUG:boto:Data: 2024-02-01T10:47:19.986 DEBUG:boto:Headers: {} 2024-02-01T10:47:19.986 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:19.986 DEBUG:boto:Port: 8001 2024-02-01T10:47:19.986 DEBUG:boto:Params: {} 2024-02-01T10:47:19.987 DEBUG:boto:Token: None 2024-02-01T10:47:19.987 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:19 GMT /vclfcn-59/?versions 2024-02-01T10:47:19.987 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:JSACQpdoYd9Us978/uN+UdPyBos= 2024-02-01T10:47:19.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:JSACQpdoYd9Us978/uN+UdPyBos=', 'Content-Length': '0'} 2024-02-01T10:47:19.991 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000254a1165ae50a162-0065bb76b7-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:19.992 DEBUG:boto:b'vclfcn-59999falseobj1nulltrue2024-02-01T10:46:46.093Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:46:46.111Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:46:46.161Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:46:46.190Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:19.992 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-02-01T10:47:19.992 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-02-01T10:47:19.993 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-02-01T10:47:19.993 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-02-01T10:47:19.993 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:47:19.993 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:47:19.993 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:47:19.993 DEBUG:boto:path=/vclfcn-59/?versions&max-keys=999 2024-02-01T10:47:19.993 DEBUG:boto:auth_path=/vclfcn-59/?versions&max-keys=999 2024-02-01T10:47:19.993 DEBUG:boto:Method: GET 2024-02-01T10:47:19.993 DEBUG:boto:Path: /vclfcn-59/?versions&max-keys=999 2024-02-01T10:47:19.993 DEBUG:boto:Data: 2024-02-01T10:47:19.993 DEBUG:boto:Headers: {} 2024-02-01T10:47:19.994 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:19.994 DEBUG:boto:Port: 8000 2024-02-01T10:47:19.994 DEBUG:boto:Params: {} 2024-02-01T10:47:19.994 DEBUG:boto:Token: None 2024-02-01T10:47:19.994 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:19 GMT /vclfcn-59/?versions 2024-02-01T10:47:19.994 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:JSACQpdoYd9Us978/uN+UdPyBos= 2024-02-01T10:47:19.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:JSACQpdoYd9Us978/uN+UdPyBos=', 'Content-Length': '0'} 2024-02-01T10:47:19.999 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f4e55a3427787c5d-0065bb76b7-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:19.999 DEBUG:boto:b'vclfcn-59999falseobj1nulltrue2024-02-01T10:46:46.093Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:46:46.111Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:46:46.161Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:46:46.190Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:19.999 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-02-01T10:47:19.999 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-02-01T10:47:19.999 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-02-01T10:47:20.000 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-02-01T10:47:20.000 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:47:20.000 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:47:20.000 DEBUG:boto:path=/vclfcn-59/?versions&max-keys=999 2024-02-01T10:47:20.000 DEBUG:boto:auth_path=/vclfcn-59/?versions&max-keys=999 2024-02-01T10:47:20.000 DEBUG:boto:Method: GET 2024-02-01T10:47:20.000 DEBUG:boto:Path: /vclfcn-59/?versions&max-keys=999 2024-02-01T10:47:20.000 DEBUG:boto:Data: 2024-02-01T10:47:20.000 DEBUG:boto:Headers: {} 2024-02-01T10:47:20.000 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:20.001 DEBUG:boto:Port: 8001 2024-02-01T10:47:20.001 DEBUG:boto:Params: {} 2024-02-01T10:47:20.001 DEBUG:boto:Token: None 2024-02-01T10:47:20.001 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:20 GMT /vclfcn-59/?versions 2024-02-01T10:47:20.001 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:TzGtwYyfV/NWOC8ZGrc3nC/T0h4= 2024-02-01T10:47:20.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:TzGtwYyfV/NWOC8ZGrc3nC/T0h4=', 'Content-Length': '0'} 2024-02-01T10:47:20.005 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ec724052d004fd8-0065bb76b8-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:20.048 DEBUG:boto:b'vclfcn-59999falseobj1nulltrue2024-02-01T10:46:46.093Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:46:46.111Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:46:46.161Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:46:46.190Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:20.049 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:47:20.049 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:47:20.049 DEBUG:boto:path=/vclfcn-59/?versions&max-keys=999 2024-02-01T10:47:20.049 DEBUG:boto:auth_path=/vclfcn-59/?versions&max-keys=999 2024-02-01T10:47:20.049 DEBUG:boto:Method: GET 2024-02-01T10:47:20.049 DEBUG:boto:Path: /vclfcn-59/?versions&max-keys=999 2024-02-01T10:47:20.049 DEBUG:boto:Data: 2024-02-01T10:47:20.049 DEBUG:boto:Headers: {} 2024-02-01T10:47:20.050 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:20.050 DEBUG:boto:Port: 8000 2024-02-01T10:47:20.050 DEBUG:boto:Params: {} 2024-02-01T10:47:20.050 DEBUG:boto:Token: None 2024-02-01T10:47:20.050 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:20 GMT /vclfcn-59/?versions 2024-02-01T10:47:20.050 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:TzGtwYyfV/NWOC8ZGrc3nC/T0h4= 2024-02-01T10:47:20.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:TzGtwYyfV/NWOC8ZGrc3nC/T0h4=', 'Content-Length': '0'} 2024-02-01T10:47:20.055 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010816288fad2e92e-0065bb76b8-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:20.055 DEBUG:boto:b'vclfcn-59999falseobj1nulltrue2024-02-01T10:46:46.093Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:46:46.111Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:46:46.161Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:46:46.190Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:20.055 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:47:20.055 DEBUG:boto:path=/vclfcn-59/obj1 2024-02-01T10:47:20.056 DEBUG:boto:auth_path=/vclfcn-59/obj1 2024-02-01T10:47:20.056 DEBUG:boto:path=/vclfcn-59/obj1?versionId=null 2024-02-01T10:47:20.056 DEBUG:boto:auth_path=/vclfcn-59/obj1?versionId=null 2024-02-01T10:47:20.056 DEBUG:boto:Method: GET 2024-02-01T10:47:20.056 DEBUG:boto:Path: /vclfcn-59/obj1?versionId=null 2024-02-01T10:47:20.056 DEBUG:boto:Data: 2024-02-01T10:47:20.056 DEBUG:boto:Headers: {} 2024-02-01T10:47:20.056 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:20.056 DEBUG:boto:Port: 8001 2024-02-01T10:47:20.056 DEBUG:boto:Params: {} 2024-02-01T10:47:20.057 DEBUG:boto:Token: None 2024-02-01T10:47:20.057 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:20 GMT /vclfcn-59/obj1?versionId=null 2024-02-01T10:47:20.057 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sghatyAePyXF3t60Ir4Z/bT8N0E= 2024-02-01T10:47:20.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sghatyAePyXF3t60Ir4Z/bT8N0E=', 'Content-Length': '0'} 2024-02-01T10:47:20.060 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000170c52b4f7e2e09c-0065bb76b8-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:20.060 DEBUG:boto:path=/vclfcn-59/obj1 2024-02-01T10:47:20.060 DEBUG:boto:auth_path=/vclfcn-59/obj1 2024-02-01T10:47:20.060 DEBUG:boto:path=/vclfcn-59/obj1?versionId=null 2024-02-01T10:47:20.060 DEBUG:boto:auth_path=/vclfcn-59/obj1?versionId=null 2024-02-01T10:47:20.060 DEBUG:boto:Method: GET 2024-02-01T10:47:20.061 DEBUG:boto:Path: /vclfcn-59/obj1?versionId=null 2024-02-01T10:47:20.061 DEBUG:boto:Data: 2024-02-01T10:47:20.061 DEBUG:boto:Headers: {} 2024-02-01T10:47:20.061 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:20.061 DEBUG:boto:Port: 8000 2024-02-01T10:47:20.061 DEBUG:boto:Params: {} 2024-02-01T10:47:20.061 DEBUG:boto:Token: None 2024-02-01T10:47:20.061 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:20 GMT /vclfcn-59/obj1?versionId=null 2024-02-01T10:47:20.061 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sghatyAePyXF3t60Ir4Z/bT8N0E= 2024-02-01T10:47:20.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sghatyAePyXF3t60Ir4Z/bT8N0E=', 'Content-Length': '0'} 2024-02-01T10:47:20.064 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e6b884399d0363ea-0065bb76b8-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:20.065 DEBUG:boto:path=/vclfcn-59/obj1 2024-02-01T10:47:20.065 DEBUG:boto:auth_path=/vclfcn-59/obj1 2024-02-01T10:47:20.065 DEBUG:boto:path=/vclfcn-59/obj1?versionId=null 2024-02-01T10:47:20.065 DEBUG:boto:auth_path=/vclfcn-59/obj1?versionId=null 2024-02-01T10:47:20.065 DEBUG:boto:Method: HEAD 2024-02-01T10:47:20.065 DEBUG:boto:Path: /vclfcn-59/obj1?versionId=null 2024-02-01T10:47:20.065 DEBUG:boto:Data: 2024-02-01T10:47:20.065 DEBUG:boto:Headers: {} 2024-02-01T10:47:20.065 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:20.065 DEBUG:boto:Port: 8001 2024-02-01T10:47:20.065 DEBUG:boto:Params: {} 2024-02-01T10:47:20.066 DEBUG:boto:Token: None 2024-02-01T10:47:20.066 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:20 GMT /vclfcn-59/obj1?versionId=null 2024-02-01T10:47:20.066 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:kZYfZLjI3r2Nj/eo93NH6yuXX28= 2024-02-01T10:47:20.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:kZYfZLjI3r2Nj/eo93NH6yuXX28=', 'Content-Length': '0'} 2024-02-01T10:47:20.069 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006e3def430f582bb0-0065bb76b8-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:20.069 DEBUG:boto:path=/vclfcn-59/obj1 2024-02-01T10:47:20.070 DEBUG:boto:auth_path=/vclfcn-59/obj1 2024-02-01T10:47:20.070 DEBUG:boto:path=/vclfcn-59/obj1?versionId=null 2024-02-01T10:47:20.070 DEBUG:boto:auth_path=/vclfcn-59/obj1?versionId=null 2024-02-01T10:47:20.070 DEBUG:boto:Method: HEAD 2024-02-01T10:47:20.070 DEBUG:boto:Path: /vclfcn-59/obj1?versionId=null 2024-02-01T10:47:20.070 DEBUG:boto:Data: 2024-02-01T10:47:20.070 DEBUG:boto:Headers: {} 2024-02-01T10:47:20.070 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:20.070 DEBUG:boto:Port: 8000 2024-02-01T10:47:20.070 DEBUG:boto:Params: {} 2024-02-01T10:47:20.070 DEBUG:boto:Token: None 2024-02-01T10:47:20.071 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:20 GMT /vclfcn-59/obj1?versionId=null 2024-02-01T10:47:20.071 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:kZYfZLjI3r2Nj/eo93NH6yuXX28= 2024-02-01T10:47:20.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:kZYfZLjI3r2Nj/eo93NH6yuXX28=', 'Content-Length': '0'} 2024-02-01T10:47:20.073 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aa570dad492c36eb-0065bb76b8-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:20.074 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:47:20.074 DEBUG:boto:path=/vclfcn-59/obj1 2024-02-01T10:47:20.074 DEBUG:boto:auth_path=/vclfcn-59/obj1 2024-02-01T10:47:20.074 DEBUG:boto:path=/vclfcn-59/obj1?versionId=null 2024-02-01T10:47:20.074 DEBUG:boto:auth_path=/vclfcn-59/obj1?versionId=null 2024-02-01T10:47:20.074 DEBUG:boto:Method: GET 2024-02-01T10:47:20.074 DEBUG:boto:Path: /vclfcn-59/obj1?versionId=null 2024-02-01T10:47:20.075 DEBUG:boto:Data: 2024-02-01T10:47:20.075 DEBUG:boto:Headers: {} 2024-02-01T10:47:20.075 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:20.075 DEBUG:boto:Port: 8001 2024-02-01T10:47:20.075 DEBUG:boto:Params: {} 2024-02-01T10:47:20.075 DEBUG:boto:Token: None 2024-02-01T10:47:20.075 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:20 GMT /vclfcn-59/obj1?versionId=null 2024-02-01T10:47:20.075 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sghatyAePyXF3t60Ir4Z/bT8N0E= 2024-02-01T10:47:20.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sghatyAePyXF3t60Ir4Z/bT8N0E=', 'Content-Length': '0'} 2024-02-01T10:47:20.078 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a2031c87ec824909-0065bb76b8-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:20.120 DEBUG:boto:path=/vclfcn-59/obj1 2024-02-01T10:47:20.120 DEBUG:boto:auth_path=/vclfcn-59/obj1 2024-02-01T10:47:20.120 DEBUG:boto:path=/vclfcn-59/obj1?versionId=null 2024-02-01T10:47:20.120 DEBUG:boto:auth_path=/vclfcn-59/obj1?versionId=null 2024-02-01T10:47:20.121 DEBUG:boto:Method: GET 2024-02-01T10:47:20.121 DEBUG:boto:Path: /vclfcn-59/obj1?versionId=null 2024-02-01T10:47:20.121 DEBUG:boto:Data: 2024-02-01T10:47:20.121 DEBUG:boto:Headers: {} 2024-02-01T10:47:20.121 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:20.121 DEBUG:boto:Port: 8000 2024-02-01T10:47:20.121 DEBUG:boto:Params: {} 2024-02-01T10:47:20.122 DEBUG:boto:Token: None 2024-02-01T10:47:20.122 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:20 GMT /vclfcn-59/obj1?versionId=null 2024-02-01T10:47:20.122 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sghatyAePyXF3t60Ir4Z/bT8N0E= 2024-02-01T10:47:20.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sghatyAePyXF3t60Ir4Z/bT8N0E=', 'Content-Length': '0'} 2024-02-01T10:47:20.125 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005adbc7343a78d678-0065bb76b8-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:20.125 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:47:20.125 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:47:20.125 DEBUG:boto:path=/vclfcn-59/?prefix=obj1 2024-02-01T10:47:20.125 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj1 2024-02-01T10:47:20.126 DEBUG:boto:Method: GET 2024-02-01T10:47:20.126 DEBUG:boto:Path: /vclfcn-59/?prefix=obj1 2024-02-01T10:47:20.126 DEBUG:boto:Data: 2024-02-01T10:47:20.126 DEBUG:boto:Headers: {} 2024-02-01T10:47:20.126 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:20.126 DEBUG:boto:Port: 8001 2024-02-01T10:47:20.126 DEBUG:boto:Params: {} 2024-02-01T10:47:20.126 DEBUG:boto:Token: None 2024-02-01T10:47:20.126 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:20 GMT /vclfcn-59/ 2024-02-01T10:47:20.126 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WBTJruCgEOkU7qnN0wNZ+a+05CI= 2024-02-01T10:47:20.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WBTJruCgEOkU7qnN0wNZ+a+05CI=', 'Content-Length': '0'} 2024-02-01T10:47:20.131 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f358b404bbc66431-0065bb76b8-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:20.131 DEBUG:boto:b'vclfcn-59obj11000falseobj12024-02-01T10:46:46.093Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:20.132 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:47:20.132 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:47:20.132 DEBUG:boto:path=/vclfcn-59/?prefix=obj1 2024-02-01T10:47:20.132 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj1 2024-02-01T10:47:20.132 DEBUG:boto:Method: GET 2024-02-01T10:47:20.132 DEBUG:boto:Path: /vclfcn-59/?prefix=obj1 2024-02-01T10:47:20.132 DEBUG:boto:Data: 2024-02-01T10:47:20.132 DEBUG:boto:Headers: {} 2024-02-01T10:47:20.132 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:20.133 DEBUG:boto:Port: 8000 2024-02-01T10:47:20.133 DEBUG:boto:Params: {} 2024-02-01T10:47:20.133 DEBUG:boto:Token: None 2024-02-01T10:47:20.133 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:20 GMT /vclfcn-59/ 2024-02-01T10:47:20.133 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WBTJruCgEOkU7qnN0wNZ+a+05CI= 2024-02-01T10:47:20.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WBTJruCgEOkU7qnN0wNZ+a+05CI=', 'Content-Length': '0'} 2024-02-01T10:47:20.137 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff1edd308a1fdafd-0065bb76b8-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:20.180 DEBUG:boto:b'vclfcn-59obj11000falseobj12024-02-01T10:46:46.093Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:20.180 DEBUG:boto:path=/vclfcn-59/obj1 2024-02-01T10:47:20.181 DEBUG:boto:auth_path=/vclfcn-59/obj1 2024-02-01T10:47:20.181 DEBUG:boto:Method: HEAD 2024-02-01T10:47:20.181 DEBUG:boto:Path: /vclfcn-59/obj1 2024-02-01T10:47:20.181 DEBUG:boto:Data: 2024-02-01T10:47:20.181 DEBUG:boto:Headers: {} 2024-02-01T10:47:20.181 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:20.181 DEBUG:boto:Port: 8001 2024-02-01T10:47:20.181 DEBUG:boto:Params: {} 2024-02-01T10:47:20.181 DEBUG:boto:Token: None 2024-02-01T10:47:20.182 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:20 GMT /vclfcn-59/obj1 2024-02-01T10:47:20.182 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:dcfOSzX9DzF83+YmOxeB5OdpUsE= 2024-02-01T10:47:20.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dcfOSzX9DzF83+YmOxeB5OdpUsE=', 'Content-Length': '0'} 2024-02-01T10:47:20.185 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000049de0857f874c3cd-0065bb76b8-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:20.185 DEBUG:boto:path=/vclfcn-59/obj1 2024-02-01T10:47:20.185 DEBUG:boto:auth_path=/vclfcn-59/obj1 2024-02-01T10:47:20.185 DEBUG:boto:Method: HEAD 2024-02-01T10:47:20.185 DEBUG:boto:Path: /vclfcn-59/obj1 2024-02-01T10:47:20.185 DEBUG:boto:Data: 2024-02-01T10:47:20.185 DEBUG:boto:Headers: {} 2024-02-01T10:47:20.185 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:20.186 DEBUG:boto:Port: 8000 2024-02-01T10:47:20.186 DEBUG:boto:Params: {} 2024-02-01T10:47:20.186 DEBUG:boto:Token: None 2024-02-01T10:47:20.186 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:20 GMT /vclfcn-59/obj1 2024-02-01T10:47:20.186 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:dcfOSzX9DzF83+YmOxeB5OdpUsE= 2024-02-01T10:47:20.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dcfOSzX9DzF83+YmOxeB5OdpUsE=', 'Content-Length': '0'} 2024-02-01T10:47:20.189 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000084785ce162544f75-0065bb76b8-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:20.189 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:47:20.189 DEBUG:boto:path=/vclfcn-59/obj1 2024-02-01T10:47:20.189 DEBUG:boto:auth_path=/vclfcn-59/obj1 2024-02-01T10:47:20.189 DEBUG:boto:Method: GET 2024-02-01T10:47:20.189 DEBUG:boto:Path: /vclfcn-59/obj1 2024-02-01T10:47:20.189 DEBUG:boto:Data: 2024-02-01T10:47:20.190 DEBUG:boto:Headers: {} 2024-02-01T10:47:20.190 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:20.190 DEBUG:boto:Port: 8001 2024-02-01T10:47:20.190 DEBUG:boto:Params: {} 2024-02-01T10:47:20.190 DEBUG:boto:Token: None 2024-02-01T10:47:20.190 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:20 GMT /vclfcn-59/obj1 2024-02-01T10:47:20.190 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:pRoZJgKOzMxChns3ECN6otNvL64= 2024-02-01T10:47:20.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:pRoZJgKOzMxChns3ECN6otNvL64=', 'Content-Length': '0'} 2024-02-01T10:47:20.193 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009dc679432b6f0564-0065bb76b8-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:20.236 DEBUG:boto:path=/vclfcn-59/obj1 2024-02-01T10:47:20.236 DEBUG:boto:auth_path=/vclfcn-59/obj1 2024-02-01T10:47:20.236 DEBUG:boto:Method: GET 2024-02-01T10:47:20.236 DEBUG:boto:Path: /vclfcn-59/obj1 2024-02-01T10:47:20.237 DEBUG:boto:Data: 2024-02-01T10:47:20.237 DEBUG:boto:Headers: {} 2024-02-01T10:47:20.237 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:20.237 DEBUG:boto:Port: 8000 2024-02-01T10:47:20.237 DEBUG:boto:Params: {} 2024-02-01T10:47:20.237 DEBUG:boto:Token: None 2024-02-01T10:47:20.237 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:20 GMT /vclfcn-59/obj1 2024-02-01T10:47:20.237 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:pRoZJgKOzMxChns3ECN6otNvL64= 2024-02-01T10:47:20.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:pRoZJgKOzMxChns3ECN6otNvL64=', 'Content-Length': '0'} 2024-02-01T10:47:20.241 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f0ff9f1d7853e05d-0065bb76b8-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:20.284 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:47:20.284 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:47:20.284 DEBUG:boto:path=/vclfcn-59/?prefix=obj1 2024-02-01T10:47:20.285 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj1 2024-02-01T10:47:20.285 DEBUG:boto:Method: GET 2024-02-01T10:47:20.285 DEBUG:boto:Path: /vclfcn-59/?prefix=obj1 2024-02-01T10:47:20.285 DEBUG:boto:Data: 2024-02-01T10:47:20.285 DEBUG:boto:Headers: {} 2024-02-01T10:47:20.285 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:20.285 DEBUG:boto:Port: 8001 2024-02-01T10:47:20.285 DEBUG:boto:Params: {} 2024-02-01T10:47:20.285 DEBUG:boto:Token: None 2024-02-01T10:47:20.285 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:20 GMT /vclfcn-59/ 2024-02-01T10:47:20.286 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WBTJruCgEOkU7qnN0wNZ+a+05CI= 2024-02-01T10:47:20.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WBTJruCgEOkU7qnN0wNZ+a+05CI=', 'Content-Length': '0'} 2024-02-01T10:47:20.290 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018ed4ccc22e533d5-0065bb76b8-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:20.336 DEBUG:boto:b'vclfcn-59obj11000falseobj12024-02-01T10:46:46.093Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:20.337 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:47:20.337 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:47:20.337 DEBUG:boto:path=/vclfcn-59/?prefix=obj1 2024-02-01T10:47:20.337 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj1 2024-02-01T10:47:20.337 DEBUG:boto:Method: GET 2024-02-01T10:47:20.337 DEBUG:boto:Path: /vclfcn-59/?prefix=obj1 2024-02-01T10:47:20.337 DEBUG:boto:Data: 2024-02-01T10:47:20.337 DEBUG:boto:Headers: {} 2024-02-01T10:47:20.337 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:20.337 DEBUG:boto:Port: 8000 2024-02-01T10:47:20.338 DEBUG:boto:Params: {} 2024-02-01T10:47:20.338 DEBUG:boto:Token: None 2024-02-01T10:47:20.338 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:20 GMT /vclfcn-59/ 2024-02-01T10:47:20.338 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WBTJruCgEOkU7qnN0wNZ+a+05CI= 2024-02-01T10:47:20.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WBTJruCgEOkU7qnN0wNZ+a+05CI=', 'Content-Length': '0'} 2024-02-01T10:47:20.342 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f71ff28d485e5506-0065bb76b8-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:20.343 DEBUG:boto:b'vclfcn-59obj11000falseobj12024-02-01T10:46:46.093Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:20.343 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:47:20.343 DEBUG:boto:path=/vclfcn-59/obj2 2024-02-01T10:47:20.343 DEBUG:boto:auth_path=/vclfcn-59/obj2 2024-02-01T10:47:20.343 DEBUG:boto:path=/vclfcn-59/obj2?versionId=null 2024-02-01T10:47:20.343 DEBUG:boto:auth_path=/vclfcn-59/obj2?versionId=null 2024-02-01T10:47:20.344 DEBUG:boto:Method: GET 2024-02-01T10:47:20.344 DEBUG:boto:Path: /vclfcn-59/obj2?versionId=null 2024-02-01T10:47:20.344 DEBUG:boto:Data: 2024-02-01T10:47:20.344 DEBUG:boto:Headers: {} 2024-02-01T10:47:20.344 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:20.344 DEBUG:boto:Port: 8001 2024-02-01T10:47:20.344 DEBUG:boto:Params: {} 2024-02-01T10:47:20.344 DEBUG:boto:Token: None 2024-02-01T10:47:20.344 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:20 GMT /vclfcn-59/obj2?versionId=null 2024-02-01T10:47:20.344 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:CIdEqcT5cl5ACma6CBKzpfqnwHA= 2024-02-01T10:47:20.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:CIdEqcT5cl5ACma6CBKzpfqnwHA=', 'Content-Length': '0'} 2024-02-01T10:47:20.347 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bc4c2e5169bbe664-0065bb76b8-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:20.348 DEBUG:boto:path=/vclfcn-59/obj2 2024-02-01T10:47:20.348 DEBUG:boto:auth_path=/vclfcn-59/obj2 2024-02-01T10:47:20.348 DEBUG:boto:path=/vclfcn-59/obj2?versionId=null 2024-02-01T10:47:20.348 DEBUG:boto:auth_path=/vclfcn-59/obj2?versionId=null 2024-02-01T10:47:20.348 DEBUG:boto:Method: GET 2024-02-01T10:47:20.348 DEBUG:boto:Path: /vclfcn-59/obj2?versionId=null 2024-02-01T10:47:20.348 DEBUG:boto:Data: 2024-02-01T10:47:20.349 DEBUG:boto:Headers: {} 2024-02-01T10:47:20.349 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:20.349 DEBUG:boto:Port: 8000 2024-02-01T10:47:20.349 DEBUG:boto:Params: {} 2024-02-01T10:47:20.349 DEBUG:boto:Token: None 2024-02-01T10:47:20.349 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:20 GMT /vclfcn-59/obj2?versionId=null 2024-02-01T10:47:20.349 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:CIdEqcT5cl5ACma6CBKzpfqnwHA= 2024-02-01T10:47:20.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:CIdEqcT5cl5ACma6CBKzpfqnwHA=', 'Content-Length': '0'} 2024-02-01T10:47:20.352 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000026eb8d9b498deb05-0065bb76b8-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:20.396 DEBUG:boto:path=/vclfcn-59/obj2 2024-02-01T10:47:20.396 DEBUG:boto:auth_path=/vclfcn-59/obj2 2024-02-01T10:47:20.396 DEBUG:boto:path=/vclfcn-59/obj2?versionId=null 2024-02-01T10:47:20.397 DEBUG:boto:auth_path=/vclfcn-59/obj2?versionId=null 2024-02-01T10:47:20.397 DEBUG:boto:Method: HEAD 2024-02-01T10:47:20.397 DEBUG:boto:Path: /vclfcn-59/obj2?versionId=null 2024-02-01T10:47:20.397 DEBUG:boto:Data: 2024-02-01T10:47:20.397 DEBUG:boto:Headers: {} 2024-02-01T10:47:20.397 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:20.397 DEBUG:boto:Port: 8001 2024-02-01T10:47:20.397 DEBUG:boto:Params: {} 2024-02-01T10:47:20.397 DEBUG:boto:Token: None 2024-02-01T10:47:20.397 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:20 GMT /vclfcn-59/obj2?versionId=null 2024-02-01T10:47:20.398 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:6zso600GjLBXdo+4JDUFEHzBgXg= 2024-02-01T10:47:20.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:6zso600GjLBXdo+4JDUFEHzBgXg=', 'Content-Length': '0'} 2024-02-01T10:47:20.401 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000479fc2c174d8a07f-0065bb76b8-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:20.401 DEBUG:boto:path=/vclfcn-59/obj2 2024-02-01T10:47:20.401 DEBUG:boto:auth_path=/vclfcn-59/obj2 2024-02-01T10:47:20.401 DEBUG:boto:path=/vclfcn-59/obj2?versionId=null 2024-02-01T10:47:20.401 DEBUG:boto:auth_path=/vclfcn-59/obj2?versionId=null 2024-02-01T10:47:20.401 DEBUG:boto:Method: HEAD 2024-02-01T10:47:20.401 DEBUG:boto:Path: /vclfcn-59/obj2?versionId=null 2024-02-01T10:47:20.401 DEBUG:boto:Data: 2024-02-01T10:47:20.402 DEBUG:boto:Headers: {} 2024-02-01T10:47:20.402 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:20.402 DEBUG:boto:Port: 8000 2024-02-01T10:47:20.402 DEBUG:boto:Params: {} 2024-02-01T10:47:20.402 DEBUG:boto:Token: None 2024-02-01T10:47:20.402 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:20 GMT /vclfcn-59/obj2?versionId=null 2024-02-01T10:47:20.402 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:6zso600GjLBXdo+4JDUFEHzBgXg= 2024-02-01T10:47:20.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:6zso600GjLBXdo+4JDUFEHzBgXg=', 'Content-Length': '0'} 2024-02-01T10:47:20.405 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f1762031bea2ae44-0065bb76b8-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:20.405 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:47:20.405 DEBUG:boto:path=/vclfcn-59/obj2 2024-02-01T10:47:20.405 DEBUG:boto:auth_path=/vclfcn-59/obj2 2024-02-01T10:47:20.405 DEBUG:boto:path=/vclfcn-59/obj2?versionId=null 2024-02-01T10:47:20.405 DEBUG:boto:auth_path=/vclfcn-59/obj2?versionId=null 2024-02-01T10:47:20.405 DEBUG:boto:Method: GET 2024-02-01T10:47:20.405 DEBUG:boto:Path: /vclfcn-59/obj2?versionId=null 2024-02-01T10:47:20.406 DEBUG:boto:Data: 2024-02-01T10:47:20.406 DEBUG:boto:Headers: {} 2024-02-01T10:47:20.406 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:20.406 DEBUG:boto:Port: 8001 2024-02-01T10:47:20.406 DEBUG:boto:Params: {} 2024-02-01T10:47:20.406 DEBUG:boto:Token: None 2024-02-01T10:47:20.406 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:20 GMT /vclfcn-59/obj2?versionId=null 2024-02-01T10:47:20.406 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:CIdEqcT5cl5ACma6CBKzpfqnwHA= 2024-02-01T10:47:20.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:CIdEqcT5cl5ACma6CBKzpfqnwHA=', 'Content-Length': '0'} 2024-02-01T10:47:20.409 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000050d0518075edaa22-0065bb76b8-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:20.452 DEBUG:boto:path=/vclfcn-59/obj2 2024-02-01T10:47:20.452 DEBUG:boto:auth_path=/vclfcn-59/obj2 2024-02-01T10:47:20.452 DEBUG:boto:path=/vclfcn-59/obj2?versionId=null 2024-02-01T10:47:20.452 DEBUG:boto:auth_path=/vclfcn-59/obj2?versionId=null 2024-02-01T10:47:20.453 DEBUG:boto:Method: GET 2024-02-01T10:47:20.453 DEBUG:boto:Path: /vclfcn-59/obj2?versionId=null 2024-02-01T10:47:20.453 DEBUG:boto:Data: 2024-02-01T10:47:20.453 DEBUG:boto:Headers: {} 2024-02-01T10:47:20.453 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:20.453 DEBUG:boto:Port: 8000 2024-02-01T10:47:20.453 DEBUG:boto:Params: {} 2024-02-01T10:47:20.453 DEBUG:boto:Token: None 2024-02-01T10:47:20.453 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:20 GMT /vclfcn-59/obj2?versionId=null 2024-02-01T10:47:20.453 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:CIdEqcT5cl5ACma6CBKzpfqnwHA= 2024-02-01T10:47:20.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:CIdEqcT5cl5ACma6CBKzpfqnwHA=', 'Content-Length': '0'} 2024-02-01T10:47:20.457 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f263d72827f97453-0065bb76b8-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:20.500 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:47:20.500 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:47:20.500 DEBUG:boto:path=/vclfcn-59/?prefix=obj2 2024-02-01T10:47:20.500 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj2 2024-02-01T10:47:20.500 DEBUG:boto:Method: GET 2024-02-01T10:47:20.501 DEBUG:boto:Path: /vclfcn-59/?prefix=obj2 2024-02-01T10:47:20.501 DEBUG:boto:Data: 2024-02-01T10:47:20.501 DEBUG:boto:Headers: {} 2024-02-01T10:47:20.501 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:20.501 DEBUG:boto:Port: 8001 2024-02-01T10:47:20.501 DEBUG:boto:Params: {} 2024-02-01T10:47:20.501 DEBUG:boto:Token: None 2024-02-01T10:47:20.501 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:20 GMT /vclfcn-59/ 2024-02-01T10:47:20.501 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WBTJruCgEOkU7qnN0wNZ+a+05CI= 2024-02-01T10:47:20.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WBTJruCgEOkU7qnN0wNZ+a+05CI=', 'Content-Length': '0'} 2024-02-01T10:47:20.505 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd883e1c012d17b1-0065bb76b8-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:20.548 DEBUG:boto:b'vclfcn-59obj21000falseobj22024-02-01T10:46:46.111Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:20.548 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:47:20.549 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:47:20.549 DEBUG:boto:path=/vclfcn-59/?prefix=obj2 2024-02-01T10:47:20.549 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj2 2024-02-01T10:47:20.549 DEBUG:boto:Method: GET 2024-02-01T10:47:20.549 DEBUG:boto:Path: /vclfcn-59/?prefix=obj2 2024-02-01T10:47:20.549 DEBUG:boto:Data: 2024-02-01T10:47:20.549 DEBUG:boto:Headers: {} 2024-02-01T10:47:20.549 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:20.549 DEBUG:boto:Port: 8000 2024-02-01T10:47:20.549 DEBUG:boto:Params: {} 2024-02-01T10:47:20.550 DEBUG:boto:Token: None 2024-02-01T10:47:20.550 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:20 GMT /vclfcn-59/ 2024-02-01T10:47:20.550 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WBTJruCgEOkU7qnN0wNZ+a+05CI= 2024-02-01T10:47:20.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WBTJruCgEOkU7qnN0wNZ+a+05CI=', 'Content-Length': '0'} 2024-02-01T10:47:20.554 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a34721cc98de1f48-0065bb76b8-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:20.554 DEBUG:boto:b'vclfcn-59obj21000falseobj22024-02-01T10:46:46.111Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:20.555 DEBUG:boto:path=/vclfcn-59/obj2 2024-02-01T10:47:20.555 DEBUG:boto:auth_path=/vclfcn-59/obj2 2024-02-01T10:47:20.555 DEBUG:boto:Method: HEAD 2024-02-01T10:47:20.555 DEBUG:boto:Path: /vclfcn-59/obj2 2024-02-01T10:47:20.555 DEBUG:boto:Data: 2024-02-01T10:47:20.555 DEBUG:boto:Headers: {} 2024-02-01T10:47:20.555 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:20.555 DEBUG:boto:Port: 8001 2024-02-01T10:47:20.555 DEBUG:boto:Params: {} 2024-02-01T10:47:20.556 DEBUG:boto:Token: None 2024-02-01T10:47:20.556 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:20 GMT /vclfcn-59/obj2 2024-02-01T10:47:20.556 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:UbPmQswjHzgV9prVaZ7WD84l62o= 2024-02-01T10:47:20.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:UbPmQswjHzgV9prVaZ7WD84l62o=', 'Content-Length': '0'} 2024-02-01T10:47:20.559 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002341c7f0da54847a-0065bb76b8-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:20.559 DEBUG:boto:path=/vclfcn-59/obj2 2024-02-01T10:47:20.559 DEBUG:boto:auth_path=/vclfcn-59/obj2 2024-02-01T10:47:20.560 DEBUG:boto:Method: HEAD 2024-02-01T10:47:20.560 DEBUG:boto:Path: /vclfcn-59/obj2 2024-02-01T10:47:20.560 DEBUG:boto:Data: 2024-02-01T10:47:20.560 DEBUG:boto:Headers: {} 2024-02-01T10:47:20.560 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:20.560 DEBUG:boto:Port: 8000 2024-02-01T10:47:20.560 DEBUG:boto:Params: {} 2024-02-01T10:47:20.560 DEBUG:boto:Token: None 2024-02-01T10:47:20.560 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:20 GMT /vclfcn-59/obj2 2024-02-01T10:47:20.561 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:UbPmQswjHzgV9prVaZ7WD84l62o= 2024-02-01T10:47:20.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:UbPmQswjHzgV9prVaZ7WD84l62o=', 'Content-Length': '0'} 2024-02-01T10:47:20.564 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009c1ab09b48190b1d-0065bb76b8-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:20.564 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:47:20.564 DEBUG:boto:path=/vclfcn-59/obj2 2024-02-01T10:47:20.564 DEBUG:boto:auth_path=/vclfcn-59/obj2 2024-02-01T10:47:20.564 DEBUG:boto:Method: GET 2024-02-01T10:47:20.564 DEBUG:boto:Path: /vclfcn-59/obj2 2024-02-01T10:47:20.564 DEBUG:boto:Data: 2024-02-01T10:47:20.565 DEBUG:boto:Headers: {} 2024-02-01T10:47:20.565 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:20.565 DEBUG:boto:Port: 8001 2024-02-01T10:47:20.565 DEBUG:boto:Params: {} 2024-02-01T10:47:20.565 DEBUG:boto:Token: None 2024-02-01T10:47:20.565 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:20 GMT /vclfcn-59/obj2 2024-02-01T10:47:20.565 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7ZuJ/5x88Wx3qefP82tOKy03EXU= 2024-02-01T10:47:20.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7ZuJ/5x88Wx3qefP82tOKy03EXU=', 'Content-Length': '0'} 2024-02-01T10:47:20.568 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000954c22359d85a8a5-0065bb76b8-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:20.612 DEBUG:boto:path=/vclfcn-59/obj2 2024-02-01T10:47:20.612 DEBUG:boto:auth_path=/vclfcn-59/obj2 2024-02-01T10:47:20.612 DEBUG:boto:Method: GET 2024-02-01T10:47:20.612 DEBUG:boto:Path: /vclfcn-59/obj2 2024-02-01T10:47:20.612 DEBUG:boto:Data: 2024-02-01T10:47:20.613 DEBUG:boto:Headers: {} 2024-02-01T10:47:20.613 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:20.613 DEBUG:boto:Port: 8000 2024-02-01T10:47:20.613 DEBUG:boto:Params: {} 2024-02-01T10:47:20.613 DEBUG:boto:Token: None 2024-02-01T10:47:20.613 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:20 GMT /vclfcn-59/obj2 2024-02-01T10:47:20.613 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7ZuJ/5x88Wx3qefP82tOKy03EXU= 2024-02-01T10:47:20.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7ZuJ/5x88Wx3qefP82tOKy03EXU=', 'Content-Length': '0'} 2024-02-01T10:47:20.616 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ce48f27d752fb4dc-0065bb76b8-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:20.660 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:47:20.660 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:47:20.660 DEBUG:boto:path=/vclfcn-59/?prefix=obj2 2024-02-01T10:47:20.661 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj2 2024-02-01T10:47:20.661 DEBUG:boto:Method: GET 2024-02-01T10:47:20.661 DEBUG:boto:Path: /vclfcn-59/?prefix=obj2 2024-02-01T10:47:20.661 DEBUG:boto:Data: 2024-02-01T10:47:20.661 DEBUG:boto:Headers: {} 2024-02-01T10:47:20.661 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:20.661 DEBUG:boto:Port: 8001 2024-02-01T10:47:20.661 DEBUG:boto:Params: {} 2024-02-01T10:47:20.661 DEBUG:boto:Token: None 2024-02-01T10:47:20.661 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:20 GMT /vclfcn-59/ 2024-02-01T10:47:20.662 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WBTJruCgEOkU7qnN0wNZ+a+05CI= 2024-02-01T10:47:20.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WBTJruCgEOkU7qnN0wNZ+a+05CI=', 'Content-Length': '0'} 2024-02-01T10:47:20.666 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3a7ff4a6635b081-0065bb76b8-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:20.708 DEBUG:boto:b'vclfcn-59obj21000falseobj22024-02-01T10:46:46.111Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:20.708 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:47:20.709 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:47:20.709 DEBUG:boto:path=/vclfcn-59/?prefix=obj2 2024-02-01T10:47:20.709 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj2 2024-02-01T10:47:20.709 DEBUG:boto:Method: GET 2024-02-01T10:47:20.709 DEBUG:boto:Path: /vclfcn-59/?prefix=obj2 2024-02-01T10:47:20.709 DEBUG:boto:Data: 2024-02-01T10:47:20.709 DEBUG:boto:Headers: {} 2024-02-01T10:47:20.709 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:20.709 DEBUG:boto:Port: 8000 2024-02-01T10:47:20.709 DEBUG:boto:Params: {} 2024-02-01T10:47:20.710 DEBUG:boto:Token: None 2024-02-01T10:47:20.710 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:20 GMT /vclfcn-59/ 2024-02-01T10:47:20.710 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WBTJruCgEOkU7qnN0wNZ+a+05CI= 2024-02-01T10:47:20.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WBTJruCgEOkU7qnN0wNZ+a+05CI=', 'Content-Length': '0'} 2024-02-01T10:47:20.714 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004417dc07e47ebd1b-0065bb76b8-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:20.756 DEBUG:boto:b'vclfcn-59obj21000falseobj22024-02-01T10:46:46.111Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:20.756 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:47:20.757 DEBUG:boto:path=/vclfcn-59/obj3 2024-02-01T10:47:20.757 DEBUG:boto:auth_path=/vclfcn-59/obj3 2024-02-01T10:47:20.757 DEBUG:boto:path=/vclfcn-59/obj3?versionId=null 2024-02-01T10:47:20.757 DEBUG:boto:auth_path=/vclfcn-59/obj3?versionId=null 2024-02-01T10:47:20.757 DEBUG:boto:Method: GET 2024-02-01T10:47:20.757 DEBUG:boto:Path: /vclfcn-59/obj3?versionId=null 2024-02-01T10:47:20.757 DEBUG:boto:Data: 2024-02-01T10:47:20.757 DEBUG:boto:Headers: {} 2024-02-01T10:47:20.757 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:20.757 DEBUG:boto:Port: 8001 2024-02-01T10:47:20.757 DEBUG:boto:Params: {} 2024-02-01T10:47:20.758 DEBUG:boto:Token: None 2024-02-01T10:47:20.758 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:20 GMT /vclfcn-59/obj3?versionId=null 2024-02-01T10:47:20.758 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NUk6NLrxcb1DXNp1RD1blzDENiY= 2024-02-01T10:47:20.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NUk6NLrxcb1DXNp1RD1blzDENiY=', 'Content-Length': '0'} 2024-02-01T10:47:20.761 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000753767cbbc232572-0065bb76b8-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:20.804 DEBUG:boto:path=/vclfcn-59/obj3 2024-02-01T10:47:20.805 DEBUG:boto:auth_path=/vclfcn-59/obj3 2024-02-01T10:47:20.805 DEBUG:boto:path=/vclfcn-59/obj3?versionId=null 2024-02-01T10:47:20.805 DEBUG:boto:auth_path=/vclfcn-59/obj3?versionId=null 2024-02-01T10:47:20.805 DEBUG:boto:Method: GET 2024-02-01T10:47:20.805 DEBUG:boto:Path: /vclfcn-59/obj3?versionId=null 2024-02-01T10:47:20.805 DEBUG:boto:Data: 2024-02-01T10:47:20.805 DEBUG:boto:Headers: {} 2024-02-01T10:47:20.805 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:20.805 DEBUG:boto:Port: 8000 2024-02-01T10:47:20.806 DEBUG:boto:Params: {} 2024-02-01T10:47:20.806 DEBUG:boto:Token: None 2024-02-01T10:47:20.806 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:20 GMT /vclfcn-59/obj3?versionId=null 2024-02-01T10:47:20.806 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NUk6NLrxcb1DXNp1RD1blzDENiY= 2024-02-01T10:47:20.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NUk6NLrxcb1DXNp1RD1blzDENiY=', 'Content-Length': '0'} 2024-02-01T10:47:20.809 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ea1a6e3db211188a-0065bb76b8-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:20.852 DEBUG:boto:path=/vclfcn-59/obj3 2024-02-01T10:47:20.852 DEBUG:boto:auth_path=/vclfcn-59/obj3 2024-02-01T10:47:20.852 DEBUG:boto:path=/vclfcn-59/obj3?versionId=null 2024-02-01T10:47:20.852 DEBUG:boto:auth_path=/vclfcn-59/obj3?versionId=null 2024-02-01T10:47:20.853 DEBUG:boto:Method: HEAD 2024-02-01T10:47:20.853 DEBUG:boto:Path: /vclfcn-59/obj3?versionId=null 2024-02-01T10:47:20.853 DEBUG:boto:Data: 2024-02-01T10:47:20.853 DEBUG:boto:Headers: {} 2024-02-01T10:47:20.853 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:20.853 DEBUG:boto:Port: 8001 2024-02-01T10:47:20.853 DEBUG:boto:Params: {} 2024-02-01T10:47:20.853 DEBUG:boto:Token: None 2024-02-01T10:47:20.853 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:20 GMT /vclfcn-59/obj3?versionId=null 2024-02-01T10:47:20.853 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:CsEuAmcUTUNvzL6LSEQ+pTdh0OY= 2024-02-01T10:47:20.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:CsEuAmcUTUNvzL6LSEQ+pTdh0OY=', 'Content-Length': '0'} 2024-02-01T10:47:20.856 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000019db5b0438fbf8f9-0065bb76b8-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:20.857 DEBUG:boto:path=/vclfcn-59/obj3 2024-02-01T10:47:20.857 DEBUG:boto:auth_path=/vclfcn-59/obj3 2024-02-01T10:47:20.857 DEBUG:boto:path=/vclfcn-59/obj3?versionId=null 2024-02-01T10:47:20.857 DEBUG:boto:auth_path=/vclfcn-59/obj3?versionId=null 2024-02-01T10:47:20.857 DEBUG:boto:Method: HEAD 2024-02-01T10:47:20.857 DEBUG:boto:Path: /vclfcn-59/obj3?versionId=null 2024-02-01T10:47:20.857 DEBUG:boto:Data: 2024-02-01T10:47:20.857 DEBUG:boto:Headers: {} 2024-02-01T10:47:20.857 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:20.858 DEBUG:boto:Port: 8000 2024-02-01T10:47:20.858 DEBUG:boto:Params: {} 2024-02-01T10:47:20.858 DEBUG:boto:Token: None 2024-02-01T10:47:20.858 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:20 GMT /vclfcn-59/obj3?versionId=null 2024-02-01T10:47:20.858 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:CsEuAmcUTUNvzL6LSEQ+pTdh0OY= 2024-02-01T10:47:20.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:CsEuAmcUTUNvzL6LSEQ+pTdh0OY=', 'Content-Length': '0'} 2024-02-01T10:47:20.861 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b6430e537fb9a6c7-0065bb76b8-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:20.861 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:47:20.861 DEBUG:boto:path=/vclfcn-59/obj3 2024-02-01T10:47:20.862 DEBUG:boto:auth_path=/vclfcn-59/obj3 2024-02-01T10:47:20.862 DEBUG:boto:path=/vclfcn-59/obj3?versionId=null 2024-02-01T10:47:20.862 DEBUG:boto:auth_path=/vclfcn-59/obj3?versionId=null 2024-02-01T10:47:20.862 DEBUG:boto:Method: GET 2024-02-01T10:47:20.862 DEBUG:boto:Path: /vclfcn-59/obj3?versionId=null 2024-02-01T10:47:20.862 DEBUG:boto:Data: 2024-02-01T10:47:20.862 DEBUG:boto:Headers: {} 2024-02-01T10:47:20.862 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:20.862 DEBUG:boto:Port: 8001 2024-02-01T10:47:20.862 DEBUG:boto:Params: {} 2024-02-01T10:47:20.862 DEBUG:boto:Token: None 2024-02-01T10:47:20.863 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:20 GMT /vclfcn-59/obj3?versionId=null 2024-02-01T10:47:20.863 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NUk6NLrxcb1DXNp1RD1blzDENiY= 2024-02-01T10:47:20.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NUk6NLrxcb1DXNp1RD1blzDENiY=', 'Content-Length': '0'} 2024-02-01T10:47:20.866 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000aa1b058cffa8fa2-0065bb76b8-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:20.908 DEBUG:boto:path=/vclfcn-59/obj3 2024-02-01T10:47:20.908 DEBUG:boto:auth_path=/vclfcn-59/obj3 2024-02-01T10:47:20.908 DEBUG:boto:path=/vclfcn-59/obj3?versionId=null 2024-02-01T10:47:20.908 DEBUG:boto:auth_path=/vclfcn-59/obj3?versionId=null 2024-02-01T10:47:20.908 DEBUG:boto:Method: GET 2024-02-01T10:47:20.909 DEBUG:boto:Path: /vclfcn-59/obj3?versionId=null 2024-02-01T10:47:20.909 DEBUG:boto:Data: 2024-02-01T10:47:20.909 DEBUG:boto:Headers: {} 2024-02-01T10:47:20.909 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:20.909 DEBUG:boto:Port: 8000 2024-02-01T10:47:20.909 DEBUG:boto:Params: {} 2024-02-01T10:47:20.909 DEBUG:boto:Token: None 2024-02-01T10:47:20.909 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:20 GMT /vclfcn-59/obj3?versionId=null 2024-02-01T10:47:20.909 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NUk6NLrxcb1DXNp1RD1blzDENiY= 2024-02-01T10:47:20.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NUk6NLrxcb1DXNp1RD1blzDENiY=', 'Content-Length': '0'} 2024-02-01T10:47:20.912 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000051b2cc1b9c226cd8-0065bb76b8-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:20.956 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:47:20.957 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:47:20.957 DEBUG:boto:path=/vclfcn-59/?prefix=obj3 2024-02-01T10:47:20.957 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj3 2024-02-01T10:47:20.957 DEBUG:boto:Method: GET 2024-02-01T10:47:20.957 DEBUG:boto:Path: /vclfcn-59/?prefix=obj3 2024-02-01T10:47:20.957 DEBUG:boto:Data: 2024-02-01T10:47:20.957 DEBUG:boto:Headers: {} 2024-02-01T10:47:20.957 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:20.958 DEBUG:boto:Port: 8001 2024-02-01T10:47:20.958 DEBUG:boto:Params: {} 2024-02-01T10:47:20.958 DEBUG:boto:Token: None 2024-02-01T10:47:20.958 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:20 GMT /vclfcn-59/ 2024-02-01T10:47:20.958 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WBTJruCgEOkU7qnN0wNZ+a+05CI= 2024-02-01T10:47:20.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WBTJruCgEOkU7qnN0wNZ+a+05CI=', 'Content-Length': '0'} 2024-02-01T10:47:20.963 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f856b44a2771bfe-0065bb76b8-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:20.963 DEBUG:boto:b'vclfcn-59obj31000falseobj32024-02-01T10:46:46.161Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:20.964 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:47:20.964 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:47:20.964 DEBUG:boto:path=/vclfcn-59/?prefix=obj3 2024-02-01T10:47:20.964 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj3 2024-02-01T10:47:20.964 DEBUG:boto:Method: GET 2024-02-01T10:47:20.964 DEBUG:boto:Path: /vclfcn-59/?prefix=obj3 2024-02-01T10:47:20.964 DEBUG:boto:Data: 2024-02-01T10:47:20.964 DEBUG:boto:Headers: {} 2024-02-01T10:47:20.965 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:20.965 DEBUG:boto:Port: 8000 2024-02-01T10:47:20.965 DEBUG:boto:Params: {} 2024-02-01T10:47:20.965 DEBUG:boto:Token: None 2024-02-01T10:47:20.965 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:20 GMT /vclfcn-59/ 2024-02-01T10:47:20.966 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WBTJruCgEOkU7qnN0wNZ+a+05CI= 2024-02-01T10:47:20.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WBTJruCgEOkU7qnN0wNZ+a+05CI=', 'Content-Length': '0'} 2024-02-01T10:47:20.971 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000854aee0edf7c8ff4-0065bb76b8-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:20.971 DEBUG:boto:b'vclfcn-59obj31000falseobj32024-02-01T10:46:46.161Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:20.971 DEBUG:boto:path=/vclfcn-59/obj3 2024-02-01T10:47:20.971 DEBUG:boto:auth_path=/vclfcn-59/obj3 2024-02-01T10:47:20.971 DEBUG:boto:Method: HEAD 2024-02-01T10:47:20.971 DEBUG:boto:Path: /vclfcn-59/obj3 2024-02-01T10:47:20.972 DEBUG:boto:Data: 2024-02-01T10:47:20.972 DEBUG:boto:Headers: {} 2024-02-01T10:47:20.972 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:20.972 DEBUG:boto:Port: 8001 2024-02-01T10:47:20.972 DEBUG:boto:Params: {} 2024-02-01T10:47:20.972 DEBUG:boto:Token: None 2024-02-01T10:47:20.972 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:20 GMT /vclfcn-59/obj3 2024-02-01T10:47:20.972 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HBVRctES2x4+y6WlIZ4HqNPC9Vc= 2024-02-01T10:47:20.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HBVRctES2x4+y6WlIZ4HqNPC9Vc=', 'Content-Length': '0'} 2024-02-01T10:47:20.975 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008bd9f1e7f3cc000e-0065bb76b8-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:20.975 DEBUG:boto:path=/vclfcn-59/obj3 2024-02-01T10:47:20.975 DEBUG:boto:auth_path=/vclfcn-59/obj3 2024-02-01T10:47:20.976 DEBUG:boto:Method: HEAD 2024-02-01T10:47:20.976 DEBUG:boto:Path: /vclfcn-59/obj3 2024-02-01T10:47:20.976 DEBUG:boto:Data: 2024-02-01T10:47:20.976 DEBUG:boto:Headers: {} 2024-02-01T10:47:20.976 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:20.976 DEBUG:boto:Port: 8000 2024-02-01T10:47:20.976 DEBUG:boto:Params: {} 2024-02-01T10:47:20.976 DEBUG:boto:Token: None 2024-02-01T10:47:20.976 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:20 GMT /vclfcn-59/obj3 2024-02-01T10:47:20.976 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HBVRctES2x4+y6WlIZ4HqNPC9Vc= 2024-02-01T10:47:20.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HBVRctES2x4+y6WlIZ4HqNPC9Vc=', 'Content-Length': '0'} 2024-02-01T10:47:20.979 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ea922c2a1ade04e2-0065bb76b8-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:20.979 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:47:20.980 DEBUG:boto:path=/vclfcn-59/obj3 2024-02-01T10:47:20.980 DEBUG:boto:auth_path=/vclfcn-59/obj3 2024-02-01T10:47:20.980 DEBUG:boto:Method: GET 2024-02-01T10:47:20.980 DEBUG:boto:Path: /vclfcn-59/obj3 2024-02-01T10:47:20.980 DEBUG:boto:Data: 2024-02-01T10:47:20.980 DEBUG:boto:Headers: {} 2024-02-01T10:47:20.980 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:20.980 DEBUG:boto:Port: 8001 2024-02-01T10:47:20.980 DEBUG:boto:Params: {} 2024-02-01T10:47:20.981 DEBUG:boto:Token: None 2024-02-01T10:47:20.981 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:20 GMT /vclfcn-59/obj3 2024-02-01T10:47:20.981 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:U70T5cGMtAPTNCXS2FkyQVmtWzk= 2024-02-01T10:47:20.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:U70T5cGMtAPTNCXS2FkyQVmtWzk=', 'Content-Length': '0'} 2024-02-01T10:47:20.983 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b7a8a31b62390206-0065bb76b8-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:21.028 DEBUG:boto:path=/vclfcn-59/obj3 2024-02-01T10:47:21.028 DEBUG:boto:auth_path=/vclfcn-59/obj3 2024-02-01T10:47:21.028 DEBUG:boto:Method: GET 2024-02-01T10:47:21.028 DEBUG:boto:Path: /vclfcn-59/obj3 2024-02-01T10:47:21.029 DEBUG:boto:Data: 2024-02-01T10:47:21.029 DEBUG:boto:Headers: {} 2024-02-01T10:47:21.029 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:21.029 DEBUG:boto:Port: 8000 2024-02-01T10:47:21.029 DEBUG:boto:Params: {} 2024-02-01T10:47:21.029 DEBUG:boto:Token: None 2024-02-01T10:47:21.029 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:21 GMT /vclfcn-59/obj3 2024-02-01T10:47:21.029 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:iSsHsi05n7e3zZSk1fcI6GQ76uY= 2024-02-01T10:47:21.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iSsHsi05n7e3zZSk1fcI6GQ76uY=', 'Content-Length': '0'} 2024-02-01T10:47:21.032 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004af8eafe8c357c90-0065bb76b9-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:21.076 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:47:21.076 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:47:21.077 DEBUG:boto:path=/vclfcn-59/?prefix=obj3 2024-02-01T10:47:21.077 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj3 2024-02-01T10:47:21.077 DEBUG:boto:Method: GET 2024-02-01T10:47:21.077 DEBUG:boto:Path: /vclfcn-59/?prefix=obj3 2024-02-01T10:47:21.077 DEBUG:boto:Data: 2024-02-01T10:47:21.077 DEBUG:boto:Headers: {} 2024-02-01T10:47:21.077 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:21.077 DEBUG:boto:Port: 8001 2024-02-01T10:47:21.077 DEBUG:boto:Params: {} 2024-02-01T10:47:21.077 DEBUG:boto:Token: None 2024-02-01T10:47:21.078 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:21 GMT /vclfcn-59/ 2024-02-01T10:47:21.078 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:eUriIJ80xTPndDEhiholfVP1W/Y= 2024-02-01T10:47:21.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:eUriIJ80xTPndDEhiholfVP1W/Y=', 'Content-Length': '0'} 2024-02-01T10:47:21.082 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f3c9f760d222aa2-0065bb76b9-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:21.082 DEBUG:boto:b'vclfcn-59obj31000falseobj32024-02-01T10:46:46.161Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:21.083 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:47:21.083 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:47:21.083 DEBUG:boto:path=/vclfcn-59/?prefix=obj3 2024-02-01T10:47:21.083 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj3 2024-02-01T10:47:21.083 DEBUG:boto:Method: GET 2024-02-01T10:47:21.083 DEBUG:boto:Path: /vclfcn-59/?prefix=obj3 2024-02-01T10:47:21.083 DEBUG:boto:Data: 2024-02-01T10:47:21.083 DEBUG:boto:Headers: {} 2024-02-01T10:47:21.084 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:21.084 DEBUG:boto:Port: 8000 2024-02-01T10:47:21.084 DEBUG:boto:Params: {} 2024-02-01T10:47:21.084 DEBUG:boto:Token: None 2024-02-01T10:47:21.084 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:21 GMT /vclfcn-59/ 2024-02-01T10:47:21.084 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:eUriIJ80xTPndDEhiholfVP1W/Y= 2024-02-01T10:47:21.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:eUriIJ80xTPndDEhiholfVP1W/Y=', 'Content-Length': '0'} 2024-02-01T10:47:21.089 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b8ab13996c7e017b-0065bb76b9-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:21.132 DEBUG:boto:b'vclfcn-59obj31000falseobj32024-02-01T10:46:46.161Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:21.132 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:47:21.133 DEBUG:boto:path=/vclfcn-59/obj4 2024-02-01T10:47:21.133 DEBUG:boto:auth_path=/vclfcn-59/obj4 2024-02-01T10:47:21.133 DEBUG:boto:path=/vclfcn-59/obj4?versionId=null 2024-02-01T10:47:21.133 DEBUG:boto:auth_path=/vclfcn-59/obj4?versionId=null 2024-02-01T10:47:21.133 DEBUG:boto:Method: GET 2024-02-01T10:47:21.133 DEBUG:boto:Path: /vclfcn-59/obj4?versionId=null 2024-02-01T10:47:21.133 DEBUG:boto:Data: 2024-02-01T10:47:21.133 DEBUG:boto:Headers: {} 2024-02-01T10:47:21.133 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:21.133 DEBUG:boto:Port: 8001 2024-02-01T10:47:21.134 DEBUG:boto:Params: {} 2024-02-01T10:47:21.134 DEBUG:boto:Token: None 2024-02-01T10:47:21.134 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:21 GMT /vclfcn-59/obj4?versionId=null 2024-02-01T10:47:21.134 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZIqoyLuJJ94cL1ljI05AvbMeivw= 2024-02-01T10:47:21.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZIqoyLuJJ94cL1ljI05AvbMeivw=', 'Content-Length': '0'} 2024-02-01T10:47:21.137 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a50c23ba6bcd383f-0065bb76b9-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:21.180 DEBUG:boto:path=/vclfcn-59/obj4 2024-02-01T10:47:21.180 DEBUG:boto:auth_path=/vclfcn-59/obj4 2024-02-01T10:47:21.180 DEBUG:boto:path=/vclfcn-59/obj4?versionId=null 2024-02-01T10:47:21.181 DEBUG:boto:auth_path=/vclfcn-59/obj4?versionId=null 2024-02-01T10:47:21.181 DEBUG:boto:Method: GET 2024-02-01T10:47:21.181 DEBUG:boto:Path: /vclfcn-59/obj4?versionId=null 2024-02-01T10:47:21.181 DEBUG:boto:Data: 2024-02-01T10:47:21.181 DEBUG:boto:Headers: {} 2024-02-01T10:47:21.181 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:21.181 DEBUG:boto:Port: 8000 2024-02-01T10:47:21.181 DEBUG:boto:Params: {} 2024-02-01T10:47:21.181 DEBUG:boto:Token: None 2024-02-01T10:47:21.182 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:21 GMT /vclfcn-59/obj4?versionId=null 2024-02-01T10:47:21.182 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZIqoyLuJJ94cL1ljI05AvbMeivw= 2024-02-01T10:47:21.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZIqoyLuJJ94cL1ljI05AvbMeivw=', 'Content-Length': '0'} 2024-02-01T10:47:21.185 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bae67acc7778f183-0065bb76b9-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:21.185 DEBUG:boto:path=/vclfcn-59/obj4 2024-02-01T10:47:21.185 DEBUG:boto:auth_path=/vclfcn-59/obj4 2024-02-01T10:47:21.185 DEBUG:boto:path=/vclfcn-59/obj4?versionId=null 2024-02-01T10:47:21.186 DEBUG:boto:auth_path=/vclfcn-59/obj4?versionId=null 2024-02-01T10:47:21.186 DEBUG:boto:Method: HEAD 2024-02-01T10:47:21.186 DEBUG:boto:Path: /vclfcn-59/obj4?versionId=null 2024-02-01T10:47:21.186 DEBUG:boto:Data: 2024-02-01T10:47:21.186 DEBUG:boto:Headers: {} 2024-02-01T10:47:21.186 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:21.186 DEBUG:boto:Port: 8001 2024-02-01T10:47:21.186 DEBUG:boto:Params: {} 2024-02-01T10:47:21.186 DEBUG:boto:Token: None 2024-02-01T10:47:21.186 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:21 GMT /vclfcn-59/obj4?versionId=null 2024-02-01T10:47:21.187 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:CkOoNROS7Oq7WD9xUIPB9YYyNsE= 2024-02-01T10:47:21.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:CkOoNROS7Oq7WD9xUIPB9YYyNsE=', 'Content-Length': '0'} 2024-02-01T10:47:21.189 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000042e75b4fbab338-0065bb76b9-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:21.190 DEBUG:boto:path=/vclfcn-59/obj4 2024-02-01T10:47:21.190 DEBUG:boto:auth_path=/vclfcn-59/obj4 2024-02-01T10:47:21.190 DEBUG:boto:path=/vclfcn-59/obj4?versionId=null 2024-02-01T10:47:21.190 DEBUG:boto:auth_path=/vclfcn-59/obj4?versionId=null 2024-02-01T10:47:21.190 DEBUG:boto:Method: HEAD 2024-02-01T10:47:21.190 DEBUG:boto:Path: /vclfcn-59/obj4?versionId=null 2024-02-01T10:47:21.190 DEBUG:boto:Data: 2024-02-01T10:47:21.190 DEBUG:boto:Headers: {} 2024-02-01T10:47:21.190 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:21.190 DEBUG:boto:Port: 8000 2024-02-01T10:47:21.191 DEBUG:boto:Params: {} 2024-02-01T10:47:21.191 DEBUG:boto:Token: None 2024-02-01T10:47:21.191 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:21 GMT /vclfcn-59/obj4?versionId=null 2024-02-01T10:47:21.191 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:CkOoNROS7Oq7WD9xUIPB9YYyNsE= 2024-02-01T10:47:21.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:CkOoNROS7Oq7WD9xUIPB9YYyNsE=', 'Content-Length': '0'} 2024-02-01T10:47:21.194 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003d374a0d7ce1127d-0065bb76b9-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:21.194 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:47:21.194 DEBUG:boto:path=/vclfcn-59/obj4 2024-02-01T10:47:21.194 DEBUG:boto:auth_path=/vclfcn-59/obj4 2024-02-01T10:47:21.194 DEBUG:boto:path=/vclfcn-59/obj4?versionId=null 2024-02-01T10:47:21.194 DEBUG:boto:auth_path=/vclfcn-59/obj4?versionId=null 2024-02-01T10:47:21.195 DEBUG:boto:Method: GET 2024-02-01T10:47:21.195 DEBUG:boto:Path: /vclfcn-59/obj4?versionId=null 2024-02-01T10:47:21.195 DEBUG:boto:Data: 2024-02-01T10:47:21.195 DEBUG:boto:Headers: {} 2024-02-01T10:47:21.195 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:21.195 DEBUG:boto:Port: 8001 2024-02-01T10:47:21.195 DEBUG:boto:Params: {} 2024-02-01T10:47:21.195 DEBUG:boto:Token: None 2024-02-01T10:47:21.195 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:21 GMT /vclfcn-59/obj4?versionId=null 2024-02-01T10:47:21.195 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZIqoyLuJJ94cL1ljI05AvbMeivw= 2024-02-01T10:47:21.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZIqoyLuJJ94cL1ljI05AvbMeivw=', 'Content-Length': '0'} 2024-02-01T10:47:21.199 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000f325da8aa7c9d71-0065bb76b9-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:21.240 DEBUG:boto:path=/vclfcn-59/obj4 2024-02-01T10:47:21.240 DEBUG:boto:auth_path=/vclfcn-59/obj4 2024-02-01T10:47:21.240 DEBUG:boto:path=/vclfcn-59/obj4?versionId=null 2024-02-01T10:47:21.240 DEBUG:boto:auth_path=/vclfcn-59/obj4?versionId=null 2024-02-01T10:47:21.241 DEBUG:boto:Method: GET 2024-02-01T10:47:21.241 DEBUG:boto:Path: /vclfcn-59/obj4?versionId=null 2024-02-01T10:47:21.241 DEBUG:boto:Data: 2024-02-01T10:47:21.241 DEBUG:boto:Headers: {} 2024-02-01T10:47:21.241 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:21.241 DEBUG:boto:Port: 8000 2024-02-01T10:47:21.241 DEBUG:boto:Params: {} 2024-02-01T10:47:21.241 DEBUG:boto:Token: None 2024-02-01T10:47:21.241 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:21 GMT /vclfcn-59/obj4?versionId=null 2024-02-01T10:47:21.242 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZIqoyLuJJ94cL1ljI05AvbMeivw= 2024-02-01T10:47:21.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZIqoyLuJJ94cL1ljI05AvbMeivw=', 'Content-Length': '0'} 2024-02-01T10:47:21.245 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000da3a906b0cd7f04-0065bb76b9-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:21.288 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:47:21.288 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:47:21.289 DEBUG:boto:path=/vclfcn-59/?prefix=obj4 2024-02-01T10:47:21.289 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj4 2024-02-01T10:47:21.289 DEBUG:boto:Method: GET 2024-02-01T10:47:21.289 DEBUG:boto:Path: /vclfcn-59/?prefix=obj4 2024-02-01T10:47:21.289 DEBUG:boto:Data: 2024-02-01T10:47:21.289 DEBUG:boto:Headers: {} 2024-02-01T10:47:21.289 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:21.290 DEBUG:boto:Port: 8001 2024-02-01T10:47:21.290 DEBUG:boto:Params: {} 2024-02-01T10:47:21.290 DEBUG:boto:Token: None 2024-02-01T10:47:21.290 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:21 GMT /vclfcn-59/ 2024-02-01T10:47:21.290 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:eUriIJ80xTPndDEhiholfVP1W/Y= 2024-02-01T10:47:21.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:eUriIJ80xTPndDEhiholfVP1W/Y=', 'Content-Length': '0'} 2024-02-01T10:47:21.295 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf57e7fccb225797-0065bb76b9-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:21.336 DEBUG:boto:b'vclfcn-59obj41000falseobj42024-02-01T10:46:46.190Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:21.337 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:47:21.337 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:47:21.337 DEBUG:boto:path=/vclfcn-59/?prefix=obj4 2024-02-01T10:47:21.337 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj4 2024-02-01T10:47:21.337 DEBUG:boto:Method: GET 2024-02-01T10:47:21.337 DEBUG:boto:Path: /vclfcn-59/?prefix=obj4 2024-02-01T10:47:21.337 DEBUG:boto:Data: 2024-02-01T10:47:21.337 DEBUG:boto:Headers: {} 2024-02-01T10:47:21.337 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:21.338 DEBUG:boto:Port: 8000 2024-02-01T10:47:21.338 DEBUG:boto:Params: {} 2024-02-01T10:47:21.338 DEBUG:boto:Token: None 2024-02-01T10:47:21.338 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:21 GMT /vclfcn-59/ 2024-02-01T10:47:21.338 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:eUriIJ80xTPndDEhiholfVP1W/Y= 2024-02-01T10:47:21.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:eUriIJ80xTPndDEhiholfVP1W/Y=', 'Content-Length': '0'} 2024-02-01T10:47:21.343 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004589ade80dcb4afc-0065bb76b9-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:21.343 DEBUG:boto:b'vclfcn-59obj41000falseobj42024-02-01T10:46:46.190Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:21.343 DEBUG:boto:path=/vclfcn-59/obj4 2024-02-01T10:47:21.343 DEBUG:boto:auth_path=/vclfcn-59/obj4 2024-02-01T10:47:21.344 DEBUG:boto:Method: HEAD 2024-02-01T10:47:21.344 DEBUG:boto:Path: /vclfcn-59/obj4 2024-02-01T10:47:21.344 DEBUG:boto:Data: 2024-02-01T10:47:21.344 DEBUG:boto:Headers: {} 2024-02-01T10:47:21.344 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:21.344 DEBUG:boto:Port: 8001 2024-02-01T10:47:21.344 DEBUG:boto:Params: {} 2024-02-01T10:47:21.344 DEBUG:boto:Token: None 2024-02-01T10:47:21.344 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:21 GMT /vclfcn-59/obj4 2024-02-01T10:47:21.345 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:bZJfJEjotMNK+YOGMRBYH8x3REE= 2024-02-01T10:47:21.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:bZJfJEjotMNK+YOGMRBYH8x3REE=', 'Content-Length': '0'} 2024-02-01T10:47:21.347 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000935476e4cf43efdc-0065bb76b9-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:21.348 DEBUG:boto:path=/vclfcn-59/obj4 2024-02-01T10:47:21.348 DEBUG:boto:auth_path=/vclfcn-59/obj4 2024-02-01T10:47:21.348 DEBUG:boto:Method: HEAD 2024-02-01T10:47:21.348 DEBUG:boto:Path: /vclfcn-59/obj4 2024-02-01T10:47:21.348 DEBUG:boto:Data: 2024-02-01T10:47:21.348 DEBUG:boto:Headers: {} 2024-02-01T10:47:21.348 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:21.348 DEBUG:boto:Port: 8000 2024-02-01T10:47:21.349 DEBUG:boto:Params: {} 2024-02-01T10:47:21.349 DEBUG:boto:Token: None 2024-02-01T10:47:21.349 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:21 GMT /vclfcn-59/obj4 2024-02-01T10:47:21.349 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:bZJfJEjotMNK+YOGMRBYH8x3REE= 2024-02-01T10:47:21.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:bZJfJEjotMNK+YOGMRBYH8x3REE=', 'Content-Length': '0'} 2024-02-01T10:47:21.352 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000098f7fca81dfbaaa8-0065bb76b9-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:21.352 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:47:21.352 DEBUG:boto:path=/vclfcn-59/obj4 2024-02-01T10:47:21.353 DEBUG:boto:auth_path=/vclfcn-59/obj4 2024-02-01T10:47:21.353 DEBUG:boto:Method: GET 2024-02-01T10:47:21.353 DEBUG:boto:Path: /vclfcn-59/obj4 2024-02-01T10:47:21.353 DEBUG:boto:Data: 2024-02-01T10:47:21.353 DEBUG:boto:Headers: {} 2024-02-01T10:47:21.353 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:21.353 DEBUG:boto:Port: 8001 2024-02-01T10:47:21.353 DEBUG:boto:Params: {} 2024-02-01T10:47:21.353 DEBUG:boto:Token: None 2024-02-01T10:47:21.353 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:21 GMT /vclfcn-59/obj4 2024-02-01T10:47:21.354 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Vyh7dvrYmibtj5PYu1Cz2qFsmdw= 2024-02-01T10:47:21.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Vyh7dvrYmibtj5PYu1Cz2qFsmdw=', 'Content-Length': '0'} 2024-02-01T10:47:21.356 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000025cb3341e7f7ff7e-0065bb76b9-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:21.400 DEBUG:boto:path=/vclfcn-59/obj4 2024-02-01T10:47:21.400 DEBUG:boto:auth_path=/vclfcn-59/obj4 2024-02-01T10:47:21.400 DEBUG:boto:Method: GET 2024-02-01T10:47:21.401 DEBUG:boto:Path: /vclfcn-59/obj4 2024-02-01T10:47:21.401 DEBUG:boto:Data: 2024-02-01T10:47:21.401 DEBUG:boto:Headers: {} 2024-02-01T10:47:21.401 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:21.401 DEBUG:boto:Port: 8000 2024-02-01T10:47:21.401 DEBUG:boto:Params: {} 2024-02-01T10:47:21.401 DEBUG:boto:Token: None 2024-02-01T10:47:21.401 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:21 GMT /vclfcn-59/obj4 2024-02-01T10:47:21.401 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Vyh7dvrYmibtj5PYu1Cz2qFsmdw= 2024-02-01T10:47:21.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Vyh7dvrYmibtj5PYu1Cz2qFsmdw=', 'Content-Length': '0'} 2024-02-01T10:47:21.404 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001a9d00b488beee9f-0065bb76b9-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:21.452 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:47:21.452 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:47:21.452 DEBUG:boto:path=/vclfcn-59/?prefix=obj4 2024-02-01T10:47:21.452 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj4 2024-02-01T10:47:21.453 DEBUG:boto:Method: GET 2024-02-01T10:47:21.453 DEBUG:boto:Path: /vclfcn-59/?prefix=obj4 2024-02-01T10:47:21.453 DEBUG:boto:Data: 2024-02-01T10:47:21.453 DEBUG:boto:Headers: {} 2024-02-01T10:47:21.453 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:21.453 DEBUG:boto:Port: 8001 2024-02-01T10:47:21.453 DEBUG:boto:Params: {} 2024-02-01T10:47:21.453 DEBUG:boto:Token: None 2024-02-01T10:47:21.453 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:21 GMT /vclfcn-59/ 2024-02-01T10:47:21.454 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:eUriIJ80xTPndDEhiholfVP1W/Y= 2024-02-01T10:47:21.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:eUriIJ80xTPndDEhiholfVP1W/Y=', 'Content-Length': '0'} 2024-02-01T10:47:21.458 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009eec1a09c1951372-0065bb76b9-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:21.458 DEBUG:boto:b'vclfcn-59obj41000falseobj42024-02-01T10:46:46.190Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:21.459 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:47:21.459 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:47:21.459 DEBUG:boto:path=/vclfcn-59/?prefix=obj4 2024-02-01T10:47:21.459 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj4 2024-02-01T10:47:21.459 DEBUG:boto:Method: GET 2024-02-01T10:47:21.459 DEBUG:boto:Path: /vclfcn-59/?prefix=obj4 2024-02-01T10:47:21.459 DEBUG:boto:Data: 2024-02-01T10:47:21.459 DEBUG:boto:Headers: {} 2024-02-01T10:47:21.460 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:21.460 DEBUG:boto:Port: 8000 2024-02-01T10:47:21.460 DEBUG:boto:Params: {} 2024-02-01T10:47:21.460 DEBUG:boto:Token: None 2024-02-01T10:47:21.460 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:21 GMT /vclfcn-59/ 2024-02-01T10:47:21.460 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:eUriIJ80xTPndDEhiholfVP1W/Y= 2024-02-01T10:47:21.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:eUriIJ80xTPndDEhiholfVP1W/Y=', 'Content-Length': '0'} 2024-02-01T10:47:21.464 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019227c0769700c19-0065bb76b9-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:21.465 DEBUG:boto:b'vclfcn-59obj41000falseobj42024-02-01T10:46:46.190Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:21.465 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-59 zones={test-zone3, test-zone2} 2024-02-01T10:47:21.465 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=vclfcn-60 2024-02-01T10:47:21.465 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:47:21.465 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:47:21.466 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket vclfcn-60 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-02-01T10:47:21.516 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:47:21.516 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:47:21.583 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:47:21.582+0000 7f5df688c980 1 bucket sync caught up with source: 2024-02-01T10:47:21.584 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [00000000001.61.6, , 00000000001.62.6, , 00000000001.51.6, , , 00000000001.42.6, , , ] 2024-02-01T10:47:21.584 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [00000000001.61.6, , 00000000001.62.6, , 00000000001.51.6, , , 00000000001.42.6, , , ] 2024-02-01T10:47:21.584 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:47:21.582+0000 7f5df688c980 0 bucket checkpoint complete 2024-02-01T10:47:21.608 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=vclfcn-60 2024-02-01T10:47:21.608 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:47:21.608 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:47:21.609 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-60 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:47:21.658 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:47:21.659 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:47:21.719 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:47:21.722+0000 7f34174e4980 1 bucket sync caught up with source: 2024-02-01T10:47:21.719 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [00000000001.61.6, , 00000000001.62.6, , 00000000001.51.6, , , 00000000001.42.6, , , ] 2024-02-01T10:47:21.719 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [00000000001.61.6, , 00000000001.62.6, , 00000000001.51.6, , , 00000000001.42.6, , , ] 2024-02-01T10:47:21.720 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:47:21.722+0000 7f34174e4980 0 bucket checkpoint complete 2024-02-01T10:47:21.742 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=vclfcn-60 2024-02-01T10:47:21.742 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-60', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:47:21.742 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-60', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:47:21.742 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-60 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:47:21.785 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:47:21.786 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:47:21.846 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:47:21.846+0000 7ffb51d56980 1 bucket sync caught up with source: 2024-02-01T10:47:21.847 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [00000000001.285.6, , 00000000001.225.6, , 00000000001.302.6, , , 00000000001.180.6, , , ] 2024-02-01T10:47:21.847 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [00000000001.285.6, , 00000000001.225.6, , 00000000001.302.6, , , 00000000001.180.6, , , ] 2024-02-01T10:47:21.847 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:47:21.846+0000 7ffb51d56980 0 bucket checkpoint complete 2024-02-01T10:47:21.869 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=vclfcn-60 2024-02-01T10:47:21.869 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-60', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:47:21.869 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-60', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:47:21.869 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-60 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:47:21.957 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:47:21.957 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:47:22.017 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:47:22.018+0000 7fbd3f078980 1 bucket sync caught up with source: 2024-02-01T10:47:22.017 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [00000000001.285.6, , 00000000001.225.6, , 00000000001.302.6, , , 00000000001.180.6, , , ] 2024-02-01T10:47:22.017 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [00000000001.285.6, , 00000000001.225.6, , 00000000001.302.6, , , 00000000001.180.6, , , ] 2024-02-01T10:47:22.017 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:47:22.018+0000 7fbd3f078980 0 bucket checkpoint complete 2024-02-01T10:47:22.039 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=vclfcn-60 2024-02-01T10:47:22.039 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-60', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:47:22.040 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-60', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:47:22.040 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-60 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:47:22.080 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:47:22.080 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:47:22.137 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:47:22.138+0000 7fa8200eb980 1 bucket sync caught up with source: 2024-02-01T10:47:22.137 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [00000000001.178.6, , 00000000001.203.6, , 00000000001.272.6, , , 00000000001.168.6, , , ] 2024-02-01T10:47:22.137 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [00000000001.178.6, , 00000000001.203.6, , 00000000001.272.6, , , 00000000001.168.6, , , ] 2024-02-01T10:47:22.137 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:47:22.138+0000 7fa8200eb980 0 bucket checkpoint complete 2024-02-01T10:47:22.161 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=vclfcn-60 2024-02-01T10:47:22.161 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-60', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:47:22.161 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-60', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:47:22.162 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket vclfcn-60 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-02-01T10:47:22.211 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:47:22.211 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:47:22.275 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:47:22.274+0000 7f0049c6c980 1 bucket sync caught up with source: 2024-02-01T10:47:22.276 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [00000000001.178.6, , 00000000001.203.6, , 00000000001.272.6, , , 00000000001.168.6, , , ] 2024-02-01T10:47:22.276 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [00000000001.178.6, , 00000000001.203.6, , 00000000001.272.6, , , 00000000001.168.6, , , ] 2024-02-01T10:47:22.276 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:47:22.274+0000 7f0049c6c980 0 bucket checkpoint complete 2024-02-01T10:47:22.299 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-60 zones={test-zone2, test-zone1} 2024-02-01T10:47:22.299 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:47:22.299 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:47:22.300 DEBUG:boto:Method: HEAD 2024-02-01T10:47:22.300 DEBUG:boto:Path: /vclfcn-60/ 2024-02-01T10:47:22.300 DEBUG:boto:Data: 2024-02-01T10:47:22.300 DEBUG:boto:Headers: {} 2024-02-01T10:47:22.300 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:22.300 DEBUG:boto:Port: 8000 2024-02-01T10:47:22.300 DEBUG:boto:Params: {} 2024-02-01T10:47:22.300 DEBUG:boto:Token: None 2024-02-01T10:47:22.300 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:22 GMT /vclfcn-60/ 2024-02-01T10:47:22.301 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:U3EWRGjfsL76Jr++Fdl1RsJKNRQ= 2024-02-01T10:47:22.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:U3EWRGjfsL76Jr++Fdl1RsJKNRQ=', 'Content-Length': '0'} 2024-02-01T10:47:22.303 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a5c96b857e411c55-0065bb76ba-5350-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:47:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:22.303 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:47:22.303 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:47:22.303 DEBUG:boto:Method: HEAD 2024-02-01T10:47:22.303 DEBUG:boto:Path: /vclfcn-60/ 2024-02-01T10:47:22.303 DEBUG:boto:Data: 2024-02-01T10:47:22.304 DEBUG:boto:Headers: {} 2024-02-01T10:47:22.304 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:22.304 DEBUG:boto:Port: 8000 2024-02-01T10:47:22.304 DEBUG:boto:Params: {} 2024-02-01T10:47:22.304 DEBUG:boto:Token: None 2024-02-01T10:47:22.304 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:22 GMT /vclfcn-60/ 2024-02-01T10:47:22.304 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:U3EWRGjfsL76Jr++Fdl1RsJKNRQ= 2024-02-01T10:47:22.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:U3EWRGjfsL76Jr++Fdl1RsJKNRQ=', 'Content-Length': '0'} 2024-02-01T10:47:22.308 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000065f9554810b82fa7-0065bb76ba-5765-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:47:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:22.308 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:47:22.309 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:47:22.309 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:47:22.309 DEBUG:boto:path=/vclfcn-60/?versions&max-keys=999 2024-02-01T10:47:22.309 DEBUG:boto:auth_path=/vclfcn-60/?versions&max-keys=999 2024-02-01T10:47:22.309 DEBUG:boto:Method: GET 2024-02-01T10:47:22.309 DEBUG:boto:Path: /vclfcn-60/?versions&max-keys=999 2024-02-01T10:47:22.309 DEBUG:boto:Data: 2024-02-01T10:47:22.309 DEBUG:boto:Headers: {} 2024-02-01T10:47:22.309 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:22.309 DEBUG:boto:Port: 8000 2024-02-01T10:47:22.310 DEBUG:boto:Params: {} 2024-02-01T10:47:22.310 DEBUG:boto:Token: None 2024-02-01T10:47:22.310 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:22 GMT /vclfcn-60/?versions 2024-02-01T10:47:22.310 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:2TOtBwphTjw05x2NMyQAyNkhJCc= 2024-02-01T10:47:22.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2TOtBwphTjw05x2NMyQAyNkhJCc=', 'Content-Length': '0'} 2024-02-01T10:47:22.312 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2028a69a193b8f5-0065bb76ba-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:22.312 DEBUG:boto:b'vclfcn-60999falseobj1nulltrue2024-02-01T10:46:46.243Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:46:46.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:46:46.285Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:46:46.336Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:22.313 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-02-01T10:47:22.313 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-02-01T10:47:22.313 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-02-01T10:47:22.313 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-02-01T10:47:22.313 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:47:22.313 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:47:22.314 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:47:22.314 DEBUG:boto:path=/vclfcn-60/?versions&max-keys=999 2024-02-01T10:47:22.314 DEBUG:boto:auth_path=/vclfcn-60/?versions&max-keys=999 2024-02-01T10:47:22.314 DEBUG:boto:Method: GET 2024-02-01T10:47:22.314 DEBUG:boto:Path: /vclfcn-60/?versions&max-keys=999 2024-02-01T10:47:22.314 DEBUG:boto:Data: 2024-02-01T10:47:22.314 DEBUG:boto:Headers: {} 2024-02-01T10:47:22.314 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:22.314 DEBUG:boto:Port: 8000 2024-02-01T10:47:22.314 DEBUG:boto:Params: {} 2024-02-01T10:47:22.314 DEBUG:boto:Token: None 2024-02-01T10:47:22.315 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:22 GMT /vclfcn-60/?versions 2024-02-01T10:47:22.315 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:2TOtBwphTjw05x2NMyQAyNkhJCc= 2024-02-01T10:47:22.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2TOtBwphTjw05x2NMyQAyNkhJCc=', 'Content-Length': '0'} 2024-02-01T10:47:22.319 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad969555bbf4aa75-0065bb76ba-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:22.319 DEBUG:boto:b'vclfcn-60999falseobj1nulltrue2024-02-01T10:46:46.243Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:46:46.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:46:46.285Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:46:46.336Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:22.320 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-02-01T10:47:22.320 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-02-01T10:47:22.320 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-02-01T10:47:22.320 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-02-01T10:47:22.320 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:47:22.320 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:47:22.320 DEBUG:boto:path=/vclfcn-60/?versions&max-keys=999 2024-02-01T10:47:22.320 DEBUG:boto:auth_path=/vclfcn-60/?versions&max-keys=999 2024-02-01T10:47:22.321 DEBUG:boto:Method: GET 2024-02-01T10:47:22.321 DEBUG:boto:Path: /vclfcn-60/?versions&max-keys=999 2024-02-01T10:47:22.321 DEBUG:boto:Data: 2024-02-01T10:47:22.321 DEBUG:boto:Headers: {} 2024-02-01T10:47:22.321 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:22.321 DEBUG:boto:Port: 8000 2024-02-01T10:47:22.321 DEBUG:boto:Params: {} 2024-02-01T10:47:22.321 DEBUG:boto:Token: None 2024-02-01T10:47:22.321 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:22 GMT /vclfcn-60/?versions 2024-02-01T10:47:22.321 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:2TOtBwphTjw05x2NMyQAyNkhJCc= 2024-02-01T10:47:22.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2TOtBwphTjw05x2NMyQAyNkhJCc=', 'Content-Length': '0'} 2024-02-01T10:47:22.324 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b67616df2be239d-0065bb76ba-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:22.324 DEBUG:boto:b'vclfcn-60999falseobj1nulltrue2024-02-01T10:46:46.243Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:46:46.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:46:46.285Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:46:46.336Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:22.324 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:47:22.324 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:47:22.324 DEBUG:boto:path=/vclfcn-60/?versions&max-keys=999 2024-02-01T10:47:22.324 DEBUG:boto:auth_path=/vclfcn-60/?versions&max-keys=999 2024-02-01T10:47:22.325 DEBUG:boto:Method: GET 2024-02-01T10:47:22.325 DEBUG:boto:Path: /vclfcn-60/?versions&max-keys=999 2024-02-01T10:47:22.325 DEBUG:boto:Data: 2024-02-01T10:47:22.325 DEBUG:boto:Headers: {} 2024-02-01T10:47:22.325 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:22.325 DEBUG:boto:Port: 8000 2024-02-01T10:47:22.325 DEBUG:boto:Params: {} 2024-02-01T10:47:22.325 DEBUG:boto:Token: None 2024-02-01T10:47:22.325 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:22 GMT /vclfcn-60/?versions 2024-02-01T10:47:22.325 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:2TOtBwphTjw05x2NMyQAyNkhJCc= 2024-02-01T10:47:22.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2TOtBwphTjw05x2NMyQAyNkhJCc=', 'Content-Length': '0'} 2024-02-01T10:47:22.330 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000202de36448a08b35-0065bb76ba-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:22.330 DEBUG:boto:b'vclfcn-60999falseobj1nulltrue2024-02-01T10:46:46.243Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:46:46.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:46:46.285Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:46:46.336Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:22.331 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:47:22.331 DEBUG:boto:path=/vclfcn-60/obj1 2024-02-01T10:47:22.331 DEBUG:boto:auth_path=/vclfcn-60/obj1 2024-02-01T10:47:22.331 DEBUG:boto:path=/vclfcn-60/obj1?versionId=null 2024-02-01T10:47:22.331 DEBUG:boto:auth_path=/vclfcn-60/obj1?versionId=null 2024-02-01T10:47:22.331 DEBUG:boto:Method: GET 2024-02-01T10:47:22.331 DEBUG:boto:Path: /vclfcn-60/obj1?versionId=null 2024-02-01T10:47:22.331 DEBUG:boto:Data: 2024-02-01T10:47:22.332 DEBUG:boto:Headers: {} 2024-02-01T10:47:22.332 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:22.332 DEBUG:boto:Port: 8000 2024-02-01T10:47:22.332 DEBUG:boto:Params: {} 2024-02-01T10:47:22.332 DEBUG:boto:Token: None 2024-02-01T10:47:22.332 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:22 GMT /vclfcn-60/obj1?versionId=null 2024-02-01T10:47:22.332 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:kwWEJJhZuPUF9MRwBB6wL/chIic= 2024-02-01T10:47:22.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:kwWEJJhZuPUF9MRwBB6wL/chIic=', 'Content-Length': '0'} 2024-02-01T10:47:22.334 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b5743b50edf65fba-0065bb76ba-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:22.376 DEBUG:boto:path=/vclfcn-60/obj1 2024-02-01T10:47:22.376 DEBUG:boto:auth_path=/vclfcn-60/obj1 2024-02-01T10:47:22.376 DEBUG:boto:path=/vclfcn-60/obj1?versionId=null 2024-02-01T10:47:22.376 DEBUG:boto:auth_path=/vclfcn-60/obj1?versionId=null 2024-02-01T10:47:22.376 DEBUG:boto:Method: GET 2024-02-01T10:47:22.377 DEBUG:boto:Path: /vclfcn-60/obj1?versionId=null 2024-02-01T10:47:22.377 DEBUG:boto:Data: 2024-02-01T10:47:22.377 DEBUG:boto:Headers: {} 2024-02-01T10:47:22.377 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:22.377 DEBUG:boto:Port: 8000 2024-02-01T10:47:22.377 DEBUG:boto:Params: {} 2024-02-01T10:47:22.377 DEBUG:boto:Token: None 2024-02-01T10:47:22.377 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:22 GMT /vclfcn-60/obj1?versionId=null 2024-02-01T10:47:22.377 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:kwWEJJhZuPUF9MRwBB6wL/chIic= 2024-02-01T10:47:22.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:kwWEJJhZuPUF9MRwBB6wL/chIic=', 'Content-Length': '0'} 2024-02-01T10:47:22.381 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f61b65bfa9007814-0065bb76ba-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:22.381 DEBUG:boto:path=/vclfcn-60/obj1 2024-02-01T10:47:22.381 DEBUG:boto:auth_path=/vclfcn-60/obj1 2024-02-01T10:47:22.381 DEBUG:boto:path=/vclfcn-60/obj1?versionId=null 2024-02-01T10:47:22.381 DEBUG:boto:auth_path=/vclfcn-60/obj1?versionId=null 2024-02-01T10:47:22.381 DEBUG:boto:Method: HEAD 2024-02-01T10:47:22.381 DEBUG:boto:Path: /vclfcn-60/obj1?versionId=null 2024-02-01T10:47:22.381 DEBUG:boto:Data: 2024-02-01T10:47:22.382 DEBUG:boto:Headers: {} 2024-02-01T10:47:22.382 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:22.382 DEBUG:boto:Port: 8000 2024-02-01T10:47:22.382 DEBUG:boto:Params: {} 2024-02-01T10:47:22.382 DEBUG:boto:Token: None 2024-02-01T10:47:22.382 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:22 GMT /vclfcn-60/obj1?versionId=null 2024-02-01T10:47:22.382 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tY9z+vxcuS2MWvY6IAMCDXRexvc= 2024-02-01T10:47:22.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tY9z+vxcuS2MWvY6IAMCDXRexvc=', 'Content-Length': '0'} 2024-02-01T10:47:22.385 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000558b92efa68a6612-0065bb76ba-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:22.385 DEBUG:boto:path=/vclfcn-60/obj1 2024-02-01T10:47:22.385 DEBUG:boto:auth_path=/vclfcn-60/obj1 2024-02-01T10:47:22.385 DEBUG:boto:path=/vclfcn-60/obj1?versionId=null 2024-02-01T10:47:22.385 DEBUG:boto:auth_path=/vclfcn-60/obj1?versionId=null 2024-02-01T10:47:22.386 DEBUG:boto:Method: HEAD 2024-02-01T10:47:22.386 DEBUG:boto:Path: /vclfcn-60/obj1?versionId=null 2024-02-01T10:47:22.386 DEBUG:boto:Data: 2024-02-01T10:47:22.386 DEBUG:boto:Headers: {} 2024-02-01T10:47:22.386 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:22.386 DEBUG:boto:Port: 8000 2024-02-01T10:47:22.386 DEBUG:boto:Params: {} 2024-02-01T10:47:22.386 DEBUG:boto:Token: None 2024-02-01T10:47:22.386 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:22 GMT /vclfcn-60/obj1?versionId=null 2024-02-01T10:47:22.386 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tY9z+vxcuS2MWvY6IAMCDXRexvc= 2024-02-01T10:47:22.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tY9z+vxcuS2MWvY6IAMCDXRexvc=', 'Content-Length': '0'} 2024-02-01T10:47:22.389 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000158c5a5f8cd155db-0065bb76ba-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:22.389 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:47:22.390 DEBUG:boto:path=/vclfcn-60/obj1 2024-02-01T10:47:22.390 DEBUG:boto:auth_path=/vclfcn-60/obj1 2024-02-01T10:47:22.390 DEBUG:boto:path=/vclfcn-60/obj1?versionId=null 2024-02-01T10:47:22.390 DEBUG:boto:auth_path=/vclfcn-60/obj1?versionId=null 2024-02-01T10:47:22.390 DEBUG:boto:Method: GET 2024-02-01T10:47:22.390 DEBUG:boto:Path: /vclfcn-60/obj1?versionId=null 2024-02-01T10:47:22.390 DEBUG:boto:Data: 2024-02-01T10:47:22.390 DEBUG:boto:Headers: {} 2024-02-01T10:47:22.390 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:22.390 DEBUG:boto:Port: 8000 2024-02-01T10:47:22.390 DEBUG:boto:Params: {} 2024-02-01T10:47:22.391 DEBUG:boto:Token: None 2024-02-01T10:47:22.391 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:22 GMT /vclfcn-60/obj1?versionId=null 2024-02-01T10:47:22.391 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:kwWEJJhZuPUF9MRwBB6wL/chIic= 2024-02-01T10:47:22.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:kwWEJJhZuPUF9MRwBB6wL/chIic=', 'Content-Length': '0'} 2024-02-01T10:47:22.393 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dc7ece07a01918f9-0065bb76ba-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:22.394 DEBUG:boto:path=/vclfcn-60/obj1 2024-02-01T10:47:22.394 DEBUG:boto:auth_path=/vclfcn-60/obj1 2024-02-01T10:47:22.394 DEBUG:boto:path=/vclfcn-60/obj1?versionId=null 2024-02-01T10:47:22.394 DEBUG:boto:auth_path=/vclfcn-60/obj1?versionId=null 2024-02-01T10:47:22.394 DEBUG:boto:Method: GET 2024-02-01T10:47:22.394 DEBUG:boto:Path: /vclfcn-60/obj1?versionId=null 2024-02-01T10:47:22.394 DEBUG:boto:Data: 2024-02-01T10:47:22.394 DEBUG:boto:Headers: {} 2024-02-01T10:47:22.395 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:22.395 DEBUG:boto:Port: 8000 2024-02-01T10:47:22.395 DEBUG:boto:Params: {} 2024-02-01T10:47:22.395 DEBUG:boto:Token: None 2024-02-01T10:47:22.395 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:22 GMT /vclfcn-60/obj1?versionId=null 2024-02-01T10:47:22.395 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:kwWEJJhZuPUF9MRwBB6wL/chIic= 2024-02-01T10:47:22.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:kwWEJJhZuPUF9MRwBB6wL/chIic=', 'Content-Length': '0'} 2024-02-01T10:47:22.398 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f99de236f2695b13-0065bb76ba-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:22.398 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:47:22.398 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:47:22.398 DEBUG:boto:path=/vclfcn-60/?prefix=obj1 2024-02-01T10:47:22.399 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj1 2024-02-01T10:47:22.399 DEBUG:boto:Method: GET 2024-02-01T10:47:22.399 DEBUG:boto:Path: /vclfcn-60/?prefix=obj1 2024-02-01T10:47:22.399 DEBUG:boto:Data: 2024-02-01T10:47:22.399 DEBUG:boto:Headers: {} 2024-02-01T10:47:22.399 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:22.399 DEBUG:boto:Port: 8000 2024-02-01T10:47:22.399 DEBUG:boto:Params: {} 2024-02-01T10:47:22.399 DEBUG:boto:Token: None 2024-02-01T10:47:22.399 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:22 GMT /vclfcn-60/ 2024-02-01T10:47:22.400 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:f16MNT44IZtusTBMK4MXEu8gOBE= 2024-02-01T10:47:22.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:f16MNT44IZtusTBMK4MXEu8gOBE=', 'Content-Length': '0'} 2024-02-01T10:47:22.405 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe6d3b4270e9ffd5-0065bb76ba-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:22.448 DEBUG:boto:b'vclfcn-60obj11000falseobj12024-02-01T10:46:46.243Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:22.448 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:47:22.449 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:47:22.449 DEBUG:boto:path=/vclfcn-60/?prefix=obj1 2024-02-01T10:47:22.449 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj1 2024-02-01T10:47:22.449 DEBUG:boto:Method: GET 2024-02-01T10:47:22.449 DEBUG:boto:Path: /vclfcn-60/?prefix=obj1 2024-02-01T10:47:22.449 DEBUG:boto:Data: 2024-02-01T10:47:22.449 DEBUG:boto:Headers: {} 2024-02-01T10:47:22.449 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:22.449 DEBUG:boto:Port: 8000 2024-02-01T10:47:22.449 DEBUG:boto:Params: {} 2024-02-01T10:47:22.449 DEBUG:boto:Token: None 2024-02-01T10:47:22.450 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:22 GMT /vclfcn-60/ 2024-02-01T10:47:22.450 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:f16MNT44IZtusTBMK4MXEu8gOBE= 2024-02-01T10:47:22.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:f16MNT44IZtusTBMK4MXEu8gOBE=', 'Content-Length': '0'} 2024-02-01T10:47:22.455 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b90836a8b0732dc-0065bb76ba-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:22.455 DEBUG:boto:b'vclfcn-60obj11000falseobj12024-02-01T10:46:46.243Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:22.455 DEBUG:boto:path=/vclfcn-60/obj1 2024-02-01T10:47:22.455 DEBUG:boto:auth_path=/vclfcn-60/obj1 2024-02-01T10:47:22.456 DEBUG:boto:Method: HEAD 2024-02-01T10:47:22.456 DEBUG:boto:Path: /vclfcn-60/obj1 2024-02-01T10:47:22.456 DEBUG:boto:Data: 2024-02-01T10:47:22.456 DEBUG:boto:Headers: {} 2024-02-01T10:47:22.456 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:22.456 DEBUG:boto:Port: 8000 2024-02-01T10:47:22.456 DEBUG:boto:Params: {} 2024-02-01T10:47:22.456 DEBUG:boto:Token: None 2024-02-01T10:47:22.456 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:22 GMT /vclfcn-60/obj1 2024-02-01T10:47:22.457 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0I1N/oKa5KE5gOUmDoFpJHM9aOo= 2024-02-01T10:47:22.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0I1N/oKa5KE5gOUmDoFpJHM9aOo=', 'Content-Length': '0'} 2024-02-01T10:47:22.459 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aebc962203598c40-0065bb76ba-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:22.460 DEBUG:boto:path=/vclfcn-60/obj1 2024-02-01T10:47:22.460 DEBUG:boto:auth_path=/vclfcn-60/obj1 2024-02-01T10:47:22.460 DEBUG:boto:Method: HEAD 2024-02-01T10:47:22.460 DEBUG:boto:Path: /vclfcn-60/obj1 2024-02-01T10:47:22.460 DEBUG:boto:Data: 2024-02-01T10:47:22.460 DEBUG:boto:Headers: {} 2024-02-01T10:47:22.460 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:22.460 DEBUG:boto:Port: 8000 2024-02-01T10:47:22.460 DEBUG:boto:Params: {} 2024-02-01T10:47:22.460 DEBUG:boto:Token: None 2024-02-01T10:47:22.461 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:22 GMT /vclfcn-60/obj1 2024-02-01T10:47:22.461 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0I1N/oKa5KE5gOUmDoFpJHM9aOo= 2024-02-01T10:47:22.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0I1N/oKa5KE5gOUmDoFpJHM9aOo=', 'Content-Length': '0'} 2024-02-01T10:47:22.463 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fa95dc2c9f007f2e-0065bb76ba-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:22.464 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:47:22.464 DEBUG:boto:path=/vclfcn-60/obj1 2024-02-01T10:47:22.464 DEBUG:boto:auth_path=/vclfcn-60/obj1 2024-02-01T10:47:22.464 DEBUG:boto:Method: GET 2024-02-01T10:47:22.464 DEBUG:boto:Path: /vclfcn-60/obj1 2024-02-01T10:47:22.464 DEBUG:boto:Data: 2024-02-01T10:47:22.464 DEBUG:boto:Headers: {} 2024-02-01T10:47:22.464 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:22.465 DEBUG:boto:Port: 8000 2024-02-01T10:47:22.465 DEBUG:boto:Params: {} 2024-02-01T10:47:22.465 DEBUG:boto:Token: None 2024-02-01T10:47:22.465 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:22 GMT /vclfcn-60/obj1 2024-02-01T10:47:22.465 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:OZ9A4pYxGdBnLdVckFR3DQaotA8= 2024-02-01T10:47:22.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:OZ9A4pYxGdBnLdVckFR3DQaotA8=', 'Content-Length': '0'} 2024-02-01T10:47:22.468 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ae60ab7fdfb6d30e-0065bb76ba-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:22.468 DEBUG:boto:path=/vclfcn-60/obj1 2024-02-01T10:47:22.468 DEBUG:boto:auth_path=/vclfcn-60/obj1 2024-02-01T10:47:22.468 DEBUG:boto:Method: GET 2024-02-01T10:47:22.469 DEBUG:boto:Path: /vclfcn-60/obj1 2024-02-01T10:47:22.469 DEBUG:boto:Data: 2024-02-01T10:47:22.469 DEBUG:boto:Headers: {} 2024-02-01T10:47:22.469 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:22.469 DEBUG:boto:Port: 8000 2024-02-01T10:47:22.469 DEBUG:boto:Params: {} 2024-02-01T10:47:22.469 DEBUG:boto:Token: None 2024-02-01T10:47:22.469 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:22 GMT /vclfcn-60/obj1 2024-02-01T10:47:22.469 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:OZ9A4pYxGdBnLdVckFR3DQaotA8= 2024-02-01T10:47:22.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:OZ9A4pYxGdBnLdVckFR3DQaotA8=', 'Content-Length': '0'} 2024-02-01T10:47:22.472 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006bf7841b5e27f970-0065bb76ba-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:22.520 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:47:22.520 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:47:22.520 DEBUG:boto:path=/vclfcn-60/?prefix=obj1 2024-02-01T10:47:22.520 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj1 2024-02-01T10:47:22.520 DEBUG:boto:Method: GET 2024-02-01T10:47:22.521 DEBUG:boto:Path: /vclfcn-60/?prefix=obj1 2024-02-01T10:47:22.521 DEBUG:boto:Data: 2024-02-01T10:47:22.521 DEBUG:boto:Headers: {} 2024-02-01T10:47:22.521 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:22.521 DEBUG:boto:Port: 8000 2024-02-01T10:47:22.521 DEBUG:boto:Params: {} 2024-02-01T10:47:22.521 DEBUG:boto:Token: None 2024-02-01T10:47:22.521 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:22 GMT /vclfcn-60/ 2024-02-01T10:47:22.521 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:f16MNT44IZtusTBMK4MXEu8gOBE= 2024-02-01T10:47:22.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:f16MNT44IZtusTBMK4MXEu8gOBE=', 'Content-Length': '0'} 2024-02-01T10:47:22.526 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a49e103878004fe0-0065bb76ba-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:22.527 DEBUG:boto:b'vclfcn-60obj11000falseobj12024-02-01T10:46:46.243Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:22.527 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:47:22.527 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:47:22.527 DEBUG:boto:path=/vclfcn-60/?prefix=obj1 2024-02-01T10:47:22.527 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj1 2024-02-01T10:47:22.527 DEBUG:boto:Method: GET 2024-02-01T10:47:22.527 DEBUG:boto:Path: /vclfcn-60/?prefix=obj1 2024-02-01T10:47:22.528 DEBUG:boto:Data: 2024-02-01T10:47:22.528 DEBUG:boto:Headers: {} 2024-02-01T10:47:22.528 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:22.528 DEBUG:boto:Port: 8000 2024-02-01T10:47:22.528 DEBUG:boto:Params: {} 2024-02-01T10:47:22.528 DEBUG:boto:Token: None 2024-02-01T10:47:22.528 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:22 GMT /vclfcn-60/ 2024-02-01T10:47:22.528 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:f16MNT44IZtusTBMK4MXEu8gOBE= 2024-02-01T10:47:22.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:f16MNT44IZtusTBMK4MXEu8gOBE=', 'Content-Length': '0'} 2024-02-01T10:47:22.533 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b63ee3d3dcc61f68-0065bb76ba-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:22.533 DEBUG:boto:b'vclfcn-60obj11000falseobj12024-02-01T10:46:46.243Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:22.534 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:47:22.534 DEBUG:boto:path=/vclfcn-60/obj2 2024-02-01T10:47:22.534 DEBUG:boto:auth_path=/vclfcn-60/obj2 2024-02-01T10:47:22.534 DEBUG:boto:path=/vclfcn-60/obj2?versionId=null 2024-02-01T10:47:22.534 DEBUG:boto:auth_path=/vclfcn-60/obj2?versionId=null 2024-02-01T10:47:22.534 DEBUG:boto:Method: GET 2024-02-01T10:47:22.534 DEBUG:boto:Path: /vclfcn-60/obj2?versionId=null 2024-02-01T10:47:22.534 DEBUG:boto:Data: 2024-02-01T10:47:22.534 DEBUG:boto:Headers: {} 2024-02-01T10:47:22.534 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:22.535 DEBUG:boto:Port: 8000 2024-02-01T10:47:22.535 DEBUG:boto:Params: {} 2024-02-01T10:47:22.535 DEBUG:boto:Token: None 2024-02-01T10:47:22.535 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:22 GMT /vclfcn-60/obj2?versionId=null 2024-02-01T10:47:22.535 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+sRmdVg0UgEPp7jU8X4TS7YweWE= 2024-02-01T10:47:22.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+sRmdVg0UgEPp7jU8X4TS7YweWE=', 'Content-Length': '0'} 2024-02-01T10:47:22.538 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000094bd74e271961515-0065bb76ba-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:22.538 DEBUG:boto:path=/vclfcn-60/obj2 2024-02-01T10:47:22.539 DEBUG:boto:auth_path=/vclfcn-60/obj2 2024-02-01T10:47:22.539 DEBUG:boto:path=/vclfcn-60/obj2?versionId=null 2024-02-01T10:47:22.539 DEBUG:boto:auth_path=/vclfcn-60/obj2?versionId=null 2024-02-01T10:47:22.539 DEBUG:boto:Method: GET 2024-02-01T10:47:22.539 DEBUG:boto:Path: /vclfcn-60/obj2?versionId=null 2024-02-01T10:47:22.539 DEBUG:boto:Data: 2024-02-01T10:47:22.539 DEBUG:boto:Headers: {} 2024-02-01T10:47:22.539 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:22.539 DEBUG:boto:Port: 8000 2024-02-01T10:47:22.539 DEBUG:boto:Params: {} 2024-02-01T10:47:22.539 DEBUG:boto:Token: None 2024-02-01T10:47:22.540 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:22 GMT /vclfcn-60/obj2?versionId=null 2024-02-01T10:47:22.540 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+sRmdVg0UgEPp7jU8X4TS7YweWE= 2024-02-01T10:47:22.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+sRmdVg0UgEPp7jU8X4TS7YweWE=', 'Content-Length': '0'} 2024-02-01T10:47:22.543 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000017841fb7686c6b5-0065bb76ba-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:22.543 DEBUG:boto:path=/vclfcn-60/obj2 2024-02-01T10:47:22.543 DEBUG:boto:auth_path=/vclfcn-60/obj2 2024-02-01T10:47:22.544 DEBUG:boto:path=/vclfcn-60/obj2?versionId=null 2024-02-01T10:47:22.544 DEBUG:boto:auth_path=/vclfcn-60/obj2?versionId=null 2024-02-01T10:47:22.544 DEBUG:boto:Method: HEAD 2024-02-01T10:47:22.544 DEBUG:boto:Path: /vclfcn-60/obj2?versionId=null 2024-02-01T10:47:22.544 DEBUG:boto:Data: 2024-02-01T10:47:22.544 DEBUG:boto:Headers: {} 2024-02-01T10:47:22.544 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:22.544 DEBUG:boto:Port: 8000 2024-02-01T10:47:22.544 DEBUG:boto:Params: {} 2024-02-01T10:47:22.544 DEBUG:boto:Token: None 2024-02-01T10:47:22.545 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:22 GMT /vclfcn-60/obj2?versionId=null 2024-02-01T10:47:22.545 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:JkNH6w8c3qYrARV4ZUdN91jcnVU= 2024-02-01T10:47:22.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:JkNH6w8c3qYrARV4ZUdN91jcnVU=', 'Content-Length': '0'} 2024-02-01T10:47:22.547 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007d4791d5565230f6-0065bb76ba-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:22.548 DEBUG:boto:path=/vclfcn-60/obj2 2024-02-01T10:47:22.548 DEBUG:boto:auth_path=/vclfcn-60/obj2 2024-02-01T10:47:22.548 DEBUG:boto:path=/vclfcn-60/obj2?versionId=null 2024-02-01T10:47:22.548 DEBUG:boto:auth_path=/vclfcn-60/obj2?versionId=null 2024-02-01T10:47:22.548 DEBUG:boto:Method: HEAD 2024-02-01T10:47:22.548 DEBUG:boto:Path: /vclfcn-60/obj2?versionId=null 2024-02-01T10:47:22.548 DEBUG:boto:Data: 2024-02-01T10:47:22.548 DEBUG:boto:Headers: {} 2024-02-01T10:47:22.548 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:22.549 DEBUG:boto:Port: 8000 2024-02-01T10:47:22.549 DEBUG:boto:Params: {} 2024-02-01T10:47:22.549 DEBUG:boto:Token: None 2024-02-01T10:47:22.549 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:22 GMT /vclfcn-60/obj2?versionId=null 2024-02-01T10:47:22.549 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:JkNH6w8c3qYrARV4ZUdN91jcnVU= 2024-02-01T10:47:22.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:JkNH6w8c3qYrARV4ZUdN91jcnVU=', 'Content-Length': '0'} 2024-02-01T10:47:22.552 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000397cd97b8161ce76-0065bb76ba-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:22.552 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:47:22.552 DEBUG:boto:path=/vclfcn-60/obj2 2024-02-01T10:47:22.552 DEBUG:boto:auth_path=/vclfcn-60/obj2 2024-02-01T10:47:22.552 DEBUG:boto:path=/vclfcn-60/obj2?versionId=null 2024-02-01T10:47:22.553 DEBUG:boto:auth_path=/vclfcn-60/obj2?versionId=null 2024-02-01T10:47:22.553 DEBUG:boto:Method: GET 2024-02-01T10:47:22.553 DEBUG:boto:Path: /vclfcn-60/obj2?versionId=null 2024-02-01T10:47:22.553 DEBUG:boto:Data: 2024-02-01T10:47:22.553 DEBUG:boto:Headers: {} 2024-02-01T10:47:22.553 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:22.553 DEBUG:boto:Port: 8000 2024-02-01T10:47:22.553 DEBUG:boto:Params: {} 2024-02-01T10:47:22.553 DEBUG:boto:Token: None 2024-02-01T10:47:22.554 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:22 GMT /vclfcn-60/obj2?versionId=null 2024-02-01T10:47:22.554 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+sRmdVg0UgEPp7jU8X4TS7YweWE= 2024-02-01T10:47:22.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+sRmdVg0UgEPp7jU8X4TS7YweWE=', 'Content-Length': '0'} 2024-02-01T10:47:22.556 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000043efe8096716f9e1-0065bb76ba-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:22.600 DEBUG:boto:path=/vclfcn-60/obj2 2024-02-01T10:47:22.600 DEBUG:boto:auth_path=/vclfcn-60/obj2 2024-02-01T10:47:22.600 DEBUG:boto:path=/vclfcn-60/obj2?versionId=null 2024-02-01T10:47:22.600 DEBUG:boto:auth_path=/vclfcn-60/obj2?versionId=null 2024-02-01T10:47:22.601 DEBUG:boto:Method: GET 2024-02-01T10:47:22.601 DEBUG:boto:Path: /vclfcn-60/obj2?versionId=null 2024-02-01T10:47:22.601 DEBUG:boto:Data: 2024-02-01T10:47:22.601 DEBUG:boto:Headers: {} 2024-02-01T10:47:22.601 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:22.601 DEBUG:boto:Port: 8000 2024-02-01T10:47:22.601 DEBUG:boto:Params: {} 2024-02-01T10:47:22.601 DEBUG:boto:Token: None 2024-02-01T10:47:22.601 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:22 GMT /vclfcn-60/obj2?versionId=null 2024-02-01T10:47:22.601 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+sRmdVg0UgEPp7jU8X4TS7YweWE= 2024-02-01T10:47:22.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+sRmdVg0UgEPp7jU8X4TS7YweWE=', 'Content-Length': '0'} 2024-02-01T10:47:22.604 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cfe1f8e639b30a34-0065bb76ba-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:22.605 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:47:22.605 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:47:22.605 DEBUG:boto:path=/vclfcn-60/?prefix=obj2 2024-02-01T10:47:22.605 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj2 2024-02-01T10:47:22.605 DEBUG:boto:Method: GET 2024-02-01T10:47:22.605 DEBUG:boto:Path: /vclfcn-60/?prefix=obj2 2024-02-01T10:47:22.605 DEBUG:boto:Data: 2024-02-01T10:47:22.605 DEBUG:boto:Headers: {} 2024-02-01T10:47:22.606 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:22.606 DEBUG:boto:Port: 8000 2024-02-01T10:47:22.606 DEBUG:boto:Params: {} 2024-02-01T10:47:22.606 DEBUG:boto:Token: None 2024-02-01T10:47:22.606 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:22 GMT /vclfcn-60/ 2024-02-01T10:47:22.606 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:f16MNT44IZtusTBMK4MXEu8gOBE= 2024-02-01T10:47:22.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:f16MNT44IZtusTBMK4MXEu8gOBE=', 'Content-Length': '0'} 2024-02-01T10:47:22.611 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f4406ca9123db086-0065bb76ba-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:22.611 DEBUG:boto:b'vclfcn-60obj21000falseobj22024-02-01T10:46:46.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:22.611 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:47:22.611 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:47:22.611 DEBUG:boto:path=/vclfcn-60/?prefix=obj2 2024-02-01T10:47:22.612 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj2 2024-02-01T10:47:22.612 DEBUG:boto:Method: GET 2024-02-01T10:47:22.612 DEBUG:boto:Path: /vclfcn-60/?prefix=obj2 2024-02-01T10:47:22.612 DEBUG:boto:Data: 2024-02-01T10:47:22.612 DEBUG:boto:Headers: {} 2024-02-01T10:47:22.612 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:22.612 DEBUG:boto:Port: 8000 2024-02-01T10:47:22.612 DEBUG:boto:Params: {} 2024-02-01T10:47:22.612 DEBUG:boto:Token: None 2024-02-01T10:47:22.612 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:22 GMT /vclfcn-60/ 2024-02-01T10:47:22.613 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:f16MNT44IZtusTBMK4MXEu8gOBE= 2024-02-01T10:47:22.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:f16MNT44IZtusTBMK4MXEu8gOBE=', 'Content-Length': '0'} 2024-02-01T10:47:22.617 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000044e75540cb7ff36a-0065bb76ba-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:22.660 DEBUG:boto:b'vclfcn-60obj21000falseobj22024-02-01T10:46:46.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:22.661 DEBUG:boto:path=/vclfcn-60/obj2 2024-02-01T10:47:22.661 DEBUG:boto:auth_path=/vclfcn-60/obj2 2024-02-01T10:47:22.661 DEBUG:boto:Method: HEAD 2024-02-01T10:47:22.661 DEBUG:boto:Path: /vclfcn-60/obj2 2024-02-01T10:47:22.661 DEBUG:boto:Data: 2024-02-01T10:47:22.661 DEBUG:boto:Headers: {} 2024-02-01T10:47:22.661 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:22.661 DEBUG:boto:Port: 8000 2024-02-01T10:47:22.662 DEBUG:boto:Params: {} 2024-02-01T10:47:22.662 DEBUG:boto:Token: None 2024-02-01T10:47:22.662 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:22 GMT /vclfcn-60/obj2 2024-02-01T10:47:22.662 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ztrzr25lVFaHaK84i9Txs7TWAbA= 2024-02-01T10:47:22.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ztrzr25lVFaHaK84i9Txs7TWAbA=', 'Content-Length': '0'} 2024-02-01T10:47:22.665 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000d7a96cda09b0f60-0065bb76ba-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:22.665 DEBUG:boto:path=/vclfcn-60/obj2 2024-02-01T10:47:22.665 DEBUG:boto:auth_path=/vclfcn-60/obj2 2024-02-01T10:47:22.666 DEBUG:boto:Method: HEAD 2024-02-01T10:47:22.666 DEBUG:boto:Path: /vclfcn-60/obj2 2024-02-01T10:47:22.666 DEBUG:boto:Data: 2024-02-01T10:47:22.666 DEBUG:boto:Headers: {} 2024-02-01T10:47:22.666 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:22.666 DEBUG:boto:Port: 8000 2024-02-01T10:47:22.666 DEBUG:boto:Params: {} 2024-02-01T10:47:22.666 DEBUG:boto:Token: None 2024-02-01T10:47:22.666 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:22 GMT /vclfcn-60/obj2 2024-02-01T10:47:22.666 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ztrzr25lVFaHaK84i9Txs7TWAbA= 2024-02-01T10:47:22.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ztrzr25lVFaHaK84i9Txs7TWAbA=', 'Content-Length': '0'} 2024-02-01T10:47:22.669 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b8c816e9cd93285e-0065bb76ba-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:22.670 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:47:22.670 DEBUG:boto:path=/vclfcn-60/obj2 2024-02-01T10:47:22.670 DEBUG:boto:auth_path=/vclfcn-60/obj2 2024-02-01T10:47:22.670 DEBUG:boto:Method: GET 2024-02-01T10:47:22.670 DEBUG:boto:Path: /vclfcn-60/obj2 2024-02-01T10:47:22.670 DEBUG:boto:Data: 2024-02-01T10:47:22.670 DEBUG:boto:Headers: {} 2024-02-01T10:47:22.670 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:22.671 DEBUG:boto:Port: 8000 2024-02-01T10:47:22.671 DEBUG:boto:Params: {} 2024-02-01T10:47:22.671 DEBUG:boto:Token: None 2024-02-01T10:47:22.671 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:22 GMT /vclfcn-60/obj2 2024-02-01T10:47:22.671 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:myamICh2v40HQ1wxf+Ddtgz4QwA= 2024-02-01T10:47:22.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:myamICh2v40HQ1wxf+Ddtgz4QwA=', 'Content-Length': '0'} 2024-02-01T10:47:22.674 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c7a023dfa6a91a5b-0065bb76ba-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:22.674 DEBUG:boto:path=/vclfcn-60/obj2 2024-02-01T10:47:22.674 DEBUG:boto:auth_path=/vclfcn-60/obj2 2024-02-01T10:47:22.674 DEBUG:boto:Method: GET 2024-02-01T10:47:22.674 DEBUG:boto:Path: /vclfcn-60/obj2 2024-02-01T10:47:22.675 DEBUG:boto:Data: 2024-02-01T10:47:22.675 DEBUG:boto:Headers: {} 2024-02-01T10:47:22.675 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:22.675 DEBUG:boto:Port: 8000 2024-02-01T10:47:22.675 DEBUG:boto:Params: {} 2024-02-01T10:47:22.675 DEBUG:boto:Token: None 2024-02-01T10:47:22.675 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:22 GMT /vclfcn-60/obj2 2024-02-01T10:47:22.675 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:myamICh2v40HQ1wxf+Ddtgz4QwA= 2024-02-01T10:47:22.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:myamICh2v40HQ1wxf+Ddtgz4QwA=', 'Content-Length': '0'} 2024-02-01T10:47:22.678 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000feb54e44677984b3-0065bb76ba-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:22.678 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:47:22.678 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:47:22.679 DEBUG:boto:path=/vclfcn-60/?prefix=obj2 2024-02-01T10:47:22.679 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj2 2024-02-01T10:47:22.679 DEBUG:boto:Method: GET 2024-02-01T10:47:22.679 DEBUG:boto:Path: /vclfcn-60/?prefix=obj2 2024-02-01T10:47:22.679 DEBUG:boto:Data: 2024-02-01T10:47:22.679 DEBUG:boto:Headers: {} 2024-02-01T10:47:22.679 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:22.679 DEBUG:boto:Port: 8000 2024-02-01T10:47:22.679 DEBUG:boto:Params: {} 2024-02-01T10:47:22.680 DEBUG:boto:Token: None 2024-02-01T10:47:22.680 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:22 GMT /vclfcn-60/ 2024-02-01T10:47:22.680 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:f16MNT44IZtusTBMK4MXEu8gOBE= 2024-02-01T10:47:22.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:f16MNT44IZtusTBMK4MXEu8gOBE=', 'Content-Length': '0'} 2024-02-01T10:47:22.685 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff8ebb9969459f24-0065bb76ba-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:22.732 DEBUG:boto:b'vclfcn-60obj21000falseobj22024-02-01T10:46:46.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:22.733 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:47:22.733 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:47:22.733 DEBUG:boto:path=/vclfcn-60/?prefix=obj2 2024-02-01T10:47:22.733 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj2 2024-02-01T10:47:22.733 DEBUG:boto:Method: GET 2024-02-01T10:47:22.733 DEBUG:boto:Path: /vclfcn-60/?prefix=obj2 2024-02-01T10:47:22.733 DEBUG:boto:Data: 2024-02-01T10:47:22.733 DEBUG:boto:Headers: {} 2024-02-01T10:47:22.733 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:22.733 DEBUG:boto:Port: 8000 2024-02-01T10:47:22.733 DEBUG:boto:Params: {} 2024-02-01T10:47:22.734 DEBUG:boto:Token: None 2024-02-01T10:47:22.734 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:22 GMT /vclfcn-60/ 2024-02-01T10:47:22.734 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:f16MNT44IZtusTBMK4MXEu8gOBE= 2024-02-01T10:47:22.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:f16MNT44IZtusTBMK4MXEu8gOBE=', 'Content-Length': '0'} 2024-02-01T10:47:22.739 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a70ce1bae13e67f2-0065bb76ba-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:22.739 DEBUG:boto:b'vclfcn-60obj21000falseobj22024-02-01T10:46:46.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:22.739 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:47:22.739 DEBUG:boto:path=/vclfcn-60/obj3 2024-02-01T10:47:22.740 DEBUG:boto:auth_path=/vclfcn-60/obj3 2024-02-01T10:47:22.740 DEBUG:boto:path=/vclfcn-60/obj3?versionId=null 2024-02-01T10:47:22.740 DEBUG:boto:auth_path=/vclfcn-60/obj3?versionId=null 2024-02-01T10:47:22.740 DEBUG:boto:Method: GET 2024-02-01T10:47:22.740 DEBUG:boto:Path: /vclfcn-60/obj3?versionId=null 2024-02-01T10:47:22.740 DEBUG:boto:Data: 2024-02-01T10:47:22.740 DEBUG:boto:Headers: {} 2024-02-01T10:47:22.741 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:22.741 DEBUG:boto:Port: 8000 2024-02-01T10:47:22.741 DEBUG:boto:Params: {} 2024-02-01T10:47:22.741 DEBUG:boto:Token: None 2024-02-01T10:47:22.741 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:22 GMT /vclfcn-60/obj3?versionId=null 2024-02-01T10:47:22.741 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ECVKOH0OLFI+5gPeFpO77HddNXU= 2024-02-01T10:47:22.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ECVKOH0OLFI+5gPeFpO77HddNXU=', 'Content-Length': '0'} 2024-02-01T10:47:22.744 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000039fd7cacfa134fc9-0065bb76ba-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:22.744 DEBUG:boto:path=/vclfcn-60/obj3 2024-02-01T10:47:22.744 DEBUG:boto:auth_path=/vclfcn-60/obj3 2024-02-01T10:47:22.744 DEBUG:boto:path=/vclfcn-60/obj3?versionId=null 2024-02-01T10:47:22.745 DEBUG:boto:auth_path=/vclfcn-60/obj3?versionId=null 2024-02-01T10:47:22.745 DEBUG:boto:Method: GET 2024-02-01T10:47:22.745 DEBUG:boto:Path: /vclfcn-60/obj3?versionId=null 2024-02-01T10:47:22.745 DEBUG:boto:Data: 2024-02-01T10:47:22.745 DEBUG:boto:Headers: {} 2024-02-01T10:47:22.745 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:22.745 DEBUG:boto:Port: 8000 2024-02-01T10:47:22.745 DEBUG:boto:Params: {} 2024-02-01T10:47:22.745 DEBUG:boto:Token: None 2024-02-01T10:47:22.745 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:22 GMT /vclfcn-60/obj3?versionId=null 2024-02-01T10:47:22.746 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ECVKOH0OLFI+5gPeFpO77HddNXU= 2024-02-01T10:47:22.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ECVKOH0OLFI+5gPeFpO77HddNXU=', 'Content-Length': '0'} 2024-02-01T10:47:22.748 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c8357fafbfd11514-0065bb76ba-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:22.792 DEBUG:boto:path=/vclfcn-60/obj3 2024-02-01T10:47:22.792 DEBUG:boto:auth_path=/vclfcn-60/obj3 2024-02-01T10:47:22.792 DEBUG:boto:path=/vclfcn-60/obj3?versionId=null 2024-02-01T10:47:22.792 DEBUG:boto:auth_path=/vclfcn-60/obj3?versionId=null 2024-02-01T10:47:22.793 DEBUG:boto:Method: HEAD 2024-02-01T10:47:22.793 DEBUG:boto:Path: /vclfcn-60/obj3?versionId=null 2024-02-01T10:47:22.793 DEBUG:boto:Data: 2024-02-01T10:47:22.793 DEBUG:boto:Headers: {} 2024-02-01T10:47:22.793 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:22.793 DEBUG:boto:Port: 8000 2024-02-01T10:47:22.793 DEBUG:boto:Params: {} 2024-02-01T10:47:22.793 DEBUG:boto:Token: None 2024-02-01T10:47:22.793 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:22 GMT /vclfcn-60/obj3?versionId=null 2024-02-01T10:47:22.794 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:iuonSsgl5NdX8oePoe2FioZfB8I= 2024-02-01T10:47:22.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iuonSsgl5NdX8oePoe2FioZfB8I=', 'Content-Length': '0'} 2024-02-01T10:47:22.796 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009602e218d68a8ba4-0065bb76ba-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:22.797 DEBUG:boto:path=/vclfcn-60/obj3 2024-02-01T10:47:22.797 DEBUG:boto:auth_path=/vclfcn-60/obj3 2024-02-01T10:47:22.797 DEBUG:boto:path=/vclfcn-60/obj3?versionId=null 2024-02-01T10:47:22.797 DEBUG:boto:auth_path=/vclfcn-60/obj3?versionId=null 2024-02-01T10:47:22.797 DEBUG:boto:Method: HEAD 2024-02-01T10:47:22.797 DEBUG:boto:Path: /vclfcn-60/obj3?versionId=null 2024-02-01T10:47:22.797 DEBUG:boto:Data: 2024-02-01T10:47:22.797 DEBUG:boto:Headers: {} 2024-02-01T10:47:22.797 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:22.797 DEBUG:boto:Port: 8000 2024-02-01T10:47:22.798 DEBUG:boto:Params: {} 2024-02-01T10:47:22.798 DEBUG:boto:Token: None 2024-02-01T10:47:22.798 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:22 GMT /vclfcn-60/obj3?versionId=null 2024-02-01T10:47:22.798 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:iuonSsgl5NdX8oePoe2FioZfB8I= 2024-02-01T10:47:22.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iuonSsgl5NdX8oePoe2FioZfB8I=', 'Content-Length': '0'} 2024-02-01T10:47:22.801 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b973c59d59998371-0065bb76ba-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:22.801 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:47:22.801 DEBUG:boto:path=/vclfcn-60/obj3 2024-02-01T10:47:22.801 DEBUG:boto:auth_path=/vclfcn-60/obj3 2024-02-01T10:47:22.801 DEBUG:boto:path=/vclfcn-60/obj3?versionId=null 2024-02-01T10:47:22.801 DEBUG:boto:auth_path=/vclfcn-60/obj3?versionId=null 2024-02-01T10:47:22.801 DEBUG:boto:Method: GET 2024-02-01T10:47:22.801 DEBUG:boto:Path: /vclfcn-60/obj3?versionId=null 2024-02-01T10:47:22.802 DEBUG:boto:Data: 2024-02-01T10:47:22.802 DEBUG:boto:Headers: {} 2024-02-01T10:47:22.802 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:22.802 DEBUG:boto:Port: 8000 2024-02-01T10:47:22.802 DEBUG:boto:Params: {} 2024-02-01T10:47:22.802 DEBUG:boto:Token: None 2024-02-01T10:47:22.802 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:22 GMT /vclfcn-60/obj3?versionId=null 2024-02-01T10:47:22.802 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ECVKOH0OLFI+5gPeFpO77HddNXU= 2024-02-01T10:47:22.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ECVKOH0OLFI+5gPeFpO77HddNXU=', 'Content-Length': '0'} 2024-02-01T10:47:22.805 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e7b2492e34ff7117-0065bb76ba-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:22.806 DEBUG:boto:path=/vclfcn-60/obj3 2024-02-01T10:47:22.806 DEBUG:boto:auth_path=/vclfcn-60/obj3 2024-02-01T10:47:22.806 DEBUG:boto:path=/vclfcn-60/obj3?versionId=null 2024-02-01T10:47:22.806 DEBUG:boto:auth_path=/vclfcn-60/obj3?versionId=null 2024-02-01T10:47:22.806 DEBUG:boto:Method: GET 2024-02-01T10:47:22.806 DEBUG:boto:Path: /vclfcn-60/obj3?versionId=null 2024-02-01T10:47:22.806 DEBUG:boto:Data: 2024-02-01T10:47:22.806 DEBUG:boto:Headers: {} 2024-02-01T10:47:22.806 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:22.806 DEBUG:boto:Port: 8000 2024-02-01T10:47:22.807 DEBUG:boto:Params: {} 2024-02-01T10:47:22.807 DEBUG:boto:Token: None 2024-02-01T10:47:22.807 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:22 GMT /vclfcn-60/obj3?versionId=null 2024-02-01T10:47:22.807 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ECVKOH0OLFI+5gPeFpO77HddNXU= 2024-02-01T10:47:22.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ECVKOH0OLFI+5gPeFpO77HddNXU=', 'Content-Length': '0'} 2024-02-01T10:47:22.810 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005c5e2c4de3e61de3-0065bb76ba-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:22.810 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:47:22.810 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:47:22.810 DEBUG:boto:path=/vclfcn-60/?prefix=obj3 2024-02-01T10:47:22.810 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj3 2024-02-01T10:47:22.810 DEBUG:boto:Method: GET 2024-02-01T10:47:22.810 DEBUG:boto:Path: /vclfcn-60/?prefix=obj3 2024-02-01T10:47:22.810 DEBUG:boto:Data: 2024-02-01T10:47:22.811 DEBUG:boto:Headers: {} 2024-02-01T10:47:22.811 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:22.811 DEBUG:boto:Port: 8000 2024-02-01T10:47:22.811 DEBUG:boto:Params: {} 2024-02-01T10:47:22.811 DEBUG:boto:Token: None 2024-02-01T10:47:22.811 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:22 GMT /vclfcn-60/ 2024-02-01T10:47:22.811 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:f16MNT44IZtusTBMK4MXEu8gOBE= 2024-02-01T10:47:22.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:f16MNT44IZtusTBMK4MXEu8gOBE=', 'Content-Length': '0'} 2024-02-01T10:47:22.815 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8d575f8a916565a-0065bb76ba-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:22.856 DEBUG:boto:b'vclfcn-60obj31000falseobj32024-02-01T10:46:46.285Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:22.857 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:47:22.857 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:47:22.857 DEBUG:boto:path=/vclfcn-60/?prefix=obj3 2024-02-01T10:47:22.857 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj3 2024-02-01T10:47:22.857 DEBUG:boto:Method: GET 2024-02-01T10:47:22.857 DEBUG:boto:Path: /vclfcn-60/?prefix=obj3 2024-02-01T10:47:22.857 DEBUG:boto:Data: 2024-02-01T10:47:22.857 DEBUG:boto:Headers: {} 2024-02-01T10:47:22.858 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:22.858 DEBUG:boto:Port: 8000 2024-02-01T10:47:22.858 DEBUG:boto:Params: {} 2024-02-01T10:47:22.858 DEBUG:boto:Token: None 2024-02-01T10:47:22.858 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:22 GMT /vclfcn-60/ 2024-02-01T10:47:22.858 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:f16MNT44IZtusTBMK4MXEu8gOBE= 2024-02-01T10:47:22.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:f16MNT44IZtusTBMK4MXEu8gOBE=', 'Content-Length': '0'} 2024-02-01T10:47:22.863 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d47ef43b63b07627-0065bb76ba-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:22.863 DEBUG:boto:b'vclfcn-60obj31000falseobj32024-02-01T10:46:46.285Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:22.864 DEBUG:boto:path=/vclfcn-60/obj3 2024-02-01T10:47:22.864 DEBUG:boto:auth_path=/vclfcn-60/obj3 2024-02-01T10:47:22.864 DEBUG:boto:Method: HEAD 2024-02-01T10:47:22.864 DEBUG:boto:Path: /vclfcn-60/obj3 2024-02-01T10:47:22.864 DEBUG:boto:Data: 2024-02-01T10:47:22.864 DEBUG:boto:Headers: {} 2024-02-01T10:47:22.864 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:22.864 DEBUG:boto:Port: 8000 2024-02-01T10:47:22.864 DEBUG:boto:Params: {} 2024-02-01T10:47:22.865 DEBUG:boto:Token: None 2024-02-01T10:47:22.865 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:22 GMT /vclfcn-60/obj3 2024-02-01T10:47:22.865 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fSdKkmWPI1CjS3lNRIrjAhccHmA= 2024-02-01T10:47:22.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fSdKkmWPI1CjS3lNRIrjAhccHmA=', 'Content-Length': '0'} 2024-02-01T10:47:22.868 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003d2f318d56007d75-0065bb76ba-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:22.868 DEBUG:boto:path=/vclfcn-60/obj3 2024-02-01T10:47:22.868 DEBUG:boto:auth_path=/vclfcn-60/obj3 2024-02-01T10:47:22.868 DEBUG:boto:Method: HEAD 2024-02-01T10:47:22.868 DEBUG:boto:Path: /vclfcn-60/obj3 2024-02-01T10:47:22.868 DEBUG:boto:Data: 2024-02-01T10:47:22.868 DEBUG:boto:Headers: {} 2024-02-01T10:47:22.868 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:22.869 DEBUG:boto:Port: 8000 2024-02-01T10:47:22.869 DEBUG:boto:Params: {} 2024-02-01T10:47:22.869 DEBUG:boto:Token: None 2024-02-01T10:47:22.869 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:22 GMT /vclfcn-60/obj3 2024-02-01T10:47:22.869 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fSdKkmWPI1CjS3lNRIrjAhccHmA= 2024-02-01T10:47:22.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fSdKkmWPI1CjS3lNRIrjAhccHmA=', 'Content-Length': '0'} 2024-02-01T10:47:22.872 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000084cf8bca18891a29-0065bb76ba-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:22.872 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:47:22.872 DEBUG:boto:path=/vclfcn-60/obj3 2024-02-01T10:47:22.872 DEBUG:boto:auth_path=/vclfcn-60/obj3 2024-02-01T10:47:22.873 DEBUG:boto:Method: GET 2024-02-01T10:47:22.873 DEBUG:boto:Path: /vclfcn-60/obj3 2024-02-01T10:47:22.873 DEBUG:boto:Data: 2024-02-01T10:47:22.873 DEBUG:boto:Headers: {} 2024-02-01T10:47:22.873 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:22.873 DEBUG:boto:Port: 8000 2024-02-01T10:47:22.873 DEBUG:boto:Params: {} 2024-02-01T10:47:22.873 DEBUG:boto:Token: None 2024-02-01T10:47:22.873 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:22 GMT /vclfcn-60/obj3 2024-02-01T10:47:22.873 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fV6eVtA5Mj+MJe+5rrfOeV4BtOY= 2024-02-01T10:47:22.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fV6eVtA5Mj+MJe+5rrfOeV4BtOY=', 'Content-Length': '0'} 2024-02-01T10:47:22.876 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e6d2ea55a6e53c1d-0065bb76ba-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:22.876 DEBUG:boto:path=/vclfcn-60/obj3 2024-02-01T10:47:22.877 DEBUG:boto:auth_path=/vclfcn-60/obj3 2024-02-01T10:47:22.877 DEBUG:boto:Method: GET 2024-02-01T10:47:22.877 DEBUG:boto:Path: /vclfcn-60/obj3 2024-02-01T10:47:22.877 DEBUG:boto:Data: 2024-02-01T10:47:22.877 DEBUG:boto:Headers: {} 2024-02-01T10:47:22.877 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:22.877 DEBUG:boto:Port: 8000 2024-02-01T10:47:22.877 DEBUG:boto:Params: {} 2024-02-01T10:47:22.877 DEBUG:boto:Token: None 2024-02-01T10:47:22.877 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:22 GMT /vclfcn-60/obj3 2024-02-01T10:47:22.878 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fV6eVtA5Mj+MJe+5rrfOeV4BtOY= 2024-02-01T10:47:22.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fV6eVtA5Mj+MJe+5rrfOeV4BtOY=', 'Content-Length': '0'} 2024-02-01T10:47:22.880 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f2a1ba6d6529b72a-0065bb76ba-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:22.928 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:47:22.928 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:47:22.928 DEBUG:boto:path=/vclfcn-60/?prefix=obj3 2024-02-01T10:47:22.928 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj3 2024-02-01T10:47:22.928 DEBUG:boto:Method: GET 2024-02-01T10:47:22.929 DEBUG:boto:Path: /vclfcn-60/?prefix=obj3 2024-02-01T10:47:22.929 DEBUG:boto:Data: 2024-02-01T10:47:22.929 DEBUG:boto:Headers: {} 2024-02-01T10:47:22.929 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:22.929 DEBUG:boto:Port: 8000 2024-02-01T10:47:22.929 DEBUG:boto:Params: {} 2024-02-01T10:47:22.929 DEBUG:boto:Token: None 2024-02-01T10:47:22.929 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:22 GMT /vclfcn-60/ 2024-02-01T10:47:22.929 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:f16MNT44IZtusTBMK4MXEu8gOBE= 2024-02-01T10:47:22.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:f16MNT44IZtusTBMK4MXEu8gOBE=', 'Content-Length': '0'} 2024-02-01T10:47:22.934 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000954c0cc31285de0c-0065bb76ba-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:22.934 DEBUG:boto:b'vclfcn-60obj31000falseobj32024-02-01T10:46:46.285Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:22.936 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:47:22.936 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:47:22.936 DEBUG:boto:path=/vclfcn-60/?prefix=obj3 2024-02-01T10:47:22.936 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj3 2024-02-01T10:47:22.936 DEBUG:boto:Method: GET 2024-02-01T10:47:22.937 DEBUG:boto:Path: /vclfcn-60/?prefix=obj3 2024-02-01T10:47:22.937 DEBUG:boto:Data: 2024-02-01T10:47:22.937 DEBUG:boto:Headers: {} 2024-02-01T10:47:22.937 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:22.937 DEBUG:boto:Port: 8000 2024-02-01T10:47:22.937 DEBUG:boto:Params: {} 2024-02-01T10:47:22.937 DEBUG:boto:Token: None 2024-02-01T10:47:22.937 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:22 GMT /vclfcn-60/ 2024-02-01T10:47:22.937 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:f16MNT44IZtusTBMK4MXEu8gOBE= 2024-02-01T10:47:22.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:f16MNT44IZtusTBMK4MXEu8gOBE=', 'Content-Length': '0'} 2024-02-01T10:47:22.942 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3684d7d9bd15a2c-0065bb76ba-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:22.942 DEBUG:boto:b'vclfcn-60obj31000falseobj32024-02-01T10:46:46.285Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:22.942 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:47:22.943 DEBUG:boto:path=/vclfcn-60/obj4 2024-02-01T10:47:22.943 DEBUG:boto:auth_path=/vclfcn-60/obj4 2024-02-01T10:47:22.943 DEBUG:boto:path=/vclfcn-60/obj4?versionId=null 2024-02-01T10:47:22.943 DEBUG:boto:auth_path=/vclfcn-60/obj4?versionId=null 2024-02-01T10:47:22.943 DEBUG:boto:Method: GET 2024-02-01T10:47:22.943 DEBUG:boto:Path: /vclfcn-60/obj4?versionId=null 2024-02-01T10:47:22.943 DEBUG:boto:Data: 2024-02-01T10:47:22.943 DEBUG:boto:Headers: {} 2024-02-01T10:47:22.943 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:22.943 DEBUG:boto:Port: 8000 2024-02-01T10:47:22.943 DEBUG:boto:Params: {} 2024-02-01T10:47:22.944 DEBUG:boto:Token: None 2024-02-01T10:47:22.944 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:22 GMT /vclfcn-60/obj4?versionId=null 2024-02-01T10:47:22.944 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:JV4TNuypYJiw66V3GbBvPOT9uiE= 2024-02-01T10:47:22.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:JV4TNuypYJiw66V3GbBvPOT9uiE=', 'Content-Length': '0'} 2024-02-01T10:47:22.947 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000044727088d8abadc2-0065bb76ba-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:22.947 DEBUG:boto:path=/vclfcn-60/obj4 2024-02-01T10:47:22.947 DEBUG:boto:auth_path=/vclfcn-60/obj4 2024-02-01T10:47:22.947 DEBUG:boto:path=/vclfcn-60/obj4?versionId=null 2024-02-01T10:47:22.947 DEBUG:boto:auth_path=/vclfcn-60/obj4?versionId=null 2024-02-01T10:47:22.947 DEBUG:boto:Method: GET 2024-02-01T10:47:22.947 DEBUG:boto:Path: /vclfcn-60/obj4?versionId=null 2024-02-01T10:47:22.947 DEBUG:boto:Data: 2024-02-01T10:47:22.948 DEBUG:boto:Headers: {} 2024-02-01T10:47:22.948 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:22.948 DEBUG:boto:Port: 8000 2024-02-01T10:47:22.948 DEBUG:boto:Params: {} 2024-02-01T10:47:22.948 DEBUG:boto:Token: None 2024-02-01T10:47:22.948 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:22 GMT /vclfcn-60/obj4?versionId=null 2024-02-01T10:47:22.948 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:JV4TNuypYJiw66V3GbBvPOT9uiE= 2024-02-01T10:47:22.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:JV4TNuypYJiw66V3GbBvPOT9uiE=', 'Content-Length': '0'} 2024-02-01T10:47:22.951 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fc3ce397921466ce-0065bb76ba-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:22.951 DEBUG:boto:path=/vclfcn-60/obj4 2024-02-01T10:47:22.951 DEBUG:boto:auth_path=/vclfcn-60/obj4 2024-02-01T10:47:22.951 DEBUG:boto:path=/vclfcn-60/obj4?versionId=null 2024-02-01T10:47:22.952 DEBUG:boto:auth_path=/vclfcn-60/obj4?versionId=null 2024-02-01T10:47:22.952 DEBUG:boto:Method: HEAD 2024-02-01T10:47:22.952 DEBUG:boto:Path: /vclfcn-60/obj4?versionId=null 2024-02-01T10:47:22.952 DEBUG:boto:Data: 2024-02-01T10:47:22.952 DEBUG:boto:Headers: {} 2024-02-01T10:47:22.952 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:22.952 DEBUG:boto:Port: 8000 2024-02-01T10:47:22.952 DEBUG:boto:Params: {} 2024-02-01T10:47:22.952 DEBUG:boto:Token: None 2024-02-01T10:47:22.952 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:22 GMT /vclfcn-60/obj4?versionId=null 2024-02-01T10:47:22.953 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8sgqSReYN8mQ+xGQbxO1tcMO/j4= 2024-02-01T10:47:22.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8sgqSReYN8mQ+xGQbxO1tcMO/j4=', 'Content-Length': '0'} 2024-02-01T10:47:22.955 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005d042ac1051684cb-0065bb76ba-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:22.955 DEBUG:boto:path=/vclfcn-60/obj4 2024-02-01T10:47:22.955 DEBUG:boto:auth_path=/vclfcn-60/obj4 2024-02-01T10:47:22.956 DEBUG:boto:path=/vclfcn-60/obj4?versionId=null 2024-02-01T10:47:22.956 DEBUG:boto:auth_path=/vclfcn-60/obj4?versionId=null 2024-02-01T10:47:22.956 DEBUG:boto:Method: HEAD 2024-02-01T10:47:22.956 DEBUG:boto:Path: /vclfcn-60/obj4?versionId=null 2024-02-01T10:47:22.956 DEBUG:boto:Data: 2024-02-01T10:47:22.956 DEBUG:boto:Headers: {} 2024-02-01T10:47:22.956 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:22.956 DEBUG:boto:Port: 8000 2024-02-01T10:47:22.956 DEBUG:boto:Params: {} 2024-02-01T10:47:22.956 DEBUG:boto:Token: None 2024-02-01T10:47:22.957 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:22 GMT /vclfcn-60/obj4?versionId=null 2024-02-01T10:47:22.957 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8sgqSReYN8mQ+xGQbxO1tcMO/j4= 2024-02-01T10:47:22.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8sgqSReYN8mQ+xGQbxO1tcMO/j4=', 'Content-Length': '0'} 2024-02-01T10:47:22.959 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fc89e2218e969901-0065bb76ba-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:22.960 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:47:22.960 DEBUG:boto:path=/vclfcn-60/obj4 2024-02-01T10:47:22.960 DEBUG:boto:auth_path=/vclfcn-60/obj4 2024-02-01T10:47:22.960 DEBUG:boto:path=/vclfcn-60/obj4?versionId=null 2024-02-01T10:47:22.960 DEBUG:boto:auth_path=/vclfcn-60/obj4?versionId=null 2024-02-01T10:47:22.960 DEBUG:boto:Method: GET 2024-02-01T10:47:22.960 DEBUG:boto:Path: /vclfcn-60/obj4?versionId=null 2024-02-01T10:47:22.960 DEBUG:boto:Data: 2024-02-01T10:47:22.960 DEBUG:boto:Headers: {} 2024-02-01T10:47:22.960 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:22.961 DEBUG:boto:Port: 8000 2024-02-01T10:47:22.961 DEBUG:boto:Params: {} 2024-02-01T10:47:22.961 DEBUG:boto:Token: None 2024-02-01T10:47:22.961 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:22 GMT /vclfcn-60/obj4?versionId=null 2024-02-01T10:47:22.961 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:JV4TNuypYJiw66V3GbBvPOT9uiE= 2024-02-01T10:47:22.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:JV4TNuypYJiw66V3GbBvPOT9uiE=', 'Content-Length': '0'} 2024-02-01T10:47:22.964 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c72659ac30fc4c7b-0065bb76ba-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:22.964 DEBUG:boto:path=/vclfcn-60/obj4 2024-02-01T10:47:22.964 DEBUG:boto:auth_path=/vclfcn-60/obj4 2024-02-01T10:47:22.964 DEBUG:boto:path=/vclfcn-60/obj4?versionId=null 2024-02-01T10:47:22.964 DEBUG:boto:auth_path=/vclfcn-60/obj4?versionId=null 2024-02-01T10:47:22.965 DEBUG:boto:Method: GET 2024-02-01T10:47:22.965 DEBUG:boto:Path: /vclfcn-60/obj4?versionId=null 2024-02-01T10:47:22.965 DEBUG:boto:Data: 2024-02-01T10:47:22.965 DEBUG:boto:Headers: {} 2024-02-01T10:47:22.965 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:22.965 DEBUG:boto:Port: 8000 2024-02-01T10:47:22.965 DEBUG:boto:Params: {} 2024-02-01T10:47:22.965 DEBUG:boto:Token: None 2024-02-01T10:47:22.965 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:22 GMT /vclfcn-60/obj4?versionId=null 2024-02-01T10:47:22.965 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:JV4TNuypYJiw66V3GbBvPOT9uiE= 2024-02-01T10:47:22.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:JV4TNuypYJiw66V3GbBvPOT9uiE=', 'Content-Length': '0'} 2024-02-01T10:47:22.968 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000032a9074023928fc2-0065bb76ba-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:22.968 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:47:22.969 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:47:22.969 DEBUG:boto:path=/vclfcn-60/?prefix=obj4 2024-02-01T10:47:22.969 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj4 2024-02-01T10:47:22.969 DEBUG:boto:Method: GET 2024-02-01T10:47:22.969 DEBUG:boto:Path: /vclfcn-60/?prefix=obj4 2024-02-01T10:47:22.969 DEBUG:boto:Data: 2024-02-01T10:47:22.969 DEBUG:boto:Headers: {} 2024-02-01T10:47:22.969 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:22.969 DEBUG:boto:Port: 8000 2024-02-01T10:47:22.969 DEBUG:boto:Params: {} 2024-02-01T10:47:22.969 DEBUG:boto:Token: None 2024-02-01T10:47:22.970 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:22 GMT /vclfcn-60/ 2024-02-01T10:47:22.970 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:f16MNT44IZtusTBMK4MXEu8gOBE= 2024-02-01T10:47:22.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:f16MNT44IZtusTBMK4MXEu8gOBE=', 'Content-Length': '0'} 2024-02-01T10:47:22.974 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000217724e9a4579fc-0065bb76ba-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:23.016 DEBUG:boto:b'vclfcn-60obj41000falseobj42024-02-01T10:46:46.336Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:23.016 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:47:23.017 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:47:23.017 DEBUG:boto:path=/vclfcn-60/?prefix=obj4 2024-02-01T10:47:23.017 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj4 2024-02-01T10:47:23.017 DEBUG:boto:Method: GET 2024-02-01T10:47:23.017 DEBUG:boto:Path: /vclfcn-60/?prefix=obj4 2024-02-01T10:47:23.017 DEBUG:boto:Data: 2024-02-01T10:47:23.017 DEBUG:boto:Headers: {} 2024-02-01T10:47:23.017 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:23.017 DEBUG:boto:Port: 8000 2024-02-01T10:47:23.017 DEBUG:boto:Params: {} 2024-02-01T10:47:23.017 DEBUG:boto:Token: None 2024-02-01T10:47:23.018 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:23 GMT /vclfcn-60/ 2024-02-01T10:47:23.018 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:T5VkXnb51nqYgqUrLSMulfbUX3o= 2024-02-01T10:47:23.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:T5VkXnb51nqYgqUrLSMulfbUX3o=', 'Content-Length': '0'} 2024-02-01T10:47:23.022 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c3227a810cc77d8-0065bb76bb-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:23.022 DEBUG:boto:b'vclfcn-60obj41000falseobj42024-02-01T10:46:46.336Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:23.023 DEBUG:boto:path=/vclfcn-60/obj4 2024-02-01T10:47:23.023 DEBUG:boto:auth_path=/vclfcn-60/obj4 2024-02-01T10:47:23.023 DEBUG:boto:Method: HEAD 2024-02-01T10:47:23.023 DEBUG:boto:Path: /vclfcn-60/obj4 2024-02-01T10:47:23.023 DEBUG:boto:Data: 2024-02-01T10:47:23.023 DEBUG:boto:Headers: {} 2024-02-01T10:47:23.023 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:23.024 DEBUG:boto:Port: 8000 2024-02-01T10:47:23.024 DEBUG:boto:Params: {} 2024-02-01T10:47:23.024 DEBUG:boto:Token: None 2024-02-01T10:47:23.024 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:23 GMT /vclfcn-60/obj4 2024-02-01T10:47:23.024 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mdUpxNME306YBrVf4B8ReXDLA6M= 2024-02-01T10:47:23.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mdUpxNME306YBrVf4B8ReXDLA6M=', 'Content-Length': '0'} 2024-02-01T10:47:23.027 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007789fa09a8d727c0-0065bb76bb-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:23.027 DEBUG:boto:path=/vclfcn-60/obj4 2024-02-01T10:47:23.027 DEBUG:boto:auth_path=/vclfcn-60/obj4 2024-02-01T10:47:23.027 DEBUG:boto:Method: HEAD 2024-02-01T10:47:23.027 DEBUG:boto:Path: /vclfcn-60/obj4 2024-02-01T10:47:23.027 DEBUG:boto:Data: 2024-02-01T10:47:23.027 DEBUG:boto:Headers: {} 2024-02-01T10:47:23.027 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:23.027 DEBUG:boto:Port: 8000 2024-02-01T10:47:23.028 DEBUG:boto:Params: {} 2024-02-01T10:47:23.028 DEBUG:boto:Token: None 2024-02-01T10:47:23.028 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:23 GMT /vclfcn-60/obj4 2024-02-01T10:47:23.028 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mdUpxNME306YBrVf4B8ReXDLA6M= 2024-02-01T10:47:23.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mdUpxNME306YBrVf4B8ReXDLA6M=', 'Content-Length': '0'} 2024-02-01T10:47:23.031 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fd6ee3aa0091a33b-0065bb76bb-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:23.031 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:47:23.031 DEBUG:boto:path=/vclfcn-60/obj4 2024-02-01T10:47:23.031 DEBUG:boto:auth_path=/vclfcn-60/obj4 2024-02-01T10:47:23.031 DEBUG:boto:Method: GET 2024-02-01T10:47:23.031 DEBUG:boto:Path: /vclfcn-60/obj4 2024-02-01T10:47:23.031 DEBUG:boto:Data: 2024-02-01T10:47:23.032 DEBUG:boto:Headers: {} 2024-02-01T10:47:23.032 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:23.032 DEBUG:boto:Port: 8000 2024-02-01T10:47:23.032 DEBUG:boto:Params: {} 2024-02-01T10:47:23.032 DEBUG:boto:Token: None 2024-02-01T10:47:23.032 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:23 GMT /vclfcn-60/obj4 2024-02-01T10:47:23.032 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:6NZHya44BljAXMpshS84MzzjLyo= 2024-02-01T10:47:23.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:6NZHya44BljAXMpshS84MzzjLyo=', 'Content-Length': '0'} 2024-02-01T10:47:23.035 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000087e308ebf533af25-0065bb76bb-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:23.036 DEBUG:boto:path=/vclfcn-60/obj4 2024-02-01T10:47:23.036 DEBUG:boto:auth_path=/vclfcn-60/obj4 2024-02-01T10:47:23.036 DEBUG:boto:Method: GET 2024-02-01T10:47:23.036 DEBUG:boto:Path: /vclfcn-60/obj4 2024-02-01T10:47:23.036 DEBUG:boto:Data: 2024-02-01T10:47:23.036 DEBUG:boto:Headers: {} 2024-02-01T10:47:23.036 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:23.036 DEBUG:boto:Port: 8000 2024-02-01T10:47:23.036 DEBUG:boto:Params: {} 2024-02-01T10:47:23.036 DEBUG:boto:Token: None 2024-02-01T10:47:23.037 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:23 GMT /vclfcn-60/obj4 2024-02-01T10:47:23.037 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:6NZHya44BljAXMpshS84MzzjLyo= 2024-02-01T10:47:23.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:6NZHya44BljAXMpshS84MzzjLyo=', 'Content-Length': '0'} 2024-02-01T10:47:23.039 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000285e04efc7b4a86e-0065bb76bb-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:23.080 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:47:23.080 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:47:23.080 DEBUG:boto:path=/vclfcn-60/?prefix=obj4 2024-02-01T10:47:23.080 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj4 2024-02-01T10:47:23.080 DEBUG:boto:Method: GET 2024-02-01T10:47:23.081 DEBUG:boto:Path: /vclfcn-60/?prefix=obj4 2024-02-01T10:47:23.081 DEBUG:boto:Data: 2024-02-01T10:47:23.081 DEBUG:boto:Headers: {} 2024-02-01T10:47:23.081 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:23.081 DEBUG:boto:Port: 8000 2024-02-01T10:47:23.081 DEBUG:boto:Params: {} 2024-02-01T10:47:23.081 DEBUG:boto:Token: None 2024-02-01T10:47:23.081 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:23 GMT /vclfcn-60/ 2024-02-01T10:47:23.081 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:T5VkXnb51nqYgqUrLSMulfbUX3o= 2024-02-01T10:47:23.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:T5VkXnb51nqYgqUrLSMulfbUX3o=', 'Content-Length': '0'} 2024-02-01T10:47:23.086 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000031e746e74623abe7-0065bb76bb-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:23.086 DEBUG:boto:b'vclfcn-60obj41000falseobj42024-02-01T10:46:46.336Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:23.087 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:47:23.087 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:47:23.087 DEBUG:boto:path=/vclfcn-60/?prefix=obj4 2024-02-01T10:47:23.087 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj4 2024-02-01T10:47:23.087 DEBUG:boto:Method: GET 2024-02-01T10:47:23.087 DEBUG:boto:Path: /vclfcn-60/?prefix=obj4 2024-02-01T10:47:23.087 DEBUG:boto:Data: 2024-02-01T10:47:23.087 DEBUG:boto:Headers: {} 2024-02-01T10:47:23.088 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:23.088 DEBUG:boto:Port: 8000 2024-02-01T10:47:23.088 DEBUG:boto:Params: {} 2024-02-01T10:47:23.088 DEBUG:boto:Token: None 2024-02-01T10:47:23.088 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:23 GMT /vclfcn-60/ 2024-02-01T10:47:23.088 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:T5VkXnb51nqYgqUrLSMulfbUX3o= 2024-02-01T10:47:23.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:T5VkXnb51nqYgqUrLSMulfbUX3o=', 'Content-Length': '0'} 2024-02-01T10:47:23.092 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e038f437f572b6ab-0065bb76bb-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:23.093 DEBUG:boto:b'vclfcn-60obj41000falseobj42024-02-01T10:46:46.336Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:23.093 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-60 zones={test-zone2, test-zone1} 2024-02-01T10:47:23.093 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-60 zones={test-zone3, test-zone1} 2024-02-01T10:47:23.093 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:47:23.093 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:47:23.094 DEBUG:boto:Method: HEAD 2024-02-01T10:47:23.094 DEBUG:boto:Path: /vclfcn-60/ 2024-02-01T10:47:23.094 DEBUG:boto:Data: 2024-02-01T10:47:23.094 DEBUG:boto:Headers: {} 2024-02-01T10:47:23.094 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:23.094 DEBUG:boto:Port: 8001 2024-02-01T10:47:23.094 DEBUG:boto:Params: {} 2024-02-01T10:47:23.094 DEBUG:boto:Token: None 2024-02-01T10:47:23.094 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:23 GMT /vclfcn-60/ 2024-02-01T10:47:23.094 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:W9xvyS07kBW01xKFmFETOumuwnc= 2024-02-01T10:47:23.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:W9xvyS07kBW01xKFmFETOumuwnc=', 'Content-Length': '0'} 2024-02-01T10:47:23.098 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000084e49366e18262fb-0065bb76bb-5762-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:47:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:23.098 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:47:23.099 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:47:23.099 DEBUG:boto:Method: HEAD 2024-02-01T10:47:23.099 DEBUG:boto:Path: /vclfcn-60/ 2024-02-01T10:47:23.099 DEBUG:boto:Data: 2024-02-01T10:47:23.099 DEBUG:boto:Headers: {} 2024-02-01T10:47:23.099 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:23.099 DEBUG:boto:Port: 8000 2024-02-01T10:47:23.099 DEBUG:boto:Params: {} 2024-02-01T10:47:23.099 DEBUG:boto:Token: None 2024-02-01T10:47:23.099 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:23 GMT /vclfcn-60/ 2024-02-01T10:47:23.100 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:W9xvyS07kBW01xKFmFETOumuwnc= 2024-02-01T10:47:23.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:W9xvyS07kBW01xKFmFETOumuwnc=', 'Content-Length': '0'} 2024-02-01T10:47:23.104 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f7c8f075732572c3-0065bb76bb-5765-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:47:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:23.104 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:47:23.104 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:47:23.104 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:47:23.104 DEBUG:boto:path=/vclfcn-60/?versions&max-keys=999 2024-02-01T10:47:23.104 DEBUG:boto:auth_path=/vclfcn-60/?versions&max-keys=999 2024-02-01T10:47:23.105 DEBUG:boto:Method: GET 2024-02-01T10:47:23.105 DEBUG:boto:Path: /vclfcn-60/?versions&max-keys=999 2024-02-01T10:47:23.105 DEBUG:boto:Data: 2024-02-01T10:47:23.105 DEBUG:boto:Headers: {} 2024-02-01T10:47:23.105 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:23.105 DEBUG:boto:Port: 8001 2024-02-01T10:47:23.105 DEBUG:boto:Params: {} 2024-02-01T10:47:23.105 DEBUG:boto:Token: None 2024-02-01T10:47:23.105 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:23 GMT /vclfcn-60/?versions 2024-02-01T10:47:23.105 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:yPR9maRp11dlUMTd2jCciyZ3Ym0= 2024-02-01T10:47:23.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:yPR9maRp11dlUMTd2jCciyZ3Ym0=', 'Content-Length': '0'} 2024-02-01T10:47:23.109 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c428cd3314cfc370-0065bb76bb-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:23.152 DEBUG:boto:b'vclfcn-60999falseobj1nulltrue2024-02-01T10:46:46.243Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:46:46.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:46:46.285Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:46:46.336Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:23.153 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-02-01T10:47:23.153 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-02-01T10:47:23.153 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-02-01T10:47:23.153 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-02-01T10:47:23.153 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:47:23.153 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:47:23.153 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:47:23.153 DEBUG:boto:path=/vclfcn-60/?versions&max-keys=999 2024-02-01T10:47:23.153 DEBUG:boto:auth_path=/vclfcn-60/?versions&max-keys=999 2024-02-01T10:47:23.153 DEBUG:boto:Method: GET 2024-02-01T10:47:23.154 DEBUG:boto:Path: /vclfcn-60/?versions&max-keys=999 2024-02-01T10:47:23.154 DEBUG:boto:Data: 2024-02-01T10:47:23.154 DEBUG:boto:Headers: {} 2024-02-01T10:47:23.154 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:23.154 DEBUG:boto:Port: 8000 2024-02-01T10:47:23.154 DEBUG:boto:Params: {} 2024-02-01T10:47:23.154 DEBUG:boto:Token: None 2024-02-01T10:47:23.154 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:23 GMT /vclfcn-60/?versions 2024-02-01T10:47:23.154 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:yPR9maRp11dlUMTd2jCciyZ3Ym0= 2024-02-01T10:47:23.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:yPR9maRp11dlUMTd2jCciyZ3Ym0=', 'Content-Length': '0'} 2024-02-01T10:47:23.160 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002212f5970d27e8e5-0065bb76bb-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:23.160 DEBUG:boto:b'vclfcn-60999falseobj1nulltrue2024-02-01T10:46:46.243Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:46:46.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:46:46.285Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:46:46.336Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:23.160 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-02-01T10:47:23.161 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-02-01T10:47:23.161 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-02-01T10:47:23.161 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-02-01T10:47:23.161 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:47:23.161 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:47:23.161 DEBUG:boto:path=/vclfcn-60/?versions&max-keys=999 2024-02-01T10:47:23.161 DEBUG:boto:auth_path=/vclfcn-60/?versions&max-keys=999 2024-02-01T10:47:23.161 DEBUG:boto:Method: GET 2024-02-01T10:47:23.161 DEBUG:boto:Path: /vclfcn-60/?versions&max-keys=999 2024-02-01T10:47:23.162 DEBUG:boto:Data: 2024-02-01T10:47:23.162 DEBUG:boto:Headers: {} 2024-02-01T10:47:23.162 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:23.162 DEBUG:boto:Port: 8001 2024-02-01T10:47:23.162 DEBUG:boto:Params: {} 2024-02-01T10:47:23.162 DEBUG:boto:Token: None 2024-02-01T10:47:23.162 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:23 GMT /vclfcn-60/?versions 2024-02-01T10:47:23.162 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:yPR9maRp11dlUMTd2jCciyZ3Ym0= 2024-02-01T10:47:23.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:yPR9maRp11dlUMTd2jCciyZ3Ym0=', 'Content-Length': '0'} 2024-02-01T10:47:23.167 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008df3e2b2d5837de4-0065bb76bb-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:23.167 DEBUG:boto:b'vclfcn-60999falseobj1nulltrue2024-02-01T10:46:46.243Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:46:46.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:46:46.285Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:46:46.336Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:23.167 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:47:23.167 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:47:23.168 DEBUG:boto:path=/vclfcn-60/?versions&max-keys=999 2024-02-01T10:47:23.168 DEBUG:boto:auth_path=/vclfcn-60/?versions&max-keys=999 2024-02-01T10:47:23.168 DEBUG:boto:Method: GET 2024-02-01T10:47:23.168 DEBUG:boto:Path: /vclfcn-60/?versions&max-keys=999 2024-02-01T10:47:23.168 DEBUG:boto:Data: 2024-02-01T10:47:23.168 DEBUG:boto:Headers: {} 2024-02-01T10:47:23.168 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:23.168 DEBUG:boto:Port: 8000 2024-02-01T10:47:23.168 DEBUG:boto:Params: {} 2024-02-01T10:47:23.168 DEBUG:boto:Token: None 2024-02-01T10:47:23.169 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:23 GMT /vclfcn-60/?versions 2024-02-01T10:47:23.169 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:yPR9maRp11dlUMTd2jCciyZ3Ym0= 2024-02-01T10:47:23.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:yPR9maRp11dlUMTd2jCciyZ3Ym0=', 'Content-Length': '0'} 2024-02-01T10:47:23.173 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000023750fd726f240d-0065bb76bb-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:23.216 DEBUG:boto:b'vclfcn-60999falseobj1nulltrue2024-02-01T10:46:46.243Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:46:46.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:46:46.285Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:46:46.336Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:23.217 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:47:23.217 DEBUG:boto:path=/vclfcn-60/obj1 2024-02-01T10:47:23.217 DEBUG:boto:auth_path=/vclfcn-60/obj1 2024-02-01T10:47:23.217 DEBUG:boto:path=/vclfcn-60/obj1?versionId=null 2024-02-01T10:47:23.217 DEBUG:boto:auth_path=/vclfcn-60/obj1?versionId=null 2024-02-01T10:47:23.217 DEBUG:boto:Method: GET 2024-02-01T10:47:23.217 DEBUG:boto:Path: /vclfcn-60/obj1?versionId=null 2024-02-01T10:47:23.217 DEBUG:boto:Data: 2024-02-01T10:47:23.217 DEBUG:boto:Headers: {} 2024-02-01T10:47:23.218 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:23.218 DEBUG:boto:Port: 8001 2024-02-01T10:47:23.218 DEBUG:boto:Params: {} 2024-02-01T10:47:23.218 DEBUG:boto:Token: None 2024-02-01T10:47:23.218 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:23 GMT /vclfcn-60/obj1?versionId=null 2024-02-01T10:47:23.218 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1YoLIKxCOh7wYJdfGB1nDh6o5go= 2024-02-01T10:47:23.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1YoLIKxCOh7wYJdfGB1nDh6o5go=', 'Content-Length': '0'} 2024-02-01T10:47:23.221 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000014a41cd1722fdfe1-0065bb76bb-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:23.222 DEBUG:boto:path=/vclfcn-60/obj1 2024-02-01T10:47:23.222 DEBUG:boto:auth_path=/vclfcn-60/obj1 2024-02-01T10:47:23.222 DEBUG:boto:path=/vclfcn-60/obj1?versionId=null 2024-02-01T10:47:23.222 DEBUG:boto:auth_path=/vclfcn-60/obj1?versionId=null 2024-02-01T10:47:23.222 DEBUG:boto:Method: GET 2024-02-01T10:47:23.222 DEBUG:boto:Path: /vclfcn-60/obj1?versionId=null 2024-02-01T10:47:23.222 DEBUG:boto:Data: 2024-02-01T10:47:23.222 DEBUG:boto:Headers: {} 2024-02-01T10:47:23.222 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:23.223 DEBUG:boto:Port: 8000 2024-02-01T10:47:23.223 DEBUG:boto:Params: {} 2024-02-01T10:47:23.223 DEBUG:boto:Token: None 2024-02-01T10:47:23.223 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:23 GMT /vclfcn-60/obj1?versionId=null 2024-02-01T10:47:23.223 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1YoLIKxCOh7wYJdfGB1nDh6o5go= 2024-02-01T10:47:23.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1YoLIKxCOh7wYJdfGB1nDh6o5go=', 'Content-Length': '0'} 2024-02-01T10:47:23.226 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001470f6a0b6dc0428-0065bb76bb-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:23.227 DEBUG:boto:path=/vclfcn-60/obj1 2024-02-01T10:47:23.227 DEBUG:boto:auth_path=/vclfcn-60/obj1 2024-02-01T10:47:23.227 DEBUG:boto:path=/vclfcn-60/obj1?versionId=null 2024-02-01T10:47:23.227 DEBUG:boto:auth_path=/vclfcn-60/obj1?versionId=null 2024-02-01T10:47:23.227 DEBUG:boto:Method: HEAD 2024-02-01T10:47:23.227 DEBUG:boto:Path: /vclfcn-60/obj1?versionId=null 2024-02-01T10:47:23.227 DEBUG:boto:Data: 2024-02-01T10:47:23.227 DEBUG:boto:Headers: {} 2024-02-01T10:47:23.227 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:23.227 DEBUG:boto:Port: 8001 2024-02-01T10:47:23.227 DEBUG:boto:Params: {} 2024-02-01T10:47:23.228 DEBUG:boto:Token: None 2024-02-01T10:47:23.228 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:23 GMT /vclfcn-60/obj1?versionId=null 2024-02-01T10:47:23.228 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:m4J4UDQgpw9uzrRfL1ezG0ovdRk= 2024-02-01T10:47:23.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:m4J4UDQgpw9uzrRfL1ezG0ovdRk=', 'Content-Length': '0'} 2024-02-01T10:47:23.231 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000296c85498193d188-0065bb76bb-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:23.231 DEBUG:boto:path=/vclfcn-60/obj1 2024-02-01T10:47:23.231 DEBUG:boto:auth_path=/vclfcn-60/obj1 2024-02-01T10:47:23.231 DEBUG:boto:path=/vclfcn-60/obj1?versionId=null 2024-02-01T10:47:23.232 DEBUG:boto:auth_path=/vclfcn-60/obj1?versionId=null 2024-02-01T10:47:23.232 DEBUG:boto:Method: HEAD 2024-02-01T10:47:23.232 DEBUG:boto:Path: /vclfcn-60/obj1?versionId=null 2024-02-01T10:47:23.232 DEBUG:boto:Data: 2024-02-01T10:47:23.232 DEBUG:boto:Headers: {} 2024-02-01T10:47:23.232 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:23.232 DEBUG:boto:Port: 8000 2024-02-01T10:47:23.232 DEBUG:boto:Params: {} 2024-02-01T10:47:23.232 DEBUG:boto:Token: None 2024-02-01T10:47:23.232 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:23 GMT /vclfcn-60/obj1?versionId=null 2024-02-01T10:47:23.233 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:m4J4UDQgpw9uzrRfL1ezG0ovdRk= 2024-02-01T10:47:23.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:m4J4UDQgpw9uzrRfL1ezG0ovdRk=', 'Content-Length': '0'} 2024-02-01T10:47:23.235 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e886498e6e78ef56-0065bb76bb-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:23.236 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:47:23.236 DEBUG:boto:path=/vclfcn-60/obj1 2024-02-01T10:47:23.236 DEBUG:boto:auth_path=/vclfcn-60/obj1 2024-02-01T10:47:23.236 DEBUG:boto:path=/vclfcn-60/obj1?versionId=null 2024-02-01T10:47:23.236 DEBUG:boto:auth_path=/vclfcn-60/obj1?versionId=null 2024-02-01T10:47:23.236 DEBUG:boto:Method: GET 2024-02-01T10:47:23.236 DEBUG:boto:Path: /vclfcn-60/obj1?versionId=null 2024-02-01T10:47:23.236 DEBUG:boto:Data: 2024-02-01T10:47:23.236 DEBUG:boto:Headers: {} 2024-02-01T10:47:23.236 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:23.237 DEBUG:boto:Port: 8001 2024-02-01T10:47:23.237 DEBUG:boto:Params: {} 2024-02-01T10:47:23.237 DEBUG:boto:Token: None 2024-02-01T10:47:23.237 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:23 GMT /vclfcn-60/obj1?versionId=null 2024-02-01T10:47:23.237 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1YoLIKxCOh7wYJdfGB1nDh6o5go= 2024-02-01T10:47:23.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1YoLIKxCOh7wYJdfGB1nDh6o5go=', 'Content-Length': '0'} 2024-02-01T10:47:23.240 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007c4a7c0c0b03c733-0065bb76bb-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:23.284 DEBUG:boto:path=/vclfcn-60/obj1 2024-02-01T10:47:23.284 DEBUG:boto:auth_path=/vclfcn-60/obj1 2024-02-01T10:47:23.284 DEBUG:boto:path=/vclfcn-60/obj1?versionId=null 2024-02-01T10:47:23.284 DEBUG:boto:auth_path=/vclfcn-60/obj1?versionId=null 2024-02-01T10:47:23.284 DEBUG:boto:Method: GET 2024-02-01T10:47:23.284 DEBUG:boto:Path: /vclfcn-60/obj1?versionId=null 2024-02-01T10:47:23.285 DEBUG:boto:Data: 2024-02-01T10:47:23.285 DEBUG:boto:Headers: {} 2024-02-01T10:47:23.285 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:23.285 DEBUG:boto:Port: 8000 2024-02-01T10:47:23.285 DEBUG:boto:Params: {} 2024-02-01T10:47:23.285 DEBUG:boto:Token: None 2024-02-01T10:47:23.285 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:23 GMT /vclfcn-60/obj1?versionId=null 2024-02-01T10:47:23.285 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1YoLIKxCOh7wYJdfGB1nDh6o5go= 2024-02-01T10:47:23.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1YoLIKxCOh7wYJdfGB1nDh6o5go=', 'Content-Length': '0'} 2024-02-01T10:47:23.289 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000371d5985f34c2e90-0065bb76bb-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:23.289 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:47:23.289 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:47:23.289 DEBUG:boto:path=/vclfcn-60/?prefix=obj1 2024-02-01T10:47:23.290 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj1 2024-02-01T10:47:23.290 DEBUG:boto:Method: GET 2024-02-01T10:47:23.290 DEBUG:boto:Path: /vclfcn-60/?prefix=obj1 2024-02-01T10:47:23.290 DEBUG:boto:Data: 2024-02-01T10:47:23.290 DEBUG:boto:Headers: {} 2024-02-01T10:47:23.290 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:23.290 DEBUG:boto:Port: 8001 2024-02-01T10:47:23.290 DEBUG:boto:Params: {} 2024-02-01T10:47:23.290 DEBUG:boto:Token: None 2024-02-01T10:47:23.290 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:23 GMT /vclfcn-60/ 2024-02-01T10:47:23.291 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:T5VkXnb51nqYgqUrLSMulfbUX3o= 2024-02-01T10:47:23.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:T5VkXnb51nqYgqUrLSMulfbUX3o=', 'Content-Length': '0'} 2024-02-01T10:47:23.299 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7c868d4ad961646-0065bb76bb-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:23.340 DEBUG:boto:b'vclfcn-60obj11000falseobj12024-02-01T10:46:46.243Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:23.340 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:47:23.340 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:47:23.341 DEBUG:boto:path=/vclfcn-60/?prefix=obj1 2024-02-01T10:47:23.341 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj1 2024-02-01T10:47:23.341 DEBUG:boto:Method: GET 2024-02-01T10:47:23.341 DEBUG:boto:Path: /vclfcn-60/?prefix=obj1 2024-02-01T10:47:23.341 DEBUG:boto:Data: 2024-02-01T10:47:23.341 DEBUG:boto:Headers: {} 2024-02-01T10:47:23.341 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:23.341 DEBUG:boto:Port: 8000 2024-02-01T10:47:23.341 DEBUG:boto:Params: {} 2024-02-01T10:47:23.341 DEBUG:boto:Token: None 2024-02-01T10:47:23.342 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:23 GMT /vclfcn-60/ 2024-02-01T10:47:23.342 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:T5VkXnb51nqYgqUrLSMulfbUX3o= 2024-02-01T10:47:23.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:T5VkXnb51nqYgqUrLSMulfbUX3o=', 'Content-Length': '0'} 2024-02-01T10:47:23.347 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aac1fe34212ad35f-0065bb76bb-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:23.347 DEBUG:boto:b'vclfcn-60obj11000falseobj12024-02-01T10:46:46.243Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:23.347 DEBUG:boto:path=/vclfcn-60/obj1 2024-02-01T10:47:23.348 DEBUG:boto:auth_path=/vclfcn-60/obj1 2024-02-01T10:47:23.348 DEBUG:boto:Method: HEAD 2024-02-01T10:47:23.348 DEBUG:boto:Path: /vclfcn-60/obj1 2024-02-01T10:47:23.348 DEBUG:boto:Data: 2024-02-01T10:47:23.348 DEBUG:boto:Headers: {} 2024-02-01T10:47:23.348 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:23.348 DEBUG:boto:Port: 8001 2024-02-01T10:47:23.348 DEBUG:boto:Params: {} 2024-02-01T10:47:23.348 DEBUG:boto:Token: None 2024-02-01T10:47:23.348 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:23 GMT /vclfcn-60/obj1 2024-02-01T10:47:23.349 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vas2pWCujt30bp6Wy4M/m6oEJck= 2024-02-01T10:47:23.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vas2pWCujt30bp6Wy4M/m6oEJck=', 'Content-Length': '0'} 2024-02-01T10:47:23.352 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f6c12fe2af5ccfdc-0065bb76bb-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:23.352 DEBUG:boto:path=/vclfcn-60/obj1 2024-02-01T10:47:23.352 DEBUG:boto:auth_path=/vclfcn-60/obj1 2024-02-01T10:47:23.352 DEBUG:boto:Method: HEAD 2024-02-01T10:47:23.352 DEBUG:boto:Path: /vclfcn-60/obj1 2024-02-01T10:47:23.352 DEBUG:boto:Data: 2024-02-01T10:47:23.352 DEBUG:boto:Headers: {} 2024-02-01T10:47:23.352 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:23.353 DEBUG:boto:Port: 8000 2024-02-01T10:47:23.353 DEBUG:boto:Params: {} 2024-02-01T10:47:23.353 DEBUG:boto:Token: None 2024-02-01T10:47:23.353 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:23 GMT /vclfcn-60/obj1 2024-02-01T10:47:23.353 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vas2pWCujt30bp6Wy4M/m6oEJck= 2024-02-01T10:47:23.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vas2pWCujt30bp6Wy4M/m6oEJck=', 'Content-Length': '0'} 2024-02-01T10:47:23.356 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005688451ebec43751-0065bb76bb-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:23.356 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:47:23.356 DEBUG:boto:path=/vclfcn-60/obj1 2024-02-01T10:47:23.356 DEBUG:boto:auth_path=/vclfcn-60/obj1 2024-02-01T10:47:23.356 DEBUG:boto:Method: GET 2024-02-01T10:47:23.356 DEBUG:boto:Path: /vclfcn-60/obj1 2024-02-01T10:47:23.357 DEBUG:boto:Data: 2024-02-01T10:47:23.357 DEBUG:boto:Headers: {} 2024-02-01T10:47:23.357 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:23.357 DEBUG:boto:Port: 8001 2024-02-01T10:47:23.357 DEBUG:boto:Params: {} 2024-02-01T10:47:23.357 DEBUG:boto:Token: None 2024-02-01T10:47:23.357 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:23 GMT /vclfcn-60/obj1 2024-02-01T10:47:23.357 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:bq5WCBNLXcybP267oZNj/F+HFOY= 2024-02-01T10:47:23.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:bq5WCBNLXcybP267oZNj/F+HFOY=', 'Content-Length': '0'} 2024-02-01T10:47:23.360 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000763e95193464e767-0065bb76bb-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:23.408 DEBUG:boto:path=/vclfcn-60/obj1 2024-02-01T10:47:23.408 DEBUG:boto:auth_path=/vclfcn-60/obj1 2024-02-01T10:47:23.408 DEBUG:boto:Method: GET 2024-02-01T10:47:23.408 DEBUG:boto:Path: /vclfcn-60/obj1 2024-02-01T10:47:23.408 DEBUG:boto:Data: 2024-02-01T10:47:23.409 DEBUG:boto:Headers: {} 2024-02-01T10:47:23.409 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:23.409 DEBUG:boto:Port: 8000 2024-02-01T10:47:23.409 DEBUG:boto:Params: {} 2024-02-01T10:47:23.409 DEBUG:boto:Token: None 2024-02-01T10:47:23.409 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:23 GMT /vclfcn-60/obj1 2024-02-01T10:47:23.409 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:bq5WCBNLXcybP267oZNj/F+HFOY= 2024-02-01T10:47:23.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:bq5WCBNLXcybP267oZNj/F+HFOY=', 'Content-Length': '0'} 2024-02-01T10:47:23.412 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000059bd5c71d70c7199-0065bb76bb-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:23.412 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:47:23.412 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:47:23.412 DEBUG:boto:path=/vclfcn-60/?prefix=obj1 2024-02-01T10:47:23.413 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj1 2024-02-01T10:47:23.413 DEBUG:boto:Method: GET 2024-02-01T10:47:23.413 DEBUG:boto:Path: /vclfcn-60/?prefix=obj1 2024-02-01T10:47:23.413 DEBUG:boto:Data: 2024-02-01T10:47:23.413 DEBUG:boto:Headers: {} 2024-02-01T10:47:23.413 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:23.413 DEBUG:boto:Port: 8001 2024-02-01T10:47:23.413 DEBUG:boto:Params: {} 2024-02-01T10:47:23.413 DEBUG:boto:Token: None 2024-02-01T10:47:23.413 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:23 GMT /vclfcn-60/ 2024-02-01T10:47:23.414 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:T5VkXnb51nqYgqUrLSMulfbUX3o= 2024-02-01T10:47:23.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:T5VkXnb51nqYgqUrLSMulfbUX3o=', 'Content-Length': '0'} 2024-02-01T10:47:23.418 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f6dbdf7fdef960b-0065bb76bb-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:23.418 DEBUG:boto:b'vclfcn-60obj11000falseobj12024-02-01T10:46:46.243Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:23.419 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:47:23.419 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:47:23.419 DEBUG:boto:path=/vclfcn-60/?prefix=obj1 2024-02-01T10:47:23.419 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj1 2024-02-01T10:47:23.419 DEBUG:boto:Method: GET 2024-02-01T10:47:23.419 DEBUG:boto:Path: /vclfcn-60/?prefix=obj1 2024-02-01T10:47:23.419 DEBUG:boto:Data: 2024-02-01T10:47:23.419 DEBUG:boto:Headers: {} 2024-02-01T10:47:23.419 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:23.419 DEBUG:boto:Port: 8000 2024-02-01T10:47:23.420 DEBUG:boto:Params: {} 2024-02-01T10:47:23.420 DEBUG:boto:Token: None 2024-02-01T10:47:23.420 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:23 GMT /vclfcn-60/ 2024-02-01T10:47:23.420 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:T5VkXnb51nqYgqUrLSMulfbUX3o= 2024-02-01T10:47:23.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:T5VkXnb51nqYgqUrLSMulfbUX3o=', 'Content-Length': '0'} 2024-02-01T10:47:23.424 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da38e53631c09595-0065bb76bb-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:23.425 DEBUG:boto:b'vclfcn-60obj11000falseobj12024-02-01T10:46:46.243Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:23.425 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:47:23.425 DEBUG:boto:path=/vclfcn-60/obj2 2024-02-01T10:47:23.425 DEBUG:boto:auth_path=/vclfcn-60/obj2 2024-02-01T10:47:23.425 DEBUG:boto:path=/vclfcn-60/obj2?versionId=null 2024-02-01T10:47:23.425 DEBUG:boto:auth_path=/vclfcn-60/obj2?versionId=null 2024-02-01T10:47:23.426 DEBUG:boto:Method: GET 2024-02-01T10:47:23.426 DEBUG:boto:Path: /vclfcn-60/obj2?versionId=null 2024-02-01T10:47:23.426 DEBUG:boto:Data: 2024-02-01T10:47:23.426 DEBUG:boto:Headers: {} 2024-02-01T10:47:23.426 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:23.426 DEBUG:boto:Port: 8001 2024-02-01T10:47:23.426 DEBUG:boto:Params: {} 2024-02-01T10:47:23.426 DEBUG:boto:Token: None 2024-02-01T10:47:23.426 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:23 GMT /vclfcn-60/obj2?versionId=null 2024-02-01T10:47:23.426 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lN6r/Sxvc6Mh/MefUDJ9GP44DXw= 2024-02-01T10:47:23.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lN6r/Sxvc6Mh/MefUDJ9GP44DXw=', 'Content-Length': '0'} 2024-02-01T10:47:23.429 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b0b04832c7e9ba2b-0065bb76bb-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:23.472 DEBUG:boto:path=/vclfcn-60/obj2 2024-02-01T10:47:23.472 DEBUG:boto:auth_path=/vclfcn-60/obj2 2024-02-01T10:47:23.472 DEBUG:boto:path=/vclfcn-60/obj2?versionId=null 2024-02-01T10:47:23.473 DEBUG:boto:auth_path=/vclfcn-60/obj2?versionId=null 2024-02-01T10:47:23.473 DEBUG:boto:Method: GET 2024-02-01T10:47:23.473 DEBUG:boto:Path: /vclfcn-60/obj2?versionId=null 2024-02-01T10:47:23.473 DEBUG:boto:Data: 2024-02-01T10:47:23.473 DEBUG:boto:Headers: {} 2024-02-01T10:47:23.473 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:23.473 DEBUG:boto:Port: 8000 2024-02-01T10:47:23.473 DEBUG:boto:Params: {} 2024-02-01T10:47:23.473 DEBUG:boto:Token: None 2024-02-01T10:47:23.473 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:23 GMT /vclfcn-60/obj2?versionId=null 2024-02-01T10:47:23.474 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lN6r/Sxvc6Mh/MefUDJ9GP44DXw= 2024-02-01T10:47:23.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lN6r/Sxvc6Mh/MefUDJ9GP44DXw=', 'Content-Length': '0'} 2024-02-01T10:47:23.477 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000df4aac25c1b0aa83-0065bb76bb-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:23.477 DEBUG:boto:path=/vclfcn-60/obj2 2024-02-01T10:47:23.477 DEBUG:boto:auth_path=/vclfcn-60/obj2 2024-02-01T10:47:23.477 DEBUG:boto:path=/vclfcn-60/obj2?versionId=null 2024-02-01T10:47:23.477 DEBUG:boto:auth_path=/vclfcn-60/obj2?versionId=null 2024-02-01T10:47:23.477 DEBUG:boto:Method: HEAD 2024-02-01T10:47:23.477 DEBUG:boto:Path: /vclfcn-60/obj2?versionId=null 2024-02-01T10:47:23.477 DEBUG:boto:Data: 2024-02-01T10:47:23.478 DEBUG:boto:Headers: {} 2024-02-01T10:47:23.478 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:23.478 DEBUG:boto:Port: 8001 2024-02-01T10:47:23.478 DEBUG:boto:Params: {} 2024-02-01T10:47:23.478 DEBUG:boto:Token: None 2024-02-01T10:47:23.478 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:23 GMT /vclfcn-60/obj2?versionId=null 2024-02-01T10:47:23.478 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:i+f3dGhpT+UCNJusNVkDIeYVbJQ= 2024-02-01T10:47:23.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:i+f3dGhpT+UCNJusNVkDIeYVbJQ=', 'Content-Length': '0'} 2024-02-01T10:47:23.481 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000028c8ff9ca4e7a5de-0065bb76bb-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:23.481 DEBUG:boto:path=/vclfcn-60/obj2 2024-02-01T10:47:23.481 DEBUG:boto:auth_path=/vclfcn-60/obj2 2024-02-01T10:47:23.481 DEBUG:boto:path=/vclfcn-60/obj2?versionId=null 2024-02-01T10:47:23.481 DEBUG:boto:auth_path=/vclfcn-60/obj2?versionId=null 2024-02-01T10:47:23.482 DEBUG:boto:Method: HEAD 2024-02-01T10:47:23.482 DEBUG:boto:Path: /vclfcn-60/obj2?versionId=null 2024-02-01T10:47:23.482 DEBUG:boto:Data: 2024-02-01T10:47:23.482 DEBUG:boto:Headers: {} 2024-02-01T10:47:23.482 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:23.482 DEBUG:boto:Port: 8000 2024-02-01T10:47:23.482 DEBUG:boto:Params: {} 2024-02-01T10:47:23.482 DEBUG:boto:Token: None 2024-02-01T10:47:23.482 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:23 GMT /vclfcn-60/obj2?versionId=null 2024-02-01T10:47:23.482 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:i+f3dGhpT+UCNJusNVkDIeYVbJQ= 2024-02-01T10:47:23.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:i+f3dGhpT+UCNJusNVkDIeYVbJQ=', 'Content-Length': '0'} 2024-02-01T10:47:23.485 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c7d574d07e7dd33c-0065bb76bb-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:23.486 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:47:23.486 DEBUG:boto:path=/vclfcn-60/obj2 2024-02-01T10:47:23.486 DEBUG:boto:auth_path=/vclfcn-60/obj2 2024-02-01T10:47:23.486 DEBUG:boto:path=/vclfcn-60/obj2?versionId=null 2024-02-01T10:47:23.486 DEBUG:boto:auth_path=/vclfcn-60/obj2?versionId=null 2024-02-01T10:47:23.486 DEBUG:boto:Method: GET 2024-02-01T10:47:23.486 DEBUG:boto:Path: /vclfcn-60/obj2?versionId=null 2024-02-01T10:47:23.486 DEBUG:boto:Data: 2024-02-01T10:47:23.486 DEBUG:boto:Headers: {} 2024-02-01T10:47:23.487 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:23.487 DEBUG:boto:Port: 8001 2024-02-01T10:47:23.487 DEBUG:boto:Params: {} 2024-02-01T10:47:23.487 DEBUG:boto:Token: None 2024-02-01T10:47:23.487 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:23 GMT /vclfcn-60/obj2?versionId=null 2024-02-01T10:47:23.487 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lN6r/Sxvc6Mh/MefUDJ9GP44DXw= 2024-02-01T10:47:23.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lN6r/Sxvc6Mh/MefUDJ9GP44DXw=', 'Content-Length': '0'} 2024-02-01T10:47:23.490 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000df2a8b3f47913441-0065bb76bb-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:23.532 DEBUG:boto:path=/vclfcn-60/obj2 2024-02-01T10:47:23.532 DEBUG:boto:auth_path=/vclfcn-60/obj2 2024-02-01T10:47:23.532 DEBUG:boto:path=/vclfcn-60/obj2?versionId=null 2024-02-01T10:47:23.532 DEBUG:boto:auth_path=/vclfcn-60/obj2?versionId=null 2024-02-01T10:47:23.533 DEBUG:boto:Method: GET 2024-02-01T10:47:23.533 DEBUG:boto:Path: /vclfcn-60/obj2?versionId=null 2024-02-01T10:47:23.533 DEBUG:boto:Data: 2024-02-01T10:47:23.533 DEBUG:boto:Headers: {} 2024-02-01T10:47:23.533 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:23.533 DEBUG:boto:Port: 8000 2024-02-01T10:47:23.533 DEBUG:boto:Params: {} 2024-02-01T10:47:23.533 DEBUG:boto:Token: None 2024-02-01T10:47:23.533 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:23 GMT /vclfcn-60/obj2?versionId=null 2024-02-01T10:47:23.534 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lN6r/Sxvc6Mh/MefUDJ9GP44DXw= 2024-02-01T10:47:23.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lN6r/Sxvc6Mh/MefUDJ9GP44DXw=', 'Content-Length': '0'} 2024-02-01T10:47:23.537 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e21568d24ed77cbd-0065bb76bb-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:23.537 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:47:23.537 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:47:23.537 DEBUG:boto:path=/vclfcn-60/?prefix=obj2 2024-02-01T10:47:23.537 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj2 2024-02-01T10:47:23.537 DEBUG:boto:Method: GET 2024-02-01T10:47:23.537 DEBUG:boto:Path: /vclfcn-60/?prefix=obj2 2024-02-01T10:47:23.538 DEBUG:boto:Data: 2024-02-01T10:47:23.538 DEBUG:boto:Headers: {} 2024-02-01T10:47:23.538 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:23.538 DEBUG:boto:Port: 8001 2024-02-01T10:47:23.538 DEBUG:boto:Params: {} 2024-02-01T10:47:23.538 DEBUG:boto:Token: None 2024-02-01T10:47:23.538 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:23 GMT /vclfcn-60/ 2024-02-01T10:47:23.538 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:T5VkXnb51nqYgqUrLSMulfbUX3o= 2024-02-01T10:47:23.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:T5VkXnb51nqYgqUrLSMulfbUX3o=', 'Content-Length': '0'} 2024-02-01T10:47:23.543 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008dda601cc050bac8-0065bb76bb-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:23.588 DEBUG:boto:b'vclfcn-60obj21000falseobj22024-02-01T10:46:46.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:23.588 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:47:23.588 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:47:23.589 DEBUG:boto:path=/vclfcn-60/?prefix=obj2 2024-02-01T10:47:23.589 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj2 2024-02-01T10:47:23.589 DEBUG:boto:Method: GET 2024-02-01T10:47:23.589 DEBUG:boto:Path: /vclfcn-60/?prefix=obj2 2024-02-01T10:47:23.589 DEBUG:boto:Data: 2024-02-01T10:47:23.589 DEBUG:boto:Headers: {} 2024-02-01T10:47:23.589 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:23.589 DEBUG:boto:Port: 8000 2024-02-01T10:47:23.589 DEBUG:boto:Params: {} 2024-02-01T10:47:23.589 DEBUG:boto:Token: None 2024-02-01T10:47:23.590 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:23 GMT /vclfcn-60/ 2024-02-01T10:47:23.590 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:T5VkXnb51nqYgqUrLSMulfbUX3o= 2024-02-01T10:47:23.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:T5VkXnb51nqYgqUrLSMulfbUX3o=', 'Content-Length': '0'} 2024-02-01T10:47:23.596 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000525cbf00448dd2b9-0065bb76bb-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:23.596 DEBUG:boto:b'vclfcn-60obj21000falseobj22024-02-01T10:46:46.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:23.596 DEBUG:boto:path=/vclfcn-60/obj2 2024-02-01T10:47:23.597 DEBUG:boto:auth_path=/vclfcn-60/obj2 2024-02-01T10:47:23.597 DEBUG:boto:Method: HEAD 2024-02-01T10:47:23.597 DEBUG:boto:Path: /vclfcn-60/obj2 2024-02-01T10:47:23.597 DEBUG:boto:Data: 2024-02-01T10:47:23.597 DEBUG:boto:Headers: {} 2024-02-01T10:47:23.597 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:23.597 DEBUG:boto:Port: 8001 2024-02-01T10:47:23.597 DEBUG:boto:Params: {} 2024-02-01T10:47:23.597 DEBUG:boto:Token: None 2024-02-01T10:47:23.598 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:23 GMT /vclfcn-60/obj2 2024-02-01T10:47:23.598 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:66XvNwzq5XzqxwWLFrfU6G8XE3E= 2024-02-01T10:47:23.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:66XvNwzq5XzqxwWLFrfU6G8XE3E=', 'Content-Length': '0'} 2024-02-01T10:47:23.601 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000067e78a3504a8c524-0065bb76bb-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:23.601 DEBUG:boto:path=/vclfcn-60/obj2 2024-02-01T10:47:23.601 DEBUG:boto:auth_path=/vclfcn-60/obj2 2024-02-01T10:47:23.601 DEBUG:boto:Method: HEAD 2024-02-01T10:47:23.601 DEBUG:boto:Path: /vclfcn-60/obj2 2024-02-01T10:47:23.601 DEBUG:boto:Data: 2024-02-01T10:47:23.601 DEBUG:boto:Headers: {} 2024-02-01T10:47:23.601 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:23.602 DEBUG:boto:Port: 8000 2024-02-01T10:47:23.602 DEBUG:boto:Params: {} 2024-02-01T10:47:23.602 DEBUG:boto:Token: None 2024-02-01T10:47:23.602 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:23 GMT /vclfcn-60/obj2 2024-02-01T10:47:23.602 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:66XvNwzq5XzqxwWLFrfU6G8XE3E= 2024-02-01T10:47:23.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:66XvNwzq5XzqxwWLFrfU6G8XE3E=', 'Content-Length': '0'} 2024-02-01T10:47:23.605 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001422ed12b16996cc-0065bb76bb-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:23.605 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:47:23.605 DEBUG:boto:path=/vclfcn-60/obj2 2024-02-01T10:47:23.605 DEBUG:boto:auth_path=/vclfcn-60/obj2 2024-02-01T10:47:23.605 DEBUG:boto:Method: GET 2024-02-01T10:47:23.606 DEBUG:boto:Path: /vclfcn-60/obj2 2024-02-01T10:47:23.606 DEBUG:boto:Data: 2024-02-01T10:47:23.606 DEBUG:boto:Headers: {} 2024-02-01T10:47:23.606 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:23.606 DEBUG:boto:Port: 8001 2024-02-01T10:47:23.606 DEBUG:boto:Params: {} 2024-02-01T10:47:23.606 DEBUG:boto:Token: None 2024-02-01T10:47:23.606 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:23 GMT /vclfcn-60/obj2 2024-02-01T10:47:23.606 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:bCmxSfsKYnx/TfCaadl2vUxkonc= 2024-02-01T10:47:23.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:bCmxSfsKYnx/TfCaadl2vUxkonc=', 'Content-Length': '0'} 2024-02-01T10:47:23.609 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000336b9b056d6f5821-0065bb76bb-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:23.652 DEBUG:boto:path=/vclfcn-60/obj2 2024-02-01T10:47:23.652 DEBUG:boto:auth_path=/vclfcn-60/obj2 2024-02-01T10:47:23.652 DEBUG:boto:Method: GET 2024-02-01T10:47:23.652 DEBUG:boto:Path: /vclfcn-60/obj2 2024-02-01T10:47:23.652 DEBUG:boto:Data: 2024-02-01T10:47:23.653 DEBUG:boto:Headers: {} 2024-02-01T10:47:23.653 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:23.653 DEBUG:boto:Port: 8000 2024-02-01T10:47:23.653 DEBUG:boto:Params: {} 2024-02-01T10:47:23.653 DEBUG:boto:Token: None 2024-02-01T10:47:23.653 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:23 GMT /vclfcn-60/obj2 2024-02-01T10:47:23.653 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:bCmxSfsKYnx/TfCaadl2vUxkonc= 2024-02-01T10:47:23.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:bCmxSfsKYnx/TfCaadl2vUxkonc=', 'Content-Length': '0'} 2024-02-01T10:47:23.656 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000ee2b6cb51134a3e-0065bb76bb-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:23.656 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:47:23.656 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:47:23.656 DEBUG:boto:path=/vclfcn-60/?prefix=obj2 2024-02-01T10:47:23.657 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj2 2024-02-01T10:47:23.657 DEBUG:boto:Method: GET 2024-02-01T10:47:23.657 DEBUG:boto:Path: /vclfcn-60/?prefix=obj2 2024-02-01T10:47:23.657 DEBUG:boto:Data: 2024-02-01T10:47:23.657 DEBUG:boto:Headers: {} 2024-02-01T10:47:23.657 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:23.657 DEBUG:boto:Port: 8001 2024-02-01T10:47:23.657 DEBUG:boto:Params: {} 2024-02-01T10:47:23.657 DEBUG:boto:Token: None 2024-02-01T10:47:23.657 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:23 GMT /vclfcn-60/ 2024-02-01T10:47:23.658 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:T5VkXnb51nqYgqUrLSMulfbUX3o= 2024-02-01T10:47:23.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:T5VkXnb51nqYgqUrLSMulfbUX3o=', 'Content-Length': '0'} 2024-02-01T10:47:23.662 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aaec471750b00e28-0065bb76bb-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:23.704 DEBUG:boto:b'vclfcn-60obj21000falseobj22024-02-01T10:46:46.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:23.705 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:47:23.705 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:47:23.705 DEBUG:boto:path=/vclfcn-60/?prefix=obj2 2024-02-01T10:47:23.705 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj2 2024-02-01T10:47:23.705 DEBUG:boto:Method: GET 2024-02-01T10:47:23.705 DEBUG:boto:Path: /vclfcn-60/?prefix=obj2 2024-02-01T10:47:23.705 DEBUG:boto:Data: 2024-02-01T10:47:23.705 DEBUG:boto:Headers: {} 2024-02-01T10:47:23.705 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:23.705 DEBUG:boto:Port: 8000 2024-02-01T10:47:23.705 DEBUG:boto:Params: {} 2024-02-01T10:47:23.706 DEBUG:boto:Token: None 2024-02-01T10:47:23.706 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:23 GMT /vclfcn-60/ 2024-02-01T10:47:23.706 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:T5VkXnb51nqYgqUrLSMulfbUX3o= 2024-02-01T10:47:23.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:T5VkXnb51nqYgqUrLSMulfbUX3o=', 'Content-Length': '0'} 2024-02-01T10:47:23.711 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067c23e37e3fe56dc-0065bb76bb-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:23.711 DEBUG:boto:b'vclfcn-60obj21000falseobj22024-02-01T10:46:46.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:23.712 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:47:23.712 DEBUG:boto:path=/vclfcn-60/obj3 2024-02-01T10:47:23.712 DEBUG:boto:auth_path=/vclfcn-60/obj3 2024-02-01T10:47:23.712 DEBUG:boto:path=/vclfcn-60/obj3?versionId=null 2024-02-01T10:47:23.712 DEBUG:boto:auth_path=/vclfcn-60/obj3?versionId=null 2024-02-01T10:47:23.712 DEBUG:boto:Method: GET 2024-02-01T10:47:23.712 DEBUG:boto:Path: /vclfcn-60/obj3?versionId=null 2024-02-01T10:47:23.712 DEBUG:boto:Data: 2024-02-01T10:47:23.712 DEBUG:boto:Headers: {} 2024-02-01T10:47:23.713 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:23.713 DEBUG:boto:Port: 8001 2024-02-01T10:47:23.713 DEBUG:boto:Params: {} 2024-02-01T10:47:23.713 DEBUG:boto:Token: None 2024-02-01T10:47:23.713 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:23 GMT /vclfcn-60/obj3?versionId=null 2024-02-01T10:47:23.713 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/MeYcSiz59RxsddFGTuwJT3TBzg= 2024-02-01T10:47:23.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/MeYcSiz59RxsddFGTuwJT3TBzg=', 'Content-Length': '0'} 2024-02-01T10:47:23.716 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001cc9a10affb18d5b-0065bb76bb-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:23.717 DEBUG:boto:path=/vclfcn-60/obj3 2024-02-01T10:47:23.717 DEBUG:boto:auth_path=/vclfcn-60/obj3 2024-02-01T10:47:23.717 DEBUG:boto:path=/vclfcn-60/obj3?versionId=null 2024-02-01T10:47:23.717 DEBUG:boto:auth_path=/vclfcn-60/obj3?versionId=null 2024-02-01T10:47:23.717 DEBUG:boto:Method: GET 2024-02-01T10:47:23.717 DEBUG:boto:Path: /vclfcn-60/obj3?versionId=null 2024-02-01T10:47:23.717 DEBUG:boto:Data: 2024-02-01T10:47:23.717 DEBUG:boto:Headers: {} 2024-02-01T10:47:23.718 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:23.718 DEBUG:boto:Port: 8000 2024-02-01T10:47:23.718 DEBUG:boto:Params: {} 2024-02-01T10:47:23.718 DEBUG:boto:Token: None 2024-02-01T10:47:23.718 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:23 GMT /vclfcn-60/obj3?versionId=null 2024-02-01T10:47:23.718 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/MeYcSiz59RxsddFGTuwJT3TBzg= 2024-02-01T10:47:23.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/MeYcSiz59RxsddFGTuwJT3TBzg=', 'Content-Length': '0'} 2024-02-01T10:47:23.721 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002c06a800a1e58219-0065bb76bb-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:23.721 DEBUG:boto:path=/vclfcn-60/obj3 2024-02-01T10:47:23.721 DEBUG:boto:auth_path=/vclfcn-60/obj3 2024-02-01T10:47:23.721 DEBUG:boto:path=/vclfcn-60/obj3?versionId=null 2024-02-01T10:47:23.721 DEBUG:boto:auth_path=/vclfcn-60/obj3?versionId=null 2024-02-01T10:47:23.722 DEBUG:boto:Method: HEAD 2024-02-01T10:47:23.722 DEBUG:boto:Path: /vclfcn-60/obj3?versionId=null 2024-02-01T10:47:23.722 DEBUG:boto:Data: 2024-02-01T10:47:23.722 DEBUG:boto:Headers: {} 2024-02-01T10:47:23.722 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:23.722 DEBUG:boto:Port: 8001 2024-02-01T10:47:23.722 DEBUG:boto:Params: {} 2024-02-01T10:47:23.722 DEBUG:boto:Token: None 2024-02-01T10:47:23.722 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:23 GMT /vclfcn-60/obj3?versionId=null 2024-02-01T10:47:23.722 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sHfGiXhyB1IJqECJQ1G0m9oCr80= 2024-02-01T10:47:23.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sHfGiXhyB1IJqECJQ1G0m9oCr80=', 'Content-Length': '0'} 2024-02-01T10:47:23.725 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e62f592c445508f6-0065bb76bb-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:23.725 DEBUG:boto:path=/vclfcn-60/obj3 2024-02-01T10:47:23.725 DEBUG:boto:auth_path=/vclfcn-60/obj3 2024-02-01T10:47:23.726 DEBUG:boto:path=/vclfcn-60/obj3?versionId=null 2024-02-01T10:47:23.726 DEBUG:boto:auth_path=/vclfcn-60/obj3?versionId=null 2024-02-01T10:47:23.726 DEBUG:boto:Method: HEAD 2024-02-01T10:47:23.726 DEBUG:boto:Path: /vclfcn-60/obj3?versionId=null 2024-02-01T10:47:23.726 DEBUG:boto:Data: 2024-02-01T10:47:23.726 DEBUG:boto:Headers: {} 2024-02-01T10:47:23.726 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:23.726 DEBUG:boto:Port: 8000 2024-02-01T10:47:23.726 DEBUG:boto:Params: {} 2024-02-01T10:47:23.726 DEBUG:boto:Token: None 2024-02-01T10:47:23.726 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:23 GMT /vclfcn-60/obj3?versionId=null 2024-02-01T10:47:23.727 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sHfGiXhyB1IJqECJQ1G0m9oCr80= 2024-02-01T10:47:23.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sHfGiXhyB1IJqECJQ1G0m9oCr80=', 'Content-Length': '0'} 2024-02-01T10:47:23.729 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c419929fc56f5a3d-0065bb76bb-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:23.730 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:47:23.730 DEBUG:boto:path=/vclfcn-60/obj3 2024-02-01T10:47:23.730 DEBUG:boto:auth_path=/vclfcn-60/obj3 2024-02-01T10:47:23.730 DEBUG:boto:path=/vclfcn-60/obj3?versionId=null 2024-02-01T10:47:23.730 DEBUG:boto:auth_path=/vclfcn-60/obj3?versionId=null 2024-02-01T10:47:23.730 DEBUG:boto:Method: GET 2024-02-01T10:47:23.730 DEBUG:boto:Path: /vclfcn-60/obj3?versionId=null 2024-02-01T10:47:23.731 DEBUG:boto:Data: 2024-02-01T10:47:23.731 DEBUG:boto:Headers: {} 2024-02-01T10:47:23.731 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:23.731 DEBUG:boto:Port: 8001 2024-02-01T10:47:23.731 DEBUG:boto:Params: {} 2024-02-01T10:47:23.731 DEBUG:boto:Token: None 2024-02-01T10:47:23.731 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:23 GMT /vclfcn-60/obj3?versionId=null 2024-02-01T10:47:23.731 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/MeYcSiz59RxsddFGTuwJT3TBzg= 2024-02-01T10:47:23.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/MeYcSiz59RxsddFGTuwJT3TBzg=', 'Content-Length': '0'} 2024-02-01T10:47:23.734 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ea287ad6669307d3-0065bb76bb-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:23.776 DEBUG:boto:path=/vclfcn-60/obj3 2024-02-01T10:47:23.776 DEBUG:boto:auth_path=/vclfcn-60/obj3 2024-02-01T10:47:23.776 DEBUG:boto:path=/vclfcn-60/obj3?versionId=null 2024-02-01T10:47:23.776 DEBUG:boto:auth_path=/vclfcn-60/obj3?versionId=null 2024-02-01T10:47:23.776 DEBUG:boto:Method: GET 2024-02-01T10:47:23.776 DEBUG:boto:Path: /vclfcn-60/obj3?versionId=null 2024-02-01T10:47:23.777 DEBUG:boto:Data: 2024-02-01T10:47:23.777 DEBUG:boto:Headers: {} 2024-02-01T10:47:23.777 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:23.777 DEBUG:boto:Port: 8000 2024-02-01T10:47:23.777 DEBUG:boto:Params: {} 2024-02-01T10:47:23.777 DEBUG:boto:Token: None 2024-02-01T10:47:23.777 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:23 GMT /vclfcn-60/obj3?versionId=null 2024-02-01T10:47:23.777 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/MeYcSiz59RxsddFGTuwJT3TBzg= 2024-02-01T10:47:23.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/MeYcSiz59RxsddFGTuwJT3TBzg=', 'Content-Length': '0'} 2024-02-01T10:47:23.780 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002dd997de199d327f-0065bb76bb-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:23.780 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:47:23.780 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:47:23.781 DEBUG:boto:path=/vclfcn-60/?prefix=obj3 2024-02-01T10:47:23.781 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj3 2024-02-01T10:47:23.781 DEBUG:boto:Method: GET 2024-02-01T10:47:23.781 DEBUG:boto:Path: /vclfcn-60/?prefix=obj3 2024-02-01T10:47:23.781 DEBUG:boto:Data: 2024-02-01T10:47:23.781 DEBUG:boto:Headers: {} 2024-02-01T10:47:23.781 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:23.781 DEBUG:boto:Port: 8001 2024-02-01T10:47:23.781 DEBUG:boto:Params: {} 2024-02-01T10:47:23.781 DEBUG:boto:Token: None 2024-02-01T10:47:23.782 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:23 GMT /vclfcn-60/ 2024-02-01T10:47:23.782 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:T5VkXnb51nqYgqUrLSMulfbUX3o= 2024-02-01T10:47:23.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:T5VkXnb51nqYgqUrLSMulfbUX3o=', 'Content-Length': '0'} 2024-02-01T10:47:23.786 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091acd5400d17c577-0065bb76bb-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:23.787 DEBUG:boto:b'vclfcn-60obj31000falseobj32024-02-01T10:46:46.285Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:23.787 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:47:23.787 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:47:23.787 DEBUG:boto:path=/vclfcn-60/?prefix=obj3 2024-02-01T10:47:23.788 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj3 2024-02-01T10:47:23.788 DEBUG:boto:Method: GET 2024-02-01T10:47:23.788 DEBUG:boto:Path: /vclfcn-60/?prefix=obj3 2024-02-01T10:47:23.788 DEBUG:boto:Data: 2024-02-01T10:47:23.788 DEBUG:boto:Headers: {} 2024-02-01T10:47:23.788 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:23.788 DEBUG:boto:Port: 8000 2024-02-01T10:47:23.788 DEBUG:boto:Params: {} 2024-02-01T10:47:23.788 DEBUG:boto:Token: None 2024-02-01T10:47:23.788 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:23 GMT /vclfcn-60/ 2024-02-01T10:47:23.789 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:T5VkXnb51nqYgqUrLSMulfbUX3o= 2024-02-01T10:47:23.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:T5VkXnb51nqYgqUrLSMulfbUX3o=', 'Content-Length': '0'} 2024-02-01T10:47:23.793 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d785549186d4541d-0065bb76bb-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:23.836 DEBUG:boto:b'vclfcn-60obj31000falseobj32024-02-01T10:46:46.285Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:23.837 DEBUG:boto:path=/vclfcn-60/obj3 2024-02-01T10:47:23.837 DEBUG:boto:auth_path=/vclfcn-60/obj3 2024-02-01T10:47:23.837 DEBUG:boto:Method: HEAD 2024-02-01T10:47:23.837 DEBUG:boto:Path: /vclfcn-60/obj3 2024-02-01T10:47:23.837 DEBUG:boto:Data: 2024-02-01T10:47:23.837 DEBUG:boto:Headers: {} 2024-02-01T10:47:23.837 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:23.837 DEBUG:boto:Port: 8001 2024-02-01T10:47:23.837 DEBUG:boto:Params: {} 2024-02-01T10:47:23.838 DEBUG:boto:Token: None 2024-02-01T10:47:23.838 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:23 GMT /vclfcn-60/obj3 2024-02-01T10:47:23.838 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:uRFBKBg5HfJlek5fwOFtanahbME= 2024-02-01T10:47:23.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:uRFBKBg5HfJlek5fwOFtanahbME=', 'Content-Length': '0'} 2024-02-01T10:47:23.841 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c81b302e4186d50a-0065bb76bb-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:23.841 DEBUG:boto:path=/vclfcn-60/obj3 2024-02-01T10:47:23.841 DEBUG:boto:auth_path=/vclfcn-60/obj3 2024-02-01T10:47:23.841 DEBUG:boto:Method: HEAD 2024-02-01T10:47:23.841 DEBUG:boto:Path: /vclfcn-60/obj3 2024-02-01T10:47:23.841 DEBUG:boto:Data: 2024-02-01T10:47:23.842 DEBUG:boto:Headers: {} 2024-02-01T10:47:23.842 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:23.842 DEBUG:boto:Port: 8000 2024-02-01T10:47:23.842 DEBUG:boto:Params: {} 2024-02-01T10:47:23.842 DEBUG:boto:Token: None 2024-02-01T10:47:23.842 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:23 GMT /vclfcn-60/obj3 2024-02-01T10:47:23.842 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:uRFBKBg5HfJlek5fwOFtanahbME= 2024-02-01T10:47:23.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:uRFBKBg5HfJlek5fwOFtanahbME=', 'Content-Length': '0'} 2024-02-01T10:47:23.845 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f295c58c4365cb14-0065bb76bb-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:23.845 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:47:23.845 DEBUG:boto:path=/vclfcn-60/obj3 2024-02-01T10:47:23.845 DEBUG:boto:auth_path=/vclfcn-60/obj3 2024-02-01T10:47:23.846 DEBUG:boto:Method: GET 2024-02-01T10:47:23.846 DEBUG:boto:Path: /vclfcn-60/obj3 2024-02-01T10:47:23.846 DEBUG:boto:Data: 2024-02-01T10:47:23.846 DEBUG:boto:Headers: {} 2024-02-01T10:47:23.846 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:23.846 DEBUG:boto:Port: 8001 2024-02-01T10:47:23.846 DEBUG:boto:Params: {} 2024-02-01T10:47:23.846 DEBUG:boto:Token: None 2024-02-01T10:47:23.846 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:23 GMT /vclfcn-60/obj3 2024-02-01T10:47:23.847 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WFrnIoULF6XQ+J2zsKrKc3y2wTQ= 2024-02-01T10:47:23.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WFrnIoULF6XQ+J2zsKrKc3y2wTQ=', 'Content-Length': '0'} 2024-02-01T10:47:23.849 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000038d9f41df6eca848-0065bb76bb-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:23.892 DEBUG:boto:path=/vclfcn-60/obj3 2024-02-01T10:47:23.892 DEBUG:boto:auth_path=/vclfcn-60/obj3 2024-02-01T10:47:23.892 DEBUG:boto:Method: GET 2024-02-01T10:47:23.892 DEBUG:boto:Path: /vclfcn-60/obj3 2024-02-01T10:47:23.893 DEBUG:boto:Data: 2024-02-01T10:47:23.893 DEBUG:boto:Headers: {} 2024-02-01T10:47:23.893 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:23.893 DEBUG:boto:Port: 8000 2024-02-01T10:47:23.893 DEBUG:boto:Params: {} 2024-02-01T10:47:23.893 DEBUG:boto:Token: None 2024-02-01T10:47:23.893 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:23 GMT /vclfcn-60/obj3 2024-02-01T10:47:23.893 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WFrnIoULF6XQ+J2zsKrKc3y2wTQ= 2024-02-01T10:47:23.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WFrnIoULF6XQ+J2zsKrKc3y2wTQ=', 'Content-Length': '0'} 2024-02-01T10:47:23.897 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000034672a117059d69b-0065bb76bb-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:23.897 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:47:23.897 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:47:23.897 DEBUG:boto:path=/vclfcn-60/?prefix=obj3 2024-02-01T10:47:23.897 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj3 2024-02-01T10:47:23.897 DEBUG:boto:Method: GET 2024-02-01T10:47:23.898 DEBUG:boto:Path: /vclfcn-60/?prefix=obj3 2024-02-01T10:47:23.898 DEBUG:boto:Data: 2024-02-01T10:47:23.898 DEBUG:boto:Headers: {} 2024-02-01T10:47:23.898 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:23.898 DEBUG:boto:Port: 8001 2024-02-01T10:47:23.898 DEBUG:boto:Params: {} 2024-02-01T10:47:23.898 DEBUG:boto:Token: None 2024-02-01T10:47:23.898 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:23 GMT /vclfcn-60/ 2024-02-01T10:47:23.898 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:T5VkXnb51nqYgqUrLSMulfbUX3o= 2024-02-01T10:47:23.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:T5VkXnb51nqYgqUrLSMulfbUX3o=', 'Content-Length': '0'} 2024-02-01T10:47:23.903 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000840e214ae7211207-0065bb76bb-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:23.944 DEBUG:boto:b'vclfcn-60obj31000falseobj32024-02-01T10:46:46.285Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:23.944 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:47:23.945 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:47:23.945 DEBUG:boto:path=/vclfcn-60/?prefix=obj3 2024-02-01T10:47:23.945 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj3 2024-02-01T10:47:23.945 DEBUG:boto:Method: GET 2024-02-01T10:47:23.945 DEBUG:boto:Path: /vclfcn-60/?prefix=obj3 2024-02-01T10:47:23.945 DEBUG:boto:Data: 2024-02-01T10:47:23.945 DEBUG:boto:Headers: {} 2024-02-01T10:47:23.945 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:23.945 DEBUG:boto:Port: 8000 2024-02-01T10:47:23.945 DEBUG:boto:Params: {} 2024-02-01T10:47:23.945 DEBUG:boto:Token: None 2024-02-01T10:47:23.946 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:23 GMT /vclfcn-60/ 2024-02-01T10:47:23.946 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:T5VkXnb51nqYgqUrLSMulfbUX3o= 2024-02-01T10:47:23.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:T5VkXnb51nqYgqUrLSMulfbUX3o=', 'Content-Length': '0'} 2024-02-01T10:47:23.951 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d53ab624ae55ee4-0065bb76bb-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:23.951 DEBUG:boto:b'vclfcn-60obj31000falseobj32024-02-01T10:46:46.285Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:23.952 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:47:23.952 DEBUG:boto:path=/vclfcn-60/obj4 2024-02-01T10:47:23.952 DEBUG:boto:auth_path=/vclfcn-60/obj4 2024-02-01T10:47:23.952 DEBUG:boto:path=/vclfcn-60/obj4?versionId=null 2024-02-01T10:47:23.952 DEBUG:boto:auth_path=/vclfcn-60/obj4?versionId=null 2024-02-01T10:47:23.953 DEBUG:boto:Method: GET 2024-02-01T10:47:23.953 DEBUG:boto:Path: /vclfcn-60/obj4?versionId=null 2024-02-01T10:47:23.953 DEBUG:boto:Data: 2024-02-01T10:47:23.953 DEBUG:boto:Headers: {} 2024-02-01T10:47:23.953 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:23.953 DEBUG:boto:Port: 8001 2024-02-01T10:47:23.953 DEBUG:boto:Params: {} 2024-02-01T10:47:23.953 DEBUG:boto:Token: None 2024-02-01T10:47:23.953 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:23 GMT /vclfcn-60/obj4?versionId=null 2024-02-01T10:47:23.954 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:31s+CR8XfXDEKNjpMG3oCY5/A+Y= 2024-02-01T10:47:23.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:31s+CR8XfXDEKNjpMG3oCY5/A+Y=', 'Content-Length': '0'} 2024-02-01T10:47:23.957 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cf3a285ba979e1ff-0065bb76bb-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:23.957 DEBUG:boto:path=/vclfcn-60/obj4 2024-02-01T10:47:23.957 DEBUG:boto:auth_path=/vclfcn-60/obj4 2024-02-01T10:47:23.957 DEBUG:boto:path=/vclfcn-60/obj4?versionId=null 2024-02-01T10:47:23.958 DEBUG:boto:auth_path=/vclfcn-60/obj4?versionId=null 2024-02-01T10:47:23.958 DEBUG:boto:Method: GET 2024-02-01T10:47:23.958 DEBUG:boto:Path: /vclfcn-60/obj4?versionId=null 2024-02-01T10:47:23.958 DEBUG:boto:Data: 2024-02-01T10:47:23.958 DEBUG:boto:Headers: {} 2024-02-01T10:47:23.958 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:23.958 DEBUG:boto:Port: 8000 2024-02-01T10:47:23.958 DEBUG:boto:Params: {} 2024-02-01T10:47:23.958 DEBUG:boto:Token: None 2024-02-01T10:47:23.958 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:23 GMT /vclfcn-60/obj4?versionId=null 2024-02-01T10:47:23.958 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:31s+CR8XfXDEKNjpMG3oCY5/A+Y= 2024-02-01T10:47:23.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:31s+CR8XfXDEKNjpMG3oCY5/A+Y=', 'Content-Length': '0'} 2024-02-01T10:47:23.961 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cde974517c04b7c2-0065bb76bb-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:23.962 DEBUG:boto:path=/vclfcn-60/obj4 2024-02-01T10:47:23.962 DEBUG:boto:auth_path=/vclfcn-60/obj4 2024-02-01T10:47:23.962 DEBUG:boto:path=/vclfcn-60/obj4?versionId=null 2024-02-01T10:47:23.962 DEBUG:boto:auth_path=/vclfcn-60/obj4?versionId=null 2024-02-01T10:47:23.962 DEBUG:boto:Method: HEAD 2024-02-01T10:47:23.962 DEBUG:boto:Path: /vclfcn-60/obj4?versionId=null 2024-02-01T10:47:23.962 DEBUG:boto:Data: 2024-02-01T10:47:23.962 DEBUG:boto:Headers: {} 2024-02-01T10:47:23.962 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:23.962 DEBUG:boto:Port: 8001 2024-02-01T10:47:23.963 DEBUG:boto:Params: {} 2024-02-01T10:47:23.963 DEBUG:boto:Token: None 2024-02-01T10:47:23.963 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:23 GMT /vclfcn-60/obj4?versionId=null 2024-02-01T10:47:23.963 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YWDs2Gre7gGD8hGR1FDd2gfGRME= 2024-02-01T10:47:23.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YWDs2Gre7gGD8hGR1FDd2gfGRME=', 'Content-Length': '0'} 2024-02-01T10:47:23.965 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ba5b721c29d65e15-0065bb76bb-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:23.966 DEBUG:boto:path=/vclfcn-60/obj4 2024-02-01T10:47:23.966 DEBUG:boto:auth_path=/vclfcn-60/obj4 2024-02-01T10:47:23.966 DEBUG:boto:path=/vclfcn-60/obj4?versionId=null 2024-02-01T10:47:23.966 DEBUG:boto:auth_path=/vclfcn-60/obj4?versionId=null 2024-02-01T10:47:23.966 DEBUG:boto:Method: HEAD 2024-02-01T10:47:23.966 DEBUG:boto:Path: /vclfcn-60/obj4?versionId=null 2024-02-01T10:47:23.966 DEBUG:boto:Data: 2024-02-01T10:47:23.966 DEBUG:boto:Headers: {} 2024-02-01T10:47:23.966 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:23.967 DEBUG:boto:Port: 8000 2024-02-01T10:47:23.967 DEBUG:boto:Params: {} 2024-02-01T10:47:23.967 DEBUG:boto:Token: None 2024-02-01T10:47:23.967 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:23 GMT /vclfcn-60/obj4?versionId=null 2024-02-01T10:47:23.967 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YWDs2Gre7gGD8hGR1FDd2gfGRME= 2024-02-01T10:47:23.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YWDs2Gre7gGD8hGR1FDd2gfGRME=', 'Content-Length': '0'} 2024-02-01T10:47:23.970 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000045b915ffddc36877-0065bb76bb-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:23.970 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:47:23.970 DEBUG:boto:path=/vclfcn-60/obj4 2024-02-01T10:47:23.970 DEBUG:boto:auth_path=/vclfcn-60/obj4 2024-02-01T10:47:23.971 DEBUG:boto:path=/vclfcn-60/obj4?versionId=null 2024-02-01T10:47:23.971 DEBUG:boto:auth_path=/vclfcn-60/obj4?versionId=null 2024-02-01T10:47:23.971 DEBUG:boto:Method: GET 2024-02-01T10:47:23.971 DEBUG:boto:Path: /vclfcn-60/obj4?versionId=null 2024-02-01T10:47:23.971 DEBUG:boto:Data: 2024-02-01T10:47:23.971 DEBUG:boto:Headers: {} 2024-02-01T10:47:23.971 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:23.971 DEBUG:boto:Port: 8001 2024-02-01T10:47:23.971 DEBUG:boto:Params: {} 2024-02-01T10:47:23.971 DEBUG:boto:Token: None 2024-02-01T10:47:23.971 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:23 GMT /vclfcn-60/obj4?versionId=null 2024-02-01T10:47:23.972 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:31s+CR8XfXDEKNjpMG3oCY5/A+Y= 2024-02-01T10:47:23.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:31s+CR8XfXDEKNjpMG3oCY5/A+Y=', 'Content-Length': '0'} 2024-02-01T10:47:23.974 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000af50550dc8c18110-0065bb76bb-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:24.016 DEBUG:boto:path=/vclfcn-60/obj4 2024-02-01T10:47:24.016 DEBUG:boto:auth_path=/vclfcn-60/obj4 2024-02-01T10:47:24.016 DEBUG:boto:path=/vclfcn-60/obj4?versionId=null 2024-02-01T10:47:24.017 DEBUG:boto:auth_path=/vclfcn-60/obj4?versionId=null 2024-02-01T10:47:24.017 DEBUG:boto:Method: GET 2024-02-01T10:47:24.017 DEBUG:boto:Path: /vclfcn-60/obj4?versionId=null 2024-02-01T10:47:24.017 DEBUG:boto:Data: 2024-02-01T10:47:24.017 DEBUG:boto:Headers: {} 2024-02-01T10:47:24.017 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:24.017 DEBUG:boto:Port: 8000 2024-02-01T10:47:24.017 DEBUG:boto:Params: {} 2024-02-01T10:47:24.017 DEBUG:boto:Token: None 2024-02-01T10:47:24.017 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:24 GMT /vclfcn-60/obj4?versionId=null 2024-02-01T10:47:24.018 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:rbpieV3+JrASJ3PneiXaxgYAq+s= 2024-02-01T10:47:24.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:rbpieV3+JrASJ3PneiXaxgYAq+s=', 'Content-Length': '0'} 2024-02-01T10:47:24.021 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008f9ca4772b72ce3f-0065bb76bc-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:24.021 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:47:24.021 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:47:24.021 DEBUG:boto:path=/vclfcn-60/?prefix=obj4 2024-02-01T10:47:24.021 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj4 2024-02-01T10:47:24.022 DEBUG:boto:Method: GET 2024-02-01T10:47:24.022 DEBUG:boto:Path: /vclfcn-60/?prefix=obj4 2024-02-01T10:47:24.022 DEBUG:boto:Data: 2024-02-01T10:47:24.022 DEBUG:boto:Headers: {} 2024-02-01T10:47:24.022 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:24.022 DEBUG:boto:Port: 8001 2024-02-01T10:47:24.022 DEBUG:boto:Params: {} 2024-02-01T10:47:24.022 DEBUG:boto:Token: None 2024-02-01T10:47:24.022 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:24 GMT /vclfcn-60/ 2024-02-01T10:47:24.023 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:JV4WWQv0vNWo81FhwML/smo88OI= 2024-02-01T10:47:24.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:JV4WWQv0vNWo81FhwML/smo88OI=', 'Content-Length': '0'} 2024-02-01T10:47:24.027 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000277fb5f9c509f26e-0065bb76bc-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:24.027 DEBUG:boto:b'vclfcn-60obj41000falseobj42024-02-01T10:46:46.336Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:24.027 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:47:24.027 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:47:24.028 DEBUG:boto:path=/vclfcn-60/?prefix=obj4 2024-02-01T10:47:24.028 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj4 2024-02-01T10:47:24.028 DEBUG:boto:Method: GET 2024-02-01T10:47:24.028 DEBUG:boto:Path: /vclfcn-60/?prefix=obj4 2024-02-01T10:47:24.028 DEBUG:boto:Data: 2024-02-01T10:47:24.028 DEBUG:boto:Headers: {} 2024-02-01T10:47:24.028 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:24.028 DEBUG:boto:Port: 8000 2024-02-01T10:47:24.028 DEBUG:boto:Params: {} 2024-02-01T10:47:24.028 DEBUG:boto:Token: None 2024-02-01T10:47:24.029 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:24 GMT /vclfcn-60/ 2024-02-01T10:47:24.029 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:JV4WWQv0vNWo81FhwML/smo88OI= 2024-02-01T10:47:24.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:JV4WWQv0vNWo81FhwML/smo88OI=', 'Content-Length': '0'} 2024-02-01T10:47:24.033 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004fa166661c0f23c2-0065bb76bc-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:24.033 DEBUG:boto:b'vclfcn-60obj41000falseobj42024-02-01T10:46:46.336Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:24.034 DEBUG:boto:path=/vclfcn-60/obj4 2024-02-01T10:47:24.034 DEBUG:boto:auth_path=/vclfcn-60/obj4 2024-02-01T10:47:24.034 DEBUG:boto:Method: HEAD 2024-02-01T10:47:24.034 DEBUG:boto:Path: /vclfcn-60/obj4 2024-02-01T10:47:24.034 DEBUG:boto:Data: 2024-02-01T10:47:24.034 DEBUG:boto:Headers: {} 2024-02-01T10:47:24.034 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:24.034 DEBUG:boto:Port: 8001 2024-02-01T10:47:24.034 DEBUG:boto:Params: {} 2024-02-01T10:47:24.035 DEBUG:boto:Token: None 2024-02-01T10:47:24.035 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:24 GMT /vclfcn-60/obj4 2024-02-01T10:47:24.035 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sCdz2o3kShiz2JVbpr3j+ezlKZk= 2024-02-01T10:47:24.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sCdz2o3kShiz2JVbpr3j+ezlKZk=', 'Content-Length': '0'} 2024-02-01T10:47:24.037 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e82a22b9abf6b665-0065bb76bc-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:24.038 DEBUG:boto:path=/vclfcn-60/obj4 2024-02-01T10:47:24.038 DEBUG:boto:auth_path=/vclfcn-60/obj4 2024-02-01T10:47:24.038 DEBUG:boto:Method: HEAD 2024-02-01T10:47:24.038 DEBUG:boto:Path: /vclfcn-60/obj4 2024-02-01T10:47:24.038 DEBUG:boto:Data: 2024-02-01T10:47:24.038 DEBUG:boto:Headers: {} 2024-02-01T10:47:24.038 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:24.038 DEBUG:boto:Port: 8000 2024-02-01T10:47:24.038 DEBUG:boto:Params: {} 2024-02-01T10:47:24.039 DEBUG:boto:Token: None 2024-02-01T10:47:24.039 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:24 GMT /vclfcn-60/obj4 2024-02-01T10:47:24.039 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sCdz2o3kShiz2JVbpr3j+ezlKZk= 2024-02-01T10:47:24.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sCdz2o3kShiz2JVbpr3j+ezlKZk=', 'Content-Length': '0'} 2024-02-01T10:47:24.042 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004a683f50da8a1355-0065bb76bc-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:24.042 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:47:24.042 DEBUG:boto:path=/vclfcn-60/obj4 2024-02-01T10:47:24.042 DEBUG:boto:auth_path=/vclfcn-60/obj4 2024-02-01T10:47:24.042 DEBUG:boto:Method: GET 2024-02-01T10:47:24.042 DEBUG:boto:Path: /vclfcn-60/obj4 2024-02-01T10:47:24.042 DEBUG:boto:Data: 2024-02-01T10:47:24.043 DEBUG:boto:Headers: {} 2024-02-01T10:47:24.043 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:24.043 DEBUG:boto:Port: 8001 2024-02-01T10:47:24.043 DEBUG:boto:Params: {} 2024-02-01T10:47:24.043 DEBUG:boto:Token: None 2024-02-01T10:47:24.043 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:24 GMT /vclfcn-60/obj4 2024-02-01T10:47:24.043 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:qXvCnZMI4kL5meWkvtQlqG4OABs= 2024-02-01T10:47:24.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:qXvCnZMI4kL5meWkvtQlqG4OABs=', 'Content-Length': '0'} 2024-02-01T10:47:24.046 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000afa0a76e7a20b2c7-0065bb76bc-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:24.088 DEBUG:boto:path=/vclfcn-60/obj4 2024-02-01T10:47:24.088 DEBUG:boto:auth_path=/vclfcn-60/obj4 2024-02-01T10:47:24.088 DEBUG:boto:Method: GET 2024-02-01T10:47:24.088 DEBUG:boto:Path: /vclfcn-60/obj4 2024-02-01T10:47:24.089 DEBUG:boto:Data: 2024-02-01T10:47:24.089 DEBUG:boto:Headers: {} 2024-02-01T10:47:24.089 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:24.089 DEBUG:boto:Port: 8000 2024-02-01T10:47:24.089 DEBUG:boto:Params: {} 2024-02-01T10:47:24.089 DEBUG:boto:Token: None 2024-02-01T10:47:24.089 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:24 GMT /vclfcn-60/obj4 2024-02-01T10:47:24.089 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:qXvCnZMI4kL5meWkvtQlqG4OABs= 2024-02-01T10:47:24.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:qXvCnZMI4kL5meWkvtQlqG4OABs=', 'Content-Length': '0'} 2024-02-01T10:47:24.092 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005cd279990911d708-0065bb76bc-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:24.093 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:47:24.093 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:47:24.093 DEBUG:boto:path=/vclfcn-60/?prefix=obj4 2024-02-01T10:47:24.093 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj4 2024-02-01T10:47:24.093 DEBUG:boto:Method: GET 2024-02-01T10:47:24.093 DEBUG:boto:Path: /vclfcn-60/?prefix=obj4 2024-02-01T10:47:24.093 DEBUG:boto:Data: 2024-02-01T10:47:24.093 DEBUG:boto:Headers: {} 2024-02-01T10:47:24.093 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:24.094 DEBUG:boto:Port: 8001 2024-02-01T10:47:24.094 DEBUG:boto:Params: {} 2024-02-01T10:47:24.094 DEBUG:boto:Token: None 2024-02-01T10:47:24.094 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:24 GMT /vclfcn-60/ 2024-02-01T10:47:24.094 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:JV4WWQv0vNWo81FhwML/smo88OI= 2024-02-01T10:47:24.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:JV4WWQv0vNWo81FhwML/smo88OI=', 'Content-Length': '0'} 2024-02-01T10:47:24.098 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e57977d97d74aee3-0065bb76bc-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:24.098 DEBUG:boto:b'vclfcn-60obj41000falseobj42024-02-01T10:46:46.336Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:24.099 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:47:24.099 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:47:24.099 DEBUG:boto:path=/vclfcn-60/?prefix=obj4 2024-02-01T10:47:24.099 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj4 2024-02-01T10:47:24.099 DEBUG:boto:Method: GET 2024-02-01T10:47:24.099 DEBUG:boto:Path: /vclfcn-60/?prefix=obj4 2024-02-01T10:47:24.099 DEBUG:boto:Data: 2024-02-01T10:47:24.100 DEBUG:boto:Headers: {} 2024-02-01T10:47:24.100 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:24.100 DEBUG:boto:Port: 8000 2024-02-01T10:47:24.100 DEBUG:boto:Params: {} 2024-02-01T10:47:24.100 DEBUG:boto:Token: None 2024-02-01T10:47:24.100 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:24 GMT /vclfcn-60/ 2024-02-01T10:47:24.100 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:JV4WWQv0vNWo81FhwML/smo88OI= 2024-02-01T10:47:24.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:JV4WWQv0vNWo81FhwML/smo88OI=', 'Content-Length': '0'} 2024-02-01T10:47:24.105 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b6696db2adbb4c5-0065bb76bc-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:24.148 DEBUG:boto:b'vclfcn-60obj41000falseobj42024-02-01T10:46:46.336Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:24.148 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-60 zones={test-zone3, test-zone1} 2024-02-01T10:47:24.149 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-60 zones={test-zone3, test-zone2} 2024-02-01T10:47:24.149 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:47:24.149 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:47:24.149 DEBUG:boto:Method: HEAD 2024-02-01T10:47:24.149 DEBUG:boto:Path: /vclfcn-60/ 2024-02-01T10:47:24.149 DEBUG:boto:Data: 2024-02-01T10:47:24.149 DEBUG:boto:Headers: {} 2024-02-01T10:47:24.149 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:24.149 DEBUG:boto:Port: 8001 2024-02-01T10:47:24.149 DEBUG:boto:Params: {} 2024-02-01T10:47:24.150 DEBUG:boto:Token: None 2024-02-01T10:47:24.150 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:24 GMT /vclfcn-60/ 2024-02-01T10:47:24.150 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FCs9vqISJZy5AQ2QeME/9RRGb2s= 2024-02-01T10:47:24.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FCs9vqISJZy5AQ2QeME/9RRGb2s=', 'Content-Length': '0'} 2024-02-01T10:47:24.154 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005f6fdc2b54014f90-0065bb76bc-5762-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:47:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:24.154 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:47:24.154 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:47:24.154 DEBUG:boto:Method: HEAD 2024-02-01T10:47:24.154 DEBUG:boto:Path: /vclfcn-60/ 2024-02-01T10:47:24.155 DEBUG:boto:Data: 2024-02-01T10:47:24.155 DEBUG:boto:Headers: {} 2024-02-01T10:47:24.155 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:24.155 DEBUG:boto:Port: 8000 2024-02-01T10:47:24.155 DEBUG:boto:Params: {} 2024-02-01T10:47:24.155 DEBUG:boto:Token: None 2024-02-01T10:47:24.155 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:24 GMT /vclfcn-60/ 2024-02-01T10:47:24.155 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FCs9vqISJZy5AQ2QeME/9RRGb2s= 2024-02-01T10:47:24.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FCs9vqISJZy5AQ2QeME/9RRGb2s=', 'Content-Length': '0'} 2024-02-01T10:47:24.160 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000007d8719286af158b-0065bb76bc-5350-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:47:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:24.160 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:47:24.161 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:47:24.161 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:47:24.161 DEBUG:boto:path=/vclfcn-60/?versions&max-keys=999 2024-02-01T10:47:24.161 DEBUG:boto:auth_path=/vclfcn-60/?versions&max-keys=999 2024-02-01T10:47:24.161 DEBUG:boto:Method: GET 2024-02-01T10:47:24.161 DEBUG:boto:Path: /vclfcn-60/?versions&max-keys=999 2024-02-01T10:47:24.161 DEBUG:boto:Data: 2024-02-01T10:47:24.161 DEBUG:boto:Headers: {} 2024-02-01T10:47:24.161 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:24.162 DEBUG:boto:Port: 8001 2024-02-01T10:47:24.162 DEBUG:boto:Params: {} 2024-02-01T10:47:24.162 DEBUG:boto:Token: None 2024-02-01T10:47:24.162 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:24 GMT /vclfcn-60/?versions 2024-02-01T10:47:24.162 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:AxKvqqydy+7Dd+wT+0iQs5Ums68= 2024-02-01T10:47:24.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:AxKvqqydy+7Dd+wT+0iQs5Ums68=', 'Content-Length': '0'} 2024-02-01T10:47:24.167 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c188fb380a299d31-0065bb76bc-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:24.167 DEBUG:boto:b'vclfcn-60999falseobj1nulltrue2024-02-01T10:46:46.243Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:46:46.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:46:46.285Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:46:46.336Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:24.168 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-02-01T10:47:24.168 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-02-01T10:47:24.168 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-02-01T10:47:24.168 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-02-01T10:47:24.168 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:47:24.168 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:47:24.168 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:47:24.168 DEBUG:boto:path=/vclfcn-60/?versions&max-keys=999 2024-02-01T10:47:24.168 DEBUG:boto:auth_path=/vclfcn-60/?versions&max-keys=999 2024-02-01T10:47:24.169 DEBUG:boto:Method: GET 2024-02-01T10:47:24.169 DEBUG:boto:Path: /vclfcn-60/?versions&max-keys=999 2024-02-01T10:47:24.169 DEBUG:boto:Data: 2024-02-01T10:47:24.169 DEBUG:boto:Headers: {} 2024-02-01T10:47:24.169 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:24.169 DEBUG:boto:Port: 8000 2024-02-01T10:47:24.169 DEBUG:boto:Params: {} 2024-02-01T10:47:24.169 DEBUG:boto:Token: None 2024-02-01T10:47:24.169 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:24 GMT /vclfcn-60/?versions 2024-02-01T10:47:24.169 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:AxKvqqydy+7Dd+wT+0iQs5Ums68= 2024-02-01T10:47:24.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:AxKvqqydy+7Dd+wT+0iQs5Ums68=', 'Content-Length': '0'} 2024-02-01T10:47:24.174 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000053234d3398b70742-0065bb76bc-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:24.174 DEBUG:boto:b'vclfcn-60999falseobj1nulltrue2024-02-01T10:46:46.243Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:46:46.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:46:46.285Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:46:46.336Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:24.175 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-02-01T10:47:24.175 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-02-01T10:47:24.175 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-02-01T10:47:24.175 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-02-01T10:47:24.175 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:47:24.175 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:47:24.175 DEBUG:boto:path=/vclfcn-60/?versions&max-keys=999 2024-02-01T10:47:24.175 DEBUG:boto:auth_path=/vclfcn-60/?versions&max-keys=999 2024-02-01T10:47:24.175 DEBUG:boto:Method: GET 2024-02-01T10:47:24.176 DEBUG:boto:Path: /vclfcn-60/?versions&max-keys=999 2024-02-01T10:47:24.176 DEBUG:boto:Data: 2024-02-01T10:47:24.176 DEBUG:boto:Headers: {} 2024-02-01T10:47:24.176 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:24.176 DEBUG:boto:Port: 8001 2024-02-01T10:47:24.176 DEBUG:boto:Params: {} 2024-02-01T10:47:24.176 DEBUG:boto:Token: None 2024-02-01T10:47:24.176 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:24 GMT /vclfcn-60/?versions 2024-02-01T10:47:24.176 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:AxKvqqydy+7Dd+wT+0iQs5Ums68= 2024-02-01T10:47:24.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:AxKvqqydy+7Dd+wT+0iQs5Ums68=', 'Content-Length': '0'} 2024-02-01T10:47:24.181 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7340aa1167f453d-0065bb76bc-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:24.224 DEBUG:boto:b'vclfcn-60999falseobj1nulltrue2024-02-01T10:46:46.243Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:46:46.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:46:46.285Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:46:46.336Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:24.225 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:47:24.225 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:47:24.225 DEBUG:boto:path=/vclfcn-60/?versions&max-keys=999 2024-02-01T10:47:24.225 DEBUG:boto:auth_path=/vclfcn-60/?versions&max-keys=999 2024-02-01T10:47:24.225 DEBUG:boto:Method: GET 2024-02-01T10:47:24.225 DEBUG:boto:Path: /vclfcn-60/?versions&max-keys=999 2024-02-01T10:47:24.225 DEBUG:boto:Data: 2024-02-01T10:47:24.225 DEBUG:boto:Headers: {} 2024-02-01T10:47:24.225 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:24.226 DEBUG:boto:Port: 8000 2024-02-01T10:47:24.226 DEBUG:boto:Params: {} 2024-02-01T10:47:24.226 DEBUG:boto:Token: None 2024-02-01T10:47:24.226 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:24 GMT /vclfcn-60/?versions 2024-02-01T10:47:24.226 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:AxKvqqydy+7Dd+wT+0iQs5Ums68= 2024-02-01T10:47:24.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:AxKvqqydy+7Dd+wT+0iQs5Ums68=', 'Content-Length': '0'} 2024-02-01T10:47:24.231 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000891a6844a4ff8979-0065bb76bc-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:24.232 DEBUG:boto:b'vclfcn-60999falseobj1nulltrue2024-02-01T10:46:46.243Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:46:46.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:46:46.285Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:46:46.336Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:24.232 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:47:24.232 DEBUG:boto:path=/vclfcn-60/obj1 2024-02-01T10:47:24.232 DEBUG:boto:auth_path=/vclfcn-60/obj1 2024-02-01T10:47:24.232 DEBUG:boto:path=/vclfcn-60/obj1?versionId=null 2024-02-01T10:47:24.233 DEBUG:boto:auth_path=/vclfcn-60/obj1?versionId=null 2024-02-01T10:47:24.233 DEBUG:boto:Method: GET 2024-02-01T10:47:24.233 DEBUG:boto:Path: /vclfcn-60/obj1?versionId=null 2024-02-01T10:47:24.233 DEBUG:boto:Data: 2024-02-01T10:47:24.233 DEBUG:boto:Headers: {} 2024-02-01T10:47:24.233 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:24.233 DEBUG:boto:Port: 8001 2024-02-01T10:47:24.233 DEBUG:boto:Params: {} 2024-02-01T10:47:24.233 DEBUG:boto:Token: None 2024-02-01T10:47:24.234 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:24 GMT /vclfcn-60/obj1?versionId=null 2024-02-01T10:47:24.234 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:700KGWgFyM5LWM9IcmdhT+RCKPM= 2024-02-01T10:47:24.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:700KGWgFyM5LWM9IcmdhT+RCKPM=', 'Content-Length': '0'} 2024-02-01T10:47:24.237 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007b64b364b2ea5102-0065bb76bc-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:24.237 DEBUG:boto:path=/vclfcn-60/obj1 2024-02-01T10:47:24.237 DEBUG:boto:auth_path=/vclfcn-60/obj1 2024-02-01T10:47:24.237 DEBUG:boto:path=/vclfcn-60/obj1?versionId=null 2024-02-01T10:47:24.238 DEBUG:boto:auth_path=/vclfcn-60/obj1?versionId=null 2024-02-01T10:47:24.238 DEBUG:boto:Method: GET 2024-02-01T10:47:24.238 DEBUG:boto:Path: /vclfcn-60/obj1?versionId=null 2024-02-01T10:47:24.238 DEBUG:boto:Data: 2024-02-01T10:47:24.238 DEBUG:boto:Headers: {} 2024-02-01T10:47:24.238 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:24.238 DEBUG:boto:Port: 8000 2024-02-01T10:47:24.238 DEBUG:boto:Params: {} 2024-02-01T10:47:24.238 DEBUG:boto:Token: None 2024-02-01T10:47:24.238 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:24 GMT /vclfcn-60/obj1?versionId=null 2024-02-01T10:47:24.239 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:700KGWgFyM5LWM9IcmdhT+RCKPM= 2024-02-01T10:47:24.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:700KGWgFyM5LWM9IcmdhT+RCKPM=', 'Content-Length': '0'} 2024-02-01T10:47:24.241 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dc411ba2fbf9d887-0065bb76bc-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:24.288 DEBUG:boto:path=/vclfcn-60/obj1 2024-02-01T10:47:24.289 DEBUG:boto:auth_path=/vclfcn-60/obj1 2024-02-01T10:47:24.289 DEBUG:boto:path=/vclfcn-60/obj1?versionId=null 2024-02-01T10:47:24.289 DEBUG:boto:auth_path=/vclfcn-60/obj1?versionId=null 2024-02-01T10:47:24.289 DEBUG:boto:Method: HEAD 2024-02-01T10:47:24.289 DEBUG:boto:Path: /vclfcn-60/obj1?versionId=null 2024-02-01T10:47:24.289 DEBUG:boto:Data: 2024-02-01T10:47:24.289 DEBUG:boto:Headers: {} 2024-02-01T10:47:24.289 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:24.289 DEBUG:boto:Port: 8001 2024-02-01T10:47:24.290 DEBUG:boto:Params: {} 2024-02-01T10:47:24.290 DEBUG:boto:Token: None 2024-02-01T10:47:24.290 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:24 GMT /vclfcn-60/obj1?versionId=null 2024-02-01T10:47:24.290 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/UIktcOkqPtqJUAzRGxB0KuUEYo= 2024-02-01T10:47:24.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/UIktcOkqPtqJUAzRGxB0KuUEYo=', 'Content-Length': '0'} 2024-02-01T10:47:24.293 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000031d5f525e44c0566-0065bb76bc-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:24.293 DEBUG:boto:path=/vclfcn-60/obj1 2024-02-01T10:47:24.293 DEBUG:boto:auth_path=/vclfcn-60/obj1 2024-02-01T10:47:24.294 DEBUG:boto:path=/vclfcn-60/obj1?versionId=null 2024-02-01T10:47:24.294 DEBUG:boto:auth_path=/vclfcn-60/obj1?versionId=null 2024-02-01T10:47:24.294 DEBUG:boto:Method: HEAD 2024-02-01T10:47:24.294 DEBUG:boto:Path: /vclfcn-60/obj1?versionId=null 2024-02-01T10:47:24.294 DEBUG:boto:Data: 2024-02-01T10:47:24.294 DEBUG:boto:Headers: {} 2024-02-01T10:47:24.294 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:24.294 DEBUG:boto:Port: 8000 2024-02-01T10:47:24.294 DEBUG:boto:Params: {} 2024-02-01T10:47:24.294 DEBUG:boto:Token: None 2024-02-01T10:47:24.295 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:24 GMT /vclfcn-60/obj1?versionId=null 2024-02-01T10:47:24.295 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/UIktcOkqPtqJUAzRGxB0KuUEYo= 2024-02-01T10:47:24.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/UIktcOkqPtqJUAzRGxB0KuUEYo=', 'Content-Length': '0'} 2024-02-01T10:47:24.297 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fa9419b237f99843-0065bb76bc-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:24.298 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:47:24.298 DEBUG:boto:path=/vclfcn-60/obj1 2024-02-01T10:47:24.298 DEBUG:boto:auth_path=/vclfcn-60/obj1 2024-02-01T10:47:24.298 DEBUG:boto:path=/vclfcn-60/obj1?versionId=null 2024-02-01T10:47:24.298 DEBUG:boto:auth_path=/vclfcn-60/obj1?versionId=null 2024-02-01T10:47:24.298 DEBUG:boto:Method: GET 2024-02-01T10:47:24.299 DEBUG:boto:Path: /vclfcn-60/obj1?versionId=null 2024-02-01T10:47:24.299 DEBUG:boto:Data: 2024-02-01T10:47:24.299 DEBUG:boto:Headers: {} 2024-02-01T10:47:24.299 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:24.299 DEBUG:boto:Port: 8001 2024-02-01T10:47:24.299 DEBUG:boto:Params: {} 2024-02-01T10:47:24.299 DEBUG:boto:Token: None 2024-02-01T10:47:24.299 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:24 GMT /vclfcn-60/obj1?versionId=null 2024-02-01T10:47:24.300 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:700KGWgFyM5LWM9IcmdhT+RCKPM= 2024-02-01T10:47:24.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:700KGWgFyM5LWM9IcmdhT+RCKPM=', 'Content-Length': '0'} 2024-02-01T10:47:24.302 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e0ecbe1aa9ba3c02-0065bb76bc-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:24.351 DEBUG:boto:path=/vclfcn-60/obj1 2024-02-01T10:47:24.351 DEBUG:boto:auth_path=/vclfcn-60/obj1 2024-02-01T10:47:24.351 DEBUG:boto:path=/vclfcn-60/obj1?versionId=null 2024-02-01T10:47:24.351 DEBUG:boto:auth_path=/vclfcn-60/obj1?versionId=null 2024-02-01T10:47:24.351 DEBUG:boto:Method: GET 2024-02-01T10:47:24.352 DEBUG:boto:Path: /vclfcn-60/obj1?versionId=null 2024-02-01T10:47:24.352 DEBUG:boto:Data: 2024-02-01T10:47:24.352 DEBUG:boto:Headers: {} 2024-02-01T10:47:24.352 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:24.352 DEBUG:boto:Port: 8000 2024-02-01T10:47:24.352 DEBUG:boto:Params: {} 2024-02-01T10:47:24.352 DEBUG:boto:Token: None 2024-02-01T10:47:24.352 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:24 GMT /vclfcn-60/obj1?versionId=null 2024-02-01T10:47:24.353 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:700KGWgFyM5LWM9IcmdhT+RCKPM= 2024-02-01T10:47:24.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:700KGWgFyM5LWM9IcmdhT+RCKPM=', 'Content-Length': '0'} 2024-02-01T10:47:24.355 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000033cf84c8aaefb3e5-0065bb76bc-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:24.356 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:47:24.356 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:47:24.356 DEBUG:boto:path=/vclfcn-60/?prefix=obj1 2024-02-01T10:47:24.356 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj1 2024-02-01T10:47:24.356 DEBUG:boto:Method: GET 2024-02-01T10:47:24.356 DEBUG:boto:Path: /vclfcn-60/?prefix=obj1 2024-02-01T10:47:24.356 DEBUG:boto:Data: 2024-02-01T10:47:24.356 DEBUG:boto:Headers: {} 2024-02-01T10:47:24.357 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:24.357 DEBUG:boto:Port: 8001 2024-02-01T10:47:24.357 DEBUG:boto:Params: {} 2024-02-01T10:47:24.357 DEBUG:boto:Token: None 2024-02-01T10:47:24.357 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:24 GMT /vclfcn-60/ 2024-02-01T10:47:24.357 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:JV4WWQv0vNWo81FhwML/smo88OI= 2024-02-01T10:47:24.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:JV4WWQv0vNWo81FhwML/smo88OI=', 'Content-Length': '0'} 2024-02-01T10:47:24.361 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051b0e4efa24a161e-0065bb76bc-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:24.404 DEBUG:boto:b'vclfcn-60obj11000falseobj12024-02-01T10:46:46.243Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:24.405 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:47:24.405 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:47:24.405 DEBUG:boto:path=/vclfcn-60/?prefix=obj1 2024-02-01T10:47:24.405 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj1 2024-02-01T10:47:24.405 DEBUG:boto:Method: GET 2024-02-01T10:47:24.405 DEBUG:boto:Path: /vclfcn-60/?prefix=obj1 2024-02-01T10:47:24.405 DEBUG:boto:Data: 2024-02-01T10:47:24.405 DEBUG:boto:Headers: {} 2024-02-01T10:47:24.405 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:24.405 DEBUG:boto:Port: 8000 2024-02-01T10:47:24.406 DEBUG:boto:Params: {} 2024-02-01T10:47:24.406 DEBUG:boto:Token: None 2024-02-01T10:47:24.406 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:24 GMT /vclfcn-60/ 2024-02-01T10:47:24.406 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:JV4WWQv0vNWo81FhwML/smo88OI= 2024-02-01T10:47:24.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:JV4WWQv0vNWo81FhwML/smo88OI=', 'Content-Length': '0'} 2024-02-01T10:47:24.411 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000166339ba3f397806-0065bb76bc-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:24.411 DEBUG:boto:b'vclfcn-60obj11000falseobj12024-02-01T10:46:46.243Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:24.412 DEBUG:boto:path=/vclfcn-60/obj1 2024-02-01T10:47:24.412 DEBUG:boto:auth_path=/vclfcn-60/obj1 2024-02-01T10:47:24.412 DEBUG:boto:Method: HEAD 2024-02-01T10:47:24.412 DEBUG:boto:Path: /vclfcn-60/obj1 2024-02-01T10:47:24.412 DEBUG:boto:Data: 2024-02-01T10:47:24.412 DEBUG:boto:Headers: {} 2024-02-01T10:47:24.412 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:24.413 DEBUG:boto:Port: 8001 2024-02-01T10:47:24.413 DEBUG:boto:Params: {} 2024-02-01T10:47:24.413 DEBUG:boto:Token: None 2024-02-01T10:47:24.413 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:24 GMT /vclfcn-60/obj1 2024-02-01T10:47:24.413 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:JkBbBxXYEMB2tl4sz3QwYphIVsM= 2024-02-01T10:47:24.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:JkBbBxXYEMB2tl4sz3QwYphIVsM=', 'Content-Length': '0'} 2024-02-01T10:47:24.416 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000080c74358b11210b7-0065bb76bc-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:24.416 DEBUG:boto:path=/vclfcn-60/obj1 2024-02-01T10:47:24.416 DEBUG:boto:auth_path=/vclfcn-60/obj1 2024-02-01T10:47:24.416 DEBUG:boto:Method: HEAD 2024-02-01T10:47:24.417 DEBUG:boto:Path: /vclfcn-60/obj1 2024-02-01T10:47:24.417 DEBUG:boto:Data: 2024-02-01T10:47:24.417 DEBUG:boto:Headers: {} 2024-02-01T10:47:24.417 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:24.417 DEBUG:boto:Port: 8000 2024-02-01T10:47:24.417 DEBUG:boto:Params: {} 2024-02-01T10:47:24.417 DEBUG:boto:Token: None 2024-02-01T10:47:24.417 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:24 GMT /vclfcn-60/obj1 2024-02-01T10:47:24.417 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:JkBbBxXYEMB2tl4sz3QwYphIVsM= 2024-02-01T10:47:24.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:JkBbBxXYEMB2tl4sz3QwYphIVsM=', 'Content-Length': '0'} 2024-02-01T10:47:24.420 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000038e0f4f88e62b6f8-0065bb76bc-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:24.420 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:47:24.420 DEBUG:boto:path=/vclfcn-60/obj1 2024-02-01T10:47:24.420 DEBUG:boto:auth_path=/vclfcn-60/obj1 2024-02-01T10:47:24.421 DEBUG:boto:Method: GET 2024-02-01T10:47:24.421 DEBUG:boto:Path: /vclfcn-60/obj1 2024-02-01T10:47:24.421 DEBUG:boto:Data: 2024-02-01T10:47:24.421 DEBUG:boto:Headers: {} 2024-02-01T10:47:24.421 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:24.421 DEBUG:boto:Port: 8001 2024-02-01T10:47:24.421 DEBUG:boto:Params: {} 2024-02-01T10:47:24.421 DEBUG:boto:Token: None 2024-02-01T10:47:24.421 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:24 GMT /vclfcn-60/obj1 2024-02-01T10:47:24.421 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wcNjCT9RULa4LA3g+lgJsCBxtyU= 2024-02-01T10:47:24.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wcNjCT9RULa4LA3g+lgJsCBxtyU=', 'Content-Length': '0'} 2024-02-01T10:47:24.424 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000055ed9a26ec91e302-0065bb76bc-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:24.468 DEBUG:boto:path=/vclfcn-60/obj1 2024-02-01T10:47:24.468 DEBUG:boto:auth_path=/vclfcn-60/obj1 2024-02-01T10:47:24.468 DEBUG:boto:Method: GET 2024-02-01T10:47:24.469 DEBUG:boto:Path: /vclfcn-60/obj1 2024-02-01T10:47:24.469 DEBUG:boto:Data: 2024-02-01T10:47:24.469 DEBUG:boto:Headers: {} 2024-02-01T10:47:24.469 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:24.469 DEBUG:boto:Port: 8000 2024-02-01T10:47:24.469 DEBUG:boto:Params: {} 2024-02-01T10:47:24.469 DEBUG:boto:Token: None 2024-02-01T10:47:24.469 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:24 GMT /vclfcn-60/obj1 2024-02-01T10:47:24.469 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wcNjCT9RULa4LA3g+lgJsCBxtyU= 2024-02-01T10:47:24.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wcNjCT9RULa4LA3g+lgJsCBxtyU=', 'Content-Length': '0'} 2024-02-01T10:47:24.472 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003072780794833da8-0065bb76bc-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:24.473 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:47:24.473 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:47:24.473 DEBUG:boto:path=/vclfcn-60/?prefix=obj1 2024-02-01T10:47:24.473 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj1 2024-02-01T10:47:24.473 DEBUG:boto:Method: GET 2024-02-01T10:47:24.474 DEBUG:boto:Path: /vclfcn-60/?prefix=obj1 2024-02-01T10:47:24.474 DEBUG:boto:Data: 2024-02-01T10:47:24.474 DEBUG:boto:Headers: {} 2024-02-01T10:47:24.474 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:24.474 DEBUG:boto:Port: 8001 2024-02-01T10:47:24.474 DEBUG:boto:Params: {} 2024-02-01T10:47:24.474 DEBUG:boto:Token: None 2024-02-01T10:47:24.474 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:24 GMT /vclfcn-60/ 2024-02-01T10:47:24.475 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:JV4WWQv0vNWo81FhwML/smo88OI= 2024-02-01T10:47:24.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:JV4WWQv0vNWo81FhwML/smo88OI=', 'Content-Length': '0'} 2024-02-01T10:47:24.479 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a173ed86fe1170d2-0065bb76bc-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:24.520 DEBUG:boto:b'vclfcn-60obj11000falseobj12024-02-01T10:46:46.243Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:24.521 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:47:24.521 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:47:24.521 DEBUG:boto:path=/vclfcn-60/?prefix=obj1 2024-02-01T10:47:24.521 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj1 2024-02-01T10:47:24.521 DEBUG:boto:Method: GET 2024-02-01T10:47:24.521 DEBUG:boto:Path: /vclfcn-60/?prefix=obj1 2024-02-01T10:47:24.521 DEBUG:boto:Data: 2024-02-01T10:47:24.521 DEBUG:boto:Headers: {} 2024-02-01T10:47:24.521 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:24.521 DEBUG:boto:Port: 8000 2024-02-01T10:47:24.521 DEBUG:boto:Params: {} 2024-02-01T10:47:24.522 DEBUG:boto:Token: None 2024-02-01T10:47:24.522 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:24 GMT /vclfcn-60/ 2024-02-01T10:47:24.522 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:JV4WWQv0vNWo81FhwML/smo88OI= 2024-02-01T10:47:24.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:JV4WWQv0vNWo81FhwML/smo88OI=', 'Content-Length': '0'} 2024-02-01T10:47:24.527 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7297e87adc1faf9-0065bb76bc-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:24.527 DEBUG:boto:b'vclfcn-60obj11000falseobj12024-02-01T10:46:46.243Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:24.527 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:47:24.528 DEBUG:boto:path=/vclfcn-60/obj2 2024-02-01T10:47:24.528 DEBUG:boto:auth_path=/vclfcn-60/obj2 2024-02-01T10:47:24.528 DEBUG:boto:path=/vclfcn-60/obj2?versionId=null 2024-02-01T10:47:24.528 DEBUG:boto:auth_path=/vclfcn-60/obj2?versionId=null 2024-02-01T10:47:24.528 DEBUG:boto:Method: GET 2024-02-01T10:47:24.528 DEBUG:boto:Path: /vclfcn-60/obj2?versionId=null 2024-02-01T10:47:24.528 DEBUG:boto:Data: 2024-02-01T10:47:24.528 DEBUG:boto:Headers: {} 2024-02-01T10:47:24.528 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:24.528 DEBUG:boto:Port: 8001 2024-02-01T10:47:24.529 DEBUG:boto:Params: {} 2024-02-01T10:47:24.529 DEBUG:boto:Token: None 2024-02-01T10:47:24.529 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:24 GMT /vclfcn-60/obj2?versionId=null 2024-02-01T10:47:24.529 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:3GdACfJ+NUSRZAN9zch+ZpKUxKM= 2024-02-01T10:47:24.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:3GdACfJ+NUSRZAN9zch+ZpKUxKM=', 'Content-Length': '0'} 2024-02-01T10:47:24.532 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000abe1cdb2bb8a8a97-0065bb76bc-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:24.532 DEBUG:boto:path=/vclfcn-60/obj2 2024-02-01T10:47:24.532 DEBUG:boto:auth_path=/vclfcn-60/obj2 2024-02-01T10:47:24.533 DEBUG:boto:path=/vclfcn-60/obj2?versionId=null 2024-02-01T10:47:24.533 DEBUG:boto:auth_path=/vclfcn-60/obj2?versionId=null 2024-02-01T10:47:24.533 DEBUG:boto:Method: GET 2024-02-01T10:47:24.533 DEBUG:boto:Path: /vclfcn-60/obj2?versionId=null 2024-02-01T10:47:24.533 DEBUG:boto:Data: 2024-02-01T10:47:24.533 DEBUG:boto:Headers: {} 2024-02-01T10:47:24.533 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:24.533 DEBUG:boto:Port: 8000 2024-02-01T10:47:24.533 DEBUG:boto:Params: {} 2024-02-01T10:47:24.533 DEBUG:boto:Token: None 2024-02-01T10:47:24.534 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:24 GMT /vclfcn-60/obj2?versionId=null 2024-02-01T10:47:24.534 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:3GdACfJ+NUSRZAN9zch+ZpKUxKM= 2024-02-01T10:47:24.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:3GdACfJ+NUSRZAN9zch+ZpKUxKM=', 'Content-Length': '0'} 2024-02-01T10:47:24.536 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b7a5efb47fa3713d-0065bb76bc-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:24.580 DEBUG:boto:path=/vclfcn-60/obj2 2024-02-01T10:47:24.580 DEBUG:boto:auth_path=/vclfcn-60/obj2 2024-02-01T10:47:24.580 DEBUG:boto:path=/vclfcn-60/obj2?versionId=null 2024-02-01T10:47:24.581 DEBUG:boto:auth_path=/vclfcn-60/obj2?versionId=null 2024-02-01T10:47:24.581 DEBUG:boto:Method: HEAD 2024-02-01T10:47:24.581 DEBUG:boto:Path: /vclfcn-60/obj2?versionId=null 2024-02-01T10:47:24.581 DEBUG:boto:Data: 2024-02-01T10:47:24.581 DEBUG:boto:Headers: {} 2024-02-01T10:47:24.581 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:24.581 DEBUG:boto:Port: 8001 2024-02-01T10:47:24.581 DEBUG:boto:Params: {} 2024-02-01T10:47:24.581 DEBUG:boto:Token: None 2024-02-01T10:47:24.582 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:24 GMT /vclfcn-60/obj2?versionId=null 2024-02-01T10:47:24.582 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:G0vNZNzW3kcbtiF6mN2iiC9Ns9A= 2024-02-01T10:47:24.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:G0vNZNzW3kcbtiF6mN2iiC9Ns9A=', 'Content-Length': '0'} 2024-02-01T10:47:24.585 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d31a7c8d136790c8-0065bb76bc-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:24.585 DEBUG:boto:path=/vclfcn-60/obj2 2024-02-01T10:47:24.585 DEBUG:boto:auth_path=/vclfcn-60/obj2 2024-02-01T10:47:24.585 DEBUG:boto:path=/vclfcn-60/obj2?versionId=null 2024-02-01T10:47:24.585 DEBUG:boto:auth_path=/vclfcn-60/obj2?versionId=null 2024-02-01T10:47:24.585 DEBUG:boto:Method: HEAD 2024-02-01T10:47:24.585 DEBUG:boto:Path: /vclfcn-60/obj2?versionId=null 2024-02-01T10:47:24.586 DEBUG:boto:Data: 2024-02-01T10:47:24.586 DEBUG:boto:Headers: {} 2024-02-01T10:47:24.586 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:24.586 DEBUG:boto:Port: 8000 2024-02-01T10:47:24.586 DEBUG:boto:Params: {} 2024-02-01T10:47:24.586 DEBUG:boto:Token: None 2024-02-01T10:47:24.586 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:24 GMT /vclfcn-60/obj2?versionId=null 2024-02-01T10:47:24.586 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:G0vNZNzW3kcbtiF6mN2iiC9Ns9A= 2024-02-01T10:47:24.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:G0vNZNzW3kcbtiF6mN2iiC9Ns9A=', 'Content-Length': '0'} 2024-02-01T10:47:24.589 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000099fd96a3c765e0e7-0065bb76bc-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:24.589 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:47:24.589 DEBUG:boto:path=/vclfcn-60/obj2 2024-02-01T10:47:24.589 DEBUG:boto:auth_path=/vclfcn-60/obj2 2024-02-01T10:47:24.590 DEBUG:boto:path=/vclfcn-60/obj2?versionId=null 2024-02-01T10:47:24.590 DEBUG:boto:auth_path=/vclfcn-60/obj2?versionId=null 2024-02-01T10:47:24.590 DEBUG:boto:Method: GET 2024-02-01T10:47:24.590 DEBUG:boto:Path: /vclfcn-60/obj2?versionId=null 2024-02-01T10:47:24.590 DEBUG:boto:Data: 2024-02-01T10:47:24.590 DEBUG:boto:Headers: {} 2024-02-01T10:47:24.590 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:24.590 DEBUG:boto:Port: 8001 2024-02-01T10:47:24.590 DEBUG:boto:Params: {} 2024-02-01T10:47:24.590 DEBUG:boto:Token: None 2024-02-01T10:47:24.591 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:24 GMT /vclfcn-60/obj2?versionId=null 2024-02-01T10:47:24.591 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:3GdACfJ+NUSRZAN9zch+ZpKUxKM= 2024-02-01T10:47:24.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:3GdACfJ+NUSRZAN9zch+ZpKUxKM=', 'Content-Length': '0'} 2024-02-01T10:47:24.594 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007e31dfbc7fe9ebac-0065bb76bc-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:24.640 DEBUG:boto:path=/vclfcn-60/obj2 2024-02-01T10:47:24.641 DEBUG:boto:auth_path=/vclfcn-60/obj2 2024-02-01T10:47:24.641 DEBUG:boto:path=/vclfcn-60/obj2?versionId=null 2024-02-01T10:47:24.641 DEBUG:boto:auth_path=/vclfcn-60/obj2?versionId=null 2024-02-01T10:47:24.641 DEBUG:boto:Method: GET 2024-02-01T10:47:24.641 DEBUG:boto:Path: /vclfcn-60/obj2?versionId=null 2024-02-01T10:47:24.641 DEBUG:boto:Data: 2024-02-01T10:47:24.642 DEBUG:boto:Headers: {} 2024-02-01T10:47:24.642 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:24.642 DEBUG:boto:Port: 8000 2024-02-01T10:47:24.642 DEBUG:boto:Params: {} 2024-02-01T10:47:24.642 DEBUG:boto:Token: None 2024-02-01T10:47:24.642 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:24 GMT /vclfcn-60/obj2?versionId=null 2024-02-01T10:47:24.642 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:3GdACfJ+NUSRZAN9zch+ZpKUxKM= 2024-02-01T10:47:24.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:3GdACfJ+NUSRZAN9zch+ZpKUxKM=', 'Content-Length': '0'} 2024-02-01T10:47:24.645 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000013a3c43cfac2466d-0065bb76bc-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:24.645 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:47:24.645 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:47:24.646 DEBUG:boto:path=/vclfcn-60/?prefix=obj2 2024-02-01T10:47:24.646 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj2 2024-02-01T10:47:24.646 DEBUG:boto:Method: GET 2024-02-01T10:47:24.646 DEBUG:boto:Path: /vclfcn-60/?prefix=obj2 2024-02-01T10:47:24.646 DEBUG:boto:Data: 2024-02-01T10:47:24.646 DEBUG:boto:Headers: {} 2024-02-01T10:47:24.646 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:24.646 DEBUG:boto:Port: 8001 2024-02-01T10:47:24.646 DEBUG:boto:Params: {} 2024-02-01T10:47:24.646 DEBUG:boto:Token: None 2024-02-01T10:47:24.647 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:24 GMT /vclfcn-60/ 2024-02-01T10:47:24.647 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:JV4WWQv0vNWo81FhwML/smo88OI= 2024-02-01T10:47:24.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:JV4WWQv0vNWo81FhwML/smo88OI=', 'Content-Length': '0'} 2024-02-01T10:47:24.651 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002023d6fca15ffec0-0065bb76bc-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:24.692 DEBUG:boto:b'vclfcn-60obj21000falseobj22024-02-01T10:46:46.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:24.692 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:47:24.693 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:47:24.693 DEBUG:boto:path=/vclfcn-60/?prefix=obj2 2024-02-01T10:47:24.693 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj2 2024-02-01T10:47:24.693 DEBUG:boto:Method: GET 2024-02-01T10:47:24.693 DEBUG:boto:Path: /vclfcn-60/?prefix=obj2 2024-02-01T10:47:24.693 DEBUG:boto:Data: 2024-02-01T10:47:24.693 DEBUG:boto:Headers: {} 2024-02-01T10:47:24.693 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:24.693 DEBUG:boto:Port: 8000 2024-02-01T10:47:24.694 DEBUG:boto:Params: {} 2024-02-01T10:47:24.694 DEBUG:boto:Token: None 2024-02-01T10:47:24.694 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:24 GMT /vclfcn-60/ 2024-02-01T10:47:24.694 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:JV4WWQv0vNWo81FhwML/smo88OI= 2024-02-01T10:47:24.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:JV4WWQv0vNWo81FhwML/smo88OI=', 'Content-Length': '0'} 2024-02-01T10:47:24.698 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008075b85e5f61b47a-0065bb76bc-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:24.699 DEBUG:boto:b'vclfcn-60obj21000falseobj22024-02-01T10:46:46.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:24.699 DEBUG:boto:path=/vclfcn-60/obj2 2024-02-01T10:47:24.699 DEBUG:boto:auth_path=/vclfcn-60/obj2 2024-02-01T10:47:24.699 DEBUG:boto:Method: HEAD 2024-02-01T10:47:24.699 DEBUG:boto:Path: /vclfcn-60/obj2 2024-02-01T10:47:24.700 DEBUG:boto:Data: 2024-02-01T10:47:24.700 DEBUG:boto:Headers: {} 2024-02-01T10:47:24.700 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:24.700 DEBUG:boto:Port: 8001 2024-02-01T10:47:24.700 DEBUG:boto:Params: {} 2024-02-01T10:47:24.700 DEBUG:boto:Token: None 2024-02-01T10:47:24.700 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:24 GMT /vclfcn-60/obj2 2024-02-01T10:47:24.700 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ji4EkI1JwnDuwY9H/dgSkJzCNiU= 2024-02-01T10:47:24.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ji4EkI1JwnDuwY9H/dgSkJzCNiU=', 'Content-Length': '0'} 2024-02-01T10:47:24.704 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ffb88d54dbaf3202-0065bb76bc-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:24.704 DEBUG:boto:path=/vclfcn-60/obj2 2024-02-01T10:47:24.704 DEBUG:boto:auth_path=/vclfcn-60/obj2 2024-02-01T10:47:24.704 DEBUG:boto:Method: HEAD 2024-02-01T10:47:24.704 DEBUG:boto:Path: /vclfcn-60/obj2 2024-02-01T10:47:24.704 DEBUG:boto:Data: 2024-02-01T10:47:24.705 DEBUG:boto:Headers: {} 2024-02-01T10:47:24.705 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:24.705 DEBUG:boto:Port: 8000 2024-02-01T10:47:24.705 DEBUG:boto:Params: {} 2024-02-01T10:47:24.705 DEBUG:boto:Token: None 2024-02-01T10:47:24.705 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:24 GMT /vclfcn-60/obj2 2024-02-01T10:47:24.705 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ji4EkI1JwnDuwY9H/dgSkJzCNiU= 2024-02-01T10:47:24.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ji4EkI1JwnDuwY9H/dgSkJzCNiU=', 'Content-Length': '0'} 2024-02-01T10:47:24.708 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cb3a3fc38d7ad9f5-0065bb76bc-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:24.708 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:47:24.709 DEBUG:boto:path=/vclfcn-60/obj2 2024-02-01T10:47:24.709 DEBUG:boto:auth_path=/vclfcn-60/obj2 2024-02-01T10:47:24.709 DEBUG:boto:Method: GET 2024-02-01T10:47:24.709 DEBUG:boto:Path: /vclfcn-60/obj2 2024-02-01T10:47:24.709 DEBUG:boto:Data: 2024-02-01T10:47:24.710 DEBUG:boto:Headers: {} 2024-02-01T10:47:24.710 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:24.710 DEBUG:boto:Port: 8001 2024-02-01T10:47:24.710 DEBUG:boto:Params: {} 2024-02-01T10:47:24.710 DEBUG:boto:Token: None 2024-02-01T10:47:24.710 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:24 GMT /vclfcn-60/obj2 2024-02-01T10:47:24.710 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:k9m0l9oAhJ96GQk8Ahya9RXa6AQ= 2024-02-01T10:47:24.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:k9m0l9oAhJ96GQk8Ahya9RXa6AQ=', 'Content-Length': '0'} 2024-02-01T10:47:24.713 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006c303a1cf3af5a99-0065bb76bc-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:24.756 DEBUG:boto:path=/vclfcn-60/obj2 2024-02-01T10:47:24.756 DEBUG:boto:auth_path=/vclfcn-60/obj2 2024-02-01T10:47:24.756 DEBUG:boto:Method: GET 2024-02-01T10:47:24.756 DEBUG:boto:Path: /vclfcn-60/obj2 2024-02-01T10:47:24.757 DEBUG:boto:Data: 2024-02-01T10:47:24.757 DEBUG:boto:Headers: {} 2024-02-01T10:47:24.757 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:24.757 DEBUG:boto:Port: 8000 2024-02-01T10:47:24.757 DEBUG:boto:Params: {} 2024-02-01T10:47:24.757 DEBUG:boto:Token: None 2024-02-01T10:47:24.757 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:24 GMT /vclfcn-60/obj2 2024-02-01T10:47:24.757 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:k9m0l9oAhJ96GQk8Ahya9RXa6AQ= 2024-02-01T10:47:24.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:k9m0l9oAhJ96GQk8Ahya9RXa6AQ=', 'Content-Length': '0'} 2024-02-01T10:47:24.760 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000014b9948c74a92d05-0065bb76bc-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:24.760 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:47:24.760 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:47:24.760 DEBUG:boto:path=/vclfcn-60/?prefix=obj2 2024-02-01T10:47:24.761 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj2 2024-02-01T10:47:24.761 DEBUG:boto:Method: GET 2024-02-01T10:47:24.761 DEBUG:boto:Path: /vclfcn-60/?prefix=obj2 2024-02-01T10:47:24.761 DEBUG:boto:Data: 2024-02-01T10:47:24.761 DEBUG:boto:Headers: {} 2024-02-01T10:47:24.761 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:24.761 DEBUG:boto:Port: 8001 2024-02-01T10:47:24.761 DEBUG:boto:Params: {} 2024-02-01T10:47:24.761 DEBUG:boto:Token: None 2024-02-01T10:47:24.761 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:24 GMT /vclfcn-60/ 2024-02-01T10:47:24.762 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:JV4WWQv0vNWo81FhwML/smo88OI= 2024-02-01T10:47:24.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:JV4WWQv0vNWo81FhwML/smo88OI=', 'Content-Length': '0'} 2024-02-01T10:47:24.766 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060dbe3e1052dada2-0065bb76bc-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:24.766 DEBUG:boto:b'vclfcn-60obj21000falseobj22024-02-01T10:46:46.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:24.766 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:47:24.767 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:47:24.767 DEBUG:boto:path=/vclfcn-60/?prefix=obj2 2024-02-01T10:47:24.767 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj2 2024-02-01T10:47:24.767 DEBUG:boto:Method: GET 2024-02-01T10:47:24.767 DEBUG:boto:Path: /vclfcn-60/?prefix=obj2 2024-02-01T10:47:24.767 DEBUG:boto:Data: 2024-02-01T10:47:24.767 DEBUG:boto:Headers: {} 2024-02-01T10:47:24.767 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:24.767 DEBUG:boto:Port: 8000 2024-02-01T10:47:24.767 DEBUG:boto:Params: {} 2024-02-01T10:47:24.768 DEBUG:boto:Token: None 2024-02-01T10:47:24.768 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:24 GMT /vclfcn-60/ 2024-02-01T10:47:24.768 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:JV4WWQv0vNWo81FhwML/smo88OI= 2024-02-01T10:47:24.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:JV4WWQv0vNWo81FhwML/smo88OI=', 'Content-Length': '0'} 2024-02-01T10:47:24.772 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cde130843eeef5d2-0065bb76bc-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:24.816 DEBUG:boto:b'vclfcn-60obj21000falseobj22024-02-01T10:46:46.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:24.820 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:47:24.820 DEBUG:boto:path=/vclfcn-60/obj3 2024-02-01T10:47:24.820 DEBUG:boto:auth_path=/vclfcn-60/obj3 2024-02-01T10:47:24.820 DEBUG:boto:path=/vclfcn-60/obj3?versionId=null 2024-02-01T10:47:24.820 DEBUG:boto:auth_path=/vclfcn-60/obj3?versionId=null 2024-02-01T10:47:24.820 DEBUG:boto:Method: GET 2024-02-01T10:47:24.821 DEBUG:boto:Path: /vclfcn-60/obj3?versionId=null 2024-02-01T10:47:24.821 DEBUG:boto:Data: 2024-02-01T10:47:24.821 DEBUG:boto:Headers: {} 2024-02-01T10:47:24.821 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:24.821 DEBUG:boto:Port: 8001 2024-02-01T10:47:24.821 DEBUG:boto:Params: {} 2024-02-01T10:47:24.821 DEBUG:boto:Token: None 2024-02-01T10:47:24.821 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:24 GMT /vclfcn-60/obj3?versionId=null 2024-02-01T10:47:24.821 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:h/RyZT9qEr6m9gFlCoEA8emr+1c= 2024-02-01T10:47:24.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:h/RyZT9qEr6m9gFlCoEA8emr+1c=', 'Content-Length': '0'} 2024-02-01T10:47:24.824 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000969978ac111f8d6-0065bb76bc-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:24.868 DEBUG:boto:path=/vclfcn-60/obj3 2024-02-01T10:47:24.868 DEBUG:boto:auth_path=/vclfcn-60/obj3 2024-02-01T10:47:24.868 DEBUG:boto:path=/vclfcn-60/obj3?versionId=null 2024-02-01T10:47:24.868 DEBUG:boto:auth_path=/vclfcn-60/obj3?versionId=null 2024-02-01T10:47:24.869 DEBUG:boto:Method: GET 2024-02-01T10:47:24.869 DEBUG:boto:Path: /vclfcn-60/obj3?versionId=null 2024-02-01T10:47:24.869 DEBUG:boto:Data: 2024-02-01T10:47:24.869 DEBUG:boto:Headers: {} 2024-02-01T10:47:24.869 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:24.869 DEBUG:boto:Port: 8000 2024-02-01T10:47:24.869 DEBUG:boto:Params: {} 2024-02-01T10:47:24.869 DEBUG:boto:Token: None 2024-02-01T10:47:24.869 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:24 GMT /vclfcn-60/obj3?versionId=null 2024-02-01T10:47:24.870 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:h/RyZT9qEr6m9gFlCoEA8emr+1c= 2024-02-01T10:47:24.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:h/RyZT9qEr6m9gFlCoEA8emr+1c=', 'Content-Length': '0'} 2024-02-01T10:47:24.872 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009def1b25820476e1-0065bb76bc-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:24.873 DEBUG:boto:path=/vclfcn-60/obj3 2024-02-01T10:47:24.873 DEBUG:boto:auth_path=/vclfcn-60/obj3 2024-02-01T10:47:24.873 DEBUG:boto:path=/vclfcn-60/obj3?versionId=null 2024-02-01T10:47:24.873 DEBUG:boto:auth_path=/vclfcn-60/obj3?versionId=null 2024-02-01T10:47:24.873 DEBUG:boto:Method: HEAD 2024-02-01T10:47:24.873 DEBUG:boto:Path: /vclfcn-60/obj3?versionId=null 2024-02-01T10:47:24.873 DEBUG:boto:Data: 2024-02-01T10:47:24.873 DEBUG:boto:Headers: {} 2024-02-01T10:47:24.874 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:24.874 DEBUG:boto:Port: 8001 2024-02-01T10:47:24.874 DEBUG:boto:Params: {} 2024-02-01T10:47:24.874 DEBUG:boto:Token: None 2024-02-01T10:47:24.874 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:24 GMT /vclfcn-60/obj3?versionId=null 2024-02-01T10:47:24.874 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:DwRtq8Xqj6ot32TgWJc+zlrRFtI= 2024-02-01T10:47:24.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:DwRtq8Xqj6ot32TgWJc+zlrRFtI=', 'Content-Length': '0'} 2024-02-01T10:47:24.877 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cd99ffecbc7d633e-0065bb76bc-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:24.877 DEBUG:boto:path=/vclfcn-60/obj3 2024-02-01T10:47:24.878 DEBUG:boto:auth_path=/vclfcn-60/obj3 2024-02-01T10:47:24.878 DEBUG:boto:path=/vclfcn-60/obj3?versionId=null 2024-02-01T10:47:24.878 DEBUG:boto:auth_path=/vclfcn-60/obj3?versionId=null 2024-02-01T10:47:24.878 DEBUG:boto:Method: HEAD 2024-02-01T10:47:24.878 DEBUG:boto:Path: /vclfcn-60/obj3?versionId=null 2024-02-01T10:47:24.878 DEBUG:boto:Data: 2024-02-01T10:47:24.878 DEBUG:boto:Headers: {} 2024-02-01T10:47:24.878 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:24.878 DEBUG:boto:Port: 8000 2024-02-01T10:47:24.878 DEBUG:boto:Params: {} 2024-02-01T10:47:24.879 DEBUG:boto:Token: None 2024-02-01T10:47:24.879 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:24 GMT /vclfcn-60/obj3?versionId=null 2024-02-01T10:47:24.879 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:DwRtq8Xqj6ot32TgWJc+zlrRFtI= 2024-02-01T10:47:24.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:DwRtq8Xqj6ot32TgWJc+zlrRFtI=', 'Content-Length': '0'} 2024-02-01T10:47:24.882 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000075c670fd172fe12e-0065bb76bc-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:24.882 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:47:24.882 DEBUG:boto:path=/vclfcn-60/obj3 2024-02-01T10:47:24.882 DEBUG:boto:auth_path=/vclfcn-60/obj3 2024-02-01T10:47:24.882 DEBUG:boto:path=/vclfcn-60/obj3?versionId=null 2024-02-01T10:47:24.882 DEBUG:boto:auth_path=/vclfcn-60/obj3?versionId=null 2024-02-01T10:47:24.883 DEBUG:boto:Method: GET 2024-02-01T10:47:24.883 DEBUG:boto:Path: /vclfcn-60/obj3?versionId=null 2024-02-01T10:47:24.883 DEBUG:boto:Data: 2024-02-01T10:47:24.883 DEBUG:boto:Headers: {} 2024-02-01T10:47:24.883 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:24.883 DEBUG:boto:Port: 8001 2024-02-01T10:47:24.883 DEBUG:boto:Params: {} 2024-02-01T10:47:24.883 DEBUG:boto:Token: None 2024-02-01T10:47:24.883 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:24 GMT /vclfcn-60/obj3?versionId=null 2024-02-01T10:47:24.883 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:h/RyZT9qEr6m9gFlCoEA8emr+1c= 2024-02-01T10:47:24.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:h/RyZT9qEr6m9gFlCoEA8emr+1c=', 'Content-Length': '0'} 2024-02-01T10:47:24.886 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c042fd59db055e11-0065bb76bc-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:24.928 DEBUG:boto:path=/vclfcn-60/obj3 2024-02-01T10:47:24.928 DEBUG:boto:auth_path=/vclfcn-60/obj3 2024-02-01T10:47:24.928 DEBUG:boto:path=/vclfcn-60/obj3?versionId=null 2024-02-01T10:47:24.928 DEBUG:boto:auth_path=/vclfcn-60/obj3?versionId=null 2024-02-01T10:47:24.928 DEBUG:boto:Method: GET 2024-02-01T10:47:24.929 DEBUG:boto:Path: /vclfcn-60/obj3?versionId=null 2024-02-01T10:47:24.929 DEBUG:boto:Data: 2024-02-01T10:47:24.929 DEBUG:boto:Headers: {} 2024-02-01T10:47:24.929 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:24.929 DEBUG:boto:Port: 8000 2024-02-01T10:47:24.929 DEBUG:boto:Params: {} 2024-02-01T10:47:24.929 DEBUG:boto:Token: None 2024-02-01T10:47:24.929 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:24 GMT /vclfcn-60/obj3?versionId=null 2024-02-01T10:47:24.929 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:h/RyZT9qEr6m9gFlCoEA8emr+1c= 2024-02-01T10:47:24.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:h/RyZT9qEr6m9gFlCoEA8emr+1c=', 'Content-Length': '0'} 2024-02-01T10:47:24.933 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004856812ead1d73bc-0065bb76bc-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:24.933 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:47:24.933 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:47:24.933 DEBUG:boto:path=/vclfcn-60/?prefix=obj3 2024-02-01T10:47:24.933 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj3 2024-02-01T10:47:24.933 DEBUG:boto:Method: GET 2024-02-01T10:47:24.933 DEBUG:boto:Path: /vclfcn-60/?prefix=obj3 2024-02-01T10:47:24.934 DEBUG:boto:Data: 2024-02-01T10:47:24.934 DEBUG:boto:Headers: {} 2024-02-01T10:47:24.934 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:24.934 DEBUG:boto:Port: 8001 2024-02-01T10:47:24.934 DEBUG:boto:Params: {} 2024-02-01T10:47:24.934 DEBUG:boto:Token: None 2024-02-01T10:47:24.934 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:24 GMT /vclfcn-60/ 2024-02-01T10:47:24.934 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:JV4WWQv0vNWo81FhwML/smo88OI= 2024-02-01T10:47:24.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:JV4WWQv0vNWo81FhwML/smo88OI=', 'Content-Length': '0'} 2024-02-01T10:47:24.939 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000111a1e4dcf0effbd-0065bb76bc-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:24.980 DEBUG:boto:b'vclfcn-60obj31000falseobj32024-02-01T10:46:46.285Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:24.980 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:47:24.981 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:47:24.981 DEBUG:boto:path=/vclfcn-60/?prefix=obj3 2024-02-01T10:47:24.981 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj3 2024-02-01T10:47:24.981 DEBUG:boto:Method: GET 2024-02-01T10:47:24.981 DEBUG:boto:Path: /vclfcn-60/?prefix=obj3 2024-02-01T10:47:24.981 DEBUG:boto:Data: 2024-02-01T10:47:24.981 DEBUG:boto:Headers: {} 2024-02-01T10:47:24.981 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:24.981 DEBUG:boto:Port: 8000 2024-02-01T10:47:24.981 DEBUG:boto:Params: {} 2024-02-01T10:47:24.981 DEBUG:boto:Token: None 2024-02-01T10:47:24.982 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:24 GMT /vclfcn-60/ 2024-02-01T10:47:24.982 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:JV4WWQv0vNWo81FhwML/smo88OI= 2024-02-01T10:47:24.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:JV4WWQv0vNWo81FhwML/smo88OI=', 'Content-Length': '0'} 2024-02-01T10:47:24.986 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b186394100244b1-0065bb76bc-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:24.987 DEBUG:boto:b'vclfcn-60obj31000falseobj32024-02-01T10:46:46.285Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:24.987 DEBUG:boto:path=/vclfcn-60/obj3 2024-02-01T10:47:24.987 DEBUG:boto:auth_path=/vclfcn-60/obj3 2024-02-01T10:47:24.987 DEBUG:boto:Method: HEAD 2024-02-01T10:47:24.987 DEBUG:boto:Path: /vclfcn-60/obj3 2024-02-01T10:47:24.988 DEBUG:boto:Data: 2024-02-01T10:47:24.988 DEBUG:boto:Headers: {} 2024-02-01T10:47:24.988 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:24.988 DEBUG:boto:Port: 8001 2024-02-01T10:47:24.988 DEBUG:boto:Params: {} 2024-02-01T10:47:24.988 DEBUG:boto:Token: None 2024-02-01T10:47:24.988 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:24 GMT /vclfcn-60/obj3 2024-02-01T10:47:24.988 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:hYAKWvKQB5+ly9sK9IxaMIrUrsA= 2024-02-01T10:47:24.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:hYAKWvKQB5+ly9sK9IxaMIrUrsA=', 'Content-Length': '0'} 2024-02-01T10:47:24.991 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000158dc3ddc63654e-0065bb76bc-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:24.992 DEBUG:boto:path=/vclfcn-60/obj3 2024-02-01T10:47:24.992 DEBUG:boto:auth_path=/vclfcn-60/obj3 2024-02-01T10:47:24.992 DEBUG:boto:Method: HEAD 2024-02-01T10:47:24.992 DEBUG:boto:Path: /vclfcn-60/obj3 2024-02-01T10:47:24.992 DEBUG:boto:Data: 2024-02-01T10:47:24.992 DEBUG:boto:Headers: {} 2024-02-01T10:47:24.992 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:24.992 DEBUG:boto:Port: 8000 2024-02-01T10:47:24.992 DEBUG:boto:Params: {} 2024-02-01T10:47:24.992 DEBUG:boto:Token: None 2024-02-01T10:47:24.993 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:24 GMT /vclfcn-60/obj3 2024-02-01T10:47:24.993 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:hYAKWvKQB5+ly9sK9IxaMIrUrsA= 2024-02-01T10:47:24.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:hYAKWvKQB5+ly9sK9IxaMIrUrsA=', 'Content-Length': '0'} 2024-02-01T10:47:24.995 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b8854bd00c562495-0065bb76bc-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:24.996 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:47:24.996 DEBUG:boto:path=/vclfcn-60/obj3 2024-02-01T10:47:24.996 DEBUG:boto:auth_path=/vclfcn-60/obj3 2024-02-01T10:47:24.996 DEBUG:boto:Method: GET 2024-02-01T10:47:24.996 DEBUG:boto:Path: /vclfcn-60/obj3 2024-02-01T10:47:24.996 DEBUG:boto:Data: 2024-02-01T10:47:24.996 DEBUG:boto:Headers: {} 2024-02-01T10:47:24.996 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:24.996 DEBUG:boto:Port: 8001 2024-02-01T10:47:24.996 DEBUG:boto:Params: {} 2024-02-01T10:47:24.997 DEBUG:boto:Token: None 2024-02-01T10:47:24.997 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:24 GMT /vclfcn-60/obj3 2024-02-01T10:47:24.997 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ODhDChXbA9W6avroA/VbrDAsx5o= 2024-02-01T10:47:24.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ODhDChXbA9W6avroA/VbrDAsx5o=', 'Content-Length': '0'} 2024-02-01T10:47:25.000 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003f15d77ae994fe8f-0065bb76bc-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:25.040 DEBUG:boto:path=/vclfcn-60/obj3 2024-02-01T10:47:25.040 DEBUG:boto:auth_path=/vclfcn-60/obj3 2024-02-01T10:47:25.040 DEBUG:boto:Method: GET 2024-02-01T10:47:25.040 DEBUG:boto:Path: /vclfcn-60/obj3 2024-02-01T10:47:25.041 DEBUG:boto:Data: 2024-02-01T10:47:25.041 DEBUG:boto:Headers: {} 2024-02-01T10:47:25.041 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:25.041 DEBUG:boto:Port: 8000 2024-02-01T10:47:25.041 DEBUG:boto:Params: {} 2024-02-01T10:47:25.041 DEBUG:boto:Token: None 2024-02-01T10:47:25.041 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:25 GMT /vclfcn-60/obj3 2024-02-01T10:47:25.041 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/iowJwM6hrgKibOGzcAcBNQ+KZ0= 2024-02-01T10:47:25.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:25 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/iowJwM6hrgKibOGzcAcBNQ+KZ0=', 'Content-Length': '0'} 2024-02-01T10:47:25.044 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ca6b35addf78ee6a-0065bb76bd-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:25.044 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:47:25.045 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:47:25.045 DEBUG:boto:path=/vclfcn-60/?prefix=obj3 2024-02-01T10:47:25.045 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj3 2024-02-01T10:47:25.045 DEBUG:boto:Method: GET 2024-02-01T10:47:25.045 DEBUG:boto:Path: /vclfcn-60/?prefix=obj3 2024-02-01T10:47:25.045 DEBUG:boto:Data: 2024-02-01T10:47:25.045 DEBUG:boto:Headers: {} 2024-02-01T10:47:25.045 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:25.045 DEBUG:boto:Port: 8001 2024-02-01T10:47:25.045 DEBUG:boto:Params: {} 2024-02-01T10:47:25.045 DEBUG:boto:Token: None 2024-02-01T10:47:25.046 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:25 GMT /vclfcn-60/ 2024-02-01T10:47:25.046 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:dJTC03LkZB4q9jmx0PdaXDdoC1k= 2024-02-01T10:47:25.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:25 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dJTC03LkZB4q9jmx0PdaXDdoC1k=', 'Content-Length': '0'} 2024-02-01T10:47:25.050 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008710af7994a03bb4-0065bb76bd-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:25.092 DEBUG:boto:b'vclfcn-60obj31000falseobj32024-02-01T10:46:46.285Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:25.092 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:47:25.093 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:47:25.093 DEBUG:boto:path=/vclfcn-60/?prefix=obj3 2024-02-01T10:47:25.093 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj3 2024-02-01T10:47:25.093 DEBUG:boto:Method: GET 2024-02-01T10:47:25.093 DEBUG:boto:Path: /vclfcn-60/?prefix=obj3 2024-02-01T10:47:25.093 DEBUG:boto:Data: 2024-02-01T10:47:25.093 DEBUG:boto:Headers: {} 2024-02-01T10:47:25.093 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:25.093 DEBUG:boto:Port: 8000 2024-02-01T10:47:25.094 DEBUG:boto:Params: {} 2024-02-01T10:47:25.094 DEBUG:boto:Token: None 2024-02-01T10:47:25.094 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:25 GMT /vclfcn-60/ 2024-02-01T10:47:25.094 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:dJTC03LkZB4q9jmx0PdaXDdoC1k= 2024-02-01T10:47:25.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:25 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dJTC03LkZB4q9jmx0PdaXDdoC1k=', 'Content-Length': '0'} 2024-02-01T10:47:25.099 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033b4342528c6ba06-0065bb76bd-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:25.099 DEBUG:boto:b'vclfcn-60obj31000falseobj32024-02-01T10:46:46.285Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:25.099 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:47:25.099 DEBUG:boto:path=/vclfcn-60/obj4 2024-02-01T10:47:25.099 DEBUG:boto:auth_path=/vclfcn-60/obj4 2024-02-01T10:47:25.100 DEBUG:boto:path=/vclfcn-60/obj4?versionId=null 2024-02-01T10:47:25.100 DEBUG:boto:auth_path=/vclfcn-60/obj4?versionId=null 2024-02-01T10:47:25.100 DEBUG:boto:Method: GET 2024-02-01T10:47:25.100 DEBUG:boto:Path: /vclfcn-60/obj4?versionId=null 2024-02-01T10:47:25.100 DEBUG:boto:Data: 2024-02-01T10:47:25.100 DEBUG:boto:Headers: {} 2024-02-01T10:47:25.100 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:25.100 DEBUG:boto:Port: 8001 2024-02-01T10:47:25.100 DEBUG:boto:Params: {} 2024-02-01T10:47:25.101 DEBUG:boto:Token: None 2024-02-01T10:47:25.101 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:25 GMT /vclfcn-60/obj4?versionId=null 2024-02-01T10:47:25.101 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:SVbctDpk3qs99Xgu/yE6sGFBzZg= 2024-02-01T10:47:25.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:25 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:SVbctDpk3qs99Xgu/yE6sGFBzZg=', 'Content-Length': '0'} 2024-02-01T10:47:25.104 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009bdf18c8382e2ee2-0065bb76bd-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:25.144 DEBUG:boto:path=/vclfcn-60/obj4 2024-02-01T10:47:25.144 DEBUG:boto:auth_path=/vclfcn-60/obj4 2024-02-01T10:47:25.144 DEBUG:boto:path=/vclfcn-60/obj4?versionId=null 2024-02-01T10:47:25.144 DEBUG:boto:auth_path=/vclfcn-60/obj4?versionId=null 2024-02-01T10:47:25.145 DEBUG:boto:Method: GET 2024-02-01T10:47:25.145 DEBUG:boto:Path: /vclfcn-60/obj4?versionId=null 2024-02-01T10:47:25.145 DEBUG:boto:Data: 2024-02-01T10:47:25.145 DEBUG:boto:Headers: {} 2024-02-01T10:47:25.145 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:25.145 DEBUG:boto:Port: 8000 2024-02-01T10:47:25.145 DEBUG:boto:Params: {} 2024-02-01T10:47:25.145 DEBUG:boto:Token: None 2024-02-01T10:47:25.145 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:25 GMT /vclfcn-60/obj4?versionId=null 2024-02-01T10:47:25.146 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:SVbctDpk3qs99Xgu/yE6sGFBzZg= 2024-02-01T10:47:25.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:25 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:SVbctDpk3qs99Xgu/yE6sGFBzZg=', 'Content-Length': '0'} 2024-02-01T10:47:25.148 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006f54c974119a51f8-0065bb76bd-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:25.149 DEBUG:boto:path=/vclfcn-60/obj4 2024-02-01T10:47:25.149 DEBUG:boto:auth_path=/vclfcn-60/obj4 2024-02-01T10:47:25.149 DEBUG:boto:path=/vclfcn-60/obj4?versionId=null 2024-02-01T10:47:25.149 DEBUG:boto:auth_path=/vclfcn-60/obj4?versionId=null 2024-02-01T10:47:25.149 DEBUG:boto:Method: HEAD 2024-02-01T10:47:25.149 DEBUG:boto:Path: /vclfcn-60/obj4?versionId=null 2024-02-01T10:47:25.149 DEBUG:boto:Data: 2024-02-01T10:47:25.149 DEBUG:boto:Headers: {} 2024-02-01T10:47:25.149 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:25.150 DEBUG:boto:Port: 8001 2024-02-01T10:47:25.150 DEBUG:boto:Params: {} 2024-02-01T10:47:25.150 DEBUG:boto:Token: None 2024-02-01T10:47:25.150 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:25 GMT /vclfcn-60/obj4?versionId=null 2024-02-01T10:47:25.150 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5UxMz/HwfzooBDGRdsE6KbOIUE8= 2024-02-01T10:47:25.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:25 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5UxMz/HwfzooBDGRdsE6KbOIUE8=', 'Content-Length': '0'} 2024-02-01T10:47:25.153 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005050b40a037681de-0065bb76bd-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:25.153 DEBUG:boto:path=/vclfcn-60/obj4 2024-02-01T10:47:25.153 DEBUG:boto:auth_path=/vclfcn-60/obj4 2024-02-01T10:47:25.153 DEBUG:boto:path=/vclfcn-60/obj4?versionId=null 2024-02-01T10:47:25.153 DEBUG:boto:auth_path=/vclfcn-60/obj4?versionId=null 2024-02-01T10:47:25.154 DEBUG:boto:Method: HEAD 2024-02-01T10:47:25.154 DEBUG:boto:Path: /vclfcn-60/obj4?versionId=null 2024-02-01T10:47:25.154 DEBUG:boto:Data: 2024-02-01T10:47:25.154 DEBUG:boto:Headers: {} 2024-02-01T10:47:25.154 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:25.154 DEBUG:boto:Port: 8000 2024-02-01T10:47:25.154 DEBUG:boto:Params: {} 2024-02-01T10:47:25.154 DEBUG:boto:Token: None 2024-02-01T10:47:25.154 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:25 GMT /vclfcn-60/obj4?versionId=null 2024-02-01T10:47:25.155 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5UxMz/HwfzooBDGRdsE6KbOIUE8= 2024-02-01T10:47:25.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:25 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5UxMz/HwfzooBDGRdsE6KbOIUE8=', 'Content-Length': '0'} 2024-02-01T10:47:25.157 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000059efc62431b556e6-0065bb76bd-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:25.158 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:47:25.158 DEBUG:boto:path=/vclfcn-60/obj4 2024-02-01T10:47:25.158 DEBUG:boto:auth_path=/vclfcn-60/obj4 2024-02-01T10:47:25.158 DEBUG:boto:path=/vclfcn-60/obj4?versionId=null 2024-02-01T10:47:25.158 DEBUG:boto:auth_path=/vclfcn-60/obj4?versionId=null 2024-02-01T10:47:25.158 DEBUG:boto:Method: GET 2024-02-01T10:47:25.158 DEBUG:boto:Path: /vclfcn-60/obj4?versionId=null 2024-02-01T10:47:25.158 DEBUG:boto:Data: 2024-02-01T10:47:25.158 DEBUG:boto:Headers: {} 2024-02-01T10:47:25.158 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:25.158 DEBUG:boto:Port: 8001 2024-02-01T10:47:25.159 DEBUG:boto:Params: {} 2024-02-01T10:47:25.159 DEBUG:boto:Token: None 2024-02-01T10:47:25.159 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:25 GMT /vclfcn-60/obj4?versionId=null 2024-02-01T10:47:25.159 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:SVbctDpk3qs99Xgu/yE6sGFBzZg= 2024-02-01T10:47:25.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:25 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:SVbctDpk3qs99Xgu/yE6sGFBzZg=', 'Content-Length': '0'} 2024-02-01T10:47:25.163 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000966a393dd601abc0-0065bb76bd-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:25.204 DEBUG:boto:path=/vclfcn-60/obj4 2024-02-01T10:47:25.204 DEBUG:boto:auth_path=/vclfcn-60/obj4 2024-02-01T10:47:25.204 DEBUG:boto:path=/vclfcn-60/obj4?versionId=null 2024-02-01T10:47:25.204 DEBUG:boto:auth_path=/vclfcn-60/obj4?versionId=null 2024-02-01T10:47:25.205 DEBUG:boto:Method: GET 2024-02-01T10:47:25.205 DEBUG:boto:Path: /vclfcn-60/obj4?versionId=null 2024-02-01T10:47:25.205 DEBUG:boto:Data: 2024-02-01T10:47:25.205 DEBUG:boto:Headers: {} 2024-02-01T10:47:25.205 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:25.205 DEBUG:boto:Port: 8000 2024-02-01T10:47:25.205 DEBUG:boto:Params: {} 2024-02-01T10:47:25.205 DEBUG:boto:Token: None 2024-02-01T10:47:25.205 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:25 GMT /vclfcn-60/obj4?versionId=null 2024-02-01T10:47:25.206 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:SVbctDpk3qs99Xgu/yE6sGFBzZg= 2024-02-01T10:47:25.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:25 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:SVbctDpk3qs99Xgu/yE6sGFBzZg=', 'Content-Length': '0'} 2024-02-01T10:47:25.208 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000100905324044aa32-0065bb76bd-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:25.209 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:47:25.209 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:47:25.209 DEBUG:boto:path=/vclfcn-60/?prefix=obj4 2024-02-01T10:47:25.209 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj4 2024-02-01T10:47:25.209 DEBUG:boto:Method: GET 2024-02-01T10:47:25.209 DEBUG:boto:Path: /vclfcn-60/?prefix=obj4 2024-02-01T10:47:25.209 DEBUG:boto:Data: 2024-02-01T10:47:25.209 DEBUG:boto:Headers: {} 2024-02-01T10:47:25.209 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:25.209 DEBUG:boto:Port: 8001 2024-02-01T10:47:25.210 DEBUG:boto:Params: {} 2024-02-01T10:47:25.210 DEBUG:boto:Token: None 2024-02-01T10:47:25.210 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:25 GMT /vclfcn-60/ 2024-02-01T10:47:25.210 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:dJTC03LkZB4q9jmx0PdaXDdoC1k= 2024-02-01T10:47:25.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:25 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dJTC03LkZB4q9jmx0PdaXDdoC1k=', 'Content-Length': '0'} 2024-02-01T10:47:25.214 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092b3e02673b4e0a6-0065bb76bd-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:25.260 DEBUG:boto:b'vclfcn-60obj41000falseobj42024-02-01T10:46:46.336Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:25.260 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:47:25.261 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:47:25.261 DEBUG:boto:path=/vclfcn-60/?prefix=obj4 2024-02-01T10:47:25.261 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj4 2024-02-01T10:47:25.261 DEBUG:boto:Method: GET 2024-02-01T10:47:25.261 DEBUG:boto:Path: /vclfcn-60/?prefix=obj4 2024-02-01T10:47:25.261 DEBUG:boto:Data: 2024-02-01T10:47:25.261 DEBUG:boto:Headers: {} 2024-02-01T10:47:25.261 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:25.261 DEBUG:boto:Port: 8000 2024-02-01T10:47:25.261 DEBUG:boto:Params: {} 2024-02-01T10:47:25.262 DEBUG:boto:Token: None 2024-02-01T10:47:25.262 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:25 GMT /vclfcn-60/ 2024-02-01T10:47:25.262 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:dJTC03LkZB4q9jmx0PdaXDdoC1k= 2024-02-01T10:47:25.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:25 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dJTC03LkZB4q9jmx0PdaXDdoC1k=', 'Content-Length': '0'} 2024-02-01T10:47:25.266 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000098fdf1e92c34d278-0065bb76bd-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:25.266 DEBUG:boto:b'vclfcn-60obj41000falseobj42024-02-01T10:46:46.336Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:25.267 DEBUG:boto:path=/vclfcn-60/obj4 2024-02-01T10:47:25.267 DEBUG:boto:auth_path=/vclfcn-60/obj4 2024-02-01T10:47:25.267 DEBUG:boto:Method: HEAD 2024-02-01T10:47:25.267 DEBUG:boto:Path: /vclfcn-60/obj4 2024-02-01T10:47:25.267 DEBUG:boto:Data: 2024-02-01T10:47:25.267 DEBUG:boto:Headers: {} 2024-02-01T10:47:25.267 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:25.267 DEBUG:boto:Port: 8001 2024-02-01T10:47:25.267 DEBUG:boto:Params: {} 2024-02-01T10:47:25.268 DEBUG:boto:Token: None 2024-02-01T10:47:25.268 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:25 GMT /vclfcn-60/obj4 2024-02-01T10:47:25.268 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8015Hx/u0ZfVn1ULl9p+muvG39o= 2024-02-01T10:47:25.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:25 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8015Hx/u0ZfVn1ULl9p+muvG39o=', 'Content-Length': '0'} 2024-02-01T10:47:25.271 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000091b0ff24afbbcf08-0065bb76bd-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:25.271 DEBUG:boto:path=/vclfcn-60/obj4 2024-02-01T10:47:25.271 DEBUG:boto:auth_path=/vclfcn-60/obj4 2024-02-01T10:47:25.271 DEBUG:boto:Method: HEAD 2024-02-01T10:47:25.271 DEBUG:boto:Path: /vclfcn-60/obj4 2024-02-01T10:47:25.271 DEBUG:boto:Data: 2024-02-01T10:47:25.272 DEBUG:boto:Headers: {} 2024-02-01T10:47:25.272 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:25.272 DEBUG:boto:Port: 8000 2024-02-01T10:47:25.272 DEBUG:boto:Params: {} 2024-02-01T10:47:25.272 DEBUG:boto:Token: None 2024-02-01T10:47:25.272 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:25 GMT /vclfcn-60/obj4 2024-02-01T10:47:25.272 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8015Hx/u0ZfVn1ULl9p+muvG39o= 2024-02-01T10:47:25.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:25 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8015Hx/u0ZfVn1ULl9p+muvG39o=', 'Content-Length': '0'} 2024-02-01T10:47:25.275 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000066be46e8914b2a56-0065bb76bd-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:25.275 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:47:25.275 DEBUG:boto:path=/vclfcn-60/obj4 2024-02-01T10:47:25.275 DEBUG:boto:auth_path=/vclfcn-60/obj4 2024-02-01T10:47:25.275 DEBUG:boto:Method: GET 2024-02-01T10:47:25.276 DEBUG:boto:Path: /vclfcn-60/obj4 2024-02-01T10:47:25.276 DEBUG:boto:Data: 2024-02-01T10:47:25.276 DEBUG:boto:Headers: {} 2024-02-01T10:47:25.276 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:25.276 DEBUG:boto:Port: 8001 2024-02-01T10:47:25.276 DEBUG:boto:Params: {} 2024-02-01T10:47:25.276 DEBUG:boto:Token: None 2024-02-01T10:47:25.276 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:25 GMT /vclfcn-60/obj4 2024-02-01T10:47:25.276 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:MdHtrzh98qwcpCv7AccXu5JTqzA= 2024-02-01T10:47:25.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:25 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:MdHtrzh98qwcpCv7AccXu5JTqzA=', 'Content-Length': '0'} 2024-02-01T10:47:25.279 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c99ffe83a7803bd0-0065bb76bd-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:25.320 DEBUG:boto:path=/vclfcn-60/obj4 2024-02-01T10:47:25.320 DEBUG:boto:auth_path=/vclfcn-60/obj4 2024-02-01T10:47:25.320 DEBUG:boto:Method: GET 2024-02-01T10:47:25.320 DEBUG:boto:Path: /vclfcn-60/obj4 2024-02-01T10:47:25.321 DEBUG:boto:Data: 2024-02-01T10:47:25.321 DEBUG:boto:Headers: {} 2024-02-01T10:47:25.321 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:25.321 DEBUG:boto:Port: 8000 2024-02-01T10:47:25.321 DEBUG:boto:Params: {} 2024-02-01T10:47:25.321 DEBUG:boto:Token: None 2024-02-01T10:47:25.321 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:25 GMT /vclfcn-60/obj4 2024-02-01T10:47:25.321 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:MdHtrzh98qwcpCv7AccXu5JTqzA= 2024-02-01T10:47:25.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:25 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:MdHtrzh98qwcpCv7AccXu5JTqzA=', 'Content-Length': '0'} 2024-02-01T10:47:25.324 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f32edf5b47dc2e87-0065bb76bd-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:25.325 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:47:25.325 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:47:25.325 DEBUG:boto:path=/vclfcn-60/?prefix=obj4 2024-02-01T10:47:25.325 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj4 2024-02-01T10:47:25.325 DEBUG:boto:Method: GET 2024-02-01T10:47:25.325 DEBUG:boto:Path: /vclfcn-60/?prefix=obj4 2024-02-01T10:47:25.325 DEBUG:boto:Data: 2024-02-01T10:47:25.325 DEBUG:boto:Headers: {} 2024-02-01T10:47:25.325 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:25.326 DEBUG:boto:Port: 8001 2024-02-01T10:47:25.326 DEBUG:boto:Params: {} 2024-02-01T10:47:25.326 DEBUG:boto:Token: None 2024-02-01T10:47:25.326 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:25 GMT /vclfcn-60/ 2024-02-01T10:47:25.326 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:dJTC03LkZB4q9jmx0PdaXDdoC1k= 2024-02-01T10:47:25.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:25 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dJTC03LkZB4q9jmx0PdaXDdoC1k=', 'Content-Length': '0'} 2024-02-01T10:47:25.331 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b7013eac89760b9-0065bb76bd-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:25.331 DEBUG:boto:b'vclfcn-60obj41000falseobj42024-02-01T10:46:46.336Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:25.331 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:47:25.331 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:47:25.332 DEBUG:boto:path=/vclfcn-60/?prefix=obj4 2024-02-01T10:47:25.332 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj4 2024-02-01T10:47:25.332 DEBUG:boto:Method: GET 2024-02-01T10:47:25.332 DEBUG:boto:Path: /vclfcn-60/?prefix=obj4 2024-02-01T10:47:25.332 DEBUG:boto:Data: 2024-02-01T10:47:25.332 DEBUG:boto:Headers: {} 2024-02-01T10:47:25.332 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:25.332 DEBUG:boto:Port: 8000 2024-02-01T10:47:25.332 DEBUG:boto:Params: {} 2024-02-01T10:47:25.332 DEBUG:boto:Token: None 2024-02-01T10:47:25.332 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:25 GMT /vclfcn-60/ 2024-02-01T10:47:25.333 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:dJTC03LkZB4q9jmx0PdaXDdoC1k= 2024-02-01T10:47:25.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:25 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dJTC03LkZB4q9jmx0PdaXDdoC1k=', 'Content-Length': '0'} 2024-02-01T10:47:25.337 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da053f6834fa3dd4-0065bb76bd-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:25 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:25.338 DEBUG:boto:b'vclfcn-60obj41000falseobj42024-02-01T10:46:46.336Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:25.338 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-60 zones={test-zone3, test-zone2} 2024-02-01T10:47:25.338 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=vclfcn-61 2024-02-01T10:47:25.338 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:47:25.339 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:47:25.339 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket vclfcn-61 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-02-01T10:47:25.390 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:47:25.390 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:47:25.455 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:47:25.454+0000 7f3f6879d980 1 bucket sync caught up with source: 2024-02-01T10:47:25.456 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [00000000001.60.6, , 00000000001.39.6, , 00000000001.163.6, , , 00000000001.45.6, , , ] 2024-02-01T10:47:25.456 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [00000000001.60.6, , 00000000001.39.6, , 00000000001.163.6, , , 00000000001.45.6, , , ] 2024-02-01T10:47:25.456 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:47:25.454+0000 7f3f6879d980 0 bucket checkpoint complete 2024-02-01T10:47:25.479 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=vclfcn-61 2024-02-01T10:47:25.479 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:47:25.479 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:47:25.480 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-61 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:47:25.531 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:47:25.531 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:47:25.593 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:47:25.594+0000 7f574471b980 1 bucket sync caught up with source: 2024-02-01T10:47:25.593 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [00000000001.60.6, , 00000000001.39.6, , 00000000001.163.6, , , 00000000001.45.6, , , ] 2024-02-01T10:47:25.593 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [00000000001.60.6, , 00000000001.39.6, , 00000000001.163.6, , , 00000000001.45.6, , , ] 2024-02-01T10:47:25.593 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:47:25.594+0000 7f574471b980 0 bucket checkpoint complete 2024-02-01T10:47:25.615 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=vclfcn-61 2024-02-01T10:47:25.615 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-61', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:47:25.616 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-61', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:47:25.616 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-61 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:47:25.657 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:47:25.657 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:47:25.718 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:47:25.718+0000 7f26881e8980 1 bucket sync caught up with source: 2024-02-01T10:47:25.718 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [00000000001.246.6, , 00000000001.230.6, , 00000000001.419.6, , , 00000000001.259.6, , , ] 2024-02-01T10:47:25.718 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [00000000001.246.6, , 00000000001.230.6, , 00000000001.419.6, , , 00000000001.259.6, , , ] 2024-02-01T10:47:25.719 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:47:25.718+0000 7f26881e8980 0 bucket checkpoint complete 2024-02-01T10:47:25.744 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=vclfcn-61 2024-02-01T10:47:25.744 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-61', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:47:25.744 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-61', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:47:25.745 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-61 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:47:25.829 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:47:25.829 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:47:25.890 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:47:25.890+0000 7fa645c35980 1 bucket sync caught up with source: 2024-02-01T10:47:25.891 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [00000000001.246.6, , 00000000001.230.6, , 00000000001.419.6, , , 00000000001.259.6, , , ] 2024-02-01T10:47:25.891 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [00000000001.246.6, , 00000000001.230.6, , 00000000001.419.6, , , 00000000001.259.6, , , ] 2024-02-01T10:47:25.891 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:47:25.890+0000 7fa645c35980 0 bucket checkpoint complete 2024-02-01T10:47:25.913 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=vclfcn-61 2024-02-01T10:47:25.913 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-61', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:47:25.913 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-61', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:47:25.913 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-61 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:47:25.998 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:47:25.998 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:47:26.057 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:47:26.058+0000 7f366b310980 1 bucket sync caught up with source: 2024-02-01T10:47:26.057 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [00000000001.204.6, , 00000000001.176.6, , 00000000001.357.6, , , 00000000001.163.6, , , ] 2024-02-01T10:47:26.057 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [00000000001.204.6, , 00000000001.176.6, , 00000000001.357.6, , , 00000000001.163.6, , , ] 2024-02-01T10:47:26.057 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:47:26.058+0000 7f366b310980 0 bucket checkpoint complete 2024-02-01T10:47:26.080 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=vclfcn-61 2024-02-01T10:47:26.080 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-61', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:47:26.081 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-61', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:47:26.081 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket vclfcn-61 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-02-01T10:47:26.130 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:47:26.131 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:47:26.195 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:47:26.194+0000 7efe3aa9b980 1 bucket sync caught up with source: 2024-02-01T10:47:26.195 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [00000000001.204.6, , 00000000001.176.6, , 00000000001.357.6, , , 00000000001.163.6, , , ] 2024-02-01T10:47:26.195 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [00000000001.204.6, , 00000000001.176.6, , 00000000001.357.6, , , 00000000001.163.6, , , ] 2024-02-01T10:47:26.195 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:47:26.194+0000 7efe3aa9b980 0 bucket checkpoint complete 2024-02-01T10:47:26.219 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-61 zones={test-zone2, test-zone1} 2024-02-01T10:47:26.220 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:47:26.220 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:47:26.220 DEBUG:boto:Method: HEAD 2024-02-01T10:47:26.220 DEBUG:boto:Path: /vclfcn-61/ 2024-02-01T10:47:26.221 DEBUG:boto:Data: 2024-02-01T10:47:26.221 DEBUG:boto:Headers: {} 2024-02-01T10:47:26.221 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:26.221 DEBUG:boto:Port: 8000 2024-02-01T10:47:26.221 DEBUG:boto:Params: {} 2024-02-01T10:47:26.222 DEBUG:boto:Token: None 2024-02-01T10:47:26.222 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:26 GMT /vclfcn-61/ 2024-02-01T10:47:26.222 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Y8555ORR8tA6oY1Ez65ghH1sFvE= 2024-02-01T10:47:26.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:26 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Y8555ORR8tA6oY1Ez65ghH1sFvE=', 'Content-Length': '0'} 2024-02-01T10:47:26.225 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005239866e346c80bb-0065bb76be-5350-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:26.225 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:47:26.225 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:47:26.225 DEBUG:boto:Method: HEAD 2024-02-01T10:47:26.225 DEBUG:boto:Path: /vclfcn-61/ 2024-02-01T10:47:26.225 DEBUG:boto:Data: 2024-02-01T10:47:26.225 DEBUG:boto:Headers: {} 2024-02-01T10:47:26.226 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:26.226 DEBUG:boto:Port: 8000 2024-02-01T10:47:26.226 DEBUG:boto:Params: {} 2024-02-01T10:47:26.226 DEBUG:boto:Token: None 2024-02-01T10:47:26.226 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:26 GMT /vclfcn-61/ 2024-02-01T10:47:26.226 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Y8555ORR8tA6oY1Ez65ghH1sFvE= 2024-02-01T10:47:26.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:26 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Y8555ORR8tA6oY1Ez65ghH1sFvE=', 'Content-Length': '0'} 2024-02-01T10:47:26.230 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003fa63c9b7711dc08-0065bb76be-5765-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:26.231 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:47:26.231 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:47:26.231 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:47:26.231 DEBUG:boto:path=/vclfcn-61/?versions&max-keys=999 2024-02-01T10:47:26.231 DEBUG:boto:auth_path=/vclfcn-61/?versions&max-keys=999 2024-02-01T10:47:26.232 DEBUG:boto:Method: GET 2024-02-01T10:47:26.232 DEBUG:boto:Path: /vclfcn-61/?versions&max-keys=999 2024-02-01T10:47:26.232 DEBUG:boto:Data: 2024-02-01T10:47:26.232 DEBUG:boto:Headers: {} 2024-02-01T10:47:26.232 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:26.232 DEBUG:boto:Port: 8000 2024-02-01T10:47:26.232 DEBUG:boto:Params: {} 2024-02-01T10:47:26.233 DEBUG:boto:Token: None 2024-02-01T10:47:26.233 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:26 GMT /vclfcn-61/?versions 2024-02-01T10:47:26.233 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Lt2ijtgueZAMpKwJOe1Gpmidd3Q= 2024-02-01T10:47:26.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:26 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Lt2ijtgueZAMpKwJOe1Gpmidd3Q=', 'Content-Length': '0'} 2024-02-01T10:47:26.235 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e76753e843d1b88e-0065bb76be-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:26.235 DEBUG:boto:b'vclfcn-61999falseobj1nulltrue2024-02-01T10:46:46.359Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:46:46.380Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:46:46.433Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:46:46.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:26.236 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-02-01T10:47:26.236 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-02-01T10:47:26.236 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-02-01T10:47:26.236 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-02-01T10:47:26.237 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:47:26.237 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:47:26.237 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:47:26.237 DEBUG:boto:path=/vclfcn-61/?versions&max-keys=999 2024-02-01T10:47:26.237 DEBUG:boto:auth_path=/vclfcn-61/?versions&max-keys=999 2024-02-01T10:47:26.237 DEBUG:boto:Method: GET 2024-02-01T10:47:26.237 DEBUG:boto:Path: /vclfcn-61/?versions&max-keys=999 2024-02-01T10:47:26.237 DEBUG:boto:Data: 2024-02-01T10:47:26.237 DEBUG:boto:Headers: {} 2024-02-01T10:47:26.237 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:26.238 DEBUG:boto:Port: 8000 2024-02-01T10:47:26.238 DEBUG:boto:Params: {} 2024-02-01T10:47:26.238 DEBUG:boto:Token: None 2024-02-01T10:47:26.238 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:26 GMT /vclfcn-61/?versions 2024-02-01T10:47:26.238 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Lt2ijtgueZAMpKwJOe1Gpmidd3Q= 2024-02-01T10:47:26.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:26 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Lt2ijtgueZAMpKwJOe1Gpmidd3Q=', 'Content-Length': '0'} 2024-02-01T10:47:26.242 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b03908f8a0168347-0065bb76be-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:26.242 DEBUG:boto:b'vclfcn-61999falseobj1nulltrue2024-02-01T10:46:46.359Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:46:46.380Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:46:46.433Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:46:46.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:26.243 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-02-01T10:47:26.243 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-02-01T10:47:26.243 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-02-01T10:47:26.243 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-02-01T10:47:26.243 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:47:26.243 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:47:26.243 DEBUG:boto:path=/vclfcn-61/?versions&max-keys=999 2024-02-01T10:47:26.243 DEBUG:boto:auth_path=/vclfcn-61/?versions&max-keys=999 2024-02-01T10:47:26.244 DEBUG:boto:Method: GET 2024-02-01T10:47:26.244 DEBUG:boto:Path: /vclfcn-61/?versions&max-keys=999 2024-02-01T10:47:26.244 DEBUG:boto:Data: 2024-02-01T10:47:26.244 DEBUG:boto:Headers: {} 2024-02-01T10:47:26.244 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:26.244 DEBUG:boto:Port: 8000 2024-02-01T10:47:26.244 DEBUG:boto:Params: {} 2024-02-01T10:47:26.244 DEBUG:boto:Token: None 2024-02-01T10:47:26.244 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:26 GMT /vclfcn-61/?versions 2024-02-01T10:47:26.245 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Lt2ijtgueZAMpKwJOe1Gpmidd3Q= 2024-02-01T10:47:26.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:26 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Lt2ijtgueZAMpKwJOe1Gpmidd3Q=', 'Content-Length': '0'} 2024-02-01T10:47:26.247 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a23c50f442def621-0065bb76be-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:26.247 DEBUG:boto:b'vclfcn-61999falseobj1nulltrue2024-02-01T10:46:46.359Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:46:46.380Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:46:46.433Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:46:46.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:26.248 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:47:26.248 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:47:26.248 DEBUG:boto:path=/vclfcn-61/?versions&max-keys=999 2024-02-01T10:47:26.248 DEBUG:boto:auth_path=/vclfcn-61/?versions&max-keys=999 2024-02-01T10:47:26.248 DEBUG:boto:Method: GET 2024-02-01T10:47:26.249 DEBUG:boto:Path: /vclfcn-61/?versions&max-keys=999 2024-02-01T10:47:26.249 DEBUG:boto:Data: 2024-02-01T10:47:26.249 DEBUG:boto:Headers: {} 2024-02-01T10:47:26.249 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:26.249 DEBUG:boto:Port: 8000 2024-02-01T10:47:26.249 DEBUG:boto:Params: {} 2024-02-01T10:47:26.249 DEBUG:boto:Token: None 2024-02-01T10:47:26.249 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:26 GMT /vclfcn-61/?versions 2024-02-01T10:47:26.250 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Lt2ijtgueZAMpKwJOe1Gpmidd3Q= 2024-02-01T10:47:26.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:26 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Lt2ijtgueZAMpKwJOe1Gpmidd3Q=', 'Content-Length': '0'} 2024-02-01T10:47:26.254 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f16c43fbb01922f8-0065bb76be-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:26.254 DEBUG:boto:b'vclfcn-61999falseobj1nulltrue2024-02-01T10:46:46.359Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:46:46.380Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:46:46.433Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:46:46.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:26.254 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:47:26.255 DEBUG:boto:path=/vclfcn-61/obj1 2024-02-01T10:47:26.255 DEBUG:boto:auth_path=/vclfcn-61/obj1 2024-02-01T10:47:26.255 DEBUG:boto:path=/vclfcn-61/obj1?versionId=null 2024-02-01T10:47:26.255 DEBUG:boto:auth_path=/vclfcn-61/obj1?versionId=null 2024-02-01T10:47:26.255 DEBUG:boto:Method: GET 2024-02-01T10:47:26.255 DEBUG:boto:Path: /vclfcn-61/obj1?versionId=null 2024-02-01T10:47:26.255 DEBUG:boto:Data: 2024-02-01T10:47:26.255 DEBUG:boto:Headers: {} 2024-02-01T10:47:26.255 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:26.255 DEBUG:boto:Port: 8000 2024-02-01T10:47:26.255 DEBUG:boto:Params: {} 2024-02-01T10:47:26.256 DEBUG:boto:Token: None 2024-02-01T10:47:26.256 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:26 GMT /vclfcn-61/obj1?versionId=null 2024-02-01T10:47:26.256 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WLhPMD/iiL5+Yk45sD2VA7TqzeI= 2024-02-01T10:47:26.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:26 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WLhPMD/iiL5+Yk45sD2VA7TqzeI=', 'Content-Length': '0'} 2024-02-01T10:47:26.260 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003764e97aa6a62aac-0065bb76be-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:26.300 DEBUG:boto:path=/vclfcn-61/obj1 2024-02-01T10:47:26.300 DEBUG:boto:auth_path=/vclfcn-61/obj1 2024-02-01T10:47:26.300 DEBUG:boto:path=/vclfcn-61/obj1?versionId=null 2024-02-01T10:47:26.300 DEBUG:boto:auth_path=/vclfcn-61/obj1?versionId=null 2024-02-01T10:47:26.301 DEBUG:boto:Method: GET 2024-02-01T10:47:26.301 DEBUG:boto:Path: /vclfcn-61/obj1?versionId=null 2024-02-01T10:47:26.301 DEBUG:boto:Data: 2024-02-01T10:47:26.301 DEBUG:boto:Headers: {} 2024-02-01T10:47:26.301 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:26.301 DEBUG:boto:Port: 8000 2024-02-01T10:47:26.301 DEBUG:boto:Params: {} 2024-02-01T10:47:26.301 DEBUG:boto:Token: None 2024-02-01T10:47:26.301 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:26 GMT /vclfcn-61/obj1?versionId=null 2024-02-01T10:47:26.302 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WLhPMD/iiL5+Yk45sD2VA7TqzeI= 2024-02-01T10:47:26.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:26 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WLhPMD/iiL5+Yk45sD2VA7TqzeI=', 'Content-Length': '0'} 2024-02-01T10:47:26.305 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001b8ee25b15229ede-0065bb76be-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:26.305 DEBUG:boto:path=/vclfcn-61/obj1 2024-02-01T10:47:26.305 DEBUG:boto:auth_path=/vclfcn-61/obj1 2024-02-01T10:47:26.305 DEBUG:boto:path=/vclfcn-61/obj1?versionId=null 2024-02-01T10:47:26.305 DEBUG:boto:auth_path=/vclfcn-61/obj1?versionId=null 2024-02-01T10:47:26.305 DEBUG:boto:Method: HEAD 2024-02-01T10:47:26.305 DEBUG:boto:Path: /vclfcn-61/obj1?versionId=null 2024-02-01T10:47:26.306 DEBUG:boto:Data: 2024-02-01T10:47:26.306 DEBUG:boto:Headers: {} 2024-02-01T10:47:26.306 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:26.306 DEBUG:boto:Port: 8000 2024-02-01T10:47:26.306 DEBUG:boto:Params: {} 2024-02-01T10:47:26.306 DEBUG:boto:Token: None 2024-02-01T10:47:26.306 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:26 GMT /vclfcn-61/obj1?versionId=null 2024-02-01T10:47:26.306 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:qaiC7M3aIStfflszyx5l9pPQvdU= 2024-02-01T10:47:26.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:26 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:qaiC7M3aIStfflszyx5l9pPQvdU=', 'Content-Length': '0'} 2024-02-01T10:47:26.309 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002883502267f77e53-0065bb76be-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:26.309 DEBUG:boto:path=/vclfcn-61/obj1 2024-02-01T10:47:26.309 DEBUG:boto:auth_path=/vclfcn-61/obj1 2024-02-01T10:47:26.309 DEBUG:boto:path=/vclfcn-61/obj1?versionId=null 2024-02-01T10:47:26.309 DEBUG:boto:auth_path=/vclfcn-61/obj1?versionId=null 2024-02-01T10:47:26.309 DEBUG:boto:Method: HEAD 2024-02-01T10:47:26.309 DEBUG:boto:Path: /vclfcn-61/obj1?versionId=null 2024-02-01T10:47:26.309 DEBUG:boto:Data: 2024-02-01T10:47:26.310 DEBUG:boto:Headers: {} 2024-02-01T10:47:26.310 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:26.310 DEBUG:boto:Port: 8000 2024-02-01T10:47:26.310 DEBUG:boto:Params: {} 2024-02-01T10:47:26.310 DEBUG:boto:Token: None 2024-02-01T10:47:26.310 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:26 GMT /vclfcn-61/obj1?versionId=null 2024-02-01T10:47:26.310 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:qaiC7M3aIStfflszyx5l9pPQvdU= 2024-02-01T10:47:26.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:26 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:qaiC7M3aIStfflszyx5l9pPQvdU=', 'Content-Length': '0'} 2024-02-01T10:47:26.313 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e0f381f721e8af46-0065bb76be-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:26.313 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:47:26.313 DEBUG:boto:path=/vclfcn-61/obj1 2024-02-01T10:47:26.313 DEBUG:boto:auth_path=/vclfcn-61/obj1 2024-02-01T10:47:26.313 DEBUG:boto:path=/vclfcn-61/obj1?versionId=null 2024-02-01T10:47:26.313 DEBUG:boto:auth_path=/vclfcn-61/obj1?versionId=null 2024-02-01T10:47:26.314 DEBUG:boto:Method: GET 2024-02-01T10:47:26.314 DEBUG:boto:Path: /vclfcn-61/obj1?versionId=null 2024-02-01T10:47:26.314 DEBUG:boto:Data: 2024-02-01T10:47:26.314 DEBUG:boto:Headers: {} 2024-02-01T10:47:26.314 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:26.314 DEBUG:boto:Port: 8000 2024-02-01T10:47:26.314 DEBUG:boto:Params: {} 2024-02-01T10:47:26.314 DEBUG:boto:Token: None 2024-02-01T10:47:26.314 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:26 GMT /vclfcn-61/obj1?versionId=null 2024-02-01T10:47:26.314 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WLhPMD/iiL5+Yk45sD2VA7TqzeI= 2024-02-01T10:47:26.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:26 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WLhPMD/iiL5+Yk45sD2VA7TqzeI=', 'Content-Length': '0'} 2024-02-01T10:47:26.317 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009d9bc78946257774-0065bb76be-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:26.317 DEBUG:boto:path=/vclfcn-61/obj1 2024-02-01T10:47:26.317 DEBUG:boto:auth_path=/vclfcn-61/obj1 2024-02-01T10:47:26.317 DEBUG:boto:path=/vclfcn-61/obj1?versionId=null 2024-02-01T10:47:26.317 DEBUG:boto:auth_path=/vclfcn-61/obj1?versionId=null 2024-02-01T10:47:26.318 DEBUG:boto:Method: GET 2024-02-01T10:47:26.318 DEBUG:boto:Path: /vclfcn-61/obj1?versionId=null 2024-02-01T10:47:26.318 DEBUG:boto:Data: 2024-02-01T10:47:26.318 DEBUG:boto:Headers: {} 2024-02-01T10:47:26.318 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:26.318 DEBUG:boto:Port: 8000 2024-02-01T10:47:26.318 DEBUG:boto:Params: {} 2024-02-01T10:47:26.318 DEBUG:boto:Token: None 2024-02-01T10:47:26.318 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:26 GMT /vclfcn-61/obj1?versionId=null 2024-02-01T10:47:26.318 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WLhPMD/iiL5+Yk45sD2VA7TqzeI= 2024-02-01T10:47:26.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:26 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WLhPMD/iiL5+Yk45sD2VA7TqzeI=', 'Content-Length': '0'} 2024-02-01T10:47:26.321 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c15209eb5ccd75b6-0065bb76be-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:26.364 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:47:26.364 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:47:26.364 DEBUG:boto:path=/vclfcn-61/?prefix=obj1 2024-02-01T10:47:26.365 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj1 2024-02-01T10:47:26.365 DEBUG:boto:Method: GET 2024-02-01T10:47:26.365 DEBUG:boto:Path: /vclfcn-61/?prefix=obj1 2024-02-01T10:47:26.365 DEBUG:boto:Data: 2024-02-01T10:47:26.365 DEBUG:boto:Headers: {} 2024-02-01T10:47:26.365 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:26.365 DEBUG:boto:Port: 8000 2024-02-01T10:47:26.365 DEBUG:boto:Params: {} 2024-02-01T10:47:26.365 DEBUG:boto:Token: None 2024-02-01T10:47:26.366 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:26 GMT /vclfcn-61/ 2024-02-01T10:47:26.366 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YCedi6sZ0ILB+0Vdi3FmEwuqNck= 2024-02-01T10:47:26.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:26 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YCedi6sZ0ILB+0Vdi3FmEwuqNck=', 'Content-Length': '0'} 2024-02-01T10:47:26.370 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c613cf98e4f8673e-0065bb76be-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:26.371 DEBUG:boto:b'vclfcn-61obj11000falseobj12024-02-01T10:46:46.359Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:26.371 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:47:26.371 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:47:26.371 DEBUG:boto:path=/vclfcn-61/?prefix=obj1 2024-02-01T10:47:26.372 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj1 2024-02-01T10:47:26.372 DEBUG:boto:Method: GET 2024-02-01T10:47:26.372 DEBUG:boto:Path: /vclfcn-61/?prefix=obj1 2024-02-01T10:47:26.372 DEBUG:boto:Data: 2024-02-01T10:47:26.372 DEBUG:boto:Headers: {} 2024-02-01T10:47:26.372 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:26.372 DEBUG:boto:Port: 8000 2024-02-01T10:47:26.372 DEBUG:boto:Params: {} 2024-02-01T10:47:26.372 DEBUG:boto:Token: None 2024-02-01T10:47:26.372 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:26 GMT /vclfcn-61/ 2024-02-01T10:47:26.373 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YCedi6sZ0ILB+0Vdi3FmEwuqNck= 2024-02-01T10:47:26.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:26 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YCedi6sZ0ILB+0Vdi3FmEwuqNck=', 'Content-Length': '0'} 2024-02-01T10:47:26.377 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000009ce70877942296-0065bb76be-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:26.377 DEBUG:boto:b'vclfcn-61obj11000falseobj12024-02-01T10:46:46.359Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:26.377 DEBUG:boto:path=/vclfcn-61/obj1 2024-02-01T10:47:26.377 DEBUG:boto:auth_path=/vclfcn-61/obj1 2024-02-01T10:47:26.377 DEBUG:boto:Method: HEAD 2024-02-01T10:47:26.377 DEBUG:boto:Path: /vclfcn-61/obj1 2024-02-01T10:47:26.377 DEBUG:boto:Data: 2024-02-01T10:47:26.378 DEBUG:boto:Headers: {} 2024-02-01T10:47:26.378 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:26.378 DEBUG:boto:Port: 8000 2024-02-01T10:47:26.378 DEBUG:boto:Params: {} 2024-02-01T10:47:26.378 DEBUG:boto:Token: None 2024-02-01T10:47:26.378 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:26 GMT /vclfcn-61/obj1 2024-02-01T10:47:26.378 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gxQkuDC/Pt6xhvLRRfMmby1iPII= 2024-02-01T10:47:26.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:26 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gxQkuDC/Pt6xhvLRRfMmby1iPII=', 'Content-Length': '0'} 2024-02-01T10:47:26.381 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003f9e307a56897776-0065bb76be-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:26.381 DEBUG:boto:path=/vclfcn-61/obj1 2024-02-01T10:47:26.381 DEBUG:boto:auth_path=/vclfcn-61/obj1 2024-02-01T10:47:26.381 DEBUG:boto:Method: HEAD 2024-02-01T10:47:26.381 DEBUG:boto:Path: /vclfcn-61/obj1 2024-02-01T10:47:26.382 DEBUG:boto:Data: 2024-02-01T10:47:26.382 DEBUG:boto:Headers: {} 2024-02-01T10:47:26.382 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:26.382 DEBUG:boto:Port: 8000 2024-02-01T10:47:26.382 DEBUG:boto:Params: {} 2024-02-01T10:47:26.382 DEBUG:boto:Token: None 2024-02-01T10:47:26.382 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:26 GMT /vclfcn-61/obj1 2024-02-01T10:47:26.382 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gxQkuDC/Pt6xhvLRRfMmby1iPII= 2024-02-01T10:47:26.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:26 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gxQkuDC/Pt6xhvLRRfMmby1iPII=', 'Content-Length': '0'} 2024-02-01T10:47:26.385 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009b619329d8857310-0065bb76be-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:26.386 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:47:26.386 DEBUG:boto:path=/vclfcn-61/obj1 2024-02-01T10:47:26.386 DEBUG:boto:auth_path=/vclfcn-61/obj1 2024-02-01T10:47:26.386 DEBUG:boto:Method: GET 2024-02-01T10:47:26.386 DEBUG:boto:Path: /vclfcn-61/obj1 2024-02-01T10:47:26.386 DEBUG:boto:Data: 2024-02-01T10:47:26.386 DEBUG:boto:Headers: {} 2024-02-01T10:47:26.386 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:26.386 DEBUG:boto:Port: 8000 2024-02-01T10:47:26.386 DEBUG:boto:Params: {} 2024-02-01T10:47:26.387 DEBUG:boto:Token: None 2024-02-01T10:47:26.387 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:26 GMT /vclfcn-61/obj1 2024-02-01T10:47:26.387 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:DOXagsLRAq1U1isOGxjFbRnq5qQ= 2024-02-01T10:47:26.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:26 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:DOXagsLRAq1U1isOGxjFbRnq5qQ=', 'Content-Length': '0'} 2024-02-01T10:47:26.390 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000051951ac761cf2e72-0065bb76be-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:26.432 DEBUG:boto:path=/vclfcn-61/obj1 2024-02-01T10:47:26.432 DEBUG:boto:auth_path=/vclfcn-61/obj1 2024-02-01T10:47:26.432 DEBUG:boto:Method: GET 2024-02-01T10:47:26.433 DEBUG:boto:Path: /vclfcn-61/obj1 2024-02-01T10:47:26.433 DEBUG:boto:Data: 2024-02-01T10:47:26.433 DEBUG:boto:Headers: {} 2024-02-01T10:47:26.433 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:26.433 DEBUG:boto:Port: 8000 2024-02-01T10:47:26.433 DEBUG:boto:Params: {} 2024-02-01T10:47:26.433 DEBUG:boto:Token: None 2024-02-01T10:47:26.433 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:26 GMT /vclfcn-61/obj1 2024-02-01T10:47:26.433 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:DOXagsLRAq1U1isOGxjFbRnq5qQ= 2024-02-01T10:47:26.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:26 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:DOXagsLRAq1U1isOGxjFbRnq5qQ=', 'Content-Length': '0'} 2024-02-01T10:47:26.436 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c911268ae5b1a3e9-0065bb76be-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:26.437 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:47:26.437 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:47:26.437 DEBUG:boto:path=/vclfcn-61/?prefix=obj1 2024-02-01T10:47:26.437 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj1 2024-02-01T10:47:26.437 DEBUG:boto:Method: GET 2024-02-01T10:47:26.437 DEBUG:boto:Path: /vclfcn-61/?prefix=obj1 2024-02-01T10:47:26.437 DEBUG:boto:Data: 2024-02-01T10:47:26.437 DEBUG:boto:Headers: {} 2024-02-01T10:47:26.437 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:26.437 DEBUG:boto:Port: 8000 2024-02-01T10:47:26.438 DEBUG:boto:Params: {} 2024-02-01T10:47:26.438 DEBUG:boto:Token: None 2024-02-01T10:47:26.438 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:26 GMT /vclfcn-61/ 2024-02-01T10:47:26.438 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YCedi6sZ0ILB+0Vdi3FmEwuqNck= 2024-02-01T10:47:26.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:26 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YCedi6sZ0ILB+0Vdi3FmEwuqNck=', 'Content-Length': '0'} 2024-02-01T10:47:26.442 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000290618717a45d2af-0065bb76be-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:26.442 DEBUG:boto:b'vclfcn-61obj11000falseobj12024-02-01T10:46:46.359Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:26.443 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:47:26.443 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:47:26.443 DEBUG:boto:path=/vclfcn-61/?prefix=obj1 2024-02-01T10:47:26.443 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj1 2024-02-01T10:47:26.443 DEBUG:boto:Method: GET 2024-02-01T10:47:26.443 DEBUG:boto:Path: /vclfcn-61/?prefix=obj1 2024-02-01T10:47:26.443 DEBUG:boto:Data: 2024-02-01T10:47:26.443 DEBUG:boto:Headers: {} 2024-02-01T10:47:26.443 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:26.443 DEBUG:boto:Port: 8000 2024-02-01T10:47:26.444 DEBUG:boto:Params: {} 2024-02-01T10:47:26.444 DEBUG:boto:Token: None 2024-02-01T10:47:26.444 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:26 GMT /vclfcn-61/ 2024-02-01T10:47:26.444 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YCedi6sZ0ILB+0Vdi3FmEwuqNck= 2024-02-01T10:47:26.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:26 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YCedi6sZ0ILB+0Vdi3FmEwuqNck=', 'Content-Length': '0'} 2024-02-01T10:47:26.448 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000642b2e1e0d165da9-0065bb76be-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:26.496 DEBUG:boto:b'vclfcn-61obj11000falseobj12024-02-01T10:46:46.359Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:26.496 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:47:26.497 DEBUG:boto:path=/vclfcn-61/obj2 2024-02-01T10:47:26.497 DEBUG:boto:auth_path=/vclfcn-61/obj2 2024-02-01T10:47:26.497 DEBUG:boto:path=/vclfcn-61/obj2?versionId=null 2024-02-01T10:47:26.497 DEBUG:boto:auth_path=/vclfcn-61/obj2?versionId=null 2024-02-01T10:47:26.497 DEBUG:boto:Method: GET 2024-02-01T10:47:26.497 DEBUG:boto:Path: /vclfcn-61/obj2?versionId=null 2024-02-01T10:47:26.497 DEBUG:boto:Data: 2024-02-01T10:47:26.497 DEBUG:boto:Headers: {} 2024-02-01T10:47:26.497 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:26.498 DEBUG:boto:Port: 8000 2024-02-01T10:47:26.498 DEBUG:boto:Params: {} 2024-02-01T10:47:26.498 DEBUG:boto:Token: None 2024-02-01T10:47:26.498 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:26 GMT /vclfcn-61/obj2?versionId=null 2024-02-01T10:47:26.498 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lJqAUTQ6Fix5nFJ/2ETBBWIaYO8= 2024-02-01T10:47:26.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:26 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lJqAUTQ6Fix5nFJ/2ETBBWIaYO8=', 'Content-Length': '0'} 2024-02-01T10:47:26.501 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008f89dce209e3de1a-0065bb76be-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:26.502 DEBUG:boto:path=/vclfcn-61/obj2 2024-02-01T10:47:26.502 DEBUG:boto:auth_path=/vclfcn-61/obj2 2024-02-01T10:47:26.502 DEBUG:boto:path=/vclfcn-61/obj2?versionId=null 2024-02-01T10:47:26.502 DEBUG:boto:auth_path=/vclfcn-61/obj2?versionId=null 2024-02-01T10:47:26.502 DEBUG:boto:Method: GET 2024-02-01T10:47:26.502 DEBUG:boto:Path: /vclfcn-61/obj2?versionId=null 2024-02-01T10:47:26.502 DEBUG:boto:Data: 2024-02-01T10:47:26.502 DEBUG:boto:Headers: {} 2024-02-01T10:47:26.503 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:26.503 DEBUG:boto:Port: 8000 2024-02-01T10:47:26.503 DEBUG:boto:Params: {} 2024-02-01T10:47:26.503 DEBUG:boto:Token: None 2024-02-01T10:47:26.503 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:26 GMT /vclfcn-61/obj2?versionId=null 2024-02-01T10:47:26.503 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lJqAUTQ6Fix5nFJ/2ETBBWIaYO8= 2024-02-01T10:47:26.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:26 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lJqAUTQ6Fix5nFJ/2ETBBWIaYO8=', 'Content-Length': '0'} 2024-02-01T10:47:26.506 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005d14da6513ca77ea-0065bb76be-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:26.507 DEBUG:boto:path=/vclfcn-61/obj2 2024-02-01T10:47:26.507 DEBUG:boto:auth_path=/vclfcn-61/obj2 2024-02-01T10:47:26.507 DEBUG:boto:path=/vclfcn-61/obj2?versionId=null 2024-02-01T10:47:26.507 DEBUG:boto:auth_path=/vclfcn-61/obj2?versionId=null 2024-02-01T10:47:26.507 DEBUG:boto:Method: HEAD 2024-02-01T10:47:26.507 DEBUG:boto:Path: /vclfcn-61/obj2?versionId=null 2024-02-01T10:47:26.507 DEBUG:boto:Data: 2024-02-01T10:47:26.507 DEBUG:boto:Headers: {} 2024-02-01T10:47:26.507 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:26.507 DEBUG:boto:Port: 8000 2024-02-01T10:47:26.507 DEBUG:boto:Params: {} 2024-02-01T10:47:26.508 DEBUG:boto:Token: None 2024-02-01T10:47:26.508 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:26 GMT /vclfcn-61/obj2?versionId=null 2024-02-01T10:47:26.508 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:IdQuEj74ge2XCN9HyyC8zaq0KiA= 2024-02-01T10:47:26.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:26 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:IdQuEj74ge2XCN9HyyC8zaq0KiA=', 'Content-Length': '0'} 2024-02-01T10:47:26.510 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000088c000c9c7eb387-0065bb76be-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:26.511 DEBUG:boto:path=/vclfcn-61/obj2 2024-02-01T10:47:26.511 DEBUG:boto:auth_path=/vclfcn-61/obj2 2024-02-01T10:47:26.511 DEBUG:boto:path=/vclfcn-61/obj2?versionId=null 2024-02-01T10:47:26.511 DEBUG:boto:auth_path=/vclfcn-61/obj2?versionId=null 2024-02-01T10:47:26.511 DEBUG:boto:Method: HEAD 2024-02-01T10:47:26.511 DEBUG:boto:Path: /vclfcn-61/obj2?versionId=null 2024-02-01T10:47:26.511 DEBUG:boto:Data: 2024-02-01T10:47:26.511 DEBUG:boto:Headers: {} 2024-02-01T10:47:26.512 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:26.512 DEBUG:boto:Port: 8000 2024-02-01T10:47:26.512 DEBUG:boto:Params: {} 2024-02-01T10:47:26.512 DEBUG:boto:Token: None 2024-02-01T10:47:26.512 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:26 GMT /vclfcn-61/obj2?versionId=null 2024-02-01T10:47:26.512 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:IdQuEj74ge2XCN9HyyC8zaq0KiA= 2024-02-01T10:47:26.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:26 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:IdQuEj74ge2XCN9HyyC8zaq0KiA=', 'Content-Length': '0'} 2024-02-01T10:47:26.515 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c7b50af8def5b2aa-0065bb76be-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:26.515 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:47:26.516 DEBUG:boto:path=/vclfcn-61/obj2 2024-02-01T10:47:26.516 DEBUG:boto:auth_path=/vclfcn-61/obj2 2024-02-01T10:47:26.516 DEBUG:boto:path=/vclfcn-61/obj2?versionId=null 2024-02-01T10:47:26.516 DEBUG:boto:auth_path=/vclfcn-61/obj2?versionId=null 2024-02-01T10:47:26.516 DEBUG:boto:Method: GET 2024-02-01T10:47:26.516 DEBUG:boto:Path: /vclfcn-61/obj2?versionId=null 2024-02-01T10:47:26.516 DEBUG:boto:Data: 2024-02-01T10:47:26.516 DEBUG:boto:Headers: {} 2024-02-01T10:47:26.516 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:26.516 DEBUG:boto:Port: 8000 2024-02-01T10:47:26.517 DEBUG:boto:Params: {} 2024-02-01T10:47:26.517 DEBUG:boto:Token: None 2024-02-01T10:47:26.517 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:26 GMT /vclfcn-61/obj2?versionId=null 2024-02-01T10:47:26.517 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lJqAUTQ6Fix5nFJ/2ETBBWIaYO8= 2024-02-01T10:47:26.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:26 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lJqAUTQ6Fix5nFJ/2ETBBWIaYO8=', 'Content-Length': '0'} 2024-02-01T10:47:26.520 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006d6fea21672a8aae-0065bb76be-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:26.560 DEBUG:boto:path=/vclfcn-61/obj2 2024-02-01T10:47:26.560 DEBUG:boto:auth_path=/vclfcn-61/obj2 2024-02-01T10:47:26.560 DEBUG:boto:path=/vclfcn-61/obj2?versionId=null 2024-02-01T10:47:26.560 DEBUG:boto:auth_path=/vclfcn-61/obj2?versionId=null 2024-02-01T10:47:26.560 DEBUG:boto:Method: GET 2024-02-01T10:47:26.561 DEBUG:boto:Path: /vclfcn-61/obj2?versionId=null 2024-02-01T10:47:26.561 DEBUG:boto:Data: 2024-02-01T10:47:26.561 DEBUG:boto:Headers: {} 2024-02-01T10:47:26.561 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:26.561 DEBUG:boto:Port: 8000 2024-02-01T10:47:26.561 DEBUG:boto:Params: {} 2024-02-01T10:47:26.561 DEBUG:boto:Token: None 2024-02-01T10:47:26.561 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:26 GMT /vclfcn-61/obj2?versionId=null 2024-02-01T10:47:26.561 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lJqAUTQ6Fix5nFJ/2ETBBWIaYO8= 2024-02-01T10:47:26.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:26 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lJqAUTQ6Fix5nFJ/2ETBBWIaYO8=', 'Content-Length': '0'} 2024-02-01T10:47:26.564 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000124ee560aa2f03d2-0065bb76be-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:26.565 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:47:26.565 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:47:26.565 DEBUG:boto:path=/vclfcn-61/?prefix=obj2 2024-02-01T10:47:26.565 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj2 2024-02-01T10:47:26.565 DEBUG:boto:Method: GET 2024-02-01T10:47:26.565 DEBUG:boto:Path: /vclfcn-61/?prefix=obj2 2024-02-01T10:47:26.565 DEBUG:boto:Data: 2024-02-01T10:47:26.565 DEBUG:boto:Headers: {} 2024-02-01T10:47:26.565 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:26.565 DEBUG:boto:Port: 8000 2024-02-01T10:47:26.566 DEBUG:boto:Params: {} 2024-02-01T10:47:26.566 DEBUG:boto:Token: None 2024-02-01T10:47:26.566 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:26 GMT /vclfcn-61/ 2024-02-01T10:47:26.566 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YCedi6sZ0ILB+0Vdi3FmEwuqNck= 2024-02-01T10:47:26.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:26 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YCedi6sZ0ILB+0Vdi3FmEwuqNck=', 'Content-Length': '0'} 2024-02-01T10:47:26.570 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a60aace6f592e747-0065bb76be-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:26.570 DEBUG:boto:b'vclfcn-61obj21000falseobj22024-02-01T10:46:46.380Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:26.571 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:47:26.571 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:47:26.571 DEBUG:boto:path=/vclfcn-61/?prefix=obj2 2024-02-01T10:47:26.571 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj2 2024-02-01T10:47:26.571 DEBUG:boto:Method: GET 2024-02-01T10:47:26.571 DEBUG:boto:Path: /vclfcn-61/?prefix=obj2 2024-02-01T10:47:26.571 DEBUG:boto:Data: 2024-02-01T10:47:26.571 DEBUG:boto:Headers: {} 2024-02-01T10:47:26.572 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:26.572 DEBUG:boto:Port: 8000 2024-02-01T10:47:26.572 DEBUG:boto:Params: {} 2024-02-01T10:47:26.572 DEBUG:boto:Token: None 2024-02-01T10:47:26.572 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:26 GMT /vclfcn-61/ 2024-02-01T10:47:26.572 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YCedi6sZ0ILB+0Vdi3FmEwuqNck= 2024-02-01T10:47:26.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:26 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YCedi6sZ0ILB+0Vdi3FmEwuqNck=', 'Content-Length': '0'} 2024-02-01T10:47:26.576 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000613c3d7e6ee2ec66-0065bb76be-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:26.577 DEBUG:boto:b'vclfcn-61obj21000falseobj22024-02-01T10:46:46.380Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:26.577 DEBUG:boto:path=/vclfcn-61/obj2 2024-02-01T10:47:26.577 DEBUG:boto:auth_path=/vclfcn-61/obj2 2024-02-01T10:47:26.577 DEBUG:boto:Method: HEAD 2024-02-01T10:47:26.577 DEBUG:boto:Path: /vclfcn-61/obj2 2024-02-01T10:47:26.577 DEBUG:boto:Data: 2024-02-01T10:47:26.578 DEBUG:boto:Headers: {} 2024-02-01T10:47:26.578 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:26.578 DEBUG:boto:Port: 8000 2024-02-01T10:47:26.578 DEBUG:boto:Params: {} 2024-02-01T10:47:26.578 DEBUG:boto:Token: None 2024-02-01T10:47:26.578 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:26 GMT /vclfcn-61/obj2 2024-02-01T10:47:26.578 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:hvl0ylz2/v862lrbfoHSuFsZsUE= 2024-02-01T10:47:26.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:26 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:hvl0ylz2/v862lrbfoHSuFsZsUE=', 'Content-Length': '0'} 2024-02-01T10:47:26.581 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000028cf3a3a56b32feb-0065bb76be-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:26.581 DEBUG:boto:path=/vclfcn-61/obj2 2024-02-01T10:47:26.581 DEBUG:boto:auth_path=/vclfcn-61/obj2 2024-02-01T10:47:26.582 DEBUG:boto:Method: HEAD 2024-02-01T10:47:26.582 DEBUG:boto:Path: /vclfcn-61/obj2 2024-02-01T10:47:26.582 DEBUG:boto:Data: 2024-02-01T10:47:26.582 DEBUG:boto:Headers: {} 2024-02-01T10:47:26.582 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:26.582 DEBUG:boto:Port: 8000 2024-02-01T10:47:26.582 DEBUG:boto:Params: {} 2024-02-01T10:47:26.582 DEBUG:boto:Token: None 2024-02-01T10:47:26.582 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:26 GMT /vclfcn-61/obj2 2024-02-01T10:47:26.582 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:hvl0ylz2/v862lrbfoHSuFsZsUE= 2024-02-01T10:47:26.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:26 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:hvl0ylz2/v862lrbfoHSuFsZsUE=', 'Content-Length': '0'} 2024-02-01T10:47:26.585 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e143532411052d9a-0065bb76be-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:26.585 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:47:26.586 DEBUG:boto:path=/vclfcn-61/obj2 2024-02-01T10:47:26.586 DEBUG:boto:auth_path=/vclfcn-61/obj2 2024-02-01T10:47:26.586 DEBUG:boto:Method: GET 2024-02-01T10:47:26.586 DEBUG:boto:Path: /vclfcn-61/obj2 2024-02-01T10:47:26.586 DEBUG:boto:Data: 2024-02-01T10:47:26.586 DEBUG:boto:Headers: {} 2024-02-01T10:47:26.586 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:26.586 DEBUG:boto:Port: 8000 2024-02-01T10:47:26.586 DEBUG:boto:Params: {} 2024-02-01T10:47:26.586 DEBUG:boto:Token: None 2024-02-01T10:47:26.587 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:26 GMT /vclfcn-61/obj2 2024-02-01T10:47:26.587 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ezVCT5MJiTH77qcl1W/pWXveYok= 2024-02-01T10:47:26.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:26 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ezVCT5MJiTH77qcl1W/pWXveYok=', 'Content-Length': '0'} 2024-02-01T10:47:26.589 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c9444ecda2154ed6-0065bb76be-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:26.632 DEBUG:boto:path=/vclfcn-61/obj2 2024-02-01T10:47:26.632 DEBUG:boto:auth_path=/vclfcn-61/obj2 2024-02-01T10:47:26.632 DEBUG:boto:Method: GET 2024-02-01T10:47:26.632 DEBUG:boto:Path: /vclfcn-61/obj2 2024-02-01T10:47:26.632 DEBUG:boto:Data: 2024-02-01T10:47:26.633 DEBUG:boto:Headers: {} 2024-02-01T10:47:26.633 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:26.633 DEBUG:boto:Port: 8000 2024-02-01T10:47:26.633 DEBUG:boto:Params: {} 2024-02-01T10:47:26.633 DEBUG:boto:Token: None 2024-02-01T10:47:26.633 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:26 GMT /vclfcn-61/obj2 2024-02-01T10:47:26.633 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ezVCT5MJiTH77qcl1W/pWXveYok= 2024-02-01T10:47:26.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:26 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ezVCT5MJiTH77qcl1W/pWXveYok=', 'Content-Length': '0'} 2024-02-01T10:47:26.636 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fe930885d3e0dfff-0065bb76be-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:26.636 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:47:26.637 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:47:26.637 DEBUG:boto:path=/vclfcn-61/?prefix=obj2 2024-02-01T10:47:26.637 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj2 2024-02-01T10:47:26.637 DEBUG:boto:Method: GET 2024-02-01T10:47:26.637 DEBUG:boto:Path: /vclfcn-61/?prefix=obj2 2024-02-01T10:47:26.637 DEBUG:boto:Data: 2024-02-01T10:47:26.637 DEBUG:boto:Headers: {} 2024-02-01T10:47:26.637 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:26.637 DEBUG:boto:Port: 8000 2024-02-01T10:47:26.637 DEBUG:boto:Params: {} 2024-02-01T10:47:26.638 DEBUG:boto:Token: None 2024-02-01T10:47:26.638 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:26 GMT /vclfcn-61/ 2024-02-01T10:47:26.638 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YCedi6sZ0ILB+0Vdi3FmEwuqNck= 2024-02-01T10:47:26.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:26 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YCedi6sZ0ILB+0Vdi3FmEwuqNck=', 'Content-Length': '0'} 2024-02-01T10:47:26.642 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7d0b0776ca2f2c7-0065bb76be-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:26.642 DEBUG:boto:b'vclfcn-61obj21000falseobj22024-02-01T10:46:46.380Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:26.643 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:47:26.643 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:47:26.643 DEBUG:boto:path=/vclfcn-61/?prefix=obj2 2024-02-01T10:47:26.643 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj2 2024-02-01T10:47:26.643 DEBUG:boto:Method: GET 2024-02-01T10:47:26.643 DEBUG:boto:Path: /vclfcn-61/?prefix=obj2 2024-02-01T10:47:26.643 DEBUG:boto:Data: 2024-02-01T10:47:26.643 DEBUG:boto:Headers: {} 2024-02-01T10:47:26.644 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:26.644 DEBUG:boto:Port: 8000 2024-02-01T10:47:26.644 DEBUG:boto:Params: {} 2024-02-01T10:47:26.644 DEBUG:boto:Token: None 2024-02-01T10:47:26.644 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:26 GMT /vclfcn-61/ 2024-02-01T10:47:26.644 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YCedi6sZ0ILB+0Vdi3FmEwuqNck= 2024-02-01T10:47:26.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:26 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YCedi6sZ0ILB+0Vdi3FmEwuqNck=', 'Content-Length': '0'} 2024-02-01T10:47:26.649 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dbbf865d0721dfc6-0065bb76be-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:26.649 DEBUG:boto:b'vclfcn-61obj21000falseobj22024-02-01T10:46:46.380Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:26.649 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:47:26.649 DEBUG:boto:path=/vclfcn-61/obj3 2024-02-01T10:47:26.649 DEBUG:boto:auth_path=/vclfcn-61/obj3 2024-02-01T10:47:26.649 DEBUG:boto:path=/vclfcn-61/obj3?versionId=null 2024-02-01T10:47:26.650 DEBUG:boto:auth_path=/vclfcn-61/obj3?versionId=null 2024-02-01T10:47:26.650 DEBUG:boto:Method: GET 2024-02-01T10:47:26.650 DEBUG:boto:Path: /vclfcn-61/obj3?versionId=null 2024-02-01T10:47:26.650 DEBUG:boto:Data: 2024-02-01T10:47:26.650 DEBUG:boto:Headers: {} 2024-02-01T10:47:26.650 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:26.650 DEBUG:boto:Port: 8000 2024-02-01T10:47:26.650 DEBUG:boto:Params: {} 2024-02-01T10:47:26.650 DEBUG:boto:Token: None 2024-02-01T10:47:26.650 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:26 GMT /vclfcn-61/obj3?versionId=null 2024-02-01T10:47:26.651 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:PQOGCo5qv4XRVtLlsGqof4omFSI= 2024-02-01T10:47:26.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:26 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:PQOGCo5qv4XRVtLlsGqof4omFSI=', 'Content-Length': '0'} 2024-02-01T10:47:26.653 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ef4068de9e4bf35c-0065bb76be-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:26.654 DEBUG:boto:path=/vclfcn-61/obj3 2024-02-01T10:47:26.654 DEBUG:boto:auth_path=/vclfcn-61/obj3 2024-02-01T10:47:26.654 DEBUG:boto:path=/vclfcn-61/obj3?versionId=null 2024-02-01T10:47:26.654 DEBUG:boto:auth_path=/vclfcn-61/obj3?versionId=null 2024-02-01T10:47:26.654 DEBUG:boto:Method: GET 2024-02-01T10:47:26.654 DEBUG:boto:Path: /vclfcn-61/obj3?versionId=null 2024-02-01T10:47:26.654 DEBUG:boto:Data: 2024-02-01T10:47:26.654 DEBUG:boto:Headers: {} 2024-02-01T10:47:26.654 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:26.655 DEBUG:boto:Port: 8000 2024-02-01T10:47:26.655 DEBUG:boto:Params: {} 2024-02-01T10:47:26.655 DEBUG:boto:Token: None 2024-02-01T10:47:26.655 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:26 GMT /vclfcn-61/obj3?versionId=null 2024-02-01T10:47:26.655 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:PQOGCo5qv4XRVtLlsGqof4omFSI= 2024-02-01T10:47:26.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:26 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:PQOGCo5qv4XRVtLlsGqof4omFSI=', 'Content-Length': '0'} 2024-02-01T10:47:26.658 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006b34ac38f15ca9af-0065bb76be-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:26.658 DEBUG:boto:path=/vclfcn-61/obj3 2024-02-01T10:47:26.658 DEBUG:boto:auth_path=/vclfcn-61/obj3 2024-02-01T10:47:26.658 DEBUG:boto:path=/vclfcn-61/obj3?versionId=null 2024-02-01T10:47:26.658 DEBUG:boto:auth_path=/vclfcn-61/obj3?versionId=null 2024-02-01T10:47:26.659 DEBUG:boto:Method: HEAD 2024-02-01T10:47:26.659 DEBUG:boto:Path: /vclfcn-61/obj3?versionId=null 2024-02-01T10:47:26.659 DEBUG:boto:Data: 2024-02-01T10:47:26.659 DEBUG:boto:Headers: {} 2024-02-01T10:47:26.659 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:26.659 DEBUG:boto:Port: 8000 2024-02-01T10:47:26.659 DEBUG:boto:Params: {} 2024-02-01T10:47:26.659 DEBUG:boto:Token: None 2024-02-01T10:47:26.659 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:26 GMT /vclfcn-61/obj3?versionId=null 2024-02-01T10:47:26.659 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:2mXBueYeNfVSYquR1769AIq3D2g= 2024-02-01T10:47:26.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:26 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2mXBueYeNfVSYquR1769AIq3D2g=', 'Content-Length': '0'} 2024-02-01T10:47:26.662 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009b27e75844ed93c6-0065bb76be-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:26.663 DEBUG:boto:path=/vclfcn-61/obj3 2024-02-01T10:47:26.663 DEBUG:boto:auth_path=/vclfcn-61/obj3 2024-02-01T10:47:26.663 DEBUG:boto:path=/vclfcn-61/obj3?versionId=null 2024-02-01T10:47:26.663 DEBUG:boto:auth_path=/vclfcn-61/obj3?versionId=null 2024-02-01T10:47:26.663 DEBUG:boto:Method: HEAD 2024-02-01T10:47:26.663 DEBUG:boto:Path: /vclfcn-61/obj3?versionId=null 2024-02-01T10:47:26.663 DEBUG:boto:Data: 2024-02-01T10:47:26.663 DEBUG:boto:Headers: {} 2024-02-01T10:47:26.663 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:26.663 DEBUG:boto:Port: 8000 2024-02-01T10:47:26.663 DEBUG:boto:Params: {} 2024-02-01T10:47:26.664 DEBUG:boto:Token: None 2024-02-01T10:47:26.664 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:26 GMT /vclfcn-61/obj3?versionId=null 2024-02-01T10:47:26.664 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:2mXBueYeNfVSYquR1769AIq3D2g= 2024-02-01T10:47:26.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:26 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2mXBueYeNfVSYquR1769AIq3D2g=', 'Content-Length': '0'} 2024-02-01T10:47:26.668 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000065630d2efe628cd3-0065bb76be-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:26.668 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:47:26.668 DEBUG:boto:path=/vclfcn-61/obj3 2024-02-01T10:47:26.669 DEBUG:boto:auth_path=/vclfcn-61/obj3 2024-02-01T10:47:26.669 DEBUG:boto:path=/vclfcn-61/obj3?versionId=null 2024-02-01T10:47:26.669 DEBUG:boto:auth_path=/vclfcn-61/obj3?versionId=null 2024-02-01T10:47:26.669 DEBUG:boto:Method: GET 2024-02-01T10:47:26.669 DEBUG:boto:Path: /vclfcn-61/obj3?versionId=null 2024-02-01T10:47:26.669 DEBUG:boto:Data: 2024-02-01T10:47:26.669 DEBUG:boto:Headers: {} 2024-02-01T10:47:26.669 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:26.669 DEBUG:boto:Port: 8000 2024-02-01T10:47:26.669 DEBUG:boto:Params: {} 2024-02-01T10:47:26.669 DEBUG:boto:Token: None 2024-02-01T10:47:26.670 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:26 GMT /vclfcn-61/obj3?versionId=null 2024-02-01T10:47:26.670 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:PQOGCo5qv4XRVtLlsGqof4omFSI= 2024-02-01T10:47:26.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:26 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:PQOGCo5qv4XRVtLlsGqof4omFSI=', 'Content-Length': '0'} 2024-02-01T10:47:26.672 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009ff026561f0bc81a-0065bb76be-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:26.716 DEBUG:boto:path=/vclfcn-61/obj3 2024-02-01T10:47:26.716 DEBUG:boto:auth_path=/vclfcn-61/obj3 2024-02-01T10:47:26.716 DEBUG:boto:path=/vclfcn-61/obj3?versionId=null 2024-02-01T10:47:26.717 DEBUG:boto:auth_path=/vclfcn-61/obj3?versionId=null 2024-02-01T10:47:26.717 DEBUG:boto:Method: GET 2024-02-01T10:47:26.717 DEBUG:boto:Path: /vclfcn-61/obj3?versionId=null 2024-02-01T10:47:26.717 DEBUG:boto:Data: 2024-02-01T10:47:26.717 DEBUG:boto:Headers: {} 2024-02-01T10:47:26.717 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:26.717 DEBUG:boto:Port: 8000 2024-02-01T10:47:26.717 DEBUG:boto:Params: {} 2024-02-01T10:47:26.717 DEBUG:boto:Token: None 2024-02-01T10:47:26.718 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:26 GMT /vclfcn-61/obj3?versionId=null 2024-02-01T10:47:26.718 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:PQOGCo5qv4XRVtLlsGqof4omFSI= 2024-02-01T10:47:26.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:26 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:PQOGCo5qv4XRVtLlsGqof4omFSI=', 'Content-Length': '0'} 2024-02-01T10:47:26.721 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000769b9377657088f2-0065bb76be-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:26.721 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:47:26.721 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:47:26.721 DEBUG:boto:path=/vclfcn-61/?prefix=obj3 2024-02-01T10:47:26.721 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj3 2024-02-01T10:47:26.721 DEBUG:boto:Method: GET 2024-02-01T10:47:26.722 DEBUG:boto:Path: /vclfcn-61/?prefix=obj3 2024-02-01T10:47:26.722 DEBUG:boto:Data: 2024-02-01T10:47:26.722 DEBUG:boto:Headers: {} 2024-02-01T10:47:26.722 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:26.722 DEBUG:boto:Port: 8000 2024-02-01T10:47:26.722 DEBUG:boto:Params: {} 2024-02-01T10:47:26.722 DEBUG:boto:Token: None 2024-02-01T10:47:26.722 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:26 GMT /vclfcn-61/ 2024-02-01T10:47:26.722 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YCedi6sZ0ILB+0Vdi3FmEwuqNck= 2024-02-01T10:47:26.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:26 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YCedi6sZ0ILB+0Vdi3FmEwuqNck=', 'Content-Length': '0'} 2024-02-01T10:47:26.726 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000811b0900a76e2b9d-0065bb76be-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:26.727 DEBUG:boto:b'vclfcn-61obj31000falseobj32024-02-01T10:46:46.433Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:26.727 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:47:26.727 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:47:26.727 DEBUG:boto:path=/vclfcn-61/?prefix=obj3 2024-02-01T10:47:26.727 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj3 2024-02-01T10:47:26.727 DEBUG:boto:Method: GET 2024-02-01T10:47:26.728 DEBUG:boto:Path: /vclfcn-61/?prefix=obj3 2024-02-01T10:47:26.728 DEBUG:boto:Data: 2024-02-01T10:47:26.728 DEBUG:boto:Headers: {} 2024-02-01T10:47:26.728 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:26.728 DEBUG:boto:Port: 8000 2024-02-01T10:47:26.728 DEBUG:boto:Params: {} 2024-02-01T10:47:26.728 DEBUG:boto:Token: None 2024-02-01T10:47:26.728 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:26 GMT /vclfcn-61/ 2024-02-01T10:47:26.728 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YCedi6sZ0ILB+0Vdi3FmEwuqNck= 2024-02-01T10:47:26.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:26 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YCedi6sZ0ILB+0Vdi3FmEwuqNck=', 'Content-Length': '0'} 2024-02-01T10:47:26.734 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c96e46ef28302d30-0065bb76be-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:26.734 DEBUG:boto:b'vclfcn-61obj31000falseobj32024-02-01T10:46:46.433Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:26.734 DEBUG:boto:path=/vclfcn-61/obj3 2024-02-01T10:47:26.735 DEBUG:boto:auth_path=/vclfcn-61/obj3 2024-02-01T10:47:26.735 DEBUG:boto:Method: HEAD 2024-02-01T10:47:26.735 DEBUG:boto:Path: /vclfcn-61/obj3 2024-02-01T10:47:26.735 DEBUG:boto:Data: 2024-02-01T10:47:26.735 DEBUG:boto:Headers: {} 2024-02-01T10:47:26.735 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:26.735 DEBUG:boto:Port: 8000 2024-02-01T10:47:26.735 DEBUG:boto:Params: {} 2024-02-01T10:47:26.735 DEBUG:boto:Token: None 2024-02-01T10:47:26.735 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:26 GMT /vclfcn-61/obj3 2024-02-01T10:47:26.736 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5u6XVIZ4oJCzz2I4BianLdn6J38= 2024-02-01T10:47:26.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:26 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5u6XVIZ4oJCzz2I4BianLdn6J38=', 'Content-Length': '0'} 2024-02-01T10:47:26.738 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000da8a9a96192f23a9-0065bb76be-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:26.739 DEBUG:boto:path=/vclfcn-61/obj3 2024-02-01T10:47:26.739 DEBUG:boto:auth_path=/vclfcn-61/obj3 2024-02-01T10:47:26.739 DEBUG:boto:Method: HEAD 2024-02-01T10:47:26.739 DEBUG:boto:Path: /vclfcn-61/obj3 2024-02-01T10:47:26.739 DEBUG:boto:Data: 2024-02-01T10:47:26.739 DEBUG:boto:Headers: {} 2024-02-01T10:47:26.739 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:26.739 DEBUG:boto:Port: 8000 2024-02-01T10:47:26.739 DEBUG:boto:Params: {} 2024-02-01T10:47:26.739 DEBUG:boto:Token: None 2024-02-01T10:47:26.740 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:26 GMT /vclfcn-61/obj3 2024-02-01T10:47:26.740 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5u6XVIZ4oJCzz2I4BianLdn6J38= 2024-02-01T10:47:26.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:26 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5u6XVIZ4oJCzz2I4BianLdn6J38=', 'Content-Length': '0'} 2024-02-01T10:47:26.743 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000002837a7701ac8fa0-0065bb76be-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:26.743 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:47:26.743 DEBUG:boto:path=/vclfcn-61/obj3 2024-02-01T10:47:26.743 DEBUG:boto:auth_path=/vclfcn-61/obj3 2024-02-01T10:47:26.744 DEBUG:boto:Method: GET 2024-02-01T10:47:26.744 DEBUG:boto:Path: /vclfcn-61/obj3 2024-02-01T10:47:26.744 DEBUG:boto:Data: 2024-02-01T10:47:26.744 DEBUG:boto:Headers: {} 2024-02-01T10:47:26.744 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:26.744 DEBUG:boto:Port: 8000 2024-02-01T10:47:26.744 DEBUG:boto:Params: {} 2024-02-01T10:47:26.744 DEBUG:boto:Token: None 2024-02-01T10:47:26.744 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:26 GMT /vclfcn-61/obj3 2024-02-01T10:47:26.745 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:z7r/77TNVATJH30RA3ZsrtSFcS8= 2024-02-01T10:47:26.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:26 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:z7r/77TNVATJH30RA3ZsrtSFcS8=', 'Content-Length': '0'} 2024-02-01T10:47:26.747 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c27dd4101be6d49c-0065bb76be-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:26.788 DEBUG:boto:path=/vclfcn-61/obj3 2024-02-01T10:47:26.788 DEBUG:boto:auth_path=/vclfcn-61/obj3 2024-02-01T10:47:26.788 DEBUG:boto:Method: GET 2024-02-01T10:47:26.788 DEBUG:boto:Path: /vclfcn-61/obj3 2024-02-01T10:47:26.789 DEBUG:boto:Data: 2024-02-01T10:47:26.789 DEBUG:boto:Headers: {} 2024-02-01T10:47:26.789 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:26.789 DEBUG:boto:Port: 8000 2024-02-01T10:47:26.789 DEBUG:boto:Params: {} 2024-02-01T10:47:26.789 DEBUG:boto:Token: None 2024-02-01T10:47:26.789 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:26 GMT /vclfcn-61/obj3 2024-02-01T10:47:26.789 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:z7r/77TNVATJH30RA3ZsrtSFcS8= 2024-02-01T10:47:26.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:26 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:z7r/77TNVATJH30RA3ZsrtSFcS8=', 'Content-Length': '0'} 2024-02-01T10:47:26.792 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f0851755216d066a-0065bb76be-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:26.793 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:47:26.793 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:47:26.793 DEBUG:boto:path=/vclfcn-61/?prefix=obj3 2024-02-01T10:47:26.793 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj3 2024-02-01T10:47:26.793 DEBUG:boto:Method: GET 2024-02-01T10:47:26.793 DEBUG:boto:Path: /vclfcn-61/?prefix=obj3 2024-02-01T10:47:26.793 DEBUG:boto:Data: 2024-02-01T10:47:26.793 DEBUG:boto:Headers: {} 2024-02-01T10:47:26.793 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:26.794 DEBUG:boto:Port: 8000 2024-02-01T10:47:26.794 DEBUG:boto:Params: {} 2024-02-01T10:47:26.794 DEBUG:boto:Token: None 2024-02-01T10:47:26.794 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:26 GMT /vclfcn-61/ 2024-02-01T10:47:26.794 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YCedi6sZ0ILB+0Vdi3FmEwuqNck= 2024-02-01T10:47:26.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:26 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YCedi6sZ0ILB+0Vdi3FmEwuqNck=', 'Content-Length': '0'} 2024-02-01T10:47:26.799 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f2ac7490658a9c7-0065bb76be-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:26.840 DEBUG:boto:b'vclfcn-61obj31000falseobj32024-02-01T10:46:46.433Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:26.840 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:47:26.841 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:47:26.841 DEBUG:boto:path=/vclfcn-61/?prefix=obj3 2024-02-01T10:47:26.841 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj3 2024-02-01T10:47:26.841 DEBUG:boto:Method: GET 2024-02-01T10:47:26.841 DEBUG:boto:Path: /vclfcn-61/?prefix=obj3 2024-02-01T10:47:26.841 DEBUG:boto:Data: 2024-02-01T10:47:26.841 DEBUG:boto:Headers: {} 2024-02-01T10:47:26.841 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:26.841 DEBUG:boto:Port: 8000 2024-02-01T10:47:26.841 DEBUG:boto:Params: {} 2024-02-01T10:47:26.842 DEBUG:boto:Token: None 2024-02-01T10:47:26.842 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:26 GMT /vclfcn-61/ 2024-02-01T10:47:26.842 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YCedi6sZ0ILB+0Vdi3FmEwuqNck= 2024-02-01T10:47:26.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:26 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YCedi6sZ0ILB+0Vdi3FmEwuqNck=', 'Content-Length': '0'} 2024-02-01T10:47:26.847 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000928bec81f67b4570-0065bb76be-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:26.847 DEBUG:boto:b'vclfcn-61obj31000falseobj32024-02-01T10:46:46.433Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:26.847 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:47:26.847 DEBUG:boto:path=/vclfcn-61/obj4 2024-02-01T10:47:26.847 DEBUG:boto:auth_path=/vclfcn-61/obj4 2024-02-01T10:47:26.847 DEBUG:boto:path=/vclfcn-61/obj4?versionId=null 2024-02-01T10:47:26.848 DEBUG:boto:auth_path=/vclfcn-61/obj4?versionId=null 2024-02-01T10:47:26.848 DEBUG:boto:Method: GET 2024-02-01T10:47:26.848 DEBUG:boto:Path: /vclfcn-61/obj4?versionId=null 2024-02-01T10:47:26.848 DEBUG:boto:Data: 2024-02-01T10:47:26.848 DEBUG:boto:Headers: {} 2024-02-01T10:47:26.848 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:26.848 DEBUG:boto:Port: 8000 2024-02-01T10:47:26.848 DEBUG:boto:Params: {} 2024-02-01T10:47:26.848 DEBUG:boto:Token: None 2024-02-01T10:47:26.849 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:26 GMT /vclfcn-61/obj4?versionId=null 2024-02-01T10:47:26.849 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:38X9/A9ASsPkbVS7BnzA8ABoxZk= 2024-02-01T10:47:26.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:26 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:38X9/A9ASsPkbVS7BnzA8ABoxZk=', 'Content-Length': '0'} 2024-02-01T10:47:26.851 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e1ad8528874ff0d4-0065bb76be-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:26.852 DEBUG:boto:path=/vclfcn-61/obj4 2024-02-01T10:47:26.852 DEBUG:boto:auth_path=/vclfcn-61/obj4 2024-02-01T10:47:26.852 DEBUG:boto:path=/vclfcn-61/obj4?versionId=null 2024-02-01T10:47:26.852 DEBUG:boto:auth_path=/vclfcn-61/obj4?versionId=null 2024-02-01T10:47:26.852 DEBUG:boto:Method: GET 2024-02-01T10:47:26.852 DEBUG:boto:Path: /vclfcn-61/obj4?versionId=null 2024-02-01T10:47:26.852 DEBUG:boto:Data: 2024-02-01T10:47:26.853 DEBUG:boto:Headers: {} 2024-02-01T10:47:26.853 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:26.853 DEBUG:boto:Port: 8000 2024-02-01T10:47:26.853 DEBUG:boto:Params: {} 2024-02-01T10:47:26.853 DEBUG:boto:Token: None 2024-02-01T10:47:26.853 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:26 GMT /vclfcn-61/obj4?versionId=null 2024-02-01T10:47:26.853 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:38X9/A9ASsPkbVS7BnzA8ABoxZk= 2024-02-01T10:47:26.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:26 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:38X9/A9ASsPkbVS7BnzA8ABoxZk=', 'Content-Length': '0'} 2024-02-01T10:47:26.856 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c174951d1404484e-0065bb76be-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:26.900 DEBUG:boto:path=/vclfcn-61/obj4 2024-02-01T10:47:26.900 DEBUG:boto:auth_path=/vclfcn-61/obj4 2024-02-01T10:47:26.900 DEBUG:boto:path=/vclfcn-61/obj4?versionId=null 2024-02-01T10:47:26.901 DEBUG:boto:auth_path=/vclfcn-61/obj4?versionId=null 2024-02-01T10:47:26.901 DEBUG:boto:Method: HEAD 2024-02-01T10:47:26.901 DEBUG:boto:Path: /vclfcn-61/obj4?versionId=null 2024-02-01T10:47:26.901 DEBUG:boto:Data: 2024-02-01T10:47:26.901 DEBUG:boto:Headers: {} 2024-02-01T10:47:26.901 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:26.901 DEBUG:boto:Port: 8000 2024-02-01T10:47:26.901 DEBUG:boto:Params: {} 2024-02-01T10:47:26.901 DEBUG:boto:Token: None 2024-02-01T10:47:26.902 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:26 GMT /vclfcn-61/obj4?versionId=null 2024-02-01T10:47:26.902 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cDKZEoDnUH5TDLDeqyjjM+ovyjg= 2024-02-01T10:47:26.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:26 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cDKZEoDnUH5TDLDeqyjjM+ovyjg=', 'Content-Length': '0'} 2024-02-01T10:47:26.905 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004fd668de4ba5844e-0065bb76be-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:26.905 DEBUG:boto:path=/vclfcn-61/obj4 2024-02-01T10:47:26.905 DEBUG:boto:auth_path=/vclfcn-61/obj4 2024-02-01T10:47:26.905 DEBUG:boto:path=/vclfcn-61/obj4?versionId=null 2024-02-01T10:47:26.905 DEBUG:boto:auth_path=/vclfcn-61/obj4?versionId=null 2024-02-01T10:47:26.905 DEBUG:boto:Method: HEAD 2024-02-01T10:47:26.905 DEBUG:boto:Path: /vclfcn-61/obj4?versionId=null 2024-02-01T10:47:26.905 DEBUG:boto:Data: 2024-02-01T10:47:26.906 DEBUG:boto:Headers: {} 2024-02-01T10:47:26.906 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:26.906 DEBUG:boto:Port: 8000 2024-02-01T10:47:26.906 DEBUG:boto:Params: {} 2024-02-01T10:47:26.906 DEBUG:boto:Token: None 2024-02-01T10:47:26.906 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:26 GMT /vclfcn-61/obj4?versionId=null 2024-02-01T10:47:26.906 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cDKZEoDnUH5TDLDeqyjjM+ovyjg= 2024-02-01T10:47:26.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:26 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cDKZEoDnUH5TDLDeqyjjM+ovyjg=', 'Content-Length': '0'} 2024-02-01T10:47:26.909 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e97327c168a54407-0065bb76be-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:26.909 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:47:26.909 DEBUG:boto:path=/vclfcn-61/obj4 2024-02-01T10:47:26.910 DEBUG:boto:auth_path=/vclfcn-61/obj4 2024-02-01T10:47:26.910 DEBUG:boto:path=/vclfcn-61/obj4?versionId=null 2024-02-01T10:47:26.910 DEBUG:boto:auth_path=/vclfcn-61/obj4?versionId=null 2024-02-01T10:47:26.910 DEBUG:boto:Method: GET 2024-02-01T10:47:26.910 DEBUG:boto:Path: /vclfcn-61/obj4?versionId=null 2024-02-01T10:47:26.910 DEBUG:boto:Data: 2024-02-01T10:47:26.910 DEBUG:boto:Headers: {} 2024-02-01T10:47:26.910 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:26.910 DEBUG:boto:Port: 8000 2024-02-01T10:47:26.910 DEBUG:boto:Params: {} 2024-02-01T10:47:26.911 DEBUG:boto:Token: None 2024-02-01T10:47:26.911 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:26 GMT /vclfcn-61/obj4?versionId=null 2024-02-01T10:47:26.911 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:38X9/A9ASsPkbVS7BnzA8ABoxZk= 2024-02-01T10:47:26.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:26 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:38X9/A9ASsPkbVS7BnzA8ABoxZk=', 'Content-Length': '0'} 2024-02-01T10:47:26.913 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000031ed1987429a6e2d-0065bb76be-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:26.956 DEBUG:boto:path=/vclfcn-61/obj4 2024-02-01T10:47:26.956 DEBUG:boto:auth_path=/vclfcn-61/obj4 2024-02-01T10:47:26.956 DEBUG:boto:path=/vclfcn-61/obj4?versionId=null 2024-02-01T10:47:26.956 DEBUG:boto:auth_path=/vclfcn-61/obj4?versionId=null 2024-02-01T10:47:26.957 DEBUG:boto:Method: GET 2024-02-01T10:47:26.957 DEBUG:boto:Path: /vclfcn-61/obj4?versionId=null 2024-02-01T10:47:26.957 DEBUG:boto:Data: 2024-02-01T10:47:26.957 DEBUG:boto:Headers: {} 2024-02-01T10:47:26.957 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:26.957 DEBUG:boto:Port: 8000 2024-02-01T10:47:26.957 DEBUG:boto:Params: {} 2024-02-01T10:47:26.957 DEBUG:boto:Token: None 2024-02-01T10:47:26.957 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:26 GMT /vclfcn-61/obj4?versionId=null 2024-02-01T10:47:26.958 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:38X9/A9ASsPkbVS7BnzA8ABoxZk= 2024-02-01T10:47:26.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:26 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:38X9/A9ASsPkbVS7BnzA8ABoxZk=', 'Content-Length': '0'} 2024-02-01T10:47:26.961 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006f354f816d19fa3f-0065bb76be-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:26.961 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:47:26.961 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:47:26.961 DEBUG:boto:path=/vclfcn-61/?prefix=obj4 2024-02-01T10:47:26.961 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj4 2024-02-01T10:47:26.962 DEBUG:boto:Method: GET 2024-02-01T10:47:26.962 DEBUG:boto:Path: /vclfcn-61/?prefix=obj4 2024-02-01T10:47:26.962 DEBUG:boto:Data: 2024-02-01T10:47:26.962 DEBUG:boto:Headers: {} 2024-02-01T10:47:26.962 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:26.962 DEBUG:boto:Port: 8000 2024-02-01T10:47:26.962 DEBUG:boto:Params: {} 2024-02-01T10:47:26.962 DEBUG:boto:Token: None 2024-02-01T10:47:26.962 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:26 GMT /vclfcn-61/ 2024-02-01T10:47:26.962 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YCedi6sZ0ILB+0Vdi3FmEwuqNck= 2024-02-01T10:47:26.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:26 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YCedi6sZ0ILB+0Vdi3FmEwuqNck=', 'Content-Length': '0'} 2024-02-01T10:47:26.967 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083bd4ad97207420b-0065bb76be-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:27.008 DEBUG:boto:b'vclfcn-61obj41000falseobj42024-02-01T10:46:46.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:27.008 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:47:27.009 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:47:27.009 DEBUG:boto:path=/vclfcn-61/?prefix=obj4 2024-02-01T10:47:27.009 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj4 2024-02-01T10:47:27.009 DEBUG:boto:Method: GET 2024-02-01T10:47:27.009 DEBUG:boto:Path: /vclfcn-61/?prefix=obj4 2024-02-01T10:47:27.009 DEBUG:boto:Data: 2024-02-01T10:47:27.009 DEBUG:boto:Headers: {} 2024-02-01T10:47:27.009 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:27.009 DEBUG:boto:Port: 8000 2024-02-01T10:47:27.009 DEBUG:boto:Params: {} 2024-02-01T10:47:27.010 DEBUG:boto:Token: None 2024-02-01T10:47:27.010 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:27 GMT /vclfcn-61/ 2024-02-01T10:47:27.010 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ww39+/07l7+WNAHvS5qWCYZMIDI= 2024-02-01T10:47:27.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ww39+/07l7+WNAHvS5qWCYZMIDI=', 'Content-Length': '0'} 2024-02-01T10:47:27.015 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e16d0d8df78d0d5-0065bb76bf-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:27.015 DEBUG:boto:b'vclfcn-61obj41000falseobj42024-02-01T10:46:46.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:27.015 DEBUG:boto:path=/vclfcn-61/obj4 2024-02-01T10:47:27.015 DEBUG:boto:auth_path=/vclfcn-61/obj4 2024-02-01T10:47:27.015 DEBUG:boto:Method: HEAD 2024-02-01T10:47:27.016 DEBUG:boto:Path: /vclfcn-61/obj4 2024-02-01T10:47:27.016 DEBUG:boto:Data: 2024-02-01T10:47:27.016 DEBUG:boto:Headers: {} 2024-02-01T10:47:27.016 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:27.016 DEBUG:boto:Port: 8000 2024-02-01T10:47:27.016 DEBUG:boto:Params: {} 2024-02-01T10:47:27.016 DEBUG:boto:Token: None 2024-02-01T10:47:27.016 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:27 GMT /vclfcn-61/obj4 2024-02-01T10:47:27.016 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:LBZs55omZ1Otp1OX9ygTzAYZwq0= 2024-02-01T10:47:27.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:LBZs55omZ1Otp1OX9ygTzAYZwq0=', 'Content-Length': '0'} 2024-02-01T10:47:27.019 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000004c16477d6507791-0065bb76bf-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:27.019 DEBUG:boto:path=/vclfcn-61/obj4 2024-02-01T10:47:27.019 DEBUG:boto:auth_path=/vclfcn-61/obj4 2024-02-01T10:47:27.020 DEBUG:boto:Method: HEAD 2024-02-01T10:47:27.020 DEBUG:boto:Path: /vclfcn-61/obj4 2024-02-01T10:47:27.020 DEBUG:boto:Data: 2024-02-01T10:47:27.020 DEBUG:boto:Headers: {} 2024-02-01T10:47:27.020 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:27.020 DEBUG:boto:Port: 8000 2024-02-01T10:47:27.020 DEBUG:boto:Params: {} 2024-02-01T10:47:27.020 DEBUG:boto:Token: None 2024-02-01T10:47:27.020 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:27 GMT /vclfcn-61/obj4 2024-02-01T10:47:27.021 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:LBZs55omZ1Otp1OX9ygTzAYZwq0= 2024-02-01T10:47:27.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:LBZs55omZ1Otp1OX9ygTzAYZwq0=', 'Content-Length': '0'} 2024-02-01T10:47:27.023 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002acc1244cc57047e-0065bb76bf-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:27.024 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:47:27.024 DEBUG:boto:path=/vclfcn-61/obj4 2024-02-01T10:47:27.024 DEBUG:boto:auth_path=/vclfcn-61/obj4 2024-02-01T10:47:27.024 DEBUG:boto:Method: GET 2024-02-01T10:47:27.024 DEBUG:boto:Path: /vclfcn-61/obj4 2024-02-01T10:47:27.024 DEBUG:boto:Data: 2024-02-01T10:47:27.024 DEBUG:boto:Headers: {} 2024-02-01T10:47:27.024 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:27.025 DEBUG:boto:Port: 8000 2024-02-01T10:47:27.025 DEBUG:boto:Params: {} 2024-02-01T10:47:27.025 DEBUG:boto:Token: None 2024-02-01T10:47:27.025 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:27 GMT /vclfcn-61/obj4 2024-02-01T10:47:27.025 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZDycMCht0nbZrtFjdxKubLJ6vzo= 2024-02-01T10:47:27.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZDycMCht0nbZrtFjdxKubLJ6vzo=', 'Content-Length': '0'} 2024-02-01T10:47:27.028 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000031683a2e9de7949-0065bb76bf-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:27.068 DEBUG:boto:path=/vclfcn-61/obj4 2024-02-01T10:47:27.068 DEBUG:boto:auth_path=/vclfcn-61/obj4 2024-02-01T10:47:27.068 DEBUG:boto:Method: GET 2024-02-01T10:47:27.068 DEBUG:boto:Path: /vclfcn-61/obj4 2024-02-01T10:47:27.068 DEBUG:boto:Data: 2024-02-01T10:47:27.069 DEBUG:boto:Headers: {} 2024-02-01T10:47:27.069 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:27.069 DEBUG:boto:Port: 8000 2024-02-01T10:47:27.069 DEBUG:boto:Params: {} 2024-02-01T10:47:27.069 DEBUG:boto:Token: None 2024-02-01T10:47:27.069 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:27 GMT /vclfcn-61/obj4 2024-02-01T10:47:27.069 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZDycMCht0nbZrtFjdxKubLJ6vzo= 2024-02-01T10:47:27.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZDycMCht0nbZrtFjdxKubLJ6vzo=', 'Content-Length': '0'} 2024-02-01T10:47:27.072 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b04de238d1495182-0065bb76bf-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:27.072 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:47:27.072 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:47:27.073 DEBUG:boto:path=/vclfcn-61/?prefix=obj4 2024-02-01T10:47:27.073 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj4 2024-02-01T10:47:27.073 DEBUG:boto:Method: GET 2024-02-01T10:47:27.073 DEBUG:boto:Path: /vclfcn-61/?prefix=obj4 2024-02-01T10:47:27.073 DEBUG:boto:Data: 2024-02-01T10:47:27.073 DEBUG:boto:Headers: {} 2024-02-01T10:47:27.073 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:27.073 DEBUG:boto:Port: 8000 2024-02-01T10:47:27.073 DEBUG:boto:Params: {} 2024-02-01T10:47:27.073 DEBUG:boto:Token: None 2024-02-01T10:47:27.073 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:27 GMT /vclfcn-61/ 2024-02-01T10:47:27.074 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ww39+/07l7+WNAHvS5qWCYZMIDI= 2024-02-01T10:47:27.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ww39+/07l7+WNAHvS5qWCYZMIDI=', 'Content-Length': '0'} 2024-02-01T10:47:27.080 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000249efe23f2028b44-0065bb76bf-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:27.080 DEBUG:boto:b'vclfcn-61obj41000falseobj42024-02-01T10:46:46.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:27.080 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:47:27.081 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:47:27.081 DEBUG:boto:path=/vclfcn-61/?prefix=obj4 2024-02-01T10:47:27.081 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj4 2024-02-01T10:47:27.081 DEBUG:boto:Method: GET 2024-02-01T10:47:27.081 DEBUG:boto:Path: /vclfcn-61/?prefix=obj4 2024-02-01T10:47:27.081 DEBUG:boto:Data: 2024-02-01T10:47:27.081 DEBUG:boto:Headers: {} 2024-02-01T10:47:27.081 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:27.081 DEBUG:boto:Port: 8000 2024-02-01T10:47:27.081 DEBUG:boto:Params: {} 2024-02-01T10:47:27.082 DEBUG:boto:Token: None 2024-02-01T10:47:27.082 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:27 GMT /vclfcn-61/ 2024-02-01T10:47:27.082 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ww39+/07l7+WNAHvS5qWCYZMIDI= 2024-02-01T10:47:27.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ww39+/07l7+WNAHvS5qWCYZMIDI=', 'Content-Length': '0'} 2024-02-01T10:47:27.086 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff6a23d62508f7a1-0065bb76bf-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:27.128 DEBUG:boto:b'vclfcn-61obj41000falseobj42024-02-01T10:46:46.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:27.128 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-61 zones={test-zone2, test-zone1} 2024-02-01T10:47:27.128 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-61 zones={test-zone3, test-zone1} 2024-02-01T10:47:27.129 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:47:27.129 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:47:27.129 DEBUG:boto:Method: HEAD 2024-02-01T10:47:27.129 DEBUG:boto:Path: /vclfcn-61/ 2024-02-01T10:47:27.129 DEBUG:boto:Data: 2024-02-01T10:47:27.129 DEBUG:boto:Headers: {} 2024-02-01T10:47:27.129 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:27.129 DEBUG:boto:Port: 8001 2024-02-01T10:47:27.129 DEBUG:boto:Params: {} 2024-02-01T10:47:27.129 DEBUG:boto:Token: None 2024-02-01T10:47:27.130 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:27 GMT /vclfcn-61/ 2024-02-01T10:47:27.130 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ahvF92v6VGxuUE3x+uM3vAKNpTY= 2024-02-01T10:47:27.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ahvF92v6VGxuUE3x+uM3vAKNpTY=', 'Content-Length': '0'} 2024-02-01T10:47:27.135 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008ab67e2efb54e069-0065bb76bf-5762-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:27.135 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:47:27.135 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:47:27.135 DEBUG:boto:Method: HEAD 2024-02-01T10:47:27.135 DEBUG:boto:Path: /vclfcn-61/ 2024-02-01T10:47:27.135 DEBUG:boto:Data: 2024-02-01T10:47:27.135 DEBUG:boto:Headers: {} 2024-02-01T10:47:27.135 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:27.135 DEBUG:boto:Port: 8000 2024-02-01T10:47:27.135 DEBUG:boto:Params: {} 2024-02-01T10:47:27.136 DEBUG:boto:Token: None 2024-02-01T10:47:27.136 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:27 GMT /vclfcn-61/ 2024-02-01T10:47:27.136 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ahvF92v6VGxuUE3x+uM3vAKNpTY= 2024-02-01T10:47:27.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ahvF92v6VGxuUE3x+uM3vAKNpTY=', 'Content-Length': '0'} 2024-02-01T10:47:27.140 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000012621ec4861bc6ce-0065bb76bf-5765-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:27.140 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:47:27.140 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:47:27.140 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:47:27.140 DEBUG:boto:path=/vclfcn-61/?versions&max-keys=999 2024-02-01T10:47:27.140 DEBUG:boto:auth_path=/vclfcn-61/?versions&max-keys=999 2024-02-01T10:47:27.141 DEBUG:boto:Method: GET 2024-02-01T10:47:27.141 DEBUG:boto:Path: /vclfcn-61/?versions&max-keys=999 2024-02-01T10:47:27.141 DEBUG:boto:Data: 2024-02-01T10:47:27.141 DEBUG:boto:Headers: {} 2024-02-01T10:47:27.141 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:27.141 DEBUG:boto:Port: 8001 2024-02-01T10:47:27.141 DEBUG:boto:Params: {} 2024-02-01T10:47:27.141 DEBUG:boto:Token: None 2024-02-01T10:47:27.141 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:27 GMT /vclfcn-61/?versions 2024-02-01T10:47:27.141 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:rCokFydKZEgUqYK3CuEpnHDKHeU= 2024-02-01T10:47:27.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:rCokFydKZEgUqYK3CuEpnHDKHeU=', 'Content-Length': '0'} 2024-02-01T10:47:27.146 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a50506f5461c260-0065bb76bf-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:27.188 DEBUG:boto:b'vclfcn-61999falseobj1nulltrue2024-02-01T10:46:46.359Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:46:46.380Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:46:46.433Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:46:46.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:27.188 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-02-01T10:47:27.189 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-02-01T10:47:27.189 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-02-01T10:47:27.189 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-02-01T10:47:27.189 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:47:27.189 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:47:27.189 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:47:27.189 DEBUG:boto:path=/vclfcn-61/?versions&max-keys=999 2024-02-01T10:47:27.189 DEBUG:boto:auth_path=/vclfcn-61/?versions&max-keys=999 2024-02-01T10:47:27.189 DEBUG:boto:Method: GET 2024-02-01T10:47:27.190 DEBUG:boto:Path: /vclfcn-61/?versions&max-keys=999 2024-02-01T10:47:27.190 DEBUG:boto:Data: 2024-02-01T10:47:27.190 DEBUG:boto:Headers: {} 2024-02-01T10:47:27.190 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:27.190 DEBUG:boto:Port: 8000 2024-02-01T10:47:27.190 DEBUG:boto:Params: {} 2024-02-01T10:47:27.190 DEBUG:boto:Token: None 2024-02-01T10:47:27.190 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:27 GMT /vclfcn-61/?versions 2024-02-01T10:47:27.190 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:rCokFydKZEgUqYK3CuEpnHDKHeU= 2024-02-01T10:47:27.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:rCokFydKZEgUqYK3CuEpnHDKHeU=', 'Content-Length': '0'} 2024-02-01T10:47:27.195 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000746e0687f5273afb-0065bb76bf-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:27.196 DEBUG:boto:b'vclfcn-61999falseobj1nulltrue2024-02-01T10:46:46.359Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:46:46.380Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:46:46.433Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:46:46.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:27.196 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-02-01T10:47:27.196 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-02-01T10:47:27.196 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-02-01T10:47:27.196 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-02-01T10:47:27.196 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:47:27.197 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:47:27.197 DEBUG:boto:path=/vclfcn-61/?versions&max-keys=999 2024-02-01T10:47:27.197 DEBUG:boto:auth_path=/vclfcn-61/?versions&max-keys=999 2024-02-01T10:47:27.197 DEBUG:boto:Method: GET 2024-02-01T10:47:27.197 DEBUG:boto:Path: /vclfcn-61/?versions&max-keys=999 2024-02-01T10:47:27.197 DEBUG:boto:Data: 2024-02-01T10:47:27.197 DEBUG:boto:Headers: {} 2024-02-01T10:47:27.197 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:27.197 DEBUG:boto:Port: 8001 2024-02-01T10:47:27.197 DEBUG:boto:Params: {} 2024-02-01T10:47:27.197 DEBUG:boto:Token: None 2024-02-01T10:47:27.198 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:27 GMT /vclfcn-61/?versions 2024-02-01T10:47:27.198 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:rCokFydKZEgUqYK3CuEpnHDKHeU= 2024-02-01T10:47:27.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:rCokFydKZEgUqYK3CuEpnHDKHeU=', 'Content-Length': '0'} 2024-02-01T10:47:27.203 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb946f2e8dd0b2b8-0065bb76bf-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:27.203 DEBUG:boto:b'vclfcn-61999falseobj1nulltrue2024-02-01T10:46:46.359Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:46:46.380Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:46:46.433Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:46:46.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:27.203 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:47:27.203 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:47:27.203 DEBUG:boto:path=/vclfcn-61/?versions&max-keys=999 2024-02-01T10:47:27.203 DEBUG:boto:auth_path=/vclfcn-61/?versions&max-keys=999 2024-02-01T10:47:27.204 DEBUG:boto:Method: GET 2024-02-01T10:47:27.204 DEBUG:boto:Path: /vclfcn-61/?versions&max-keys=999 2024-02-01T10:47:27.204 DEBUG:boto:Data: 2024-02-01T10:47:27.204 DEBUG:boto:Headers: {} 2024-02-01T10:47:27.204 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:27.204 DEBUG:boto:Port: 8000 2024-02-01T10:47:27.204 DEBUG:boto:Params: {} 2024-02-01T10:47:27.204 DEBUG:boto:Token: None 2024-02-01T10:47:27.204 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:27 GMT /vclfcn-61/?versions 2024-02-01T10:47:27.204 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:rCokFydKZEgUqYK3CuEpnHDKHeU= 2024-02-01T10:47:27.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:rCokFydKZEgUqYK3CuEpnHDKHeU=', 'Content-Length': '0'} 2024-02-01T10:47:27.209 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ffc3ffdd3aae279-0065bb76bf-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:27.209 DEBUG:boto:b'vclfcn-61999falseobj1nulltrue2024-02-01T10:46:46.359Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:46:46.380Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:46:46.433Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:46:46.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:27.209 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:47:27.209 DEBUG:boto:path=/vclfcn-61/obj1 2024-02-01T10:47:27.210 DEBUG:boto:auth_path=/vclfcn-61/obj1 2024-02-01T10:47:27.210 DEBUG:boto:path=/vclfcn-61/obj1?versionId=null 2024-02-01T10:47:27.210 DEBUG:boto:auth_path=/vclfcn-61/obj1?versionId=null 2024-02-01T10:47:27.210 DEBUG:boto:Method: GET 2024-02-01T10:47:27.210 DEBUG:boto:Path: /vclfcn-61/obj1?versionId=null 2024-02-01T10:47:27.210 DEBUG:boto:Data: 2024-02-01T10:47:27.210 DEBUG:boto:Headers: {} 2024-02-01T10:47:27.210 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:27.210 DEBUG:boto:Port: 8001 2024-02-01T10:47:27.210 DEBUG:boto:Params: {} 2024-02-01T10:47:27.210 DEBUG:boto:Token: None 2024-02-01T10:47:27.211 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:27 GMT /vclfcn-61/obj1?versionId=null 2024-02-01T10:47:27.211 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/uBIkh1KyNcc2C+0M6MX17xWNe4= 2024-02-01T10:47:27.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/uBIkh1KyNcc2C+0M6MX17xWNe4=', 'Content-Length': '0'} 2024-02-01T10:47:27.214 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001dbb8c79111927c5-0065bb76bf-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:27.214 DEBUG:boto:path=/vclfcn-61/obj1 2024-02-01T10:47:27.214 DEBUG:boto:auth_path=/vclfcn-61/obj1 2024-02-01T10:47:27.214 DEBUG:boto:path=/vclfcn-61/obj1?versionId=null 2024-02-01T10:47:27.214 DEBUG:boto:auth_path=/vclfcn-61/obj1?versionId=null 2024-02-01T10:47:27.214 DEBUG:boto:Method: GET 2024-02-01T10:47:27.215 DEBUG:boto:Path: /vclfcn-61/obj1?versionId=null 2024-02-01T10:47:27.215 DEBUG:boto:Data: 2024-02-01T10:47:27.215 DEBUG:boto:Headers: {} 2024-02-01T10:47:27.215 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:27.215 DEBUG:boto:Port: 8000 2024-02-01T10:47:27.215 DEBUG:boto:Params: {} 2024-02-01T10:47:27.215 DEBUG:boto:Token: None 2024-02-01T10:47:27.215 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:27 GMT /vclfcn-61/obj1?versionId=null 2024-02-01T10:47:27.215 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/uBIkh1KyNcc2C+0M6MX17xWNe4= 2024-02-01T10:47:27.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/uBIkh1KyNcc2C+0M6MX17xWNe4=', 'Content-Length': '0'} 2024-02-01T10:47:27.218 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000084947c66ce96bca9-0065bb76bf-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:27.260 DEBUG:boto:path=/vclfcn-61/obj1 2024-02-01T10:47:27.260 DEBUG:boto:auth_path=/vclfcn-61/obj1 2024-02-01T10:47:27.260 DEBUG:boto:path=/vclfcn-61/obj1?versionId=null 2024-02-01T10:47:27.260 DEBUG:boto:auth_path=/vclfcn-61/obj1?versionId=null 2024-02-01T10:47:27.260 DEBUG:boto:Method: HEAD 2024-02-01T10:47:27.261 DEBUG:boto:Path: /vclfcn-61/obj1?versionId=null 2024-02-01T10:47:27.261 DEBUG:boto:Data: 2024-02-01T10:47:27.261 DEBUG:boto:Headers: {} 2024-02-01T10:47:27.261 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:27.261 DEBUG:boto:Port: 8001 2024-02-01T10:47:27.261 DEBUG:boto:Params: {} 2024-02-01T10:47:27.261 DEBUG:boto:Token: None 2024-02-01T10:47:27.261 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:27 GMT /vclfcn-61/obj1?versionId=null 2024-02-01T10:47:27.261 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:G/Z8ApmAxFTeCHNnkIEKwLZO8l8= 2024-02-01T10:47:27.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:G/Z8ApmAxFTeCHNnkIEKwLZO8l8=', 'Content-Length': '0'} 2024-02-01T10:47:27.264 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000c0d8e59ebc38207-0065bb76bf-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:27.265 DEBUG:boto:path=/vclfcn-61/obj1 2024-02-01T10:47:27.265 DEBUG:boto:auth_path=/vclfcn-61/obj1 2024-02-01T10:47:27.265 DEBUG:boto:path=/vclfcn-61/obj1?versionId=null 2024-02-01T10:47:27.265 DEBUG:boto:auth_path=/vclfcn-61/obj1?versionId=null 2024-02-01T10:47:27.265 DEBUG:boto:Method: HEAD 2024-02-01T10:47:27.265 DEBUG:boto:Path: /vclfcn-61/obj1?versionId=null 2024-02-01T10:47:27.265 DEBUG:boto:Data: 2024-02-01T10:47:27.265 DEBUG:boto:Headers: {} 2024-02-01T10:47:27.265 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:27.265 DEBUG:boto:Port: 8000 2024-02-01T10:47:27.266 DEBUG:boto:Params: {} 2024-02-01T10:47:27.266 DEBUG:boto:Token: None 2024-02-01T10:47:27.266 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:27 GMT /vclfcn-61/obj1?versionId=null 2024-02-01T10:47:27.266 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:G/Z8ApmAxFTeCHNnkIEKwLZO8l8= 2024-02-01T10:47:27.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:G/Z8ApmAxFTeCHNnkIEKwLZO8l8=', 'Content-Length': '0'} 2024-02-01T10:47:27.269 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d38e65de854db74e-0065bb76bf-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:27.269 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:47:27.269 DEBUG:boto:path=/vclfcn-61/obj1 2024-02-01T10:47:27.270 DEBUG:boto:auth_path=/vclfcn-61/obj1 2024-02-01T10:47:27.270 DEBUG:boto:path=/vclfcn-61/obj1?versionId=null 2024-02-01T10:47:27.270 DEBUG:boto:auth_path=/vclfcn-61/obj1?versionId=null 2024-02-01T10:47:27.270 DEBUG:boto:Method: GET 2024-02-01T10:47:27.270 DEBUG:boto:Path: /vclfcn-61/obj1?versionId=null 2024-02-01T10:47:27.270 DEBUG:boto:Data: 2024-02-01T10:47:27.270 DEBUG:boto:Headers: {} 2024-02-01T10:47:27.270 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:27.270 DEBUG:boto:Port: 8001 2024-02-01T10:47:27.270 DEBUG:boto:Params: {} 2024-02-01T10:47:27.270 DEBUG:boto:Token: None 2024-02-01T10:47:27.271 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:27 GMT /vclfcn-61/obj1?versionId=null 2024-02-01T10:47:27.271 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/uBIkh1KyNcc2C+0M6MX17xWNe4= 2024-02-01T10:47:27.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/uBIkh1KyNcc2C+0M6MX17xWNe4=', 'Content-Length': '0'} 2024-02-01T10:47:27.274 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ec9f684ef87858af-0065bb76bf-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:27.316 DEBUG:boto:path=/vclfcn-61/obj1 2024-02-01T10:47:27.316 DEBUG:boto:auth_path=/vclfcn-61/obj1 2024-02-01T10:47:27.316 DEBUG:boto:path=/vclfcn-61/obj1?versionId=null 2024-02-01T10:47:27.316 DEBUG:boto:auth_path=/vclfcn-61/obj1?versionId=null 2024-02-01T10:47:27.317 DEBUG:boto:Method: GET 2024-02-01T10:47:27.317 DEBUG:boto:Path: /vclfcn-61/obj1?versionId=null 2024-02-01T10:47:27.317 DEBUG:boto:Data: 2024-02-01T10:47:27.317 DEBUG:boto:Headers: {} 2024-02-01T10:47:27.317 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:27.317 DEBUG:boto:Port: 8000 2024-02-01T10:47:27.317 DEBUG:boto:Params: {} 2024-02-01T10:47:27.318 DEBUG:boto:Token: None 2024-02-01T10:47:27.318 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:27 GMT /vclfcn-61/obj1?versionId=null 2024-02-01T10:47:27.318 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/uBIkh1KyNcc2C+0M6MX17xWNe4= 2024-02-01T10:47:27.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/uBIkh1KyNcc2C+0M6MX17xWNe4=', 'Content-Length': '0'} 2024-02-01T10:47:27.321 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f2629a13055c0d40-0065bb76bf-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:27.322 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:47:27.322 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:47:27.322 DEBUG:boto:path=/vclfcn-61/?prefix=obj1 2024-02-01T10:47:27.322 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj1 2024-02-01T10:47:27.322 DEBUG:boto:Method: GET 2024-02-01T10:47:27.322 DEBUG:boto:Path: /vclfcn-61/?prefix=obj1 2024-02-01T10:47:27.322 DEBUG:boto:Data: 2024-02-01T10:47:27.322 DEBUG:boto:Headers: {} 2024-02-01T10:47:27.322 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:27.322 DEBUG:boto:Port: 8001 2024-02-01T10:47:27.323 DEBUG:boto:Params: {} 2024-02-01T10:47:27.323 DEBUG:boto:Token: None 2024-02-01T10:47:27.323 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:27 GMT /vclfcn-61/ 2024-02-01T10:47:27.323 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ww39+/07l7+WNAHvS5qWCYZMIDI= 2024-02-01T10:47:27.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ww39+/07l7+WNAHvS5qWCYZMIDI=', 'Content-Length': '0'} 2024-02-01T10:47:27.327 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039fc0961bba89b1a-0065bb76bf-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:27.328 DEBUG:boto:b'vclfcn-61obj11000falseobj12024-02-01T10:46:46.359Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:27.328 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:47:27.328 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:47:27.328 DEBUG:boto:path=/vclfcn-61/?prefix=obj1 2024-02-01T10:47:27.329 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj1 2024-02-01T10:47:27.329 DEBUG:boto:Method: GET 2024-02-01T10:47:27.329 DEBUG:boto:Path: /vclfcn-61/?prefix=obj1 2024-02-01T10:47:27.329 DEBUG:boto:Data: 2024-02-01T10:47:27.329 DEBUG:boto:Headers: {} 2024-02-01T10:47:27.329 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:27.329 DEBUG:boto:Port: 8000 2024-02-01T10:47:27.329 DEBUG:boto:Params: {} 2024-02-01T10:47:27.329 DEBUG:boto:Token: None 2024-02-01T10:47:27.329 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:27 GMT /vclfcn-61/ 2024-02-01T10:47:27.330 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ww39+/07l7+WNAHvS5qWCYZMIDI= 2024-02-01T10:47:27.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ww39+/07l7+WNAHvS5qWCYZMIDI=', 'Content-Length': '0'} 2024-02-01T10:47:27.334 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf8a38d5390cd1f0-0065bb76bf-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:27.334 DEBUG:boto:b'vclfcn-61obj11000falseobj12024-02-01T10:46:46.359Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:27.335 DEBUG:boto:path=/vclfcn-61/obj1 2024-02-01T10:47:27.335 DEBUG:boto:auth_path=/vclfcn-61/obj1 2024-02-01T10:47:27.335 DEBUG:boto:Method: HEAD 2024-02-01T10:47:27.335 DEBUG:boto:Path: /vclfcn-61/obj1 2024-02-01T10:47:27.335 DEBUG:boto:Data: 2024-02-01T10:47:27.335 DEBUG:boto:Headers: {} 2024-02-01T10:47:27.335 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:27.335 DEBUG:boto:Port: 8001 2024-02-01T10:47:27.335 DEBUG:boto:Params: {} 2024-02-01T10:47:27.335 DEBUG:boto:Token: None 2024-02-01T10:47:27.336 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:27 GMT /vclfcn-61/obj1 2024-02-01T10:47:27.336 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Cqmj2+NFivL9wdwlm8XCIlYq0z8= 2024-02-01T10:47:27.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Cqmj2+NFivL9wdwlm8XCIlYq0z8=', 'Content-Length': '0'} 2024-02-01T10:47:27.338 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000088ee73b61a25ec9b-0065bb76bf-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:27.339 DEBUG:boto:path=/vclfcn-61/obj1 2024-02-01T10:47:27.339 DEBUG:boto:auth_path=/vclfcn-61/obj1 2024-02-01T10:47:27.339 DEBUG:boto:Method: HEAD 2024-02-01T10:47:27.339 DEBUG:boto:Path: /vclfcn-61/obj1 2024-02-01T10:47:27.339 DEBUG:boto:Data: 2024-02-01T10:47:27.339 DEBUG:boto:Headers: {} 2024-02-01T10:47:27.339 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:27.339 DEBUG:boto:Port: 8000 2024-02-01T10:47:27.339 DEBUG:boto:Params: {} 2024-02-01T10:47:27.340 DEBUG:boto:Token: None 2024-02-01T10:47:27.340 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:27 GMT /vclfcn-61/obj1 2024-02-01T10:47:27.340 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Cqmj2+NFivL9wdwlm8XCIlYq0z8= 2024-02-01T10:47:27.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Cqmj2+NFivL9wdwlm8XCIlYq0z8=', 'Content-Length': '0'} 2024-02-01T10:47:27.343 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bf1038a007ad5111-0065bb76bf-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:27.343 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:47:27.343 DEBUG:boto:path=/vclfcn-61/obj1 2024-02-01T10:47:27.343 DEBUG:boto:auth_path=/vclfcn-61/obj1 2024-02-01T10:47:27.343 DEBUG:boto:Method: GET 2024-02-01T10:47:27.343 DEBUG:boto:Path: /vclfcn-61/obj1 2024-02-01T10:47:27.344 DEBUG:boto:Data: 2024-02-01T10:47:27.344 DEBUG:boto:Headers: {} 2024-02-01T10:47:27.344 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:27.344 DEBUG:boto:Port: 8001 2024-02-01T10:47:27.344 DEBUG:boto:Params: {} 2024-02-01T10:47:27.344 DEBUG:boto:Token: None 2024-02-01T10:47:27.344 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:27 GMT /vclfcn-61/obj1 2024-02-01T10:47:27.344 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:3P+pv6W7/8X8uOw2eknTZ116Nws= 2024-02-01T10:47:27.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:3P+pv6W7/8X8uOw2eknTZ116Nws=', 'Content-Length': '0'} 2024-02-01T10:47:27.347 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000696fa81a9d4f657-0065bb76bf-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:27.347 DEBUG:boto:path=/vclfcn-61/obj1 2024-02-01T10:47:27.348 DEBUG:boto:auth_path=/vclfcn-61/obj1 2024-02-01T10:47:27.348 DEBUG:boto:Method: GET 2024-02-01T10:47:27.348 DEBUG:boto:Path: /vclfcn-61/obj1 2024-02-01T10:47:27.348 DEBUG:boto:Data: 2024-02-01T10:47:27.348 DEBUG:boto:Headers: {} 2024-02-01T10:47:27.348 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:27.348 DEBUG:boto:Port: 8000 2024-02-01T10:47:27.348 DEBUG:boto:Params: {} 2024-02-01T10:47:27.348 DEBUG:boto:Token: None 2024-02-01T10:47:27.349 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:27 GMT /vclfcn-61/obj1 2024-02-01T10:47:27.349 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:3P+pv6W7/8X8uOw2eknTZ116Nws= 2024-02-01T10:47:27.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:3P+pv6W7/8X8uOw2eknTZ116Nws=', 'Content-Length': '0'} 2024-02-01T10:47:27.352 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004b28fe53b3ff9253-0065bb76bf-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:27.396 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:47:27.396 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:47:27.396 DEBUG:boto:path=/vclfcn-61/?prefix=obj1 2024-02-01T10:47:27.396 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj1 2024-02-01T10:47:27.397 DEBUG:boto:Method: GET 2024-02-01T10:47:27.397 DEBUG:boto:Path: /vclfcn-61/?prefix=obj1 2024-02-01T10:47:27.397 DEBUG:boto:Data: 2024-02-01T10:47:27.397 DEBUG:boto:Headers: {} 2024-02-01T10:47:27.397 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:27.397 DEBUG:boto:Port: 8001 2024-02-01T10:47:27.397 DEBUG:boto:Params: {} 2024-02-01T10:47:27.397 DEBUG:boto:Token: None 2024-02-01T10:47:27.398 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:27 GMT /vclfcn-61/ 2024-02-01T10:47:27.398 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ww39+/07l7+WNAHvS5qWCYZMIDI= 2024-02-01T10:47:27.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ww39+/07l7+WNAHvS5qWCYZMIDI=', 'Content-Length': '0'} 2024-02-01T10:47:27.402 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003bbdddf4ed774865-0065bb76bf-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:27.403 DEBUG:boto:b'vclfcn-61obj11000falseobj12024-02-01T10:46:46.359Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:27.403 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:47:27.403 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:47:27.403 DEBUG:boto:path=/vclfcn-61/?prefix=obj1 2024-02-01T10:47:27.403 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj1 2024-02-01T10:47:27.404 DEBUG:boto:Method: GET 2024-02-01T10:47:27.404 DEBUG:boto:Path: /vclfcn-61/?prefix=obj1 2024-02-01T10:47:27.404 DEBUG:boto:Data: 2024-02-01T10:47:27.404 DEBUG:boto:Headers: {} 2024-02-01T10:47:27.404 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:27.404 DEBUG:boto:Port: 8000 2024-02-01T10:47:27.404 DEBUG:boto:Params: {} 2024-02-01T10:47:27.404 DEBUG:boto:Token: None 2024-02-01T10:47:27.404 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:27 GMT /vclfcn-61/ 2024-02-01T10:47:27.405 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ww39+/07l7+WNAHvS5qWCYZMIDI= 2024-02-01T10:47:27.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ww39+/07l7+WNAHvS5qWCYZMIDI=', 'Content-Length': '0'} 2024-02-01T10:47:27.409 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d7d5c5c4c80267d3-0065bb76bf-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:27.409 DEBUG:boto:b'vclfcn-61obj11000falseobj12024-02-01T10:46:46.359Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:27.410 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:47:27.410 DEBUG:boto:path=/vclfcn-61/obj2 2024-02-01T10:47:27.410 DEBUG:boto:auth_path=/vclfcn-61/obj2 2024-02-01T10:47:27.410 DEBUG:boto:path=/vclfcn-61/obj2?versionId=null 2024-02-01T10:47:27.410 DEBUG:boto:auth_path=/vclfcn-61/obj2?versionId=null 2024-02-01T10:47:27.410 DEBUG:boto:Method: GET 2024-02-01T10:47:27.410 DEBUG:boto:Path: /vclfcn-61/obj2?versionId=null 2024-02-01T10:47:27.410 DEBUG:boto:Data: 2024-02-01T10:47:27.411 DEBUG:boto:Headers: {} 2024-02-01T10:47:27.411 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:27.411 DEBUG:boto:Port: 8001 2024-02-01T10:47:27.411 DEBUG:boto:Params: {} 2024-02-01T10:47:27.411 DEBUG:boto:Token: None 2024-02-01T10:47:27.411 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:27 GMT /vclfcn-61/obj2?versionId=null 2024-02-01T10:47:27.411 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:D5MCu96OhDR4vDqmGmIwjFp2xEE= 2024-02-01T10:47:27.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:D5MCu96OhDR4vDqmGmIwjFp2xEE=', 'Content-Length': '0'} 2024-02-01T10:47:27.414 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000040eddcd4fcc41dc4-0065bb76bf-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:27.456 DEBUG:boto:path=/vclfcn-61/obj2 2024-02-01T10:47:27.456 DEBUG:boto:auth_path=/vclfcn-61/obj2 2024-02-01T10:47:27.457 DEBUG:boto:path=/vclfcn-61/obj2?versionId=null 2024-02-01T10:47:27.457 DEBUG:boto:auth_path=/vclfcn-61/obj2?versionId=null 2024-02-01T10:47:27.457 DEBUG:boto:Method: GET 2024-02-01T10:47:27.457 DEBUG:boto:Path: /vclfcn-61/obj2?versionId=null 2024-02-01T10:47:27.457 DEBUG:boto:Data: 2024-02-01T10:47:27.457 DEBUG:boto:Headers: {} 2024-02-01T10:47:27.457 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:27.457 DEBUG:boto:Port: 8000 2024-02-01T10:47:27.457 DEBUG:boto:Params: {} 2024-02-01T10:47:27.458 DEBUG:boto:Token: None 2024-02-01T10:47:27.458 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:27 GMT /vclfcn-61/obj2?versionId=null 2024-02-01T10:47:27.458 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:D5MCu96OhDR4vDqmGmIwjFp2xEE= 2024-02-01T10:47:27.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:D5MCu96OhDR4vDqmGmIwjFp2xEE=', 'Content-Length': '0'} 2024-02-01T10:47:27.461 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000056d8b4deff33c523-0065bb76bf-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:27.461 DEBUG:boto:path=/vclfcn-61/obj2 2024-02-01T10:47:27.461 DEBUG:boto:auth_path=/vclfcn-61/obj2 2024-02-01T10:47:27.461 DEBUG:boto:path=/vclfcn-61/obj2?versionId=null 2024-02-01T10:47:27.462 DEBUG:boto:auth_path=/vclfcn-61/obj2?versionId=null 2024-02-01T10:47:27.462 DEBUG:boto:Method: HEAD 2024-02-01T10:47:27.462 DEBUG:boto:Path: /vclfcn-61/obj2?versionId=null 2024-02-01T10:47:27.462 DEBUG:boto:Data: 2024-02-01T10:47:27.462 DEBUG:boto:Headers: {} 2024-02-01T10:47:27.462 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:27.462 DEBUG:boto:Port: 8001 2024-02-01T10:47:27.462 DEBUG:boto:Params: {} 2024-02-01T10:47:27.462 DEBUG:boto:Token: None 2024-02-01T10:47:27.463 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:27 GMT /vclfcn-61/obj2?versionId=null 2024-02-01T10:47:27.463 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7O07phqF522YgYCZJwZtf7WY3fM= 2024-02-01T10:47:27.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7O07phqF522YgYCZJwZtf7WY3fM=', 'Content-Length': '0'} 2024-02-01T10:47:27.465 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000002f23df4cfba4ad8-0065bb76bf-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:27.466 DEBUG:boto:path=/vclfcn-61/obj2 2024-02-01T10:47:27.466 DEBUG:boto:auth_path=/vclfcn-61/obj2 2024-02-01T10:47:27.466 DEBUG:boto:path=/vclfcn-61/obj2?versionId=null 2024-02-01T10:47:27.466 DEBUG:boto:auth_path=/vclfcn-61/obj2?versionId=null 2024-02-01T10:47:27.466 DEBUG:boto:Method: HEAD 2024-02-01T10:47:27.466 DEBUG:boto:Path: /vclfcn-61/obj2?versionId=null 2024-02-01T10:47:27.466 DEBUG:boto:Data: 2024-02-01T10:47:27.466 DEBUG:boto:Headers: {} 2024-02-01T10:47:27.466 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:27.467 DEBUG:boto:Port: 8000 2024-02-01T10:47:27.467 DEBUG:boto:Params: {} 2024-02-01T10:47:27.467 DEBUG:boto:Token: None 2024-02-01T10:47:27.467 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:27 GMT /vclfcn-61/obj2?versionId=null 2024-02-01T10:47:27.467 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7O07phqF522YgYCZJwZtf7WY3fM= 2024-02-01T10:47:27.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7O07phqF522YgYCZJwZtf7WY3fM=', 'Content-Length': '0'} 2024-02-01T10:47:27.470 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000112fc6c4ea869739-0065bb76bf-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:27.470 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:47:27.470 DEBUG:boto:path=/vclfcn-61/obj2 2024-02-01T10:47:27.470 DEBUG:boto:auth_path=/vclfcn-61/obj2 2024-02-01T10:47:27.470 DEBUG:boto:path=/vclfcn-61/obj2?versionId=null 2024-02-01T10:47:27.470 DEBUG:boto:auth_path=/vclfcn-61/obj2?versionId=null 2024-02-01T10:47:27.471 DEBUG:boto:Method: GET 2024-02-01T10:47:27.471 DEBUG:boto:Path: /vclfcn-61/obj2?versionId=null 2024-02-01T10:47:27.471 DEBUG:boto:Data: 2024-02-01T10:47:27.471 DEBUG:boto:Headers: {} 2024-02-01T10:47:27.471 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:27.471 DEBUG:boto:Port: 8001 2024-02-01T10:47:27.471 DEBUG:boto:Params: {} 2024-02-01T10:47:27.471 DEBUG:boto:Token: None 2024-02-01T10:47:27.471 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:27 GMT /vclfcn-61/obj2?versionId=null 2024-02-01T10:47:27.471 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:D5MCu96OhDR4vDqmGmIwjFp2xEE= 2024-02-01T10:47:27.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:D5MCu96OhDR4vDqmGmIwjFp2xEE=', 'Content-Length': '0'} 2024-02-01T10:47:27.474 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f6095018e2b1c0bb-0065bb76bf-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:27.475 DEBUG:boto:path=/vclfcn-61/obj2 2024-02-01T10:47:27.475 DEBUG:boto:auth_path=/vclfcn-61/obj2 2024-02-01T10:47:27.475 DEBUG:boto:path=/vclfcn-61/obj2?versionId=null 2024-02-01T10:47:27.475 DEBUG:boto:auth_path=/vclfcn-61/obj2?versionId=null 2024-02-01T10:47:27.475 DEBUG:boto:Method: GET 2024-02-01T10:47:27.475 DEBUG:boto:Path: /vclfcn-61/obj2?versionId=null 2024-02-01T10:47:27.475 DEBUG:boto:Data: 2024-02-01T10:47:27.475 DEBUG:boto:Headers: {} 2024-02-01T10:47:27.476 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:27.476 DEBUG:boto:Port: 8000 2024-02-01T10:47:27.476 DEBUG:boto:Params: {} 2024-02-01T10:47:27.476 DEBUG:boto:Token: None 2024-02-01T10:47:27.476 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:27 GMT /vclfcn-61/obj2?versionId=null 2024-02-01T10:47:27.476 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:D5MCu96OhDR4vDqmGmIwjFp2xEE= 2024-02-01T10:47:27.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:D5MCu96OhDR4vDqmGmIwjFp2xEE=', 'Content-Length': '0'} 2024-02-01T10:47:27.479 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000589a09b96f821d99-0065bb76bf-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:27.520 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:47:27.520 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:47:27.521 DEBUG:boto:path=/vclfcn-61/?prefix=obj2 2024-02-01T10:47:27.521 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj2 2024-02-01T10:47:27.521 DEBUG:boto:Method: GET 2024-02-01T10:47:27.521 DEBUG:boto:Path: /vclfcn-61/?prefix=obj2 2024-02-01T10:47:27.521 DEBUG:boto:Data: 2024-02-01T10:47:27.521 DEBUG:boto:Headers: {} 2024-02-01T10:47:27.521 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:27.521 DEBUG:boto:Port: 8001 2024-02-01T10:47:27.521 DEBUG:boto:Params: {} 2024-02-01T10:47:27.521 DEBUG:boto:Token: None 2024-02-01T10:47:27.522 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:27 GMT /vclfcn-61/ 2024-02-01T10:47:27.522 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ww39+/07l7+WNAHvS5qWCYZMIDI= 2024-02-01T10:47:27.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ww39+/07l7+WNAHvS5qWCYZMIDI=', 'Content-Length': '0'} 2024-02-01T10:47:27.527 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000036ad4661f91b39b5-0065bb76bf-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:27.527 DEBUG:boto:b'vclfcn-61obj21000falseobj22024-02-01T10:46:46.380Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:27.528 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:47:27.528 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:47:27.528 DEBUG:boto:path=/vclfcn-61/?prefix=obj2 2024-02-01T10:47:27.528 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj2 2024-02-01T10:47:27.528 DEBUG:boto:Method: GET 2024-02-01T10:47:27.528 DEBUG:boto:Path: /vclfcn-61/?prefix=obj2 2024-02-01T10:47:27.528 DEBUG:boto:Data: 2024-02-01T10:47:27.528 DEBUG:boto:Headers: {} 2024-02-01T10:47:27.529 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:27.529 DEBUG:boto:Port: 8000 2024-02-01T10:47:27.529 DEBUG:boto:Params: {} 2024-02-01T10:47:27.529 DEBUG:boto:Token: None 2024-02-01T10:47:27.529 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:27 GMT /vclfcn-61/ 2024-02-01T10:47:27.529 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ww39+/07l7+WNAHvS5qWCYZMIDI= 2024-02-01T10:47:27.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ww39+/07l7+WNAHvS5qWCYZMIDI=', 'Content-Length': '0'} 2024-02-01T10:47:27.534 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f6064d865795882-0065bb76bf-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:27.535 DEBUG:boto:b'vclfcn-61obj21000falseobj22024-02-01T10:46:46.380Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:27.535 DEBUG:boto:path=/vclfcn-61/obj2 2024-02-01T10:47:27.535 DEBUG:boto:auth_path=/vclfcn-61/obj2 2024-02-01T10:47:27.535 DEBUG:boto:Method: HEAD 2024-02-01T10:47:27.535 DEBUG:boto:Path: /vclfcn-61/obj2 2024-02-01T10:47:27.535 DEBUG:boto:Data: 2024-02-01T10:47:27.535 DEBUG:boto:Headers: {} 2024-02-01T10:47:27.536 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:27.536 DEBUG:boto:Port: 8001 2024-02-01T10:47:27.536 DEBUG:boto:Params: {} 2024-02-01T10:47:27.536 DEBUG:boto:Token: None 2024-02-01T10:47:27.536 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:27 GMT /vclfcn-61/obj2 2024-02-01T10:47:27.536 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:rZ/OKFeYaCP+1yXYnmPeD8cyjj8= 2024-02-01T10:47:27.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:rZ/OKFeYaCP+1yXYnmPeD8cyjj8=', 'Content-Length': '0'} 2024-02-01T10:47:27.539 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000046ca4082757abc6e-0065bb76bf-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:27.539 DEBUG:boto:path=/vclfcn-61/obj2 2024-02-01T10:47:27.539 DEBUG:boto:auth_path=/vclfcn-61/obj2 2024-02-01T10:47:27.539 DEBUG:boto:Method: HEAD 2024-02-01T10:47:27.539 DEBUG:boto:Path: /vclfcn-61/obj2 2024-02-01T10:47:27.539 DEBUG:boto:Data: 2024-02-01T10:47:27.540 DEBUG:boto:Headers: {} 2024-02-01T10:47:27.540 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:27.540 DEBUG:boto:Port: 8000 2024-02-01T10:47:27.540 DEBUG:boto:Params: {} 2024-02-01T10:47:27.540 DEBUG:boto:Token: None 2024-02-01T10:47:27.540 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:27 GMT /vclfcn-61/obj2 2024-02-01T10:47:27.540 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:rZ/OKFeYaCP+1yXYnmPeD8cyjj8= 2024-02-01T10:47:27.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:rZ/OKFeYaCP+1yXYnmPeD8cyjj8=', 'Content-Length': '0'} 2024-02-01T10:47:27.543 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000040ca653d4c50c3af-0065bb76bf-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:27.543 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:47:27.543 DEBUG:boto:path=/vclfcn-61/obj2 2024-02-01T10:47:27.544 DEBUG:boto:auth_path=/vclfcn-61/obj2 2024-02-01T10:47:27.544 DEBUG:boto:Method: GET 2024-02-01T10:47:27.544 DEBUG:boto:Path: /vclfcn-61/obj2 2024-02-01T10:47:27.544 DEBUG:boto:Data: 2024-02-01T10:47:27.544 DEBUG:boto:Headers: {} 2024-02-01T10:47:27.544 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:27.544 DEBUG:boto:Port: 8001 2024-02-01T10:47:27.544 DEBUG:boto:Params: {} 2024-02-01T10:47:27.544 DEBUG:boto:Token: None 2024-02-01T10:47:27.544 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:27 GMT /vclfcn-61/obj2 2024-02-01T10:47:27.545 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:xOOBpFfsms/xUj6PVccvRnueGRU= 2024-02-01T10:47:27.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:xOOBpFfsms/xUj6PVccvRnueGRU=', 'Content-Length': '0'} 2024-02-01T10:47:27.547 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e58a2f9f90542030-0065bb76bf-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:27.588 DEBUG:boto:path=/vclfcn-61/obj2 2024-02-01T10:47:27.588 DEBUG:boto:auth_path=/vclfcn-61/obj2 2024-02-01T10:47:27.588 DEBUG:boto:Method: GET 2024-02-01T10:47:27.588 DEBUG:boto:Path: /vclfcn-61/obj2 2024-02-01T10:47:27.588 DEBUG:boto:Data: 2024-02-01T10:47:27.589 DEBUG:boto:Headers: {} 2024-02-01T10:47:27.589 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:27.589 DEBUG:boto:Port: 8000 2024-02-01T10:47:27.589 DEBUG:boto:Params: {} 2024-02-01T10:47:27.589 DEBUG:boto:Token: None 2024-02-01T10:47:27.589 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:27 GMT /vclfcn-61/obj2 2024-02-01T10:47:27.589 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:xOOBpFfsms/xUj6PVccvRnueGRU= 2024-02-01T10:47:27.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:xOOBpFfsms/xUj6PVccvRnueGRU=', 'Content-Length': '0'} 2024-02-01T10:47:27.592 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d451cea83c69607b-0065bb76bf-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:27.592 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:47:27.593 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:47:27.593 DEBUG:boto:path=/vclfcn-61/?prefix=obj2 2024-02-01T10:47:27.593 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj2 2024-02-01T10:47:27.593 DEBUG:boto:Method: GET 2024-02-01T10:47:27.593 DEBUG:boto:Path: /vclfcn-61/?prefix=obj2 2024-02-01T10:47:27.593 DEBUG:boto:Data: 2024-02-01T10:47:27.593 DEBUG:boto:Headers: {} 2024-02-01T10:47:27.593 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:27.593 DEBUG:boto:Port: 8001 2024-02-01T10:47:27.593 DEBUG:boto:Params: {} 2024-02-01T10:47:27.593 DEBUG:boto:Token: None 2024-02-01T10:47:27.594 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:27 GMT /vclfcn-61/ 2024-02-01T10:47:27.594 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ww39+/07l7+WNAHvS5qWCYZMIDI= 2024-02-01T10:47:27.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ww39+/07l7+WNAHvS5qWCYZMIDI=', 'Content-Length': '0'} 2024-02-01T10:47:27.598 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9bd88be8dc6c475-0065bb76bf-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:27.599 DEBUG:boto:b'vclfcn-61obj21000falseobj22024-02-01T10:46:46.380Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:27.599 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:47:27.599 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:47:27.599 DEBUG:boto:path=/vclfcn-61/?prefix=obj2 2024-02-01T10:47:27.599 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj2 2024-02-01T10:47:27.600 DEBUG:boto:Method: GET 2024-02-01T10:47:27.600 DEBUG:boto:Path: /vclfcn-61/?prefix=obj2 2024-02-01T10:47:27.600 DEBUG:boto:Data: 2024-02-01T10:47:27.600 DEBUG:boto:Headers: {} 2024-02-01T10:47:27.600 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:27.600 DEBUG:boto:Port: 8000 2024-02-01T10:47:27.600 DEBUG:boto:Params: {} 2024-02-01T10:47:27.600 DEBUG:boto:Token: None 2024-02-01T10:47:27.600 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:27 GMT /vclfcn-61/ 2024-02-01T10:47:27.601 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ww39+/07l7+WNAHvS5qWCYZMIDI= 2024-02-01T10:47:27.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ww39+/07l7+WNAHvS5qWCYZMIDI=', 'Content-Length': '0'} 2024-02-01T10:47:27.605 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005eb247b49a3bc10d-0065bb76bf-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:27.648 DEBUG:boto:b'vclfcn-61obj21000falseobj22024-02-01T10:46:46.380Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:27.648 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:47:27.649 DEBUG:boto:path=/vclfcn-61/obj3 2024-02-01T10:47:27.649 DEBUG:boto:auth_path=/vclfcn-61/obj3 2024-02-01T10:47:27.649 DEBUG:boto:path=/vclfcn-61/obj3?versionId=null 2024-02-01T10:47:27.649 DEBUG:boto:auth_path=/vclfcn-61/obj3?versionId=null 2024-02-01T10:47:27.649 DEBUG:boto:Method: GET 2024-02-01T10:47:27.649 DEBUG:boto:Path: /vclfcn-61/obj3?versionId=null 2024-02-01T10:47:27.649 DEBUG:boto:Data: 2024-02-01T10:47:27.649 DEBUG:boto:Headers: {} 2024-02-01T10:47:27.649 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:27.650 DEBUG:boto:Port: 8001 2024-02-01T10:47:27.650 DEBUG:boto:Params: {} 2024-02-01T10:47:27.650 DEBUG:boto:Token: None 2024-02-01T10:47:27.650 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:27 GMT /vclfcn-61/obj3?versionId=null 2024-02-01T10:47:27.650 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:pcESn8YTywItku5G5o1Fj2/gM4A= 2024-02-01T10:47:27.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:pcESn8YTywItku5G5o1Fj2/gM4A=', 'Content-Length': '0'} 2024-02-01T10:47:27.653 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000098c5cce561091316-0065bb76bf-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:27.653 DEBUG:boto:path=/vclfcn-61/obj3 2024-02-01T10:47:27.653 DEBUG:boto:auth_path=/vclfcn-61/obj3 2024-02-01T10:47:27.653 DEBUG:boto:path=/vclfcn-61/obj3?versionId=null 2024-02-01T10:47:27.654 DEBUG:boto:auth_path=/vclfcn-61/obj3?versionId=null 2024-02-01T10:47:27.654 DEBUG:boto:Method: GET 2024-02-01T10:47:27.654 DEBUG:boto:Path: /vclfcn-61/obj3?versionId=null 2024-02-01T10:47:27.654 DEBUG:boto:Data: 2024-02-01T10:47:27.654 DEBUG:boto:Headers: {} 2024-02-01T10:47:27.654 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:27.654 DEBUG:boto:Port: 8000 2024-02-01T10:47:27.654 DEBUG:boto:Params: {} 2024-02-01T10:47:27.654 DEBUG:boto:Token: None 2024-02-01T10:47:27.655 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:27 GMT /vclfcn-61/obj3?versionId=null 2024-02-01T10:47:27.655 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:pcESn8YTywItku5G5o1Fj2/gM4A= 2024-02-01T10:47:27.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:pcESn8YTywItku5G5o1Fj2/gM4A=', 'Content-Length': '0'} 2024-02-01T10:47:27.658 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e59b78a9edb39ccb-0065bb76bf-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:27.658 DEBUG:boto:path=/vclfcn-61/obj3 2024-02-01T10:47:27.658 DEBUG:boto:auth_path=/vclfcn-61/obj3 2024-02-01T10:47:27.658 DEBUG:boto:path=/vclfcn-61/obj3?versionId=null 2024-02-01T10:47:27.658 DEBUG:boto:auth_path=/vclfcn-61/obj3?versionId=null 2024-02-01T10:47:27.658 DEBUG:boto:Method: HEAD 2024-02-01T10:47:27.659 DEBUG:boto:Path: /vclfcn-61/obj3?versionId=null 2024-02-01T10:47:27.659 DEBUG:boto:Data: 2024-02-01T10:47:27.659 DEBUG:boto:Headers: {} 2024-02-01T10:47:27.659 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:27.659 DEBUG:boto:Port: 8001 2024-02-01T10:47:27.659 DEBUG:boto:Params: {} 2024-02-01T10:47:27.659 DEBUG:boto:Token: None 2024-02-01T10:47:27.659 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:27 GMT /vclfcn-61/obj3?versionId=null 2024-02-01T10:47:27.659 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:skE6Abx4qbEy/IrinmDmjGhkZFw= 2024-02-01T10:47:27.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:skE6Abx4qbEy/IrinmDmjGhkZFw=', 'Content-Length': '0'} 2024-02-01T10:47:27.662 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e68cc6fdae88fc65-0065bb76bf-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:27.662 DEBUG:boto:path=/vclfcn-61/obj3 2024-02-01T10:47:27.663 DEBUG:boto:auth_path=/vclfcn-61/obj3 2024-02-01T10:47:27.663 DEBUG:boto:path=/vclfcn-61/obj3?versionId=null 2024-02-01T10:47:27.663 DEBUG:boto:auth_path=/vclfcn-61/obj3?versionId=null 2024-02-01T10:47:27.663 DEBUG:boto:Method: HEAD 2024-02-01T10:47:27.663 DEBUG:boto:Path: /vclfcn-61/obj3?versionId=null 2024-02-01T10:47:27.663 DEBUG:boto:Data: 2024-02-01T10:47:27.663 DEBUG:boto:Headers: {} 2024-02-01T10:47:27.663 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:27.663 DEBUG:boto:Port: 8000 2024-02-01T10:47:27.663 DEBUG:boto:Params: {} 2024-02-01T10:47:27.663 DEBUG:boto:Token: None 2024-02-01T10:47:27.664 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:27 GMT /vclfcn-61/obj3?versionId=null 2024-02-01T10:47:27.664 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:skE6Abx4qbEy/IrinmDmjGhkZFw= 2024-02-01T10:47:27.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:skE6Abx4qbEy/IrinmDmjGhkZFw=', 'Content-Length': '0'} 2024-02-01T10:47:27.667 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000018a5b2e9ea73d092-0065bb76bf-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:27.667 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:47:27.667 DEBUG:boto:path=/vclfcn-61/obj3 2024-02-01T10:47:27.667 DEBUG:boto:auth_path=/vclfcn-61/obj3 2024-02-01T10:47:27.668 DEBUG:boto:path=/vclfcn-61/obj3?versionId=null 2024-02-01T10:47:27.668 DEBUG:boto:auth_path=/vclfcn-61/obj3?versionId=null 2024-02-01T10:47:27.668 DEBUG:boto:Method: GET 2024-02-01T10:47:27.668 DEBUG:boto:Path: /vclfcn-61/obj3?versionId=null 2024-02-01T10:47:27.668 DEBUG:boto:Data: 2024-02-01T10:47:27.668 DEBUG:boto:Headers: {} 2024-02-01T10:47:27.668 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:27.668 DEBUG:boto:Port: 8001 2024-02-01T10:47:27.668 DEBUG:boto:Params: {} 2024-02-01T10:47:27.668 DEBUG:boto:Token: None 2024-02-01T10:47:27.669 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:27 GMT /vclfcn-61/obj3?versionId=null 2024-02-01T10:47:27.669 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:pcESn8YTywItku5G5o1Fj2/gM4A= 2024-02-01T10:47:27.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:pcESn8YTywItku5G5o1Fj2/gM4A=', 'Content-Length': '0'} 2024-02-01T10:47:27.672 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b9d168b9df223f8e-0065bb76bf-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:27.672 DEBUG:boto:path=/vclfcn-61/obj3 2024-02-01T10:47:27.672 DEBUG:boto:auth_path=/vclfcn-61/obj3 2024-02-01T10:47:27.672 DEBUG:boto:path=/vclfcn-61/obj3?versionId=null 2024-02-01T10:47:27.672 DEBUG:boto:auth_path=/vclfcn-61/obj3?versionId=null 2024-02-01T10:47:27.672 DEBUG:boto:Method: GET 2024-02-01T10:47:27.672 DEBUG:boto:Path: /vclfcn-61/obj3?versionId=null 2024-02-01T10:47:27.673 DEBUG:boto:Data: 2024-02-01T10:47:27.673 DEBUG:boto:Headers: {} 2024-02-01T10:47:27.673 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:27.673 DEBUG:boto:Port: 8000 2024-02-01T10:47:27.673 DEBUG:boto:Params: {} 2024-02-01T10:47:27.673 DEBUG:boto:Token: None 2024-02-01T10:47:27.673 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:27 GMT /vclfcn-61/obj3?versionId=null 2024-02-01T10:47:27.673 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:pcESn8YTywItku5G5o1Fj2/gM4A= 2024-02-01T10:47:27.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:pcESn8YTywItku5G5o1Fj2/gM4A=', 'Content-Length': '0'} 2024-02-01T10:47:27.676 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000077973a21f477f6bd-0065bb76bf-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:27.676 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:47:27.677 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:47:27.677 DEBUG:boto:path=/vclfcn-61/?prefix=obj3 2024-02-01T10:47:27.677 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj3 2024-02-01T10:47:27.677 DEBUG:boto:Method: GET 2024-02-01T10:47:27.677 DEBUG:boto:Path: /vclfcn-61/?prefix=obj3 2024-02-01T10:47:27.677 DEBUG:boto:Data: 2024-02-01T10:47:27.677 DEBUG:boto:Headers: {} 2024-02-01T10:47:27.677 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:27.677 DEBUG:boto:Port: 8001 2024-02-01T10:47:27.677 DEBUG:boto:Params: {} 2024-02-01T10:47:27.677 DEBUG:boto:Token: None 2024-02-01T10:47:27.678 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:27 GMT /vclfcn-61/ 2024-02-01T10:47:27.678 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ww39+/07l7+WNAHvS5qWCYZMIDI= 2024-02-01T10:47:27.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ww39+/07l7+WNAHvS5qWCYZMIDI=', 'Content-Length': '0'} 2024-02-01T10:47:27.682 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1df4bca2e570a07-0065bb76bf-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:27.683 DEBUG:boto:b'vclfcn-61obj31000falseobj32024-02-01T10:46:46.433Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:27.683 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:47:27.683 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:47:27.683 DEBUG:boto:path=/vclfcn-61/?prefix=obj3 2024-02-01T10:47:27.683 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj3 2024-02-01T10:47:27.683 DEBUG:boto:Method: GET 2024-02-01T10:47:27.683 DEBUG:boto:Path: /vclfcn-61/?prefix=obj3 2024-02-01T10:47:27.684 DEBUG:boto:Data: 2024-02-01T10:47:27.684 DEBUG:boto:Headers: {} 2024-02-01T10:47:27.684 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:27.684 DEBUG:boto:Port: 8000 2024-02-01T10:47:27.684 DEBUG:boto:Params: {} 2024-02-01T10:47:27.684 DEBUG:boto:Token: None 2024-02-01T10:47:27.684 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:27 GMT /vclfcn-61/ 2024-02-01T10:47:27.684 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ww39+/07l7+WNAHvS5qWCYZMIDI= 2024-02-01T10:47:27.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ww39+/07l7+WNAHvS5qWCYZMIDI=', 'Content-Length': '0'} 2024-02-01T10:47:27.689 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084fa8da1a6e8e920-0065bb76bf-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:27.732 DEBUG:boto:b'vclfcn-61obj31000falseobj32024-02-01T10:46:46.433Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:27.732 DEBUG:boto:path=/vclfcn-61/obj3 2024-02-01T10:47:27.732 DEBUG:boto:auth_path=/vclfcn-61/obj3 2024-02-01T10:47:27.733 DEBUG:boto:Method: HEAD 2024-02-01T10:47:27.733 DEBUG:boto:Path: /vclfcn-61/obj3 2024-02-01T10:47:27.733 DEBUG:boto:Data: 2024-02-01T10:47:27.733 DEBUG:boto:Headers: {} 2024-02-01T10:47:27.733 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:27.733 DEBUG:boto:Port: 8001 2024-02-01T10:47:27.733 DEBUG:boto:Params: {} 2024-02-01T10:47:27.733 DEBUG:boto:Token: None 2024-02-01T10:47:27.733 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:27 GMT /vclfcn-61/obj3 2024-02-01T10:47:27.734 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BDKGjRemL0PyELHGCVXb9uXZxpg= 2024-02-01T10:47:27.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BDKGjRemL0PyELHGCVXb9uXZxpg=', 'Content-Length': '0'} 2024-02-01T10:47:27.736 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f6296103d49ee5ca-0065bb76bf-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:27.737 DEBUG:boto:path=/vclfcn-61/obj3 2024-02-01T10:47:27.737 DEBUG:boto:auth_path=/vclfcn-61/obj3 2024-02-01T10:47:27.737 DEBUG:boto:Method: HEAD 2024-02-01T10:47:27.737 DEBUG:boto:Path: /vclfcn-61/obj3 2024-02-01T10:47:27.737 DEBUG:boto:Data: 2024-02-01T10:47:27.737 DEBUG:boto:Headers: {} 2024-02-01T10:47:27.737 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:27.737 DEBUG:boto:Port: 8000 2024-02-01T10:47:27.737 DEBUG:boto:Params: {} 2024-02-01T10:47:27.738 DEBUG:boto:Token: None 2024-02-01T10:47:27.738 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:27 GMT /vclfcn-61/obj3 2024-02-01T10:47:27.738 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BDKGjRemL0PyELHGCVXb9uXZxpg= 2024-02-01T10:47:27.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BDKGjRemL0PyELHGCVXb9uXZxpg=', 'Content-Length': '0'} 2024-02-01T10:47:27.740 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a77d00a0e17708e1-0065bb76bf-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:27.741 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:47:27.741 DEBUG:boto:path=/vclfcn-61/obj3 2024-02-01T10:47:27.741 DEBUG:boto:auth_path=/vclfcn-61/obj3 2024-02-01T10:47:27.741 DEBUG:boto:Method: GET 2024-02-01T10:47:27.741 DEBUG:boto:Path: /vclfcn-61/obj3 2024-02-01T10:47:27.741 DEBUG:boto:Data: 2024-02-01T10:47:27.741 DEBUG:boto:Headers: {} 2024-02-01T10:47:27.741 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:27.741 DEBUG:boto:Port: 8001 2024-02-01T10:47:27.742 DEBUG:boto:Params: {} 2024-02-01T10:47:27.742 DEBUG:boto:Token: None 2024-02-01T10:47:27.742 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:27 GMT /vclfcn-61/obj3 2024-02-01T10:47:27.742 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:hcMDpHvzSML7Tdg7ZdtULFtEc5k= 2024-02-01T10:47:27.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:hcMDpHvzSML7Tdg7ZdtULFtEc5k=', 'Content-Length': '0'} 2024-02-01T10:47:27.745 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e87200138701c0b3-0065bb76bf-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:27.788 DEBUG:boto:path=/vclfcn-61/obj3 2024-02-01T10:47:27.788 DEBUG:boto:auth_path=/vclfcn-61/obj3 2024-02-01T10:47:27.788 DEBUG:boto:Method: GET 2024-02-01T10:47:27.788 DEBUG:boto:Path: /vclfcn-61/obj3 2024-02-01T10:47:27.789 DEBUG:boto:Data: 2024-02-01T10:47:27.789 DEBUG:boto:Headers: {} 2024-02-01T10:47:27.789 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:27.789 DEBUG:boto:Port: 8000 2024-02-01T10:47:27.789 DEBUG:boto:Params: {} 2024-02-01T10:47:27.789 DEBUG:boto:Token: None 2024-02-01T10:47:27.789 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:27 GMT /vclfcn-61/obj3 2024-02-01T10:47:27.789 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:hcMDpHvzSML7Tdg7ZdtULFtEc5k= 2024-02-01T10:47:27.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:hcMDpHvzSML7Tdg7ZdtULFtEc5k=', 'Content-Length': '0'} 2024-02-01T10:47:27.792 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f9120aad3ab1b045-0065bb76bf-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:27.793 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:47:27.793 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:47:27.793 DEBUG:boto:path=/vclfcn-61/?prefix=obj3 2024-02-01T10:47:27.793 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj3 2024-02-01T10:47:27.793 DEBUG:boto:Method: GET 2024-02-01T10:47:27.793 DEBUG:boto:Path: /vclfcn-61/?prefix=obj3 2024-02-01T10:47:27.793 DEBUG:boto:Data: 2024-02-01T10:47:27.794 DEBUG:boto:Headers: {} 2024-02-01T10:47:27.794 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:27.794 DEBUG:boto:Port: 8001 2024-02-01T10:47:27.794 DEBUG:boto:Params: {} 2024-02-01T10:47:27.794 DEBUG:boto:Token: None 2024-02-01T10:47:27.794 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:27 GMT /vclfcn-61/ 2024-02-01T10:47:27.794 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ww39+/07l7+WNAHvS5qWCYZMIDI= 2024-02-01T10:47:27.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ww39+/07l7+WNAHvS5qWCYZMIDI=', 'Content-Length': '0'} 2024-02-01T10:47:27.798 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f92cfb8c0bd1f1b-0065bb76bf-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:27.799 DEBUG:boto:b'vclfcn-61obj31000falseobj32024-02-01T10:46:46.433Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:27.799 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:47:27.799 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:47:27.799 DEBUG:boto:path=/vclfcn-61/?prefix=obj3 2024-02-01T10:47:27.799 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj3 2024-02-01T10:47:27.800 DEBUG:boto:Method: GET 2024-02-01T10:47:27.800 DEBUG:boto:Path: /vclfcn-61/?prefix=obj3 2024-02-01T10:47:27.800 DEBUG:boto:Data: 2024-02-01T10:47:27.800 DEBUG:boto:Headers: {} 2024-02-01T10:47:27.800 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:27.800 DEBUG:boto:Port: 8000 2024-02-01T10:47:27.800 DEBUG:boto:Params: {} 2024-02-01T10:47:27.800 DEBUG:boto:Token: None 2024-02-01T10:47:27.800 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:27 GMT /vclfcn-61/ 2024-02-01T10:47:27.801 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ww39+/07l7+WNAHvS5qWCYZMIDI= 2024-02-01T10:47:27.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ww39+/07l7+WNAHvS5qWCYZMIDI=', 'Content-Length': '0'} 2024-02-01T10:47:27.805 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000061a61b0ab9dbdcf0-0065bb76bf-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:27.805 DEBUG:boto:b'vclfcn-61obj31000falseobj32024-02-01T10:46:46.433Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:27.805 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:47:27.805 DEBUG:boto:path=/vclfcn-61/obj4 2024-02-01T10:47:27.806 DEBUG:boto:auth_path=/vclfcn-61/obj4 2024-02-01T10:47:27.806 DEBUG:boto:path=/vclfcn-61/obj4?versionId=null 2024-02-01T10:47:27.806 DEBUG:boto:auth_path=/vclfcn-61/obj4?versionId=null 2024-02-01T10:47:27.806 DEBUG:boto:Method: GET 2024-02-01T10:47:27.806 DEBUG:boto:Path: /vclfcn-61/obj4?versionId=null 2024-02-01T10:47:27.806 DEBUG:boto:Data: 2024-02-01T10:47:27.806 DEBUG:boto:Headers: {} 2024-02-01T10:47:27.806 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:27.806 DEBUG:boto:Port: 8001 2024-02-01T10:47:27.806 DEBUG:boto:Params: {} 2024-02-01T10:47:27.807 DEBUG:boto:Token: None 2024-02-01T10:47:27.807 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:27 GMT /vclfcn-61/obj4?versionId=null 2024-02-01T10:47:27.807 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8FK+PPduXbZABzzkSBvvNm4ETzI= 2024-02-01T10:47:27.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8FK+PPduXbZABzzkSBvvNm4ETzI=', 'Content-Length': '0'} 2024-02-01T10:47:27.810 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001a12aa6b969aa1f7-0065bb76bf-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:27.810 DEBUG:boto:path=/vclfcn-61/obj4 2024-02-01T10:47:27.810 DEBUG:boto:auth_path=/vclfcn-61/obj4 2024-02-01T10:47:27.810 DEBUG:boto:path=/vclfcn-61/obj4?versionId=null 2024-02-01T10:47:27.810 DEBUG:boto:auth_path=/vclfcn-61/obj4?versionId=null 2024-02-01T10:47:27.810 DEBUG:boto:Method: GET 2024-02-01T10:47:27.810 DEBUG:boto:Path: /vclfcn-61/obj4?versionId=null 2024-02-01T10:47:27.810 DEBUG:boto:Data: 2024-02-01T10:47:27.811 DEBUG:boto:Headers: {} 2024-02-01T10:47:27.811 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:27.811 DEBUG:boto:Port: 8000 2024-02-01T10:47:27.811 DEBUG:boto:Params: {} 2024-02-01T10:47:27.811 DEBUG:boto:Token: None 2024-02-01T10:47:27.811 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:27 GMT /vclfcn-61/obj4?versionId=null 2024-02-01T10:47:27.811 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8FK+PPduXbZABzzkSBvvNm4ETzI= 2024-02-01T10:47:27.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8FK+PPduXbZABzzkSBvvNm4ETzI=', 'Content-Length': '0'} 2024-02-01T10:47:27.814 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fa33ab40c3959691-0065bb76bf-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:27.856 DEBUG:boto:path=/vclfcn-61/obj4 2024-02-01T10:47:27.856 DEBUG:boto:auth_path=/vclfcn-61/obj4 2024-02-01T10:47:27.856 DEBUG:boto:path=/vclfcn-61/obj4?versionId=null 2024-02-01T10:47:27.856 DEBUG:boto:auth_path=/vclfcn-61/obj4?versionId=null 2024-02-01T10:47:27.857 DEBUG:boto:Method: HEAD 2024-02-01T10:47:27.857 DEBUG:boto:Path: /vclfcn-61/obj4?versionId=null 2024-02-01T10:47:27.857 DEBUG:boto:Data: 2024-02-01T10:47:27.857 DEBUG:boto:Headers: {} 2024-02-01T10:47:27.857 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:27.857 DEBUG:boto:Port: 8001 2024-02-01T10:47:27.857 DEBUG:boto:Params: {} 2024-02-01T10:47:27.857 DEBUG:boto:Token: None 2024-02-01T10:47:27.857 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:27 GMT /vclfcn-61/obj4?versionId=null 2024-02-01T10:47:27.858 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mn7+ao2S62k9JpoTCsMI/5RP7Oo= 2024-02-01T10:47:27.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mn7+ao2S62k9JpoTCsMI/5RP7Oo=', 'Content-Length': '0'} 2024-02-01T10:47:27.861 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009686659548d743a8-0065bb76bf-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:27.861 DEBUG:boto:path=/vclfcn-61/obj4 2024-02-01T10:47:27.861 DEBUG:boto:auth_path=/vclfcn-61/obj4 2024-02-01T10:47:27.861 DEBUG:boto:path=/vclfcn-61/obj4?versionId=null 2024-02-01T10:47:27.861 DEBUG:boto:auth_path=/vclfcn-61/obj4?versionId=null 2024-02-01T10:47:27.861 DEBUG:boto:Method: HEAD 2024-02-01T10:47:27.862 DEBUG:boto:Path: /vclfcn-61/obj4?versionId=null 2024-02-01T10:47:27.862 DEBUG:boto:Data: 2024-02-01T10:47:27.862 DEBUG:boto:Headers: {} 2024-02-01T10:47:27.862 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:27.862 DEBUG:boto:Port: 8000 2024-02-01T10:47:27.862 DEBUG:boto:Params: {} 2024-02-01T10:47:27.862 DEBUG:boto:Token: None 2024-02-01T10:47:27.862 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:27 GMT /vclfcn-61/obj4?versionId=null 2024-02-01T10:47:27.862 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mn7+ao2S62k9JpoTCsMI/5RP7Oo= 2024-02-01T10:47:27.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mn7+ao2S62k9JpoTCsMI/5RP7Oo=', 'Content-Length': '0'} 2024-02-01T10:47:27.865 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007dd9c91a52de7a7e-0065bb76bf-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:27.865 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:47:27.865 DEBUG:boto:path=/vclfcn-61/obj4 2024-02-01T10:47:27.866 DEBUG:boto:auth_path=/vclfcn-61/obj4 2024-02-01T10:47:27.866 DEBUG:boto:path=/vclfcn-61/obj4?versionId=null 2024-02-01T10:47:27.866 DEBUG:boto:auth_path=/vclfcn-61/obj4?versionId=null 2024-02-01T10:47:27.866 DEBUG:boto:Method: GET 2024-02-01T10:47:27.866 DEBUG:boto:Path: /vclfcn-61/obj4?versionId=null 2024-02-01T10:47:27.866 DEBUG:boto:Data: 2024-02-01T10:47:27.866 DEBUG:boto:Headers: {} 2024-02-01T10:47:27.866 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:27.866 DEBUG:boto:Port: 8001 2024-02-01T10:47:27.866 DEBUG:boto:Params: {} 2024-02-01T10:47:27.867 DEBUG:boto:Token: None 2024-02-01T10:47:27.867 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:27 GMT /vclfcn-61/obj4?versionId=null 2024-02-01T10:47:27.867 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8FK+PPduXbZABzzkSBvvNm4ETzI= 2024-02-01T10:47:27.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8FK+PPduXbZABzzkSBvvNm4ETzI=', 'Content-Length': '0'} 2024-02-01T10:47:27.870 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005872a6ef3c92310d-0065bb76bf-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:27.912 DEBUG:boto:path=/vclfcn-61/obj4 2024-02-01T10:47:27.912 DEBUG:boto:auth_path=/vclfcn-61/obj4 2024-02-01T10:47:27.912 DEBUG:boto:path=/vclfcn-61/obj4?versionId=null 2024-02-01T10:47:27.912 DEBUG:boto:auth_path=/vclfcn-61/obj4?versionId=null 2024-02-01T10:47:27.912 DEBUG:boto:Method: GET 2024-02-01T10:47:27.913 DEBUG:boto:Path: /vclfcn-61/obj4?versionId=null 2024-02-01T10:47:27.913 DEBUG:boto:Data: 2024-02-01T10:47:27.913 DEBUG:boto:Headers: {} 2024-02-01T10:47:27.913 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:27.913 DEBUG:boto:Port: 8000 2024-02-01T10:47:27.913 DEBUG:boto:Params: {} 2024-02-01T10:47:27.913 DEBUG:boto:Token: None 2024-02-01T10:47:27.913 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:27 GMT /vclfcn-61/obj4?versionId=null 2024-02-01T10:47:27.913 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8FK+PPduXbZABzzkSBvvNm4ETzI= 2024-02-01T10:47:27.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8FK+PPduXbZABzzkSBvvNm4ETzI=', 'Content-Length': '0'} 2024-02-01T10:47:27.916 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000416da48c84e6bfd2-0065bb76bf-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:27.917 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:47:27.917 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:47:27.917 DEBUG:boto:path=/vclfcn-61/?prefix=obj4 2024-02-01T10:47:27.917 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj4 2024-02-01T10:47:27.917 DEBUG:boto:Method: GET 2024-02-01T10:47:27.917 DEBUG:boto:Path: /vclfcn-61/?prefix=obj4 2024-02-01T10:47:27.917 DEBUG:boto:Data: 2024-02-01T10:47:27.917 DEBUG:boto:Headers: {} 2024-02-01T10:47:27.917 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:27.917 DEBUG:boto:Port: 8001 2024-02-01T10:47:27.918 DEBUG:boto:Params: {} 2024-02-01T10:47:27.918 DEBUG:boto:Token: None 2024-02-01T10:47:27.918 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:27 GMT /vclfcn-61/ 2024-02-01T10:47:27.918 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ww39+/07l7+WNAHvS5qWCYZMIDI= 2024-02-01T10:47:27.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ww39+/07l7+WNAHvS5qWCYZMIDI=', 'Content-Length': '0'} 2024-02-01T10:47:27.922 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de195f2e2f0dc33f-0065bb76bf-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:27.922 DEBUG:boto:b'vclfcn-61obj41000falseobj42024-02-01T10:46:46.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:27.923 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:47:27.923 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:47:27.923 DEBUG:boto:path=/vclfcn-61/?prefix=obj4 2024-02-01T10:47:27.923 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj4 2024-02-01T10:47:27.923 DEBUG:boto:Method: GET 2024-02-01T10:47:27.923 DEBUG:boto:Path: /vclfcn-61/?prefix=obj4 2024-02-01T10:47:27.923 DEBUG:boto:Data: 2024-02-01T10:47:27.924 DEBUG:boto:Headers: {} 2024-02-01T10:47:27.924 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:27.924 DEBUG:boto:Port: 8000 2024-02-01T10:47:27.924 DEBUG:boto:Params: {} 2024-02-01T10:47:27.924 DEBUG:boto:Token: None 2024-02-01T10:47:27.924 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:27 GMT /vclfcn-61/ 2024-02-01T10:47:27.924 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ww39+/07l7+WNAHvS5qWCYZMIDI= 2024-02-01T10:47:27.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ww39+/07l7+WNAHvS5qWCYZMIDI=', 'Content-Length': '0'} 2024-02-01T10:47:27.928 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ad19cfe83edfa39-0065bb76bf-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:27.929 DEBUG:boto:b'vclfcn-61obj41000falseobj42024-02-01T10:46:46.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:27.929 DEBUG:boto:path=/vclfcn-61/obj4 2024-02-01T10:47:27.929 DEBUG:boto:auth_path=/vclfcn-61/obj4 2024-02-01T10:47:27.929 DEBUG:boto:Method: HEAD 2024-02-01T10:47:27.929 DEBUG:boto:Path: /vclfcn-61/obj4 2024-02-01T10:47:27.929 DEBUG:boto:Data: 2024-02-01T10:47:27.929 DEBUG:boto:Headers: {} 2024-02-01T10:47:27.929 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:27.929 DEBUG:boto:Port: 8001 2024-02-01T10:47:27.930 DEBUG:boto:Params: {} 2024-02-01T10:47:27.930 DEBUG:boto:Token: None 2024-02-01T10:47:27.930 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:27 GMT /vclfcn-61/obj4 2024-02-01T10:47:27.930 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:LBZs55omZ1Otp1OX9ygTzAYZwq0= 2024-02-01T10:47:27.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:LBZs55omZ1Otp1OX9ygTzAYZwq0=', 'Content-Length': '0'} 2024-02-01T10:47:27.933 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003c64a25df42dca6a-0065bb76bf-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:27.933 DEBUG:boto:path=/vclfcn-61/obj4 2024-02-01T10:47:27.933 DEBUG:boto:auth_path=/vclfcn-61/obj4 2024-02-01T10:47:27.933 DEBUG:boto:Method: HEAD 2024-02-01T10:47:27.933 DEBUG:boto:Path: /vclfcn-61/obj4 2024-02-01T10:47:27.933 DEBUG:boto:Data: 2024-02-01T10:47:27.934 DEBUG:boto:Headers: {} 2024-02-01T10:47:27.934 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:27.934 DEBUG:boto:Port: 8000 2024-02-01T10:47:27.934 DEBUG:boto:Params: {} 2024-02-01T10:47:27.934 DEBUG:boto:Token: None 2024-02-01T10:47:27.934 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:27 GMT /vclfcn-61/obj4 2024-02-01T10:47:27.934 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:LBZs55omZ1Otp1OX9ygTzAYZwq0= 2024-02-01T10:47:27.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:LBZs55omZ1Otp1OX9ygTzAYZwq0=', 'Content-Length': '0'} 2024-02-01T10:47:27.937 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000067214290d4aee14b-0065bb76bf-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:27.937 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:47:27.937 DEBUG:boto:path=/vclfcn-61/obj4 2024-02-01T10:47:27.937 DEBUG:boto:auth_path=/vclfcn-61/obj4 2024-02-01T10:47:27.937 DEBUG:boto:Method: GET 2024-02-01T10:47:27.937 DEBUG:boto:Path: /vclfcn-61/obj4 2024-02-01T10:47:27.938 DEBUG:boto:Data: 2024-02-01T10:47:27.938 DEBUG:boto:Headers: {} 2024-02-01T10:47:27.938 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:27.938 DEBUG:boto:Port: 8001 2024-02-01T10:47:27.938 DEBUG:boto:Params: {} 2024-02-01T10:47:27.938 DEBUG:boto:Token: None 2024-02-01T10:47:27.938 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:27 GMT /vclfcn-61/obj4 2024-02-01T10:47:27.938 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZDycMCht0nbZrtFjdxKubLJ6vzo= 2024-02-01T10:47:27.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZDycMCht0nbZrtFjdxKubLJ6vzo=', 'Content-Length': '0'} 2024-02-01T10:47:27.941 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007d346164fefc9edd-0065bb76bf-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:27.984 DEBUG:boto:path=/vclfcn-61/obj4 2024-02-01T10:47:27.984 DEBUG:boto:auth_path=/vclfcn-61/obj4 2024-02-01T10:47:27.984 DEBUG:boto:Method: GET 2024-02-01T10:47:27.984 DEBUG:boto:Path: /vclfcn-61/obj4 2024-02-01T10:47:27.984 DEBUG:boto:Data: 2024-02-01T10:47:27.985 DEBUG:boto:Headers: {} 2024-02-01T10:47:27.985 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:27.985 DEBUG:boto:Port: 8000 2024-02-01T10:47:27.985 DEBUG:boto:Params: {} 2024-02-01T10:47:27.985 DEBUG:boto:Token: None 2024-02-01T10:47:27.985 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:27 GMT /vclfcn-61/obj4 2024-02-01T10:47:27.985 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZDycMCht0nbZrtFjdxKubLJ6vzo= 2024-02-01T10:47:27.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZDycMCht0nbZrtFjdxKubLJ6vzo=', 'Content-Length': '0'} 2024-02-01T10:47:27.988 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bdc5e876a12ce73f-0065bb76bf-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:27.989 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:47:27.989 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:47:27.989 DEBUG:boto:path=/vclfcn-61/?prefix=obj4 2024-02-01T10:47:27.989 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj4 2024-02-01T10:47:27.989 DEBUG:boto:Method: GET 2024-02-01T10:47:27.989 DEBUG:boto:Path: /vclfcn-61/?prefix=obj4 2024-02-01T10:47:27.989 DEBUG:boto:Data: 2024-02-01T10:47:27.989 DEBUG:boto:Headers: {} 2024-02-01T10:47:27.989 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:27.989 DEBUG:boto:Port: 8001 2024-02-01T10:47:27.989 DEBUG:boto:Params: {} 2024-02-01T10:47:27.990 DEBUG:boto:Token: None 2024-02-01T10:47:27.990 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:27 GMT /vclfcn-61/ 2024-02-01T10:47:27.990 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ww39+/07l7+WNAHvS5qWCYZMIDI= 2024-02-01T10:47:27.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ww39+/07l7+WNAHvS5qWCYZMIDI=', 'Content-Length': '0'} 2024-02-01T10:47:27.994 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e099dbd9a8139fa-0065bb76bf-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:27.995 DEBUG:boto:b'vclfcn-61obj41000falseobj42024-02-01T10:46:46.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:27.995 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:47:27.995 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:47:27.995 DEBUG:boto:path=/vclfcn-61/?prefix=obj4 2024-02-01T10:47:27.995 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj4 2024-02-01T10:47:27.996 DEBUG:boto:Method: GET 2024-02-01T10:47:27.996 DEBUG:boto:Path: /vclfcn-61/?prefix=obj4 2024-02-01T10:47:27.996 DEBUG:boto:Data: 2024-02-01T10:47:27.996 DEBUG:boto:Headers: {} 2024-02-01T10:47:27.996 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:27.996 DEBUG:boto:Port: 8000 2024-02-01T10:47:27.996 DEBUG:boto:Params: {} 2024-02-01T10:47:27.996 DEBUG:boto:Token: None 2024-02-01T10:47:27.996 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:27 GMT /vclfcn-61/ 2024-02-01T10:47:27.996 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ww39+/07l7+WNAHvS5qWCYZMIDI= 2024-02-01T10:47:27.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ww39+/07l7+WNAHvS5qWCYZMIDI=', 'Content-Length': '0'} 2024-02-01T10:47:28.001 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b08ee74429ca3af-0065bb76bf-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:28 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:28.001 DEBUG:boto:b'vclfcn-61obj41000falseobj42024-02-01T10:46:46.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:28.002 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-61 zones={test-zone3, test-zone1} 2024-02-01T10:47:28.002 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-61 zones={test-zone3, test-zone2} 2024-02-01T10:47:28.002 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:47:28.002 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:47:28.002 DEBUG:boto:Method: HEAD 2024-02-01T10:47:28.002 DEBUG:boto:Path: /vclfcn-61/ 2024-02-01T10:47:28.002 DEBUG:boto:Data: 2024-02-01T10:47:28.003 DEBUG:boto:Headers: {} 2024-02-01T10:47:28.003 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:28.003 DEBUG:boto:Port: 8001 2024-02-01T10:47:28.003 DEBUG:boto:Params: {} 2024-02-01T10:47:28.003 DEBUG:boto:Token: None 2024-02-01T10:47:28.003 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:28 GMT /vclfcn-61/ 2024-02-01T10:47:28.003 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/D7tYCRfGbId6p175w3eUyqew5g= 2024-02-01T10:47:28.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:28 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/D7tYCRfGbId6p175w3eUyqew5g=', 'Content-Length': '0'} 2024-02-01T10:47:28.007 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000fc6650f1510352a-0065bb76c0-5762-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:47:28 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:28.008 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:47:28.008 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:47:28.008 DEBUG:boto:Method: HEAD 2024-02-01T10:47:28.008 DEBUG:boto:Path: /vclfcn-61/ 2024-02-01T10:47:28.008 DEBUG:boto:Data: 2024-02-01T10:47:28.008 DEBUG:boto:Headers: {} 2024-02-01T10:47:28.008 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:28.008 DEBUG:boto:Port: 8000 2024-02-01T10:47:28.009 DEBUG:boto:Params: {} 2024-02-01T10:47:28.009 DEBUG:boto:Token: None 2024-02-01T10:47:28.009 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:28 GMT /vclfcn-61/ 2024-02-01T10:47:28.009 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/D7tYCRfGbId6p175w3eUyqew5g= 2024-02-01T10:47:28.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:28 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/D7tYCRfGbId6p175w3eUyqew5g=', 'Content-Length': '0'} 2024-02-01T10:47:28.014 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003b765035bb3f91e6-0065bb76c0-5350-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:47:28 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:28.014 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:47:28.014 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:47:28.014 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:47:28.014 DEBUG:boto:path=/vclfcn-61/?versions&max-keys=999 2024-02-01T10:47:28.014 DEBUG:boto:auth_path=/vclfcn-61/?versions&max-keys=999 2024-02-01T10:47:28.014 DEBUG:boto:Method: GET 2024-02-01T10:47:28.015 DEBUG:boto:Path: /vclfcn-61/?versions&max-keys=999 2024-02-01T10:47:28.015 DEBUG:boto:Data: 2024-02-01T10:47:28.015 DEBUG:boto:Headers: {} 2024-02-01T10:47:28.015 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:28.015 DEBUG:boto:Port: 8001 2024-02-01T10:47:28.015 DEBUG:boto:Params: {} 2024-02-01T10:47:28.015 DEBUG:boto:Token: None 2024-02-01T10:47:28.015 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:28 GMT /vclfcn-61/?versions 2024-02-01T10:47:28.015 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:h+KyHy72wC3JGxPZhf1qVfhnF/o= 2024-02-01T10:47:28.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:28 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:h+KyHy72wC3JGxPZhf1qVfhnF/o=', 'Content-Length': '0'} 2024-02-01T10:47:28.020 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d012be333cc279a8-0065bb76c0-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:28 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:28.020 DEBUG:boto:b'vclfcn-61999falseobj1nulltrue2024-02-01T10:46:46.359Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:46:46.380Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:46:46.433Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:46:46.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:28.021 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-02-01T10:47:28.021 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-02-01T10:47:28.021 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-02-01T10:47:28.021 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-02-01T10:47:28.021 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:47:28.021 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:47:28.022 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:47:28.022 DEBUG:boto:path=/vclfcn-61/?versions&max-keys=999 2024-02-01T10:47:28.022 DEBUG:boto:auth_path=/vclfcn-61/?versions&max-keys=999 2024-02-01T10:47:28.022 DEBUG:boto:Method: GET 2024-02-01T10:47:28.022 DEBUG:boto:Path: /vclfcn-61/?versions&max-keys=999 2024-02-01T10:47:28.022 DEBUG:boto:Data: 2024-02-01T10:47:28.022 DEBUG:boto:Headers: {} 2024-02-01T10:47:28.022 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:28.022 DEBUG:boto:Port: 8000 2024-02-01T10:47:28.022 DEBUG:boto:Params: {} 2024-02-01T10:47:28.022 DEBUG:boto:Token: None 2024-02-01T10:47:28.023 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:28 GMT /vclfcn-61/?versions 2024-02-01T10:47:28.023 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:h+KyHy72wC3JGxPZhf1qVfhnF/o= 2024-02-01T10:47:28.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:28 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:h+KyHy72wC3JGxPZhf1qVfhnF/o=', 'Content-Length': '0'} 2024-02-01T10:47:28.027 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006acea4a19dfbf0fb-0065bb76c0-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:28 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:28.028 DEBUG:boto:b'vclfcn-61999falseobj1nulltrue2024-02-01T10:46:46.359Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:46:46.380Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:46:46.433Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:46:46.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:28.028 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-02-01T10:47:28.028 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-02-01T10:47:28.028 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-02-01T10:47:28.028 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-02-01T10:47:28.028 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:47:28.029 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:47:28.029 DEBUG:boto:path=/vclfcn-61/?versions&max-keys=999 2024-02-01T10:47:28.029 DEBUG:boto:auth_path=/vclfcn-61/?versions&max-keys=999 2024-02-01T10:47:28.029 DEBUG:boto:Method: GET 2024-02-01T10:47:28.029 DEBUG:boto:Path: /vclfcn-61/?versions&max-keys=999 2024-02-01T10:47:28.029 DEBUG:boto:Data: 2024-02-01T10:47:28.029 DEBUG:boto:Headers: {} 2024-02-01T10:47:28.029 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:28.029 DEBUG:boto:Port: 8001 2024-02-01T10:47:28.029 DEBUG:boto:Params: {} 2024-02-01T10:47:28.030 DEBUG:boto:Token: None 2024-02-01T10:47:28.030 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:28 GMT /vclfcn-61/?versions 2024-02-01T10:47:28.030 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:h+KyHy72wC3JGxPZhf1qVfhnF/o= 2024-02-01T10:47:28.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:28 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:h+KyHy72wC3JGxPZhf1qVfhnF/o=', 'Content-Length': '0'} 2024-02-01T10:47:28.035 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008afb674c528a1a11-0065bb76c0-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:28 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:28.035 DEBUG:boto:b'vclfcn-61999falseobj1nulltrue2024-02-01T10:46:46.359Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:46:46.380Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:46:46.433Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:46:46.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:28.035 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:47:28.035 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:47:28.035 DEBUG:boto:path=/vclfcn-61/?versions&max-keys=999 2024-02-01T10:47:28.036 DEBUG:boto:auth_path=/vclfcn-61/?versions&max-keys=999 2024-02-01T10:47:28.036 DEBUG:boto:Method: GET 2024-02-01T10:47:28.036 DEBUG:boto:Path: /vclfcn-61/?versions&max-keys=999 2024-02-01T10:47:28.036 DEBUG:boto:Data: 2024-02-01T10:47:28.036 DEBUG:boto:Headers: {} 2024-02-01T10:47:28.036 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:28.036 DEBUG:boto:Port: 8000 2024-02-01T10:47:28.036 DEBUG:boto:Params: {} 2024-02-01T10:47:28.036 DEBUG:boto:Token: None 2024-02-01T10:47:28.036 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:28 GMT /vclfcn-61/?versions 2024-02-01T10:47:28.037 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:h+KyHy72wC3JGxPZhf1qVfhnF/o= 2024-02-01T10:47:28.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:28 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:h+KyHy72wC3JGxPZhf1qVfhnF/o=', 'Content-Length': '0'} 2024-02-01T10:47:28.041 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b948d69cd4e32ba0-0065bb76c0-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:28 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:28.041 DEBUG:boto:b'vclfcn-61999falseobj1nulltrue2024-02-01T10:46:46.359Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:46:46.380Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:46:46.433Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:46:46.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:28.042 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:47:28.042 DEBUG:boto:path=/vclfcn-61/obj1 2024-02-01T10:47:28.042 DEBUG:boto:auth_path=/vclfcn-61/obj1 2024-02-01T10:47:28.042 DEBUG:boto:path=/vclfcn-61/obj1?versionId=null 2024-02-01T10:47:28.042 DEBUG:boto:auth_path=/vclfcn-61/obj1?versionId=null 2024-02-01T10:47:28.042 DEBUG:boto:Method: GET 2024-02-01T10:47:28.042 DEBUG:boto:Path: /vclfcn-61/obj1?versionId=null 2024-02-01T10:47:28.042 DEBUG:boto:Data: 2024-02-01T10:47:28.042 DEBUG:boto:Headers: {} 2024-02-01T10:47:28.042 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:28.042 DEBUG:boto:Port: 8001 2024-02-01T10:47:28.043 DEBUG:boto:Params: {} 2024-02-01T10:47:28.043 DEBUG:boto:Token: None 2024-02-01T10:47:28.043 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:28 GMT /vclfcn-61/obj1?versionId=null 2024-02-01T10:47:28.043 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cIf0v2/NdY8fPiBR+QYeNZnSUmg= 2024-02-01T10:47:28.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:28 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cIf0v2/NdY8fPiBR+QYeNZnSUmg=', 'Content-Length': '0'} 2024-02-01T10:47:28.046 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000008223bb55e732b4f-0065bb76c0-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:28 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:28.088 DEBUG:boto:path=/vclfcn-61/obj1 2024-02-01T10:47:28.088 DEBUG:boto:auth_path=/vclfcn-61/obj1 2024-02-01T10:47:28.088 DEBUG:boto:path=/vclfcn-61/obj1?versionId=null 2024-02-01T10:47:28.088 DEBUG:boto:auth_path=/vclfcn-61/obj1?versionId=null 2024-02-01T10:47:28.088 DEBUG:boto:Method: GET 2024-02-01T10:47:28.089 DEBUG:boto:Path: /vclfcn-61/obj1?versionId=null 2024-02-01T10:47:28.089 DEBUG:boto:Data: 2024-02-01T10:47:28.089 DEBUG:boto:Headers: {} 2024-02-01T10:47:28.089 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:28.089 DEBUG:boto:Port: 8000 2024-02-01T10:47:28.089 DEBUG:boto:Params: {} 2024-02-01T10:47:28.089 DEBUG:boto:Token: None 2024-02-01T10:47:28.089 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:28 GMT /vclfcn-61/obj1?versionId=null 2024-02-01T10:47:28.089 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cIf0v2/NdY8fPiBR+QYeNZnSUmg= 2024-02-01T10:47:28.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:28 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cIf0v2/NdY8fPiBR+QYeNZnSUmg=', 'Content-Length': '0'} 2024-02-01T10:47:28.092 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002e29aae62a7fcb21-0065bb76c0-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:28 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:28.092 DEBUG:boto:path=/vclfcn-61/obj1 2024-02-01T10:47:28.093 DEBUG:boto:auth_path=/vclfcn-61/obj1 2024-02-01T10:47:28.093 DEBUG:boto:path=/vclfcn-61/obj1?versionId=null 2024-02-01T10:47:28.093 DEBUG:boto:auth_path=/vclfcn-61/obj1?versionId=null 2024-02-01T10:47:28.093 DEBUG:boto:Method: HEAD 2024-02-01T10:47:28.093 DEBUG:boto:Path: /vclfcn-61/obj1?versionId=null 2024-02-01T10:47:28.093 DEBUG:boto:Data: 2024-02-01T10:47:28.093 DEBUG:boto:Headers: {} 2024-02-01T10:47:28.093 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:28.093 DEBUG:boto:Port: 8001 2024-02-01T10:47:28.093 DEBUG:boto:Params: {} 2024-02-01T10:47:28.094 DEBUG:boto:Token: None 2024-02-01T10:47:28.094 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:28 GMT /vclfcn-61/obj1?versionId=null 2024-02-01T10:47:28.094 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WH1oYUN66SrBQjQAukpi5X/Btlw= 2024-02-01T10:47:28.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:28 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WH1oYUN66SrBQjQAukpi5X/Btlw=', 'Content-Length': '0'} 2024-02-01T10:47:28.101 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bafe293d9c9572eb-0065bb76c0-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:28 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:28.101 DEBUG:boto:path=/vclfcn-61/obj1 2024-02-01T10:47:28.102 DEBUG:boto:auth_path=/vclfcn-61/obj1 2024-02-01T10:47:28.102 DEBUG:boto:path=/vclfcn-61/obj1?versionId=null 2024-02-01T10:47:28.102 DEBUG:boto:auth_path=/vclfcn-61/obj1?versionId=null 2024-02-01T10:47:28.102 DEBUG:boto:Method: HEAD 2024-02-01T10:47:28.102 DEBUG:boto:Path: /vclfcn-61/obj1?versionId=null 2024-02-01T10:47:28.102 DEBUG:boto:Data: 2024-02-01T10:47:28.102 DEBUG:boto:Headers: {} 2024-02-01T10:47:28.102 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:28.102 DEBUG:boto:Port: 8000 2024-02-01T10:47:28.102 DEBUG:boto:Params: {} 2024-02-01T10:47:28.103 DEBUG:boto:Token: None 2024-02-01T10:47:28.103 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:28 GMT /vclfcn-61/obj1?versionId=null 2024-02-01T10:47:28.103 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WH1oYUN66SrBQjQAukpi5X/Btlw= 2024-02-01T10:47:28.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:28 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WH1oYUN66SrBQjQAukpi5X/Btlw=', 'Content-Length': '0'} 2024-02-01T10:47:28.106 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000077b3c588980db5fb-0065bb76c0-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:28 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:28.106 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:47:28.106 DEBUG:boto:path=/vclfcn-61/obj1 2024-02-01T10:47:28.106 DEBUG:boto:auth_path=/vclfcn-61/obj1 2024-02-01T10:47:28.106 DEBUG:boto:path=/vclfcn-61/obj1?versionId=null 2024-02-01T10:47:28.106 DEBUG:boto:auth_path=/vclfcn-61/obj1?versionId=null 2024-02-01T10:47:28.107 DEBUG:boto:Method: GET 2024-02-01T10:47:28.107 DEBUG:boto:Path: /vclfcn-61/obj1?versionId=null 2024-02-01T10:47:28.107 DEBUG:boto:Data: 2024-02-01T10:47:28.107 DEBUG:boto:Headers: {} 2024-02-01T10:47:28.107 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:28.107 DEBUG:boto:Port: 8001 2024-02-01T10:47:28.107 DEBUG:boto:Params: {} 2024-02-01T10:47:28.107 DEBUG:boto:Token: None 2024-02-01T10:47:28.107 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:28 GMT /vclfcn-61/obj1?versionId=null 2024-02-01T10:47:28.107 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cIf0v2/NdY8fPiBR+QYeNZnSUmg= 2024-02-01T10:47:28.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:28 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cIf0v2/NdY8fPiBR+QYeNZnSUmg=', 'Content-Length': '0'} 2024-02-01T10:47:28.110 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005ba58f943655e307-0065bb76c0-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:28 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:28.110 DEBUG:boto:path=/vclfcn-61/obj1 2024-02-01T10:47:28.110 DEBUG:boto:auth_path=/vclfcn-61/obj1 2024-02-01T10:47:28.111 DEBUG:boto:path=/vclfcn-61/obj1?versionId=null 2024-02-01T10:47:28.111 DEBUG:boto:auth_path=/vclfcn-61/obj1?versionId=null 2024-02-01T10:47:28.111 DEBUG:boto:Method: GET 2024-02-01T10:47:28.111 DEBUG:boto:Path: /vclfcn-61/obj1?versionId=null 2024-02-01T10:47:28.111 DEBUG:boto:Data: 2024-02-01T10:47:28.111 DEBUG:boto:Headers: {} 2024-02-01T10:47:28.111 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:28.111 DEBUG:boto:Port: 8000 2024-02-01T10:47:28.111 DEBUG:boto:Params: {} 2024-02-01T10:47:28.111 DEBUG:boto:Token: None 2024-02-01T10:47:28.112 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:28 GMT /vclfcn-61/obj1?versionId=null 2024-02-01T10:47:28.112 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cIf0v2/NdY8fPiBR+QYeNZnSUmg= 2024-02-01T10:47:28.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:28 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cIf0v2/NdY8fPiBR+QYeNZnSUmg=', 'Content-Length': '0'} 2024-02-01T10:47:28.115 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001eba8f09d91fd80f-0065bb76c0-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:28 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:28.115 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:47:28.115 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:47:28.115 DEBUG:boto:path=/vclfcn-61/?prefix=obj1 2024-02-01T10:47:28.116 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj1 2024-02-01T10:47:28.116 DEBUG:boto:Method: GET 2024-02-01T10:47:28.116 DEBUG:boto:Path: /vclfcn-61/?prefix=obj1 2024-02-01T10:47:28.116 DEBUG:boto:Data: 2024-02-01T10:47:28.116 DEBUG:boto:Headers: {} 2024-02-01T10:47:28.116 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:28.116 DEBUG:boto:Port: 8001 2024-02-01T10:47:28.116 DEBUG:boto:Params: {} 2024-02-01T10:47:28.116 DEBUG:boto:Token: None 2024-02-01T10:47:28.116 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:28 GMT /vclfcn-61/ 2024-02-01T10:47:28.117 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ruQAEcZvaFom2nDJQzldE6X8Llw= 2024-02-01T10:47:28.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:28 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ruQAEcZvaFom2nDJQzldE6X8Llw=', 'Content-Length': '0'} 2024-02-01T10:47:28.121 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e9de2e344bf7852-0065bb76c0-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:28 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:28.121 DEBUG:boto:b'vclfcn-61obj11000falseobj12024-02-01T10:46:46.359Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:28.122 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:47:28.122 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:47:28.122 DEBUG:boto:path=/vclfcn-61/?prefix=obj1 2024-02-01T10:47:28.122 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj1 2024-02-01T10:47:28.122 DEBUG:boto:Method: GET 2024-02-01T10:47:28.122 DEBUG:boto:Path: /vclfcn-61/?prefix=obj1 2024-02-01T10:47:28.123 DEBUG:boto:Data: 2024-02-01T10:47:28.123 DEBUG:boto:Headers: {} 2024-02-01T10:47:28.123 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:28.123 DEBUG:boto:Port: 8000 2024-02-01T10:47:28.123 DEBUG:boto:Params: {} 2024-02-01T10:47:28.123 DEBUG:boto:Token: None 2024-02-01T10:47:28.123 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:28 GMT /vclfcn-61/ 2024-02-01T10:47:28.123 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ruQAEcZvaFom2nDJQzldE6X8Llw= 2024-02-01T10:47:28.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:28 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ruQAEcZvaFom2nDJQzldE6X8Llw=', 'Content-Length': '0'} 2024-02-01T10:47:28.128 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d137decb912bf06a-0065bb76c0-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:28 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:28.128 DEBUG:boto:b'vclfcn-61obj11000falseobj12024-02-01T10:46:46.359Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:28.129 DEBUG:boto:path=/vclfcn-61/obj1 2024-02-01T10:47:28.129 DEBUG:boto:auth_path=/vclfcn-61/obj1 2024-02-01T10:47:28.129 DEBUG:boto:Method: HEAD 2024-02-01T10:47:28.129 DEBUG:boto:Path: /vclfcn-61/obj1 2024-02-01T10:47:28.129 DEBUG:boto:Data: 2024-02-01T10:47:28.129 DEBUG:boto:Headers: {} 2024-02-01T10:47:28.129 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:28.129 DEBUG:boto:Port: 8001 2024-02-01T10:47:28.129 DEBUG:boto:Params: {} 2024-02-01T10:47:28.130 DEBUG:boto:Token: None 2024-02-01T10:47:28.130 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:28 GMT /vclfcn-61/obj1 2024-02-01T10:47:28.130 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:EQ+K7XtXBTa/EfZHz33GkAuhD9c= 2024-02-01T10:47:28.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:28 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:EQ+K7XtXBTa/EfZHz33GkAuhD9c=', 'Content-Length': '0'} 2024-02-01T10:47:28.132 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004d15f59305048b28-0065bb76c0-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:28 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:28.133 DEBUG:boto:path=/vclfcn-61/obj1 2024-02-01T10:47:28.133 DEBUG:boto:auth_path=/vclfcn-61/obj1 2024-02-01T10:47:28.133 DEBUG:boto:Method: HEAD 2024-02-01T10:47:28.133 DEBUG:boto:Path: /vclfcn-61/obj1 2024-02-01T10:47:28.133 DEBUG:boto:Data: 2024-02-01T10:47:28.133 DEBUG:boto:Headers: {} 2024-02-01T10:47:28.133 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:28.133 DEBUG:boto:Port: 8000 2024-02-01T10:47:28.133 DEBUG:boto:Params: {} 2024-02-01T10:47:28.134 DEBUG:boto:Token: None 2024-02-01T10:47:28.134 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:28 GMT /vclfcn-61/obj1 2024-02-01T10:47:28.134 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:EQ+K7XtXBTa/EfZHz33GkAuhD9c= 2024-02-01T10:47:28.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:28 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:EQ+K7XtXBTa/EfZHz33GkAuhD9c=', 'Content-Length': '0'} 2024-02-01T10:47:28.137 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006381afeda0851176-0065bb76c0-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:28 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:28.137 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:47:28.137 DEBUG:boto:path=/vclfcn-61/obj1 2024-02-01T10:47:28.137 DEBUG:boto:auth_path=/vclfcn-61/obj1 2024-02-01T10:47:28.137 DEBUG:boto:Method: GET 2024-02-01T10:47:28.137 DEBUG:boto:Path: /vclfcn-61/obj1 2024-02-01T10:47:28.137 DEBUG:boto:Data: 2024-02-01T10:47:28.137 DEBUG:boto:Headers: {} 2024-02-01T10:47:28.137 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:28.138 DEBUG:boto:Port: 8001 2024-02-01T10:47:28.138 DEBUG:boto:Params: {} 2024-02-01T10:47:28.138 DEBUG:boto:Token: None 2024-02-01T10:47:28.138 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:28 GMT /vclfcn-61/obj1 2024-02-01T10:47:28.138 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:LLpocSTkJysqeE1zuPzyE2PeSys= 2024-02-01T10:47:28.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:28 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:LLpocSTkJysqeE1zuPzyE2PeSys=', 'Content-Length': '0'} 2024-02-01T10:47:28.141 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000065455b3bc3bd81d1-0065bb76c0-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:28 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:28.184 DEBUG:boto:path=/vclfcn-61/obj1 2024-02-01T10:47:28.184 DEBUG:boto:auth_path=/vclfcn-61/obj1 2024-02-01T10:47:28.184 DEBUG:boto:Method: GET 2024-02-01T10:47:28.184 DEBUG:boto:Path: /vclfcn-61/obj1 2024-02-01T10:47:28.184 DEBUG:boto:Data: 2024-02-01T10:47:28.185 DEBUG:boto:Headers: {} 2024-02-01T10:47:28.185 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:28.185 DEBUG:boto:Port: 8000 2024-02-01T10:47:28.185 DEBUG:boto:Params: {} 2024-02-01T10:47:28.185 DEBUG:boto:Token: None 2024-02-01T10:47:28.185 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:28 GMT /vclfcn-61/obj1 2024-02-01T10:47:28.185 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:LLpocSTkJysqeE1zuPzyE2PeSys= 2024-02-01T10:47:28.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:28 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:LLpocSTkJysqeE1zuPzyE2PeSys=', 'Content-Length': '0'} 2024-02-01T10:47:28.188 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000896426ad76c04f1f-0065bb76c0-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:28 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:28.236 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:47:28.236 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:47:28.236 DEBUG:boto:path=/vclfcn-61/?prefix=obj1 2024-02-01T10:47:28.236 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj1 2024-02-01T10:47:28.237 DEBUG:boto:Method: GET 2024-02-01T10:47:28.237 DEBUG:boto:Path: /vclfcn-61/?prefix=obj1 2024-02-01T10:47:28.237 DEBUG:boto:Data: 2024-02-01T10:47:28.237 DEBUG:boto:Headers: {} 2024-02-01T10:47:28.237 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:28.237 DEBUG:boto:Port: 8001 2024-02-01T10:47:28.237 DEBUG:boto:Params: {} 2024-02-01T10:47:28.237 DEBUG:boto:Token: None 2024-02-01T10:47:28.237 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:28 GMT /vclfcn-61/ 2024-02-01T10:47:28.238 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ruQAEcZvaFom2nDJQzldE6X8Llw= 2024-02-01T10:47:28.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:28 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ruQAEcZvaFom2nDJQzldE6X8Llw=', 'Content-Length': '0'} 2024-02-01T10:47:28.242 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b2bc4b8b0518bbb4-0065bb76c0-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:28 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:28.243 DEBUG:boto:b'vclfcn-61obj11000falseobj12024-02-01T10:46:46.359Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:28.243 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:47:28.243 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:47:28.243 DEBUG:boto:path=/vclfcn-61/?prefix=obj1 2024-02-01T10:47:28.243 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj1 2024-02-01T10:47:28.244 DEBUG:boto:Method: GET 2024-02-01T10:47:28.244 DEBUG:boto:Path: /vclfcn-61/?prefix=obj1 2024-02-01T10:47:28.244 DEBUG:boto:Data: 2024-02-01T10:47:28.244 DEBUG:boto:Headers: {} 2024-02-01T10:47:28.244 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:28.244 DEBUG:boto:Port: 8000 2024-02-01T10:47:28.244 DEBUG:boto:Params: {} 2024-02-01T10:47:28.244 DEBUG:boto:Token: None 2024-02-01T10:47:28.244 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:28 GMT /vclfcn-61/ 2024-02-01T10:47:28.244 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ruQAEcZvaFom2nDJQzldE6X8Llw= 2024-02-01T10:47:28.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:28 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ruQAEcZvaFom2nDJQzldE6X8Llw=', 'Content-Length': '0'} 2024-02-01T10:47:28.248 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b512e08d80601853-0065bb76c0-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:28 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:28.249 DEBUG:boto:b'vclfcn-61obj11000falseobj12024-02-01T10:46:46.359Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:28.249 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:47:28.249 DEBUG:boto:path=/vclfcn-61/obj2 2024-02-01T10:47:28.249 DEBUG:boto:auth_path=/vclfcn-61/obj2 2024-02-01T10:47:28.249 DEBUG:boto:path=/vclfcn-61/obj2?versionId=null 2024-02-01T10:47:28.249 DEBUG:boto:auth_path=/vclfcn-61/obj2?versionId=null 2024-02-01T10:47:28.250 DEBUG:boto:Method: GET 2024-02-01T10:47:28.250 DEBUG:boto:Path: /vclfcn-61/obj2?versionId=null 2024-02-01T10:47:28.250 DEBUG:boto:Data: 2024-02-01T10:47:28.250 DEBUG:boto:Headers: {} 2024-02-01T10:47:28.250 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:28.250 DEBUG:boto:Port: 8001 2024-02-01T10:47:28.250 DEBUG:boto:Params: {} 2024-02-01T10:47:28.250 DEBUG:boto:Token: None 2024-02-01T10:47:28.251 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:28 GMT /vclfcn-61/obj2?versionId=null 2024-02-01T10:47:28.251 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QeHzy5hDvx8qmw5UI9PZjsGaU08= 2024-02-01T10:47:28.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:28 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QeHzy5hDvx8qmw5UI9PZjsGaU08=', 'Content-Length': '0'} 2024-02-01T10:47:28.253 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000003fc658d4eecc77e-0065bb76c0-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:28 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:28.254 DEBUG:boto:path=/vclfcn-61/obj2 2024-02-01T10:47:28.254 DEBUG:boto:auth_path=/vclfcn-61/obj2 2024-02-01T10:47:28.254 DEBUG:boto:path=/vclfcn-61/obj2?versionId=null 2024-02-01T10:47:28.254 DEBUG:boto:auth_path=/vclfcn-61/obj2?versionId=null 2024-02-01T10:47:28.254 DEBUG:boto:Method: GET 2024-02-01T10:47:28.254 DEBUG:boto:Path: /vclfcn-61/obj2?versionId=null 2024-02-01T10:47:28.254 DEBUG:boto:Data: 2024-02-01T10:47:28.254 DEBUG:boto:Headers: {} 2024-02-01T10:47:28.255 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:28.255 DEBUG:boto:Port: 8000 2024-02-01T10:47:28.255 DEBUG:boto:Params: {} 2024-02-01T10:47:28.255 DEBUG:boto:Token: None 2024-02-01T10:47:28.255 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:28 GMT /vclfcn-61/obj2?versionId=null 2024-02-01T10:47:28.255 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QeHzy5hDvx8qmw5UI9PZjsGaU08= 2024-02-01T10:47:28.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:28 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QeHzy5hDvx8qmw5UI9PZjsGaU08=', 'Content-Length': '0'} 2024-02-01T10:47:28.258 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000def22b6e5e3fdd7-0065bb76c0-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:28 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:28.258 DEBUG:boto:path=/vclfcn-61/obj2 2024-02-01T10:47:28.258 DEBUG:boto:auth_path=/vclfcn-61/obj2 2024-02-01T10:47:28.258 DEBUG:boto:path=/vclfcn-61/obj2?versionId=null 2024-02-01T10:47:28.259 DEBUG:boto:auth_path=/vclfcn-61/obj2?versionId=null 2024-02-01T10:47:28.259 DEBUG:boto:Method: HEAD 2024-02-01T10:47:28.259 DEBUG:boto:Path: /vclfcn-61/obj2?versionId=null 2024-02-01T10:47:28.259 DEBUG:boto:Data: 2024-02-01T10:47:28.259 DEBUG:boto:Headers: {} 2024-02-01T10:47:28.259 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:28.259 DEBUG:boto:Port: 8001 2024-02-01T10:47:28.259 DEBUG:boto:Params: {} 2024-02-01T10:47:28.259 DEBUG:boto:Token: None 2024-02-01T10:47:28.260 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:28 GMT /vclfcn-61/obj2?versionId=null 2024-02-01T10:47:28.260 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:XWXEohlEnHHU1AaSFrHxszT8rac= 2024-02-01T10:47:28.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:28 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:XWXEohlEnHHU1AaSFrHxszT8rac=', 'Content-Length': '0'} 2024-02-01T10:47:28.262 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ee602e5c1f8a413d-0065bb76c0-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:28 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:28.263 DEBUG:boto:path=/vclfcn-61/obj2 2024-02-01T10:47:28.263 DEBUG:boto:auth_path=/vclfcn-61/obj2 2024-02-01T10:47:28.263 DEBUG:boto:path=/vclfcn-61/obj2?versionId=null 2024-02-01T10:47:28.263 DEBUG:boto:auth_path=/vclfcn-61/obj2?versionId=null 2024-02-01T10:47:28.263 DEBUG:boto:Method: HEAD 2024-02-01T10:47:28.263 DEBUG:boto:Path: /vclfcn-61/obj2?versionId=null 2024-02-01T10:47:28.263 DEBUG:boto:Data: 2024-02-01T10:47:28.263 DEBUG:boto:Headers: {} 2024-02-01T10:47:28.263 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:28.263 DEBUG:boto:Port: 8000 2024-02-01T10:47:28.264 DEBUG:boto:Params: {} 2024-02-01T10:47:28.264 DEBUG:boto:Token: None 2024-02-01T10:47:28.264 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:28 GMT /vclfcn-61/obj2?versionId=null 2024-02-01T10:47:28.264 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:XWXEohlEnHHU1AaSFrHxszT8rac= 2024-02-01T10:47:28.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:28 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:XWXEohlEnHHU1AaSFrHxszT8rac=', 'Content-Length': '0'} 2024-02-01T10:47:28.267 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dd3f0d2bb7fac626-0065bb76c0-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:28 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:28.267 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:47:28.267 DEBUG:boto:path=/vclfcn-61/obj2 2024-02-01T10:47:28.267 DEBUG:boto:auth_path=/vclfcn-61/obj2 2024-02-01T10:47:28.267 DEBUG:boto:path=/vclfcn-61/obj2?versionId=null 2024-02-01T10:47:28.267 DEBUG:boto:auth_path=/vclfcn-61/obj2?versionId=null 2024-02-01T10:47:28.268 DEBUG:boto:Method: GET 2024-02-01T10:47:28.268 DEBUG:boto:Path: /vclfcn-61/obj2?versionId=null 2024-02-01T10:47:28.268 DEBUG:boto:Data: 2024-02-01T10:47:28.268 DEBUG:boto:Headers: {} 2024-02-01T10:47:28.268 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:28.268 DEBUG:boto:Port: 8001 2024-02-01T10:47:28.268 DEBUG:boto:Params: {} 2024-02-01T10:47:28.268 DEBUG:boto:Token: None 2024-02-01T10:47:28.268 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:28 GMT /vclfcn-61/obj2?versionId=null 2024-02-01T10:47:28.269 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QeHzy5hDvx8qmw5UI9PZjsGaU08= 2024-02-01T10:47:28.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:28 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QeHzy5hDvx8qmw5UI9PZjsGaU08=', 'Content-Length': '0'} 2024-02-01T10:47:28.272 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000021aee83c3d667b30-0065bb76c0-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:28 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:28.272 DEBUG:boto:path=/vclfcn-61/obj2 2024-02-01T10:47:28.272 DEBUG:boto:auth_path=/vclfcn-61/obj2 2024-02-01T10:47:28.272 DEBUG:boto:path=/vclfcn-61/obj2?versionId=null 2024-02-01T10:47:28.272 DEBUG:boto:auth_path=/vclfcn-61/obj2?versionId=null 2024-02-01T10:47:28.272 DEBUG:boto:Method: GET 2024-02-01T10:47:28.272 DEBUG:boto:Path: /vclfcn-61/obj2?versionId=null 2024-02-01T10:47:28.272 DEBUG:boto:Data: 2024-02-01T10:47:28.273 DEBUG:boto:Headers: {} 2024-02-01T10:47:28.273 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:28.273 DEBUG:boto:Port: 8000 2024-02-01T10:47:28.273 DEBUG:boto:Params: {} 2024-02-01T10:47:28.273 DEBUG:boto:Token: None 2024-02-01T10:47:28.273 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:28 GMT /vclfcn-61/obj2?versionId=null 2024-02-01T10:47:28.273 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QeHzy5hDvx8qmw5UI9PZjsGaU08= 2024-02-01T10:47:28.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:28 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QeHzy5hDvx8qmw5UI9PZjsGaU08=', 'Content-Length': '0'} 2024-02-01T10:47:28.276 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fc039799afa17527-0065bb76c0-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:28 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:28.320 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:47:28.320 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:47:28.320 DEBUG:boto:path=/vclfcn-61/?prefix=obj2 2024-02-01T10:47:28.320 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj2 2024-02-01T10:47:28.321 DEBUG:boto:Method: GET 2024-02-01T10:47:28.321 DEBUG:boto:Path: /vclfcn-61/?prefix=obj2 2024-02-01T10:47:28.321 DEBUG:boto:Data: 2024-02-01T10:47:28.321 DEBUG:boto:Headers: {} 2024-02-01T10:47:28.321 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:28.321 DEBUG:boto:Port: 8001 2024-02-01T10:47:28.321 DEBUG:boto:Params: {} 2024-02-01T10:47:28.321 DEBUG:boto:Token: None 2024-02-01T10:47:28.321 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:28 GMT /vclfcn-61/ 2024-02-01T10:47:28.322 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ruQAEcZvaFom2nDJQzldE6X8Llw= 2024-02-01T10:47:28.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:28 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ruQAEcZvaFom2nDJQzldE6X8Llw=', 'Content-Length': '0'} 2024-02-01T10:47:28.326 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051635c60deffb11e-0065bb76c0-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:28 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:28.327 DEBUG:boto:b'vclfcn-61obj21000falseobj22024-02-01T10:46:46.380Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:28.327 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:47:28.327 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:47:28.327 DEBUG:boto:path=/vclfcn-61/?prefix=obj2 2024-02-01T10:47:28.328 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj2 2024-02-01T10:47:28.328 DEBUG:boto:Method: GET 2024-02-01T10:47:28.328 DEBUG:boto:Path: /vclfcn-61/?prefix=obj2 2024-02-01T10:47:28.328 DEBUG:boto:Data: 2024-02-01T10:47:28.328 DEBUG:boto:Headers: {} 2024-02-01T10:47:28.328 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:28.328 DEBUG:boto:Port: 8000 2024-02-01T10:47:28.328 DEBUG:boto:Params: {} 2024-02-01T10:47:28.328 DEBUG:boto:Token: None 2024-02-01T10:47:28.329 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:28 GMT /vclfcn-61/ 2024-02-01T10:47:28.329 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ruQAEcZvaFom2nDJQzldE6X8Llw= 2024-02-01T10:47:28.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:28 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ruQAEcZvaFom2nDJQzldE6X8Llw=', 'Content-Length': '0'} 2024-02-01T10:47:28.333 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f9e79653bef7084-0065bb76c0-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:28 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:28.333 DEBUG:boto:b'vclfcn-61obj21000falseobj22024-02-01T10:46:46.380Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:28.334 DEBUG:boto:path=/vclfcn-61/obj2 2024-02-01T10:47:28.334 DEBUG:boto:auth_path=/vclfcn-61/obj2 2024-02-01T10:47:28.334 DEBUG:boto:Method: HEAD 2024-02-01T10:47:28.334 DEBUG:boto:Path: /vclfcn-61/obj2 2024-02-01T10:47:28.334 DEBUG:boto:Data: 2024-02-01T10:47:28.334 DEBUG:boto:Headers: {} 2024-02-01T10:47:28.334 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:28.334 DEBUG:boto:Port: 8001 2024-02-01T10:47:28.335 DEBUG:boto:Params: {} 2024-02-01T10:47:28.335 DEBUG:boto:Token: None 2024-02-01T10:47:28.335 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:28 GMT /vclfcn-61/obj2 2024-02-01T10:47:28.335 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Qa+sAssl9hwABFA8En/hUdJxd60= 2024-02-01T10:47:28.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:28 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Qa+sAssl9hwABFA8En/hUdJxd60=', 'Content-Length': '0'} 2024-02-01T10:47:28.337 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000892a82d7e740b2ad-0065bb76c0-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:28 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:28.338 DEBUG:boto:path=/vclfcn-61/obj2 2024-02-01T10:47:28.338 DEBUG:boto:auth_path=/vclfcn-61/obj2 2024-02-01T10:47:28.338 DEBUG:boto:Method: HEAD 2024-02-01T10:47:28.338 DEBUG:boto:Path: /vclfcn-61/obj2 2024-02-01T10:47:28.338 DEBUG:boto:Data: 2024-02-01T10:47:28.338 DEBUG:boto:Headers: {} 2024-02-01T10:47:28.338 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:28.338 DEBUG:boto:Port: 8000 2024-02-01T10:47:28.338 DEBUG:boto:Params: {} 2024-02-01T10:47:28.339 DEBUG:boto:Token: None 2024-02-01T10:47:28.339 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:28 GMT /vclfcn-61/obj2 2024-02-01T10:47:28.339 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Qa+sAssl9hwABFA8En/hUdJxd60= 2024-02-01T10:47:28.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:28 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Qa+sAssl9hwABFA8En/hUdJxd60=', 'Content-Length': '0'} 2024-02-01T10:47:28.342 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000defdd033d6ec4040-0065bb76c0-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:28 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:28.342 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:47:28.342 DEBUG:boto:path=/vclfcn-61/obj2 2024-02-01T10:47:28.342 DEBUG:boto:auth_path=/vclfcn-61/obj2 2024-02-01T10:47:28.342 DEBUG:boto:Method: GET 2024-02-01T10:47:28.342 DEBUG:boto:Path: /vclfcn-61/obj2 2024-02-01T10:47:28.343 DEBUG:boto:Data: 2024-02-01T10:47:28.343 DEBUG:boto:Headers: {} 2024-02-01T10:47:28.343 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:28.343 DEBUG:boto:Port: 8001 2024-02-01T10:47:28.343 DEBUG:boto:Params: {} 2024-02-01T10:47:28.343 DEBUG:boto:Token: None 2024-02-01T10:47:28.343 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:28 GMT /vclfcn-61/obj2 2024-02-01T10:47:28.343 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8O66os9iYNYlVZwWDBfv0PzuWcU= 2024-02-01T10:47:28.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:28 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8O66os9iYNYlVZwWDBfv0PzuWcU=', 'Content-Length': '0'} 2024-02-01T10:47:28.346 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000343045cbc9ac49cc-0065bb76c0-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:28 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:28.388 DEBUG:boto:path=/vclfcn-61/obj2 2024-02-01T10:47:28.388 DEBUG:boto:auth_path=/vclfcn-61/obj2 2024-02-01T10:47:28.388 DEBUG:boto:Method: GET 2024-02-01T10:47:28.388 DEBUG:boto:Path: /vclfcn-61/obj2 2024-02-01T10:47:28.388 DEBUG:boto:Data: 2024-02-01T10:47:28.389 DEBUG:boto:Headers: {} 2024-02-01T10:47:28.389 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:28.389 DEBUG:boto:Port: 8000 2024-02-01T10:47:28.389 DEBUG:boto:Params: {} 2024-02-01T10:47:28.389 DEBUG:boto:Token: None 2024-02-01T10:47:28.389 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:28 GMT /vclfcn-61/obj2 2024-02-01T10:47:28.389 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8O66os9iYNYlVZwWDBfv0PzuWcU= 2024-02-01T10:47:28.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:28 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8O66os9iYNYlVZwWDBfv0PzuWcU=', 'Content-Length': '0'} 2024-02-01T10:47:28.392 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000883655bd2d96d6e4-0065bb76c0-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:28 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:28.432 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:47:28.432 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:47:28.432 DEBUG:boto:path=/vclfcn-61/?prefix=obj2 2024-02-01T10:47:28.433 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj2 2024-02-01T10:47:28.433 DEBUG:boto:Method: GET 2024-02-01T10:47:28.433 DEBUG:boto:Path: /vclfcn-61/?prefix=obj2 2024-02-01T10:47:28.433 DEBUG:boto:Data: 2024-02-01T10:47:28.433 DEBUG:boto:Headers: {} 2024-02-01T10:47:28.433 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:28.433 DEBUG:boto:Port: 8001 2024-02-01T10:47:28.433 DEBUG:boto:Params: {} 2024-02-01T10:47:28.433 DEBUG:boto:Token: None 2024-02-01T10:47:28.433 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:28 GMT /vclfcn-61/ 2024-02-01T10:47:28.434 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ruQAEcZvaFom2nDJQzldE6X8Llw= 2024-02-01T10:47:28.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:28 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ruQAEcZvaFom2nDJQzldE6X8Llw=', 'Content-Length': '0'} 2024-02-01T10:47:28.438 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c15a46f716757a8-0065bb76c0-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:28 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:28.439 DEBUG:boto:b'vclfcn-61obj21000falseobj22024-02-01T10:46:46.380Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:28.439 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:47:28.439 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:47:28.439 DEBUG:boto:path=/vclfcn-61/?prefix=obj2 2024-02-01T10:47:28.439 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj2 2024-02-01T10:47:28.439 DEBUG:boto:Method: GET 2024-02-01T10:47:28.440 DEBUG:boto:Path: /vclfcn-61/?prefix=obj2 2024-02-01T10:47:28.440 DEBUG:boto:Data: 2024-02-01T10:47:28.440 DEBUG:boto:Headers: {} 2024-02-01T10:47:28.440 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:28.440 DEBUG:boto:Port: 8000 2024-02-01T10:47:28.440 DEBUG:boto:Params: {} 2024-02-01T10:47:28.440 DEBUG:boto:Token: None 2024-02-01T10:47:28.440 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:28 GMT /vclfcn-61/ 2024-02-01T10:47:28.440 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ruQAEcZvaFom2nDJQzldE6X8Llw= 2024-02-01T10:47:28.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:28 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ruQAEcZvaFom2nDJQzldE6X8Llw=', 'Content-Length': '0'} 2024-02-01T10:47:28.445 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000048b0125d9c4c78ae-0065bb76c0-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:28 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:28.488 DEBUG:boto:b'vclfcn-61obj21000falseobj22024-02-01T10:46:46.380Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:28.488 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:47:28.489 DEBUG:boto:path=/vclfcn-61/obj3 2024-02-01T10:47:28.489 DEBUG:boto:auth_path=/vclfcn-61/obj3 2024-02-01T10:47:28.489 DEBUG:boto:path=/vclfcn-61/obj3?versionId=null 2024-02-01T10:47:28.489 DEBUG:boto:auth_path=/vclfcn-61/obj3?versionId=null 2024-02-01T10:47:28.489 DEBUG:boto:Method: GET 2024-02-01T10:47:28.489 DEBUG:boto:Path: /vclfcn-61/obj3?versionId=null 2024-02-01T10:47:28.489 DEBUG:boto:Data: 2024-02-01T10:47:28.489 DEBUG:boto:Headers: {} 2024-02-01T10:47:28.489 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:28.490 DEBUG:boto:Port: 8001 2024-02-01T10:47:28.490 DEBUG:boto:Params: {} 2024-02-01T10:47:28.490 DEBUG:boto:Token: None 2024-02-01T10:47:28.490 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:28 GMT /vclfcn-61/obj3?versionId=null 2024-02-01T10:47:28.490 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Zxea6D1/HSkjxl7mFmD75yv39qo= 2024-02-01T10:47:28.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:28 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Zxea6D1/HSkjxl7mFmD75yv39qo=', 'Content-Length': '0'} 2024-02-01T10:47:28.493 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c428cd8b94fd2685-0065bb76c0-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:28 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:28.493 DEBUG:boto:path=/vclfcn-61/obj3 2024-02-01T10:47:28.494 DEBUG:boto:auth_path=/vclfcn-61/obj3 2024-02-01T10:47:28.494 DEBUG:boto:path=/vclfcn-61/obj3?versionId=null 2024-02-01T10:47:28.494 DEBUG:boto:auth_path=/vclfcn-61/obj3?versionId=null 2024-02-01T10:47:28.494 DEBUG:boto:Method: GET 2024-02-01T10:47:28.494 DEBUG:boto:Path: /vclfcn-61/obj3?versionId=null 2024-02-01T10:47:28.494 DEBUG:boto:Data: 2024-02-01T10:47:28.494 DEBUG:boto:Headers: {} 2024-02-01T10:47:28.494 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:28.494 DEBUG:boto:Port: 8000 2024-02-01T10:47:28.494 DEBUG:boto:Params: {} 2024-02-01T10:47:28.494 DEBUG:boto:Token: None 2024-02-01T10:47:28.495 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:28 GMT /vclfcn-61/obj3?versionId=null 2024-02-01T10:47:28.495 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Zxea6D1/HSkjxl7mFmD75yv39qo= 2024-02-01T10:47:28.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:28 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Zxea6D1/HSkjxl7mFmD75yv39qo=', 'Content-Length': '0'} 2024-02-01T10:47:28.498 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000729d6ef6314bb2ea-0065bb76c0-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:28 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:28.498 DEBUG:boto:path=/vclfcn-61/obj3 2024-02-01T10:47:28.498 DEBUG:boto:auth_path=/vclfcn-61/obj3 2024-02-01T10:47:28.498 DEBUG:boto:path=/vclfcn-61/obj3?versionId=null 2024-02-01T10:47:28.498 DEBUG:boto:auth_path=/vclfcn-61/obj3?versionId=null 2024-02-01T10:47:28.498 DEBUG:boto:Method: HEAD 2024-02-01T10:47:28.498 DEBUG:boto:Path: /vclfcn-61/obj3?versionId=null 2024-02-01T10:47:28.498 DEBUG:boto:Data: 2024-02-01T10:47:28.499 DEBUG:boto:Headers: {} 2024-02-01T10:47:28.499 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:28.499 DEBUG:boto:Port: 8001 2024-02-01T10:47:28.499 DEBUG:boto:Params: {} 2024-02-01T10:47:28.499 DEBUG:boto:Token: None 2024-02-01T10:47:28.499 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:28 GMT /vclfcn-61/obj3?versionId=null 2024-02-01T10:47:28.499 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mocfS04lSbT0g5gZg9y6QV+ZRJE= 2024-02-01T10:47:28.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:28 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mocfS04lSbT0g5gZg9y6QV+ZRJE=', 'Content-Length': '0'} 2024-02-01T10:47:28.502 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000049d062f0f8df3ef9-0065bb76c0-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:28 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:28.502 DEBUG:boto:path=/vclfcn-61/obj3 2024-02-01T10:47:28.503 DEBUG:boto:auth_path=/vclfcn-61/obj3 2024-02-01T10:47:28.503 DEBUG:boto:path=/vclfcn-61/obj3?versionId=null 2024-02-01T10:47:28.503 DEBUG:boto:auth_path=/vclfcn-61/obj3?versionId=null 2024-02-01T10:47:28.503 DEBUG:boto:Method: HEAD 2024-02-01T10:47:28.503 DEBUG:boto:Path: /vclfcn-61/obj3?versionId=null 2024-02-01T10:47:28.503 DEBUG:boto:Data: 2024-02-01T10:47:28.503 DEBUG:boto:Headers: {} 2024-02-01T10:47:28.503 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:28.503 DEBUG:boto:Port: 8000 2024-02-01T10:47:28.503 DEBUG:boto:Params: {} 2024-02-01T10:47:28.504 DEBUG:boto:Token: None 2024-02-01T10:47:28.504 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:28 GMT /vclfcn-61/obj3?versionId=null 2024-02-01T10:47:28.504 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mocfS04lSbT0g5gZg9y6QV+ZRJE= 2024-02-01T10:47:28.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:28 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mocfS04lSbT0g5gZg9y6QV+ZRJE=', 'Content-Length': '0'} 2024-02-01T10:47:28.507 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001e04811d7b5fcb77-0065bb76c0-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:28 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:28.507 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:47:28.507 DEBUG:boto:path=/vclfcn-61/obj3 2024-02-01T10:47:28.507 DEBUG:boto:auth_path=/vclfcn-61/obj3 2024-02-01T10:47:28.507 DEBUG:boto:path=/vclfcn-61/obj3?versionId=null 2024-02-01T10:47:28.507 DEBUG:boto:auth_path=/vclfcn-61/obj3?versionId=null 2024-02-01T10:47:28.507 DEBUG:boto:Method: GET 2024-02-01T10:47:28.508 DEBUG:boto:Path: /vclfcn-61/obj3?versionId=null 2024-02-01T10:47:28.508 DEBUG:boto:Data: 2024-02-01T10:47:28.508 DEBUG:boto:Headers: {} 2024-02-01T10:47:28.508 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:28.508 DEBUG:boto:Port: 8001 2024-02-01T10:47:28.512 DEBUG:boto:Params: {} 2024-02-01T10:47:28.512 DEBUG:boto:Token: None 2024-02-01T10:47:28.512 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:28 GMT /vclfcn-61/obj3?versionId=null 2024-02-01T10:47:28.512 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Zxea6D1/HSkjxl7mFmD75yv39qo= 2024-02-01T10:47:28.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:28 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Zxea6D1/HSkjxl7mFmD75yv39qo=', 'Content-Length': '0'} 2024-02-01T10:47:28.515 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000029893e6d3c9f870-0065bb76c0-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:28 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:28.515 DEBUG:boto:path=/vclfcn-61/obj3 2024-02-01T10:47:28.515 DEBUG:boto:auth_path=/vclfcn-61/obj3 2024-02-01T10:47:28.516 DEBUG:boto:path=/vclfcn-61/obj3?versionId=null 2024-02-01T10:47:28.516 DEBUG:boto:auth_path=/vclfcn-61/obj3?versionId=null 2024-02-01T10:47:28.516 DEBUG:boto:Method: GET 2024-02-01T10:47:28.516 DEBUG:boto:Path: /vclfcn-61/obj3?versionId=null 2024-02-01T10:47:28.516 DEBUG:boto:Data: 2024-02-01T10:47:28.516 DEBUG:boto:Headers: {} 2024-02-01T10:47:28.516 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:28.516 DEBUG:boto:Port: 8000 2024-02-01T10:47:28.516 DEBUG:boto:Params: {} 2024-02-01T10:47:28.516 DEBUG:boto:Token: None 2024-02-01T10:47:28.517 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:28 GMT /vclfcn-61/obj3?versionId=null 2024-02-01T10:47:28.517 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Zxea6D1/HSkjxl7mFmD75yv39qo= 2024-02-01T10:47:28.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:28 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Zxea6D1/HSkjxl7mFmD75yv39qo=', 'Content-Length': '0'} 2024-02-01T10:47:28.519 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008e354423dd467bd1-0065bb76c0-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:28 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:28.564 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:47:28.564 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:47:28.564 DEBUG:boto:path=/vclfcn-61/?prefix=obj3 2024-02-01T10:47:28.565 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj3 2024-02-01T10:47:28.565 DEBUG:boto:Method: GET 2024-02-01T10:47:28.565 DEBUG:boto:Path: /vclfcn-61/?prefix=obj3 2024-02-01T10:47:28.565 DEBUG:boto:Data: 2024-02-01T10:47:28.565 DEBUG:boto:Headers: {} 2024-02-01T10:47:28.565 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:28.565 DEBUG:boto:Port: 8001 2024-02-01T10:47:28.565 DEBUG:boto:Params: {} 2024-02-01T10:47:28.565 DEBUG:boto:Token: None 2024-02-01T10:47:28.565 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:28 GMT /vclfcn-61/ 2024-02-01T10:47:28.566 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ruQAEcZvaFom2nDJQzldE6X8Llw= 2024-02-01T10:47:28.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:28 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ruQAEcZvaFom2nDJQzldE6X8Llw=', 'Content-Length': '0'} 2024-02-01T10:47:28.570 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a8b2f74e585fa4a-0065bb76c0-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:28 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:28.570 DEBUG:boto:b'vclfcn-61obj31000falseobj32024-02-01T10:46:46.433Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:28.571 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:47:28.571 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:47:28.571 DEBUG:boto:path=/vclfcn-61/?prefix=obj3 2024-02-01T10:47:28.571 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj3 2024-02-01T10:47:28.571 DEBUG:boto:Method: GET 2024-02-01T10:47:28.571 DEBUG:boto:Path: /vclfcn-61/?prefix=obj3 2024-02-01T10:47:28.572 DEBUG:boto:Data: 2024-02-01T10:47:28.572 DEBUG:boto:Headers: {} 2024-02-01T10:47:28.572 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:28.572 DEBUG:boto:Port: 8000 2024-02-01T10:47:28.572 DEBUG:boto:Params: {} 2024-02-01T10:47:28.572 DEBUG:boto:Token: None 2024-02-01T10:47:28.572 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:28 GMT /vclfcn-61/ 2024-02-01T10:47:28.572 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ruQAEcZvaFom2nDJQzldE6X8Llw= 2024-02-01T10:47:28.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:28 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ruQAEcZvaFom2nDJQzldE6X8Llw=', 'Content-Length': '0'} 2024-02-01T10:47:28.577 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089127c3794eaa6af-0065bb76c0-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:28 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:28.620 DEBUG:boto:b'vclfcn-61obj31000falseobj32024-02-01T10:46:46.433Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:28.620 DEBUG:boto:path=/vclfcn-61/obj3 2024-02-01T10:47:28.621 DEBUG:boto:auth_path=/vclfcn-61/obj3 2024-02-01T10:47:28.621 DEBUG:boto:Method: HEAD 2024-02-01T10:47:28.621 DEBUG:boto:Path: /vclfcn-61/obj3 2024-02-01T10:47:28.621 DEBUG:boto:Data: 2024-02-01T10:47:28.621 DEBUG:boto:Headers: {} 2024-02-01T10:47:28.621 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:28.621 DEBUG:boto:Port: 8001 2024-02-01T10:47:28.621 DEBUG:boto:Params: {} 2024-02-01T10:47:28.621 DEBUG:boto:Token: None 2024-02-01T10:47:28.622 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:28 GMT /vclfcn-61/obj3 2024-02-01T10:47:28.622 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:OTem7j0Wz18dnEuud+tT8H4i+4g= 2024-02-01T10:47:28.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:28 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:OTem7j0Wz18dnEuud+tT8H4i+4g=', 'Content-Length': '0'} 2024-02-01T10:47:28.624 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001935520492625e4e-0065bb76c0-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:28 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:28.625 DEBUG:boto:path=/vclfcn-61/obj3 2024-02-01T10:47:28.625 DEBUG:boto:auth_path=/vclfcn-61/obj3 2024-02-01T10:47:28.625 DEBUG:boto:Method: HEAD 2024-02-01T10:47:28.625 DEBUG:boto:Path: /vclfcn-61/obj3 2024-02-01T10:47:28.625 DEBUG:boto:Data: 2024-02-01T10:47:28.625 DEBUG:boto:Headers: {} 2024-02-01T10:47:28.626 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:28.626 DEBUG:boto:Port: 8000 2024-02-01T10:47:28.626 DEBUG:boto:Params: {} 2024-02-01T10:47:28.626 DEBUG:boto:Token: None 2024-02-01T10:47:28.626 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:28 GMT /vclfcn-61/obj3 2024-02-01T10:47:28.626 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:OTem7j0Wz18dnEuud+tT8H4i+4g= 2024-02-01T10:47:28.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:28 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:OTem7j0Wz18dnEuud+tT8H4i+4g=', 'Content-Length': '0'} 2024-02-01T10:47:28.629 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d3dd30096d88290d-0065bb76c0-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:28 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:28.629 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:47:28.629 DEBUG:boto:path=/vclfcn-61/obj3 2024-02-01T10:47:28.629 DEBUG:boto:auth_path=/vclfcn-61/obj3 2024-02-01T10:47:28.629 DEBUG:boto:Method: GET 2024-02-01T10:47:28.629 DEBUG:boto:Path: /vclfcn-61/obj3 2024-02-01T10:47:28.630 DEBUG:boto:Data: 2024-02-01T10:47:28.630 DEBUG:boto:Headers: {} 2024-02-01T10:47:28.630 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:28.630 DEBUG:boto:Port: 8001 2024-02-01T10:47:28.630 DEBUG:boto:Params: {} 2024-02-01T10:47:28.630 DEBUG:boto:Token: None 2024-02-01T10:47:28.630 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:28 GMT /vclfcn-61/obj3 2024-02-01T10:47:28.630 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:OCOaT6yHxCU6DepXqRUbWqVbvmM= 2024-02-01T10:47:28.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:28 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:OCOaT6yHxCU6DepXqRUbWqVbvmM=', 'Content-Length': '0'} 2024-02-01T10:47:28.633 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000534179b89237f4d3-0065bb76c0-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:28 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:28.676 DEBUG:boto:path=/vclfcn-61/obj3 2024-02-01T10:47:28.676 DEBUG:boto:auth_path=/vclfcn-61/obj3 2024-02-01T10:47:28.676 DEBUG:boto:Method: GET 2024-02-01T10:47:28.677 DEBUG:boto:Path: /vclfcn-61/obj3 2024-02-01T10:47:28.677 DEBUG:boto:Data: 2024-02-01T10:47:28.677 DEBUG:boto:Headers: {} 2024-02-01T10:47:28.677 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:28.677 DEBUG:boto:Port: 8000 2024-02-01T10:47:28.677 DEBUG:boto:Params: {} 2024-02-01T10:47:28.677 DEBUG:boto:Token: None 2024-02-01T10:47:28.677 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:28 GMT /vclfcn-61/obj3 2024-02-01T10:47:28.677 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:OCOaT6yHxCU6DepXqRUbWqVbvmM= 2024-02-01T10:47:28.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:28 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:OCOaT6yHxCU6DepXqRUbWqVbvmM=', 'Content-Length': '0'} 2024-02-01T10:47:28.680 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003920fcb164d89db4-0065bb76c0-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:28 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:28.728 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:47:28.728 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:47:28.728 DEBUG:boto:path=/vclfcn-61/?prefix=obj3 2024-02-01T10:47:28.729 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj3 2024-02-01T10:47:28.729 DEBUG:boto:Method: GET 2024-02-01T10:47:28.729 DEBUG:boto:Path: /vclfcn-61/?prefix=obj3 2024-02-01T10:47:28.729 DEBUG:boto:Data: 2024-02-01T10:47:28.729 DEBUG:boto:Headers: {} 2024-02-01T10:47:28.729 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:28.729 DEBUG:boto:Port: 8001 2024-02-01T10:47:28.729 DEBUG:boto:Params: {} 2024-02-01T10:47:28.729 DEBUG:boto:Token: None 2024-02-01T10:47:28.730 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:28 GMT /vclfcn-61/ 2024-02-01T10:47:28.730 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ruQAEcZvaFom2nDJQzldE6X8Llw= 2024-02-01T10:47:28.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:28 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ruQAEcZvaFom2nDJQzldE6X8Llw=', 'Content-Length': '0'} 2024-02-01T10:47:28.734 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2c6eee019c10507-0065bb76c0-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:28 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:28.735 DEBUG:boto:b'vclfcn-61obj31000falseobj32024-02-01T10:46:46.433Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:28.735 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:47:28.735 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:47:28.735 DEBUG:boto:path=/vclfcn-61/?prefix=obj3 2024-02-01T10:47:28.735 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj3 2024-02-01T10:47:28.735 DEBUG:boto:Method: GET 2024-02-01T10:47:28.735 DEBUG:boto:Path: /vclfcn-61/?prefix=obj3 2024-02-01T10:47:28.736 DEBUG:boto:Data: 2024-02-01T10:47:28.736 DEBUG:boto:Headers: {} 2024-02-01T10:47:28.736 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:28.736 DEBUG:boto:Port: 8000 2024-02-01T10:47:28.736 DEBUG:boto:Params: {} 2024-02-01T10:47:28.736 DEBUG:boto:Token: None 2024-02-01T10:47:28.736 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:28 GMT /vclfcn-61/ 2024-02-01T10:47:28.736 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ruQAEcZvaFom2nDJQzldE6X8Llw= 2024-02-01T10:47:28.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:28 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ruQAEcZvaFom2nDJQzldE6X8Llw=', 'Content-Length': '0'} 2024-02-01T10:47:28.741 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3a4d759a7dcc802-0065bb76c0-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:28 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:28.741 DEBUG:boto:b'vclfcn-61obj31000falseobj32024-02-01T10:46:46.433Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:28.741 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:47:28.741 DEBUG:boto:path=/vclfcn-61/obj4 2024-02-01T10:47:28.742 DEBUG:boto:auth_path=/vclfcn-61/obj4 2024-02-01T10:47:28.742 DEBUG:boto:path=/vclfcn-61/obj4?versionId=null 2024-02-01T10:47:28.742 DEBUG:boto:auth_path=/vclfcn-61/obj4?versionId=null 2024-02-01T10:47:28.742 DEBUG:boto:Method: GET 2024-02-01T10:47:28.742 DEBUG:boto:Path: /vclfcn-61/obj4?versionId=null 2024-02-01T10:47:28.742 DEBUG:boto:Data: 2024-02-01T10:47:28.742 DEBUG:boto:Headers: {} 2024-02-01T10:47:28.742 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:28.743 DEBUG:boto:Port: 8001 2024-02-01T10:47:28.743 DEBUG:boto:Params: {} 2024-02-01T10:47:28.743 DEBUG:boto:Token: None 2024-02-01T10:47:28.743 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:28 GMT /vclfcn-61/obj4?versionId=null 2024-02-01T10:47:28.743 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:TI5oWycInDRTHw2kLiC3QLHZ4ZM= 2024-02-01T10:47:28.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:28 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:TI5oWycInDRTHw2kLiC3QLHZ4ZM=', 'Content-Length': '0'} 2024-02-01T10:47:28.746 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f8a7230867414b25-0065bb76c0-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:28 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:28.746 DEBUG:boto:path=/vclfcn-61/obj4 2024-02-01T10:47:28.746 DEBUG:boto:auth_path=/vclfcn-61/obj4 2024-02-01T10:47:28.746 DEBUG:boto:path=/vclfcn-61/obj4?versionId=null 2024-02-01T10:47:28.746 DEBUG:boto:auth_path=/vclfcn-61/obj4?versionId=null 2024-02-01T10:47:28.747 DEBUG:boto:Method: GET 2024-02-01T10:47:28.747 DEBUG:boto:Path: /vclfcn-61/obj4?versionId=null 2024-02-01T10:47:28.747 DEBUG:boto:Data: 2024-02-01T10:47:28.747 DEBUG:boto:Headers: {} 2024-02-01T10:47:28.747 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:28.747 DEBUG:boto:Port: 8000 2024-02-01T10:47:28.747 DEBUG:boto:Params: {} 2024-02-01T10:47:28.747 DEBUG:boto:Token: None 2024-02-01T10:47:28.747 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:28 GMT /vclfcn-61/obj4?versionId=null 2024-02-01T10:47:28.747 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:TI5oWycInDRTHw2kLiC3QLHZ4ZM= 2024-02-01T10:47:28.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:28 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:TI5oWycInDRTHw2kLiC3QLHZ4ZM=', 'Content-Length': '0'} 2024-02-01T10:47:28.751 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ffec25ecd1af1627-0065bb76c0-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:28 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:28.792 DEBUG:boto:path=/vclfcn-61/obj4 2024-02-01T10:47:28.792 DEBUG:boto:auth_path=/vclfcn-61/obj4 2024-02-01T10:47:28.792 DEBUG:boto:path=/vclfcn-61/obj4?versionId=null 2024-02-01T10:47:28.792 DEBUG:boto:auth_path=/vclfcn-61/obj4?versionId=null 2024-02-01T10:47:28.793 DEBUG:boto:Method: HEAD 2024-02-01T10:47:28.793 DEBUG:boto:Path: /vclfcn-61/obj4?versionId=null 2024-02-01T10:47:28.793 DEBUG:boto:Data: 2024-02-01T10:47:28.793 DEBUG:boto:Headers: {} 2024-02-01T10:47:28.793 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:28.793 DEBUG:boto:Port: 8001 2024-02-01T10:47:28.793 DEBUG:boto:Params: {} 2024-02-01T10:47:28.793 DEBUG:boto:Token: None 2024-02-01T10:47:28.794 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:28 GMT /vclfcn-61/obj4?versionId=null 2024-02-01T10:47:28.794 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:aBXxbEsR5IV9GpX901sw2A7y2fM= 2024-02-01T10:47:28.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:28 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:aBXxbEsR5IV9GpX901sw2A7y2fM=', 'Content-Length': '0'} 2024-02-01T10:47:28.796 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d9cc91152363609c-0065bb76c0-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:28 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:28.797 DEBUG:boto:path=/vclfcn-61/obj4 2024-02-01T10:47:28.797 DEBUG:boto:auth_path=/vclfcn-61/obj4 2024-02-01T10:47:28.797 DEBUG:boto:path=/vclfcn-61/obj4?versionId=null 2024-02-01T10:47:28.797 DEBUG:boto:auth_path=/vclfcn-61/obj4?versionId=null 2024-02-01T10:47:28.797 DEBUG:boto:Method: HEAD 2024-02-01T10:47:28.797 DEBUG:boto:Path: /vclfcn-61/obj4?versionId=null 2024-02-01T10:47:28.797 DEBUG:boto:Data: 2024-02-01T10:47:28.797 DEBUG:boto:Headers: {} 2024-02-01T10:47:28.798 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:28.798 DEBUG:boto:Port: 8000 2024-02-01T10:47:28.798 DEBUG:boto:Params: {} 2024-02-01T10:47:28.798 DEBUG:boto:Token: None 2024-02-01T10:47:28.798 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:28 GMT /vclfcn-61/obj4?versionId=null 2024-02-01T10:47:28.798 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:aBXxbEsR5IV9GpX901sw2A7y2fM= 2024-02-01T10:47:28.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:28 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:aBXxbEsR5IV9GpX901sw2A7y2fM=', 'Content-Length': '0'} 2024-02-01T10:47:28.801 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000092d6ca1e5b897e4-0065bb76c0-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:28 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:28.801 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:47:28.801 DEBUG:boto:path=/vclfcn-61/obj4 2024-02-01T10:47:28.801 DEBUG:boto:auth_path=/vclfcn-61/obj4 2024-02-01T10:47:28.802 DEBUG:boto:path=/vclfcn-61/obj4?versionId=null 2024-02-01T10:47:28.802 DEBUG:boto:auth_path=/vclfcn-61/obj4?versionId=null 2024-02-01T10:47:28.802 DEBUG:boto:Method: GET 2024-02-01T10:47:28.802 DEBUG:boto:Path: /vclfcn-61/obj4?versionId=null 2024-02-01T10:47:28.802 DEBUG:boto:Data: 2024-02-01T10:47:28.802 DEBUG:boto:Headers: {} 2024-02-01T10:47:28.802 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:28.802 DEBUG:boto:Port: 8001 2024-02-01T10:47:28.802 DEBUG:boto:Params: {} 2024-02-01T10:47:28.802 DEBUG:boto:Token: None 2024-02-01T10:47:28.803 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:28 GMT /vclfcn-61/obj4?versionId=null 2024-02-01T10:47:28.803 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:TI5oWycInDRTHw2kLiC3QLHZ4ZM= 2024-02-01T10:47:28.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:28 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:TI5oWycInDRTHw2kLiC3QLHZ4ZM=', 'Content-Length': '0'} 2024-02-01T10:47:28.805 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e7136b3cbb80b425-0065bb76c0-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:28 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:28.806 DEBUG:boto:path=/vclfcn-61/obj4 2024-02-01T10:47:28.806 DEBUG:boto:auth_path=/vclfcn-61/obj4 2024-02-01T10:47:28.806 DEBUG:boto:path=/vclfcn-61/obj4?versionId=null 2024-02-01T10:47:28.806 DEBUG:boto:auth_path=/vclfcn-61/obj4?versionId=null 2024-02-01T10:47:28.806 DEBUG:boto:Method: GET 2024-02-01T10:47:28.806 DEBUG:boto:Path: /vclfcn-61/obj4?versionId=null 2024-02-01T10:47:28.806 DEBUG:boto:Data: 2024-02-01T10:47:28.806 DEBUG:boto:Headers: {} 2024-02-01T10:47:28.806 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:28.806 DEBUG:boto:Port: 8000 2024-02-01T10:47:28.807 DEBUG:boto:Params: {} 2024-02-01T10:47:28.807 DEBUG:boto:Token: None 2024-02-01T10:47:28.807 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:28 GMT /vclfcn-61/obj4?versionId=null 2024-02-01T10:47:28.807 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:TI5oWycInDRTHw2kLiC3QLHZ4ZM= 2024-02-01T10:47:28.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:28 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:TI5oWycInDRTHw2kLiC3QLHZ4ZM=', 'Content-Length': '0'} 2024-02-01T10:47:28.810 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000b52df072e155a49-0065bb76c0-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:28 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:28.852 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:47:28.852 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:47:28.852 DEBUG:boto:path=/vclfcn-61/?prefix=obj4 2024-02-01T10:47:28.852 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj4 2024-02-01T10:47:28.852 DEBUG:boto:Method: GET 2024-02-01T10:47:28.853 DEBUG:boto:Path: /vclfcn-61/?prefix=obj4 2024-02-01T10:47:28.853 DEBUG:boto:Data: 2024-02-01T10:47:28.853 DEBUG:boto:Headers: {} 2024-02-01T10:47:28.853 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:28.853 DEBUG:boto:Port: 8001 2024-02-01T10:47:28.853 DEBUG:boto:Params: {} 2024-02-01T10:47:28.853 DEBUG:boto:Token: None 2024-02-01T10:47:28.853 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:28 GMT /vclfcn-61/ 2024-02-01T10:47:28.853 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ruQAEcZvaFom2nDJQzldE6X8Llw= 2024-02-01T10:47:28.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:28 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ruQAEcZvaFom2nDJQzldE6X8Llw=', 'Content-Length': '0'} 2024-02-01T10:47:28.858 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000beb7fd9747f8f3f-0065bb76c0-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:28 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:28.858 DEBUG:boto:b'vclfcn-61obj41000falseobj42024-02-01T10:46:46.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:28.858 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:47:28.859 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:47:28.859 DEBUG:boto:path=/vclfcn-61/?prefix=obj4 2024-02-01T10:47:28.859 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj4 2024-02-01T10:47:28.859 DEBUG:boto:Method: GET 2024-02-01T10:47:28.859 DEBUG:boto:Path: /vclfcn-61/?prefix=obj4 2024-02-01T10:47:28.859 DEBUG:boto:Data: 2024-02-01T10:47:28.859 DEBUG:boto:Headers: {} 2024-02-01T10:47:28.859 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:28.859 DEBUG:boto:Port: 8000 2024-02-01T10:47:28.859 DEBUG:boto:Params: {} 2024-02-01T10:47:28.860 DEBUG:boto:Token: None 2024-02-01T10:47:28.860 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:28 GMT /vclfcn-61/ 2024-02-01T10:47:28.860 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ruQAEcZvaFom2nDJQzldE6X8Llw= 2024-02-01T10:47:28.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:28 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ruQAEcZvaFom2nDJQzldE6X8Llw=', 'Content-Length': '0'} 2024-02-01T10:47:28.864 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf12374c0d92f2cb-0065bb76c0-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:28 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:28.912 DEBUG:boto:b'vclfcn-61obj41000falseobj42024-02-01T10:46:46.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:28.912 DEBUG:boto:path=/vclfcn-61/obj4 2024-02-01T10:47:28.913 DEBUG:boto:auth_path=/vclfcn-61/obj4 2024-02-01T10:47:28.913 DEBUG:boto:Method: HEAD 2024-02-01T10:47:28.913 DEBUG:boto:Path: /vclfcn-61/obj4 2024-02-01T10:47:28.913 DEBUG:boto:Data: 2024-02-01T10:47:28.913 DEBUG:boto:Headers: {} 2024-02-01T10:47:28.913 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:28.913 DEBUG:boto:Port: 8001 2024-02-01T10:47:28.913 DEBUG:boto:Params: {} 2024-02-01T10:47:28.913 DEBUG:boto:Token: None 2024-02-01T10:47:28.914 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:28 GMT /vclfcn-61/obj4 2024-02-01T10:47:28.914 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8vlfmMdyyrg2YzAPaS9kqm6Gkq8= 2024-02-01T10:47:28.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:28 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8vlfmMdyyrg2YzAPaS9kqm6Gkq8=', 'Content-Length': '0'} 2024-02-01T10:47:28.917 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002a2f2ab6d47e6ad1-0065bb76c0-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:28 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:28.917 DEBUG:boto:path=/vclfcn-61/obj4 2024-02-01T10:47:28.917 DEBUG:boto:auth_path=/vclfcn-61/obj4 2024-02-01T10:47:28.917 DEBUG:boto:Method: HEAD 2024-02-01T10:47:28.917 DEBUG:boto:Path: /vclfcn-61/obj4 2024-02-01T10:47:28.917 DEBUG:boto:Data: 2024-02-01T10:47:28.918 DEBUG:boto:Headers: {} 2024-02-01T10:47:28.918 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:28.918 DEBUG:boto:Port: 8000 2024-02-01T10:47:28.918 DEBUG:boto:Params: {} 2024-02-01T10:47:28.918 DEBUG:boto:Token: None 2024-02-01T10:47:28.918 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:28 GMT /vclfcn-61/obj4 2024-02-01T10:47:28.918 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8vlfmMdyyrg2YzAPaS9kqm6Gkq8= 2024-02-01T10:47:28.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:28 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8vlfmMdyyrg2YzAPaS9kqm6Gkq8=', 'Content-Length': '0'} 2024-02-01T10:47:28.921 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fb2a03ee1a2c0235-0065bb76c0-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:28 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:28.921 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:47:28.922 DEBUG:boto:path=/vclfcn-61/obj4 2024-02-01T10:47:28.922 DEBUG:boto:auth_path=/vclfcn-61/obj4 2024-02-01T10:47:28.922 DEBUG:boto:Method: GET 2024-02-01T10:47:28.922 DEBUG:boto:Path: /vclfcn-61/obj4 2024-02-01T10:47:28.922 DEBUG:boto:Data: 2024-02-01T10:47:28.922 DEBUG:boto:Headers: {} 2024-02-01T10:47:28.922 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:28.922 DEBUG:boto:Port: 8001 2024-02-01T10:47:28.922 DEBUG:boto:Params: {} 2024-02-01T10:47:28.923 DEBUG:boto:Token: None 2024-02-01T10:47:28.923 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:28 GMT /vclfcn-61/obj4 2024-02-01T10:47:28.923 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mBTECShShngtUaplpBtdbRVdJnA= 2024-02-01T10:47:28.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:28 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mBTECShShngtUaplpBtdbRVdJnA=', 'Content-Length': '0'} 2024-02-01T10:47:28.925 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000087c0dee26fd0abc3-0065bb76c0-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:28 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:28.972 DEBUG:boto:path=/vclfcn-61/obj4 2024-02-01T10:47:28.972 DEBUG:boto:auth_path=/vclfcn-61/obj4 2024-02-01T10:47:28.972 DEBUG:boto:Method: GET 2024-02-01T10:47:28.972 DEBUG:boto:Path: /vclfcn-61/obj4 2024-02-01T10:47:28.973 DEBUG:boto:Data: 2024-02-01T10:47:28.973 DEBUG:boto:Headers: {} 2024-02-01T10:47:28.973 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:28.973 DEBUG:boto:Port: 8000 2024-02-01T10:47:28.973 DEBUG:boto:Params: {} 2024-02-01T10:47:28.973 DEBUG:boto:Token: None 2024-02-01T10:47:28.973 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:28 GMT /vclfcn-61/obj4 2024-02-01T10:47:28.973 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mBTECShShngtUaplpBtdbRVdJnA= 2024-02-01T10:47:28.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:28 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mBTECShShngtUaplpBtdbRVdJnA=', 'Content-Length': '0'} 2024-02-01T10:47:28.976 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002c1047b9116916f8-0065bb76c0-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:28 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:29.024 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:47:29.024 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:47:29.024 DEBUG:boto:path=/vclfcn-61/?prefix=obj4 2024-02-01T10:47:29.025 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj4 2024-02-01T10:47:29.025 DEBUG:boto:Method: GET 2024-02-01T10:47:29.025 DEBUG:boto:Path: /vclfcn-61/?prefix=obj4 2024-02-01T10:47:29.025 DEBUG:boto:Data: 2024-02-01T10:47:29.025 DEBUG:boto:Headers: {} 2024-02-01T10:47:29.025 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:29.025 DEBUG:boto:Port: 8001 2024-02-01T10:47:29.025 DEBUG:boto:Params: {} 2024-02-01T10:47:29.025 DEBUG:boto:Token: None 2024-02-01T10:47:29.026 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:29 GMT /vclfcn-61/ 2024-02-01T10:47:29.026 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:n9NNjuTpogNYFzVAC4tyWZGdb90= 2024-02-01T10:47:29.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:29 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:n9NNjuTpogNYFzVAC4tyWZGdb90=', 'Content-Length': '0'} 2024-02-01T10:47:29.031 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce37ba152057ed15-0065bb76c1-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:29 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:29.031 DEBUG:boto:b'vclfcn-61obj41000falseobj42024-02-01T10:46:46.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:29.032 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:47:29.032 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:47:29.032 DEBUG:boto:path=/vclfcn-61/?prefix=obj4 2024-02-01T10:47:29.032 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj4 2024-02-01T10:47:29.032 DEBUG:boto:Method: GET 2024-02-01T10:47:29.032 DEBUG:boto:Path: /vclfcn-61/?prefix=obj4 2024-02-01T10:47:29.032 DEBUG:boto:Data: 2024-02-01T10:47:29.032 DEBUG:boto:Headers: {} 2024-02-01T10:47:29.032 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:29.032 DEBUG:boto:Port: 8000 2024-02-01T10:47:29.033 DEBUG:boto:Params: {} 2024-02-01T10:47:29.033 DEBUG:boto:Token: None 2024-02-01T10:47:29.033 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:47:29 GMT /vclfcn-61/ 2024-02-01T10:47:29.033 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:n9NNjuTpogNYFzVAC4tyWZGdb90= 2024-02-01T10:47:29.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:29 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:n9NNjuTpogNYFzVAC4tyWZGdb90=', 'Content-Length': '0'} 2024-02-01T10:47:29.038 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c457f0fdc1da1ed-0065bb76c1-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:29 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:29.080 DEBUG:boto:b'vclfcn-61obj41000falseobj42024-02-01T10:46:46.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:47:29.080 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-61 zones={test-zone3, test-zone2} 2024-02-01T10:47:29.081 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'vclfcn-59', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:47:29.081 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'vclfcn-59', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:47:29.081 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket vclfcn-59 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-02-01T10:47:29.125 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:47:29.125 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:47:29.235 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'vclfcn-59', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:47:29.235 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'vclfcn-59', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:47:29.235 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket vclfcn-59 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-02-01T10:47:29.319 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:47:29.319 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:47:29.429 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'vclfcn-60', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:47:29.429 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'vclfcn-60', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:47:29.430 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket vclfcn-60 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-02-01T10:47:29.509 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:47:29.509 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:47:29.613 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'vclfcn-60', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:47:29.613 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'vclfcn-60', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:47:29.613 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket vclfcn-60 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-02-01T10:47:29.694 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:47:29.694 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:47:29.802 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'vclfcn-61', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:47:29.802 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'vclfcn-61', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:47:29.802 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket vclfcn-61 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-02-01T10:47:29.886 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:47:29.887 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:47:29.993 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'vclfcn-61', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:47:29.993 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'vclfcn-61', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:47:29.993 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket vclfcn-61 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-02-01T10:47:30.071 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:47:30.071 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:47:30.195 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:47:30.195 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:47:30.196 DEBUG:boto:Method: HEAD 2024-02-01T10:47:30.196 DEBUG:boto:Path: /vclfcn-59/ 2024-02-01T10:47:30.196 DEBUG:boto:Data: 2024-02-01T10:47:30.196 DEBUG:boto:Headers: {} 2024-02-01T10:47:30.196 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:30.196 DEBUG:boto:Port: 8000 2024-02-01T10:47:30.196 DEBUG:boto:Params: {} 2024-02-01T10:47:30.196 DEBUG:boto:Token: None 2024-02-01T10:47:30.197 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:30 GMT /vclfcn-59/ 2024-02-01T10:47:30.197 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:2qo0ZFfwDpVrVqRVc20724jpjJU= 2024-02-01T10:47:30.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:30 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2qo0ZFfwDpVrVqRVc20724jpjJU=', 'Content-Length': '0'} 2024-02-01T10:47:30.199 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009fd0648955b1dbb7-0065bb76c2-5765-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:30.200 DEBUG:boto:path=/vclfcn-59/obj5 2024-02-01T10:47:30.200 DEBUG:boto:auth_path=/vclfcn-59/obj5 2024-02-01T10:47:30.200 DEBUG:boto:Method: PUT 2024-02-01T10:47:30.200 DEBUG:boto:Path: /vclfcn-59/obj5 2024-02-01T10:47:30.200 DEBUG:boto:Data: 2024-02-01T10:47:30.200 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-02-01T10:47:30.200 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:30.200 DEBUG:boto:Port: 8000 2024-02-01T10:47:30.200 DEBUG:boto:Params: {} 2024-02-01T10:47:30.201 DEBUG:boto:Token: None 2024-02-01T10:47:30.201 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 01 Feb 2024 10:47:30 GMT /vclfcn-59/obj5 2024-02-01T10:47:30.201 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:r/6Guv0yKeIKc5DcQ1mBR1EdnYg= 2024-02-01T10:47:30.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:47:30 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:r/6Guv0yKeIKc5DcQ1mBR1EdnYg='} 2024-02-01T10:47:30.208 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fca9cb636c2c8a90-0065bb76c2-5765-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:30.208 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:47:30.208 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:47:30.208 DEBUG:boto:Method: HEAD 2024-02-01T10:47:30.208 DEBUG:boto:Path: /vclfcn-59/ 2024-02-01T10:47:30.208 DEBUG:boto:Data: 2024-02-01T10:47:30.208 DEBUG:boto:Headers: {} 2024-02-01T10:47:30.208 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:30.208 DEBUG:boto:Port: 8000 2024-02-01T10:47:30.209 DEBUG:boto:Params: {} 2024-02-01T10:47:30.209 DEBUG:boto:Token: None 2024-02-01T10:47:30.209 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:30 GMT /vclfcn-59/ 2024-02-01T10:47:30.209 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:2qo0ZFfwDpVrVqRVc20724jpjJU= 2024-02-01T10:47:30.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:30 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2qo0ZFfwDpVrVqRVc20724jpjJU=', 'Content-Length': '0'} 2024-02-01T10:47:30.211 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000bc7d7b2ab24fed4-0065bb76c2-5765-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:30.211 DEBUG:boto:path=/vclfcn-59/obj6 2024-02-01T10:47:30.211 DEBUG:boto:auth_path=/vclfcn-59/obj6 2024-02-01T10:47:30.211 DEBUG:boto:Method: PUT 2024-02-01T10:47:30.212 DEBUG:boto:Path: /vclfcn-59/obj6 2024-02-01T10:47:30.212 DEBUG:boto:Data: 2024-02-01T10:47:30.212 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-02-01T10:47:30.212 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:30.212 DEBUG:boto:Port: 8000 2024-02-01T10:47:30.212 DEBUG:boto:Params: {} 2024-02-01T10:47:30.212 DEBUG:boto:Token: None 2024-02-01T10:47:30.212 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 01 Feb 2024 10:47:30 GMT /vclfcn-59/obj6 2024-02-01T10:47:30.212 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:CjHv9qlUd7EvZCUraCNv9QDzri4= 2024-02-01T10:47:30.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:47:30 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:CjHv9qlUd7EvZCUraCNv9QDzri4='} 2024-02-01T10:47:30.217 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000060592c3efb6c70a2-0065bb76c2-5765-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:30.218 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:47:30.218 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:47:30.218 DEBUG:boto:Method: HEAD 2024-02-01T10:47:30.218 DEBUG:boto:Path: /vclfcn-59/ 2024-02-01T10:47:30.218 DEBUG:boto:Data: 2024-02-01T10:47:30.218 DEBUG:boto:Headers: {} 2024-02-01T10:47:30.218 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:30.218 DEBUG:boto:Port: 8000 2024-02-01T10:47:30.218 DEBUG:boto:Params: {} 2024-02-01T10:47:30.218 DEBUG:boto:Token: None 2024-02-01T10:47:30.219 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:30 GMT /vclfcn-59/ 2024-02-01T10:47:30.219 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:2qo0ZFfwDpVrVqRVc20724jpjJU= 2024-02-01T10:47:30.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:30 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2qo0ZFfwDpVrVqRVc20724jpjJU=', 'Content-Length': '0'} 2024-02-01T10:47:30.221 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c5a417809978e7dc-0065bb76c2-5765-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:30.221 DEBUG:boto:path=/vclfcn-59/obj7 2024-02-01T10:47:30.221 DEBUG:boto:auth_path=/vclfcn-59/obj7 2024-02-01T10:47:30.222 DEBUG:boto:Method: PUT 2024-02-01T10:47:30.222 DEBUG:boto:Path: /vclfcn-59/obj7 2024-02-01T10:47:30.222 DEBUG:boto:Data: 2024-02-01T10:47:30.222 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-02-01T10:47:30.222 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:30.222 DEBUG:boto:Port: 8000 2024-02-01T10:47:30.222 DEBUG:boto:Params: {} 2024-02-01T10:47:30.222 DEBUG:boto:Token: None 2024-02-01T10:47:30.222 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 01 Feb 2024 10:47:30 GMT /vclfcn-59/obj7 2024-02-01T10:47:30.222 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Q4SZzKfgti5v7zUj3UcP08cuw6Y= 2024-02-01T10:47:30.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:47:30 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Q4SZzKfgti5v7zUj3UcP08cuw6Y='} 2024-02-01T10:47:30.264 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000034ad1ef8d0e5184e-0065bb76c2-5765-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:30.264 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:47:30.265 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:47:30.265 DEBUG:boto:Method: HEAD 2024-02-01T10:47:30.265 DEBUG:boto:Path: /vclfcn-59/ 2024-02-01T10:47:30.265 DEBUG:boto:Data: 2024-02-01T10:47:30.265 DEBUG:boto:Headers: {} 2024-02-01T10:47:30.265 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:30.265 DEBUG:boto:Port: 8000 2024-02-01T10:47:30.265 DEBUG:boto:Params: {} 2024-02-01T10:47:30.265 DEBUG:boto:Token: None 2024-02-01T10:47:30.265 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:30 GMT /vclfcn-59/ 2024-02-01T10:47:30.266 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:2qo0ZFfwDpVrVqRVc20724jpjJU= 2024-02-01T10:47:30.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:30 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2qo0ZFfwDpVrVqRVc20724jpjJU=', 'Content-Length': '0'} 2024-02-01T10:47:30.269 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ae34306c8fe044bd-0065bb76c2-5765-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:30.269 DEBUG:boto:path=/vclfcn-59/obj8 2024-02-01T10:47:30.269 DEBUG:boto:auth_path=/vclfcn-59/obj8 2024-02-01T10:47:30.269 DEBUG:boto:Method: PUT 2024-02-01T10:47:30.270 DEBUG:boto:Path: /vclfcn-59/obj8 2024-02-01T10:47:30.270 DEBUG:boto:Data: 2024-02-01T10:47:30.270 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-02-01T10:47:30.270 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:47:30.270 DEBUG:boto:Port: 8000 2024-02-01T10:47:30.270 DEBUG:boto:Params: {} 2024-02-01T10:47:30.270 DEBUG:boto:Token: None 2024-02-01T10:47:30.270 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 01 Feb 2024 10:47:30 GMT /vclfcn-59/obj8 2024-02-01T10:47:30.270 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sy53JP3NNF3lKB1W2yd2i/zy1sQ= 2024-02-01T10:47:30.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:47:30 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sy53JP3NNF3lKB1W2yd2i/zy1sQ='} 2024-02-01T10:47:30.280 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a55222c6518c15ff-0065bb76c2-5765-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:30.280 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:47:30.280 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:47:30.280 DEBUG:boto:Method: HEAD 2024-02-01T10:47:30.280 DEBUG:boto:Path: /vclfcn-60/ 2024-02-01T10:47:30.280 DEBUG:boto:Data: 2024-02-01T10:47:30.281 DEBUG:boto:Headers: {} 2024-02-01T10:47:30.281 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:30.281 DEBUG:boto:Port: 8000 2024-02-01T10:47:30.281 DEBUG:boto:Params: {} 2024-02-01T10:47:30.281 DEBUG:boto:Token: None 2024-02-01T10:47:30.281 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:30 GMT /vclfcn-60/ 2024-02-01T10:47:30.281 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:iNG5VAYP31oMoKPE7zrdmCkUYY0= 2024-02-01T10:47:30.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:30 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iNG5VAYP31oMoKPE7zrdmCkUYY0=', 'Content-Length': '0'} 2024-02-01T10:47:30.285 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002d62de24b64195e6-0065bb76c2-5350-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:30.286 DEBUG:boto:path=/vclfcn-60/obj5 2024-02-01T10:47:30.286 DEBUG:boto:auth_path=/vclfcn-60/obj5 2024-02-01T10:47:30.286 DEBUG:boto:Method: PUT 2024-02-01T10:47:30.286 DEBUG:boto:Path: /vclfcn-60/obj5 2024-02-01T10:47:30.286 DEBUG:boto:Data: 2024-02-01T10:47:30.286 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-02-01T10:47:30.286 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:30.286 DEBUG:boto:Port: 8000 2024-02-01T10:47:30.287 DEBUG:boto:Params: {} 2024-02-01T10:47:30.287 DEBUG:boto:Token: None 2024-02-01T10:47:30.287 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 01 Feb 2024 10:47:30 GMT /vclfcn-60/obj5 2024-02-01T10:47:30.287 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:VdbD0injjXU2UkdEyMve41zmd4I= 2024-02-01T10:47:30.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:47:30 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:VdbD0injjXU2UkdEyMve41zmd4I='} 2024-02-01T10:47:30.332 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d8fa0cbf2cd11ce8-0065bb76c2-5350-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:30.332 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:47:30.333 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:47:30.333 DEBUG:boto:Method: HEAD 2024-02-01T10:47:30.333 DEBUG:boto:Path: /vclfcn-60/ 2024-02-01T10:47:30.333 DEBUG:boto:Data: 2024-02-01T10:47:30.333 DEBUG:boto:Headers: {} 2024-02-01T10:47:30.333 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:30.333 DEBUG:boto:Port: 8000 2024-02-01T10:47:30.333 DEBUG:boto:Params: {} 2024-02-01T10:47:30.333 DEBUG:boto:Token: None 2024-02-01T10:47:30.333 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:30 GMT /vclfcn-60/ 2024-02-01T10:47:30.334 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:iNG5VAYP31oMoKPE7zrdmCkUYY0= 2024-02-01T10:47:30.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:30 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iNG5VAYP31oMoKPE7zrdmCkUYY0=', 'Content-Length': '0'} 2024-02-01T10:47:30.338 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004ac13181782ee735-0065bb76c2-5350-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:30.338 DEBUG:boto:path=/vclfcn-60/obj6 2024-02-01T10:47:30.338 DEBUG:boto:auth_path=/vclfcn-60/obj6 2024-02-01T10:47:30.338 DEBUG:boto:Method: PUT 2024-02-01T10:47:30.338 DEBUG:boto:Path: /vclfcn-60/obj6 2024-02-01T10:47:30.338 DEBUG:boto:Data: 2024-02-01T10:47:30.339 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-02-01T10:47:30.339 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:30.339 DEBUG:boto:Port: 8000 2024-02-01T10:47:30.339 DEBUG:boto:Params: {} 2024-02-01T10:47:30.339 DEBUG:boto:Token: None 2024-02-01T10:47:30.339 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 01 Feb 2024 10:47:30 GMT /vclfcn-60/obj6 2024-02-01T10:47:30.339 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:3OhPzdxZebXt3qWgtqEkO3KRz6o= 2024-02-01T10:47:30.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:47:30 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:3OhPzdxZebXt3qWgtqEkO3KRz6o='} 2024-02-01T10:47:30.347 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000081b3efc791b8b6c0-0065bb76c2-5350-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:30.348 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:47:30.348 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:47:30.348 DEBUG:boto:Method: HEAD 2024-02-01T10:47:30.348 DEBUG:boto:Path: /vclfcn-60/ 2024-02-01T10:47:30.348 DEBUG:boto:Data: 2024-02-01T10:47:30.348 DEBUG:boto:Headers: {} 2024-02-01T10:47:30.348 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:30.348 DEBUG:boto:Port: 8000 2024-02-01T10:47:30.348 DEBUG:boto:Params: {} 2024-02-01T10:47:30.348 DEBUG:boto:Token: None 2024-02-01T10:47:30.349 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:30 GMT /vclfcn-60/ 2024-02-01T10:47:30.349 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:iNG5VAYP31oMoKPE7zrdmCkUYY0= 2024-02-01T10:47:30.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:30 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iNG5VAYP31oMoKPE7zrdmCkUYY0=', 'Content-Length': '0'} 2024-02-01T10:47:30.353 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000005ca5cd239fe924a-0065bb76c2-5350-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:30.353 DEBUG:boto:path=/vclfcn-60/obj7 2024-02-01T10:47:30.353 DEBUG:boto:auth_path=/vclfcn-60/obj7 2024-02-01T10:47:30.353 DEBUG:boto:Method: PUT 2024-02-01T10:47:30.353 DEBUG:boto:Path: /vclfcn-60/obj7 2024-02-01T10:47:30.353 DEBUG:boto:Data: 2024-02-01T10:47:30.353 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-02-01T10:47:30.354 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:30.354 DEBUG:boto:Port: 8000 2024-02-01T10:47:30.354 DEBUG:boto:Params: {} 2024-02-01T10:47:30.354 DEBUG:boto:Token: None 2024-02-01T10:47:30.354 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 01 Feb 2024 10:47:30 GMT /vclfcn-60/obj7 2024-02-01T10:47:30.354 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Bt054FbkvX0AJekwj+PrWUmKTaY= 2024-02-01T10:47:30.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:47:30 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Bt054FbkvX0AJekwj+PrWUmKTaY='} 2024-02-01T10:47:30.396 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000551ce40be30d6d5e-0065bb76c2-5350-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:30.397 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:47:30.397 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:47:30.397 DEBUG:boto:Method: HEAD 2024-02-01T10:47:30.397 DEBUG:boto:Path: /vclfcn-60/ 2024-02-01T10:47:30.397 DEBUG:boto:Data: 2024-02-01T10:47:30.397 DEBUG:boto:Headers: {} 2024-02-01T10:47:30.397 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:30.397 DEBUG:boto:Port: 8000 2024-02-01T10:47:30.397 DEBUG:boto:Params: {} 2024-02-01T10:47:30.397 DEBUG:boto:Token: None 2024-02-01T10:47:30.398 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:30 GMT /vclfcn-60/ 2024-02-01T10:47:30.398 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:iNG5VAYP31oMoKPE7zrdmCkUYY0= 2024-02-01T10:47:30.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:30 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iNG5VAYP31oMoKPE7zrdmCkUYY0=', 'Content-Length': '0'} 2024-02-01T10:47:30.402 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008ad3783057a3e976-0065bb76c2-5350-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:30.402 DEBUG:boto:path=/vclfcn-60/obj8 2024-02-01T10:47:30.402 DEBUG:boto:auth_path=/vclfcn-60/obj8 2024-02-01T10:47:30.403 DEBUG:boto:Method: PUT 2024-02-01T10:47:30.403 DEBUG:boto:Path: /vclfcn-60/obj8 2024-02-01T10:47:30.403 DEBUG:boto:Data: 2024-02-01T10:47:30.403 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-02-01T10:47:30.403 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:47:30.403 DEBUG:boto:Port: 8000 2024-02-01T10:47:30.403 DEBUG:boto:Params: {} 2024-02-01T10:47:30.403 DEBUG:boto:Token: None 2024-02-01T10:47:30.403 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 01 Feb 2024 10:47:30 GMT /vclfcn-60/obj8 2024-02-01T10:47:30.404 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:dMnxzIijZaa8fow6igfbY4GRW2c= 2024-02-01T10:47:30.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:47:30 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dMnxzIijZaa8fow6igfbY4GRW2c='} 2024-02-01T10:47:30.412 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f542e64aa1327ebe-0065bb76c2-5350-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:30.412 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:47:30.413 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:47:30.413 DEBUG:boto:Method: HEAD 2024-02-01T10:47:30.413 DEBUG:boto:Path: /vclfcn-61/ 2024-02-01T10:47:30.413 DEBUG:boto:Data: 2024-02-01T10:47:30.413 DEBUG:boto:Headers: {} 2024-02-01T10:47:30.413 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:30.413 DEBUG:boto:Port: 8001 2024-02-01T10:47:30.413 DEBUG:boto:Params: {} 2024-02-01T10:47:30.413 DEBUG:boto:Token: None 2024-02-01T10:47:30.413 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:30 GMT /vclfcn-61/ 2024-02-01T10:47:30.414 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:42a7RXZRqthU/558UdZ+tURi2zw= 2024-02-01T10:47:30.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:30 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:42a7RXZRqthU/558UdZ+tURi2zw=', 'Content-Length': '0'} 2024-02-01T10:47:30.418 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000102e08e23bda8992-0065bb76c2-5762-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:30.418 DEBUG:boto:path=/vclfcn-61/obj5 2024-02-01T10:47:30.418 DEBUG:boto:auth_path=/vclfcn-61/obj5 2024-02-01T10:47:30.419 DEBUG:boto:Method: PUT 2024-02-01T10:47:30.419 DEBUG:boto:Path: /vclfcn-61/obj5 2024-02-01T10:47:30.419 DEBUG:boto:Data: 2024-02-01T10:47:30.419 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-02-01T10:47:30.419 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:30.419 DEBUG:boto:Port: 8001 2024-02-01T10:47:30.419 DEBUG:boto:Params: {} 2024-02-01T10:47:30.419 DEBUG:boto:Token: None 2024-02-01T10:47:30.419 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 01 Feb 2024 10:47:30 GMT /vclfcn-61/obj5 2024-02-01T10:47:30.419 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:hK45rmdoCNnbVdY/dDeC1/P8OsQ= 2024-02-01T10:47:30.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:47:30 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:hK45rmdoCNnbVdY/dDeC1/P8OsQ='} 2024-02-01T10:47:30.428 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002d08c7ab82b72b90-0065bb76c2-5762-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:30.428 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:47:30.428 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:47:30.429 DEBUG:boto:Method: HEAD 2024-02-01T10:47:30.429 DEBUG:boto:Path: /vclfcn-61/ 2024-02-01T10:47:30.429 DEBUG:boto:Data: 2024-02-01T10:47:30.429 DEBUG:boto:Headers: {} 2024-02-01T10:47:30.429 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:30.429 DEBUG:boto:Port: 8001 2024-02-01T10:47:30.429 DEBUG:boto:Params: {} 2024-02-01T10:47:30.429 DEBUG:boto:Token: None 2024-02-01T10:47:30.430 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:30 GMT /vclfcn-61/ 2024-02-01T10:47:30.430 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:42a7RXZRqthU/558UdZ+tURi2zw= 2024-02-01T10:47:30.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:30 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:42a7RXZRqthU/558UdZ+tURi2zw=', 'Content-Length': '0'} 2024-02-01T10:47:30.433 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001c9557cf2f1aef73-0065bb76c2-5762-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:30.434 DEBUG:boto:path=/vclfcn-61/obj6 2024-02-01T10:47:30.434 DEBUG:boto:auth_path=/vclfcn-61/obj6 2024-02-01T10:47:30.434 DEBUG:boto:Method: PUT 2024-02-01T10:47:30.434 DEBUG:boto:Path: /vclfcn-61/obj6 2024-02-01T10:47:30.434 DEBUG:boto:Data: 2024-02-01T10:47:30.434 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-02-01T10:47:30.434 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:30.434 DEBUG:boto:Port: 8001 2024-02-01T10:47:30.434 DEBUG:boto:Params: {} 2024-02-01T10:47:30.435 DEBUG:boto:Token: None 2024-02-01T10:47:30.435 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 01 Feb 2024 10:47:30 GMT /vclfcn-61/obj6 2024-02-01T10:47:30.435 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fonGhRNitz8FwuRXarLaGGBGR/M= 2024-02-01T10:47:30.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:47:30 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fonGhRNitz8FwuRXarLaGGBGR/M='} 2024-02-01T10:47:30.480 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003507f0f6649a8472-0065bb76c2-5762-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:30.481 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:47:30.481 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:47:30.481 DEBUG:boto:Method: HEAD 2024-02-01T10:47:30.481 DEBUG:boto:Path: /vclfcn-61/ 2024-02-01T10:47:30.481 DEBUG:boto:Data: 2024-02-01T10:47:30.481 DEBUG:boto:Headers: {} 2024-02-01T10:47:30.481 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:30.481 DEBUG:boto:Port: 8001 2024-02-01T10:47:30.481 DEBUG:boto:Params: {} 2024-02-01T10:47:30.481 DEBUG:boto:Token: None 2024-02-01T10:47:30.482 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:30 GMT /vclfcn-61/ 2024-02-01T10:47:30.482 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:42a7RXZRqthU/558UdZ+tURi2zw= 2024-02-01T10:47:30.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:30 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:42a7RXZRqthU/558UdZ+tURi2zw=', 'Content-Length': '0'} 2024-02-01T10:47:30.486 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009fd912eb49706a14-0065bb76c2-5762-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:30.486 DEBUG:boto:path=/vclfcn-61/obj7 2024-02-01T10:47:30.486 DEBUG:boto:auth_path=/vclfcn-61/obj7 2024-02-01T10:47:30.486 DEBUG:boto:Method: PUT 2024-02-01T10:47:30.487 DEBUG:boto:Path: /vclfcn-61/obj7 2024-02-01T10:47:30.487 DEBUG:boto:Data: 2024-02-01T10:47:30.487 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-02-01T10:47:30.487 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:30.487 DEBUG:boto:Port: 8001 2024-02-01T10:47:30.487 DEBUG:boto:Params: {} 2024-02-01T10:47:30.487 DEBUG:boto:Token: None 2024-02-01T10:47:30.487 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 01 Feb 2024 10:47:30 GMT /vclfcn-61/obj7 2024-02-01T10:47:30.487 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lykYs6kjTcyIajeyB+7DqNA2Oy8= 2024-02-01T10:47:30.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:47:30 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lykYs6kjTcyIajeyB+7DqNA2Oy8='} 2024-02-01T10:47:30.496 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007dfc4da606ac2d74-0065bb76c2-5762-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:30.496 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:47:30.496 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:47:30.496 DEBUG:boto:Method: HEAD 2024-02-01T10:47:30.496 DEBUG:boto:Path: /vclfcn-61/ 2024-02-01T10:47:30.496 DEBUG:boto:Data: 2024-02-01T10:47:30.496 DEBUG:boto:Headers: {} 2024-02-01T10:47:30.496 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:30.497 DEBUG:boto:Port: 8001 2024-02-01T10:47:30.497 DEBUG:boto:Params: {} 2024-02-01T10:47:30.497 DEBUG:boto:Token: None 2024-02-01T10:47:30.497 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:47:30 GMT /vclfcn-61/ 2024-02-01T10:47:30.497 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:42a7RXZRqthU/558UdZ+tURi2zw= 2024-02-01T10:47:30.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:47:30 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:42a7RXZRqthU/558UdZ+tURi2zw=', 'Content-Length': '0'} 2024-02-01T10:47:30.501 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000790c5518cd1e28a2-0065bb76c2-5762-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:30.501 DEBUG:boto:path=/vclfcn-61/obj8 2024-02-01T10:47:30.501 DEBUG:boto:auth_path=/vclfcn-61/obj8 2024-02-01T10:47:30.502 DEBUG:boto:Method: PUT 2024-02-01T10:47:30.502 DEBUG:boto:Path: /vclfcn-61/obj8 2024-02-01T10:47:30.502 DEBUG:boto:Data: 2024-02-01T10:47:30.502 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-02-01T10:47:30.502 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:47:30.502 DEBUG:boto:Port: 8001 2024-02-01T10:47:30.502 DEBUG:boto:Params: {} 2024-02-01T10:47:30.502 DEBUG:boto:Token: None 2024-02-01T10:47:30.502 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 01 Feb 2024 10:47:30 GMT /vclfcn-61/obj8 2024-02-01T10:47:30.503 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ipgOIU+ImoycGAXE8On8ZK7mmSU= 2024-02-01T10:47:30.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:47:30 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ipgOIU+ImoycGAXE8On8ZK7mmSU='} 2024-02-01T10:47:30.548 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000073550f8edd8e6ae1-0065bb76c2-5762-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:47:30.549 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=vclfcn-59 2024-02-01T10:47:30.549 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:47:30.549 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:47:30.549 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket vclfcn-59 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-02-01T10:47:30.599 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:47:30.599 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:47:30.664 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:47:30.662+0000 7f0236fef980 1 waiting for incremental sync to catch up: 2024-02-01T10:47:30.664 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [00000000001.40.6, , 00000000001.40.6, , 00000000001.38.6, , , 00000000001.52.6, , , ] 2024-02-01T10:47:30.664 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [00000000004.50.6, 00000000002.45.1, 00000000004.44.6, 00000000002.45.1, 00000000004.48.6, 00000000002.81.1, 00000000002.46.1, 00000000004.60.6, 00000000002.32.1, 00000000002.46.1, 00000000002.65.1] 2024-02-01T10:47:35.667 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:47:35.666+0000 7f0236fef980 1 bucket sync caught up with source: 2024-02-01T10:47:35.667 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [00000000004.50.6, 00000000002.45.1, 00000000004.44.6, 00000000002.45.1, 00000000004.48.6, 00000000002.81.1, 00000000002.46.1, 00000000004.60.6, 00000000002.32.1, 00000000002.46.1, 00000000002.65.1] 2024-02-01T10:47:35.667 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [00000000004.50.6, 00000000002.45.1, 00000000004.44.6, 00000000002.45.1, 00000000004.48.6, 00000000002.81.1, 00000000002.46.1, 00000000004.60.6, 00000000002.32.1, 00000000002.46.1, 00000000002.65.1] 2024-02-01T10:47:35.667 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:47:35.666+0000 7f0236fef980 0 bucket checkpoint complete 2024-02-01T10:47:35.703 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=vclfcn-59 2024-02-01T10:47:35.703 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:47:35.703 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:47:35.703 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-59 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:47:35.758 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:47:35.758 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:47:35.823 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:47:35.825+0000 7fe3ac949980 1 waiting for incremental sync to catch up: 2024-02-01T10:47:35.824 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [00000000001.40.6, , 00000000001.40.6, , 00000000001.38.6, , , 00000000001.52.6, , , ] 2024-02-01T10:47:35.824 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [00000000004.50.6, 00000000002.45.1, 00000000004.44.6, 00000000002.45.1, 00000000004.48.6, 00000000002.81.1, 00000000002.46.1, 00000000004.60.6, 00000000002.32.1, 00000000002.46.1, 00000000002.65.1] 2024-02-01T10:47:40.826 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:47:40.825+0000 7fe3ac949980 1 waiting for incremental sync to catch up: 2024-02-01T10:47:40.826 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [00000000001.40.6, , 00000000001.40.6, , 00000000001.38.6, , , 00000000001.52.6, , , ] 2024-02-01T10:47:40.826 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [00000000004.50.6, 00000000002.45.1, 00000000004.44.6, 00000000002.45.1, 00000000004.48.6, 00000000002.81.1, 00000000002.46.1, 00000000004.60.6, 00000000002.32.1, 00000000002.46.1, 00000000002.65.1] 2024-02-01T10:47:45.828 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:47:45.829+0000 7fe3ac949980 1 bucket sync caught up with source: 2024-02-01T10:47:45.828 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [00000000004.50.6, 00000000002.45.1, 00000000004.44.6, 00000000002.45.1, 00000000004.48.6, 00000000002.81.1, 00000000002.46.1, 00000000004.60.6, 00000000002.32.1, 00000000002.46.1, 00000000002.65.1] 2024-02-01T10:47:45.828 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [00000000004.50.6, 00000000002.45.1, 00000000004.44.6, 00000000002.45.1, 00000000004.48.6, 00000000002.81.1, 00000000002.46.1, 00000000004.60.6, 00000000002.32.1, 00000000002.46.1, 00000000002.65.1] 2024-02-01T10:47:45.828 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:47:45.829+0000 7fe3ac949980 0 bucket checkpoint complete 2024-02-01T10:47:45.861 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=vclfcn-59 2024-02-01T10:47:45.862 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-59', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:47:45.862 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-59', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:47:45.862 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-59 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:47:45.905 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:47:45.905 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:47:45.966 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:47:45.965+0000 7eff529c4980 1 bucket sync caught up with source: 2024-02-01T10:47:45.967 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [00000000002.272.6, , 00000000002.205.6, , 00000000002.270.6, , , 00000000002.289.6, , , ] 2024-02-01T10:47:45.967 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [00000000002.272.6, , 00000000002.205.6, , 00000000002.270.6, , , 00000000002.289.6, , , ] 2024-02-01T10:47:45.967 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:47:45.965+0000 7eff529c4980 0 bucket checkpoint complete 2024-02-01T10:47:45.989 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=vclfcn-59 2024-02-01T10:47:45.990 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-59', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:47:45.990 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-59', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:47:45.990 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-59 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:47:46.075 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:47:46.076 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:47:46.135 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:47:46.137+0000 7f90df803980 1 bucket sync caught up with source: 2024-02-01T10:47:46.136 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [00000000002.272.6, , 00000000002.205.6, , 00000000002.270.6, , , 00000000002.289.6, , , ] 2024-02-01T10:47:46.136 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [00000000002.272.6, , 00000000002.205.6, , 00000000002.270.6, , , 00000000002.289.6, , , ] 2024-02-01T10:47:46.136 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:47:46.137+0000 7f90df803980 0 bucket checkpoint complete 2024-02-01T10:47:46.158 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=vclfcn-59 2024-02-01T10:47:46.158 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-59', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:47:46.158 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-59', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:47:46.158 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-59 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:47:46.199 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:47:46.199 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:47:46.259 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:47:46.257+0000 7f2a4bece980 1 waiting for incremental sync to catch up: 2024-02-01T10:47:46.259 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [00000000003.179.1, 00000000002.197.1, 00000000004.79.1, 00000000002.171.1, 00000000004.183.1, 00000000002.315.1, 00000000002.175.1, 00000000003.178.1, 00000000002.240.1, 00000000002.200.1, 00000000002.202.1] 2024-02-01T10:47:46.259 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [00000000004.223.6, 00000000002.197.1, 00000000005.91.6, 00000000002.171.1, 00000000005.222.6, 00000000002.315.1, 00000000002.175.1, 00000000004.223.6, 00000000002.240.1, 00000000002.200.1, 00000000002.202.1] 2024-02-01T10:47:51.262 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:47:51.261+0000 7f2a4bece980 1 waiting for incremental sync to catch up: 2024-02-01T10:47:51.262 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [00000000003.179.1, 00000000002.197.1, 00000000004.79.1, 00000000002.171.1, 00000000004.183.1, 00000000002.315.1, 00000000002.175.1, 00000000003.178.1, 00000000002.240.1, 00000000002.200.1, 00000000002.202.1] 2024-02-01T10:47:51.262 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [00000000004.223.6, 00000000002.197.1, 00000000005.91.6, 00000000002.171.1, 00000000005.222.6, 00000000002.315.1, 00000000002.175.1, 00000000004.223.6, 00000000002.240.1, 00000000002.200.1, 00000000002.202.1] 2024-02-01T10:47:56.264 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:47:56.265+0000 7f2a4bece980 1 waiting for incremental sync to catch up: 2024-02-01T10:47:56.264 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [00000000004.223.6, 00000000002.197.1, 00000000004.79.1, 00000000002.171.1, 00000000005.222.6, 00000000002.315.1, 00000000002.175.1, 00000000004.223.6, 00000000002.240.1, 00000000002.200.1, 00000000002.202.1] 2024-02-01T10:47:56.264 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [00000000004.223.6, 00000000002.197.1, 00000000005.91.6, 00000000002.171.1, 00000000005.222.6, 00000000002.315.1, 00000000002.175.1, 00000000004.223.6, 00000000002.240.1, 00000000002.200.1, 00000000002.202.1] 2024-02-01T10:48:01.266 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:48:01.265+0000 7f2a4bece980 1 bucket sync caught up with source: 2024-02-01T10:48:01.266 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [00000000004.223.6, 00000000002.197.1, 00000000005.91.6, 00000000002.171.1, 00000000005.222.6, 00000000002.315.1, 00000000002.175.1, 00000000004.223.6, 00000000002.240.1, 00000000002.200.1, 00000000002.202.1] 2024-02-01T10:48:01.266 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [00000000004.223.6, 00000000002.197.1, 00000000005.91.6, 00000000002.171.1, 00000000005.222.6, 00000000002.315.1, 00000000002.175.1, 00000000004.223.6, 00000000002.240.1, 00000000002.200.1, 00000000002.202.1] 2024-02-01T10:48:01.266 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:48:01.265+0000 7f2a4bece980 0 bucket checkpoint complete 2024-02-01T10:48:01.310 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=vclfcn-59 2024-02-01T10:48:01.310 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-59', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:48:01.310 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-59', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:48:01.311 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket vclfcn-59 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-02-01T10:48:01.361 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:48:01.361 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:48:01.425 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:48:01.422+0000 7f9edea13980 1 bucket sync caught up with source: 2024-02-01T10:48:01.425 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [00000000004.223.6, 00000000002.197.1, 00000000005.91.6, 00000000002.171.1, 00000000005.222.6, 00000000002.315.1, 00000000002.175.1, 00000000004.223.6, 00000000002.240.1, 00000000002.200.1, 00000000002.202.1] 2024-02-01T10:48:01.425 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [00000000004.223.6, 00000000002.197.1, 00000000005.91.6, 00000000002.171.1, 00000000005.222.6, 00000000002.315.1, 00000000002.175.1, 00000000004.223.6, 00000000002.240.1, 00000000002.200.1, 00000000002.202.1] 2024-02-01T10:48:01.425 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:48:01.422+0000 7f9edea13980 0 bucket checkpoint complete 2024-02-01T10:48:01.449 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-59 zones={test-zone2, test-zone1} 2024-02-01T10:48:01.449 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:01.449 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:01.449 DEBUG:boto:Method: HEAD 2024-02-01T10:48:01.449 DEBUG:boto:Path: /vclfcn-59/ 2024-02-01T10:48:01.449 DEBUG:boto:Data: 2024-02-01T10:48:01.449 DEBUG:boto:Headers: {} 2024-02-01T10:48:01.449 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:01.450 DEBUG:boto:Port: 8000 2024-02-01T10:48:01.450 DEBUG:boto:Params: {} 2024-02-01T10:48:01.450 DEBUG:boto:Token: None 2024-02-01T10:48:01.450 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:01 GMT /vclfcn-59/ 2024-02-01T10:48:01.450 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RA0Dj20BgDHCBp42Z9aH3s087Q8= 2024-02-01T10:48:01.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RA0Dj20BgDHCBp42Z9aH3s087Q8=', 'Content-Length': '0'} 2024-02-01T10:48:01.453 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000224ccd35b7365871-0065bb76e1-5350-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:01.453 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:01.453 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:01.453 DEBUG:boto:Method: HEAD 2024-02-01T10:48:01.453 DEBUG:boto:Path: /vclfcn-59/ 2024-02-01T10:48:01.453 DEBUG:boto:Data: 2024-02-01T10:48:01.454 DEBUG:boto:Headers: {} 2024-02-01T10:48:01.454 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:01.454 DEBUG:boto:Port: 8000 2024-02-01T10:48:01.454 DEBUG:boto:Params: {} 2024-02-01T10:48:01.454 DEBUG:boto:Token: None 2024-02-01T10:48:01.454 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:01 GMT /vclfcn-59/ 2024-02-01T10:48:01.454 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RA0Dj20BgDHCBp42Z9aH3s087Q8= 2024-02-01T10:48:01.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RA0Dj20BgDHCBp42Z9aH3s087Q8=', 'Content-Length': '0'} 2024-02-01T10:48:01.458 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000acdb2c131eeb34fc-0065bb76e1-5765-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:01.458 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:48:01.459 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:01.459 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:01.459 DEBUG:boto:path=/vclfcn-59/?versions&max-keys=999 2024-02-01T10:48:01.459 DEBUG:boto:auth_path=/vclfcn-59/?versions&max-keys=999 2024-02-01T10:48:01.459 DEBUG:boto:Method: GET 2024-02-01T10:48:01.459 DEBUG:boto:Path: /vclfcn-59/?versions&max-keys=999 2024-02-01T10:48:01.459 DEBUG:boto:Data: 2024-02-01T10:48:01.459 DEBUG:boto:Headers: {} 2024-02-01T10:48:01.459 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:01.459 DEBUG:boto:Port: 8000 2024-02-01T10:48:01.460 DEBUG:boto:Params: {} 2024-02-01T10:48:01.460 DEBUG:boto:Token: None 2024-02-01T10:48:01.460 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:01 GMT /vclfcn-59/?versions 2024-02-01T10:48:01.460 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:yQcwa5ZmtqZ11bYD74dqrU+wV/o= 2024-02-01T10:48:01.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:yQcwa5ZmtqZ11bYD74dqrU+wV/o=', 'Content-Length': '0'} 2024-02-01T10:48:01.462 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080542615a7d251e3-0065bb76e1-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:01.463 DEBUG:boto:b'vclfcn-59999falseobj1nulltrue2024-02-01T10:46:46.093Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:46:46.111Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:46:46.161Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:46:46.190Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-02-01T10:47:30.206Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-02-01T10:47:30.217Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-02-01T10:47:30.227Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-02-01T10:47:30.275Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:01.464 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-02-01T10:48:01.464 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-02-01T10:48:01.464 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-02-01T10:48:01.464 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-02-01T10:48:01.464 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-02-01T10:48:01.464 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-02-01T10:48:01.464 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-02-01T10:48:01.464 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-02-01T10:48:01.465 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:48:01.465 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:01.465 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:01.465 DEBUG:boto:path=/vclfcn-59/?versions&max-keys=999 2024-02-01T10:48:01.465 DEBUG:boto:auth_path=/vclfcn-59/?versions&max-keys=999 2024-02-01T10:48:01.465 DEBUG:boto:Method: GET 2024-02-01T10:48:01.465 DEBUG:boto:Path: /vclfcn-59/?versions&max-keys=999 2024-02-01T10:48:01.465 DEBUG:boto:Data: 2024-02-01T10:48:01.465 DEBUG:boto:Headers: {} 2024-02-01T10:48:01.465 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:01.465 DEBUG:boto:Port: 8000 2024-02-01T10:48:01.465 DEBUG:boto:Params: {} 2024-02-01T10:48:01.466 DEBUG:boto:Token: None 2024-02-01T10:48:01.466 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:01 GMT /vclfcn-59/?versions 2024-02-01T10:48:01.466 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:yQcwa5ZmtqZ11bYD74dqrU+wV/o= 2024-02-01T10:48:01.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:yQcwa5ZmtqZ11bYD74dqrU+wV/o=', 'Content-Length': '0'} 2024-02-01T10:48:01.470 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de2f135050813b29-0065bb76e1-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:01.512 DEBUG:boto:b'vclfcn-59999falseobj1nulltrue2024-02-01T10:46:46.093Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:46:46.111Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:46:46.161Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:46:46.190Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-02-01T10:47:30.206Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-02-01T10:47:30.217Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-02-01T10:47:30.227Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-02-01T10:47:30.275Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:01.513 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-02-01T10:48:01.513 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-02-01T10:48:01.513 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-02-01T10:48:01.513 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-02-01T10:48:01.513 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-02-01T10:48:01.513 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-02-01T10:48:01.514 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-02-01T10:48:01.514 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-02-01T10:48:01.514 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:01.514 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:01.514 DEBUG:boto:path=/vclfcn-59/?versions&max-keys=999 2024-02-01T10:48:01.514 DEBUG:boto:auth_path=/vclfcn-59/?versions&max-keys=999 2024-02-01T10:48:01.514 DEBUG:boto:Method: GET 2024-02-01T10:48:01.514 DEBUG:boto:Path: /vclfcn-59/?versions&max-keys=999 2024-02-01T10:48:01.514 DEBUG:boto:Data: 2024-02-01T10:48:01.515 DEBUG:boto:Headers: {} 2024-02-01T10:48:01.515 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:01.515 DEBUG:boto:Port: 8000 2024-02-01T10:48:01.515 DEBUG:boto:Params: {} 2024-02-01T10:48:01.515 DEBUG:boto:Token: None 2024-02-01T10:48:01.515 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:01 GMT /vclfcn-59/?versions 2024-02-01T10:48:01.515 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:yQcwa5ZmtqZ11bYD74dqrU+wV/o= 2024-02-01T10:48:01.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:yQcwa5ZmtqZ11bYD74dqrU+wV/o=', 'Content-Length': '0'} 2024-02-01T10:48:01.519 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018d914f044777a28-0065bb76e1-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:01.519 DEBUG:boto:b'vclfcn-59999falseobj1nulltrue2024-02-01T10:46:46.093Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:46:46.111Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:46:46.161Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:46:46.190Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-02-01T10:47:30.206Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-02-01T10:47:30.217Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-02-01T10:47:30.227Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-02-01T10:47:30.275Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:01.520 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:01.520 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:01.520 DEBUG:boto:path=/vclfcn-59/?versions&max-keys=999 2024-02-01T10:48:01.520 DEBUG:boto:auth_path=/vclfcn-59/?versions&max-keys=999 2024-02-01T10:48:01.520 DEBUG:boto:Method: GET 2024-02-01T10:48:01.520 DEBUG:boto:Path: /vclfcn-59/?versions&max-keys=999 2024-02-01T10:48:01.520 DEBUG:boto:Data: 2024-02-01T10:48:01.521 DEBUG:boto:Headers: {} 2024-02-01T10:48:01.521 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:01.521 DEBUG:boto:Port: 8000 2024-02-01T10:48:01.521 DEBUG:boto:Params: {} 2024-02-01T10:48:01.521 DEBUG:boto:Token: None 2024-02-01T10:48:01.521 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:01 GMT /vclfcn-59/?versions 2024-02-01T10:48:01.521 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:yQcwa5ZmtqZ11bYD74dqrU+wV/o= 2024-02-01T10:48:01.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:yQcwa5ZmtqZ11bYD74dqrU+wV/o=', 'Content-Length': '0'} 2024-02-01T10:48:01.525 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fee7d79c525d4057-0065bb76e1-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:01.526 DEBUG:boto:b'vclfcn-59999falseobj1nulltrue2024-02-01T10:46:46.093Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:46:46.111Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:46:46.161Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:46:46.190Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-02-01T10:47:30.206Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-02-01T10:47:30.217Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-02-01T10:47:30.227Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-02-01T10:47:30.275Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:01.526 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:48:01.526 DEBUG:boto:path=/vclfcn-59/obj1 2024-02-01T10:48:01.527 DEBUG:boto:auth_path=/vclfcn-59/obj1 2024-02-01T10:48:01.527 DEBUG:boto:path=/vclfcn-59/obj1?versionId=null 2024-02-01T10:48:01.527 DEBUG:boto:auth_path=/vclfcn-59/obj1?versionId=null 2024-02-01T10:48:01.527 DEBUG:boto:Method: GET 2024-02-01T10:48:01.527 DEBUG:boto:Path: /vclfcn-59/obj1?versionId=null 2024-02-01T10:48:01.527 DEBUG:boto:Data: 2024-02-01T10:48:01.527 DEBUG:boto:Headers: {} 2024-02-01T10:48:01.527 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:01.527 DEBUG:boto:Port: 8000 2024-02-01T10:48:01.527 DEBUG:boto:Params: {} 2024-02-01T10:48:01.528 DEBUG:boto:Token: None 2024-02-01T10:48:01.528 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:01 GMT /vclfcn-59/obj1?versionId=null 2024-02-01T10:48:01.528 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:CLX6NuLB9A/EcU/nFtGK9kcA3MU= 2024-02-01T10:48:01.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:CLX6NuLB9A/EcU/nFtGK9kcA3MU=', 'Content-Length': '0'} 2024-02-01T10:48:01.530 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000094c49383985152a6-0065bb76e1-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:01.531 DEBUG:boto:path=/vclfcn-59/obj1 2024-02-01T10:48:01.531 DEBUG:boto:auth_path=/vclfcn-59/obj1 2024-02-01T10:48:01.531 DEBUG:boto:path=/vclfcn-59/obj1?versionId=null 2024-02-01T10:48:01.531 DEBUG:boto:auth_path=/vclfcn-59/obj1?versionId=null 2024-02-01T10:48:01.531 DEBUG:boto:Method: GET 2024-02-01T10:48:01.531 DEBUG:boto:Path: /vclfcn-59/obj1?versionId=null 2024-02-01T10:48:01.531 DEBUG:boto:Data: 2024-02-01T10:48:01.531 DEBUG:boto:Headers: {} 2024-02-01T10:48:01.532 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:01.532 DEBUG:boto:Port: 8000 2024-02-01T10:48:01.532 DEBUG:boto:Params: {} 2024-02-01T10:48:01.532 DEBUG:boto:Token: None 2024-02-01T10:48:01.532 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:01 GMT /vclfcn-59/obj1?versionId=null 2024-02-01T10:48:01.532 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:CLX6NuLB9A/EcU/nFtGK9kcA3MU= 2024-02-01T10:48:01.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:CLX6NuLB9A/EcU/nFtGK9kcA3MU=', 'Content-Length': '0'} 2024-02-01T10:48:01.535 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007f86f943ee93a4b5-0065bb76e1-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:01.536 DEBUG:boto:path=/vclfcn-59/obj1 2024-02-01T10:48:01.536 DEBUG:boto:auth_path=/vclfcn-59/obj1 2024-02-01T10:48:01.536 DEBUG:boto:path=/vclfcn-59/obj1?versionId=null 2024-02-01T10:48:01.536 DEBUG:boto:auth_path=/vclfcn-59/obj1?versionId=null 2024-02-01T10:48:01.536 DEBUG:boto:Method: HEAD 2024-02-01T10:48:01.536 DEBUG:boto:Path: /vclfcn-59/obj1?versionId=null 2024-02-01T10:48:01.536 DEBUG:boto:Data: 2024-02-01T10:48:01.536 DEBUG:boto:Headers: {} 2024-02-01T10:48:01.536 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:01.536 DEBUG:boto:Port: 8000 2024-02-01T10:48:01.537 DEBUG:boto:Params: {} 2024-02-01T10:48:01.537 DEBUG:boto:Token: None 2024-02-01T10:48:01.537 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:01 GMT /vclfcn-59/obj1?versionId=null 2024-02-01T10:48:01.537 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:29X63Li77QalRKWhbDnulVi++Qs= 2024-02-01T10:48:01.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:29X63Li77QalRKWhbDnulVi++Qs=', 'Content-Length': '0'} 2024-02-01T10:48:01.539 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ae2e37fe80062aeb-0065bb76e1-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:01.540 DEBUG:boto:path=/vclfcn-59/obj1 2024-02-01T10:48:01.540 DEBUG:boto:auth_path=/vclfcn-59/obj1 2024-02-01T10:48:01.540 DEBUG:boto:path=/vclfcn-59/obj1?versionId=null 2024-02-01T10:48:01.540 DEBUG:boto:auth_path=/vclfcn-59/obj1?versionId=null 2024-02-01T10:48:01.540 DEBUG:boto:Method: HEAD 2024-02-01T10:48:01.540 DEBUG:boto:Path: /vclfcn-59/obj1?versionId=null 2024-02-01T10:48:01.540 DEBUG:boto:Data: 2024-02-01T10:48:01.540 DEBUG:boto:Headers: {} 2024-02-01T10:48:01.540 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:01.540 DEBUG:boto:Port: 8000 2024-02-01T10:48:01.540 DEBUG:boto:Params: {} 2024-02-01T10:48:01.541 DEBUG:boto:Token: None 2024-02-01T10:48:01.541 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:01 GMT /vclfcn-59/obj1?versionId=null 2024-02-01T10:48:01.541 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:29X63Li77QalRKWhbDnulVi++Qs= 2024-02-01T10:48:01.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:29X63Li77QalRKWhbDnulVi++Qs=', 'Content-Length': '0'} 2024-02-01T10:48:01.544 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000136a5980088ab490-0065bb76e1-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:01.544 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:48:01.544 DEBUG:boto:path=/vclfcn-59/obj1 2024-02-01T10:48:01.544 DEBUG:boto:auth_path=/vclfcn-59/obj1 2024-02-01T10:48:01.544 DEBUG:boto:path=/vclfcn-59/obj1?versionId=null 2024-02-01T10:48:01.545 DEBUG:boto:auth_path=/vclfcn-59/obj1?versionId=null 2024-02-01T10:48:01.545 DEBUG:boto:Method: GET 2024-02-01T10:48:01.545 DEBUG:boto:Path: /vclfcn-59/obj1?versionId=null 2024-02-01T10:48:01.545 DEBUG:boto:Data: 2024-02-01T10:48:01.545 DEBUG:boto:Headers: {} 2024-02-01T10:48:01.545 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:01.545 DEBUG:boto:Port: 8000 2024-02-01T10:48:01.545 DEBUG:boto:Params: {} 2024-02-01T10:48:01.545 DEBUG:boto:Token: None 2024-02-01T10:48:01.545 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:01 GMT /vclfcn-59/obj1?versionId=null 2024-02-01T10:48:01.546 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:CLX6NuLB9A/EcU/nFtGK9kcA3MU= 2024-02-01T10:48:01.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:CLX6NuLB9A/EcU/nFtGK9kcA3MU=', 'Content-Length': '0'} 2024-02-01T10:48:01.548 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e7b301019345cebb-0065bb76e1-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:01.592 DEBUG:boto:path=/vclfcn-59/obj1 2024-02-01T10:48:01.592 DEBUG:boto:auth_path=/vclfcn-59/obj1 2024-02-01T10:48:01.592 DEBUG:boto:path=/vclfcn-59/obj1?versionId=null 2024-02-01T10:48:01.593 DEBUG:boto:auth_path=/vclfcn-59/obj1?versionId=null 2024-02-01T10:48:01.593 DEBUG:boto:Method: GET 2024-02-01T10:48:01.593 DEBUG:boto:Path: /vclfcn-59/obj1?versionId=null 2024-02-01T10:48:01.593 DEBUG:boto:Data: 2024-02-01T10:48:01.593 DEBUG:boto:Headers: {} 2024-02-01T10:48:01.593 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:01.593 DEBUG:boto:Port: 8000 2024-02-01T10:48:01.593 DEBUG:boto:Params: {} 2024-02-01T10:48:01.593 DEBUG:boto:Token: None 2024-02-01T10:48:01.593 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:01 GMT /vclfcn-59/obj1?versionId=null 2024-02-01T10:48:01.594 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:CLX6NuLB9A/EcU/nFtGK9kcA3MU= 2024-02-01T10:48:01.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:CLX6NuLB9A/EcU/nFtGK9kcA3MU=', 'Content-Length': '0'} 2024-02-01T10:48:01.597 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000095d8a500ad119a89-0065bb76e1-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:01.597 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:01.597 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:01.597 DEBUG:boto:path=/vclfcn-59/?prefix=obj1 2024-02-01T10:48:01.597 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj1 2024-02-01T10:48:01.597 DEBUG:boto:Method: GET 2024-02-01T10:48:01.597 DEBUG:boto:Path: /vclfcn-59/?prefix=obj1 2024-02-01T10:48:01.598 DEBUG:boto:Data: 2024-02-01T10:48:01.598 DEBUG:boto:Headers: {} 2024-02-01T10:48:01.598 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:01.598 DEBUG:boto:Port: 8000 2024-02-01T10:48:01.598 DEBUG:boto:Params: {} 2024-02-01T10:48:01.598 DEBUG:boto:Token: None 2024-02-01T10:48:01.598 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:01 GMT /vclfcn-59/ 2024-02-01T10:48:01.598 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5Vy8Xp+v9QzdkDyVOw8noS7fwz8= 2024-02-01T10:48:01.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5Vy8Xp+v9QzdkDyVOw8noS7fwz8=', 'Content-Length': '0'} 2024-02-01T10:48:01.603 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000057071159c5ed71e1-0065bb76e1-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:01.603 DEBUG:boto:b'vclfcn-59obj11000falseobj12024-02-01T10:46:46.093Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:01.603 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:01.603 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:01.603 DEBUG:boto:path=/vclfcn-59/?prefix=obj1 2024-02-01T10:48:01.604 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj1 2024-02-01T10:48:01.604 DEBUG:boto:Method: GET 2024-02-01T10:48:01.604 DEBUG:boto:Path: /vclfcn-59/?prefix=obj1 2024-02-01T10:48:01.604 DEBUG:boto:Data: 2024-02-01T10:48:01.604 DEBUG:boto:Headers: {} 2024-02-01T10:48:01.604 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:01.604 DEBUG:boto:Port: 8000 2024-02-01T10:48:01.604 DEBUG:boto:Params: {} 2024-02-01T10:48:01.604 DEBUG:boto:Token: None 2024-02-01T10:48:01.604 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:01 GMT /vclfcn-59/ 2024-02-01T10:48:01.605 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5Vy8Xp+v9QzdkDyVOw8noS7fwz8= 2024-02-01T10:48:01.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5Vy8Xp+v9QzdkDyVOw8noS7fwz8=', 'Content-Length': '0'} 2024-02-01T10:48:01.609 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ae5913a1faf02c7-0065bb76e1-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:01.652 DEBUG:boto:b'vclfcn-59obj11000falseobj12024-02-01T10:46:46.093Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:01.652 DEBUG:boto:path=/vclfcn-59/obj1 2024-02-01T10:48:01.653 DEBUG:boto:auth_path=/vclfcn-59/obj1 2024-02-01T10:48:01.653 DEBUG:boto:Method: HEAD 2024-02-01T10:48:01.653 DEBUG:boto:Path: /vclfcn-59/obj1 2024-02-01T10:48:01.653 DEBUG:boto:Data: 2024-02-01T10:48:01.653 DEBUG:boto:Headers: {} 2024-02-01T10:48:01.653 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:01.653 DEBUG:boto:Port: 8000 2024-02-01T10:48:01.653 DEBUG:boto:Params: {} 2024-02-01T10:48:01.653 DEBUG:boto:Token: None 2024-02-01T10:48:01.654 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:01 GMT /vclfcn-59/obj1 2024-02-01T10:48:01.654 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:T6RLoq7T01R9XhQLYlFeD2UkpZ0= 2024-02-01T10:48:01.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:T6RLoq7T01R9XhQLYlFeD2UkpZ0=', 'Content-Length': '0'} 2024-02-01T10:48:01.657 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007e0a021e0dfc01d8-0065bb76e1-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:01.657 DEBUG:boto:path=/vclfcn-59/obj1 2024-02-01T10:48:01.657 DEBUG:boto:auth_path=/vclfcn-59/obj1 2024-02-01T10:48:01.657 DEBUG:boto:Method: HEAD 2024-02-01T10:48:01.657 DEBUG:boto:Path: /vclfcn-59/obj1 2024-02-01T10:48:01.657 DEBUG:boto:Data: 2024-02-01T10:48:01.658 DEBUG:boto:Headers: {} 2024-02-01T10:48:01.658 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:01.658 DEBUG:boto:Port: 8000 2024-02-01T10:48:01.658 DEBUG:boto:Params: {} 2024-02-01T10:48:01.658 DEBUG:boto:Token: None 2024-02-01T10:48:01.658 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:01 GMT /vclfcn-59/obj1 2024-02-01T10:48:01.658 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:T6RLoq7T01R9XhQLYlFeD2UkpZ0= 2024-02-01T10:48:01.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:T6RLoq7T01R9XhQLYlFeD2UkpZ0=', 'Content-Length': '0'} 2024-02-01T10:48:01.661 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000998bfc1c047b9d22-0065bb76e1-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:01.661 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:48:01.661 DEBUG:boto:path=/vclfcn-59/obj1 2024-02-01T10:48:01.661 DEBUG:boto:auth_path=/vclfcn-59/obj1 2024-02-01T10:48:01.661 DEBUG:boto:Method: GET 2024-02-01T10:48:01.662 DEBUG:boto:Path: /vclfcn-59/obj1 2024-02-01T10:48:01.662 DEBUG:boto:Data: 2024-02-01T10:48:01.662 DEBUG:boto:Headers: {} 2024-02-01T10:48:01.662 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:01.662 DEBUG:boto:Port: 8000 2024-02-01T10:48:01.662 DEBUG:boto:Params: {} 2024-02-01T10:48:01.662 DEBUG:boto:Token: None 2024-02-01T10:48:01.662 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:01 GMT /vclfcn-59/obj1 2024-02-01T10:48:01.662 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:AXSbbMt/OqnE91jhiF8+2dNa2D0= 2024-02-01T10:48:01.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:AXSbbMt/OqnE91jhiF8+2dNa2D0=', 'Content-Length': '0'} 2024-02-01T10:48:01.665 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f61c1c1e86878ae5-0065bb76e1-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:01.708 DEBUG:boto:path=/vclfcn-59/obj1 2024-02-01T10:48:01.708 DEBUG:boto:auth_path=/vclfcn-59/obj1 2024-02-01T10:48:01.708 DEBUG:boto:Method: GET 2024-02-01T10:48:01.708 DEBUG:boto:Path: /vclfcn-59/obj1 2024-02-01T10:48:01.709 DEBUG:boto:Data: 2024-02-01T10:48:01.709 DEBUG:boto:Headers: {} 2024-02-01T10:48:01.709 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:01.709 DEBUG:boto:Port: 8000 2024-02-01T10:48:01.709 DEBUG:boto:Params: {} 2024-02-01T10:48:01.709 DEBUG:boto:Token: None 2024-02-01T10:48:01.709 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:01 GMT /vclfcn-59/obj1 2024-02-01T10:48:01.709 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:AXSbbMt/OqnE91jhiF8+2dNa2D0= 2024-02-01T10:48:01.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:AXSbbMt/OqnE91jhiF8+2dNa2D0=', 'Content-Length': '0'} 2024-02-01T10:48:01.712 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000527f0f06dd6fdf5-0065bb76e1-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:01.712 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:01.713 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:01.713 DEBUG:boto:path=/vclfcn-59/?prefix=obj1 2024-02-01T10:48:01.713 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj1 2024-02-01T10:48:01.713 DEBUG:boto:Method: GET 2024-02-01T10:48:01.713 DEBUG:boto:Path: /vclfcn-59/?prefix=obj1 2024-02-01T10:48:01.713 DEBUG:boto:Data: 2024-02-01T10:48:01.713 DEBUG:boto:Headers: {} 2024-02-01T10:48:01.713 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:01.713 DEBUG:boto:Port: 8000 2024-02-01T10:48:01.714 DEBUG:boto:Params: {} 2024-02-01T10:48:01.714 DEBUG:boto:Token: None 2024-02-01T10:48:01.714 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:01 GMT /vclfcn-59/ 2024-02-01T10:48:01.714 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5Vy8Xp+v9QzdkDyVOw8noS7fwz8= 2024-02-01T10:48:01.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5Vy8Xp+v9QzdkDyVOw8noS7fwz8=', 'Content-Length': '0'} 2024-02-01T10:48:01.718 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b60352715028b132-0065bb76e1-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:01.719 DEBUG:boto:b'vclfcn-59obj11000falseobj12024-02-01T10:46:46.093Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:01.719 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:01.719 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:01.719 DEBUG:boto:path=/vclfcn-59/?prefix=obj1 2024-02-01T10:48:01.719 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj1 2024-02-01T10:48:01.719 DEBUG:boto:Method: GET 2024-02-01T10:48:01.720 DEBUG:boto:Path: /vclfcn-59/?prefix=obj1 2024-02-01T10:48:01.720 DEBUG:boto:Data: 2024-02-01T10:48:01.720 DEBUG:boto:Headers: {} 2024-02-01T10:48:01.720 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:01.720 DEBUG:boto:Port: 8000 2024-02-01T10:48:01.720 DEBUG:boto:Params: {} 2024-02-01T10:48:01.720 DEBUG:boto:Token: None 2024-02-01T10:48:01.720 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:01 GMT /vclfcn-59/ 2024-02-01T10:48:01.720 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5Vy8Xp+v9QzdkDyVOw8noS7fwz8= 2024-02-01T10:48:01.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5Vy8Xp+v9QzdkDyVOw8noS7fwz8=', 'Content-Length': '0'} 2024-02-01T10:48:01.724 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000437371c31296f726-0065bb76e1-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:01.724 DEBUG:boto:b'vclfcn-59obj11000falseobj12024-02-01T10:46:46.093Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:01.725 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:48:01.725 DEBUG:boto:path=/vclfcn-59/obj2 2024-02-01T10:48:01.725 DEBUG:boto:auth_path=/vclfcn-59/obj2 2024-02-01T10:48:01.725 DEBUG:boto:path=/vclfcn-59/obj2?versionId=null 2024-02-01T10:48:01.725 DEBUG:boto:auth_path=/vclfcn-59/obj2?versionId=null 2024-02-01T10:48:01.725 DEBUG:boto:Method: GET 2024-02-01T10:48:01.725 DEBUG:boto:Path: /vclfcn-59/obj2?versionId=null 2024-02-01T10:48:01.725 DEBUG:boto:Data: 2024-02-01T10:48:01.726 DEBUG:boto:Headers: {} 2024-02-01T10:48:01.726 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:01.726 DEBUG:boto:Port: 8000 2024-02-01T10:48:01.726 DEBUG:boto:Params: {} 2024-02-01T10:48:01.726 DEBUG:boto:Token: None 2024-02-01T10:48:01.726 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:01 GMT /vclfcn-59/obj2?versionId=null 2024-02-01T10:48:01.726 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:N7SRr7NFVfnjSvOc4rUb/cvrCJ0= 2024-02-01T10:48:01.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:N7SRr7NFVfnjSvOc4rUb/cvrCJ0=', 'Content-Length': '0'} 2024-02-01T10:48:01.729 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cbbe1cfb2dcc3307-0065bb76e1-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:01.729 DEBUG:boto:path=/vclfcn-59/obj2 2024-02-01T10:48:01.730 DEBUG:boto:auth_path=/vclfcn-59/obj2 2024-02-01T10:48:01.730 DEBUG:boto:path=/vclfcn-59/obj2?versionId=null 2024-02-01T10:48:01.730 DEBUG:boto:auth_path=/vclfcn-59/obj2?versionId=null 2024-02-01T10:48:01.730 DEBUG:boto:Method: GET 2024-02-01T10:48:01.730 DEBUG:boto:Path: /vclfcn-59/obj2?versionId=null 2024-02-01T10:48:01.730 DEBUG:boto:Data: 2024-02-01T10:48:01.730 DEBUG:boto:Headers: {} 2024-02-01T10:48:01.730 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:01.730 DEBUG:boto:Port: 8000 2024-02-01T10:48:01.730 DEBUG:boto:Params: {} 2024-02-01T10:48:01.730 DEBUG:boto:Token: None 2024-02-01T10:48:01.731 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:01 GMT /vclfcn-59/obj2?versionId=null 2024-02-01T10:48:01.731 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:N7SRr7NFVfnjSvOc4rUb/cvrCJ0= 2024-02-01T10:48:01.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:N7SRr7NFVfnjSvOc4rUb/cvrCJ0=', 'Content-Length': '0'} 2024-02-01T10:48:01.734 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d877c141da86119b-0065bb76e1-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:01.776 DEBUG:boto:path=/vclfcn-59/obj2 2024-02-01T10:48:01.776 DEBUG:boto:auth_path=/vclfcn-59/obj2 2024-02-01T10:48:01.776 DEBUG:boto:path=/vclfcn-59/obj2?versionId=null 2024-02-01T10:48:01.777 DEBUG:boto:auth_path=/vclfcn-59/obj2?versionId=null 2024-02-01T10:48:01.777 DEBUG:boto:Method: HEAD 2024-02-01T10:48:01.777 DEBUG:boto:Path: /vclfcn-59/obj2?versionId=null 2024-02-01T10:48:01.777 DEBUG:boto:Data: 2024-02-01T10:48:01.777 DEBUG:boto:Headers: {} 2024-02-01T10:48:01.777 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:01.777 DEBUG:boto:Port: 8000 2024-02-01T10:48:01.777 DEBUG:boto:Params: {} 2024-02-01T10:48:01.778 DEBUG:boto:Token: None 2024-02-01T10:48:01.778 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:01 GMT /vclfcn-59/obj2?versionId=null 2024-02-01T10:48:01.778 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:N8N0rD66o/557zMPkT8hVOiWKOA= 2024-02-01T10:48:01.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:N8N0rD66o/557zMPkT8hVOiWKOA=', 'Content-Length': '0'} 2024-02-01T10:48:01.781 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d815a40289321e5b-0065bb76e1-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:01.781 DEBUG:boto:path=/vclfcn-59/obj2 2024-02-01T10:48:01.781 DEBUG:boto:auth_path=/vclfcn-59/obj2 2024-02-01T10:48:01.781 DEBUG:boto:path=/vclfcn-59/obj2?versionId=null 2024-02-01T10:48:01.781 DEBUG:boto:auth_path=/vclfcn-59/obj2?versionId=null 2024-02-01T10:48:01.781 DEBUG:boto:Method: HEAD 2024-02-01T10:48:01.782 DEBUG:boto:Path: /vclfcn-59/obj2?versionId=null 2024-02-01T10:48:01.782 DEBUG:boto:Data: 2024-02-01T10:48:01.782 DEBUG:boto:Headers: {} 2024-02-01T10:48:01.782 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:01.782 DEBUG:boto:Port: 8000 2024-02-01T10:48:01.782 DEBUG:boto:Params: {} 2024-02-01T10:48:01.782 DEBUG:boto:Token: None 2024-02-01T10:48:01.782 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:01 GMT /vclfcn-59/obj2?versionId=null 2024-02-01T10:48:01.782 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:N8N0rD66o/557zMPkT8hVOiWKOA= 2024-02-01T10:48:01.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:N8N0rD66o/557zMPkT8hVOiWKOA=', 'Content-Length': '0'} 2024-02-01T10:48:01.787 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e6e5df2a5fa8bf1d-0065bb76e1-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:01.787 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:48:01.787 DEBUG:boto:path=/vclfcn-59/obj2 2024-02-01T10:48:01.787 DEBUG:boto:auth_path=/vclfcn-59/obj2 2024-02-01T10:48:01.787 DEBUG:boto:path=/vclfcn-59/obj2?versionId=null 2024-02-01T10:48:01.787 DEBUG:boto:auth_path=/vclfcn-59/obj2?versionId=null 2024-02-01T10:48:01.787 DEBUG:boto:Method: GET 2024-02-01T10:48:01.788 DEBUG:boto:Path: /vclfcn-59/obj2?versionId=null 2024-02-01T10:48:01.788 DEBUG:boto:Data: 2024-02-01T10:48:01.788 DEBUG:boto:Headers: {} 2024-02-01T10:48:01.788 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:01.788 DEBUG:boto:Port: 8000 2024-02-01T10:48:01.788 DEBUG:boto:Params: {} 2024-02-01T10:48:01.788 DEBUG:boto:Token: None 2024-02-01T10:48:01.788 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:01 GMT /vclfcn-59/obj2?versionId=null 2024-02-01T10:48:01.788 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:N7SRr7NFVfnjSvOc4rUb/cvrCJ0= 2024-02-01T10:48:01.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:N7SRr7NFVfnjSvOc4rUb/cvrCJ0=', 'Content-Length': '0'} 2024-02-01T10:48:01.791 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000051d3da7c100bf65f-0065bb76e1-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:01.836 DEBUG:boto:path=/vclfcn-59/obj2 2024-02-01T10:48:01.836 DEBUG:boto:auth_path=/vclfcn-59/obj2 2024-02-01T10:48:01.836 DEBUG:boto:path=/vclfcn-59/obj2?versionId=null 2024-02-01T10:48:01.837 DEBUG:boto:auth_path=/vclfcn-59/obj2?versionId=null 2024-02-01T10:48:01.837 DEBUG:boto:Method: GET 2024-02-01T10:48:01.837 DEBUG:boto:Path: /vclfcn-59/obj2?versionId=null 2024-02-01T10:48:01.837 DEBUG:boto:Data: 2024-02-01T10:48:01.837 DEBUG:boto:Headers: {} 2024-02-01T10:48:01.837 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:01.837 DEBUG:boto:Port: 8000 2024-02-01T10:48:01.837 DEBUG:boto:Params: {} 2024-02-01T10:48:01.837 DEBUG:boto:Token: None 2024-02-01T10:48:01.837 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:01 GMT /vclfcn-59/obj2?versionId=null 2024-02-01T10:48:01.838 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:N7SRr7NFVfnjSvOc4rUb/cvrCJ0= 2024-02-01T10:48:01.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:N7SRr7NFVfnjSvOc4rUb/cvrCJ0=', 'Content-Length': '0'} 2024-02-01T10:48:01.841 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ba98ab4f8fce0011-0065bb76e1-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:01.841 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:01.841 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:01.841 DEBUG:boto:path=/vclfcn-59/?prefix=obj2 2024-02-01T10:48:01.841 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj2 2024-02-01T10:48:01.841 DEBUG:boto:Method: GET 2024-02-01T10:48:01.841 DEBUG:boto:Path: /vclfcn-59/?prefix=obj2 2024-02-01T10:48:01.842 DEBUG:boto:Data: 2024-02-01T10:48:01.842 DEBUG:boto:Headers: {} 2024-02-01T10:48:01.842 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:01.842 DEBUG:boto:Port: 8000 2024-02-01T10:48:01.842 DEBUG:boto:Params: {} 2024-02-01T10:48:01.842 DEBUG:boto:Token: None 2024-02-01T10:48:01.842 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:01 GMT /vclfcn-59/ 2024-02-01T10:48:01.842 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5Vy8Xp+v9QzdkDyVOw8noS7fwz8= 2024-02-01T10:48:01.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5Vy8Xp+v9QzdkDyVOw8noS7fwz8=', 'Content-Length': '0'} 2024-02-01T10:48:01.847 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000024641711bcc01a5-0065bb76e1-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:01.847 DEBUG:boto:b'vclfcn-59obj21000falseobj22024-02-01T10:46:46.111Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:01.848 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:01.848 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:01.848 DEBUG:boto:path=/vclfcn-59/?prefix=obj2 2024-02-01T10:48:01.848 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj2 2024-02-01T10:48:01.848 DEBUG:boto:Method: GET 2024-02-01T10:48:01.848 DEBUG:boto:Path: /vclfcn-59/?prefix=obj2 2024-02-01T10:48:01.848 DEBUG:boto:Data: 2024-02-01T10:48:01.849 DEBUG:boto:Headers: {} 2024-02-01T10:48:01.849 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:01.849 DEBUG:boto:Port: 8000 2024-02-01T10:48:01.849 DEBUG:boto:Params: {} 2024-02-01T10:48:01.849 DEBUG:boto:Token: None 2024-02-01T10:48:01.849 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:01 GMT /vclfcn-59/ 2024-02-01T10:48:01.849 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5Vy8Xp+v9QzdkDyVOw8noS7fwz8= 2024-02-01T10:48:01.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5Vy8Xp+v9QzdkDyVOw8noS7fwz8=', 'Content-Length': '0'} 2024-02-01T10:48:01.853 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f377a32f191a0474-0065bb76e1-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:01.853 DEBUG:boto:b'vclfcn-59obj21000falseobj22024-02-01T10:46:46.111Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:01.854 DEBUG:boto:path=/vclfcn-59/obj2 2024-02-01T10:48:01.854 DEBUG:boto:auth_path=/vclfcn-59/obj2 2024-02-01T10:48:01.854 DEBUG:boto:Method: HEAD 2024-02-01T10:48:01.854 DEBUG:boto:Path: /vclfcn-59/obj2 2024-02-01T10:48:01.854 DEBUG:boto:Data: 2024-02-01T10:48:01.854 DEBUG:boto:Headers: {} 2024-02-01T10:48:01.854 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:01.854 DEBUG:boto:Port: 8000 2024-02-01T10:48:01.854 DEBUG:boto:Params: {} 2024-02-01T10:48:01.854 DEBUG:boto:Token: None 2024-02-01T10:48:01.855 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:01 GMT /vclfcn-59/obj2 2024-02-01T10:48:01.855 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/jgQVV1urE74e/qowJl33N6MBfs= 2024-02-01T10:48:01.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/jgQVV1urE74e/qowJl33N6MBfs=', 'Content-Length': '0'} 2024-02-01T10:48:01.857 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b2f129b7a211ba47-0065bb76e1-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:01.858 DEBUG:boto:path=/vclfcn-59/obj2 2024-02-01T10:48:01.858 DEBUG:boto:auth_path=/vclfcn-59/obj2 2024-02-01T10:48:01.858 DEBUG:boto:Method: HEAD 2024-02-01T10:48:01.858 DEBUG:boto:Path: /vclfcn-59/obj2 2024-02-01T10:48:01.858 DEBUG:boto:Data: 2024-02-01T10:48:01.858 DEBUG:boto:Headers: {} 2024-02-01T10:48:01.858 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:01.858 DEBUG:boto:Port: 8000 2024-02-01T10:48:01.858 DEBUG:boto:Params: {} 2024-02-01T10:48:01.859 DEBUG:boto:Token: None 2024-02-01T10:48:01.859 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:01 GMT /vclfcn-59/obj2 2024-02-01T10:48:01.859 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/jgQVV1urE74e/qowJl33N6MBfs= 2024-02-01T10:48:01.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/jgQVV1urE74e/qowJl33N6MBfs=', 'Content-Length': '0'} 2024-02-01T10:48:01.861 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d222a33d778f104f-0065bb76e1-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:01.862 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:48:01.862 DEBUG:boto:path=/vclfcn-59/obj2 2024-02-01T10:48:01.862 DEBUG:boto:auth_path=/vclfcn-59/obj2 2024-02-01T10:48:01.862 DEBUG:boto:Method: GET 2024-02-01T10:48:01.862 DEBUG:boto:Path: /vclfcn-59/obj2 2024-02-01T10:48:01.862 DEBUG:boto:Data: 2024-02-01T10:48:01.862 DEBUG:boto:Headers: {} 2024-02-01T10:48:01.862 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:01.863 DEBUG:boto:Port: 8000 2024-02-01T10:48:01.863 DEBUG:boto:Params: {} 2024-02-01T10:48:01.863 DEBUG:boto:Token: None 2024-02-01T10:48:01.863 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:01 GMT /vclfcn-59/obj2 2024-02-01T10:48:01.863 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:F6/tYI43Kpcz3WL+snextCT4mB8= 2024-02-01T10:48:01.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:F6/tYI43Kpcz3WL+snextCT4mB8=', 'Content-Length': '0'} 2024-02-01T10:48:01.866 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004549d5cbcfc03cfd-0065bb76e1-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:01.912 DEBUG:boto:path=/vclfcn-59/obj2 2024-02-01T10:48:01.912 DEBUG:boto:auth_path=/vclfcn-59/obj2 2024-02-01T10:48:01.912 DEBUG:boto:Method: GET 2024-02-01T10:48:01.912 DEBUG:boto:Path: /vclfcn-59/obj2 2024-02-01T10:48:01.912 DEBUG:boto:Data: 2024-02-01T10:48:01.913 DEBUG:boto:Headers: {} 2024-02-01T10:48:01.913 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:01.913 DEBUG:boto:Port: 8000 2024-02-01T10:48:01.913 DEBUG:boto:Params: {} 2024-02-01T10:48:01.913 DEBUG:boto:Token: None 2024-02-01T10:48:01.913 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:01 GMT /vclfcn-59/obj2 2024-02-01T10:48:01.913 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:F6/tYI43Kpcz3WL+snextCT4mB8= 2024-02-01T10:48:01.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:F6/tYI43Kpcz3WL+snextCT4mB8=', 'Content-Length': '0'} 2024-02-01T10:48:01.916 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f8fcd07e57c8d199-0065bb76e1-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:01.916 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:01.917 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:01.917 DEBUG:boto:path=/vclfcn-59/?prefix=obj2 2024-02-01T10:48:01.917 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj2 2024-02-01T10:48:01.917 DEBUG:boto:Method: GET 2024-02-01T10:48:01.917 DEBUG:boto:Path: /vclfcn-59/?prefix=obj2 2024-02-01T10:48:01.917 DEBUG:boto:Data: 2024-02-01T10:48:01.917 DEBUG:boto:Headers: {} 2024-02-01T10:48:01.917 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:01.917 DEBUG:boto:Port: 8000 2024-02-01T10:48:01.917 DEBUG:boto:Params: {} 2024-02-01T10:48:01.918 DEBUG:boto:Token: None 2024-02-01T10:48:01.918 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:01 GMT /vclfcn-59/ 2024-02-01T10:48:01.918 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5Vy8Xp+v9QzdkDyVOw8noS7fwz8= 2024-02-01T10:48:01.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5Vy8Xp+v9QzdkDyVOw8noS7fwz8=', 'Content-Length': '0'} 2024-02-01T10:48:01.922 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d8e7902d31ba64a1-0065bb76e1-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:01.922 DEBUG:boto:b'vclfcn-59obj21000falseobj22024-02-01T10:46:46.111Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:01.923 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:01.923 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:01.923 DEBUG:boto:path=/vclfcn-59/?prefix=obj2 2024-02-01T10:48:01.923 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj2 2024-02-01T10:48:01.923 DEBUG:boto:Method: GET 2024-02-01T10:48:01.923 DEBUG:boto:Path: /vclfcn-59/?prefix=obj2 2024-02-01T10:48:01.923 DEBUG:boto:Data: 2024-02-01T10:48:01.923 DEBUG:boto:Headers: {} 2024-02-01T10:48:01.924 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:01.924 DEBUG:boto:Port: 8000 2024-02-01T10:48:01.924 DEBUG:boto:Params: {} 2024-02-01T10:48:01.924 DEBUG:boto:Token: None 2024-02-01T10:48:01.924 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:01 GMT /vclfcn-59/ 2024-02-01T10:48:01.924 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5Vy8Xp+v9QzdkDyVOw8noS7fwz8= 2024-02-01T10:48:01.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5Vy8Xp+v9QzdkDyVOw8noS7fwz8=', 'Content-Length': '0'} 2024-02-01T10:48:01.928 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004bf57d95943b2239-0065bb76e1-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:01.929 DEBUG:boto:b'vclfcn-59obj21000falseobj22024-02-01T10:46:46.111Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:01.929 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:48:01.929 DEBUG:boto:path=/vclfcn-59/obj3 2024-02-01T10:48:01.929 DEBUG:boto:auth_path=/vclfcn-59/obj3 2024-02-01T10:48:01.929 DEBUG:boto:path=/vclfcn-59/obj3?versionId=null 2024-02-01T10:48:01.929 DEBUG:boto:auth_path=/vclfcn-59/obj3?versionId=null 2024-02-01T10:48:01.929 DEBUG:boto:Method: GET 2024-02-01T10:48:01.930 DEBUG:boto:Path: /vclfcn-59/obj3?versionId=null 2024-02-01T10:48:01.930 DEBUG:boto:Data: 2024-02-01T10:48:01.930 DEBUG:boto:Headers: {} 2024-02-01T10:48:01.930 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:01.930 DEBUG:boto:Port: 8000 2024-02-01T10:48:01.930 DEBUG:boto:Params: {} 2024-02-01T10:48:01.930 DEBUG:boto:Token: None 2024-02-01T10:48:01.930 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:01 GMT /vclfcn-59/obj3?versionId=null 2024-02-01T10:48:01.930 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:yR6sq2tv8kow2H8e1LuH6mePJDw= 2024-02-01T10:48:01.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:yR6sq2tv8kow2H8e1LuH6mePJDw=', 'Content-Length': '0'} 2024-02-01T10:48:01.933 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000056095ac5a828ee15-0065bb76e1-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:01.933 DEBUG:boto:path=/vclfcn-59/obj3 2024-02-01T10:48:01.933 DEBUG:boto:auth_path=/vclfcn-59/obj3 2024-02-01T10:48:01.934 DEBUG:boto:path=/vclfcn-59/obj3?versionId=null 2024-02-01T10:48:01.934 DEBUG:boto:auth_path=/vclfcn-59/obj3?versionId=null 2024-02-01T10:48:01.934 DEBUG:boto:Method: GET 2024-02-01T10:48:01.934 DEBUG:boto:Path: /vclfcn-59/obj3?versionId=null 2024-02-01T10:48:01.934 DEBUG:boto:Data: 2024-02-01T10:48:01.934 DEBUG:boto:Headers: {} 2024-02-01T10:48:01.934 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:01.934 DEBUG:boto:Port: 8000 2024-02-01T10:48:01.934 DEBUG:boto:Params: {} 2024-02-01T10:48:01.934 DEBUG:boto:Token: None 2024-02-01T10:48:01.935 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:01 GMT /vclfcn-59/obj3?versionId=null 2024-02-01T10:48:01.935 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:yR6sq2tv8kow2H8e1LuH6mePJDw= 2024-02-01T10:48:01.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:yR6sq2tv8kow2H8e1LuH6mePJDw=', 'Content-Length': '0'} 2024-02-01T10:48:01.938 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aea155f709a2feb6-0065bb76e1-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:01.938 DEBUG:boto:path=/vclfcn-59/obj3 2024-02-01T10:48:01.938 DEBUG:boto:auth_path=/vclfcn-59/obj3 2024-02-01T10:48:01.938 DEBUG:boto:path=/vclfcn-59/obj3?versionId=null 2024-02-01T10:48:01.938 DEBUG:boto:auth_path=/vclfcn-59/obj3?versionId=null 2024-02-01T10:48:01.938 DEBUG:boto:Method: HEAD 2024-02-01T10:48:01.938 DEBUG:boto:Path: /vclfcn-59/obj3?versionId=null 2024-02-01T10:48:01.939 DEBUG:boto:Data: 2024-02-01T10:48:01.939 DEBUG:boto:Headers: {} 2024-02-01T10:48:01.939 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:01.939 DEBUG:boto:Port: 8000 2024-02-01T10:48:01.939 DEBUG:boto:Params: {} 2024-02-01T10:48:01.939 DEBUG:boto:Token: None 2024-02-01T10:48:01.939 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:01 GMT /vclfcn-59/obj3?versionId=null 2024-02-01T10:48:01.939 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0hepAZ+E5Xc0/MpTmE2kVLvgiWA= 2024-02-01T10:48:01.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0hepAZ+E5Xc0/MpTmE2kVLvgiWA=', 'Content-Length': '0'} 2024-02-01T10:48:01.942 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003874a9870479d33a-0065bb76e1-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:01.942 DEBUG:boto:path=/vclfcn-59/obj3 2024-02-01T10:48:01.943 DEBUG:boto:auth_path=/vclfcn-59/obj3 2024-02-01T10:48:01.943 DEBUG:boto:path=/vclfcn-59/obj3?versionId=null 2024-02-01T10:48:01.943 DEBUG:boto:auth_path=/vclfcn-59/obj3?versionId=null 2024-02-01T10:48:01.943 DEBUG:boto:Method: HEAD 2024-02-01T10:48:01.943 DEBUG:boto:Path: /vclfcn-59/obj3?versionId=null 2024-02-01T10:48:01.943 DEBUG:boto:Data: 2024-02-01T10:48:01.943 DEBUG:boto:Headers: {} 2024-02-01T10:48:01.943 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:01.943 DEBUG:boto:Port: 8000 2024-02-01T10:48:01.943 DEBUG:boto:Params: {} 2024-02-01T10:48:01.944 DEBUG:boto:Token: None 2024-02-01T10:48:01.944 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:01 GMT /vclfcn-59/obj3?versionId=null 2024-02-01T10:48:01.944 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0hepAZ+E5Xc0/MpTmE2kVLvgiWA= 2024-02-01T10:48:01.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0hepAZ+E5Xc0/MpTmE2kVLvgiWA=', 'Content-Length': '0'} 2024-02-01T10:48:01.947 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006760f5afffc1c728-0065bb76e1-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:01.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:48:01.947 DEBUG:boto:path=/vclfcn-59/obj3 2024-02-01T10:48:01.947 DEBUG:boto:auth_path=/vclfcn-59/obj3 2024-02-01T10:48:01.947 DEBUG:boto:path=/vclfcn-59/obj3?versionId=null 2024-02-01T10:48:01.947 DEBUG:boto:auth_path=/vclfcn-59/obj3?versionId=null 2024-02-01T10:48:01.947 DEBUG:boto:Method: GET 2024-02-01T10:48:01.948 DEBUG:boto:Path: /vclfcn-59/obj3?versionId=null 2024-02-01T10:48:01.948 DEBUG:boto:Data: 2024-02-01T10:48:01.948 DEBUG:boto:Headers: {} 2024-02-01T10:48:01.948 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:01.948 DEBUG:boto:Port: 8000 2024-02-01T10:48:01.948 DEBUG:boto:Params: {} 2024-02-01T10:48:01.948 DEBUG:boto:Token: None 2024-02-01T10:48:01.948 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:01 GMT /vclfcn-59/obj3?versionId=null 2024-02-01T10:48:01.948 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:yR6sq2tv8kow2H8e1LuH6mePJDw= 2024-02-01T10:48:01.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:yR6sq2tv8kow2H8e1LuH6mePJDw=', 'Content-Length': '0'} 2024-02-01T10:48:01.951 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007299f3b8e839dc45-0065bb76e1-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:01.992 DEBUG:boto:path=/vclfcn-59/obj3 2024-02-01T10:48:01.992 DEBUG:boto:auth_path=/vclfcn-59/obj3 2024-02-01T10:48:01.992 DEBUG:boto:path=/vclfcn-59/obj3?versionId=null 2024-02-01T10:48:01.992 DEBUG:boto:auth_path=/vclfcn-59/obj3?versionId=null 2024-02-01T10:48:01.992 DEBUG:boto:Method: GET 2024-02-01T10:48:01.993 DEBUG:boto:Path: /vclfcn-59/obj3?versionId=null 2024-02-01T10:48:01.993 DEBUG:boto:Data: 2024-02-01T10:48:01.993 DEBUG:boto:Headers: {} 2024-02-01T10:48:01.993 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:01.993 DEBUG:boto:Port: 8000 2024-02-01T10:48:01.993 DEBUG:boto:Params: {} 2024-02-01T10:48:01.993 DEBUG:boto:Token: None 2024-02-01T10:48:01.993 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:01 GMT /vclfcn-59/obj3?versionId=null 2024-02-01T10:48:01.993 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:yR6sq2tv8kow2H8e1LuH6mePJDw= 2024-02-01T10:48:01.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:yR6sq2tv8kow2H8e1LuH6mePJDw=', 'Content-Length': '0'} 2024-02-01T10:48:01.996 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004c443344f1f475c5-0065bb76e1-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:01.996 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:01.997 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:01.997 DEBUG:boto:path=/vclfcn-59/?prefix=obj3 2024-02-01T10:48:01.997 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj3 2024-02-01T10:48:01.997 DEBUG:boto:Method: GET 2024-02-01T10:48:01.997 DEBUG:boto:Path: /vclfcn-59/?prefix=obj3 2024-02-01T10:48:01.997 DEBUG:boto:Data: 2024-02-01T10:48:01.997 DEBUG:boto:Headers: {} 2024-02-01T10:48:01.997 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:01.997 DEBUG:boto:Port: 8000 2024-02-01T10:48:01.997 DEBUG:boto:Params: {} 2024-02-01T10:48:01.997 DEBUG:boto:Token: None 2024-02-01T10:48:01.998 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:01 GMT /vclfcn-59/ 2024-02-01T10:48:01.998 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5Vy8Xp+v9QzdkDyVOw8noS7fwz8= 2024-02-01T10:48:01.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5Vy8Xp+v9QzdkDyVOw8noS7fwz8=', 'Content-Length': '0'} 2024-02-01T10:48:02.002 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d2d36daf2243b78-0065bb76e1-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:02.002 DEBUG:boto:b'vclfcn-59obj31000falseobj32024-02-01T10:46:46.161Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:02.002 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:02.003 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:02.003 DEBUG:boto:path=/vclfcn-59/?prefix=obj3 2024-02-01T10:48:02.003 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj3 2024-02-01T10:48:02.003 DEBUG:boto:Method: GET 2024-02-01T10:48:02.003 DEBUG:boto:Path: /vclfcn-59/?prefix=obj3 2024-02-01T10:48:02.003 DEBUG:boto:Data: 2024-02-01T10:48:02.003 DEBUG:boto:Headers: {} 2024-02-01T10:48:02.003 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:02.003 DEBUG:boto:Port: 8000 2024-02-01T10:48:02.003 DEBUG:boto:Params: {} 2024-02-01T10:48:02.003 DEBUG:boto:Token: None 2024-02-01T10:48:02.004 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:02 GMT /vclfcn-59/ 2024-02-01T10:48:02.004 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:c2RgMycaHPRZMhj4ypc9JNhplUI= 2024-02-01T10:48:02.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:c2RgMycaHPRZMhj4ypc9JNhplUI=', 'Content-Length': '0'} 2024-02-01T10:48:02.008 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000801708f65040c809-0065bb76e2-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:02.008 DEBUG:boto:b'vclfcn-59obj31000falseobj32024-02-01T10:46:46.161Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:02.008 DEBUG:boto:path=/vclfcn-59/obj3 2024-02-01T10:48:02.008 DEBUG:boto:auth_path=/vclfcn-59/obj3 2024-02-01T10:48:02.008 DEBUG:boto:Method: HEAD 2024-02-01T10:48:02.009 DEBUG:boto:Path: /vclfcn-59/obj3 2024-02-01T10:48:02.009 DEBUG:boto:Data: 2024-02-01T10:48:02.009 DEBUG:boto:Headers: {} 2024-02-01T10:48:02.009 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:02.009 DEBUG:boto:Port: 8000 2024-02-01T10:48:02.009 DEBUG:boto:Params: {} 2024-02-01T10:48:02.009 DEBUG:boto:Token: None 2024-02-01T10:48:02.009 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:02 GMT /vclfcn-59/obj3 2024-02-01T10:48:02.009 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:4ynQ0gpjLVQFJ8GlIw+tsoADBwI= 2024-02-01T10:48:02.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:4ynQ0gpjLVQFJ8GlIw+tsoADBwI=', 'Content-Length': '0'} 2024-02-01T10:48:02.012 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f221dc6c5199075d-0065bb76e2-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:02.012 DEBUG:boto:path=/vclfcn-59/obj3 2024-02-01T10:48:02.012 DEBUG:boto:auth_path=/vclfcn-59/obj3 2024-02-01T10:48:02.012 DEBUG:boto:Method: HEAD 2024-02-01T10:48:02.013 DEBUG:boto:Path: /vclfcn-59/obj3 2024-02-01T10:48:02.013 DEBUG:boto:Data: 2024-02-01T10:48:02.013 DEBUG:boto:Headers: {} 2024-02-01T10:48:02.013 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:02.013 DEBUG:boto:Port: 8000 2024-02-01T10:48:02.013 DEBUG:boto:Params: {} 2024-02-01T10:48:02.013 DEBUG:boto:Token: None 2024-02-01T10:48:02.013 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:02 GMT /vclfcn-59/obj3 2024-02-01T10:48:02.013 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:4ynQ0gpjLVQFJ8GlIw+tsoADBwI= 2024-02-01T10:48:02.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:4ynQ0gpjLVQFJ8GlIw+tsoADBwI=', 'Content-Length': '0'} 2024-02-01T10:48:02.016 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000075c7d81d4f5d5215-0065bb76e2-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:02.016 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:48:02.016 DEBUG:boto:path=/vclfcn-59/obj3 2024-02-01T10:48:02.016 DEBUG:boto:auth_path=/vclfcn-59/obj3 2024-02-01T10:48:02.017 DEBUG:boto:Method: GET 2024-02-01T10:48:02.017 DEBUG:boto:Path: /vclfcn-59/obj3 2024-02-01T10:48:02.017 DEBUG:boto:Data: 2024-02-01T10:48:02.017 DEBUG:boto:Headers: {} 2024-02-01T10:48:02.017 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:02.017 DEBUG:boto:Port: 8000 2024-02-01T10:48:02.017 DEBUG:boto:Params: {} 2024-02-01T10:48:02.017 DEBUG:boto:Token: None 2024-02-01T10:48:02.017 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:02 GMT /vclfcn-59/obj3 2024-02-01T10:48:02.017 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Z4rx9DV943jhO1A8Fh7daQZXyxY= 2024-02-01T10:48:02.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Z4rx9DV943jhO1A8Fh7daQZXyxY=', 'Content-Length': '0'} 2024-02-01T10:48:02.020 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ba85febf7c1a6e5e-0065bb76e2-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:02.064 DEBUG:boto:path=/vclfcn-59/obj3 2024-02-01T10:48:02.064 DEBUG:boto:auth_path=/vclfcn-59/obj3 2024-02-01T10:48:02.064 DEBUG:boto:Method: GET 2024-02-01T10:48:02.064 DEBUG:boto:Path: /vclfcn-59/obj3 2024-02-01T10:48:02.065 DEBUG:boto:Data: 2024-02-01T10:48:02.065 DEBUG:boto:Headers: {} 2024-02-01T10:48:02.065 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:02.065 DEBUG:boto:Port: 8000 2024-02-01T10:48:02.065 DEBUG:boto:Params: {} 2024-02-01T10:48:02.065 DEBUG:boto:Token: None 2024-02-01T10:48:02.065 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:02 GMT /vclfcn-59/obj3 2024-02-01T10:48:02.065 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Z4rx9DV943jhO1A8Fh7daQZXyxY= 2024-02-01T10:48:02.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Z4rx9DV943jhO1A8Fh7daQZXyxY=', 'Content-Length': '0'} 2024-02-01T10:48:02.068 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d0e0d46aedb38eac-0065bb76e2-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:02.068 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:02.069 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:02.069 DEBUG:boto:path=/vclfcn-59/?prefix=obj3 2024-02-01T10:48:02.069 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj3 2024-02-01T10:48:02.069 DEBUG:boto:Method: GET 2024-02-01T10:48:02.069 DEBUG:boto:Path: /vclfcn-59/?prefix=obj3 2024-02-01T10:48:02.069 DEBUG:boto:Data: 2024-02-01T10:48:02.069 DEBUG:boto:Headers: {} 2024-02-01T10:48:02.069 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:02.069 DEBUG:boto:Port: 8000 2024-02-01T10:48:02.069 DEBUG:boto:Params: {} 2024-02-01T10:48:02.070 DEBUG:boto:Token: None 2024-02-01T10:48:02.070 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:02 GMT /vclfcn-59/ 2024-02-01T10:48:02.070 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:c2RgMycaHPRZMhj4ypc9JNhplUI= 2024-02-01T10:48:02.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:c2RgMycaHPRZMhj4ypc9JNhplUI=', 'Content-Length': '0'} 2024-02-01T10:48:02.074 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002619f4c7cbc3828b-0065bb76e2-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:02.116 DEBUG:boto:b'vclfcn-59obj31000falseobj32024-02-01T10:46:46.161Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:02.116 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:02.117 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:02.117 DEBUG:boto:path=/vclfcn-59/?prefix=obj3 2024-02-01T10:48:02.117 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj3 2024-02-01T10:48:02.117 DEBUG:boto:Method: GET 2024-02-01T10:48:02.117 DEBUG:boto:Path: /vclfcn-59/?prefix=obj3 2024-02-01T10:48:02.117 DEBUG:boto:Data: 2024-02-01T10:48:02.117 DEBUG:boto:Headers: {} 2024-02-01T10:48:02.117 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:02.117 DEBUG:boto:Port: 8000 2024-02-01T10:48:02.117 DEBUG:boto:Params: {} 2024-02-01T10:48:02.118 DEBUG:boto:Token: None 2024-02-01T10:48:02.118 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:02 GMT /vclfcn-59/ 2024-02-01T10:48:02.118 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:c2RgMycaHPRZMhj4ypc9JNhplUI= 2024-02-01T10:48:02.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:c2RgMycaHPRZMhj4ypc9JNhplUI=', 'Content-Length': '0'} 2024-02-01T10:48:02.123 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a298be7329af870-0065bb76e2-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:02.123 DEBUG:boto:b'vclfcn-59obj31000falseobj32024-02-01T10:46:46.161Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:02.123 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:48:02.124 DEBUG:boto:path=/vclfcn-59/obj4 2024-02-01T10:48:02.124 DEBUG:boto:auth_path=/vclfcn-59/obj4 2024-02-01T10:48:02.124 DEBUG:boto:path=/vclfcn-59/obj4?versionId=null 2024-02-01T10:48:02.124 DEBUG:boto:auth_path=/vclfcn-59/obj4?versionId=null 2024-02-01T10:48:02.124 DEBUG:boto:Method: GET 2024-02-01T10:48:02.124 DEBUG:boto:Path: /vclfcn-59/obj4?versionId=null 2024-02-01T10:48:02.124 DEBUG:boto:Data: 2024-02-01T10:48:02.124 DEBUG:boto:Headers: {} 2024-02-01T10:48:02.124 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:02.124 DEBUG:boto:Port: 8000 2024-02-01T10:48:02.125 DEBUG:boto:Params: {} 2024-02-01T10:48:02.125 DEBUG:boto:Token: None 2024-02-01T10:48:02.125 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:02 GMT /vclfcn-59/obj4?versionId=null 2024-02-01T10:48:02.125 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fi4+NXaKp04HjJpHTMf7JtKXzQ0= 2024-02-01T10:48:02.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fi4+NXaKp04HjJpHTMf7JtKXzQ0=', 'Content-Length': '0'} 2024-02-01T10:48:02.128 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fe65701bf1b4932c-0065bb76e2-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:02.172 DEBUG:boto:path=/vclfcn-59/obj4 2024-02-01T10:48:02.173 DEBUG:boto:auth_path=/vclfcn-59/obj4 2024-02-01T10:48:02.173 DEBUG:boto:path=/vclfcn-59/obj4?versionId=null 2024-02-01T10:48:02.173 DEBUG:boto:auth_path=/vclfcn-59/obj4?versionId=null 2024-02-01T10:48:02.173 DEBUG:boto:Method: GET 2024-02-01T10:48:02.173 DEBUG:boto:Path: /vclfcn-59/obj4?versionId=null 2024-02-01T10:48:02.173 DEBUG:boto:Data: 2024-02-01T10:48:02.173 DEBUG:boto:Headers: {} 2024-02-01T10:48:02.173 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:02.173 DEBUG:boto:Port: 8000 2024-02-01T10:48:02.173 DEBUG:boto:Params: {} 2024-02-01T10:48:02.174 DEBUG:boto:Token: None 2024-02-01T10:48:02.174 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:02 GMT /vclfcn-59/obj4?versionId=null 2024-02-01T10:48:02.174 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fi4+NXaKp04HjJpHTMf7JtKXzQ0= 2024-02-01T10:48:02.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fi4+NXaKp04HjJpHTMf7JtKXzQ0=', 'Content-Length': '0'} 2024-02-01T10:48:02.177 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000926a04a6930c012-0065bb76e2-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:02.177 DEBUG:boto:path=/vclfcn-59/obj4 2024-02-01T10:48:02.177 DEBUG:boto:auth_path=/vclfcn-59/obj4 2024-02-01T10:48:02.178 DEBUG:boto:path=/vclfcn-59/obj4?versionId=null 2024-02-01T10:48:02.178 DEBUG:boto:auth_path=/vclfcn-59/obj4?versionId=null 2024-02-01T10:48:02.178 DEBUG:boto:Method: HEAD 2024-02-01T10:48:02.178 DEBUG:boto:Path: /vclfcn-59/obj4?versionId=null 2024-02-01T10:48:02.178 DEBUG:boto:Data: 2024-02-01T10:48:02.178 DEBUG:boto:Headers: {} 2024-02-01T10:48:02.178 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:02.178 DEBUG:boto:Port: 8000 2024-02-01T10:48:02.178 DEBUG:boto:Params: {} 2024-02-01T10:48:02.178 DEBUG:boto:Token: None 2024-02-01T10:48:02.179 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:02 GMT /vclfcn-59/obj4?versionId=null 2024-02-01T10:48:02.179 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gw5TCHUNVwbXsAYGOFIiM83zm4I= 2024-02-01T10:48:02.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gw5TCHUNVwbXsAYGOFIiM83zm4I=', 'Content-Length': '0'} 2024-02-01T10:48:02.182 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000043ee075df0948772-0065bb76e2-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:02.182 DEBUG:boto:path=/vclfcn-59/obj4 2024-02-01T10:48:02.182 DEBUG:boto:auth_path=/vclfcn-59/obj4 2024-02-01T10:48:02.182 DEBUG:boto:path=/vclfcn-59/obj4?versionId=null 2024-02-01T10:48:02.182 DEBUG:boto:auth_path=/vclfcn-59/obj4?versionId=null 2024-02-01T10:48:02.183 DEBUG:boto:Method: HEAD 2024-02-01T10:48:02.183 DEBUG:boto:Path: /vclfcn-59/obj4?versionId=null 2024-02-01T10:48:02.183 DEBUG:boto:Data: 2024-02-01T10:48:02.183 DEBUG:boto:Headers: {} 2024-02-01T10:48:02.183 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:02.183 DEBUG:boto:Port: 8000 2024-02-01T10:48:02.183 DEBUG:boto:Params: {} 2024-02-01T10:48:02.183 DEBUG:boto:Token: None 2024-02-01T10:48:02.183 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:02 GMT /vclfcn-59/obj4?versionId=null 2024-02-01T10:48:02.184 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gw5TCHUNVwbXsAYGOFIiM83zm4I= 2024-02-01T10:48:02.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gw5TCHUNVwbXsAYGOFIiM83zm4I=', 'Content-Length': '0'} 2024-02-01T10:48:02.186 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bfbfda869a4b11e6-0065bb76e2-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:02.187 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:48:02.187 DEBUG:boto:path=/vclfcn-59/obj4 2024-02-01T10:48:02.187 DEBUG:boto:auth_path=/vclfcn-59/obj4 2024-02-01T10:48:02.187 DEBUG:boto:path=/vclfcn-59/obj4?versionId=null 2024-02-01T10:48:02.187 DEBUG:boto:auth_path=/vclfcn-59/obj4?versionId=null 2024-02-01T10:48:02.187 DEBUG:boto:Method: GET 2024-02-01T10:48:02.187 DEBUG:boto:Path: /vclfcn-59/obj4?versionId=null 2024-02-01T10:48:02.188 DEBUG:boto:Data: 2024-02-01T10:48:02.188 DEBUG:boto:Headers: {} 2024-02-01T10:48:02.188 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:02.188 DEBUG:boto:Port: 8000 2024-02-01T10:48:02.188 DEBUG:boto:Params: {} 2024-02-01T10:48:02.188 DEBUG:boto:Token: None 2024-02-01T10:48:02.188 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:02 GMT /vclfcn-59/obj4?versionId=null 2024-02-01T10:48:02.188 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fi4+NXaKp04HjJpHTMf7JtKXzQ0= 2024-02-01T10:48:02.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fi4+NXaKp04HjJpHTMf7JtKXzQ0=', 'Content-Length': '0'} 2024-02-01T10:48:02.191 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009161e6a21f53f17e-0065bb76e2-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:02.232 DEBUG:boto:path=/vclfcn-59/obj4 2024-02-01T10:48:02.232 DEBUG:boto:auth_path=/vclfcn-59/obj4 2024-02-01T10:48:02.232 DEBUG:boto:path=/vclfcn-59/obj4?versionId=null 2024-02-01T10:48:02.233 DEBUG:boto:auth_path=/vclfcn-59/obj4?versionId=null 2024-02-01T10:48:02.233 DEBUG:boto:Method: GET 2024-02-01T10:48:02.233 DEBUG:boto:Path: /vclfcn-59/obj4?versionId=null 2024-02-01T10:48:02.233 DEBUG:boto:Data: 2024-02-01T10:48:02.233 DEBUG:boto:Headers: {} 2024-02-01T10:48:02.233 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:02.233 DEBUG:boto:Port: 8000 2024-02-01T10:48:02.233 DEBUG:boto:Params: {} 2024-02-01T10:48:02.233 DEBUG:boto:Token: None 2024-02-01T10:48:02.234 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:02 GMT /vclfcn-59/obj4?versionId=null 2024-02-01T10:48:02.234 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fi4+NXaKp04HjJpHTMf7JtKXzQ0= 2024-02-01T10:48:02.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fi4+NXaKp04HjJpHTMf7JtKXzQ0=', 'Content-Length': '0'} 2024-02-01T10:48:02.237 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b9e24025823c8732-0065bb76e2-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:02.238 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:02.238 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:02.238 DEBUG:boto:path=/vclfcn-59/?prefix=obj4 2024-02-01T10:48:02.238 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj4 2024-02-01T10:48:02.238 DEBUG:boto:Method: GET 2024-02-01T10:48:02.238 DEBUG:boto:Path: /vclfcn-59/?prefix=obj4 2024-02-01T10:48:02.238 DEBUG:boto:Data: 2024-02-01T10:48:02.238 DEBUG:boto:Headers: {} 2024-02-01T10:48:02.238 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:02.238 DEBUG:boto:Port: 8000 2024-02-01T10:48:02.239 DEBUG:boto:Params: {} 2024-02-01T10:48:02.239 DEBUG:boto:Token: None 2024-02-01T10:48:02.239 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:02 GMT /vclfcn-59/ 2024-02-01T10:48:02.239 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:c2RgMycaHPRZMhj4ypc9JNhplUI= 2024-02-01T10:48:02.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:c2RgMycaHPRZMhj4ypc9JNhplUI=', 'Content-Length': '0'} 2024-02-01T10:48:02.243 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026bb9048ad0d797b-0065bb76e2-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:02.244 DEBUG:boto:b'vclfcn-59obj41000falseobj42024-02-01T10:46:46.190Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:02.244 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:02.244 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:02.244 DEBUG:boto:path=/vclfcn-59/?prefix=obj4 2024-02-01T10:48:02.245 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj4 2024-02-01T10:48:02.245 DEBUG:boto:Method: GET 2024-02-01T10:48:02.245 DEBUG:boto:Path: /vclfcn-59/?prefix=obj4 2024-02-01T10:48:02.245 DEBUG:boto:Data: 2024-02-01T10:48:02.245 DEBUG:boto:Headers: {} 2024-02-01T10:48:02.245 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:02.245 DEBUG:boto:Port: 8000 2024-02-01T10:48:02.245 DEBUG:boto:Params: {} 2024-02-01T10:48:02.245 DEBUG:boto:Token: None 2024-02-01T10:48:02.246 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:02 GMT /vclfcn-59/ 2024-02-01T10:48:02.246 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:c2RgMycaHPRZMhj4ypc9JNhplUI= 2024-02-01T10:48:02.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:c2RgMycaHPRZMhj4ypc9JNhplUI=', 'Content-Length': '0'} 2024-02-01T10:48:02.250 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006db9334efccbfaa0-0065bb76e2-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:02.250 DEBUG:boto:b'vclfcn-59obj41000falseobj42024-02-01T10:46:46.190Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:02.251 DEBUG:boto:path=/vclfcn-59/obj4 2024-02-01T10:48:02.251 DEBUG:boto:auth_path=/vclfcn-59/obj4 2024-02-01T10:48:02.251 DEBUG:boto:Method: HEAD 2024-02-01T10:48:02.251 DEBUG:boto:Path: /vclfcn-59/obj4 2024-02-01T10:48:02.251 DEBUG:boto:Data: 2024-02-01T10:48:02.251 DEBUG:boto:Headers: {} 2024-02-01T10:48:02.251 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:02.252 DEBUG:boto:Port: 8000 2024-02-01T10:48:02.252 DEBUG:boto:Params: {} 2024-02-01T10:48:02.252 DEBUG:boto:Token: None 2024-02-01T10:48:02.252 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:02 GMT /vclfcn-59/obj4 2024-02-01T10:48:02.252 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:3W97WJXbfosyD4rAhMSrIuv5Has= 2024-02-01T10:48:02.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:3W97WJXbfosyD4rAhMSrIuv5Has=', 'Content-Length': '0'} 2024-02-01T10:48:02.255 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000010c2dcc88f52556-0065bb76e2-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:02.255 DEBUG:boto:path=/vclfcn-59/obj4 2024-02-01T10:48:02.256 DEBUG:boto:auth_path=/vclfcn-59/obj4 2024-02-01T10:48:02.256 DEBUG:boto:Method: HEAD 2024-02-01T10:48:02.256 DEBUG:boto:Path: /vclfcn-59/obj4 2024-02-01T10:48:02.256 DEBUG:boto:Data: 2024-02-01T10:48:02.256 DEBUG:boto:Headers: {} 2024-02-01T10:48:02.256 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:02.256 DEBUG:boto:Port: 8000 2024-02-01T10:48:02.256 DEBUG:boto:Params: {} 2024-02-01T10:48:02.257 DEBUG:boto:Token: None 2024-02-01T10:48:02.257 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:02 GMT /vclfcn-59/obj4 2024-02-01T10:48:02.257 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:3W97WJXbfosyD4rAhMSrIuv5Has= 2024-02-01T10:48:02.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:3W97WJXbfosyD4rAhMSrIuv5Has=', 'Content-Length': '0'} 2024-02-01T10:48:02.260 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fabac50d0cd67013-0065bb76e2-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:02.260 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:48:02.260 DEBUG:boto:path=/vclfcn-59/obj4 2024-02-01T10:48:02.260 DEBUG:boto:auth_path=/vclfcn-59/obj4 2024-02-01T10:48:02.260 DEBUG:boto:Method: GET 2024-02-01T10:48:02.261 DEBUG:boto:Path: /vclfcn-59/obj4 2024-02-01T10:48:02.261 DEBUG:boto:Data: 2024-02-01T10:48:02.261 DEBUG:boto:Headers: {} 2024-02-01T10:48:02.261 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:02.261 DEBUG:boto:Port: 8000 2024-02-01T10:48:02.261 DEBUG:boto:Params: {} 2024-02-01T10:48:02.261 DEBUG:boto:Token: None 2024-02-01T10:48:02.261 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:02 GMT /vclfcn-59/obj4 2024-02-01T10:48:02.261 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8G1Kw+W0DL3O45ea9d2sGRr8LBY= 2024-02-01T10:48:02.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8G1Kw+W0DL3O45ea9d2sGRr8LBY=', 'Content-Length': '0'} 2024-02-01T10:48:02.264 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005b33bc195636d004-0065bb76e2-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:02.312 DEBUG:boto:path=/vclfcn-59/obj4 2024-02-01T10:48:02.312 DEBUG:boto:auth_path=/vclfcn-59/obj4 2024-02-01T10:48:02.312 DEBUG:boto:Method: GET 2024-02-01T10:48:02.313 DEBUG:boto:Path: /vclfcn-59/obj4 2024-02-01T10:48:02.313 DEBUG:boto:Data: 2024-02-01T10:48:02.313 DEBUG:boto:Headers: {} 2024-02-01T10:48:02.313 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:02.313 DEBUG:boto:Port: 8000 2024-02-01T10:48:02.313 DEBUG:boto:Params: {} 2024-02-01T10:48:02.313 DEBUG:boto:Token: None 2024-02-01T10:48:02.313 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:02 GMT /vclfcn-59/obj4 2024-02-01T10:48:02.313 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8G1Kw+W0DL3O45ea9d2sGRr8LBY= 2024-02-01T10:48:02.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8G1Kw+W0DL3O45ea9d2sGRr8LBY=', 'Content-Length': '0'} 2024-02-01T10:48:02.319 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cf83df64777265e5-0065bb76e2-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:02.319 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:02.320 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:02.320 DEBUG:boto:path=/vclfcn-59/?prefix=obj4 2024-02-01T10:48:02.320 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj4 2024-02-01T10:48:02.320 DEBUG:boto:Method: GET 2024-02-01T10:48:02.320 DEBUG:boto:Path: /vclfcn-59/?prefix=obj4 2024-02-01T10:48:02.320 DEBUG:boto:Data: 2024-02-01T10:48:02.320 DEBUG:boto:Headers: {} 2024-02-01T10:48:02.320 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:02.320 DEBUG:boto:Port: 8000 2024-02-01T10:48:02.320 DEBUG:boto:Params: {} 2024-02-01T10:48:02.321 DEBUG:boto:Token: None 2024-02-01T10:48:02.321 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:02 GMT /vclfcn-59/ 2024-02-01T10:48:02.321 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:c2RgMycaHPRZMhj4ypc9JNhplUI= 2024-02-01T10:48:02.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:c2RgMycaHPRZMhj4ypc9JNhplUI=', 'Content-Length': '0'} 2024-02-01T10:48:02.325 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003020a85f1f99fd1f-0065bb76e2-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:02.373 DEBUG:boto:b'vclfcn-59obj41000falseobj42024-02-01T10:46:46.190Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:02.374 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:02.374 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:02.374 DEBUG:boto:path=/vclfcn-59/?prefix=obj4 2024-02-01T10:48:02.374 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj4 2024-02-01T10:48:02.374 DEBUG:boto:Method: GET 2024-02-01T10:48:02.374 DEBUG:boto:Path: /vclfcn-59/?prefix=obj4 2024-02-01T10:48:02.374 DEBUG:boto:Data: 2024-02-01T10:48:02.375 DEBUG:boto:Headers: {} 2024-02-01T10:48:02.375 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:02.375 DEBUG:boto:Port: 8000 2024-02-01T10:48:02.375 DEBUG:boto:Params: {} 2024-02-01T10:48:02.375 DEBUG:boto:Token: None 2024-02-01T10:48:02.375 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:02 GMT /vclfcn-59/ 2024-02-01T10:48:02.375 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:c2RgMycaHPRZMhj4ypc9JNhplUI= 2024-02-01T10:48:02.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:c2RgMycaHPRZMhj4ypc9JNhplUI=', 'Content-Length': '0'} 2024-02-01T10:48:02.380 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037d62a37bb8398c2-0065bb76e2-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:02.380 DEBUG:boto:b'vclfcn-59obj41000falseobj42024-02-01T10:46:46.190Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:02.380 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-02-01T10:48:02.381 DEBUG:boto:path=/vclfcn-59/obj5 2024-02-01T10:48:02.381 DEBUG:boto:auth_path=/vclfcn-59/obj5 2024-02-01T10:48:02.381 DEBUG:boto:path=/vclfcn-59/obj5?versionId=null 2024-02-01T10:48:02.381 DEBUG:boto:auth_path=/vclfcn-59/obj5?versionId=null 2024-02-01T10:48:02.381 DEBUG:boto:Method: GET 2024-02-01T10:48:02.381 DEBUG:boto:Path: /vclfcn-59/obj5?versionId=null 2024-02-01T10:48:02.382 DEBUG:boto:Data: 2024-02-01T10:48:02.382 DEBUG:boto:Headers: {} 2024-02-01T10:48:02.382 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:02.382 DEBUG:boto:Port: 8000 2024-02-01T10:48:02.382 DEBUG:boto:Params: {} 2024-02-01T10:48:02.383 DEBUG:boto:Token: None 2024-02-01T10:48:02.383 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:02 GMT /vclfcn-59/obj5?versionId=null 2024-02-01T10:48:02.383 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:MCTzjuje/Snd3uOjM/XIHyCxY7M= 2024-02-01T10:48:02.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:MCTzjuje/Snd3uOjM/XIHyCxY7M=', 'Content-Length': '0'} 2024-02-01T10:48:02.386 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006a5c696488a699e4-0065bb76e2-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:02.386 DEBUG:boto:path=/vclfcn-59/obj5 2024-02-01T10:48:02.387 DEBUG:boto:auth_path=/vclfcn-59/obj5 2024-02-01T10:48:02.387 DEBUG:boto:path=/vclfcn-59/obj5?versionId=null 2024-02-01T10:48:02.387 DEBUG:boto:auth_path=/vclfcn-59/obj5?versionId=null 2024-02-01T10:48:02.387 DEBUG:boto:Method: GET 2024-02-01T10:48:02.387 DEBUG:boto:Path: /vclfcn-59/obj5?versionId=null 2024-02-01T10:48:02.387 DEBUG:boto:Data: 2024-02-01T10:48:02.387 DEBUG:boto:Headers: {} 2024-02-01T10:48:02.387 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:02.387 DEBUG:boto:Port: 8000 2024-02-01T10:48:02.387 DEBUG:boto:Params: {} 2024-02-01T10:48:02.387 DEBUG:boto:Token: None 2024-02-01T10:48:02.388 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:02 GMT /vclfcn-59/obj5?versionId=null 2024-02-01T10:48:02.388 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:MCTzjuje/Snd3uOjM/XIHyCxY7M= 2024-02-01T10:48:02.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:MCTzjuje/Snd3uOjM/XIHyCxY7M=', 'Content-Length': '0'} 2024-02-01T10:48:02.391 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000b6d69a94ab25338-0065bb76e2-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:02.432 DEBUG:boto:path=/vclfcn-59/obj5 2024-02-01T10:48:02.432 DEBUG:boto:auth_path=/vclfcn-59/obj5 2024-02-01T10:48:02.432 DEBUG:boto:path=/vclfcn-59/obj5?versionId=null 2024-02-01T10:48:02.433 DEBUG:boto:auth_path=/vclfcn-59/obj5?versionId=null 2024-02-01T10:48:02.433 DEBUG:boto:Method: HEAD 2024-02-01T10:48:02.434 DEBUG:boto:Path: /vclfcn-59/obj5?versionId=null 2024-02-01T10:48:02.435 DEBUG:boto:Data: 2024-02-01T10:48:02.435 DEBUG:boto:Headers: {} 2024-02-01T10:48:02.436 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:02.436 DEBUG:boto:Port: 8000 2024-02-01T10:48:02.436 DEBUG:boto:Params: {} 2024-02-01T10:48:02.436 DEBUG:boto:Token: None 2024-02-01T10:48:02.437 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:02 GMT /vclfcn-59/obj5?versionId=null 2024-02-01T10:48:02.437 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:LaZK9bL4JHXXFtS7aMi+m53eHbc= 2024-02-01T10:48:02.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:LaZK9bL4JHXXFtS7aMi+m53eHbc=', 'Content-Length': '0'} 2024-02-01T10:48:02.440 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000306e9780177efff9-0065bb76e2-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:02.441 DEBUG:boto:path=/vclfcn-59/obj5 2024-02-01T10:48:02.441 DEBUG:boto:auth_path=/vclfcn-59/obj5 2024-02-01T10:48:02.441 DEBUG:boto:path=/vclfcn-59/obj5?versionId=null 2024-02-01T10:48:02.441 DEBUG:boto:auth_path=/vclfcn-59/obj5?versionId=null 2024-02-01T10:48:02.441 DEBUG:boto:Method: HEAD 2024-02-01T10:48:02.441 DEBUG:boto:Path: /vclfcn-59/obj5?versionId=null 2024-02-01T10:48:02.441 DEBUG:boto:Data: 2024-02-01T10:48:02.441 DEBUG:boto:Headers: {} 2024-02-01T10:48:02.441 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:02.442 DEBUG:boto:Port: 8000 2024-02-01T10:48:02.442 DEBUG:boto:Params: {} 2024-02-01T10:48:02.442 DEBUG:boto:Token: None 2024-02-01T10:48:02.442 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:02 GMT /vclfcn-59/obj5?versionId=null 2024-02-01T10:48:02.442 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:LaZK9bL4JHXXFtS7aMi+m53eHbc= 2024-02-01T10:48:02.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:LaZK9bL4JHXXFtS7aMi+m53eHbc=', 'Content-Length': '0'} 2024-02-01T10:48:02.445 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000018c45f1ea570b704-0065bb76e2-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:02.445 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-02-01T10:48:02.445 DEBUG:boto:path=/vclfcn-59/obj5 2024-02-01T10:48:02.445 DEBUG:boto:auth_path=/vclfcn-59/obj5 2024-02-01T10:48:02.446 DEBUG:boto:path=/vclfcn-59/obj5?versionId=null 2024-02-01T10:48:02.446 DEBUG:boto:auth_path=/vclfcn-59/obj5?versionId=null 2024-02-01T10:48:02.446 DEBUG:boto:Method: GET 2024-02-01T10:48:02.446 DEBUG:boto:Path: /vclfcn-59/obj5?versionId=null 2024-02-01T10:48:02.446 DEBUG:boto:Data: 2024-02-01T10:48:02.446 DEBUG:boto:Headers: {} 2024-02-01T10:48:02.446 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:02.446 DEBUG:boto:Port: 8000 2024-02-01T10:48:02.446 DEBUG:boto:Params: {} 2024-02-01T10:48:02.446 DEBUG:boto:Token: None 2024-02-01T10:48:02.447 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:02 GMT /vclfcn-59/obj5?versionId=null 2024-02-01T10:48:02.447 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:MCTzjuje/Snd3uOjM/XIHyCxY7M= 2024-02-01T10:48:02.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:MCTzjuje/Snd3uOjM/XIHyCxY7M=', 'Content-Length': '0'} 2024-02-01T10:48:02.450 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000055b9abd04a98ba4c-0065bb76e2-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:02.492 DEBUG:boto:path=/vclfcn-59/obj5 2024-02-01T10:48:02.492 DEBUG:boto:auth_path=/vclfcn-59/obj5 2024-02-01T10:48:02.493 DEBUG:boto:path=/vclfcn-59/obj5?versionId=null 2024-02-01T10:48:02.493 DEBUG:boto:auth_path=/vclfcn-59/obj5?versionId=null 2024-02-01T10:48:02.493 DEBUG:boto:Method: GET 2024-02-01T10:48:02.493 DEBUG:boto:Path: /vclfcn-59/obj5?versionId=null 2024-02-01T10:48:02.493 DEBUG:boto:Data: 2024-02-01T10:48:02.494 DEBUG:boto:Headers: {} 2024-02-01T10:48:02.494 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:02.494 DEBUG:boto:Port: 8000 2024-02-01T10:48:02.494 DEBUG:boto:Params: {} 2024-02-01T10:48:02.494 DEBUG:boto:Token: None 2024-02-01T10:48:02.495 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:02 GMT /vclfcn-59/obj5?versionId=null 2024-02-01T10:48:02.495 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:MCTzjuje/Snd3uOjM/XIHyCxY7M= 2024-02-01T10:48:02.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:MCTzjuje/Snd3uOjM/XIHyCxY7M=', 'Content-Length': '0'} 2024-02-01T10:48:02.499 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002b81abf36e79fecf-0065bb76e2-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:02.499 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:02.499 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:02.500 DEBUG:boto:path=/vclfcn-59/?prefix=obj5 2024-02-01T10:48:02.500 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj5 2024-02-01T10:48:02.500 DEBUG:boto:Method: GET 2024-02-01T10:48:02.500 DEBUG:boto:Path: /vclfcn-59/?prefix=obj5 2024-02-01T10:48:02.500 DEBUG:boto:Data: 2024-02-01T10:48:02.501 DEBUG:boto:Headers: {} 2024-02-01T10:48:02.501 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:02.501 DEBUG:boto:Port: 8000 2024-02-01T10:48:02.501 DEBUG:boto:Params: {} 2024-02-01T10:48:02.502 DEBUG:boto:Token: None 2024-02-01T10:48:02.502 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:02 GMT /vclfcn-59/ 2024-02-01T10:48:02.502 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:c2RgMycaHPRZMhj4ypc9JNhplUI= 2024-02-01T10:48:02.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:c2RgMycaHPRZMhj4ypc9JNhplUI=', 'Content-Length': '0'} 2024-02-01T10:48:02.507 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ab443e7b6dc1893-0065bb76e2-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:02.552 DEBUG:boto:b'vclfcn-59obj51000falseobj52024-02-01T10:47:30.206Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:02.553 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:02.553 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:02.553 DEBUG:boto:path=/vclfcn-59/?prefix=obj5 2024-02-01T10:48:02.554 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj5 2024-02-01T10:48:02.554 DEBUG:boto:Method: GET 2024-02-01T10:48:02.554 DEBUG:boto:Path: /vclfcn-59/?prefix=obj5 2024-02-01T10:48:02.554 DEBUG:boto:Data: 2024-02-01T10:48:02.554 DEBUG:boto:Headers: {} 2024-02-01T10:48:02.555 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:02.555 DEBUG:boto:Port: 8000 2024-02-01T10:48:02.555 DEBUG:boto:Params: {} 2024-02-01T10:48:02.555 DEBUG:boto:Token: None 2024-02-01T10:48:02.556 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:02 GMT /vclfcn-59/ 2024-02-01T10:48:02.556 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:c2RgMycaHPRZMhj4ypc9JNhplUI= 2024-02-01T10:48:02.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:c2RgMycaHPRZMhj4ypc9JNhplUI=', 'Content-Length': '0'} 2024-02-01T10:48:02.561 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a530c12432eff709-0065bb76e2-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:02.561 DEBUG:boto:b'vclfcn-59obj51000falseobj52024-02-01T10:47:30.206Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:02.561 DEBUG:boto:path=/vclfcn-59/obj5 2024-02-01T10:48:02.562 DEBUG:boto:auth_path=/vclfcn-59/obj5 2024-02-01T10:48:02.562 DEBUG:boto:Method: HEAD 2024-02-01T10:48:02.562 DEBUG:boto:Path: /vclfcn-59/obj5 2024-02-01T10:48:02.562 DEBUG:boto:Data: 2024-02-01T10:48:02.562 DEBUG:boto:Headers: {} 2024-02-01T10:48:02.563 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:02.563 DEBUG:boto:Port: 8000 2024-02-01T10:48:02.563 DEBUG:boto:Params: {} 2024-02-01T10:48:02.563 DEBUG:boto:Token: None 2024-02-01T10:48:02.564 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:02 GMT /vclfcn-59/obj5 2024-02-01T10:48:02.564 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ubTeoLSpJLRQRS209hOTgqxzZ9g= 2024-02-01T10:48:02.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ubTeoLSpJLRQRS209hOTgqxzZ9g=', 'Content-Length': '0'} 2024-02-01T10:48:02.567 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000894924bb9cf2e90f-0065bb76e2-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:02.567 DEBUG:boto:path=/vclfcn-59/obj5 2024-02-01T10:48:02.567 DEBUG:boto:auth_path=/vclfcn-59/obj5 2024-02-01T10:48:02.568 DEBUG:boto:Method: HEAD 2024-02-01T10:48:02.568 DEBUG:boto:Path: /vclfcn-59/obj5 2024-02-01T10:48:02.568 DEBUG:boto:Data: 2024-02-01T10:48:02.568 DEBUG:boto:Headers: {} 2024-02-01T10:48:02.568 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:02.568 DEBUG:boto:Port: 8000 2024-02-01T10:48:02.568 DEBUG:boto:Params: {} 2024-02-01T10:48:02.569 DEBUG:boto:Token: None 2024-02-01T10:48:02.569 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:02 GMT /vclfcn-59/obj5 2024-02-01T10:48:02.569 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ubTeoLSpJLRQRS209hOTgqxzZ9g= 2024-02-01T10:48:02.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ubTeoLSpJLRQRS209hOTgqxzZ9g=', 'Content-Length': '0'} 2024-02-01T10:48:02.572 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000006dd5cb6c26e78f1-0065bb76e2-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:02.572 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-02-01T10:48:02.573 DEBUG:boto:path=/vclfcn-59/obj5 2024-02-01T10:48:02.573 DEBUG:boto:auth_path=/vclfcn-59/obj5 2024-02-01T10:48:02.573 DEBUG:boto:Method: GET 2024-02-01T10:48:02.573 DEBUG:boto:Path: /vclfcn-59/obj5 2024-02-01T10:48:02.573 DEBUG:boto:Data: 2024-02-01T10:48:02.573 DEBUG:boto:Headers: {} 2024-02-01T10:48:02.573 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:02.574 DEBUG:boto:Port: 8000 2024-02-01T10:48:02.574 DEBUG:boto:Params: {} 2024-02-01T10:48:02.574 DEBUG:boto:Token: None 2024-02-01T10:48:02.574 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:02 GMT /vclfcn-59/obj5 2024-02-01T10:48:02.574 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nJLvWWBYUp8tsVFroyAH46nURgQ= 2024-02-01T10:48:02.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nJLvWWBYUp8tsVFroyAH46nURgQ=', 'Content-Length': '0'} 2024-02-01T10:48:02.577 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000944b00a6bf0608ed-0065bb76e2-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:02.620 DEBUG:boto:path=/vclfcn-59/obj5 2024-02-01T10:48:02.621 DEBUG:boto:auth_path=/vclfcn-59/obj5 2024-02-01T10:48:02.621 DEBUG:boto:Method: GET 2024-02-01T10:48:02.621 DEBUG:boto:Path: /vclfcn-59/obj5 2024-02-01T10:48:02.621 DEBUG:boto:Data: 2024-02-01T10:48:02.621 DEBUG:boto:Headers: {} 2024-02-01T10:48:02.622 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:02.622 DEBUG:boto:Port: 8000 2024-02-01T10:48:02.622 DEBUG:boto:Params: {} 2024-02-01T10:48:02.622 DEBUG:boto:Token: None 2024-02-01T10:48:02.623 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:02 GMT /vclfcn-59/obj5 2024-02-01T10:48:02.623 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nJLvWWBYUp8tsVFroyAH46nURgQ= 2024-02-01T10:48:02.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nJLvWWBYUp8tsVFroyAH46nURgQ=', 'Content-Length': '0'} 2024-02-01T10:48:02.626 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008f52ca844b92b65b-0065bb76e2-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:02.627 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:02.627 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:02.628 DEBUG:boto:path=/vclfcn-59/?prefix=obj5 2024-02-01T10:48:02.628 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj5 2024-02-01T10:48:02.628 DEBUG:boto:Method: GET 2024-02-01T10:48:02.628 DEBUG:boto:Path: /vclfcn-59/?prefix=obj5 2024-02-01T10:48:02.628 DEBUG:boto:Data: 2024-02-01T10:48:02.629 DEBUG:boto:Headers: {} 2024-02-01T10:48:02.629 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:02.629 DEBUG:boto:Port: 8000 2024-02-01T10:48:02.629 DEBUG:boto:Params: {} 2024-02-01T10:48:02.629 DEBUG:boto:Token: None 2024-02-01T10:48:02.630 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:02 GMT /vclfcn-59/ 2024-02-01T10:48:02.630 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:c2RgMycaHPRZMhj4ypc9JNhplUI= 2024-02-01T10:48:02.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:c2RgMycaHPRZMhj4ypc9JNhplUI=', 'Content-Length': '0'} 2024-02-01T10:48:02.635 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6b148dcc78b9397-0065bb76e2-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:02.635 DEBUG:boto:b'vclfcn-59obj51000falseobj52024-02-01T10:47:30.206Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:02.636 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:02.636 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:02.636 DEBUG:boto:path=/vclfcn-59/?prefix=obj5 2024-02-01T10:48:02.637 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj5 2024-02-01T10:48:02.637 DEBUG:boto:Method: GET 2024-02-01T10:48:02.637 DEBUG:boto:Path: /vclfcn-59/?prefix=obj5 2024-02-01T10:48:02.637 DEBUG:boto:Data: 2024-02-01T10:48:02.637 DEBUG:boto:Headers: {} 2024-02-01T10:48:02.638 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:02.638 DEBUG:boto:Port: 8000 2024-02-01T10:48:02.638 DEBUG:boto:Params: {} 2024-02-01T10:48:02.638 DEBUG:boto:Token: None 2024-02-01T10:48:02.639 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:02 GMT /vclfcn-59/ 2024-02-01T10:48:02.639 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:c2RgMycaHPRZMhj4ypc9JNhplUI= 2024-02-01T10:48:02.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:c2RgMycaHPRZMhj4ypc9JNhplUI=', 'Content-Length': '0'} 2024-02-01T10:48:02.644 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010042bd5c3a37ff0-0065bb76e2-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:02.684 DEBUG:boto:b'vclfcn-59obj51000falseobj52024-02-01T10:47:30.206Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:02.685 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-02-01T10:48:02.685 DEBUG:boto:path=/vclfcn-59/obj6 2024-02-01T10:48:02.685 DEBUG:boto:auth_path=/vclfcn-59/obj6 2024-02-01T10:48:02.685 DEBUG:boto:path=/vclfcn-59/obj6?versionId=null 2024-02-01T10:48:02.686 DEBUG:boto:auth_path=/vclfcn-59/obj6?versionId=null 2024-02-01T10:48:02.686 DEBUG:boto:Method: GET 2024-02-01T10:48:02.686 DEBUG:boto:Path: /vclfcn-59/obj6?versionId=null 2024-02-01T10:48:02.686 DEBUG:boto:Data: 2024-02-01T10:48:02.686 DEBUG:boto:Headers: {} 2024-02-01T10:48:02.686 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:02.687 DEBUG:boto:Port: 8000 2024-02-01T10:48:02.687 DEBUG:boto:Params: {} 2024-02-01T10:48:02.687 DEBUG:boto:Token: None 2024-02-01T10:48:02.687 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:02 GMT /vclfcn-59/obj6?versionId=null 2024-02-01T10:48:02.688 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:VyoZFVAItTWPg0DwnkhvHHw4aIg= 2024-02-01T10:48:02.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:VyoZFVAItTWPg0DwnkhvHHw4aIg=', 'Content-Length': '0'} 2024-02-01T10:48:02.691 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000086c34c9d64aad227-0065bb76e2-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:02.736 DEBUG:boto:path=/vclfcn-59/obj6 2024-02-01T10:48:02.736 DEBUG:boto:auth_path=/vclfcn-59/obj6 2024-02-01T10:48:02.736 DEBUG:boto:path=/vclfcn-59/obj6?versionId=null 2024-02-01T10:48:02.736 DEBUG:boto:auth_path=/vclfcn-59/obj6?versionId=null 2024-02-01T10:48:02.737 DEBUG:boto:Method: GET 2024-02-01T10:48:02.737 DEBUG:boto:Path: /vclfcn-59/obj6?versionId=null 2024-02-01T10:48:02.737 DEBUG:boto:Data: 2024-02-01T10:48:02.737 DEBUG:boto:Headers: {} 2024-02-01T10:48:02.737 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:02.737 DEBUG:boto:Port: 8000 2024-02-01T10:48:02.737 DEBUG:boto:Params: {} 2024-02-01T10:48:02.737 DEBUG:boto:Token: None 2024-02-01T10:48:02.737 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:02 GMT /vclfcn-59/obj6?versionId=null 2024-02-01T10:48:02.738 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:VyoZFVAItTWPg0DwnkhvHHw4aIg= 2024-02-01T10:48:02.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:VyoZFVAItTWPg0DwnkhvHHw4aIg=', 'Content-Length': '0'} 2024-02-01T10:48:02.740 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ca59b331e4a3c280-0065bb76e2-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:02.741 DEBUG:boto:path=/vclfcn-59/obj6 2024-02-01T10:48:02.741 DEBUG:boto:auth_path=/vclfcn-59/obj6 2024-02-01T10:48:02.741 DEBUG:boto:path=/vclfcn-59/obj6?versionId=null 2024-02-01T10:48:02.741 DEBUG:boto:auth_path=/vclfcn-59/obj6?versionId=null 2024-02-01T10:48:02.741 DEBUG:boto:Method: HEAD 2024-02-01T10:48:02.741 DEBUG:boto:Path: /vclfcn-59/obj6?versionId=null 2024-02-01T10:48:02.741 DEBUG:boto:Data: 2024-02-01T10:48:02.741 DEBUG:boto:Headers: {} 2024-02-01T10:48:02.742 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:02.742 DEBUG:boto:Port: 8000 2024-02-01T10:48:02.742 DEBUG:boto:Params: {} 2024-02-01T10:48:02.742 DEBUG:boto:Token: None 2024-02-01T10:48:02.742 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:02 GMT /vclfcn-59/obj6?versionId=null 2024-02-01T10:48:02.742 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:UxQWUMxX6w9/YACKqfjq8IPt2vY= 2024-02-01T10:48:02.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:UxQWUMxX6w9/YACKqfjq8IPt2vY=', 'Content-Length': '0'} 2024-02-01T10:48:02.745 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000af2fdc58a199bbf7-0065bb76e2-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:02.745 DEBUG:boto:path=/vclfcn-59/obj6 2024-02-01T10:48:02.745 DEBUG:boto:auth_path=/vclfcn-59/obj6 2024-02-01T10:48:02.745 DEBUG:boto:path=/vclfcn-59/obj6?versionId=null 2024-02-01T10:48:02.746 DEBUG:boto:auth_path=/vclfcn-59/obj6?versionId=null 2024-02-01T10:48:02.746 DEBUG:boto:Method: HEAD 2024-02-01T10:48:02.746 DEBUG:boto:Path: /vclfcn-59/obj6?versionId=null 2024-02-01T10:48:02.746 DEBUG:boto:Data: 2024-02-01T10:48:02.746 DEBUG:boto:Headers: {} 2024-02-01T10:48:02.746 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:02.746 DEBUG:boto:Port: 8000 2024-02-01T10:48:02.746 DEBUG:boto:Params: {} 2024-02-01T10:48:02.746 DEBUG:boto:Token: None 2024-02-01T10:48:02.746 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:02 GMT /vclfcn-59/obj6?versionId=null 2024-02-01T10:48:02.747 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:UxQWUMxX6w9/YACKqfjq8IPt2vY= 2024-02-01T10:48:02.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:UxQWUMxX6w9/YACKqfjq8IPt2vY=', 'Content-Length': '0'} 2024-02-01T10:48:02.749 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ec7d206d63c72cdd-0065bb76e2-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:02.749 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-02-01T10:48:02.750 DEBUG:boto:path=/vclfcn-59/obj6 2024-02-01T10:48:02.750 DEBUG:boto:auth_path=/vclfcn-59/obj6 2024-02-01T10:48:02.750 DEBUG:boto:path=/vclfcn-59/obj6?versionId=null 2024-02-01T10:48:02.750 DEBUG:boto:auth_path=/vclfcn-59/obj6?versionId=null 2024-02-01T10:48:02.750 DEBUG:boto:Method: GET 2024-02-01T10:48:02.750 DEBUG:boto:Path: /vclfcn-59/obj6?versionId=null 2024-02-01T10:48:02.750 DEBUG:boto:Data: 2024-02-01T10:48:02.750 DEBUG:boto:Headers: {} 2024-02-01T10:48:02.750 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:02.750 DEBUG:boto:Port: 8000 2024-02-01T10:48:02.751 DEBUG:boto:Params: {} 2024-02-01T10:48:02.751 DEBUG:boto:Token: None 2024-02-01T10:48:02.751 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:02 GMT /vclfcn-59/obj6?versionId=null 2024-02-01T10:48:02.751 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:VyoZFVAItTWPg0DwnkhvHHw4aIg= 2024-02-01T10:48:02.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:VyoZFVAItTWPg0DwnkhvHHw4aIg=', 'Content-Length': '0'} 2024-02-01T10:48:02.754 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000348ff026597fd7d7-0065bb76e2-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:02.796 DEBUG:boto:path=/vclfcn-59/obj6 2024-02-01T10:48:02.796 DEBUG:boto:auth_path=/vclfcn-59/obj6 2024-02-01T10:48:02.796 DEBUG:boto:path=/vclfcn-59/obj6?versionId=null 2024-02-01T10:48:02.796 DEBUG:boto:auth_path=/vclfcn-59/obj6?versionId=null 2024-02-01T10:48:02.797 DEBUG:boto:Method: GET 2024-02-01T10:48:02.797 DEBUG:boto:Path: /vclfcn-59/obj6?versionId=null 2024-02-01T10:48:02.797 DEBUG:boto:Data: 2024-02-01T10:48:02.797 DEBUG:boto:Headers: {} 2024-02-01T10:48:02.797 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:02.797 DEBUG:boto:Port: 8000 2024-02-01T10:48:02.797 DEBUG:boto:Params: {} 2024-02-01T10:48:02.797 DEBUG:boto:Token: None 2024-02-01T10:48:02.797 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:02 GMT /vclfcn-59/obj6?versionId=null 2024-02-01T10:48:02.798 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:VyoZFVAItTWPg0DwnkhvHHw4aIg= 2024-02-01T10:48:02.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:VyoZFVAItTWPg0DwnkhvHHw4aIg=', 'Content-Length': '0'} 2024-02-01T10:48:02.800 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fe9f0a387c37b524-0065bb76e2-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:02.801 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:02.801 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:02.801 DEBUG:boto:path=/vclfcn-59/?prefix=obj6 2024-02-01T10:48:02.802 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj6 2024-02-01T10:48:02.802 DEBUG:boto:Method: GET 2024-02-01T10:48:02.802 DEBUG:boto:Path: /vclfcn-59/?prefix=obj6 2024-02-01T10:48:02.802 DEBUG:boto:Data: 2024-02-01T10:48:02.802 DEBUG:boto:Headers: {} 2024-02-01T10:48:02.802 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:02.802 DEBUG:boto:Port: 8000 2024-02-01T10:48:02.802 DEBUG:boto:Params: {} 2024-02-01T10:48:02.803 DEBUG:boto:Token: None 2024-02-01T10:48:02.803 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:02 GMT /vclfcn-59/ 2024-02-01T10:48:02.803 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:c2RgMycaHPRZMhj4ypc9JNhplUI= 2024-02-01T10:48:02.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:c2RgMycaHPRZMhj4ypc9JNhplUI=', 'Content-Length': '0'} 2024-02-01T10:48:02.807 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6481cf637ad9cfb-0065bb76e2-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:02.848 DEBUG:boto:b'vclfcn-59obj61000falseobj62024-02-01T10:47:30.217Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:02.849 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:02.849 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:02.849 DEBUG:boto:path=/vclfcn-59/?prefix=obj6 2024-02-01T10:48:02.849 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj6 2024-02-01T10:48:02.849 DEBUG:boto:Method: GET 2024-02-01T10:48:02.849 DEBUG:boto:Path: /vclfcn-59/?prefix=obj6 2024-02-01T10:48:02.849 DEBUG:boto:Data: 2024-02-01T10:48:02.850 DEBUG:boto:Headers: {} 2024-02-01T10:48:02.850 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:02.850 DEBUG:boto:Port: 8000 2024-02-01T10:48:02.850 DEBUG:boto:Params: {} 2024-02-01T10:48:02.850 DEBUG:boto:Token: None 2024-02-01T10:48:02.850 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:02 GMT /vclfcn-59/ 2024-02-01T10:48:02.851 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:c2RgMycaHPRZMhj4ypc9JNhplUI= 2024-02-01T10:48:02.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:c2RgMycaHPRZMhj4ypc9JNhplUI=', 'Content-Length': '0'} 2024-02-01T10:48:02.855 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d3cfea4c9f18157-0065bb76e2-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:02.855 DEBUG:boto:b'vclfcn-59obj61000falseobj62024-02-01T10:47:30.217Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:02.856 DEBUG:boto:path=/vclfcn-59/obj6 2024-02-01T10:48:02.856 DEBUG:boto:auth_path=/vclfcn-59/obj6 2024-02-01T10:48:02.856 DEBUG:boto:Method: HEAD 2024-02-01T10:48:02.856 DEBUG:boto:Path: /vclfcn-59/obj6 2024-02-01T10:48:02.856 DEBUG:boto:Data: 2024-02-01T10:48:02.857 DEBUG:boto:Headers: {} 2024-02-01T10:48:02.857 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:02.857 DEBUG:boto:Port: 8000 2024-02-01T10:48:02.857 DEBUG:boto:Params: {} 2024-02-01T10:48:02.857 DEBUG:boto:Token: None 2024-02-01T10:48:02.857 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:02 GMT /vclfcn-59/obj6 2024-02-01T10:48:02.857 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cRDsNbJaK2H5oSY9bE5C/KbAS1w= 2024-02-01T10:48:02.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cRDsNbJaK2H5oSY9bE5C/KbAS1w=', 'Content-Length': '0'} 2024-02-01T10:48:02.860 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a379578958e11d78-0065bb76e2-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:02.860 DEBUG:boto:path=/vclfcn-59/obj6 2024-02-01T10:48:02.861 DEBUG:boto:auth_path=/vclfcn-59/obj6 2024-02-01T10:48:02.861 DEBUG:boto:Method: HEAD 2024-02-01T10:48:02.861 DEBUG:boto:Path: /vclfcn-59/obj6 2024-02-01T10:48:02.861 DEBUG:boto:Data: 2024-02-01T10:48:02.861 DEBUG:boto:Headers: {} 2024-02-01T10:48:02.861 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:02.861 DEBUG:boto:Port: 8000 2024-02-01T10:48:02.861 DEBUG:boto:Params: {} 2024-02-01T10:48:02.862 DEBUG:boto:Token: None 2024-02-01T10:48:02.862 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:02 GMT /vclfcn-59/obj6 2024-02-01T10:48:02.862 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cRDsNbJaK2H5oSY9bE5C/KbAS1w= 2024-02-01T10:48:02.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cRDsNbJaK2H5oSY9bE5C/KbAS1w=', 'Content-Length': '0'} 2024-02-01T10:48:02.864 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f2c9f5c58728e6c3-0065bb76e2-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:02.865 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-02-01T10:48:02.865 DEBUG:boto:path=/vclfcn-59/obj6 2024-02-01T10:48:02.865 DEBUG:boto:auth_path=/vclfcn-59/obj6 2024-02-01T10:48:02.865 DEBUG:boto:Method: GET 2024-02-01T10:48:02.865 DEBUG:boto:Path: /vclfcn-59/obj6 2024-02-01T10:48:02.865 DEBUG:boto:Data: 2024-02-01T10:48:02.865 DEBUG:boto:Headers: {} 2024-02-01T10:48:02.865 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:02.866 DEBUG:boto:Port: 8000 2024-02-01T10:48:02.866 DEBUG:boto:Params: {} 2024-02-01T10:48:02.866 DEBUG:boto:Token: None 2024-02-01T10:48:02.866 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:02 GMT /vclfcn-59/obj6 2024-02-01T10:48:02.866 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:hQ3BA5EIGk4hnhdYNzsTTPp4BfU= 2024-02-01T10:48:02.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:hQ3BA5EIGk4hnhdYNzsTTPp4BfU=', 'Content-Length': '0'} 2024-02-01T10:48:02.869 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000481d63bde28a5f7d-0065bb76e2-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:02.912 DEBUG:boto:path=/vclfcn-59/obj6 2024-02-01T10:48:02.912 DEBUG:boto:auth_path=/vclfcn-59/obj6 2024-02-01T10:48:02.912 DEBUG:boto:Method: GET 2024-02-01T10:48:02.912 DEBUG:boto:Path: /vclfcn-59/obj6 2024-02-01T10:48:02.913 DEBUG:boto:Data: 2024-02-01T10:48:02.913 DEBUG:boto:Headers: {} 2024-02-01T10:48:02.913 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:02.913 DEBUG:boto:Port: 8000 2024-02-01T10:48:02.913 DEBUG:boto:Params: {} 2024-02-01T10:48:02.913 DEBUG:boto:Token: None 2024-02-01T10:48:02.913 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:02 GMT /vclfcn-59/obj6 2024-02-01T10:48:02.913 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:hQ3BA5EIGk4hnhdYNzsTTPp4BfU= 2024-02-01T10:48:02.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:hQ3BA5EIGk4hnhdYNzsTTPp4BfU=', 'Content-Length': '0'} 2024-02-01T10:48:02.916 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cdabcf305eb318f9-0065bb76e2-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:02.917 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:02.917 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:02.917 DEBUG:boto:path=/vclfcn-59/?prefix=obj6 2024-02-01T10:48:02.917 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj6 2024-02-01T10:48:02.917 DEBUG:boto:Method: GET 2024-02-01T10:48:02.917 DEBUG:boto:Path: /vclfcn-59/?prefix=obj6 2024-02-01T10:48:02.918 DEBUG:boto:Data: 2024-02-01T10:48:02.918 DEBUG:boto:Headers: {} 2024-02-01T10:48:02.918 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:02.918 DEBUG:boto:Port: 8000 2024-02-01T10:48:02.918 DEBUG:boto:Params: {} 2024-02-01T10:48:02.918 DEBUG:boto:Token: None 2024-02-01T10:48:02.918 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:02 GMT /vclfcn-59/ 2024-02-01T10:48:02.918 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:c2RgMycaHPRZMhj4ypc9JNhplUI= 2024-02-01T10:48:02.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:c2RgMycaHPRZMhj4ypc9JNhplUI=', 'Content-Length': '0'} 2024-02-01T10:48:02.923 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000313f353b94838b82-0065bb76e2-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:02.968 DEBUG:boto:b'vclfcn-59obj61000falseobj62024-02-01T10:47:30.217Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:02.968 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:02.969 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:02.969 DEBUG:boto:path=/vclfcn-59/?prefix=obj6 2024-02-01T10:48:02.969 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj6 2024-02-01T10:48:02.969 DEBUG:boto:Method: GET 2024-02-01T10:48:02.969 DEBUG:boto:Path: /vclfcn-59/?prefix=obj6 2024-02-01T10:48:02.969 DEBUG:boto:Data: 2024-02-01T10:48:02.969 DEBUG:boto:Headers: {} 2024-02-01T10:48:02.969 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:02.969 DEBUG:boto:Port: 8000 2024-02-01T10:48:02.969 DEBUG:boto:Params: {} 2024-02-01T10:48:02.969 DEBUG:boto:Token: None 2024-02-01T10:48:02.970 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:02 GMT /vclfcn-59/ 2024-02-01T10:48:02.970 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:c2RgMycaHPRZMhj4ypc9JNhplUI= 2024-02-01T10:48:02.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:c2RgMycaHPRZMhj4ypc9JNhplUI=', 'Content-Length': '0'} 2024-02-01T10:48:02.974 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000059eaa2970f07a779-0065bb76e2-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:02.974 DEBUG:boto:b'vclfcn-59obj61000falseobj62024-02-01T10:47:30.217Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:02.975 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-02-01T10:48:02.975 DEBUG:boto:path=/vclfcn-59/obj7 2024-02-01T10:48:02.975 DEBUG:boto:auth_path=/vclfcn-59/obj7 2024-02-01T10:48:02.975 DEBUG:boto:path=/vclfcn-59/obj7?versionId=null 2024-02-01T10:48:02.975 DEBUG:boto:auth_path=/vclfcn-59/obj7?versionId=null 2024-02-01T10:48:02.975 DEBUG:boto:Method: GET 2024-02-01T10:48:02.975 DEBUG:boto:Path: /vclfcn-59/obj7?versionId=null 2024-02-01T10:48:02.975 DEBUG:boto:Data: 2024-02-01T10:48:02.975 DEBUG:boto:Headers: {} 2024-02-01T10:48:02.975 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:02.975 DEBUG:boto:Port: 8000 2024-02-01T10:48:02.976 DEBUG:boto:Params: {} 2024-02-01T10:48:02.976 DEBUG:boto:Token: None 2024-02-01T10:48:02.976 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:02 GMT /vclfcn-59/obj7?versionId=null 2024-02-01T10:48:02.976 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:VLr0c+k3XoxN5wk7jXscEzxFLIA= 2024-02-01T10:48:02.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:VLr0c+k3XoxN5wk7jXscEzxFLIA=', 'Content-Length': '0'} 2024-02-01T10:48:02.979 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005779b11c0ecfe66a-0065bb76e2-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:02.979 DEBUG:boto:path=/vclfcn-59/obj7 2024-02-01T10:48:02.979 DEBUG:boto:auth_path=/vclfcn-59/obj7 2024-02-01T10:48:02.980 DEBUG:boto:path=/vclfcn-59/obj7?versionId=null 2024-02-01T10:48:02.980 DEBUG:boto:auth_path=/vclfcn-59/obj7?versionId=null 2024-02-01T10:48:02.980 DEBUG:boto:Method: GET 2024-02-01T10:48:02.980 DEBUG:boto:Path: /vclfcn-59/obj7?versionId=null 2024-02-01T10:48:02.980 DEBUG:boto:Data: 2024-02-01T10:48:02.980 DEBUG:boto:Headers: {} 2024-02-01T10:48:02.980 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:02.980 DEBUG:boto:Port: 8000 2024-02-01T10:48:02.980 DEBUG:boto:Params: {} 2024-02-01T10:48:02.981 DEBUG:boto:Token: None 2024-02-01T10:48:02.981 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:02 GMT /vclfcn-59/obj7?versionId=null 2024-02-01T10:48:02.981 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:VLr0c+k3XoxN5wk7jXscEzxFLIA= 2024-02-01T10:48:02.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:VLr0c+k3XoxN5wk7jXscEzxFLIA=', 'Content-Length': '0'} 2024-02-01T10:48:02.983 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000046176bd3bef3b420-0065bb76e2-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:03.024 DEBUG:boto:path=/vclfcn-59/obj7 2024-02-01T10:48:03.024 DEBUG:boto:auth_path=/vclfcn-59/obj7 2024-02-01T10:48:03.024 DEBUG:boto:path=/vclfcn-59/obj7?versionId=null 2024-02-01T10:48:03.024 DEBUG:boto:auth_path=/vclfcn-59/obj7?versionId=null 2024-02-01T10:48:03.025 DEBUG:boto:Method: HEAD 2024-02-01T10:48:03.025 DEBUG:boto:Path: /vclfcn-59/obj7?versionId=null 2024-02-01T10:48:03.025 DEBUG:boto:Data: 2024-02-01T10:48:03.025 DEBUG:boto:Headers: {} 2024-02-01T10:48:03.025 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:03.025 DEBUG:boto:Port: 8000 2024-02-01T10:48:03.025 DEBUG:boto:Params: {} 2024-02-01T10:48:03.026 DEBUG:boto:Token: None 2024-02-01T10:48:03.026 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:03 GMT /vclfcn-59/obj7?versionId=null 2024-02-01T10:48:03.026 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:a8XBcygcxp+uP5X307wDb8G2KPY= 2024-02-01T10:48:03.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:a8XBcygcxp+uP5X307wDb8G2KPY=', 'Content-Length': '0'} 2024-02-01T10:48:03.029 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004dfee81bfc76c2b1-0065bb76e3-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:03.029 DEBUG:boto:path=/vclfcn-59/obj7 2024-02-01T10:48:03.029 DEBUG:boto:auth_path=/vclfcn-59/obj7 2024-02-01T10:48:03.029 DEBUG:boto:path=/vclfcn-59/obj7?versionId=null 2024-02-01T10:48:03.030 DEBUG:boto:auth_path=/vclfcn-59/obj7?versionId=null 2024-02-01T10:48:03.030 DEBUG:boto:Method: HEAD 2024-02-01T10:48:03.030 DEBUG:boto:Path: /vclfcn-59/obj7?versionId=null 2024-02-01T10:48:03.030 DEBUG:boto:Data: 2024-02-01T10:48:03.030 DEBUG:boto:Headers: {} 2024-02-01T10:48:03.030 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:03.030 DEBUG:boto:Port: 8000 2024-02-01T10:48:03.030 DEBUG:boto:Params: {} 2024-02-01T10:48:03.030 DEBUG:boto:Token: None 2024-02-01T10:48:03.031 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:03 GMT /vclfcn-59/obj7?versionId=null 2024-02-01T10:48:03.031 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:a8XBcygcxp+uP5X307wDb8G2KPY= 2024-02-01T10:48:03.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:a8XBcygcxp+uP5X307wDb8G2KPY=', 'Content-Length': '0'} 2024-02-01T10:48:03.034 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000094e9830a0990632d-0065bb76e3-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:03.034 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-02-01T10:48:03.034 DEBUG:boto:path=/vclfcn-59/obj7 2024-02-01T10:48:03.034 DEBUG:boto:auth_path=/vclfcn-59/obj7 2024-02-01T10:48:03.034 DEBUG:boto:path=/vclfcn-59/obj7?versionId=null 2024-02-01T10:48:03.034 DEBUG:boto:auth_path=/vclfcn-59/obj7?versionId=null 2024-02-01T10:48:03.035 DEBUG:boto:Method: GET 2024-02-01T10:48:03.035 DEBUG:boto:Path: /vclfcn-59/obj7?versionId=null 2024-02-01T10:48:03.035 DEBUG:boto:Data: 2024-02-01T10:48:03.035 DEBUG:boto:Headers: {} 2024-02-01T10:48:03.035 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:03.035 DEBUG:boto:Port: 8000 2024-02-01T10:48:03.035 DEBUG:boto:Params: {} 2024-02-01T10:48:03.035 DEBUG:boto:Token: None 2024-02-01T10:48:03.035 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:03 GMT /vclfcn-59/obj7?versionId=null 2024-02-01T10:48:03.036 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:DE84i1Zr1fiAdYYv4mrPsDDrOy8= 2024-02-01T10:48:03.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:DE84i1Zr1fiAdYYv4mrPsDDrOy8=', 'Content-Length': '0'} 2024-02-01T10:48:03.038 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000428109c8c14fb844-0065bb76e3-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:03.080 DEBUG:boto:path=/vclfcn-59/obj7 2024-02-01T10:48:03.080 DEBUG:boto:auth_path=/vclfcn-59/obj7 2024-02-01T10:48:03.080 DEBUG:boto:path=/vclfcn-59/obj7?versionId=null 2024-02-01T10:48:03.081 DEBUG:boto:auth_path=/vclfcn-59/obj7?versionId=null 2024-02-01T10:48:03.081 DEBUG:boto:Method: GET 2024-02-01T10:48:03.081 DEBUG:boto:Path: /vclfcn-59/obj7?versionId=null 2024-02-01T10:48:03.081 DEBUG:boto:Data: 2024-02-01T10:48:03.081 DEBUG:boto:Headers: {} 2024-02-01T10:48:03.081 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:03.081 DEBUG:boto:Port: 8000 2024-02-01T10:48:03.081 DEBUG:boto:Params: {} 2024-02-01T10:48:03.081 DEBUG:boto:Token: None 2024-02-01T10:48:03.082 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:03 GMT /vclfcn-59/obj7?versionId=null 2024-02-01T10:48:03.082 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:DE84i1Zr1fiAdYYv4mrPsDDrOy8= 2024-02-01T10:48:03.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:DE84i1Zr1fiAdYYv4mrPsDDrOy8=', 'Content-Length': '0'} 2024-02-01T10:48:03.085 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a4162b6e46a48d4b-0065bb76e3-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:03.085 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:03.085 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:03.085 DEBUG:boto:path=/vclfcn-59/?prefix=obj7 2024-02-01T10:48:03.085 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj7 2024-02-01T10:48:03.085 DEBUG:boto:Method: GET 2024-02-01T10:48:03.086 DEBUG:boto:Path: /vclfcn-59/?prefix=obj7 2024-02-01T10:48:03.086 DEBUG:boto:Data: 2024-02-01T10:48:03.086 DEBUG:boto:Headers: {} 2024-02-01T10:48:03.086 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:03.086 DEBUG:boto:Port: 8000 2024-02-01T10:48:03.086 DEBUG:boto:Params: {} 2024-02-01T10:48:03.086 DEBUG:boto:Token: None 2024-02-01T10:48:03.086 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:03 GMT /vclfcn-59/ 2024-02-01T10:48:03.086 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nMZ+QSOdClyZcZD5pxyoNs9MRBQ= 2024-02-01T10:48:03.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nMZ+QSOdClyZcZD5pxyoNs9MRBQ=', 'Content-Length': '0'} 2024-02-01T10:48:03.091 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000751f576264c86a5a-0065bb76e3-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:03.132 DEBUG:boto:b'vclfcn-59obj71000falseobj72024-02-01T10:47:30.227Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:03.133 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:03.133 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:03.133 DEBUG:boto:path=/vclfcn-59/?prefix=obj7 2024-02-01T10:48:03.133 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj7 2024-02-01T10:48:03.133 DEBUG:boto:Method: GET 2024-02-01T10:48:03.133 DEBUG:boto:Path: /vclfcn-59/?prefix=obj7 2024-02-01T10:48:03.133 DEBUG:boto:Data: 2024-02-01T10:48:03.133 DEBUG:boto:Headers: {} 2024-02-01T10:48:03.133 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:03.133 DEBUG:boto:Port: 8000 2024-02-01T10:48:03.133 DEBUG:boto:Params: {} 2024-02-01T10:48:03.134 DEBUG:boto:Token: None 2024-02-01T10:48:03.134 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:03 GMT /vclfcn-59/ 2024-02-01T10:48:03.134 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nMZ+QSOdClyZcZD5pxyoNs9MRBQ= 2024-02-01T10:48:03.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nMZ+QSOdClyZcZD5pxyoNs9MRBQ=', 'Content-Length': '0'} 2024-02-01T10:48:03.138 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d72a0334b1d5b4d-0065bb76e3-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:03.139 DEBUG:boto:b'vclfcn-59obj71000falseobj72024-02-01T10:47:30.227Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:03.139 DEBUG:boto:path=/vclfcn-59/obj7 2024-02-01T10:48:03.139 DEBUG:boto:auth_path=/vclfcn-59/obj7 2024-02-01T10:48:03.139 DEBUG:boto:Method: HEAD 2024-02-01T10:48:03.139 DEBUG:boto:Path: /vclfcn-59/obj7 2024-02-01T10:48:03.139 DEBUG:boto:Data: 2024-02-01T10:48:03.139 DEBUG:boto:Headers: {} 2024-02-01T10:48:03.140 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:03.140 DEBUG:boto:Port: 8000 2024-02-01T10:48:03.140 DEBUG:boto:Params: {} 2024-02-01T10:48:03.140 DEBUG:boto:Token: None 2024-02-01T10:48:03.140 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:03 GMT /vclfcn-59/obj7 2024-02-01T10:48:03.140 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NoA+QyQJ2Y6SmkN8/CXUkQwiW94= 2024-02-01T10:48:03.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NoA+QyQJ2Y6SmkN8/CXUkQwiW94=', 'Content-Length': '0'} 2024-02-01T10:48:03.143 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009911aa85496003a4-0065bb76e3-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:03.144 DEBUG:boto:path=/vclfcn-59/obj7 2024-02-01T10:48:03.144 DEBUG:boto:auth_path=/vclfcn-59/obj7 2024-02-01T10:48:03.144 DEBUG:boto:Method: HEAD 2024-02-01T10:48:03.144 DEBUG:boto:Path: /vclfcn-59/obj7 2024-02-01T10:48:03.144 DEBUG:boto:Data: 2024-02-01T10:48:03.144 DEBUG:boto:Headers: {} 2024-02-01T10:48:03.144 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:03.144 DEBUG:boto:Port: 8000 2024-02-01T10:48:03.144 DEBUG:boto:Params: {} 2024-02-01T10:48:03.145 DEBUG:boto:Token: None 2024-02-01T10:48:03.145 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:03 GMT /vclfcn-59/obj7 2024-02-01T10:48:03.145 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NoA+QyQJ2Y6SmkN8/CXUkQwiW94= 2024-02-01T10:48:03.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NoA+QyQJ2Y6SmkN8/CXUkQwiW94=', 'Content-Length': '0'} 2024-02-01T10:48:03.147 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000940bbc1ccca9592-0065bb76e3-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:03.148 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-02-01T10:48:03.148 DEBUG:boto:path=/vclfcn-59/obj7 2024-02-01T10:48:03.148 DEBUG:boto:auth_path=/vclfcn-59/obj7 2024-02-01T10:48:03.148 DEBUG:boto:Method: GET 2024-02-01T10:48:03.148 DEBUG:boto:Path: /vclfcn-59/obj7 2024-02-01T10:48:03.148 DEBUG:boto:Data: 2024-02-01T10:48:03.148 DEBUG:boto:Headers: {} 2024-02-01T10:48:03.148 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:03.148 DEBUG:boto:Port: 8000 2024-02-01T10:48:03.149 DEBUG:boto:Params: {} 2024-02-01T10:48:03.149 DEBUG:boto:Token: None 2024-02-01T10:48:03.149 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:03 GMT /vclfcn-59/obj7 2024-02-01T10:48:03.149 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NoT89XTaPMQ+QUCPVFJmxcPJJyQ= 2024-02-01T10:48:03.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NoT89XTaPMQ+QUCPVFJmxcPJJyQ=', 'Content-Length': '0'} 2024-02-01T10:48:03.152 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000054bd5811e2ce08b8-0065bb76e3-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:03.196 DEBUG:boto:path=/vclfcn-59/obj7 2024-02-01T10:48:03.196 DEBUG:boto:auth_path=/vclfcn-59/obj7 2024-02-01T10:48:03.196 DEBUG:boto:Method: GET 2024-02-01T10:48:03.196 DEBUG:boto:Path: /vclfcn-59/obj7 2024-02-01T10:48:03.196 DEBUG:boto:Data: 2024-02-01T10:48:03.196 DEBUG:boto:Headers: {} 2024-02-01T10:48:03.197 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:03.197 DEBUG:boto:Port: 8000 2024-02-01T10:48:03.197 DEBUG:boto:Params: {} 2024-02-01T10:48:03.197 DEBUG:boto:Token: None 2024-02-01T10:48:03.197 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:03 GMT /vclfcn-59/obj7 2024-02-01T10:48:03.197 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NoT89XTaPMQ+QUCPVFJmxcPJJyQ= 2024-02-01T10:48:03.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NoT89XTaPMQ+QUCPVFJmxcPJJyQ=', 'Content-Length': '0'} 2024-02-01T10:48:03.200 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000020aff9aa7fa10f9c-0065bb76e3-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:03.201 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:03.201 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:03.201 DEBUG:boto:path=/vclfcn-59/?prefix=obj7 2024-02-01T10:48:03.201 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj7 2024-02-01T10:48:03.201 DEBUG:boto:Method: GET 2024-02-01T10:48:03.201 DEBUG:boto:Path: /vclfcn-59/?prefix=obj7 2024-02-01T10:48:03.201 DEBUG:boto:Data: 2024-02-01T10:48:03.201 DEBUG:boto:Headers: {} 2024-02-01T10:48:03.202 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:03.202 DEBUG:boto:Port: 8000 2024-02-01T10:48:03.202 DEBUG:boto:Params: {} 2024-02-01T10:48:03.202 DEBUG:boto:Token: None 2024-02-01T10:48:03.202 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:03 GMT /vclfcn-59/ 2024-02-01T10:48:03.202 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nMZ+QSOdClyZcZD5pxyoNs9MRBQ= 2024-02-01T10:48:03.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nMZ+QSOdClyZcZD5pxyoNs9MRBQ=', 'Content-Length': '0'} 2024-02-01T10:48:03.207 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ca4dd7d65000047-0065bb76e3-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:03.207 DEBUG:boto:b'vclfcn-59obj71000falseobj72024-02-01T10:47:30.227Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:03.207 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:03.208 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:03.208 DEBUG:boto:path=/vclfcn-59/?prefix=obj7 2024-02-01T10:48:03.208 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj7 2024-02-01T10:48:03.208 DEBUG:boto:Method: GET 2024-02-01T10:48:03.208 DEBUG:boto:Path: /vclfcn-59/?prefix=obj7 2024-02-01T10:48:03.208 DEBUG:boto:Data: 2024-02-01T10:48:03.208 DEBUG:boto:Headers: {} 2024-02-01T10:48:03.208 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:03.208 DEBUG:boto:Port: 8000 2024-02-01T10:48:03.208 DEBUG:boto:Params: {} 2024-02-01T10:48:03.208 DEBUG:boto:Token: None 2024-02-01T10:48:03.209 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:03 GMT /vclfcn-59/ 2024-02-01T10:48:03.209 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nMZ+QSOdClyZcZD5pxyoNs9MRBQ= 2024-02-01T10:48:03.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nMZ+QSOdClyZcZD5pxyoNs9MRBQ=', 'Content-Length': '0'} 2024-02-01T10:48:03.213 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a714faf20893a99-0065bb76e3-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:03.260 DEBUG:boto:b'vclfcn-59obj71000falseobj72024-02-01T10:47:30.227Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:03.261 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-02-01T10:48:03.261 DEBUG:boto:path=/vclfcn-59/obj8 2024-02-01T10:48:03.261 DEBUG:boto:auth_path=/vclfcn-59/obj8 2024-02-01T10:48:03.261 DEBUG:boto:path=/vclfcn-59/obj8?versionId=null 2024-02-01T10:48:03.261 DEBUG:boto:auth_path=/vclfcn-59/obj8?versionId=null 2024-02-01T10:48:03.261 DEBUG:boto:Method: GET 2024-02-01T10:48:03.261 DEBUG:boto:Path: /vclfcn-59/obj8?versionId=null 2024-02-01T10:48:03.261 DEBUG:boto:Data: 2024-02-01T10:48:03.262 DEBUG:boto:Headers: {} 2024-02-01T10:48:03.262 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:03.262 DEBUG:boto:Port: 8000 2024-02-01T10:48:03.262 DEBUG:boto:Params: {} 2024-02-01T10:48:03.262 DEBUG:boto:Token: None 2024-02-01T10:48:03.263 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:03 GMT /vclfcn-59/obj8?versionId=null 2024-02-01T10:48:03.263 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Gj+WFRQi+J3c7t1TzqOD5JIumvs= 2024-02-01T10:48:03.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Gj+WFRQi+J3c7t1TzqOD5JIumvs=', 'Content-Length': '0'} 2024-02-01T10:48:03.266 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000016757e091d756249-0065bb76e3-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:03.312 DEBUG:boto:path=/vclfcn-59/obj8 2024-02-01T10:48:03.312 DEBUG:boto:auth_path=/vclfcn-59/obj8 2024-02-01T10:48:03.312 DEBUG:boto:path=/vclfcn-59/obj8?versionId=null 2024-02-01T10:48:03.312 DEBUG:boto:auth_path=/vclfcn-59/obj8?versionId=null 2024-02-01T10:48:03.313 DEBUG:boto:Method: GET 2024-02-01T10:48:03.313 DEBUG:boto:Path: /vclfcn-59/obj8?versionId=null 2024-02-01T10:48:03.313 DEBUG:boto:Data: 2024-02-01T10:48:03.313 DEBUG:boto:Headers: {} 2024-02-01T10:48:03.313 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:03.313 DEBUG:boto:Port: 8000 2024-02-01T10:48:03.313 DEBUG:boto:Params: {} 2024-02-01T10:48:03.313 DEBUG:boto:Token: None 2024-02-01T10:48:03.313 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:03 GMT /vclfcn-59/obj8?versionId=null 2024-02-01T10:48:03.314 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Gj+WFRQi+J3c7t1TzqOD5JIumvs= 2024-02-01T10:48:03.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Gj+WFRQi+J3c7t1TzqOD5JIumvs=', 'Content-Length': '0'} 2024-02-01T10:48:03.317 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000948f35b539e3cae4-0065bb76e3-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:03.317 DEBUG:boto:path=/vclfcn-59/obj8 2024-02-01T10:48:03.317 DEBUG:boto:auth_path=/vclfcn-59/obj8 2024-02-01T10:48:03.317 DEBUG:boto:path=/vclfcn-59/obj8?versionId=null 2024-02-01T10:48:03.317 DEBUG:boto:auth_path=/vclfcn-59/obj8?versionId=null 2024-02-01T10:48:03.317 DEBUG:boto:Method: HEAD 2024-02-01T10:48:03.317 DEBUG:boto:Path: /vclfcn-59/obj8?versionId=null 2024-02-01T10:48:03.318 DEBUG:boto:Data: 2024-02-01T10:48:03.318 DEBUG:boto:Headers: {} 2024-02-01T10:48:03.318 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:03.318 DEBUG:boto:Port: 8000 2024-02-01T10:48:03.318 DEBUG:boto:Params: {} 2024-02-01T10:48:03.318 DEBUG:boto:Token: None 2024-02-01T10:48:03.318 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:03 GMT /vclfcn-59/obj8?versionId=null 2024-02-01T10:48:03.318 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+ftrw6Q/SfGywUauUJz0IZo48a8= 2024-02-01T10:48:03.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+ftrw6Q/SfGywUauUJz0IZo48a8=', 'Content-Length': '0'} 2024-02-01T10:48:03.321 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c3d6ebb196bf4f2d-0065bb76e3-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:03.321 DEBUG:boto:path=/vclfcn-59/obj8 2024-02-01T10:48:03.321 DEBUG:boto:auth_path=/vclfcn-59/obj8 2024-02-01T10:48:03.322 DEBUG:boto:path=/vclfcn-59/obj8?versionId=null 2024-02-01T10:48:03.322 DEBUG:boto:auth_path=/vclfcn-59/obj8?versionId=null 2024-02-01T10:48:03.322 DEBUG:boto:Method: HEAD 2024-02-01T10:48:03.322 DEBUG:boto:Path: /vclfcn-59/obj8?versionId=null 2024-02-01T10:48:03.322 DEBUG:boto:Data: 2024-02-01T10:48:03.322 DEBUG:boto:Headers: {} 2024-02-01T10:48:03.322 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:03.322 DEBUG:boto:Port: 8000 2024-02-01T10:48:03.322 DEBUG:boto:Params: {} 2024-02-01T10:48:03.322 DEBUG:boto:Token: None 2024-02-01T10:48:03.323 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:03 GMT /vclfcn-59/obj8?versionId=null 2024-02-01T10:48:03.323 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+ftrw6Q/SfGywUauUJz0IZo48a8= 2024-02-01T10:48:03.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+ftrw6Q/SfGywUauUJz0IZo48a8=', 'Content-Length': '0'} 2024-02-01T10:48:03.325 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b4e455ef21fcead2-0065bb76e3-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:03.326 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-02-01T10:48:03.326 DEBUG:boto:path=/vclfcn-59/obj8 2024-02-01T10:48:03.326 DEBUG:boto:auth_path=/vclfcn-59/obj8 2024-02-01T10:48:03.326 DEBUG:boto:path=/vclfcn-59/obj8?versionId=null 2024-02-01T10:48:03.326 DEBUG:boto:auth_path=/vclfcn-59/obj8?versionId=null 2024-02-01T10:48:03.326 DEBUG:boto:Method: GET 2024-02-01T10:48:03.327 DEBUG:boto:Path: /vclfcn-59/obj8?versionId=null 2024-02-01T10:48:03.327 DEBUG:boto:Data: 2024-02-01T10:48:03.327 DEBUG:boto:Headers: {} 2024-02-01T10:48:03.327 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:03.327 DEBUG:boto:Port: 8000 2024-02-01T10:48:03.327 DEBUG:boto:Params: {} 2024-02-01T10:48:03.327 DEBUG:boto:Token: None 2024-02-01T10:48:03.327 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:03 GMT /vclfcn-59/obj8?versionId=null 2024-02-01T10:48:03.328 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Gj+WFRQi+J3c7t1TzqOD5JIumvs= 2024-02-01T10:48:03.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Gj+WFRQi+J3c7t1TzqOD5JIumvs=', 'Content-Length': '0'} 2024-02-01T10:48:03.330 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000007d84c4b7390fa05-0065bb76e3-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:03.372 DEBUG:boto:path=/vclfcn-59/obj8 2024-02-01T10:48:03.372 DEBUG:boto:auth_path=/vclfcn-59/obj8 2024-02-01T10:48:03.372 DEBUG:boto:path=/vclfcn-59/obj8?versionId=null 2024-02-01T10:48:03.373 DEBUG:boto:auth_path=/vclfcn-59/obj8?versionId=null 2024-02-01T10:48:03.373 DEBUG:boto:Method: GET 2024-02-01T10:48:03.373 DEBUG:boto:Path: /vclfcn-59/obj8?versionId=null 2024-02-01T10:48:03.373 DEBUG:boto:Data: 2024-02-01T10:48:03.373 DEBUG:boto:Headers: {} 2024-02-01T10:48:03.373 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:03.373 DEBUG:boto:Port: 8000 2024-02-01T10:48:03.373 DEBUG:boto:Params: {} 2024-02-01T10:48:03.374 DEBUG:boto:Token: None 2024-02-01T10:48:03.374 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:03 GMT /vclfcn-59/obj8?versionId=null 2024-02-01T10:48:03.374 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Gj+WFRQi+J3c7t1TzqOD5JIumvs= 2024-02-01T10:48:03.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Gj+WFRQi+J3c7t1TzqOD5JIumvs=', 'Content-Length': '0'} 2024-02-01T10:48:03.377 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000007420510e1ad315c-0065bb76e3-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:03.377 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:03.377 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:03.378 DEBUG:boto:path=/vclfcn-59/?prefix=obj8 2024-02-01T10:48:03.378 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj8 2024-02-01T10:48:03.378 DEBUG:boto:Method: GET 2024-02-01T10:48:03.378 DEBUG:boto:Path: /vclfcn-59/?prefix=obj8 2024-02-01T10:48:03.378 DEBUG:boto:Data: 2024-02-01T10:48:03.378 DEBUG:boto:Headers: {} 2024-02-01T10:48:03.378 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:03.378 DEBUG:boto:Port: 8000 2024-02-01T10:48:03.378 DEBUG:boto:Params: {} 2024-02-01T10:48:03.378 DEBUG:boto:Token: None 2024-02-01T10:48:03.379 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:03 GMT /vclfcn-59/ 2024-02-01T10:48:03.379 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nMZ+QSOdClyZcZD5pxyoNs9MRBQ= 2024-02-01T10:48:03.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nMZ+QSOdClyZcZD5pxyoNs9MRBQ=', 'Content-Length': '0'} 2024-02-01T10:48:03.384 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ccea7faf3b24e57-0065bb76e3-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:03.428 DEBUG:boto:b'vclfcn-59obj81000falseobj82024-02-01T10:47:30.275Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:03.429 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:03.429 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:03.429 DEBUG:boto:path=/vclfcn-59/?prefix=obj8 2024-02-01T10:48:03.429 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj8 2024-02-01T10:48:03.429 DEBUG:boto:Method: GET 2024-02-01T10:48:03.429 DEBUG:boto:Path: /vclfcn-59/?prefix=obj8 2024-02-01T10:48:03.429 DEBUG:boto:Data: 2024-02-01T10:48:03.429 DEBUG:boto:Headers: {} 2024-02-01T10:48:03.429 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:03.429 DEBUG:boto:Port: 8000 2024-02-01T10:48:03.429 DEBUG:boto:Params: {} 2024-02-01T10:48:03.430 DEBUG:boto:Token: None 2024-02-01T10:48:03.430 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:03 GMT /vclfcn-59/ 2024-02-01T10:48:03.430 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nMZ+QSOdClyZcZD5pxyoNs9MRBQ= 2024-02-01T10:48:03.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nMZ+QSOdClyZcZD5pxyoNs9MRBQ=', 'Content-Length': '0'} 2024-02-01T10:48:03.434 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e0214d63543dbfa6-0065bb76e3-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:03.435 DEBUG:boto:b'vclfcn-59obj81000falseobj82024-02-01T10:47:30.275Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:03.435 DEBUG:boto:path=/vclfcn-59/obj8 2024-02-01T10:48:03.435 DEBUG:boto:auth_path=/vclfcn-59/obj8 2024-02-01T10:48:03.435 DEBUG:boto:Method: HEAD 2024-02-01T10:48:03.435 DEBUG:boto:Path: /vclfcn-59/obj8 2024-02-01T10:48:03.435 DEBUG:boto:Data: 2024-02-01T10:48:03.436 DEBUG:boto:Headers: {} 2024-02-01T10:48:03.436 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:03.436 DEBUG:boto:Port: 8000 2024-02-01T10:48:03.436 DEBUG:boto:Params: {} 2024-02-01T10:48:03.436 DEBUG:boto:Token: None 2024-02-01T10:48:03.436 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:03 GMT /vclfcn-59/obj8 2024-02-01T10:48:03.436 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ab82SdykEJxuY110j7lyGYOsduo= 2024-02-01T10:48:03.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ab82SdykEJxuY110j7lyGYOsduo=', 'Content-Length': '0'} 2024-02-01T10:48:03.439 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000133a42f15f1943ae-0065bb76e3-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:03.440 DEBUG:boto:path=/vclfcn-59/obj8 2024-02-01T10:48:03.440 DEBUG:boto:auth_path=/vclfcn-59/obj8 2024-02-01T10:48:03.440 DEBUG:boto:Method: HEAD 2024-02-01T10:48:03.440 DEBUG:boto:Path: /vclfcn-59/obj8 2024-02-01T10:48:03.440 DEBUG:boto:Data: 2024-02-01T10:48:03.440 DEBUG:boto:Headers: {} 2024-02-01T10:48:03.440 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:03.440 DEBUG:boto:Port: 8000 2024-02-01T10:48:03.441 DEBUG:boto:Params: {} 2024-02-01T10:48:03.441 DEBUG:boto:Token: None 2024-02-01T10:48:03.441 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:03 GMT /vclfcn-59/obj8 2024-02-01T10:48:03.441 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ab82SdykEJxuY110j7lyGYOsduo= 2024-02-01T10:48:03.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ab82SdykEJxuY110j7lyGYOsduo=', 'Content-Length': '0'} 2024-02-01T10:48:03.444 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000620dab9156c077bb-0065bb76e3-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:03.444 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-02-01T10:48:03.444 DEBUG:boto:path=/vclfcn-59/obj8 2024-02-01T10:48:03.444 DEBUG:boto:auth_path=/vclfcn-59/obj8 2024-02-01T10:48:03.444 DEBUG:boto:Method: GET 2024-02-01T10:48:03.444 DEBUG:boto:Path: /vclfcn-59/obj8 2024-02-01T10:48:03.444 DEBUG:boto:Data: 2024-02-01T10:48:03.445 DEBUG:boto:Headers: {} 2024-02-01T10:48:03.445 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:03.445 DEBUG:boto:Port: 8000 2024-02-01T10:48:03.445 DEBUG:boto:Params: {} 2024-02-01T10:48:03.445 DEBUG:boto:Token: None 2024-02-01T10:48:03.445 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:03 GMT /vclfcn-59/obj8 2024-02-01T10:48:03.445 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:xjvAebI1gi5UizQf24x4u9r8Abs= 2024-02-01T10:48:03.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:xjvAebI1gi5UizQf24x4u9r8Abs=', 'Content-Length': '0'} 2024-02-01T10:48:03.448 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009e0f3c15fc290e07-0065bb76e3-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:03.488 DEBUG:boto:path=/vclfcn-59/obj8 2024-02-01T10:48:03.488 DEBUG:boto:auth_path=/vclfcn-59/obj8 2024-02-01T10:48:03.488 DEBUG:boto:Method: GET 2024-02-01T10:48:03.489 DEBUG:boto:Path: /vclfcn-59/obj8 2024-02-01T10:48:03.489 DEBUG:boto:Data: 2024-02-01T10:48:03.489 DEBUG:boto:Headers: {} 2024-02-01T10:48:03.489 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:03.489 DEBUG:boto:Port: 8000 2024-02-01T10:48:03.489 DEBUG:boto:Params: {} 2024-02-01T10:48:03.489 DEBUG:boto:Token: None 2024-02-01T10:48:03.489 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:03 GMT /vclfcn-59/obj8 2024-02-01T10:48:03.489 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:xjvAebI1gi5UizQf24x4u9r8Abs= 2024-02-01T10:48:03.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:xjvAebI1gi5UizQf24x4u9r8Abs=', 'Content-Length': '0'} 2024-02-01T10:48:03.493 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004f3b5d13c7cb7ba6-0065bb76e3-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:03.494 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:03.494 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:03.494 DEBUG:boto:path=/vclfcn-59/?prefix=obj8 2024-02-01T10:48:03.494 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj8 2024-02-01T10:48:03.494 DEBUG:boto:Method: GET 2024-02-01T10:48:03.494 DEBUG:boto:Path: /vclfcn-59/?prefix=obj8 2024-02-01T10:48:03.495 DEBUG:boto:Data: 2024-02-01T10:48:03.495 DEBUG:boto:Headers: {} 2024-02-01T10:48:03.495 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:03.495 DEBUG:boto:Port: 8000 2024-02-01T10:48:03.496 DEBUG:boto:Params: {} 2024-02-01T10:48:03.496 DEBUG:boto:Token: None 2024-02-01T10:48:03.496 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:03 GMT /vclfcn-59/ 2024-02-01T10:48:03.496 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nMZ+QSOdClyZcZD5pxyoNs9MRBQ= 2024-02-01T10:48:03.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nMZ+QSOdClyZcZD5pxyoNs9MRBQ=', 'Content-Length': '0'} 2024-02-01T10:48:03.501 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d43c04048f6ae0fc-0065bb76e3-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:03.501 DEBUG:boto:b'vclfcn-59obj81000falseobj82024-02-01T10:47:30.275Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:03.502 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:03.502 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:03.502 DEBUG:boto:path=/vclfcn-59/?prefix=obj8 2024-02-01T10:48:03.502 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj8 2024-02-01T10:48:03.502 DEBUG:boto:Method: GET 2024-02-01T10:48:03.502 DEBUG:boto:Path: /vclfcn-59/?prefix=obj8 2024-02-01T10:48:03.502 DEBUG:boto:Data: 2024-02-01T10:48:03.503 DEBUG:boto:Headers: {} 2024-02-01T10:48:03.503 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:03.503 DEBUG:boto:Port: 8000 2024-02-01T10:48:03.503 DEBUG:boto:Params: {} 2024-02-01T10:48:03.503 DEBUG:boto:Token: None 2024-02-01T10:48:03.503 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:03 GMT /vclfcn-59/ 2024-02-01T10:48:03.503 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nMZ+QSOdClyZcZD5pxyoNs9MRBQ= 2024-02-01T10:48:03.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nMZ+QSOdClyZcZD5pxyoNs9MRBQ=', 'Content-Length': '0'} 2024-02-01T10:48:03.508 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b649e20c801aad2-0065bb76e3-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:03.552 DEBUG:boto:b'vclfcn-59obj81000falseobj82024-02-01T10:47:30.275Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:03.553 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-59 zones={test-zone2, test-zone1} 2024-02-01T10:48:03.553 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-59 zones={test-zone3, test-zone1} 2024-02-01T10:48:03.553 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:03.553 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:03.553 DEBUG:boto:Method: HEAD 2024-02-01T10:48:03.553 DEBUG:boto:Path: /vclfcn-59/ 2024-02-01T10:48:03.553 DEBUG:boto:Data: 2024-02-01T10:48:03.553 DEBUG:boto:Headers: {} 2024-02-01T10:48:03.553 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:03.554 DEBUG:boto:Port: 8001 2024-02-01T10:48:03.554 DEBUG:boto:Params: {} 2024-02-01T10:48:03.554 DEBUG:boto:Token: None 2024-02-01T10:48:03.554 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:03 GMT /vclfcn-59/ 2024-02-01T10:48:03.554 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5eXPVAZQ/hKh5vxkwew3WF8viF0= 2024-02-01T10:48:03.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5eXPVAZQ/hKh5vxkwew3WF8viF0=', 'Content-Length': '0'} 2024-02-01T10:48:03.558 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a9067a1b2c5dd75e-0065bb76e3-5762-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:03.558 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:03.558 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:03.559 DEBUG:boto:Method: HEAD 2024-02-01T10:48:03.559 DEBUG:boto:Path: /vclfcn-59/ 2024-02-01T10:48:03.559 DEBUG:boto:Data: 2024-02-01T10:48:03.559 DEBUG:boto:Headers: {} 2024-02-01T10:48:03.559 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:03.559 DEBUG:boto:Port: 8000 2024-02-01T10:48:03.559 DEBUG:boto:Params: {} 2024-02-01T10:48:03.559 DEBUG:boto:Token: None 2024-02-01T10:48:03.559 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:03 GMT /vclfcn-59/ 2024-02-01T10:48:03.559 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5eXPVAZQ/hKh5vxkwew3WF8viF0= 2024-02-01T10:48:03.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5eXPVAZQ/hKh5vxkwew3WF8viF0=', 'Content-Length': '0'} 2024-02-01T10:48:03.563 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001733adc426dc21c0-0065bb76e3-5765-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:03.563 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:48:03.564 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:03.564 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:03.564 DEBUG:boto:path=/vclfcn-59/?versions&max-keys=999 2024-02-01T10:48:03.564 DEBUG:boto:auth_path=/vclfcn-59/?versions&max-keys=999 2024-02-01T10:48:03.564 DEBUG:boto:Method: GET 2024-02-01T10:48:03.564 DEBUG:boto:Path: /vclfcn-59/?versions&max-keys=999 2024-02-01T10:48:03.564 DEBUG:boto:Data: 2024-02-01T10:48:03.564 DEBUG:boto:Headers: {} 2024-02-01T10:48:03.564 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:03.564 DEBUG:boto:Port: 8001 2024-02-01T10:48:03.564 DEBUG:boto:Params: {} 2024-02-01T10:48:03.565 DEBUG:boto:Token: None 2024-02-01T10:48:03.565 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:03 GMT /vclfcn-59/?versions 2024-02-01T10:48:03.565 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:KN8sO+ACAmiRLD7GgF8Zs2ntz0A= 2024-02-01T10:48:03.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:KN8sO+ACAmiRLD7GgF8Zs2ntz0A=', 'Content-Length': '0'} 2024-02-01T10:48:03.569 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f26e3cfefd17e984-0065bb76e3-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:03.569 DEBUG:boto:b'vclfcn-59999falseobj1nulltrue2024-02-01T10:46:46.093Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:46:46.111Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:46:46.161Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:46:46.190Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-02-01T10:47:30.206Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-02-01T10:47:30.217Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-02-01T10:47:30.227Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-02-01T10:47:30.275Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:03.570 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-02-01T10:48:03.570 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-02-01T10:48:03.570 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-02-01T10:48:03.570 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-02-01T10:48:03.570 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-02-01T10:48:03.570 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-02-01T10:48:03.570 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-02-01T10:48:03.570 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-02-01T10:48:03.570 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:48:03.571 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:03.571 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:03.571 DEBUG:boto:path=/vclfcn-59/?versions&max-keys=999 2024-02-01T10:48:03.571 DEBUG:boto:auth_path=/vclfcn-59/?versions&max-keys=999 2024-02-01T10:48:03.571 DEBUG:boto:Method: GET 2024-02-01T10:48:03.571 DEBUG:boto:Path: /vclfcn-59/?versions&max-keys=999 2024-02-01T10:48:03.571 DEBUG:boto:Data: 2024-02-01T10:48:03.571 DEBUG:boto:Headers: {} 2024-02-01T10:48:03.571 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:03.571 DEBUG:boto:Port: 8000 2024-02-01T10:48:03.571 DEBUG:boto:Params: {} 2024-02-01T10:48:03.572 DEBUG:boto:Token: None 2024-02-01T10:48:03.572 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:03 GMT /vclfcn-59/?versions 2024-02-01T10:48:03.572 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:KN8sO+ACAmiRLD7GgF8Zs2ntz0A= 2024-02-01T10:48:03.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:KN8sO+ACAmiRLD7GgF8Zs2ntz0A=', 'Content-Length': '0'} 2024-02-01T10:48:03.577 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092cf148c6762383e-0065bb76e3-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:03.577 DEBUG:boto:b'vclfcn-59999falseobj1nulltrue2024-02-01T10:46:46.093Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:46:46.111Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:46:46.161Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:46:46.190Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-02-01T10:47:30.206Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-02-01T10:47:30.217Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-02-01T10:47:30.227Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-02-01T10:47:30.275Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:03.577 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-02-01T10:48:03.577 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-02-01T10:48:03.577 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-02-01T10:48:03.578 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-02-01T10:48:03.578 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-02-01T10:48:03.578 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-02-01T10:48:03.578 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-02-01T10:48:03.578 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-02-01T10:48:03.578 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:03.578 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:03.578 DEBUG:boto:path=/vclfcn-59/?versions&max-keys=999 2024-02-01T10:48:03.578 DEBUG:boto:auth_path=/vclfcn-59/?versions&max-keys=999 2024-02-01T10:48:03.578 DEBUG:boto:Method: GET 2024-02-01T10:48:03.578 DEBUG:boto:Path: /vclfcn-59/?versions&max-keys=999 2024-02-01T10:48:03.579 DEBUG:boto:Data: 2024-02-01T10:48:03.579 DEBUG:boto:Headers: {} 2024-02-01T10:48:03.579 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:03.579 DEBUG:boto:Port: 8001 2024-02-01T10:48:03.579 DEBUG:boto:Params: {} 2024-02-01T10:48:03.579 DEBUG:boto:Token: None 2024-02-01T10:48:03.579 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:03 GMT /vclfcn-59/?versions 2024-02-01T10:48:03.579 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:KN8sO+ACAmiRLD7GgF8Zs2ntz0A= 2024-02-01T10:48:03.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:KN8sO+ACAmiRLD7GgF8Zs2ntz0A=', 'Content-Length': '0'} 2024-02-01T10:48:03.584 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009313b3bb98ad491a-0065bb76e3-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:03.584 DEBUG:boto:b'vclfcn-59999falseobj1nulltrue2024-02-01T10:46:46.093Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:46:46.111Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:46:46.161Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:46:46.190Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-02-01T10:47:30.206Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-02-01T10:47:30.217Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-02-01T10:47:30.227Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-02-01T10:47:30.275Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:03.585 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:03.585 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:03.585 DEBUG:boto:path=/vclfcn-59/?versions&max-keys=999 2024-02-01T10:48:03.585 DEBUG:boto:auth_path=/vclfcn-59/?versions&max-keys=999 2024-02-01T10:48:03.585 DEBUG:boto:Method: GET 2024-02-01T10:48:03.585 DEBUG:boto:Path: /vclfcn-59/?versions&max-keys=999 2024-02-01T10:48:03.585 DEBUG:boto:Data: 2024-02-01T10:48:03.585 DEBUG:boto:Headers: {} 2024-02-01T10:48:03.585 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:03.585 DEBUG:boto:Port: 8000 2024-02-01T10:48:03.585 DEBUG:boto:Params: {} 2024-02-01T10:48:03.585 DEBUG:boto:Token: None 2024-02-01T10:48:03.586 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:03 GMT /vclfcn-59/?versions 2024-02-01T10:48:03.586 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:KN8sO+ACAmiRLD7GgF8Zs2ntz0A= 2024-02-01T10:48:03.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:KN8sO+ACAmiRLD7GgF8Zs2ntz0A=', 'Content-Length': '0'} 2024-02-01T10:48:03.591 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082036b1a54cd946a-0065bb76e3-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:03.591 DEBUG:boto:b'vclfcn-59999falseobj1nulltrue2024-02-01T10:46:46.093Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:46:46.111Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:46:46.161Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:46:46.190Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-02-01T10:47:30.206Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-02-01T10:47:30.217Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-02-01T10:47:30.227Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-02-01T10:47:30.275Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:03.592 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:48:03.592 DEBUG:boto:path=/vclfcn-59/obj1 2024-02-01T10:48:03.592 DEBUG:boto:auth_path=/vclfcn-59/obj1 2024-02-01T10:48:03.592 DEBUG:boto:path=/vclfcn-59/obj1?versionId=null 2024-02-01T10:48:03.592 DEBUG:boto:auth_path=/vclfcn-59/obj1?versionId=null 2024-02-01T10:48:03.592 DEBUG:boto:Method: GET 2024-02-01T10:48:03.592 DEBUG:boto:Path: /vclfcn-59/obj1?versionId=null 2024-02-01T10:48:03.592 DEBUG:boto:Data: 2024-02-01T10:48:03.592 DEBUG:boto:Headers: {} 2024-02-01T10:48:03.593 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:03.593 DEBUG:boto:Port: 8001 2024-02-01T10:48:03.593 DEBUG:boto:Params: {} 2024-02-01T10:48:03.593 DEBUG:boto:Token: None 2024-02-01T10:48:03.593 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:03 GMT /vclfcn-59/obj1?versionId=null 2024-02-01T10:48:03.593 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:pTOtm44/c2YFvaI6eQoXNo7Ga5k= 2024-02-01T10:48:03.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:pTOtm44/c2YFvaI6eQoXNo7Ga5k=', 'Content-Length': '0'} 2024-02-01T10:48:03.596 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004be587ba64ceffdb-0065bb76e3-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:03.644 DEBUG:boto:path=/vclfcn-59/obj1 2024-02-01T10:48:03.644 DEBUG:boto:auth_path=/vclfcn-59/obj1 2024-02-01T10:48:03.644 DEBUG:boto:path=/vclfcn-59/obj1?versionId=null 2024-02-01T10:48:03.644 DEBUG:boto:auth_path=/vclfcn-59/obj1?versionId=null 2024-02-01T10:48:03.645 DEBUG:boto:Method: GET 2024-02-01T10:48:03.645 DEBUG:boto:Path: /vclfcn-59/obj1?versionId=null 2024-02-01T10:48:03.645 DEBUG:boto:Data: 2024-02-01T10:48:03.645 DEBUG:boto:Headers: {} 2024-02-01T10:48:03.645 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:03.645 DEBUG:boto:Port: 8000 2024-02-01T10:48:03.645 DEBUG:boto:Params: {} 2024-02-01T10:48:03.645 DEBUG:boto:Token: None 2024-02-01T10:48:03.645 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:03 GMT /vclfcn-59/obj1?versionId=null 2024-02-01T10:48:03.646 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:pTOtm44/c2YFvaI6eQoXNo7Ga5k= 2024-02-01T10:48:03.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:pTOtm44/c2YFvaI6eQoXNo7Ga5k=', 'Content-Length': '0'} 2024-02-01T10:48:03.648 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ce11fa56444fc2b8-0065bb76e3-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:03.649 DEBUG:boto:path=/vclfcn-59/obj1 2024-02-01T10:48:03.649 DEBUG:boto:auth_path=/vclfcn-59/obj1 2024-02-01T10:48:03.649 DEBUG:boto:path=/vclfcn-59/obj1?versionId=null 2024-02-01T10:48:03.649 DEBUG:boto:auth_path=/vclfcn-59/obj1?versionId=null 2024-02-01T10:48:03.649 DEBUG:boto:Method: HEAD 2024-02-01T10:48:03.649 DEBUG:boto:Path: /vclfcn-59/obj1?versionId=null 2024-02-01T10:48:03.649 DEBUG:boto:Data: 2024-02-01T10:48:03.649 DEBUG:boto:Headers: {} 2024-02-01T10:48:03.649 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:03.649 DEBUG:boto:Port: 8001 2024-02-01T10:48:03.650 DEBUG:boto:Params: {} 2024-02-01T10:48:03.650 DEBUG:boto:Token: None 2024-02-01T10:48:03.650 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:03 GMT /vclfcn-59/obj1?versionId=null 2024-02-01T10:48:03.650 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:18BOJsYbEmkVtcaw3xixFg0gdcw= 2024-02-01T10:48:03.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:18BOJsYbEmkVtcaw3xixFg0gdcw=', 'Content-Length': '0'} 2024-02-01T10:48:03.653 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c9e211a70a64eed7-0065bb76e3-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:03.654 DEBUG:boto:path=/vclfcn-59/obj1 2024-02-01T10:48:03.654 DEBUG:boto:auth_path=/vclfcn-59/obj1 2024-02-01T10:48:03.654 DEBUG:boto:path=/vclfcn-59/obj1?versionId=null 2024-02-01T10:48:03.654 DEBUG:boto:auth_path=/vclfcn-59/obj1?versionId=null 2024-02-01T10:48:03.654 DEBUG:boto:Method: HEAD 2024-02-01T10:48:03.654 DEBUG:boto:Path: /vclfcn-59/obj1?versionId=null 2024-02-01T10:48:03.654 DEBUG:boto:Data: 2024-02-01T10:48:03.654 DEBUG:boto:Headers: {} 2024-02-01T10:48:03.654 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:03.654 DEBUG:boto:Port: 8000 2024-02-01T10:48:03.655 DEBUG:boto:Params: {} 2024-02-01T10:48:03.655 DEBUG:boto:Token: None 2024-02-01T10:48:03.655 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:03 GMT /vclfcn-59/obj1?versionId=null 2024-02-01T10:48:03.655 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:18BOJsYbEmkVtcaw3xixFg0gdcw= 2024-02-01T10:48:03.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:18BOJsYbEmkVtcaw3xixFg0gdcw=', 'Content-Length': '0'} 2024-02-01T10:48:03.658 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dbbe4bd265d2143c-0065bb76e3-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:03.658 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:48:03.658 DEBUG:boto:path=/vclfcn-59/obj1 2024-02-01T10:48:03.658 DEBUG:boto:auth_path=/vclfcn-59/obj1 2024-02-01T10:48:03.658 DEBUG:boto:path=/vclfcn-59/obj1?versionId=null 2024-02-01T10:48:03.658 DEBUG:boto:auth_path=/vclfcn-59/obj1?versionId=null 2024-02-01T10:48:03.658 DEBUG:boto:Method: GET 2024-02-01T10:48:03.658 DEBUG:boto:Path: /vclfcn-59/obj1?versionId=null 2024-02-01T10:48:03.658 DEBUG:boto:Data: 2024-02-01T10:48:03.659 DEBUG:boto:Headers: {} 2024-02-01T10:48:03.659 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:03.659 DEBUG:boto:Port: 8001 2024-02-01T10:48:03.659 DEBUG:boto:Params: {} 2024-02-01T10:48:03.659 DEBUG:boto:Token: None 2024-02-01T10:48:03.659 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:03 GMT /vclfcn-59/obj1?versionId=null 2024-02-01T10:48:03.659 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:pTOtm44/c2YFvaI6eQoXNo7Ga5k= 2024-02-01T10:48:03.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:pTOtm44/c2YFvaI6eQoXNo7Ga5k=', 'Content-Length': '0'} 2024-02-01T10:48:03.662 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000015c9e7869dd71860-0065bb76e3-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:03.704 DEBUG:boto:path=/vclfcn-59/obj1 2024-02-01T10:48:03.704 DEBUG:boto:auth_path=/vclfcn-59/obj1 2024-02-01T10:48:03.704 DEBUG:boto:path=/vclfcn-59/obj1?versionId=null 2024-02-01T10:48:03.704 DEBUG:boto:auth_path=/vclfcn-59/obj1?versionId=null 2024-02-01T10:48:03.704 DEBUG:boto:Method: GET 2024-02-01T10:48:03.705 DEBUG:boto:Path: /vclfcn-59/obj1?versionId=null 2024-02-01T10:48:03.705 DEBUG:boto:Data: 2024-02-01T10:48:03.705 DEBUG:boto:Headers: {} 2024-02-01T10:48:03.705 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:03.705 DEBUG:boto:Port: 8000 2024-02-01T10:48:03.705 DEBUG:boto:Params: {} 2024-02-01T10:48:03.705 DEBUG:boto:Token: None 2024-02-01T10:48:03.705 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:03 GMT /vclfcn-59/obj1?versionId=null 2024-02-01T10:48:03.705 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:pTOtm44/c2YFvaI6eQoXNo7Ga5k= 2024-02-01T10:48:03.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:pTOtm44/c2YFvaI6eQoXNo7Ga5k=', 'Content-Length': '0'} 2024-02-01T10:48:03.708 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008efe37c4ec084c29-0065bb76e3-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:03.708 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:03.709 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:03.709 DEBUG:boto:path=/vclfcn-59/?prefix=obj1 2024-02-01T10:48:03.709 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj1 2024-02-01T10:48:03.709 DEBUG:boto:Method: GET 2024-02-01T10:48:03.709 DEBUG:boto:Path: /vclfcn-59/?prefix=obj1 2024-02-01T10:48:03.709 DEBUG:boto:Data: 2024-02-01T10:48:03.709 DEBUG:boto:Headers: {} 2024-02-01T10:48:03.709 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:03.709 DEBUG:boto:Port: 8001 2024-02-01T10:48:03.709 DEBUG:boto:Params: {} 2024-02-01T10:48:03.709 DEBUG:boto:Token: None 2024-02-01T10:48:03.710 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:03 GMT /vclfcn-59/ 2024-02-01T10:48:03.710 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nMZ+QSOdClyZcZD5pxyoNs9MRBQ= 2024-02-01T10:48:03.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nMZ+QSOdClyZcZD5pxyoNs9MRBQ=', 'Content-Length': '0'} 2024-02-01T10:48:03.714 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003cc257ed032c7320-0065bb76e3-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:03.760 DEBUG:boto:b'vclfcn-59obj11000falseobj12024-02-01T10:46:46.093Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:03.761 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:03.761 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:03.761 DEBUG:boto:path=/vclfcn-59/?prefix=obj1 2024-02-01T10:48:03.761 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj1 2024-02-01T10:48:03.762 DEBUG:boto:Method: GET 2024-02-01T10:48:03.762 DEBUG:boto:Path: /vclfcn-59/?prefix=obj1 2024-02-01T10:48:03.762 DEBUG:boto:Data: 2024-02-01T10:48:03.762 DEBUG:boto:Headers: {} 2024-02-01T10:48:03.762 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:03.762 DEBUG:boto:Port: 8000 2024-02-01T10:48:03.762 DEBUG:boto:Params: {} 2024-02-01T10:48:03.762 DEBUG:boto:Token: None 2024-02-01T10:48:03.762 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:03 GMT /vclfcn-59/ 2024-02-01T10:48:03.763 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nMZ+QSOdClyZcZD5pxyoNs9MRBQ= 2024-02-01T10:48:03.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nMZ+QSOdClyZcZD5pxyoNs9MRBQ=', 'Content-Length': '0'} 2024-02-01T10:48:03.767 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000202e392ff4a07a88-0065bb76e3-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:03.768 DEBUG:boto:b'vclfcn-59obj11000falseobj12024-02-01T10:46:46.093Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:03.768 DEBUG:boto:path=/vclfcn-59/obj1 2024-02-01T10:48:03.768 DEBUG:boto:auth_path=/vclfcn-59/obj1 2024-02-01T10:48:03.768 DEBUG:boto:Method: HEAD 2024-02-01T10:48:03.768 DEBUG:boto:Path: /vclfcn-59/obj1 2024-02-01T10:48:03.768 DEBUG:boto:Data: 2024-02-01T10:48:03.769 DEBUG:boto:Headers: {} 2024-02-01T10:48:03.769 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:03.769 DEBUG:boto:Port: 8001 2024-02-01T10:48:03.769 DEBUG:boto:Params: {} 2024-02-01T10:48:03.769 DEBUG:boto:Token: None 2024-02-01T10:48:03.769 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:03 GMT /vclfcn-59/obj1 2024-02-01T10:48:03.769 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:DRL70hYOeBViphQ7vt9pzwmKjgQ= 2024-02-01T10:48:03.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:DRL70hYOeBViphQ7vt9pzwmKjgQ=', 'Content-Length': '0'} 2024-02-01T10:48:03.772 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000798a757c7c267066-0065bb76e3-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:03.772 DEBUG:boto:path=/vclfcn-59/obj1 2024-02-01T10:48:03.772 DEBUG:boto:auth_path=/vclfcn-59/obj1 2024-02-01T10:48:03.773 DEBUG:boto:Method: HEAD 2024-02-01T10:48:03.773 DEBUG:boto:Path: /vclfcn-59/obj1 2024-02-01T10:48:03.773 DEBUG:boto:Data: 2024-02-01T10:48:03.773 DEBUG:boto:Headers: {} 2024-02-01T10:48:03.773 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:03.773 DEBUG:boto:Port: 8000 2024-02-01T10:48:03.773 DEBUG:boto:Params: {} 2024-02-01T10:48:03.773 DEBUG:boto:Token: None 2024-02-01T10:48:03.773 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:03 GMT /vclfcn-59/obj1 2024-02-01T10:48:03.774 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:DRL70hYOeBViphQ7vt9pzwmKjgQ= 2024-02-01T10:48:03.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:DRL70hYOeBViphQ7vt9pzwmKjgQ=', 'Content-Length': '0'} 2024-02-01T10:48:03.777 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002edd565b4f2c7c51-0065bb76e3-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:03.777 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:48:03.777 DEBUG:boto:path=/vclfcn-59/obj1 2024-02-01T10:48:03.777 DEBUG:boto:auth_path=/vclfcn-59/obj1 2024-02-01T10:48:03.777 DEBUG:boto:Method: GET 2024-02-01T10:48:03.777 DEBUG:boto:Path: /vclfcn-59/obj1 2024-02-01T10:48:03.777 DEBUG:boto:Data: 2024-02-01T10:48:03.777 DEBUG:boto:Headers: {} 2024-02-01T10:48:03.778 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:03.778 DEBUG:boto:Port: 8001 2024-02-01T10:48:03.778 DEBUG:boto:Params: {} 2024-02-01T10:48:03.778 DEBUG:boto:Token: None 2024-02-01T10:48:03.778 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:03 GMT /vclfcn-59/obj1 2024-02-01T10:48:03.778 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:us728yHOg9AiBMF0YewT1eUjOaU= 2024-02-01T10:48:03.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:us728yHOg9AiBMF0YewT1eUjOaU=', 'Content-Length': '0'} 2024-02-01T10:48:03.781 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000050da9b2e3c9b0079-0065bb76e3-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:03.824 DEBUG:boto:path=/vclfcn-59/obj1 2024-02-01T10:48:03.824 DEBUG:boto:auth_path=/vclfcn-59/obj1 2024-02-01T10:48:03.825 DEBUG:boto:Method: GET 2024-02-01T10:48:03.825 DEBUG:boto:Path: /vclfcn-59/obj1 2024-02-01T10:48:03.825 DEBUG:boto:Data: 2024-02-01T10:48:03.825 DEBUG:boto:Headers: {} 2024-02-01T10:48:03.825 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:03.825 DEBUG:boto:Port: 8000 2024-02-01T10:48:03.825 DEBUG:boto:Params: {} 2024-02-01T10:48:03.825 DEBUG:boto:Token: None 2024-02-01T10:48:03.825 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:03 GMT /vclfcn-59/obj1 2024-02-01T10:48:03.826 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:us728yHOg9AiBMF0YewT1eUjOaU= 2024-02-01T10:48:03.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:us728yHOg9AiBMF0YewT1eUjOaU=', 'Content-Length': '0'} 2024-02-01T10:48:03.828 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000008ecd35bc49d35c0-0065bb76e3-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:03.829 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:03.829 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:03.829 DEBUG:boto:path=/vclfcn-59/?prefix=obj1 2024-02-01T10:48:03.829 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj1 2024-02-01T10:48:03.829 DEBUG:boto:Method: GET 2024-02-01T10:48:03.829 DEBUG:boto:Path: /vclfcn-59/?prefix=obj1 2024-02-01T10:48:03.829 DEBUG:boto:Data: 2024-02-01T10:48:03.830 DEBUG:boto:Headers: {} 2024-02-01T10:48:03.830 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:03.830 DEBUG:boto:Port: 8001 2024-02-01T10:48:03.830 DEBUG:boto:Params: {} 2024-02-01T10:48:03.830 DEBUG:boto:Token: None 2024-02-01T10:48:03.830 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:03 GMT /vclfcn-59/ 2024-02-01T10:48:03.830 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nMZ+QSOdClyZcZD5pxyoNs9MRBQ= 2024-02-01T10:48:03.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nMZ+QSOdClyZcZD5pxyoNs9MRBQ=', 'Content-Length': '0'} 2024-02-01T10:48:03.834 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002056ffad5dfc1379-0065bb76e3-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:03.876 DEBUG:boto:b'vclfcn-59obj11000falseobj12024-02-01T10:46:46.093Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:03.877 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:03.877 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:03.877 DEBUG:boto:path=/vclfcn-59/?prefix=obj1 2024-02-01T10:48:03.877 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj1 2024-02-01T10:48:03.877 DEBUG:boto:Method: GET 2024-02-01T10:48:03.877 DEBUG:boto:Path: /vclfcn-59/?prefix=obj1 2024-02-01T10:48:03.877 DEBUG:boto:Data: 2024-02-01T10:48:03.877 DEBUG:boto:Headers: {} 2024-02-01T10:48:03.877 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:03.877 DEBUG:boto:Port: 8000 2024-02-01T10:48:03.877 DEBUG:boto:Params: {} 2024-02-01T10:48:03.878 DEBUG:boto:Token: None 2024-02-01T10:48:03.878 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:03 GMT /vclfcn-59/ 2024-02-01T10:48:03.878 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nMZ+QSOdClyZcZD5pxyoNs9MRBQ= 2024-02-01T10:48:03.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nMZ+QSOdClyZcZD5pxyoNs9MRBQ=', 'Content-Length': '0'} 2024-02-01T10:48:03.883 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008af7c91df78cd6c2-0065bb76e3-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:03.883 DEBUG:boto:b'vclfcn-59obj11000falseobj12024-02-01T10:46:46.093Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:03.883 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:48:03.883 DEBUG:boto:path=/vclfcn-59/obj2 2024-02-01T10:48:03.883 DEBUG:boto:auth_path=/vclfcn-59/obj2 2024-02-01T10:48:03.883 DEBUG:boto:path=/vclfcn-59/obj2?versionId=null 2024-02-01T10:48:03.884 DEBUG:boto:auth_path=/vclfcn-59/obj2?versionId=null 2024-02-01T10:48:03.884 DEBUG:boto:Method: GET 2024-02-01T10:48:03.884 DEBUG:boto:Path: /vclfcn-59/obj2?versionId=null 2024-02-01T10:48:03.884 DEBUG:boto:Data: 2024-02-01T10:48:03.884 DEBUG:boto:Headers: {} 2024-02-01T10:48:03.884 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:03.884 DEBUG:boto:Port: 8001 2024-02-01T10:48:03.884 DEBUG:boto:Params: {} 2024-02-01T10:48:03.884 DEBUG:boto:Token: None 2024-02-01T10:48:03.884 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:03 GMT /vclfcn-59/obj2?versionId=null 2024-02-01T10:48:03.885 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:6OMPpdyNFY8v5+Djkp5LY9XVCU0= 2024-02-01T10:48:03.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:6OMPpdyNFY8v5+Djkp5LY9XVCU0=', 'Content-Length': '0'} 2024-02-01T10:48:03.887 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b771dec6cbed03e1-0065bb76e3-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:03.888 DEBUG:boto:path=/vclfcn-59/obj2 2024-02-01T10:48:03.888 DEBUG:boto:auth_path=/vclfcn-59/obj2 2024-02-01T10:48:03.888 DEBUG:boto:path=/vclfcn-59/obj2?versionId=null 2024-02-01T10:48:03.888 DEBUG:boto:auth_path=/vclfcn-59/obj2?versionId=null 2024-02-01T10:48:03.888 DEBUG:boto:Method: GET 2024-02-01T10:48:03.888 DEBUG:boto:Path: /vclfcn-59/obj2?versionId=null 2024-02-01T10:48:03.888 DEBUG:boto:Data: 2024-02-01T10:48:03.888 DEBUG:boto:Headers: {} 2024-02-01T10:48:03.888 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:03.888 DEBUG:boto:Port: 8000 2024-02-01T10:48:03.889 DEBUG:boto:Params: {} 2024-02-01T10:48:03.889 DEBUG:boto:Token: None 2024-02-01T10:48:03.889 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:03 GMT /vclfcn-59/obj2?versionId=null 2024-02-01T10:48:03.889 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:6OMPpdyNFY8v5+Djkp5LY9XVCU0= 2024-02-01T10:48:03.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:6OMPpdyNFY8v5+Djkp5LY9XVCU0=', 'Content-Length': '0'} 2024-02-01T10:48:03.892 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000540ccb0a579a6d58-0065bb76e3-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:03.892 DEBUG:boto:path=/vclfcn-59/obj2 2024-02-01T10:48:03.892 DEBUG:boto:auth_path=/vclfcn-59/obj2 2024-02-01T10:48:03.892 DEBUG:boto:path=/vclfcn-59/obj2?versionId=null 2024-02-01T10:48:03.893 DEBUG:boto:auth_path=/vclfcn-59/obj2?versionId=null 2024-02-01T10:48:03.893 DEBUG:boto:Method: HEAD 2024-02-01T10:48:03.893 DEBUG:boto:Path: /vclfcn-59/obj2?versionId=null 2024-02-01T10:48:03.893 DEBUG:boto:Data: 2024-02-01T10:48:03.893 DEBUG:boto:Headers: {} 2024-02-01T10:48:03.893 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:03.893 DEBUG:boto:Port: 8001 2024-02-01T10:48:03.893 DEBUG:boto:Params: {} 2024-02-01T10:48:03.893 DEBUG:boto:Token: None 2024-02-01T10:48:03.893 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:03 GMT /vclfcn-59/obj2?versionId=null 2024-02-01T10:48:03.894 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZkU8nhBJb5nmnisGgo02jvL6e6U= 2024-02-01T10:48:03.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZkU8nhBJb5nmnisGgo02jvL6e6U=', 'Content-Length': '0'} 2024-02-01T10:48:03.896 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003b4f8584565025ef-0065bb76e3-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:03.896 DEBUG:boto:path=/vclfcn-59/obj2 2024-02-01T10:48:03.897 DEBUG:boto:auth_path=/vclfcn-59/obj2 2024-02-01T10:48:03.897 DEBUG:boto:path=/vclfcn-59/obj2?versionId=null 2024-02-01T10:48:03.897 DEBUG:boto:auth_path=/vclfcn-59/obj2?versionId=null 2024-02-01T10:48:03.897 DEBUG:boto:Method: HEAD 2024-02-01T10:48:03.897 DEBUG:boto:Path: /vclfcn-59/obj2?versionId=null 2024-02-01T10:48:03.897 DEBUG:boto:Data: 2024-02-01T10:48:03.897 DEBUG:boto:Headers: {} 2024-02-01T10:48:03.897 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:03.897 DEBUG:boto:Port: 8000 2024-02-01T10:48:03.897 DEBUG:boto:Params: {} 2024-02-01T10:48:03.897 DEBUG:boto:Token: None 2024-02-01T10:48:03.898 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:03 GMT /vclfcn-59/obj2?versionId=null 2024-02-01T10:48:03.898 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZkU8nhBJb5nmnisGgo02jvL6e6U= 2024-02-01T10:48:03.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZkU8nhBJb5nmnisGgo02jvL6e6U=', 'Content-Length': '0'} 2024-02-01T10:48:03.901 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004258d073a058a689-0065bb76e3-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:03.901 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:48:03.901 DEBUG:boto:path=/vclfcn-59/obj2 2024-02-01T10:48:03.901 DEBUG:boto:auth_path=/vclfcn-59/obj2 2024-02-01T10:48:03.901 DEBUG:boto:path=/vclfcn-59/obj2?versionId=null 2024-02-01T10:48:03.901 DEBUG:boto:auth_path=/vclfcn-59/obj2?versionId=null 2024-02-01T10:48:03.902 DEBUG:boto:Method: GET 2024-02-01T10:48:03.902 DEBUG:boto:Path: /vclfcn-59/obj2?versionId=null 2024-02-01T10:48:03.902 DEBUG:boto:Data: 2024-02-01T10:48:03.902 DEBUG:boto:Headers: {} 2024-02-01T10:48:03.902 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:03.902 DEBUG:boto:Port: 8001 2024-02-01T10:48:03.902 DEBUG:boto:Params: {} 2024-02-01T10:48:03.902 DEBUG:boto:Token: None 2024-02-01T10:48:03.902 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:03 GMT /vclfcn-59/obj2?versionId=null 2024-02-01T10:48:03.902 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:6OMPpdyNFY8v5+Djkp5LY9XVCU0= 2024-02-01T10:48:03.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:6OMPpdyNFY8v5+Djkp5LY9XVCU0=', 'Content-Length': '0'} 2024-02-01T10:48:03.905 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004ad116d11f1edec0-0065bb76e3-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:03.952 DEBUG:boto:path=/vclfcn-59/obj2 2024-02-01T10:48:03.952 DEBUG:boto:auth_path=/vclfcn-59/obj2 2024-02-01T10:48:03.952 DEBUG:boto:path=/vclfcn-59/obj2?versionId=null 2024-02-01T10:48:03.952 DEBUG:boto:auth_path=/vclfcn-59/obj2?versionId=null 2024-02-01T10:48:03.953 DEBUG:boto:Method: GET 2024-02-01T10:48:03.953 DEBUG:boto:Path: /vclfcn-59/obj2?versionId=null 2024-02-01T10:48:03.953 DEBUG:boto:Data: 2024-02-01T10:48:03.953 DEBUG:boto:Headers: {} 2024-02-01T10:48:03.953 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:03.953 DEBUG:boto:Port: 8000 2024-02-01T10:48:03.953 DEBUG:boto:Params: {} 2024-02-01T10:48:03.953 DEBUG:boto:Token: None 2024-02-01T10:48:03.953 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:03 GMT /vclfcn-59/obj2?versionId=null 2024-02-01T10:48:03.954 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:6OMPpdyNFY8v5+Djkp5LY9XVCU0= 2024-02-01T10:48:03.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:6OMPpdyNFY8v5+Djkp5LY9XVCU0=', 'Content-Length': '0'} 2024-02-01T10:48:03.957 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000df660cda4c16fc4c-0065bb76e3-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:03.959 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:03.959 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:03.959 DEBUG:boto:path=/vclfcn-59/?prefix=obj2 2024-02-01T10:48:03.959 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj2 2024-02-01T10:48:03.959 DEBUG:boto:Method: GET 2024-02-01T10:48:03.959 DEBUG:boto:Path: /vclfcn-59/?prefix=obj2 2024-02-01T10:48:03.960 DEBUG:boto:Data: 2024-02-01T10:48:03.960 DEBUG:boto:Headers: {} 2024-02-01T10:48:03.960 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:03.960 DEBUG:boto:Port: 8001 2024-02-01T10:48:03.960 DEBUG:boto:Params: {} 2024-02-01T10:48:03.960 DEBUG:boto:Token: None 2024-02-01T10:48:03.960 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:03 GMT /vclfcn-59/ 2024-02-01T10:48:03.960 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nMZ+QSOdClyZcZD5pxyoNs9MRBQ= 2024-02-01T10:48:03.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nMZ+QSOdClyZcZD5pxyoNs9MRBQ=', 'Content-Length': '0'} 2024-02-01T10:48:03.965 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003387cf97e47d8217-0065bb76e3-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:04.008 DEBUG:boto:b'vclfcn-59obj21000falseobj22024-02-01T10:46:46.111Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:04.009 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:04.009 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:04.009 DEBUG:boto:path=/vclfcn-59/?prefix=obj2 2024-02-01T10:48:04.009 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj2 2024-02-01T10:48:04.009 DEBUG:boto:Method: GET 2024-02-01T10:48:04.009 DEBUG:boto:Path: /vclfcn-59/?prefix=obj2 2024-02-01T10:48:04.009 DEBUG:boto:Data: 2024-02-01T10:48:04.009 DEBUG:boto:Headers: {} 2024-02-01T10:48:04.009 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:04.009 DEBUG:boto:Port: 8000 2024-02-01T10:48:04.009 DEBUG:boto:Params: {} 2024-02-01T10:48:04.010 DEBUG:boto:Token: None 2024-02-01T10:48:04.010 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:04 GMT /vclfcn-59/ 2024-02-01T10:48:04.010 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:c5xribnwSqgeG1HrL3Lia8JSSZU= 2024-02-01T10:48:04.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:c5xribnwSqgeG1HrL3Lia8JSSZU=', 'Content-Length': '0'} 2024-02-01T10:48:04.014 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e21a70b96356fa88-0065bb76e4-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:04.015 DEBUG:boto:b'vclfcn-59obj21000falseobj22024-02-01T10:46:46.111Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:04.015 DEBUG:boto:path=/vclfcn-59/obj2 2024-02-01T10:48:04.015 DEBUG:boto:auth_path=/vclfcn-59/obj2 2024-02-01T10:48:04.015 DEBUG:boto:Method: HEAD 2024-02-01T10:48:04.015 DEBUG:boto:Path: /vclfcn-59/obj2 2024-02-01T10:48:04.015 DEBUG:boto:Data: 2024-02-01T10:48:04.015 DEBUG:boto:Headers: {} 2024-02-01T10:48:04.015 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:04.016 DEBUG:boto:Port: 8001 2024-02-01T10:48:04.016 DEBUG:boto:Params: {} 2024-02-01T10:48:04.016 DEBUG:boto:Token: None 2024-02-01T10:48:04.016 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:04 GMT /vclfcn-59/obj2 2024-02-01T10:48:04.016 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ABoqCRb+jjKvAzQa/eqa+tTGIAs= 2024-02-01T10:48:04.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ABoqCRb+jjKvAzQa/eqa+tTGIAs=', 'Content-Length': '0'} 2024-02-01T10:48:04.019 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d8c050f591087648-0065bb76e4-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:04.019 DEBUG:boto:path=/vclfcn-59/obj2 2024-02-01T10:48:04.019 DEBUG:boto:auth_path=/vclfcn-59/obj2 2024-02-01T10:48:04.019 DEBUG:boto:Method: HEAD 2024-02-01T10:48:04.019 DEBUG:boto:Path: /vclfcn-59/obj2 2024-02-01T10:48:04.019 DEBUG:boto:Data: 2024-02-01T10:48:04.019 DEBUG:boto:Headers: {} 2024-02-01T10:48:04.020 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:04.020 DEBUG:boto:Port: 8000 2024-02-01T10:48:04.020 DEBUG:boto:Params: {} 2024-02-01T10:48:04.020 DEBUG:boto:Token: None 2024-02-01T10:48:04.020 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:04 GMT /vclfcn-59/obj2 2024-02-01T10:48:04.020 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ABoqCRb+jjKvAzQa/eqa+tTGIAs= 2024-02-01T10:48:04.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ABoqCRb+jjKvAzQa/eqa+tTGIAs=', 'Content-Length': '0'} 2024-02-01T10:48:04.023 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f3c8bfb132230a01-0065bb76e4-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:04.023 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:48:04.023 DEBUG:boto:path=/vclfcn-59/obj2 2024-02-01T10:48:04.023 DEBUG:boto:auth_path=/vclfcn-59/obj2 2024-02-01T10:48:04.023 DEBUG:boto:Method: GET 2024-02-01T10:48:04.023 DEBUG:boto:Path: /vclfcn-59/obj2 2024-02-01T10:48:04.024 DEBUG:boto:Data: 2024-02-01T10:48:04.024 DEBUG:boto:Headers: {} 2024-02-01T10:48:04.024 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:04.024 DEBUG:boto:Port: 8001 2024-02-01T10:48:04.024 DEBUG:boto:Params: {} 2024-02-01T10:48:04.024 DEBUG:boto:Token: None 2024-02-01T10:48:04.024 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:04 GMT /vclfcn-59/obj2 2024-02-01T10:48:04.024 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cP/H+mIVDdtSMh8MlLzrtr6YtoA= 2024-02-01T10:48:04.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cP/H+mIVDdtSMh8MlLzrtr6YtoA=', 'Content-Length': '0'} 2024-02-01T10:48:04.027 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000005a502b892177a45-0065bb76e4-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:04.068 DEBUG:boto:path=/vclfcn-59/obj2 2024-02-01T10:48:04.068 DEBUG:boto:auth_path=/vclfcn-59/obj2 2024-02-01T10:48:04.068 DEBUG:boto:Method: GET 2024-02-01T10:48:04.069 DEBUG:boto:Path: /vclfcn-59/obj2 2024-02-01T10:48:04.069 DEBUG:boto:Data: 2024-02-01T10:48:04.069 DEBUG:boto:Headers: {} 2024-02-01T10:48:04.069 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:04.069 DEBUG:boto:Port: 8000 2024-02-01T10:48:04.069 DEBUG:boto:Params: {} 2024-02-01T10:48:04.069 DEBUG:boto:Token: None 2024-02-01T10:48:04.069 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:04 GMT /vclfcn-59/obj2 2024-02-01T10:48:04.069 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cP/H+mIVDdtSMh8MlLzrtr6YtoA= 2024-02-01T10:48:04.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cP/H+mIVDdtSMh8MlLzrtr6YtoA=', 'Content-Length': '0'} 2024-02-01T10:48:04.072 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004497603f4a035c29-0065bb76e4-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:04.073 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:04.073 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:04.073 DEBUG:boto:path=/vclfcn-59/?prefix=obj2 2024-02-01T10:48:04.073 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj2 2024-02-01T10:48:04.073 DEBUG:boto:Method: GET 2024-02-01T10:48:04.073 DEBUG:boto:Path: /vclfcn-59/?prefix=obj2 2024-02-01T10:48:04.073 DEBUG:boto:Data: 2024-02-01T10:48:04.073 DEBUG:boto:Headers: {} 2024-02-01T10:48:04.074 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:04.074 DEBUG:boto:Port: 8001 2024-02-01T10:48:04.074 DEBUG:boto:Params: {} 2024-02-01T10:48:04.074 DEBUG:boto:Token: None 2024-02-01T10:48:04.074 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:04 GMT /vclfcn-59/ 2024-02-01T10:48:04.074 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:c5xribnwSqgeG1HrL3Lia8JSSZU= 2024-02-01T10:48:04.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:c5xribnwSqgeG1HrL3Lia8JSSZU=', 'Content-Length': '0'} 2024-02-01T10:48:04.078 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fea47503c36eb67d-0065bb76e4-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:04.078 DEBUG:boto:b'vclfcn-59obj21000falseobj22024-02-01T10:46:46.111Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:04.079 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:04.079 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:04.079 DEBUG:boto:path=/vclfcn-59/?prefix=obj2 2024-02-01T10:48:04.079 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj2 2024-02-01T10:48:04.079 DEBUG:boto:Method: GET 2024-02-01T10:48:04.079 DEBUG:boto:Path: /vclfcn-59/?prefix=obj2 2024-02-01T10:48:04.079 DEBUG:boto:Data: 2024-02-01T10:48:04.079 DEBUG:boto:Headers: {} 2024-02-01T10:48:04.079 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:04.079 DEBUG:boto:Port: 8000 2024-02-01T10:48:04.079 DEBUG:boto:Params: {} 2024-02-01T10:48:04.080 DEBUG:boto:Token: None 2024-02-01T10:48:04.080 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:04 GMT /vclfcn-59/ 2024-02-01T10:48:04.080 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:c5xribnwSqgeG1HrL3Lia8JSSZU= 2024-02-01T10:48:04.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:c5xribnwSqgeG1HrL3Lia8JSSZU=', 'Content-Length': '0'} 2024-02-01T10:48:04.084 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086ee957a1776f936-0065bb76e4-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:04.085 DEBUG:boto:b'vclfcn-59obj21000falseobj22024-02-01T10:46:46.111Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:04.085 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:48:04.085 DEBUG:boto:path=/vclfcn-59/obj3 2024-02-01T10:48:04.086 DEBUG:boto:auth_path=/vclfcn-59/obj3 2024-02-01T10:48:04.086 DEBUG:boto:path=/vclfcn-59/obj3?versionId=null 2024-02-01T10:48:04.086 DEBUG:boto:auth_path=/vclfcn-59/obj3?versionId=null 2024-02-01T10:48:04.086 DEBUG:boto:Method: GET 2024-02-01T10:48:04.086 DEBUG:boto:Path: /vclfcn-59/obj3?versionId=null 2024-02-01T10:48:04.086 DEBUG:boto:Data: 2024-02-01T10:48:04.086 DEBUG:boto:Headers: {} 2024-02-01T10:48:04.086 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:04.086 DEBUG:boto:Port: 8001 2024-02-01T10:48:04.086 DEBUG:boto:Params: {} 2024-02-01T10:48:04.087 DEBUG:boto:Token: None 2024-02-01T10:48:04.087 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:04 GMT /vclfcn-59/obj3?versionId=null 2024-02-01T10:48:04.087 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Qe5NKacCUZxGJAJp+z7aFFb62j8= 2024-02-01T10:48:04.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Qe5NKacCUZxGJAJp+z7aFFb62j8=', 'Content-Length': '0'} 2024-02-01T10:48:04.090 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003e86ccad0db4b616-0065bb76e4-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:04.136 DEBUG:boto:path=/vclfcn-59/obj3 2024-02-01T10:48:04.136 DEBUG:boto:auth_path=/vclfcn-59/obj3 2024-02-01T10:48:04.136 DEBUG:boto:path=/vclfcn-59/obj3?versionId=null 2024-02-01T10:48:04.137 DEBUG:boto:auth_path=/vclfcn-59/obj3?versionId=null 2024-02-01T10:48:04.137 DEBUG:boto:Method: GET 2024-02-01T10:48:04.137 DEBUG:boto:Path: /vclfcn-59/obj3?versionId=null 2024-02-01T10:48:04.137 DEBUG:boto:Data: 2024-02-01T10:48:04.137 DEBUG:boto:Headers: {} 2024-02-01T10:48:04.137 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:04.137 DEBUG:boto:Port: 8000 2024-02-01T10:48:04.137 DEBUG:boto:Params: {} 2024-02-01T10:48:04.137 DEBUG:boto:Token: None 2024-02-01T10:48:04.138 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:04 GMT /vclfcn-59/obj3?versionId=null 2024-02-01T10:48:04.138 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Qe5NKacCUZxGJAJp+z7aFFb62j8= 2024-02-01T10:48:04.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Qe5NKacCUZxGJAJp+z7aFFb62j8=', 'Content-Length': '0'} 2024-02-01T10:48:04.141 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c8c13b66e9443f2f-0065bb76e4-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:04.141 DEBUG:boto:path=/vclfcn-59/obj3 2024-02-01T10:48:04.142 DEBUG:boto:auth_path=/vclfcn-59/obj3 2024-02-01T10:48:04.142 DEBUG:boto:path=/vclfcn-59/obj3?versionId=null 2024-02-01T10:48:04.142 DEBUG:boto:auth_path=/vclfcn-59/obj3?versionId=null 2024-02-01T10:48:04.142 DEBUG:boto:Method: HEAD 2024-02-01T10:48:04.142 DEBUG:boto:Path: /vclfcn-59/obj3?versionId=null 2024-02-01T10:48:04.142 DEBUG:boto:Data: 2024-02-01T10:48:04.142 DEBUG:boto:Headers: {} 2024-02-01T10:48:04.142 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:04.142 DEBUG:boto:Port: 8001 2024-02-01T10:48:04.142 DEBUG:boto:Params: {} 2024-02-01T10:48:04.143 DEBUG:boto:Token: None 2024-02-01T10:48:04.143 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:04 GMT /vclfcn-59/obj3?versionId=null 2024-02-01T10:48:04.143 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:VPykChc5bs4vJ+blRjoWip1QxjU= 2024-02-01T10:48:04.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:VPykChc5bs4vJ+blRjoWip1QxjU=', 'Content-Length': '0'} 2024-02-01T10:48:04.146 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000997aea224359be72-0065bb76e4-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:04.146 DEBUG:boto:path=/vclfcn-59/obj3 2024-02-01T10:48:04.146 DEBUG:boto:auth_path=/vclfcn-59/obj3 2024-02-01T10:48:04.146 DEBUG:boto:path=/vclfcn-59/obj3?versionId=null 2024-02-01T10:48:04.146 DEBUG:boto:auth_path=/vclfcn-59/obj3?versionId=null 2024-02-01T10:48:04.146 DEBUG:boto:Method: HEAD 2024-02-01T10:48:04.146 DEBUG:boto:Path: /vclfcn-59/obj3?versionId=null 2024-02-01T10:48:04.146 DEBUG:boto:Data: 2024-02-01T10:48:04.147 DEBUG:boto:Headers: {} 2024-02-01T10:48:04.147 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:04.147 DEBUG:boto:Port: 8000 2024-02-01T10:48:04.147 DEBUG:boto:Params: {} 2024-02-01T10:48:04.147 DEBUG:boto:Token: None 2024-02-01T10:48:04.147 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:04 GMT /vclfcn-59/obj3?versionId=null 2024-02-01T10:48:04.147 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:VPykChc5bs4vJ+blRjoWip1QxjU= 2024-02-01T10:48:04.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:VPykChc5bs4vJ+blRjoWip1QxjU=', 'Content-Length': '0'} 2024-02-01T10:48:04.150 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008038c45e9ce76de1-0065bb76e4-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:04.150 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:48:04.150 DEBUG:boto:path=/vclfcn-59/obj3 2024-02-01T10:48:04.150 DEBUG:boto:auth_path=/vclfcn-59/obj3 2024-02-01T10:48:04.150 DEBUG:boto:path=/vclfcn-59/obj3?versionId=null 2024-02-01T10:48:04.151 DEBUG:boto:auth_path=/vclfcn-59/obj3?versionId=null 2024-02-01T10:48:04.151 DEBUG:boto:Method: GET 2024-02-01T10:48:04.151 DEBUG:boto:Path: /vclfcn-59/obj3?versionId=null 2024-02-01T10:48:04.151 DEBUG:boto:Data: 2024-02-01T10:48:04.151 DEBUG:boto:Headers: {} 2024-02-01T10:48:04.151 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:04.151 DEBUG:boto:Port: 8001 2024-02-01T10:48:04.151 DEBUG:boto:Params: {} 2024-02-01T10:48:04.151 DEBUG:boto:Token: None 2024-02-01T10:48:04.151 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:04 GMT /vclfcn-59/obj3?versionId=null 2024-02-01T10:48:04.152 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Qe5NKacCUZxGJAJp+z7aFFb62j8= 2024-02-01T10:48:04.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Qe5NKacCUZxGJAJp+z7aFFb62j8=', 'Content-Length': '0'} 2024-02-01T10:48:04.154 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000036ef55e4da23bed7-0065bb76e4-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:04.200 DEBUG:boto:path=/vclfcn-59/obj3 2024-02-01T10:48:04.200 DEBUG:boto:auth_path=/vclfcn-59/obj3 2024-02-01T10:48:04.200 DEBUG:boto:path=/vclfcn-59/obj3?versionId=null 2024-02-01T10:48:04.201 DEBUG:boto:auth_path=/vclfcn-59/obj3?versionId=null 2024-02-01T10:48:04.201 DEBUG:boto:Method: GET 2024-02-01T10:48:04.201 DEBUG:boto:Path: /vclfcn-59/obj3?versionId=null 2024-02-01T10:48:04.201 DEBUG:boto:Data: 2024-02-01T10:48:04.201 DEBUG:boto:Headers: {} 2024-02-01T10:48:04.201 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:04.201 DEBUG:boto:Port: 8000 2024-02-01T10:48:04.201 DEBUG:boto:Params: {} 2024-02-01T10:48:04.201 DEBUG:boto:Token: None 2024-02-01T10:48:04.202 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:04 GMT /vclfcn-59/obj3?versionId=null 2024-02-01T10:48:04.202 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Qe5NKacCUZxGJAJp+z7aFFb62j8= 2024-02-01T10:48:04.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Qe5NKacCUZxGJAJp+z7aFFb62j8=', 'Content-Length': '0'} 2024-02-01T10:48:04.205 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d5f99dff9604d653-0065bb76e4-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:04.205 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:04.205 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:04.205 DEBUG:boto:path=/vclfcn-59/?prefix=obj3 2024-02-01T10:48:04.205 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj3 2024-02-01T10:48:04.206 DEBUG:boto:Method: GET 2024-02-01T10:48:04.206 DEBUG:boto:Path: /vclfcn-59/?prefix=obj3 2024-02-01T10:48:04.206 DEBUG:boto:Data: 2024-02-01T10:48:04.206 DEBUG:boto:Headers: {} 2024-02-01T10:48:04.206 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:04.206 DEBUG:boto:Port: 8001 2024-02-01T10:48:04.206 DEBUG:boto:Params: {} 2024-02-01T10:48:04.206 DEBUG:boto:Token: None 2024-02-01T10:48:04.206 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:04 GMT /vclfcn-59/ 2024-02-01T10:48:04.207 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:c5xribnwSqgeG1HrL3Lia8JSSZU= 2024-02-01T10:48:04.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:c5xribnwSqgeG1HrL3Lia8JSSZU=', 'Content-Length': '0'} 2024-02-01T10:48:04.211 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f8ac735a20fa035-0065bb76e4-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:04.256 DEBUG:boto:b'vclfcn-59obj31000falseobj32024-02-01T10:46:46.161Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:04.257 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:04.257 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:04.257 DEBUG:boto:path=/vclfcn-59/?prefix=obj3 2024-02-01T10:48:04.257 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj3 2024-02-01T10:48:04.257 DEBUG:boto:Method: GET 2024-02-01T10:48:04.257 DEBUG:boto:Path: /vclfcn-59/?prefix=obj3 2024-02-01T10:48:04.257 DEBUG:boto:Data: 2024-02-01T10:48:04.257 DEBUG:boto:Headers: {} 2024-02-01T10:48:04.257 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:04.257 DEBUG:boto:Port: 8000 2024-02-01T10:48:04.258 DEBUG:boto:Params: {} 2024-02-01T10:48:04.258 DEBUG:boto:Token: None 2024-02-01T10:48:04.258 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:04 GMT /vclfcn-59/ 2024-02-01T10:48:04.258 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:c5xribnwSqgeG1HrL3Lia8JSSZU= 2024-02-01T10:48:04.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:c5xribnwSqgeG1HrL3Lia8JSSZU=', 'Content-Length': '0'} 2024-02-01T10:48:04.262 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dabc61b62078131c-0065bb76e4-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:04.262 DEBUG:boto:b'vclfcn-59obj31000falseobj32024-02-01T10:46:46.161Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:04.263 DEBUG:boto:path=/vclfcn-59/obj3 2024-02-01T10:48:04.263 DEBUG:boto:auth_path=/vclfcn-59/obj3 2024-02-01T10:48:04.263 DEBUG:boto:Method: HEAD 2024-02-01T10:48:04.263 DEBUG:boto:Path: /vclfcn-59/obj3 2024-02-01T10:48:04.263 DEBUG:boto:Data: 2024-02-01T10:48:04.263 DEBUG:boto:Headers: {} 2024-02-01T10:48:04.264 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:04.264 DEBUG:boto:Port: 8001 2024-02-01T10:48:04.264 DEBUG:boto:Params: {} 2024-02-01T10:48:04.264 DEBUG:boto:Token: None 2024-02-01T10:48:04.264 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:04 GMT /vclfcn-59/obj3 2024-02-01T10:48:04.264 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ebR/XVFp4BsR1CvuTqgu0XzdIgE= 2024-02-01T10:48:04.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ebR/XVFp4BsR1CvuTqgu0XzdIgE=', 'Content-Length': '0'} 2024-02-01T10:48:04.267 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003526c2f4b2f873a3-0065bb76e4-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:04.267 DEBUG:boto:path=/vclfcn-59/obj3 2024-02-01T10:48:04.267 DEBUG:boto:auth_path=/vclfcn-59/obj3 2024-02-01T10:48:04.267 DEBUG:boto:Method: HEAD 2024-02-01T10:48:04.267 DEBUG:boto:Path: /vclfcn-59/obj3 2024-02-01T10:48:04.268 DEBUG:boto:Data: 2024-02-01T10:48:04.268 DEBUG:boto:Headers: {} 2024-02-01T10:48:04.268 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:04.268 DEBUG:boto:Port: 8000 2024-02-01T10:48:04.268 DEBUG:boto:Params: {} 2024-02-01T10:48:04.268 DEBUG:boto:Token: None 2024-02-01T10:48:04.268 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:04 GMT /vclfcn-59/obj3 2024-02-01T10:48:04.268 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ebR/XVFp4BsR1CvuTqgu0XzdIgE= 2024-02-01T10:48:04.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ebR/XVFp4BsR1CvuTqgu0XzdIgE=', 'Content-Length': '0'} 2024-02-01T10:48:04.271 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000055ab11594d552ed7-0065bb76e4-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:04.271 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:48:04.271 DEBUG:boto:path=/vclfcn-59/obj3 2024-02-01T10:48:04.271 DEBUG:boto:auth_path=/vclfcn-59/obj3 2024-02-01T10:48:04.271 DEBUG:boto:Method: GET 2024-02-01T10:48:04.272 DEBUG:boto:Path: /vclfcn-59/obj3 2024-02-01T10:48:04.272 DEBUG:boto:Data: 2024-02-01T10:48:04.272 DEBUG:boto:Headers: {} 2024-02-01T10:48:04.272 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:04.272 DEBUG:boto:Port: 8001 2024-02-01T10:48:04.272 DEBUG:boto:Params: {} 2024-02-01T10:48:04.272 DEBUG:boto:Token: None 2024-02-01T10:48:04.272 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:04 GMT /vclfcn-59/obj3 2024-02-01T10:48:04.272 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BYcmPGsyGOKwvMAZw/XIYXZK/hY= 2024-02-01T10:48:04.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BYcmPGsyGOKwvMAZw/XIYXZK/hY=', 'Content-Length': '0'} 2024-02-01T10:48:04.275 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001646785d7ff6fc06-0065bb76e4-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:04.316 DEBUG:boto:path=/vclfcn-59/obj3 2024-02-01T10:48:04.316 DEBUG:boto:auth_path=/vclfcn-59/obj3 2024-02-01T10:48:04.316 DEBUG:boto:Method: GET 2024-02-01T10:48:04.317 DEBUG:boto:Path: /vclfcn-59/obj3 2024-02-01T10:48:04.317 DEBUG:boto:Data: 2024-02-01T10:48:04.317 DEBUG:boto:Headers: {} 2024-02-01T10:48:04.317 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:04.317 DEBUG:boto:Port: 8000 2024-02-01T10:48:04.317 DEBUG:boto:Params: {} 2024-02-01T10:48:04.317 DEBUG:boto:Token: None 2024-02-01T10:48:04.317 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:04 GMT /vclfcn-59/obj3 2024-02-01T10:48:04.317 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BYcmPGsyGOKwvMAZw/XIYXZK/hY= 2024-02-01T10:48:04.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BYcmPGsyGOKwvMAZw/XIYXZK/hY=', 'Content-Length': '0'} 2024-02-01T10:48:04.321 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e1f27b08e510a3da-0065bb76e4-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:04.321 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:04.321 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:04.321 DEBUG:boto:path=/vclfcn-59/?prefix=obj3 2024-02-01T10:48:04.321 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj3 2024-02-01T10:48:04.321 DEBUG:boto:Method: GET 2024-02-01T10:48:04.321 DEBUG:boto:Path: /vclfcn-59/?prefix=obj3 2024-02-01T10:48:04.321 DEBUG:boto:Data: 2024-02-01T10:48:04.322 DEBUG:boto:Headers: {} 2024-02-01T10:48:04.322 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:04.322 DEBUG:boto:Port: 8001 2024-02-01T10:48:04.322 DEBUG:boto:Params: {} 2024-02-01T10:48:04.322 DEBUG:boto:Token: None 2024-02-01T10:48:04.322 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:04 GMT /vclfcn-59/ 2024-02-01T10:48:04.322 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:c5xribnwSqgeG1HrL3Lia8JSSZU= 2024-02-01T10:48:04.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:c5xribnwSqgeG1HrL3Lia8JSSZU=', 'Content-Length': '0'} 2024-02-01T10:48:04.327 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc78ebd49ba59d35-0065bb76e4-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:04.327 DEBUG:boto:b'vclfcn-59obj31000falseobj32024-02-01T10:46:46.161Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:04.327 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:04.327 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:04.327 DEBUG:boto:path=/vclfcn-59/?prefix=obj3 2024-02-01T10:48:04.328 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj3 2024-02-01T10:48:04.328 DEBUG:boto:Method: GET 2024-02-01T10:48:04.328 DEBUG:boto:Path: /vclfcn-59/?prefix=obj3 2024-02-01T10:48:04.328 DEBUG:boto:Data: 2024-02-01T10:48:04.328 DEBUG:boto:Headers: {} 2024-02-01T10:48:04.328 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:04.328 DEBUG:boto:Port: 8000 2024-02-01T10:48:04.328 DEBUG:boto:Params: {} 2024-02-01T10:48:04.328 DEBUG:boto:Token: None 2024-02-01T10:48:04.329 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:04 GMT /vclfcn-59/ 2024-02-01T10:48:04.329 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:c5xribnwSqgeG1HrL3Lia8JSSZU= 2024-02-01T10:48:04.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:c5xribnwSqgeG1HrL3Lia8JSSZU=', 'Content-Length': '0'} 2024-02-01T10:48:04.333 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a86c2ff40bd89a0-0065bb76e4-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:04.333 DEBUG:boto:b'vclfcn-59obj31000falseobj32024-02-01T10:46:46.161Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:04.333 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:48:04.334 DEBUG:boto:path=/vclfcn-59/obj4 2024-02-01T10:48:04.334 DEBUG:boto:auth_path=/vclfcn-59/obj4 2024-02-01T10:48:04.334 DEBUG:boto:path=/vclfcn-59/obj4?versionId=null 2024-02-01T10:48:04.334 DEBUG:boto:auth_path=/vclfcn-59/obj4?versionId=null 2024-02-01T10:48:04.334 DEBUG:boto:Method: GET 2024-02-01T10:48:04.334 DEBUG:boto:Path: /vclfcn-59/obj4?versionId=null 2024-02-01T10:48:04.334 DEBUG:boto:Data: 2024-02-01T10:48:04.334 DEBUG:boto:Headers: {} 2024-02-01T10:48:04.334 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:04.334 DEBUG:boto:Port: 8001 2024-02-01T10:48:04.335 DEBUG:boto:Params: {} 2024-02-01T10:48:04.335 DEBUG:boto:Token: None 2024-02-01T10:48:04.335 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:04 GMT /vclfcn-59/obj4?versionId=null 2024-02-01T10:48:04.335 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:u3WXnErkMUW+5JQH8G7SQCz4SEI= 2024-02-01T10:48:04.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:u3WXnErkMUW+5JQH8G7SQCz4SEI=', 'Content-Length': '0'} 2024-02-01T10:48:04.338 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000accec13740a5bd57-0065bb76e4-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:04.380 DEBUG:boto:path=/vclfcn-59/obj4 2024-02-01T10:48:04.380 DEBUG:boto:auth_path=/vclfcn-59/obj4 2024-02-01T10:48:04.380 DEBUG:boto:path=/vclfcn-59/obj4?versionId=null 2024-02-01T10:48:04.380 DEBUG:boto:auth_path=/vclfcn-59/obj4?versionId=null 2024-02-01T10:48:04.381 DEBUG:boto:Method: GET 2024-02-01T10:48:04.381 DEBUG:boto:Path: /vclfcn-59/obj4?versionId=null 2024-02-01T10:48:04.381 DEBUG:boto:Data: 2024-02-01T10:48:04.381 DEBUG:boto:Headers: {} 2024-02-01T10:48:04.381 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:04.381 DEBUG:boto:Port: 8000 2024-02-01T10:48:04.381 DEBUG:boto:Params: {} 2024-02-01T10:48:04.381 DEBUG:boto:Token: None 2024-02-01T10:48:04.381 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:04 GMT /vclfcn-59/obj4?versionId=null 2024-02-01T10:48:04.382 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:u3WXnErkMUW+5JQH8G7SQCz4SEI= 2024-02-01T10:48:04.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:u3WXnErkMUW+5JQH8G7SQCz4SEI=', 'Content-Length': '0'} 2024-02-01T10:48:04.384 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000369c5d234cdcd213-0065bb76e4-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:04.385 DEBUG:boto:path=/vclfcn-59/obj4 2024-02-01T10:48:04.385 DEBUG:boto:auth_path=/vclfcn-59/obj4 2024-02-01T10:48:04.385 DEBUG:boto:path=/vclfcn-59/obj4?versionId=null 2024-02-01T10:48:04.385 DEBUG:boto:auth_path=/vclfcn-59/obj4?versionId=null 2024-02-01T10:48:04.385 DEBUG:boto:Method: HEAD 2024-02-01T10:48:04.385 DEBUG:boto:Path: /vclfcn-59/obj4?versionId=null 2024-02-01T10:48:04.385 DEBUG:boto:Data: 2024-02-01T10:48:04.386 DEBUG:boto:Headers: {} 2024-02-01T10:48:04.386 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:04.386 DEBUG:boto:Port: 8001 2024-02-01T10:48:04.386 DEBUG:boto:Params: {} 2024-02-01T10:48:04.386 DEBUG:boto:Token: None 2024-02-01T10:48:04.386 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:04 GMT /vclfcn-59/obj4?versionId=null 2024-02-01T10:48:04.386 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:n6Wp++USSMlM7NfDZvdNEiyS6YA= 2024-02-01T10:48:04.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:n6Wp++USSMlM7NfDZvdNEiyS6YA=', 'Content-Length': '0'} 2024-02-01T10:48:04.389 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002d55d70107d65b16-0065bb76e4-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:04.389 DEBUG:boto:path=/vclfcn-59/obj4 2024-02-01T10:48:04.389 DEBUG:boto:auth_path=/vclfcn-59/obj4 2024-02-01T10:48:04.389 DEBUG:boto:path=/vclfcn-59/obj4?versionId=null 2024-02-01T10:48:04.390 DEBUG:boto:auth_path=/vclfcn-59/obj4?versionId=null 2024-02-01T10:48:04.390 DEBUG:boto:Method: HEAD 2024-02-01T10:48:04.390 DEBUG:boto:Path: /vclfcn-59/obj4?versionId=null 2024-02-01T10:48:04.390 DEBUG:boto:Data: 2024-02-01T10:48:04.390 DEBUG:boto:Headers: {} 2024-02-01T10:48:04.390 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:04.390 DEBUG:boto:Port: 8000 2024-02-01T10:48:04.390 DEBUG:boto:Params: {} 2024-02-01T10:48:04.390 DEBUG:boto:Token: None 2024-02-01T10:48:04.390 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:04 GMT /vclfcn-59/obj4?versionId=null 2024-02-01T10:48:04.391 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:n6Wp++USSMlM7NfDZvdNEiyS6YA= 2024-02-01T10:48:04.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:n6Wp++USSMlM7NfDZvdNEiyS6YA=', 'Content-Length': '0'} 2024-02-01T10:48:04.393 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fc40c0830d3e83bf-0065bb76e4-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:04.393 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:48:04.394 DEBUG:boto:path=/vclfcn-59/obj4 2024-02-01T10:48:04.394 DEBUG:boto:auth_path=/vclfcn-59/obj4 2024-02-01T10:48:04.394 DEBUG:boto:path=/vclfcn-59/obj4?versionId=null 2024-02-01T10:48:04.394 DEBUG:boto:auth_path=/vclfcn-59/obj4?versionId=null 2024-02-01T10:48:04.394 DEBUG:boto:Method: GET 2024-02-01T10:48:04.394 DEBUG:boto:Path: /vclfcn-59/obj4?versionId=null 2024-02-01T10:48:04.394 DEBUG:boto:Data: 2024-02-01T10:48:04.394 DEBUG:boto:Headers: {} 2024-02-01T10:48:04.394 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:04.394 DEBUG:boto:Port: 8001 2024-02-01T10:48:04.394 DEBUG:boto:Params: {} 2024-02-01T10:48:04.395 DEBUG:boto:Token: None 2024-02-01T10:48:04.395 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:04 GMT /vclfcn-59/obj4?versionId=null 2024-02-01T10:48:04.395 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:u3WXnErkMUW+5JQH8G7SQCz4SEI= 2024-02-01T10:48:04.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:u3WXnErkMUW+5JQH8G7SQCz4SEI=', 'Content-Length': '0'} 2024-02-01T10:48:04.397 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d6be55f4790462ef-0065bb76e4-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:04.440 DEBUG:boto:path=/vclfcn-59/obj4 2024-02-01T10:48:04.440 DEBUG:boto:auth_path=/vclfcn-59/obj4 2024-02-01T10:48:04.440 DEBUG:boto:path=/vclfcn-59/obj4?versionId=null 2024-02-01T10:48:04.441 DEBUG:boto:auth_path=/vclfcn-59/obj4?versionId=null 2024-02-01T10:48:04.441 DEBUG:boto:Method: GET 2024-02-01T10:48:04.441 DEBUG:boto:Path: /vclfcn-59/obj4?versionId=null 2024-02-01T10:48:04.441 DEBUG:boto:Data: 2024-02-01T10:48:04.441 DEBUG:boto:Headers: {} 2024-02-01T10:48:04.441 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:04.441 DEBUG:boto:Port: 8000 2024-02-01T10:48:04.441 DEBUG:boto:Params: {} 2024-02-01T10:48:04.441 DEBUG:boto:Token: None 2024-02-01T10:48:04.442 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:04 GMT /vclfcn-59/obj4?versionId=null 2024-02-01T10:48:04.442 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:u3WXnErkMUW+5JQH8G7SQCz4SEI= 2024-02-01T10:48:04.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:u3WXnErkMUW+5JQH8G7SQCz4SEI=', 'Content-Length': '0'} 2024-02-01T10:48:04.445 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e446bf04b0364116-0065bb76e4-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:04.445 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:04.445 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:04.445 DEBUG:boto:path=/vclfcn-59/?prefix=obj4 2024-02-01T10:48:04.445 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj4 2024-02-01T10:48:04.445 DEBUG:boto:Method: GET 2024-02-01T10:48:04.445 DEBUG:boto:Path: /vclfcn-59/?prefix=obj4 2024-02-01T10:48:04.446 DEBUG:boto:Data: 2024-02-01T10:48:04.446 DEBUG:boto:Headers: {} 2024-02-01T10:48:04.446 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:04.446 DEBUG:boto:Port: 8001 2024-02-01T10:48:04.446 DEBUG:boto:Params: {} 2024-02-01T10:48:04.446 DEBUG:boto:Token: None 2024-02-01T10:48:04.446 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:04 GMT /vclfcn-59/ 2024-02-01T10:48:04.446 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:c5xribnwSqgeG1HrL3Lia8JSSZU= 2024-02-01T10:48:04.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:c5xribnwSqgeG1HrL3Lia8JSSZU=', 'Content-Length': '0'} 2024-02-01T10:48:04.450 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000198441a68dd38973-0065bb76e4-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:04.492 DEBUG:boto:b'vclfcn-59obj41000falseobj42024-02-01T10:46:46.190Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:04.492 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:04.493 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:04.493 DEBUG:boto:path=/vclfcn-59/?prefix=obj4 2024-02-01T10:48:04.493 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj4 2024-02-01T10:48:04.493 DEBUG:boto:Method: GET 2024-02-01T10:48:04.493 DEBUG:boto:Path: /vclfcn-59/?prefix=obj4 2024-02-01T10:48:04.493 DEBUG:boto:Data: 2024-02-01T10:48:04.493 DEBUG:boto:Headers: {} 2024-02-01T10:48:04.493 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:04.493 DEBUG:boto:Port: 8000 2024-02-01T10:48:04.494 DEBUG:boto:Params: {} 2024-02-01T10:48:04.494 DEBUG:boto:Token: None 2024-02-01T10:48:04.494 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:04 GMT /vclfcn-59/ 2024-02-01T10:48:04.494 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:c5xribnwSqgeG1HrL3Lia8JSSZU= 2024-02-01T10:48:04.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:c5xribnwSqgeG1HrL3Lia8JSSZU=', 'Content-Length': '0'} 2024-02-01T10:48:04.499 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006de7d69ce7d8195c-0065bb76e4-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:04.499 DEBUG:boto:b'vclfcn-59obj41000falseobj42024-02-01T10:46:46.190Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:04.499 DEBUG:boto:path=/vclfcn-59/obj4 2024-02-01T10:48:04.499 DEBUG:boto:auth_path=/vclfcn-59/obj4 2024-02-01T10:48:04.499 DEBUG:boto:Method: HEAD 2024-02-01T10:48:04.500 DEBUG:boto:Path: /vclfcn-59/obj4 2024-02-01T10:48:04.500 DEBUG:boto:Data: 2024-02-01T10:48:04.500 DEBUG:boto:Headers: {} 2024-02-01T10:48:04.500 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:04.500 DEBUG:boto:Port: 8001 2024-02-01T10:48:04.500 DEBUG:boto:Params: {} 2024-02-01T10:48:04.500 DEBUG:boto:Token: None 2024-02-01T10:48:04.500 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:04 GMT /vclfcn-59/obj4 2024-02-01T10:48:04.500 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:MAkXpgfbniaNBt1XVcJNQ9EGllo= 2024-02-01T10:48:04.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:MAkXpgfbniaNBt1XVcJNQ9EGllo=', 'Content-Length': '0'} 2024-02-01T10:48:04.503 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000409048d5fefbd8ae-0065bb76e4-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:04.503 DEBUG:boto:path=/vclfcn-59/obj4 2024-02-01T10:48:04.504 DEBUG:boto:auth_path=/vclfcn-59/obj4 2024-02-01T10:48:04.504 DEBUG:boto:Method: HEAD 2024-02-01T10:48:04.504 DEBUG:boto:Path: /vclfcn-59/obj4 2024-02-01T10:48:04.504 DEBUG:boto:Data: 2024-02-01T10:48:04.504 DEBUG:boto:Headers: {} 2024-02-01T10:48:04.504 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:04.504 DEBUG:boto:Port: 8000 2024-02-01T10:48:04.504 DEBUG:boto:Params: {} 2024-02-01T10:48:04.504 DEBUG:boto:Token: None 2024-02-01T10:48:04.505 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:04 GMT /vclfcn-59/obj4 2024-02-01T10:48:04.505 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:MAkXpgfbniaNBt1XVcJNQ9EGllo= 2024-02-01T10:48:04.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:MAkXpgfbniaNBt1XVcJNQ9EGllo=', 'Content-Length': '0'} 2024-02-01T10:48:04.507 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b0f3fcd980ed2611-0065bb76e4-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:04.508 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:48:04.508 DEBUG:boto:path=/vclfcn-59/obj4 2024-02-01T10:48:04.508 DEBUG:boto:auth_path=/vclfcn-59/obj4 2024-02-01T10:48:04.508 DEBUG:boto:Method: GET 2024-02-01T10:48:04.508 DEBUG:boto:Path: /vclfcn-59/obj4 2024-02-01T10:48:04.508 DEBUG:boto:Data: 2024-02-01T10:48:04.508 DEBUG:boto:Headers: {} 2024-02-01T10:48:04.508 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:04.508 DEBUG:boto:Port: 8001 2024-02-01T10:48:04.509 DEBUG:boto:Params: {} 2024-02-01T10:48:04.509 DEBUG:boto:Token: None 2024-02-01T10:48:04.509 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:04 GMT /vclfcn-59/obj4 2024-02-01T10:48:04.509 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wW7ljTZEKujs0TWJZ5u4Gaef1SQ= 2024-02-01T10:48:04.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wW7ljTZEKujs0TWJZ5u4Gaef1SQ=', 'Content-Length': '0'} 2024-02-01T10:48:04.512 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e3d5ad19335b900f-0065bb76e4-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:04.552 DEBUG:boto:path=/vclfcn-59/obj4 2024-02-01T10:48:04.552 DEBUG:boto:auth_path=/vclfcn-59/obj4 2024-02-01T10:48:04.552 DEBUG:boto:Method: GET 2024-02-01T10:48:04.553 DEBUG:boto:Path: /vclfcn-59/obj4 2024-02-01T10:48:04.553 DEBUG:boto:Data: 2024-02-01T10:48:04.553 DEBUG:boto:Headers: {} 2024-02-01T10:48:04.553 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:04.553 DEBUG:boto:Port: 8000 2024-02-01T10:48:04.553 DEBUG:boto:Params: {} 2024-02-01T10:48:04.553 DEBUG:boto:Token: None 2024-02-01T10:48:04.553 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:04 GMT /vclfcn-59/obj4 2024-02-01T10:48:04.553 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wW7ljTZEKujs0TWJZ5u4Gaef1SQ= 2024-02-01T10:48:04.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wW7ljTZEKujs0TWJZ5u4Gaef1SQ=', 'Content-Length': '0'} 2024-02-01T10:48:04.556 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000597e9b83a82ee28d-0065bb76e4-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:04.557 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:04.557 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:04.557 DEBUG:boto:path=/vclfcn-59/?prefix=obj4 2024-02-01T10:48:04.557 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj4 2024-02-01T10:48:04.557 DEBUG:boto:Method: GET 2024-02-01T10:48:04.557 DEBUG:boto:Path: /vclfcn-59/?prefix=obj4 2024-02-01T10:48:04.557 DEBUG:boto:Data: 2024-02-01T10:48:04.557 DEBUG:boto:Headers: {} 2024-02-01T10:48:04.557 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:04.557 DEBUG:boto:Port: 8001 2024-02-01T10:48:04.557 DEBUG:boto:Params: {} 2024-02-01T10:48:04.558 DEBUG:boto:Token: None 2024-02-01T10:48:04.558 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:04 GMT /vclfcn-59/ 2024-02-01T10:48:04.558 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:c5xribnwSqgeG1HrL3Lia8JSSZU= 2024-02-01T10:48:04.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:c5xribnwSqgeG1HrL3Lia8JSSZU=', 'Content-Length': '0'} 2024-02-01T10:48:04.563 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000003a278d730cd3886-0065bb76e4-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:04.563 DEBUG:boto:b'vclfcn-59obj41000falseobj42024-02-01T10:46:46.190Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:04.564 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:04.564 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:04.564 DEBUG:boto:path=/vclfcn-59/?prefix=obj4 2024-02-01T10:48:04.564 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj4 2024-02-01T10:48:04.564 DEBUG:boto:Method: GET 2024-02-01T10:48:04.564 DEBUG:boto:Path: /vclfcn-59/?prefix=obj4 2024-02-01T10:48:04.564 DEBUG:boto:Data: 2024-02-01T10:48:04.565 DEBUG:boto:Headers: {} 2024-02-01T10:48:04.565 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:04.565 DEBUG:boto:Port: 8000 2024-02-01T10:48:04.565 DEBUG:boto:Params: {} 2024-02-01T10:48:04.565 DEBUG:boto:Token: None 2024-02-01T10:48:04.565 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:04 GMT /vclfcn-59/ 2024-02-01T10:48:04.565 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:c5xribnwSqgeG1HrL3Lia8JSSZU= 2024-02-01T10:48:04.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:c5xribnwSqgeG1HrL3Lia8JSSZU=', 'Content-Length': '0'} 2024-02-01T10:48:04.570 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d9af6d75b1d0767-0065bb76e4-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:04.570 DEBUG:boto:b'vclfcn-59obj41000falseobj42024-02-01T10:46:46.190Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:04.570 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-02-01T10:48:04.570 DEBUG:boto:path=/vclfcn-59/obj5 2024-02-01T10:48:04.571 DEBUG:boto:auth_path=/vclfcn-59/obj5 2024-02-01T10:48:04.571 DEBUG:boto:path=/vclfcn-59/obj5?versionId=null 2024-02-01T10:48:04.571 DEBUG:boto:auth_path=/vclfcn-59/obj5?versionId=null 2024-02-01T10:48:04.571 DEBUG:boto:Method: GET 2024-02-01T10:48:04.571 DEBUG:boto:Path: /vclfcn-59/obj5?versionId=null 2024-02-01T10:48:04.571 DEBUG:boto:Data: 2024-02-01T10:48:04.571 DEBUG:boto:Headers: {} 2024-02-01T10:48:04.572 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:04.572 DEBUG:boto:Port: 8001 2024-02-01T10:48:04.572 DEBUG:boto:Params: {} 2024-02-01T10:48:04.572 DEBUG:boto:Token: None 2024-02-01T10:48:04.572 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:04 GMT /vclfcn-59/obj5?versionId=null 2024-02-01T10:48:04.572 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:taZTUBfw7HJXamtMzYgWm3OmCgI= 2024-02-01T10:48:04.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:taZTUBfw7HJXamtMzYgWm3OmCgI=', 'Content-Length': '0'} 2024-02-01T10:48:04.575 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004790da4c4f954df8-0065bb76e4-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:04.616 DEBUG:boto:path=/vclfcn-59/obj5 2024-02-01T10:48:04.616 DEBUG:boto:auth_path=/vclfcn-59/obj5 2024-02-01T10:48:04.616 DEBUG:boto:path=/vclfcn-59/obj5?versionId=null 2024-02-01T10:48:04.616 DEBUG:boto:auth_path=/vclfcn-59/obj5?versionId=null 2024-02-01T10:48:04.617 DEBUG:boto:Method: GET 2024-02-01T10:48:04.617 DEBUG:boto:Path: /vclfcn-59/obj5?versionId=null 2024-02-01T10:48:04.617 DEBUG:boto:Data: 2024-02-01T10:48:04.617 DEBUG:boto:Headers: {} 2024-02-01T10:48:04.617 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:04.617 DEBUG:boto:Port: 8000 2024-02-01T10:48:04.617 DEBUG:boto:Params: {} 2024-02-01T10:48:04.617 DEBUG:boto:Token: None 2024-02-01T10:48:04.617 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:04 GMT /vclfcn-59/obj5?versionId=null 2024-02-01T10:48:04.617 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:taZTUBfw7HJXamtMzYgWm3OmCgI= 2024-02-01T10:48:04.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:taZTUBfw7HJXamtMzYgWm3OmCgI=', 'Content-Length': '0'} 2024-02-01T10:48:04.621 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002e29c96f3a28b9bb-0065bb76e4-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:04.621 DEBUG:boto:path=/vclfcn-59/obj5 2024-02-01T10:48:04.621 DEBUG:boto:auth_path=/vclfcn-59/obj5 2024-02-01T10:48:04.622 DEBUG:boto:path=/vclfcn-59/obj5?versionId=null 2024-02-01T10:48:04.622 DEBUG:boto:auth_path=/vclfcn-59/obj5?versionId=null 2024-02-01T10:48:04.622 DEBUG:boto:Method: HEAD 2024-02-01T10:48:04.622 DEBUG:boto:Path: /vclfcn-59/obj5?versionId=null 2024-02-01T10:48:04.622 DEBUG:boto:Data: 2024-02-01T10:48:04.622 DEBUG:boto:Headers: {} 2024-02-01T10:48:04.622 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:04.622 DEBUG:boto:Port: 8001 2024-02-01T10:48:04.622 DEBUG:boto:Params: {} 2024-02-01T10:48:04.622 DEBUG:boto:Token: None 2024-02-01T10:48:04.623 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:04 GMT /vclfcn-59/obj5?versionId=null 2024-02-01T10:48:04.623 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:R+E/4o8IkJ2nHz6bOEeCJIXsUSw= 2024-02-01T10:48:04.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:R+E/4o8IkJ2nHz6bOEeCJIXsUSw=', 'Content-Length': '0'} 2024-02-01T10:48:04.626 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000634cee7f9249dd74-0065bb76e4-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:04.626 DEBUG:boto:path=/vclfcn-59/obj5 2024-02-01T10:48:04.626 DEBUG:boto:auth_path=/vclfcn-59/obj5 2024-02-01T10:48:04.626 DEBUG:boto:path=/vclfcn-59/obj5?versionId=null 2024-02-01T10:48:04.626 DEBUG:boto:auth_path=/vclfcn-59/obj5?versionId=null 2024-02-01T10:48:04.626 DEBUG:boto:Method: HEAD 2024-02-01T10:48:04.626 DEBUG:boto:Path: /vclfcn-59/obj5?versionId=null 2024-02-01T10:48:04.626 DEBUG:boto:Data: 2024-02-01T10:48:04.627 DEBUG:boto:Headers: {} 2024-02-01T10:48:04.627 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:04.627 DEBUG:boto:Port: 8000 2024-02-01T10:48:04.627 DEBUG:boto:Params: {} 2024-02-01T10:48:04.627 DEBUG:boto:Token: None 2024-02-01T10:48:04.627 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:04 GMT /vclfcn-59/obj5?versionId=null 2024-02-01T10:48:04.627 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:R+E/4o8IkJ2nHz6bOEeCJIXsUSw= 2024-02-01T10:48:04.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:R+E/4o8IkJ2nHz6bOEeCJIXsUSw=', 'Content-Length': '0'} 2024-02-01T10:48:04.630 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000081440c5f30028098-0065bb76e4-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:04.630 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-02-01T10:48:04.630 DEBUG:boto:path=/vclfcn-59/obj5 2024-02-01T10:48:04.631 DEBUG:boto:auth_path=/vclfcn-59/obj5 2024-02-01T10:48:04.631 DEBUG:boto:path=/vclfcn-59/obj5?versionId=null 2024-02-01T10:48:04.631 DEBUG:boto:auth_path=/vclfcn-59/obj5?versionId=null 2024-02-01T10:48:04.631 DEBUG:boto:Method: GET 2024-02-01T10:48:04.631 DEBUG:boto:Path: /vclfcn-59/obj5?versionId=null 2024-02-01T10:48:04.631 DEBUG:boto:Data: 2024-02-01T10:48:04.631 DEBUG:boto:Headers: {} 2024-02-01T10:48:04.631 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:04.631 DEBUG:boto:Port: 8001 2024-02-01T10:48:04.631 DEBUG:boto:Params: {} 2024-02-01T10:48:04.631 DEBUG:boto:Token: None 2024-02-01T10:48:04.632 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:04 GMT /vclfcn-59/obj5?versionId=null 2024-02-01T10:48:04.632 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:taZTUBfw7HJXamtMzYgWm3OmCgI= 2024-02-01T10:48:04.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:taZTUBfw7HJXamtMzYgWm3OmCgI=', 'Content-Length': '0'} 2024-02-01T10:48:04.635 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bff57ed61ce13f98-0065bb76e4-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:04.680 DEBUG:boto:path=/vclfcn-59/obj5 2024-02-01T10:48:04.680 DEBUG:boto:auth_path=/vclfcn-59/obj5 2024-02-01T10:48:04.680 DEBUG:boto:path=/vclfcn-59/obj5?versionId=null 2024-02-01T10:48:04.681 DEBUG:boto:auth_path=/vclfcn-59/obj5?versionId=null 2024-02-01T10:48:04.681 DEBUG:boto:Method: GET 2024-02-01T10:48:04.681 DEBUG:boto:Path: /vclfcn-59/obj5?versionId=null 2024-02-01T10:48:04.681 DEBUG:boto:Data: 2024-02-01T10:48:04.681 DEBUG:boto:Headers: {} 2024-02-01T10:48:04.681 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:04.681 DEBUG:boto:Port: 8000 2024-02-01T10:48:04.681 DEBUG:boto:Params: {} 2024-02-01T10:48:04.681 DEBUG:boto:Token: None 2024-02-01T10:48:04.681 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:04 GMT /vclfcn-59/obj5?versionId=null 2024-02-01T10:48:04.682 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:taZTUBfw7HJXamtMzYgWm3OmCgI= 2024-02-01T10:48:04.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:taZTUBfw7HJXamtMzYgWm3OmCgI=', 'Content-Length': '0'} 2024-02-01T10:48:04.685 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000003ebe67ad924251-0065bb76e4-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:04.685 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:04.686 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:04.686 DEBUG:boto:path=/vclfcn-59/?prefix=obj5 2024-02-01T10:48:04.686 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj5 2024-02-01T10:48:04.686 DEBUG:boto:Method: GET 2024-02-01T10:48:04.686 DEBUG:boto:Path: /vclfcn-59/?prefix=obj5 2024-02-01T10:48:04.686 DEBUG:boto:Data: 2024-02-01T10:48:04.686 DEBUG:boto:Headers: {} 2024-02-01T10:48:04.686 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:04.686 DEBUG:boto:Port: 8001 2024-02-01T10:48:04.686 DEBUG:boto:Params: {} 2024-02-01T10:48:04.686 DEBUG:boto:Token: None 2024-02-01T10:48:04.687 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:04 GMT /vclfcn-59/ 2024-02-01T10:48:04.687 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:c5xribnwSqgeG1HrL3Lia8JSSZU= 2024-02-01T10:48:04.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:c5xribnwSqgeG1HrL3Lia8JSSZU=', 'Content-Length': '0'} 2024-02-01T10:48:04.691 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084eed3a5a359323c-0065bb76e4-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:04.732 DEBUG:boto:b'vclfcn-59obj51000falseobj52024-02-01T10:47:30.206Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:04.732 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:04.733 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:04.733 DEBUG:boto:path=/vclfcn-59/?prefix=obj5 2024-02-01T10:48:04.733 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj5 2024-02-01T10:48:04.733 DEBUG:boto:Method: GET 2024-02-01T10:48:04.733 DEBUG:boto:Path: /vclfcn-59/?prefix=obj5 2024-02-01T10:48:04.733 DEBUG:boto:Data: 2024-02-01T10:48:04.733 DEBUG:boto:Headers: {} 2024-02-01T10:48:04.733 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:04.733 DEBUG:boto:Port: 8000 2024-02-01T10:48:04.733 DEBUG:boto:Params: {} 2024-02-01T10:48:04.734 DEBUG:boto:Token: None 2024-02-01T10:48:04.734 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:04 GMT /vclfcn-59/ 2024-02-01T10:48:04.734 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:c5xribnwSqgeG1HrL3Lia8JSSZU= 2024-02-01T10:48:04.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:c5xribnwSqgeG1HrL3Lia8JSSZU=', 'Content-Length': '0'} 2024-02-01T10:48:04.738 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d067338a8d382bca-0065bb76e4-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:04.738 DEBUG:boto:b'vclfcn-59obj51000falseobj52024-02-01T10:47:30.206Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:04.739 DEBUG:boto:path=/vclfcn-59/obj5 2024-02-01T10:48:04.739 DEBUG:boto:auth_path=/vclfcn-59/obj5 2024-02-01T10:48:04.739 DEBUG:boto:Method: HEAD 2024-02-01T10:48:04.739 DEBUG:boto:Path: /vclfcn-59/obj5 2024-02-01T10:48:04.739 DEBUG:boto:Data: 2024-02-01T10:48:04.739 DEBUG:boto:Headers: {} 2024-02-01T10:48:04.739 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:04.739 DEBUG:boto:Port: 8001 2024-02-01T10:48:04.739 DEBUG:boto:Params: {} 2024-02-01T10:48:04.739 DEBUG:boto:Token: None 2024-02-01T10:48:04.740 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:04 GMT /vclfcn-59/obj5 2024-02-01T10:48:04.740 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:4+Swly9V+Hdc8sYa9TpX5ReVAEI= 2024-02-01T10:48:04.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:4+Swly9V+Hdc8sYa9TpX5ReVAEI=', 'Content-Length': '0'} 2024-02-01T10:48:04.743 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008433df4dc245880c-0065bb76e4-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:04.743 DEBUG:boto:path=/vclfcn-59/obj5 2024-02-01T10:48:04.743 DEBUG:boto:auth_path=/vclfcn-59/obj5 2024-02-01T10:48:04.743 DEBUG:boto:Method: HEAD 2024-02-01T10:48:04.743 DEBUG:boto:Path: /vclfcn-59/obj5 2024-02-01T10:48:04.743 DEBUG:boto:Data: 2024-02-01T10:48:04.743 DEBUG:boto:Headers: {} 2024-02-01T10:48:04.743 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:04.744 DEBUG:boto:Port: 8000 2024-02-01T10:48:04.744 DEBUG:boto:Params: {} 2024-02-01T10:48:04.744 DEBUG:boto:Token: None 2024-02-01T10:48:04.744 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:04 GMT /vclfcn-59/obj5 2024-02-01T10:48:04.744 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:4+Swly9V+Hdc8sYa9TpX5ReVAEI= 2024-02-01T10:48:04.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:4+Swly9V+Hdc8sYa9TpX5ReVAEI=', 'Content-Length': '0'} 2024-02-01T10:48:04.746 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000446b767944b176c0-0065bb76e4-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:04.747 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-02-01T10:48:04.747 DEBUG:boto:path=/vclfcn-59/obj5 2024-02-01T10:48:04.747 DEBUG:boto:auth_path=/vclfcn-59/obj5 2024-02-01T10:48:04.747 DEBUG:boto:Method: GET 2024-02-01T10:48:04.747 DEBUG:boto:Path: /vclfcn-59/obj5 2024-02-01T10:48:04.747 DEBUG:boto:Data: 2024-02-01T10:48:04.747 DEBUG:boto:Headers: {} 2024-02-01T10:48:04.747 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:04.747 DEBUG:boto:Port: 8001 2024-02-01T10:48:04.748 DEBUG:boto:Params: {} 2024-02-01T10:48:04.748 DEBUG:boto:Token: None 2024-02-01T10:48:04.748 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:04 GMT /vclfcn-59/obj5 2024-02-01T10:48:04.748 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:JguhGVwfyXP1OsJ5427/QgAnaD8= 2024-02-01T10:48:04.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:JguhGVwfyXP1OsJ5427/QgAnaD8=', 'Content-Length': '0'} 2024-02-01T10:48:04.750 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000006a6e0d6d14b09f9-0065bb76e4-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:04.792 DEBUG:boto:path=/vclfcn-59/obj5 2024-02-01T10:48:04.792 DEBUG:boto:auth_path=/vclfcn-59/obj5 2024-02-01T10:48:04.792 DEBUG:boto:Method: GET 2024-02-01T10:48:04.793 DEBUG:boto:Path: /vclfcn-59/obj5 2024-02-01T10:48:04.793 DEBUG:boto:Data: 2024-02-01T10:48:04.793 DEBUG:boto:Headers: {} 2024-02-01T10:48:04.793 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:04.793 DEBUG:boto:Port: 8000 2024-02-01T10:48:04.793 DEBUG:boto:Params: {} 2024-02-01T10:48:04.793 DEBUG:boto:Token: None 2024-02-01T10:48:04.793 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:04 GMT /vclfcn-59/obj5 2024-02-01T10:48:04.793 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:JguhGVwfyXP1OsJ5427/QgAnaD8= 2024-02-01T10:48:04.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:JguhGVwfyXP1OsJ5427/QgAnaD8=', 'Content-Length': '0'} 2024-02-01T10:48:04.796 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000024a64e26b210f384-0065bb76e4-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:04.796 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:04.796 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:04.797 DEBUG:boto:path=/vclfcn-59/?prefix=obj5 2024-02-01T10:48:04.797 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj5 2024-02-01T10:48:04.797 DEBUG:boto:Method: GET 2024-02-01T10:48:04.797 DEBUG:boto:Path: /vclfcn-59/?prefix=obj5 2024-02-01T10:48:04.797 DEBUG:boto:Data: 2024-02-01T10:48:04.797 DEBUG:boto:Headers: {} 2024-02-01T10:48:04.797 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:04.797 DEBUG:boto:Port: 8001 2024-02-01T10:48:04.797 DEBUG:boto:Params: {} 2024-02-01T10:48:04.797 DEBUG:boto:Token: None 2024-02-01T10:48:04.798 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:04 GMT /vclfcn-59/ 2024-02-01T10:48:04.798 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:c5xribnwSqgeG1HrL3Lia8JSSZU= 2024-02-01T10:48:04.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:c5xribnwSqgeG1HrL3Lia8JSSZU=', 'Content-Length': '0'} 2024-02-01T10:48:04.802 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b2f9b0d981af3024-0065bb76e4-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:04.802 DEBUG:boto:b'vclfcn-59obj51000falseobj52024-02-01T10:47:30.206Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:04.802 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:04.802 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:04.802 DEBUG:boto:path=/vclfcn-59/?prefix=obj5 2024-02-01T10:48:04.803 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj5 2024-02-01T10:48:04.803 DEBUG:boto:Method: GET 2024-02-01T10:48:04.803 DEBUG:boto:Path: /vclfcn-59/?prefix=obj5 2024-02-01T10:48:04.803 DEBUG:boto:Data: 2024-02-01T10:48:04.803 DEBUG:boto:Headers: {} 2024-02-01T10:48:04.803 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:04.803 DEBUG:boto:Port: 8000 2024-02-01T10:48:04.803 DEBUG:boto:Params: {} 2024-02-01T10:48:04.803 DEBUG:boto:Token: None 2024-02-01T10:48:04.803 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:04 GMT /vclfcn-59/ 2024-02-01T10:48:04.803 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:c5xribnwSqgeG1HrL3Lia8JSSZU= 2024-02-01T10:48:04.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:c5xribnwSqgeG1HrL3Lia8JSSZU=', 'Content-Length': '0'} 2024-02-01T10:48:04.808 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000165c356da5fed2bc-0065bb76e4-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:04.808 DEBUG:boto:b'vclfcn-59obj51000falseobj52024-02-01T10:47:30.206Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:04.809 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-02-01T10:48:04.809 DEBUG:boto:path=/vclfcn-59/obj6 2024-02-01T10:48:04.809 DEBUG:boto:auth_path=/vclfcn-59/obj6 2024-02-01T10:48:04.809 DEBUG:boto:path=/vclfcn-59/obj6?versionId=null 2024-02-01T10:48:04.809 DEBUG:boto:auth_path=/vclfcn-59/obj6?versionId=null 2024-02-01T10:48:04.809 DEBUG:boto:Method: GET 2024-02-01T10:48:04.809 DEBUG:boto:Path: /vclfcn-59/obj6?versionId=null 2024-02-01T10:48:04.809 DEBUG:boto:Data: 2024-02-01T10:48:04.810 DEBUG:boto:Headers: {} 2024-02-01T10:48:04.810 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:04.810 DEBUG:boto:Port: 8001 2024-02-01T10:48:04.810 DEBUG:boto:Params: {} 2024-02-01T10:48:04.810 DEBUG:boto:Token: None 2024-02-01T10:48:04.810 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:04 GMT /vclfcn-59/obj6?versionId=null 2024-02-01T10:48:04.810 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:JEW7DmfdBQmAdP9qGfQZs9V+hEE= 2024-02-01T10:48:04.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:JEW7DmfdBQmAdP9qGfQZs9V+hEE=', 'Content-Length': '0'} 2024-02-01T10:48:04.813 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000006d9534bb071feff-0065bb76e4-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:04.856 DEBUG:boto:path=/vclfcn-59/obj6 2024-02-01T10:48:04.856 DEBUG:boto:auth_path=/vclfcn-59/obj6 2024-02-01T10:48:04.856 DEBUG:boto:path=/vclfcn-59/obj6?versionId=null 2024-02-01T10:48:04.856 DEBUG:boto:auth_path=/vclfcn-59/obj6?versionId=null 2024-02-01T10:48:04.857 DEBUG:boto:Method: GET 2024-02-01T10:48:04.857 DEBUG:boto:Path: /vclfcn-59/obj6?versionId=null 2024-02-01T10:48:04.857 DEBUG:boto:Data: 2024-02-01T10:48:04.857 DEBUG:boto:Headers: {} 2024-02-01T10:48:04.857 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:04.857 DEBUG:boto:Port: 8000 2024-02-01T10:48:04.857 DEBUG:boto:Params: {} 2024-02-01T10:48:04.857 DEBUG:boto:Token: None 2024-02-01T10:48:04.857 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:04 GMT /vclfcn-59/obj6?versionId=null 2024-02-01T10:48:04.857 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:JEW7DmfdBQmAdP9qGfQZs9V+hEE= 2024-02-01T10:48:04.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:JEW7DmfdBQmAdP9qGfQZs9V+hEE=', 'Content-Length': '0'} 2024-02-01T10:48:04.860 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000615a8aecf944d9f6-0065bb76e4-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:04.861 DEBUG:boto:path=/vclfcn-59/obj6 2024-02-01T10:48:04.861 DEBUG:boto:auth_path=/vclfcn-59/obj6 2024-02-01T10:48:04.861 DEBUG:boto:path=/vclfcn-59/obj6?versionId=null 2024-02-01T10:48:04.861 DEBUG:boto:auth_path=/vclfcn-59/obj6?versionId=null 2024-02-01T10:48:04.861 DEBUG:boto:Method: HEAD 2024-02-01T10:48:04.861 DEBUG:boto:Path: /vclfcn-59/obj6?versionId=null 2024-02-01T10:48:04.861 DEBUG:boto:Data: 2024-02-01T10:48:04.862 DEBUG:boto:Headers: {} 2024-02-01T10:48:04.862 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:04.862 DEBUG:boto:Port: 8001 2024-02-01T10:48:04.862 DEBUG:boto:Params: {} 2024-02-01T10:48:04.862 DEBUG:boto:Token: None 2024-02-01T10:48:04.862 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:04 GMT /vclfcn-59/obj6?versionId=null 2024-02-01T10:48:04.863 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FK4Y0gfkSL4+4QEGFyJVtSAKQxc= 2024-02-01T10:48:04.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FK4Y0gfkSL4+4QEGFyJVtSAKQxc=', 'Content-Length': '0'} 2024-02-01T10:48:04.865 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003a588203281e1ea8-0065bb76e4-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:04.865 DEBUG:boto:path=/vclfcn-59/obj6 2024-02-01T10:48:04.865 DEBUG:boto:auth_path=/vclfcn-59/obj6 2024-02-01T10:48:04.866 DEBUG:boto:path=/vclfcn-59/obj6?versionId=null 2024-02-01T10:48:04.866 DEBUG:boto:auth_path=/vclfcn-59/obj6?versionId=null 2024-02-01T10:48:04.866 DEBUG:boto:Method: HEAD 2024-02-01T10:48:04.866 DEBUG:boto:Path: /vclfcn-59/obj6?versionId=null 2024-02-01T10:48:04.866 DEBUG:boto:Data: 2024-02-01T10:48:04.866 DEBUG:boto:Headers: {} 2024-02-01T10:48:04.866 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:04.866 DEBUG:boto:Port: 8000 2024-02-01T10:48:04.866 DEBUG:boto:Params: {} 2024-02-01T10:48:04.866 DEBUG:boto:Token: None 2024-02-01T10:48:04.866 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:04 GMT /vclfcn-59/obj6?versionId=null 2024-02-01T10:48:04.867 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FK4Y0gfkSL4+4QEGFyJVtSAKQxc= 2024-02-01T10:48:04.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FK4Y0gfkSL4+4QEGFyJVtSAKQxc=', 'Content-Length': '0'} 2024-02-01T10:48:04.869 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f719d2d2245c77f2-0065bb76e4-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:04.869 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-02-01T10:48:04.870 DEBUG:boto:path=/vclfcn-59/obj6 2024-02-01T10:48:04.870 DEBUG:boto:auth_path=/vclfcn-59/obj6 2024-02-01T10:48:04.870 DEBUG:boto:path=/vclfcn-59/obj6?versionId=null 2024-02-01T10:48:04.870 DEBUG:boto:auth_path=/vclfcn-59/obj6?versionId=null 2024-02-01T10:48:04.870 DEBUG:boto:Method: GET 2024-02-01T10:48:04.870 DEBUG:boto:Path: /vclfcn-59/obj6?versionId=null 2024-02-01T10:48:04.870 DEBUG:boto:Data: 2024-02-01T10:48:04.870 DEBUG:boto:Headers: {} 2024-02-01T10:48:04.870 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:04.870 DEBUG:boto:Port: 8001 2024-02-01T10:48:04.870 DEBUG:boto:Params: {} 2024-02-01T10:48:04.870 DEBUG:boto:Token: None 2024-02-01T10:48:04.871 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:04 GMT /vclfcn-59/obj6?versionId=null 2024-02-01T10:48:04.871 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:JEW7DmfdBQmAdP9qGfQZs9V+hEE= 2024-02-01T10:48:04.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:JEW7DmfdBQmAdP9qGfQZs9V+hEE=', 'Content-Length': '0'} 2024-02-01T10:48:04.873 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000947c92d92b29e193-0065bb76e4-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:04.916 DEBUG:boto:path=/vclfcn-59/obj6 2024-02-01T10:48:04.916 DEBUG:boto:auth_path=/vclfcn-59/obj6 2024-02-01T10:48:04.916 DEBUG:boto:path=/vclfcn-59/obj6?versionId=null 2024-02-01T10:48:04.916 DEBUG:boto:auth_path=/vclfcn-59/obj6?versionId=null 2024-02-01T10:48:04.916 DEBUG:boto:Method: GET 2024-02-01T10:48:04.917 DEBUG:boto:Path: /vclfcn-59/obj6?versionId=null 2024-02-01T10:48:04.917 DEBUG:boto:Data: 2024-02-01T10:48:04.917 DEBUG:boto:Headers: {} 2024-02-01T10:48:04.917 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:04.917 DEBUG:boto:Port: 8000 2024-02-01T10:48:04.917 DEBUG:boto:Params: {} 2024-02-01T10:48:04.917 DEBUG:boto:Token: None 2024-02-01T10:48:04.917 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:04 GMT /vclfcn-59/obj6?versionId=null 2024-02-01T10:48:04.917 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:JEW7DmfdBQmAdP9qGfQZs9V+hEE= 2024-02-01T10:48:04.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:JEW7DmfdBQmAdP9qGfQZs9V+hEE=', 'Content-Length': '0'} 2024-02-01T10:48:04.920 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001a797c37d1aeecfe-0065bb76e4-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:04.920 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:04.921 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:04.921 DEBUG:boto:path=/vclfcn-59/?prefix=obj6 2024-02-01T10:48:04.921 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj6 2024-02-01T10:48:04.921 DEBUG:boto:Method: GET 2024-02-01T10:48:04.921 DEBUG:boto:Path: /vclfcn-59/?prefix=obj6 2024-02-01T10:48:04.921 DEBUG:boto:Data: 2024-02-01T10:48:04.921 DEBUG:boto:Headers: {} 2024-02-01T10:48:04.921 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:04.921 DEBUG:boto:Port: 8001 2024-02-01T10:48:04.921 DEBUG:boto:Params: {} 2024-02-01T10:48:04.921 DEBUG:boto:Token: None 2024-02-01T10:48:04.922 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:04 GMT /vclfcn-59/ 2024-02-01T10:48:04.922 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:c5xribnwSqgeG1HrL3Lia8JSSZU= 2024-02-01T10:48:04.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:c5xribnwSqgeG1HrL3Lia8JSSZU=', 'Content-Length': '0'} 2024-02-01T10:48:04.926 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000062adedc006f1a6f6-0065bb76e4-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:04.968 DEBUG:boto:b'vclfcn-59obj61000falseobj62024-02-01T10:47:30.217Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:04.969 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:04.969 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:04.969 DEBUG:boto:path=/vclfcn-59/?prefix=obj6 2024-02-01T10:48:04.969 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj6 2024-02-01T10:48:04.969 DEBUG:boto:Method: GET 2024-02-01T10:48:04.969 DEBUG:boto:Path: /vclfcn-59/?prefix=obj6 2024-02-01T10:48:04.969 DEBUG:boto:Data: 2024-02-01T10:48:04.969 DEBUG:boto:Headers: {} 2024-02-01T10:48:04.969 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:04.969 DEBUG:boto:Port: 8000 2024-02-01T10:48:04.969 DEBUG:boto:Params: {} 2024-02-01T10:48:04.970 DEBUG:boto:Token: None 2024-02-01T10:48:04.970 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:04 GMT /vclfcn-59/ 2024-02-01T10:48:04.970 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:c5xribnwSqgeG1HrL3Lia8JSSZU= 2024-02-01T10:48:04.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:c5xribnwSqgeG1HrL3Lia8JSSZU=', 'Content-Length': '0'} 2024-02-01T10:48:04.974 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc02c9bbc60c5262-0065bb76e4-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:04.975 DEBUG:boto:b'vclfcn-59obj61000falseobj62024-02-01T10:47:30.217Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:04.975 DEBUG:boto:path=/vclfcn-59/obj6 2024-02-01T10:48:04.975 DEBUG:boto:auth_path=/vclfcn-59/obj6 2024-02-01T10:48:04.975 DEBUG:boto:Method: HEAD 2024-02-01T10:48:04.975 DEBUG:boto:Path: /vclfcn-59/obj6 2024-02-01T10:48:04.975 DEBUG:boto:Data: 2024-02-01T10:48:04.976 DEBUG:boto:Headers: {} 2024-02-01T10:48:04.976 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:04.976 DEBUG:boto:Port: 8001 2024-02-01T10:48:04.976 DEBUG:boto:Params: {} 2024-02-01T10:48:04.976 DEBUG:boto:Token: None 2024-02-01T10:48:04.976 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:04 GMT /vclfcn-59/obj6 2024-02-01T10:48:04.976 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1x2o3jn7cCTM/RjUYewI8BRr03U= 2024-02-01T10:48:04.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1x2o3jn7cCTM/RjUYewI8BRr03U=', 'Content-Length': '0'} 2024-02-01T10:48:04.979 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000050e17c4bc3ca2b87-0065bb76e4-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:04.979 DEBUG:boto:path=/vclfcn-59/obj6 2024-02-01T10:48:04.979 DEBUG:boto:auth_path=/vclfcn-59/obj6 2024-02-01T10:48:04.979 DEBUG:boto:Method: HEAD 2024-02-01T10:48:04.979 DEBUG:boto:Path: /vclfcn-59/obj6 2024-02-01T10:48:04.980 DEBUG:boto:Data: 2024-02-01T10:48:04.980 DEBUG:boto:Headers: {} 2024-02-01T10:48:04.980 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:04.980 DEBUG:boto:Port: 8000 2024-02-01T10:48:04.980 DEBUG:boto:Params: {} 2024-02-01T10:48:04.980 DEBUG:boto:Token: None 2024-02-01T10:48:04.980 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:04 GMT /vclfcn-59/obj6 2024-02-01T10:48:04.980 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1x2o3jn7cCTM/RjUYewI8BRr03U= 2024-02-01T10:48:04.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1x2o3jn7cCTM/RjUYewI8BRr03U=', 'Content-Length': '0'} 2024-02-01T10:48:04.983 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000018d79be0031709e1-0065bb76e4-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:04.983 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-02-01T10:48:04.983 DEBUG:boto:path=/vclfcn-59/obj6 2024-02-01T10:48:04.983 DEBUG:boto:auth_path=/vclfcn-59/obj6 2024-02-01T10:48:04.983 DEBUG:boto:Method: GET 2024-02-01T10:48:04.983 DEBUG:boto:Path: /vclfcn-59/obj6 2024-02-01T10:48:04.984 DEBUG:boto:Data: 2024-02-01T10:48:04.984 DEBUG:boto:Headers: {} 2024-02-01T10:48:04.984 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:04.984 DEBUG:boto:Port: 8001 2024-02-01T10:48:04.984 DEBUG:boto:Params: {} 2024-02-01T10:48:04.984 DEBUG:boto:Token: None 2024-02-01T10:48:04.984 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:04 GMT /vclfcn-59/obj6 2024-02-01T10:48:04.984 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ou/To56+/DFNS5UdZagnQUunvsg= 2024-02-01T10:48:04.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ou/To56+/DFNS5UdZagnQUunvsg=', 'Content-Length': '0'} 2024-02-01T10:48:04.987 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003b20f450f48cd25c-0065bb76e4-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:05.032 DEBUG:boto:path=/vclfcn-59/obj6 2024-02-01T10:48:05.032 DEBUG:boto:auth_path=/vclfcn-59/obj6 2024-02-01T10:48:05.032 DEBUG:boto:Method: GET 2024-02-01T10:48:05.032 DEBUG:boto:Path: /vclfcn-59/obj6 2024-02-01T10:48:05.033 DEBUG:boto:Data: 2024-02-01T10:48:05.033 DEBUG:boto:Headers: {} 2024-02-01T10:48:05.033 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:05.033 DEBUG:boto:Port: 8000 2024-02-01T10:48:05.033 DEBUG:boto:Params: {} 2024-02-01T10:48:05.033 DEBUG:boto:Token: None 2024-02-01T10:48:05.033 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:05 GMT /vclfcn-59/obj6 2024-02-01T10:48:05.033 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:kVYWVqPmVkqCLIat0fmIeBg5Ozk= 2024-02-01T10:48:05.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:kVYWVqPmVkqCLIat0fmIeBg5Ozk=', 'Content-Length': '0'} 2024-02-01T10:48:05.036 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000011febf460f847057-0065bb76e5-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:05.036 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:05.037 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:05.037 DEBUG:boto:path=/vclfcn-59/?prefix=obj6 2024-02-01T10:48:05.037 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj6 2024-02-01T10:48:05.037 DEBUG:boto:Method: GET 2024-02-01T10:48:05.037 DEBUG:boto:Path: /vclfcn-59/?prefix=obj6 2024-02-01T10:48:05.037 DEBUG:boto:Data: 2024-02-01T10:48:05.037 DEBUG:boto:Headers: {} 2024-02-01T10:48:05.037 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:05.037 DEBUG:boto:Port: 8001 2024-02-01T10:48:05.037 DEBUG:boto:Params: {} 2024-02-01T10:48:05.037 DEBUG:boto:Token: None 2024-02-01T10:48:05.037 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:05 GMT /vclfcn-59/ 2024-02-01T10:48:05.038 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Q9LrdrhS2DRPqHDne9rbiEMNiH8= 2024-02-01T10:48:05.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Q9LrdrhS2DRPqHDne9rbiEMNiH8=', 'Content-Length': '0'} 2024-02-01T10:48:05.042 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089b012fde69fd032-0065bb76e5-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:05.042 DEBUG:boto:b'vclfcn-59obj61000falseobj62024-02-01T10:47:30.217Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:05.043 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:05.043 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:05.043 DEBUG:boto:path=/vclfcn-59/?prefix=obj6 2024-02-01T10:48:05.043 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj6 2024-02-01T10:48:05.043 DEBUG:boto:Method: GET 2024-02-01T10:48:05.043 DEBUG:boto:Path: /vclfcn-59/?prefix=obj6 2024-02-01T10:48:05.043 DEBUG:boto:Data: 2024-02-01T10:48:05.043 DEBUG:boto:Headers: {} 2024-02-01T10:48:05.043 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:05.044 DEBUG:boto:Port: 8000 2024-02-01T10:48:05.044 DEBUG:boto:Params: {} 2024-02-01T10:48:05.044 DEBUG:boto:Token: None 2024-02-01T10:48:05.044 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:05 GMT /vclfcn-59/ 2024-02-01T10:48:05.044 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Q9LrdrhS2DRPqHDne9rbiEMNiH8= 2024-02-01T10:48:05.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Q9LrdrhS2DRPqHDne9rbiEMNiH8=', 'Content-Length': '0'} 2024-02-01T10:48:05.048 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006045253a360b1bc1-0065bb76e5-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:05.048 DEBUG:boto:b'vclfcn-59obj61000falseobj62024-02-01T10:47:30.217Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:05.049 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-02-01T10:48:05.049 DEBUG:boto:path=/vclfcn-59/obj7 2024-02-01T10:48:05.049 DEBUG:boto:auth_path=/vclfcn-59/obj7 2024-02-01T10:48:05.049 DEBUG:boto:path=/vclfcn-59/obj7?versionId=null 2024-02-01T10:48:05.049 DEBUG:boto:auth_path=/vclfcn-59/obj7?versionId=null 2024-02-01T10:48:05.049 DEBUG:boto:Method: GET 2024-02-01T10:48:05.049 DEBUG:boto:Path: /vclfcn-59/obj7?versionId=null 2024-02-01T10:48:05.049 DEBUG:boto:Data: 2024-02-01T10:48:05.049 DEBUG:boto:Headers: {} 2024-02-01T10:48:05.050 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:05.050 DEBUG:boto:Port: 8001 2024-02-01T10:48:05.050 DEBUG:boto:Params: {} 2024-02-01T10:48:05.050 DEBUG:boto:Token: None 2024-02-01T10:48:05.050 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:05 GMT /vclfcn-59/obj7?versionId=null 2024-02-01T10:48:05.050 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:uB/Fs5WSqvZxLibWPxeG7E0r1G8= 2024-02-01T10:48:05.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:uB/Fs5WSqvZxLibWPxeG7E0r1G8=', 'Content-Length': '0'} 2024-02-01T10:48:05.053 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000055fae50d3dbd7d31-0065bb76e5-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:05.100 DEBUG:boto:path=/vclfcn-59/obj7 2024-02-01T10:48:05.100 DEBUG:boto:auth_path=/vclfcn-59/obj7 2024-02-01T10:48:05.100 DEBUG:boto:path=/vclfcn-59/obj7?versionId=null 2024-02-01T10:48:05.100 DEBUG:boto:auth_path=/vclfcn-59/obj7?versionId=null 2024-02-01T10:48:05.101 DEBUG:boto:Method: GET 2024-02-01T10:48:05.101 DEBUG:boto:Path: /vclfcn-59/obj7?versionId=null 2024-02-01T10:48:05.101 DEBUG:boto:Data: 2024-02-01T10:48:05.101 DEBUG:boto:Headers: {} 2024-02-01T10:48:05.101 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:05.101 DEBUG:boto:Port: 8000 2024-02-01T10:48:05.101 DEBUG:boto:Params: {} 2024-02-01T10:48:05.101 DEBUG:boto:Token: None 2024-02-01T10:48:05.101 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:05 GMT /vclfcn-59/obj7?versionId=null 2024-02-01T10:48:05.102 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:uB/Fs5WSqvZxLibWPxeG7E0r1G8= 2024-02-01T10:48:05.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:uB/Fs5WSqvZxLibWPxeG7E0r1G8=', 'Content-Length': '0'} 2024-02-01T10:48:05.104 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000087d17ac6746367bc-0065bb76e5-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:05.105 DEBUG:boto:path=/vclfcn-59/obj7 2024-02-01T10:48:05.105 DEBUG:boto:auth_path=/vclfcn-59/obj7 2024-02-01T10:48:05.105 DEBUG:boto:path=/vclfcn-59/obj7?versionId=null 2024-02-01T10:48:05.105 DEBUG:boto:auth_path=/vclfcn-59/obj7?versionId=null 2024-02-01T10:48:05.105 DEBUG:boto:Method: HEAD 2024-02-01T10:48:05.105 DEBUG:boto:Path: /vclfcn-59/obj7?versionId=null 2024-02-01T10:48:05.105 DEBUG:boto:Data: 2024-02-01T10:48:05.105 DEBUG:boto:Headers: {} 2024-02-01T10:48:05.106 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:05.106 DEBUG:boto:Port: 8001 2024-02-01T10:48:05.106 DEBUG:boto:Params: {} 2024-02-01T10:48:05.106 DEBUG:boto:Token: None 2024-02-01T10:48:05.106 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:05 GMT /vclfcn-59/obj7?versionId=null 2024-02-01T10:48:05.106 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:3fj66YN8UlN/40pAvPNrZc3AE5E= 2024-02-01T10:48:05.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:3fj66YN8UlN/40pAvPNrZc3AE5E=', 'Content-Length': '0'} 2024-02-01T10:48:05.109 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006f4e4893bbc62f8c-0065bb76e5-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:05.109 DEBUG:boto:path=/vclfcn-59/obj7 2024-02-01T10:48:05.109 DEBUG:boto:auth_path=/vclfcn-59/obj7 2024-02-01T10:48:05.109 DEBUG:boto:path=/vclfcn-59/obj7?versionId=null 2024-02-01T10:48:05.109 DEBUG:boto:auth_path=/vclfcn-59/obj7?versionId=null 2024-02-01T10:48:05.109 DEBUG:boto:Method: HEAD 2024-02-01T10:48:05.109 DEBUG:boto:Path: /vclfcn-59/obj7?versionId=null 2024-02-01T10:48:05.109 DEBUG:boto:Data: 2024-02-01T10:48:05.110 DEBUG:boto:Headers: {} 2024-02-01T10:48:05.110 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:05.110 DEBUG:boto:Port: 8000 2024-02-01T10:48:05.110 DEBUG:boto:Params: {} 2024-02-01T10:48:05.110 DEBUG:boto:Token: None 2024-02-01T10:48:05.110 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:05 GMT /vclfcn-59/obj7?versionId=null 2024-02-01T10:48:05.110 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:3fj66YN8UlN/40pAvPNrZc3AE5E= 2024-02-01T10:48:05.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:3fj66YN8UlN/40pAvPNrZc3AE5E=', 'Content-Length': '0'} 2024-02-01T10:48:05.113 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000562b8219ac284880-0065bb76e5-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:05.113 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-02-01T10:48:05.113 DEBUG:boto:path=/vclfcn-59/obj7 2024-02-01T10:48:05.113 DEBUG:boto:auth_path=/vclfcn-59/obj7 2024-02-01T10:48:05.113 DEBUG:boto:path=/vclfcn-59/obj7?versionId=null 2024-02-01T10:48:05.113 DEBUG:boto:auth_path=/vclfcn-59/obj7?versionId=null 2024-02-01T10:48:05.113 DEBUG:boto:Method: GET 2024-02-01T10:48:05.113 DEBUG:boto:Path: /vclfcn-59/obj7?versionId=null 2024-02-01T10:48:05.113 DEBUG:boto:Data: 2024-02-01T10:48:05.114 DEBUG:boto:Headers: {} 2024-02-01T10:48:05.114 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:05.114 DEBUG:boto:Port: 8001 2024-02-01T10:48:05.114 DEBUG:boto:Params: {} 2024-02-01T10:48:05.114 DEBUG:boto:Token: None 2024-02-01T10:48:05.114 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:05 GMT /vclfcn-59/obj7?versionId=null 2024-02-01T10:48:05.114 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:uB/Fs5WSqvZxLibWPxeG7E0r1G8= 2024-02-01T10:48:05.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:uB/Fs5WSqvZxLibWPxeG7E0r1G8=', 'Content-Length': '0'} 2024-02-01T10:48:05.117 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ab7e7a68cb3598cd-0065bb76e5-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:05.160 DEBUG:boto:path=/vclfcn-59/obj7 2024-02-01T10:48:05.160 DEBUG:boto:auth_path=/vclfcn-59/obj7 2024-02-01T10:48:05.160 DEBUG:boto:path=/vclfcn-59/obj7?versionId=null 2024-02-01T10:48:05.160 DEBUG:boto:auth_path=/vclfcn-59/obj7?versionId=null 2024-02-01T10:48:05.161 DEBUG:boto:Method: GET 2024-02-01T10:48:05.161 DEBUG:boto:Path: /vclfcn-59/obj7?versionId=null 2024-02-01T10:48:05.161 DEBUG:boto:Data: 2024-02-01T10:48:05.161 DEBUG:boto:Headers: {} 2024-02-01T10:48:05.161 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:05.161 DEBUG:boto:Port: 8000 2024-02-01T10:48:05.161 DEBUG:boto:Params: {} 2024-02-01T10:48:05.161 DEBUG:boto:Token: None 2024-02-01T10:48:05.161 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:05 GMT /vclfcn-59/obj7?versionId=null 2024-02-01T10:48:05.162 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:uB/Fs5WSqvZxLibWPxeG7E0r1G8= 2024-02-01T10:48:05.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:uB/Fs5WSqvZxLibWPxeG7E0r1G8=', 'Content-Length': '0'} 2024-02-01T10:48:05.164 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004807f91b66c864ee-0065bb76e5-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:05.165 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:05.165 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:05.165 DEBUG:boto:path=/vclfcn-59/?prefix=obj7 2024-02-01T10:48:05.165 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj7 2024-02-01T10:48:05.165 DEBUG:boto:Method: GET 2024-02-01T10:48:05.165 DEBUG:boto:Path: /vclfcn-59/?prefix=obj7 2024-02-01T10:48:05.165 DEBUG:boto:Data: 2024-02-01T10:48:05.165 DEBUG:boto:Headers: {} 2024-02-01T10:48:05.165 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:05.166 DEBUG:boto:Port: 8001 2024-02-01T10:48:05.166 DEBUG:boto:Params: {} 2024-02-01T10:48:05.166 DEBUG:boto:Token: None 2024-02-01T10:48:05.166 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:05 GMT /vclfcn-59/ 2024-02-01T10:48:05.166 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Q9LrdrhS2DRPqHDne9rbiEMNiH8= 2024-02-01T10:48:05.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Q9LrdrhS2DRPqHDne9rbiEMNiH8=', 'Content-Length': '0'} 2024-02-01T10:48:05.170 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000beffc9db9a3ebeb3-0065bb76e5-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:05.212 DEBUG:boto:b'vclfcn-59obj71000falseobj72024-02-01T10:47:30.227Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:05.213 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:05.213 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:05.213 DEBUG:boto:path=/vclfcn-59/?prefix=obj7 2024-02-01T10:48:05.213 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj7 2024-02-01T10:48:05.213 DEBUG:boto:Method: GET 2024-02-01T10:48:05.213 DEBUG:boto:Path: /vclfcn-59/?prefix=obj7 2024-02-01T10:48:05.213 DEBUG:boto:Data: 2024-02-01T10:48:05.213 DEBUG:boto:Headers: {} 2024-02-01T10:48:05.213 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:05.213 DEBUG:boto:Port: 8000 2024-02-01T10:48:05.213 DEBUG:boto:Params: {} 2024-02-01T10:48:05.214 DEBUG:boto:Token: None 2024-02-01T10:48:05.214 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:05 GMT /vclfcn-59/ 2024-02-01T10:48:05.214 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Q9LrdrhS2DRPqHDne9rbiEMNiH8= 2024-02-01T10:48:05.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Q9LrdrhS2DRPqHDne9rbiEMNiH8=', 'Content-Length': '0'} 2024-02-01T10:48:05.219 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009fe2f76edc3bc2a1-0065bb76e5-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:05.219 DEBUG:boto:b'vclfcn-59obj71000falseobj72024-02-01T10:47:30.227Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:05.219 DEBUG:boto:path=/vclfcn-59/obj7 2024-02-01T10:48:05.219 DEBUG:boto:auth_path=/vclfcn-59/obj7 2024-02-01T10:48:05.219 DEBUG:boto:Method: HEAD 2024-02-01T10:48:05.219 DEBUG:boto:Path: /vclfcn-59/obj7 2024-02-01T10:48:05.219 DEBUG:boto:Data: 2024-02-01T10:48:05.219 DEBUG:boto:Headers: {} 2024-02-01T10:48:05.220 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:05.220 DEBUG:boto:Port: 8001 2024-02-01T10:48:05.220 DEBUG:boto:Params: {} 2024-02-01T10:48:05.220 DEBUG:boto:Token: None 2024-02-01T10:48:05.220 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:05 GMT /vclfcn-59/obj7 2024-02-01T10:48:05.220 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FLSG3reR2kcQcKDDcsp5RwzUbLQ= 2024-02-01T10:48:05.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FLSG3reR2kcQcKDDcsp5RwzUbLQ=', 'Content-Length': '0'} 2024-02-01T10:48:05.223 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f1ec2129c33dde6d-0065bb76e5-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:05.223 DEBUG:boto:path=/vclfcn-59/obj7 2024-02-01T10:48:05.223 DEBUG:boto:auth_path=/vclfcn-59/obj7 2024-02-01T10:48:05.223 DEBUG:boto:Method: HEAD 2024-02-01T10:48:05.223 DEBUG:boto:Path: /vclfcn-59/obj7 2024-02-01T10:48:05.224 DEBUG:boto:Data: 2024-02-01T10:48:05.224 DEBUG:boto:Headers: {} 2024-02-01T10:48:05.224 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:05.224 DEBUG:boto:Port: 8000 2024-02-01T10:48:05.224 DEBUG:boto:Params: {} 2024-02-01T10:48:05.224 DEBUG:boto:Token: None 2024-02-01T10:48:05.224 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:05 GMT /vclfcn-59/obj7 2024-02-01T10:48:05.224 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FLSG3reR2kcQcKDDcsp5RwzUbLQ= 2024-02-01T10:48:05.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FLSG3reR2kcQcKDDcsp5RwzUbLQ=', 'Content-Length': '0'} 2024-02-01T10:48:05.227 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bd9420a11986e2d5-0065bb76e5-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:05.227 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-02-01T10:48:05.227 DEBUG:boto:path=/vclfcn-59/obj7 2024-02-01T10:48:05.227 DEBUG:boto:auth_path=/vclfcn-59/obj7 2024-02-01T10:48:05.228 DEBUG:boto:Method: GET 2024-02-01T10:48:05.228 DEBUG:boto:Path: /vclfcn-59/obj7 2024-02-01T10:48:05.228 DEBUG:boto:Data: 2024-02-01T10:48:05.228 DEBUG:boto:Headers: {} 2024-02-01T10:48:05.228 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:05.228 DEBUG:boto:Port: 8001 2024-02-01T10:48:05.228 DEBUG:boto:Params: {} 2024-02-01T10:48:05.228 DEBUG:boto:Token: None 2024-02-01T10:48:05.228 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:05 GMT /vclfcn-59/obj7 2024-02-01T10:48:05.228 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:n1Pmp1d/RMZQi+jD1CS3rUyN/Ig= 2024-02-01T10:48:05.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:n1Pmp1d/RMZQi+jD1CS3rUyN/Ig=', 'Content-Length': '0'} 2024-02-01T10:48:05.231 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000faac567ea617d0f9-0065bb76e5-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:05.272 DEBUG:boto:path=/vclfcn-59/obj7 2024-02-01T10:48:05.272 DEBUG:boto:auth_path=/vclfcn-59/obj7 2024-02-01T10:48:05.273 DEBUG:boto:Method: GET 2024-02-01T10:48:05.273 DEBUG:boto:Path: /vclfcn-59/obj7 2024-02-01T10:48:05.273 DEBUG:boto:Data: 2024-02-01T10:48:05.273 DEBUG:boto:Headers: {} 2024-02-01T10:48:05.273 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:05.273 DEBUG:boto:Port: 8000 2024-02-01T10:48:05.273 DEBUG:boto:Params: {} 2024-02-01T10:48:05.273 DEBUG:boto:Token: None 2024-02-01T10:48:05.273 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:05 GMT /vclfcn-59/obj7 2024-02-01T10:48:05.274 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:n1Pmp1d/RMZQi+jD1CS3rUyN/Ig= 2024-02-01T10:48:05.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:n1Pmp1d/RMZQi+jD1CS3rUyN/Ig=', 'Content-Length': '0'} 2024-02-01T10:48:05.276 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000bca0b53cc334e60-0065bb76e5-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:05.277 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:05.277 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:05.277 DEBUG:boto:path=/vclfcn-59/?prefix=obj7 2024-02-01T10:48:05.277 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj7 2024-02-01T10:48:05.277 DEBUG:boto:Method: GET 2024-02-01T10:48:05.277 DEBUG:boto:Path: /vclfcn-59/?prefix=obj7 2024-02-01T10:48:05.278 DEBUG:boto:Data: 2024-02-01T10:48:05.278 DEBUG:boto:Headers: {} 2024-02-01T10:48:05.278 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:05.278 DEBUG:boto:Port: 8001 2024-02-01T10:48:05.278 DEBUG:boto:Params: {} 2024-02-01T10:48:05.278 DEBUG:boto:Token: None 2024-02-01T10:48:05.278 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:05 GMT /vclfcn-59/ 2024-02-01T10:48:05.278 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Q9LrdrhS2DRPqHDne9rbiEMNiH8= 2024-02-01T10:48:05.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Q9LrdrhS2DRPqHDne9rbiEMNiH8=', 'Content-Length': '0'} 2024-02-01T10:48:05.283 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000455928b0ab9c346-0065bb76e5-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:05.324 DEBUG:boto:b'vclfcn-59obj71000falseobj72024-02-01T10:47:30.227Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:05.325 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:05.325 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:05.325 DEBUG:boto:path=/vclfcn-59/?prefix=obj7 2024-02-01T10:48:05.325 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj7 2024-02-01T10:48:05.325 DEBUG:boto:Method: GET 2024-02-01T10:48:05.325 DEBUG:boto:Path: /vclfcn-59/?prefix=obj7 2024-02-01T10:48:05.325 DEBUG:boto:Data: 2024-02-01T10:48:05.325 DEBUG:boto:Headers: {} 2024-02-01T10:48:05.326 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:05.326 DEBUG:boto:Port: 8000 2024-02-01T10:48:05.326 DEBUG:boto:Params: {} 2024-02-01T10:48:05.326 DEBUG:boto:Token: None 2024-02-01T10:48:05.326 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:05 GMT /vclfcn-59/ 2024-02-01T10:48:05.326 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Q9LrdrhS2DRPqHDne9rbiEMNiH8= 2024-02-01T10:48:05.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Q9LrdrhS2DRPqHDne9rbiEMNiH8=', 'Content-Length': '0'} 2024-02-01T10:48:05.330 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad0942db41e622fb-0065bb76e5-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:05.331 DEBUG:boto:b'vclfcn-59obj71000falseobj72024-02-01T10:47:30.227Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:05.331 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-02-01T10:48:05.331 DEBUG:boto:path=/vclfcn-59/obj8 2024-02-01T10:48:05.331 DEBUG:boto:auth_path=/vclfcn-59/obj8 2024-02-01T10:48:05.331 DEBUG:boto:path=/vclfcn-59/obj8?versionId=null 2024-02-01T10:48:05.331 DEBUG:boto:auth_path=/vclfcn-59/obj8?versionId=null 2024-02-01T10:48:05.332 DEBUG:boto:Method: GET 2024-02-01T10:48:05.332 DEBUG:boto:Path: /vclfcn-59/obj8?versionId=null 2024-02-01T10:48:05.332 DEBUG:boto:Data: 2024-02-01T10:48:05.332 DEBUG:boto:Headers: {} 2024-02-01T10:48:05.332 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:05.332 DEBUG:boto:Port: 8001 2024-02-01T10:48:05.332 DEBUG:boto:Params: {} 2024-02-01T10:48:05.332 DEBUG:boto:Token: None 2024-02-01T10:48:05.332 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:05 GMT /vclfcn-59/obj8?versionId=null 2024-02-01T10:48:05.333 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+XqT+Feyp5s70xLgua5U7FwUqT0= 2024-02-01T10:48:05.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+XqT+Feyp5s70xLgua5U7FwUqT0=', 'Content-Length': '0'} 2024-02-01T10:48:05.335 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000914e08a0eefb4220-0065bb76e5-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:05.376 DEBUG:boto:path=/vclfcn-59/obj8 2024-02-01T10:48:05.376 DEBUG:boto:auth_path=/vclfcn-59/obj8 2024-02-01T10:48:05.376 DEBUG:boto:path=/vclfcn-59/obj8?versionId=null 2024-02-01T10:48:05.376 DEBUG:boto:auth_path=/vclfcn-59/obj8?versionId=null 2024-02-01T10:48:05.376 DEBUG:boto:Method: GET 2024-02-01T10:48:05.377 DEBUG:boto:Path: /vclfcn-59/obj8?versionId=null 2024-02-01T10:48:05.377 DEBUG:boto:Data: 2024-02-01T10:48:05.377 DEBUG:boto:Headers: {} 2024-02-01T10:48:05.377 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:05.377 DEBUG:boto:Port: 8000 2024-02-01T10:48:05.377 DEBUG:boto:Params: {} 2024-02-01T10:48:05.377 DEBUG:boto:Token: None 2024-02-01T10:48:05.377 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:05 GMT /vclfcn-59/obj8?versionId=null 2024-02-01T10:48:05.378 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+XqT+Feyp5s70xLgua5U7FwUqT0= 2024-02-01T10:48:05.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+XqT+Feyp5s70xLgua5U7FwUqT0=', 'Content-Length': '0'} 2024-02-01T10:48:05.381 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fa72d47ea7f23cbe-0065bb76e5-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:05.381 DEBUG:boto:path=/vclfcn-59/obj8 2024-02-01T10:48:05.381 DEBUG:boto:auth_path=/vclfcn-59/obj8 2024-02-01T10:48:05.381 DEBUG:boto:path=/vclfcn-59/obj8?versionId=null 2024-02-01T10:48:05.381 DEBUG:boto:auth_path=/vclfcn-59/obj8?versionId=null 2024-02-01T10:48:05.382 DEBUG:boto:Method: HEAD 2024-02-01T10:48:05.382 DEBUG:boto:Path: /vclfcn-59/obj8?versionId=null 2024-02-01T10:48:05.382 DEBUG:boto:Data: 2024-02-01T10:48:05.382 DEBUG:boto:Headers: {} 2024-02-01T10:48:05.382 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:05.382 DEBUG:boto:Port: 8001 2024-02-01T10:48:05.382 DEBUG:boto:Params: {} 2024-02-01T10:48:05.382 DEBUG:boto:Token: None 2024-02-01T10:48:05.383 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:05 GMT /vclfcn-59/obj8?versionId=null 2024-02-01T10:48:05.383 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BSzx3kvItrHjwQ+xgtLAJQNYaKU= 2024-02-01T10:48:05.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BSzx3kvItrHjwQ+xgtLAJQNYaKU=', 'Content-Length': '0'} 2024-02-01T10:48:05.385 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000013eebd396dd77537-0065bb76e5-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:05.386 DEBUG:boto:path=/vclfcn-59/obj8 2024-02-01T10:48:05.386 DEBUG:boto:auth_path=/vclfcn-59/obj8 2024-02-01T10:48:05.386 DEBUG:boto:path=/vclfcn-59/obj8?versionId=null 2024-02-01T10:48:05.386 DEBUG:boto:auth_path=/vclfcn-59/obj8?versionId=null 2024-02-01T10:48:05.386 DEBUG:boto:Method: HEAD 2024-02-01T10:48:05.386 DEBUG:boto:Path: /vclfcn-59/obj8?versionId=null 2024-02-01T10:48:05.386 DEBUG:boto:Data: 2024-02-01T10:48:05.386 DEBUG:boto:Headers: {} 2024-02-01T10:48:05.387 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:05.387 DEBUG:boto:Port: 8000 2024-02-01T10:48:05.387 DEBUG:boto:Params: {} 2024-02-01T10:48:05.387 DEBUG:boto:Token: None 2024-02-01T10:48:05.387 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:05 GMT /vclfcn-59/obj8?versionId=null 2024-02-01T10:48:05.387 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BSzx3kvItrHjwQ+xgtLAJQNYaKU= 2024-02-01T10:48:05.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BSzx3kvItrHjwQ+xgtLAJQNYaKU=', 'Content-Length': '0'} 2024-02-01T10:48:05.390 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000435c3a2daa6f6557-0065bb76e5-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:05.390 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-02-01T10:48:05.390 DEBUG:boto:path=/vclfcn-59/obj8 2024-02-01T10:48:05.390 DEBUG:boto:auth_path=/vclfcn-59/obj8 2024-02-01T10:48:05.390 DEBUG:boto:path=/vclfcn-59/obj8?versionId=null 2024-02-01T10:48:05.390 DEBUG:boto:auth_path=/vclfcn-59/obj8?versionId=null 2024-02-01T10:48:05.391 DEBUG:boto:Method: GET 2024-02-01T10:48:05.391 DEBUG:boto:Path: /vclfcn-59/obj8?versionId=null 2024-02-01T10:48:05.391 DEBUG:boto:Data: 2024-02-01T10:48:05.391 DEBUG:boto:Headers: {} 2024-02-01T10:48:05.391 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:05.391 DEBUG:boto:Port: 8001 2024-02-01T10:48:05.391 DEBUG:boto:Params: {} 2024-02-01T10:48:05.391 DEBUG:boto:Token: None 2024-02-01T10:48:05.391 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:05 GMT /vclfcn-59/obj8?versionId=null 2024-02-01T10:48:05.392 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+XqT+Feyp5s70xLgua5U7FwUqT0= 2024-02-01T10:48:05.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+XqT+Feyp5s70xLgua5U7FwUqT0=', 'Content-Length': '0'} 2024-02-01T10:48:05.394 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000539fd39ef57ee555-0065bb76e5-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:05.440 DEBUG:boto:path=/vclfcn-59/obj8 2024-02-01T10:48:05.440 DEBUG:boto:auth_path=/vclfcn-59/obj8 2024-02-01T10:48:05.440 DEBUG:boto:path=/vclfcn-59/obj8?versionId=null 2024-02-01T10:48:05.440 DEBUG:boto:auth_path=/vclfcn-59/obj8?versionId=null 2024-02-01T10:48:05.440 DEBUG:boto:Method: GET 2024-02-01T10:48:05.440 DEBUG:boto:Path: /vclfcn-59/obj8?versionId=null 2024-02-01T10:48:05.441 DEBUG:boto:Data: 2024-02-01T10:48:05.441 DEBUG:boto:Headers: {} 2024-02-01T10:48:05.441 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:05.441 DEBUG:boto:Port: 8000 2024-02-01T10:48:05.441 DEBUG:boto:Params: {} 2024-02-01T10:48:05.441 DEBUG:boto:Token: None 2024-02-01T10:48:05.441 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:05 GMT /vclfcn-59/obj8?versionId=null 2024-02-01T10:48:05.441 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+XqT+Feyp5s70xLgua5U7FwUqT0= 2024-02-01T10:48:05.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+XqT+Feyp5s70xLgua5U7FwUqT0=', 'Content-Length': '0'} 2024-02-01T10:48:05.444 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000479b0699073413c-0065bb76e5-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:05.444 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:05.444 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:05.444 DEBUG:boto:path=/vclfcn-59/?prefix=obj8 2024-02-01T10:48:05.444 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj8 2024-02-01T10:48:05.445 DEBUG:boto:Method: GET 2024-02-01T10:48:05.445 DEBUG:boto:Path: /vclfcn-59/?prefix=obj8 2024-02-01T10:48:05.445 DEBUG:boto:Data: 2024-02-01T10:48:05.445 DEBUG:boto:Headers: {} 2024-02-01T10:48:05.445 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:05.445 DEBUG:boto:Port: 8001 2024-02-01T10:48:05.445 DEBUG:boto:Params: {} 2024-02-01T10:48:05.445 DEBUG:boto:Token: None 2024-02-01T10:48:05.445 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:05 GMT /vclfcn-59/ 2024-02-01T10:48:05.445 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Q9LrdrhS2DRPqHDne9rbiEMNiH8= 2024-02-01T10:48:05.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Q9LrdrhS2DRPqHDne9rbiEMNiH8=', 'Content-Length': '0'} 2024-02-01T10:48:05.450 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000036a5155fc5a90919-0065bb76e5-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:05.492 DEBUG:boto:b'vclfcn-59obj81000falseobj82024-02-01T10:47:30.275Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:05.492 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:05.493 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:05.493 DEBUG:boto:path=/vclfcn-59/?prefix=obj8 2024-02-01T10:48:05.493 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj8 2024-02-01T10:48:05.493 DEBUG:boto:Method: GET 2024-02-01T10:48:05.493 DEBUG:boto:Path: /vclfcn-59/?prefix=obj8 2024-02-01T10:48:05.493 DEBUG:boto:Data: 2024-02-01T10:48:05.493 DEBUG:boto:Headers: {} 2024-02-01T10:48:05.493 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:05.493 DEBUG:boto:Port: 8000 2024-02-01T10:48:05.493 DEBUG:boto:Params: {} 2024-02-01T10:48:05.494 DEBUG:boto:Token: None 2024-02-01T10:48:05.494 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:05 GMT /vclfcn-59/ 2024-02-01T10:48:05.494 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Q9LrdrhS2DRPqHDne9rbiEMNiH8= 2024-02-01T10:48:05.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Q9LrdrhS2DRPqHDne9rbiEMNiH8=', 'Content-Length': '0'} 2024-02-01T10:48:05.498 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000087789851ded19c7-0065bb76e5-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:05.499 DEBUG:boto:b'vclfcn-59obj81000falseobj82024-02-01T10:47:30.275Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:05.499 DEBUG:boto:path=/vclfcn-59/obj8 2024-02-01T10:48:05.499 DEBUG:boto:auth_path=/vclfcn-59/obj8 2024-02-01T10:48:05.499 DEBUG:boto:Method: HEAD 2024-02-01T10:48:05.499 DEBUG:boto:Path: /vclfcn-59/obj8 2024-02-01T10:48:05.499 DEBUG:boto:Data: 2024-02-01T10:48:05.499 DEBUG:boto:Headers: {} 2024-02-01T10:48:05.499 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:05.500 DEBUG:boto:Port: 8001 2024-02-01T10:48:05.500 DEBUG:boto:Params: {} 2024-02-01T10:48:05.500 DEBUG:boto:Token: None 2024-02-01T10:48:05.500 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:05 GMT /vclfcn-59/obj8 2024-02-01T10:48:05.500 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:eF2cJ2kV5vzl7GJnWvhoum8WPEI= 2024-02-01T10:48:05.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:eF2cJ2kV5vzl7GJnWvhoum8WPEI=', 'Content-Length': '0'} 2024-02-01T10:48:05.503 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e1c014400be65432-0065bb76e5-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:05.503 DEBUG:boto:path=/vclfcn-59/obj8 2024-02-01T10:48:05.503 DEBUG:boto:auth_path=/vclfcn-59/obj8 2024-02-01T10:48:05.503 DEBUG:boto:Method: HEAD 2024-02-01T10:48:05.503 DEBUG:boto:Path: /vclfcn-59/obj8 2024-02-01T10:48:05.504 DEBUG:boto:Data: 2024-02-01T10:48:05.504 DEBUG:boto:Headers: {} 2024-02-01T10:48:05.504 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:05.504 DEBUG:boto:Port: 8000 2024-02-01T10:48:05.504 DEBUG:boto:Params: {} 2024-02-01T10:48:05.504 DEBUG:boto:Token: None 2024-02-01T10:48:05.504 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:05 GMT /vclfcn-59/obj8 2024-02-01T10:48:05.504 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:eF2cJ2kV5vzl7GJnWvhoum8WPEI= 2024-02-01T10:48:05.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:eF2cJ2kV5vzl7GJnWvhoum8WPEI=', 'Content-Length': '0'} 2024-02-01T10:48:05.507 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004998a666e34820dc-0065bb76e5-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:05.507 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-02-01T10:48:05.507 DEBUG:boto:path=/vclfcn-59/obj8 2024-02-01T10:48:05.507 DEBUG:boto:auth_path=/vclfcn-59/obj8 2024-02-01T10:48:05.507 DEBUG:boto:Method: GET 2024-02-01T10:48:05.507 DEBUG:boto:Path: /vclfcn-59/obj8 2024-02-01T10:48:05.508 DEBUG:boto:Data: 2024-02-01T10:48:05.508 DEBUG:boto:Headers: {} 2024-02-01T10:48:05.508 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:05.508 DEBUG:boto:Port: 8001 2024-02-01T10:48:05.508 DEBUG:boto:Params: {} 2024-02-01T10:48:05.508 DEBUG:boto:Token: None 2024-02-01T10:48:05.508 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:05 GMT /vclfcn-59/obj8 2024-02-01T10:48:05.508 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8WmekNq0011T0TlOOfc5q95xcZc= 2024-02-01T10:48:05.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8WmekNq0011T0TlOOfc5q95xcZc=', 'Content-Length': '0'} 2024-02-01T10:48:05.511 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000061267425428e2d96-0065bb76e5-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:05.552 DEBUG:boto:path=/vclfcn-59/obj8 2024-02-01T10:48:05.552 DEBUG:boto:auth_path=/vclfcn-59/obj8 2024-02-01T10:48:05.552 DEBUG:boto:Method: GET 2024-02-01T10:48:05.553 DEBUG:boto:Path: /vclfcn-59/obj8 2024-02-01T10:48:05.553 DEBUG:boto:Data: 2024-02-01T10:48:05.553 DEBUG:boto:Headers: {} 2024-02-01T10:48:05.553 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:05.553 DEBUG:boto:Port: 8000 2024-02-01T10:48:05.553 DEBUG:boto:Params: {} 2024-02-01T10:48:05.553 DEBUG:boto:Token: None 2024-02-01T10:48:05.553 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:05 GMT /vclfcn-59/obj8 2024-02-01T10:48:05.553 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8WmekNq0011T0TlOOfc5q95xcZc= 2024-02-01T10:48:05.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8WmekNq0011T0TlOOfc5q95xcZc=', 'Content-Length': '0'} 2024-02-01T10:48:05.556 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000d3de8bd01fe4cdd-0065bb76e5-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:05.557 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:05.557 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:05.557 DEBUG:boto:path=/vclfcn-59/?prefix=obj8 2024-02-01T10:48:05.557 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj8 2024-02-01T10:48:05.557 DEBUG:boto:Method: GET 2024-02-01T10:48:05.557 DEBUG:boto:Path: /vclfcn-59/?prefix=obj8 2024-02-01T10:48:05.557 DEBUG:boto:Data: 2024-02-01T10:48:05.557 DEBUG:boto:Headers: {} 2024-02-01T10:48:05.557 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:05.557 DEBUG:boto:Port: 8001 2024-02-01T10:48:05.557 DEBUG:boto:Params: {} 2024-02-01T10:48:05.558 DEBUG:boto:Token: None 2024-02-01T10:48:05.558 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:05 GMT /vclfcn-59/ 2024-02-01T10:48:05.558 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Q9LrdrhS2DRPqHDne9rbiEMNiH8= 2024-02-01T10:48:05.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Q9LrdrhS2DRPqHDne9rbiEMNiH8=', 'Content-Length': '0'} 2024-02-01T10:48:05.562 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006845b6b650a7a5ce-0065bb76e5-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:05.562 DEBUG:boto:b'vclfcn-59obj81000falseobj82024-02-01T10:47:30.275Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:05.563 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:05.563 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:05.563 DEBUG:boto:path=/vclfcn-59/?prefix=obj8 2024-02-01T10:48:05.563 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj8 2024-02-01T10:48:05.563 DEBUG:boto:Method: GET 2024-02-01T10:48:05.563 DEBUG:boto:Path: /vclfcn-59/?prefix=obj8 2024-02-01T10:48:05.563 DEBUG:boto:Data: 2024-02-01T10:48:05.563 DEBUG:boto:Headers: {} 2024-02-01T10:48:05.563 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:05.563 DEBUG:boto:Port: 8000 2024-02-01T10:48:05.563 DEBUG:boto:Params: {} 2024-02-01T10:48:05.564 DEBUG:boto:Token: None 2024-02-01T10:48:05.564 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:05 GMT /vclfcn-59/ 2024-02-01T10:48:05.564 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Q9LrdrhS2DRPqHDne9rbiEMNiH8= 2024-02-01T10:48:05.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Q9LrdrhS2DRPqHDne9rbiEMNiH8=', 'Content-Length': '0'} 2024-02-01T10:48:05.568 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003517226a2f4d2c0a-0065bb76e5-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:05.568 DEBUG:boto:b'vclfcn-59obj81000falseobj82024-02-01T10:47:30.275Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:05.569 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-59 zones={test-zone3, test-zone1} 2024-02-01T10:48:05.569 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-59 zones={test-zone3, test-zone2} 2024-02-01T10:48:05.569 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:05.569 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:05.569 DEBUG:boto:Method: HEAD 2024-02-01T10:48:05.569 DEBUG:boto:Path: /vclfcn-59/ 2024-02-01T10:48:05.569 DEBUG:boto:Data: 2024-02-01T10:48:05.569 DEBUG:boto:Headers: {} 2024-02-01T10:48:05.570 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:05.570 DEBUG:boto:Port: 8001 2024-02-01T10:48:05.570 DEBUG:boto:Params: {} 2024-02-01T10:48:05.570 DEBUG:boto:Token: None 2024-02-01T10:48:05.570 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:05 GMT /vclfcn-59/ 2024-02-01T10:48:05.570 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Oev9saGCkek51fDjCtsLu0+94mc= 2024-02-01T10:48:05.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Oev9saGCkek51fDjCtsLu0+94mc=', 'Content-Length': '0'} 2024-02-01T10:48:05.574 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c4b9e12bfbf7f4ff-0065bb76e5-5762-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:48:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:05.574 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:05.574 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:05.574 DEBUG:boto:Method: HEAD 2024-02-01T10:48:05.574 DEBUG:boto:Path: /vclfcn-59/ 2024-02-01T10:48:05.575 DEBUG:boto:Data: 2024-02-01T10:48:05.575 DEBUG:boto:Headers: {} 2024-02-01T10:48:05.575 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:05.575 DEBUG:boto:Port: 8000 2024-02-01T10:48:05.575 DEBUG:boto:Params: {} 2024-02-01T10:48:05.575 DEBUG:boto:Token: None 2024-02-01T10:48:05.575 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:05 GMT /vclfcn-59/ 2024-02-01T10:48:05.575 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Oev9saGCkek51fDjCtsLu0+94mc= 2024-02-01T10:48:05.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Oev9saGCkek51fDjCtsLu0+94mc=', 'Content-Length': '0'} 2024-02-01T10:48:05.579 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a202911ca5023b1b-0065bb76e5-5350-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:48:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:05.579 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:48:05.580 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:05.580 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:05.580 DEBUG:boto:path=/vclfcn-59/?versions&max-keys=999 2024-02-01T10:48:05.580 DEBUG:boto:auth_path=/vclfcn-59/?versions&max-keys=999 2024-02-01T10:48:05.580 DEBUG:boto:Method: GET 2024-02-01T10:48:05.580 DEBUG:boto:Path: /vclfcn-59/?versions&max-keys=999 2024-02-01T10:48:05.580 DEBUG:boto:Data: 2024-02-01T10:48:05.580 DEBUG:boto:Headers: {} 2024-02-01T10:48:05.580 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:05.580 DEBUG:boto:Port: 8001 2024-02-01T10:48:05.580 DEBUG:boto:Params: {} 2024-02-01T10:48:05.581 DEBUG:boto:Token: None 2024-02-01T10:48:05.581 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:05 GMT /vclfcn-59/?versions 2024-02-01T10:48:05.581 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Yd6uAn1yDEIeXYz2gq3OmINv1DA= 2024-02-01T10:48:05.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Yd6uAn1yDEIeXYz2gq3OmINv1DA=', 'Content-Length': '0'} 2024-02-01T10:48:05.586 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050fa8ee11f8bc9b2-0065bb76e5-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:05.587 DEBUG:boto:b'vclfcn-59999falseobj1nulltrue2024-02-01T10:46:46.093Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:46:46.111Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:46:46.161Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:46:46.190Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-02-01T10:47:30.206Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-02-01T10:47:30.217Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-02-01T10:47:30.227Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-02-01T10:47:30.275Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:05.587 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-02-01T10:48:05.587 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-02-01T10:48:05.587 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-02-01T10:48:05.588 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-02-01T10:48:05.588 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-02-01T10:48:05.588 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-02-01T10:48:05.588 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-02-01T10:48:05.588 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-02-01T10:48:05.588 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:48:05.588 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:05.588 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:05.588 DEBUG:boto:path=/vclfcn-59/?versions&max-keys=999 2024-02-01T10:48:05.588 DEBUG:boto:auth_path=/vclfcn-59/?versions&max-keys=999 2024-02-01T10:48:05.589 DEBUG:boto:Method: GET 2024-02-01T10:48:05.589 DEBUG:boto:Path: /vclfcn-59/?versions&max-keys=999 2024-02-01T10:48:05.589 DEBUG:boto:Data: 2024-02-01T10:48:05.589 DEBUG:boto:Headers: {} 2024-02-01T10:48:05.589 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:05.589 DEBUG:boto:Port: 8000 2024-02-01T10:48:05.589 DEBUG:boto:Params: {} 2024-02-01T10:48:05.589 DEBUG:boto:Token: None 2024-02-01T10:48:05.589 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:05 GMT /vclfcn-59/?versions 2024-02-01T10:48:05.589 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Yd6uAn1yDEIeXYz2gq3OmINv1DA= 2024-02-01T10:48:05.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Yd6uAn1yDEIeXYz2gq3OmINv1DA=', 'Content-Length': '0'} 2024-02-01T10:48:05.594 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e8f25267565ce66-0065bb76e5-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:05.596 DEBUG:boto:b'vclfcn-59999falseobj1nulltrue2024-02-01T10:46:46.093Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:46:46.111Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:46:46.161Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:46:46.190Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-02-01T10:47:30.206Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-02-01T10:47:30.217Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-02-01T10:47:30.227Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-02-01T10:47:30.275Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:05.599 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-02-01T10:48:05.599 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-02-01T10:48:05.599 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-02-01T10:48:05.599 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-02-01T10:48:05.600 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-02-01T10:48:05.600 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-02-01T10:48:05.600 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-02-01T10:48:05.600 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-02-01T10:48:05.600 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:05.600 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:05.600 DEBUG:boto:path=/vclfcn-59/?versions&max-keys=999 2024-02-01T10:48:05.600 DEBUG:boto:auth_path=/vclfcn-59/?versions&max-keys=999 2024-02-01T10:48:05.600 DEBUG:boto:Method: GET 2024-02-01T10:48:05.600 DEBUG:boto:Path: /vclfcn-59/?versions&max-keys=999 2024-02-01T10:48:05.601 DEBUG:boto:Data: 2024-02-01T10:48:05.601 DEBUG:boto:Headers: {} 2024-02-01T10:48:05.601 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:05.601 DEBUG:boto:Port: 8001 2024-02-01T10:48:05.601 DEBUG:boto:Params: {} 2024-02-01T10:48:05.601 DEBUG:boto:Token: None 2024-02-01T10:48:05.601 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:05 GMT /vclfcn-59/?versions 2024-02-01T10:48:05.601 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Yd6uAn1yDEIeXYz2gq3OmINv1DA= 2024-02-01T10:48:05.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Yd6uAn1yDEIeXYz2gq3OmINv1DA=', 'Content-Length': '0'} 2024-02-01T10:48:05.606 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b12e9b5c1117718-0065bb76e5-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:05.606 DEBUG:boto:b'vclfcn-59999falseobj1nulltrue2024-02-01T10:46:46.093Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:46:46.111Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:46:46.161Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:46:46.190Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-02-01T10:47:30.206Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-02-01T10:47:30.217Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-02-01T10:47:30.227Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-02-01T10:47:30.275Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:05.607 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:05.607 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:05.607 DEBUG:boto:path=/vclfcn-59/?versions&max-keys=999 2024-02-01T10:48:05.607 DEBUG:boto:auth_path=/vclfcn-59/?versions&max-keys=999 2024-02-01T10:48:05.607 DEBUG:boto:Method: GET 2024-02-01T10:48:05.607 DEBUG:boto:Path: /vclfcn-59/?versions&max-keys=999 2024-02-01T10:48:05.607 DEBUG:boto:Data: 2024-02-01T10:48:05.607 DEBUG:boto:Headers: {} 2024-02-01T10:48:05.608 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:05.608 DEBUG:boto:Port: 8000 2024-02-01T10:48:05.608 DEBUG:boto:Params: {} 2024-02-01T10:48:05.608 DEBUG:boto:Token: None 2024-02-01T10:48:05.608 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:05 GMT /vclfcn-59/?versions 2024-02-01T10:48:05.608 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Yd6uAn1yDEIeXYz2gq3OmINv1DA= 2024-02-01T10:48:05.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Yd6uAn1yDEIeXYz2gq3OmINv1DA=', 'Content-Length': '0'} 2024-02-01T10:48:05.613 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b434d55b209759a-0065bb76e5-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:05.614 DEBUG:boto:b'vclfcn-59999falseobj1nulltrue2024-02-01T10:46:46.093Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:46:46.111Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:46:46.161Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:46:46.190Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-02-01T10:47:30.206Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-02-01T10:47:30.217Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-02-01T10:47:30.227Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-02-01T10:47:30.275Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:05.614 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:48:05.614 DEBUG:boto:path=/vclfcn-59/obj1 2024-02-01T10:48:05.615 DEBUG:boto:auth_path=/vclfcn-59/obj1 2024-02-01T10:48:05.615 DEBUG:boto:path=/vclfcn-59/obj1?versionId=null 2024-02-01T10:48:05.615 DEBUG:boto:auth_path=/vclfcn-59/obj1?versionId=null 2024-02-01T10:48:05.615 DEBUG:boto:Method: GET 2024-02-01T10:48:05.615 DEBUG:boto:Path: /vclfcn-59/obj1?versionId=null 2024-02-01T10:48:05.615 DEBUG:boto:Data: 2024-02-01T10:48:05.615 DEBUG:boto:Headers: {} 2024-02-01T10:48:05.615 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:05.615 DEBUG:boto:Port: 8001 2024-02-01T10:48:05.615 DEBUG:boto:Params: {} 2024-02-01T10:48:05.615 DEBUG:boto:Token: None 2024-02-01T10:48:05.616 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:05 GMT /vclfcn-59/obj1?versionId=null 2024-02-01T10:48:05.616 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:kkobTVkqZIckK0YR0DQb3rMtnKA= 2024-02-01T10:48:05.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:kkobTVkqZIckK0YR0DQb3rMtnKA=', 'Content-Length': '0'} 2024-02-01T10:48:05.619 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000be24f80ae1a02468-0065bb76e5-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:05.660 DEBUG:boto:path=/vclfcn-59/obj1 2024-02-01T10:48:05.660 DEBUG:boto:auth_path=/vclfcn-59/obj1 2024-02-01T10:48:05.661 DEBUG:boto:path=/vclfcn-59/obj1?versionId=null 2024-02-01T10:48:05.661 DEBUG:boto:auth_path=/vclfcn-59/obj1?versionId=null 2024-02-01T10:48:05.661 DEBUG:boto:Method: GET 2024-02-01T10:48:05.661 DEBUG:boto:Path: /vclfcn-59/obj1?versionId=null 2024-02-01T10:48:05.661 DEBUG:boto:Data: 2024-02-01T10:48:05.661 DEBUG:boto:Headers: {} 2024-02-01T10:48:05.661 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:05.661 DEBUG:boto:Port: 8000 2024-02-01T10:48:05.661 DEBUG:boto:Params: {} 2024-02-01T10:48:05.661 DEBUG:boto:Token: None 2024-02-01T10:48:05.662 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:05 GMT /vclfcn-59/obj1?versionId=null 2024-02-01T10:48:05.662 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:kkobTVkqZIckK0YR0DQb3rMtnKA= 2024-02-01T10:48:05.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:kkobTVkqZIckK0YR0DQb3rMtnKA=', 'Content-Length': '0'} 2024-02-01T10:48:05.665 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000db513f26f8d0a7b9-0065bb76e5-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:05.665 DEBUG:boto:path=/vclfcn-59/obj1 2024-02-01T10:48:05.665 DEBUG:boto:auth_path=/vclfcn-59/obj1 2024-02-01T10:48:05.665 DEBUG:boto:path=/vclfcn-59/obj1?versionId=null 2024-02-01T10:48:05.665 DEBUG:boto:auth_path=/vclfcn-59/obj1?versionId=null 2024-02-01T10:48:05.665 DEBUG:boto:Method: HEAD 2024-02-01T10:48:05.665 DEBUG:boto:Path: /vclfcn-59/obj1?versionId=null 2024-02-01T10:48:05.666 DEBUG:boto:Data: 2024-02-01T10:48:05.666 DEBUG:boto:Headers: {} 2024-02-01T10:48:05.666 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:05.666 DEBUG:boto:Port: 8001 2024-02-01T10:48:05.666 DEBUG:boto:Params: {} 2024-02-01T10:48:05.666 DEBUG:boto:Token: None 2024-02-01T10:48:05.666 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:05 GMT /vclfcn-59/obj1?versionId=null 2024-02-01T10:48:05.666 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wzbORyPjnBEHJIochGF5GugLiSQ= 2024-02-01T10:48:05.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wzbORyPjnBEHJIochGF5GugLiSQ=', 'Content-Length': '0'} 2024-02-01T10:48:05.669 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d5af7e8c927fe5c9-0065bb76e5-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:05.669 DEBUG:boto:path=/vclfcn-59/obj1 2024-02-01T10:48:05.670 DEBUG:boto:auth_path=/vclfcn-59/obj1 2024-02-01T10:48:05.670 DEBUG:boto:path=/vclfcn-59/obj1?versionId=null 2024-02-01T10:48:05.670 DEBUG:boto:auth_path=/vclfcn-59/obj1?versionId=null 2024-02-01T10:48:05.670 DEBUG:boto:Method: HEAD 2024-02-01T10:48:05.670 DEBUG:boto:Path: /vclfcn-59/obj1?versionId=null 2024-02-01T10:48:05.670 DEBUG:boto:Data: 2024-02-01T10:48:05.670 DEBUG:boto:Headers: {} 2024-02-01T10:48:05.670 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:05.670 DEBUG:boto:Port: 8000 2024-02-01T10:48:05.670 DEBUG:boto:Params: {} 2024-02-01T10:48:05.670 DEBUG:boto:Token: None 2024-02-01T10:48:05.671 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:05 GMT /vclfcn-59/obj1?versionId=null 2024-02-01T10:48:05.671 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wzbORyPjnBEHJIochGF5GugLiSQ= 2024-02-01T10:48:05.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wzbORyPjnBEHJIochGF5GugLiSQ=', 'Content-Length': '0'} 2024-02-01T10:48:05.673 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fa5b8357b2add260-0065bb76e5-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:05.674 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:48:05.674 DEBUG:boto:path=/vclfcn-59/obj1 2024-02-01T10:48:05.674 DEBUG:boto:auth_path=/vclfcn-59/obj1 2024-02-01T10:48:05.674 DEBUG:boto:path=/vclfcn-59/obj1?versionId=null 2024-02-01T10:48:05.674 DEBUG:boto:auth_path=/vclfcn-59/obj1?versionId=null 2024-02-01T10:48:05.674 DEBUG:boto:Method: GET 2024-02-01T10:48:05.674 DEBUG:boto:Path: /vclfcn-59/obj1?versionId=null 2024-02-01T10:48:05.674 DEBUG:boto:Data: 2024-02-01T10:48:05.675 DEBUG:boto:Headers: {} 2024-02-01T10:48:05.675 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:05.675 DEBUG:boto:Port: 8001 2024-02-01T10:48:05.675 DEBUG:boto:Params: {} 2024-02-01T10:48:05.675 DEBUG:boto:Token: None 2024-02-01T10:48:05.675 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:05 GMT /vclfcn-59/obj1?versionId=null 2024-02-01T10:48:05.675 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:kkobTVkqZIckK0YR0DQb3rMtnKA= 2024-02-01T10:48:05.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:kkobTVkqZIckK0YR0DQb3rMtnKA=', 'Content-Length': '0'} 2024-02-01T10:48:05.678 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002a20944a594dab8b-0065bb76e5-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:05.720 DEBUG:boto:path=/vclfcn-59/obj1 2024-02-01T10:48:05.720 DEBUG:boto:auth_path=/vclfcn-59/obj1 2024-02-01T10:48:05.720 DEBUG:boto:path=/vclfcn-59/obj1?versionId=null 2024-02-01T10:48:05.721 DEBUG:boto:auth_path=/vclfcn-59/obj1?versionId=null 2024-02-01T10:48:05.721 DEBUG:boto:Method: GET 2024-02-01T10:48:05.721 DEBUG:boto:Path: /vclfcn-59/obj1?versionId=null 2024-02-01T10:48:05.721 DEBUG:boto:Data: 2024-02-01T10:48:05.721 DEBUG:boto:Headers: {} 2024-02-01T10:48:05.721 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:05.721 DEBUG:boto:Port: 8000 2024-02-01T10:48:05.721 DEBUG:boto:Params: {} 2024-02-01T10:48:05.721 DEBUG:boto:Token: None 2024-02-01T10:48:05.722 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:05 GMT /vclfcn-59/obj1?versionId=null 2024-02-01T10:48:05.722 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:kkobTVkqZIckK0YR0DQb3rMtnKA= 2024-02-01T10:48:05.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:kkobTVkqZIckK0YR0DQb3rMtnKA=', 'Content-Length': '0'} 2024-02-01T10:48:05.725 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005d4a34a6f066138f-0065bb76e5-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:05.725 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:05.725 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:05.725 DEBUG:boto:path=/vclfcn-59/?prefix=obj1 2024-02-01T10:48:05.725 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj1 2024-02-01T10:48:05.725 DEBUG:boto:Method: GET 2024-02-01T10:48:05.725 DEBUG:boto:Path: /vclfcn-59/?prefix=obj1 2024-02-01T10:48:05.726 DEBUG:boto:Data: 2024-02-01T10:48:05.726 DEBUG:boto:Headers: {} 2024-02-01T10:48:05.726 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:05.726 DEBUG:boto:Port: 8001 2024-02-01T10:48:05.726 DEBUG:boto:Params: {} 2024-02-01T10:48:05.726 DEBUG:boto:Token: None 2024-02-01T10:48:05.726 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:05 GMT /vclfcn-59/ 2024-02-01T10:48:05.726 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Q9LrdrhS2DRPqHDne9rbiEMNiH8= 2024-02-01T10:48:05.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Q9LrdrhS2DRPqHDne9rbiEMNiH8=', 'Content-Length': '0'} 2024-02-01T10:48:05.730 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000387f95301e39aec5-0065bb76e5-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:05.776 DEBUG:boto:b'vclfcn-59obj11000falseobj12024-02-01T10:46:46.093Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:05.777 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:05.777 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:05.777 DEBUG:boto:path=/vclfcn-59/?prefix=obj1 2024-02-01T10:48:05.777 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj1 2024-02-01T10:48:05.777 DEBUG:boto:Method: GET 2024-02-01T10:48:05.777 DEBUG:boto:Path: /vclfcn-59/?prefix=obj1 2024-02-01T10:48:05.778 DEBUG:boto:Data: 2024-02-01T10:48:05.778 DEBUG:boto:Headers: {} 2024-02-01T10:48:05.778 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:05.778 DEBUG:boto:Port: 8000 2024-02-01T10:48:05.778 DEBUG:boto:Params: {} 2024-02-01T10:48:05.778 DEBUG:boto:Token: None 2024-02-01T10:48:05.778 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:05 GMT /vclfcn-59/ 2024-02-01T10:48:05.778 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Q9LrdrhS2DRPqHDne9rbiEMNiH8= 2024-02-01T10:48:05.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Q9LrdrhS2DRPqHDne9rbiEMNiH8=', 'Content-Length': '0'} 2024-02-01T10:48:05.783 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e709c554c8d1070b-0065bb76e5-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:05.783 DEBUG:boto:b'vclfcn-59obj11000falseobj12024-02-01T10:46:46.093Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:05.784 DEBUG:boto:path=/vclfcn-59/obj1 2024-02-01T10:48:05.784 DEBUG:boto:auth_path=/vclfcn-59/obj1 2024-02-01T10:48:05.784 DEBUG:boto:Method: HEAD 2024-02-01T10:48:05.784 DEBUG:boto:Path: /vclfcn-59/obj1 2024-02-01T10:48:05.784 DEBUG:boto:Data: 2024-02-01T10:48:05.784 DEBUG:boto:Headers: {} 2024-02-01T10:48:05.784 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:05.784 DEBUG:boto:Port: 8001 2024-02-01T10:48:05.784 DEBUG:boto:Params: {} 2024-02-01T10:48:05.785 DEBUG:boto:Token: None 2024-02-01T10:48:05.785 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:05 GMT /vclfcn-59/obj1 2024-02-01T10:48:05.785 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:q7H/bNwWq5BedoZxfIAFqq3uNgc= 2024-02-01T10:48:05.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:q7H/bNwWq5BedoZxfIAFqq3uNgc=', 'Content-Length': '0'} 2024-02-01T10:48:05.788 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009ec5159b3f093fd9-0065bb76e5-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:05.791 DEBUG:boto:path=/vclfcn-59/obj1 2024-02-01T10:48:05.791 DEBUG:boto:auth_path=/vclfcn-59/obj1 2024-02-01T10:48:05.791 DEBUG:boto:Method: HEAD 2024-02-01T10:48:05.791 DEBUG:boto:Path: /vclfcn-59/obj1 2024-02-01T10:48:05.791 DEBUG:boto:Data: 2024-02-01T10:48:05.791 DEBUG:boto:Headers: {} 2024-02-01T10:48:05.792 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:05.792 DEBUG:boto:Port: 8000 2024-02-01T10:48:05.792 DEBUG:boto:Params: {} 2024-02-01T10:48:05.792 DEBUG:boto:Token: None 2024-02-01T10:48:05.792 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:05 GMT /vclfcn-59/obj1 2024-02-01T10:48:05.792 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:q7H/bNwWq5BedoZxfIAFqq3uNgc= 2024-02-01T10:48:05.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:q7H/bNwWq5BedoZxfIAFqq3uNgc=', 'Content-Length': '0'} 2024-02-01T10:48:05.795 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a8144b7bf880ff70-0065bb76e5-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:05.795 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:48:05.796 DEBUG:boto:path=/vclfcn-59/obj1 2024-02-01T10:48:05.796 DEBUG:boto:auth_path=/vclfcn-59/obj1 2024-02-01T10:48:05.796 DEBUG:boto:Method: GET 2024-02-01T10:48:05.796 DEBUG:boto:Path: /vclfcn-59/obj1 2024-02-01T10:48:05.796 DEBUG:boto:Data: 2024-02-01T10:48:05.796 DEBUG:boto:Headers: {} 2024-02-01T10:48:05.797 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:05.797 DEBUG:boto:Port: 8001 2024-02-01T10:48:05.797 DEBUG:boto:Params: {} 2024-02-01T10:48:05.797 DEBUG:boto:Token: None 2024-02-01T10:48:05.797 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:05 GMT /vclfcn-59/obj1 2024-02-01T10:48:05.797 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vKItSnZqD8XZg6W2UYS550Q5LGE= 2024-02-01T10:48:05.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vKItSnZqD8XZg6W2UYS550Q5LGE=', 'Content-Length': '0'} 2024-02-01T10:48:05.800 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000087a31e242d957913-0065bb76e5-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:05.848 DEBUG:boto:path=/vclfcn-59/obj1 2024-02-01T10:48:05.848 DEBUG:boto:auth_path=/vclfcn-59/obj1 2024-02-01T10:48:05.848 DEBUG:boto:Method: GET 2024-02-01T10:48:05.849 DEBUG:boto:Path: /vclfcn-59/obj1 2024-02-01T10:48:05.849 DEBUG:boto:Data: 2024-02-01T10:48:05.849 DEBUG:boto:Headers: {} 2024-02-01T10:48:05.849 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:05.849 DEBUG:boto:Port: 8000 2024-02-01T10:48:05.849 DEBUG:boto:Params: {} 2024-02-01T10:48:05.849 DEBUG:boto:Token: None 2024-02-01T10:48:05.849 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:05 GMT /vclfcn-59/obj1 2024-02-01T10:48:05.849 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vKItSnZqD8XZg6W2UYS550Q5LGE= 2024-02-01T10:48:05.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vKItSnZqD8XZg6W2UYS550Q5LGE=', 'Content-Length': '0'} 2024-02-01T10:48:05.852 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006def5a814281e54b-0065bb76e5-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:05.896 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:05.896 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:05.896 DEBUG:boto:path=/vclfcn-59/?prefix=obj1 2024-02-01T10:48:05.896 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj1 2024-02-01T10:48:05.897 DEBUG:boto:Method: GET 2024-02-01T10:48:05.897 DEBUG:boto:Path: /vclfcn-59/?prefix=obj1 2024-02-01T10:48:05.897 DEBUG:boto:Data: 2024-02-01T10:48:05.897 DEBUG:boto:Headers: {} 2024-02-01T10:48:05.897 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:05.897 DEBUG:boto:Port: 8001 2024-02-01T10:48:05.897 DEBUG:boto:Params: {} 2024-02-01T10:48:05.897 DEBUG:boto:Token: None 2024-02-01T10:48:05.897 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:05 GMT /vclfcn-59/ 2024-02-01T10:48:05.898 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Q9LrdrhS2DRPqHDne9rbiEMNiH8= 2024-02-01T10:48:05.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Q9LrdrhS2DRPqHDne9rbiEMNiH8=', 'Content-Length': '0'} 2024-02-01T10:48:05.902 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bec531e28d8a657d-0065bb76e5-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:05.944 DEBUG:boto:b'vclfcn-59obj11000falseobj12024-02-01T10:46:46.093Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:05.944 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:05.945 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:05.945 DEBUG:boto:path=/vclfcn-59/?prefix=obj1 2024-02-01T10:48:05.945 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj1 2024-02-01T10:48:05.945 DEBUG:boto:Method: GET 2024-02-01T10:48:05.945 DEBUG:boto:Path: /vclfcn-59/?prefix=obj1 2024-02-01T10:48:05.945 DEBUG:boto:Data: 2024-02-01T10:48:05.945 DEBUG:boto:Headers: {} 2024-02-01T10:48:05.945 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:05.945 DEBUG:boto:Port: 8000 2024-02-01T10:48:05.945 DEBUG:boto:Params: {} 2024-02-01T10:48:05.946 DEBUG:boto:Token: None 2024-02-01T10:48:05.946 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:05 GMT /vclfcn-59/ 2024-02-01T10:48:05.946 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Q9LrdrhS2DRPqHDne9rbiEMNiH8= 2024-02-01T10:48:05.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Q9LrdrhS2DRPqHDne9rbiEMNiH8=', 'Content-Length': '0'} 2024-02-01T10:48:05.950 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae13d77df904842b-0065bb76e5-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:05.992 DEBUG:boto:b'vclfcn-59obj11000falseobj12024-02-01T10:46:46.093Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:05.993 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:48:05.993 DEBUG:boto:path=/vclfcn-59/obj2 2024-02-01T10:48:05.993 DEBUG:boto:auth_path=/vclfcn-59/obj2 2024-02-01T10:48:05.993 DEBUG:boto:path=/vclfcn-59/obj2?versionId=null 2024-02-01T10:48:05.993 DEBUG:boto:auth_path=/vclfcn-59/obj2?versionId=null 2024-02-01T10:48:05.993 DEBUG:boto:Method: GET 2024-02-01T10:48:05.993 DEBUG:boto:Path: /vclfcn-59/obj2?versionId=null 2024-02-01T10:48:05.993 DEBUG:boto:Data: 2024-02-01T10:48:05.994 DEBUG:boto:Headers: {} 2024-02-01T10:48:05.994 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:05.994 DEBUG:boto:Port: 8001 2024-02-01T10:48:05.994 DEBUG:boto:Params: {} 2024-02-01T10:48:05.994 DEBUG:boto:Token: None 2024-02-01T10:48:05.994 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:05 GMT /vclfcn-59/obj2?versionId=null 2024-02-01T10:48:05.994 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:y4rNt2u95XBeu2E+0an7xTpX5W4= 2024-02-01T10:48:05.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:y4rNt2u95XBeu2E+0an7xTpX5W4=', 'Content-Length': '0'} 2024-02-01T10:48:05.997 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000073f745130dd46faf-0065bb76e5-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:05.998 DEBUG:boto:path=/vclfcn-59/obj2 2024-02-01T10:48:05.998 DEBUG:boto:auth_path=/vclfcn-59/obj2 2024-02-01T10:48:05.998 DEBUG:boto:path=/vclfcn-59/obj2?versionId=null 2024-02-01T10:48:05.998 DEBUG:boto:auth_path=/vclfcn-59/obj2?versionId=null 2024-02-01T10:48:05.998 DEBUG:boto:Method: GET 2024-02-01T10:48:05.998 DEBUG:boto:Path: /vclfcn-59/obj2?versionId=null 2024-02-01T10:48:05.998 DEBUG:boto:Data: 2024-02-01T10:48:05.999 DEBUG:boto:Headers: {} 2024-02-01T10:48:05.999 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:05.999 DEBUG:boto:Port: 8000 2024-02-01T10:48:05.999 DEBUG:boto:Params: {} 2024-02-01T10:48:05.999 DEBUG:boto:Token: None 2024-02-01T10:48:05.999 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:05 GMT /vclfcn-59/obj2?versionId=null 2024-02-01T10:48:05.999 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:y4rNt2u95XBeu2E+0an7xTpX5W4= 2024-02-01T10:48:05.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:y4rNt2u95XBeu2E+0an7xTpX5W4=', 'Content-Length': '0'} 2024-02-01T10:48:06.002 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dc5c5447f1d4a755-0065bb76e5-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:06.003 DEBUG:boto:path=/vclfcn-59/obj2 2024-02-01T10:48:06.003 DEBUG:boto:auth_path=/vclfcn-59/obj2 2024-02-01T10:48:06.003 DEBUG:boto:path=/vclfcn-59/obj2?versionId=null 2024-02-01T10:48:06.003 DEBUG:boto:auth_path=/vclfcn-59/obj2?versionId=null 2024-02-01T10:48:06.003 DEBUG:boto:Method: HEAD 2024-02-01T10:48:06.003 DEBUG:boto:Path: /vclfcn-59/obj2?versionId=null 2024-02-01T10:48:06.003 DEBUG:boto:Data: 2024-02-01T10:48:06.003 DEBUG:boto:Headers: {} 2024-02-01T10:48:06.003 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:06.004 DEBUG:boto:Port: 8001 2024-02-01T10:48:06.004 DEBUG:boto:Params: {} 2024-02-01T10:48:06.004 DEBUG:boto:Token: None 2024-02-01T10:48:06.004 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:06 GMT /vclfcn-59/obj2?versionId=null 2024-02-01T10:48:06.004 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lzBMtoKsrlF4IoV0wOZy2k4Oz08= 2024-02-01T10:48:06.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lzBMtoKsrlF4IoV0wOZy2k4Oz08=', 'Content-Length': '0'} 2024-02-01T10:48:06.007 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fa3d138296a73e09-0065bb76e6-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:06.007 DEBUG:boto:path=/vclfcn-59/obj2 2024-02-01T10:48:06.007 DEBUG:boto:auth_path=/vclfcn-59/obj2 2024-02-01T10:48:06.007 DEBUG:boto:path=/vclfcn-59/obj2?versionId=null 2024-02-01T10:48:06.007 DEBUG:boto:auth_path=/vclfcn-59/obj2?versionId=null 2024-02-01T10:48:06.008 DEBUG:boto:Method: HEAD 2024-02-01T10:48:06.008 DEBUG:boto:Path: /vclfcn-59/obj2?versionId=null 2024-02-01T10:48:06.008 DEBUG:boto:Data: 2024-02-01T10:48:06.008 DEBUG:boto:Headers: {} 2024-02-01T10:48:06.008 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:06.008 DEBUG:boto:Port: 8000 2024-02-01T10:48:06.008 DEBUG:boto:Params: {} 2024-02-01T10:48:06.008 DEBUG:boto:Token: None 2024-02-01T10:48:06.008 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:06 GMT /vclfcn-59/obj2?versionId=null 2024-02-01T10:48:06.009 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lzBMtoKsrlF4IoV0wOZy2k4Oz08= 2024-02-01T10:48:06.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lzBMtoKsrlF4IoV0wOZy2k4Oz08=', 'Content-Length': '0'} 2024-02-01T10:48:06.011 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000be00be8bcdbb888d-0065bb76e6-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:06.012 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:48:06.012 DEBUG:boto:path=/vclfcn-59/obj2 2024-02-01T10:48:06.012 DEBUG:boto:auth_path=/vclfcn-59/obj2 2024-02-01T10:48:06.012 DEBUG:boto:path=/vclfcn-59/obj2?versionId=null 2024-02-01T10:48:06.012 DEBUG:boto:auth_path=/vclfcn-59/obj2?versionId=null 2024-02-01T10:48:06.012 DEBUG:boto:Method: GET 2024-02-01T10:48:06.012 DEBUG:boto:Path: /vclfcn-59/obj2?versionId=null 2024-02-01T10:48:06.012 DEBUG:boto:Data: 2024-02-01T10:48:06.012 DEBUG:boto:Headers: {} 2024-02-01T10:48:06.013 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:06.013 DEBUG:boto:Port: 8001 2024-02-01T10:48:06.013 DEBUG:boto:Params: {} 2024-02-01T10:48:06.013 DEBUG:boto:Token: None 2024-02-01T10:48:06.013 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:06 GMT /vclfcn-59/obj2?versionId=null 2024-02-01T10:48:06.013 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:i+PDpnrVX0GdaEkipL3Vo/cSkr8= 2024-02-01T10:48:06.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:i+PDpnrVX0GdaEkipL3Vo/cSkr8=', 'Content-Length': '0'} 2024-02-01T10:48:06.016 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000003aa804f4ef1c93e-0065bb76e6-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:06.060 DEBUG:boto:path=/vclfcn-59/obj2 2024-02-01T10:48:06.060 DEBUG:boto:auth_path=/vclfcn-59/obj2 2024-02-01T10:48:06.060 DEBUG:boto:path=/vclfcn-59/obj2?versionId=null 2024-02-01T10:48:06.061 DEBUG:boto:auth_path=/vclfcn-59/obj2?versionId=null 2024-02-01T10:48:06.061 DEBUG:boto:Method: GET 2024-02-01T10:48:06.061 DEBUG:boto:Path: /vclfcn-59/obj2?versionId=null 2024-02-01T10:48:06.061 DEBUG:boto:Data: 2024-02-01T10:48:06.061 DEBUG:boto:Headers: {} 2024-02-01T10:48:06.061 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:06.061 DEBUG:boto:Port: 8000 2024-02-01T10:48:06.061 DEBUG:boto:Params: {} 2024-02-01T10:48:06.061 DEBUG:boto:Token: None 2024-02-01T10:48:06.062 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:06 GMT /vclfcn-59/obj2?versionId=null 2024-02-01T10:48:06.062 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:i+PDpnrVX0GdaEkipL3Vo/cSkr8= 2024-02-01T10:48:06.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:i+PDpnrVX0GdaEkipL3Vo/cSkr8=', 'Content-Length': '0'} 2024-02-01T10:48:06.065 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aa5b3455ea27deca-0065bb76e6-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:06.112 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:06.112 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:06.112 DEBUG:boto:path=/vclfcn-59/?prefix=obj2 2024-02-01T10:48:06.113 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj2 2024-02-01T10:48:06.113 DEBUG:boto:Method: GET 2024-02-01T10:48:06.113 DEBUG:boto:Path: /vclfcn-59/?prefix=obj2 2024-02-01T10:48:06.114 DEBUG:boto:Data: 2024-02-01T10:48:06.114 DEBUG:boto:Headers: {} 2024-02-01T10:48:06.114 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:06.114 DEBUG:boto:Port: 8001 2024-02-01T10:48:06.114 DEBUG:boto:Params: {} 2024-02-01T10:48:06.114 DEBUG:boto:Token: None 2024-02-01T10:48:06.114 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:06 GMT /vclfcn-59/ 2024-02-01T10:48:06.114 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:R/xyYuuFAaegFnbq0SJjJejgA30= 2024-02-01T10:48:06.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:R/xyYuuFAaegFnbq0SJjJejgA30=', 'Content-Length': '0'} 2024-02-01T10:48:06.119 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000020f9fde2f52ef017-0065bb76e6-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:06.160 DEBUG:boto:b'vclfcn-59obj21000falseobj22024-02-01T10:46:46.111Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:06.161 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:06.161 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:06.161 DEBUG:boto:path=/vclfcn-59/?prefix=obj2 2024-02-01T10:48:06.162 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj2 2024-02-01T10:48:06.162 DEBUG:boto:Method: GET 2024-02-01T10:48:06.162 DEBUG:boto:Path: /vclfcn-59/?prefix=obj2 2024-02-01T10:48:06.162 DEBUG:boto:Data: 2024-02-01T10:48:06.162 DEBUG:boto:Headers: {} 2024-02-01T10:48:06.163 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:06.163 DEBUG:boto:Port: 8000 2024-02-01T10:48:06.163 DEBUG:boto:Params: {} 2024-02-01T10:48:06.163 DEBUG:boto:Token: None 2024-02-01T10:48:06.163 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:06 GMT /vclfcn-59/ 2024-02-01T10:48:06.163 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:R/xyYuuFAaegFnbq0SJjJejgA30= 2024-02-01T10:48:06.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:R/xyYuuFAaegFnbq0SJjJejgA30=', 'Content-Length': '0'} 2024-02-01T10:48:06.168 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006bc3ed7c692b1245-0065bb76e6-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:06.168 DEBUG:boto:b'vclfcn-59obj21000falseobj22024-02-01T10:46:46.111Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:06.168 DEBUG:boto:path=/vclfcn-59/obj2 2024-02-01T10:48:06.168 DEBUG:boto:auth_path=/vclfcn-59/obj2 2024-02-01T10:48:06.169 DEBUG:boto:Method: HEAD 2024-02-01T10:48:06.169 DEBUG:boto:Path: /vclfcn-59/obj2 2024-02-01T10:48:06.169 DEBUG:boto:Data: 2024-02-01T10:48:06.169 DEBUG:boto:Headers: {} 2024-02-01T10:48:06.169 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:06.169 DEBUG:boto:Port: 8001 2024-02-01T10:48:06.169 DEBUG:boto:Params: {} 2024-02-01T10:48:06.169 DEBUG:boto:Token: None 2024-02-01T10:48:06.169 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:06 GMT /vclfcn-59/obj2 2024-02-01T10:48:06.169 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Y49R5dMPkUgB18TMaoXnGm+67tw= 2024-02-01T10:48:06.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Y49R5dMPkUgB18TMaoXnGm+67tw=', 'Content-Length': '0'} 2024-02-01T10:48:06.172 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000655996d5a4168a6a-0065bb76e6-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:06.173 DEBUG:boto:path=/vclfcn-59/obj2 2024-02-01T10:48:06.173 DEBUG:boto:auth_path=/vclfcn-59/obj2 2024-02-01T10:48:06.173 DEBUG:boto:Method: HEAD 2024-02-01T10:48:06.173 DEBUG:boto:Path: /vclfcn-59/obj2 2024-02-01T10:48:06.174 DEBUG:boto:Data: 2024-02-01T10:48:06.174 DEBUG:boto:Headers: {} 2024-02-01T10:48:06.174 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:06.174 DEBUG:boto:Port: 8000 2024-02-01T10:48:06.174 DEBUG:boto:Params: {} 2024-02-01T10:48:06.174 DEBUG:boto:Token: None 2024-02-01T10:48:06.174 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:06 GMT /vclfcn-59/obj2 2024-02-01T10:48:06.175 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Y49R5dMPkUgB18TMaoXnGm+67tw= 2024-02-01T10:48:06.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Y49R5dMPkUgB18TMaoXnGm+67tw=', 'Content-Length': '0'} 2024-02-01T10:48:06.179 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ea64f233e9226198-0065bb76e6-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:06.179 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:48:06.179 DEBUG:boto:path=/vclfcn-59/obj2 2024-02-01T10:48:06.180 DEBUG:boto:auth_path=/vclfcn-59/obj2 2024-02-01T10:48:06.180 DEBUG:boto:Method: GET 2024-02-01T10:48:06.180 DEBUG:boto:Path: /vclfcn-59/obj2 2024-02-01T10:48:06.180 DEBUG:boto:Data: 2024-02-01T10:48:06.180 DEBUG:boto:Headers: {} 2024-02-01T10:48:06.180 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:06.180 DEBUG:boto:Port: 8001 2024-02-01T10:48:06.180 DEBUG:boto:Params: {} 2024-02-01T10:48:06.180 DEBUG:boto:Token: None 2024-02-01T10:48:06.181 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:06 GMT /vclfcn-59/obj2 2024-02-01T10:48:06.181 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nvLuAVpOEFog09PtHkVkhML1Ozg= 2024-02-01T10:48:06.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nvLuAVpOEFog09PtHkVkhML1Ozg=', 'Content-Length': '0'} 2024-02-01T10:48:06.183 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c20d03b8c0add45d-0065bb76e6-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:06.224 DEBUG:boto:path=/vclfcn-59/obj2 2024-02-01T10:48:06.224 DEBUG:boto:auth_path=/vclfcn-59/obj2 2024-02-01T10:48:06.224 DEBUG:boto:Method: GET 2024-02-01T10:48:06.225 DEBUG:boto:Path: /vclfcn-59/obj2 2024-02-01T10:48:06.225 DEBUG:boto:Data: 2024-02-01T10:48:06.225 DEBUG:boto:Headers: {} 2024-02-01T10:48:06.225 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:06.225 DEBUG:boto:Port: 8000 2024-02-01T10:48:06.225 DEBUG:boto:Params: {} 2024-02-01T10:48:06.225 DEBUG:boto:Token: None 2024-02-01T10:48:06.226 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:06 GMT /vclfcn-59/obj2 2024-02-01T10:48:06.226 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nvLuAVpOEFog09PtHkVkhML1Ozg= 2024-02-01T10:48:06.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nvLuAVpOEFog09PtHkVkhML1Ozg=', 'Content-Length': '0'} 2024-02-01T10:48:06.229 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a16aec1c5fbd46da-0065bb76e6-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:06.276 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:06.276 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:06.277 DEBUG:boto:path=/vclfcn-59/?prefix=obj2 2024-02-01T10:48:06.277 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj2 2024-02-01T10:48:06.277 DEBUG:boto:Method: GET 2024-02-01T10:48:06.277 DEBUG:boto:Path: /vclfcn-59/?prefix=obj2 2024-02-01T10:48:06.277 DEBUG:boto:Data: 2024-02-01T10:48:06.277 DEBUG:boto:Headers: {} 2024-02-01T10:48:06.277 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:06.277 DEBUG:boto:Port: 8001 2024-02-01T10:48:06.278 DEBUG:boto:Params: {} 2024-02-01T10:48:06.278 DEBUG:boto:Token: None 2024-02-01T10:48:06.278 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:06 GMT /vclfcn-59/ 2024-02-01T10:48:06.278 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:R/xyYuuFAaegFnbq0SJjJejgA30= 2024-02-01T10:48:06.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:R/xyYuuFAaegFnbq0SJjJejgA30=', 'Content-Length': '0'} 2024-02-01T10:48:06.283 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035b476935ed87b85-0065bb76e6-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:06.324 DEBUG:boto:b'vclfcn-59obj21000falseobj22024-02-01T10:46:46.111Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:06.325 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:06.325 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:06.325 DEBUG:boto:path=/vclfcn-59/?prefix=obj2 2024-02-01T10:48:06.325 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj2 2024-02-01T10:48:06.325 DEBUG:boto:Method: GET 2024-02-01T10:48:06.325 DEBUG:boto:Path: /vclfcn-59/?prefix=obj2 2024-02-01T10:48:06.325 DEBUG:boto:Data: 2024-02-01T10:48:06.325 DEBUG:boto:Headers: {} 2024-02-01T10:48:06.326 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:06.326 DEBUG:boto:Port: 8000 2024-02-01T10:48:06.326 DEBUG:boto:Params: {} 2024-02-01T10:48:06.326 DEBUG:boto:Token: None 2024-02-01T10:48:06.326 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:06 GMT /vclfcn-59/ 2024-02-01T10:48:06.326 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:R/xyYuuFAaegFnbq0SJjJejgA30= 2024-02-01T10:48:06.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:R/xyYuuFAaegFnbq0SJjJejgA30=', 'Content-Length': '0'} 2024-02-01T10:48:06.331 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000add0ad83071ef058-0065bb76e6-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:06.331 DEBUG:boto:b'vclfcn-59obj21000falseobj22024-02-01T10:46:46.111Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:06.331 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:48:06.332 DEBUG:boto:path=/vclfcn-59/obj3 2024-02-01T10:48:06.332 DEBUG:boto:auth_path=/vclfcn-59/obj3 2024-02-01T10:48:06.332 DEBUG:boto:path=/vclfcn-59/obj3?versionId=null 2024-02-01T10:48:06.332 DEBUG:boto:auth_path=/vclfcn-59/obj3?versionId=null 2024-02-01T10:48:06.332 DEBUG:boto:Method: GET 2024-02-01T10:48:06.332 DEBUG:boto:Path: /vclfcn-59/obj3?versionId=null 2024-02-01T10:48:06.332 DEBUG:boto:Data: 2024-02-01T10:48:06.333 DEBUG:boto:Headers: {} 2024-02-01T10:48:06.333 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:06.333 DEBUG:boto:Port: 8001 2024-02-01T10:48:06.333 DEBUG:boto:Params: {} 2024-02-01T10:48:06.333 DEBUG:boto:Token: None 2024-02-01T10:48:06.333 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:06 GMT /vclfcn-59/obj3?versionId=null 2024-02-01T10:48:06.333 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:dzirF59gi2p0sxu4QviZ8xHyzk8= 2024-02-01T10:48:06.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dzirF59gi2p0sxu4QviZ8xHyzk8=', 'Content-Length': '0'} 2024-02-01T10:48:06.337 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bc103f9bad420530-0065bb76e6-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:06.380 DEBUG:boto:path=/vclfcn-59/obj3 2024-02-01T10:48:06.380 DEBUG:boto:auth_path=/vclfcn-59/obj3 2024-02-01T10:48:06.380 DEBUG:boto:path=/vclfcn-59/obj3?versionId=null 2024-02-01T10:48:06.380 DEBUG:boto:auth_path=/vclfcn-59/obj3?versionId=null 2024-02-01T10:48:06.381 DEBUG:boto:Method: GET 2024-02-01T10:48:06.381 DEBUG:boto:Path: /vclfcn-59/obj3?versionId=null 2024-02-01T10:48:06.381 DEBUG:boto:Data: 2024-02-01T10:48:06.381 DEBUG:boto:Headers: {} 2024-02-01T10:48:06.381 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:06.381 DEBUG:boto:Port: 8000 2024-02-01T10:48:06.381 DEBUG:boto:Params: {} 2024-02-01T10:48:06.381 DEBUG:boto:Token: None 2024-02-01T10:48:06.381 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:06 GMT /vclfcn-59/obj3?versionId=null 2024-02-01T10:48:06.382 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:dzirF59gi2p0sxu4QviZ8xHyzk8= 2024-02-01T10:48:06.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dzirF59gi2p0sxu4QviZ8xHyzk8=', 'Content-Length': '0'} 2024-02-01T10:48:06.384 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a5103dde865d7451-0065bb76e6-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:06.385 DEBUG:boto:path=/vclfcn-59/obj3 2024-02-01T10:48:06.385 DEBUG:boto:auth_path=/vclfcn-59/obj3 2024-02-01T10:48:06.385 DEBUG:boto:path=/vclfcn-59/obj3?versionId=null 2024-02-01T10:48:06.385 DEBUG:boto:auth_path=/vclfcn-59/obj3?versionId=null 2024-02-01T10:48:06.386 DEBUG:boto:Method: HEAD 2024-02-01T10:48:06.386 DEBUG:boto:Path: /vclfcn-59/obj3?versionId=null 2024-02-01T10:48:06.386 DEBUG:boto:Data: 2024-02-01T10:48:06.386 DEBUG:boto:Headers: {} 2024-02-01T10:48:06.386 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:06.386 DEBUG:boto:Port: 8001 2024-02-01T10:48:06.386 DEBUG:boto:Params: {} 2024-02-01T10:48:06.386 DEBUG:boto:Token: None 2024-02-01T10:48:06.386 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:06 GMT /vclfcn-59/obj3?versionId=null 2024-02-01T10:48:06.387 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Mg2Uo/w+kjXKVElGK6JQ1D7T7p4= 2024-02-01T10:48:06.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Mg2Uo/w+kjXKVElGK6JQ1D7T7p4=', 'Content-Length': '0'} 2024-02-01T10:48:06.389 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f18d1292834c1fc0-0065bb76e6-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:06.390 DEBUG:boto:path=/vclfcn-59/obj3 2024-02-01T10:48:06.390 DEBUG:boto:auth_path=/vclfcn-59/obj3 2024-02-01T10:48:06.390 DEBUG:boto:path=/vclfcn-59/obj3?versionId=null 2024-02-01T10:48:06.390 DEBUG:boto:auth_path=/vclfcn-59/obj3?versionId=null 2024-02-01T10:48:06.390 DEBUG:boto:Method: HEAD 2024-02-01T10:48:06.390 DEBUG:boto:Path: /vclfcn-59/obj3?versionId=null 2024-02-01T10:48:06.390 DEBUG:boto:Data: 2024-02-01T10:48:06.390 DEBUG:boto:Headers: {} 2024-02-01T10:48:06.391 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:06.391 DEBUG:boto:Port: 8000 2024-02-01T10:48:06.391 DEBUG:boto:Params: {} 2024-02-01T10:48:06.391 DEBUG:boto:Token: None 2024-02-01T10:48:06.391 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:06 GMT /vclfcn-59/obj3?versionId=null 2024-02-01T10:48:06.391 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Mg2Uo/w+kjXKVElGK6JQ1D7T7p4= 2024-02-01T10:48:06.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Mg2Uo/w+kjXKVElGK6JQ1D7T7p4=', 'Content-Length': '0'} 2024-02-01T10:48:06.394 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e8dae333518c86b9-0065bb76e6-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:06.394 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:48:06.395 DEBUG:boto:path=/vclfcn-59/obj3 2024-02-01T10:48:06.395 DEBUG:boto:auth_path=/vclfcn-59/obj3 2024-02-01T10:48:06.395 DEBUG:boto:path=/vclfcn-59/obj3?versionId=null 2024-02-01T10:48:06.395 DEBUG:boto:auth_path=/vclfcn-59/obj3?versionId=null 2024-02-01T10:48:06.395 DEBUG:boto:Method: GET 2024-02-01T10:48:06.395 DEBUG:boto:Path: /vclfcn-59/obj3?versionId=null 2024-02-01T10:48:06.395 DEBUG:boto:Data: 2024-02-01T10:48:06.395 DEBUG:boto:Headers: {} 2024-02-01T10:48:06.395 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:06.396 DEBUG:boto:Port: 8001 2024-02-01T10:48:06.396 DEBUG:boto:Params: {} 2024-02-01T10:48:06.396 DEBUG:boto:Token: None 2024-02-01T10:48:06.396 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:06 GMT /vclfcn-59/obj3?versionId=null 2024-02-01T10:48:06.396 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:dzirF59gi2p0sxu4QviZ8xHyzk8= 2024-02-01T10:48:06.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dzirF59gi2p0sxu4QviZ8xHyzk8=', 'Content-Length': '0'} 2024-02-01T10:48:06.399 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000075d000e37e9773f-0065bb76e6-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:06.440 DEBUG:boto:path=/vclfcn-59/obj3 2024-02-01T10:48:06.440 DEBUG:boto:auth_path=/vclfcn-59/obj3 2024-02-01T10:48:06.440 DEBUG:boto:path=/vclfcn-59/obj3?versionId=null 2024-02-01T10:48:06.441 DEBUG:boto:auth_path=/vclfcn-59/obj3?versionId=null 2024-02-01T10:48:06.441 DEBUG:boto:Method: GET 2024-02-01T10:48:06.441 DEBUG:boto:Path: /vclfcn-59/obj3?versionId=null 2024-02-01T10:48:06.441 DEBUG:boto:Data: 2024-02-01T10:48:06.441 DEBUG:boto:Headers: {} 2024-02-01T10:48:06.441 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:06.441 DEBUG:boto:Port: 8000 2024-02-01T10:48:06.441 DEBUG:boto:Params: {} 2024-02-01T10:48:06.441 DEBUG:boto:Token: None 2024-02-01T10:48:06.441 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:06 GMT /vclfcn-59/obj3?versionId=null 2024-02-01T10:48:06.442 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:dzirF59gi2p0sxu4QviZ8xHyzk8= 2024-02-01T10:48:06.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dzirF59gi2p0sxu4QviZ8xHyzk8=', 'Content-Length': '0'} 2024-02-01T10:48:06.445 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e2b5e2b09b979333-0065bb76e6-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:06.488 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:06.488 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:06.488 DEBUG:boto:path=/vclfcn-59/?prefix=obj3 2024-02-01T10:48:06.489 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj3 2024-02-01T10:48:06.489 DEBUG:boto:Method: GET 2024-02-01T10:48:06.489 DEBUG:boto:Path: /vclfcn-59/?prefix=obj3 2024-02-01T10:48:06.489 DEBUG:boto:Data: 2024-02-01T10:48:06.489 DEBUG:boto:Headers: {} 2024-02-01T10:48:06.489 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:06.489 DEBUG:boto:Port: 8001 2024-02-01T10:48:06.489 DEBUG:boto:Params: {} 2024-02-01T10:48:06.489 DEBUG:boto:Token: None 2024-02-01T10:48:06.489 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:06 GMT /vclfcn-59/ 2024-02-01T10:48:06.490 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:R/xyYuuFAaegFnbq0SJjJejgA30= 2024-02-01T10:48:06.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:R/xyYuuFAaegFnbq0SJjJejgA30=', 'Content-Length': '0'} 2024-02-01T10:48:06.494 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7858da5441919b7-0065bb76e6-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:06.494 DEBUG:boto:b'vclfcn-59obj31000falseobj32024-02-01T10:46:46.161Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:06.494 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:06.494 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:06.495 DEBUG:boto:path=/vclfcn-59/?prefix=obj3 2024-02-01T10:48:06.495 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj3 2024-02-01T10:48:06.495 DEBUG:boto:Method: GET 2024-02-01T10:48:06.495 DEBUG:boto:Path: /vclfcn-59/?prefix=obj3 2024-02-01T10:48:06.495 DEBUG:boto:Data: 2024-02-01T10:48:06.495 DEBUG:boto:Headers: {} 2024-02-01T10:48:06.495 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:06.495 DEBUG:boto:Port: 8000 2024-02-01T10:48:06.495 DEBUG:boto:Params: {} 2024-02-01T10:48:06.495 DEBUG:boto:Token: None 2024-02-01T10:48:06.495 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:06 GMT /vclfcn-59/ 2024-02-01T10:48:06.496 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:R/xyYuuFAaegFnbq0SJjJejgA30= 2024-02-01T10:48:06.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:R/xyYuuFAaegFnbq0SJjJejgA30=', 'Content-Length': '0'} 2024-02-01T10:48:06.500 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b9c9e9eb8713ec55-0065bb76e6-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:06.501 DEBUG:boto:b'vclfcn-59obj31000falseobj32024-02-01T10:46:46.161Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:06.501 DEBUG:boto:path=/vclfcn-59/obj3 2024-02-01T10:48:06.501 DEBUG:boto:auth_path=/vclfcn-59/obj3 2024-02-01T10:48:06.501 DEBUG:boto:Method: HEAD 2024-02-01T10:48:06.501 DEBUG:boto:Path: /vclfcn-59/obj3 2024-02-01T10:48:06.501 DEBUG:boto:Data: 2024-02-01T10:48:06.501 DEBUG:boto:Headers: {} 2024-02-01T10:48:06.501 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:06.502 DEBUG:boto:Port: 8001 2024-02-01T10:48:06.502 DEBUG:boto:Params: {} 2024-02-01T10:48:06.502 DEBUG:boto:Token: None 2024-02-01T10:48:06.502 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:06 GMT /vclfcn-59/obj3 2024-02-01T10:48:06.502 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5oli8QMu81mNC5wuUkHRQa5cqrQ= 2024-02-01T10:48:06.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5oli8QMu81mNC5wuUkHRQa5cqrQ=', 'Content-Length': '0'} 2024-02-01T10:48:06.505 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002a018662ec363000-0065bb76e6-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:06.505 DEBUG:boto:path=/vclfcn-59/obj3 2024-02-01T10:48:06.505 DEBUG:boto:auth_path=/vclfcn-59/obj3 2024-02-01T10:48:06.505 DEBUG:boto:Method: HEAD 2024-02-01T10:48:06.505 DEBUG:boto:Path: /vclfcn-59/obj3 2024-02-01T10:48:06.505 DEBUG:boto:Data: 2024-02-01T10:48:06.506 DEBUG:boto:Headers: {} 2024-02-01T10:48:06.506 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:06.506 DEBUG:boto:Port: 8000 2024-02-01T10:48:06.506 DEBUG:boto:Params: {} 2024-02-01T10:48:06.506 DEBUG:boto:Token: None 2024-02-01T10:48:06.506 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:06 GMT /vclfcn-59/obj3 2024-02-01T10:48:06.506 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5oli8QMu81mNC5wuUkHRQa5cqrQ= 2024-02-01T10:48:06.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5oli8QMu81mNC5wuUkHRQa5cqrQ=', 'Content-Length': '0'} 2024-02-01T10:48:06.509 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c30af4bd7c6b0fd4-0065bb76e6-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:06.509 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:48:06.509 DEBUG:boto:path=/vclfcn-59/obj3 2024-02-01T10:48:06.509 DEBUG:boto:auth_path=/vclfcn-59/obj3 2024-02-01T10:48:06.509 DEBUG:boto:Method: GET 2024-02-01T10:48:06.510 DEBUG:boto:Path: /vclfcn-59/obj3 2024-02-01T10:48:06.510 DEBUG:boto:Data: 2024-02-01T10:48:06.510 DEBUG:boto:Headers: {} 2024-02-01T10:48:06.510 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:06.510 DEBUG:boto:Port: 8001 2024-02-01T10:48:06.510 DEBUG:boto:Params: {} 2024-02-01T10:48:06.510 DEBUG:boto:Token: None 2024-02-01T10:48:06.510 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:06 GMT /vclfcn-59/obj3 2024-02-01T10:48:06.510 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jzg64YbV/eOGoA2i+GNA4iEuiLQ= 2024-02-01T10:48:06.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jzg64YbV/eOGoA2i+GNA4iEuiLQ=', 'Content-Length': '0'} 2024-02-01T10:48:06.513 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cb3ef9e5fa184f73-0065bb76e6-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:06.556 DEBUG:boto:path=/vclfcn-59/obj3 2024-02-01T10:48:06.556 DEBUG:boto:auth_path=/vclfcn-59/obj3 2024-02-01T10:48:06.557 DEBUG:boto:Method: GET 2024-02-01T10:48:06.557 DEBUG:boto:Path: /vclfcn-59/obj3 2024-02-01T10:48:06.557 DEBUG:boto:Data: 2024-02-01T10:48:06.557 DEBUG:boto:Headers: {} 2024-02-01T10:48:06.557 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:06.557 DEBUG:boto:Port: 8000 2024-02-01T10:48:06.557 DEBUG:boto:Params: {} 2024-02-01T10:48:06.557 DEBUG:boto:Token: None 2024-02-01T10:48:06.557 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:06 GMT /vclfcn-59/obj3 2024-02-01T10:48:06.557 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jzg64YbV/eOGoA2i+GNA4iEuiLQ= 2024-02-01T10:48:06.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jzg64YbV/eOGoA2i+GNA4iEuiLQ=', 'Content-Length': '0'} 2024-02-01T10:48:06.560 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eb1b6945fccf0387-0065bb76e6-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:06.604 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:06.604 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:06.604 DEBUG:boto:path=/vclfcn-59/?prefix=obj3 2024-02-01T10:48:06.605 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj3 2024-02-01T10:48:06.605 DEBUG:boto:Method: GET 2024-02-01T10:48:06.605 DEBUG:boto:Path: /vclfcn-59/?prefix=obj3 2024-02-01T10:48:06.605 DEBUG:boto:Data: 2024-02-01T10:48:06.605 DEBUG:boto:Headers: {} 2024-02-01T10:48:06.605 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:06.605 DEBUG:boto:Port: 8001 2024-02-01T10:48:06.605 DEBUG:boto:Params: {} 2024-02-01T10:48:06.606 DEBUG:boto:Token: None 2024-02-01T10:48:06.606 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:06 GMT /vclfcn-59/ 2024-02-01T10:48:06.606 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:R/xyYuuFAaegFnbq0SJjJejgA30= 2024-02-01T10:48:06.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:R/xyYuuFAaegFnbq0SJjJejgA30=', 'Content-Length': '0'} 2024-02-01T10:48:06.610 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee3b5010b7215cc4-0065bb76e6-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:06.652 DEBUG:boto:b'vclfcn-59obj31000falseobj32024-02-01T10:46:46.161Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:06.653 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:06.653 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:06.653 DEBUG:boto:path=/vclfcn-59/?prefix=obj3 2024-02-01T10:48:06.653 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj3 2024-02-01T10:48:06.653 DEBUG:boto:Method: GET 2024-02-01T10:48:06.653 DEBUG:boto:Path: /vclfcn-59/?prefix=obj3 2024-02-01T10:48:06.653 DEBUG:boto:Data: 2024-02-01T10:48:06.653 DEBUG:boto:Headers: {} 2024-02-01T10:48:06.653 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:06.653 DEBUG:boto:Port: 8000 2024-02-01T10:48:06.654 DEBUG:boto:Params: {} 2024-02-01T10:48:06.654 DEBUG:boto:Token: None 2024-02-01T10:48:06.654 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:06 GMT /vclfcn-59/ 2024-02-01T10:48:06.654 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:R/xyYuuFAaegFnbq0SJjJejgA30= 2024-02-01T10:48:06.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:R/xyYuuFAaegFnbq0SJjJejgA30=', 'Content-Length': '0'} 2024-02-01T10:48:06.658 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030354369b0e52dbd-0065bb76e6-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:06.700 DEBUG:boto:b'vclfcn-59obj31000falseobj32024-02-01T10:46:46.161Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:06.700 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:48:06.701 DEBUG:boto:path=/vclfcn-59/obj4 2024-02-01T10:48:06.701 DEBUG:boto:auth_path=/vclfcn-59/obj4 2024-02-01T10:48:06.701 DEBUG:boto:path=/vclfcn-59/obj4?versionId=null 2024-02-01T10:48:06.701 DEBUG:boto:auth_path=/vclfcn-59/obj4?versionId=null 2024-02-01T10:48:06.701 DEBUG:boto:Method: GET 2024-02-01T10:48:06.701 DEBUG:boto:Path: /vclfcn-59/obj4?versionId=null 2024-02-01T10:48:06.701 DEBUG:boto:Data: 2024-02-01T10:48:06.701 DEBUG:boto:Headers: {} 2024-02-01T10:48:06.701 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:06.701 DEBUG:boto:Port: 8001 2024-02-01T10:48:06.701 DEBUG:boto:Params: {} 2024-02-01T10:48:06.702 DEBUG:boto:Token: None 2024-02-01T10:48:06.702 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:06 GMT /vclfcn-59/obj4?versionId=null 2024-02-01T10:48:06.702 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YtrkiJ+R7LD9TLJ47v87dmFQ+2Y= 2024-02-01T10:48:06.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YtrkiJ+R7LD9TLJ47v87dmFQ+2Y=', 'Content-Length': '0'} 2024-02-01T10:48:06.705 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000253ced31f0fdcff2-0065bb76e6-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:06.748 DEBUG:boto:path=/vclfcn-59/obj4 2024-02-01T10:48:06.748 DEBUG:boto:auth_path=/vclfcn-59/obj4 2024-02-01T10:48:06.748 DEBUG:boto:path=/vclfcn-59/obj4?versionId=null 2024-02-01T10:48:06.748 DEBUG:boto:auth_path=/vclfcn-59/obj4?versionId=null 2024-02-01T10:48:06.748 DEBUG:boto:Method: GET 2024-02-01T10:48:06.749 DEBUG:boto:Path: /vclfcn-59/obj4?versionId=null 2024-02-01T10:48:06.749 DEBUG:boto:Data: 2024-02-01T10:48:06.749 DEBUG:boto:Headers: {} 2024-02-01T10:48:06.749 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:06.749 DEBUG:boto:Port: 8000 2024-02-01T10:48:06.749 DEBUG:boto:Params: {} 2024-02-01T10:48:06.749 DEBUG:boto:Token: None 2024-02-01T10:48:06.749 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:06 GMT /vclfcn-59/obj4?versionId=null 2024-02-01T10:48:06.749 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YtrkiJ+R7LD9TLJ47v87dmFQ+2Y= 2024-02-01T10:48:06.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YtrkiJ+R7LD9TLJ47v87dmFQ+2Y=', 'Content-Length': '0'} 2024-02-01T10:48:06.753 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e85b3fe5dc16a94f-0065bb76e6-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:06.753 DEBUG:boto:path=/vclfcn-59/obj4 2024-02-01T10:48:06.753 DEBUG:boto:auth_path=/vclfcn-59/obj4 2024-02-01T10:48:06.753 DEBUG:boto:path=/vclfcn-59/obj4?versionId=null 2024-02-01T10:48:06.753 DEBUG:boto:auth_path=/vclfcn-59/obj4?versionId=null 2024-02-01T10:48:06.753 DEBUG:boto:Method: HEAD 2024-02-01T10:48:06.753 DEBUG:boto:Path: /vclfcn-59/obj4?versionId=null 2024-02-01T10:48:06.754 DEBUG:boto:Data: 2024-02-01T10:48:06.754 DEBUG:boto:Headers: {} 2024-02-01T10:48:06.754 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:06.754 DEBUG:boto:Port: 8001 2024-02-01T10:48:06.754 DEBUG:boto:Params: {} 2024-02-01T10:48:06.754 DEBUG:boto:Token: None 2024-02-01T10:48:06.754 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:06 GMT /vclfcn-59/obj4?versionId=null 2024-02-01T10:48:06.754 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:SDTgE2kshxHnuteZ3plYPCNG/xY= 2024-02-01T10:48:06.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:SDTgE2kshxHnuteZ3plYPCNG/xY=', 'Content-Length': '0'} 2024-02-01T10:48:06.757 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f38a7580c02ac63c-0065bb76e6-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:06.758 DEBUG:boto:path=/vclfcn-59/obj4 2024-02-01T10:48:06.758 DEBUG:boto:auth_path=/vclfcn-59/obj4 2024-02-01T10:48:06.758 DEBUG:boto:path=/vclfcn-59/obj4?versionId=null 2024-02-01T10:48:06.758 DEBUG:boto:auth_path=/vclfcn-59/obj4?versionId=null 2024-02-01T10:48:06.758 DEBUG:boto:Method: HEAD 2024-02-01T10:48:06.758 DEBUG:boto:Path: /vclfcn-59/obj4?versionId=null 2024-02-01T10:48:06.758 DEBUG:boto:Data: 2024-02-01T10:48:06.758 DEBUG:boto:Headers: {} 2024-02-01T10:48:06.758 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:06.758 DEBUG:boto:Port: 8000 2024-02-01T10:48:06.758 DEBUG:boto:Params: {} 2024-02-01T10:48:06.759 DEBUG:boto:Token: None 2024-02-01T10:48:06.759 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:06 GMT /vclfcn-59/obj4?versionId=null 2024-02-01T10:48:06.759 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:SDTgE2kshxHnuteZ3plYPCNG/xY= 2024-02-01T10:48:06.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:SDTgE2kshxHnuteZ3plYPCNG/xY=', 'Content-Length': '0'} 2024-02-01T10:48:06.762 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000faa607d6ea051c62-0065bb76e6-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:06.762 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:48:06.762 DEBUG:boto:path=/vclfcn-59/obj4 2024-02-01T10:48:06.762 DEBUG:boto:auth_path=/vclfcn-59/obj4 2024-02-01T10:48:06.762 DEBUG:boto:path=/vclfcn-59/obj4?versionId=null 2024-02-01T10:48:06.762 DEBUG:boto:auth_path=/vclfcn-59/obj4?versionId=null 2024-02-01T10:48:06.762 DEBUG:boto:Method: GET 2024-02-01T10:48:06.763 DEBUG:boto:Path: /vclfcn-59/obj4?versionId=null 2024-02-01T10:48:06.763 DEBUG:boto:Data: 2024-02-01T10:48:06.763 DEBUG:boto:Headers: {} 2024-02-01T10:48:06.763 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:06.763 DEBUG:boto:Port: 8001 2024-02-01T10:48:06.763 DEBUG:boto:Params: {} 2024-02-01T10:48:06.763 DEBUG:boto:Token: None 2024-02-01T10:48:06.763 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:06 GMT /vclfcn-59/obj4?versionId=null 2024-02-01T10:48:06.763 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YtrkiJ+R7LD9TLJ47v87dmFQ+2Y= 2024-02-01T10:48:06.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YtrkiJ+R7LD9TLJ47v87dmFQ+2Y=', 'Content-Length': '0'} 2024-02-01T10:48:06.766 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006798bc0d9a62f6e8-0065bb76e6-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:06.808 DEBUG:boto:path=/vclfcn-59/obj4 2024-02-01T10:48:06.808 DEBUG:boto:auth_path=/vclfcn-59/obj4 2024-02-01T10:48:06.808 DEBUG:boto:path=/vclfcn-59/obj4?versionId=null 2024-02-01T10:48:06.808 DEBUG:boto:auth_path=/vclfcn-59/obj4?versionId=null 2024-02-01T10:48:06.809 DEBUG:boto:Method: GET 2024-02-01T10:48:06.809 DEBUG:boto:Path: /vclfcn-59/obj4?versionId=null 2024-02-01T10:48:06.809 DEBUG:boto:Data: 2024-02-01T10:48:06.809 DEBUG:boto:Headers: {} 2024-02-01T10:48:06.809 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:06.809 DEBUG:boto:Port: 8000 2024-02-01T10:48:06.809 DEBUG:boto:Params: {} 2024-02-01T10:48:06.809 DEBUG:boto:Token: None 2024-02-01T10:48:06.809 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:06 GMT /vclfcn-59/obj4?versionId=null 2024-02-01T10:48:06.809 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YtrkiJ+R7LD9TLJ47v87dmFQ+2Y= 2024-02-01T10:48:06.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YtrkiJ+R7LD9TLJ47v87dmFQ+2Y=', 'Content-Length': '0'} 2024-02-01T10:48:06.812 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004200b42370634569-0065bb76e6-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:06.856 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:06.856 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:06.856 DEBUG:boto:path=/vclfcn-59/?prefix=obj4 2024-02-01T10:48:06.857 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj4 2024-02-01T10:48:06.857 DEBUG:boto:Method: GET 2024-02-01T10:48:06.857 DEBUG:boto:Path: /vclfcn-59/?prefix=obj4 2024-02-01T10:48:06.857 DEBUG:boto:Data: 2024-02-01T10:48:06.857 DEBUG:boto:Headers: {} 2024-02-01T10:48:06.857 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:06.857 DEBUG:boto:Port: 8001 2024-02-01T10:48:06.857 DEBUG:boto:Params: {} 2024-02-01T10:48:06.857 DEBUG:boto:Token: None 2024-02-01T10:48:06.857 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:06 GMT /vclfcn-59/ 2024-02-01T10:48:06.858 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:R/xyYuuFAaegFnbq0SJjJejgA30= 2024-02-01T10:48:06.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:R/xyYuuFAaegFnbq0SJjJejgA30=', 'Content-Length': '0'} 2024-02-01T10:48:06.862 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bdde21228d2d4d10-0065bb76e6-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:06.904 DEBUG:boto:b'vclfcn-59obj41000falseobj42024-02-01T10:46:46.190Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:06.905 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:06.905 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:06.905 DEBUG:boto:path=/vclfcn-59/?prefix=obj4 2024-02-01T10:48:06.905 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj4 2024-02-01T10:48:06.905 DEBUG:boto:Method: GET 2024-02-01T10:48:06.905 DEBUG:boto:Path: /vclfcn-59/?prefix=obj4 2024-02-01T10:48:06.905 DEBUG:boto:Data: 2024-02-01T10:48:06.905 DEBUG:boto:Headers: {} 2024-02-01T10:48:06.905 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:06.905 DEBUG:boto:Port: 8000 2024-02-01T10:48:06.905 DEBUG:boto:Params: {} 2024-02-01T10:48:06.906 DEBUG:boto:Token: None 2024-02-01T10:48:06.906 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:06 GMT /vclfcn-59/ 2024-02-01T10:48:06.906 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:R/xyYuuFAaegFnbq0SJjJejgA30= 2024-02-01T10:48:06.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:R/xyYuuFAaegFnbq0SJjJejgA30=', 'Content-Length': '0'} 2024-02-01T10:48:06.911 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f14080bbbeae62f-0065bb76e6-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:06.911 DEBUG:boto:b'vclfcn-59obj41000falseobj42024-02-01T10:46:46.190Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:06.911 DEBUG:boto:path=/vclfcn-59/obj4 2024-02-01T10:48:06.912 DEBUG:boto:auth_path=/vclfcn-59/obj4 2024-02-01T10:48:06.912 DEBUG:boto:Method: HEAD 2024-02-01T10:48:06.912 DEBUG:boto:Path: /vclfcn-59/obj4 2024-02-01T10:48:06.912 DEBUG:boto:Data: 2024-02-01T10:48:06.912 DEBUG:boto:Headers: {} 2024-02-01T10:48:06.912 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:06.912 DEBUG:boto:Port: 8001 2024-02-01T10:48:06.912 DEBUG:boto:Params: {} 2024-02-01T10:48:06.912 DEBUG:boto:Token: None 2024-02-01T10:48:06.912 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:06 GMT /vclfcn-59/obj4 2024-02-01T10:48:06.913 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NrX+WzdFXB1BTV9vE3v4ehskwPQ= 2024-02-01T10:48:06.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NrX+WzdFXB1BTV9vE3v4ehskwPQ=', 'Content-Length': '0'} 2024-02-01T10:48:06.916 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009695ed8bbfe7b479-0065bb76e6-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:06.916 DEBUG:boto:path=/vclfcn-59/obj4 2024-02-01T10:48:06.916 DEBUG:boto:auth_path=/vclfcn-59/obj4 2024-02-01T10:48:06.916 DEBUG:boto:Method: HEAD 2024-02-01T10:48:06.916 DEBUG:boto:Path: /vclfcn-59/obj4 2024-02-01T10:48:06.916 DEBUG:boto:Data: 2024-02-01T10:48:06.916 DEBUG:boto:Headers: {} 2024-02-01T10:48:06.916 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:06.917 DEBUG:boto:Port: 8000 2024-02-01T10:48:06.917 DEBUG:boto:Params: {} 2024-02-01T10:48:06.917 DEBUG:boto:Token: None 2024-02-01T10:48:06.917 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:06 GMT /vclfcn-59/obj4 2024-02-01T10:48:06.917 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NrX+WzdFXB1BTV9vE3v4ehskwPQ= 2024-02-01T10:48:06.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NrX+WzdFXB1BTV9vE3v4ehskwPQ=', 'Content-Length': '0'} 2024-02-01T10:48:06.920 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002d17d4303a78976c-0065bb76e6-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:06.920 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:48:06.920 DEBUG:boto:path=/vclfcn-59/obj4 2024-02-01T10:48:06.920 DEBUG:boto:auth_path=/vclfcn-59/obj4 2024-02-01T10:48:06.920 DEBUG:boto:Method: GET 2024-02-01T10:48:06.921 DEBUG:boto:Path: /vclfcn-59/obj4 2024-02-01T10:48:06.921 DEBUG:boto:Data: 2024-02-01T10:48:06.921 DEBUG:boto:Headers: {} 2024-02-01T10:48:06.921 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:06.921 DEBUG:boto:Port: 8001 2024-02-01T10:48:06.921 DEBUG:boto:Params: {} 2024-02-01T10:48:06.921 DEBUG:boto:Token: None 2024-02-01T10:48:06.921 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:06 GMT /vclfcn-59/obj4 2024-02-01T10:48:06.921 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gW6tAF97RCe3TaoGoPR4/M/ba7Y= 2024-02-01T10:48:06.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gW6tAF97RCe3TaoGoPR4/M/ba7Y=', 'Content-Length': '0'} 2024-02-01T10:48:06.924 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000095d6cd8abb47f06b-0065bb76e6-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:06.968 DEBUG:boto:path=/vclfcn-59/obj4 2024-02-01T10:48:06.968 DEBUG:boto:auth_path=/vclfcn-59/obj4 2024-02-01T10:48:06.968 DEBUG:boto:Method: GET 2024-02-01T10:48:06.968 DEBUG:boto:Path: /vclfcn-59/obj4 2024-02-01T10:48:06.969 DEBUG:boto:Data: 2024-02-01T10:48:06.969 DEBUG:boto:Headers: {} 2024-02-01T10:48:06.969 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:06.969 DEBUG:boto:Port: 8000 2024-02-01T10:48:06.969 DEBUG:boto:Params: {} 2024-02-01T10:48:06.969 DEBUG:boto:Token: None 2024-02-01T10:48:06.969 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:06 GMT /vclfcn-59/obj4 2024-02-01T10:48:06.970 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gW6tAF97RCe3TaoGoPR4/M/ba7Y= 2024-02-01T10:48:06.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gW6tAF97RCe3TaoGoPR4/M/ba7Y=', 'Content-Length': '0'} 2024-02-01T10:48:06.973 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007530eca5693ed8ca-0065bb76e6-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:07.016 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:07.016 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:07.016 DEBUG:boto:path=/vclfcn-59/?prefix=obj4 2024-02-01T10:48:07.016 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj4 2024-02-01T10:48:07.017 DEBUG:boto:Method: GET 2024-02-01T10:48:07.017 DEBUG:boto:Path: /vclfcn-59/?prefix=obj4 2024-02-01T10:48:07.017 DEBUG:boto:Data: 2024-02-01T10:48:07.017 DEBUG:boto:Headers: {} 2024-02-01T10:48:07.017 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:07.017 DEBUG:boto:Port: 8001 2024-02-01T10:48:07.017 DEBUG:boto:Params: {} 2024-02-01T10:48:07.017 DEBUG:boto:Token: None 2024-02-01T10:48:07.017 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:07 GMT /vclfcn-59/ 2024-02-01T10:48:07.018 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NwdGjBpCSV0xKw6akevMJFGrGXM= 2024-02-01T10:48:07.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NwdGjBpCSV0xKw6akevMJFGrGXM=', 'Content-Length': '0'} 2024-02-01T10:48:07.022 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000920d6b4015c521da-0065bb76e7-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:07.022 DEBUG:boto:b'vclfcn-59obj41000falseobj42024-02-01T10:46:46.190Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:07.022 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:07.022 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:07.023 DEBUG:boto:path=/vclfcn-59/?prefix=obj4 2024-02-01T10:48:07.023 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj4 2024-02-01T10:48:07.023 DEBUG:boto:Method: GET 2024-02-01T10:48:07.023 DEBUG:boto:Path: /vclfcn-59/?prefix=obj4 2024-02-01T10:48:07.023 DEBUG:boto:Data: 2024-02-01T10:48:07.023 DEBUG:boto:Headers: {} 2024-02-01T10:48:07.023 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:07.023 DEBUG:boto:Port: 8000 2024-02-01T10:48:07.023 DEBUG:boto:Params: {} 2024-02-01T10:48:07.023 DEBUG:boto:Token: None 2024-02-01T10:48:07.024 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:07 GMT /vclfcn-59/ 2024-02-01T10:48:07.024 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NwdGjBpCSV0xKw6akevMJFGrGXM= 2024-02-01T10:48:07.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NwdGjBpCSV0xKw6akevMJFGrGXM=', 'Content-Length': '0'} 2024-02-01T10:48:07.028 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd657106e82cb49b-0065bb76e7-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:07.028 DEBUG:boto:b'vclfcn-59obj41000falseobj42024-02-01T10:46:46.190Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:07.029 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-02-01T10:48:07.029 DEBUG:boto:path=/vclfcn-59/obj5 2024-02-01T10:48:07.029 DEBUG:boto:auth_path=/vclfcn-59/obj5 2024-02-01T10:48:07.029 DEBUG:boto:path=/vclfcn-59/obj5?versionId=null 2024-02-01T10:48:07.029 DEBUG:boto:auth_path=/vclfcn-59/obj5?versionId=null 2024-02-01T10:48:07.029 DEBUG:boto:Method: GET 2024-02-01T10:48:07.029 DEBUG:boto:Path: /vclfcn-59/obj5?versionId=null 2024-02-01T10:48:07.030 DEBUG:boto:Data: 2024-02-01T10:48:07.030 DEBUG:boto:Headers: {} 2024-02-01T10:48:07.030 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:07.030 DEBUG:boto:Port: 8001 2024-02-01T10:48:07.030 DEBUG:boto:Params: {} 2024-02-01T10:48:07.030 DEBUG:boto:Token: None 2024-02-01T10:48:07.030 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:07 GMT /vclfcn-59/obj5?versionId=null 2024-02-01T10:48:07.030 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:i33VzYYS0uO+zt3F7w+N9If8BL4= 2024-02-01T10:48:07.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:i33VzYYS0uO+zt3F7w+N9If8BL4=', 'Content-Length': '0'} 2024-02-01T10:48:07.033 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b11d92d5d4b2b6b9-0065bb76e7-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:07.076 DEBUG:boto:path=/vclfcn-59/obj5 2024-02-01T10:48:07.076 DEBUG:boto:auth_path=/vclfcn-59/obj5 2024-02-01T10:48:07.077 DEBUG:boto:path=/vclfcn-59/obj5?versionId=null 2024-02-01T10:48:07.077 DEBUG:boto:auth_path=/vclfcn-59/obj5?versionId=null 2024-02-01T10:48:07.077 DEBUG:boto:Method: GET 2024-02-01T10:48:07.077 DEBUG:boto:Path: /vclfcn-59/obj5?versionId=null 2024-02-01T10:48:07.077 DEBUG:boto:Data: 2024-02-01T10:48:07.077 DEBUG:boto:Headers: {} 2024-02-01T10:48:07.077 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:07.077 DEBUG:boto:Port: 8000 2024-02-01T10:48:07.077 DEBUG:boto:Params: {} 2024-02-01T10:48:07.077 DEBUG:boto:Token: None 2024-02-01T10:48:07.078 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:07 GMT /vclfcn-59/obj5?versionId=null 2024-02-01T10:48:07.078 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:i33VzYYS0uO+zt3F7w+N9If8BL4= 2024-02-01T10:48:07.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:i33VzYYS0uO+zt3F7w+N9If8BL4=', 'Content-Length': '0'} 2024-02-01T10:48:07.081 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e8c0f82686c72f82-0065bb76e7-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:07.124 DEBUG:boto:path=/vclfcn-59/obj5 2024-02-01T10:48:07.124 DEBUG:boto:auth_path=/vclfcn-59/obj5 2024-02-01T10:48:07.124 DEBUG:boto:path=/vclfcn-59/obj5?versionId=null 2024-02-01T10:48:07.125 DEBUG:boto:auth_path=/vclfcn-59/obj5?versionId=null 2024-02-01T10:48:07.125 DEBUG:boto:Method: HEAD 2024-02-01T10:48:07.125 DEBUG:boto:Path: /vclfcn-59/obj5?versionId=null 2024-02-01T10:48:07.125 DEBUG:boto:Data: 2024-02-01T10:48:07.125 DEBUG:boto:Headers: {} 2024-02-01T10:48:07.125 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:07.125 DEBUG:boto:Port: 8001 2024-02-01T10:48:07.125 DEBUG:boto:Params: {} 2024-02-01T10:48:07.125 DEBUG:boto:Token: None 2024-02-01T10:48:07.125 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:07 GMT /vclfcn-59/obj5?versionId=null 2024-02-01T10:48:07.126 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:r5swrRARzUgw9C5ZOV2bwOWbLcs= 2024-02-01T10:48:07.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:r5swrRARzUgw9C5ZOV2bwOWbLcs=', 'Content-Length': '0'} 2024-02-01T10:48:07.129 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009d7bcd105fbb2559-0065bb76e7-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:07.129 DEBUG:boto:path=/vclfcn-59/obj5 2024-02-01T10:48:07.129 DEBUG:boto:auth_path=/vclfcn-59/obj5 2024-02-01T10:48:07.129 DEBUG:boto:path=/vclfcn-59/obj5?versionId=null 2024-02-01T10:48:07.129 DEBUG:boto:auth_path=/vclfcn-59/obj5?versionId=null 2024-02-01T10:48:07.129 DEBUG:boto:Method: HEAD 2024-02-01T10:48:07.129 DEBUG:boto:Path: /vclfcn-59/obj5?versionId=null 2024-02-01T10:48:07.129 DEBUG:boto:Data: 2024-02-01T10:48:07.129 DEBUG:boto:Headers: {} 2024-02-01T10:48:07.130 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:07.130 DEBUG:boto:Port: 8000 2024-02-01T10:48:07.130 DEBUG:boto:Params: {} 2024-02-01T10:48:07.130 DEBUG:boto:Token: None 2024-02-01T10:48:07.130 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:07 GMT /vclfcn-59/obj5?versionId=null 2024-02-01T10:48:07.130 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:r5swrRARzUgw9C5ZOV2bwOWbLcs= 2024-02-01T10:48:07.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:r5swrRARzUgw9C5ZOV2bwOWbLcs=', 'Content-Length': '0'} 2024-02-01T10:48:07.133 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009ae2a0953db15621-0065bb76e7-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:07.133 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-02-01T10:48:07.133 DEBUG:boto:path=/vclfcn-59/obj5 2024-02-01T10:48:07.133 DEBUG:boto:auth_path=/vclfcn-59/obj5 2024-02-01T10:48:07.134 DEBUG:boto:path=/vclfcn-59/obj5?versionId=null 2024-02-01T10:48:07.134 DEBUG:boto:auth_path=/vclfcn-59/obj5?versionId=null 2024-02-01T10:48:07.134 DEBUG:boto:Method: GET 2024-02-01T10:48:07.134 DEBUG:boto:Path: /vclfcn-59/obj5?versionId=null 2024-02-01T10:48:07.134 DEBUG:boto:Data: 2024-02-01T10:48:07.134 DEBUG:boto:Headers: {} 2024-02-01T10:48:07.134 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:07.134 DEBUG:boto:Port: 8001 2024-02-01T10:48:07.134 DEBUG:boto:Params: {} 2024-02-01T10:48:07.134 DEBUG:boto:Token: None 2024-02-01T10:48:07.135 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:07 GMT /vclfcn-59/obj5?versionId=null 2024-02-01T10:48:07.135 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:i33VzYYS0uO+zt3F7w+N9If8BL4= 2024-02-01T10:48:07.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:i33VzYYS0uO+zt3F7w+N9If8BL4=', 'Content-Length': '0'} 2024-02-01T10:48:07.138 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a1e689e245353495-0065bb76e7-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:07.180 DEBUG:boto:path=/vclfcn-59/obj5 2024-02-01T10:48:07.180 DEBUG:boto:auth_path=/vclfcn-59/obj5 2024-02-01T10:48:07.180 DEBUG:boto:path=/vclfcn-59/obj5?versionId=null 2024-02-01T10:48:07.180 DEBUG:boto:auth_path=/vclfcn-59/obj5?versionId=null 2024-02-01T10:48:07.181 DEBUG:boto:Method: GET 2024-02-01T10:48:07.181 DEBUG:boto:Path: /vclfcn-59/obj5?versionId=null 2024-02-01T10:48:07.181 DEBUG:boto:Data: 2024-02-01T10:48:07.181 DEBUG:boto:Headers: {} 2024-02-01T10:48:07.181 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:07.181 DEBUG:boto:Port: 8000 2024-02-01T10:48:07.181 DEBUG:boto:Params: {} 2024-02-01T10:48:07.181 DEBUG:boto:Token: None 2024-02-01T10:48:07.181 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:07 GMT /vclfcn-59/obj5?versionId=null 2024-02-01T10:48:07.182 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:i33VzYYS0uO+zt3F7w+N9If8BL4= 2024-02-01T10:48:07.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:i33VzYYS0uO+zt3F7w+N9If8BL4=', 'Content-Length': '0'} 2024-02-01T10:48:07.185 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006edb377cb8c47970-0065bb76e7-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:07.228 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:07.228 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:07.228 DEBUG:boto:path=/vclfcn-59/?prefix=obj5 2024-02-01T10:48:07.228 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj5 2024-02-01T10:48:07.229 DEBUG:boto:Method: GET 2024-02-01T10:48:07.229 DEBUG:boto:Path: /vclfcn-59/?prefix=obj5 2024-02-01T10:48:07.229 DEBUG:boto:Data: 2024-02-01T10:48:07.229 DEBUG:boto:Headers: {} 2024-02-01T10:48:07.229 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:07.229 DEBUG:boto:Port: 8001 2024-02-01T10:48:07.229 DEBUG:boto:Params: {} 2024-02-01T10:48:07.229 DEBUG:boto:Token: None 2024-02-01T10:48:07.229 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:07 GMT /vclfcn-59/ 2024-02-01T10:48:07.229 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NwdGjBpCSV0xKw6akevMJFGrGXM= 2024-02-01T10:48:07.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NwdGjBpCSV0xKw6akevMJFGrGXM=', 'Content-Length': '0'} 2024-02-01T10:48:07.235 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ca52b85ed04d7d3-0065bb76e7-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:07.235 DEBUG:boto:b'vclfcn-59obj51000falseobj52024-02-01T10:47:30.206Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:07.236 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:07.236 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:07.236 DEBUG:boto:path=/vclfcn-59/?prefix=obj5 2024-02-01T10:48:07.236 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj5 2024-02-01T10:48:07.236 DEBUG:boto:Method: GET 2024-02-01T10:48:07.236 DEBUG:boto:Path: /vclfcn-59/?prefix=obj5 2024-02-01T10:48:07.236 DEBUG:boto:Data: 2024-02-01T10:48:07.236 DEBUG:boto:Headers: {} 2024-02-01T10:48:07.237 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:07.237 DEBUG:boto:Port: 8000 2024-02-01T10:48:07.237 DEBUG:boto:Params: {} 2024-02-01T10:48:07.237 DEBUG:boto:Token: None 2024-02-01T10:48:07.237 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:07 GMT /vclfcn-59/ 2024-02-01T10:48:07.237 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NwdGjBpCSV0xKw6akevMJFGrGXM= 2024-02-01T10:48:07.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NwdGjBpCSV0xKw6akevMJFGrGXM=', 'Content-Length': '0'} 2024-02-01T10:48:07.242 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d334e1f11d0ce56-0065bb76e7-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:07.243 DEBUG:boto:b'vclfcn-59obj51000falseobj52024-02-01T10:47:30.206Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:07.243 DEBUG:boto:path=/vclfcn-59/obj5 2024-02-01T10:48:07.243 DEBUG:boto:auth_path=/vclfcn-59/obj5 2024-02-01T10:48:07.243 DEBUG:boto:Method: HEAD 2024-02-01T10:48:07.243 DEBUG:boto:Path: /vclfcn-59/obj5 2024-02-01T10:48:07.243 DEBUG:boto:Data: 2024-02-01T10:48:07.244 DEBUG:boto:Headers: {} 2024-02-01T10:48:07.244 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:07.244 DEBUG:boto:Port: 8001 2024-02-01T10:48:07.244 DEBUG:boto:Params: {} 2024-02-01T10:48:07.244 DEBUG:boto:Token: None 2024-02-01T10:48:07.244 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:07 GMT /vclfcn-59/obj5 2024-02-01T10:48:07.245 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:F8UBVcb1YkhTF3euISJF+kS3Jg8= 2024-02-01T10:48:07.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:F8UBVcb1YkhTF3euISJF+kS3Jg8=', 'Content-Length': '0'} 2024-02-01T10:48:07.247 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000c56d218fe9bd488-0065bb76e7-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:07.248 DEBUG:boto:path=/vclfcn-59/obj5 2024-02-01T10:48:07.248 DEBUG:boto:auth_path=/vclfcn-59/obj5 2024-02-01T10:48:07.248 DEBUG:boto:Method: HEAD 2024-02-01T10:48:07.248 DEBUG:boto:Path: /vclfcn-59/obj5 2024-02-01T10:48:07.248 DEBUG:boto:Data: 2024-02-01T10:48:07.248 DEBUG:boto:Headers: {} 2024-02-01T10:48:07.248 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:07.248 DEBUG:boto:Port: 8000 2024-02-01T10:48:07.248 DEBUG:boto:Params: {} 2024-02-01T10:48:07.249 DEBUG:boto:Token: None 2024-02-01T10:48:07.249 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:07 GMT /vclfcn-59/obj5 2024-02-01T10:48:07.249 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:F8UBVcb1YkhTF3euISJF+kS3Jg8= 2024-02-01T10:48:07.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:F8UBVcb1YkhTF3euISJF+kS3Jg8=', 'Content-Length': '0'} 2024-02-01T10:48:07.251 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005d58af0cdd37a71a-0065bb76e7-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:07.252 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-02-01T10:48:07.252 DEBUG:boto:path=/vclfcn-59/obj5 2024-02-01T10:48:07.252 DEBUG:boto:auth_path=/vclfcn-59/obj5 2024-02-01T10:48:07.252 DEBUG:boto:Method: GET 2024-02-01T10:48:07.252 DEBUG:boto:Path: /vclfcn-59/obj5 2024-02-01T10:48:07.252 DEBUG:boto:Data: 2024-02-01T10:48:07.252 DEBUG:boto:Headers: {} 2024-02-01T10:48:07.253 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:07.253 DEBUG:boto:Port: 8001 2024-02-01T10:48:07.253 DEBUG:boto:Params: {} 2024-02-01T10:48:07.253 DEBUG:boto:Token: None 2024-02-01T10:48:07.253 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:07 GMT /vclfcn-59/obj5 2024-02-01T10:48:07.253 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:dwW+vAwPBL2U1C64ceNy69oGDFI= 2024-02-01T10:48:07.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dwW+vAwPBL2U1C64ceNy69oGDFI=', 'Content-Length': '0'} 2024-02-01T10:48:07.256 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000084652e380be85788-0065bb76e7-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:07.300 DEBUG:boto:path=/vclfcn-59/obj5 2024-02-01T10:48:07.300 DEBUG:boto:auth_path=/vclfcn-59/obj5 2024-02-01T10:48:07.300 DEBUG:boto:Method: GET 2024-02-01T10:48:07.300 DEBUG:boto:Path: /vclfcn-59/obj5 2024-02-01T10:48:07.301 DEBUG:boto:Data: 2024-02-01T10:48:07.301 DEBUG:boto:Headers: {} 2024-02-01T10:48:07.301 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:07.301 DEBUG:boto:Port: 8000 2024-02-01T10:48:07.301 DEBUG:boto:Params: {} 2024-02-01T10:48:07.301 DEBUG:boto:Token: None 2024-02-01T10:48:07.301 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:07 GMT /vclfcn-59/obj5 2024-02-01T10:48:07.301 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:dwW+vAwPBL2U1C64ceNy69oGDFI= 2024-02-01T10:48:07.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dwW+vAwPBL2U1C64ceNy69oGDFI=', 'Content-Length': '0'} 2024-02-01T10:48:07.304 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000021df5921152c4c94-0065bb76e7-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:07.348 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:07.348 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:07.348 DEBUG:boto:path=/vclfcn-59/?prefix=obj5 2024-02-01T10:48:07.348 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj5 2024-02-01T10:48:07.349 DEBUG:boto:Method: GET 2024-02-01T10:48:07.349 DEBUG:boto:Path: /vclfcn-59/?prefix=obj5 2024-02-01T10:48:07.349 DEBUG:boto:Data: 2024-02-01T10:48:07.349 DEBUG:boto:Headers: {} 2024-02-01T10:48:07.349 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:07.349 DEBUG:boto:Port: 8001 2024-02-01T10:48:07.349 DEBUG:boto:Params: {} 2024-02-01T10:48:07.349 DEBUG:boto:Token: None 2024-02-01T10:48:07.349 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:07 GMT /vclfcn-59/ 2024-02-01T10:48:07.350 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NwdGjBpCSV0xKw6akevMJFGrGXM= 2024-02-01T10:48:07.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NwdGjBpCSV0xKw6akevMJFGrGXM=', 'Content-Length': '0'} 2024-02-01T10:48:07.354 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a71ead9d01d88eb-0065bb76e7-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:07.354 DEBUG:boto:b'vclfcn-59obj51000falseobj52024-02-01T10:47:30.206Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:07.355 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:07.355 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:07.355 DEBUG:boto:path=/vclfcn-59/?prefix=obj5 2024-02-01T10:48:07.355 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj5 2024-02-01T10:48:07.355 DEBUG:boto:Method: GET 2024-02-01T10:48:07.355 DEBUG:boto:Path: /vclfcn-59/?prefix=obj5 2024-02-01T10:48:07.355 DEBUG:boto:Data: 2024-02-01T10:48:07.355 DEBUG:boto:Headers: {} 2024-02-01T10:48:07.355 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:07.356 DEBUG:boto:Port: 8000 2024-02-01T10:48:07.356 DEBUG:boto:Params: {} 2024-02-01T10:48:07.356 DEBUG:boto:Token: None 2024-02-01T10:48:07.356 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:07 GMT /vclfcn-59/ 2024-02-01T10:48:07.356 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NwdGjBpCSV0xKw6akevMJFGrGXM= 2024-02-01T10:48:07.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NwdGjBpCSV0xKw6akevMJFGrGXM=', 'Content-Length': '0'} 2024-02-01T10:48:07.361 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064cef97f8d820d00-0065bb76e7-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:07.361 DEBUG:boto:b'vclfcn-59obj51000falseobj52024-02-01T10:47:30.206Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:07.361 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-02-01T10:48:07.361 DEBUG:boto:path=/vclfcn-59/obj6 2024-02-01T10:48:07.362 DEBUG:boto:auth_path=/vclfcn-59/obj6 2024-02-01T10:48:07.362 DEBUG:boto:path=/vclfcn-59/obj6?versionId=null 2024-02-01T10:48:07.362 DEBUG:boto:auth_path=/vclfcn-59/obj6?versionId=null 2024-02-01T10:48:07.362 DEBUG:boto:Method: GET 2024-02-01T10:48:07.362 DEBUG:boto:Path: /vclfcn-59/obj6?versionId=null 2024-02-01T10:48:07.362 DEBUG:boto:Data: 2024-02-01T10:48:07.362 DEBUG:boto:Headers: {} 2024-02-01T10:48:07.362 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:07.362 DEBUG:boto:Port: 8001 2024-02-01T10:48:07.362 DEBUG:boto:Params: {} 2024-02-01T10:48:07.362 DEBUG:boto:Token: None 2024-02-01T10:48:07.363 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:07 GMT /vclfcn-59/obj6?versionId=null 2024-02-01T10:48:07.363 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ggBqNpUathg4DKPykz2vv/UsKLk= 2024-02-01T10:48:07.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ggBqNpUathg4DKPykz2vv/UsKLk=', 'Content-Length': '0'} 2024-02-01T10:48:07.365 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e007064bda442c40-0065bb76e7-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:07.408 DEBUG:boto:path=/vclfcn-59/obj6 2024-02-01T10:48:07.408 DEBUG:boto:auth_path=/vclfcn-59/obj6 2024-02-01T10:48:07.409 DEBUG:boto:path=/vclfcn-59/obj6?versionId=null 2024-02-01T10:48:07.409 DEBUG:boto:auth_path=/vclfcn-59/obj6?versionId=null 2024-02-01T10:48:07.409 DEBUG:boto:Method: GET 2024-02-01T10:48:07.409 DEBUG:boto:Path: /vclfcn-59/obj6?versionId=null 2024-02-01T10:48:07.409 DEBUG:boto:Data: 2024-02-01T10:48:07.409 DEBUG:boto:Headers: {} 2024-02-01T10:48:07.409 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:07.409 DEBUG:boto:Port: 8000 2024-02-01T10:48:07.409 DEBUG:boto:Params: {} 2024-02-01T10:48:07.410 DEBUG:boto:Token: None 2024-02-01T10:48:07.410 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:07 GMT /vclfcn-59/obj6?versionId=null 2024-02-01T10:48:07.410 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ggBqNpUathg4DKPykz2vv/UsKLk= 2024-02-01T10:48:07.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ggBqNpUathg4DKPykz2vv/UsKLk=', 'Content-Length': '0'} 2024-02-01T10:48:07.413 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f4c16c7c3a70deae-0065bb76e7-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:07.413 DEBUG:boto:path=/vclfcn-59/obj6 2024-02-01T10:48:07.414 DEBUG:boto:auth_path=/vclfcn-59/obj6 2024-02-01T10:48:07.414 DEBUG:boto:path=/vclfcn-59/obj6?versionId=null 2024-02-01T10:48:07.414 DEBUG:boto:auth_path=/vclfcn-59/obj6?versionId=null 2024-02-01T10:48:07.414 DEBUG:boto:Method: HEAD 2024-02-01T10:48:07.414 DEBUG:boto:Path: /vclfcn-59/obj6?versionId=null 2024-02-01T10:48:07.414 DEBUG:boto:Data: 2024-02-01T10:48:07.414 DEBUG:boto:Headers: {} 2024-02-01T10:48:07.414 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:07.414 DEBUG:boto:Port: 8001 2024-02-01T10:48:07.414 DEBUG:boto:Params: {} 2024-02-01T10:48:07.414 DEBUG:boto:Token: None 2024-02-01T10:48:07.415 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:07 GMT /vclfcn-59/obj6?versionId=null 2024-02-01T10:48:07.415 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cfHik4jnY+BXoyz/Li7GdVNUBUA= 2024-02-01T10:48:07.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cfHik4jnY+BXoyz/Li7GdVNUBUA=', 'Content-Length': '0'} 2024-02-01T10:48:07.418 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000635d20136056aca7-0065bb76e7-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:07.418 DEBUG:boto:path=/vclfcn-59/obj6 2024-02-01T10:48:07.418 DEBUG:boto:auth_path=/vclfcn-59/obj6 2024-02-01T10:48:07.418 DEBUG:boto:path=/vclfcn-59/obj6?versionId=null 2024-02-01T10:48:07.418 DEBUG:boto:auth_path=/vclfcn-59/obj6?versionId=null 2024-02-01T10:48:07.418 DEBUG:boto:Method: HEAD 2024-02-01T10:48:07.418 DEBUG:boto:Path: /vclfcn-59/obj6?versionId=null 2024-02-01T10:48:07.418 DEBUG:boto:Data: 2024-02-01T10:48:07.419 DEBUG:boto:Headers: {} 2024-02-01T10:48:07.419 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:07.419 DEBUG:boto:Port: 8000 2024-02-01T10:48:07.419 DEBUG:boto:Params: {} 2024-02-01T10:48:07.419 DEBUG:boto:Token: None 2024-02-01T10:48:07.419 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:07 GMT /vclfcn-59/obj6?versionId=null 2024-02-01T10:48:07.419 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cfHik4jnY+BXoyz/Li7GdVNUBUA= 2024-02-01T10:48:07.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cfHik4jnY+BXoyz/Li7GdVNUBUA=', 'Content-Length': '0'} 2024-02-01T10:48:07.422 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c422b226f334cdf2-0065bb76e7-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:07.422 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-02-01T10:48:07.422 DEBUG:boto:path=/vclfcn-59/obj6 2024-02-01T10:48:07.422 DEBUG:boto:auth_path=/vclfcn-59/obj6 2024-02-01T10:48:07.423 DEBUG:boto:path=/vclfcn-59/obj6?versionId=null 2024-02-01T10:48:07.423 DEBUG:boto:auth_path=/vclfcn-59/obj6?versionId=null 2024-02-01T10:48:07.423 DEBUG:boto:Method: GET 2024-02-01T10:48:07.423 DEBUG:boto:Path: /vclfcn-59/obj6?versionId=null 2024-02-01T10:48:07.423 DEBUG:boto:Data: 2024-02-01T10:48:07.423 DEBUG:boto:Headers: {} 2024-02-01T10:48:07.423 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:07.423 DEBUG:boto:Port: 8001 2024-02-01T10:48:07.423 DEBUG:boto:Params: {} 2024-02-01T10:48:07.423 DEBUG:boto:Token: None 2024-02-01T10:48:07.423 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:07 GMT /vclfcn-59/obj6?versionId=null 2024-02-01T10:48:07.424 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ggBqNpUathg4DKPykz2vv/UsKLk= 2024-02-01T10:48:07.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ggBqNpUathg4DKPykz2vv/UsKLk=', 'Content-Length': '0'} 2024-02-01T10:48:07.426 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bd8ba90f41e49c98-0065bb76e7-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:07.468 DEBUG:boto:path=/vclfcn-59/obj6 2024-02-01T10:48:07.468 DEBUG:boto:auth_path=/vclfcn-59/obj6 2024-02-01T10:48:07.468 DEBUG:boto:path=/vclfcn-59/obj6?versionId=null 2024-02-01T10:48:07.469 DEBUG:boto:auth_path=/vclfcn-59/obj6?versionId=null 2024-02-01T10:48:07.469 DEBUG:boto:Method: GET 2024-02-01T10:48:07.469 DEBUG:boto:Path: /vclfcn-59/obj6?versionId=null 2024-02-01T10:48:07.469 DEBUG:boto:Data: 2024-02-01T10:48:07.469 DEBUG:boto:Headers: {} 2024-02-01T10:48:07.469 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:07.469 DEBUG:boto:Port: 8000 2024-02-01T10:48:07.469 DEBUG:boto:Params: {} 2024-02-01T10:48:07.470 DEBUG:boto:Token: None 2024-02-01T10:48:07.470 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:07 GMT /vclfcn-59/obj6?versionId=null 2024-02-01T10:48:07.470 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ggBqNpUathg4DKPykz2vv/UsKLk= 2024-02-01T10:48:07.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ggBqNpUathg4DKPykz2vv/UsKLk=', 'Content-Length': '0'} 2024-02-01T10:48:07.473 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003012eddb153086e9-0065bb76e7-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:07.516 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:07.516 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:07.516 DEBUG:boto:path=/vclfcn-59/?prefix=obj6 2024-02-01T10:48:07.516 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj6 2024-02-01T10:48:07.516 DEBUG:boto:Method: GET 2024-02-01T10:48:07.517 DEBUG:boto:Path: /vclfcn-59/?prefix=obj6 2024-02-01T10:48:07.517 DEBUG:boto:Data: 2024-02-01T10:48:07.517 DEBUG:boto:Headers: {} 2024-02-01T10:48:07.517 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:07.517 DEBUG:boto:Port: 8001 2024-02-01T10:48:07.517 DEBUG:boto:Params: {} 2024-02-01T10:48:07.517 DEBUG:boto:Token: None 2024-02-01T10:48:07.517 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:07 GMT /vclfcn-59/ 2024-02-01T10:48:07.518 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NwdGjBpCSV0xKw6akevMJFGrGXM= 2024-02-01T10:48:07.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NwdGjBpCSV0xKw6akevMJFGrGXM=', 'Content-Length': '0'} 2024-02-01T10:48:07.522 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001db1bdec09914bf9-0065bb76e7-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:07.523 DEBUG:boto:b'vclfcn-59obj61000falseobj62024-02-01T10:47:30.217Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:07.523 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:07.523 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:07.523 DEBUG:boto:path=/vclfcn-59/?prefix=obj6 2024-02-01T10:48:07.523 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj6 2024-02-01T10:48:07.524 DEBUG:boto:Method: GET 2024-02-01T10:48:07.524 DEBUG:boto:Path: /vclfcn-59/?prefix=obj6 2024-02-01T10:48:07.524 DEBUG:boto:Data: 2024-02-01T10:48:07.524 DEBUG:boto:Headers: {} 2024-02-01T10:48:07.524 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:07.524 DEBUG:boto:Port: 8000 2024-02-01T10:48:07.524 DEBUG:boto:Params: {} 2024-02-01T10:48:07.524 DEBUG:boto:Token: None 2024-02-01T10:48:07.524 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:07 GMT /vclfcn-59/ 2024-02-01T10:48:07.525 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NwdGjBpCSV0xKw6akevMJFGrGXM= 2024-02-01T10:48:07.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NwdGjBpCSV0xKw6akevMJFGrGXM=', 'Content-Length': '0'} 2024-02-01T10:48:07.529 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ea0c989419d1651-0065bb76e7-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:07.529 DEBUG:boto:b'vclfcn-59obj61000falseobj62024-02-01T10:47:30.217Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:07.529 DEBUG:boto:path=/vclfcn-59/obj6 2024-02-01T10:48:07.529 DEBUG:boto:auth_path=/vclfcn-59/obj6 2024-02-01T10:48:07.530 DEBUG:boto:Method: HEAD 2024-02-01T10:48:07.530 DEBUG:boto:Path: /vclfcn-59/obj6 2024-02-01T10:48:07.530 DEBUG:boto:Data: 2024-02-01T10:48:07.530 DEBUG:boto:Headers: {} 2024-02-01T10:48:07.530 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:07.530 DEBUG:boto:Port: 8001 2024-02-01T10:48:07.530 DEBUG:boto:Params: {} 2024-02-01T10:48:07.530 DEBUG:boto:Token: None 2024-02-01T10:48:07.530 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:07 GMT /vclfcn-59/obj6 2024-02-01T10:48:07.530 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:INmIhXQVkFUMrDFdGVx1mSEh+Ds= 2024-02-01T10:48:07.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:INmIhXQVkFUMrDFdGVx1mSEh+Ds=', 'Content-Length': '0'} 2024-02-01T10:48:07.533 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004adec40ca2b86481-0065bb76e7-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:07.534 DEBUG:boto:path=/vclfcn-59/obj6 2024-02-01T10:48:07.534 DEBUG:boto:auth_path=/vclfcn-59/obj6 2024-02-01T10:48:07.534 DEBUG:boto:Method: HEAD 2024-02-01T10:48:07.534 DEBUG:boto:Path: /vclfcn-59/obj6 2024-02-01T10:48:07.534 DEBUG:boto:Data: 2024-02-01T10:48:07.534 DEBUG:boto:Headers: {} 2024-02-01T10:48:07.534 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:07.534 DEBUG:boto:Port: 8000 2024-02-01T10:48:07.535 DEBUG:boto:Params: {} 2024-02-01T10:48:07.535 DEBUG:boto:Token: None 2024-02-01T10:48:07.535 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:07 GMT /vclfcn-59/obj6 2024-02-01T10:48:07.535 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:INmIhXQVkFUMrDFdGVx1mSEh+Ds= 2024-02-01T10:48:07.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:INmIhXQVkFUMrDFdGVx1mSEh+Ds=', 'Content-Length': '0'} 2024-02-01T10:48:07.538 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ccb69b3e9e3d3e09-0065bb76e7-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:07.538 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-02-01T10:48:07.538 DEBUG:boto:path=/vclfcn-59/obj6 2024-02-01T10:48:07.538 DEBUG:boto:auth_path=/vclfcn-59/obj6 2024-02-01T10:48:07.539 DEBUG:boto:Method: GET 2024-02-01T10:48:07.539 DEBUG:boto:Path: /vclfcn-59/obj6 2024-02-01T10:48:07.539 DEBUG:boto:Data: 2024-02-01T10:48:07.539 DEBUG:boto:Headers: {} 2024-02-01T10:48:07.539 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:07.539 DEBUG:boto:Port: 8001 2024-02-01T10:48:07.539 DEBUG:boto:Params: {} 2024-02-01T10:48:07.539 DEBUG:boto:Token: None 2024-02-01T10:48:07.539 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:07 GMT /vclfcn-59/obj6 2024-02-01T10:48:07.539 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:i5LTGz0WRM/EZgFIJ48SDwD/ouA= 2024-02-01T10:48:07.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:i5LTGz0WRM/EZgFIJ48SDwD/ouA=', 'Content-Length': '0'} 2024-02-01T10:48:07.542 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003d3676631053e559-0065bb76e7-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:07.584 DEBUG:boto:path=/vclfcn-59/obj6 2024-02-01T10:48:07.584 DEBUG:boto:auth_path=/vclfcn-59/obj6 2024-02-01T10:48:07.584 DEBUG:boto:Method: GET 2024-02-01T10:48:07.584 DEBUG:boto:Path: /vclfcn-59/obj6 2024-02-01T10:48:07.584 DEBUG:boto:Data: 2024-02-01T10:48:07.585 DEBUG:boto:Headers: {} 2024-02-01T10:48:07.585 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:07.585 DEBUG:boto:Port: 8000 2024-02-01T10:48:07.585 DEBUG:boto:Params: {} 2024-02-01T10:48:07.585 DEBUG:boto:Token: None 2024-02-01T10:48:07.585 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:07 GMT /vclfcn-59/obj6 2024-02-01T10:48:07.585 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:i5LTGz0WRM/EZgFIJ48SDwD/ouA= 2024-02-01T10:48:07.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:i5LTGz0WRM/EZgFIJ48SDwD/ouA=', 'Content-Length': '0'} 2024-02-01T10:48:07.588 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000183e538ae6c4a1b2-0065bb76e7-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:07.632 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:07.632 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:07.632 DEBUG:boto:path=/vclfcn-59/?prefix=obj6 2024-02-01T10:48:07.632 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj6 2024-02-01T10:48:07.633 DEBUG:boto:Method: GET 2024-02-01T10:48:07.633 DEBUG:boto:Path: /vclfcn-59/?prefix=obj6 2024-02-01T10:48:07.633 DEBUG:boto:Data: 2024-02-01T10:48:07.633 DEBUG:boto:Headers: {} 2024-02-01T10:48:07.633 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:07.633 DEBUG:boto:Port: 8001 2024-02-01T10:48:07.633 DEBUG:boto:Params: {} 2024-02-01T10:48:07.633 DEBUG:boto:Token: None 2024-02-01T10:48:07.633 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:07 GMT /vclfcn-59/ 2024-02-01T10:48:07.634 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NwdGjBpCSV0xKw6akevMJFGrGXM= 2024-02-01T10:48:07.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NwdGjBpCSV0xKw6akevMJFGrGXM=', 'Content-Length': '0'} 2024-02-01T10:48:07.638 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7e4b608b1470a03-0065bb76e7-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:07.680 DEBUG:boto:b'vclfcn-59obj61000falseobj62024-02-01T10:47:30.217Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:07.681 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:07.681 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:07.681 DEBUG:boto:path=/vclfcn-59/?prefix=obj6 2024-02-01T10:48:07.681 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj6 2024-02-01T10:48:07.681 DEBUG:boto:Method: GET 2024-02-01T10:48:07.681 DEBUG:boto:Path: /vclfcn-59/?prefix=obj6 2024-02-01T10:48:07.681 DEBUG:boto:Data: 2024-02-01T10:48:07.681 DEBUG:boto:Headers: {} 2024-02-01T10:48:07.681 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:07.682 DEBUG:boto:Port: 8000 2024-02-01T10:48:07.682 DEBUG:boto:Params: {} 2024-02-01T10:48:07.682 DEBUG:boto:Token: None 2024-02-01T10:48:07.682 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:07 GMT /vclfcn-59/ 2024-02-01T10:48:07.682 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NwdGjBpCSV0xKw6akevMJFGrGXM= 2024-02-01T10:48:07.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NwdGjBpCSV0xKw6akevMJFGrGXM=', 'Content-Length': '0'} 2024-02-01T10:48:07.686 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002fc66d0a592898d6-0065bb76e7-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:07.728 DEBUG:boto:b'vclfcn-59obj61000falseobj62024-02-01T10:47:30.217Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:07.728 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-02-01T10:48:07.728 DEBUG:boto:path=/vclfcn-59/obj7 2024-02-01T10:48:07.729 DEBUG:boto:auth_path=/vclfcn-59/obj7 2024-02-01T10:48:07.729 DEBUG:boto:path=/vclfcn-59/obj7?versionId=null 2024-02-01T10:48:07.729 DEBUG:boto:auth_path=/vclfcn-59/obj7?versionId=null 2024-02-01T10:48:07.729 DEBUG:boto:Method: GET 2024-02-01T10:48:07.729 DEBUG:boto:Path: /vclfcn-59/obj7?versionId=null 2024-02-01T10:48:07.729 DEBUG:boto:Data: 2024-02-01T10:48:07.729 DEBUG:boto:Headers: {} 2024-02-01T10:48:07.729 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:07.729 DEBUG:boto:Port: 8001 2024-02-01T10:48:07.729 DEBUG:boto:Params: {} 2024-02-01T10:48:07.729 DEBUG:boto:Token: None 2024-02-01T10:48:07.730 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:07 GMT /vclfcn-59/obj7?versionId=null 2024-02-01T10:48:07.730 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:C1YIzMbIvh9eLuZagSP9aP+mDgo= 2024-02-01T10:48:07.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:C1YIzMbIvh9eLuZagSP9aP+mDgo=', 'Content-Length': '0'} 2024-02-01T10:48:07.734 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000849f8b197a4e9573-0065bb76e7-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:07.734 DEBUG:boto:path=/vclfcn-59/obj7 2024-02-01T10:48:07.734 DEBUG:boto:auth_path=/vclfcn-59/obj7 2024-02-01T10:48:07.734 DEBUG:boto:path=/vclfcn-59/obj7?versionId=null 2024-02-01T10:48:07.734 DEBUG:boto:auth_path=/vclfcn-59/obj7?versionId=null 2024-02-01T10:48:07.734 DEBUG:boto:Method: GET 2024-02-01T10:48:07.735 DEBUG:boto:Path: /vclfcn-59/obj7?versionId=null 2024-02-01T10:48:07.735 DEBUG:boto:Data: 2024-02-01T10:48:07.735 DEBUG:boto:Headers: {} 2024-02-01T10:48:07.735 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:07.735 DEBUG:boto:Port: 8000 2024-02-01T10:48:07.735 DEBUG:boto:Params: {} 2024-02-01T10:48:07.735 DEBUG:boto:Token: None 2024-02-01T10:48:07.735 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:07 GMT /vclfcn-59/obj7?versionId=null 2024-02-01T10:48:07.735 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:C1YIzMbIvh9eLuZagSP9aP+mDgo= 2024-02-01T10:48:07.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:C1YIzMbIvh9eLuZagSP9aP+mDgo=', 'Content-Length': '0'} 2024-02-01T10:48:07.738 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bcf898a955fdac85-0065bb76e7-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:07.739 DEBUG:boto:path=/vclfcn-59/obj7 2024-02-01T10:48:07.739 DEBUG:boto:auth_path=/vclfcn-59/obj7 2024-02-01T10:48:07.739 DEBUG:boto:path=/vclfcn-59/obj7?versionId=null 2024-02-01T10:48:07.739 DEBUG:boto:auth_path=/vclfcn-59/obj7?versionId=null 2024-02-01T10:48:07.739 DEBUG:boto:Method: HEAD 2024-02-01T10:48:07.739 DEBUG:boto:Path: /vclfcn-59/obj7?versionId=null 2024-02-01T10:48:07.739 DEBUG:boto:Data: 2024-02-01T10:48:07.739 DEBUG:boto:Headers: {} 2024-02-01T10:48:07.740 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:07.740 DEBUG:boto:Port: 8001 2024-02-01T10:48:07.740 DEBUG:boto:Params: {} 2024-02-01T10:48:07.740 DEBUG:boto:Token: None 2024-02-01T10:48:07.740 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:07 GMT /vclfcn-59/obj7?versionId=null 2024-02-01T10:48:07.740 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zU+bRNCxCW8whbImWeXyQV8llDU= 2024-02-01T10:48:07.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zU+bRNCxCW8whbImWeXyQV8llDU=', 'Content-Length': '0'} 2024-02-01T10:48:07.743 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006177c62b7708f5a5-0065bb76e7-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:07.744 DEBUG:boto:path=/vclfcn-59/obj7 2024-02-01T10:48:07.744 DEBUG:boto:auth_path=/vclfcn-59/obj7 2024-02-01T10:48:07.744 DEBUG:boto:path=/vclfcn-59/obj7?versionId=null 2024-02-01T10:48:07.744 DEBUG:boto:auth_path=/vclfcn-59/obj7?versionId=null 2024-02-01T10:48:07.744 DEBUG:boto:Method: HEAD 2024-02-01T10:48:07.744 DEBUG:boto:Path: /vclfcn-59/obj7?versionId=null 2024-02-01T10:48:07.744 DEBUG:boto:Data: 2024-02-01T10:48:07.744 DEBUG:boto:Headers: {} 2024-02-01T10:48:07.744 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:07.744 DEBUG:boto:Port: 8000 2024-02-01T10:48:07.745 DEBUG:boto:Params: {} 2024-02-01T10:48:07.745 DEBUG:boto:Token: None 2024-02-01T10:48:07.745 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:07 GMT /vclfcn-59/obj7?versionId=null 2024-02-01T10:48:07.745 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zU+bRNCxCW8whbImWeXyQV8llDU= 2024-02-01T10:48:07.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zU+bRNCxCW8whbImWeXyQV8llDU=', 'Content-Length': '0'} 2024-02-01T10:48:07.748 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000086bc0e922dd341f2-0065bb76e7-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:07.748 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-02-01T10:48:07.748 DEBUG:boto:path=/vclfcn-59/obj7 2024-02-01T10:48:07.748 DEBUG:boto:auth_path=/vclfcn-59/obj7 2024-02-01T10:48:07.748 DEBUG:boto:path=/vclfcn-59/obj7?versionId=null 2024-02-01T10:48:07.748 DEBUG:boto:auth_path=/vclfcn-59/obj7?versionId=null 2024-02-01T10:48:07.748 DEBUG:boto:Method: GET 2024-02-01T10:48:07.749 DEBUG:boto:Path: /vclfcn-59/obj7?versionId=null 2024-02-01T10:48:07.749 DEBUG:boto:Data: 2024-02-01T10:48:07.749 DEBUG:boto:Headers: {} 2024-02-01T10:48:07.749 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:07.749 DEBUG:boto:Port: 8001 2024-02-01T10:48:07.749 DEBUG:boto:Params: {} 2024-02-01T10:48:07.749 DEBUG:boto:Token: None 2024-02-01T10:48:07.749 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:07 GMT /vclfcn-59/obj7?versionId=null 2024-02-01T10:48:07.749 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:C1YIzMbIvh9eLuZagSP9aP+mDgo= 2024-02-01T10:48:07.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:C1YIzMbIvh9eLuZagSP9aP+mDgo=', 'Content-Length': '0'} 2024-02-01T10:48:07.752 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000949f4f0dfb836809-0065bb76e7-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:07.796 DEBUG:boto:path=/vclfcn-59/obj7 2024-02-01T10:48:07.796 DEBUG:boto:auth_path=/vclfcn-59/obj7 2024-02-01T10:48:07.796 DEBUG:boto:path=/vclfcn-59/obj7?versionId=null 2024-02-01T10:48:07.796 DEBUG:boto:auth_path=/vclfcn-59/obj7?versionId=null 2024-02-01T10:48:07.797 DEBUG:boto:Method: GET 2024-02-01T10:48:07.797 DEBUG:boto:Path: /vclfcn-59/obj7?versionId=null 2024-02-01T10:48:07.797 DEBUG:boto:Data: 2024-02-01T10:48:07.797 DEBUG:boto:Headers: {} 2024-02-01T10:48:07.797 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:07.797 DEBUG:boto:Port: 8000 2024-02-01T10:48:07.797 DEBUG:boto:Params: {} 2024-02-01T10:48:07.797 DEBUG:boto:Token: None 2024-02-01T10:48:07.797 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:07 GMT /vclfcn-59/obj7?versionId=null 2024-02-01T10:48:07.797 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:C1YIzMbIvh9eLuZagSP9aP+mDgo= 2024-02-01T10:48:07.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:C1YIzMbIvh9eLuZagSP9aP+mDgo=', 'Content-Length': '0'} 2024-02-01T10:48:07.800 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000068c9a0d07794e17f-0065bb76e7-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:07.848 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:07.848 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:07.848 DEBUG:boto:path=/vclfcn-59/?prefix=obj7 2024-02-01T10:48:07.848 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj7 2024-02-01T10:48:07.849 DEBUG:boto:Method: GET 2024-02-01T10:48:07.849 DEBUG:boto:Path: /vclfcn-59/?prefix=obj7 2024-02-01T10:48:07.849 DEBUG:boto:Data: 2024-02-01T10:48:07.849 DEBUG:boto:Headers: {} 2024-02-01T10:48:07.849 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:07.849 DEBUG:boto:Port: 8001 2024-02-01T10:48:07.849 DEBUG:boto:Params: {} 2024-02-01T10:48:07.849 DEBUG:boto:Token: None 2024-02-01T10:48:07.849 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:07 GMT /vclfcn-59/ 2024-02-01T10:48:07.849 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NwdGjBpCSV0xKw6akevMJFGrGXM= 2024-02-01T10:48:07.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NwdGjBpCSV0xKw6akevMJFGrGXM=', 'Content-Length': '0'} 2024-02-01T10:48:07.854 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c55a90c66459802-0065bb76e7-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:07.854 DEBUG:boto:b'vclfcn-59obj71000falseobj72024-02-01T10:47:30.227Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:07.855 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:07.855 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:07.855 DEBUG:boto:path=/vclfcn-59/?prefix=obj7 2024-02-01T10:48:07.855 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj7 2024-02-01T10:48:07.855 DEBUG:boto:Method: GET 2024-02-01T10:48:07.855 DEBUG:boto:Path: /vclfcn-59/?prefix=obj7 2024-02-01T10:48:07.855 DEBUG:boto:Data: 2024-02-01T10:48:07.855 DEBUG:boto:Headers: {} 2024-02-01T10:48:07.856 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:07.856 DEBUG:boto:Port: 8000 2024-02-01T10:48:07.856 DEBUG:boto:Params: {} 2024-02-01T10:48:07.856 DEBUG:boto:Token: None 2024-02-01T10:48:07.856 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:07 GMT /vclfcn-59/ 2024-02-01T10:48:07.856 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NwdGjBpCSV0xKw6akevMJFGrGXM= 2024-02-01T10:48:07.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NwdGjBpCSV0xKw6akevMJFGrGXM=', 'Content-Length': '0'} 2024-02-01T10:48:07.861 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c25fa54d2149c90-0065bb76e7-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:07.861 DEBUG:boto:b'vclfcn-59obj71000falseobj72024-02-01T10:47:30.227Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:07.861 DEBUG:boto:path=/vclfcn-59/obj7 2024-02-01T10:48:07.862 DEBUG:boto:auth_path=/vclfcn-59/obj7 2024-02-01T10:48:07.862 DEBUG:boto:Method: HEAD 2024-02-01T10:48:07.862 DEBUG:boto:Path: /vclfcn-59/obj7 2024-02-01T10:48:07.862 DEBUG:boto:Data: 2024-02-01T10:48:07.862 DEBUG:boto:Headers: {} 2024-02-01T10:48:07.862 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:07.862 DEBUG:boto:Port: 8001 2024-02-01T10:48:07.862 DEBUG:boto:Params: {} 2024-02-01T10:48:07.862 DEBUG:boto:Token: None 2024-02-01T10:48:07.862 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:07 GMT /vclfcn-59/obj7 2024-02-01T10:48:07.863 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RP5GTZ5rfSI55JiBh7+5kww1a10= 2024-02-01T10:48:07.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RP5GTZ5rfSI55JiBh7+5kww1a10=', 'Content-Length': '0'} 2024-02-01T10:48:07.865 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000cacb8e08c88a10b-0065bb76e7-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:07.866 DEBUG:boto:path=/vclfcn-59/obj7 2024-02-01T10:48:07.866 DEBUG:boto:auth_path=/vclfcn-59/obj7 2024-02-01T10:48:07.866 DEBUG:boto:Method: HEAD 2024-02-01T10:48:07.866 DEBUG:boto:Path: /vclfcn-59/obj7 2024-02-01T10:48:07.866 DEBUG:boto:Data: 2024-02-01T10:48:07.866 DEBUG:boto:Headers: {} 2024-02-01T10:48:07.866 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:07.866 DEBUG:boto:Port: 8000 2024-02-01T10:48:07.866 DEBUG:boto:Params: {} 2024-02-01T10:48:07.866 DEBUG:boto:Token: None 2024-02-01T10:48:07.867 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:07 GMT /vclfcn-59/obj7 2024-02-01T10:48:07.867 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RP5GTZ5rfSI55JiBh7+5kww1a10= 2024-02-01T10:48:07.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RP5GTZ5rfSI55JiBh7+5kww1a10=', 'Content-Length': '0'} 2024-02-01T10:48:07.869 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e5124e6e1570b2e6-0065bb76e7-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:07.870 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-02-01T10:48:07.870 DEBUG:boto:path=/vclfcn-59/obj7 2024-02-01T10:48:07.870 DEBUG:boto:auth_path=/vclfcn-59/obj7 2024-02-01T10:48:07.870 DEBUG:boto:Method: GET 2024-02-01T10:48:07.870 DEBUG:boto:Path: /vclfcn-59/obj7 2024-02-01T10:48:07.870 DEBUG:boto:Data: 2024-02-01T10:48:07.870 DEBUG:boto:Headers: {} 2024-02-01T10:48:07.870 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:07.870 DEBUG:boto:Port: 8001 2024-02-01T10:48:07.871 DEBUG:boto:Params: {} 2024-02-01T10:48:07.871 DEBUG:boto:Token: None 2024-02-01T10:48:07.871 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:07 GMT /vclfcn-59/obj7 2024-02-01T10:48:07.871 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:f1vDSMiKEzXTuM99t+O/nQwiNSY= 2024-02-01T10:48:07.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:f1vDSMiKEzXTuM99t+O/nQwiNSY=', 'Content-Length': '0'} 2024-02-01T10:48:07.874 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a7423cf8be5b8215-0065bb76e7-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:07.916 DEBUG:boto:path=/vclfcn-59/obj7 2024-02-01T10:48:07.916 DEBUG:boto:auth_path=/vclfcn-59/obj7 2024-02-01T10:48:07.916 DEBUG:boto:Method: GET 2024-02-01T10:48:07.916 DEBUG:boto:Path: /vclfcn-59/obj7 2024-02-01T10:48:07.917 DEBUG:boto:Data: 2024-02-01T10:48:07.917 DEBUG:boto:Headers: {} 2024-02-01T10:48:07.917 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:07.917 DEBUG:boto:Port: 8000 2024-02-01T10:48:07.917 DEBUG:boto:Params: {} 2024-02-01T10:48:07.917 DEBUG:boto:Token: None 2024-02-01T10:48:07.917 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:07 GMT /vclfcn-59/obj7 2024-02-01T10:48:07.917 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:f1vDSMiKEzXTuM99t+O/nQwiNSY= 2024-02-01T10:48:07.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:f1vDSMiKEzXTuM99t+O/nQwiNSY=', 'Content-Length': '0'} 2024-02-01T10:48:07.920 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d610e9000821c610-0065bb76e7-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:07.968 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:07.968 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:07.968 DEBUG:boto:path=/vclfcn-59/?prefix=obj7 2024-02-01T10:48:07.968 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj7 2024-02-01T10:48:07.968 DEBUG:boto:Method: GET 2024-02-01T10:48:07.969 DEBUG:boto:Path: /vclfcn-59/?prefix=obj7 2024-02-01T10:48:07.969 DEBUG:boto:Data: 2024-02-01T10:48:07.969 DEBUG:boto:Headers: {} 2024-02-01T10:48:07.969 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:07.969 DEBUG:boto:Port: 8001 2024-02-01T10:48:07.969 DEBUG:boto:Params: {} 2024-02-01T10:48:07.969 DEBUG:boto:Token: None 2024-02-01T10:48:07.969 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:07 GMT /vclfcn-59/ 2024-02-01T10:48:07.969 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NwdGjBpCSV0xKw6akevMJFGrGXM= 2024-02-01T10:48:07.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NwdGjBpCSV0xKw6akevMJFGrGXM=', 'Content-Length': '0'} 2024-02-01T10:48:07.973 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b68f0246f383c5f8-0065bb76e7-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:08.016 DEBUG:boto:b'vclfcn-59obj71000falseobj72024-02-01T10:47:30.227Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:08.016 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:08.016 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:08.017 DEBUG:boto:path=/vclfcn-59/?prefix=obj7 2024-02-01T10:48:08.017 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj7 2024-02-01T10:48:08.017 DEBUG:boto:Method: GET 2024-02-01T10:48:08.017 DEBUG:boto:Path: /vclfcn-59/?prefix=obj7 2024-02-01T10:48:08.017 DEBUG:boto:Data: 2024-02-01T10:48:08.017 DEBUG:boto:Headers: {} 2024-02-01T10:48:08.017 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:08.017 DEBUG:boto:Port: 8000 2024-02-01T10:48:08.017 DEBUG:boto:Params: {} 2024-02-01T10:48:08.017 DEBUG:boto:Token: None 2024-02-01T10:48:08.018 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:08 GMT /vclfcn-59/ 2024-02-01T10:48:08.018 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1jao1FKZ87N/Rfw+SOfaAaYnhZs= 2024-02-01T10:48:08.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1jao1FKZ87N/Rfw+SOfaAaYnhZs=', 'Content-Length': '0'} 2024-02-01T10:48:08.022 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071e280b35356cb72-0065bb76e8-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:08.022 DEBUG:boto:b'vclfcn-59obj71000falseobj72024-02-01T10:47:30.227Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:08.022 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-02-01T10:48:08.022 DEBUG:boto:path=/vclfcn-59/obj8 2024-02-01T10:48:08.022 DEBUG:boto:auth_path=/vclfcn-59/obj8 2024-02-01T10:48:08.022 DEBUG:boto:path=/vclfcn-59/obj8?versionId=null 2024-02-01T10:48:08.023 DEBUG:boto:auth_path=/vclfcn-59/obj8?versionId=null 2024-02-01T10:48:08.023 DEBUG:boto:Method: GET 2024-02-01T10:48:08.023 DEBUG:boto:Path: /vclfcn-59/obj8?versionId=null 2024-02-01T10:48:08.023 DEBUG:boto:Data: 2024-02-01T10:48:08.023 DEBUG:boto:Headers: {} 2024-02-01T10:48:08.023 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:08.023 DEBUG:boto:Port: 8001 2024-02-01T10:48:08.023 DEBUG:boto:Params: {} 2024-02-01T10:48:08.023 DEBUG:boto:Token: None 2024-02-01T10:48:08.024 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:08 GMT /vclfcn-59/obj8?versionId=null 2024-02-01T10:48:08.024 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zoeWfaH/nnb78FzOi1ORzv0E+Gc= 2024-02-01T10:48:08.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zoeWfaH/nnb78FzOi1ORzv0E+Gc=', 'Content-Length': '0'} 2024-02-01T10:48:08.026 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000416647773e23d713-0065bb76e8-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:08.072 DEBUG:boto:path=/vclfcn-59/obj8 2024-02-01T10:48:08.073 DEBUG:boto:auth_path=/vclfcn-59/obj8 2024-02-01T10:48:08.073 DEBUG:boto:path=/vclfcn-59/obj8?versionId=null 2024-02-01T10:48:08.073 DEBUG:boto:auth_path=/vclfcn-59/obj8?versionId=null 2024-02-01T10:48:08.073 DEBUG:boto:Method: GET 2024-02-01T10:48:08.073 DEBUG:boto:Path: /vclfcn-59/obj8?versionId=null 2024-02-01T10:48:08.073 DEBUG:boto:Data: 2024-02-01T10:48:08.073 DEBUG:boto:Headers: {} 2024-02-01T10:48:08.073 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:08.073 DEBUG:boto:Port: 8000 2024-02-01T10:48:08.074 DEBUG:boto:Params: {} 2024-02-01T10:48:08.074 DEBUG:boto:Token: None 2024-02-01T10:48:08.074 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:08 GMT /vclfcn-59/obj8?versionId=null 2024-02-01T10:48:08.074 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zoeWfaH/nnb78FzOi1ORzv0E+Gc= 2024-02-01T10:48:08.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zoeWfaH/nnb78FzOi1ORzv0E+Gc=', 'Content-Length': '0'} 2024-02-01T10:48:08.077 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f29c408496323eca-0065bb76e8-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:08.078 DEBUG:boto:path=/vclfcn-59/obj8 2024-02-01T10:48:08.078 DEBUG:boto:auth_path=/vclfcn-59/obj8 2024-02-01T10:48:08.078 DEBUG:boto:path=/vclfcn-59/obj8?versionId=null 2024-02-01T10:48:08.078 DEBUG:boto:auth_path=/vclfcn-59/obj8?versionId=null 2024-02-01T10:48:08.078 DEBUG:boto:Method: HEAD 2024-02-01T10:48:08.078 DEBUG:boto:Path: /vclfcn-59/obj8?versionId=null 2024-02-01T10:48:08.078 DEBUG:boto:Data: 2024-02-01T10:48:08.078 DEBUG:boto:Headers: {} 2024-02-01T10:48:08.078 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:08.078 DEBUG:boto:Port: 8001 2024-02-01T10:48:08.079 DEBUG:boto:Params: {} 2024-02-01T10:48:08.079 DEBUG:boto:Token: None 2024-02-01T10:48:08.079 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:08 GMT /vclfcn-59/obj8?versionId=null 2024-02-01T10:48:08.079 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FFVOg9Ep2ImWbno2YGalzmrQLXg= 2024-02-01T10:48:08.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FFVOg9Ep2ImWbno2YGalzmrQLXg=', 'Content-Length': '0'} 2024-02-01T10:48:08.082 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007169d78171e14c82-0065bb76e8-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:08.082 DEBUG:boto:path=/vclfcn-59/obj8 2024-02-01T10:48:08.082 DEBUG:boto:auth_path=/vclfcn-59/obj8 2024-02-01T10:48:08.082 DEBUG:boto:path=/vclfcn-59/obj8?versionId=null 2024-02-01T10:48:08.082 DEBUG:boto:auth_path=/vclfcn-59/obj8?versionId=null 2024-02-01T10:48:08.082 DEBUG:boto:Method: HEAD 2024-02-01T10:48:08.082 DEBUG:boto:Path: /vclfcn-59/obj8?versionId=null 2024-02-01T10:48:08.082 DEBUG:boto:Data: 2024-02-01T10:48:08.083 DEBUG:boto:Headers: {} 2024-02-01T10:48:08.083 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:08.083 DEBUG:boto:Port: 8000 2024-02-01T10:48:08.083 DEBUG:boto:Params: {} 2024-02-01T10:48:08.083 DEBUG:boto:Token: None 2024-02-01T10:48:08.083 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:08 GMT /vclfcn-59/obj8?versionId=null 2024-02-01T10:48:08.083 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FFVOg9Ep2ImWbno2YGalzmrQLXg= 2024-02-01T10:48:08.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FFVOg9Ep2ImWbno2YGalzmrQLXg=', 'Content-Length': '0'} 2024-02-01T10:48:08.086 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002e79df624c8aacf1-0065bb76e8-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:08.087 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-02-01T10:48:08.087 DEBUG:boto:path=/vclfcn-59/obj8 2024-02-01T10:48:08.087 DEBUG:boto:auth_path=/vclfcn-59/obj8 2024-02-01T10:48:08.087 DEBUG:boto:path=/vclfcn-59/obj8?versionId=null 2024-02-01T10:48:08.087 DEBUG:boto:auth_path=/vclfcn-59/obj8?versionId=null 2024-02-01T10:48:08.087 DEBUG:boto:Method: GET 2024-02-01T10:48:08.088 DEBUG:boto:Path: /vclfcn-59/obj8?versionId=null 2024-02-01T10:48:08.088 DEBUG:boto:Data: 2024-02-01T10:48:08.088 DEBUG:boto:Headers: {} 2024-02-01T10:48:08.088 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:08.088 DEBUG:boto:Port: 8001 2024-02-01T10:48:08.088 DEBUG:boto:Params: {} 2024-02-01T10:48:08.088 DEBUG:boto:Token: None 2024-02-01T10:48:08.088 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:08 GMT /vclfcn-59/obj8?versionId=null 2024-02-01T10:48:08.088 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zoeWfaH/nnb78FzOi1ORzv0E+Gc= 2024-02-01T10:48:08.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zoeWfaH/nnb78FzOi1ORzv0E+Gc=', 'Content-Length': '0'} 2024-02-01T10:48:08.091 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a736eaddf01236e1-0065bb76e8-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:08.132 DEBUG:boto:path=/vclfcn-59/obj8 2024-02-01T10:48:08.132 DEBUG:boto:auth_path=/vclfcn-59/obj8 2024-02-01T10:48:08.132 DEBUG:boto:path=/vclfcn-59/obj8?versionId=null 2024-02-01T10:48:08.132 DEBUG:boto:auth_path=/vclfcn-59/obj8?versionId=null 2024-02-01T10:48:08.132 DEBUG:boto:Method: GET 2024-02-01T10:48:08.133 DEBUG:boto:Path: /vclfcn-59/obj8?versionId=null 2024-02-01T10:48:08.133 DEBUG:boto:Data: 2024-02-01T10:48:08.133 DEBUG:boto:Headers: {} 2024-02-01T10:48:08.133 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:08.133 DEBUG:boto:Port: 8000 2024-02-01T10:48:08.133 DEBUG:boto:Params: {} 2024-02-01T10:48:08.133 DEBUG:boto:Token: None 2024-02-01T10:48:08.133 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:08 GMT /vclfcn-59/obj8?versionId=null 2024-02-01T10:48:08.133 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zoeWfaH/nnb78FzOi1ORzv0E+Gc= 2024-02-01T10:48:08.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zoeWfaH/nnb78FzOi1ORzv0E+Gc=', 'Content-Length': '0'} 2024-02-01T10:48:08.136 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005938e6e8689d1ad2-0065bb76e8-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:08.180 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:08.180 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:08.180 DEBUG:boto:path=/vclfcn-59/?prefix=obj8 2024-02-01T10:48:08.180 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj8 2024-02-01T10:48:08.181 DEBUG:boto:Method: GET 2024-02-01T10:48:08.181 DEBUG:boto:Path: /vclfcn-59/?prefix=obj8 2024-02-01T10:48:08.181 DEBUG:boto:Data: 2024-02-01T10:48:08.181 DEBUG:boto:Headers: {} 2024-02-01T10:48:08.181 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:08.181 DEBUG:boto:Port: 8001 2024-02-01T10:48:08.181 DEBUG:boto:Params: {} 2024-02-01T10:48:08.181 DEBUG:boto:Token: None 2024-02-01T10:48:08.181 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:08 GMT /vclfcn-59/ 2024-02-01T10:48:08.181 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1jao1FKZ87N/Rfw+SOfaAaYnhZs= 2024-02-01T10:48:08.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1jao1FKZ87N/Rfw+SOfaAaYnhZs=', 'Content-Length': '0'} 2024-02-01T10:48:08.186 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009998b19789064562-0065bb76e8-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:08.186 DEBUG:boto:b'vclfcn-59obj81000falseobj82024-02-01T10:47:30.275Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:08.186 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:08.187 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:08.187 DEBUG:boto:path=/vclfcn-59/?prefix=obj8 2024-02-01T10:48:08.187 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj8 2024-02-01T10:48:08.187 DEBUG:boto:Method: GET 2024-02-01T10:48:08.187 DEBUG:boto:Path: /vclfcn-59/?prefix=obj8 2024-02-01T10:48:08.187 DEBUG:boto:Data: 2024-02-01T10:48:08.187 DEBUG:boto:Headers: {} 2024-02-01T10:48:08.187 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:08.187 DEBUG:boto:Port: 8000 2024-02-01T10:48:08.187 DEBUG:boto:Params: {} 2024-02-01T10:48:08.188 DEBUG:boto:Token: None 2024-02-01T10:48:08.188 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:08 GMT /vclfcn-59/ 2024-02-01T10:48:08.188 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1jao1FKZ87N/Rfw+SOfaAaYnhZs= 2024-02-01T10:48:08.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1jao1FKZ87N/Rfw+SOfaAaYnhZs=', 'Content-Length': '0'} 2024-02-01T10:48:08.192 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d505a996796e73ff-0065bb76e8-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:08.193 DEBUG:boto:b'vclfcn-59obj81000falseobj82024-02-01T10:47:30.275Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:08.193 DEBUG:boto:path=/vclfcn-59/obj8 2024-02-01T10:48:08.193 DEBUG:boto:auth_path=/vclfcn-59/obj8 2024-02-01T10:48:08.193 DEBUG:boto:Method: HEAD 2024-02-01T10:48:08.193 DEBUG:boto:Path: /vclfcn-59/obj8 2024-02-01T10:48:08.193 DEBUG:boto:Data: 2024-02-01T10:48:08.193 DEBUG:boto:Headers: {} 2024-02-01T10:48:08.194 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:08.194 DEBUG:boto:Port: 8001 2024-02-01T10:48:08.194 DEBUG:boto:Params: {} 2024-02-01T10:48:08.194 DEBUG:boto:Token: None 2024-02-01T10:48:08.194 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:08 GMT /vclfcn-59/obj8 2024-02-01T10:48:08.194 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:4AKUfR/7lwGi3HMx5q6derqFSvQ= 2024-02-01T10:48:08.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:4AKUfR/7lwGi3HMx5q6derqFSvQ=', 'Content-Length': '0'} 2024-02-01T10:48:08.197 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000184a4cab0ab604fe-0065bb76e8-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:08.197 DEBUG:boto:path=/vclfcn-59/obj8 2024-02-01T10:48:08.197 DEBUG:boto:auth_path=/vclfcn-59/obj8 2024-02-01T10:48:08.197 DEBUG:boto:Method: HEAD 2024-02-01T10:48:08.198 DEBUG:boto:Path: /vclfcn-59/obj8 2024-02-01T10:48:08.198 DEBUG:boto:Data: 2024-02-01T10:48:08.198 DEBUG:boto:Headers: {} 2024-02-01T10:48:08.198 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:08.198 DEBUG:boto:Port: 8000 2024-02-01T10:48:08.198 DEBUG:boto:Params: {} 2024-02-01T10:48:08.198 DEBUG:boto:Token: None 2024-02-01T10:48:08.198 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:08 GMT /vclfcn-59/obj8 2024-02-01T10:48:08.198 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:4AKUfR/7lwGi3HMx5q6derqFSvQ= 2024-02-01T10:48:08.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:4AKUfR/7lwGi3HMx5q6derqFSvQ=', 'Content-Length': '0'} 2024-02-01T10:48:08.201 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009e2e8b810ada05bb-0065bb76e8-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:08.201 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-02-01T10:48:08.202 DEBUG:boto:path=/vclfcn-59/obj8 2024-02-01T10:48:08.202 DEBUG:boto:auth_path=/vclfcn-59/obj8 2024-02-01T10:48:08.202 DEBUG:boto:Method: GET 2024-02-01T10:48:08.202 DEBUG:boto:Path: /vclfcn-59/obj8 2024-02-01T10:48:08.202 DEBUG:boto:Data: 2024-02-01T10:48:08.202 DEBUG:boto:Headers: {} 2024-02-01T10:48:08.202 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:08.202 DEBUG:boto:Port: 8001 2024-02-01T10:48:08.202 DEBUG:boto:Params: {} 2024-02-01T10:48:08.202 DEBUG:boto:Token: None 2024-02-01T10:48:08.203 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:08 GMT /vclfcn-59/obj8 2024-02-01T10:48:08.203 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wCYhqMzbnVfjkPqgxCUiyABnp5k= 2024-02-01T10:48:08.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wCYhqMzbnVfjkPqgxCUiyABnp5k=', 'Content-Length': '0'} 2024-02-01T10:48:08.205 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a319e9946806f544-0065bb76e8-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:08.248 DEBUG:boto:path=/vclfcn-59/obj8 2024-02-01T10:48:08.248 DEBUG:boto:auth_path=/vclfcn-59/obj8 2024-02-01T10:48:08.249 DEBUG:boto:Method: GET 2024-02-01T10:48:08.249 DEBUG:boto:Path: /vclfcn-59/obj8 2024-02-01T10:48:08.249 DEBUG:boto:Data: 2024-02-01T10:48:08.249 DEBUG:boto:Headers: {} 2024-02-01T10:48:08.249 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:08.249 DEBUG:boto:Port: 8000 2024-02-01T10:48:08.249 DEBUG:boto:Params: {} 2024-02-01T10:48:08.249 DEBUG:boto:Token: None 2024-02-01T10:48:08.249 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:08 GMT /vclfcn-59/obj8 2024-02-01T10:48:08.249 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wCYhqMzbnVfjkPqgxCUiyABnp5k= 2024-02-01T10:48:08.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wCYhqMzbnVfjkPqgxCUiyABnp5k=', 'Content-Length': '0'} 2024-02-01T10:48:08.252 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-59:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004a6c778d84894a56-0065bb76e8-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:08.296 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:08.296 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:08.296 DEBUG:boto:path=/vclfcn-59/?prefix=obj8 2024-02-01T10:48:08.297 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj8 2024-02-01T10:48:08.297 DEBUG:boto:Method: GET 2024-02-01T10:48:08.297 DEBUG:boto:Path: /vclfcn-59/?prefix=obj8 2024-02-01T10:48:08.297 DEBUG:boto:Data: 2024-02-01T10:48:08.297 DEBUG:boto:Headers: {} 2024-02-01T10:48:08.297 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:08.297 DEBUG:boto:Port: 8001 2024-02-01T10:48:08.297 DEBUG:boto:Params: {} 2024-02-01T10:48:08.297 DEBUG:boto:Token: None 2024-02-01T10:48:08.298 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:08 GMT /vclfcn-59/ 2024-02-01T10:48:08.298 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1jao1FKZ87N/Rfw+SOfaAaYnhZs= 2024-02-01T10:48:08.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1jao1FKZ87N/Rfw+SOfaAaYnhZs=', 'Content-Length': '0'} 2024-02-01T10:48:08.303 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077ba8dc7127b9393-0065bb76e8-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:08.344 DEBUG:boto:b'vclfcn-59obj81000falseobj82024-02-01T10:47:30.275Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:08.344 DEBUG:boto:path=/vclfcn-59/ 2024-02-01T10:48:08.345 DEBUG:boto:auth_path=/vclfcn-59/ 2024-02-01T10:48:08.345 DEBUG:boto:path=/vclfcn-59/?prefix=obj8 2024-02-01T10:48:08.345 DEBUG:boto:auth_path=/vclfcn-59/?prefix=obj8 2024-02-01T10:48:08.345 DEBUG:boto:Method: GET 2024-02-01T10:48:08.345 DEBUG:boto:Path: /vclfcn-59/?prefix=obj8 2024-02-01T10:48:08.345 DEBUG:boto:Data: 2024-02-01T10:48:08.345 DEBUG:boto:Headers: {} 2024-02-01T10:48:08.345 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:08.345 DEBUG:boto:Port: 8000 2024-02-01T10:48:08.345 DEBUG:boto:Params: {} 2024-02-01T10:48:08.346 DEBUG:boto:Token: None 2024-02-01T10:48:08.346 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:08 GMT /vclfcn-59/ 2024-02-01T10:48:08.346 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1jao1FKZ87N/Rfw+SOfaAaYnhZs= 2024-02-01T10:48:08.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1jao1FKZ87N/Rfw+SOfaAaYnhZs=', 'Content-Length': '0'} 2024-02-01T10:48:08.350 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2097002ab0e5b9e-0065bb76e8-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:08.392 DEBUG:boto:b'vclfcn-59obj81000falseobj82024-02-01T10:47:30.275Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:08.392 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-59 zones={test-zone3, test-zone2} 2024-02-01T10:48:08.393 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=vclfcn-60 2024-02-01T10:48:08.393 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:48:08.393 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:48:08.393 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket vclfcn-60 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-02-01T10:48:08.443 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:48:08.443 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:48:08.512 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:48:08.510+0000 7f0e6abd1980 1 bucket sync caught up with source: 2024-02-01T10:48:08.512 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [00000000004.65.6, 00000000002.79.1, 00000000004.68.6, 00000000002.48.1, 00000000004.59.6, 00000000002.56.1, 00000000002.57.1, 00000000004.50.6, 00000000002.47.1, 00000000002.46.1, 00000000002.38.1] 2024-02-01T10:48:08.512 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [00000000004.65.6, 00000000002.79.1, 00000000004.68.6, 00000000002.48.1, 00000000004.59.6, 00000000002.56.1, 00000000002.57.1, 00000000004.50.6, 00000000002.47.1, 00000000002.46.1, 00000000002.38.1] 2024-02-01T10:48:08.512 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:48:08.510+0000 7f0e6abd1980 0 bucket checkpoint complete 2024-02-01T10:48:08.535 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=vclfcn-60 2024-02-01T10:48:08.535 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:48:08.536 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:48:08.536 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-60 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:48:08.587 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:48:08.587 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:48:08.647 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:48:08.648+0000 7f4704eec980 1 bucket sync caught up with source: 2024-02-01T10:48:08.647 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [00000000004.65.6, 00000000002.79.1, 00000000004.68.6, 00000000002.48.1, 00000000004.59.6, 00000000002.56.1, 00000000002.57.1, 00000000004.50.6, 00000000002.47.1, 00000000002.46.1, 00000000002.38.1] 2024-02-01T10:48:08.648 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [00000000004.65.6, 00000000002.79.1, 00000000004.68.6, 00000000002.48.1, 00000000004.59.6, 00000000002.56.1, 00000000002.57.1, 00000000004.50.6, 00000000002.47.1, 00000000002.46.1, 00000000002.38.1] 2024-02-01T10:48:08.648 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:48:08.648+0000 7f4704eec980 0 bucket checkpoint complete 2024-02-01T10:48:08.670 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=vclfcn-60 2024-02-01T10:48:08.670 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-60', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:48:08.670 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-60', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:48:08.670 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-60 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:48:08.710 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:48:08.711 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:48:08.773 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:48:08.772+0000 7f2f322b9980 1 bucket sync caught up with source: 2024-02-01T10:48:08.773 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [00000000005.294.1, 00000000003.304.1, 00000000005.238.1, 00000000003.298.1, 00000000005.323.1, 00000000003.324.1, 00000000003.325.1, 00000000005.199.1, 00000000003.264.1, 00000000003.238.1, 00000000003.207.1] 2024-02-01T10:48:08.773 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [00000000005.294.1, 00000000003.304.1, 00000000005.238.1, 00000000003.298.1, 00000000005.323.1, 00000000003.324.1, 00000000003.325.1, 00000000005.199.1, 00000000003.264.1, 00000000003.238.1, 00000000003.207.1] 2024-02-01T10:48:08.774 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:48:08.772+0000 7f2f322b9980 0 bucket checkpoint complete 2024-02-01T10:48:08.797 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=vclfcn-60 2024-02-01T10:48:08.797 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-60', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:48:08.797 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-60', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:48:08.797 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-60 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:48:08.880 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:48:08.881 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:48:08.941 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:48:08.940+0000 7fbb851d0980 1 bucket sync caught up with source: 2024-02-01T10:48:08.942 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [00000000005.294.1, 00000000003.304.1, 00000000005.238.1, 00000000003.298.1, 00000000005.323.1, 00000000003.324.1, 00000000003.325.1, 00000000005.199.1, 00000000003.264.1, 00000000003.238.1, 00000000003.207.1] 2024-02-01T10:48:08.942 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [00000000005.294.1, 00000000003.304.1, 00000000005.238.1, 00000000003.298.1, 00000000005.323.1, 00000000003.324.1, 00000000003.325.1, 00000000005.199.1, 00000000003.264.1, 00000000003.238.1, 00000000003.207.1] 2024-02-01T10:48:08.942 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:48:08.940+0000 7fbb851d0980 0 bucket checkpoint complete 2024-02-01T10:48:08.964 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=vclfcn-60 2024-02-01T10:48:08.964 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-60', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:48:08.964 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-60', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:48:08.964 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-60 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:48:09.053 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:48:09.054 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:48:09.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:48:09.112+0000 7f7478a98980 1 bucket sync caught up with source: 2024-02-01T10:48:09.113 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [00000000002.199.6, , 00000000002.242.6, , 00000000002.331.6, , , 00000000002.213.6, , , ] 2024-02-01T10:48:09.113 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [00000000002.199.6, , 00000000002.242.6, , 00000000002.331.6, , , 00000000002.213.6, , , ] 2024-02-01T10:48:09.113 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:48:09.112+0000 7f7478a98980 0 bucket checkpoint complete 2024-02-01T10:48:09.135 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=vclfcn-60 2024-02-01T10:48:09.136 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-60', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:48:09.136 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-60', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:48:09.136 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket vclfcn-60 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-02-01T10:48:09.186 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:48:09.186 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:48:09.252 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:48:09.250+0000 7f23c0c15980 1 bucket sync caught up with source: 2024-02-01T10:48:09.252 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [00000000002.199.6, , 00000000002.242.6, , 00000000002.331.6, , , 00000000002.213.6, , , ] 2024-02-01T10:48:09.252 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [00000000002.199.6, , 00000000002.242.6, , 00000000002.331.6, , , 00000000002.213.6, , , ] 2024-02-01T10:48:09.252 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:48:09.250+0000 7f23c0c15980 0 bucket checkpoint complete 2024-02-01T10:48:09.278 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-60 zones={test-zone2, test-zone1} 2024-02-01T10:48:09.278 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:09.279 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:09.279 DEBUG:boto:Method: HEAD 2024-02-01T10:48:09.279 DEBUG:boto:Path: /vclfcn-60/ 2024-02-01T10:48:09.279 DEBUG:boto:Data: 2024-02-01T10:48:09.280 DEBUG:boto:Headers: {} 2024-02-01T10:48:09.280 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:09.280 DEBUG:boto:Port: 8000 2024-02-01T10:48:09.280 DEBUG:boto:Params: {} 2024-02-01T10:48:09.281 DEBUG:boto:Token: None 2024-02-01T10:48:09.281 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:09 GMT /vclfcn-60/ 2024-02-01T10:48:09.281 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:UEEZliqkyNMH0pUS7mVYRmBt+tk= 2024-02-01T10:48:09.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:UEEZliqkyNMH0pUS7mVYRmBt+tk=', 'Content-Length': '0'} 2024-02-01T10:48:09.284 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fb329240fd795d96-0065bb76e9-5350-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:48:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:09.284 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:09.285 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:09.285 DEBUG:boto:Method: HEAD 2024-02-01T10:48:09.285 DEBUG:boto:Path: /vclfcn-60/ 2024-02-01T10:48:09.285 DEBUG:boto:Data: 2024-02-01T10:48:09.285 DEBUG:boto:Headers: {} 2024-02-01T10:48:09.285 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:09.286 DEBUG:boto:Port: 8000 2024-02-01T10:48:09.286 DEBUG:boto:Params: {} 2024-02-01T10:48:09.286 DEBUG:boto:Token: None 2024-02-01T10:48:09.286 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:09 GMT /vclfcn-60/ 2024-02-01T10:48:09.287 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:UEEZliqkyNMH0pUS7mVYRmBt+tk= 2024-02-01T10:48:09.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:UEEZliqkyNMH0pUS7mVYRmBt+tk=', 'Content-Length': '0'} 2024-02-01T10:48:09.291 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000072bbe5c750e74019-0065bb76e9-5765-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:48:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:09.292 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:48:09.292 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:09.292 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:09.293 DEBUG:boto:path=/vclfcn-60/?versions&max-keys=999 2024-02-01T10:48:09.293 DEBUG:boto:auth_path=/vclfcn-60/?versions&max-keys=999 2024-02-01T10:48:09.293 DEBUG:boto:Method: GET 2024-02-01T10:48:09.293 DEBUG:boto:Path: /vclfcn-60/?versions&max-keys=999 2024-02-01T10:48:09.294 DEBUG:boto:Data: 2024-02-01T10:48:09.294 DEBUG:boto:Headers: {} 2024-02-01T10:48:09.294 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:09.294 DEBUG:boto:Port: 8000 2024-02-01T10:48:09.294 DEBUG:boto:Params: {} 2024-02-01T10:48:09.295 DEBUG:boto:Token: None 2024-02-01T10:48:09.295 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:09 GMT /vclfcn-60/?versions 2024-02-01T10:48:09.295 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:U1h8v6NOoFkEwn99r6fDWhsG0Fg= 2024-02-01T10:48:09.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:U1h8v6NOoFkEwn99r6fDWhsG0Fg=', 'Content-Length': '0'} 2024-02-01T10:48:09.298 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ffbb8adc6448d1b0-0065bb76e9-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:09.298 DEBUG:boto:b'vclfcn-60999falseobj1nulltrue2024-02-01T10:46:46.243Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:46:46.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:46:46.285Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:46:46.336Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-02-01T10:47:30.291Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-02-01T10:47:30.343Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-02-01T10:47:30.358Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-02-01T10:47:30.408Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:09.299 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-02-01T10:48:09.299 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-02-01T10:48:09.299 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-02-01T10:48:09.299 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-02-01T10:48:09.300 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-02-01T10:48:09.300 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-02-01T10:48:09.300 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-02-01T10:48:09.300 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-02-01T10:48:09.300 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:48:09.300 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:09.300 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:09.300 DEBUG:boto:path=/vclfcn-60/?versions&max-keys=999 2024-02-01T10:48:09.300 DEBUG:boto:auth_path=/vclfcn-60/?versions&max-keys=999 2024-02-01T10:48:09.301 DEBUG:boto:Method: GET 2024-02-01T10:48:09.301 DEBUG:boto:Path: /vclfcn-60/?versions&max-keys=999 2024-02-01T10:48:09.301 DEBUG:boto:Data: 2024-02-01T10:48:09.301 DEBUG:boto:Headers: {} 2024-02-01T10:48:09.301 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:09.301 DEBUG:boto:Port: 8000 2024-02-01T10:48:09.301 DEBUG:boto:Params: {} 2024-02-01T10:48:09.301 DEBUG:boto:Token: None 2024-02-01T10:48:09.301 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:09 GMT /vclfcn-60/?versions 2024-02-01T10:48:09.301 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:U1h8v6NOoFkEwn99r6fDWhsG0Fg= 2024-02-01T10:48:09.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:U1h8v6NOoFkEwn99r6fDWhsG0Fg=', 'Content-Length': '0'} 2024-02-01T10:48:09.306 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c598c759cc2fc78a-0065bb76e9-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:09.307 DEBUG:boto:b'vclfcn-60999falseobj1nulltrue2024-02-01T10:46:46.243Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:46:46.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:46:46.285Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:46:46.336Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-02-01T10:47:30.291Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-02-01T10:47:30.343Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-02-01T10:47:30.358Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-02-01T10:47:30.408Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:09.307 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-02-01T10:48:09.307 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-02-01T10:48:09.308 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-02-01T10:48:09.308 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-02-01T10:48:09.308 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-02-01T10:48:09.308 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-02-01T10:48:09.308 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-02-01T10:48:09.308 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-02-01T10:48:09.308 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:09.308 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:09.308 DEBUG:boto:path=/vclfcn-60/?versions&max-keys=999 2024-02-01T10:48:09.309 DEBUG:boto:auth_path=/vclfcn-60/?versions&max-keys=999 2024-02-01T10:48:09.309 DEBUG:boto:Method: GET 2024-02-01T10:48:09.309 DEBUG:boto:Path: /vclfcn-60/?versions&max-keys=999 2024-02-01T10:48:09.309 DEBUG:boto:Data: 2024-02-01T10:48:09.309 DEBUG:boto:Headers: {} 2024-02-01T10:48:09.309 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:09.309 DEBUG:boto:Port: 8000 2024-02-01T10:48:09.309 DEBUG:boto:Params: {} 2024-02-01T10:48:09.309 DEBUG:boto:Token: None 2024-02-01T10:48:09.309 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:09 GMT /vclfcn-60/?versions 2024-02-01T10:48:09.310 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:U1h8v6NOoFkEwn99r6fDWhsG0Fg= 2024-02-01T10:48:09.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:U1h8v6NOoFkEwn99r6fDWhsG0Fg=', 'Content-Length': '0'} 2024-02-01T10:48:09.312 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ecedad579c6ba6f4-0065bb76e9-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:09.312 DEBUG:boto:b'vclfcn-60999falseobj1nulltrue2024-02-01T10:46:46.243Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:46:46.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:46:46.285Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:46:46.336Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-02-01T10:47:30.291Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-02-01T10:47:30.343Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-02-01T10:47:30.358Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-02-01T10:47:30.408Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:09.313 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:09.313 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:09.313 DEBUG:boto:path=/vclfcn-60/?versions&max-keys=999 2024-02-01T10:48:09.313 DEBUG:boto:auth_path=/vclfcn-60/?versions&max-keys=999 2024-02-01T10:48:09.313 DEBUG:boto:Method: GET 2024-02-01T10:48:09.313 DEBUG:boto:Path: /vclfcn-60/?versions&max-keys=999 2024-02-01T10:48:09.313 DEBUG:boto:Data: 2024-02-01T10:48:09.314 DEBUG:boto:Headers: {} 2024-02-01T10:48:09.314 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:09.314 DEBUG:boto:Port: 8000 2024-02-01T10:48:09.314 DEBUG:boto:Params: {} 2024-02-01T10:48:09.314 DEBUG:boto:Token: None 2024-02-01T10:48:09.314 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:09 GMT /vclfcn-60/?versions 2024-02-01T10:48:09.314 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:U1h8v6NOoFkEwn99r6fDWhsG0Fg= 2024-02-01T10:48:09.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:U1h8v6NOoFkEwn99r6fDWhsG0Fg=', 'Content-Length': '0'} 2024-02-01T10:48:09.319 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000283d67fe0df7b5fe-0065bb76e9-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:09.319 DEBUG:boto:b'vclfcn-60999falseobj1nulltrue2024-02-01T10:46:46.243Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:46:46.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:46:46.285Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:46:46.336Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-02-01T10:47:30.291Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-02-01T10:47:30.343Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-02-01T10:47:30.358Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-02-01T10:47:30.408Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:09.320 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:48:09.320 DEBUG:boto:path=/vclfcn-60/obj1 2024-02-01T10:48:09.320 DEBUG:boto:auth_path=/vclfcn-60/obj1 2024-02-01T10:48:09.320 DEBUG:boto:path=/vclfcn-60/obj1?versionId=null 2024-02-01T10:48:09.320 DEBUG:boto:auth_path=/vclfcn-60/obj1?versionId=null 2024-02-01T10:48:09.320 DEBUG:boto:Method: GET 2024-02-01T10:48:09.320 DEBUG:boto:Path: /vclfcn-60/obj1?versionId=null 2024-02-01T10:48:09.320 DEBUG:boto:Data: 2024-02-01T10:48:09.320 DEBUG:boto:Headers: {} 2024-02-01T10:48:09.321 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:09.321 DEBUG:boto:Port: 8000 2024-02-01T10:48:09.321 DEBUG:boto:Params: {} 2024-02-01T10:48:09.321 DEBUG:boto:Token: None 2024-02-01T10:48:09.321 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:09 GMT /vclfcn-60/obj1?versionId=null 2024-02-01T10:48:09.321 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:U5iYQjUFynjtryzzRS+UMruAyvk= 2024-02-01T10:48:09.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:U5iYQjUFynjtryzzRS+UMruAyvk=', 'Content-Length': '0'} 2024-02-01T10:48:09.323 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009828a290c6f19888-0065bb76e9-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:09.324 DEBUG:boto:path=/vclfcn-60/obj1 2024-02-01T10:48:09.324 DEBUG:boto:auth_path=/vclfcn-60/obj1 2024-02-01T10:48:09.324 DEBUG:boto:path=/vclfcn-60/obj1?versionId=null 2024-02-01T10:48:09.324 DEBUG:boto:auth_path=/vclfcn-60/obj1?versionId=null 2024-02-01T10:48:09.324 DEBUG:boto:Method: GET 2024-02-01T10:48:09.324 DEBUG:boto:Path: /vclfcn-60/obj1?versionId=null 2024-02-01T10:48:09.324 DEBUG:boto:Data: 2024-02-01T10:48:09.324 DEBUG:boto:Headers: {} 2024-02-01T10:48:09.324 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:09.325 DEBUG:boto:Port: 8000 2024-02-01T10:48:09.325 DEBUG:boto:Params: {} 2024-02-01T10:48:09.325 DEBUG:boto:Token: None 2024-02-01T10:48:09.325 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:09 GMT /vclfcn-60/obj1?versionId=null 2024-02-01T10:48:09.325 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:U5iYQjUFynjtryzzRS+UMruAyvk= 2024-02-01T10:48:09.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:U5iYQjUFynjtryzzRS+UMruAyvk=', 'Content-Length': '0'} 2024-02-01T10:48:09.328 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001c4e85805d03b754-0065bb76e9-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:09.368 DEBUG:boto:path=/vclfcn-60/obj1 2024-02-01T10:48:09.368 DEBUG:boto:auth_path=/vclfcn-60/obj1 2024-02-01T10:48:09.368 DEBUG:boto:path=/vclfcn-60/obj1?versionId=null 2024-02-01T10:48:09.369 DEBUG:boto:auth_path=/vclfcn-60/obj1?versionId=null 2024-02-01T10:48:09.369 DEBUG:boto:Method: HEAD 2024-02-01T10:48:09.369 DEBUG:boto:Path: /vclfcn-60/obj1?versionId=null 2024-02-01T10:48:09.369 DEBUG:boto:Data: 2024-02-01T10:48:09.369 DEBUG:boto:Headers: {} 2024-02-01T10:48:09.369 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:09.369 DEBUG:boto:Port: 8000 2024-02-01T10:48:09.369 DEBUG:boto:Params: {} 2024-02-01T10:48:09.369 DEBUG:boto:Token: None 2024-02-01T10:48:09.369 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:09 GMT /vclfcn-60/obj1?versionId=null 2024-02-01T10:48:09.370 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:hFs9QRkJyQbM7UkHreQAftew6G4= 2024-02-01T10:48:09.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:hFs9QRkJyQbM7UkHreQAftew6G4=', 'Content-Length': '0'} 2024-02-01T10:48:09.372 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001883fcc0a07e2d96-0065bb76e9-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:09.373 DEBUG:boto:path=/vclfcn-60/obj1 2024-02-01T10:48:09.373 DEBUG:boto:auth_path=/vclfcn-60/obj1 2024-02-01T10:48:09.373 DEBUG:boto:path=/vclfcn-60/obj1?versionId=null 2024-02-01T10:48:09.373 DEBUG:boto:auth_path=/vclfcn-60/obj1?versionId=null 2024-02-01T10:48:09.373 DEBUG:boto:Method: HEAD 2024-02-01T10:48:09.373 DEBUG:boto:Path: /vclfcn-60/obj1?versionId=null 2024-02-01T10:48:09.373 DEBUG:boto:Data: 2024-02-01T10:48:09.373 DEBUG:boto:Headers: {} 2024-02-01T10:48:09.373 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:09.373 DEBUG:boto:Port: 8000 2024-02-01T10:48:09.374 DEBUG:boto:Params: {} 2024-02-01T10:48:09.374 DEBUG:boto:Token: None 2024-02-01T10:48:09.374 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:09 GMT /vclfcn-60/obj1?versionId=null 2024-02-01T10:48:09.374 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:hFs9QRkJyQbM7UkHreQAftew6G4= 2024-02-01T10:48:09.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:hFs9QRkJyQbM7UkHreQAftew6G4=', 'Content-Length': '0'} 2024-02-01T10:48:09.377 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e951c9bc4be345cd-0065bb76e9-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:09.377 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:48:09.377 DEBUG:boto:path=/vclfcn-60/obj1 2024-02-01T10:48:09.378 DEBUG:boto:auth_path=/vclfcn-60/obj1 2024-02-01T10:48:09.378 DEBUG:boto:path=/vclfcn-60/obj1?versionId=null 2024-02-01T10:48:09.378 DEBUG:boto:auth_path=/vclfcn-60/obj1?versionId=null 2024-02-01T10:48:09.378 DEBUG:boto:Method: GET 2024-02-01T10:48:09.378 DEBUG:boto:Path: /vclfcn-60/obj1?versionId=null 2024-02-01T10:48:09.378 DEBUG:boto:Data: 2024-02-01T10:48:09.378 DEBUG:boto:Headers: {} 2024-02-01T10:48:09.378 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:09.378 DEBUG:boto:Port: 8000 2024-02-01T10:48:09.378 DEBUG:boto:Params: {} 2024-02-01T10:48:09.379 DEBUG:boto:Token: None 2024-02-01T10:48:09.379 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:09 GMT /vclfcn-60/obj1?versionId=null 2024-02-01T10:48:09.379 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:U5iYQjUFynjtryzzRS+UMruAyvk= 2024-02-01T10:48:09.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:U5iYQjUFynjtryzzRS+UMruAyvk=', 'Content-Length': '0'} 2024-02-01T10:48:09.381 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000493de9b85a8008e8-0065bb76e9-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:09.424 DEBUG:boto:path=/vclfcn-60/obj1 2024-02-01T10:48:09.424 DEBUG:boto:auth_path=/vclfcn-60/obj1 2024-02-01T10:48:09.424 DEBUG:boto:path=/vclfcn-60/obj1?versionId=null 2024-02-01T10:48:09.425 DEBUG:boto:auth_path=/vclfcn-60/obj1?versionId=null 2024-02-01T10:48:09.425 DEBUG:boto:Method: GET 2024-02-01T10:48:09.425 DEBUG:boto:Path: /vclfcn-60/obj1?versionId=null 2024-02-01T10:48:09.425 DEBUG:boto:Data: 2024-02-01T10:48:09.425 DEBUG:boto:Headers: {} 2024-02-01T10:48:09.425 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:09.425 DEBUG:boto:Port: 8000 2024-02-01T10:48:09.425 DEBUG:boto:Params: {} 2024-02-01T10:48:09.425 DEBUG:boto:Token: None 2024-02-01T10:48:09.426 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:09 GMT /vclfcn-60/obj1?versionId=null 2024-02-01T10:48:09.426 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:U5iYQjUFynjtryzzRS+UMruAyvk= 2024-02-01T10:48:09.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:U5iYQjUFynjtryzzRS+UMruAyvk=', 'Content-Length': '0'} 2024-02-01T10:48:09.429 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000053353f584ccff507-0065bb76e9-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:09.429 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:09.429 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:09.429 DEBUG:boto:path=/vclfcn-60/?prefix=obj1 2024-02-01T10:48:09.429 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj1 2024-02-01T10:48:09.429 DEBUG:boto:Method: GET 2024-02-01T10:48:09.430 DEBUG:boto:Path: /vclfcn-60/?prefix=obj1 2024-02-01T10:48:09.430 DEBUG:boto:Data: 2024-02-01T10:48:09.430 DEBUG:boto:Headers: {} 2024-02-01T10:48:09.430 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:09.430 DEBUG:boto:Port: 8000 2024-02-01T10:48:09.430 DEBUG:boto:Params: {} 2024-02-01T10:48:09.430 DEBUG:boto:Token: None 2024-02-01T10:48:09.430 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:09 GMT /vclfcn-60/ 2024-02-01T10:48:09.430 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:LxpWTwF+jQEoYejGBeeZg02cQAE= 2024-02-01T10:48:09.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:LxpWTwF+jQEoYejGBeeZg02cQAE=', 'Content-Length': '0'} 2024-02-01T10:48:09.435 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005efeeaaae7c6be77-0065bb76e9-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:09.435 DEBUG:boto:b'vclfcn-60obj11000falseobj12024-02-01T10:46:46.243Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:09.435 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:09.435 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:09.435 DEBUG:boto:path=/vclfcn-60/?prefix=obj1 2024-02-01T10:48:09.436 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj1 2024-02-01T10:48:09.436 DEBUG:boto:Method: GET 2024-02-01T10:48:09.436 DEBUG:boto:Path: /vclfcn-60/?prefix=obj1 2024-02-01T10:48:09.436 DEBUG:boto:Data: 2024-02-01T10:48:09.436 DEBUG:boto:Headers: {} 2024-02-01T10:48:09.436 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:09.436 DEBUG:boto:Port: 8000 2024-02-01T10:48:09.436 DEBUG:boto:Params: {} 2024-02-01T10:48:09.436 DEBUG:boto:Token: None 2024-02-01T10:48:09.437 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:09 GMT /vclfcn-60/ 2024-02-01T10:48:09.437 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:LxpWTwF+jQEoYejGBeeZg02cQAE= 2024-02-01T10:48:09.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:LxpWTwF+jQEoYejGBeeZg02cQAE=', 'Content-Length': '0'} 2024-02-01T10:48:09.442 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003627024d9d750dbe-0065bb76e9-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:09.442 DEBUG:boto:b'vclfcn-60obj11000falseobj12024-02-01T10:46:46.243Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:09.442 DEBUG:boto:path=/vclfcn-60/obj1 2024-02-01T10:48:09.443 DEBUG:boto:auth_path=/vclfcn-60/obj1 2024-02-01T10:48:09.443 DEBUG:boto:Method: HEAD 2024-02-01T10:48:09.443 DEBUG:boto:Path: /vclfcn-60/obj1 2024-02-01T10:48:09.443 DEBUG:boto:Data: 2024-02-01T10:48:09.444 DEBUG:boto:Headers: {} 2024-02-01T10:48:09.444 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:09.444 DEBUG:boto:Port: 8000 2024-02-01T10:48:09.444 DEBUG:boto:Params: {} 2024-02-01T10:48:09.444 DEBUG:boto:Token: None 2024-02-01T10:48:09.445 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:09 GMT /vclfcn-60/obj1 2024-02-01T10:48:09.445 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1QVturovOhkT5YBDsPUvTqyrevA= 2024-02-01T10:48:09.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1QVturovOhkT5YBDsPUvTqyrevA=', 'Content-Length': '0'} 2024-02-01T10:48:09.448 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008b1b8a7ecdd71e4b-0065bb76e9-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:09.449 DEBUG:boto:path=/vclfcn-60/obj1 2024-02-01T10:48:09.449 DEBUG:boto:auth_path=/vclfcn-60/obj1 2024-02-01T10:48:09.449 DEBUG:boto:Method: HEAD 2024-02-01T10:48:09.449 DEBUG:boto:Path: /vclfcn-60/obj1 2024-02-01T10:48:09.449 DEBUG:boto:Data: 2024-02-01T10:48:09.450 DEBUG:boto:Headers: {} 2024-02-01T10:48:09.450 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:09.450 DEBUG:boto:Port: 8000 2024-02-01T10:48:09.450 DEBUG:boto:Params: {} 2024-02-01T10:48:09.450 DEBUG:boto:Token: None 2024-02-01T10:48:09.451 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:09 GMT /vclfcn-60/obj1 2024-02-01T10:48:09.451 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1QVturovOhkT5YBDsPUvTqyrevA= 2024-02-01T10:48:09.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1QVturovOhkT5YBDsPUvTqyrevA=', 'Content-Length': '0'} 2024-02-01T10:48:09.454 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a6532dd424817313-0065bb76e9-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:09.455 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:48:09.455 DEBUG:boto:path=/vclfcn-60/obj1 2024-02-01T10:48:09.455 DEBUG:boto:auth_path=/vclfcn-60/obj1 2024-02-01T10:48:09.455 DEBUG:boto:Method: GET 2024-02-01T10:48:09.456 DEBUG:boto:Path: /vclfcn-60/obj1 2024-02-01T10:48:09.456 DEBUG:boto:Data: 2024-02-01T10:48:09.456 DEBUG:boto:Headers: {} 2024-02-01T10:48:09.456 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:09.456 DEBUG:boto:Port: 8000 2024-02-01T10:48:09.457 DEBUG:boto:Params: {} 2024-02-01T10:48:09.457 DEBUG:boto:Token: None 2024-02-01T10:48:09.457 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:09 GMT /vclfcn-60/obj1 2024-02-01T10:48:09.457 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:2yB4NqsLNIHIsqZrPFh3++WpwSo= 2024-02-01T10:48:09.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2yB4NqsLNIHIsqZrPFh3++WpwSo=', 'Content-Length': '0'} 2024-02-01T10:48:09.461 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e0cd76ad9f22cf60-0065bb76e9-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:09.505 DEBUG:boto:path=/vclfcn-60/obj1 2024-02-01T10:48:09.505 DEBUG:boto:auth_path=/vclfcn-60/obj1 2024-02-01T10:48:09.505 DEBUG:boto:Method: GET 2024-02-01T10:48:09.505 DEBUG:boto:Path: /vclfcn-60/obj1 2024-02-01T10:48:09.505 DEBUG:boto:Data: 2024-02-01T10:48:09.505 DEBUG:boto:Headers: {} 2024-02-01T10:48:09.505 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:09.505 DEBUG:boto:Port: 8000 2024-02-01T10:48:09.505 DEBUG:boto:Params: {} 2024-02-01T10:48:09.506 DEBUG:boto:Token: None 2024-02-01T10:48:09.506 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:09 GMT /vclfcn-60/obj1 2024-02-01T10:48:09.506 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:2yB4NqsLNIHIsqZrPFh3++WpwSo= 2024-02-01T10:48:09.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2yB4NqsLNIHIsqZrPFh3++WpwSo=', 'Content-Length': '0'} 2024-02-01T10:48:09.509 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009e8e1c658d9ed5aa-0065bb76e9-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:09.509 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:09.509 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:09.509 DEBUG:boto:path=/vclfcn-60/?prefix=obj1 2024-02-01T10:48:09.509 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj1 2024-02-01T10:48:09.510 DEBUG:boto:Method: GET 2024-02-01T10:48:09.510 DEBUG:boto:Path: /vclfcn-60/?prefix=obj1 2024-02-01T10:48:09.510 DEBUG:boto:Data: 2024-02-01T10:48:09.510 DEBUG:boto:Headers: {} 2024-02-01T10:48:09.510 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:09.510 DEBUG:boto:Port: 8000 2024-02-01T10:48:09.510 DEBUG:boto:Params: {} 2024-02-01T10:48:09.510 DEBUG:boto:Token: None 2024-02-01T10:48:09.510 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:09 GMT /vclfcn-60/ 2024-02-01T10:48:09.511 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:LxpWTwF+jQEoYejGBeeZg02cQAE= 2024-02-01T10:48:09.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:LxpWTwF+jQEoYejGBeeZg02cQAE=', 'Content-Length': '0'} 2024-02-01T10:48:09.516 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb919cc2db527aaa-0065bb76e9-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:09.516 DEBUG:boto:b'vclfcn-60obj11000falseobj12024-02-01T10:46:46.243Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:09.517 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:09.517 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:09.517 DEBUG:boto:path=/vclfcn-60/?prefix=obj1 2024-02-01T10:48:09.517 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj1 2024-02-01T10:48:09.517 DEBUG:boto:Method: GET 2024-02-01T10:48:09.517 DEBUG:boto:Path: /vclfcn-60/?prefix=obj1 2024-02-01T10:48:09.517 DEBUG:boto:Data: 2024-02-01T10:48:09.517 DEBUG:boto:Headers: {} 2024-02-01T10:48:09.517 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:09.518 DEBUG:boto:Port: 8000 2024-02-01T10:48:09.518 DEBUG:boto:Params: {} 2024-02-01T10:48:09.518 DEBUG:boto:Token: None 2024-02-01T10:48:09.518 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:09 GMT /vclfcn-60/ 2024-02-01T10:48:09.518 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:LxpWTwF+jQEoYejGBeeZg02cQAE= 2024-02-01T10:48:09.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:LxpWTwF+jQEoYejGBeeZg02cQAE=', 'Content-Length': '0'} 2024-02-01T10:48:09.523 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025350ddb676dbbc3-0065bb76e9-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:09.524 DEBUG:boto:b'vclfcn-60obj11000falseobj12024-02-01T10:46:46.243Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:09.524 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:48:09.525 DEBUG:boto:path=/vclfcn-60/obj2 2024-02-01T10:48:09.525 DEBUG:boto:auth_path=/vclfcn-60/obj2 2024-02-01T10:48:09.525 DEBUG:boto:path=/vclfcn-60/obj2?versionId=null 2024-02-01T10:48:09.525 DEBUG:boto:auth_path=/vclfcn-60/obj2?versionId=null 2024-02-01T10:48:09.525 DEBUG:boto:Method: GET 2024-02-01T10:48:09.526 DEBUG:boto:Path: /vclfcn-60/obj2?versionId=null 2024-02-01T10:48:09.526 DEBUG:boto:Data: 2024-02-01T10:48:09.526 DEBUG:boto:Headers: {} 2024-02-01T10:48:09.526 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:09.526 DEBUG:boto:Port: 8000 2024-02-01T10:48:09.527 DEBUG:boto:Params: {} 2024-02-01T10:48:09.527 DEBUG:boto:Token: None 2024-02-01T10:48:09.527 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:09 GMT /vclfcn-60/obj2?versionId=null 2024-02-01T10:48:09.527 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:3379xqWBuMKgsBlYBv1GnZq/U5k= 2024-02-01T10:48:09.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:3379xqWBuMKgsBlYBv1GnZq/U5k=', 'Content-Length': '0'} 2024-02-01T10:48:09.530 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000061472c0b8a1e49d9-0065bb76e9-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:09.531 DEBUG:boto:path=/vclfcn-60/obj2 2024-02-01T10:48:09.531 DEBUG:boto:auth_path=/vclfcn-60/obj2 2024-02-01T10:48:09.531 DEBUG:boto:path=/vclfcn-60/obj2?versionId=null 2024-02-01T10:48:09.531 DEBUG:boto:auth_path=/vclfcn-60/obj2?versionId=null 2024-02-01T10:48:09.531 DEBUG:boto:Method: GET 2024-02-01T10:48:09.531 DEBUG:boto:Path: /vclfcn-60/obj2?versionId=null 2024-02-01T10:48:09.531 DEBUG:boto:Data: 2024-02-01T10:48:09.532 DEBUG:boto:Headers: {} 2024-02-01T10:48:09.532 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:09.532 DEBUG:boto:Port: 8000 2024-02-01T10:48:09.532 DEBUG:boto:Params: {} 2024-02-01T10:48:09.532 DEBUG:boto:Token: None 2024-02-01T10:48:09.532 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:09 GMT /vclfcn-60/obj2?versionId=null 2024-02-01T10:48:09.532 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:3379xqWBuMKgsBlYBv1GnZq/U5k= 2024-02-01T10:48:09.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:3379xqWBuMKgsBlYBv1GnZq/U5k=', 'Content-Length': '0'} 2024-02-01T10:48:09.535 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000017689b59cbbc4fad-0065bb76e9-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:09.535 DEBUG:boto:path=/vclfcn-60/obj2 2024-02-01T10:48:09.536 DEBUG:boto:auth_path=/vclfcn-60/obj2 2024-02-01T10:48:09.536 DEBUG:boto:path=/vclfcn-60/obj2?versionId=null 2024-02-01T10:48:09.536 DEBUG:boto:auth_path=/vclfcn-60/obj2?versionId=null 2024-02-01T10:48:09.536 DEBUG:boto:Method: HEAD 2024-02-01T10:48:09.537 DEBUG:boto:Path: /vclfcn-60/obj2?versionId=null 2024-02-01T10:48:09.537 DEBUG:boto:Data: 2024-02-01T10:48:09.537 DEBUG:boto:Headers: {} 2024-02-01T10:48:09.537 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:09.537 DEBUG:boto:Port: 8000 2024-02-01T10:48:09.538 DEBUG:boto:Params: {} 2024-02-01T10:48:09.538 DEBUG:boto:Token: None 2024-02-01T10:48:09.538 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:09 GMT /vclfcn-60/obj2?versionId=null 2024-02-01T10:48:09.538 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:t5XmDgEKnZ9jyTCrnOx8PqcCrm0= 2024-02-01T10:48:09.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:t5XmDgEKnZ9jyTCrnOx8PqcCrm0=', 'Content-Length': '0'} 2024-02-01T10:48:09.541 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003af993dfc2d11e8a-0065bb76e9-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:09.542 DEBUG:boto:path=/vclfcn-60/obj2 2024-02-01T10:48:09.542 DEBUG:boto:auth_path=/vclfcn-60/obj2 2024-02-01T10:48:09.542 DEBUG:boto:path=/vclfcn-60/obj2?versionId=null 2024-02-01T10:48:09.542 DEBUG:boto:auth_path=/vclfcn-60/obj2?versionId=null 2024-02-01T10:48:09.542 DEBUG:boto:Method: HEAD 2024-02-01T10:48:09.542 DEBUG:boto:Path: /vclfcn-60/obj2?versionId=null 2024-02-01T10:48:09.542 DEBUG:boto:Data: 2024-02-01T10:48:09.543 DEBUG:boto:Headers: {} 2024-02-01T10:48:09.543 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:09.543 DEBUG:boto:Port: 8000 2024-02-01T10:48:09.543 DEBUG:boto:Params: {} 2024-02-01T10:48:09.543 DEBUG:boto:Token: None 2024-02-01T10:48:09.543 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:09 GMT /vclfcn-60/obj2?versionId=null 2024-02-01T10:48:09.543 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:t5XmDgEKnZ9jyTCrnOx8PqcCrm0= 2024-02-01T10:48:09.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:t5XmDgEKnZ9jyTCrnOx8PqcCrm0=', 'Content-Length': '0'} 2024-02-01T10:48:09.546 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000023a363de69562005-0065bb76e9-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:09.547 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:48:09.547 DEBUG:boto:path=/vclfcn-60/obj2 2024-02-01T10:48:09.547 DEBUG:boto:auth_path=/vclfcn-60/obj2 2024-02-01T10:48:09.547 DEBUG:boto:path=/vclfcn-60/obj2?versionId=null 2024-02-01T10:48:09.547 DEBUG:boto:auth_path=/vclfcn-60/obj2?versionId=null 2024-02-01T10:48:09.547 DEBUG:boto:Method: GET 2024-02-01T10:48:09.547 DEBUG:boto:Path: /vclfcn-60/obj2?versionId=null 2024-02-01T10:48:09.547 DEBUG:boto:Data: 2024-02-01T10:48:09.547 DEBUG:boto:Headers: {} 2024-02-01T10:48:09.547 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:09.547 DEBUG:boto:Port: 8000 2024-02-01T10:48:09.548 DEBUG:boto:Params: {} 2024-02-01T10:48:09.548 DEBUG:boto:Token: None 2024-02-01T10:48:09.548 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:09 GMT /vclfcn-60/obj2?versionId=null 2024-02-01T10:48:09.548 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:3379xqWBuMKgsBlYBv1GnZq/U5k= 2024-02-01T10:48:09.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:3379xqWBuMKgsBlYBv1GnZq/U5k=', 'Content-Length': '0'} 2024-02-01T10:48:09.551 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000022b47fa9658108be-0065bb76e9-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:09.592 DEBUG:boto:path=/vclfcn-60/obj2 2024-02-01T10:48:09.592 DEBUG:boto:auth_path=/vclfcn-60/obj2 2024-02-01T10:48:09.592 DEBUG:boto:path=/vclfcn-60/obj2?versionId=null 2024-02-01T10:48:09.593 DEBUG:boto:auth_path=/vclfcn-60/obj2?versionId=null 2024-02-01T10:48:09.593 DEBUG:boto:Method: GET 2024-02-01T10:48:09.593 DEBUG:boto:Path: /vclfcn-60/obj2?versionId=null 2024-02-01T10:48:09.593 DEBUG:boto:Data: 2024-02-01T10:48:09.593 DEBUG:boto:Headers: {} 2024-02-01T10:48:09.593 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:09.593 DEBUG:boto:Port: 8000 2024-02-01T10:48:09.593 DEBUG:boto:Params: {} 2024-02-01T10:48:09.594 DEBUG:boto:Token: None 2024-02-01T10:48:09.594 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:09 GMT /vclfcn-60/obj2?versionId=null 2024-02-01T10:48:09.594 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:3379xqWBuMKgsBlYBv1GnZq/U5k= 2024-02-01T10:48:09.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:3379xqWBuMKgsBlYBv1GnZq/U5k=', 'Content-Length': '0'} 2024-02-01T10:48:09.597 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001274e753a82aed37-0065bb76e9-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:09.597 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:09.598 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:09.598 DEBUG:boto:path=/vclfcn-60/?prefix=obj2 2024-02-01T10:48:09.598 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj2 2024-02-01T10:48:09.598 DEBUG:boto:Method: GET 2024-02-01T10:48:09.598 DEBUG:boto:Path: /vclfcn-60/?prefix=obj2 2024-02-01T10:48:09.598 DEBUG:boto:Data: 2024-02-01T10:48:09.598 DEBUG:boto:Headers: {} 2024-02-01T10:48:09.598 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:09.598 DEBUG:boto:Port: 8000 2024-02-01T10:48:09.599 DEBUG:boto:Params: {} 2024-02-01T10:48:09.599 DEBUG:boto:Token: None 2024-02-01T10:48:09.599 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:09 GMT /vclfcn-60/ 2024-02-01T10:48:09.599 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:LxpWTwF+jQEoYejGBeeZg02cQAE= 2024-02-01T10:48:09.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:LxpWTwF+jQEoYejGBeeZg02cQAE=', 'Content-Length': '0'} 2024-02-01T10:48:09.604 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004dcd550c0fe10fde-0065bb76e9-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:09.605 DEBUG:boto:b'vclfcn-60obj21000falseobj22024-02-01T10:46:46.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:09.605 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:09.605 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:09.605 DEBUG:boto:path=/vclfcn-60/?prefix=obj2 2024-02-01T10:48:09.605 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj2 2024-02-01T10:48:09.605 DEBUG:boto:Method: GET 2024-02-01T10:48:09.606 DEBUG:boto:Path: /vclfcn-60/?prefix=obj2 2024-02-01T10:48:09.606 DEBUG:boto:Data: 2024-02-01T10:48:09.606 DEBUG:boto:Headers: {} 2024-02-01T10:48:09.606 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:09.606 DEBUG:boto:Port: 8000 2024-02-01T10:48:09.606 DEBUG:boto:Params: {} 2024-02-01T10:48:09.606 DEBUG:boto:Token: None 2024-02-01T10:48:09.606 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:09 GMT /vclfcn-60/ 2024-02-01T10:48:09.606 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:LxpWTwF+jQEoYejGBeeZg02cQAE= 2024-02-01T10:48:09.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:LxpWTwF+jQEoYejGBeeZg02cQAE=', 'Content-Length': '0'} 2024-02-01T10:48:09.611 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004700a563dad308af-0065bb76e9-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:09.656 DEBUG:boto:b'vclfcn-60obj21000falseobj22024-02-01T10:46:46.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:09.656 DEBUG:boto:path=/vclfcn-60/obj2 2024-02-01T10:48:09.657 DEBUG:boto:auth_path=/vclfcn-60/obj2 2024-02-01T10:48:09.657 DEBUG:boto:Method: HEAD 2024-02-01T10:48:09.657 DEBUG:boto:Path: /vclfcn-60/obj2 2024-02-01T10:48:09.657 DEBUG:boto:Data: 2024-02-01T10:48:09.657 DEBUG:boto:Headers: {} 2024-02-01T10:48:09.657 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:09.657 DEBUG:boto:Port: 8000 2024-02-01T10:48:09.657 DEBUG:boto:Params: {} 2024-02-01T10:48:09.657 DEBUG:boto:Token: None 2024-02-01T10:48:09.657 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:09 GMT /vclfcn-60/obj2 2024-02-01T10:48:09.658 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:xaYlMIwxC6IFmFxAjMJa88kbDi8= 2024-02-01T10:48:09.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:xaYlMIwxC6IFmFxAjMJa88kbDi8=', 'Content-Length': '0'} 2024-02-01T10:48:09.660 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000079f034ef2a87a10d-0065bb76e9-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:09.660 DEBUG:boto:path=/vclfcn-60/obj2 2024-02-01T10:48:09.661 DEBUG:boto:auth_path=/vclfcn-60/obj2 2024-02-01T10:48:09.661 DEBUG:boto:Method: HEAD 2024-02-01T10:48:09.661 DEBUG:boto:Path: /vclfcn-60/obj2 2024-02-01T10:48:09.661 DEBUG:boto:Data: 2024-02-01T10:48:09.661 DEBUG:boto:Headers: {} 2024-02-01T10:48:09.661 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:09.661 DEBUG:boto:Port: 8000 2024-02-01T10:48:09.661 DEBUG:boto:Params: {} 2024-02-01T10:48:09.661 DEBUG:boto:Token: None 2024-02-01T10:48:09.661 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:09 GMT /vclfcn-60/obj2 2024-02-01T10:48:09.662 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:xaYlMIwxC6IFmFxAjMJa88kbDi8= 2024-02-01T10:48:09.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:xaYlMIwxC6IFmFxAjMJa88kbDi8=', 'Content-Length': '0'} 2024-02-01T10:48:09.664 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b37a97f69d35d60a-0065bb76e9-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:09.665 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:48:09.665 DEBUG:boto:path=/vclfcn-60/obj2 2024-02-01T10:48:09.665 DEBUG:boto:auth_path=/vclfcn-60/obj2 2024-02-01T10:48:09.665 DEBUG:boto:Method: GET 2024-02-01T10:48:09.666 DEBUG:boto:Path: /vclfcn-60/obj2 2024-02-01T10:48:09.666 DEBUG:boto:Data: 2024-02-01T10:48:09.666 DEBUG:boto:Headers: {} 2024-02-01T10:48:09.666 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:09.666 DEBUG:boto:Port: 8000 2024-02-01T10:48:09.666 DEBUG:boto:Params: {} 2024-02-01T10:48:09.666 DEBUG:boto:Token: None 2024-02-01T10:48:09.666 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:09 GMT /vclfcn-60/obj2 2024-02-01T10:48:09.666 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:4Ehvz0VIAY/jIUyKKne23oacH2Y= 2024-02-01T10:48:09.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:4Ehvz0VIAY/jIUyKKne23oacH2Y=', 'Content-Length': '0'} 2024-02-01T10:48:09.669 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c9a2a59883c446e5-0065bb76e9-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:09.669 DEBUG:boto:path=/vclfcn-60/obj2 2024-02-01T10:48:09.670 DEBUG:boto:auth_path=/vclfcn-60/obj2 2024-02-01T10:48:09.670 DEBUG:boto:Method: GET 2024-02-01T10:48:09.670 DEBUG:boto:Path: /vclfcn-60/obj2 2024-02-01T10:48:09.670 DEBUG:boto:Data: 2024-02-01T10:48:09.670 DEBUG:boto:Headers: {} 2024-02-01T10:48:09.670 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:09.670 DEBUG:boto:Port: 8000 2024-02-01T10:48:09.670 DEBUG:boto:Params: {} 2024-02-01T10:48:09.670 DEBUG:boto:Token: None 2024-02-01T10:48:09.670 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:09 GMT /vclfcn-60/obj2 2024-02-01T10:48:09.671 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:4Ehvz0VIAY/jIUyKKne23oacH2Y= 2024-02-01T10:48:09.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:4Ehvz0VIAY/jIUyKKne23oacH2Y=', 'Content-Length': '0'} 2024-02-01T10:48:09.674 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dbbaef83ac9d3a08-0065bb76e9-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:09.674 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:09.674 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:09.674 DEBUG:boto:path=/vclfcn-60/?prefix=obj2 2024-02-01T10:48:09.674 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj2 2024-02-01T10:48:09.674 DEBUG:boto:Method: GET 2024-02-01T10:48:09.674 DEBUG:boto:Path: /vclfcn-60/?prefix=obj2 2024-02-01T10:48:09.674 DEBUG:boto:Data: 2024-02-01T10:48:09.675 DEBUG:boto:Headers: {} 2024-02-01T10:48:09.675 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:09.675 DEBUG:boto:Port: 8000 2024-02-01T10:48:09.675 DEBUG:boto:Params: {} 2024-02-01T10:48:09.675 DEBUG:boto:Token: None 2024-02-01T10:48:09.675 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:09 GMT /vclfcn-60/ 2024-02-01T10:48:09.675 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:LxpWTwF+jQEoYejGBeeZg02cQAE= 2024-02-01T10:48:09.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:LxpWTwF+jQEoYejGBeeZg02cQAE=', 'Content-Length': '0'} 2024-02-01T10:48:09.680 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000000bedda1e2990b6-0065bb76e9-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:09.724 DEBUG:boto:b'vclfcn-60obj21000falseobj22024-02-01T10:46:46.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:09.725 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:09.725 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:09.725 DEBUG:boto:path=/vclfcn-60/?prefix=obj2 2024-02-01T10:48:09.726 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj2 2024-02-01T10:48:09.726 DEBUG:boto:Method: GET 2024-02-01T10:48:09.726 DEBUG:boto:Path: /vclfcn-60/?prefix=obj2 2024-02-01T10:48:09.726 DEBUG:boto:Data: 2024-02-01T10:48:09.727 DEBUG:boto:Headers: {} 2024-02-01T10:48:09.727 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:09.727 DEBUG:boto:Port: 8000 2024-02-01T10:48:09.727 DEBUG:boto:Params: {} 2024-02-01T10:48:09.727 DEBUG:boto:Token: None 2024-02-01T10:48:09.728 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:09 GMT /vclfcn-60/ 2024-02-01T10:48:09.728 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:LxpWTwF+jQEoYejGBeeZg02cQAE= 2024-02-01T10:48:09.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:LxpWTwF+jQEoYejGBeeZg02cQAE=', 'Content-Length': '0'} 2024-02-01T10:48:09.734 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003bc73ac43aaa4aea-0065bb76e9-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:09.734 DEBUG:boto:b'vclfcn-60obj21000falseobj22024-02-01T10:46:46.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:09.734 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:48:09.734 DEBUG:boto:path=/vclfcn-60/obj3 2024-02-01T10:48:09.735 DEBUG:boto:auth_path=/vclfcn-60/obj3 2024-02-01T10:48:09.735 DEBUG:boto:path=/vclfcn-60/obj3?versionId=null 2024-02-01T10:48:09.735 DEBUG:boto:auth_path=/vclfcn-60/obj3?versionId=null 2024-02-01T10:48:09.735 DEBUG:boto:Method: GET 2024-02-01T10:48:09.735 DEBUG:boto:Path: /vclfcn-60/obj3?versionId=null 2024-02-01T10:48:09.735 DEBUG:boto:Data: 2024-02-01T10:48:09.735 DEBUG:boto:Headers: {} 2024-02-01T10:48:09.735 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:09.735 DEBUG:boto:Port: 8000 2024-02-01T10:48:09.735 DEBUG:boto:Params: {} 2024-02-01T10:48:09.736 DEBUG:boto:Token: None 2024-02-01T10:48:09.736 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:09 GMT /vclfcn-60/obj3?versionId=null 2024-02-01T10:48:09.736 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sEr6cReIq9eap7F86EUEgFU5Yec= 2024-02-01T10:48:09.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sEr6cReIq9eap7F86EUEgFU5Yec=', 'Content-Length': '0'} 2024-02-01T10:48:09.739 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ca7591e7afa90393-0065bb76e9-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:09.739 DEBUG:boto:path=/vclfcn-60/obj3 2024-02-01T10:48:09.739 DEBUG:boto:auth_path=/vclfcn-60/obj3 2024-02-01T10:48:09.740 DEBUG:boto:path=/vclfcn-60/obj3?versionId=null 2024-02-01T10:48:09.740 DEBUG:boto:auth_path=/vclfcn-60/obj3?versionId=null 2024-02-01T10:48:09.740 DEBUG:boto:Method: GET 2024-02-01T10:48:09.740 DEBUG:boto:Path: /vclfcn-60/obj3?versionId=null 2024-02-01T10:48:09.740 DEBUG:boto:Data: 2024-02-01T10:48:09.740 DEBUG:boto:Headers: {} 2024-02-01T10:48:09.740 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:09.740 DEBUG:boto:Port: 8000 2024-02-01T10:48:09.740 DEBUG:boto:Params: {} 2024-02-01T10:48:09.741 DEBUG:boto:Token: None 2024-02-01T10:48:09.741 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:09 GMT /vclfcn-60/obj3?versionId=null 2024-02-01T10:48:09.741 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sEr6cReIq9eap7F86EUEgFU5Yec= 2024-02-01T10:48:09.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sEr6cReIq9eap7F86EUEgFU5Yec=', 'Content-Length': '0'} 2024-02-01T10:48:09.744 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c7fd63fbf96c3878-0065bb76e9-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:09.784 DEBUG:boto:path=/vclfcn-60/obj3 2024-02-01T10:48:09.784 DEBUG:boto:auth_path=/vclfcn-60/obj3 2024-02-01T10:48:09.784 DEBUG:boto:path=/vclfcn-60/obj3?versionId=null 2024-02-01T10:48:09.784 DEBUG:boto:auth_path=/vclfcn-60/obj3?versionId=null 2024-02-01T10:48:09.785 DEBUG:boto:Method: HEAD 2024-02-01T10:48:09.785 DEBUG:boto:Path: /vclfcn-60/obj3?versionId=null 2024-02-01T10:48:09.785 DEBUG:boto:Data: 2024-02-01T10:48:09.785 DEBUG:boto:Headers: {} 2024-02-01T10:48:09.785 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:09.785 DEBUG:boto:Port: 8000 2024-02-01T10:48:09.785 DEBUG:boto:Params: {} 2024-02-01T10:48:09.785 DEBUG:boto:Token: None 2024-02-01T10:48:09.785 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:09 GMT /vclfcn-60/obj3?versionId=null 2024-02-01T10:48:09.786 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:rqqC8YelyBg7tzHEZl1C1EINglk= 2024-02-01T10:48:09.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:rqqC8YelyBg7tzHEZl1C1EINglk=', 'Content-Length': '0'} 2024-02-01T10:48:09.788 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cae3702688257f3a-0065bb76e9-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:09.788 DEBUG:boto:path=/vclfcn-60/obj3 2024-02-01T10:48:09.789 DEBUG:boto:auth_path=/vclfcn-60/obj3 2024-02-01T10:48:09.789 DEBUG:boto:path=/vclfcn-60/obj3?versionId=null 2024-02-01T10:48:09.789 DEBUG:boto:auth_path=/vclfcn-60/obj3?versionId=null 2024-02-01T10:48:09.789 DEBUG:boto:Method: HEAD 2024-02-01T10:48:09.789 DEBUG:boto:Path: /vclfcn-60/obj3?versionId=null 2024-02-01T10:48:09.789 DEBUG:boto:Data: 2024-02-01T10:48:09.789 DEBUG:boto:Headers: {} 2024-02-01T10:48:09.789 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:09.789 DEBUG:boto:Port: 8000 2024-02-01T10:48:09.789 DEBUG:boto:Params: {} 2024-02-01T10:48:09.790 DEBUG:boto:Token: None 2024-02-01T10:48:09.790 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:09 GMT /vclfcn-60/obj3?versionId=null 2024-02-01T10:48:09.790 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:rqqC8YelyBg7tzHEZl1C1EINglk= 2024-02-01T10:48:09.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:rqqC8YelyBg7tzHEZl1C1EINglk=', 'Content-Length': '0'} 2024-02-01T10:48:09.793 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007ca1e1bf5d5207f0-0065bb76e9-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:09.793 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:48:09.793 DEBUG:boto:path=/vclfcn-60/obj3 2024-02-01T10:48:09.793 DEBUG:boto:auth_path=/vclfcn-60/obj3 2024-02-01T10:48:09.793 DEBUG:boto:path=/vclfcn-60/obj3?versionId=null 2024-02-01T10:48:09.793 DEBUG:boto:auth_path=/vclfcn-60/obj3?versionId=null 2024-02-01T10:48:09.793 DEBUG:boto:Method: GET 2024-02-01T10:48:09.794 DEBUG:boto:Path: /vclfcn-60/obj3?versionId=null 2024-02-01T10:48:09.794 DEBUG:boto:Data: 2024-02-01T10:48:09.794 DEBUG:boto:Headers: {} 2024-02-01T10:48:09.794 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:09.794 DEBUG:boto:Port: 8000 2024-02-01T10:48:09.794 DEBUG:boto:Params: {} 2024-02-01T10:48:09.794 DEBUG:boto:Token: None 2024-02-01T10:48:09.794 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:09 GMT /vclfcn-60/obj3?versionId=null 2024-02-01T10:48:09.794 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sEr6cReIq9eap7F86EUEgFU5Yec= 2024-02-01T10:48:09.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sEr6cReIq9eap7F86EUEgFU5Yec=', 'Content-Length': '0'} 2024-02-01T10:48:09.797 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002be0a858251661bd-0065bb76e9-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:09.797 DEBUG:boto:path=/vclfcn-60/obj3 2024-02-01T10:48:09.797 DEBUG:boto:auth_path=/vclfcn-60/obj3 2024-02-01T10:48:09.797 DEBUG:boto:path=/vclfcn-60/obj3?versionId=null 2024-02-01T10:48:09.798 DEBUG:boto:auth_path=/vclfcn-60/obj3?versionId=null 2024-02-01T10:48:09.798 DEBUG:boto:Method: GET 2024-02-01T10:48:09.798 DEBUG:boto:Path: /vclfcn-60/obj3?versionId=null 2024-02-01T10:48:09.798 DEBUG:boto:Data: 2024-02-01T10:48:09.798 DEBUG:boto:Headers: {} 2024-02-01T10:48:09.798 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:09.798 DEBUG:boto:Port: 8000 2024-02-01T10:48:09.798 DEBUG:boto:Params: {} 2024-02-01T10:48:09.798 DEBUG:boto:Token: None 2024-02-01T10:48:09.799 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:09 GMT /vclfcn-60/obj3?versionId=null 2024-02-01T10:48:09.799 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sEr6cReIq9eap7F86EUEgFU5Yec= 2024-02-01T10:48:09.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sEr6cReIq9eap7F86EUEgFU5Yec=', 'Content-Length': '0'} 2024-02-01T10:48:09.802 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001f5123cd92c72552-0065bb76e9-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:09.802 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:09.802 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:09.802 DEBUG:boto:path=/vclfcn-60/?prefix=obj3 2024-02-01T10:48:09.802 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj3 2024-02-01T10:48:09.802 DEBUG:boto:Method: GET 2024-02-01T10:48:09.803 DEBUG:boto:Path: /vclfcn-60/?prefix=obj3 2024-02-01T10:48:09.803 DEBUG:boto:Data: 2024-02-01T10:48:09.803 DEBUG:boto:Headers: {} 2024-02-01T10:48:09.803 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:09.803 DEBUG:boto:Port: 8000 2024-02-01T10:48:09.803 DEBUG:boto:Params: {} 2024-02-01T10:48:09.803 DEBUG:boto:Token: None 2024-02-01T10:48:09.803 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:09 GMT /vclfcn-60/ 2024-02-01T10:48:09.803 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:LxpWTwF+jQEoYejGBeeZg02cQAE= 2024-02-01T10:48:09.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:LxpWTwF+jQEoYejGBeeZg02cQAE=', 'Content-Length': '0'} 2024-02-01T10:48:09.808 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff84a4183c53d0d0-0065bb76e9-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:09.852 DEBUG:boto:b'vclfcn-60obj31000falseobj32024-02-01T10:46:46.285Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:09.852 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:09.853 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:09.853 DEBUG:boto:path=/vclfcn-60/?prefix=obj3 2024-02-01T10:48:09.853 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj3 2024-02-01T10:48:09.853 DEBUG:boto:Method: GET 2024-02-01T10:48:09.853 DEBUG:boto:Path: /vclfcn-60/?prefix=obj3 2024-02-01T10:48:09.853 DEBUG:boto:Data: 2024-02-01T10:48:09.853 DEBUG:boto:Headers: {} 2024-02-01T10:48:09.853 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:09.853 DEBUG:boto:Port: 8000 2024-02-01T10:48:09.853 DEBUG:boto:Params: {} 2024-02-01T10:48:09.854 DEBUG:boto:Token: None 2024-02-01T10:48:09.854 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:09 GMT /vclfcn-60/ 2024-02-01T10:48:09.854 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:LxpWTwF+jQEoYejGBeeZg02cQAE= 2024-02-01T10:48:09.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:LxpWTwF+jQEoYejGBeeZg02cQAE=', 'Content-Length': '0'} 2024-02-01T10:48:09.858 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc16ce312a1e0efe-0065bb76e9-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:09.858 DEBUG:boto:b'vclfcn-60obj31000falseobj32024-02-01T10:46:46.285Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:09.859 DEBUG:boto:path=/vclfcn-60/obj3 2024-02-01T10:48:09.859 DEBUG:boto:auth_path=/vclfcn-60/obj3 2024-02-01T10:48:09.859 DEBUG:boto:Method: HEAD 2024-02-01T10:48:09.859 DEBUG:boto:Path: /vclfcn-60/obj3 2024-02-01T10:48:09.859 DEBUG:boto:Data: 2024-02-01T10:48:09.859 DEBUG:boto:Headers: {} 2024-02-01T10:48:09.859 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:09.860 DEBUG:boto:Port: 8000 2024-02-01T10:48:09.860 DEBUG:boto:Params: {} 2024-02-01T10:48:09.860 DEBUG:boto:Token: None 2024-02-01T10:48:09.860 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:09 GMT /vclfcn-60/obj3 2024-02-01T10:48:09.860 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:dkLp7ITorp78ayEUcKnw/AU3I/U= 2024-02-01T10:48:09.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dkLp7ITorp78ayEUcKnw/AU3I/U=', 'Content-Length': '0'} 2024-02-01T10:48:09.863 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000c7bc1fef00bdf2a-0065bb76e9-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:09.863 DEBUG:boto:path=/vclfcn-60/obj3 2024-02-01T10:48:09.863 DEBUG:boto:auth_path=/vclfcn-60/obj3 2024-02-01T10:48:09.863 DEBUG:boto:Method: HEAD 2024-02-01T10:48:09.864 DEBUG:boto:Path: /vclfcn-60/obj3 2024-02-01T10:48:09.864 DEBUG:boto:Data: 2024-02-01T10:48:09.864 DEBUG:boto:Headers: {} 2024-02-01T10:48:09.864 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:09.864 DEBUG:boto:Port: 8000 2024-02-01T10:48:09.864 DEBUG:boto:Params: {} 2024-02-01T10:48:09.864 DEBUG:boto:Token: None 2024-02-01T10:48:09.864 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:09 GMT /vclfcn-60/obj3 2024-02-01T10:48:09.864 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:dkLp7ITorp78ayEUcKnw/AU3I/U= 2024-02-01T10:48:09.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dkLp7ITorp78ayEUcKnw/AU3I/U=', 'Content-Length': '0'} 2024-02-01T10:48:09.867 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000096c61e1af6afaff-0065bb76e9-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:09.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:48:09.867 DEBUG:boto:path=/vclfcn-60/obj3 2024-02-01T10:48:09.867 DEBUG:boto:auth_path=/vclfcn-60/obj3 2024-02-01T10:48:09.868 DEBUG:boto:Method: GET 2024-02-01T10:48:09.868 DEBUG:boto:Path: /vclfcn-60/obj3 2024-02-01T10:48:09.868 DEBUG:boto:Data: 2024-02-01T10:48:09.868 DEBUG:boto:Headers: {} 2024-02-01T10:48:09.868 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:09.868 DEBUG:boto:Port: 8000 2024-02-01T10:48:09.868 DEBUG:boto:Params: {} 2024-02-01T10:48:09.868 DEBUG:boto:Token: None 2024-02-01T10:48:09.868 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:09 GMT /vclfcn-60/obj3 2024-02-01T10:48:09.869 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:B0d2papmJSSKPL/HYCCZR/gjgPg= 2024-02-01T10:48:09.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:B0d2papmJSSKPL/HYCCZR/gjgPg=', 'Content-Length': '0'} 2024-02-01T10:48:09.872 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000709d2ee7fc7202d1-0065bb76e9-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:09.872 DEBUG:boto:path=/vclfcn-60/obj3 2024-02-01T10:48:09.872 DEBUG:boto:auth_path=/vclfcn-60/obj3 2024-02-01T10:48:09.872 DEBUG:boto:Method: GET 2024-02-01T10:48:09.872 DEBUG:boto:Path: /vclfcn-60/obj3 2024-02-01T10:48:09.872 DEBUG:boto:Data: 2024-02-01T10:48:09.873 DEBUG:boto:Headers: {} 2024-02-01T10:48:09.873 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:09.873 DEBUG:boto:Port: 8000 2024-02-01T10:48:09.873 DEBUG:boto:Params: {} 2024-02-01T10:48:09.873 DEBUG:boto:Token: None 2024-02-01T10:48:09.873 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:09 GMT /vclfcn-60/obj3 2024-02-01T10:48:09.873 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:B0d2papmJSSKPL/HYCCZR/gjgPg= 2024-02-01T10:48:09.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:B0d2papmJSSKPL/HYCCZR/gjgPg=', 'Content-Length': '0'} 2024-02-01T10:48:09.876 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000025b9dfc1053b1ac8-0065bb76e9-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:09.920 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:09.920 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:09.920 DEBUG:boto:path=/vclfcn-60/?prefix=obj3 2024-02-01T10:48:09.920 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj3 2024-02-01T10:48:09.921 DEBUG:boto:Method: GET 2024-02-01T10:48:09.921 DEBUG:boto:Path: /vclfcn-60/?prefix=obj3 2024-02-01T10:48:09.921 DEBUG:boto:Data: 2024-02-01T10:48:09.921 DEBUG:boto:Headers: {} 2024-02-01T10:48:09.921 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:09.921 DEBUG:boto:Port: 8000 2024-02-01T10:48:09.921 DEBUG:boto:Params: {} 2024-02-01T10:48:09.921 DEBUG:boto:Token: None 2024-02-01T10:48:09.921 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:09 GMT /vclfcn-60/ 2024-02-01T10:48:09.922 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:LxpWTwF+jQEoYejGBeeZg02cQAE= 2024-02-01T10:48:09.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:LxpWTwF+jQEoYejGBeeZg02cQAE=', 'Content-Length': '0'} 2024-02-01T10:48:09.926 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d24df2b0add4c09-0065bb76e9-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:09.926 DEBUG:boto:b'vclfcn-60obj31000falseobj32024-02-01T10:46:46.285Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:09.926 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:09.926 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:09.926 DEBUG:boto:path=/vclfcn-60/?prefix=obj3 2024-02-01T10:48:09.926 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj3 2024-02-01T10:48:09.927 DEBUG:boto:Method: GET 2024-02-01T10:48:09.927 DEBUG:boto:Path: /vclfcn-60/?prefix=obj3 2024-02-01T10:48:09.927 DEBUG:boto:Data: 2024-02-01T10:48:09.927 DEBUG:boto:Headers: {} 2024-02-01T10:48:09.927 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:09.927 DEBUG:boto:Port: 8000 2024-02-01T10:48:09.927 DEBUG:boto:Params: {} 2024-02-01T10:48:09.927 DEBUG:boto:Token: None 2024-02-01T10:48:09.927 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:09 GMT /vclfcn-60/ 2024-02-01T10:48:09.927 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:LxpWTwF+jQEoYejGBeeZg02cQAE= 2024-02-01T10:48:09.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:LxpWTwF+jQEoYejGBeeZg02cQAE=', 'Content-Length': '0'} 2024-02-01T10:48:09.932 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b50fb132de3918a7-0065bb76e9-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:09.932 DEBUG:boto:b'vclfcn-60obj31000falseobj32024-02-01T10:46:46.285Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:09.933 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:48:09.933 DEBUG:boto:path=/vclfcn-60/obj4 2024-02-01T10:48:09.933 DEBUG:boto:auth_path=/vclfcn-60/obj4 2024-02-01T10:48:09.933 DEBUG:boto:path=/vclfcn-60/obj4?versionId=null 2024-02-01T10:48:09.933 DEBUG:boto:auth_path=/vclfcn-60/obj4?versionId=null 2024-02-01T10:48:09.933 DEBUG:boto:Method: GET 2024-02-01T10:48:09.933 DEBUG:boto:Path: /vclfcn-60/obj4?versionId=null 2024-02-01T10:48:09.933 DEBUG:boto:Data: 2024-02-01T10:48:09.933 DEBUG:boto:Headers: {} 2024-02-01T10:48:09.933 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:09.934 DEBUG:boto:Port: 8000 2024-02-01T10:48:09.934 DEBUG:boto:Params: {} 2024-02-01T10:48:09.934 DEBUG:boto:Token: None 2024-02-01T10:48:09.934 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:09 GMT /vclfcn-60/obj4?versionId=null 2024-02-01T10:48:09.934 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:dWaOidZ3vz3lxfmMuCNwikANYjE= 2024-02-01T10:48:09.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dWaOidZ3vz3lxfmMuCNwikANYjE=', 'Content-Length': '0'} 2024-02-01T10:48:09.937 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c39a4c147f8bd48d-0065bb76e9-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:09.937 DEBUG:boto:path=/vclfcn-60/obj4 2024-02-01T10:48:09.937 DEBUG:boto:auth_path=/vclfcn-60/obj4 2024-02-01T10:48:09.937 DEBUG:boto:path=/vclfcn-60/obj4?versionId=null 2024-02-01T10:48:09.937 DEBUG:boto:auth_path=/vclfcn-60/obj4?versionId=null 2024-02-01T10:48:09.938 DEBUG:boto:Method: GET 2024-02-01T10:48:09.938 DEBUG:boto:Path: /vclfcn-60/obj4?versionId=null 2024-02-01T10:48:09.938 DEBUG:boto:Data: 2024-02-01T10:48:09.938 DEBUG:boto:Headers: {} 2024-02-01T10:48:09.938 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:09.938 DEBUG:boto:Port: 8000 2024-02-01T10:48:09.938 DEBUG:boto:Params: {} 2024-02-01T10:48:09.938 DEBUG:boto:Token: None 2024-02-01T10:48:09.938 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:09 GMT /vclfcn-60/obj4?versionId=null 2024-02-01T10:48:09.938 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:dWaOidZ3vz3lxfmMuCNwikANYjE= 2024-02-01T10:48:09.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dWaOidZ3vz3lxfmMuCNwikANYjE=', 'Content-Length': '0'} 2024-02-01T10:48:09.941 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007938d2121dedf56f-0065bb76e9-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:09.942 DEBUG:boto:path=/vclfcn-60/obj4 2024-02-01T10:48:09.942 DEBUG:boto:auth_path=/vclfcn-60/obj4 2024-02-01T10:48:09.942 DEBUG:boto:path=/vclfcn-60/obj4?versionId=null 2024-02-01T10:48:09.942 DEBUG:boto:auth_path=/vclfcn-60/obj4?versionId=null 2024-02-01T10:48:09.942 DEBUG:boto:Method: HEAD 2024-02-01T10:48:09.942 DEBUG:boto:Path: /vclfcn-60/obj4?versionId=null 2024-02-01T10:48:09.942 DEBUG:boto:Data: 2024-02-01T10:48:09.942 DEBUG:boto:Headers: {} 2024-02-01T10:48:09.942 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:09.943 DEBUG:boto:Port: 8000 2024-02-01T10:48:09.943 DEBUG:boto:Params: {} 2024-02-01T10:48:09.943 DEBUG:boto:Token: None 2024-02-01T10:48:09.943 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:09 GMT /vclfcn-60/obj4?versionId=null 2024-02-01T10:48:09.943 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5ArfsIDU2PsBY14PSn2p5Vkzf3Q= 2024-02-01T10:48:09.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5ArfsIDU2PsBY14PSn2p5Vkzf3Q=', 'Content-Length': '0'} 2024-02-01T10:48:09.946 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bec90d6b87fef61b-0065bb76e9-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:09.946 DEBUG:boto:path=/vclfcn-60/obj4 2024-02-01T10:48:09.946 DEBUG:boto:auth_path=/vclfcn-60/obj4 2024-02-01T10:48:09.946 DEBUG:boto:path=/vclfcn-60/obj4?versionId=null 2024-02-01T10:48:09.946 DEBUG:boto:auth_path=/vclfcn-60/obj4?versionId=null 2024-02-01T10:48:09.946 DEBUG:boto:Method: HEAD 2024-02-01T10:48:09.946 DEBUG:boto:Path: /vclfcn-60/obj4?versionId=null 2024-02-01T10:48:09.946 DEBUG:boto:Data: 2024-02-01T10:48:09.947 DEBUG:boto:Headers: {} 2024-02-01T10:48:09.947 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:09.947 DEBUG:boto:Port: 8000 2024-02-01T10:48:09.947 DEBUG:boto:Params: {} 2024-02-01T10:48:09.947 DEBUG:boto:Token: None 2024-02-01T10:48:09.947 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:09 GMT /vclfcn-60/obj4?versionId=null 2024-02-01T10:48:09.947 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5ArfsIDU2PsBY14PSn2p5Vkzf3Q= 2024-02-01T10:48:09.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5ArfsIDU2PsBY14PSn2p5Vkzf3Q=', 'Content-Length': '0'} 2024-02-01T10:48:09.950 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000499093b43f612015-0065bb76e9-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:09.950 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:48:09.950 DEBUG:boto:path=/vclfcn-60/obj4 2024-02-01T10:48:09.950 DEBUG:boto:auth_path=/vclfcn-60/obj4 2024-02-01T10:48:09.950 DEBUG:boto:path=/vclfcn-60/obj4?versionId=null 2024-02-01T10:48:09.951 DEBUG:boto:auth_path=/vclfcn-60/obj4?versionId=null 2024-02-01T10:48:09.951 DEBUG:boto:Method: GET 2024-02-01T10:48:09.951 DEBUG:boto:Path: /vclfcn-60/obj4?versionId=null 2024-02-01T10:48:09.951 DEBUG:boto:Data: 2024-02-01T10:48:09.951 DEBUG:boto:Headers: {} 2024-02-01T10:48:09.951 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:09.951 DEBUG:boto:Port: 8000 2024-02-01T10:48:09.951 DEBUG:boto:Params: {} 2024-02-01T10:48:09.951 DEBUG:boto:Token: None 2024-02-01T10:48:09.952 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:09 GMT /vclfcn-60/obj4?versionId=null 2024-02-01T10:48:09.952 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:dWaOidZ3vz3lxfmMuCNwikANYjE= 2024-02-01T10:48:09.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dWaOidZ3vz3lxfmMuCNwikANYjE=', 'Content-Length': '0'} 2024-02-01T10:48:09.955 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005dcf198097ebc91c-0065bb76e9-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:09.996 DEBUG:boto:path=/vclfcn-60/obj4 2024-02-01T10:48:09.996 DEBUG:boto:auth_path=/vclfcn-60/obj4 2024-02-01T10:48:09.996 DEBUG:boto:path=/vclfcn-60/obj4?versionId=null 2024-02-01T10:48:09.997 DEBUG:boto:auth_path=/vclfcn-60/obj4?versionId=null 2024-02-01T10:48:09.997 DEBUG:boto:Method: GET 2024-02-01T10:48:09.997 DEBUG:boto:Path: /vclfcn-60/obj4?versionId=null 2024-02-01T10:48:09.997 DEBUG:boto:Data: 2024-02-01T10:48:09.997 DEBUG:boto:Headers: {} 2024-02-01T10:48:09.997 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:09.997 DEBUG:boto:Port: 8000 2024-02-01T10:48:09.998 DEBUG:boto:Params: {} 2024-02-01T10:48:09.998 DEBUG:boto:Token: None 2024-02-01T10:48:09.998 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:09 GMT /vclfcn-60/obj4?versionId=null 2024-02-01T10:48:09.998 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:dWaOidZ3vz3lxfmMuCNwikANYjE= 2024-02-01T10:48:09.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dWaOidZ3vz3lxfmMuCNwikANYjE=', 'Content-Length': '0'} 2024-02-01T10:48:10.001 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000074914a818b218dbe-0065bb76ea-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:10.001 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:10.001 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:10.001 DEBUG:boto:path=/vclfcn-60/?prefix=obj4 2024-02-01T10:48:10.001 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj4 2024-02-01T10:48:10.001 DEBUG:boto:Method: GET 2024-02-01T10:48:10.001 DEBUG:boto:Path: /vclfcn-60/?prefix=obj4 2024-02-01T10:48:10.001 DEBUG:boto:Data: 2024-02-01T10:48:10.002 DEBUG:boto:Headers: {} 2024-02-01T10:48:10.002 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:10.002 DEBUG:boto:Port: 8000 2024-02-01T10:48:10.002 DEBUG:boto:Params: {} 2024-02-01T10:48:10.002 DEBUG:boto:Token: None 2024-02-01T10:48:10.002 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:10 GMT /vclfcn-60/ 2024-02-01T10:48:10.002 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fM7RjZAk6oZ3BsuaYN7sqUQ0eWs= 2024-02-01T10:48:10.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fM7RjZAk6oZ3BsuaYN7sqUQ0eWs=', 'Content-Length': '0'} 2024-02-01T10:48:10.007 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6467104bc360932-0065bb76ea-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:10.007 DEBUG:boto:b'vclfcn-60obj41000falseobj42024-02-01T10:46:46.336Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:10.008 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:10.008 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:10.008 DEBUG:boto:path=/vclfcn-60/?prefix=obj4 2024-02-01T10:48:10.008 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj4 2024-02-01T10:48:10.008 DEBUG:boto:Method: GET 2024-02-01T10:48:10.008 DEBUG:boto:Path: /vclfcn-60/?prefix=obj4 2024-02-01T10:48:10.008 DEBUG:boto:Data: 2024-02-01T10:48:10.008 DEBUG:boto:Headers: {} 2024-02-01T10:48:10.008 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:10.008 DEBUG:boto:Port: 8000 2024-02-01T10:48:10.009 DEBUG:boto:Params: {} 2024-02-01T10:48:10.009 DEBUG:boto:Token: None 2024-02-01T10:48:10.009 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:10 GMT /vclfcn-60/ 2024-02-01T10:48:10.009 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fM7RjZAk6oZ3BsuaYN7sqUQ0eWs= 2024-02-01T10:48:10.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fM7RjZAk6oZ3BsuaYN7sqUQ0eWs=', 'Content-Length': '0'} 2024-02-01T10:48:10.013 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000adb0bbfda54bcf20-0065bb76ea-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:10.013 DEBUG:boto:b'vclfcn-60obj41000falseobj42024-02-01T10:46:46.336Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:10.013 DEBUG:boto:path=/vclfcn-60/obj4 2024-02-01T10:48:10.013 DEBUG:boto:auth_path=/vclfcn-60/obj4 2024-02-01T10:48:10.014 DEBUG:boto:Method: HEAD 2024-02-01T10:48:10.014 DEBUG:boto:Path: /vclfcn-60/obj4 2024-02-01T10:48:10.014 DEBUG:boto:Data: 2024-02-01T10:48:10.014 DEBUG:boto:Headers: {} 2024-02-01T10:48:10.014 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:10.015 DEBUG:boto:Port: 8000 2024-02-01T10:48:10.015 DEBUG:boto:Params: {} 2024-02-01T10:48:10.015 DEBUG:boto:Token: None 2024-02-01T10:48:10.015 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:10 GMT /vclfcn-60/obj4 2024-02-01T10:48:10.016 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:95sTbBE6P7Gz7yB2b5QSdlH5ucU= 2024-02-01T10:48:10.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:95sTbBE6P7Gz7yB2b5QSdlH5ucU=', 'Content-Length': '0'} 2024-02-01T10:48:10.018 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f088524049829101-0065bb76ea-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:10.018 DEBUG:boto:path=/vclfcn-60/obj4 2024-02-01T10:48:10.018 DEBUG:boto:auth_path=/vclfcn-60/obj4 2024-02-01T10:48:10.019 DEBUG:boto:Method: HEAD 2024-02-01T10:48:10.019 DEBUG:boto:Path: /vclfcn-60/obj4 2024-02-01T10:48:10.019 DEBUG:boto:Data: 2024-02-01T10:48:10.019 DEBUG:boto:Headers: {} 2024-02-01T10:48:10.019 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:10.019 DEBUG:boto:Port: 8000 2024-02-01T10:48:10.019 DEBUG:boto:Params: {} 2024-02-01T10:48:10.019 DEBUG:boto:Token: None 2024-02-01T10:48:10.019 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:10 GMT /vclfcn-60/obj4 2024-02-01T10:48:10.019 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:95sTbBE6P7Gz7yB2b5QSdlH5ucU= 2024-02-01T10:48:10.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:95sTbBE6P7Gz7yB2b5QSdlH5ucU=', 'Content-Length': '0'} 2024-02-01T10:48:10.022 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000089bda6a5dd1b76d0-0065bb76ea-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:10.022 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:48:10.022 DEBUG:boto:path=/vclfcn-60/obj4 2024-02-01T10:48:10.023 DEBUG:boto:auth_path=/vclfcn-60/obj4 2024-02-01T10:48:10.023 DEBUG:boto:Method: GET 2024-02-01T10:48:10.023 DEBUG:boto:Path: /vclfcn-60/obj4 2024-02-01T10:48:10.023 DEBUG:boto:Data: 2024-02-01T10:48:10.023 DEBUG:boto:Headers: {} 2024-02-01T10:48:10.023 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:10.023 DEBUG:boto:Port: 8000 2024-02-01T10:48:10.023 DEBUG:boto:Params: {} 2024-02-01T10:48:10.024 DEBUG:boto:Token: None 2024-02-01T10:48:10.024 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:10 GMT /vclfcn-60/obj4 2024-02-01T10:48:10.024 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mnF+TBfS5nZ4w3otOHf1290yS20= 2024-02-01T10:48:10.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mnF+TBfS5nZ4w3otOHf1290yS20=', 'Content-Length': '0'} 2024-02-01T10:48:10.027 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000304050bfdbea4c4a-0065bb76ea-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:10.027 DEBUG:boto:path=/vclfcn-60/obj4 2024-02-01T10:48:10.027 DEBUG:boto:auth_path=/vclfcn-60/obj4 2024-02-01T10:48:10.027 DEBUG:boto:Method: GET 2024-02-01T10:48:10.027 DEBUG:boto:Path: /vclfcn-60/obj4 2024-02-01T10:48:10.028 DEBUG:boto:Data: 2024-02-01T10:48:10.028 DEBUG:boto:Headers: {} 2024-02-01T10:48:10.028 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:10.028 DEBUG:boto:Port: 8000 2024-02-01T10:48:10.028 DEBUG:boto:Params: {} 2024-02-01T10:48:10.028 DEBUG:boto:Token: None 2024-02-01T10:48:10.028 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:10 GMT /vclfcn-60/obj4 2024-02-01T10:48:10.028 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mnF+TBfS5nZ4w3otOHf1290yS20= 2024-02-01T10:48:10.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mnF+TBfS5nZ4w3otOHf1290yS20=', 'Content-Length': '0'} 2024-02-01T10:48:10.031 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000069fed81b8c423f6a-0065bb76ea-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:10.031 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:10.031 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:10.032 DEBUG:boto:path=/vclfcn-60/?prefix=obj4 2024-02-01T10:48:10.032 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj4 2024-02-01T10:48:10.032 DEBUG:boto:Method: GET 2024-02-01T10:48:10.033 DEBUG:boto:Path: /vclfcn-60/?prefix=obj4 2024-02-01T10:48:10.033 DEBUG:boto:Data: 2024-02-01T10:48:10.033 DEBUG:boto:Headers: {} 2024-02-01T10:48:10.033 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:10.033 DEBUG:boto:Port: 8000 2024-02-01T10:48:10.034 DEBUG:boto:Params: {} 2024-02-01T10:48:10.034 DEBUG:boto:Token: None 2024-02-01T10:48:10.034 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:10 GMT /vclfcn-60/ 2024-02-01T10:48:10.034 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fM7RjZAk6oZ3BsuaYN7sqUQ0eWs= 2024-02-01T10:48:10.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fM7RjZAk6oZ3BsuaYN7sqUQ0eWs=', 'Content-Length': '0'} 2024-02-01T10:48:10.039 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000073fbf5eef5b0234c-0065bb76ea-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:10.039 DEBUG:boto:b'vclfcn-60obj41000falseobj42024-02-01T10:46:46.336Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:10.039 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:10.039 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:10.039 DEBUG:boto:path=/vclfcn-60/?prefix=obj4 2024-02-01T10:48:10.040 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj4 2024-02-01T10:48:10.040 DEBUG:boto:Method: GET 2024-02-01T10:48:10.040 DEBUG:boto:Path: /vclfcn-60/?prefix=obj4 2024-02-01T10:48:10.040 DEBUG:boto:Data: 2024-02-01T10:48:10.040 DEBUG:boto:Headers: {} 2024-02-01T10:48:10.040 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:10.040 DEBUG:boto:Port: 8000 2024-02-01T10:48:10.040 DEBUG:boto:Params: {} 2024-02-01T10:48:10.040 DEBUG:boto:Token: None 2024-02-01T10:48:10.040 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:10 GMT /vclfcn-60/ 2024-02-01T10:48:10.041 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fM7RjZAk6oZ3BsuaYN7sqUQ0eWs= 2024-02-01T10:48:10.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fM7RjZAk6oZ3BsuaYN7sqUQ0eWs=', 'Content-Length': '0'} 2024-02-01T10:48:10.045 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004334e7ff0003d83e-0065bb76ea-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:10.045 DEBUG:boto:b'vclfcn-60obj41000falseobj42024-02-01T10:46:46.336Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:10.046 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-02-01T10:48:10.046 DEBUG:boto:path=/vclfcn-60/obj5 2024-02-01T10:48:10.046 DEBUG:boto:auth_path=/vclfcn-60/obj5 2024-02-01T10:48:10.046 DEBUG:boto:path=/vclfcn-60/obj5?versionId=null 2024-02-01T10:48:10.046 DEBUG:boto:auth_path=/vclfcn-60/obj5?versionId=null 2024-02-01T10:48:10.046 DEBUG:boto:Method: GET 2024-02-01T10:48:10.046 DEBUG:boto:Path: /vclfcn-60/obj5?versionId=null 2024-02-01T10:48:10.046 DEBUG:boto:Data: 2024-02-01T10:48:10.047 DEBUG:boto:Headers: {} 2024-02-01T10:48:10.047 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:10.047 DEBUG:boto:Port: 8000 2024-02-01T10:48:10.047 DEBUG:boto:Params: {} 2024-02-01T10:48:10.047 DEBUG:boto:Token: None 2024-02-01T10:48:10.047 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:10 GMT /vclfcn-60/obj5?versionId=null 2024-02-01T10:48:10.047 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+HXTYHwo1IFGK6qqQLXAzIQHAcA= 2024-02-01T10:48:10.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+HXTYHwo1IFGK6qqQLXAzIQHAcA=', 'Content-Length': '0'} 2024-02-01T10:48:10.050 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000035972c7fd9791744-0065bb76ea-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:10.096 DEBUG:boto:path=/vclfcn-60/obj5 2024-02-01T10:48:10.097 DEBUG:boto:auth_path=/vclfcn-60/obj5 2024-02-01T10:48:10.097 DEBUG:boto:path=/vclfcn-60/obj5?versionId=null 2024-02-01T10:48:10.097 DEBUG:boto:auth_path=/vclfcn-60/obj5?versionId=null 2024-02-01T10:48:10.097 DEBUG:boto:Method: GET 2024-02-01T10:48:10.097 DEBUG:boto:Path: /vclfcn-60/obj5?versionId=null 2024-02-01T10:48:10.097 DEBUG:boto:Data: 2024-02-01T10:48:10.097 DEBUG:boto:Headers: {} 2024-02-01T10:48:10.097 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:10.098 DEBUG:boto:Port: 8000 2024-02-01T10:48:10.098 DEBUG:boto:Params: {} 2024-02-01T10:48:10.098 DEBUG:boto:Token: None 2024-02-01T10:48:10.098 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:10 GMT /vclfcn-60/obj5?versionId=null 2024-02-01T10:48:10.098 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+HXTYHwo1IFGK6qqQLXAzIQHAcA= 2024-02-01T10:48:10.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+HXTYHwo1IFGK6qqQLXAzIQHAcA=', 'Content-Length': '0'} 2024-02-01T10:48:10.102 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007f9db586977c592b-0065bb76ea-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:10.102 DEBUG:boto:path=/vclfcn-60/obj5 2024-02-01T10:48:10.102 DEBUG:boto:auth_path=/vclfcn-60/obj5 2024-02-01T10:48:10.102 DEBUG:boto:path=/vclfcn-60/obj5?versionId=null 2024-02-01T10:48:10.102 DEBUG:boto:auth_path=/vclfcn-60/obj5?versionId=null 2024-02-01T10:48:10.103 DEBUG:boto:Method: HEAD 2024-02-01T10:48:10.103 DEBUG:boto:Path: /vclfcn-60/obj5?versionId=null 2024-02-01T10:48:10.103 DEBUG:boto:Data: 2024-02-01T10:48:10.103 DEBUG:boto:Headers: {} 2024-02-01T10:48:10.103 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:10.103 DEBUG:boto:Port: 8000 2024-02-01T10:48:10.104 DEBUG:boto:Params: {} 2024-02-01T10:48:10.104 DEBUG:boto:Token: None 2024-02-01T10:48:10.107 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:10 GMT /vclfcn-60/obj5?versionId=null 2024-02-01T10:48:10.107 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:EdNPEp0e8Ij0a87YiPX7NR+OQeM= 2024-02-01T10:48:10.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:EdNPEp0e8Ij0a87YiPX7NR+OQeM=', 'Content-Length': '0'} 2024-02-01T10:48:10.111 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001c9c6c8bd0c36377-0065bb76ea-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:10.112 DEBUG:boto:path=/vclfcn-60/obj5 2024-02-01T10:48:10.112 DEBUG:boto:auth_path=/vclfcn-60/obj5 2024-02-01T10:48:10.112 DEBUG:boto:path=/vclfcn-60/obj5?versionId=null 2024-02-01T10:48:10.112 DEBUG:boto:auth_path=/vclfcn-60/obj5?versionId=null 2024-02-01T10:48:10.112 DEBUG:boto:Method: HEAD 2024-02-01T10:48:10.112 DEBUG:boto:Path: /vclfcn-60/obj5?versionId=null 2024-02-01T10:48:10.112 DEBUG:boto:Data: 2024-02-01T10:48:10.113 DEBUG:boto:Headers: {} 2024-02-01T10:48:10.113 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:10.113 DEBUG:boto:Port: 8000 2024-02-01T10:48:10.113 DEBUG:boto:Params: {} 2024-02-01T10:48:10.113 DEBUG:boto:Token: None 2024-02-01T10:48:10.113 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:10 GMT /vclfcn-60/obj5?versionId=null 2024-02-01T10:48:10.113 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:EdNPEp0e8Ij0a87YiPX7NR+OQeM= 2024-02-01T10:48:10.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:EdNPEp0e8Ij0a87YiPX7NR+OQeM=', 'Content-Length': '0'} 2024-02-01T10:48:10.116 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f1ad52bf0d75723e-0065bb76ea-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:10.117 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-02-01T10:48:10.117 DEBUG:boto:path=/vclfcn-60/obj5 2024-02-01T10:48:10.117 DEBUG:boto:auth_path=/vclfcn-60/obj5 2024-02-01T10:48:10.117 DEBUG:boto:path=/vclfcn-60/obj5?versionId=null 2024-02-01T10:48:10.117 DEBUG:boto:auth_path=/vclfcn-60/obj5?versionId=null 2024-02-01T10:48:10.117 DEBUG:boto:Method: GET 2024-02-01T10:48:10.117 DEBUG:boto:Path: /vclfcn-60/obj5?versionId=null 2024-02-01T10:48:10.117 DEBUG:boto:Data: 2024-02-01T10:48:10.117 DEBUG:boto:Headers: {} 2024-02-01T10:48:10.117 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:10.117 DEBUG:boto:Port: 8000 2024-02-01T10:48:10.118 DEBUG:boto:Params: {} 2024-02-01T10:48:10.118 DEBUG:boto:Token: None 2024-02-01T10:48:10.118 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:10 GMT /vclfcn-60/obj5?versionId=null 2024-02-01T10:48:10.118 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+HXTYHwo1IFGK6qqQLXAzIQHAcA= 2024-02-01T10:48:10.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+HXTYHwo1IFGK6qqQLXAzIQHAcA=', 'Content-Length': '0'} 2024-02-01T10:48:10.121 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f69c7247d7b7afe8-0065bb76ea-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:10.121 DEBUG:boto:path=/vclfcn-60/obj5 2024-02-01T10:48:10.121 DEBUG:boto:auth_path=/vclfcn-60/obj5 2024-02-01T10:48:10.121 DEBUG:boto:path=/vclfcn-60/obj5?versionId=null 2024-02-01T10:48:10.121 DEBUG:boto:auth_path=/vclfcn-60/obj5?versionId=null 2024-02-01T10:48:10.122 DEBUG:boto:Method: GET 2024-02-01T10:48:10.122 DEBUG:boto:Path: /vclfcn-60/obj5?versionId=null 2024-02-01T10:48:10.122 DEBUG:boto:Data: 2024-02-01T10:48:10.122 DEBUG:boto:Headers: {} 2024-02-01T10:48:10.122 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:10.122 DEBUG:boto:Port: 8000 2024-02-01T10:48:10.122 DEBUG:boto:Params: {} 2024-02-01T10:48:10.122 DEBUG:boto:Token: None 2024-02-01T10:48:10.122 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:10 GMT /vclfcn-60/obj5?versionId=null 2024-02-01T10:48:10.122 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+HXTYHwo1IFGK6qqQLXAzIQHAcA= 2024-02-01T10:48:10.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+HXTYHwo1IFGK6qqQLXAzIQHAcA=', 'Content-Length': '0'} 2024-02-01T10:48:10.125 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003ced7ff925246c6a-0065bb76ea-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:10.168 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:10.168 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:10.169 DEBUG:boto:path=/vclfcn-60/?prefix=obj5 2024-02-01T10:48:10.169 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj5 2024-02-01T10:48:10.169 DEBUG:boto:Method: GET 2024-02-01T10:48:10.169 DEBUG:boto:Path: /vclfcn-60/?prefix=obj5 2024-02-01T10:48:10.169 DEBUG:boto:Data: 2024-02-01T10:48:10.169 DEBUG:boto:Headers: {} 2024-02-01T10:48:10.169 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:10.169 DEBUG:boto:Port: 8000 2024-02-01T10:48:10.169 DEBUG:boto:Params: {} 2024-02-01T10:48:10.169 DEBUG:boto:Token: None 2024-02-01T10:48:10.170 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:10 GMT /vclfcn-60/ 2024-02-01T10:48:10.170 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fM7RjZAk6oZ3BsuaYN7sqUQ0eWs= 2024-02-01T10:48:10.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fM7RjZAk6oZ3BsuaYN7sqUQ0eWs=', 'Content-Length': '0'} 2024-02-01T10:48:10.174 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1f4aa7d23fa02db-0065bb76ea-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:10.175 DEBUG:boto:b'vclfcn-60obj51000falseobj52024-02-01T10:47:30.291Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:10.175 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:10.176 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:10.176 DEBUG:boto:path=/vclfcn-60/?prefix=obj5 2024-02-01T10:48:10.176 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj5 2024-02-01T10:48:10.176 DEBUG:boto:Method: GET 2024-02-01T10:48:10.176 DEBUG:boto:Path: /vclfcn-60/?prefix=obj5 2024-02-01T10:48:10.176 DEBUG:boto:Data: 2024-02-01T10:48:10.176 DEBUG:boto:Headers: {} 2024-02-01T10:48:10.176 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:10.176 DEBUG:boto:Port: 8000 2024-02-01T10:48:10.176 DEBUG:boto:Params: {} 2024-02-01T10:48:10.176 DEBUG:boto:Token: None 2024-02-01T10:48:10.177 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:10 GMT /vclfcn-60/ 2024-02-01T10:48:10.177 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fM7RjZAk6oZ3BsuaYN7sqUQ0eWs= 2024-02-01T10:48:10.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fM7RjZAk6oZ3BsuaYN7sqUQ0eWs=', 'Content-Length': '0'} 2024-02-01T10:48:10.181 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e753f546d4d3ee0-0065bb76ea-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:10.182 DEBUG:boto:b'vclfcn-60obj51000falseobj52024-02-01T10:47:30.291Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:10.182 DEBUG:boto:path=/vclfcn-60/obj5 2024-02-01T10:48:10.182 DEBUG:boto:auth_path=/vclfcn-60/obj5 2024-02-01T10:48:10.182 DEBUG:boto:Method: HEAD 2024-02-01T10:48:10.182 DEBUG:boto:Path: /vclfcn-60/obj5 2024-02-01T10:48:10.182 DEBUG:boto:Data: 2024-02-01T10:48:10.182 DEBUG:boto:Headers: {} 2024-02-01T10:48:10.183 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:10.183 DEBUG:boto:Port: 8000 2024-02-01T10:48:10.183 DEBUG:boto:Params: {} 2024-02-01T10:48:10.183 DEBUG:boto:Token: None 2024-02-01T10:48:10.183 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:10 GMT /vclfcn-60/obj5 2024-02-01T10:48:10.183 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7UBtf+csgLbObisWKg8QvqrHQJA= 2024-02-01T10:48:10.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7UBtf+csgLbObisWKg8QvqrHQJA=', 'Content-Length': '0'} 2024-02-01T10:48:10.186 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004afc8385644ca89a-0065bb76ea-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:10.186 DEBUG:boto:path=/vclfcn-60/obj5 2024-02-01T10:48:10.186 DEBUG:boto:auth_path=/vclfcn-60/obj5 2024-02-01T10:48:10.186 DEBUG:boto:Method: HEAD 2024-02-01T10:48:10.187 DEBUG:boto:Path: /vclfcn-60/obj5 2024-02-01T10:48:10.187 DEBUG:boto:Data: 2024-02-01T10:48:10.187 DEBUG:boto:Headers: {} 2024-02-01T10:48:10.187 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:10.187 DEBUG:boto:Port: 8000 2024-02-01T10:48:10.187 DEBUG:boto:Params: {} 2024-02-01T10:48:10.187 DEBUG:boto:Token: None 2024-02-01T10:48:10.187 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:10 GMT /vclfcn-60/obj5 2024-02-01T10:48:10.187 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7UBtf+csgLbObisWKg8QvqrHQJA= 2024-02-01T10:48:10.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7UBtf+csgLbObisWKg8QvqrHQJA=', 'Content-Length': '0'} 2024-02-01T10:48:10.190 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dceed940163a4141-0065bb76ea-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:10.190 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-02-01T10:48:10.190 DEBUG:boto:path=/vclfcn-60/obj5 2024-02-01T10:48:10.191 DEBUG:boto:auth_path=/vclfcn-60/obj5 2024-02-01T10:48:10.191 DEBUG:boto:Method: GET 2024-02-01T10:48:10.191 DEBUG:boto:Path: /vclfcn-60/obj5 2024-02-01T10:48:10.191 DEBUG:boto:Data: 2024-02-01T10:48:10.191 DEBUG:boto:Headers: {} 2024-02-01T10:48:10.191 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:10.191 DEBUG:boto:Port: 8000 2024-02-01T10:48:10.191 DEBUG:boto:Params: {} 2024-02-01T10:48:10.191 DEBUG:boto:Token: None 2024-02-01T10:48:10.191 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:10 GMT /vclfcn-60/obj5 2024-02-01T10:48:10.192 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zKc283ft9YhHODgL0bJ9bxM9PRQ= 2024-02-01T10:48:10.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zKc283ft9YhHODgL0bJ9bxM9PRQ=', 'Content-Length': '0'} 2024-02-01T10:48:10.194 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004969cc7bdc39e1be-0065bb76ea-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:10.236 DEBUG:boto:path=/vclfcn-60/obj5 2024-02-01T10:48:10.236 DEBUG:boto:auth_path=/vclfcn-60/obj5 2024-02-01T10:48:10.237 DEBUG:boto:Method: GET 2024-02-01T10:48:10.237 DEBUG:boto:Path: /vclfcn-60/obj5 2024-02-01T10:48:10.237 DEBUG:boto:Data: 2024-02-01T10:48:10.237 DEBUG:boto:Headers: {} 2024-02-01T10:48:10.237 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:10.237 DEBUG:boto:Port: 8000 2024-02-01T10:48:10.237 DEBUG:boto:Params: {} 2024-02-01T10:48:10.237 DEBUG:boto:Token: None 2024-02-01T10:48:10.237 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:10 GMT /vclfcn-60/obj5 2024-02-01T10:48:10.238 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zKc283ft9YhHODgL0bJ9bxM9PRQ= 2024-02-01T10:48:10.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zKc283ft9YhHODgL0bJ9bxM9PRQ=', 'Content-Length': '0'} 2024-02-01T10:48:10.240 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007f3b7c6537dbb097-0065bb76ea-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:10.241 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:10.241 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:10.241 DEBUG:boto:path=/vclfcn-60/?prefix=obj5 2024-02-01T10:48:10.241 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj5 2024-02-01T10:48:10.241 DEBUG:boto:Method: GET 2024-02-01T10:48:10.241 DEBUG:boto:Path: /vclfcn-60/?prefix=obj5 2024-02-01T10:48:10.241 DEBUG:boto:Data: 2024-02-01T10:48:10.241 DEBUG:boto:Headers: {} 2024-02-01T10:48:10.241 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:10.242 DEBUG:boto:Port: 8000 2024-02-01T10:48:10.242 DEBUG:boto:Params: {} 2024-02-01T10:48:10.242 DEBUG:boto:Token: None 2024-02-01T10:48:10.242 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:10 GMT /vclfcn-60/ 2024-02-01T10:48:10.242 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fM7RjZAk6oZ3BsuaYN7sqUQ0eWs= 2024-02-01T10:48:10.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fM7RjZAk6oZ3BsuaYN7sqUQ0eWs=', 'Content-Length': '0'} 2024-02-01T10:48:10.246 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a069c8fa006eb7ff-0065bb76ea-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:10.247 DEBUG:boto:b'vclfcn-60obj51000falseobj52024-02-01T10:47:30.291Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:10.247 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:10.247 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:10.247 DEBUG:boto:path=/vclfcn-60/?prefix=obj5 2024-02-01T10:48:10.247 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj5 2024-02-01T10:48:10.248 DEBUG:boto:Method: GET 2024-02-01T10:48:10.248 DEBUG:boto:Path: /vclfcn-60/?prefix=obj5 2024-02-01T10:48:10.248 DEBUG:boto:Data: 2024-02-01T10:48:10.248 DEBUG:boto:Headers: {} 2024-02-01T10:48:10.248 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:10.248 DEBUG:boto:Port: 8000 2024-02-01T10:48:10.248 DEBUG:boto:Params: {} 2024-02-01T10:48:10.248 DEBUG:boto:Token: None 2024-02-01T10:48:10.248 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:10 GMT /vclfcn-60/ 2024-02-01T10:48:10.249 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fM7RjZAk6oZ3BsuaYN7sqUQ0eWs= 2024-02-01T10:48:10.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fM7RjZAk6oZ3BsuaYN7sqUQ0eWs=', 'Content-Length': '0'} 2024-02-01T10:48:10.254 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012f3ccf1bc59690b-0065bb76ea-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:10.300 DEBUG:boto:b'vclfcn-60obj51000falseobj52024-02-01T10:47:30.291Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:10.301 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-02-01T10:48:10.301 DEBUG:boto:path=/vclfcn-60/obj6 2024-02-01T10:48:10.301 DEBUG:boto:auth_path=/vclfcn-60/obj6 2024-02-01T10:48:10.301 DEBUG:boto:path=/vclfcn-60/obj6?versionId=null 2024-02-01T10:48:10.301 DEBUG:boto:auth_path=/vclfcn-60/obj6?versionId=null 2024-02-01T10:48:10.301 DEBUG:boto:Method: GET 2024-02-01T10:48:10.301 DEBUG:boto:Path: /vclfcn-60/obj6?versionId=null 2024-02-01T10:48:10.301 DEBUG:boto:Data: 2024-02-01T10:48:10.301 DEBUG:boto:Headers: {} 2024-02-01T10:48:10.302 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:10.302 DEBUG:boto:Port: 8000 2024-02-01T10:48:10.302 DEBUG:boto:Params: {} 2024-02-01T10:48:10.302 DEBUG:boto:Token: None 2024-02-01T10:48:10.302 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:10 GMT /vclfcn-60/obj6?versionId=null 2024-02-01T10:48:10.302 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:LgM8HwwYeN+0AlxW5wDS8GeQeX8= 2024-02-01T10:48:10.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:LgM8HwwYeN+0AlxW5wDS8GeQeX8=', 'Content-Length': '0'} 2024-02-01T10:48:10.305 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000054d9e5ce879676d2-0065bb76ea-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:10.306 DEBUG:boto:path=/vclfcn-60/obj6 2024-02-01T10:48:10.306 DEBUG:boto:auth_path=/vclfcn-60/obj6 2024-02-01T10:48:10.306 DEBUG:boto:path=/vclfcn-60/obj6?versionId=null 2024-02-01T10:48:10.306 DEBUG:boto:auth_path=/vclfcn-60/obj6?versionId=null 2024-02-01T10:48:10.306 DEBUG:boto:Method: GET 2024-02-01T10:48:10.306 DEBUG:boto:Path: /vclfcn-60/obj6?versionId=null 2024-02-01T10:48:10.306 DEBUG:boto:Data: 2024-02-01T10:48:10.306 DEBUG:boto:Headers: {} 2024-02-01T10:48:10.306 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:10.307 DEBUG:boto:Port: 8000 2024-02-01T10:48:10.307 DEBUG:boto:Params: {} 2024-02-01T10:48:10.307 DEBUG:boto:Token: None 2024-02-01T10:48:10.307 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:10 GMT /vclfcn-60/obj6?versionId=null 2024-02-01T10:48:10.307 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:LgM8HwwYeN+0AlxW5wDS8GeQeX8= 2024-02-01T10:48:10.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:LgM8HwwYeN+0AlxW5wDS8GeQeX8=', 'Content-Length': '0'} 2024-02-01T10:48:10.310 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000fb78f43795f45c8-0065bb76ea-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:10.311 DEBUG:boto:path=/vclfcn-60/obj6 2024-02-01T10:48:10.311 DEBUG:boto:auth_path=/vclfcn-60/obj6 2024-02-01T10:48:10.311 DEBUG:boto:path=/vclfcn-60/obj6?versionId=null 2024-02-01T10:48:10.311 DEBUG:boto:auth_path=/vclfcn-60/obj6?versionId=null 2024-02-01T10:48:10.311 DEBUG:boto:Method: HEAD 2024-02-01T10:48:10.311 DEBUG:boto:Path: /vclfcn-60/obj6?versionId=null 2024-02-01T10:48:10.311 DEBUG:boto:Data: 2024-02-01T10:48:10.311 DEBUG:boto:Headers: {} 2024-02-01T10:48:10.311 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:10.311 DEBUG:boto:Port: 8000 2024-02-01T10:48:10.311 DEBUG:boto:Params: {} 2024-02-01T10:48:10.312 DEBUG:boto:Token: None 2024-02-01T10:48:10.312 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:10 GMT /vclfcn-60/obj6?versionId=null 2024-02-01T10:48:10.312 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fQvkuRVvq+5P5AeTeNMIFtPWMNs= 2024-02-01T10:48:10.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fQvkuRVvq+5P5AeTeNMIFtPWMNs=', 'Content-Length': '0'} 2024-02-01T10:48:10.318 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f61a173a7d7174fb-0065bb76ea-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:10.318 DEBUG:boto:path=/vclfcn-60/obj6 2024-02-01T10:48:10.318 DEBUG:boto:auth_path=/vclfcn-60/obj6 2024-02-01T10:48:10.318 DEBUG:boto:path=/vclfcn-60/obj6?versionId=null 2024-02-01T10:48:10.318 DEBUG:boto:auth_path=/vclfcn-60/obj6?versionId=null 2024-02-01T10:48:10.319 DEBUG:boto:Method: HEAD 2024-02-01T10:48:10.319 DEBUG:boto:Path: /vclfcn-60/obj6?versionId=null 2024-02-01T10:48:10.319 DEBUG:boto:Data: 2024-02-01T10:48:10.319 DEBUG:boto:Headers: {} 2024-02-01T10:48:10.319 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:10.319 DEBUG:boto:Port: 8000 2024-02-01T10:48:10.319 DEBUG:boto:Params: {} 2024-02-01T10:48:10.319 DEBUG:boto:Token: None 2024-02-01T10:48:10.319 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:10 GMT /vclfcn-60/obj6?versionId=null 2024-02-01T10:48:10.319 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fQvkuRVvq+5P5AeTeNMIFtPWMNs= 2024-02-01T10:48:10.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fQvkuRVvq+5P5AeTeNMIFtPWMNs=', 'Content-Length': '0'} 2024-02-01T10:48:10.323 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005c61635589fcdc91-0065bb76ea-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:10.323 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-02-01T10:48:10.323 DEBUG:boto:path=/vclfcn-60/obj6 2024-02-01T10:48:10.323 DEBUG:boto:auth_path=/vclfcn-60/obj6 2024-02-01T10:48:10.323 DEBUG:boto:path=/vclfcn-60/obj6?versionId=null 2024-02-01T10:48:10.323 DEBUG:boto:auth_path=/vclfcn-60/obj6?versionId=null 2024-02-01T10:48:10.324 DEBUG:boto:Method: GET 2024-02-01T10:48:10.324 DEBUG:boto:Path: /vclfcn-60/obj6?versionId=null 2024-02-01T10:48:10.324 DEBUG:boto:Data: 2024-02-01T10:48:10.324 DEBUG:boto:Headers: {} 2024-02-01T10:48:10.324 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:10.324 DEBUG:boto:Port: 8000 2024-02-01T10:48:10.324 DEBUG:boto:Params: {} 2024-02-01T10:48:10.324 DEBUG:boto:Token: None 2024-02-01T10:48:10.324 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:10 GMT /vclfcn-60/obj6?versionId=null 2024-02-01T10:48:10.324 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:LgM8HwwYeN+0AlxW5wDS8GeQeX8= 2024-02-01T10:48:10.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:LgM8HwwYeN+0AlxW5wDS8GeQeX8=', 'Content-Length': '0'} 2024-02-01T10:48:10.328 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f27936011dafea23-0065bb76ea-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:10.376 DEBUG:boto:path=/vclfcn-60/obj6 2024-02-01T10:48:10.376 DEBUG:boto:auth_path=/vclfcn-60/obj6 2024-02-01T10:48:10.377 DEBUG:boto:path=/vclfcn-60/obj6?versionId=null 2024-02-01T10:48:10.377 DEBUG:boto:auth_path=/vclfcn-60/obj6?versionId=null 2024-02-01T10:48:10.377 DEBUG:boto:Method: GET 2024-02-01T10:48:10.377 DEBUG:boto:Path: /vclfcn-60/obj6?versionId=null 2024-02-01T10:48:10.377 DEBUG:boto:Data: 2024-02-01T10:48:10.377 DEBUG:boto:Headers: {} 2024-02-01T10:48:10.377 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:10.377 DEBUG:boto:Port: 8000 2024-02-01T10:48:10.377 DEBUG:boto:Params: {} 2024-02-01T10:48:10.377 DEBUG:boto:Token: None 2024-02-01T10:48:10.378 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:10 GMT /vclfcn-60/obj6?versionId=null 2024-02-01T10:48:10.378 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:LgM8HwwYeN+0AlxW5wDS8GeQeX8= 2024-02-01T10:48:10.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:LgM8HwwYeN+0AlxW5wDS8GeQeX8=', 'Content-Length': '0'} 2024-02-01T10:48:10.381 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000068ce85bf7a255b9a-0065bb76ea-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:10.381 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:10.381 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:10.381 DEBUG:boto:path=/vclfcn-60/?prefix=obj6 2024-02-01T10:48:10.382 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj6 2024-02-01T10:48:10.382 DEBUG:boto:Method: GET 2024-02-01T10:48:10.382 DEBUG:boto:Path: /vclfcn-60/?prefix=obj6 2024-02-01T10:48:10.382 DEBUG:boto:Data: 2024-02-01T10:48:10.382 DEBUG:boto:Headers: {} 2024-02-01T10:48:10.382 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:10.382 DEBUG:boto:Port: 8000 2024-02-01T10:48:10.382 DEBUG:boto:Params: {} 2024-02-01T10:48:10.382 DEBUG:boto:Token: None 2024-02-01T10:48:10.383 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:10 GMT /vclfcn-60/ 2024-02-01T10:48:10.383 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fM7RjZAk6oZ3BsuaYN7sqUQ0eWs= 2024-02-01T10:48:10.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fM7RjZAk6oZ3BsuaYN7sqUQ0eWs=', 'Content-Length': '0'} 2024-02-01T10:48:10.388 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd5f00ff9afcbd8f-0065bb76ea-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:10.388 DEBUG:boto:b'vclfcn-60obj61000falseobj62024-02-01T10:47:30.343Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:10.388 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:10.389 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:10.389 DEBUG:boto:path=/vclfcn-60/?prefix=obj6 2024-02-01T10:48:10.389 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj6 2024-02-01T10:48:10.389 DEBUG:boto:Method: GET 2024-02-01T10:48:10.389 DEBUG:boto:Path: /vclfcn-60/?prefix=obj6 2024-02-01T10:48:10.389 DEBUG:boto:Data: 2024-02-01T10:48:10.389 DEBUG:boto:Headers: {} 2024-02-01T10:48:10.389 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:10.389 DEBUG:boto:Port: 8000 2024-02-01T10:48:10.389 DEBUG:boto:Params: {} 2024-02-01T10:48:10.389 DEBUG:boto:Token: None 2024-02-01T10:48:10.390 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:10 GMT /vclfcn-60/ 2024-02-01T10:48:10.390 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fM7RjZAk6oZ3BsuaYN7sqUQ0eWs= 2024-02-01T10:48:10.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fM7RjZAk6oZ3BsuaYN7sqUQ0eWs=', 'Content-Length': '0'} 2024-02-01T10:48:10.395 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007de2126d12c92585-0065bb76ea-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:10.436 DEBUG:boto:b'vclfcn-60obj61000falseobj62024-02-01T10:47:30.343Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:10.437 DEBUG:boto:path=/vclfcn-60/obj6 2024-02-01T10:48:10.437 DEBUG:boto:auth_path=/vclfcn-60/obj6 2024-02-01T10:48:10.437 DEBUG:boto:Method: HEAD 2024-02-01T10:48:10.437 DEBUG:boto:Path: /vclfcn-60/obj6 2024-02-01T10:48:10.437 DEBUG:boto:Data: 2024-02-01T10:48:10.437 DEBUG:boto:Headers: {} 2024-02-01T10:48:10.437 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:10.437 DEBUG:boto:Port: 8000 2024-02-01T10:48:10.437 DEBUG:boto:Params: {} 2024-02-01T10:48:10.438 DEBUG:boto:Token: None 2024-02-01T10:48:10.438 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:10 GMT /vclfcn-60/obj6 2024-02-01T10:48:10.438 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:++RtPLsZ83R7nkgPTxFj6n9muSc= 2024-02-01T10:48:10.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:++RtPLsZ83R7nkgPTxFj6n9muSc=', 'Content-Length': '0'} 2024-02-01T10:48:10.441 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000df44e7724c5df41a-0065bb76ea-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:10.441 DEBUG:boto:path=/vclfcn-60/obj6 2024-02-01T10:48:10.441 DEBUG:boto:auth_path=/vclfcn-60/obj6 2024-02-01T10:48:10.441 DEBUG:boto:Method: HEAD 2024-02-01T10:48:10.441 DEBUG:boto:Path: /vclfcn-60/obj6 2024-02-01T10:48:10.441 DEBUG:boto:Data: 2024-02-01T10:48:10.442 DEBUG:boto:Headers: {} 2024-02-01T10:48:10.442 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:10.442 DEBUG:boto:Port: 8000 2024-02-01T10:48:10.442 DEBUG:boto:Params: {} 2024-02-01T10:48:10.442 DEBUG:boto:Token: None 2024-02-01T10:48:10.442 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:10 GMT /vclfcn-60/obj6 2024-02-01T10:48:10.442 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:++RtPLsZ83R7nkgPTxFj6n9muSc= 2024-02-01T10:48:10.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:++RtPLsZ83R7nkgPTxFj6n9muSc=', 'Content-Length': '0'} 2024-02-01T10:48:10.445 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000027ec53a6f91f4f85-0065bb76ea-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:10.445 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-02-01T10:48:10.446 DEBUG:boto:path=/vclfcn-60/obj6 2024-02-01T10:48:10.446 DEBUG:boto:auth_path=/vclfcn-60/obj6 2024-02-01T10:48:10.446 DEBUG:boto:Method: GET 2024-02-01T10:48:10.446 DEBUG:boto:Path: /vclfcn-60/obj6 2024-02-01T10:48:10.446 DEBUG:boto:Data: 2024-02-01T10:48:10.446 DEBUG:boto:Headers: {} 2024-02-01T10:48:10.446 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:10.446 DEBUG:boto:Port: 8000 2024-02-01T10:48:10.446 DEBUG:boto:Params: {} 2024-02-01T10:48:10.446 DEBUG:boto:Token: None 2024-02-01T10:48:10.447 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:10 GMT /vclfcn-60/obj6 2024-02-01T10:48:10.447 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wsBkUbo4eFA4/Lc9Wfskajzt+lU= 2024-02-01T10:48:10.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wsBkUbo4eFA4/Lc9Wfskajzt+lU=', 'Content-Length': '0'} 2024-02-01T10:48:10.449 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000397580153c82c4f3-0065bb76ea-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:10.450 DEBUG:boto:path=/vclfcn-60/obj6 2024-02-01T10:48:10.450 DEBUG:boto:auth_path=/vclfcn-60/obj6 2024-02-01T10:48:10.450 DEBUG:boto:Method: GET 2024-02-01T10:48:10.450 DEBUG:boto:Path: /vclfcn-60/obj6 2024-02-01T10:48:10.450 DEBUG:boto:Data: 2024-02-01T10:48:10.450 DEBUG:boto:Headers: {} 2024-02-01T10:48:10.450 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:10.450 DEBUG:boto:Port: 8000 2024-02-01T10:48:10.450 DEBUG:boto:Params: {} 2024-02-01T10:48:10.451 DEBUG:boto:Token: None 2024-02-01T10:48:10.451 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:10 GMT /vclfcn-60/obj6 2024-02-01T10:48:10.451 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wsBkUbo4eFA4/Lc9Wfskajzt+lU= 2024-02-01T10:48:10.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wsBkUbo4eFA4/Lc9Wfskajzt+lU=', 'Content-Length': '0'} 2024-02-01T10:48:10.454 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000084c0e98a9747b53f-0065bb76ea-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:10.454 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:10.454 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:10.454 DEBUG:boto:path=/vclfcn-60/?prefix=obj6 2024-02-01T10:48:10.454 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj6 2024-02-01T10:48:10.455 DEBUG:boto:Method: GET 2024-02-01T10:48:10.455 DEBUG:boto:Path: /vclfcn-60/?prefix=obj6 2024-02-01T10:48:10.455 DEBUG:boto:Data: 2024-02-01T10:48:10.455 DEBUG:boto:Headers: {} 2024-02-01T10:48:10.455 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:10.455 DEBUG:boto:Port: 8000 2024-02-01T10:48:10.455 DEBUG:boto:Params: {} 2024-02-01T10:48:10.455 DEBUG:boto:Token: None 2024-02-01T10:48:10.455 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:10 GMT /vclfcn-60/ 2024-02-01T10:48:10.455 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fM7RjZAk6oZ3BsuaYN7sqUQ0eWs= 2024-02-01T10:48:10.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fM7RjZAk6oZ3BsuaYN7sqUQ0eWs=', 'Content-Length': '0'} 2024-02-01T10:48:10.460 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba6decaf23e705a6-0065bb76ea-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:10.508 DEBUG:boto:b'vclfcn-60obj61000falseobj62024-02-01T10:47:30.343Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:10.509 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:10.509 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:10.509 DEBUG:boto:path=/vclfcn-60/?prefix=obj6 2024-02-01T10:48:10.509 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj6 2024-02-01T10:48:10.509 DEBUG:boto:Method: GET 2024-02-01T10:48:10.509 DEBUG:boto:Path: /vclfcn-60/?prefix=obj6 2024-02-01T10:48:10.509 DEBUG:boto:Data: 2024-02-01T10:48:10.509 DEBUG:boto:Headers: {} 2024-02-01T10:48:10.509 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:10.509 DEBUG:boto:Port: 8000 2024-02-01T10:48:10.510 DEBUG:boto:Params: {} 2024-02-01T10:48:10.510 DEBUG:boto:Token: None 2024-02-01T10:48:10.510 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:10 GMT /vclfcn-60/ 2024-02-01T10:48:10.510 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fM7RjZAk6oZ3BsuaYN7sqUQ0eWs= 2024-02-01T10:48:10.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fM7RjZAk6oZ3BsuaYN7sqUQ0eWs=', 'Content-Length': '0'} 2024-02-01T10:48:10.515 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000014bde05128aca156-0065bb76ea-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:10.515 DEBUG:boto:b'vclfcn-60obj61000falseobj62024-02-01T10:47:30.343Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:10.516 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-02-01T10:48:10.516 DEBUG:boto:path=/vclfcn-60/obj7 2024-02-01T10:48:10.519 DEBUG:boto:auth_path=/vclfcn-60/obj7 2024-02-01T10:48:10.519 DEBUG:boto:path=/vclfcn-60/obj7?versionId=null 2024-02-01T10:48:10.519 DEBUG:boto:auth_path=/vclfcn-60/obj7?versionId=null 2024-02-01T10:48:10.519 DEBUG:boto:Method: GET 2024-02-01T10:48:10.519 DEBUG:boto:Path: /vclfcn-60/obj7?versionId=null 2024-02-01T10:48:10.520 DEBUG:boto:Data: 2024-02-01T10:48:10.520 DEBUG:boto:Headers: {} 2024-02-01T10:48:10.520 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:10.520 DEBUG:boto:Port: 8000 2024-02-01T10:48:10.520 DEBUG:boto:Params: {} 2024-02-01T10:48:10.520 DEBUG:boto:Token: None 2024-02-01T10:48:10.520 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:10 GMT /vclfcn-60/obj7?versionId=null 2024-02-01T10:48:10.520 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0TbAi3KtAE2FijlW2H76bvh1qEI= 2024-02-01T10:48:10.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0TbAi3KtAE2FijlW2H76bvh1qEI=', 'Content-Length': '0'} 2024-02-01T10:48:10.523 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bb62d7ad3ef2ee0d-0065bb76ea-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:10.524 DEBUG:boto:path=/vclfcn-60/obj7 2024-02-01T10:48:10.524 DEBUG:boto:auth_path=/vclfcn-60/obj7 2024-02-01T10:48:10.524 DEBUG:boto:path=/vclfcn-60/obj7?versionId=null 2024-02-01T10:48:10.524 DEBUG:boto:auth_path=/vclfcn-60/obj7?versionId=null 2024-02-01T10:48:10.524 DEBUG:boto:Method: GET 2024-02-01T10:48:10.524 DEBUG:boto:Path: /vclfcn-60/obj7?versionId=null 2024-02-01T10:48:10.524 DEBUG:boto:Data: 2024-02-01T10:48:10.524 DEBUG:boto:Headers: {} 2024-02-01T10:48:10.524 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:10.525 DEBUG:boto:Port: 8000 2024-02-01T10:48:10.525 DEBUG:boto:Params: {} 2024-02-01T10:48:10.525 DEBUG:boto:Token: None 2024-02-01T10:48:10.525 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:10 GMT /vclfcn-60/obj7?versionId=null 2024-02-01T10:48:10.525 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0TbAi3KtAE2FijlW2H76bvh1qEI= 2024-02-01T10:48:10.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0TbAi3KtAE2FijlW2H76bvh1qEI=', 'Content-Length': '0'} 2024-02-01T10:48:10.528 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008049cbe858aef831-0065bb76ea-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:10.572 DEBUG:boto:path=/vclfcn-60/obj7 2024-02-01T10:48:10.572 DEBUG:boto:auth_path=/vclfcn-60/obj7 2024-02-01T10:48:10.573 DEBUG:boto:path=/vclfcn-60/obj7?versionId=null 2024-02-01T10:48:10.573 DEBUG:boto:auth_path=/vclfcn-60/obj7?versionId=null 2024-02-01T10:48:10.573 DEBUG:boto:Method: HEAD 2024-02-01T10:48:10.573 DEBUG:boto:Path: /vclfcn-60/obj7?versionId=null 2024-02-01T10:48:10.573 DEBUG:boto:Data: 2024-02-01T10:48:10.573 DEBUG:boto:Headers: {} 2024-02-01T10:48:10.573 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:10.573 DEBUG:boto:Port: 8000 2024-02-01T10:48:10.573 DEBUG:boto:Params: {} 2024-02-01T10:48:10.573 DEBUG:boto:Token: None 2024-02-01T10:48:10.574 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:10 GMT /vclfcn-60/obj7?versionId=null 2024-02-01T10:48:10.574 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:kOBspU4ucGjIlZ/TXWHZCazY+jg= 2024-02-01T10:48:10.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:kOBspU4ucGjIlZ/TXWHZCazY+jg=', 'Content-Length': '0'} 2024-02-01T10:48:10.577 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002454a6aad1c21558-0065bb76ea-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:10.577 DEBUG:boto:path=/vclfcn-60/obj7 2024-02-01T10:48:10.577 DEBUG:boto:auth_path=/vclfcn-60/obj7 2024-02-01T10:48:10.577 DEBUG:boto:path=/vclfcn-60/obj7?versionId=null 2024-02-01T10:48:10.577 DEBUG:boto:auth_path=/vclfcn-60/obj7?versionId=null 2024-02-01T10:48:10.577 DEBUG:boto:Method: HEAD 2024-02-01T10:48:10.578 DEBUG:boto:Path: /vclfcn-60/obj7?versionId=null 2024-02-01T10:48:10.578 DEBUG:boto:Data: 2024-02-01T10:48:10.578 DEBUG:boto:Headers: {} 2024-02-01T10:48:10.578 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:10.578 DEBUG:boto:Port: 8000 2024-02-01T10:48:10.578 DEBUG:boto:Params: {} 2024-02-01T10:48:10.578 DEBUG:boto:Token: None 2024-02-01T10:48:10.578 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:10 GMT /vclfcn-60/obj7?versionId=null 2024-02-01T10:48:10.578 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:kOBspU4ucGjIlZ/TXWHZCazY+jg= 2024-02-01T10:48:10.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:kOBspU4ucGjIlZ/TXWHZCazY+jg=', 'Content-Length': '0'} 2024-02-01T10:48:10.582 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cd83b3b94d0d436a-0065bb76ea-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:10.582 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-02-01T10:48:10.582 DEBUG:boto:path=/vclfcn-60/obj7 2024-02-01T10:48:10.582 DEBUG:boto:auth_path=/vclfcn-60/obj7 2024-02-01T10:48:10.582 DEBUG:boto:path=/vclfcn-60/obj7?versionId=null 2024-02-01T10:48:10.582 DEBUG:boto:auth_path=/vclfcn-60/obj7?versionId=null 2024-02-01T10:48:10.582 DEBUG:boto:Method: GET 2024-02-01T10:48:10.583 DEBUG:boto:Path: /vclfcn-60/obj7?versionId=null 2024-02-01T10:48:10.583 DEBUG:boto:Data: 2024-02-01T10:48:10.583 DEBUG:boto:Headers: {} 2024-02-01T10:48:10.583 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:10.583 DEBUG:boto:Port: 8000 2024-02-01T10:48:10.583 DEBUG:boto:Params: {} 2024-02-01T10:48:10.583 DEBUG:boto:Token: None 2024-02-01T10:48:10.583 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:10 GMT /vclfcn-60/obj7?versionId=null 2024-02-01T10:48:10.583 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0TbAi3KtAE2FijlW2H76bvh1qEI= 2024-02-01T10:48:10.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0TbAi3KtAE2FijlW2H76bvh1qEI=', 'Content-Length': '0'} 2024-02-01T10:48:10.586 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000073302917f0f4e755-0065bb76ea-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:10.586 DEBUG:boto:path=/vclfcn-60/obj7 2024-02-01T10:48:10.587 DEBUG:boto:auth_path=/vclfcn-60/obj7 2024-02-01T10:48:10.587 DEBUG:boto:path=/vclfcn-60/obj7?versionId=null 2024-02-01T10:48:10.587 DEBUG:boto:auth_path=/vclfcn-60/obj7?versionId=null 2024-02-01T10:48:10.587 DEBUG:boto:Method: GET 2024-02-01T10:48:10.587 DEBUG:boto:Path: /vclfcn-60/obj7?versionId=null 2024-02-01T10:48:10.587 DEBUG:boto:Data: 2024-02-01T10:48:10.587 DEBUG:boto:Headers: {} 2024-02-01T10:48:10.587 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:10.587 DEBUG:boto:Port: 8000 2024-02-01T10:48:10.587 DEBUG:boto:Params: {} 2024-02-01T10:48:10.587 DEBUG:boto:Token: None 2024-02-01T10:48:10.588 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:10 GMT /vclfcn-60/obj7?versionId=null 2024-02-01T10:48:10.588 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0TbAi3KtAE2FijlW2H76bvh1qEI= 2024-02-01T10:48:10.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0TbAi3KtAE2FijlW2H76bvh1qEI=', 'Content-Length': '0'} 2024-02-01T10:48:10.590 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b9a4aef96f270c3b-0065bb76ea-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:10.632 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:10.632 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:10.632 DEBUG:boto:path=/vclfcn-60/?prefix=obj7 2024-02-01T10:48:10.633 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj7 2024-02-01T10:48:10.633 DEBUG:boto:Method: GET 2024-02-01T10:48:10.633 DEBUG:boto:Path: /vclfcn-60/?prefix=obj7 2024-02-01T10:48:10.633 DEBUG:boto:Data: 2024-02-01T10:48:10.633 DEBUG:boto:Headers: {} 2024-02-01T10:48:10.633 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:10.633 DEBUG:boto:Port: 8000 2024-02-01T10:48:10.633 DEBUG:boto:Params: {} 2024-02-01T10:48:10.633 DEBUG:boto:Token: None 2024-02-01T10:48:10.634 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:10 GMT /vclfcn-60/ 2024-02-01T10:48:10.634 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fM7RjZAk6oZ3BsuaYN7sqUQ0eWs= 2024-02-01T10:48:10.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fM7RjZAk6oZ3BsuaYN7sqUQ0eWs=', 'Content-Length': '0'} 2024-02-01T10:48:10.639 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b2623e5cd652620c-0065bb76ea-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:10.639 DEBUG:boto:b'vclfcn-60obj71000falseobj72024-02-01T10:47:30.358Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:10.639 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:10.640 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:10.640 DEBUG:boto:path=/vclfcn-60/?prefix=obj7 2024-02-01T10:48:10.640 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj7 2024-02-01T10:48:10.640 DEBUG:boto:Method: GET 2024-02-01T10:48:10.640 DEBUG:boto:Path: /vclfcn-60/?prefix=obj7 2024-02-01T10:48:10.640 DEBUG:boto:Data: 2024-02-01T10:48:10.640 DEBUG:boto:Headers: {} 2024-02-01T10:48:10.640 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:10.640 DEBUG:boto:Port: 8000 2024-02-01T10:48:10.640 DEBUG:boto:Params: {} 2024-02-01T10:48:10.640 DEBUG:boto:Token: None 2024-02-01T10:48:10.641 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:10 GMT /vclfcn-60/ 2024-02-01T10:48:10.641 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fM7RjZAk6oZ3BsuaYN7sqUQ0eWs= 2024-02-01T10:48:10.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fM7RjZAk6oZ3BsuaYN7sqUQ0eWs=', 'Content-Length': '0'} 2024-02-01T10:48:10.646 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002132660762c7b85e-0065bb76ea-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:10.688 DEBUG:boto:b'vclfcn-60obj71000falseobj72024-02-01T10:47:30.358Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:10.689 DEBUG:boto:path=/vclfcn-60/obj7 2024-02-01T10:48:10.689 DEBUG:boto:auth_path=/vclfcn-60/obj7 2024-02-01T10:48:10.689 DEBUG:boto:Method: HEAD 2024-02-01T10:48:10.689 DEBUG:boto:Path: /vclfcn-60/obj7 2024-02-01T10:48:10.689 DEBUG:boto:Data: 2024-02-01T10:48:10.689 DEBUG:boto:Headers: {} 2024-02-01T10:48:10.689 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:10.689 DEBUG:boto:Port: 8000 2024-02-01T10:48:10.689 DEBUG:boto:Params: {} 2024-02-01T10:48:10.690 DEBUG:boto:Token: None 2024-02-01T10:48:10.690 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:10 GMT /vclfcn-60/obj7 2024-02-01T10:48:10.690 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:4NbnILo+Aa7JL9IwKrkpiGJk1PU= 2024-02-01T10:48:10.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:4NbnILo+Aa7JL9IwKrkpiGJk1PU=', 'Content-Length': '0'} 2024-02-01T10:48:10.693 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f7143a20459b3e78-0065bb76ea-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:10.693 DEBUG:boto:path=/vclfcn-60/obj7 2024-02-01T10:48:10.693 DEBUG:boto:auth_path=/vclfcn-60/obj7 2024-02-01T10:48:10.693 DEBUG:boto:Method: HEAD 2024-02-01T10:48:10.694 DEBUG:boto:Path: /vclfcn-60/obj7 2024-02-01T10:48:10.694 DEBUG:boto:Data: 2024-02-01T10:48:10.694 DEBUG:boto:Headers: {} 2024-02-01T10:48:10.694 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:10.694 DEBUG:boto:Port: 8000 2024-02-01T10:48:10.694 DEBUG:boto:Params: {} 2024-02-01T10:48:10.694 DEBUG:boto:Token: None 2024-02-01T10:48:10.694 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:10 GMT /vclfcn-60/obj7 2024-02-01T10:48:10.694 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:4NbnILo+Aa7JL9IwKrkpiGJk1PU= 2024-02-01T10:48:10.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:4NbnILo+Aa7JL9IwKrkpiGJk1PU=', 'Content-Length': '0'} 2024-02-01T10:48:10.697 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000626409fd2209074c-0065bb76ea-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:10.698 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-02-01T10:48:10.698 DEBUG:boto:path=/vclfcn-60/obj7 2024-02-01T10:48:10.698 DEBUG:boto:auth_path=/vclfcn-60/obj7 2024-02-01T10:48:10.698 DEBUG:boto:Method: GET 2024-02-01T10:48:10.698 DEBUG:boto:Path: /vclfcn-60/obj7 2024-02-01T10:48:10.698 DEBUG:boto:Data: 2024-02-01T10:48:10.698 DEBUG:boto:Headers: {} 2024-02-01T10:48:10.698 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:10.698 DEBUG:boto:Port: 8000 2024-02-01T10:48:10.699 DEBUG:boto:Params: {} 2024-02-01T10:48:10.699 DEBUG:boto:Token: None 2024-02-01T10:48:10.699 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:10 GMT /vclfcn-60/obj7 2024-02-01T10:48:10.699 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:iugaYT6Lc4ZD9ygYIaxHeFzix5U= 2024-02-01T10:48:10.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iugaYT6Lc4ZD9ygYIaxHeFzix5U=', 'Content-Length': '0'} 2024-02-01T10:48:10.702 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007e918a280fc67347-0065bb76ea-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:10.744 DEBUG:boto:path=/vclfcn-60/obj7 2024-02-01T10:48:10.744 DEBUG:boto:auth_path=/vclfcn-60/obj7 2024-02-01T10:48:10.744 DEBUG:boto:Method: GET 2024-02-01T10:48:10.745 DEBUG:boto:Path: /vclfcn-60/obj7 2024-02-01T10:48:10.745 DEBUG:boto:Data: 2024-02-01T10:48:10.745 DEBUG:boto:Headers: {} 2024-02-01T10:48:10.745 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:10.745 DEBUG:boto:Port: 8000 2024-02-01T10:48:10.745 DEBUG:boto:Params: {} 2024-02-01T10:48:10.745 DEBUG:boto:Token: None 2024-02-01T10:48:10.745 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:10 GMT /vclfcn-60/obj7 2024-02-01T10:48:10.746 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:iugaYT6Lc4ZD9ygYIaxHeFzix5U= 2024-02-01T10:48:10.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iugaYT6Lc4ZD9ygYIaxHeFzix5U=', 'Content-Length': '0'} 2024-02-01T10:48:10.748 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fc582c70903e80a4-0065bb76ea-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:10.792 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:10.792 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:10.792 DEBUG:boto:path=/vclfcn-60/?prefix=obj7 2024-02-01T10:48:10.793 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj7 2024-02-01T10:48:10.793 DEBUG:boto:Method: GET 2024-02-01T10:48:10.793 DEBUG:boto:Path: /vclfcn-60/?prefix=obj7 2024-02-01T10:48:10.793 DEBUG:boto:Data: 2024-02-01T10:48:10.793 DEBUG:boto:Headers: {} 2024-02-01T10:48:10.793 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:10.793 DEBUG:boto:Port: 8000 2024-02-01T10:48:10.793 DEBUG:boto:Params: {} 2024-02-01T10:48:10.793 DEBUG:boto:Token: None 2024-02-01T10:48:10.794 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:10 GMT /vclfcn-60/ 2024-02-01T10:48:10.794 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fM7RjZAk6oZ3BsuaYN7sqUQ0eWs= 2024-02-01T10:48:10.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fM7RjZAk6oZ3BsuaYN7sqUQ0eWs=', 'Content-Length': '0'} 2024-02-01T10:48:10.798 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000292fe21ce88ce0ed-0065bb76ea-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:10.799 DEBUG:boto:b'vclfcn-60obj71000falseobj72024-02-01T10:47:30.358Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:10.799 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:10.799 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:10.799 DEBUG:boto:path=/vclfcn-60/?prefix=obj7 2024-02-01T10:48:10.799 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj7 2024-02-01T10:48:10.799 DEBUG:boto:Method: GET 2024-02-01T10:48:10.800 DEBUG:boto:Path: /vclfcn-60/?prefix=obj7 2024-02-01T10:48:10.800 DEBUG:boto:Data: 2024-02-01T10:48:10.800 DEBUG:boto:Headers: {} 2024-02-01T10:48:10.800 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:10.800 DEBUG:boto:Port: 8000 2024-02-01T10:48:10.800 DEBUG:boto:Params: {} 2024-02-01T10:48:10.800 DEBUG:boto:Token: None 2024-02-01T10:48:10.800 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:10 GMT /vclfcn-60/ 2024-02-01T10:48:10.800 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fM7RjZAk6oZ3BsuaYN7sqUQ0eWs= 2024-02-01T10:48:10.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fM7RjZAk6oZ3BsuaYN7sqUQ0eWs=', 'Content-Length': '0'} 2024-02-01T10:48:10.806 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d0e07a2b3f81493-0065bb76ea-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:10.807 DEBUG:boto:b'vclfcn-60obj71000falseobj72024-02-01T10:47:30.358Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:10.807 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-02-01T10:48:10.807 DEBUG:boto:path=/vclfcn-60/obj8 2024-02-01T10:48:10.807 DEBUG:boto:auth_path=/vclfcn-60/obj8 2024-02-01T10:48:10.807 DEBUG:boto:path=/vclfcn-60/obj8?versionId=null 2024-02-01T10:48:10.807 DEBUG:boto:auth_path=/vclfcn-60/obj8?versionId=null 2024-02-01T10:48:10.808 DEBUG:boto:Method: GET 2024-02-01T10:48:10.808 DEBUG:boto:Path: /vclfcn-60/obj8?versionId=null 2024-02-01T10:48:10.808 DEBUG:boto:Data: 2024-02-01T10:48:10.808 DEBUG:boto:Headers: {} 2024-02-01T10:48:10.808 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:10.808 DEBUG:boto:Port: 8000 2024-02-01T10:48:10.808 DEBUG:boto:Params: {} 2024-02-01T10:48:10.808 DEBUG:boto:Token: None 2024-02-01T10:48:10.808 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:10 GMT /vclfcn-60/obj8?versionId=null 2024-02-01T10:48:10.808 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FFGg6xFpQyLNUMJ0Fs6hDeVCv/o= 2024-02-01T10:48:10.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FFGg6xFpQyLNUMJ0Fs6hDeVCv/o=', 'Content-Length': '0'} 2024-02-01T10:48:10.811 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c39903b3572303dc-0065bb76ea-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:10.812 DEBUG:boto:path=/vclfcn-60/obj8 2024-02-01T10:48:10.812 DEBUG:boto:auth_path=/vclfcn-60/obj8 2024-02-01T10:48:10.812 DEBUG:boto:path=/vclfcn-60/obj8?versionId=null 2024-02-01T10:48:10.812 DEBUG:boto:auth_path=/vclfcn-60/obj8?versionId=null 2024-02-01T10:48:10.812 DEBUG:boto:Method: GET 2024-02-01T10:48:10.812 DEBUG:boto:Path: /vclfcn-60/obj8?versionId=null 2024-02-01T10:48:10.812 DEBUG:boto:Data: 2024-02-01T10:48:10.812 DEBUG:boto:Headers: {} 2024-02-01T10:48:10.812 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:10.812 DEBUG:boto:Port: 8000 2024-02-01T10:48:10.813 DEBUG:boto:Params: {} 2024-02-01T10:48:10.813 DEBUG:boto:Token: None 2024-02-01T10:48:10.813 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:10 GMT /vclfcn-60/obj8?versionId=null 2024-02-01T10:48:10.813 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FFGg6xFpQyLNUMJ0Fs6hDeVCv/o= 2024-02-01T10:48:10.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FFGg6xFpQyLNUMJ0Fs6hDeVCv/o=', 'Content-Length': '0'} 2024-02-01T10:48:10.816 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c802ac182ed96495-0065bb76ea-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:10.860 DEBUG:boto:path=/vclfcn-60/obj8 2024-02-01T10:48:10.860 DEBUG:boto:auth_path=/vclfcn-60/obj8 2024-02-01T10:48:10.860 DEBUG:boto:path=/vclfcn-60/obj8?versionId=null 2024-02-01T10:48:10.860 DEBUG:boto:auth_path=/vclfcn-60/obj8?versionId=null 2024-02-01T10:48:10.861 DEBUG:boto:Method: HEAD 2024-02-01T10:48:10.861 DEBUG:boto:Path: /vclfcn-60/obj8?versionId=null 2024-02-01T10:48:10.861 DEBUG:boto:Data: 2024-02-01T10:48:10.861 DEBUG:boto:Headers: {} 2024-02-01T10:48:10.861 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:10.861 DEBUG:boto:Port: 8000 2024-02-01T10:48:10.861 DEBUG:boto:Params: {} 2024-02-01T10:48:10.862 DEBUG:boto:Token: None 2024-02-01T10:48:10.862 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:10 GMT /vclfcn-60/obj8?versionId=null 2024-02-01T10:48:10.862 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jOxCzlpYgNlvTLVOK1icumrfaY8= 2024-02-01T10:48:10.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jOxCzlpYgNlvTLVOK1icumrfaY8=', 'Content-Length': '0'} 2024-02-01T10:48:10.865 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e947d03eae55eefc-0065bb76ea-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:10.865 DEBUG:boto:path=/vclfcn-60/obj8 2024-02-01T10:48:10.865 DEBUG:boto:auth_path=/vclfcn-60/obj8 2024-02-01T10:48:10.865 DEBUG:boto:path=/vclfcn-60/obj8?versionId=null 2024-02-01T10:48:10.865 DEBUG:boto:auth_path=/vclfcn-60/obj8?versionId=null 2024-02-01T10:48:10.865 DEBUG:boto:Method: HEAD 2024-02-01T10:48:10.866 DEBUG:boto:Path: /vclfcn-60/obj8?versionId=null 2024-02-01T10:48:10.866 DEBUG:boto:Data: 2024-02-01T10:48:10.866 DEBUG:boto:Headers: {} 2024-02-01T10:48:10.866 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:10.866 DEBUG:boto:Port: 8000 2024-02-01T10:48:10.866 DEBUG:boto:Params: {} 2024-02-01T10:48:10.866 DEBUG:boto:Token: None 2024-02-01T10:48:10.866 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:10 GMT /vclfcn-60/obj8?versionId=null 2024-02-01T10:48:10.866 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jOxCzlpYgNlvTLVOK1icumrfaY8= 2024-02-01T10:48:10.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jOxCzlpYgNlvTLVOK1icumrfaY8=', 'Content-Length': '0'} 2024-02-01T10:48:10.869 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000376e6d3787cfb4b2-0065bb76ea-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:10.870 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-02-01T10:48:10.870 DEBUG:boto:path=/vclfcn-60/obj8 2024-02-01T10:48:10.870 DEBUG:boto:auth_path=/vclfcn-60/obj8 2024-02-01T10:48:10.870 DEBUG:boto:path=/vclfcn-60/obj8?versionId=null 2024-02-01T10:48:10.870 DEBUG:boto:auth_path=/vclfcn-60/obj8?versionId=null 2024-02-01T10:48:10.871 DEBUG:boto:Method: GET 2024-02-01T10:48:10.871 DEBUG:boto:Path: /vclfcn-60/obj8?versionId=null 2024-02-01T10:48:10.871 DEBUG:boto:Data: 2024-02-01T10:48:10.871 DEBUG:boto:Headers: {} 2024-02-01T10:48:10.871 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:10.871 DEBUG:boto:Port: 8000 2024-02-01T10:48:10.871 DEBUG:boto:Params: {} 2024-02-01T10:48:10.871 DEBUG:boto:Token: None 2024-02-01T10:48:10.872 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:10 GMT /vclfcn-60/obj8?versionId=null 2024-02-01T10:48:10.872 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FFGg6xFpQyLNUMJ0Fs6hDeVCv/o= 2024-02-01T10:48:10.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FFGg6xFpQyLNUMJ0Fs6hDeVCv/o=', 'Content-Length': '0'} 2024-02-01T10:48:10.875 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bddb685ba7d60e87-0065bb76ea-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:10.875 DEBUG:boto:path=/vclfcn-60/obj8 2024-02-01T10:48:10.875 DEBUG:boto:auth_path=/vclfcn-60/obj8 2024-02-01T10:48:10.875 DEBUG:boto:path=/vclfcn-60/obj8?versionId=null 2024-02-01T10:48:10.875 DEBUG:boto:auth_path=/vclfcn-60/obj8?versionId=null 2024-02-01T10:48:10.875 DEBUG:boto:Method: GET 2024-02-01T10:48:10.875 DEBUG:boto:Path: /vclfcn-60/obj8?versionId=null 2024-02-01T10:48:10.876 DEBUG:boto:Data: 2024-02-01T10:48:10.876 DEBUG:boto:Headers: {} 2024-02-01T10:48:10.876 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:10.876 DEBUG:boto:Port: 8000 2024-02-01T10:48:10.876 DEBUG:boto:Params: {} 2024-02-01T10:48:10.876 DEBUG:boto:Token: None 2024-02-01T10:48:10.876 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:10 GMT /vclfcn-60/obj8?versionId=null 2024-02-01T10:48:10.876 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FFGg6xFpQyLNUMJ0Fs6hDeVCv/o= 2024-02-01T10:48:10.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FFGg6xFpQyLNUMJ0Fs6hDeVCv/o=', 'Content-Length': '0'} 2024-02-01T10:48:10.879 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005b857ace426e8faa-0065bb76ea-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:10.920 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:10.920 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:10.920 DEBUG:boto:path=/vclfcn-60/?prefix=obj8 2024-02-01T10:48:10.920 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj8 2024-02-01T10:48:10.920 DEBUG:boto:Method: GET 2024-02-01T10:48:10.921 DEBUG:boto:Path: /vclfcn-60/?prefix=obj8 2024-02-01T10:48:10.921 DEBUG:boto:Data: 2024-02-01T10:48:10.921 DEBUG:boto:Headers: {} 2024-02-01T10:48:10.921 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:10.921 DEBUG:boto:Port: 8000 2024-02-01T10:48:10.921 DEBUG:boto:Params: {} 2024-02-01T10:48:10.922 DEBUG:boto:Token: None 2024-02-01T10:48:10.922 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:10 GMT /vclfcn-60/ 2024-02-01T10:48:10.922 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fM7RjZAk6oZ3BsuaYN7sqUQ0eWs= 2024-02-01T10:48:10.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fM7RjZAk6oZ3BsuaYN7sqUQ0eWs=', 'Content-Length': '0'} 2024-02-01T10:48:10.926 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd859bbf47552582-0065bb76ea-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:10.927 DEBUG:boto:b'vclfcn-60obj81000falseobj82024-02-01T10:47:30.408Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:10.927 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:10.927 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:10.927 DEBUG:boto:path=/vclfcn-60/?prefix=obj8 2024-02-01T10:48:10.927 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj8 2024-02-01T10:48:10.928 DEBUG:boto:Method: GET 2024-02-01T10:48:10.928 DEBUG:boto:Path: /vclfcn-60/?prefix=obj8 2024-02-01T10:48:10.928 DEBUG:boto:Data: 2024-02-01T10:48:10.928 DEBUG:boto:Headers: {} 2024-02-01T10:48:10.928 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:10.928 DEBUG:boto:Port: 8000 2024-02-01T10:48:10.928 DEBUG:boto:Params: {} 2024-02-01T10:48:10.931 DEBUG:boto:Token: None 2024-02-01T10:48:10.931 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:10 GMT /vclfcn-60/ 2024-02-01T10:48:10.931 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fM7RjZAk6oZ3BsuaYN7sqUQ0eWs= 2024-02-01T10:48:10.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fM7RjZAk6oZ3BsuaYN7sqUQ0eWs=', 'Content-Length': '0'} 2024-02-01T10:48:10.936 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000006d55bb6a79b2ee9-0065bb76ea-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:10.980 DEBUG:boto:b'vclfcn-60obj81000falseobj82024-02-01T10:47:30.408Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:10.981 DEBUG:boto:path=/vclfcn-60/obj8 2024-02-01T10:48:10.981 DEBUG:boto:auth_path=/vclfcn-60/obj8 2024-02-01T10:48:10.981 DEBUG:boto:Method: HEAD 2024-02-01T10:48:10.981 DEBUG:boto:Path: /vclfcn-60/obj8 2024-02-01T10:48:10.981 DEBUG:boto:Data: 2024-02-01T10:48:10.981 DEBUG:boto:Headers: {} 2024-02-01T10:48:10.981 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:10.981 DEBUG:boto:Port: 8000 2024-02-01T10:48:10.981 DEBUG:boto:Params: {} 2024-02-01T10:48:10.981 DEBUG:boto:Token: None 2024-02-01T10:48:10.982 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:10 GMT /vclfcn-60/obj8 2024-02-01T10:48:10.982 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:35/A2MbT9wYYTay+kvYazthU9Hs= 2024-02-01T10:48:10.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:35/A2MbT9wYYTay+kvYazthU9Hs=', 'Content-Length': '0'} 2024-02-01T10:48:10.985 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008f2434b4e77239b0-0065bb76ea-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:10.985 DEBUG:boto:path=/vclfcn-60/obj8 2024-02-01T10:48:10.985 DEBUG:boto:auth_path=/vclfcn-60/obj8 2024-02-01T10:48:10.985 DEBUG:boto:Method: HEAD 2024-02-01T10:48:10.985 DEBUG:boto:Path: /vclfcn-60/obj8 2024-02-01T10:48:10.985 DEBUG:boto:Data: 2024-02-01T10:48:10.985 DEBUG:boto:Headers: {} 2024-02-01T10:48:10.985 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:10.986 DEBUG:boto:Port: 8000 2024-02-01T10:48:10.986 DEBUG:boto:Params: {} 2024-02-01T10:48:10.986 DEBUG:boto:Token: None 2024-02-01T10:48:10.986 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:10 GMT /vclfcn-60/obj8 2024-02-01T10:48:10.986 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:35/A2MbT9wYYTay+kvYazthU9Hs= 2024-02-01T10:48:10.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:35/A2MbT9wYYTay+kvYazthU9Hs=', 'Content-Length': '0'} 2024-02-01T10:48:10.989 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000987d9809591d3734-0065bb76ea-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:10.989 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-02-01T10:48:10.990 DEBUG:boto:path=/vclfcn-60/obj8 2024-02-01T10:48:10.990 DEBUG:boto:auth_path=/vclfcn-60/obj8 2024-02-01T10:48:10.990 DEBUG:boto:Method: GET 2024-02-01T10:48:10.990 DEBUG:boto:Path: /vclfcn-60/obj8 2024-02-01T10:48:10.990 DEBUG:boto:Data: 2024-02-01T10:48:10.990 DEBUG:boto:Headers: {} 2024-02-01T10:48:10.990 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:10.990 DEBUG:boto:Port: 8000 2024-02-01T10:48:10.990 DEBUG:boto:Params: {} 2024-02-01T10:48:10.990 DEBUG:boto:Token: None 2024-02-01T10:48:10.991 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:10 GMT /vclfcn-60/obj8 2024-02-01T10:48:10.991 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:IHhZDaBHYWC8Ww+QEQ9P90PU7PI= 2024-02-01T10:48:10.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:IHhZDaBHYWC8Ww+QEQ9P90PU7PI=', 'Content-Length': '0'} 2024-02-01T10:48:10.993 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c35f83d0da1324a8-0065bb76ea-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:11.036 DEBUG:boto:path=/vclfcn-60/obj8 2024-02-01T10:48:11.036 DEBUG:boto:auth_path=/vclfcn-60/obj8 2024-02-01T10:48:11.036 DEBUG:boto:Method: GET 2024-02-01T10:48:11.037 DEBUG:boto:Path: /vclfcn-60/obj8 2024-02-01T10:48:11.037 DEBUG:boto:Data: 2024-02-01T10:48:11.037 DEBUG:boto:Headers: {} 2024-02-01T10:48:11.037 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:11.037 DEBUG:boto:Port: 8000 2024-02-01T10:48:11.037 DEBUG:boto:Params: {} 2024-02-01T10:48:11.037 DEBUG:boto:Token: None 2024-02-01T10:48:11.037 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:11 GMT /vclfcn-60/obj8 2024-02-01T10:48:11.037 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+LQhEtucrQiOOxb4EMzusgIJPZE= 2024-02-01T10:48:11.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+LQhEtucrQiOOxb4EMzusgIJPZE=', 'Content-Length': '0'} 2024-02-01T10:48:11.040 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b0b0228b3499df9a-0065bb76eb-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:11.084 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:11.084 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:11.084 DEBUG:boto:path=/vclfcn-60/?prefix=obj8 2024-02-01T10:48:11.085 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj8 2024-02-01T10:48:11.085 DEBUG:boto:Method: GET 2024-02-01T10:48:11.085 DEBUG:boto:Path: /vclfcn-60/?prefix=obj8 2024-02-01T10:48:11.085 DEBUG:boto:Data: 2024-02-01T10:48:11.085 DEBUG:boto:Headers: {} 2024-02-01T10:48:11.085 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:11.085 DEBUG:boto:Port: 8000 2024-02-01T10:48:11.085 DEBUG:boto:Params: {} 2024-02-01T10:48:11.085 DEBUG:boto:Token: None 2024-02-01T10:48:11.085 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:11 GMT /vclfcn-60/ 2024-02-01T10:48:11.086 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jgwG3hvw7E8Y2dW3Vt7DKQlNJng= 2024-02-01T10:48:11.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jgwG3hvw7E8Y2dW3Vt7DKQlNJng=', 'Content-Length': '0'} 2024-02-01T10:48:11.090 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c4c2a39a7a04171-0065bb76eb-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:11.091 DEBUG:boto:b'vclfcn-60obj81000falseobj82024-02-01T10:47:30.408Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:11.091 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:11.091 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:11.091 DEBUG:boto:path=/vclfcn-60/?prefix=obj8 2024-02-01T10:48:11.091 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj8 2024-02-01T10:48:11.092 DEBUG:boto:Method: GET 2024-02-01T10:48:11.092 DEBUG:boto:Path: /vclfcn-60/?prefix=obj8 2024-02-01T10:48:11.092 DEBUG:boto:Data: 2024-02-01T10:48:11.092 DEBUG:boto:Headers: {} 2024-02-01T10:48:11.092 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:11.092 DEBUG:boto:Port: 8000 2024-02-01T10:48:11.092 DEBUG:boto:Params: {} 2024-02-01T10:48:11.092 DEBUG:boto:Token: None 2024-02-01T10:48:11.092 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:11 GMT /vclfcn-60/ 2024-02-01T10:48:11.092 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jgwG3hvw7E8Y2dW3Vt7DKQlNJng= 2024-02-01T10:48:11.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jgwG3hvw7E8Y2dW3Vt7DKQlNJng=', 'Content-Length': '0'} 2024-02-01T10:48:11.097 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b607f8023239c021-0065bb76eb-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:11.140 DEBUG:boto:b'vclfcn-60obj81000falseobj82024-02-01T10:47:30.408Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:11.141 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-60 zones={test-zone2, test-zone1} 2024-02-01T10:48:11.141 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-60 zones={test-zone3, test-zone1} 2024-02-01T10:48:11.141 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:11.141 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:11.141 DEBUG:boto:Method: HEAD 2024-02-01T10:48:11.141 DEBUG:boto:Path: /vclfcn-60/ 2024-02-01T10:48:11.141 DEBUG:boto:Data: 2024-02-01T10:48:11.142 DEBUG:boto:Headers: {} 2024-02-01T10:48:11.142 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:11.142 DEBUG:boto:Port: 8001 2024-02-01T10:48:11.142 DEBUG:boto:Params: {} 2024-02-01T10:48:11.142 DEBUG:boto:Token: None 2024-02-01T10:48:11.142 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:11 GMT /vclfcn-60/ 2024-02-01T10:48:11.142 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vEmHKLKl29aj9Gm+VzKQ4omYNy0= 2024-02-01T10:48:11.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vEmHKLKl29aj9Gm+VzKQ4omYNy0=', 'Content-Length': '0'} 2024-02-01T10:48:11.147 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000076e7d5f201807029-0065bb76eb-5762-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:48:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:11.147 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:11.147 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:11.147 DEBUG:boto:Method: HEAD 2024-02-01T10:48:11.147 DEBUG:boto:Path: /vclfcn-60/ 2024-02-01T10:48:11.147 DEBUG:boto:Data: 2024-02-01T10:48:11.147 DEBUG:boto:Headers: {} 2024-02-01T10:48:11.147 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:11.147 DEBUG:boto:Port: 8000 2024-02-01T10:48:11.148 DEBUG:boto:Params: {} 2024-02-01T10:48:11.148 DEBUG:boto:Token: None 2024-02-01T10:48:11.148 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:11 GMT /vclfcn-60/ 2024-02-01T10:48:11.148 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vEmHKLKl29aj9Gm+VzKQ4omYNy0= 2024-02-01T10:48:11.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vEmHKLKl29aj9Gm+VzKQ4omYNy0=', 'Content-Length': '0'} 2024-02-01T10:48:11.152 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c7aecadb2a58119d-0065bb76eb-5765-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:48:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:11.152 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:48:11.152 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:11.152 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:11.152 DEBUG:boto:path=/vclfcn-60/?versions&max-keys=999 2024-02-01T10:48:11.152 DEBUG:boto:auth_path=/vclfcn-60/?versions&max-keys=999 2024-02-01T10:48:11.153 DEBUG:boto:Method: GET 2024-02-01T10:48:11.153 DEBUG:boto:Path: /vclfcn-60/?versions&max-keys=999 2024-02-01T10:48:11.153 DEBUG:boto:Data: 2024-02-01T10:48:11.153 DEBUG:boto:Headers: {} 2024-02-01T10:48:11.153 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:11.153 DEBUG:boto:Port: 8001 2024-02-01T10:48:11.153 DEBUG:boto:Params: {} 2024-02-01T10:48:11.153 DEBUG:boto:Token: None 2024-02-01T10:48:11.153 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:11 GMT /vclfcn-60/?versions 2024-02-01T10:48:11.153 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Aam37cq53n9nWCqY/R0ciI1QWUI= 2024-02-01T10:48:11.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Aam37cq53n9nWCqY/R0ciI1QWUI=', 'Content-Length': '0'} 2024-02-01T10:48:11.158 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec4ea11b8b2f68f9-0065bb76eb-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:11.158 DEBUG:boto:b'vclfcn-60999falseobj1nulltrue2024-02-01T10:46:46.243Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:46:46.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:46:46.285Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:46:46.336Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-02-01T10:47:30.291Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-02-01T10:47:30.343Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-02-01T10:47:30.358Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-02-01T10:47:30.408Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:11.159 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-02-01T10:48:11.159 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-02-01T10:48:11.159 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-02-01T10:48:11.159 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-02-01T10:48:11.159 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-02-01T10:48:11.159 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-02-01T10:48:11.159 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-02-01T10:48:11.159 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-02-01T10:48:11.159 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:48:11.160 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:11.160 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:11.160 DEBUG:boto:path=/vclfcn-60/?versions&max-keys=999 2024-02-01T10:48:11.160 DEBUG:boto:auth_path=/vclfcn-60/?versions&max-keys=999 2024-02-01T10:48:11.160 DEBUG:boto:Method: GET 2024-02-01T10:48:11.160 DEBUG:boto:Path: /vclfcn-60/?versions&max-keys=999 2024-02-01T10:48:11.160 DEBUG:boto:Data: 2024-02-01T10:48:11.160 DEBUG:boto:Headers: {} 2024-02-01T10:48:11.160 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:11.160 DEBUG:boto:Port: 8000 2024-02-01T10:48:11.161 DEBUG:boto:Params: {} 2024-02-01T10:48:11.161 DEBUG:boto:Token: None 2024-02-01T10:48:11.161 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:11 GMT /vclfcn-60/?versions 2024-02-01T10:48:11.161 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Aam37cq53n9nWCqY/R0ciI1QWUI= 2024-02-01T10:48:11.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Aam37cq53n9nWCqY/R0ciI1QWUI=', 'Content-Length': '0'} 2024-02-01T10:48:11.166 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000202e5e39879f4d6d-0065bb76eb-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:11.166 DEBUG:boto:b'vclfcn-60999falseobj1nulltrue2024-02-01T10:46:46.243Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:46:46.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:46:46.285Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:46:46.336Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-02-01T10:47:30.291Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-02-01T10:47:30.343Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-02-01T10:47:30.358Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-02-01T10:47:30.408Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:11.167 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-02-01T10:48:11.167 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-02-01T10:48:11.167 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-02-01T10:48:11.167 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-02-01T10:48:11.167 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-02-01T10:48:11.167 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-02-01T10:48:11.167 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-02-01T10:48:11.168 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-02-01T10:48:11.168 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:11.168 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:11.168 DEBUG:boto:path=/vclfcn-60/?versions&max-keys=999 2024-02-01T10:48:11.168 DEBUG:boto:auth_path=/vclfcn-60/?versions&max-keys=999 2024-02-01T10:48:11.168 DEBUG:boto:Method: GET 2024-02-01T10:48:11.168 DEBUG:boto:Path: /vclfcn-60/?versions&max-keys=999 2024-02-01T10:48:11.168 DEBUG:boto:Data: 2024-02-01T10:48:11.169 DEBUG:boto:Headers: {} 2024-02-01T10:48:11.169 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:11.169 DEBUG:boto:Port: 8001 2024-02-01T10:48:11.169 DEBUG:boto:Params: {} 2024-02-01T10:48:11.169 DEBUG:boto:Token: None 2024-02-01T10:48:11.169 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:11 GMT /vclfcn-60/?versions 2024-02-01T10:48:11.169 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Aam37cq53n9nWCqY/R0ciI1QWUI= 2024-02-01T10:48:11.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Aam37cq53n9nWCqY/R0ciI1QWUI=', 'Content-Length': '0'} 2024-02-01T10:48:11.174 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e4e9388d83610ed-0065bb76eb-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:11.174 DEBUG:boto:b'vclfcn-60999falseobj1nulltrue2024-02-01T10:46:46.243Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:46:46.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:46:46.285Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:46:46.336Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-02-01T10:47:30.291Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-02-01T10:47:30.343Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-02-01T10:47:30.358Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-02-01T10:47:30.408Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:11.175 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:11.175 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:11.175 DEBUG:boto:path=/vclfcn-60/?versions&max-keys=999 2024-02-01T10:48:11.175 DEBUG:boto:auth_path=/vclfcn-60/?versions&max-keys=999 2024-02-01T10:48:11.175 DEBUG:boto:Method: GET 2024-02-01T10:48:11.175 DEBUG:boto:Path: /vclfcn-60/?versions&max-keys=999 2024-02-01T10:48:11.175 DEBUG:boto:Data: 2024-02-01T10:48:11.175 DEBUG:boto:Headers: {} 2024-02-01T10:48:11.175 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:11.175 DEBUG:boto:Port: 8000 2024-02-01T10:48:11.176 DEBUG:boto:Params: {} 2024-02-01T10:48:11.176 DEBUG:boto:Token: None 2024-02-01T10:48:11.176 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:11 GMT /vclfcn-60/?versions 2024-02-01T10:48:11.176 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Aam37cq53n9nWCqY/R0ciI1QWUI= 2024-02-01T10:48:11.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Aam37cq53n9nWCqY/R0ciI1QWUI=', 'Content-Length': '0'} 2024-02-01T10:48:11.183 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8ce7628122870e8-0065bb76eb-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:11.183 DEBUG:boto:b'vclfcn-60999falseobj1nulltrue2024-02-01T10:46:46.243Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:46:46.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:46:46.285Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:46:46.336Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-02-01T10:47:30.291Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-02-01T10:47:30.343Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-02-01T10:47:30.358Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-02-01T10:47:30.408Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:11.184 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:48:11.184 DEBUG:boto:path=/vclfcn-60/obj1 2024-02-01T10:48:11.184 DEBUG:boto:auth_path=/vclfcn-60/obj1 2024-02-01T10:48:11.184 DEBUG:boto:path=/vclfcn-60/obj1?versionId=null 2024-02-01T10:48:11.184 DEBUG:boto:auth_path=/vclfcn-60/obj1?versionId=null 2024-02-01T10:48:11.184 DEBUG:boto:Method: GET 2024-02-01T10:48:11.184 DEBUG:boto:Path: /vclfcn-60/obj1?versionId=null 2024-02-01T10:48:11.184 DEBUG:boto:Data: 2024-02-01T10:48:11.184 DEBUG:boto:Headers: {} 2024-02-01T10:48:11.185 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:11.185 DEBUG:boto:Port: 8001 2024-02-01T10:48:11.185 DEBUG:boto:Params: {} 2024-02-01T10:48:11.185 DEBUG:boto:Token: None 2024-02-01T10:48:11.185 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:11 GMT /vclfcn-60/obj1?versionId=null 2024-02-01T10:48:11.185 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:k2HCkd46Ouuq1dmBDa8ewD505rA= 2024-02-01T10:48:11.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:k2HCkd46Ouuq1dmBDa8ewD505rA=', 'Content-Length': '0'} 2024-02-01T10:48:11.188 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001dc62da804a68ac0-0065bb76eb-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:11.232 DEBUG:boto:path=/vclfcn-60/obj1 2024-02-01T10:48:11.232 DEBUG:boto:auth_path=/vclfcn-60/obj1 2024-02-01T10:48:11.232 DEBUG:boto:path=/vclfcn-60/obj1?versionId=null 2024-02-01T10:48:11.232 DEBUG:boto:auth_path=/vclfcn-60/obj1?versionId=null 2024-02-01T10:48:11.233 DEBUG:boto:Method: GET 2024-02-01T10:48:11.233 DEBUG:boto:Path: /vclfcn-60/obj1?versionId=null 2024-02-01T10:48:11.233 DEBUG:boto:Data: 2024-02-01T10:48:11.233 DEBUG:boto:Headers: {} 2024-02-01T10:48:11.233 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:11.233 DEBUG:boto:Port: 8000 2024-02-01T10:48:11.233 DEBUG:boto:Params: {} 2024-02-01T10:48:11.233 DEBUG:boto:Token: None 2024-02-01T10:48:11.233 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:11 GMT /vclfcn-60/obj1?versionId=null 2024-02-01T10:48:11.234 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:k2HCkd46Ouuq1dmBDa8ewD505rA= 2024-02-01T10:48:11.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:k2HCkd46Ouuq1dmBDa8ewD505rA=', 'Content-Length': '0'} 2024-02-01T10:48:11.237 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c758e9e91f693551-0065bb76eb-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:11.238 DEBUG:boto:path=/vclfcn-60/obj1 2024-02-01T10:48:11.238 DEBUG:boto:auth_path=/vclfcn-60/obj1 2024-02-01T10:48:11.238 DEBUG:boto:path=/vclfcn-60/obj1?versionId=null 2024-02-01T10:48:11.238 DEBUG:boto:auth_path=/vclfcn-60/obj1?versionId=null 2024-02-01T10:48:11.238 DEBUG:boto:Method: HEAD 2024-02-01T10:48:11.238 DEBUG:boto:Path: /vclfcn-60/obj1?versionId=null 2024-02-01T10:48:11.238 DEBUG:boto:Data: 2024-02-01T10:48:11.238 DEBUG:boto:Headers: {} 2024-02-01T10:48:11.239 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:11.239 DEBUG:boto:Port: 8001 2024-02-01T10:48:11.239 DEBUG:boto:Params: {} 2024-02-01T10:48:11.240 DEBUG:boto:Token: None 2024-02-01T10:48:11.240 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:11 GMT /vclfcn-60/obj1?versionId=null 2024-02-01T10:48:11.240 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:59eFBQgpSqGtdcigrriSJI+kpsw= 2024-02-01T10:48:11.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:59eFBQgpSqGtdcigrriSJI+kpsw=', 'Content-Length': '0'} 2024-02-01T10:48:11.243 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008491550c308a0cfd-0065bb76eb-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:11.243 DEBUG:boto:path=/vclfcn-60/obj1 2024-02-01T10:48:11.243 DEBUG:boto:auth_path=/vclfcn-60/obj1 2024-02-01T10:48:11.244 DEBUG:boto:path=/vclfcn-60/obj1?versionId=null 2024-02-01T10:48:11.244 DEBUG:boto:auth_path=/vclfcn-60/obj1?versionId=null 2024-02-01T10:48:11.244 DEBUG:boto:Method: HEAD 2024-02-01T10:48:11.244 DEBUG:boto:Path: /vclfcn-60/obj1?versionId=null 2024-02-01T10:48:11.244 DEBUG:boto:Data: 2024-02-01T10:48:11.244 DEBUG:boto:Headers: {} 2024-02-01T10:48:11.244 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:11.244 DEBUG:boto:Port: 8000 2024-02-01T10:48:11.244 DEBUG:boto:Params: {} 2024-02-01T10:48:11.245 DEBUG:boto:Token: None 2024-02-01T10:48:11.245 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:11 GMT /vclfcn-60/obj1?versionId=null 2024-02-01T10:48:11.245 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:59eFBQgpSqGtdcigrriSJI+kpsw= 2024-02-01T10:48:11.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:59eFBQgpSqGtdcigrriSJI+kpsw=', 'Content-Length': '0'} 2024-02-01T10:48:11.248 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008891a4e7817e790b-0065bb76eb-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:11.248 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:48:11.248 DEBUG:boto:path=/vclfcn-60/obj1 2024-02-01T10:48:11.248 DEBUG:boto:auth_path=/vclfcn-60/obj1 2024-02-01T10:48:11.248 DEBUG:boto:path=/vclfcn-60/obj1?versionId=null 2024-02-01T10:48:11.248 DEBUG:boto:auth_path=/vclfcn-60/obj1?versionId=null 2024-02-01T10:48:11.248 DEBUG:boto:Method: GET 2024-02-01T10:48:11.249 DEBUG:boto:Path: /vclfcn-60/obj1?versionId=null 2024-02-01T10:48:11.249 DEBUG:boto:Data: 2024-02-01T10:48:11.249 DEBUG:boto:Headers: {} 2024-02-01T10:48:11.249 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:11.249 DEBUG:boto:Port: 8001 2024-02-01T10:48:11.249 DEBUG:boto:Params: {} 2024-02-01T10:48:11.249 DEBUG:boto:Token: None 2024-02-01T10:48:11.249 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:11 GMT /vclfcn-60/obj1?versionId=null 2024-02-01T10:48:11.250 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:k2HCkd46Ouuq1dmBDa8ewD505rA= 2024-02-01T10:48:11.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:k2HCkd46Ouuq1dmBDa8ewD505rA=', 'Content-Length': '0'} 2024-02-01T10:48:11.252 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008aafd4e690c7d7ed-0065bb76eb-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:11.300 DEBUG:boto:path=/vclfcn-60/obj1 2024-02-01T10:48:11.301 DEBUG:boto:auth_path=/vclfcn-60/obj1 2024-02-01T10:48:11.301 DEBUG:boto:path=/vclfcn-60/obj1?versionId=null 2024-02-01T10:48:11.301 DEBUG:boto:auth_path=/vclfcn-60/obj1?versionId=null 2024-02-01T10:48:11.301 DEBUG:boto:Method: GET 2024-02-01T10:48:11.301 DEBUG:boto:Path: /vclfcn-60/obj1?versionId=null 2024-02-01T10:48:11.301 DEBUG:boto:Data: 2024-02-01T10:48:11.301 DEBUG:boto:Headers: {} 2024-02-01T10:48:11.301 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:11.301 DEBUG:boto:Port: 8000 2024-02-01T10:48:11.302 DEBUG:boto:Params: {} 2024-02-01T10:48:11.302 DEBUG:boto:Token: None 2024-02-01T10:48:11.302 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:11 GMT /vclfcn-60/obj1?versionId=null 2024-02-01T10:48:11.302 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:k2HCkd46Ouuq1dmBDa8ewD505rA= 2024-02-01T10:48:11.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:k2HCkd46Ouuq1dmBDa8ewD505rA=', 'Content-Length': '0'} 2024-02-01T10:48:11.305 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bcb7606fe35dff23-0065bb76eb-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:11.352 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:11.353 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:11.353 DEBUG:boto:path=/vclfcn-60/?prefix=obj1 2024-02-01T10:48:11.353 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj1 2024-02-01T10:48:11.353 DEBUG:boto:Method: GET 2024-02-01T10:48:11.353 DEBUG:boto:Path: /vclfcn-60/?prefix=obj1 2024-02-01T10:48:11.353 DEBUG:boto:Data: 2024-02-01T10:48:11.353 DEBUG:boto:Headers: {} 2024-02-01T10:48:11.353 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:11.353 DEBUG:boto:Port: 8001 2024-02-01T10:48:11.354 DEBUG:boto:Params: {} 2024-02-01T10:48:11.354 DEBUG:boto:Token: None 2024-02-01T10:48:11.354 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:11 GMT /vclfcn-60/ 2024-02-01T10:48:11.354 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jgwG3hvw7E8Y2dW3Vt7DKQlNJng= 2024-02-01T10:48:11.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jgwG3hvw7E8Y2dW3Vt7DKQlNJng=', 'Content-Length': '0'} 2024-02-01T10:48:11.359 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e305f6712cd8caaa-0065bb76eb-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:11.359 DEBUG:boto:b'vclfcn-60obj11000falseobj12024-02-01T10:46:46.243Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:11.360 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:11.360 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:11.360 DEBUG:boto:path=/vclfcn-60/?prefix=obj1 2024-02-01T10:48:11.360 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj1 2024-02-01T10:48:11.360 DEBUG:boto:Method: GET 2024-02-01T10:48:11.361 DEBUG:boto:Path: /vclfcn-60/?prefix=obj1 2024-02-01T10:48:11.361 DEBUG:boto:Data: 2024-02-01T10:48:11.361 DEBUG:boto:Headers: {} 2024-02-01T10:48:11.361 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:11.361 DEBUG:boto:Port: 8000 2024-02-01T10:48:11.361 DEBUG:boto:Params: {} 2024-02-01T10:48:11.361 DEBUG:boto:Token: None 2024-02-01T10:48:11.361 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:11 GMT /vclfcn-60/ 2024-02-01T10:48:11.361 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jgwG3hvw7E8Y2dW3Vt7DKQlNJng= 2024-02-01T10:48:11.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jgwG3hvw7E8Y2dW3Vt7DKQlNJng=', 'Content-Length': '0'} 2024-02-01T10:48:11.366 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0c1f7147983fdfc-0065bb76eb-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:11.366 DEBUG:boto:b'vclfcn-60obj11000falseobj12024-02-01T10:46:46.243Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:11.367 DEBUG:boto:path=/vclfcn-60/obj1 2024-02-01T10:48:11.367 DEBUG:boto:auth_path=/vclfcn-60/obj1 2024-02-01T10:48:11.367 DEBUG:boto:Method: HEAD 2024-02-01T10:48:11.367 DEBUG:boto:Path: /vclfcn-60/obj1 2024-02-01T10:48:11.367 DEBUG:boto:Data: 2024-02-01T10:48:11.367 DEBUG:boto:Headers: {} 2024-02-01T10:48:11.367 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:11.367 DEBUG:boto:Port: 8001 2024-02-01T10:48:11.367 DEBUG:boto:Params: {} 2024-02-01T10:48:11.368 DEBUG:boto:Token: None 2024-02-01T10:48:11.368 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:11 GMT /vclfcn-60/obj1 2024-02-01T10:48:11.368 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:uvEOvhAPpkEPrcSYuJXOmuOotbs= 2024-02-01T10:48:11.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:uvEOvhAPpkEPrcSYuJXOmuOotbs=', 'Content-Length': '0'} 2024-02-01T10:48:11.371 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000184c598617e1cd4e-0065bb76eb-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:11.371 DEBUG:boto:path=/vclfcn-60/obj1 2024-02-01T10:48:11.371 DEBUG:boto:auth_path=/vclfcn-60/obj1 2024-02-01T10:48:11.371 DEBUG:boto:Method: HEAD 2024-02-01T10:48:11.371 DEBUG:boto:Path: /vclfcn-60/obj1 2024-02-01T10:48:11.372 DEBUG:boto:Data: 2024-02-01T10:48:11.372 DEBUG:boto:Headers: {} 2024-02-01T10:48:11.372 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:11.372 DEBUG:boto:Port: 8000 2024-02-01T10:48:11.372 DEBUG:boto:Params: {} 2024-02-01T10:48:11.372 DEBUG:boto:Token: None 2024-02-01T10:48:11.372 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:11 GMT /vclfcn-60/obj1 2024-02-01T10:48:11.372 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:uvEOvhAPpkEPrcSYuJXOmuOotbs= 2024-02-01T10:48:11.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:uvEOvhAPpkEPrcSYuJXOmuOotbs=', 'Content-Length': '0'} 2024-02-01T10:48:11.375 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cde24351cbddcf60-0065bb76eb-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:11.375 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:48:11.376 DEBUG:boto:path=/vclfcn-60/obj1 2024-02-01T10:48:11.376 DEBUG:boto:auth_path=/vclfcn-60/obj1 2024-02-01T10:48:11.376 DEBUG:boto:Method: GET 2024-02-01T10:48:11.376 DEBUG:boto:Path: /vclfcn-60/obj1 2024-02-01T10:48:11.376 DEBUG:boto:Data: 2024-02-01T10:48:11.376 DEBUG:boto:Headers: {} 2024-02-01T10:48:11.376 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:11.376 DEBUG:boto:Port: 8001 2024-02-01T10:48:11.377 DEBUG:boto:Params: {} 2024-02-01T10:48:11.377 DEBUG:boto:Token: None 2024-02-01T10:48:11.377 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:11 GMT /vclfcn-60/obj1 2024-02-01T10:48:11.377 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:PWR4MLoIMCpKmV51nBRiSgQyLpc= 2024-02-01T10:48:11.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:PWR4MLoIMCpKmV51nBRiSgQyLpc=', 'Content-Length': '0'} 2024-02-01T10:48:11.379 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000028d429335e69f090-0065bb76eb-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:11.420 DEBUG:boto:path=/vclfcn-60/obj1 2024-02-01T10:48:11.420 DEBUG:boto:auth_path=/vclfcn-60/obj1 2024-02-01T10:48:11.420 DEBUG:boto:Method: GET 2024-02-01T10:48:11.420 DEBUG:boto:Path: /vclfcn-60/obj1 2024-02-01T10:48:11.421 DEBUG:boto:Data: 2024-02-01T10:48:11.421 DEBUG:boto:Headers: {} 2024-02-01T10:48:11.421 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:11.421 DEBUG:boto:Port: 8000 2024-02-01T10:48:11.421 DEBUG:boto:Params: {} 2024-02-01T10:48:11.421 DEBUG:boto:Token: None 2024-02-01T10:48:11.421 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:11 GMT /vclfcn-60/obj1 2024-02-01T10:48:11.421 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:PWR4MLoIMCpKmV51nBRiSgQyLpc= 2024-02-01T10:48:11.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:PWR4MLoIMCpKmV51nBRiSgQyLpc=', 'Content-Length': '0'} 2024-02-01T10:48:11.425 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008d7b7c149632b817-0065bb76eb-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:11.468 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:11.468 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:11.468 DEBUG:boto:path=/vclfcn-60/?prefix=obj1 2024-02-01T10:48:11.468 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj1 2024-02-01T10:48:11.469 DEBUG:boto:Method: GET 2024-02-01T10:48:11.469 DEBUG:boto:Path: /vclfcn-60/?prefix=obj1 2024-02-01T10:48:11.469 DEBUG:boto:Data: 2024-02-01T10:48:11.469 DEBUG:boto:Headers: {} 2024-02-01T10:48:11.469 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:11.469 DEBUG:boto:Port: 8001 2024-02-01T10:48:11.469 DEBUG:boto:Params: {} 2024-02-01T10:48:11.469 DEBUG:boto:Token: None 2024-02-01T10:48:11.469 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:11 GMT /vclfcn-60/ 2024-02-01T10:48:11.469 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jgwG3hvw7E8Y2dW3Vt7DKQlNJng= 2024-02-01T10:48:11.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jgwG3hvw7E8Y2dW3Vt7DKQlNJng=', 'Content-Length': '0'} 2024-02-01T10:48:11.474 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c1338cb33a054d6-0065bb76eb-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:11.474 DEBUG:boto:b'vclfcn-60obj11000falseobj12024-02-01T10:46:46.243Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:11.475 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:11.475 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:11.475 DEBUG:boto:path=/vclfcn-60/?prefix=obj1 2024-02-01T10:48:11.475 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj1 2024-02-01T10:48:11.475 DEBUG:boto:Method: GET 2024-02-01T10:48:11.475 DEBUG:boto:Path: /vclfcn-60/?prefix=obj1 2024-02-01T10:48:11.475 DEBUG:boto:Data: 2024-02-01T10:48:11.476 DEBUG:boto:Headers: {} 2024-02-01T10:48:11.476 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:11.476 DEBUG:boto:Port: 8000 2024-02-01T10:48:11.476 DEBUG:boto:Params: {} 2024-02-01T10:48:11.476 DEBUG:boto:Token: None 2024-02-01T10:48:11.476 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:11 GMT /vclfcn-60/ 2024-02-01T10:48:11.476 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jgwG3hvw7E8Y2dW3Vt7DKQlNJng= 2024-02-01T10:48:11.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jgwG3hvw7E8Y2dW3Vt7DKQlNJng=', 'Content-Length': '0'} 2024-02-01T10:48:11.481 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f18f2beb16dfbcd7-0065bb76eb-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:11.481 DEBUG:boto:b'vclfcn-60obj11000falseobj12024-02-01T10:46:46.243Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:11.481 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:48:11.482 DEBUG:boto:path=/vclfcn-60/obj2 2024-02-01T10:48:11.482 DEBUG:boto:auth_path=/vclfcn-60/obj2 2024-02-01T10:48:11.482 DEBUG:boto:path=/vclfcn-60/obj2?versionId=null 2024-02-01T10:48:11.482 DEBUG:boto:auth_path=/vclfcn-60/obj2?versionId=null 2024-02-01T10:48:11.482 DEBUG:boto:Method: GET 2024-02-01T10:48:11.482 DEBUG:boto:Path: /vclfcn-60/obj2?versionId=null 2024-02-01T10:48:11.482 DEBUG:boto:Data: 2024-02-01T10:48:11.482 DEBUG:boto:Headers: {} 2024-02-01T10:48:11.483 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:11.483 DEBUG:boto:Port: 8001 2024-02-01T10:48:11.483 DEBUG:boto:Params: {} 2024-02-01T10:48:11.483 DEBUG:boto:Token: None 2024-02-01T10:48:11.483 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:11 GMT /vclfcn-60/obj2?versionId=null 2024-02-01T10:48:11.483 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:TYToYJ0Bt79qrv+neWtJgoZEw4Y= 2024-02-01T10:48:11.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:TYToYJ0Bt79qrv+neWtJgoZEw4Y=', 'Content-Length': '0'} 2024-02-01T10:48:11.486 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000772c0c5e7d11b14c-0065bb76eb-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:11.486 DEBUG:boto:path=/vclfcn-60/obj2 2024-02-01T10:48:11.487 DEBUG:boto:auth_path=/vclfcn-60/obj2 2024-02-01T10:48:11.487 DEBUG:boto:path=/vclfcn-60/obj2?versionId=null 2024-02-01T10:48:11.487 DEBUG:boto:auth_path=/vclfcn-60/obj2?versionId=null 2024-02-01T10:48:11.487 DEBUG:boto:Method: GET 2024-02-01T10:48:11.487 DEBUG:boto:Path: /vclfcn-60/obj2?versionId=null 2024-02-01T10:48:11.487 DEBUG:boto:Data: 2024-02-01T10:48:11.487 DEBUG:boto:Headers: {} 2024-02-01T10:48:11.487 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:11.487 DEBUG:boto:Port: 8000 2024-02-01T10:48:11.487 DEBUG:boto:Params: {} 2024-02-01T10:48:11.488 DEBUG:boto:Token: None 2024-02-01T10:48:11.488 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:11 GMT /vclfcn-60/obj2?versionId=null 2024-02-01T10:48:11.488 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:TYToYJ0Bt79qrv+neWtJgoZEw4Y= 2024-02-01T10:48:11.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:TYToYJ0Bt79qrv+neWtJgoZEw4Y=', 'Content-Length': '0'} 2024-02-01T10:48:11.491 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e3541e9541fad162-0065bb76eb-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:11.491 DEBUG:boto:path=/vclfcn-60/obj2 2024-02-01T10:48:11.491 DEBUG:boto:auth_path=/vclfcn-60/obj2 2024-02-01T10:48:11.491 DEBUG:boto:path=/vclfcn-60/obj2?versionId=null 2024-02-01T10:48:11.491 DEBUG:boto:auth_path=/vclfcn-60/obj2?versionId=null 2024-02-01T10:48:11.492 DEBUG:boto:Method: HEAD 2024-02-01T10:48:11.492 DEBUG:boto:Path: /vclfcn-60/obj2?versionId=null 2024-02-01T10:48:11.492 DEBUG:boto:Data: 2024-02-01T10:48:11.492 DEBUG:boto:Headers: {} 2024-02-01T10:48:11.492 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:11.492 DEBUG:boto:Port: 8001 2024-02-01T10:48:11.492 DEBUG:boto:Params: {} 2024-02-01T10:48:11.492 DEBUG:boto:Token: None 2024-02-01T10:48:11.492 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:11 GMT /vclfcn-60/obj2?versionId=null 2024-02-01T10:48:11.493 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:CDtYuAFrOrrMIuLRXhVE5twD3oU= 2024-02-01T10:48:11.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:CDtYuAFrOrrMIuLRXhVE5twD3oU=', 'Content-Length': '0'} 2024-02-01T10:48:11.496 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006b16d22322225d34-0065bb76eb-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:11.496 DEBUG:boto:path=/vclfcn-60/obj2 2024-02-01T10:48:11.496 DEBUG:boto:auth_path=/vclfcn-60/obj2 2024-02-01T10:48:11.496 DEBUG:boto:path=/vclfcn-60/obj2?versionId=null 2024-02-01T10:48:11.496 DEBUG:boto:auth_path=/vclfcn-60/obj2?versionId=null 2024-02-01T10:48:11.496 DEBUG:boto:Method: HEAD 2024-02-01T10:48:11.496 DEBUG:boto:Path: /vclfcn-60/obj2?versionId=null 2024-02-01T10:48:11.497 DEBUG:boto:Data: 2024-02-01T10:48:11.497 DEBUG:boto:Headers: {} 2024-02-01T10:48:11.497 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:11.497 DEBUG:boto:Port: 8000 2024-02-01T10:48:11.497 DEBUG:boto:Params: {} 2024-02-01T10:48:11.497 DEBUG:boto:Token: None 2024-02-01T10:48:11.497 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:11 GMT /vclfcn-60/obj2?versionId=null 2024-02-01T10:48:11.497 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:CDtYuAFrOrrMIuLRXhVE5twD3oU= 2024-02-01T10:48:11.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:CDtYuAFrOrrMIuLRXhVE5twD3oU=', 'Content-Length': '0'} 2024-02-01T10:48:11.500 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000899fe2f5195d5287-0065bb76eb-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:11.501 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:48:11.501 DEBUG:boto:path=/vclfcn-60/obj2 2024-02-01T10:48:11.501 DEBUG:boto:auth_path=/vclfcn-60/obj2 2024-02-01T10:48:11.501 DEBUG:boto:path=/vclfcn-60/obj2?versionId=null 2024-02-01T10:48:11.501 DEBUG:boto:auth_path=/vclfcn-60/obj2?versionId=null 2024-02-01T10:48:11.501 DEBUG:boto:Method: GET 2024-02-01T10:48:11.501 DEBUG:boto:Path: /vclfcn-60/obj2?versionId=null 2024-02-01T10:48:11.501 DEBUG:boto:Data: 2024-02-01T10:48:11.501 DEBUG:boto:Headers: {} 2024-02-01T10:48:11.501 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:11.502 DEBUG:boto:Port: 8001 2024-02-01T10:48:11.502 DEBUG:boto:Params: {} 2024-02-01T10:48:11.502 DEBUG:boto:Token: None 2024-02-01T10:48:11.502 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:11 GMT /vclfcn-60/obj2?versionId=null 2024-02-01T10:48:11.502 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:TYToYJ0Bt79qrv+neWtJgoZEw4Y= 2024-02-01T10:48:11.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:TYToYJ0Bt79qrv+neWtJgoZEw4Y=', 'Content-Length': '0'} 2024-02-01T10:48:11.505 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000034505c3f3e6fe1d0-0065bb76eb-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:11.548 DEBUG:boto:path=/vclfcn-60/obj2 2024-02-01T10:48:11.548 DEBUG:boto:auth_path=/vclfcn-60/obj2 2024-02-01T10:48:11.549 DEBUG:boto:path=/vclfcn-60/obj2?versionId=null 2024-02-01T10:48:11.549 DEBUG:boto:auth_path=/vclfcn-60/obj2?versionId=null 2024-02-01T10:48:11.549 DEBUG:boto:Method: GET 2024-02-01T10:48:11.549 DEBUG:boto:Path: /vclfcn-60/obj2?versionId=null 2024-02-01T10:48:11.550 DEBUG:boto:Data: 2024-02-01T10:48:11.550 DEBUG:boto:Headers: {} 2024-02-01T10:48:11.550 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:11.550 DEBUG:boto:Port: 8000 2024-02-01T10:48:11.550 DEBUG:boto:Params: {} 2024-02-01T10:48:11.551 DEBUG:boto:Token: None 2024-02-01T10:48:11.551 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:11 GMT /vclfcn-60/obj2?versionId=null 2024-02-01T10:48:11.551 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:TYToYJ0Bt79qrv+neWtJgoZEw4Y= 2024-02-01T10:48:11.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:TYToYJ0Bt79qrv+neWtJgoZEw4Y=', 'Content-Length': '0'} 2024-02-01T10:48:11.555 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007edeb36c22322d3e-0065bb76eb-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:11.600 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:11.600 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:11.600 DEBUG:boto:path=/vclfcn-60/?prefix=obj2 2024-02-01T10:48:11.600 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj2 2024-02-01T10:48:11.600 DEBUG:boto:Method: GET 2024-02-01T10:48:11.600 DEBUG:boto:Path: /vclfcn-60/?prefix=obj2 2024-02-01T10:48:11.601 DEBUG:boto:Data: 2024-02-01T10:48:11.601 DEBUG:boto:Headers: {} 2024-02-01T10:48:11.601 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:11.601 DEBUG:boto:Port: 8001 2024-02-01T10:48:11.601 DEBUG:boto:Params: {} 2024-02-01T10:48:11.601 DEBUG:boto:Token: None 2024-02-01T10:48:11.601 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:11 GMT /vclfcn-60/ 2024-02-01T10:48:11.601 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jgwG3hvw7E8Y2dW3Vt7DKQlNJng= 2024-02-01T10:48:11.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jgwG3hvw7E8Y2dW3Vt7DKQlNJng=', 'Content-Length': '0'} 2024-02-01T10:48:11.606 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff7fb8eb368fcec9-0065bb76eb-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:11.652 DEBUG:boto:b'vclfcn-60obj21000falseobj22024-02-01T10:46:46.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:11.653 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:11.653 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:11.653 DEBUG:boto:path=/vclfcn-60/?prefix=obj2 2024-02-01T10:48:11.653 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj2 2024-02-01T10:48:11.653 DEBUG:boto:Method: GET 2024-02-01T10:48:11.653 DEBUG:boto:Path: /vclfcn-60/?prefix=obj2 2024-02-01T10:48:11.653 DEBUG:boto:Data: 2024-02-01T10:48:11.653 DEBUG:boto:Headers: {} 2024-02-01T10:48:11.654 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:11.654 DEBUG:boto:Port: 8000 2024-02-01T10:48:11.654 DEBUG:boto:Params: {} 2024-02-01T10:48:11.654 DEBUG:boto:Token: None 2024-02-01T10:48:11.654 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:11 GMT /vclfcn-60/ 2024-02-01T10:48:11.654 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jgwG3hvw7E8Y2dW3Vt7DKQlNJng= 2024-02-01T10:48:11.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jgwG3hvw7E8Y2dW3Vt7DKQlNJng=', 'Content-Length': '0'} 2024-02-01T10:48:11.659 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045d89c098877457c-0065bb76eb-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:11.700 DEBUG:boto:b'vclfcn-60obj21000falseobj22024-02-01T10:46:46.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:11.700 DEBUG:boto:path=/vclfcn-60/obj2 2024-02-01T10:48:11.701 DEBUG:boto:auth_path=/vclfcn-60/obj2 2024-02-01T10:48:11.701 DEBUG:boto:Method: HEAD 2024-02-01T10:48:11.701 DEBUG:boto:Path: /vclfcn-60/obj2 2024-02-01T10:48:11.701 DEBUG:boto:Data: 2024-02-01T10:48:11.701 DEBUG:boto:Headers: {} 2024-02-01T10:48:11.701 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:11.701 DEBUG:boto:Port: 8001 2024-02-01T10:48:11.701 DEBUG:boto:Params: {} 2024-02-01T10:48:11.702 DEBUG:boto:Token: None 2024-02-01T10:48:11.702 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:11 GMT /vclfcn-60/obj2 2024-02-01T10:48:11.702 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tbMeVpA6on8Ljk2jvk8QISQ4vD4= 2024-02-01T10:48:11.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tbMeVpA6on8Ljk2jvk8QISQ4vD4=', 'Content-Length': '0'} 2024-02-01T10:48:11.705 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c066485c749b546e-0065bb76eb-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:11.705 DEBUG:boto:path=/vclfcn-60/obj2 2024-02-01T10:48:11.705 DEBUG:boto:auth_path=/vclfcn-60/obj2 2024-02-01T10:48:11.706 DEBUG:boto:Method: HEAD 2024-02-01T10:48:11.706 DEBUG:boto:Path: /vclfcn-60/obj2 2024-02-01T10:48:11.706 DEBUG:boto:Data: 2024-02-01T10:48:11.706 DEBUG:boto:Headers: {} 2024-02-01T10:48:11.706 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:11.706 DEBUG:boto:Port: 8000 2024-02-01T10:48:11.706 DEBUG:boto:Params: {} 2024-02-01T10:48:11.706 DEBUG:boto:Token: None 2024-02-01T10:48:11.706 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:11 GMT /vclfcn-60/obj2 2024-02-01T10:48:11.706 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tbMeVpA6on8Ljk2jvk8QISQ4vD4= 2024-02-01T10:48:11.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tbMeVpA6on8Ljk2jvk8QISQ4vD4=', 'Content-Length': '0'} 2024-02-01T10:48:11.709 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004524a9f5bf930efb-0065bb76eb-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:11.709 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:48:11.710 DEBUG:boto:path=/vclfcn-60/obj2 2024-02-01T10:48:11.710 DEBUG:boto:auth_path=/vclfcn-60/obj2 2024-02-01T10:48:11.710 DEBUG:boto:Method: GET 2024-02-01T10:48:11.710 DEBUG:boto:Path: /vclfcn-60/obj2 2024-02-01T10:48:11.710 DEBUG:boto:Data: 2024-02-01T10:48:11.710 DEBUG:boto:Headers: {} 2024-02-01T10:48:11.710 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:11.710 DEBUG:boto:Port: 8001 2024-02-01T10:48:11.710 DEBUG:boto:Params: {} 2024-02-01T10:48:11.710 DEBUG:boto:Token: None 2024-02-01T10:48:11.711 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:11 GMT /vclfcn-60/obj2 2024-02-01T10:48:11.711 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:p8hkXueEWe2GEI/LByv5mHJYiv0= 2024-02-01T10:48:11.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:p8hkXueEWe2GEI/LByv5mHJYiv0=', 'Content-Length': '0'} 2024-02-01T10:48:11.714 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007c051b68f6826ece-0065bb76eb-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:11.756 DEBUG:boto:path=/vclfcn-60/obj2 2024-02-01T10:48:11.756 DEBUG:boto:auth_path=/vclfcn-60/obj2 2024-02-01T10:48:11.757 DEBUG:boto:Method: GET 2024-02-01T10:48:11.757 DEBUG:boto:Path: /vclfcn-60/obj2 2024-02-01T10:48:11.757 DEBUG:boto:Data: 2024-02-01T10:48:11.757 DEBUG:boto:Headers: {} 2024-02-01T10:48:11.757 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:11.758 DEBUG:boto:Port: 8000 2024-02-01T10:48:11.758 DEBUG:boto:Params: {} 2024-02-01T10:48:11.758 DEBUG:boto:Token: None 2024-02-01T10:48:11.758 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:11 GMT /vclfcn-60/obj2 2024-02-01T10:48:11.759 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:p8hkXueEWe2GEI/LByv5mHJYiv0= 2024-02-01T10:48:11.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:p8hkXueEWe2GEI/LByv5mHJYiv0=', 'Content-Length': '0'} 2024-02-01T10:48:11.762 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d6c4659b94c75e3b-0065bb76eb-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:11.804 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:11.804 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:11.804 DEBUG:boto:path=/vclfcn-60/?prefix=obj2 2024-02-01T10:48:11.804 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj2 2024-02-01T10:48:11.805 DEBUG:boto:Method: GET 2024-02-01T10:48:11.805 DEBUG:boto:Path: /vclfcn-60/?prefix=obj2 2024-02-01T10:48:11.805 DEBUG:boto:Data: 2024-02-01T10:48:11.805 DEBUG:boto:Headers: {} 2024-02-01T10:48:11.805 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:11.805 DEBUG:boto:Port: 8001 2024-02-01T10:48:11.805 DEBUG:boto:Params: {} 2024-02-01T10:48:11.805 DEBUG:boto:Token: None 2024-02-01T10:48:11.806 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:11 GMT /vclfcn-60/ 2024-02-01T10:48:11.806 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jgwG3hvw7E8Y2dW3Vt7DKQlNJng= 2024-02-01T10:48:11.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jgwG3hvw7E8Y2dW3Vt7DKQlNJng=', 'Content-Length': '0'} 2024-02-01T10:48:11.810 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f5f8fb46029142ac-0065bb76eb-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:11.852 DEBUG:boto:b'vclfcn-60obj21000falseobj22024-02-01T10:46:46.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:11.853 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:11.853 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:11.853 DEBUG:boto:path=/vclfcn-60/?prefix=obj2 2024-02-01T10:48:11.853 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj2 2024-02-01T10:48:11.853 DEBUG:boto:Method: GET 2024-02-01T10:48:11.853 DEBUG:boto:Path: /vclfcn-60/?prefix=obj2 2024-02-01T10:48:11.853 DEBUG:boto:Data: 2024-02-01T10:48:11.853 DEBUG:boto:Headers: {} 2024-02-01T10:48:11.853 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:11.854 DEBUG:boto:Port: 8000 2024-02-01T10:48:11.854 DEBUG:boto:Params: {} 2024-02-01T10:48:11.854 DEBUG:boto:Token: None 2024-02-01T10:48:11.854 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:11 GMT /vclfcn-60/ 2024-02-01T10:48:11.854 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jgwG3hvw7E8Y2dW3Vt7DKQlNJng= 2024-02-01T10:48:11.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jgwG3hvw7E8Y2dW3Vt7DKQlNJng=', 'Content-Length': '0'} 2024-02-01T10:48:11.859 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bcb6a45482e58caf-0065bb76eb-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:11.860 DEBUG:boto:b'vclfcn-60obj21000falseobj22024-02-01T10:46:46.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:11.860 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:48:11.860 DEBUG:boto:path=/vclfcn-60/obj3 2024-02-01T10:48:11.860 DEBUG:boto:auth_path=/vclfcn-60/obj3 2024-02-01T10:48:11.860 DEBUG:boto:path=/vclfcn-60/obj3?versionId=null 2024-02-01T10:48:11.860 DEBUG:boto:auth_path=/vclfcn-60/obj3?versionId=null 2024-02-01T10:48:11.860 DEBUG:boto:Method: GET 2024-02-01T10:48:11.861 DEBUG:boto:Path: /vclfcn-60/obj3?versionId=null 2024-02-01T10:48:11.861 DEBUG:boto:Data: 2024-02-01T10:48:11.861 DEBUG:boto:Headers: {} 2024-02-01T10:48:11.861 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:11.861 DEBUG:boto:Port: 8001 2024-02-01T10:48:11.861 DEBUG:boto:Params: {} 2024-02-01T10:48:11.861 DEBUG:boto:Token: None 2024-02-01T10:48:11.861 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:11 GMT /vclfcn-60/obj3?versionId=null 2024-02-01T10:48:11.861 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cpp5x/Jeemj44UpsVjaUDZtxLi0= 2024-02-01T10:48:11.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cpp5x/Jeemj44UpsVjaUDZtxLi0=', 'Content-Length': '0'} 2024-02-01T10:48:11.864 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ee760c26d65b3f02-0065bb76eb-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:11.865 DEBUG:boto:path=/vclfcn-60/obj3 2024-02-01T10:48:11.865 DEBUG:boto:auth_path=/vclfcn-60/obj3 2024-02-01T10:48:11.865 DEBUG:boto:path=/vclfcn-60/obj3?versionId=null 2024-02-01T10:48:11.865 DEBUG:boto:auth_path=/vclfcn-60/obj3?versionId=null 2024-02-01T10:48:11.865 DEBUG:boto:Method: GET 2024-02-01T10:48:11.865 DEBUG:boto:Path: /vclfcn-60/obj3?versionId=null 2024-02-01T10:48:11.865 DEBUG:boto:Data: 2024-02-01T10:48:11.866 DEBUG:boto:Headers: {} 2024-02-01T10:48:11.866 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:11.866 DEBUG:boto:Port: 8000 2024-02-01T10:48:11.866 DEBUG:boto:Params: {} 2024-02-01T10:48:11.866 DEBUG:boto:Token: None 2024-02-01T10:48:11.866 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:11 GMT /vclfcn-60/obj3?versionId=null 2024-02-01T10:48:11.866 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cpp5x/Jeemj44UpsVjaUDZtxLi0= 2024-02-01T10:48:11.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cpp5x/Jeemj44UpsVjaUDZtxLi0=', 'Content-Length': '0'} 2024-02-01T10:48:11.869 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f24211fa56502c8a-0065bb76eb-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:11.912 DEBUG:boto:path=/vclfcn-60/obj3 2024-02-01T10:48:11.912 DEBUG:boto:auth_path=/vclfcn-60/obj3 2024-02-01T10:48:11.912 DEBUG:boto:path=/vclfcn-60/obj3?versionId=null 2024-02-01T10:48:11.912 DEBUG:boto:auth_path=/vclfcn-60/obj3?versionId=null 2024-02-01T10:48:11.913 DEBUG:boto:Method: HEAD 2024-02-01T10:48:11.913 DEBUG:boto:Path: /vclfcn-60/obj3?versionId=null 2024-02-01T10:48:11.913 DEBUG:boto:Data: 2024-02-01T10:48:11.913 DEBUG:boto:Headers: {} 2024-02-01T10:48:11.913 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:11.913 DEBUG:boto:Port: 8001 2024-02-01T10:48:11.913 DEBUG:boto:Params: {} 2024-02-01T10:48:11.913 DEBUG:boto:Token: None 2024-02-01T10:48:11.914 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:11 GMT /vclfcn-60/obj3?versionId=null 2024-02-01T10:48:11.914 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7CZM+4JI+ijWjhlY09DWeegkgy4= 2024-02-01T10:48:11.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7CZM+4JI+ijWjhlY09DWeegkgy4=', 'Content-Length': '0'} 2024-02-01T10:48:11.917 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ceb10621d5529054-0065bb76eb-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:11.917 DEBUG:boto:path=/vclfcn-60/obj3 2024-02-01T10:48:11.917 DEBUG:boto:auth_path=/vclfcn-60/obj3 2024-02-01T10:48:11.917 DEBUG:boto:path=/vclfcn-60/obj3?versionId=null 2024-02-01T10:48:11.917 DEBUG:boto:auth_path=/vclfcn-60/obj3?versionId=null 2024-02-01T10:48:11.918 DEBUG:boto:Method: HEAD 2024-02-01T10:48:11.918 DEBUG:boto:Path: /vclfcn-60/obj3?versionId=null 2024-02-01T10:48:11.918 DEBUG:boto:Data: 2024-02-01T10:48:11.918 DEBUG:boto:Headers: {} 2024-02-01T10:48:11.918 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:11.918 DEBUG:boto:Port: 8000 2024-02-01T10:48:11.918 DEBUG:boto:Params: {} 2024-02-01T10:48:11.918 DEBUG:boto:Token: None 2024-02-01T10:48:11.919 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:11 GMT /vclfcn-60/obj3?versionId=null 2024-02-01T10:48:11.919 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7CZM+4JI+ijWjhlY09DWeegkgy4= 2024-02-01T10:48:11.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7CZM+4JI+ijWjhlY09DWeegkgy4=', 'Content-Length': '0'} 2024-02-01T10:48:11.922 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a03688c3b7c24489-0065bb76eb-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:11.922 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:48:11.922 DEBUG:boto:path=/vclfcn-60/obj3 2024-02-01T10:48:11.922 DEBUG:boto:auth_path=/vclfcn-60/obj3 2024-02-01T10:48:11.922 DEBUG:boto:path=/vclfcn-60/obj3?versionId=null 2024-02-01T10:48:11.922 DEBUG:boto:auth_path=/vclfcn-60/obj3?versionId=null 2024-02-01T10:48:11.922 DEBUG:boto:Method: GET 2024-02-01T10:48:11.923 DEBUG:boto:Path: /vclfcn-60/obj3?versionId=null 2024-02-01T10:48:11.923 DEBUG:boto:Data: 2024-02-01T10:48:11.923 DEBUG:boto:Headers: {} 2024-02-01T10:48:11.923 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:11.923 DEBUG:boto:Port: 8001 2024-02-01T10:48:11.923 DEBUG:boto:Params: {} 2024-02-01T10:48:11.923 DEBUG:boto:Token: None 2024-02-01T10:48:11.923 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:11 GMT /vclfcn-60/obj3?versionId=null 2024-02-01T10:48:11.923 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cpp5x/Jeemj44UpsVjaUDZtxLi0= 2024-02-01T10:48:11.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cpp5x/Jeemj44UpsVjaUDZtxLi0=', 'Content-Length': '0'} 2024-02-01T10:48:11.926 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000023371aaa60935a8e-0065bb76eb-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:11.969 DEBUG:boto:path=/vclfcn-60/obj3 2024-02-01T10:48:11.969 DEBUG:boto:auth_path=/vclfcn-60/obj3 2024-02-01T10:48:11.969 DEBUG:boto:path=/vclfcn-60/obj3?versionId=null 2024-02-01T10:48:11.969 DEBUG:boto:auth_path=/vclfcn-60/obj3?versionId=null 2024-02-01T10:48:11.969 DEBUG:boto:Method: GET 2024-02-01T10:48:11.969 DEBUG:boto:Path: /vclfcn-60/obj3?versionId=null 2024-02-01T10:48:11.969 DEBUG:boto:Data: 2024-02-01T10:48:11.970 DEBUG:boto:Headers: {} 2024-02-01T10:48:11.970 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:11.970 DEBUG:boto:Port: 8000 2024-02-01T10:48:11.970 DEBUG:boto:Params: {} 2024-02-01T10:48:11.970 DEBUG:boto:Token: None 2024-02-01T10:48:11.970 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:11 GMT /vclfcn-60/obj3?versionId=null 2024-02-01T10:48:11.970 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cpp5x/Jeemj44UpsVjaUDZtxLi0= 2024-02-01T10:48:11.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cpp5x/Jeemj44UpsVjaUDZtxLi0=', 'Content-Length': '0'} 2024-02-01T10:48:11.973 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006c7aa85a1f9c6f87-0065bb76eb-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:12.016 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:12.016 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:12.016 DEBUG:boto:path=/vclfcn-60/?prefix=obj3 2024-02-01T10:48:12.016 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj3 2024-02-01T10:48:12.017 DEBUG:boto:Method: GET 2024-02-01T10:48:12.017 DEBUG:boto:Path: /vclfcn-60/?prefix=obj3 2024-02-01T10:48:12.017 DEBUG:boto:Data: 2024-02-01T10:48:12.017 DEBUG:boto:Headers: {} 2024-02-01T10:48:12.017 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:12.017 DEBUG:boto:Port: 8001 2024-02-01T10:48:12.017 DEBUG:boto:Params: {} 2024-02-01T10:48:12.017 DEBUG:boto:Token: None 2024-02-01T10:48:12.017 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:12 GMT /vclfcn-60/ 2024-02-01T10:48:12.018 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:c8YGS1uhrw5y8f6UwmOI7ZGnJOU= 2024-02-01T10:48:12.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:c8YGS1uhrw5y8f6UwmOI7ZGnJOU=', 'Content-Length': '0'} 2024-02-01T10:48:12.022 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000051202e0aeadfd3e-0065bb76ec-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:12.064 DEBUG:boto:b'vclfcn-60obj31000falseobj32024-02-01T10:46:46.285Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:12.065 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:12.065 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:12.065 DEBUG:boto:path=/vclfcn-60/?prefix=obj3 2024-02-01T10:48:12.065 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj3 2024-02-01T10:48:12.065 DEBUG:boto:Method: GET 2024-02-01T10:48:12.065 DEBUG:boto:Path: /vclfcn-60/?prefix=obj3 2024-02-01T10:48:12.065 DEBUG:boto:Data: 2024-02-01T10:48:12.065 DEBUG:boto:Headers: {} 2024-02-01T10:48:12.065 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:12.066 DEBUG:boto:Port: 8000 2024-02-01T10:48:12.066 DEBUG:boto:Params: {} 2024-02-01T10:48:12.066 DEBUG:boto:Token: None 2024-02-01T10:48:12.066 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:12 GMT /vclfcn-60/ 2024-02-01T10:48:12.066 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:c8YGS1uhrw5y8f6UwmOI7ZGnJOU= 2024-02-01T10:48:12.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:c8YGS1uhrw5y8f6UwmOI7ZGnJOU=', 'Content-Length': '0'} 2024-02-01T10:48:12.071 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e4065a332a17309-0065bb76ec-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:12.071 DEBUG:boto:b'vclfcn-60obj31000falseobj32024-02-01T10:46:46.285Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:12.071 DEBUG:boto:path=/vclfcn-60/obj3 2024-02-01T10:48:12.071 DEBUG:boto:auth_path=/vclfcn-60/obj3 2024-02-01T10:48:12.071 DEBUG:boto:Method: HEAD 2024-02-01T10:48:12.071 DEBUG:boto:Path: /vclfcn-60/obj3 2024-02-01T10:48:12.072 DEBUG:boto:Data: 2024-02-01T10:48:12.072 DEBUG:boto:Headers: {} 2024-02-01T10:48:12.072 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:12.072 DEBUG:boto:Port: 8001 2024-02-01T10:48:12.072 DEBUG:boto:Params: {} 2024-02-01T10:48:12.072 DEBUG:boto:Token: None 2024-02-01T10:48:12.072 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:12 GMT /vclfcn-60/obj3 2024-02-01T10:48:12.073 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+W+iFupDEvb8v4ODC71UU/+oUOM= 2024-02-01T10:48:12.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+W+iFupDEvb8v4ODC71UU/+oUOM=', 'Content-Length': '0'} 2024-02-01T10:48:12.075 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000098dd6c26fe5ba09d-0065bb76ec-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:12.076 DEBUG:boto:path=/vclfcn-60/obj3 2024-02-01T10:48:12.076 DEBUG:boto:auth_path=/vclfcn-60/obj3 2024-02-01T10:48:12.076 DEBUG:boto:Method: HEAD 2024-02-01T10:48:12.076 DEBUG:boto:Path: /vclfcn-60/obj3 2024-02-01T10:48:12.076 DEBUG:boto:Data: 2024-02-01T10:48:12.076 DEBUG:boto:Headers: {} 2024-02-01T10:48:12.076 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:12.076 DEBUG:boto:Port: 8000 2024-02-01T10:48:12.076 DEBUG:boto:Params: {} 2024-02-01T10:48:12.076 DEBUG:boto:Token: None 2024-02-01T10:48:12.077 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:12 GMT /vclfcn-60/obj3 2024-02-01T10:48:12.077 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+W+iFupDEvb8v4ODC71UU/+oUOM= 2024-02-01T10:48:12.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+W+iFupDEvb8v4ODC71UU/+oUOM=', 'Content-Length': '0'} 2024-02-01T10:48:12.080 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007666469e8fb54701-0065bb76ec-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:12.080 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:48:12.080 DEBUG:boto:path=/vclfcn-60/obj3 2024-02-01T10:48:12.081 DEBUG:boto:auth_path=/vclfcn-60/obj3 2024-02-01T10:48:12.081 DEBUG:boto:Method: GET 2024-02-01T10:48:12.081 DEBUG:boto:Path: /vclfcn-60/obj3 2024-02-01T10:48:12.081 DEBUG:boto:Data: 2024-02-01T10:48:12.081 DEBUG:boto:Headers: {} 2024-02-01T10:48:12.081 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:12.081 DEBUG:boto:Port: 8001 2024-02-01T10:48:12.081 DEBUG:boto:Params: {} 2024-02-01T10:48:12.081 DEBUG:boto:Token: None 2024-02-01T10:48:12.082 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:12 GMT /vclfcn-60/obj3 2024-02-01T10:48:12.082 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:l5eD9J8Tno1iHFAXBHY6OZjeFSI= 2024-02-01T10:48:12.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:l5eD9J8Tno1iHFAXBHY6OZjeFSI=', 'Content-Length': '0'} 2024-02-01T10:48:12.085 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000074fdae482184f90e-0065bb76ec-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:12.128 DEBUG:boto:path=/vclfcn-60/obj3 2024-02-01T10:48:12.128 DEBUG:boto:auth_path=/vclfcn-60/obj3 2024-02-01T10:48:12.128 DEBUG:boto:Method: GET 2024-02-01T10:48:12.128 DEBUG:boto:Path: /vclfcn-60/obj3 2024-02-01T10:48:12.129 DEBUG:boto:Data: 2024-02-01T10:48:12.129 DEBUG:boto:Headers: {} 2024-02-01T10:48:12.129 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:12.129 DEBUG:boto:Port: 8000 2024-02-01T10:48:12.129 DEBUG:boto:Params: {} 2024-02-01T10:48:12.129 DEBUG:boto:Token: None 2024-02-01T10:48:12.129 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:12 GMT /vclfcn-60/obj3 2024-02-01T10:48:12.129 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:l5eD9J8Tno1iHFAXBHY6OZjeFSI= 2024-02-01T10:48:12.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:l5eD9J8Tno1iHFAXBHY6OZjeFSI=', 'Content-Length': '0'} 2024-02-01T10:48:12.133 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000096df973b07074e5a-0065bb76ec-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:12.176 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:12.176 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:12.176 DEBUG:boto:path=/vclfcn-60/?prefix=obj3 2024-02-01T10:48:12.177 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj3 2024-02-01T10:48:12.177 DEBUG:boto:Method: GET 2024-02-01T10:48:12.177 DEBUG:boto:Path: /vclfcn-60/?prefix=obj3 2024-02-01T10:48:12.177 DEBUG:boto:Data: 2024-02-01T10:48:12.177 DEBUG:boto:Headers: {} 2024-02-01T10:48:12.177 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:12.177 DEBUG:boto:Port: 8001 2024-02-01T10:48:12.178 DEBUG:boto:Params: {} 2024-02-01T10:48:12.178 DEBUG:boto:Token: None 2024-02-01T10:48:12.178 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:12 GMT /vclfcn-60/ 2024-02-01T10:48:12.178 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:c8YGS1uhrw5y8f6UwmOI7ZGnJOU= 2024-02-01T10:48:12.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:c8YGS1uhrw5y8f6UwmOI7ZGnJOU=', 'Content-Length': '0'} 2024-02-01T10:48:12.183 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050a243c35795fa51-0065bb76ec-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:12.183 DEBUG:boto:b'vclfcn-60obj31000falseobj32024-02-01T10:46:46.285Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:12.183 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:12.184 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:12.184 DEBUG:boto:path=/vclfcn-60/?prefix=obj3 2024-02-01T10:48:12.184 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj3 2024-02-01T10:48:12.184 DEBUG:boto:Method: GET 2024-02-01T10:48:12.184 DEBUG:boto:Path: /vclfcn-60/?prefix=obj3 2024-02-01T10:48:12.184 DEBUG:boto:Data: 2024-02-01T10:48:12.184 DEBUG:boto:Headers: {} 2024-02-01T10:48:12.184 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:12.184 DEBUG:boto:Port: 8000 2024-02-01T10:48:12.185 DEBUG:boto:Params: {} 2024-02-01T10:48:12.185 DEBUG:boto:Token: None 2024-02-01T10:48:12.185 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:12 GMT /vclfcn-60/ 2024-02-01T10:48:12.185 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:c8YGS1uhrw5y8f6UwmOI7ZGnJOU= 2024-02-01T10:48:12.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:c8YGS1uhrw5y8f6UwmOI7ZGnJOU=', 'Content-Length': '0'} 2024-02-01T10:48:12.190 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b16991704df35f30-0065bb76ec-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:12.190 DEBUG:boto:b'vclfcn-60obj31000falseobj32024-02-01T10:46:46.285Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:12.191 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:48:12.191 DEBUG:boto:path=/vclfcn-60/obj4 2024-02-01T10:48:12.191 DEBUG:boto:auth_path=/vclfcn-60/obj4 2024-02-01T10:48:12.191 DEBUG:boto:path=/vclfcn-60/obj4?versionId=null 2024-02-01T10:48:12.191 DEBUG:boto:auth_path=/vclfcn-60/obj4?versionId=null 2024-02-01T10:48:12.191 DEBUG:boto:Method: GET 2024-02-01T10:48:12.191 DEBUG:boto:Path: /vclfcn-60/obj4?versionId=null 2024-02-01T10:48:12.191 DEBUG:boto:Data: 2024-02-01T10:48:12.191 DEBUG:boto:Headers: {} 2024-02-01T10:48:12.192 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:12.192 DEBUG:boto:Port: 8001 2024-02-01T10:48:12.192 DEBUG:boto:Params: {} 2024-02-01T10:48:12.192 DEBUG:boto:Token: None 2024-02-01T10:48:12.192 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:12 GMT /vclfcn-60/obj4?versionId=null 2024-02-01T10:48:12.192 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:llr9Dw/7viOUhYulqlIfK+RU4lI= 2024-02-01T10:48:12.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:llr9Dw/7viOUhYulqlIfK+RU4lI=', 'Content-Length': '0'} 2024-02-01T10:48:12.195 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000093578dbce4d9124f-0065bb76ec-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:12.236 DEBUG:boto:path=/vclfcn-60/obj4 2024-02-01T10:48:12.236 DEBUG:boto:auth_path=/vclfcn-60/obj4 2024-02-01T10:48:12.236 DEBUG:boto:path=/vclfcn-60/obj4?versionId=null 2024-02-01T10:48:12.236 DEBUG:boto:auth_path=/vclfcn-60/obj4?versionId=null 2024-02-01T10:48:12.237 DEBUG:boto:Method: GET 2024-02-01T10:48:12.237 DEBUG:boto:Path: /vclfcn-60/obj4?versionId=null 2024-02-01T10:48:12.237 DEBUG:boto:Data: 2024-02-01T10:48:12.237 DEBUG:boto:Headers: {} 2024-02-01T10:48:12.237 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:12.237 DEBUG:boto:Port: 8000 2024-02-01T10:48:12.237 DEBUG:boto:Params: {} 2024-02-01T10:48:12.237 DEBUG:boto:Token: None 2024-02-01T10:48:12.237 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:12 GMT /vclfcn-60/obj4?versionId=null 2024-02-01T10:48:12.238 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:llr9Dw/7viOUhYulqlIfK+RU4lI= 2024-02-01T10:48:12.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:llr9Dw/7viOUhYulqlIfK+RU4lI=', 'Content-Length': '0'} 2024-02-01T10:48:12.241 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c5cd02881591aeb1-0065bb76ec-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:12.241 DEBUG:boto:path=/vclfcn-60/obj4 2024-02-01T10:48:12.241 DEBUG:boto:auth_path=/vclfcn-60/obj4 2024-02-01T10:48:12.241 DEBUG:boto:path=/vclfcn-60/obj4?versionId=null 2024-02-01T10:48:12.241 DEBUG:boto:auth_path=/vclfcn-60/obj4?versionId=null 2024-02-01T10:48:12.241 DEBUG:boto:Method: HEAD 2024-02-01T10:48:12.241 DEBUG:boto:Path: /vclfcn-60/obj4?versionId=null 2024-02-01T10:48:12.242 DEBUG:boto:Data: 2024-02-01T10:48:12.242 DEBUG:boto:Headers: {} 2024-02-01T10:48:12.242 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:12.242 DEBUG:boto:Port: 8001 2024-02-01T10:48:12.242 DEBUG:boto:Params: {} 2024-02-01T10:48:12.242 DEBUG:boto:Token: None 2024-02-01T10:48:12.242 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:12 GMT /vclfcn-60/obj4?versionId=null 2024-02-01T10:48:12.242 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZOsKjqvdqlTggczf/LVmHuVFt2Q= 2024-02-01T10:48:12.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZOsKjqvdqlTggczf/LVmHuVFt2Q=', 'Content-Length': '0'} 2024-02-01T10:48:12.245 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000030b11c75b4c30fb4-0065bb76ec-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:12.245 DEBUG:boto:path=/vclfcn-60/obj4 2024-02-01T10:48:12.245 DEBUG:boto:auth_path=/vclfcn-60/obj4 2024-02-01T10:48:12.245 DEBUG:boto:path=/vclfcn-60/obj4?versionId=null 2024-02-01T10:48:12.245 DEBUG:boto:auth_path=/vclfcn-60/obj4?versionId=null 2024-02-01T10:48:12.246 DEBUG:boto:Method: HEAD 2024-02-01T10:48:12.246 DEBUG:boto:Path: /vclfcn-60/obj4?versionId=null 2024-02-01T10:48:12.246 DEBUG:boto:Data: 2024-02-01T10:48:12.246 DEBUG:boto:Headers: {} 2024-02-01T10:48:12.246 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:12.246 DEBUG:boto:Port: 8000 2024-02-01T10:48:12.246 DEBUG:boto:Params: {} 2024-02-01T10:48:12.246 DEBUG:boto:Token: None 2024-02-01T10:48:12.246 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:12 GMT /vclfcn-60/obj4?versionId=null 2024-02-01T10:48:12.246 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZOsKjqvdqlTggczf/LVmHuVFt2Q= 2024-02-01T10:48:12.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZOsKjqvdqlTggczf/LVmHuVFt2Q=', 'Content-Length': '0'} 2024-02-01T10:48:12.249 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004aca51eca669072c-0065bb76ec-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:12.250 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:48:12.250 DEBUG:boto:path=/vclfcn-60/obj4 2024-02-01T10:48:12.250 DEBUG:boto:auth_path=/vclfcn-60/obj4 2024-02-01T10:48:12.250 DEBUG:boto:path=/vclfcn-60/obj4?versionId=null 2024-02-01T10:48:12.250 DEBUG:boto:auth_path=/vclfcn-60/obj4?versionId=null 2024-02-01T10:48:12.250 DEBUG:boto:Method: GET 2024-02-01T10:48:12.250 DEBUG:boto:Path: /vclfcn-60/obj4?versionId=null 2024-02-01T10:48:12.250 DEBUG:boto:Data: 2024-02-01T10:48:12.250 DEBUG:boto:Headers: {} 2024-02-01T10:48:12.250 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:12.251 DEBUG:boto:Port: 8001 2024-02-01T10:48:12.251 DEBUG:boto:Params: {} 2024-02-01T10:48:12.251 DEBUG:boto:Token: None 2024-02-01T10:48:12.251 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:12 GMT /vclfcn-60/obj4?versionId=null 2024-02-01T10:48:12.251 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:llr9Dw/7viOUhYulqlIfK+RU4lI= 2024-02-01T10:48:12.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:llr9Dw/7viOUhYulqlIfK+RU4lI=', 'Content-Length': '0'} 2024-02-01T10:48:12.254 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000123cd5d16700576a-0065bb76ec-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:12.300 DEBUG:boto:path=/vclfcn-60/obj4 2024-02-01T10:48:12.300 DEBUG:boto:auth_path=/vclfcn-60/obj4 2024-02-01T10:48:12.300 DEBUG:boto:path=/vclfcn-60/obj4?versionId=null 2024-02-01T10:48:12.301 DEBUG:boto:auth_path=/vclfcn-60/obj4?versionId=null 2024-02-01T10:48:12.301 DEBUG:boto:Method: GET 2024-02-01T10:48:12.301 DEBUG:boto:Path: /vclfcn-60/obj4?versionId=null 2024-02-01T10:48:12.301 DEBUG:boto:Data: 2024-02-01T10:48:12.301 DEBUG:boto:Headers: {} 2024-02-01T10:48:12.301 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:12.301 DEBUG:boto:Port: 8000 2024-02-01T10:48:12.301 DEBUG:boto:Params: {} 2024-02-01T10:48:12.301 DEBUG:boto:Token: None 2024-02-01T10:48:12.301 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:12 GMT /vclfcn-60/obj4?versionId=null 2024-02-01T10:48:12.302 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:llr9Dw/7viOUhYulqlIfK+RU4lI= 2024-02-01T10:48:12.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:llr9Dw/7viOUhYulqlIfK+RU4lI=', 'Content-Length': '0'} 2024-02-01T10:48:12.305 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004808b2b2c5f64f94-0065bb76ec-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:12.348 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:12.348 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:12.348 DEBUG:boto:path=/vclfcn-60/?prefix=obj4 2024-02-01T10:48:12.349 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj4 2024-02-01T10:48:12.349 DEBUG:boto:Method: GET 2024-02-01T10:48:12.349 DEBUG:boto:Path: /vclfcn-60/?prefix=obj4 2024-02-01T10:48:12.349 DEBUG:boto:Data: 2024-02-01T10:48:12.349 DEBUG:boto:Headers: {} 2024-02-01T10:48:12.349 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:12.349 DEBUG:boto:Port: 8001 2024-02-01T10:48:12.349 DEBUG:boto:Params: {} 2024-02-01T10:48:12.349 DEBUG:boto:Token: None 2024-02-01T10:48:12.349 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:12 GMT /vclfcn-60/ 2024-02-01T10:48:12.350 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:c8YGS1uhrw5y8f6UwmOI7ZGnJOU= 2024-02-01T10:48:12.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:c8YGS1uhrw5y8f6UwmOI7ZGnJOU=', 'Content-Length': '0'} 2024-02-01T10:48:12.354 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000738e649344a3528-0065bb76ec-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:12.396 DEBUG:boto:b'vclfcn-60obj41000falseobj42024-02-01T10:46:46.336Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:12.397 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:12.397 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:12.397 DEBUG:boto:path=/vclfcn-60/?prefix=obj4 2024-02-01T10:48:12.397 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj4 2024-02-01T10:48:12.397 DEBUG:boto:Method: GET 2024-02-01T10:48:12.397 DEBUG:boto:Path: /vclfcn-60/?prefix=obj4 2024-02-01T10:48:12.397 DEBUG:boto:Data: 2024-02-01T10:48:12.397 DEBUG:boto:Headers: {} 2024-02-01T10:48:12.397 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:12.397 DEBUG:boto:Port: 8000 2024-02-01T10:48:12.398 DEBUG:boto:Params: {} 2024-02-01T10:48:12.398 DEBUG:boto:Token: None 2024-02-01T10:48:12.398 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:12 GMT /vclfcn-60/ 2024-02-01T10:48:12.398 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:c8YGS1uhrw5y8f6UwmOI7ZGnJOU= 2024-02-01T10:48:12.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:c8YGS1uhrw5y8f6UwmOI7ZGnJOU=', 'Content-Length': '0'} 2024-02-01T10:48:12.404 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd428f95c4ba12ae-0065bb76ec-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:12.404 DEBUG:boto:b'vclfcn-60obj41000falseobj42024-02-01T10:46:46.336Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:12.405 DEBUG:boto:path=/vclfcn-60/obj4 2024-02-01T10:48:12.405 DEBUG:boto:auth_path=/vclfcn-60/obj4 2024-02-01T10:48:12.405 DEBUG:boto:Method: HEAD 2024-02-01T10:48:12.405 DEBUG:boto:Path: /vclfcn-60/obj4 2024-02-01T10:48:12.405 DEBUG:boto:Data: 2024-02-01T10:48:12.405 DEBUG:boto:Headers: {} 2024-02-01T10:48:12.405 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:12.405 DEBUG:boto:Port: 8001 2024-02-01T10:48:12.406 DEBUG:boto:Params: {} 2024-02-01T10:48:12.406 DEBUG:boto:Token: None 2024-02-01T10:48:12.406 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:12 GMT /vclfcn-60/obj4 2024-02-01T10:48:12.406 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:4BOCMz78D8WWd3LaGwHqPKPnNUM= 2024-02-01T10:48:12.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:4BOCMz78D8WWd3LaGwHqPKPnNUM=', 'Content-Length': '0'} 2024-02-01T10:48:12.409 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c1c5108f5a734d94-0065bb76ec-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:12.409 DEBUG:boto:path=/vclfcn-60/obj4 2024-02-01T10:48:12.409 DEBUG:boto:auth_path=/vclfcn-60/obj4 2024-02-01T10:48:12.409 DEBUG:boto:Method: HEAD 2024-02-01T10:48:12.409 DEBUG:boto:Path: /vclfcn-60/obj4 2024-02-01T10:48:12.410 DEBUG:boto:Data: 2024-02-01T10:48:12.410 DEBUG:boto:Headers: {} 2024-02-01T10:48:12.410 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:12.410 DEBUG:boto:Port: 8000 2024-02-01T10:48:12.410 DEBUG:boto:Params: {} 2024-02-01T10:48:12.410 DEBUG:boto:Token: None 2024-02-01T10:48:12.410 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:12 GMT /vclfcn-60/obj4 2024-02-01T10:48:12.410 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:4BOCMz78D8WWd3LaGwHqPKPnNUM= 2024-02-01T10:48:12.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:4BOCMz78D8WWd3LaGwHqPKPnNUM=', 'Content-Length': '0'} 2024-02-01T10:48:12.413 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003771d6fe47ed1d3c-0065bb76ec-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:12.413 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:48:12.413 DEBUG:boto:path=/vclfcn-60/obj4 2024-02-01T10:48:12.414 DEBUG:boto:auth_path=/vclfcn-60/obj4 2024-02-01T10:48:12.414 DEBUG:boto:Method: GET 2024-02-01T10:48:12.414 DEBUG:boto:Path: /vclfcn-60/obj4 2024-02-01T10:48:12.414 DEBUG:boto:Data: 2024-02-01T10:48:12.414 DEBUG:boto:Headers: {} 2024-02-01T10:48:12.414 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:12.414 DEBUG:boto:Port: 8001 2024-02-01T10:48:12.414 DEBUG:boto:Params: {} 2024-02-01T10:48:12.414 DEBUG:boto:Token: None 2024-02-01T10:48:12.414 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:12 GMT /vclfcn-60/obj4 2024-02-01T10:48:12.415 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:A9Box9O9kqb1SvAaieLqY5f/dJ0= 2024-02-01T10:48:12.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:A9Box9O9kqb1SvAaieLqY5f/dJ0=', 'Content-Length': '0'} 2024-02-01T10:48:12.418 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c91a8e64388db3ac-0065bb76ec-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:12.464 DEBUG:boto:path=/vclfcn-60/obj4 2024-02-01T10:48:12.464 DEBUG:boto:auth_path=/vclfcn-60/obj4 2024-02-01T10:48:12.464 DEBUG:boto:Method: GET 2024-02-01T10:48:12.465 DEBUG:boto:Path: /vclfcn-60/obj4 2024-02-01T10:48:12.465 DEBUG:boto:Data: 2024-02-01T10:48:12.465 DEBUG:boto:Headers: {} 2024-02-01T10:48:12.465 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:12.465 DEBUG:boto:Port: 8000 2024-02-01T10:48:12.465 DEBUG:boto:Params: {} 2024-02-01T10:48:12.465 DEBUG:boto:Token: None 2024-02-01T10:48:12.465 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:12 GMT /vclfcn-60/obj4 2024-02-01T10:48:12.465 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:A9Box9O9kqb1SvAaieLqY5f/dJ0= 2024-02-01T10:48:12.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:A9Box9O9kqb1SvAaieLqY5f/dJ0=', 'Content-Length': '0'} 2024-02-01T10:48:12.469 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003380ac563b62cca1-0065bb76ec-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:12.512 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:12.512 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:12.512 DEBUG:boto:path=/vclfcn-60/?prefix=obj4 2024-02-01T10:48:12.513 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj4 2024-02-01T10:48:12.513 DEBUG:boto:Method: GET 2024-02-01T10:48:12.513 DEBUG:boto:Path: /vclfcn-60/?prefix=obj4 2024-02-01T10:48:12.513 DEBUG:boto:Data: 2024-02-01T10:48:12.513 DEBUG:boto:Headers: {} 2024-02-01T10:48:12.513 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:12.513 DEBUG:boto:Port: 8001 2024-02-01T10:48:12.513 DEBUG:boto:Params: {} 2024-02-01T10:48:12.513 DEBUG:boto:Token: None 2024-02-01T10:48:12.513 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:12 GMT /vclfcn-60/ 2024-02-01T10:48:12.514 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:c8YGS1uhrw5y8f6UwmOI7ZGnJOU= 2024-02-01T10:48:12.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:c8YGS1uhrw5y8f6UwmOI7ZGnJOU=', 'Content-Length': '0'} 2024-02-01T10:48:12.518 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8b279b117df7a64-0065bb76ec-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:12.560 DEBUG:boto:b'vclfcn-60obj41000falseobj42024-02-01T10:46:46.336Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:12.563 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:12.563 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:12.563 DEBUG:boto:path=/vclfcn-60/?prefix=obj4 2024-02-01T10:48:12.564 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj4 2024-02-01T10:48:12.564 DEBUG:boto:Method: GET 2024-02-01T10:48:12.564 DEBUG:boto:Path: /vclfcn-60/?prefix=obj4 2024-02-01T10:48:12.564 DEBUG:boto:Data: 2024-02-01T10:48:12.564 DEBUG:boto:Headers: {} 2024-02-01T10:48:12.564 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:12.564 DEBUG:boto:Port: 8000 2024-02-01T10:48:12.564 DEBUG:boto:Params: {} 2024-02-01T10:48:12.564 DEBUG:boto:Token: None 2024-02-01T10:48:12.565 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:12 GMT /vclfcn-60/ 2024-02-01T10:48:12.565 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:c8YGS1uhrw5y8f6UwmOI7ZGnJOU= 2024-02-01T10:48:12.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:c8YGS1uhrw5y8f6UwmOI7ZGnJOU=', 'Content-Length': '0'} 2024-02-01T10:48:12.569 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f716b49a0afed219-0065bb76ec-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:12.612 DEBUG:boto:b'vclfcn-60obj41000falseobj42024-02-01T10:46:46.336Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:12.612 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-02-01T10:48:12.613 DEBUG:boto:path=/vclfcn-60/obj5 2024-02-01T10:48:12.613 DEBUG:boto:auth_path=/vclfcn-60/obj5 2024-02-01T10:48:12.613 DEBUG:boto:path=/vclfcn-60/obj5?versionId=null 2024-02-01T10:48:12.613 DEBUG:boto:auth_path=/vclfcn-60/obj5?versionId=null 2024-02-01T10:48:12.613 DEBUG:boto:Method: GET 2024-02-01T10:48:12.613 DEBUG:boto:Path: /vclfcn-60/obj5?versionId=null 2024-02-01T10:48:12.613 DEBUG:boto:Data: 2024-02-01T10:48:12.613 DEBUG:boto:Headers: {} 2024-02-01T10:48:12.613 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:12.613 DEBUG:boto:Port: 8001 2024-02-01T10:48:12.613 DEBUG:boto:Params: {} 2024-02-01T10:48:12.614 DEBUG:boto:Token: None 2024-02-01T10:48:12.614 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:12 GMT /vclfcn-60/obj5?versionId=null 2024-02-01T10:48:12.614 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QG3ipw58snpg5GETsJCiQNk+vW8= 2024-02-01T10:48:12.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QG3ipw58snpg5GETsJCiQNk+vW8=', 'Content-Length': '0'} 2024-02-01T10:48:12.617 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b1770ad4b6f6a225-0065bb76ec-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:12.617 DEBUG:boto:path=/vclfcn-60/obj5 2024-02-01T10:48:12.617 DEBUG:boto:auth_path=/vclfcn-60/obj5 2024-02-01T10:48:12.618 DEBUG:boto:path=/vclfcn-60/obj5?versionId=null 2024-02-01T10:48:12.618 DEBUG:boto:auth_path=/vclfcn-60/obj5?versionId=null 2024-02-01T10:48:12.618 DEBUG:boto:Method: GET 2024-02-01T10:48:12.618 DEBUG:boto:Path: /vclfcn-60/obj5?versionId=null 2024-02-01T10:48:12.618 DEBUG:boto:Data: 2024-02-01T10:48:12.618 DEBUG:boto:Headers: {} 2024-02-01T10:48:12.618 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:12.618 DEBUG:boto:Port: 8000 2024-02-01T10:48:12.618 DEBUG:boto:Params: {} 2024-02-01T10:48:12.618 DEBUG:boto:Token: None 2024-02-01T10:48:12.619 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:12 GMT /vclfcn-60/obj5?versionId=null 2024-02-01T10:48:12.619 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QG3ipw58snpg5GETsJCiQNk+vW8= 2024-02-01T10:48:12.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QG3ipw58snpg5GETsJCiQNk+vW8=', 'Content-Length': '0'} 2024-02-01T10:48:12.622 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ddea598d1194bf85-0065bb76ec-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:12.622 DEBUG:boto:path=/vclfcn-60/obj5 2024-02-01T10:48:12.622 DEBUG:boto:auth_path=/vclfcn-60/obj5 2024-02-01T10:48:12.622 DEBUG:boto:path=/vclfcn-60/obj5?versionId=null 2024-02-01T10:48:12.623 DEBUG:boto:auth_path=/vclfcn-60/obj5?versionId=null 2024-02-01T10:48:12.623 DEBUG:boto:Method: HEAD 2024-02-01T10:48:12.623 DEBUG:boto:Path: /vclfcn-60/obj5?versionId=null 2024-02-01T10:48:12.623 DEBUG:boto:Data: 2024-02-01T10:48:12.623 DEBUG:boto:Headers: {} 2024-02-01T10:48:12.623 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:12.623 DEBUG:boto:Port: 8001 2024-02-01T10:48:12.623 DEBUG:boto:Params: {} 2024-02-01T10:48:12.623 DEBUG:boto:Token: None 2024-02-01T10:48:12.623 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:12 GMT /vclfcn-60/obj5?versionId=null 2024-02-01T10:48:12.623 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:EVyjzJylxsmuJ5gp4yC/V8OycAo= 2024-02-01T10:48:12.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:EVyjzJylxsmuJ5gp4yC/V8OycAo=', 'Content-Length': '0'} 2024-02-01T10:48:12.626 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004f0088541bebbbe9-0065bb76ec-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:12.626 DEBUG:boto:path=/vclfcn-60/obj5 2024-02-01T10:48:12.627 DEBUG:boto:auth_path=/vclfcn-60/obj5 2024-02-01T10:48:12.627 DEBUG:boto:path=/vclfcn-60/obj5?versionId=null 2024-02-01T10:48:12.627 DEBUG:boto:auth_path=/vclfcn-60/obj5?versionId=null 2024-02-01T10:48:12.627 DEBUG:boto:Method: HEAD 2024-02-01T10:48:12.627 DEBUG:boto:Path: /vclfcn-60/obj5?versionId=null 2024-02-01T10:48:12.627 DEBUG:boto:Data: 2024-02-01T10:48:12.627 DEBUG:boto:Headers: {} 2024-02-01T10:48:12.627 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:12.627 DEBUG:boto:Port: 8000 2024-02-01T10:48:12.627 DEBUG:boto:Params: {} 2024-02-01T10:48:12.628 DEBUG:boto:Token: None 2024-02-01T10:48:12.628 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:12 GMT /vclfcn-60/obj5?versionId=null 2024-02-01T10:48:12.628 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:EVyjzJylxsmuJ5gp4yC/V8OycAo= 2024-02-01T10:48:12.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:EVyjzJylxsmuJ5gp4yC/V8OycAo=', 'Content-Length': '0'} 2024-02-01T10:48:12.631 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003851d925a9c7f40d-0065bb76ec-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:12.631 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-02-01T10:48:12.631 DEBUG:boto:path=/vclfcn-60/obj5 2024-02-01T10:48:12.631 DEBUG:boto:auth_path=/vclfcn-60/obj5 2024-02-01T10:48:12.631 DEBUG:boto:path=/vclfcn-60/obj5?versionId=null 2024-02-01T10:48:12.631 DEBUG:boto:auth_path=/vclfcn-60/obj5?versionId=null 2024-02-01T10:48:12.631 DEBUG:boto:Method: GET 2024-02-01T10:48:12.632 DEBUG:boto:Path: /vclfcn-60/obj5?versionId=null 2024-02-01T10:48:12.632 DEBUG:boto:Data: 2024-02-01T10:48:12.632 DEBUG:boto:Headers: {} 2024-02-01T10:48:12.632 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:12.632 DEBUG:boto:Port: 8001 2024-02-01T10:48:12.632 DEBUG:boto:Params: {} 2024-02-01T10:48:12.632 DEBUG:boto:Token: None 2024-02-01T10:48:12.632 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:12 GMT /vclfcn-60/obj5?versionId=null 2024-02-01T10:48:12.632 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QG3ipw58snpg5GETsJCiQNk+vW8= 2024-02-01T10:48:12.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QG3ipw58snpg5GETsJCiQNk+vW8=', 'Content-Length': '0'} 2024-02-01T10:48:12.635 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c0a64af3c86f9afb-0065bb76ec-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:12.676 DEBUG:boto:path=/vclfcn-60/obj5 2024-02-01T10:48:12.676 DEBUG:boto:auth_path=/vclfcn-60/obj5 2024-02-01T10:48:12.676 DEBUG:boto:path=/vclfcn-60/obj5?versionId=null 2024-02-01T10:48:12.676 DEBUG:boto:auth_path=/vclfcn-60/obj5?versionId=null 2024-02-01T10:48:12.677 DEBUG:boto:Method: GET 2024-02-01T10:48:12.677 DEBUG:boto:Path: /vclfcn-60/obj5?versionId=null 2024-02-01T10:48:12.677 DEBUG:boto:Data: 2024-02-01T10:48:12.677 DEBUG:boto:Headers: {} 2024-02-01T10:48:12.677 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:12.677 DEBUG:boto:Port: 8000 2024-02-01T10:48:12.677 DEBUG:boto:Params: {} 2024-02-01T10:48:12.678 DEBUG:boto:Token: None 2024-02-01T10:48:12.678 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:12 GMT /vclfcn-60/obj5?versionId=null 2024-02-01T10:48:12.678 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QG3ipw58snpg5GETsJCiQNk+vW8= 2024-02-01T10:48:12.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QG3ipw58snpg5GETsJCiQNk+vW8=', 'Content-Length': '0'} 2024-02-01T10:48:12.681 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000018132bce44f5af89-0065bb76ec-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:12.724 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:12.724 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:12.724 DEBUG:boto:path=/vclfcn-60/?prefix=obj5 2024-02-01T10:48:12.724 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj5 2024-02-01T10:48:12.725 DEBUG:boto:Method: GET 2024-02-01T10:48:12.725 DEBUG:boto:Path: /vclfcn-60/?prefix=obj5 2024-02-01T10:48:12.725 DEBUG:boto:Data: 2024-02-01T10:48:12.725 DEBUG:boto:Headers: {} 2024-02-01T10:48:12.725 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:12.725 DEBUG:boto:Port: 8001 2024-02-01T10:48:12.725 DEBUG:boto:Params: {} 2024-02-01T10:48:12.725 DEBUG:boto:Token: None 2024-02-01T10:48:12.725 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:12 GMT /vclfcn-60/ 2024-02-01T10:48:12.725 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:c8YGS1uhrw5y8f6UwmOI7ZGnJOU= 2024-02-01T10:48:12.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:c8YGS1uhrw5y8f6UwmOI7ZGnJOU=', 'Content-Length': '0'} 2024-02-01T10:48:12.735 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d5f2b4f50adebe9-0065bb76ec-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:12.776 DEBUG:boto:b'vclfcn-60obj51000falseobj52024-02-01T10:47:30.291Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:12.777 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:12.777 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:12.777 DEBUG:boto:path=/vclfcn-60/?prefix=obj5 2024-02-01T10:48:12.777 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj5 2024-02-01T10:48:12.777 DEBUG:boto:Method: GET 2024-02-01T10:48:12.778 DEBUG:boto:Path: /vclfcn-60/?prefix=obj5 2024-02-01T10:48:12.778 DEBUG:boto:Data: 2024-02-01T10:48:12.778 DEBUG:boto:Headers: {} 2024-02-01T10:48:12.778 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:12.778 DEBUG:boto:Port: 8000 2024-02-01T10:48:12.778 DEBUG:boto:Params: {} 2024-02-01T10:48:12.778 DEBUG:boto:Token: None 2024-02-01T10:48:12.778 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:12 GMT /vclfcn-60/ 2024-02-01T10:48:12.778 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:c8YGS1uhrw5y8f6UwmOI7ZGnJOU= 2024-02-01T10:48:12.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:c8YGS1uhrw5y8f6UwmOI7ZGnJOU=', 'Content-Length': '0'} 2024-02-01T10:48:12.784 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000934625b77798c0a-0065bb76ec-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:12.824 DEBUG:boto:b'vclfcn-60obj51000falseobj52024-02-01T10:47:30.291Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:12.824 DEBUG:boto:path=/vclfcn-60/obj5 2024-02-01T10:48:12.824 DEBUG:boto:auth_path=/vclfcn-60/obj5 2024-02-01T10:48:12.825 DEBUG:boto:Method: HEAD 2024-02-01T10:48:12.825 DEBUG:boto:Path: /vclfcn-60/obj5 2024-02-01T10:48:12.825 DEBUG:boto:Data: 2024-02-01T10:48:12.825 DEBUG:boto:Headers: {} 2024-02-01T10:48:12.825 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:12.825 DEBUG:boto:Port: 8001 2024-02-01T10:48:12.825 DEBUG:boto:Params: {} 2024-02-01T10:48:12.825 DEBUG:boto:Token: None 2024-02-01T10:48:12.825 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:12 GMT /vclfcn-60/obj5 2024-02-01T10:48:12.826 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WvMGGX8+J6ZyzDU59x50S6PNuZk= 2024-02-01T10:48:12.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WvMGGX8+J6ZyzDU59x50S6PNuZk=', 'Content-Length': '0'} 2024-02-01T10:48:12.828 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d88d4c99670e1f7c-0065bb76ec-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:12.829 DEBUG:boto:path=/vclfcn-60/obj5 2024-02-01T10:48:12.829 DEBUG:boto:auth_path=/vclfcn-60/obj5 2024-02-01T10:48:12.829 DEBUG:boto:Method: HEAD 2024-02-01T10:48:12.829 DEBUG:boto:Path: /vclfcn-60/obj5 2024-02-01T10:48:12.829 DEBUG:boto:Data: 2024-02-01T10:48:12.829 DEBUG:boto:Headers: {} 2024-02-01T10:48:12.829 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:12.829 DEBUG:boto:Port: 8000 2024-02-01T10:48:12.830 DEBUG:boto:Params: {} 2024-02-01T10:48:12.830 DEBUG:boto:Token: None 2024-02-01T10:48:12.830 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:12 GMT /vclfcn-60/obj5 2024-02-01T10:48:12.830 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WvMGGX8+J6ZyzDU59x50S6PNuZk= 2024-02-01T10:48:12.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WvMGGX8+J6ZyzDU59x50S6PNuZk=', 'Content-Length': '0'} 2024-02-01T10:48:12.833 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000050af79f71e23ffc1-0065bb76ec-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:12.833 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-02-01T10:48:12.833 DEBUG:boto:path=/vclfcn-60/obj5 2024-02-01T10:48:12.833 DEBUG:boto:auth_path=/vclfcn-60/obj5 2024-02-01T10:48:12.834 DEBUG:boto:Method: GET 2024-02-01T10:48:12.834 DEBUG:boto:Path: /vclfcn-60/obj5 2024-02-01T10:48:12.834 DEBUG:boto:Data: 2024-02-01T10:48:12.834 DEBUG:boto:Headers: {} 2024-02-01T10:48:12.834 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:12.834 DEBUG:boto:Port: 8001 2024-02-01T10:48:12.834 DEBUG:boto:Params: {} 2024-02-01T10:48:12.834 DEBUG:boto:Token: None 2024-02-01T10:48:12.834 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:12 GMT /vclfcn-60/obj5 2024-02-01T10:48:12.834 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ehdVOn8B9cvc9OFLlCZadft68Jc= 2024-02-01T10:48:12.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ehdVOn8B9cvc9OFLlCZadft68Jc=', 'Content-Length': '0'} 2024-02-01T10:48:12.838 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009a31d22710f31dfa-0065bb76ec-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:12.884 DEBUG:boto:path=/vclfcn-60/obj5 2024-02-01T10:48:12.884 DEBUG:boto:auth_path=/vclfcn-60/obj5 2024-02-01T10:48:12.884 DEBUG:boto:Method: GET 2024-02-01T10:48:12.884 DEBUG:boto:Path: /vclfcn-60/obj5 2024-02-01T10:48:12.885 DEBUG:boto:Data: 2024-02-01T10:48:12.885 DEBUG:boto:Headers: {} 2024-02-01T10:48:12.885 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:12.885 DEBUG:boto:Port: 8000 2024-02-01T10:48:12.885 DEBUG:boto:Params: {} 2024-02-01T10:48:12.885 DEBUG:boto:Token: None 2024-02-01T10:48:12.885 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:12 GMT /vclfcn-60/obj5 2024-02-01T10:48:12.885 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ehdVOn8B9cvc9OFLlCZadft68Jc= 2024-02-01T10:48:12.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ehdVOn8B9cvc9OFLlCZadft68Jc=', 'Content-Length': '0'} 2024-02-01T10:48:12.888 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bff7e6c5d8c2d955-0065bb76ec-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:12.932 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:12.932 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:12.932 DEBUG:boto:path=/vclfcn-60/?prefix=obj5 2024-02-01T10:48:12.932 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj5 2024-02-01T10:48:12.932 DEBUG:boto:Method: GET 2024-02-01T10:48:12.933 DEBUG:boto:Path: /vclfcn-60/?prefix=obj5 2024-02-01T10:48:12.933 DEBUG:boto:Data: 2024-02-01T10:48:12.933 DEBUG:boto:Headers: {} 2024-02-01T10:48:12.933 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:12.933 DEBUG:boto:Port: 8001 2024-02-01T10:48:12.933 DEBUG:boto:Params: {} 2024-02-01T10:48:12.933 DEBUG:boto:Token: None 2024-02-01T10:48:12.933 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:12 GMT /vclfcn-60/ 2024-02-01T10:48:12.933 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:c8YGS1uhrw5y8f6UwmOI7ZGnJOU= 2024-02-01T10:48:12.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:c8YGS1uhrw5y8f6UwmOI7ZGnJOU=', 'Content-Length': '0'} 2024-02-01T10:48:12.938 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b629192fafc79fea-0065bb76ec-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:12.980 DEBUG:boto:b'vclfcn-60obj51000falseobj52024-02-01T10:47:30.291Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:12.981 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:12.981 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:12.981 DEBUG:boto:path=/vclfcn-60/?prefix=obj5 2024-02-01T10:48:12.981 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj5 2024-02-01T10:48:12.981 DEBUG:boto:Method: GET 2024-02-01T10:48:12.981 DEBUG:boto:Path: /vclfcn-60/?prefix=obj5 2024-02-01T10:48:12.981 DEBUG:boto:Data: 2024-02-01T10:48:12.981 DEBUG:boto:Headers: {} 2024-02-01T10:48:12.981 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:12.981 DEBUG:boto:Port: 8000 2024-02-01T10:48:12.982 DEBUG:boto:Params: {} 2024-02-01T10:48:12.982 DEBUG:boto:Token: None 2024-02-01T10:48:12.982 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:12 GMT /vclfcn-60/ 2024-02-01T10:48:12.982 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:c8YGS1uhrw5y8f6UwmOI7ZGnJOU= 2024-02-01T10:48:12.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:c8YGS1uhrw5y8f6UwmOI7ZGnJOU=', 'Content-Length': '0'} 2024-02-01T10:48:12.987 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082dc2140f8a8318f-0065bb76ec-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:13.028 DEBUG:boto:b'vclfcn-60obj51000falseobj52024-02-01T10:47:30.291Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:13.028 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-02-01T10:48:13.029 DEBUG:boto:path=/vclfcn-60/obj6 2024-02-01T10:48:13.029 DEBUG:boto:auth_path=/vclfcn-60/obj6 2024-02-01T10:48:13.029 DEBUG:boto:path=/vclfcn-60/obj6?versionId=null 2024-02-01T10:48:13.029 DEBUG:boto:auth_path=/vclfcn-60/obj6?versionId=null 2024-02-01T10:48:13.029 DEBUG:boto:Method: GET 2024-02-01T10:48:13.029 DEBUG:boto:Path: /vclfcn-60/obj6?versionId=null 2024-02-01T10:48:13.030 DEBUG:boto:Data: 2024-02-01T10:48:13.030 DEBUG:boto:Headers: {} 2024-02-01T10:48:13.030 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:13.030 DEBUG:boto:Port: 8001 2024-02-01T10:48:13.030 DEBUG:boto:Params: {} 2024-02-01T10:48:13.030 DEBUG:boto:Token: None 2024-02-01T10:48:13.030 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:13 GMT /vclfcn-60/obj6?versionId=null 2024-02-01T10:48:13.030 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GKi6IVu7W30beHcmdzGmA3yIJn4= 2024-02-01T10:48:13.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GKi6IVu7W30beHcmdzGmA3yIJn4=', 'Content-Length': '0'} 2024-02-01T10:48:13.034 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000044045602e1457095-0065bb76ed-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:13.076 DEBUG:boto:path=/vclfcn-60/obj6 2024-02-01T10:48:13.076 DEBUG:boto:auth_path=/vclfcn-60/obj6 2024-02-01T10:48:13.076 DEBUG:boto:path=/vclfcn-60/obj6?versionId=null 2024-02-01T10:48:13.077 DEBUG:boto:auth_path=/vclfcn-60/obj6?versionId=null 2024-02-01T10:48:13.077 DEBUG:boto:Method: GET 2024-02-01T10:48:13.077 DEBUG:boto:Path: /vclfcn-60/obj6?versionId=null 2024-02-01T10:48:13.077 DEBUG:boto:Data: 2024-02-01T10:48:13.077 DEBUG:boto:Headers: {} 2024-02-01T10:48:13.077 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:13.077 DEBUG:boto:Port: 8000 2024-02-01T10:48:13.077 DEBUG:boto:Params: {} 2024-02-01T10:48:13.078 DEBUG:boto:Token: None 2024-02-01T10:48:13.078 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:13 GMT /vclfcn-60/obj6?versionId=null 2024-02-01T10:48:13.078 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GKi6IVu7W30beHcmdzGmA3yIJn4= 2024-02-01T10:48:13.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GKi6IVu7W30beHcmdzGmA3yIJn4=', 'Content-Length': '0'} 2024-02-01T10:48:13.081 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000100a6eb1f9ac39b-0065bb76ed-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:13.124 DEBUG:boto:path=/vclfcn-60/obj6 2024-02-01T10:48:13.124 DEBUG:boto:auth_path=/vclfcn-60/obj6 2024-02-01T10:48:13.124 DEBUG:boto:path=/vclfcn-60/obj6?versionId=null 2024-02-01T10:48:13.124 DEBUG:boto:auth_path=/vclfcn-60/obj6?versionId=null 2024-02-01T10:48:13.125 DEBUG:boto:Method: HEAD 2024-02-01T10:48:13.125 DEBUG:boto:Path: /vclfcn-60/obj6?versionId=null 2024-02-01T10:48:13.125 DEBUG:boto:Data: 2024-02-01T10:48:13.125 DEBUG:boto:Headers: {} 2024-02-01T10:48:13.125 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:13.125 DEBUG:boto:Port: 8001 2024-02-01T10:48:13.125 DEBUG:boto:Params: {} 2024-02-01T10:48:13.125 DEBUG:boto:Token: None 2024-02-01T10:48:13.125 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:13 GMT /vclfcn-60/obj6?versionId=null 2024-02-01T10:48:13.126 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:iabMLnuvEvKTLBSQ+rPbZ93T3pE= 2024-02-01T10:48:13.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iabMLnuvEvKTLBSQ+rPbZ93T3pE=', 'Content-Length': '0'} 2024-02-01T10:48:13.128 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000086dd026e94239551-0065bb76ed-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:13.129 DEBUG:boto:path=/vclfcn-60/obj6 2024-02-01T10:48:13.129 DEBUG:boto:auth_path=/vclfcn-60/obj6 2024-02-01T10:48:13.129 DEBUG:boto:path=/vclfcn-60/obj6?versionId=null 2024-02-01T10:48:13.129 DEBUG:boto:auth_path=/vclfcn-60/obj6?versionId=null 2024-02-01T10:48:13.129 DEBUG:boto:Method: HEAD 2024-02-01T10:48:13.129 DEBUG:boto:Path: /vclfcn-60/obj6?versionId=null 2024-02-01T10:48:13.129 DEBUG:boto:Data: 2024-02-01T10:48:13.129 DEBUG:boto:Headers: {} 2024-02-01T10:48:13.129 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:13.130 DEBUG:boto:Port: 8000 2024-02-01T10:48:13.130 DEBUG:boto:Params: {} 2024-02-01T10:48:13.130 DEBUG:boto:Token: None 2024-02-01T10:48:13.130 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:13 GMT /vclfcn-60/obj6?versionId=null 2024-02-01T10:48:13.130 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:iabMLnuvEvKTLBSQ+rPbZ93T3pE= 2024-02-01T10:48:13.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iabMLnuvEvKTLBSQ+rPbZ93T3pE=', 'Content-Length': '0'} 2024-02-01T10:48:13.133 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008ae9f3b05f561c8d-0065bb76ed-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:13.133 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-02-01T10:48:13.133 DEBUG:boto:path=/vclfcn-60/obj6 2024-02-01T10:48:13.133 DEBUG:boto:auth_path=/vclfcn-60/obj6 2024-02-01T10:48:13.133 DEBUG:boto:path=/vclfcn-60/obj6?versionId=null 2024-02-01T10:48:13.133 DEBUG:boto:auth_path=/vclfcn-60/obj6?versionId=null 2024-02-01T10:48:13.134 DEBUG:boto:Method: GET 2024-02-01T10:48:13.134 DEBUG:boto:Path: /vclfcn-60/obj6?versionId=null 2024-02-01T10:48:13.134 DEBUG:boto:Data: 2024-02-01T10:48:13.134 DEBUG:boto:Headers: {} 2024-02-01T10:48:13.134 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:13.134 DEBUG:boto:Port: 8001 2024-02-01T10:48:13.134 DEBUG:boto:Params: {} 2024-02-01T10:48:13.134 DEBUG:boto:Token: None 2024-02-01T10:48:13.134 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:13 GMT /vclfcn-60/obj6?versionId=null 2024-02-01T10:48:13.134 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GKi6IVu7W30beHcmdzGmA3yIJn4= 2024-02-01T10:48:13.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GKi6IVu7W30beHcmdzGmA3yIJn4=', 'Content-Length': '0'} 2024-02-01T10:48:13.137 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000997ec4e03d663362-0065bb76ed-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:13.180 DEBUG:boto:path=/vclfcn-60/obj6 2024-02-01T10:48:13.180 DEBUG:boto:auth_path=/vclfcn-60/obj6 2024-02-01T10:48:13.181 DEBUG:boto:path=/vclfcn-60/obj6?versionId=null 2024-02-01T10:48:13.181 DEBUG:boto:auth_path=/vclfcn-60/obj6?versionId=null 2024-02-01T10:48:13.181 DEBUG:boto:Method: GET 2024-02-01T10:48:13.181 DEBUG:boto:Path: /vclfcn-60/obj6?versionId=null 2024-02-01T10:48:13.181 DEBUG:boto:Data: 2024-02-01T10:48:13.181 DEBUG:boto:Headers: {} 2024-02-01T10:48:13.181 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:13.181 DEBUG:boto:Port: 8000 2024-02-01T10:48:13.181 DEBUG:boto:Params: {} 2024-02-01T10:48:13.181 DEBUG:boto:Token: None 2024-02-01T10:48:13.182 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:13 GMT /vclfcn-60/obj6?versionId=null 2024-02-01T10:48:13.182 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GKi6IVu7W30beHcmdzGmA3yIJn4= 2024-02-01T10:48:13.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GKi6IVu7W30beHcmdzGmA3yIJn4=', 'Content-Length': '0'} 2024-02-01T10:48:13.185 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000022d0b8fadefe4eea-0065bb76ed-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:13.232 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:13.232 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:13.233 DEBUG:boto:path=/vclfcn-60/?prefix=obj6 2024-02-01T10:48:13.233 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj6 2024-02-01T10:48:13.233 DEBUG:boto:Method: GET 2024-02-01T10:48:13.233 DEBUG:boto:Path: /vclfcn-60/?prefix=obj6 2024-02-01T10:48:13.233 DEBUG:boto:Data: 2024-02-01T10:48:13.233 DEBUG:boto:Headers: {} 2024-02-01T10:48:13.233 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:13.233 DEBUG:boto:Port: 8001 2024-02-01T10:48:13.233 DEBUG:boto:Params: {} 2024-02-01T10:48:13.233 DEBUG:boto:Token: None 2024-02-01T10:48:13.234 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:13 GMT /vclfcn-60/ 2024-02-01T10:48:13.234 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RPp22OOQzcmqjI/fhJP5M9F21WY= 2024-02-01T10:48:13.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RPp22OOQzcmqjI/fhJP5M9F21WY=', 'Content-Length': '0'} 2024-02-01T10:48:13.238 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dbffa05366f284c9-0065bb76ed-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:13.239 DEBUG:boto:b'vclfcn-60obj61000falseobj62024-02-01T10:47:30.343Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:13.239 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:13.239 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:13.239 DEBUG:boto:path=/vclfcn-60/?prefix=obj6 2024-02-01T10:48:13.239 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj6 2024-02-01T10:48:13.239 DEBUG:boto:Method: GET 2024-02-01T10:48:13.240 DEBUG:boto:Path: /vclfcn-60/?prefix=obj6 2024-02-01T10:48:13.240 DEBUG:boto:Data: 2024-02-01T10:48:13.240 DEBUG:boto:Headers: {} 2024-02-01T10:48:13.240 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:13.240 DEBUG:boto:Port: 8000 2024-02-01T10:48:13.240 DEBUG:boto:Params: {} 2024-02-01T10:48:13.240 DEBUG:boto:Token: None 2024-02-01T10:48:13.240 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:13 GMT /vclfcn-60/ 2024-02-01T10:48:13.240 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RPp22OOQzcmqjI/fhJP5M9F21WY= 2024-02-01T10:48:13.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RPp22OOQzcmqjI/fhJP5M9F21WY=', 'Content-Length': '0'} 2024-02-01T10:48:13.245 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a4d828cbc35ea399-0065bb76ed-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:13.288 DEBUG:boto:b'vclfcn-60obj61000falseobj62024-02-01T10:47:30.343Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:13.289 DEBUG:boto:path=/vclfcn-60/obj6 2024-02-01T10:48:13.289 DEBUG:boto:auth_path=/vclfcn-60/obj6 2024-02-01T10:48:13.289 DEBUG:boto:Method: HEAD 2024-02-01T10:48:13.289 DEBUG:boto:Path: /vclfcn-60/obj6 2024-02-01T10:48:13.289 DEBUG:boto:Data: 2024-02-01T10:48:13.289 DEBUG:boto:Headers: {} 2024-02-01T10:48:13.289 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:13.289 DEBUG:boto:Port: 8001 2024-02-01T10:48:13.289 DEBUG:boto:Params: {} 2024-02-01T10:48:13.290 DEBUG:boto:Token: None 2024-02-01T10:48:13.290 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:13 GMT /vclfcn-60/obj6 2024-02-01T10:48:13.290 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Hyuz4H7m1HpQuqS/zAEAPvHx/bg= 2024-02-01T10:48:13.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Hyuz4H7m1HpQuqS/zAEAPvHx/bg=', 'Content-Length': '0'} 2024-02-01T10:48:13.293 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c3839065d6ef06df-0065bb76ed-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:13.293 DEBUG:boto:path=/vclfcn-60/obj6 2024-02-01T10:48:13.293 DEBUG:boto:auth_path=/vclfcn-60/obj6 2024-02-01T10:48:13.293 DEBUG:boto:Method: HEAD 2024-02-01T10:48:13.293 DEBUG:boto:Path: /vclfcn-60/obj6 2024-02-01T10:48:13.294 DEBUG:boto:Data: 2024-02-01T10:48:13.294 DEBUG:boto:Headers: {} 2024-02-01T10:48:13.294 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:13.294 DEBUG:boto:Port: 8000 2024-02-01T10:48:13.294 DEBUG:boto:Params: {} 2024-02-01T10:48:13.294 DEBUG:boto:Token: None 2024-02-01T10:48:13.294 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:13 GMT /vclfcn-60/obj6 2024-02-01T10:48:13.294 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Hyuz4H7m1HpQuqS/zAEAPvHx/bg= 2024-02-01T10:48:13.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Hyuz4H7m1HpQuqS/zAEAPvHx/bg=', 'Content-Length': '0'} 2024-02-01T10:48:13.297 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d6e256576f9f28b2-0065bb76ed-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:13.297 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-02-01T10:48:13.298 DEBUG:boto:path=/vclfcn-60/obj6 2024-02-01T10:48:13.298 DEBUG:boto:auth_path=/vclfcn-60/obj6 2024-02-01T10:48:13.298 DEBUG:boto:Method: GET 2024-02-01T10:48:13.298 DEBUG:boto:Path: /vclfcn-60/obj6 2024-02-01T10:48:13.298 DEBUG:boto:Data: 2024-02-01T10:48:13.298 DEBUG:boto:Headers: {} 2024-02-01T10:48:13.298 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:13.298 DEBUG:boto:Port: 8001 2024-02-01T10:48:13.298 DEBUG:boto:Params: {} 2024-02-01T10:48:13.298 DEBUG:boto:Token: None 2024-02-01T10:48:13.299 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:13 GMT /vclfcn-60/obj6 2024-02-01T10:48:13.299 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:oIwNZRBWgPVUBdEzCqLv1iY427s= 2024-02-01T10:48:13.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:oIwNZRBWgPVUBdEzCqLv1iY427s=', 'Content-Length': '0'} 2024-02-01T10:48:13.301 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b86e758cd617ec20-0065bb76ed-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:13.344 DEBUG:boto:path=/vclfcn-60/obj6 2024-02-01T10:48:13.344 DEBUG:boto:auth_path=/vclfcn-60/obj6 2024-02-01T10:48:13.344 DEBUG:boto:Method: GET 2024-02-01T10:48:13.345 DEBUG:boto:Path: /vclfcn-60/obj6 2024-02-01T10:48:13.345 DEBUG:boto:Data: 2024-02-01T10:48:13.345 DEBUG:boto:Headers: {} 2024-02-01T10:48:13.345 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:13.345 DEBUG:boto:Port: 8000 2024-02-01T10:48:13.345 DEBUG:boto:Params: {} 2024-02-01T10:48:13.345 DEBUG:boto:Token: None 2024-02-01T10:48:13.345 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:13 GMT /vclfcn-60/obj6 2024-02-01T10:48:13.345 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:oIwNZRBWgPVUBdEzCqLv1iY427s= 2024-02-01T10:48:13.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:oIwNZRBWgPVUBdEzCqLv1iY427s=', 'Content-Length': '0'} 2024-02-01T10:48:13.349 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a98efd21aee00b4b-0065bb76ed-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:13.392 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:13.392 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:13.392 DEBUG:boto:path=/vclfcn-60/?prefix=obj6 2024-02-01T10:48:13.393 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj6 2024-02-01T10:48:13.393 DEBUG:boto:Method: GET 2024-02-01T10:48:13.393 DEBUG:boto:Path: /vclfcn-60/?prefix=obj6 2024-02-01T10:48:13.393 DEBUG:boto:Data: 2024-02-01T10:48:13.393 DEBUG:boto:Headers: {} 2024-02-01T10:48:13.393 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:13.393 DEBUG:boto:Port: 8001 2024-02-01T10:48:13.393 DEBUG:boto:Params: {} 2024-02-01T10:48:13.393 DEBUG:boto:Token: None 2024-02-01T10:48:13.394 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:13 GMT /vclfcn-60/ 2024-02-01T10:48:13.394 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RPp22OOQzcmqjI/fhJP5M9F21WY= 2024-02-01T10:48:13.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RPp22OOQzcmqjI/fhJP5M9F21WY=', 'Content-Length': '0'} 2024-02-01T10:48:13.399 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028f01d5a346e374d-0065bb76ed-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:13.399 DEBUG:boto:b'vclfcn-60obj61000falseobj62024-02-01T10:47:30.343Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:13.399 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:13.399 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:13.400 DEBUG:boto:path=/vclfcn-60/?prefix=obj6 2024-02-01T10:48:13.400 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj6 2024-02-01T10:48:13.400 DEBUG:boto:Method: GET 2024-02-01T10:48:13.400 DEBUG:boto:Path: /vclfcn-60/?prefix=obj6 2024-02-01T10:48:13.400 DEBUG:boto:Data: 2024-02-01T10:48:13.400 DEBUG:boto:Headers: {} 2024-02-01T10:48:13.400 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:13.400 DEBUG:boto:Port: 8000 2024-02-01T10:48:13.400 DEBUG:boto:Params: {} 2024-02-01T10:48:13.400 DEBUG:boto:Token: None 2024-02-01T10:48:13.400 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:13 GMT /vclfcn-60/ 2024-02-01T10:48:13.401 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RPp22OOQzcmqjI/fhJP5M9F21WY= 2024-02-01T10:48:13.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RPp22OOQzcmqjI/fhJP5M9F21WY=', 'Content-Length': '0'} 2024-02-01T10:48:13.406 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b69be0628cfb627-0065bb76ed-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:13.406 DEBUG:boto:b'vclfcn-60obj61000falseobj62024-02-01T10:47:30.343Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:13.406 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-02-01T10:48:13.407 DEBUG:boto:path=/vclfcn-60/obj7 2024-02-01T10:48:13.407 DEBUG:boto:auth_path=/vclfcn-60/obj7 2024-02-01T10:48:13.407 DEBUG:boto:path=/vclfcn-60/obj7?versionId=null 2024-02-01T10:48:13.407 DEBUG:boto:auth_path=/vclfcn-60/obj7?versionId=null 2024-02-01T10:48:13.407 DEBUG:boto:Method: GET 2024-02-01T10:48:13.407 DEBUG:boto:Path: /vclfcn-60/obj7?versionId=null 2024-02-01T10:48:13.407 DEBUG:boto:Data: 2024-02-01T10:48:13.407 DEBUG:boto:Headers: {} 2024-02-01T10:48:13.407 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:13.407 DEBUG:boto:Port: 8001 2024-02-01T10:48:13.408 DEBUG:boto:Params: {} 2024-02-01T10:48:13.408 DEBUG:boto:Token: None 2024-02-01T10:48:13.408 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:13 GMT /vclfcn-60/obj7?versionId=null 2024-02-01T10:48:13.408 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:6tWopnR5kDIs/Bh2aw2FTtZq6MI= 2024-02-01T10:48:13.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:6tWopnR5kDIs/Bh2aw2FTtZq6MI=', 'Content-Length': '0'} 2024-02-01T10:48:13.411 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e40dd1507e1e6ed4-0065bb76ed-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:13.456 DEBUG:boto:path=/vclfcn-60/obj7 2024-02-01T10:48:13.456 DEBUG:boto:auth_path=/vclfcn-60/obj7 2024-02-01T10:48:13.456 DEBUG:boto:path=/vclfcn-60/obj7?versionId=null 2024-02-01T10:48:13.456 DEBUG:boto:auth_path=/vclfcn-60/obj7?versionId=null 2024-02-01T10:48:13.457 DEBUG:boto:Method: GET 2024-02-01T10:48:13.457 DEBUG:boto:Path: /vclfcn-60/obj7?versionId=null 2024-02-01T10:48:13.457 DEBUG:boto:Data: 2024-02-01T10:48:13.457 DEBUG:boto:Headers: {} 2024-02-01T10:48:13.457 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:13.457 DEBUG:boto:Port: 8000 2024-02-01T10:48:13.457 DEBUG:boto:Params: {} 2024-02-01T10:48:13.457 DEBUG:boto:Token: None 2024-02-01T10:48:13.457 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:13 GMT /vclfcn-60/obj7?versionId=null 2024-02-01T10:48:13.458 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:6tWopnR5kDIs/Bh2aw2FTtZq6MI= 2024-02-01T10:48:13.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:6tWopnR5kDIs/Bh2aw2FTtZq6MI=', 'Content-Length': '0'} 2024-02-01T10:48:13.460 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fe1134cb3f013c11-0065bb76ed-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:13.505 DEBUG:boto:path=/vclfcn-60/obj7 2024-02-01T10:48:13.505 DEBUG:boto:auth_path=/vclfcn-60/obj7 2024-02-01T10:48:13.505 DEBUG:boto:path=/vclfcn-60/obj7?versionId=null 2024-02-01T10:48:13.505 DEBUG:boto:auth_path=/vclfcn-60/obj7?versionId=null 2024-02-01T10:48:13.505 DEBUG:boto:Method: HEAD 2024-02-01T10:48:13.505 DEBUG:boto:Path: /vclfcn-60/obj7?versionId=null 2024-02-01T10:48:13.505 DEBUG:boto:Data: 2024-02-01T10:48:13.506 DEBUG:boto:Headers: {} 2024-02-01T10:48:13.506 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:13.506 DEBUG:boto:Port: 8001 2024-02-01T10:48:13.506 DEBUG:boto:Params: {} 2024-02-01T10:48:13.506 DEBUG:boto:Token: None 2024-02-01T10:48:13.506 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:13 GMT /vclfcn-60/obj7?versionId=null 2024-02-01T10:48:13.506 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ciYN5iYTnCdZMF2HfxuNA/Wvb2E= 2024-02-01T10:48:13.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ciYN5iYTnCdZMF2HfxuNA/Wvb2E=', 'Content-Length': '0'} 2024-02-01T10:48:13.509 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000045a22aae1cb8ae51-0065bb76ed-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:13.509 DEBUG:boto:path=/vclfcn-60/obj7 2024-02-01T10:48:13.510 DEBUG:boto:auth_path=/vclfcn-60/obj7 2024-02-01T10:48:13.510 DEBUG:boto:path=/vclfcn-60/obj7?versionId=null 2024-02-01T10:48:13.510 DEBUG:boto:auth_path=/vclfcn-60/obj7?versionId=null 2024-02-01T10:48:13.510 DEBUG:boto:Method: HEAD 2024-02-01T10:48:13.510 DEBUG:boto:Path: /vclfcn-60/obj7?versionId=null 2024-02-01T10:48:13.510 DEBUG:boto:Data: 2024-02-01T10:48:13.510 DEBUG:boto:Headers: {} 2024-02-01T10:48:13.510 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:13.510 DEBUG:boto:Port: 8000 2024-02-01T10:48:13.510 DEBUG:boto:Params: {} 2024-02-01T10:48:13.510 DEBUG:boto:Token: None 2024-02-01T10:48:13.511 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:13 GMT /vclfcn-60/obj7?versionId=null 2024-02-01T10:48:13.511 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ciYN5iYTnCdZMF2HfxuNA/Wvb2E= 2024-02-01T10:48:13.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ciYN5iYTnCdZMF2HfxuNA/Wvb2E=', 'Content-Length': '0'} 2024-02-01T10:48:13.513 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009943ebae8f3ed4b1-0065bb76ed-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:13.514 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-02-01T10:48:13.514 DEBUG:boto:path=/vclfcn-60/obj7 2024-02-01T10:48:13.514 DEBUG:boto:auth_path=/vclfcn-60/obj7 2024-02-01T10:48:13.514 DEBUG:boto:path=/vclfcn-60/obj7?versionId=null 2024-02-01T10:48:13.514 DEBUG:boto:auth_path=/vclfcn-60/obj7?versionId=null 2024-02-01T10:48:13.514 DEBUG:boto:Method: GET 2024-02-01T10:48:13.514 DEBUG:boto:Path: /vclfcn-60/obj7?versionId=null 2024-02-01T10:48:13.515 DEBUG:boto:Data: 2024-02-01T10:48:13.515 DEBUG:boto:Headers: {} 2024-02-01T10:48:13.515 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:13.515 DEBUG:boto:Port: 8001 2024-02-01T10:48:13.515 DEBUG:boto:Params: {} 2024-02-01T10:48:13.515 DEBUG:boto:Token: None 2024-02-01T10:48:13.515 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:13 GMT /vclfcn-60/obj7?versionId=null 2024-02-01T10:48:13.515 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:6tWopnR5kDIs/Bh2aw2FTtZq6MI= 2024-02-01T10:48:13.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:6tWopnR5kDIs/Bh2aw2FTtZq6MI=', 'Content-Length': '0'} 2024-02-01T10:48:13.518 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000062af7dd6d420d0f8-0065bb76ed-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:13.560 DEBUG:boto:path=/vclfcn-60/obj7 2024-02-01T10:48:13.560 DEBUG:boto:auth_path=/vclfcn-60/obj7 2024-02-01T10:48:13.560 DEBUG:boto:path=/vclfcn-60/obj7?versionId=null 2024-02-01T10:48:13.561 DEBUG:boto:auth_path=/vclfcn-60/obj7?versionId=null 2024-02-01T10:48:13.561 DEBUG:boto:Method: GET 2024-02-01T10:48:13.561 DEBUG:boto:Path: /vclfcn-60/obj7?versionId=null 2024-02-01T10:48:13.561 DEBUG:boto:Data: 2024-02-01T10:48:13.561 DEBUG:boto:Headers: {} 2024-02-01T10:48:13.561 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:13.561 DEBUG:boto:Port: 8000 2024-02-01T10:48:13.561 DEBUG:boto:Params: {} 2024-02-01T10:48:13.561 DEBUG:boto:Token: None 2024-02-01T10:48:13.561 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:13 GMT /vclfcn-60/obj7?versionId=null 2024-02-01T10:48:13.562 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:6tWopnR5kDIs/Bh2aw2FTtZq6MI= 2024-02-01T10:48:13.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:6tWopnR5kDIs/Bh2aw2FTtZq6MI=', 'Content-Length': '0'} 2024-02-01T10:48:13.565 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006d148df00c948f81-0065bb76ed-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:13.612 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:13.613 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:13.613 DEBUG:boto:path=/vclfcn-60/?prefix=obj7 2024-02-01T10:48:13.613 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj7 2024-02-01T10:48:13.613 DEBUG:boto:Method: GET 2024-02-01T10:48:13.613 DEBUG:boto:Path: /vclfcn-60/?prefix=obj7 2024-02-01T10:48:13.613 DEBUG:boto:Data: 2024-02-01T10:48:13.613 DEBUG:boto:Headers: {} 2024-02-01T10:48:13.613 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:13.613 DEBUG:boto:Port: 8001 2024-02-01T10:48:13.614 DEBUG:boto:Params: {} 2024-02-01T10:48:13.614 DEBUG:boto:Token: None 2024-02-01T10:48:13.614 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:13 GMT /vclfcn-60/ 2024-02-01T10:48:13.614 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RPp22OOQzcmqjI/fhJP5M9F21WY= 2024-02-01T10:48:13.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RPp22OOQzcmqjI/fhJP5M9F21WY=', 'Content-Length': '0'} 2024-02-01T10:48:13.619 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009291ea6b89252775-0065bb76ed-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:13.619 DEBUG:boto:b'vclfcn-60obj71000falseobj72024-02-01T10:47:30.358Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:13.620 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:13.620 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:13.620 DEBUG:boto:path=/vclfcn-60/?prefix=obj7 2024-02-01T10:48:13.620 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj7 2024-02-01T10:48:13.620 DEBUG:boto:Method: GET 2024-02-01T10:48:13.621 DEBUG:boto:Path: /vclfcn-60/?prefix=obj7 2024-02-01T10:48:13.621 DEBUG:boto:Data: 2024-02-01T10:48:13.621 DEBUG:boto:Headers: {} 2024-02-01T10:48:13.621 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:13.621 DEBUG:boto:Port: 8000 2024-02-01T10:48:13.621 DEBUG:boto:Params: {} 2024-02-01T10:48:13.621 DEBUG:boto:Token: None 2024-02-01T10:48:13.621 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:13 GMT /vclfcn-60/ 2024-02-01T10:48:13.621 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RPp22OOQzcmqjI/fhJP5M9F21WY= 2024-02-01T10:48:13.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RPp22OOQzcmqjI/fhJP5M9F21WY=', 'Content-Length': '0'} 2024-02-01T10:48:13.626 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee291f6862f95422-0065bb76ed-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:13.627 DEBUG:boto:b'vclfcn-60obj71000falseobj72024-02-01T10:47:30.358Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:13.627 DEBUG:boto:path=/vclfcn-60/obj7 2024-02-01T10:48:13.627 DEBUG:boto:auth_path=/vclfcn-60/obj7 2024-02-01T10:48:13.627 DEBUG:boto:Method: HEAD 2024-02-01T10:48:13.627 DEBUG:boto:Path: /vclfcn-60/obj7 2024-02-01T10:48:13.627 DEBUG:boto:Data: 2024-02-01T10:48:13.628 DEBUG:boto:Headers: {} 2024-02-01T10:48:13.628 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:13.628 DEBUG:boto:Port: 8001 2024-02-01T10:48:13.628 DEBUG:boto:Params: {} 2024-02-01T10:48:13.628 DEBUG:boto:Token: None 2024-02-01T10:48:13.628 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:13 GMT /vclfcn-60/obj7 2024-02-01T10:48:13.628 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Z3ijVoVp+eVlyVVOUXiOm+CJqGA= 2024-02-01T10:48:13.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Z3ijVoVp+eVlyVVOUXiOm+CJqGA=', 'Content-Length': '0'} 2024-02-01T10:48:13.631 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a98248d5aaf10946-0065bb76ed-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:13.631 DEBUG:boto:path=/vclfcn-60/obj7 2024-02-01T10:48:13.632 DEBUG:boto:auth_path=/vclfcn-60/obj7 2024-02-01T10:48:13.632 DEBUG:boto:Method: HEAD 2024-02-01T10:48:13.632 DEBUG:boto:Path: /vclfcn-60/obj7 2024-02-01T10:48:13.632 DEBUG:boto:Data: 2024-02-01T10:48:13.632 DEBUG:boto:Headers: {} 2024-02-01T10:48:13.632 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:13.632 DEBUG:boto:Port: 8000 2024-02-01T10:48:13.632 DEBUG:boto:Params: {} 2024-02-01T10:48:13.632 DEBUG:boto:Token: None 2024-02-01T10:48:13.633 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:13 GMT /vclfcn-60/obj7 2024-02-01T10:48:13.633 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Z3ijVoVp+eVlyVVOUXiOm+CJqGA= 2024-02-01T10:48:13.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Z3ijVoVp+eVlyVVOUXiOm+CJqGA=', 'Content-Length': '0'} 2024-02-01T10:48:13.635 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000879af0be7bfd6934-0065bb76ed-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:13.636 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-02-01T10:48:13.636 DEBUG:boto:path=/vclfcn-60/obj7 2024-02-01T10:48:13.636 DEBUG:boto:auth_path=/vclfcn-60/obj7 2024-02-01T10:48:13.636 DEBUG:boto:Method: GET 2024-02-01T10:48:13.636 DEBUG:boto:Path: /vclfcn-60/obj7 2024-02-01T10:48:13.636 DEBUG:boto:Data: 2024-02-01T10:48:13.636 DEBUG:boto:Headers: {} 2024-02-01T10:48:13.636 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:13.637 DEBUG:boto:Port: 8001 2024-02-01T10:48:13.637 DEBUG:boto:Params: {} 2024-02-01T10:48:13.637 DEBUG:boto:Token: None 2024-02-01T10:48:13.637 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:13 GMT /vclfcn-60/obj7 2024-02-01T10:48:13.637 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:u3a1FlBsZEwi7INZpLA95GyesXs= 2024-02-01T10:48:13.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:u3a1FlBsZEwi7INZpLA95GyesXs=', 'Content-Length': '0'} 2024-02-01T10:48:13.640 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004da2fa99d75a94bf-0065bb76ed-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:13.680 DEBUG:boto:path=/vclfcn-60/obj7 2024-02-01T10:48:13.680 DEBUG:boto:auth_path=/vclfcn-60/obj7 2024-02-01T10:48:13.680 DEBUG:boto:Method: GET 2024-02-01T10:48:13.680 DEBUG:boto:Path: /vclfcn-60/obj7 2024-02-01T10:48:13.681 DEBUG:boto:Data: 2024-02-01T10:48:13.681 DEBUG:boto:Headers: {} 2024-02-01T10:48:13.681 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:13.681 DEBUG:boto:Port: 8000 2024-02-01T10:48:13.681 DEBUG:boto:Params: {} 2024-02-01T10:48:13.681 DEBUG:boto:Token: None 2024-02-01T10:48:13.681 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:13 GMT /vclfcn-60/obj7 2024-02-01T10:48:13.681 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:u3a1FlBsZEwi7INZpLA95GyesXs= 2024-02-01T10:48:13.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:u3a1FlBsZEwi7INZpLA95GyesXs=', 'Content-Length': '0'} 2024-02-01T10:48:13.685 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f9c919a01c425c3f-0065bb76ed-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:13.732 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:13.733 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:13.733 DEBUG:boto:path=/vclfcn-60/?prefix=obj7 2024-02-01T10:48:13.733 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj7 2024-02-01T10:48:13.734 DEBUG:boto:Method: GET 2024-02-01T10:48:13.734 DEBUG:boto:Path: /vclfcn-60/?prefix=obj7 2024-02-01T10:48:13.734 DEBUG:boto:Data: 2024-02-01T10:48:13.734 DEBUG:boto:Headers: {} 2024-02-01T10:48:13.734 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:13.734 DEBUG:boto:Port: 8001 2024-02-01T10:48:13.734 DEBUG:boto:Params: {} 2024-02-01T10:48:13.734 DEBUG:boto:Token: None 2024-02-01T10:48:13.735 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:13 GMT /vclfcn-60/ 2024-02-01T10:48:13.735 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RPp22OOQzcmqjI/fhJP5M9F21WY= 2024-02-01T10:48:13.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RPp22OOQzcmqjI/fhJP5M9F21WY=', 'Content-Length': '0'} 2024-02-01T10:48:13.746 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bbe5465e234b80bd-0065bb76ed-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:13.796 DEBUG:boto:b'vclfcn-60obj71000falseobj72024-02-01T10:47:30.358Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:13.796 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:13.796 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:13.797 DEBUG:boto:path=/vclfcn-60/?prefix=obj7 2024-02-01T10:48:13.797 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj7 2024-02-01T10:48:13.797 DEBUG:boto:Method: GET 2024-02-01T10:48:13.798 DEBUG:boto:Path: /vclfcn-60/?prefix=obj7 2024-02-01T10:48:13.798 DEBUG:boto:Data: 2024-02-01T10:48:13.798 DEBUG:boto:Headers: {} 2024-02-01T10:48:13.798 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:13.798 DEBUG:boto:Port: 8000 2024-02-01T10:48:13.798 DEBUG:boto:Params: {} 2024-02-01T10:48:13.798 DEBUG:boto:Token: None 2024-02-01T10:48:13.798 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:13 GMT /vclfcn-60/ 2024-02-01T10:48:13.799 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RPp22OOQzcmqjI/fhJP5M9F21WY= 2024-02-01T10:48:13.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RPp22OOQzcmqjI/fhJP5M9F21WY=', 'Content-Length': '0'} 2024-02-01T10:48:13.805 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035676bcb5c0f0566-0065bb76ed-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:13.806 DEBUG:boto:b'vclfcn-60obj71000falseobj72024-02-01T10:47:30.358Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:13.806 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-02-01T10:48:13.807 DEBUG:boto:path=/vclfcn-60/obj8 2024-02-01T10:48:13.807 DEBUG:boto:auth_path=/vclfcn-60/obj8 2024-02-01T10:48:13.807 DEBUG:boto:path=/vclfcn-60/obj8?versionId=null 2024-02-01T10:48:13.807 DEBUG:boto:auth_path=/vclfcn-60/obj8?versionId=null 2024-02-01T10:48:13.807 DEBUG:boto:Method: GET 2024-02-01T10:48:13.807 DEBUG:boto:Path: /vclfcn-60/obj8?versionId=null 2024-02-01T10:48:13.807 DEBUG:boto:Data: 2024-02-01T10:48:13.807 DEBUG:boto:Headers: {} 2024-02-01T10:48:13.807 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:13.808 DEBUG:boto:Port: 8001 2024-02-01T10:48:13.808 DEBUG:boto:Params: {} 2024-02-01T10:48:13.808 DEBUG:boto:Token: None 2024-02-01T10:48:13.808 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:13 GMT /vclfcn-60/obj8?versionId=null 2024-02-01T10:48:13.808 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZN2ZjclFpViI6oQBOqOCMQJICVc= 2024-02-01T10:48:13.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZN2ZjclFpViI6oQBOqOCMQJICVc=', 'Content-Length': '0'} 2024-02-01T10:48:13.811 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000077b7a99896e1d29f-0065bb76ed-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:13.856 DEBUG:boto:path=/vclfcn-60/obj8 2024-02-01T10:48:13.856 DEBUG:boto:auth_path=/vclfcn-60/obj8 2024-02-01T10:48:13.857 DEBUG:boto:path=/vclfcn-60/obj8?versionId=null 2024-02-01T10:48:13.857 DEBUG:boto:auth_path=/vclfcn-60/obj8?versionId=null 2024-02-01T10:48:13.857 DEBUG:boto:Method: GET 2024-02-01T10:48:13.857 DEBUG:boto:Path: /vclfcn-60/obj8?versionId=null 2024-02-01T10:48:13.857 DEBUG:boto:Data: 2024-02-01T10:48:13.857 DEBUG:boto:Headers: {} 2024-02-01T10:48:13.857 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:13.857 DEBUG:boto:Port: 8000 2024-02-01T10:48:13.857 DEBUG:boto:Params: {} 2024-02-01T10:48:13.857 DEBUG:boto:Token: None 2024-02-01T10:48:13.858 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:13 GMT /vclfcn-60/obj8?versionId=null 2024-02-01T10:48:13.858 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZN2ZjclFpViI6oQBOqOCMQJICVc= 2024-02-01T10:48:13.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZN2ZjclFpViI6oQBOqOCMQJICVc=', 'Content-Length': '0'} 2024-02-01T10:48:13.861 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001ed48be7ad088a56-0065bb76ed-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:13.861 DEBUG:boto:path=/vclfcn-60/obj8 2024-02-01T10:48:13.862 DEBUG:boto:auth_path=/vclfcn-60/obj8 2024-02-01T10:48:13.862 DEBUG:boto:path=/vclfcn-60/obj8?versionId=null 2024-02-01T10:48:13.862 DEBUG:boto:auth_path=/vclfcn-60/obj8?versionId=null 2024-02-01T10:48:13.862 DEBUG:boto:Method: HEAD 2024-02-01T10:48:13.862 DEBUG:boto:Path: /vclfcn-60/obj8?versionId=null 2024-02-01T10:48:13.862 DEBUG:boto:Data: 2024-02-01T10:48:13.862 DEBUG:boto:Headers: {} 2024-02-01T10:48:13.862 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:13.862 DEBUG:boto:Port: 8001 2024-02-01T10:48:13.862 DEBUG:boto:Params: {} 2024-02-01T10:48:13.863 DEBUG:boto:Token: None 2024-02-01T10:48:13.863 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:13 GMT /vclfcn-60/obj8?versionId=null 2024-02-01T10:48:13.863 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0vDXyorFrH/IYlzgO5QnOYp120M= 2024-02-01T10:48:13.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0vDXyorFrH/IYlzgO5QnOYp120M=', 'Content-Length': '0'} 2024-02-01T10:48:13.866 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000051e85ae293ad8795-0065bb76ed-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:13.866 DEBUG:boto:path=/vclfcn-60/obj8 2024-02-01T10:48:13.866 DEBUG:boto:auth_path=/vclfcn-60/obj8 2024-02-01T10:48:13.866 DEBUG:boto:path=/vclfcn-60/obj8?versionId=null 2024-02-01T10:48:13.866 DEBUG:boto:auth_path=/vclfcn-60/obj8?versionId=null 2024-02-01T10:48:13.866 DEBUG:boto:Method: HEAD 2024-02-01T10:48:13.866 DEBUG:boto:Path: /vclfcn-60/obj8?versionId=null 2024-02-01T10:48:13.867 DEBUG:boto:Data: 2024-02-01T10:48:13.867 DEBUG:boto:Headers: {} 2024-02-01T10:48:13.867 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:13.867 DEBUG:boto:Port: 8000 2024-02-01T10:48:13.867 DEBUG:boto:Params: {} 2024-02-01T10:48:13.867 DEBUG:boto:Token: None 2024-02-01T10:48:13.867 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:13 GMT /vclfcn-60/obj8?versionId=null 2024-02-01T10:48:13.867 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0vDXyorFrH/IYlzgO5QnOYp120M= 2024-02-01T10:48:13.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0vDXyorFrH/IYlzgO5QnOYp120M=', 'Content-Length': '0'} 2024-02-01T10:48:13.870 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000188a215346fbb775-0065bb76ed-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:13.870 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-02-01T10:48:13.871 DEBUG:boto:path=/vclfcn-60/obj8 2024-02-01T10:48:13.871 DEBUG:boto:auth_path=/vclfcn-60/obj8 2024-02-01T10:48:13.871 DEBUG:boto:path=/vclfcn-60/obj8?versionId=null 2024-02-01T10:48:13.871 DEBUG:boto:auth_path=/vclfcn-60/obj8?versionId=null 2024-02-01T10:48:13.871 DEBUG:boto:Method: GET 2024-02-01T10:48:13.871 DEBUG:boto:Path: /vclfcn-60/obj8?versionId=null 2024-02-01T10:48:13.871 DEBUG:boto:Data: 2024-02-01T10:48:13.871 DEBUG:boto:Headers: {} 2024-02-01T10:48:13.871 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:13.872 DEBUG:boto:Port: 8001 2024-02-01T10:48:13.872 DEBUG:boto:Params: {} 2024-02-01T10:48:13.872 DEBUG:boto:Token: None 2024-02-01T10:48:13.872 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:13 GMT /vclfcn-60/obj8?versionId=null 2024-02-01T10:48:13.872 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZN2ZjclFpViI6oQBOqOCMQJICVc= 2024-02-01T10:48:13.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZN2ZjclFpViI6oQBOqOCMQJICVc=', 'Content-Length': '0'} 2024-02-01T10:48:13.875 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001185dc87108b27a5-0065bb76ed-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:13.916 DEBUG:boto:path=/vclfcn-60/obj8 2024-02-01T10:48:13.916 DEBUG:boto:auth_path=/vclfcn-60/obj8 2024-02-01T10:48:13.916 DEBUG:boto:path=/vclfcn-60/obj8?versionId=null 2024-02-01T10:48:13.916 DEBUG:boto:auth_path=/vclfcn-60/obj8?versionId=null 2024-02-01T10:48:13.917 DEBUG:boto:Method: GET 2024-02-01T10:48:13.917 DEBUG:boto:Path: /vclfcn-60/obj8?versionId=null 2024-02-01T10:48:13.917 DEBUG:boto:Data: 2024-02-01T10:48:13.917 DEBUG:boto:Headers: {} 2024-02-01T10:48:13.917 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:13.917 DEBUG:boto:Port: 8000 2024-02-01T10:48:13.917 DEBUG:boto:Params: {} 2024-02-01T10:48:13.917 DEBUG:boto:Token: None 2024-02-01T10:48:13.917 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:13 GMT /vclfcn-60/obj8?versionId=null 2024-02-01T10:48:13.917 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZN2ZjclFpViI6oQBOqOCMQJICVc= 2024-02-01T10:48:13.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZN2ZjclFpViI6oQBOqOCMQJICVc=', 'Content-Length': '0'} 2024-02-01T10:48:13.920 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009fc294fb407e329d-0065bb76ed-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:13.968 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:13.968 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:13.968 DEBUG:boto:path=/vclfcn-60/?prefix=obj8 2024-02-01T10:48:13.969 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj8 2024-02-01T10:48:13.969 DEBUG:boto:Method: GET 2024-02-01T10:48:13.969 DEBUG:boto:Path: /vclfcn-60/?prefix=obj8 2024-02-01T10:48:13.969 DEBUG:boto:Data: 2024-02-01T10:48:13.969 DEBUG:boto:Headers: {} 2024-02-01T10:48:13.969 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:13.969 DEBUG:boto:Port: 8001 2024-02-01T10:48:13.969 DEBUG:boto:Params: {} 2024-02-01T10:48:13.969 DEBUG:boto:Token: None 2024-02-01T10:48:13.969 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:13 GMT /vclfcn-60/ 2024-02-01T10:48:13.970 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RPp22OOQzcmqjI/fhJP5M9F21WY= 2024-02-01T10:48:13.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RPp22OOQzcmqjI/fhJP5M9F21WY=', 'Content-Length': '0'} 2024-02-01T10:48:13.974 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b00ab563dded5cab-0065bb76ed-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:13.974 DEBUG:boto:b'vclfcn-60obj81000falseobj82024-02-01T10:47:30.408Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:13.975 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:13.975 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:13.975 DEBUG:boto:path=/vclfcn-60/?prefix=obj8 2024-02-01T10:48:13.975 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj8 2024-02-01T10:48:13.975 DEBUG:boto:Method: GET 2024-02-01T10:48:13.975 DEBUG:boto:Path: /vclfcn-60/?prefix=obj8 2024-02-01T10:48:13.975 DEBUG:boto:Data: 2024-02-01T10:48:13.975 DEBUG:boto:Headers: {} 2024-02-01T10:48:13.975 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:13.976 DEBUG:boto:Port: 8000 2024-02-01T10:48:13.976 DEBUG:boto:Params: {} 2024-02-01T10:48:13.976 DEBUG:boto:Token: None 2024-02-01T10:48:13.976 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:13 GMT /vclfcn-60/ 2024-02-01T10:48:13.976 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RPp22OOQzcmqjI/fhJP5M9F21WY= 2024-02-01T10:48:13.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RPp22OOQzcmqjI/fhJP5M9F21WY=', 'Content-Length': '0'} 2024-02-01T10:48:13.981 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003545b8df93c2890b-0065bb76ed-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:13.982 DEBUG:boto:b'vclfcn-60obj81000falseobj82024-02-01T10:47:30.408Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:13.982 DEBUG:boto:path=/vclfcn-60/obj8 2024-02-01T10:48:13.982 DEBUG:boto:auth_path=/vclfcn-60/obj8 2024-02-01T10:48:13.982 DEBUG:boto:Method: HEAD 2024-02-01T10:48:13.982 DEBUG:boto:Path: /vclfcn-60/obj8 2024-02-01T10:48:13.983 DEBUG:boto:Data: 2024-02-01T10:48:13.983 DEBUG:boto:Headers: {} 2024-02-01T10:48:13.983 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:13.983 DEBUG:boto:Port: 8001 2024-02-01T10:48:13.983 DEBUG:boto:Params: {} 2024-02-01T10:48:13.983 DEBUG:boto:Token: None 2024-02-01T10:48:13.983 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:13 GMT /vclfcn-60/obj8 2024-02-01T10:48:13.983 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:6Y+bkZkMu35myxTR+2BV1biW7+s= 2024-02-01T10:48:13.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:6Y+bkZkMu35myxTR+2BV1biW7+s=', 'Content-Length': '0'} 2024-02-01T10:48:13.986 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a974bdcbb8e0ba54-0065bb76ed-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:13.986 DEBUG:boto:path=/vclfcn-60/obj8 2024-02-01T10:48:13.987 DEBUG:boto:auth_path=/vclfcn-60/obj8 2024-02-01T10:48:13.987 DEBUG:boto:Method: HEAD 2024-02-01T10:48:13.987 DEBUG:boto:Path: /vclfcn-60/obj8 2024-02-01T10:48:13.987 DEBUG:boto:Data: 2024-02-01T10:48:13.987 DEBUG:boto:Headers: {} 2024-02-01T10:48:13.987 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:13.987 DEBUG:boto:Port: 8000 2024-02-01T10:48:13.987 DEBUG:boto:Params: {} 2024-02-01T10:48:13.988 DEBUG:boto:Token: None 2024-02-01T10:48:13.988 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:13 GMT /vclfcn-60/obj8 2024-02-01T10:48:13.988 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:6Y+bkZkMu35myxTR+2BV1biW7+s= 2024-02-01T10:48:13.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:6Y+bkZkMu35myxTR+2BV1biW7+s=', 'Content-Length': '0'} 2024-02-01T10:48:13.990 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004a4f69719deae220-0065bb76ed-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:13.991 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-02-01T10:48:13.991 DEBUG:boto:path=/vclfcn-60/obj8 2024-02-01T10:48:13.991 DEBUG:boto:auth_path=/vclfcn-60/obj8 2024-02-01T10:48:13.991 DEBUG:boto:Method: GET 2024-02-01T10:48:13.991 DEBUG:boto:Path: /vclfcn-60/obj8 2024-02-01T10:48:13.991 DEBUG:boto:Data: 2024-02-01T10:48:13.991 DEBUG:boto:Headers: {} 2024-02-01T10:48:13.991 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:13.992 DEBUG:boto:Port: 8001 2024-02-01T10:48:13.992 DEBUG:boto:Params: {} 2024-02-01T10:48:13.992 DEBUG:boto:Token: None 2024-02-01T10:48:13.992 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:13 GMT /vclfcn-60/obj8 2024-02-01T10:48:13.992 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:bjOzEbjzS9PMmdUgxAR92q/D7m0= 2024-02-01T10:48:13.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:bjOzEbjzS9PMmdUgxAR92q/D7m0=', 'Content-Length': '0'} 2024-02-01T10:48:13.995 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dc4b8ae755ee0fdf-0065bb76ed-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:14.040 DEBUG:boto:path=/vclfcn-60/obj8 2024-02-01T10:48:14.040 DEBUG:boto:auth_path=/vclfcn-60/obj8 2024-02-01T10:48:14.040 DEBUG:boto:Method: GET 2024-02-01T10:48:14.040 DEBUG:boto:Path: /vclfcn-60/obj8 2024-02-01T10:48:14.041 DEBUG:boto:Data: 2024-02-01T10:48:14.041 DEBUG:boto:Headers: {} 2024-02-01T10:48:14.041 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:14.041 DEBUG:boto:Port: 8000 2024-02-01T10:48:14.041 DEBUG:boto:Params: {} 2024-02-01T10:48:14.041 DEBUG:boto:Token: None 2024-02-01T10:48:14.041 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:14 GMT /vclfcn-60/obj8 2024-02-01T10:48:14.041 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ycdDk5DyZV4lxbrIzNz1igQZwYI= 2024-02-01T10:48:14.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ycdDk5DyZV4lxbrIzNz1igQZwYI=', 'Content-Length': '0'} 2024-02-01T10:48:14.044 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d183aebcf27ea107-0065bb76ee-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:14.088 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:14.088 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:14.088 DEBUG:boto:path=/vclfcn-60/?prefix=obj8 2024-02-01T10:48:14.088 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj8 2024-02-01T10:48:14.089 DEBUG:boto:Method: GET 2024-02-01T10:48:14.089 DEBUG:boto:Path: /vclfcn-60/?prefix=obj8 2024-02-01T10:48:14.089 DEBUG:boto:Data: 2024-02-01T10:48:14.089 DEBUG:boto:Headers: {} 2024-02-01T10:48:14.089 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:14.089 DEBUG:boto:Port: 8001 2024-02-01T10:48:14.089 DEBUG:boto:Params: {} 2024-02-01T10:48:14.089 DEBUG:boto:Token: None 2024-02-01T10:48:14.089 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:14 GMT /vclfcn-60/ 2024-02-01T10:48:14.090 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8rc2OZObw/0ljsgnZFAojGvH+Ro= 2024-02-01T10:48:14.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8rc2OZObw/0ljsgnZFAojGvH+Ro=', 'Content-Length': '0'} 2024-02-01T10:48:14.094 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097c6f64daf8f83c6-0065bb76ee-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:14.094 DEBUG:boto:b'vclfcn-60obj81000falseobj82024-02-01T10:47:30.408Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:14.094 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:14.095 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:14.095 DEBUG:boto:path=/vclfcn-60/?prefix=obj8 2024-02-01T10:48:14.095 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj8 2024-02-01T10:48:14.095 DEBUG:boto:Method: GET 2024-02-01T10:48:14.095 DEBUG:boto:Path: /vclfcn-60/?prefix=obj8 2024-02-01T10:48:14.095 DEBUG:boto:Data: 2024-02-01T10:48:14.095 DEBUG:boto:Headers: {} 2024-02-01T10:48:14.095 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:14.095 DEBUG:boto:Port: 8000 2024-02-01T10:48:14.095 DEBUG:boto:Params: {} 2024-02-01T10:48:14.095 DEBUG:boto:Token: None 2024-02-01T10:48:14.096 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:14 GMT /vclfcn-60/ 2024-02-01T10:48:14.096 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8rc2OZObw/0ljsgnZFAojGvH+Ro= 2024-02-01T10:48:14.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8rc2OZObw/0ljsgnZFAojGvH+Ro=', 'Content-Length': '0'} 2024-02-01T10:48:14.101 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b56e457e5f72995b-0065bb76ee-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:14.144 DEBUG:boto:b'vclfcn-60obj81000falseobj82024-02-01T10:47:30.408Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:14.145 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-60 zones={test-zone3, test-zone1} 2024-02-01T10:48:14.145 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-60 zones={test-zone3, test-zone2} 2024-02-01T10:48:14.145 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:14.145 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:14.145 DEBUG:boto:Method: HEAD 2024-02-01T10:48:14.145 DEBUG:boto:Path: /vclfcn-60/ 2024-02-01T10:48:14.145 DEBUG:boto:Data: 2024-02-01T10:48:14.145 DEBUG:boto:Headers: {} 2024-02-01T10:48:14.145 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:14.145 DEBUG:boto:Port: 8001 2024-02-01T10:48:14.145 DEBUG:boto:Params: {} 2024-02-01T10:48:14.146 DEBUG:boto:Token: None 2024-02-01T10:48:14.146 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:14 GMT /vclfcn-60/ 2024-02-01T10:48:14.146 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wodKeG7+3PrlE1GmhkqJiL1cD2E= 2024-02-01T10:48:14.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wodKeG7+3PrlE1GmhkqJiL1cD2E=', 'Content-Length': '0'} 2024-02-01T10:48:14.150 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008183d3b965752edd-0065bb76ee-5762-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:14.150 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:14.151 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:14.151 DEBUG:boto:Method: HEAD 2024-02-01T10:48:14.151 DEBUG:boto:Path: /vclfcn-60/ 2024-02-01T10:48:14.151 DEBUG:boto:Data: 2024-02-01T10:48:14.151 DEBUG:boto:Headers: {} 2024-02-01T10:48:14.151 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:14.151 DEBUG:boto:Port: 8000 2024-02-01T10:48:14.151 DEBUG:boto:Params: {} 2024-02-01T10:48:14.151 DEBUG:boto:Token: None 2024-02-01T10:48:14.152 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:14 GMT /vclfcn-60/ 2024-02-01T10:48:14.152 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wodKeG7+3PrlE1GmhkqJiL1cD2E= 2024-02-01T10:48:14.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wodKeG7+3PrlE1GmhkqJiL1cD2E=', 'Content-Length': '0'} 2024-02-01T10:48:14.156 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005e534928e2848833-0065bb76ee-5350-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:14.156 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:48:14.157 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:14.157 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:14.157 DEBUG:boto:path=/vclfcn-60/?versions&max-keys=999 2024-02-01T10:48:14.157 DEBUG:boto:auth_path=/vclfcn-60/?versions&max-keys=999 2024-02-01T10:48:14.157 DEBUG:boto:Method: GET 2024-02-01T10:48:14.157 DEBUG:boto:Path: /vclfcn-60/?versions&max-keys=999 2024-02-01T10:48:14.157 DEBUG:boto:Data: 2024-02-01T10:48:14.157 DEBUG:boto:Headers: {} 2024-02-01T10:48:14.157 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:14.158 DEBUG:boto:Port: 8001 2024-02-01T10:48:14.158 DEBUG:boto:Params: {} 2024-02-01T10:48:14.158 DEBUG:boto:Token: None 2024-02-01T10:48:14.158 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:14 GMT /vclfcn-60/?versions 2024-02-01T10:48:14.158 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cWApfdvVpc/nxtuZkELdorRkGJ4= 2024-02-01T10:48:14.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cWApfdvVpc/nxtuZkELdorRkGJ4=', 'Content-Length': '0'} 2024-02-01T10:48:14.163 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000006ee2cdae1cd0f6a-0065bb76ee-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:14.163 DEBUG:boto:b'vclfcn-60999falseobj1nulltrue2024-02-01T10:46:46.243Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:46:46.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:46:46.285Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:46:46.336Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-02-01T10:47:30.291Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-02-01T10:47:30.343Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-02-01T10:47:30.358Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-02-01T10:47:30.408Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:14.164 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-02-01T10:48:14.164 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-02-01T10:48:14.164 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-02-01T10:48:14.164 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-02-01T10:48:14.164 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-02-01T10:48:14.164 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-02-01T10:48:14.164 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-02-01T10:48:14.164 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-02-01T10:48:14.165 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:48:14.165 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:14.165 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:14.165 DEBUG:boto:path=/vclfcn-60/?versions&max-keys=999 2024-02-01T10:48:14.165 DEBUG:boto:auth_path=/vclfcn-60/?versions&max-keys=999 2024-02-01T10:48:14.165 DEBUG:boto:Method: GET 2024-02-01T10:48:14.165 DEBUG:boto:Path: /vclfcn-60/?versions&max-keys=999 2024-02-01T10:48:14.165 DEBUG:boto:Data: 2024-02-01T10:48:14.165 DEBUG:boto:Headers: {} 2024-02-01T10:48:14.165 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:14.166 DEBUG:boto:Port: 8000 2024-02-01T10:48:14.166 DEBUG:boto:Params: {} 2024-02-01T10:48:14.166 DEBUG:boto:Token: None 2024-02-01T10:48:14.166 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:14 GMT /vclfcn-60/?versions 2024-02-01T10:48:14.166 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cWApfdvVpc/nxtuZkELdorRkGJ4= 2024-02-01T10:48:14.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cWApfdvVpc/nxtuZkELdorRkGJ4=', 'Content-Length': '0'} 2024-02-01T10:48:14.171 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed968c99b77af6b5-0065bb76ee-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:14.171 DEBUG:boto:b'vclfcn-60999falseobj1nulltrue2024-02-01T10:46:46.243Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:46:46.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:46:46.285Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:46:46.336Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-02-01T10:47:30.291Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-02-01T10:47:30.343Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-02-01T10:47:30.358Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-02-01T10:47:30.408Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:14.172 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-02-01T10:48:14.172 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-02-01T10:48:14.172 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-02-01T10:48:14.172 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-02-01T10:48:14.172 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-02-01T10:48:14.172 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-02-01T10:48:14.172 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-02-01T10:48:14.172 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-02-01T10:48:14.172 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:14.172 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:14.173 DEBUG:boto:path=/vclfcn-60/?versions&max-keys=999 2024-02-01T10:48:14.173 DEBUG:boto:auth_path=/vclfcn-60/?versions&max-keys=999 2024-02-01T10:48:14.173 DEBUG:boto:Method: GET 2024-02-01T10:48:14.173 DEBUG:boto:Path: /vclfcn-60/?versions&max-keys=999 2024-02-01T10:48:14.173 DEBUG:boto:Data: 2024-02-01T10:48:14.173 DEBUG:boto:Headers: {} 2024-02-01T10:48:14.173 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:14.173 DEBUG:boto:Port: 8001 2024-02-01T10:48:14.173 DEBUG:boto:Params: {} 2024-02-01T10:48:14.173 DEBUG:boto:Token: None 2024-02-01T10:48:14.174 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:14 GMT /vclfcn-60/?versions 2024-02-01T10:48:14.174 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cWApfdvVpc/nxtuZkELdorRkGJ4= 2024-02-01T10:48:14.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cWApfdvVpc/nxtuZkELdorRkGJ4=', 'Content-Length': '0'} 2024-02-01T10:48:14.178 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000328a0fb341f21b86-0065bb76ee-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:14.179 DEBUG:boto:b'vclfcn-60999falseobj1nulltrue2024-02-01T10:46:46.243Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:46:46.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:46:46.285Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:46:46.336Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-02-01T10:47:30.291Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-02-01T10:47:30.343Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-02-01T10:47:30.358Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-02-01T10:47:30.408Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:14.179 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:14.179 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:14.179 DEBUG:boto:path=/vclfcn-60/?versions&max-keys=999 2024-02-01T10:48:14.180 DEBUG:boto:auth_path=/vclfcn-60/?versions&max-keys=999 2024-02-01T10:48:14.180 DEBUG:boto:Method: GET 2024-02-01T10:48:14.180 DEBUG:boto:Path: /vclfcn-60/?versions&max-keys=999 2024-02-01T10:48:14.180 DEBUG:boto:Data: 2024-02-01T10:48:14.180 DEBUG:boto:Headers: {} 2024-02-01T10:48:14.180 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:14.180 DEBUG:boto:Port: 8000 2024-02-01T10:48:14.180 DEBUG:boto:Params: {} 2024-02-01T10:48:14.180 DEBUG:boto:Token: None 2024-02-01T10:48:14.180 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:14 GMT /vclfcn-60/?versions 2024-02-01T10:48:14.181 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cWApfdvVpc/nxtuZkELdorRkGJ4= 2024-02-01T10:48:14.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cWApfdvVpc/nxtuZkELdorRkGJ4=', 'Content-Length': '0'} 2024-02-01T10:48:14.186 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4bdca615d6f2b44-0065bb76ee-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:14.186 DEBUG:boto:b'vclfcn-60999falseobj1nulltrue2024-02-01T10:46:46.243Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:46:46.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:46:46.285Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:46:46.336Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-02-01T10:47:30.291Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-02-01T10:47:30.343Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-02-01T10:47:30.358Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-02-01T10:47:30.408Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:14.186 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:48:14.187 DEBUG:boto:path=/vclfcn-60/obj1 2024-02-01T10:48:14.187 DEBUG:boto:auth_path=/vclfcn-60/obj1 2024-02-01T10:48:14.187 DEBUG:boto:path=/vclfcn-60/obj1?versionId=null 2024-02-01T10:48:14.187 DEBUG:boto:auth_path=/vclfcn-60/obj1?versionId=null 2024-02-01T10:48:14.187 DEBUG:boto:Method: GET 2024-02-01T10:48:14.187 DEBUG:boto:Path: /vclfcn-60/obj1?versionId=null 2024-02-01T10:48:14.187 DEBUG:boto:Data: 2024-02-01T10:48:14.187 DEBUG:boto:Headers: {} 2024-02-01T10:48:14.187 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:14.187 DEBUG:boto:Port: 8001 2024-02-01T10:48:14.187 DEBUG:boto:Params: {} 2024-02-01T10:48:14.188 DEBUG:boto:Token: None 2024-02-01T10:48:14.188 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:14 GMT /vclfcn-60/obj1?versionId=null 2024-02-01T10:48:14.188 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ITbJrHWBHgcxirZ+FGFnhEZpipY= 2024-02-01T10:48:14.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ITbJrHWBHgcxirZ+FGFnhEZpipY=', 'Content-Length': '0'} 2024-02-01T10:48:14.191 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fac4eb614162c20b-0065bb76ee-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:14.232 DEBUG:boto:path=/vclfcn-60/obj1 2024-02-01T10:48:14.232 DEBUG:boto:auth_path=/vclfcn-60/obj1 2024-02-01T10:48:14.232 DEBUG:boto:path=/vclfcn-60/obj1?versionId=null 2024-02-01T10:48:14.233 DEBUG:boto:auth_path=/vclfcn-60/obj1?versionId=null 2024-02-01T10:48:14.233 DEBUG:boto:Method: GET 2024-02-01T10:48:14.233 DEBUG:boto:Path: /vclfcn-60/obj1?versionId=null 2024-02-01T10:48:14.233 DEBUG:boto:Data: 2024-02-01T10:48:14.233 DEBUG:boto:Headers: {} 2024-02-01T10:48:14.233 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:14.233 DEBUG:boto:Port: 8000 2024-02-01T10:48:14.233 DEBUG:boto:Params: {} 2024-02-01T10:48:14.233 DEBUG:boto:Token: None 2024-02-01T10:48:14.234 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:14 GMT /vclfcn-60/obj1?versionId=null 2024-02-01T10:48:14.234 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ITbJrHWBHgcxirZ+FGFnhEZpipY= 2024-02-01T10:48:14.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ITbJrHWBHgcxirZ+FGFnhEZpipY=', 'Content-Length': '0'} 2024-02-01T10:48:14.237 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dea29f8992a930bf-0065bb76ee-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:14.237 DEBUG:boto:path=/vclfcn-60/obj1 2024-02-01T10:48:14.237 DEBUG:boto:auth_path=/vclfcn-60/obj1 2024-02-01T10:48:14.237 DEBUG:boto:path=/vclfcn-60/obj1?versionId=null 2024-02-01T10:48:14.237 DEBUG:boto:auth_path=/vclfcn-60/obj1?versionId=null 2024-02-01T10:48:14.237 DEBUG:boto:Method: HEAD 2024-02-01T10:48:14.238 DEBUG:boto:Path: /vclfcn-60/obj1?versionId=null 2024-02-01T10:48:14.238 DEBUG:boto:Data: 2024-02-01T10:48:14.238 DEBUG:boto:Headers: {} 2024-02-01T10:48:14.238 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:14.238 DEBUG:boto:Port: 8001 2024-02-01T10:48:14.238 DEBUG:boto:Params: {} 2024-02-01T10:48:14.238 DEBUG:boto:Token: None 2024-02-01T10:48:14.238 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:14 GMT /vclfcn-60/obj1?versionId=null 2024-02-01T10:48:14.239 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:o4FFSBW1YX5QW6tABlpYw6zBVzU= 2024-02-01T10:48:14.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:o4FFSBW1YX5QW6tABlpYw6zBVzU=', 'Content-Length': '0'} 2024-02-01T10:48:14.241 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e2c2e4b747637ca1-0065bb76ee-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:14.242 DEBUG:boto:path=/vclfcn-60/obj1 2024-02-01T10:48:14.242 DEBUG:boto:auth_path=/vclfcn-60/obj1 2024-02-01T10:48:14.242 DEBUG:boto:path=/vclfcn-60/obj1?versionId=null 2024-02-01T10:48:14.242 DEBUG:boto:auth_path=/vclfcn-60/obj1?versionId=null 2024-02-01T10:48:14.242 DEBUG:boto:Method: HEAD 2024-02-01T10:48:14.242 DEBUG:boto:Path: /vclfcn-60/obj1?versionId=null 2024-02-01T10:48:14.243 DEBUG:boto:Data: 2024-02-01T10:48:14.243 DEBUG:boto:Headers: {} 2024-02-01T10:48:14.243 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:14.243 DEBUG:boto:Port: 8000 2024-02-01T10:48:14.243 DEBUG:boto:Params: {} 2024-02-01T10:48:14.243 DEBUG:boto:Token: None 2024-02-01T10:48:14.243 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:14 GMT /vclfcn-60/obj1?versionId=null 2024-02-01T10:48:14.243 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:o4FFSBW1YX5QW6tABlpYw6zBVzU= 2024-02-01T10:48:14.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:o4FFSBW1YX5QW6tABlpYw6zBVzU=', 'Content-Length': '0'} 2024-02-01T10:48:14.247 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000008376933bada0d76-0065bb76ee-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:14.247 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:48:14.247 DEBUG:boto:path=/vclfcn-60/obj1 2024-02-01T10:48:14.247 DEBUG:boto:auth_path=/vclfcn-60/obj1 2024-02-01T10:48:14.247 DEBUG:boto:path=/vclfcn-60/obj1?versionId=null 2024-02-01T10:48:14.247 DEBUG:boto:auth_path=/vclfcn-60/obj1?versionId=null 2024-02-01T10:48:14.248 DEBUG:boto:Method: GET 2024-02-01T10:48:14.248 DEBUG:boto:Path: /vclfcn-60/obj1?versionId=null 2024-02-01T10:48:14.248 DEBUG:boto:Data: 2024-02-01T10:48:14.248 DEBUG:boto:Headers: {} 2024-02-01T10:48:14.248 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:14.248 DEBUG:boto:Port: 8001 2024-02-01T10:48:14.248 DEBUG:boto:Params: {} 2024-02-01T10:48:14.248 DEBUG:boto:Token: None 2024-02-01T10:48:14.248 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:14 GMT /vclfcn-60/obj1?versionId=null 2024-02-01T10:48:14.249 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ITbJrHWBHgcxirZ+FGFnhEZpipY= 2024-02-01T10:48:14.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ITbJrHWBHgcxirZ+FGFnhEZpipY=', 'Content-Length': '0'} 2024-02-01T10:48:14.252 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004ded850e91c03838-0065bb76ee-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:14.292 DEBUG:boto:path=/vclfcn-60/obj1 2024-02-01T10:48:14.292 DEBUG:boto:auth_path=/vclfcn-60/obj1 2024-02-01T10:48:14.292 DEBUG:boto:path=/vclfcn-60/obj1?versionId=null 2024-02-01T10:48:14.292 DEBUG:boto:auth_path=/vclfcn-60/obj1?versionId=null 2024-02-01T10:48:14.292 DEBUG:boto:Method: GET 2024-02-01T10:48:14.292 DEBUG:boto:Path: /vclfcn-60/obj1?versionId=null 2024-02-01T10:48:14.293 DEBUG:boto:Data: 2024-02-01T10:48:14.293 DEBUG:boto:Headers: {} 2024-02-01T10:48:14.293 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:14.293 DEBUG:boto:Port: 8000 2024-02-01T10:48:14.293 DEBUG:boto:Params: {} 2024-02-01T10:48:14.293 DEBUG:boto:Token: None 2024-02-01T10:48:14.293 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:14 GMT /vclfcn-60/obj1?versionId=null 2024-02-01T10:48:14.293 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ITbJrHWBHgcxirZ+FGFnhEZpipY= 2024-02-01T10:48:14.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ITbJrHWBHgcxirZ+FGFnhEZpipY=', 'Content-Length': '0'} 2024-02-01T10:48:14.296 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008da7e7b65e1dbb19-0065bb76ee-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:14.297 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:14.297 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:14.297 DEBUG:boto:path=/vclfcn-60/?prefix=obj1 2024-02-01T10:48:14.297 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj1 2024-02-01T10:48:14.297 DEBUG:boto:Method: GET 2024-02-01T10:48:14.297 DEBUG:boto:Path: /vclfcn-60/?prefix=obj1 2024-02-01T10:48:14.297 DEBUG:boto:Data: 2024-02-01T10:48:14.297 DEBUG:boto:Headers: {} 2024-02-01T10:48:14.297 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:14.298 DEBUG:boto:Port: 8001 2024-02-01T10:48:14.298 DEBUG:boto:Params: {} 2024-02-01T10:48:14.298 DEBUG:boto:Token: None 2024-02-01T10:48:14.298 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:14 GMT /vclfcn-60/ 2024-02-01T10:48:14.298 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8rc2OZObw/0ljsgnZFAojGvH+Ro= 2024-02-01T10:48:14.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8rc2OZObw/0ljsgnZFAojGvH+Ro=', 'Content-Length': '0'} 2024-02-01T10:48:14.302 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030df8278806015c2-0065bb76ee-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:14.348 DEBUG:boto:b'vclfcn-60obj11000falseobj12024-02-01T10:46:46.243Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:14.348 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:14.349 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:14.349 DEBUG:boto:path=/vclfcn-60/?prefix=obj1 2024-02-01T10:48:14.349 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj1 2024-02-01T10:48:14.349 DEBUG:boto:Method: GET 2024-02-01T10:48:14.349 DEBUG:boto:Path: /vclfcn-60/?prefix=obj1 2024-02-01T10:48:14.349 DEBUG:boto:Data: 2024-02-01T10:48:14.349 DEBUG:boto:Headers: {} 2024-02-01T10:48:14.349 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:14.349 DEBUG:boto:Port: 8000 2024-02-01T10:48:14.349 DEBUG:boto:Params: {} 2024-02-01T10:48:14.350 DEBUG:boto:Token: None 2024-02-01T10:48:14.350 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:14 GMT /vclfcn-60/ 2024-02-01T10:48:14.350 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8rc2OZObw/0ljsgnZFAojGvH+Ro= 2024-02-01T10:48:14.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8rc2OZObw/0ljsgnZFAojGvH+Ro=', 'Content-Length': '0'} 2024-02-01T10:48:14.354 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c259272d4cd4b1d8-0065bb76ee-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:14.354 DEBUG:boto:b'vclfcn-60obj11000falseobj12024-02-01T10:46:46.243Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:14.355 DEBUG:boto:path=/vclfcn-60/obj1 2024-02-01T10:48:14.355 DEBUG:boto:auth_path=/vclfcn-60/obj1 2024-02-01T10:48:14.355 DEBUG:boto:Method: HEAD 2024-02-01T10:48:14.355 DEBUG:boto:Path: /vclfcn-60/obj1 2024-02-01T10:48:14.355 DEBUG:boto:Data: 2024-02-01T10:48:14.355 DEBUG:boto:Headers: {} 2024-02-01T10:48:14.355 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:14.355 DEBUG:boto:Port: 8001 2024-02-01T10:48:14.356 DEBUG:boto:Params: {} 2024-02-01T10:48:14.356 DEBUG:boto:Token: None 2024-02-01T10:48:14.356 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:14 GMT /vclfcn-60/obj1 2024-02-01T10:48:14.356 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:69j/XI/l1Xbc0v0NsHb6lwtXh04= 2024-02-01T10:48:14.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:69j/XI/l1Xbc0v0NsHb6lwtXh04=', 'Content-Length': '0'} 2024-02-01T10:48:14.358 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dd3aacbff95fa2e9-0065bb76ee-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:14.359 DEBUG:boto:path=/vclfcn-60/obj1 2024-02-01T10:48:14.359 DEBUG:boto:auth_path=/vclfcn-60/obj1 2024-02-01T10:48:14.360 DEBUG:boto:Method: HEAD 2024-02-01T10:48:14.360 DEBUG:boto:Path: /vclfcn-60/obj1 2024-02-01T10:48:14.360 DEBUG:boto:Data: 2024-02-01T10:48:14.360 DEBUG:boto:Headers: {} 2024-02-01T10:48:14.360 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:14.360 DEBUG:boto:Port: 8000 2024-02-01T10:48:14.360 DEBUG:boto:Params: {} 2024-02-01T10:48:14.360 DEBUG:boto:Token: None 2024-02-01T10:48:14.361 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:14 GMT /vclfcn-60/obj1 2024-02-01T10:48:14.361 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:69j/XI/l1Xbc0v0NsHb6lwtXh04= 2024-02-01T10:48:14.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:69j/XI/l1Xbc0v0NsHb6lwtXh04=', 'Content-Length': '0'} 2024-02-01T10:48:14.364 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007aacbfee5c284c84-0065bb76ee-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:14.364 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:48:14.364 DEBUG:boto:path=/vclfcn-60/obj1 2024-02-01T10:48:14.364 DEBUG:boto:auth_path=/vclfcn-60/obj1 2024-02-01T10:48:14.365 DEBUG:boto:Method: GET 2024-02-01T10:48:14.365 DEBUG:boto:Path: /vclfcn-60/obj1 2024-02-01T10:48:14.365 DEBUG:boto:Data: 2024-02-01T10:48:14.365 DEBUG:boto:Headers: {} 2024-02-01T10:48:14.365 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:14.365 DEBUG:boto:Port: 8001 2024-02-01T10:48:14.365 DEBUG:boto:Params: {} 2024-02-01T10:48:14.365 DEBUG:boto:Token: None 2024-02-01T10:48:14.365 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:14 GMT /vclfcn-60/obj1 2024-02-01T10:48:14.365 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vH7QhyK/FaC/OmoYDMGLLelvcr8= 2024-02-01T10:48:14.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vH7QhyK/FaC/OmoYDMGLLelvcr8=', 'Content-Length': '0'} 2024-02-01T10:48:14.369 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006ff8acfb1e682ca6-0065bb76ee-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:14.420 DEBUG:boto:path=/vclfcn-60/obj1 2024-02-01T10:48:14.420 DEBUG:boto:auth_path=/vclfcn-60/obj1 2024-02-01T10:48:14.420 DEBUG:boto:Method: GET 2024-02-01T10:48:14.420 DEBUG:boto:Path: /vclfcn-60/obj1 2024-02-01T10:48:14.420 DEBUG:boto:Data: 2024-02-01T10:48:14.420 DEBUG:boto:Headers: {} 2024-02-01T10:48:14.420 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:14.421 DEBUG:boto:Port: 8000 2024-02-01T10:48:14.421 DEBUG:boto:Params: {} 2024-02-01T10:48:14.421 DEBUG:boto:Token: None 2024-02-01T10:48:14.421 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:14 GMT /vclfcn-60/obj1 2024-02-01T10:48:14.421 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vH7QhyK/FaC/OmoYDMGLLelvcr8= 2024-02-01T10:48:14.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vH7QhyK/FaC/OmoYDMGLLelvcr8=', 'Content-Length': '0'} 2024-02-01T10:48:14.424 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ee4924744ecfedbb-0065bb76ee-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:14.425 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:14.425 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:14.426 DEBUG:boto:path=/vclfcn-60/?prefix=obj1 2024-02-01T10:48:14.426 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj1 2024-02-01T10:48:14.426 DEBUG:boto:Method: GET 2024-02-01T10:48:14.426 DEBUG:boto:Path: /vclfcn-60/?prefix=obj1 2024-02-01T10:48:14.427 DEBUG:boto:Data: 2024-02-01T10:48:14.427 DEBUG:boto:Headers: {} 2024-02-01T10:48:14.427 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:14.427 DEBUG:boto:Port: 8001 2024-02-01T10:48:14.427 DEBUG:boto:Params: {} 2024-02-01T10:48:14.428 DEBUG:boto:Token: None 2024-02-01T10:48:14.428 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:14 GMT /vclfcn-60/ 2024-02-01T10:48:14.428 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8rc2OZObw/0ljsgnZFAojGvH+Ro= 2024-02-01T10:48:14.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8rc2OZObw/0ljsgnZFAojGvH+Ro=', 'Content-Length': '0'} 2024-02-01T10:48:14.433 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd8e8aafdc2f8a36-0065bb76ee-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:14.433 DEBUG:boto:b'vclfcn-60obj11000falseobj12024-02-01T10:46:46.243Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:14.434 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:14.434 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:14.435 DEBUG:boto:path=/vclfcn-60/?prefix=obj1 2024-02-01T10:48:14.435 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj1 2024-02-01T10:48:14.435 DEBUG:boto:Method: GET 2024-02-01T10:48:14.435 DEBUG:boto:Path: /vclfcn-60/?prefix=obj1 2024-02-01T10:48:14.435 DEBUG:boto:Data: 2024-02-01T10:48:14.436 DEBUG:boto:Headers: {} 2024-02-01T10:48:14.436 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:14.436 DEBUG:boto:Port: 8000 2024-02-01T10:48:14.436 DEBUG:boto:Params: {} 2024-02-01T10:48:14.436 DEBUG:boto:Token: None 2024-02-01T10:48:14.437 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:14 GMT /vclfcn-60/ 2024-02-01T10:48:14.437 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8rc2OZObw/0ljsgnZFAojGvH+Ro= 2024-02-01T10:48:14.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8rc2OZObw/0ljsgnZFAojGvH+Ro=', 'Content-Length': '0'} 2024-02-01T10:48:14.442 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000133fa9e824592a74-0065bb76ee-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:14.442 DEBUG:boto:b'vclfcn-60obj11000falseobj12024-02-01T10:46:46.243Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:14.443 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:48:14.443 DEBUG:boto:path=/vclfcn-60/obj2 2024-02-01T10:48:14.443 DEBUG:boto:auth_path=/vclfcn-60/obj2 2024-02-01T10:48:14.443 DEBUG:boto:path=/vclfcn-60/obj2?versionId=null 2024-02-01T10:48:14.444 DEBUG:boto:auth_path=/vclfcn-60/obj2?versionId=null 2024-02-01T10:48:14.444 DEBUG:boto:Method: GET 2024-02-01T10:48:14.444 DEBUG:boto:Path: /vclfcn-60/obj2?versionId=null 2024-02-01T10:48:14.444 DEBUG:boto:Data: 2024-02-01T10:48:14.444 DEBUG:boto:Headers: {} 2024-02-01T10:48:14.444 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:14.445 DEBUG:boto:Port: 8001 2024-02-01T10:48:14.445 DEBUG:boto:Params: {} 2024-02-01T10:48:14.445 DEBUG:boto:Token: None 2024-02-01T10:48:14.445 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:14 GMT /vclfcn-60/obj2?versionId=null 2024-02-01T10:48:14.445 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vLJp+ySBoeJJ7V6vi0bWyaVyGuQ= 2024-02-01T10:48:14.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vLJp+ySBoeJJ7V6vi0bWyaVyGuQ=', 'Content-Length': '0'} 2024-02-01T10:48:14.449 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005ae5a154c7f08567-0065bb76ee-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:14.492 DEBUG:boto:path=/vclfcn-60/obj2 2024-02-01T10:48:14.492 DEBUG:boto:auth_path=/vclfcn-60/obj2 2024-02-01T10:48:14.492 DEBUG:boto:path=/vclfcn-60/obj2?versionId=null 2024-02-01T10:48:14.493 DEBUG:boto:auth_path=/vclfcn-60/obj2?versionId=null 2024-02-01T10:48:14.493 DEBUG:boto:Method: GET 2024-02-01T10:48:14.493 DEBUG:boto:Path: /vclfcn-60/obj2?versionId=null 2024-02-01T10:48:14.493 DEBUG:boto:Data: 2024-02-01T10:48:14.493 DEBUG:boto:Headers: {} 2024-02-01T10:48:14.493 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:14.493 DEBUG:boto:Port: 8000 2024-02-01T10:48:14.493 DEBUG:boto:Params: {} 2024-02-01T10:48:14.493 DEBUG:boto:Token: None 2024-02-01T10:48:14.493 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:14 GMT /vclfcn-60/obj2?versionId=null 2024-02-01T10:48:14.494 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vLJp+ySBoeJJ7V6vi0bWyaVyGuQ= 2024-02-01T10:48:14.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vLJp+ySBoeJJ7V6vi0bWyaVyGuQ=', 'Content-Length': '0'} 2024-02-01T10:48:14.497 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000068f87cccffb2ae2c-0065bb76ee-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:14.497 DEBUG:boto:path=/vclfcn-60/obj2 2024-02-01T10:48:14.498 DEBUG:boto:auth_path=/vclfcn-60/obj2 2024-02-01T10:48:14.498 DEBUG:boto:path=/vclfcn-60/obj2?versionId=null 2024-02-01T10:48:14.498 DEBUG:boto:auth_path=/vclfcn-60/obj2?versionId=null 2024-02-01T10:48:14.498 DEBUG:boto:Method: HEAD 2024-02-01T10:48:14.498 DEBUG:boto:Path: /vclfcn-60/obj2?versionId=null 2024-02-01T10:48:14.498 DEBUG:boto:Data: 2024-02-01T10:48:14.499 DEBUG:boto:Headers: {} 2024-02-01T10:48:14.499 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:14.499 DEBUG:boto:Port: 8001 2024-02-01T10:48:14.499 DEBUG:boto:Params: {} 2024-02-01T10:48:14.499 DEBUG:boto:Token: None 2024-02-01T10:48:14.500 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:14 GMT /vclfcn-60/obj2?versionId=null 2024-02-01T10:48:14.500 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ne5fnPgeYYmFRoL9FTdPYYLnCjQ= 2024-02-01T10:48:14.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ne5fnPgeYYmFRoL9FTdPYYLnCjQ=', 'Content-Length': '0'} 2024-02-01T10:48:14.503 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007c2022f653300651-0065bb76ee-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:14.504 DEBUG:boto:path=/vclfcn-60/obj2 2024-02-01T10:48:14.504 DEBUG:boto:auth_path=/vclfcn-60/obj2 2024-02-01T10:48:14.504 DEBUG:boto:path=/vclfcn-60/obj2?versionId=null 2024-02-01T10:48:14.504 DEBUG:boto:auth_path=/vclfcn-60/obj2?versionId=null 2024-02-01T10:48:14.505 DEBUG:boto:Method: HEAD 2024-02-01T10:48:14.505 DEBUG:boto:Path: /vclfcn-60/obj2?versionId=null 2024-02-01T10:48:14.505 DEBUG:boto:Data: 2024-02-01T10:48:14.505 DEBUG:boto:Headers: {} 2024-02-01T10:48:14.505 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:14.505 DEBUG:boto:Port: 8000 2024-02-01T10:48:14.506 DEBUG:boto:Params: {} 2024-02-01T10:48:14.506 DEBUG:boto:Token: None 2024-02-01T10:48:14.506 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:14 GMT /vclfcn-60/obj2?versionId=null 2024-02-01T10:48:14.507 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ne5fnPgeYYmFRoL9FTdPYYLnCjQ= 2024-02-01T10:48:14.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ne5fnPgeYYmFRoL9FTdPYYLnCjQ=', 'Content-Length': '0'} 2024-02-01T10:48:14.510 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005cf62bb057855878-0065bb76ee-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:14.510 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:48:14.510 DEBUG:boto:path=/vclfcn-60/obj2 2024-02-01T10:48:14.511 DEBUG:boto:auth_path=/vclfcn-60/obj2 2024-02-01T10:48:14.511 DEBUG:boto:path=/vclfcn-60/obj2?versionId=null 2024-02-01T10:48:14.511 DEBUG:boto:auth_path=/vclfcn-60/obj2?versionId=null 2024-02-01T10:48:14.511 DEBUG:boto:Method: GET 2024-02-01T10:48:14.511 DEBUG:boto:Path: /vclfcn-60/obj2?versionId=null 2024-02-01T10:48:14.511 DEBUG:boto:Data: 2024-02-01T10:48:14.511 DEBUG:boto:Headers: {} 2024-02-01T10:48:14.511 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:14.512 DEBUG:boto:Port: 8001 2024-02-01T10:48:14.512 DEBUG:boto:Params: {} 2024-02-01T10:48:14.512 DEBUG:boto:Token: None 2024-02-01T10:48:14.512 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:14 GMT /vclfcn-60/obj2?versionId=null 2024-02-01T10:48:14.512 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vLJp+ySBoeJJ7V6vi0bWyaVyGuQ= 2024-02-01T10:48:14.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vLJp+ySBoeJJ7V6vi0bWyaVyGuQ=', 'Content-Length': '0'} 2024-02-01T10:48:14.515 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000404e8af047ce41ee-0065bb76ee-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:14.560 DEBUG:boto:path=/vclfcn-60/obj2 2024-02-01T10:48:14.560 DEBUG:boto:auth_path=/vclfcn-60/obj2 2024-02-01T10:48:14.560 DEBUG:boto:path=/vclfcn-60/obj2?versionId=null 2024-02-01T10:48:14.561 DEBUG:boto:auth_path=/vclfcn-60/obj2?versionId=null 2024-02-01T10:48:14.561 DEBUG:boto:Method: GET 2024-02-01T10:48:14.561 DEBUG:boto:Path: /vclfcn-60/obj2?versionId=null 2024-02-01T10:48:14.561 DEBUG:boto:Data: 2024-02-01T10:48:14.561 DEBUG:boto:Headers: {} 2024-02-01T10:48:14.561 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:14.561 DEBUG:boto:Port: 8000 2024-02-01T10:48:14.561 DEBUG:boto:Params: {} 2024-02-01T10:48:14.561 DEBUG:boto:Token: None 2024-02-01T10:48:14.562 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:14 GMT /vclfcn-60/obj2?versionId=null 2024-02-01T10:48:14.562 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vLJp+ySBoeJJ7V6vi0bWyaVyGuQ= 2024-02-01T10:48:14.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vLJp+ySBoeJJ7V6vi0bWyaVyGuQ=', 'Content-Length': '0'} 2024-02-01T10:48:14.565 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003f684fd53502188d-0065bb76ee-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:14.565 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:14.565 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:14.566 DEBUG:boto:path=/vclfcn-60/?prefix=obj2 2024-02-01T10:48:14.566 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj2 2024-02-01T10:48:14.566 DEBUG:boto:Method: GET 2024-02-01T10:48:14.566 DEBUG:boto:Path: /vclfcn-60/?prefix=obj2 2024-02-01T10:48:14.566 DEBUG:boto:Data: 2024-02-01T10:48:14.567 DEBUG:boto:Headers: {} 2024-02-01T10:48:14.567 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:14.567 DEBUG:boto:Port: 8001 2024-02-01T10:48:14.567 DEBUG:boto:Params: {} 2024-02-01T10:48:14.567 DEBUG:boto:Token: None 2024-02-01T10:48:14.568 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:14 GMT /vclfcn-60/ 2024-02-01T10:48:14.568 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8rc2OZObw/0ljsgnZFAojGvH+Ro= 2024-02-01T10:48:14.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8rc2OZObw/0ljsgnZFAojGvH+Ro=', 'Content-Length': '0'} 2024-02-01T10:48:14.573 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015071da6383b1cd9-0065bb76ee-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:14.573 DEBUG:boto:b'vclfcn-60obj21000falseobj22024-02-01T10:46:46.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:14.574 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:14.574 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:14.574 DEBUG:boto:path=/vclfcn-60/?prefix=obj2 2024-02-01T10:48:14.575 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj2 2024-02-01T10:48:14.575 DEBUG:boto:Method: GET 2024-02-01T10:48:14.575 DEBUG:boto:Path: /vclfcn-60/?prefix=obj2 2024-02-01T10:48:14.575 DEBUG:boto:Data: 2024-02-01T10:48:14.575 DEBUG:boto:Headers: {} 2024-02-01T10:48:14.575 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:14.575 DEBUG:boto:Port: 8000 2024-02-01T10:48:14.576 DEBUG:boto:Params: {} 2024-02-01T10:48:14.576 DEBUG:boto:Token: None 2024-02-01T10:48:14.576 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:14 GMT /vclfcn-60/ 2024-02-01T10:48:14.576 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8rc2OZObw/0ljsgnZFAojGvH+Ro= 2024-02-01T10:48:14.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8rc2OZObw/0ljsgnZFAojGvH+Ro=', 'Content-Length': '0'} 2024-02-01T10:48:14.581 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a14fc626f377ad47-0065bb76ee-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:14.581 DEBUG:boto:b'vclfcn-60obj21000falseobj22024-02-01T10:46:46.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:14.582 DEBUG:boto:path=/vclfcn-60/obj2 2024-02-01T10:48:14.582 DEBUG:boto:auth_path=/vclfcn-60/obj2 2024-02-01T10:48:14.582 DEBUG:boto:Method: HEAD 2024-02-01T10:48:14.582 DEBUG:boto:Path: /vclfcn-60/obj2 2024-02-01T10:48:14.582 DEBUG:boto:Data: 2024-02-01T10:48:14.582 DEBUG:boto:Headers: {} 2024-02-01T10:48:14.582 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:14.582 DEBUG:boto:Port: 8001 2024-02-01T10:48:14.582 DEBUG:boto:Params: {} 2024-02-01T10:48:14.582 DEBUG:boto:Token: None 2024-02-01T10:48:14.583 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:14 GMT /vclfcn-60/obj2 2024-02-01T10:48:14.583 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Kt8jfKlYuXG8PDP4XuHxK3LHqMg= 2024-02-01T10:48:14.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Kt8jfKlYuXG8PDP4XuHxK3LHqMg=', 'Content-Length': '0'} 2024-02-01T10:48:14.586 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009424714ff887149f-0065bb76ee-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:14.587 DEBUG:boto:path=/vclfcn-60/obj2 2024-02-01T10:48:14.587 DEBUG:boto:auth_path=/vclfcn-60/obj2 2024-02-01T10:48:14.587 DEBUG:boto:Method: HEAD 2024-02-01T10:48:14.588 DEBUG:boto:Path: /vclfcn-60/obj2 2024-02-01T10:48:14.588 DEBUG:boto:Data: 2024-02-01T10:48:14.588 DEBUG:boto:Headers: {} 2024-02-01T10:48:14.588 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:14.588 DEBUG:boto:Port: 8000 2024-02-01T10:48:14.589 DEBUG:boto:Params: {} 2024-02-01T10:48:14.590 DEBUG:boto:Token: None 2024-02-01T10:48:14.590 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:14 GMT /vclfcn-60/obj2 2024-02-01T10:48:14.590 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Kt8jfKlYuXG8PDP4XuHxK3LHqMg= 2024-02-01T10:48:14.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Kt8jfKlYuXG8PDP4XuHxK3LHqMg=', 'Content-Length': '0'} 2024-02-01T10:48:14.593 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000086a590a3a4e1cf6a-0065bb76ee-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:14.593 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:48:14.594 DEBUG:boto:path=/vclfcn-60/obj2 2024-02-01T10:48:14.594 DEBUG:boto:auth_path=/vclfcn-60/obj2 2024-02-01T10:48:14.594 DEBUG:boto:Method: GET 2024-02-01T10:48:14.594 DEBUG:boto:Path: /vclfcn-60/obj2 2024-02-01T10:48:14.594 DEBUG:boto:Data: 2024-02-01T10:48:14.594 DEBUG:boto:Headers: {} 2024-02-01T10:48:14.595 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:14.595 DEBUG:boto:Port: 8001 2024-02-01T10:48:14.595 DEBUG:boto:Params: {} 2024-02-01T10:48:14.595 DEBUG:boto:Token: None 2024-02-01T10:48:14.595 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:14 GMT /vclfcn-60/obj2 2024-02-01T10:48:14.595 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:kzlcCEurZKxbhdFlsCvHVxMtqLc= 2024-02-01T10:48:14.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:kzlcCEurZKxbhdFlsCvHVxMtqLc=', 'Content-Length': '0'} 2024-02-01T10:48:14.599 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000049fb3944f0ee2f6d-0065bb76ee-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:14.640 DEBUG:boto:path=/vclfcn-60/obj2 2024-02-01T10:48:14.641 DEBUG:boto:auth_path=/vclfcn-60/obj2 2024-02-01T10:48:14.641 DEBUG:boto:Method: GET 2024-02-01T10:48:14.641 DEBUG:boto:Path: /vclfcn-60/obj2 2024-02-01T10:48:14.641 DEBUG:boto:Data: 2024-02-01T10:48:14.641 DEBUG:boto:Headers: {} 2024-02-01T10:48:14.641 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:14.641 DEBUG:boto:Port: 8000 2024-02-01T10:48:14.642 DEBUG:boto:Params: {} 2024-02-01T10:48:14.642 DEBUG:boto:Token: None 2024-02-01T10:48:14.642 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:14 GMT /vclfcn-60/obj2 2024-02-01T10:48:14.642 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:kzlcCEurZKxbhdFlsCvHVxMtqLc= 2024-02-01T10:48:14.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:kzlcCEurZKxbhdFlsCvHVxMtqLc=', 'Content-Length': '0'} 2024-02-01T10:48:14.646 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000042e61daf97b6ae72-0065bb76ee-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:14.646 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:14.646 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:14.647 DEBUG:boto:path=/vclfcn-60/?prefix=obj2 2024-02-01T10:48:14.647 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj2 2024-02-01T10:48:14.647 DEBUG:boto:Method: GET 2024-02-01T10:48:14.647 DEBUG:boto:Path: /vclfcn-60/?prefix=obj2 2024-02-01T10:48:14.647 DEBUG:boto:Data: 2024-02-01T10:48:14.647 DEBUG:boto:Headers: {} 2024-02-01T10:48:14.647 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:14.648 DEBUG:boto:Port: 8001 2024-02-01T10:48:14.648 DEBUG:boto:Params: {} 2024-02-01T10:48:14.648 DEBUG:boto:Token: None 2024-02-01T10:48:14.648 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:14 GMT /vclfcn-60/ 2024-02-01T10:48:14.648 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8rc2OZObw/0ljsgnZFAojGvH+Ro= 2024-02-01T10:48:14.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8rc2OZObw/0ljsgnZFAojGvH+Ro=', 'Content-Length': '0'} 2024-02-01T10:48:14.653 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2ee6c39da500d8e-0065bb76ee-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:14.653 DEBUG:boto:b'vclfcn-60obj21000falseobj22024-02-01T10:46:46.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:14.654 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:14.654 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:14.654 DEBUG:boto:path=/vclfcn-60/?prefix=obj2 2024-02-01T10:48:14.654 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj2 2024-02-01T10:48:14.654 DEBUG:boto:Method: GET 2024-02-01T10:48:14.655 DEBUG:boto:Path: /vclfcn-60/?prefix=obj2 2024-02-01T10:48:14.655 DEBUG:boto:Data: 2024-02-01T10:48:14.655 DEBUG:boto:Headers: {} 2024-02-01T10:48:14.655 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:14.655 DEBUG:boto:Port: 8000 2024-02-01T10:48:14.656 DEBUG:boto:Params: {} 2024-02-01T10:48:14.656 DEBUG:boto:Token: None 2024-02-01T10:48:14.656 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:14 GMT /vclfcn-60/ 2024-02-01T10:48:14.657 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8rc2OZObw/0ljsgnZFAojGvH+Ro= 2024-02-01T10:48:14.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8rc2OZObw/0ljsgnZFAojGvH+Ro=', 'Content-Length': '0'} 2024-02-01T10:48:14.661 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d582591c6b4ac33b-0065bb76ee-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:14.704 DEBUG:boto:b'vclfcn-60obj21000falseobj22024-02-01T10:46:46.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:14.705 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:48:14.705 DEBUG:boto:path=/vclfcn-60/obj3 2024-02-01T10:48:14.705 DEBUG:boto:auth_path=/vclfcn-60/obj3 2024-02-01T10:48:14.706 DEBUG:boto:path=/vclfcn-60/obj3?versionId=null 2024-02-01T10:48:14.706 DEBUG:boto:auth_path=/vclfcn-60/obj3?versionId=null 2024-02-01T10:48:14.706 DEBUG:boto:Method: GET 2024-02-01T10:48:14.706 DEBUG:boto:Path: /vclfcn-60/obj3?versionId=null 2024-02-01T10:48:14.706 DEBUG:boto:Data: 2024-02-01T10:48:14.707 DEBUG:boto:Headers: {} 2024-02-01T10:48:14.707 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:14.707 DEBUG:boto:Port: 8001 2024-02-01T10:48:14.707 DEBUG:boto:Params: {} 2024-02-01T10:48:14.707 DEBUG:boto:Token: None 2024-02-01T10:48:14.707 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:14 GMT /vclfcn-60/obj3?versionId=null 2024-02-01T10:48:14.707 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:2svcwrcYsxOZ67VoZ5PVUWiAWTk= 2024-02-01T10:48:14.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2svcwrcYsxOZ67VoZ5PVUWiAWTk=', 'Content-Length': '0'} 2024-02-01T10:48:14.711 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000088872ec124029e5f-0065bb76ee-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:14.711 DEBUG:boto:path=/vclfcn-60/obj3 2024-02-01T10:48:14.711 DEBUG:boto:auth_path=/vclfcn-60/obj3 2024-02-01T10:48:14.712 DEBUG:boto:path=/vclfcn-60/obj3?versionId=null 2024-02-01T10:48:14.712 DEBUG:boto:auth_path=/vclfcn-60/obj3?versionId=null 2024-02-01T10:48:14.712 DEBUG:boto:Method: GET 2024-02-01T10:48:14.712 DEBUG:boto:Path: /vclfcn-60/obj3?versionId=null 2024-02-01T10:48:14.713 DEBUG:boto:Data: 2024-02-01T10:48:14.713 DEBUG:boto:Headers: {} 2024-02-01T10:48:14.713 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:14.713 DEBUG:boto:Port: 8000 2024-02-01T10:48:14.713 DEBUG:boto:Params: {} 2024-02-01T10:48:14.713 DEBUG:boto:Token: None 2024-02-01T10:48:14.714 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:14 GMT /vclfcn-60/obj3?versionId=null 2024-02-01T10:48:14.714 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:2svcwrcYsxOZ67VoZ5PVUWiAWTk= 2024-02-01T10:48:14.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2svcwrcYsxOZ67VoZ5PVUWiAWTk=', 'Content-Length': '0'} 2024-02-01T10:48:14.717 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fdcfe433bfe26983-0065bb76ee-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:14.718 DEBUG:boto:path=/vclfcn-60/obj3 2024-02-01T10:48:14.718 DEBUG:boto:auth_path=/vclfcn-60/obj3 2024-02-01T10:48:14.718 DEBUG:boto:path=/vclfcn-60/obj3?versionId=null 2024-02-01T10:48:14.718 DEBUG:boto:auth_path=/vclfcn-60/obj3?versionId=null 2024-02-01T10:48:14.718 DEBUG:boto:Method: HEAD 2024-02-01T10:48:14.719 DEBUG:boto:Path: /vclfcn-60/obj3?versionId=null 2024-02-01T10:48:14.719 DEBUG:boto:Data: 2024-02-01T10:48:14.719 DEBUG:boto:Headers: {} 2024-02-01T10:48:14.719 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:14.719 DEBUG:boto:Port: 8001 2024-02-01T10:48:14.719 DEBUG:boto:Params: {} 2024-02-01T10:48:14.719 DEBUG:boto:Token: None 2024-02-01T10:48:14.720 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:14 GMT /vclfcn-60/obj3?versionId=null 2024-02-01T10:48:14.720 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mzgjG7SPGynqBFStVza0EE8SnpM= 2024-02-01T10:48:14.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mzgjG7SPGynqBFStVza0EE8SnpM=', 'Content-Length': '0'} 2024-02-01T10:48:14.723 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002b45c279b4af134d-0065bb76ee-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:14.723 DEBUG:boto:path=/vclfcn-60/obj3 2024-02-01T10:48:14.723 DEBUG:boto:auth_path=/vclfcn-60/obj3 2024-02-01T10:48:14.724 DEBUG:boto:path=/vclfcn-60/obj3?versionId=null 2024-02-01T10:48:14.724 DEBUG:boto:auth_path=/vclfcn-60/obj3?versionId=null 2024-02-01T10:48:14.724 DEBUG:boto:Method: HEAD 2024-02-01T10:48:14.724 DEBUG:boto:Path: /vclfcn-60/obj3?versionId=null 2024-02-01T10:48:14.724 DEBUG:boto:Data: 2024-02-01T10:48:14.724 DEBUG:boto:Headers: {} 2024-02-01T10:48:14.724 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:14.724 DEBUG:boto:Port: 8000 2024-02-01T10:48:14.725 DEBUG:boto:Params: {} 2024-02-01T10:48:14.725 DEBUG:boto:Token: None 2024-02-01T10:48:14.725 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:14 GMT /vclfcn-60/obj3?versionId=null 2024-02-01T10:48:14.725 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mzgjG7SPGynqBFStVza0EE8SnpM= 2024-02-01T10:48:14.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mzgjG7SPGynqBFStVza0EE8SnpM=', 'Content-Length': '0'} 2024-02-01T10:48:14.728 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a185bcdb204f5b2c-0065bb76ee-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:14.728 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:48:14.729 DEBUG:boto:path=/vclfcn-60/obj3 2024-02-01T10:48:14.729 DEBUG:boto:auth_path=/vclfcn-60/obj3 2024-02-01T10:48:14.729 DEBUG:boto:path=/vclfcn-60/obj3?versionId=null 2024-02-01T10:48:14.729 DEBUG:boto:auth_path=/vclfcn-60/obj3?versionId=null 2024-02-01T10:48:14.729 DEBUG:boto:Method: GET 2024-02-01T10:48:14.730 DEBUG:boto:Path: /vclfcn-60/obj3?versionId=null 2024-02-01T10:48:14.730 DEBUG:boto:Data: 2024-02-01T10:48:14.730 DEBUG:boto:Headers: {} 2024-02-01T10:48:14.731 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:14.731 DEBUG:boto:Port: 8001 2024-02-01T10:48:14.731 DEBUG:boto:Params: {} 2024-02-01T10:48:14.731 DEBUG:boto:Token: None 2024-02-01T10:48:14.731 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:14 GMT /vclfcn-60/obj3?versionId=null 2024-02-01T10:48:14.731 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:2svcwrcYsxOZ67VoZ5PVUWiAWTk= 2024-02-01T10:48:14.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2svcwrcYsxOZ67VoZ5PVUWiAWTk=', 'Content-Length': '0'} 2024-02-01T10:48:14.734 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d507c6d88f03af15-0065bb76ee-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:14.776 DEBUG:boto:path=/vclfcn-60/obj3 2024-02-01T10:48:14.776 DEBUG:boto:auth_path=/vclfcn-60/obj3 2024-02-01T10:48:14.776 DEBUG:boto:path=/vclfcn-60/obj3?versionId=null 2024-02-01T10:48:14.776 DEBUG:boto:auth_path=/vclfcn-60/obj3?versionId=null 2024-02-01T10:48:14.777 DEBUG:boto:Method: GET 2024-02-01T10:48:14.777 DEBUG:boto:Path: /vclfcn-60/obj3?versionId=null 2024-02-01T10:48:14.777 DEBUG:boto:Data: 2024-02-01T10:48:14.777 DEBUG:boto:Headers: {} 2024-02-01T10:48:14.777 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:14.777 DEBUG:boto:Port: 8000 2024-02-01T10:48:14.777 DEBUG:boto:Params: {} 2024-02-01T10:48:14.778 DEBUG:boto:Token: None 2024-02-01T10:48:14.778 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:14 GMT /vclfcn-60/obj3?versionId=null 2024-02-01T10:48:14.778 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:2svcwrcYsxOZ67VoZ5PVUWiAWTk= 2024-02-01T10:48:14.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2svcwrcYsxOZ67VoZ5PVUWiAWTk=', 'Content-Length': '0'} 2024-02-01T10:48:14.781 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007d7777c1e6d2ad04-0065bb76ee-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:14.782 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:14.782 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:14.782 DEBUG:boto:path=/vclfcn-60/?prefix=obj3 2024-02-01T10:48:14.782 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj3 2024-02-01T10:48:14.783 DEBUG:boto:Method: GET 2024-02-01T10:48:14.783 DEBUG:boto:Path: /vclfcn-60/?prefix=obj3 2024-02-01T10:48:14.783 DEBUG:boto:Data: 2024-02-01T10:48:14.783 DEBUG:boto:Headers: {} 2024-02-01T10:48:14.784 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:14.784 DEBUG:boto:Port: 8001 2024-02-01T10:48:14.784 DEBUG:boto:Params: {} 2024-02-01T10:48:14.784 DEBUG:boto:Token: None 2024-02-01T10:48:14.785 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:14 GMT /vclfcn-60/ 2024-02-01T10:48:14.785 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8rc2OZObw/0ljsgnZFAojGvH+Ro= 2024-02-01T10:48:14.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8rc2OZObw/0ljsgnZFAojGvH+Ro=', 'Content-Length': '0'} 2024-02-01T10:48:14.789 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e53d5158a47f63f-0065bb76ee-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:14.836 DEBUG:boto:b'vclfcn-60obj31000falseobj32024-02-01T10:46:46.285Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:14.837 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:14.837 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:14.837 DEBUG:boto:path=/vclfcn-60/?prefix=obj3 2024-02-01T10:48:14.837 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj3 2024-02-01T10:48:14.837 DEBUG:boto:Method: GET 2024-02-01T10:48:14.837 DEBUG:boto:Path: /vclfcn-60/?prefix=obj3 2024-02-01T10:48:14.838 DEBUG:boto:Data: 2024-02-01T10:48:14.838 DEBUG:boto:Headers: {} 2024-02-01T10:48:14.838 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:14.838 DEBUG:boto:Port: 8000 2024-02-01T10:48:14.838 DEBUG:boto:Params: {} 2024-02-01T10:48:14.838 DEBUG:boto:Token: None 2024-02-01T10:48:14.839 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:14 GMT /vclfcn-60/ 2024-02-01T10:48:14.839 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8rc2OZObw/0ljsgnZFAojGvH+Ro= 2024-02-01T10:48:14.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8rc2OZObw/0ljsgnZFAojGvH+Ro=', 'Content-Length': '0'} 2024-02-01T10:48:14.843 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026c89b5a5d7ca117-0065bb76ee-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:14.844 DEBUG:boto:b'vclfcn-60obj31000falseobj32024-02-01T10:46:46.285Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:14.844 DEBUG:boto:path=/vclfcn-60/obj3 2024-02-01T10:48:14.844 DEBUG:boto:auth_path=/vclfcn-60/obj3 2024-02-01T10:48:14.845 DEBUG:boto:Method: HEAD 2024-02-01T10:48:14.845 DEBUG:boto:Path: /vclfcn-60/obj3 2024-02-01T10:48:14.845 DEBUG:boto:Data: 2024-02-01T10:48:14.845 DEBUG:boto:Headers: {} 2024-02-01T10:48:14.845 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:14.845 DEBUG:boto:Port: 8001 2024-02-01T10:48:14.845 DEBUG:boto:Params: {} 2024-02-01T10:48:14.846 DEBUG:boto:Token: None 2024-02-01T10:48:14.846 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:14 GMT /vclfcn-60/obj3 2024-02-01T10:48:14.846 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:as3IaaBivpWVWRX68R3SjlZoj8M= 2024-02-01T10:48:14.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:as3IaaBivpWVWRX68R3SjlZoj8M=', 'Content-Length': '0'} 2024-02-01T10:48:14.849 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000090b7e90ff16eee9c-0065bb76ee-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:14.850 DEBUG:boto:path=/vclfcn-60/obj3 2024-02-01T10:48:14.850 DEBUG:boto:auth_path=/vclfcn-60/obj3 2024-02-01T10:48:14.850 DEBUG:boto:Method: HEAD 2024-02-01T10:48:14.850 DEBUG:boto:Path: /vclfcn-60/obj3 2024-02-01T10:48:14.850 DEBUG:boto:Data: 2024-02-01T10:48:14.851 DEBUG:boto:Headers: {} 2024-02-01T10:48:14.851 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:14.851 DEBUG:boto:Port: 8000 2024-02-01T10:48:14.851 DEBUG:boto:Params: {} 2024-02-01T10:48:14.851 DEBUG:boto:Token: None 2024-02-01T10:48:14.852 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:14 GMT /vclfcn-60/obj3 2024-02-01T10:48:14.852 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:as3IaaBivpWVWRX68R3SjlZoj8M= 2024-02-01T10:48:14.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:as3IaaBivpWVWRX68R3SjlZoj8M=', 'Content-Length': '0'} 2024-02-01T10:48:14.855 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003a64703f4679b00c-0065bb76ee-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:14.855 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:48:14.856 DEBUG:boto:path=/vclfcn-60/obj3 2024-02-01T10:48:14.856 DEBUG:boto:auth_path=/vclfcn-60/obj3 2024-02-01T10:48:14.856 DEBUG:boto:Method: GET 2024-02-01T10:48:14.856 DEBUG:boto:Path: /vclfcn-60/obj3 2024-02-01T10:48:14.856 DEBUG:boto:Data: 2024-02-01T10:48:14.856 DEBUG:boto:Headers: {} 2024-02-01T10:48:14.856 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:14.856 DEBUG:boto:Port: 8001 2024-02-01T10:48:14.857 DEBUG:boto:Params: {} 2024-02-01T10:48:14.857 DEBUG:boto:Token: None 2024-02-01T10:48:14.857 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:14 GMT /vclfcn-60/obj3 2024-02-01T10:48:14.857 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/WhmHWnUA5dIUu5acgNWl5IM+e0= 2024-02-01T10:48:14.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/WhmHWnUA5dIUu5acgNWl5IM+e0=', 'Content-Length': '0'} 2024-02-01T10:48:14.861 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000666fdb8159aa279a-0065bb76ee-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:14.904 DEBUG:boto:path=/vclfcn-60/obj3 2024-02-01T10:48:14.904 DEBUG:boto:auth_path=/vclfcn-60/obj3 2024-02-01T10:48:14.905 DEBUG:boto:Method: GET 2024-02-01T10:48:14.905 DEBUG:boto:Path: /vclfcn-60/obj3 2024-02-01T10:48:14.905 DEBUG:boto:Data: 2024-02-01T10:48:14.905 DEBUG:boto:Headers: {} 2024-02-01T10:48:14.905 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:14.905 DEBUG:boto:Port: 8000 2024-02-01T10:48:14.905 DEBUG:boto:Params: {} 2024-02-01T10:48:14.905 DEBUG:boto:Token: None 2024-02-01T10:48:14.905 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:14 GMT /vclfcn-60/obj3 2024-02-01T10:48:14.906 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/WhmHWnUA5dIUu5acgNWl5IM+e0= 2024-02-01T10:48:14.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/WhmHWnUA5dIUu5acgNWl5IM+e0=', 'Content-Length': '0'} 2024-02-01T10:48:14.909 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ac3e77a4414a3f45-0065bb76ee-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:14.909 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:14.909 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:14.910 DEBUG:boto:path=/vclfcn-60/?prefix=obj3 2024-02-01T10:48:14.910 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj3 2024-02-01T10:48:14.910 DEBUG:boto:Method: GET 2024-02-01T10:48:14.910 DEBUG:boto:Path: /vclfcn-60/?prefix=obj3 2024-02-01T10:48:14.910 DEBUG:boto:Data: 2024-02-01T10:48:14.910 DEBUG:boto:Headers: {} 2024-02-01T10:48:14.911 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:14.911 DEBUG:boto:Port: 8001 2024-02-01T10:48:14.911 DEBUG:boto:Params: {} 2024-02-01T10:48:14.911 DEBUG:boto:Token: None 2024-02-01T10:48:14.912 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:14 GMT /vclfcn-60/ 2024-02-01T10:48:14.912 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8rc2OZObw/0ljsgnZFAojGvH+Ro= 2024-02-01T10:48:14.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8rc2OZObw/0ljsgnZFAojGvH+Ro=', 'Content-Length': '0'} 2024-02-01T10:48:14.916 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de3260cd021c56a9-0065bb76ee-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:14.960 DEBUG:boto:b'vclfcn-60obj31000falseobj32024-02-01T10:46:46.285Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:14.961 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:14.961 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:14.961 DEBUG:boto:path=/vclfcn-60/?prefix=obj3 2024-02-01T10:48:14.961 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj3 2024-02-01T10:48:14.962 DEBUG:boto:Method: GET 2024-02-01T10:48:14.962 DEBUG:boto:Path: /vclfcn-60/?prefix=obj3 2024-02-01T10:48:14.962 DEBUG:boto:Data: 2024-02-01T10:48:14.962 DEBUG:boto:Headers: {} 2024-02-01T10:48:14.962 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:14.962 DEBUG:boto:Port: 8000 2024-02-01T10:48:14.962 DEBUG:boto:Params: {} 2024-02-01T10:48:14.963 DEBUG:boto:Token: None 2024-02-01T10:48:14.963 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:14 GMT /vclfcn-60/ 2024-02-01T10:48:14.963 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8rc2OZObw/0ljsgnZFAojGvH+Ro= 2024-02-01T10:48:14.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8rc2OZObw/0ljsgnZFAojGvH+Ro=', 'Content-Length': '0'} 2024-02-01T10:48:14.970 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b804f60ebb5b9748-0065bb76ee-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:14.970 DEBUG:boto:b'vclfcn-60obj31000falseobj32024-02-01T10:46:46.285Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:14.971 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:48:14.971 DEBUG:boto:path=/vclfcn-60/obj4 2024-02-01T10:48:14.971 DEBUG:boto:auth_path=/vclfcn-60/obj4 2024-02-01T10:48:14.971 DEBUG:boto:path=/vclfcn-60/obj4?versionId=null 2024-02-01T10:48:14.971 DEBUG:boto:auth_path=/vclfcn-60/obj4?versionId=null 2024-02-01T10:48:14.971 DEBUG:boto:Method: GET 2024-02-01T10:48:14.971 DEBUG:boto:Path: /vclfcn-60/obj4?versionId=null 2024-02-01T10:48:14.972 DEBUG:boto:Data: 2024-02-01T10:48:14.972 DEBUG:boto:Headers: {} 2024-02-01T10:48:14.972 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:14.972 DEBUG:boto:Port: 8001 2024-02-01T10:48:14.972 DEBUG:boto:Params: {} 2024-02-01T10:48:14.972 DEBUG:boto:Token: None 2024-02-01T10:48:14.973 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:14 GMT /vclfcn-60/obj4?versionId=null 2024-02-01T10:48:14.973 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:rhJ9NErYkPUTcXM3Szx67c8FNZQ= 2024-02-01T10:48:14.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:rhJ9NErYkPUTcXM3Szx67c8FNZQ=', 'Content-Length': '0'} 2024-02-01T10:48:14.976 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001c7103b40d908cda-0065bb76ee-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:14.977 DEBUG:boto:path=/vclfcn-60/obj4 2024-02-01T10:48:14.977 DEBUG:boto:auth_path=/vclfcn-60/obj4 2024-02-01T10:48:14.977 DEBUG:boto:path=/vclfcn-60/obj4?versionId=null 2024-02-01T10:48:14.977 DEBUG:boto:auth_path=/vclfcn-60/obj4?versionId=null 2024-02-01T10:48:14.978 DEBUG:boto:Method: GET 2024-02-01T10:48:14.978 DEBUG:boto:Path: /vclfcn-60/obj4?versionId=null 2024-02-01T10:48:14.978 DEBUG:boto:Data: 2024-02-01T10:48:14.978 DEBUG:boto:Headers: {} 2024-02-01T10:48:14.978 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:14.978 DEBUG:boto:Port: 8000 2024-02-01T10:48:14.978 DEBUG:boto:Params: {} 2024-02-01T10:48:14.978 DEBUG:boto:Token: None 2024-02-01T10:48:14.978 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:14 GMT /vclfcn-60/obj4?versionId=null 2024-02-01T10:48:14.979 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:rhJ9NErYkPUTcXM3Szx67c8FNZQ= 2024-02-01T10:48:14.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:rhJ9NErYkPUTcXM3Szx67c8FNZQ=', 'Content-Length': '0'} 2024-02-01T10:48:14.983 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a7ffbc636d5618c6-0065bb76ee-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:14.983 DEBUG:boto:path=/vclfcn-60/obj4 2024-02-01T10:48:14.983 DEBUG:boto:auth_path=/vclfcn-60/obj4 2024-02-01T10:48:14.984 DEBUG:boto:path=/vclfcn-60/obj4?versionId=null 2024-02-01T10:48:14.984 DEBUG:boto:auth_path=/vclfcn-60/obj4?versionId=null 2024-02-01T10:48:14.984 DEBUG:boto:Method: HEAD 2024-02-01T10:48:14.984 DEBUG:boto:Path: /vclfcn-60/obj4?versionId=null 2024-02-01T10:48:14.984 DEBUG:boto:Data: 2024-02-01T10:48:14.985 DEBUG:boto:Headers: {} 2024-02-01T10:48:14.985 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:14.985 DEBUG:boto:Port: 8001 2024-02-01T10:48:14.985 DEBUG:boto:Params: {} 2024-02-01T10:48:14.986 DEBUG:boto:Token: None 2024-02-01T10:48:14.986 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:14 GMT /vclfcn-60/obj4?versionId=null 2024-02-01T10:48:14.986 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:pptLxYSU5X4CgA9FHfusRMJyj/A= 2024-02-01T10:48:14.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:pptLxYSU5X4CgA9FHfusRMJyj/A=', 'Content-Length': '0'} 2024-02-01T10:48:14.989 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d9e7ca8299a502df-0065bb76ee-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:14.989 DEBUG:boto:path=/vclfcn-60/obj4 2024-02-01T10:48:14.989 DEBUG:boto:auth_path=/vclfcn-60/obj4 2024-02-01T10:48:14.989 DEBUG:boto:path=/vclfcn-60/obj4?versionId=null 2024-02-01T10:48:14.990 DEBUG:boto:auth_path=/vclfcn-60/obj4?versionId=null 2024-02-01T10:48:14.990 DEBUG:boto:Method: HEAD 2024-02-01T10:48:14.990 DEBUG:boto:Path: /vclfcn-60/obj4?versionId=null 2024-02-01T10:48:14.990 DEBUG:boto:Data: 2024-02-01T10:48:14.990 DEBUG:boto:Headers: {} 2024-02-01T10:48:14.990 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:14.990 DEBUG:boto:Port: 8000 2024-02-01T10:48:14.990 DEBUG:boto:Params: {} 2024-02-01T10:48:14.990 DEBUG:boto:Token: None 2024-02-01T10:48:14.991 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:14 GMT /vclfcn-60/obj4?versionId=null 2024-02-01T10:48:14.991 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:pptLxYSU5X4CgA9FHfusRMJyj/A= 2024-02-01T10:48:14.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:pptLxYSU5X4CgA9FHfusRMJyj/A=', 'Content-Length': '0'} 2024-02-01T10:48:14.993 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000054c728d1eb76a042-0065bb76ee-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:14.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:48:14.994 DEBUG:boto:path=/vclfcn-60/obj4 2024-02-01T10:48:14.994 DEBUG:boto:auth_path=/vclfcn-60/obj4 2024-02-01T10:48:14.994 DEBUG:boto:path=/vclfcn-60/obj4?versionId=null 2024-02-01T10:48:14.994 DEBUG:boto:auth_path=/vclfcn-60/obj4?versionId=null 2024-02-01T10:48:14.994 DEBUG:boto:Method: GET 2024-02-01T10:48:14.995 DEBUG:boto:Path: /vclfcn-60/obj4?versionId=null 2024-02-01T10:48:14.995 DEBUG:boto:Data: 2024-02-01T10:48:14.995 DEBUG:boto:Headers: {} 2024-02-01T10:48:14.995 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:14.995 DEBUG:boto:Port: 8001 2024-02-01T10:48:14.995 DEBUG:boto:Params: {} 2024-02-01T10:48:14.995 DEBUG:boto:Token: None 2024-02-01T10:48:14.995 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:14 GMT /vclfcn-60/obj4?versionId=null 2024-02-01T10:48:14.995 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:rhJ9NErYkPUTcXM3Szx67c8FNZQ= 2024-02-01T10:48:14.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:rhJ9NErYkPUTcXM3Szx67c8FNZQ=', 'Content-Length': '0'} 2024-02-01T10:48:14.998 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000e4b8ec7cbe7ee20-0065bb76ee-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:15.040 DEBUG:boto:path=/vclfcn-60/obj4 2024-02-01T10:48:15.041 DEBUG:boto:auth_path=/vclfcn-60/obj4 2024-02-01T10:48:15.041 DEBUG:boto:path=/vclfcn-60/obj4?versionId=null 2024-02-01T10:48:15.041 DEBUG:boto:auth_path=/vclfcn-60/obj4?versionId=null 2024-02-01T10:48:15.041 DEBUG:boto:Method: GET 2024-02-01T10:48:15.041 DEBUG:boto:Path: /vclfcn-60/obj4?versionId=null 2024-02-01T10:48:15.041 DEBUG:boto:Data: 2024-02-01T10:48:15.041 DEBUG:boto:Headers: {} 2024-02-01T10:48:15.041 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:15.041 DEBUG:boto:Port: 8000 2024-02-01T10:48:15.041 DEBUG:boto:Params: {} 2024-02-01T10:48:15.042 DEBUG:boto:Token: None 2024-02-01T10:48:15.042 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:15 GMT /vclfcn-60/obj4?versionId=null 2024-02-01T10:48:15.042 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:kZm42ThmE+zXYO+AZFZtwBLBRIc= 2024-02-01T10:48:15.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:kZm42ThmE+zXYO+AZFZtwBLBRIc=', 'Content-Length': '0'} 2024-02-01T10:48:15.045 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c9265e4e475c8b5c-0065bb76ef-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:15.045 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:15.045 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:15.046 DEBUG:boto:path=/vclfcn-60/?prefix=obj4 2024-02-01T10:48:15.046 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj4 2024-02-01T10:48:15.046 DEBUG:boto:Method: GET 2024-02-01T10:48:15.046 DEBUG:boto:Path: /vclfcn-60/?prefix=obj4 2024-02-01T10:48:15.046 DEBUG:boto:Data: 2024-02-01T10:48:15.046 DEBUG:boto:Headers: {} 2024-02-01T10:48:15.046 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:15.046 DEBUG:boto:Port: 8001 2024-02-01T10:48:15.046 DEBUG:boto:Params: {} 2024-02-01T10:48:15.046 DEBUG:boto:Token: None 2024-02-01T10:48:15.046 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:15 GMT /vclfcn-60/ 2024-02-01T10:48:15.047 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Fiih1wD6PHvuNivmKQ49rkh2r34= 2024-02-01T10:48:15.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Fiih1wD6PHvuNivmKQ49rkh2r34=', 'Content-Length': '0'} 2024-02-01T10:48:15.052 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001dff7b87e810adbb-0065bb76ef-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:15.052 DEBUG:boto:b'vclfcn-60obj41000falseobj42024-02-01T10:46:46.336Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:15.053 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:15.053 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:15.053 DEBUG:boto:path=/vclfcn-60/?prefix=obj4 2024-02-01T10:48:15.053 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj4 2024-02-01T10:48:15.053 DEBUG:boto:Method: GET 2024-02-01T10:48:15.053 DEBUG:boto:Path: /vclfcn-60/?prefix=obj4 2024-02-01T10:48:15.053 DEBUG:boto:Data: 2024-02-01T10:48:15.053 DEBUG:boto:Headers: {} 2024-02-01T10:48:15.053 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:15.053 DEBUG:boto:Port: 8000 2024-02-01T10:48:15.053 DEBUG:boto:Params: {} 2024-02-01T10:48:15.054 DEBUG:boto:Token: None 2024-02-01T10:48:15.054 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:15 GMT /vclfcn-60/ 2024-02-01T10:48:15.054 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Fiih1wD6PHvuNivmKQ49rkh2r34= 2024-02-01T10:48:15.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Fiih1wD6PHvuNivmKQ49rkh2r34=', 'Content-Length': '0'} 2024-02-01T10:48:15.059 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002fd4423f71425936-0065bb76ef-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:15.100 DEBUG:boto:b'vclfcn-60obj41000falseobj42024-02-01T10:46:46.336Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:15.101 DEBUG:boto:path=/vclfcn-60/obj4 2024-02-01T10:48:15.101 DEBUG:boto:auth_path=/vclfcn-60/obj4 2024-02-01T10:48:15.101 DEBUG:boto:Method: HEAD 2024-02-01T10:48:15.101 DEBUG:boto:Path: /vclfcn-60/obj4 2024-02-01T10:48:15.101 DEBUG:boto:Data: 2024-02-01T10:48:15.101 DEBUG:boto:Headers: {} 2024-02-01T10:48:15.101 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:15.101 DEBUG:boto:Port: 8001 2024-02-01T10:48:15.101 DEBUG:boto:Params: {} 2024-02-01T10:48:15.101 DEBUG:boto:Token: None 2024-02-01T10:48:15.102 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:15 GMT /vclfcn-60/obj4 2024-02-01T10:48:15.102 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0hGDHW5IjOAfYolh8IWqaWL8xvc= 2024-02-01T10:48:15.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0hGDHW5IjOAfYolh8IWqaWL8xvc=', 'Content-Length': '0'} 2024-02-01T10:48:15.105 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003cc425986ff0a935-0065bb76ef-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:15.105 DEBUG:boto:path=/vclfcn-60/obj4 2024-02-01T10:48:15.105 DEBUG:boto:auth_path=/vclfcn-60/obj4 2024-02-01T10:48:15.105 DEBUG:boto:Method: HEAD 2024-02-01T10:48:15.105 DEBUG:boto:Path: /vclfcn-60/obj4 2024-02-01T10:48:15.105 DEBUG:boto:Data: 2024-02-01T10:48:15.106 DEBUG:boto:Headers: {} 2024-02-01T10:48:15.106 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:15.106 DEBUG:boto:Port: 8000 2024-02-01T10:48:15.106 DEBUG:boto:Params: {} 2024-02-01T10:48:15.106 DEBUG:boto:Token: None 2024-02-01T10:48:15.106 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:15 GMT /vclfcn-60/obj4 2024-02-01T10:48:15.106 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0hGDHW5IjOAfYolh8IWqaWL8xvc= 2024-02-01T10:48:15.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0hGDHW5IjOAfYolh8IWqaWL8xvc=', 'Content-Length': '0'} 2024-02-01T10:48:15.109 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000046c6e4f6bca9a1f2-0065bb76ef-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:15.109 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:48:15.109 DEBUG:boto:path=/vclfcn-60/obj4 2024-02-01T10:48:15.109 DEBUG:boto:auth_path=/vclfcn-60/obj4 2024-02-01T10:48:15.109 DEBUG:boto:Method: GET 2024-02-01T10:48:15.110 DEBUG:boto:Path: /vclfcn-60/obj4 2024-02-01T10:48:15.110 DEBUG:boto:Data: 2024-02-01T10:48:15.110 DEBUG:boto:Headers: {} 2024-02-01T10:48:15.110 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:15.110 DEBUG:boto:Port: 8001 2024-02-01T10:48:15.110 DEBUG:boto:Params: {} 2024-02-01T10:48:15.110 DEBUG:boto:Token: None 2024-02-01T10:48:15.110 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:15 GMT /vclfcn-60/obj4 2024-02-01T10:48:15.110 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:qt78gw3i53PFGjCNh6utHQ+qUr4= 2024-02-01T10:48:15.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:qt78gw3i53PFGjCNh6utHQ+qUr4=', 'Content-Length': '0'} 2024-02-01T10:48:15.113 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000464e692f727f04f1-0065bb76ef-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:15.156 DEBUG:boto:path=/vclfcn-60/obj4 2024-02-01T10:48:15.157 DEBUG:boto:auth_path=/vclfcn-60/obj4 2024-02-01T10:48:15.157 DEBUG:boto:Method: GET 2024-02-01T10:48:15.157 DEBUG:boto:Path: /vclfcn-60/obj4 2024-02-01T10:48:15.157 DEBUG:boto:Data: 2024-02-01T10:48:15.157 DEBUG:boto:Headers: {} 2024-02-01T10:48:15.157 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:15.157 DEBUG:boto:Port: 8000 2024-02-01T10:48:15.157 DEBUG:boto:Params: {} 2024-02-01T10:48:15.157 DEBUG:boto:Token: None 2024-02-01T10:48:15.158 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:15 GMT /vclfcn-60/obj4 2024-02-01T10:48:15.158 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:qt78gw3i53PFGjCNh6utHQ+qUr4= 2024-02-01T10:48:15.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:qt78gw3i53PFGjCNh6utHQ+qUr4=', 'Content-Length': '0'} 2024-02-01T10:48:15.161 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002228f6bb5619a963-0065bb76ef-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:15.161 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:15.161 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:15.161 DEBUG:boto:path=/vclfcn-60/?prefix=obj4 2024-02-01T10:48:15.161 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj4 2024-02-01T10:48:15.161 DEBUG:boto:Method: GET 2024-02-01T10:48:15.162 DEBUG:boto:Path: /vclfcn-60/?prefix=obj4 2024-02-01T10:48:15.162 DEBUG:boto:Data: 2024-02-01T10:48:15.162 DEBUG:boto:Headers: {} 2024-02-01T10:48:15.162 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:15.162 DEBUG:boto:Port: 8001 2024-02-01T10:48:15.162 DEBUG:boto:Params: {} 2024-02-01T10:48:15.162 DEBUG:boto:Token: None 2024-02-01T10:48:15.163 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:15 GMT /vclfcn-60/ 2024-02-01T10:48:15.163 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Fiih1wD6PHvuNivmKQ49rkh2r34= 2024-02-01T10:48:15.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Fiih1wD6PHvuNivmKQ49rkh2r34=', 'Content-Length': '0'} 2024-02-01T10:48:15.167 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000216c98b1bb879d63-0065bb76ef-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:15.167 DEBUG:boto:b'vclfcn-60obj41000falseobj42024-02-01T10:46:46.336Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:15.168 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:15.168 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:15.168 DEBUG:boto:path=/vclfcn-60/?prefix=obj4 2024-02-01T10:48:15.168 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj4 2024-02-01T10:48:15.168 DEBUG:boto:Method: GET 2024-02-01T10:48:15.168 DEBUG:boto:Path: /vclfcn-60/?prefix=obj4 2024-02-01T10:48:15.168 DEBUG:boto:Data: 2024-02-01T10:48:15.168 DEBUG:boto:Headers: {} 2024-02-01T10:48:15.168 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:15.168 DEBUG:boto:Port: 8000 2024-02-01T10:48:15.169 DEBUG:boto:Params: {} 2024-02-01T10:48:15.169 DEBUG:boto:Token: None 2024-02-01T10:48:15.169 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:15 GMT /vclfcn-60/ 2024-02-01T10:48:15.169 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Fiih1wD6PHvuNivmKQ49rkh2r34= 2024-02-01T10:48:15.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Fiih1wD6PHvuNivmKQ49rkh2r34=', 'Content-Length': '0'} 2024-02-01T10:48:15.174 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000868bed607eed54f2-0065bb76ef-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:15.174 DEBUG:boto:b'vclfcn-60obj41000falseobj42024-02-01T10:46:46.336Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:15.175 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-02-01T10:48:15.175 DEBUG:boto:path=/vclfcn-60/obj5 2024-02-01T10:48:15.175 DEBUG:boto:auth_path=/vclfcn-60/obj5 2024-02-01T10:48:15.175 DEBUG:boto:path=/vclfcn-60/obj5?versionId=null 2024-02-01T10:48:15.175 DEBUG:boto:auth_path=/vclfcn-60/obj5?versionId=null 2024-02-01T10:48:15.175 DEBUG:boto:Method: GET 2024-02-01T10:48:15.175 DEBUG:boto:Path: /vclfcn-60/obj5?versionId=null 2024-02-01T10:48:15.175 DEBUG:boto:Data: 2024-02-01T10:48:15.175 DEBUG:boto:Headers: {} 2024-02-01T10:48:15.176 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:15.176 DEBUG:boto:Port: 8001 2024-02-01T10:48:15.176 DEBUG:boto:Params: {} 2024-02-01T10:48:15.176 DEBUG:boto:Token: None 2024-02-01T10:48:15.176 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:15 GMT /vclfcn-60/obj5?versionId=null 2024-02-01T10:48:15.176 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:AIKGyRaZFcEUbZsoD+hKj3/M2zs= 2024-02-01T10:48:15.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:AIKGyRaZFcEUbZsoD+hKj3/M2zs=', 'Content-Length': '0'} 2024-02-01T10:48:15.180 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000425d927490ff3dd8-0065bb76ef-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:15.180 DEBUG:boto:path=/vclfcn-60/obj5 2024-02-01T10:48:15.180 DEBUG:boto:auth_path=/vclfcn-60/obj5 2024-02-01T10:48:15.180 DEBUG:boto:path=/vclfcn-60/obj5?versionId=null 2024-02-01T10:48:15.181 DEBUG:boto:auth_path=/vclfcn-60/obj5?versionId=null 2024-02-01T10:48:15.181 DEBUG:boto:Method: GET 2024-02-01T10:48:15.181 DEBUG:boto:Path: /vclfcn-60/obj5?versionId=null 2024-02-01T10:48:15.181 DEBUG:boto:Data: 2024-02-01T10:48:15.181 DEBUG:boto:Headers: {} 2024-02-01T10:48:15.181 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:15.181 DEBUG:boto:Port: 8000 2024-02-01T10:48:15.181 DEBUG:boto:Params: {} 2024-02-01T10:48:15.181 DEBUG:boto:Token: None 2024-02-01T10:48:15.182 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:15 GMT /vclfcn-60/obj5?versionId=null 2024-02-01T10:48:15.182 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:AIKGyRaZFcEUbZsoD+hKj3/M2zs= 2024-02-01T10:48:15.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:AIKGyRaZFcEUbZsoD+hKj3/M2zs=', 'Content-Length': '0'} 2024-02-01T10:48:15.185 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000066797e40cabb317f-0065bb76ef-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:15.235 DEBUG:boto:path=/vclfcn-60/obj5 2024-02-01T10:48:15.235 DEBUG:boto:auth_path=/vclfcn-60/obj5 2024-02-01T10:48:15.236 DEBUG:boto:path=/vclfcn-60/obj5?versionId=null 2024-02-01T10:48:15.236 DEBUG:boto:auth_path=/vclfcn-60/obj5?versionId=null 2024-02-01T10:48:15.236 DEBUG:boto:Method: HEAD 2024-02-01T10:48:15.236 DEBUG:boto:Path: /vclfcn-60/obj5?versionId=null 2024-02-01T10:48:15.236 DEBUG:boto:Data: 2024-02-01T10:48:15.236 DEBUG:boto:Headers: {} 2024-02-01T10:48:15.236 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:15.236 DEBUG:boto:Port: 8001 2024-02-01T10:48:15.236 DEBUG:boto:Params: {} 2024-02-01T10:48:15.236 DEBUG:boto:Token: None 2024-02-01T10:48:15.237 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:15 GMT /vclfcn-60/obj5?versionId=null 2024-02-01T10:48:15.237 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WmiUgzks4WtgQj3f/h6YH/vnEUo= 2024-02-01T10:48:15.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WmiUgzks4WtgQj3f/h6YH/vnEUo=', 'Content-Length': '0'} 2024-02-01T10:48:15.240 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006bc20ff6f445b92c-0065bb76ef-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:15.240 DEBUG:boto:path=/vclfcn-60/obj5 2024-02-01T10:48:15.240 DEBUG:boto:auth_path=/vclfcn-60/obj5 2024-02-01T10:48:15.240 DEBUG:boto:path=/vclfcn-60/obj5?versionId=null 2024-02-01T10:48:15.240 DEBUG:boto:auth_path=/vclfcn-60/obj5?versionId=null 2024-02-01T10:48:15.241 DEBUG:boto:Method: HEAD 2024-02-01T10:48:15.241 DEBUG:boto:Path: /vclfcn-60/obj5?versionId=null 2024-02-01T10:48:15.241 DEBUG:boto:Data: 2024-02-01T10:48:15.241 DEBUG:boto:Headers: {} 2024-02-01T10:48:15.241 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:15.241 DEBUG:boto:Port: 8000 2024-02-01T10:48:15.241 DEBUG:boto:Params: {} 2024-02-01T10:48:15.241 DEBUG:boto:Token: None 2024-02-01T10:48:15.241 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:15 GMT /vclfcn-60/obj5?versionId=null 2024-02-01T10:48:15.242 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WmiUgzks4WtgQj3f/h6YH/vnEUo= 2024-02-01T10:48:15.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WmiUgzks4WtgQj3f/h6YH/vnEUo=', 'Content-Length': '0'} 2024-02-01T10:48:15.244 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b492fd9aea466637-0065bb76ef-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:15.245 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-02-01T10:48:15.245 DEBUG:boto:path=/vclfcn-60/obj5 2024-02-01T10:48:15.245 DEBUG:boto:auth_path=/vclfcn-60/obj5 2024-02-01T10:48:15.245 DEBUG:boto:path=/vclfcn-60/obj5?versionId=null 2024-02-01T10:48:15.245 DEBUG:boto:auth_path=/vclfcn-60/obj5?versionId=null 2024-02-01T10:48:15.245 DEBUG:boto:Method: GET 2024-02-01T10:48:15.245 DEBUG:boto:Path: /vclfcn-60/obj5?versionId=null 2024-02-01T10:48:15.245 DEBUG:boto:Data: 2024-02-01T10:48:15.245 DEBUG:boto:Headers: {} 2024-02-01T10:48:15.246 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:15.246 DEBUG:boto:Port: 8001 2024-02-01T10:48:15.246 DEBUG:boto:Params: {} 2024-02-01T10:48:15.246 DEBUG:boto:Token: None 2024-02-01T10:48:15.246 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:15 GMT /vclfcn-60/obj5?versionId=null 2024-02-01T10:48:15.246 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:AIKGyRaZFcEUbZsoD+hKj3/M2zs= 2024-02-01T10:48:15.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:AIKGyRaZFcEUbZsoD+hKj3/M2zs=', 'Content-Length': '0'} 2024-02-01T10:48:15.250 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ac16a3a6544c3f56-0065bb76ef-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:15.292 DEBUG:boto:path=/vclfcn-60/obj5 2024-02-01T10:48:15.292 DEBUG:boto:auth_path=/vclfcn-60/obj5 2024-02-01T10:48:15.292 DEBUG:boto:path=/vclfcn-60/obj5?versionId=null 2024-02-01T10:48:15.292 DEBUG:boto:auth_path=/vclfcn-60/obj5?versionId=null 2024-02-01T10:48:15.293 DEBUG:boto:Method: GET 2024-02-01T10:48:15.293 DEBUG:boto:Path: /vclfcn-60/obj5?versionId=null 2024-02-01T10:48:15.293 DEBUG:boto:Data: 2024-02-01T10:48:15.293 DEBUG:boto:Headers: {} 2024-02-01T10:48:15.293 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:15.293 DEBUG:boto:Port: 8000 2024-02-01T10:48:15.293 DEBUG:boto:Params: {} 2024-02-01T10:48:15.293 DEBUG:boto:Token: None 2024-02-01T10:48:15.293 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:15 GMT /vclfcn-60/obj5?versionId=null 2024-02-01T10:48:15.294 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:AIKGyRaZFcEUbZsoD+hKj3/M2zs= 2024-02-01T10:48:15.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:AIKGyRaZFcEUbZsoD+hKj3/M2zs=', 'Content-Length': '0'} 2024-02-01T10:48:15.297 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000076723b553163a8f4-0065bb76ef-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:15.297 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:15.297 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:15.297 DEBUG:boto:path=/vclfcn-60/?prefix=obj5 2024-02-01T10:48:15.298 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj5 2024-02-01T10:48:15.298 DEBUG:boto:Method: GET 2024-02-01T10:48:15.298 DEBUG:boto:Path: /vclfcn-60/?prefix=obj5 2024-02-01T10:48:15.298 DEBUG:boto:Data: 2024-02-01T10:48:15.298 DEBUG:boto:Headers: {} 2024-02-01T10:48:15.298 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:15.298 DEBUG:boto:Port: 8001 2024-02-01T10:48:15.298 DEBUG:boto:Params: {} 2024-02-01T10:48:15.298 DEBUG:boto:Token: None 2024-02-01T10:48:15.298 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:15 GMT /vclfcn-60/ 2024-02-01T10:48:15.299 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Fiih1wD6PHvuNivmKQ49rkh2r34= 2024-02-01T10:48:15.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Fiih1wD6PHvuNivmKQ49rkh2r34=', 'Content-Length': '0'} 2024-02-01T10:48:15.303 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f5d24db3ac916051-0065bb76ef-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:15.344 DEBUG:boto:b'vclfcn-60obj51000falseobj52024-02-01T10:47:30.291Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:15.345 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:15.345 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:15.345 DEBUG:boto:path=/vclfcn-60/?prefix=obj5 2024-02-01T10:48:15.345 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj5 2024-02-01T10:48:15.345 DEBUG:boto:Method: GET 2024-02-01T10:48:15.345 DEBUG:boto:Path: /vclfcn-60/?prefix=obj5 2024-02-01T10:48:15.345 DEBUG:boto:Data: 2024-02-01T10:48:15.346 DEBUG:boto:Headers: {} 2024-02-01T10:48:15.346 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:15.346 DEBUG:boto:Port: 8000 2024-02-01T10:48:15.346 DEBUG:boto:Params: {} 2024-02-01T10:48:15.346 DEBUG:boto:Token: None 2024-02-01T10:48:15.346 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:15 GMT /vclfcn-60/ 2024-02-01T10:48:15.346 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Fiih1wD6PHvuNivmKQ49rkh2r34= 2024-02-01T10:48:15.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Fiih1wD6PHvuNivmKQ49rkh2r34=', 'Content-Length': '0'} 2024-02-01T10:48:15.351 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006cdd5da0bee7e9cd-0065bb76ef-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:15.351 DEBUG:boto:b'vclfcn-60obj51000falseobj52024-02-01T10:47:30.291Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:15.351 DEBUG:boto:path=/vclfcn-60/obj5 2024-02-01T10:48:15.351 DEBUG:boto:auth_path=/vclfcn-60/obj5 2024-02-01T10:48:15.352 DEBUG:boto:Method: HEAD 2024-02-01T10:48:15.352 DEBUG:boto:Path: /vclfcn-60/obj5 2024-02-01T10:48:15.352 DEBUG:boto:Data: 2024-02-01T10:48:15.352 DEBUG:boto:Headers: {} 2024-02-01T10:48:15.352 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:15.352 DEBUG:boto:Port: 8001 2024-02-01T10:48:15.352 DEBUG:boto:Params: {} 2024-02-01T10:48:15.352 DEBUG:boto:Token: None 2024-02-01T10:48:15.352 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:15 GMT /vclfcn-60/obj5 2024-02-01T10:48:15.353 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:B5DemTfzv98npIArMinTOPUkmgs= 2024-02-01T10:48:15.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:B5DemTfzv98npIArMinTOPUkmgs=', 'Content-Length': '0'} 2024-02-01T10:48:15.355 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000086f834bb86bd86af-0065bb76ef-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:15.355 DEBUG:boto:path=/vclfcn-60/obj5 2024-02-01T10:48:15.356 DEBUG:boto:auth_path=/vclfcn-60/obj5 2024-02-01T10:48:15.356 DEBUG:boto:Method: HEAD 2024-02-01T10:48:15.356 DEBUG:boto:Path: /vclfcn-60/obj5 2024-02-01T10:48:15.356 DEBUG:boto:Data: 2024-02-01T10:48:15.356 DEBUG:boto:Headers: {} 2024-02-01T10:48:15.356 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:15.356 DEBUG:boto:Port: 8000 2024-02-01T10:48:15.356 DEBUG:boto:Params: {} 2024-02-01T10:48:15.356 DEBUG:boto:Token: None 2024-02-01T10:48:15.357 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:15 GMT /vclfcn-60/obj5 2024-02-01T10:48:15.357 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:B5DemTfzv98npIArMinTOPUkmgs= 2024-02-01T10:48:15.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:B5DemTfzv98npIArMinTOPUkmgs=', 'Content-Length': '0'} 2024-02-01T10:48:15.359 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000038dca79ed98f8e95-0065bb76ef-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:15.360 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-02-01T10:48:15.360 DEBUG:boto:path=/vclfcn-60/obj5 2024-02-01T10:48:15.360 DEBUG:boto:auth_path=/vclfcn-60/obj5 2024-02-01T10:48:15.360 DEBUG:boto:Method: GET 2024-02-01T10:48:15.360 DEBUG:boto:Path: /vclfcn-60/obj5 2024-02-01T10:48:15.360 DEBUG:boto:Data: 2024-02-01T10:48:15.360 DEBUG:boto:Headers: {} 2024-02-01T10:48:15.361 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:15.361 DEBUG:boto:Port: 8001 2024-02-01T10:48:15.361 DEBUG:boto:Params: {} 2024-02-01T10:48:15.361 DEBUG:boto:Token: None 2024-02-01T10:48:15.361 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:15 GMT /vclfcn-60/obj5 2024-02-01T10:48:15.361 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:yjVM2Tc2MeYaXOpYvzWckCjgnOM= 2024-02-01T10:48:15.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:yjVM2Tc2MeYaXOpYvzWckCjgnOM=', 'Content-Length': '0'} 2024-02-01T10:48:15.364 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dadc82641a7a8f89-0065bb76ef-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:15.408 DEBUG:boto:path=/vclfcn-60/obj5 2024-02-01T10:48:15.409 DEBUG:boto:auth_path=/vclfcn-60/obj5 2024-02-01T10:48:15.409 DEBUG:boto:Method: GET 2024-02-01T10:48:15.409 DEBUG:boto:Path: /vclfcn-60/obj5 2024-02-01T10:48:15.409 DEBUG:boto:Data: 2024-02-01T10:48:15.409 DEBUG:boto:Headers: {} 2024-02-01T10:48:15.409 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:15.409 DEBUG:boto:Port: 8000 2024-02-01T10:48:15.409 DEBUG:boto:Params: {} 2024-02-01T10:48:15.409 DEBUG:boto:Token: None 2024-02-01T10:48:15.410 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:15 GMT /vclfcn-60/obj5 2024-02-01T10:48:15.410 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:yjVM2Tc2MeYaXOpYvzWckCjgnOM= 2024-02-01T10:48:15.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:yjVM2Tc2MeYaXOpYvzWckCjgnOM=', 'Content-Length': '0'} 2024-02-01T10:48:15.413 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000682bf6339bb3387b-0065bb76ef-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:15.413 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:15.414 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:15.414 DEBUG:boto:path=/vclfcn-60/?prefix=obj5 2024-02-01T10:48:15.414 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj5 2024-02-01T10:48:15.414 DEBUG:boto:Method: GET 2024-02-01T10:48:15.414 DEBUG:boto:Path: /vclfcn-60/?prefix=obj5 2024-02-01T10:48:15.414 DEBUG:boto:Data: 2024-02-01T10:48:15.415 DEBUG:boto:Headers: {} 2024-02-01T10:48:15.415 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:15.415 DEBUG:boto:Port: 8001 2024-02-01T10:48:15.415 DEBUG:boto:Params: {} 2024-02-01T10:48:15.415 DEBUG:boto:Token: None 2024-02-01T10:48:15.415 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:15 GMT /vclfcn-60/ 2024-02-01T10:48:15.416 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Fiih1wD6PHvuNivmKQ49rkh2r34= 2024-02-01T10:48:15.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Fiih1wD6PHvuNivmKQ49rkh2r34=', 'Content-Length': '0'} 2024-02-01T10:48:15.421 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b40fd07d24450104-0065bb76ef-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:15.421 DEBUG:boto:b'vclfcn-60obj51000falseobj52024-02-01T10:47:30.291Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:15.422 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:15.422 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:15.423 DEBUG:boto:path=/vclfcn-60/?prefix=obj5 2024-02-01T10:48:15.423 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj5 2024-02-01T10:48:15.423 DEBUG:boto:Method: GET 2024-02-01T10:48:15.423 DEBUG:boto:Path: /vclfcn-60/?prefix=obj5 2024-02-01T10:48:15.423 DEBUG:boto:Data: 2024-02-01T10:48:15.423 DEBUG:boto:Headers: {} 2024-02-01T10:48:15.423 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:15.423 DEBUG:boto:Port: 8000 2024-02-01T10:48:15.423 DEBUG:boto:Params: {} 2024-02-01T10:48:15.423 DEBUG:boto:Token: None 2024-02-01T10:48:15.424 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:15 GMT /vclfcn-60/ 2024-02-01T10:48:15.424 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Fiih1wD6PHvuNivmKQ49rkh2r34= 2024-02-01T10:48:15.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Fiih1wD6PHvuNivmKQ49rkh2r34=', 'Content-Length': '0'} 2024-02-01T10:48:15.428 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7e8774d1e41ad16-0065bb76ef-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:15.472 DEBUG:boto:b'vclfcn-60obj51000falseobj52024-02-01T10:47:30.291Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:15.473 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-02-01T10:48:15.473 DEBUG:boto:path=/vclfcn-60/obj6 2024-02-01T10:48:15.473 DEBUG:boto:auth_path=/vclfcn-60/obj6 2024-02-01T10:48:15.473 DEBUG:boto:path=/vclfcn-60/obj6?versionId=null 2024-02-01T10:48:15.473 DEBUG:boto:auth_path=/vclfcn-60/obj6?versionId=null 2024-02-01T10:48:15.473 DEBUG:boto:Method: GET 2024-02-01T10:48:15.473 DEBUG:boto:Path: /vclfcn-60/obj6?versionId=null 2024-02-01T10:48:15.473 DEBUG:boto:Data: 2024-02-01T10:48:15.474 DEBUG:boto:Headers: {} 2024-02-01T10:48:15.474 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:15.474 DEBUG:boto:Port: 8001 2024-02-01T10:48:15.474 DEBUG:boto:Params: {} 2024-02-01T10:48:15.474 DEBUG:boto:Token: None 2024-02-01T10:48:15.474 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:15 GMT /vclfcn-60/obj6?versionId=null 2024-02-01T10:48:15.474 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:2y1/9LahmRzjN19kQzPp3dyvv7Q= 2024-02-01T10:48:15.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2y1/9LahmRzjN19kQzPp3dyvv7Q=', 'Content-Length': '0'} 2024-02-01T10:48:15.478 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000de00ffa6af1fcf7b-0065bb76ef-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:15.524 DEBUG:boto:path=/vclfcn-60/obj6 2024-02-01T10:48:15.524 DEBUG:boto:auth_path=/vclfcn-60/obj6 2024-02-01T10:48:15.524 DEBUG:boto:path=/vclfcn-60/obj6?versionId=null 2024-02-01T10:48:15.524 DEBUG:boto:auth_path=/vclfcn-60/obj6?versionId=null 2024-02-01T10:48:15.525 DEBUG:boto:Method: GET 2024-02-01T10:48:15.525 DEBUG:boto:Path: /vclfcn-60/obj6?versionId=null 2024-02-01T10:48:15.525 DEBUG:boto:Data: 2024-02-01T10:48:15.525 DEBUG:boto:Headers: {} 2024-02-01T10:48:15.525 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:15.525 DEBUG:boto:Port: 8000 2024-02-01T10:48:15.525 DEBUG:boto:Params: {} 2024-02-01T10:48:15.525 DEBUG:boto:Token: None 2024-02-01T10:48:15.525 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:15 GMT /vclfcn-60/obj6?versionId=null 2024-02-01T10:48:15.526 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:2y1/9LahmRzjN19kQzPp3dyvv7Q= 2024-02-01T10:48:15.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2y1/9LahmRzjN19kQzPp3dyvv7Q=', 'Content-Length': '0'} 2024-02-01T10:48:15.528 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000061b7fe26e63ed71a-0065bb76ef-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:15.529 DEBUG:boto:path=/vclfcn-60/obj6 2024-02-01T10:48:15.529 DEBUG:boto:auth_path=/vclfcn-60/obj6 2024-02-01T10:48:15.529 DEBUG:boto:path=/vclfcn-60/obj6?versionId=null 2024-02-01T10:48:15.529 DEBUG:boto:auth_path=/vclfcn-60/obj6?versionId=null 2024-02-01T10:48:15.529 DEBUG:boto:Method: HEAD 2024-02-01T10:48:15.529 DEBUG:boto:Path: /vclfcn-60/obj6?versionId=null 2024-02-01T10:48:15.529 DEBUG:boto:Data: 2024-02-01T10:48:15.529 DEBUG:boto:Headers: {} 2024-02-01T10:48:15.529 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:15.530 DEBUG:boto:Port: 8001 2024-02-01T10:48:15.530 DEBUG:boto:Params: {} 2024-02-01T10:48:15.530 DEBUG:boto:Token: None 2024-02-01T10:48:15.530 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:15 GMT /vclfcn-60/obj6?versionId=null 2024-02-01T10:48:15.530 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:aTPs9mHsi/3ami5q2bLE5fS5IeM= 2024-02-01T10:48:15.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:aTPs9mHsi/3ami5q2bLE5fS5IeM=', 'Content-Length': '0'} 2024-02-01T10:48:15.533 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008a325952b64562a4-0065bb76ef-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:15.534 DEBUG:boto:path=/vclfcn-60/obj6 2024-02-01T10:48:15.534 DEBUG:boto:auth_path=/vclfcn-60/obj6 2024-02-01T10:48:15.534 DEBUG:boto:path=/vclfcn-60/obj6?versionId=null 2024-02-01T10:48:15.534 DEBUG:boto:auth_path=/vclfcn-60/obj6?versionId=null 2024-02-01T10:48:15.534 DEBUG:boto:Method: HEAD 2024-02-01T10:48:15.534 DEBUG:boto:Path: /vclfcn-60/obj6?versionId=null 2024-02-01T10:48:15.535 DEBUG:boto:Data: 2024-02-01T10:48:15.535 DEBUG:boto:Headers: {} 2024-02-01T10:48:15.535 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:15.535 DEBUG:boto:Port: 8000 2024-02-01T10:48:15.535 DEBUG:boto:Params: {} 2024-02-01T10:48:15.535 DEBUG:boto:Token: None 2024-02-01T10:48:15.535 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:15 GMT /vclfcn-60/obj6?versionId=null 2024-02-01T10:48:15.535 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:aTPs9mHsi/3ami5q2bLE5fS5IeM= 2024-02-01T10:48:15.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:aTPs9mHsi/3ami5q2bLE5fS5IeM=', 'Content-Length': '0'} 2024-02-01T10:48:15.538 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000053f069bdc99c279e-0065bb76ef-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:15.538 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-02-01T10:48:15.538 DEBUG:boto:path=/vclfcn-60/obj6 2024-02-01T10:48:15.539 DEBUG:boto:auth_path=/vclfcn-60/obj6 2024-02-01T10:48:15.539 DEBUG:boto:path=/vclfcn-60/obj6?versionId=null 2024-02-01T10:48:15.539 DEBUG:boto:auth_path=/vclfcn-60/obj6?versionId=null 2024-02-01T10:48:15.539 DEBUG:boto:Method: GET 2024-02-01T10:48:15.539 DEBUG:boto:Path: /vclfcn-60/obj6?versionId=null 2024-02-01T10:48:15.539 DEBUG:boto:Data: 2024-02-01T10:48:15.539 DEBUG:boto:Headers: {} 2024-02-01T10:48:15.539 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:15.539 DEBUG:boto:Port: 8001 2024-02-01T10:48:15.539 DEBUG:boto:Params: {} 2024-02-01T10:48:15.540 DEBUG:boto:Token: None 2024-02-01T10:48:15.540 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:15 GMT /vclfcn-60/obj6?versionId=null 2024-02-01T10:48:15.540 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:2y1/9LahmRzjN19kQzPp3dyvv7Q= 2024-02-01T10:48:15.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2y1/9LahmRzjN19kQzPp3dyvv7Q=', 'Content-Length': '0'} 2024-02-01T10:48:15.543 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000081064cf405356a6b-0065bb76ef-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:15.584 DEBUG:boto:path=/vclfcn-60/obj6 2024-02-01T10:48:15.584 DEBUG:boto:auth_path=/vclfcn-60/obj6 2024-02-01T10:48:15.584 DEBUG:boto:path=/vclfcn-60/obj6?versionId=null 2024-02-01T10:48:15.584 DEBUG:boto:auth_path=/vclfcn-60/obj6?versionId=null 2024-02-01T10:48:15.584 DEBUG:boto:Method: GET 2024-02-01T10:48:15.584 DEBUG:boto:Path: /vclfcn-60/obj6?versionId=null 2024-02-01T10:48:15.585 DEBUG:boto:Data: 2024-02-01T10:48:15.585 DEBUG:boto:Headers: {} 2024-02-01T10:48:15.585 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:15.585 DEBUG:boto:Port: 8000 2024-02-01T10:48:15.585 DEBUG:boto:Params: {} 2024-02-01T10:48:15.585 DEBUG:boto:Token: None 2024-02-01T10:48:15.585 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:15 GMT /vclfcn-60/obj6?versionId=null 2024-02-01T10:48:15.585 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:2y1/9LahmRzjN19kQzPp3dyvv7Q= 2024-02-01T10:48:15.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2y1/9LahmRzjN19kQzPp3dyvv7Q=', 'Content-Length': '0'} 2024-02-01T10:48:15.588 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006f54e3771ef15fd6-0065bb76ef-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:15.588 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:15.589 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:15.589 DEBUG:boto:path=/vclfcn-60/?prefix=obj6 2024-02-01T10:48:15.589 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj6 2024-02-01T10:48:15.589 DEBUG:boto:Method: GET 2024-02-01T10:48:15.589 DEBUG:boto:Path: /vclfcn-60/?prefix=obj6 2024-02-01T10:48:15.589 DEBUG:boto:Data: 2024-02-01T10:48:15.589 DEBUG:boto:Headers: {} 2024-02-01T10:48:15.589 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:15.589 DEBUG:boto:Port: 8001 2024-02-01T10:48:15.589 DEBUG:boto:Params: {} 2024-02-01T10:48:15.589 DEBUG:boto:Token: None 2024-02-01T10:48:15.590 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:15 GMT /vclfcn-60/ 2024-02-01T10:48:15.590 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Fiih1wD6PHvuNivmKQ49rkh2r34= 2024-02-01T10:48:15.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Fiih1wD6PHvuNivmKQ49rkh2r34=', 'Content-Length': '0'} 2024-02-01T10:48:15.594 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b75bda2c01345dfc-0065bb76ef-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:15.636 DEBUG:boto:b'vclfcn-60obj61000falseobj62024-02-01T10:47:30.343Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:15.637 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:15.637 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:15.638 DEBUG:boto:path=/vclfcn-60/?prefix=obj6 2024-02-01T10:48:15.638 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj6 2024-02-01T10:48:15.638 DEBUG:boto:Method: GET 2024-02-01T10:48:15.638 DEBUG:boto:Path: /vclfcn-60/?prefix=obj6 2024-02-01T10:48:15.638 DEBUG:boto:Data: 2024-02-01T10:48:15.639 DEBUG:boto:Headers: {} 2024-02-01T10:48:15.639 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:15.639 DEBUG:boto:Port: 8000 2024-02-01T10:48:15.639 DEBUG:boto:Params: {} 2024-02-01T10:48:15.640 DEBUG:boto:Token: None 2024-02-01T10:48:15.640 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:15 GMT /vclfcn-60/ 2024-02-01T10:48:15.640 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Fiih1wD6PHvuNivmKQ49rkh2r34= 2024-02-01T10:48:15.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Fiih1wD6PHvuNivmKQ49rkh2r34=', 'Content-Length': '0'} 2024-02-01T10:48:15.646 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003dede72ca4119cd4-0065bb76ef-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:15.646 DEBUG:boto:b'vclfcn-60obj61000falseobj62024-02-01T10:47:30.343Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:15.648 DEBUG:boto:path=/vclfcn-60/obj6 2024-02-01T10:48:15.648 DEBUG:boto:auth_path=/vclfcn-60/obj6 2024-02-01T10:48:15.648 DEBUG:boto:Method: HEAD 2024-02-01T10:48:15.648 DEBUG:boto:Path: /vclfcn-60/obj6 2024-02-01T10:48:15.648 DEBUG:boto:Data: 2024-02-01T10:48:15.649 DEBUG:boto:Headers: {} 2024-02-01T10:48:15.649 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:15.649 DEBUG:boto:Port: 8001 2024-02-01T10:48:15.649 DEBUG:boto:Params: {} 2024-02-01T10:48:15.649 DEBUG:boto:Token: None 2024-02-01T10:48:15.649 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:15 GMT /vclfcn-60/obj6 2024-02-01T10:48:15.649 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:efYyG8Nd/gxQ6Tdu/mtJYOnXTI8= 2024-02-01T10:48:15.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:efYyG8Nd/gxQ6Tdu/mtJYOnXTI8=', 'Content-Length': '0'} 2024-02-01T10:48:15.652 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c6559c33a3b0dd50-0065bb76ef-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:15.653 DEBUG:boto:path=/vclfcn-60/obj6 2024-02-01T10:48:15.653 DEBUG:boto:auth_path=/vclfcn-60/obj6 2024-02-01T10:48:15.653 DEBUG:boto:Method: HEAD 2024-02-01T10:48:15.653 DEBUG:boto:Path: /vclfcn-60/obj6 2024-02-01T10:48:15.653 DEBUG:boto:Data: 2024-02-01T10:48:15.653 DEBUG:boto:Headers: {} 2024-02-01T10:48:15.653 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:15.653 DEBUG:boto:Port: 8000 2024-02-01T10:48:15.653 DEBUG:boto:Params: {} 2024-02-01T10:48:15.653 DEBUG:boto:Token: None 2024-02-01T10:48:15.654 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:15 GMT /vclfcn-60/obj6 2024-02-01T10:48:15.654 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:efYyG8Nd/gxQ6Tdu/mtJYOnXTI8= 2024-02-01T10:48:15.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:efYyG8Nd/gxQ6Tdu/mtJYOnXTI8=', 'Content-Length': '0'} 2024-02-01T10:48:15.658 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000971b83b5c25c9b26-0065bb76ef-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:15.658 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-02-01T10:48:15.658 DEBUG:boto:path=/vclfcn-60/obj6 2024-02-01T10:48:15.658 DEBUG:boto:auth_path=/vclfcn-60/obj6 2024-02-01T10:48:15.658 DEBUG:boto:Method: GET 2024-02-01T10:48:15.658 DEBUG:boto:Path: /vclfcn-60/obj6 2024-02-01T10:48:15.658 DEBUG:boto:Data: 2024-02-01T10:48:15.659 DEBUG:boto:Headers: {} 2024-02-01T10:48:15.659 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:15.659 DEBUG:boto:Port: 8001 2024-02-01T10:48:15.659 DEBUG:boto:Params: {} 2024-02-01T10:48:15.659 DEBUG:boto:Token: None 2024-02-01T10:48:15.659 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:15 GMT /vclfcn-60/obj6 2024-02-01T10:48:15.659 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NcZwkZFjDM1zc62aDUmJ6LScpow= 2024-02-01T10:48:15.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NcZwkZFjDM1zc62aDUmJ6LScpow=', 'Content-Length': '0'} 2024-02-01T10:48:15.662 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bd5aafcc242f4608-0065bb76ef-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:15.704 DEBUG:boto:path=/vclfcn-60/obj6 2024-02-01T10:48:15.704 DEBUG:boto:auth_path=/vclfcn-60/obj6 2024-02-01T10:48:15.704 DEBUG:boto:Method: GET 2024-02-01T10:48:15.705 DEBUG:boto:Path: /vclfcn-60/obj6 2024-02-01T10:48:15.705 DEBUG:boto:Data: 2024-02-01T10:48:15.705 DEBUG:boto:Headers: {} 2024-02-01T10:48:15.705 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:15.705 DEBUG:boto:Port: 8000 2024-02-01T10:48:15.705 DEBUG:boto:Params: {} 2024-02-01T10:48:15.706 DEBUG:boto:Token: None 2024-02-01T10:48:15.706 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:15 GMT /vclfcn-60/obj6 2024-02-01T10:48:15.706 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NcZwkZFjDM1zc62aDUmJ6LScpow= 2024-02-01T10:48:15.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NcZwkZFjDM1zc62aDUmJ6LScpow=', 'Content-Length': '0'} 2024-02-01T10:48:15.709 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006872d8e676325982-0065bb76ef-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:15.709 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:15.710 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:15.710 DEBUG:boto:path=/vclfcn-60/?prefix=obj6 2024-02-01T10:48:15.710 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj6 2024-02-01T10:48:15.710 DEBUG:boto:Method: GET 2024-02-01T10:48:15.710 DEBUG:boto:Path: /vclfcn-60/?prefix=obj6 2024-02-01T10:48:15.710 DEBUG:boto:Data: 2024-02-01T10:48:15.710 DEBUG:boto:Headers: {} 2024-02-01T10:48:15.710 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:15.710 DEBUG:boto:Port: 8001 2024-02-01T10:48:15.710 DEBUG:boto:Params: {} 2024-02-01T10:48:15.711 DEBUG:boto:Token: None 2024-02-01T10:48:15.711 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:15 GMT /vclfcn-60/ 2024-02-01T10:48:15.711 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Fiih1wD6PHvuNivmKQ49rkh2r34= 2024-02-01T10:48:15.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Fiih1wD6PHvuNivmKQ49rkh2r34=', 'Content-Length': '0'} 2024-02-01T10:48:15.716 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000003314707a397a397-0065bb76ef-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:15.756 DEBUG:boto:b'vclfcn-60obj61000falseobj62024-02-01T10:47:30.343Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:15.756 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:15.757 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:15.757 DEBUG:boto:path=/vclfcn-60/?prefix=obj6 2024-02-01T10:48:15.757 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj6 2024-02-01T10:48:15.757 DEBUG:boto:Method: GET 2024-02-01T10:48:15.757 DEBUG:boto:Path: /vclfcn-60/?prefix=obj6 2024-02-01T10:48:15.757 DEBUG:boto:Data: 2024-02-01T10:48:15.757 DEBUG:boto:Headers: {} 2024-02-01T10:48:15.757 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:15.757 DEBUG:boto:Port: 8000 2024-02-01T10:48:15.757 DEBUG:boto:Params: {} 2024-02-01T10:48:15.758 DEBUG:boto:Token: None 2024-02-01T10:48:15.758 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:15 GMT /vclfcn-60/ 2024-02-01T10:48:15.758 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Fiih1wD6PHvuNivmKQ49rkh2r34= 2024-02-01T10:48:15.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Fiih1wD6PHvuNivmKQ49rkh2r34=', 'Content-Length': '0'} 2024-02-01T10:48:15.762 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000538b923b122dd258-0065bb76ef-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:15.763 DEBUG:boto:b'vclfcn-60obj61000falseobj62024-02-01T10:47:30.343Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:15.763 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-02-01T10:48:15.763 DEBUG:boto:path=/vclfcn-60/obj7 2024-02-01T10:48:15.763 DEBUG:boto:auth_path=/vclfcn-60/obj7 2024-02-01T10:48:15.763 DEBUG:boto:path=/vclfcn-60/obj7?versionId=null 2024-02-01T10:48:15.763 DEBUG:boto:auth_path=/vclfcn-60/obj7?versionId=null 2024-02-01T10:48:15.764 DEBUG:boto:Method: GET 2024-02-01T10:48:15.764 DEBUG:boto:Path: /vclfcn-60/obj7?versionId=null 2024-02-01T10:48:15.764 DEBUG:boto:Data: 2024-02-01T10:48:15.764 DEBUG:boto:Headers: {} 2024-02-01T10:48:15.764 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:15.764 DEBUG:boto:Port: 8001 2024-02-01T10:48:15.764 DEBUG:boto:Params: {} 2024-02-01T10:48:15.764 DEBUG:boto:Token: None 2024-02-01T10:48:15.764 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:15 GMT /vclfcn-60/obj7?versionId=null 2024-02-01T10:48:15.765 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RPiEYQs4UgX3WDQCMtlME244IWg= 2024-02-01T10:48:15.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RPiEYQs4UgX3WDQCMtlME244IWg=', 'Content-Length': '0'} 2024-02-01T10:48:15.767 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002d6d25d7fbeb2a54-0065bb76ef-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:15.767 DEBUG:boto:path=/vclfcn-60/obj7 2024-02-01T10:48:15.768 DEBUG:boto:auth_path=/vclfcn-60/obj7 2024-02-01T10:48:15.768 DEBUG:boto:path=/vclfcn-60/obj7?versionId=null 2024-02-01T10:48:15.768 DEBUG:boto:auth_path=/vclfcn-60/obj7?versionId=null 2024-02-01T10:48:15.768 DEBUG:boto:Method: GET 2024-02-01T10:48:15.768 DEBUG:boto:Path: /vclfcn-60/obj7?versionId=null 2024-02-01T10:48:15.768 DEBUG:boto:Data: 2024-02-01T10:48:15.768 DEBUG:boto:Headers: {} 2024-02-01T10:48:15.768 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:15.768 DEBUG:boto:Port: 8000 2024-02-01T10:48:15.768 DEBUG:boto:Params: {} 2024-02-01T10:48:15.768 DEBUG:boto:Token: None 2024-02-01T10:48:15.769 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:15 GMT /vclfcn-60/obj7?versionId=null 2024-02-01T10:48:15.769 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RPiEYQs4UgX3WDQCMtlME244IWg= 2024-02-01T10:48:15.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RPiEYQs4UgX3WDQCMtlME244IWg=', 'Content-Length': '0'} 2024-02-01T10:48:15.771 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004a3762ffcbacaf13-0065bb76ef-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:15.812 DEBUG:boto:path=/vclfcn-60/obj7 2024-02-01T10:48:15.812 DEBUG:boto:auth_path=/vclfcn-60/obj7 2024-02-01T10:48:15.812 DEBUG:boto:path=/vclfcn-60/obj7?versionId=null 2024-02-01T10:48:15.813 DEBUG:boto:auth_path=/vclfcn-60/obj7?versionId=null 2024-02-01T10:48:15.813 DEBUG:boto:Method: HEAD 2024-02-01T10:48:15.813 DEBUG:boto:Path: /vclfcn-60/obj7?versionId=null 2024-02-01T10:48:15.813 DEBUG:boto:Data: 2024-02-01T10:48:15.813 DEBUG:boto:Headers: {} 2024-02-01T10:48:15.813 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:15.813 DEBUG:boto:Port: 8001 2024-02-01T10:48:15.813 DEBUG:boto:Params: {} 2024-02-01T10:48:15.813 DEBUG:boto:Token: None 2024-02-01T10:48:15.813 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:15 GMT /vclfcn-60/obj7?versionId=null 2024-02-01T10:48:15.814 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lxnJ82pNq5oFbewPPLqIV84tfsQ= 2024-02-01T10:48:15.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lxnJ82pNq5oFbewPPLqIV84tfsQ=', 'Content-Length': '0'} 2024-02-01T10:48:15.817 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e1b2618d857f550b-0065bb76ef-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:15.817 DEBUG:boto:path=/vclfcn-60/obj7 2024-02-01T10:48:15.817 DEBUG:boto:auth_path=/vclfcn-60/obj7 2024-02-01T10:48:15.817 DEBUG:boto:path=/vclfcn-60/obj7?versionId=null 2024-02-01T10:48:15.817 DEBUG:boto:auth_path=/vclfcn-60/obj7?versionId=null 2024-02-01T10:48:15.817 DEBUG:boto:Method: HEAD 2024-02-01T10:48:15.818 DEBUG:boto:Path: /vclfcn-60/obj7?versionId=null 2024-02-01T10:48:15.818 DEBUG:boto:Data: 2024-02-01T10:48:15.818 DEBUG:boto:Headers: {} 2024-02-01T10:48:15.818 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:15.818 DEBUG:boto:Port: 8000 2024-02-01T10:48:15.818 DEBUG:boto:Params: {} 2024-02-01T10:48:15.818 DEBUG:boto:Token: None 2024-02-01T10:48:15.818 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:15 GMT /vclfcn-60/obj7?versionId=null 2024-02-01T10:48:15.818 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lxnJ82pNq5oFbewPPLqIV84tfsQ= 2024-02-01T10:48:15.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lxnJ82pNq5oFbewPPLqIV84tfsQ=', 'Content-Length': '0'} 2024-02-01T10:48:15.821 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b2fa056c2d0e99fd-0065bb76ef-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:15.821 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-02-01T10:48:15.822 DEBUG:boto:path=/vclfcn-60/obj7 2024-02-01T10:48:15.822 DEBUG:boto:auth_path=/vclfcn-60/obj7 2024-02-01T10:48:15.822 DEBUG:boto:path=/vclfcn-60/obj7?versionId=null 2024-02-01T10:48:15.822 DEBUG:boto:auth_path=/vclfcn-60/obj7?versionId=null 2024-02-01T10:48:15.822 DEBUG:boto:Method: GET 2024-02-01T10:48:15.822 DEBUG:boto:Path: /vclfcn-60/obj7?versionId=null 2024-02-01T10:48:15.822 DEBUG:boto:Data: 2024-02-01T10:48:15.822 DEBUG:boto:Headers: {} 2024-02-01T10:48:15.822 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:15.822 DEBUG:boto:Port: 8001 2024-02-01T10:48:15.822 DEBUG:boto:Params: {} 2024-02-01T10:48:15.823 DEBUG:boto:Token: None 2024-02-01T10:48:15.823 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:15 GMT /vclfcn-60/obj7?versionId=null 2024-02-01T10:48:15.823 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RPiEYQs4UgX3WDQCMtlME244IWg= 2024-02-01T10:48:15.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RPiEYQs4UgX3WDQCMtlME244IWg=', 'Content-Length': '0'} 2024-02-01T10:48:15.826 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cad7a9bb7e356340-0065bb76ef-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:15.868 DEBUG:boto:path=/vclfcn-60/obj7 2024-02-01T10:48:15.868 DEBUG:boto:auth_path=/vclfcn-60/obj7 2024-02-01T10:48:15.868 DEBUG:boto:path=/vclfcn-60/obj7?versionId=null 2024-02-01T10:48:15.868 DEBUG:boto:auth_path=/vclfcn-60/obj7?versionId=null 2024-02-01T10:48:15.869 DEBUG:boto:Method: GET 2024-02-01T10:48:15.869 DEBUG:boto:Path: /vclfcn-60/obj7?versionId=null 2024-02-01T10:48:15.869 DEBUG:boto:Data: 2024-02-01T10:48:15.869 DEBUG:boto:Headers: {} 2024-02-01T10:48:15.869 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:15.869 DEBUG:boto:Port: 8000 2024-02-01T10:48:15.869 DEBUG:boto:Params: {} 2024-02-01T10:48:15.869 DEBUG:boto:Token: None 2024-02-01T10:48:15.869 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:15 GMT /vclfcn-60/obj7?versionId=null 2024-02-01T10:48:15.870 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RPiEYQs4UgX3WDQCMtlME244IWg= 2024-02-01T10:48:15.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RPiEYQs4UgX3WDQCMtlME244IWg=', 'Content-Length': '0'} 2024-02-01T10:48:15.872 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005f7b4ebf32f22a64-0065bb76ef-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:15.873 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:15.873 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:15.873 DEBUG:boto:path=/vclfcn-60/?prefix=obj7 2024-02-01T10:48:15.873 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj7 2024-02-01T10:48:15.873 DEBUG:boto:Method: GET 2024-02-01T10:48:15.873 DEBUG:boto:Path: /vclfcn-60/?prefix=obj7 2024-02-01T10:48:15.873 DEBUG:boto:Data: 2024-02-01T10:48:15.874 DEBUG:boto:Headers: {} 2024-02-01T10:48:15.874 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:15.874 DEBUG:boto:Port: 8001 2024-02-01T10:48:15.874 DEBUG:boto:Params: {} 2024-02-01T10:48:15.874 DEBUG:boto:Token: None 2024-02-01T10:48:15.874 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:15 GMT /vclfcn-60/ 2024-02-01T10:48:15.874 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Fiih1wD6PHvuNivmKQ49rkh2r34= 2024-02-01T10:48:15.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Fiih1wD6PHvuNivmKQ49rkh2r34=', 'Content-Length': '0'} 2024-02-01T10:48:15.879 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac65169c8fcd5cc2-0065bb76ef-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:15.924 DEBUG:boto:b'vclfcn-60obj71000falseobj72024-02-01T10:47:30.358Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:15.924 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:15.925 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:15.925 DEBUG:boto:path=/vclfcn-60/?prefix=obj7 2024-02-01T10:48:15.925 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj7 2024-02-01T10:48:15.925 DEBUG:boto:Method: GET 2024-02-01T10:48:15.925 DEBUG:boto:Path: /vclfcn-60/?prefix=obj7 2024-02-01T10:48:15.925 DEBUG:boto:Data: 2024-02-01T10:48:15.925 DEBUG:boto:Headers: {} 2024-02-01T10:48:15.925 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:15.925 DEBUG:boto:Port: 8000 2024-02-01T10:48:15.925 DEBUG:boto:Params: {} 2024-02-01T10:48:15.926 DEBUG:boto:Token: None 2024-02-01T10:48:15.926 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:15 GMT /vclfcn-60/ 2024-02-01T10:48:15.926 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Fiih1wD6PHvuNivmKQ49rkh2r34= 2024-02-01T10:48:15.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Fiih1wD6PHvuNivmKQ49rkh2r34=', 'Content-Length': '0'} 2024-02-01T10:48:15.931 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb6e42f17a24b1ef-0065bb76ef-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:15.931 DEBUG:boto:b'vclfcn-60obj71000falseobj72024-02-01T10:47:30.358Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:15.931 DEBUG:boto:path=/vclfcn-60/obj7 2024-02-01T10:48:15.931 DEBUG:boto:auth_path=/vclfcn-60/obj7 2024-02-01T10:48:15.932 DEBUG:boto:Method: HEAD 2024-02-01T10:48:15.932 DEBUG:boto:Path: /vclfcn-60/obj7 2024-02-01T10:48:15.932 DEBUG:boto:Data: 2024-02-01T10:48:15.932 DEBUG:boto:Headers: {} 2024-02-01T10:48:15.932 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:15.932 DEBUG:boto:Port: 8001 2024-02-01T10:48:15.932 DEBUG:boto:Params: {} 2024-02-01T10:48:15.933 DEBUG:boto:Token: None 2024-02-01T10:48:15.933 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:15 GMT /vclfcn-60/obj7 2024-02-01T10:48:15.933 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jJ5uPWCEyCC+NmRJ4/eNnl6Wnpo= 2024-02-01T10:48:15.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jJ5uPWCEyCC+NmRJ4/eNnl6Wnpo=', 'Content-Length': '0'} 2024-02-01T10:48:15.936 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e1f4a6f8022ce598-0065bb76ef-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:15.937 DEBUG:boto:path=/vclfcn-60/obj7 2024-02-01T10:48:15.937 DEBUG:boto:auth_path=/vclfcn-60/obj7 2024-02-01T10:48:15.937 DEBUG:boto:Method: HEAD 2024-02-01T10:48:15.938 DEBUG:boto:Path: /vclfcn-60/obj7 2024-02-01T10:48:15.938 DEBUG:boto:Data: 2024-02-01T10:48:15.938 DEBUG:boto:Headers: {} 2024-02-01T10:48:15.938 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:15.938 DEBUG:boto:Port: 8000 2024-02-01T10:48:15.939 DEBUG:boto:Params: {} 2024-02-01T10:48:15.939 DEBUG:boto:Token: None 2024-02-01T10:48:15.939 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:15 GMT /vclfcn-60/obj7 2024-02-01T10:48:15.939 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jJ5uPWCEyCC+NmRJ4/eNnl6Wnpo= 2024-02-01T10:48:15.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jJ5uPWCEyCC+NmRJ4/eNnl6Wnpo=', 'Content-Length': '0'} 2024-02-01T10:48:15.942 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000008e39c600ad09e84-0065bb76ef-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:15.943 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-02-01T10:48:15.943 DEBUG:boto:path=/vclfcn-60/obj7 2024-02-01T10:48:15.943 DEBUG:boto:auth_path=/vclfcn-60/obj7 2024-02-01T10:48:15.943 DEBUG:boto:Method: GET 2024-02-01T10:48:15.943 DEBUG:boto:Path: /vclfcn-60/obj7 2024-02-01T10:48:15.943 DEBUG:boto:Data: 2024-02-01T10:48:15.943 DEBUG:boto:Headers: {} 2024-02-01T10:48:15.943 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:15.943 DEBUG:boto:Port: 8001 2024-02-01T10:48:15.943 DEBUG:boto:Params: {} 2024-02-01T10:48:15.944 DEBUG:boto:Token: None 2024-02-01T10:48:15.944 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:15 GMT /vclfcn-60/obj7 2024-02-01T10:48:15.944 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0gwtEpI1b13k7hewgbA9hcBKjuo= 2024-02-01T10:48:15.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0gwtEpI1b13k7hewgbA9hcBKjuo=', 'Content-Length': '0'} 2024-02-01T10:48:15.947 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000df4201df500f2a2-0065bb76ef-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:15.988 DEBUG:boto:path=/vclfcn-60/obj7 2024-02-01T10:48:15.988 DEBUG:boto:auth_path=/vclfcn-60/obj7 2024-02-01T10:48:15.988 DEBUG:boto:Method: GET 2024-02-01T10:48:15.988 DEBUG:boto:Path: /vclfcn-60/obj7 2024-02-01T10:48:15.989 DEBUG:boto:Data: 2024-02-01T10:48:15.989 DEBUG:boto:Headers: {} 2024-02-01T10:48:15.989 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:15.989 DEBUG:boto:Port: 8000 2024-02-01T10:48:15.989 DEBUG:boto:Params: {} 2024-02-01T10:48:15.989 DEBUG:boto:Token: None 2024-02-01T10:48:15.989 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:15 GMT /vclfcn-60/obj7 2024-02-01T10:48:15.989 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0gwtEpI1b13k7hewgbA9hcBKjuo= 2024-02-01T10:48:15.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0gwtEpI1b13k7hewgbA9hcBKjuo=', 'Content-Length': '0'} 2024-02-01T10:48:15.992 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b985acfd082c439e-0065bb76ef-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:15.993 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:15.993 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:15.993 DEBUG:boto:path=/vclfcn-60/?prefix=obj7 2024-02-01T10:48:15.993 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj7 2024-02-01T10:48:15.993 DEBUG:boto:Method: GET 2024-02-01T10:48:15.993 DEBUG:boto:Path: /vclfcn-60/?prefix=obj7 2024-02-01T10:48:15.993 DEBUG:boto:Data: 2024-02-01T10:48:15.993 DEBUG:boto:Headers: {} 2024-02-01T10:48:15.993 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:15.994 DEBUG:boto:Port: 8001 2024-02-01T10:48:15.994 DEBUG:boto:Params: {} 2024-02-01T10:48:15.994 DEBUG:boto:Token: None 2024-02-01T10:48:15.994 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:15 GMT /vclfcn-60/ 2024-02-01T10:48:15.994 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Fiih1wD6PHvuNivmKQ49rkh2r34= 2024-02-01T10:48:15.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Fiih1wD6PHvuNivmKQ49rkh2r34=', 'Content-Length': '0'} 2024-02-01T10:48:15.999 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd056b176960ca5e-0065bb76ef-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:16.040 DEBUG:boto:b'vclfcn-60obj71000falseobj72024-02-01T10:47:30.358Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:16.041 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:16.041 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:16.041 DEBUG:boto:path=/vclfcn-60/?prefix=obj7 2024-02-01T10:48:16.041 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj7 2024-02-01T10:48:16.041 DEBUG:boto:Method: GET 2024-02-01T10:48:16.041 DEBUG:boto:Path: /vclfcn-60/?prefix=obj7 2024-02-01T10:48:16.041 DEBUG:boto:Data: 2024-02-01T10:48:16.041 DEBUG:boto:Headers: {} 2024-02-01T10:48:16.041 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:16.042 DEBUG:boto:Port: 8000 2024-02-01T10:48:16.042 DEBUG:boto:Params: {} 2024-02-01T10:48:16.042 DEBUG:boto:Token: None 2024-02-01T10:48:16.042 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:16 GMT /vclfcn-60/ 2024-02-01T10:48:16.042 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:aTGQxyTJ0XGHlcwUAb+vEx1YqrY= 2024-02-01T10:48:16.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:aTGQxyTJ0XGHlcwUAb+vEx1YqrY=', 'Content-Length': '0'} 2024-02-01T10:48:16.047 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000032f38ea45c66209c-0065bb76f0-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:16.047 DEBUG:boto:b'vclfcn-60obj71000falseobj72024-02-01T10:47:30.358Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:16.048 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-02-01T10:48:16.048 DEBUG:boto:path=/vclfcn-60/obj8 2024-02-01T10:48:16.048 DEBUG:boto:auth_path=/vclfcn-60/obj8 2024-02-01T10:48:16.048 DEBUG:boto:path=/vclfcn-60/obj8?versionId=null 2024-02-01T10:48:16.048 DEBUG:boto:auth_path=/vclfcn-60/obj8?versionId=null 2024-02-01T10:48:16.052 DEBUG:boto:Method: GET 2024-02-01T10:48:16.052 DEBUG:boto:Path: /vclfcn-60/obj8?versionId=null 2024-02-01T10:48:16.052 DEBUG:boto:Data: 2024-02-01T10:48:16.052 DEBUG:boto:Headers: {} 2024-02-01T10:48:16.052 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:16.052 DEBUG:boto:Port: 8001 2024-02-01T10:48:16.052 DEBUG:boto:Params: {} 2024-02-01T10:48:16.052 DEBUG:boto:Token: None 2024-02-01T10:48:16.053 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:16 GMT /vclfcn-60/obj8?versionId=null 2024-02-01T10:48:16.053 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BnVmtJlGuVw2Hqj5ROrlraDMklI= 2024-02-01T10:48:16.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BnVmtJlGuVw2Hqj5ROrlraDMklI=', 'Content-Length': '0'} 2024-02-01T10:48:16.056 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006175cc1e738206e1-0065bb76f0-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:16.056 DEBUG:boto:path=/vclfcn-60/obj8 2024-02-01T10:48:16.056 DEBUG:boto:auth_path=/vclfcn-60/obj8 2024-02-01T10:48:16.056 DEBUG:boto:path=/vclfcn-60/obj8?versionId=null 2024-02-01T10:48:16.056 DEBUG:boto:auth_path=/vclfcn-60/obj8?versionId=null 2024-02-01T10:48:16.056 DEBUG:boto:Method: GET 2024-02-01T10:48:16.056 DEBUG:boto:Path: /vclfcn-60/obj8?versionId=null 2024-02-01T10:48:16.056 DEBUG:boto:Data: 2024-02-01T10:48:16.057 DEBUG:boto:Headers: {} 2024-02-01T10:48:16.057 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:16.057 DEBUG:boto:Port: 8000 2024-02-01T10:48:16.057 DEBUG:boto:Params: {} 2024-02-01T10:48:16.057 DEBUG:boto:Token: None 2024-02-01T10:48:16.057 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:16 GMT /vclfcn-60/obj8?versionId=null 2024-02-01T10:48:16.057 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BnVmtJlGuVw2Hqj5ROrlraDMklI= 2024-02-01T10:48:16.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BnVmtJlGuVw2Hqj5ROrlraDMklI=', 'Content-Length': '0'} 2024-02-01T10:48:16.060 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c84011b3ab975647-0065bb76f0-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:16.104 DEBUG:boto:path=/vclfcn-60/obj8 2024-02-01T10:48:16.104 DEBUG:boto:auth_path=/vclfcn-60/obj8 2024-02-01T10:48:16.104 DEBUG:boto:path=/vclfcn-60/obj8?versionId=null 2024-02-01T10:48:16.104 DEBUG:boto:auth_path=/vclfcn-60/obj8?versionId=null 2024-02-01T10:48:16.105 DEBUG:boto:Method: HEAD 2024-02-01T10:48:16.105 DEBUG:boto:Path: /vclfcn-60/obj8?versionId=null 2024-02-01T10:48:16.105 DEBUG:boto:Data: 2024-02-01T10:48:16.105 DEBUG:boto:Headers: {} 2024-02-01T10:48:16.105 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:16.105 DEBUG:boto:Port: 8001 2024-02-01T10:48:16.105 DEBUG:boto:Params: {} 2024-02-01T10:48:16.105 DEBUG:boto:Token: None 2024-02-01T10:48:16.105 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:16 GMT /vclfcn-60/obj8?versionId=null 2024-02-01T10:48:16.105 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5/ejUSGz/Ijk8UmlEW+2+l0Tlvk= 2024-02-01T10:48:16.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5/ejUSGz/Ijk8UmlEW+2+l0Tlvk=', 'Content-Length': '0'} 2024-02-01T10:48:16.108 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000083ae04559c8753d3-0065bb76f0-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:16.109 DEBUG:boto:path=/vclfcn-60/obj8 2024-02-01T10:48:16.109 DEBUG:boto:auth_path=/vclfcn-60/obj8 2024-02-01T10:48:16.109 DEBUG:boto:path=/vclfcn-60/obj8?versionId=null 2024-02-01T10:48:16.109 DEBUG:boto:auth_path=/vclfcn-60/obj8?versionId=null 2024-02-01T10:48:16.109 DEBUG:boto:Method: HEAD 2024-02-01T10:48:16.109 DEBUG:boto:Path: /vclfcn-60/obj8?versionId=null 2024-02-01T10:48:16.109 DEBUG:boto:Data: 2024-02-01T10:48:16.109 DEBUG:boto:Headers: {} 2024-02-01T10:48:16.110 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:16.110 DEBUG:boto:Port: 8000 2024-02-01T10:48:16.110 DEBUG:boto:Params: {} 2024-02-01T10:48:16.110 DEBUG:boto:Token: None 2024-02-01T10:48:16.110 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:16 GMT /vclfcn-60/obj8?versionId=null 2024-02-01T10:48:16.110 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5/ejUSGz/Ijk8UmlEW+2+l0Tlvk= 2024-02-01T10:48:16.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5/ejUSGz/Ijk8UmlEW+2+l0Tlvk=', 'Content-Length': '0'} 2024-02-01T10:48:16.113 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001a7d1ee62efb3ef3-0065bb76f0-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:16.113 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-02-01T10:48:16.113 DEBUG:boto:path=/vclfcn-60/obj8 2024-02-01T10:48:16.114 DEBUG:boto:auth_path=/vclfcn-60/obj8 2024-02-01T10:48:16.114 DEBUG:boto:path=/vclfcn-60/obj8?versionId=null 2024-02-01T10:48:16.114 DEBUG:boto:auth_path=/vclfcn-60/obj8?versionId=null 2024-02-01T10:48:16.114 DEBUG:boto:Method: GET 2024-02-01T10:48:16.114 DEBUG:boto:Path: /vclfcn-60/obj8?versionId=null 2024-02-01T10:48:16.114 DEBUG:boto:Data: 2024-02-01T10:48:16.114 DEBUG:boto:Headers: {} 2024-02-01T10:48:16.114 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:16.114 DEBUG:boto:Port: 8001 2024-02-01T10:48:16.114 DEBUG:boto:Params: {} 2024-02-01T10:48:16.115 DEBUG:boto:Token: None 2024-02-01T10:48:16.115 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:16 GMT /vclfcn-60/obj8?versionId=null 2024-02-01T10:48:16.115 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BnVmtJlGuVw2Hqj5ROrlraDMklI= 2024-02-01T10:48:16.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BnVmtJlGuVw2Hqj5ROrlraDMklI=', 'Content-Length': '0'} 2024-02-01T10:48:16.118 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000680f19f74ea89960-0065bb76f0-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:16.160 DEBUG:boto:path=/vclfcn-60/obj8 2024-02-01T10:48:16.160 DEBUG:boto:auth_path=/vclfcn-60/obj8 2024-02-01T10:48:16.160 DEBUG:boto:path=/vclfcn-60/obj8?versionId=null 2024-02-01T10:48:16.160 DEBUG:boto:auth_path=/vclfcn-60/obj8?versionId=null 2024-02-01T10:48:16.160 DEBUG:boto:Method: GET 2024-02-01T10:48:16.161 DEBUG:boto:Path: /vclfcn-60/obj8?versionId=null 2024-02-01T10:48:16.161 DEBUG:boto:Data: 2024-02-01T10:48:16.161 DEBUG:boto:Headers: {} 2024-02-01T10:48:16.161 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:16.161 DEBUG:boto:Port: 8000 2024-02-01T10:48:16.161 DEBUG:boto:Params: {} 2024-02-01T10:48:16.161 DEBUG:boto:Token: None 2024-02-01T10:48:16.162 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:16 GMT /vclfcn-60/obj8?versionId=null 2024-02-01T10:48:16.162 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BnVmtJlGuVw2Hqj5ROrlraDMklI= 2024-02-01T10:48:16.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BnVmtJlGuVw2Hqj5ROrlraDMklI=', 'Content-Length': '0'} 2024-02-01T10:48:16.165 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000087fa8db084f09d8a-0065bb76f0-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:16.165 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:16.165 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:16.165 DEBUG:boto:path=/vclfcn-60/?prefix=obj8 2024-02-01T10:48:16.165 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj8 2024-02-01T10:48:16.165 DEBUG:boto:Method: GET 2024-02-01T10:48:16.166 DEBUG:boto:Path: /vclfcn-60/?prefix=obj8 2024-02-01T10:48:16.166 DEBUG:boto:Data: 2024-02-01T10:48:16.166 DEBUG:boto:Headers: {} 2024-02-01T10:48:16.166 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:16.166 DEBUG:boto:Port: 8001 2024-02-01T10:48:16.166 DEBUG:boto:Params: {} 2024-02-01T10:48:16.166 DEBUG:boto:Token: None 2024-02-01T10:48:16.166 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:16 GMT /vclfcn-60/ 2024-02-01T10:48:16.166 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:aTGQxyTJ0XGHlcwUAb+vEx1YqrY= 2024-02-01T10:48:16.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:aTGQxyTJ0XGHlcwUAb+vEx1YqrY=', 'Content-Length': '0'} 2024-02-01T10:48:16.171 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f97f1b3e29b5c16-0065bb76f0-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:16.216 DEBUG:boto:b'vclfcn-60obj81000falseobj82024-02-01T10:47:30.408Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:16.217 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:16.217 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:16.217 DEBUG:boto:path=/vclfcn-60/?prefix=obj8 2024-02-01T10:48:16.217 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj8 2024-02-01T10:48:16.217 DEBUG:boto:Method: GET 2024-02-01T10:48:16.217 DEBUG:boto:Path: /vclfcn-60/?prefix=obj8 2024-02-01T10:48:16.217 DEBUG:boto:Data: 2024-02-01T10:48:16.217 DEBUG:boto:Headers: {} 2024-02-01T10:48:16.217 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:16.217 DEBUG:boto:Port: 8000 2024-02-01T10:48:16.217 DEBUG:boto:Params: {} 2024-02-01T10:48:16.218 DEBUG:boto:Token: None 2024-02-01T10:48:16.218 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:16 GMT /vclfcn-60/ 2024-02-01T10:48:16.218 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:aTGQxyTJ0XGHlcwUAb+vEx1YqrY= 2024-02-01T10:48:16.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:aTGQxyTJ0XGHlcwUAb+vEx1YqrY=', 'Content-Length': '0'} 2024-02-01T10:48:16.222 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072e459e0100158d7-0065bb76f0-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:16.223 DEBUG:boto:b'vclfcn-60obj81000falseobj82024-02-01T10:47:30.408Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:16.223 DEBUG:boto:path=/vclfcn-60/obj8 2024-02-01T10:48:16.223 DEBUG:boto:auth_path=/vclfcn-60/obj8 2024-02-01T10:48:16.223 DEBUG:boto:Method: HEAD 2024-02-01T10:48:16.224 DEBUG:boto:Path: /vclfcn-60/obj8 2024-02-01T10:48:16.224 DEBUG:boto:Data: 2024-02-01T10:48:16.224 DEBUG:boto:Headers: {} 2024-02-01T10:48:16.224 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:16.224 DEBUG:boto:Port: 8001 2024-02-01T10:48:16.224 DEBUG:boto:Params: {} 2024-02-01T10:48:16.224 DEBUG:boto:Token: None 2024-02-01T10:48:16.224 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:16 GMT /vclfcn-60/obj8 2024-02-01T10:48:16.224 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:SRdUIrUsD60kYjyc/opr+RLSJyw= 2024-02-01T10:48:16.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:SRdUIrUsD60kYjyc/opr+RLSJyw=', 'Content-Length': '0'} 2024-02-01T10:48:16.227 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000960906db7d20b363-0065bb76f0-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:16.227 DEBUG:boto:path=/vclfcn-60/obj8 2024-02-01T10:48:16.228 DEBUG:boto:auth_path=/vclfcn-60/obj8 2024-02-01T10:48:16.228 DEBUG:boto:Method: HEAD 2024-02-01T10:48:16.228 DEBUG:boto:Path: /vclfcn-60/obj8 2024-02-01T10:48:16.228 DEBUG:boto:Data: 2024-02-01T10:48:16.228 DEBUG:boto:Headers: {} 2024-02-01T10:48:16.228 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:16.228 DEBUG:boto:Port: 8000 2024-02-01T10:48:16.228 DEBUG:boto:Params: {} 2024-02-01T10:48:16.228 DEBUG:boto:Token: None 2024-02-01T10:48:16.229 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:16 GMT /vclfcn-60/obj8 2024-02-01T10:48:16.229 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:SRdUIrUsD60kYjyc/opr+RLSJyw= 2024-02-01T10:48:16.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:SRdUIrUsD60kYjyc/opr+RLSJyw=', 'Content-Length': '0'} 2024-02-01T10:48:16.232 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f69be4518ee98326-0065bb76f0-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:16.233 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-02-01T10:48:16.233 DEBUG:boto:path=/vclfcn-60/obj8 2024-02-01T10:48:16.233 DEBUG:boto:auth_path=/vclfcn-60/obj8 2024-02-01T10:48:16.233 DEBUG:boto:Method: GET 2024-02-01T10:48:16.233 DEBUG:boto:Path: /vclfcn-60/obj8 2024-02-01T10:48:16.233 DEBUG:boto:Data: 2024-02-01T10:48:16.233 DEBUG:boto:Headers: {} 2024-02-01T10:48:16.233 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:16.233 DEBUG:boto:Port: 8001 2024-02-01T10:48:16.234 DEBUG:boto:Params: {} 2024-02-01T10:48:16.234 DEBUG:boto:Token: None 2024-02-01T10:48:16.234 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:16 GMT /vclfcn-60/obj8 2024-02-01T10:48:16.234 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:S+qCXXfySDg+1fwj9VNDExC7qWI= 2024-02-01T10:48:16.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:S+qCXXfySDg+1fwj9VNDExC7qWI=', 'Content-Length': '0'} 2024-02-01T10:48:16.237 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-60:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a2d614101e28b0d7-0065bb76f0-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:16.280 DEBUG:boto:path=/vclfcn-60/obj8 2024-02-01T10:48:16.280 DEBUG:boto:auth_path=/vclfcn-60/obj8 2024-02-01T10:48:16.281 DEBUG:boto:Method: GET 2024-02-01T10:48:16.281 DEBUG:boto:Path: /vclfcn-60/obj8 2024-02-01T10:48:16.281 DEBUG:boto:Data: 2024-02-01T10:48:16.281 DEBUG:boto:Headers: {} 2024-02-01T10:48:16.281 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:16.281 DEBUG:boto:Port: 8000 2024-02-01T10:48:16.281 DEBUG:boto:Params: {} 2024-02-01T10:48:16.281 DEBUG:boto:Token: None 2024-02-01T10:48:16.281 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:16 GMT /vclfcn-60/obj8 2024-02-01T10:48:16.282 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:S+qCXXfySDg+1fwj9VNDExC7qWI= 2024-02-01T10:48:16.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:S+qCXXfySDg+1fwj9VNDExC7qWI=', 'Content-Length': '0'} 2024-02-01T10:48:16.285 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f79448c0721b593b-0065bb76f0-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:16.286 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:16.286 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:16.287 DEBUG:boto:path=/vclfcn-60/?prefix=obj8 2024-02-01T10:48:16.287 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj8 2024-02-01T10:48:16.287 DEBUG:boto:Method: GET 2024-02-01T10:48:16.287 DEBUG:boto:Path: /vclfcn-60/?prefix=obj8 2024-02-01T10:48:16.287 DEBUG:boto:Data: 2024-02-01T10:48:16.287 DEBUG:boto:Headers: {} 2024-02-01T10:48:16.287 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:16.287 DEBUG:boto:Port: 8001 2024-02-01T10:48:16.287 DEBUG:boto:Params: {} 2024-02-01T10:48:16.287 DEBUG:boto:Token: None 2024-02-01T10:48:16.287 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:16 GMT /vclfcn-60/ 2024-02-01T10:48:16.288 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:aTGQxyTJ0XGHlcwUAb+vEx1YqrY= 2024-02-01T10:48:16.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:aTGQxyTJ0XGHlcwUAb+vEx1YqrY=', 'Content-Length': '0'} 2024-02-01T10:48:16.292 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000231011dcdbc9a67f-0065bb76f0-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:16.292 DEBUG:boto:b'vclfcn-60obj81000falseobj82024-02-01T10:47:30.408Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:16.292 DEBUG:boto:path=/vclfcn-60/ 2024-02-01T10:48:16.293 DEBUG:boto:auth_path=/vclfcn-60/ 2024-02-01T10:48:16.293 DEBUG:boto:path=/vclfcn-60/?prefix=obj8 2024-02-01T10:48:16.293 DEBUG:boto:auth_path=/vclfcn-60/?prefix=obj8 2024-02-01T10:48:16.293 DEBUG:boto:Method: GET 2024-02-01T10:48:16.293 DEBUG:boto:Path: /vclfcn-60/?prefix=obj8 2024-02-01T10:48:16.293 DEBUG:boto:Data: 2024-02-01T10:48:16.293 DEBUG:boto:Headers: {} 2024-02-01T10:48:16.293 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:16.293 DEBUG:boto:Port: 8000 2024-02-01T10:48:16.294 DEBUG:boto:Params: {} 2024-02-01T10:48:16.294 DEBUG:boto:Token: None 2024-02-01T10:48:16.294 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:16 GMT /vclfcn-60/ 2024-02-01T10:48:16.294 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:aTGQxyTJ0XGHlcwUAb+vEx1YqrY= 2024-02-01T10:48:16.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:aTGQxyTJ0XGHlcwUAb+vEx1YqrY=', 'Content-Length': '0'} 2024-02-01T10:48:16.298 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d71830d78e1206f-0065bb76f0-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:16.340 DEBUG:boto:b'vclfcn-60obj81000falseobj82024-02-01T10:47:30.408Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:16.341 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-60 zones={test-zone3, test-zone2} 2024-02-01T10:48:16.341 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=vclfcn-61 2024-02-01T10:48:16.341 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:48:16.341 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:48:16.341 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket vclfcn-61 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-02-01T10:48:16.391 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:48:16.392 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:48:16.456 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:48:16.453+0000 7fe6e1944980 1 bucket sync caught up with source: 2024-02-01T10:48:16.456 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [00000000004.64.6, 00000000002.66.1, 00000000004.45.6, 00000000002.43.1, 00000000004.167.6, 00000000002.67.1, 00000000002.57.1, 00000000004.51.6, 00000000002.81.1, 00000000002.83.1, 00000000002.58.1] 2024-02-01T10:48:16.457 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [00000000004.64.6, 00000000002.66.1, 00000000004.45.6, 00000000002.43.1, 00000000004.167.6, 00000000002.67.1, 00000000002.57.1, 00000000004.51.6, 00000000002.81.1, 00000000002.83.1, 00000000002.58.1] 2024-02-01T10:48:16.457 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:48:16.453+0000 7fe6e1944980 0 bucket checkpoint complete 2024-02-01T10:48:16.481 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=vclfcn-61 2024-02-01T10:48:16.482 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:48:16.482 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:48:16.482 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-61 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:48:16.532 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:48:16.532 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:48:16.594 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:48:16.592+0000 7f4dde86d980 1 bucket sync caught up with source: 2024-02-01T10:48:16.594 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [00000000004.64.6, 00000000002.66.1, 00000000004.45.6, 00000000002.43.1, 00000000004.167.6, 00000000002.67.1, 00000000002.57.1, 00000000004.51.6, 00000000002.81.1, 00000000002.83.1, 00000000002.58.1] 2024-02-01T10:48:16.594 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [00000000004.64.6, 00000000002.66.1, 00000000004.45.6, 00000000002.43.1, 00000000004.167.6, 00000000002.67.1, 00000000002.57.1, 00000000004.51.6, 00000000002.81.1, 00000000002.83.1, 00000000002.58.1] 2024-02-01T10:48:16.594 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:48:16.592+0000 7f4dde86d980 0 bucket checkpoint complete 2024-02-01T10:48:16.617 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=vclfcn-61 2024-02-01T10:48:16.617 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-61', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:48:16.617 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-61', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:48:16.617 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-61 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:48:16.657 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:48:16.657 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:48:16.719 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:48:16.716+0000 7f3e491fb980 1 bucket sync caught up with source: 2024-02-01T10:48:16.720 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [00000000002.254.6, , 00000000002.244.6, , 00000000002.427.6, , , 00000000002.284.6, , , ] 2024-02-01T10:48:16.720 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [00000000002.254.6, , 00000000002.244.6, , 00000000002.427.6, , , 00000000002.284.6, , , ] 2024-02-01T10:48:16.720 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:48:16.716+0000 7f3e491fb980 0 bucket checkpoint complete 2024-02-01T10:48:16.743 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=vclfcn-61 2024-02-01T10:48:16.743 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-61', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:48:16.743 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-61', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:48:16.743 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-61 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:48:16.828 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:48:16.829 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:48:16.889 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:48:16.888+0000 7fcda1ed5980 1 bucket sync caught up with source: 2024-02-01T10:48:16.889 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [00000000002.254.6, , 00000000002.244.6, , 00000000002.427.6, , , 00000000002.284.6, , , ] 2024-02-01T10:48:16.890 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [00000000002.254.6, , 00000000002.244.6, , 00000000002.427.6, , , 00000000002.284.6, , , ] 2024-02-01T10:48:16.890 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:48:16.888+0000 7fcda1ed5980 0 bucket checkpoint complete 2024-02-01T10:48:16.912 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=vclfcn-61 2024-02-01T10:48:16.912 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-61', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:48:16.912 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-61', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:48:16.912 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-61 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:48:16.952 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:48:16.952 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:48:17.010 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:48:17.008+0000 7fd564ce9980 1 bucket sync caught up with source: 2024-02-01T10:48:17.010 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [00000000004.213.1, 00000000002.205.1, 00000000004.191.1, 00000000002.192.1, 00000000004.366.1, 00000000002.204.1, 00000000002.181.1, 00000000004.176.1, 00000000002.304.1, 00000000002.317.1, 00000000002.183.1] 2024-02-01T10:48:17.010 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [00000000004.213.1, 00000000002.205.1, 00000000004.191.1, 00000000002.192.1, 00000000004.366.1, 00000000002.204.1, 00000000002.181.1, 00000000004.176.1, 00000000002.304.1, 00000000002.317.1, 00000000002.183.1] 2024-02-01T10:48:17.010 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:48:17.008+0000 7fd564ce9980 0 bucket checkpoint complete 2024-02-01T10:48:17.033 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=vclfcn-61 2024-02-01T10:48:17.033 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-61', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:48:17.034 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-61', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:48:17.034 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket vclfcn-61 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-02-01T10:48:17.084 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:48:17.085 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:48:17.150 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:48:17.149+0000 7f03869c6980 1 bucket sync caught up with source: 2024-02-01T10:48:17.151 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [00000000004.213.1, 00000000002.205.1, 00000000004.191.1, 00000000002.192.1, 00000000004.366.1, 00000000002.204.1, 00000000002.181.1, 00000000004.176.1, 00000000002.304.1, 00000000002.317.1, 00000000002.183.1] 2024-02-01T10:48:17.151 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [00000000004.213.1, 00000000002.205.1, 00000000004.191.1, 00000000002.192.1, 00000000004.366.1, 00000000002.204.1, 00000000002.181.1, 00000000004.176.1, 00000000002.304.1, 00000000002.317.1, 00000000002.183.1] 2024-02-01T10:48:17.151 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:48:17.149+0000 7f03869c6980 0 bucket checkpoint complete 2024-02-01T10:48:17.174 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-61 zones={test-zone2, test-zone1} 2024-02-01T10:48:17.174 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:17.175 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:17.175 DEBUG:boto:Method: HEAD 2024-02-01T10:48:17.175 DEBUG:boto:Path: /vclfcn-61/ 2024-02-01T10:48:17.175 DEBUG:boto:Data: 2024-02-01T10:48:17.175 DEBUG:boto:Headers: {} 2024-02-01T10:48:17.175 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:17.175 DEBUG:boto:Port: 8000 2024-02-01T10:48:17.175 DEBUG:boto:Params: {} 2024-02-01T10:48:17.175 DEBUG:boto:Token: None 2024-02-01T10:48:17.176 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:17 GMT /vclfcn-61/ 2024-02-01T10:48:17.176 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:yNOPKQMv3QcG6vTBH4/JTU6oXEk= 2024-02-01T10:48:17.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:yNOPKQMv3QcG6vTBH4/JTU6oXEk=', 'Content-Length': '0'} 2024-02-01T10:48:17.178 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dbd55b70b97ab3a5-0065bb76f1-5350-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:17.178 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:17.178 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:17.179 DEBUG:boto:Method: HEAD 2024-02-01T10:48:17.179 DEBUG:boto:Path: /vclfcn-61/ 2024-02-01T10:48:17.179 DEBUG:boto:Data: 2024-02-01T10:48:17.179 DEBUG:boto:Headers: {} 2024-02-01T10:48:17.179 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:17.179 DEBUG:boto:Port: 8000 2024-02-01T10:48:17.179 DEBUG:boto:Params: {} 2024-02-01T10:48:17.179 DEBUG:boto:Token: None 2024-02-01T10:48:17.179 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:17 GMT /vclfcn-61/ 2024-02-01T10:48:17.179 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:yNOPKQMv3QcG6vTBH4/JTU6oXEk= 2024-02-01T10:48:17.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:yNOPKQMv3QcG6vTBH4/JTU6oXEk=', 'Content-Length': '0'} 2024-02-01T10:48:17.183 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f30ebf3e4e240f04-0065bb76f1-5765-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:17.184 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:48:17.184 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:17.184 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:17.184 DEBUG:boto:path=/vclfcn-61/?versions&max-keys=999 2024-02-01T10:48:17.184 DEBUG:boto:auth_path=/vclfcn-61/?versions&max-keys=999 2024-02-01T10:48:17.184 DEBUG:boto:Method: GET 2024-02-01T10:48:17.184 DEBUG:boto:Path: /vclfcn-61/?versions&max-keys=999 2024-02-01T10:48:17.185 DEBUG:boto:Data: 2024-02-01T10:48:17.185 DEBUG:boto:Headers: {} 2024-02-01T10:48:17.185 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:17.185 DEBUG:boto:Port: 8000 2024-02-01T10:48:17.185 DEBUG:boto:Params: {} 2024-02-01T10:48:17.185 DEBUG:boto:Token: None 2024-02-01T10:48:17.185 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:17 GMT /vclfcn-61/?versions 2024-02-01T10:48:17.185 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Q01hnIxKyga+0YrHzPyNQhHrydg= 2024-02-01T10:48:17.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Q01hnIxKyga+0YrHzPyNQhHrydg=', 'Content-Length': '0'} 2024-02-01T10:48:17.188 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f501d916c4c5f1a-0065bb76f1-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:17.188 DEBUG:boto:b'vclfcn-61999falseobj1nulltrue2024-02-01T10:46:46.359Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:46:46.380Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:46:46.433Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:46:46.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-02-01T10:47:30.425Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-02-01T10:47:30.440Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-02-01T10:47:30.492Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-02-01T10:47:30.507Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:17.189 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-02-01T10:48:17.189 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-02-01T10:48:17.189 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-02-01T10:48:17.189 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-02-01T10:48:17.189 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-02-01T10:48:17.189 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-02-01T10:48:17.189 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-02-01T10:48:17.189 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-02-01T10:48:17.190 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:48:17.190 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:17.190 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:17.190 DEBUG:boto:path=/vclfcn-61/?versions&max-keys=999 2024-02-01T10:48:17.190 DEBUG:boto:auth_path=/vclfcn-61/?versions&max-keys=999 2024-02-01T10:48:17.190 DEBUG:boto:Method: GET 2024-02-01T10:48:17.190 DEBUG:boto:Path: /vclfcn-61/?versions&max-keys=999 2024-02-01T10:48:17.190 DEBUG:boto:Data: 2024-02-01T10:48:17.191 DEBUG:boto:Headers: {} 2024-02-01T10:48:17.191 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:17.191 DEBUG:boto:Port: 8000 2024-02-01T10:48:17.191 DEBUG:boto:Params: {} 2024-02-01T10:48:17.191 DEBUG:boto:Token: None 2024-02-01T10:48:17.191 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:17 GMT /vclfcn-61/?versions 2024-02-01T10:48:17.191 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Q01hnIxKyga+0YrHzPyNQhHrydg= 2024-02-01T10:48:17.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Q01hnIxKyga+0YrHzPyNQhHrydg=', 'Content-Length': '0'} 2024-02-01T10:48:17.195 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051585f19bc37258d-0065bb76f1-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:17.196 DEBUG:boto:b'vclfcn-61999falseobj1nulltrue2024-02-01T10:46:46.359Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:46:46.380Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:46:46.433Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:46:46.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-02-01T10:47:30.425Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-02-01T10:47:30.440Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-02-01T10:47:30.492Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-02-01T10:47:30.507Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:17.196 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-02-01T10:48:17.197 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-02-01T10:48:17.197 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-02-01T10:48:17.197 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-02-01T10:48:17.197 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-02-01T10:48:17.197 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-02-01T10:48:17.197 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-02-01T10:48:17.197 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-02-01T10:48:17.197 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:17.197 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:17.197 DEBUG:boto:path=/vclfcn-61/?versions&max-keys=999 2024-02-01T10:48:17.198 DEBUG:boto:auth_path=/vclfcn-61/?versions&max-keys=999 2024-02-01T10:48:17.198 DEBUG:boto:Method: GET 2024-02-01T10:48:17.198 DEBUG:boto:Path: /vclfcn-61/?versions&max-keys=999 2024-02-01T10:48:17.198 DEBUG:boto:Data: 2024-02-01T10:48:17.198 DEBUG:boto:Headers: {} 2024-02-01T10:48:17.198 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:17.198 DEBUG:boto:Port: 8000 2024-02-01T10:48:17.198 DEBUG:boto:Params: {} 2024-02-01T10:48:17.199 DEBUG:boto:Token: None 2024-02-01T10:48:17.199 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:17 GMT /vclfcn-61/?versions 2024-02-01T10:48:17.199 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Q01hnIxKyga+0YrHzPyNQhHrydg= 2024-02-01T10:48:17.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Q01hnIxKyga+0YrHzPyNQhHrydg=', 'Content-Length': '0'} 2024-02-01T10:48:17.201 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9a55b9ae7b02b21-0065bb76f1-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:17.201 DEBUG:boto:b'vclfcn-61999falseobj1nulltrue2024-02-01T10:46:46.359Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:46:46.380Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:46:46.433Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:46:46.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-02-01T10:47:30.425Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-02-01T10:47:30.440Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-02-01T10:47:30.492Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-02-01T10:47:30.507Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:17.202 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:17.202 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:17.202 DEBUG:boto:path=/vclfcn-61/?versions&max-keys=999 2024-02-01T10:48:17.202 DEBUG:boto:auth_path=/vclfcn-61/?versions&max-keys=999 2024-02-01T10:48:17.203 DEBUG:boto:Method: GET 2024-02-01T10:48:17.203 DEBUG:boto:Path: /vclfcn-61/?versions&max-keys=999 2024-02-01T10:48:17.203 DEBUG:boto:Data: 2024-02-01T10:48:17.203 DEBUG:boto:Headers: {} 2024-02-01T10:48:17.203 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:17.203 DEBUG:boto:Port: 8000 2024-02-01T10:48:17.203 DEBUG:boto:Params: {} 2024-02-01T10:48:17.203 DEBUG:boto:Token: None 2024-02-01T10:48:17.203 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:17 GMT /vclfcn-61/?versions 2024-02-01T10:48:17.204 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Q01hnIxKyga+0YrHzPyNQhHrydg= 2024-02-01T10:48:17.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Q01hnIxKyga+0YrHzPyNQhHrydg=', 'Content-Length': '0'} 2024-02-01T10:48:17.208 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e501de062743dca-0065bb76f1-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:17.209 DEBUG:boto:b'vclfcn-61999falseobj1nulltrue2024-02-01T10:46:46.359Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:46:46.380Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:46:46.433Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:46:46.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-02-01T10:47:30.425Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-02-01T10:47:30.440Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-02-01T10:47:30.492Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-02-01T10:47:30.507Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:17.209 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:48:17.210 DEBUG:boto:path=/vclfcn-61/obj1 2024-02-01T10:48:17.210 DEBUG:boto:auth_path=/vclfcn-61/obj1 2024-02-01T10:48:17.210 DEBUG:boto:path=/vclfcn-61/obj1?versionId=null 2024-02-01T10:48:17.210 DEBUG:boto:auth_path=/vclfcn-61/obj1?versionId=null 2024-02-01T10:48:17.210 DEBUG:boto:Method: GET 2024-02-01T10:48:17.210 DEBUG:boto:Path: /vclfcn-61/obj1?versionId=null 2024-02-01T10:48:17.210 DEBUG:boto:Data: 2024-02-01T10:48:17.210 DEBUG:boto:Headers: {} 2024-02-01T10:48:17.210 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:17.210 DEBUG:boto:Port: 8000 2024-02-01T10:48:17.211 DEBUG:boto:Params: {} 2024-02-01T10:48:17.211 DEBUG:boto:Token: None 2024-02-01T10:48:17.211 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:17 GMT /vclfcn-61/obj1?versionId=null 2024-02-01T10:48:17.211 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:d6LEnlVXe2HWQAMAUaLHRr0/QxE= 2024-02-01T10:48:17.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:d6LEnlVXe2HWQAMAUaLHRr0/QxE=', 'Content-Length': '0'} 2024-02-01T10:48:17.213 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009f66871c8ac9546e-0065bb76f1-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:17.213 DEBUG:boto:path=/vclfcn-61/obj1 2024-02-01T10:48:17.213 DEBUG:boto:auth_path=/vclfcn-61/obj1 2024-02-01T10:48:17.214 DEBUG:boto:path=/vclfcn-61/obj1?versionId=null 2024-02-01T10:48:17.214 DEBUG:boto:auth_path=/vclfcn-61/obj1?versionId=null 2024-02-01T10:48:17.214 DEBUG:boto:Method: GET 2024-02-01T10:48:17.214 DEBUG:boto:Path: /vclfcn-61/obj1?versionId=null 2024-02-01T10:48:17.214 DEBUG:boto:Data: 2024-02-01T10:48:17.214 DEBUG:boto:Headers: {} 2024-02-01T10:48:17.214 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:17.214 DEBUG:boto:Port: 8000 2024-02-01T10:48:17.214 DEBUG:boto:Params: {} 2024-02-01T10:48:17.214 DEBUG:boto:Token: None 2024-02-01T10:48:17.215 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:17 GMT /vclfcn-61/obj1?versionId=null 2024-02-01T10:48:17.215 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:d6LEnlVXe2HWQAMAUaLHRr0/QxE= 2024-02-01T10:48:17.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:d6LEnlVXe2HWQAMAUaLHRr0/QxE=', 'Content-Length': '0'} 2024-02-01T10:48:17.218 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a4dc58f16937bef4-0065bb76f1-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:17.218 DEBUG:boto:path=/vclfcn-61/obj1 2024-02-01T10:48:17.218 DEBUG:boto:auth_path=/vclfcn-61/obj1 2024-02-01T10:48:17.218 DEBUG:boto:path=/vclfcn-61/obj1?versionId=null 2024-02-01T10:48:17.218 DEBUG:boto:auth_path=/vclfcn-61/obj1?versionId=null 2024-02-01T10:48:17.218 DEBUG:boto:Method: HEAD 2024-02-01T10:48:17.219 DEBUG:boto:Path: /vclfcn-61/obj1?versionId=null 2024-02-01T10:48:17.219 DEBUG:boto:Data: 2024-02-01T10:48:17.219 DEBUG:boto:Headers: {} 2024-02-01T10:48:17.219 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:17.219 DEBUG:boto:Port: 8000 2024-02-01T10:48:17.219 DEBUG:boto:Params: {} 2024-02-01T10:48:17.219 DEBUG:boto:Token: None 2024-02-01T10:48:17.219 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:17 GMT /vclfcn-61/obj1?versionId=null 2024-02-01T10:48:17.219 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:x24Egw68P4d1Iid12S+7qByLxzY= 2024-02-01T10:48:17.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:x24Egw68P4d1Iid12S+7qByLxzY=', 'Content-Length': '0'} 2024-02-01T10:48:17.222 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003dfb2d85f2e77d9b-0065bb76f1-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:17.222 DEBUG:boto:path=/vclfcn-61/obj1 2024-02-01T10:48:17.222 DEBUG:boto:auth_path=/vclfcn-61/obj1 2024-02-01T10:48:17.222 DEBUG:boto:path=/vclfcn-61/obj1?versionId=null 2024-02-01T10:48:17.222 DEBUG:boto:auth_path=/vclfcn-61/obj1?versionId=null 2024-02-01T10:48:17.222 DEBUG:boto:Method: HEAD 2024-02-01T10:48:17.222 DEBUG:boto:Path: /vclfcn-61/obj1?versionId=null 2024-02-01T10:48:17.223 DEBUG:boto:Data: 2024-02-01T10:48:17.223 DEBUG:boto:Headers: {} 2024-02-01T10:48:17.223 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:17.223 DEBUG:boto:Port: 8000 2024-02-01T10:48:17.223 DEBUG:boto:Params: {} 2024-02-01T10:48:17.223 DEBUG:boto:Token: None 2024-02-01T10:48:17.223 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:17 GMT /vclfcn-61/obj1?versionId=null 2024-02-01T10:48:17.223 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:x24Egw68P4d1Iid12S+7qByLxzY= 2024-02-01T10:48:17.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:x24Egw68P4d1Iid12S+7qByLxzY=', 'Content-Length': '0'} 2024-02-01T10:48:17.226 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000826bc1c0d74dd476-0065bb76f1-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:17.226 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:48:17.227 DEBUG:boto:path=/vclfcn-61/obj1 2024-02-01T10:48:17.227 DEBUG:boto:auth_path=/vclfcn-61/obj1 2024-02-01T10:48:17.227 DEBUG:boto:path=/vclfcn-61/obj1?versionId=null 2024-02-01T10:48:17.227 DEBUG:boto:auth_path=/vclfcn-61/obj1?versionId=null 2024-02-01T10:48:17.227 DEBUG:boto:Method: GET 2024-02-01T10:48:17.227 DEBUG:boto:Path: /vclfcn-61/obj1?versionId=null 2024-02-01T10:48:17.227 DEBUG:boto:Data: 2024-02-01T10:48:17.227 DEBUG:boto:Headers: {} 2024-02-01T10:48:17.227 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:17.227 DEBUG:boto:Port: 8000 2024-02-01T10:48:17.227 DEBUG:boto:Params: {} 2024-02-01T10:48:17.228 DEBUG:boto:Token: None 2024-02-01T10:48:17.228 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:17 GMT /vclfcn-61/obj1?versionId=null 2024-02-01T10:48:17.228 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:d6LEnlVXe2HWQAMAUaLHRr0/QxE= 2024-02-01T10:48:17.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:d6LEnlVXe2HWQAMAUaLHRr0/QxE=', 'Content-Length': '0'} 2024-02-01T10:48:17.230 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000025ce184fe4045f8d-0065bb76f1-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:17.231 DEBUG:boto:path=/vclfcn-61/obj1 2024-02-01T10:48:17.231 DEBUG:boto:auth_path=/vclfcn-61/obj1 2024-02-01T10:48:17.231 DEBUG:boto:path=/vclfcn-61/obj1?versionId=null 2024-02-01T10:48:17.231 DEBUG:boto:auth_path=/vclfcn-61/obj1?versionId=null 2024-02-01T10:48:17.231 DEBUG:boto:Method: GET 2024-02-01T10:48:17.231 DEBUG:boto:Path: /vclfcn-61/obj1?versionId=null 2024-02-01T10:48:17.231 DEBUG:boto:Data: 2024-02-01T10:48:17.231 DEBUG:boto:Headers: {} 2024-02-01T10:48:17.231 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:17.231 DEBUG:boto:Port: 8000 2024-02-01T10:48:17.231 DEBUG:boto:Params: {} 2024-02-01T10:48:17.232 DEBUG:boto:Token: None 2024-02-01T10:48:17.232 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:17 GMT /vclfcn-61/obj1?versionId=null 2024-02-01T10:48:17.232 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:d6LEnlVXe2HWQAMAUaLHRr0/QxE= 2024-02-01T10:48:17.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:d6LEnlVXe2HWQAMAUaLHRr0/QxE=', 'Content-Length': '0'} 2024-02-01T10:48:17.235 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000474457b592a48aa9-0065bb76f1-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:17.279 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:17.279 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:17.280 DEBUG:boto:path=/vclfcn-61/?prefix=obj1 2024-02-01T10:48:17.280 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj1 2024-02-01T10:48:17.280 DEBUG:boto:Method: GET 2024-02-01T10:48:17.280 DEBUG:boto:Path: /vclfcn-61/?prefix=obj1 2024-02-01T10:48:17.280 DEBUG:boto:Data: 2024-02-01T10:48:17.280 DEBUG:boto:Headers: {} 2024-02-01T10:48:17.280 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:17.280 DEBUG:boto:Port: 8000 2024-02-01T10:48:17.280 DEBUG:boto:Params: {} 2024-02-01T10:48:17.281 DEBUG:boto:Token: None 2024-02-01T10:48:17.281 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:17 GMT /vclfcn-61/ 2024-02-01T10:48:17.281 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NckkrdISkXpttVAEOYSklUKFayw= 2024-02-01T10:48:17.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NckkrdISkXpttVAEOYSklUKFayw=', 'Content-Length': '0'} 2024-02-01T10:48:17.285 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a8af39ee6d67e7d-0065bb76f1-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:17.286 DEBUG:boto:b'vclfcn-61obj11000falseobj12024-02-01T10:46:46.359Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:17.286 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:17.286 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:17.286 DEBUG:boto:path=/vclfcn-61/?prefix=obj1 2024-02-01T10:48:17.286 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj1 2024-02-01T10:48:17.286 DEBUG:boto:Method: GET 2024-02-01T10:48:17.287 DEBUG:boto:Path: /vclfcn-61/?prefix=obj1 2024-02-01T10:48:17.287 DEBUG:boto:Data: 2024-02-01T10:48:17.287 DEBUG:boto:Headers: {} 2024-02-01T10:48:17.287 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:17.287 DEBUG:boto:Port: 8000 2024-02-01T10:48:17.287 DEBUG:boto:Params: {} 2024-02-01T10:48:17.287 DEBUG:boto:Token: None 2024-02-01T10:48:17.287 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:17 GMT /vclfcn-61/ 2024-02-01T10:48:17.287 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NckkrdISkXpttVAEOYSklUKFayw= 2024-02-01T10:48:17.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NckkrdISkXpttVAEOYSklUKFayw=', 'Content-Length': '0'} 2024-02-01T10:48:17.291 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035a70e417e965dc3-0065bb76f1-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:17.292 DEBUG:boto:b'vclfcn-61obj11000falseobj12024-02-01T10:46:46.359Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:17.292 DEBUG:boto:path=/vclfcn-61/obj1 2024-02-01T10:48:17.292 DEBUG:boto:auth_path=/vclfcn-61/obj1 2024-02-01T10:48:17.292 DEBUG:boto:Method: HEAD 2024-02-01T10:48:17.292 DEBUG:boto:Path: /vclfcn-61/obj1 2024-02-01T10:48:17.292 DEBUG:boto:Data: 2024-02-01T10:48:17.292 DEBUG:boto:Headers: {} 2024-02-01T10:48:17.293 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:17.293 DEBUG:boto:Port: 8000 2024-02-01T10:48:17.293 DEBUG:boto:Params: {} 2024-02-01T10:48:17.293 DEBUG:boto:Token: None 2024-02-01T10:48:17.293 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:17 GMT /vclfcn-61/obj1 2024-02-01T10:48:17.293 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Wjwahk1pcj42a2nmamkSQaFCJas= 2024-02-01T10:48:17.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Wjwahk1pcj42a2nmamkSQaFCJas=', 'Content-Length': '0'} 2024-02-01T10:48:17.296 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aaad46a98141e5c1-0065bb76f1-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:17.296 DEBUG:boto:path=/vclfcn-61/obj1 2024-02-01T10:48:17.296 DEBUG:boto:auth_path=/vclfcn-61/obj1 2024-02-01T10:48:17.296 DEBUG:boto:Method: HEAD 2024-02-01T10:48:17.297 DEBUG:boto:Path: /vclfcn-61/obj1 2024-02-01T10:48:17.297 DEBUG:boto:Data: 2024-02-01T10:48:17.297 DEBUG:boto:Headers: {} 2024-02-01T10:48:17.297 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:17.297 DEBUG:boto:Port: 8000 2024-02-01T10:48:17.297 DEBUG:boto:Params: {} 2024-02-01T10:48:17.297 DEBUG:boto:Token: None 2024-02-01T10:48:17.297 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:17 GMT /vclfcn-61/obj1 2024-02-01T10:48:17.297 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Wjwahk1pcj42a2nmamkSQaFCJas= 2024-02-01T10:48:17.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Wjwahk1pcj42a2nmamkSQaFCJas=', 'Content-Length': '0'} 2024-02-01T10:48:17.300 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f7042ff40bc3880a-0065bb76f1-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:17.300 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:48:17.301 DEBUG:boto:path=/vclfcn-61/obj1 2024-02-01T10:48:17.301 DEBUG:boto:auth_path=/vclfcn-61/obj1 2024-02-01T10:48:17.301 DEBUG:boto:Method: GET 2024-02-01T10:48:17.301 DEBUG:boto:Path: /vclfcn-61/obj1 2024-02-01T10:48:17.301 DEBUG:boto:Data: 2024-02-01T10:48:17.301 DEBUG:boto:Headers: {} 2024-02-01T10:48:17.301 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:17.301 DEBUG:boto:Port: 8000 2024-02-01T10:48:17.301 DEBUG:boto:Params: {} 2024-02-01T10:48:17.302 DEBUG:boto:Token: None 2024-02-01T10:48:17.302 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:17 GMT /vclfcn-61/obj1 2024-02-01T10:48:17.302 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lJ3VE9agouk3aHgpJG32y5rXTVU= 2024-02-01T10:48:17.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lJ3VE9agouk3aHgpJG32y5rXTVU=', 'Content-Length': '0'} 2024-02-01T10:48:17.305 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a4645a0d5f196491-0065bb76f1-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:17.348 DEBUG:boto:path=/vclfcn-61/obj1 2024-02-01T10:48:17.348 DEBUG:boto:auth_path=/vclfcn-61/obj1 2024-02-01T10:48:17.348 DEBUG:boto:Method: GET 2024-02-01T10:48:17.348 DEBUG:boto:Path: /vclfcn-61/obj1 2024-02-01T10:48:17.349 DEBUG:boto:Data: 2024-02-01T10:48:17.349 DEBUG:boto:Headers: {} 2024-02-01T10:48:17.349 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:17.349 DEBUG:boto:Port: 8000 2024-02-01T10:48:17.349 DEBUG:boto:Params: {} 2024-02-01T10:48:17.349 DEBUG:boto:Token: None 2024-02-01T10:48:17.349 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:17 GMT /vclfcn-61/obj1 2024-02-01T10:48:17.349 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lJ3VE9agouk3aHgpJG32y5rXTVU= 2024-02-01T10:48:17.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lJ3VE9agouk3aHgpJG32y5rXTVU=', 'Content-Length': '0'} 2024-02-01T10:48:17.353 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002f016cadd10727cf-0065bb76f1-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:17.396 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:17.396 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:17.396 DEBUG:boto:path=/vclfcn-61/?prefix=obj1 2024-02-01T10:48:17.396 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj1 2024-02-01T10:48:17.396 DEBUG:boto:Method: GET 2024-02-01T10:48:17.396 DEBUG:boto:Path: /vclfcn-61/?prefix=obj1 2024-02-01T10:48:17.397 DEBUG:boto:Data: 2024-02-01T10:48:17.397 DEBUG:boto:Headers: {} 2024-02-01T10:48:17.397 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:17.397 DEBUG:boto:Port: 8000 2024-02-01T10:48:17.397 DEBUG:boto:Params: {} 2024-02-01T10:48:17.397 DEBUG:boto:Token: None 2024-02-01T10:48:17.397 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:17 GMT /vclfcn-61/ 2024-02-01T10:48:17.397 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NckkrdISkXpttVAEOYSklUKFayw= 2024-02-01T10:48:17.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NckkrdISkXpttVAEOYSklUKFayw=', 'Content-Length': '0'} 2024-02-01T10:48:17.402 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f5b27bed5b1f4ad-0065bb76f1-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:17.402 DEBUG:boto:b'vclfcn-61obj11000falseobj12024-02-01T10:46:46.359Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:17.402 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:17.403 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:17.403 DEBUG:boto:path=/vclfcn-61/?prefix=obj1 2024-02-01T10:48:17.403 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj1 2024-02-01T10:48:17.403 DEBUG:boto:Method: GET 2024-02-01T10:48:17.403 DEBUG:boto:Path: /vclfcn-61/?prefix=obj1 2024-02-01T10:48:17.403 DEBUG:boto:Data: 2024-02-01T10:48:17.403 DEBUG:boto:Headers: {} 2024-02-01T10:48:17.403 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:17.403 DEBUG:boto:Port: 8000 2024-02-01T10:48:17.403 DEBUG:boto:Params: {} 2024-02-01T10:48:17.404 DEBUG:boto:Token: None 2024-02-01T10:48:17.404 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:17 GMT /vclfcn-61/ 2024-02-01T10:48:17.404 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NckkrdISkXpttVAEOYSklUKFayw= 2024-02-01T10:48:17.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NckkrdISkXpttVAEOYSklUKFayw=', 'Content-Length': '0'} 2024-02-01T10:48:17.408 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3c1473bdd5510ea-0065bb76f1-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:17.408 DEBUG:boto:b'vclfcn-61obj11000falseobj12024-02-01T10:46:46.359Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:17.408 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:48:17.409 DEBUG:boto:path=/vclfcn-61/obj2 2024-02-01T10:48:17.409 DEBUG:boto:auth_path=/vclfcn-61/obj2 2024-02-01T10:48:17.409 DEBUG:boto:path=/vclfcn-61/obj2?versionId=null 2024-02-01T10:48:17.409 DEBUG:boto:auth_path=/vclfcn-61/obj2?versionId=null 2024-02-01T10:48:17.409 DEBUG:boto:Method: GET 2024-02-01T10:48:17.409 DEBUG:boto:Path: /vclfcn-61/obj2?versionId=null 2024-02-01T10:48:17.409 DEBUG:boto:Data: 2024-02-01T10:48:17.409 DEBUG:boto:Headers: {} 2024-02-01T10:48:17.409 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:17.409 DEBUG:boto:Port: 8000 2024-02-01T10:48:17.410 DEBUG:boto:Params: {} 2024-02-01T10:48:17.410 DEBUG:boto:Token: None 2024-02-01T10:48:17.410 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:17 GMT /vclfcn-61/obj2?versionId=null 2024-02-01T10:48:17.410 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Dl4kVRAqZqasCjfbwmMogwIvefY= 2024-02-01T10:48:17.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Dl4kVRAqZqasCjfbwmMogwIvefY=', 'Content-Length': '0'} 2024-02-01T10:48:17.413 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a78cde14845f687a-0065bb76f1-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:17.413 DEBUG:boto:path=/vclfcn-61/obj2 2024-02-01T10:48:17.413 DEBUG:boto:auth_path=/vclfcn-61/obj2 2024-02-01T10:48:17.413 DEBUG:boto:path=/vclfcn-61/obj2?versionId=null 2024-02-01T10:48:17.413 DEBUG:boto:auth_path=/vclfcn-61/obj2?versionId=null 2024-02-01T10:48:17.413 DEBUG:boto:Method: GET 2024-02-01T10:48:17.413 DEBUG:boto:Path: /vclfcn-61/obj2?versionId=null 2024-02-01T10:48:17.414 DEBUG:boto:Data: 2024-02-01T10:48:17.414 DEBUG:boto:Headers: {} 2024-02-01T10:48:17.414 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:17.414 DEBUG:boto:Port: 8000 2024-02-01T10:48:17.414 DEBUG:boto:Params: {} 2024-02-01T10:48:17.414 DEBUG:boto:Token: None 2024-02-01T10:48:17.414 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:17 GMT /vclfcn-61/obj2?versionId=null 2024-02-01T10:48:17.414 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Dl4kVRAqZqasCjfbwmMogwIvefY= 2024-02-01T10:48:17.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Dl4kVRAqZqasCjfbwmMogwIvefY=', 'Content-Length': '0'} 2024-02-01T10:48:17.417 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006d4c40929d175a33-0065bb76f1-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:17.460 DEBUG:boto:path=/vclfcn-61/obj2 2024-02-01T10:48:17.460 DEBUG:boto:auth_path=/vclfcn-61/obj2 2024-02-01T10:48:17.460 DEBUG:boto:path=/vclfcn-61/obj2?versionId=null 2024-02-01T10:48:17.460 DEBUG:boto:auth_path=/vclfcn-61/obj2?versionId=null 2024-02-01T10:48:17.461 DEBUG:boto:Method: HEAD 2024-02-01T10:48:17.461 DEBUG:boto:Path: /vclfcn-61/obj2?versionId=null 2024-02-01T10:48:17.461 DEBUG:boto:Data: 2024-02-01T10:48:17.461 DEBUG:boto:Headers: {} 2024-02-01T10:48:17.461 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:17.461 DEBUG:boto:Port: 8000 2024-02-01T10:48:17.461 DEBUG:boto:Params: {} 2024-02-01T10:48:17.461 DEBUG:boto:Token: None 2024-02-01T10:48:17.461 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:17 GMT /vclfcn-61/obj2?versionId=null 2024-02-01T10:48:17.462 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7vCQF+N0zmC/nb/b7R3UsQ9RvJg= 2024-02-01T10:48:17.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7vCQF+N0zmC/nb/b7R3UsQ9RvJg=', 'Content-Length': '0'} 2024-02-01T10:48:17.464 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000058a6cefa7c236570-0065bb76f1-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:17.465 DEBUG:boto:path=/vclfcn-61/obj2 2024-02-01T10:48:17.465 DEBUG:boto:auth_path=/vclfcn-61/obj2 2024-02-01T10:48:17.465 DEBUG:boto:path=/vclfcn-61/obj2?versionId=null 2024-02-01T10:48:17.465 DEBUG:boto:auth_path=/vclfcn-61/obj2?versionId=null 2024-02-01T10:48:17.465 DEBUG:boto:Method: HEAD 2024-02-01T10:48:17.465 DEBUG:boto:Path: /vclfcn-61/obj2?versionId=null 2024-02-01T10:48:17.465 DEBUG:boto:Data: 2024-02-01T10:48:17.465 DEBUG:boto:Headers: {} 2024-02-01T10:48:17.465 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:17.465 DEBUG:boto:Port: 8000 2024-02-01T10:48:17.466 DEBUG:boto:Params: {} 2024-02-01T10:48:17.466 DEBUG:boto:Token: None 2024-02-01T10:48:17.466 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:17 GMT /vclfcn-61/obj2?versionId=null 2024-02-01T10:48:17.466 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7vCQF+N0zmC/nb/b7R3UsQ9RvJg= 2024-02-01T10:48:17.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7vCQF+N0zmC/nb/b7R3UsQ9RvJg=', 'Content-Length': '0'} 2024-02-01T10:48:17.469 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000018a8136022805656-0065bb76f1-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:17.469 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:48:17.469 DEBUG:boto:path=/vclfcn-61/obj2 2024-02-01T10:48:17.469 DEBUG:boto:auth_path=/vclfcn-61/obj2 2024-02-01T10:48:17.469 DEBUG:boto:path=/vclfcn-61/obj2?versionId=null 2024-02-01T10:48:17.469 DEBUG:boto:auth_path=/vclfcn-61/obj2?versionId=null 2024-02-01T10:48:17.470 DEBUG:boto:Method: GET 2024-02-01T10:48:17.470 DEBUG:boto:Path: /vclfcn-61/obj2?versionId=null 2024-02-01T10:48:17.470 DEBUG:boto:Data: 2024-02-01T10:48:17.470 DEBUG:boto:Headers: {} 2024-02-01T10:48:17.470 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:17.470 DEBUG:boto:Port: 8000 2024-02-01T10:48:17.470 DEBUG:boto:Params: {} 2024-02-01T10:48:17.470 DEBUG:boto:Token: None 2024-02-01T10:48:17.470 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:17 GMT /vclfcn-61/obj2?versionId=null 2024-02-01T10:48:17.470 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Dl4kVRAqZqasCjfbwmMogwIvefY= 2024-02-01T10:48:17.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Dl4kVRAqZqasCjfbwmMogwIvefY=', 'Content-Length': '0'} 2024-02-01T10:48:17.473 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000470882566328fe50-0065bb76f1-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:17.523 DEBUG:boto:path=/vclfcn-61/obj2 2024-02-01T10:48:17.523 DEBUG:boto:auth_path=/vclfcn-61/obj2 2024-02-01T10:48:17.523 DEBUG:boto:path=/vclfcn-61/obj2?versionId=null 2024-02-01T10:48:17.523 DEBUG:boto:auth_path=/vclfcn-61/obj2?versionId=null 2024-02-01T10:48:17.523 DEBUG:boto:Method: GET 2024-02-01T10:48:17.523 DEBUG:boto:Path: /vclfcn-61/obj2?versionId=null 2024-02-01T10:48:17.524 DEBUG:boto:Data: 2024-02-01T10:48:17.524 DEBUG:boto:Headers: {} 2024-02-01T10:48:17.524 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:17.524 DEBUG:boto:Port: 8000 2024-02-01T10:48:17.524 DEBUG:boto:Params: {} 2024-02-01T10:48:17.524 DEBUG:boto:Token: None 2024-02-01T10:48:17.524 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:17 GMT /vclfcn-61/obj2?versionId=null 2024-02-01T10:48:17.524 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Dl4kVRAqZqasCjfbwmMogwIvefY= 2024-02-01T10:48:17.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Dl4kVRAqZqasCjfbwmMogwIvefY=', 'Content-Length': '0'} 2024-02-01T10:48:17.527 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c0fcc760a31b59d6-0065bb76f1-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:17.568 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:17.568 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:17.568 DEBUG:boto:path=/vclfcn-61/?prefix=obj2 2024-02-01T10:48:17.569 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj2 2024-02-01T10:48:17.569 DEBUG:boto:Method: GET 2024-02-01T10:48:17.569 DEBUG:boto:Path: /vclfcn-61/?prefix=obj2 2024-02-01T10:48:17.569 DEBUG:boto:Data: 2024-02-01T10:48:17.569 DEBUG:boto:Headers: {} 2024-02-01T10:48:17.569 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:17.569 DEBUG:boto:Port: 8000 2024-02-01T10:48:17.569 DEBUG:boto:Params: {} 2024-02-01T10:48:17.569 DEBUG:boto:Token: None 2024-02-01T10:48:17.569 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:17 GMT /vclfcn-61/ 2024-02-01T10:48:17.570 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NckkrdISkXpttVAEOYSklUKFayw= 2024-02-01T10:48:17.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NckkrdISkXpttVAEOYSklUKFayw=', 'Content-Length': '0'} 2024-02-01T10:48:17.574 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea414c8d21100a2d-0065bb76f1-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:17.616 DEBUG:boto:b'vclfcn-61obj21000falseobj22024-02-01T10:46:46.380Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:17.617 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:17.617 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:17.617 DEBUG:boto:path=/vclfcn-61/?prefix=obj2 2024-02-01T10:48:17.617 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj2 2024-02-01T10:48:17.617 DEBUG:boto:Method: GET 2024-02-01T10:48:17.617 DEBUG:boto:Path: /vclfcn-61/?prefix=obj2 2024-02-01T10:48:17.617 DEBUG:boto:Data: 2024-02-01T10:48:17.617 DEBUG:boto:Headers: {} 2024-02-01T10:48:17.617 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:17.617 DEBUG:boto:Port: 8000 2024-02-01T10:48:17.617 DEBUG:boto:Params: {} 2024-02-01T10:48:17.618 DEBUG:boto:Token: None 2024-02-01T10:48:17.618 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:17 GMT /vclfcn-61/ 2024-02-01T10:48:17.618 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NckkrdISkXpttVAEOYSklUKFayw= 2024-02-01T10:48:17.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NckkrdISkXpttVAEOYSklUKFayw=', 'Content-Length': '0'} 2024-02-01T10:48:17.622 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000081aacbd061acaaab-0065bb76f1-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:17.623 DEBUG:boto:b'vclfcn-61obj21000falseobj22024-02-01T10:46:46.380Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:17.623 DEBUG:boto:path=/vclfcn-61/obj2 2024-02-01T10:48:17.623 DEBUG:boto:auth_path=/vclfcn-61/obj2 2024-02-01T10:48:17.623 DEBUG:boto:Method: HEAD 2024-02-01T10:48:17.623 DEBUG:boto:Path: /vclfcn-61/obj2 2024-02-01T10:48:17.623 DEBUG:boto:Data: 2024-02-01T10:48:17.624 DEBUG:boto:Headers: {} 2024-02-01T10:48:17.624 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:17.624 DEBUG:boto:Port: 8000 2024-02-01T10:48:17.624 DEBUG:boto:Params: {} 2024-02-01T10:48:17.624 DEBUG:boto:Token: None 2024-02-01T10:48:17.624 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:17 GMT /vclfcn-61/obj2 2024-02-01T10:48:17.624 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:2Up+Doxe509li2JWh8UQ9mliXBU= 2024-02-01T10:48:17.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2Up+Doxe509li2JWh8UQ9mliXBU=', 'Content-Length': '0'} 2024-02-01T10:48:17.627 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e3e12afc98195e5e-0065bb76f1-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:17.627 DEBUG:boto:path=/vclfcn-61/obj2 2024-02-01T10:48:17.627 DEBUG:boto:auth_path=/vclfcn-61/obj2 2024-02-01T10:48:17.628 DEBUG:boto:Method: HEAD 2024-02-01T10:48:17.628 DEBUG:boto:Path: /vclfcn-61/obj2 2024-02-01T10:48:17.628 DEBUG:boto:Data: 2024-02-01T10:48:17.628 DEBUG:boto:Headers: {} 2024-02-01T10:48:17.628 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:17.628 DEBUG:boto:Port: 8000 2024-02-01T10:48:17.628 DEBUG:boto:Params: {} 2024-02-01T10:48:17.629 DEBUG:boto:Token: None 2024-02-01T10:48:17.629 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:17 GMT /vclfcn-61/obj2 2024-02-01T10:48:17.629 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:2Up+Doxe509li2JWh8UQ9mliXBU= 2024-02-01T10:48:17.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2Up+Doxe509li2JWh8UQ9mliXBU=', 'Content-Length': '0'} 2024-02-01T10:48:17.631 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000043792dae8368cbbf-0065bb76f1-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:17.632 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:48:17.632 DEBUG:boto:path=/vclfcn-61/obj2 2024-02-01T10:48:17.632 DEBUG:boto:auth_path=/vclfcn-61/obj2 2024-02-01T10:48:17.632 DEBUG:boto:Method: GET 2024-02-01T10:48:17.632 DEBUG:boto:Path: /vclfcn-61/obj2 2024-02-01T10:48:17.632 DEBUG:boto:Data: 2024-02-01T10:48:17.633 DEBUG:boto:Headers: {} 2024-02-01T10:48:17.633 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:17.633 DEBUG:boto:Port: 8000 2024-02-01T10:48:17.633 DEBUG:boto:Params: {} 2024-02-01T10:48:17.633 DEBUG:boto:Token: None 2024-02-01T10:48:17.633 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:17 GMT /vclfcn-61/obj2 2024-02-01T10:48:17.633 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:s6n2P3tNUw641PNwjN5wyupx4+c= 2024-02-01T10:48:17.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:s6n2P3tNUw641PNwjN5wyupx4+c=', 'Content-Length': '0'} 2024-02-01T10:48:17.636 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ef4d482cb84969f5-0065bb76f1-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:17.680 DEBUG:boto:path=/vclfcn-61/obj2 2024-02-01T10:48:17.680 DEBUG:boto:auth_path=/vclfcn-61/obj2 2024-02-01T10:48:17.680 DEBUG:boto:Method: GET 2024-02-01T10:48:17.680 DEBUG:boto:Path: /vclfcn-61/obj2 2024-02-01T10:48:17.681 DEBUG:boto:Data: 2024-02-01T10:48:17.681 DEBUG:boto:Headers: {} 2024-02-01T10:48:17.681 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:17.681 DEBUG:boto:Port: 8000 2024-02-01T10:48:17.681 DEBUG:boto:Params: {} 2024-02-01T10:48:17.681 DEBUG:boto:Token: None 2024-02-01T10:48:17.681 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:17 GMT /vclfcn-61/obj2 2024-02-01T10:48:17.681 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:s6n2P3tNUw641PNwjN5wyupx4+c= 2024-02-01T10:48:17.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:s6n2P3tNUw641PNwjN5wyupx4+c=', 'Content-Length': '0'} 2024-02-01T10:48:17.687 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000060d4dc7bea5c99c3-0065bb76f1-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:17.729 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:17.730 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:17.730 DEBUG:boto:path=/vclfcn-61/?prefix=obj2 2024-02-01T10:48:17.730 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj2 2024-02-01T10:48:17.731 DEBUG:boto:Method: GET 2024-02-01T10:48:17.731 DEBUG:boto:Path: /vclfcn-61/?prefix=obj2 2024-02-01T10:48:17.731 DEBUG:boto:Data: 2024-02-01T10:48:17.732 DEBUG:boto:Headers: {} 2024-02-01T10:48:17.732 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:17.732 DEBUG:boto:Port: 8000 2024-02-01T10:48:17.733 DEBUG:boto:Params: {} 2024-02-01T10:48:17.733 DEBUG:boto:Token: None 2024-02-01T10:48:17.734 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:17 GMT /vclfcn-61/ 2024-02-01T10:48:17.734 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NckkrdISkXpttVAEOYSklUKFayw= 2024-02-01T10:48:17.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NckkrdISkXpttVAEOYSklUKFayw=', 'Content-Length': '0'} 2024-02-01T10:48:17.739 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002123f937fcee577a-0065bb76f1-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:17.739 DEBUG:boto:b'vclfcn-61obj21000falseobj22024-02-01T10:46:46.380Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:17.741 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:17.741 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:17.742 DEBUG:boto:path=/vclfcn-61/?prefix=obj2 2024-02-01T10:48:17.742 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj2 2024-02-01T10:48:17.742 DEBUG:boto:Method: GET 2024-02-01T10:48:17.742 DEBUG:boto:Path: /vclfcn-61/?prefix=obj2 2024-02-01T10:48:17.742 DEBUG:boto:Data: 2024-02-01T10:48:17.743 DEBUG:boto:Headers: {} 2024-02-01T10:48:17.743 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:17.743 DEBUG:boto:Port: 8000 2024-02-01T10:48:17.743 DEBUG:boto:Params: {} 2024-02-01T10:48:17.744 DEBUG:boto:Token: None 2024-02-01T10:48:17.745 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:17 GMT /vclfcn-61/ 2024-02-01T10:48:17.745 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NckkrdISkXpttVAEOYSklUKFayw= 2024-02-01T10:48:17.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NckkrdISkXpttVAEOYSklUKFayw=', 'Content-Length': '0'} 2024-02-01T10:48:17.750 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000494f4df245d0afbe-0065bb76f1-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:17.751 DEBUG:boto:b'vclfcn-61obj21000falseobj22024-02-01T10:46:46.380Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:17.752 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:48:17.752 DEBUG:boto:path=/vclfcn-61/obj3 2024-02-01T10:48:17.753 DEBUG:boto:auth_path=/vclfcn-61/obj3 2024-02-01T10:48:17.753 DEBUG:boto:path=/vclfcn-61/obj3?versionId=null 2024-02-01T10:48:17.753 DEBUG:boto:auth_path=/vclfcn-61/obj3?versionId=null 2024-02-01T10:48:17.754 DEBUG:boto:Method: GET 2024-02-01T10:48:17.754 DEBUG:boto:Path: /vclfcn-61/obj3?versionId=null 2024-02-01T10:48:17.754 DEBUG:boto:Data: 2024-02-01T10:48:17.754 DEBUG:boto:Headers: {} 2024-02-01T10:48:17.754 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:17.754 DEBUG:boto:Port: 8000 2024-02-01T10:48:17.754 DEBUG:boto:Params: {} 2024-02-01T10:48:17.755 DEBUG:boto:Token: None 2024-02-01T10:48:17.755 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:17 GMT /vclfcn-61/obj3?versionId=null 2024-02-01T10:48:17.756 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:OdaXfYKNOV9Wlb/UOsNhw2o8fNE= 2024-02-01T10:48:17.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:OdaXfYKNOV9Wlb/UOsNhw2o8fNE=', 'Content-Length': '0'} 2024-02-01T10:48:17.759 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f200ac914c8420ed-0065bb76f1-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:17.800 DEBUG:boto:path=/vclfcn-61/obj3 2024-02-01T10:48:17.800 DEBUG:boto:auth_path=/vclfcn-61/obj3 2024-02-01T10:48:17.800 DEBUG:boto:path=/vclfcn-61/obj3?versionId=null 2024-02-01T10:48:17.801 DEBUG:boto:auth_path=/vclfcn-61/obj3?versionId=null 2024-02-01T10:48:17.801 DEBUG:boto:Method: GET 2024-02-01T10:48:17.801 DEBUG:boto:Path: /vclfcn-61/obj3?versionId=null 2024-02-01T10:48:17.801 DEBUG:boto:Data: 2024-02-01T10:48:17.801 DEBUG:boto:Headers: {} 2024-02-01T10:48:17.801 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:17.801 DEBUG:boto:Port: 8000 2024-02-01T10:48:17.802 DEBUG:boto:Params: {} 2024-02-01T10:48:17.802 DEBUG:boto:Token: None 2024-02-01T10:48:17.802 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:17 GMT /vclfcn-61/obj3?versionId=null 2024-02-01T10:48:17.803 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:OdaXfYKNOV9Wlb/UOsNhw2o8fNE= 2024-02-01T10:48:17.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:OdaXfYKNOV9Wlb/UOsNhw2o8fNE=', 'Content-Length': '0'} 2024-02-01T10:48:17.807 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bd90faaa7c30846b-0065bb76f1-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:17.807 DEBUG:boto:path=/vclfcn-61/obj3 2024-02-01T10:48:17.807 DEBUG:boto:auth_path=/vclfcn-61/obj3 2024-02-01T10:48:17.808 DEBUG:boto:path=/vclfcn-61/obj3?versionId=null 2024-02-01T10:48:17.808 DEBUG:boto:auth_path=/vclfcn-61/obj3?versionId=null 2024-02-01T10:48:17.808 DEBUG:boto:Method: HEAD 2024-02-01T10:48:17.808 DEBUG:boto:Path: /vclfcn-61/obj3?versionId=null 2024-02-01T10:48:17.808 DEBUG:boto:Data: 2024-02-01T10:48:17.808 DEBUG:boto:Headers: {} 2024-02-01T10:48:17.808 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:17.808 DEBUG:boto:Port: 8000 2024-02-01T10:48:17.808 DEBUG:boto:Params: {} 2024-02-01T10:48:17.809 DEBUG:boto:Token: None 2024-02-01T10:48:17.809 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:17 GMT /vclfcn-61/obj3?versionId=null 2024-02-01T10:48:17.809 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:P7jHgoEzmmGyUzierdjiWcu1PRQ= 2024-02-01T10:48:17.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:P7jHgoEzmmGyUzierdjiWcu1PRQ=', 'Content-Length': '0'} 2024-02-01T10:48:17.812 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002fb8ab2acaf92fad-0065bb76f1-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:17.812 DEBUG:boto:path=/vclfcn-61/obj3 2024-02-01T10:48:17.812 DEBUG:boto:auth_path=/vclfcn-61/obj3 2024-02-01T10:48:17.813 DEBUG:boto:path=/vclfcn-61/obj3?versionId=null 2024-02-01T10:48:17.813 DEBUG:boto:auth_path=/vclfcn-61/obj3?versionId=null 2024-02-01T10:48:17.813 DEBUG:boto:Method: HEAD 2024-02-01T10:48:17.813 DEBUG:boto:Path: /vclfcn-61/obj3?versionId=null 2024-02-01T10:48:17.813 DEBUG:boto:Data: 2024-02-01T10:48:17.814 DEBUG:boto:Headers: {} 2024-02-01T10:48:17.814 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:17.814 DEBUG:boto:Port: 8000 2024-02-01T10:48:17.814 DEBUG:boto:Params: {} 2024-02-01T10:48:17.814 DEBUG:boto:Token: None 2024-02-01T10:48:17.815 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:17 GMT /vclfcn-61/obj3?versionId=null 2024-02-01T10:48:17.815 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:P7jHgoEzmmGyUzierdjiWcu1PRQ= 2024-02-01T10:48:17.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:P7jHgoEzmmGyUzierdjiWcu1PRQ=', 'Content-Length': '0'} 2024-02-01T10:48:17.817 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000269eb710e18f072d-0065bb76f1-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:17.818 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:48:17.818 DEBUG:boto:path=/vclfcn-61/obj3 2024-02-01T10:48:17.818 DEBUG:boto:auth_path=/vclfcn-61/obj3 2024-02-01T10:48:17.818 DEBUG:boto:path=/vclfcn-61/obj3?versionId=null 2024-02-01T10:48:17.818 DEBUG:boto:auth_path=/vclfcn-61/obj3?versionId=null 2024-02-01T10:48:17.818 DEBUG:boto:Method: GET 2024-02-01T10:48:17.819 DEBUG:boto:Path: /vclfcn-61/obj3?versionId=null 2024-02-01T10:48:17.819 DEBUG:boto:Data: 2024-02-01T10:48:17.819 DEBUG:boto:Headers: {} 2024-02-01T10:48:17.819 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:17.819 DEBUG:boto:Port: 8000 2024-02-01T10:48:17.819 DEBUG:boto:Params: {} 2024-02-01T10:48:17.819 DEBUG:boto:Token: None 2024-02-01T10:48:17.820 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:17 GMT /vclfcn-61/obj3?versionId=null 2024-02-01T10:48:17.820 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:OdaXfYKNOV9Wlb/UOsNhw2o8fNE= 2024-02-01T10:48:17.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:OdaXfYKNOV9Wlb/UOsNhw2o8fNE=', 'Content-Length': '0'} 2024-02-01T10:48:17.823 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000067c67c0d10dc3572-0065bb76f1-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:17.864 DEBUG:boto:path=/vclfcn-61/obj3 2024-02-01T10:48:17.864 DEBUG:boto:auth_path=/vclfcn-61/obj3 2024-02-01T10:48:17.864 DEBUG:boto:path=/vclfcn-61/obj3?versionId=null 2024-02-01T10:48:17.864 DEBUG:boto:auth_path=/vclfcn-61/obj3?versionId=null 2024-02-01T10:48:17.864 DEBUG:boto:Method: GET 2024-02-01T10:48:17.865 DEBUG:boto:Path: /vclfcn-61/obj3?versionId=null 2024-02-01T10:48:17.865 DEBUG:boto:Data: 2024-02-01T10:48:17.865 DEBUG:boto:Headers: {} 2024-02-01T10:48:17.865 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:17.865 DEBUG:boto:Port: 8000 2024-02-01T10:48:17.865 DEBUG:boto:Params: {} 2024-02-01T10:48:17.865 DEBUG:boto:Token: None 2024-02-01T10:48:17.865 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:17 GMT /vclfcn-61/obj3?versionId=null 2024-02-01T10:48:17.865 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:OdaXfYKNOV9Wlb/UOsNhw2o8fNE= 2024-02-01T10:48:17.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:OdaXfYKNOV9Wlb/UOsNhw2o8fNE=', 'Content-Length': '0'} 2024-02-01T10:48:17.869 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ff57595001f7992c-0065bb76f1-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:17.913 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:17.913 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:17.913 DEBUG:boto:path=/vclfcn-61/?prefix=obj3 2024-02-01T10:48:17.913 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj3 2024-02-01T10:48:17.913 DEBUG:boto:Method: GET 2024-02-01T10:48:17.913 DEBUG:boto:Path: /vclfcn-61/?prefix=obj3 2024-02-01T10:48:17.914 DEBUG:boto:Data: 2024-02-01T10:48:17.914 DEBUG:boto:Headers: {} 2024-02-01T10:48:17.914 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:17.914 DEBUG:boto:Port: 8000 2024-02-01T10:48:17.914 DEBUG:boto:Params: {} 2024-02-01T10:48:17.914 DEBUG:boto:Token: None 2024-02-01T10:48:17.914 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:17 GMT /vclfcn-61/ 2024-02-01T10:48:17.915 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NckkrdISkXpttVAEOYSklUKFayw= 2024-02-01T10:48:17.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NckkrdISkXpttVAEOYSklUKFayw=', 'Content-Length': '0'} 2024-02-01T10:48:17.919 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b528b4f8b26281fd-0065bb76f1-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:17.920 DEBUG:boto:b'vclfcn-61obj31000falseobj32024-02-01T10:46:46.433Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:17.920 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:17.920 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:17.920 DEBUG:boto:path=/vclfcn-61/?prefix=obj3 2024-02-01T10:48:17.921 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj3 2024-02-01T10:48:17.921 DEBUG:boto:Method: GET 2024-02-01T10:48:17.921 DEBUG:boto:Path: /vclfcn-61/?prefix=obj3 2024-02-01T10:48:17.921 DEBUG:boto:Data: 2024-02-01T10:48:17.921 DEBUG:boto:Headers: {} 2024-02-01T10:48:17.921 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:17.921 DEBUG:boto:Port: 8000 2024-02-01T10:48:17.921 DEBUG:boto:Params: {} 2024-02-01T10:48:17.921 DEBUG:boto:Token: None 2024-02-01T10:48:17.921 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:17 GMT /vclfcn-61/ 2024-02-01T10:48:17.922 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NckkrdISkXpttVAEOYSklUKFayw= 2024-02-01T10:48:17.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NckkrdISkXpttVAEOYSklUKFayw=', 'Content-Length': '0'} 2024-02-01T10:48:17.926 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a13f4d2200c7646a-0065bb76f1-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:17.926 DEBUG:boto:b'vclfcn-61obj31000falseobj32024-02-01T10:46:46.433Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:17.926 DEBUG:boto:path=/vclfcn-61/obj3 2024-02-01T10:48:17.926 DEBUG:boto:auth_path=/vclfcn-61/obj3 2024-02-01T10:48:17.927 DEBUG:boto:Method: HEAD 2024-02-01T10:48:17.927 DEBUG:boto:Path: /vclfcn-61/obj3 2024-02-01T10:48:17.927 DEBUG:boto:Data: 2024-02-01T10:48:17.927 DEBUG:boto:Headers: {} 2024-02-01T10:48:17.927 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:17.927 DEBUG:boto:Port: 8000 2024-02-01T10:48:17.927 DEBUG:boto:Params: {} 2024-02-01T10:48:17.927 DEBUG:boto:Token: None 2024-02-01T10:48:17.927 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:17 GMT /vclfcn-61/obj3 2024-02-01T10:48:17.928 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:z9vVfyjD/2BULc32g6ywOG4SMSw= 2024-02-01T10:48:17.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:z9vVfyjD/2BULc32g6ywOG4SMSw=', 'Content-Length': '0'} 2024-02-01T10:48:17.930 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e0a7db5bd5c8f385-0065bb76f1-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:17.930 DEBUG:boto:path=/vclfcn-61/obj3 2024-02-01T10:48:17.931 DEBUG:boto:auth_path=/vclfcn-61/obj3 2024-02-01T10:48:17.931 DEBUG:boto:Method: HEAD 2024-02-01T10:48:17.931 DEBUG:boto:Path: /vclfcn-61/obj3 2024-02-01T10:48:17.931 DEBUG:boto:Data: 2024-02-01T10:48:17.931 DEBUG:boto:Headers: {} 2024-02-01T10:48:17.931 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:17.931 DEBUG:boto:Port: 8000 2024-02-01T10:48:17.931 DEBUG:boto:Params: {} 2024-02-01T10:48:17.931 DEBUG:boto:Token: None 2024-02-01T10:48:17.932 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:17 GMT /vclfcn-61/obj3 2024-02-01T10:48:17.932 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:z9vVfyjD/2BULc32g6ywOG4SMSw= 2024-02-01T10:48:17.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:z9vVfyjD/2BULc32g6ywOG4SMSw=', 'Content-Length': '0'} 2024-02-01T10:48:17.934 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b2264e1f990f7787-0065bb76f1-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:17.935 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:48:17.935 DEBUG:boto:path=/vclfcn-61/obj3 2024-02-01T10:48:17.935 DEBUG:boto:auth_path=/vclfcn-61/obj3 2024-02-01T10:48:17.935 DEBUG:boto:Method: GET 2024-02-01T10:48:17.935 DEBUG:boto:Path: /vclfcn-61/obj3 2024-02-01T10:48:17.935 DEBUG:boto:Data: 2024-02-01T10:48:17.935 DEBUG:boto:Headers: {} 2024-02-01T10:48:17.935 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:17.936 DEBUG:boto:Port: 8000 2024-02-01T10:48:17.936 DEBUG:boto:Params: {} 2024-02-01T10:48:17.936 DEBUG:boto:Token: None 2024-02-01T10:48:17.936 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:17 GMT /vclfcn-61/obj3 2024-02-01T10:48:17.936 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:xUNkAPVoOXSBRLISDuxCKEmbmu8= 2024-02-01T10:48:17.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:xUNkAPVoOXSBRLISDuxCKEmbmu8=', 'Content-Length': '0'} 2024-02-01T10:48:17.939 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c69c217dce4842fc-0065bb76f1-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:17.985 DEBUG:boto:path=/vclfcn-61/obj3 2024-02-01T10:48:17.985 DEBUG:boto:auth_path=/vclfcn-61/obj3 2024-02-01T10:48:17.985 DEBUG:boto:Method: GET 2024-02-01T10:48:17.985 DEBUG:boto:Path: /vclfcn-61/obj3 2024-02-01T10:48:17.985 DEBUG:boto:Data: 2024-02-01T10:48:17.985 DEBUG:boto:Headers: {} 2024-02-01T10:48:17.985 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:17.985 DEBUG:boto:Port: 8000 2024-02-01T10:48:17.986 DEBUG:boto:Params: {} 2024-02-01T10:48:17.986 DEBUG:boto:Token: None 2024-02-01T10:48:17.986 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:17 GMT /vclfcn-61/obj3 2024-02-01T10:48:17.986 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:xUNkAPVoOXSBRLISDuxCKEmbmu8= 2024-02-01T10:48:17.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:xUNkAPVoOXSBRLISDuxCKEmbmu8=', 'Content-Length': '0'} 2024-02-01T10:48:17.989 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009266d812f24e9034-0065bb76f1-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:18.032 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:18.032 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:18.032 DEBUG:boto:path=/vclfcn-61/?prefix=obj3 2024-02-01T10:48:18.033 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj3 2024-02-01T10:48:18.033 DEBUG:boto:Method: GET 2024-02-01T10:48:18.033 DEBUG:boto:Path: /vclfcn-61/?prefix=obj3 2024-02-01T10:48:18.033 DEBUG:boto:Data: 2024-02-01T10:48:18.033 DEBUG:boto:Headers: {} 2024-02-01T10:48:18.033 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:18.033 DEBUG:boto:Port: 8000 2024-02-01T10:48:18.033 DEBUG:boto:Params: {} 2024-02-01T10:48:18.033 DEBUG:boto:Token: None 2024-02-01T10:48:18.034 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:18 GMT /vclfcn-61/ 2024-02-01T10:48:18.034 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Lr6O0kGCwbM0XrNfMdDh9GMwVKA= 2024-02-01T10:48:18.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Lr6O0kGCwbM0XrNfMdDh9GMwVKA=', 'Content-Length': '0'} 2024-02-01T10:48:18.038 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec44f2ff944ac0d7-0065bb76f2-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:18.038 DEBUG:boto:b'vclfcn-61obj31000falseobj32024-02-01T10:46:46.433Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:18.039 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:18.039 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:18.039 DEBUG:boto:path=/vclfcn-61/?prefix=obj3 2024-02-01T10:48:18.039 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj3 2024-02-01T10:48:18.039 DEBUG:boto:Method: GET 2024-02-01T10:48:18.039 DEBUG:boto:Path: /vclfcn-61/?prefix=obj3 2024-02-01T10:48:18.039 DEBUG:boto:Data: 2024-02-01T10:48:18.039 DEBUG:boto:Headers: {} 2024-02-01T10:48:18.040 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:18.040 DEBUG:boto:Port: 8000 2024-02-01T10:48:18.040 DEBUG:boto:Params: {} 2024-02-01T10:48:18.040 DEBUG:boto:Token: None 2024-02-01T10:48:18.040 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:18 GMT /vclfcn-61/ 2024-02-01T10:48:18.040 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Lr6O0kGCwbM0XrNfMdDh9GMwVKA= 2024-02-01T10:48:18.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Lr6O0kGCwbM0XrNfMdDh9GMwVKA=', 'Content-Length': '0'} 2024-02-01T10:48:18.045 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc7ec17199a4b39d-0065bb76f2-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:18.088 DEBUG:boto:b'vclfcn-61obj31000falseobj32024-02-01T10:46:46.433Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:18.088 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:48:18.089 DEBUG:boto:path=/vclfcn-61/obj4 2024-02-01T10:48:18.089 DEBUG:boto:auth_path=/vclfcn-61/obj4 2024-02-01T10:48:18.089 DEBUG:boto:path=/vclfcn-61/obj4?versionId=null 2024-02-01T10:48:18.089 DEBUG:boto:auth_path=/vclfcn-61/obj4?versionId=null 2024-02-01T10:48:18.089 DEBUG:boto:Method: GET 2024-02-01T10:48:18.089 DEBUG:boto:Path: /vclfcn-61/obj4?versionId=null 2024-02-01T10:48:18.089 DEBUG:boto:Data: 2024-02-01T10:48:18.089 DEBUG:boto:Headers: {} 2024-02-01T10:48:18.089 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:18.089 DEBUG:boto:Port: 8000 2024-02-01T10:48:18.090 DEBUG:boto:Params: {} 2024-02-01T10:48:18.090 DEBUG:boto:Token: None 2024-02-01T10:48:18.090 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:18 GMT /vclfcn-61/obj4?versionId=null 2024-02-01T10:48:18.090 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gGiIAIQsgg83xukMfime6PQz5cQ= 2024-02-01T10:48:18.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gGiIAIQsgg83xukMfime6PQz5cQ=', 'Content-Length': '0'} 2024-02-01T10:48:18.093 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006a35c8beb94737c0-0065bb76f2-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:18.094 DEBUG:boto:path=/vclfcn-61/obj4 2024-02-01T10:48:18.094 DEBUG:boto:auth_path=/vclfcn-61/obj4 2024-02-01T10:48:18.094 DEBUG:boto:path=/vclfcn-61/obj4?versionId=null 2024-02-01T10:48:18.094 DEBUG:boto:auth_path=/vclfcn-61/obj4?versionId=null 2024-02-01T10:48:18.094 DEBUG:boto:Method: GET 2024-02-01T10:48:18.094 DEBUG:boto:Path: /vclfcn-61/obj4?versionId=null 2024-02-01T10:48:18.094 DEBUG:boto:Data: 2024-02-01T10:48:18.094 DEBUG:boto:Headers: {} 2024-02-01T10:48:18.095 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:18.095 DEBUG:boto:Port: 8000 2024-02-01T10:48:18.095 DEBUG:boto:Params: {} 2024-02-01T10:48:18.095 DEBUG:boto:Token: None 2024-02-01T10:48:18.095 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:18 GMT /vclfcn-61/obj4?versionId=null 2024-02-01T10:48:18.095 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gGiIAIQsgg83xukMfime6PQz5cQ= 2024-02-01T10:48:18.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gGiIAIQsgg83xukMfime6PQz5cQ=', 'Content-Length': '0'} 2024-02-01T10:48:18.099 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e230ab6349c8edc3-0065bb76f2-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:18.099 DEBUG:boto:path=/vclfcn-61/obj4 2024-02-01T10:48:18.099 DEBUG:boto:auth_path=/vclfcn-61/obj4 2024-02-01T10:48:18.099 DEBUG:boto:path=/vclfcn-61/obj4?versionId=null 2024-02-01T10:48:18.099 DEBUG:boto:auth_path=/vclfcn-61/obj4?versionId=null 2024-02-01T10:48:18.099 DEBUG:boto:Method: HEAD 2024-02-01T10:48:18.099 DEBUG:boto:Path: /vclfcn-61/obj4?versionId=null 2024-02-01T10:48:18.100 DEBUG:boto:Data: 2024-02-01T10:48:18.100 DEBUG:boto:Headers: {} 2024-02-01T10:48:18.100 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:18.100 DEBUG:boto:Port: 8000 2024-02-01T10:48:18.100 DEBUG:boto:Params: {} 2024-02-01T10:48:18.100 DEBUG:boto:Token: None 2024-02-01T10:48:18.100 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:18 GMT /vclfcn-61/obj4?versionId=null 2024-02-01T10:48:18.100 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sX0OTHjVl5ZO0MmZe79Qf+nBfjA= 2024-02-01T10:48:18.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sX0OTHjVl5ZO0MmZe79Qf+nBfjA=', 'Content-Length': '0'} 2024-02-01T10:48:18.103 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000659eba5a993fde23-0065bb76f2-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:18.103 DEBUG:boto:path=/vclfcn-61/obj4 2024-02-01T10:48:18.103 DEBUG:boto:auth_path=/vclfcn-61/obj4 2024-02-01T10:48:18.104 DEBUG:boto:path=/vclfcn-61/obj4?versionId=null 2024-02-01T10:48:18.104 DEBUG:boto:auth_path=/vclfcn-61/obj4?versionId=null 2024-02-01T10:48:18.104 DEBUG:boto:Method: HEAD 2024-02-01T10:48:18.104 DEBUG:boto:Path: /vclfcn-61/obj4?versionId=null 2024-02-01T10:48:18.104 DEBUG:boto:Data: 2024-02-01T10:48:18.104 DEBUG:boto:Headers: {} 2024-02-01T10:48:18.104 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:18.104 DEBUG:boto:Port: 8000 2024-02-01T10:48:18.104 DEBUG:boto:Params: {} 2024-02-01T10:48:18.104 DEBUG:boto:Token: None 2024-02-01T10:48:18.105 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:18 GMT /vclfcn-61/obj4?versionId=null 2024-02-01T10:48:18.105 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sX0OTHjVl5ZO0MmZe79Qf+nBfjA= 2024-02-01T10:48:18.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sX0OTHjVl5ZO0MmZe79Qf+nBfjA=', 'Content-Length': '0'} 2024-02-01T10:48:18.108 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eb1987bd05dab6eb-0065bb76f2-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:18.108 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:48:18.108 DEBUG:boto:path=/vclfcn-61/obj4 2024-02-01T10:48:18.108 DEBUG:boto:auth_path=/vclfcn-61/obj4 2024-02-01T10:48:18.108 DEBUG:boto:path=/vclfcn-61/obj4?versionId=null 2024-02-01T10:48:18.108 DEBUG:boto:auth_path=/vclfcn-61/obj4?versionId=null 2024-02-01T10:48:18.108 DEBUG:boto:Method: GET 2024-02-01T10:48:18.109 DEBUG:boto:Path: /vclfcn-61/obj4?versionId=null 2024-02-01T10:48:18.109 DEBUG:boto:Data: 2024-02-01T10:48:18.109 DEBUG:boto:Headers: {} 2024-02-01T10:48:18.109 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:18.109 DEBUG:boto:Port: 8000 2024-02-01T10:48:18.109 DEBUG:boto:Params: {} 2024-02-01T10:48:18.109 DEBUG:boto:Token: None 2024-02-01T10:48:18.109 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:18 GMT /vclfcn-61/obj4?versionId=null 2024-02-01T10:48:18.109 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gGiIAIQsgg83xukMfime6PQz5cQ= 2024-02-01T10:48:18.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gGiIAIQsgg83xukMfime6PQz5cQ=', 'Content-Length': '0'} 2024-02-01T10:48:18.112 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c25256d7ce40b89a-0065bb76f2-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:18.156 DEBUG:boto:path=/vclfcn-61/obj4 2024-02-01T10:48:18.156 DEBUG:boto:auth_path=/vclfcn-61/obj4 2024-02-01T10:48:18.157 DEBUG:boto:path=/vclfcn-61/obj4?versionId=null 2024-02-01T10:48:18.157 DEBUG:boto:auth_path=/vclfcn-61/obj4?versionId=null 2024-02-01T10:48:18.157 DEBUG:boto:Method: GET 2024-02-01T10:48:18.157 DEBUG:boto:Path: /vclfcn-61/obj4?versionId=null 2024-02-01T10:48:18.157 DEBUG:boto:Data: 2024-02-01T10:48:18.157 DEBUG:boto:Headers: {} 2024-02-01T10:48:18.157 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:18.157 DEBUG:boto:Port: 8000 2024-02-01T10:48:18.157 DEBUG:boto:Params: {} 2024-02-01T10:48:18.157 DEBUG:boto:Token: None 2024-02-01T10:48:18.158 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:18 GMT /vclfcn-61/obj4?versionId=null 2024-02-01T10:48:18.158 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gGiIAIQsgg83xukMfime6PQz5cQ= 2024-02-01T10:48:18.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gGiIAIQsgg83xukMfime6PQz5cQ=', 'Content-Length': '0'} 2024-02-01T10:48:18.161 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004bf744607d7879e9-0065bb76f2-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:18.204 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:18.204 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:18.204 DEBUG:boto:path=/vclfcn-61/?prefix=obj4 2024-02-01T10:48:18.204 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj4 2024-02-01T10:48:18.204 DEBUG:boto:Method: GET 2024-02-01T10:48:18.205 DEBUG:boto:Path: /vclfcn-61/?prefix=obj4 2024-02-01T10:48:18.205 DEBUG:boto:Data: 2024-02-01T10:48:18.205 DEBUG:boto:Headers: {} 2024-02-01T10:48:18.205 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:18.205 DEBUG:boto:Port: 8000 2024-02-01T10:48:18.205 DEBUG:boto:Params: {} 2024-02-01T10:48:18.205 DEBUG:boto:Token: None 2024-02-01T10:48:18.205 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:18 GMT /vclfcn-61/ 2024-02-01T10:48:18.206 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Lr6O0kGCwbM0XrNfMdDh9GMwVKA= 2024-02-01T10:48:18.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Lr6O0kGCwbM0XrNfMdDh9GMwVKA=', 'Content-Length': '0'} 2024-02-01T10:48:18.210 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000301d7f4e00ac6b32-0065bb76f2-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:18.252 DEBUG:boto:b'vclfcn-61obj41000falseobj42024-02-01T10:46:46.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:18.253 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:18.253 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:18.253 DEBUG:boto:path=/vclfcn-61/?prefix=obj4 2024-02-01T10:48:18.253 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj4 2024-02-01T10:48:18.253 DEBUG:boto:Method: GET 2024-02-01T10:48:18.253 DEBUG:boto:Path: /vclfcn-61/?prefix=obj4 2024-02-01T10:48:18.253 DEBUG:boto:Data: 2024-02-01T10:48:18.253 DEBUG:boto:Headers: {} 2024-02-01T10:48:18.253 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:18.254 DEBUG:boto:Port: 8000 2024-02-01T10:48:18.254 DEBUG:boto:Params: {} 2024-02-01T10:48:18.254 DEBUG:boto:Token: None 2024-02-01T10:48:18.254 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:18 GMT /vclfcn-61/ 2024-02-01T10:48:18.254 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Lr6O0kGCwbM0XrNfMdDh9GMwVKA= 2024-02-01T10:48:18.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Lr6O0kGCwbM0XrNfMdDh9GMwVKA=', 'Content-Length': '0'} 2024-02-01T10:48:18.258 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6e71393ddd3d1b4-0065bb76f2-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:18.300 DEBUG:boto:b'vclfcn-61obj41000falseobj42024-02-01T10:46:46.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:18.300 DEBUG:boto:path=/vclfcn-61/obj4 2024-02-01T10:48:18.300 DEBUG:boto:auth_path=/vclfcn-61/obj4 2024-02-01T10:48:18.301 DEBUG:boto:Method: HEAD 2024-02-01T10:48:18.301 DEBUG:boto:Path: /vclfcn-61/obj4 2024-02-01T10:48:18.301 DEBUG:boto:Data: 2024-02-01T10:48:18.301 DEBUG:boto:Headers: {} 2024-02-01T10:48:18.301 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:18.301 DEBUG:boto:Port: 8000 2024-02-01T10:48:18.301 DEBUG:boto:Params: {} 2024-02-01T10:48:18.301 DEBUG:boto:Token: None 2024-02-01T10:48:18.301 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:18 GMT /vclfcn-61/obj4 2024-02-01T10:48:18.302 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zZh8MA9bvS4wqhp3h2sMVoJHyRU= 2024-02-01T10:48:18.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zZh8MA9bvS4wqhp3h2sMVoJHyRU=', 'Content-Length': '0'} 2024-02-01T10:48:18.304 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005acec28e0d544ac9-0065bb76f2-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:18.305 DEBUG:boto:path=/vclfcn-61/obj4 2024-02-01T10:48:18.305 DEBUG:boto:auth_path=/vclfcn-61/obj4 2024-02-01T10:48:18.305 DEBUG:boto:Method: HEAD 2024-02-01T10:48:18.305 DEBUG:boto:Path: /vclfcn-61/obj4 2024-02-01T10:48:18.305 DEBUG:boto:Data: 2024-02-01T10:48:18.305 DEBUG:boto:Headers: {} 2024-02-01T10:48:18.305 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:18.305 DEBUG:boto:Port: 8000 2024-02-01T10:48:18.306 DEBUG:boto:Params: {} 2024-02-01T10:48:18.306 DEBUG:boto:Token: None 2024-02-01T10:48:18.306 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:18 GMT /vclfcn-61/obj4 2024-02-01T10:48:18.306 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zZh8MA9bvS4wqhp3h2sMVoJHyRU= 2024-02-01T10:48:18.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zZh8MA9bvS4wqhp3h2sMVoJHyRU=', 'Content-Length': '0'} 2024-02-01T10:48:18.309 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000039923afb78747d8c-0065bb76f2-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:18.309 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:48:18.309 DEBUG:boto:path=/vclfcn-61/obj4 2024-02-01T10:48:18.309 DEBUG:boto:auth_path=/vclfcn-61/obj4 2024-02-01T10:48:18.309 DEBUG:boto:Method: GET 2024-02-01T10:48:18.309 DEBUG:boto:Path: /vclfcn-61/obj4 2024-02-01T10:48:18.309 DEBUG:boto:Data: 2024-02-01T10:48:18.310 DEBUG:boto:Headers: {} 2024-02-01T10:48:18.310 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:18.310 DEBUG:boto:Port: 8000 2024-02-01T10:48:18.310 DEBUG:boto:Params: {} 2024-02-01T10:48:18.310 DEBUG:boto:Token: None 2024-02-01T10:48:18.310 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:18 GMT /vclfcn-61/obj4 2024-02-01T10:48:18.310 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:G/k1b8M3O4nEW5e1lSiyUjOgIv4= 2024-02-01T10:48:18.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:G/k1b8M3O4nEW5e1lSiyUjOgIv4=', 'Content-Length': '0'} 2024-02-01T10:48:18.313 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007aab9ca546c5be69-0065bb76f2-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:18.356 DEBUG:boto:path=/vclfcn-61/obj4 2024-02-01T10:48:18.356 DEBUG:boto:auth_path=/vclfcn-61/obj4 2024-02-01T10:48:18.356 DEBUG:boto:Method: GET 2024-02-01T10:48:18.356 DEBUG:boto:Path: /vclfcn-61/obj4 2024-02-01T10:48:18.356 DEBUG:boto:Data: 2024-02-01T10:48:18.357 DEBUG:boto:Headers: {} 2024-02-01T10:48:18.357 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:18.357 DEBUG:boto:Port: 8000 2024-02-01T10:48:18.357 DEBUG:boto:Params: {} 2024-02-01T10:48:18.357 DEBUG:boto:Token: None 2024-02-01T10:48:18.357 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:18 GMT /vclfcn-61/obj4 2024-02-01T10:48:18.357 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:G/k1b8M3O4nEW5e1lSiyUjOgIv4= 2024-02-01T10:48:18.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:G/k1b8M3O4nEW5e1lSiyUjOgIv4=', 'Content-Length': '0'} 2024-02-01T10:48:18.360 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fb47384d946a046e-0065bb76f2-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:18.408 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:18.408 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:18.408 DEBUG:boto:path=/vclfcn-61/?prefix=obj4 2024-02-01T10:48:18.408 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj4 2024-02-01T10:48:18.408 DEBUG:boto:Method: GET 2024-02-01T10:48:18.409 DEBUG:boto:Path: /vclfcn-61/?prefix=obj4 2024-02-01T10:48:18.409 DEBUG:boto:Data: 2024-02-01T10:48:18.409 DEBUG:boto:Headers: {} 2024-02-01T10:48:18.409 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:18.409 DEBUG:boto:Port: 8000 2024-02-01T10:48:18.409 DEBUG:boto:Params: {} 2024-02-01T10:48:18.409 DEBUG:boto:Token: None 2024-02-01T10:48:18.409 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:18 GMT /vclfcn-61/ 2024-02-01T10:48:18.410 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Lr6O0kGCwbM0XrNfMdDh9GMwVKA= 2024-02-01T10:48:18.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Lr6O0kGCwbM0XrNfMdDh9GMwVKA=', 'Content-Length': '0'} 2024-02-01T10:48:18.414 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004914ab9633c8acea-0065bb76f2-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:18.456 DEBUG:boto:b'vclfcn-61obj41000falseobj42024-02-01T10:46:46.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:18.457 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:18.457 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:18.457 DEBUG:boto:path=/vclfcn-61/?prefix=obj4 2024-02-01T10:48:18.457 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj4 2024-02-01T10:48:18.457 DEBUG:boto:Method: GET 2024-02-01T10:48:18.457 DEBUG:boto:Path: /vclfcn-61/?prefix=obj4 2024-02-01T10:48:18.457 DEBUG:boto:Data: 2024-02-01T10:48:18.457 DEBUG:boto:Headers: {} 2024-02-01T10:48:18.457 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:18.457 DEBUG:boto:Port: 8000 2024-02-01T10:48:18.458 DEBUG:boto:Params: {} 2024-02-01T10:48:18.458 DEBUG:boto:Token: None 2024-02-01T10:48:18.458 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:18 GMT /vclfcn-61/ 2024-02-01T10:48:18.458 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Lr6O0kGCwbM0XrNfMdDh9GMwVKA= 2024-02-01T10:48:18.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Lr6O0kGCwbM0XrNfMdDh9GMwVKA=', 'Content-Length': '0'} 2024-02-01T10:48:18.462 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c736b422940157f-0065bb76f2-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:18.508 DEBUG:boto:b'vclfcn-61obj41000falseobj42024-02-01T10:46:46.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:18.508 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-02-01T10:48:18.509 DEBUG:boto:path=/vclfcn-61/obj5 2024-02-01T10:48:18.509 DEBUG:boto:auth_path=/vclfcn-61/obj5 2024-02-01T10:48:18.509 DEBUG:boto:path=/vclfcn-61/obj5?versionId=null 2024-02-01T10:48:18.509 DEBUG:boto:auth_path=/vclfcn-61/obj5?versionId=null 2024-02-01T10:48:18.511 DEBUG:boto:Method: GET 2024-02-01T10:48:18.511 DEBUG:boto:Path: /vclfcn-61/obj5?versionId=null 2024-02-01T10:48:18.511 DEBUG:boto:Data: 2024-02-01T10:48:18.511 DEBUG:boto:Headers: {} 2024-02-01T10:48:18.511 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:18.511 DEBUG:boto:Port: 8000 2024-02-01T10:48:18.512 DEBUG:boto:Params: {} 2024-02-01T10:48:18.512 DEBUG:boto:Token: None 2024-02-01T10:48:18.512 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:18 GMT /vclfcn-61/obj5?versionId=null 2024-02-01T10:48:18.512 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tdCYuNXt1iS3Qnft7+Lq0wnl2y8= 2024-02-01T10:48:18.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tdCYuNXt1iS3Qnft7+Lq0wnl2y8=', 'Content-Length': '0'} 2024-02-01T10:48:18.517 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007665842c2bfcbc0d-0065bb76f2-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:18.518 DEBUG:boto:path=/vclfcn-61/obj5 2024-02-01T10:48:18.518 DEBUG:boto:auth_path=/vclfcn-61/obj5 2024-02-01T10:48:18.518 DEBUG:boto:path=/vclfcn-61/obj5?versionId=null 2024-02-01T10:48:18.518 DEBUG:boto:auth_path=/vclfcn-61/obj5?versionId=null 2024-02-01T10:48:18.518 DEBUG:boto:Method: GET 2024-02-01T10:48:18.518 DEBUG:boto:Path: /vclfcn-61/obj5?versionId=null 2024-02-01T10:48:18.518 DEBUG:boto:Data: 2024-02-01T10:48:18.518 DEBUG:boto:Headers: {} 2024-02-01T10:48:18.518 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:18.519 DEBUG:boto:Port: 8000 2024-02-01T10:48:18.519 DEBUG:boto:Params: {} 2024-02-01T10:48:18.519 DEBUG:boto:Token: None 2024-02-01T10:48:18.519 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:18 GMT /vclfcn-61/obj5?versionId=null 2024-02-01T10:48:18.519 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tdCYuNXt1iS3Qnft7+Lq0wnl2y8= 2024-02-01T10:48:18.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tdCYuNXt1iS3Qnft7+Lq0wnl2y8=', 'Content-Length': '0'} 2024-02-01T10:48:18.522 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000081e0addb83ebe4b6-0065bb76f2-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:18.564 DEBUG:boto:path=/vclfcn-61/obj5 2024-02-01T10:48:18.564 DEBUG:boto:auth_path=/vclfcn-61/obj5 2024-02-01T10:48:18.564 DEBUG:boto:path=/vclfcn-61/obj5?versionId=null 2024-02-01T10:48:18.564 DEBUG:boto:auth_path=/vclfcn-61/obj5?versionId=null 2024-02-01T10:48:18.565 DEBUG:boto:Method: HEAD 2024-02-01T10:48:18.565 DEBUG:boto:Path: /vclfcn-61/obj5?versionId=null 2024-02-01T10:48:18.565 DEBUG:boto:Data: 2024-02-01T10:48:18.565 DEBUG:boto:Headers: {} 2024-02-01T10:48:18.565 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:18.565 DEBUG:boto:Port: 8000 2024-02-01T10:48:18.565 DEBUG:boto:Params: {} 2024-02-01T10:48:18.565 DEBUG:boto:Token: None 2024-02-01T10:48:18.565 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:18 GMT /vclfcn-61/obj5?versionId=null 2024-02-01T10:48:18.566 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ICeVahiiR8DrNvd0UYZ2lHDEmxw= 2024-02-01T10:48:18.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ICeVahiiR8DrNvd0UYZ2lHDEmxw=', 'Content-Length': '0'} 2024-02-01T10:48:18.569 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b23b1069be7bc0a6-0065bb76f2-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:18.569 DEBUG:boto:path=/vclfcn-61/obj5 2024-02-01T10:48:18.569 DEBUG:boto:auth_path=/vclfcn-61/obj5 2024-02-01T10:48:18.569 DEBUG:boto:path=/vclfcn-61/obj5?versionId=null 2024-02-01T10:48:18.569 DEBUG:boto:auth_path=/vclfcn-61/obj5?versionId=null 2024-02-01T10:48:18.569 DEBUG:boto:Method: HEAD 2024-02-01T10:48:18.569 DEBUG:boto:Path: /vclfcn-61/obj5?versionId=null 2024-02-01T10:48:18.569 DEBUG:boto:Data: 2024-02-01T10:48:18.569 DEBUG:boto:Headers: {} 2024-02-01T10:48:18.570 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:18.570 DEBUG:boto:Port: 8000 2024-02-01T10:48:18.570 DEBUG:boto:Params: {} 2024-02-01T10:48:18.570 DEBUG:boto:Token: None 2024-02-01T10:48:18.570 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:18 GMT /vclfcn-61/obj5?versionId=null 2024-02-01T10:48:18.570 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ICeVahiiR8DrNvd0UYZ2lHDEmxw= 2024-02-01T10:48:18.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ICeVahiiR8DrNvd0UYZ2lHDEmxw=', 'Content-Length': '0'} 2024-02-01T10:48:18.573 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cc0a53f343f4ff05-0065bb76f2-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:18.574 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-02-01T10:48:18.574 DEBUG:boto:path=/vclfcn-61/obj5 2024-02-01T10:48:18.574 DEBUG:boto:auth_path=/vclfcn-61/obj5 2024-02-01T10:48:18.574 DEBUG:boto:path=/vclfcn-61/obj5?versionId=null 2024-02-01T10:48:18.574 DEBUG:boto:auth_path=/vclfcn-61/obj5?versionId=null 2024-02-01T10:48:18.574 DEBUG:boto:Method: GET 2024-02-01T10:48:18.574 DEBUG:boto:Path: /vclfcn-61/obj5?versionId=null 2024-02-01T10:48:18.574 DEBUG:boto:Data: 2024-02-01T10:48:18.574 DEBUG:boto:Headers: {} 2024-02-01T10:48:18.575 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:18.575 DEBUG:boto:Port: 8000 2024-02-01T10:48:18.575 DEBUG:boto:Params: {} 2024-02-01T10:48:18.575 DEBUG:boto:Token: None 2024-02-01T10:48:18.575 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:18 GMT /vclfcn-61/obj5?versionId=null 2024-02-01T10:48:18.575 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tdCYuNXt1iS3Qnft7+Lq0wnl2y8= 2024-02-01T10:48:18.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tdCYuNXt1iS3Qnft7+Lq0wnl2y8=', 'Content-Length': '0'} 2024-02-01T10:48:18.578 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bee3581183cb3a8c-0065bb76f2-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:18.620 DEBUG:boto:path=/vclfcn-61/obj5 2024-02-01T10:48:18.620 DEBUG:boto:auth_path=/vclfcn-61/obj5 2024-02-01T10:48:18.620 DEBUG:boto:path=/vclfcn-61/obj5?versionId=null 2024-02-01T10:48:18.620 DEBUG:boto:auth_path=/vclfcn-61/obj5?versionId=null 2024-02-01T10:48:18.621 DEBUG:boto:Method: GET 2024-02-01T10:48:18.621 DEBUG:boto:Path: /vclfcn-61/obj5?versionId=null 2024-02-01T10:48:18.621 DEBUG:boto:Data: 2024-02-01T10:48:18.621 DEBUG:boto:Headers: {} 2024-02-01T10:48:18.621 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:18.621 DEBUG:boto:Port: 8000 2024-02-01T10:48:18.621 DEBUG:boto:Params: {} 2024-02-01T10:48:18.621 DEBUG:boto:Token: None 2024-02-01T10:48:18.621 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:18 GMT /vclfcn-61/obj5?versionId=null 2024-02-01T10:48:18.621 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tdCYuNXt1iS3Qnft7+Lq0wnl2y8= 2024-02-01T10:48:18.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tdCYuNXt1iS3Qnft7+Lq0wnl2y8=', 'Content-Length': '0'} 2024-02-01T10:48:18.624 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009b07676b1a125223-0065bb76f2-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:18.668 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:18.668 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:18.668 DEBUG:boto:path=/vclfcn-61/?prefix=obj5 2024-02-01T10:48:18.668 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj5 2024-02-01T10:48:18.668 DEBUG:boto:Method: GET 2024-02-01T10:48:18.669 DEBUG:boto:Path: /vclfcn-61/?prefix=obj5 2024-02-01T10:48:18.669 DEBUG:boto:Data: 2024-02-01T10:48:18.669 DEBUG:boto:Headers: {} 2024-02-01T10:48:18.669 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:18.669 DEBUG:boto:Port: 8000 2024-02-01T10:48:18.669 DEBUG:boto:Params: {} 2024-02-01T10:48:18.669 DEBUG:boto:Token: None 2024-02-01T10:48:18.669 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:18 GMT /vclfcn-61/ 2024-02-01T10:48:18.670 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Lr6O0kGCwbM0XrNfMdDh9GMwVKA= 2024-02-01T10:48:18.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Lr6O0kGCwbM0XrNfMdDh9GMwVKA=', 'Content-Length': '0'} 2024-02-01T10:48:18.674 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f5ce229f09512d3a-0065bb76f2-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:18.674 DEBUG:boto:b'vclfcn-61obj51000falseobj52024-02-01T10:47:30.425Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:18.674 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:18.674 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:18.675 DEBUG:boto:path=/vclfcn-61/?prefix=obj5 2024-02-01T10:48:18.675 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj5 2024-02-01T10:48:18.675 DEBUG:boto:Method: GET 2024-02-01T10:48:18.675 DEBUG:boto:Path: /vclfcn-61/?prefix=obj5 2024-02-01T10:48:18.675 DEBUG:boto:Data: 2024-02-01T10:48:18.675 DEBUG:boto:Headers: {} 2024-02-01T10:48:18.675 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:18.675 DEBUG:boto:Port: 8000 2024-02-01T10:48:18.675 DEBUG:boto:Params: {} 2024-02-01T10:48:18.675 DEBUG:boto:Token: None 2024-02-01T10:48:18.676 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:18 GMT /vclfcn-61/ 2024-02-01T10:48:18.676 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Lr6O0kGCwbM0XrNfMdDh9GMwVKA= 2024-02-01T10:48:18.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Lr6O0kGCwbM0XrNfMdDh9GMwVKA=', 'Content-Length': '0'} 2024-02-01T10:48:18.680 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005168d723ae66b4dc-0065bb76f2-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:18.728 DEBUG:boto:b'vclfcn-61obj51000falseobj52024-02-01T10:47:30.425Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:18.728 DEBUG:boto:path=/vclfcn-61/obj5 2024-02-01T10:48:18.729 DEBUG:boto:auth_path=/vclfcn-61/obj5 2024-02-01T10:48:18.729 DEBUG:boto:Method: HEAD 2024-02-01T10:48:18.729 DEBUG:boto:Path: /vclfcn-61/obj5 2024-02-01T10:48:18.729 DEBUG:boto:Data: 2024-02-01T10:48:18.729 DEBUG:boto:Headers: {} 2024-02-01T10:48:18.729 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:18.729 DEBUG:boto:Port: 8000 2024-02-01T10:48:18.729 DEBUG:boto:Params: {} 2024-02-01T10:48:18.729 DEBUG:boto:Token: None 2024-02-01T10:48:18.730 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:18 GMT /vclfcn-61/obj5 2024-02-01T10:48:18.730 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tTgjKF764a4CZ3PB1NAwAcjn23M= 2024-02-01T10:48:18.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tTgjKF764a4CZ3PB1NAwAcjn23M=', 'Content-Length': '0'} 2024-02-01T10:48:18.733 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000889770e3fa8da58e-0065bb76f2-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:18.733 DEBUG:boto:path=/vclfcn-61/obj5 2024-02-01T10:48:18.733 DEBUG:boto:auth_path=/vclfcn-61/obj5 2024-02-01T10:48:18.733 DEBUG:boto:Method: HEAD 2024-02-01T10:48:18.734 DEBUG:boto:Path: /vclfcn-61/obj5 2024-02-01T10:48:18.734 DEBUG:boto:Data: 2024-02-01T10:48:18.734 DEBUG:boto:Headers: {} 2024-02-01T10:48:18.734 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:18.734 DEBUG:boto:Port: 8000 2024-02-01T10:48:18.734 DEBUG:boto:Params: {} 2024-02-01T10:48:18.734 DEBUG:boto:Token: None 2024-02-01T10:48:18.734 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:18 GMT /vclfcn-61/obj5 2024-02-01T10:48:18.734 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tTgjKF764a4CZ3PB1NAwAcjn23M= 2024-02-01T10:48:18.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tTgjKF764a4CZ3PB1NAwAcjn23M=', 'Content-Length': '0'} 2024-02-01T10:48:18.737 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008a8d266447749e90-0065bb76f2-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:18.738 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-02-01T10:48:18.738 DEBUG:boto:path=/vclfcn-61/obj5 2024-02-01T10:48:18.738 DEBUG:boto:auth_path=/vclfcn-61/obj5 2024-02-01T10:48:18.738 DEBUG:boto:Method: GET 2024-02-01T10:48:18.738 DEBUG:boto:Path: /vclfcn-61/obj5 2024-02-01T10:48:18.738 DEBUG:boto:Data: 2024-02-01T10:48:18.738 DEBUG:boto:Headers: {} 2024-02-01T10:48:18.738 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:18.739 DEBUG:boto:Port: 8000 2024-02-01T10:48:18.739 DEBUG:boto:Params: {} 2024-02-01T10:48:18.739 DEBUG:boto:Token: None 2024-02-01T10:48:18.739 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:18 GMT /vclfcn-61/obj5 2024-02-01T10:48:18.739 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:11Z2WZCeaYrEKV+c7fs9R8KX3hc= 2024-02-01T10:48:18.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:11Z2WZCeaYrEKV+c7fs9R8KX3hc=', 'Content-Length': '0'} 2024-02-01T10:48:18.742 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fa41f4ba975041e4-0065bb76f2-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:18.784 DEBUG:boto:path=/vclfcn-61/obj5 2024-02-01T10:48:18.784 DEBUG:boto:auth_path=/vclfcn-61/obj5 2024-02-01T10:48:18.785 DEBUG:boto:Method: GET 2024-02-01T10:48:18.785 DEBUG:boto:Path: /vclfcn-61/obj5 2024-02-01T10:48:18.785 DEBUG:boto:Data: 2024-02-01T10:48:18.785 DEBUG:boto:Headers: {} 2024-02-01T10:48:18.785 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:18.785 DEBUG:boto:Port: 8000 2024-02-01T10:48:18.785 DEBUG:boto:Params: {} 2024-02-01T10:48:18.785 DEBUG:boto:Token: None 2024-02-01T10:48:18.785 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:18 GMT /vclfcn-61/obj5 2024-02-01T10:48:18.786 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:11Z2WZCeaYrEKV+c7fs9R8KX3hc= 2024-02-01T10:48:18.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:11Z2WZCeaYrEKV+c7fs9R8KX3hc=', 'Content-Length': '0'} 2024-02-01T10:48:18.789 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ccc3f8097895f1c5-0065bb76f2-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:18.832 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:18.832 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:18.832 DEBUG:boto:path=/vclfcn-61/?prefix=obj5 2024-02-01T10:48:18.833 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj5 2024-02-01T10:48:18.833 DEBUG:boto:Method: GET 2024-02-01T10:48:18.833 DEBUG:boto:Path: /vclfcn-61/?prefix=obj5 2024-02-01T10:48:18.833 DEBUG:boto:Data: 2024-02-01T10:48:18.833 DEBUG:boto:Headers: {} 2024-02-01T10:48:18.833 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:18.833 DEBUG:boto:Port: 8000 2024-02-01T10:48:18.833 DEBUG:boto:Params: {} 2024-02-01T10:48:18.833 DEBUG:boto:Token: None 2024-02-01T10:48:18.833 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:18 GMT /vclfcn-61/ 2024-02-01T10:48:18.834 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Lr6O0kGCwbM0XrNfMdDh9GMwVKA= 2024-02-01T10:48:18.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Lr6O0kGCwbM0XrNfMdDh9GMwVKA=', 'Content-Length': '0'} 2024-02-01T10:48:18.839 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e56d37fe353ce6e3-0065bb76f2-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:18.884 DEBUG:boto:b'vclfcn-61obj51000falseobj52024-02-01T10:47:30.425Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:18.885 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:18.885 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:18.885 DEBUG:boto:path=/vclfcn-61/?prefix=obj5 2024-02-01T10:48:18.885 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj5 2024-02-01T10:48:18.885 DEBUG:boto:Method: GET 2024-02-01T10:48:18.885 DEBUG:boto:Path: /vclfcn-61/?prefix=obj5 2024-02-01T10:48:18.885 DEBUG:boto:Data: 2024-02-01T10:48:18.885 DEBUG:boto:Headers: {} 2024-02-01T10:48:18.885 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:18.886 DEBUG:boto:Port: 8000 2024-02-01T10:48:18.886 DEBUG:boto:Params: {} 2024-02-01T10:48:18.886 DEBUG:boto:Token: None 2024-02-01T10:48:18.886 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:18 GMT /vclfcn-61/ 2024-02-01T10:48:18.886 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Lr6O0kGCwbM0XrNfMdDh9GMwVKA= 2024-02-01T10:48:18.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Lr6O0kGCwbM0XrNfMdDh9GMwVKA=', 'Content-Length': '0'} 2024-02-01T10:48:18.891 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000075ea58c9812e87f0-0065bb76f2-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:18.891 DEBUG:boto:b'vclfcn-61obj51000falseobj52024-02-01T10:47:30.425Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:18.891 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-02-01T10:48:18.891 DEBUG:boto:path=/vclfcn-61/obj6 2024-02-01T10:48:18.891 DEBUG:boto:auth_path=/vclfcn-61/obj6 2024-02-01T10:48:18.892 DEBUG:boto:path=/vclfcn-61/obj6?versionId=null 2024-02-01T10:48:18.892 DEBUG:boto:auth_path=/vclfcn-61/obj6?versionId=null 2024-02-01T10:48:18.892 DEBUG:boto:Method: GET 2024-02-01T10:48:18.892 DEBUG:boto:Path: /vclfcn-61/obj6?versionId=null 2024-02-01T10:48:18.892 DEBUG:boto:Data: 2024-02-01T10:48:18.892 DEBUG:boto:Headers: {} 2024-02-01T10:48:18.892 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:18.892 DEBUG:boto:Port: 8000 2024-02-01T10:48:18.892 DEBUG:boto:Params: {} 2024-02-01T10:48:18.892 DEBUG:boto:Token: None 2024-02-01T10:48:18.893 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:18 GMT /vclfcn-61/obj6?versionId=null 2024-02-01T10:48:18.893 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:xjaDQeFdL0J4i3FU7idJYlAcsGw= 2024-02-01T10:48:18.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:xjaDQeFdL0J4i3FU7idJYlAcsGw=', 'Content-Length': '0'} 2024-02-01T10:48:18.895 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000011f4d211784401a0-0065bb76f2-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:18.896 DEBUG:boto:path=/vclfcn-61/obj6 2024-02-01T10:48:18.896 DEBUG:boto:auth_path=/vclfcn-61/obj6 2024-02-01T10:48:18.896 DEBUG:boto:path=/vclfcn-61/obj6?versionId=null 2024-02-01T10:48:18.896 DEBUG:boto:auth_path=/vclfcn-61/obj6?versionId=null 2024-02-01T10:48:18.896 DEBUG:boto:Method: GET 2024-02-01T10:48:18.896 DEBUG:boto:Path: /vclfcn-61/obj6?versionId=null 2024-02-01T10:48:18.896 DEBUG:boto:Data: 2024-02-01T10:48:18.896 DEBUG:boto:Headers: {} 2024-02-01T10:48:18.896 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:18.897 DEBUG:boto:Port: 8000 2024-02-01T10:48:18.897 DEBUG:boto:Params: {} 2024-02-01T10:48:18.897 DEBUG:boto:Token: None 2024-02-01T10:48:18.897 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:18 GMT /vclfcn-61/obj6?versionId=null 2024-02-01T10:48:18.897 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:xjaDQeFdL0J4i3FU7idJYlAcsGw= 2024-02-01T10:48:18.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:xjaDQeFdL0J4i3FU7idJYlAcsGw=', 'Content-Length': '0'} 2024-02-01T10:48:18.900 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ae29a2a1a6446704-0065bb76f2-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:18.948 DEBUG:boto:path=/vclfcn-61/obj6 2024-02-01T10:48:18.948 DEBUG:boto:auth_path=/vclfcn-61/obj6 2024-02-01T10:48:18.949 DEBUG:boto:path=/vclfcn-61/obj6?versionId=null 2024-02-01T10:48:18.949 DEBUG:boto:auth_path=/vclfcn-61/obj6?versionId=null 2024-02-01T10:48:18.949 DEBUG:boto:Method: HEAD 2024-02-01T10:48:18.949 DEBUG:boto:Path: /vclfcn-61/obj6?versionId=null 2024-02-01T10:48:18.949 DEBUG:boto:Data: 2024-02-01T10:48:18.949 DEBUG:boto:Headers: {} 2024-02-01T10:48:18.949 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:18.949 DEBUG:boto:Port: 8000 2024-02-01T10:48:18.949 DEBUG:boto:Params: {} 2024-02-01T10:48:18.949 DEBUG:boto:Token: None 2024-02-01T10:48:18.950 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:18 GMT /vclfcn-61/obj6?versionId=null 2024-02-01T10:48:18.950 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:k3nk/5s5AgEx0t/x6YBg8CF0gRQ= 2024-02-01T10:48:18.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:k3nk/5s5AgEx0t/x6YBg8CF0gRQ=', 'Content-Length': '0'} 2024-02-01T10:48:18.953 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009defd2b06025a610-0065bb76f2-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:18.953 DEBUG:boto:path=/vclfcn-61/obj6 2024-02-01T10:48:18.953 DEBUG:boto:auth_path=/vclfcn-61/obj6 2024-02-01T10:48:18.953 DEBUG:boto:path=/vclfcn-61/obj6?versionId=null 2024-02-01T10:48:18.953 DEBUG:boto:auth_path=/vclfcn-61/obj6?versionId=null 2024-02-01T10:48:18.953 DEBUG:boto:Method: HEAD 2024-02-01T10:48:18.953 DEBUG:boto:Path: /vclfcn-61/obj6?versionId=null 2024-02-01T10:48:18.954 DEBUG:boto:Data: 2024-02-01T10:48:18.954 DEBUG:boto:Headers: {} 2024-02-01T10:48:18.954 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:18.954 DEBUG:boto:Port: 8000 2024-02-01T10:48:18.954 DEBUG:boto:Params: {} 2024-02-01T10:48:18.954 DEBUG:boto:Token: None 2024-02-01T10:48:18.954 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:18 GMT /vclfcn-61/obj6?versionId=null 2024-02-01T10:48:18.954 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:k3nk/5s5AgEx0t/x6YBg8CF0gRQ= 2024-02-01T10:48:18.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:k3nk/5s5AgEx0t/x6YBg8CF0gRQ=', 'Content-Length': '0'} 2024-02-01T10:48:18.957 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bd4693f65420b817-0065bb76f2-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:18.958 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-02-01T10:48:18.958 DEBUG:boto:path=/vclfcn-61/obj6 2024-02-01T10:48:18.958 DEBUG:boto:auth_path=/vclfcn-61/obj6 2024-02-01T10:48:18.958 DEBUG:boto:path=/vclfcn-61/obj6?versionId=null 2024-02-01T10:48:18.958 DEBUG:boto:auth_path=/vclfcn-61/obj6?versionId=null 2024-02-01T10:48:18.958 DEBUG:boto:Method: GET 2024-02-01T10:48:18.958 DEBUG:boto:Path: /vclfcn-61/obj6?versionId=null 2024-02-01T10:48:18.958 DEBUG:boto:Data: 2024-02-01T10:48:18.958 DEBUG:boto:Headers: {} 2024-02-01T10:48:18.959 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:18.959 DEBUG:boto:Port: 8000 2024-02-01T10:48:18.959 DEBUG:boto:Params: {} 2024-02-01T10:48:18.959 DEBUG:boto:Token: None 2024-02-01T10:48:18.959 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:18 GMT /vclfcn-61/obj6?versionId=null 2024-02-01T10:48:18.959 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:xjaDQeFdL0J4i3FU7idJYlAcsGw= 2024-02-01T10:48:18.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:18 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:xjaDQeFdL0J4i3FU7idJYlAcsGw=', 'Content-Length': '0'} 2024-02-01T10:48:18.962 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c3e399a2976be9e4-0065bb76f2-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:18 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:19.008 DEBUG:boto:path=/vclfcn-61/obj6 2024-02-01T10:48:19.008 DEBUG:boto:auth_path=/vclfcn-61/obj6 2024-02-01T10:48:19.008 DEBUG:boto:path=/vclfcn-61/obj6?versionId=null 2024-02-01T10:48:19.009 DEBUG:boto:auth_path=/vclfcn-61/obj6?versionId=null 2024-02-01T10:48:19.009 DEBUG:boto:Method: GET 2024-02-01T10:48:19.009 DEBUG:boto:Path: /vclfcn-61/obj6?versionId=null 2024-02-01T10:48:19.009 DEBUG:boto:Data: 2024-02-01T10:48:19.009 DEBUG:boto:Headers: {} 2024-02-01T10:48:19.009 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:19.009 DEBUG:boto:Port: 8000 2024-02-01T10:48:19.009 DEBUG:boto:Params: {} 2024-02-01T10:48:19.009 DEBUG:boto:Token: None 2024-02-01T10:48:19.009 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:19 GMT /vclfcn-61/obj6?versionId=null 2024-02-01T10:48:19.010 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9seH4tjIvfTMchS/vH5Ta+dIEFc= 2024-02-01T10:48:19.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9seH4tjIvfTMchS/vH5Ta+dIEFc=', 'Content-Length': '0'} 2024-02-01T10:48:19.012 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003fcda3f6ac598f24-0065bb76f3-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:19.056 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:19.056 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:19.056 DEBUG:boto:path=/vclfcn-61/?prefix=obj6 2024-02-01T10:48:19.056 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj6 2024-02-01T10:48:19.056 DEBUG:boto:Method: GET 2024-02-01T10:48:19.057 DEBUG:boto:Path: /vclfcn-61/?prefix=obj6 2024-02-01T10:48:19.057 DEBUG:boto:Data: 2024-02-01T10:48:19.057 DEBUG:boto:Headers: {} 2024-02-01T10:48:19.057 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:19.057 DEBUG:boto:Port: 8000 2024-02-01T10:48:19.057 DEBUG:boto:Params: {} 2024-02-01T10:48:19.057 DEBUG:boto:Token: None 2024-02-01T10:48:19.057 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:19 GMT /vclfcn-61/ 2024-02-01T10:48:19.057 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tgDZeB7mWfYALFkrBKG3K//eUvo= 2024-02-01T10:48:19.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tgDZeB7mWfYALFkrBKG3K//eUvo=', 'Content-Length': '0'} 2024-02-01T10:48:19.062 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cedfabc0418e4a2f-0065bb76f3-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:19.104 DEBUG:boto:b'vclfcn-61obj61000falseobj62024-02-01T10:47:30.440Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:19.105 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:19.105 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:19.105 DEBUG:boto:path=/vclfcn-61/?prefix=obj6 2024-02-01T10:48:19.105 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj6 2024-02-01T10:48:19.105 DEBUG:boto:Method: GET 2024-02-01T10:48:19.105 DEBUG:boto:Path: /vclfcn-61/?prefix=obj6 2024-02-01T10:48:19.105 DEBUG:boto:Data: 2024-02-01T10:48:19.105 DEBUG:boto:Headers: {} 2024-02-01T10:48:19.105 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:19.105 DEBUG:boto:Port: 8000 2024-02-01T10:48:19.105 DEBUG:boto:Params: {} 2024-02-01T10:48:19.106 DEBUG:boto:Token: None 2024-02-01T10:48:19.106 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:19 GMT /vclfcn-61/ 2024-02-01T10:48:19.106 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tgDZeB7mWfYALFkrBKG3K//eUvo= 2024-02-01T10:48:19.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tgDZeB7mWfYALFkrBKG3K//eUvo=', 'Content-Length': '0'} 2024-02-01T10:48:19.110 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004686baddd492ab76-0065bb76f3-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:19.110 DEBUG:boto:b'vclfcn-61obj61000falseobj62024-02-01T10:47:30.440Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:19.111 DEBUG:boto:path=/vclfcn-61/obj6 2024-02-01T10:48:19.111 DEBUG:boto:auth_path=/vclfcn-61/obj6 2024-02-01T10:48:19.111 DEBUG:boto:Method: HEAD 2024-02-01T10:48:19.111 DEBUG:boto:Path: /vclfcn-61/obj6 2024-02-01T10:48:19.111 DEBUG:boto:Data: 2024-02-01T10:48:19.111 DEBUG:boto:Headers: {} 2024-02-01T10:48:19.111 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:19.111 DEBUG:boto:Port: 8000 2024-02-01T10:48:19.111 DEBUG:boto:Params: {} 2024-02-01T10:48:19.112 DEBUG:boto:Token: None 2024-02-01T10:48:19.112 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:19 GMT /vclfcn-61/obj6 2024-02-01T10:48:19.112 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Y1r4KvEtU6zfSHFUanQ6UO4hG6o= 2024-02-01T10:48:19.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Y1r4KvEtU6zfSHFUanQ6UO4hG6o=', 'Content-Length': '0'} 2024-02-01T10:48:19.115 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000081a75e2b1f9389f6-0065bb76f3-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:19.115 DEBUG:boto:path=/vclfcn-61/obj6 2024-02-01T10:48:19.115 DEBUG:boto:auth_path=/vclfcn-61/obj6 2024-02-01T10:48:19.115 DEBUG:boto:Method: HEAD 2024-02-01T10:48:19.115 DEBUG:boto:Path: /vclfcn-61/obj6 2024-02-01T10:48:19.115 DEBUG:boto:Data: 2024-02-01T10:48:19.115 DEBUG:boto:Headers: {} 2024-02-01T10:48:19.115 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:19.116 DEBUG:boto:Port: 8000 2024-02-01T10:48:19.116 DEBUG:boto:Params: {} 2024-02-01T10:48:19.116 DEBUG:boto:Token: None 2024-02-01T10:48:19.116 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:19 GMT /vclfcn-61/obj6 2024-02-01T10:48:19.116 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Y1r4KvEtU6zfSHFUanQ6UO4hG6o= 2024-02-01T10:48:19.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Y1r4KvEtU6zfSHFUanQ6UO4hG6o=', 'Content-Length': '0'} 2024-02-01T10:48:19.119 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ce8c40d6f9bc2780-0065bb76f3-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:19.119 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-02-01T10:48:19.120 DEBUG:boto:path=/vclfcn-61/obj6 2024-02-01T10:48:19.120 DEBUG:boto:auth_path=/vclfcn-61/obj6 2024-02-01T10:48:19.120 DEBUG:boto:Method: GET 2024-02-01T10:48:19.123 DEBUG:boto:Path: /vclfcn-61/obj6 2024-02-01T10:48:19.123 DEBUG:boto:Data: 2024-02-01T10:48:19.123 DEBUG:boto:Headers: {} 2024-02-01T10:48:19.124 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:19.124 DEBUG:boto:Port: 8000 2024-02-01T10:48:19.124 DEBUG:boto:Params: {} 2024-02-01T10:48:19.124 DEBUG:boto:Token: None 2024-02-01T10:48:19.124 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:19 GMT /vclfcn-61/obj6 2024-02-01T10:48:19.124 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:pRAb85zH68JFfc6CDx11q/iKnCM= 2024-02-01T10:48:19.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:pRAb85zH68JFfc6CDx11q/iKnCM=', 'Content-Length': '0'} 2024-02-01T10:48:19.127 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d970a097fcbfa376-0065bb76f3-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:19.169 DEBUG:boto:path=/vclfcn-61/obj6 2024-02-01T10:48:19.169 DEBUG:boto:auth_path=/vclfcn-61/obj6 2024-02-01T10:48:19.169 DEBUG:boto:Method: GET 2024-02-01T10:48:19.169 DEBUG:boto:Path: /vclfcn-61/obj6 2024-02-01T10:48:19.169 DEBUG:boto:Data: 2024-02-01T10:48:19.169 DEBUG:boto:Headers: {} 2024-02-01T10:48:19.169 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:19.169 DEBUG:boto:Port: 8000 2024-02-01T10:48:19.169 DEBUG:boto:Params: {} 2024-02-01T10:48:19.170 DEBUG:boto:Token: None 2024-02-01T10:48:19.170 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:19 GMT /vclfcn-61/obj6 2024-02-01T10:48:19.170 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:pRAb85zH68JFfc6CDx11q/iKnCM= 2024-02-01T10:48:19.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:pRAb85zH68JFfc6CDx11q/iKnCM=', 'Content-Length': '0'} 2024-02-01T10:48:19.173 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000927c6bab831243df-0065bb76f3-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:19.220 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:19.220 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:19.220 DEBUG:boto:path=/vclfcn-61/?prefix=obj6 2024-02-01T10:48:19.221 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj6 2024-02-01T10:48:19.221 DEBUG:boto:Method: GET 2024-02-01T10:48:19.221 DEBUG:boto:Path: /vclfcn-61/?prefix=obj6 2024-02-01T10:48:19.221 DEBUG:boto:Data: 2024-02-01T10:48:19.221 DEBUG:boto:Headers: {} 2024-02-01T10:48:19.221 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:19.221 DEBUG:boto:Port: 8000 2024-02-01T10:48:19.221 DEBUG:boto:Params: {} 2024-02-01T10:48:19.221 DEBUG:boto:Token: None 2024-02-01T10:48:19.221 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:19 GMT /vclfcn-61/ 2024-02-01T10:48:19.222 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tgDZeB7mWfYALFkrBKG3K//eUvo= 2024-02-01T10:48:19.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tgDZeB7mWfYALFkrBKG3K//eUvo=', 'Content-Length': '0'} 2024-02-01T10:48:19.226 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f72e0afd02f273cf-0065bb76f3-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:19.227 DEBUG:boto:b'vclfcn-61obj61000falseobj62024-02-01T10:47:30.440Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:19.228 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:19.228 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:19.228 DEBUG:boto:path=/vclfcn-61/?prefix=obj6 2024-02-01T10:48:19.228 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj6 2024-02-01T10:48:19.228 DEBUG:boto:Method: GET 2024-02-01T10:48:19.228 DEBUG:boto:Path: /vclfcn-61/?prefix=obj6 2024-02-01T10:48:19.228 DEBUG:boto:Data: 2024-02-01T10:48:19.228 DEBUG:boto:Headers: {} 2024-02-01T10:48:19.228 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:19.228 DEBUG:boto:Port: 8000 2024-02-01T10:48:19.228 DEBUG:boto:Params: {} 2024-02-01T10:48:19.229 DEBUG:boto:Token: None 2024-02-01T10:48:19.229 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:19 GMT /vclfcn-61/ 2024-02-01T10:48:19.229 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tgDZeB7mWfYALFkrBKG3K//eUvo= 2024-02-01T10:48:19.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tgDZeB7mWfYALFkrBKG3K//eUvo=', 'Content-Length': '0'} 2024-02-01T10:48:19.234 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005916e0b07598a67f-0065bb76f3-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:19.234 DEBUG:boto:b'vclfcn-61obj61000falseobj62024-02-01T10:47:30.440Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:19.234 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-02-01T10:48:19.234 DEBUG:boto:path=/vclfcn-61/obj7 2024-02-01T10:48:19.235 DEBUG:boto:auth_path=/vclfcn-61/obj7 2024-02-01T10:48:19.235 DEBUG:boto:path=/vclfcn-61/obj7?versionId=null 2024-02-01T10:48:19.235 DEBUG:boto:auth_path=/vclfcn-61/obj7?versionId=null 2024-02-01T10:48:19.235 DEBUG:boto:Method: GET 2024-02-01T10:48:19.235 DEBUG:boto:Path: /vclfcn-61/obj7?versionId=null 2024-02-01T10:48:19.235 DEBUG:boto:Data: 2024-02-01T10:48:19.235 DEBUG:boto:Headers: {} 2024-02-01T10:48:19.235 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:19.235 DEBUG:boto:Port: 8000 2024-02-01T10:48:19.235 DEBUG:boto:Params: {} 2024-02-01T10:48:19.235 DEBUG:boto:Token: None 2024-02-01T10:48:19.236 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:19 GMT /vclfcn-61/obj7?versionId=null 2024-02-01T10:48:19.236 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:14SZGBF7C/wfdx5/NJlrYJ9uG8k= 2024-02-01T10:48:19.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:14SZGBF7C/wfdx5/NJlrYJ9uG8k=', 'Content-Length': '0'} 2024-02-01T10:48:19.239 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001ef6546680653a89-0065bb76f3-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:19.280 DEBUG:boto:path=/vclfcn-61/obj7 2024-02-01T10:48:19.280 DEBUG:boto:auth_path=/vclfcn-61/obj7 2024-02-01T10:48:19.280 DEBUG:boto:path=/vclfcn-61/obj7?versionId=null 2024-02-01T10:48:19.280 DEBUG:boto:auth_path=/vclfcn-61/obj7?versionId=null 2024-02-01T10:48:19.280 DEBUG:boto:Method: GET 2024-02-01T10:48:19.281 DEBUG:boto:Path: /vclfcn-61/obj7?versionId=null 2024-02-01T10:48:19.281 DEBUG:boto:Data: 2024-02-01T10:48:19.281 DEBUG:boto:Headers: {} 2024-02-01T10:48:19.281 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:19.281 DEBUG:boto:Port: 8000 2024-02-01T10:48:19.281 DEBUG:boto:Params: {} 2024-02-01T10:48:19.281 DEBUG:boto:Token: None 2024-02-01T10:48:19.281 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:19 GMT /vclfcn-61/obj7?versionId=null 2024-02-01T10:48:19.281 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:14SZGBF7C/wfdx5/NJlrYJ9uG8k= 2024-02-01T10:48:19.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:14SZGBF7C/wfdx5/NJlrYJ9uG8k=', 'Content-Length': '0'} 2024-02-01T10:48:19.285 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b43a55147268db7b-0065bb76f3-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:19.285 DEBUG:boto:path=/vclfcn-61/obj7 2024-02-01T10:48:19.285 DEBUG:boto:auth_path=/vclfcn-61/obj7 2024-02-01T10:48:19.286 DEBUG:boto:path=/vclfcn-61/obj7?versionId=null 2024-02-01T10:48:19.286 DEBUG:boto:auth_path=/vclfcn-61/obj7?versionId=null 2024-02-01T10:48:19.286 DEBUG:boto:Method: HEAD 2024-02-01T10:48:19.286 DEBUG:boto:Path: /vclfcn-61/obj7?versionId=null 2024-02-01T10:48:19.286 DEBUG:boto:Data: 2024-02-01T10:48:19.286 DEBUG:boto:Headers: {} 2024-02-01T10:48:19.286 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:19.286 DEBUG:boto:Port: 8000 2024-02-01T10:48:19.286 DEBUG:boto:Params: {} 2024-02-01T10:48:19.287 DEBUG:boto:Token: None 2024-02-01T10:48:19.287 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:19 GMT /vclfcn-61/obj7?versionId=null 2024-02-01T10:48:19.287 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0zGvjMbIiAAx3ND6dWCrjbiyHYM= 2024-02-01T10:48:19.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0zGvjMbIiAAx3ND6dWCrjbiyHYM=', 'Content-Length': '0'} 2024-02-01T10:48:19.290 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002b69734c027be7cb-0065bb76f3-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:19.290 DEBUG:boto:path=/vclfcn-61/obj7 2024-02-01T10:48:19.290 DEBUG:boto:auth_path=/vclfcn-61/obj7 2024-02-01T10:48:19.290 DEBUG:boto:path=/vclfcn-61/obj7?versionId=null 2024-02-01T10:48:19.290 DEBUG:boto:auth_path=/vclfcn-61/obj7?versionId=null 2024-02-01T10:48:19.290 DEBUG:boto:Method: HEAD 2024-02-01T10:48:19.290 DEBUG:boto:Path: /vclfcn-61/obj7?versionId=null 2024-02-01T10:48:19.291 DEBUG:boto:Data: 2024-02-01T10:48:19.291 DEBUG:boto:Headers: {} 2024-02-01T10:48:19.291 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:19.291 DEBUG:boto:Port: 8000 2024-02-01T10:48:19.291 DEBUG:boto:Params: {} 2024-02-01T10:48:19.291 DEBUG:boto:Token: None 2024-02-01T10:48:19.291 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:19 GMT /vclfcn-61/obj7?versionId=null 2024-02-01T10:48:19.291 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0zGvjMbIiAAx3ND6dWCrjbiyHYM= 2024-02-01T10:48:19.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0zGvjMbIiAAx3ND6dWCrjbiyHYM=', 'Content-Length': '0'} 2024-02-01T10:48:19.294 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000be40d806ad4cd75d-0065bb76f3-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:19.294 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-02-01T10:48:19.294 DEBUG:boto:path=/vclfcn-61/obj7 2024-02-01T10:48:19.295 DEBUG:boto:auth_path=/vclfcn-61/obj7 2024-02-01T10:48:19.295 DEBUG:boto:path=/vclfcn-61/obj7?versionId=null 2024-02-01T10:48:19.295 DEBUG:boto:auth_path=/vclfcn-61/obj7?versionId=null 2024-02-01T10:48:19.295 DEBUG:boto:Method: GET 2024-02-01T10:48:19.295 DEBUG:boto:Path: /vclfcn-61/obj7?versionId=null 2024-02-01T10:48:19.295 DEBUG:boto:Data: 2024-02-01T10:48:19.295 DEBUG:boto:Headers: {} 2024-02-01T10:48:19.295 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:19.295 DEBUG:boto:Port: 8000 2024-02-01T10:48:19.295 DEBUG:boto:Params: {} 2024-02-01T10:48:19.295 DEBUG:boto:Token: None 2024-02-01T10:48:19.296 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:19 GMT /vclfcn-61/obj7?versionId=null 2024-02-01T10:48:19.296 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:14SZGBF7C/wfdx5/NJlrYJ9uG8k= 2024-02-01T10:48:19.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:14SZGBF7C/wfdx5/NJlrYJ9uG8k=', 'Content-Length': '0'} 2024-02-01T10:48:19.299 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b7499d8026e311d7-0065bb76f3-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:19.340 DEBUG:boto:path=/vclfcn-61/obj7 2024-02-01T10:48:19.340 DEBUG:boto:auth_path=/vclfcn-61/obj7 2024-02-01T10:48:19.340 DEBUG:boto:path=/vclfcn-61/obj7?versionId=null 2024-02-01T10:48:19.341 DEBUG:boto:auth_path=/vclfcn-61/obj7?versionId=null 2024-02-01T10:48:19.341 DEBUG:boto:Method: GET 2024-02-01T10:48:19.341 DEBUG:boto:Path: /vclfcn-61/obj7?versionId=null 2024-02-01T10:48:19.341 DEBUG:boto:Data: 2024-02-01T10:48:19.341 DEBUG:boto:Headers: {} 2024-02-01T10:48:19.342 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:19.342 DEBUG:boto:Port: 8000 2024-02-01T10:48:19.342 DEBUG:boto:Params: {} 2024-02-01T10:48:19.342 DEBUG:boto:Token: None 2024-02-01T10:48:19.342 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:19 GMT /vclfcn-61/obj7?versionId=null 2024-02-01T10:48:19.343 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:14SZGBF7C/wfdx5/NJlrYJ9uG8k= 2024-02-01T10:48:19.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:14SZGBF7C/wfdx5/NJlrYJ9uG8k=', 'Content-Length': '0'} 2024-02-01T10:48:19.346 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000b4522aa22b95059-0065bb76f3-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:19.388 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:19.388 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:19.389 DEBUG:boto:path=/vclfcn-61/?prefix=obj7 2024-02-01T10:48:19.389 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj7 2024-02-01T10:48:19.389 DEBUG:boto:Method: GET 2024-02-01T10:48:19.389 DEBUG:boto:Path: /vclfcn-61/?prefix=obj7 2024-02-01T10:48:19.390 DEBUG:boto:Data: 2024-02-01T10:48:19.390 DEBUG:boto:Headers: {} 2024-02-01T10:48:19.390 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:19.390 DEBUG:boto:Port: 8000 2024-02-01T10:48:19.390 DEBUG:boto:Params: {} 2024-02-01T10:48:19.390 DEBUG:boto:Token: None 2024-02-01T10:48:19.391 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:19 GMT /vclfcn-61/ 2024-02-01T10:48:19.391 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tgDZeB7mWfYALFkrBKG3K//eUvo= 2024-02-01T10:48:19.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tgDZeB7mWfYALFkrBKG3K//eUvo=', 'Content-Length': '0'} 2024-02-01T10:48:19.396 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f5919c797f2d35a5-0065bb76f3-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:19.444 DEBUG:boto:b'vclfcn-61obj71000falseobj72024-02-01T10:47:30.492Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:19.445 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:19.445 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:19.445 DEBUG:boto:path=/vclfcn-61/?prefix=obj7 2024-02-01T10:48:19.445 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj7 2024-02-01T10:48:19.445 DEBUG:boto:Method: GET 2024-02-01T10:48:19.446 DEBUG:boto:Path: /vclfcn-61/?prefix=obj7 2024-02-01T10:48:19.446 DEBUG:boto:Data: 2024-02-01T10:48:19.446 DEBUG:boto:Headers: {} 2024-02-01T10:48:19.446 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:19.446 DEBUG:boto:Port: 8000 2024-02-01T10:48:19.446 DEBUG:boto:Params: {} 2024-02-01T10:48:19.446 DEBUG:boto:Token: None 2024-02-01T10:48:19.447 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:19 GMT /vclfcn-61/ 2024-02-01T10:48:19.447 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tgDZeB7mWfYALFkrBKG3K//eUvo= 2024-02-01T10:48:19.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tgDZeB7mWfYALFkrBKG3K//eUvo=', 'Content-Length': '0'} 2024-02-01T10:48:19.451 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ee5fcba9ecf2066-0065bb76f3-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:19.452 DEBUG:boto:b'vclfcn-61obj71000falseobj72024-02-01T10:47:30.492Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:19.452 DEBUG:boto:path=/vclfcn-61/obj7 2024-02-01T10:48:19.452 DEBUG:boto:auth_path=/vclfcn-61/obj7 2024-02-01T10:48:19.452 DEBUG:boto:Method: HEAD 2024-02-01T10:48:19.453 DEBUG:boto:Path: /vclfcn-61/obj7 2024-02-01T10:48:19.453 DEBUG:boto:Data: 2024-02-01T10:48:19.453 DEBUG:boto:Headers: {} 2024-02-01T10:48:19.453 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:19.453 DEBUG:boto:Port: 8000 2024-02-01T10:48:19.453 DEBUG:boto:Params: {} 2024-02-01T10:48:19.453 DEBUG:boto:Token: None 2024-02-01T10:48:19.454 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:19 GMT /vclfcn-61/obj7 2024-02-01T10:48:19.454 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BrwwiXlb4CzgByiSFt1X0KkZQek= 2024-02-01T10:48:19.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BrwwiXlb4CzgByiSFt1X0KkZQek=', 'Content-Length': '0'} 2024-02-01T10:48:19.457 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000052b26a09b3d6deaa-0065bb76f3-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:19.457 DEBUG:boto:path=/vclfcn-61/obj7 2024-02-01T10:48:19.457 DEBUG:boto:auth_path=/vclfcn-61/obj7 2024-02-01T10:48:19.458 DEBUG:boto:Method: HEAD 2024-02-01T10:48:19.458 DEBUG:boto:Path: /vclfcn-61/obj7 2024-02-01T10:48:19.458 DEBUG:boto:Data: 2024-02-01T10:48:19.458 DEBUG:boto:Headers: {} 2024-02-01T10:48:19.458 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:19.459 DEBUG:boto:Port: 8000 2024-02-01T10:48:19.459 DEBUG:boto:Params: {} 2024-02-01T10:48:19.459 DEBUG:boto:Token: None 2024-02-01T10:48:19.459 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:19 GMT /vclfcn-61/obj7 2024-02-01T10:48:19.460 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BrwwiXlb4CzgByiSFt1X0KkZQek= 2024-02-01T10:48:19.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BrwwiXlb4CzgByiSFt1X0KkZQek=', 'Content-Length': '0'} 2024-02-01T10:48:19.463 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e85e1512cffe4f54-0065bb76f3-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:19.463 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-02-01T10:48:19.463 DEBUG:boto:path=/vclfcn-61/obj7 2024-02-01T10:48:19.463 DEBUG:boto:auth_path=/vclfcn-61/obj7 2024-02-01T10:48:19.464 DEBUG:boto:Method: GET 2024-02-01T10:48:19.464 DEBUG:boto:Path: /vclfcn-61/obj7 2024-02-01T10:48:19.464 DEBUG:boto:Data: 2024-02-01T10:48:19.464 DEBUG:boto:Headers: {} 2024-02-01T10:48:19.464 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:19.464 DEBUG:boto:Port: 8000 2024-02-01T10:48:19.464 DEBUG:boto:Params: {} 2024-02-01T10:48:19.464 DEBUG:boto:Token: None 2024-02-01T10:48:19.465 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:19 GMT /vclfcn-61/obj7 2024-02-01T10:48:19.465 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gFDznmIC94TjByfBJrJX/SYIKy4= 2024-02-01T10:48:19.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gFDznmIC94TjByfBJrJX/SYIKy4=', 'Content-Length': '0'} 2024-02-01T10:48:19.468 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009084ddba43e0e731-0065bb76f3-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:19.516 DEBUG:boto:path=/vclfcn-61/obj7 2024-02-01T10:48:19.516 DEBUG:boto:auth_path=/vclfcn-61/obj7 2024-02-01T10:48:19.516 DEBUG:boto:Method: GET 2024-02-01T10:48:19.516 DEBUG:boto:Path: /vclfcn-61/obj7 2024-02-01T10:48:19.516 DEBUG:boto:Data: 2024-02-01T10:48:19.517 DEBUG:boto:Headers: {} 2024-02-01T10:48:19.517 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:19.517 DEBUG:boto:Port: 8000 2024-02-01T10:48:19.517 DEBUG:boto:Params: {} 2024-02-01T10:48:19.517 DEBUG:boto:Token: None 2024-02-01T10:48:19.517 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:19 GMT /vclfcn-61/obj7 2024-02-01T10:48:19.517 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gFDznmIC94TjByfBJrJX/SYIKy4= 2024-02-01T10:48:19.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gFDznmIC94TjByfBJrJX/SYIKy4=', 'Content-Length': '0'} 2024-02-01T10:48:19.520 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001b3a1e741c62c8c1-0065bb76f3-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:19.568 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:19.568 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:19.569 DEBUG:boto:path=/vclfcn-61/?prefix=obj7 2024-02-01T10:48:19.569 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj7 2024-02-01T10:48:19.569 DEBUG:boto:Method: GET 2024-02-01T10:48:19.569 DEBUG:boto:Path: /vclfcn-61/?prefix=obj7 2024-02-01T10:48:19.569 DEBUG:boto:Data: 2024-02-01T10:48:19.569 DEBUG:boto:Headers: {} 2024-02-01T10:48:19.569 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:19.569 DEBUG:boto:Port: 8000 2024-02-01T10:48:19.569 DEBUG:boto:Params: {} 2024-02-01T10:48:19.570 DEBUG:boto:Token: None 2024-02-01T10:48:19.570 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:19 GMT /vclfcn-61/ 2024-02-01T10:48:19.570 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tgDZeB7mWfYALFkrBKG3K//eUvo= 2024-02-01T10:48:19.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tgDZeB7mWfYALFkrBKG3K//eUvo=', 'Content-Length': '0'} 2024-02-01T10:48:19.574 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089680342f1be92da-0065bb76f3-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:19.616 DEBUG:boto:b'vclfcn-61obj71000falseobj72024-02-01T10:47:30.492Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:19.617 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:19.617 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:19.617 DEBUG:boto:path=/vclfcn-61/?prefix=obj7 2024-02-01T10:48:19.617 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj7 2024-02-01T10:48:19.617 DEBUG:boto:Method: GET 2024-02-01T10:48:19.617 DEBUG:boto:Path: /vclfcn-61/?prefix=obj7 2024-02-01T10:48:19.617 DEBUG:boto:Data: 2024-02-01T10:48:19.617 DEBUG:boto:Headers: {} 2024-02-01T10:48:19.617 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:19.618 DEBUG:boto:Port: 8000 2024-02-01T10:48:19.618 DEBUG:boto:Params: {} 2024-02-01T10:48:19.618 DEBUG:boto:Token: None 2024-02-01T10:48:19.618 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:19 GMT /vclfcn-61/ 2024-02-01T10:48:19.618 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tgDZeB7mWfYALFkrBKG3K//eUvo= 2024-02-01T10:48:19.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tgDZeB7mWfYALFkrBKG3K//eUvo=', 'Content-Length': '0'} 2024-02-01T10:48:19.622 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000990eed5de1ca1fb6-0065bb76f3-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:19.664 DEBUG:boto:b'vclfcn-61obj71000falseobj72024-02-01T10:47:30.492Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:19.664 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-02-01T10:48:19.665 DEBUG:boto:path=/vclfcn-61/obj8 2024-02-01T10:48:19.665 DEBUG:boto:auth_path=/vclfcn-61/obj8 2024-02-01T10:48:19.665 DEBUG:boto:path=/vclfcn-61/obj8?versionId=null 2024-02-01T10:48:19.665 DEBUG:boto:auth_path=/vclfcn-61/obj8?versionId=null 2024-02-01T10:48:19.665 DEBUG:boto:Method: GET 2024-02-01T10:48:19.665 DEBUG:boto:Path: /vclfcn-61/obj8?versionId=null 2024-02-01T10:48:19.665 DEBUG:boto:Data: 2024-02-01T10:48:19.665 DEBUG:boto:Headers: {} 2024-02-01T10:48:19.666 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:19.666 DEBUG:boto:Port: 8000 2024-02-01T10:48:19.666 DEBUG:boto:Params: {} 2024-02-01T10:48:19.666 DEBUG:boto:Token: None 2024-02-01T10:48:19.666 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:19 GMT /vclfcn-61/obj8?versionId=null 2024-02-01T10:48:19.666 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HhUd1Dhq4ESUa2Wrpv8/PqKzlRE= 2024-02-01T10:48:19.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HhUd1Dhq4ESUa2Wrpv8/PqKzlRE=', 'Content-Length': '0'} 2024-02-01T10:48:19.669 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c4d9ee586bf947d7-0065bb76f3-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:19.670 DEBUG:boto:path=/vclfcn-61/obj8 2024-02-01T10:48:19.670 DEBUG:boto:auth_path=/vclfcn-61/obj8 2024-02-01T10:48:19.670 DEBUG:boto:path=/vclfcn-61/obj8?versionId=null 2024-02-01T10:48:19.670 DEBUG:boto:auth_path=/vclfcn-61/obj8?versionId=null 2024-02-01T10:48:19.670 DEBUG:boto:Method: GET 2024-02-01T10:48:19.670 DEBUG:boto:Path: /vclfcn-61/obj8?versionId=null 2024-02-01T10:48:19.670 DEBUG:boto:Data: 2024-02-01T10:48:19.670 DEBUG:boto:Headers: {} 2024-02-01T10:48:19.671 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:19.671 DEBUG:boto:Port: 8000 2024-02-01T10:48:19.671 DEBUG:boto:Params: {} 2024-02-01T10:48:19.671 DEBUG:boto:Token: None 2024-02-01T10:48:19.671 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:19 GMT /vclfcn-61/obj8?versionId=null 2024-02-01T10:48:19.671 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HhUd1Dhq4ESUa2Wrpv8/PqKzlRE= 2024-02-01T10:48:19.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HhUd1Dhq4ESUa2Wrpv8/PqKzlRE=', 'Content-Length': '0'} 2024-02-01T10:48:19.674 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000082de2c7484fac8f4-0065bb76f3-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:19.675 DEBUG:boto:path=/vclfcn-61/obj8 2024-02-01T10:48:19.675 DEBUG:boto:auth_path=/vclfcn-61/obj8 2024-02-01T10:48:19.675 DEBUG:boto:path=/vclfcn-61/obj8?versionId=null 2024-02-01T10:48:19.675 DEBUG:boto:auth_path=/vclfcn-61/obj8?versionId=null 2024-02-01T10:48:19.675 DEBUG:boto:Method: HEAD 2024-02-01T10:48:19.675 DEBUG:boto:Path: /vclfcn-61/obj8?versionId=null 2024-02-01T10:48:19.675 DEBUG:boto:Data: 2024-02-01T10:48:19.675 DEBUG:boto:Headers: {} 2024-02-01T10:48:19.675 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:19.676 DEBUG:boto:Port: 8000 2024-02-01T10:48:19.676 DEBUG:boto:Params: {} 2024-02-01T10:48:19.676 DEBUG:boto:Token: None 2024-02-01T10:48:19.676 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:19 GMT /vclfcn-61/obj8?versionId=null 2024-02-01T10:48:19.676 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wccKGVtj2vcwMwGZTyWBW6CrnyM= 2024-02-01T10:48:19.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wccKGVtj2vcwMwGZTyWBW6CrnyM=', 'Content-Length': '0'} 2024-02-01T10:48:19.679 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002ba5509fb60b9cab-0065bb76f3-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:19.679 DEBUG:boto:path=/vclfcn-61/obj8 2024-02-01T10:48:19.680 DEBUG:boto:auth_path=/vclfcn-61/obj8 2024-02-01T10:48:19.680 DEBUG:boto:path=/vclfcn-61/obj8?versionId=null 2024-02-01T10:48:19.680 DEBUG:boto:auth_path=/vclfcn-61/obj8?versionId=null 2024-02-01T10:48:19.680 DEBUG:boto:Method: HEAD 2024-02-01T10:48:19.680 DEBUG:boto:Path: /vclfcn-61/obj8?versionId=null 2024-02-01T10:48:19.680 DEBUG:boto:Data: 2024-02-01T10:48:19.680 DEBUG:boto:Headers: {} 2024-02-01T10:48:19.680 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:19.680 DEBUG:boto:Port: 8000 2024-02-01T10:48:19.681 DEBUG:boto:Params: {} 2024-02-01T10:48:19.681 DEBUG:boto:Token: None 2024-02-01T10:48:19.681 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:19 GMT /vclfcn-61/obj8?versionId=null 2024-02-01T10:48:19.681 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wccKGVtj2vcwMwGZTyWBW6CrnyM= 2024-02-01T10:48:19.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wccKGVtj2vcwMwGZTyWBW6CrnyM=', 'Content-Length': '0'} 2024-02-01T10:48:19.684 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000655d906aadf3ee80-0065bb76f3-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:19.684 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-02-01T10:48:19.684 DEBUG:boto:path=/vclfcn-61/obj8 2024-02-01T10:48:19.685 DEBUG:boto:auth_path=/vclfcn-61/obj8 2024-02-01T10:48:19.685 DEBUG:boto:path=/vclfcn-61/obj8?versionId=null 2024-02-01T10:48:19.685 DEBUG:boto:auth_path=/vclfcn-61/obj8?versionId=null 2024-02-01T10:48:19.685 DEBUG:boto:Method: GET 2024-02-01T10:48:19.685 DEBUG:boto:Path: /vclfcn-61/obj8?versionId=null 2024-02-01T10:48:19.685 DEBUG:boto:Data: 2024-02-01T10:48:19.685 DEBUG:boto:Headers: {} 2024-02-01T10:48:19.685 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:19.685 DEBUG:boto:Port: 8000 2024-02-01T10:48:19.685 DEBUG:boto:Params: {} 2024-02-01T10:48:19.686 DEBUG:boto:Token: None 2024-02-01T10:48:19.686 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:19 GMT /vclfcn-61/obj8?versionId=null 2024-02-01T10:48:19.686 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HhUd1Dhq4ESUa2Wrpv8/PqKzlRE= 2024-02-01T10:48:19.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HhUd1Dhq4ESUa2Wrpv8/PqKzlRE=', 'Content-Length': '0'} 2024-02-01T10:48:19.689 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000024157ebeee12c6df-0065bb76f3-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:19.732 DEBUG:boto:path=/vclfcn-61/obj8 2024-02-01T10:48:19.732 DEBUG:boto:auth_path=/vclfcn-61/obj8 2024-02-01T10:48:19.732 DEBUG:boto:path=/vclfcn-61/obj8?versionId=null 2024-02-01T10:48:19.732 DEBUG:boto:auth_path=/vclfcn-61/obj8?versionId=null 2024-02-01T10:48:19.732 DEBUG:boto:Method: GET 2024-02-01T10:48:19.733 DEBUG:boto:Path: /vclfcn-61/obj8?versionId=null 2024-02-01T10:48:19.733 DEBUG:boto:Data: 2024-02-01T10:48:19.733 DEBUG:boto:Headers: {} 2024-02-01T10:48:19.733 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:19.733 DEBUG:boto:Port: 8000 2024-02-01T10:48:19.733 DEBUG:boto:Params: {} 2024-02-01T10:48:19.735 DEBUG:boto:Token: None 2024-02-01T10:48:19.735 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:19 GMT /vclfcn-61/obj8?versionId=null 2024-02-01T10:48:19.735 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HhUd1Dhq4ESUa2Wrpv8/PqKzlRE= 2024-02-01T10:48:19.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HhUd1Dhq4ESUa2Wrpv8/PqKzlRE=', 'Content-Length': '0'} 2024-02-01T10:48:19.738 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008ce9b58353731e98-0065bb76f3-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:19.780 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:19.780 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:19.780 DEBUG:boto:path=/vclfcn-61/?prefix=obj8 2024-02-01T10:48:19.780 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj8 2024-02-01T10:48:19.780 DEBUG:boto:Method: GET 2024-02-01T10:48:19.780 DEBUG:boto:Path: /vclfcn-61/?prefix=obj8 2024-02-01T10:48:19.781 DEBUG:boto:Data: 2024-02-01T10:48:19.781 DEBUG:boto:Headers: {} 2024-02-01T10:48:19.781 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:19.781 DEBUG:boto:Port: 8000 2024-02-01T10:48:19.781 DEBUG:boto:Params: {} 2024-02-01T10:48:19.781 DEBUG:boto:Token: None 2024-02-01T10:48:19.781 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:19 GMT /vclfcn-61/ 2024-02-01T10:48:19.782 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tgDZeB7mWfYALFkrBKG3K//eUvo= 2024-02-01T10:48:19.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tgDZeB7mWfYALFkrBKG3K//eUvo=', 'Content-Length': '0'} 2024-02-01T10:48:19.786 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058ad0aa12e7718bd-0065bb76f3-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:19.832 DEBUG:boto:b'vclfcn-61obj81000falseobj82024-02-01T10:47:30.507Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:19.833 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:19.833 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:19.833 DEBUG:boto:path=/vclfcn-61/?prefix=obj8 2024-02-01T10:48:19.833 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj8 2024-02-01T10:48:19.833 DEBUG:boto:Method: GET 2024-02-01T10:48:19.833 DEBUG:boto:Path: /vclfcn-61/?prefix=obj8 2024-02-01T10:48:19.833 DEBUG:boto:Data: 2024-02-01T10:48:19.833 DEBUG:boto:Headers: {} 2024-02-01T10:48:19.833 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:19.834 DEBUG:boto:Port: 8000 2024-02-01T10:48:19.834 DEBUG:boto:Params: {} 2024-02-01T10:48:19.834 DEBUG:boto:Token: None 2024-02-01T10:48:19.834 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:19 GMT /vclfcn-61/ 2024-02-01T10:48:19.834 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tgDZeB7mWfYALFkrBKG3K//eUvo= 2024-02-01T10:48:19.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tgDZeB7mWfYALFkrBKG3K//eUvo=', 'Content-Length': '0'} 2024-02-01T10:48:19.839 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000563adec00d02028b-0065bb76f3-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:19.880 DEBUG:boto:b'vclfcn-61obj81000falseobj82024-02-01T10:47:30.507Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:19.880 DEBUG:boto:path=/vclfcn-61/obj8 2024-02-01T10:48:19.881 DEBUG:boto:auth_path=/vclfcn-61/obj8 2024-02-01T10:48:19.881 DEBUG:boto:Method: HEAD 2024-02-01T10:48:19.881 DEBUG:boto:Path: /vclfcn-61/obj8 2024-02-01T10:48:19.881 DEBUG:boto:Data: 2024-02-01T10:48:19.881 DEBUG:boto:Headers: {} 2024-02-01T10:48:19.881 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:19.881 DEBUG:boto:Port: 8000 2024-02-01T10:48:19.881 DEBUG:boto:Params: {} 2024-02-01T10:48:19.881 DEBUG:boto:Token: None 2024-02-01T10:48:19.882 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:19 GMT /vclfcn-61/obj8 2024-02-01T10:48:19.882 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:pnr/b1cOomb2k9jiWNX/LdtiJXA= 2024-02-01T10:48:19.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:pnr/b1cOomb2k9jiWNX/LdtiJXA=', 'Content-Length': '0'} 2024-02-01T10:48:19.885 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001630ae07d320beb9-0065bb76f3-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:19.885 DEBUG:boto:path=/vclfcn-61/obj8 2024-02-01T10:48:19.885 DEBUG:boto:auth_path=/vclfcn-61/obj8 2024-02-01T10:48:19.885 DEBUG:boto:Method: HEAD 2024-02-01T10:48:19.885 DEBUG:boto:Path: /vclfcn-61/obj8 2024-02-01T10:48:19.885 DEBUG:boto:Data: 2024-02-01T10:48:19.885 DEBUG:boto:Headers: {} 2024-02-01T10:48:19.886 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:19.886 DEBUG:boto:Port: 8000 2024-02-01T10:48:19.886 DEBUG:boto:Params: {} 2024-02-01T10:48:19.886 DEBUG:boto:Token: None 2024-02-01T10:48:19.886 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:19 GMT /vclfcn-61/obj8 2024-02-01T10:48:19.886 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:pnr/b1cOomb2k9jiWNX/LdtiJXA= 2024-02-01T10:48:19.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:pnr/b1cOomb2k9jiWNX/LdtiJXA=', 'Content-Length': '0'} 2024-02-01T10:48:19.889 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a2b56b12de3e1eee-0065bb76f3-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:19.889 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-02-01T10:48:19.889 DEBUG:boto:path=/vclfcn-61/obj8 2024-02-01T10:48:19.889 DEBUG:boto:auth_path=/vclfcn-61/obj8 2024-02-01T10:48:19.890 DEBUG:boto:Method: GET 2024-02-01T10:48:19.890 DEBUG:boto:Path: /vclfcn-61/obj8 2024-02-01T10:48:19.890 DEBUG:boto:Data: 2024-02-01T10:48:19.890 DEBUG:boto:Headers: {} 2024-02-01T10:48:19.890 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:19.890 DEBUG:boto:Port: 8000 2024-02-01T10:48:19.890 DEBUG:boto:Params: {} 2024-02-01T10:48:19.890 DEBUG:boto:Token: None 2024-02-01T10:48:19.890 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:19 GMT /vclfcn-61/obj8 2024-02-01T10:48:19.891 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:v5Se5zxPbzB4XHfjyX2NEUC2JMs= 2024-02-01T10:48:19.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:v5Se5zxPbzB4XHfjyX2NEUC2JMs=', 'Content-Length': '0'} 2024-02-01T10:48:19.894 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d999d8c7206a2fdc-0065bb76f3-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:19.936 DEBUG:boto:path=/vclfcn-61/obj8 2024-02-01T10:48:19.936 DEBUG:boto:auth_path=/vclfcn-61/obj8 2024-02-01T10:48:19.936 DEBUG:boto:Method: GET 2024-02-01T10:48:19.936 DEBUG:boto:Path: /vclfcn-61/obj8 2024-02-01T10:48:19.936 DEBUG:boto:Data: 2024-02-01T10:48:19.937 DEBUG:boto:Headers: {} 2024-02-01T10:48:19.937 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:19.937 DEBUG:boto:Port: 8000 2024-02-01T10:48:19.937 DEBUG:boto:Params: {} 2024-02-01T10:48:19.937 DEBUG:boto:Token: None 2024-02-01T10:48:19.937 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:19 GMT /vclfcn-61/obj8 2024-02-01T10:48:19.937 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:v5Se5zxPbzB4XHfjyX2NEUC2JMs= 2024-02-01T10:48:19.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:v5Se5zxPbzB4XHfjyX2NEUC2JMs=', 'Content-Length': '0'} 2024-02-01T10:48:19.940 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006689297b53e19ff8-0065bb76f3-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:19.984 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:19.984 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:19.984 DEBUG:boto:path=/vclfcn-61/?prefix=obj8 2024-02-01T10:48:19.984 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj8 2024-02-01T10:48:19.985 DEBUG:boto:Method: GET 2024-02-01T10:48:19.985 DEBUG:boto:Path: /vclfcn-61/?prefix=obj8 2024-02-01T10:48:19.985 DEBUG:boto:Data: 2024-02-01T10:48:19.985 DEBUG:boto:Headers: {} 2024-02-01T10:48:19.985 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:19.985 DEBUG:boto:Port: 8000 2024-02-01T10:48:19.985 DEBUG:boto:Params: {} 2024-02-01T10:48:19.985 DEBUG:boto:Token: None 2024-02-01T10:48:19.985 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:19 GMT /vclfcn-61/ 2024-02-01T10:48:19.986 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tgDZeB7mWfYALFkrBKG3K//eUvo= 2024-02-01T10:48:19.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tgDZeB7mWfYALFkrBKG3K//eUvo=', 'Content-Length': '0'} 2024-02-01T10:48:19.990 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095932427080acd3d-0065bb76f3-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:20.032 DEBUG:boto:b'vclfcn-61obj81000falseobj82024-02-01T10:47:30.507Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:20.033 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:20.033 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:20.033 DEBUG:boto:path=/vclfcn-61/?prefix=obj8 2024-02-01T10:48:20.033 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj8 2024-02-01T10:48:20.033 DEBUG:boto:Method: GET 2024-02-01T10:48:20.033 DEBUG:boto:Path: /vclfcn-61/?prefix=obj8 2024-02-01T10:48:20.033 DEBUG:boto:Data: 2024-02-01T10:48:20.033 DEBUG:boto:Headers: {} 2024-02-01T10:48:20.034 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:20.034 DEBUG:boto:Port: 8000 2024-02-01T10:48:20.034 DEBUG:boto:Params: {} 2024-02-01T10:48:20.034 DEBUG:boto:Token: None 2024-02-01T10:48:20.034 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:20 GMT /vclfcn-61/ 2024-02-01T10:48:20.034 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lr/jBd4QIloMXQ3awULOR1Oec94= 2024-02-01T10:48:20.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lr/jBd4QIloMXQ3awULOR1Oec94=', 'Content-Length': '0'} 2024-02-01T10:48:20.038 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000643b11eeb341e97e-0065bb76f4-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:20.038 DEBUG:boto:b'vclfcn-61obj81000falseobj82024-02-01T10:47:30.507Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:20.039 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-61 zones={test-zone2, test-zone1} 2024-02-01T10:48:20.039 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-61 zones={test-zone3, test-zone1} 2024-02-01T10:48:20.039 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:20.039 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:20.039 DEBUG:boto:Method: HEAD 2024-02-01T10:48:20.039 DEBUG:boto:Path: /vclfcn-61/ 2024-02-01T10:48:20.040 DEBUG:boto:Data: 2024-02-01T10:48:20.040 DEBUG:boto:Headers: {} 2024-02-01T10:48:20.040 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:20.040 DEBUG:boto:Port: 8001 2024-02-01T10:48:20.040 DEBUG:boto:Params: {} 2024-02-01T10:48:20.040 DEBUG:boto:Token: None 2024-02-01T10:48:20.040 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:20 GMT /vclfcn-61/ 2024-02-01T10:48:20.040 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:MsjIkuTKA1DAaFxqXp/vPrQCl9k= 2024-02-01T10:48:20.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:MsjIkuTKA1DAaFxqXp/vPrQCl9k=', 'Content-Length': '0'} 2024-02-01T10:48:20.044 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001c72685b44795bc7-0065bb76f4-5762-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:48:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:20.045 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:20.045 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:20.045 DEBUG:boto:Method: HEAD 2024-02-01T10:48:20.045 DEBUG:boto:Path: /vclfcn-61/ 2024-02-01T10:48:20.045 DEBUG:boto:Data: 2024-02-01T10:48:20.045 DEBUG:boto:Headers: {} 2024-02-01T10:48:20.045 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:20.045 DEBUG:boto:Port: 8000 2024-02-01T10:48:20.045 DEBUG:boto:Params: {} 2024-02-01T10:48:20.045 DEBUG:boto:Token: None 2024-02-01T10:48:20.046 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:20 GMT /vclfcn-61/ 2024-02-01T10:48:20.046 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:MsjIkuTKA1DAaFxqXp/vPrQCl9k= 2024-02-01T10:48:20.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:MsjIkuTKA1DAaFxqXp/vPrQCl9k=', 'Content-Length': '0'} 2024-02-01T10:48:20.050 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000816fdf47a99a717a-0065bb76f4-5765-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:48:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:20.050 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:48:20.050 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:20.050 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:20.051 DEBUG:boto:path=/vclfcn-61/?versions&max-keys=999 2024-02-01T10:48:20.051 DEBUG:boto:auth_path=/vclfcn-61/?versions&max-keys=999 2024-02-01T10:48:20.051 DEBUG:boto:Method: GET 2024-02-01T10:48:20.051 DEBUG:boto:Path: /vclfcn-61/?versions&max-keys=999 2024-02-01T10:48:20.051 DEBUG:boto:Data: 2024-02-01T10:48:20.051 DEBUG:boto:Headers: {} 2024-02-01T10:48:20.051 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:20.051 DEBUG:boto:Port: 8001 2024-02-01T10:48:20.051 DEBUG:boto:Params: {} 2024-02-01T10:48:20.051 DEBUG:boto:Token: None 2024-02-01T10:48:20.052 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:20 GMT /vclfcn-61/?versions 2024-02-01T10:48:20.052 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:PLo/adVbtSwXM34t6fJy0+akWsE= 2024-02-01T10:48:20.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:PLo/adVbtSwXM34t6fJy0+akWsE=', 'Content-Length': '0'} 2024-02-01T10:48:20.057 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee111db29328b36d-0065bb76f4-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:20.057 DEBUG:boto:b'vclfcn-61999falseobj1nulltrue2024-02-01T10:46:46.359Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:46:46.380Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:46:46.433Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:46:46.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-02-01T10:47:30.425Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-02-01T10:47:30.440Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-02-01T10:47:30.492Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-02-01T10:47:30.507Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:20.058 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-02-01T10:48:20.058 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-02-01T10:48:20.058 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-02-01T10:48:20.058 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-02-01T10:48:20.058 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-02-01T10:48:20.058 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-02-01T10:48:20.058 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-02-01T10:48:20.058 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-02-01T10:48:20.059 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:48:20.059 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:20.059 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:20.059 DEBUG:boto:path=/vclfcn-61/?versions&max-keys=999 2024-02-01T10:48:20.059 DEBUG:boto:auth_path=/vclfcn-61/?versions&max-keys=999 2024-02-01T10:48:20.059 DEBUG:boto:Method: GET 2024-02-01T10:48:20.059 DEBUG:boto:Path: /vclfcn-61/?versions&max-keys=999 2024-02-01T10:48:20.059 DEBUG:boto:Data: 2024-02-01T10:48:20.059 DEBUG:boto:Headers: {} 2024-02-01T10:48:20.060 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:20.060 DEBUG:boto:Port: 8000 2024-02-01T10:48:20.060 DEBUG:boto:Params: {} 2024-02-01T10:48:20.060 DEBUG:boto:Token: None 2024-02-01T10:48:20.060 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:20 GMT /vclfcn-61/?versions 2024-02-01T10:48:20.060 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:PLo/adVbtSwXM34t6fJy0+akWsE= 2024-02-01T10:48:20.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:PLo/adVbtSwXM34t6fJy0+akWsE=', 'Content-Length': '0'} 2024-02-01T10:48:20.064 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd4f419b077ebb4c-0065bb76f4-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:20.065 DEBUG:boto:b'vclfcn-61999falseobj1nulltrue2024-02-01T10:46:46.359Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:46:46.380Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:46:46.433Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:46:46.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-02-01T10:47:30.425Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-02-01T10:47:30.440Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-02-01T10:47:30.492Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-02-01T10:47:30.507Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:20.066 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-02-01T10:48:20.066 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-02-01T10:48:20.066 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-02-01T10:48:20.066 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-02-01T10:48:20.067 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-02-01T10:48:20.067 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-02-01T10:48:20.067 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-02-01T10:48:20.067 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-02-01T10:48:20.067 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:20.067 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:20.067 DEBUG:boto:path=/vclfcn-61/?versions&max-keys=999 2024-02-01T10:48:20.067 DEBUG:boto:auth_path=/vclfcn-61/?versions&max-keys=999 2024-02-01T10:48:20.067 DEBUG:boto:Method: GET 2024-02-01T10:48:20.068 DEBUG:boto:Path: /vclfcn-61/?versions&max-keys=999 2024-02-01T10:48:20.068 DEBUG:boto:Data: 2024-02-01T10:48:20.068 DEBUG:boto:Headers: {} 2024-02-01T10:48:20.068 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:20.068 DEBUG:boto:Port: 8001 2024-02-01T10:48:20.068 DEBUG:boto:Params: {} 2024-02-01T10:48:20.068 DEBUG:boto:Token: None 2024-02-01T10:48:20.068 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:20 GMT /vclfcn-61/?versions 2024-02-01T10:48:20.068 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:PLo/adVbtSwXM34t6fJy0+akWsE= 2024-02-01T10:48:20.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:PLo/adVbtSwXM34t6fJy0+akWsE=', 'Content-Length': '0'} 2024-02-01T10:48:20.073 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000559296883f68bb54-0065bb76f4-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:20.073 DEBUG:boto:b'vclfcn-61999falseobj1nulltrue2024-02-01T10:46:46.359Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:46:46.380Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:46:46.433Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:46:46.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-02-01T10:47:30.425Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-02-01T10:47:30.440Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-02-01T10:47:30.492Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-02-01T10:47:30.507Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:20.074 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:20.074 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:20.074 DEBUG:boto:path=/vclfcn-61/?versions&max-keys=999 2024-02-01T10:48:20.074 DEBUG:boto:auth_path=/vclfcn-61/?versions&max-keys=999 2024-02-01T10:48:20.074 DEBUG:boto:Method: GET 2024-02-01T10:48:20.074 DEBUG:boto:Path: /vclfcn-61/?versions&max-keys=999 2024-02-01T10:48:20.074 DEBUG:boto:Data: 2024-02-01T10:48:20.075 DEBUG:boto:Headers: {} 2024-02-01T10:48:20.075 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:20.075 DEBUG:boto:Port: 8000 2024-02-01T10:48:20.075 DEBUG:boto:Params: {} 2024-02-01T10:48:20.075 DEBUG:boto:Token: None 2024-02-01T10:48:20.075 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:20 GMT /vclfcn-61/?versions 2024-02-01T10:48:20.075 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:PLo/adVbtSwXM34t6fJy0+akWsE= 2024-02-01T10:48:20.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:PLo/adVbtSwXM34t6fJy0+akWsE=', 'Content-Length': '0'} 2024-02-01T10:48:20.080 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003661d1ed46b3f23b-0065bb76f4-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:20.081 DEBUG:boto:b'vclfcn-61999falseobj1nulltrue2024-02-01T10:46:46.359Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:46:46.380Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:46:46.433Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:46:46.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-02-01T10:47:30.425Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-02-01T10:47:30.440Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-02-01T10:47:30.492Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-02-01T10:47:30.507Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:20.081 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:48:20.082 DEBUG:boto:path=/vclfcn-61/obj1 2024-02-01T10:48:20.082 DEBUG:boto:auth_path=/vclfcn-61/obj1 2024-02-01T10:48:20.082 DEBUG:boto:path=/vclfcn-61/obj1?versionId=null 2024-02-01T10:48:20.082 DEBUG:boto:auth_path=/vclfcn-61/obj1?versionId=null 2024-02-01T10:48:20.082 DEBUG:boto:Method: GET 2024-02-01T10:48:20.082 DEBUG:boto:Path: /vclfcn-61/obj1?versionId=null 2024-02-01T10:48:20.082 DEBUG:boto:Data: 2024-02-01T10:48:20.082 DEBUG:boto:Headers: {} 2024-02-01T10:48:20.082 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:20.083 DEBUG:boto:Port: 8001 2024-02-01T10:48:20.083 DEBUG:boto:Params: {} 2024-02-01T10:48:20.083 DEBUG:boto:Token: None 2024-02-01T10:48:20.083 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:20 GMT /vclfcn-61/obj1?versionId=null 2024-02-01T10:48:20.083 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7y2iaChn/0RwtGEW2G7Z8e4eq6M= 2024-02-01T10:48:20.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7y2iaChn/0RwtGEW2G7Z8e4eq6M=', 'Content-Length': '0'} 2024-02-01T10:48:20.086 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001e409844cc97b3a0-0065bb76f4-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:20.128 DEBUG:boto:path=/vclfcn-61/obj1 2024-02-01T10:48:20.128 DEBUG:boto:auth_path=/vclfcn-61/obj1 2024-02-01T10:48:20.128 DEBUG:boto:path=/vclfcn-61/obj1?versionId=null 2024-02-01T10:48:20.129 DEBUG:boto:auth_path=/vclfcn-61/obj1?versionId=null 2024-02-01T10:48:20.129 DEBUG:boto:Method: GET 2024-02-01T10:48:20.129 DEBUG:boto:Path: /vclfcn-61/obj1?versionId=null 2024-02-01T10:48:20.129 DEBUG:boto:Data: 2024-02-01T10:48:20.129 DEBUG:boto:Headers: {} 2024-02-01T10:48:20.129 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:20.129 DEBUG:boto:Port: 8000 2024-02-01T10:48:20.129 DEBUG:boto:Params: {} 2024-02-01T10:48:20.129 DEBUG:boto:Token: None 2024-02-01T10:48:20.129 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:20 GMT /vclfcn-61/obj1?versionId=null 2024-02-01T10:48:20.130 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7y2iaChn/0RwtGEW2G7Z8e4eq6M= 2024-02-01T10:48:20.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7y2iaChn/0RwtGEW2G7Z8e4eq6M=', 'Content-Length': '0'} 2024-02-01T10:48:20.133 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000024ef1e5c09a883c6-0065bb76f4-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:20.133 DEBUG:boto:path=/vclfcn-61/obj1 2024-02-01T10:48:20.133 DEBUG:boto:auth_path=/vclfcn-61/obj1 2024-02-01T10:48:20.133 DEBUG:boto:path=/vclfcn-61/obj1?versionId=null 2024-02-01T10:48:20.133 DEBUG:boto:auth_path=/vclfcn-61/obj1?versionId=null 2024-02-01T10:48:20.133 DEBUG:boto:Method: HEAD 2024-02-01T10:48:20.133 DEBUG:boto:Path: /vclfcn-61/obj1?versionId=null 2024-02-01T10:48:20.134 DEBUG:boto:Data: 2024-02-01T10:48:20.134 DEBUG:boto:Headers: {} 2024-02-01T10:48:20.134 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:20.134 DEBUG:boto:Port: 8001 2024-02-01T10:48:20.134 DEBUG:boto:Params: {} 2024-02-01T10:48:20.134 DEBUG:boto:Token: None 2024-02-01T10:48:20.134 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:20 GMT /vclfcn-61/obj1?versionId=null 2024-02-01T10:48:20.134 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1bhP3LE3ERELisCbXERR/9J9M/8= 2024-02-01T10:48:20.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1bhP3LE3ERELisCbXERR/9J9M/8=', 'Content-Length': '0'} 2024-02-01T10:48:20.137 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003ea69ee45f799f1c-0065bb76f4-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:20.137 DEBUG:boto:path=/vclfcn-61/obj1 2024-02-01T10:48:20.137 DEBUG:boto:auth_path=/vclfcn-61/obj1 2024-02-01T10:48:20.138 DEBUG:boto:path=/vclfcn-61/obj1?versionId=null 2024-02-01T10:48:20.138 DEBUG:boto:auth_path=/vclfcn-61/obj1?versionId=null 2024-02-01T10:48:20.138 DEBUG:boto:Method: HEAD 2024-02-01T10:48:20.138 DEBUG:boto:Path: /vclfcn-61/obj1?versionId=null 2024-02-01T10:48:20.138 DEBUG:boto:Data: 2024-02-01T10:48:20.138 DEBUG:boto:Headers: {} 2024-02-01T10:48:20.138 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:20.138 DEBUG:boto:Port: 8000 2024-02-01T10:48:20.138 DEBUG:boto:Params: {} 2024-02-01T10:48:20.138 DEBUG:boto:Token: None 2024-02-01T10:48:20.139 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:20 GMT /vclfcn-61/obj1?versionId=null 2024-02-01T10:48:20.139 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1bhP3LE3ERELisCbXERR/9J9M/8= 2024-02-01T10:48:20.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1bhP3LE3ERELisCbXERR/9J9M/8=', 'Content-Length': '0'} 2024-02-01T10:48:20.143 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002e0c808cb565c0a4-0065bb76f4-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:20.144 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:48:20.144 DEBUG:boto:path=/vclfcn-61/obj1 2024-02-01T10:48:20.144 DEBUG:boto:auth_path=/vclfcn-61/obj1 2024-02-01T10:48:20.144 DEBUG:boto:path=/vclfcn-61/obj1?versionId=null 2024-02-01T10:48:20.144 DEBUG:boto:auth_path=/vclfcn-61/obj1?versionId=null 2024-02-01T10:48:20.144 DEBUG:boto:Method: GET 2024-02-01T10:48:20.144 DEBUG:boto:Path: /vclfcn-61/obj1?versionId=null 2024-02-01T10:48:20.144 DEBUG:boto:Data: 2024-02-01T10:48:20.145 DEBUG:boto:Headers: {} 2024-02-01T10:48:20.145 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:20.145 DEBUG:boto:Port: 8001 2024-02-01T10:48:20.145 DEBUG:boto:Params: {} 2024-02-01T10:48:20.145 DEBUG:boto:Token: None 2024-02-01T10:48:20.145 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:20 GMT /vclfcn-61/obj1?versionId=null 2024-02-01T10:48:20.145 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7y2iaChn/0RwtGEW2G7Z8e4eq6M= 2024-02-01T10:48:20.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7y2iaChn/0RwtGEW2G7Z8e4eq6M=', 'Content-Length': '0'} 2024-02-01T10:48:20.148 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000185fb86e070bf472-0065bb76f4-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:20.149 DEBUG:boto:path=/vclfcn-61/obj1 2024-02-01T10:48:20.149 DEBUG:boto:auth_path=/vclfcn-61/obj1 2024-02-01T10:48:20.149 DEBUG:boto:path=/vclfcn-61/obj1?versionId=null 2024-02-01T10:48:20.149 DEBUG:boto:auth_path=/vclfcn-61/obj1?versionId=null 2024-02-01T10:48:20.149 DEBUG:boto:Method: GET 2024-02-01T10:48:20.149 DEBUG:boto:Path: /vclfcn-61/obj1?versionId=null 2024-02-01T10:48:20.149 DEBUG:boto:Data: 2024-02-01T10:48:20.149 DEBUG:boto:Headers: {} 2024-02-01T10:48:20.149 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:20.150 DEBUG:boto:Port: 8000 2024-02-01T10:48:20.150 DEBUG:boto:Params: {} 2024-02-01T10:48:20.150 DEBUG:boto:Token: None 2024-02-01T10:48:20.150 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:20 GMT /vclfcn-61/obj1?versionId=null 2024-02-01T10:48:20.150 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7y2iaChn/0RwtGEW2G7Z8e4eq6M= 2024-02-01T10:48:20.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7y2iaChn/0RwtGEW2G7Z8e4eq6M=', 'Content-Length': '0'} 2024-02-01T10:48:20.153 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c980aca0978b6712-0065bb76f4-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:20.196 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:20.196 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:20.196 DEBUG:boto:path=/vclfcn-61/?prefix=obj1 2024-02-01T10:48:20.196 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj1 2024-02-01T10:48:20.196 DEBUG:boto:Method: GET 2024-02-01T10:48:20.197 DEBUG:boto:Path: /vclfcn-61/?prefix=obj1 2024-02-01T10:48:20.197 DEBUG:boto:Data: 2024-02-01T10:48:20.197 DEBUG:boto:Headers: {} 2024-02-01T10:48:20.197 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:20.197 DEBUG:boto:Port: 8001 2024-02-01T10:48:20.197 DEBUG:boto:Params: {} 2024-02-01T10:48:20.197 DEBUG:boto:Token: None 2024-02-01T10:48:20.197 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:20 GMT /vclfcn-61/ 2024-02-01T10:48:20.198 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lr/jBd4QIloMXQ3awULOR1Oec94= 2024-02-01T10:48:20.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lr/jBd4QIloMXQ3awULOR1Oec94=', 'Content-Length': '0'} 2024-02-01T10:48:20.202 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d9cc598a8708461-0065bb76f4-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:20.202 DEBUG:boto:b'vclfcn-61obj11000falseobj12024-02-01T10:46:46.359Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:20.203 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:20.203 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:20.203 DEBUG:boto:path=/vclfcn-61/?prefix=obj1 2024-02-01T10:48:20.203 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj1 2024-02-01T10:48:20.203 DEBUG:boto:Method: GET 2024-02-01T10:48:20.203 DEBUG:boto:Path: /vclfcn-61/?prefix=obj1 2024-02-01T10:48:20.203 DEBUG:boto:Data: 2024-02-01T10:48:20.203 DEBUG:boto:Headers: {} 2024-02-01T10:48:20.204 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:20.204 DEBUG:boto:Port: 8000 2024-02-01T10:48:20.204 DEBUG:boto:Params: {} 2024-02-01T10:48:20.204 DEBUG:boto:Token: None 2024-02-01T10:48:20.204 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:20 GMT /vclfcn-61/ 2024-02-01T10:48:20.204 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lr/jBd4QIloMXQ3awULOR1Oec94= 2024-02-01T10:48:20.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lr/jBd4QIloMXQ3awULOR1Oec94=', 'Content-Length': '0'} 2024-02-01T10:48:20.209 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c3e0504667f1f6b-0065bb76f4-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:20.252 DEBUG:boto:b'vclfcn-61obj11000falseobj12024-02-01T10:46:46.359Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:20.252 DEBUG:boto:path=/vclfcn-61/obj1 2024-02-01T10:48:20.253 DEBUG:boto:auth_path=/vclfcn-61/obj1 2024-02-01T10:48:20.253 DEBUG:boto:Method: HEAD 2024-02-01T10:48:20.253 DEBUG:boto:Path: /vclfcn-61/obj1 2024-02-01T10:48:20.253 DEBUG:boto:Data: 2024-02-01T10:48:20.253 DEBUG:boto:Headers: {} 2024-02-01T10:48:20.253 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:20.253 DEBUG:boto:Port: 8001 2024-02-01T10:48:20.253 DEBUG:boto:Params: {} 2024-02-01T10:48:20.253 DEBUG:boto:Token: None 2024-02-01T10:48:20.254 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:20 GMT /vclfcn-61/obj1 2024-02-01T10:48:20.254 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GQGx2ixheD1a7RI99dQGw/hI6qw= 2024-02-01T10:48:20.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GQGx2ixheD1a7RI99dQGw/hI6qw=', 'Content-Length': '0'} 2024-02-01T10:48:20.257 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d905cbb9b2f080dd-0065bb76f4-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:20.257 DEBUG:boto:path=/vclfcn-61/obj1 2024-02-01T10:48:20.257 DEBUG:boto:auth_path=/vclfcn-61/obj1 2024-02-01T10:48:20.257 DEBUG:boto:Method: HEAD 2024-02-01T10:48:20.257 DEBUG:boto:Path: /vclfcn-61/obj1 2024-02-01T10:48:20.257 DEBUG:boto:Data: 2024-02-01T10:48:20.258 DEBUG:boto:Headers: {} 2024-02-01T10:48:20.258 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:20.258 DEBUG:boto:Port: 8000 2024-02-01T10:48:20.258 DEBUG:boto:Params: {} 2024-02-01T10:48:20.258 DEBUG:boto:Token: None 2024-02-01T10:48:20.258 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:20 GMT /vclfcn-61/obj1 2024-02-01T10:48:20.258 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GQGx2ixheD1a7RI99dQGw/hI6qw= 2024-02-01T10:48:20.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GQGx2ixheD1a7RI99dQGw/hI6qw=', 'Content-Length': '0'} 2024-02-01T10:48:20.261 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001d1ba673902bd90a-0065bb76f4-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:20.261 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:48:20.261 DEBUG:boto:path=/vclfcn-61/obj1 2024-02-01T10:48:20.261 DEBUG:boto:auth_path=/vclfcn-61/obj1 2024-02-01T10:48:20.262 DEBUG:boto:Method: GET 2024-02-01T10:48:20.262 DEBUG:boto:Path: /vclfcn-61/obj1 2024-02-01T10:48:20.262 DEBUG:boto:Data: 2024-02-01T10:48:20.262 DEBUG:boto:Headers: {} 2024-02-01T10:48:20.262 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:20.262 DEBUG:boto:Port: 8001 2024-02-01T10:48:20.262 DEBUG:boto:Params: {} 2024-02-01T10:48:20.262 DEBUG:boto:Token: None 2024-02-01T10:48:20.262 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:20 GMT /vclfcn-61/obj1 2024-02-01T10:48:20.263 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tgr8dmj9bKc4pGu1hsrzTw0d43Q= 2024-02-01T10:48:20.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tgr8dmj9bKc4pGu1hsrzTw0d43Q=', 'Content-Length': '0'} 2024-02-01T10:48:20.265 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b6387cf09445f71e-0065bb76f4-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:20.308 DEBUG:boto:path=/vclfcn-61/obj1 2024-02-01T10:48:20.308 DEBUG:boto:auth_path=/vclfcn-61/obj1 2024-02-01T10:48:20.308 DEBUG:boto:Method: GET 2024-02-01T10:48:20.308 DEBUG:boto:Path: /vclfcn-61/obj1 2024-02-01T10:48:20.309 DEBUG:boto:Data: 2024-02-01T10:48:20.309 DEBUG:boto:Headers: {} 2024-02-01T10:48:20.309 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:20.309 DEBUG:boto:Port: 8000 2024-02-01T10:48:20.309 DEBUG:boto:Params: {} 2024-02-01T10:48:20.309 DEBUG:boto:Token: None 2024-02-01T10:48:20.309 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:20 GMT /vclfcn-61/obj1 2024-02-01T10:48:20.309 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tgr8dmj9bKc4pGu1hsrzTw0d43Q= 2024-02-01T10:48:20.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tgr8dmj9bKc4pGu1hsrzTw0d43Q=', 'Content-Length': '0'} 2024-02-01T10:48:20.312 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ca62ba3285d50a2e-0065bb76f4-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:20.356 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:20.356 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:20.356 DEBUG:boto:path=/vclfcn-61/?prefix=obj1 2024-02-01T10:48:20.356 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj1 2024-02-01T10:48:20.356 DEBUG:boto:Method: GET 2024-02-01T10:48:20.357 DEBUG:boto:Path: /vclfcn-61/?prefix=obj1 2024-02-01T10:48:20.357 DEBUG:boto:Data: 2024-02-01T10:48:20.357 DEBUG:boto:Headers: {} 2024-02-01T10:48:20.357 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:20.357 DEBUG:boto:Port: 8001 2024-02-01T10:48:20.357 DEBUG:boto:Params: {} 2024-02-01T10:48:20.357 DEBUG:boto:Token: None 2024-02-01T10:48:20.358 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:20 GMT /vclfcn-61/ 2024-02-01T10:48:20.358 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lr/jBd4QIloMXQ3awULOR1Oec94= 2024-02-01T10:48:20.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lr/jBd4QIloMXQ3awULOR1Oec94=', 'Content-Length': '0'} 2024-02-01T10:48:20.362 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b7c35eb417fd8e3-0065bb76f4-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:20.363 DEBUG:boto:b'vclfcn-61obj11000falseobj12024-02-01T10:46:46.359Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:20.363 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:20.363 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:20.363 DEBUG:boto:path=/vclfcn-61/?prefix=obj1 2024-02-01T10:48:20.363 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj1 2024-02-01T10:48:20.363 DEBUG:boto:Method: GET 2024-02-01T10:48:20.364 DEBUG:boto:Path: /vclfcn-61/?prefix=obj1 2024-02-01T10:48:20.364 DEBUG:boto:Data: 2024-02-01T10:48:20.364 DEBUG:boto:Headers: {} 2024-02-01T10:48:20.364 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:20.364 DEBUG:boto:Port: 8000 2024-02-01T10:48:20.364 DEBUG:boto:Params: {} 2024-02-01T10:48:20.364 DEBUG:boto:Token: None 2024-02-01T10:48:20.364 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:20 GMT /vclfcn-61/ 2024-02-01T10:48:20.364 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lr/jBd4QIloMXQ3awULOR1Oec94= 2024-02-01T10:48:20.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lr/jBd4QIloMXQ3awULOR1Oec94=', 'Content-Length': '0'} 2024-02-01T10:48:20.369 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f881ca66c01b07d0-0065bb76f4-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:20.369 DEBUG:boto:b'vclfcn-61obj11000falseobj12024-02-01T10:46:46.359Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:20.369 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:48:20.369 DEBUG:boto:path=/vclfcn-61/obj2 2024-02-01T10:48:20.369 DEBUG:boto:auth_path=/vclfcn-61/obj2 2024-02-01T10:48:20.369 DEBUG:boto:path=/vclfcn-61/obj2?versionId=null 2024-02-01T10:48:20.370 DEBUG:boto:auth_path=/vclfcn-61/obj2?versionId=null 2024-02-01T10:48:20.370 DEBUG:boto:Method: GET 2024-02-01T10:48:20.370 DEBUG:boto:Path: /vclfcn-61/obj2?versionId=null 2024-02-01T10:48:20.370 DEBUG:boto:Data: 2024-02-01T10:48:20.370 DEBUG:boto:Headers: {} 2024-02-01T10:48:20.370 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:20.370 DEBUG:boto:Port: 8001 2024-02-01T10:48:20.370 DEBUG:boto:Params: {} 2024-02-01T10:48:20.370 DEBUG:boto:Token: None 2024-02-01T10:48:20.371 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:20 GMT /vclfcn-61/obj2?versionId=null 2024-02-01T10:48:20.371 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Fs+ak5wnpR/O4LieQDTPDCF/pIY= 2024-02-01T10:48:20.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Fs+ak5wnpR/O4LieQDTPDCF/pIY=', 'Content-Length': '0'} 2024-02-01T10:48:20.374 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000031491315a3e43a90-0065bb76f4-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:20.375 DEBUG:boto:path=/vclfcn-61/obj2 2024-02-01T10:48:20.375 DEBUG:boto:auth_path=/vclfcn-61/obj2 2024-02-01T10:48:20.375 DEBUG:boto:path=/vclfcn-61/obj2?versionId=null 2024-02-01T10:48:20.375 DEBUG:boto:auth_path=/vclfcn-61/obj2?versionId=null 2024-02-01T10:48:20.375 DEBUG:boto:Method: GET 2024-02-01T10:48:20.375 DEBUG:boto:Path: /vclfcn-61/obj2?versionId=null 2024-02-01T10:48:20.375 DEBUG:boto:Data: 2024-02-01T10:48:20.375 DEBUG:boto:Headers: {} 2024-02-01T10:48:20.375 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:20.375 DEBUG:boto:Port: 8000 2024-02-01T10:48:20.375 DEBUG:boto:Params: {} 2024-02-01T10:48:20.376 DEBUG:boto:Token: None 2024-02-01T10:48:20.376 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:20 GMT /vclfcn-61/obj2?versionId=null 2024-02-01T10:48:20.376 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Fs+ak5wnpR/O4LieQDTPDCF/pIY= 2024-02-01T10:48:20.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Fs+ak5wnpR/O4LieQDTPDCF/pIY=', 'Content-Length': '0'} 2024-02-01T10:48:20.379 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001f372e00427def50-0065bb76f4-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:20.420 DEBUG:boto:path=/vclfcn-61/obj2 2024-02-01T10:48:20.420 DEBUG:boto:auth_path=/vclfcn-61/obj2 2024-02-01T10:48:20.420 DEBUG:boto:path=/vclfcn-61/obj2?versionId=null 2024-02-01T10:48:20.421 DEBUG:boto:auth_path=/vclfcn-61/obj2?versionId=null 2024-02-01T10:48:20.421 DEBUG:boto:Method: HEAD 2024-02-01T10:48:20.421 DEBUG:boto:Path: /vclfcn-61/obj2?versionId=null 2024-02-01T10:48:20.421 DEBUG:boto:Data: 2024-02-01T10:48:20.421 DEBUG:boto:Headers: {} 2024-02-01T10:48:20.421 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:20.421 DEBUG:boto:Port: 8001 2024-02-01T10:48:20.421 DEBUG:boto:Params: {} 2024-02-01T10:48:20.421 DEBUG:boto:Token: None 2024-02-01T10:48:20.422 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:20 GMT /vclfcn-61/obj2?versionId=null 2024-02-01T10:48:20.422 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5+Q0KPRontu+GIcEbTraWz/lG98= 2024-02-01T10:48:20.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5+Q0KPRontu+GIcEbTraWz/lG98=', 'Content-Length': '0'} 2024-02-01T10:48:20.425 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cc6abf5ab6ca01d3-0065bb76f4-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:20.425 DEBUG:boto:path=/vclfcn-61/obj2 2024-02-01T10:48:20.425 DEBUG:boto:auth_path=/vclfcn-61/obj2 2024-02-01T10:48:20.425 DEBUG:boto:path=/vclfcn-61/obj2?versionId=null 2024-02-01T10:48:20.425 DEBUG:boto:auth_path=/vclfcn-61/obj2?versionId=null 2024-02-01T10:48:20.425 DEBUG:boto:Method: HEAD 2024-02-01T10:48:20.425 DEBUG:boto:Path: /vclfcn-61/obj2?versionId=null 2024-02-01T10:48:20.426 DEBUG:boto:Data: 2024-02-01T10:48:20.426 DEBUG:boto:Headers: {} 2024-02-01T10:48:20.426 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:20.426 DEBUG:boto:Port: 8000 2024-02-01T10:48:20.426 DEBUG:boto:Params: {} 2024-02-01T10:48:20.426 DEBUG:boto:Token: None 2024-02-01T10:48:20.426 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:20 GMT /vclfcn-61/obj2?versionId=null 2024-02-01T10:48:20.426 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5+Q0KPRontu+GIcEbTraWz/lG98= 2024-02-01T10:48:20.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5+Q0KPRontu+GIcEbTraWz/lG98=', 'Content-Length': '0'} 2024-02-01T10:48:20.429 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f3148b224726aaf2-0065bb76f4-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:20.429 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:48:20.429 DEBUG:boto:path=/vclfcn-61/obj2 2024-02-01T10:48:20.430 DEBUG:boto:auth_path=/vclfcn-61/obj2 2024-02-01T10:48:20.430 DEBUG:boto:path=/vclfcn-61/obj2?versionId=null 2024-02-01T10:48:20.430 DEBUG:boto:auth_path=/vclfcn-61/obj2?versionId=null 2024-02-01T10:48:20.431 DEBUG:boto:Method: GET 2024-02-01T10:48:20.431 DEBUG:boto:Path: /vclfcn-61/obj2?versionId=null 2024-02-01T10:48:20.431 DEBUG:boto:Data: 2024-02-01T10:48:20.431 DEBUG:boto:Headers: {} 2024-02-01T10:48:20.431 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:20.431 DEBUG:boto:Port: 8001 2024-02-01T10:48:20.431 DEBUG:boto:Params: {} 2024-02-01T10:48:20.431 DEBUG:boto:Token: None 2024-02-01T10:48:20.431 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:20 GMT /vclfcn-61/obj2?versionId=null 2024-02-01T10:48:20.431 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Fs+ak5wnpR/O4LieQDTPDCF/pIY= 2024-02-01T10:48:20.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Fs+ak5wnpR/O4LieQDTPDCF/pIY=', 'Content-Length': '0'} 2024-02-01T10:48:20.435 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005121402df775711c-0065bb76f4-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:20.435 DEBUG:boto:path=/vclfcn-61/obj2 2024-02-01T10:48:20.435 DEBUG:boto:auth_path=/vclfcn-61/obj2 2024-02-01T10:48:20.435 DEBUG:boto:path=/vclfcn-61/obj2?versionId=null 2024-02-01T10:48:20.436 DEBUG:boto:auth_path=/vclfcn-61/obj2?versionId=null 2024-02-01T10:48:20.436 DEBUG:boto:Method: GET 2024-02-01T10:48:20.436 DEBUG:boto:Path: /vclfcn-61/obj2?versionId=null 2024-02-01T10:48:20.436 DEBUG:boto:Data: 2024-02-01T10:48:20.436 DEBUG:boto:Headers: {} 2024-02-01T10:48:20.436 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:20.436 DEBUG:boto:Port: 8000 2024-02-01T10:48:20.436 DEBUG:boto:Params: {} 2024-02-01T10:48:20.436 DEBUG:boto:Token: None 2024-02-01T10:48:20.437 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:20 GMT /vclfcn-61/obj2?versionId=null 2024-02-01T10:48:20.437 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Fs+ak5wnpR/O4LieQDTPDCF/pIY= 2024-02-01T10:48:20.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Fs+ak5wnpR/O4LieQDTPDCF/pIY=', 'Content-Length': '0'} 2024-02-01T10:48:20.439 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000af8ac15cbf10fb7c-0065bb76f4-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:20.484 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:20.484 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:20.484 DEBUG:boto:path=/vclfcn-61/?prefix=obj2 2024-02-01T10:48:20.484 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj2 2024-02-01T10:48:20.485 DEBUG:boto:Method: GET 2024-02-01T10:48:20.485 DEBUG:boto:Path: /vclfcn-61/?prefix=obj2 2024-02-01T10:48:20.485 DEBUG:boto:Data: 2024-02-01T10:48:20.485 DEBUG:boto:Headers: {} 2024-02-01T10:48:20.485 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:20.485 DEBUG:boto:Port: 8001 2024-02-01T10:48:20.485 DEBUG:boto:Params: {} 2024-02-01T10:48:20.485 DEBUG:boto:Token: None 2024-02-01T10:48:20.485 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:20 GMT /vclfcn-61/ 2024-02-01T10:48:20.486 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lr/jBd4QIloMXQ3awULOR1Oec94= 2024-02-01T10:48:20.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lr/jBd4QIloMXQ3awULOR1Oec94=', 'Content-Length': '0'} 2024-02-01T10:48:20.490 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd2cc93d99efb528-0065bb76f4-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:20.490 DEBUG:boto:b'vclfcn-61obj21000falseobj22024-02-01T10:46:46.380Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:20.491 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:20.491 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:20.491 DEBUG:boto:path=/vclfcn-61/?prefix=obj2 2024-02-01T10:48:20.492 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj2 2024-02-01T10:48:20.492 DEBUG:boto:Method: GET 2024-02-01T10:48:20.492 DEBUG:boto:Path: /vclfcn-61/?prefix=obj2 2024-02-01T10:48:20.492 DEBUG:boto:Data: 2024-02-01T10:48:20.492 DEBUG:boto:Headers: {} 2024-02-01T10:48:20.492 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:20.492 DEBUG:boto:Port: 8000 2024-02-01T10:48:20.492 DEBUG:boto:Params: {} 2024-02-01T10:48:20.492 DEBUG:boto:Token: None 2024-02-01T10:48:20.493 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:20 GMT /vclfcn-61/ 2024-02-01T10:48:20.493 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lr/jBd4QIloMXQ3awULOR1Oec94= 2024-02-01T10:48:20.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lr/jBd4QIloMXQ3awULOR1Oec94=', 'Content-Length': '0'} 2024-02-01T10:48:20.497 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000119cee786564ec0c-0065bb76f4-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:20.540 DEBUG:boto:b'vclfcn-61obj21000falseobj22024-02-01T10:46:46.380Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:20.540 DEBUG:boto:path=/vclfcn-61/obj2 2024-02-01T10:48:20.541 DEBUG:boto:auth_path=/vclfcn-61/obj2 2024-02-01T10:48:20.541 DEBUG:boto:Method: HEAD 2024-02-01T10:48:20.541 DEBUG:boto:Path: /vclfcn-61/obj2 2024-02-01T10:48:20.541 DEBUG:boto:Data: 2024-02-01T10:48:20.541 DEBUG:boto:Headers: {} 2024-02-01T10:48:20.541 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:20.541 DEBUG:boto:Port: 8001 2024-02-01T10:48:20.541 DEBUG:boto:Params: {} 2024-02-01T10:48:20.541 DEBUG:boto:Token: None 2024-02-01T10:48:20.541 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:20 GMT /vclfcn-61/obj2 2024-02-01T10:48:20.542 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ap82+xcwSWXOgtZErDv/ZkhQMIo= 2024-02-01T10:48:20.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ap82+xcwSWXOgtZErDv/ZkhQMIo=', 'Content-Length': '0'} 2024-02-01T10:48:20.545 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ad42168ee6a8713e-0065bb76f4-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:20.545 DEBUG:boto:path=/vclfcn-61/obj2 2024-02-01T10:48:20.545 DEBUG:boto:auth_path=/vclfcn-61/obj2 2024-02-01T10:48:20.545 DEBUG:boto:Method: HEAD 2024-02-01T10:48:20.546 DEBUG:boto:Path: /vclfcn-61/obj2 2024-02-01T10:48:20.546 DEBUG:boto:Data: 2024-02-01T10:48:20.546 DEBUG:boto:Headers: {} 2024-02-01T10:48:20.546 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:20.546 DEBUG:boto:Port: 8000 2024-02-01T10:48:20.546 DEBUG:boto:Params: {} 2024-02-01T10:48:20.546 DEBUG:boto:Token: None 2024-02-01T10:48:20.546 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:20 GMT /vclfcn-61/obj2 2024-02-01T10:48:20.546 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ap82+xcwSWXOgtZErDv/ZkhQMIo= 2024-02-01T10:48:20.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ap82+xcwSWXOgtZErDv/ZkhQMIo=', 'Content-Length': '0'} 2024-02-01T10:48:20.552 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000042db67d5198f4dbe-0065bb76f4-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:20.552 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:48:20.552 DEBUG:boto:path=/vclfcn-61/obj2 2024-02-01T10:48:20.552 DEBUG:boto:auth_path=/vclfcn-61/obj2 2024-02-01T10:48:20.552 DEBUG:boto:Method: GET 2024-02-01T10:48:20.553 DEBUG:boto:Path: /vclfcn-61/obj2 2024-02-01T10:48:20.553 DEBUG:boto:Data: 2024-02-01T10:48:20.553 DEBUG:boto:Headers: {} 2024-02-01T10:48:20.553 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:20.553 DEBUG:boto:Port: 8001 2024-02-01T10:48:20.553 DEBUG:boto:Params: {} 2024-02-01T10:48:20.553 DEBUG:boto:Token: None 2024-02-01T10:48:20.553 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:20 GMT /vclfcn-61/obj2 2024-02-01T10:48:20.553 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:dwIpHZohefC/xmmYciwXrzvl0Cg= 2024-02-01T10:48:20.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dwIpHZohefC/xmmYciwXrzvl0Cg=', 'Content-Length': '0'} 2024-02-01T10:48:20.556 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e96216eda819eaad-0065bb76f4-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:20.600 DEBUG:boto:path=/vclfcn-61/obj2 2024-02-01T10:48:20.600 DEBUG:boto:auth_path=/vclfcn-61/obj2 2024-02-01T10:48:20.601 DEBUG:boto:Method: GET 2024-02-01T10:48:20.601 DEBUG:boto:Path: /vclfcn-61/obj2 2024-02-01T10:48:20.601 DEBUG:boto:Data: 2024-02-01T10:48:20.601 DEBUG:boto:Headers: {} 2024-02-01T10:48:20.601 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:20.601 DEBUG:boto:Port: 8000 2024-02-01T10:48:20.601 DEBUG:boto:Params: {} 2024-02-01T10:48:20.601 DEBUG:boto:Token: None 2024-02-01T10:48:20.601 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:20 GMT /vclfcn-61/obj2 2024-02-01T10:48:20.601 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:dwIpHZohefC/xmmYciwXrzvl0Cg= 2024-02-01T10:48:20.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dwIpHZohefC/xmmYciwXrzvl0Cg=', 'Content-Length': '0'} 2024-02-01T10:48:20.605 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ec20284335e9b0b4-0065bb76f4-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:20.653 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:20.653 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:20.653 DEBUG:boto:path=/vclfcn-61/?prefix=obj2 2024-02-01T10:48:20.653 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj2 2024-02-01T10:48:20.653 DEBUG:boto:Method: GET 2024-02-01T10:48:20.653 DEBUG:boto:Path: /vclfcn-61/?prefix=obj2 2024-02-01T10:48:20.653 DEBUG:boto:Data: 2024-02-01T10:48:20.653 DEBUG:boto:Headers: {} 2024-02-01T10:48:20.654 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:20.654 DEBUG:boto:Port: 8001 2024-02-01T10:48:20.654 DEBUG:boto:Params: {} 2024-02-01T10:48:20.654 DEBUG:boto:Token: None 2024-02-01T10:48:20.654 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:20 GMT /vclfcn-61/ 2024-02-01T10:48:20.654 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lr/jBd4QIloMXQ3awULOR1Oec94= 2024-02-01T10:48:20.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lr/jBd4QIloMXQ3awULOR1Oec94=', 'Content-Length': '0'} 2024-02-01T10:48:20.659 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000964ec37240ca7c51-0065bb76f4-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:20.659 DEBUG:boto:b'vclfcn-61obj21000falseobj22024-02-01T10:46:46.380Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:20.660 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:20.660 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:20.660 DEBUG:boto:path=/vclfcn-61/?prefix=obj2 2024-02-01T10:48:20.660 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj2 2024-02-01T10:48:20.660 DEBUG:boto:Method: GET 2024-02-01T10:48:20.660 DEBUG:boto:Path: /vclfcn-61/?prefix=obj2 2024-02-01T10:48:20.661 DEBUG:boto:Data: 2024-02-01T10:48:20.661 DEBUG:boto:Headers: {} 2024-02-01T10:48:20.661 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:20.661 DEBUG:boto:Port: 8000 2024-02-01T10:48:20.661 DEBUG:boto:Params: {} 2024-02-01T10:48:20.661 DEBUG:boto:Token: None 2024-02-01T10:48:20.661 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:20 GMT /vclfcn-61/ 2024-02-01T10:48:20.661 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lr/jBd4QIloMXQ3awULOR1Oec94= 2024-02-01T10:48:20.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lr/jBd4QIloMXQ3awULOR1Oec94=', 'Content-Length': '0'} 2024-02-01T10:48:20.666 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e4fafd36a03648f-0065bb76f4-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:20.712 DEBUG:boto:b'vclfcn-61obj21000falseobj22024-02-01T10:46:46.380Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:20.713 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:48:20.713 DEBUG:boto:path=/vclfcn-61/obj3 2024-02-01T10:48:20.713 DEBUG:boto:auth_path=/vclfcn-61/obj3 2024-02-01T10:48:20.713 DEBUG:boto:path=/vclfcn-61/obj3?versionId=null 2024-02-01T10:48:20.713 DEBUG:boto:auth_path=/vclfcn-61/obj3?versionId=null 2024-02-01T10:48:20.713 DEBUG:boto:Method: GET 2024-02-01T10:48:20.713 DEBUG:boto:Path: /vclfcn-61/obj3?versionId=null 2024-02-01T10:48:20.713 DEBUG:boto:Data: 2024-02-01T10:48:20.713 DEBUG:boto:Headers: {} 2024-02-01T10:48:20.714 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:20.714 DEBUG:boto:Port: 8001 2024-02-01T10:48:20.714 DEBUG:boto:Params: {} 2024-02-01T10:48:20.714 DEBUG:boto:Token: None 2024-02-01T10:48:20.714 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:20 GMT /vclfcn-61/obj3?versionId=null 2024-02-01T10:48:20.714 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zkS473IYRQjyg3zt5XS6ZN9jvsk= 2024-02-01T10:48:20.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zkS473IYRQjyg3zt5XS6ZN9jvsk=', 'Content-Length': '0'} 2024-02-01T10:48:20.717 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002b26aed444a5b4d6-0065bb76f4-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:20.718 DEBUG:boto:path=/vclfcn-61/obj3 2024-02-01T10:48:20.718 DEBUG:boto:auth_path=/vclfcn-61/obj3 2024-02-01T10:48:20.718 DEBUG:boto:path=/vclfcn-61/obj3?versionId=null 2024-02-01T10:48:20.719 DEBUG:boto:auth_path=/vclfcn-61/obj3?versionId=null 2024-02-01T10:48:20.719 DEBUG:boto:Method: GET 2024-02-01T10:48:20.719 DEBUG:boto:Path: /vclfcn-61/obj3?versionId=null 2024-02-01T10:48:20.719 DEBUG:boto:Data: 2024-02-01T10:48:20.719 DEBUG:boto:Headers: {} 2024-02-01T10:48:20.719 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:20.719 DEBUG:boto:Port: 8000 2024-02-01T10:48:20.719 DEBUG:boto:Params: {} 2024-02-01T10:48:20.720 DEBUG:boto:Token: None 2024-02-01T10:48:20.720 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:20 GMT /vclfcn-61/obj3?versionId=null 2024-02-01T10:48:20.720 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zkS473IYRQjyg3zt5XS6ZN9jvsk= 2024-02-01T10:48:20.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zkS473IYRQjyg3zt5XS6ZN9jvsk=', 'Content-Length': '0'} 2024-02-01T10:48:20.723 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000025d18ae928a80c70-0065bb76f4-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:20.768 DEBUG:boto:path=/vclfcn-61/obj3 2024-02-01T10:48:20.768 DEBUG:boto:auth_path=/vclfcn-61/obj3 2024-02-01T10:48:20.768 DEBUG:boto:path=/vclfcn-61/obj3?versionId=null 2024-02-01T10:48:20.769 DEBUG:boto:auth_path=/vclfcn-61/obj3?versionId=null 2024-02-01T10:48:20.769 DEBUG:boto:Method: HEAD 2024-02-01T10:48:20.769 DEBUG:boto:Path: /vclfcn-61/obj3?versionId=null 2024-02-01T10:48:20.769 DEBUG:boto:Data: 2024-02-01T10:48:20.769 DEBUG:boto:Headers: {} 2024-02-01T10:48:20.769 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:20.769 DEBUG:boto:Port: 8001 2024-02-01T10:48:20.769 DEBUG:boto:Params: {} 2024-02-01T10:48:20.769 DEBUG:boto:Token: None 2024-02-01T10:48:20.769 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:20 GMT /vclfcn-61/obj3?versionId=null 2024-02-01T10:48:20.770 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:PABfwwPwayRu3OhF+rD5omEG7dc= 2024-02-01T10:48:20.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:PABfwwPwayRu3OhF+rD5omEG7dc=', 'Content-Length': '0'} 2024-02-01T10:48:20.773 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001279b92d20f1e29b-0065bb76f4-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:20.773 DEBUG:boto:path=/vclfcn-61/obj3 2024-02-01T10:48:20.774 DEBUG:boto:auth_path=/vclfcn-61/obj3 2024-02-01T10:48:20.774 DEBUG:boto:path=/vclfcn-61/obj3?versionId=null 2024-02-01T10:48:20.774 DEBUG:boto:auth_path=/vclfcn-61/obj3?versionId=null 2024-02-01T10:48:20.774 DEBUG:boto:Method: HEAD 2024-02-01T10:48:20.774 DEBUG:boto:Path: /vclfcn-61/obj3?versionId=null 2024-02-01T10:48:20.774 DEBUG:boto:Data: 2024-02-01T10:48:20.774 DEBUG:boto:Headers: {} 2024-02-01T10:48:20.774 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:20.774 DEBUG:boto:Port: 8000 2024-02-01T10:48:20.775 DEBUG:boto:Params: {} 2024-02-01T10:48:20.775 DEBUG:boto:Token: None 2024-02-01T10:48:20.775 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:20 GMT /vclfcn-61/obj3?versionId=null 2024-02-01T10:48:20.775 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:PABfwwPwayRu3OhF+rD5omEG7dc= 2024-02-01T10:48:20.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:PABfwwPwayRu3OhF+rD5omEG7dc=', 'Content-Length': '0'} 2024-02-01T10:48:20.778 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000736ce34795c907ce-0065bb76f4-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:20.778 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:48:20.778 DEBUG:boto:path=/vclfcn-61/obj3 2024-02-01T10:48:20.778 DEBUG:boto:auth_path=/vclfcn-61/obj3 2024-02-01T10:48:20.778 DEBUG:boto:path=/vclfcn-61/obj3?versionId=null 2024-02-01T10:48:20.779 DEBUG:boto:auth_path=/vclfcn-61/obj3?versionId=null 2024-02-01T10:48:20.779 DEBUG:boto:Method: GET 2024-02-01T10:48:20.779 DEBUG:boto:Path: /vclfcn-61/obj3?versionId=null 2024-02-01T10:48:20.779 DEBUG:boto:Data: 2024-02-01T10:48:20.779 DEBUG:boto:Headers: {} 2024-02-01T10:48:20.779 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:20.779 DEBUG:boto:Port: 8001 2024-02-01T10:48:20.779 DEBUG:boto:Params: {} 2024-02-01T10:48:20.779 DEBUG:boto:Token: None 2024-02-01T10:48:20.779 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:20 GMT /vclfcn-61/obj3?versionId=null 2024-02-01T10:48:20.780 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zkS473IYRQjyg3zt5XS6ZN9jvsk= 2024-02-01T10:48:20.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zkS473IYRQjyg3zt5XS6ZN9jvsk=', 'Content-Length': '0'} 2024-02-01T10:48:20.783 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000144e8bfa1055babe-0065bb76f4-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:20.783 DEBUG:boto:path=/vclfcn-61/obj3 2024-02-01T10:48:20.783 DEBUG:boto:auth_path=/vclfcn-61/obj3 2024-02-01T10:48:20.783 DEBUG:boto:path=/vclfcn-61/obj3?versionId=null 2024-02-01T10:48:20.783 DEBUG:boto:auth_path=/vclfcn-61/obj3?versionId=null 2024-02-01T10:48:20.783 DEBUG:boto:Method: GET 2024-02-01T10:48:20.783 DEBUG:boto:Path: /vclfcn-61/obj3?versionId=null 2024-02-01T10:48:20.784 DEBUG:boto:Data: 2024-02-01T10:48:20.784 DEBUG:boto:Headers: {} 2024-02-01T10:48:20.784 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:20.784 DEBUG:boto:Port: 8000 2024-02-01T10:48:20.784 DEBUG:boto:Params: {} 2024-02-01T10:48:20.784 DEBUG:boto:Token: None 2024-02-01T10:48:20.784 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:20 GMT /vclfcn-61/obj3?versionId=null 2024-02-01T10:48:20.784 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zkS473IYRQjyg3zt5XS6ZN9jvsk= 2024-02-01T10:48:20.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zkS473IYRQjyg3zt5XS6ZN9jvsk=', 'Content-Length': '0'} 2024-02-01T10:48:20.787 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008aca52c5a6d9577e-0065bb76f4-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:20.832 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:20.832 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:20.833 DEBUG:boto:path=/vclfcn-61/?prefix=obj3 2024-02-01T10:48:20.833 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj3 2024-02-01T10:48:20.833 DEBUG:boto:Method: GET 2024-02-01T10:48:20.833 DEBUG:boto:Path: /vclfcn-61/?prefix=obj3 2024-02-01T10:48:20.833 DEBUG:boto:Data: 2024-02-01T10:48:20.833 DEBUG:boto:Headers: {} 2024-02-01T10:48:20.833 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:20.833 DEBUG:boto:Port: 8001 2024-02-01T10:48:20.833 DEBUG:boto:Params: {} 2024-02-01T10:48:20.834 DEBUG:boto:Token: None 2024-02-01T10:48:20.834 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:20 GMT /vclfcn-61/ 2024-02-01T10:48:20.834 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lr/jBd4QIloMXQ3awULOR1Oec94= 2024-02-01T10:48:20.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lr/jBd4QIloMXQ3awULOR1Oec94=', 'Content-Length': '0'} 2024-02-01T10:48:20.840 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029cc27688d2e8be0-0065bb76f4-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:20.840 DEBUG:boto:b'vclfcn-61obj31000falseobj32024-02-01T10:46:46.433Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:20.841 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:20.841 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:20.841 DEBUG:boto:path=/vclfcn-61/?prefix=obj3 2024-02-01T10:48:20.841 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj3 2024-02-01T10:48:20.841 DEBUG:boto:Method: GET 2024-02-01T10:48:20.841 DEBUG:boto:Path: /vclfcn-61/?prefix=obj3 2024-02-01T10:48:20.841 DEBUG:boto:Data: 2024-02-01T10:48:20.841 DEBUG:boto:Headers: {} 2024-02-01T10:48:20.841 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:20.841 DEBUG:boto:Port: 8000 2024-02-01T10:48:20.842 DEBUG:boto:Params: {} 2024-02-01T10:48:20.842 DEBUG:boto:Token: None 2024-02-01T10:48:20.842 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:20 GMT /vclfcn-61/ 2024-02-01T10:48:20.842 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lr/jBd4QIloMXQ3awULOR1Oec94= 2024-02-01T10:48:20.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lr/jBd4QIloMXQ3awULOR1Oec94=', 'Content-Length': '0'} 2024-02-01T10:48:20.846 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000105cc84ab5505777-0065bb76f4-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:20.888 DEBUG:boto:b'vclfcn-61obj31000falseobj32024-02-01T10:46:46.433Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:20.889 DEBUG:boto:path=/vclfcn-61/obj3 2024-02-01T10:48:20.889 DEBUG:boto:auth_path=/vclfcn-61/obj3 2024-02-01T10:48:20.889 DEBUG:boto:Method: HEAD 2024-02-01T10:48:20.889 DEBUG:boto:Path: /vclfcn-61/obj3 2024-02-01T10:48:20.889 DEBUG:boto:Data: 2024-02-01T10:48:20.889 DEBUG:boto:Headers: {} 2024-02-01T10:48:20.889 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:20.889 DEBUG:boto:Port: 8001 2024-02-01T10:48:20.889 DEBUG:boto:Params: {} 2024-02-01T10:48:20.890 DEBUG:boto:Token: None 2024-02-01T10:48:20.890 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:20 GMT /vclfcn-61/obj3 2024-02-01T10:48:20.890 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5LUUq6sKURn1Tgm7a44LtGpKEls= 2024-02-01T10:48:20.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5LUUq6sKURn1Tgm7a44LtGpKEls=', 'Content-Length': '0'} 2024-02-01T10:48:20.893 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000117d62f7b7e395a1-0065bb76f4-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:20.893 DEBUG:boto:path=/vclfcn-61/obj3 2024-02-01T10:48:20.894 DEBUG:boto:auth_path=/vclfcn-61/obj3 2024-02-01T10:48:20.894 DEBUG:boto:Method: HEAD 2024-02-01T10:48:20.894 DEBUG:boto:Path: /vclfcn-61/obj3 2024-02-01T10:48:20.894 DEBUG:boto:Data: 2024-02-01T10:48:20.894 DEBUG:boto:Headers: {} 2024-02-01T10:48:20.894 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:20.894 DEBUG:boto:Port: 8000 2024-02-01T10:48:20.894 DEBUG:boto:Params: {} 2024-02-01T10:48:20.894 DEBUG:boto:Token: None 2024-02-01T10:48:20.894 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:20 GMT /vclfcn-61/obj3 2024-02-01T10:48:20.895 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5LUUq6sKURn1Tgm7a44LtGpKEls= 2024-02-01T10:48:20.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5LUUq6sKURn1Tgm7a44LtGpKEls=', 'Content-Length': '0'} 2024-02-01T10:48:20.897 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000903ddd4dda23420b-0065bb76f4-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:20.897 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:48:20.898 DEBUG:boto:path=/vclfcn-61/obj3 2024-02-01T10:48:20.898 DEBUG:boto:auth_path=/vclfcn-61/obj3 2024-02-01T10:48:20.898 DEBUG:boto:Method: GET 2024-02-01T10:48:20.898 DEBUG:boto:Path: /vclfcn-61/obj3 2024-02-01T10:48:20.898 DEBUG:boto:Data: 2024-02-01T10:48:20.898 DEBUG:boto:Headers: {} 2024-02-01T10:48:20.898 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:20.898 DEBUG:boto:Port: 8001 2024-02-01T10:48:20.898 DEBUG:boto:Params: {} 2024-02-01T10:48:20.898 DEBUG:boto:Token: None 2024-02-01T10:48:20.899 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:20 GMT /vclfcn-61/obj3 2024-02-01T10:48:20.899 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FuQp4rdiQ9Ab912NnuxKNzyt14o= 2024-02-01T10:48:20.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FuQp4rdiQ9Ab912NnuxKNzyt14o=', 'Content-Length': '0'} 2024-02-01T10:48:20.902 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cb5f046be8bdec50-0065bb76f4-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:20.902 DEBUG:boto:path=/vclfcn-61/obj3 2024-02-01T10:48:20.902 DEBUG:boto:auth_path=/vclfcn-61/obj3 2024-02-01T10:48:20.902 DEBUG:boto:Method: GET 2024-02-01T10:48:20.902 DEBUG:boto:Path: /vclfcn-61/obj3 2024-02-01T10:48:20.902 DEBUG:boto:Data: 2024-02-01T10:48:20.903 DEBUG:boto:Headers: {} 2024-02-01T10:48:20.903 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:20.903 DEBUG:boto:Port: 8000 2024-02-01T10:48:20.903 DEBUG:boto:Params: {} 2024-02-01T10:48:20.903 DEBUG:boto:Token: None 2024-02-01T10:48:20.903 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:20 GMT /vclfcn-61/obj3 2024-02-01T10:48:20.904 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FuQp4rdiQ9Ab912NnuxKNzyt14o= 2024-02-01T10:48:20.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FuQp4rdiQ9Ab912NnuxKNzyt14o=', 'Content-Length': '0'} 2024-02-01T10:48:20.907 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009c3e457f07927f84-0065bb76f4-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:20.948 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:20.948 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:20.948 DEBUG:boto:path=/vclfcn-61/?prefix=obj3 2024-02-01T10:48:20.949 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj3 2024-02-01T10:48:20.949 DEBUG:boto:Method: GET 2024-02-01T10:48:20.949 DEBUG:boto:Path: /vclfcn-61/?prefix=obj3 2024-02-01T10:48:20.949 DEBUG:boto:Data: 2024-02-01T10:48:20.949 DEBUG:boto:Headers: {} 2024-02-01T10:48:20.949 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:20.949 DEBUG:boto:Port: 8001 2024-02-01T10:48:20.949 DEBUG:boto:Params: {} 2024-02-01T10:48:20.949 DEBUG:boto:Token: None 2024-02-01T10:48:20.950 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:20 GMT /vclfcn-61/ 2024-02-01T10:48:20.950 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lr/jBd4QIloMXQ3awULOR1Oec94= 2024-02-01T10:48:20.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lr/jBd4QIloMXQ3awULOR1Oec94=', 'Content-Length': '0'} 2024-02-01T10:48:20.955 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008fef49eb8e013fc1-0065bb76f4-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:20.956 DEBUG:boto:b'vclfcn-61obj31000falseobj32024-02-01T10:46:46.433Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:20.956 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:20.956 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:20.957 DEBUG:boto:path=/vclfcn-61/?prefix=obj3 2024-02-01T10:48:20.957 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj3 2024-02-01T10:48:20.959 DEBUG:boto:Method: GET 2024-02-01T10:48:20.960 DEBUG:boto:Path: /vclfcn-61/?prefix=obj3 2024-02-01T10:48:20.960 DEBUG:boto:Data: 2024-02-01T10:48:20.960 DEBUG:boto:Headers: {} 2024-02-01T10:48:20.960 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:20.961 DEBUG:boto:Port: 8000 2024-02-01T10:48:20.961 DEBUG:boto:Params: {} 2024-02-01T10:48:20.961 DEBUG:boto:Token: None 2024-02-01T10:48:20.961 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:20 GMT /vclfcn-61/ 2024-02-01T10:48:20.961 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lr/jBd4QIloMXQ3awULOR1Oec94= 2024-02-01T10:48:20.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:20 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lr/jBd4QIloMXQ3awULOR1Oec94=', 'Content-Length': '0'} 2024-02-01T10:48:20.966 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087ab8807fe33c104-0065bb76f4-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:20 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:21.008 DEBUG:boto:b'vclfcn-61obj31000falseobj32024-02-01T10:46:46.433Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:21.008 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:48:21.009 DEBUG:boto:path=/vclfcn-61/obj4 2024-02-01T10:48:21.009 DEBUG:boto:auth_path=/vclfcn-61/obj4 2024-02-01T10:48:21.009 DEBUG:boto:path=/vclfcn-61/obj4?versionId=null 2024-02-01T10:48:21.009 DEBUG:boto:auth_path=/vclfcn-61/obj4?versionId=null 2024-02-01T10:48:21.009 DEBUG:boto:Method: GET 2024-02-01T10:48:21.009 DEBUG:boto:Path: /vclfcn-61/obj4?versionId=null 2024-02-01T10:48:21.009 DEBUG:boto:Data: 2024-02-01T10:48:21.009 DEBUG:boto:Headers: {} 2024-02-01T10:48:21.009 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:21.009 DEBUG:boto:Port: 8001 2024-02-01T10:48:21.010 DEBUG:boto:Params: {} 2024-02-01T10:48:21.010 DEBUG:boto:Token: None 2024-02-01T10:48:21.010 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:21 GMT /vclfcn-61/obj4?versionId=null 2024-02-01T10:48:21.010 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mKjwnlJ+BCNf3rS0KVTUGOsnUZY= 2024-02-01T10:48:21.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mKjwnlJ+BCNf3rS0KVTUGOsnUZY=', 'Content-Length': '0'} 2024-02-01T10:48:21.014 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008a9b9e7fb3196912-0065bb76f5-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:21.015 DEBUG:boto:path=/vclfcn-61/obj4 2024-02-01T10:48:21.015 DEBUG:boto:auth_path=/vclfcn-61/obj4 2024-02-01T10:48:21.015 DEBUG:boto:path=/vclfcn-61/obj4?versionId=null 2024-02-01T10:48:21.015 DEBUG:boto:auth_path=/vclfcn-61/obj4?versionId=null 2024-02-01T10:48:21.015 DEBUG:boto:Method: GET 2024-02-01T10:48:21.015 DEBUG:boto:Path: /vclfcn-61/obj4?versionId=null 2024-02-01T10:48:21.015 DEBUG:boto:Data: 2024-02-01T10:48:21.015 DEBUG:boto:Headers: {} 2024-02-01T10:48:21.016 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:21.016 DEBUG:boto:Port: 8000 2024-02-01T10:48:21.016 DEBUG:boto:Params: {} 2024-02-01T10:48:21.016 DEBUG:boto:Token: None 2024-02-01T10:48:21.016 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:21 GMT /vclfcn-61/obj4?versionId=null 2024-02-01T10:48:21.016 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mKjwnlJ+BCNf3rS0KVTUGOsnUZY= 2024-02-01T10:48:21.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mKjwnlJ+BCNf3rS0KVTUGOsnUZY=', 'Content-Length': '0'} 2024-02-01T10:48:21.020 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000022f445e11a5efd30-0065bb76f5-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:21.020 DEBUG:boto:path=/vclfcn-61/obj4 2024-02-01T10:48:21.020 DEBUG:boto:auth_path=/vclfcn-61/obj4 2024-02-01T10:48:21.021 DEBUG:boto:path=/vclfcn-61/obj4?versionId=null 2024-02-01T10:48:21.021 DEBUG:boto:auth_path=/vclfcn-61/obj4?versionId=null 2024-02-01T10:48:21.021 DEBUG:boto:Method: HEAD 2024-02-01T10:48:21.021 DEBUG:boto:Path: /vclfcn-61/obj4?versionId=null 2024-02-01T10:48:21.021 DEBUG:boto:Data: 2024-02-01T10:48:21.021 DEBUG:boto:Headers: {} 2024-02-01T10:48:21.022 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:21.022 DEBUG:boto:Port: 8001 2024-02-01T10:48:21.022 DEBUG:boto:Params: {} 2024-02-01T10:48:21.022 DEBUG:boto:Token: None 2024-02-01T10:48:21.022 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:21 GMT /vclfcn-61/obj4?versionId=null 2024-02-01T10:48:21.023 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tmlMeQLmD+kVV4foEIISlnvlVYE= 2024-02-01T10:48:21.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tmlMeQLmD+kVV4foEIISlnvlVYE=', 'Content-Length': '0'} 2024-02-01T10:48:21.026 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000052991f3f3b41e22d-0065bb76f5-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:21.026 DEBUG:boto:path=/vclfcn-61/obj4 2024-02-01T10:48:21.026 DEBUG:boto:auth_path=/vclfcn-61/obj4 2024-02-01T10:48:21.026 DEBUG:boto:path=/vclfcn-61/obj4?versionId=null 2024-02-01T10:48:21.026 DEBUG:boto:auth_path=/vclfcn-61/obj4?versionId=null 2024-02-01T10:48:21.026 DEBUG:boto:Method: HEAD 2024-02-01T10:48:21.026 DEBUG:boto:Path: /vclfcn-61/obj4?versionId=null 2024-02-01T10:48:21.027 DEBUG:boto:Data: 2024-02-01T10:48:21.027 DEBUG:boto:Headers: {} 2024-02-01T10:48:21.027 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:21.027 DEBUG:boto:Port: 8000 2024-02-01T10:48:21.027 DEBUG:boto:Params: {} 2024-02-01T10:48:21.027 DEBUG:boto:Token: None 2024-02-01T10:48:21.027 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:21 GMT /vclfcn-61/obj4?versionId=null 2024-02-01T10:48:21.027 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tmlMeQLmD+kVV4foEIISlnvlVYE= 2024-02-01T10:48:21.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tmlMeQLmD+kVV4foEIISlnvlVYE=', 'Content-Length': '0'} 2024-02-01T10:48:21.030 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004d48b33e0abddfca-0065bb76f5-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:21.030 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:48:21.031 DEBUG:boto:path=/vclfcn-61/obj4 2024-02-01T10:48:21.031 DEBUG:boto:auth_path=/vclfcn-61/obj4 2024-02-01T10:48:21.031 DEBUG:boto:path=/vclfcn-61/obj4?versionId=null 2024-02-01T10:48:21.031 DEBUG:boto:auth_path=/vclfcn-61/obj4?versionId=null 2024-02-01T10:48:21.031 DEBUG:boto:Method: GET 2024-02-01T10:48:21.031 DEBUG:boto:Path: /vclfcn-61/obj4?versionId=null 2024-02-01T10:48:21.031 DEBUG:boto:Data: 2024-02-01T10:48:21.031 DEBUG:boto:Headers: {} 2024-02-01T10:48:21.031 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:21.031 DEBUG:boto:Port: 8001 2024-02-01T10:48:21.032 DEBUG:boto:Params: {} 2024-02-01T10:48:21.032 DEBUG:boto:Token: None 2024-02-01T10:48:21.032 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:21 GMT /vclfcn-61/obj4?versionId=null 2024-02-01T10:48:21.032 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mKjwnlJ+BCNf3rS0KVTUGOsnUZY= 2024-02-01T10:48:21.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mKjwnlJ+BCNf3rS0KVTUGOsnUZY=', 'Content-Length': '0'} 2024-02-01T10:48:21.036 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000023eb49f7c8ddb02e-0065bb76f5-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:21.076 DEBUG:boto:path=/vclfcn-61/obj4 2024-02-01T10:48:21.076 DEBUG:boto:auth_path=/vclfcn-61/obj4 2024-02-01T10:48:21.076 DEBUG:boto:path=/vclfcn-61/obj4?versionId=null 2024-02-01T10:48:21.077 DEBUG:boto:auth_path=/vclfcn-61/obj4?versionId=null 2024-02-01T10:48:21.077 DEBUG:boto:Method: GET 2024-02-01T10:48:21.077 DEBUG:boto:Path: /vclfcn-61/obj4?versionId=null 2024-02-01T10:48:21.077 DEBUG:boto:Data: 2024-02-01T10:48:21.077 DEBUG:boto:Headers: {} 2024-02-01T10:48:21.078 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:21.078 DEBUG:boto:Port: 8000 2024-02-01T10:48:21.078 DEBUG:boto:Params: {} 2024-02-01T10:48:21.078 DEBUG:boto:Token: None 2024-02-01T10:48:21.078 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:21 GMT /vclfcn-61/obj4?versionId=null 2024-02-01T10:48:21.078 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mKjwnlJ+BCNf3rS0KVTUGOsnUZY= 2024-02-01T10:48:21.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mKjwnlJ+BCNf3rS0KVTUGOsnUZY=', 'Content-Length': '0'} 2024-02-01T10:48:21.081 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000438de3b8a28c88b8-0065bb76f5-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:21.124 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:21.125 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:21.125 DEBUG:boto:path=/vclfcn-61/?prefix=obj4 2024-02-01T10:48:21.125 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj4 2024-02-01T10:48:21.125 DEBUG:boto:Method: GET 2024-02-01T10:48:21.125 DEBUG:boto:Path: /vclfcn-61/?prefix=obj4 2024-02-01T10:48:21.126 DEBUG:boto:Data: 2024-02-01T10:48:21.126 DEBUG:boto:Headers: {} 2024-02-01T10:48:21.126 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:21.126 DEBUG:boto:Port: 8001 2024-02-01T10:48:21.127 DEBUG:boto:Params: {} 2024-02-01T10:48:21.127 DEBUG:boto:Token: None 2024-02-01T10:48:21.127 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:21 GMT /vclfcn-61/ 2024-02-01T10:48:21.127 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZhinEpH8inHnaTrD74abEQSbAjw= 2024-02-01T10:48:21.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZhinEpH8inHnaTrD74abEQSbAjw=', 'Content-Length': '0'} 2024-02-01T10:48:21.132 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dadcd08bc28f5d74-0065bb76f5-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:21.132 DEBUG:boto:b'vclfcn-61obj41000falseobj42024-02-01T10:46:46.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:21.133 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:21.133 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:21.133 DEBUG:boto:path=/vclfcn-61/?prefix=obj4 2024-02-01T10:48:21.133 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj4 2024-02-01T10:48:21.133 DEBUG:boto:Method: GET 2024-02-01T10:48:21.133 DEBUG:boto:Path: /vclfcn-61/?prefix=obj4 2024-02-01T10:48:21.133 DEBUG:boto:Data: 2024-02-01T10:48:21.133 DEBUG:boto:Headers: {} 2024-02-01T10:48:21.134 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:21.134 DEBUG:boto:Port: 8000 2024-02-01T10:48:21.134 DEBUG:boto:Params: {} 2024-02-01T10:48:21.134 DEBUG:boto:Token: None 2024-02-01T10:48:21.134 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:21 GMT /vclfcn-61/ 2024-02-01T10:48:21.135 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZhinEpH8inHnaTrD74abEQSbAjw= 2024-02-01T10:48:21.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZhinEpH8inHnaTrD74abEQSbAjw=', 'Content-Length': '0'} 2024-02-01T10:48:21.139 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c5f7e82b15910a0-0065bb76f5-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:21.140 DEBUG:boto:b'vclfcn-61obj41000falseobj42024-02-01T10:46:46.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:21.140 DEBUG:boto:path=/vclfcn-61/obj4 2024-02-01T10:48:21.140 DEBUG:boto:auth_path=/vclfcn-61/obj4 2024-02-01T10:48:21.141 DEBUG:boto:Method: HEAD 2024-02-01T10:48:21.141 DEBUG:boto:Path: /vclfcn-61/obj4 2024-02-01T10:48:21.141 DEBUG:boto:Data: 2024-02-01T10:48:21.141 DEBUG:boto:Headers: {} 2024-02-01T10:48:21.141 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:21.142 DEBUG:boto:Port: 8001 2024-02-01T10:48:21.142 DEBUG:boto:Params: {} 2024-02-01T10:48:21.142 DEBUG:boto:Token: None 2024-02-01T10:48:21.142 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:21 GMT /vclfcn-61/obj4 2024-02-01T10:48:21.142 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:VVwQKZ7kTvtNUq2Ehl3DtAU/Awo= 2024-02-01T10:48:21.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:VVwQKZ7kTvtNUq2Ehl3DtAU/Awo=', 'Content-Length': '0'} 2024-02-01T10:48:21.146 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d781e3ce4321f2bc-0065bb76f5-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:21.146 DEBUG:boto:path=/vclfcn-61/obj4 2024-02-01T10:48:21.146 DEBUG:boto:auth_path=/vclfcn-61/obj4 2024-02-01T10:48:21.146 DEBUG:boto:Method: HEAD 2024-02-01T10:48:21.147 DEBUG:boto:Path: /vclfcn-61/obj4 2024-02-01T10:48:21.147 DEBUG:boto:Data: 2024-02-01T10:48:21.147 DEBUG:boto:Headers: {} 2024-02-01T10:48:21.147 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:21.147 DEBUG:boto:Port: 8000 2024-02-01T10:48:21.147 DEBUG:boto:Params: {} 2024-02-01T10:48:21.148 DEBUG:boto:Token: None 2024-02-01T10:48:21.148 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:21 GMT /vclfcn-61/obj4 2024-02-01T10:48:21.148 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:VVwQKZ7kTvtNUq2Ehl3DtAU/Awo= 2024-02-01T10:48:21.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:VVwQKZ7kTvtNUq2Ehl3DtAU/Awo=', 'Content-Length': '0'} 2024-02-01T10:48:21.152 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000078fdee4aea590707-0065bb76f5-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:21.152 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:48:21.152 DEBUG:boto:path=/vclfcn-61/obj4 2024-02-01T10:48:21.152 DEBUG:boto:auth_path=/vclfcn-61/obj4 2024-02-01T10:48:21.153 DEBUG:boto:Method: GET 2024-02-01T10:48:21.153 DEBUG:boto:Path: /vclfcn-61/obj4 2024-02-01T10:48:21.153 DEBUG:boto:Data: 2024-02-01T10:48:21.153 DEBUG:boto:Headers: {} 2024-02-01T10:48:21.153 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:21.153 DEBUG:boto:Port: 8001 2024-02-01T10:48:21.153 DEBUG:boto:Params: {} 2024-02-01T10:48:21.154 DEBUG:boto:Token: None 2024-02-01T10:48:21.154 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:21 GMT /vclfcn-61/obj4 2024-02-01T10:48:21.154 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:SL2D9gf33aOhxos5fOi6atkWUak= 2024-02-01T10:48:21.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:SL2D9gf33aOhxos5fOi6atkWUak=', 'Content-Length': '0'} 2024-02-01T10:48:21.158 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004aa7ace761f3c497-0065bb76f5-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:21.158 DEBUG:boto:path=/vclfcn-61/obj4 2024-02-01T10:48:21.158 DEBUG:boto:auth_path=/vclfcn-61/obj4 2024-02-01T10:48:21.158 DEBUG:boto:Method: GET 2024-02-01T10:48:21.159 DEBUG:boto:Path: /vclfcn-61/obj4 2024-02-01T10:48:21.159 DEBUG:boto:Data: 2024-02-01T10:48:21.159 DEBUG:boto:Headers: {} 2024-02-01T10:48:21.159 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:21.159 DEBUG:boto:Port: 8000 2024-02-01T10:48:21.159 DEBUG:boto:Params: {} 2024-02-01T10:48:21.159 DEBUG:boto:Token: None 2024-02-01T10:48:21.159 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:21 GMT /vclfcn-61/obj4 2024-02-01T10:48:21.160 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:SL2D9gf33aOhxos5fOi6atkWUak= 2024-02-01T10:48:21.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:SL2D9gf33aOhxos5fOi6atkWUak=', 'Content-Length': '0'} 2024-02-01T10:48:21.166 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000076afbc203c12c4e1-0065bb76f5-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:21.208 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:21.209 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:21.209 DEBUG:boto:path=/vclfcn-61/?prefix=obj4 2024-02-01T10:48:21.209 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj4 2024-02-01T10:48:21.209 DEBUG:boto:Method: GET 2024-02-01T10:48:21.209 DEBUG:boto:Path: /vclfcn-61/?prefix=obj4 2024-02-01T10:48:21.209 DEBUG:boto:Data: 2024-02-01T10:48:21.210 DEBUG:boto:Headers: {} 2024-02-01T10:48:21.210 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:21.210 DEBUG:boto:Port: 8001 2024-02-01T10:48:21.210 DEBUG:boto:Params: {} 2024-02-01T10:48:21.210 DEBUG:boto:Token: None 2024-02-01T10:48:21.210 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:21 GMT /vclfcn-61/ 2024-02-01T10:48:21.210 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZhinEpH8inHnaTrD74abEQSbAjw= 2024-02-01T10:48:21.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZhinEpH8inHnaTrD74abEQSbAjw=', 'Content-Length': '0'} 2024-02-01T10:48:21.215 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006367ec1769dbb1da-0065bb76f5-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:21.215 DEBUG:boto:b'vclfcn-61obj41000falseobj42024-02-01T10:46:46.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:21.215 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:21.216 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:21.216 DEBUG:boto:path=/vclfcn-61/?prefix=obj4 2024-02-01T10:48:21.216 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj4 2024-02-01T10:48:21.216 DEBUG:boto:Method: GET 2024-02-01T10:48:21.216 DEBUG:boto:Path: /vclfcn-61/?prefix=obj4 2024-02-01T10:48:21.216 DEBUG:boto:Data: 2024-02-01T10:48:21.216 DEBUG:boto:Headers: {} 2024-02-01T10:48:21.216 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:21.216 DEBUG:boto:Port: 8000 2024-02-01T10:48:21.217 DEBUG:boto:Params: {} 2024-02-01T10:48:21.217 DEBUG:boto:Token: None 2024-02-01T10:48:21.217 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:21 GMT /vclfcn-61/ 2024-02-01T10:48:21.217 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZhinEpH8inHnaTrD74abEQSbAjw= 2024-02-01T10:48:21.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZhinEpH8inHnaTrD74abEQSbAjw=', 'Content-Length': '0'} 2024-02-01T10:48:21.221 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000072c1247a018282c-0065bb76f5-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:21.222 DEBUG:boto:b'vclfcn-61obj41000falseobj42024-02-01T10:46:46.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:21.222 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-02-01T10:48:21.222 DEBUG:boto:path=/vclfcn-61/obj5 2024-02-01T10:48:21.223 DEBUG:boto:auth_path=/vclfcn-61/obj5 2024-02-01T10:48:21.223 DEBUG:boto:path=/vclfcn-61/obj5?versionId=null 2024-02-01T10:48:21.223 DEBUG:boto:auth_path=/vclfcn-61/obj5?versionId=null 2024-02-01T10:48:21.223 DEBUG:boto:Method: GET 2024-02-01T10:48:21.223 DEBUG:boto:Path: /vclfcn-61/obj5?versionId=null 2024-02-01T10:48:21.223 DEBUG:boto:Data: 2024-02-01T10:48:21.223 DEBUG:boto:Headers: {} 2024-02-01T10:48:21.223 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:21.223 DEBUG:boto:Port: 8001 2024-02-01T10:48:21.223 DEBUG:boto:Params: {} 2024-02-01T10:48:21.224 DEBUG:boto:Token: None 2024-02-01T10:48:21.224 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:21 GMT /vclfcn-61/obj5?versionId=null 2024-02-01T10:48:21.224 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+71A3drPUKfc4QcWx7TpmPCeLvI= 2024-02-01T10:48:21.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+71A3drPUKfc4QcWx7TpmPCeLvI=', 'Content-Length': '0'} 2024-02-01T10:48:21.228 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a426d2c05113e110-0065bb76f5-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:21.268 DEBUG:boto:path=/vclfcn-61/obj5 2024-02-01T10:48:21.268 DEBUG:boto:auth_path=/vclfcn-61/obj5 2024-02-01T10:48:21.269 DEBUG:boto:path=/vclfcn-61/obj5?versionId=null 2024-02-01T10:48:21.269 DEBUG:boto:auth_path=/vclfcn-61/obj5?versionId=null 2024-02-01T10:48:21.269 DEBUG:boto:Method: GET 2024-02-01T10:48:21.269 DEBUG:boto:Path: /vclfcn-61/obj5?versionId=null 2024-02-01T10:48:21.269 DEBUG:boto:Data: 2024-02-01T10:48:21.269 DEBUG:boto:Headers: {} 2024-02-01T10:48:21.269 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:21.270 DEBUG:boto:Port: 8000 2024-02-01T10:48:21.270 DEBUG:boto:Params: {} 2024-02-01T10:48:21.270 DEBUG:boto:Token: None 2024-02-01T10:48:21.270 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:21 GMT /vclfcn-61/obj5?versionId=null 2024-02-01T10:48:21.270 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+71A3drPUKfc4QcWx7TpmPCeLvI= 2024-02-01T10:48:21.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+71A3drPUKfc4QcWx7TpmPCeLvI=', 'Content-Length': '0'} 2024-02-01T10:48:21.274 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006fd1d72e33c6783c-0065bb76f5-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:21.275 DEBUG:boto:path=/vclfcn-61/obj5 2024-02-01T10:48:21.275 DEBUG:boto:auth_path=/vclfcn-61/obj5 2024-02-01T10:48:21.275 DEBUG:boto:path=/vclfcn-61/obj5?versionId=null 2024-02-01T10:48:21.275 DEBUG:boto:auth_path=/vclfcn-61/obj5?versionId=null 2024-02-01T10:48:21.275 DEBUG:boto:Method: HEAD 2024-02-01T10:48:21.275 DEBUG:boto:Path: /vclfcn-61/obj5?versionId=null 2024-02-01T10:48:21.276 DEBUG:boto:Data: 2024-02-01T10:48:21.276 DEBUG:boto:Headers: {} 2024-02-01T10:48:21.276 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:21.276 DEBUG:boto:Port: 8001 2024-02-01T10:48:21.276 DEBUG:boto:Params: {} 2024-02-01T10:48:21.277 DEBUG:boto:Token: None 2024-02-01T10:48:21.277 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:21 GMT /vclfcn-61/obj5?versionId=null 2024-02-01T10:48:21.277 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Mlpy8IzzhPfS4WcHlNON/tMHAPE= 2024-02-01T10:48:21.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Mlpy8IzzhPfS4WcHlNON/tMHAPE=', 'Content-Length': '0'} 2024-02-01T10:48:21.280 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bd38d95849a9ec5d-0065bb76f5-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:21.281 DEBUG:boto:path=/vclfcn-61/obj5 2024-02-01T10:48:21.281 DEBUG:boto:auth_path=/vclfcn-61/obj5 2024-02-01T10:48:21.281 DEBUG:boto:path=/vclfcn-61/obj5?versionId=null 2024-02-01T10:48:21.281 DEBUG:boto:auth_path=/vclfcn-61/obj5?versionId=null 2024-02-01T10:48:21.281 DEBUG:boto:Method: HEAD 2024-02-01T10:48:21.282 DEBUG:boto:Path: /vclfcn-61/obj5?versionId=null 2024-02-01T10:48:21.282 DEBUG:boto:Data: 2024-02-01T10:48:21.282 DEBUG:boto:Headers: {} 2024-02-01T10:48:21.282 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:21.282 DEBUG:boto:Port: 8000 2024-02-01T10:48:21.282 DEBUG:boto:Params: {} 2024-02-01T10:48:21.282 DEBUG:boto:Token: None 2024-02-01T10:48:21.283 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:21 GMT /vclfcn-61/obj5?versionId=null 2024-02-01T10:48:21.283 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Mlpy8IzzhPfS4WcHlNON/tMHAPE= 2024-02-01T10:48:21.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Mlpy8IzzhPfS4WcHlNON/tMHAPE=', 'Content-Length': '0'} 2024-02-01T10:48:21.286 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d8aff223abc7d929-0065bb76f5-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:21.286 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-02-01T10:48:21.287 DEBUG:boto:path=/vclfcn-61/obj5 2024-02-01T10:48:21.287 DEBUG:boto:auth_path=/vclfcn-61/obj5 2024-02-01T10:48:21.287 DEBUG:boto:path=/vclfcn-61/obj5?versionId=null 2024-02-01T10:48:21.287 DEBUG:boto:auth_path=/vclfcn-61/obj5?versionId=null 2024-02-01T10:48:21.287 DEBUG:boto:Method: GET 2024-02-01T10:48:21.287 DEBUG:boto:Path: /vclfcn-61/obj5?versionId=null 2024-02-01T10:48:21.287 DEBUG:boto:Data: 2024-02-01T10:48:21.288 DEBUG:boto:Headers: {} 2024-02-01T10:48:21.288 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:21.288 DEBUG:boto:Port: 8001 2024-02-01T10:48:21.288 DEBUG:boto:Params: {} 2024-02-01T10:48:21.288 DEBUG:boto:Token: None 2024-02-01T10:48:21.288 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:21 GMT /vclfcn-61/obj5?versionId=null 2024-02-01T10:48:21.289 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+71A3drPUKfc4QcWx7TpmPCeLvI= 2024-02-01T10:48:21.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+71A3drPUKfc4QcWx7TpmPCeLvI=', 'Content-Length': '0'} 2024-02-01T10:48:21.292 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000996e7bf2635f5485-0065bb76f5-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:21.292 DEBUG:boto:path=/vclfcn-61/obj5 2024-02-01T10:48:21.292 DEBUG:boto:auth_path=/vclfcn-61/obj5 2024-02-01T10:48:21.293 DEBUG:boto:path=/vclfcn-61/obj5?versionId=null 2024-02-01T10:48:21.293 DEBUG:boto:auth_path=/vclfcn-61/obj5?versionId=null 2024-02-01T10:48:21.293 DEBUG:boto:Method: GET 2024-02-01T10:48:21.293 DEBUG:boto:Path: /vclfcn-61/obj5?versionId=null 2024-02-01T10:48:21.294 DEBUG:boto:Data: 2024-02-01T10:48:21.294 DEBUG:boto:Headers: {} 2024-02-01T10:48:21.294 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:21.294 DEBUG:boto:Port: 8000 2024-02-01T10:48:21.294 DEBUG:boto:Params: {} 2024-02-01T10:48:21.294 DEBUG:boto:Token: None 2024-02-01T10:48:21.294 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:21 GMT /vclfcn-61/obj5?versionId=null 2024-02-01T10:48:21.294 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+71A3drPUKfc4QcWx7TpmPCeLvI= 2024-02-01T10:48:21.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+71A3drPUKfc4QcWx7TpmPCeLvI=', 'Content-Length': '0'} 2024-02-01T10:48:21.298 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000099f5e6d0d63414e1-0065bb76f5-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:21.340 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:21.341 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:21.341 DEBUG:boto:path=/vclfcn-61/?prefix=obj5 2024-02-01T10:48:21.341 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj5 2024-02-01T10:48:21.341 DEBUG:boto:Method: GET 2024-02-01T10:48:21.341 DEBUG:boto:Path: /vclfcn-61/?prefix=obj5 2024-02-01T10:48:21.341 DEBUG:boto:Data: 2024-02-01T10:48:21.341 DEBUG:boto:Headers: {} 2024-02-01T10:48:21.341 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:21.341 DEBUG:boto:Port: 8001 2024-02-01T10:48:21.341 DEBUG:boto:Params: {} 2024-02-01T10:48:21.342 DEBUG:boto:Token: None 2024-02-01T10:48:21.342 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:21 GMT /vclfcn-61/ 2024-02-01T10:48:21.342 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZhinEpH8inHnaTrD74abEQSbAjw= 2024-02-01T10:48:21.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZhinEpH8inHnaTrD74abEQSbAjw=', 'Content-Length': '0'} 2024-02-01T10:48:21.347 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090ee21a2400e40ad-0065bb76f5-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:21.347 DEBUG:boto:b'vclfcn-61obj51000falseobj52024-02-01T10:47:30.425Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:21.347 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:21.348 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:21.348 DEBUG:boto:path=/vclfcn-61/?prefix=obj5 2024-02-01T10:48:21.348 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj5 2024-02-01T10:48:21.348 DEBUG:boto:Method: GET 2024-02-01T10:48:21.348 DEBUG:boto:Path: /vclfcn-61/?prefix=obj5 2024-02-01T10:48:21.348 DEBUG:boto:Data: 2024-02-01T10:48:21.348 DEBUG:boto:Headers: {} 2024-02-01T10:48:21.348 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:21.348 DEBUG:boto:Port: 8000 2024-02-01T10:48:21.348 DEBUG:boto:Params: {} 2024-02-01T10:48:21.349 DEBUG:boto:Token: None 2024-02-01T10:48:21.349 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:21 GMT /vclfcn-61/ 2024-02-01T10:48:21.349 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZhinEpH8inHnaTrD74abEQSbAjw= 2024-02-01T10:48:21.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZhinEpH8inHnaTrD74abEQSbAjw=', 'Content-Length': '0'} 2024-02-01T10:48:21.353 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b8e17a16d2c4029-0065bb76f5-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:21.354 DEBUG:boto:b'vclfcn-61obj51000falseobj52024-02-01T10:47:30.425Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:21.354 DEBUG:boto:path=/vclfcn-61/obj5 2024-02-01T10:48:21.354 DEBUG:boto:auth_path=/vclfcn-61/obj5 2024-02-01T10:48:21.354 DEBUG:boto:Method: HEAD 2024-02-01T10:48:21.354 DEBUG:boto:Path: /vclfcn-61/obj5 2024-02-01T10:48:21.354 DEBUG:boto:Data: 2024-02-01T10:48:21.354 DEBUG:boto:Headers: {} 2024-02-01T10:48:21.355 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:21.355 DEBUG:boto:Port: 8001 2024-02-01T10:48:21.355 DEBUG:boto:Params: {} 2024-02-01T10:48:21.355 DEBUG:boto:Token: None 2024-02-01T10:48:21.355 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:21 GMT /vclfcn-61/obj5 2024-02-01T10:48:21.355 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:t/TrNykNV8LuPvDaDI5Y9Do3SP0= 2024-02-01T10:48:21.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:t/TrNykNV8LuPvDaDI5Y9Do3SP0=', 'Content-Length': '0'} 2024-02-01T10:48:21.358 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000097f333725fa2de00-0065bb76f5-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:21.358 DEBUG:boto:path=/vclfcn-61/obj5 2024-02-01T10:48:21.358 DEBUG:boto:auth_path=/vclfcn-61/obj5 2024-02-01T10:48:21.358 DEBUG:boto:Method: HEAD 2024-02-01T10:48:21.358 DEBUG:boto:Path: /vclfcn-61/obj5 2024-02-01T10:48:21.358 DEBUG:boto:Data: 2024-02-01T10:48:21.359 DEBUG:boto:Headers: {} 2024-02-01T10:48:21.359 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:21.359 DEBUG:boto:Port: 8000 2024-02-01T10:48:21.359 DEBUG:boto:Params: {} 2024-02-01T10:48:21.359 DEBUG:boto:Token: None 2024-02-01T10:48:21.359 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:21 GMT /vclfcn-61/obj5 2024-02-01T10:48:21.359 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:t/TrNykNV8LuPvDaDI5Y9Do3SP0= 2024-02-01T10:48:21.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:t/TrNykNV8LuPvDaDI5Y9Do3SP0=', 'Content-Length': '0'} 2024-02-01T10:48:21.362 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002f6e0963849d5535-0065bb76f5-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:21.362 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-02-01T10:48:21.362 DEBUG:boto:path=/vclfcn-61/obj5 2024-02-01T10:48:21.363 DEBUG:boto:auth_path=/vclfcn-61/obj5 2024-02-01T10:48:21.363 DEBUG:boto:Method: GET 2024-02-01T10:48:21.363 DEBUG:boto:Path: /vclfcn-61/obj5 2024-02-01T10:48:21.363 DEBUG:boto:Data: 2024-02-01T10:48:21.363 DEBUG:boto:Headers: {} 2024-02-01T10:48:21.363 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:21.363 DEBUG:boto:Port: 8001 2024-02-01T10:48:21.363 DEBUG:boto:Params: {} 2024-02-01T10:48:21.363 DEBUG:boto:Token: None 2024-02-01T10:48:21.364 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:21 GMT /vclfcn-61/obj5 2024-02-01T10:48:21.364 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tahZ916wrJ/vBBd0lcJy8twwv3M= 2024-02-01T10:48:21.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tahZ916wrJ/vBBd0lcJy8twwv3M=', 'Content-Length': '0'} 2024-02-01T10:48:21.366 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003d723ab637111e94-0065bb76f5-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:21.408 DEBUG:boto:path=/vclfcn-61/obj5 2024-02-01T10:48:21.408 DEBUG:boto:auth_path=/vclfcn-61/obj5 2024-02-01T10:48:21.408 DEBUG:boto:Method: GET 2024-02-01T10:48:21.408 DEBUG:boto:Path: /vclfcn-61/obj5 2024-02-01T10:48:21.408 DEBUG:boto:Data: 2024-02-01T10:48:21.409 DEBUG:boto:Headers: {} 2024-02-01T10:48:21.409 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:21.409 DEBUG:boto:Port: 8000 2024-02-01T10:48:21.409 DEBUG:boto:Params: {} 2024-02-01T10:48:21.409 DEBUG:boto:Token: None 2024-02-01T10:48:21.409 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:21 GMT /vclfcn-61/obj5 2024-02-01T10:48:21.409 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tahZ916wrJ/vBBd0lcJy8twwv3M= 2024-02-01T10:48:21.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tahZ916wrJ/vBBd0lcJy8twwv3M=', 'Content-Length': '0'} 2024-02-01T10:48:21.412 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e0f03231ef218fab-0065bb76f5-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:21.460 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:21.460 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:21.460 DEBUG:boto:path=/vclfcn-61/?prefix=obj5 2024-02-01T10:48:21.461 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj5 2024-02-01T10:48:21.461 DEBUG:boto:Method: GET 2024-02-01T10:48:21.461 DEBUG:boto:Path: /vclfcn-61/?prefix=obj5 2024-02-01T10:48:21.461 DEBUG:boto:Data: 2024-02-01T10:48:21.461 DEBUG:boto:Headers: {} 2024-02-01T10:48:21.461 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:21.461 DEBUG:boto:Port: 8001 2024-02-01T10:48:21.461 DEBUG:boto:Params: {} 2024-02-01T10:48:21.461 DEBUG:boto:Token: None 2024-02-01T10:48:21.462 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:21 GMT /vclfcn-61/ 2024-02-01T10:48:21.462 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZhinEpH8inHnaTrD74abEQSbAjw= 2024-02-01T10:48:21.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZhinEpH8inHnaTrD74abEQSbAjw=', 'Content-Length': '0'} 2024-02-01T10:48:21.467 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b516110ba31c96c2-0065bb76f5-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:21.468 DEBUG:boto:b'vclfcn-61obj51000falseobj52024-02-01T10:47:30.425Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:21.469 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:21.469 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:21.469 DEBUG:boto:path=/vclfcn-61/?prefix=obj5 2024-02-01T10:48:21.469 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj5 2024-02-01T10:48:21.469 DEBUG:boto:Method: GET 2024-02-01T10:48:21.469 DEBUG:boto:Path: /vclfcn-61/?prefix=obj5 2024-02-01T10:48:21.469 DEBUG:boto:Data: 2024-02-01T10:48:21.469 DEBUG:boto:Headers: {} 2024-02-01T10:48:21.469 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:21.470 DEBUG:boto:Port: 8000 2024-02-01T10:48:21.470 DEBUG:boto:Params: {} 2024-02-01T10:48:21.470 DEBUG:boto:Token: None 2024-02-01T10:48:21.470 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:21 GMT /vclfcn-61/ 2024-02-01T10:48:21.470 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZhinEpH8inHnaTrD74abEQSbAjw= 2024-02-01T10:48:21.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZhinEpH8inHnaTrD74abEQSbAjw=', 'Content-Length': '0'} 2024-02-01T10:48:21.475 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004edba164af146c08-0065bb76f5-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:21.516 DEBUG:boto:b'vclfcn-61obj51000falseobj52024-02-01T10:47:30.425Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:21.517 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-02-01T10:48:21.517 DEBUG:boto:path=/vclfcn-61/obj6 2024-02-01T10:48:21.517 DEBUG:boto:auth_path=/vclfcn-61/obj6 2024-02-01T10:48:21.517 DEBUG:boto:path=/vclfcn-61/obj6?versionId=null 2024-02-01T10:48:21.517 DEBUG:boto:auth_path=/vclfcn-61/obj6?versionId=null 2024-02-01T10:48:21.517 DEBUG:boto:Method: GET 2024-02-01T10:48:21.517 DEBUG:boto:Path: /vclfcn-61/obj6?versionId=null 2024-02-01T10:48:21.517 DEBUG:boto:Data: 2024-02-01T10:48:21.517 DEBUG:boto:Headers: {} 2024-02-01T10:48:21.518 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:21.518 DEBUG:boto:Port: 8001 2024-02-01T10:48:21.518 DEBUG:boto:Params: {} 2024-02-01T10:48:21.518 DEBUG:boto:Token: None 2024-02-01T10:48:21.518 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:21 GMT /vclfcn-61/obj6?versionId=null 2024-02-01T10:48:21.518 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8Vjm+mC+sF/D6KWp+b6Ij6blpmY= 2024-02-01T10:48:21.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8Vjm+mC+sF/D6KWp+b6Ij6blpmY=', 'Content-Length': '0'} 2024-02-01T10:48:21.521 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001e43d1e27401f3aa-0065bb76f5-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:21.522 DEBUG:boto:path=/vclfcn-61/obj6 2024-02-01T10:48:21.522 DEBUG:boto:auth_path=/vclfcn-61/obj6 2024-02-01T10:48:21.522 DEBUG:boto:path=/vclfcn-61/obj6?versionId=null 2024-02-01T10:48:21.522 DEBUG:boto:auth_path=/vclfcn-61/obj6?versionId=null 2024-02-01T10:48:21.523 DEBUG:boto:Method: GET 2024-02-01T10:48:21.523 DEBUG:boto:Path: /vclfcn-61/obj6?versionId=null 2024-02-01T10:48:21.523 DEBUG:boto:Data: 2024-02-01T10:48:21.523 DEBUG:boto:Headers: {} 2024-02-01T10:48:21.523 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:21.523 DEBUG:boto:Port: 8000 2024-02-01T10:48:21.523 DEBUG:boto:Params: {} 2024-02-01T10:48:21.523 DEBUG:boto:Token: None 2024-02-01T10:48:21.523 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:21 GMT /vclfcn-61/obj6?versionId=null 2024-02-01T10:48:21.523 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8Vjm+mC+sF/D6KWp+b6Ij6blpmY= 2024-02-01T10:48:21.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8Vjm+mC+sF/D6KWp+b6Ij6blpmY=', 'Content-Length': '0'} 2024-02-01T10:48:21.527 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003063a1c4f89546e0-0065bb76f5-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:21.528 DEBUG:boto:path=/vclfcn-61/obj6 2024-02-01T10:48:21.528 DEBUG:boto:auth_path=/vclfcn-61/obj6 2024-02-01T10:48:21.528 DEBUG:boto:path=/vclfcn-61/obj6?versionId=null 2024-02-01T10:48:21.528 DEBUG:boto:auth_path=/vclfcn-61/obj6?versionId=null 2024-02-01T10:48:21.528 DEBUG:boto:Method: HEAD 2024-02-01T10:48:21.528 DEBUG:boto:Path: /vclfcn-61/obj6?versionId=null 2024-02-01T10:48:21.528 DEBUG:boto:Data: 2024-02-01T10:48:21.529 DEBUG:boto:Headers: {} 2024-02-01T10:48:21.529 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:21.529 DEBUG:boto:Port: 8001 2024-02-01T10:48:21.529 DEBUG:boto:Params: {} 2024-02-01T10:48:21.529 DEBUG:boto:Token: None 2024-02-01T10:48:21.529 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:21 GMT /vclfcn-61/obj6?versionId=null 2024-02-01T10:48:21.529 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:c7p+TGsKqwG//bsFhcU4/BkrOHI= 2024-02-01T10:48:21.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:c7p+TGsKqwG//bsFhcU4/BkrOHI=', 'Content-Length': '0'} 2024-02-01T10:48:21.532 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d39e9d48d6593ba8-0065bb76f5-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:21.532 DEBUG:boto:path=/vclfcn-61/obj6 2024-02-01T10:48:21.532 DEBUG:boto:auth_path=/vclfcn-61/obj6 2024-02-01T10:48:21.532 DEBUG:boto:path=/vclfcn-61/obj6?versionId=null 2024-02-01T10:48:21.533 DEBUG:boto:auth_path=/vclfcn-61/obj6?versionId=null 2024-02-01T10:48:21.533 DEBUG:boto:Method: HEAD 2024-02-01T10:48:21.533 DEBUG:boto:Path: /vclfcn-61/obj6?versionId=null 2024-02-01T10:48:21.533 DEBUG:boto:Data: 2024-02-01T10:48:21.533 DEBUG:boto:Headers: {} 2024-02-01T10:48:21.533 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:21.533 DEBUG:boto:Port: 8000 2024-02-01T10:48:21.533 DEBUG:boto:Params: {} 2024-02-01T10:48:21.533 DEBUG:boto:Token: None 2024-02-01T10:48:21.533 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:21 GMT /vclfcn-61/obj6?versionId=null 2024-02-01T10:48:21.534 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:c7p+TGsKqwG//bsFhcU4/BkrOHI= 2024-02-01T10:48:21.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:c7p+TGsKqwG//bsFhcU4/BkrOHI=', 'Content-Length': '0'} 2024-02-01T10:48:21.536 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002922a2891d2fc417-0065bb76f5-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:21.536 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-02-01T10:48:21.537 DEBUG:boto:path=/vclfcn-61/obj6 2024-02-01T10:48:21.537 DEBUG:boto:auth_path=/vclfcn-61/obj6 2024-02-01T10:48:21.537 DEBUG:boto:path=/vclfcn-61/obj6?versionId=null 2024-02-01T10:48:21.537 DEBUG:boto:auth_path=/vclfcn-61/obj6?versionId=null 2024-02-01T10:48:21.537 DEBUG:boto:Method: GET 2024-02-01T10:48:21.537 DEBUG:boto:Path: /vclfcn-61/obj6?versionId=null 2024-02-01T10:48:21.537 DEBUG:boto:Data: 2024-02-01T10:48:21.537 DEBUG:boto:Headers: {} 2024-02-01T10:48:21.537 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:21.537 DEBUG:boto:Port: 8001 2024-02-01T10:48:21.537 DEBUG:boto:Params: {} 2024-02-01T10:48:21.537 DEBUG:boto:Token: None 2024-02-01T10:48:21.538 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:21 GMT /vclfcn-61/obj6?versionId=null 2024-02-01T10:48:21.538 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8Vjm+mC+sF/D6KWp+b6Ij6blpmY= 2024-02-01T10:48:21.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8Vjm+mC+sF/D6KWp+b6Ij6blpmY=', 'Content-Length': '0'} 2024-02-01T10:48:21.541 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d65a80b44608fe09-0065bb76f5-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:21.541 DEBUG:boto:path=/vclfcn-61/obj6 2024-02-01T10:48:21.541 DEBUG:boto:auth_path=/vclfcn-61/obj6 2024-02-01T10:48:21.541 DEBUG:boto:path=/vclfcn-61/obj6?versionId=null 2024-02-01T10:48:21.541 DEBUG:boto:auth_path=/vclfcn-61/obj6?versionId=null 2024-02-01T10:48:21.541 DEBUG:boto:Method: GET 2024-02-01T10:48:21.541 DEBUG:boto:Path: /vclfcn-61/obj6?versionId=null 2024-02-01T10:48:21.541 DEBUG:boto:Data: 2024-02-01T10:48:21.542 DEBUG:boto:Headers: {} 2024-02-01T10:48:21.542 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:21.542 DEBUG:boto:Port: 8000 2024-02-01T10:48:21.542 DEBUG:boto:Params: {} 2024-02-01T10:48:21.542 DEBUG:boto:Token: None 2024-02-01T10:48:21.542 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:21 GMT /vclfcn-61/obj6?versionId=null 2024-02-01T10:48:21.542 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8Vjm+mC+sF/D6KWp+b6Ij6blpmY= 2024-02-01T10:48:21.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8Vjm+mC+sF/D6KWp+b6Ij6blpmY=', 'Content-Length': '0'} 2024-02-01T10:48:21.545 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005bbed1feaa9f956a-0065bb76f5-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:21.588 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:21.588 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:21.588 DEBUG:boto:path=/vclfcn-61/?prefix=obj6 2024-02-01T10:48:21.589 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj6 2024-02-01T10:48:21.589 DEBUG:boto:Method: GET 2024-02-01T10:48:21.589 DEBUG:boto:Path: /vclfcn-61/?prefix=obj6 2024-02-01T10:48:21.589 DEBUG:boto:Data: 2024-02-01T10:48:21.589 DEBUG:boto:Headers: {} 2024-02-01T10:48:21.589 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:21.589 DEBUG:boto:Port: 8001 2024-02-01T10:48:21.589 DEBUG:boto:Params: {} 2024-02-01T10:48:21.589 DEBUG:boto:Token: None 2024-02-01T10:48:21.589 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:21 GMT /vclfcn-61/ 2024-02-01T10:48:21.590 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZhinEpH8inHnaTrD74abEQSbAjw= 2024-02-01T10:48:21.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZhinEpH8inHnaTrD74abEQSbAjw=', 'Content-Length': '0'} 2024-02-01T10:48:21.594 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000016b541f3512503d4-0065bb76f5-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:21.594 DEBUG:boto:b'vclfcn-61obj61000falseobj62024-02-01T10:47:30.440Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:21.595 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:21.595 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:21.596 DEBUG:boto:path=/vclfcn-61/?prefix=obj6 2024-02-01T10:48:21.596 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj6 2024-02-01T10:48:21.596 DEBUG:boto:Method: GET 2024-02-01T10:48:21.596 DEBUG:boto:Path: /vclfcn-61/?prefix=obj6 2024-02-01T10:48:21.596 DEBUG:boto:Data: 2024-02-01T10:48:21.596 DEBUG:boto:Headers: {} 2024-02-01T10:48:21.596 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:21.596 DEBUG:boto:Port: 8000 2024-02-01T10:48:21.596 DEBUG:boto:Params: {} 2024-02-01T10:48:21.596 DEBUG:boto:Token: None 2024-02-01T10:48:21.597 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:21 GMT /vclfcn-61/ 2024-02-01T10:48:21.597 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZhinEpH8inHnaTrD74abEQSbAjw= 2024-02-01T10:48:21.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZhinEpH8inHnaTrD74abEQSbAjw=', 'Content-Length': '0'} 2024-02-01T10:48:21.601 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015340682405f009f-0065bb76f5-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:21.601 DEBUG:boto:b'vclfcn-61obj61000falseobj62024-02-01T10:47:30.440Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:21.602 DEBUG:boto:path=/vclfcn-61/obj6 2024-02-01T10:48:21.602 DEBUG:boto:auth_path=/vclfcn-61/obj6 2024-02-01T10:48:21.602 DEBUG:boto:Method: HEAD 2024-02-01T10:48:21.602 DEBUG:boto:Path: /vclfcn-61/obj6 2024-02-01T10:48:21.602 DEBUG:boto:Data: 2024-02-01T10:48:21.602 DEBUG:boto:Headers: {} 2024-02-01T10:48:21.602 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:21.603 DEBUG:boto:Port: 8001 2024-02-01T10:48:21.603 DEBUG:boto:Params: {} 2024-02-01T10:48:21.603 DEBUG:boto:Token: None 2024-02-01T10:48:21.603 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:21 GMT /vclfcn-61/obj6 2024-02-01T10:48:21.603 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:UNrPvjDBiGoUKIE50X68eQqfIHw= 2024-02-01T10:48:21.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:UNrPvjDBiGoUKIE50X68eQqfIHw=', 'Content-Length': '0'} 2024-02-01T10:48:21.606 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fef81479a63430ef-0065bb76f5-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:21.606 DEBUG:boto:path=/vclfcn-61/obj6 2024-02-01T10:48:21.606 DEBUG:boto:auth_path=/vclfcn-61/obj6 2024-02-01T10:48:21.606 DEBUG:boto:Method: HEAD 2024-02-01T10:48:21.606 DEBUG:boto:Path: /vclfcn-61/obj6 2024-02-01T10:48:21.606 DEBUG:boto:Data: 2024-02-01T10:48:21.607 DEBUG:boto:Headers: {} 2024-02-01T10:48:21.607 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:21.607 DEBUG:boto:Port: 8000 2024-02-01T10:48:21.607 DEBUG:boto:Params: {} 2024-02-01T10:48:21.608 DEBUG:boto:Token: None 2024-02-01T10:48:21.608 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:21 GMT /vclfcn-61/obj6 2024-02-01T10:48:21.608 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:UNrPvjDBiGoUKIE50X68eQqfIHw= 2024-02-01T10:48:21.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:UNrPvjDBiGoUKIE50X68eQqfIHw=', 'Content-Length': '0'} 2024-02-01T10:48:21.611 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000099c3e376b0080eaa-0065bb76f5-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:21.611 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-02-01T10:48:21.612 DEBUG:boto:path=/vclfcn-61/obj6 2024-02-01T10:48:21.612 DEBUG:boto:auth_path=/vclfcn-61/obj6 2024-02-01T10:48:21.612 DEBUG:boto:Method: GET 2024-02-01T10:48:21.612 DEBUG:boto:Path: /vclfcn-61/obj6 2024-02-01T10:48:21.612 DEBUG:boto:Data: 2024-02-01T10:48:21.612 DEBUG:boto:Headers: {} 2024-02-01T10:48:21.612 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:21.613 DEBUG:boto:Port: 8001 2024-02-01T10:48:21.613 DEBUG:boto:Params: {} 2024-02-01T10:48:21.613 DEBUG:boto:Token: None 2024-02-01T10:48:21.613 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:21 GMT /vclfcn-61/obj6 2024-02-01T10:48:21.613 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sZRNrHwgFD+nF/akuyCs3EDv6Vw= 2024-02-01T10:48:21.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sZRNrHwgFD+nF/akuyCs3EDv6Vw=', 'Content-Length': '0'} 2024-02-01T10:48:21.616 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000040bc911334c17b6-0065bb76f5-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:21.656 DEBUG:boto:path=/vclfcn-61/obj6 2024-02-01T10:48:21.656 DEBUG:boto:auth_path=/vclfcn-61/obj6 2024-02-01T10:48:21.657 DEBUG:boto:Method: GET 2024-02-01T10:48:21.657 DEBUG:boto:Path: /vclfcn-61/obj6 2024-02-01T10:48:21.657 DEBUG:boto:Data: 2024-02-01T10:48:21.657 DEBUG:boto:Headers: {} 2024-02-01T10:48:21.657 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:21.657 DEBUG:boto:Port: 8000 2024-02-01T10:48:21.657 DEBUG:boto:Params: {} 2024-02-01T10:48:21.657 DEBUG:boto:Token: None 2024-02-01T10:48:21.657 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:21 GMT /vclfcn-61/obj6 2024-02-01T10:48:21.658 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sZRNrHwgFD+nF/akuyCs3EDv6Vw= 2024-02-01T10:48:21.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sZRNrHwgFD+nF/akuyCs3EDv6Vw=', 'Content-Length': '0'} 2024-02-01T10:48:21.660 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d079e3572294a27a-0065bb76f5-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:21.704 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:21.704 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:21.705 DEBUG:boto:path=/vclfcn-61/?prefix=obj6 2024-02-01T10:48:21.705 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj6 2024-02-01T10:48:21.705 DEBUG:boto:Method: GET 2024-02-01T10:48:21.705 DEBUG:boto:Path: /vclfcn-61/?prefix=obj6 2024-02-01T10:48:21.705 DEBUG:boto:Data: 2024-02-01T10:48:21.705 DEBUG:boto:Headers: {} 2024-02-01T10:48:21.705 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:21.705 DEBUG:boto:Port: 8001 2024-02-01T10:48:21.705 DEBUG:boto:Params: {} 2024-02-01T10:48:21.706 DEBUG:boto:Token: None 2024-02-01T10:48:21.706 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:21 GMT /vclfcn-61/ 2024-02-01T10:48:21.706 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZhinEpH8inHnaTrD74abEQSbAjw= 2024-02-01T10:48:21.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZhinEpH8inHnaTrD74abEQSbAjw=', 'Content-Length': '0'} 2024-02-01T10:48:21.711 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004abe615086d5a608-0065bb76f5-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:21.711 DEBUG:boto:b'vclfcn-61obj61000falseobj62024-02-01T10:47:30.440Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:21.711 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:21.712 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:21.712 DEBUG:boto:path=/vclfcn-61/?prefix=obj6 2024-02-01T10:48:21.712 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj6 2024-02-01T10:48:21.712 DEBUG:boto:Method: GET 2024-02-01T10:48:21.712 DEBUG:boto:Path: /vclfcn-61/?prefix=obj6 2024-02-01T10:48:21.712 DEBUG:boto:Data: 2024-02-01T10:48:21.712 DEBUG:boto:Headers: {} 2024-02-01T10:48:21.712 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:21.712 DEBUG:boto:Port: 8000 2024-02-01T10:48:21.712 DEBUG:boto:Params: {} 2024-02-01T10:48:21.713 DEBUG:boto:Token: None 2024-02-01T10:48:21.713 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:21 GMT /vclfcn-61/ 2024-02-01T10:48:21.713 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZhinEpH8inHnaTrD74abEQSbAjw= 2024-02-01T10:48:21.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZhinEpH8inHnaTrD74abEQSbAjw=', 'Content-Length': '0'} 2024-02-01T10:48:21.718 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082454ec29f589ce3-0065bb76f5-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:21.760 DEBUG:boto:b'vclfcn-61obj61000falseobj62024-02-01T10:47:30.440Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:21.760 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-02-01T10:48:21.760 DEBUG:boto:path=/vclfcn-61/obj7 2024-02-01T10:48:21.761 DEBUG:boto:auth_path=/vclfcn-61/obj7 2024-02-01T10:48:21.761 DEBUG:boto:path=/vclfcn-61/obj7?versionId=null 2024-02-01T10:48:21.761 DEBUG:boto:auth_path=/vclfcn-61/obj7?versionId=null 2024-02-01T10:48:21.761 DEBUG:boto:Method: GET 2024-02-01T10:48:21.761 DEBUG:boto:Path: /vclfcn-61/obj7?versionId=null 2024-02-01T10:48:21.761 DEBUG:boto:Data: 2024-02-01T10:48:21.761 DEBUG:boto:Headers: {} 2024-02-01T10:48:21.761 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:21.761 DEBUG:boto:Port: 8001 2024-02-01T10:48:21.761 DEBUG:boto:Params: {} 2024-02-01T10:48:21.761 DEBUG:boto:Token: None 2024-02-01T10:48:21.761 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:21 GMT /vclfcn-61/obj7?versionId=null 2024-02-01T10:48:21.762 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:dd9X3cGrwtwdzFpk5LdQJPgAgj8= 2024-02-01T10:48:21.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dd9X3cGrwtwdzFpk5LdQJPgAgj8=', 'Content-Length': '0'} 2024-02-01T10:48:21.764 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002f7fcb9c2022631c-0065bb76f5-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:21.765 DEBUG:boto:path=/vclfcn-61/obj7 2024-02-01T10:48:21.765 DEBUG:boto:auth_path=/vclfcn-61/obj7 2024-02-01T10:48:21.765 DEBUG:boto:path=/vclfcn-61/obj7?versionId=null 2024-02-01T10:48:21.765 DEBUG:boto:auth_path=/vclfcn-61/obj7?versionId=null 2024-02-01T10:48:21.765 DEBUG:boto:Method: GET 2024-02-01T10:48:21.765 DEBUG:boto:Path: /vclfcn-61/obj7?versionId=null 2024-02-01T10:48:21.765 DEBUG:boto:Data: 2024-02-01T10:48:21.765 DEBUG:boto:Headers: {} 2024-02-01T10:48:21.766 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:21.766 DEBUG:boto:Port: 8000 2024-02-01T10:48:21.766 DEBUG:boto:Params: {} 2024-02-01T10:48:21.766 DEBUG:boto:Token: None 2024-02-01T10:48:21.766 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:21 GMT /vclfcn-61/obj7?versionId=null 2024-02-01T10:48:21.766 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:dd9X3cGrwtwdzFpk5LdQJPgAgj8= 2024-02-01T10:48:21.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dd9X3cGrwtwdzFpk5LdQJPgAgj8=', 'Content-Length': '0'} 2024-02-01T10:48:21.770 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000efb5bc2549d35b5b-0065bb76f5-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:21.770 DEBUG:boto:path=/vclfcn-61/obj7 2024-02-01T10:48:21.770 DEBUG:boto:auth_path=/vclfcn-61/obj7 2024-02-01T10:48:21.770 DEBUG:boto:path=/vclfcn-61/obj7?versionId=null 2024-02-01T10:48:21.770 DEBUG:boto:auth_path=/vclfcn-61/obj7?versionId=null 2024-02-01T10:48:21.770 DEBUG:boto:Method: HEAD 2024-02-01T10:48:21.770 DEBUG:boto:Path: /vclfcn-61/obj7?versionId=null 2024-02-01T10:48:21.771 DEBUG:boto:Data: 2024-02-01T10:48:21.771 DEBUG:boto:Headers: {} 2024-02-01T10:48:21.771 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:21.771 DEBUG:boto:Port: 8001 2024-02-01T10:48:21.771 DEBUG:boto:Params: {} 2024-02-01T10:48:21.771 DEBUG:boto:Token: None 2024-02-01T10:48:21.771 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:21 GMT /vclfcn-61/obj7?versionId=null 2024-02-01T10:48:21.771 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sUIGwFbUmMDQP8YsSq3qsbgwr6k= 2024-02-01T10:48:21.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sUIGwFbUmMDQP8YsSq3qsbgwr6k=', 'Content-Length': '0'} 2024-02-01T10:48:21.776 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006be9c789d2b102d3-0065bb76f5-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:21.776 DEBUG:boto:path=/vclfcn-61/obj7 2024-02-01T10:48:21.776 DEBUG:boto:auth_path=/vclfcn-61/obj7 2024-02-01T10:48:21.776 DEBUG:boto:path=/vclfcn-61/obj7?versionId=null 2024-02-01T10:48:21.776 DEBUG:boto:auth_path=/vclfcn-61/obj7?versionId=null 2024-02-01T10:48:21.776 DEBUG:boto:Method: HEAD 2024-02-01T10:48:21.776 DEBUG:boto:Path: /vclfcn-61/obj7?versionId=null 2024-02-01T10:48:21.777 DEBUG:boto:Data: 2024-02-01T10:48:21.777 DEBUG:boto:Headers: {} 2024-02-01T10:48:21.777 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:21.777 DEBUG:boto:Port: 8000 2024-02-01T10:48:21.777 DEBUG:boto:Params: {} 2024-02-01T10:48:21.777 DEBUG:boto:Token: None 2024-02-01T10:48:21.777 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:21 GMT /vclfcn-61/obj7?versionId=null 2024-02-01T10:48:21.777 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sUIGwFbUmMDQP8YsSq3qsbgwr6k= 2024-02-01T10:48:21.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sUIGwFbUmMDQP8YsSq3qsbgwr6k=', 'Content-Length': '0'} 2024-02-01T10:48:21.780 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b0a132328b4b1c76-0065bb76f5-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:21.780 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-02-01T10:48:21.780 DEBUG:boto:path=/vclfcn-61/obj7 2024-02-01T10:48:21.780 DEBUG:boto:auth_path=/vclfcn-61/obj7 2024-02-01T10:48:21.781 DEBUG:boto:path=/vclfcn-61/obj7?versionId=null 2024-02-01T10:48:21.781 DEBUG:boto:auth_path=/vclfcn-61/obj7?versionId=null 2024-02-01T10:48:21.781 DEBUG:boto:Method: GET 2024-02-01T10:48:21.781 DEBUG:boto:Path: /vclfcn-61/obj7?versionId=null 2024-02-01T10:48:21.781 DEBUG:boto:Data: 2024-02-01T10:48:21.781 DEBUG:boto:Headers: {} 2024-02-01T10:48:21.781 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:21.781 DEBUG:boto:Port: 8001 2024-02-01T10:48:21.781 DEBUG:boto:Params: {} 2024-02-01T10:48:21.781 DEBUG:boto:Token: None 2024-02-01T10:48:21.782 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:21 GMT /vclfcn-61/obj7?versionId=null 2024-02-01T10:48:21.782 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:dd9X3cGrwtwdzFpk5LdQJPgAgj8= 2024-02-01T10:48:21.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dd9X3cGrwtwdzFpk5LdQJPgAgj8=', 'Content-Length': '0'} 2024-02-01T10:48:21.784 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006927d69b48570de8-0065bb76f5-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:21.785 DEBUG:boto:path=/vclfcn-61/obj7 2024-02-01T10:48:21.785 DEBUG:boto:auth_path=/vclfcn-61/obj7 2024-02-01T10:48:21.785 DEBUG:boto:path=/vclfcn-61/obj7?versionId=null 2024-02-01T10:48:21.785 DEBUG:boto:auth_path=/vclfcn-61/obj7?versionId=null 2024-02-01T10:48:21.785 DEBUG:boto:Method: GET 2024-02-01T10:48:21.785 DEBUG:boto:Path: /vclfcn-61/obj7?versionId=null 2024-02-01T10:48:21.785 DEBUG:boto:Data: 2024-02-01T10:48:21.785 DEBUG:boto:Headers: {} 2024-02-01T10:48:21.786 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:21.786 DEBUG:boto:Port: 8000 2024-02-01T10:48:21.786 DEBUG:boto:Params: {} 2024-02-01T10:48:21.786 DEBUG:boto:Token: None 2024-02-01T10:48:21.786 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:21 GMT /vclfcn-61/obj7?versionId=null 2024-02-01T10:48:21.786 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:dd9X3cGrwtwdzFpk5LdQJPgAgj8= 2024-02-01T10:48:21.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dd9X3cGrwtwdzFpk5LdQJPgAgj8=', 'Content-Length': '0'} 2024-02-01T10:48:21.789 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000042078b08725492a2-0065bb76f5-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:21.832 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:21.832 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:21.832 DEBUG:boto:path=/vclfcn-61/?prefix=obj7 2024-02-01T10:48:21.832 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj7 2024-02-01T10:48:21.833 DEBUG:boto:Method: GET 2024-02-01T10:48:21.833 DEBUG:boto:Path: /vclfcn-61/?prefix=obj7 2024-02-01T10:48:21.833 DEBUG:boto:Data: 2024-02-01T10:48:21.833 DEBUG:boto:Headers: {} 2024-02-01T10:48:21.833 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:21.833 DEBUG:boto:Port: 8001 2024-02-01T10:48:21.833 DEBUG:boto:Params: {} 2024-02-01T10:48:21.833 DEBUG:boto:Token: None 2024-02-01T10:48:21.833 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:21 GMT /vclfcn-61/ 2024-02-01T10:48:21.834 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZhinEpH8inHnaTrD74abEQSbAjw= 2024-02-01T10:48:21.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZhinEpH8inHnaTrD74abEQSbAjw=', 'Content-Length': '0'} 2024-02-01T10:48:21.839 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012cf867621cb950d-0065bb76f5-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:21.840 DEBUG:boto:b'vclfcn-61obj71000falseobj72024-02-01T10:47:30.492Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:21.840 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:21.840 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:21.840 DEBUG:boto:path=/vclfcn-61/?prefix=obj7 2024-02-01T10:48:21.841 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj7 2024-02-01T10:48:21.841 DEBUG:boto:Method: GET 2024-02-01T10:48:21.841 DEBUG:boto:Path: /vclfcn-61/?prefix=obj7 2024-02-01T10:48:21.841 DEBUG:boto:Data: 2024-02-01T10:48:21.841 DEBUG:boto:Headers: {} 2024-02-01T10:48:21.841 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:21.841 DEBUG:boto:Port: 8000 2024-02-01T10:48:21.841 DEBUG:boto:Params: {} 2024-02-01T10:48:21.841 DEBUG:boto:Token: None 2024-02-01T10:48:21.841 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:21 GMT /vclfcn-61/ 2024-02-01T10:48:21.842 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZhinEpH8inHnaTrD74abEQSbAjw= 2024-02-01T10:48:21.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZhinEpH8inHnaTrD74abEQSbAjw=', 'Content-Length': '0'} 2024-02-01T10:48:21.846 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca2e6195ad5b1596-0065bb76f5-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:21.846 DEBUG:boto:b'vclfcn-61obj71000falseobj72024-02-01T10:47:30.492Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:21.846 DEBUG:boto:path=/vclfcn-61/obj7 2024-02-01T10:48:21.846 DEBUG:boto:auth_path=/vclfcn-61/obj7 2024-02-01T10:48:21.846 DEBUG:boto:Method: HEAD 2024-02-01T10:48:21.847 DEBUG:boto:Path: /vclfcn-61/obj7 2024-02-01T10:48:21.847 DEBUG:boto:Data: 2024-02-01T10:48:21.847 DEBUG:boto:Headers: {} 2024-02-01T10:48:21.847 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:21.847 DEBUG:boto:Port: 8001 2024-02-01T10:48:21.847 DEBUG:boto:Params: {} 2024-02-01T10:48:21.847 DEBUG:boto:Token: None 2024-02-01T10:48:21.847 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:21 GMT /vclfcn-61/obj7 2024-02-01T10:48:21.847 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HKT1QcsGzL7O0p+lEZLlf7cG1SQ= 2024-02-01T10:48:21.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HKT1QcsGzL7O0p+lEZLlf7cG1SQ=', 'Content-Length': '0'} 2024-02-01T10:48:21.850 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c34ce5434d8ee72e-0065bb76f5-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:21.850 DEBUG:boto:path=/vclfcn-61/obj7 2024-02-01T10:48:21.850 DEBUG:boto:auth_path=/vclfcn-61/obj7 2024-02-01T10:48:21.850 DEBUG:boto:Method: HEAD 2024-02-01T10:48:21.851 DEBUG:boto:Path: /vclfcn-61/obj7 2024-02-01T10:48:21.851 DEBUG:boto:Data: 2024-02-01T10:48:21.851 DEBUG:boto:Headers: {} 2024-02-01T10:48:21.851 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:21.851 DEBUG:boto:Port: 8000 2024-02-01T10:48:21.851 DEBUG:boto:Params: {} 2024-02-01T10:48:21.851 DEBUG:boto:Token: None 2024-02-01T10:48:21.851 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:21 GMT /vclfcn-61/obj7 2024-02-01T10:48:21.851 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HKT1QcsGzL7O0p+lEZLlf7cG1SQ= 2024-02-01T10:48:21.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HKT1QcsGzL7O0p+lEZLlf7cG1SQ=', 'Content-Length': '0'} 2024-02-01T10:48:21.854 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000641a69eef3d99b02-0065bb76f5-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:21.855 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-02-01T10:48:21.855 DEBUG:boto:path=/vclfcn-61/obj7 2024-02-01T10:48:21.855 DEBUG:boto:auth_path=/vclfcn-61/obj7 2024-02-01T10:48:21.855 DEBUG:boto:Method: GET 2024-02-01T10:48:21.855 DEBUG:boto:Path: /vclfcn-61/obj7 2024-02-01T10:48:21.855 DEBUG:boto:Data: 2024-02-01T10:48:21.855 DEBUG:boto:Headers: {} 2024-02-01T10:48:21.855 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:21.855 DEBUG:boto:Port: 8001 2024-02-01T10:48:21.855 DEBUG:boto:Params: {} 2024-02-01T10:48:21.856 DEBUG:boto:Token: None 2024-02-01T10:48:21.856 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:21 GMT /vclfcn-61/obj7 2024-02-01T10:48:21.856 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HT93eNIVzMnYaKqE1KF17N/Hgmg= 2024-02-01T10:48:21.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HT93eNIVzMnYaKqE1KF17N/Hgmg=', 'Content-Length': '0'} 2024-02-01T10:48:21.858 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004e2994b15f1d2694-0065bb76f5-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:21.900 DEBUG:boto:path=/vclfcn-61/obj7 2024-02-01T10:48:21.900 DEBUG:boto:auth_path=/vclfcn-61/obj7 2024-02-01T10:48:21.901 DEBUG:boto:Method: GET 2024-02-01T10:48:21.901 DEBUG:boto:Path: /vclfcn-61/obj7 2024-02-01T10:48:21.901 DEBUG:boto:Data: 2024-02-01T10:48:21.901 DEBUG:boto:Headers: {} 2024-02-01T10:48:21.901 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:21.901 DEBUG:boto:Port: 8000 2024-02-01T10:48:21.901 DEBUG:boto:Params: {} 2024-02-01T10:48:21.901 DEBUG:boto:Token: None 2024-02-01T10:48:21.901 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:21 GMT /vclfcn-61/obj7 2024-02-01T10:48:21.902 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HT93eNIVzMnYaKqE1KF17N/Hgmg= 2024-02-01T10:48:21.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HT93eNIVzMnYaKqE1KF17N/Hgmg=', 'Content-Length': '0'} 2024-02-01T10:48:21.905 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000edf44e004c8c93dc-0065bb76f5-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:21.949 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:21.949 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:21.949 DEBUG:boto:path=/vclfcn-61/?prefix=obj7 2024-02-01T10:48:21.949 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj7 2024-02-01T10:48:21.949 DEBUG:boto:Method: GET 2024-02-01T10:48:21.949 DEBUG:boto:Path: /vclfcn-61/?prefix=obj7 2024-02-01T10:48:21.949 DEBUG:boto:Data: 2024-02-01T10:48:21.949 DEBUG:boto:Headers: {} 2024-02-01T10:48:21.949 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:21.949 DEBUG:boto:Port: 8001 2024-02-01T10:48:21.950 DEBUG:boto:Params: {} 2024-02-01T10:48:21.950 DEBUG:boto:Token: None 2024-02-01T10:48:21.950 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:21 GMT /vclfcn-61/ 2024-02-01T10:48:21.950 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZhinEpH8inHnaTrD74abEQSbAjw= 2024-02-01T10:48:21.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZhinEpH8inHnaTrD74abEQSbAjw=', 'Content-Length': '0'} 2024-02-01T10:48:21.954 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000031c6c41a99cef583-0065bb76f5-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:21.955 DEBUG:boto:b'vclfcn-61obj71000falseobj72024-02-01T10:47:30.492Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:21.955 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:21.955 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:21.955 DEBUG:boto:path=/vclfcn-61/?prefix=obj7 2024-02-01T10:48:21.955 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj7 2024-02-01T10:48:21.955 DEBUG:boto:Method: GET 2024-02-01T10:48:21.956 DEBUG:boto:Path: /vclfcn-61/?prefix=obj7 2024-02-01T10:48:21.956 DEBUG:boto:Data: 2024-02-01T10:48:21.956 DEBUG:boto:Headers: {} 2024-02-01T10:48:21.956 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:21.956 DEBUG:boto:Port: 8000 2024-02-01T10:48:21.956 DEBUG:boto:Params: {} 2024-02-01T10:48:21.956 DEBUG:boto:Token: None 2024-02-01T10:48:21.956 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:21 GMT /vclfcn-61/ 2024-02-01T10:48:21.956 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZhinEpH8inHnaTrD74abEQSbAjw= 2024-02-01T10:48:21.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZhinEpH8inHnaTrD74abEQSbAjw=', 'Content-Length': '0'} 2024-02-01T10:48:21.960 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d88c3a605a8ed1d-0065bb76f5-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:22.004 DEBUG:boto:b'vclfcn-61obj71000falseobj72024-02-01T10:47:30.492Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:22.004 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-02-01T10:48:22.005 DEBUG:boto:path=/vclfcn-61/obj8 2024-02-01T10:48:22.005 DEBUG:boto:auth_path=/vclfcn-61/obj8 2024-02-01T10:48:22.005 DEBUG:boto:path=/vclfcn-61/obj8?versionId=null 2024-02-01T10:48:22.005 DEBUG:boto:auth_path=/vclfcn-61/obj8?versionId=null 2024-02-01T10:48:22.005 DEBUG:boto:Method: GET 2024-02-01T10:48:22.005 DEBUG:boto:Path: /vclfcn-61/obj8?versionId=null 2024-02-01T10:48:22.005 DEBUG:boto:Data: 2024-02-01T10:48:22.005 DEBUG:boto:Headers: {} 2024-02-01T10:48:22.005 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:22.005 DEBUG:boto:Port: 8001 2024-02-01T10:48:22.006 DEBUG:boto:Params: {} 2024-02-01T10:48:22.006 DEBUG:boto:Token: None 2024-02-01T10:48:22.006 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:22 GMT /vclfcn-61/obj8?versionId=null 2024-02-01T10:48:22.006 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:dkpFEA/aX+zwh0PgmKwQDCC1++Y= 2024-02-01T10:48:22.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dkpFEA/aX+zwh0PgmKwQDCC1++Y=', 'Content-Length': '0'} 2024-02-01T10:48:22.009 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007f5ec720d14d3d6e-0065bb76f6-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:22.009 DEBUG:boto:path=/vclfcn-61/obj8 2024-02-01T10:48:22.009 DEBUG:boto:auth_path=/vclfcn-61/obj8 2024-02-01T10:48:22.009 DEBUG:boto:path=/vclfcn-61/obj8?versionId=null 2024-02-01T10:48:22.009 DEBUG:boto:auth_path=/vclfcn-61/obj8?versionId=null 2024-02-01T10:48:22.010 DEBUG:boto:Method: GET 2024-02-01T10:48:22.010 DEBUG:boto:Path: /vclfcn-61/obj8?versionId=null 2024-02-01T10:48:22.010 DEBUG:boto:Data: 2024-02-01T10:48:22.010 DEBUG:boto:Headers: {} 2024-02-01T10:48:22.010 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:22.010 DEBUG:boto:Port: 8000 2024-02-01T10:48:22.010 DEBUG:boto:Params: {} 2024-02-01T10:48:22.010 DEBUG:boto:Token: None 2024-02-01T10:48:22.010 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:22 GMT /vclfcn-61/obj8?versionId=null 2024-02-01T10:48:22.010 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:dkpFEA/aX+zwh0PgmKwQDCC1++Y= 2024-02-01T10:48:22.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dkpFEA/aX+zwh0PgmKwQDCC1++Y=', 'Content-Length': '0'} 2024-02-01T10:48:22.013 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a8a179265fae72ca-0065bb76f6-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:22.014 DEBUG:boto:path=/vclfcn-61/obj8 2024-02-01T10:48:22.014 DEBUG:boto:auth_path=/vclfcn-61/obj8 2024-02-01T10:48:22.014 DEBUG:boto:path=/vclfcn-61/obj8?versionId=null 2024-02-01T10:48:22.014 DEBUG:boto:auth_path=/vclfcn-61/obj8?versionId=null 2024-02-01T10:48:22.014 DEBUG:boto:Method: HEAD 2024-02-01T10:48:22.014 DEBUG:boto:Path: /vclfcn-61/obj8?versionId=null 2024-02-01T10:48:22.014 DEBUG:boto:Data: 2024-02-01T10:48:22.014 DEBUG:boto:Headers: {} 2024-02-01T10:48:22.015 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:22.015 DEBUG:boto:Port: 8001 2024-02-01T10:48:22.015 DEBUG:boto:Params: {} 2024-02-01T10:48:22.015 DEBUG:boto:Token: None 2024-02-01T10:48:22.015 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:22 GMT /vclfcn-61/obj8?versionId=null 2024-02-01T10:48:22.015 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Bm1oW8ZNTVMq2BjxLxIHoWuPpOU= 2024-02-01T10:48:22.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Bm1oW8ZNTVMq2BjxLxIHoWuPpOU=', 'Content-Length': '0'} 2024-02-01T10:48:22.018 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001661c1b4fe61f5ed-0065bb76f6-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:22.018 DEBUG:boto:path=/vclfcn-61/obj8 2024-02-01T10:48:22.018 DEBUG:boto:auth_path=/vclfcn-61/obj8 2024-02-01T10:48:22.018 DEBUG:boto:path=/vclfcn-61/obj8?versionId=null 2024-02-01T10:48:22.018 DEBUG:boto:auth_path=/vclfcn-61/obj8?versionId=null 2024-02-01T10:48:22.018 DEBUG:boto:Method: HEAD 2024-02-01T10:48:22.018 DEBUG:boto:Path: /vclfcn-61/obj8?versionId=null 2024-02-01T10:48:22.018 DEBUG:boto:Data: 2024-02-01T10:48:22.019 DEBUG:boto:Headers: {} 2024-02-01T10:48:22.019 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:22.019 DEBUG:boto:Port: 8000 2024-02-01T10:48:22.019 DEBUG:boto:Params: {} 2024-02-01T10:48:22.019 DEBUG:boto:Token: None 2024-02-01T10:48:22.019 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:22 GMT /vclfcn-61/obj8?versionId=null 2024-02-01T10:48:22.019 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Bm1oW8ZNTVMq2BjxLxIHoWuPpOU= 2024-02-01T10:48:22.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Bm1oW8ZNTVMq2BjxLxIHoWuPpOU=', 'Content-Length': '0'} 2024-02-01T10:48:22.022 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000088fa8bcdcdf8e9f2-0065bb76f6-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:22.022 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-02-01T10:48:22.022 DEBUG:boto:path=/vclfcn-61/obj8 2024-02-01T10:48:22.022 DEBUG:boto:auth_path=/vclfcn-61/obj8 2024-02-01T10:48:22.023 DEBUG:boto:path=/vclfcn-61/obj8?versionId=null 2024-02-01T10:48:22.023 DEBUG:boto:auth_path=/vclfcn-61/obj8?versionId=null 2024-02-01T10:48:22.023 DEBUG:boto:Method: GET 2024-02-01T10:48:22.023 DEBUG:boto:Path: /vclfcn-61/obj8?versionId=null 2024-02-01T10:48:22.023 DEBUG:boto:Data: 2024-02-01T10:48:22.023 DEBUG:boto:Headers: {} 2024-02-01T10:48:22.023 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:22.023 DEBUG:boto:Port: 8001 2024-02-01T10:48:22.023 DEBUG:boto:Params: {} 2024-02-01T10:48:22.023 DEBUG:boto:Token: None 2024-02-01T10:48:22.023 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:22 GMT /vclfcn-61/obj8?versionId=null 2024-02-01T10:48:22.024 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:dkpFEA/aX+zwh0PgmKwQDCC1++Y= 2024-02-01T10:48:22.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dkpFEA/aX+zwh0PgmKwQDCC1++Y=', 'Content-Length': '0'} 2024-02-01T10:48:22.027 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000009bfe4fb83a52ed9-0065bb76f6-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:22.027 DEBUG:boto:path=/vclfcn-61/obj8 2024-02-01T10:48:22.027 DEBUG:boto:auth_path=/vclfcn-61/obj8 2024-02-01T10:48:22.027 DEBUG:boto:path=/vclfcn-61/obj8?versionId=null 2024-02-01T10:48:22.027 DEBUG:boto:auth_path=/vclfcn-61/obj8?versionId=null 2024-02-01T10:48:22.027 DEBUG:boto:Method: GET 2024-02-01T10:48:22.027 DEBUG:boto:Path: /vclfcn-61/obj8?versionId=null 2024-02-01T10:48:22.027 DEBUG:boto:Data: 2024-02-01T10:48:22.028 DEBUG:boto:Headers: {} 2024-02-01T10:48:22.028 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:22.028 DEBUG:boto:Port: 8000 2024-02-01T10:48:22.028 DEBUG:boto:Params: {} 2024-02-01T10:48:22.028 DEBUG:boto:Token: None 2024-02-01T10:48:22.028 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:22 GMT /vclfcn-61/obj8?versionId=null 2024-02-01T10:48:22.028 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:dkpFEA/aX+zwh0PgmKwQDCC1++Y= 2024-02-01T10:48:22.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dkpFEA/aX+zwh0PgmKwQDCC1++Y=', 'Content-Length': '0'} 2024-02-01T10:48:22.031 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aac55626582356fc-0065bb76f6-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:22.072 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:22.072 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:22.072 DEBUG:boto:path=/vclfcn-61/?prefix=obj8 2024-02-01T10:48:22.072 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj8 2024-02-01T10:48:22.073 DEBUG:boto:Method: GET 2024-02-01T10:48:22.073 DEBUG:boto:Path: /vclfcn-61/?prefix=obj8 2024-02-01T10:48:22.073 DEBUG:boto:Data: 2024-02-01T10:48:22.073 DEBUG:boto:Headers: {} 2024-02-01T10:48:22.073 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:22.073 DEBUG:boto:Port: 8001 2024-02-01T10:48:22.073 DEBUG:boto:Params: {} 2024-02-01T10:48:22.073 DEBUG:boto:Token: None 2024-02-01T10:48:22.073 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:22 GMT /vclfcn-61/ 2024-02-01T10:48:22.074 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:DrEManxPYhWBnK0wAKjS1z5Q4pQ= 2024-02-01T10:48:22.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:DrEManxPYhWBnK0wAKjS1z5Q4pQ=', 'Content-Length': '0'} 2024-02-01T10:48:22.079 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ca27f194e0525ab-0065bb76f6-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:22.079 DEBUG:boto:b'vclfcn-61obj81000falseobj82024-02-01T10:47:30.507Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:22.080 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:22.080 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:22.080 DEBUG:boto:path=/vclfcn-61/?prefix=obj8 2024-02-01T10:48:22.080 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj8 2024-02-01T10:48:22.080 DEBUG:boto:Method: GET 2024-02-01T10:48:22.080 DEBUG:boto:Path: /vclfcn-61/?prefix=obj8 2024-02-01T10:48:22.080 DEBUG:boto:Data: 2024-02-01T10:48:22.081 DEBUG:boto:Headers: {} 2024-02-01T10:48:22.081 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:22.081 DEBUG:boto:Port: 8000 2024-02-01T10:48:22.081 DEBUG:boto:Params: {} 2024-02-01T10:48:22.081 DEBUG:boto:Token: None 2024-02-01T10:48:22.081 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:22 GMT /vclfcn-61/ 2024-02-01T10:48:22.081 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:DrEManxPYhWBnK0wAKjS1z5Q4pQ= 2024-02-01T10:48:22.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:DrEManxPYhWBnK0wAKjS1z5Q4pQ=', 'Content-Length': '0'} 2024-02-01T10:48:22.086 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa2a3adf6c5935d8-0065bb76f6-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:22.086 DEBUG:boto:b'vclfcn-61obj81000falseobj82024-02-01T10:47:30.507Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:22.086 DEBUG:boto:path=/vclfcn-61/obj8 2024-02-01T10:48:22.086 DEBUG:boto:auth_path=/vclfcn-61/obj8 2024-02-01T10:48:22.087 DEBUG:boto:Method: HEAD 2024-02-01T10:48:22.087 DEBUG:boto:Path: /vclfcn-61/obj8 2024-02-01T10:48:22.087 DEBUG:boto:Data: 2024-02-01T10:48:22.087 DEBUG:boto:Headers: {} 2024-02-01T10:48:22.087 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:22.087 DEBUG:boto:Port: 8001 2024-02-01T10:48:22.087 DEBUG:boto:Params: {} 2024-02-01T10:48:22.087 DEBUG:boto:Token: None 2024-02-01T10:48:22.087 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:22 GMT /vclfcn-61/obj8 2024-02-01T10:48:22.088 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:uUM6F9sswZ67jQCje/bxkiSxTyg= 2024-02-01T10:48:22.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:uUM6F9sswZ67jQCje/bxkiSxTyg=', 'Content-Length': '0'} 2024-02-01T10:48:22.090 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a4486481885e07a2-0065bb76f6-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:22.090 DEBUG:boto:path=/vclfcn-61/obj8 2024-02-01T10:48:22.091 DEBUG:boto:auth_path=/vclfcn-61/obj8 2024-02-01T10:48:22.091 DEBUG:boto:Method: HEAD 2024-02-01T10:48:22.091 DEBUG:boto:Path: /vclfcn-61/obj8 2024-02-01T10:48:22.091 DEBUG:boto:Data: 2024-02-01T10:48:22.091 DEBUG:boto:Headers: {} 2024-02-01T10:48:22.091 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:22.091 DEBUG:boto:Port: 8000 2024-02-01T10:48:22.091 DEBUG:boto:Params: {} 2024-02-01T10:48:22.091 DEBUG:boto:Token: None 2024-02-01T10:48:22.091 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:22 GMT /vclfcn-61/obj8 2024-02-01T10:48:22.092 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:uUM6F9sswZ67jQCje/bxkiSxTyg= 2024-02-01T10:48:22.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:uUM6F9sswZ67jQCje/bxkiSxTyg=', 'Content-Length': '0'} 2024-02-01T10:48:22.094 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000602eca583f133056-0065bb76f6-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:22.095 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-02-01T10:48:22.095 DEBUG:boto:path=/vclfcn-61/obj8 2024-02-01T10:48:22.095 DEBUG:boto:auth_path=/vclfcn-61/obj8 2024-02-01T10:48:22.095 DEBUG:boto:Method: GET 2024-02-01T10:48:22.095 DEBUG:boto:Path: /vclfcn-61/obj8 2024-02-01T10:48:22.095 DEBUG:boto:Data: 2024-02-01T10:48:22.095 DEBUG:boto:Headers: {} 2024-02-01T10:48:22.095 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:22.096 DEBUG:boto:Port: 8001 2024-02-01T10:48:22.096 DEBUG:boto:Params: {} 2024-02-01T10:48:22.096 DEBUG:boto:Token: None 2024-02-01T10:48:22.096 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:22 GMT /vclfcn-61/obj8 2024-02-01T10:48:22.096 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:76OK/I0U6y4cmqnLvWGNs19bx2s= 2024-02-01T10:48:22.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:76OK/I0U6y4cmqnLvWGNs19bx2s=', 'Content-Length': '0'} 2024-02-01T10:48:22.099 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000295ad3ea6eb2ef00-0065bb76f6-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:22.140 DEBUG:boto:path=/vclfcn-61/obj8 2024-02-01T10:48:22.140 DEBUG:boto:auth_path=/vclfcn-61/obj8 2024-02-01T10:48:22.140 DEBUG:boto:Method: GET 2024-02-01T10:48:22.141 DEBUG:boto:Path: /vclfcn-61/obj8 2024-02-01T10:48:22.141 DEBUG:boto:Data: 2024-02-01T10:48:22.141 DEBUG:boto:Headers: {} 2024-02-01T10:48:22.141 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:22.141 DEBUG:boto:Port: 8000 2024-02-01T10:48:22.141 DEBUG:boto:Params: {} 2024-02-01T10:48:22.141 DEBUG:boto:Token: None 2024-02-01T10:48:22.141 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:22 GMT /vclfcn-61/obj8 2024-02-01T10:48:22.141 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:76OK/I0U6y4cmqnLvWGNs19bx2s= 2024-02-01T10:48:22.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:76OK/I0U6y4cmqnLvWGNs19bx2s=', 'Content-Length': '0'} 2024-02-01T10:48:22.144 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000038c4f611b2db14bb-0065bb76f6-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:22.188 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:22.188 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:22.188 DEBUG:boto:path=/vclfcn-61/?prefix=obj8 2024-02-01T10:48:22.189 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj8 2024-02-01T10:48:22.189 DEBUG:boto:Method: GET 2024-02-01T10:48:22.189 DEBUG:boto:Path: /vclfcn-61/?prefix=obj8 2024-02-01T10:48:22.189 DEBUG:boto:Data: 2024-02-01T10:48:22.189 DEBUG:boto:Headers: {} 2024-02-01T10:48:22.189 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:22.189 DEBUG:boto:Port: 8001 2024-02-01T10:48:22.189 DEBUG:boto:Params: {} 2024-02-01T10:48:22.189 DEBUG:boto:Token: None 2024-02-01T10:48:22.190 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:22 GMT /vclfcn-61/ 2024-02-01T10:48:22.190 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:DrEManxPYhWBnK0wAKjS1z5Q4pQ= 2024-02-01T10:48:22.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:DrEManxPYhWBnK0wAKjS1z5Q4pQ=', 'Content-Length': '0'} 2024-02-01T10:48:22.194 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee2b3711f5afe7a3-0065bb76f6-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:22.194 DEBUG:boto:b'vclfcn-61obj81000falseobj82024-02-01T10:47:30.507Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:22.195 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:22.195 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:22.195 DEBUG:boto:path=/vclfcn-61/?prefix=obj8 2024-02-01T10:48:22.195 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj8 2024-02-01T10:48:22.196 DEBUG:boto:Method: GET 2024-02-01T10:48:22.196 DEBUG:boto:Path: /vclfcn-61/?prefix=obj8 2024-02-01T10:48:22.196 DEBUG:boto:Data: 2024-02-01T10:48:22.196 DEBUG:boto:Headers: {} 2024-02-01T10:48:22.196 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:22.196 DEBUG:boto:Port: 8000 2024-02-01T10:48:22.196 DEBUG:boto:Params: {} 2024-02-01T10:48:22.196 DEBUG:boto:Token: None 2024-02-01T10:48:22.197 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:22 GMT /vclfcn-61/ 2024-02-01T10:48:22.197 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:DrEManxPYhWBnK0wAKjS1z5Q4pQ= 2024-02-01T10:48:22.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:DrEManxPYhWBnK0wAKjS1z5Q4pQ=', 'Content-Length': '0'} 2024-02-01T10:48:22.201 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2f4f70ff540558f-0065bb76f6-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:22.244 DEBUG:boto:b'vclfcn-61obj81000falseobj82024-02-01T10:47:30.507Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:22.245 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-61 zones={test-zone3, test-zone1} 2024-02-01T10:48:22.245 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-61 zones={test-zone3, test-zone2} 2024-02-01T10:48:22.245 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:22.245 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:22.245 DEBUG:boto:Method: HEAD 2024-02-01T10:48:22.245 DEBUG:boto:Path: /vclfcn-61/ 2024-02-01T10:48:22.245 DEBUG:boto:Data: 2024-02-01T10:48:22.245 DEBUG:boto:Headers: {} 2024-02-01T10:48:22.245 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:22.246 DEBUG:boto:Port: 8001 2024-02-01T10:48:22.246 DEBUG:boto:Params: {} 2024-02-01T10:48:22.246 DEBUG:boto:Token: None 2024-02-01T10:48:22.246 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:22 GMT /vclfcn-61/ 2024-02-01T10:48:22.247 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:c4ts0UIAdJyk7wFX67esUZWwAy8= 2024-02-01T10:48:22.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:c4ts0UIAdJyk7wFX67esUZWwAy8=', 'Content-Length': '0'} 2024-02-01T10:48:22.251 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ba640e6d1450bee2-0065bb76f6-5762-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:22.251 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:22.251 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:22.252 DEBUG:boto:Method: HEAD 2024-02-01T10:48:22.252 DEBUG:boto:Path: /vclfcn-61/ 2024-02-01T10:48:22.252 DEBUG:boto:Data: 2024-02-01T10:48:22.252 DEBUG:boto:Headers: {} 2024-02-01T10:48:22.252 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:22.252 DEBUG:boto:Port: 8000 2024-02-01T10:48:22.253 DEBUG:boto:Params: {} 2024-02-01T10:48:22.253 DEBUG:boto:Token: None 2024-02-01T10:48:22.253 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:22 GMT /vclfcn-61/ 2024-02-01T10:48:22.253 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:c4ts0UIAdJyk7wFX67esUZWwAy8= 2024-02-01T10:48:22.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:c4ts0UIAdJyk7wFX67esUZWwAy8=', 'Content-Length': '0'} 2024-02-01T10:48:22.257 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005cddc2af52bb7b15-0065bb76f6-5350-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:22.258 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:48:22.258 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:22.258 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:22.258 DEBUG:boto:path=/vclfcn-61/?versions&max-keys=999 2024-02-01T10:48:22.258 DEBUG:boto:auth_path=/vclfcn-61/?versions&max-keys=999 2024-02-01T10:48:22.259 DEBUG:boto:Method: GET 2024-02-01T10:48:22.259 DEBUG:boto:Path: /vclfcn-61/?versions&max-keys=999 2024-02-01T10:48:22.259 DEBUG:boto:Data: 2024-02-01T10:48:22.259 DEBUG:boto:Headers: {} 2024-02-01T10:48:22.259 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:22.259 DEBUG:boto:Port: 8001 2024-02-01T10:48:22.260 DEBUG:boto:Params: {} 2024-02-01T10:48:22.260 DEBUG:boto:Token: None 2024-02-01T10:48:22.260 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:22 GMT /vclfcn-61/?versions 2024-02-01T10:48:22.260 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NY5cDQ+KvtIdkvhOZHp8IRMZnsQ= 2024-02-01T10:48:22.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NY5cDQ+KvtIdkvhOZHp8IRMZnsQ=', 'Content-Length': '0'} 2024-02-01T10:48:22.266 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001077e6c29dbe8e43-0065bb76f6-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:22.267 DEBUG:boto:b'vclfcn-61999falseobj1nulltrue2024-02-01T10:46:46.359Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:46:46.380Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:46:46.433Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:46:46.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-02-01T10:47:30.425Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-02-01T10:47:30.440Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-02-01T10:47:30.492Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-02-01T10:47:30.507Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:22.268 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-02-01T10:48:22.268 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-02-01T10:48:22.269 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-02-01T10:48:22.269 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-02-01T10:48:22.269 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-02-01T10:48:22.269 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-02-01T10:48:22.269 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-02-01T10:48:22.269 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-02-01T10:48:22.269 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:48:22.270 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:22.270 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:22.270 DEBUG:boto:path=/vclfcn-61/?versions&max-keys=999 2024-02-01T10:48:22.270 DEBUG:boto:auth_path=/vclfcn-61/?versions&max-keys=999 2024-02-01T10:48:22.270 DEBUG:boto:Method: GET 2024-02-01T10:48:22.270 DEBUG:boto:Path: /vclfcn-61/?versions&max-keys=999 2024-02-01T10:48:22.270 DEBUG:boto:Data: 2024-02-01T10:48:22.270 DEBUG:boto:Headers: {} 2024-02-01T10:48:22.271 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:22.271 DEBUG:boto:Port: 8000 2024-02-01T10:48:22.271 DEBUG:boto:Params: {} 2024-02-01T10:48:22.271 DEBUG:boto:Token: None 2024-02-01T10:48:22.271 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:22 GMT /vclfcn-61/?versions 2024-02-01T10:48:22.271 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NY5cDQ+KvtIdkvhOZHp8IRMZnsQ= 2024-02-01T10:48:22.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NY5cDQ+KvtIdkvhOZHp8IRMZnsQ=', 'Content-Length': '0'} 2024-02-01T10:48:22.276 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003068274bca37f4eb-0065bb76f6-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:22.276 DEBUG:boto:b'vclfcn-61999falseobj1nulltrue2024-02-01T10:46:46.359Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:46:46.380Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:46:46.433Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:46:46.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-02-01T10:47:30.425Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-02-01T10:47:30.440Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-02-01T10:47:30.492Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-02-01T10:47:30.507Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:22.276 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-02-01T10:48:22.277 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-02-01T10:48:22.277 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-02-01T10:48:22.277 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-02-01T10:48:22.277 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-02-01T10:48:22.277 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-02-01T10:48:22.277 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-02-01T10:48:22.277 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-02-01T10:48:22.277 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:22.277 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:22.277 DEBUG:boto:path=/vclfcn-61/?versions&max-keys=999 2024-02-01T10:48:22.278 DEBUG:boto:auth_path=/vclfcn-61/?versions&max-keys=999 2024-02-01T10:48:22.278 DEBUG:boto:Method: GET 2024-02-01T10:48:22.278 DEBUG:boto:Path: /vclfcn-61/?versions&max-keys=999 2024-02-01T10:48:22.278 DEBUG:boto:Data: 2024-02-01T10:48:22.278 DEBUG:boto:Headers: {} 2024-02-01T10:48:22.278 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:22.278 DEBUG:boto:Port: 8001 2024-02-01T10:48:22.278 DEBUG:boto:Params: {} 2024-02-01T10:48:22.278 DEBUG:boto:Token: None 2024-02-01T10:48:22.278 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:22 GMT /vclfcn-61/?versions 2024-02-01T10:48:22.279 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NY5cDQ+KvtIdkvhOZHp8IRMZnsQ= 2024-02-01T10:48:22.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NY5cDQ+KvtIdkvhOZHp8IRMZnsQ=', 'Content-Length': '0'} 2024-02-01T10:48:22.283 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000645661b134f77f53-0065bb76f6-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:22.283 DEBUG:boto:b'vclfcn-61999falseobj1nulltrue2024-02-01T10:46:46.359Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:46:46.380Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:46:46.433Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:46:46.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-02-01T10:47:30.425Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-02-01T10:47:30.440Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-02-01T10:47:30.492Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-02-01T10:47:30.507Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:22.284 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:22.284 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:22.284 DEBUG:boto:path=/vclfcn-61/?versions&max-keys=999 2024-02-01T10:48:22.285 DEBUG:boto:auth_path=/vclfcn-61/?versions&max-keys=999 2024-02-01T10:48:22.285 DEBUG:boto:Method: GET 2024-02-01T10:48:22.285 DEBUG:boto:Path: /vclfcn-61/?versions&max-keys=999 2024-02-01T10:48:22.285 DEBUG:boto:Data: 2024-02-01T10:48:22.286 DEBUG:boto:Headers: {} 2024-02-01T10:48:22.286 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:22.286 DEBUG:boto:Port: 8000 2024-02-01T10:48:22.286 DEBUG:boto:Params: {} 2024-02-01T10:48:22.286 DEBUG:boto:Token: None 2024-02-01T10:48:22.287 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:22 GMT /vclfcn-61/?versions 2024-02-01T10:48:22.287 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NY5cDQ+KvtIdkvhOZHp8IRMZnsQ= 2024-02-01T10:48:22.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NY5cDQ+KvtIdkvhOZHp8IRMZnsQ=', 'Content-Length': '0'} 2024-02-01T10:48:22.292 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018099bc8dd7f3d68-0065bb76f6-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:22.292 DEBUG:boto:b'vclfcn-61999falseobj1nulltrue2024-02-01T10:46:46.359Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:46:46.380Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:46:46.433Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:46:46.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-02-01T10:47:30.425Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-02-01T10:47:30.440Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-02-01T10:47:30.492Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-02-01T10:47:30.507Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:22.292 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:48:22.293 DEBUG:boto:path=/vclfcn-61/obj1 2024-02-01T10:48:22.293 DEBUG:boto:auth_path=/vclfcn-61/obj1 2024-02-01T10:48:22.293 DEBUG:boto:path=/vclfcn-61/obj1?versionId=null 2024-02-01T10:48:22.293 DEBUG:boto:auth_path=/vclfcn-61/obj1?versionId=null 2024-02-01T10:48:22.293 DEBUG:boto:Method: GET 2024-02-01T10:48:22.293 DEBUG:boto:Path: /vclfcn-61/obj1?versionId=null 2024-02-01T10:48:22.293 DEBUG:boto:Data: 2024-02-01T10:48:22.293 DEBUG:boto:Headers: {} 2024-02-01T10:48:22.293 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:22.294 DEBUG:boto:Port: 8001 2024-02-01T10:48:22.294 DEBUG:boto:Params: {} 2024-02-01T10:48:22.294 DEBUG:boto:Token: None 2024-02-01T10:48:22.294 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:22 GMT /vclfcn-61/obj1?versionId=null 2024-02-01T10:48:22.294 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:uAPstWZF+3CWO085Yvr0Y9QDTCI= 2024-02-01T10:48:22.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:uAPstWZF+3CWO085Yvr0Y9QDTCI=', 'Content-Length': '0'} 2024-02-01T10:48:22.297 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a7cb9654aec96b3a-0065bb76f6-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:22.340 DEBUG:boto:path=/vclfcn-61/obj1 2024-02-01T10:48:22.340 DEBUG:boto:auth_path=/vclfcn-61/obj1 2024-02-01T10:48:22.340 DEBUG:boto:path=/vclfcn-61/obj1?versionId=null 2024-02-01T10:48:22.340 DEBUG:boto:auth_path=/vclfcn-61/obj1?versionId=null 2024-02-01T10:48:22.340 DEBUG:boto:Method: GET 2024-02-01T10:48:22.341 DEBUG:boto:Path: /vclfcn-61/obj1?versionId=null 2024-02-01T10:48:22.341 DEBUG:boto:Data: 2024-02-01T10:48:22.341 DEBUG:boto:Headers: {} 2024-02-01T10:48:22.341 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:22.341 DEBUG:boto:Port: 8000 2024-02-01T10:48:22.341 DEBUG:boto:Params: {} 2024-02-01T10:48:22.341 DEBUG:boto:Token: None 2024-02-01T10:48:22.341 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:22 GMT /vclfcn-61/obj1?versionId=null 2024-02-01T10:48:22.341 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:uAPstWZF+3CWO085Yvr0Y9QDTCI= 2024-02-01T10:48:22.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:uAPstWZF+3CWO085Yvr0Y9QDTCI=', 'Content-Length': '0'} 2024-02-01T10:48:22.344 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000070eb6fa19e2fa540-0065bb76f6-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:22.345 DEBUG:boto:path=/vclfcn-61/obj1 2024-02-01T10:48:22.345 DEBUG:boto:auth_path=/vclfcn-61/obj1 2024-02-01T10:48:22.345 DEBUG:boto:path=/vclfcn-61/obj1?versionId=null 2024-02-01T10:48:22.345 DEBUG:boto:auth_path=/vclfcn-61/obj1?versionId=null 2024-02-01T10:48:22.345 DEBUG:boto:Method: HEAD 2024-02-01T10:48:22.345 DEBUG:boto:Path: /vclfcn-61/obj1?versionId=null 2024-02-01T10:48:22.345 DEBUG:boto:Data: 2024-02-01T10:48:22.345 DEBUG:boto:Headers: {} 2024-02-01T10:48:22.345 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:22.346 DEBUG:boto:Port: 8001 2024-02-01T10:48:22.346 DEBUG:boto:Params: {} 2024-02-01T10:48:22.346 DEBUG:boto:Token: None 2024-02-01T10:48:22.346 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:22 GMT /vclfcn-61/obj1?versionId=null 2024-02-01T10:48:22.346 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YvHOyIdMWWUkVeu7EiKu6H4DDb8= 2024-02-01T10:48:22.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YvHOyIdMWWUkVeu7EiKu6H4DDb8=', 'Content-Length': '0'} 2024-02-01T10:48:22.349 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c2fcc8deb7df444f-0065bb76f6-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:22.349 DEBUG:boto:path=/vclfcn-61/obj1 2024-02-01T10:48:22.349 DEBUG:boto:auth_path=/vclfcn-61/obj1 2024-02-01T10:48:22.349 DEBUG:boto:path=/vclfcn-61/obj1?versionId=null 2024-02-01T10:48:22.349 DEBUG:boto:auth_path=/vclfcn-61/obj1?versionId=null 2024-02-01T10:48:22.349 DEBUG:boto:Method: HEAD 2024-02-01T10:48:22.350 DEBUG:boto:Path: /vclfcn-61/obj1?versionId=null 2024-02-01T10:48:22.350 DEBUG:boto:Data: 2024-02-01T10:48:22.350 DEBUG:boto:Headers: {} 2024-02-01T10:48:22.350 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:22.350 DEBUG:boto:Port: 8000 2024-02-01T10:48:22.350 DEBUG:boto:Params: {} 2024-02-01T10:48:22.350 DEBUG:boto:Token: None 2024-02-01T10:48:22.350 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:22 GMT /vclfcn-61/obj1?versionId=null 2024-02-01T10:48:22.350 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YvHOyIdMWWUkVeu7EiKu6H4DDb8= 2024-02-01T10:48:22.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YvHOyIdMWWUkVeu7EiKu6H4DDb8=', 'Content-Length': '0'} 2024-02-01T10:48:22.353 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000067c46049fc0b6612-0065bb76f6-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:22.353 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:48:22.354 DEBUG:boto:path=/vclfcn-61/obj1 2024-02-01T10:48:22.354 DEBUG:boto:auth_path=/vclfcn-61/obj1 2024-02-01T10:48:22.354 DEBUG:boto:path=/vclfcn-61/obj1?versionId=null 2024-02-01T10:48:22.354 DEBUG:boto:auth_path=/vclfcn-61/obj1?versionId=null 2024-02-01T10:48:22.354 DEBUG:boto:Method: GET 2024-02-01T10:48:22.354 DEBUG:boto:Path: /vclfcn-61/obj1?versionId=null 2024-02-01T10:48:22.354 DEBUG:boto:Data: 2024-02-01T10:48:22.354 DEBUG:boto:Headers: {} 2024-02-01T10:48:22.354 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:22.354 DEBUG:boto:Port: 8001 2024-02-01T10:48:22.355 DEBUG:boto:Params: {} 2024-02-01T10:48:22.355 DEBUG:boto:Token: None 2024-02-01T10:48:22.355 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:22 GMT /vclfcn-61/obj1?versionId=null 2024-02-01T10:48:22.355 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:uAPstWZF+3CWO085Yvr0Y9QDTCI= 2024-02-01T10:48:22.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:uAPstWZF+3CWO085Yvr0Y9QDTCI=', 'Content-Length': '0'} 2024-02-01T10:48:22.358 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009232bf7c3aaa70fe-0065bb76f6-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:22.358 DEBUG:boto:path=/vclfcn-61/obj1 2024-02-01T10:48:22.358 DEBUG:boto:auth_path=/vclfcn-61/obj1 2024-02-01T10:48:22.358 DEBUG:boto:path=/vclfcn-61/obj1?versionId=null 2024-02-01T10:48:22.358 DEBUG:boto:auth_path=/vclfcn-61/obj1?versionId=null 2024-02-01T10:48:22.359 DEBUG:boto:Method: GET 2024-02-01T10:48:22.359 DEBUG:boto:Path: /vclfcn-61/obj1?versionId=null 2024-02-01T10:48:22.359 DEBUG:boto:Data: 2024-02-01T10:48:22.359 DEBUG:boto:Headers: {} 2024-02-01T10:48:22.359 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:22.359 DEBUG:boto:Port: 8000 2024-02-01T10:48:22.359 DEBUG:boto:Params: {} 2024-02-01T10:48:22.359 DEBUG:boto:Token: None 2024-02-01T10:48:22.359 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:22 GMT /vclfcn-61/obj1?versionId=null 2024-02-01T10:48:22.360 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:uAPstWZF+3CWO085Yvr0Y9QDTCI= 2024-02-01T10:48:22.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:uAPstWZF+3CWO085Yvr0Y9QDTCI=', 'Content-Length': '0'} 2024-02-01T10:48:22.362 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000035e1fd96b8184d37-0065bb76f6-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:22.404 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:22.404 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:22.404 DEBUG:boto:path=/vclfcn-61/?prefix=obj1 2024-02-01T10:48:22.404 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj1 2024-02-01T10:48:22.404 DEBUG:boto:Method: GET 2024-02-01T10:48:22.405 DEBUG:boto:Path: /vclfcn-61/?prefix=obj1 2024-02-01T10:48:22.405 DEBUG:boto:Data: 2024-02-01T10:48:22.405 DEBUG:boto:Headers: {} 2024-02-01T10:48:22.405 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:22.405 DEBUG:boto:Port: 8001 2024-02-01T10:48:22.405 DEBUG:boto:Params: {} 2024-02-01T10:48:22.405 DEBUG:boto:Token: None 2024-02-01T10:48:22.405 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:22 GMT /vclfcn-61/ 2024-02-01T10:48:22.405 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:DrEManxPYhWBnK0wAKjS1z5Q4pQ= 2024-02-01T10:48:22.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:DrEManxPYhWBnK0wAKjS1z5Q4pQ=', 'Content-Length': '0'} 2024-02-01T10:48:22.410 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d97ae85e4272f662-0065bb76f6-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:22.411 DEBUG:boto:b'vclfcn-61obj11000falseobj12024-02-01T10:46:46.359Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:22.411 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:22.411 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:22.411 DEBUG:boto:path=/vclfcn-61/?prefix=obj1 2024-02-01T10:48:22.412 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj1 2024-02-01T10:48:22.412 DEBUG:boto:Method: GET 2024-02-01T10:48:22.412 DEBUG:boto:Path: /vclfcn-61/?prefix=obj1 2024-02-01T10:48:22.412 DEBUG:boto:Data: 2024-02-01T10:48:22.412 DEBUG:boto:Headers: {} 2024-02-01T10:48:22.412 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:22.412 DEBUG:boto:Port: 8000 2024-02-01T10:48:22.412 DEBUG:boto:Params: {} 2024-02-01T10:48:22.412 DEBUG:boto:Token: None 2024-02-01T10:48:22.412 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:22 GMT /vclfcn-61/ 2024-02-01T10:48:22.413 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:DrEManxPYhWBnK0wAKjS1z5Q4pQ= 2024-02-01T10:48:22.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:DrEManxPYhWBnK0wAKjS1z5Q4pQ=', 'Content-Length': '0'} 2024-02-01T10:48:22.416 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d8aeeac3a4b59768-0065bb76f6-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:22.417 DEBUG:boto:b'vclfcn-61obj11000falseobj12024-02-01T10:46:46.359Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:22.417 DEBUG:boto:path=/vclfcn-61/obj1 2024-02-01T10:48:22.417 DEBUG:boto:auth_path=/vclfcn-61/obj1 2024-02-01T10:48:22.417 DEBUG:boto:Method: HEAD 2024-02-01T10:48:22.417 DEBUG:boto:Path: /vclfcn-61/obj1 2024-02-01T10:48:22.417 DEBUG:boto:Data: 2024-02-01T10:48:22.418 DEBUG:boto:Headers: {} 2024-02-01T10:48:22.418 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:22.418 DEBUG:boto:Port: 8001 2024-02-01T10:48:22.418 DEBUG:boto:Params: {} 2024-02-01T10:48:22.418 DEBUG:boto:Token: None 2024-02-01T10:48:22.418 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:22 GMT /vclfcn-61/obj1 2024-02-01T10:48:22.418 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0z7JzHjhMwoR0/v9n95MV9VwJgo= 2024-02-01T10:48:22.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0z7JzHjhMwoR0/v9n95MV9VwJgo=', 'Content-Length': '0'} 2024-02-01T10:48:22.421 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000052e77e61ec9809f6-0065bb76f6-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:22.421 DEBUG:boto:path=/vclfcn-61/obj1 2024-02-01T10:48:22.422 DEBUG:boto:auth_path=/vclfcn-61/obj1 2024-02-01T10:48:22.422 DEBUG:boto:Method: HEAD 2024-02-01T10:48:22.422 DEBUG:boto:Path: /vclfcn-61/obj1 2024-02-01T10:48:22.422 DEBUG:boto:Data: 2024-02-01T10:48:22.422 DEBUG:boto:Headers: {} 2024-02-01T10:48:22.422 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:22.422 DEBUG:boto:Port: 8000 2024-02-01T10:48:22.422 DEBUG:boto:Params: {} 2024-02-01T10:48:22.422 DEBUG:boto:Token: None 2024-02-01T10:48:22.422 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:22 GMT /vclfcn-61/obj1 2024-02-01T10:48:22.423 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0z7JzHjhMwoR0/v9n95MV9VwJgo= 2024-02-01T10:48:22.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0z7JzHjhMwoR0/v9n95MV9VwJgo=', 'Content-Length': '0'} 2024-02-01T10:48:22.425 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003b50f42392557ce6-0065bb76f6-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:22.426 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:48:22.426 DEBUG:boto:path=/vclfcn-61/obj1 2024-02-01T10:48:22.426 DEBUG:boto:auth_path=/vclfcn-61/obj1 2024-02-01T10:48:22.426 DEBUG:boto:Method: GET 2024-02-01T10:48:22.426 DEBUG:boto:Path: /vclfcn-61/obj1 2024-02-01T10:48:22.426 DEBUG:boto:Data: 2024-02-01T10:48:22.426 DEBUG:boto:Headers: {} 2024-02-01T10:48:22.426 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:22.426 DEBUG:boto:Port: 8001 2024-02-01T10:48:22.426 DEBUG:boto:Params: {} 2024-02-01T10:48:22.427 DEBUG:boto:Token: None 2024-02-01T10:48:22.427 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:22 GMT /vclfcn-61/obj1 2024-02-01T10:48:22.427 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:DaLO59upcbMRLMAZ9BHVCeZ2FTk= 2024-02-01T10:48:22.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:DaLO59upcbMRLMAZ9BHVCeZ2FTk=', 'Content-Length': '0'} 2024-02-01T10:48:22.430 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f486b7f61f17dc10-0065bb76f6-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:22.472 DEBUG:boto:path=/vclfcn-61/obj1 2024-02-01T10:48:22.472 DEBUG:boto:auth_path=/vclfcn-61/obj1 2024-02-01T10:48:22.472 DEBUG:boto:Method: GET 2024-02-01T10:48:22.473 DEBUG:boto:Path: /vclfcn-61/obj1 2024-02-01T10:48:22.473 DEBUG:boto:Data: 2024-02-01T10:48:22.473 DEBUG:boto:Headers: {} 2024-02-01T10:48:22.473 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:22.473 DEBUG:boto:Port: 8000 2024-02-01T10:48:22.473 DEBUG:boto:Params: {} 2024-02-01T10:48:22.473 DEBUG:boto:Token: None 2024-02-01T10:48:22.473 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:22 GMT /vclfcn-61/obj1 2024-02-01T10:48:22.474 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:DaLO59upcbMRLMAZ9BHVCeZ2FTk= 2024-02-01T10:48:22.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:DaLO59upcbMRLMAZ9BHVCeZ2FTk=', 'Content-Length': '0'} 2024-02-01T10:48:22.477 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006418bf0318e76ec8-0065bb76f6-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:22.520 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:22.520 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:22.520 DEBUG:boto:path=/vclfcn-61/?prefix=obj1 2024-02-01T10:48:22.521 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj1 2024-02-01T10:48:22.521 DEBUG:boto:Method: GET 2024-02-01T10:48:22.521 DEBUG:boto:Path: /vclfcn-61/?prefix=obj1 2024-02-01T10:48:22.521 DEBUG:boto:Data: 2024-02-01T10:48:22.521 DEBUG:boto:Headers: {} 2024-02-01T10:48:22.521 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:22.521 DEBUG:boto:Port: 8001 2024-02-01T10:48:22.521 DEBUG:boto:Params: {} 2024-02-01T10:48:22.521 DEBUG:boto:Token: None 2024-02-01T10:48:22.522 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:22 GMT /vclfcn-61/ 2024-02-01T10:48:22.522 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:DrEManxPYhWBnK0wAKjS1z5Q4pQ= 2024-02-01T10:48:22.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:DrEManxPYhWBnK0wAKjS1z5Q4pQ=', 'Content-Length': '0'} 2024-02-01T10:48:22.526 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000074a49f637527bd26-0065bb76f6-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:22.527 DEBUG:boto:b'vclfcn-61obj11000falseobj12024-02-01T10:46:46.359Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:22.527 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:22.527 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:22.527 DEBUG:boto:path=/vclfcn-61/?prefix=obj1 2024-02-01T10:48:22.527 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj1 2024-02-01T10:48:22.527 DEBUG:boto:Method: GET 2024-02-01T10:48:22.528 DEBUG:boto:Path: /vclfcn-61/?prefix=obj1 2024-02-01T10:48:22.528 DEBUG:boto:Data: 2024-02-01T10:48:22.528 DEBUG:boto:Headers: {} 2024-02-01T10:48:22.528 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:22.528 DEBUG:boto:Port: 8000 2024-02-01T10:48:22.528 DEBUG:boto:Params: {} 2024-02-01T10:48:22.528 DEBUG:boto:Token: None 2024-02-01T10:48:22.528 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:22 GMT /vclfcn-61/ 2024-02-01T10:48:22.529 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:DrEManxPYhWBnK0wAKjS1z5Q4pQ= 2024-02-01T10:48:22.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:DrEManxPYhWBnK0wAKjS1z5Q4pQ=', 'Content-Length': '0'} 2024-02-01T10:48:22.533 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6cd540aa95f5900-0065bb76f6-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:22.576 DEBUG:boto:b'vclfcn-61obj11000falseobj12024-02-01T10:46:46.359Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:22.577 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:48:22.577 DEBUG:boto:path=/vclfcn-61/obj2 2024-02-01T10:48:22.577 DEBUG:boto:auth_path=/vclfcn-61/obj2 2024-02-01T10:48:22.577 DEBUG:boto:path=/vclfcn-61/obj2?versionId=null 2024-02-01T10:48:22.577 DEBUG:boto:auth_path=/vclfcn-61/obj2?versionId=null 2024-02-01T10:48:22.577 DEBUG:boto:Method: GET 2024-02-01T10:48:22.577 DEBUG:boto:Path: /vclfcn-61/obj2?versionId=null 2024-02-01T10:48:22.577 DEBUG:boto:Data: 2024-02-01T10:48:22.578 DEBUG:boto:Headers: {} 2024-02-01T10:48:22.578 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:22.578 DEBUG:boto:Port: 8001 2024-02-01T10:48:22.578 DEBUG:boto:Params: {} 2024-02-01T10:48:22.578 DEBUG:boto:Token: None 2024-02-01T10:48:22.578 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:22 GMT /vclfcn-61/obj2?versionId=null 2024-02-01T10:48:22.578 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Bp3Co72d/oBpwiRINd/a2i4uiZg= 2024-02-01T10:48:22.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Bp3Co72d/oBpwiRINd/a2i4uiZg=', 'Content-Length': '0'} 2024-02-01T10:48:22.581 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000947385f306f2750e-0065bb76f6-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:22.582 DEBUG:boto:path=/vclfcn-61/obj2 2024-02-01T10:48:22.582 DEBUG:boto:auth_path=/vclfcn-61/obj2 2024-02-01T10:48:22.582 DEBUG:boto:path=/vclfcn-61/obj2?versionId=null 2024-02-01T10:48:22.582 DEBUG:boto:auth_path=/vclfcn-61/obj2?versionId=null 2024-02-01T10:48:22.582 DEBUG:boto:Method: GET 2024-02-01T10:48:22.582 DEBUG:boto:Path: /vclfcn-61/obj2?versionId=null 2024-02-01T10:48:22.582 DEBUG:boto:Data: 2024-02-01T10:48:22.582 DEBUG:boto:Headers: {} 2024-02-01T10:48:22.582 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:22.582 DEBUG:boto:Port: 8000 2024-02-01T10:48:22.582 DEBUG:boto:Params: {} 2024-02-01T10:48:22.583 DEBUG:boto:Token: None 2024-02-01T10:48:22.583 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:22 GMT /vclfcn-61/obj2?versionId=null 2024-02-01T10:48:22.583 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Bp3Co72d/oBpwiRINd/a2i4uiZg= 2024-02-01T10:48:22.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Bp3Co72d/oBpwiRINd/a2i4uiZg=', 'Content-Length': '0'} 2024-02-01T10:48:22.586 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000320d6423d21e150b-0065bb76f6-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:22.586 DEBUG:boto:path=/vclfcn-61/obj2 2024-02-01T10:48:22.586 DEBUG:boto:auth_path=/vclfcn-61/obj2 2024-02-01T10:48:22.586 DEBUG:boto:path=/vclfcn-61/obj2?versionId=null 2024-02-01T10:48:22.586 DEBUG:boto:auth_path=/vclfcn-61/obj2?versionId=null 2024-02-01T10:48:22.587 DEBUG:boto:Method: HEAD 2024-02-01T10:48:22.587 DEBUG:boto:Path: /vclfcn-61/obj2?versionId=null 2024-02-01T10:48:22.587 DEBUG:boto:Data: 2024-02-01T10:48:22.587 DEBUG:boto:Headers: {} 2024-02-01T10:48:22.587 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:22.587 DEBUG:boto:Port: 8001 2024-02-01T10:48:22.587 DEBUG:boto:Params: {} 2024-02-01T10:48:22.587 DEBUG:boto:Token: None 2024-02-01T10:48:22.587 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:22 GMT /vclfcn-61/obj2?versionId=null 2024-02-01T10:48:22.588 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:edEli88I5+EhVkVXeROJaLI4nIA= 2024-02-01T10:48:22.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:edEli88I5+EhVkVXeROJaLI4nIA=', 'Content-Length': '0'} 2024-02-01T10:48:22.591 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000007b944ab09b1a400-0065bb76f6-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:22.592 DEBUG:boto:path=/vclfcn-61/obj2 2024-02-01T10:48:22.592 DEBUG:boto:auth_path=/vclfcn-61/obj2 2024-02-01T10:48:22.592 DEBUG:boto:path=/vclfcn-61/obj2?versionId=null 2024-02-01T10:48:22.592 DEBUG:boto:auth_path=/vclfcn-61/obj2?versionId=null 2024-02-01T10:48:22.592 DEBUG:boto:Method: HEAD 2024-02-01T10:48:22.592 DEBUG:boto:Path: /vclfcn-61/obj2?versionId=null 2024-02-01T10:48:22.592 DEBUG:boto:Data: 2024-02-01T10:48:22.592 DEBUG:boto:Headers: {} 2024-02-01T10:48:22.592 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:22.593 DEBUG:boto:Port: 8000 2024-02-01T10:48:22.593 DEBUG:boto:Params: {} 2024-02-01T10:48:22.593 DEBUG:boto:Token: None 2024-02-01T10:48:22.593 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:22 GMT /vclfcn-61/obj2?versionId=null 2024-02-01T10:48:22.593 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:edEli88I5+EhVkVXeROJaLI4nIA= 2024-02-01T10:48:22.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:edEli88I5+EhVkVXeROJaLI4nIA=', 'Content-Length': '0'} 2024-02-01T10:48:22.596 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002e0652c20a2de495-0065bb76f6-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:22.597 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:48:22.597 DEBUG:boto:path=/vclfcn-61/obj2 2024-02-01T10:48:22.597 DEBUG:boto:auth_path=/vclfcn-61/obj2 2024-02-01T10:48:22.597 DEBUG:boto:path=/vclfcn-61/obj2?versionId=null 2024-02-01T10:48:22.598 DEBUG:boto:auth_path=/vclfcn-61/obj2?versionId=null 2024-02-01T10:48:22.598 DEBUG:boto:Method: GET 2024-02-01T10:48:22.598 DEBUG:boto:Path: /vclfcn-61/obj2?versionId=null 2024-02-01T10:48:22.598 DEBUG:boto:Data: 2024-02-01T10:48:22.598 DEBUG:boto:Headers: {} 2024-02-01T10:48:22.598 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:22.599 DEBUG:boto:Port: 8001 2024-02-01T10:48:22.599 DEBUG:boto:Params: {} 2024-02-01T10:48:22.599 DEBUG:boto:Token: None 2024-02-01T10:48:22.599 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:22 GMT /vclfcn-61/obj2?versionId=null 2024-02-01T10:48:22.602 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Bp3Co72d/oBpwiRINd/a2i4uiZg= 2024-02-01T10:48:22.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Bp3Co72d/oBpwiRINd/a2i4uiZg=', 'Content-Length': '0'} 2024-02-01T10:48:22.605 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000053bd1a8b79406b50-0065bb76f6-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:22.605 DEBUG:boto:path=/vclfcn-61/obj2 2024-02-01T10:48:22.605 DEBUG:boto:auth_path=/vclfcn-61/obj2 2024-02-01T10:48:22.605 DEBUG:boto:path=/vclfcn-61/obj2?versionId=null 2024-02-01T10:48:22.605 DEBUG:boto:auth_path=/vclfcn-61/obj2?versionId=null 2024-02-01T10:48:22.605 DEBUG:boto:Method: GET 2024-02-01T10:48:22.606 DEBUG:boto:Path: /vclfcn-61/obj2?versionId=null 2024-02-01T10:48:22.606 DEBUG:boto:Data: 2024-02-01T10:48:22.606 DEBUG:boto:Headers: {} 2024-02-01T10:48:22.606 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:22.606 DEBUG:boto:Port: 8000 2024-02-01T10:48:22.606 DEBUG:boto:Params: {} 2024-02-01T10:48:22.606 DEBUG:boto:Token: None 2024-02-01T10:48:22.606 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:22 GMT /vclfcn-61/obj2?versionId=null 2024-02-01T10:48:22.606 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Bp3Co72d/oBpwiRINd/a2i4uiZg= 2024-02-01T10:48:22.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Bp3Co72d/oBpwiRINd/a2i4uiZg=', 'Content-Length': '0'} 2024-02-01T10:48:22.609 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d5ea3f21b40811bd-0065bb76f6-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:22.652 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:22.652 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:22.652 DEBUG:boto:path=/vclfcn-61/?prefix=obj2 2024-02-01T10:48:22.652 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj2 2024-02-01T10:48:22.653 DEBUG:boto:Method: GET 2024-02-01T10:48:22.653 DEBUG:boto:Path: /vclfcn-61/?prefix=obj2 2024-02-01T10:48:22.653 DEBUG:boto:Data: 2024-02-01T10:48:22.653 DEBUG:boto:Headers: {} 2024-02-01T10:48:22.653 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:22.653 DEBUG:boto:Port: 8001 2024-02-01T10:48:22.653 DEBUG:boto:Params: {} 2024-02-01T10:48:22.653 DEBUG:boto:Token: None 2024-02-01T10:48:22.653 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:22 GMT /vclfcn-61/ 2024-02-01T10:48:22.654 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:DrEManxPYhWBnK0wAKjS1z5Q4pQ= 2024-02-01T10:48:22.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:DrEManxPYhWBnK0wAKjS1z5Q4pQ=', 'Content-Length': '0'} 2024-02-01T10:48:22.658 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b838e74a5ef25d2-0065bb76f6-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:22.658 DEBUG:boto:b'vclfcn-61obj21000falseobj22024-02-01T10:46:46.380Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:22.659 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:22.659 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:22.659 DEBUG:boto:path=/vclfcn-61/?prefix=obj2 2024-02-01T10:48:22.659 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj2 2024-02-01T10:48:22.659 DEBUG:boto:Method: GET 2024-02-01T10:48:22.659 DEBUG:boto:Path: /vclfcn-61/?prefix=obj2 2024-02-01T10:48:22.659 DEBUG:boto:Data: 2024-02-01T10:48:22.659 DEBUG:boto:Headers: {} 2024-02-01T10:48:22.659 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:22.660 DEBUG:boto:Port: 8000 2024-02-01T10:48:22.660 DEBUG:boto:Params: {} 2024-02-01T10:48:22.660 DEBUG:boto:Token: None 2024-02-01T10:48:22.660 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:22 GMT /vclfcn-61/ 2024-02-01T10:48:22.660 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:DrEManxPYhWBnK0wAKjS1z5Q4pQ= 2024-02-01T10:48:22.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:DrEManxPYhWBnK0wAKjS1z5Q4pQ=', 'Content-Length': '0'} 2024-02-01T10:48:22.664 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000740397c9dfb7fda2-0065bb76f6-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:22.664 DEBUG:boto:b'vclfcn-61obj21000falseobj22024-02-01T10:46:46.380Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:22.665 DEBUG:boto:path=/vclfcn-61/obj2 2024-02-01T10:48:22.665 DEBUG:boto:auth_path=/vclfcn-61/obj2 2024-02-01T10:48:22.665 DEBUG:boto:Method: HEAD 2024-02-01T10:48:22.665 DEBUG:boto:Path: /vclfcn-61/obj2 2024-02-01T10:48:22.665 DEBUG:boto:Data: 2024-02-01T10:48:22.665 DEBUG:boto:Headers: {} 2024-02-01T10:48:22.665 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:22.665 DEBUG:boto:Port: 8001 2024-02-01T10:48:22.665 DEBUG:boto:Params: {} 2024-02-01T10:48:22.665 DEBUG:boto:Token: None 2024-02-01T10:48:22.666 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:22 GMT /vclfcn-61/obj2 2024-02-01T10:48:22.666 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fP7wvp9lJwRoLIzBSUXKsaQZcRM= 2024-02-01T10:48:22.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fP7wvp9lJwRoLIzBSUXKsaQZcRM=', 'Content-Length': '0'} 2024-02-01T10:48:22.668 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000079e08daca36a4044-0065bb76f6-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:22.669 DEBUG:boto:path=/vclfcn-61/obj2 2024-02-01T10:48:22.669 DEBUG:boto:auth_path=/vclfcn-61/obj2 2024-02-01T10:48:22.669 DEBUG:boto:Method: HEAD 2024-02-01T10:48:22.669 DEBUG:boto:Path: /vclfcn-61/obj2 2024-02-01T10:48:22.669 DEBUG:boto:Data: 2024-02-01T10:48:22.669 DEBUG:boto:Headers: {} 2024-02-01T10:48:22.669 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:22.669 DEBUG:boto:Port: 8000 2024-02-01T10:48:22.669 DEBUG:boto:Params: {} 2024-02-01T10:48:22.669 DEBUG:boto:Token: None 2024-02-01T10:48:22.670 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:22 GMT /vclfcn-61/obj2 2024-02-01T10:48:22.670 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fP7wvp9lJwRoLIzBSUXKsaQZcRM= 2024-02-01T10:48:22.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fP7wvp9lJwRoLIzBSUXKsaQZcRM=', 'Content-Length': '0'} 2024-02-01T10:48:22.672 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a07174b0ef84be47-0065bb76f6-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:22.673 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:48:22.673 DEBUG:boto:path=/vclfcn-61/obj2 2024-02-01T10:48:22.673 DEBUG:boto:auth_path=/vclfcn-61/obj2 2024-02-01T10:48:22.673 DEBUG:boto:Method: GET 2024-02-01T10:48:22.673 DEBUG:boto:Path: /vclfcn-61/obj2 2024-02-01T10:48:22.673 DEBUG:boto:Data: 2024-02-01T10:48:22.673 DEBUG:boto:Headers: {} 2024-02-01T10:48:22.673 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:22.673 DEBUG:boto:Port: 8001 2024-02-01T10:48:22.673 DEBUG:boto:Params: {} 2024-02-01T10:48:22.674 DEBUG:boto:Token: None 2024-02-01T10:48:22.674 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:22 GMT /vclfcn-61/obj2 2024-02-01T10:48:22.674 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gktndRR4x5qoZ2US7VHUJ39x4hg= 2024-02-01T10:48:22.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gktndRR4x5qoZ2US7VHUJ39x4hg=', 'Content-Length': '0'} 2024-02-01T10:48:22.677 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a374f0bf97024ab5-0065bb76f6-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:22.720 DEBUG:boto:path=/vclfcn-61/obj2 2024-02-01T10:48:22.720 DEBUG:boto:auth_path=/vclfcn-61/obj2 2024-02-01T10:48:22.721 DEBUG:boto:Method: GET 2024-02-01T10:48:22.721 DEBUG:boto:Path: /vclfcn-61/obj2 2024-02-01T10:48:22.721 DEBUG:boto:Data: 2024-02-01T10:48:22.721 DEBUG:boto:Headers: {} 2024-02-01T10:48:22.721 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:22.721 DEBUG:boto:Port: 8000 2024-02-01T10:48:22.721 DEBUG:boto:Params: {} 2024-02-01T10:48:22.721 DEBUG:boto:Token: None 2024-02-01T10:48:22.721 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:22 GMT /vclfcn-61/obj2 2024-02-01T10:48:22.722 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gktndRR4x5qoZ2US7VHUJ39x4hg= 2024-02-01T10:48:22.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gktndRR4x5qoZ2US7VHUJ39x4hg=', 'Content-Length': '0'} 2024-02-01T10:48:22.725 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009f471bb26dbcc804-0065bb76f6-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:22.768 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:22.768 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:22.768 DEBUG:boto:path=/vclfcn-61/?prefix=obj2 2024-02-01T10:48:22.769 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj2 2024-02-01T10:48:22.769 DEBUG:boto:Method: GET 2024-02-01T10:48:22.769 DEBUG:boto:Path: /vclfcn-61/?prefix=obj2 2024-02-01T10:48:22.769 DEBUG:boto:Data: 2024-02-01T10:48:22.769 DEBUG:boto:Headers: {} 2024-02-01T10:48:22.769 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:22.769 DEBUG:boto:Port: 8001 2024-02-01T10:48:22.769 DEBUG:boto:Params: {} 2024-02-01T10:48:22.769 DEBUG:boto:Token: None 2024-02-01T10:48:22.769 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:22 GMT /vclfcn-61/ 2024-02-01T10:48:22.770 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:DrEManxPYhWBnK0wAKjS1z5Q4pQ= 2024-02-01T10:48:22.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:DrEManxPYhWBnK0wAKjS1z5Q4pQ=', 'Content-Length': '0'} 2024-02-01T10:48:22.776 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000001c9ca0346b43159-0065bb76f6-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:22.776 DEBUG:boto:b'vclfcn-61obj21000falseobj22024-02-01T10:46:46.380Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:22.777 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:22.777 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:22.777 DEBUG:boto:path=/vclfcn-61/?prefix=obj2 2024-02-01T10:48:22.777 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj2 2024-02-01T10:48:22.777 DEBUG:boto:Method: GET 2024-02-01T10:48:22.777 DEBUG:boto:Path: /vclfcn-61/?prefix=obj2 2024-02-01T10:48:22.778 DEBUG:boto:Data: 2024-02-01T10:48:22.778 DEBUG:boto:Headers: {} 2024-02-01T10:48:22.778 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:22.778 DEBUG:boto:Port: 8000 2024-02-01T10:48:22.778 DEBUG:boto:Params: {} 2024-02-01T10:48:22.778 DEBUG:boto:Token: None 2024-02-01T10:48:22.778 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:22 GMT /vclfcn-61/ 2024-02-01T10:48:22.778 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:DrEManxPYhWBnK0wAKjS1z5Q4pQ= 2024-02-01T10:48:22.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:DrEManxPYhWBnK0wAKjS1z5Q4pQ=', 'Content-Length': '0'} 2024-02-01T10:48:22.783 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000136e681c9d310351-0065bb76f6-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:22.828 DEBUG:boto:b'vclfcn-61obj21000falseobj22024-02-01T10:46:46.380Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:22.828 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:48:22.829 DEBUG:boto:path=/vclfcn-61/obj3 2024-02-01T10:48:22.829 DEBUG:boto:auth_path=/vclfcn-61/obj3 2024-02-01T10:48:22.829 DEBUG:boto:path=/vclfcn-61/obj3?versionId=null 2024-02-01T10:48:22.829 DEBUG:boto:auth_path=/vclfcn-61/obj3?versionId=null 2024-02-01T10:48:22.829 DEBUG:boto:Method: GET 2024-02-01T10:48:22.829 DEBUG:boto:Path: /vclfcn-61/obj3?versionId=null 2024-02-01T10:48:22.829 DEBUG:boto:Data: 2024-02-01T10:48:22.829 DEBUG:boto:Headers: {} 2024-02-01T10:48:22.829 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:22.829 DEBUG:boto:Port: 8001 2024-02-01T10:48:22.830 DEBUG:boto:Params: {} 2024-02-01T10:48:22.830 DEBUG:boto:Token: None 2024-02-01T10:48:22.830 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:22 GMT /vclfcn-61/obj3?versionId=null 2024-02-01T10:48:22.830 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fBQKtUC9gTGMQCKcaKKkuK9+mhY= 2024-02-01T10:48:22.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fBQKtUC9gTGMQCKcaKKkuK9+mhY=', 'Content-Length': '0'} 2024-02-01T10:48:22.833 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000093906e92905750c0-0065bb76f6-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:22.834 DEBUG:boto:path=/vclfcn-61/obj3 2024-02-01T10:48:22.834 DEBUG:boto:auth_path=/vclfcn-61/obj3 2024-02-01T10:48:22.834 DEBUG:boto:path=/vclfcn-61/obj3?versionId=null 2024-02-01T10:48:22.834 DEBUG:boto:auth_path=/vclfcn-61/obj3?versionId=null 2024-02-01T10:48:22.834 DEBUG:boto:Method: GET 2024-02-01T10:48:22.834 DEBUG:boto:Path: /vclfcn-61/obj3?versionId=null 2024-02-01T10:48:22.834 DEBUG:boto:Data: 2024-02-01T10:48:22.834 DEBUG:boto:Headers: {} 2024-02-01T10:48:22.834 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:22.834 DEBUG:boto:Port: 8000 2024-02-01T10:48:22.834 DEBUG:boto:Params: {} 2024-02-01T10:48:22.835 DEBUG:boto:Token: None 2024-02-01T10:48:22.835 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:22 GMT /vclfcn-61/obj3?versionId=null 2024-02-01T10:48:22.835 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fBQKtUC9gTGMQCKcaKKkuK9+mhY= 2024-02-01T10:48:22.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fBQKtUC9gTGMQCKcaKKkuK9+mhY=', 'Content-Length': '0'} 2024-02-01T10:48:22.837 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000690bfdc8005667ce-0065bb76f6-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:22.838 DEBUG:boto:path=/vclfcn-61/obj3 2024-02-01T10:48:22.838 DEBUG:boto:auth_path=/vclfcn-61/obj3 2024-02-01T10:48:22.838 DEBUG:boto:path=/vclfcn-61/obj3?versionId=null 2024-02-01T10:48:22.838 DEBUG:boto:auth_path=/vclfcn-61/obj3?versionId=null 2024-02-01T10:48:22.838 DEBUG:boto:Method: HEAD 2024-02-01T10:48:22.838 DEBUG:boto:Path: /vclfcn-61/obj3?versionId=null 2024-02-01T10:48:22.838 DEBUG:boto:Data: 2024-02-01T10:48:22.838 DEBUG:boto:Headers: {} 2024-02-01T10:48:22.839 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:22.839 DEBUG:boto:Port: 8001 2024-02-01T10:48:22.839 DEBUG:boto:Params: {} 2024-02-01T10:48:22.839 DEBUG:boto:Token: None 2024-02-01T10:48:22.839 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:22 GMT /vclfcn-61/obj3?versionId=null 2024-02-01T10:48:22.839 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:hj4cygZ2+GByFx8SRFyhMSxgtqQ= 2024-02-01T10:48:22.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:hj4cygZ2+GByFx8SRFyhMSxgtqQ=', 'Content-Length': '0'} 2024-02-01T10:48:22.842 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a0d2f0936e1775a4-0065bb76f6-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:22.842 DEBUG:boto:path=/vclfcn-61/obj3 2024-02-01T10:48:22.842 DEBUG:boto:auth_path=/vclfcn-61/obj3 2024-02-01T10:48:22.842 DEBUG:boto:path=/vclfcn-61/obj3?versionId=null 2024-02-01T10:48:22.842 DEBUG:boto:auth_path=/vclfcn-61/obj3?versionId=null 2024-02-01T10:48:22.842 DEBUG:boto:Method: HEAD 2024-02-01T10:48:22.843 DEBUG:boto:Path: /vclfcn-61/obj3?versionId=null 2024-02-01T10:48:22.843 DEBUG:boto:Data: 2024-02-01T10:48:22.843 DEBUG:boto:Headers: {} 2024-02-01T10:48:22.843 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:22.843 DEBUG:boto:Port: 8000 2024-02-01T10:48:22.843 DEBUG:boto:Params: {} 2024-02-01T10:48:22.843 DEBUG:boto:Token: None 2024-02-01T10:48:22.843 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:22 GMT /vclfcn-61/obj3?versionId=null 2024-02-01T10:48:22.843 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:hj4cygZ2+GByFx8SRFyhMSxgtqQ= 2024-02-01T10:48:22.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:hj4cygZ2+GByFx8SRFyhMSxgtqQ=', 'Content-Length': '0'} 2024-02-01T10:48:22.846 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003d7352fb46cd7187-0065bb76f6-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:22.846 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:48:22.846 DEBUG:boto:path=/vclfcn-61/obj3 2024-02-01T10:48:22.846 DEBUG:boto:auth_path=/vclfcn-61/obj3 2024-02-01T10:48:22.847 DEBUG:boto:path=/vclfcn-61/obj3?versionId=null 2024-02-01T10:48:22.847 DEBUG:boto:auth_path=/vclfcn-61/obj3?versionId=null 2024-02-01T10:48:22.847 DEBUG:boto:Method: GET 2024-02-01T10:48:22.847 DEBUG:boto:Path: /vclfcn-61/obj3?versionId=null 2024-02-01T10:48:22.847 DEBUG:boto:Data: 2024-02-01T10:48:22.847 DEBUG:boto:Headers: {} 2024-02-01T10:48:22.847 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:22.847 DEBUG:boto:Port: 8001 2024-02-01T10:48:22.847 DEBUG:boto:Params: {} 2024-02-01T10:48:22.847 DEBUG:boto:Token: None 2024-02-01T10:48:22.847 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:22 GMT /vclfcn-61/obj3?versionId=null 2024-02-01T10:48:22.848 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fBQKtUC9gTGMQCKcaKKkuK9+mhY= 2024-02-01T10:48:22.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fBQKtUC9gTGMQCKcaKKkuK9+mhY=', 'Content-Length': '0'} 2024-02-01T10:48:22.850 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000068f78d1a04c4cb6d-0065bb76f6-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:22.851 DEBUG:boto:path=/vclfcn-61/obj3 2024-02-01T10:48:22.851 DEBUG:boto:auth_path=/vclfcn-61/obj3 2024-02-01T10:48:22.851 DEBUG:boto:path=/vclfcn-61/obj3?versionId=null 2024-02-01T10:48:22.851 DEBUG:boto:auth_path=/vclfcn-61/obj3?versionId=null 2024-02-01T10:48:22.851 DEBUG:boto:Method: GET 2024-02-01T10:48:22.851 DEBUG:boto:Path: /vclfcn-61/obj3?versionId=null 2024-02-01T10:48:22.851 DEBUG:boto:Data: 2024-02-01T10:48:22.851 DEBUG:boto:Headers: {} 2024-02-01T10:48:22.852 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:22.852 DEBUG:boto:Port: 8000 2024-02-01T10:48:22.852 DEBUG:boto:Params: {} 2024-02-01T10:48:22.852 DEBUG:boto:Token: None 2024-02-01T10:48:22.852 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:22 GMT /vclfcn-61/obj3?versionId=null 2024-02-01T10:48:22.852 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fBQKtUC9gTGMQCKcaKKkuK9+mhY= 2024-02-01T10:48:22.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fBQKtUC9gTGMQCKcaKKkuK9+mhY=', 'Content-Length': '0'} 2024-02-01T10:48:22.855 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004d0217aed4babb2c-0065bb76f6-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:22.896 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:22.896 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:22.896 DEBUG:boto:path=/vclfcn-61/?prefix=obj3 2024-02-01T10:48:22.897 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj3 2024-02-01T10:48:22.897 DEBUG:boto:Method: GET 2024-02-01T10:48:22.897 DEBUG:boto:Path: /vclfcn-61/?prefix=obj3 2024-02-01T10:48:22.897 DEBUG:boto:Data: 2024-02-01T10:48:22.897 DEBUG:boto:Headers: {} 2024-02-01T10:48:22.897 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:22.897 DEBUG:boto:Port: 8001 2024-02-01T10:48:22.897 DEBUG:boto:Params: {} 2024-02-01T10:48:22.897 DEBUG:boto:Token: None 2024-02-01T10:48:22.897 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:22 GMT /vclfcn-61/ 2024-02-01T10:48:22.898 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:DrEManxPYhWBnK0wAKjS1z5Q4pQ= 2024-02-01T10:48:22.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:DrEManxPYhWBnK0wAKjS1z5Q4pQ=', 'Content-Length': '0'} 2024-02-01T10:48:22.902 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000550b68bd6be69d1b-0065bb76f6-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:22.902 DEBUG:boto:b'vclfcn-61obj31000falseobj32024-02-01T10:46:46.433Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:22.903 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:22.903 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:22.903 DEBUG:boto:path=/vclfcn-61/?prefix=obj3 2024-02-01T10:48:22.903 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj3 2024-02-01T10:48:22.903 DEBUG:boto:Method: GET 2024-02-01T10:48:22.903 DEBUG:boto:Path: /vclfcn-61/?prefix=obj3 2024-02-01T10:48:22.903 DEBUG:boto:Data: 2024-02-01T10:48:22.904 DEBUG:boto:Headers: {} 2024-02-01T10:48:22.904 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:22.904 DEBUG:boto:Port: 8000 2024-02-01T10:48:22.904 DEBUG:boto:Params: {} 2024-02-01T10:48:22.904 DEBUG:boto:Token: None 2024-02-01T10:48:22.904 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:22 GMT /vclfcn-61/ 2024-02-01T10:48:22.904 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:DrEManxPYhWBnK0wAKjS1z5Q4pQ= 2024-02-01T10:48:22.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:DrEManxPYhWBnK0wAKjS1z5Q4pQ=', 'Content-Length': '0'} 2024-02-01T10:48:22.909 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cca6348e34f33d73-0065bb76f6-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:22.953 DEBUG:boto:b'vclfcn-61obj31000falseobj32024-02-01T10:46:46.433Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:22.953 DEBUG:boto:path=/vclfcn-61/obj3 2024-02-01T10:48:22.953 DEBUG:boto:auth_path=/vclfcn-61/obj3 2024-02-01T10:48:22.953 DEBUG:boto:Method: HEAD 2024-02-01T10:48:22.953 DEBUG:boto:Path: /vclfcn-61/obj3 2024-02-01T10:48:22.954 DEBUG:boto:Data: 2024-02-01T10:48:22.954 DEBUG:boto:Headers: {} 2024-02-01T10:48:22.954 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:22.954 DEBUG:boto:Port: 8001 2024-02-01T10:48:22.954 DEBUG:boto:Params: {} 2024-02-01T10:48:22.954 DEBUG:boto:Token: None 2024-02-01T10:48:22.954 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:22 GMT /vclfcn-61/obj3 2024-02-01T10:48:22.954 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:AlcV1C2yRCyO0emMY64T2dsbkbk= 2024-02-01T10:48:22.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:AlcV1C2yRCyO0emMY64T2dsbkbk=', 'Content-Length': '0'} 2024-02-01T10:48:22.957 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008d64edd504e31b25-0065bb76f6-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:22.958 DEBUG:boto:path=/vclfcn-61/obj3 2024-02-01T10:48:22.958 DEBUG:boto:auth_path=/vclfcn-61/obj3 2024-02-01T10:48:22.958 DEBUG:boto:Method: HEAD 2024-02-01T10:48:22.958 DEBUG:boto:Path: /vclfcn-61/obj3 2024-02-01T10:48:22.958 DEBUG:boto:Data: 2024-02-01T10:48:22.958 DEBUG:boto:Headers: {} 2024-02-01T10:48:22.958 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:22.958 DEBUG:boto:Port: 8000 2024-02-01T10:48:22.958 DEBUG:boto:Params: {} 2024-02-01T10:48:22.959 DEBUG:boto:Token: None 2024-02-01T10:48:22.959 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:22 GMT /vclfcn-61/obj3 2024-02-01T10:48:22.959 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:AlcV1C2yRCyO0emMY64T2dsbkbk= 2024-02-01T10:48:22.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:AlcV1C2yRCyO0emMY64T2dsbkbk=', 'Content-Length': '0'} 2024-02-01T10:48:22.961 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d72b7c99f3583fb6-0065bb76f6-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:22.962 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:48:22.962 DEBUG:boto:path=/vclfcn-61/obj3 2024-02-01T10:48:22.962 DEBUG:boto:auth_path=/vclfcn-61/obj3 2024-02-01T10:48:22.962 DEBUG:boto:Method: GET 2024-02-01T10:48:22.962 DEBUG:boto:Path: /vclfcn-61/obj3 2024-02-01T10:48:22.962 DEBUG:boto:Data: 2024-02-01T10:48:22.962 DEBUG:boto:Headers: {} 2024-02-01T10:48:22.962 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:22.963 DEBUG:boto:Port: 8001 2024-02-01T10:48:22.963 DEBUG:boto:Params: {} 2024-02-01T10:48:22.963 DEBUG:boto:Token: None 2024-02-01T10:48:22.963 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:22 GMT /vclfcn-61/obj3 2024-02-01T10:48:22.963 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:rn7mvFLMOXr32xy3IdYyPNxIVHc= 2024-02-01T10:48:22.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:rn7mvFLMOXr32xy3IdYyPNxIVHc=', 'Content-Length': '0'} 2024-02-01T10:48:22.966 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004f7f45849b5e3df0-0065bb76f6-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:23.012 DEBUG:boto:path=/vclfcn-61/obj3 2024-02-01T10:48:23.012 DEBUG:boto:auth_path=/vclfcn-61/obj3 2024-02-01T10:48:23.012 DEBUG:boto:Method: GET 2024-02-01T10:48:23.013 DEBUG:boto:Path: /vclfcn-61/obj3 2024-02-01T10:48:23.013 DEBUG:boto:Data: 2024-02-01T10:48:23.013 DEBUG:boto:Headers: {} 2024-02-01T10:48:23.013 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:23.013 DEBUG:boto:Port: 8000 2024-02-01T10:48:23.013 DEBUG:boto:Params: {} 2024-02-01T10:48:23.013 DEBUG:boto:Token: None 2024-02-01T10:48:23.013 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:23 GMT /vclfcn-61/obj3 2024-02-01T10:48:23.013 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:G9NUrcMHekT4fhHeOPyKQRpCo84= 2024-02-01T10:48:23.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:G9NUrcMHekT4fhHeOPyKQRpCo84=', 'Content-Length': '0'} 2024-02-01T10:48:23.016 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008111e2d8f642ac03-0065bb76f7-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:23.060 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:23.060 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:23.060 DEBUG:boto:path=/vclfcn-61/?prefix=obj3 2024-02-01T10:48:23.061 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj3 2024-02-01T10:48:23.061 DEBUG:boto:Method: GET 2024-02-01T10:48:23.061 DEBUG:boto:Path: /vclfcn-61/?prefix=obj3 2024-02-01T10:48:23.061 DEBUG:boto:Data: 2024-02-01T10:48:23.061 DEBUG:boto:Headers: {} 2024-02-01T10:48:23.061 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:23.061 DEBUG:boto:Port: 8001 2024-02-01T10:48:23.061 DEBUG:boto:Params: {} 2024-02-01T10:48:23.061 DEBUG:boto:Token: None 2024-02-01T10:48:23.062 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:23 GMT /vclfcn-61/ 2024-02-01T10:48:23.062 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HXFkyexdu3c5953UR2gnypKP/8Y= 2024-02-01T10:48:23.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HXFkyexdu3c5953UR2gnypKP/8Y=', 'Content-Length': '0'} 2024-02-01T10:48:23.066 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b93d7e48037a9ab-0065bb76f7-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:23.066 DEBUG:boto:b'vclfcn-61obj31000falseobj32024-02-01T10:46:46.433Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:23.067 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:23.067 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:23.067 DEBUG:boto:path=/vclfcn-61/?prefix=obj3 2024-02-01T10:48:23.067 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj3 2024-02-01T10:48:23.067 DEBUG:boto:Method: GET 2024-02-01T10:48:23.067 DEBUG:boto:Path: /vclfcn-61/?prefix=obj3 2024-02-01T10:48:23.067 DEBUG:boto:Data: 2024-02-01T10:48:23.067 DEBUG:boto:Headers: {} 2024-02-01T10:48:23.067 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:23.067 DEBUG:boto:Port: 8000 2024-02-01T10:48:23.068 DEBUG:boto:Params: {} 2024-02-01T10:48:23.068 DEBUG:boto:Token: None 2024-02-01T10:48:23.068 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:23 GMT /vclfcn-61/ 2024-02-01T10:48:23.068 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HXFkyexdu3c5953UR2gnypKP/8Y= 2024-02-01T10:48:23.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HXFkyexdu3c5953UR2gnypKP/8Y=', 'Content-Length': '0'} 2024-02-01T10:48:23.072 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000791c4f747ed9dcf-0065bb76f7-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:23.073 DEBUG:boto:b'vclfcn-61obj31000falseobj32024-02-01T10:46:46.433Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:23.073 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:48:23.073 DEBUG:boto:path=/vclfcn-61/obj4 2024-02-01T10:48:23.073 DEBUG:boto:auth_path=/vclfcn-61/obj4 2024-02-01T10:48:23.073 DEBUG:boto:path=/vclfcn-61/obj4?versionId=null 2024-02-01T10:48:23.073 DEBUG:boto:auth_path=/vclfcn-61/obj4?versionId=null 2024-02-01T10:48:23.073 DEBUG:boto:Method: GET 2024-02-01T10:48:23.073 DEBUG:boto:Path: /vclfcn-61/obj4?versionId=null 2024-02-01T10:48:23.074 DEBUG:boto:Data: 2024-02-01T10:48:23.074 DEBUG:boto:Headers: {} 2024-02-01T10:48:23.074 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:23.074 DEBUG:boto:Port: 8001 2024-02-01T10:48:23.074 DEBUG:boto:Params: {} 2024-02-01T10:48:23.074 DEBUG:boto:Token: None 2024-02-01T10:48:23.074 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:23 GMT /vclfcn-61/obj4?versionId=null 2024-02-01T10:48:23.074 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:3cf+25gVGhAFWP8RmuBo03Jtbpc= 2024-02-01T10:48:23.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:3cf+25gVGhAFWP8RmuBo03Jtbpc=', 'Content-Length': '0'} 2024-02-01T10:48:23.077 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003460f28dc0258012-0065bb76f7-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:23.077 DEBUG:boto:path=/vclfcn-61/obj4 2024-02-01T10:48:23.077 DEBUG:boto:auth_path=/vclfcn-61/obj4 2024-02-01T10:48:23.077 DEBUG:boto:path=/vclfcn-61/obj4?versionId=null 2024-02-01T10:48:23.078 DEBUG:boto:auth_path=/vclfcn-61/obj4?versionId=null 2024-02-01T10:48:23.078 DEBUG:boto:Method: GET 2024-02-01T10:48:23.078 DEBUG:boto:Path: /vclfcn-61/obj4?versionId=null 2024-02-01T10:48:23.078 DEBUG:boto:Data: 2024-02-01T10:48:23.078 DEBUG:boto:Headers: {} 2024-02-01T10:48:23.078 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:23.078 DEBUG:boto:Port: 8000 2024-02-01T10:48:23.078 DEBUG:boto:Params: {} 2024-02-01T10:48:23.078 DEBUG:boto:Token: None 2024-02-01T10:48:23.078 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:23 GMT /vclfcn-61/obj4?versionId=null 2024-02-01T10:48:23.078 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:3cf+25gVGhAFWP8RmuBo03Jtbpc= 2024-02-01T10:48:23.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:3cf+25gVGhAFWP8RmuBo03Jtbpc=', 'Content-Length': '0'} 2024-02-01T10:48:23.082 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000956e28fb90d2f004-0065bb76f7-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:23.124 DEBUG:boto:path=/vclfcn-61/obj4 2024-02-01T10:48:23.124 DEBUG:boto:auth_path=/vclfcn-61/obj4 2024-02-01T10:48:23.124 DEBUG:boto:path=/vclfcn-61/obj4?versionId=null 2024-02-01T10:48:23.124 DEBUG:boto:auth_path=/vclfcn-61/obj4?versionId=null 2024-02-01T10:48:23.125 DEBUG:boto:Method: HEAD 2024-02-01T10:48:23.125 DEBUG:boto:Path: /vclfcn-61/obj4?versionId=null 2024-02-01T10:48:23.125 DEBUG:boto:Data: 2024-02-01T10:48:23.125 DEBUG:boto:Headers: {} 2024-02-01T10:48:23.125 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:23.125 DEBUG:boto:Port: 8001 2024-02-01T10:48:23.125 DEBUG:boto:Params: {} 2024-02-01T10:48:23.125 DEBUG:boto:Token: None 2024-02-01T10:48:23.126 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:23 GMT /vclfcn-61/obj4?versionId=null 2024-02-01T10:48:23.126 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cMxE3frrHGXzVLnS386lb2W0bdk= 2024-02-01T10:48:23.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cMxE3frrHGXzVLnS386lb2W0bdk=', 'Content-Length': '0'} 2024-02-01T10:48:23.129 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000023cb237bcbe11c35-0065bb76f7-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:23.129 DEBUG:boto:path=/vclfcn-61/obj4 2024-02-01T10:48:23.129 DEBUG:boto:auth_path=/vclfcn-61/obj4 2024-02-01T10:48:23.129 DEBUG:boto:path=/vclfcn-61/obj4?versionId=null 2024-02-01T10:48:23.129 DEBUG:boto:auth_path=/vclfcn-61/obj4?versionId=null 2024-02-01T10:48:23.129 DEBUG:boto:Method: HEAD 2024-02-01T10:48:23.129 DEBUG:boto:Path: /vclfcn-61/obj4?versionId=null 2024-02-01T10:48:23.130 DEBUG:boto:Data: 2024-02-01T10:48:23.130 DEBUG:boto:Headers: {} 2024-02-01T10:48:23.130 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:23.130 DEBUG:boto:Port: 8000 2024-02-01T10:48:23.130 DEBUG:boto:Params: {} 2024-02-01T10:48:23.130 DEBUG:boto:Token: None 2024-02-01T10:48:23.130 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:23 GMT /vclfcn-61/obj4?versionId=null 2024-02-01T10:48:23.130 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cMxE3frrHGXzVLnS386lb2W0bdk= 2024-02-01T10:48:23.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cMxE3frrHGXzVLnS386lb2W0bdk=', 'Content-Length': '0'} 2024-02-01T10:48:23.133 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009f3f222fd6c0e0c3-0065bb76f7-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:23.133 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:48:23.134 DEBUG:boto:path=/vclfcn-61/obj4 2024-02-01T10:48:23.134 DEBUG:boto:auth_path=/vclfcn-61/obj4 2024-02-01T10:48:23.134 DEBUG:boto:path=/vclfcn-61/obj4?versionId=null 2024-02-01T10:48:23.134 DEBUG:boto:auth_path=/vclfcn-61/obj4?versionId=null 2024-02-01T10:48:23.134 DEBUG:boto:Method: GET 2024-02-01T10:48:23.134 DEBUG:boto:Path: /vclfcn-61/obj4?versionId=null 2024-02-01T10:48:23.134 DEBUG:boto:Data: 2024-02-01T10:48:23.134 DEBUG:boto:Headers: {} 2024-02-01T10:48:23.134 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:23.134 DEBUG:boto:Port: 8001 2024-02-01T10:48:23.134 DEBUG:boto:Params: {} 2024-02-01T10:48:23.135 DEBUG:boto:Token: None 2024-02-01T10:48:23.135 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:23 GMT /vclfcn-61/obj4?versionId=null 2024-02-01T10:48:23.135 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:3cf+25gVGhAFWP8RmuBo03Jtbpc= 2024-02-01T10:48:23.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:3cf+25gVGhAFWP8RmuBo03Jtbpc=', 'Content-Length': '0'} 2024-02-01T10:48:23.138 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000caa23510140de6f6-0065bb76f7-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:23.138 DEBUG:boto:path=/vclfcn-61/obj4 2024-02-01T10:48:23.138 DEBUG:boto:auth_path=/vclfcn-61/obj4 2024-02-01T10:48:23.138 DEBUG:boto:path=/vclfcn-61/obj4?versionId=null 2024-02-01T10:48:23.139 DEBUG:boto:auth_path=/vclfcn-61/obj4?versionId=null 2024-02-01T10:48:23.139 DEBUG:boto:Method: GET 2024-02-01T10:48:23.139 DEBUG:boto:Path: /vclfcn-61/obj4?versionId=null 2024-02-01T10:48:23.139 DEBUG:boto:Data: 2024-02-01T10:48:23.139 DEBUG:boto:Headers: {} 2024-02-01T10:48:23.139 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:23.139 DEBUG:boto:Port: 8000 2024-02-01T10:48:23.139 DEBUG:boto:Params: {} 2024-02-01T10:48:23.139 DEBUG:boto:Token: None 2024-02-01T10:48:23.139 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:23 GMT /vclfcn-61/obj4?versionId=null 2024-02-01T10:48:23.140 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:3cf+25gVGhAFWP8RmuBo03Jtbpc= 2024-02-01T10:48:23.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:3cf+25gVGhAFWP8RmuBo03Jtbpc=', 'Content-Length': '0'} 2024-02-01T10:48:23.142 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000075746fb608d42d54-0065bb76f7-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:23.185 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:23.185 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:23.185 DEBUG:boto:path=/vclfcn-61/?prefix=obj4 2024-02-01T10:48:23.185 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj4 2024-02-01T10:48:23.185 DEBUG:boto:Method: GET 2024-02-01T10:48:23.185 DEBUG:boto:Path: /vclfcn-61/?prefix=obj4 2024-02-01T10:48:23.185 DEBUG:boto:Data: 2024-02-01T10:48:23.185 DEBUG:boto:Headers: {} 2024-02-01T10:48:23.186 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:23.186 DEBUG:boto:Port: 8001 2024-02-01T10:48:23.186 DEBUG:boto:Params: {} 2024-02-01T10:48:23.186 DEBUG:boto:Token: None 2024-02-01T10:48:23.186 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:23 GMT /vclfcn-61/ 2024-02-01T10:48:23.186 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HXFkyexdu3c5953UR2gnypKP/8Y= 2024-02-01T10:48:23.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HXFkyexdu3c5953UR2gnypKP/8Y=', 'Content-Length': '0'} 2024-02-01T10:48:23.191 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4b17c568df0fb09-0065bb76f7-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:23.191 DEBUG:boto:b'vclfcn-61obj41000falseobj42024-02-01T10:46:46.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:23.191 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:23.191 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:23.192 DEBUG:boto:path=/vclfcn-61/?prefix=obj4 2024-02-01T10:48:23.192 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj4 2024-02-01T10:48:23.192 DEBUG:boto:Method: GET 2024-02-01T10:48:23.192 DEBUG:boto:Path: /vclfcn-61/?prefix=obj4 2024-02-01T10:48:23.192 DEBUG:boto:Data: 2024-02-01T10:48:23.192 DEBUG:boto:Headers: {} 2024-02-01T10:48:23.192 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:23.192 DEBUG:boto:Port: 8000 2024-02-01T10:48:23.192 DEBUG:boto:Params: {} 2024-02-01T10:48:23.192 DEBUG:boto:Token: None 2024-02-01T10:48:23.193 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:23 GMT /vclfcn-61/ 2024-02-01T10:48:23.193 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HXFkyexdu3c5953UR2gnypKP/8Y= 2024-02-01T10:48:23.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HXFkyexdu3c5953UR2gnypKP/8Y=', 'Content-Length': '0'} 2024-02-01T10:48:23.197 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ea55cf4fa60ac17-0065bb76f7-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:23.241 DEBUG:boto:b'vclfcn-61obj41000falseobj42024-02-01T10:46:46.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:23.241 DEBUG:boto:path=/vclfcn-61/obj4 2024-02-01T10:48:23.241 DEBUG:boto:auth_path=/vclfcn-61/obj4 2024-02-01T10:48:23.241 DEBUG:boto:Method: HEAD 2024-02-01T10:48:23.241 DEBUG:boto:Path: /vclfcn-61/obj4 2024-02-01T10:48:23.242 DEBUG:boto:Data: 2024-02-01T10:48:23.242 DEBUG:boto:Headers: {} 2024-02-01T10:48:23.242 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:23.242 DEBUG:boto:Port: 8001 2024-02-01T10:48:23.242 DEBUG:boto:Params: {} 2024-02-01T10:48:23.242 DEBUG:boto:Token: None 2024-02-01T10:48:23.242 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:23 GMT /vclfcn-61/obj4 2024-02-01T10:48:23.242 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ITioXaLGrlelePn1X+zsOK61VT8= 2024-02-01T10:48:23.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ITioXaLGrlelePn1X+zsOK61VT8=', 'Content-Length': '0'} 2024-02-01T10:48:23.245 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e9cd9518c8bc6489-0065bb76f7-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:23.246 DEBUG:boto:path=/vclfcn-61/obj4 2024-02-01T10:48:23.246 DEBUG:boto:auth_path=/vclfcn-61/obj4 2024-02-01T10:48:23.246 DEBUG:boto:Method: HEAD 2024-02-01T10:48:23.246 DEBUG:boto:Path: /vclfcn-61/obj4 2024-02-01T10:48:23.246 DEBUG:boto:Data: 2024-02-01T10:48:23.246 DEBUG:boto:Headers: {} 2024-02-01T10:48:23.246 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:23.246 DEBUG:boto:Port: 8000 2024-02-01T10:48:23.247 DEBUG:boto:Params: {} 2024-02-01T10:48:23.247 DEBUG:boto:Token: None 2024-02-01T10:48:23.247 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:23 GMT /vclfcn-61/obj4 2024-02-01T10:48:23.247 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ITioXaLGrlelePn1X+zsOK61VT8= 2024-02-01T10:48:23.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ITioXaLGrlelePn1X+zsOK61VT8=', 'Content-Length': '0'} 2024-02-01T10:48:23.250 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009901f949bd2555e1-0065bb76f7-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:23.250 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:48:23.251 DEBUG:boto:path=/vclfcn-61/obj4 2024-02-01T10:48:23.251 DEBUG:boto:auth_path=/vclfcn-61/obj4 2024-02-01T10:48:23.251 DEBUG:boto:Method: GET 2024-02-01T10:48:23.251 DEBUG:boto:Path: /vclfcn-61/obj4 2024-02-01T10:48:23.251 DEBUG:boto:Data: 2024-02-01T10:48:23.251 DEBUG:boto:Headers: {} 2024-02-01T10:48:23.251 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:23.251 DEBUG:boto:Port: 8001 2024-02-01T10:48:23.251 DEBUG:boto:Params: {} 2024-02-01T10:48:23.251 DEBUG:boto:Token: None 2024-02-01T10:48:23.252 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:23 GMT /vclfcn-61/obj4 2024-02-01T10:48:23.252 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HK4q20I0s8FU1GA/zSeyl55rzx4= 2024-02-01T10:48:23.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HK4q20I0s8FU1GA/zSeyl55rzx4=', 'Content-Length': '0'} 2024-02-01T10:48:23.255 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000afbe35492242838e-0065bb76f7-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:23.296 DEBUG:boto:path=/vclfcn-61/obj4 2024-02-01T10:48:23.296 DEBUG:boto:auth_path=/vclfcn-61/obj4 2024-02-01T10:48:23.296 DEBUG:boto:Method: GET 2024-02-01T10:48:23.297 DEBUG:boto:Path: /vclfcn-61/obj4 2024-02-01T10:48:23.297 DEBUG:boto:Data: 2024-02-01T10:48:23.297 DEBUG:boto:Headers: {} 2024-02-01T10:48:23.297 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:23.297 DEBUG:boto:Port: 8000 2024-02-01T10:48:23.297 DEBUG:boto:Params: {} 2024-02-01T10:48:23.297 DEBUG:boto:Token: None 2024-02-01T10:48:23.297 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:23 GMT /vclfcn-61/obj4 2024-02-01T10:48:23.297 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HK4q20I0s8FU1GA/zSeyl55rzx4= 2024-02-01T10:48:23.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HK4q20I0s8FU1GA/zSeyl55rzx4=', 'Content-Length': '0'} 2024-02-01T10:48:23.300 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:46:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e3a260cde47cdf87-0065bb76f7-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:23.344 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:23.344 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:23.344 DEBUG:boto:path=/vclfcn-61/?prefix=obj4 2024-02-01T10:48:23.344 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj4 2024-02-01T10:48:23.345 DEBUG:boto:Method: GET 2024-02-01T10:48:23.345 DEBUG:boto:Path: /vclfcn-61/?prefix=obj4 2024-02-01T10:48:23.345 DEBUG:boto:Data: 2024-02-01T10:48:23.345 DEBUG:boto:Headers: {} 2024-02-01T10:48:23.345 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:23.345 DEBUG:boto:Port: 8001 2024-02-01T10:48:23.345 DEBUG:boto:Params: {} 2024-02-01T10:48:23.345 DEBUG:boto:Token: None 2024-02-01T10:48:23.345 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:23 GMT /vclfcn-61/ 2024-02-01T10:48:23.345 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HXFkyexdu3c5953UR2gnypKP/8Y= 2024-02-01T10:48:23.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HXFkyexdu3c5953UR2gnypKP/8Y=', 'Content-Length': '0'} 2024-02-01T10:48:23.350 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f2a5c750f8ff09e-0065bb76f7-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:23.351 DEBUG:boto:b'vclfcn-61obj41000falseobj42024-02-01T10:46:46.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:23.352 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:23.352 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:23.352 DEBUG:boto:path=/vclfcn-61/?prefix=obj4 2024-02-01T10:48:23.352 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj4 2024-02-01T10:48:23.352 DEBUG:boto:Method: GET 2024-02-01T10:48:23.352 DEBUG:boto:Path: /vclfcn-61/?prefix=obj4 2024-02-01T10:48:23.352 DEBUG:boto:Data: 2024-02-01T10:48:23.352 DEBUG:boto:Headers: {} 2024-02-01T10:48:23.352 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:23.352 DEBUG:boto:Port: 8000 2024-02-01T10:48:23.352 DEBUG:boto:Params: {} 2024-02-01T10:48:23.353 DEBUG:boto:Token: None 2024-02-01T10:48:23.353 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:23 GMT /vclfcn-61/ 2024-02-01T10:48:23.353 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HXFkyexdu3c5953UR2gnypKP/8Y= 2024-02-01T10:48:23.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HXFkyexdu3c5953UR2gnypKP/8Y=', 'Content-Length': '0'} 2024-02-01T10:48:23.357 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b7c8ed98f1cf1fd-0065bb76f7-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:23.357 DEBUG:boto:b'vclfcn-61obj41000falseobj42024-02-01T10:46:46.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:23.357 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-02-01T10:48:23.357 DEBUG:boto:path=/vclfcn-61/obj5 2024-02-01T10:48:23.358 DEBUG:boto:auth_path=/vclfcn-61/obj5 2024-02-01T10:48:23.358 DEBUG:boto:path=/vclfcn-61/obj5?versionId=null 2024-02-01T10:48:23.358 DEBUG:boto:auth_path=/vclfcn-61/obj5?versionId=null 2024-02-01T10:48:23.358 DEBUG:boto:Method: GET 2024-02-01T10:48:23.358 DEBUG:boto:Path: /vclfcn-61/obj5?versionId=null 2024-02-01T10:48:23.358 DEBUG:boto:Data: 2024-02-01T10:48:23.358 DEBUG:boto:Headers: {} 2024-02-01T10:48:23.358 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:23.358 DEBUG:boto:Port: 8001 2024-02-01T10:48:23.358 DEBUG:boto:Params: {} 2024-02-01T10:48:23.358 DEBUG:boto:Token: None 2024-02-01T10:48:23.359 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:23 GMT /vclfcn-61/obj5?versionId=null 2024-02-01T10:48:23.359 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0Y/yVcVNMHN/e6BYVZJEJGyWxHs= 2024-02-01T10:48:23.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0Y/yVcVNMHN/e6BYVZJEJGyWxHs=', 'Content-Length': '0'} 2024-02-01T10:48:23.362 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005c75635c4bc873ac-0065bb76f7-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:23.362 DEBUG:boto:path=/vclfcn-61/obj5 2024-02-01T10:48:23.362 DEBUG:boto:auth_path=/vclfcn-61/obj5 2024-02-01T10:48:23.362 DEBUG:boto:path=/vclfcn-61/obj5?versionId=null 2024-02-01T10:48:23.362 DEBUG:boto:auth_path=/vclfcn-61/obj5?versionId=null 2024-02-01T10:48:23.362 DEBUG:boto:Method: GET 2024-02-01T10:48:23.362 DEBUG:boto:Path: /vclfcn-61/obj5?versionId=null 2024-02-01T10:48:23.362 DEBUG:boto:Data: 2024-02-01T10:48:23.363 DEBUG:boto:Headers: {} 2024-02-01T10:48:23.363 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:23.363 DEBUG:boto:Port: 8000 2024-02-01T10:48:23.363 DEBUG:boto:Params: {} 2024-02-01T10:48:23.363 DEBUG:boto:Token: None 2024-02-01T10:48:23.363 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:23 GMT /vclfcn-61/obj5?versionId=null 2024-02-01T10:48:23.363 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0Y/yVcVNMHN/e6BYVZJEJGyWxHs= 2024-02-01T10:48:23.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0Y/yVcVNMHN/e6BYVZJEJGyWxHs=', 'Content-Length': '0'} 2024-02-01T10:48:23.366 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a7b932f4c09e9167-0065bb76f7-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:23.408 DEBUG:boto:path=/vclfcn-61/obj5 2024-02-01T10:48:23.408 DEBUG:boto:auth_path=/vclfcn-61/obj5 2024-02-01T10:48:23.408 DEBUG:boto:path=/vclfcn-61/obj5?versionId=null 2024-02-01T10:48:23.409 DEBUG:boto:auth_path=/vclfcn-61/obj5?versionId=null 2024-02-01T10:48:23.409 DEBUG:boto:Method: HEAD 2024-02-01T10:48:23.409 DEBUG:boto:Path: /vclfcn-61/obj5?versionId=null 2024-02-01T10:48:23.409 DEBUG:boto:Data: 2024-02-01T10:48:23.409 DEBUG:boto:Headers: {} 2024-02-01T10:48:23.409 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:23.409 DEBUG:boto:Port: 8001 2024-02-01T10:48:23.409 DEBUG:boto:Params: {} 2024-02-01T10:48:23.409 DEBUG:boto:Token: None 2024-02-01T10:48:23.409 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:23 GMT /vclfcn-61/obj5?versionId=null 2024-02-01T10:48:23.410 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:UdfQ6Tg1ReisFnqp0+izBZX1rTw= 2024-02-01T10:48:23.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:UdfQ6Tg1ReisFnqp0+izBZX1rTw=', 'Content-Length': '0'} 2024-02-01T10:48:23.413 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000050d7d9f352edf468-0065bb76f7-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:23.413 DEBUG:boto:path=/vclfcn-61/obj5 2024-02-01T10:48:23.413 DEBUG:boto:auth_path=/vclfcn-61/obj5 2024-02-01T10:48:23.413 DEBUG:boto:path=/vclfcn-61/obj5?versionId=null 2024-02-01T10:48:23.413 DEBUG:boto:auth_path=/vclfcn-61/obj5?versionId=null 2024-02-01T10:48:23.413 DEBUG:boto:Method: HEAD 2024-02-01T10:48:23.413 DEBUG:boto:Path: /vclfcn-61/obj5?versionId=null 2024-02-01T10:48:23.414 DEBUG:boto:Data: 2024-02-01T10:48:23.414 DEBUG:boto:Headers: {} 2024-02-01T10:48:23.414 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:23.414 DEBUG:boto:Port: 8000 2024-02-01T10:48:23.414 DEBUG:boto:Params: {} 2024-02-01T10:48:23.414 DEBUG:boto:Token: None 2024-02-01T10:48:23.414 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:23 GMT /vclfcn-61/obj5?versionId=null 2024-02-01T10:48:23.414 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:UdfQ6Tg1ReisFnqp0+izBZX1rTw= 2024-02-01T10:48:23.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:UdfQ6Tg1ReisFnqp0+izBZX1rTw=', 'Content-Length': '0'} 2024-02-01T10:48:23.420 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ee38238e966ff92d-0065bb76f7-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:23.421 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-02-01T10:48:23.421 DEBUG:boto:path=/vclfcn-61/obj5 2024-02-01T10:48:23.421 DEBUG:boto:auth_path=/vclfcn-61/obj5 2024-02-01T10:48:23.421 DEBUG:boto:path=/vclfcn-61/obj5?versionId=null 2024-02-01T10:48:23.421 DEBUG:boto:auth_path=/vclfcn-61/obj5?versionId=null 2024-02-01T10:48:23.421 DEBUG:boto:Method: GET 2024-02-01T10:48:23.421 DEBUG:boto:Path: /vclfcn-61/obj5?versionId=null 2024-02-01T10:48:23.421 DEBUG:boto:Data: 2024-02-01T10:48:23.422 DEBUG:boto:Headers: {} 2024-02-01T10:48:23.422 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:23.422 DEBUG:boto:Port: 8001 2024-02-01T10:48:23.422 DEBUG:boto:Params: {} 2024-02-01T10:48:23.422 DEBUG:boto:Token: None 2024-02-01T10:48:23.422 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:23 GMT /vclfcn-61/obj5?versionId=null 2024-02-01T10:48:23.422 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0Y/yVcVNMHN/e6BYVZJEJGyWxHs= 2024-02-01T10:48:23.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0Y/yVcVNMHN/e6BYVZJEJGyWxHs=', 'Content-Length': '0'} 2024-02-01T10:48:23.425 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004c2d1acc99c1b958-0065bb76f7-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:23.425 DEBUG:boto:path=/vclfcn-61/obj5 2024-02-01T10:48:23.425 DEBUG:boto:auth_path=/vclfcn-61/obj5 2024-02-01T10:48:23.426 DEBUG:boto:path=/vclfcn-61/obj5?versionId=null 2024-02-01T10:48:23.426 DEBUG:boto:auth_path=/vclfcn-61/obj5?versionId=null 2024-02-01T10:48:23.426 DEBUG:boto:Method: GET 2024-02-01T10:48:23.426 DEBUG:boto:Path: /vclfcn-61/obj5?versionId=null 2024-02-01T10:48:23.426 DEBUG:boto:Data: 2024-02-01T10:48:23.426 DEBUG:boto:Headers: {} 2024-02-01T10:48:23.426 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:23.426 DEBUG:boto:Port: 8000 2024-02-01T10:48:23.426 DEBUG:boto:Params: {} 2024-02-01T10:48:23.426 DEBUG:boto:Token: None 2024-02-01T10:48:23.427 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:23 GMT /vclfcn-61/obj5?versionId=null 2024-02-01T10:48:23.427 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0Y/yVcVNMHN/e6BYVZJEJGyWxHs= 2024-02-01T10:48:23.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0Y/yVcVNMHN/e6BYVZJEJGyWxHs=', 'Content-Length': '0'} 2024-02-01T10:48:23.430 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007ff8c6b24b98929b-0065bb76f7-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:23.472 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:23.472 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:23.472 DEBUG:boto:path=/vclfcn-61/?prefix=obj5 2024-02-01T10:48:23.473 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj5 2024-02-01T10:48:23.473 DEBUG:boto:Method: GET 2024-02-01T10:48:23.473 DEBUG:boto:Path: /vclfcn-61/?prefix=obj5 2024-02-01T10:48:23.473 DEBUG:boto:Data: 2024-02-01T10:48:23.473 DEBUG:boto:Headers: {} 2024-02-01T10:48:23.473 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:23.473 DEBUG:boto:Port: 8001 2024-02-01T10:48:23.473 DEBUG:boto:Params: {} 2024-02-01T10:48:23.473 DEBUG:boto:Token: None 2024-02-01T10:48:23.473 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:23 GMT /vclfcn-61/ 2024-02-01T10:48:23.474 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HXFkyexdu3c5953UR2gnypKP/8Y= 2024-02-01T10:48:23.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HXFkyexdu3c5953UR2gnypKP/8Y=', 'Content-Length': '0'} 2024-02-01T10:48:23.478 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b583ee0b7af57933-0065bb76f7-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:23.479 DEBUG:boto:b'vclfcn-61obj51000falseobj52024-02-01T10:47:30.425Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:23.480 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:23.480 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:23.480 DEBUG:boto:path=/vclfcn-61/?prefix=obj5 2024-02-01T10:48:23.480 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj5 2024-02-01T10:48:23.480 DEBUG:boto:Method: GET 2024-02-01T10:48:23.480 DEBUG:boto:Path: /vclfcn-61/?prefix=obj5 2024-02-01T10:48:23.480 DEBUG:boto:Data: 2024-02-01T10:48:23.481 DEBUG:boto:Headers: {} 2024-02-01T10:48:23.481 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:23.481 DEBUG:boto:Port: 8000 2024-02-01T10:48:23.481 DEBUG:boto:Params: {} 2024-02-01T10:48:23.481 DEBUG:boto:Token: None 2024-02-01T10:48:23.481 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:23 GMT /vclfcn-61/ 2024-02-01T10:48:23.481 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HXFkyexdu3c5953UR2gnypKP/8Y= 2024-02-01T10:48:23.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HXFkyexdu3c5953UR2gnypKP/8Y=', 'Content-Length': '0'} 2024-02-01T10:48:23.486 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a553d4c1301cf98f-0065bb76f7-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:23.532 DEBUG:boto:b'vclfcn-61obj51000falseobj52024-02-01T10:47:30.425Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:23.532 DEBUG:boto:path=/vclfcn-61/obj5 2024-02-01T10:48:23.533 DEBUG:boto:auth_path=/vclfcn-61/obj5 2024-02-01T10:48:23.533 DEBUG:boto:Method: HEAD 2024-02-01T10:48:23.533 DEBUG:boto:Path: /vclfcn-61/obj5 2024-02-01T10:48:23.533 DEBUG:boto:Data: 2024-02-01T10:48:23.533 DEBUG:boto:Headers: {} 2024-02-01T10:48:23.533 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:23.533 DEBUG:boto:Port: 8001 2024-02-01T10:48:23.533 DEBUG:boto:Params: {} 2024-02-01T10:48:23.533 DEBUG:boto:Token: None 2024-02-01T10:48:23.534 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:23 GMT /vclfcn-61/obj5 2024-02-01T10:48:23.534 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BT7L+eo6dCXFVM4oZA8rDzS7Qp4= 2024-02-01T10:48:23.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BT7L+eo6dCXFVM4oZA8rDzS7Qp4=', 'Content-Length': '0'} 2024-02-01T10:48:23.537 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000057e87db7618a9e4f-0065bb76f7-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:23.537 DEBUG:boto:path=/vclfcn-61/obj5 2024-02-01T10:48:23.537 DEBUG:boto:auth_path=/vclfcn-61/obj5 2024-02-01T10:48:23.537 DEBUG:boto:Method: HEAD 2024-02-01T10:48:23.537 DEBUG:boto:Path: /vclfcn-61/obj5 2024-02-01T10:48:23.537 DEBUG:boto:Data: 2024-02-01T10:48:23.537 DEBUG:boto:Headers: {} 2024-02-01T10:48:23.537 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:23.537 DEBUG:boto:Port: 8000 2024-02-01T10:48:23.538 DEBUG:boto:Params: {} 2024-02-01T10:48:23.538 DEBUG:boto:Token: None 2024-02-01T10:48:23.538 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:23 GMT /vclfcn-61/obj5 2024-02-01T10:48:23.538 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BT7L+eo6dCXFVM4oZA8rDzS7Qp4= 2024-02-01T10:48:23.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BT7L+eo6dCXFVM4oZA8rDzS7Qp4=', 'Content-Length': '0'} 2024-02-01T10:48:23.541 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000030cdd2b16e50712a-0065bb76f7-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:23.541 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-02-01T10:48:23.541 DEBUG:boto:path=/vclfcn-61/obj5 2024-02-01T10:48:23.542 DEBUG:boto:auth_path=/vclfcn-61/obj5 2024-02-01T10:48:23.542 DEBUG:boto:Method: GET 2024-02-01T10:48:23.542 DEBUG:boto:Path: /vclfcn-61/obj5 2024-02-01T10:48:23.542 DEBUG:boto:Data: 2024-02-01T10:48:23.542 DEBUG:boto:Headers: {} 2024-02-01T10:48:23.542 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:23.542 DEBUG:boto:Port: 8001 2024-02-01T10:48:23.542 DEBUG:boto:Params: {} 2024-02-01T10:48:23.542 DEBUG:boto:Token: None 2024-02-01T10:48:23.542 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:23 GMT /vclfcn-61/obj5 2024-02-01T10:48:23.543 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mbSPntMtWp7PSOXrIpc3q32haO4= 2024-02-01T10:48:23.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mbSPntMtWp7PSOXrIpc3q32haO4=', 'Content-Length': '0'} 2024-02-01T10:48:23.545 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002ae768dc518d92e5-0065bb76f7-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:23.588 DEBUG:boto:path=/vclfcn-61/obj5 2024-02-01T10:48:23.589 DEBUG:boto:auth_path=/vclfcn-61/obj5 2024-02-01T10:48:23.589 DEBUG:boto:Method: GET 2024-02-01T10:48:23.589 DEBUG:boto:Path: /vclfcn-61/obj5 2024-02-01T10:48:23.589 DEBUG:boto:Data: 2024-02-01T10:48:23.589 DEBUG:boto:Headers: {} 2024-02-01T10:48:23.589 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:23.589 DEBUG:boto:Port: 8000 2024-02-01T10:48:23.589 DEBUG:boto:Params: {} 2024-02-01T10:48:23.589 DEBUG:boto:Token: None 2024-02-01T10:48:23.590 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:23 GMT /vclfcn-61/obj5 2024-02-01T10:48:23.590 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mbSPntMtWp7PSOXrIpc3q32haO4= 2024-02-01T10:48:23.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mbSPntMtWp7PSOXrIpc3q32haO4=', 'Content-Length': '0'} 2024-02-01T10:48:23.593 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001d1d30cdda69e50f-0065bb76f7-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:23.636 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:23.636 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:23.637 DEBUG:boto:path=/vclfcn-61/?prefix=obj5 2024-02-01T10:48:23.637 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj5 2024-02-01T10:48:23.637 DEBUG:boto:Method: GET 2024-02-01T10:48:23.637 DEBUG:boto:Path: /vclfcn-61/?prefix=obj5 2024-02-01T10:48:23.637 DEBUG:boto:Data: 2024-02-01T10:48:23.637 DEBUG:boto:Headers: {} 2024-02-01T10:48:23.637 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:23.637 DEBUG:boto:Port: 8001 2024-02-01T10:48:23.637 DEBUG:boto:Params: {} 2024-02-01T10:48:23.638 DEBUG:boto:Token: None 2024-02-01T10:48:23.638 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:23 GMT /vclfcn-61/ 2024-02-01T10:48:23.638 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HXFkyexdu3c5953UR2gnypKP/8Y= 2024-02-01T10:48:23.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HXFkyexdu3c5953UR2gnypKP/8Y=', 'Content-Length': '0'} 2024-02-01T10:48:23.643 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e1279601d737da04-0065bb76f7-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:23.643 DEBUG:boto:b'vclfcn-61obj51000falseobj52024-02-01T10:47:30.425Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:23.644 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:23.644 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:23.644 DEBUG:boto:path=/vclfcn-61/?prefix=obj5 2024-02-01T10:48:23.644 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj5 2024-02-01T10:48:23.644 DEBUG:boto:Method: GET 2024-02-01T10:48:23.644 DEBUG:boto:Path: /vclfcn-61/?prefix=obj5 2024-02-01T10:48:23.644 DEBUG:boto:Data: 2024-02-01T10:48:23.644 DEBUG:boto:Headers: {} 2024-02-01T10:48:23.644 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:23.644 DEBUG:boto:Port: 8000 2024-02-01T10:48:23.644 DEBUG:boto:Params: {} 2024-02-01T10:48:23.645 DEBUG:boto:Token: None 2024-02-01T10:48:23.645 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:23 GMT /vclfcn-61/ 2024-02-01T10:48:23.645 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HXFkyexdu3c5953UR2gnypKP/8Y= 2024-02-01T10:48:23.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HXFkyexdu3c5953UR2gnypKP/8Y=', 'Content-Length': '0'} 2024-02-01T10:48:23.650 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000194ee2e5aee9a418-0065bb76f7-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:23.650 DEBUG:boto:b'vclfcn-61obj51000falseobj52024-02-01T10:47:30.425Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:23.650 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-02-01T10:48:23.650 DEBUG:boto:path=/vclfcn-61/obj6 2024-02-01T10:48:23.651 DEBUG:boto:auth_path=/vclfcn-61/obj6 2024-02-01T10:48:23.651 DEBUG:boto:path=/vclfcn-61/obj6?versionId=null 2024-02-01T10:48:23.651 DEBUG:boto:auth_path=/vclfcn-61/obj6?versionId=null 2024-02-01T10:48:23.651 DEBUG:boto:Method: GET 2024-02-01T10:48:23.651 DEBUG:boto:Path: /vclfcn-61/obj6?versionId=null 2024-02-01T10:48:23.651 DEBUG:boto:Data: 2024-02-01T10:48:23.651 DEBUG:boto:Headers: {} 2024-02-01T10:48:23.651 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:23.651 DEBUG:boto:Port: 8001 2024-02-01T10:48:23.651 DEBUG:boto:Params: {} 2024-02-01T10:48:23.651 DEBUG:boto:Token: None 2024-02-01T10:48:23.652 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:23 GMT /vclfcn-61/obj6?versionId=null 2024-02-01T10:48:23.652 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:pcTIU6WPASUoxlR5t/qSlTITJFs= 2024-02-01T10:48:23.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:pcTIU6WPASUoxlR5t/qSlTITJFs=', 'Content-Length': '0'} 2024-02-01T10:48:23.655 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002465ce521e024255-0065bb76f7-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:23.655 DEBUG:boto:path=/vclfcn-61/obj6 2024-02-01T10:48:23.655 DEBUG:boto:auth_path=/vclfcn-61/obj6 2024-02-01T10:48:23.655 DEBUG:boto:path=/vclfcn-61/obj6?versionId=null 2024-02-01T10:48:23.655 DEBUG:boto:auth_path=/vclfcn-61/obj6?versionId=null 2024-02-01T10:48:23.655 DEBUG:boto:Method: GET 2024-02-01T10:48:23.655 DEBUG:boto:Path: /vclfcn-61/obj6?versionId=null 2024-02-01T10:48:23.656 DEBUG:boto:Data: 2024-02-01T10:48:23.656 DEBUG:boto:Headers: {} 2024-02-01T10:48:23.656 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:23.656 DEBUG:boto:Port: 8000 2024-02-01T10:48:23.656 DEBUG:boto:Params: {} 2024-02-01T10:48:23.656 DEBUG:boto:Token: None 2024-02-01T10:48:23.656 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:23 GMT /vclfcn-61/obj6?versionId=null 2024-02-01T10:48:23.656 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:pcTIU6WPASUoxlR5t/qSlTITJFs= 2024-02-01T10:48:23.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:pcTIU6WPASUoxlR5t/qSlTITJFs=', 'Content-Length': '0'} 2024-02-01T10:48:23.659 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000be8ef3ac3c068604-0065bb76f7-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:23.704 DEBUG:boto:path=/vclfcn-61/obj6 2024-02-01T10:48:23.704 DEBUG:boto:auth_path=/vclfcn-61/obj6 2024-02-01T10:48:23.704 DEBUG:boto:path=/vclfcn-61/obj6?versionId=null 2024-02-01T10:48:23.704 DEBUG:boto:auth_path=/vclfcn-61/obj6?versionId=null 2024-02-01T10:48:23.705 DEBUG:boto:Method: HEAD 2024-02-01T10:48:23.705 DEBUG:boto:Path: /vclfcn-61/obj6?versionId=null 2024-02-01T10:48:23.705 DEBUG:boto:Data: 2024-02-01T10:48:23.705 DEBUG:boto:Headers: {} 2024-02-01T10:48:23.705 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:23.705 DEBUG:boto:Port: 8001 2024-02-01T10:48:23.705 DEBUG:boto:Params: {} 2024-02-01T10:48:23.705 DEBUG:boto:Token: None 2024-02-01T10:48:23.706 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:23 GMT /vclfcn-61/obj6?versionId=null 2024-02-01T10:48:23.706 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lIhmYFsUDvJsw5+CCE3ldnXzuw4= 2024-02-01T10:48:23.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lIhmYFsUDvJsw5+CCE3ldnXzuw4=', 'Content-Length': '0'} 2024-02-01T10:48:23.708 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003f537ce5870c90bd-0065bb76f7-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:23.709 DEBUG:boto:path=/vclfcn-61/obj6 2024-02-01T10:48:23.709 DEBUG:boto:auth_path=/vclfcn-61/obj6 2024-02-01T10:48:23.709 DEBUG:boto:path=/vclfcn-61/obj6?versionId=null 2024-02-01T10:48:23.709 DEBUG:boto:auth_path=/vclfcn-61/obj6?versionId=null 2024-02-01T10:48:23.709 DEBUG:boto:Method: HEAD 2024-02-01T10:48:23.709 DEBUG:boto:Path: /vclfcn-61/obj6?versionId=null 2024-02-01T10:48:23.709 DEBUG:boto:Data: 2024-02-01T10:48:23.709 DEBUG:boto:Headers: {} 2024-02-01T10:48:23.709 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:23.709 DEBUG:boto:Port: 8000 2024-02-01T10:48:23.710 DEBUG:boto:Params: {} 2024-02-01T10:48:23.710 DEBUG:boto:Token: None 2024-02-01T10:48:23.710 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:23 GMT /vclfcn-61/obj6?versionId=null 2024-02-01T10:48:23.710 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lIhmYFsUDvJsw5+CCE3ldnXzuw4= 2024-02-01T10:48:23.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lIhmYFsUDvJsw5+CCE3ldnXzuw4=', 'Content-Length': '0'} 2024-02-01T10:48:23.713 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000231575461f37e2d-0065bb76f7-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:23.713 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-02-01T10:48:23.713 DEBUG:boto:path=/vclfcn-61/obj6 2024-02-01T10:48:23.713 DEBUG:boto:auth_path=/vclfcn-61/obj6 2024-02-01T10:48:23.714 DEBUG:boto:path=/vclfcn-61/obj6?versionId=null 2024-02-01T10:48:23.714 DEBUG:boto:auth_path=/vclfcn-61/obj6?versionId=null 2024-02-01T10:48:23.714 DEBUG:boto:Method: GET 2024-02-01T10:48:23.714 DEBUG:boto:Path: /vclfcn-61/obj6?versionId=null 2024-02-01T10:48:23.714 DEBUG:boto:Data: 2024-02-01T10:48:23.714 DEBUG:boto:Headers: {} 2024-02-01T10:48:23.714 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:23.714 DEBUG:boto:Port: 8001 2024-02-01T10:48:23.714 DEBUG:boto:Params: {} 2024-02-01T10:48:23.714 DEBUG:boto:Token: None 2024-02-01T10:48:23.714 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:23 GMT /vclfcn-61/obj6?versionId=null 2024-02-01T10:48:23.715 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:pcTIU6WPASUoxlR5t/qSlTITJFs= 2024-02-01T10:48:23.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:pcTIU6WPASUoxlR5t/qSlTITJFs=', 'Content-Length': '0'} 2024-02-01T10:48:23.718 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002ffeaa00309bd886-0065bb76f7-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:23.719 DEBUG:boto:path=/vclfcn-61/obj6 2024-02-01T10:48:23.719 DEBUG:boto:auth_path=/vclfcn-61/obj6 2024-02-01T10:48:23.719 DEBUG:boto:path=/vclfcn-61/obj6?versionId=null 2024-02-01T10:48:23.719 DEBUG:boto:auth_path=/vclfcn-61/obj6?versionId=null 2024-02-01T10:48:23.720 DEBUG:boto:Method: GET 2024-02-01T10:48:23.720 DEBUG:boto:Path: /vclfcn-61/obj6?versionId=null 2024-02-01T10:48:23.720 DEBUG:boto:Data: 2024-02-01T10:48:23.720 DEBUG:boto:Headers: {} 2024-02-01T10:48:23.720 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:23.720 DEBUG:boto:Port: 8000 2024-02-01T10:48:23.720 DEBUG:boto:Params: {} 2024-02-01T10:48:23.720 DEBUG:boto:Token: None 2024-02-01T10:48:23.720 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:23 GMT /vclfcn-61/obj6?versionId=null 2024-02-01T10:48:23.720 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:pcTIU6WPASUoxlR5t/qSlTITJFs= 2024-02-01T10:48:23.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:pcTIU6WPASUoxlR5t/qSlTITJFs=', 'Content-Length': '0'} 2024-02-01T10:48:23.723 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000009cea9f516b5cb87-0065bb76f7-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:23.764 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:23.764 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:23.764 DEBUG:boto:path=/vclfcn-61/?prefix=obj6 2024-02-01T10:48:23.764 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj6 2024-02-01T10:48:23.765 DEBUG:boto:Method: GET 2024-02-01T10:48:23.765 DEBUG:boto:Path: /vclfcn-61/?prefix=obj6 2024-02-01T10:48:23.765 DEBUG:boto:Data: 2024-02-01T10:48:23.765 DEBUG:boto:Headers: {} 2024-02-01T10:48:23.765 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:23.765 DEBUG:boto:Port: 8001 2024-02-01T10:48:23.765 DEBUG:boto:Params: {} 2024-02-01T10:48:23.765 DEBUG:boto:Token: None 2024-02-01T10:48:23.765 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:23 GMT /vclfcn-61/ 2024-02-01T10:48:23.765 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HXFkyexdu3c5953UR2gnypKP/8Y= 2024-02-01T10:48:23.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HXFkyexdu3c5953UR2gnypKP/8Y=', 'Content-Length': '0'} 2024-02-01T10:48:23.771 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f39e2e8f5d90b001-0065bb76f7-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:23.771 DEBUG:boto:b'vclfcn-61obj61000falseobj62024-02-01T10:47:30.440Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:23.771 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:23.771 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:23.771 DEBUG:boto:path=/vclfcn-61/?prefix=obj6 2024-02-01T10:48:23.772 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj6 2024-02-01T10:48:23.772 DEBUG:boto:Method: GET 2024-02-01T10:48:23.772 DEBUG:boto:Path: /vclfcn-61/?prefix=obj6 2024-02-01T10:48:23.772 DEBUG:boto:Data: 2024-02-01T10:48:23.772 DEBUG:boto:Headers: {} 2024-02-01T10:48:23.772 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:23.772 DEBUG:boto:Port: 8000 2024-02-01T10:48:23.772 DEBUG:boto:Params: {} 2024-02-01T10:48:23.772 DEBUG:boto:Token: None 2024-02-01T10:48:23.772 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:23 GMT /vclfcn-61/ 2024-02-01T10:48:23.773 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HXFkyexdu3c5953UR2gnypKP/8Y= 2024-02-01T10:48:23.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HXFkyexdu3c5953UR2gnypKP/8Y=', 'Content-Length': '0'} 2024-02-01T10:48:23.777 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001813d7179ccd8da3-0065bb76f7-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:23.820 DEBUG:boto:b'vclfcn-61obj61000falseobj62024-02-01T10:47:30.440Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:23.821 DEBUG:boto:path=/vclfcn-61/obj6 2024-02-01T10:48:23.821 DEBUG:boto:auth_path=/vclfcn-61/obj6 2024-02-01T10:48:23.821 DEBUG:boto:Method: HEAD 2024-02-01T10:48:23.821 DEBUG:boto:Path: /vclfcn-61/obj6 2024-02-01T10:48:23.821 DEBUG:boto:Data: 2024-02-01T10:48:23.821 DEBUG:boto:Headers: {} 2024-02-01T10:48:23.821 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:23.821 DEBUG:boto:Port: 8001 2024-02-01T10:48:23.822 DEBUG:boto:Params: {} 2024-02-01T10:48:23.822 DEBUG:boto:Token: None 2024-02-01T10:48:23.822 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:23 GMT /vclfcn-61/obj6 2024-02-01T10:48:23.822 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:oI3bpbO8sI2fPj9LxcZNiHeoYik= 2024-02-01T10:48:23.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:oI3bpbO8sI2fPj9LxcZNiHeoYik=', 'Content-Length': '0'} 2024-02-01T10:48:23.825 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002c17113b0c7cdc93-0065bb76f7-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:23.825 DEBUG:boto:path=/vclfcn-61/obj6 2024-02-01T10:48:23.825 DEBUG:boto:auth_path=/vclfcn-61/obj6 2024-02-01T10:48:23.825 DEBUG:boto:Method: HEAD 2024-02-01T10:48:23.825 DEBUG:boto:Path: /vclfcn-61/obj6 2024-02-01T10:48:23.825 DEBUG:boto:Data: 2024-02-01T10:48:23.825 DEBUG:boto:Headers: {} 2024-02-01T10:48:23.826 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:23.826 DEBUG:boto:Port: 8000 2024-02-01T10:48:23.826 DEBUG:boto:Params: {} 2024-02-01T10:48:23.826 DEBUG:boto:Token: None 2024-02-01T10:48:23.826 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:23 GMT /vclfcn-61/obj6 2024-02-01T10:48:23.826 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:oI3bpbO8sI2fPj9LxcZNiHeoYik= 2024-02-01T10:48:23.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:oI3bpbO8sI2fPj9LxcZNiHeoYik=', 'Content-Length': '0'} 2024-02-01T10:48:23.831 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000040e94813a3521950-0065bb76f7-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:23.832 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-02-01T10:48:23.832 DEBUG:boto:path=/vclfcn-61/obj6 2024-02-01T10:48:23.832 DEBUG:boto:auth_path=/vclfcn-61/obj6 2024-02-01T10:48:23.832 DEBUG:boto:Method: GET 2024-02-01T10:48:23.832 DEBUG:boto:Path: /vclfcn-61/obj6 2024-02-01T10:48:23.832 DEBUG:boto:Data: 2024-02-01T10:48:23.833 DEBUG:boto:Headers: {} 2024-02-01T10:48:23.833 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:23.833 DEBUG:boto:Port: 8001 2024-02-01T10:48:23.833 DEBUG:boto:Params: {} 2024-02-01T10:48:23.833 DEBUG:boto:Token: None 2024-02-01T10:48:23.833 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:23 GMT /vclfcn-61/obj6 2024-02-01T10:48:23.833 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:t7d5WEsFjaWOjoJp6QtAoJ3RvIQ= 2024-02-01T10:48:23.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:t7d5WEsFjaWOjoJp6QtAoJ3RvIQ=', 'Content-Length': '0'} 2024-02-01T10:48:23.836 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006e1c0c17ae4d3e7a-0065bb76f7-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:23.880 DEBUG:boto:path=/vclfcn-61/obj6 2024-02-01T10:48:23.880 DEBUG:boto:auth_path=/vclfcn-61/obj6 2024-02-01T10:48:23.880 DEBUG:boto:Method: GET 2024-02-01T10:48:23.880 DEBUG:boto:Path: /vclfcn-61/obj6 2024-02-01T10:48:23.881 DEBUG:boto:Data: 2024-02-01T10:48:23.881 DEBUG:boto:Headers: {} 2024-02-01T10:48:23.881 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:23.881 DEBUG:boto:Port: 8000 2024-02-01T10:48:23.881 DEBUG:boto:Params: {} 2024-02-01T10:48:23.881 DEBUG:boto:Token: None 2024-02-01T10:48:23.881 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:23 GMT /vclfcn-61/obj6 2024-02-01T10:48:23.881 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:t7d5WEsFjaWOjoJp6QtAoJ3RvIQ= 2024-02-01T10:48:23.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:t7d5WEsFjaWOjoJp6QtAoJ3RvIQ=', 'Content-Length': '0'} 2024-02-01T10:48:23.884 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000079277a99b9280308-0065bb76f7-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:23.928 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:23.928 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:23.928 DEBUG:boto:path=/vclfcn-61/?prefix=obj6 2024-02-01T10:48:23.929 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj6 2024-02-01T10:48:23.929 DEBUG:boto:Method: GET 2024-02-01T10:48:23.929 DEBUG:boto:Path: /vclfcn-61/?prefix=obj6 2024-02-01T10:48:23.929 DEBUG:boto:Data: 2024-02-01T10:48:23.929 DEBUG:boto:Headers: {} 2024-02-01T10:48:23.929 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:23.929 DEBUG:boto:Port: 8001 2024-02-01T10:48:23.929 DEBUG:boto:Params: {} 2024-02-01T10:48:23.929 DEBUG:boto:Token: None 2024-02-01T10:48:23.929 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:23 GMT /vclfcn-61/ 2024-02-01T10:48:23.930 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HXFkyexdu3c5953UR2gnypKP/8Y= 2024-02-01T10:48:23.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HXFkyexdu3c5953UR2gnypKP/8Y=', 'Content-Length': '0'} 2024-02-01T10:48:23.935 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006dca03279c03d8c6-0065bb76f7-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:23.936 DEBUG:boto:b'vclfcn-61obj61000falseobj62024-02-01T10:47:30.440Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:23.936 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:23.936 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:23.936 DEBUG:boto:path=/vclfcn-61/?prefix=obj6 2024-02-01T10:48:23.936 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj6 2024-02-01T10:48:23.937 DEBUG:boto:Method: GET 2024-02-01T10:48:23.937 DEBUG:boto:Path: /vclfcn-61/?prefix=obj6 2024-02-01T10:48:23.937 DEBUG:boto:Data: 2024-02-01T10:48:23.937 DEBUG:boto:Headers: {} 2024-02-01T10:48:23.937 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:23.937 DEBUG:boto:Port: 8000 2024-02-01T10:48:23.937 DEBUG:boto:Params: {} 2024-02-01T10:48:23.937 DEBUG:boto:Token: None 2024-02-01T10:48:23.937 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:23 GMT /vclfcn-61/ 2024-02-01T10:48:23.937 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HXFkyexdu3c5953UR2gnypKP/8Y= 2024-02-01T10:48:23.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HXFkyexdu3c5953UR2gnypKP/8Y=', 'Content-Length': '0'} 2024-02-01T10:48:23.941 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000016691e4ec4e58879-0065bb76f7-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:23.984 DEBUG:boto:b'vclfcn-61obj61000falseobj62024-02-01T10:47:30.440Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:23.984 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-02-01T10:48:23.985 DEBUG:boto:path=/vclfcn-61/obj7 2024-02-01T10:48:23.985 DEBUG:boto:auth_path=/vclfcn-61/obj7 2024-02-01T10:48:23.985 DEBUG:boto:path=/vclfcn-61/obj7?versionId=null 2024-02-01T10:48:23.985 DEBUG:boto:auth_path=/vclfcn-61/obj7?versionId=null 2024-02-01T10:48:23.985 DEBUG:boto:Method: GET 2024-02-01T10:48:23.985 DEBUG:boto:Path: /vclfcn-61/obj7?versionId=null 2024-02-01T10:48:23.985 DEBUG:boto:Data: 2024-02-01T10:48:23.985 DEBUG:boto:Headers: {} 2024-02-01T10:48:23.985 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:23.986 DEBUG:boto:Port: 8001 2024-02-01T10:48:23.986 DEBUG:boto:Params: {} 2024-02-01T10:48:23.986 DEBUG:boto:Token: None 2024-02-01T10:48:23.986 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:23 GMT /vclfcn-61/obj7?versionId=null 2024-02-01T10:48:23.986 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:CpKSUd+hCuIPgj4NOcmLhyg4liE= 2024-02-01T10:48:23.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:CpKSUd+hCuIPgj4NOcmLhyg4liE=', 'Content-Length': '0'} 2024-02-01T10:48:23.989 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000082d1fbec449ee7f2-0065bb76f7-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:23.990 DEBUG:boto:path=/vclfcn-61/obj7 2024-02-01T10:48:23.990 DEBUG:boto:auth_path=/vclfcn-61/obj7 2024-02-01T10:48:23.990 DEBUG:boto:path=/vclfcn-61/obj7?versionId=null 2024-02-01T10:48:23.991 DEBUG:boto:auth_path=/vclfcn-61/obj7?versionId=null 2024-02-01T10:48:23.991 DEBUG:boto:Method: GET 2024-02-01T10:48:23.991 DEBUG:boto:Path: /vclfcn-61/obj7?versionId=null 2024-02-01T10:48:23.991 DEBUG:boto:Data: 2024-02-01T10:48:23.991 DEBUG:boto:Headers: {} 2024-02-01T10:48:23.992 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:23.992 DEBUG:boto:Port: 8000 2024-02-01T10:48:23.992 DEBUG:boto:Params: {} 2024-02-01T10:48:23.992 DEBUG:boto:Token: None 2024-02-01T10:48:23.993 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:23 GMT /vclfcn-61/obj7?versionId=null 2024-02-01T10:48:23.993 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:CpKSUd+hCuIPgj4NOcmLhyg4liE= 2024-02-01T10:48:23.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:23 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:CpKSUd+hCuIPgj4NOcmLhyg4liE=', 'Content-Length': '0'} 2024-02-01T10:48:23.997 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000071141f117bb8abca-0065bb76f7-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:23 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:24.040 DEBUG:boto:path=/vclfcn-61/obj7 2024-02-01T10:48:24.040 DEBUG:boto:auth_path=/vclfcn-61/obj7 2024-02-01T10:48:24.040 DEBUG:boto:path=/vclfcn-61/obj7?versionId=null 2024-02-01T10:48:24.040 DEBUG:boto:auth_path=/vclfcn-61/obj7?versionId=null 2024-02-01T10:48:24.040 DEBUG:boto:Method: HEAD 2024-02-01T10:48:24.041 DEBUG:boto:Path: /vclfcn-61/obj7?versionId=null 2024-02-01T10:48:24.041 DEBUG:boto:Data: 2024-02-01T10:48:24.041 DEBUG:boto:Headers: {} 2024-02-01T10:48:24.041 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:24.041 DEBUG:boto:Port: 8001 2024-02-01T10:48:24.041 DEBUG:boto:Params: {} 2024-02-01T10:48:24.041 DEBUG:boto:Token: None 2024-02-01T10:48:24.041 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:24 GMT /vclfcn-61/obj7?versionId=null 2024-02-01T10:48:24.041 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RSpAZ7bES/6IfEeWREiuTxflrnQ= 2024-02-01T10:48:24.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RSpAZ7bES/6IfEeWREiuTxflrnQ=', 'Content-Length': '0'} 2024-02-01T10:48:24.044 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bb0351a141be0a33-0065bb76f8-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:24.044 DEBUG:boto:path=/vclfcn-61/obj7 2024-02-01T10:48:24.045 DEBUG:boto:auth_path=/vclfcn-61/obj7 2024-02-01T10:48:24.045 DEBUG:boto:path=/vclfcn-61/obj7?versionId=null 2024-02-01T10:48:24.045 DEBUG:boto:auth_path=/vclfcn-61/obj7?versionId=null 2024-02-01T10:48:24.045 DEBUG:boto:Method: HEAD 2024-02-01T10:48:24.045 DEBUG:boto:Path: /vclfcn-61/obj7?versionId=null 2024-02-01T10:48:24.045 DEBUG:boto:Data: 2024-02-01T10:48:24.045 DEBUG:boto:Headers: {} 2024-02-01T10:48:24.045 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:24.045 DEBUG:boto:Port: 8000 2024-02-01T10:48:24.045 DEBUG:boto:Params: {} 2024-02-01T10:48:24.045 DEBUG:boto:Token: None 2024-02-01T10:48:24.046 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:24 GMT /vclfcn-61/obj7?versionId=null 2024-02-01T10:48:24.046 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RSpAZ7bES/6IfEeWREiuTxflrnQ= 2024-02-01T10:48:24.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RSpAZ7bES/6IfEeWREiuTxflrnQ=', 'Content-Length': '0'} 2024-02-01T10:48:24.049 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c6b01f130ed310c8-0065bb76f8-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:24.049 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-02-01T10:48:24.049 DEBUG:boto:path=/vclfcn-61/obj7 2024-02-01T10:48:24.049 DEBUG:boto:auth_path=/vclfcn-61/obj7 2024-02-01T10:48:24.049 DEBUG:boto:path=/vclfcn-61/obj7?versionId=null 2024-02-01T10:48:24.049 DEBUG:boto:auth_path=/vclfcn-61/obj7?versionId=null 2024-02-01T10:48:24.049 DEBUG:boto:Method: GET 2024-02-01T10:48:24.050 DEBUG:boto:Path: /vclfcn-61/obj7?versionId=null 2024-02-01T10:48:24.050 DEBUG:boto:Data: 2024-02-01T10:48:24.050 DEBUG:boto:Headers: {} 2024-02-01T10:48:24.050 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:24.050 DEBUG:boto:Port: 8001 2024-02-01T10:48:24.050 DEBUG:boto:Params: {} 2024-02-01T10:48:24.050 DEBUG:boto:Token: None 2024-02-01T10:48:24.050 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:24 GMT /vclfcn-61/obj7?versionId=null 2024-02-01T10:48:24.051 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZB+/LAWo+5LoRBlqFKXeXQ6tQiM= 2024-02-01T10:48:24.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZB+/LAWo+5LoRBlqFKXeXQ6tQiM=', 'Content-Length': '0'} 2024-02-01T10:48:24.054 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003e2f87024bd4114e-0065bb76f8-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:24.055 DEBUG:boto:path=/vclfcn-61/obj7 2024-02-01T10:48:24.055 DEBUG:boto:auth_path=/vclfcn-61/obj7 2024-02-01T10:48:24.055 DEBUG:boto:path=/vclfcn-61/obj7?versionId=null 2024-02-01T10:48:24.055 DEBUG:boto:auth_path=/vclfcn-61/obj7?versionId=null 2024-02-01T10:48:24.055 DEBUG:boto:Method: GET 2024-02-01T10:48:24.055 DEBUG:boto:Path: /vclfcn-61/obj7?versionId=null 2024-02-01T10:48:24.056 DEBUG:boto:Data: 2024-02-01T10:48:24.056 DEBUG:boto:Headers: {} 2024-02-01T10:48:24.056 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:24.056 DEBUG:boto:Port: 8000 2024-02-01T10:48:24.056 DEBUG:boto:Params: {} 2024-02-01T10:48:24.056 DEBUG:boto:Token: None 2024-02-01T10:48:24.056 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:24 GMT /vclfcn-61/obj7?versionId=null 2024-02-01T10:48:24.056 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZB+/LAWo+5LoRBlqFKXeXQ6tQiM= 2024-02-01T10:48:24.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZB+/LAWo+5LoRBlqFKXeXQ6tQiM=', 'Content-Length': '0'} 2024-02-01T10:48:24.060 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003961d71aa75bbcf6-0065bb76f8-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:24.104 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:24.104 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:24.104 DEBUG:boto:path=/vclfcn-61/?prefix=obj7 2024-02-01T10:48:24.105 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj7 2024-02-01T10:48:24.105 DEBUG:boto:Method: GET 2024-02-01T10:48:24.105 DEBUG:boto:Path: /vclfcn-61/?prefix=obj7 2024-02-01T10:48:24.105 DEBUG:boto:Data: 2024-02-01T10:48:24.105 DEBUG:boto:Headers: {} 2024-02-01T10:48:24.105 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:24.105 DEBUG:boto:Port: 8001 2024-02-01T10:48:24.105 DEBUG:boto:Params: {} 2024-02-01T10:48:24.106 DEBUG:boto:Token: None 2024-02-01T10:48:24.106 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:24 GMT /vclfcn-61/ 2024-02-01T10:48:24.106 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FGMx2KVvcpJEq35XE0X4YHL0QGo= 2024-02-01T10:48:24.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FGMx2KVvcpJEq35XE0X4YHL0QGo=', 'Content-Length': '0'} 2024-02-01T10:48:24.111 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004cd54e86dcd91d6e-0065bb76f8-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:24.111 DEBUG:boto:b'vclfcn-61obj71000falseobj72024-02-01T10:47:30.492Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:24.111 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:24.111 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:24.111 DEBUG:boto:path=/vclfcn-61/?prefix=obj7 2024-02-01T10:48:24.112 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj7 2024-02-01T10:48:24.112 DEBUG:boto:Method: GET 2024-02-01T10:48:24.112 DEBUG:boto:Path: /vclfcn-61/?prefix=obj7 2024-02-01T10:48:24.112 DEBUG:boto:Data: 2024-02-01T10:48:24.112 DEBUG:boto:Headers: {} 2024-02-01T10:48:24.112 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:24.112 DEBUG:boto:Port: 8000 2024-02-01T10:48:24.112 DEBUG:boto:Params: {} 2024-02-01T10:48:24.113 DEBUG:boto:Token: None 2024-02-01T10:48:24.113 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:24 GMT /vclfcn-61/ 2024-02-01T10:48:24.113 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FGMx2KVvcpJEq35XE0X4YHL0QGo= 2024-02-01T10:48:24.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FGMx2KVvcpJEq35XE0X4YHL0QGo=', 'Content-Length': '0'} 2024-02-01T10:48:24.117 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051c5651aa5cd23a1-0065bb76f8-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:24.160 DEBUG:boto:b'vclfcn-61obj71000falseobj72024-02-01T10:47:30.492Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:24.161 DEBUG:boto:path=/vclfcn-61/obj7 2024-02-01T10:48:24.161 DEBUG:boto:auth_path=/vclfcn-61/obj7 2024-02-01T10:48:24.161 DEBUG:boto:Method: HEAD 2024-02-01T10:48:24.161 DEBUG:boto:Path: /vclfcn-61/obj7 2024-02-01T10:48:24.161 DEBUG:boto:Data: 2024-02-01T10:48:24.162 DEBUG:boto:Headers: {} 2024-02-01T10:48:24.162 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:24.162 DEBUG:boto:Port: 8001 2024-02-01T10:48:24.162 DEBUG:boto:Params: {} 2024-02-01T10:48:24.163 DEBUG:boto:Token: None 2024-02-01T10:48:24.163 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:24 GMT /vclfcn-61/obj7 2024-02-01T10:48:24.163 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vBXiW/Bf4/Mr6kD8jaHTwosqauc= 2024-02-01T10:48:24.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vBXiW/Bf4/Mr6kD8jaHTwosqauc=', 'Content-Length': '0'} 2024-02-01T10:48:24.166 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c0c698035934d017-0065bb76f8-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:24.167 DEBUG:boto:path=/vclfcn-61/obj7 2024-02-01T10:48:24.167 DEBUG:boto:auth_path=/vclfcn-61/obj7 2024-02-01T10:48:24.167 DEBUG:boto:Method: HEAD 2024-02-01T10:48:24.167 DEBUG:boto:Path: /vclfcn-61/obj7 2024-02-01T10:48:24.167 DEBUG:boto:Data: 2024-02-01T10:48:24.167 DEBUG:boto:Headers: {} 2024-02-01T10:48:24.167 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:24.167 DEBUG:boto:Port: 8000 2024-02-01T10:48:24.168 DEBUG:boto:Params: {} 2024-02-01T10:48:24.168 DEBUG:boto:Token: None 2024-02-01T10:48:24.168 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:24 GMT /vclfcn-61/obj7 2024-02-01T10:48:24.168 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vBXiW/Bf4/Mr6kD8jaHTwosqauc= 2024-02-01T10:48:24.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vBXiW/Bf4/Mr6kD8jaHTwosqauc=', 'Content-Length': '0'} 2024-02-01T10:48:24.171 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002b2eba5c64dfe3d4-0065bb76f8-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:24.171 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-02-01T10:48:24.171 DEBUG:boto:path=/vclfcn-61/obj7 2024-02-01T10:48:24.171 DEBUG:boto:auth_path=/vclfcn-61/obj7 2024-02-01T10:48:24.171 DEBUG:boto:Method: GET 2024-02-01T10:48:24.171 DEBUG:boto:Path: /vclfcn-61/obj7 2024-02-01T10:48:24.171 DEBUG:boto:Data: 2024-02-01T10:48:24.172 DEBUG:boto:Headers: {} 2024-02-01T10:48:24.172 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:24.172 DEBUG:boto:Port: 8001 2024-02-01T10:48:24.172 DEBUG:boto:Params: {} 2024-02-01T10:48:24.172 DEBUG:boto:Token: None 2024-02-01T10:48:24.172 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:24 GMT /vclfcn-61/obj7 2024-02-01T10:48:24.172 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:p0Hzu2Bqm2OToO6qU6Rqg0rp2eg= 2024-02-01T10:48:24.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:p0Hzu2Bqm2OToO6qU6Rqg0rp2eg=', 'Content-Length': '0'} 2024-02-01T10:48:24.175 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000af89a4062cbb3cfa-0065bb76f8-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:24.176 DEBUG:boto:path=/vclfcn-61/obj7 2024-02-01T10:48:24.176 DEBUG:boto:auth_path=/vclfcn-61/obj7 2024-02-01T10:48:24.176 DEBUG:boto:Method: GET 2024-02-01T10:48:24.176 DEBUG:boto:Path: /vclfcn-61/obj7 2024-02-01T10:48:24.176 DEBUG:boto:Data: 2024-02-01T10:48:24.176 DEBUG:boto:Headers: {} 2024-02-01T10:48:24.176 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:24.176 DEBUG:boto:Port: 8000 2024-02-01T10:48:24.176 DEBUG:boto:Params: {} 2024-02-01T10:48:24.176 DEBUG:boto:Token: None 2024-02-01T10:48:24.177 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:24 GMT /vclfcn-61/obj7 2024-02-01T10:48:24.177 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:p0Hzu2Bqm2OToO6qU6Rqg0rp2eg= 2024-02-01T10:48:24.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:p0Hzu2Bqm2OToO6qU6Rqg0rp2eg=', 'Content-Length': '0'} 2024-02-01T10:48:24.179 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000166c609bf0c419be-0065bb76f8-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:24.224 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:24.224 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:24.225 DEBUG:boto:path=/vclfcn-61/?prefix=obj7 2024-02-01T10:48:24.225 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj7 2024-02-01T10:48:24.225 DEBUG:boto:Method: GET 2024-02-01T10:48:24.225 DEBUG:boto:Path: /vclfcn-61/?prefix=obj7 2024-02-01T10:48:24.225 DEBUG:boto:Data: 2024-02-01T10:48:24.226 DEBUG:boto:Headers: {} 2024-02-01T10:48:24.226 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:24.226 DEBUG:boto:Port: 8001 2024-02-01T10:48:24.226 DEBUG:boto:Params: {} 2024-02-01T10:48:24.227 DEBUG:boto:Token: None 2024-02-01T10:48:24.227 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:24 GMT /vclfcn-61/ 2024-02-01T10:48:24.227 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FGMx2KVvcpJEq35XE0X4YHL0QGo= 2024-02-01T10:48:24.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FGMx2KVvcpJEq35XE0X4YHL0QGo=', 'Content-Length': '0'} 2024-02-01T10:48:24.232 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b9896665f12096fc-0065bb76f8-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:24.232 DEBUG:boto:b'vclfcn-61obj71000falseobj72024-02-01T10:47:30.492Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:24.232 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:24.232 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:24.233 DEBUG:boto:path=/vclfcn-61/?prefix=obj7 2024-02-01T10:48:24.233 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj7 2024-02-01T10:48:24.233 DEBUG:boto:Method: GET 2024-02-01T10:48:24.233 DEBUG:boto:Path: /vclfcn-61/?prefix=obj7 2024-02-01T10:48:24.233 DEBUG:boto:Data: 2024-02-01T10:48:24.234 DEBUG:boto:Headers: {} 2024-02-01T10:48:24.234 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:24.234 DEBUG:boto:Port: 8000 2024-02-01T10:48:24.234 DEBUG:boto:Params: {} 2024-02-01T10:48:24.235 DEBUG:boto:Token: None 2024-02-01T10:48:24.235 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:24 GMT /vclfcn-61/ 2024-02-01T10:48:24.235 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FGMx2KVvcpJEq35XE0X4YHL0QGo= 2024-02-01T10:48:24.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FGMx2KVvcpJEq35XE0X4YHL0QGo=', 'Content-Length': '0'} 2024-02-01T10:48:24.240 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a4d31b1dabe8a548-0065bb76f8-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:24.240 DEBUG:boto:b'vclfcn-61obj71000falseobj72024-02-01T10:47:30.492Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:24.240 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-02-01T10:48:24.240 DEBUG:boto:path=/vclfcn-61/obj8 2024-02-01T10:48:24.241 DEBUG:boto:auth_path=/vclfcn-61/obj8 2024-02-01T10:48:24.241 DEBUG:boto:path=/vclfcn-61/obj8?versionId=null 2024-02-01T10:48:24.241 DEBUG:boto:auth_path=/vclfcn-61/obj8?versionId=null 2024-02-01T10:48:24.241 DEBUG:boto:Method: GET 2024-02-01T10:48:24.241 DEBUG:boto:Path: /vclfcn-61/obj8?versionId=null 2024-02-01T10:48:24.241 DEBUG:boto:Data: 2024-02-01T10:48:24.241 DEBUG:boto:Headers: {} 2024-02-01T10:48:24.241 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:24.241 DEBUG:boto:Port: 8001 2024-02-01T10:48:24.241 DEBUG:boto:Params: {} 2024-02-01T10:48:24.241 DEBUG:boto:Token: None 2024-02-01T10:48:24.242 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:24 GMT /vclfcn-61/obj8?versionId=null 2024-02-01T10:48:24.242 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:htO1O50xr5rNXr4T/EJAoGXb1XA= 2024-02-01T10:48:24.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:htO1O50xr5rNXr4T/EJAoGXb1XA=', 'Content-Length': '0'} 2024-02-01T10:48:24.244 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ac80d6284ad71db1-0065bb76f8-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:24.245 DEBUG:boto:path=/vclfcn-61/obj8 2024-02-01T10:48:24.245 DEBUG:boto:auth_path=/vclfcn-61/obj8 2024-02-01T10:48:24.245 DEBUG:boto:path=/vclfcn-61/obj8?versionId=null 2024-02-01T10:48:24.245 DEBUG:boto:auth_path=/vclfcn-61/obj8?versionId=null 2024-02-01T10:48:24.245 DEBUG:boto:Method: GET 2024-02-01T10:48:24.245 DEBUG:boto:Path: /vclfcn-61/obj8?versionId=null 2024-02-01T10:48:24.245 DEBUG:boto:Data: 2024-02-01T10:48:24.245 DEBUG:boto:Headers: {} 2024-02-01T10:48:24.245 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:24.246 DEBUG:boto:Port: 8000 2024-02-01T10:48:24.246 DEBUG:boto:Params: {} 2024-02-01T10:48:24.246 DEBUG:boto:Token: None 2024-02-01T10:48:24.246 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:24 GMT /vclfcn-61/obj8?versionId=null 2024-02-01T10:48:24.246 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:htO1O50xr5rNXr4T/EJAoGXb1XA= 2024-02-01T10:48:24.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:htO1O50xr5rNXr4T/EJAoGXb1XA=', 'Content-Length': '0'} 2024-02-01T10:48:24.249 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000051c4b18449b37845-0065bb76f8-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:24.292 DEBUG:boto:path=/vclfcn-61/obj8 2024-02-01T10:48:24.292 DEBUG:boto:auth_path=/vclfcn-61/obj8 2024-02-01T10:48:24.292 DEBUG:boto:path=/vclfcn-61/obj8?versionId=null 2024-02-01T10:48:24.293 DEBUG:boto:auth_path=/vclfcn-61/obj8?versionId=null 2024-02-01T10:48:24.293 DEBUG:boto:Method: HEAD 2024-02-01T10:48:24.293 DEBUG:boto:Path: /vclfcn-61/obj8?versionId=null 2024-02-01T10:48:24.293 DEBUG:boto:Data: 2024-02-01T10:48:24.293 DEBUG:boto:Headers: {} 2024-02-01T10:48:24.293 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:24.293 DEBUG:boto:Port: 8001 2024-02-01T10:48:24.293 DEBUG:boto:Params: {} 2024-02-01T10:48:24.293 DEBUG:boto:Token: None 2024-02-01T10:48:24.294 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:24 GMT /vclfcn-61/obj8?versionId=null 2024-02-01T10:48:24.294 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:uAgnnzbz+KnN4DJ1X3zm7GiiPLE= 2024-02-01T10:48:24.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:uAgnnzbz+KnN4DJ1X3zm7GiiPLE=', 'Content-Length': '0'} 2024-02-01T10:48:24.296 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007e54e291747bbc62-0065bb76f8-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:24.297 DEBUG:boto:path=/vclfcn-61/obj8 2024-02-01T10:48:24.297 DEBUG:boto:auth_path=/vclfcn-61/obj8 2024-02-01T10:48:24.297 DEBUG:boto:path=/vclfcn-61/obj8?versionId=null 2024-02-01T10:48:24.297 DEBUG:boto:auth_path=/vclfcn-61/obj8?versionId=null 2024-02-01T10:48:24.297 DEBUG:boto:Method: HEAD 2024-02-01T10:48:24.297 DEBUG:boto:Path: /vclfcn-61/obj8?versionId=null 2024-02-01T10:48:24.298 DEBUG:boto:Data: 2024-02-01T10:48:24.298 DEBUG:boto:Headers: {} 2024-02-01T10:48:24.298 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:24.298 DEBUG:boto:Port: 8000 2024-02-01T10:48:24.298 DEBUG:boto:Params: {} 2024-02-01T10:48:24.298 DEBUG:boto:Token: None 2024-02-01T10:48:24.298 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:24 GMT /vclfcn-61/obj8?versionId=null 2024-02-01T10:48:24.299 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:uAgnnzbz+KnN4DJ1X3zm7GiiPLE= 2024-02-01T10:48:24.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:uAgnnzbz+KnN4DJ1X3zm7GiiPLE=', 'Content-Length': '0'} 2024-02-01T10:48:24.302 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007579050b62921890-0065bb76f8-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:24.302 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-02-01T10:48:24.302 DEBUG:boto:path=/vclfcn-61/obj8 2024-02-01T10:48:24.302 DEBUG:boto:auth_path=/vclfcn-61/obj8 2024-02-01T10:48:24.302 DEBUG:boto:path=/vclfcn-61/obj8?versionId=null 2024-02-01T10:48:24.303 DEBUG:boto:auth_path=/vclfcn-61/obj8?versionId=null 2024-02-01T10:48:24.303 DEBUG:boto:Method: GET 2024-02-01T10:48:24.303 DEBUG:boto:Path: /vclfcn-61/obj8?versionId=null 2024-02-01T10:48:24.303 DEBUG:boto:Data: 2024-02-01T10:48:24.303 DEBUG:boto:Headers: {} 2024-02-01T10:48:24.303 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:24.303 DEBUG:boto:Port: 8001 2024-02-01T10:48:24.303 DEBUG:boto:Params: {} 2024-02-01T10:48:24.303 DEBUG:boto:Token: None 2024-02-01T10:48:24.304 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:24 GMT /vclfcn-61/obj8?versionId=null 2024-02-01T10:48:24.304 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:htO1O50xr5rNXr4T/EJAoGXb1XA= 2024-02-01T10:48:24.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:htO1O50xr5rNXr4T/EJAoGXb1XA=', 'Content-Length': '0'} 2024-02-01T10:48:24.307 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bf13882eba583bc5-0065bb76f8-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:24.307 DEBUG:boto:path=/vclfcn-61/obj8 2024-02-01T10:48:24.307 DEBUG:boto:auth_path=/vclfcn-61/obj8 2024-02-01T10:48:24.307 DEBUG:boto:path=/vclfcn-61/obj8?versionId=null 2024-02-01T10:48:24.307 DEBUG:boto:auth_path=/vclfcn-61/obj8?versionId=null 2024-02-01T10:48:24.308 DEBUG:boto:Method: GET 2024-02-01T10:48:24.308 DEBUG:boto:Path: /vclfcn-61/obj8?versionId=null 2024-02-01T10:48:24.308 DEBUG:boto:Data: 2024-02-01T10:48:24.308 DEBUG:boto:Headers: {} 2024-02-01T10:48:24.308 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:24.308 DEBUG:boto:Port: 8000 2024-02-01T10:48:24.308 DEBUG:boto:Params: {} 2024-02-01T10:48:24.308 DEBUG:boto:Token: None 2024-02-01T10:48:24.308 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:24 GMT /vclfcn-61/obj8?versionId=null 2024-02-01T10:48:24.309 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:htO1O50xr5rNXr4T/EJAoGXb1XA= 2024-02-01T10:48:24.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:htO1O50xr5rNXr4T/EJAoGXb1XA=', 'Content-Length': '0'} 2024-02-01T10:48:24.311 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000025a7f02dd2735f92-0065bb76f8-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:24.352 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:24.352 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:24.353 DEBUG:boto:path=/vclfcn-61/?prefix=obj8 2024-02-01T10:48:24.353 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj8 2024-02-01T10:48:24.353 DEBUG:boto:Method: GET 2024-02-01T10:48:24.353 DEBUG:boto:Path: /vclfcn-61/?prefix=obj8 2024-02-01T10:48:24.353 DEBUG:boto:Data: 2024-02-01T10:48:24.353 DEBUG:boto:Headers: {} 2024-02-01T10:48:24.353 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:24.353 DEBUG:boto:Port: 8001 2024-02-01T10:48:24.353 DEBUG:boto:Params: {} 2024-02-01T10:48:24.354 DEBUG:boto:Token: None 2024-02-01T10:48:24.354 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:24 GMT /vclfcn-61/ 2024-02-01T10:48:24.354 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FGMx2KVvcpJEq35XE0X4YHL0QGo= 2024-02-01T10:48:24.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FGMx2KVvcpJEq35XE0X4YHL0QGo=', 'Content-Length': '0'} 2024-02-01T10:48:24.358 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aea6d36dda6d2822-0065bb76f8-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:24.358 DEBUG:boto:b'vclfcn-61obj81000falseobj82024-02-01T10:47:30.507Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:24.359 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:24.359 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:24.359 DEBUG:boto:path=/vclfcn-61/?prefix=obj8 2024-02-01T10:48:24.359 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj8 2024-02-01T10:48:24.359 DEBUG:boto:Method: GET 2024-02-01T10:48:24.359 DEBUG:boto:Path: /vclfcn-61/?prefix=obj8 2024-02-01T10:48:24.360 DEBUG:boto:Data: 2024-02-01T10:48:24.360 DEBUG:boto:Headers: {} 2024-02-01T10:48:24.360 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:24.360 DEBUG:boto:Port: 8000 2024-02-01T10:48:24.360 DEBUG:boto:Params: {} 2024-02-01T10:48:24.360 DEBUG:boto:Token: None 2024-02-01T10:48:24.360 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:24 GMT /vclfcn-61/ 2024-02-01T10:48:24.360 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FGMx2KVvcpJEq35XE0X4YHL0QGo= 2024-02-01T10:48:24.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FGMx2KVvcpJEq35XE0X4YHL0QGo=', 'Content-Length': '0'} 2024-02-01T10:48:24.365 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008923e69f8366a0d1-0065bb76f8-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:24.412 DEBUG:boto:b'vclfcn-61obj81000falseobj82024-02-01T10:47:30.507Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:24.413 DEBUG:boto:path=/vclfcn-61/obj8 2024-02-01T10:48:24.413 DEBUG:boto:auth_path=/vclfcn-61/obj8 2024-02-01T10:48:24.413 DEBUG:boto:Method: HEAD 2024-02-01T10:48:24.413 DEBUG:boto:Path: /vclfcn-61/obj8 2024-02-01T10:48:24.413 DEBUG:boto:Data: 2024-02-01T10:48:24.413 DEBUG:boto:Headers: {} 2024-02-01T10:48:24.413 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:24.413 DEBUG:boto:Port: 8001 2024-02-01T10:48:24.413 DEBUG:boto:Params: {} 2024-02-01T10:48:24.413 DEBUG:boto:Token: None 2024-02-01T10:48:24.414 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:24 GMT /vclfcn-61/obj8 2024-02-01T10:48:24.414 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:KWZYqfeHaU0UfhtuVdI9r3Z63Dw= 2024-02-01T10:48:24.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:KWZYqfeHaU0UfhtuVdI9r3Z63Dw=', 'Content-Length': '0'} 2024-02-01T10:48:24.416 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007e1cb3772423d0a5-0065bb76f8-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:24.417 DEBUG:boto:path=/vclfcn-61/obj8 2024-02-01T10:48:24.417 DEBUG:boto:auth_path=/vclfcn-61/obj8 2024-02-01T10:48:24.417 DEBUG:boto:Method: HEAD 2024-02-01T10:48:24.417 DEBUG:boto:Path: /vclfcn-61/obj8 2024-02-01T10:48:24.417 DEBUG:boto:Data: 2024-02-01T10:48:24.417 DEBUG:boto:Headers: {} 2024-02-01T10:48:24.417 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:24.417 DEBUG:boto:Port: 8000 2024-02-01T10:48:24.417 DEBUG:boto:Params: {} 2024-02-01T10:48:24.418 DEBUG:boto:Token: None 2024-02-01T10:48:24.418 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:24 GMT /vclfcn-61/obj8 2024-02-01T10:48:24.418 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:KWZYqfeHaU0UfhtuVdI9r3Z63Dw= 2024-02-01T10:48:24.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:KWZYqfeHaU0UfhtuVdI9r3Z63Dw=', 'Content-Length': '0'} 2024-02-01T10:48:24.421 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a7c4ad1a237110ca-0065bb76f8-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:24.421 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-02-01T10:48:24.421 DEBUG:boto:path=/vclfcn-61/obj8 2024-02-01T10:48:24.421 DEBUG:boto:auth_path=/vclfcn-61/obj8 2024-02-01T10:48:24.421 DEBUG:boto:Method: GET 2024-02-01T10:48:24.421 DEBUG:boto:Path: /vclfcn-61/obj8 2024-02-01T10:48:24.422 DEBUG:boto:Data: 2024-02-01T10:48:24.422 DEBUG:boto:Headers: {} 2024-02-01T10:48:24.422 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:24.422 DEBUG:boto:Port: 8001 2024-02-01T10:48:24.422 DEBUG:boto:Params: {} 2024-02-01T10:48:24.422 DEBUG:boto:Token: None 2024-02-01T10:48:24.422 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:24 GMT /vclfcn-61/obj8 2024-02-01T10:48:24.422 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:N6RH5GQusZxOCT9DvRSOP0f5Gso= 2024-02-01T10:48:24.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:N6RH5GQusZxOCT9DvRSOP0f5Gso=', 'Content-Length': '0'} 2024-02-01T10:48:24.425 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000088e9b4a92b030bd4-0065bb76f8-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:24.472 DEBUG:boto:path=/vclfcn-61/obj8 2024-02-01T10:48:24.472 DEBUG:boto:auth_path=/vclfcn-61/obj8 2024-02-01T10:48:24.473 DEBUG:boto:Method: GET 2024-02-01T10:48:24.473 DEBUG:boto:Path: /vclfcn-61/obj8 2024-02-01T10:48:24.473 DEBUG:boto:Data: 2024-02-01T10:48:24.473 DEBUG:boto:Headers: {} 2024-02-01T10:48:24.474 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:24.474 DEBUG:boto:Port: 8000 2024-02-01T10:48:24.474 DEBUG:boto:Params: {} 2024-02-01T10:48:24.474 DEBUG:boto:Token: None 2024-02-01T10:48:24.474 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:24 GMT /vclfcn-61/obj8 2024-02-01T10:48:24.475 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:N6RH5GQusZxOCT9DvRSOP0f5Gso= 2024-02-01T10:48:24.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:N6RH5GQusZxOCT9DvRSOP0f5Gso=', 'Content-Length': '0'} 2024-02-01T10:48:24.478 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:47:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-61:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000012a9906da9ec1cf1-0065bb76f8-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:24.524 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:24.524 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:24.524 DEBUG:boto:path=/vclfcn-61/?prefix=obj8 2024-02-01T10:48:24.524 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj8 2024-02-01T10:48:24.525 DEBUG:boto:Method: GET 2024-02-01T10:48:24.525 DEBUG:boto:Path: /vclfcn-61/?prefix=obj8 2024-02-01T10:48:24.525 DEBUG:boto:Data: 2024-02-01T10:48:24.525 DEBUG:boto:Headers: {} 2024-02-01T10:48:24.525 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:24.525 DEBUG:boto:Port: 8001 2024-02-01T10:48:24.525 DEBUG:boto:Params: {} 2024-02-01T10:48:24.525 DEBUG:boto:Token: None 2024-02-01T10:48:24.525 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:24 GMT /vclfcn-61/ 2024-02-01T10:48:24.526 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FGMx2KVvcpJEq35XE0X4YHL0QGo= 2024-02-01T10:48:24.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FGMx2KVvcpJEq35XE0X4YHL0QGo=', 'Content-Length': '0'} 2024-02-01T10:48:24.530 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f4a2c3fe015b6e7d-0065bb76f8-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:24.530 DEBUG:boto:b'vclfcn-61obj81000falseobj82024-02-01T10:47:30.507Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:24.531 DEBUG:boto:path=/vclfcn-61/ 2024-02-01T10:48:24.531 DEBUG:boto:auth_path=/vclfcn-61/ 2024-02-01T10:48:24.531 DEBUG:boto:path=/vclfcn-61/?prefix=obj8 2024-02-01T10:48:24.531 DEBUG:boto:auth_path=/vclfcn-61/?prefix=obj8 2024-02-01T10:48:24.531 DEBUG:boto:Method: GET 2024-02-01T10:48:24.531 DEBUG:boto:Path: /vclfcn-61/?prefix=obj8 2024-02-01T10:48:24.531 DEBUG:boto:Data: 2024-02-01T10:48:24.532 DEBUG:boto:Headers: {} 2024-02-01T10:48:24.532 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:24.532 DEBUG:boto:Port: 8000 2024-02-01T10:48:24.532 DEBUG:boto:Params: {} 2024-02-01T10:48:24.532 DEBUG:boto:Token: None 2024-02-01T10:48:24.532 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:24 GMT /vclfcn-61/ 2024-02-01T10:48:24.532 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FGMx2KVvcpJEq35XE0X4YHL0QGo= 2024-02-01T10:48:24.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:24 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FGMx2KVvcpJEq35XE0X4YHL0QGo=', 'Content-Length': '0'} 2024-02-01T10:48:24.537 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1f36cbaa10434d7-0065bb76f8-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:24 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:24.537 DEBUG:boto:b'vclfcn-61obj81000falseobj82024-02-01T10:47:30.507Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:24.537 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-61 zones={test-zone3, test-zone2} 2024-02-01T10:48:24.538 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone1 target=test-zone2 2024-02-01T10:48:24.538 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:48:24.538 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:48:24.538 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:48:24.580 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:48:24.580 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:48:24.669 INFO:teuthology.orchestra.run.smithi148.stdout:[{"marker":"00000000000000000000:00000000000000005582","last_update":"2024-02-01T10:47:43.534912Z"},{"marker":"00000000000000000000:00000000000000010370","last_update":"2024-02-01T10:47:43.527045Z"},{"marker":"00000000000000000000:00000000000000010088","last_update":"2024-02-01T10:47:30.156139Z"},{"marker":"00000000000000000000:00000000000000005450","last_update":"2024-02-01T10:47:43.519439Z"}] 2024-02-01T10:48:24.670 DEBUG:rgw_multi.tests:data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000005582', 1: '00000000000000000000:00000000000000010370', 2: '00000000000000000000:00000000000000010088', 3: '00000000000000000000:00000000000000005450'} 2024-02-01T10:48:24.670 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-02-01T10:48:24.670 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:48:24.670 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:48:24.671 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:48:24.717 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:48:24.717 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:48:24.804 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":12760245309917388665},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000005582","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:47:43.534912Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000010370","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:47:43.527045Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000010088","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:47:30.156139Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000005450","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:47:43.519439Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-02-01T10:48:24Z"} 2024-02-01T10:48:24.804 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":12760245309917388665},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000005582","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:47:43.534912Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000010370","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:47:43.527045Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000010088","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:47:30.156139Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000005450","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:47:43.519439Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-02-01T10:48:24Z"} 2024-02-01T10:48:24.805 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005582', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:47:43.534912Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000010370', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:47:43.527045Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000010088', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:47:30.156139Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005450', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:47:43.519439Z'}}] 2024-02-01T10:48:24.805 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000005582', 1: '00000000000000000000:00000000000000010370', 2: '00000000000000000000:00000000000000010088', 3: '00000000000000000000:00000000000000005450'} 2024-02-01T10:48:24.805 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000005582', 1: '00000000000000000000:00000000000000010370', 2: '00000000000000000000:00000000000000010088', 3: '00000000000000000000:00000000000000005450'} 2024-02-01T10:48:24.805 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-02-01T10:48:24.805 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone1 target=test-zone3 2024-02-01T10:48:24.805 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:48:24.806 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:48:24.806 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:48:24.848 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:48:24.848 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:48:24.935 INFO:teuthology.orchestra.run.smithi148.stdout:[{"marker":"00000000000000000000:00000000000000005582","last_update":"2024-02-01T10:47:43.534912Z"},{"marker":"00000000000000000000:00000000000000010370","last_update":"2024-02-01T10:47:43.527045Z"},{"marker":"00000000000000000000:00000000000000010088","last_update":"2024-02-01T10:47:30.156139Z"},{"marker":"00000000000000000000:00000000000000005450","last_update":"2024-02-01T10:47:43.519439Z"}] 2024-02-01T10:48:24.936 DEBUG:rgw_multi.tests:data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000005582', 1: '00000000000000000000:00000000000000010370', 2: '00000000000000000000:00000000000000010088', 3: '00000000000000000000:00000000000000005450'} 2024-02-01T10:48:24.936 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2024-02-01T10:48:24.936 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:48:24.936 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:48:24.937 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:48:25.024 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:48:25.025 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:48:25.103 INFO:teuthology.orchestra.run.smithi148.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":8365149014541277257},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000005582","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:47:43.534912Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000010370","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:47:43.527045Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000010088","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:47:30.156139Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000005450","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:47:43.519439Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-02-01T10:48:25Z"} 2024-02-01T10:48:25.103 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":8365149014541277257},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000005582","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:47:43.534912Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000010370","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:47:43.527045Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000010088","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:47:30.156139Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000005450","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:47:43.519439Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-02-01T10:48:25Z"} 2024-02-01T10:48:25.103 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005582', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:47:43.534912Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000010370', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:47:43.527045Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000010088', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:47:30.156139Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005450', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:47:43.519439Z'}}] 2024-02-01T10:48:25.103 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000005582', 1: '00000000000000000000:00000000000000010370', 2: '00000000000000000000:00000000000000010088', 3: '00000000000000000000:00000000000000005450'} 2024-02-01T10:48:25.104 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000005582', 1: '00000000000000000000:00000000000000010370', 2: '00000000000000000000:00000000000000010088', 3: '00000000000000000000:00000000000000005450'} 2024-02-01T10:48:25.104 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2024-02-01T10:48:25.104 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone2 target=test-zone1 2024-02-01T10:48:25.104 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:48:25.104 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:48:25.105 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:48:25.147 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:48:25.147 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:48:25.234 INFO:teuthology.orchestra.run.smithi187.stdout:[{"marker":"00000000000000000000:00000000000000004595","last_update":"2024-02-01T10:47:33.649329Z"},{"marker":"00000000000000000000:00000000000000008819","last_update":"2024-02-01T10:47:33.651154Z"},{"marker":"00000000000000000000:00000000000000006847","last_update":"2024-02-01T10:47:31.056921Z"},{"marker":"00000000000000000000:00000000000000003613","last_update":"2024-02-01T10:47:33.591006Z"}] 2024-02-01T10:48:25.234 DEBUG:rgw_multi.tests:data markers for zone=test-zone2 markers={0: '00000000000000000000:00000000000000004595', 1: '00000000000000000000:00000000000000008819', 2: '00000000000000000000:00000000000000006847', 3: '00000000000000000000:00000000000000003613'} 2024-02-01T10:48:25.234 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2024-02-01T10:48:25.235 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:48:25.235 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:48:25.235 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:48:25.282 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:48:25.282 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:48:25.365 INFO:teuthology.orchestra.run.smithi148.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":8595394196901466828},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004595","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:47:33.649329Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000008819","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:47:33.651154Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000006847","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:47:31.056921Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003613","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:47:33.591006Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-02-01T10:48:25Z"} 2024-02-01T10:48:25.365 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":8595394196901466828},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004595","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:47:33.649329Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000008819","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:47:33.651154Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000006847","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:47:31.056921Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003613","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:47:33.591006Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-02-01T10:48:25Z"} 2024-02-01T10:48:25.365 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004595', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:47:33.649329Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008819', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:47:33.651154Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006847', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:47:31.056921Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003613', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:47:33.591006Z'}}] 2024-02-01T10:48:25.365 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004595', 1: '00000000000000000000:00000000000000008819', 2: '00000000000000000000:00000000000000006847', 3: '00000000000000000000:00000000000000003613'} 2024-02-01T10:48:25.365 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000004595', 1: '00000000000000000000:00000000000000008819', 2: '00000000000000000000:00000000000000006847', 3: '00000000000000000000:00000000000000003613'} 2024-02-01T10:48:25.366 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2024-02-01T10:48:25.366 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone2 target=test-zone3 2024-02-01T10:48:25.366 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:48:25.366 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:48:25.366 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:48:25.408 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:48:25.408 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:48:25.498 INFO:teuthology.orchestra.run.smithi187.stdout:[{"marker":"00000000000000000000:00000000000000004595","last_update":"2024-02-01T10:47:33.649329Z"},{"marker":"00000000000000000000:00000000000000008819","last_update":"2024-02-01T10:47:33.651154Z"},{"marker":"00000000000000000000:00000000000000006847","last_update":"2024-02-01T10:47:31.056921Z"},{"marker":"00000000000000000000:00000000000000003613","last_update":"2024-02-01T10:47:33.591006Z"}] 2024-02-01T10:48:25.499 DEBUG:rgw_multi.tests:data markers for zone=test-zone2 markers={0: '00000000000000000000:00000000000000004595', 1: '00000000000000000000:00000000000000008819', 2: '00000000000000000000:00000000000000006847', 3: '00000000000000000000:00000000000000003613'} 2024-02-01T10:48:25.499 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2024-02-01T10:48:25.499 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:48:25.499 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:48:25.499 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:48:25.544 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:48:25.544 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:48:25.625 INFO:teuthology.orchestra.run.smithi148.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4461638765891857795},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004595","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:47:33.649329Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000008819","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:47:33.651154Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000006847","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:47:31.056921Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003613","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:47:33.591006Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-02-01T10:48:25Z"} 2024-02-01T10:48:25.625 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4461638765891857795},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004595","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:47:33.649329Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000008819","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:47:33.651154Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000006847","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:47:31.056921Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003613","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:47:33.591006Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-02-01T10:48:25Z"} 2024-02-01T10:48:25.626 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004595', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:47:33.649329Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008819', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:47:33.651154Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006847', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:47:31.056921Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003613', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:47:33.591006Z'}}] 2024-02-01T10:48:25.626 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004595', 1: '00000000000000000000:00000000000000008819', 2: '00000000000000000000:00000000000000006847', 3: '00000000000000000000:00000000000000003613'} 2024-02-01T10:48:25.626 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000004595', 1: '00000000000000000000:00000000000000008819', 2: '00000000000000000000:00000000000000006847', 3: '00000000000000000000:00000000000000003613'} 2024-02-01T10:48:25.626 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2024-02-01T10:48:25.626 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone3 target=test-zone1 2024-02-01T10:48:25.626 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:48:25.626 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:48:25.627 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:48:25.659 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:48:25.659 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:48:25.741 INFO:teuthology.orchestra.run.smithi148.stdout:[{"marker":"00000000000000000000:00000000000000005582","last_update":"2024-02-01T10:47:53.025485Z"},{"marker":"00000000000000000000:00000000000000009383","last_update":"2024-02-01T10:47:53.028944Z"},{"marker":"00000000000000000000:00000000000000008820","last_update":"2024-02-01T10:47:53.031817Z"},{"marker":"00000000000000000000:00000000000000004601","last_update":"2024-02-01T10:47:53.035069Z"}] 2024-02-01T10:48:25.741 DEBUG:rgw_multi.tests:data markers for zone=test-zone3 markers={0: '00000000000000000000:00000000000000005582', 1: '00000000000000000000:00000000000000009383', 2: '00000000000000000000:00000000000000008820', 3: '00000000000000000000:00000000000000004601'} 2024-02-01T10:48:25.741 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2024-02-01T10:48:25.741 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:48:25.741 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:48:25.742 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:48:25.818 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:48:25.818 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:48:25.892 INFO:teuthology.orchestra.run.smithi148.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":8579032341556541049},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000005582","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:47:53.025485Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000009383","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:47:53.028944Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000008820","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:47:53.031817Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004601","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:47:53.035069Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-02-01T10:48:25Z"} 2024-02-01T10:48:25.892 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":8579032341556541049},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000005582","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:47:53.025485Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000009383","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:47:53.028944Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000008820","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:47:53.031817Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004601","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:47:53.035069Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-02-01T10:48:25Z"} 2024-02-01T10:48:25.892 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005582', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:47:53.025485Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000009383', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:47:53.028944Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008820', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:47:53.031817Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004601', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:47:53.035069Z'}}] 2024-02-01T10:48:25.893 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000005582', 1: '00000000000000000000:00000000000000009383', 2: '00000000000000000000:00000000000000008820', 3: '00000000000000000000:00000000000000004601'} 2024-02-01T10:48:25.893 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000005582', 1: '00000000000000000000:00000000000000009383', 2: '00000000000000000000:00000000000000008820', 3: '00000000000000000000:00000000000000004601'} 2024-02-01T10:48:25.893 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2024-02-01T10:48:25.893 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone3 target=test-zone2 2024-02-01T10:48:25.893 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:48:25.893 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:48:25.893 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:48:25.967 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:48:25.967 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:48:26.040 INFO:teuthology.orchestra.run.smithi148.stdout:[{"marker":"00000000000000000000:00000000000000005582","last_update":"2024-02-01T10:47:53.025485Z"},{"marker":"00000000000000000000:00000000000000009383","last_update":"2024-02-01T10:47:53.028944Z"},{"marker":"00000000000000000000:00000000000000008820","last_update":"2024-02-01T10:47:53.031817Z"},{"marker":"00000000000000000000:00000000000000004601","last_update":"2024-02-01T10:47:53.035069Z"}] 2024-02-01T10:48:26.040 DEBUG:rgw_multi.tests:data markers for zone=test-zone3 markers={0: '00000000000000000000:00000000000000005582', 1: '00000000000000000000:00000000000000009383', 2: '00000000000000000000:00000000000000008820', 3: '00000000000000000000:00000000000000004601'} 2024-02-01T10:48:26.040 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2024-02-01T10:48:26.041 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:48:26.041 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:48:26.041 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:48:26.086 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:48:26.086 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:48:26.171 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":843725689139632598},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000005582","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:47:53.025485Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000009383","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:47:53.028944Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000008820","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:47:53.031817Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004601","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:47:53.035069Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-02-01T10:48:26Z"} 2024-02-01T10:48:26.171 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":843725689139632598},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000005582","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:47:53.025485Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000009383","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:47:53.028944Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000008820","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:47:53.031817Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004601","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:47:53.035069Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-02-01T10:48:26Z"} 2024-02-01T10:48:26.171 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005582', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:47:53.025485Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000009383', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:47:53.028944Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008820', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:47:53.031817Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004601', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:47:53.035069Z'}}] 2024-02-01T10:48:26.172 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000005582', 1: '00000000000000000000:00000000000000009383', 2: '00000000000000000000:00000000000000008820', 3: '00000000000000000000:00000000000000004601'} 2024-02-01T10:48:26.172 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000005582', 1: '00000000000000000000:00000000000000009383', 2: '00000000000000000000:00000000000000008820', 3: '00000000000000000000:00000000000000004601'} 2024-02-01T10:48:26.172 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2024-02-01T10:48:26.172 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_enable_right_after_disable ... ok 2024-02-01T10:48:26.175 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=vclfcn-62 2024-02-01T10:48:26.176 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:48:26.176 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:48:26.176 DEBUG:boto:Method: PUT 2024-02-01T10:48:26.176 DEBUG:boto:Path: /vclfcn-62/ 2024-02-01T10:48:26.176 DEBUG:boto:Data: 2024-02-01T10:48:26.176 DEBUG:boto:Headers: {} 2024-02-01T10:48:26.176 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:26.176 DEBUG:boto:Port: 8000 2024-02-01T10:48:26.176 DEBUG:boto:Params: {} 2024-02-01T10:48:26.177 DEBUG:boto:Token: None 2024-02-01T10:48:26.177 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:48:26 GMT /vclfcn-62/ 2024-02-01T10:48:26.177 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:K2NFRo/sQp0OFdXvsTcFgQWu8hU= 2024-02-01T10:48:26.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:26 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:K2NFRo/sQp0OFdXvsTcFgQWu8hU=', 'Content-Length': '0'} 2024-02-01T10:48:26.209 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000009cc5e90e977d49d-0065bb76fa-5765-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:26.209 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=vclfcn-63 2024-02-01T10:48:26.210 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:48:26.210 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:48:26.210 DEBUG:boto:Method: PUT 2024-02-01T10:48:26.210 DEBUG:boto:Path: /vclfcn-63/ 2024-02-01T10:48:26.210 DEBUG:boto:Data: 2024-02-01T10:48:26.210 DEBUG:boto:Headers: {} 2024-02-01T10:48:26.210 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:26.210 DEBUG:boto:Port: 8000 2024-02-01T10:48:26.210 DEBUG:boto:Params: {} 2024-02-01T10:48:26.210 DEBUG:boto:Token: None 2024-02-01T10:48:26.210 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:48:26 GMT /vclfcn-63/ 2024-02-01T10:48:26.211 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0kPBzKVeLlbNqr4zUMF42ZtgdM0= 2024-02-01T10:48:26.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:26 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0kPBzKVeLlbNqr4zUMF42ZtgdM0=', 'Content-Length': '0'} 2024-02-01T10:48:26.251 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000033d8d1596fce08d9-0065bb76fa-5350-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:26.251 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=vclfcn-64 2024-02-01T10:48:26.251 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:48:26.251 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:48:26.251 DEBUG:boto:Method: PUT 2024-02-01T10:48:26.251 DEBUG:boto:Path: /vclfcn-64/ 2024-02-01T10:48:26.251 DEBUG:boto:Data: 2024-02-01T10:48:26.251 DEBUG:boto:Headers: {} 2024-02-01T10:48:26.252 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:26.252 DEBUG:boto:Port: 8001 2024-02-01T10:48:26.252 DEBUG:boto:Params: {} 2024-02-01T10:48:26.252 DEBUG:boto:Token: None 2024-02-01T10:48:26.252 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:48:26 GMT /vclfcn-64/ 2024-02-01T10:48:26.252 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:a4NN18u0glGSCgu02z6qM39bgUk= 2024-02-01T10:48:26.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:26 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:a4NN18u0glGSCgu02z6qM39bgUk=', 'Content-Length': '0'} 2024-02-01T10:48:26.299 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f24aadf0c09c6554-0065bb76fa-5762-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:26.300 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:48:26.300 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:48:26.300 DEBUG:boto:Method: HEAD 2024-02-01T10:48:26.301 DEBUG:boto:Path: /vclfcn-62/ 2024-02-01T10:48:26.301 DEBUG:boto:Data: 2024-02-01T10:48:26.301 DEBUG:boto:Headers: {} 2024-02-01T10:48:26.301 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:26.301 DEBUG:boto:Port: 8000 2024-02-01T10:48:26.302 DEBUG:boto:Params: {} 2024-02-01T10:48:26.302 DEBUG:boto:Token: None 2024-02-01T10:48:26.302 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:26 GMT /vclfcn-62/ 2024-02-01T10:48:26.303 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:N8FIcuHqFf8yWX5DTAwi5W0J5dg= 2024-02-01T10:48:26.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:26 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:N8FIcuHqFf8yWX5DTAwi5W0J5dg=', 'Content-Length': '0'} 2024-02-01T10:48:26.307 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000161bd0df9fed176e-0065bb76fa-5765-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:26.307 DEBUG:boto:path=/vclfcn-62/obj1 2024-02-01T10:48:26.308 DEBUG:boto:auth_path=/vclfcn-62/obj1 2024-02-01T10:48:26.308 DEBUG:boto:Method: PUT 2024-02-01T10:48:26.308 DEBUG:boto:Path: /vclfcn-62/obj1 2024-02-01T10:48:26.308 DEBUG:boto:Data: 2024-02-01T10:48:26.309 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-02-01T10:48:26.309 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:26.309 DEBUG:boto:Port: 8000 2024-02-01T10:48:26.309 DEBUG:boto:Params: {} 2024-02-01T10:48:26.310 DEBUG:boto:Token: None 2024-02-01T10:48:26.310 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 01 Feb 2024 10:48:26 GMT /vclfcn-62/obj1 2024-02-01T10:48:26.310 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:njM9zhPoxYMwr0GaQBxAZgilMsk= 2024-02-01T10:48:26.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:48:26 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:njM9zhPoxYMwr0GaQBxAZgilMsk='} 2024-02-01T10:48:26.322 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000986c83c396ce0c3-0065bb76fa-5765-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:26.322 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:48:26.322 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:48:26.322 DEBUG:boto:Method: HEAD 2024-02-01T10:48:26.322 DEBUG:boto:Path: /vclfcn-62/ 2024-02-01T10:48:26.322 DEBUG:boto:Data: 2024-02-01T10:48:26.322 DEBUG:boto:Headers: {} 2024-02-01T10:48:26.322 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:26.323 DEBUG:boto:Port: 8000 2024-02-01T10:48:26.323 DEBUG:boto:Params: {} 2024-02-01T10:48:26.323 DEBUG:boto:Token: None 2024-02-01T10:48:26.323 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:26 GMT /vclfcn-62/ 2024-02-01T10:48:26.323 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:N8FIcuHqFf8yWX5DTAwi5W0J5dg= 2024-02-01T10:48:26.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:26 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:N8FIcuHqFf8yWX5DTAwi5W0J5dg=', 'Content-Length': '0'} 2024-02-01T10:48:26.326 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d618fd9ead05190f-0065bb76fa-5765-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:26.327 DEBUG:boto:path=/vclfcn-62/obj2 2024-02-01T10:48:26.327 DEBUG:boto:auth_path=/vclfcn-62/obj2 2024-02-01T10:48:26.327 DEBUG:boto:Method: PUT 2024-02-01T10:48:26.327 DEBUG:boto:Path: /vclfcn-62/obj2 2024-02-01T10:48:26.327 DEBUG:boto:Data: 2024-02-01T10:48:26.327 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-02-01T10:48:26.327 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:26.327 DEBUG:boto:Port: 8000 2024-02-01T10:48:26.327 DEBUG:boto:Params: {} 2024-02-01T10:48:26.328 DEBUG:boto:Token: None 2024-02-01T10:48:26.328 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 01 Feb 2024 10:48:26 GMT /vclfcn-62/obj2 2024-02-01T10:48:26.328 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:qhmD3ifVPqBo38Bby2IxBCQcsjw= 2024-02-01T10:48:26.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:48:26 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:qhmD3ifVPqBo38Bby2IxBCQcsjw='} 2024-02-01T10:48:26.372 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008e1f7505adfc0169-0065bb76fa-5765-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:26.373 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:48:26.373 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:48:26.373 DEBUG:boto:Method: HEAD 2024-02-01T10:48:26.373 DEBUG:boto:Path: /vclfcn-62/ 2024-02-01T10:48:26.373 DEBUG:boto:Data: 2024-02-01T10:48:26.373 DEBUG:boto:Headers: {} 2024-02-01T10:48:26.373 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:26.373 DEBUG:boto:Port: 8000 2024-02-01T10:48:26.373 DEBUG:boto:Params: {} 2024-02-01T10:48:26.374 DEBUG:boto:Token: None 2024-02-01T10:48:26.374 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:26 GMT /vclfcn-62/ 2024-02-01T10:48:26.374 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:N8FIcuHqFf8yWX5DTAwi5W0J5dg= 2024-02-01T10:48:26.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:26 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:N8FIcuHqFf8yWX5DTAwi5W0J5dg=', 'Content-Length': '0'} 2024-02-01T10:48:26.377 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004b3b534d1ac7cf05-0065bb76fa-5765-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:26.378 DEBUG:boto:path=/vclfcn-62/obj3 2024-02-01T10:48:26.378 DEBUG:boto:auth_path=/vclfcn-62/obj3 2024-02-01T10:48:26.378 DEBUG:boto:Method: PUT 2024-02-01T10:48:26.378 DEBUG:boto:Path: /vclfcn-62/obj3 2024-02-01T10:48:26.378 DEBUG:boto:Data: 2024-02-01T10:48:26.378 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-02-01T10:48:26.378 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:26.378 DEBUG:boto:Port: 8000 2024-02-01T10:48:26.378 DEBUG:boto:Params: {} 2024-02-01T10:48:26.379 DEBUG:boto:Token: None 2024-02-01T10:48:26.379 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 01 Feb 2024 10:48:26 GMT /vclfcn-62/obj3 2024-02-01T10:48:26.379 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fwenWEGRpaj0chhpyh2isFf9jJE= 2024-02-01T10:48:26.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:48:26 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fwenWEGRpaj0chhpyh2isFf9jJE='} 2024-02-01T10:48:26.389 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000052f1f41149d6dd3c-0065bb76fa-5765-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:26.390 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:48:26.390 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:48:26.390 DEBUG:boto:Method: HEAD 2024-02-01T10:48:26.390 DEBUG:boto:Path: /vclfcn-62/ 2024-02-01T10:48:26.390 DEBUG:boto:Data: 2024-02-01T10:48:26.390 DEBUG:boto:Headers: {} 2024-02-01T10:48:26.390 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:26.390 DEBUG:boto:Port: 8000 2024-02-01T10:48:26.390 DEBUG:boto:Params: {} 2024-02-01T10:48:26.390 DEBUG:boto:Token: None 2024-02-01T10:48:26.391 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:26 GMT /vclfcn-62/ 2024-02-01T10:48:26.391 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:N8FIcuHqFf8yWX5DTAwi5W0J5dg= 2024-02-01T10:48:26.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:26 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:N8FIcuHqFf8yWX5DTAwi5W0J5dg=', 'Content-Length': '0'} 2024-02-01T10:48:26.394 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000056eadca74ecedeab-0065bb76fa-5765-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:26.395 DEBUG:boto:path=/vclfcn-62/obj4 2024-02-01T10:48:26.395 DEBUG:boto:auth_path=/vclfcn-62/obj4 2024-02-01T10:48:26.395 DEBUG:boto:Method: PUT 2024-02-01T10:48:26.395 DEBUG:boto:Path: /vclfcn-62/obj4 2024-02-01T10:48:26.395 DEBUG:boto:Data: 2024-02-01T10:48:26.395 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-02-01T10:48:26.395 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:26.395 DEBUG:boto:Port: 8000 2024-02-01T10:48:26.395 DEBUG:boto:Params: {} 2024-02-01T10:48:26.395 DEBUG:boto:Token: None 2024-02-01T10:48:26.396 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 01 Feb 2024 10:48:26 GMT /vclfcn-62/obj4 2024-02-01T10:48:26.396 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GY0/9DaCCnRU/q1KP47KSPo+NIw= 2024-02-01T10:48:26.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:48:26 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GY0/9DaCCnRU/q1KP47KSPo+NIw='} 2024-02-01T10:48:26.440 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e7ed747f0d4469be-0065bb76fa-5765-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:26.441 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:48:26.441 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:48:26.441 DEBUG:boto:Method: HEAD 2024-02-01T10:48:26.441 DEBUG:boto:Path: /vclfcn-63/ 2024-02-01T10:48:26.441 DEBUG:boto:Data: 2024-02-01T10:48:26.441 DEBUG:boto:Headers: {} 2024-02-01T10:48:26.441 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:26.441 DEBUG:boto:Port: 8000 2024-02-01T10:48:26.441 DEBUG:boto:Params: {} 2024-02-01T10:48:26.441 DEBUG:boto:Token: None 2024-02-01T10:48:26.442 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:26 GMT /vclfcn-63/ 2024-02-01T10:48:26.442 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:aB3IjEzgYFX7uxY9/BqigAGu018= 2024-02-01T10:48:26.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:26 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:aB3IjEzgYFX7uxY9/BqigAGu018=', 'Content-Length': '0'} 2024-02-01T10:48:26.446 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000026d69e610d16162c-0065bb76fa-5350-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:26.446 DEBUG:boto:path=/vclfcn-63/obj1 2024-02-01T10:48:26.446 DEBUG:boto:auth_path=/vclfcn-63/obj1 2024-02-01T10:48:26.446 DEBUG:boto:Method: PUT 2024-02-01T10:48:26.446 DEBUG:boto:Path: /vclfcn-63/obj1 2024-02-01T10:48:26.446 DEBUG:boto:Data: 2024-02-01T10:48:26.447 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-02-01T10:48:26.447 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:26.447 DEBUG:boto:Port: 8000 2024-02-01T10:48:26.447 DEBUG:boto:Params: {} 2024-02-01T10:48:26.447 DEBUG:boto:Token: None 2024-02-01T10:48:26.447 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 01 Feb 2024 10:48:26 GMT /vclfcn-63/obj1 2024-02-01T10:48:26.447 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:V+2FD0726IjxCbYZ3do6oo8zfDk= 2024-02-01T10:48:26.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:48:26 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:V+2FD0726IjxCbYZ3do6oo8zfDk='} 2024-02-01T10:48:26.495 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000508745027f6b936b-0065bb76fa-5350-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:26.496 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:48:26.496 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:48:26.496 DEBUG:boto:Method: HEAD 2024-02-01T10:48:26.496 DEBUG:boto:Path: /vclfcn-63/ 2024-02-01T10:48:26.496 DEBUG:boto:Data: 2024-02-01T10:48:26.496 DEBUG:boto:Headers: {} 2024-02-01T10:48:26.496 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:26.496 DEBUG:boto:Port: 8000 2024-02-01T10:48:26.497 DEBUG:boto:Params: {} 2024-02-01T10:48:26.497 DEBUG:boto:Token: None 2024-02-01T10:48:26.497 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:26 GMT /vclfcn-63/ 2024-02-01T10:48:26.497 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:aB3IjEzgYFX7uxY9/BqigAGu018= 2024-02-01T10:48:26.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:26 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:aB3IjEzgYFX7uxY9/BqigAGu018=', 'Content-Length': '0'} 2024-02-01T10:48:26.500 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f22504256fd62dda-0065bb76fa-5350-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:26.501 DEBUG:boto:path=/vclfcn-63/obj2 2024-02-01T10:48:26.501 DEBUG:boto:auth_path=/vclfcn-63/obj2 2024-02-01T10:48:26.501 DEBUG:boto:Method: PUT 2024-02-01T10:48:26.501 DEBUG:boto:Path: /vclfcn-63/obj2 2024-02-01T10:48:26.502 DEBUG:boto:Data: 2024-02-01T10:48:26.502 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-02-01T10:48:26.502 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:26.502 DEBUG:boto:Port: 8000 2024-02-01T10:48:26.502 DEBUG:boto:Params: {} 2024-02-01T10:48:26.502 DEBUG:boto:Token: None 2024-02-01T10:48:26.502 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 01 Feb 2024 10:48:26 GMT /vclfcn-63/obj2 2024-02-01T10:48:26.502 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ah9cN/7PU99k++utSW03MC9jVSM= 2024-02-01T10:48:26.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:48:26 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ah9cN/7PU99k++utSW03MC9jVSM='} 2024-02-01T10:48:26.513 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000090b79c41ca0b5110-0065bb76fa-5350-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:26.513 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:48:26.513 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:48:26.513 DEBUG:boto:Method: HEAD 2024-02-01T10:48:26.514 DEBUG:boto:Path: /vclfcn-63/ 2024-02-01T10:48:26.514 DEBUG:boto:Data: 2024-02-01T10:48:26.514 DEBUG:boto:Headers: {} 2024-02-01T10:48:26.514 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:26.514 DEBUG:boto:Port: 8000 2024-02-01T10:48:26.514 DEBUG:boto:Params: {} 2024-02-01T10:48:26.514 DEBUG:boto:Token: None 2024-02-01T10:48:26.514 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:26 GMT /vclfcn-63/ 2024-02-01T10:48:26.514 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:aB3IjEzgYFX7uxY9/BqigAGu018= 2024-02-01T10:48:26.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:26 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:aB3IjEzgYFX7uxY9/BqigAGu018=', 'Content-Length': '0'} 2024-02-01T10:48:26.518 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f38d484632d6c3d6-0065bb76fa-5350-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:26.518 DEBUG:boto:path=/vclfcn-63/obj3 2024-02-01T10:48:26.518 DEBUG:boto:auth_path=/vclfcn-63/obj3 2024-02-01T10:48:26.518 DEBUG:boto:Method: PUT 2024-02-01T10:48:26.518 DEBUG:boto:Path: /vclfcn-63/obj3 2024-02-01T10:48:26.519 DEBUG:boto:Data: 2024-02-01T10:48:26.519 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-02-01T10:48:26.519 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:26.519 DEBUG:boto:Port: 8000 2024-02-01T10:48:26.519 DEBUG:boto:Params: {} 2024-02-01T10:48:26.519 DEBUG:boto:Token: None 2024-02-01T10:48:26.519 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 01 Feb 2024 10:48:26 GMT /vclfcn-63/obj3 2024-02-01T10:48:26.519 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:yqoiHgjIH+VScu0Z8hKyDUkpG+M= 2024-02-01T10:48:26.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:48:26 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:yqoiHgjIH+VScu0Z8hKyDUkpG+M='} 2024-02-01T10:48:26.564 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006dbfb12b511431e4-0065bb76fa-5350-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:26.565 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:48:26.565 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:48:26.565 DEBUG:boto:Method: HEAD 2024-02-01T10:48:26.565 DEBUG:boto:Path: /vclfcn-63/ 2024-02-01T10:48:26.565 DEBUG:boto:Data: 2024-02-01T10:48:26.565 DEBUG:boto:Headers: {} 2024-02-01T10:48:26.565 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:26.565 DEBUG:boto:Port: 8000 2024-02-01T10:48:26.565 DEBUG:boto:Params: {} 2024-02-01T10:48:26.565 DEBUG:boto:Token: None 2024-02-01T10:48:26.566 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:26 GMT /vclfcn-63/ 2024-02-01T10:48:26.566 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:aB3IjEzgYFX7uxY9/BqigAGu018= 2024-02-01T10:48:26.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:26 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:aB3IjEzgYFX7uxY9/BqigAGu018=', 'Content-Length': '0'} 2024-02-01T10:48:26.570 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000acc2d3a7a2a2a350-0065bb76fa-5350-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:26.570 DEBUG:boto:path=/vclfcn-63/obj4 2024-02-01T10:48:26.570 DEBUG:boto:auth_path=/vclfcn-63/obj4 2024-02-01T10:48:26.570 DEBUG:boto:Method: PUT 2024-02-01T10:48:26.570 DEBUG:boto:Path: /vclfcn-63/obj4 2024-02-01T10:48:26.570 DEBUG:boto:Data: 2024-02-01T10:48:26.570 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-02-01T10:48:26.571 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:26.571 DEBUG:boto:Port: 8000 2024-02-01T10:48:26.571 DEBUG:boto:Params: {} 2024-02-01T10:48:26.571 DEBUG:boto:Token: None 2024-02-01T10:48:26.571 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 01 Feb 2024 10:48:26 GMT /vclfcn-63/obj4 2024-02-01T10:48:26.571 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GRcyMKIeWnXGW1eFV7kgnegC0No= 2024-02-01T10:48:26.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:48:26 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GRcyMKIeWnXGW1eFV7kgnegC0No='} 2024-02-01T10:48:26.582 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000145824567c131b05-0065bb76fa-5350-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:26.583 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:48:26.583 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:48:26.583 DEBUG:boto:Method: HEAD 2024-02-01T10:48:26.583 DEBUG:boto:Path: /vclfcn-64/ 2024-02-01T10:48:26.583 DEBUG:boto:Data: 2024-02-01T10:48:26.583 DEBUG:boto:Headers: {} 2024-02-01T10:48:26.583 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:26.583 DEBUG:boto:Port: 8001 2024-02-01T10:48:26.583 DEBUG:boto:Params: {} 2024-02-01T10:48:26.584 DEBUG:boto:Token: None 2024-02-01T10:48:26.584 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:26 GMT /vclfcn-64/ 2024-02-01T10:48:26.584 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:clWHG4RlPe57YZq+w7/O6Sll90w= 2024-02-01T10:48:26.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:26 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:clWHG4RlPe57YZq+w7/O6Sll90w=', 'Content-Length': '0'} 2024-02-01T10:48:26.588 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000004b4cf5a625f447d-0065bb76fa-5762-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:26.588 DEBUG:boto:path=/vclfcn-64/obj1 2024-02-01T10:48:26.588 DEBUG:boto:auth_path=/vclfcn-64/obj1 2024-02-01T10:48:26.588 DEBUG:boto:Method: PUT 2024-02-01T10:48:26.588 DEBUG:boto:Path: /vclfcn-64/obj1 2024-02-01T10:48:26.588 DEBUG:boto:Data: 2024-02-01T10:48:26.589 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-02-01T10:48:26.589 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:26.589 DEBUG:boto:Port: 8001 2024-02-01T10:48:26.589 DEBUG:boto:Params: {} 2024-02-01T10:48:26.589 DEBUG:boto:Token: None 2024-02-01T10:48:26.589 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 01 Feb 2024 10:48:26 GMT /vclfcn-64/obj1 2024-02-01T10:48:26.589 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/n/AiJ8vgWNB9hTSxSCy4KOr0w4= 2024-02-01T10:48:26.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:48:26 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/n/AiJ8vgWNB9hTSxSCy4KOr0w4='} 2024-02-01T10:48:26.603 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000070ca1ea4881cc4fc-0065bb76fa-5762-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:26.603 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:48:26.604 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:48:26.604 DEBUG:boto:Method: HEAD 2024-02-01T10:48:26.604 DEBUG:boto:Path: /vclfcn-64/ 2024-02-01T10:48:26.604 DEBUG:boto:Data: 2024-02-01T10:48:26.604 DEBUG:boto:Headers: {} 2024-02-01T10:48:26.604 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:26.604 DEBUG:boto:Port: 8001 2024-02-01T10:48:26.604 DEBUG:boto:Params: {} 2024-02-01T10:48:26.604 DEBUG:boto:Token: None 2024-02-01T10:48:26.604 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:26 GMT /vclfcn-64/ 2024-02-01T10:48:26.605 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:clWHG4RlPe57YZq+w7/O6Sll90w= 2024-02-01T10:48:26.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:26 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:clWHG4RlPe57YZq+w7/O6Sll90w=', 'Content-Length': '0'} 2024-02-01T10:48:26.609 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001749262329a98a4c-0065bb76fa-5762-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:26.609 DEBUG:boto:path=/vclfcn-64/obj2 2024-02-01T10:48:26.609 DEBUG:boto:auth_path=/vclfcn-64/obj2 2024-02-01T10:48:26.609 DEBUG:boto:Method: PUT 2024-02-01T10:48:26.609 DEBUG:boto:Path: /vclfcn-64/obj2 2024-02-01T10:48:26.609 DEBUG:boto:Data: 2024-02-01T10:48:26.609 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-02-01T10:48:26.609 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:26.610 DEBUG:boto:Port: 8001 2024-02-01T10:48:26.610 DEBUG:boto:Params: {} 2024-02-01T10:48:26.610 DEBUG:boto:Token: None 2024-02-01T10:48:26.610 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 01 Feb 2024 10:48:26 GMT /vclfcn-64/obj2 2024-02-01T10:48:26.610 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YavV1UrKjG46xiPEvQE6BvikRDU= 2024-02-01T10:48:26.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:48:26 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YavV1UrKjG46xiPEvQE6BvikRDU='} 2024-02-01T10:48:26.622 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000049d167b97e9d3a6f-0065bb76fa-5762-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:26.622 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:48:26.622 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:48:26.622 DEBUG:boto:Method: HEAD 2024-02-01T10:48:26.622 DEBUG:boto:Path: /vclfcn-64/ 2024-02-01T10:48:26.622 DEBUG:boto:Data: 2024-02-01T10:48:26.622 DEBUG:boto:Headers: {} 2024-02-01T10:48:26.623 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:26.623 DEBUG:boto:Port: 8001 2024-02-01T10:48:26.623 DEBUG:boto:Params: {} 2024-02-01T10:48:26.623 DEBUG:boto:Token: None 2024-02-01T10:48:26.623 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:26 GMT /vclfcn-64/ 2024-02-01T10:48:26.623 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:clWHG4RlPe57YZq+w7/O6Sll90w= 2024-02-01T10:48:26.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:26 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:clWHG4RlPe57YZq+w7/O6Sll90w=', 'Content-Length': '0'} 2024-02-01T10:48:26.627 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000034f28e76ed04cd01-0065bb76fa-5762-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:26.628 DEBUG:boto:path=/vclfcn-64/obj3 2024-02-01T10:48:26.628 DEBUG:boto:auth_path=/vclfcn-64/obj3 2024-02-01T10:48:26.628 DEBUG:boto:Method: PUT 2024-02-01T10:48:26.628 DEBUG:boto:Path: /vclfcn-64/obj3 2024-02-01T10:48:26.628 DEBUG:boto:Data: 2024-02-01T10:48:26.628 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-02-01T10:48:26.628 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:26.628 DEBUG:boto:Port: 8001 2024-02-01T10:48:26.628 DEBUG:boto:Params: {} 2024-02-01T10:48:26.629 DEBUG:boto:Token: None 2024-02-01T10:48:26.629 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 01 Feb 2024 10:48:26 GMT /vclfcn-64/obj3 2024-02-01T10:48:26.629 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:p4bWT2E4FbB8P8LNu9ZSz+l6+W4= 2024-02-01T10:48:26.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:48:26 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:p4bWT2E4FbB8P8LNu9ZSz+l6+W4='} 2024-02-01T10:48:26.677 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000045e9e04fe364d3c9-0065bb76fa-5762-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:26.677 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:48:26.677 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:48:26.677 DEBUG:boto:Method: HEAD 2024-02-01T10:48:26.678 DEBUG:boto:Path: /vclfcn-64/ 2024-02-01T10:48:26.678 DEBUG:boto:Data: 2024-02-01T10:48:26.678 DEBUG:boto:Headers: {} 2024-02-01T10:48:26.678 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:26.678 DEBUG:boto:Port: 8001 2024-02-01T10:48:26.678 DEBUG:boto:Params: {} 2024-02-01T10:48:26.678 DEBUG:boto:Token: None 2024-02-01T10:48:26.678 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:26 GMT /vclfcn-64/ 2024-02-01T10:48:26.679 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:clWHG4RlPe57YZq+w7/O6Sll90w= 2024-02-01T10:48:26.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:26 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:clWHG4RlPe57YZq+w7/O6Sll90w=', 'Content-Length': '0'} 2024-02-01T10:48:26.683 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000073215e8bdb08b6ce-0065bb76fa-5762-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:26.683 DEBUG:boto:path=/vclfcn-64/obj4 2024-02-01T10:48:26.683 DEBUG:boto:auth_path=/vclfcn-64/obj4 2024-02-01T10:48:26.683 DEBUG:boto:Method: PUT 2024-02-01T10:48:26.683 DEBUG:boto:Path: /vclfcn-64/obj4 2024-02-01T10:48:26.683 DEBUG:boto:Data: 2024-02-01T10:48:26.683 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-02-01T10:48:26.683 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:26.684 DEBUG:boto:Port: 8001 2024-02-01T10:48:26.684 DEBUG:boto:Params: {} 2024-02-01T10:48:26.684 DEBUG:boto:Token: None 2024-02-01T10:48:26.684 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 01 Feb 2024 10:48:26 GMT /vclfcn-64/obj4 2024-02-01T10:48:26.684 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:upN98GflRi+pahyIUFlaLlfwrfg= 2024-02-01T10:48:26.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:48:26 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:upN98GflRi+pahyIUFlaLlfwrfg='} 2024-02-01T10:48:26.696 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001c1e47e31f8a90c1-0065bb76fa-5762-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:26.696 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:48:26.696 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:48:26.697 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:48:26.737 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:48:26.738 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:48:26.795 INFO:teuthology.orchestra.run.smithi148.stderr:No --period given, using current period=f2e6c290-e7aa-4279-b77e-15403b18d61f 2024-02-01T10:48:26.821 INFO:teuthology.orchestra.run.smithi148.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1706784506.281191_286.1","last_update":"2024-02-01T10:48:26.281191Z"},{"marker":"","last_update":"0.000000"}] 2024-02-01T10:48:26.821 DEBUG:rgw_multi.tests:master meta markers={0: '', 1: '', 2: '1_1706784506.281191_286.1', 3: ''} 2024-02-01T10:48:26.821 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-02-01T10:48:26.822 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:48:26.822 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:48:26.822 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:48:26.864 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:48:26.865 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:48:26.920 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:48:26.917+0000 7fd28d65f980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:48:26.922 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:48:26.917+0000 7fd28d65f980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:48:26.946 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706784506.276382_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:48:26.276382Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:48:26Z"} 2024-02-01T10:48:26.947 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706784506.276382_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:48:26.276382Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:48:26Z"} 2024-02-01T10:48:26.947 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706784506.276382_285.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:48:26.276382Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:48:26.947 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1706784506.281191_286.1', 3: ''} 2024-02-01T10:48:26.947 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1706784506.276382_285.1', 3: ''} 2024-02-01T10:48:26.947 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1706784506.281191_286.1 target=1_1706784506.276382_285.1 2024-02-01T10:48:31.948 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:48:31.948 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:48:31.949 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:48:31.991 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:48:31.992 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:48:32.047 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:48:32.045+0000 7fcac1be3980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:48:32.048 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:48:32.045+0000 7fcac1be3980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:48:32.073 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706784506.281191_286.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:48:26.281191Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:48:32Z"} 2024-02-01T10:48:32.073 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706784506.281191_286.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:48:26.281191Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:48:32Z"} 2024-02-01T10:48:32.073 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706784506.281191_286.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:48:26.281191Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:48:32.074 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1706784506.281191_286.1', 3: ''} 2024-02-01T10:48:32.074 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1706784506.281191_286.1', 3: ''} 2024-02-01T10:48:32.074 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-02-01T10:48:32.074 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-02-01T10:48:32.074 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:48:32.074 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:48:32.074 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:48:32.118 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:48:32.118 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:48:32.171 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:48:32.168+0000 7f9255b90980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:48:32.172 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:48:32.172+0000 7f9255b90980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:48:32.196 INFO:teuthology.orchestra.run.smithi148.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706784506.281191_286.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:48:26.281191Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:48:32Z"} 2024-02-01T10:48:32.196 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706784506.281191_286.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:48:26.281191Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:48:32Z"} 2024-02-01T10:48:32.197 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706784506.281191_286.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:48:26.281191Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:48:32.197 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1706784506.281191_286.1', 3: ''} 2024-02-01T10:48:32.197 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1706784506.281191_286.1', 3: ''} 2024-02-01T10:48:32.197 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-02-01T10:48:32.197 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=vclfcn-62 2024-02-01T10:48:32.197 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:48:32.197 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:48:32.198 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket vclfcn-62 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-02-01T10:48:32.247 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:48:32.247 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:48:32.309 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:48:32.305+0000 7f2019ea0980 1 waiting to reach incremental sync.. 2024-02-01T10:48:37.313 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:48:37.309+0000 7f2019ea0980 1 bucket sync caught up with source: 2024-02-01T10:48:37.313 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [00000000001.55.6, , 00000000001.49.6, , 00000000001.48.6, , , 00000000001.51.6, , , ] 2024-02-01T10:48:37.313 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [00000000001.55.6, , 00000000001.49.6, , 00000000001.48.6, , , 00000000001.51.6, , , ] 2024-02-01T10:48:37.313 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:48:37.309+0000 7f2019ea0980 0 bucket checkpoint complete 2024-02-01T10:48:37.356 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=vclfcn-62 2024-02-01T10:48:37.356 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:48:37.356 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:48:37.357 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-62 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:48:37.407 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:48:37.407 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:48:37.468 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:48:37.468+0000 7f7bad1db980 1 bucket sync caught up with source: 2024-02-01T10:48:37.468 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [00000000001.55.6, , 00000000001.49.6, , 00000000001.48.6, , , 00000000001.51.6, , , ] 2024-02-01T10:48:37.468 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [00000000001.55.6, , 00000000001.49.6, , 00000000001.48.6, , , 00000000001.51.6, , , ] 2024-02-01T10:48:37.468 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:48:37.468+0000 7f7bad1db980 0 bucket checkpoint complete 2024-02-01T10:48:37.490 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=vclfcn-62 2024-02-01T10:48:37.490 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-62', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:48:37.490 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-62', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:48:37.491 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-62 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:48:37.531 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:48:37.531 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:48:37.590 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:48:37.588+0000 7f84540ab980 1 waiting to reach incremental sync.. 2024-02-01T10:48:42.591 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:48:42.588+0000 7f84540ab980 1 waiting to reach incremental sync.. 2024-02-01T10:48:47.594 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:48:47.591+0000 7f84540ab980 1 bucket sync caught up with source: 2024-02-01T10:48:47.595 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [00000000001.294.6, , 00000000001.213.6, , 00000000001.240.6, , , 00000000001.215.6, , , ] 2024-02-01T10:48:47.595 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [00000000001.294.6, , 00000000001.213.6, , 00000000001.240.6, , , 00000000001.215.6, , , ] 2024-02-01T10:48:47.595 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:48:47.591+0000 7f84540ab980 0 bucket checkpoint complete 2024-02-01T10:48:47.632 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=vclfcn-62 2024-02-01T10:48:47.632 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-62', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:48:47.633 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-62', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:48:47.633 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-62 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:48:47.725 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:48:47.725 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:48:47.795 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:48:47.791+0000 7f444ad26980 1 bucket sync caught up with source: 2024-02-01T10:48:47.795 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [00000000001.294.6, , 00000000001.213.6, , 00000000001.240.6, , , 00000000001.215.6, , , ] 2024-02-01T10:48:47.796 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [00000000001.294.6, , 00000000001.213.6, , 00000000001.240.6, , , 00000000001.215.6, , , ] 2024-02-01T10:48:47.796 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:48:47.791+0000 7f444ad26980 0 bucket checkpoint complete 2024-02-01T10:48:47.818 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=vclfcn-62 2024-02-01T10:48:47.819 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-62', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:48:47.819 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-62', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:48:47.819 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-62 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:48:47.859 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:48:47.859 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:48:47.918 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:48:47.915+0000 7fd074a25980 1 bucket sync caught up with source: 2024-02-01T10:48:47.918 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [00000000001.216.6, , 00000000001.96.6, , 00000000001.148.6, , , 00000000001.98.6, , , ] 2024-02-01T10:48:47.918 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [00000000001.216.6, , 00000000001.96.6, , 00000000001.148.6, , , 00000000001.98.6, , , ] 2024-02-01T10:48:47.919 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:48:47.915+0000 7fd074a25980 0 bucket checkpoint complete 2024-02-01T10:48:47.944 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=vclfcn-62 2024-02-01T10:48:47.944 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-62', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:48:47.944 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-62', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:48:47.944 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket vclfcn-62 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-02-01T10:48:47.994 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:48:47.994 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:48:48.059 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:48:48.057+0000 7f856e61c980 1 bucket sync caught up with source: 2024-02-01T10:48:48.059 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [00000000001.216.6, , 00000000001.96.6, , 00000000001.148.6, , , 00000000001.98.6, , , ] 2024-02-01T10:48:48.060 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [00000000001.216.6, , 00000000001.96.6, , 00000000001.148.6, , , 00000000001.98.6, , , ] 2024-02-01T10:48:48.060 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:48:48.057+0000 7f856e61c980 0 bucket checkpoint complete 2024-02-01T10:48:48.084 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-62 zones={test-zone2, test-zone1} 2024-02-01T10:48:48.084 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:48:48.084 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:48:48.084 DEBUG:boto:Method: HEAD 2024-02-01T10:48:48.084 DEBUG:boto:Path: /vclfcn-62/ 2024-02-01T10:48:48.085 DEBUG:boto:Data: 2024-02-01T10:48:48.085 DEBUG:boto:Headers: {} 2024-02-01T10:48:48.085 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:48.085 DEBUG:boto:Port: 8000 2024-02-01T10:48:48.085 DEBUG:boto:Params: {} 2024-02-01T10:48:48.085 DEBUG:boto:Token: None 2024-02-01T10:48:48.085 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:48 GMT /vclfcn-62/ 2024-02-01T10:48:48.085 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Hmzf3am+MrvrA1d+t5z4uic/zew= 2024-02-01T10:48:48.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Hmzf3am+MrvrA1d+t5z4uic/zew=', 'Content-Length': '0'} 2024-02-01T10:48:48.088 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dd5c4e15b8a747e8-0065bb7710-5350-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:48:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:48.088 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:48:48.088 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:48:48.089 DEBUG:boto:Method: HEAD 2024-02-01T10:48:48.089 DEBUG:boto:Path: /vclfcn-62/ 2024-02-01T10:48:48.089 DEBUG:boto:Data: 2024-02-01T10:48:48.089 DEBUG:boto:Headers: {} 2024-02-01T10:48:48.089 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:48.089 DEBUG:boto:Port: 8000 2024-02-01T10:48:48.089 DEBUG:boto:Params: {} 2024-02-01T10:48:48.089 DEBUG:boto:Token: None 2024-02-01T10:48:48.090 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:48 GMT /vclfcn-62/ 2024-02-01T10:48:48.090 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Hmzf3am+MrvrA1d+t5z4uic/zew= 2024-02-01T10:48:48.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Hmzf3am+MrvrA1d+t5z4uic/zew=', 'Content-Length': '0'} 2024-02-01T10:48:48.094 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000076406d05266ad3fb-0065bb7710-5765-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:48:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:48.095 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:48:48.095 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:48:48.095 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:48:48.095 DEBUG:boto:path=/vclfcn-62/?versions&max-keys=999 2024-02-01T10:48:48.095 DEBUG:boto:auth_path=/vclfcn-62/?versions&max-keys=999 2024-02-01T10:48:48.095 DEBUG:boto:Method: GET 2024-02-01T10:48:48.095 DEBUG:boto:Path: /vclfcn-62/?versions&max-keys=999 2024-02-01T10:48:48.095 DEBUG:boto:Data: 2024-02-01T10:48:48.095 DEBUG:boto:Headers: {} 2024-02-01T10:48:48.095 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:48.096 DEBUG:boto:Port: 8000 2024-02-01T10:48:48.096 DEBUG:boto:Params: {} 2024-02-01T10:48:48.096 DEBUG:boto:Token: None 2024-02-01T10:48:48.096 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:48 GMT /vclfcn-62/?versions 2024-02-01T10:48:48.096 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:o9qX5n2jv+esPt/xQj7G0CWyQlU= 2024-02-01T10:48:48.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:o9qX5n2jv+esPt/xQj7G0CWyQlU=', 'Content-Length': '0'} 2024-02-01T10:48:48.098 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bcedf44260c971b4-0065bb7710-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:48.099 DEBUG:boto:b'vclfcn-62999falseobj1nulltrue2024-02-01T10:48:26.315Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:48:26.333Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:48:26.384Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:48:26.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:48.099 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-02-01T10:48:48.100 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-02-01T10:48:48.100 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-02-01T10:48:48.100 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-02-01T10:48:48.100 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:48:48.100 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:48:48.100 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:48:48.100 DEBUG:boto:path=/vclfcn-62/?versions&max-keys=999 2024-02-01T10:48:48.100 DEBUG:boto:auth_path=/vclfcn-62/?versions&max-keys=999 2024-02-01T10:48:48.101 DEBUG:boto:Method: GET 2024-02-01T10:48:48.101 DEBUG:boto:Path: /vclfcn-62/?versions&max-keys=999 2024-02-01T10:48:48.101 DEBUG:boto:Data: 2024-02-01T10:48:48.101 DEBUG:boto:Headers: {} 2024-02-01T10:48:48.101 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:48.101 DEBUG:boto:Port: 8000 2024-02-01T10:48:48.101 DEBUG:boto:Params: {} 2024-02-01T10:48:48.101 DEBUG:boto:Token: None 2024-02-01T10:48:48.101 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:48 GMT /vclfcn-62/?versions 2024-02-01T10:48:48.102 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:o9qX5n2jv+esPt/xQj7G0CWyQlU= 2024-02-01T10:48:48.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:o9qX5n2jv+esPt/xQj7G0CWyQlU=', 'Content-Length': '0'} 2024-02-01T10:48:48.106 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001580f3b333fba418-0065bb7710-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:48.106 DEBUG:boto:b'vclfcn-62999falseobj1nulltrue2024-02-01T10:48:26.315Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:48:26.333Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:48:26.384Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:48:26.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:48.107 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-02-01T10:48:48.107 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-02-01T10:48:48.107 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-02-01T10:48:48.107 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-02-01T10:48:48.107 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:48:48.107 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:48:48.107 DEBUG:boto:path=/vclfcn-62/?versions&max-keys=999 2024-02-01T10:48:48.107 DEBUG:boto:auth_path=/vclfcn-62/?versions&max-keys=999 2024-02-01T10:48:48.108 DEBUG:boto:Method: GET 2024-02-01T10:48:48.108 DEBUG:boto:Path: /vclfcn-62/?versions&max-keys=999 2024-02-01T10:48:48.108 DEBUG:boto:Data: 2024-02-01T10:48:48.108 DEBUG:boto:Headers: {} 2024-02-01T10:48:48.108 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:48.108 DEBUG:boto:Port: 8000 2024-02-01T10:48:48.108 DEBUG:boto:Params: {} 2024-02-01T10:48:48.108 DEBUG:boto:Token: None 2024-02-01T10:48:48.108 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:48 GMT /vclfcn-62/?versions 2024-02-01T10:48:48.109 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:o9qX5n2jv+esPt/xQj7G0CWyQlU= 2024-02-01T10:48:48.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:o9qX5n2jv+esPt/xQj7G0CWyQlU=', 'Content-Length': '0'} 2024-02-01T10:48:48.111 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002592e99c0a0304ee-0065bb7710-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:48.111 DEBUG:boto:b'vclfcn-62999falseobj1nulltrue2024-02-01T10:48:26.315Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:48:26.333Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:48:26.384Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:48:26.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:48.112 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:48:48.112 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:48:48.112 DEBUG:boto:path=/vclfcn-62/?versions&max-keys=999 2024-02-01T10:48:48.112 DEBUG:boto:auth_path=/vclfcn-62/?versions&max-keys=999 2024-02-01T10:48:48.112 DEBUG:boto:Method: GET 2024-02-01T10:48:48.112 DEBUG:boto:Path: /vclfcn-62/?versions&max-keys=999 2024-02-01T10:48:48.112 DEBUG:boto:Data: 2024-02-01T10:48:48.113 DEBUG:boto:Headers: {} 2024-02-01T10:48:48.113 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:48.113 DEBUG:boto:Port: 8000 2024-02-01T10:48:48.113 DEBUG:boto:Params: {} 2024-02-01T10:48:48.113 DEBUG:boto:Token: None 2024-02-01T10:48:48.113 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:48 GMT /vclfcn-62/?versions 2024-02-01T10:48:48.113 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:o9qX5n2jv+esPt/xQj7G0CWyQlU= 2024-02-01T10:48:48.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:o9qX5n2jv+esPt/xQj7G0CWyQlU=', 'Content-Length': '0'} 2024-02-01T10:48:48.118 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006076095648e7600b-0065bb7710-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:48.118 DEBUG:boto:b'vclfcn-62999falseobj1nulltrue2024-02-01T10:48:26.315Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:48:26.333Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:48:26.384Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:48:26.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:48.118 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:48:48.119 DEBUG:boto:path=/vclfcn-62/obj1 2024-02-01T10:48:48.119 DEBUG:boto:auth_path=/vclfcn-62/obj1 2024-02-01T10:48:48.119 DEBUG:boto:path=/vclfcn-62/obj1?versionId=null 2024-02-01T10:48:48.119 DEBUG:boto:auth_path=/vclfcn-62/obj1?versionId=null 2024-02-01T10:48:48.119 DEBUG:boto:Method: GET 2024-02-01T10:48:48.119 DEBUG:boto:Path: /vclfcn-62/obj1?versionId=null 2024-02-01T10:48:48.119 DEBUG:boto:Data: 2024-02-01T10:48:48.119 DEBUG:boto:Headers: {} 2024-02-01T10:48:48.119 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:48.119 DEBUG:boto:Port: 8000 2024-02-01T10:48:48.120 DEBUG:boto:Params: {} 2024-02-01T10:48:48.120 DEBUG:boto:Token: None 2024-02-01T10:48:48.120 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:48 GMT /vclfcn-62/obj1?versionId=null 2024-02-01T10:48:48.120 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jiVr+8HNsjvRYurv3n0bNnD9eXs= 2024-02-01T10:48:48.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jiVr+8HNsjvRYurv3n0bNnD9eXs=', 'Content-Length': '0'} 2024-02-01T10:48:48.122 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004a972773e8e316a3-0065bb7710-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:48.164 DEBUG:boto:path=/vclfcn-62/obj1 2024-02-01T10:48:48.164 DEBUG:boto:auth_path=/vclfcn-62/obj1 2024-02-01T10:48:48.164 DEBUG:boto:path=/vclfcn-62/obj1?versionId=null 2024-02-01T10:48:48.165 DEBUG:boto:auth_path=/vclfcn-62/obj1?versionId=null 2024-02-01T10:48:48.165 DEBUG:boto:Method: GET 2024-02-01T10:48:48.165 DEBUG:boto:Path: /vclfcn-62/obj1?versionId=null 2024-02-01T10:48:48.165 DEBUG:boto:Data: 2024-02-01T10:48:48.165 DEBUG:boto:Headers: {} 2024-02-01T10:48:48.165 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:48.165 DEBUG:boto:Port: 8000 2024-02-01T10:48:48.165 DEBUG:boto:Params: {} 2024-02-01T10:48:48.165 DEBUG:boto:Token: None 2024-02-01T10:48:48.165 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:48 GMT /vclfcn-62/obj1?versionId=null 2024-02-01T10:48:48.166 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jiVr+8HNsjvRYurv3n0bNnD9eXs= 2024-02-01T10:48:48.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jiVr+8HNsjvRYurv3n0bNnD9eXs=', 'Content-Length': '0'} 2024-02-01T10:48:48.169 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b4780f681a99725d-0065bb7710-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:48.169 DEBUG:boto:path=/vclfcn-62/obj1 2024-02-01T10:48:48.169 DEBUG:boto:auth_path=/vclfcn-62/obj1 2024-02-01T10:48:48.169 DEBUG:boto:path=/vclfcn-62/obj1?versionId=null 2024-02-01T10:48:48.169 DEBUG:boto:auth_path=/vclfcn-62/obj1?versionId=null 2024-02-01T10:48:48.169 DEBUG:boto:Method: HEAD 2024-02-01T10:48:48.170 DEBUG:boto:Path: /vclfcn-62/obj1?versionId=null 2024-02-01T10:48:48.170 DEBUG:boto:Data: 2024-02-01T10:48:48.170 DEBUG:boto:Headers: {} 2024-02-01T10:48:48.170 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:48.170 DEBUG:boto:Port: 8000 2024-02-01T10:48:48.170 DEBUG:boto:Params: {} 2024-02-01T10:48:48.170 DEBUG:boto:Token: None 2024-02-01T10:48:48.170 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:48 GMT /vclfcn-62/obj1?versionId=null 2024-02-01T10:48:48.170 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:06n+cIvjXFAykrkuxEIGJgX/m04= 2024-02-01T10:48:48.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:06n+cIvjXFAykrkuxEIGJgX/m04=', 'Content-Length': '0'} 2024-02-01T10:48:48.173 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000a1e4f6cef680889-0065bb7710-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:48.174 DEBUG:boto:path=/vclfcn-62/obj1 2024-02-01T10:48:48.174 DEBUG:boto:auth_path=/vclfcn-62/obj1 2024-02-01T10:48:48.174 DEBUG:boto:path=/vclfcn-62/obj1?versionId=null 2024-02-01T10:48:48.174 DEBUG:boto:auth_path=/vclfcn-62/obj1?versionId=null 2024-02-01T10:48:48.174 DEBUG:boto:Method: HEAD 2024-02-01T10:48:48.174 DEBUG:boto:Path: /vclfcn-62/obj1?versionId=null 2024-02-01T10:48:48.175 DEBUG:boto:Data: 2024-02-01T10:48:48.175 DEBUG:boto:Headers: {} 2024-02-01T10:48:48.175 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:48.175 DEBUG:boto:Port: 8000 2024-02-01T10:48:48.175 DEBUG:boto:Params: {} 2024-02-01T10:48:48.175 DEBUG:boto:Token: None 2024-02-01T10:48:48.175 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:48 GMT /vclfcn-62/obj1?versionId=null 2024-02-01T10:48:48.176 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:06n+cIvjXFAykrkuxEIGJgX/m04= 2024-02-01T10:48:48.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:06n+cIvjXFAykrkuxEIGJgX/m04=', 'Content-Length': '0'} 2024-02-01T10:48:48.179 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002d964abb15affd1c-0065bb7710-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:48.179 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:48:48.179 DEBUG:boto:path=/vclfcn-62/obj1 2024-02-01T10:48:48.179 DEBUG:boto:auth_path=/vclfcn-62/obj1 2024-02-01T10:48:48.179 DEBUG:boto:path=/vclfcn-62/obj1?versionId=null 2024-02-01T10:48:48.179 DEBUG:boto:auth_path=/vclfcn-62/obj1?versionId=null 2024-02-01T10:48:48.180 DEBUG:boto:Method: GET 2024-02-01T10:48:48.180 DEBUG:boto:Path: /vclfcn-62/obj1?versionId=null 2024-02-01T10:48:48.180 DEBUG:boto:Data: 2024-02-01T10:48:48.180 DEBUG:boto:Headers: {} 2024-02-01T10:48:48.180 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:48.180 DEBUG:boto:Port: 8000 2024-02-01T10:48:48.180 DEBUG:boto:Params: {} 2024-02-01T10:48:48.180 DEBUG:boto:Token: None 2024-02-01T10:48:48.180 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:48 GMT /vclfcn-62/obj1?versionId=null 2024-02-01T10:48:48.181 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jiVr+8HNsjvRYurv3n0bNnD9eXs= 2024-02-01T10:48:48.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jiVr+8HNsjvRYurv3n0bNnD9eXs=', 'Content-Length': '0'} 2024-02-01T10:48:48.183 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f1fa012fe0e3b237-0065bb7710-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:48.184 DEBUG:boto:path=/vclfcn-62/obj1 2024-02-01T10:48:48.184 DEBUG:boto:auth_path=/vclfcn-62/obj1 2024-02-01T10:48:48.184 DEBUG:boto:path=/vclfcn-62/obj1?versionId=null 2024-02-01T10:48:48.184 DEBUG:boto:auth_path=/vclfcn-62/obj1?versionId=null 2024-02-01T10:48:48.184 DEBUG:boto:Method: GET 2024-02-01T10:48:48.184 DEBUG:boto:Path: /vclfcn-62/obj1?versionId=null 2024-02-01T10:48:48.184 DEBUG:boto:Data: 2024-02-01T10:48:48.185 DEBUG:boto:Headers: {} 2024-02-01T10:48:48.185 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:48.185 DEBUG:boto:Port: 8000 2024-02-01T10:48:48.185 DEBUG:boto:Params: {} 2024-02-01T10:48:48.185 DEBUG:boto:Token: None 2024-02-01T10:48:48.185 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:48 GMT /vclfcn-62/obj1?versionId=null 2024-02-01T10:48:48.185 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jiVr+8HNsjvRYurv3n0bNnD9eXs= 2024-02-01T10:48:48.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jiVr+8HNsjvRYurv3n0bNnD9eXs=', 'Content-Length': '0'} 2024-02-01T10:48:48.188 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004a51b03248c7d747-0065bb7710-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:48.228 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:48:48.228 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:48:48.229 DEBUG:boto:path=/vclfcn-62/?prefix=obj1 2024-02-01T10:48:48.229 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj1 2024-02-01T10:48:48.229 DEBUG:boto:Method: GET 2024-02-01T10:48:48.229 DEBUG:boto:Path: /vclfcn-62/?prefix=obj1 2024-02-01T10:48:48.229 DEBUG:boto:Data: 2024-02-01T10:48:48.229 DEBUG:boto:Headers: {} 2024-02-01T10:48:48.229 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:48.229 DEBUG:boto:Port: 8000 2024-02-01T10:48:48.229 DEBUG:boto:Params: {} 2024-02-01T10:48:48.230 DEBUG:boto:Token: None 2024-02-01T10:48:48.230 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:48 GMT /vclfcn-62/ 2024-02-01T10:48:48.230 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ci/9jktEulOllMe5jPmCFggWOZw= 2024-02-01T10:48:48.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ci/9jktEulOllMe5jPmCFggWOZw=', 'Content-Length': '0'} 2024-02-01T10:48:48.234 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b40b5f23dd06fac-0065bb7710-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:48.234 DEBUG:boto:b'vclfcn-62obj11000falseobj12024-02-01T10:48:26.315Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:48.235 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:48:48.235 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:48:48.235 DEBUG:boto:path=/vclfcn-62/?prefix=obj1 2024-02-01T10:48:48.235 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj1 2024-02-01T10:48:48.235 DEBUG:boto:Method: GET 2024-02-01T10:48:48.235 DEBUG:boto:Path: /vclfcn-62/?prefix=obj1 2024-02-01T10:48:48.235 DEBUG:boto:Data: 2024-02-01T10:48:48.236 DEBUG:boto:Headers: {} 2024-02-01T10:48:48.236 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:48.236 DEBUG:boto:Port: 8000 2024-02-01T10:48:48.236 DEBUG:boto:Params: {} 2024-02-01T10:48:48.236 DEBUG:boto:Token: None 2024-02-01T10:48:48.236 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:48 GMT /vclfcn-62/ 2024-02-01T10:48:48.237 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ci/9jktEulOllMe5jPmCFggWOZw= 2024-02-01T10:48:48.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ci/9jktEulOllMe5jPmCFggWOZw=', 'Content-Length': '0'} 2024-02-01T10:48:48.241 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d7715961c78469ce-0065bb7710-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:48.241 DEBUG:boto:b'vclfcn-62obj11000falseobj12024-02-01T10:48:26.315Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:48.241 DEBUG:boto:path=/vclfcn-62/obj1 2024-02-01T10:48:48.241 DEBUG:boto:auth_path=/vclfcn-62/obj1 2024-02-01T10:48:48.241 DEBUG:boto:Method: HEAD 2024-02-01T10:48:48.242 DEBUG:boto:Path: /vclfcn-62/obj1 2024-02-01T10:48:48.242 DEBUG:boto:Data: 2024-02-01T10:48:48.242 DEBUG:boto:Headers: {} 2024-02-01T10:48:48.242 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:48.242 DEBUG:boto:Port: 8000 2024-02-01T10:48:48.242 DEBUG:boto:Params: {} 2024-02-01T10:48:48.242 DEBUG:boto:Token: None 2024-02-01T10:48:48.242 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:48 GMT /vclfcn-62/obj1 2024-02-01T10:48:48.243 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nfCT9GUB6IG8QPZttfxpjz+j8zs= 2024-02-01T10:48:48.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nfCT9GUB6IG8QPZttfxpjz+j8zs=', 'Content-Length': '0'} 2024-02-01T10:48:48.245 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000046618906c1513e2a-0065bb7710-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:48.246 DEBUG:boto:path=/vclfcn-62/obj1 2024-02-01T10:48:48.246 DEBUG:boto:auth_path=/vclfcn-62/obj1 2024-02-01T10:48:48.246 DEBUG:boto:Method: HEAD 2024-02-01T10:48:48.246 DEBUG:boto:Path: /vclfcn-62/obj1 2024-02-01T10:48:48.246 DEBUG:boto:Data: 2024-02-01T10:48:48.246 DEBUG:boto:Headers: {} 2024-02-01T10:48:48.246 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:48.246 DEBUG:boto:Port: 8000 2024-02-01T10:48:48.247 DEBUG:boto:Params: {} 2024-02-01T10:48:48.247 DEBUG:boto:Token: None 2024-02-01T10:48:48.247 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:48 GMT /vclfcn-62/obj1 2024-02-01T10:48:48.247 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nfCT9GUB6IG8QPZttfxpjz+j8zs= 2024-02-01T10:48:48.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nfCT9GUB6IG8QPZttfxpjz+j8zs=', 'Content-Length': '0'} 2024-02-01T10:48:48.250 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008523faec65cfa053-0065bb7710-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:48.250 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:48:48.250 DEBUG:boto:path=/vclfcn-62/obj1 2024-02-01T10:48:48.250 DEBUG:boto:auth_path=/vclfcn-62/obj1 2024-02-01T10:48:48.250 DEBUG:boto:Method: GET 2024-02-01T10:48:48.250 DEBUG:boto:Path: /vclfcn-62/obj1 2024-02-01T10:48:48.251 DEBUG:boto:Data: 2024-02-01T10:48:48.251 DEBUG:boto:Headers: {} 2024-02-01T10:48:48.251 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:48.251 DEBUG:boto:Port: 8000 2024-02-01T10:48:48.251 DEBUG:boto:Params: {} 2024-02-01T10:48:48.251 DEBUG:boto:Token: None 2024-02-01T10:48:48.251 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:48 GMT /vclfcn-62/obj1 2024-02-01T10:48:48.251 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:kW9SXYXtFvEUTgZpfGBDeJCd9co= 2024-02-01T10:48:48.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:kW9SXYXtFvEUTgZpfGBDeJCd9co=', 'Content-Length': '0'} 2024-02-01T10:48:48.254 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002dc62b69237ff999-0065bb7710-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:48.296 DEBUG:boto:path=/vclfcn-62/obj1 2024-02-01T10:48:48.296 DEBUG:boto:auth_path=/vclfcn-62/obj1 2024-02-01T10:48:48.296 DEBUG:boto:Method: GET 2024-02-01T10:48:48.296 DEBUG:boto:Path: /vclfcn-62/obj1 2024-02-01T10:48:48.297 DEBUG:boto:Data: 2024-02-01T10:48:48.297 DEBUG:boto:Headers: {} 2024-02-01T10:48:48.297 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:48.297 DEBUG:boto:Port: 8000 2024-02-01T10:48:48.297 DEBUG:boto:Params: {} 2024-02-01T10:48:48.297 DEBUG:boto:Token: None 2024-02-01T10:48:48.297 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:48 GMT /vclfcn-62/obj1 2024-02-01T10:48:48.297 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:kW9SXYXtFvEUTgZpfGBDeJCd9co= 2024-02-01T10:48:48.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:kW9SXYXtFvEUTgZpfGBDeJCd9co=', 'Content-Length': '0'} 2024-02-01T10:48:48.300 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003b068f686b48ad8c-0065bb7710-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:48.301 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:48:48.301 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:48:48.301 DEBUG:boto:path=/vclfcn-62/?prefix=obj1 2024-02-01T10:48:48.301 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj1 2024-02-01T10:48:48.301 DEBUG:boto:Method: GET 2024-02-01T10:48:48.301 DEBUG:boto:Path: /vclfcn-62/?prefix=obj1 2024-02-01T10:48:48.301 DEBUG:boto:Data: 2024-02-01T10:48:48.301 DEBUG:boto:Headers: {} 2024-02-01T10:48:48.301 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:48.301 DEBUG:boto:Port: 8000 2024-02-01T10:48:48.302 DEBUG:boto:Params: {} 2024-02-01T10:48:48.302 DEBUG:boto:Token: None 2024-02-01T10:48:48.302 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:48 GMT /vclfcn-62/ 2024-02-01T10:48:48.302 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ci/9jktEulOllMe5jPmCFggWOZw= 2024-02-01T10:48:48.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ci/9jktEulOllMe5jPmCFggWOZw=', 'Content-Length': '0'} 2024-02-01T10:48:48.306 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a28d8620688dd92c-0065bb7710-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:48.307 DEBUG:boto:b'vclfcn-62obj11000falseobj12024-02-01T10:48:26.315Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:48.307 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:48:48.307 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:48:48.307 DEBUG:boto:path=/vclfcn-62/?prefix=obj1 2024-02-01T10:48:48.307 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj1 2024-02-01T10:48:48.307 DEBUG:boto:Method: GET 2024-02-01T10:48:48.307 DEBUG:boto:Path: /vclfcn-62/?prefix=obj1 2024-02-01T10:48:48.308 DEBUG:boto:Data: 2024-02-01T10:48:48.308 DEBUG:boto:Headers: {} 2024-02-01T10:48:48.308 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:48.308 DEBUG:boto:Port: 8000 2024-02-01T10:48:48.308 DEBUG:boto:Params: {} 2024-02-01T10:48:48.308 DEBUG:boto:Token: None 2024-02-01T10:48:48.308 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:48 GMT /vclfcn-62/ 2024-02-01T10:48:48.308 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ci/9jktEulOllMe5jPmCFggWOZw= 2024-02-01T10:48:48.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ci/9jktEulOllMe5jPmCFggWOZw=', 'Content-Length': '0'} 2024-02-01T10:48:48.313 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f2d1ae6d7cebfdf-0065bb7710-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:48.356 DEBUG:boto:b'vclfcn-62obj11000falseobj12024-02-01T10:48:26.315Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:48.356 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:48:48.357 DEBUG:boto:path=/vclfcn-62/obj2 2024-02-01T10:48:48.357 DEBUG:boto:auth_path=/vclfcn-62/obj2 2024-02-01T10:48:48.357 DEBUG:boto:path=/vclfcn-62/obj2?versionId=null 2024-02-01T10:48:48.357 DEBUG:boto:auth_path=/vclfcn-62/obj2?versionId=null 2024-02-01T10:48:48.357 DEBUG:boto:Method: GET 2024-02-01T10:48:48.357 DEBUG:boto:Path: /vclfcn-62/obj2?versionId=null 2024-02-01T10:48:48.357 DEBUG:boto:Data: 2024-02-01T10:48:48.357 DEBUG:boto:Headers: {} 2024-02-01T10:48:48.357 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:48.357 DEBUG:boto:Port: 8000 2024-02-01T10:48:48.358 DEBUG:boto:Params: {} 2024-02-01T10:48:48.358 DEBUG:boto:Token: None 2024-02-01T10:48:48.358 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:48 GMT /vclfcn-62/obj2?versionId=null 2024-02-01T10:48:48.358 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:DqA+P58pHH8bTGGTv/0rUhYtGWs= 2024-02-01T10:48:48.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:DqA+P58pHH8bTGGTv/0rUhYtGWs=', 'Content-Length': '0'} 2024-02-01T10:48:48.361 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004a05b03b4ed47493-0065bb7710-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:48.404 DEBUG:boto:path=/vclfcn-62/obj2 2024-02-01T10:48:48.404 DEBUG:boto:auth_path=/vclfcn-62/obj2 2024-02-01T10:48:48.404 DEBUG:boto:path=/vclfcn-62/obj2?versionId=null 2024-02-01T10:48:48.404 DEBUG:boto:auth_path=/vclfcn-62/obj2?versionId=null 2024-02-01T10:48:48.405 DEBUG:boto:Method: GET 2024-02-01T10:48:48.405 DEBUG:boto:Path: /vclfcn-62/obj2?versionId=null 2024-02-01T10:48:48.405 DEBUG:boto:Data: 2024-02-01T10:48:48.405 DEBUG:boto:Headers: {} 2024-02-01T10:48:48.405 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:48.405 DEBUG:boto:Port: 8000 2024-02-01T10:48:48.405 DEBUG:boto:Params: {} 2024-02-01T10:48:48.405 DEBUG:boto:Token: None 2024-02-01T10:48:48.405 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:48 GMT /vclfcn-62/obj2?versionId=null 2024-02-01T10:48:48.406 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:DqA+P58pHH8bTGGTv/0rUhYtGWs= 2024-02-01T10:48:48.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:DqA+P58pHH8bTGGTv/0rUhYtGWs=', 'Content-Length': '0'} 2024-02-01T10:48:48.409 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fcfb40b2b2c0225f-0065bb7710-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:48.409 DEBUG:boto:path=/vclfcn-62/obj2 2024-02-01T10:48:48.409 DEBUG:boto:auth_path=/vclfcn-62/obj2 2024-02-01T10:48:48.409 DEBUG:boto:path=/vclfcn-62/obj2?versionId=null 2024-02-01T10:48:48.410 DEBUG:boto:auth_path=/vclfcn-62/obj2?versionId=null 2024-02-01T10:48:48.410 DEBUG:boto:Method: HEAD 2024-02-01T10:48:48.410 DEBUG:boto:Path: /vclfcn-62/obj2?versionId=null 2024-02-01T10:48:48.410 DEBUG:boto:Data: 2024-02-01T10:48:48.410 DEBUG:boto:Headers: {} 2024-02-01T10:48:48.410 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:48.410 DEBUG:boto:Port: 8000 2024-02-01T10:48:48.410 DEBUG:boto:Params: {} 2024-02-01T10:48:48.410 DEBUG:boto:Token: None 2024-02-01T10:48:48.410 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:48 GMT /vclfcn-62/obj2?versionId=null 2024-02-01T10:48:48.410 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sxd2Bf+trSQTRIjxZx/P1NMZBug= 2024-02-01T10:48:48.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sxd2Bf+trSQTRIjxZx/P1NMZBug=', 'Content-Length': '0'} 2024-02-01T10:48:48.413 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000055b53501008233cc-0065bb7710-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:48.414 DEBUG:boto:path=/vclfcn-62/obj2 2024-02-01T10:48:48.414 DEBUG:boto:auth_path=/vclfcn-62/obj2 2024-02-01T10:48:48.414 DEBUG:boto:path=/vclfcn-62/obj2?versionId=null 2024-02-01T10:48:48.414 DEBUG:boto:auth_path=/vclfcn-62/obj2?versionId=null 2024-02-01T10:48:48.414 DEBUG:boto:Method: HEAD 2024-02-01T10:48:48.414 DEBUG:boto:Path: /vclfcn-62/obj2?versionId=null 2024-02-01T10:48:48.414 DEBUG:boto:Data: 2024-02-01T10:48:48.414 DEBUG:boto:Headers: {} 2024-02-01T10:48:48.414 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:48.414 DEBUG:boto:Port: 8000 2024-02-01T10:48:48.415 DEBUG:boto:Params: {} 2024-02-01T10:48:48.415 DEBUG:boto:Token: None 2024-02-01T10:48:48.415 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:48 GMT /vclfcn-62/obj2?versionId=null 2024-02-01T10:48:48.415 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sxd2Bf+trSQTRIjxZx/P1NMZBug= 2024-02-01T10:48:48.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sxd2Bf+trSQTRIjxZx/P1NMZBug=', 'Content-Length': '0'} 2024-02-01T10:48:48.418 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006ed0f77ef70f419f-0065bb7710-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:48.418 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:48:48.418 DEBUG:boto:path=/vclfcn-62/obj2 2024-02-01T10:48:48.418 DEBUG:boto:auth_path=/vclfcn-62/obj2 2024-02-01T10:48:48.418 DEBUG:boto:path=/vclfcn-62/obj2?versionId=null 2024-02-01T10:48:48.418 DEBUG:boto:auth_path=/vclfcn-62/obj2?versionId=null 2024-02-01T10:48:48.419 DEBUG:boto:Method: GET 2024-02-01T10:48:48.419 DEBUG:boto:Path: /vclfcn-62/obj2?versionId=null 2024-02-01T10:48:48.419 DEBUG:boto:Data: 2024-02-01T10:48:48.419 DEBUG:boto:Headers: {} 2024-02-01T10:48:48.419 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:48.419 DEBUG:boto:Port: 8000 2024-02-01T10:48:48.419 DEBUG:boto:Params: {} 2024-02-01T10:48:48.419 DEBUG:boto:Token: None 2024-02-01T10:48:48.419 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:48 GMT /vclfcn-62/obj2?versionId=null 2024-02-01T10:48:48.419 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:DqA+P58pHH8bTGGTv/0rUhYtGWs= 2024-02-01T10:48:48.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:DqA+P58pHH8bTGGTv/0rUhYtGWs=', 'Content-Length': '0'} 2024-02-01T10:48:48.423 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006ce7c24e136dcc7f-0065bb7710-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:48.468 DEBUG:boto:path=/vclfcn-62/obj2 2024-02-01T10:48:48.468 DEBUG:boto:auth_path=/vclfcn-62/obj2 2024-02-01T10:48:48.468 DEBUG:boto:path=/vclfcn-62/obj2?versionId=null 2024-02-01T10:48:48.468 DEBUG:boto:auth_path=/vclfcn-62/obj2?versionId=null 2024-02-01T10:48:48.468 DEBUG:boto:Method: GET 2024-02-01T10:48:48.469 DEBUG:boto:Path: /vclfcn-62/obj2?versionId=null 2024-02-01T10:48:48.469 DEBUG:boto:Data: 2024-02-01T10:48:48.469 DEBUG:boto:Headers: {} 2024-02-01T10:48:48.469 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:48.469 DEBUG:boto:Port: 8000 2024-02-01T10:48:48.469 DEBUG:boto:Params: {} 2024-02-01T10:48:48.469 DEBUG:boto:Token: None 2024-02-01T10:48:48.469 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:48 GMT /vclfcn-62/obj2?versionId=null 2024-02-01T10:48:48.469 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:DqA+P58pHH8bTGGTv/0rUhYtGWs= 2024-02-01T10:48:48.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:DqA+P58pHH8bTGGTv/0rUhYtGWs=', 'Content-Length': '0'} 2024-02-01T10:48:48.473 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000077a1f6ccf61e0136-0065bb7710-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:48.473 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:48:48.473 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:48:48.473 DEBUG:boto:path=/vclfcn-62/?prefix=obj2 2024-02-01T10:48:48.473 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj2 2024-02-01T10:48:48.473 DEBUG:boto:Method: GET 2024-02-01T10:48:48.473 DEBUG:boto:Path: /vclfcn-62/?prefix=obj2 2024-02-01T10:48:48.474 DEBUG:boto:Data: 2024-02-01T10:48:48.474 DEBUG:boto:Headers: {} 2024-02-01T10:48:48.474 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:48.474 DEBUG:boto:Port: 8000 2024-02-01T10:48:48.474 DEBUG:boto:Params: {} 2024-02-01T10:48:48.474 DEBUG:boto:Token: None 2024-02-01T10:48:48.474 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:48 GMT /vclfcn-62/ 2024-02-01T10:48:48.474 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ci/9jktEulOllMe5jPmCFggWOZw= 2024-02-01T10:48:48.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ci/9jktEulOllMe5jPmCFggWOZw=', 'Content-Length': '0'} 2024-02-01T10:48:48.479 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c711f0de397496da-0065bb7710-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:48.479 DEBUG:boto:b'vclfcn-62obj21000falseobj22024-02-01T10:48:26.333Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:48.479 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:48:48.479 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:48:48.480 DEBUG:boto:path=/vclfcn-62/?prefix=obj2 2024-02-01T10:48:48.480 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj2 2024-02-01T10:48:48.480 DEBUG:boto:Method: GET 2024-02-01T10:48:48.480 DEBUG:boto:Path: /vclfcn-62/?prefix=obj2 2024-02-01T10:48:48.480 DEBUG:boto:Data: 2024-02-01T10:48:48.480 DEBUG:boto:Headers: {} 2024-02-01T10:48:48.480 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:48.480 DEBUG:boto:Port: 8000 2024-02-01T10:48:48.480 DEBUG:boto:Params: {} 2024-02-01T10:48:48.480 DEBUG:boto:Token: None 2024-02-01T10:48:48.481 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:48 GMT /vclfcn-62/ 2024-02-01T10:48:48.481 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ci/9jktEulOllMe5jPmCFggWOZw= 2024-02-01T10:48:48.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ci/9jktEulOllMe5jPmCFggWOZw=', 'Content-Length': '0'} 2024-02-01T10:48:48.485 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fdb9cada6a86490d-0065bb7710-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:48.485 DEBUG:boto:b'vclfcn-62obj21000falseobj22024-02-01T10:48:26.333Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:48.485 DEBUG:boto:path=/vclfcn-62/obj2 2024-02-01T10:48:48.486 DEBUG:boto:auth_path=/vclfcn-62/obj2 2024-02-01T10:48:48.486 DEBUG:boto:Method: HEAD 2024-02-01T10:48:48.486 DEBUG:boto:Path: /vclfcn-62/obj2 2024-02-01T10:48:48.486 DEBUG:boto:Data: 2024-02-01T10:48:48.486 DEBUG:boto:Headers: {} 2024-02-01T10:48:48.486 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:48.486 DEBUG:boto:Port: 8000 2024-02-01T10:48:48.486 DEBUG:boto:Params: {} 2024-02-01T10:48:48.486 DEBUG:boto:Token: None 2024-02-01T10:48:48.487 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:48 GMT /vclfcn-62/obj2 2024-02-01T10:48:48.487 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:x0OwHMXthBLkwvWKYGmFarAu8LU= 2024-02-01T10:48:48.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:x0OwHMXthBLkwvWKYGmFarAu8LU=', 'Content-Length': '0'} 2024-02-01T10:48:48.489 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000010e265b8bb82bae-0065bb7710-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:48.490 DEBUG:boto:path=/vclfcn-62/obj2 2024-02-01T10:48:48.490 DEBUG:boto:auth_path=/vclfcn-62/obj2 2024-02-01T10:48:48.490 DEBUG:boto:Method: HEAD 2024-02-01T10:48:48.490 DEBUG:boto:Path: /vclfcn-62/obj2 2024-02-01T10:48:48.490 DEBUG:boto:Data: 2024-02-01T10:48:48.490 DEBUG:boto:Headers: {} 2024-02-01T10:48:48.490 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:48.490 DEBUG:boto:Port: 8000 2024-02-01T10:48:48.490 DEBUG:boto:Params: {} 2024-02-01T10:48:48.490 DEBUG:boto:Token: None 2024-02-01T10:48:48.491 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:48 GMT /vclfcn-62/obj2 2024-02-01T10:48:48.491 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:x0OwHMXthBLkwvWKYGmFarAu8LU= 2024-02-01T10:48:48.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:x0OwHMXthBLkwvWKYGmFarAu8LU=', 'Content-Length': '0'} 2024-02-01T10:48:48.493 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b170a97af14db1c0-0065bb7710-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:48.494 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:48:48.494 DEBUG:boto:path=/vclfcn-62/obj2 2024-02-01T10:48:48.494 DEBUG:boto:auth_path=/vclfcn-62/obj2 2024-02-01T10:48:48.494 DEBUG:boto:Method: GET 2024-02-01T10:48:48.494 DEBUG:boto:Path: /vclfcn-62/obj2 2024-02-01T10:48:48.494 DEBUG:boto:Data: 2024-02-01T10:48:48.494 DEBUG:boto:Headers: {} 2024-02-01T10:48:48.494 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:48.494 DEBUG:boto:Port: 8000 2024-02-01T10:48:48.494 DEBUG:boto:Params: {} 2024-02-01T10:48:48.495 DEBUG:boto:Token: None 2024-02-01T10:48:48.495 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:48 GMT /vclfcn-62/obj2 2024-02-01T10:48:48.495 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Qomg2yjbchXQ4nZa410nmQY1s9U= 2024-02-01T10:48:48.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Qomg2yjbchXQ4nZa410nmQY1s9U=', 'Content-Length': '0'} 2024-02-01T10:48:48.498 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000099d2eeffe7487f53-0065bb7710-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:48.540 DEBUG:boto:path=/vclfcn-62/obj2 2024-02-01T10:48:48.540 DEBUG:boto:auth_path=/vclfcn-62/obj2 2024-02-01T10:48:48.540 DEBUG:boto:Method: GET 2024-02-01T10:48:48.541 DEBUG:boto:Path: /vclfcn-62/obj2 2024-02-01T10:48:48.541 DEBUG:boto:Data: 2024-02-01T10:48:48.541 DEBUG:boto:Headers: {} 2024-02-01T10:48:48.541 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:48.541 DEBUG:boto:Port: 8000 2024-02-01T10:48:48.541 DEBUG:boto:Params: {} 2024-02-01T10:48:48.541 DEBUG:boto:Token: None 2024-02-01T10:48:48.541 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:48 GMT /vclfcn-62/obj2 2024-02-01T10:48:48.541 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Qomg2yjbchXQ4nZa410nmQY1s9U= 2024-02-01T10:48:48.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Qomg2yjbchXQ4nZa410nmQY1s9U=', 'Content-Length': '0'} 2024-02-01T10:48:48.544 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000acd57229af20faf7-0065bb7710-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:48.544 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:48:48.545 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:48:48.545 DEBUG:boto:path=/vclfcn-62/?prefix=obj2 2024-02-01T10:48:48.545 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj2 2024-02-01T10:48:48.545 DEBUG:boto:Method: GET 2024-02-01T10:48:48.545 DEBUG:boto:Path: /vclfcn-62/?prefix=obj2 2024-02-01T10:48:48.545 DEBUG:boto:Data: 2024-02-01T10:48:48.545 DEBUG:boto:Headers: {} 2024-02-01T10:48:48.545 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:48.545 DEBUG:boto:Port: 8000 2024-02-01T10:48:48.545 DEBUG:boto:Params: {} 2024-02-01T10:48:48.545 DEBUG:boto:Token: None 2024-02-01T10:48:48.546 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:48 GMT /vclfcn-62/ 2024-02-01T10:48:48.546 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ci/9jktEulOllMe5jPmCFggWOZw= 2024-02-01T10:48:48.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ci/9jktEulOllMe5jPmCFggWOZw=', 'Content-Length': '0'} 2024-02-01T10:48:48.550 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000706e31eee36977a7-0065bb7710-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:48.550 DEBUG:boto:b'vclfcn-62obj21000falseobj22024-02-01T10:48:26.333Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:48.550 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:48:48.550 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:48:48.550 DEBUG:boto:path=/vclfcn-62/?prefix=obj2 2024-02-01T10:48:48.551 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj2 2024-02-01T10:48:48.551 DEBUG:boto:Method: GET 2024-02-01T10:48:48.551 DEBUG:boto:Path: /vclfcn-62/?prefix=obj2 2024-02-01T10:48:48.551 DEBUG:boto:Data: 2024-02-01T10:48:48.551 DEBUG:boto:Headers: {} 2024-02-01T10:48:48.551 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:48.551 DEBUG:boto:Port: 8000 2024-02-01T10:48:48.551 DEBUG:boto:Params: {} 2024-02-01T10:48:48.551 DEBUG:boto:Token: None 2024-02-01T10:48:48.551 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:48 GMT /vclfcn-62/ 2024-02-01T10:48:48.552 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ci/9jktEulOllMe5jPmCFggWOZw= 2024-02-01T10:48:48.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ci/9jktEulOllMe5jPmCFggWOZw=', 'Content-Length': '0'} 2024-02-01T10:48:48.556 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063ae547f85a359d8-0065bb7710-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:48.556 DEBUG:boto:b'vclfcn-62obj21000falseobj22024-02-01T10:48:26.333Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:48.556 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:48:48.556 DEBUG:boto:path=/vclfcn-62/obj3 2024-02-01T10:48:48.556 DEBUG:boto:auth_path=/vclfcn-62/obj3 2024-02-01T10:48:48.557 DEBUG:boto:path=/vclfcn-62/obj3?versionId=null 2024-02-01T10:48:48.557 DEBUG:boto:auth_path=/vclfcn-62/obj3?versionId=null 2024-02-01T10:48:48.557 DEBUG:boto:Method: GET 2024-02-01T10:48:48.557 DEBUG:boto:Path: /vclfcn-62/obj3?versionId=null 2024-02-01T10:48:48.557 DEBUG:boto:Data: 2024-02-01T10:48:48.557 DEBUG:boto:Headers: {} 2024-02-01T10:48:48.557 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:48.557 DEBUG:boto:Port: 8000 2024-02-01T10:48:48.557 DEBUG:boto:Params: {} 2024-02-01T10:48:48.558 DEBUG:boto:Token: None 2024-02-01T10:48:48.558 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:48 GMT /vclfcn-62/obj3?versionId=null 2024-02-01T10:48:48.558 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0PWXPOoW+ObaCuseC2i+axEuD/c= 2024-02-01T10:48:48.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0PWXPOoW+ObaCuseC2i+axEuD/c=', 'Content-Length': '0'} 2024-02-01T10:48:48.561 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000030f9ad3cebd3002a-0065bb7710-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:48.604 DEBUG:boto:path=/vclfcn-62/obj3 2024-02-01T10:48:48.604 DEBUG:boto:auth_path=/vclfcn-62/obj3 2024-02-01T10:48:48.604 DEBUG:boto:path=/vclfcn-62/obj3?versionId=null 2024-02-01T10:48:48.604 DEBUG:boto:auth_path=/vclfcn-62/obj3?versionId=null 2024-02-01T10:48:48.605 DEBUG:boto:Method: GET 2024-02-01T10:48:48.605 DEBUG:boto:Path: /vclfcn-62/obj3?versionId=null 2024-02-01T10:48:48.605 DEBUG:boto:Data: 2024-02-01T10:48:48.605 DEBUG:boto:Headers: {} 2024-02-01T10:48:48.605 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:48.605 DEBUG:boto:Port: 8000 2024-02-01T10:48:48.605 DEBUG:boto:Params: {} 2024-02-01T10:48:48.605 DEBUG:boto:Token: None 2024-02-01T10:48:48.605 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:48 GMT /vclfcn-62/obj3?versionId=null 2024-02-01T10:48:48.606 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0PWXPOoW+ObaCuseC2i+axEuD/c= 2024-02-01T10:48:48.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0PWXPOoW+ObaCuseC2i+axEuD/c=', 'Content-Length': '0'} 2024-02-01T10:48:48.608 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004f9ef7efdcd4d3a4-0065bb7710-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:48.609 DEBUG:boto:path=/vclfcn-62/obj3 2024-02-01T10:48:48.609 DEBUG:boto:auth_path=/vclfcn-62/obj3 2024-02-01T10:48:48.609 DEBUG:boto:path=/vclfcn-62/obj3?versionId=null 2024-02-01T10:48:48.609 DEBUG:boto:auth_path=/vclfcn-62/obj3?versionId=null 2024-02-01T10:48:48.609 DEBUG:boto:Method: HEAD 2024-02-01T10:48:48.609 DEBUG:boto:Path: /vclfcn-62/obj3?versionId=null 2024-02-01T10:48:48.609 DEBUG:boto:Data: 2024-02-01T10:48:48.609 DEBUG:boto:Headers: {} 2024-02-01T10:48:48.610 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:48.610 DEBUG:boto:Port: 8000 2024-02-01T10:48:48.610 DEBUG:boto:Params: {} 2024-02-01T10:48:48.610 DEBUG:boto:Token: None 2024-02-01T10:48:48.610 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:48 GMT /vclfcn-62/obj3?versionId=null 2024-02-01T10:48:48.610 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Iu3r+Cgw2BJwir/yuRt3EbmgbzU= 2024-02-01T10:48:48.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Iu3r+Cgw2BJwir/yuRt3EbmgbzU=', 'Content-Length': '0'} 2024-02-01T10:48:48.613 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000024de7e8e831429ce-0065bb7710-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:48.613 DEBUG:boto:path=/vclfcn-62/obj3 2024-02-01T10:48:48.613 DEBUG:boto:auth_path=/vclfcn-62/obj3 2024-02-01T10:48:48.613 DEBUG:boto:path=/vclfcn-62/obj3?versionId=null 2024-02-01T10:48:48.613 DEBUG:boto:auth_path=/vclfcn-62/obj3?versionId=null 2024-02-01T10:48:48.614 DEBUG:boto:Method: HEAD 2024-02-01T10:48:48.614 DEBUG:boto:Path: /vclfcn-62/obj3?versionId=null 2024-02-01T10:48:48.614 DEBUG:boto:Data: 2024-02-01T10:48:48.614 DEBUG:boto:Headers: {} 2024-02-01T10:48:48.614 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:48.614 DEBUG:boto:Port: 8000 2024-02-01T10:48:48.614 DEBUG:boto:Params: {} 2024-02-01T10:48:48.614 DEBUG:boto:Token: None 2024-02-01T10:48:48.614 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:48 GMT /vclfcn-62/obj3?versionId=null 2024-02-01T10:48:48.614 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Iu3r+Cgw2BJwir/yuRt3EbmgbzU= 2024-02-01T10:48:48.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Iu3r+Cgw2BJwir/yuRt3EbmgbzU=', 'Content-Length': '0'} 2024-02-01T10:48:48.617 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000ecd7eb6ef3e96bb-0065bb7710-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:48.617 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:48:48.618 DEBUG:boto:path=/vclfcn-62/obj3 2024-02-01T10:48:48.618 DEBUG:boto:auth_path=/vclfcn-62/obj3 2024-02-01T10:48:48.618 DEBUG:boto:path=/vclfcn-62/obj3?versionId=null 2024-02-01T10:48:48.618 DEBUG:boto:auth_path=/vclfcn-62/obj3?versionId=null 2024-02-01T10:48:48.618 DEBUG:boto:Method: GET 2024-02-01T10:48:48.618 DEBUG:boto:Path: /vclfcn-62/obj3?versionId=null 2024-02-01T10:48:48.618 DEBUG:boto:Data: 2024-02-01T10:48:48.618 DEBUG:boto:Headers: {} 2024-02-01T10:48:48.618 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:48.618 DEBUG:boto:Port: 8000 2024-02-01T10:48:48.619 DEBUG:boto:Params: {} 2024-02-01T10:48:48.619 DEBUG:boto:Token: None 2024-02-01T10:48:48.619 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:48 GMT /vclfcn-62/obj3?versionId=null 2024-02-01T10:48:48.619 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0PWXPOoW+ObaCuseC2i+axEuD/c= 2024-02-01T10:48:48.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0PWXPOoW+ObaCuseC2i+axEuD/c=', 'Content-Length': '0'} 2024-02-01T10:48:48.622 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000421a17ecf5ca3a42-0065bb7710-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:48.664 DEBUG:boto:path=/vclfcn-62/obj3 2024-02-01T10:48:48.664 DEBUG:boto:auth_path=/vclfcn-62/obj3 2024-02-01T10:48:48.664 DEBUG:boto:path=/vclfcn-62/obj3?versionId=null 2024-02-01T10:48:48.664 DEBUG:boto:auth_path=/vclfcn-62/obj3?versionId=null 2024-02-01T10:48:48.664 DEBUG:boto:Method: GET 2024-02-01T10:48:48.664 DEBUG:boto:Path: /vclfcn-62/obj3?versionId=null 2024-02-01T10:48:48.665 DEBUG:boto:Data: 2024-02-01T10:48:48.665 DEBUG:boto:Headers: {} 2024-02-01T10:48:48.665 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:48.665 DEBUG:boto:Port: 8000 2024-02-01T10:48:48.665 DEBUG:boto:Params: {} 2024-02-01T10:48:48.665 DEBUG:boto:Token: None 2024-02-01T10:48:48.665 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:48 GMT /vclfcn-62/obj3?versionId=null 2024-02-01T10:48:48.666 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0PWXPOoW+ObaCuseC2i+axEuD/c= 2024-02-01T10:48:48.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0PWXPOoW+ObaCuseC2i+axEuD/c=', 'Content-Length': '0'} 2024-02-01T10:48:48.668 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002ec5a4f0288930a5-0065bb7710-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:48.669 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:48:48.669 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:48:48.669 DEBUG:boto:path=/vclfcn-62/?prefix=obj3 2024-02-01T10:48:48.669 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj3 2024-02-01T10:48:48.669 DEBUG:boto:Method: GET 2024-02-01T10:48:48.669 DEBUG:boto:Path: /vclfcn-62/?prefix=obj3 2024-02-01T10:48:48.669 DEBUG:boto:Data: 2024-02-01T10:48:48.669 DEBUG:boto:Headers: {} 2024-02-01T10:48:48.670 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:48.670 DEBUG:boto:Port: 8000 2024-02-01T10:48:48.670 DEBUG:boto:Params: {} 2024-02-01T10:48:48.670 DEBUG:boto:Token: None 2024-02-01T10:48:48.670 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:48 GMT /vclfcn-62/ 2024-02-01T10:48:48.670 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ci/9jktEulOllMe5jPmCFggWOZw= 2024-02-01T10:48:48.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ci/9jktEulOllMe5jPmCFggWOZw=', 'Content-Length': '0'} 2024-02-01T10:48:48.674 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad588977e9f2e3af-0065bb7710-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:48.675 DEBUG:boto:b'vclfcn-62obj31000falseobj32024-02-01T10:48:26.384Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:48.675 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:48:48.675 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:48:48.675 DEBUG:boto:path=/vclfcn-62/?prefix=obj3 2024-02-01T10:48:48.676 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj3 2024-02-01T10:48:48.676 DEBUG:boto:Method: GET 2024-02-01T10:48:48.676 DEBUG:boto:Path: /vclfcn-62/?prefix=obj3 2024-02-01T10:48:48.676 DEBUG:boto:Data: 2024-02-01T10:48:48.676 DEBUG:boto:Headers: {} 2024-02-01T10:48:48.676 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:48.676 DEBUG:boto:Port: 8000 2024-02-01T10:48:48.676 DEBUG:boto:Params: {} 2024-02-01T10:48:48.676 DEBUG:boto:Token: None 2024-02-01T10:48:48.677 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:48 GMT /vclfcn-62/ 2024-02-01T10:48:48.677 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ci/9jktEulOllMe5jPmCFggWOZw= 2024-02-01T10:48:48.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ci/9jktEulOllMe5jPmCFggWOZw=', 'Content-Length': '0'} 2024-02-01T10:48:48.681 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077ef6a642ea7f1d1-0065bb7710-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:48.728 DEBUG:boto:b'vclfcn-62obj31000falseobj32024-02-01T10:48:26.384Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:48.728 DEBUG:boto:path=/vclfcn-62/obj3 2024-02-01T10:48:48.728 DEBUG:boto:auth_path=/vclfcn-62/obj3 2024-02-01T10:48:48.729 DEBUG:boto:Method: HEAD 2024-02-01T10:48:48.729 DEBUG:boto:Path: /vclfcn-62/obj3 2024-02-01T10:48:48.729 DEBUG:boto:Data: 2024-02-01T10:48:48.729 DEBUG:boto:Headers: {} 2024-02-01T10:48:48.729 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:48.729 DEBUG:boto:Port: 8000 2024-02-01T10:48:48.729 DEBUG:boto:Params: {} 2024-02-01T10:48:48.729 DEBUG:boto:Token: None 2024-02-01T10:48:48.729 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:48 GMT /vclfcn-62/obj3 2024-02-01T10:48:48.730 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Z+w266X4RsiH44KAd+VWvy+jOak= 2024-02-01T10:48:48.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Z+w266X4RsiH44KAd+VWvy+jOak=', 'Content-Length': '0'} 2024-02-01T10:48:48.733 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000567cf8d6f3775275-0065bb7710-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:48.733 DEBUG:boto:path=/vclfcn-62/obj3 2024-02-01T10:48:48.733 DEBUG:boto:auth_path=/vclfcn-62/obj3 2024-02-01T10:48:48.733 DEBUG:boto:Method: HEAD 2024-02-01T10:48:48.733 DEBUG:boto:Path: /vclfcn-62/obj3 2024-02-01T10:48:48.733 DEBUG:boto:Data: 2024-02-01T10:48:48.734 DEBUG:boto:Headers: {} 2024-02-01T10:48:48.734 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:48.734 DEBUG:boto:Port: 8000 2024-02-01T10:48:48.734 DEBUG:boto:Params: {} 2024-02-01T10:48:48.734 DEBUG:boto:Token: None 2024-02-01T10:48:48.734 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:48 GMT /vclfcn-62/obj3 2024-02-01T10:48:48.734 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Z+w266X4RsiH44KAd+VWvy+jOak= 2024-02-01T10:48:48.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Z+w266X4RsiH44KAd+VWvy+jOak=', 'Content-Length': '0'} 2024-02-01T10:48:48.737 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008fa613df022f3a59-0065bb7710-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:48.737 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:48:48.737 DEBUG:boto:path=/vclfcn-62/obj3 2024-02-01T10:48:48.737 DEBUG:boto:auth_path=/vclfcn-62/obj3 2024-02-01T10:48:48.737 DEBUG:boto:Method: GET 2024-02-01T10:48:48.738 DEBUG:boto:Path: /vclfcn-62/obj3 2024-02-01T10:48:48.738 DEBUG:boto:Data: 2024-02-01T10:48:48.738 DEBUG:boto:Headers: {} 2024-02-01T10:48:48.738 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:48.738 DEBUG:boto:Port: 8000 2024-02-01T10:48:48.738 DEBUG:boto:Params: {} 2024-02-01T10:48:48.738 DEBUG:boto:Token: None 2024-02-01T10:48:48.738 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:48 GMT /vclfcn-62/obj3 2024-02-01T10:48:48.738 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fzLjtmyp3SmraQqYe9w6tjys5wA= 2024-02-01T10:48:48.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fzLjtmyp3SmraQqYe9w6tjys5wA=', 'Content-Length': '0'} 2024-02-01T10:48:48.741 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bf0bb067b4a90737-0065bb7710-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:48.788 DEBUG:boto:path=/vclfcn-62/obj3 2024-02-01T10:48:48.791 DEBUG:boto:auth_path=/vclfcn-62/obj3 2024-02-01T10:48:48.791 DEBUG:boto:Method: GET 2024-02-01T10:48:48.791 DEBUG:boto:Path: /vclfcn-62/obj3 2024-02-01T10:48:48.791 DEBUG:boto:Data: 2024-02-01T10:48:48.791 DEBUG:boto:Headers: {} 2024-02-01T10:48:48.791 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:48.791 DEBUG:boto:Port: 8000 2024-02-01T10:48:48.791 DEBUG:boto:Params: {} 2024-02-01T10:48:48.792 DEBUG:boto:Token: None 2024-02-01T10:48:48.792 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:48 GMT /vclfcn-62/obj3 2024-02-01T10:48:48.792 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fzLjtmyp3SmraQqYe9w6tjys5wA= 2024-02-01T10:48:48.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fzLjtmyp3SmraQqYe9w6tjys5wA=', 'Content-Length': '0'} 2024-02-01T10:48:48.795 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c6ceebd05a7ef689-0065bb7710-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:48.796 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:48:48.796 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:48:48.796 DEBUG:boto:path=/vclfcn-62/?prefix=obj3 2024-02-01T10:48:48.796 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj3 2024-02-01T10:48:48.796 DEBUG:boto:Method: GET 2024-02-01T10:48:48.796 DEBUG:boto:Path: /vclfcn-62/?prefix=obj3 2024-02-01T10:48:48.796 DEBUG:boto:Data: 2024-02-01T10:48:48.796 DEBUG:boto:Headers: {} 2024-02-01T10:48:48.796 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:48.796 DEBUG:boto:Port: 8000 2024-02-01T10:48:48.797 DEBUG:boto:Params: {} 2024-02-01T10:48:48.797 DEBUG:boto:Token: None 2024-02-01T10:48:48.797 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:48 GMT /vclfcn-62/ 2024-02-01T10:48:48.797 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ci/9jktEulOllMe5jPmCFggWOZw= 2024-02-01T10:48:48.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ci/9jktEulOllMe5jPmCFggWOZw=', 'Content-Length': '0'} 2024-02-01T10:48:48.802 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4749f4ef73ed3d6-0065bb7710-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:48.844 DEBUG:boto:b'vclfcn-62obj31000falseobj32024-02-01T10:48:26.384Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:48.844 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:48:48.845 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:48:48.845 DEBUG:boto:path=/vclfcn-62/?prefix=obj3 2024-02-01T10:48:48.845 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj3 2024-02-01T10:48:48.845 DEBUG:boto:Method: GET 2024-02-01T10:48:48.845 DEBUG:boto:Path: /vclfcn-62/?prefix=obj3 2024-02-01T10:48:48.845 DEBUG:boto:Data: 2024-02-01T10:48:48.845 DEBUG:boto:Headers: {} 2024-02-01T10:48:48.845 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:48.845 DEBUG:boto:Port: 8000 2024-02-01T10:48:48.845 DEBUG:boto:Params: {} 2024-02-01T10:48:48.846 DEBUG:boto:Token: None 2024-02-01T10:48:48.846 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:48 GMT /vclfcn-62/ 2024-02-01T10:48:48.846 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ci/9jktEulOllMe5jPmCFggWOZw= 2024-02-01T10:48:48.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ci/9jktEulOllMe5jPmCFggWOZw=', 'Content-Length': '0'} 2024-02-01T10:48:48.851 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008bbd83f3df505b69-0065bb7710-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:48.851 DEBUG:boto:b'vclfcn-62obj31000falseobj32024-02-01T10:48:26.384Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:48.852 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:48:48.852 DEBUG:boto:path=/vclfcn-62/obj4 2024-02-01T10:48:48.852 DEBUG:boto:auth_path=/vclfcn-62/obj4 2024-02-01T10:48:48.852 DEBUG:boto:path=/vclfcn-62/obj4?versionId=null 2024-02-01T10:48:48.852 DEBUG:boto:auth_path=/vclfcn-62/obj4?versionId=null 2024-02-01T10:48:48.852 DEBUG:boto:Method: GET 2024-02-01T10:48:48.852 DEBUG:boto:Path: /vclfcn-62/obj4?versionId=null 2024-02-01T10:48:48.853 DEBUG:boto:Data: 2024-02-01T10:48:48.853 DEBUG:boto:Headers: {} 2024-02-01T10:48:48.853 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:48.853 DEBUG:boto:Port: 8000 2024-02-01T10:48:48.853 DEBUG:boto:Params: {} 2024-02-01T10:48:48.853 DEBUG:boto:Token: None 2024-02-01T10:48:48.853 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:48 GMT /vclfcn-62/obj4?versionId=null 2024-02-01T10:48:48.853 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:P89zXefZzxmL4tkbShLWkBlP6s4= 2024-02-01T10:48:48.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:P89zXefZzxmL4tkbShLWkBlP6s4=', 'Content-Length': '0'} 2024-02-01T10:48:48.856 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ae198b0de58d1bd1-0065bb7710-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:48.900 DEBUG:boto:path=/vclfcn-62/obj4 2024-02-01T10:48:48.900 DEBUG:boto:auth_path=/vclfcn-62/obj4 2024-02-01T10:48:48.900 DEBUG:boto:path=/vclfcn-62/obj4?versionId=null 2024-02-01T10:48:48.901 DEBUG:boto:auth_path=/vclfcn-62/obj4?versionId=null 2024-02-01T10:48:48.901 DEBUG:boto:Method: GET 2024-02-01T10:48:48.901 DEBUG:boto:Path: /vclfcn-62/obj4?versionId=null 2024-02-01T10:48:48.901 DEBUG:boto:Data: 2024-02-01T10:48:48.901 DEBUG:boto:Headers: {} 2024-02-01T10:48:48.901 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:48.901 DEBUG:boto:Port: 8000 2024-02-01T10:48:48.901 DEBUG:boto:Params: {} 2024-02-01T10:48:48.901 DEBUG:boto:Token: None 2024-02-01T10:48:48.902 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:48 GMT /vclfcn-62/obj4?versionId=null 2024-02-01T10:48:48.902 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:P89zXefZzxmL4tkbShLWkBlP6s4= 2024-02-01T10:48:48.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:P89zXefZzxmL4tkbShLWkBlP6s4=', 'Content-Length': '0'} 2024-02-01T10:48:48.905 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004bad1f2376fb7efb-0065bb7710-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:48.906 DEBUG:boto:path=/vclfcn-62/obj4 2024-02-01T10:48:48.906 DEBUG:boto:auth_path=/vclfcn-62/obj4 2024-02-01T10:48:48.906 DEBUG:boto:path=/vclfcn-62/obj4?versionId=null 2024-02-01T10:48:48.906 DEBUG:boto:auth_path=/vclfcn-62/obj4?versionId=null 2024-02-01T10:48:48.906 DEBUG:boto:Method: HEAD 2024-02-01T10:48:48.906 DEBUG:boto:Path: /vclfcn-62/obj4?versionId=null 2024-02-01T10:48:48.906 DEBUG:boto:Data: 2024-02-01T10:48:48.906 DEBUG:boto:Headers: {} 2024-02-01T10:48:48.907 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:48.907 DEBUG:boto:Port: 8000 2024-02-01T10:48:48.907 DEBUG:boto:Params: {} 2024-02-01T10:48:48.907 DEBUG:boto:Token: None 2024-02-01T10:48:48.907 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:48 GMT /vclfcn-62/obj4?versionId=null 2024-02-01T10:48:48.907 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9Ld835Q9w2j/mP9A4rVwlD2nzaI= 2024-02-01T10:48:48.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9Ld835Q9w2j/mP9A4rVwlD2nzaI=', 'Content-Length': '0'} 2024-02-01T10:48:48.910 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000411b41fd44a6d382-0065bb7710-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:48.910 DEBUG:boto:path=/vclfcn-62/obj4 2024-02-01T10:48:48.910 DEBUG:boto:auth_path=/vclfcn-62/obj4 2024-02-01T10:48:48.911 DEBUG:boto:path=/vclfcn-62/obj4?versionId=null 2024-02-01T10:48:48.911 DEBUG:boto:auth_path=/vclfcn-62/obj4?versionId=null 2024-02-01T10:48:48.911 DEBUG:boto:Method: HEAD 2024-02-01T10:48:48.911 DEBUG:boto:Path: /vclfcn-62/obj4?versionId=null 2024-02-01T10:48:48.911 DEBUG:boto:Data: 2024-02-01T10:48:48.911 DEBUG:boto:Headers: {} 2024-02-01T10:48:48.911 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:48.911 DEBUG:boto:Port: 8000 2024-02-01T10:48:48.911 DEBUG:boto:Params: {} 2024-02-01T10:48:48.911 DEBUG:boto:Token: None 2024-02-01T10:48:48.912 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:48 GMT /vclfcn-62/obj4?versionId=null 2024-02-01T10:48:48.912 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9Ld835Q9w2j/mP9A4rVwlD2nzaI= 2024-02-01T10:48:48.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9Ld835Q9w2j/mP9A4rVwlD2nzaI=', 'Content-Length': '0'} 2024-02-01T10:48:48.914 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fadcdedf89e89e8f-0065bb7710-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:48.915 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:48:48.915 DEBUG:boto:path=/vclfcn-62/obj4 2024-02-01T10:48:48.915 DEBUG:boto:auth_path=/vclfcn-62/obj4 2024-02-01T10:48:48.915 DEBUG:boto:path=/vclfcn-62/obj4?versionId=null 2024-02-01T10:48:48.915 DEBUG:boto:auth_path=/vclfcn-62/obj4?versionId=null 2024-02-01T10:48:48.915 DEBUG:boto:Method: GET 2024-02-01T10:48:48.915 DEBUG:boto:Path: /vclfcn-62/obj4?versionId=null 2024-02-01T10:48:48.916 DEBUG:boto:Data: 2024-02-01T10:48:48.916 DEBUG:boto:Headers: {} 2024-02-01T10:48:48.916 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:48.916 DEBUG:boto:Port: 8000 2024-02-01T10:48:48.916 DEBUG:boto:Params: {} 2024-02-01T10:48:48.916 DEBUG:boto:Token: None 2024-02-01T10:48:48.916 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:48 GMT /vclfcn-62/obj4?versionId=null 2024-02-01T10:48:48.916 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:P89zXefZzxmL4tkbShLWkBlP6s4= 2024-02-01T10:48:48.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:P89zXefZzxmL4tkbShLWkBlP6s4=', 'Content-Length': '0'} 2024-02-01T10:48:48.919 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e127188916deebaf-0065bb7710-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:48.964 DEBUG:boto:path=/vclfcn-62/obj4 2024-02-01T10:48:48.964 DEBUG:boto:auth_path=/vclfcn-62/obj4 2024-02-01T10:48:48.964 DEBUG:boto:path=/vclfcn-62/obj4?versionId=null 2024-02-01T10:48:48.964 DEBUG:boto:auth_path=/vclfcn-62/obj4?versionId=null 2024-02-01T10:48:48.965 DEBUG:boto:Method: GET 2024-02-01T10:48:48.965 DEBUG:boto:Path: /vclfcn-62/obj4?versionId=null 2024-02-01T10:48:48.965 DEBUG:boto:Data: 2024-02-01T10:48:48.965 DEBUG:boto:Headers: {} 2024-02-01T10:48:48.965 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:48.965 DEBUG:boto:Port: 8000 2024-02-01T10:48:48.965 DEBUG:boto:Params: {} 2024-02-01T10:48:48.965 DEBUG:boto:Token: None 2024-02-01T10:48:48.965 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:48 GMT /vclfcn-62/obj4?versionId=null 2024-02-01T10:48:48.966 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:P89zXefZzxmL4tkbShLWkBlP6s4= 2024-02-01T10:48:48.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:P89zXefZzxmL4tkbShLWkBlP6s4=', 'Content-Length': '0'} 2024-02-01T10:48:48.968 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005b641e0c26527974-0065bb7710-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:48.969 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:48:48.969 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:48:48.969 DEBUG:boto:path=/vclfcn-62/?prefix=obj4 2024-02-01T10:48:48.969 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj4 2024-02-01T10:48:48.969 DEBUG:boto:Method: GET 2024-02-01T10:48:48.969 DEBUG:boto:Path: /vclfcn-62/?prefix=obj4 2024-02-01T10:48:48.969 DEBUG:boto:Data: 2024-02-01T10:48:48.969 DEBUG:boto:Headers: {} 2024-02-01T10:48:48.970 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:48.970 DEBUG:boto:Port: 8000 2024-02-01T10:48:48.970 DEBUG:boto:Params: {} 2024-02-01T10:48:48.970 DEBUG:boto:Token: None 2024-02-01T10:48:48.970 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:48 GMT /vclfcn-62/ 2024-02-01T10:48:48.970 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ci/9jktEulOllMe5jPmCFggWOZw= 2024-02-01T10:48:48.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ci/9jktEulOllMe5jPmCFggWOZw=', 'Content-Length': '0'} 2024-02-01T10:48:48.974 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000016d2a27c620a907-0065bb7710-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:48.975 DEBUG:boto:b'vclfcn-62obj41000falseobj42024-02-01T10:48:26.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:48.975 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:48:48.975 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:48:48.975 DEBUG:boto:path=/vclfcn-62/?prefix=obj4 2024-02-01T10:48:48.975 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj4 2024-02-01T10:48:48.976 DEBUG:boto:Method: GET 2024-02-01T10:48:48.976 DEBUG:boto:Path: /vclfcn-62/?prefix=obj4 2024-02-01T10:48:48.976 DEBUG:boto:Data: 2024-02-01T10:48:48.976 DEBUG:boto:Headers: {} 2024-02-01T10:48:48.976 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:48.976 DEBUG:boto:Port: 8000 2024-02-01T10:48:48.976 DEBUG:boto:Params: {} 2024-02-01T10:48:48.976 DEBUG:boto:Token: None 2024-02-01T10:48:48.976 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:48 GMT /vclfcn-62/ 2024-02-01T10:48:48.976 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ci/9jktEulOllMe5jPmCFggWOZw= 2024-02-01T10:48:48.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ci/9jktEulOllMe5jPmCFggWOZw=', 'Content-Length': '0'} 2024-02-01T10:48:48.981 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000302730bc794540ad-0065bb7710-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:48 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:49.024 DEBUG:boto:b'vclfcn-62obj41000falseobj42024-02-01T10:48:26.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:49.024 DEBUG:boto:path=/vclfcn-62/obj4 2024-02-01T10:48:49.025 DEBUG:boto:auth_path=/vclfcn-62/obj4 2024-02-01T10:48:49.025 DEBUG:boto:Method: HEAD 2024-02-01T10:48:49.025 DEBUG:boto:Path: /vclfcn-62/obj4 2024-02-01T10:48:49.025 DEBUG:boto:Data: 2024-02-01T10:48:49.025 DEBUG:boto:Headers: {} 2024-02-01T10:48:49.025 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:49.025 DEBUG:boto:Port: 8000 2024-02-01T10:48:49.025 DEBUG:boto:Params: {} 2024-02-01T10:48:49.026 DEBUG:boto:Token: None 2024-02-01T10:48:49.026 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:49 GMT /vclfcn-62/obj4 2024-02-01T10:48:49.026 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8wYDX6ZJq99BYdW7BtUM281iAgo= 2024-02-01T10:48:49.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8wYDX6ZJq99BYdW7BtUM281iAgo=', 'Content-Length': '0'} 2024-02-01T10:48:49.029 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cad6e80a9adfd315-0065bb7711-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:49.029 DEBUG:boto:path=/vclfcn-62/obj4 2024-02-01T10:48:49.029 DEBUG:boto:auth_path=/vclfcn-62/obj4 2024-02-01T10:48:49.029 DEBUG:boto:Method: HEAD 2024-02-01T10:48:49.029 DEBUG:boto:Path: /vclfcn-62/obj4 2024-02-01T10:48:49.029 DEBUG:boto:Data: 2024-02-01T10:48:49.029 DEBUG:boto:Headers: {} 2024-02-01T10:48:49.029 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:49.030 DEBUG:boto:Port: 8000 2024-02-01T10:48:49.030 DEBUG:boto:Params: {} 2024-02-01T10:48:49.030 DEBUG:boto:Token: None 2024-02-01T10:48:49.030 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:49 GMT /vclfcn-62/obj4 2024-02-01T10:48:49.030 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8wYDX6ZJq99BYdW7BtUM281iAgo= 2024-02-01T10:48:49.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8wYDX6ZJq99BYdW7BtUM281iAgo=', 'Content-Length': '0'} 2024-02-01T10:48:49.033 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c16c50aa5699a607-0065bb7711-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:49.033 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:48:49.033 DEBUG:boto:path=/vclfcn-62/obj4 2024-02-01T10:48:49.033 DEBUG:boto:auth_path=/vclfcn-62/obj4 2024-02-01T10:48:49.033 DEBUG:boto:Method: GET 2024-02-01T10:48:49.033 DEBUG:boto:Path: /vclfcn-62/obj4 2024-02-01T10:48:49.034 DEBUG:boto:Data: 2024-02-01T10:48:49.034 DEBUG:boto:Headers: {} 2024-02-01T10:48:49.034 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:49.034 DEBUG:boto:Port: 8000 2024-02-01T10:48:49.034 DEBUG:boto:Params: {} 2024-02-01T10:48:49.034 DEBUG:boto:Token: None 2024-02-01T10:48:49.034 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:49 GMT /vclfcn-62/obj4 2024-02-01T10:48:49.034 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:izAsOvwJFXJ1N2IduUtQtRC0xr8= 2024-02-01T10:48:49.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:izAsOvwJFXJ1N2IduUtQtRC0xr8=', 'Content-Length': '0'} 2024-02-01T10:48:49.037 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f23e568f6c543e98-0065bb7711-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:49.084 DEBUG:boto:path=/vclfcn-62/obj4 2024-02-01T10:48:49.084 DEBUG:boto:auth_path=/vclfcn-62/obj4 2024-02-01T10:48:49.084 DEBUG:boto:Method: GET 2024-02-01T10:48:49.084 DEBUG:boto:Path: /vclfcn-62/obj4 2024-02-01T10:48:49.084 DEBUG:boto:Data: 2024-02-01T10:48:49.085 DEBUG:boto:Headers: {} 2024-02-01T10:48:49.085 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:49.085 DEBUG:boto:Port: 8000 2024-02-01T10:48:49.085 DEBUG:boto:Params: {} 2024-02-01T10:48:49.085 DEBUG:boto:Token: None 2024-02-01T10:48:49.085 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:49 GMT /vclfcn-62/obj4 2024-02-01T10:48:49.085 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:izAsOvwJFXJ1N2IduUtQtRC0xr8= 2024-02-01T10:48:49.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:izAsOvwJFXJ1N2IduUtQtRC0xr8=', 'Content-Length': '0'} 2024-02-01T10:48:49.088 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000046a70399e31fe5ef-0065bb7711-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:49.089 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:48:49.089 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:48:49.089 DEBUG:boto:path=/vclfcn-62/?prefix=obj4 2024-02-01T10:48:49.089 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj4 2024-02-01T10:48:49.089 DEBUG:boto:Method: GET 2024-02-01T10:48:49.089 DEBUG:boto:Path: /vclfcn-62/?prefix=obj4 2024-02-01T10:48:49.089 DEBUG:boto:Data: 2024-02-01T10:48:49.089 DEBUG:boto:Headers: {} 2024-02-01T10:48:49.089 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:49.089 DEBUG:boto:Port: 8000 2024-02-01T10:48:49.090 DEBUG:boto:Params: {} 2024-02-01T10:48:49.090 DEBUG:boto:Token: None 2024-02-01T10:48:49.090 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:49 GMT /vclfcn-62/ 2024-02-01T10:48:49.090 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5o2GaAhuOVfi8KdCqOyMNa3u4m4= 2024-02-01T10:48:49.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5o2GaAhuOVfi8KdCqOyMNa3u4m4=', 'Content-Length': '0'} 2024-02-01T10:48:49.094 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008eea7592c9f293a1-0065bb7711-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:49.136 DEBUG:boto:b'vclfcn-62obj41000falseobj42024-02-01T10:48:26.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:49.136 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:48:49.137 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:48:49.137 DEBUG:boto:path=/vclfcn-62/?prefix=obj4 2024-02-01T10:48:49.137 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj4 2024-02-01T10:48:49.137 DEBUG:boto:Method: GET 2024-02-01T10:48:49.137 DEBUG:boto:Path: /vclfcn-62/?prefix=obj4 2024-02-01T10:48:49.137 DEBUG:boto:Data: 2024-02-01T10:48:49.137 DEBUG:boto:Headers: {} 2024-02-01T10:48:49.137 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:49.137 DEBUG:boto:Port: 8000 2024-02-01T10:48:49.137 DEBUG:boto:Params: {} 2024-02-01T10:48:49.138 DEBUG:boto:Token: None 2024-02-01T10:48:49.138 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:49 GMT /vclfcn-62/ 2024-02-01T10:48:49.138 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5o2GaAhuOVfi8KdCqOyMNa3u4m4= 2024-02-01T10:48:49.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5o2GaAhuOVfi8KdCqOyMNa3u4m4=', 'Content-Length': '0'} 2024-02-01T10:48:49.142 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001bfeb1f96cb27cfc-0065bb7711-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:49.143 DEBUG:boto:b'vclfcn-62obj41000falseobj42024-02-01T10:48:26.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:49.143 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-62 zones={test-zone2, test-zone1} 2024-02-01T10:48:49.143 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-62 zones={test-zone3, test-zone1} 2024-02-01T10:48:49.143 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:48:49.143 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:48:49.143 DEBUG:boto:Method: HEAD 2024-02-01T10:48:49.143 DEBUG:boto:Path: /vclfcn-62/ 2024-02-01T10:48:49.144 DEBUG:boto:Data: 2024-02-01T10:48:49.144 DEBUG:boto:Headers: {} 2024-02-01T10:48:49.144 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:49.144 DEBUG:boto:Port: 8001 2024-02-01T10:48:49.144 DEBUG:boto:Params: {} 2024-02-01T10:48:49.144 DEBUG:boto:Token: None 2024-02-01T10:48:49.144 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:49 GMT /vclfcn-62/ 2024-02-01T10:48:49.144 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QiN+s9wrf8kBIU9/4ptOAZhJyGE= 2024-02-01T10:48:49.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QiN+s9wrf8kBIU9/4ptOAZhJyGE=', 'Content-Length': '0'} 2024-02-01T10:48:49.148 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002f590fb642c9a21e-0065bb7711-5762-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:49.148 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:48:49.149 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:48:49.149 DEBUG:boto:Method: HEAD 2024-02-01T10:48:49.149 DEBUG:boto:Path: /vclfcn-62/ 2024-02-01T10:48:49.149 DEBUG:boto:Data: 2024-02-01T10:48:49.149 DEBUG:boto:Headers: {} 2024-02-01T10:48:49.149 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:49.149 DEBUG:boto:Port: 8000 2024-02-01T10:48:49.149 DEBUG:boto:Params: {} 2024-02-01T10:48:49.149 DEBUG:boto:Token: None 2024-02-01T10:48:49.149 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:49 GMT /vclfcn-62/ 2024-02-01T10:48:49.150 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QiN+s9wrf8kBIU9/4ptOAZhJyGE= 2024-02-01T10:48:49.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QiN+s9wrf8kBIU9/4ptOAZhJyGE=', 'Content-Length': '0'} 2024-02-01T10:48:49.154 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bdbc3b7d2add3a96-0065bb7711-5765-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:49.154 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:48:49.154 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:48:49.154 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:48:49.154 DEBUG:boto:path=/vclfcn-62/?versions&max-keys=999 2024-02-01T10:48:49.154 DEBUG:boto:auth_path=/vclfcn-62/?versions&max-keys=999 2024-02-01T10:48:49.154 DEBUG:boto:Method: GET 2024-02-01T10:48:49.155 DEBUG:boto:Path: /vclfcn-62/?versions&max-keys=999 2024-02-01T10:48:49.155 DEBUG:boto:Data: 2024-02-01T10:48:49.155 DEBUG:boto:Headers: {} 2024-02-01T10:48:49.155 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:49.155 DEBUG:boto:Port: 8001 2024-02-01T10:48:49.155 DEBUG:boto:Params: {} 2024-02-01T10:48:49.155 DEBUG:boto:Token: None 2024-02-01T10:48:49.155 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:49 GMT /vclfcn-62/?versions 2024-02-01T10:48:49.155 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FiyAUVrX0t/aCtFIce4DjSZSYOQ= 2024-02-01T10:48:49.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FiyAUVrX0t/aCtFIce4DjSZSYOQ=', 'Content-Length': '0'} 2024-02-01T10:48:49.160 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c049e3136491b237-0065bb7711-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:49.160 DEBUG:boto:b'vclfcn-62999falseobj1nulltrue2024-02-01T10:48:26.315Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:48:26.333Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:48:26.384Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:48:26.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:49.161 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-02-01T10:48:49.161 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-02-01T10:48:49.161 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-02-01T10:48:49.161 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-02-01T10:48:49.161 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:48:49.161 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:48:49.161 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:48:49.161 DEBUG:boto:path=/vclfcn-62/?versions&max-keys=999 2024-02-01T10:48:49.161 DEBUG:boto:auth_path=/vclfcn-62/?versions&max-keys=999 2024-02-01T10:48:49.161 DEBUG:boto:Method: GET 2024-02-01T10:48:49.162 DEBUG:boto:Path: /vclfcn-62/?versions&max-keys=999 2024-02-01T10:48:49.162 DEBUG:boto:Data: 2024-02-01T10:48:49.162 DEBUG:boto:Headers: {} 2024-02-01T10:48:49.162 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:49.162 DEBUG:boto:Port: 8000 2024-02-01T10:48:49.162 DEBUG:boto:Params: {} 2024-02-01T10:48:49.162 DEBUG:boto:Token: None 2024-02-01T10:48:49.162 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:49 GMT /vclfcn-62/?versions 2024-02-01T10:48:49.162 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FiyAUVrX0t/aCtFIce4DjSZSYOQ= 2024-02-01T10:48:49.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FiyAUVrX0t/aCtFIce4DjSZSYOQ=', 'Content-Length': '0'} 2024-02-01T10:48:49.166 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008350765921d412f6-0065bb7711-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:49.166 DEBUG:boto:b'vclfcn-62999falseobj1nulltrue2024-02-01T10:48:26.315Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:48:26.333Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:48:26.384Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:48:26.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:49.167 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-02-01T10:48:49.167 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-02-01T10:48:49.167 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-02-01T10:48:49.168 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-02-01T10:48:49.168 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:48:49.168 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:48:49.168 DEBUG:boto:path=/vclfcn-62/?versions&max-keys=999 2024-02-01T10:48:49.168 DEBUG:boto:auth_path=/vclfcn-62/?versions&max-keys=999 2024-02-01T10:48:49.168 DEBUG:boto:Method: GET 2024-02-01T10:48:49.168 DEBUG:boto:Path: /vclfcn-62/?versions&max-keys=999 2024-02-01T10:48:49.168 DEBUG:boto:Data: 2024-02-01T10:48:49.168 DEBUG:boto:Headers: {} 2024-02-01T10:48:49.169 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:49.169 DEBUG:boto:Port: 8001 2024-02-01T10:48:49.169 DEBUG:boto:Params: {} 2024-02-01T10:48:49.169 DEBUG:boto:Token: None 2024-02-01T10:48:49.169 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:49 GMT /vclfcn-62/?versions 2024-02-01T10:48:49.169 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FiyAUVrX0t/aCtFIce4DjSZSYOQ= 2024-02-01T10:48:49.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FiyAUVrX0t/aCtFIce4DjSZSYOQ=', 'Content-Length': '0'} 2024-02-01T10:48:49.173 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000afc101ed91c9d89e-0065bb7711-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:49.174 DEBUG:boto:b'vclfcn-62999falseobj1nulltrue2024-02-01T10:48:26.315Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:48:26.333Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:48:26.384Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:48:26.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:49.174 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:48:49.174 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:48:49.174 DEBUG:boto:path=/vclfcn-62/?versions&max-keys=999 2024-02-01T10:48:49.174 DEBUG:boto:auth_path=/vclfcn-62/?versions&max-keys=999 2024-02-01T10:48:49.174 DEBUG:boto:Method: GET 2024-02-01T10:48:49.174 DEBUG:boto:Path: /vclfcn-62/?versions&max-keys=999 2024-02-01T10:48:49.175 DEBUG:boto:Data: 2024-02-01T10:48:49.175 DEBUG:boto:Headers: {} 2024-02-01T10:48:49.175 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:49.175 DEBUG:boto:Port: 8000 2024-02-01T10:48:49.175 DEBUG:boto:Params: {} 2024-02-01T10:48:49.175 DEBUG:boto:Token: None 2024-02-01T10:48:49.175 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:49 GMT /vclfcn-62/?versions 2024-02-01T10:48:49.175 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FiyAUVrX0t/aCtFIce4DjSZSYOQ= 2024-02-01T10:48:49.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FiyAUVrX0t/aCtFIce4DjSZSYOQ=', 'Content-Length': '0'} 2024-02-01T10:48:49.180 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c525cc9554b540a-0065bb7711-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:49.180 DEBUG:boto:b'vclfcn-62999falseobj1nulltrue2024-02-01T10:48:26.315Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:48:26.333Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:48:26.384Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:48:26.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:49.180 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:48:49.181 DEBUG:boto:path=/vclfcn-62/obj1 2024-02-01T10:48:49.181 DEBUG:boto:auth_path=/vclfcn-62/obj1 2024-02-01T10:48:49.181 DEBUG:boto:path=/vclfcn-62/obj1?versionId=null 2024-02-01T10:48:49.181 DEBUG:boto:auth_path=/vclfcn-62/obj1?versionId=null 2024-02-01T10:48:49.181 DEBUG:boto:Method: GET 2024-02-01T10:48:49.181 DEBUG:boto:Path: /vclfcn-62/obj1?versionId=null 2024-02-01T10:48:49.181 DEBUG:boto:Data: 2024-02-01T10:48:49.181 DEBUG:boto:Headers: {} 2024-02-01T10:48:49.181 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:49.181 DEBUG:boto:Port: 8001 2024-02-01T10:48:49.181 DEBUG:boto:Params: {} 2024-02-01T10:48:49.182 DEBUG:boto:Token: None 2024-02-01T10:48:49.182 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:49 GMT /vclfcn-62/obj1?versionId=null 2024-02-01T10:48:49.182 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Szxfs6M22m0KhbW+w0UKiqstv5U= 2024-02-01T10:48:49.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Szxfs6M22m0KhbW+w0UKiqstv5U=', 'Content-Length': '0'} 2024-02-01T10:48:49.185 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f6ca461c585ca904-0065bb7711-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:49.228 DEBUG:boto:path=/vclfcn-62/obj1 2024-02-01T10:48:49.228 DEBUG:boto:auth_path=/vclfcn-62/obj1 2024-02-01T10:48:49.228 DEBUG:boto:path=/vclfcn-62/obj1?versionId=null 2024-02-01T10:48:49.228 DEBUG:boto:auth_path=/vclfcn-62/obj1?versionId=null 2024-02-01T10:48:49.229 DEBUG:boto:Method: GET 2024-02-01T10:48:49.229 DEBUG:boto:Path: /vclfcn-62/obj1?versionId=null 2024-02-01T10:48:49.229 DEBUG:boto:Data: 2024-02-01T10:48:49.229 DEBUG:boto:Headers: {} 2024-02-01T10:48:49.229 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:49.229 DEBUG:boto:Port: 8000 2024-02-01T10:48:49.229 DEBUG:boto:Params: {} 2024-02-01T10:48:49.229 DEBUG:boto:Token: None 2024-02-01T10:48:49.229 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:49 GMT /vclfcn-62/obj1?versionId=null 2024-02-01T10:48:49.229 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Szxfs6M22m0KhbW+w0UKiqstv5U= 2024-02-01T10:48:49.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Szxfs6M22m0KhbW+w0UKiqstv5U=', 'Content-Length': '0'} 2024-02-01T10:48:49.232 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dd80daf0774bceb6-0065bb7711-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:49.232 DEBUG:boto:path=/vclfcn-62/obj1 2024-02-01T10:48:49.233 DEBUG:boto:auth_path=/vclfcn-62/obj1 2024-02-01T10:48:49.233 DEBUG:boto:path=/vclfcn-62/obj1?versionId=null 2024-02-01T10:48:49.233 DEBUG:boto:auth_path=/vclfcn-62/obj1?versionId=null 2024-02-01T10:48:49.233 DEBUG:boto:Method: HEAD 2024-02-01T10:48:49.233 DEBUG:boto:Path: /vclfcn-62/obj1?versionId=null 2024-02-01T10:48:49.233 DEBUG:boto:Data: 2024-02-01T10:48:49.233 DEBUG:boto:Headers: {} 2024-02-01T10:48:49.233 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:49.233 DEBUG:boto:Port: 8001 2024-02-01T10:48:49.233 DEBUG:boto:Params: {} 2024-02-01T10:48:49.234 DEBUG:boto:Token: None 2024-02-01T10:48:49.234 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:49 GMT /vclfcn-62/obj1?versionId=null 2024-02-01T10:48:49.234 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mpXKnizOzXYQlTfxHcBny71jASw= 2024-02-01T10:48:49.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mpXKnizOzXYQlTfxHcBny71jASw=', 'Content-Length': '0'} 2024-02-01T10:48:49.237 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000141a9801eb794cd9-0065bb7711-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:49.237 DEBUG:boto:path=/vclfcn-62/obj1 2024-02-01T10:48:49.237 DEBUG:boto:auth_path=/vclfcn-62/obj1 2024-02-01T10:48:49.237 DEBUG:boto:path=/vclfcn-62/obj1?versionId=null 2024-02-01T10:48:49.238 DEBUG:boto:auth_path=/vclfcn-62/obj1?versionId=null 2024-02-01T10:48:49.238 DEBUG:boto:Method: HEAD 2024-02-01T10:48:49.238 DEBUG:boto:Path: /vclfcn-62/obj1?versionId=null 2024-02-01T10:48:49.238 DEBUG:boto:Data: 2024-02-01T10:48:49.238 DEBUG:boto:Headers: {} 2024-02-01T10:48:49.238 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:49.238 DEBUG:boto:Port: 8000 2024-02-01T10:48:49.238 DEBUG:boto:Params: {} 2024-02-01T10:48:49.238 DEBUG:boto:Token: None 2024-02-01T10:48:49.238 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:49 GMT /vclfcn-62/obj1?versionId=null 2024-02-01T10:48:49.238 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mpXKnizOzXYQlTfxHcBny71jASw= 2024-02-01T10:48:49.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mpXKnizOzXYQlTfxHcBny71jASw=', 'Content-Length': '0'} 2024-02-01T10:48:49.241 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008dfc7e513fa25398-0065bb7711-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:49.242 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:48:49.242 DEBUG:boto:path=/vclfcn-62/obj1 2024-02-01T10:48:49.242 DEBUG:boto:auth_path=/vclfcn-62/obj1 2024-02-01T10:48:49.242 DEBUG:boto:path=/vclfcn-62/obj1?versionId=null 2024-02-01T10:48:49.242 DEBUG:boto:auth_path=/vclfcn-62/obj1?versionId=null 2024-02-01T10:48:49.242 DEBUG:boto:Method: GET 2024-02-01T10:48:49.242 DEBUG:boto:Path: /vclfcn-62/obj1?versionId=null 2024-02-01T10:48:49.242 DEBUG:boto:Data: 2024-02-01T10:48:49.242 DEBUG:boto:Headers: {} 2024-02-01T10:48:49.242 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:49.243 DEBUG:boto:Port: 8001 2024-02-01T10:48:49.243 DEBUG:boto:Params: {} 2024-02-01T10:48:49.243 DEBUG:boto:Token: None 2024-02-01T10:48:49.243 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:49 GMT /vclfcn-62/obj1?versionId=null 2024-02-01T10:48:49.243 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Szxfs6M22m0KhbW+w0UKiqstv5U= 2024-02-01T10:48:49.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Szxfs6M22m0KhbW+w0UKiqstv5U=', 'Content-Length': '0'} 2024-02-01T10:48:49.246 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007423f35277e94cb8-0065bb7711-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:49.247 DEBUG:boto:path=/vclfcn-62/obj1 2024-02-01T10:48:49.247 DEBUG:boto:auth_path=/vclfcn-62/obj1 2024-02-01T10:48:49.247 DEBUG:boto:path=/vclfcn-62/obj1?versionId=null 2024-02-01T10:48:49.247 DEBUG:boto:auth_path=/vclfcn-62/obj1?versionId=null 2024-02-01T10:48:49.247 DEBUG:boto:Method: GET 2024-02-01T10:48:49.247 DEBUG:boto:Path: /vclfcn-62/obj1?versionId=null 2024-02-01T10:48:49.247 DEBUG:boto:Data: 2024-02-01T10:48:49.247 DEBUG:boto:Headers: {} 2024-02-01T10:48:49.247 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:49.247 DEBUG:boto:Port: 8000 2024-02-01T10:48:49.247 DEBUG:boto:Params: {} 2024-02-01T10:48:49.248 DEBUG:boto:Token: None 2024-02-01T10:48:49.248 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:49 GMT /vclfcn-62/obj1?versionId=null 2024-02-01T10:48:49.248 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Szxfs6M22m0KhbW+w0UKiqstv5U= 2024-02-01T10:48:49.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Szxfs6M22m0KhbW+w0UKiqstv5U=', 'Content-Length': '0'} 2024-02-01T10:48:49.251 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000876159e77327c9d3-0065bb7711-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:49.251 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:48:49.251 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:48:49.251 DEBUG:boto:path=/vclfcn-62/?prefix=obj1 2024-02-01T10:48:49.251 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj1 2024-02-01T10:48:49.252 DEBUG:boto:Method: GET 2024-02-01T10:48:49.252 DEBUG:boto:Path: /vclfcn-62/?prefix=obj1 2024-02-01T10:48:49.252 DEBUG:boto:Data: 2024-02-01T10:48:49.252 DEBUG:boto:Headers: {} 2024-02-01T10:48:49.252 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:49.252 DEBUG:boto:Port: 8001 2024-02-01T10:48:49.252 DEBUG:boto:Params: {} 2024-02-01T10:48:49.252 DEBUG:boto:Token: None 2024-02-01T10:48:49.252 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:49 GMT /vclfcn-62/ 2024-02-01T10:48:49.252 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5o2GaAhuOVfi8KdCqOyMNa3u4m4= 2024-02-01T10:48:49.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5o2GaAhuOVfi8KdCqOyMNa3u4m4=', 'Content-Length': '0'} 2024-02-01T10:48:49.256 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b18922bffe6f0ecb-0065bb7711-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:49.300 DEBUG:boto:b'vclfcn-62obj11000falseobj12024-02-01T10:48:26.315Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:49.301 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:48:49.301 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:48:49.301 DEBUG:boto:path=/vclfcn-62/?prefix=obj1 2024-02-01T10:48:49.301 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj1 2024-02-01T10:48:49.301 DEBUG:boto:Method: GET 2024-02-01T10:48:49.301 DEBUG:boto:Path: /vclfcn-62/?prefix=obj1 2024-02-01T10:48:49.301 DEBUG:boto:Data: 2024-02-01T10:48:49.301 DEBUG:boto:Headers: {} 2024-02-01T10:48:49.301 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:49.302 DEBUG:boto:Port: 8000 2024-02-01T10:48:49.302 DEBUG:boto:Params: {} 2024-02-01T10:48:49.302 DEBUG:boto:Token: None 2024-02-01T10:48:49.302 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:49 GMT /vclfcn-62/ 2024-02-01T10:48:49.302 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5o2GaAhuOVfi8KdCqOyMNa3u4m4= 2024-02-01T10:48:49.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5o2GaAhuOVfi8KdCqOyMNa3u4m4=', 'Content-Length': '0'} 2024-02-01T10:48:49.307 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000797e0db29f74539b-0065bb7711-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:49.307 DEBUG:boto:b'vclfcn-62obj11000falseobj12024-02-01T10:48:26.315Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:49.307 DEBUG:boto:path=/vclfcn-62/obj1 2024-02-01T10:48:49.307 DEBUG:boto:auth_path=/vclfcn-62/obj1 2024-02-01T10:48:49.308 DEBUG:boto:Method: HEAD 2024-02-01T10:48:49.308 DEBUG:boto:Path: /vclfcn-62/obj1 2024-02-01T10:48:49.308 DEBUG:boto:Data: 2024-02-01T10:48:49.308 DEBUG:boto:Headers: {} 2024-02-01T10:48:49.308 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:49.308 DEBUG:boto:Port: 8001 2024-02-01T10:48:49.308 DEBUG:boto:Params: {} 2024-02-01T10:48:49.308 DEBUG:boto:Token: None 2024-02-01T10:48:49.309 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:49 GMT /vclfcn-62/obj1 2024-02-01T10:48:49.309 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7uiy2EjeG2Fz5pEwKBZynlr5B6E= 2024-02-01T10:48:49.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7uiy2EjeG2Fz5pEwKBZynlr5B6E=', 'Content-Length': '0'} 2024-02-01T10:48:49.312 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000081f72461091077e4-0065bb7711-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:49.312 DEBUG:boto:path=/vclfcn-62/obj1 2024-02-01T10:48:49.313 DEBUG:boto:auth_path=/vclfcn-62/obj1 2024-02-01T10:48:49.313 DEBUG:boto:Method: HEAD 2024-02-01T10:48:49.313 DEBUG:boto:Path: /vclfcn-62/obj1 2024-02-01T10:48:49.313 DEBUG:boto:Data: 2024-02-01T10:48:49.314 DEBUG:boto:Headers: {} 2024-02-01T10:48:49.314 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:49.314 DEBUG:boto:Port: 8000 2024-02-01T10:48:49.314 DEBUG:boto:Params: {} 2024-02-01T10:48:49.315 DEBUG:boto:Token: None 2024-02-01T10:48:49.315 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:49 GMT /vclfcn-62/obj1 2024-02-01T10:48:49.315 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7uiy2EjeG2Fz5pEwKBZynlr5B6E= 2024-02-01T10:48:49.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7uiy2EjeG2Fz5pEwKBZynlr5B6E=', 'Content-Length': '0'} 2024-02-01T10:48:49.318 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000714f8639f2ae373c-0065bb7711-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:49.318 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:48:49.318 DEBUG:boto:path=/vclfcn-62/obj1 2024-02-01T10:48:49.318 DEBUG:boto:auth_path=/vclfcn-62/obj1 2024-02-01T10:48:49.319 DEBUG:boto:Method: GET 2024-02-01T10:48:49.319 DEBUG:boto:Path: /vclfcn-62/obj1 2024-02-01T10:48:49.319 DEBUG:boto:Data: 2024-02-01T10:48:49.319 DEBUG:boto:Headers: {} 2024-02-01T10:48:49.319 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:49.319 DEBUG:boto:Port: 8001 2024-02-01T10:48:49.320 DEBUG:boto:Params: {} 2024-02-01T10:48:49.320 DEBUG:boto:Token: None 2024-02-01T10:48:49.320 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:49 GMT /vclfcn-62/obj1 2024-02-01T10:48:49.320 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:edNxG1eT1NYv3ixd1pfSTHISZTo= 2024-02-01T10:48:49.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:edNxG1eT1NYv3ixd1pfSTHISZTo=', 'Content-Length': '0'} 2024-02-01T10:48:49.323 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b821ab58746c9b24-0065bb7711-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:49.364 DEBUG:boto:path=/vclfcn-62/obj1 2024-02-01T10:48:49.364 DEBUG:boto:auth_path=/vclfcn-62/obj1 2024-02-01T10:48:49.365 DEBUG:boto:Method: GET 2024-02-01T10:48:49.365 DEBUG:boto:Path: /vclfcn-62/obj1 2024-02-01T10:48:49.365 DEBUG:boto:Data: 2024-02-01T10:48:49.365 DEBUG:boto:Headers: {} 2024-02-01T10:48:49.365 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:49.365 DEBUG:boto:Port: 8000 2024-02-01T10:48:49.365 DEBUG:boto:Params: {} 2024-02-01T10:48:49.365 DEBUG:boto:Token: None 2024-02-01T10:48:49.366 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:49 GMT /vclfcn-62/obj1 2024-02-01T10:48:49.366 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:edNxG1eT1NYv3ixd1pfSTHISZTo= 2024-02-01T10:48:49.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:edNxG1eT1NYv3ixd1pfSTHISZTo=', 'Content-Length': '0'} 2024-02-01T10:48:49.369 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bf24912776dbfd2b-0065bb7711-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:49.369 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:48:49.369 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:48:49.369 DEBUG:boto:path=/vclfcn-62/?prefix=obj1 2024-02-01T10:48:49.370 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj1 2024-02-01T10:48:49.370 DEBUG:boto:Method: GET 2024-02-01T10:48:49.370 DEBUG:boto:Path: /vclfcn-62/?prefix=obj1 2024-02-01T10:48:49.370 DEBUG:boto:Data: 2024-02-01T10:48:49.370 DEBUG:boto:Headers: {} 2024-02-01T10:48:49.370 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:49.370 DEBUG:boto:Port: 8001 2024-02-01T10:48:49.370 DEBUG:boto:Params: {} 2024-02-01T10:48:49.371 DEBUG:boto:Token: None 2024-02-01T10:48:49.371 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:49 GMT /vclfcn-62/ 2024-02-01T10:48:49.371 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5o2GaAhuOVfi8KdCqOyMNa3u4m4= 2024-02-01T10:48:49.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5o2GaAhuOVfi8KdCqOyMNa3u4m4=', 'Content-Length': '0'} 2024-02-01T10:48:49.375 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000044a7fb5dce30ce8b-0065bb7711-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:49.416 DEBUG:boto:b'vclfcn-62obj11000falseobj12024-02-01T10:48:26.315Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:49.417 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:48:49.417 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:48:49.417 DEBUG:boto:path=/vclfcn-62/?prefix=obj1 2024-02-01T10:48:49.417 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj1 2024-02-01T10:48:49.417 DEBUG:boto:Method: GET 2024-02-01T10:48:49.417 DEBUG:boto:Path: /vclfcn-62/?prefix=obj1 2024-02-01T10:48:49.418 DEBUG:boto:Data: 2024-02-01T10:48:49.418 DEBUG:boto:Headers: {} 2024-02-01T10:48:49.418 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:49.418 DEBUG:boto:Port: 8000 2024-02-01T10:48:49.418 DEBUG:boto:Params: {} 2024-02-01T10:48:49.419 DEBUG:boto:Token: None 2024-02-01T10:48:49.419 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:49 GMT /vclfcn-62/ 2024-02-01T10:48:49.419 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5o2GaAhuOVfi8KdCqOyMNa3u4m4= 2024-02-01T10:48:49.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5o2GaAhuOVfi8KdCqOyMNa3u4m4=', 'Content-Length': '0'} 2024-02-01T10:48:49.424 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004eaf506974d4924d-0065bb7711-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:49.424 DEBUG:boto:b'vclfcn-62obj11000falseobj12024-02-01T10:48:26.315Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:49.425 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:48:49.425 DEBUG:boto:path=/vclfcn-62/obj2 2024-02-01T10:48:49.425 DEBUG:boto:auth_path=/vclfcn-62/obj2 2024-02-01T10:48:49.425 DEBUG:boto:path=/vclfcn-62/obj2?versionId=null 2024-02-01T10:48:49.426 DEBUG:boto:auth_path=/vclfcn-62/obj2?versionId=null 2024-02-01T10:48:49.426 DEBUG:boto:Method: GET 2024-02-01T10:48:49.426 DEBUG:boto:Path: /vclfcn-62/obj2?versionId=null 2024-02-01T10:48:49.426 DEBUG:boto:Data: 2024-02-01T10:48:49.426 DEBUG:boto:Headers: {} 2024-02-01T10:48:49.427 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:49.427 DEBUG:boto:Port: 8001 2024-02-01T10:48:49.427 DEBUG:boto:Params: {} 2024-02-01T10:48:49.427 DEBUG:boto:Token: None 2024-02-01T10:48:49.428 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:49 GMT /vclfcn-62/obj2?versionId=null 2024-02-01T10:48:49.428 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:VWfN/Z/EVkbEBEt6cW0xmfDXZW0= 2024-02-01T10:48:49.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:VWfN/Z/EVkbEBEt6cW0xmfDXZW0=', 'Content-Length': '0'} 2024-02-01T10:48:49.432 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003b594a4c58d55fc9-0065bb7711-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:49.432 DEBUG:boto:path=/vclfcn-62/obj2 2024-02-01T10:48:49.432 DEBUG:boto:auth_path=/vclfcn-62/obj2 2024-02-01T10:48:49.432 DEBUG:boto:path=/vclfcn-62/obj2?versionId=null 2024-02-01T10:48:49.432 DEBUG:boto:auth_path=/vclfcn-62/obj2?versionId=null 2024-02-01T10:48:49.433 DEBUG:boto:Method: GET 2024-02-01T10:48:49.433 DEBUG:boto:Path: /vclfcn-62/obj2?versionId=null 2024-02-01T10:48:49.433 DEBUG:boto:Data: 2024-02-01T10:48:49.433 DEBUG:boto:Headers: {} 2024-02-01T10:48:49.433 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:49.433 DEBUG:boto:Port: 8000 2024-02-01T10:48:49.433 DEBUG:boto:Params: {} 2024-02-01T10:48:49.434 DEBUG:boto:Token: None 2024-02-01T10:48:49.434 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:49 GMT /vclfcn-62/obj2?versionId=null 2024-02-01T10:48:49.434 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:VWfN/Z/EVkbEBEt6cW0xmfDXZW0= 2024-02-01T10:48:49.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:VWfN/Z/EVkbEBEt6cW0xmfDXZW0=', 'Content-Length': '0'} 2024-02-01T10:48:49.437 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002c03e40eb6d40822-0065bb7711-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:49.438 DEBUG:boto:path=/vclfcn-62/obj2 2024-02-01T10:48:49.438 DEBUG:boto:auth_path=/vclfcn-62/obj2 2024-02-01T10:48:49.438 DEBUG:boto:path=/vclfcn-62/obj2?versionId=null 2024-02-01T10:48:49.439 DEBUG:boto:auth_path=/vclfcn-62/obj2?versionId=null 2024-02-01T10:48:49.439 DEBUG:boto:Method: HEAD 2024-02-01T10:48:49.439 DEBUG:boto:Path: /vclfcn-62/obj2?versionId=null 2024-02-01T10:48:49.439 DEBUG:boto:Data: 2024-02-01T10:48:49.440 DEBUG:boto:Headers: {} 2024-02-01T10:48:49.440 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:49.440 DEBUG:boto:Port: 8001 2024-02-01T10:48:49.440 DEBUG:boto:Params: {} 2024-02-01T10:48:49.440 DEBUG:boto:Token: None 2024-02-01T10:48:49.440 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:49 GMT /vclfcn-62/obj2?versionId=null 2024-02-01T10:48:49.440 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:rYquLr3yvdYABhsbENLO6i325no= 2024-02-01T10:48:49.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:rYquLr3yvdYABhsbENLO6i325no=', 'Content-Length': '0'} 2024-02-01T10:48:49.443 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aca2df69bf6a4f43-0065bb7711-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:49.443 DEBUG:boto:path=/vclfcn-62/obj2 2024-02-01T10:48:49.444 DEBUG:boto:auth_path=/vclfcn-62/obj2 2024-02-01T10:48:49.444 DEBUG:boto:path=/vclfcn-62/obj2?versionId=null 2024-02-01T10:48:49.444 DEBUG:boto:auth_path=/vclfcn-62/obj2?versionId=null 2024-02-01T10:48:49.444 DEBUG:boto:Method: HEAD 2024-02-01T10:48:49.444 DEBUG:boto:Path: /vclfcn-62/obj2?versionId=null 2024-02-01T10:48:49.445 DEBUG:boto:Data: 2024-02-01T10:48:49.445 DEBUG:boto:Headers: {} 2024-02-01T10:48:49.445 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:49.445 DEBUG:boto:Port: 8000 2024-02-01T10:48:49.445 DEBUG:boto:Params: {} 2024-02-01T10:48:49.446 DEBUG:boto:Token: None 2024-02-01T10:48:49.446 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:49 GMT /vclfcn-62/obj2?versionId=null 2024-02-01T10:48:49.446 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:rYquLr3yvdYABhsbENLO6i325no= 2024-02-01T10:48:49.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:rYquLr3yvdYABhsbENLO6i325no=', 'Content-Length': '0'} 2024-02-01T10:48:49.449 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b0c3688f97006a7b-0065bb7711-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:49.450 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:48:49.450 DEBUG:boto:path=/vclfcn-62/obj2 2024-02-01T10:48:49.450 DEBUG:boto:auth_path=/vclfcn-62/obj2 2024-02-01T10:48:49.450 DEBUG:boto:path=/vclfcn-62/obj2?versionId=null 2024-02-01T10:48:49.450 DEBUG:boto:auth_path=/vclfcn-62/obj2?versionId=null 2024-02-01T10:48:49.450 DEBUG:boto:Method: GET 2024-02-01T10:48:49.451 DEBUG:boto:Path: /vclfcn-62/obj2?versionId=null 2024-02-01T10:48:49.451 DEBUG:boto:Data: 2024-02-01T10:48:49.451 DEBUG:boto:Headers: {} 2024-02-01T10:48:49.451 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:49.451 DEBUG:boto:Port: 8001 2024-02-01T10:48:49.451 DEBUG:boto:Params: {} 2024-02-01T10:48:49.452 DEBUG:boto:Token: None 2024-02-01T10:48:49.452 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:49 GMT /vclfcn-62/obj2?versionId=null 2024-02-01T10:48:49.452 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:VWfN/Z/EVkbEBEt6cW0xmfDXZW0= 2024-02-01T10:48:49.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:VWfN/Z/EVkbEBEt6cW0xmfDXZW0=', 'Content-Length': '0'} 2024-02-01T10:48:49.456 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dc0c1b91cd9f2067-0065bb7711-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:49.496 DEBUG:boto:path=/vclfcn-62/obj2 2024-02-01T10:48:49.496 DEBUG:boto:auth_path=/vclfcn-62/obj2 2024-02-01T10:48:49.497 DEBUG:boto:path=/vclfcn-62/obj2?versionId=null 2024-02-01T10:48:49.497 DEBUG:boto:auth_path=/vclfcn-62/obj2?versionId=null 2024-02-01T10:48:49.497 DEBUG:boto:Method: GET 2024-02-01T10:48:49.497 DEBUG:boto:Path: /vclfcn-62/obj2?versionId=null 2024-02-01T10:48:49.497 DEBUG:boto:Data: 2024-02-01T10:48:49.498 DEBUG:boto:Headers: {} 2024-02-01T10:48:49.498 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:49.498 DEBUG:boto:Port: 8000 2024-02-01T10:48:49.498 DEBUG:boto:Params: {} 2024-02-01T10:48:49.498 DEBUG:boto:Token: None 2024-02-01T10:48:49.499 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:49 GMT /vclfcn-62/obj2?versionId=null 2024-02-01T10:48:49.499 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:VWfN/Z/EVkbEBEt6cW0xmfDXZW0= 2024-02-01T10:48:49.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:VWfN/Z/EVkbEBEt6cW0xmfDXZW0=', 'Content-Length': '0'} 2024-02-01T10:48:49.502 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006e8f69285845d8db-0065bb7711-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:49.503 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:48:49.503 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:48:49.503 DEBUG:boto:path=/vclfcn-62/?prefix=obj2 2024-02-01T10:48:49.503 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj2 2024-02-01T10:48:49.503 DEBUG:boto:Method: GET 2024-02-01T10:48:49.503 DEBUG:boto:Path: /vclfcn-62/?prefix=obj2 2024-02-01T10:48:49.504 DEBUG:boto:Data: 2024-02-01T10:48:49.504 DEBUG:boto:Headers: {} 2024-02-01T10:48:49.504 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:49.504 DEBUG:boto:Port: 8001 2024-02-01T10:48:49.504 DEBUG:boto:Params: {} 2024-02-01T10:48:49.504 DEBUG:boto:Token: None 2024-02-01T10:48:49.504 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:49 GMT /vclfcn-62/ 2024-02-01T10:48:49.504 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5o2GaAhuOVfi8KdCqOyMNa3u4m4= 2024-02-01T10:48:49.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5o2GaAhuOVfi8KdCqOyMNa3u4m4=', 'Content-Length': '0'} 2024-02-01T10:48:49.509 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e77d2d1f7849e930-0065bb7711-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:49.509 DEBUG:boto:b'vclfcn-62obj21000falseobj22024-02-01T10:48:26.333Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:49.510 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:48:49.510 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:48:49.510 DEBUG:boto:path=/vclfcn-62/?prefix=obj2 2024-02-01T10:48:49.510 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj2 2024-02-01T10:48:49.510 DEBUG:boto:Method: GET 2024-02-01T10:48:49.510 DEBUG:boto:Path: /vclfcn-62/?prefix=obj2 2024-02-01T10:48:49.510 DEBUG:boto:Data: 2024-02-01T10:48:49.510 DEBUG:boto:Headers: {} 2024-02-01T10:48:49.510 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:49.510 DEBUG:boto:Port: 8000 2024-02-01T10:48:49.510 DEBUG:boto:Params: {} 2024-02-01T10:48:49.511 DEBUG:boto:Token: None 2024-02-01T10:48:49.511 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:49 GMT /vclfcn-62/ 2024-02-01T10:48:49.511 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5o2GaAhuOVfi8KdCqOyMNa3u4m4= 2024-02-01T10:48:49.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5o2GaAhuOVfi8KdCqOyMNa3u4m4=', 'Content-Length': '0'} 2024-02-01T10:48:49.515 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028df24017fedda27-0065bb7711-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:49.556 DEBUG:boto:b'vclfcn-62obj21000falseobj22024-02-01T10:48:26.333Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:49.557 DEBUG:boto:path=/vclfcn-62/obj2 2024-02-01T10:48:49.557 DEBUG:boto:auth_path=/vclfcn-62/obj2 2024-02-01T10:48:49.557 DEBUG:boto:Method: HEAD 2024-02-01T10:48:49.558 DEBUG:boto:Path: /vclfcn-62/obj2 2024-02-01T10:48:49.558 DEBUG:boto:Data: 2024-02-01T10:48:49.558 DEBUG:boto:Headers: {} 2024-02-01T10:48:49.558 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:49.558 DEBUG:boto:Port: 8001 2024-02-01T10:48:49.558 DEBUG:boto:Params: {} 2024-02-01T10:48:49.559 DEBUG:boto:Token: None 2024-02-01T10:48:49.559 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:49 GMT /vclfcn-62/obj2 2024-02-01T10:48:49.559 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Glpwey2WAcInVb62gxE7LNmkdr8= 2024-02-01T10:48:49.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Glpwey2WAcInVb62gxE7LNmkdr8=', 'Content-Length': '0'} 2024-02-01T10:48:49.562 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000095f19196f556bffe-0065bb7711-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:49.562 DEBUG:boto:path=/vclfcn-62/obj2 2024-02-01T10:48:49.563 DEBUG:boto:auth_path=/vclfcn-62/obj2 2024-02-01T10:48:49.563 DEBUG:boto:Method: HEAD 2024-02-01T10:48:49.563 DEBUG:boto:Path: /vclfcn-62/obj2 2024-02-01T10:48:49.563 DEBUG:boto:Data: 2024-02-01T10:48:49.563 DEBUG:boto:Headers: {} 2024-02-01T10:48:49.563 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:49.563 DEBUG:boto:Port: 8000 2024-02-01T10:48:49.563 DEBUG:boto:Params: {} 2024-02-01T10:48:49.563 DEBUG:boto:Token: None 2024-02-01T10:48:49.564 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:49 GMT /vclfcn-62/obj2 2024-02-01T10:48:49.564 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Glpwey2WAcInVb62gxE7LNmkdr8= 2024-02-01T10:48:49.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Glpwey2WAcInVb62gxE7LNmkdr8=', 'Content-Length': '0'} 2024-02-01T10:48:49.567 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000567ec94f61ab3ecc-0065bb7711-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:49.567 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:48:49.567 DEBUG:boto:path=/vclfcn-62/obj2 2024-02-01T10:48:49.567 DEBUG:boto:auth_path=/vclfcn-62/obj2 2024-02-01T10:48:49.567 DEBUG:boto:Method: GET 2024-02-01T10:48:49.567 DEBUG:boto:Path: /vclfcn-62/obj2 2024-02-01T10:48:49.568 DEBUG:boto:Data: 2024-02-01T10:48:49.568 DEBUG:boto:Headers: {} 2024-02-01T10:48:49.568 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:49.568 DEBUG:boto:Port: 8001 2024-02-01T10:48:49.568 DEBUG:boto:Params: {} 2024-02-01T10:48:49.568 DEBUG:boto:Token: None 2024-02-01T10:48:49.568 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:49 GMT /vclfcn-62/obj2 2024-02-01T10:48:49.569 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:61Ste9viqHIScFWFeEXm9IVj7zU= 2024-02-01T10:48:49.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:61Ste9viqHIScFWFeEXm9IVj7zU=', 'Content-Length': '0'} 2024-02-01T10:48:49.571 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000021157bd40e93ec83-0065bb7711-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:49.616 DEBUG:boto:path=/vclfcn-62/obj2 2024-02-01T10:48:49.616 DEBUG:boto:auth_path=/vclfcn-62/obj2 2024-02-01T10:48:49.617 DEBUG:boto:Method: GET 2024-02-01T10:48:49.617 DEBUG:boto:Path: /vclfcn-62/obj2 2024-02-01T10:48:49.617 DEBUG:boto:Data: 2024-02-01T10:48:49.617 DEBUG:boto:Headers: {} 2024-02-01T10:48:49.617 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:49.617 DEBUG:boto:Port: 8000 2024-02-01T10:48:49.617 DEBUG:boto:Params: {} 2024-02-01T10:48:49.618 DEBUG:boto:Token: None 2024-02-01T10:48:49.618 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:49 GMT /vclfcn-62/obj2 2024-02-01T10:48:49.618 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:61Ste9viqHIScFWFeEXm9IVj7zU= 2024-02-01T10:48:49.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:61Ste9viqHIScFWFeEXm9IVj7zU=', 'Content-Length': '0'} 2024-02-01T10:48:49.621 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000afa5b7e69304dd7c-0065bb7711-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:49.622 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:48:49.622 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:48:49.622 DEBUG:boto:path=/vclfcn-62/?prefix=obj2 2024-02-01T10:48:49.622 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj2 2024-02-01T10:48:49.622 DEBUG:boto:Method: GET 2024-02-01T10:48:49.622 DEBUG:boto:Path: /vclfcn-62/?prefix=obj2 2024-02-01T10:48:49.622 DEBUG:boto:Data: 2024-02-01T10:48:49.622 DEBUG:boto:Headers: {} 2024-02-01T10:48:49.622 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:49.622 DEBUG:boto:Port: 8001 2024-02-01T10:48:49.623 DEBUG:boto:Params: {} 2024-02-01T10:48:49.623 DEBUG:boto:Token: None 2024-02-01T10:48:49.623 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:49 GMT /vclfcn-62/ 2024-02-01T10:48:49.623 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5o2GaAhuOVfi8KdCqOyMNa3u4m4= 2024-02-01T10:48:49.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5o2GaAhuOVfi8KdCqOyMNa3u4m4=', 'Content-Length': '0'} 2024-02-01T10:48:49.627 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024c9fd957f2776bf-0065bb7711-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:49.628 DEBUG:boto:b'vclfcn-62obj21000falseobj22024-02-01T10:48:26.333Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:49.628 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:48:49.628 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:48:49.629 DEBUG:boto:path=/vclfcn-62/?prefix=obj2 2024-02-01T10:48:49.629 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj2 2024-02-01T10:48:49.630 DEBUG:boto:Method: GET 2024-02-01T10:48:49.630 DEBUG:boto:Path: /vclfcn-62/?prefix=obj2 2024-02-01T10:48:49.630 DEBUG:boto:Data: 2024-02-01T10:48:49.630 DEBUG:boto:Headers: {} 2024-02-01T10:48:49.630 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:49.630 DEBUG:boto:Port: 8000 2024-02-01T10:48:49.630 DEBUG:boto:Params: {} 2024-02-01T10:48:49.630 DEBUG:boto:Token: None 2024-02-01T10:48:49.630 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:49 GMT /vclfcn-62/ 2024-02-01T10:48:49.630 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5o2GaAhuOVfi8KdCqOyMNa3u4m4= 2024-02-01T10:48:49.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5o2GaAhuOVfi8KdCqOyMNa3u4m4=', 'Content-Length': '0'} 2024-02-01T10:48:49.635 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4862abd76aaad59-0065bb7711-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:49.635 DEBUG:boto:b'vclfcn-62obj21000falseobj22024-02-01T10:48:26.333Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:49.635 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:48:49.636 DEBUG:boto:path=/vclfcn-62/obj3 2024-02-01T10:48:49.636 DEBUG:boto:auth_path=/vclfcn-62/obj3 2024-02-01T10:48:49.636 DEBUG:boto:path=/vclfcn-62/obj3?versionId=null 2024-02-01T10:48:49.636 DEBUG:boto:auth_path=/vclfcn-62/obj3?versionId=null 2024-02-01T10:48:49.636 DEBUG:boto:Method: GET 2024-02-01T10:48:49.636 DEBUG:boto:Path: /vclfcn-62/obj3?versionId=null 2024-02-01T10:48:49.636 DEBUG:boto:Data: 2024-02-01T10:48:49.636 DEBUG:boto:Headers: {} 2024-02-01T10:48:49.636 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:49.636 DEBUG:boto:Port: 8001 2024-02-01T10:48:49.637 DEBUG:boto:Params: {} 2024-02-01T10:48:49.637 DEBUG:boto:Token: None 2024-02-01T10:48:49.637 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:49 GMT /vclfcn-62/obj3?versionId=null 2024-02-01T10:48:49.637 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:MSAblj0GsHNJyP5kDoMh2Iyt0CA= 2024-02-01T10:48:49.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:MSAblj0GsHNJyP5kDoMh2Iyt0CA=', 'Content-Length': '0'} 2024-02-01T10:48:49.640 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009df43cdda3c19fbc-0065bb7711-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:49.640 DEBUG:boto:path=/vclfcn-62/obj3 2024-02-01T10:48:49.640 DEBUG:boto:auth_path=/vclfcn-62/obj3 2024-02-01T10:48:49.640 DEBUG:boto:path=/vclfcn-62/obj3?versionId=null 2024-02-01T10:48:49.641 DEBUG:boto:auth_path=/vclfcn-62/obj3?versionId=null 2024-02-01T10:48:49.641 DEBUG:boto:Method: GET 2024-02-01T10:48:49.641 DEBUG:boto:Path: /vclfcn-62/obj3?versionId=null 2024-02-01T10:48:49.641 DEBUG:boto:Data: 2024-02-01T10:48:49.641 DEBUG:boto:Headers: {} 2024-02-01T10:48:49.641 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:49.641 DEBUG:boto:Port: 8000 2024-02-01T10:48:49.641 DEBUG:boto:Params: {} 2024-02-01T10:48:49.641 DEBUG:boto:Token: None 2024-02-01T10:48:49.641 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:49 GMT /vclfcn-62/obj3?versionId=null 2024-02-01T10:48:49.642 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:MSAblj0GsHNJyP5kDoMh2Iyt0CA= 2024-02-01T10:48:49.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:MSAblj0GsHNJyP5kDoMh2Iyt0CA=', 'Content-Length': '0'} 2024-02-01T10:48:49.644 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d1f46431bfedf44d-0065bb7711-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:49.688 DEBUG:boto:path=/vclfcn-62/obj3 2024-02-01T10:48:49.688 DEBUG:boto:auth_path=/vclfcn-62/obj3 2024-02-01T10:48:49.688 DEBUG:boto:path=/vclfcn-62/obj3?versionId=null 2024-02-01T10:48:49.688 DEBUG:boto:auth_path=/vclfcn-62/obj3?versionId=null 2024-02-01T10:48:49.689 DEBUG:boto:Method: HEAD 2024-02-01T10:48:49.689 DEBUG:boto:Path: /vclfcn-62/obj3?versionId=null 2024-02-01T10:48:49.689 DEBUG:boto:Data: 2024-02-01T10:48:49.689 DEBUG:boto:Headers: {} 2024-02-01T10:48:49.689 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:49.689 DEBUG:boto:Port: 8001 2024-02-01T10:48:49.690 DEBUG:boto:Params: {} 2024-02-01T10:48:49.690 DEBUG:boto:Token: None 2024-02-01T10:48:49.690 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:49 GMT /vclfcn-62/obj3?versionId=null 2024-02-01T10:48:49.690 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mTg3z0KSxNAwIismBDXUs5+zCm0= 2024-02-01T10:48:49.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mTg3z0KSxNAwIismBDXUs5+zCm0=', 'Content-Length': '0'} 2024-02-01T10:48:49.693 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000949cca0430b516c8-0065bb7711-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:49.693 DEBUG:boto:path=/vclfcn-62/obj3 2024-02-01T10:48:49.694 DEBUG:boto:auth_path=/vclfcn-62/obj3 2024-02-01T10:48:49.694 DEBUG:boto:path=/vclfcn-62/obj3?versionId=null 2024-02-01T10:48:49.694 DEBUG:boto:auth_path=/vclfcn-62/obj3?versionId=null 2024-02-01T10:48:49.694 DEBUG:boto:Method: HEAD 2024-02-01T10:48:49.694 DEBUG:boto:Path: /vclfcn-62/obj3?versionId=null 2024-02-01T10:48:49.694 DEBUG:boto:Data: 2024-02-01T10:48:49.694 DEBUG:boto:Headers: {} 2024-02-01T10:48:49.694 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:49.694 DEBUG:boto:Port: 8000 2024-02-01T10:48:49.694 DEBUG:boto:Params: {} 2024-02-01T10:48:49.695 DEBUG:boto:Token: None 2024-02-01T10:48:49.695 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:49 GMT /vclfcn-62/obj3?versionId=null 2024-02-01T10:48:49.695 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mTg3z0KSxNAwIismBDXUs5+zCm0= 2024-02-01T10:48:49.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mTg3z0KSxNAwIismBDXUs5+zCm0=', 'Content-Length': '0'} 2024-02-01T10:48:49.698 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f29d35e8940b59b3-0065bb7711-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:49.698 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:48:49.698 DEBUG:boto:path=/vclfcn-62/obj3 2024-02-01T10:48:49.698 DEBUG:boto:auth_path=/vclfcn-62/obj3 2024-02-01T10:48:49.698 DEBUG:boto:path=/vclfcn-62/obj3?versionId=null 2024-02-01T10:48:49.699 DEBUG:boto:auth_path=/vclfcn-62/obj3?versionId=null 2024-02-01T10:48:49.699 DEBUG:boto:Method: GET 2024-02-01T10:48:49.699 DEBUG:boto:Path: /vclfcn-62/obj3?versionId=null 2024-02-01T10:48:49.699 DEBUG:boto:Data: 2024-02-01T10:48:49.699 DEBUG:boto:Headers: {} 2024-02-01T10:48:49.699 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:49.699 DEBUG:boto:Port: 8001 2024-02-01T10:48:49.699 DEBUG:boto:Params: {} 2024-02-01T10:48:49.699 DEBUG:boto:Token: None 2024-02-01T10:48:49.699 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:49 GMT /vclfcn-62/obj3?versionId=null 2024-02-01T10:48:49.700 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:MSAblj0GsHNJyP5kDoMh2Iyt0CA= 2024-02-01T10:48:49.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:MSAblj0GsHNJyP5kDoMh2Iyt0CA=', 'Content-Length': '0'} 2024-02-01T10:48:49.703 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f2c0f2bb67ffe7d6-0065bb7711-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:49.748 DEBUG:boto:path=/vclfcn-62/obj3 2024-02-01T10:48:49.748 DEBUG:boto:auth_path=/vclfcn-62/obj3 2024-02-01T10:48:49.748 DEBUG:boto:path=/vclfcn-62/obj3?versionId=null 2024-02-01T10:48:49.748 DEBUG:boto:auth_path=/vclfcn-62/obj3?versionId=null 2024-02-01T10:48:49.748 DEBUG:boto:Method: GET 2024-02-01T10:48:49.749 DEBUG:boto:Path: /vclfcn-62/obj3?versionId=null 2024-02-01T10:48:49.749 DEBUG:boto:Data: 2024-02-01T10:48:49.749 DEBUG:boto:Headers: {} 2024-02-01T10:48:49.749 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:49.749 DEBUG:boto:Port: 8000 2024-02-01T10:48:49.749 DEBUG:boto:Params: {} 2024-02-01T10:48:49.749 DEBUG:boto:Token: None 2024-02-01T10:48:49.749 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:49 GMT /vclfcn-62/obj3?versionId=null 2024-02-01T10:48:49.749 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:MSAblj0GsHNJyP5kDoMh2Iyt0CA= 2024-02-01T10:48:49.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:MSAblj0GsHNJyP5kDoMh2Iyt0CA=', 'Content-Length': '0'} 2024-02-01T10:48:49.752 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000026b6d84dc18ad9e8-0065bb7711-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:49.753 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:48:49.753 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:48:49.753 DEBUG:boto:path=/vclfcn-62/?prefix=obj3 2024-02-01T10:48:49.753 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj3 2024-02-01T10:48:49.753 DEBUG:boto:Method: GET 2024-02-01T10:48:49.753 DEBUG:boto:Path: /vclfcn-62/?prefix=obj3 2024-02-01T10:48:49.753 DEBUG:boto:Data: 2024-02-01T10:48:49.753 DEBUG:boto:Headers: {} 2024-02-01T10:48:49.753 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:49.753 DEBUG:boto:Port: 8001 2024-02-01T10:48:49.754 DEBUG:boto:Params: {} 2024-02-01T10:48:49.754 DEBUG:boto:Token: None 2024-02-01T10:48:49.754 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:49 GMT /vclfcn-62/ 2024-02-01T10:48:49.754 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5o2GaAhuOVfi8KdCqOyMNa3u4m4= 2024-02-01T10:48:49.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5o2GaAhuOVfi8KdCqOyMNa3u4m4=', 'Content-Length': '0'} 2024-02-01T10:48:49.758 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2a55f150378a3a0-0065bb7711-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:49.759 DEBUG:boto:b'vclfcn-62obj31000falseobj32024-02-01T10:48:26.384Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:49.759 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:48:49.759 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:48:49.759 DEBUG:boto:path=/vclfcn-62/?prefix=obj3 2024-02-01T10:48:49.759 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj3 2024-02-01T10:48:49.759 DEBUG:boto:Method: GET 2024-02-01T10:48:49.759 DEBUG:boto:Path: /vclfcn-62/?prefix=obj3 2024-02-01T10:48:49.760 DEBUG:boto:Data: 2024-02-01T10:48:49.760 DEBUG:boto:Headers: {} 2024-02-01T10:48:49.760 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:49.760 DEBUG:boto:Port: 8000 2024-02-01T10:48:49.760 DEBUG:boto:Params: {} 2024-02-01T10:48:49.760 DEBUG:boto:Token: None 2024-02-01T10:48:49.760 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:49 GMT /vclfcn-62/ 2024-02-01T10:48:49.760 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5o2GaAhuOVfi8KdCqOyMNa3u4m4= 2024-02-01T10:48:49.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5o2GaAhuOVfi8KdCqOyMNa3u4m4=', 'Content-Length': '0'} 2024-02-01T10:48:49.766 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000838c606a5bec6f1d-0065bb7711-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:49.767 DEBUG:boto:b'vclfcn-62obj31000falseobj32024-02-01T10:48:26.384Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:49.767 DEBUG:boto:path=/vclfcn-62/obj3 2024-02-01T10:48:49.767 DEBUG:boto:auth_path=/vclfcn-62/obj3 2024-02-01T10:48:49.768 DEBUG:boto:Method: HEAD 2024-02-01T10:48:49.768 DEBUG:boto:Path: /vclfcn-62/obj3 2024-02-01T10:48:49.768 DEBUG:boto:Data: 2024-02-01T10:48:49.768 DEBUG:boto:Headers: {} 2024-02-01T10:48:49.768 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:49.768 DEBUG:boto:Port: 8001 2024-02-01T10:48:49.768 DEBUG:boto:Params: {} 2024-02-01T10:48:49.768 DEBUG:boto:Token: None 2024-02-01T10:48:49.768 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:49 GMT /vclfcn-62/obj3 2024-02-01T10:48:49.769 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Y4QeBpol07IJVC/gaOmmRjq/wys= 2024-02-01T10:48:49.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Y4QeBpol07IJVC/gaOmmRjq/wys=', 'Content-Length': '0'} 2024-02-01T10:48:49.771 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000e81b0b18ee83856-0065bb7711-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:49.771 DEBUG:boto:path=/vclfcn-62/obj3 2024-02-01T10:48:49.771 DEBUG:boto:auth_path=/vclfcn-62/obj3 2024-02-01T10:48:49.772 DEBUG:boto:Method: HEAD 2024-02-01T10:48:49.772 DEBUG:boto:Path: /vclfcn-62/obj3 2024-02-01T10:48:49.772 DEBUG:boto:Data: 2024-02-01T10:48:49.772 DEBUG:boto:Headers: {} 2024-02-01T10:48:49.772 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:49.772 DEBUG:boto:Port: 8000 2024-02-01T10:48:49.772 DEBUG:boto:Params: {} 2024-02-01T10:48:49.772 DEBUG:boto:Token: None 2024-02-01T10:48:49.772 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:49 GMT /vclfcn-62/obj3 2024-02-01T10:48:49.773 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Y4QeBpol07IJVC/gaOmmRjq/wys= 2024-02-01T10:48:49.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Y4QeBpol07IJVC/gaOmmRjq/wys=', 'Content-Length': '0'} 2024-02-01T10:48:49.775 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001fd9fbb3c0d0405a-0065bb7711-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:49.775 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:48:49.775 DEBUG:boto:path=/vclfcn-62/obj3 2024-02-01T10:48:49.776 DEBUG:boto:auth_path=/vclfcn-62/obj3 2024-02-01T10:48:49.776 DEBUG:boto:Method: GET 2024-02-01T10:48:49.776 DEBUG:boto:Path: /vclfcn-62/obj3 2024-02-01T10:48:49.776 DEBUG:boto:Data: 2024-02-01T10:48:49.776 DEBUG:boto:Headers: {} 2024-02-01T10:48:49.776 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:49.776 DEBUG:boto:Port: 8001 2024-02-01T10:48:49.776 DEBUG:boto:Params: {} 2024-02-01T10:48:49.776 DEBUG:boto:Token: None 2024-02-01T10:48:49.776 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:49 GMT /vclfcn-62/obj3 2024-02-01T10:48:49.777 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:DfJtvO+ZbnbhhJHhnMK8Fefq0o4= 2024-02-01T10:48:49.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:DfJtvO+ZbnbhhJHhnMK8Fefq0o4=', 'Content-Length': '0'} 2024-02-01T10:48:49.779 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a2e469f478417273-0065bb7711-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:49.820 DEBUG:boto:path=/vclfcn-62/obj3 2024-02-01T10:48:49.820 DEBUG:boto:auth_path=/vclfcn-62/obj3 2024-02-01T10:48:49.820 DEBUG:boto:Method: GET 2024-02-01T10:48:49.820 DEBUG:boto:Path: /vclfcn-62/obj3 2024-02-01T10:48:49.821 DEBUG:boto:Data: 2024-02-01T10:48:49.821 DEBUG:boto:Headers: {} 2024-02-01T10:48:49.821 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:49.821 DEBUG:boto:Port: 8000 2024-02-01T10:48:49.821 DEBUG:boto:Params: {} 2024-02-01T10:48:49.822 DEBUG:boto:Token: None 2024-02-01T10:48:49.822 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:49 GMT /vclfcn-62/obj3 2024-02-01T10:48:49.822 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:DfJtvO+ZbnbhhJHhnMK8Fefq0o4= 2024-02-01T10:48:49.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:DfJtvO+ZbnbhhJHhnMK8Fefq0o4=', 'Content-Length': '0'} 2024-02-01T10:48:49.825 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000faa9f47d6d40b807-0065bb7711-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:49.825 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:48:49.826 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:48:49.826 DEBUG:boto:path=/vclfcn-62/?prefix=obj3 2024-02-01T10:48:49.826 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj3 2024-02-01T10:48:49.826 DEBUG:boto:Method: GET 2024-02-01T10:48:49.826 DEBUG:boto:Path: /vclfcn-62/?prefix=obj3 2024-02-01T10:48:49.826 DEBUG:boto:Data: 2024-02-01T10:48:49.826 DEBUG:boto:Headers: {} 2024-02-01T10:48:49.826 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:49.827 DEBUG:boto:Port: 8001 2024-02-01T10:48:49.827 DEBUG:boto:Params: {} 2024-02-01T10:48:49.827 DEBUG:boto:Token: None 2024-02-01T10:48:49.827 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:49 GMT /vclfcn-62/ 2024-02-01T10:48:49.827 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5o2GaAhuOVfi8KdCqOyMNa3u4m4= 2024-02-01T10:48:49.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5o2GaAhuOVfi8KdCqOyMNa3u4m4=', 'Content-Length': '0'} 2024-02-01T10:48:49.831 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b70f597b453cd929-0065bb7711-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:49.876 DEBUG:boto:b'vclfcn-62obj31000falseobj32024-02-01T10:48:26.384Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:49.877 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:48:49.877 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:48:49.877 DEBUG:boto:path=/vclfcn-62/?prefix=obj3 2024-02-01T10:48:49.877 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj3 2024-02-01T10:48:49.877 DEBUG:boto:Method: GET 2024-02-01T10:48:49.877 DEBUG:boto:Path: /vclfcn-62/?prefix=obj3 2024-02-01T10:48:49.878 DEBUG:boto:Data: 2024-02-01T10:48:49.878 DEBUG:boto:Headers: {} 2024-02-01T10:48:49.878 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:49.878 DEBUG:boto:Port: 8000 2024-02-01T10:48:49.878 DEBUG:boto:Params: {} 2024-02-01T10:48:49.878 DEBUG:boto:Token: None 2024-02-01T10:48:49.878 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:49 GMT /vclfcn-62/ 2024-02-01T10:48:49.878 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5o2GaAhuOVfi8KdCqOyMNa3u4m4= 2024-02-01T10:48:49.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5o2GaAhuOVfi8KdCqOyMNa3u4m4=', 'Content-Length': '0'} 2024-02-01T10:48:49.883 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f467b881b97f748-0065bb7711-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:49.883 DEBUG:boto:b'vclfcn-62obj31000falseobj32024-02-01T10:48:26.384Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:49.884 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:48:49.884 DEBUG:boto:path=/vclfcn-62/obj4 2024-02-01T10:48:49.884 DEBUG:boto:auth_path=/vclfcn-62/obj4 2024-02-01T10:48:49.884 DEBUG:boto:path=/vclfcn-62/obj4?versionId=null 2024-02-01T10:48:49.884 DEBUG:boto:auth_path=/vclfcn-62/obj4?versionId=null 2024-02-01T10:48:49.884 DEBUG:boto:Method: GET 2024-02-01T10:48:49.884 DEBUG:boto:Path: /vclfcn-62/obj4?versionId=null 2024-02-01T10:48:49.885 DEBUG:boto:Data: 2024-02-01T10:48:49.885 DEBUG:boto:Headers: {} 2024-02-01T10:48:49.885 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:49.885 DEBUG:boto:Port: 8001 2024-02-01T10:48:49.885 DEBUG:boto:Params: {} 2024-02-01T10:48:49.885 DEBUG:boto:Token: None 2024-02-01T10:48:49.885 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:49 GMT /vclfcn-62/obj4?versionId=null 2024-02-01T10:48:49.885 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ddeD1JcvtiX1hwP1fA3Xg4ibx5M= 2024-02-01T10:48:49.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ddeD1JcvtiX1hwP1fA3Xg4ibx5M=', 'Content-Length': '0'} 2024-02-01T10:48:49.888 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ca33dbe9fc89af89-0065bb7711-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:49.889 DEBUG:boto:path=/vclfcn-62/obj4 2024-02-01T10:48:49.889 DEBUG:boto:auth_path=/vclfcn-62/obj4 2024-02-01T10:48:49.889 DEBUG:boto:path=/vclfcn-62/obj4?versionId=null 2024-02-01T10:48:49.889 DEBUG:boto:auth_path=/vclfcn-62/obj4?versionId=null 2024-02-01T10:48:49.889 DEBUG:boto:Method: GET 2024-02-01T10:48:49.889 DEBUG:boto:Path: /vclfcn-62/obj4?versionId=null 2024-02-01T10:48:49.889 DEBUG:boto:Data: 2024-02-01T10:48:49.889 DEBUG:boto:Headers: {} 2024-02-01T10:48:49.889 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:49.889 DEBUG:boto:Port: 8000 2024-02-01T10:48:49.890 DEBUG:boto:Params: {} 2024-02-01T10:48:49.890 DEBUG:boto:Token: None 2024-02-01T10:48:49.890 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:49 GMT /vclfcn-62/obj4?versionId=null 2024-02-01T10:48:49.890 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ddeD1JcvtiX1hwP1fA3Xg4ibx5M= 2024-02-01T10:48:49.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ddeD1JcvtiX1hwP1fA3Xg4ibx5M=', 'Content-Length': '0'} 2024-02-01T10:48:49.893 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cb2d0cdfc89f77ff-0065bb7711-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:49.894 DEBUG:boto:path=/vclfcn-62/obj4 2024-02-01T10:48:49.894 DEBUG:boto:auth_path=/vclfcn-62/obj4 2024-02-01T10:48:49.894 DEBUG:boto:path=/vclfcn-62/obj4?versionId=null 2024-02-01T10:48:49.894 DEBUG:boto:auth_path=/vclfcn-62/obj4?versionId=null 2024-02-01T10:48:49.894 DEBUG:boto:Method: HEAD 2024-02-01T10:48:49.894 DEBUG:boto:Path: /vclfcn-62/obj4?versionId=null 2024-02-01T10:48:49.894 DEBUG:boto:Data: 2024-02-01T10:48:49.894 DEBUG:boto:Headers: {} 2024-02-01T10:48:49.895 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:49.895 DEBUG:boto:Port: 8001 2024-02-01T10:48:49.895 DEBUG:boto:Params: {} 2024-02-01T10:48:49.895 DEBUG:boto:Token: None 2024-02-01T10:48:49.895 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:49 GMT /vclfcn-62/obj4?versionId=null 2024-02-01T10:48:49.895 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0srbVUy90M9Bf5YX426824YA16E= 2024-02-01T10:48:49.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0srbVUy90M9Bf5YX426824YA16E=', 'Content-Length': '0'} 2024-02-01T10:48:49.898 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000018cafe9b8af78687-0065bb7711-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:49.898 DEBUG:boto:path=/vclfcn-62/obj4 2024-02-01T10:48:49.898 DEBUG:boto:auth_path=/vclfcn-62/obj4 2024-02-01T10:48:49.898 DEBUG:boto:path=/vclfcn-62/obj4?versionId=null 2024-02-01T10:48:49.899 DEBUG:boto:auth_path=/vclfcn-62/obj4?versionId=null 2024-02-01T10:48:49.899 DEBUG:boto:Method: HEAD 2024-02-01T10:48:49.899 DEBUG:boto:Path: /vclfcn-62/obj4?versionId=null 2024-02-01T10:48:49.899 DEBUG:boto:Data: 2024-02-01T10:48:49.899 DEBUG:boto:Headers: {} 2024-02-01T10:48:49.899 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:49.899 DEBUG:boto:Port: 8000 2024-02-01T10:48:49.899 DEBUG:boto:Params: {} 2024-02-01T10:48:49.899 DEBUG:boto:Token: None 2024-02-01T10:48:49.899 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:49 GMT /vclfcn-62/obj4?versionId=null 2024-02-01T10:48:49.900 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0srbVUy90M9Bf5YX426824YA16E= 2024-02-01T10:48:49.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0srbVUy90M9Bf5YX426824YA16E=', 'Content-Length': '0'} 2024-02-01T10:48:49.902 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000410cc8915855763b-0065bb7711-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:49.903 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:48:49.903 DEBUG:boto:path=/vclfcn-62/obj4 2024-02-01T10:48:49.903 DEBUG:boto:auth_path=/vclfcn-62/obj4 2024-02-01T10:48:49.903 DEBUG:boto:path=/vclfcn-62/obj4?versionId=null 2024-02-01T10:48:49.903 DEBUG:boto:auth_path=/vclfcn-62/obj4?versionId=null 2024-02-01T10:48:49.903 DEBUG:boto:Method: GET 2024-02-01T10:48:49.903 DEBUG:boto:Path: /vclfcn-62/obj4?versionId=null 2024-02-01T10:48:49.903 DEBUG:boto:Data: 2024-02-01T10:48:49.903 DEBUG:boto:Headers: {} 2024-02-01T10:48:49.903 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:49.904 DEBUG:boto:Port: 8001 2024-02-01T10:48:49.904 DEBUG:boto:Params: {} 2024-02-01T10:48:49.904 DEBUG:boto:Token: None 2024-02-01T10:48:49.904 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:49 GMT /vclfcn-62/obj4?versionId=null 2024-02-01T10:48:49.904 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ddeD1JcvtiX1hwP1fA3Xg4ibx5M= 2024-02-01T10:48:49.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ddeD1JcvtiX1hwP1fA3Xg4ibx5M=', 'Content-Length': '0'} 2024-02-01T10:48:49.907 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000df1da82baba49611-0065bb7711-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:49.952 DEBUG:boto:path=/vclfcn-62/obj4 2024-02-01T10:48:49.952 DEBUG:boto:auth_path=/vclfcn-62/obj4 2024-02-01T10:48:49.952 DEBUG:boto:path=/vclfcn-62/obj4?versionId=null 2024-02-01T10:48:49.952 DEBUG:boto:auth_path=/vclfcn-62/obj4?versionId=null 2024-02-01T10:48:49.953 DEBUG:boto:Method: GET 2024-02-01T10:48:49.953 DEBUG:boto:Path: /vclfcn-62/obj4?versionId=null 2024-02-01T10:48:49.953 DEBUG:boto:Data: 2024-02-01T10:48:49.953 DEBUG:boto:Headers: {} 2024-02-01T10:48:49.953 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:49.953 DEBUG:boto:Port: 8000 2024-02-01T10:48:49.953 DEBUG:boto:Params: {} 2024-02-01T10:48:49.953 DEBUG:boto:Token: None 2024-02-01T10:48:49.953 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:49 GMT /vclfcn-62/obj4?versionId=null 2024-02-01T10:48:49.954 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ddeD1JcvtiX1hwP1fA3Xg4ibx5M= 2024-02-01T10:48:49.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ddeD1JcvtiX1hwP1fA3Xg4ibx5M=', 'Content-Length': '0'} 2024-02-01T10:48:49.957 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009dc6a13cbfef4003-0065bb7711-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:49.957 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:48:49.958 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:48:49.958 DEBUG:boto:path=/vclfcn-62/?prefix=obj4 2024-02-01T10:48:49.958 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj4 2024-02-01T10:48:49.958 DEBUG:boto:Method: GET 2024-02-01T10:48:49.958 DEBUG:boto:Path: /vclfcn-62/?prefix=obj4 2024-02-01T10:48:49.958 DEBUG:boto:Data: 2024-02-01T10:48:49.958 DEBUG:boto:Headers: {} 2024-02-01T10:48:49.958 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:49.959 DEBUG:boto:Port: 8001 2024-02-01T10:48:49.959 DEBUG:boto:Params: {} 2024-02-01T10:48:49.959 DEBUG:boto:Token: None 2024-02-01T10:48:49.959 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:49 GMT /vclfcn-62/ 2024-02-01T10:48:49.959 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5o2GaAhuOVfi8KdCqOyMNa3u4m4= 2024-02-01T10:48:49.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5o2GaAhuOVfi8KdCqOyMNa3u4m4=', 'Content-Length': '0'} 2024-02-01T10:48:49.963 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ef525c64531cfdf-0065bb7711-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:49.964 DEBUG:boto:b'vclfcn-62obj41000falseobj42024-02-01T10:48:26.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:49.964 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:48:49.964 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:48:49.964 DEBUG:boto:path=/vclfcn-62/?prefix=obj4 2024-02-01T10:48:49.965 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj4 2024-02-01T10:48:49.965 DEBUG:boto:Method: GET 2024-02-01T10:48:49.965 DEBUG:boto:Path: /vclfcn-62/?prefix=obj4 2024-02-01T10:48:49.965 DEBUG:boto:Data: 2024-02-01T10:48:49.965 DEBUG:boto:Headers: {} 2024-02-01T10:48:49.965 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:49.965 DEBUG:boto:Port: 8000 2024-02-01T10:48:49.965 DEBUG:boto:Params: {} 2024-02-01T10:48:49.965 DEBUG:boto:Token: None 2024-02-01T10:48:49.965 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:49 GMT /vclfcn-62/ 2024-02-01T10:48:49.966 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5o2GaAhuOVfi8KdCqOyMNa3u4m4= 2024-02-01T10:48:49.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:49 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5o2GaAhuOVfi8KdCqOyMNa3u4m4=', 'Content-Length': '0'} 2024-02-01T10:48:49.970 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000108a8ecdaf20c284-0065bb7711-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:50.012 DEBUG:boto:b'vclfcn-62obj41000falseobj42024-02-01T10:48:26.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:50.013 DEBUG:boto:path=/vclfcn-62/obj4 2024-02-01T10:48:50.013 DEBUG:boto:auth_path=/vclfcn-62/obj4 2024-02-01T10:48:50.013 DEBUG:boto:Method: HEAD 2024-02-01T10:48:50.013 DEBUG:boto:Path: /vclfcn-62/obj4 2024-02-01T10:48:50.014 DEBUG:boto:Data: 2024-02-01T10:48:50.014 DEBUG:boto:Headers: {} 2024-02-01T10:48:50.014 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:50.014 DEBUG:boto:Port: 8001 2024-02-01T10:48:50.014 DEBUG:boto:Params: {} 2024-02-01T10:48:50.014 DEBUG:boto:Token: None 2024-02-01T10:48:50.015 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:50 GMT /vclfcn-62/obj4 2024-02-01T10:48:50.015 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Jq1jNLy6PomlMxZMuQojrPHF1DQ= 2024-02-01T10:48:50.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Jq1jNLy6PomlMxZMuQojrPHF1DQ=', 'Content-Length': '0'} 2024-02-01T10:48:50.018 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005ee0a954639897d7-0065bb7712-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:50.018 DEBUG:boto:path=/vclfcn-62/obj4 2024-02-01T10:48:50.018 DEBUG:boto:auth_path=/vclfcn-62/obj4 2024-02-01T10:48:50.018 DEBUG:boto:Method: HEAD 2024-02-01T10:48:50.018 DEBUG:boto:Path: /vclfcn-62/obj4 2024-02-01T10:48:50.019 DEBUG:boto:Data: 2024-02-01T10:48:50.019 DEBUG:boto:Headers: {} 2024-02-01T10:48:50.019 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:50.019 DEBUG:boto:Port: 8000 2024-02-01T10:48:50.019 DEBUG:boto:Params: {} 2024-02-01T10:48:50.019 DEBUG:boto:Token: None 2024-02-01T10:48:50.019 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:50 GMT /vclfcn-62/obj4 2024-02-01T10:48:50.019 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Jq1jNLy6PomlMxZMuQojrPHF1DQ= 2024-02-01T10:48:50.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Jq1jNLy6PomlMxZMuQojrPHF1DQ=', 'Content-Length': '0'} 2024-02-01T10:48:50.023 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008ea366acd8c892dc-0065bb7712-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:50.024 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:48:50.024 DEBUG:boto:path=/vclfcn-62/obj4 2024-02-01T10:48:50.024 DEBUG:boto:auth_path=/vclfcn-62/obj4 2024-02-01T10:48:50.024 DEBUG:boto:Method: GET 2024-02-01T10:48:50.024 DEBUG:boto:Path: /vclfcn-62/obj4 2024-02-01T10:48:50.024 DEBUG:boto:Data: 2024-02-01T10:48:50.024 DEBUG:boto:Headers: {} 2024-02-01T10:48:50.024 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:50.025 DEBUG:boto:Port: 8001 2024-02-01T10:48:50.025 DEBUG:boto:Params: {} 2024-02-01T10:48:50.025 DEBUG:boto:Token: None 2024-02-01T10:48:50.025 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:50 GMT /vclfcn-62/obj4 2024-02-01T10:48:50.025 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gvWsAKfWbq4zyIOt/rObybuHRGg= 2024-02-01T10:48:50.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gvWsAKfWbq4zyIOt/rObybuHRGg=', 'Content-Length': '0'} 2024-02-01T10:48:50.028 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000338bd50af470ee59-0065bb7712-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:50.072 DEBUG:boto:path=/vclfcn-62/obj4 2024-02-01T10:48:50.072 DEBUG:boto:auth_path=/vclfcn-62/obj4 2024-02-01T10:48:50.072 DEBUG:boto:Method: GET 2024-02-01T10:48:50.072 DEBUG:boto:Path: /vclfcn-62/obj4 2024-02-01T10:48:50.073 DEBUG:boto:Data: 2024-02-01T10:48:50.073 DEBUG:boto:Headers: {} 2024-02-01T10:48:50.073 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:50.073 DEBUG:boto:Port: 8000 2024-02-01T10:48:50.073 DEBUG:boto:Params: {} 2024-02-01T10:48:50.073 DEBUG:boto:Token: None 2024-02-01T10:48:50.073 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:50 GMT /vclfcn-62/obj4 2024-02-01T10:48:50.073 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gvWsAKfWbq4zyIOt/rObybuHRGg= 2024-02-01T10:48:50.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gvWsAKfWbq4zyIOt/rObybuHRGg=', 'Content-Length': '0'} 2024-02-01T10:48:50.076 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cb8c88ba409a150a-0065bb7712-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:50.077 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:48:50.077 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:48:50.077 DEBUG:boto:path=/vclfcn-62/?prefix=obj4 2024-02-01T10:48:50.077 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj4 2024-02-01T10:48:50.077 DEBUG:boto:Method: GET 2024-02-01T10:48:50.077 DEBUG:boto:Path: /vclfcn-62/?prefix=obj4 2024-02-01T10:48:50.077 DEBUG:boto:Data: 2024-02-01T10:48:50.078 DEBUG:boto:Headers: {} 2024-02-01T10:48:50.078 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:50.078 DEBUG:boto:Port: 8001 2024-02-01T10:48:50.078 DEBUG:boto:Params: {} 2024-02-01T10:48:50.078 DEBUG:boto:Token: None 2024-02-01T10:48:50.078 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:50 GMT /vclfcn-62/ 2024-02-01T10:48:50.078 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:pJ6p/CfmqNl3fCHPPxzKZlU2Rjs= 2024-02-01T10:48:50.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:pJ6p/CfmqNl3fCHPPxzKZlU2Rjs=', 'Content-Length': '0'} 2024-02-01T10:48:50.082 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b9471965708f1313-0065bb7712-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:50.124 DEBUG:boto:b'vclfcn-62obj41000falseobj42024-02-01T10:48:26.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:50.124 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:48:50.125 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:48:50.125 DEBUG:boto:path=/vclfcn-62/?prefix=obj4 2024-02-01T10:48:50.125 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj4 2024-02-01T10:48:50.125 DEBUG:boto:Method: GET 2024-02-01T10:48:50.125 DEBUG:boto:Path: /vclfcn-62/?prefix=obj4 2024-02-01T10:48:50.125 DEBUG:boto:Data: 2024-02-01T10:48:50.125 DEBUG:boto:Headers: {} 2024-02-01T10:48:50.125 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:48:50.125 DEBUG:boto:Port: 8000 2024-02-01T10:48:50.125 DEBUG:boto:Params: {} 2024-02-01T10:48:50.126 DEBUG:boto:Token: None 2024-02-01T10:48:50.126 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:50 GMT /vclfcn-62/ 2024-02-01T10:48:50.126 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:pJ6p/CfmqNl3fCHPPxzKZlU2Rjs= 2024-02-01T10:48:50.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:pJ6p/CfmqNl3fCHPPxzKZlU2Rjs=', 'Content-Length': '0'} 2024-02-01T10:48:50.130 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001cb4368a0400c0fb-0065bb7712-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:50.130 DEBUG:boto:b'vclfcn-62obj41000falseobj42024-02-01T10:48:26.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:50.131 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-62 zones={test-zone3, test-zone1} 2024-02-01T10:48:50.131 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-62 zones={test-zone3, test-zone2} 2024-02-01T10:48:50.131 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:48:50.131 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:48:50.131 DEBUG:boto:Method: HEAD 2024-02-01T10:48:50.131 DEBUG:boto:Path: /vclfcn-62/ 2024-02-01T10:48:50.132 DEBUG:boto:Data: 2024-02-01T10:48:50.132 DEBUG:boto:Headers: {} 2024-02-01T10:48:50.132 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:50.132 DEBUG:boto:Port: 8001 2024-02-01T10:48:50.132 DEBUG:boto:Params: {} 2024-02-01T10:48:50.132 DEBUG:boto:Token: None 2024-02-01T10:48:50.132 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:50 GMT /vclfcn-62/ 2024-02-01T10:48:50.132 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HbMbpqmh6Ygyn/XL8EK0MBvaGls= 2024-02-01T10:48:50.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HbMbpqmh6Ygyn/XL8EK0MBvaGls=', 'Content-Length': '0'} 2024-02-01T10:48:50.136 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000603057f88aff423d-0065bb7712-5762-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:48:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:50.136 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:48:50.137 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:48:50.137 DEBUG:boto:Method: HEAD 2024-02-01T10:48:50.137 DEBUG:boto:Path: /vclfcn-62/ 2024-02-01T10:48:50.137 DEBUG:boto:Data: 2024-02-01T10:48:50.137 DEBUG:boto:Headers: {} 2024-02-01T10:48:50.137 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:50.137 DEBUG:boto:Port: 8000 2024-02-01T10:48:50.137 DEBUG:boto:Params: {} 2024-02-01T10:48:50.138 DEBUG:boto:Token: None 2024-02-01T10:48:50.138 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:50 GMT /vclfcn-62/ 2024-02-01T10:48:50.138 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HbMbpqmh6Ygyn/XL8EK0MBvaGls= 2024-02-01T10:48:50.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HbMbpqmh6Ygyn/XL8EK0MBvaGls=', 'Content-Length': '0'} 2024-02-01T10:48:50.143 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002dd85d74785e537f-0065bb7712-5350-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:48:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:50.144 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:48:50.144 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:48:50.144 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:48:50.144 DEBUG:boto:path=/vclfcn-62/?versions&max-keys=999 2024-02-01T10:48:50.144 DEBUG:boto:auth_path=/vclfcn-62/?versions&max-keys=999 2024-02-01T10:48:50.144 DEBUG:boto:Method: GET 2024-02-01T10:48:50.144 DEBUG:boto:Path: /vclfcn-62/?versions&max-keys=999 2024-02-01T10:48:50.144 DEBUG:boto:Data: 2024-02-01T10:48:50.144 DEBUG:boto:Headers: {} 2024-02-01T10:48:50.145 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:50.145 DEBUG:boto:Port: 8001 2024-02-01T10:48:50.145 DEBUG:boto:Params: {} 2024-02-01T10:48:50.145 DEBUG:boto:Token: None 2024-02-01T10:48:50.145 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:50 GMT /vclfcn-62/?versions 2024-02-01T10:48:50.145 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:w/ZTqoY2XPItbBLwjUHqxcsBQhA= 2024-02-01T10:48:50.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:w/ZTqoY2XPItbBLwjUHqxcsBQhA=', 'Content-Length': '0'} 2024-02-01T10:48:50.149 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5d46e10a3bcf98e-0065bb7712-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:50.150 DEBUG:boto:b'vclfcn-62999falseobj1nulltrue2024-02-01T10:48:26.315Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:48:26.333Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:48:26.384Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:48:26.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:50.150 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-02-01T10:48:50.151 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-02-01T10:48:50.151 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-02-01T10:48:50.151 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-02-01T10:48:50.151 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:48:50.151 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:48:50.151 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:48:50.151 DEBUG:boto:path=/vclfcn-62/?versions&max-keys=999 2024-02-01T10:48:50.152 DEBUG:boto:auth_path=/vclfcn-62/?versions&max-keys=999 2024-02-01T10:48:50.152 DEBUG:boto:Method: GET 2024-02-01T10:48:50.152 DEBUG:boto:Path: /vclfcn-62/?versions&max-keys=999 2024-02-01T10:48:50.152 DEBUG:boto:Data: 2024-02-01T10:48:50.152 DEBUG:boto:Headers: {} 2024-02-01T10:48:50.152 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:50.152 DEBUG:boto:Port: 8000 2024-02-01T10:48:50.152 DEBUG:boto:Params: {} 2024-02-01T10:48:50.152 DEBUG:boto:Token: None 2024-02-01T10:48:50.152 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:50 GMT /vclfcn-62/?versions 2024-02-01T10:48:50.153 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:w/ZTqoY2XPItbBLwjUHqxcsBQhA= 2024-02-01T10:48:50.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:w/ZTqoY2XPItbBLwjUHqxcsBQhA=', 'Content-Length': '0'} 2024-02-01T10:48:50.157 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b8c1819aed449281-0065bb7712-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:50.157 DEBUG:boto:b'vclfcn-62999falseobj1nulltrue2024-02-01T10:48:26.315Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:48:26.333Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:48:26.384Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:48:26.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:50.158 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-02-01T10:48:50.158 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-02-01T10:48:50.158 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-02-01T10:48:50.158 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-02-01T10:48:50.158 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:48:50.158 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:48:50.158 DEBUG:boto:path=/vclfcn-62/?versions&max-keys=999 2024-02-01T10:48:50.158 DEBUG:boto:auth_path=/vclfcn-62/?versions&max-keys=999 2024-02-01T10:48:50.159 DEBUG:boto:Method: GET 2024-02-01T10:48:50.159 DEBUG:boto:Path: /vclfcn-62/?versions&max-keys=999 2024-02-01T10:48:50.159 DEBUG:boto:Data: 2024-02-01T10:48:50.159 DEBUG:boto:Headers: {} 2024-02-01T10:48:50.159 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:50.159 DEBUG:boto:Port: 8001 2024-02-01T10:48:50.159 DEBUG:boto:Params: {} 2024-02-01T10:48:50.159 DEBUG:boto:Token: None 2024-02-01T10:48:50.159 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:50 GMT /vclfcn-62/?versions 2024-02-01T10:48:50.160 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:w/ZTqoY2XPItbBLwjUHqxcsBQhA= 2024-02-01T10:48:50.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:w/ZTqoY2XPItbBLwjUHqxcsBQhA=', 'Content-Length': '0'} 2024-02-01T10:48:50.164 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051e745a0057c5adb-0065bb7712-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:50.164 DEBUG:boto:b'vclfcn-62999falseobj1nulltrue2024-02-01T10:48:26.315Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:48:26.333Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:48:26.384Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:48:26.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:50.164 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:48:50.165 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:48:50.165 DEBUG:boto:path=/vclfcn-62/?versions&max-keys=999 2024-02-01T10:48:50.165 DEBUG:boto:auth_path=/vclfcn-62/?versions&max-keys=999 2024-02-01T10:48:50.165 DEBUG:boto:Method: GET 2024-02-01T10:48:50.165 DEBUG:boto:Path: /vclfcn-62/?versions&max-keys=999 2024-02-01T10:48:50.165 DEBUG:boto:Data: 2024-02-01T10:48:50.165 DEBUG:boto:Headers: {} 2024-02-01T10:48:50.165 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:50.165 DEBUG:boto:Port: 8000 2024-02-01T10:48:50.165 DEBUG:boto:Params: {} 2024-02-01T10:48:50.166 DEBUG:boto:Token: None 2024-02-01T10:48:50.166 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:50 GMT /vclfcn-62/?versions 2024-02-01T10:48:50.166 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:w/ZTqoY2XPItbBLwjUHqxcsBQhA= 2024-02-01T10:48:50.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:w/ZTqoY2XPItbBLwjUHqxcsBQhA=', 'Content-Length': '0'} 2024-02-01T10:48:50.170 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002919aff88679b4c9-0065bb7712-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:50.171 DEBUG:boto:b'vclfcn-62999falseobj1nulltrue2024-02-01T10:48:26.315Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:48:26.333Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:48:26.384Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:48:26.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:50.171 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:48:50.172 DEBUG:boto:path=/vclfcn-62/obj1 2024-02-01T10:48:50.172 DEBUG:boto:auth_path=/vclfcn-62/obj1 2024-02-01T10:48:50.172 DEBUG:boto:path=/vclfcn-62/obj1?versionId=null 2024-02-01T10:48:50.172 DEBUG:boto:auth_path=/vclfcn-62/obj1?versionId=null 2024-02-01T10:48:50.172 DEBUG:boto:Method: GET 2024-02-01T10:48:50.172 DEBUG:boto:Path: /vclfcn-62/obj1?versionId=null 2024-02-01T10:48:50.172 DEBUG:boto:Data: 2024-02-01T10:48:50.172 DEBUG:boto:Headers: {} 2024-02-01T10:48:50.172 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:50.172 DEBUG:boto:Port: 8001 2024-02-01T10:48:50.173 DEBUG:boto:Params: {} 2024-02-01T10:48:50.173 DEBUG:boto:Token: None 2024-02-01T10:48:50.173 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:50 GMT /vclfcn-62/obj1?versionId=null 2024-02-01T10:48:50.173 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:I9Lo03PDgv4ShCFOX7NODIUGubs= 2024-02-01T10:48:50.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:I9Lo03PDgv4ShCFOX7NODIUGubs=', 'Content-Length': '0'} 2024-02-01T10:48:50.176 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cd9331963eef7ab6-0065bb7712-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:50.224 DEBUG:boto:path=/vclfcn-62/obj1 2024-02-01T10:48:50.224 DEBUG:boto:auth_path=/vclfcn-62/obj1 2024-02-01T10:48:50.225 DEBUG:boto:path=/vclfcn-62/obj1?versionId=null 2024-02-01T10:48:50.225 DEBUG:boto:auth_path=/vclfcn-62/obj1?versionId=null 2024-02-01T10:48:50.225 DEBUG:boto:Method: GET 2024-02-01T10:48:50.225 DEBUG:boto:Path: /vclfcn-62/obj1?versionId=null 2024-02-01T10:48:50.225 DEBUG:boto:Data: 2024-02-01T10:48:50.226 DEBUG:boto:Headers: {} 2024-02-01T10:48:50.226 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:50.226 DEBUG:boto:Port: 8000 2024-02-01T10:48:50.226 DEBUG:boto:Params: {} 2024-02-01T10:48:50.227 DEBUG:boto:Token: None 2024-02-01T10:48:50.227 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:50 GMT /vclfcn-62/obj1?versionId=null 2024-02-01T10:48:50.227 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:I9Lo03PDgv4ShCFOX7NODIUGubs= 2024-02-01T10:48:50.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:I9Lo03PDgv4ShCFOX7NODIUGubs=', 'Content-Length': '0'} 2024-02-01T10:48:50.230 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009bdb1186274d4fca-0065bb7712-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:50.231 DEBUG:boto:path=/vclfcn-62/obj1 2024-02-01T10:48:50.231 DEBUG:boto:auth_path=/vclfcn-62/obj1 2024-02-01T10:48:50.231 DEBUG:boto:path=/vclfcn-62/obj1?versionId=null 2024-02-01T10:48:50.231 DEBUG:boto:auth_path=/vclfcn-62/obj1?versionId=null 2024-02-01T10:48:50.232 DEBUG:boto:Method: HEAD 2024-02-01T10:48:50.232 DEBUG:boto:Path: /vclfcn-62/obj1?versionId=null 2024-02-01T10:48:50.232 DEBUG:boto:Data: 2024-02-01T10:48:50.232 DEBUG:boto:Headers: {} 2024-02-01T10:48:50.232 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:50.232 DEBUG:boto:Port: 8001 2024-02-01T10:48:50.233 DEBUG:boto:Params: {} 2024-02-01T10:48:50.233 DEBUG:boto:Token: None 2024-02-01T10:48:50.233 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:50 GMT /vclfcn-62/obj1?versionId=null 2024-02-01T10:48:50.234 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YCEfnDLmNf+vbHL3CU1THpIhQ3Q= 2024-02-01T10:48:50.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YCEfnDLmNf+vbHL3CU1THpIhQ3Q=', 'Content-Length': '0'} 2024-02-01T10:48:50.237 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a35455cbde4aa40b-0065bb7712-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:50.237 DEBUG:boto:path=/vclfcn-62/obj1 2024-02-01T10:48:50.237 DEBUG:boto:auth_path=/vclfcn-62/obj1 2024-02-01T10:48:50.238 DEBUG:boto:path=/vclfcn-62/obj1?versionId=null 2024-02-01T10:48:50.238 DEBUG:boto:auth_path=/vclfcn-62/obj1?versionId=null 2024-02-01T10:48:50.238 DEBUG:boto:Method: HEAD 2024-02-01T10:48:50.238 DEBUG:boto:Path: /vclfcn-62/obj1?versionId=null 2024-02-01T10:48:50.238 DEBUG:boto:Data: 2024-02-01T10:48:50.238 DEBUG:boto:Headers: {} 2024-02-01T10:48:50.238 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:50.238 DEBUG:boto:Port: 8000 2024-02-01T10:48:50.239 DEBUG:boto:Params: {} 2024-02-01T10:48:50.239 DEBUG:boto:Token: None 2024-02-01T10:48:50.239 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:50 GMT /vclfcn-62/obj1?versionId=null 2024-02-01T10:48:50.239 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YCEfnDLmNf+vbHL3CU1THpIhQ3Q= 2024-02-01T10:48:50.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YCEfnDLmNf+vbHL3CU1THpIhQ3Q=', 'Content-Length': '0'} 2024-02-01T10:48:50.242 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006b7742ed97ba97a0-0065bb7712-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:50.243 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:48:50.243 DEBUG:boto:path=/vclfcn-62/obj1 2024-02-01T10:48:50.244 DEBUG:boto:auth_path=/vclfcn-62/obj1 2024-02-01T10:48:50.244 DEBUG:boto:path=/vclfcn-62/obj1?versionId=null 2024-02-01T10:48:50.244 DEBUG:boto:auth_path=/vclfcn-62/obj1?versionId=null 2024-02-01T10:48:50.244 DEBUG:boto:Method: GET 2024-02-01T10:48:50.244 DEBUG:boto:Path: /vclfcn-62/obj1?versionId=null 2024-02-01T10:48:50.245 DEBUG:boto:Data: 2024-02-01T10:48:50.245 DEBUG:boto:Headers: {} 2024-02-01T10:48:50.245 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:50.245 DEBUG:boto:Port: 8001 2024-02-01T10:48:50.245 DEBUG:boto:Params: {} 2024-02-01T10:48:50.245 DEBUG:boto:Token: None 2024-02-01T10:48:50.245 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:50 GMT /vclfcn-62/obj1?versionId=null 2024-02-01T10:48:50.246 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:I9Lo03PDgv4ShCFOX7NODIUGubs= 2024-02-01T10:48:50.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:I9Lo03PDgv4ShCFOX7NODIUGubs=', 'Content-Length': '0'} 2024-02-01T10:48:50.248 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003f377708a7f2c7be-0065bb7712-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:50.292 DEBUG:boto:path=/vclfcn-62/obj1 2024-02-01T10:48:50.292 DEBUG:boto:auth_path=/vclfcn-62/obj1 2024-02-01T10:48:50.292 DEBUG:boto:path=/vclfcn-62/obj1?versionId=null 2024-02-01T10:48:50.293 DEBUG:boto:auth_path=/vclfcn-62/obj1?versionId=null 2024-02-01T10:48:50.293 DEBUG:boto:Method: GET 2024-02-01T10:48:50.293 DEBUG:boto:Path: /vclfcn-62/obj1?versionId=null 2024-02-01T10:48:50.293 DEBUG:boto:Data: 2024-02-01T10:48:50.293 DEBUG:boto:Headers: {} 2024-02-01T10:48:50.293 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:50.293 DEBUG:boto:Port: 8000 2024-02-01T10:48:50.293 DEBUG:boto:Params: {} 2024-02-01T10:48:50.293 DEBUG:boto:Token: None 2024-02-01T10:48:50.294 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:50 GMT /vclfcn-62/obj1?versionId=null 2024-02-01T10:48:50.294 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:I9Lo03PDgv4ShCFOX7NODIUGubs= 2024-02-01T10:48:50.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:I9Lo03PDgv4ShCFOX7NODIUGubs=', 'Content-Length': '0'} 2024-02-01T10:48:50.297 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000071512233c5f63b82-0065bb7712-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:50.340 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:48:50.340 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:48:50.340 DEBUG:boto:path=/vclfcn-62/?prefix=obj1 2024-02-01T10:48:50.340 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj1 2024-02-01T10:48:50.341 DEBUG:boto:Method: GET 2024-02-01T10:48:50.341 DEBUG:boto:Path: /vclfcn-62/?prefix=obj1 2024-02-01T10:48:50.341 DEBUG:boto:Data: 2024-02-01T10:48:50.341 DEBUG:boto:Headers: {} 2024-02-01T10:48:50.341 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:50.341 DEBUG:boto:Port: 8001 2024-02-01T10:48:50.341 DEBUG:boto:Params: {} 2024-02-01T10:48:50.341 DEBUG:boto:Token: None 2024-02-01T10:48:50.341 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:50 GMT /vclfcn-62/ 2024-02-01T10:48:50.342 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:pJ6p/CfmqNl3fCHPPxzKZlU2Rjs= 2024-02-01T10:48:50.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:pJ6p/CfmqNl3fCHPPxzKZlU2Rjs=', 'Content-Length': '0'} 2024-02-01T10:48:50.346 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c23ddab34cce2878-0065bb7712-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:50.388 DEBUG:boto:b'vclfcn-62obj11000falseobj12024-02-01T10:48:26.315Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:50.389 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:48:50.389 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:48:50.389 DEBUG:boto:path=/vclfcn-62/?prefix=obj1 2024-02-01T10:48:50.389 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj1 2024-02-01T10:48:50.389 DEBUG:boto:Method: GET 2024-02-01T10:48:50.389 DEBUG:boto:Path: /vclfcn-62/?prefix=obj1 2024-02-01T10:48:50.389 DEBUG:boto:Data: 2024-02-01T10:48:50.389 DEBUG:boto:Headers: {} 2024-02-01T10:48:50.390 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:50.390 DEBUG:boto:Port: 8000 2024-02-01T10:48:50.390 DEBUG:boto:Params: {} 2024-02-01T10:48:50.390 DEBUG:boto:Token: None 2024-02-01T10:48:50.390 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:50 GMT /vclfcn-62/ 2024-02-01T10:48:50.390 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:pJ6p/CfmqNl3fCHPPxzKZlU2Rjs= 2024-02-01T10:48:50.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:pJ6p/CfmqNl3fCHPPxzKZlU2Rjs=', 'Content-Length': '0'} 2024-02-01T10:48:50.395 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8416d881df8fb16-0065bb7712-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:50.395 DEBUG:boto:b'vclfcn-62obj11000falseobj12024-02-01T10:48:26.315Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:50.396 DEBUG:boto:path=/vclfcn-62/obj1 2024-02-01T10:48:50.396 DEBUG:boto:auth_path=/vclfcn-62/obj1 2024-02-01T10:48:50.396 DEBUG:boto:Method: HEAD 2024-02-01T10:48:50.396 DEBUG:boto:Path: /vclfcn-62/obj1 2024-02-01T10:48:50.396 DEBUG:boto:Data: 2024-02-01T10:48:50.397 DEBUG:boto:Headers: {} 2024-02-01T10:48:50.397 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:50.397 DEBUG:boto:Port: 8001 2024-02-01T10:48:50.397 DEBUG:boto:Params: {} 2024-02-01T10:48:50.397 DEBUG:boto:Token: None 2024-02-01T10:48:50.397 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:50 GMT /vclfcn-62/obj1 2024-02-01T10:48:50.397 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:JLwa6f9C5s2/xuomWAYwq6S9T3A= 2024-02-01T10:48:50.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:JLwa6f9C5s2/xuomWAYwq6S9T3A=', 'Content-Length': '0'} 2024-02-01T10:48:50.400 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fc0f6b28c04ec989-0065bb7712-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:50.401 DEBUG:boto:path=/vclfcn-62/obj1 2024-02-01T10:48:50.401 DEBUG:boto:auth_path=/vclfcn-62/obj1 2024-02-01T10:48:50.401 DEBUG:boto:Method: HEAD 2024-02-01T10:48:50.401 DEBUG:boto:Path: /vclfcn-62/obj1 2024-02-01T10:48:50.401 DEBUG:boto:Data: 2024-02-01T10:48:50.401 DEBUG:boto:Headers: {} 2024-02-01T10:48:50.401 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:50.401 DEBUG:boto:Port: 8000 2024-02-01T10:48:50.401 DEBUG:boto:Params: {} 2024-02-01T10:48:50.401 DEBUG:boto:Token: None 2024-02-01T10:48:50.402 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:50 GMT /vclfcn-62/obj1 2024-02-01T10:48:50.402 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:JLwa6f9C5s2/xuomWAYwq6S9T3A= 2024-02-01T10:48:50.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:JLwa6f9C5s2/xuomWAYwq6S9T3A=', 'Content-Length': '0'} 2024-02-01T10:48:50.405 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003d85007557af04dd-0065bb7712-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:50.405 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:48:50.405 DEBUG:boto:path=/vclfcn-62/obj1 2024-02-01T10:48:50.406 DEBUG:boto:auth_path=/vclfcn-62/obj1 2024-02-01T10:48:50.406 DEBUG:boto:Method: GET 2024-02-01T10:48:50.406 DEBUG:boto:Path: /vclfcn-62/obj1 2024-02-01T10:48:50.406 DEBUG:boto:Data: 2024-02-01T10:48:50.406 DEBUG:boto:Headers: {} 2024-02-01T10:48:50.406 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:50.406 DEBUG:boto:Port: 8001 2024-02-01T10:48:50.406 DEBUG:boto:Params: {} 2024-02-01T10:48:50.406 DEBUG:boto:Token: None 2024-02-01T10:48:50.406 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:50 GMT /vclfcn-62/obj1 2024-02-01T10:48:50.407 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:TVuSzZfyWnw+H6N1HHVpgZ+JbkY= 2024-02-01T10:48:50.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:TVuSzZfyWnw+H6N1HHVpgZ+JbkY=', 'Content-Length': '0'} 2024-02-01T10:48:50.410 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008ea60414b0160949-0065bb7712-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:50.456 DEBUG:boto:path=/vclfcn-62/obj1 2024-02-01T10:48:50.456 DEBUG:boto:auth_path=/vclfcn-62/obj1 2024-02-01T10:48:50.456 DEBUG:boto:Method: GET 2024-02-01T10:48:50.456 DEBUG:boto:Path: /vclfcn-62/obj1 2024-02-01T10:48:50.457 DEBUG:boto:Data: 2024-02-01T10:48:50.457 DEBUG:boto:Headers: {} 2024-02-01T10:48:50.457 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:50.457 DEBUG:boto:Port: 8000 2024-02-01T10:48:50.457 DEBUG:boto:Params: {} 2024-02-01T10:48:50.457 DEBUG:boto:Token: None 2024-02-01T10:48:50.457 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:50 GMT /vclfcn-62/obj1 2024-02-01T10:48:50.457 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:TVuSzZfyWnw+H6N1HHVpgZ+JbkY= 2024-02-01T10:48:50.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:TVuSzZfyWnw+H6N1HHVpgZ+JbkY=', 'Content-Length': '0'} 2024-02-01T10:48:50.460 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fbdfb0f3c639d706-0065bb7712-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:50.504 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:48:50.504 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:48:50.504 DEBUG:boto:path=/vclfcn-62/?prefix=obj1 2024-02-01T10:48:50.504 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj1 2024-02-01T10:48:50.505 DEBUG:boto:Method: GET 2024-02-01T10:48:50.505 DEBUG:boto:Path: /vclfcn-62/?prefix=obj1 2024-02-01T10:48:50.505 DEBUG:boto:Data: 2024-02-01T10:48:50.505 DEBUG:boto:Headers: {} 2024-02-01T10:48:50.505 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:50.505 DEBUG:boto:Port: 8001 2024-02-01T10:48:50.505 DEBUG:boto:Params: {} 2024-02-01T10:48:50.505 DEBUG:boto:Token: None 2024-02-01T10:48:50.506 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:50 GMT /vclfcn-62/ 2024-02-01T10:48:50.506 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:pJ6p/CfmqNl3fCHPPxzKZlU2Rjs= 2024-02-01T10:48:50.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:pJ6p/CfmqNl3fCHPPxzKZlU2Rjs=', 'Content-Length': '0'} 2024-02-01T10:48:50.510 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000085eab3abcfcb06fa-0065bb7712-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:50.552 DEBUG:boto:b'vclfcn-62obj11000falseobj12024-02-01T10:48:26.315Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:50.552 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:48:50.553 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:48:50.553 DEBUG:boto:path=/vclfcn-62/?prefix=obj1 2024-02-01T10:48:50.553 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj1 2024-02-01T10:48:50.553 DEBUG:boto:Method: GET 2024-02-01T10:48:50.553 DEBUG:boto:Path: /vclfcn-62/?prefix=obj1 2024-02-01T10:48:50.553 DEBUG:boto:Data: 2024-02-01T10:48:50.553 DEBUG:boto:Headers: {} 2024-02-01T10:48:50.553 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:50.553 DEBUG:boto:Port: 8000 2024-02-01T10:48:50.553 DEBUG:boto:Params: {} 2024-02-01T10:48:50.554 DEBUG:boto:Token: None 2024-02-01T10:48:50.554 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:50 GMT /vclfcn-62/ 2024-02-01T10:48:50.554 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:pJ6p/CfmqNl3fCHPPxzKZlU2Rjs= 2024-02-01T10:48:50.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:pJ6p/CfmqNl3fCHPPxzKZlU2Rjs=', 'Content-Length': '0'} 2024-02-01T10:48:50.558 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009692c90d23522abf-0065bb7712-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:50.604 DEBUG:boto:b'vclfcn-62obj11000falseobj12024-02-01T10:48:26.315Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:50.604 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:48:50.605 DEBUG:boto:path=/vclfcn-62/obj2 2024-02-01T10:48:50.605 DEBUG:boto:auth_path=/vclfcn-62/obj2 2024-02-01T10:48:50.605 DEBUG:boto:path=/vclfcn-62/obj2?versionId=null 2024-02-01T10:48:50.605 DEBUG:boto:auth_path=/vclfcn-62/obj2?versionId=null 2024-02-01T10:48:50.605 DEBUG:boto:Method: GET 2024-02-01T10:48:50.605 DEBUG:boto:Path: /vclfcn-62/obj2?versionId=null 2024-02-01T10:48:50.605 DEBUG:boto:Data: 2024-02-01T10:48:50.605 DEBUG:boto:Headers: {} 2024-02-01T10:48:50.605 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:50.605 DEBUG:boto:Port: 8001 2024-02-01T10:48:50.605 DEBUG:boto:Params: {} 2024-02-01T10:48:50.606 DEBUG:boto:Token: None 2024-02-01T10:48:50.606 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:50 GMT /vclfcn-62/obj2?versionId=null 2024-02-01T10:48:50.606 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vP8xv4GBQWYJYS40EZ8IOh91reI= 2024-02-01T10:48:50.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vP8xv4GBQWYJYS40EZ8IOh91reI=', 'Content-Length': '0'} 2024-02-01T10:48:50.609 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009b01855c15b14cc7-0065bb7712-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:50.609 DEBUG:boto:path=/vclfcn-62/obj2 2024-02-01T10:48:50.609 DEBUG:boto:auth_path=/vclfcn-62/obj2 2024-02-01T10:48:50.609 DEBUG:boto:path=/vclfcn-62/obj2?versionId=null 2024-02-01T10:48:50.610 DEBUG:boto:auth_path=/vclfcn-62/obj2?versionId=null 2024-02-01T10:48:50.610 DEBUG:boto:Method: GET 2024-02-01T10:48:50.610 DEBUG:boto:Path: /vclfcn-62/obj2?versionId=null 2024-02-01T10:48:50.610 DEBUG:boto:Data: 2024-02-01T10:48:50.610 DEBUG:boto:Headers: {} 2024-02-01T10:48:50.610 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:50.610 DEBUG:boto:Port: 8000 2024-02-01T10:48:50.610 DEBUG:boto:Params: {} 2024-02-01T10:48:50.610 DEBUG:boto:Token: None 2024-02-01T10:48:50.611 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:50 GMT /vclfcn-62/obj2?versionId=null 2024-02-01T10:48:50.611 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vP8xv4GBQWYJYS40EZ8IOh91reI= 2024-02-01T10:48:50.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vP8xv4GBQWYJYS40EZ8IOh91reI=', 'Content-Length': '0'} 2024-02-01T10:48:50.614 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000227f3cea94f441f7-0065bb7712-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:50.656 DEBUG:boto:path=/vclfcn-62/obj2 2024-02-01T10:48:50.656 DEBUG:boto:auth_path=/vclfcn-62/obj2 2024-02-01T10:48:50.656 DEBUG:boto:path=/vclfcn-62/obj2?versionId=null 2024-02-01T10:48:50.656 DEBUG:boto:auth_path=/vclfcn-62/obj2?versionId=null 2024-02-01T10:48:50.657 DEBUG:boto:Method: HEAD 2024-02-01T10:48:50.657 DEBUG:boto:Path: /vclfcn-62/obj2?versionId=null 2024-02-01T10:48:50.657 DEBUG:boto:Data: 2024-02-01T10:48:50.657 DEBUG:boto:Headers: {} 2024-02-01T10:48:50.657 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:50.657 DEBUG:boto:Port: 8001 2024-02-01T10:48:50.657 DEBUG:boto:Params: {} 2024-02-01T10:48:50.657 DEBUG:boto:Token: None 2024-02-01T10:48:50.657 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:50 GMT /vclfcn-62/obj2?versionId=null 2024-02-01T10:48:50.658 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:KiQNd5k/wL0Wx0m1zkbkgloM1nw= 2024-02-01T10:48:50.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:KiQNd5k/wL0Wx0m1zkbkgloM1nw=', 'Content-Length': '0'} 2024-02-01T10:48:50.660 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e11b23e6f53b0d29-0065bb7712-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:50.661 DEBUG:boto:path=/vclfcn-62/obj2 2024-02-01T10:48:50.661 DEBUG:boto:auth_path=/vclfcn-62/obj2 2024-02-01T10:48:50.661 DEBUG:boto:path=/vclfcn-62/obj2?versionId=null 2024-02-01T10:48:50.661 DEBUG:boto:auth_path=/vclfcn-62/obj2?versionId=null 2024-02-01T10:48:50.661 DEBUG:boto:Method: HEAD 2024-02-01T10:48:50.661 DEBUG:boto:Path: /vclfcn-62/obj2?versionId=null 2024-02-01T10:48:50.661 DEBUG:boto:Data: 2024-02-01T10:48:50.661 DEBUG:boto:Headers: {} 2024-02-01T10:48:50.662 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:50.662 DEBUG:boto:Port: 8000 2024-02-01T10:48:50.662 DEBUG:boto:Params: {} 2024-02-01T10:48:50.662 DEBUG:boto:Token: None 2024-02-01T10:48:50.662 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:50 GMT /vclfcn-62/obj2?versionId=null 2024-02-01T10:48:50.662 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:KiQNd5k/wL0Wx0m1zkbkgloM1nw= 2024-02-01T10:48:50.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:KiQNd5k/wL0Wx0m1zkbkgloM1nw=', 'Content-Length': '0'} 2024-02-01T10:48:50.665 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003955eabe09f5d307-0065bb7712-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:50.665 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:48:50.665 DEBUG:boto:path=/vclfcn-62/obj2 2024-02-01T10:48:50.665 DEBUG:boto:auth_path=/vclfcn-62/obj2 2024-02-01T10:48:50.665 DEBUG:boto:path=/vclfcn-62/obj2?versionId=null 2024-02-01T10:48:50.665 DEBUG:boto:auth_path=/vclfcn-62/obj2?versionId=null 2024-02-01T10:48:50.666 DEBUG:boto:Method: GET 2024-02-01T10:48:50.666 DEBUG:boto:Path: /vclfcn-62/obj2?versionId=null 2024-02-01T10:48:50.666 DEBUG:boto:Data: 2024-02-01T10:48:50.666 DEBUG:boto:Headers: {} 2024-02-01T10:48:50.666 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:50.666 DEBUG:boto:Port: 8001 2024-02-01T10:48:50.666 DEBUG:boto:Params: {} 2024-02-01T10:48:50.666 DEBUG:boto:Token: None 2024-02-01T10:48:50.666 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:50 GMT /vclfcn-62/obj2?versionId=null 2024-02-01T10:48:50.666 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vP8xv4GBQWYJYS40EZ8IOh91reI= 2024-02-01T10:48:50.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vP8xv4GBQWYJYS40EZ8IOh91reI=', 'Content-Length': '0'} 2024-02-01T10:48:50.669 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000031508fb7fad56911-0065bb7712-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:50.712 DEBUG:boto:path=/vclfcn-62/obj2 2024-02-01T10:48:50.712 DEBUG:boto:auth_path=/vclfcn-62/obj2 2024-02-01T10:48:50.712 DEBUG:boto:path=/vclfcn-62/obj2?versionId=null 2024-02-01T10:48:50.712 DEBUG:boto:auth_path=/vclfcn-62/obj2?versionId=null 2024-02-01T10:48:50.712 DEBUG:boto:Method: GET 2024-02-01T10:48:50.713 DEBUG:boto:Path: /vclfcn-62/obj2?versionId=null 2024-02-01T10:48:50.713 DEBUG:boto:Data: 2024-02-01T10:48:50.713 DEBUG:boto:Headers: {} 2024-02-01T10:48:50.713 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:50.713 DEBUG:boto:Port: 8000 2024-02-01T10:48:50.713 DEBUG:boto:Params: {} 2024-02-01T10:48:50.713 DEBUG:boto:Token: None 2024-02-01T10:48:50.713 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:50 GMT /vclfcn-62/obj2?versionId=null 2024-02-01T10:48:50.713 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vP8xv4GBQWYJYS40EZ8IOh91reI= 2024-02-01T10:48:50.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vP8xv4GBQWYJYS40EZ8IOh91reI=', 'Content-Length': '0'} 2024-02-01T10:48:50.716 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ee7b6c3a1812bd85-0065bb7712-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:50.760 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:48:50.760 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:48:50.760 DEBUG:boto:path=/vclfcn-62/?prefix=obj2 2024-02-01T10:48:50.760 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj2 2024-02-01T10:48:50.760 DEBUG:boto:Method: GET 2024-02-01T10:48:50.761 DEBUG:boto:Path: /vclfcn-62/?prefix=obj2 2024-02-01T10:48:50.761 DEBUG:boto:Data: 2024-02-01T10:48:50.761 DEBUG:boto:Headers: {} 2024-02-01T10:48:50.761 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:50.761 DEBUG:boto:Port: 8001 2024-02-01T10:48:50.761 DEBUG:boto:Params: {} 2024-02-01T10:48:50.761 DEBUG:boto:Token: None 2024-02-01T10:48:50.761 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:50 GMT /vclfcn-62/ 2024-02-01T10:48:50.761 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:pJ6p/CfmqNl3fCHPPxzKZlU2Rjs= 2024-02-01T10:48:50.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:pJ6p/CfmqNl3fCHPPxzKZlU2Rjs=', 'Content-Length': '0'} 2024-02-01T10:48:50.766 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eeaa3899982eafd0-0065bb7712-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:50.766 DEBUG:boto:b'vclfcn-62obj21000falseobj22024-02-01T10:48:26.333Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:50.767 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:48:50.767 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:48:50.767 DEBUG:boto:path=/vclfcn-62/?prefix=obj2 2024-02-01T10:48:50.767 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj2 2024-02-01T10:48:50.767 DEBUG:boto:Method: GET 2024-02-01T10:48:50.767 DEBUG:boto:Path: /vclfcn-62/?prefix=obj2 2024-02-01T10:48:50.767 DEBUG:boto:Data: 2024-02-01T10:48:50.767 DEBUG:boto:Headers: {} 2024-02-01T10:48:50.767 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:50.768 DEBUG:boto:Port: 8000 2024-02-01T10:48:50.768 DEBUG:boto:Params: {} 2024-02-01T10:48:50.768 DEBUG:boto:Token: None 2024-02-01T10:48:50.768 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:50 GMT /vclfcn-62/ 2024-02-01T10:48:50.768 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:pJ6p/CfmqNl3fCHPPxzKZlU2Rjs= 2024-02-01T10:48:50.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:pJ6p/CfmqNl3fCHPPxzKZlU2Rjs=', 'Content-Length': '0'} 2024-02-01T10:48:50.773 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d6da7e8dea08d40-0065bb7712-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:50.773 DEBUG:boto:b'vclfcn-62obj21000falseobj22024-02-01T10:48:26.333Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:50.774 DEBUG:boto:path=/vclfcn-62/obj2 2024-02-01T10:48:50.774 DEBUG:boto:auth_path=/vclfcn-62/obj2 2024-02-01T10:48:50.774 DEBUG:boto:Method: HEAD 2024-02-01T10:48:50.774 DEBUG:boto:Path: /vclfcn-62/obj2 2024-02-01T10:48:50.774 DEBUG:boto:Data: 2024-02-01T10:48:50.774 DEBUG:boto:Headers: {} 2024-02-01T10:48:50.774 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:50.774 DEBUG:boto:Port: 8001 2024-02-01T10:48:50.774 DEBUG:boto:Params: {} 2024-02-01T10:48:50.775 DEBUG:boto:Token: None 2024-02-01T10:48:50.775 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:50 GMT /vclfcn-62/obj2 2024-02-01T10:48:50.775 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ONzn8U60oyep7tvFaFA2qnV+7L8= 2024-02-01T10:48:50.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ONzn8U60oyep7tvFaFA2qnV+7L8=', 'Content-Length': '0'} 2024-02-01T10:48:50.778 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000035c3a77ba943984a-0065bb7712-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:50.778 DEBUG:boto:path=/vclfcn-62/obj2 2024-02-01T10:48:50.778 DEBUG:boto:auth_path=/vclfcn-62/obj2 2024-02-01T10:48:50.778 DEBUG:boto:Method: HEAD 2024-02-01T10:48:50.779 DEBUG:boto:Path: /vclfcn-62/obj2 2024-02-01T10:48:50.779 DEBUG:boto:Data: 2024-02-01T10:48:50.779 DEBUG:boto:Headers: {} 2024-02-01T10:48:50.779 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:50.779 DEBUG:boto:Port: 8000 2024-02-01T10:48:50.779 DEBUG:boto:Params: {} 2024-02-01T10:48:50.779 DEBUG:boto:Token: None 2024-02-01T10:48:50.779 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:50 GMT /vclfcn-62/obj2 2024-02-01T10:48:50.779 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ONzn8U60oyep7tvFaFA2qnV+7L8= 2024-02-01T10:48:50.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ONzn8U60oyep7tvFaFA2qnV+7L8=', 'Content-Length': '0'} 2024-02-01T10:48:50.782 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000870da31e846c1863-0065bb7712-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:50.782 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:48:50.783 DEBUG:boto:path=/vclfcn-62/obj2 2024-02-01T10:48:50.783 DEBUG:boto:auth_path=/vclfcn-62/obj2 2024-02-01T10:48:50.783 DEBUG:boto:Method: GET 2024-02-01T10:48:50.783 DEBUG:boto:Path: /vclfcn-62/obj2 2024-02-01T10:48:50.783 DEBUG:boto:Data: 2024-02-01T10:48:50.783 DEBUG:boto:Headers: {} 2024-02-01T10:48:50.783 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:50.783 DEBUG:boto:Port: 8001 2024-02-01T10:48:50.783 DEBUG:boto:Params: {} 2024-02-01T10:48:50.784 DEBUG:boto:Token: None 2024-02-01T10:48:50.784 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:50 GMT /vclfcn-62/obj2 2024-02-01T10:48:50.784 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sEUSkd1dtxY6s3XBi+RBdU4yPT8= 2024-02-01T10:48:50.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sEUSkd1dtxY6s3XBi+RBdU4yPT8=', 'Content-Length': '0'} 2024-02-01T10:48:50.786 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fbd91188e0bc0111-0065bb7712-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:50.828 DEBUG:boto:path=/vclfcn-62/obj2 2024-02-01T10:48:50.828 DEBUG:boto:auth_path=/vclfcn-62/obj2 2024-02-01T10:48:50.828 DEBUG:boto:Method: GET 2024-02-01T10:48:50.828 DEBUG:boto:Path: /vclfcn-62/obj2 2024-02-01T10:48:50.828 DEBUG:boto:Data: 2024-02-01T10:48:50.829 DEBUG:boto:Headers: {} 2024-02-01T10:48:50.829 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:50.829 DEBUG:boto:Port: 8000 2024-02-01T10:48:50.829 DEBUG:boto:Params: {} 2024-02-01T10:48:50.829 DEBUG:boto:Token: None 2024-02-01T10:48:50.829 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:50 GMT /vclfcn-62/obj2 2024-02-01T10:48:50.830 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sEUSkd1dtxY6s3XBi+RBdU4yPT8= 2024-02-01T10:48:50.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sEUSkd1dtxY6s3XBi+RBdU4yPT8=', 'Content-Length': '0'} 2024-02-01T10:48:50.833 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000cdf3461f053992e-0065bb7712-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:50.880 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:48:50.880 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:48:50.880 DEBUG:boto:path=/vclfcn-62/?prefix=obj2 2024-02-01T10:48:50.880 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj2 2024-02-01T10:48:50.881 DEBUG:boto:Method: GET 2024-02-01T10:48:50.881 DEBUG:boto:Path: /vclfcn-62/?prefix=obj2 2024-02-01T10:48:50.881 DEBUG:boto:Data: 2024-02-01T10:48:50.881 DEBUG:boto:Headers: {} 2024-02-01T10:48:50.881 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:50.881 DEBUG:boto:Port: 8001 2024-02-01T10:48:50.881 DEBUG:boto:Params: {} 2024-02-01T10:48:50.881 DEBUG:boto:Token: None 2024-02-01T10:48:50.881 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:50 GMT /vclfcn-62/ 2024-02-01T10:48:50.882 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:pJ6p/CfmqNl3fCHPPxzKZlU2Rjs= 2024-02-01T10:48:50.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:pJ6p/CfmqNl3fCHPPxzKZlU2Rjs=', 'Content-Length': '0'} 2024-02-01T10:48:50.886 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b6300a333f34e82-0065bb7712-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:50.928 DEBUG:boto:b'vclfcn-62obj21000falseobj22024-02-01T10:48:26.333Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:50.928 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:48:50.929 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:48:50.929 DEBUG:boto:path=/vclfcn-62/?prefix=obj2 2024-02-01T10:48:50.929 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj2 2024-02-01T10:48:50.929 DEBUG:boto:Method: GET 2024-02-01T10:48:50.929 DEBUG:boto:Path: /vclfcn-62/?prefix=obj2 2024-02-01T10:48:50.929 DEBUG:boto:Data: 2024-02-01T10:48:50.929 DEBUG:boto:Headers: {} 2024-02-01T10:48:50.929 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:50.929 DEBUG:boto:Port: 8000 2024-02-01T10:48:50.929 DEBUG:boto:Params: {} 2024-02-01T10:48:50.930 DEBUG:boto:Token: None 2024-02-01T10:48:50.930 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:50 GMT /vclfcn-62/ 2024-02-01T10:48:50.930 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:pJ6p/CfmqNl3fCHPPxzKZlU2Rjs= 2024-02-01T10:48:50.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:pJ6p/CfmqNl3fCHPPxzKZlU2Rjs=', 'Content-Length': '0'} 2024-02-01T10:48:50.934 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b375060cc1f4d04-0065bb7712-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:50.976 DEBUG:boto:b'vclfcn-62obj21000falseobj22024-02-01T10:48:26.333Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:50.977 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:48:50.977 DEBUG:boto:path=/vclfcn-62/obj3 2024-02-01T10:48:50.977 DEBUG:boto:auth_path=/vclfcn-62/obj3 2024-02-01T10:48:50.977 DEBUG:boto:path=/vclfcn-62/obj3?versionId=null 2024-02-01T10:48:50.977 DEBUG:boto:auth_path=/vclfcn-62/obj3?versionId=null 2024-02-01T10:48:50.977 DEBUG:boto:Method: GET 2024-02-01T10:48:50.978 DEBUG:boto:Path: /vclfcn-62/obj3?versionId=null 2024-02-01T10:48:50.978 DEBUG:boto:Data: 2024-02-01T10:48:50.978 DEBUG:boto:Headers: {} 2024-02-01T10:48:50.978 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:50.978 DEBUG:boto:Port: 8001 2024-02-01T10:48:50.978 DEBUG:boto:Params: {} 2024-02-01T10:48:50.978 DEBUG:boto:Token: None 2024-02-01T10:48:50.978 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:50 GMT /vclfcn-62/obj3?versionId=null 2024-02-01T10:48:50.978 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:h4rOOOhQ0UJ9lJ3F8qOB5G0voYM= 2024-02-01T10:48:50.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:h4rOOOhQ0UJ9lJ3F8qOB5G0voYM=', 'Content-Length': '0'} 2024-02-01T10:48:50.981 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000c776fe5b8c91746-0065bb7712-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:50.982 DEBUG:boto:path=/vclfcn-62/obj3 2024-02-01T10:48:50.982 DEBUG:boto:auth_path=/vclfcn-62/obj3 2024-02-01T10:48:50.982 DEBUG:boto:path=/vclfcn-62/obj3?versionId=null 2024-02-01T10:48:50.982 DEBUG:boto:auth_path=/vclfcn-62/obj3?versionId=null 2024-02-01T10:48:50.982 DEBUG:boto:Method: GET 2024-02-01T10:48:50.982 DEBUG:boto:Path: /vclfcn-62/obj3?versionId=null 2024-02-01T10:48:50.982 DEBUG:boto:Data: 2024-02-01T10:48:50.983 DEBUG:boto:Headers: {} 2024-02-01T10:48:50.983 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:50.983 DEBUG:boto:Port: 8000 2024-02-01T10:48:50.983 DEBUG:boto:Params: {} 2024-02-01T10:48:50.983 DEBUG:boto:Token: None 2024-02-01T10:48:50.983 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:50 GMT /vclfcn-62/obj3?versionId=null 2024-02-01T10:48:50.983 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:h4rOOOhQ0UJ9lJ3F8qOB5G0voYM= 2024-02-01T10:48:50.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:50 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:h4rOOOhQ0UJ9lJ3F8qOB5G0voYM=', 'Content-Length': '0'} 2024-02-01T10:48:50.986 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005ec4bac310716ed1-0065bb7712-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:50 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:51.033 DEBUG:boto:path=/vclfcn-62/obj3 2024-02-01T10:48:51.033 DEBUG:boto:auth_path=/vclfcn-62/obj3 2024-02-01T10:48:51.033 DEBUG:boto:path=/vclfcn-62/obj3?versionId=null 2024-02-01T10:48:51.033 DEBUG:boto:auth_path=/vclfcn-62/obj3?versionId=null 2024-02-01T10:48:51.033 DEBUG:boto:Method: HEAD 2024-02-01T10:48:51.033 DEBUG:boto:Path: /vclfcn-62/obj3?versionId=null 2024-02-01T10:48:51.034 DEBUG:boto:Data: 2024-02-01T10:48:51.034 DEBUG:boto:Headers: {} 2024-02-01T10:48:51.034 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:51.034 DEBUG:boto:Port: 8001 2024-02-01T10:48:51.034 DEBUG:boto:Params: {} 2024-02-01T10:48:51.034 DEBUG:boto:Token: None 2024-02-01T10:48:51.034 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:51 GMT /vclfcn-62/obj3?versionId=null 2024-02-01T10:48:51.035 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Hr/LkEKIZeBNuVZTjpsPbAcDBag= 2024-02-01T10:48:51.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Hr/LkEKIZeBNuVZTjpsPbAcDBag=', 'Content-Length': '0'} 2024-02-01T10:48:51.037 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c64ef3ff39b1609c-0065bb7713-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:51.038 DEBUG:boto:path=/vclfcn-62/obj3 2024-02-01T10:48:51.038 DEBUG:boto:auth_path=/vclfcn-62/obj3 2024-02-01T10:48:51.038 DEBUG:boto:path=/vclfcn-62/obj3?versionId=null 2024-02-01T10:48:51.038 DEBUG:boto:auth_path=/vclfcn-62/obj3?versionId=null 2024-02-01T10:48:51.038 DEBUG:boto:Method: HEAD 2024-02-01T10:48:51.038 DEBUG:boto:Path: /vclfcn-62/obj3?versionId=null 2024-02-01T10:48:51.038 DEBUG:boto:Data: 2024-02-01T10:48:51.039 DEBUG:boto:Headers: {} 2024-02-01T10:48:51.039 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:51.039 DEBUG:boto:Port: 8000 2024-02-01T10:48:51.039 DEBUG:boto:Params: {} 2024-02-01T10:48:51.039 DEBUG:boto:Token: None 2024-02-01T10:48:51.039 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:51 GMT /vclfcn-62/obj3?versionId=null 2024-02-01T10:48:51.039 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Hr/LkEKIZeBNuVZTjpsPbAcDBag= 2024-02-01T10:48:51.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Hr/LkEKIZeBNuVZTjpsPbAcDBag=', 'Content-Length': '0'} 2024-02-01T10:48:51.042 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003e9b426b0bdef917-0065bb7713-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:51.042 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:48:51.042 DEBUG:boto:path=/vclfcn-62/obj3 2024-02-01T10:48:51.042 DEBUG:boto:auth_path=/vclfcn-62/obj3 2024-02-01T10:48:51.043 DEBUG:boto:path=/vclfcn-62/obj3?versionId=null 2024-02-01T10:48:51.043 DEBUG:boto:auth_path=/vclfcn-62/obj3?versionId=null 2024-02-01T10:48:51.043 DEBUG:boto:Method: GET 2024-02-01T10:48:51.043 DEBUG:boto:Path: /vclfcn-62/obj3?versionId=null 2024-02-01T10:48:51.043 DEBUG:boto:Data: 2024-02-01T10:48:51.043 DEBUG:boto:Headers: {} 2024-02-01T10:48:51.043 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:51.044 DEBUG:boto:Port: 8001 2024-02-01T10:48:51.044 DEBUG:boto:Params: {} 2024-02-01T10:48:51.044 DEBUG:boto:Token: None 2024-02-01T10:48:51.044 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:51 GMT /vclfcn-62/obj3?versionId=null 2024-02-01T10:48:51.044 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5m83mtinucVS6aV8N45Rpqofs4U= 2024-02-01T10:48:51.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5m83mtinucVS6aV8N45Rpqofs4U=', 'Content-Length': '0'} 2024-02-01T10:48:51.047 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000038daaa5f5464dbb8-0065bb7713-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:51.088 DEBUG:boto:path=/vclfcn-62/obj3 2024-02-01T10:48:51.088 DEBUG:boto:auth_path=/vclfcn-62/obj3 2024-02-01T10:48:51.088 DEBUG:boto:path=/vclfcn-62/obj3?versionId=null 2024-02-01T10:48:51.089 DEBUG:boto:auth_path=/vclfcn-62/obj3?versionId=null 2024-02-01T10:48:51.089 DEBUG:boto:Method: GET 2024-02-01T10:48:51.089 DEBUG:boto:Path: /vclfcn-62/obj3?versionId=null 2024-02-01T10:48:51.089 DEBUG:boto:Data: 2024-02-01T10:48:51.089 DEBUG:boto:Headers: {} 2024-02-01T10:48:51.090 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:51.090 DEBUG:boto:Port: 8000 2024-02-01T10:48:51.090 DEBUG:boto:Params: {} 2024-02-01T10:48:51.090 DEBUG:boto:Token: None 2024-02-01T10:48:51.090 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:51 GMT /vclfcn-62/obj3?versionId=null 2024-02-01T10:48:51.091 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5m83mtinucVS6aV8N45Rpqofs4U= 2024-02-01T10:48:51.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5m83mtinucVS6aV8N45Rpqofs4U=', 'Content-Length': '0'} 2024-02-01T10:48:51.094 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bb37ae261c38d7db-0065bb7713-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:51.136 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:48:51.136 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:48:51.137 DEBUG:boto:path=/vclfcn-62/?prefix=obj3 2024-02-01T10:48:51.137 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj3 2024-02-01T10:48:51.137 DEBUG:boto:Method: GET 2024-02-01T10:48:51.137 DEBUG:boto:Path: /vclfcn-62/?prefix=obj3 2024-02-01T10:48:51.137 DEBUG:boto:Data: 2024-02-01T10:48:51.137 DEBUG:boto:Headers: {} 2024-02-01T10:48:51.137 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:51.137 DEBUG:boto:Port: 8001 2024-02-01T10:48:51.137 DEBUG:boto:Params: {} 2024-02-01T10:48:51.138 DEBUG:boto:Token: None 2024-02-01T10:48:51.138 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:51 GMT /vclfcn-62/ 2024-02-01T10:48:51.138 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:xEm2r0CAGoktq+zd17N/yCB3Wl4= 2024-02-01T10:48:51.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:xEm2r0CAGoktq+zd17N/yCB3Wl4=', 'Content-Length': '0'} 2024-02-01T10:48:51.143 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b884e17555cb16ec-0065bb7713-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:51.143 DEBUG:boto:b'vclfcn-62obj31000falseobj32024-02-01T10:48:26.384Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:51.143 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:48:51.144 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:48:51.144 DEBUG:boto:path=/vclfcn-62/?prefix=obj3 2024-02-01T10:48:51.144 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj3 2024-02-01T10:48:51.144 DEBUG:boto:Method: GET 2024-02-01T10:48:51.144 DEBUG:boto:Path: /vclfcn-62/?prefix=obj3 2024-02-01T10:48:51.144 DEBUG:boto:Data: 2024-02-01T10:48:51.144 DEBUG:boto:Headers: {} 2024-02-01T10:48:51.145 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:51.145 DEBUG:boto:Port: 8000 2024-02-01T10:48:51.145 DEBUG:boto:Params: {} 2024-02-01T10:48:51.145 DEBUG:boto:Token: None 2024-02-01T10:48:51.145 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:51 GMT /vclfcn-62/ 2024-02-01T10:48:51.145 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:xEm2r0CAGoktq+zd17N/yCB3Wl4= 2024-02-01T10:48:51.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:xEm2r0CAGoktq+zd17N/yCB3Wl4=', 'Content-Length': '0'} 2024-02-01T10:48:51.150 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ccd1686ddcfbdc60-0065bb7713-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:51.150 DEBUG:boto:b'vclfcn-62obj31000falseobj32024-02-01T10:48:26.384Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:51.150 DEBUG:boto:path=/vclfcn-62/obj3 2024-02-01T10:48:51.151 DEBUG:boto:auth_path=/vclfcn-62/obj3 2024-02-01T10:48:51.151 DEBUG:boto:Method: HEAD 2024-02-01T10:48:51.151 DEBUG:boto:Path: /vclfcn-62/obj3 2024-02-01T10:48:51.151 DEBUG:boto:Data: 2024-02-01T10:48:51.151 DEBUG:boto:Headers: {} 2024-02-01T10:48:51.151 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:51.151 DEBUG:boto:Port: 8001 2024-02-01T10:48:51.151 DEBUG:boto:Params: {} 2024-02-01T10:48:51.151 DEBUG:boto:Token: None 2024-02-01T10:48:51.152 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:51 GMT /vclfcn-62/obj3 2024-02-01T10:48:51.152 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lYwFEWH6OsfkvomhnCz+20qniQ8= 2024-02-01T10:48:51.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lYwFEWH6OsfkvomhnCz+20qniQ8=', 'Content-Length': '0'} 2024-02-01T10:48:51.155 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002eefe1af51d4beac-0065bb7713-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:51.155 DEBUG:boto:path=/vclfcn-62/obj3 2024-02-01T10:48:51.155 DEBUG:boto:auth_path=/vclfcn-62/obj3 2024-02-01T10:48:51.156 DEBUG:boto:Method: HEAD 2024-02-01T10:48:51.156 DEBUG:boto:Path: /vclfcn-62/obj3 2024-02-01T10:48:51.156 DEBUG:boto:Data: 2024-02-01T10:48:51.156 DEBUG:boto:Headers: {} 2024-02-01T10:48:51.156 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:51.156 DEBUG:boto:Port: 8000 2024-02-01T10:48:51.156 DEBUG:boto:Params: {} 2024-02-01T10:48:51.157 DEBUG:boto:Token: None 2024-02-01T10:48:51.157 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:51 GMT /vclfcn-62/obj3 2024-02-01T10:48:51.157 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lYwFEWH6OsfkvomhnCz+20qniQ8= 2024-02-01T10:48:51.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lYwFEWH6OsfkvomhnCz+20qniQ8=', 'Content-Length': '0'} 2024-02-01T10:48:51.159 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a4c4dbf55d78aa04-0065bb7713-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:51.160 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:48:51.160 DEBUG:boto:path=/vclfcn-62/obj3 2024-02-01T10:48:51.160 DEBUG:boto:auth_path=/vclfcn-62/obj3 2024-02-01T10:48:51.160 DEBUG:boto:Method: GET 2024-02-01T10:48:51.160 DEBUG:boto:Path: /vclfcn-62/obj3 2024-02-01T10:48:51.160 DEBUG:boto:Data: 2024-02-01T10:48:51.160 DEBUG:boto:Headers: {} 2024-02-01T10:48:51.161 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:51.161 DEBUG:boto:Port: 8001 2024-02-01T10:48:51.161 DEBUG:boto:Params: {} 2024-02-01T10:48:51.161 DEBUG:boto:Token: None 2024-02-01T10:48:51.161 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:51 GMT /vclfcn-62/obj3 2024-02-01T10:48:51.161 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GYEuC23VAxpgmNPU2hbpUw4yrTc= 2024-02-01T10:48:51.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GYEuC23VAxpgmNPU2hbpUw4yrTc=', 'Content-Length': '0'} 2024-02-01T10:48:51.164 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006dfde79e38ec86bc-0065bb7713-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:51.204 DEBUG:boto:path=/vclfcn-62/obj3 2024-02-01T10:48:51.204 DEBUG:boto:auth_path=/vclfcn-62/obj3 2024-02-01T10:48:51.205 DEBUG:boto:Method: GET 2024-02-01T10:48:51.205 DEBUG:boto:Path: /vclfcn-62/obj3 2024-02-01T10:48:51.205 DEBUG:boto:Data: 2024-02-01T10:48:51.205 DEBUG:boto:Headers: {} 2024-02-01T10:48:51.206 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:51.206 DEBUG:boto:Port: 8000 2024-02-01T10:48:51.206 DEBUG:boto:Params: {} 2024-02-01T10:48:51.206 DEBUG:boto:Token: None 2024-02-01T10:48:51.207 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:51 GMT /vclfcn-62/obj3 2024-02-01T10:48:51.207 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GYEuC23VAxpgmNPU2hbpUw4yrTc= 2024-02-01T10:48:51.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GYEuC23VAxpgmNPU2hbpUw4yrTc=', 'Content-Length': '0'} 2024-02-01T10:48:51.210 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000927fc502bac1ddae-0065bb7713-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:51.256 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:48:51.256 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:48:51.256 DEBUG:boto:path=/vclfcn-62/?prefix=obj3 2024-02-01T10:48:51.257 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj3 2024-02-01T10:48:51.257 DEBUG:boto:Method: GET 2024-02-01T10:48:51.257 DEBUG:boto:Path: /vclfcn-62/?prefix=obj3 2024-02-01T10:48:51.257 DEBUG:boto:Data: 2024-02-01T10:48:51.257 DEBUG:boto:Headers: {} 2024-02-01T10:48:51.257 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:51.257 DEBUG:boto:Port: 8001 2024-02-01T10:48:51.257 DEBUG:boto:Params: {} 2024-02-01T10:48:51.257 DEBUG:boto:Token: None 2024-02-01T10:48:51.258 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:51 GMT /vclfcn-62/ 2024-02-01T10:48:51.258 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:xEm2r0CAGoktq+zd17N/yCB3Wl4= 2024-02-01T10:48:51.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:xEm2r0CAGoktq+zd17N/yCB3Wl4=', 'Content-Length': '0'} 2024-02-01T10:48:51.262 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e1b56599117062e1-0065bb7713-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:51.262 DEBUG:boto:b'vclfcn-62obj31000falseobj32024-02-01T10:48:26.384Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:51.263 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:48:51.263 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:48:51.263 DEBUG:boto:path=/vclfcn-62/?prefix=obj3 2024-02-01T10:48:51.263 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj3 2024-02-01T10:48:51.263 DEBUG:boto:Method: GET 2024-02-01T10:48:51.263 DEBUG:boto:Path: /vclfcn-62/?prefix=obj3 2024-02-01T10:48:51.263 DEBUG:boto:Data: 2024-02-01T10:48:51.263 DEBUG:boto:Headers: {} 2024-02-01T10:48:51.263 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:51.263 DEBUG:boto:Port: 8000 2024-02-01T10:48:51.263 DEBUG:boto:Params: {} 2024-02-01T10:48:51.264 DEBUG:boto:Token: None 2024-02-01T10:48:51.264 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:51 GMT /vclfcn-62/ 2024-02-01T10:48:51.264 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:xEm2r0CAGoktq+zd17N/yCB3Wl4= 2024-02-01T10:48:51.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:xEm2r0CAGoktq+zd17N/yCB3Wl4=', 'Content-Length': '0'} 2024-02-01T10:48:51.269 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000036c57129ddfbef9e-0065bb7713-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:51.316 DEBUG:boto:b'vclfcn-62obj31000falseobj32024-02-01T10:48:26.384Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:51.317 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:48:51.317 DEBUG:boto:path=/vclfcn-62/obj4 2024-02-01T10:48:51.317 DEBUG:boto:auth_path=/vclfcn-62/obj4 2024-02-01T10:48:51.317 DEBUG:boto:path=/vclfcn-62/obj4?versionId=null 2024-02-01T10:48:51.317 DEBUG:boto:auth_path=/vclfcn-62/obj4?versionId=null 2024-02-01T10:48:51.317 DEBUG:boto:Method: GET 2024-02-01T10:48:51.317 DEBUG:boto:Path: /vclfcn-62/obj4?versionId=null 2024-02-01T10:48:51.317 DEBUG:boto:Data: 2024-02-01T10:48:51.317 DEBUG:boto:Headers: {} 2024-02-01T10:48:51.318 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:51.318 DEBUG:boto:Port: 8001 2024-02-01T10:48:51.318 DEBUG:boto:Params: {} 2024-02-01T10:48:51.318 DEBUG:boto:Token: None 2024-02-01T10:48:51.318 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:51 GMT /vclfcn-62/obj4?versionId=null 2024-02-01T10:48:51.318 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GUE/YwS8SqkfIwl+gsINHke0RNg= 2024-02-01T10:48:51.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GUE/YwS8SqkfIwl+gsINHke0RNg=', 'Content-Length': '0'} 2024-02-01T10:48:51.321 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000915cab48f0723ebf-0065bb7713-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:51.364 DEBUG:boto:path=/vclfcn-62/obj4 2024-02-01T10:48:51.364 DEBUG:boto:auth_path=/vclfcn-62/obj4 2024-02-01T10:48:51.364 DEBUG:boto:path=/vclfcn-62/obj4?versionId=null 2024-02-01T10:48:51.365 DEBUG:boto:auth_path=/vclfcn-62/obj4?versionId=null 2024-02-01T10:48:51.365 DEBUG:boto:Method: GET 2024-02-01T10:48:51.365 DEBUG:boto:Path: /vclfcn-62/obj4?versionId=null 2024-02-01T10:48:51.365 DEBUG:boto:Data: 2024-02-01T10:48:51.365 DEBUG:boto:Headers: {} 2024-02-01T10:48:51.365 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:51.365 DEBUG:boto:Port: 8000 2024-02-01T10:48:51.365 DEBUG:boto:Params: {} 2024-02-01T10:48:51.365 DEBUG:boto:Token: None 2024-02-01T10:48:51.365 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:51 GMT /vclfcn-62/obj4?versionId=null 2024-02-01T10:48:51.366 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GUE/YwS8SqkfIwl+gsINHke0RNg= 2024-02-01T10:48:51.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GUE/YwS8SqkfIwl+gsINHke0RNg=', 'Content-Length': '0'} 2024-02-01T10:48:51.369 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ea239ec0e88c6917-0065bb7713-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:51.416 DEBUG:boto:path=/vclfcn-62/obj4 2024-02-01T10:48:51.416 DEBUG:boto:auth_path=/vclfcn-62/obj4 2024-02-01T10:48:51.416 DEBUG:boto:path=/vclfcn-62/obj4?versionId=null 2024-02-01T10:48:51.416 DEBUG:boto:auth_path=/vclfcn-62/obj4?versionId=null 2024-02-01T10:48:51.417 DEBUG:boto:Method: HEAD 2024-02-01T10:48:51.417 DEBUG:boto:Path: /vclfcn-62/obj4?versionId=null 2024-02-01T10:48:51.417 DEBUG:boto:Data: 2024-02-01T10:48:51.417 DEBUG:boto:Headers: {} 2024-02-01T10:48:51.417 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:51.417 DEBUG:boto:Port: 8001 2024-02-01T10:48:51.417 DEBUG:boto:Params: {} 2024-02-01T10:48:51.417 DEBUG:boto:Token: None 2024-02-01T10:48:51.418 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:51 GMT /vclfcn-62/obj4?versionId=null 2024-02-01T10:48:51.418 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wAg3xqbp+xj1Pvga3WbxRtuRNq4= 2024-02-01T10:48:51.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wAg3xqbp+xj1Pvga3WbxRtuRNq4=', 'Content-Length': '0'} 2024-02-01T10:48:51.421 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007856d073b5c56136-0065bb7713-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:51.421 DEBUG:boto:path=/vclfcn-62/obj4 2024-02-01T10:48:51.421 DEBUG:boto:auth_path=/vclfcn-62/obj4 2024-02-01T10:48:51.421 DEBUG:boto:path=/vclfcn-62/obj4?versionId=null 2024-02-01T10:48:51.421 DEBUG:boto:auth_path=/vclfcn-62/obj4?versionId=null 2024-02-01T10:48:51.422 DEBUG:boto:Method: HEAD 2024-02-01T10:48:51.422 DEBUG:boto:Path: /vclfcn-62/obj4?versionId=null 2024-02-01T10:48:51.422 DEBUG:boto:Data: 2024-02-01T10:48:51.422 DEBUG:boto:Headers: {} 2024-02-01T10:48:51.422 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:51.422 DEBUG:boto:Port: 8000 2024-02-01T10:48:51.422 DEBUG:boto:Params: {} 2024-02-01T10:48:51.422 DEBUG:boto:Token: None 2024-02-01T10:48:51.422 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:51 GMT /vclfcn-62/obj4?versionId=null 2024-02-01T10:48:51.423 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wAg3xqbp+xj1Pvga3WbxRtuRNq4= 2024-02-01T10:48:51.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wAg3xqbp+xj1Pvga3WbxRtuRNq4=', 'Content-Length': '0'} 2024-02-01T10:48:51.426 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000070d7645dd6a0d929-0065bb7713-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:51.426 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:48:51.426 DEBUG:boto:path=/vclfcn-62/obj4 2024-02-01T10:48:51.426 DEBUG:boto:auth_path=/vclfcn-62/obj4 2024-02-01T10:48:51.426 DEBUG:boto:path=/vclfcn-62/obj4?versionId=null 2024-02-01T10:48:51.427 DEBUG:boto:auth_path=/vclfcn-62/obj4?versionId=null 2024-02-01T10:48:51.427 DEBUG:boto:Method: GET 2024-02-01T10:48:51.427 DEBUG:boto:Path: /vclfcn-62/obj4?versionId=null 2024-02-01T10:48:51.427 DEBUG:boto:Data: 2024-02-01T10:48:51.427 DEBUG:boto:Headers: {} 2024-02-01T10:48:51.427 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:51.427 DEBUG:boto:Port: 8001 2024-02-01T10:48:51.427 DEBUG:boto:Params: {} 2024-02-01T10:48:51.428 DEBUG:boto:Token: None 2024-02-01T10:48:51.428 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:51 GMT /vclfcn-62/obj4?versionId=null 2024-02-01T10:48:51.428 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GUE/YwS8SqkfIwl+gsINHke0RNg= 2024-02-01T10:48:51.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GUE/YwS8SqkfIwl+gsINHke0RNg=', 'Content-Length': '0'} 2024-02-01T10:48:51.431 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000303161b7334798a0-0065bb7713-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:51.472 DEBUG:boto:path=/vclfcn-62/obj4 2024-02-01T10:48:51.472 DEBUG:boto:auth_path=/vclfcn-62/obj4 2024-02-01T10:48:51.473 DEBUG:boto:path=/vclfcn-62/obj4?versionId=null 2024-02-01T10:48:51.473 DEBUG:boto:auth_path=/vclfcn-62/obj4?versionId=null 2024-02-01T10:48:51.473 DEBUG:boto:Method: GET 2024-02-01T10:48:51.473 DEBUG:boto:Path: /vclfcn-62/obj4?versionId=null 2024-02-01T10:48:51.473 DEBUG:boto:Data: 2024-02-01T10:48:51.473 DEBUG:boto:Headers: {} 2024-02-01T10:48:51.473 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:51.473 DEBUG:boto:Port: 8000 2024-02-01T10:48:51.474 DEBUG:boto:Params: {} 2024-02-01T10:48:51.474 DEBUG:boto:Token: None 2024-02-01T10:48:51.474 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:51 GMT /vclfcn-62/obj4?versionId=null 2024-02-01T10:48:51.474 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GUE/YwS8SqkfIwl+gsINHke0RNg= 2024-02-01T10:48:51.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GUE/YwS8SqkfIwl+gsINHke0RNg=', 'Content-Length': '0'} 2024-02-01T10:48:51.477 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000002abe10a39d76f47-0065bb7713-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:51.520 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:48:51.520 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:48:51.520 DEBUG:boto:path=/vclfcn-62/?prefix=obj4 2024-02-01T10:48:51.520 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj4 2024-02-01T10:48:51.520 DEBUG:boto:Method: GET 2024-02-01T10:48:51.520 DEBUG:boto:Path: /vclfcn-62/?prefix=obj4 2024-02-01T10:48:51.521 DEBUG:boto:Data: 2024-02-01T10:48:51.521 DEBUG:boto:Headers: {} 2024-02-01T10:48:51.521 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:51.521 DEBUG:boto:Port: 8001 2024-02-01T10:48:51.521 DEBUG:boto:Params: {} 2024-02-01T10:48:51.521 DEBUG:boto:Token: None 2024-02-01T10:48:51.521 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:51 GMT /vclfcn-62/ 2024-02-01T10:48:51.521 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:xEm2r0CAGoktq+zd17N/yCB3Wl4= 2024-02-01T10:48:51.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:xEm2r0CAGoktq+zd17N/yCB3Wl4=', 'Content-Length': '0'} 2024-02-01T10:48:51.526 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6801d68701855ce-0065bb7713-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:51.526 DEBUG:boto:b'vclfcn-62obj41000falseobj42024-02-01T10:48:26.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:51.526 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:48:51.526 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:48:51.527 DEBUG:boto:path=/vclfcn-62/?prefix=obj4 2024-02-01T10:48:51.527 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj4 2024-02-01T10:48:51.527 DEBUG:boto:Method: GET 2024-02-01T10:48:51.527 DEBUG:boto:Path: /vclfcn-62/?prefix=obj4 2024-02-01T10:48:51.527 DEBUG:boto:Data: 2024-02-01T10:48:51.527 DEBUG:boto:Headers: {} 2024-02-01T10:48:51.527 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:51.527 DEBUG:boto:Port: 8000 2024-02-01T10:48:51.527 DEBUG:boto:Params: {} 2024-02-01T10:48:51.527 DEBUG:boto:Token: None 2024-02-01T10:48:51.528 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:51 GMT /vclfcn-62/ 2024-02-01T10:48:51.528 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:xEm2r0CAGoktq+zd17N/yCB3Wl4= 2024-02-01T10:48:51.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:xEm2r0CAGoktq+zd17N/yCB3Wl4=', 'Content-Length': '0'} 2024-02-01T10:48:51.532 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc1e627d6d328585-0065bb7713-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:51.576 DEBUG:boto:b'vclfcn-62obj41000falseobj42024-02-01T10:48:26.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:51.576 DEBUG:boto:path=/vclfcn-62/obj4 2024-02-01T10:48:51.576 DEBUG:boto:auth_path=/vclfcn-62/obj4 2024-02-01T10:48:51.577 DEBUG:boto:Method: HEAD 2024-02-01T10:48:51.577 DEBUG:boto:Path: /vclfcn-62/obj4 2024-02-01T10:48:51.577 DEBUG:boto:Data: 2024-02-01T10:48:51.577 DEBUG:boto:Headers: {} 2024-02-01T10:48:51.577 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:51.577 DEBUG:boto:Port: 8001 2024-02-01T10:48:51.577 DEBUG:boto:Params: {} 2024-02-01T10:48:51.577 DEBUG:boto:Token: None 2024-02-01T10:48:51.577 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:51 GMT /vclfcn-62/obj4 2024-02-01T10:48:51.577 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:M6DUoBbBafkSpUIjuiWW/N+FbQw= 2024-02-01T10:48:51.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:M6DUoBbBafkSpUIjuiWW/N+FbQw=', 'Content-Length': '0'} 2024-02-01T10:48:51.581 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000317df0eac0b09afe-0065bb7713-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:51.581 DEBUG:boto:path=/vclfcn-62/obj4 2024-02-01T10:48:51.581 DEBUG:boto:auth_path=/vclfcn-62/obj4 2024-02-01T10:48:51.581 DEBUG:boto:Method: HEAD 2024-02-01T10:48:51.581 DEBUG:boto:Path: /vclfcn-62/obj4 2024-02-01T10:48:51.581 DEBUG:boto:Data: 2024-02-01T10:48:51.581 DEBUG:boto:Headers: {} 2024-02-01T10:48:51.581 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:51.581 DEBUG:boto:Port: 8000 2024-02-01T10:48:51.582 DEBUG:boto:Params: {} 2024-02-01T10:48:51.582 DEBUG:boto:Token: None 2024-02-01T10:48:51.582 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:48:51 GMT /vclfcn-62/obj4 2024-02-01T10:48:51.582 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:M6DUoBbBafkSpUIjuiWW/N+FbQw= 2024-02-01T10:48:51.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:M6DUoBbBafkSpUIjuiWW/N+FbQw=', 'Content-Length': '0'} 2024-02-01T10:48:51.585 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004e4099f32d75301a-0065bb7713-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:51.585 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:48:51.585 DEBUG:boto:path=/vclfcn-62/obj4 2024-02-01T10:48:51.585 DEBUG:boto:auth_path=/vclfcn-62/obj4 2024-02-01T10:48:51.585 DEBUG:boto:Method: GET 2024-02-01T10:48:51.585 DEBUG:boto:Path: /vclfcn-62/obj4 2024-02-01T10:48:51.586 DEBUG:boto:Data: 2024-02-01T10:48:51.586 DEBUG:boto:Headers: {} 2024-02-01T10:48:51.586 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:51.586 DEBUG:boto:Port: 8001 2024-02-01T10:48:51.586 DEBUG:boto:Params: {} 2024-02-01T10:48:51.586 DEBUG:boto:Token: None 2024-02-01T10:48:51.586 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:51 GMT /vclfcn-62/obj4 2024-02-01T10:48:51.586 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:G18xF7DDAvM9Sk4PY3iir73aFqs= 2024-02-01T10:48:51.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:G18xF7DDAvM9Sk4PY3iir73aFqs=', 'Content-Length': '0'} 2024-02-01T10:48:51.589 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005c9f7db663701fdb-0065bb7713-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:51.632 DEBUG:boto:path=/vclfcn-62/obj4 2024-02-01T10:48:51.632 DEBUG:boto:auth_path=/vclfcn-62/obj4 2024-02-01T10:48:51.632 DEBUG:boto:Method: GET 2024-02-01T10:48:51.632 DEBUG:boto:Path: /vclfcn-62/obj4 2024-02-01T10:48:51.632 DEBUG:boto:Data: 2024-02-01T10:48:51.632 DEBUG:boto:Headers: {} 2024-02-01T10:48:51.633 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:51.633 DEBUG:boto:Port: 8000 2024-02-01T10:48:51.633 DEBUG:boto:Params: {} 2024-02-01T10:48:51.633 DEBUG:boto:Token: None 2024-02-01T10:48:51.633 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:51 GMT /vclfcn-62/obj4 2024-02-01T10:48:51.633 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:G18xF7DDAvM9Sk4PY3iir73aFqs= 2024-02-01T10:48:51.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:G18xF7DDAvM9Sk4PY3iir73aFqs=', 'Content-Length': '0'} 2024-02-01T10:48:51.636 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000020559c60a7c539db-0065bb7713-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:51.680 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:48:51.680 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:48:51.680 DEBUG:boto:path=/vclfcn-62/?prefix=obj4 2024-02-01T10:48:51.680 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj4 2024-02-01T10:48:51.681 DEBUG:boto:Method: GET 2024-02-01T10:48:51.681 DEBUG:boto:Path: /vclfcn-62/?prefix=obj4 2024-02-01T10:48:51.681 DEBUG:boto:Data: 2024-02-01T10:48:51.681 DEBUG:boto:Headers: {} 2024-02-01T10:48:51.681 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:48:51.681 DEBUG:boto:Port: 8001 2024-02-01T10:48:51.681 DEBUG:boto:Params: {} 2024-02-01T10:48:51.681 DEBUG:boto:Token: None 2024-02-01T10:48:51.681 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:51 GMT /vclfcn-62/ 2024-02-01T10:48:51.681 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:xEm2r0CAGoktq+zd17N/yCB3Wl4= 2024-02-01T10:48:51.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:xEm2r0CAGoktq+zd17N/yCB3Wl4=', 'Content-Length': '0'} 2024-02-01T10:48:51.686 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003cd9a8bff845c936-0065bb7713-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:51.686 DEBUG:boto:b'vclfcn-62obj41000falseobj42024-02-01T10:48:26.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:51.687 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:48:51.687 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:48:51.687 DEBUG:boto:path=/vclfcn-62/?prefix=obj4 2024-02-01T10:48:51.687 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj4 2024-02-01T10:48:51.687 DEBUG:boto:Method: GET 2024-02-01T10:48:51.687 DEBUG:boto:Path: /vclfcn-62/?prefix=obj4 2024-02-01T10:48:51.687 DEBUG:boto:Data: 2024-02-01T10:48:51.687 DEBUG:boto:Headers: {} 2024-02-01T10:48:51.687 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:48:51.687 DEBUG:boto:Port: 8000 2024-02-01T10:48:51.688 DEBUG:boto:Params: {} 2024-02-01T10:48:51.688 DEBUG:boto:Token: None 2024-02-01T10:48:51.688 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:48:51 GMT /vclfcn-62/ 2024-02-01T10:48:51.688 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:xEm2r0CAGoktq+zd17N/yCB3Wl4= 2024-02-01T10:48:51.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:48:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:xEm2r0CAGoktq+zd17N/yCB3Wl4=', 'Content-Length': '0'} 2024-02-01T10:48:51.692 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000062a0e266565d50e4-0065bb7713-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:48:51 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:48:51.692 DEBUG:boto:b'vclfcn-62obj41000falseobj42024-02-01T10:48:26.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:48:51.693 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-62 zones={test-zone3, test-zone2} 2024-02-01T10:48:51.693 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=vclfcn-63 2024-02-01T10:48:51.693 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:48:51.693 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:48:51.693 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket vclfcn-63 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-02-01T10:48:51.744 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:48:51.744 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:48:51.810 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:48:51.805+0000 7f87745e6980 1 waiting to reach incremental sync.. 2024-02-01T10:48:56.811 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:48:56.809+0000 7f87745e6980 1 waiting to reach incremental sync.. 2024-02-01T10:49:01.815 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:49:01.813+0000 7f87745e6980 1 bucket sync caught up with source: 2024-02-01T10:49:01.815 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [00000000001.48.6, , 00000000001.52.6, , 00000000001.72.6, , , 00000000001.52.6, , , ] 2024-02-01T10:49:01.815 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [00000000001.48.6, , 00000000001.52.6, , 00000000001.72.6, , , 00000000001.52.6, , , ] 2024-02-01T10:49:01.815 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:49:01.813+0000 7f87745e6980 0 bucket checkpoint complete 2024-02-01T10:49:01.863 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=vclfcn-63 2024-02-01T10:49:01.864 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:49:01.864 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:49:01.864 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-63 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:49:01.915 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:49:01.915 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:49:01.975 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:49:01.971+0000 7f4802b84980 1 bucket sync caught up with source: 2024-02-01T10:49:01.975 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [00000000001.48.6, , 00000000001.52.6, , 00000000001.72.6, , , 00000000001.52.6, , , ] 2024-02-01T10:49:01.975 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [00000000001.48.6, , 00000000001.52.6, , 00000000001.72.6, , , 00000000001.52.6, , , ] 2024-02-01T10:49:01.975 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:49:01.971+0000 7f4802b84980 0 bucket checkpoint complete 2024-02-01T10:49:01.998 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=vclfcn-63 2024-02-01T10:49:01.998 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-63', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:49:01.998 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-63', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:49:01.998 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-63 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:49:02.082 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:49:02.082 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:49:02.145 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:49:02.143+0000 7f9612587980 1 bucket sync caught up with source: 2024-02-01T10:49:02.145 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [00000000001.252.6, , 00000000001.332.6, , 00000000001.312.6, , , 00000000001.318.6, , , ] 2024-02-01T10:49:02.145 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [00000000001.252.6, , 00000000001.332.6, , 00000000001.312.6, , , 00000000001.318.6, , , ] 2024-02-01T10:49:02.145 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:49:02.143+0000 7f9612587980 0 bucket checkpoint complete 2024-02-01T10:49:02.168 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=vclfcn-63 2024-02-01T10:49:02.168 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-63', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:49:02.168 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-63', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:49:02.169 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-63 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:49:02.258 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:49:02.258 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:49:02.320 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:49:02.319+0000 7f369e819980 1 bucket sync caught up with source: 2024-02-01T10:49:02.320 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [00000000001.252.6, , 00000000001.332.6, , 00000000001.312.6, , , 00000000001.318.6, , , ] 2024-02-01T10:49:02.320 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [00000000001.252.6, , 00000000001.332.6, , 00000000001.312.6, , , 00000000001.318.6, , , ] 2024-02-01T10:49:02.321 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:49:02.319+0000 7f369e819980 0 bucket checkpoint complete 2024-02-01T10:49:02.344 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=vclfcn-63 2024-02-01T10:49:02.344 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-63', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:49:02.345 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-63', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:49:02.345 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-63 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:49:02.386 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:49:02.386 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:49:02.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:49:02.439+0000 7fc5cb563980 1 bucket sync caught up with source: 2024-02-01T10:49:02.443 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [00000000001.225.6, , 00000000001.199.6, , 00000000001.207.6, , , 00000000001.235.6, , , ] 2024-02-01T10:49:02.443 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [00000000001.225.6, , 00000000001.199.6, , 00000000001.207.6, , , 00000000001.235.6, , , ] 2024-02-01T10:49:02.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:49:02.439+0000 7fc5cb563980 0 bucket checkpoint complete 2024-02-01T10:49:02.467 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=vclfcn-63 2024-02-01T10:49:02.467 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-63', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:49:02.467 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-63', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:49:02.467 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket vclfcn-63 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-02-01T10:49:02.518 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:49:02.518 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:49:02.584 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:49:02.581+0000 7f56cc1a9980 1 bucket sync caught up with source: 2024-02-01T10:49:02.584 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [00000000001.225.6, , 00000000001.199.6, , 00000000001.207.6, , , 00000000001.235.6, , , ] 2024-02-01T10:49:02.584 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [00000000001.225.6, , 00000000001.199.6, , 00000000001.207.6, , , 00000000001.235.6, , , ] 2024-02-01T10:49:02.584 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:49:02.581+0000 7f56cc1a9980 0 bucket checkpoint complete 2024-02-01T10:49:02.608 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-63 zones={test-zone2, test-zone1} 2024-02-01T10:49:02.608 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:49:02.608 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:49:02.608 DEBUG:boto:Method: HEAD 2024-02-01T10:49:02.608 DEBUG:boto:Path: /vclfcn-63/ 2024-02-01T10:49:02.608 DEBUG:boto:Data: 2024-02-01T10:49:02.608 DEBUG:boto:Headers: {} 2024-02-01T10:49:02.609 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:02.609 DEBUG:boto:Port: 8000 2024-02-01T10:49:02.609 DEBUG:boto:Params: {} 2024-02-01T10:49:02.609 DEBUG:boto:Token: None 2024-02-01T10:49:02.609 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:02 GMT /vclfcn-63/ 2024-02-01T10:49:02.609 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:76HiLQo1YfuOvZxuOeqXi7VejzM= 2024-02-01T10:49:02.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:76HiLQo1YfuOvZxuOeqXi7VejzM=', 'Content-Length': '0'} 2024-02-01T10:49:02.612 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d3ba36d7fde7cc1a-0065bb771e-5350-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:49:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:02.612 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:49:02.612 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:49:02.612 DEBUG:boto:Method: HEAD 2024-02-01T10:49:02.613 DEBUG:boto:Path: /vclfcn-63/ 2024-02-01T10:49:02.613 DEBUG:boto:Data: 2024-02-01T10:49:02.613 DEBUG:boto:Headers: {} 2024-02-01T10:49:02.613 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:02.613 DEBUG:boto:Port: 8000 2024-02-01T10:49:02.613 DEBUG:boto:Params: {} 2024-02-01T10:49:02.613 DEBUG:boto:Token: None 2024-02-01T10:49:02.613 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:02 GMT /vclfcn-63/ 2024-02-01T10:49:02.613 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:76HiLQo1YfuOvZxuOeqXi7VejzM= 2024-02-01T10:49:02.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:76HiLQo1YfuOvZxuOeqXi7VejzM=', 'Content-Length': '0'} 2024-02-01T10:49:02.617 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c970e3827f10ea56-0065bb771e-5765-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:49:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:02.618 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:49:02.618 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:49:02.618 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:49:02.618 DEBUG:boto:path=/vclfcn-63/?versions&max-keys=999 2024-02-01T10:49:02.618 DEBUG:boto:auth_path=/vclfcn-63/?versions&max-keys=999 2024-02-01T10:49:02.618 DEBUG:boto:Method: GET 2024-02-01T10:49:02.618 DEBUG:boto:Path: /vclfcn-63/?versions&max-keys=999 2024-02-01T10:49:02.618 DEBUG:boto:Data: 2024-02-01T10:49:02.618 DEBUG:boto:Headers: {} 2024-02-01T10:49:02.618 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:02.619 DEBUG:boto:Port: 8000 2024-02-01T10:49:02.619 DEBUG:boto:Params: {} 2024-02-01T10:49:02.619 DEBUG:boto:Token: None 2024-02-01T10:49:02.619 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:02 GMT /vclfcn-63/?versions 2024-02-01T10:49:02.619 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:iqcW1bnkOehPHxflUTAVQ360YXo= 2024-02-01T10:49:02.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iqcW1bnkOehPHxflUTAVQ360YXo=', 'Content-Length': '0'} 2024-02-01T10:49:02.621 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007900048c7c56fa78-0065bb771e-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:02.622 DEBUG:boto:b'vclfcn-63999falseobj1nulltrue2024-02-01T10:48:26.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:48:26.507Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:48:26.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:48:26.575Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:02.622 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-02-01T10:49:02.622 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-02-01T10:49:02.622 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-02-01T10:49:02.623 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-02-01T10:49:02.623 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:49:02.623 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:49:02.623 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:49:02.623 DEBUG:boto:path=/vclfcn-63/?versions&max-keys=999 2024-02-01T10:49:02.623 DEBUG:boto:auth_path=/vclfcn-63/?versions&max-keys=999 2024-02-01T10:49:02.623 DEBUG:boto:Method: GET 2024-02-01T10:49:02.623 DEBUG:boto:Path: /vclfcn-63/?versions&max-keys=999 2024-02-01T10:49:02.624 DEBUG:boto:Data: 2024-02-01T10:49:02.624 DEBUG:boto:Headers: {} 2024-02-01T10:49:02.624 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:02.624 DEBUG:boto:Port: 8000 2024-02-01T10:49:02.624 DEBUG:boto:Params: {} 2024-02-01T10:49:02.624 DEBUG:boto:Token: None 2024-02-01T10:49:02.624 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:02 GMT /vclfcn-63/?versions 2024-02-01T10:49:02.624 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:iqcW1bnkOehPHxflUTAVQ360YXo= 2024-02-01T10:49:02.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iqcW1bnkOehPHxflUTAVQ360YXo=', 'Content-Length': '0'} 2024-02-01T10:49:02.629 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad4b6f0531a06563-0065bb771e-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:02.629 DEBUG:boto:b'vclfcn-63999falseobj1nulltrue2024-02-01T10:48:26.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:48:26.507Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:48:26.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:48:26.575Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:02.629 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-02-01T10:49:02.630 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-02-01T10:49:02.630 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-02-01T10:49:02.630 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-02-01T10:49:02.630 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:49:02.630 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:49:02.630 DEBUG:boto:path=/vclfcn-63/?versions&max-keys=999 2024-02-01T10:49:02.630 DEBUG:boto:auth_path=/vclfcn-63/?versions&max-keys=999 2024-02-01T10:49:02.630 DEBUG:boto:Method: GET 2024-02-01T10:49:02.630 DEBUG:boto:Path: /vclfcn-63/?versions&max-keys=999 2024-02-01T10:49:02.630 DEBUG:boto:Data: 2024-02-01T10:49:02.631 DEBUG:boto:Headers: {} 2024-02-01T10:49:02.631 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:02.631 DEBUG:boto:Port: 8000 2024-02-01T10:49:02.631 DEBUG:boto:Params: {} 2024-02-01T10:49:02.631 DEBUG:boto:Token: None 2024-02-01T10:49:02.631 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:02 GMT /vclfcn-63/?versions 2024-02-01T10:49:02.631 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:iqcW1bnkOehPHxflUTAVQ360YXo= 2024-02-01T10:49:02.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iqcW1bnkOehPHxflUTAVQ360YXo=', 'Content-Length': '0'} 2024-02-01T10:49:02.634 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094aa5214a13e9b75-0065bb771e-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:02.634 DEBUG:boto:b'vclfcn-63999falseobj1nulltrue2024-02-01T10:48:26.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:48:26.507Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:48:26.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:48:26.575Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:02.634 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:49:02.634 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:49:02.634 DEBUG:boto:path=/vclfcn-63/?versions&max-keys=999 2024-02-01T10:49:02.635 DEBUG:boto:auth_path=/vclfcn-63/?versions&max-keys=999 2024-02-01T10:49:02.635 DEBUG:boto:Method: GET 2024-02-01T10:49:02.635 DEBUG:boto:Path: /vclfcn-63/?versions&max-keys=999 2024-02-01T10:49:02.635 DEBUG:boto:Data: 2024-02-01T10:49:02.635 DEBUG:boto:Headers: {} 2024-02-01T10:49:02.635 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:02.635 DEBUG:boto:Port: 8000 2024-02-01T10:49:02.635 DEBUG:boto:Params: {} 2024-02-01T10:49:02.635 DEBUG:boto:Token: None 2024-02-01T10:49:02.635 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:02 GMT /vclfcn-63/?versions 2024-02-01T10:49:02.635 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:iqcW1bnkOehPHxflUTAVQ360YXo= 2024-02-01T10:49:02.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iqcW1bnkOehPHxflUTAVQ360YXo=', 'Content-Length': '0'} 2024-02-01T10:49:02.640 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004bf2ab9121f21f69-0065bb771e-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:02.640 DEBUG:boto:b'vclfcn-63999falseobj1nulltrue2024-02-01T10:48:26.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:48:26.507Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:48:26.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:48:26.575Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:02.640 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:49:02.641 DEBUG:boto:path=/vclfcn-63/obj1 2024-02-01T10:49:02.641 DEBUG:boto:auth_path=/vclfcn-63/obj1 2024-02-01T10:49:02.641 DEBUG:boto:path=/vclfcn-63/obj1?versionId=null 2024-02-01T10:49:02.641 DEBUG:boto:auth_path=/vclfcn-63/obj1?versionId=null 2024-02-01T10:49:02.641 DEBUG:boto:Method: GET 2024-02-01T10:49:02.641 DEBUG:boto:Path: /vclfcn-63/obj1?versionId=null 2024-02-01T10:49:02.641 DEBUG:boto:Data: 2024-02-01T10:49:02.641 DEBUG:boto:Headers: {} 2024-02-01T10:49:02.642 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:02.642 DEBUG:boto:Port: 8000 2024-02-01T10:49:02.642 DEBUG:boto:Params: {} 2024-02-01T10:49:02.642 DEBUG:boto:Token: None 2024-02-01T10:49:02.642 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:02 GMT /vclfcn-63/obj1?versionId=null 2024-02-01T10:49:02.643 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:odSMI24LM4o8tich1yGxXN4ZybI= 2024-02-01T10:49:02.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:odSMI24LM4o8tich1yGxXN4ZybI=', 'Content-Length': '0'} 2024-02-01T10:49:02.645 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000651ba1efdddc0dae-0065bb771e-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:02.688 DEBUG:boto:path=/vclfcn-63/obj1 2024-02-01T10:49:02.688 DEBUG:boto:auth_path=/vclfcn-63/obj1 2024-02-01T10:49:02.688 DEBUG:boto:path=/vclfcn-63/obj1?versionId=null 2024-02-01T10:49:02.688 DEBUG:boto:auth_path=/vclfcn-63/obj1?versionId=null 2024-02-01T10:49:02.689 DEBUG:boto:Method: GET 2024-02-01T10:49:02.689 DEBUG:boto:Path: /vclfcn-63/obj1?versionId=null 2024-02-01T10:49:02.689 DEBUG:boto:Data: 2024-02-01T10:49:02.689 DEBUG:boto:Headers: {} 2024-02-01T10:49:02.689 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:02.689 DEBUG:boto:Port: 8000 2024-02-01T10:49:02.689 DEBUG:boto:Params: {} 2024-02-01T10:49:02.689 DEBUG:boto:Token: None 2024-02-01T10:49:02.689 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:02 GMT /vclfcn-63/obj1?versionId=null 2024-02-01T10:49:02.690 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:odSMI24LM4o8tich1yGxXN4ZybI= 2024-02-01T10:49:02.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:odSMI24LM4o8tich1yGxXN4ZybI=', 'Content-Length': '0'} 2024-02-01T10:49:02.693 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000910815258f2aada5-0065bb771e-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:02.693 DEBUG:boto:path=/vclfcn-63/obj1 2024-02-01T10:49:02.693 DEBUG:boto:auth_path=/vclfcn-63/obj1 2024-02-01T10:49:02.693 DEBUG:boto:path=/vclfcn-63/obj1?versionId=null 2024-02-01T10:49:02.693 DEBUG:boto:auth_path=/vclfcn-63/obj1?versionId=null 2024-02-01T10:49:02.694 DEBUG:boto:Method: HEAD 2024-02-01T10:49:02.694 DEBUG:boto:Path: /vclfcn-63/obj1?versionId=null 2024-02-01T10:49:02.694 DEBUG:boto:Data: 2024-02-01T10:49:02.694 DEBUG:boto:Headers: {} 2024-02-01T10:49:02.694 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:02.694 DEBUG:boto:Port: 8000 2024-02-01T10:49:02.694 DEBUG:boto:Params: {} 2024-02-01T10:49:02.694 DEBUG:boto:Token: None 2024-02-01T10:49:02.694 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:02 GMT /vclfcn-63/obj1?versionId=null 2024-02-01T10:49:02.694 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:TqqDsX+9KgfsXtCxPmytH8e++Tg= 2024-02-01T10:49:02.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:TqqDsX+9KgfsXtCxPmytH8e++Tg=', 'Content-Length': '0'} 2024-02-01T10:49:02.698 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006750fc5a4e8cb23b-0065bb771e-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:02.698 DEBUG:boto:path=/vclfcn-63/obj1 2024-02-01T10:49:02.698 DEBUG:boto:auth_path=/vclfcn-63/obj1 2024-02-01T10:49:02.698 DEBUG:boto:path=/vclfcn-63/obj1?versionId=null 2024-02-01T10:49:02.698 DEBUG:boto:auth_path=/vclfcn-63/obj1?versionId=null 2024-02-01T10:49:02.698 DEBUG:boto:Method: HEAD 2024-02-01T10:49:02.699 DEBUG:boto:Path: /vclfcn-63/obj1?versionId=null 2024-02-01T10:49:02.699 DEBUG:boto:Data: 2024-02-01T10:49:02.699 DEBUG:boto:Headers: {} 2024-02-01T10:49:02.699 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:02.699 DEBUG:boto:Port: 8000 2024-02-01T10:49:02.699 DEBUG:boto:Params: {} 2024-02-01T10:49:02.699 DEBUG:boto:Token: None 2024-02-01T10:49:02.699 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:02 GMT /vclfcn-63/obj1?versionId=null 2024-02-01T10:49:02.699 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:TqqDsX+9KgfsXtCxPmytH8e++Tg= 2024-02-01T10:49:02.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:TqqDsX+9KgfsXtCxPmytH8e++Tg=', 'Content-Length': '0'} 2024-02-01T10:49:02.702 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cc2dd1def7a72962-0065bb771e-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:02.703 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:49:02.703 DEBUG:boto:path=/vclfcn-63/obj1 2024-02-01T10:49:02.703 DEBUG:boto:auth_path=/vclfcn-63/obj1 2024-02-01T10:49:02.703 DEBUG:boto:path=/vclfcn-63/obj1?versionId=null 2024-02-01T10:49:02.703 DEBUG:boto:auth_path=/vclfcn-63/obj1?versionId=null 2024-02-01T10:49:02.703 DEBUG:boto:Method: GET 2024-02-01T10:49:02.703 DEBUG:boto:Path: /vclfcn-63/obj1?versionId=null 2024-02-01T10:49:02.704 DEBUG:boto:Data: 2024-02-01T10:49:02.704 DEBUG:boto:Headers: {} 2024-02-01T10:49:02.704 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:02.704 DEBUG:boto:Port: 8000 2024-02-01T10:49:02.704 DEBUG:boto:Params: {} 2024-02-01T10:49:02.704 DEBUG:boto:Token: None 2024-02-01T10:49:02.704 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:02 GMT /vclfcn-63/obj1?versionId=null 2024-02-01T10:49:02.704 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:odSMI24LM4o8tich1yGxXN4ZybI= 2024-02-01T10:49:02.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:odSMI24LM4o8tich1yGxXN4ZybI=', 'Content-Length': '0'} 2024-02-01T10:49:02.708 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a2a2f497e646ee4d-0065bb771e-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:02.708 DEBUG:boto:path=/vclfcn-63/obj1 2024-02-01T10:49:02.709 DEBUG:boto:auth_path=/vclfcn-63/obj1 2024-02-01T10:49:02.709 DEBUG:boto:path=/vclfcn-63/obj1?versionId=null 2024-02-01T10:49:02.709 DEBUG:boto:auth_path=/vclfcn-63/obj1?versionId=null 2024-02-01T10:49:02.709 DEBUG:boto:Method: GET 2024-02-01T10:49:02.709 DEBUG:boto:Path: /vclfcn-63/obj1?versionId=null 2024-02-01T10:49:02.709 DEBUG:boto:Data: 2024-02-01T10:49:02.709 DEBUG:boto:Headers: {} 2024-02-01T10:49:02.709 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:02.709 DEBUG:boto:Port: 8000 2024-02-01T10:49:02.710 DEBUG:boto:Params: {} 2024-02-01T10:49:02.710 DEBUG:boto:Token: None 2024-02-01T10:49:02.710 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:02 GMT /vclfcn-63/obj1?versionId=null 2024-02-01T10:49:02.710 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:odSMI24LM4o8tich1yGxXN4ZybI= 2024-02-01T10:49:02.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:odSMI24LM4o8tich1yGxXN4ZybI=', 'Content-Length': '0'} 2024-02-01T10:49:02.714 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004d6075a63a1749f2-0065bb771e-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:02.756 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:49:02.756 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:49:02.756 DEBUG:boto:path=/vclfcn-63/?prefix=obj1 2024-02-01T10:49:02.757 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj1 2024-02-01T10:49:02.757 DEBUG:boto:Method: GET 2024-02-01T10:49:02.757 DEBUG:boto:Path: /vclfcn-63/?prefix=obj1 2024-02-01T10:49:02.757 DEBUG:boto:Data: 2024-02-01T10:49:02.757 DEBUG:boto:Headers: {} 2024-02-01T10:49:02.757 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:02.757 DEBUG:boto:Port: 8000 2024-02-01T10:49:02.757 DEBUG:boto:Params: {} 2024-02-01T10:49:02.757 DEBUG:boto:Token: None 2024-02-01T10:49:02.758 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:02 GMT /vclfcn-63/ 2024-02-01T10:49:02.758 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:U5JlTvL+TOl19qxCXQrspNLKS04= 2024-02-01T10:49:02.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:U5JlTvL+TOl19qxCXQrspNLKS04=', 'Content-Length': '0'} 2024-02-01T10:49:02.762 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000560d60b7385d3a68-0065bb771e-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:02.763 DEBUG:boto:b'vclfcn-63obj11000falseobj12024-02-01T10:48:26.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:02.763 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:49:02.763 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:49:02.763 DEBUG:boto:path=/vclfcn-63/?prefix=obj1 2024-02-01T10:49:02.763 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj1 2024-02-01T10:49:02.764 DEBUG:boto:Method: GET 2024-02-01T10:49:02.764 DEBUG:boto:Path: /vclfcn-63/?prefix=obj1 2024-02-01T10:49:02.764 DEBUG:boto:Data: 2024-02-01T10:49:02.764 DEBUG:boto:Headers: {} 2024-02-01T10:49:02.764 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:02.764 DEBUG:boto:Port: 8000 2024-02-01T10:49:02.764 DEBUG:boto:Params: {} 2024-02-01T10:49:02.764 DEBUG:boto:Token: None 2024-02-01T10:49:02.764 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:02 GMT /vclfcn-63/ 2024-02-01T10:49:02.765 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:U5JlTvL+TOl19qxCXQrspNLKS04= 2024-02-01T10:49:02.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:U5JlTvL+TOl19qxCXQrspNLKS04=', 'Content-Length': '0'} 2024-02-01T10:49:02.769 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8848945a8925149-0065bb771e-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:02.769 DEBUG:boto:b'vclfcn-63obj11000falseobj12024-02-01T10:48:26.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:02.769 DEBUG:boto:path=/vclfcn-63/obj1 2024-02-01T10:49:02.770 DEBUG:boto:auth_path=/vclfcn-63/obj1 2024-02-01T10:49:02.770 DEBUG:boto:Method: HEAD 2024-02-01T10:49:02.770 DEBUG:boto:Path: /vclfcn-63/obj1 2024-02-01T10:49:02.770 DEBUG:boto:Data: 2024-02-01T10:49:02.770 DEBUG:boto:Headers: {} 2024-02-01T10:49:02.770 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:02.770 DEBUG:boto:Port: 8000 2024-02-01T10:49:02.770 DEBUG:boto:Params: {} 2024-02-01T10:49:02.770 DEBUG:boto:Token: None 2024-02-01T10:49:02.770 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:02 GMT /vclfcn-63/obj1 2024-02-01T10:49:02.771 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wT8VZelFGOhVAb6mTOWoVIdFJhk= 2024-02-01T10:49:02.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wT8VZelFGOhVAb6mTOWoVIdFJhk=', 'Content-Length': '0'} 2024-02-01T10:49:02.773 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000041a4d2c3517fff4b-0065bb771e-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:02.774 DEBUG:boto:path=/vclfcn-63/obj1 2024-02-01T10:49:02.774 DEBUG:boto:auth_path=/vclfcn-63/obj1 2024-02-01T10:49:02.774 DEBUG:boto:Method: HEAD 2024-02-01T10:49:02.774 DEBUG:boto:Path: /vclfcn-63/obj1 2024-02-01T10:49:02.774 DEBUG:boto:Data: 2024-02-01T10:49:02.774 DEBUG:boto:Headers: {} 2024-02-01T10:49:02.774 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:02.774 DEBUG:boto:Port: 8000 2024-02-01T10:49:02.774 DEBUG:boto:Params: {} 2024-02-01T10:49:02.775 DEBUG:boto:Token: None 2024-02-01T10:49:02.775 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:02 GMT /vclfcn-63/obj1 2024-02-01T10:49:02.775 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wT8VZelFGOhVAb6mTOWoVIdFJhk= 2024-02-01T10:49:02.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wT8VZelFGOhVAb6mTOWoVIdFJhk=', 'Content-Length': '0'} 2024-02-01T10:49:02.778 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b2967b8f20bca8b5-0065bb771e-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:02.779 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:49:02.779 DEBUG:boto:path=/vclfcn-63/obj1 2024-02-01T10:49:02.779 DEBUG:boto:auth_path=/vclfcn-63/obj1 2024-02-01T10:49:02.779 DEBUG:boto:Method: GET 2024-02-01T10:49:02.779 DEBUG:boto:Path: /vclfcn-63/obj1 2024-02-01T10:49:02.779 DEBUG:boto:Data: 2024-02-01T10:49:02.779 DEBUG:boto:Headers: {} 2024-02-01T10:49:02.779 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:02.779 DEBUG:boto:Port: 8000 2024-02-01T10:49:02.779 DEBUG:boto:Params: {} 2024-02-01T10:49:02.780 DEBUG:boto:Token: None 2024-02-01T10:49:02.780 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:02 GMT /vclfcn-63/obj1 2024-02-01T10:49:02.780 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wtfiSHNqr3H/cf77N0J+067uJME= 2024-02-01T10:49:02.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wtfiSHNqr3H/cf77N0J+067uJME=', 'Content-Length': '0'} 2024-02-01T10:49:02.783 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b225f100694daf91-0065bb771e-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:02.824 DEBUG:boto:path=/vclfcn-63/obj1 2024-02-01T10:49:02.824 DEBUG:boto:auth_path=/vclfcn-63/obj1 2024-02-01T10:49:02.824 DEBUG:boto:Method: GET 2024-02-01T10:49:02.825 DEBUG:boto:Path: /vclfcn-63/obj1 2024-02-01T10:49:02.825 DEBUG:boto:Data: 2024-02-01T10:49:02.825 DEBUG:boto:Headers: {} 2024-02-01T10:49:02.825 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:02.825 DEBUG:boto:Port: 8000 2024-02-01T10:49:02.825 DEBUG:boto:Params: {} 2024-02-01T10:49:02.825 DEBUG:boto:Token: None 2024-02-01T10:49:02.825 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:02 GMT /vclfcn-63/obj1 2024-02-01T10:49:02.825 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wtfiSHNqr3H/cf77N0J+067uJME= 2024-02-01T10:49:02.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wtfiSHNqr3H/cf77N0J+067uJME=', 'Content-Length': '0'} 2024-02-01T10:49:02.829 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000057cdf2c2c4c35607-0065bb771e-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:02.876 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:49:02.876 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:49:02.876 DEBUG:boto:path=/vclfcn-63/?prefix=obj1 2024-02-01T10:49:02.876 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj1 2024-02-01T10:49:02.877 DEBUG:boto:Method: GET 2024-02-01T10:49:02.877 DEBUG:boto:Path: /vclfcn-63/?prefix=obj1 2024-02-01T10:49:02.877 DEBUG:boto:Data: 2024-02-01T10:49:02.877 DEBUG:boto:Headers: {} 2024-02-01T10:49:02.877 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:02.877 DEBUG:boto:Port: 8000 2024-02-01T10:49:02.877 DEBUG:boto:Params: {} 2024-02-01T10:49:02.877 DEBUG:boto:Token: None 2024-02-01T10:49:02.877 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:02 GMT /vclfcn-63/ 2024-02-01T10:49:02.878 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:U5JlTvL+TOl19qxCXQrspNLKS04= 2024-02-01T10:49:02.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:U5JlTvL+TOl19qxCXQrspNLKS04=', 'Content-Length': '0'} 2024-02-01T10:49:02.882 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf8a5eac2c747506-0065bb771e-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:02.882 DEBUG:boto:b'vclfcn-63obj11000falseobj12024-02-01T10:48:26.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:02.883 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:49:02.883 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:49:02.883 DEBUG:boto:path=/vclfcn-63/?prefix=obj1 2024-02-01T10:49:02.883 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj1 2024-02-01T10:49:02.883 DEBUG:boto:Method: GET 2024-02-01T10:49:02.884 DEBUG:boto:Path: /vclfcn-63/?prefix=obj1 2024-02-01T10:49:02.884 DEBUG:boto:Data: 2024-02-01T10:49:02.884 DEBUG:boto:Headers: {} 2024-02-01T10:49:02.884 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:02.884 DEBUG:boto:Port: 8000 2024-02-01T10:49:02.884 DEBUG:boto:Params: {} 2024-02-01T10:49:02.884 DEBUG:boto:Token: None 2024-02-01T10:49:02.885 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:02 GMT /vclfcn-63/ 2024-02-01T10:49:02.885 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:U5JlTvL+TOl19qxCXQrspNLKS04= 2024-02-01T10:49:02.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:U5JlTvL+TOl19qxCXQrspNLKS04=', 'Content-Length': '0'} 2024-02-01T10:49:02.889 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de08e59fbc16b47d-0065bb771e-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:02.932 DEBUG:boto:b'vclfcn-63obj11000falseobj12024-02-01T10:48:26.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:02.933 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:49:02.933 DEBUG:boto:path=/vclfcn-63/obj2 2024-02-01T10:49:02.933 DEBUG:boto:auth_path=/vclfcn-63/obj2 2024-02-01T10:49:02.933 DEBUG:boto:path=/vclfcn-63/obj2?versionId=null 2024-02-01T10:49:02.933 DEBUG:boto:auth_path=/vclfcn-63/obj2?versionId=null 2024-02-01T10:49:02.933 DEBUG:boto:Method: GET 2024-02-01T10:49:02.933 DEBUG:boto:Path: /vclfcn-63/obj2?versionId=null 2024-02-01T10:49:02.933 DEBUG:boto:Data: 2024-02-01T10:49:02.934 DEBUG:boto:Headers: {} 2024-02-01T10:49:02.934 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:02.934 DEBUG:boto:Port: 8000 2024-02-01T10:49:02.934 DEBUG:boto:Params: {} 2024-02-01T10:49:02.934 DEBUG:boto:Token: None 2024-02-01T10:49:02.934 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:02 GMT /vclfcn-63/obj2?versionId=null 2024-02-01T10:49:02.934 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ir3G7envEYZyglfdUlEQg3aAC+k= 2024-02-01T10:49:02.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ir3G7envEYZyglfdUlEQg3aAC+k=', 'Content-Length': '0'} 2024-02-01T10:49:02.937 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000075e992ce1ba59abc-0065bb771e-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:02.937 DEBUG:boto:path=/vclfcn-63/obj2 2024-02-01T10:49:02.938 DEBUG:boto:auth_path=/vclfcn-63/obj2 2024-02-01T10:49:02.938 DEBUG:boto:path=/vclfcn-63/obj2?versionId=null 2024-02-01T10:49:02.938 DEBUG:boto:auth_path=/vclfcn-63/obj2?versionId=null 2024-02-01T10:49:02.938 DEBUG:boto:Method: GET 2024-02-01T10:49:02.938 DEBUG:boto:Path: /vclfcn-63/obj2?versionId=null 2024-02-01T10:49:02.938 DEBUG:boto:Data: 2024-02-01T10:49:02.938 DEBUG:boto:Headers: {} 2024-02-01T10:49:02.938 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:02.938 DEBUG:boto:Port: 8000 2024-02-01T10:49:02.938 DEBUG:boto:Params: {} 2024-02-01T10:49:02.938 DEBUG:boto:Token: None 2024-02-01T10:49:02.939 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:02 GMT /vclfcn-63/obj2?versionId=null 2024-02-01T10:49:02.939 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ir3G7envEYZyglfdUlEQg3aAC+k= 2024-02-01T10:49:02.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ir3G7envEYZyglfdUlEQg3aAC+k=', 'Content-Length': '0'} 2024-02-01T10:49:02.941 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000de3121d05e84ec2b-0065bb771e-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:02.988 DEBUG:boto:path=/vclfcn-63/obj2 2024-02-01T10:49:02.988 DEBUG:boto:auth_path=/vclfcn-63/obj2 2024-02-01T10:49:02.988 DEBUG:boto:path=/vclfcn-63/obj2?versionId=null 2024-02-01T10:49:02.988 DEBUG:boto:auth_path=/vclfcn-63/obj2?versionId=null 2024-02-01T10:49:02.989 DEBUG:boto:Method: HEAD 2024-02-01T10:49:02.989 DEBUG:boto:Path: /vclfcn-63/obj2?versionId=null 2024-02-01T10:49:02.989 DEBUG:boto:Data: 2024-02-01T10:49:02.989 DEBUG:boto:Headers: {} 2024-02-01T10:49:02.989 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:02.989 DEBUG:boto:Port: 8000 2024-02-01T10:49:02.989 DEBUG:boto:Params: {} 2024-02-01T10:49:02.989 DEBUG:boto:Token: None 2024-02-01T10:49:02.989 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:02 GMT /vclfcn-63/obj2?versionId=null 2024-02-01T10:49:02.990 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FR7yuifgno1XYFhhAyY0Ie5C7Es= 2024-02-01T10:49:02.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FR7yuifgno1XYFhhAyY0Ie5C7Es=', 'Content-Length': '0'} 2024-02-01T10:49:02.992 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001493dd929d205a9a-0065bb771e-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:02.993 DEBUG:boto:path=/vclfcn-63/obj2 2024-02-01T10:49:02.993 DEBUG:boto:auth_path=/vclfcn-63/obj2 2024-02-01T10:49:02.993 DEBUG:boto:path=/vclfcn-63/obj2?versionId=null 2024-02-01T10:49:02.993 DEBUG:boto:auth_path=/vclfcn-63/obj2?versionId=null 2024-02-01T10:49:02.993 DEBUG:boto:Method: HEAD 2024-02-01T10:49:02.993 DEBUG:boto:Path: /vclfcn-63/obj2?versionId=null 2024-02-01T10:49:02.993 DEBUG:boto:Data: 2024-02-01T10:49:02.993 DEBUG:boto:Headers: {} 2024-02-01T10:49:02.993 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:02.993 DEBUG:boto:Port: 8000 2024-02-01T10:49:02.993 DEBUG:boto:Params: {} 2024-02-01T10:49:02.994 DEBUG:boto:Token: None 2024-02-01T10:49:02.994 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:02 GMT /vclfcn-63/obj2?versionId=null 2024-02-01T10:49:02.994 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FR7yuifgno1XYFhhAyY0Ie5C7Es= 2024-02-01T10:49:02.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FR7yuifgno1XYFhhAyY0Ie5C7Es=', 'Content-Length': '0'} 2024-02-01T10:49:02.997 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000088afcf3679dd2100-0065bb771e-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:02.997 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:49:02.997 DEBUG:boto:path=/vclfcn-63/obj2 2024-02-01T10:49:02.997 DEBUG:boto:auth_path=/vclfcn-63/obj2 2024-02-01T10:49:02.997 DEBUG:boto:path=/vclfcn-63/obj2?versionId=null 2024-02-01T10:49:02.997 DEBUG:boto:auth_path=/vclfcn-63/obj2?versionId=null 2024-02-01T10:49:02.997 DEBUG:boto:Method: GET 2024-02-01T10:49:02.997 DEBUG:boto:Path: /vclfcn-63/obj2?versionId=null 2024-02-01T10:49:02.997 DEBUG:boto:Data: 2024-02-01T10:49:02.998 DEBUG:boto:Headers: {} 2024-02-01T10:49:02.998 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:02.998 DEBUG:boto:Port: 8000 2024-02-01T10:49:02.998 DEBUG:boto:Params: {} 2024-02-01T10:49:02.998 DEBUG:boto:Token: None 2024-02-01T10:49:02.998 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:02 GMT /vclfcn-63/obj2?versionId=null 2024-02-01T10:49:02.998 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ir3G7envEYZyglfdUlEQg3aAC+k= 2024-02-01T10:49:02.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ir3G7envEYZyglfdUlEQg3aAC+k=', 'Content-Length': '0'} 2024-02-01T10:49:03.001 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004849891406682c86-0065bb771e-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:03.001 DEBUG:boto:path=/vclfcn-63/obj2 2024-02-01T10:49:03.001 DEBUG:boto:auth_path=/vclfcn-63/obj2 2024-02-01T10:49:03.001 DEBUG:boto:path=/vclfcn-63/obj2?versionId=null 2024-02-01T10:49:03.001 DEBUG:boto:auth_path=/vclfcn-63/obj2?versionId=null 2024-02-01T10:49:03.001 DEBUG:boto:Method: GET 2024-02-01T10:49:03.002 DEBUG:boto:Path: /vclfcn-63/obj2?versionId=null 2024-02-01T10:49:03.002 DEBUG:boto:Data: 2024-02-01T10:49:03.002 DEBUG:boto:Headers: {} 2024-02-01T10:49:03.002 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:03.002 DEBUG:boto:Port: 8000 2024-02-01T10:49:03.002 DEBUG:boto:Params: {} 2024-02-01T10:49:03.002 DEBUG:boto:Token: None 2024-02-01T10:49:03.002 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:03 GMT /vclfcn-63/obj2?versionId=null 2024-02-01T10:49:03.002 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Id16SzEtk1Z/kzNfqugG0YY0XfM= 2024-02-01T10:49:03.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Id16SzEtk1Z/kzNfqugG0YY0XfM=', 'Content-Length': '0'} 2024-02-01T10:49:03.005 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d32906b4884db6c2-0065bb771f-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:03.048 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:49:03.048 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:49:03.048 DEBUG:boto:path=/vclfcn-63/?prefix=obj2 2024-02-01T10:49:03.048 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj2 2024-02-01T10:49:03.049 DEBUG:boto:Method: GET 2024-02-01T10:49:03.049 DEBUG:boto:Path: /vclfcn-63/?prefix=obj2 2024-02-01T10:49:03.049 DEBUG:boto:Data: 2024-02-01T10:49:03.049 DEBUG:boto:Headers: {} 2024-02-01T10:49:03.049 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:03.049 DEBUG:boto:Port: 8000 2024-02-01T10:49:03.049 DEBUG:boto:Params: {} 2024-02-01T10:49:03.049 DEBUG:boto:Token: None 2024-02-01T10:49:03.049 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:03 GMT /vclfcn-63/ 2024-02-01T10:49:03.049 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:uLqgyey3yk7v8jzNLr/8DxsAGkE= 2024-02-01T10:49:03.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:uLqgyey3yk7v8jzNLr/8DxsAGkE=', 'Content-Length': '0'} 2024-02-01T10:49:03.053 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e90d21df1624a8a5-0065bb771f-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:03.054 DEBUG:boto:b'vclfcn-63obj21000falseobj22024-02-01T10:48:26.507Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:03.054 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:49:03.054 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:49:03.054 DEBUG:boto:path=/vclfcn-63/?prefix=obj2 2024-02-01T10:49:03.054 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj2 2024-02-01T10:49:03.054 DEBUG:boto:Method: GET 2024-02-01T10:49:03.055 DEBUG:boto:Path: /vclfcn-63/?prefix=obj2 2024-02-01T10:49:03.055 DEBUG:boto:Data: 2024-02-01T10:49:03.055 DEBUG:boto:Headers: {} 2024-02-01T10:49:03.055 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:03.055 DEBUG:boto:Port: 8000 2024-02-01T10:49:03.055 DEBUG:boto:Params: {} 2024-02-01T10:49:03.055 DEBUG:boto:Token: None 2024-02-01T10:49:03.055 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:03 GMT /vclfcn-63/ 2024-02-01T10:49:03.055 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:uLqgyey3yk7v8jzNLr/8DxsAGkE= 2024-02-01T10:49:03.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:uLqgyey3yk7v8jzNLr/8DxsAGkE=', 'Content-Length': '0'} 2024-02-01T10:49:03.059 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035b3e2e3131a78ff-0065bb771f-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:03.060 DEBUG:boto:b'vclfcn-63obj21000falseobj22024-02-01T10:48:26.507Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:03.060 DEBUG:boto:path=/vclfcn-63/obj2 2024-02-01T10:49:03.060 DEBUG:boto:auth_path=/vclfcn-63/obj2 2024-02-01T10:49:03.060 DEBUG:boto:Method: HEAD 2024-02-01T10:49:03.060 DEBUG:boto:Path: /vclfcn-63/obj2 2024-02-01T10:49:03.060 DEBUG:boto:Data: 2024-02-01T10:49:03.060 DEBUG:boto:Headers: {} 2024-02-01T10:49:03.061 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:03.061 DEBUG:boto:Port: 8000 2024-02-01T10:49:03.061 DEBUG:boto:Params: {} 2024-02-01T10:49:03.061 DEBUG:boto:Token: None 2024-02-01T10:49:03.061 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:03 GMT /vclfcn-63/obj2 2024-02-01T10:49:03.061 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:d/JSRXDXzNABQUm/yAiDRBZtw10= 2024-02-01T10:49:03.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:d/JSRXDXzNABQUm/yAiDRBZtw10=', 'Content-Length': '0'} 2024-02-01T10:49:03.064 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008b5792db9779206b-0065bb771f-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:03.064 DEBUG:boto:path=/vclfcn-63/obj2 2024-02-01T10:49:03.064 DEBUG:boto:auth_path=/vclfcn-63/obj2 2024-02-01T10:49:03.065 DEBUG:boto:Method: HEAD 2024-02-01T10:49:03.065 DEBUG:boto:Path: /vclfcn-63/obj2 2024-02-01T10:49:03.065 DEBUG:boto:Data: 2024-02-01T10:49:03.065 DEBUG:boto:Headers: {} 2024-02-01T10:49:03.065 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:03.065 DEBUG:boto:Port: 8000 2024-02-01T10:49:03.065 DEBUG:boto:Params: {} 2024-02-01T10:49:03.065 DEBUG:boto:Token: None 2024-02-01T10:49:03.065 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:03 GMT /vclfcn-63/obj2 2024-02-01T10:49:03.065 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:d/JSRXDXzNABQUm/yAiDRBZtw10= 2024-02-01T10:49:03.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:d/JSRXDXzNABQUm/yAiDRBZtw10=', 'Content-Length': '0'} 2024-02-01T10:49:03.068 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a87a21823275ef94-0065bb771f-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:03.069 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:49:03.069 DEBUG:boto:path=/vclfcn-63/obj2 2024-02-01T10:49:03.069 DEBUG:boto:auth_path=/vclfcn-63/obj2 2024-02-01T10:49:03.069 DEBUG:boto:Method: GET 2024-02-01T10:49:03.069 DEBUG:boto:Path: /vclfcn-63/obj2 2024-02-01T10:49:03.069 DEBUG:boto:Data: 2024-02-01T10:49:03.069 DEBUG:boto:Headers: {} 2024-02-01T10:49:03.069 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:03.069 DEBUG:boto:Port: 8000 2024-02-01T10:49:03.069 DEBUG:boto:Params: {} 2024-02-01T10:49:03.070 DEBUG:boto:Token: None 2024-02-01T10:49:03.070 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:03 GMT /vclfcn-63/obj2 2024-02-01T10:49:03.070 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/WcH4a9r/35DmdVQ24zrp0jNVOw= 2024-02-01T10:49:03.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/WcH4a9r/35DmdVQ24zrp0jNVOw=', 'Content-Length': '0'} 2024-02-01T10:49:03.072 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000edd5afe398ca085a-0065bb771f-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:03.116 DEBUG:boto:path=/vclfcn-63/obj2 2024-02-01T10:49:03.116 DEBUG:boto:auth_path=/vclfcn-63/obj2 2024-02-01T10:49:03.116 DEBUG:boto:Method: GET 2024-02-01T10:49:03.116 DEBUG:boto:Path: /vclfcn-63/obj2 2024-02-01T10:49:03.117 DEBUG:boto:Data: 2024-02-01T10:49:03.117 DEBUG:boto:Headers: {} 2024-02-01T10:49:03.117 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:03.117 DEBUG:boto:Port: 8000 2024-02-01T10:49:03.117 DEBUG:boto:Params: {} 2024-02-01T10:49:03.117 DEBUG:boto:Token: None 2024-02-01T10:49:03.117 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:03 GMT /vclfcn-63/obj2 2024-02-01T10:49:03.117 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/WcH4a9r/35DmdVQ24zrp0jNVOw= 2024-02-01T10:49:03.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/WcH4a9r/35DmdVQ24zrp0jNVOw=', 'Content-Length': '0'} 2024-02-01T10:49:03.120 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d8ab4e464a96229d-0065bb771f-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:03.168 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:49:03.168 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:49:03.168 DEBUG:boto:path=/vclfcn-63/?prefix=obj2 2024-02-01T10:49:03.168 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj2 2024-02-01T10:49:03.169 DEBUG:boto:Method: GET 2024-02-01T10:49:03.169 DEBUG:boto:Path: /vclfcn-63/?prefix=obj2 2024-02-01T10:49:03.169 DEBUG:boto:Data: 2024-02-01T10:49:03.169 DEBUG:boto:Headers: {} 2024-02-01T10:49:03.169 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:03.169 DEBUG:boto:Port: 8000 2024-02-01T10:49:03.169 DEBUG:boto:Params: {} 2024-02-01T10:49:03.170 DEBUG:boto:Token: None 2024-02-01T10:49:03.170 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:03 GMT /vclfcn-63/ 2024-02-01T10:49:03.170 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:uLqgyey3yk7v8jzNLr/8DxsAGkE= 2024-02-01T10:49:03.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:uLqgyey3yk7v8jzNLr/8DxsAGkE=', 'Content-Length': '0'} 2024-02-01T10:49:03.174 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002a8e8f6b409bf21-0065bb771f-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:03.175 DEBUG:boto:b'vclfcn-63obj21000falseobj22024-02-01T10:48:26.507Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:03.175 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:49:03.175 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:49:03.175 DEBUG:boto:path=/vclfcn-63/?prefix=obj2 2024-02-01T10:49:03.175 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj2 2024-02-01T10:49:03.175 DEBUG:boto:Method: GET 2024-02-01T10:49:03.176 DEBUG:boto:Path: /vclfcn-63/?prefix=obj2 2024-02-01T10:49:03.176 DEBUG:boto:Data: 2024-02-01T10:49:03.176 DEBUG:boto:Headers: {} 2024-02-01T10:49:03.176 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:03.176 DEBUG:boto:Port: 8000 2024-02-01T10:49:03.176 DEBUG:boto:Params: {} 2024-02-01T10:49:03.176 DEBUG:boto:Token: None 2024-02-01T10:49:03.176 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:03 GMT /vclfcn-63/ 2024-02-01T10:49:03.176 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:uLqgyey3yk7v8jzNLr/8DxsAGkE= 2024-02-01T10:49:03.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:uLqgyey3yk7v8jzNLr/8DxsAGkE=', 'Content-Length': '0'} 2024-02-01T10:49:03.180 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f5d34d522fa95fa-0065bb771f-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:03.224 DEBUG:boto:b'vclfcn-63obj21000falseobj22024-02-01T10:48:26.507Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:03.224 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:49:03.225 DEBUG:boto:path=/vclfcn-63/obj3 2024-02-01T10:49:03.225 DEBUG:boto:auth_path=/vclfcn-63/obj3 2024-02-01T10:49:03.225 DEBUG:boto:path=/vclfcn-63/obj3?versionId=null 2024-02-01T10:49:03.225 DEBUG:boto:auth_path=/vclfcn-63/obj3?versionId=null 2024-02-01T10:49:03.225 DEBUG:boto:Method: GET 2024-02-01T10:49:03.225 DEBUG:boto:Path: /vclfcn-63/obj3?versionId=null 2024-02-01T10:49:03.225 DEBUG:boto:Data: 2024-02-01T10:49:03.225 DEBUG:boto:Headers: {} 2024-02-01T10:49:03.225 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:03.225 DEBUG:boto:Port: 8000 2024-02-01T10:49:03.225 DEBUG:boto:Params: {} 2024-02-01T10:49:03.226 DEBUG:boto:Token: None 2024-02-01T10:49:03.226 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:03 GMT /vclfcn-63/obj3?versionId=null 2024-02-01T10:49:03.226 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fCtPs5ZBja8dW8Op7tFh4PJ4i0A= 2024-02-01T10:49:03.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fCtPs5ZBja8dW8Op7tFh4PJ4i0A=', 'Content-Length': '0'} 2024-02-01T10:49:03.229 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fb0c63ec68babb82-0065bb771f-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:03.229 DEBUG:boto:path=/vclfcn-63/obj3 2024-02-01T10:49:03.229 DEBUG:boto:auth_path=/vclfcn-63/obj3 2024-02-01T10:49:03.229 DEBUG:boto:path=/vclfcn-63/obj3?versionId=null 2024-02-01T10:49:03.230 DEBUG:boto:auth_path=/vclfcn-63/obj3?versionId=null 2024-02-01T10:49:03.230 DEBUG:boto:Method: GET 2024-02-01T10:49:03.230 DEBUG:boto:Path: /vclfcn-63/obj3?versionId=null 2024-02-01T10:49:03.230 DEBUG:boto:Data: 2024-02-01T10:49:03.230 DEBUG:boto:Headers: {} 2024-02-01T10:49:03.230 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:03.230 DEBUG:boto:Port: 8000 2024-02-01T10:49:03.230 DEBUG:boto:Params: {} 2024-02-01T10:49:03.230 DEBUG:boto:Token: None 2024-02-01T10:49:03.230 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:03 GMT /vclfcn-63/obj3?versionId=null 2024-02-01T10:49:03.230 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fCtPs5ZBja8dW8Op7tFh4PJ4i0A= 2024-02-01T10:49:03.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fCtPs5ZBja8dW8Op7tFh4PJ4i0A=', 'Content-Length': '0'} 2024-02-01T10:49:03.234 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000443964bb17241ba1-0065bb771f-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:03.276 DEBUG:boto:path=/vclfcn-63/obj3 2024-02-01T10:49:03.276 DEBUG:boto:auth_path=/vclfcn-63/obj3 2024-02-01T10:49:03.276 DEBUG:boto:path=/vclfcn-63/obj3?versionId=null 2024-02-01T10:49:03.277 DEBUG:boto:auth_path=/vclfcn-63/obj3?versionId=null 2024-02-01T10:49:03.277 DEBUG:boto:Method: HEAD 2024-02-01T10:49:03.277 DEBUG:boto:Path: /vclfcn-63/obj3?versionId=null 2024-02-01T10:49:03.277 DEBUG:boto:Data: 2024-02-01T10:49:03.277 DEBUG:boto:Headers: {} 2024-02-01T10:49:03.277 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:03.277 DEBUG:boto:Port: 8000 2024-02-01T10:49:03.277 DEBUG:boto:Params: {} 2024-02-01T10:49:03.277 DEBUG:boto:Token: None 2024-02-01T10:49:03.278 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:03 GMT /vclfcn-63/obj3?versionId=null 2024-02-01T10:49:03.278 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:TGpECB/xBh+jCfIo1XewA0zJ/2Y= 2024-02-01T10:49:03.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:TGpECB/xBh+jCfIo1XewA0zJ/2Y=', 'Content-Length': '0'} 2024-02-01T10:49:03.280 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000500881a3fccdbe18-0065bb771f-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:03.281 DEBUG:boto:path=/vclfcn-63/obj3 2024-02-01T10:49:03.281 DEBUG:boto:auth_path=/vclfcn-63/obj3 2024-02-01T10:49:03.281 DEBUG:boto:path=/vclfcn-63/obj3?versionId=null 2024-02-01T10:49:03.281 DEBUG:boto:auth_path=/vclfcn-63/obj3?versionId=null 2024-02-01T10:49:03.281 DEBUG:boto:Method: HEAD 2024-02-01T10:49:03.281 DEBUG:boto:Path: /vclfcn-63/obj3?versionId=null 2024-02-01T10:49:03.281 DEBUG:boto:Data: 2024-02-01T10:49:03.281 DEBUG:boto:Headers: {} 2024-02-01T10:49:03.282 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:03.282 DEBUG:boto:Port: 8000 2024-02-01T10:49:03.282 DEBUG:boto:Params: {} 2024-02-01T10:49:03.282 DEBUG:boto:Token: None 2024-02-01T10:49:03.282 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:03 GMT /vclfcn-63/obj3?versionId=null 2024-02-01T10:49:03.282 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:TGpECB/xBh+jCfIo1XewA0zJ/2Y= 2024-02-01T10:49:03.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:TGpECB/xBh+jCfIo1XewA0zJ/2Y=', 'Content-Length': '0'} 2024-02-01T10:49:03.285 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000075e81850d98a3b99-0065bb771f-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:03.285 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:49:03.286 DEBUG:boto:path=/vclfcn-63/obj3 2024-02-01T10:49:03.286 DEBUG:boto:auth_path=/vclfcn-63/obj3 2024-02-01T10:49:03.286 DEBUG:boto:path=/vclfcn-63/obj3?versionId=null 2024-02-01T10:49:03.286 DEBUG:boto:auth_path=/vclfcn-63/obj3?versionId=null 2024-02-01T10:49:03.286 DEBUG:boto:Method: GET 2024-02-01T10:49:03.286 DEBUG:boto:Path: /vclfcn-63/obj3?versionId=null 2024-02-01T10:49:03.286 DEBUG:boto:Data: 2024-02-01T10:49:03.286 DEBUG:boto:Headers: {} 2024-02-01T10:49:03.286 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:03.287 DEBUG:boto:Port: 8000 2024-02-01T10:49:03.287 DEBUG:boto:Params: {} 2024-02-01T10:49:03.287 DEBUG:boto:Token: None 2024-02-01T10:49:03.287 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:03 GMT /vclfcn-63/obj3?versionId=null 2024-02-01T10:49:03.287 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fCtPs5ZBja8dW8Op7tFh4PJ4i0A= 2024-02-01T10:49:03.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fCtPs5ZBja8dW8Op7tFh4PJ4i0A=', 'Content-Length': '0'} 2024-02-01T10:49:03.290 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e45520dc90701195-0065bb771f-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:03.290 DEBUG:boto:path=/vclfcn-63/obj3 2024-02-01T10:49:03.290 DEBUG:boto:auth_path=/vclfcn-63/obj3 2024-02-01T10:49:03.290 DEBUG:boto:path=/vclfcn-63/obj3?versionId=null 2024-02-01T10:49:03.290 DEBUG:boto:auth_path=/vclfcn-63/obj3?versionId=null 2024-02-01T10:49:03.290 DEBUG:boto:Method: GET 2024-02-01T10:49:03.291 DEBUG:boto:Path: /vclfcn-63/obj3?versionId=null 2024-02-01T10:49:03.291 DEBUG:boto:Data: 2024-02-01T10:49:03.291 DEBUG:boto:Headers: {} 2024-02-01T10:49:03.291 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:03.291 DEBUG:boto:Port: 8000 2024-02-01T10:49:03.291 DEBUG:boto:Params: {} 2024-02-01T10:49:03.291 DEBUG:boto:Token: None 2024-02-01T10:49:03.291 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:03 GMT /vclfcn-63/obj3?versionId=null 2024-02-01T10:49:03.291 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fCtPs5ZBja8dW8Op7tFh4PJ4i0A= 2024-02-01T10:49:03.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fCtPs5ZBja8dW8Op7tFh4PJ4i0A=', 'Content-Length': '0'} 2024-02-01T10:49:03.295 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000006fc9f00ca638075-0065bb771f-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:03.336 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:49:03.336 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:49:03.336 DEBUG:boto:path=/vclfcn-63/?prefix=obj3 2024-02-01T10:49:03.336 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj3 2024-02-01T10:49:03.337 DEBUG:boto:Method: GET 2024-02-01T10:49:03.337 DEBUG:boto:Path: /vclfcn-63/?prefix=obj3 2024-02-01T10:49:03.337 DEBUG:boto:Data: 2024-02-01T10:49:03.337 DEBUG:boto:Headers: {} 2024-02-01T10:49:03.337 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:03.337 DEBUG:boto:Port: 8000 2024-02-01T10:49:03.337 DEBUG:boto:Params: {} 2024-02-01T10:49:03.337 DEBUG:boto:Token: None 2024-02-01T10:49:03.337 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:03 GMT /vclfcn-63/ 2024-02-01T10:49:03.338 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:uLqgyey3yk7v8jzNLr/8DxsAGkE= 2024-02-01T10:49:03.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:uLqgyey3yk7v8jzNLr/8DxsAGkE=', 'Content-Length': '0'} 2024-02-01T10:49:03.342 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f3f9a9d548314cb-0065bb771f-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:03.343 DEBUG:boto:b'vclfcn-63obj31000falseobj32024-02-01T10:48:26.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:03.343 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:49:03.343 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:49:03.343 DEBUG:boto:path=/vclfcn-63/?prefix=obj3 2024-02-01T10:49:03.343 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj3 2024-02-01T10:49:03.343 DEBUG:boto:Method: GET 2024-02-01T10:49:03.344 DEBUG:boto:Path: /vclfcn-63/?prefix=obj3 2024-02-01T10:49:03.344 DEBUG:boto:Data: 2024-02-01T10:49:03.344 DEBUG:boto:Headers: {} 2024-02-01T10:49:03.344 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:03.344 DEBUG:boto:Port: 8000 2024-02-01T10:49:03.344 DEBUG:boto:Params: {} 2024-02-01T10:49:03.344 DEBUG:boto:Token: None 2024-02-01T10:49:03.344 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:03 GMT /vclfcn-63/ 2024-02-01T10:49:03.344 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:uLqgyey3yk7v8jzNLr/8DxsAGkE= 2024-02-01T10:49:03.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:uLqgyey3yk7v8jzNLr/8DxsAGkE=', 'Content-Length': '0'} 2024-02-01T10:49:03.349 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f8cb7e980cafe30-0065bb771f-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:03.392 DEBUG:boto:b'vclfcn-63obj31000falseobj32024-02-01T10:48:26.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:03.393 DEBUG:boto:path=/vclfcn-63/obj3 2024-02-01T10:49:03.393 DEBUG:boto:auth_path=/vclfcn-63/obj3 2024-02-01T10:49:03.393 DEBUG:boto:Method: HEAD 2024-02-01T10:49:03.393 DEBUG:boto:Path: /vclfcn-63/obj3 2024-02-01T10:49:03.393 DEBUG:boto:Data: 2024-02-01T10:49:03.394 DEBUG:boto:Headers: {} 2024-02-01T10:49:03.394 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:03.394 DEBUG:boto:Port: 8000 2024-02-01T10:49:03.394 DEBUG:boto:Params: {} 2024-02-01T10:49:03.394 DEBUG:boto:Token: None 2024-02-01T10:49:03.394 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:03 GMT /vclfcn-63/obj3 2024-02-01T10:49:03.394 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:pIfB0Zv7IFju1PtSsCdM56mUXTQ= 2024-02-01T10:49:03.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:pIfB0Zv7IFju1PtSsCdM56mUXTQ=', 'Content-Length': '0'} 2024-02-01T10:49:03.397 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003a1e5617d2d3e9d9-0065bb771f-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:03.397 DEBUG:boto:path=/vclfcn-63/obj3 2024-02-01T10:49:03.397 DEBUG:boto:auth_path=/vclfcn-63/obj3 2024-02-01T10:49:03.398 DEBUG:boto:Method: HEAD 2024-02-01T10:49:03.398 DEBUG:boto:Path: /vclfcn-63/obj3 2024-02-01T10:49:03.398 DEBUG:boto:Data: 2024-02-01T10:49:03.398 DEBUG:boto:Headers: {} 2024-02-01T10:49:03.398 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:03.398 DEBUG:boto:Port: 8000 2024-02-01T10:49:03.398 DEBUG:boto:Params: {} 2024-02-01T10:49:03.398 DEBUG:boto:Token: None 2024-02-01T10:49:03.398 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:03 GMT /vclfcn-63/obj3 2024-02-01T10:49:03.399 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:pIfB0Zv7IFju1PtSsCdM56mUXTQ= 2024-02-01T10:49:03.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:pIfB0Zv7IFju1PtSsCdM56mUXTQ=', 'Content-Length': '0'} 2024-02-01T10:49:03.401 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000673f0bce9b45c756-0065bb771f-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:03.402 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:49:03.402 DEBUG:boto:path=/vclfcn-63/obj3 2024-02-01T10:49:03.402 DEBUG:boto:auth_path=/vclfcn-63/obj3 2024-02-01T10:49:03.402 DEBUG:boto:Method: GET 2024-02-01T10:49:03.402 DEBUG:boto:Path: /vclfcn-63/obj3 2024-02-01T10:49:03.402 DEBUG:boto:Data: 2024-02-01T10:49:03.402 DEBUG:boto:Headers: {} 2024-02-01T10:49:03.402 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:03.403 DEBUG:boto:Port: 8000 2024-02-01T10:49:03.403 DEBUG:boto:Params: {} 2024-02-01T10:49:03.403 DEBUG:boto:Token: None 2024-02-01T10:49:03.403 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:03 GMT /vclfcn-63/obj3 2024-02-01T10:49:03.403 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BKuju2U72p/qhUg3abADEaFwBCM= 2024-02-01T10:49:03.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BKuju2U72p/qhUg3abADEaFwBCM=', 'Content-Length': '0'} 2024-02-01T10:49:03.406 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ea9abcff7f752322-0065bb771f-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:03.406 DEBUG:boto:path=/vclfcn-63/obj3 2024-02-01T10:49:03.406 DEBUG:boto:auth_path=/vclfcn-63/obj3 2024-02-01T10:49:03.406 DEBUG:boto:Method: GET 2024-02-01T10:49:03.406 DEBUG:boto:Path: /vclfcn-63/obj3 2024-02-01T10:49:03.407 DEBUG:boto:Data: 2024-02-01T10:49:03.407 DEBUG:boto:Headers: {} 2024-02-01T10:49:03.407 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:03.407 DEBUG:boto:Port: 8000 2024-02-01T10:49:03.407 DEBUG:boto:Params: {} 2024-02-01T10:49:03.407 DEBUG:boto:Token: None 2024-02-01T10:49:03.407 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:03 GMT /vclfcn-63/obj3 2024-02-01T10:49:03.407 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BKuju2U72p/qhUg3abADEaFwBCM= 2024-02-01T10:49:03.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BKuju2U72p/qhUg3abADEaFwBCM=', 'Content-Length': '0'} 2024-02-01T10:49:03.410 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000012a3b907ab02a76f-0065bb771f-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:03.452 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:49:03.452 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:49:03.453 DEBUG:boto:path=/vclfcn-63/?prefix=obj3 2024-02-01T10:49:03.453 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj3 2024-02-01T10:49:03.453 DEBUG:boto:Method: GET 2024-02-01T10:49:03.453 DEBUG:boto:Path: /vclfcn-63/?prefix=obj3 2024-02-01T10:49:03.453 DEBUG:boto:Data: 2024-02-01T10:49:03.453 DEBUG:boto:Headers: {} 2024-02-01T10:49:03.453 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:03.453 DEBUG:boto:Port: 8000 2024-02-01T10:49:03.453 DEBUG:boto:Params: {} 2024-02-01T10:49:03.453 DEBUG:boto:Token: None 2024-02-01T10:49:03.454 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:03 GMT /vclfcn-63/ 2024-02-01T10:49:03.454 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:uLqgyey3yk7v8jzNLr/8DxsAGkE= 2024-02-01T10:49:03.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:uLqgyey3yk7v8jzNLr/8DxsAGkE=', 'Content-Length': '0'} 2024-02-01T10:49:03.458 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a1c9c1190225d94-0065bb771f-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:03.458 DEBUG:boto:b'vclfcn-63obj31000falseobj32024-02-01T10:48:26.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:03.459 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:49:03.459 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:49:03.459 DEBUG:boto:path=/vclfcn-63/?prefix=obj3 2024-02-01T10:49:03.459 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj3 2024-02-01T10:49:03.459 DEBUG:boto:Method: GET 2024-02-01T10:49:03.459 DEBUG:boto:Path: /vclfcn-63/?prefix=obj3 2024-02-01T10:49:03.459 DEBUG:boto:Data: 2024-02-01T10:49:03.459 DEBUG:boto:Headers: {} 2024-02-01T10:49:03.460 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:03.460 DEBUG:boto:Port: 8000 2024-02-01T10:49:03.460 DEBUG:boto:Params: {} 2024-02-01T10:49:03.460 DEBUG:boto:Token: None 2024-02-01T10:49:03.460 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:03 GMT /vclfcn-63/ 2024-02-01T10:49:03.460 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:uLqgyey3yk7v8jzNLr/8DxsAGkE= 2024-02-01T10:49:03.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:uLqgyey3yk7v8jzNLr/8DxsAGkE=', 'Content-Length': '0'} 2024-02-01T10:49:03.464 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000deb3fd367a9d8a2d-0065bb771f-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:03.512 DEBUG:boto:b'vclfcn-63obj31000falseobj32024-02-01T10:48:26.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:03.513 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:49:03.513 DEBUG:boto:path=/vclfcn-63/obj4 2024-02-01T10:49:03.513 DEBUG:boto:auth_path=/vclfcn-63/obj4 2024-02-01T10:49:03.513 DEBUG:boto:path=/vclfcn-63/obj4?versionId=null 2024-02-01T10:49:03.513 DEBUG:boto:auth_path=/vclfcn-63/obj4?versionId=null 2024-02-01T10:49:03.513 DEBUG:boto:Method: GET 2024-02-01T10:49:03.513 DEBUG:boto:Path: /vclfcn-63/obj4?versionId=null 2024-02-01T10:49:03.513 DEBUG:boto:Data: 2024-02-01T10:49:03.513 DEBUG:boto:Headers: {} 2024-02-01T10:49:03.514 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:03.514 DEBUG:boto:Port: 8000 2024-02-01T10:49:03.514 DEBUG:boto:Params: {} 2024-02-01T10:49:03.514 DEBUG:boto:Token: None 2024-02-01T10:49:03.514 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:03 GMT /vclfcn-63/obj4?versionId=null 2024-02-01T10:49:03.514 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:JWTfADH2BML6hy1dYdzZi4h2A3o= 2024-02-01T10:49:03.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:JWTfADH2BML6hy1dYdzZi4h2A3o=', 'Content-Length': '0'} 2024-02-01T10:49:03.517 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f9d0336b4082b1f4-0065bb771f-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:03.517 DEBUG:boto:path=/vclfcn-63/obj4 2024-02-01T10:49:03.517 DEBUG:boto:auth_path=/vclfcn-63/obj4 2024-02-01T10:49:03.517 DEBUG:boto:path=/vclfcn-63/obj4?versionId=null 2024-02-01T10:49:03.518 DEBUG:boto:auth_path=/vclfcn-63/obj4?versionId=null 2024-02-01T10:49:03.518 DEBUG:boto:Method: GET 2024-02-01T10:49:03.518 DEBUG:boto:Path: /vclfcn-63/obj4?versionId=null 2024-02-01T10:49:03.518 DEBUG:boto:Data: 2024-02-01T10:49:03.518 DEBUG:boto:Headers: {} 2024-02-01T10:49:03.518 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:03.518 DEBUG:boto:Port: 8000 2024-02-01T10:49:03.518 DEBUG:boto:Params: {} 2024-02-01T10:49:03.518 DEBUG:boto:Token: None 2024-02-01T10:49:03.518 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:03 GMT /vclfcn-63/obj4?versionId=null 2024-02-01T10:49:03.519 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:JWTfADH2BML6hy1dYdzZi4h2A3o= 2024-02-01T10:49:03.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:JWTfADH2BML6hy1dYdzZi4h2A3o=', 'Content-Length': '0'} 2024-02-01T10:49:03.522 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000082e7cfaaf648da3f-0065bb771f-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:03.522 DEBUG:boto:path=/vclfcn-63/obj4 2024-02-01T10:49:03.522 DEBUG:boto:auth_path=/vclfcn-63/obj4 2024-02-01T10:49:03.522 DEBUG:boto:path=/vclfcn-63/obj4?versionId=null 2024-02-01T10:49:03.523 DEBUG:boto:auth_path=/vclfcn-63/obj4?versionId=null 2024-02-01T10:49:03.523 DEBUG:boto:Method: HEAD 2024-02-01T10:49:03.523 DEBUG:boto:Path: /vclfcn-63/obj4?versionId=null 2024-02-01T10:49:03.523 DEBUG:boto:Data: 2024-02-01T10:49:03.523 DEBUG:boto:Headers: {} 2024-02-01T10:49:03.523 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:03.523 DEBUG:boto:Port: 8000 2024-02-01T10:49:03.523 DEBUG:boto:Params: {} 2024-02-01T10:49:03.523 DEBUG:boto:Token: None 2024-02-01T10:49:03.523 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:03 GMT /vclfcn-63/obj4?versionId=null 2024-02-01T10:49:03.524 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WA13e9wHGn7aVglMJzWcrrg75pA= 2024-02-01T10:49:03.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WA13e9wHGn7aVglMJzWcrrg75pA=', 'Content-Length': '0'} 2024-02-01T10:49:03.527 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008c939f252d1e4b6e-0065bb771f-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:03.527 DEBUG:boto:path=/vclfcn-63/obj4 2024-02-01T10:49:03.527 DEBUG:boto:auth_path=/vclfcn-63/obj4 2024-02-01T10:49:03.527 DEBUG:boto:path=/vclfcn-63/obj4?versionId=null 2024-02-01T10:49:03.527 DEBUG:boto:auth_path=/vclfcn-63/obj4?versionId=null 2024-02-01T10:49:03.527 DEBUG:boto:Method: HEAD 2024-02-01T10:49:03.527 DEBUG:boto:Path: /vclfcn-63/obj4?versionId=null 2024-02-01T10:49:03.527 DEBUG:boto:Data: 2024-02-01T10:49:03.527 DEBUG:boto:Headers: {} 2024-02-01T10:49:03.528 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:03.528 DEBUG:boto:Port: 8000 2024-02-01T10:49:03.528 DEBUG:boto:Params: {} 2024-02-01T10:49:03.528 DEBUG:boto:Token: None 2024-02-01T10:49:03.528 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:03 GMT /vclfcn-63/obj4?versionId=null 2024-02-01T10:49:03.528 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WA13e9wHGn7aVglMJzWcrrg75pA= 2024-02-01T10:49:03.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WA13e9wHGn7aVglMJzWcrrg75pA=', 'Content-Length': '0'} 2024-02-01T10:49:03.531 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ed5586d25eba75ed-0065bb771f-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:03.531 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:49:03.531 DEBUG:boto:path=/vclfcn-63/obj4 2024-02-01T10:49:03.531 DEBUG:boto:auth_path=/vclfcn-63/obj4 2024-02-01T10:49:03.531 DEBUG:boto:path=/vclfcn-63/obj4?versionId=null 2024-02-01T10:49:03.532 DEBUG:boto:auth_path=/vclfcn-63/obj4?versionId=null 2024-02-01T10:49:03.532 DEBUG:boto:Method: GET 2024-02-01T10:49:03.532 DEBUG:boto:Path: /vclfcn-63/obj4?versionId=null 2024-02-01T10:49:03.532 DEBUG:boto:Data: 2024-02-01T10:49:03.532 DEBUG:boto:Headers: {} 2024-02-01T10:49:03.532 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:03.532 DEBUG:boto:Port: 8000 2024-02-01T10:49:03.532 DEBUG:boto:Params: {} 2024-02-01T10:49:03.532 DEBUG:boto:Token: None 2024-02-01T10:49:03.532 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:03 GMT /vclfcn-63/obj4?versionId=null 2024-02-01T10:49:03.533 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:JWTfADH2BML6hy1dYdzZi4h2A3o= 2024-02-01T10:49:03.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:JWTfADH2BML6hy1dYdzZi4h2A3o=', 'Content-Length': '0'} 2024-02-01T10:49:03.535 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b537e0d8fa38d0cc-0065bb771f-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:03.576 DEBUG:boto:path=/vclfcn-63/obj4 2024-02-01T10:49:03.576 DEBUG:boto:auth_path=/vclfcn-63/obj4 2024-02-01T10:49:03.576 DEBUG:boto:path=/vclfcn-63/obj4?versionId=null 2024-02-01T10:49:03.577 DEBUG:boto:auth_path=/vclfcn-63/obj4?versionId=null 2024-02-01T10:49:03.577 DEBUG:boto:Method: GET 2024-02-01T10:49:03.577 DEBUG:boto:Path: /vclfcn-63/obj4?versionId=null 2024-02-01T10:49:03.577 DEBUG:boto:Data: 2024-02-01T10:49:03.577 DEBUG:boto:Headers: {} 2024-02-01T10:49:03.577 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:03.577 DEBUG:boto:Port: 8000 2024-02-01T10:49:03.577 DEBUG:boto:Params: {} 2024-02-01T10:49:03.577 DEBUG:boto:Token: None 2024-02-01T10:49:03.578 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:03 GMT /vclfcn-63/obj4?versionId=null 2024-02-01T10:49:03.578 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:JWTfADH2BML6hy1dYdzZi4h2A3o= 2024-02-01T10:49:03.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:JWTfADH2BML6hy1dYdzZi4h2A3o=', 'Content-Length': '0'} 2024-02-01T10:49:03.581 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000011d0a3b7ea16cf1c-0065bb771f-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:03.628 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:49:03.628 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:49:03.628 DEBUG:boto:path=/vclfcn-63/?prefix=obj4 2024-02-01T10:49:03.629 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj4 2024-02-01T10:49:03.629 DEBUG:boto:Method: GET 2024-02-01T10:49:03.629 DEBUG:boto:Path: /vclfcn-63/?prefix=obj4 2024-02-01T10:49:03.629 DEBUG:boto:Data: 2024-02-01T10:49:03.629 DEBUG:boto:Headers: {} 2024-02-01T10:49:03.629 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:03.629 DEBUG:boto:Port: 8000 2024-02-01T10:49:03.629 DEBUG:boto:Params: {} 2024-02-01T10:49:03.629 DEBUG:boto:Token: None 2024-02-01T10:49:03.630 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:03 GMT /vclfcn-63/ 2024-02-01T10:49:03.630 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:uLqgyey3yk7v8jzNLr/8DxsAGkE= 2024-02-01T10:49:03.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:uLqgyey3yk7v8jzNLr/8DxsAGkE=', 'Content-Length': '0'} 2024-02-01T10:49:03.634 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001077aec4810764d7-0065bb771f-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:03.635 DEBUG:boto:b'vclfcn-63obj41000falseobj42024-02-01T10:48:26.575Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:03.636 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:49:03.636 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:49:03.636 DEBUG:boto:path=/vclfcn-63/?prefix=obj4 2024-02-01T10:49:03.636 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj4 2024-02-01T10:49:03.636 DEBUG:boto:Method: GET 2024-02-01T10:49:03.636 DEBUG:boto:Path: /vclfcn-63/?prefix=obj4 2024-02-01T10:49:03.636 DEBUG:boto:Data: 2024-02-01T10:49:03.636 DEBUG:boto:Headers: {} 2024-02-01T10:49:03.636 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:03.636 DEBUG:boto:Port: 8000 2024-02-01T10:49:03.637 DEBUG:boto:Params: {} 2024-02-01T10:49:03.637 DEBUG:boto:Token: None 2024-02-01T10:49:03.637 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:03 GMT /vclfcn-63/ 2024-02-01T10:49:03.637 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:uLqgyey3yk7v8jzNLr/8DxsAGkE= 2024-02-01T10:49:03.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:uLqgyey3yk7v8jzNLr/8DxsAGkE=', 'Content-Length': '0'} 2024-02-01T10:49:03.641 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000227ecaea5a7d0c51-0065bb771f-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:03.684 DEBUG:boto:b'vclfcn-63obj41000falseobj42024-02-01T10:48:26.575Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:03.685 DEBUG:boto:path=/vclfcn-63/obj4 2024-02-01T10:49:03.685 DEBUG:boto:auth_path=/vclfcn-63/obj4 2024-02-01T10:49:03.685 DEBUG:boto:Method: HEAD 2024-02-01T10:49:03.685 DEBUG:boto:Path: /vclfcn-63/obj4 2024-02-01T10:49:03.685 DEBUG:boto:Data: 2024-02-01T10:49:03.685 DEBUG:boto:Headers: {} 2024-02-01T10:49:03.685 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:03.685 DEBUG:boto:Port: 8000 2024-02-01T10:49:03.685 DEBUG:boto:Params: {} 2024-02-01T10:49:03.686 DEBUG:boto:Token: None 2024-02-01T10:49:03.686 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:03 GMT /vclfcn-63/obj4 2024-02-01T10:49:03.686 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:E1lurnFbdxQdl1q7Bo3m1kQeaDA= 2024-02-01T10:49:03.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:E1lurnFbdxQdl1q7Bo3m1kQeaDA=', 'Content-Length': '0'} 2024-02-01T10:49:03.689 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002d735b32ab03e603-0065bb771f-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:03.689 DEBUG:boto:path=/vclfcn-63/obj4 2024-02-01T10:49:03.689 DEBUG:boto:auth_path=/vclfcn-63/obj4 2024-02-01T10:49:03.689 DEBUG:boto:Method: HEAD 2024-02-01T10:49:03.689 DEBUG:boto:Path: /vclfcn-63/obj4 2024-02-01T10:49:03.689 DEBUG:boto:Data: 2024-02-01T10:49:03.690 DEBUG:boto:Headers: {} 2024-02-01T10:49:03.690 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:03.690 DEBUG:boto:Port: 8000 2024-02-01T10:49:03.690 DEBUG:boto:Params: {} 2024-02-01T10:49:03.690 DEBUG:boto:Token: None 2024-02-01T10:49:03.690 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:03 GMT /vclfcn-63/obj4 2024-02-01T10:49:03.690 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:E1lurnFbdxQdl1q7Bo3m1kQeaDA= 2024-02-01T10:49:03.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:E1lurnFbdxQdl1q7Bo3m1kQeaDA=', 'Content-Length': '0'} 2024-02-01T10:49:03.694 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000031419e91511d0061-0065bb771f-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:03.694 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:49:03.694 DEBUG:boto:path=/vclfcn-63/obj4 2024-02-01T10:49:03.694 DEBUG:boto:auth_path=/vclfcn-63/obj4 2024-02-01T10:49:03.694 DEBUG:boto:Method: GET 2024-02-01T10:49:03.694 DEBUG:boto:Path: /vclfcn-63/obj4 2024-02-01T10:49:03.694 DEBUG:boto:Data: 2024-02-01T10:49:03.694 DEBUG:boto:Headers: {} 2024-02-01T10:49:03.695 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:03.695 DEBUG:boto:Port: 8000 2024-02-01T10:49:03.695 DEBUG:boto:Params: {} 2024-02-01T10:49:03.695 DEBUG:boto:Token: None 2024-02-01T10:49:03.695 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:03 GMT /vclfcn-63/obj4 2024-02-01T10:49:03.695 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cUHEq576jHwYXWfwbVi49DyIhJU= 2024-02-01T10:49:03.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cUHEq576jHwYXWfwbVi49DyIhJU=', 'Content-Length': '0'} 2024-02-01T10:49:03.698 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e415828f700ae812-0065bb771f-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:03.698 DEBUG:boto:path=/vclfcn-63/obj4 2024-02-01T10:49:03.699 DEBUG:boto:auth_path=/vclfcn-63/obj4 2024-02-01T10:49:03.699 DEBUG:boto:Method: GET 2024-02-01T10:49:03.699 DEBUG:boto:Path: /vclfcn-63/obj4 2024-02-01T10:49:03.699 DEBUG:boto:Data: 2024-02-01T10:49:03.699 DEBUG:boto:Headers: {} 2024-02-01T10:49:03.699 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:03.699 DEBUG:boto:Port: 8000 2024-02-01T10:49:03.699 DEBUG:boto:Params: {} 2024-02-01T10:49:03.699 DEBUG:boto:Token: None 2024-02-01T10:49:03.699 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:03 GMT /vclfcn-63/obj4 2024-02-01T10:49:03.700 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cUHEq576jHwYXWfwbVi49DyIhJU= 2024-02-01T10:49:03.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cUHEq576jHwYXWfwbVi49DyIhJU=', 'Content-Length': '0'} 2024-02-01T10:49:03.703 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fc3aa2fdea5eec47-0065bb771f-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:03.748 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:49:03.748 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:49:03.748 DEBUG:boto:path=/vclfcn-63/?prefix=obj4 2024-02-01T10:49:03.748 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj4 2024-02-01T10:49:03.749 DEBUG:boto:Method: GET 2024-02-01T10:49:03.749 DEBUG:boto:Path: /vclfcn-63/?prefix=obj4 2024-02-01T10:49:03.749 DEBUG:boto:Data: 2024-02-01T10:49:03.749 DEBUG:boto:Headers: {} 2024-02-01T10:49:03.749 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:03.749 DEBUG:boto:Port: 8000 2024-02-01T10:49:03.749 DEBUG:boto:Params: {} 2024-02-01T10:49:03.749 DEBUG:boto:Token: None 2024-02-01T10:49:03.749 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:03 GMT /vclfcn-63/ 2024-02-01T10:49:03.750 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:uLqgyey3yk7v8jzNLr/8DxsAGkE= 2024-02-01T10:49:03.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:uLqgyey3yk7v8jzNLr/8DxsAGkE=', 'Content-Length': '0'} 2024-02-01T10:49:03.754 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000440467aa6a534180-0065bb771f-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:03.755 DEBUG:boto:b'vclfcn-63obj41000falseobj42024-02-01T10:48:26.575Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:03.755 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:49:03.755 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:49:03.755 DEBUG:boto:path=/vclfcn-63/?prefix=obj4 2024-02-01T10:49:03.755 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj4 2024-02-01T10:49:03.755 DEBUG:boto:Method: GET 2024-02-01T10:49:03.756 DEBUG:boto:Path: /vclfcn-63/?prefix=obj4 2024-02-01T10:49:03.756 DEBUG:boto:Data: 2024-02-01T10:49:03.756 DEBUG:boto:Headers: {} 2024-02-01T10:49:03.756 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:03.756 DEBUG:boto:Port: 8000 2024-02-01T10:49:03.756 DEBUG:boto:Params: {} 2024-02-01T10:49:03.756 DEBUG:boto:Token: None 2024-02-01T10:49:03.756 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:03 GMT /vclfcn-63/ 2024-02-01T10:49:03.756 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:uLqgyey3yk7v8jzNLr/8DxsAGkE= 2024-02-01T10:49:03.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:uLqgyey3yk7v8jzNLr/8DxsAGkE=', 'Content-Length': '0'} 2024-02-01T10:49:03.763 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b4aebe51a60b99d-0065bb771f-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:03.805 DEBUG:boto:b'vclfcn-63obj41000falseobj42024-02-01T10:48:26.575Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:03.805 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-63 zones={test-zone2, test-zone1} 2024-02-01T10:49:03.805 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-63 zones={test-zone3, test-zone1} 2024-02-01T10:49:03.806 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:49:03.806 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:49:03.806 DEBUG:boto:Method: HEAD 2024-02-01T10:49:03.806 DEBUG:boto:Path: /vclfcn-63/ 2024-02-01T10:49:03.806 DEBUG:boto:Data: 2024-02-01T10:49:03.806 DEBUG:boto:Headers: {} 2024-02-01T10:49:03.806 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:03.806 DEBUG:boto:Port: 8001 2024-02-01T10:49:03.806 DEBUG:boto:Params: {} 2024-02-01T10:49:03.806 DEBUG:boto:Token: None 2024-02-01T10:49:03.807 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:03 GMT /vclfcn-63/ 2024-02-01T10:49:03.807 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:pHGJl7K1gu+a+AvDbi4RRK9pEFc= 2024-02-01T10:49:03.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:pHGJl7K1gu+a+AvDbi4RRK9pEFc=', 'Content-Length': '0'} 2024-02-01T10:49:03.812 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000df4dfd5a75340133-0065bb771f-5762-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:49:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:03.812 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:49:03.812 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:49:03.812 DEBUG:boto:Method: HEAD 2024-02-01T10:49:03.812 DEBUG:boto:Path: /vclfcn-63/ 2024-02-01T10:49:03.813 DEBUG:boto:Data: 2024-02-01T10:49:03.813 DEBUG:boto:Headers: {} 2024-02-01T10:49:03.813 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:03.813 DEBUG:boto:Port: 8000 2024-02-01T10:49:03.813 DEBUG:boto:Params: {} 2024-02-01T10:49:03.813 DEBUG:boto:Token: None 2024-02-01T10:49:03.813 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:03 GMT /vclfcn-63/ 2024-02-01T10:49:03.813 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:pHGJl7K1gu+a+AvDbi4RRK9pEFc= 2024-02-01T10:49:03.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:pHGJl7K1gu+a+AvDbi4RRK9pEFc=', 'Content-Length': '0'} 2024-02-01T10:49:03.817 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000691d936084f8af75-0065bb771f-5765-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:49:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:03.818 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:49:03.818 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:49:03.818 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:49:03.818 DEBUG:boto:path=/vclfcn-63/?versions&max-keys=999 2024-02-01T10:49:03.818 DEBUG:boto:auth_path=/vclfcn-63/?versions&max-keys=999 2024-02-01T10:49:03.818 DEBUG:boto:Method: GET 2024-02-01T10:49:03.818 DEBUG:boto:Path: /vclfcn-63/?versions&max-keys=999 2024-02-01T10:49:03.818 DEBUG:boto:Data: 2024-02-01T10:49:03.819 DEBUG:boto:Headers: {} 2024-02-01T10:49:03.819 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:03.819 DEBUG:boto:Port: 8001 2024-02-01T10:49:03.819 DEBUG:boto:Params: {} 2024-02-01T10:49:03.819 DEBUG:boto:Token: None 2024-02-01T10:49:03.819 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:03 GMT /vclfcn-63/?versions 2024-02-01T10:49:03.819 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:SmO0zjZ5aYjl4hYjmn9j+paRYzo= 2024-02-01T10:49:03.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:SmO0zjZ5aYjl4hYjmn9j+paRYzo=', 'Content-Length': '0'} 2024-02-01T10:49:03.824 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000355c21de11f71e33-0065bb771f-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:03.824 DEBUG:boto:b'vclfcn-63999falseobj1nulltrue2024-02-01T10:48:26.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:48:26.507Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:48:26.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:48:26.575Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:03.824 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-02-01T10:49:03.824 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-02-01T10:49:03.825 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-02-01T10:49:03.825 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-02-01T10:49:03.825 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:49:03.825 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:49:03.825 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:49:03.825 DEBUG:boto:path=/vclfcn-63/?versions&max-keys=999 2024-02-01T10:49:03.825 DEBUG:boto:auth_path=/vclfcn-63/?versions&max-keys=999 2024-02-01T10:49:03.825 DEBUG:boto:Method: GET 2024-02-01T10:49:03.825 DEBUG:boto:Path: /vclfcn-63/?versions&max-keys=999 2024-02-01T10:49:03.826 DEBUG:boto:Data: 2024-02-01T10:49:03.826 DEBUG:boto:Headers: {} 2024-02-01T10:49:03.826 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:03.826 DEBUG:boto:Port: 8000 2024-02-01T10:49:03.826 DEBUG:boto:Params: {} 2024-02-01T10:49:03.826 DEBUG:boto:Token: None 2024-02-01T10:49:03.826 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:03 GMT /vclfcn-63/?versions 2024-02-01T10:49:03.826 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:SmO0zjZ5aYjl4hYjmn9j+paRYzo= 2024-02-01T10:49:03.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:SmO0zjZ5aYjl4hYjmn9j+paRYzo=', 'Content-Length': '0'} 2024-02-01T10:49:03.831 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091a3cabe6a1a777f-0065bb771f-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:03.831 DEBUG:boto:b'vclfcn-63999falseobj1nulltrue2024-02-01T10:48:26.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:48:26.507Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:48:26.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:48:26.575Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:03.832 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-02-01T10:49:03.832 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-02-01T10:49:03.832 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-02-01T10:49:03.832 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-02-01T10:49:03.832 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:49:03.832 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:49:03.832 DEBUG:boto:path=/vclfcn-63/?versions&max-keys=999 2024-02-01T10:49:03.832 DEBUG:boto:auth_path=/vclfcn-63/?versions&max-keys=999 2024-02-01T10:49:03.833 DEBUG:boto:Method: GET 2024-02-01T10:49:03.833 DEBUG:boto:Path: /vclfcn-63/?versions&max-keys=999 2024-02-01T10:49:03.833 DEBUG:boto:Data: 2024-02-01T10:49:03.833 DEBUG:boto:Headers: {} 2024-02-01T10:49:03.833 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:03.833 DEBUG:boto:Port: 8001 2024-02-01T10:49:03.833 DEBUG:boto:Params: {} 2024-02-01T10:49:03.833 DEBUG:boto:Token: None 2024-02-01T10:49:03.834 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:03 GMT /vclfcn-63/?versions 2024-02-01T10:49:03.834 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:SmO0zjZ5aYjl4hYjmn9j+paRYzo= 2024-02-01T10:49:03.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:SmO0zjZ5aYjl4hYjmn9j+paRYzo=', 'Content-Length': '0'} 2024-02-01T10:49:03.838 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dca070b76011ee56-0065bb771f-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:03.838 DEBUG:boto:b'vclfcn-63999falseobj1nulltrue2024-02-01T10:48:26.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:48:26.507Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:48:26.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:48:26.575Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:03.839 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:49:03.839 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:49:03.839 DEBUG:boto:path=/vclfcn-63/?versions&max-keys=999 2024-02-01T10:49:03.839 DEBUG:boto:auth_path=/vclfcn-63/?versions&max-keys=999 2024-02-01T10:49:03.839 DEBUG:boto:Method: GET 2024-02-01T10:49:03.839 DEBUG:boto:Path: /vclfcn-63/?versions&max-keys=999 2024-02-01T10:49:03.839 DEBUG:boto:Data: 2024-02-01T10:49:03.840 DEBUG:boto:Headers: {} 2024-02-01T10:49:03.840 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:03.840 DEBUG:boto:Port: 8000 2024-02-01T10:49:03.840 DEBUG:boto:Params: {} 2024-02-01T10:49:03.840 DEBUG:boto:Token: None 2024-02-01T10:49:03.840 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:03 GMT /vclfcn-63/?versions 2024-02-01T10:49:03.840 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:SmO0zjZ5aYjl4hYjmn9j+paRYzo= 2024-02-01T10:49:03.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:SmO0zjZ5aYjl4hYjmn9j+paRYzo=', 'Content-Length': '0'} 2024-02-01T10:49:03.845 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f29286af194a4689-0065bb771f-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:03.845 DEBUG:boto:b'vclfcn-63999falseobj1nulltrue2024-02-01T10:48:26.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:48:26.507Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:48:26.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:48:26.575Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:03.846 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:49:03.846 DEBUG:boto:path=/vclfcn-63/obj1 2024-02-01T10:49:03.846 DEBUG:boto:auth_path=/vclfcn-63/obj1 2024-02-01T10:49:03.846 DEBUG:boto:path=/vclfcn-63/obj1?versionId=null 2024-02-01T10:49:03.847 DEBUG:boto:auth_path=/vclfcn-63/obj1?versionId=null 2024-02-01T10:49:03.847 DEBUG:boto:Method: GET 2024-02-01T10:49:03.847 DEBUG:boto:Path: /vclfcn-63/obj1?versionId=null 2024-02-01T10:49:03.847 DEBUG:boto:Data: 2024-02-01T10:49:03.847 DEBUG:boto:Headers: {} 2024-02-01T10:49:03.847 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:03.847 DEBUG:boto:Port: 8001 2024-02-01T10:49:03.847 DEBUG:boto:Params: {} 2024-02-01T10:49:03.847 DEBUG:boto:Token: None 2024-02-01T10:49:03.848 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:03 GMT /vclfcn-63/obj1?versionId=null 2024-02-01T10:49:03.848 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ewv5MPLPkPkDpQxAEsoc6S3QbP4= 2024-02-01T10:49:03.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ewv5MPLPkPkDpQxAEsoc6S3QbP4=', 'Content-Length': '0'} 2024-02-01T10:49:03.851 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ffa907a9e5cc0d84-0065bb771f-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:03.892 DEBUG:boto:path=/vclfcn-63/obj1 2024-02-01T10:49:03.892 DEBUG:boto:auth_path=/vclfcn-63/obj1 2024-02-01T10:49:03.892 DEBUG:boto:path=/vclfcn-63/obj1?versionId=null 2024-02-01T10:49:03.892 DEBUG:boto:auth_path=/vclfcn-63/obj1?versionId=null 2024-02-01T10:49:03.893 DEBUG:boto:Method: GET 2024-02-01T10:49:03.893 DEBUG:boto:Path: /vclfcn-63/obj1?versionId=null 2024-02-01T10:49:03.893 DEBUG:boto:Data: 2024-02-01T10:49:03.893 DEBUG:boto:Headers: {} 2024-02-01T10:49:03.893 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:03.893 DEBUG:boto:Port: 8000 2024-02-01T10:49:03.893 DEBUG:boto:Params: {} 2024-02-01T10:49:03.893 DEBUG:boto:Token: None 2024-02-01T10:49:03.894 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:03 GMT /vclfcn-63/obj1?versionId=null 2024-02-01T10:49:03.894 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ewv5MPLPkPkDpQxAEsoc6S3QbP4= 2024-02-01T10:49:03.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ewv5MPLPkPkDpQxAEsoc6S3QbP4=', 'Content-Length': '0'} 2024-02-01T10:49:03.897 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000072b2a574124fed6c-0065bb771f-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:03.898 DEBUG:boto:path=/vclfcn-63/obj1 2024-02-01T10:49:03.898 DEBUG:boto:auth_path=/vclfcn-63/obj1 2024-02-01T10:49:03.898 DEBUG:boto:path=/vclfcn-63/obj1?versionId=null 2024-02-01T10:49:03.899 DEBUG:boto:auth_path=/vclfcn-63/obj1?versionId=null 2024-02-01T10:49:03.899 DEBUG:boto:Method: HEAD 2024-02-01T10:49:03.899 DEBUG:boto:Path: /vclfcn-63/obj1?versionId=null 2024-02-01T10:49:03.899 DEBUG:boto:Data: 2024-02-01T10:49:03.899 DEBUG:boto:Headers: {} 2024-02-01T10:49:03.899 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:03.901 DEBUG:boto:Port: 8001 2024-02-01T10:49:03.901 DEBUG:boto:Params: {} 2024-02-01T10:49:03.901 DEBUG:boto:Token: None 2024-02-01T10:49:03.901 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:03 GMT /vclfcn-63/obj1?versionId=null 2024-02-01T10:49:03.901 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:aW1zP8pSiwa2iMtS2kGinB9GGTY= 2024-02-01T10:49:03.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:aW1zP8pSiwa2iMtS2kGinB9GGTY=', 'Content-Length': '0'} 2024-02-01T10:49:03.904 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009cdb62919adda0a4-0065bb771f-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:03.904 DEBUG:boto:path=/vclfcn-63/obj1 2024-02-01T10:49:03.904 DEBUG:boto:auth_path=/vclfcn-63/obj1 2024-02-01T10:49:03.904 DEBUG:boto:path=/vclfcn-63/obj1?versionId=null 2024-02-01T10:49:03.905 DEBUG:boto:auth_path=/vclfcn-63/obj1?versionId=null 2024-02-01T10:49:03.905 DEBUG:boto:Method: HEAD 2024-02-01T10:49:03.905 DEBUG:boto:Path: /vclfcn-63/obj1?versionId=null 2024-02-01T10:49:03.905 DEBUG:boto:Data: 2024-02-01T10:49:03.905 DEBUG:boto:Headers: {} 2024-02-01T10:49:03.905 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:03.905 DEBUG:boto:Port: 8000 2024-02-01T10:49:03.905 DEBUG:boto:Params: {} 2024-02-01T10:49:03.906 DEBUG:boto:Token: None 2024-02-01T10:49:03.906 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:03 GMT /vclfcn-63/obj1?versionId=null 2024-02-01T10:49:03.906 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:aW1zP8pSiwa2iMtS2kGinB9GGTY= 2024-02-01T10:49:03.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:aW1zP8pSiwa2iMtS2kGinB9GGTY=', 'Content-Length': '0'} 2024-02-01T10:49:03.909 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000047691dc46e9f7825-0065bb771f-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:03.909 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:49:03.909 DEBUG:boto:path=/vclfcn-63/obj1 2024-02-01T10:49:03.909 DEBUG:boto:auth_path=/vclfcn-63/obj1 2024-02-01T10:49:03.909 DEBUG:boto:path=/vclfcn-63/obj1?versionId=null 2024-02-01T10:49:03.909 DEBUG:boto:auth_path=/vclfcn-63/obj1?versionId=null 2024-02-01T10:49:03.910 DEBUG:boto:Method: GET 2024-02-01T10:49:03.910 DEBUG:boto:Path: /vclfcn-63/obj1?versionId=null 2024-02-01T10:49:03.910 DEBUG:boto:Data: 2024-02-01T10:49:03.910 DEBUG:boto:Headers: {} 2024-02-01T10:49:03.910 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:03.910 DEBUG:boto:Port: 8001 2024-02-01T10:49:03.910 DEBUG:boto:Params: {} 2024-02-01T10:49:03.910 DEBUG:boto:Token: None 2024-02-01T10:49:03.910 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:03 GMT /vclfcn-63/obj1?versionId=null 2024-02-01T10:49:03.911 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ewv5MPLPkPkDpQxAEsoc6S3QbP4= 2024-02-01T10:49:03.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ewv5MPLPkPkDpQxAEsoc6S3QbP4=', 'Content-Length': '0'} 2024-02-01T10:49:03.913 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000067b0c3e46d584027-0065bb771f-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:03.956 DEBUG:boto:path=/vclfcn-63/obj1 2024-02-01T10:49:03.956 DEBUG:boto:auth_path=/vclfcn-63/obj1 2024-02-01T10:49:03.956 DEBUG:boto:path=/vclfcn-63/obj1?versionId=null 2024-02-01T10:49:03.956 DEBUG:boto:auth_path=/vclfcn-63/obj1?versionId=null 2024-02-01T10:49:03.956 DEBUG:boto:Method: GET 2024-02-01T10:49:03.957 DEBUG:boto:Path: /vclfcn-63/obj1?versionId=null 2024-02-01T10:49:03.957 DEBUG:boto:Data: 2024-02-01T10:49:03.957 DEBUG:boto:Headers: {} 2024-02-01T10:49:03.957 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:03.957 DEBUG:boto:Port: 8000 2024-02-01T10:49:03.957 DEBUG:boto:Params: {} 2024-02-01T10:49:03.958 DEBUG:boto:Token: None 2024-02-01T10:49:03.958 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:03 GMT /vclfcn-63/obj1?versionId=null 2024-02-01T10:49:03.958 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ewv5MPLPkPkDpQxAEsoc6S3QbP4= 2024-02-01T10:49:03.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ewv5MPLPkPkDpQxAEsoc6S3QbP4=', 'Content-Length': '0'} 2024-02-01T10:49:03.961 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000751bf18f26b1c0b3-0065bb771f-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:04.008 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:49:04.008 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:49:04.009 DEBUG:boto:path=/vclfcn-63/?prefix=obj1 2024-02-01T10:49:04.009 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj1 2024-02-01T10:49:04.009 DEBUG:boto:Method: GET 2024-02-01T10:49:04.009 DEBUG:boto:Path: /vclfcn-63/?prefix=obj1 2024-02-01T10:49:04.009 DEBUG:boto:Data: 2024-02-01T10:49:04.009 DEBUG:boto:Headers: {} 2024-02-01T10:49:04.009 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:04.009 DEBUG:boto:Port: 8001 2024-02-01T10:49:04.009 DEBUG:boto:Params: {} 2024-02-01T10:49:04.009 DEBUG:boto:Token: None 2024-02-01T10:49:04.010 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:04 GMT /vclfcn-63/ 2024-02-01T10:49:04.010 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:n8HmNEkRmswUPn2QUKXL7697o/c= 2024-02-01T10:49:04.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:n8HmNEkRmswUPn2QUKXL7697o/c=', 'Content-Length': '0'} 2024-02-01T10:49:04.014 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000863ec7dbee6daed6-0065bb7720-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:04.060 DEBUG:boto:b'vclfcn-63obj11000falseobj12024-02-01T10:48:26.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:04.061 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:49:04.061 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:49:04.061 DEBUG:boto:path=/vclfcn-63/?prefix=obj1 2024-02-01T10:49:04.061 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj1 2024-02-01T10:49:04.061 DEBUG:boto:Method: GET 2024-02-01T10:49:04.061 DEBUG:boto:Path: /vclfcn-63/?prefix=obj1 2024-02-01T10:49:04.061 DEBUG:boto:Data: 2024-02-01T10:49:04.061 DEBUG:boto:Headers: {} 2024-02-01T10:49:04.061 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:04.061 DEBUG:boto:Port: 8000 2024-02-01T10:49:04.062 DEBUG:boto:Params: {} 2024-02-01T10:49:04.062 DEBUG:boto:Token: None 2024-02-01T10:49:04.062 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:04 GMT /vclfcn-63/ 2024-02-01T10:49:04.062 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:n8HmNEkRmswUPn2QUKXL7697o/c= 2024-02-01T10:49:04.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:n8HmNEkRmswUPn2QUKXL7697o/c=', 'Content-Length': '0'} 2024-02-01T10:49:04.066 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000107a1993be627355-0065bb7720-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:04.066 DEBUG:boto:b'vclfcn-63obj11000falseobj12024-02-01T10:48:26.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:04.067 DEBUG:boto:path=/vclfcn-63/obj1 2024-02-01T10:49:04.067 DEBUG:boto:auth_path=/vclfcn-63/obj1 2024-02-01T10:49:04.067 DEBUG:boto:Method: HEAD 2024-02-01T10:49:04.067 DEBUG:boto:Path: /vclfcn-63/obj1 2024-02-01T10:49:04.067 DEBUG:boto:Data: 2024-02-01T10:49:04.067 DEBUG:boto:Headers: {} 2024-02-01T10:49:04.067 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:04.067 DEBUG:boto:Port: 8001 2024-02-01T10:49:04.067 DEBUG:boto:Params: {} 2024-02-01T10:49:04.067 DEBUG:boto:Token: None 2024-02-01T10:49:04.068 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:04 GMT /vclfcn-63/obj1 2024-02-01T10:49:04.068 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cpQt4xZKNOJQr5hpUZxJaWNEMyc= 2024-02-01T10:49:04.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cpQt4xZKNOJQr5hpUZxJaWNEMyc=', 'Content-Length': '0'} 2024-02-01T10:49:04.071 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000001184a61bf15a774-0065bb7720-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:04.071 DEBUG:boto:path=/vclfcn-63/obj1 2024-02-01T10:49:04.071 DEBUG:boto:auth_path=/vclfcn-63/obj1 2024-02-01T10:49:04.071 DEBUG:boto:Method: HEAD 2024-02-01T10:49:04.071 DEBUG:boto:Path: /vclfcn-63/obj1 2024-02-01T10:49:04.071 DEBUG:boto:Data: 2024-02-01T10:49:04.071 DEBUG:boto:Headers: {} 2024-02-01T10:49:04.071 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:04.072 DEBUG:boto:Port: 8000 2024-02-01T10:49:04.072 DEBUG:boto:Params: {} 2024-02-01T10:49:04.072 DEBUG:boto:Token: None 2024-02-01T10:49:04.072 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:04 GMT /vclfcn-63/obj1 2024-02-01T10:49:04.072 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cpQt4xZKNOJQr5hpUZxJaWNEMyc= 2024-02-01T10:49:04.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cpQt4xZKNOJQr5hpUZxJaWNEMyc=', 'Content-Length': '0'} 2024-02-01T10:49:04.075 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000071b71f3013e650c2-0065bb7720-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:04.075 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:49:04.075 DEBUG:boto:path=/vclfcn-63/obj1 2024-02-01T10:49:04.075 DEBUG:boto:auth_path=/vclfcn-63/obj1 2024-02-01T10:49:04.075 DEBUG:boto:Method: GET 2024-02-01T10:49:04.075 DEBUG:boto:Path: /vclfcn-63/obj1 2024-02-01T10:49:04.075 DEBUG:boto:Data: 2024-02-01T10:49:04.076 DEBUG:boto:Headers: {} 2024-02-01T10:49:04.076 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:04.076 DEBUG:boto:Port: 8001 2024-02-01T10:49:04.076 DEBUG:boto:Params: {} 2024-02-01T10:49:04.076 DEBUG:boto:Token: None 2024-02-01T10:49:04.076 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:04 GMT /vclfcn-63/obj1 2024-02-01T10:49:04.076 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:B9fAjKkftrTJx3a3wWn2LkzziZY= 2024-02-01T10:49:04.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:B9fAjKkftrTJx3a3wWn2LkzziZY=', 'Content-Length': '0'} 2024-02-01T10:49:04.079 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005d3506359382d3d0-0065bb7720-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:04.120 DEBUG:boto:path=/vclfcn-63/obj1 2024-02-01T10:49:04.120 DEBUG:boto:auth_path=/vclfcn-63/obj1 2024-02-01T10:49:04.120 DEBUG:boto:Method: GET 2024-02-01T10:49:04.121 DEBUG:boto:Path: /vclfcn-63/obj1 2024-02-01T10:49:04.121 DEBUG:boto:Data: 2024-02-01T10:49:04.121 DEBUG:boto:Headers: {} 2024-02-01T10:49:04.121 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:04.121 DEBUG:boto:Port: 8000 2024-02-01T10:49:04.121 DEBUG:boto:Params: {} 2024-02-01T10:49:04.121 DEBUG:boto:Token: None 2024-02-01T10:49:04.121 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:04 GMT /vclfcn-63/obj1 2024-02-01T10:49:04.122 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:B9fAjKkftrTJx3a3wWn2LkzziZY= 2024-02-01T10:49:04.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:B9fAjKkftrTJx3a3wWn2LkzziZY=', 'Content-Length': '0'} 2024-02-01T10:49:04.124 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005e02b5540f940ef4-0065bb7720-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:04.168 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:49:04.168 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:49:04.169 DEBUG:boto:path=/vclfcn-63/?prefix=obj1 2024-02-01T10:49:04.169 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj1 2024-02-01T10:49:04.169 DEBUG:boto:Method: GET 2024-02-01T10:49:04.169 DEBUG:boto:Path: /vclfcn-63/?prefix=obj1 2024-02-01T10:49:04.169 DEBUG:boto:Data: 2024-02-01T10:49:04.169 DEBUG:boto:Headers: {} 2024-02-01T10:49:04.169 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:04.169 DEBUG:boto:Port: 8001 2024-02-01T10:49:04.169 DEBUG:boto:Params: {} 2024-02-01T10:49:04.170 DEBUG:boto:Token: None 2024-02-01T10:49:04.170 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:04 GMT /vclfcn-63/ 2024-02-01T10:49:04.170 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:n8HmNEkRmswUPn2QUKXL7697o/c= 2024-02-01T10:49:04.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:n8HmNEkRmswUPn2QUKXL7697o/c=', 'Content-Length': '0'} 2024-02-01T10:49:04.175 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee71c7d4262905ff-0065bb7720-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:04.216 DEBUG:boto:b'vclfcn-63obj11000falseobj12024-02-01T10:48:26.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:04.217 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:49:04.217 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:49:04.217 DEBUG:boto:path=/vclfcn-63/?prefix=obj1 2024-02-01T10:49:04.217 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj1 2024-02-01T10:49:04.217 DEBUG:boto:Method: GET 2024-02-01T10:49:04.217 DEBUG:boto:Path: /vclfcn-63/?prefix=obj1 2024-02-01T10:49:04.217 DEBUG:boto:Data: 2024-02-01T10:49:04.217 DEBUG:boto:Headers: {} 2024-02-01T10:49:04.217 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:04.218 DEBUG:boto:Port: 8000 2024-02-01T10:49:04.218 DEBUG:boto:Params: {} 2024-02-01T10:49:04.218 DEBUG:boto:Token: None 2024-02-01T10:49:04.218 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:04 GMT /vclfcn-63/ 2024-02-01T10:49:04.218 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:n8HmNEkRmswUPn2QUKXL7697o/c= 2024-02-01T10:49:04.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:n8HmNEkRmswUPn2QUKXL7697o/c=', 'Content-Length': '0'} 2024-02-01T10:49:04.222 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000020cc08d563803181-0065bb7720-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:04.223 DEBUG:boto:b'vclfcn-63obj11000falseobj12024-02-01T10:48:26.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:04.223 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:49:04.223 DEBUG:boto:path=/vclfcn-63/obj2 2024-02-01T10:49:04.223 DEBUG:boto:auth_path=/vclfcn-63/obj2 2024-02-01T10:49:04.223 DEBUG:boto:path=/vclfcn-63/obj2?versionId=null 2024-02-01T10:49:04.223 DEBUG:boto:auth_path=/vclfcn-63/obj2?versionId=null 2024-02-01T10:49:04.223 DEBUG:boto:Method: GET 2024-02-01T10:49:04.223 DEBUG:boto:Path: /vclfcn-63/obj2?versionId=null 2024-02-01T10:49:04.224 DEBUG:boto:Data: 2024-02-01T10:49:04.224 DEBUG:boto:Headers: {} 2024-02-01T10:49:04.224 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:04.224 DEBUG:boto:Port: 8001 2024-02-01T10:49:04.224 DEBUG:boto:Params: {} 2024-02-01T10:49:04.224 DEBUG:boto:Token: None 2024-02-01T10:49:04.224 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:04 GMT /vclfcn-63/obj2?versionId=null 2024-02-01T10:49:04.224 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0lWf67smHz++cg1d7qVvpdT7mNA= 2024-02-01T10:49:04.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0lWf67smHz++cg1d7qVvpdT7mNA=', 'Content-Length': '0'} 2024-02-01T10:49:04.228 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000018a62d19a91db273-0065bb7720-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:04.268 DEBUG:boto:path=/vclfcn-63/obj2 2024-02-01T10:49:04.268 DEBUG:boto:auth_path=/vclfcn-63/obj2 2024-02-01T10:49:04.268 DEBUG:boto:path=/vclfcn-63/obj2?versionId=null 2024-02-01T10:49:04.269 DEBUG:boto:auth_path=/vclfcn-63/obj2?versionId=null 2024-02-01T10:49:04.269 DEBUG:boto:Method: GET 2024-02-01T10:49:04.269 DEBUG:boto:Path: /vclfcn-63/obj2?versionId=null 2024-02-01T10:49:04.269 DEBUG:boto:Data: 2024-02-01T10:49:04.269 DEBUG:boto:Headers: {} 2024-02-01T10:49:04.269 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:04.269 DEBUG:boto:Port: 8000 2024-02-01T10:49:04.269 DEBUG:boto:Params: {} 2024-02-01T10:49:04.269 DEBUG:boto:Token: None 2024-02-01T10:49:04.270 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:04 GMT /vclfcn-63/obj2?versionId=null 2024-02-01T10:49:04.270 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0lWf67smHz++cg1d7qVvpdT7mNA= 2024-02-01T10:49:04.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0lWf67smHz++cg1d7qVvpdT7mNA=', 'Content-Length': '0'} 2024-02-01T10:49:04.273 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c83a61d8c0f56ff3-0065bb7720-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:04.273 DEBUG:boto:path=/vclfcn-63/obj2 2024-02-01T10:49:04.273 DEBUG:boto:auth_path=/vclfcn-63/obj2 2024-02-01T10:49:04.273 DEBUG:boto:path=/vclfcn-63/obj2?versionId=null 2024-02-01T10:49:04.273 DEBUG:boto:auth_path=/vclfcn-63/obj2?versionId=null 2024-02-01T10:49:04.274 DEBUG:boto:Method: HEAD 2024-02-01T10:49:04.274 DEBUG:boto:Path: /vclfcn-63/obj2?versionId=null 2024-02-01T10:49:04.274 DEBUG:boto:Data: 2024-02-01T10:49:04.274 DEBUG:boto:Headers: {} 2024-02-01T10:49:04.274 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:04.274 DEBUG:boto:Port: 8001 2024-02-01T10:49:04.274 DEBUG:boto:Params: {} 2024-02-01T10:49:04.274 DEBUG:boto:Token: None 2024-02-01T10:49:04.274 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:04 GMT /vclfcn-63/obj2?versionId=null 2024-02-01T10:49:04.275 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:utWKh/MQH1NseTwuyvSkGguiy9M= 2024-02-01T10:49:04.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:utWKh/MQH1NseTwuyvSkGguiy9M=', 'Content-Length': '0'} 2024-02-01T10:49:04.277 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000168941e840675f79-0065bb7720-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:04.277 DEBUG:boto:path=/vclfcn-63/obj2 2024-02-01T10:49:04.277 DEBUG:boto:auth_path=/vclfcn-63/obj2 2024-02-01T10:49:04.278 DEBUG:boto:path=/vclfcn-63/obj2?versionId=null 2024-02-01T10:49:04.278 DEBUG:boto:auth_path=/vclfcn-63/obj2?versionId=null 2024-02-01T10:49:04.278 DEBUG:boto:Method: HEAD 2024-02-01T10:49:04.278 DEBUG:boto:Path: /vclfcn-63/obj2?versionId=null 2024-02-01T10:49:04.278 DEBUG:boto:Data: 2024-02-01T10:49:04.278 DEBUG:boto:Headers: {} 2024-02-01T10:49:04.278 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:04.278 DEBUG:boto:Port: 8000 2024-02-01T10:49:04.278 DEBUG:boto:Params: {} 2024-02-01T10:49:04.278 DEBUG:boto:Token: None 2024-02-01T10:49:04.279 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:04 GMT /vclfcn-63/obj2?versionId=null 2024-02-01T10:49:04.279 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:utWKh/MQH1NseTwuyvSkGguiy9M= 2024-02-01T10:49:04.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:utWKh/MQH1NseTwuyvSkGguiy9M=', 'Content-Length': '0'} 2024-02-01T10:49:04.282 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002d9024366e330d49-0065bb7720-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:04.282 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:49:04.282 DEBUG:boto:path=/vclfcn-63/obj2 2024-02-01T10:49:04.282 DEBUG:boto:auth_path=/vclfcn-63/obj2 2024-02-01T10:49:04.282 DEBUG:boto:path=/vclfcn-63/obj2?versionId=null 2024-02-01T10:49:04.282 DEBUG:boto:auth_path=/vclfcn-63/obj2?versionId=null 2024-02-01T10:49:04.282 DEBUG:boto:Method: GET 2024-02-01T10:49:04.282 DEBUG:boto:Path: /vclfcn-63/obj2?versionId=null 2024-02-01T10:49:04.283 DEBUG:boto:Data: 2024-02-01T10:49:04.283 DEBUG:boto:Headers: {} 2024-02-01T10:49:04.283 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:04.283 DEBUG:boto:Port: 8001 2024-02-01T10:49:04.283 DEBUG:boto:Params: {} 2024-02-01T10:49:04.283 DEBUG:boto:Token: None 2024-02-01T10:49:04.283 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:04 GMT /vclfcn-63/obj2?versionId=null 2024-02-01T10:49:04.283 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0lWf67smHz++cg1d7qVvpdT7mNA= 2024-02-01T10:49:04.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0lWf67smHz++cg1d7qVvpdT7mNA=', 'Content-Length': '0'} 2024-02-01T10:49:04.286 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b8bc8bb6e201d294-0065bb7720-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:04.328 DEBUG:boto:path=/vclfcn-63/obj2 2024-02-01T10:49:04.328 DEBUG:boto:auth_path=/vclfcn-63/obj2 2024-02-01T10:49:04.328 DEBUG:boto:path=/vclfcn-63/obj2?versionId=null 2024-02-01T10:49:04.328 DEBUG:boto:auth_path=/vclfcn-63/obj2?versionId=null 2024-02-01T10:49:04.329 DEBUG:boto:Method: GET 2024-02-01T10:49:04.329 DEBUG:boto:Path: /vclfcn-63/obj2?versionId=null 2024-02-01T10:49:04.329 DEBUG:boto:Data: 2024-02-01T10:49:04.329 DEBUG:boto:Headers: {} 2024-02-01T10:49:04.329 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:04.329 DEBUG:boto:Port: 8000 2024-02-01T10:49:04.329 DEBUG:boto:Params: {} 2024-02-01T10:49:04.329 DEBUG:boto:Token: None 2024-02-01T10:49:04.329 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:04 GMT /vclfcn-63/obj2?versionId=null 2024-02-01T10:49:04.330 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0lWf67smHz++cg1d7qVvpdT7mNA= 2024-02-01T10:49:04.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0lWf67smHz++cg1d7qVvpdT7mNA=', 'Content-Length': '0'} 2024-02-01T10:49:04.333 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006cb637a92b29499a-0065bb7720-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:04.376 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:49:04.376 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:49:04.376 DEBUG:boto:path=/vclfcn-63/?prefix=obj2 2024-02-01T10:49:04.376 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj2 2024-02-01T10:49:04.377 DEBUG:boto:Method: GET 2024-02-01T10:49:04.377 DEBUG:boto:Path: /vclfcn-63/?prefix=obj2 2024-02-01T10:49:04.377 DEBUG:boto:Data: 2024-02-01T10:49:04.377 DEBUG:boto:Headers: {} 2024-02-01T10:49:04.377 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:04.377 DEBUG:boto:Port: 8001 2024-02-01T10:49:04.377 DEBUG:boto:Params: {} 2024-02-01T10:49:04.377 DEBUG:boto:Token: None 2024-02-01T10:49:04.377 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:04 GMT /vclfcn-63/ 2024-02-01T10:49:04.378 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:n8HmNEkRmswUPn2QUKXL7697o/c= 2024-02-01T10:49:04.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:n8HmNEkRmswUPn2QUKXL7697o/c=', 'Content-Length': '0'} 2024-02-01T10:49:04.382 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000975b5af77ff758d6-0065bb7720-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:04.424 DEBUG:boto:b'vclfcn-63obj21000falseobj22024-02-01T10:48:26.507Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:04.425 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:49:04.425 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:49:04.425 DEBUG:boto:path=/vclfcn-63/?prefix=obj2 2024-02-01T10:49:04.425 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj2 2024-02-01T10:49:04.425 DEBUG:boto:Method: GET 2024-02-01T10:49:04.425 DEBUG:boto:Path: /vclfcn-63/?prefix=obj2 2024-02-01T10:49:04.425 DEBUG:boto:Data: 2024-02-01T10:49:04.425 DEBUG:boto:Headers: {} 2024-02-01T10:49:04.425 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:04.426 DEBUG:boto:Port: 8000 2024-02-01T10:49:04.426 DEBUG:boto:Params: {} 2024-02-01T10:49:04.426 DEBUG:boto:Token: None 2024-02-01T10:49:04.426 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:04 GMT /vclfcn-63/ 2024-02-01T10:49:04.426 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:n8HmNEkRmswUPn2QUKXL7697o/c= 2024-02-01T10:49:04.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:n8HmNEkRmswUPn2QUKXL7697o/c=', 'Content-Length': '0'} 2024-02-01T10:49:04.431 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fda204ffb968484c-0065bb7720-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:04.431 DEBUG:boto:b'vclfcn-63obj21000falseobj22024-02-01T10:48:26.507Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:04.431 DEBUG:boto:path=/vclfcn-63/obj2 2024-02-01T10:49:04.432 DEBUG:boto:auth_path=/vclfcn-63/obj2 2024-02-01T10:49:04.432 DEBUG:boto:Method: HEAD 2024-02-01T10:49:04.432 DEBUG:boto:Path: /vclfcn-63/obj2 2024-02-01T10:49:04.432 DEBUG:boto:Data: 2024-02-01T10:49:04.432 DEBUG:boto:Headers: {} 2024-02-01T10:49:04.432 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:04.432 DEBUG:boto:Port: 8001 2024-02-01T10:49:04.432 DEBUG:boto:Params: {} 2024-02-01T10:49:04.432 DEBUG:boto:Token: None 2024-02-01T10:49:04.433 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:04 GMT /vclfcn-63/obj2 2024-02-01T10:49:04.433 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:MWidjTVwZLx98f5O1uO3RQckIOw= 2024-02-01T10:49:04.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:MWidjTVwZLx98f5O1uO3RQckIOw=', 'Content-Length': '0'} 2024-02-01T10:49:04.436 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000077fadb368cb20490-0065bb7720-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:04.436 DEBUG:boto:path=/vclfcn-63/obj2 2024-02-01T10:49:04.436 DEBUG:boto:auth_path=/vclfcn-63/obj2 2024-02-01T10:49:04.436 DEBUG:boto:Method: HEAD 2024-02-01T10:49:04.436 DEBUG:boto:Path: /vclfcn-63/obj2 2024-02-01T10:49:04.436 DEBUG:boto:Data: 2024-02-01T10:49:04.436 DEBUG:boto:Headers: {} 2024-02-01T10:49:04.437 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:04.437 DEBUG:boto:Port: 8000 2024-02-01T10:49:04.437 DEBUG:boto:Params: {} 2024-02-01T10:49:04.437 DEBUG:boto:Token: None 2024-02-01T10:49:04.437 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:04 GMT /vclfcn-63/obj2 2024-02-01T10:49:04.437 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:MWidjTVwZLx98f5O1uO3RQckIOw= 2024-02-01T10:49:04.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:MWidjTVwZLx98f5O1uO3RQckIOw=', 'Content-Length': '0'} 2024-02-01T10:49:04.440 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b3f06eed0c461d44-0065bb7720-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:04.441 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:49:04.441 DEBUG:boto:path=/vclfcn-63/obj2 2024-02-01T10:49:04.441 DEBUG:boto:auth_path=/vclfcn-63/obj2 2024-02-01T10:49:04.441 DEBUG:boto:Method: GET 2024-02-01T10:49:04.441 DEBUG:boto:Path: /vclfcn-63/obj2 2024-02-01T10:49:04.441 DEBUG:boto:Data: 2024-02-01T10:49:04.441 DEBUG:boto:Headers: {} 2024-02-01T10:49:04.441 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:04.442 DEBUG:boto:Port: 8001 2024-02-01T10:49:04.442 DEBUG:boto:Params: {} 2024-02-01T10:49:04.442 DEBUG:boto:Token: None 2024-02-01T10:49:04.442 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:04 GMT /vclfcn-63/obj2 2024-02-01T10:49:04.442 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:c3QeY0lUkDNtKew84G0+W2s93L0= 2024-02-01T10:49:04.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:c3QeY0lUkDNtKew84G0+W2s93L0=', 'Content-Length': '0'} 2024-02-01T10:49:04.445 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009fb0c2ef9cdb98b8-0065bb7720-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:04.488 DEBUG:boto:path=/vclfcn-63/obj2 2024-02-01T10:49:04.488 DEBUG:boto:auth_path=/vclfcn-63/obj2 2024-02-01T10:49:04.488 DEBUG:boto:Method: GET 2024-02-01T10:49:04.488 DEBUG:boto:Path: /vclfcn-63/obj2 2024-02-01T10:49:04.488 DEBUG:boto:Data: 2024-02-01T10:49:04.489 DEBUG:boto:Headers: {} 2024-02-01T10:49:04.489 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:04.489 DEBUG:boto:Port: 8000 2024-02-01T10:49:04.489 DEBUG:boto:Params: {} 2024-02-01T10:49:04.489 DEBUG:boto:Token: None 2024-02-01T10:49:04.489 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:04 GMT /vclfcn-63/obj2 2024-02-01T10:49:04.489 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:c3QeY0lUkDNtKew84G0+W2s93L0= 2024-02-01T10:49:04.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:c3QeY0lUkDNtKew84G0+W2s93L0=', 'Content-Length': '0'} 2024-02-01T10:49:04.492 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cbf6a9012dcbbefd-0065bb7720-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:04.536 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:49:04.536 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:49:04.536 DEBUG:boto:path=/vclfcn-63/?prefix=obj2 2024-02-01T10:49:04.536 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj2 2024-02-01T10:49:04.537 DEBUG:boto:Method: GET 2024-02-01T10:49:04.537 DEBUG:boto:Path: /vclfcn-63/?prefix=obj2 2024-02-01T10:49:04.537 DEBUG:boto:Data: 2024-02-01T10:49:04.537 DEBUG:boto:Headers: {} 2024-02-01T10:49:04.537 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:04.537 DEBUG:boto:Port: 8001 2024-02-01T10:49:04.537 DEBUG:boto:Params: {} 2024-02-01T10:49:04.537 DEBUG:boto:Token: None 2024-02-01T10:49:04.537 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:04 GMT /vclfcn-63/ 2024-02-01T10:49:04.538 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:n8HmNEkRmswUPn2QUKXL7697o/c= 2024-02-01T10:49:04.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:n8HmNEkRmswUPn2QUKXL7697o/c=', 'Content-Length': '0'} 2024-02-01T10:49:04.542 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000665ec61816c80e5-0065bb7720-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:04.542 DEBUG:boto:b'vclfcn-63obj21000falseobj22024-02-01T10:48:26.507Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:04.542 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:49:04.543 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:49:04.543 DEBUG:boto:path=/vclfcn-63/?prefix=obj2 2024-02-01T10:49:04.543 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj2 2024-02-01T10:49:04.543 DEBUG:boto:Method: GET 2024-02-01T10:49:04.543 DEBUG:boto:Path: /vclfcn-63/?prefix=obj2 2024-02-01T10:49:04.543 DEBUG:boto:Data: 2024-02-01T10:49:04.543 DEBUG:boto:Headers: {} 2024-02-01T10:49:04.543 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:04.543 DEBUG:boto:Port: 8000 2024-02-01T10:49:04.543 DEBUG:boto:Params: {} 2024-02-01T10:49:04.544 DEBUG:boto:Token: None 2024-02-01T10:49:04.544 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:04 GMT /vclfcn-63/ 2024-02-01T10:49:04.544 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:n8HmNEkRmswUPn2QUKXL7697o/c= 2024-02-01T10:49:04.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:n8HmNEkRmswUPn2QUKXL7697o/c=', 'Content-Length': '0'} 2024-02-01T10:49:04.549 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c01f74e3a1eb89d1-0065bb7720-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:04.549 DEBUG:boto:b'vclfcn-63obj21000falseobj22024-02-01T10:48:26.507Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:04.549 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:49:04.549 DEBUG:boto:path=/vclfcn-63/obj3 2024-02-01T10:49:04.549 DEBUG:boto:auth_path=/vclfcn-63/obj3 2024-02-01T10:49:04.549 DEBUG:boto:path=/vclfcn-63/obj3?versionId=null 2024-02-01T10:49:04.550 DEBUG:boto:auth_path=/vclfcn-63/obj3?versionId=null 2024-02-01T10:49:04.550 DEBUG:boto:Method: GET 2024-02-01T10:49:04.550 DEBUG:boto:Path: /vclfcn-63/obj3?versionId=null 2024-02-01T10:49:04.550 DEBUG:boto:Data: 2024-02-01T10:49:04.550 DEBUG:boto:Headers: {} 2024-02-01T10:49:04.550 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:04.550 DEBUG:boto:Port: 8001 2024-02-01T10:49:04.550 DEBUG:boto:Params: {} 2024-02-01T10:49:04.550 DEBUG:boto:Token: None 2024-02-01T10:49:04.550 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:04 GMT /vclfcn-63/obj3?versionId=null 2024-02-01T10:49:04.551 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nIw+5HibFTG6Pb+8RpboQmX9DmU= 2024-02-01T10:49:04.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nIw+5HibFTG6Pb+8RpboQmX9DmU=', 'Content-Length': '0'} 2024-02-01T10:49:04.553 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002d4eae825c0beafe-0065bb7720-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:04.596 DEBUG:boto:path=/vclfcn-63/obj3 2024-02-01T10:49:04.596 DEBUG:boto:auth_path=/vclfcn-63/obj3 2024-02-01T10:49:04.596 DEBUG:boto:path=/vclfcn-63/obj3?versionId=null 2024-02-01T10:49:04.596 DEBUG:boto:auth_path=/vclfcn-63/obj3?versionId=null 2024-02-01T10:49:04.597 DEBUG:boto:Method: GET 2024-02-01T10:49:04.597 DEBUG:boto:Path: /vclfcn-63/obj3?versionId=null 2024-02-01T10:49:04.597 DEBUG:boto:Data: 2024-02-01T10:49:04.597 DEBUG:boto:Headers: {} 2024-02-01T10:49:04.597 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:04.597 DEBUG:boto:Port: 8000 2024-02-01T10:49:04.597 DEBUG:boto:Params: {} 2024-02-01T10:49:04.597 DEBUG:boto:Token: None 2024-02-01T10:49:04.597 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:04 GMT /vclfcn-63/obj3?versionId=null 2024-02-01T10:49:04.598 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nIw+5HibFTG6Pb+8RpboQmX9DmU= 2024-02-01T10:49:04.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nIw+5HibFTG6Pb+8RpboQmX9DmU=', 'Content-Length': '0'} 2024-02-01T10:49:04.600 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d1646fe0ec4be14f-0065bb7720-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:04.601 DEBUG:boto:path=/vclfcn-63/obj3 2024-02-01T10:49:04.601 DEBUG:boto:auth_path=/vclfcn-63/obj3 2024-02-01T10:49:04.601 DEBUG:boto:path=/vclfcn-63/obj3?versionId=null 2024-02-01T10:49:04.601 DEBUG:boto:auth_path=/vclfcn-63/obj3?versionId=null 2024-02-01T10:49:04.601 DEBUG:boto:Method: HEAD 2024-02-01T10:49:04.601 DEBUG:boto:Path: /vclfcn-63/obj3?versionId=null 2024-02-01T10:49:04.601 DEBUG:boto:Data: 2024-02-01T10:49:04.601 DEBUG:boto:Headers: {} 2024-02-01T10:49:04.602 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:04.602 DEBUG:boto:Port: 8001 2024-02-01T10:49:04.602 DEBUG:boto:Params: {} 2024-02-01T10:49:04.602 DEBUG:boto:Token: None 2024-02-01T10:49:04.602 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:04 GMT /vclfcn-63/obj3?versionId=null 2024-02-01T10:49:04.602 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tr9CWlW/aDW1kp/NZDlMbolfG78= 2024-02-01T10:49:04.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tr9CWlW/aDW1kp/NZDlMbolfG78=', 'Content-Length': '0'} 2024-02-01T10:49:04.605 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ac343a2b00df8f41-0065bb7720-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:04.605 DEBUG:boto:path=/vclfcn-63/obj3 2024-02-01T10:49:04.605 DEBUG:boto:auth_path=/vclfcn-63/obj3 2024-02-01T10:49:04.605 DEBUG:boto:path=/vclfcn-63/obj3?versionId=null 2024-02-01T10:49:04.606 DEBUG:boto:auth_path=/vclfcn-63/obj3?versionId=null 2024-02-01T10:49:04.606 DEBUG:boto:Method: HEAD 2024-02-01T10:49:04.606 DEBUG:boto:Path: /vclfcn-63/obj3?versionId=null 2024-02-01T10:49:04.606 DEBUG:boto:Data: 2024-02-01T10:49:04.606 DEBUG:boto:Headers: {} 2024-02-01T10:49:04.606 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:04.606 DEBUG:boto:Port: 8000 2024-02-01T10:49:04.606 DEBUG:boto:Params: {} 2024-02-01T10:49:04.606 DEBUG:boto:Token: None 2024-02-01T10:49:04.606 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:04 GMT /vclfcn-63/obj3?versionId=null 2024-02-01T10:49:04.606 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tr9CWlW/aDW1kp/NZDlMbolfG78= 2024-02-01T10:49:04.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tr9CWlW/aDW1kp/NZDlMbolfG78=', 'Content-Length': '0'} 2024-02-01T10:49:04.609 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000030bf1e0d63a04db3-0065bb7720-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:04.610 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:49:04.610 DEBUG:boto:path=/vclfcn-63/obj3 2024-02-01T10:49:04.610 DEBUG:boto:auth_path=/vclfcn-63/obj3 2024-02-01T10:49:04.610 DEBUG:boto:path=/vclfcn-63/obj3?versionId=null 2024-02-01T10:49:04.610 DEBUG:boto:auth_path=/vclfcn-63/obj3?versionId=null 2024-02-01T10:49:04.610 DEBUG:boto:Method: GET 2024-02-01T10:49:04.610 DEBUG:boto:Path: /vclfcn-63/obj3?versionId=null 2024-02-01T10:49:04.610 DEBUG:boto:Data: 2024-02-01T10:49:04.610 DEBUG:boto:Headers: {} 2024-02-01T10:49:04.610 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:04.610 DEBUG:boto:Port: 8001 2024-02-01T10:49:04.611 DEBUG:boto:Params: {} 2024-02-01T10:49:04.611 DEBUG:boto:Token: None 2024-02-01T10:49:04.611 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:04 GMT /vclfcn-63/obj3?versionId=null 2024-02-01T10:49:04.611 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nIw+5HibFTG6Pb+8RpboQmX9DmU= 2024-02-01T10:49:04.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nIw+5HibFTG6Pb+8RpboQmX9DmU=', 'Content-Length': '0'} 2024-02-01T10:49:04.614 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000571e11419186e163-0065bb7720-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:04.656 DEBUG:boto:path=/vclfcn-63/obj3 2024-02-01T10:49:04.657 DEBUG:boto:auth_path=/vclfcn-63/obj3 2024-02-01T10:49:04.657 DEBUG:boto:path=/vclfcn-63/obj3?versionId=null 2024-02-01T10:49:04.657 DEBUG:boto:auth_path=/vclfcn-63/obj3?versionId=null 2024-02-01T10:49:04.658 DEBUG:boto:Method: GET 2024-02-01T10:49:04.658 DEBUG:boto:Path: /vclfcn-63/obj3?versionId=null 2024-02-01T10:49:04.658 DEBUG:boto:Data: 2024-02-01T10:49:04.658 DEBUG:boto:Headers: {} 2024-02-01T10:49:04.658 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:04.659 DEBUG:boto:Port: 8000 2024-02-01T10:49:04.659 DEBUG:boto:Params: {} 2024-02-01T10:49:04.659 DEBUG:boto:Token: None 2024-02-01T10:49:04.659 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:04 GMT /vclfcn-63/obj3?versionId=null 2024-02-01T10:49:04.660 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nIw+5HibFTG6Pb+8RpboQmX9DmU= 2024-02-01T10:49:04.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nIw+5HibFTG6Pb+8RpboQmX9DmU=', 'Content-Length': '0'} 2024-02-01T10:49:04.663 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000beb64a1ce8a11431-0065bb7720-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:04.704 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:49:04.705 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:49:04.705 DEBUG:boto:path=/vclfcn-63/?prefix=obj3 2024-02-01T10:49:04.705 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj3 2024-02-01T10:49:04.705 DEBUG:boto:Method: GET 2024-02-01T10:49:04.705 DEBUG:boto:Path: /vclfcn-63/?prefix=obj3 2024-02-01T10:49:04.706 DEBUG:boto:Data: 2024-02-01T10:49:04.706 DEBUG:boto:Headers: {} 2024-02-01T10:49:04.706 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:04.706 DEBUG:boto:Port: 8001 2024-02-01T10:49:04.706 DEBUG:boto:Params: {} 2024-02-01T10:49:04.707 DEBUG:boto:Token: None 2024-02-01T10:49:04.708 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:04 GMT /vclfcn-63/ 2024-02-01T10:49:04.708 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:n8HmNEkRmswUPn2QUKXL7697o/c= 2024-02-01T10:49:04.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:n8HmNEkRmswUPn2QUKXL7697o/c=', 'Content-Length': '0'} 2024-02-01T10:49:04.713 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003946f2145c633579-0065bb7720-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:04.756 DEBUG:boto:b'vclfcn-63obj31000falseobj32024-02-01T10:48:26.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:04.757 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:49:04.757 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:49:04.757 DEBUG:boto:path=/vclfcn-63/?prefix=obj3 2024-02-01T10:49:04.757 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj3 2024-02-01T10:49:04.757 DEBUG:boto:Method: GET 2024-02-01T10:49:04.757 DEBUG:boto:Path: /vclfcn-63/?prefix=obj3 2024-02-01T10:49:04.757 DEBUG:boto:Data: 2024-02-01T10:49:04.757 DEBUG:boto:Headers: {} 2024-02-01T10:49:04.757 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:04.757 DEBUG:boto:Port: 8000 2024-02-01T10:49:04.758 DEBUG:boto:Params: {} 2024-02-01T10:49:04.758 DEBUG:boto:Token: None 2024-02-01T10:49:04.758 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:04 GMT /vclfcn-63/ 2024-02-01T10:49:04.758 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:n8HmNEkRmswUPn2QUKXL7697o/c= 2024-02-01T10:49:04.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:n8HmNEkRmswUPn2QUKXL7697o/c=', 'Content-Length': '0'} 2024-02-01T10:49:04.763 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004036bc561233bb52-0065bb7720-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:04.763 DEBUG:boto:b'vclfcn-63obj31000falseobj32024-02-01T10:48:26.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:04.763 DEBUG:boto:path=/vclfcn-63/obj3 2024-02-01T10:49:04.763 DEBUG:boto:auth_path=/vclfcn-63/obj3 2024-02-01T10:49:04.763 DEBUG:boto:Method: HEAD 2024-02-01T10:49:04.764 DEBUG:boto:Path: /vclfcn-63/obj3 2024-02-01T10:49:04.764 DEBUG:boto:Data: 2024-02-01T10:49:04.764 DEBUG:boto:Headers: {} 2024-02-01T10:49:04.764 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:04.764 DEBUG:boto:Port: 8001 2024-02-01T10:49:04.764 DEBUG:boto:Params: {} 2024-02-01T10:49:04.764 DEBUG:boto:Token: None 2024-02-01T10:49:04.764 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:04 GMT /vclfcn-63/obj3 2024-02-01T10:49:04.765 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WkL3Plfq45J9VjWKzDKs/yEM/Vk= 2024-02-01T10:49:04.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WkL3Plfq45J9VjWKzDKs/yEM/Vk=', 'Content-Length': '0'} 2024-02-01T10:49:04.767 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001c7497d8e3684e90-0065bb7720-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:04.768 DEBUG:boto:path=/vclfcn-63/obj3 2024-02-01T10:49:04.768 DEBUG:boto:auth_path=/vclfcn-63/obj3 2024-02-01T10:49:04.768 DEBUG:boto:Method: HEAD 2024-02-01T10:49:04.768 DEBUG:boto:Path: /vclfcn-63/obj3 2024-02-01T10:49:04.768 DEBUG:boto:Data: 2024-02-01T10:49:04.768 DEBUG:boto:Headers: {} 2024-02-01T10:49:04.768 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:04.769 DEBUG:boto:Port: 8000 2024-02-01T10:49:04.769 DEBUG:boto:Params: {} 2024-02-01T10:49:04.769 DEBUG:boto:Token: None 2024-02-01T10:49:04.769 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:04 GMT /vclfcn-63/obj3 2024-02-01T10:49:04.769 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WkL3Plfq45J9VjWKzDKs/yEM/Vk= 2024-02-01T10:49:04.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WkL3Plfq45J9VjWKzDKs/yEM/Vk=', 'Content-Length': '0'} 2024-02-01T10:49:04.772 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b3cdb395bc5e70fb-0065bb7720-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:04.772 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:49:04.772 DEBUG:boto:path=/vclfcn-63/obj3 2024-02-01T10:49:04.772 DEBUG:boto:auth_path=/vclfcn-63/obj3 2024-02-01T10:49:04.772 DEBUG:boto:Method: GET 2024-02-01T10:49:04.773 DEBUG:boto:Path: /vclfcn-63/obj3 2024-02-01T10:49:04.773 DEBUG:boto:Data: 2024-02-01T10:49:04.773 DEBUG:boto:Headers: {} 2024-02-01T10:49:04.773 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:04.773 DEBUG:boto:Port: 8001 2024-02-01T10:49:04.773 DEBUG:boto:Params: {} 2024-02-01T10:49:04.773 DEBUG:boto:Token: None 2024-02-01T10:49:04.773 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:04 GMT /vclfcn-63/obj3 2024-02-01T10:49:04.773 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:pbRr058e7LXT9NyFbloFOSKBe0c= 2024-02-01T10:49:04.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:pbRr058e7LXT9NyFbloFOSKBe0c=', 'Content-Length': '0'} 2024-02-01T10:49:04.778 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e8e2a1ef4a63dd19-0065bb7720-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:04.820 DEBUG:boto:path=/vclfcn-63/obj3 2024-02-01T10:49:04.820 DEBUG:boto:auth_path=/vclfcn-63/obj3 2024-02-01T10:49:04.820 DEBUG:boto:Method: GET 2024-02-01T10:49:04.821 DEBUG:boto:Path: /vclfcn-63/obj3 2024-02-01T10:49:04.821 DEBUG:boto:Data: 2024-02-01T10:49:04.821 DEBUG:boto:Headers: {} 2024-02-01T10:49:04.821 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:04.821 DEBUG:boto:Port: 8000 2024-02-01T10:49:04.821 DEBUG:boto:Params: {} 2024-02-01T10:49:04.821 DEBUG:boto:Token: None 2024-02-01T10:49:04.821 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:04 GMT /vclfcn-63/obj3 2024-02-01T10:49:04.821 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:pbRr058e7LXT9NyFbloFOSKBe0c= 2024-02-01T10:49:04.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:pbRr058e7LXT9NyFbloFOSKBe0c=', 'Content-Length': '0'} 2024-02-01T10:49:04.825 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000016fcf14194929aa9-0065bb7720-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:04.868 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:49:04.868 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:49:04.868 DEBUG:boto:path=/vclfcn-63/?prefix=obj3 2024-02-01T10:49:04.868 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj3 2024-02-01T10:49:04.869 DEBUG:boto:Method: GET 2024-02-01T10:49:04.869 DEBUG:boto:Path: /vclfcn-63/?prefix=obj3 2024-02-01T10:49:04.869 DEBUG:boto:Data: 2024-02-01T10:49:04.869 DEBUG:boto:Headers: {} 2024-02-01T10:49:04.869 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:04.869 DEBUG:boto:Port: 8001 2024-02-01T10:49:04.869 DEBUG:boto:Params: {} 2024-02-01T10:49:04.869 DEBUG:boto:Token: None 2024-02-01T10:49:04.869 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:04 GMT /vclfcn-63/ 2024-02-01T10:49:04.869 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:n8HmNEkRmswUPn2QUKXL7697o/c= 2024-02-01T10:49:04.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:n8HmNEkRmswUPn2QUKXL7697o/c=', 'Content-Length': '0'} 2024-02-01T10:49:04.874 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000053d53a1e5861834-0065bb7720-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:04.875 DEBUG:boto:b'vclfcn-63obj31000falseobj32024-02-01T10:48:26.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:04.875 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:49:04.875 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:49:04.876 DEBUG:boto:path=/vclfcn-63/?prefix=obj3 2024-02-01T10:49:04.876 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj3 2024-02-01T10:49:04.876 DEBUG:boto:Method: GET 2024-02-01T10:49:04.876 DEBUG:boto:Path: /vclfcn-63/?prefix=obj3 2024-02-01T10:49:04.876 DEBUG:boto:Data: 2024-02-01T10:49:04.876 DEBUG:boto:Headers: {} 2024-02-01T10:49:04.876 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:04.876 DEBUG:boto:Port: 8000 2024-02-01T10:49:04.876 DEBUG:boto:Params: {} 2024-02-01T10:49:04.876 DEBUG:boto:Token: None 2024-02-01T10:49:04.877 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:04 GMT /vclfcn-63/ 2024-02-01T10:49:04.877 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:n8HmNEkRmswUPn2QUKXL7697o/c= 2024-02-01T10:49:04.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:n8HmNEkRmswUPn2QUKXL7697o/c=', 'Content-Length': '0'} 2024-02-01T10:49:04.881 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000074b594b5a7979ceb-0065bb7720-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:04.928 DEBUG:boto:b'vclfcn-63obj31000falseobj32024-02-01T10:48:26.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:04.929 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:49:04.929 DEBUG:boto:path=/vclfcn-63/obj4 2024-02-01T10:49:04.929 DEBUG:boto:auth_path=/vclfcn-63/obj4 2024-02-01T10:49:04.929 DEBUG:boto:path=/vclfcn-63/obj4?versionId=null 2024-02-01T10:49:04.929 DEBUG:boto:auth_path=/vclfcn-63/obj4?versionId=null 2024-02-01T10:49:04.929 DEBUG:boto:Method: GET 2024-02-01T10:49:04.929 DEBUG:boto:Path: /vclfcn-63/obj4?versionId=null 2024-02-01T10:49:04.929 DEBUG:boto:Data: 2024-02-01T10:49:04.929 DEBUG:boto:Headers: {} 2024-02-01T10:49:04.929 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:04.930 DEBUG:boto:Port: 8001 2024-02-01T10:49:04.930 DEBUG:boto:Params: {} 2024-02-01T10:49:04.930 DEBUG:boto:Token: None 2024-02-01T10:49:04.930 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:04 GMT /vclfcn-63/obj4?versionId=null 2024-02-01T10:49:04.930 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NoBOyuGD1hDmcaBLdvTv/be+Y70= 2024-02-01T10:49:04.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NoBOyuGD1hDmcaBLdvTv/be+Y70=', 'Content-Length': '0'} 2024-02-01T10:49:04.933 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008d0e1fbd00553fdc-0065bb7720-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:04.976 DEBUG:boto:path=/vclfcn-63/obj4 2024-02-01T10:49:04.976 DEBUG:boto:auth_path=/vclfcn-63/obj4 2024-02-01T10:49:04.977 DEBUG:boto:path=/vclfcn-63/obj4?versionId=null 2024-02-01T10:49:04.977 DEBUG:boto:auth_path=/vclfcn-63/obj4?versionId=null 2024-02-01T10:49:04.977 DEBUG:boto:Method: GET 2024-02-01T10:49:04.977 DEBUG:boto:Path: /vclfcn-63/obj4?versionId=null 2024-02-01T10:49:04.977 DEBUG:boto:Data: 2024-02-01T10:49:04.977 DEBUG:boto:Headers: {} 2024-02-01T10:49:04.977 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:04.977 DEBUG:boto:Port: 8000 2024-02-01T10:49:04.978 DEBUG:boto:Params: {} 2024-02-01T10:49:04.978 DEBUG:boto:Token: None 2024-02-01T10:49:04.978 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:04 GMT /vclfcn-63/obj4?versionId=null 2024-02-01T10:49:04.978 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NoBOyuGD1hDmcaBLdvTv/be+Y70= 2024-02-01T10:49:04.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NoBOyuGD1hDmcaBLdvTv/be+Y70=', 'Content-Length': '0'} 2024-02-01T10:49:04.981 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000be07c369e7408ac0-0065bb7720-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:04.982 DEBUG:boto:path=/vclfcn-63/obj4 2024-02-01T10:49:04.982 DEBUG:boto:auth_path=/vclfcn-63/obj4 2024-02-01T10:49:04.982 DEBUG:boto:path=/vclfcn-63/obj4?versionId=null 2024-02-01T10:49:04.983 DEBUG:boto:auth_path=/vclfcn-63/obj4?versionId=null 2024-02-01T10:49:04.983 DEBUG:boto:Method: HEAD 2024-02-01T10:49:04.983 DEBUG:boto:Path: /vclfcn-63/obj4?versionId=null 2024-02-01T10:49:04.983 DEBUG:boto:Data: 2024-02-01T10:49:04.983 DEBUG:boto:Headers: {} 2024-02-01T10:49:04.983 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:04.983 DEBUG:boto:Port: 8001 2024-02-01T10:49:04.984 DEBUG:boto:Params: {} 2024-02-01T10:49:04.984 DEBUG:boto:Token: None 2024-02-01T10:49:04.984 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:04 GMT /vclfcn-63/obj4?versionId=null 2024-02-01T10:49:04.984 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NxO/Ir6tHWLt4FHGECOs/nTD39g= 2024-02-01T10:49:04.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NxO/Ir6tHWLt4FHGECOs/nTD39g=', 'Content-Length': '0'} 2024-02-01T10:49:04.987 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000689605a4004240ed-0065bb7720-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:04.988 DEBUG:boto:path=/vclfcn-63/obj4 2024-02-01T10:49:04.988 DEBUG:boto:auth_path=/vclfcn-63/obj4 2024-02-01T10:49:04.988 DEBUG:boto:path=/vclfcn-63/obj4?versionId=null 2024-02-01T10:49:04.988 DEBUG:boto:auth_path=/vclfcn-63/obj4?versionId=null 2024-02-01T10:49:04.988 DEBUG:boto:Method: HEAD 2024-02-01T10:49:04.989 DEBUG:boto:Path: /vclfcn-63/obj4?versionId=null 2024-02-01T10:49:04.989 DEBUG:boto:Data: 2024-02-01T10:49:04.989 DEBUG:boto:Headers: {} 2024-02-01T10:49:04.989 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:04.989 DEBUG:boto:Port: 8000 2024-02-01T10:49:04.989 DEBUG:boto:Params: {} 2024-02-01T10:49:04.989 DEBUG:boto:Token: None 2024-02-01T10:49:04.989 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:04 GMT /vclfcn-63/obj4?versionId=null 2024-02-01T10:49:04.989 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NxO/Ir6tHWLt4FHGECOs/nTD39g= 2024-02-01T10:49:04.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NxO/Ir6tHWLt4FHGECOs/nTD39g=', 'Content-Length': '0'} 2024-02-01T10:49:04.992 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003b7c7d20e604ceda-0065bb7720-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:04.992 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:49:04.995 DEBUG:boto:path=/vclfcn-63/obj4 2024-02-01T10:49:04.995 DEBUG:boto:auth_path=/vclfcn-63/obj4 2024-02-01T10:49:04.995 DEBUG:boto:path=/vclfcn-63/obj4?versionId=null 2024-02-01T10:49:04.995 DEBUG:boto:auth_path=/vclfcn-63/obj4?versionId=null 2024-02-01T10:49:04.996 DEBUG:boto:Method: GET 2024-02-01T10:49:04.996 DEBUG:boto:Path: /vclfcn-63/obj4?versionId=null 2024-02-01T10:49:04.996 DEBUG:boto:Data: 2024-02-01T10:49:04.996 DEBUG:boto:Headers: {} 2024-02-01T10:49:04.996 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:04.996 DEBUG:boto:Port: 8001 2024-02-01T10:49:04.996 DEBUG:boto:Params: {} 2024-02-01T10:49:04.996 DEBUG:boto:Token: None 2024-02-01T10:49:04.996 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:04 GMT /vclfcn-63/obj4?versionId=null 2024-02-01T10:49:04.997 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NoBOyuGD1hDmcaBLdvTv/be+Y70= 2024-02-01T10:49:04.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NoBOyuGD1hDmcaBLdvTv/be+Y70=', 'Content-Length': '0'} 2024-02-01T10:49:05.000 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000080a2f89f14578a73-0065bb7720-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:05.044 DEBUG:boto:path=/vclfcn-63/obj4 2024-02-01T10:49:05.044 DEBUG:boto:auth_path=/vclfcn-63/obj4 2024-02-01T10:49:05.044 DEBUG:boto:path=/vclfcn-63/obj4?versionId=null 2024-02-01T10:49:05.044 DEBUG:boto:auth_path=/vclfcn-63/obj4?versionId=null 2024-02-01T10:49:05.045 DEBUG:boto:Method: GET 2024-02-01T10:49:05.045 DEBUG:boto:Path: /vclfcn-63/obj4?versionId=null 2024-02-01T10:49:05.045 DEBUG:boto:Data: 2024-02-01T10:49:05.045 DEBUG:boto:Headers: {} 2024-02-01T10:49:05.045 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:05.045 DEBUG:boto:Port: 8000 2024-02-01T10:49:05.045 DEBUG:boto:Params: {} 2024-02-01T10:49:05.045 DEBUG:boto:Token: None 2024-02-01T10:49:05.045 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:05 GMT /vclfcn-63/obj4?versionId=null 2024-02-01T10:49:05.046 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Y/S8EY9PcJWqQkH4eZ95LgcGaTs= 2024-02-01T10:49:05.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Y/S8EY9PcJWqQkH4eZ95LgcGaTs=', 'Content-Length': '0'} 2024-02-01T10:49:05.048 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007aed838d92fb35bc-0065bb7721-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:05.092 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:49:05.092 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:49:05.092 DEBUG:boto:path=/vclfcn-63/?prefix=obj4 2024-02-01T10:49:05.092 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj4 2024-02-01T10:49:05.093 DEBUG:boto:Method: GET 2024-02-01T10:49:05.093 DEBUG:boto:Path: /vclfcn-63/?prefix=obj4 2024-02-01T10:49:05.093 DEBUG:boto:Data: 2024-02-01T10:49:05.093 DEBUG:boto:Headers: {} 2024-02-01T10:49:05.093 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:05.093 DEBUG:boto:Port: 8001 2024-02-01T10:49:05.093 DEBUG:boto:Params: {} 2024-02-01T10:49:05.093 DEBUG:boto:Token: None 2024-02-01T10:49:05.093 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:05 GMT /vclfcn-63/ 2024-02-01T10:49:05.094 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0gAczpIV9i1WJPU8kC3CyH/Ylio= 2024-02-01T10:49:05.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0gAczpIV9i1WJPU8kC3CyH/Ylio=', 'Content-Length': '0'} 2024-02-01T10:49:05.098 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e0ed8d9e14268286-0065bb7721-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:05.098 DEBUG:boto:b'vclfcn-63obj41000falseobj42024-02-01T10:48:26.575Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:05.098 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:49:05.099 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:49:05.099 DEBUG:boto:path=/vclfcn-63/?prefix=obj4 2024-02-01T10:49:05.099 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj4 2024-02-01T10:49:05.099 DEBUG:boto:Method: GET 2024-02-01T10:49:05.099 DEBUG:boto:Path: /vclfcn-63/?prefix=obj4 2024-02-01T10:49:05.099 DEBUG:boto:Data: 2024-02-01T10:49:05.099 DEBUG:boto:Headers: {} 2024-02-01T10:49:05.099 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:05.099 DEBUG:boto:Port: 8000 2024-02-01T10:49:05.100 DEBUG:boto:Params: {} 2024-02-01T10:49:05.100 DEBUG:boto:Token: None 2024-02-01T10:49:05.100 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:05 GMT /vclfcn-63/ 2024-02-01T10:49:05.100 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0gAczpIV9i1WJPU8kC3CyH/Ylio= 2024-02-01T10:49:05.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0gAczpIV9i1WJPU8kC3CyH/Ylio=', 'Content-Length': '0'} 2024-02-01T10:49:05.105 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000111fadbca9100133-0065bb7721-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:05.148 DEBUG:boto:b'vclfcn-63obj41000falseobj42024-02-01T10:48:26.575Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:05.149 DEBUG:boto:path=/vclfcn-63/obj4 2024-02-01T10:49:05.149 DEBUG:boto:auth_path=/vclfcn-63/obj4 2024-02-01T10:49:05.149 DEBUG:boto:Method: HEAD 2024-02-01T10:49:05.149 DEBUG:boto:Path: /vclfcn-63/obj4 2024-02-01T10:49:05.149 DEBUG:boto:Data: 2024-02-01T10:49:05.149 DEBUG:boto:Headers: {} 2024-02-01T10:49:05.149 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:05.149 DEBUG:boto:Port: 8001 2024-02-01T10:49:05.149 DEBUG:boto:Params: {} 2024-02-01T10:49:05.149 DEBUG:boto:Token: None 2024-02-01T10:49:05.150 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:05 GMT /vclfcn-63/obj4 2024-02-01T10:49:05.150 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:smRpv8IGC8IQWws1NP2NJXGCeH8= 2024-02-01T10:49:05.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:smRpv8IGC8IQWws1NP2NJXGCeH8=', 'Content-Length': '0'} 2024-02-01T10:49:05.153 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006441b5ba1afe64d8-0065bb7721-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:05.153 DEBUG:boto:path=/vclfcn-63/obj4 2024-02-01T10:49:05.153 DEBUG:boto:auth_path=/vclfcn-63/obj4 2024-02-01T10:49:05.153 DEBUG:boto:Method: HEAD 2024-02-01T10:49:05.153 DEBUG:boto:Path: /vclfcn-63/obj4 2024-02-01T10:49:05.153 DEBUG:boto:Data: 2024-02-01T10:49:05.153 DEBUG:boto:Headers: {} 2024-02-01T10:49:05.153 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:05.154 DEBUG:boto:Port: 8000 2024-02-01T10:49:05.154 DEBUG:boto:Params: {} 2024-02-01T10:49:05.154 DEBUG:boto:Token: None 2024-02-01T10:49:05.154 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:05 GMT /vclfcn-63/obj4 2024-02-01T10:49:05.154 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:smRpv8IGC8IQWws1NP2NJXGCeH8= 2024-02-01T10:49:05.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:smRpv8IGC8IQWws1NP2NJXGCeH8=', 'Content-Length': '0'} 2024-02-01T10:49:05.157 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d50b6f7c6f5e5a81-0065bb7721-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:05.157 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:49:05.157 DEBUG:boto:path=/vclfcn-63/obj4 2024-02-01T10:49:05.157 DEBUG:boto:auth_path=/vclfcn-63/obj4 2024-02-01T10:49:05.157 DEBUG:boto:Method: GET 2024-02-01T10:49:05.157 DEBUG:boto:Path: /vclfcn-63/obj4 2024-02-01T10:49:05.158 DEBUG:boto:Data: 2024-02-01T10:49:05.158 DEBUG:boto:Headers: {} 2024-02-01T10:49:05.158 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:05.158 DEBUG:boto:Port: 8001 2024-02-01T10:49:05.158 DEBUG:boto:Params: {} 2024-02-01T10:49:05.158 DEBUG:boto:Token: None 2024-02-01T10:49:05.158 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:05 GMT /vclfcn-63/obj4 2024-02-01T10:49:05.158 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lPRtE6gc/J2O9sedCIx8g2cz/T4= 2024-02-01T10:49:05.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lPRtE6gc/J2O9sedCIx8g2cz/T4=', 'Content-Length': '0'} 2024-02-01T10:49:05.161 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cf597c833a986c0a-0065bb7721-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:05.208 DEBUG:boto:path=/vclfcn-63/obj4 2024-02-01T10:49:05.208 DEBUG:boto:auth_path=/vclfcn-63/obj4 2024-02-01T10:49:05.208 DEBUG:boto:Method: GET 2024-02-01T10:49:05.209 DEBUG:boto:Path: /vclfcn-63/obj4 2024-02-01T10:49:05.209 DEBUG:boto:Data: 2024-02-01T10:49:05.209 DEBUG:boto:Headers: {} 2024-02-01T10:49:05.209 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:05.209 DEBUG:boto:Port: 8000 2024-02-01T10:49:05.209 DEBUG:boto:Params: {} 2024-02-01T10:49:05.209 DEBUG:boto:Token: None 2024-02-01T10:49:05.209 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:05 GMT /vclfcn-63/obj4 2024-02-01T10:49:05.209 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lPRtE6gc/J2O9sedCIx8g2cz/T4= 2024-02-01T10:49:05.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lPRtE6gc/J2O9sedCIx8g2cz/T4=', 'Content-Length': '0'} 2024-02-01T10:49:05.212 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000246d2dc4e5fb70f5-0065bb7721-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:05.256 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:49:05.256 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:49:05.256 DEBUG:boto:path=/vclfcn-63/?prefix=obj4 2024-02-01T10:49:05.256 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj4 2024-02-01T10:49:05.257 DEBUG:boto:Method: GET 2024-02-01T10:49:05.257 DEBUG:boto:Path: /vclfcn-63/?prefix=obj4 2024-02-01T10:49:05.257 DEBUG:boto:Data: 2024-02-01T10:49:05.257 DEBUG:boto:Headers: {} 2024-02-01T10:49:05.257 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:05.257 DEBUG:boto:Port: 8001 2024-02-01T10:49:05.257 DEBUG:boto:Params: {} 2024-02-01T10:49:05.257 DEBUG:boto:Token: None 2024-02-01T10:49:05.257 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:05 GMT /vclfcn-63/ 2024-02-01T10:49:05.258 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0gAczpIV9i1WJPU8kC3CyH/Ylio= 2024-02-01T10:49:05.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0gAczpIV9i1WJPU8kC3CyH/Ylio=', 'Content-Length': '0'} 2024-02-01T10:49:05.262 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000022dfe0a424ab8279-0065bb7721-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:05.262 DEBUG:boto:b'vclfcn-63obj41000falseobj42024-02-01T10:48:26.575Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:05.263 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:49:05.263 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:49:05.263 DEBUG:boto:path=/vclfcn-63/?prefix=obj4 2024-02-01T10:49:05.263 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj4 2024-02-01T10:49:05.263 DEBUG:boto:Method: GET 2024-02-01T10:49:05.263 DEBUG:boto:Path: /vclfcn-63/?prefix=obj4 2024-02-01T10:49:05.263 DEBUG:boto:Data: 2024-02-01T10:49:05.263 DEBUG:boto:Headers: {} 2024-02-01T10:49:05.264 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:05.264 DEBUG:boto:Port: 8000 2024-02-01T10:49:05.264 DEBUG:boto:Params: {} 2024-02-01T10:49:05.264 DEBUG:boto:Token: None 2024-02-01T10:49:05.264 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:05 GMT /vclfcn-63/ 2024-02-01T10:49:05.264 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0gAczpIV9i1WJPU8kC3CyH/Ylio= 2024-02-01T10:49:05.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0gAczpIV9i1WJPU8kC3CyH/Ylio=', 'Content-Length': '0'} 2024-02-01T10:49:05.269 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed96db3bdfced92f-0065bb7721-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:05.269 DEBUG:boto:b'vclfcn-63obj41000falseobj42024-02-01T10:48:26.575Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:05.270 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-63 zones={test-zone3, test-zone1} 2024-02-01T10:49:05.270 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-63 zones={test-zone3, test-zone2} 2024-02-01T10:49:05.270 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:49:05.270 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:49:05.271 DEBUG:boto:Method: HEAD 2024-02-01T10:49:05.271 DEBUG:boto:Path: /vclfcn-63/ 2024-02-01T10:49:05.271 DEBUG:boto:Data: 2024-02-01T10:49:05.271 DEBUG:boto:Headers: {} 2024-02-01T10:49:05.271 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:05.271 DEBUG:boto:Port: 8001 2024-02-01T10:49:05.271 DEBUG:boto:Params: {} 2024-02-01T10:49:05.272 DEBUG:boto:Token: None 2024-02-01T10:49:05.272 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:05 GMT /vclfcn-63/ 2024-02-01T10:49:05.272 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:W7741mA7dGLqCiBX4C+/kqcr5Z4= 2024-02-01T10:49:05.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:W7741mA7dGLqCiBX4C+/kqcr5Z4=', 'Content-Length': '0'} 2024-02-01T10:49:05.276 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005d6c94d0918b5dd7-0065bb7721-5762-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:49:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:05.276 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:49:05.276 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:49:05.276 DEBUG:boto:Method: HEAD 2024-02-01T10:49:05.276 DEBUG:boto:Path: /vclfcn-63/ 2024-02-01T10:49:05.277 DEBUG:boto:Data: 2024-02-01T10:49:05.277 DEBUG:boto:Headers: {} 2024-02-01T10:49:05.277 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:05.277 DEBUG:boto:Port: 8000 2024-02-01T10:49:05.277 DEBUG:boto:Params: {} 2024-02-01T10:49:05.277 DEBUG:boto:Token: None 2024-02-01T10:49:05.277 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:05 GMT /vclfcn-63/ 2024-02-01T10:49:05.277 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:W7741mA7dGLqCiBX4C+/kqcr5Z4= 2024-02-01T10:49:05.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:W7741mA7dGLqCiBX4C+/kqcr5Z4=', 'Content-Length': '0'} 2024-02-01T10:49:05.281 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ebbd70f3959304d4-0065bb7721-5350-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:49:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:05.282 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:49:05.282 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:49:05.282 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:49:05.282 DEBUG:boto:path=/vclfcn-63/?versions&max-keys=999 2024-02-01T10:49:05.282 DEBUG:boto:auth_path=/vclfcn-63/?versions&max-keys=999 2024-02-01T10:49:05.282 DEBUG:boto:Method: GET 2024-02-01T10:49:05.282 DEBUG:boto:Path: /vclfcn-63/?versions&max-keys=999 2024-02-01T10:49:05.282 DEBUG:boto:Data: 2024-02-01T10:49:05.282 DEBUG:boto:Headers: {} 2024-02-01T10:49:05.283 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:05.283 DEBUG:boto:Port: 8001 2024-02-01T10:49:05.283 DEBUG:boto:Params: {} 2024-02-01T10:49:05.283 DEBUG:boto:Token: None 2024-02-01T10:49:05.283 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:05 GMT /vclfcn-63/?versions 2024-02-01T10:49:05.283 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Pi4/hQ7kc4gpxZUIgTMa5mdOj7s= 2024-02-01T10:49:05.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Pi4/hQ7kc4gpxZUIgTMa5mdOj7s=', 'Content-Length': '0'} 2024-02-01T10:49:05.288 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002664491cea67a40f-0065bb7721-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:05.288 DEBUG:boto:b'vclfcn-63999falseobj1nulltrue2024-02-01T10:48:26.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:48:26.507Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:48:26.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:48:26.575Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:05.289 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-02-01T10:49:05.289 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-02-01T10:49:05.290 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-02-01T10:49:05.290 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-02-01T10:49:05.290 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:49:05.290 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:49:05.290 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:49:05.290 DEBUG:boto:path=/vclfcn-63/?versions&max-keys=999 2024-02-01T10:49:05.290 DEBUG:boto:auth_path=/vclfcn-63/?versions&max-keys=999 2024-02-01T10:49:05.290 DEBUG:boto:Method: GET 2024-02-01T10:49:05.291 DEBUG:boto:Path: /vclfcn-63/?versions&max-keys=999 2024-02-01T10:49:05.291 DEBUG:boto:Data: 2024-02-01T10:49:05.291 DEBUG:boto:Headers: {} 2024-02-01T10:49:05.291 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:05.291 DEBUG:boto:Port: 8000 2024-02-01T10:49:05.291 DEBUG:boto:Params: {} 2024-02-01T10:49:05.291 DEBUG:boto:Token: None 2024-02-01T10:49:05.291 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:05 GMT /vclfcn-63/?versions 2024-02-01T10:49:05.291 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Pi4/hQ7kc4gpxZUIgTMa5mdOj7s= 2024-02-01T10:49:05.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Pi4/hQ7kc4gpxZUIgTMa5mdOj7s=', 'Content-Length': '0'} 2024-02-01T10:49:05.297 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9c99d3cbfc12aed-0065bb7721-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:05.297 DEBUG:boto:b'vclfcn-63999falseobj1nulltrue2024-02-01T10:48:26.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:48:26.507Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:48:26.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:48:26.575Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:05.297 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-02-01T10:49:05.297 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-02-01T10:49:05.298 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-02-01T10:49:05.298 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-02-01T10:49:05.298 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:49:05.298 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:49:05.298 DEBUG:boto:path=/vclfcn-63/?versions&max-keys=999 2024-02-01T10:49:05.298 DEBUG:boto:auth_path=/vclfcn-63/?versions&max-keys=999 2024-02-01T10:49:05.298 DEBUG:boto:Method: GET 2024-02-01T10:49:05.298 DEBUG:boto:Path: /vclfcn-63/?versions&max-keys=999 2024-02-01T10:49:05.298 DEBUG:boto:Data: 2024-02-01T10:49:05.299 DEBUG:boto:Headers: {} 2024-02-01T10:49:05.299 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:05.299 DEBUG:boto:Port: 8001 2024-02-01T10:49:05.299 DEBUG:boto:Params: {} 2024-02-01T10:49:05.299 DEBUG:boto:Token: None 2024-02-01T10:49:05.299 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:05 GMT /vclfcn-63/?versions 2024-02-01T10:49:05.299 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Pi4/hQ7kc4gpxZUIgTMa5mdOj7s= 2024-02-01T10:49:05.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Pi4/hQ7kc4gpxZUIgTMa5mdOj7s=', 'Content-Length': '0'} 2024-02-01T10:49:05.304 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fee24659776bf9f3-0065bb7721-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:05.352 DEBUG:boto:b'vclfcn-63999falseobj1nulltrue2024-02-01T10:48:26.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:48:26.507Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:48:26.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:48:26.575Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:05.353 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:49:05.353 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:49:05.353 DEBUG:boto:path=/vclfcn-63/?versions&max-keys=999 2024-02-01T10:49:05.353 DEBUG:boto:auth_path=/vclfcn-63/?versions&max-keys=999 2024-02-01T10:49:05.353 DEBUG:boto:Method: GET 2024-02-01T10:49:05.353 DEBUG:boto:Path: /vclfcn-63/?versions&max-keys=999 2024-02-01T10:49:05.353 DEBUG:boto:Data: 2024-02-01T10:49:05.353 DEBUG:boto:Headers: {} 2024-02-01T10:49:05.353 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:05.354 DEBUG:boto:Port: 8000 2024-02-01T10:49:05.354 DEBUG:boto:Params: {} 2024-02-01T10:49:05.354 DEBUG:boto:Token: None 2024-02-01T10:49:05.354 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:05 GMT /vclfcn-63/?versions 2024-02-01T10:49:05.354 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Pi4/hQ7kc4gpxZUIgTMa5mdOj7s= 2024-02-01T10:49:05.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Pi4/hQ7kc4gpxZUIgTMa5mdOj7s=', 'Content-Length': '0'} 2024-02-01T10:49:05.359 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000046ad7b923462bd4b-0065bb7721-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:05.359 DEBUG:boto:b'vclfcn-63999falseobj1nulltrue2024-02-01T10:48:26.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:48:26.507Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:48:26.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:48:26.575Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:05.359 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:49:05.360 DEBUG:boto:path=/vclfcn-63/obj1 2024-02-01T10:49:05.360 DEBUG:boto:auth_path=/vclfcn-63/obj1 2024-02-01T10:49:05.360 DEBUG:boto:path=/vclfcn-63/obj1?versionId=null 2024-02-01T10:49:05.360 DEBUG:boto:auth_path=/vclfcn-63/obj1?versionId=null 2024-02-01T10:49:05.360 DEBUG:boto:Method: GET 2024-02-01T10:49:05.360 DEBUG:boto:Path: /vclfcn-63/obj1?versionId=null 2024-02-01T10:49:05.360 DEBUG:boto:Data: 2024-02-01T10:49:05.360 DEBUG:boto:Headers: {} 2024-02-01T10:49:05.360 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:05.360 DEBUG:boto:Port: 8001 2024-02-01T10:49:05.361 DEBUG:boto:Params: {} 2024-02-01T10:49:05.361 DEBUG:boto:Token: None 2024-02-01T10:49:05.361 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:05 GMT /vclfcn-63/obj1?versionId=null 2024-02-01T10:49:05.361 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:2ECDJur3uYNvbW7eK3zW2WEt+3s= 2024-02-01T10:49:05.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2ECDJur3uYNvbW7eK3zW2WEt+3s=', 'Content-Length': '0'} 2024-02-01T10:49:05.364 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e364408007ffe061-0065bb7721-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:05.365 DEBUG:boto:path=/vclfcn-63/obj1 2024-02-01T10:49:05.365 DEBUG:boto:auth_path=/vclfcn-63/obj1 2024-02-01T10:49:05.365 DEBUG:boto:path=/vclfcn-63/obj1?versionId=null 2024-02-01T10:49:05.365 DEBUG:boto:auth_path=/vclfcn-63/obj1?versionId=null 2024-02-01T10:49:05.365 DEBUG:boto:Method: GET 2024-02-01T10:49:05.366 DEBUG:boto:Path: /vclfcn-63/obj1?versionId=null 2024-02-01T10:49:05.366 DEBUG:boto:Data: 2024-02-01T10:49:05.366 DEBUG:boto:Headers: {} 2024-02-01T10:49:05.366 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:05.366 DEBUG:boto:Port: 8000 2024-02-01T10:49:05.366 DEBUG:boto:Params: {} 2024-02-01T10:49:05.366 DEBUG:boto:Token: None 2024-02-01T10:49:05.366 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:05 GMT /vclfcn-63/obj1?versionId=null 2024-02-01T10:49:05.367 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:2ECDJur3uYNvbW7eK3zW2WEt+3s= 2024-02-01T10:49:05.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2ECDJur3uYNvbW7eK3zW2WEt+3s=', 'Content-Length': '0'} 2024-02-01T10:49:05.370 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c8257b1588d4f280-0065bb7721-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:05.412 DEBUG:boto:path=/vclfcn-63/obj1 2024-02-01T10:49:05.412 DEBUG:boto:auth_path=/vclfcn-63/obj1 2024-02-01T10:49:05.412 DEBUG:boto:path=/vclfcn-63/obj1?versionId=null 2024-02-01T10:49:05.412 DEBUG:boto:auth_path=/vclfcn-63/obj1?versionId=null 2024-02-01T10:49:05.413 DEBUG:boto:Method: HEAD 2024-02-01T10:49:05.413 DEBUG:boto:Path: /vclfcn-63/obj1?versionId=null 2024-02-01T10:49:05.413 DEBUG:boto:Data: 2024-02-01T10:49:05.413 DEBUG:boto:Headers: {} 2024-02-01T10:49:05.413 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:05.413 DEBUG:boto:Port: 8001 2024-02-01T10:49:05.413 DEBUG:boto:Params: {} 2024-02-01T10:49:05.413 DEBUG:boto:Token: None 2024-02-01T10:49:05.413 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:05 GMT /vclfcn-63/obj1?versionId=null 2024-02-01T10:49:05.414 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BGWbYqJ5JTDOvrvpT3EuvyrWYVw= 2024-02-01T10:49:05.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BGWbYqJ5JTDOvrvpT3EuvyrWYVw=', 'Content-Length': '0'} 2024-02-01T10:49:05.416 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000863db80e6308b348-0065bb7721-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:05.417 DEBUG:boto:path=/vclfcn-63/obj1 2024-02-01T10:49:05.417 DEBUG:boto:auth_path=/vclfcn-63/obj1 2024-02-01T10:49:05.417 DEBUG:boto:path=/vclfcn-63/obj1?versionId=null 2024-02-01T10:49:05.417 DEBUG:boto:auth_path=/vclfcn-63/obj1?versionId=null 2024-02-01T10:49:05.417 DEBUG:boto:Method: HEAD 2024-02-01T10:49:05.417 DEBUG:boto:Path: /vclfcn-63/obj1?versionId=null 2024-02-01T10:49:05.417 DEBUG:boto:Data: 2024-02-01T10:49:05.417 DEBUG:boto:Headers: {} 2024-02-01T10:49:05.417 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:05.417 DEBUG:boto:Port: 8000 2024-02-01T10:49:05.417 DEBUG:boto:Params: {} 2024-02-01T10:49:05.418 DEBUG:boto:Token: None 2024-02-01T10:49:05.418 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:05 GMT /vclfcn-63/obj1?versionId=null 2024-02-01T10:49:05.418 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BGWbYqJ5JTDOvrvpT3EuvyrWYVw= 2024-02-01T10:49:05.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BGWbYqJ5JTDOvrvpT3EuvyrWYVw=', 'Content-Length': '0'} 2024-02-01T10:49:05.421 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000efa860dc70409d33-0065bb7721-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:05.421 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:49:05.421 DEBUG:boto:path=/vclfcn-63/obj1 2024-02-01T10:49:05.421 DEBUG:boto:auth_path=/vclfcn-63/obj1 2024-02-01T10:49:05.421 DEBUG:boto:path=/vclfcn-63/obj1?versionId=null 2024-02-01T10:49:05.421 DEBUG:boto:auth_path=/vclfcn-63/obj1?versionId=null 2024-02-01T10:49:05.421 DEBUG:boto:Method: GET 2024-02-01T10:49:05.421 DEBUG:boto:Path: /vclfcn-63/obj1?versionId=null 2024-02-01T10:49:05.422 DEBUG:boto:Data: 2024-02-01T10:49:05.422 DEBUG:boto:Headers: {} 2024-02-01T10:49:05.422 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:05.422 DEBUG:boto:Port: 8001 2024-02-01T10:49:05.422 DEBUG:boto:Params: {} 2024-02-01T10:49:05.422 DEBUG:boto:Token: None 2024-02-01T10:49:05.422 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:05 GMT /vclfcn-63/obj1?versionId=null 2024-02-01T10:49:05.422 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:2ECDJur3uYNvbW7eK3zW2WEt+3s= 2024-02-01T10:49:05.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2ECDJur3uYNvbW7eK3zW2WEt+3s=', 'Content-Length': '0'} 2024-02-01T10:49:05.425 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003898f6d9228964b9-0065bb7721-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:05.468 DEBUG:boto:path=/vclfcn-63/obj1 2024-02-01T10:49:05.468 DEBUG:boto:auth_path=/vclfcn-63/obj1 2024-02-01T10:49:05.468 DEBUG:boto:path=/vclfcn-63/obj1?versionId=null 2024-02-01T10:49:05.469 DEBUG:boto:auth_path=/vclfcn-63/obj1?versionId=null 2024-02-01T10:49:05.469 DEBUG:boto:Method: GET 2024-02-01T10:49:05.469 DEBUG:boto:Path: /vclfcn-63/obj1?versionId=null 2024-02-01T10:49:05.469 DEBUG:boto:Data: 2024-02-01T10:49:05.469 DEBUG:boto:Headers: {} 2024-02-01T10:49:05.469 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:05.469 DEBUG:boto:Port: 8000 2024-02-01T10:49:05.469 DEBUG:boto:Params: {} 2024-02-01T10:49:05.469 DEBUG:boto:Token: None 2024-02-01T10:49:05.470 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:05 GMT /vclfcn-63/obj1?versionId=null 2024-02-01T10:49:05.470 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:2ECDJur3uYNvbW7eK3zW2WEt+3s= 2024-02-01T10:49:05.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2ECDJur3uYNvbW7eK3zW2WEt+3s=', 'Content-Length': '0'} 2024-02-01T10:49:05.473 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f75f0bf12499d82f-0065bb7721-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:05.473 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:49:05.473 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:49:05.473 DEBUG:boto:path=/vclfcn-63/?prefix=obj1 2024-02-01T10:49:05.473 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj1 2024-02-01T10:49:05.474 DEBUG:boto:Method: GET 2024-02-01T10:49:05.474 DEBUG:boto:Path: /vclfcn-63/?prefix=obj1 2024-02-01T10:49:05.474 DEBUG:boto:Data: 2024-02-01T10:49:05.474 DEBUG:boto:Headers: {} 2024-02-01T10:49:05.474 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:05.474 DEBUG:boto:Port: 8001 2024-02-01T10:49:05.474 DEBUG:boto:Params: {} 2024-02-01T10:49:05.474 DEBUG:boto:Token: None 2024-02-01T10:49:05.474 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:05 GMT /vclfcn-63/ 2024-02-01T10:49:05.474 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0gAczpIV9i1WJPU8kC3CyH/Ylio= 2024-02-01T10:49:05.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0gAczpIV9i1WJPU8kC3CyH/Ylio=', 'Content-Length': '0'} 2024-02-01T10:49:05.479 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da4fbb6ae7f75e59-0065bb7721-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:05.520 DEBUG:boto:b'vclfcn-63obj11000falseobj12024-02-01T10:48:26.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:05.521 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:49:05.521 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:49:05.521 DEBUG:boto:path=/vclfcn-63/?prefix=obj1 2024-02-01T10:49:05.521 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj1 2024-02-01T10:49:05.521 DEBUG:boto:Method: GET 2024-02-01T10:49:05.521 DEBUG:boto:Path: /vclfcn-63/?prefix=obj1 2024-02-01T10:49:05.521 DEBUG:boto:Data: 2024-02-01T10:49:05.521 DEBUG:boto:Headers: {} 2024-02-01T10:49:05.522 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:05.522 DEBUG:boto:Port: 8000 2024-02-01T10:49:05.522 DEBUG:boto:Params: {} 2024-02-01T10:49:05.522 DEBUG:boto:Token: None 2024-02-01T10:49:05.522 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:05 GMT /vclfcn-63/ 2024-02-01T10:49:05.522 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0gAczpIV9i1WJPU8kC3CyH/Ylio= 2024-02-01T10:49:05.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0gAczpIV9i1WJPU8kC3CyH/Ylio=', 'Content-Length': '0'} 2024-02-01T10:49:05.526 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3b8bdce9c8ae270-0065bb7721-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:05.527 DEBUG:boto:b'vclfcn-63obj11000falseobj12024-02-01T10:48:26.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:05.527 DEBUG:boto:path=/vclfcn-63/obj1 2024-02-01T10:49:05.527 DEBUG:boto:auth_path=/vclfcn-63/obj1 2024-02-01T10:49:05.527 DEBUG:boto:Method: HEAD 2024-02-01T10:49:05.527 DEBUG:boto:Path: /vclfcn-63/obj1 2024-02-01T10:49:05.527 DEBUG:boto:Data: 2024-02-01T10:49:05.527 DEBUG:boto:Headers: {} 2024-02-01T10:49:05.527 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:05.528 DEBUG:boto:Port: 8001 2024-02-01T10:49:05.528 DEBUG:boto:Params: {} 2024-02-01T10:49:05.528 DEBUG:boto:Token: None 2024-02-01T10:49:05.528 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:05 GMT /vclfcn-63/obj1 2024-02-01T10:49:05.528 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:84KAuGJ/Lrehm/1xLen1mUPojy4= 2024-02-01T10:49:05.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:84KAuGJ/Lrehm/1xLen1mUPojy4=', 'Content-Length': '0'} 2024-02-01T10:49:05.531 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000336b1a7eab851c1a-0065bb7721-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:05.531 DEBUG:boto:path=/vclfcn-63/obj1 2024-02-01T10:49:05.531 DEBUG:boto:auth_path=/vclfcn-63/obj1 2024-02-01T10:49:05.531 DEBUG:boto:Method: HEAD 2024-02-01T10:49:05.532 DEBUG:boto:Path: /vclfcn-63/obj1 2024-02-01T10:49:05.532 DEBUG:boto:Data: 2024-02-01T10:49:05.532 DEBUG:boto:Headers: {} 2024-02-01T10:49:05.532 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:05.532 DEBUG:boto:Port: 8000 2024-02-01T10:49:05.532 DEBUG:boto:Params: {} 2024-02-01T10:49:05.532 DEBUG:boto:Token: None 2024-02-01T10:49:05.532 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:05 GMT /vclfcn-63/obj1 2024-02-01T10:49:05.533 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:84KAuGJ/Lrehm/1xLen1mUPojy4= 2024-02-01T10:49:05.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:84KAuGJ/Lrehm/1xLen1mUPojy4=', 'Content-Length': '0'} 2024-02-01T10:49:05.535 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a31276ec1633d8ad-0065bb7721-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:05.536 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:49:05.536 DEBUG:boto:path=/vclfcn-63/obj1 2024-02-01T10:49:05.536 DEBUG:boto:auth_path=/vclfcn-63/obj1 2024-02-01T10:49:05.536 DEBUG:boto:Method: GET 2024-02-01T10:49:05.536 DEBUG:boto:Path: /vclfcn-63/obj1 2024-02-01T10:49:05.536 DEBUG:boto:Data: 2024-02-01T10:49:05.536 DEBUG:boto:Headers: {} 2024-02-01T10:49:05.536 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:05.537 DEBUG:boto:Port: 8001 2024-02-01T10:49:05.537 DEBUG:boto:Params: {} 2024-02-01T10:49:05.537 DEBUG:boto:Token: None 2024-02-01T10:49:05.537 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:05 GMT /vclfcn-63/obj1 2024-02-01T10:49:05.537 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:JQQBNWAjYyC89k3RiZn1gEd6+mc= 2024-02-01T10:49:05.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:JQQBNWAjYyC89k3RiZn1gEd6+mc=', 'Content-Length': '0'} 2024-02-01T10:49:05.540 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f313144506e6f669-0065bb7721-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:05.588 DEBUG:boto:path=/vclfcn-63/obj1 2024-02-01T10:49:05.588 DEBUG:boto:auth_path=/vclfcn-63/obj1 2024-02-01T10:49:05.589 DEBUG:boto:Method: GET 2024-02-01T10:49:05.589 DEBUG:boto:Path: /vclfcn-63/obj1 2024-02-01T10:49:05.589 DEBUG:boto:Data: 2024-02-01T10:49:05.589 DEBUG:boto:Headers: {} 2024-02-01T10:49:05.589 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:05.589 DEBUG:boto:Port: 8000 2024-02-01T10:49:05.589 DEBUG:boto:Params: {} 2024-02-01T10:49:05.589 DEBUG:boto:Token: None 2024-02-01T10:49:05.589 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:05 GMT /vclfcn-63/obj1 2024-02-01T10:49:05.590 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:JQQBNWAjYyC89k3RiZn1gEd6+mc= 2024-02-01T10:49:05.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:JQQBNWAjYyC89k3RiZn1gEd6+mc=', 'Content-Length': '0'} 2024-02-01T10:49:05.592 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004fa0ae6a3db400e9-0065bb7721-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:05.593 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:49:05.593 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:49:05.593 DEBUG:boto:path=/vclfcn-63/?prefix=obj1 2024-02-01T10:49:05.593 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj1 2024-02-01T10:49:05.593 DEBUG:boto:Method: GET 2024-02-01T10:49:05.593 DEBUG:boto:Path: /vclfcn-63/?prefix=obj1 2024-02-01T10:49:05.593 DEBUG:boto:Data: 2024-02-01T10:49:05.593 DEBUG:boto:Headers: {} 2024-02-01T10:49:05.594 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:05.594 DEBUG:boto:Port: 8001 2024-02-01T10:49:05.594 DEBUG:boto:Params: {} 2024-02-01T10:49:05.594 DEBUG:boto:Token: None 2024-02-01T10:49:05.594 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:05 GMT /vclfcn-63/ 2024-02-01T10:49:05.594 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0gAczpIV9i1WJPU8kC3CyH/Ylio= 2024-02-01T10:49:05.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0gAczpIV9i1WJPU8kC3CyH/Ylio=', 'Content-Length': '0'} 2024-02-01T10:49:05.599 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012c704024494c522-0065bb7721-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:05.599 DEBUG:boto:b'vclfcn-63obj11000falseobj12024-02-01T10:48:26.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:05.599 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:49:05.600 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:49:05.600 DEBUG:boto:path=/vclfcn-63/?prefix=obj1 2024-02-01T10:49:05.600 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj1 2024-02-01T10:49:05.600 DEBUG:boto:Method: GET 2024-02-01T10:49:05.600 DEBUG:boto:Path: /vclfcn-63/?prefix=obj1 2024-02-01T10:49:05.600 DEBUG:boto:Data: 2024-02-01T10:49:05.600 DEBUG:boto:Headers: {} 2024-02-01T10:49:05.600 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:05.600 DEBUG:boto:Port: 8000 2024-02-01T10:49:05.601 DEBUG:boto:Params: {} 2024-02-01T10:49:05.601 DEBUG:boto:Token: None 2024-02-01T10:49:05.601 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:05 GMT /vclfcn-63/ 2024-02-01T10:49:05.601 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0gAczpIV9i1WJPU8kC3CyH/Ylio= 2024-02-01T10:49:05.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0gAczpIV9i1WJPU8kC3CyH/Ylio=', 'Content-Length': '0'} 2024-02-01T10:49:05.607 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000074ec53dcbb3c150f-0065bb7721-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:05.652 DEBUG:boto:b'vclfcn-63obj11000falseobj12024-02-01T10:48:26.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:05.653 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:49:05.653 DEBUG:boto:path=/vclfcn-63/obj2 2024-02-01T10:49:05.653 DEBUG:boto:auth_path=/vclfcn-63/obj2 2024-02-01T10:49:05.653 DEBUG:boto:path=/vclfcn-63/obj2?versionId=null 2024-02-01T10:49:05.653 DEBUG:boto:auth_path=/vclfcn-63/obj2?versionId=null 2024-02-01T10:49:05.653 DEBUG:boto:Method: GET 2024-02-01T10:49:05.653 DEBUG:boto:Path: /vclfcn-63/obj2?versionId=null 2024-02-01T10:49:05.653 DEBUG:boto:Data: 2024-02-01T10:49:05.654 DEBUG:boto:Headers: {} 2024-02-01T10:49:05.654 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:05.654 DEBUG:boto:Port: 8001 2024-02-01T10:49:05.654 DEBUG:boto:Params: {} 2024-02-01T10:49:05.654 DEBUG:boto:Token: None 2024-02-01T10:49:05.654 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:05 GMT /vclfcn-63/obj2?versionId=null 2024-02-01T10:49:05.654 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mjmlXjKBxPbQsASObT5BEAPT5Xo= 2024-02-01T10:49:05.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mjmlXjKBxPbQsASObT5BEAPT5Xo=', 'Content-Length': '0'} 2024-02-01T10:49:05.657 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005a175367fa6db711-0065bb7721-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:05.704 DEBUG:boto:path=/vclfcn-63/obj2 2024-02-01T10:49:05.704 DEBUG:boto:auth_path=/vclfcn-63/obj2 2024-02-01T10:49:05.704 DEBUG:boto:path=/vclfcn-63/obj2?versionId=null 2024-02-01T10:49:05.704 DEBUG:boto:auth_path=/vclfcn-63/obj2?versionId=null 2024-02-01T10:49:05.705 DEBUG:boto:Method: GET 2024-02-01T10:49:05.705 DEBUG:boto:Path: /vclfcn-63/obj2?versionId=null 2024-02-01T10:49:05.705 DEBUG:boto:Data: 2024-02-01T10:49:05.705 DEBUG:boto:Headers: {} 2024-02-01T10:49:05.705 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:05.705 DEBUG:boto:Port: 8000 2024-02-01T10:49:05.705 DEBUG:boto:Params: {} 2024-02-01T10:49:05.705 DEBUG:boto:Token: None 2024-02-01T10:49:05.706 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:05 GMT /vclfcn-63/obj2?versionId=null 2024-02-01T10:49:05.706 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mjmlXjKBxPbQsASObT5BEAPT5Xo= 2024-02-01T10:49:05.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mjmlXjKBxPbQsASObT5BEAPT5Xo=', 'Content-Length': '0'} 2024-02-01T10:49:05.709 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000395fbc7d932172ec-0065bb7721-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:05.709 DEBUG:boto:path=/vclfcn-63/obj2 2024-02-01T10:49:05.709 DEBUG:boto:auth_path=/vclfcn-63/obj2 2024-02-01T10:49:05.709 DEBUG:boto:path=/vclfcn-63/obj2?versionId=null 2024-02-01T10:49:05.709 DEBUG:boto:auth_path=/vclfcn-63/obj2?versionId=null 2024-02-01T10:49:05.710 DEBUG:boto:Method: HEAD 2024-02-01T10:49:05.710 DEBUG:boto:Path: /vclfcn-63/obj2?versionId=null 2024-02-01T10:49:05.710 DEBUG:boto:Data: 2024-02-01T10:49:05.710 DEBUG:boto:Headers: {} 2024-02-01T10:49:05.710 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:05.710 DEBUG:boto:Port: 8001 2024-02-01T10:49:05.710 DEBUG:boto:Params: {} 2024-02-01T10:49:05.710 DEBUG:boto:Token: None 2024-02-01T10:49:05.710 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:05 GMT /vclfcn-63/obj2?versionId=null 2024-02-01T10:49:05.710 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1Bydk4vhtJJ+7ypGwMy1pjzzbqc= 2024-02-01T10:49:05.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1Bydk4vhtJJ+7ypGwMy1pjzzbqc=', 'Content-Length': '0'} 2024-02-01T10:49:05.713 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000795ebc168fa73dd-0065bb7721-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:05.713 DEBUG:boto:path=/vclfcn-63/obj2 2024-02-01T10:49:05.713 DEBUG:boto:auth_path=/vclfcn-63/obj2 2024-02-01T10:49:05.714 DEBUG:boto:path=/vclfcn-63/obj2?versionId=null 2024-02-01T10:49:05.714 DEBUG:boto:auth_path=/vclfcn-63/obj2?versionId=null 2024-02-01T10:49:05.714 DEBUG:boto:Method: HEAD 2024-02-01T10:49:05.714 DEBUG:boto:Path: /vclfcn-63/obj2?versionId=null 2024-02-01T10:49:05.714 DEBUG:boto:Data: 2024-02-01T10:49:05.714 DEBUG:boto:Headers: {} 2024-02-01T10:49:05.714 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:05.714 DEBUG:boto:Port: 8000 2024-02-01T10:49:05.714 DEBUG:boto:Params: {} 2024-02-01T10:49:05.714 DEBUG:boto:Token: None 2024-02-01T10:49:05.715 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:05 GMT /vclfcn-63/obj2?versionId=null 2024-02-01T10:49:05.715 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1Bydk4vhtJJ+7ypGwMy1pjzzbqc= 2024-02-01T10:49:05.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1Bydk4vhtJJ+7ypGwMy1pjzzbqc=', 'Content-Length': '0'} 2024-02-01T10:49:05.717 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000771b7dfd7f6ce63-0065bb7721-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:05.718 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:49:05.718 DEBUG:boto:path=/vclfcn-63/obj2 2024-02-01T10:49:05.718 DEBUG:boto:auth_path=/vclfcn-63/obj2 2024-02-01T10:49:05.718 DEBUG:boto:path=/vclfcn-63/obj2?versionId=null 2024-02-01T10:49:05.718 DEBUG:boto:auth_path=/vclfcn-63/obj2?versionId=null 2024-02-01T10:49:05.718 DEBUG:boto:Method: GET 2024-02-01T10:49:05.718 DEBUG:boto:Path: /vclfcn-63/obj2?versionId=null 2024-02-01T10:49:05.718 DEBUG:boto:Data: 2024-02-01T10:49:05.719 DEBUG:boto:Headers: {} 2024-02-01T10:49:05.719 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:05.719 DEBUG:boto:Port: 8001 2024-02-01T10:49:05.719 DEBUG:boto:Params: {} 2024-02-01T10:49:05.719 DEBUG:boto:Token: None 2024-02-01T10:49:05.719 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:05 GMT /vclfcn-63/obj2?versionId=null 2024-02-01T10:49:05.719 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mjmlXjKBxPbQsASObT5BEAPT5Xo= 2024-02-01T10:49:05.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mjmlXjKBxPbQsASObT5BEAPT5Xo=', 'Content-Length': '0'} 2024-02-01T10:49:05.722 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000139273b21989dd2d-0065bb7721-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:05.768 DEBUG:boto:path=/vclfcn-63/obj2 2024-02-01T10:49:05.768 DEBUG:boto:auth_path=/vclfcn-63/obj2 2024-02-01T10:49:05.768 DEBUG:boto:path=/vclfcn-63/obj2?versionId=null 2024-02-01T10:49:05.768 DEBUG:boto:auth_path=/vclfcn-63/obj2?versionId=null 2024-02-01T10:49:05.769 DEBUG:boto:Method: GET 2024-02-01T10:49:05.769 DEBUG:boto:Path: /vclfcn-63/obj2?versionId=null 2024-02-01T10:49:05.769 DEBUG:boto:Data: 2024-02-01T10:49:05.769 DEBUG:boto:Headers: {} 2024-02-01T10:49:05.769 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:05.769 DEBUG:boto:Port: 8000 2024-02-01T10:49:05.769 DEBUG:boto:Params: {} 2024-02-01T10:49:05.770 DEBUG:boto:Token: None 2024-02-01T10:49:05.770 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:05 GMT /vclfcn-63/obj2?versionId=null 2024-02-01T10:49:05.770 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mjmlXjKBxPbQsASObT5BEAPT5Xo= 2024-02-01T10:49:05.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mjmlXjKBxPbQsASObT5BEAPT5Xo=', 'Content-Length': '0'} 2024-02-01T10:49:05.773 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003d11ef35d498898c-0065bb7721-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:05.773 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:49:05.774 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:49:05.774 DEBUG:boto:path=/vclfcn-63/?prefix=obj2 2024-02-01T10:49:05.774 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj2 2024-02-01T10:49:05.774 DEBUG:boto:Method: GET 2024-02-01T10:49:05.774 DEBUG:boto:Path: /vclfcn-63/?prefix=obj2 2024-02-01T10:49:05.774 DEBUG:boto:Data: 2024-02-01T10:49:05.774 DEBUG:boto:Headers: {} 2024-02-01T10:49:05.774 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:05.774 DEBUG:boto:Port: 8001 2024-02-01T10:49:05.774 DEBUG:boto:Params: {} 2024-02-01T10:49:05.774 DEBUG:boto:Token: None 2024-02-01T10:49:05.775 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:05 GMT /vclfcn-63/ 2024-02-01T10:49:05.775 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0gAczpIV9i1WJPU8kC3CyH/Ylio= 2024-02-01T10:49:05.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0gAczpIV9i1WJPU8kC3CyH/Ylio=', 'Content-Length': '0'} 2024-02-01T10:49:05.779 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b71bd65aacbf5051-0065bb7721-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:05.820 DEBUG:boto:b'vclfcn-63obj21000falseobj22024-02-01T10:48:26.507Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:05.821 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:49:05.821 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:49:05.821 DEBUG:boto:path=/vclfcn-63/?prefix=obj2 2024-02-01T10:49:05.821 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj2 2024-02-01T10:49:05.821 DEBUG:boto:Method: GET 2024-02-01T10:49:05.821 DEBUG:boto:Path: /vclfcn-63/?prefix=obj2 2024-02-01T10:49:05.821 DEBUG:boto:Data: 2024-02-01T10:49:05.821 DEBUG:boto:Headers: {} 2024-02-01T10:49:05.821 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:05.821 DEBUG:boto:Port: 8000 2024-02-01T10:49:05.821 DEBUG:boto:Params: {} 2024-02-01T10:49:05.822 DEBUG:boto:Token: None 2024-02-01T10:49:05.822 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:05 GMT /vclfcn-63/ 2024-02-01T10:49:05.822 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0gAczpIV9i1WJPU8kC3CyH/Ylio= 2024-02-01T10:49:05.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0gAczpIV9i1WJPU8kC3CyH/Ylio=', 'Content-Length': '0'} 2024-02-01T10:49:05.827 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e92491ed9b42786e-0065bb7721-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:05.827 DEBUG:boto:b'vclfcn-63obj21000falseobj22024-02-01T10:48:26.507Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:05.827 DEBUG:boto:path=/vclfcn-63/obj2 2024-02-01T10:49:05.828 DEBUG:boto:auth_path=/vclfcn-63/obj2 2024-02-01T10:49:05.828 DEBUG:boto:Method: HEAD 2024-02-01T10:49:05.828 DEBUG:boto:Path: /vclfcn-63/obj2 2024-02-01T10:49:05.828 DEBUG:boto:Data: 2024-02-01T10:49:05.828 DEBUG:boto:Headers: {} 2024-02-01T10:49:05.828 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:05.828 DEBUG:boto:Port: 8001 2024-02-01T10:49:05.828 DEBUG:boto:Params: {} 2024-02-01T10:49:05.828 DEBUG:boto:Token: None 2024-02-01T10:49:05.828 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:05 GMT /vclfcn-63/obj2 2024-02-01T10:49:05.829 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HaFuCkwHDQpX2BhL1s4UzSI+kto= 2024-02-01T10:49:05.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HaFuCkwHDQpX2BhL1s4UzSI+kto=', 'Content-Length': '0'} 2024-02-01T10:49:05.832 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000038db1c9d0efb59ff-0065bb7721-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:05.832 DEBUG:boto:path=/vclfcn-63/obj2 2024-02-01T10:49:05.832 DEBUG:boto:auth_path=/vclfcn-63/obj2 2024-02-01T10:49:05.832 DEBUG:boto:Method: HEAD 2024-02-01T10:49:05.832 DEBUG:boto:Path: /vclfcn-63/obj2 2024-02-01T10:49:05.832 DEBUG:boto:Data: 2024-02-01T10:49:05.832 DEBUG:boto:Headers: {} 2024-02-01T10:49:05.833 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:05.833 DEBUG:boto:Port: 8000 2024-02-01T10:49:05.833 DEBUG:boto:Params: {} 2024-02-01T10:49:05.833 DEBUG:boto:Token: None 2024-02-01T10:49:05.833 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:05 GMT /vclfcn-63/obj2 2024-02-01T10:49:05.833 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HaFuCkwHDQpX2BhL1s4UzSI+kto= 2024-02-01T10:49:05.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HaFuCkwHDQpX2BhL1s4UzSI+kto=', 'Content-Length': '0'} 2024-02-01T10:49:05.836 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c095eb8862e00357-0065bb7721-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:05.836 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:49:05.836 DEBUG:boto:path=/vclfcn-63/obj2 2024-02-01T10:49:05.836 DEBUG:boto:auth_path=/vclfcn-63/obj2 2024-02-01T10:49:05.836 DEBUG:boto:Method: GET 2024-02-01T10:49:05.836 DEBUG:boto:Path: /vclfcn-63/obj2 2024-02-01T10:49:05.836 DEBUG:boto:Data: 2024-02-01T10:49:05.837 DEBUG:boto:Headers: {} 2024-02-01T10:49:05.837 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:05.837 DEBUG:boto:Port: 8001 2024-02-01T10:49:05.837 DEBUG:boto:Params: {} 2024-02-01T10:49:05.837 DEBUG:boto:Token: None 2024-02-01T10:49:05.837 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:05 GMT /vclfcn-63/obj2 2024-02-01T10:49:05.837 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RIvnpLYlbt3/kkY/kNAMFhxGgRQ= 2024-02-01T10:49:05.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RIvnpLYlbt3/kkY/kNAMFhxGgRQ=', 'Content-Length': '0'} 2024-02-01T10:49:05.840 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f2f63e2f3d041395-0065bb7721-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:05.884 DEBUG:boto:path=/vclfcn-63/obj2 2024-02-01T10:49:05.885 DEBUG:boto:auth_path=/vclfcn-63/obj2 2024-02-01T10:49:05.885 DEBUG:boto:Method: GET 2024-02-01T10:49:05.885 DEBUG:boto:Path: /vclfcn-63/obj2 2024-02-01T10:49:05.885 DEBUG:boto:Data: 2024-02-01T10:49:05.885 DEBUG:boto:Headers: {} 2024-02-01T10:49:05.885 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:05.886 DEBUG:boto:Port: 8000 2024-02-01T10:49:05.886 DEBUG:boto:Params: {} 2024-02-01T10:49:05.886 DEBUG:boto:Token: None 2024-02-01T10:49:05.886 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:05 GMT /vclfcn-63/obj2 2024-02-01T10:49:05.886 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RIvnpLYlbt3/kkY/kNAMFhxGgRQ= 2024-02-01T10:49:05.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RIvnpLYlbt3/kkY/kNAMFhxGgRQ=', 'Content-Length': '0'} 2024-02-01T10:49:05.889 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cd0efb6770fe053e-0065bb7721-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:05.889 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:49:05.890 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:49:05.890 DEBUG:boto:path=/vclfcn-63/?prefix=obj2 2024-02-01T10:49:05.890 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj2 2024-02-01T10:49:05.890 DEBUG:boto:Method: GET 2024-02-01T10:49:05.890 DEBUG:boto:Path: /vclfcn-63/?prefix=obj2 2024-02-01T10:49:05.890 DEBUG:boto:Data: 2024-02-01T10:49:05.890 DEBUG:boto:Headers: {} 2024-02-01T10:49:05.890 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:05.890 DEBUG:boto:Port: 8001 2024-02-01T10:49:05.890 DEBUG:boto:Params: {} 2024-02-01T10:49:05.891 DEBUG:boto:Token: None 2024-02-01T10:49:05.891 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:05 GMT /vclfcn-63/ 2024-02-01T10:49:05.891 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0gAczpIV9i1WJPU8kC3CyH/Ylio= 2024-02-01T10:49:05.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0gAczpIV9i1WJPU8kC3CyH/Ylio=', 'Content-Length': '0'} 2024-02-01T10:49:05.895 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080dbc400b4d5395c-0065bb7721-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:05.940 DEBUG:boto:b'vclfcn-63obj21000falseobj22024-02-01T10:48:26.507Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:05.941 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:49:05.941 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:49:05.941 DEBUG:boto:path=/vclfcn-63/?prefix=obj2 2024-02-01T10:49:05.941 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj2 2024-02-01T10:49:05.941 DEBUG:boto:Method: GET 2024-02-01T10:49:05.941 DEBUG:boto:Path: /vclfcn-63/?prefix=obj2 2024-02-01T10:49:05.941 DEBUG:boto:Data: 2024-02-01T10:49:05.941 DEBUG:boto:Headers: {} 2024-02-01T10:49:05.941 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:05.941 DEBUG:boto:Port: 8000 2024-02-01T10:49:05.942 DEBUG:boto:Params: {} 2024-02-01T10:49:05.942 DEBUG:boto:Token: None 2024-02-01T10:49:05.942 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:05 GMT /vclfcn-63/ 2024-02-01T10:49:05.942 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0gAczpIV9i1WJPU8kC3CyH/Ylio= 2024-02-01T10:49:05.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0gAczpIV9i1WJPU8kC3CyH/Ylio=', 'Content-Length': '0'} 2024-02-01T10:49:05.947 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090522188bee92c2e-0065bb7721-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:05.947 DEBUG:boto:b'vclfcn-63obj21000falseobj22024-02-01T10:48:26.507Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:05.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:49:05.948 DEBUG:boto:path=/vclfcn-63/obj3 2024-02-01T10:49:05.948 DEBUG:boto:auth_path=/vclfcn-63/obj3 2024-02-01T10:49:05.948 DEBUG:boto:path=/vclfcn-63/obj3?versionId=null 2024-02-01T10:49:05.948 DEBUG:boto:auth_path=/vclfcn-63/obj3?versionId=null 2024-02-01T10:49:05.948 DEBUG:boto:Method: GET 2024-02-01T10:49:05.948 DEBUG:boto:Path: /vclfcn-63/obj3?versionId=null 2024-02-01T10:49:05.948 DEBUG:boto:Data: 2024-02-01T10:49:05.948 DEBUG:boto:Headers: {} 2024-02-01T10:49:05.948 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:05.948 DEBUG:boto:Port: 8001 2024-02-01T10:49:05.948 DEBUG:boto:Params: {} 2024-02-01T10:49:05.949 DEBUG:boto:Token: None 2024-02-01T10:49:05.949 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:05 GMT /vclfcn-63/obj3?versionId=null 2024-02-01T10:49:05.949 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BigtkckoXI8lSDbB+OhFiJqHIEI= 2024-02-01T10:49:05.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BigtkckoXI8lSDbB+OhFiJqHIEI=', 'Content-Length': '0'} 2024-02-01T10:49:05.952 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dbd0bd205b21bedb-0065bb7721-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:05.992 DEBUG:boto:path=/vclfcn-63/obj3 2024-02-01T10:49:05.992 DEBUG:boto:auth_path=/vclfcn-63/obj3 2024-02-01T10:49:05.992 DEBUG:boto:path=/vclfcn-63/obj3?versionId=null 2024-02-01T10:49:05.993 DEBUG:boto:auth_path=/vclfcn-63/obj3?versionId=null 2024-02-01T10:49:05.993 DEBUG:boto:Method: GET 2024-02-01T10:49:05.993 DEBUG:boto:Path: /vclfcn-63/obj3?versionId=null 2024-02-01T10:49:05.993 DEBUG:boto:Data: 2024-02-01T10:49:05.993 DEBUG:boto:Headers: {} 2024-02-01T10:49:05.993 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:05.993 DEBUG:boto:Port: 8000 2024-02-01T10:49:05.993 DEBUG:boto:Params: {} 2024-02-01T10:49:05.993 DEBUG:boto:Token: None 2024-02-01T10:49:05.994 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:05 GMT /vclfcn-63/obj3?versionId=null 2024-02-01T10:49:05.994 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BigtkckoXI8lSDbB+OhFiJqHIEI= 2024-02-01T10:49:05.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BigtkckoXI8lSDbB+OhFiJqHIEI=', 'Content-Length': '0'} 2024-02-01T10:49:05.997 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d1a795b85c12185f-0065bb7721-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:05.997 DEBUG:boto:path=/vclfcn-63/obj3 2024-02-01T10:49:05.997 DEBUG:boto:auth_path=/vclfcn-63/obj3 2024-02-01T10:49:05.997 DEBUG:boto:path=/vclfcn-63/obj3?versionId=null 2024-02-01T10:49:05.997 DEBUG:boto:auth_path=/vclfcn-63/obj3?versionId=null 2024-02-01T10:49:05.998 DEBUG:boto:Method: HEAD 2024-02-01T10:49:05.998 DEBUG:boto:Path: /vclfcn-63/obj3?versionId=null 2024-02-01T10:49:05.998 DEBUG:boto:Data: 2024-02-01T10:49:05.998 DEBUG:boto:Headers: {} 2024-02-01T10:49:05.998 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:05.998 DEBUG:boto:Port: 8001 2024-02-01T10:49:05.998 DEBUG:boto:Params: {} 2024-02-01T10:49:05.998 DEBUG:boto:Token: None 2024-02-01T10:49:05.998 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:05 GMT /vclfcn-63/obj3?versionId=null 2024-02-01T10:49:05.998 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:K3Rr8tVj2QB/VsUqM8S7LaX5W6w= 2024-02-01T10:49:05.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:K3Rr8tVj2QB/VsUqM8S7LaX5W6w=', 'Content-Length': '0'} 2024-02-01T10:49:06.001 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f487e246b8158a7c-0065bb7721-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:06.002 DEBUG:boto:path=/vclfcn-63/obj3 2024-02-01T10:49:06.002 DEBUG:boto:auth_path=/vclfcn-63/obj3 2024-02-01T10:49:06.002 DEBUG:boto:path=/vclfcn-63/obj3?versionId=null 2024-02-01T10:49:06.002 DEBUG:boto:auth_path=/vclfcn-63/obj3?versionId=null 2024-02-01T10:49:06.002 DEBUG:boto:Method: HEAD 2024-02-01T10:49:06.002 DEBUG:boto:Path: /vclfcn-63/obj3?versionId=null 2024-02-01T10:49:06.002 DEBUG:boto:Data: 2024-02-01T10:49:06.002 DEBUG:boto:Headers: {} 2024-02-01T10:49:06.002 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:06.002 DEBUG:boto:Port: 8000 2024-02-01T10:49:06.003 DEBUG:boto:Params: {} 2024-02-01T10:49:06.003 DEBUG:boto:Token: None 2024-02-01T10:49:06.003 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:06 GMT /vclfcn-63/obj3?versionId=null 2024-02-01T10:49:06.003 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:SxUSjK5QOoHerV8wvtgF4THCAus= 2024-02-01T10:49:06.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:SxUSjK5QOoHerV8wvtgF4THCAus=', 'Content-Length': '0'} 2024-02-01T10:49:06.006 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fcd4b9e870a38c2e-0065bb7722-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:06.006 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:49:06.006 DEBUG:boto:path=/vclfcn-63/obj3 2024-02-01T10:49:06.006 DEBUG:boto:auth_path=/vclfcn-63/obj3 2024-02-01T10:49:06.007 DEBUG:boto:path=/vclfcn-63/obj3?versionId=null 2024-02-01T10:49:06.007 DEBUG:boto:auth_path=/vclfcn-63/obj3?versionId=null 2024-02-01T10:49:06.007 DEBUG:boto:Method: GET 2024-02-01T10:49:06.007 DEBUG:boto:Path: /vclfcn-63/obj3?versionId=null 2024-02-01T10:49:06.007 DEBUG:boto:Data: 2024-02-01T10:49:06.007 DEBUG:boto:Headers: {} 2024-02-01T10:49:06.007 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:06.007 DEBUG:boto:Port: 8001 2024-02-01T10:49:06.007 DEBUG:boto:Params: {} 2024-02-01T10:49:06.007 DEBUG:boto:Token: None 2024-02-01T10:49:06.008 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:06 GMT /vclfcn-63/obj3?versionId=null 2024-02-01T10:49:06.008 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:4XsFZFy6hfadARj/b8vpliw5ioc= 2024-02-01T10:49:06.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:4XsFZFy6hfadARj/b8vpliw5ioc=', 'Content-Length': '0'} 2024-02-01T10:49:06.011 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000090f3ab489717fcae-0065bb7722-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:06.056 DEBUG:boto:path=/vclfcn-63/obj3 2024-02-01T10:49:06.056 DEBUG:boto:auth_path=/vclfcn-63/obj3 2024-02-01T10:49:06.056 DEBUG:boto:path=/vclfcn-63/obj3?versionId=null 2024-02-01T10:49:06.056 DEBUG:boto:auth_path=/vclfcn-63/obj3?versionId=null 2024-02-01T10:49:06.057 DEBUG:boto:Method: GET 2024-02-01T10:49:06.057 DEBUG:boto:Path: /vclfcn-63/obj3?versionId=null 2024-02-01T10:49:06.057 DEBUG:boto:Data: 2024-02-01T10:49:06.057 DEBUG:boto:Headers: {} 2024-02-01T10:49:06.057 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:06.057 DEBUG:boto:Port: 8000 2024-02-01T10:49:06.057 DEBUG:boto:Params: {} 2024-02-01T10:49:06.057 DEBUG:boto:Token: None 2024-02-01T10:49:06.057 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:06 GMT /vclfcn-63/obj3?versionId=null 2024-02-01T10:49:06.058 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:4XsFZFy6hfadARj/b8vpliw5ioc= 2024-02-01T10:49:06.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:4XsFZFy6hfadARj/b8vpliw5ioc=', 'Content-Length': '0'} 2024-02-01T10:49:06.061 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a13da57ea78444a5-0065bb7722-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:06.061 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:49:06.061 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:49:06.061 DEBUG:boto:path=/vclfcn-63/?prefix=obj3 2024-02-01T10:49:06.061 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj3 2024-02-01T10:49:06.061 DEBUG:boto:Method: GET 2024-02-01T10:49:06.061 DEBUG:boto:Path: /vclfcn-63/?prefix=obj3 2024-02-01T10:49:06.061 DEBUG:boto:Data: 2024-02-01T10:49:06.062 DEBUG:boto:Headers: {} 2024-02-01T10:49:06.062 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:06.062 DEBUG:boto:Port: 8001 2024-02-01T10:49:06.062 DEBUG:boto:Params: {} 2024-02-01T10:49:06.062 DEBUG:boto:Token: None 2024-02-01T10:49:06.062 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:06 GMT /vclfcn-63/ 2024-02-01T10:49:06.062 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9QIjbFPjM4qH0s4RK4qNKMf/9i8= 2024-02-01T10:49:06.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9QIjbFPjM4qH0s4RK4qNKMf/9i8=', 'Content-Length': '0'} 2024-02-01T10:49:06.066 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fee9e238de626716-0065bb7722-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:06.108 DEBUG:boto:b'vclfcn-63obj31000falseobj32024-02-01T10:48:26.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:06.109 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:49:06.110 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:49:06.110 DEBUG:boto:path=/vclfcn-63/?prefix=obj3 2024-02-01T10:49:06.110 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj3 2024-02-01T10:49:06.110 DEBUG:boto:Method: GET 2024-02-01T10:49:06.110 DEBUG:boto:Path: /vclfcn-63/?prefix=obj3 2024-02-01T10:49:06.110 DEBUG:boto:Data: 2024-02-01T10:49:06.110 DEBUG:boto:Headers: {} 2024-02-01T10:49:06.110 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:06.110 DEBUG:boto:Port: 8000 2024-02-01T10:49:06.111 DEBUG:boto:Params: {} 2024-02-01T10:49:06.111 DEBUG:boto:Token: None 2024-02-01T10:49:06.111 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:06 GMT /vclfcn-63/ 2024-02-01T10:49:06.111 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9QIjbFPjM4qH0s4RK4qNKMf/9i8= 2024-02-01T10:49:06.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9QIjbFPjM4qH0s4RK4qNKMf/9i8=', 'Content-Length': '0'} 2024-02-01T10:49:06.116 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1e1a77763ef52b1-0065bb7722-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:06.116 DEBUG:boto:b'vclfcn-63obj31000falseobj32024-02-01T10:48:26.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:06.117 DEBUG:boto:path=/vclfcn-63/obj3 2024-02-01T10:49:06.117 DEBUG:boto:auth_path=/vclfcn-63/obj3 2024-02-01T10:49:06.117 DEBUG:boto:Method: HEAD 2024-02-01T10:49:06.117 DEBUG:boto:Path: /vclfcn-63/obj3 2024-02-01T10:49:06.117 DEBUG:boto:Data: 2024-02-01T10:49:06.117 DEBUG:boto:Headers: {} 2024-02-01T10:49:06.117 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:06.118 DEBUG:boto:Port: 8001 2024-02-01T10:49:06.118 DEBUG:boto:Params: {} 2024-02-01T10:49:06.118 DEBUG:boto:Token: None 2024-02-01T10:49:06.118 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:06 GMT /vclfcn-63/obj3 2024-02-01T10:49:06.119 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:k1zN68uIFh7zX0ZsmaAqiY5ELz0= 2024-02-01T10:49:06.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:k1zN68uIFh7zX0ZsmaAqiY5ELz0=', 'Content-Length': '0'} 2024-02-01T10:49:06.122 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a0c4b94f23b3afee-0065bb7722-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:06.122 DEBUG:boto:path=/vclfcn-63/obj3 2024-02-01T10:49:06.122 DEBUG:boto:auth_path=/vclfcn-63/obj3 2024-02-01T10:49:06.122 DEBUG:boto:Method: HEAD 2024-02-01T10:49:06.123 DEBUG:boto:Path: /vclfcn-63/obj3 2024-02-01T10:49:06.123 DEBUG:boto:Data: 2024-02-01T10:49:06.123 DEBUG:boto:Headers: {} 2024-02-01T10:49:06.123 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:06.123 DEBUG:boto:Port: 8000 2024-02-01T10:49:06.123 DEBUG:boto:Params: {} 2024-02-01T10:49:06.123 DEBUG:boto:Token: None 2024-02-01T10:49:06.123 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:06 GMT /vclfcn-63/obj3 2024-02-01T10:49:06.123 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:k1zN68uIFh7zX0ZsmaAqiY5ELz0= 2024-02-01T10:49:06.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:k1zN68uIFh7zX0ZsmaAqiY5ELz0=', 'Content-Length': '0'} 2024-02-01T10:49:06.126 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000039d19f5ffb26f05a-0065bb7722-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:06.126 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:49:06.126 DEBUG:boto:path=/vclfcn-63/obj3 2024-02-01T10:49:06.127 DEBUG:boto:auth_path=/vclfcn-63/obj3 2024-02-01T10:49:06.127 DEBUG:boto:Method: GET 2024-02-01T10:49:06.127 DEBUG:boto:Path: /vclfcn-63/obj3 2024-02-01T10:49:06.127 DEBUG:boto:Data: 2024-02-01T10:49:06.127 DEBUG:boto:Headers: {} 2024-02-01T10:49:06.127 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:06.127 DEBUG:boto:Port: 8001 2024-02-01T10:49:06.127 DEBUG:boto:Params: {} 2024-02-01T10:49:06.127 DEBUG:boto:Token: None 2024-02-01T10:49:06.127 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:06 GMT /vclfcn-63/obj3 2024-02-01T10:49:06.128 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:UQQ3MI/xPGmoRU9kIW8SbeN43fk= 2024-02-01T10:49:06.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:UQQ3MI/xPGmoRU9kIW8SbeN43fk=', 'Content-Length': '0'} 2024-02-01T10:49:06.131 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000036387576906e52aa-0065bb7722-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:06.176 DEBUG:boto:path=/vclfcn-63/obj3 2024-02-01T10:49:06.176 DEBUG:boto:auth_path=/vclfcn-63/obj3 2024-02-01T10:49:06.177 DEBUG:boto:Method: GET 2024-02-01T10:49:06.177 DEBUG:boto:Path: /vclfcn-63/obj3 2024-02-01T10:49:06.177 DEBUG:boto:Data: 2024-02-01T10:49:06.177 DEBUG:boto:Headers: {} 2024-02-01T10:49:06.177 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:06.177 DEBUG:boto:Port: 8000 2024-02-01T10:49:06.177 DEBUG:boto:Params: {} 2024-02-01T10:49:06.177 DEBUG:boto:Token: None 2024-02-01T10:49:06.177 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:06 GMT /vclfcn-63/obj3 2024-02-01T10:49:06.178 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:UQQ3MI/xPGmoRU9kIW8SbeN43fk= 2024-02-01T10:49:06.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:UQQ3MI/xPGmoRU9kIW8SbeN43fk=', 'Content-Length': '0'} 2024-02-01T10:49:06.181 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000d2d9d9626d39e86-0065bb7722-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:06.181 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:49:06.181 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:49:06.181 DEBUG:boto:path=/vclfcn-63/?prefix=obj3 2024-02-01T10:49:06.182 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj3 2024-02-01T10:49:06.182 DEBUG:boto:Method: GET 2024-02-01T10:49:06.182 DEBUG:boto:Path: /vclfcn-63/?prefix=obj3 2024-02-01T10:49:06.182 DEBUG:boto:Data: 2024-02-01T10:49:06.182 DEBUG:boto:Headers: {} 2024-02-01T10:49:06.182 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:06.182 DEBUG:boto:Port: 8001 2024-02-01T10:49:06.182 DEBUG:boto:Params: {} 2024-02-01T10:49:06.182 DEBUG:boto:Token: None 2024-02-01T10:49:06.183 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:06 GMT /vclfcn-63/ 2024-02-01T10:49:06.183 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9QIjbFPjM4qH0s4RK4qNKMf/9i8= 2024-02-01T10:49:06.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9QIjbFPjM4qH0s4RK4qNKMf/9i8=', 'Content-Length': '0'} 2024-02-01T10:49:06.188 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083bda6f0a141a5b1-0065bb7722-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:06.236 DEBUG:boto:b'vclfcn-63obj31000falseobj32024-02-01T10:48:26.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:06.237 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:49:06.237 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:49:06.237 DEBUG:boto:path=/vclfcn-63/?prefix=obj3 2024-02-01T10:49:06.237 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj3 2024-02-01T10:49:06.237 DEBUG:boto:Method: GET 2024-02-01T10:49:06.237 DEBUG:boto:Path: /vclfcn-63/?prefix=obj3 2024-02-01T10:49:06.237 DEBUG:boto:Data: 2024-02-01T10:49:06.237 DEBUG:boto:Headers: {} 2024-02-01T10:49:06.237 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:06.237 DEBUG:boto:Port: 8000 2024-02-01T10:49:06.238 DEBUG:boto:Params: {} 2024-02-01T10:49:06.238 DEBUG:boto:Token: None 2024-02-01T10:49:06.238 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:06 GMT /vclfcn-63/ 2024-02-01T10:49:06.238 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9QIjbFPjM4qH0s4RK4qNKMf/9i8= 2024-02-01T10:49:06.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9QIjbFPjM4qH0s4RK4qNKMf/9i8=', 'Content-Length': '0'} 2024-02-01T10:49:06.243 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a21a74fc6585784e-0065bb7722-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:06.243 DEBUG:boto:b'vclfcn-63obj31000falseobj32024-02-01T10:48:26.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:06.243 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:49:06.243 DEBUG:boto:path=/vclfcn-63/obj4 2024-02-01T10:49:06.243 DEBUG:boto:auth_path=/vclfcn-63/obj4 2024-02-01T10:49:06.244 DEBUG:boto:path=/vclfcn-63/obj4?versionId=null 2024-02-01T10:49:06.244 DEBUG:boto:auth_path=/vclfcn-63/obj4?versionId=null 2024-02-01T10:49:06.244 DEBUG:boto:Method: GET 2024-02-01T10:49:06.244 DEBUG:boto:Path: /vclfcn-63/obj4?versionId=null 2024-02-01T10:49:06.244 DEBUG:boto:Data: 2024-02-01T10:49:06.244 DEBUG:boto:Headers: {} 2024-02-01T10:49:06.244 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:06.244 DEBUG:boto:Port: 8001 2024-02-01T10:49:06.244 DEBUG:boto:Params: {} 2024-02-01T10:49:06.244 DEBUG:boto:Token: None 2024-02-01T10:49:06.244 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:06 GMT /vclfcn-63/obj4?versionId=null 2024-02-01T10:49:06.245 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:25J8+RSPiYwczQK0O0K5I7f3WdA= 2024-02-01T10:49:06.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:25J8+RSPiYwczQK0O0K5I7f3WdA=', 'Content-Length': '0'} 2024-02-01T10:49:06.248 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000facd452b89bf1477-0065bb7722-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:06.248 DEBUG:boto:path=/vclfcn-63/obj4 2024-02-01T10:49:06.248 DEBUG:boto:auth_path=/vclfcn-63/obj4 2024-02-01T10:49:06.248 DEBUG:boto:path=/vclfcn-63/obj4?versionId=null 2024-02-01T10:49:06.249 DEBUG:boto:auth_path=/vclfcn-63/obj4?versionId=null 2024-02-01T10:49:06.249 DEBUG:boto:Method: GET 2024-02-01T10:49:06.249 DEBUG:boto:Path: /vclfcn-63/obj4?versionId=null 2024-02-01T10:49:06.249 DEBUG:boto:Data: 2024-02-01T10:49:06.249 DEBUG:boto:Headers: {} 2024-02-01T10:49:06.249 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:06.249 DEBUG:boto:Port: 8000 2024-02-01T10:49:06.249 DEBUG:boto:Params: {} 2024-02-01T10:49:06.249 DEBUG:boto:Token: None 2024-02-01T10:49:06.249 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:06 GMT /vclfcn-63/obj4?versionId=null 2024-02-01T10:49:06.250 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:25J8+RSPiYwczQK0O0K5I7f3WdA= 2024-02-01T10:49:06.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:25J8+RSPiYwczQK0O0K5I7f3WdA=', 'Content-Length': '0'} 2024-02-01T10:49:06.253 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000092ecc7e822bfe34f-0065bb7722-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:06.253 DEBUG:boto:path=/vclfcn-63/obj4 2024-02-01T10:49:06.253 DEBUG:boto:auth_path=/vclfcn-63/obj4 2024-02-01T10:49:06.253 DEBUG:boto:path=/vclfcn-63/obj4?versionId=null 2024-02-01T10:49:06.253 DEBUG:boto:auth_path=/vclfcn-63/obj4?versionId=null 2024-02-01T10:49:06.253 DEBUG:boto:Method: HEAD 2024-02-01T10:49:06.254 DEBUG:boto:Path: /vclfcn-63/obj4?versionId=null 2024-02-01T10:49:06.254 DEBUG:boto:Data: 2024-02-01T10:49:06.254 DEBUG:boto:Headers: {} 2024-02-01T10:49:06.254 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:06.254 DEBUG:boto:Port: 8001 2024-02-01T10:49:06.254 DEBUG:boto:Params: {} 2024-02-01T10:49:06.254 DEBUG:boto:Token: None 2024-02-01T10:49:06.254 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:06 GMT /vclfcn-63/obj4?versionId=null 2024-02-01T10:49:06.254 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:yTI9KgSiq87KhDQ+0IDduBS1wS0= 2024-02-01T10:49:06.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:yTI9KgSiq87KhDQ+0IDduBS1wS0=', 'Content-Length': '0'} 2024-02-01T10:49:06.257 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000000898ec80bd3e003-0065bb7722-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:06.257 DEBUG:boto:path=/vclfcn-63/obj4 2024-02-01T10:49:06.257 DEBUG:boto:auth_path=/vclfcn-63/obj4 2024-02-01T10:49:06.257 DEBUG:boto:path=/vclfcn-63/obj4?versionId=null 2024-02-01T10:49:06.258 DEBUG:boto:auth_path=/vclfcn-63/obj4?versionId=null 2024-02-01T10:49:06.258 DEBUG:boto:Method: HEAD 2024-02-01T10:49:06.258 DEBUG:boto:Path: /vclfcn-63/obj4?versionId=null 2024-02-01T10:49:06.258 DEBUG:boto:Data: 2024-02-01T10:49:06.258 DEBUG:boto:Headers: {} 2024-02-01T10:49:06.258 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:06.258 DEBUG:boto:Port: 8000 2024-02-01T10:49:06.258 DEBUG:boto:Params: {} 2024-02-01T10:49:06.258 DEBUG:boto:Token: None 2024-02-01T10:49:06.259 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:06 GMT /vclfcn-63/obj4?versionId=null 2024-02-01T10:49:06.259 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:yTI9KgSiq87KhDQ+0IDduBS1wS0= 2024-02-01T10:49:06.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:yTI9KgSiq87KhDQ+0IDduBS1wS0=', 'Content-Length': '0'} 2024-02-01T10:49:06.261 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f6f862e3fe6b115e-0065bb7722-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:06.262 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:49:06.262 DEBUG:boto:path=/vclfcn-63/obj4 2024-02-01T10:49:06.262 DEBUG:boto:auth_path=/vclfcn-63/obj4 2024-02-01T10:49:06.262 DEBUG:boto:path=/vclfcn-63/obj4?versionId=null 2024-02-01T10:49:06.262 DEBUG:boto:auth_path=/vclfcn-63/obj4?versionId=null 2024-02-01T10:49:06.262 DEBUG:boto:Method: GET 2024-02-01T10:49:06.262 DEBUG:boto:Path: /vclfcn-63/obj4?versionId=null 2024-02-01T10:49:06.262 DEBUG:boto:Data: 2024-02-01T10:49:06.262 DEBUG:boto:Headers: {} 2024-02-01T10:49:06.262 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:06.263 DEBUG:boto:Port: 8001 2024-02-01T10:49:06.263 DEBUG:boto:Params: {} 2024-02-01T10:49:06.263 DEBUG:boto:Token: None 2024-02-01T10:49:06.263 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:06 GMT /vclfcn-63/obj4?versionId=null 2024-02-01T10:49:06.263 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:25J8+RSPiYwczQK0O0K5I7f3WdA= 2024-02-01T10:49:06.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:25J8+RSPiYwczQK0O0K5I7f3WdA=', 'Content-Length': '0'} 2024-02-01T10:49:06.266 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000897b7345e9ae7ccc-0065bb7722-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:06.308 DEBUG:boto:path=/vclfcn-63/obj4 2024-02-01T10:49:06.308 DEBUG:boto:auth_path=/vclfcn-63/obj4 2024-02-01T10:49:06.308 DEBUG:boto:path=/vclfcn-63/obj4?versionId=null 2024-02-01T10:49:06.309 DEBUG:boto:auth_path=/vclfcn-63/obj4?versionId=null 2024-02-01T10:49:06.309 DEBUG:boto:Method: GET 2024-02-01T10:49:06.309 DEBUG:boto:Path: /vclfcn-63/obj4?versionId=null 2024-02-01T10:49:06.309 DEBUG:boto:Data: 2024-02-01T10:49:06.309 DEBUG:boto:Headers: {} 2024-02-01T10:49:06.309 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:06.309 DEBUG:boto:Port: 8000 2024-02-01T10:49:06.309 DEBUG:boto:Params: {} 2024-02-01T10:49:06.309 DEBUG:boto:Token: None 2024-02-01T10:49:06.310 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:06 GMT /vclfcn-63/obj4?versionId=null 2024-02-01T10:49:06.310 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:25J8+RSPiYwczQK0O0K5I7f3WdA= 2024-02-01T10:49:06.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:25J8+RSPiYwczQK0O0K5I7f3WdA=', 'Content-Length': '0'} 2024-02-01T10:49:06.313 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005bb565b3d57be6b2-0065bb7722-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:06.313 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:49:06.314 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:49:06.314 DEBUG:boto:path=/vclfcn-63/?prefix=obj4 2024-02-01T10:49:06.314 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj4 2024-02-01T10:49:06.314 DEBUG:boto:Method: GET 2024-02-01T10:49:06.314 DEBUG:boto:Path: /vclfcn-63/?prefix=obj4 2024-02-01T10:49:06.314 DEBUG:boto:Data: 2024-02-01T10:49:06.314 DEBUG:boto:Headers: {} 2024-02-01T10:49:06.314 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:06.314 DEBUG:boto:Port: 8001 2024-02-01T10:49:06.314 DEBUG:boto:Params: {} 2024-02-01T10:49:06.315 DEBUG:boto:Token: None 2024-02-01T10:49:06.315 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:06 GMT /vclfcn-63/ 2024-02-01T10:49:06.315 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9QIjbFPjM4qH0s4RK4qNKMf/9i8= 2024-02-01T10:49:06.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9QIjbFPjM4qH0s4RK4qNKMf/9i8=', 'Content-Length': '0'} 2024-02-01T10:49:06.319 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000138f114b5d34f809-0065bb7722-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:06.319 DEBUG:boto:b'vclfcn-63obj41000falseobj42024-02-01T10:48:26.575Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:06.320 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:49:06.320 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:49:06.320 DEBUG:boto:path=/vclfcn-63/?prefix=obj4 2024-02-01T10:49:06.320 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj4 2024-02-01T10:49:06.320 DEBUG:boto:Method: GET 2024-02-01T10:49:06.320 DEBUG:boto:Path: /vclfcn-63/?prefix=obj4 2024-02-01T10:49:06.321 DEBUG:boto:Data: 2024-02-01T10:49:06.321 DEBUG:boto:Headers: {} 2024-02-01T10:49:06.321 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:06.321 DEBUG:boto:Port: 8000 2024-02-01T10:49:06.321 DEBUG:boto:Params: {} 2024-02-01T10:49:06.321 DEBUG:boto:Token: None 2024-02-01T10:49:06.321 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:06 GMT /vclfcn-63/ 2024-02-01T10:49:06.321 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9QIjbFPjM4qH0s4RK4qNKMf/9i8= 2024-02-01T10:49:06.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9QIjbFPjM4qH0s4RK4qNKMf/9i8=', 'Content-Length': '0'} 2024-02-01T10:49:06.326 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4de5c22316f6240-0065bb7722-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:06.326 DEBUG:boto:b'vclfcn-63obj41000falseobj42024-02-01T10:48:26.575Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:06.327 DEBUG:boto:path=/vclfcn-63/obj4 2024-02-01T10:49:06.327 DEBUG:boto:auth_path=/vclfcn-63/obj4 2024-02-01T10:49:06.327 DEBUG:boto:Method: HEAD 2024-02-01T10:49:06.327 DEBUG:boto:Path: /vclfcn-63/obj4 2024-02-01T10:49:06.327 DEBUG:boto:Data: 2024-02-01T10:49:06.327 DEBUG:boto:Headers: {} 2024-02-01T10:49:06.327 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:06.327 DEBUG:boto:Port: 8001 2024-02-01T10:49:06.327 DEBUG:boto:Params: {} 2024-02-01T10:49:06.327 DEBUG:boto:Token: None 2024-02-01T10:49:06.328 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:06 GMT /vclfcn-63/obj4 2024-02-01T10:49:06.328 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ar4YUKobOFMxQFzuIBTRCUR7mAw= 2024-02-01T10:49:06.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ar4YUKobOFMxQFzuIBTRCUR7mAw=', 'Content-Length': '0'} 2024-02-01T10:49:06.330 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008534956b69ab461a-0065bb7722-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:06.331 DEBUG:boto:path=/vclfcn-63/obj4 2024-02-01T10:49:06.331 DEBUG:boto:auth_path=/vclfcn-63/obj4 2024-02-01T10:49:06.331 DEBUG:boto:Method: HEAD 2024-02-01T10:49:06.331 DEBUG:boto:Path: /vclfcn-63/obj4 2024-02-01T10:49:06.331 DEBUG:boto:Data: 2024-02-01T10:49:06.331 DEBUG:boto:Headers: {} 2024-02-01T10:49:06.331 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:06.331 DEBUG:boto:Port: 8000 2024-02-01T10:49:06.331 DEBUG:boto:Params: {} 2024-02-01T10:49:06.332 DEBUG:boto:Token: None 2024-02-01T10:49:06.332 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:06 GMT /vclfcn-63/obj4 2024-02-01T10:49:06.332 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ar4YUKobOFMxQFzuIBTRCUR7mAw= 2024-02-01T10:49:06.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ar4YUKobOFMxQFzuIBTRCUR7mAw=', 'Content-Length': '0'} 2024-02-01T10:49:06.334 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000029d4d7128195510d-0065bb7722-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:06.335 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:49:06.335 DEBUG:boto:path=/vclfcn-63/obj4 2024-02-01T10:49:06.335 DEBUG:boto:auth_path=/vclfcn-63/obj4 2024-02-01T10:49:06.335 DEBUG:boto:Method: GET 2024-02-01T10:49:06.335 DEBUG:boto:Path: /vclfcn-63/obj4 2024-02-01T10:49:06.335 DEBUG:boto:Data: 2024-02-01T10:49:06.335 DEBUG:boto:Headers: {} 2024-02-01T10:49:06.335 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:06.336 DEBUG:boto:Port: 8001 2024-02-01T10:49:06.336 DEBUG:boto:Params: {} 2024-02-01T10:49:06.336 DEBUG:boto:Token: None 2024-02-01T10:49:06.336 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:06 GMT /vclfcn-63/obj4 2024-02-01T10:49:06.336 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ApAFtOvl0zRjS1QiLFMAtltujyY= 2024-02-01T10:49:06.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ApAFtOvl0zRjS1QiLFMAtltujyY=', 'Content-Length': '0'} 2024-02-01T10:49:06.339 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cbef908c8a3b0fc9-0065bb7722-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:06.380 DEBUG:boto:path=/vclfcn-63/obj4 2024-02-01T10:49:06.380 DEBUG:boto:auth_path=/vclfcn-63/obj4 2024-02-01T10:49:06.380 DEBUG:boto:Method: GET 2024-02-01T10:49:06.381 DEBUG:boto:Path: /vclfcn-63/obj4 2024-02-01T10:49:06.381 DEBUG:boto:Data: 2024-02-01T10:49:06.381 DEBUG:boto:Headers: {} 2024-02-01T10:49:06.381 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:06.381 DEBUG:boto:Port: 8000 2024-02-01T10:49:06.381 DEBUG:boto:Params: {} 2024-02-01T10:49:06.381 DEBUG:boto:Token: None 2024-02-01T10:49:06.381 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:06 GMT /vclfcn-63/obj4 2024-02-01T10:49:06.382 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ApAFtOvl0zRjS1QiLFMAtltujyY= 2024-02-01T10:49:06.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ApAFtOvl0zRjS1QiLFMAtltujyY=', 'Content-Length': '0'} 2024-02-01T10:49:06.385 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008448385731b72160-0065bb7722-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:06.385 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:49:06.386 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:49:06.386 DEBUG:boto:path=/vclfcn-63/?prefix=obj4 2024-02-01T10:49:06.386 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj4 2024-02-01T10:49:06.386 DEBUG:boto:Method: GET 2024-02-01T10:49:06.386 DEBUG:boto:Path: /vclfcn-63/?prefix=obj4 2024-02-01T10:49:06.386 DEBUG:boto:Data: 2024-02-01T10:49:06.386 DEBUG:boto:Headers: {} 2024-02-01T10:49:06.387 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:06.387 DEBUG:boto:Port: 8001 2024-02-01T10:49:06.387 DEBUG:boto:Params: {} 2024-02-01T10:49:06.387 DEBUG:boto:Token: None 2024-02-01T10:49:06.387 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:06 GMT /vclfcn-63/ 2024-02-01T10:49:06.387 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9QIjbFPjM4qH0s4RK4qNKMf/9i8= 2024-02-01T10:49:06.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9QIjbFPjM4qH0s4RK4qNKMf/9i8=', 'Content-Length': '0'} 2024-02-01T10:49:06.392 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000756ca36e890a8a21-0065bb7722-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:06.440 DEBUG:boto:b'vclfcn-63obj41000falseobj42024-02-01T10:48:26.575Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:06.441 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:49:06.441 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:49:06.441 DEBUG:boto:path=/vclfcn-63/?prefix=obj4 2024-02-01T10:49:06.441 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj4 2024-02-01T10:49:06.441 DEBUG:boto:Method: GET 2024-02-01T10:49:06.441 DEBUG:boto:Path: /vclfcn-63/?prefix=obj4 2024-02-01T10:49:06.441 DEBUG:boto:Data: 2024-02-01T10:49:06.441 DEBUG:boto:Headers: {} 2024-02-01T10:49:06.441 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:06.441 DEBUG:boto:Port: 8000 2024-02-01T10:49:06.442 DEBUG:boto:Params: {} 2024-02-01T10:49:06.442 DEBUG:boto:Token: None 2024-02-01T10:49:06.442 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:06 GMT /vclfcn-63/ 2024-02-01T10:49:06.442 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9QIjbFPjM4qH0s4RK4qNKMf/9i8= 2024-02-01T10:49:06.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9QIjbFPjM4qH0s4RK4qNKMf/9i8=', 'Content-Length': '0'} 2024-02-01T10:49:06.447 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d5485311a97e914-0065bb7722-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:06.447 DEBUG:boto:b'vclfcn-63obj41000falseobj42024-02-01T10:48:26.575Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:06.447 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-63 zones={test-zone3, test-zone2} 2024-02-01T10:49:06.448 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=vclfcn-64 2024-02-01T10:49:06.448 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:49:06.448 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:49:06.448 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket vclfcn-64 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-02-01T10:49:06.500 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:49:06.500 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:49:06.564 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:49:06.561+0000 7fbe2ef3b980 1 bucket sync caught up with source: 2024-02-01T10:49:06.564 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [00000000001.70.6, , 00000000001.48.6, , 00000000001.50.6, , , 00000000001.57.6, , , ] 2024-02-01T10:49:06.565 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [00000000001.70.6, , 00000000001.48.6, , 00000000001.50.6, , , 00000000001.57.6, , , ] 2024-02-01T10:49:06.565 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:49:06.561+0000 7fbe2ef3b980 0 bucket checkpoint complete 2024-02-01T10:49:06.588 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=vclfcn-64 2024-02-01T10:49:06.589 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:49:06.589 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:49:06.589 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-64 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:49:06.640 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:49:06.641 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:49:06.701 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:49:06.699+0000 7fc3879a5980 1 bucket sync caught up with source: 2024-02-01T10:49:06.701 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [00000000001.70.6, , 00000000001.48.6, , 00000000001.50.6, , , 00000000001.57.6, , , ] 2024-02-01T10:49:06.701 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [00000000001.70.6, , 00000000001.48.6, , 00000000001.50.6, , , 00000000001.57.6, , , ] 2024-02-01T10:49:06.701 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:49:06.699+0000 7fc3879a5980 0 bucket checkpoint complete 2024-02-01T10:49:06.724 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=vclfcn-64 2024-02-01T10:49:06.724 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-64', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:49:06.725 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-64', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:49:06.725 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-64 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:49:06.765 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:49:06.765 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:49:06.826 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:49:06.823+0000 7fca6ecf5980 1 bucket sync caught up with source: 2024-02-01T10:49:06.827 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [00000000001.316.6, , 00000000001.241.6, , 00000000001.286.6, , , 00000000001.292.6, , , ] 2024-02-01T10:49:06.827 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [00000000001.316.6, , 00000000001.241.6, , 00000000001.286.6, , , 00000000001.292.6, , , ] 2024-02-01T10:49:06.827 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:49:06.823+0000 7fca6ecf5980 0 bucket checkpoint complete 2024-02-01T10:49:06.850 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=vclfcn-64 2024-02-01T10:49:06.850 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-64', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:49:06.850 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-64', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:49:06.850 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-64 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:49:06.937 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:49:06.937 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:49:06.999 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:49:06.995+0000 7fe37771a980 1 bucket sync caught up with source: 2024-02-01T10:49:06.999 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [00000000001.316.6, , 00000000001.241.6, , 00000000001.286.6, , , 00000000001.292.6, , , ] 2024-02-01T10:49:06.999 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [00000000001.316.6, , 00000000001.241.6, , 00000000001.286.6, , , 00000000001.292.6, , , ] 2024-02-01T10:49:06.999 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:49:06.995+0000 7fe37771a980 0 bucket checkpoint complete 2024-02-01T10:49:07.021 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=vclfcn-64 2024-02-01T10:49:07.022 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-64', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:49:07.022 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-64', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:49:07.022 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-64 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:49:07.107 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:49:07.108 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:49:07.169 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:49:07.167+0000 7ff23396f980 1 bucket sync caught up with source: 2024-02-01T10:49:07.169 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [00000000001.205.6, , 00000000001.205.6, , 00000000001.224.6, , , 00000000001.214.6, , , ] 2024-02-01T10:49:07.169 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [00000000001.205.6, , 00000000001.205.6, , 00000000001.224.6, , , 00000000001.214.6, , , ] 2024-02-01T10:49:07.169 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:49:07.167+0000 7ff23396f980 0 bucket checkpoint complete 2024-02-01T10:49:07.193 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=vclfcn-64 2024-02-01T10:49:07.193 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-64', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:49:07.193 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-64', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:49:07.193 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket vclfcn-64 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-02-01T10:49:07.243 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:49:07.244 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:49:07.308 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:49:07.305+0000 7fe780098980 1 bucket sync caught up with source: 2024-02-01T10:49:07.308 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [00000000001.205.6, , 00000000001.205.6, , 00000000001.224.6, , , 00000000001.214.6, , , ] 2024-02-01T10:49:07.308 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [00000000001.205.6, , 00000000001.205.6, , 00000000001.224.6, , , 00000000001.214.6, , , ] 2024-02-01T10:49:07.308 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:49:07.305+0000 7fe780098980 0 bucket checkpoint complete 2024-02-01T10:49:07.332 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-64 zones={test-zone2, test-zone1} 2024-02-01T10:49:07.333 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:49:07.333 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:49:07.333 DEBUG:boto:Method: HEAD 2024-02-01T10:49:07.333 DEBUG:boto:Path: /vclfcn-64/ 2024-02-01T10:49:07.333 DEBUG:boto:Data: 2024-02-01T10:49:07.333 DEBUG:boto:Headers: {} 2024-02-01T10:49:07.333 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:07.333 DEBUG:boto:Port: 8000 2024-02-01T10:49:07.333 DEBUG:boto:Params: {} 2024-02-01T10:49:07.333 DEBUG:boto:Token: None 2024-02-01T10:49:07.334 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:07 GMT /vclfcn-64/ 2024-02-01T10:49:07.334 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:eKnTzeQTkUQr0octOibtiHpCtww= 2024-02-01T10:49:07.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:eKnTzeQTkUQr0octOibtiHpCtww=', 'Content-Length': '0'} 2024-02-01T10:49:07.336 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b1f5e60145ca04aa-0065bb7723-5350-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:49:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:07.336 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:49:07.336 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:49:07.337 DEBUG:boto:Method: HEAD 2024-02-01T10:49:07.337 DEBUG:boto:Path: /vclfcn-64/ 2024-02-01T10:49:07.337 DEBUG:boto:Data: 2024-02-01T10:49:07.337 DEBUG:boto:Headers: {} 2024-02-01T10:49:07.337 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:07.337 DEBUG:boto:Port: 8000 2024-02-01T10:49:07.337 DEBUG:boto:Params: {} 2024-02-01T10:49:07.337 DEBUG:boto:Token: None 2024-02-01T10:49:07.337 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:07 GMT /vclfcn-64/ 2024-02-01T10:49:07.338 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:eKnTzeQTkUQr0octOibtiHpCtww= 2024-02-01T10:49:07.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:eKnTzeQTkUQr0octOibtiHpCtww=', 'Content-Length': '0'} 2024-02-01T10:49:07.342 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a42add24e3904fa1-0065bb7723-5765-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:49:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:07.342 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:49:07.342 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:49:07.342 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:49:07.342 DEBUG:boto:path=/vclfcn-64/?versions&max-keys=999 2024-02-01T10:49:07.342 DEBUG:boto:auth_path=/vclfcn-64/?versions&max-keys=999 2024-02-01T10:49:07.342 DEBUG:boto:Method: GET 2024-02-01T10:49:07.343 DEBUG:boto:Path: /vclfcn-64/?versions&max-keys=999 2024-02-01T10:49:07.343 DEBUG:boto:Data: 2024-02-01T10:49:07.343 DEBUG:boto:Headers: {} 2024-02-01T10:49:07.343 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:07.343 DEBUG:boto:Port: 8000 2024-02-01T10:49:07.343 DEBUG:boto:Params: {} 2024-02-01T10:49:07.343 DEBUG:boto:Token: None 2024-02-01T10:49:07.343 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:07 GMT /vclfcn-64/?versions 2024-02-01T10:49:07.343 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ID3Aydn7GuMI6+9U/7d6/ARpfoo= 2024-02-01T10:49:07.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ID3Aydn7GuMI6+9U/7d6/ARpfoo=', 'Content-Length': '0'} 2024-02-01T10:49:07.346 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000766becb475548b63-0065bb7723-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:07.346 DEBUG:boto:b'vclfcn-64999falseobj1nulltrue2024-02-01T10:48:26.595Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:48:26.615Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:48:26.634Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:48:26.689Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:07.347 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-02-01T10:49:07.347 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-02-01T10:49:07.347 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-02-01T10:49:07.347 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-02-01T10:49:07.347 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:49:07.347 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:49:07.348 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:49:07.348 DEBUG:boto:path=/vclfcn-64/?versions&max-keys=999 2024-02-01T10:49:07.348 DEBUG:boto:auth_path=/vclfcn-64/?versions&max-keys=999 2024-02-01T10:49:07.348 DEBUG:boto:Method: GET 2024-02-01T10:49:07.348 DEBUG:boto:Path: /vclfcn-64/?versions&max-keys=999 2024-02-01T10:49:07.348 DEBUG:boto:Data: 2024-02-01T10:49:07.348 DEBUG:boto:Headers: {} 2024-02-01T10:49:07.348 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:07.348 DEBUG:boto:Port: 8000 2024-02-01T10:49:07.349 DEBUG:boto:Params: {} 2024-02-01T10:49:07.349 DEBUG:boto:Token: None 2024-02-01T10:49:07.349 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:07 GMT /vclfcn-64/?versions 2024-02-01T10:49:07.350 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ID3Aydn7GuMI6+9U/7d6/ARpfoo= 2024-02-01T10:49:07.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ID3Aydn7GuMI6+9U/7d6/ARpfoo=', 'Content-Length': '0'} 2024-02-01T10:49:07.355 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5c3638dc66fdccd-0065bb7723-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:07.355 DEBUG:boto:b'vclfcn-64999falseobj1nulltrue2024-02-01T10:48:26.595Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:48:26.615Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:48:26.634Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:48:26.689Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:07.356 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-02-01T10:49:07.356 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-02-01T10:49:07.356 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-02-01T10:49:07.356 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-02-01T10:49:07.356 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:49:07.356 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:49:07.356 DEBUG:boto:path=/vclfcn-64/?versions&max-keys=999 2024-02-01T10:49:07.356 DEBUG:boto:auth_path=/vclfcn-64/?versions&max-keys=999 2024-02-01T10:49:07.356 DEBUG:boto:Method: GET 2024-02-01T10:49:07.357 DEBUG:boto:Path: /vclfcn-64/?versions&max-keys=999 2024-02-01T10:49:07.357 DEBUG:boto:Data: 2024-02-01T10:49:07.357 DEBUG:boto:Headers: {} 2024-02-01T10:49:07.357 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:07.357 DEBUG:boto:Port: 8000 2024-02-01T10:49:07.357 DEBUG:boto:Params: {} 2024-02-01T10:49:07.357 DEBUG:boto:Token: None 2024-02-01T10:49:07.357 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:07 GMT /vclfcn-64/?versions 2024-02-01T10:49:07.357 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ID3Aydn7GuMI6+9U/7d6/ARpfoo= 2024-02-01T10:49:07.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ID3Aydn7GuMI6+9U/7d6/ARpfoo=', 'Content-Length': '0'} 2024-02-01T10:49:07.360 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008315223ab4a22668-0065bb7723-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:07.360 DEBUG:boto:b'vclfcn-64999falseobj1nulltrue2024-02-01T10:48:26.595Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:48:26.615Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:48:26.634Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:48:26.689Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:07.361 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:49:07.361 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:49:07.361 DEBUG:boto:path=/vclfcn-64/?versions&max-keys=999 2024-02-01T10:49:07.361 DEBUG:boto:auth_path=/vclfcn-64/?versions&max-keys=999 2024-02-01T10:49:07.361 DEBUG:boto:Method: GET 2024-02-01T10:49:07.361 DEBUG:boto:Path: /vclfcn-64/?versions&max-keys=999 2024-02-01T10:49:07.361 DEBUG:boto:Data: 2024-02-01T10:49:07.361 DEBUG:boto:Headers: {} 2024-02-01T10:49:07.361 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:07.362 DEBUG:boto:Port: 8000 2024-02-01T10:49:07.362 DEBUG:boto:Params: {} 2024-02-01T10:49:07.362 DEBUG:boto:Token: None 2024-02-01T10:49:07.362 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:07 GMT /vclfcn-64/?versions 2024-02-01T10:49:07.362 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ID3Aydn7GuMI6+9U/7d6/ARpfoo= 2024-02-01T10:49:07.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ID3Aydn7GuMI6+9U/7d6/ARpfoo=', 'Content-Length': '0'} 2024-02-01T10:49:07.367 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008dbb4b28cf9c11c6-0065bb7723-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:07.367 DEBUG:boto:b'vclfcn-64999falseobj1nulltrue2024-02-01T10:48:26.595Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:48:26.615Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:48:26.634Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:48:26.689Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:07.368 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:49:07.368 DEBUG:boto:path=/vclfcn-64/obj1 2024-02-01T10:49:07.368 DEBUG:boto:auth_path=/vclfcn-64/obj1 2024-02-01T10:49:07.368 DEBUG:boto:path=/vclfcn-64/obj1?versionId=null 2024-02-01T10:49:07.368 DEBUG:boto:auth_path=/vclfcn-64/obj1?versionId=null 2024-02-01T10:49:07.369 DEBUG:boto:Method: GET 2024-02-01T10:49:07.369 DEBUG:boto:Path: /vclfcn-64/obj1?versionId=null 2024-02-01T10:49:07.369 DEBUG:boto:Data: 2024-02-01T10:49:07.370 DEBUG:boto:Headers: {} 2024-02-01T10:49:07.370 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:07.370 DEBUG:boto:Port: 8000 2024-02-01T10:49:07.370 DEBUG:boto:Params: {} 2024-02-01T10:49:07.370 DEBUG:boto:Token: None 2024-02-01T10:49:07.370 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:07 GMT /vclfcn-64/obj1?versionId=null 2024-02-01T10:49:07.370 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+PjZS1znxPgMBbCgLKG6+zWjPbM= 2024-02-01T10:49:07.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+PjZS1znxPgMBbCgLKG6+zWjPbM=', 'Content-Length': '0'} 2024-02-01T10:49:07.372 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009230e95e49913250-0065bb7723-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:07.416 DEBUG:boto:path=/vclfcn-64/obj1 2024-02-01T10:49:07.416 DEBUG:boto:auth_path=/vclfcn-64/obj1 2024-02-01T10:49:07.416 DEBUG:boto:path=/vclfcn-64/obj1?versionId=null 2024-02-01T10:49:07.416 DEBUG:boto:auth_path=/vclfcn-64/obj1?versionId=null 2024-02-01T10:49:07.417 DEBUG:boto:Method: GET 2024-02-01T10:49:07.417 DEBUG:boto:Path: /vclfcn-64/obj1?versionId=null 2024-02-01T10:49:07.417 DEBUG:boto:Data: 2024-02-01T10:49:07.417 DEBUG:boto:Headers: {} 2024-02-01T10:49:07.417 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:07.417 DEBUG:boto:Port: 8000 2024-02-01T10:49:07.417 DEBUG:boto:Params: {} 2024-02-01T10:49:07.417 DEBUG:boto:Token: None 2024-02-01T10:49:07.417 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:07 GMT /vclfcn-64/obj1?versionId=null 2024-02-01T10:49:07.418 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+PjZS1znxPgMBbCgLKG6+zWjPbM= 2024-02-01T10:49:07.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+PjZS1znxPgMBbCgLKG6+zWjPbM=', 'Content-Length': '0'} 2024-02-01T10:49:07.421 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f3a9bdf1cb478a06-0065bb7723-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:07.422 DEBUG:boto:path=/vclfcn-64/obj1 2024-02-01T10:49:07.422 DEBUG:boto:auth_path=/vclfcn-64/obj1 2024-02-01T10:49:07.422 DEBUG:boto:path=/vclfcn-64/obj1?versionId=null 2024-02-01T10:49:07.422 DEBUG:boto:auth_path=/vclfcn-64/obj1?versionId=null 2024-02-01T10:49:07.422 DEBUG:boto:Method: HEAD 2024-02-01T10:49:07.422 DEBUG:boto:Path: /vclfcn-64/obj1?versionId=null 2024-02-01T10:49:07.422 DEBUG:boto:Data: 2024-02-01T10:49:07.422 DEBUG:boto:Headers: {} 2024-02-01T10:49:07.422 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:07.423 DEBUG:boto:Port: 8000 2024-02-01T10:49:07.423 DEBUG:boto:Params: {} 2024-02-01T10:49:07.423 DEBUG:boto:Token: None 2024-02-01T10:49:07.423 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:07 GMT /vclfcn-64/obj1?versionId=null 2024-02-01T10:49:07.423 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:I/DbMLp/xBnKIz/yriSkoOi3Hd0= 2024-02-01T10:49:07.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:I/DbMLp/xBnKIz/yriSkoOi3Hd0=', 'Content-Length': '0'} 2024-02-01T10:49:07.426 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ef22f487033f8b73-0065bb7723-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:07.426 DEBUG:boto:path=/vclfcn-64/obj1 2024-02-01T10:49:07.426 DEBUG:boto:auth_path=/vclfcn-64/obj1 2024-02-01T10:49:07.426 DEBUG:boto:path=/vclfcn-64/obj1?versionId=null 2024-02-01T10:49:07.426 DEBUG:boto:auth_path=/vclfcn-64/obj1?versionId=null 2024-02-01T10:49:07.426 DEBUG:boto:Method: HEAD 2024-02-01T10:49:07.426 DEBUG:boto:Path: /vclfcn-64/obj1?versionId=null 2024-02-01T10:49:07.426 DEBUG:boto:Data: 2024-02-01T10:49:07.427 DEBUG:boto:Headers: {} 2024-02-01T10:49:07.427 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:07.427 DEBUG:boto:Port: 8000 2024-02-01T10:49:07.427 DEBUG:boto:Params: {} 2024-02-01T10:49:07.427 DEBUG:boto:Token: None 2024-02-01T10:49:07.427 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:07 GMT /vclfcn-64/obj1?versionId=null 2024-02-01T10:49:07.427 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:I/DbMLp/xBnKIz/yriSkoOi3Hd0= 2024-02-01T10:49:07.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:I/DbMLp/xBnKIz/yriSkoOi3Hd0=', 'Content-Length': '0'} 2024-02-01T10:49:07.430 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006cce2515f092b235-0065bb7723-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:07.430 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:49:07.430 DEBUG:boto:path=/vclfcn-64/obj1 2024-02-01T10:49:07.430 DEBUG:boto:auth_path=/vclfcn-64/obj1 2024-02-01T10:49:07.430 DEBUG:boto:path=/vclfcn-64/obj1?versionId=null 2024-02-01T10:49:07.430 DEBUG:boto:auth_path=/vclfcn-64/obj1?versionId=null 2024-02-01T10:49:07.431 DEBUG:boto:Method: GET 2024-02-01T10:49:07.431 DEBUG:boto:Path: /vclfcn-64/obj1?versionId=null 2024-02-01T10:49:07.431 DEBUG:boto:Data: 2024-02-01T10:49:07.431 DEBUG:boto:Headers: {} 2024-02-01T10:49:07.431 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:07.431 DEBUG:boto:Port: 8000 2024-02-01T10:49:07.431 DEBUG:boto:Params: {} 2024-02-01T10:49:07.431 DEBUG:boto:Token: None 2024-02-01T10:49:07.431 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:07 GMT /vclfcn-64/obj1?versionId=null 2024-02-01T10:49:07.431 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+PjZS1znxPgMBbCgLKG6+zWjPbM= 2024-02-01T10:49:07.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+PjZS1znxPgMBbCgLKG6+zWjPbM=', 'Content-Length': '0'} 2024-02-01T10:49:07.435 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000071fb8f1b2cf3ebf6-0065bb7723-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:07.435 DEBUG:boto:path=/vclfcn-64/obj1 2024-02-01T10:49:07.435 DEBUG:boto:auth_path=/vclfcn-64/obj1 2024-02-01T10:49:07.435 DEBUG:boto:path=/vclfcn-64/obj1?versionId=null 2024-02-01T10:49:07.435 DEBUG:boto:auth_path=/vclfcn-64/obj1?versionId=null 2024-02-01T10:49:07.435 DEBUG:boto:Method: GET 2024-02-01T10:49:07.435 DEBUG:boto:Path: /vclfcn-64/obj1?versionId=null 2024-02-01T10:49:07.435 DEBUG:boto:Data: 2024-02-01T10:49:07.435 DEBUG:boto:Headers: {} 2024-02-01T10:49:07.436 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:07.436 DEBUG:boto:Port: 8000 2024-02-01T10:49:07.436 DEBUG:boto:Params: {} 2024-02-01T10:49:07.436 DEBUG:boto:Token: None 2024-02-01T10:49:07.436 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:07 GMT /vclfcn-64/obj1?versionId=null 2024-02-01T10:49:07.436 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+PjZS1znxPgMBbCgLKG6+zWjPbM= 2024-02-01T10:49:07.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+PjZS1znxPgMBbCgLKG6+zWjPbM=', 'Content-Length': '0'} 2024-02-01T10:49:07.439 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fbc9039424f3dad4-0065bb7723-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:07.440 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:49:07.440 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:49:07.440 DEBUG:boto:path=/vclfcn-64/?prefix=obj1 2024-02-01T10:49:07.440 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj1 2024-02-01T10:49:07.440 DEBUG:boto:Method: GET 2024-02-01T10:49:07.440 DEBUG:boto:Path: /vclfcn-64/?prefix=obj1 2024-02-01T10:49:07.440 DEBUG:boto:Data: 2024-02-01T10:49:07.440 DEBUG:boto:Headers: {} 2024-02-01T10:49:07.440 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:07.440 DEBUG:boto:Port: 8000 2024-02-01T10:49:07.441 DEBUG:boto:Params: {} 2024-02-01T10:49:07.441 DEBUG:boto:Token: None 2024-02-01T10:49:07.442 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:07 GMT /vclfcn-64/ 2024-02-01T10:49:07.442 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:2JFVdLJn9rONZzRyf4EF/wiDgkU= 2024-02-01T10:49:07.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2JFVdLJn9rONZzRyf4EF/wiDgkU=', 'Content-Length': '0'} 2024-02-01T10:49:07.446 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d3793f142c565de-0065bb7723-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:07.446 DEBUG:boto:b'vclfcn-64obj11000falseobj12024-02-01T10:48:26.595Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:07.446 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:49:07.447 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:49:07.447 DEBUG:boto:path=/vclfcn-64/?prefix=obj1 2024-02-01T10:49:07.447 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj1 2024-02-01T10:49:07.447 DEBUG:boto:Method: GET 2024-02-01T10:49:07.447 DEBUG:boto:Path: /vclfcn-64/?prefix=obj1 2024-02-01T10:49:07.447 DEBUG:boto:Data: 2024-02-01T10:49:07.447 DEBUG:boto:Headers: {} 2024-02-01T10:49:07.447 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:07.447 DEBUG:boto:Port: 8000 2024-02-01T10:49:07.447 DEBUG:boto:Params: {} 2024-02-01T10:49:07.447 DEBUG:boto:Token: None 2024-02-01T10:49:07.448 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:07 GMT /vclfcn-64/ 2024-02-01T10:49:07.448 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:2JFVdLJn9rONZzRyf4EF/wiDgkU= 2024-02-01T10:49:07.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2JFVdLJn9rONZzRyf4EF/wiDgkU=', 'Content-Length': '0'} 2024-02-01T10:49:07.456 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e54cc7511233cdc0-0065bb7723-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:07.456 DEBUG:boto:b'vclfcn-64obj11000falseobj12024-02-01T10:48:26.595Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:07.457 DEBUG:boto:path=/vclfcn-64/obj1 2024-02-01T10:49:07.457 DEBUG:boto:auth_path=/vclfcn-64/obj1 2024-02-01T10:49:07.457 DEBUG:boto:Method: HEAD 2024-02-01T10:49:07.457 DEBUG:boto:Path: /vclfcn-64/obj1 2024-02-01T10:49:07.457 DEBUG:boto:Data: 2024-02-01T10:49:07.457 DEBUG:boto:Headers: {} 2024-02-01T10:49:07.458 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:07.458 DEBUG:boto:Port: 8000 2024-02-01T10:49:07.458 DEBUG:boto:Params: {} 2024-02-01T10:49:07.458 DEBUG:boto:Token: None 2024-02-01T10:49:07.458 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:07 GMT /vclfcn-64/obj1 2024-02-01T10:49:07.458 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:i52TAHLIyg2RhcWkMeFg4ryn3VM= 2024-02-01T10:49:07.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:i52TAHLIyg2RhcWkMeFg4ryn3VM=', 'Content-Length': '0'} 2024-02-01T10:49:07.461 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000097f43f4ce9011aaa-0065bb7723-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:07.461 DEBUG:boto:path=/vclfcn-64/obj1 2024-02-01T10:49:07.462 DEBUG:boto:auth_path=/vclfcn-64/obj1 2024-02-01T10:49:07.462 DEBUG:boto:Method: HEAD 2024-02-01T10:49:07.462 DEBUG:boto:Path: /vclfcn-64/obj1 2024-02-01T10:49:07.462 DEBUG:boto:Data: 2024-02-01T10:49:07.462 DEBUG:boto:Headers: {} 2024-02-01T10:49:07.462 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:07.462 DEBUG:boto:Port: 8000 2024-02-01T10:49:07.462 DEBUG:boto:Params: {} 2024-02-01T10:49:07.462 DEBUG:boto:Token: None 2024-02-01T10:49:07.462 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:07 GMT /vclfcn-64/obj1 2024-02-01T10:49:07.463 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:i52TAHLIyg2RhcWkMeFg4ryn3VM= 2024-02-01T10:49:07.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:i52TAHLIyg2RhcWkMeFg4ryn3VM=', 'Content-Length': '0'} 2024-02-01T10:49:07.465 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000384761844b89bad3-0065bb7723-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:07.466 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:49:07.466 DEBUG:boto:path=/vclfcn-64/obj1 2024-02-01T10:49:07.466 DEBUG:boto:auth_path=/vclfcn-64/obj1 2024-02-01T10:49:07.466 DEBUG:boto:Method: GET 2024-02-01T10:49:07.466 DEBUG:boto:Path: /vclfcn-64/obj1 2024-02-01T10:49:07.466 DEBUG:boto:Data: 2024-02-01T10:49:07.466 DEBUG:boto:Headers: {} 2024-02-01T10:49:07.466 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:07.467 DEBUG:boto:Port: 8000 2024-02-01T10:49:07.467 DEBUG:boto:Params: {} 2024-02-01T10:49:07.467 DEBUG:boto:Token: None 2024-02-01T10:49:07.467 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:07 GMT /vclfcn-64/obj1 2024-02-01T10:49:07.467 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QcOk/KAxl8SmFcP/FNvc+iZ3MUw= 2024-02-01T10:49:07.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QcOk/KAxl8SmFcP/FNvc+iZ3MUw=', 'Content-Length': '0'} 2024-02-01T10:49:07.470 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000071e1b074e88a9de0-0065bb7723-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:07.516 DEBUG:boto:path=/vclfcn-64/obj1 2024-02-01T10:49:07.516 DEBUG:boto:auth_path=/vclfcn-64/obj1 2024-02-01T10:49:07.516 DEBUG:boto:Method: GET 2024-02-01T10:49:07.516 DEBUG:boto:Path: /vclfcn-64/obj1 2024-02-01T10:49:07.517 DEBUG:boto:Data: 2024-02-01T10:49:07.517 DEBUG:boto:Headers: {} 2024-02-01T10:49:07.517 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:07.517 DEBUG:boto:Port: 8000 2024-02-01T10:49:07.517 DEBUG:boto:Params: {} 2024-02-01T10:49:07.517 DEBUG:boto:Token: None 2024-02-01T10:49:07.517 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:07 GMT /vclfcn-64/obj1 2024-02-01T10:49:07.517 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QcOk/KAxl8SmFcP/FNvc+iZ3MUw= 2024-02-01T10:49:07.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QcOk/KAxl8SmFcP/FNvc+iZ3MUw=', 'Content-Length': '0'} 2024-02-01T10:49:07.520 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001cb61ef4fe7cb0eb-0065bb7723-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:07.564 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:49:07.564 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:49:07.564 DEBUG:boto:path=/vclfcn-64/?prefix=obj1 2024-02-01T10:49:07.565 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj1 2024-02-01T10:49:07.565 DEBUG:boto:Method: GET 2024-02-01T10:49:07.565 DEBUG:boto:Path: /vclfcn-64/?prefix=obj1 2024-02-01T10:49:07.565 DEBUG:boto:Data: 2024-02-01T10:49:07.565 DEBUG:boto:Headers: {} 2024-02-01T10:49:07.565 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:07.565 DEBUG:boto:Port: 8000 2024-02-01T10:49:07.565 DEBUG:boto:Params: {} 2024-02-01T10:49:07.565 DEBUG:boto:Token: None 2024-02-01T10:49:07.566 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:07 GMT /vclfcn-64/ 2024-02-01T10:49:07.566 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:2JFVdLJn9rONZzRyf4EF/wiDgkU= 2024-02-01T10:49:07.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2JFVdLJn9rONZzRyf4EF/wiDgkU=', 'Content-Length': '0'} 2024-02-01T10:49:07.570 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e211f86a8b54084f-0065bb7723-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:07.616 DEBUG:boto:b'vclfcn-64obj11000falseobj12024-02-01T10:48:26.595Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:07.617 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:49:07.617 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:49:07.617 DEBUG:boto:path=/vclfcn-64/?prefix=obj1 2024-02-01T10:49:07.617 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj1 2024-02-01T10:49:07.617 DEBUG:boto:Method: GET 2024-02-01T10:49:07.617 DEBUG:boto:Path: /vclfcn-64/?prefix=obj1 2024-02-01T10:49:07.617 DEBUG:boto:Data: 2024-02-01T10:49:07.617 DEBUG:boto:Headers: {} 2024-02-01T10:49:07.617 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:07.617 DEBUG:boto:Port: 8000 2024-02-01T10:49:07.618 DEBUG:boto:Params: {} 2024-02-01T10:49:07.618 DEBUG:boto:Token: None 2024-02-01T10:49:07.618 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:07 GMT /vclfcn-64/ 2024-02-01T10:49:07.618 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:2JFVdLJn9rONZzRyf4EF/wiDgkU= 2024-02-01T10:49:07.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2JFVdLJn9rONZzRyf4EF/wiDgkU=', 'Content-Length': '0'} 2024-02-01T10:49:07.622 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc8a413b5f5baffe-0065bb7723-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:07.622 DEBUG:boto:b'vclfcn-64obj11000falseobj12024-02-01T10:48:26.595Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:07.623 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:49:07.623 DEBUG:boto:path=/vclfcn-64/obj2 2024-02-01T10:49:07.623 DEBUG:boto:auth_path=/vclfcn-64/obj2 2024-02-01T10:49:07.623 DEBUG:boto:path=/vclfcn-64/obj2?versionId=null 2024-02-01T10:49:07.623 DEBUG:boto:auth_path=/vclfcn-64/obj2?versionId=null 2024-02-01T10:49:07.623 DEBUG:boto:Method: GET 2024-02-01T10:49:07.623 DEBUG:boto:Path: /vclfcn-64/obj2?versionId=null 2024-02-01T10:49:07.623 DEBUG:boto:Data: 2024-02-01T10:49:07.624 DEBUG:boto:Headers: {} 2024-02-01T10:49:07.624 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:07.624 DEBUG:boto:Port: 8000 2024-02-01T10:49:07.624 DEBUG:boto:Params: {} 2024-02-01T10:49:07.624 DEBUG:boto:Token: None 2024-02-01T10:49:07.624 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:07 GMT /vclfcn-64/obj2?versionId=null 2024-02-01T10:49:07.624 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Mh/wTzAY1YiLRYhD7dGN4KjId30= 2024-02-01T10:49:07.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Mh/wTzAY1YiLRYhD7dGN4KjId30=', 'Content-Length': '0'} 2024-02-01T10:49:07.627 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d31f15a7ef14a39b-0065bb7723-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:07.668 DEBUG:boto:path=/vclfcn-64/obj2 2024-02-01T10:49:07.668 DEBUG:boto:auth_path=/vclfcn-64/obj2 2024-02-01T10:49:07.668 DEBUG:boto:path=/vclfcn-64/obj2?versionId=null 2024-02-01T10:49:07.668 DEBUG:boto:auth_path=/vclfcn-64/obj2?versionId=null 2024-02-01T10:49:07.669 DEBUG:boto:Method: GET 2024-02-01T10:49:07.669 DEBUG:boto:Path: /vclfcn-64/obj2?versionId=null 2024-02-01T10:49:07.669 DEBUG:boto:Data: 2024-02-01T10:49:07.669 DEBUG:boto:Headers: {} 2024-02-01T10:49:07.669 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:07.669 DEBUG:boto:Port: 8000 2024-02-01T10:49:07.669 DEBUG:boto:Params: {} 2024-02-01T10:49:07.669 DEBUG:boto:Token: None 2024-02-01T10:49:07.669 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:07 GMT /vclfcn-64/obj2?versionId=null 2024-02-01T10:49:07.670 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Mh/wTzAY1YiLRYhD7dGN4KjId30= 2024-02-01T10:49:07.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Mh/wTzAY1YiLRYhD7dGN4KjId30=', 'Content-Length': '0'} 2024-02-01T10:49:07.673 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000347227a8134b27a7-0065bb7723-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:07.673 DEBUG:boto:path=/vclfcn-64/obj2 2024-02-01T10:49:07.673 DEBUG:boto:auth_path=/vclfcn-64/obj2 2024-02-01T10:49:07.673 DEBUG:boto:path=/vclfcn-64/obj2?versionId=null 2024-02-01T10:49:07.674 DEBUG:boto:auth_path=/vclfcn-64/obj2?versionId=null 2024-02-01T10:49:07.674 DEBUG:boto:Method: HEAD 2024-02-01T10:49:07.674 DEBUG:boto:Path: /vclfcn-64/obj2?versionId=null 2024-02-01T10:49:07.674 DEBUG:boto:Data: 2024-02-01T10:49:07.674 DEBUG:boto:Headers: {} 2024-02-01T10:49:07.674 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:07.674 DEBUG:boto:Port: 8000 2024-02-01T10:49:07.674 DEBUG:boto:Params: {} 2024-02-01T10:49:07.674 DEBUG:boto:Token: None 2024-02-01T10:49:07.675 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:07 GMT /vclfcn-64/obj2?versionId=null 2024-02-01T10:49:07.675 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:qm+rLxrBjlVUxi42pSTMKB4IXdA= 2024-02-01T10:49:07.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:qm+rLxrBjlVUxi42pSTMKB4IXdA=', 'Content-Length': '0'} 2024-02-01T10:49:07.678 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000073eca6009b5bfccb-0065bb7723-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:07.678 DEBUG:boto:path=/vclfcn-64/obj2 2024-02-01T10:49:07.678 DEBUG:boto:auth_path=/vclfcn-64/obj2 2024-02-01T10:49:07.678 DEBUG:boto:path=/vclfcn-64/obj2?versionId=null 2024-02-01T10:49:07.678 DEBUG:boto:auth_path=/vclfcn-64/obj2?versionId=null 2024-02-01T10:49:07.679 DEBUG:boto:Method: HEAD 2024-02-01T10:49:07.679 DEBUG:boto:Path: /vclfcn-64/obj2?versionId=null 2024-02-01T10:49:07.679 DEBUG:boto:Data: 2024-02-01T10:49:07.679 DEBUG:boto:Headers: {} 2024-02-01T10:49:07.679 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:07.679 DEBUG:boto:Port: 8000 2024-02-01T10:49:07.679 DEBUG:boto:Params: {} 2024-02-01T10:49:07.679 DEBUG:boto:Token: None 2024-02-01T10:49:07.679 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:07 GMT /vclfcn-64/obj2?versionId=null 2024-02-01T10:49:07.679 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:qm+rLxrBjlVUxi42pSTMKB4IXdA= 2024-02-01T10:49:07.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:qm+rLxrBjlVUxi42pSTMKB4IXdA=', 'Content-Length': '0'} 2024-02-01T10:49:07.682 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ed7fb89c8267387c-0065bb7723-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:07.683 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:49:07.683 DEBUG:boto:path=/vclfcn-64/obj2 2024-02-01T10:49:07.683 DEBUG:boto:auth_path=/vclfcn-64/obj2 2024-02-01T10:49:07.683 DEBUG:boto:path=/vclfcn-64/obj2?versionId=null 2024-02-01T10:49:07.683 DEBUG:boto:auth_path=/vclfcn-64/obj2?versionId=null 2024-02-01T10:49:07.683 DEBUG:boto:Method: GET 2024-02-01T10:49:07.683 DEBUG:boto:Path: /vclfcn-64/obj2?versionId=null 2024-02-01T10:49:07.683 DEBUG:boto:Data: 2024-02-01T10:49:07.684 DEBUG:boto:Headers: {} 2024-02-01T10:49:07.684 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:07.684 DEBUG:boto:Port: 8000 2024-02-01T10:49:07.684 DEBUG:boto:Params: {} 2024-02-01T10:49:07.684 DEBUG:boto:Token: None 2024-02-01T10:49:07.684 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:07 GMT /vclfcn-64/obj2?versionId=null 2024-02-01T10:49:07.684 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Mh/wTzAY1YiLRYhD7dGN4KjId30= 2024-02-01T10:49:07.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Mh/wTzAY1YiLRYhD7dGN4KjId30=', 'Content-Length': '0'} 2024-02-01T10:49:07.687 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fcfbb0acec2d6ea3-0065bb7723-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:07.728 DEBUG:boto:path=/vclfcn-64/obj2 2024-02-01T10:49:07.728 DEBUG:boto:auth_path=/vclfcn-64/obj2 2024-02-01T10:49:07.728 DEBUG:boto:path=/vclfcn-64/obj2?versionId=null 2024-02-01T10:49:07.728 DEBUG:boto:auth_path=/vclfcn-64/obj2?versionId=null 2024-02-01T10:49:07.729 DEBUG:boto:Method: GET 2024-02-01T10:49:07.729 DEBUG:boto:Path: /vclfcn-64/obj2?versionId=null 2024-02-01T10:49:07.729 DEBUG:boto:Data: 2024-02-01T10:49:07.729 DEBUG:boto:Headers: {} 2024-02-01T10:49:07.729 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:07.729 DEBUG:boto:Port: 8000 2024-02-01T10:49:07.729 DEBUG:boto:Params: {} 2024-02-01T10:49:07.729 DEBUG:boto:Token: None 2024-02-01T10:49:07.729 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:07 GMT /vclfcn-64/obj2?versionId=null 2024-02-01T10:49:07.730 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Mh/wTzAY1YiLRYhD7dGN4KjId30= 2024-02-01T10:49:07.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Mh/wTzAY1YiLRYhD7dGN4KjId30=', 'Content-Length': '0'} 2024-02-01T10:49:07.733 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008ca55c00c5e33573-0065bb7723-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:07.780 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:49:07.780 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:49:07.780 DEBUG:boto:path=/vclfcn-64/?prefix=obj2 2024-02-01T10:49:07.780 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj2 2024-02-01T10:49:07.780 DEBUG:boto:Method: GET 2024-02-01T10:49:07.781 DEBUG:boto:Path: /vclfcn-64/?prefix=obj2 2024-02-01T10:49:07.781 DEBUG:boto:Data: 2024-02-01T10:49:07.781 DEBUG:boto:Headers: {} 2024-02-01T10:49:07.781 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:07.781 DEBUG:boto:Port: 8000 2024-02-01T10:49:07.781 DEBUG:boto:Params: {} 2024-02-01T10:49:07.781 DEBUG:boto:Token: None 2024-02-01T10:49:07.781 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:07 GMT /vclfcn-64/ 2024-02-01T10:49:07.782 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:2JFVdLJn9rONZzRyf4EF/wiDgkU= 2024-02-01T10:49:07.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2JFVdLJn9rONZzRyf4EF/wiDgkU=', 'Content-Length': '0'} 2024-02-01T10:49:07.787 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a726ac179b407def-0065bb7723-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:07.787 DEBUG:boto:b'vclfcn-64obj21000falseobj22024-02-01T10:48:26.615Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:07.787 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:49:07.787 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:49:07.788 DEBUG:boto:path=/vclfcn-64/?prefix=obj2 2024-02-01T10:49:07.788 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj2 2024-02-01T10:49:07.788 DEBUG:boto:Method: GET 2024-02-01T10:49:07.788 DEBUG:boto:Path: /vclfcn-64/?prefix=obj2 2024-02-01T10:49:07.788 DEBUG:boto:Data: 2024-02-01T10:49:07.788 DEBUG:boto:Headers: {} 2024-02-01T10:49:07.788 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:07.788 DEBUG:boto:Port: 8000 2024-02-01T10:49:07.788 DEBUG:boto:Params: {} 2024-02-01T10:49:07.788 DEBUG:boto:Token: None 2024-02-01T10:49:07.789 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:07 GMT /vclfcn-64/ 2024-02-01T10:49:07.789 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:2JFVdLJn9rONZzRyf4EF/wiDgkU= 2024-02-01T10:49:07.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2JFVdLJn9rONZzRyf4EF/wiDgkU=', 'Content-Length': '0'} 2024-02-01T10:49:07.795 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d7378b27a259123-0065bb7723-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:07.796 DEBUG:boto:b'vclfcn-64obj21000falseobj22024-02-01T10:48:26.615Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:07.796 DEBUG:boto:path=/vclfcn-64/obj2 2024-02-01T10:49:07.796 DEBUG:boto:auth_path=/vclfcn-64/obj2 2024-02-01T10:49:07.796 DEBUG:boto:Method: HEAD 2024-02-01T10:49:07.796 DEBUG:boto:Path: /vclfcn-64/obj2 2024-02-01T10:49:07.796 DEBUG:boto:Data: 2024-02-01T10:49:07.797 DEBUG:boto:Headers: {} 2024-02-01T10:49:07.797 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:07.798 DEBUG:boto:Port: 8000 2024-02-01T10:49:07.798 DEBUG:boto:Params: {} 2024-02-01T10:49:07.798 DEBUG:boto:Token: None 2024-02-01T10:49:07.798 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:07 GMT /vclfcn-64/obj2 2024-02-01T10:49:07.798 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0gFxr0Za3HaLjJMTv6lPHmLwVgs= 2024-02-01T10:49:07.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0gFxr0Za3HaLjJMTv6lPHmLwVgs=', 'Content-Length': '0'} 2024-02-01T10:49:07.801 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e3df45459f005276-0065bb7723-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:07.801 DEBUG:boto:path=/vclfcn-64/obj2 2024-02-01T10:49:07.801 DEBUG:boto:auth_path=/vclfcn-64/obj2 2024-02-01T10:49:07.801 DEBUG:boto:Method: HEAD 2024-02-01T10:49:07.801 DEBUG:boto:Path: /vclfcn-64/obj2 2024-02-01T10:49:07.801 DEBUG:boto:Data: 2024-02-01T10:49:07.802 DEBUG:boto:Headers: {} 2024-02-01T10:49:07.802 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:07.802 DEBUG:boto:Port: 8000 2024-02-01T10:49:07.802 DEBUG:boto:Params: {} 2024-02-01T10:49:07.802 DEBUG:boto:Token: None 2024-02-01T10:49:07.802 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:07 GMT /vclfcn-64/obj2 2024-02-01T10:49:07.802 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0gFxr0Za3HaLjJMTv6lPHmLwVgs= 2024-02-01T10:49:07.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0gFxr0Za3HaLjJMTv6lPHmLwVgs=', 'Content-Length': '0'} 2024-02-01T10:49:07.805 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000071fc0294e82cfce-0065bb7723-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:07.805 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:49:07.805 DEBUG:boto:path=/vclfcn-64/obj2 2024-02-01T10:49:07.805 DEBUG:boto:auth_path=/vclfcn-64/obj2 2024-02-01T10:49:07.805 DEBUG:boto:Method: GET 2024-02-01T10:49:07.806 DEBUG:boto:Path: /vclfcn-64/obj2 2024-02-01T10:49:07.806 DEBUG:boto:Data: 2024-02-01T10:49:07.806 DEBUG:boto:Headers: {} 2024-02-01T10:49:07.806 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:07.806 DEBUG:boto:Port: 8000 2024-02-01T10:49:07.806 DEBUG:boto:Params: {} 2024-02-01T10:49:07.806 DEBUG:boto:Token: None 2024-02-01T10:49:07.806 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:07 GMT /vclfcn-64/obj2 2024-02-01T10:49:07.806 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/cKR+h0v9B4LnHldbVeZQUnEWLw= 2024-02-01T10:49:07.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/cKR+h0v9B4LnHldbVeZQUnEWLw=', 'Content-Length': '0'} 2024-02-01T10:49:07.809 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007971832230ec5449-0065bb7723-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:07.852 DEBUG:boto:path=/vclfcn-64/obj2 2024-02-01T10:49:07.852 DEBUG:boto:auth_path=/vclfcn-64/obj2 2024-02-01T10:49:07.852 DEBUG:boto:Method: GET 2024-02-01T10:49:07.852 DEBUG:boto:Path: /vclfcn-64/obj2 2024-02-01T10:49:07.853 DEBUG:boto:Data: 2024-02-01T10:49:07.853 DEBUG:boto:Headers: {} 2024-02-01T10:49:07.853 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:07.853 DEBUG:boto:Port: 8000 2024-02-01T10:49:07.853 DEBUG:boto:Params: {} 2024-02-01T10:49:07.853 DEBUG:boto:Token: None 2024-02-01T10:49:07.853 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:07 GMT /vclfcn-64/obj2 2024-02-01T10:49:07.853 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/cKR+h0v9B4LnHldbVeZQUnEWLw= 2024-02-01T10:49:07.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/cKR+h0v9B4LnHldbVeZQUnEWLw=', 'Content-Length': '0'} 2024-02-01T10:49:07.860 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d1325218b891aba6-0065bb7723-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:07.900 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:49:07.900 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:49:07.900 DEBUG:boto:path=/vclfcn-64/?prefix=obj2 2024-02-01T10:49:07.900 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj2 2024-02-01T10:49:07.901 DEBUG:boto:Method: GET 2024-02-01T10:49:07.901 DEBUG:boto:Path: /vclfcn-64/?prefix=obj2 2024-02-01T10:49:07.901 DEBUG:boto:Data: 2024-02-01T10:49:07.901 DEBUG:boto:Headers: {} 2024-02-01T10:49:07.901 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:07.901 DEBUG:boto:Port: 8000 2024-02-01T10:49:07.901 DEBUG:boto:Params: {} 2024-02-01T10:49:07.901 DEBUG:boto:Token: None 2024-02-01T10:49:07.901 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:07 GMT /vclfcn-64/ 2024-02-01T10:49:07.901 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:2JFVdLJn9rONZzRyf4EF/wiDgkU= 2024-02-01T10:49:07.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2JFVdLJn9rONZzRyf4EF/wiDgkU=', 'Content-Length': '0'} 2024-02-01T10:49:07.906 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001bbb3a70d30715c2-0065bb7723-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:07.948 DEBUG:boto:b'vclfcn-64obj21000falseobj22024-02-01T10:48:26.615Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:07.949 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:49:07.949 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:49:07.949 DEBUG:boto:path=/vclfcn-64/?prefix=obj2 2024-02-01T10:49:07.949 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj2 2024-02-01T10:49:07.949 DEBUG:boto:Method: GET 2024-02-01T10:49:07.949 DEBUG:boto:Path: /vclfcn-64/?prefix=obj2 2024-02-01T10:49:07.949 DEBUG:boto:Data: 2024-02-01T10:49:07.949 DEBUG:boto:Headers: {} 2024-02-01T10:49:07.949 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:07.950 DEBUG:boto:Port: 8000 2024-02-01T10:49:07.950 DEBUG:boto:Params: {} 2024-02-01T10:49:07.950 DEBUG:boto:Token: None 2024-02-01T10:49:07.950 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:07 GMT /vclfcn-64/ 2024-02-01T10:49:07.950 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:2JFVdLJn9rONZzRyf4EF/wiDgkU= 2024-02-01T10:49:07.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2JFVdLJn9rONZzRyf4EF/wiDgkU=', 'Content-Length': '0'} 2024-02-01T10:49:07.954 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d1c229eafd96c69-0065bb7723-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:07.996 DEBUG:boto:b'vclfcn-64obj21000falseobj22024-02-01T10:48:26.615Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:07.997 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:49:07.997 DEBUG:boto:path=/vclfcn-64/obj3 2024-02-01T10:49:07.997 DEBUG:boto:auth_path=/vclfcn-64/obj3 2024-02-01T10:49:07.997 DEBUG:boto:path=/vclfcn-64/obj3?versionId=null 2024-02-01T10:49:07.997 DEBUG:boto:auth_path=/vclfcn-64/obj3?versionId=null 2024-02-01T10:49:07.997 DEBUG:boto:Method: GET 2024-02-01T10:49:07.997 DEBUG:boto:Path: /vclfcn-64/obj3?versionId=null 2024-02-01T10:49:07.997 DEBUG:boto:Data: 2024-02-01T10:49:07.997 DEBUG:boto:Headers: {} 2024-02-01T10:49:07.998 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:07.998 DEBUG:boto:Port: 8000 2024-02-01T10:49:07.998 DEBUG:boto:Params: {} 2024-02-01T10:49:07.998 DEBUG:boto:Token: None 2024-02-01T10:49:07.998 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:07 GMT /vclfcn-64/obj3?versionId=null 2024-02-01T10:49:07.998 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:pn235mDiFlRnt0cwlmOziqWbOcQ= 2024-02-01T10:49:07.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:pn235mDiFlRnt0cwlmOziqWbOcQ=', 'Content-Length': '0'} 2024-02-01T10:49:08.002 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000070596bd2bc079dd4-0065bb7723-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:08.002 DEBUG:boto:path=/vclfcn-64/obj3 2024-02-01T10:49:08.002 DEBUG:boto:auth_path=/vclfcn-64/obj3 2024-02-01T10:49:08.002 DEBUG:boto:path=/vclfcn-64/obj3?versionId=null 2024-02-01T10:49:08.002 DEBUG:boto:auth_path=/vclfcn-64/obj3?versionId=null 2024-02-01T10:49:08.002 DEBUG:boto:Method: GET 2024-02-01T10:49:08.003 DEBUG:boto:Path: /vclfcn-64/obj3?versionId=null 2024-02-01T10:49:08.003 DEBUG:boto:Data: 2024-02-01T10:49:08.003 DEBUG:boto:Headers: {} 2024-02-01T10:49:08.003 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:08.003 DEBUG:boto:Port: 8000 2024-02-01T10:49:08.003 DEBUG:boto:Params: {} 2024-02-01T10:49:08.003 DEBUG:boto:Token: None 2024-02-01T10:49:08.003 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:08 GMT /vclfcn-64/obj3?versionId=null 2024-02-01T10:49:08.003 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:a0IF11z3Fn5ZVZUz6cC9kEXHaSc= 2024-02-01T10:49:08.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:a0IF11z3Fn5ZVZUz6cC9kEXHaSc=', 'Content-Length': '0'} 2024-02-01T10:49:08.006 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000087682964714cbafc-0065bb7724-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:08.007 DEBUG:boto:path=/vclfcn-64/obj3 2024-02-01T10:49:08.007 DEBUG:boto:auth_path=/vclfcn-64/obj3 2024-02-01T10:49:08.007 DEBUG:boto:path=/vclfcn-64/obj3?versionId=null 2024-02-01T10:49:08.007 DEBUG:boto:auth_path=/vclfcn-64/obj3?versionId=null 2024-02-01T10:49:08.007 DEBUG:boto:Method: HEAD 2024-02-01T10:49:08.007 DEBUG:boto:Path: /vclfcn-64/obj3?versionId=null 2024-02-01T10:49:08.007 DEBUG:boto:Data: 2024-02-01T10:49:08.007 DEBUG:boto:Headers: {} 2024-02-01T10:49:08.007 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:08.008 DEBUG:boto:Port: 8000 2024-02-01T10:49:08.008 DEBUG:boto:Params: {} 2024-02-01T10:49:08.008 DEBUG:boto:Token: None 2024-02-01T10:49:08.008 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:08 GMT /vclfcn-64/obj3?versionId=null 2024-02-01T10:49:08.008 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Y1QbOpPejRrPGBljtf/NhZbKMw4= 2024-02-01T10:49:08.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Y1QbOpPejRrPGBljtf/NhZbKMw4=', 'Content-Length': '0'} 2024-02-01T10:49:08.011 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000761667df19a2e4d4-0065bb7724-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:08.012 DEBUG:boto:path=/vclfcn-64/obj3 2024-02-01T10:49:08.012 DEBUG:boto:auth_path=/vclfcn-64/obj3 2024-02-01T10:49:08.012 DEBUG:boto:path=/vclfcn-64/obj3?versionId=null 2024-02-01T10:49:08.012 DEBUG:boto:auth_path=/vclfcn-64/obj3?versionId=null 2024-02-01T10:49:08.012 DEBUG:boto:Method: HEAD 2024-02-01T10:49:08.012 DEBUG:boto:Path: /vclfcn-64/obj3?versionId=null 2024-02-01T10:49:08.012 DEBUG:boto:Data: 2024-02-01T10:49:08.012 DEBUG:boto:Headers: {} 2024-02-01T10:49:08.012 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:08.013 DEBUG:boto:Port: 8000 2024-02-01T10:49:08.013 DEBUG:boto:Params: {} 2024-02-01T10:49:08.013 DEBUG:boto:Token: None 2024-02-01T10:49:08.013 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:08 GMT /vclfcn-64/obj3?versionId=null 2024-02-01T10:49:08.013 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Y1QbOpPejRrPGBljtf/NhZbKMw4= 2024-02-01T10:49:08.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Y1QbOpPejRrPGBljtf/NhZbKMw4=', 'Content-Length': '0'} 2024-02-01T10:49:08.016 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a67106c073232d0f-0065bb7724-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:08.016 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:49:08.016 DEBUG:boto:path=/vclfcn-64/obj3 2024-02-01T10:49:08.016 DEBUG:boto:auth_path=/vclfcn-64/obj3 2024-02-01T10:49:08.017 DEBUG:boto:path=/vclfcn-64/obj3?versionId=null 2024-02-01T10:49:08.017 DEBUG:boto:auth_path=/vclfcn-64/obj3?versionId=null 2024-02-01T10:49:08.017 DEBUG:boto:Method: GET 2024-02-01T10:49:08.017 DEBUG:boto:Path: /vclfcn-64/obj3?versionId=null 2024-02-01T10:49:08.017 DEBUG:boto:Data: 2024-02-01T10:49:08.017 DEBUG:boto:Headers: {} 2024-02-01T10:49:08.017 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:08.017 DEBUG:boto:Port: 8000 2024-02-01T10:49:08.017 DEBUG:boto:Params: {} 2024-02-01T10:49:08.017 DEBUG:boto:Token: None 2024-02-01T10:49:08.018 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:08 GMT /vclfcn-64/obj3?versionId=null 2024-02-01T10:49:08.018 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:a0IF11z3Fn5ZVZUz6cC9kEXHaSc= 2024-02-01T10:49:08.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:a0IF11z3Fn5ZVZUz6cC9kEXHaSc=', 'Content-Length': '0'} 2024-02-01T10:49:08.021 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007b20afced16b8560-0065bb7724-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:08.064 DEBUG:boto:path=/vclfcn-64/obj3 2024-02-01T10:49:08.064 DEBUG:boto:auth_path=/vclfcn-64/obj3 2024-02-01T10:49:08.064 DEBUG:boto:path=/vclfcn-64/obj3?versionId=null 2024-02-01T10:49:08.064 DEBUG:boto:auth_path=/vclfcn-64/obj3?versionId=null 2024-02-01T10:49:08.065 DEBUG:boto:Method: GET 2024-02-01T10:49:08.065 DEBUG:boto:Path: /vclfcn-64/obj3?versionId=null 2024-02-01T10:49:08.065 DEBUG:boto:Data: 2024-02-01T10:49:08.065 DEBUG:boto:Headers: {} 2024-02-01T10:49:08.065 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:08.065 DEBUG:boto:Port: 8000 2024-02-01T10:49:08.065 DEBUG:boto:Params: {} 2024-02-01T10:49:08.065 DEBUG:boto:Token: None 2024-02-01T10:49:08.066 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:08 GMT /vclfcn-64/obj3?versionId=null 2024-02-01T10:49:08.066 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:a0IF11z3Fn5ZVZUz6cC9kEXHaSc= 2024-02-01T10:49:08.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:a0IF11z3Fn5ZVZUz6cC9kEXHaSc=', 'Content-Length': '0'} 2024-02-01T10:49:08.069 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007b5ffc07c7b8b91b-0065bb7724-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:08.112 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:49:08.112 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:49:08.112 DEBUG:boto:path=/vclfcn-64/?prefix=obj3 2024-02-01T10:49:08.112 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj3 2024-02-01T10:49:08.113 DEBUG:boto:Method: GET 2024-02-01T10:49:08.113 DEBUG:boto:Path: /vclfcn-64/?prefix=obj3 2024-02-01T10:49:08.113 DEBUG:boto:Data: 2024-02-01T10:49:08.113 DEBUG:boto:Headers: {} 2024-02-01T10:49:08.113 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:08.113 DEBUG:boto:Port: 8000 2024-02-01T10:49:08.113 DEBUG:boto:Params: {} 2024-02-01T10:49:08.113 DEBUG:boto:Token: None 2024-02-01T10:49:08.113 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:08 GMT /vclfcn-64/ 2024-02-01T10:49:08.114 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:f0BzGQrCVXluuLGn1s8cbpxOQUE= 2024-02-01T10:49:08.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:f0BzGQrCVXluuLGn1s8cbpxOQUE=', 'Content-Length': '0'} 2024-02-01T10:49:08.118 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e4345932c0f0504-0065bb7724-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:08.164 DEBUG:boto:b'vclfcn-64obj31000falseobj32024-02-01T10:48:26.634Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:08.165 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:49:08.165 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:49:08.165 DEBUG:boto:path=/vclfcn-64/?prefix=obj3 2024-02-01T10:49:08.165 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj3 2024-02-01T10:49:08.166 DEBUG:boto:Method: GET 2024-02-01T10:49:08.166 DEBUG:boto:Path: /vclfcn-64/?prefix=obj3 2024-02-01T10:49:08.166 DEBUG:boto:Data: 2024-02-01T10:49:08.166 DEBUG:boto:Headers: {} 2024-02-01T10:49:08.166 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:08.166 DEBUG:boto:Port: 8000 2024-02-01T10:49:08.166 DEBUG:boto:Params: {} 2024-02-01T10:49:08.167 DEBUG:boto:Token: None 2024-02-01T10:49:08.167 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:08 GMT /vclfcn-64/ 2024-02-01T10:49:08.167 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:f0BzGQrCVXluuLGn1s8cbpxOQUE= 2024-02-01T10:49:08.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:f0BzGQrCVXluuLGn1s8cbpxOQUE=', 'Content-Length': '0'} 2024-02-01T10:49:08.172 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5beda7692f880f4-0065bb7724-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:08.172 DEBUG:boto:b'vclfcn-64obj31000falseobj32024-02-01T10:48:26.634Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:08.173 DEBUG:boto:path=/vclfcn-64/obj3 2024-02-01T10:49:08.173 DEBUG:boto:auth_path=/vclfcn-64/obj3 2024-02-01T10:49:08.173 DEBUG:boto:Method: HEAD 2024-02-01T10:49:08.173 DEBUG:boto:Path: /vclfcn-64/obj3 2024-02-01T10:49:08.173 DEBUG:boto:Data: 2024-02-01T10:49:08.173 DEBUG:boto:Headers: {} 2024-02-01T10:49:08.173 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:08.173 DEBUG:boto:Port: 8000 2024-02-01T10:49:08.173 DEBUG:boto:Params: {} 2024-02-01T10:49:08.174 DEBUG:boto:Token: None 2024-02-01T10:49:08.174 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:08 GMT /vclfcn-64/obj3 2024-02-01T10:49:08.174 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tCxVASve9bM9uAjKLFfAZwDE5cU= 2024-02-01T10:49:08.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tCxVASve9bM9uAjKLFfAZwDE5cU=', 'Content-Length': '0'} 2024-02-01T10:49:08.177 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000072d53531cbdbb47c-0065bb7724-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:08.177 DEBUG:boto:path=/vclfcn-64/obj3 2024-02-01T10:49:08.177 DEBUG:boto:auth_path=/vclfcn-64/obj3 2024-02-01T10:49:08.177 DEBUG:boto:Method: HEAD 2024-02-01T10:49:08.177 DEBUG:boto:Path: /vclfcn-64/obj3 2024-02-01T10:49:08.177 DEBUG:boto:Data: 2024-02-01T10:49:08.177 DEBUG:boto:Headers: {} 2024-02-01T10:49:08.177 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:08.178 DEBUG:boto:Port: 8000 2024-02-01T10:49:08.178 DEBUG:boto:Params: {} 2024-02-01T10:49:08.178 DEBUG:boto:Token: None 2024-02-01T10:49:08.178 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:08 GMT /vclfcn-64/obj3 2024-02-01T10:49:08.178 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tCxVASve9bM9uAjKLFfAZwDE5cU= 2024-02-01T10:49:08.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tCxVASve9bM9uAjKLFfAZwDE5cU=', 'Content-Length': '0'} 2024-02-01T10:49:08.181 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000102a501b53398cae-0065bb7724-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:08.181 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:49:08.181 DEBUG:boto:path=/vclfcn-64/obj3 2024-02-01T10:49:08.181 DEBUG:boto:auth_path=/vclfcn-64/obj3 2024-02-01T10:49:08.181 DEBUG:boto:Method: GET 2024-02-01T10:49:08.181 DEBUG:boto:Path: /vclfcn-64/obj3 2024-02-01T10:49:08.181 DEBUG:boto:Data: 2024-02-01T10:49:08.181 DEBUG:boto:Headers: {} 2024-02-01T10:49:08.182 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:08.182 DEBUG:boto:Port: 8000 2024-02-01T10:49:08.182 DEBUG:boto:Params: {} 2024-02-01T10:49:08.182 DEBUG:boto:Token: None 2024-02-01T10:49:08.182 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:08 GMT /vclfcn-64/obj3 2024-02-01T10:49:08.182 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:CTJ0vCL15Ua27UnTJcAFcxQAU1I= 2024-02-01T10:49:08.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:CTJ0vCL15Ua27UnTJcAFcxQAU1I=', 'Content-Length': '0'} 2024-02-01T10:49:08.185 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000029cae7ade935b076-0065bb7724-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:08.228 DEBUG:boto:path=/vclfcn-64/obj3 2024-02-01T10:49:08.228 DEBUG:boto:auth_path=/vclfcn-64/obj3 2024-02-01T10:49:08.228 DEBUG:boto:Method: GET 2024-02-01T10:49:08.228 DEBUG:boto:Path: /vclfcn-64/obj3 2024-02-01T10:49:08.229 DEBUG:boto:Data: 2024-02-01T10:49:08.229 DEBUG:boto:Headers: {} 2024-02-01T10:49:08.229 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:08.229 DEBUG:boto:Port: 8000 2024-02-01T10:49:08.229 DEBUG:boto:Params: {} 2024-02-01T10:49:08.229 DEBUG:boto:Token: None 2024-02-01T10:49:08.229 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:08 GMT /vclfcn-64/obj3 2024-02-01T10:49:08.229 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:CTJ0vCL15Ua27UnTJcAFcxQAU1I= 2024-02-01T10:49:08.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:CTJ0vCL15Ua27UnTJcAFcxQAU1I=', 'Content-Length': '0'} 2024-02-01T10:49:08.232 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000075677b1663e7d8ac-0065bb7724-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:08.276 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:49:08.276 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:49:08.276 DEBUG:boto:path=/vclfcn-64/?prefix=obj3 2024-02-01T10:49:08.277 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj3 2024-02-01T10:49:08.277 DEBUG:boto:Method: GET 2024-02-01T10:49:08.277 DEBUG:boto:Path: /vclfcn-64/?prefix=obj3 2024-02-01T10:49:08.277 DEBUG:boto:Data: 2024-02-01T10:49:08.277 DEBUG:boto:Headers: {} 2024-02-01T10:49:08.277 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:08.277 DEBUG:boto:Port: 8000 2024-02-01T10:49:08.277 DEBUG:boto:Params: {} 2024-02-01T10:49:08.277 DEBUG:boto:Token: None 2024-02-01T10:49:08.277 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:08 GMT /vclfcn-64/ 2024-02-01T10:49:08.278 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:f0BzGQrCVXluuLGn1s8cbpxOQUE= 2024-02-01T10:49:08.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:f0BzGQrCVXluuLGn1s8cbpxOQUE=', 'Content-Length': '0'} 2024-02-01T10:49:08.282 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a569dbb8fc0be556-0065bb7724-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:08.324 DEBUG:boto:b'vclfcn-64obj31000falseobj32024-02-01T10:48:26.634Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:08.325 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:49:08.325 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:49:08.325 DEBUG:boto:path=/vclfcn-64/?prefix=obj3 2024-02-01T10:49:08.325 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj3 2024-02-01T10:49:08.325 DEBUG:boto:Method: GET 2024-02-01T10:49:08.325 DEBUG:boto:Path: /vclfcn-64/?prefix=obj3 2024-02-01T10:49:08.325 DEBUG:boto:Data: 2024-02-01T10:49:08.325 DEBUG:boto:Headers: {} 2024-02-01T10:49:08.325 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:08.325 DEBUG:boto:Port: 8000 2024-02-01T10:49:08.325 DEBUG:boto:Params: {} 2024-02-01T10:49:08.326 DEBUG:boto:Token: None 2024-02-01T10:49:08.326 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:08 GMT /vclfcn-64/ 2024-02-01T10:49:08.326 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:f0BzGQrCVXluuLGn1s8cbpxOQUE= 2024-02-01T10:49:08.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:f0BzGQrCVXluuLGn1s8cbpxOQUE=', 'Content-Length': '0'} 2024-02-01T10:49:08.331 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000421d6821a8ca8898-0065bb7724-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:08.331 DEBUG:boto:b'vclfcn-64obj31000falseobj32024-02-01T10:48:26.634Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:08.331 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:49:08.331 DEBUG:boto:path=/vclfcn-64/obj4 2024-02-01T10:49:08.331 DEBUG:boto:auth_path=/vclfcn-64/obj4 2024-02-01T10:49:08.331 DEBUG:boto:path=/vclfcn-64/obj4?versionId=null 2024-02-01T10:49:08.332 DEBUG:boto:auth_path=/vclfcn-64/obj4?versionId=null 2024-02-01T10:49:08.332 DEBUG:boto:Method: GET 2024-02-01T10:49:08.332 DEBUG:boto:Path: /vclfcn-64/obj4?versionId=null 2024-02-01T10:49:08.332 DEBUG:boto:Data: 2024-02-01T10:49:08.332 DEBUG:boto:Headers: {} 2024-02-01T10:49:08.332 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:08.332 DEBUG:boto:Port: 8000 2024-02-01T10:49:08.332 DEBUG:boto:Params: {} 2024-02-01T10:49:08.332 DEBUG:boto:Token: None 2024-02-01T10:49:08.333 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:08 GMT /vclfcn-64/obj4?versionId=null 2024-02-01T10:49:08.333 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:SAK+EbpK9i7ugBM682NnpT/OJF8= 2024-02-01T10:49:08.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:SAK+EbpK9i7ugBM682NnpT/OJF8=', 'Content-Length': '0'} 2024-02-01T10:49:08.336 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f8175c9aab27e66b-0065bb7724-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:08.336 DEBUG:boto:path=/vclfcn-64/obj4 2024-02-01T10:49:08.336 DEBUG:boto:auth_path=/vclfcn-64/obj4 2024-02-01T10:49:08.336 DEBUG:boto:path=/vclfcn-64/obj4?versionId=null 2024-02-01T10:49:08.337 DEBUG:boto:auth_path=/vclfcn-64/obj4?versionId=null 2024-02-01T10:49:08.337 DEBUG:boto:Method: GET 2024-02-01T10:49:08.337 DEBUG:boto:Path: /vclfcn-64/obj4?versionId=null 2024-02-01T10:49:08.337 DEBUG:boto:Data: 2024-02-01T10:49:08.337 DEBUG:boto:Headers: {} 2024-02-01T10:49:08.337 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:08.337 DEBUG:boto:Port: 8000 2024-02-01T10:49:08.337 DEBUG:boto:Params: {} 2024-02-01T10:49:08.337 DEBUG:boto:Token: None 2024-02-01T10:49:08.338 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:08 GMT /vclfcn-64/obj4?versionId=null 2024-02-01T10:49:08.338 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:SAK+EbpK9i7ugBM682NnpT/OJF8= 2024-02-01T10:49:08.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:SAK+EbpK9i7ugBM682NnpT/OJF8=', 'Content-Length': '0'} 2024-02-01T10:49:08.341 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000360d8a8e0dd171c9-0065bb7724-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:08.341 DEBUG:boto:path=/vclfcn-64/obj4 2024-02-01T10:49:08.341 DEBUG:boto:auth_path=/vclfcn-64/obj4 2024-02-01T10:49:08.341 DEBUG:boto:path=/vclfcn-64/obj4?versionId=null 2024-02-01T10:49:08.341 DEBUG:boto:auth_path=/vclfcn-64/obj4?versionId=null 2024-02-01T10:49:08.342 DEBUG:boto:Method: HEAD 2024-02-01T10:49:08.342 DEBUG:boto:Path: /vclfcn-64/obj4?versionId=null 2024-02-01T10:49:08.342 DEBUG:boto:Data: 2024-02-01T10:49:08.342 DEBUG:boto:Headers: {} 2024-02-01T10:49:08.342 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:08.342 DEBUG:boto:Port: 8000 2024-02-01T10:49:08.342 DEBUG:boto:Params: {} 2024-02-01T10:49:08.342 DEBUG:boto:Token: None 2024-02-01T10:49:08.342 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:08 GMT /vclfcn-64/obj4?versionId=null 2024-02-01T10:49:08.343 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:H0Pb7eZ6b6cnHpwU/o8e1GFYs6I= 2024-02-01T10:49:08.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:H0Pb7eZ6b6cnHpwU/o8e1GFYs6I=', 'Content-Length': '0'} 2024-02-01T10:49:08.346 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000045cea1a66a743bd4-0065bb7724-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:08.346 DEBUG:boto:path=/vclfcn-64/obj4 2024-02-01T10:49:08.346 DEBUG:boto:auth_path=/vclfcn-64/obj4 2024-02-01T10:49:08.346 DEBUG:boto:path=/vclfcn-64/obj4?versionId=null 2024-02-01T10:49:08.346 DEBUG:boto:auth_path=/vclfcn-64/obj4?versionId=null 2024-02-01T10:49:08.346 DEBUG:boto:Method: HEAD 2024-02-01T10:49:08.346 DEBUG:boto:Path: /vclfcn-64/obj4?versionId=null 2024-02-01T10:49:08.347 DEBUG:boto:Data: 2024-02-01T10:49:08.347 DEBUG:boto:Headers: {} 2024-02-01T10:49:08.347 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:08.347 DEBUG:boto:Port: 8000 2024-02-01T10:49:08.347 DEBUG:boto:Params: {} 2024-02-01T10:49:08.347 DEBUG:boto:Token: None 2024-02-01T10:49:08.347 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:08 GMT /vclfcn-64/obj4?versionId=null 2024-02-01T10:49:08.347 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:H0Pb7eZ6b6cnHpwU/o8e1GFYs6I= 2024-02-01T10:49:08.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:H0Pb7eZ6b6cnHpwU/o8e1GFYs6I=', 'Content-Length': '0'} 2024-02-01T10:49:08.350 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000057d24f69816f51ac-0065bb7724-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:08.350 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:49:08.350 DEBUG:boto:path=/vclfcn-64/obj4 2024-02-01T10:49:08.350 DEBUG:boto:auth_path=/vclfcn-64/obj4 2024-02-01T10:49:08.351 DEBUG:boto:path=/vclfcn-64/obj4?versionId=null 2024-02-01T10:49:08.351 DEBUG:boto:auth_path=/vclfcn-64/obj4?versionId=null 2024-02-01T10:49:08.351 DEBUG:boto:Method: GET 2024-02-01T10:49:08.351 DEBUG:boto:Path: /vclfcn-64/obj4?versionId=null 2024-02-01T10:49:08.351 DEBUG:boto:Data: 2024-02-01T10:49:08.351 DEBUG:boto:Headers: {} 2024-02-01T10:49:08.351 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:08.351 DEBUG:boto:Port: 8000 2024-02-01T10:49:08.351 DEBUG:boto:Params: {} 2024-02-01T10:49:08.351 DEBUG:boto:Token: None 2024-02-01T10:49:08.352 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:08 GMT /vclfcn-64/obj4?versionId=null 2024-02-01T10:49:08.352 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:SAK+EbpK9i7ugBM682NnpT/OJF8= 2024-02-01T10:49:08.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:SAK+EbpK9i7ugBM682NnpT/OJF8=', 'Content-Length': '0'} 2024-02-01T10:49:08.355 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f91c49377ba86233-0065bb7724-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:08.396 DEBUG:boto:path=/vclfcn-64/obj4 2024-02-01T10:49:08.396 DEBUG:boto:auth_path=/vclfcn-64/obj4 2024-02-01T10:49:08.396 DEBUG:boto:path=/vclfcn-64/obj4?versionId=null 2024-02-01T10:49:08.396 DEBUG:boto:auth_path=/vclfcn-64/obj4?versionId=null 2024-02-01T10:49:08.397 DEBUG:boto:Method: GET 2024-02-01T10:49:08.397 DEBUG:boto:Path: /vclfcn-64/obj4?versionId=null 2024-02-01T10:49:08.397 DEBUG:boto:Data: 2024-02-01T10:49:08.397 DEBUG:boto:Headers: {} 2024-02-01T10:49:08.397 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:08.397 DEBUG:boto:Port: 8000 2024-02-01T10:49:08.397 DEBUG:boto:Params: {} 2024-02-01T10:49:08.397 DEBUG:boto:Token: None 2024-02-01T10:49:08.397 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:08 GMT /vclfcn-64/obj4?versionId=null 2024-02-01T10:49:08.397 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:SAK+EbpK9i7ugBM682NnpT/OJF8= 2024-02-01T10:49:08.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:SAK+EbpK9i7ugBM682NnpT/OJF8=', 'Content-Length': '0'} 2024-02-01T10:49:08.400 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008e713254197be5ee-0065bb7724-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:08.444 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:49:08.444 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:49:08.444 DEBUG:boto:path=/vclfcn-64/?prefix=obj4 2024-02-01T10:49:08.444 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj4 2024-02-01T10:49:08.445 DEBUG:boto:Method: GET 2024-02-01T10:49:08.445 DEBUG:boto:Path: /vclfcn-64/?prefix=obj4 2024-02-01T10:49:08.445 DEBUG:boto:Data: 2024-02-01T10:49:08.445 DEBUG:boto:Headers: {} 2024-02-01T10:49:08.445 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:08.445 DEBUG:boto:Port: 8000 2024-02-01T10:49:08.445 DEBUG:boto:Params: {} 2024-02-01T10:49:08.445 DEBUG:boto:Token: None 2024-02-01T10:49:08.445 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:08 GMT /vclfcn-64/ 2024-02-01T10:49:08.446 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:f0BzGQrCVXluuLGn1s8cbpxOQUE= 2024-02-01T10:49:08.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:f0BzGQrCVXluuLGn1s8cbpxOQUE=', 'Content-Length': '0'} 2024-02-01T10:49:08.452 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a3466e90bcf29bef-0065bb7724-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:08.452 DEBUG:boto:b'vclfcn-64obj41000falseobj42024-02-01T10:48:26.689Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:08.453 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:49:08.453 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:49:08.453 DEBUG:boto:path=/vclfcn-64/?prefix=obj4 2024-02-01T10:49:08.453 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj4 2024-02-01T10:49:08.453 DEBUG:boto:Method: GET 2024-02-01T10:49:08.453 DEBUG:boto:Path: /vclfcn-64/?prefix=obj4 2024-02-01T10:49:08.453 DEBUG:boto:Data: 2024-02-01T10:49:08.453 DEBUG:boto:Headers: {} 2024-02-01T10:49:08.453 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:08.453 DEBUG:boto:Port: 8000 2024-02-01T10:49:08.454 DEBUG:boto:Params: {} 2024-02-01T10:49:08.454 DEBUG:boto:Token: None 2024-02-01T10:49:08.454 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:08 GMT /vclfcn-64/ 2024-02-01T10:49:08.454 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:f0BzGQrCVXluuLGn1s8cbpxOQUE= 2024-02-01T10:49:08.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:f0BzGQrCVXluuLGn1s8cbpxOQUE=', 'Content-Length': '0'} 2024-02-01T10:49:08.458 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000077a9d1767d4b89d-0065bb7724-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:08.458 DEBUG:boto:b'vclfcn-64obj41000falseobj42024-02-01T10:48:26.689Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:08.458 DEBUG:boto:path=/vclfcn-64/obj4 2024-02-01T10:49:08.459 DEBUG:boto:auth_path=/vclfcn-64/obj4 2024-02-01T10:49:08.459 DEBUG:boto:Method: HEAD 2024-02-01T10:49:08.459 DEBUG:boto:Path: /vclfcn-64/obj4 2024-02-01T10:49:08.459 DEBUG:boto:Data: 2024-02-01T10:49:08.459 DEBUG:boto:Headers: {} 2024-02-01T10:49:08.459 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:08.459 DEBUG:boto:Port: 8000 2024-02-01T10:49:08.459 DEBUG:boto:Params: {} 2024-02-01T10:49:08.460 DEBUG:boto:Token: None 2024-02-01T10:49:08.460 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:08 GMT /vclfcn-64/obj4 2024-02-01T10:49:08.460 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5NGv5ajr6mxnJRS77NDHXDcG50Y= 2024-02-01T10:49:08.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5NGv5ajr6mxnJRS77NDHXDcG50Y=', 'Content-Length': '0'} 2024-02-01T10:49:08.463 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bb426f0d200acfac-0065bb7724-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:08.463 DEBUG:boto:path=/vclfcn-64/obj4 2024-02-01T10:49:08.463 DEBUG:boto:auth_path=/vclfcn-64/obj4 2024-02-01T10:49:08.464 DEBUG:boto:Method: HEAD 2024-02-01T10:49:08.464 DEBUG:boto:Path: /vclfcn-64/obj4 2024-02-01T10:49:08.464 DEBUG:boto:Data: 2024-02-01T10:49:08.464 DEBUG:boto:Headers: {} 2024-02-01T10:49:08.464 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:08.464 DEBUG:boto:Port: 8000 2024-02-01T10:49:08.464 DEBUG:boto:Params: {} 2024-02-01T10:49:08.464 DEBUG:boto:Token: None 2024-02-01T10:49:08.464 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:08 GMT /vclfcn-64/obj4 2024-02-01T10:49:08.464 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5NGv5ajr6mxnJRS77NDHXDcG50Y= 2024-02-01T10:49:08.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5NGv5ajr6mxnJRS77NDHXDcG50Y=', 'Content-Length': '0'} 2024-02-01T10:49:08.467 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f3efb5edad66eb80-0065bb7724-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:08.468 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:49:08.468 DEBUG:boto:path=/vclfcn-64/obj4 2024-02-01T10:49:08.468 DEBUG:boto:auth_path=/vclfcn-64/obj4 2024-02-01T10:49:08.468 DEBUG:boto:Method: GET 2024-02-01T10:49:08.468 DEBUG:boto:Path: /vclfcn-64/obj4 2024-02-01T10:49:08.468 DEBUG:boto:Data: 2024-02-01T10:49:08.468 DEBUG:boto:Headers: {} 2024-02-01T10:49:08.468 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:08.468 DEBUG:boto:Port: 8000 2024-02-01T10:49:08.469 DEBUG:boto:Params: {} 2024-02-01T10:49:08.469 DEBUG:boto:Token: None 2024-02-01T10:49:08.469 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:08 GMT /vclfcn-64/obj4 2024-02-01T10:49:08.469 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GRt6hDs+Jteh2Ffx1FGjA44HaVk= 2024-02-01T10:49:08.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GRt6hDs+Jteh2Ffx1FGjA44HaVk=', 'Content-Length': '0'} 2024-02-01T10:49:08.471 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000089649b79b2a67617-0065bb7724-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:08.512 DEBUG:boto:path=/vclfcn-64/obj4 2024-02-01T10:49:08.513 DEBUG:boto:auth_path=/vclfcn-64/obj4 2024-02-01T10:49:08.513 DEBUG:boto:Method: GET 2024-02-01T10:49:08.513 DEBUG:boto:Path: /vclfcn-64/obj4 2024-02-01T10:49:08.513 DEBUG:boto:Data: 2024-02-01T10:49:08.513 DEBUG:boto:Headers: {} 2024-02-01T10:49:08.513 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:08.513 DEBUG:boto:Port: 8000 2024-02-01T10:49:08.513 DEBUG:boto:Params: {} 2024-02-01T10:49:08.513 DEBUG:boto:Token: None 2024-02-01T10:49:08.514 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:08 GMT /vclfcn-64/obj4 2024-02-01T10:49:08.514 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GRt6hDs+Jteh2Ffx1FGjA44HaVk= 2024-02-01T10:49:08.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GRt6hDs+Jteh2Ffx1FGjA44HaVk=', 'Content-Length': '0'} 2024-02-01T10:49:08.517 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ed2a0cbf92b17342-0065bb7724-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:08.560 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:49:08.560 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:49:08.560 DEBUG:boto:path=/vclfcn-64/?prefix=obj4 2024-02-01T10:49:08.560 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj4 2024-02-01T10:49:08.561 DEBUG:boto:Method: GET 2024-02-01T10:49:08.561 DEBUG:boto:Path: /vclfcn-64/?prefix=obj4 2024-02-01T10:49:08.561 DEBUG:boto:Data: 2024-02-01T10:49:08.561 DEBUG:boto:Headers: {} 2024-02-01T10:49:08.561 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:08.561 DEBUG:boto:Port: 8000 2024-02-01T10:49:08.561 DEBUG:boto:Params: {} 2024-02-01T10:49:08.561 DEBUG:boto:Token: None 2024-02-01T10:49:08.561 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:08 GMT /vclfcn-64/ 2024-02-01T10:49:08.562 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:f0BzGQrCVXluuLGn1s8cbpxOQUE= 2024-02-01T10:49:08.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:f0BzGQrCVXluuLGn1s8cbpxOQUE=', 'Content-Length': '0'} 2024-02-01T10:49:08.566 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9dd37738d55816f-0065bb7724-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:08.566 DEBUG:boto:b'vclfcn-64obj41000falseobj42024-02-01T10:48:26.689Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:08.566 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:49:08.567 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:49:08.567 DEBUG:boto:path=/vclfcn-64/?prefix=obj4 2024-02-01T10:49:08.567 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj4 2024-02-01T10:49:08.567 DEBUG:boto:Method: GET 2024-02-01T10:49:08.567 DEBUG:boto:Path: /vclfcn-64/?prefix=obj4 2024-02-01T10:49:08.567 DEBUG:boto:Data: 2024-02-01T10:49:08.567 DEBUG:boto:Headers: {} 2024-02-01T10:49:08.567 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:08.567 DEBUG:boto:Port: 8000 2024-02-01T10:49:08.567 DEBUG:boto:Params: {} 2024-02-01T10:49:08.567 DEBUG:boto:Token: None 2024-02-01T10:49:08.568 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:08 GMT /vclfcn-64/ 2024-02-01T10:49:08.568 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:f0BzGQrCVXluuLGn1s8cbpxOQUE= 2024-02-01T10:49:08.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:f0BzGQrCVXluuLGn1s8cbpxOQUE=', 'Content-Length': '0'} 2024-02-01T10:49:08.572 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f11554039a4c2ca-0065bb7724-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:08.616 DEBUG:boto:b'vclfcn-64obj41000falseobj42024-02-01T10:48:26.689Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:08.616 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-64 zones={test-zone2, test-zone1} 2024-02-01T10:49:08.617 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-64 zones={test-zone3, test-zone1} 2024-02-01T10:49:08.617 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:49:08.617 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:49:08.617 DEBUG:boto:Method: HEAD 2024-02-01T10:49:08.617 DEBUG:boto:Path: /vclfcn-64/ 2024-02-01T10:49:08.617 DEBUG:boto:Data: 2024-02-01T10:49:08.617 DEBUG:boto:Headers: {} 2024-02-01T10:49:08.617 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:08.617 DEBUG:boto:Port: 8001 2024-02-01T10:49:08.617 DEBUG:boto:Params: {} 2024-02-01T10:49:08.618 DEBUG:boto:Token: None 2024-02-01T10:49:08.618 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:08 GMT /vclfcn-64/ 2024-02-01T10:49:08.618 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:n6ZDTZsFH1bIDC2yssSIwooI42s= 2024-02-01T10:49:08.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:n6ZDTZsFH1bIDC2yssSIwooI42s=', 'Content-Length': '0'} 2024-02-01T10:49:08.622 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000092ab3633823c675f-0065bb7724-5762-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:49:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:08.623 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:49:08.623 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:49:08.623 DEBUG:boto:Method: HEAD 2024-02-01T10:49:08.623 DEBUG:boto:Path: /vclfcn-64/ 2024-02-01T10:49:08.623 DEBUG:boto:Data: 2024-02-01T10:49:08.623 DEBUG:boto:Headers: {} 2024-02-01T10:49:08.623 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:08.623 DEBUG:boto:Port: 8000 2024-02-01T10:49:08.623 DEBUG:boto:Params: {} 2024-02-01T10:49:08.623 DEBUG:boto:Token: None 2024-02-01T10:49:08.624 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:08 GMT /vclfcn-64/ 2024-02-01T10:49:08.624 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:n6ZDTZsFH1bIDC2yssSIwooI42s= 2024-02-01T10:49:08.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:n6ZDTZsFH1bIDC2yssSIwooI42s=', 'Content-Length': '0'} 2024-02-01T10:49:08.628 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000ab3a58d05a54e62-0065bb7724-5765-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:49:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:08.628 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:49:08.628 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:49:08.628 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:49:08.628 DEBUG:boto:path=/vclfcn-64/?versions&max-keys=999 2024-02-01T10:49:08.628 DEBUG:boto:auth_path=/vclfcn-64/?versions&max-keys=999 2024-02-01T10:49:08.628 DEBUG:boto:Method: GET 2024-02-01T10:49:08.629 DEBUG:boto:Path: /vclfcn-64/?versions&max-keys=999 2024-02-01T10:49:08.629 DEBUG:boto:Data: 2024-02-01T10:49:08.629 DEBUG:boto:Headers: {} 2024-02-01T10:49:08.629 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:08.629 DEBUG:boto:Port: 8001 2024-02-01T10:49:08.629 DEBUG:boto:Params: {} 2024-02-01T10:49:08.629 DEBUG:boto:Token: None 2024-02-01T10:49:08.629 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:08 GMT /vclfcn-64/?versions 2024-02-01T10:49:08.629 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QIvAUl2KKYwY8A9NneXRacXN7hs= 2024-02-01T10:49:08.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QIvAUl2KKYwY8A9NneXRacXN7hs=', 'Content-Length': '0'} 2024-02-01T10:49:08.633 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000079025d24305a5e6b-0065bb7724-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:08.633 DEBUG:boto:b'vclfcn-64999falseobj1nulltrue2024-02-01T10:48:26.595Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:48:26.615Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:48:26.634Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:48:26.689Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:08.634 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-02-01T10:49:08.634 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-02-01T10:49:08.634 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-02-01T10:49:08.634 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-02-01T10:49:08.634 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:49:08.634 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:49:08.634 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:49:08.635 DEBUG:boto:path=/vclfcn-64/?versions&max-keys=999 2024-02-01T10:49:08.635 DEBUG:boto:auth_path=/vclfcn-64/?versions&max-keys=999 2024-02-01T10:49:08.635 DEBUG:boto:Method: GET 2024-02-01T10:49:08.635 DEBUG:boto:Path: /vclfcn-64/?versions&max-keys=999 2024-02-01T10:49:08.635 DEBUG:boto:Data: 2024-02-01T10:49:08.635 DEBUG:boto:Headers: {} 2024-02-01T10:49:08.635 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:08.635 DEBUG:boto:Port: 8000 2024-02-01T10:49:08.635 DEBUG:boto:Params: {} 2024-02-01T10:49:08.635 DEBUG:boto:Token: None 2024-02-01T10:49:08.636 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:08 GMT /vclfcn-64/?versions 2024-02-01T10:49:08.636 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QIvAUl2KKYwY8A9NneXRacXN7hs= 2024-02-01T10:49:08.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QIvAUl2KKYwY8A9NneXRacXN7hs=', 'Content-Length': '0'} 2024-02-01T10:49:08.640 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7129151ebbec745-0065bb7724-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:08.640 DEBUG:boto:b'vclfcn-64999falseobj1nulltrue2024-02-01T10:48:26.595Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:48:26.615Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:48:26.634Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:48:26.689Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:08.641 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-02-01T10:49:08.641 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-02-01T10:49:08.641 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-02-01T10:49:08.641 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-02-01T10:49:08.641 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:49:08.641 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:49:08.641 DEBUG:boto:path=/vclfcn-64/?versions&max-keys=999 2024-02-01T10:49:08.641 DEBUG:boto:auth_path=/vclfcn-64/?versions&max-keys=999 2024-02-01T10:49:08.641 DEBUG:boto:Method: GET 2024-02-01T10:49:08.641 DEBUG:boto:Path: /vclfcn-64/?versions&max-keys=999 2024-02-01T10:49:08.642 DEBUG:boto:Data: 2024-02-01T10:49:08.642 DEBUG:boto:Headers: {} 2024-02-01T10:49:08.642 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:08.642 DEBUG:boto:Port: 8001 2024-02-01T10:49:08.642 DEBUG:boto:Params: {} 2024-02-01T10:49:08.642 DEBUG:boto:Token: None 2024-02-01T10:49:08.642 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:08 GMT /vclfcn-64/?versions 2024-02-01T10:49:08.642 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QIvAUl2KKYwY8A9NneXRacXN7hs= 2024-02-01T10:49:08.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QIvAUl2KKYwY8A9NneXRacXN7hs=', 'Content-Length': '0'} 2024-02-01T10:49:08.646 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000027093524fedcfda-0065bb7724-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:08.646 DEBUG:boto:b'vclfcn-64999falseobj1nulltrue2024-02-01T10:48:26.595Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:48:26.615Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:48:26.634Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:48:26.689Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:08.647 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:49:08.647 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:49:08.647 DEBUG:boto:path=/vclfcn-64/?versions&max-keys=999 2024-02-01T10:49:08.647 DEBUG:boto:auth_path=/vclfcn-64/?versions&max-keys=999 2024-02-01T10:49:08.647 DEBUG:boto:Method: GET 2024-02-01T10:49:08.647 DEBUG:boto:Path: /vclfcn-64/?versions&max-keys=999 2024-02-01T10:49:08.647 DEBUG:boto:Data: 2024-02-01T10:49:08.648 DEBUG:boto:Headers: {} 2024-02-01T10:49:08.648 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:08.648 DEBUG:boto:Port: 8000 2024-02-01T10:49:08.648 DEBUG:boto:Params: {} 2024-02-01T10:49:08.648 DEBUG:boto:Token: None 2024-02-01T10:49:08.648 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:08 GMT /vclfcn-64/?versions 2024-02-01T10:49:08.648 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QIvAUl2KKYwY8A9NneXRacXN7hs= 2024-02-01T10:49:08.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QIvAUl2KKYwY8A9NneXRacXN7hs=', 'Content-Length': '0'} 2024-02-01T10:49:08.652 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b389adae59af454a-0065bb7724-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:08.653 DEBUG:boto:b'vclfcn-64999falseobj1nulltrue2024-02-01T10:48:26.595Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:48:26.615Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:48:26.634Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:48:26.689Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:08.655 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:49:08.655 DEBUG:boto:path=/vclfcn-64/obj1 2024-02-01T10:49:08.655 DEBUG:boto:auth_path=/vclfcn-64/obj1 2024-02-01T10:49:08.655 DEBUG:boto:path=/vclfcn-64/obj1?versionId=null 2024-02-01T10:49:08.655 DEBUG:boto:auth_path=/vclfcn-64/obj1?versionId=null 2024-02-01T10:49:08.655 DEBUG:boto:Method: GET 2024-02-01T10:49:08.655 DEBUG:boto:Path: /vclfcn-64/obj1?versionId=null 2024-02-01T10:49:08.656 DEBUG:boto:Data: 2024-02-01T10:49:08.656 DEBUG:boto:Headers: {} 2024-02-01T10:49:08.656 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:08.656 DEBUG:boto:Port: 8001 2024-02-01T10:49:08.656 DEBUG:boto:Params: {} 2024-02-01T10:49:08.656 DEBUG:boto:Token: None 2024-02-01T10:49:08.656 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:08 GMT /vclfcn-64/obj1?versionId=null 2024-02-01T10:49:08.656 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0J/dRm6TVOkrrW7mh6g+7Yk/tb8= 2024-02-01T10:49:08.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0J/dRm6TVOkrrW7mh6g+7Yk/tb8=', 'Content-Length': '0'} 2024-02-01T10:49:08.659 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d87bd6cd72baae72-0065bb7724-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:08.700 DEBUG:boto:path=/vclfcn-64/obj1 2024-02-01T10:49:08.700 DEBUG:boto:auth_path=/vclfcn-64/obj1 2024-02-01T10:49:08.700 DEBUG:boto:path=/vclfcn-64/obj1?versionId=null 2024-02-01T10:49:08.700 DEBUG:boto:auth_path=/vclfcn-64/obj1?versionId=null 2024-02-01T10:49:08.701 DEBUG:boto:Method: GET 2024-02-01T10:49:08.701 DEBUG:boto:Path: /vclfcn-64/obj1?versionId=null 2024-02-01T10:49:08.701 DEBUG:boto:Data: 2024-02-01T10:49:08.701 DEBUG:boto:Headers: {} 2024-02-01T10:49:08.701 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:08.701 DEBUG:boto:Port: 8000 2024-02-01T10:49:08.701 DEBUG:boto:Params: {} 2024-02-01T10:49:08.701 DEBUG:boto:Token: None 2024-02-01T10:49:08.701 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:08 GMT /vclfcn-64/obj1?versionId=null 2024-02-01T10:49:08.702 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0J/dRm6TVOkrrW7mh6g+7Yk/tb8= 2024-02-01T10:49:08.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0J/dRm6TVOkrrW7mh6g+7Yk/tb8=', 'Content-Length': '0'} 2024-02-01T10:49:08.704 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000067037ae90ee87442-0065bb7724-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:08.705 DEBUG:boto:path=/vclfcn-64/obj1 2024-02-01T10:49:08.705 DEBUG:boto:auth_path=/vclfcn-64/obj1 2024-02-01T10:49:08.705 DEBUG:boto:path=/vclfcn-64/obj1?versionId=null 2024-02-01T10:49:08.705 DEBUG:boto:auth_path=/vclfcn-64/obj1?versionId=null 2024-02-01T10:49:08.705 DEBUG:boto:Method: HEAD 2024-02-01T10:49:08.705 DEBUG:boto:Path: /vclfcn-64/obj1?versionId=null 2024-02-01T10:49:08.705 DEBUG:boto:Data: 2024-02-01T10:49:08.705 DEBUG:boto:Headers: {} 2024-02-01T10:49:08.706 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:08.706 DEBUG:boto:Port: 8001 2024-02-01T10:49:08.706 DEBUG:boto:Params: {} 2024-02-01T10:49:08.706 DEBUG:boto:Token: None 2024-02-01T10:49:08.706 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:08 GMT /vclfcn-64/obj1?versionId=null 2024-02-01T10:49:08.706 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:f9XSKplhu0D0RiMa5eyWe6Etb0c= 2024-02-01T10:49:08.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:f9XSKplhu0D0RiMa5eyWe6Etb0c=', 'Content-Length': '0'} 2024-02-01T10:49:08.709 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000038a4b8a82a1c579b-0065bb7724-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:08.709 DEBUG:boto:path=/vclfcn-64/obj1 2024-02-01T10:49:08.709 DEBUG:boto:auth_path=/vclfcn-64/obj1 2024-02-01T10:49:08.709 DEBUG:boto:path=/vclfcn-64/obj1?versionId=null 2024-02-01T10:49:08.709 DEBUG:boto:auth_path=/vclfcn-64/obj1?versionId=null 2024-02-01T10:49:08.709 DEBUG:boto:Method: HEAD 2024-02-01T10:49:08.709 DEBUG:boto:Path: /vclfcn-64/obj1?versionId=null 2024-02-01T10:49:08.710 DEBUG:boto:Data: 2024-02-01T10:49:08.710 DEBUG:boto:Headers: {} 2024-02-01T10:49:08.710 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:08.710 DEBUG:boto:Port: 8000 2024-02-01T10:49:08.710 DEBUG:boto:Params: {} 2024-02-01T10:49:08.710 DEBUG:boto:Token: None 2024-02-01T10:49:08.710 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:08 GMT /vclfcn-64/obj1?versionId=null 2024-02-01T10:49:08.710 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:f9XSKplhu0D0RiMa5eyWe6Etb0c= 2024-02-01T10:49:08.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:f9XSKplhu0D0RiMa5eyWe6Etb0c=', 'Content-Length': '0'} 2024-02-01T10:49:08.713 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cde3aa4103f1dd77-0065bb7724-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:08.714 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:49:08.714 DEBUG:boto:path=/vclfcn-64/obj1 2024-02-01T10:49:08.714 DEBUG:boto:auth_path=/vclfcn-64/obj1 2024-02-01T10:49:08.714 DEBUG:boto:path=/vclfcn-64/obj1?versionId=null 2024-02-01T10:49:08.714 DEBUG:boto:auth_path=/vclfcn-64/obj1?versionId=null 2024-02-01T10:49:08.715 DEBUG:boto:Method: GET 2024-02-01T10:49:08.715 DEBUG:boto:Path: /vclfcn-64/obj1?versionId=null 2024-02-01T10:49:08.715 DEBUG:boto:Data: 2024-02-01T10:49:08.715 DEBUG:boto:Headers: {} 2024-02-01T10:49:08.715 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:08.715 DEBUG:boto:Port: 8001 2024-02-01T10:49:08.715 DEBUG:boto:Params: {} 2024-02-01T10:49:08.715 DEBUG:boto:Token: None 2024-02-01T10:49:08.716 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:08 GMT /vclfcn-64/obj1?versionId=null 2024-02-01T10:49:08.716 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0J/dRm6TVOkrrW7mh6g+7Yk/tb8= 2024-02-01T10:49:08.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0J/dRm6TVOkrrW7mh6g+7Yk/tb8=', 'Content-Length': '0'} 2024-02-01T10:49:08.719 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000084bec577fe33d064-0065bb7724-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:08.719 DEBUG:boto:path=/vclfcn-64/obj1 2024-02-01T10:49:08.720 DEBUG:boto:auth_path=/vclfcn-64/obj1 2024-02-01T10:49:08.720 DEBUG:boto:path=/vclfcn-64/obj1?versionId=null 2024-02-01T10:49:08.720 DEBUG:boto:auth_path=/vclfcn-64/obj1?versionId=null 2024-02-01T10:49:08.720 DEBUG:boto:Method: GET 2024-02-01T10:49:08.720 DEBUG:boto:Path: /vclfcn-64/obj1?versionId=null 2024-02-01T10:49:08.720 DEBUG:boto:Data: 2024-02-01T10:49:08.720 DEBUG:boto:Headers: {} 2024-02-01T10:49:08.720 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:08.720 DEBUG:boto:Port: 8000 2024-02-01T10:49:08.720 DEBUG:boto:Params: {} 2024-02-01T10:49:08.720 DEBUG:boto:Token: None 2024-02-01T10:49:08.721 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:08 GMT /vclfcn-64/obj1?versionId=null 2024-02-01T10:49:08.721 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0J/dRm6TVOkrrW7mh6g+7Yk/tb8= 2024-02-01T10:49:08.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0J/dRm6TVOkrrW7mh6g+7Yk/tb8=', 'Content-Length': '0'} 2024-02-01T10:49:08.724 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000017fb57cd8c95e426-0065bb7724-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:08.764 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:49:08.764 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:49:08.764 DEBUG:boto:path=/vclfcn-64/?prefix=obj1 2024-02-01T10:49:08.764 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj1 2024-02-01T10:49:08.765 DEBUG:boto:Method: GET 2024-02-01T10:49:08.765 DEBUG:boto:Path: /vclfcn-64/?prefix=obj1 2024-02-01T10:49:08.765 DEBUG:boto:Data: 2024-02-01T10:49:08.765 DEBUG:boto:Headers: {} 2024-02-01T10:49:08.765 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:08.765 DEBUG:boto:Port: 8001 2024-02-01T10:49:08.765 DEBUG:boto:Params: {} 2024-02-01T10:49:08.765 DEBUG:boto:Token: None 2024-02-01T10:49:08.765 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:08 GMT /vclfcn-64/ 2024-02-01T10:49:08.766 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:f0BzGQrCVXluuLGn1s8cbpxOQUE= 2024-02-01T10:49:08.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:f0BzGQrCVXluuLGn1s8cbpxOQUE=', 'Content-Length': '0'} 2024-02-01T10:49:08.770 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2532915fa12d60d-0065bb7724-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:08.771 DEBUG:boto:b'vclfcn-64obj11000falseobj12024-02-01T10:48:26.595Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:08.772 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:49:08.772 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:49:08.772 DEBUG:boto:path=/vclfcn-64/?prefix=obj1 2024-02-01T10:49:08.772 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj1 2024-02-01T10:49:08.772 DEBUG:boto:Method: GET 2024-02-01T10:49:08.772 DEBUG:boto:Path: /vclfcn-64/?prefix=obj1 2024-02-01T10:49:08.773 DEBUG:boto:Data: 2024-02-01T10:49:08.773 DEBUG:boto:Headers: {} 2024-02-01T10:49:08.773 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:08.773 DEBUG:boto:Port: 8000 2024-02-01T10:49:08.773 DEBUG:boto:Params: {} 2024-02-01T10:49:08.773 DEBUG:boto:Token: None 2024-02-01T10:49:08.774 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:08 GMT /vclfcn-64/ 2024-02-01T10:49:08.774 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:f0BzGQrCVXluuLGn1s8cbpxOQUE= 2024-02-01T10:49:08.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:f0BzGQrCVXluuLGn1s8cbpxOQUE=', 'Content-Length': '0'} 2024-02-01T10:49:08.778 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000888c860fe13b483b-0065bb7724-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:08.778 DEBUG:boto:b'vclfcn-64obj11000falseobj12024-02-01T10:48:26.595Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:08.779 DEBUG:boto:path=/vclfcn-64/obj1 2024-02-01T10:49:08.779 DEBUG:boto:auth_path=/vclfcn-64/obj1 2024-02-01T10:49:08.779 DEBUG:boto:Method: HEAD 2024-02-01T10:49:08.779 DEBUG:boto:Path: /vclfcn-64/obj1 2024-02-01T10:49:08.779 DEBUG:boto:Data: 2024-02-01T10:49:08.779 DEBUG:boto:Headers: {} 2024-02-01T10:49:08.780 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:08.780 DEBUG:boto:Port: 8001 2024-02-01T10:49:08.780 DEBUG:boto:Params: {} 2024-02-01T10:49:08.780 DEBUG:boto:Token: None 2024-02-01T10:49:08.780 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:08 GMT /vclfcn-64/obj1 2024-02-01T10:49:08.780 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ssnAIOqXTM3dCer9pvreXiii9qU= 2024-02-01T10:49:08.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ssnAIOqXTM3dCer9pvreXiii9qU=', 'Content-Length': '0'} 2024-02-01T10:49:08.783 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a26033522a147f72-0065bb7724-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:08.784 DEBUG:boto:path=/vclfcn-64/obj1 2024-02-01T10:49:08.784 DEBUG:boto:auth_path=/vclfcn-64/obj1 2024-02-01T10:49:08.784 DEBUG:boto:Method: HEAD 2024-02-01T10:49:08.784 DEBUG:boto:Path: /vclfcn-64/obj1 2024-02-01T10:49:08.784 DEBUG:boto:Data: 2024-02-01T10:49:08.784 DEBUG:boto:Headers: {} 2024-02-01T10:49:08.784 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:08.784 DEBUG:boto:Port: 8000 2024-02-01T10:49:08.784 DEBUG:boto:Params: {} 2024-02-01T10:49:08.784 DEBUG:boto:Token: None 2024-02-01T10:49:08.785 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:08 GMT /vclfcn-64/obj1 2024-02-01T10:49:08.785 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ssnAIOqXTM3dCer9pvreXiii9qU= 2024-02-01T10:49:08.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ssnAIOqXTM3dCer9pvreXiii9qU=', 'Content-Length': '0'} 2024-02-01T10:49:08.788 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b8c5920082d5405c-0065bb7724-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:08.788 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:49:08.788 DEBUG:boto:path=/vclfcn-64/obj1 2024-02-01T10:49:08.788 DEBUG:boto:auth_path=/vclfcn-64/obj1 2024-02-01T10:49:08.788 DEBUG:boto:Method: GET 2024-02-01T10:49:08.788 DEBUG:boto:Path: /vclfcn-64/obj1 2024-02-01T10:49:08.788 DEBUG:boto:Data: 2024-02-01T10:49:08.789 DEBUG:boto:Headers: {} 2024-02-01T10:49:08.789 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:08.789 DEBUG:boto:Port: 8001 2024-02-01T10:49:08.789 DEBUG:boto:Params: {} 2024-02-01T10:49:08.790 DEBUG:boto:Token: None 2024-02-01T10:49:08.790 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:08 GMT /vclfcn-64/obj1 2024-02-01T10:49:08.790 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8wI3Y5WnDZPoKNmYekhIrogxPvg= 2024-02-01T10:49:08.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8wI3Y5WnDZPoKNmYekhIrogxPvg=', 'Content-Length': '0'} 2024-02-01T10:49:08.794 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d91d48d2bebe752f-0065bb7724-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:08.836 DEBUG:boto:path=/vclfcn-64/obj1 2024-02-01T10:49:08.836 DEBUG:boto:auth_path=/vclfcn-64/obj1 2024-02-01T10:49:08.836 DEBUG:boto:Method: GET 2024-02-01T10:49:08.837 DEBUG:boto:Path: /vclfcn-64/obj1 2024-02-01T10:49:08.837 DEBUG:boto:Data: 2024-02-01T10:49:08.837 DEBUG:boto:Headers: {} 2024-02-01T10:49:08.837 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:08.837 DEBUG:boto:Port: 8000 2024-02-01T10:49:08.837 DEBUG:boto:Params: {} 2024-02-01T10:49:08.837 DEBUG:boto:Token: None 2024-02-01T10:49:08.838 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:08 GMT /vclfcn-64/obj1 2024-02-01T10:49:08.838 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8wI3Y5WnDZPoKNmYekhIrogxPvg= 2024-02-01T10:49:08.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8wI3Y5WnDZPoKNmYekhIrogxPvg=', 'Content-Length': '0'} 2024-02-01T10:49:08.841 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000048c8660496231c0d-0065bb7724-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:08.884 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:49:08.884 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:49:08.884 DEBUG:boto:path=/vclfcn-64/?prefix=obj1 2024-02-01T10:49:08.885 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj1 2024-02-01T10:49:08.885 DEBUG:boto:Method: GET 2024-02-01T10:49:08.885 DEBUG:boto:Path: /vclfcn-64/?prefix=obj1 2024-02-01T10:49:08.885 DEBUG:boto:Data: 2024-02-01T10:49:08.885 DEBUG:boto:Headers: {} 2024-02-01T10:49:08.885 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:08.885 DEBUG:boto:Port: 8001 2024-02-01T10:49:08.885 DEBUG:boto:Params: {} 2024-02-01T10:49:08.885 DEBUG:boto:Token: None 2024-02-01T10:49:08.885 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:08 GMT /vclfcn-64/ 2024-02-01T10:49:08.886 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:f0BzGQrCVXluuLGn1s8cbpxOQUE= 2024-02-01T10:49:08.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:f0BzGQrCVXluuLGn1s8cbpxOQUE=', 'Content-Length': '0'} 2024-02-01T10:49:08.890 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000236e06958b2d0f5a-0065bb7724-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:08.890 DEBUG:boto:b'vclfcn-64obj11000falseobj12024-02-01T10:48:26.595Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:08.891 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:49:08.891 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:49:08.891 DEBUG:boto:path=/vclfcn-64/?prefix=obj1 2024-02-01T10:49:08.891 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj1 2024-02-01T10:49:08.891 DEBUG:boto:Method: GET 2024-02-01T10:49:08.891 DEBUG:boto:Path: /vclfcn-64/?prefix=obj1 2024-02-01T10:49:08.891 DEBUG:boto:Data: 2024-02-01T10:49:08.891 DEBUG:boto:Headers: {} 2024-02-01T10:49:08.892 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:08.892 DEBUG:boto:Port: 8000 2024-02-01T10:49:08.892 DEBUG:boto:Params: {} 2024-02-01T10:49:08.892 DEBUG:boto:Token: None 2024-02-01T10:49:08.892 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:08 GMT /vclfcn-64/ 2024-02-01T10:49:08.892 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:f0BzGQrCVXluuLGn1s8cbpxOQUE= 2024-02-01T10:49:08.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:f0BzGQrCVXluuLGn1s8cbpxOQUE=', 'Content-Length': '0'} 2024-02-01T10:49:08.896 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000699879160c0af6ea-0065bb7724-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:08.940 DEBUG:boto:b'vclfcn-64obj11000falseobj12024-02-01T10:48:26.595Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:08.941 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:49:08.941 DEBUG:boto:path=/vclfcn-64/obj2 2024-02-01T10:49:08.941 DEBUG:boto:auth_path=/vclfcn-64/obj2 2024-02-01T10:49:08.941 DEBUG:boto:path=/vclfcn-64/obj2?versionId=null 2024-02-01T10:49:08.941 DEBUG:boto:auth_path=/vclfcn-64/obj2?versionId=null 2024-02-01T10:49:08.941 DEBUG:boto:Method: GET 2024-02-01T10:49:08.941 DEBUG:boto:Path: /vclfcn-64/obj2?versionId=null 2024-02-01T10:49:08.941 DEBUG:boto:Data: 2024-02-01T10:49:08.941 DEBUG:boto:Headers: {} 2024-02-01T10:49:08.941 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:08.942 DEBUG:boto:Port: 8001 2024-02-01T10:49:08.942 DEBUG:boto:Params: {} 2024-02-01T10:49:08.942 DEBUG:boto:Token: None 2024-02-01T10:49:08.942 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:08 GMT /vclfcn-64/obj2?versionId=null 2024-02-01T10:49:08.942 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/BsuL7gEyNSg+CBYNX8tJkG4x2M= 2024-02-01T10:49:08.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/BsuL7gEyNSg+CBYNX8tJkG4x2M=', 'Content-Length': '0'} 2024-02-01T10:49:08.945 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004958ae4514d100c6-0065bb7724-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:08.945 DEBUG:boto:path=/vclfcn-64/obj2 2024-02-01T10:49:08.946 DEBUG:boto:auth_path=/vclfcn-64/obj2 2024-02-01T10:49:08.946 DEBUG:boto:path=/vclfcn-64/obj2?versionId=null 2024-02-01T10:49:08.946 DEBUG:boto:auth_path=/vclfcn-64/obj2?versionId=null 2024-02-01T10:49:08.946 DEBUG:boto:Method: GET 2024-02-01T10:49:08.946 DEBUG:boto:Path: /vclfcn-64/obj2?versionId=null 2024-02-01T10:49:08.946 DEBUG:boto:Data: 2024-02-01T10:49:08.946 DEBUG:boto:Headers: {} 2024-02-01T10:49:08.946 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:08.946 DEBUG:boto:Port: 8000 2024-02-01T10:49:08.946 DEBUG:boto:Params: {} 2024-02-01T10:49:08.946 DEBUG:boto:Token: None 2024-02-01T10:49:08.947 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:08 GMT /vclfcn-64/obj2?versionId=null 2024-02-01T10:49:08.947 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/BsuL7gEyNSg+CBYNX8tJkG4x2M= 2024-02-01T10:49:08.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/BsuL7gEyNSg+CBYNX8tJkG4x2M=', 'Content-Length': '0'} 2024-02-01T10:49:08.949 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ef84027f4973bf2c-0065bb7724-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:08.950 DEBUG:boto:path=/vclfcn-64/obj2 2024-02-01T10:49:08.950 DEBUG:boto:auth_path=/vclfcn-64/obj2 2024-02-01T10:49:08.950 DEBUG:boto:path=/vclfcn-64/obj2?versionId=null 2024-02-01T10:49:08.950 DEBUG:boto:auth_path=/vclfcn-64/obj2?versionId=null 2024-02-01T10:49:08.950 DEBUG:boto:Method: HEAD 2024-02-01T10:49:08.950 DEBUG:boto:Path: /vclfcn-64/obj2?versionId=null 2024-02-01T10:49:08.950 DEBUG:boto:Data: 2024-02-01T10:49:08.950 DEBUG:boto:Headers: {} 2024-02-01T10:49:08.951 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:08.951 DEBUG:boto:Port: 8001 2024-02-01T10:49:08.951 DEBUG:boto:Params: {} 2024-02-01T10:49:08.951 DEBUG:boto:Token: None 2024-02-01T10:49:08.951 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:08 GMT /vclfcn-64/obj2?versionId=null 2024-02-01T10:49:08.951 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:MhKQf5MknX1hDsOIXbC0kWYKudM= 2024-02-01T10:49:08.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:MhKQf5MknX1hDsOIXbC0kWYKudM=', 'Content-Length': '0'} 2024-02-01T10:49:08.954 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000026c9aff6eb58ed7b-0065bb7724-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:08.954 DEBUG:boto:path=/vclfcn-64/obj2 2024-02-01T10:49:08.954 DEBUG:boto:auth_path=/vclfcn-64/obj2 2024-02-01T10:49:08.954 DEBUG:boto:path=/vclfcn-64/obj2?versionId=null 2024-02-01T10:49:08.954 DEBUG:boto:auth_path=/vclfcn-64/obj2?versionId=null 2024-02-01T10:49:08.954 DEBUG:boto:Method: HEAD 2024-02-01T10:49:08.954 DEBUG:boto:Path: /vclfcn-64/obj2?versionId=null 2024-02-01T10:49:08.955 DEBUG:boto:Data: 2024-02-01T10:49:08.955 DEBUG:boto:Headers: {} 2024-02-01T10:49:08.955 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:08.955 DEBUG:boto:Port: 8000 2024-02-01T10:49:08.955 DEBUG:boto:Params: {} 2024-02-01T10:49:08.955 DEBUG:boto:Token: None 2024-02-01T10:49:08.955 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:08 GMT /vclfcn-64/obj2?versionId=null 2024-02-01T10:49:08.955 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:MhKQf5MknX1hDsOIXbC0kWYKudM= 2024-02-01T10:49:08.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:MhKQf5MknX1hDsOIXbC0kWYKudM=', 'Content-Length': '0'} 2024-02-01T10:49:08.958 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000954fda4685c99093-0065bb7724-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:08.958 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:49:08.958 DEBUG:boto:path=/vclfcn-64/obj2 2024-02-01T10:49:08.958 DEBUG:boto:auth_path=/vclfcn-64/obj2 2024-02-01T10:49:08.959 DEBUG:boto:path=/vclfcn-64/obj2?versionId=null 2024-02-01T10:49:08.959 DEBUG:boto:auth_path=/vclfcn-64/obj2?versionId=null 2024-02-01T10:49:08.959 DEBUG:boto:Method: GET 2024-02-01T10:49:08.959 DEBUG:boto:Path: /vclfcn-64/obj2?versionId=null 2024-02-01T10:49:08.959 DEBUG:boto:Data: 2024-02-01T10:49:08.959 DEBUG:boto:Headers: {} 2024-02-01T10:49:08.959 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:08.959 DEBUG:boto:Port: 8001 2024-02-01T10:49:08.959 DEBUG:boto:Params: {} 2024-02-01T10:49:08.959 DEBUG:boto:Token: None 2024-02-01T10:49:08.960 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:08 GMT /vclfcn-64/obj2?versionId=null 2024-02-01T10:49:08.960 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/BsuL7gEyNSg+CBYNX8tJkG4x2M= 2024-02-01T10:49:08.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/BsuL7gEyNSg+CBYNX8tJkG4x2M=', 'Content-Length': '0'} 2024-02-01T10:49:08.962 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007464ead74959e86d-0065bb7724-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:08.963 DEBUG:boto:path=/vclfcn-64/obj2 2024-02-01T10:49:08.963 DEBUG:boto:auth_path=/vclfcn-64/obj2 2024-02-01T10:49:08.964 DEBUG:boto:path=/vclfcn-64/obj2?versionId=null 2024-02-01T10:49:08.964 DEBUG:boto:auth_path=/vclfcn-64/obj2?versionId=null 2024-02-01T10:49:08.964 DEBUG:boto:Method: GET 2024-02-01T10:49:08.964 DEBUG:boto:Path: /vclfcn-64/obj2?versionId=null 2024-02-01T10:49:08.964 DEBUG:boto:Data: 2024-02-01T10:49:08.964 DEBUG:boto:Headers: {} 2024-02-01T10:49:08.964 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:08.964 DEBUG:boto:Port: 8000 2024-02-01T10:49:08.965 DEBUG:boto:Params: {} 2024-02-01T10:49:08.965 DEBUG:boto:Token: None 2024-02-01T10:49:08.965 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:08 GMT /vclfcn-64/obj2?versionId=null 2024-02-01T10:49:08.965 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/BsuL7gEyNSg+CBYNX8tJkG4x2M= 2024-02-01T10:49:08.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/BsuL7gEyNSg+CBYNX8tJkG4x2M=', 'Content-Length': '0'} 2024-02-01T10:49:08.968 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008d64fea60d82ea8f-0065bb7724-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:09.008 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:49:09.008 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:49:09.008 DEBUG:boto:path=/vclfcn-64/?prefix=obj2 2024-02-01T10:49:09.008 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj2 2024-02-01T10:49:09.009 DEBUG:boto:Method: GET 2024-02-01T10:49:09.009 DEBUG:boto:Path: /vclfcn-64/?prefix=obj2 2024-02-01T10:49:09.009 DEBUG:boto:Data: 2024-02-01T10:49:09.009 DEBUG:boto:Headers: {} 2024-02-01T10:49:09.009 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:09.009 DEBUG:boto:Port: 8001 2024-02-01T10:49:09.009 DEBUG:boto:Params: {} 2024-02-01T10:49:09.009 DEBUG:boto:Token: None 2024-02-01T10:49:09.009 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:09 GMT /vclfcn-64/ 2024-02-01T10:49:09.010 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tr7c5TgwY1ZfwySxIrBV0bT5tDA= 2024-02-01T10:49:09.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tr7c5TgwY1ZfwySxIrBV0bT5tDA=', 'Content-Length': '0'} 2024-02-01T10:49:09.014 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d7af0fb1eda60d2d-0065bb7725-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:09.014 DEBUG:boto:b'vclfcn-64obj21000falseobj22024-02-01T10:48:26.615Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:09.015 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:49:09.015 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:49:09.015 DEBUG:boto:path=/vclfcn-64/?prefix=obj2 2024-02-01T10:49:09.015 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj2 2024-02-01T10:49:09.016 DEBUG:boto:Method: GET 2024-02-01T10:49:09.016 DEBUG:boto:Path: /vclfcn-64/?prefix=obj2 2024-02-01T10:49:09.016 DEBUG:boto:Data: 2024-02-01T10:49:09.016 DEBUG:boto:Headers: {} 2024-02-01T10:49:09.016 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:09.016 DEBUG:boto:Port: 8000 2024-02-01T10:49:09.016 DEBUG:boto:Params: {} 2024-02-01T10:49:09.017 DEBUG:boto:Token: None 2024-02-01T10:49:09.017 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:09 GMT /vclfcn-64/ 2024-02-01T10:49:09.018 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tr7c5TgwY1ZfwySxIrBV0bT5tDA= 2024-02-01T10:49:09.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tr7c5TgwY1ZfwySxIrBV0bT5tDA=', 'Content-Length': '0'} 2024-02-01T10:49:09.022 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f81bc6db6ecd9e94-0065bb7725-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:09.022 DEBUG:boto:b'vclfcn-64obj21000falseobj22024-02-01T10:48:26.615Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:09.022 DEBUG:boto:path=/vclfcn-64/obj2 2024-02-01T10:49:09.022 DEBUG:boto:auth_path=/vclfcn-64/obj2 2024-02-01T10:49:09.023 DEBUG:boto:Method: HEAD 2024-02-01T10:49:09.023 DEBUG:boto:Path: /vclfcn-64/obj2 2024-02-01T10:49:09.023 DEBUG:boto:Data: 2024-02-01T10:49:09.023 DEBUG:boto:Headers: {} 2024-02-01T10:49:09.023 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:09.023 DEBUG:boto:Port: 8001 2024-02-01T10:49:09.023 DEBUG:boto:Params: {} 2024-02-01T10:49:09.023 DEBUG:boto:Token: None 2024-02-01T10:49:09.023 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:09 GMT /vclfcn-64/obj2 2024-02-01T10:49:09.023 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GStft1wkhSBFx8QXrsIE1b2B2z4= 2024-02-01T10:49:09.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GStft1wkhSBFx8QXrsIE1b2B2z4=', 'Content-Length': '0'} 2024-02-01T10:49:09.026 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000000a4724b5605acfc-0065bb7725-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:09.026 DEBUG:boto:path=/vclfcn-64/obj2 2024-02-01T10:49:09.026 DEBUG:boto:auth_path=/vclfcn-64/obj2 2024-02-01T10:49:09.027 DEBUG:boto:Method: HEAD 2024-02-01T10:49:09.027 DEBUG:boto:Path: /vclfcn-64/obj2 2024-02-01T10:49:09.027 DEBUG:boto:Data: 2024-02-01T10:49:09.027 DEBUG:boto:Headers: {} 2024-02-01T10:49:09.027 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:09.027 DEBUG:boto:Port: 8000 2024-02-01T10:49:09.027 DEBUG:boto:Params: {} 2024-02-01T10:49:09.027 DEBUG:boto:Token: None 2024-02-01T10:49:09.027 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:09 GMT /vclfcn-64/obj2 2024-02-01T10:49:09.027 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GStft1wkhSBFx8QXrsIE1b2B2z4= 2024-02-01T10:49:09.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GStft1wkhSBFx8QXrsIE1b2B2z4=', 'Content-Length': '0'} 2024-02-01T10:49:09.030 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004d2595a0886fb79f-0065bb7725-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:09.030 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:49:09.031 DEBUG:boto:path=/vclfcn-64/obj2 2024-02-01T10:49:09.031 DEBUG:boto:auth_path=/vclfcn-64/obj2 2024-02-01T10:49:09.031 DEBUG:boto:Method: GET 2024-02-01T10:49:09.031 DEBUG:boto:Path: /vclfcn-64/obj2 2024-02-01T10:49:09.031 DEBUG:boto:Data: 2024-02-01T10:49:09.031 DEBUG:boto:Headers: {} 2024-02-01T10:49:09.031 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:09.032 DEBUG:boto:Port: 8001 2024-02-01T10:49:09.032 DEBUG:boto:Params: {} 2024-02-01T10:49:09.032 DEBUG:boto:Token: None 2024-02-01T10:49:09.032 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:09 GMT /vclfcn-64/obj2 2024-02-01T10:49:09.032 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:xghkqYXuXyeLnZvlMmVGr0v+WjM= 2024-02-01T10:49:09.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:xghkqYXuXyeLnZvlMmVGr0v+WjM=', 'Content-Length': '0'} 2024-02-01T10:49:09.035 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ef790b22a9b3a73e-0065bb7725-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:09.076 DEBUG:boto:path=/vclfcn-64/obj2 2024-02-01T10:49:09.076 DEBUG:boto:auth_path=/vclfcn-64/obj2 2024-02-01T10:49:09.076 DEBUG:boto:Method: GET 2024-02-01T10:49:09.076 DEBUG:boto:Path: /vclfcn-64/obj2 2024-02-01T10:49:09.077 DEBUG:boto:Data: 2024-02-01T10:49:09.077 DEBUG:boto:Headers: {} 2024-02-01T10:49:09.077 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:09.077 DEBUG:boto:Port: 8000 2024-02-01T10:49:09.077 DEBUG:boto:Params: {} 2024-02-01T10:49:09.077 DEBUG:boto:Token: None 2024-02-01T10:49:09.077 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:09 GMT /vclfcn-64/obj2 2024-02-01T10:49:09.077 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:xghkqYXuXyeLnZvlMmVGr0v+WjM= 2024-02-01T10:49:09.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:xghkqYXuXyeLnZvlMmVGr0v+WjM=', 'Content-Length': '0'} 2024-02-01T10:49:09.083 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b4348adc7a41bf45-0065bb7725-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:09.124 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:49:09.124 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:49:09.124 DEBUG:boto:path=/vclfcn-64/?prefix=obj2 2024-02-01T10:49:09.125 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj2 2024-02-01T10:49:09.125 DEBUG:boto:Method: GET 2024-02-01T10:49:09.125 DEBUG:boto:Path: /vclfcn-64/?prefix=obj2 2024-02-01T10:49:09.125 DEBUG:boto:Data: 2024-02-01T10:49:09.125 DEBUG:boto:Headers: {} 2024-02-01T10:49:09.125 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:09.125 DEBUG:boto:Port: 8001 2024-02-01T10:49:09.125 DEBUG:boto:Params: {} 2024-02-01T10:49:09.125 DEBUG:boto:Token: None 2024-02-01T10:49:09.125 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:09 GMT /vclfcn-64/ 2024-02-01T10:49:09.126 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tr7c5TgwY1ZfwySxIrBV0bT5tDA= 2024-02-01T10:49:09.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tr7c5TgwY1ZfwySxIrBV0bT5tDA=', 'Content-Length': '0'} 2024-02-01T10:49:09.130 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008234d76d2d855d9b-0065bb7725-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:09.130 DEBUG:boto:b'vclfcn-64obj21000falseobj22024-02-01T10:48:26.615Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:09.131 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:49:09.131 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:49:09.131 DEBUG:boto:path=/vclfcn-64/?prefix=obj2 2024-02-01T10:49:09.131 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj2 2024-02-01T10:49:09.132 DEBUG:boto:Method: GET 2024-02-01T10:49:09.132 DEBUG:boto:Path: /vclfcn-64/?prefix=obj2 2024-02-01T10:49:09.132 DEBUG:boto:Data: 2024-02-01T10:49:09.132 DEBUG:boto:Headers: {} 2024-02-01T10:49:09.132 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:09.132 DEBUG:boto:Port: 8000 2024-02-01T10:49:09.132 DEBUG:boto:Params: {} 2024-02-01T10:49:09.132 DEBUG:boto:Token: None 2024-02-01T10:49:09.132 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:09 GMT /vclfcn-64/ 2024-02-01T10:49:09.133 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tr7c5TgwY1ZfwySxIrBV0bT5tDA= 2024-02-01T10:49:09.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tr7c5TgwY1ZfwySxIrBV0bT5tDA=', 'Content-Length': '0'} 2024-02-01T10:49:09.137 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e27d0793f3086596-0065bb7725-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:09.180 DEBUG:boto:b'vclfcn-64obj21000falseobj22024-02-01T10:48:26.615Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:09.180 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:49:09.181 DEBUG:boto:path=/vclfcn-64/obj3 2024-02-01T10:49:09.181 DEBUG:boto:auth_path=/vclfcn-64/obj3 2024-02-01T10:49:09.181 DEBUG:boto:path=/vclfcn-64/obj3?versionId=null 2024-02-01T10:49:09.181 DEBUG:boto:auth_path=/vclfcn-64/obj3?versionId=null 2024-02-01T10:49:09.181 DEBUG:boto:Method: GET 2024-02-01T10:49:09.181 DEBUG:boto:Path: /vclfcn-64/obj3?versionId=null 2024-02-01T10:49:09.181 DEBUG:boto:Data: 2024-02-01T10:49:09.181 DEBUG:boto:Headers: {} 2024-02-01T10:49:09.181 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:09.181 DEBUG:boto:Port: 8001 2024-02-01T10:49:09.181 DEBUG:boto:Params: {} 2024-02-01T10:49:09.182 DEBUG:boto:Token: None 2024-02-01T10:49:09.182 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:09 GMT /vclfcn-64/obj3?versionId=null 2024-02-01T10:49:09.182 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:qYmpWYaIXCfeXryZByuPPV+qtN4= 2024-02-01T10:49:09.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:qYmpWYaIXCfeXryZByuPPV+qtN4=', 'Content-Length': '0'} 2024-02-01T10:49:09.186 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bbaaa8bb13af8663-0065bb7725-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:09.187 DEBUG:boto:path=/vclfcn-64/obj3 2024-02-01T10:49:09.187 DEBUG:boto:auth_path=/vclfcn-64/obj3 2024-02-01T10:49:09.187 DEBUG:boto:path=/vclfcn-64/obj3?versionId=null 2024-02-01T10:49:09.187 DEBUG:boto:auth_path=/vclfcn-64/obj3?versionId=null 2024-02-01T10:49:09.187 DEBUG:boto:Method: GET 2024-02-01T10:49:09.187 DEBUG:boto:Path: /vclfcn-64/obj3?versionId=null 2024-02-01T10:49:09.187 DEBUG:boto:Data: 2024-02-01T10:49:09.187 DEBUG:boto:Headers: {} 2024-02-01T10:49:09.187 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:09.187 DEBUG:boto:Port: 8000 2024-02-01T10:49:09.187 DEBUG:boto:Params: {} 2024-02-01T10:49:09.188 DEBUG:boto:Token: None 2024-02-01T10:49:09.188 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:09 GMT /vclfcn-64/obj3?versionId=null 2024-02-01T10:49:09.188 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:qYmpWYaIXCfeXryZByuPPV+qtN4= 2024-02-01T10:49:09.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:qYmpWYaIXCfeXryZByuPPV+qtN4=', 'Content-Length': '0'} 2024-02-01T10:49:09.191 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d01b3537141becd9-0065bb7725-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:09.191 DEBUG:boto:path=/vclfcn-64/obj3 2024-02-01T10:49:09.191 DEBUG:boto:auth_path=/vclfcn-64/obj3 2024-02-01T10:49:09.192 DEBUG:boto:path=/vclfcn-64/obj3?versionId=null 2024-02-01T10:49:09.192 DEBUG:boto:auth_path=/vclfcn-64/obj3?versionId=null 2024-02-01T10:49:09.192 DEBUG:boto:Method: HEAD 2024-02-01T10:49:09.192 DEBUG:boto:Path: /vclfcn-64/obj3?versionId=null 2024-02-01T10:49:09.192 DEBUG:boto:Data: 2024-02-01T10:49:09.192 DEBUG:boto:Headers: {} 2024-02-01T10:49:09.192 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:09.192 DEBUG:boto:Port: 8001 2024-02-01T10:49:09.192 DEBUG:boto:Params: {} 2024-02-01T10:49:09.193 DEBUG:boto:Token: None 2024-02-01T10:49:09.193 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:09 GMT /vclfcn-64/obj3?versionId=null 2024-02-01T10:49:09.193 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:y+lyGU67D38zbvm+GxXB/FRXlf8= 2024-02-01T10:49:09.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:y+lyGU67D38zbvm+GxXB/FRXlf8=', 'Content-Length': '0'} 2024-02-01T10:49:09.196 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ea92379b20bc268c-0065bb7725-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:09.196 DEBUG:boto:path=/vclfcn-64/obj3 2024-02-01T10:49:09.196 DEBUG:boto:auth_path=/vclfcn-64/obj3 2024-02-01T10:49:09.196 DEBUG:boto:path=/vclfcn-64/obj3?versionId=null 2024-02-01T10:49:09.196 DEBUG:boto:auth_path=/vclfcn-64/obj3?versionId=null 2024-02-01T10:49:09.196 DEBUG:boto:Method: HEAD 2024-02-01T10:49:09.196 DEBUG:boto:Path: /vclfcn-64/obj3?versionId=null 2024-02-01T10:49:09.196 DEBUG:boto:Data: 2024-02-01T10:49:09.196 DEBUG:boto:Headers: {} 2024-02-01T10:49:09.197 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:09.197 DEBUG:boto:Port: 8000 2024-02-01T10:49:09.197 DEBUG:boto:Params: {} 2024-02-01T10:49:09.197 DEBUG:boto:Token: None 2024-02-01T10:49:09.197 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:09 GMT /vclfcn-64/obj3?versionId=null 2024-02-01T10:49:09.197 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:y+lyGU67D38zbvm+GxXB/FRXlf8= 2024-02-01T10:49:09.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:y+lyGU67D38zbvm+GxXB/FRXlf8=', 'Content-Length': '0'} 2024-02-01T10:49:09.200 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006949102dd267a9e3-0065bb7725-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:09.200 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:49:09.200 DEBUG:boto:path=/vclfcn-64/obj3 2024-02-01T10:49:09.200 DEBUG:boto:auth_path=/vclfcn-64/obj3 2024-02-01T10:49:09.200 DEBUG:boto:path=/vclfcn-64/obj3?versionId=null 2024-02-01T10:49:09.200 DEBUG:boto:auth_path=/vclfcn-64/obj3?versionId=null 2024-02-01T10:49:09.201 DEBUG:boto:Method: GET 2024-02-01T10:49:09.201 DEBUG:boto:Path: /vclfcn-64/obj3?versionId=null 2024-02-01T10:49:09.201 DEBUG:boto:Data: 2024-02-01T10:49:09.201 DEBUG:boto:Headers: {} 2024-02-01T10:49:09.201 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:09.201 DEBUG:boto:Port: 8001 2024-02-01T10:49:09.201 DEBUG:boto:Params: {} 2024-02-01T10:49:09.201 DEBUG:boto:Token: None 2024-02-01T10:49:09.201 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:09 GMT /vclfcn-64/obj3?versionId=null 2024-02-01T10:49:09.201 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:qYmpWYaIXCfeXryZByuPPV+qtN4= 2024-02-01T10:49:09.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:qYmpWYaIXCfeXryZByuPPV+qtN4=', 'Content-Length': '0'} 2024-02-01T10:49:09.204 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002e45b0448958b40f-0065bb7725-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:09.204 DEBUG:boto:path=/vclfcn-64/obj3 2024-02-01T10:49:09.204 DEBUG:boto:auth_path=/vclfcn-64/obj3 2024-02-01T10:49:09.204 DEBUG:boto:path=/vclfcn-64/obj3?versionId=null 2024-02-01T10:49:09.205 DEBUG:boto:auth_path=/vclfcn-64/obj3?versionId=null 2024-02-01T10:49:09.205 DEBUG:boto:Method: GET 2024-02-01T10:49:09.205 DEBUG:boto:Path: /vclfcn-64/obj3?versionId=null 2024-02-01T10:49:09.205 DEBUG:boto:Data: 2024-02-01T10:49:09.205 DEBUG:boto:Headers: {} 2024-02-01T10:49:09.205 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:09.205 DEBUG:boto:Port: 8000 2024-02-01T10:49:09.205 DEBUG:boto:Params: {} 2024-02-01T10:49:09.205 DEBUG:boto:Token: None 2024-02-01T10:49:09.205 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:09 GMT /vclfcn-64/obj3?versionId=null 2024-02-01T10:49:09.206 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:qYmpWYaIXCfeXryZByuPPV+qtN4= 2024-02-01T10:49:09.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:qYmpWYaIXCfeXryZByuPPV+qtN4=', 'Content-Length': '0'} 2024-02-01T10:49:09.209 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006452e21c69ab5f3d-0065bb7725-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:09.252 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:49:09.252 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:49:09.252 DEBUG:boto:path=/vclfcn-64/?prefix=obj3 2024-02-01T10:49:09.252 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj3 2024-02-01T10:49:09.253 DEBUG:boto:Method: GET 2024-02-01T10:49:09.253 DEBUG:boto:Path: /vclfcn-64/?prefix=obj3 2024-02-01T10:49:09.253 DEBUG:boto:Data: 2024-02-01T10:49:09.253 DEBUG:boto:Headers: {} 2024-02-01T10:49:09.253 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:09.253 DEBUG:boto:Port: 8001 2024-02-01T10:49:09.253 DEBUG:boto:Params: {} 2024-02-01T10:49:09.253 DEBUG:boto:Token: None 2024-02-01T10:49:09.253 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:09 GMT /vclfcn-64/ 2024-02-01T10:49:09.253 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tr7c5TgwY1ZfwySxIrBV0bT5tDA= 2024-02-01T10:49:09.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tr7c5TgwY1ZfwySxIrBV0bT5tDA=', 'Content-Length': '0'} 2024-02-01T10:49:09.258 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007241dea5bbfb0b6d-0065bb7725-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:09.258 DEBUG:boto:b'vclfcn-64obj31000falseobj32024-02-01T10:48:26.634Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:09.259 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:49:09.259 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:49:09.259 DEBUG:boto:path=/vclfcn-64/?prefix=obj3 2024-02-01T10:49:09.259 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj3 2024-02-01T10:49:09.259 DEBUG:boto:Method: GET 2024-02-01T10:49:09.259 DEBUG:boto:Path: /vclfcn-64/?prefix=obj3 2024-02-01T10:49:09.259 DEBUG:boto:Data: 2024-02-01T10:49:09.259 DEBUG:boto:Headers: {} 2024-02-01T10:49:09.259 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:09.259 DEBUG:boto:Port: 8000 2024-02-01T10:49:09.259 DEBUG:boto:Params: {} 2024-02-01T10:49:09.260 DEBUG:boto:Token: None 2024-02-01T10:49:09.260 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:09 GMT /vclfcn-64/ 2024-02-01T10:49:09.260 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tr7c5TgwY1ZfwySxIrBV0bT5tDA= 2024-02-01T10:49:09.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tr7c5TgwY1ZfwySxIrBV0bT5tDA=', 'Content-Length': '0'} 2024-02-01T10:49:09.264 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001fdd2d0163f72944-0065bb7725-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:09.264 DEBUG:boto:b'vclfcn-64obj31000falseobj32024-02-01T10:48:26.634Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:09.265 DEBUG:boto:path=/vclfcn-64/obj3 2024-02-01T10:49:09.265 DEBUG:boto:auth_path=/vclfcn-64/obj3 2024-02-01T10:49:09.265 DEBUG:boto:Method: HEAD 2024-02-01T10:49:09.265 DEBUG:boto:Path: /vclfcn-64/obj3 2024-02-01T10:49:09.265 DEBUG:boto:Data: 2024-02-01T10:49:09.265 DEBUG:boto:Headers: {} 2024-02-01T10:49:09.265 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:09.265 DEBUG:boto:Port: 8001 2024-02-01T10:49:09.265 DEBUG:boto:Params: {} 2024-02-01T10:49:09.265 DEBUG:boto:Token: None 2024-02-01T10:49:09.266 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:09 GMT /vclfcn-64/obj3 2024-02-01T10:49:09.266 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:AzvtNlAl9BOT+JHCcTzTQt6Cr8I= 2024-02-01T10:49:09.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:AzvtNlAl9BOT+JHCcTzTQt6Cr8I=', 'Content-Length': '0'} 2024-02-01T10:49:09.269 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004c55f80f0ecab63b-0065bb7725-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:09.269 DEBUG:boto:path=/vclfcn-64/obj3 2024-02-01T10:49:09.269 DEBUG:boto:auth_path=/vclfcn-64/obj3 2024-02-01T10:49:09.269 DEBUG:boto:Method: HEAD 2024-02-01T10:49:09.269 DEBUG:boto:Path: /vclfcn-64/obj3 2024-02-01T10:49:09.269 DEBUG:boto:Data: 2024-02-01T10:49:09.269 DEBUG:boto:Headers: {} 2024-02-01T10:49:09.269 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:09.270 DEBUG:boto:Port: 8000 2024-02-01T10:49:09.270 DEBUG:boto:Params: {} 2024-02-01T10:49:09.270 DEBUG:boto:Token: None 2024-02-01T10:49:09.270 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:09 GMT /vclfcn-64/obj3 2024-02-01T10:49:09.270 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:AzvtNlAl9BOT+JHCcTzTQt6Cr8I= 2024-02-01T10:49:09.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:AzvtNlAl9BOT+JHCcTzTQt6Cr8I=', 'Content-Length': '0'} 2024-02-01T10:49:09.273 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000343b1e31598ac7ae-0065bb7725-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:09.273 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:49:09.273 DEBUG:boto:path=/vclfcn-64/obj3 2024-02-01T10:49:09.273 DEBUG:boto:auth_path=/vclfcn-64/obj3 2024-02-01T10:49:09.273 DEBUG:boto:Method: GET 2024-02-01T10:49:09.273 DEBUG:boto:Path: /vclfcn-64/obj3 2024-02-01T10:49:09.274 DEBUG:boto:Data: 2024-02-01T10:49:09.274 DEBUG:boto:Headers: {} 2024-02-01T10:49:09.274 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:09.274 DEBUG:boto:Port: 8001 2024-02-01T10:49:09.274 DEBUG:boto:Params: {} 2024-02-01T10:49:09.274 DEBUG:boto:Token: None 2024-02-01T10:49:09.274 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:09 GMT /vclfcn-64/obj3 2024-02-01T10:49:09.274 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:aKWEjAJtdmmnyn1GmrgEPZ6xSWM= 2024-02-01T10:49:09.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:aKWEjAJtdmmnyn1GmrgEPZ6xSWM=', 'Content-Length': '0'} 2024-02-01T10:49:09.277 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008c3b6b55f11c643e-0065bb7725-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:09.324 DEBUG:boto:path=/vclfcn-64/obj3 2024-02-01T10:49:09.324 DEBUG:boto:auth_path=/vclfcn-64/obj3 2024-02-01T10:49:09.324 DEBUG:boto:Method: GET 2024-02-01T10:49:09.325 DEBUG:boto:Path: /vclfcn-64/obj3 2024-02-01T10:49:09.325 DEBUG:boto:Data: 2024-02-01T10:49:09.325 DEBUG:boto:Headers: {} 2024-02-01T10:49:09.325 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:09.325 DEBUG:boto:Port: 8000 2024-02-01T10:49:09.325 DEBUG:boto:Params: {} 2024-02-01T10:49:09.325 DEBUG:boto:Token: None 2024-02-01T10:49:09.326 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:09 GMT /vclfcn-64/obj3 2024-02-01T10:49:09.326 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:aKWEjAJtdmmnyn1GmrgEPZ6xSWM= 2024-02-01T10:49:09.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:aKWEjAJtdmmnyn1GmrgEPZ6xSWM=', 'Content-Length': '0'} 2024-02-01T10:49:09.329 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001ab8aeadbac5e55b-0065bb7725-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:09.376 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:49:09.376 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:49:09.376 DEBUG:boto:path=/vclfcn-64/?prefix=obj3 2024-02-01T10:49:09.377 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj3 2024-02-01T10:49:09.377 DEBUG:boto:Method: GET 2024-02-01T10:49:09.377 DEBUG:boto:Path: /vclfcn-64/?prefix=obj3 2024-02-01T10:49:09.377 DEBUG:boto:Data: 2024-02-01T10:49:09.377 DEBUG:boto:Headers: {} 2024-02-01T10:49:09.377 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:09.377 DEBUG:boto:Port: 8001 2024-02-01T10:49:09.377 DEBUG:boto:Params: {} 2024-02-01T10:49:09.377 DEBUG:boto:Token: None 2024-02-01T10:49:09.378 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:09 GMT /vclfcn-64/ 2024-02-01T10:49:09.378 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tr7c5TgwY1ZfwySxIrBV0bT5tDA= 2024-02-01T10:49:09.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tr7c5TgwY1ZfwySxIrBV0bT5tDA=', 'Content-Length': '0'} 2024-02-01T10:49:09.382 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005411a53dd4a9dac0-0065bb7725-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:09.382 DEBUG:boto:b'vclfcn-64obj31000falseobj32024-02-01T10:48:26.634Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:09.382 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:49:09.383 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:49:09.383 DEBUG:boto:path=/vclfcn-64/?prefix=obj3 2024-02-01T10:49:09.383 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj3 2024-02-01T10:49:09.383 DEBUG:boto:Method: GET 2024-02-01T10:49:09.383 DEBUG:boto:Path: /vclfcn-64/?prefix=obj3 2024-02-01T10:49:09.383 DEBUG:boto:Data: 2024-02-01T10:49:09.383 DEBUG:boto:Headers: {} 2024-02-01T10:49:09.383 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:09.383 DEBUG:boto:Port: 8000 2024-02-01T10:49:09.383 DEBUG:boto:Params: {} 2024-02-01T10:49:09.383 DEBUG:boto:Token: None 2024-02-01T10:49:09.384 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:09 GMT /vclfcn-64/ 2024-02-01T10:49:09.384 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tr7c5TgwY1ZfwySxIrBV0bT5tDA= 2024-02-01T10:49:09.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tr7c5TgwY1ZfwySxIrBV0bT5tDA=', 'Content-Length': '0'} 2024-02-01T10:49:09.388 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2d744149a13942e-0065bb7725-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:09.388 DEBUG:boto:b'vclfcn-64obj31000falseobj32024-02-01T10:48:26.634Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:09.388 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:49:09.388 DEBUG:boto:path=/vclfcn-64/obj4 2024-02-01T10:49:09.389 DEBUG:boto:auth_path=/vclfcn-64/obj4 2024-02-01T10:49:09.389 DEBUG:boto:path=/vclfcn-64/obj4?versionId=null 2024-02-01T10:49:09.389 DEBUG:boto:auth_path=/vclfcn-64/obj4?versionId=null 2024-02-01T10:49:09.389 DEBUG:boto:Method: GET 2024-02-01T10:49:09.389 DEBUG:boto:Path: /vclfcn-64/obj4?versionId=null 2024-02-01T10:49:09.389 DEBUG:boto:Data: 2024-02-01T10:49:09.389 DEBUG:boto:Headers: {} 2024-02-01T10:49:09.389 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:09.389 DEBUG:boto:Port: 8001 2024-02-01T10:49:09.389 DEBUG:boto:Params: {} 2024-02-01T10:49:09.389 DEBUG:boto:Token: None 2024-02-01T10:49:09.390 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:09 GMT /vclfcn-64/obj4?versionId=null 2024-02-01T10:49:09.390 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:whi87ZBs2N88oEGytI/gE0krg20= 2024-02-01T10:49:09.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:whi87ZBs2N88oEGytI/gE0krg20=', 'Content-Length': '0'} 2024-02-01T10:49:09.393 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a0953530455ba1c7-0065bb7725-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:09.393 DEBUG:boto:path=/vclfcn-64/obj4 2024-02-01T10:49:09.393 DEBUG:boto:auth_path=/vclfcn-64/obj4 2024-02-01T10:49:09.393 DEBUG:boto:path=/vclfcn-64/obj4?versionId=null 2024-02-01T10:49:09.393 DEBUG:boto:auth_path=/vclfcn-64/obj4?versionId=null 2024-02-01T10:49:09.393 DEBUG:boto:Method: GET 2024-02-01T10:49:09.394 DEBUG:boto:Path: /vclfcn-64/obj4?versionId=null 2024-02-01T10:49:09.394 DEBUG:boto:Data: 2024-02-01T10:49:09.394 DEBUG:boto:Headers: {} 2024-02-01T10:49:09.394 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:09.394 DEBUG:boto:Port: 8000 2024-02-01T10:49:09.394 DEBUG:boto:Params: {} 2024-02-01T10:49:09.394 DEBUG:boto:Token: None 2024-02-01T10:49:09.394 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:09 GMT /vclfcn-64/obj4?versionId=null 2024-02-01T10:49:09.394 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:whi87ZBs2N88oEGytI/gE0krg20= 2024-02-01T10:49:09.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:whi87ZBs2N88oEGytI/gE0krg20=', 'Content-Length': '0'} 2024-02-01T10:49:09.397 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000020cd63af15620287-0065bb7725-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:09.398 DEBUG:boto:path=/vclfcn-64/obj4 2024-02-01T10:49:09.398 DEBUG:boto:auth_path=/vclfcn-64/obj4 2024-02-01T10:49:09.398 DEBUG:boto:path=/vclfcn-64/obj4?versionId=null 2024-02-01T10:49:09.398 DEBUG:boto:auth_path=/vclfcn-64/obj4?versionId=null 2024-02-01T10:49:09.398 DEBUG:boto:Method: HEAD 2024-02-01T10:49:09.398 DEBUG:boto:Path: /vclfcn-64/obj4?versionId=null 2024-02-01T10:49:09.398 DEBUG:boto:Data: 2024-02-01T10:49:09.398 DEBUG:boto:Headers: {} 2024-02-01T10:49:09.399 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:09.399 DEBUG:boto:Port: 8001 2024-02-01T10:49:09.399 DEBUG:boto:Params: {} 2024-02-01T10:49:09.399 DEBUG:boto:Token: None 2024-02-01T10:49:09.399 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:09 GMT /vclfcn-64/obj4?versionId=null 2024-02-01T10:49:09.399 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:eP3ia86Wftujmrhz5jCfIXIaC1U= 2024-02-01T10:49:09.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:eP3ia86Wftujmrhz5jCfIXIaC1U=', 'Content-Length': '0'} 2024-02-01T10:49:09.402 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000041b9e8e7a1ed136e-0065bb7725-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:09.402 DEBUG:boto:path=/vclfcn-64/obj4 2024-02-01T10:49:09.402 DEBUG:boto:auth_path=/vclfcn-64/obj4 2024-02-01T10:49:09.402 DEBUG:boto:path=/vclfcn-64/obj4?versionId=null 2024-02-01T10:49:09.402 DEBUG:boto:auth_path=/vclfcn-64/obj4?versionId=null 2024-02-01T10:49:09.402 DEBUG:boto:Method: HEAD 2024-02-01T10:49:09.402 DEBUG:boto:Path: /vclfcn-64/obj4?versionId=null 2024-02-01T10:49:09.402 DEBUG:boto:Data: 2024-02-01T10:49:09.403 DEBUG:boto:Headers: {} 2024-02-01T10:49:09.403 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:09.403 DEBUG:boto:Port: 8000 2024-02-01T10:49:09.403 DEBUG:boto:Params: {} 2024-02-01T10:49:09.403 DEBUG:boto:Token: None 2024-02-01T10:49:09.403 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:09 GMT /vclfcn-64/obj4?versionId=null 2024-02-01T10:49:09.403 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:eP3ia86Wftujmrhz5jCfIXIaC1U= 2024-02-01T10:49:09.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:eP3ia86Wftujmrhz5jCfIXIaC1U=', 'Content-Length': '0'} 2024-02-01T10:49:09.406 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001196ddaf490d36ec-0065bb7725-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:09.406 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:49:09.406 DEBUG:boto:path=/vclfcn-64/obj4 2024-02-01T10:49:09.406 DEBUG:boto:auth_path=/vclfcn-64/obj4 2024-02-01T10:49:09.406 DEBUG:boto:path=/vclfcn-64/obj4?versionId=null 2024-02-01T10:49:09.406 DEBUG:boto:auth_path=/vclfcn-64/obj4?versionId=null 2024-02-01T10:49:09.406 DEBUG:boto:Method: GET 2024-02-01T10:49:09.407 DEBUG:boto:Path: /vclfcn-64/obj4?versionId=null 2024-02-01T10:49:09.407 DEBUG:boto:Data: 2024-02-01T10:49:09.407 DEBUG:boto:Headers: {} 2024-02-01T10:49:09.407 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:09.407 DEBUG:boto:Port: 8001 2024-02-01T10:49:09.407 DEBUG:boto:Params: {} 2024-02-01T10:49:09.407 DEBUG:boto:Token: None 2024-02-01T10:49:09.407 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:09 GMT /vclfcn-64/obj4?versionId=null 2024-02-01T10:49:09.407 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:whi87ZBs2N88oEGytI/gE0krg20= 2024-02-01T10:49:09.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:whi87ZBs2N88oEGytI/gE0krg20=', 'Content-Length': '0'} 2024-02-01T10:49:09.410 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a30fe7a2c623d0eb-0065bb7725-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:09.452 DEBUG:boto:path=/vclfcn-64/obj4 2024-02-01T10:49:09.452 DEBUG:boto:auth_path=/vclfcn-64/obj4 2024-02-01T10:49:09.452 DEBUG:boto:path=/vclfcn-64/obj4?versionId=null 2024-02-01T10:49:09.452 DEBUG:boto:auth_path=/vclfcn-64/obj4?versionId=null 2024-02-01T10:49:09.453 DEBUG:boto:Method: GET 2024-02-01T10:49:09.453 DEBUG:boto:Path: /vclfcn-64/obj4?versionId=null 2024-02-01T10:49:09.453 DEBUG:boto:Data: 2024-02-01T10:49:09.453 DEBUG:boto:Headers: {} 2024-02-01T10:49:09.453 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:09.453 DEBUG:boto:Port: 8000 2024-02-01T10:49:09.453 DEBUG:boto:Params: {} 2024-02-01T10:49:09.453 DEBUG:boto:Token: None 2024-02-01T10:49:09.453 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:09 GMT /vclfcn-64/obj4?versionId=null 2024-02-01T10:49:09.453 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:whi87ZBs2N88oEGytI/gE0krg20= 2024-02-01T10:49:09.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:whi87ZBs2N88oEGytI/gE0krg20=', 'Content-Length': '0'} 2024-02-01T10:49:09.457 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000267b50cd39feed06-0065bb7725-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:09.500 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:49:09.500 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:49:09.500 DEBUG:boto:path=/vclfcn-64/?prefix=obj4 2024-02-01T10:49:09.501 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj4 2024-02-01T10:49:09.501 DEBUG:boto:Method: GET 2024-02-01T10:49:09.501 DEBUG:boto:Path: /vclfcn-64/?prefix=obj4 2024-02-01T10:49:09.501 DEBUG:boto:Data: 2024-02-01T10:49:09.501 DEBUG:boto:Headers: {} 2024-02-01T10:49:09.501 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:09.501 DEBUG:boto:Port: 8001 2024-02-01T10:49:09.501 DEBUG:boto:Params: {} 2024-02-01T10:49:09.501 DEBUG:boto:Token: None 2024-02-01T10:49:09.501 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:09 GMT /vclfcn-64/ 2024-02-01T10:49:09.502 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tr7c5TgwY1ZfwySxIrBV0bT5tDA= 2024-02-01T10:49:09.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tr7c5TgwY1ZfwySxIrBV0bT5tDA=', 'Content-Length': '0'} 2024-02-01T10:49:09.506 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd910a6b04459a28-0065bb7725-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:09.506 DEBUG:boto:b'vclfcn-64obj41000falseobj42024-02-01T10:48:26.689Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:09.507 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:49:09.507 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:49:09.507 DEBUG:boto:path=/vclfcn-64/?prefix=obj4 2024-02-01T10:49:09.507 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj4 2024-02-01T10:49:09.507 DEBUG:boto:Method: GET 2024-02-01T10:49:09.507 DEBUG:boto:Path: /vclfcn-64/?prefix=obj4 2024-02-01T10:49:09.507 DEBUG:boto:Data: 2024-02-01T10:49:09.507 DEBUG:boto:Headers: {} 2024-02-01T10:49:09.507 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:09.507 DEBUG:boto:Port: 8000 2024-02-01T10:49:09.508 DEBUG:boto:Params: {} 2024-02-01T10:49:09.508 DEBUG:boto:Token: None 2024-02-01T10:49:09.508 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:09 GMT /vclfcn-64/ 2024-02-01T10:49:09.508 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tr7c5TgwY1ZfwySxIrBV0bT5tDA= 2024-02-01T10:49:09.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tr7c5TgwY1ZfwySxIrBV0bT5tDA=', 'Content-Length': '0'} 2024-02-01T10:49:09.512 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2fbd85d7c881e7d-0065bb7725-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:09.513 DEBUG:boto:b'vclfcn-64obj41000falseobj42024-02-01T10:48:26.689Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:09.513 DEBUG:boto:path=/vclfcn-64/obj4 2024-02-01T10:49:09.513 DEBUG:boto:auth_path=/vclfcn-64/obj4 2024-02-01T10:49:09.513 DEBUG:boto:Method: HEAD 2024-02-01T10:49:09.513 DEBUG:boto:Path: /vclfcn-64/obj4 2024-02-01T10:49:09.513 DEBUG:boto:Data: 2024-02-01T10:49:09.513 DEBUG:boto:Headers: {} 2024-02-01T10:49:09.513 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:09.514 DEBUG:boto:Port: 8001 2024-02-01T10:49:09.514 DEBUG:boto:Params: {} 2024-02-01T10:49:09.514 DEBUG:boto:Token: None 2024-02-01T10:49:09.514 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:09 GMT /vclfcn-64/obj4 2024-02-01T10:49:09.514 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:oWUBSjDHJqQhX7YoKzOz4Gq7ubM= 2024-02-01T10:49:09.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:oWUBSjDHJqQhX7YoKzOz4Gq7ubM=', 'Content-Length': '0'} 2024-02-01T10:49:09.517 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b54de6364555366a-0065bb7725-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:09.517 DEBUG:boto:path=/vclfcn-64/obj4 2024-02-01T10:49:09.517 DEBUG:boto:auth_path=/vclfcn-64/obj4 2024-02-01T10:49:09.517 DEBUG:boto:Method: HEAD 2024-02-01T10:49:09.517 DEBUG:boto:Path: /vclfcn-64/obj4 2024-02-01T10:49:09.518 DEBUG:boto:Data: 2024-02-01T10:49:09.518 DEBUG:boto:Headers: {} 2024-02-01T10:49:09.518 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:09.518 DEBUG:boto:Port: 8000 2024-02-01T10:49:09.518 DEBUG:boto:Params: {} 2024-02-01T10:49:09.518 DEBUG:boto:Token: None 2024-02-01T10:49:09.518 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:09 GMT /vclfcn-64/obj4 2024-02-01T10:49:09.518 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:oWUBSjDHJqQhX7YoKzOz4Gq7ubM= 2024-02-01T10:49:09.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:oWUBSjDHJqQhX7YoKzOz4Gq7ubM=', 'Content-Length': '0'} 2024-02-01T10:49:09.521 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000417b2654f2379baf-0065bb7725-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:09.521 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:49:09.521 DEBUG:boto:path=/vclfcn-64/obj4 2024-02-01T10:49:09.522 DEBUG:boto:auth_path=/vclfcn-64/obj4 2024-02-01T10:49:09.522 DEBUG:boto:Method: GET 2024-02-01T10:49:09.522 DEBUG:boto:Path: /vclfcn-64/obj4 2024-02-01T10:49:09.522 DEBUG:boto:Data: 2024-02-01T10:49:09.522 DEBUG:boto:Headers: {} 2024-02-01T10:49:09.522 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:09.522 DEBUG:boto:Port: 8001 2024-02-01T10:49:09.522 DEBUG:boto:Params: {} 2024-02-01T10:49:09.522 DEBUG:boto:Token: None 2024-02-01T10:49:09.522 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:09 GMT /vclfcn-64/obj4 2024-02-01T10:49:09.523 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GEEFO7e2r7aKZfL+jbLByhfDV2Y= 2024-02-01T10:49:09.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GEEFO7e2r7aKZfL+jbLByhfDV2Y=', 'Content-Length': '0'} 2024-02-01T10:49:09.525 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f67e03fa3c7a24e2-0065bb7725-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:09.525 DEBUG:boto:path=/vclfcn-64/obj4 2024-02-01T10:49:09.525 DEBUG:boto:auth_path=/vclfcn-64/obj4 2024-02-01T10:49:09.526 DEBUG:boto:Method: GET 2024-02-01T10:49:09.526 DEBUG:boto:Path: /vclfcn-64/obj4 2024-02-01T10:49:09.526 DEBUG:boto:Data: 2024-02-01T10:49:09.526 DEBUG:boto:Headers: {} 2024-02-01T10:49:09.526 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:09.526 DEBUG:boto:Port: 8000 2024-02-01T10:49:09.526 DEBUG:boto:Params: {} 2024-02-01T10:49:09.526 DEBUG:boto:Token: None 2024-02-01T10:49:09.526 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:09 GMT /vclfcn-64/obj4 2024-02-01T10:49:09.526 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GEEFO7e2r7aKZfL+jbLByhfDV2Y= 2024-02-01T10:49:09.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GEEFO7e2r7aKZfL+jbLByhfDV2Y=', 'Content-Length': '0'} 2024-02-01T10:49:09.530 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000031eba622842263f9-0065bb7725-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:09.572 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:49:09.572 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:49:09.572 DEBUG:boto:path=/vclfcn-64/?prefix=obj4 2024-02-01T10:49:09.572 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj4 2024-02-01T10:49:09.573 DEBUG:boto:Method: GET 2024-02-01T10:49:09.573 DEBUG:boto:Path: /vclfcn-64/?prefix=obj4 2024-02-01T10:49:09.573 DEBUG:boto:Data: 2024-02-01T10:49:09.573 DEBUG:boto:Headers: {} 2024-02-01T10:49:09.573 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:09.573 DEBUG:boto:Port: 8001 2024-02-01T10:49:09.573 DEBUG:boto:Params: {} 2024-02-01T10:49:09.573 DEBUG:boto:Token: None 2024-02-01T10:49:09.573 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:09 GMT /vclfcn-64/ 2024-02-01T10:49:09.574 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tr7c5TgwY1ZfwySxIrBV0bT5tDA= 2024-02-01T10:49:09.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tr7c5TgwY1ZfwySxIrBV0bT5tDA=', 'Content-Length': '0'} 2024-02-01T10:49:09.578 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c72af0d08bbf83d-0065bb7725-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:09.578 DEBUG:boto:b'vclfcn-64obj41000falseobj42024-02-01T10:48:26.689Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:09.579 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:49:09.579 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:49:09.579 DEBUG:boto:path=/vclfcn-64/?prefix=obj4 2024-02-01T10:49:09.579 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj4 2024-02-01T10:49:09.579 DEBUG:boto:Method: GET 2024-02-01T10:49:09.579 DEBUG:boto:Path: /vclfcn-64/?prefix=obj4 2024-02-01T10:49:09.579 DEBUG:boto:Data: 2024-02-01T10:49:09.579 DEBUG:boto:Headers: {} 2024-02-01T10:49:09.579 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:09.579 DEBUG:boto:Port: 8000 2024-02-01T10:49:09.580 DEBUG:boto:Params: {} 2024-02-01T10:49:09.580 DEBUG:boto:Token: None 2024-02-01T10:49:09.580 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:09 GMT /vclfcn-64/ 2024-02-01T10:49:09.580 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tr7c5TgwY1ZfwySxIrBV0bT5tDA= 2024-02-01T10:49:09.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tr7c5TgwY1ZfwySxIrBV0bT5tDA=', 'Content-Length': '0'} 2024-02-01T10:49:09.584 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009e0f5057868efdf-0065bb7725-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:09.628 DEBUG:boto:b'vclfcn-64obj41000falseobj42024-02-01T10:48:26.689Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:09.628 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-64 zones={test-zone3, test-zone1} 2024-02-01T10:49:09.629 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-64 zones={test-zone3, test-zone2} 2024-02-01T10:49:09.629 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:49:09.629 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:49:09.629 DEBUG:boto:Method: HEAD 2024-02-01T10:49:09.629 DEBUG:boto:Path: /vclfcn-64/ 2024-02-01T10:49:09.629 DEBUG:boto:Data: 2024-02-01T10:49:09.629 DEBUG:boto:Headers: {} 2024-02-01T10:49:09.629 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:09.629 DEBUG:boto:Port: 8001 2024-02-01T10:49:09.629 DEBUG:boto:Params: {} 2024-02-01T10:49:09.630 DEBUG:boto:Token: None 2024-02-01T10:49:09.630 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:09 GMT /vclfcn-64/ 2024-02-01T10:49:09.630 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:eTzncpM0uswYIA0KxJSWpFhzzjo= 2024-02-01T10:49:09.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:eTzncpM0uswYIA0KxJSWpFhzzjo=', 'Content-Length': '0'} 2024-02-01T10:49:09.634 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000bd4fabee7e085ff-0065bb7725-5762-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:49:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:09.635 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:49:09.635 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:49:09.635 DEBUG:boto:Method: HEAD 2024-02-01T10:49:09.635 DEBUG:boto:Path: /vclfcn-64/ 2024-02-01T10:49:09.635 DEBUG:boto:Data: 2024-02-01T10:49:09.635 DEBUG:boto:Headers: {} 2024-02-01T10:49:09.635 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:09.635 DEBUG:boto:Port: 8000 2024-02-01T10:49:09.636 DEBUG:boto:Params: {} 2024-02-01T10:49:09.636 DEBUG:boto:Token: None 2024-02-01T10:49:09.636 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:09 GMT /vclfcn-64/ 2024-02-01T10:49:09.636 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:eTzncpM0uswYIA0KxJSWpFhzzjo= 2024-02-01T10:49:09.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:eTzncpM0uswYIA0KxJSWpFhzzjo=', 'Content-Length': '0'} 2024-02-01T10:49:09.640 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000687ae22795414954-0065bb7725-5350-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:49:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:09.640 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:49:09.641 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:49:09.641 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:49:09.641 DEBUG:boto:path=/vclfcn-64/?versions&max-keys=999 2024-02-01T10:49:09.641 DEBUG:boto:auth_path=/vclfcn-64/?versions&max-keys=999 2024-02-01T10:49:09.641 DEBUG:boto:Method: GET 2024-02-01T10:49:09.641 DEBUG:boto:Path: /vclfcn-64/?versions&max-keys=999 2024-02-01T10:49:09.641 DEBUG:boto:Data: 2024-02-01T10:49:09.641 DEBUG:boto:Headers: {} 2024-02-01T10:49:09.641 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:09.641 DEBUG:boto:Port: 8001 2024-02-01T10:49:09.641 DEBUG:boto:Params: {} 2024-02-01T10:49:09.642 DEBUG:boto:Token: None 2024-02-01T10:49:09.642 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:09 GMT /vclfcn-64/?versions 2024-02-01T10:49:09.642 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:3frpXjVh+XhsTO6P84jih1q8SsM= 2024-02-01T10:49:09.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:3frpXjVh+XhsTO6P84jih1q8SsM=', 'Content-Length': '0'} 2024-02-01T10:49:09.646 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000871418afd4734753-0065bb7725-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:09.688 DEBUG:boto:b'vclfcn-64999falseobj1nulltrue2024-02-01T10:48:26.595Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:48:26.615Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:48:26.634Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:48:26.689Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:09.689 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-02-01T10:49:09.689 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-02-01T10:49:09.689 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-02-01T10:49:09.689 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-02-01T10:49:09.689 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:49:09.689 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:49:09.689 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:49:09.689 DEBUG:boto:path=/vclfcn-64/?versions&max-keys=999 2024-02-01T10:49:09.689 DEBUG:boto:auth_path=/vclfcn-64/?versions&max-keys=999 2024-02-01T10:49:09.690 DEBUG:boto:Method: GET 2024-02-01T10:49:09.690 DEBUG:boto:Path: /vclfcn-64/?versions&max-keys=999 2024-02-01T10:49:09.690 DEBUG:boto:Data: 2024-02-01T10:49:09.690 DEBUG:boto:Headers: {} 2024-02-01T10:49:09.690 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:09.690 DEBUG:boto:Port: 8000 2024-02-01T10:49:09.690 DEBUG:boto:Params: {} 2024-02-01T10:49:09.690 DEBUG:boto:Token: None 2024-02-01T10:49:09.690 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:09 GMT /vclfcn-64/?versions 2024-02-01T10:49:09.691 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:3frpXjVh+XhsTO6P84jih1q8SsM= 2024-02-01T10:49:09.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:3frpXjVh+XhsTO6P84jih1q8SsM=', 'Content-Length': '0'} 2024-02-01T10:49:09.695 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000123791823385e9e9-0065bb7725-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:09.740 DEBUG:boto:b'vclfcn-64999falseobj1nulltrue2024-02-01T10:48:26.595Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:48:26.615Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:48:26.634Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:48:26.689Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:09.740 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-02-01T10:49:09.741 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-02-01T10:49:09.741 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-02-01T10:49:09.741 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-02-01T10:49:09.741 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:49:09.741 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:49:09.741 DEBUG:boto:path=/vclfcn-64/?versions&max-keys=999 2024-02-01T10:49:09.741 DEBUG:boto:auth_path=/vclfcn-64/?versions&max-keys=999 2024-02-01T10:49:09.741 DEBUG:boto:Method: GET 2024-02-01T10:49:09.741 DEBUG:boto:Path: /vclfcn-64/?versions&max-keys=999 2024-02-01T10:49:09.742 DEBUG:boto:Data: 2024-02-01T10:49:09.742 DEBUG:boto:Headers: {} 2024-02-01T10:49:09.742 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:09.742 DEBUG:boto:Port: 8001 2024-02-01T10:49:09.742 DEBUG:boto:Params: {} 2024-02-01T10:49:09.742 DEBUG:boto:Token: None 2024-02-01T10:49:09.742 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:09 GMT /vclfcn-64/?versions 2024-02-01T10:49:09.742 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:3frpXjVh+XhsTO6P84jih1q8SsM= 2024-02-01T10:49:09.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:3frpXjVh+XhsTO6P84jih1q8SsM=', 'Content-Length': '0'} 2024-02-01T10:49:09.750 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5ba0ecd1d5eec08-0065bb7725-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:09.751 DEBUG:boto:b'vclfcn-64999falseobj1nulltrue2024-02-01T10:48:26.595Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:48:26.615Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:48:26.634Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:48:26.689Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:09.751 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:49:09.751 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:49:09.751 DEBUG:boto:path=/vclfcn-64/?versions&max-keys=999 2024-02-01T10:49:09.751 DEBUG:boto:auth_path=/vclfcn-64/?versions&max-keys=999 2024-02-01T10:49:09.751 DEBUG:boto:Method: GET 2024-02-01T10:49:09.752 DEBUG:boto:Path: /vclfcn-64/?versions&max-keys=999 2024-02-01T10:49:09.752 DEBUG:boto:Data: 2024-02-01T10:49:09.752 DEBUG:boto:Headers: {} 2024-02-01T10:49:09.752 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:09.752 DEBUG:boto:Port: 8000 2024-02-01T10:49:09.752 DEBUG:boto:Params: {} 2024-02-01T10:49:09.752 DEBUG:boto:Token: None 2024-02-01T10:49:09.752 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:09 GMT /vclfcn-64/?versions 2024-02-01T10:49:09.752 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:3frpXjVh+XhsTO6P84jih1q8SsM= 2024-02-01T10:49:09.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:3frpXjVh+XhsTO6P84jih1q8SsM=', 'Content-Length': '0'} 2024-02-01T10:49:09.757 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008fce51582548395-0065bb7725-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:09.757 DEBUG:boto:b'vclfcn-64999falseobj1nulltrue2024-02-01T10:48:26.595Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:48:26.615Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:48:26.634Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:48:26.689Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:09.757 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:49:09.758 DEBUG:boto:path=/vclfcn-64/obj1 2024-02-01T10:49:09.758 DEBUG:boto:auth_path=/vclfcn-64/obj1 2024-02-01T10:49:09.758 DEBUG:boto:path=/vclfcn-64/obj1?versionId=null 2024-02-01T10:49:09.758 DEBUG:boto:auth_path=/vclfcn-64/obj1?versionId=null 2024-02-01T10:49:09.758 DEBUG:boto:Method: GET 2024-02-01T10:49:09.758 DEBUG:boto:Path: /vclfcn-64/obj1?versionId=null 2024-02-01T10:49:09.758 DEBUG:boto:Data: 2024-02-01T10:49:09.758 DEBUG:boto:Headers: {} 2024-02-01T10:49:09.758 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:09.758 DEBUG:boto:Port: 8001 2024-02-01T10:49:09.759 DEBUG:boto:Params: {} 2024-02-01T10:49:09.759 DEBUG:boto:Token: None 2024-02-01T10:49:09.759 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:09 GMT /vclfcn-64/obj1?versionId=null 2024-02-01T10:49:09.759 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ohGn7ENI0hEtsXHfROQ/OEVPCnQ= 2024-02-01T10:49:09.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ohGn7ENI0hEtsXHfROQ/OEVPCnQ=', 'Content-Length': '0'} 2024-02-01T10:49:09.762 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000095b77ff02d877f5-0065bb7725-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:09.762 DEBUG:boto:path=/vclfcn-64/obj1 2024-02-01T10:49:09.762 DEBUG:boto:auth_path=/vclfcn-64/obj1 2024-02-01T10:49:09.762 DEBUG:boto:path=/vclfcn-64/obj1?versionId=null 2024-02-01T10:49:09.762 DEBUG:boto:auth_path=/vclfcn-64/obj1?versionId=null 2024-02-01T10:49:09.762 DEBUG:boto:Method: GET 2024-02-01T10:49:09.763 DEBUG:boto:Path: /vclfcn-64/obj1?versionId=null 2024-02-01T10:49:09.763 DEBUG:boto:Data: 2024-02-01T10:49:09.763 DEBUG:boto:Headers: {} 2024-02-01T10:49:09.763 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:09.763 DEBUG:boto:Port: 8000 2024-02-01T10:49:09.763 DEBUG:boto:Params: {} 2024-02-01T10:49:09.763 DEBUG:boto:Token: None 2024-02-01T10:49:09.763 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:09 GMT /vclfcn-64/obj1?versionId=null 2024-02-01T10:49:09.763 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ohGn7ENI0hEtsXHfROQ/OEVPCnQ= 2024-02-01T10:49:09.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ohGn7ENI0hEtsXHfROQ/OEVPCnQ=', 'Content-Length': '0'} 2024-02-01T10:49:09.766 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e4230754dee41811-0065bb7725-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:09.767 DEBUG:boto:path=/vclfcn-64/obj1 2024-02-01T10:49:09.767 DEBUG:boto:auth_path=/vclfcn-64/obj1 2024-02-01T10:49:09.767 DEBUG:boto:path=/vclfcn-64/obj1?versionId=null 2024-02-01T10:49:09.767 DEBUG:boto:auth_path=/vclfcn-64/obj1?versionId=null 2024-02-01T10:49:09.767 DEBUG:boto:Method: HEAD 2024-02-01T10:49:09.767 DEBUG:boto:Path: /vclfcn-64/obj1?versionId=null 2024-02-01T10:49:09.767 DEBUG:boto:Data: 2024-02-01T10:49:09.767 DEBUG:boto:Headers: {} 2024-02-01T10:49:09.767 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:09.768 DEBUG:boto:Port: 8001 2024-02-01T10:49:09.768 DEBUG:boto:Params: {} 2024-02-01T10:49:09.768 DEBUG:boto:Token: None 2024-02-01T10:49:09.768 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:09 GMT /vclfcn-64/obj1?versionId=null 2024-02-01T10:49:09.768 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:JZ+hXBBtkA8basRgyZAb6iRiU7Y= 2024-02-01T10:49:09.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:JZ+hXBBtkA8basRgyZAb6iRiU7Y=', 'Content-Length': '0'} 2024-02-01T10:49:09.771 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008c12f580311dd238-0065bb7725-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:09.772 DEBUG:boto:path=/vclfcn-64/obj1 2024-02-01T10:49:09.772 DEBUG:boto:auth_path=/vclfcn-64/obj1 2024-02-01T10:49:09.772 DEBUG:boto:path=/vclfcn-64/obj1?versionId=null 2024-02-01T10:49:09.772 DEBUG:boto:auth_path=/vclfcn-64/obj1?versionId=null 2024-02-01T10:49:09.772 DEBUG:boto:Method: HEAD 2024-02-01T10:49:09.772 DEBUG:boto:Path: /vclfcn-64/obj1?versionId=null 2024-02-01T10:49:09.772 DEBUG:boto:Data: 2024-02-01T10:49:09.772 DEBUG:boto:Headers: {} 2024-02-01T10:49:09.772 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:09.773 DEBUG:boto:Port: 8000 2024-02-01T10:49:09.773 DEBUG:boto:Params: {} 2024-02-01T10:49:09.773 DEBUG:boto:Token: None 2024-02-01T10:49:09.773 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:09 GMT /vclfcn-64/obj1?versionId=null 2024-02-01T10:49:09.773 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:JZ+hXBBtkA8basRgyZAb6iRiU7Y= 2024-02-01T10:49:09.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:JZ+hXBBtkA8basRgyZAb6iRiU7Y=', 'Content-Length': '0'} 2024-02-01T10:49:09.776 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006f4fd88188e0945a-0065bb7725-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:09.776 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:49:09.776 DEBUG:boto:path=/vclfcn-64/obj1 2024-02-01T10:49:09.776 DEBUG:boto:auth_path=/vclfcn-64/obj1 2024-02-01T10:49:09.776 DEBUG:boto:path=/vclfcn-64/obj1?versionId=null 2024-02-01T10:49:09.776 DEBUG:boto:auth_path=/vclfcn-64/obj1?versionId=null 2024-02-01T10:49:09.776 DEBUG:boto:Method: GET 2024-02-01T10:49:09.777 DEBUG:boto:Path: /vclfcn-64/obj1?versionId=null 2024-02-01T10:49:09.777 DEBUG:boto:Data: 2024-02-01T10:49:09.777 DEBUG:boto:Headers: {} 2024-02-01T10:49:09.777 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:09.777 DEBUG:boto:Port: 8001 2024-02-01T10:49:09.777 DEBUG:boto:Params: {} 2024-02-01T10:49:09.777 DEBUG:boto:Token: None 2024-02-01T10:49:09.777 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:09 GMT /vclfcn-64/obj1?versionId=null 2024-02-01T10:49:09.777 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ohGn7ENI0hEtsXHfROQ/OEVPCnQ= 2024-02-01T10:49:09.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ohGn7ENI0hEtsXHfROQ/OEVPCnQ=', 'Content-Length': '0'} 2024-02-01T10:49:09.780 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004f7a5a8615b214da-0065bb7725-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:09.781 DEBUG:boto:path=/vclfcn-64/obj1 2024-02-01T10:49:09.781 DEBUG:boto:auth_path=/vclfcn-64/obj1 2024-02-01T10:49:09.781 DEBUG:boto:path=/vclfcn-64/obj1?versionId=null 2024-02-01T10:49:09.781 DEBUG:boto:auth_path=/vclfcn-64/obj1?versionId=null 2024-02-01T10:49:09.781 DEBUG:boto:Method: GET 2024-02-01T10:49:09.781 DEBUG:boto:Path: /vclfcn-64/obj1?versionId=null 2024-02-01T10:49:09.781 DEBUG:boto:Data: 2024-02-01T10:49:09.781 DEBUG:boto:Headers: {} 2024-02-01T10:49:09.782 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:09.782 DEBUG:boto:Port: 8000 2024-02-01T10:49:09.782 DEBUG:boto:Params: {} 2024-02-01T10:49:09.782 DEBUG:boto:Token: None 2024-02-01T10:49:09.782 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:09 GMT /vclfcn-64/obj1?versionId=null 2024-02-01T10:49:09.782 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ohGn7ENI0hEtsXHfROQ/OEVPCnQ= 2024-02-01T10:49:09.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ohGn7ENI0hEtsXHfROQ/OEVPCnQ=', 'Content-Length': '0'} 2024-02-01T10:49:09.785 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cbdd2fac9d586ef6-0065bb7725-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:09.828 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:49:09.828 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:49:09.828 DEBUG:boto:path=/vclfcn-64/?prefix=obj1 2024-02-01T10:49:09.828 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj1 2024-02-01T10:49:09.829 DEBUG:boto:Method: GET 2024-02-01T10:49:09.829 DEBUG:boto:Path: /vclfcn-64/?prefix=obj1 2024-02-01T10:49:09.829 DEBUG:boto:Data: 2024-02-01T10:49:09.829 DEBUG:boto:Headers: {} 2024-02-01T10:49:09.829 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:09.829 DEBUG:boto:Port: 8001 2024-02-01T10:49:09.829 DEBUG:boto:Params: {} 2024-02-01T10:49:09.829 DEBUG:boto:Token: None 2024-02-01T10:49:09.829 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:09 GMT /vclfcn-64/ 2024-02-01T10:49:09.830 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tr7c5TgwY1ZfwySxIrBV0bT5tDA= 2024-02-01T10:49:09.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tr7c5TgwY1ZfwySxIrBV0bT5tDA=', 'Content-Length': '0'} 2024-02-01T10:49:09.834 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f814881a3798b7c-0065bb7725-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:09.834 DEBUG:boto:b'vclfcn-64obj11000falseobj12024-02-01T10:48:26.595Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:09.835 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:49:09.835 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:49:09.835 DEBUG:boto:path=/vclfcn-64/?prefix=obj1 2024-02-01T10:49:09.835 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj1 2024-02-01T10:49:09.835 DEBUG:boto:Method: GET 2024-02-01T10:49:09.835 DEBUG:boto:Path: /vclfcn-64/?prefix=obj1 2024-02-01T10:49:09.835 DEBUG:boto:Data: 2024-02-01T10:49:09.835 DEBUG:boto:Headers: {} 2024-02-01T10:49:09.835 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:09.835 DEBUG:boto:Port: 8000 2024-02-01T10:49:09.836 DEBUG:boto:Params: {} 2024-02-01T10:49:09.836 DEBUG:boto:Token: None 2024-02-01T10:49:09.836 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:09 GMT /vclfcn-64/ 2024-02-01T10:49:09.836 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tr7c5TgwY1ZfwySxIrBV0bT5tDA= 2024-02-01T10:49:09.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tr7c5TgwY1ZfwySxIrBV0bT5tDA=', 'Content-Length': '0'} 2024-02-01T10:49:09.840 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ff0e285df8e6bec-0065bb7725-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:09.884 DEBUG:boto:b'vclfcn-64obj11000falseobj12024-02-01T10:48:26.595Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:09.884 DEBUG:boto:path=/vclfcn-64/obj1 2024-02-01T10:49:09.885 DEBUG:boto:auth_path=/vclfcn-64/obj1 2024-02-01T10:49:09.885 DEBUG:boto:Method: HEAD 2024-02-01T10:49:09.885 DEBUG:boto:Path: /vclfcn-64/obj1 2024-02-01T10:49:09.885 DEBUG:boto:Data: 2024-02-01T10:49:09.885 DEBUG:boto:Headers: {} 2024-02-01T10:49:09.885 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:09.885 DEBUG:boto:Port: 8001 2024-02-01T10:49:09.885 DEBUG:boto:Params: {} 2024-02-01T10:49:09.885 DEBUG:boto:Token: None 2024-02-01T10:49:09.885 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:09 GMT /vclfcn-64/obj1 2024-02-01T10:49:09.886 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GBCEcJkuvuhJghx0zlCURl2oV6A= 2024-02-01T10:49:09.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GBCEcJkuvuhJghx0zlCURl2oV6A=', 'Content-Length': '0'} 2024-02-01T10:49:09.888 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000661ad6b43fa1dd01-0065bb7725-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:09.889 DEBUG:boto:path=/vclfcn-64/obj1 2024-02-01T10:49:09.889 DEBUG:boto:auth_path=/vclfcn-64/obj1 2024-02-01T10:49:09.889 DEBUG:boto:Method: HEAD 2024-02-01T10:49:09.889 DEBUG:boto:Path: /vclfcn-64/obj1 2024-02-01T10:49:09.889 DEBUG:boto:Data: 2024-02-01T10:49:09.889 DEBUG:boto:Headers: {} 2024-02-01T10:49:09.889 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:09.889 DEBUG:boto:Port: 8000 2024-02-01T10:49:09.889 DEBUG:boto:Params: {} 2024-02-01T10:49:09.890 DEBUG:boto:Token: None 2024-02-01T10:49:09.890 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:09 GMT /vclfcn-64/obj1 2024-02-01T10:49:09.890 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GBCEcJkuvuhJghx0zlCURl2oV6A= 2024-02-01T10:49:09.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GBCEcJkuvuhJghx0zlCURl2oV6A=', 'Content-Length': '0'} 2024-02-01T10:49:09.892 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005b926c93e5e4834e-0065bb7725-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:09.893 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:49:09.893 DEBUG:boto:path=/vclfcn-64/obj1 2024-02-01T10:49:09.893 DEBUG:boto:auth_path=/vclfcn-64/obj1 2024-02-01T10:49:09.893 DEBUG:boto:Method: GET 2024-02-01T10:49:09.893 DEBUG:boto:Path: /vclfcn-64/obj1 2024-02-01T10:49:09.893 DEBUG:boto:Data: 2024-02-01T10:49:09.893 DEBUG:boto:Headers: {} 2024-02-01T10:49:09.893 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:09.894 DEBUG:boto:Port: 8001 2024-02-01T10:49:09.894 DEBUG:boto:Params: {} 2024-02-01T10:49:09.894 DEBUG:boto:Token: None 2024-02-01T10:49:09.894 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:09 GMT /vclfcn-64/obj1 2024-02-01T10:49:09.894 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:LlC/Q5wUkMgmtG3Gy1VlHUcuGiI= 2024-02-01T10:49:09.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:LlC/Q5wUkMgmtG3Gy1VlHUcuGiI=', 'Content-Length': '0'} 2024-02-01T10:49:09.899 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000da6a98542ec6b674-0065bb7725-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:09.940 DEBUG:boto:path=/vclfcn-64/obj1 2024-02-01T10:49:09.940 DEBUG:boto:auth_path=/vclfcn-64/obj1 2024-02-01T10:49:09.940 DEBUG:boto:Method: GET 2024-02-01T10:49:09.940 DEBUG:boto:Path: /vclfcn-64/obj1 2024-02-01T10:49:09.941 DEBUG:boto:Data: 2024-02-01T10:49:09.941 DEBUG:boto:Headers: {} 2024-02-01T10:49:09.941 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:09.941 DEBUG:boto:Port: 8000 2024-02-01T10:49:09.941 DEBUG:boto:Params: {} 2024-02-01T10:49:09.941 DEBUG:boto:Token: None 2024-02-01T10:49:09.941 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:09 GMT /vclfcn-64/obj1 2024-02-01T10:49:09.941 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:LlC/Q5wUkMgmtG3Gy1VlHUcuGiI= 2024-02-01T10:49:09.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:LlC/Q5wUkMgmtG3Gy1VlHUcuGiI=', 'Content-Length': '0'} 2024-02-01T10:49:09.945 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008b7c120f4f1d3ff3-0065bb7725-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:09.988 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:49:09.988 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:49:09.988 DEBUG:boto:path=/vclfcn-64/?prefix=obj1 2024-02-01T10:49:09.989 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj1 2024-02-01T10:49:09.989 DEBUG:boto:Method: GET 2024-02-01T10:49:09.989 DEBUG:boto:Path: /vclfcn-64/?prefix=obj1 2024-02-01T10:49:09.989 DEBUG:boto:Data: 2024-02-01T10:49:09.989 DEBUG:boto:Headers: {} 2024-02-01T10:49:09.989 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:09.989 DEBUG:boto:Port: 8001 2024-02-01T10:49:09.989 DEBUG:boto:Params: {} 2024-02-01T10:49:09.989 DEBUG:boto:Token: None 2024-02-01T10:49:09.989 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:09 GMT /vclfcn-64/ 2024-02-01T10:49:09.990 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tr7c5TgwY1ZfwySxIrBV0bT5tDA= 2024-02-01T10:49:09.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tr7c5TgwY1ZfwySxIrBV0bT5tDA=', 'Content-Length': '0'} 2024-02-01T10:49:09.994 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089142cabf2b345de-0065bb7725-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:09.994 DEBUG:boto:b'vclfcn-64obj11000falseobj12024-02-01T10:48:26.595Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:09.994 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:49:09.994 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:49:09.995 DEBUG:boto:path=/vclfcn-64/?prefix=obj1 2024-02-01T10:49:09.995 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj1 2024-02-01T10:49:09.995 DEBUG:boto:Method: GET 2024-02-01T10:49:09.995 DEBUG:boto:Path: /vclfcn-64/?prefix=obj1 2024-02-01T10:49:09.995 DEBUG:boto:Data: 2024-02-01T10:49:09.995 DEBUG:boto:Headers: {} 2024-02-01T10:49:09.995 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:09.995 DEBUG:boto:Port: 8000 2024-02-01T10:49:09.995 DEBUG:boto:Params: {} 2024-02-01T10:49:09.995 DEBUG:boto:Token: None 2024-02-01T10:49:09.996 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:09 GMT /vclfcn-64/ 2024-02-01T10:49:09.996 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tr7c5TgwY1ZfwySxIrBV0bT5tDA= 2024-02-01T10:49:09.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tr7c5TgwY1ZfwySxIrBV0bT5tDA=', 'Content-Length': '0'} 2024-02-01T10:49:10.000 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008be8b993083e388b-0065bb7725-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:10.000 DEBUG:boto:b'vclfcn-64obj11000falseobj12024-02-01T10:48:26.595Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:10.001 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:49:10.001 DEBUG:boto:path=/vclfcn-64/obj2 2024-02-01T10:49:10.001 DEBUG:boto:auth_path=/vclfcn-64/obj2 2024-02-01T10:49:10.001 DEBUG:boto:path=/vclfcn-64/obj2?versionId=null 2024-02-01T10:49:10.001 DEBUG:boto:auth_path=/vclfcn-64/obj2?versionId=null 2024-02-01T10:49:10.001 DEBUG:boto:Method: GET 2024-02-01T10:49:10.001 DEBUG:boto:Path: /vclfcn-64/obj2?versionId=null 2024-02-01T10:49:10.001 DEBUG:boto:Data: 2024-02-01T10:49:10.001 DEBUG:boto:Headers: {} 2024-02-01T10:49:10.001 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:10.001 DEBUG:boto:Port: 8001 2024-02-01T10:49:10.002 DEBUG:boto:Params: {} 2024-02-01T10:49:10.002 DEBUG:boto:Token: None 2024-02-01T10:49:10.002 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:10 GMT /vclfcn-64/obj2?versionId=null 2024-02-01T10:49:10.002 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GhBMRKUDgVnNTCen2vrOSzzqbQM= 2024-02-01T10:49:10.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GhBMRKUDgVnNTCen2vrOSzzqbQM=', 'Content-Length': '0'} 2024-02-01T10:49:10.010 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d567713a9edb15cb-0065bb7725-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:10.011 DEBUG:boto:path=/vclfcn-64/obj2 2024-02-01T10:49:10.011 DEBUG:boto:auth_path=/vclfcn-64/obj2 2024-02-01T10:49:10.011 DEBUG:boto:path=/vclfcn-64/obj2?versionId=null 2024-02-01T10:49:10.011 DEBUG:boto:auth_path=/vclfcn-64/obj2?versionId=null 2024-02-01T10:49:10.011 DEBUG:boto:Method: GET 2024-02-01T10:49:10.011 DEBUG:boto:Path: /vclfcn-64/obj2?versionId=null 2024-02-01T10:49:10.012 DEBUG:boto:Data: 2024-02-01T10:49:10.012 DEBUG:boto:Headers: {} 2024-02-01T10:49:10.012 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:10.012 DEBUG:boto:Port: 8000 2024-02-01T10:49:10.012 DEBUG:boto:Params: {} 2024-02-01T10:49:10.013 DEBUG:boto:Token: None 2024-02-01T10:49:10.013 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:10 GMT /vclfcn-64/obj2?versionId=null 2024-02-01T10:49:10.014 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GhBMRKUDgVnNTCen2vrOSzzqbQM= 2024-02-01T10:49:10.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GhBMRKUDgVnNTCen2vrOSzzqbQM=', 'Content-Length': '0'} 2024-02-01T10:49:10.016 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000096b62451c53edbd6-0065bb7726-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:10.064 DEBUG:boto:path=/vclfcn-64/obj2 2024-02-01T10:49:10.064 DEBUG:boto:auth_path=/vclfcn-64/obj2 2024-02-01T10:49:10.064 DEBUG:boto:path=/vclfcn-64/obj2?versionId=null 2024-02-01T10:49:10.064 DEBUG:boto:auth_path=/vclfcn-64/obj2?versionId=null 2024-02-01T10:49:10.065 DEBUG:boto:Method: HEAD 2024-02-01T10:49:10.065 DEBUG:boto:Path: /vclfcn-64/obj2?versionId=null 2024-02-01T10:49:10.065 DEBUG:boto:Data: 2024-02-01T10:49:10.065 DEBUG:boto:Headers: {} 2024-02-01T10:49:10.065 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:10.065 DEBUG:boto:Port: 8001 2024-02-01T10:49:10.065 DEBUG:boto:Params: {} 2024-02-01T10:49:10.065 DEBUG:boto:Token: None 2024-02-01T10:49:10.065 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:10 GMT /vclfcn-64/obj2?versionId=null 2024-02-01T10:49:10.066 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:KrKylpOq+2x83R+exW8F/yvv/+c= 2024-02-01T10:49:10.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:KrKylpOq+2x83R+exW8F/yvv/+c=', 'Content-Length': '0'} 2024-02-01T10:49:10.068 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000007771584186f0d76-0065bb7726-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:10.069 DEBUG:boto:path=/vclfcn-64/obj2 2024-02-01T10:49:10.069 DEBUG:boto:auth_path=/vclfcn-64/obj2 2024-02-01T10:49:10.069 DEBUG:boto:path=/vclfcn-64/obj2?versionId=null 2024-02-01T10:49:10.069 DEBUG:boto:auth_path=/vclfcn-64/obj2?versionId=null 2024-02-01T10:49:10.069 DEBUG:boto:Method: HEAD 2024-02-01T10:49:10.069 DEBUG:boto:Path: /vclfcn-64/obj2?versionId=null 2024-02-01T10:49:10.069 DEBUG:boto:Data: 2024-02-01T10:49:10.069 DEBUG:boto:Headers: {} 2024-02-01T10:49:10.069 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:10.069 DEBUG:boto:Port: 8000 2024-02-01T10:49:10.070 DEBUG:boto:Params: {} 2024-02-01T10:49:10.070 DEBUG:boto:Token: None 2024-02-01T10:49:10.070 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:10 GMT /vclfcn-64/obj2?versionId=null 2024-02-01T10:49:10.070 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:KrKylpOq+2x83R+exW8F/yvv/+c= 2024-02-01T10:49:10.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:KrKylpOq+2x83R+exW8F/yvv/+c=', 'Content-Length': '0'} 2024-02-01T10:49:10.073 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000feb187117772d6f1-0065bb7726-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:10.073 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:49:10.073 DEBUG:boto:path=/vclfcn-64/obj2 2024-02-01T10:49:10.073 DEBUG:boto:auth_path=/vclfcn-64/obj2 2024-02-01T10:49:10.073 DEBUG:boto:path=/vclfcn-64/obj2?versionId=null 2024-02-01T10:49:10.073 DEBUG:boto:auth_path=/vclfcn-64/obj2?versionId=null 2024-02-01T10:49:10.074 DEBUG:boto:Method: GET 2024-02-01T10:49:10.074 DEBUG:boto:Path: /vclfcn-64/obj2?versionId=null 2024-02-01T10:49:10.074 DEBUG:boto:Data: 2024-02-01T10:49:10.074 DEBUG:boto:Headers: {} 2024-02-01T10:49:10.074 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:10.074 DEBUG:boto:Port: 8001 2024-02-01T10:49:10.074 DEBUG:boto:Params: {} 2024-02-01T10:49:10.074 DEBUG:boto:Token: None 2024-02-01T10:49:10.074 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:10 GMT /vclfcn-64/obj2?versionId=null 2024-02-01T10:49:10.075 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GhBMRKUDgVnNTCen2vrOSzzqbQM= 2024-02-01T10:49:10.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GhBMRKUDgVnNTCen2vrOSzzqbQM=', 'Content-Length': '0'} 2024-02-01T10:49:10.078 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cee2c9eb5ae54546-0065bb7726-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:10.078 DEBUG:boto:path=/vclfcn-64/obj2 2024-02-01T10:49:10.078 DEBUG:boto:auth_path=/vclfcn-64/obj2 2024-02-01T10:49:10.078 DEBUG:boto:path=/vclfcn-64/obj2?versionId=null 2024-02-01T10:49:10.078 DEBUG:boto:auth_path=/vclfcn-64/obj2?versionId=null 2024-02-01T10:49:10.078 DEBUG:boto:Method: GET 2024-02-01T10:49:10.079 DEBUG:boto:Path: /vclfcn-64/obj2?versionId=null 2024-02-01T10:49:10.079 DEBUG:boto:Data: 2024-02-01T10:49:10.079 DEBUG:boto:Headers: {} 2024-02-01T10:49:10.079 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:10.079 DEBUG:boto:Port: 8000 2024-02-01T10:49:10.079 DEBUG:boto:Params: {} 2024-02-01T10:49:10.079 DEBUG:boto:Token: None 2024-02-01T10:49:10.079 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:10 GMT /vclfcn-64/obj2?versionId=null 2024-02-01T10:49:10.079 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GhBMRKUDgVnNTCen2vrOSzzqbQM= 2024-02-01T10:49:10.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GhBMRKUDgVnNTCen2vrOSzzqbQM=', 'Content-Length': '0'} 2024-02-01T10:49:10.082 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e1e24f731a168a13-0065bb7726-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:10.124 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:49:10.124 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:49:10.125 DEBUG:boto:path=/vclfcn-64/?prefix=obj2 2024-02-01T10:49:10.125 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj2 2024-02-01T10:49:10.125 DEBUG:boto:Method: GET 2024-02-01T10:49:10.125 DEBUG:boto:Path: /vclfcn-64/?prefix=obj2 2024-02-01T10:49:10.125 DEBUG:boto:Data: 2024-02-01T10:49:10.125 DEBUG:boto:Headers: {} 2024-02-01T10:49:10.125 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:10.125 DEBUG:boto:Port: 8001 2024-02-01T10:49:10.125 DEBUG:boto:Params: {} 2024-02-01T10:49:10.126 DEBUG:boto:Token: None 2024-02-01T10:49:10.126 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:10 GMT /vclfcn-64/ 2024-02-01T10:49:10.126 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:f48i+ScGVYX5F7GTecJbJuACoXU= 2024-02-01T10:49:10.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:f48i+ScGVYX5F7GTecJbJuACoXU=', 'Content-Length': '0'} 2024-02-01T10:49:10.131 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b61384689e8b263-0065bb7726-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:10.131 DEBUG:boto:b'vclfcn-64obj21000falseobj22024-02-01T10:48:26.615Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:10.131 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:49:10.132 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:49:10.132 DEBUG:boto:path=/vclfcn-64/?prefix=obj2 2024-02-01T10:49:10.132 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj2 2024-02-01T10:49:10.132 DEBUG:boto:Method: GET 2024-02-01T10:49:10.132 DEBUG:boto:Path: /vclfcn-64/?prefix=obj2 2024-02-01T10:49:10.132 DEBUG:boto:Data: 2024-02-01T10:49:10.132 DEBUG:boto:Headers: {} 2024-02-01T10:49:10.132 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:10.132 DEBUG:boto:Port: 8000 2024-02-01T10:49:10.132 DEBUG:boto:Params: {} 2024-02-01T10:49:10.133 DEBUG:boto:Token: None 2024-02-01T10:49:10.133 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:10 GMT /vclfcn-64/ 2024-02-01T10:49:10.133 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:f48i+ScGVYX5F7GTecJbJuACoXU= 2024-02-01T10:49:10.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:f48i+ScGVYX5F7GTecJbJuACoXU=', 'Content-Length': '0'} 2024-02-01T10:49:10.138 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fbc4489c37e8e71f-0065bb7726-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:10.180 DEBUG:boto:b'vclfcn-64obj21000falseobj22024-02-01T10:48:26.615Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:10.181 DEBUG:boto:path=/vclfcn-64/obj2 2024-02-01T10:49:10.181 DEBUG:boto:auth_path=/vclfcn-64/obj2 2024-02-01T10:49:10.181 DEBUG:boto:Method: HEAD 2024-02-01T10:49:10.181 DEBUG:boto:Path: /vclfcn-64/obj2 2024-02-01T10:49:10.181 DEBUG:boto:Data: 2024-02-01T10:49:10.182 DEBUG:boto:Headers: {} 2024-02-01T10:49:10.182 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:10.182 DEBUG:boto:Port: 8001 2024-02-01T10:49:10.182 DEBUG:boto:Params: {} 2024-02-01T10:49:10.182 DEBUG:boto:Token: None 2024-02-01T10:49:10.182 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:10 GMT /vclfcn-64/obj2 2024-02-01T10:49:10.182 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:v+x5vy1g5NP/QFdCXdGnqDbVYF8= 2024-02-01T10:49:10.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:v+x5vy1g5NP/QFdCXdGnqDbVYF8=', 'Content-Length': '0'} 2024-02-01T10:49:10.185 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007be794b2d20a5c14-0065bb7726-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:10.186 DEBUG:boto:path=/vclfcn-64/obj2 2024-02-01T10:49:10.186 DEBUG:boto:auth_path=/vclfcn-64/obj2 2024-02-01T10:49:10.186 DEBUG:boto:Method: HEAD 2024-02-01T10:49:10.186 DEBUG:boto:Path: /vclfcn-64/obj2 2024-02-01T10:49:10.186 DEBUG:boto:Data: 2024-02-01T10:49:10.186 DEBUG:boto:Headers: {} 2024-02-01T10:49:10.187 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:10.187 DEBUG:boto:Port: 8000 2024-02-01T10:49:10.187 DEBUG:boto:Params: {} 2024-02-01T10:49:10.187 DEBUG:boto:Token: None 2024-02-01T10:49:10.187 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:10 GMT /vclfcn-64/obj2 2024-02-01T10:49:10.187 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:v+x5vy1g5NP/QFdCXdGnqDbVYF8= 2024-02-01T10:49:10.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:v+x5vy1g5NP/QFdCXdGnqDbVYF8=', 'Content-Length': '0'} 2024-02-01T10:49:10.190 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000084857e8d1e5b0f3b-0065bb7726-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:10.190 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:49:10.190 DEBUG:boto:path=/vclfcn-64/obj2 2024-02-01T10:49:10.190 DEBUG:boto:auth_path=/vclfcn-64/obj2 2024-02-01T10:49:10.190 DEBUG:boto:Method: GET 2024-02-01T10:49:10.190 DEBUG:boto:Path: /vclfcn-64/obj2 2024-02-01T10:49:10.191 DEBUG:boto:Data: 2024-02-01T10:49:10.191 DEBUG:boto:Headers: {} 2024-02-01T10:49:10.191 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:10.191 DEBUG:boto:Port: 8001 2024-02-01T10:49:10.191 DEBUG:boto:Params: {} 2024-02-01T10:49:10.191 DEBUG:boto:Token: None 2024-02-01T10:49:10.191 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:10 GMT /vclfcn-64/obj2 2024-02-01T10:49:10.191 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wky1qbgNfSb1aEIyF5z97nTTctw= 2024-02-01T10:49:10.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wky1qbgNfSb1aEIyF5z97nTTctw=', 'Content-Length': '0'} 2024-02-01T10:49:10.194 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bac4d05cb3869ea6-0065bb7726-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:10.236 DEBUG:boto:path=/vclfcn-64/obj2 2024-02-01T10:49:10.236 DEBUG:boto:auth_path=/vclfcn-64/obj2 2024-02-01T10:49:10.236 DEBUG:boto:Method: GET 2024-02-01T10:49:10.237 DEBUG:boto:Path: /vclfcn-64/obj2 2024-02-01T10:49:10.237 DEBUG:boto:Data: 2024-02-01T10:49:10.237 DEBUG:boto:Headers: {} 2024-02-01T10:49:10.237 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:10.237 DEBUG:boto:Port: 8000 2024-02-01T10:49:10.237 DEBUG:boto:Params: {} 2024-02-01T10:49:10.237 DEBUG:boto:Token: None 2024-02-01T10:49:10.238 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:10 GMT /vclfcn-64/obj2 2024-02-01T10:49:10.238 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wky1qbgNfSb1aEIyF5z97nTTctw= 2024-02-01T10:49:10.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wky1qbgNfSb1aEIyF5z97nTTctw=', 'Content-Length': '0'} 2024-02-01T10:49:10.241 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b2c6651cb319944d-0065bb7726-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:10.284 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:49:10.284 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:49:10.284 DEBUG:boto:path=/vclfcn-64/?prefix=obj2 2024-02-01T10:49:10.284 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj2 2024-02-01T10:49:10.285 DEBUG:boto:Method: GET 2024-02-01T10:49:10.285 DEBUG:boto:Path: /vclfcn-64/?prefix=obj2 2024-02-01T10:49:10.285 DEBUG:boto:Data: 2024-02-01T10:49:10.285 DEBUG:boto:Headers: {} 2024-02-01T10:49:10.285 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:10.285 DEBUG:boto:Port: 8001 2024-02-01T10:49:10.285 DEBUG:boto:Params: {} 2024-02-01T10:49:10.285 DEBUG:boto:Token: None 2024-02-01T10:49:10.285 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:10 GMT /vclfcn-64/ 2024-02-01T10:49:10.286 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:f48i+ScGVYX5F7GTecJbJuACoXU= 2024-02-01T10:49:10.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:f48i+ScGVYX5F7GTecJbJuACoXU=', 'Content-Length': '0'} 2024-02-01T10:49:10.290 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b639bda4b8bf08cd-0065bb7726-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:10.290 DEBUG:boto:b'vclfcn-64obj21000falseobj22024-02-01T10:48:26.615Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:10.290 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:49:10.290 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:49:10.291 DEBUG:boto:path=/vclfcn-64/?prefix=obj2 2024-02-01T10:49:10.291 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj2 2024-02-01T10:49:10.291 DEBUG:boto:Method: GET 2024-02-01T10:49:10.291 DEBUG:boto:Path: /vclfcn-64/?prefix=obj2 2024-02-01T10:49:10.291 DEBUG:boto:Data: 2024-02-01T10:49:10.291 DEBUG:boto:Headers: {} 2024-02-01T10:49:10.291 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:10.291 DEBUG:boto:Port: 8000 2024-02-01T10:49:10.291 DEBUG:boto:Params: {} 2024-02-01T10:49:10.291 DEBUG:boto:Token: None 2024-02-01T10:49:10.292 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:10 GMT /vclfcn-64/ 2024-02-01T10:49:10.292 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:f48i+ScGVYX5F7GTecJbJuACoXU= 2024-02-01T10:49:10.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:f48i+ScGVYX5F7GTecJbJuACoXU=', 'Content-Length': '0'} 2024-02-01T10:49:10.296 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a18166e26710162d-0065bb7726-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:10.296 DEBUG:boto:b'vclfcn-64obj21000falseobj22024-02-01T10:48:26.615Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:10.297 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:49:10.297 DEBUG:boto:path=/vclfcn-64/obj3 2024-02-01T10:49:10.297 DEBUG:boto:auth_path=/vclfcn-64/obj3 2024-02-01T10:49:10.297 DEBUG:boto:path=/vclfcn-64/obj3?versionId=null 2024-02-01T10:49:10.298 DEBUG:boto:auth_path=/vclfcn-64/obj3?versionId=null 2024-02-01T10:49:10.298 DEBUG:boto:Method: GET 2024-02-01T10:49:10.298 DEBUG:boto:Path: /vclfcn-64/obj3?versionId=null 2024-02-01T10:49:10.298 DEBUG:boto:Data: 2024-02-01T10:49:10.298 DEBUG:boto:Headers: {} 2024-02-01T10:49:10.298 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:10.298 DEBUG:boto:Port: 8001 2024-02-01T10:49:10.299 DEBUG:boto:Params: {} 2024-02-01T10:49:10.299 DEBUG:boto:Token: None 2024-02-01T10:49:10.299 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:10 GMT /vclfcn-64/obj3?versionId=null 2024-02-01T10:49:10.299 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lkR2rEXRCBQBVRWDbqzy4wlPYlY= 2024-02-01T10:49:10.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lkR2rEXRCBQBVRWDbqzy4wlPYlY=', 'Content-Length': '0'} 2024-02-01T10:49:10.303 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ebbe531e0990249e-0065bb7726-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:10.304 DEBUG:boto:path=/vclfcn-64/obj3 2024-02-01T10:49:10.304 DEBUG:boto:auth_path=/vclfcn-64/obj3 2024-02-01T10:49:10.304 DEBUG:boto:path=/vclfcn-64/obj3?versionId=null 2024-02-01T10:49:10.304 DEBUG:boto:auth_path=/vclfcn-64/obj3?versionId=null 2024-02-01T10:49:10.304 DEBUG:boto:Method: GET 2024-02-01T10:49:10.304 DEBUG:boto:Path: /vclfcn-64/obj3?versionId=null 2024-02-01T10:49:10.304 DEBUG:boto:Data: 2024-02-01T10:49:10.305 DEBUG:boto:Headers: {} 2024-02-01T10:49:10.305 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:10.305 DEBUG:boto:Port: 8000 2024-02-01T10:49:10.305 DEBUG:boto:Params: {} 2024-02-01T10:49:10.305 DEBUG:boto:Token: None 2024-02-01T10:49:10.305 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:10 GMT /vclfcn-64/obj3?versionId=null 2024-02-01T10:49:10.305 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lkR2rEXRCBQBVRWDbqzy4wlPYlY= 2024-02-01T10:49:10.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lkR2rEXRCBQBVRWDbqzy4wlPYlY=', 'Content-Length': '0'} 2024-02-01T10:49:10.308 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004ec8b507faa8c3b8-0065bb7726-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:10.352 DEBUG:boto:path=/vclfcn-64/obj3 2024-02-01T10:49:10.352 DEBUG:boto:auth_path=/vclfcn-64/obj3 2024-02-01T10:49:10.352 DEBUG:boto:path=/vclfcn-64/obj3?versionId=null 2024-02-01T10:49:10.353 DEBUG:boto:auth_path=/vclfcn-64/obj3?versionId=null 2024-02-01T10:49:10.353 DEBUG:boto:Method: HEAD 2024-02-01T10:49:10.353 DEBUG:boto:Path: /vclfcn-64/obj3?versionId=null 2024-02-01T10:49:10.353 DEBUG:boto:Data: 2024-02-01T10:49:10.353 DEBUG:boto:Headers: {} 2024-02-01T10:49:10.353 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:10.353 DEBUG:boto:Port: 8001 2024-02-01T10:49:10.353 DEBUG:boto:Params: {} 2024-02-01T10:49:10.353 DEBUG:boto:Token: None 2024-02-01T10:49:10.353 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:10 GMT /vclfcn-64/obj3?versionId=null 2024-02-01T10:49:10.354 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:OJNTiEkDXjkzvWilOxEJZFOwL/Y= 2024-02-01T10:49:10.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:OJNTiEkDXjkzvWilOxEJZFOwL/Y=', 'Content-Length': '0'} 2024-02-01T10:49:10.356 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c46a9698d1e2eb8a-0065bb7726-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:10.357 DEBUG:boto:path=/vclfcn-64/obj3 2024-02-01T10:49:10.357 DEBUG:boto:auth_path=/vclfcn-64/obj3 2024-02-01T10:49:10.357 DEBUG:boto:path=/vclfcn-64/obj3?versionId=null 2024-02-01T10:49:10.357 DEBUG:boto:auth_path=/vclfcn-64/obj3?versionId=null 2024-02-01T10:49:10.357 DEBUG:boto:Method: HEAD 2024-02-01T10:49:10.357 DEBUG:boto:Path: /vclfcn-64/obj3?versionId=null 2024-02-01T10:49:10.357 DEBUG:boto:Data: 2024-02-01T10:49:10.357 DEBUG:boto:Headers: {} 2024-02-01T10:49:10.357 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:10.358 DEBUG:boto:Port: 8000 2024-02-01T10:49:10.358 DEBUG:boto:Params: {} 2024-02-01T10:49:10.358 DEBUG:boto:Token: None 2024-02-01T10:49:10.358 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:10 GMT /vclfcn-64/obj3?versionId=null 2024-02-01T10:49:10.358 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:OJNTiEkDXjkzvWilOxEJZFOwL/Y= 2024-02-01T10:49:10.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:OJNTiEkDXjkzvWilOxEJZFOwL/Y=', 'Content-Length': '0'} 2024-02-01T10:49:10.361 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000319644963a89ced-0065bb7726-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:10.361 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:49:10.361 DEBUG:boto:path=/vclfcn-64/obj3 2024-02-01T10:49:10.361 DEBUG:boto:auth_path=/vclfcn-64/obj3 2024-02-01T10:49:10.361 DEBUG:boto:path=/vclfcn-64/obj3?versionId=null 2024-02-01T10:49:10.361 DEBUG:boto:auth_path=/vclfcn-64/obj3?versionId=null 2024-02-01T10:49:10.361 DEBUG:boto:Method: GET 2024-02-01T10:49:10.361 DEBUG:boto:Path: /vclfcn-64/obj3?versionId=null 2024-02-01T10:49:10.362 DEBUG:boto:Data: 2024-02-01T10:49:10.362 DEBUG:boto:Headers: {} 2024-02-01T10:49:10.362 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:10.362 DEBUG:boto:Port: 8001 2024-02-01T10:49:10.362 DEBUG:boto:Params: {} 2024-02-01T10:49:10.362 DEBUG:boto:Token: None 2024-02-01T10:49:10.362 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:10 GMT /vclfcn-64/obj3?versionId=null 2024-02-01T10:49:10.362 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lkR2rEXRCBQBVRWDbqzy4wlPYlY= 2024-02-01T10:49:10.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lkR2rEXRCBQBVRWDbqzy4wlPYlY=', 'Content-Length': '0'} 2024-02-01T10:49:10.365 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000033347334241da0d9-0065bb7726-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:10.365 DEBUG:boto:path=/vclfcn-64/obj3 2024-02-01T10:49:10.365 DEBUG:boto:auth_path=/vclfcn-64/obj3 2024-02-01T10:49:10.365 DEBUG:boto:path=/vclfcn-64/obj3?versionId=null 2024-02-01T10:49:10.366 DEBUG:boto:auth_path=/vclfcn-64/obj3?versionId=null 2024-02-01T10:49:10.366 DEBUG:boto:Method: GET 2024-02-01T10:49:10.366 DEBUG:boto:Path: /vclfcn-64/obj3?versionId=null 2024-02-01T10:49:10.366 DEBUG:boto:Data: 2024-02-01T10:49:10.366 DEBUG:boto:Headers: {} 2024-02-01T10:49:10.366 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:10.366 DEBUG:boto:Port: 8000 2024-02-01T10:49:10.366 DEBUG:boto:Params: {} 2024-02-01T10:49:10.366 DEBUG:boto:Token: None 2024-02-01T10:49:10.366 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:10 GMT /vclfcn-64/obj3?versionId=null 2024-02-01T10:49:10.367 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lkR2rEXRCBQBVRWDbqzy4wlPYlY= 2024-02-01T10:49:10.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lkR2rEXRCBQBVRWDbqzy4wlPYlY=', 'Content-Length': '0'} 2024-02-01T10:49:10.369 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c448b894737cb240-0065bb7726-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:10.412 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:49:10.412 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:49:10.412 DEBUG:boto:path=/vclfcn-64/?prefix=obj3 2024-02-01T10:49:10.412 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj3 2024-02-01T10:49:10.413 DEBUG:boto:Method: GET 2024-02-01T10:49:10.413 DEBUG:boto:Path: /vclfcn-64/?prefix=obj3 2024-02-01T10:49:10.413 DEBUG:boto:Data: 2024-02-01T10:49:10.413 DEBUG:boto:Headers: {} 2024-02-01T10:49:10.413 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:10.413 DEBUG:boto:Port: 8001 2024-02-01T10:49:10.413 DEBUG:boto:Params: {} 2024-02-01T10:49:10.413 DEBUG:boto:Token: None 2024-02-01T10:49:10.413 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:10 GMT /vclfcn-64/ 2024-02-01T10:49:10.414 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:f48i+ScGVYX5F7GTecJbJuACoXU= 2024-02-01T10:49:10.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:f48i+ScGVYX5F7GTecJbJuACoXU=', 'Content-Length': '0'} 2024-02-01T10:49:10.418 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c020eb76620e864-0065bb7726-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:10.419 DEBUG:boto:b'vclfcn-64obj31000falseobj32024-02-01T10:48:26.634Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:10.419 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:49:10.419 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:49:10.419 DEBUG:boto:path=/vclfcn-64/?prefix=obj3 2024-02-01T10:49:10.419 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj3 2024-02-01T10:49:10.419 DEBUG:boto:Method: GET 2024-02-01T10:49:10.420 DEBUG:boto:Path: /vclfcn-64/?prefix=obj3 2024-02-01T10:49:10.420 DEBUG:boto:Data: 2024-02-01T10:49:10.420 DEBUG:boto:Headers: {} 2024-02-01T10:49:10.420 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:10.420 DEBUG:boto:Port: 8000 2024-02-01T10:49:10.420 DEBUG:boto:Params: {} 2024-02-01T10:49:10.420 DEBUG:boto:Token: None 2024-02-01T10:49:10.420 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:10 GMT /vclfcn-64/ 2024-02-01T10:49:10.420 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:f48i+ScGVYX5F7GTecJbJuACoXU= 2024-02-01T10:49:10.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:f48i+ScGVYX5F7GTecJbJuACoXU=', 'Content-Length': '0'} 2024-02-01T10:49:10.425 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9e337ca813e927d-0065bb7726-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:10.472 DEBUG:boto:b'vclfcn-64obj31000falseobj32024-02-01T10:48:26.634Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:10.473 DEBUG:boto:path=/vclfcn-64/obj3 2024-02-01T10:49:10.473 DEBUG:boto:auth_path=/vclfcn-64/obj3 2024-02-01T10:49:10.473 DEBUG:boto:Method: HEAD 2024-02-01T10:49:10.473 DEBUG:boto:Path: /vclfcn-64/obj3 2024-02-01T10:49:10.473 DEBUG:boto:Data: 2024-02-01T10:49:10.473 DEBUG:boto:Headers: {} 2024-02-01T10:49:10.473 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:10.473 DEBUG:boto:Port: 8001 2024-02-01T10:49:10.473 DEBUG:boto:Params: {} 2024-02-01T10:49:10.474 DEBUG:boto:Token: None 2024-02-01T10:49:10.474 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:10 GMT /vclfcn-64/obj3 2024-02-01T10:49:10.474 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vqNDVETf3gyF/BZiPhT3TsYcZpo= 2024-02-01T10:49:10.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vqNDVETf3gyF/BZiPhT3TsYcZpo=', 'Content-Length': '0'} 2024-02-01T10:49:10.477 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000012da9e2dd2239849-0065bb7726-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:10.477 DEBUG:boto:path=/vclfcn-64/obj3 2024-02-01T10:49:10.477 DEBUG:boto:auth_path=/vclfcn-64/obj3 2024-02-01T10:49:10.477 DEBUG:boto:Method: HEAD 2024-02-01T10:49:10.477 DEBUG:boto:Path: /vclfcn-64/obj3 2024-02-01T10:49:10.477 DEBUG:boto:Data: 2024-02-01T10:49:10.477 DEBUG:boto:Headers: {} 2024-02-01T10:49:10.477 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:10.478 DEBUG:boto:Port: 8000 2024-02-01T10:49:10.478 DEBUG:boto:Params: {} 2024-02-01T10:49:10.478 DEBUG:boto:Token: None 2024-02-01T10:49:10.478 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:10 GMT /vclfcn-64/obj3 2024-02-01T10:49:10.478 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vqNDVETf3gyF/BZiPhT3TsYcZpo= 2024-02-01T10:49:10.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vqNDVETf3gyF/BZiPhT3TsYcZpo=', 'Content-Length': '0'} 2024-02-01T10:49:10.481 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b06d1869d30a0ecc-0065bb7726-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:10.481 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:49:10.481 DEBUG:boto:path=/vclfcn-64/obj3 2024-02-01T10:49:10.481 DEBUG:boto:auth_path=/vclfcn-64/obj3 2024-02-01T10:49:10.481 DEBUG:boto:Method: GET 2024-02-01T10:49:10.482 DEBUG:boto:Path: /vclfcn-64/obj3 2024-02-01T10:49:10.482 DEBUG:boto:Data: 2024-02-01T10:49:10.482 DEBUG:boto:Headers: {} 2024-02-01T10:49:10.482 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:10.482 DEBUG:boto:Port: 8001 2024-02-01T10:49:10.482 DEBUG:boto:Params: {} 2024-02-01T10:49:10.482 DEBUG:boto:Token: None 2024-02-01T10:49:10.482 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:10 GMT /vclfcn-64/obj3 2024-02-01T10:49:10.482 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WTvky4dMQbJe24jCqvoZur+Modo= 2024-02-01T10:49:10.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WTvky4dMQbJe24jCqvoZur+Modo=', 'Content-Length': '0'} 2024-02-01T10:49:10.485 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ab61635e6002d1cc-0065bb7726-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:10.528 DEBUG:boto:path=/vclfcn-64/obj3 2024-02-01T10:49:10.528 DEBUG:boto:auth_path=/vclfcn-64/obj3 2024-02-01T10:49:10.528 DEBUG:boto:Method: GET 2024-02-01T10:49:10.528 DEBUG:boto:Path: /vclfcn-64/obj3 2024-02-01T10:49:10.529 DEBUG:boto:Data: 2024-02-01T10:49:10.529 DEBUG:boto:Headers: {} 2024-02-01T10:49:10.529 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:10.529 DEBUG:boto:Port: 8000 2024-02-01T10:49:10.529 DEBUG:boto:Params: {} 2024-02-01T10:49:10.529 DEBUG:boto:Token: None 2024-02-01T10:49:10.529 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:10 GMT /vclfcn-64/obj3 2024-02-01T10:49:10.529 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WTvky4dMQbJe24jCqvoZur+Modo= 2024-02-01T10:49:10.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WTvky4dMQbJe24jCqvoZur+Modo=', 'Content-Length': '0'} 2024-02-01T10:49:10.533 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c3e2aaba9ce622d3-0065bb7726-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:10.576 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:49:10.576 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:49:10.576 DEBUG:boto:path=/vclfcn-64/?prefix=obj3 2024-02-01T10:49:10.576 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj3 2024-02-01T10:49:10.577 DEBUG:boto:Method: GET 2024-02-01T10:49:10.577 DEBUG:boto:Path: /vclfcn-64/?prefix=obj3 2024-02-01T10:49:10.577 DEBUG:boto:Data: 2024-02-01T10:49:10.577 DEBUG:boto:Headers: {} 2024-02-01T10:49:10.577 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:10.577 DEBUG:boto:Port: 8001 2024-02-01T10:49:10.577 DEBUG:boto:Params: {} 2024-02-01T10:49:10.577 DEBUG:boto:Token: None 2024-02-01T10:49:10.577 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:10 GMT /vclfcn-64/ 2024-02-01T10:49:10.577 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:f48i+ScGVYX5F7GTecJbJuACoXU= 2024-02-01T10:49:10.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:f48i+ScGVYX5F7GTecJbJuACoXU=', 'Content-Length': '0'} 2024-02-01T10:49:10.581 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e98ffa6ab0c6a3fe-0065bb7726-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:10.582 DEBUG:boto:b'vclfcn-64obj31000falseobj32024-02-01T10:48:26.634Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:10.582 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:49:10.582 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:49:10.582 DEBUG:boto:path=/vclfcn-64/?prefix=obj3 2024-02-01T10:49:10.582 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj3 2024-02-01T10:49:10.582 DEBUG:boto:Method: GET 2024-02-01T10:49:10.583 DEBUG:boto:Path: /vclfcn-64/?prefix=obj3 2024-02-01T10:49:10.583 DEBUG:boto:Data: 2024-02-01T10:49:10.583 DEBUG:boto:Headers: {} 2024-02-01T10:49:10.583 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:10.583 DEBUG:boto:Port: 8000 2024-02-01T10:49:10.583 DEBUG:boto:Params: {} 2024-02-01T10:49:10.583 DEBUG:boto:Token: None 2024-02-01T10:49:10.583 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:10 GMT /vclfcn-64/ 2024-02-01T10:49:10.583 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:f48i+ScGVYX5F7GTecJbJuACoXU= 2024-02-01T10:49:10.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:f48i+ScGVYX5F7GTecJbJuACoXU=', 'Content-Length': '0'} 2024-02-01T10:49:10.587 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001313165947f49796-0065bb7726-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:10.628 DEBUG:boto:b'vclfcn-64obj31000falseobj32024-02-01T10:48:26.634Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:10.628 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:49:10.629 DEBUG:boto:path=/vclfcn-64/obj4 2024-02-01T10:49:10.629 DEBUG:boto:auth_path=/vclfcn-64/obj4 2024-02-01T10:49:10.629 DEBUG:boto:path=/vclfcn-64/obj4?versionId=null 2024-02-01T10:49:10.629 DEBUG:boto:auth_path=/vclfcn-64/obj4?versionId=null 2024-02-01T10:49:10.629 DEBUG:boto:Method: GET 2024-02-01T10:49:10.629 DEBUG:boto:Path: /vclfcn-64/obj4?versionId=null 2024-02-01T10:49:10.629 DEBUG:boto:Data: 2024-02-01T10:49:10.629 DEBUG:boto:Headers: {} 2024-02-01T10:49:10.629 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:10.630 DEBUG:boto:Port: 8001 2024-02-01T10:49:10.630 DEBUG:boto:Params: {} 2024-02-01T10:49:10.630 DEBUG:boto:Token: None 2024-02-01T10:49:10.630 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:10 GMT /vclfcn-64/obj4?versionId=null 2024-02-01T10:49:10.630 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:P1yw5d8X8YvnWX4SjnYmErfWRUo= 2024-02-01T10:49:10.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:P1yw5d8X8YvnWX4SjnYmErfWRUo=', 'Content-Length': '0'} 2024-02-01T10:49:10.633 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007355d044c063417e-0065bb7726-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:10.633 DEBUG:boto:path=/vclfcn-64/obj4 2024-02-01T10:49:10.634 DEBUG:boto:auth_path=/vclfcn-64/obj4 2024-02-01T10:49:10.634 DEBUG:boto:path=/vclfcn-64/obj4?versionId=null 2024-02-01T10:49:10.634 DEBUG:boto:auth_path=/vclfcn-64/obj4?versionId=null 2024-02-01T10:49:10.634 DEBUG:boto:Method: GET 2024-02-01T10:49:10.634 DEBUG:boto:Path: /vclfcn-64/obj4?versionId=null 2024-02-01T10:49:10.634 DEBUG:boto:Data: 2024-02-01T10:49:10.634 DEBUG:boto:Headers: {} 2024-02-01T10:49:10.634 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:10.634 DEBUG:boto:Port: 8000 2024-02-01T10:49:10.634 DEBUG:boto:Params: {} 2024-02-01T10:49:10.635 DEBUG:boto:Token: None 2024-02-01T10:49:10.635 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:10 GMT /vclfcn-64/obj4?versionId=null 2024-02-01T10:49:10.635 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:P1yw5d8X8YvnWX4SjnYmErfWRUo= 2024-02-01T10:49:10.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:P1yw5d8X8YvnWX4SjnYmErfWRUo=', 'Content-Length': '0'} 2024-02-01T10:49:10.638 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000001cac91f49894372-0065bb7726-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:10.680 DEBUG:boto:path=/vclfcn-64/obj4 2024-02-01T10:49:10.680 DEBUG:boto:auth_path=/vclfcn-64/obj4 2024-02-01T10:49:10.680 DEBUG:boto:path=/vclfcn-64/obj4?versionId=null 2024-02-01T10:49:10.680 DEBUG:boto:auth_path=/vclfcn-64/obj4?versionId=null 2024-02-01T10:49:10.681 DEBUG:boto:Method: HEAD 2024-02-01T10:49:10.681 DEBUG:boto:Path: /vclfcn-64/obj4?versionId=null 2024-02-01T10:49:10.681 DEBUG:boto:Data: 2024-02-01T10:49:10.681 DEBUG:boto:Headers: {} 2024-02-01T10:49:10.681 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:10.681 DEBUG:boto:Port: 8001 2024-02-01T10:49:10.681 DEBUG:boto:Params: {} 2024-02-01T10:49:10.681 DEBUG:boto:Token: None 2024-02-01T10:49:10.682 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:10 GMT /vclfcn-64/obj4?versionId=null 2024-02-01T10:49:10.682 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:eEJpT3+K0OQwXw8C1wLdK68f3Sc= 2024-02-01T10:49:10.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:eEJpT3+K0OQwXw8C1wLdK68f3Sc=', 'Content-Length': '0'} 2024-02-01T10:49:10.685 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003355b262310e6e47-0065bb7726-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:10.685 DEBUG:boto:path=/vclfcn-64/obj4 2024-02-01T10:49:10.685 DEBUG:boto:auth_path=/vclfcn-64/obj4 2024-02-01T10:49:10.685 DEBUG:boto:path=/vclfcn-64/obj4?versionId=null 2024-02-01T10:49:10.685 DEBUG:boto:auth_path=/vclfcn-64/obj4?versionId=null 2024-02-01T10:49:10.686 DEBUG:boto:Method: HEAD 2024-02-01T10:49:10.686 DEBUG:boto:Path: /vclfcn-64/obj4?versionId=null 2024-02-01T10:49:10.686 DEBUG:boto:Data: 2024-02-01T10:49:10.686 DEBUG:boto:Headers: {} 2024-02-01T10:49:10.686 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:10.686 DEBUG:boto:Port: 8000 2024-02-01T10:49:10.686 DEBUG:boto:Params: {} 2024-02-01T10:49:10.686 DEBUG:boto:Token: None 2024-02-01T10:49:10.686 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:10 GMT /vclfcn-64/obj4?versionId=null 2024-02-01T10:49:10.687 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:eEJpT3+K0OQwXw8C1wLdK68f3Sc= 2024-02-01T10:49:10.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:eEJpT3+K0OQwXw8C1wLdK68f3Sc=', 'Content-Length': '0'} 2024-02-01T10:49:10.689 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a5904f9fb478599e-0065bb7726-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:10.690 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:49:10.690 DEBUG:boto:path=/vclfcn-64/obj4 2024-02-01T10:49:10.690 DEBUG:boto:auth_path=/vclfcn-64/obj4 2024-02-01T10:49:10.690 DEBUG:boto:path=/vclfcn-64/obj4?versionId=null 2024-02-01T10:49:10.690 DEBUG:boto:auth_path=/vclfcn-64/obj4?versionId=null 2024-02-01T10:49:10.690 DEBUG:boto:Method: GET 2024-02-01T10:49:10.690 DEBUG:boto:Path: /vclfcn-64/obj4?versionId=null 2024-02-01T10:49:10.691 DEBUG:boto:Data: 2024-02-01T10:49:10.691 DEBUG:boto:Headers: {} 2024-02-01T10:49:10.691 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:10.691 DEBUG:boto:Port: 8001 2024-02-01T10:49:10.691 DEBUG:boto:Params: {} 2024-02-01T10:49:10.691 DEBUG:boto:Token: None 2024-02-01T10:49:10.691 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:10 GMT /vclfcn-64/obj4?versionId=null 2024-02-01T10:49:10.691 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:P1yw5d8X8YvnWX4SjnYmErfWRUo= 2024-02-01T10:49:10.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:P1yw5d8X8YvnWX4SjnYmErfWRUo=', 'Content-Length': '0'} 2024-02-01T10:49:10.694 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008ca45dad4374f6cf-0065bb7726-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:10.695 DEBUG:boto:path=/vclfcn-64/obj4 2024-02-01T10:49:10.696 DEBUG:boto:auth_path=/vclfcn-64/obj4 2024-02-01T10:49:10.696 DEBUG:boto:path=/vclfcn-64/obj4?versionId=null 2024-02-01T10:49:10.696 DEBUG:boto:auth_path=/vclfcn-64/obj4?versionId=null 2024-02-01T10:49:10.696 DEBUG:boto:Method: GET 2024-02-01T10:49:10.696 DEBUG:boto:Path: /vclfcn-64/obj4?versionId=null 2024-02-01T10:49:10.696 DEBUG:boto:Data: 2024-02-01T10:49:10.697 DEBUG:boto:Headers: {} 2024-02-01T10:49:10.697 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:10.697 DEBUG:boto:Port: 8000 2024-02-01T10:49:10.697 DEBUG:boto:Params: {} 2024-02-01T10:49:10.697 DEBUG:boto:Token: None 2024-02-01T10:49:10.697 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:10 GMT /vclfcn-64/obj4?versionId=null 2024-02-01T10:49:10.697 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:P1yw5d8X8YvnWX4SjnYmErfWRUo= 2024-02-01T10:49:10.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:P1yw5d8X8YvnWX4SjnYmErfWRUo=', 'Content-Length': '0'} 2024-02-01T10:49:10.700 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000006f004c7d3d3c0ee-0065bb7726-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:10.744 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:49:10.744 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:49:10.744 DEBUG:boto:path=/vclfcn-64/?prefix=obj4 2024-02-01T10:49:10.745 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj4 2024-02-01T10:49:10.745 DEBUG:boto:Method: GET 2024-02-01T10:49:10.745 DEBUG:boto:Path: /vclfcn-64/?prefix=obj4 2024-02-01T10:49:10.745 DEBUG:boto:Data: 2024-02-01T10:49:10.745 DEBUG:boto:Headers: {} 2024-02-01T10:49:10.745 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:10.745 DEBUG:boto:Port: 8001 2024-02-01T10:49:10.745 DEBUG:boto:Params: {} 2024-02-01T10:49:10.745 DEBUG:boto:Token: None 2024-02-01T10:49:10.745 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:10 GMT /vclfcn-64/ 2024-02-01T10:49:10.746 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:f48i+ScGVYX5F7GTecJbJuACoXU= 2024-02-01T10:49:10.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:f48i+ScGVYX5F7GTecJbJuACoXU=', 'Content-Length': '0'} 2024-02-01T10:49:10.751 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094641f62a102ddb9-0065bb7726-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:10.752 DEBUG:boto:b'vclfcn-64obj41000falseobj42024-02-01T10:48:26.689Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:10.752 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:49:10.752 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:49:10.753 DEBUG:boto:path=/vclfcn-64/?prefix=obj4 2024-02-01T10:49:10.753 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj4 2024-02-01T10:49:10.753 DEBUG:boto:Method: GET 2024-02-01T10:49:10.753 DEBUG:boto:Path: /vclfcn-64/?prefix=obj4 2024-02-01T10:49:10.753 DEBUG:boto:Data: 2024-02-01T10:49:10.753 DEBUG:boto:Headers: {} 2024-02-01T10:49:10.753 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:10.753 DEBUG:boto:Port: 8000 2024-02-01T10:49:10.753 DEBUG:boto:Params: {} 2024-02-01T10:49:10.753 DEBUG:boto:Token: None 2024-02-01T10:49:10.754 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:10 GMT /vclfcn-64/ 2024-02-01T10:49:10.754 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:f48i+ScGVYX5F7GTecJbJuACoXU= 2024-02-01T10:49:10.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:f48i+ScGVYX5F7GTecJbJuACoXU=', 'Content-Length': '0'} 2024-02-01T10:49:10.759 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a4f9121e24474d14-0065bb7726-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:10.800 DEBUG:boto:b'vclfcn-64obj41000falseobj42024-02-01T10:48:26.689Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:10.800 DEBUG:boto:path=/vclfcn-64/obj4 2024-02-01T10:49:10.801 DEBUG:boto:auth_path=/vclfcn-64/obj4 2024-02-01T10:49:10.801 DEBUG:boto:Method: HEAD 2024-02-01T10:49:10.801 DEBUG:boto:Path: /vclfcn-64/obj4 2024-02-01T10:49:10.801 DEBUG:boto:Data: 2024-02-01T10:49:10.801 DEBUG:boto:Headers: {} 2024-02-01T10:49:10.801 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:10.801 DEBUG:boto:Port: 8001 2024-02-01T10:49:10.801 DEBUG:boto:Params: {} 2024-02-01T10:49:10.801 DEBUG:boto:Token: None 2024-02-01T10:49:10.801 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:10 GMT /vclfcn-64/obj4 2024-02-01T10:49:10.802 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:pNEyKq+ri4/cGRtmeKCQYuP4TtI= 2024-02-01T10:49:10.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:pNEyKq+ri4/cGRtmeKCQYuP4TtI=', 'Content-Length': '0'} 2024-02-01T10:49:10.805 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a900d994c9989cf7-0065bb7726-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:10.805 DEBUG:boto:path=/vclfcn-64/obj4 2024-02-01T10:49:10.805 DEBUG:boto:auth_path=/vclfcn-64/obj4 2024-02-01T10:49:10.806 DEBUG:boto:Method: HEAD 2024-02-01T10:49:10.806 DEBUG:boto:Path: /vclfcn-64/obj4 2024-02-01T10:49:10.806 DEBUG:boto:Data: 2024-02-01T10:49:10.806 DEBUG:boto:Headers: {} 2024-02-01T10:49:10.806 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:10.806 DEBUG:boto:Port: 8000 2024-02-01T10:49:10.806 DEBUG:boto:Params: {} 2024-02-01T10:49:10.806 DEBUG:boto:Token: None 2024-02-01T10:49:10.806 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:10 GMT /vclfcn-64/obj4 2024-02-01T10:49:10.806 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:pNEyKq+ri4/cGRtmeKCQYuP4TtI= 2024-02-01T10:49:10.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:pNEyKq+ri4/cGRtmeKCQYuP4TtI=', 'Content-Length': '0'} 2024-02-01T10:49:10.809 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000250de0cc500c0845-0065bb7726-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:10.809 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:49:10.809 DEBUG:boto:path=/vclfcn-64/obj4 2024-02-01T10:49:10.810 DEBUG:boto:auth_path=/vclfcn-64/obj4 2024-02-01T10:49:10.810 DEBUG:boto:Method: GET 2024-02-01T10:49:10.810 DEBUG:boto:Path: /vclfcn-64/obj4 2024-02-01T10:49:10.810 DEBUG:boto:Data: 2024-02-01T10:49:10.810 DEBUG:boto:Headers: {} 2024-02-01T10:49:10.810 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:10.810 DEBUG:boto:Port: 8001 2024-02-01T10:49:10.810 DEBUG:boto:Params: {} 2024-02-01T10:49:10.811 DEBUG:boto:Token: None 2024-02-01T10:49:10.811 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:10 GMT /vclfcn-64/obj4 2024-02-01T10:49:10.811 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WsZsDv1HREGoOAf3SN7kvZAJnr4= 2024-02-01T10:49:10.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WsZsDv1HREGoOAf3SN7kvZAJnr4=', 'Content-Length': '0'} 2024-02-01T10:49:10.814 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004f027ec01d963b23-0065bb7726-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:10.814 DEBUG:boto:path=/vclfcn-64/obj4 2024-02-01T10:49:10.814 DEBUG:boto:auth_path=/vclfcn-64/obj4 2024-02-01T10:49:10.814 DEBUG:boto:Method: GET 2024-02-01T10:49:10.814 DEBUG:boto:Path: /vclfcn-64/obj4 2024-02-01T10:49:10.815 DEBUG:boto:Data: 2024-02-01T10:49:10.815 DEBUG:boto:Headers: {} 2024-02-01T10:49:10.815 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:10.815 DEBUG:boto:Port: 8000 2024-02-01T10:49:10.815 DEBUG:boto:Params: {} 2024-02-01T10:49:10.815 DEBUG:boto:Token: None 2024-02-01T10:49:10.815 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:10 GMT /vclfcn-64/obj4 2024-02-01T10:49:10.815 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WsZsDv1HREGoOAf3SN7kvZAJnr4= 2024-02-01T10:49:10.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WsZsDv1HREGoOAf3SN7kvZAJnr4=', 'Content-Length': '0'} 2024-02-01T10:49:10.818 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000e9ef41010d70d45-0065bb7726-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:10.860 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:49:10.860 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:49:10.860 DEBUG:boto:path=/vclfcn-64/?prefix=obj4 2024-02-01T10:49:10.860 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj4 2024-02-01T10:49:10.861 DEBUG:boto:Method: GET 2024-02-01T10:49:10.861 DEBUG:boto:Path: /vclfcn-64/?prefix=obj4 2024-02-01T10:49:10.861 DEBUG:boto:Data: 2024-02-01T10:49:10.861 DEBUG:boto:Headers: {} 2024-02-01T10:49:10.861 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:10.861 DEBUG:boto:Port: 8001 2024-02-01T10:49:10.861 DEBUG:boto:Params: {} 2024-02-01T10:49:10.861 DEBUG:boto:Token: None 2024-02-01T10:49:10.861 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:10 GMT /vclfcn-64/ 2024-02-01T10:49:10.862 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:f48i+ScGVYX5F7GTecJbJuACoXU= 2024-02-01T10:49:10.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:f48i+ScGVYX5F7GTecJbJuACoXU=', 'Content-Length': '0'} 2024-02-01T10:49:10.866 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000000ab141dbcbf2b5c-0065bb7726-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:10.866 DEBUG:boto:b'vclfcn-64obj41000falseobj42024-02-01T10:48:26.689Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:10.867 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:49:10.867 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:49:10.867 DEBUG:boto:path=/vclfcn-64/?prefix=obj4 2024-02-01T10:49:10.867 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj4 2024-02-01T10:49:10.867 DEBUG:boto:Method: GET 2024-02-01T10:49:10.867 DEBUG:boto:Path: /vclfcn-64/?prefix=obj4 2024-02-01T10:49:10.867 DEBUG:boto:Data: 2024-02-01T10:49:10.868 DEBUG:boto:Headers: {} 2024-02-01T10:49:10.868 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:10.868 DEBUG:boto:Port: 8000 2024-02-01T10:49:10.868 DEBUG:boto:Params: {} 2024-02-01T10:49:10.868 DEBUG:boto:Token: None 2024-02-01T10:49:10.868 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:10 GMT /vclfcn-64/ 2024-02-01T10:49:10.868 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:f48i+ScGVYX5F7GTecJbJuACoXU= 2024-02-01T10:49:10.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:f48i+ScGVYX5F7GTecJbJuACoXU=', 'Content-Length': '0'} 2024-02-01T10:49:10.872 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072bca3aab2494cb7-0065bb7726-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:10.916 DEBUG:boto:b'vclfcn-64obj41000falseobj42024-02-01T10:48:26.689Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:10.916 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-64 zones={test-zone3, test-zone2} 2024-02-01T10:49:10.917 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'vclfcn-62', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:49:10.917 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'vclfcn-62', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:49:10.917 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket vclfcn-62 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-02-01T10:49:10.961 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:49:10.961 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:49:11.070 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'vclfcn-63', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:49:11.070 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'vclfcn-63', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:49:11.071 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket vclfcn-63 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-02-01T10:49:11.151 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:49:11.151 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:49:11.262 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'vclfcn-64', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:49:11.262 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'vclfcn-64', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:49:11.262 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket vclfcn-64 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-02-01T10:49:11.339 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:49:11.339 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:49:11.445 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:49:11.445 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:49:11.445 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:49:11.523 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:49:11.523 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:49:11.576 INFO:teuthology.orchestra.run.smithi148.stderr:No --period given, using current period=f2e6c290-e7aa-4279-b77e-15403b18d61f 2024-02-01T10:49:11.600 INFO:teuthology.orchestra.run.smithi148.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1706784551.423106_315.1","last_update":"2024-02-01T10:49:11.423106Z"},{"marker":"","last_update":"0.000000"}] 2024-02-01T10:49:11.601 DEBUG:rgw_multi.tests:master meta markers={0: '', 1: '', 2: '1_1706784551.423106_315.1', 3: ''} 2024-02-01T10:49:11.601 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-02-01T10:49:11.601 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:49:11.601 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:49:11.601 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:49:11.645 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:49:11.645 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:49:11.700 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:49:11.697+0000 7fa95c95d980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:49:11.701 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:49:11.697+0000 7fa95c95d980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:49:11.725 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706784506.281191_286.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:48:26.281191Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:49:11Z"} 2024-02-01T10:49:11.726 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706784506.281191_286.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:48:26.281191Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:49:11Z"} 2024-02-01T10:49:11.726 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706784506.281191_286.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:48:26.281191Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:49:11.726 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1706784551.423106_315.1', 3: ''} 2024-02-01T10:49:11.726 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1706784506.281191_286.1', 3: ''} 2024-02-01T10:49:11.726 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1706784551.423106_315.1 target=1_1706784506.281191_286.1 2024-02-01T10:49:16.727 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:49:16.727 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:49:16.728 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:49:16.774 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:49:16.774 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:49:16.833 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:49:16.828+0000 7fd6a0378980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:49:16.834 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:49:16.828+0000 7fd6a0378980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:49:16.858 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706784551.423106_315.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:49:11.423106Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:49:16Z"} 2024-02-01T10:49:16.859 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706784551.423106_315.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:49:11.423106Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:49:16Z"} 2024-02-01T10:49:16.859 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706784551.423106_315.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:49:11.423106Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:49:16.859 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1706784551.423106_315.1', 3: ''} 2024-02-01T10:49:16.859 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1706784551.423106_315.1', 3: ''} 2024-02-01T10:49:16.860 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-02-01T10:49:16.860 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-02-01T10:49:16.860 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:49:16.860 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:49:16.860 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:49:16.903 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:49:16.903 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:49:16.959 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:49:16.955+0000 7f4768c2c980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:49:16.959 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:49:16.959+0000 7f4768c2c980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:49:16.983 INFO:teuthology.orchestra.run.smithi148.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706784506.281191_286.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:48:26.281191Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:49:16Z"} 2024-02-01T10:49:16.983 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706784506.281191_286.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:48:26.281191Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:49:16Z"} 2024-02-01T10:49:16.984 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706784506.281191_286.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:48:26.281191Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:49:16.984 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1706784551.423106_315.1', 3: ''} 2024-02-01T10:49:16.984 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1706784506.281191_286.1', 3: ''} 2024-02-01T10:49:16.984 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1706784551.423106_315.1 target=1_1706784506.281191_286.1 2024-02-01T10:49:21.985 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:49:21.986 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:49:21.986 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:49:22.028 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:49:22.028 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:49:22.087 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:49:22.083+0000 7f7ee971e980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:49:22.088 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:49:22.087+0000 7f7ee971e980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:49:22.111 INFO:teuthology.orchestra.run.smithi148.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706784551.423106_315.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:49:11.423106Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:49:22Z"} 2024-02-01T10:49:22.112 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706784551.423106_315.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:49:11.423106Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:49:22Z"} 2024-02-01T10:49:22.112 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706784551.423106_315.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:49:11.423106Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:49:22.112 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1706784551.423106_315.1', 3: ''} 2024-02-01T10:49:22.113 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1706784551.423106_315.1', 3: ''} 2024-02-01T10:49:22.113 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-02-01T10:49:22.113 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:22.113 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:22.113 DEBUG:boto:Method: HEAD 2024-02-01T10:49:22.113 DEBUG:boto:Path: /vclfcn-62/ 2024-02-01T10:49:22.113 DEBUG:boto:Data: 2024-02-01T10:49:22.114 DEBUG:boto:Headers: {} 2024-02-01T10:49:22.114 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:22.114 DEBUG:boto:Port: 8000 2024-02-01T10:49:22.114 DEBUG:boto:Params: {} 2024-02-01T10:49:22.114 DEBUG:boto:Token: None 2024-02-01T10:49:22.115 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:22 GMT /vclfcn-62/ 2024-02-01T10:49:22.115 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Qeypx52YbNtwHxYW1ZxiNJ1I2gM= 2024-02-01T10:49:22.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Qeypx52YbNtwHxYW1ZxiNJ1I2gM=', 'Content-Length': '0'} 2024-02-01T10:49:22.117 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000708bcd4b1278745d-0065bb7732-5765-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:22.118 DEBUG:boto:path=/vclfcn-62/obj5 2024-02-01T10:49:22.118 DEBUG:boto:auth_path=/vclfcn-62/obj5 2024-02-01T10:49:22.118 DEBUG:boto:Method: PUT 2024-02-01T10:49:22.118 DEBUG:boto:Path: /vclfcn-62/obj5 2024-02-01T10:49:22.118 DEBUG:boto:Data: 2024-02-01T10:49:22.118 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-02-01T10:49:22.118 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:22.118 DEBUG:boto:Port: 8000 2024-02-01T10:49:22.118 DEBUG:boto:Params: {} 2024-02-01T10:49:22.119 DEBUG:boto:Token: None 2024-02-01T10:49:22.119 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 01 Feb 2024 10:49:22 GMT /vclfcn-62/obj5 2024-02-01T10:49:22.119 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+1Ls7OCTj9HxMlf1H1QUr02h/h0= 2024-02-01T10:49:22.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:49:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+1Ls7OCTj9HxMlf1H1QUr02h/h0='} 2024-02-01T10:49:22.124 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e32e9f8a8721b8b3-0065bb7732-5765-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:22.125 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:22.125 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:22.125 DEBUG:boto:Method: HEAD 2024-02-01T10:49:22.125 DEBUG:boto:Path: /vclfcn-62/ 2024-02-01T10:49:22.125 DEBUG:boto:Data: 2024-02-01T10:49:22.125 DEBUG:boto:Headers: {} 2024-02-01T10:49:22.125 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:22.126 DEBUG:boto:Port: 8000 2024-02-01T10:49:22.126 DEBUG:boto:Params: {} 2024-02-01T10:49:22.126 DEBUG:boto:Token: None 2024-02-01T10:49:22.126 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:22 GMT /vclfcn-62/ 2024-02-01T10:49:22.126 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Qeypx52YbNtwHxYW1ZxiNJ1I2gM= 2024-02-01T10:49:22.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Qeypx52YbNtwHxYW1ZxiNJ1I2gM=', 'Content-Length': '0'} 2024-02-01T10:49:22.129 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bd77518ad33a82b6-0065bb7732-5765-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:22.129 DEBUG:boto:path=/vclfcn-62/obj6 2024-02-01T10:49:22.129 DEBUG:boto:auth_path=/vclfcn-62/obj6 2024-02-01T10:49:22.129 DEBUG:boto:Method: PUT 2024-02-01T10:49:22.129 DEBUG:boto:Path: /vclfcn-62/obj6 2024-02-01T10:49:22.129 DEBUG:boto:Data: 2024-02-01T10:49:22.129 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-02-01T10:49:22.130 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:22.130 DEBUG:boto:Port: 8000 2024-02-01T10:49:22.130 DEBUG:boto:Params: {} 2024-02-01T10:49:22.130 DEBUG:boto:Token: None 2024-02-01T10:49:22.130 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 01 Feb 2024 10:49:22 GMT /vclfcn-62/obj6 2024-02-01T10:49:22.131 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Ya0WF0ppUpJeHUE7ppKRtxVnAyU= 2024-02-01T10:49:22.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:49:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Ya0WF0ppUpJeHUE7ppKRtxVnAyU='} 2024-02-01T10:49:22.176 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009d393d123cce8ba9-0065bb7732-5765-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:22.176 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:22.177 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:22.177 DEBUG:boto:Method: HEAD 2024-02-01T10:49:22.177 DEBUG:boto:Path: /vclfcn-62/ 2024-02-01T10:49:22.177 DEBUG:boto:Data: 2024-02-01T10:49:22.177 DEBUG:boto:Headers: {} 2024-02-01T10:49:22.177 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:22.177 DEBUG:boto:Port: 8000 2024-02-01T10:49:22.177 DEBUG:boto:Params: {} 2024-02-01T10:49:22.177 DEBUG:boto:Token: None 2024-02-01T10:49:22.178 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:22 GMT /vclfcn-62/ 2024-02-01T10:49:22.178 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Qeypx52YbNtwHxYW1ZxiNJ1I2gM= 2024-02-01T10:49:22.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Qeypx52YbNtwHxYW1ZxiNJ1I2gM=', 'Content-Length': '0'} 2024-02-01T10:49:22.180 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000754c5b07faa928df-0065bb7732-5765-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:22.181 DEBUG:boto:path=/vclfcn-62/obj7 2024-02-01T10:49:22.181 DEBUG:boto:auth_path=/vclfcn-62/obj7 2024-02-01T10:49:22.181 DEBUG:boto:Method: PUT 2024-02-01T10:49:22.181 DEBUG:boto:Path: /vclfcn-62/obj7 2024-02-01T10:49:22.181 DEBUG:boto:Data: 2024-02-01T10:49:22.181 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-02-01T10:49:22.181 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:22.181 DEBUG:boto:Port: 8000 2024-02-01T10:49:22.182 DEBUG:boto:Params: {} 2024-02-01T10:49:22.182 DEBUG:boto:Token: None 2024-02-01T10:49:22.182 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 01 Feb 2024 10:49:22 GMT /vclfcn-62/obj7 2024-02-01T10:49:22.182 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vBv5mbj8oB7fVhfKc0R96o3tMME= 2024-02-01T10:49:22.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:49:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vBv5mbj8oB7fVhfKc0R96o3tMME='} 2024-02-01T10:49:22.190 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003f4e5129e02cb85f-0065bb7732-5765-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:22.190 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:22.190 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:22.190 DEBUG:boto:Method: HEAD 2024-02-01T10:49:22.190 DEBUG:boto:Path: /vclfcn-62/ 2024-02-01T10:49:22.190 DEBUG:boto:Data: 2024-02-01T10:49:22.190 DEBUG:boto:Headers: {} 2024-02-01T10:49:22.190 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:22.190 DEBUG:boto:Port: 8000 2024-02-01T10:49:22.191 DEBUG:boto:Params: {} 2024-02-01T10:49:22.191 DEBUG:boto:Token: None 2024-02-01T10:49:22.191 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:22 GMT /vclfcn-62/ 2024-02-01T10:49:22.191 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Qeypx52YbNtwHxYW1ZxiNJ1I2gM= 2024-02-01T10:49:22.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Qeypx52YbNtwHxYW1ZxiNJ1I2gM=', 'Content-Length': '0'} 2024-02-01T10:49:22.194 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b67c6cde6bb01c30-0065bb7732-5765-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:22.194 DEBUG:boto:path=/vclfcn-62/obj8 2024-02-01T10:49:22.194 DEBUG:boto:auth_path=/vclfcn-62/obj8 2024-02-01T10:49:22.195 DEBUG:boto:Method: PUT 2024-02-01T10:49:22.195 DEBUG:boto:Path: /vclfcn-62/obj8 2024-02-01T10:49:22.195 DEBUG:boto:Data: 2024-02-01T10:49:22.195 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-02-01T10:49:22.195 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:22.195 DEBUG:boto:Port: 8000 2024-02-01T10:49:22.195 DEBUG:boto:Params: {} 2024-02-01T10:49:22.195 DEBUG:boto:Token: None 2024-02-01T10:49:22.195 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 01 Feb 2024 10:49:22 GMT /vclfcn-62/obj8 2024-02-01T10:49:22.195 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Hryjm9e2UiVynfdXnqadqj0cEYo= 2024-02-01T10:49:22.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:49:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Hryjm9e2UiVynfdXnqadqj0cEYo='} 2024-02-01T10:49:22.240 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000031eb776dec2e864c-0065bb7732-5765-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:22.241 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:49:22.241 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:49:22.241 DEBUG:boto:Method: HEAD 2024-02-01T10:49:22.241 DEBUG:boto:Path: /vclfcn-63/ 2024-02-01T10:49:22.241 DEBUG:boto:Data: 2024-02-01T10:49:22.241 DEBUG:boto:Headers: {} 2024-02-01T10:49:22.241 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:22.242 DEBUG:boto:Port: 8000 2024-02-01T10:49:22.242 DEBUG:boto:Params: {} 2024-02-01T10:49:22.242 DEBUG:boto:Token: None 2024-02-01T10:49:22.242 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:22 GMT /vclfcn-63/ 2024-02-01T10:49:22.242 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fGPt44/b6uDX+0Az7hEhXA9RCYk= 2024-02-01T10:49:22.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fGPt44/b6uDX+0Az7hEhXA9RCYk=', 'Content-Length': '0'} 2024-02-01T10:49:22.247 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c2b6380c875df153-0065bb7732-5350-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:22.248 DEBUG:boto:path=/vclfcn-63/obj5 2024-02-01T10:49:22.248 DEBUG:boto:auth_path=/vclfcn-63/obj5 2024-02-01T10:49:22.248 DEBUG:boto:Method: PUT 2024-02-01T10:49:22.248 DEBUG:boto:Path: /vclfcn-63/obj5 2024-02-01T10:49:22.249 DEBUG:boto:Data: 2024-02-01T10:49:22.249 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-02-01T10:49:22.249 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:22.249 DEBUG:boto:Port: 8000 2024-02-01T10:49:22.249 DEBUG:boto:Params: {} 2024-02-01T10:49:22.250 DEBUG:boto:Token: None 2024-02-01T10:49:22.250 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 01 Feb 2024 10:49:22 GMT /vclfcn-63/obj5 2024-02-01T10:49:22.250 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Wr18a4Yh9bRxRNLCMYxOii+/Zxw= 2024-02-01T10:49:22.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:49:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Wr18a4Yh9bRxRNLCMYxOii+/Zxw='} 2024-02-01T10:49:22.259 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000037cd834d4b995970-0065bb7732-5350-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:22.260 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:49:22.260 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:49:22.260 DEBUG:boto:Method: HEAD 2024-02-01T10:49:22.260 DEBUG:boto:Path: /vclfcn-63/ 2024-02-01T10:49:22.261 DEBUG:boto:Data: 2024-02-01T10:49:22.261 DEBUG:boto:Headers: {} 2024-02-01T10:49:22.261 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:22.261 DEBUG:boto:Port: 8000 2024-02-01T10:49:22.261 DEBUG:boto:Params: {} 2024-02-01T10:49:22.262 DEBUG:boto:Token: None 2024-02-01T10:49:22.262 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:22 GMT /vclfcn-63/ 2024-02-01T10:49:22.262 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fGPt44/b6uDX+0Az7hEhXA9RCYk= 2024-02-01T10:49:22.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fGPt44/b6uDX+0Az7hEhXA9RCYk=', 'Content-Length': '0'} 2024-02-01T10:49:22.266 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e52947025d4849c5-0065bb7732-5350-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:22.266 DEBUG:boto:path=/vclfcn-63/obj6 2024-02-01T10:49:22.266 DEBUG:boto:auth_path=/vclfcn-63/obj6 2024-02-01T10:49:22.267 DEBUG:boto:Method: PUT 2024-02-01T10:49:22.267 DEBUG:boto:Path: /vclfcn-63/obj6 2024-02-01T10:49:22.267 DEBUG:boto:Data: 2024-02-01T10:49:22.267 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-02-01T10:49:22.267 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:22.267 DEBUG:boto:Port: 8000 2024-02-01T10:49:22.267 DEBUG:boto:Params: {} 2024-02-01T10:49:22.267 DEBUG:boto:Token: None 2024-02-01T10:49:22.267 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 01 Feb 2024 10:49:22 GMT /vclfcn-63/obj6 2024-02-01T10:49:22.268 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:LNLyfYYS8wILugf+2HbTjA0/PNY= 2024-02-01T10:49:22.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:49:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:LNLyfYYS8wILugf+2HbTjA0/PNY='} 2024-02-01T10:49:22.316 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000084fbe2a7d5cd8edd-0065bb7732-5350-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:22.317 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:49:22.317 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:49:22.317 DEBUG:boto:Method: HEAD 2024-02-01T10:49:22.317 DEBUG:boto:Path: /vclfcn-63/ 2024-02-01T10:49:22.317 DEBUG:boto:Data: 2024-02-01T10:49:22.317 DEBUG:boto:Headers: {} 2024-02-01T10:49:22.317 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:22.317 DEBUG:boto:Port: 8000 2024-02-01T10:49:22.317 DEBUG:boto:Params: {} 2024-02-01T10:49:22.318 DEBUG:boto:Token: None 2024-02-01T10:49:22.318 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:22 GMT /vclfcn-63/ 2024-02-01T10:49:22.318 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fGPt44/b6uDX+0Az7hEhXA9RCYk= 2024-02-01T10:49:22.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fGPt44/b6uDX+0Az7hEhXA9RCYk=', 'Content-Length': '0'} 2024-02-01T10:49:22.322 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f3a9a9fcc10b6634-0065bb7732-5350-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:22.322 DEBUG:boto:path=/vclfcn-63/obj7 2024-02-01T10:49:22.322 DEBUG:boto:auth_path=/vclfcn-63/obj7 2024-02-01T10:49:22.322 DEBUG:boto:Method: PUT 2024-02-01T10:49:22.323 DEBUG:boto:Path: /vclfcn-63/obj7 2024-02-01T10:49:22.323 DEBUG:boto:Data: 2024-02-01T10:49:22.323 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-02-01T10:49:22.323 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:22.323 DEBUG:boto:Port: 8000 2024-02-01T10:49:22.323 DEBUG:boto:Params: {} 2024-02-01T10:49:22.323 DEBUG:boto:Token: None 2024-02-01T10:49:22.323 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 01 Feb 2024 10:49:22 GMT /vclfcn-63/obj7 2024-02-01T10:49:22.323 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZMAEaHWHJ6IRNKJD/pKE5TCz9WY= 2024-02-01T10:49:22.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:49:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZMAEaHWHJ6IRNKJD/pKE5TCz9WY='} 2024-02-01T10:49:22.332 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001349fbd56f4161a6-0065bb7732-5350-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:22.332 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:49:22.332 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:49:22.332 DEBUG:boto:Method: HEAD 2024-02-01T10:49:22.332 DEBUG:boto:Path: /vclfcn-63/ 2024-02-01T10:49:22.333 DEBUG:boto:Data: 2024-02-01T10:49:22.333 DEBUG:boto:Headers: {} 2024-02-01T10:49:22.333 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:22.333 DEBUG:boto:Port: 8000 2024-02-01T10:49:22.333 DEBUG:boto:Params: {} 2024-02-01T10:49:22.333 DEBUG:boto:Token: None 2024-02-01T10:49:22.333 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:22 GMT /vclfcn-63/ 2024-02-01T10:49:22.333 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fGPt44/b6uDX+0Az7hEhXA9RCYk= 2024-02-01T10:49:22.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fGPt44/b6uDX+0Az7hEhXA9RCYk=', 'Content-Length': '0'} 2024-02-01T10:49:22.338 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000085b0cb821d40ae40-0065bb7732-5350-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:22.338 DEBUG:boto:path=/vclfcn-63/obj8 2024-02-01T10:49:22.338 DEBUG:boto:auth_path=/vclfcn-63/obj8 2024-02-01T10:49:22.338 DEBUG:boto:Method: PUT 2024-02-01T10:49:22.338 DEBUG:boto:Path: /vclfcn-63/obj8 2024-02-01T10:49:22.338 DEBUG:boto:Data: 2024-02-01T10:49:22.338 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-02-01T10:49:22.339 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:22.339 DEBUG:boto:Port: 8000 2024-02-01T10:49:22.339 DEBUG:boto:Params: {} 2024-02-01T10:49:22.339 DEBUG:boto:Token: None 2024-02-01T10:49:22.339 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 01 Feb 2024 10:49:22 GMT /vclfcn-63/obj8 2024-02-01T10:49:22.339 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7woTjqQUx1naqY637kGBc9viUIw= 2024-02-01T10:49:22.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:49:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7woTjqQUx1naqY637kGBc9viUIw='} 2024-02-01T10:49:22.388 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c8668cce67b5bd3d-0065bb7732-5350-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:22.388 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:49:22.389 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:49:22.389 DEBUG:boto:Method: HEAD 2024-02-01T10:49:22.389 DEBUG:boto:Path: /vclfcn-64/ 2024-02-01T10:49:22.389 DEBUG:boto:Data: 2024-02-01T10:49:22.390 DEBUG:boto:Headers: {} 2024-02-01T10:49:22.390 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:22.390 DEBUG:boto:Port: 8001 2024-02-01T10:49:22.390 DEBUG:boto:Params: {} 2024-02-01T10:49:22.390 DEBUG:boto:Token: None 2024-02-01T10:49:22.391 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:22 GMT /vclfcn-64/ 2024-02-01T10:49:22.391 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+GO4Tk7lzqac0juuVOO1dVYJwAo= 2024-02-01T10:49:22.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+GO4Tk7lzqac0juuVOO1dVYJwAo=', 'Content-Length': '0'} 2024-02-01T10:49:22.396 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000446d3f58aeffe5dd-0065bb7732-5762-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:22.396 DEBUG:boto:path=/vclfcn-64/obj5 2024-02-01T10:49:22.396 DEBUG:boto:auth_path=/vclfcn-64/obj5 2024-02-01T10:49:22.396 DEBUG:boto:Method: PUT 2024-02-01T10:49:22.397 DEBUG:boto:Path: /vclfcn-64/obj5 2024-02-01T10:49:22.397 DEBUG:boto:Data: 2024-02-01T10:49:22.397 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-02-01T10:49:22.397 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:22.397 DEBUG:boto:Port: 8001 2024-02-01T10:49:22.397 DEBUG:boto:Params: {} 2024-02-01T10:49:22.397 DEBUG:boto:Token: None 2024-02-01T10:49:22.397 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 01 Feb 2024 10:49:22 GMT /vclfcn-64/obj5 2024-02-01T10:49:22.398 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:C6nv/SENjXWaQFirATdG3vRy/F8= 2024-02-01T10:49:22.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:49:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:C6nv/SENjXWaQFirATdG3vRy/F8='} 2024-02-01T10:49:22.442 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ebf16d983ee23a16-0065bb7732-5762-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:22.442 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:49:22.442 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:49:22.443 DEBUG:boto:Method: HEAD 2024-02-01T10:49:22.443 DEBUG:boto:Path: /vclfcn-64/ 2024-02-01T10:49:22.443 DEBUG:boto:Data: 2024-02-01T10:49:22.443 DEBUG:boto:Headers: {} 2024-02-01T10:49:22.443 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:22.443 DEBUG:boto:Port: 8001 2024-02-01T10:49:22.443 DEBUG:boto:Params: {} 2024-02-01T10:49:22.443 DEBUG:boto:Token: None 2024-02-01T10:49:22.444 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:22 GMT /vclfcn-64/ 2024-02-01T10:49:22.444 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+GO4Tk7lzqac0juuVOO1dVYJwAo= 2024-02-01T10:49:22.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+GO4Tk7lzqac0juuVOO1dVYJwAo=', 'Content-Length': '0'} 2024-02-01T10:49:22.448 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000055c7b145fb716ed1-0065bb7732-5762-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:22.449 DEBUG:boto:path=/vclfcn-64/obj6 2024-02-01T10:49:22.449 DEBUG:boto:auth_path=/vclfcn-64/obj6 2024-02-01T10:49:22.449 DEBUG:boto:Method: PUT 2024-02-01T10:49:22.449 DEBUG:boto:Path: /vclfcn-64/obj6 2024-02-01T10:49:22.449 DEBUG:boto:Data: 2024-02-01T10:49:22.449 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-02-01T10:49:22.449 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:22.449 DEBUG:boto:Port: 8001 2024-02-01T10:49:22.449 DEBUG:boto:Params: {} 2024-02-01T10:49:22.450 DEBUG:boto:Token: None 2024-02-01T10:49:22.450 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 01 Feb 2024 10:49:22 GMT /vclfcn-64/obj6 2024-02-01T10:49:22.450 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zye1Y3diKM3khcPhxkXQLoDbY7Q= 2024-02-01T10:49:22.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:49:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zye1Y3diKM3khcPhxkXQLoDbY7Q='} 2024-02-01T10:49:22.459 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001786ac453715b65f-0065bb7732-5762-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:22.459 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:49:22.459 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:49:22.460 DEBUG:boto:Method: HEAD 2024-02-01T10:49:22.460 DEBUG:boto:Path: /vclfcn-64/ 2024-02-01T10:49:22.460 DEBUG:boto:Data: 2024-02-01T10:49:22.460 DEBUG:boto:Headers: {} 2024-02-01T10:49:22.460 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:22.460 DEBUG:boto:Port: 8001 2024-02-01T10:49:22.460 DEBUG:boto:Params: {} 2024-02-01T10:49:22.460 DEBUG:boto:Token: None 2024-02-01T10:49:22.460 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:22 GMT /vclfcn-64/ 2024-02-01T10:49:22.461 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+GO4Tk7lzqac0juuVOO1dVYJwAo= 2024-02-01T10:49:22.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+GO4Tk7lzqac0juuVOO1dVYJwAo=', 'Content-Length': '0'} 2024-02-01T10:49:22.465 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001a9569cda4555c77-0065bb7732-5762-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:22.465 DEBUG:boto:path=/vclfcn-64/obj7 2024-02-01T10:49:22.465 DEBUG:boto:auth_path=/vclfcn-64/obj7 2024-02-01T10:49:22.465 DEBUG:boto:Method: PUT 2024-02-01T10:49:22.465 DEBUG:boto:Path: /vclfcn-64/obj7 2024-02-01T10:49:22.465 DEBUG:boto:Data: 2024-02-01T10:49:22.465 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-02-01T10:49:22.465 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:22.466 DEBUG:boto:Port: 8001 2024-02-01T10:49:22.466 DEBUG:boto:Params: {} 2024-02-01T10:49:22.466 DEBUG:boto:Token: None 2024-02-01T10:49:22.466 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 01 Feb 2024 10:49:22 GMT /vclfcn-64/obj7 2024-02-01T10:49:22.466 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:LN7DPh5cPjVGT0I6F6VIaOMcqRw= 2024-02-01T10:49:22.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:49:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:LN7DPh5cPjVGT0I6F6VIaOMcqRw='} 2024-02-01T10:49:22.508 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e5499d6372d52696-0065bb7732-5762-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:22.508 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:49:22.509 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:49:22.509 DEBUG:boto:Method: HEAD 2024-02-01T10:49:22.509 DEBUG:boto:Path: /vclfcn-64/ 2024-02-01T10:49:22.509 DEBUG:boto:Data: 2024-02-01T10:49:22.509 DEBUG:boto:Headers: {} 2024-02-01T10:49:22.509 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:22.509 DEBUG:boto:Port: 8001 2024-02-01T10:49:22.509 DEBUG:boto:Params: {} 2024-02-01T10:49:22.509 DEBUG:boto:Token: None 2024-02-01T10:49:22.510 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:22 GMT /vclfcn-64/ 2024-02-01T10:49:22.510 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+GO4Tk7lzqac0juuVOO1dVYJwAo= 2024-02-01T10:49:22.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+GO4Tk7lzqac0juuVOO1dVYJwAo=', 'Content-Length': '0'} 2024-02-01T10:49:22.513 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005f944d150451b15a-0065bb7732-5762-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:22.514 DEBUG:boto:path=/vclfcn-64/obj8 2024-02-01T10:49:22.514 DEBUG:boto:auth_path=/vclfcn-64/obj8 2024-02-01T10:49:22.514 DEBUG:boto:Method: PUT 2024-02-01T10:49:22.514 DEBUG:boto:Path: /vclfcn-64/obj8 2024-02-01T10:49:22.514 DEBUG:boto:Data: 2024-02-01T10:49:22.514 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-02-01T10:49:22.514 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:22.514 DEBUG:boto:Port: 8001 2024-02-01T10:49:22.514 DEBUG:boto:Params: {} 2024-02-01T10:49:22.514 DEBUG:boto:Token: None 2024-02-01T10:49:22.515 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 01 Feb 2024 10:49:22 GMT /vclfcn-64/obj8 2024-02-01T10:49:22.515 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9KvF8SD1TVIC10JZECgiBnuZC1I= 2024-02-01T10:49:22.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:49:22 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9KvF8SD1TVIC10JZECgiBnuZC1I='} 2024-02-01T10:49:22.522 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b8aba9cb398501f0-0065bb7732-5762-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:22.523 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'vclfcn-62', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:49:22.523 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'vclfcn-62', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:49:22.523 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket vclfcn-62 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-02-01T10:49:22.566 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:49:22.566 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:49:22.674 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'vclfcn-63', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:49:22.675 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'vclfcn-63', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:49:22.675 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket vclfcn-63 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-02-01T10:49:22.751 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:49:22.751 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:49:22.858 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'vclfcn-64', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:49:22.858 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'vclfcn-64', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-02-01T10:49:22.858 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket vclfcn-64 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-02-01T10:49:22.938 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:49:22.939 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:49:23.046 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=vclfcn-62 2024-02-01T10:49:23.047 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:49:23.047 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:49:23.047 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket vclfcn-62 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-02-01T10:49:23.097 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:49:23.097 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:49:23.165 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:49:23.160+0000 7f58fd42e980 1 waiting to reach incremental sync.. 2024-02-01T10:49:24.782 INFO:tasks.rgw.c1.client.1.smithi148.stdout:2024-02-01T10:49:24.778+0000 7f8357755640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-02-01T10:49:28.166 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:49:28.160+0000 7f58fd42e980 1 waiting to reach incremental sync.. 2024-02-01T10:49:33.167 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:49:33.168+0000 7f58fd42e980 1 waiting to reach incremental sync.. 2024-02-01T10:49:38.170 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:49:38.168+0000 7f58fd42e980 1 bucket sync caught up with source: 2024-02-01T10:49:38.171 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [00000000004.62.1, 00000000002.43.1, 00000000004.59.1, 00000000002.76.1, 00000000004.53.1, 00000000002.45.1, 00000000002.64.1, 00000000004.60.1, 00000000002.52.1, 00000000002.46.1, 00000000002.170.1] 2024-02-01T10:49:38.171 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [00000000004.62.1, 00000000002.43.1, 00000000004.59.1, 00000000002.76.1, 00000000004.53.1, 00000000002.45.1, 00000000002.64.1, 00000000004.60.1, 00000000002.52.1, 00000000002.46.1, 00000000002.170.1] 2024-02-01T10:49:38.171 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:49:38.168+0000 7f58fd42e980 0 bucket checkpoint complete 2024-02-01T10:49:38.212 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=vclfcn-62 2024-02-01T10:49:38.212 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:49:38.213 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:49:38.213 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-62 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:49:38.263 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:49:38.263 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:49:38.324 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:49:38.322+0000 7efc76b43980 1 bucket sync caught up with source: 2024-02-01T10:49:38.324 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [00000000004.62.1, 00000000002.43.1, 00000000004.59.1, 00000000002.76.1, 00000000004.53.1, 00000000002.45.1, 00000000002.64.1, 00000000004.60.1, 00000000002.52.1, 00000000002.46.1, 00000000002.170.1] 2024-02-01T10:49:38.324 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [00000000004.62.1, 00000000002.43.1, 00000000004.59.1, 00000000002.76.1, 00000000004.53.1, 00000000002.45.1, 00000000002.64.1, 00000000004.60.1, 00000000002.52.1, 00000000002.46.1, 00000000002.170.1] 2024-02-01T10:49:38.324 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:49:38.322+0000 7efc76b43980 0 bucket checkpoint complete 2024-02-01T10:49:38.346 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=vclfcn-62 2024-02-01T10:49:38.347 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-62', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:49:38.347 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-62', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:49:38.347 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-62 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:49:38.387 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:49:38.388 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:49:38.449 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:49:38.446+0000 7f236b3e5980 1 waiting for incremental sync to catch up: 2024-02-01T10:49:38.449 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [00000000003.300.1, 00000000002.247.1, 00000000003.223.1, 00000000002.328.1, 00000000003.246.1, 00000000002.161.1, 00000000002.375.1, 00000000003.226.1, 00000000002.250.1, 00000000002.164.1, 00000000002.432.1] 2024-02-01T10:49:38.449 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [00000000004.309.6, 00000000002.247.1, 00000000004.233.6, 00000000002.328.1, 00000000004.253.6, 00000000002.161.1, 00000000002.375.1, 00000000004.235.6, 00000000002.250.1, 00000000002.164.1, 00000000002.432.1] 2024-02-01T10:49:43.451 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:49:43.446+0000 7f236b3e5980 1 waiting for incremental sync to catch up: 2024-02-01T10:49:43.451 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [00000000003.300.1, 00000000002.247.1, 00000000003.223.1, 00000000002.328.1, 00000000003.246.1, 00000000002.161.1, 00000000002.375.1, 00000000003.226.1, 00000000002.250.1, 00000000002.164.1, 00000000002.432.1] 2024-02-01T10:49:43.451 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [00000000004.309.6, 00000000002.247.1, 00000000004.233.6, 00000000002.328.1, 00000000004.253.6, 00000000002.161.1, 00000000002.375.1, 00000000004.235.6, 00000000002.250.1, 00000000002.164.1, 00000000002.432.1] 2024-02-01T10:49:48.454 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:49:48.454+0000 7f236b3e5980 1 waiting for incremental sync to catch up: 2024-02-01T10:49:48.454 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [00000000004.309.6, 00000000002.247.1, 00000000003.223.1, 00000000002.328.1, 00000000004.253.6, 00000000002.161.1, 00000000002.375.1, 00000000004.235.6, 00000000002.250.1, 00000000002.164.1, 00000000002.432.1] 2024-02-01T10:49:48.454 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [00000000004.309.6, 00000000002.247.1, 00000000004.233.6, 00000000002.328.1, 00000000004.253.6, 00000000002.161.1, 00000000002.375.1, 00000000004.235.6, 00000000002.250.1, 00000000002.164.1, 00000000002.432.1] 2024-02-01T10:49:53.456 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:49:53.458+0000 7f236b3e5980 1 bucket sync caught up with source: 2024-02-01T10:49:53.456 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [00000000004.309.6, 00000000002.247.1, 00000000004.233.6, 00000000002.328.1, 00000000004.253.6, 00000000002.161.1, 00000000002.375.1, 00000000004.235.6, 00000000002.250.1, 00000000002.164.1, 00000000002.432.1] 2024-02-01T10:49:53.456 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [00000000004.309.6, 00000000002.247.1, 00000000004.233.6, 00000000002.328.1, 00000000004.253.6, 00000000002.161.1, 00000000002.375.1, 00000000004.235.6, 00000000002.250.1, 00000000002.164.1, 00000000002.432.1] 2024-02-01T10:49:53.457 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:49:53.458+0000 7f236b3e5980 0 bucket checkpoint complete 2024-02-01T10:49:53.504 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=vclfcn-62 2024-02-01T10:49:53.504 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-62', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:49:53.504 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-62', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:49:53.504 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-62 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:49:53.600 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:49:53.601 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:49:53.665 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:49:53.666+0000 7efc777d4980 1 bucket sync caught up with source: 2024-02-01T10:49:53.665 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [00000000004.309.6, 00000000002.247.1, 00000000004.233.6, 00000000002.328.1, 00000000004.253.6, 00000000002.161.1, 00000000002.375.1, 00000000004.235.6, 00000000002.250.1, 00000000002.164.1, 00000000002.432.1] 2024-02-01T10:49:53.665 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [00000000004.309.6, 00000000002.247.1, 00000000004.233.6, 00000000002.328.1, 00000000004.253.6, 00000000002.161.1, 00000000002.375.1, 00000000004.235.6, 00000000002.250.1, 00000000002.164.1, 00000000002.432.1] 2024-02-01T10:49:53.665 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:49:53.666+0000 7efc777d4980 0 bucket checkpoint complete 2024-02-01T10:49:53.688 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=vclfcn-62 2024-02-01T10:49:53.688 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-62', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:49:53.688 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-62', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:49:53.688 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-62 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:49:53.729 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:49:53.729 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:49:53.788 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:49:53.790+0000 7fcbd19f9980 1 bucket sync caught up with source: 2024-02-01T10:49:53.788 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [00000000004.229.1, 00000000002.153.1, 00000000004.114.1, 00000000002.223.1, 00000000004.159.1, 00000000002.113.1, 00000000002.343.1, 00000000004.116.1, 00000000002.221.1, 00000000002.118.1, 00000000002.384.1] 2024-02-01T10:49:53.788 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [00000000004.229.1, 00000000002.153.1, 00000000004.114.1, 00000000002.223.1, 00000000004.159.1, 00000000002.113.1, 00000000002.343.1, 00000000004.116.1, 00000000002.221.1, 00000000002.118.1, 00000000002.384.1] 2024-02-01T10:49:53.788 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:49:53.790+0000 7fcbd19f9980 0 bucket checkpoint complete 2024-02-01T10:49:53.813 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=vclfcn-62 2024-02-01T10:49:53.813 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-62', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:49:53.813 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-62', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:49:53.813 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket vclfcn-62 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-02-01T10:49:53.863 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:49:53.864 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:49:53.929 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:49:53.928+0000 7f0be80b5980 1 bucket sync caught up with source: 2024-02-01T10:49:53.929 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [00000000004.229.1, 00000000002.153.1, 00000000004.114.1, 00000000002.223.1, 00000000004.159.1, 00000000002.113.1, 00000000002.343.1, 00000000004.116.1, 00000000002.221.1, 00000000002.118.1, 00000000002.384.1] 2024-02-01T10:49:53.930 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [00000000004.229.1, 00000000002.153.1, 00000000004.114.1, 00000000002.223.1, 00000000004.159.1, 00000000002.113.1, 00000000002.343.1, 00000000004.116.1, 00000000002.221.1, 00000000002.118.1, 00000000002.384.1] 2024-02-01T10:49:53.930 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:49:53.928+0000 7f0be80b5980 0 bucket checkpoint complete 2024-02-01T10:49:53.953 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-62 zones={test-zone2, test-zone1} 2024-02-01T10:49:53.953 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:53.954 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:53.954 DEBUG:boto:Method: HEAD 2024-02-01T10:49:53.954 DEBUG:boto:Path: /vclfcn-62/ 2024-02-01T10:49:53.954 DEBUG:boto:Data: 2024-02-01T10:49:53.954 DEBUG:boto:Headers: {} 2024-02-01T10:49:53.954 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:53.954 DEBUG:boto:Port: 8000 2024-02-01T10:49:53.954 DEBUG:boto:Params: {} 2024-02-01T10:49:53.955 DEBUG:boto:Token: None 2024-02-01T10:49:53.955 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:53 GMT /vclfcn-62/ 2024-02-01T10:49:53.955 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:qF2FzePMPmICWXxg+NKmRQbeazY= 2024-02-01T10:49:53.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:qF2FzePMPmICWXxg+NKmRQbeazY=', 'Content-Length': '0'} 2024-02-01T10:49:53.958 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000067366fc830f159ec-0065bb7751-5350-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:49:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:53.958 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:53.958 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:53.958 DEBUG:boto:Method: HEAD 2024-02-01T10:49:53.959 DEBUG:boto:Path: /vclfcn-62/ 2024-02-01T10:49:53.959 DEBUG:boto:Data: 2024-02-01T10:49:53.959 DEBUG:boto:Headers: {} 2024-02-01T10:49:53.959 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:53.959 DEBUG:boto:Port: 8000 2024-02-01T10:49:53.959 DEBUG:boto:Params: {} 2024-02-01T10:49:53.959 DEBUG:boto:Token: None 2024-02-01T10:49:53.959 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:53 GMT /vclfcn-62/ 2024-02-01T10:49:53.959 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:qF2FzePMPmICWXxg+NKmRQbeazY= 2024-02-01T10:49:53.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:qF2FzePMPmICWXxg+NKmRQbeazY=', 'Content-Length': '0'} 2024-02-01T10:49:53.963 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d8f9e5d818dd677c-0065bb7751-5765-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:49:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:53.964 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:49:53.964 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:53.964 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:53.964 DEBUG:boto:path=/vclfcn-62/?versions&max-keys=999 2024-02-01T10:49:53.964 DEBUG:boto:auth_path=/vclfcn-62/?versions&max-keys=999 2024-02-01T10:49:53.964 DEBUG:boto:Method: GET 2024-02-01T10:49:53.964 DEBUG:boto:Path: /vclfcn-62/?versions&max-keys=999 2024-02-01T10:49:53.964 DEBUG:boto:Data: 2024-02-01T10:49:53.965 DEBUG:boto:Headers: {} 2024-02-01T10:49:53.965 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:53.965 DEBUG:boto:Port: 8000 2024-02-01T10:49:53.965 DEBUG:boto:Params: {} 2024-02-01T10:49:53.965 DEBUG:boto:Token: None 2024-02-01T10:49:53.965 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:53 GMT /vclfcn-62/?versions 2024-02-01T10:49:53.965 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WlDGAIo1NaISIO/YE77oWNQtdcE= 2024-02-01T10:49:53.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WlDGAIo1NaISIO/YE77oWNQtdcE=', 'Content-Length': '0'} 2024-02-01T10:49:53.968 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000059c0180e3dd7c259-0065bb7751-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:53.968 DEBUG:boto:b'vclfcn-62999falseobj1nulltrue2024-02-01T10:48:26.315Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:48:26.333Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:48:26.384Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:48:26.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-02-01T10:49:22.123Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-02-01T10:49:22.135Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-02-01T10:49:22.186Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-02-01T10:49:22.200Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:53.969 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-02-01T10:49:53.969 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-02-01T10:49:53.969 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-02-01T10:49:53.969 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-02-01T10:49:53.969 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-02-01T10:49:53.969 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-02-01T10:49:53.970 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-02-01T10:49:53.970 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-02-01T10:49:53.970 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:49:53.970 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:53.970 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:53.970 DEBUG:boto:path=/vclfcn-62/?versions&max-keys=999 2024-02-01T10:49:53.970 DEBUG:boto:auth_path=/vclfcn-62/?versions&max-keys=999 2024-02-01T10:49:53.970 DEBUG:boto:Method: GET 2024-02-01T10:49:53.970 DEBUG:boto:Path: /vclfcn-62/?versions&max-keys=999 2024-02-01T10:49:53.970 DEBUG:boto:Data: 2024-02-01T10:49:53.970 DEBUG:boto:Headers: {} 2024-02-01T10:49:53.971 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:53.971 DEBUG:boto:Port: 8000 2024-02-01T10:49:53.971 DEBUG:boto:Params: {} 2024-02-01T10:49:53.971 DEBUG:boto:Token: None 2024-02-01T10:49:53.971 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:53 GMT /vclfcn-62/?versions 2024-02-01T10:49:53.971 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WlDGAIo1NaISIO/YE77oWNQtdcE= 2024-02-01T10:49:53.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WlDGAIo1NaISIO/YE77oWNQtdcE=', 'Content-Length': '0'} 2024-02-01T10:49:53.975 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d42eb6d63d1407a3-0065bb7751-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:53.975 DEBUG:boto:b'vclfcn-62999falseobj1nulltrue2024-02-01T10:48:26.315Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:48:26.333Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:48:26.384Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:48:26.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-02-01T10:49:22.123Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-02-01T10:49:22.135Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-02-01T10:49:22.186Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-02-01T10:49:22.200Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:53.976 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-02-01T10:49:53.976 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-02-01T10:49:53.976 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-02-01T10:49:53.976 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-02-01T10:49:53.977 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-02-01T10:49:53.977 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-02-01T10:49:53.977 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-02-01T10:49:53.977 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-02-01T10:49:53.977 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:53.977 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:53.977 DEBUG:boto:path=/vclfcn-62/?versions&max-keys=999 2024-02-01T10:49:53.977 DEBUG:boto:auth_path=/vclfcn-62/?versions&max-keys=999 2024-02-01T10:49:53.977 DEBUG:boto:Method: GET 2024-02-01T10:49:53.977 DEBUG:boto:Path: /vclfcn-62/?versions&max-keys=999 2024-02-01T10:49:53.978 DEBUG:boto:Data: 2024-02-01T10:49:53.978 DEBUG:boto:Headers: {} 2024-02-01T10:49:53.978 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:53.978 DEBUG:boto:Port: 8000 2024-02-01T10:49:53.978 DEBUG:boto:Params: {} 2024-02-01T10:49:53.978 DEBUG:boto:Token: None 2024-02-01T10:49:53.978 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:53 GMT /vclfcn-62/?versions 2024-02-01T10:49:53.978 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WlDGAIo1NaISIO/YE77oWNQtdcE= 2024-02-01T10:49:53.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WlDGAIo1NaISIO/YE77oWNQtdcE=', 'Content-Length': '0'} 2024-02-01T10:49:53.981 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000048ee294a73fa6070-0065bb7751-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:53.981 DEBUG:boto:b'vclfcn-62999falseobj1nulltrue2024-02-01T10:48:26.315Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:48:26.333Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:48:26.384Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:48:26.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-02-01T10:49:22.123Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-02-01T10:49:22.135Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-02-01T10:49:22.186Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-02-01T10:49:22.200Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:53.982 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:53.982 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:53.982 DEBUG:boto:path=/vclfcn-62/?versions&max-keys=999 2024-02-01T10:49:53.982 DEBUG:boto:auth_path=/vclfcn-62/?versions&max-keys=999 2024-02-01T10:49:53.982 DEBUG:boto:Method: GET 2024-02-01T10:49:53.982 DEBUG:boto:Path: /vclfcn-62/?versions&max-keys=999 2024-02-01T10:49:53.982 DEBUG:boto:Data: 2024-02-01T10:49:53.982 DEBUG:boto:Headers: {} 2024-02-01T10:49:53.982 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:53.982 DEBUG:boto:Port: 8000 2024-02-01T10:49:53.983 DEBUG:boto:Params: {} 2024-02-01T10:49:53.983 DEBUG:boto:Token: None 2024-02-01T10:49:53.983 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:53 GMT /vclfcn-62/?versions 2024-02-01T10:49:53.983 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WlDGAIo1NaISIO/YE77oWNQtdcE= 2024-02-01T10:49:53.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WlDGAIo1NaISIO/YE77oWNQtdcE=', 'Content-Length': '0'} 2024-02-01T10:49:53.987 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088d7e465884a4dbf-0065bb7751-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:53.988 DEBUG:boto:b'vclfcn-62999falseobj1nulltrue2024-02-01T10:48:26.315Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:48:26.333Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:48:26.384Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:48:26.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-02-01T10:49:22.123Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-02-01T10:49:22.135Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-02-01T10:49:22.186Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-02-01T10:49:22.200Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:53.988 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:49:53.989 DEBUG:boto:path=/vclfcn-62/obj1 2024-02-01T10:49:53.989 DEBUG:boto:auth_path=/vclfcn-62/obj1 2024-02-01T10:49:53.989 DEBUG:boto:path=/vclfcn-62/obj1?versionId=null 2024-02-01T10:49:53.989 DEBUG:boto:auth_path=/vclfcn-62/obj1?versionId=null 2024-02-01T10:49:53.989 DEBUG:boto:Method: GET 2024-02-01T10:49:53.989 DEBUG:boto:Path: /vclfcn-62/obj1?versionId=null 2024-02-01T10:49:53.989 DEBUG:boto:Data: 2024-02-01T10:49:53.989 DEBUG:boto:Headers: {} 2024-02-01T10:49:53.989 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:53.989 DEBUG:boto:Port: 8000 2024-02-01T10:49:53.989 DEBUG:boto:Params: {} 2024-02-01T10:49:53.990 DEBUG:boto:Token: None 2024-02-01T10:49:53.990 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:53 GMT /vclfcn-62/obj1?versionId=null 2024-02-01T10:49:53.990 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:juxGb8d/hp6UCy/UcRIGk+IYDCs= 2024-02-01T10:49:53.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:juxGb8d/hp6UCy/UcRIGk+IYDCs=', 'Content-Length': '0'} 2024-02-01T10:49:53.992 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009fb7f2e74c7b1cca-0065bb7751-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:53.992 DEBUG:boto:path=/vclfcn-62/obj1 2024-02-01T10:49:53.992 DEBUG:boto:auth_path=/vclfcn-62/obj1 2024-02-01T10:49:53.992 DEBUG:boto:path=/vclfcn-62/obj1?versionId=null 2024-02-01T10:49:53.993 DEBUG:boto:auth_path=/vclfcn-62/obj1?versionId=null 2024-02-01T10:49:53.993 DEBUG:boto:Method: GET 2024-02-01T10:49:53.993 DEBUG:boto:Path: /vclfcn-62/obj1?versionId=null 2024-02-01T10:49:53.993 DEBUG:boto:Data: 2024-02-01T10:49:53.993 DEBUG:boto:Headers: {} 2024-02-01T10:49:53.993 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:53.993 DEBUG:boto:Port: 8000 2024-02-01T10:49:53.993 DEBUG:boto:Params: {} 2024-02-01T10:49:53.994 DEBUG:boto:Token: None 2024-02-01T10:49:53.994 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:53 GMT /vclfcn-62/obj1?versionId=null 2024-02-01T10:49:53.994 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:juxGb8d/hp6UCy/UcRIGk+IYDCs= 2024-02-01T10:49:53.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:juxGb8d/hp6UCy/UcRIGk+IYDCs=', 'Content-Length': '0'} 2024-02-01T10:49:53.997 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009095172895f4cb0d-0065bb7751-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:53 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:53.997 DEBUG:boto:path=/vclfcn-62/obj1 2024-02-01T10:49:53.997 DEBUG:boto:auth_path=/vclfcn-62/obj1 2024-02-01T10:49:53.997 DEBUG:boto:path=/vclfcn-62/obj1?versionId=null 2024-02-01T10:49:53.997 DEBUG:boto:auth_path=/vclfcn-62/obj1?versionId=null 2024-02-01T10:49:53.997 DEBUG:boto:Method: HEAD 2024-02-01T10:49:53.997 DEBUG:boto:Path: /vclfcn-62/obj1?versionId=null 2024-02-01T10:49:53.997 DEBUG:boto:Data: 2024-02-01T10:49:53.998 DEBUG:boto:Headers: {} 2024-02-01T10:49:53.998 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:53.998 DEBUG:boto:Port: 8000 2024-02-01T10:49:53.998 DEBUG:boto:Params: {} 2024-02-01T10:49:53.998 DEBUG:boto:Token: None 2024-02-01T10:49:53.998 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:53 GMT /vclfcn-62/obj1?versionId=null 2024-02-01T10:49:53.998 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:v1Hct4MdDAsojjkQqBT66lfIc6M= 2024-02-01T10:49:53.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:53 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:v1Hct4MdDAsojjkQqBT66lfIc6M=', 'Content-Length': '0'} 2024-02-01T10:49:54.000 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000047e0e4625be17aa6-0065bb7752-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:54.000 DEBUG:boto:path=/vclfcn-62/obj1 2024-02-01T10:49:54.001 DEBUG:boto:auth_path=/vclfcn-62/obj1 2024-02-01T10:49:54.001 DEBUG:boto:path=/vclfcn-62/obj1?versionId=null 2024-02-01T10:49:54.001 DEBUG:boto:auth_path=/vclfcn-62/obj1?versionId=null 2024-02-01T10:49:54.001 DEBUG:boto:Method: HEAD 2024-02-01T10:49:54.001 DEBUG:boto:Path: /vclfcn-62/obj1?versionId=null 2024-02-01T10:49:54.001 DEBUG:boto:Data: 2024-02-01T10:49:54.001 DEBUG:boto:Headers: {} 2024-02-01T10:49:54.001 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:54.001 DEBUG:boto:Port: 8000 2024-02-01T10:49:54.001 DEBUG:boto:Params: {} 2024-02-01T10:49:54.002 DEBUG:boto:Token: None 2024-02-01T10:49:54.002 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:54 GMT /vclfcn-62/obj1?versionId=null 2024-02-01T10:49:54.002 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:CJ+siqlQajHt7oHELqhy+FY8EZc= 2024-02-01T10:49:54.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:CJ+siqlQajHt7oHELqhy+FY8EZc=', 'Content-Length': '0'} 2024-02-01T10:49:54.005 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fe2e9050cd530146-0065bb7752-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:54.005 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:49:54.005 DEBUG:boto:path=/vclfcn-62/obj1 2024-02-01T10:49:54.005 DEBUG:boto:auth_path=/vclfcn-62/obj1 2024-02-01T10:49:54.005 DEBUG:boto:path=/vclfcn-62/obj1?versionId=null 2024-02-01T10:49:54.005 DEBUG:boto:auth_path=/vclfcn-62/obj1?versionId=null 2024-02-01T10:49:54.005 DEBUG:boto:Method: GET 2024-02-01T10:49:54.005 DEBUG:boto:Path: /vclfcn-62/obj1?versionId=null 2024-02-01T10:49:54.005 DEBUG:boto:Data: 2024-02-01T10:49:54.006 DEBUG:boto:Headers: {} 2024-02-01T10:49:54.006 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:54.006 DEBUG:boto:Port: 8000 2024-02-01T10:49:54.006 DEBUG:boto:Params: {} 2024-02-01T10:49:54.006 DEBUG:boto:Token: None 2024-02-01T10:49:54.006 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:54 GMT /vclfcn-62/obj1?versionId=null 2024-02-01T10:49:54.006 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:kitTD5yZeRwCfddeItJd1ZbrRV0= 2024-02-01T10:49:54.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:kitTD5yZeRwCfddeItJd1ZbrRV0=', 'Content-Length': '0'} 2024-02-01T10:49:54.008 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000218f739f9ba2b053-0065bb7752-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:54.052 DEBUG:boto:path=/vclfcn-62/obj1 2024-02-01T10:49:54.052 DEBUG:boto:auth_path=/vclfcn-62/obj1 2024-02-01T10:49:54.052 DEBUG:boto:path=/vclfcn-62/obj1?versionId=null 2024-02-01T10:49:54.053 DEBUG:boto:auth_path=/vclfcn-62/obj1?versionId=null 2024-02-01T10:49:54.053 DEBUG:boto:Method: GET 2024-02-01T10:49:54.053 DEBUG:boto:Path: /vclfcn-62/obj1?versionId=null 2024-02-01T10:49:54.053 DEBUG:boto:Data: 2024-02-01T10:49:54.053 DEBUG:boto:Headers: {} 2024-02-01T10:49:54.053 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:54.053 DEBUG:boto:Port: 8000 2024-02-01T10:49:54.053 DEBUG:boto:Params: {} 2024-02-01T10:49:54.053 DEBUG:boto:Token: None 2024-02-01T10:49:54.054 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:54 GMT /vclfcn-62/obj1?versionId=null 2024-02-01T10:49:54.054 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:kitTD5yZeRwCfddeItJd1ZbrRV0= 2024-02-01T10:49:54.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:kitTD5yZeRwCfddeItJd1ZbrRV0=', 'Content-Length': '0'} 2024-02-01T10:49:54.057 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fd54119cbf54445e-0065bb7752-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:54.057 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:54.057 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:54.057 DEBUG:boto:path=/vclfcn-62/?prefix=obj1 2024-02-01T10:49:54.057 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj1 2024-02-01T10:49:54.057 DEBUG:boto:Method: GET 2024-02-01T10:49:54.057 DEBUG:boto:Path: /vclfcn-62/?prefix=obj1 2024-02-01T10:49:54.058 DEBUG:boto:Data: 2024-02-01T10:49:54.058 DEBUG:boto:Headers: {} 2024-02-01T10:49:54.058 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:54.059 DEBUG:boto:Port: 8000 2024-02-01T10:49:54.059 DEBUG:boto:Params: {} 2024-02-01T10:49:54.059 DEBUG:boto:Token: None 2024-02-01T10:49:54.059 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:54 GMT /vclfcn-62/ 2024-02-01T10:49:54.059 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HhKpJHgR2UFZClMtRS3LPwCK2sA= 2024-02-01T10:49:54.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HhKpJHgR2UFZClMtRS3LPwCK2sA=', 'Content-Length': '0'} 2024-02-01T10:49:54.063 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007874a72c03459a84-0065bb7752-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:54.064 DEBUG:boto:b'vclfcn-62obj11000falseobj12024-02-01T10:48:26.315Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:54.065 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:54.065 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:54.065 DEBUG:boto:path=/vclfcn-62/?prefix=obj1 2024-02-01T10:49:54.065 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj1 2024-02-01T10:49:54.066 DEBUG:boto:Method: GET 2024-02-01T10:49:54.066 DEBUG:boto:Path: /vclfcn-62/?prefix=obj1 2024-02-01T10:49:54.066 DEBUG:boto:Data: 2024-02-01T10:49:54.066 DEBUG:boto:Headers: {} 2024-02-01T10:49:54.066 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:54.067 DEBUG:boto:Port: 8000 2024-02-01T10:49:54.068 DEBUG:boto:Params: {} 2024-02-01T10:49:54.068 DEBUG:boto:Token: None 2024-02-01T10:49:54.068 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:54 GMT /vclfcn-62/ 2024-02-01T10:49:54.068 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HhKpJHgR2UFZClMtRS3LPwCK2sA= 2024-02-01T10:49:54.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HhKpJHgR2UFZClMtRS3LPwCK2sA=', 'Content-Length': '0'} 2024-02-01T10:49:54.072 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000485df8cf817d6b4e-0065bb7752-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:54.073 DEBUG:boto:b'vclfcn-62obj11000falseobj12024-02-01T10:48:26.315Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:54.073 DEBUG:boto:path=/vclfcn-62/obj1 2024-02-01T10:49:54.073 DEBUG:boto:auth_path=/vclfcn-62/obj1 2024-02-01T10:49:54.074 DEBUG:boto:Method: HEAD 2024-02-01T10:49:54.074 DEBUG:boto:Path: /vclfcn-62/obj1 2024-02-01T10:49:54.074 DEBUG:boto:Data: 2024-02-01T10:49:54.074 DEBUG:boto:Headers: {} 2024-02-01T10:49:54.074 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:54.074 DEBUG:boto:Port: 8000 2024-02-01T10:49:54.074 DEBUG:boto:Params: {} 2024-02-01T10:49:54.074 DEBUG:boto:Token: None 2024-02-01T10:49:54.074 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:54 GMT /vclfcn-62/obj1 2024-02-01T10:49:54.074 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:3JweZgrOgKfVb1A2xCiTR1py7yM= 2024-02-01T10:49:54.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:3JweZgrOgKfVb1A2xCiTR1py7yM=', 'Content-Length': '0'} 2024-02-01T10:49:54.077 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000860f4f1115e4e2f7-0065bb7752-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:54.077 DEBUG:boto:path=/vclfcn-62/obj1 2024-02-01T10:49:54.077 DEBUG:boto:auth_path=/vclfcn-62/obj1 2024-02-01T10:49:54.078 DEBUG:boto:Method: HEAD 2024-02-01T10:49:54.078 DEBUG:boto:Path: /vclfcn-62/obj1 2024-02-01T10:49:54.078 DEBUG:boto:Data: 2024-02-01T10:49:54.078 DEBUG:boto:Headers: {} 2024-02-01T10:49:54.078 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:54.078 DEBUG:boto:Port: 8000 2024-02-01T10:49:54.078 DEBUG:boto:Params: {} 2024-02-01T10:49:54.078 DEBUG:boto:Token: None 2024-02-01T10:49:54.078 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:54 GMT /vclfcn-62/obj1 2024-02-01T10:49:54.078 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:3JweZgrOgKfVb1A2xCiTR1py7yM= 2024-02-01T10:49:54.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:3JweZgrOgKfVb1A2xCiTR1py7yM=', 'Content-Length': '0'} 2024-02-01T10:49:54.081 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000106ab6175a06fbc8-0065bb7752-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:54.081 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:49:54.082 DEBUG:boto:path=/vclfcn-62/obj1 2024-02-01T10:49:54.082 DEBUG:boto:auth_path=/vclfcn-62/obj1 2024-02-01T10:49:54.082 DEBUG:boto:Method: GET 2024-02-01T10:49:54.082 DEBUG:boto:Path: /vclfcn-62/obj1 2024-02-01T10:49:54.082 DEBUG:boto:Data: 2024-02-01T10:49:54.082 DEBUG:boto:Headers: {} 2024-02-01T10:49:54.082 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:54.082 DEBUG:boto:Port: 8000 2024-02-01T10:49:54.082 DEBUG:boto:Params: {} 2024-02-01T10:49:54.083 DEBUG:boto:Token: None 2024-02-01T10:49:54.083 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:54 GMT /vclfcn-62/obj1 2024-02-01T10:49:54.083 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:r4YcK6Dep4LlxxxljpjJ/VXgUvg= 2024-02-01T10:49:54.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:r4YcK6Dep4LlxxxljpjJ/VXgUvg=', 'Content-Length': '0'} 2024-02-01T10:49:54.085 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dcff112ee78977c2-0065bb7752-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:54.128 DEBUG:boto:path=/vclfcn-62/obj1 2024-02-01T10:49:54.128 DEBUG:boto:auth_path=/vclfcn-62/obj1 2024-02-01T10:49:54.129 DEBUG:boto:Method: GET 2024-02-01T10:49:54.129 DEBUG:boto:Path: /vclfcn-62/obj1 2024-02-01T10:49:54.129 DEBUG:boto:Data: 2024-02-01T10:49:54.129 DEBUG:boto:Headers: {} 2024-02-01T10:49:54.129 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:54.129 DEBUG:boto:Port: 8000 2024-02-01T10:49:54.129 DEBUG:boto:Params: {} 2024-02-01T10:49:54.129 DEBUG:boto:Token: None 2024-02-01T10:49:54.129 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:54 GMT /vclfcn-62/obj1 2024-02-01T10:49:54.129 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:r4YcK6Dep4LlxxxljpjJ/VXgUvg= 2024-02-01T10:49:54.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:r4YcK6Dep4LlxxxljpjJ/VXgUvg=', 'Content-Length': '0'} 2024-02-01T10:49:54.132 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009e6940a72684d4f3-0065bb7752-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:54.133 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:54.133 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:54.133 DEBUG:boto:path=/vclfcn-62/?prefix=obj1 2024-02-01T10:49:54.133 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj1 2024-02-01T10:49:54.133 DEBUG:boto:Method: GET 2024-02-01T10:49:54.133 DEBUG:boto:Path: /vclfcn-62/?prefix=obj1 2024-02-01T10:49:54.133 DEBUG:boto:Data: 2024-02-01T10:49:54.133 DEBUG:boto:Headers: {} 2024-02-01T10:49:54.133 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:54.134 DEBUG:boto:Port: 8000 2024-02-01T10:49:54.134 DEBUG:boto:Params: {} 2024-02-01T10:49:54.134 DEBUG:boto:Token: None 2024-02-01T10:49:54.134 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:54 GMT /vclfcn-62/ 2024-02-01T10:49:54.134 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HhKpJHgR2UFZClMtRS3LPwCK2sA= 2024-02-01T10:49:54.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HhKpJHgR2UFZClMtRS3LPwCK2sA=', 'Content-Length': '0'} 2024-02-01T10:49:54.138 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000660de8959e0b6ca3-0065bb7752-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:54.138 DEBUG:boto:b'vclfcn-62obj11000falseobj12024-02-01T10:48:26.315Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:54.139 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:54.139 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:54.139 DEBUG:boto:path=/vclfcn-62/?prefix=obj1 2024-02-01T10:49:54.139 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj1 2024-02-01T10:49:54.139 DEBUG:boto:Method: GET 2024-02-01T10:49:54.140 DEBUG:boto:Path: /vclfcn-62/?prefix=obj1 2024-02-01T10:49:54.140 DEBUG:boto:Data: 2024-02-01T10:49:54.140 DEBUG:boto:Headers: {} 2024-02-01T10:49:54.140 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:54.140 DEBUG:boto:Port: 8000 2024-02-01T10:49:54.140 DEBUG:boto:Params: {} 2024-02-01T10:49:54.140 DEBUG:boto:Token: None 2024-02-01T10:49:54.140 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:54 GMT /vclfcn-62/ 2024-02-01T10:49:54.140 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HhKpJHgR2UFZClMtRS3LPwCK2sA= 2024-02-01T10:49:54.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HhKpJHgR2UFZClMtRS3LPwCK2sA=', 'Content-Length': '0'} 2024-02-01T10:49:54.145 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000057547cb77a3f53b7-0065bb7752-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:54.188 DEBUG:boto:b'vclfcn-62obj11000falseobj12024-02-01T10:48:26.315Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:54.189 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:49:54.189 DEBUG:boto:path=/vclfcn-62/obj2 2024-02-01T10:49:54.189 DEBUG:boto:auth_path=/vclfcn-62/obj2 2024-02-01T10:49:54.189 DEBUG:boto:path=/vclfcn-62/obj2?versionId=null 2024-02-01T10:49:54.189 DEBUG:boto:auth_path=/vclfcn-62/obj2?versionId=null 2024-02-01T10:49:54.189 DEBUG:boto:Method: GET 2024-02-01T10:49:54.189 DEBUG:boto:Path: /vclfcn-62/obj2?versionId=null 2024-02-01T10:49:54.189 DEBUG:boto:Data: 2024-02-01T10:49:54.190 DEBUG:boto:Headers: {} 2024-02-01T10:49:54.190 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:54.190 DEBUG:boto:Port: 8000 2024-02-01T10:49:54.190 DEBUG:boto:Params: {} 2024-02-01T10:49:54.190 DEBUG:boto:Token: None 2024-02-01T10:49:54.190 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:54 GMT /vclfcn-62/obj2?versionId=null 2024-02-01T10:49:54.190 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:l7u4MHfXJEfjWpZIb6yMPZSGroY= 2024-02-01T10:49:54.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:l7u4MHfXJEfjWpZIb6yMPZSGroY=', 'Content-Length': '0'} 2024-02-01T10:49:54.193 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008c88f542b51a486a-0065bb7752-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:54.240 DEBUG:boto:path=/vclfcn-62/obj2 2024-02-01T10:49:54.240 DEBUG:boto:auth_path=/vclfcn-62/obj2 2024-02-01T10:49:54.240 DEBUG:boto:path=/vclfcn-62/obj2?versionId=null 2024-02-01T10:49:54.240 DEBUG:boto:auth_path=/vclfcn-62/obj2?versionId=null 2024-02-01T10:49:54.241 DEBUG:boto:Method: GET 2024-02-01T10:49:54.241 DEBUG:boto:Path: /vclfcn-62/obj2?versionId=null 2024-02-01T10:49:54.241 DEBUG:boto:Data: 2024-02-01T10:49:54.241 DEBUG:boto:Headers: {} 2024-02-01T10:49:54.241 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:54.241 DEBUG:boto:Port: 8000 2024-02-01T10:49:54.241 DEBUG:boto:Params: {} 2024-02-01T10:49:54.241 DEBUG:boto:Token: None 2024-02-01T10:49:54.241 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:54 GMT /vclfcn-62/obj2?versionId=null 2024-02-01T10:49:54.242 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:l7u4MHfXJEfjWpZIb6yMPZSGroY= 2024-02-01T10:49:54.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:l7u4MHfXJEfjWpZIb6yMPZSGroY=', 'Content-Length': '0'} 2024-02-01T10:49:54.245 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e3b5d0d0dc6cc979-0065bb7752-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:54.245 DEBUG:boto:path=/vclfcn-62/obj2 2024-02-01T10:49:54.245 DEBUG:boto:auth_path=/vclfcn-62/obj2 2024-02-01T10:49:54.245 DEBUG:boto:path=/vclfcn-62/obj2?versionId=null 2024-02-01T10:49:54.245 DEBUG:boto:auth_path=/vclfcn-62/obj2?versionId=null 2024-02-01T10:49:54.246 DEBUG:boto:Method: HEAD 2024-02-01T10:49:54.246 DEBUG:boto:Path: /vclfcn-62/obj2?versionId=null 2024-02-01T10:49:54.246 DEBUG:boto:Data: 2024-02-01T10:49:54.246 DEBUG:boto:Headers: {} 2024-02-01T10:49:54.246 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:54.246 DEBUG:boto:Port: 8000 2024-02-01T10:49:54.246 DEBUG:boto:Params: {} 2024-02-01T10:49:54.246 DEBUG:boto:Token: None 2024-02-01T10:49:54.246 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:54 GMT /vclfcn-62/obj2?versionId=null 2024-02-01T10:49:54.246 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:EsNYflrJlEGC6TNA3ekEURuGjmc= 2024-02-01T10:49:54.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:EsNYflrJlEGC6TNA3ekEURuGjmc=', 'Content-Length': '0'} 2024-02-01T10:49:54.249 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006a21d2dbfcaf3cb3-0065bb7752-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:54.250 DEBUG:boto:path=/vclfcn-62/obj2 2024-02-01T10:49:54.250 DEBUG:boto:auth_path=/vclfcn-62/obj2 2024-02-01T10:49:54.250 DEBUG:boto:path=/vclfcn-62/obj2?versionId=null 2024-02-01T10:49:54.250 DEBUG:boto:auth_path=/vclfcn-62/obj2?versionId=null 2024-02-01T10:49:54.250 DEBUG:boto:Method: HEAD 2024-02-01T10:49:54.250 DEBUG:boto:Path: /vclfcn-62/obj2?versionId=null 2024-02-01T10:49:54.250 DEBUG:boto:Data: 2024-02-01T10:49:54.250 DEBUG:boto:Headers: {} 2024-02-01T10:49:54.251 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:54.251 DEBUG:boto:Port: 8000 2024-02-01T10:49:54.251 DEBUG:boto:Params: {} 2024-02-01T10:49:54.251 DEBUG:boto:Token: None 2024-02-01T10:49:54.251 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:54 GMT /vclfcn-62/obj2?versionId=null 2024-02-01T10:49:54.251 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:EsNYflrJlEGC6TNA3ekEURuGjmc= 2024-02-01T10:49:54.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:EsNYflrJlEGC6TNA3ekEURuGjmc=', 'Content-Length': '0'} 2024-02-01T10:49:54.254 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003e16dbfaa52dca8a-0065bb7752-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:54.254 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:49:54.254 DEBUG:boto:path=/vclfcn-62/obj2 2024-02-01T10:49:54.254 DEBUG:boto:auth_path=/vclfcn-62/obj2 2024-02-01T10:49:54.254 DEBUG:boto:path=/vclfcn-62/obj2?versionId=null 2024-02-01T10:49:54.254 DEBUG:boto:auth_path=/vclfcn-62/obj2?versionId=null 2024-02-01T10:49:54.255 DEBUG:boto:Method: GET 2024-02-01T10:49:54.255 DEBUG:boto:Path: /vclfcn-62/obj2?versionId=null 2024-02-01T10:49:54.255 DEBUG:boto:Data: 2024-02-01T10:49:54.255 DEBUG:boto:Headers: {} 2024-02-01T10:49:54.255 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:54.255 DEBUG:boto:Port: 8000 2024-02-01T10:49:54.255 DEBUG:boto:Params: {} 2024-02-01T10:49:54.255 DEBUG:boto:Token: None 2024-02-01T10:49:54.255 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:54 GMT /vclfcn-62/obj2?versionId=null 2024-02-01T10:49:54.255 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:l7u4MHfXJEfjWpZIb6yMPZSGroY= 2024-02-01T10:49:54.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:l7u4MHfXJEfjWpZIb6yMPZSGroY=', 'Content-Length': '0'} 2024-02-01T10:49:54.258 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009f4fa0004be6f099-0065bb7752-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:54.304 DEBUG:boto:path=/vclfcn-62/obj2 2024-02-01T10:49:54.304 DEBUG:boto:auth_path=/vclfcn-62/obj2 2024-02-01T10:49:54.304 DEBUG:boto:path=/vclfcn-62/obj2?versionId=null 2024-02-01T10:49:54.304 DEBUG:boto:auth_path=/vclfcn-62/obj2?versionId=null 2024-02-01T10:49:54.305 DEBUG:boto:Method: GET 2024-02-01T10:49:54.305 DEBUG:boto:Path: /vclfcn-62/obj2?versionId=null 2024-02-01T10:49:54.305 DEBUG:boto:Data: 2024-02-01T10:49:54.305 DEBUG:boto:Headers: {} 2024-02-01T10:49:54.305 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:54.305 DEBUG:boto:Port: 8000 2024-02-01T10:49:54.305 DEBUG:boto:Params: {} 2024-02-01T10:49:54.305 DEBUG:boto:Token: None 2024-02-01T10:49:54.305 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:54 GMT /vclfcn-62/obj2?versionId=null 2024-02-01T10:49:54.306 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:l7u4MHfXJEfjWpZIb6yMPZSGroY= 2024-02-01T10:49:54.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:l7u4MHfXJEfjWpZIb6yMPZSGroY=', 'Content-Length': '0'} 2024-02-01T10:49:54.312 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006d7ffc35a5100086-0065bb7752-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:54.312 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:54.312 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:54.312 DEBUG:boto:path=/vclfcn-62/?prefix=obj2 2024-02-01T10:49:54.312 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj2 2024-02-01T10:49:54.312 DEBUG:boto:Method: GET 2024-02-01T10:49:54.313 DEBUG:boto:Path: /vclfcn-62/?prefix=obj2 2024-02-01T10:49:54.313 DEBUG:boto:Data: 2024-02-01T10:49:54.313 DEBUG:boto:Headers: {} 2024-02-01T10:49:54.313 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:54.313 DEBUG:boto:Port: 8000 2024-02-01T10:49:54.313 DEBUG:boto:Params: {} 2024-02-01T10:49:54.313 DEBUG:boto:Token: None 2024-02-01T10:49:54.313 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:54 GMT /vclfcn-62/ 2024-02-01T10:49:54.313 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HhKpJHgR2UFZClMtRS3LPwCK2sA= 2024-02-01T10:49:54.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HhKpJHgR2UFZClMtRS3LPwCK2sA=', 'Content-Length': '0'} 2024-02-01T10:49:54.318 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e80d5ce7ea6b5339-0065bb7752-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:54.318 DEBUG:boto:b'vclfcn-62obj21000falseobj22024-02-01T10:48:26.333Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:54.318 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:54.319 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:54.319 DEBUG:boto:path=/vclfcn-62/?prefix=obj2 2024-02-01T10:49:54.319 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj2 2024-02-01T10:49:54.319 DEBUG:boto:Method: GET 2024-02-01T10:49:54.319 DEBUG:boto:Path: /vclfcn-62/?prefix=obj2 2024-02-01T10:49:54.319 DEBUG:boto:Data: 2024-02-01T10:49:54.319 DEBUG:boto:Headers: {} 2024-02-01T10:49:54.319 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:54.319 DEBUG:boto:Port: 8000 2024-02-01T10:49:54.319 DEBUG:boto:Params: {} 2024-02-01T10:49:54.320 DEBUG:boto:Token: None 2024-02-01T10:49:54.320 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:54 GMT /vclfcn-62/ 2024-02-01T10:49:54.320 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HhKpJHgR2UFZClMtRS3LPwCK2sA= 2024-02-01T10:49:54.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HhKpJHgR2UFZClMtRS3LPwCK2sA=', 'Content-Length': '0'} 2024-02-01T10:49:54.324 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fba1454ec3194a14-0065bb7752-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:54.324 DEBUG:boto:b'vclfcn-62obj21000falseobj22024-02-01T10:48:26.333Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:54.325 DEBUG:boto:path=/vclfcn-62/obj2 2024-02-01T10:49:54.325 DEBUG:boto:auth_path=/vclfcn-62/obj2 2024-02-01T10:49:54.325 DEBUG:boto:Method: HEAD 2024-02-01T10:49:54.325 DEBUG:boto:Path: /vclfcn-62/obj2 2024-02-01T10:49:54.326 DEBUG:boto:Data: 2024-02-01T10:49:54.326 DEBUG:boto:Headers: {} 2024-02-01T10:49:54.326 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:54.326 DEBUG:boto:Port: 8000 2024-02-01T10:49:54.326 DEBUG:boto:Params: {} 2024-02-01T10:49:54.326 DEBUG:boto:Token: None 2024-02-01T10:49:54.326 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:54 GMT /vclfcn-62/obj2 2024-02-01T10:49:54.327 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0jSx/j3cSjMdI38Zj9YWAD5rQRA= 2024-02-01T10:49:54.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0jSx/j3cSjMdI38Zj9YWAD5rQRA=', 'Content-Length': '0'} 2024-02-01T10:49:54.329 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000083c4f1b1de9e0482-0065bb7752-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:54.330 DEBUG:boto:path=/vclfcn-62/obj2 2024-02-01T10:49:54.330 DEBUG:boto:auth_path=/vclfcn-62/obj2 2024-02-01T10:49:54.330 DEBUG:boto:Method: HEAD 2024-02-01T10:49:54.330 DEBUG:boto:Path: /vclfcn-62/obj2 2024-02-01T10:49:54.330 DEBUG:boto:Data: 2024-02-01T10:49:54.330 DEBUG:boto:Headers: {} 2024-02-01T10:49:54.330 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:54.330 DEBUG:boto:Port: 8000 2024-02-01T10:49:54.330 DEBUG:boto:Params: {} 2024-02-01T10:49:54.330 DEBUG:boto:Token: None 2024-02-01T10:49:54.331 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:54 GMT /vclfcn-62/obj2 2024-02-01T10:49:54.331 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0jSx/j3cSjMdI38Zj9YWAD5rQRA= 2024-02-01T10:49:54.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0jSx/j3cSjMdI38Zj9YWAD5rQRA=', 'Content-Length': '0'} 2024-02-01T10:49:54.333 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c13891734ccb3aec-0065bb7752-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:54.334 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:49:54.334 DEBUG:boto:path=/vclfcn-62/obj2 2024-02-01T10:49:54.334 DEBUG:boto:auth_path=/vclfcn-62/obj2 2024-02-01T10:49:54.334 DEBUG:boto:Method: GET 2024-02-01T10:49:54.334 DEBUG:boto:Path: /vclfcn-62/obj2 2024-02-01T10:49:54.334 DEBUG:boto:Data: 2024-02-01T10:49:54.334 DEBUG:boto:Headers: {} 2024-02-01T10:49:54.334 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:54.335 DEBUG:boto:Port: 8000 2024-02-01T10:49:54.335 DEBUG:boto:Params: {} 2024-02-01T10:49:54.335 DEBUG:boto:Token: None 2024-02-01T10:49:54.335 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:54 GMT /vclfcn-62/obj2 2024-02-01T10:49:54.336 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:KyNl/z1SAYWw1xz5nib0qthdNUI= 2024-02-01T10:49:54.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:KyNl/z1SAYWw1xz5nib0qthdNUI=', 'Content-Length': '0'} 2024-02-01T10:49:54.339 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aecec2325526ba98-0065bb7752-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:54.380 DEBUG:boto:path=/vclfcn-62/obj2 2024-02-01T10:49:54.380 DEBUG:boto:auth_path=/vclfcn-62/obj2 2024-02-01T10:49:54.380 DEBUG:boto:Method: GET 2024-02-01T10:49:54.381 DEBUG:boto:Path: /vclfcn-62/obj2 2024-02-01T10:49:54.381 DEBUG:boto:Data: 2024-02-01T10:49:54.381 DEBUG:boto:Headers: {} 2024-02-01T10:49:54.381 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:54.381 DEBUG:boto:Port: 8000 2024-02-01T10:49:54.381 DEBUG:boto:Params: {} 2024-02-01T10:49:54.381 DEBUG:boto:Token: None 2024-02-01T10:49:54.381 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:54 GMT /vclfcn-62/obj2 2024-02-01T10:49:54.381 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:KyNl/z1SAYWw1xz5nib0qthdNUI= 2024-02-01T10:49:54.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:KyNl/z1SAYWw1xz5nib0qthdNUI=', 'Content-Length': '0'} 2024-02-01T10:49:54.384 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b5824c84a4d4b32b-0065bb7752-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:54.385 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:54.385 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:54.385 DEBUG:boto:path=/vclfcn-62/?prefix=obj2 2024-02-01T10:49:54.385 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj2 2024-02-01T10:49:54.385 DEBUG:boto:Method: GET 2024-02-01T10:49:54.385 DEBUG:boto:Path: /vclfcn-62/?prefix=obj2 2024-02-01T10:49:54.385 DEBUG:boto:Data: 2024-02-01T10:49:54.385 DEBUG:boto:Headers: {} 2024-02-01T10:49:54.386 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:54.386 DEBUG:boto:Port: 8000 2024-02-01T10:49:54.386 DEBUG:boto:Params: {} 2024-02-01T10:49:54.386 DEBUG:boto:Token: None 2024-02-01T10:49:54.386 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:54 GMT /vclfcn-62/ 2024-02-01T10:49:54.386 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HhKpJHgR2UFZClMtRS3LPwCK2sA= 2024-02-01T10:49:54.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HhKpJHgR2UFZClMtRS3LPwCK2sA=', 'Content-Length': '0'} 2024-02-01T10:49:54.390 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095f441cf564a99d6-0065bb7752-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:54.391 DEBUG:boto:b'vclfcn-62obj21000falseobj22024-02-01T10:48:26.333Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:54.391 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:54.391 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:54.391 DEBUG:boto:path=/vclfcn-62/?prefix=obj2 2024-02-01T10:49:54.391 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj2 2024-02-01T10:49:54.391 DEBUG:boto:Method: GET 2024-02-01T10:49:54.391 DEBUG:boto:Path: /vclfcn-62/?prefix=obj2 2024-02-01T10:49:54.392 DEBUG:boto:Data: 2024-02-01T10:49:54.392 DEBUG:boto:Headers: {} 2024-02-01T10:49:54.392 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:54.392 DEBUG:boto:Port: 8000 2024-02-01T10:49:54.392 DEBUG:boto:Params: {} 2024-02-01T10:49:54.392 DEBUG:boto:Token: None 2024-02-01T10:49:54.392 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:54 GMT /vclfcn-62/ 2024-02-01T10:49:54.393 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HhKpJHgR2UFZClMtRS3LPwCK2sA= 2024-02-01T10:49:54.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HhKpJHgR2UFZClMtRS3LPwCK2sA=', 'Content-Length': '0'} 2024-02-01T10:49:54.396 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ca13b0bc7d2947c-0065bb7752-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:54.397 DEBUG:boto:b'vclfcn-62obj21000falseobj22024-02-01T10:48:26.333Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:54.397 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:49:54.397 DEBUG:boto:path=/vclfcn-62/obj3 2024-02-01T10:49:54.397 DEBUG:boto:auth_path=/vclfcn-62/obj3 2024-02-01T10:49:54.397 DEBUG:boto:path=/vclfcn-62/obj3?versionId=null 2024-02-01T10:49:54.397 DEBUG:boto:auth_path=/vclfcn-62/obj3?versionId=null 2024-02-01T10:49:54.398 DEBUG:boto:Method: GET 2024-02-01T10:49:54.398 DEBUG:boto:Path: /vclfcn-62/obj3?versionId=null 2024-02-01T10:49:54.398 DEBUG:boto:Data: 2024-02-01T10:49:54.398 DEBUG:boto:Headers: {} 2024-02-01T10:49:54.398 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:54.398 DEBUG:boto:Port: 8000 2024-02-01T10:49:54.398 DEBUG:boto:Params: {} 2024-02-01T10:49:54.399 DEBUG:boto:Token: None 2024-02-01T10:49:54.399 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:54 GMT /vclfcn-62/obj3?versionId=null 2024-02-01T10:49:54.399 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:slM2M2HnIRK/yjj90XL0i+vJMJM= 2024-02-01T10:49:54.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:slM2M2HnIRK/yjj90XL0i+vJMJM=', 'Content-Length': '0'} 2024-02-01T10:49:54.402 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ec372553800e4911-0065bb7752-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:54.448 DEBUG:boto:path=/vclfcn-62/obj3 2024-02-01T10:49:54.448 DEBUG:boto:auth_path=/vclfcn-62/obj3 2024-02-01T10:49:54.448 DEBUG:boto:path=/vclfcn-62/obj3?versionId=null 2024-02-01T10:49:54.449 DEBUG:boto:auth_path=/vclfcn-62/obj3?versionId=null 2024-02-01T10:49:54.449 DEBUG:boto:Method: GET 2024-02-01T10:49:54.449 DEBUG:boto:Path: /vclfcn-62/obj3?versionId=null 2024-02-01T10:49:54.449 DEBUG:boto:Data: 2024-02-01T10:49:54.449 DEBUG:boto:Headers: {} 2024-02-01T10:49:54.449 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:54.449 DEBUG:boto:Port: 8000 2024-02-01T10:49:54.449 DEBUG:boto:Params: {} 2024-02-01T10:49:54.449 DEBUG:boto:Token: None 2024-02-01T10:49:54.450 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:54 GMT /vclfcn-62/obj3?versionId=null 2024-02-01T10:49:54.450 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:slM2M2HnIRK/yjj90XL0i+vJMJM= 2024-02-01T10:49:54.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:slM2M2HnIRK/yjj90XL0i+vJMJM=', 'Content-Length': '0'} 2024-02-01T10:49:54.453 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e3cdd94a2fb56931-0065bb7752-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:54.453 DEBUG:boto:path=/vclfcn-62/obj3 2024-02-01T10:49:54.453 DEBUG:boto:auth_path=/vclfcn-62/obj3 2024-02-01T10:49:54.453 DEBUG:boto:path=/vclfcn-62/obj3?versionId=null 2024-02-01T10:49:54.453 DEBUG:boto:auth_path=/vclfcn-62/obj3?versionId=null 2024-02-01T10:49:54.453 DEBUG:boto:Method: HEAD 2024-02-01T10:49:54.454 DEBUG:boto:Path: /vclfcn-62/obj3?versionId=null 2024-02-01T10:49:54.454 DEBUG:boto:Data: 2024-02-01T10:49:54.454 DEBUG:boto:Headers: {} 2024-02-01T10:49:54.454 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:54.454 DEBUG:boto:Port: 8000 2024-02-01T10:49:54.454 DEBUG:boto:Params: {} 2024-02-01T10:49:54.454 DEBUG:boto:Token: None 2024-02-01T10:49:54.454 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:54 GMT /vclfcn-62/obj3?versionId=null 2024-02-01T10:49:54.455 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:qavnew7j25+56rfsidXb/cfYWqo= 2024-02-01T10:49:54.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:qavnew7j25+56rfsidXb/cfYWqo=', 'Content-Length': '0'} 2024-02-01T10:49:54.458 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c229febc9d816fa3-0065bb7752-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:54.458 DEBUG:boto:path=/vclfcn-62/obj3 2024-02-01T10:49:54.458 DEBUG:boto:auth_path=/vclfcn-62/obj3 2024-02-01T10:49:54.458 DEBUG:boto:path=/vclfcn-62/obj3?versionId=null 2024-02-01T10:49:54.458 DEBUG:boto:auth_path=/vclfcn-62/obj3?versionId=null 2024-02-01T10:49:54.458 DEBUG:boto:Method: HEAD 2024-02-01T10:49:54.458 DEBUG:boto:Path: /vclfcn-62/obj3?versionId=null 2024-02-01T10:49:54.458 DEBUG:boto:Data: 2024-02-01T10:49:54.459 DEBUG:boto:Headers: {} 2024-02-01T10:49:54.459 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:54.459 DEBUG:boto:Port: 8000 2024-02-01T10:49:54.459 DEBUG:boto:Params: {} 2024-02-01T10:49:54.459 DEBUG:boto:Token: None 2024-02-01T10:49:54.459 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:54 GMT /vclfcn-62/obj3?versionId=null 2024-02-01T10:49:54.459 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:qavnew7j25+56rfsidXb/cfYWqo= 2024-02-01T10:49:54.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:qavnew7j25+56rfsidXb/cfYWqo=', 'Content-Length': '0'} 2024-02-01T10:49:54.462 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000666b1472e66c738a-0065bb7752-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:54.463 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:49:54.463 DEBUG:boto:path=/vclfcn-62/obj3 2024-02-01T10:49:54.463 DEBUG:boto:auth_path=/vclfcn-62/obj3 2024-02-01T10:49:54.463 DEBUG:boto:path=/vclfcn-62/obj3?versionId=null 2024-02-01T10:49:54.463 DEBUG:boto:auth_path=/vclfcn-62/obj3?versionId=null 2024-02-01T10:49:54.463 DEBUG:boto:Method: GET 2024-02-01T10:49:54.463 DEBUG:boto:Path: /vclfcn-62/obj3?versionId=null 2024-02-01T10:49:54.463 DEBUG:boto:Data: 2024-02-01T10:49:54.463 DEBUG:boto:Headers: {} 2024-02-01T10:49:54.464 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:54.464 DEBUG:boto:Port: 8000 2024-02-01T10:49:54.464 DEBUG:boto:Params: {} 2024-02-01T10:49:54.464 DEBUG:boto:Token: None 2024-02-01T10:49:54.464 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:54 GMT /vclfcn-62/obj3?versionId=null 2024-02-01T10:49:54.464 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:slM2M2HnIRK/yjj90XL0i+vJMJM= 2024-02-01T10:49:54.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:slM2M2HnIRK/yjj90XL0i+vJMJM=', 'Content-Length': '0'} 2024-02-01T10:49:54.467 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003ae7bab81b89228d-0065bb7752-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:54.516 DEBUG:boto:path=/vclfcn-62/obj3 2024-02-01T10:49:54.516 DEBUG:boto:auth_path=/vclfcn-62/obj3 2024-02-01T10:49:54.516 DEBUG:boto:path=/vclfcn-62/obj3?versionId=null 2024-02-01T10:49:54.516 DEBUG:boto:auth_path=/vclfcn-62/obj3?versionId=null 2024-02-01T10:49:54.517 DEBUG:boto:Method: GET 2024-02-01T10:49:54.517 DEBUG:boto:Path: /vclfcn-62/obj3?versionId=null 2024-02-01T10:49:54.517 DEBUG:boto:Data: 2024-02-01T10:49:54.517 DEBUG:boto:Headers: {} 2024-02-01T10:49:54.517 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:54.517 DEBUG:boto:Port: 8000 2024-02-01T10:49:54.517 DEBUG:boto:Params: {} 2024-02-01T10:49:54.517 DEBUG:boto:Token: None 2024-02-01T10:49:54.517 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:54 GMT /vclfcn-62/obj3?versionId=null 2024-02-01T10:49:54.518 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:slM2M2HnIRK/yjj90XL0i+vJMJM= 2024-02-01T10:49:54.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:slM2M2HnIRK/yjj90XL0i+vJMJM=', 'Content-Length': '0'} 2024-02-01T10:49:54.521 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bd0f35b4eee3f46f-0065bb7752-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:54.521 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:54.521 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:54.521 DEBUG:boto:path=/vclfcn-62/?prefix=obj3 2024-02-01T10:49:54.521 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj3 2024-02-01T10:49:54.521 DEBUG:boto:Method: GET 2024-02-01T10:49:54.521 DEBUG:boto:Path: /vclfcn-62/?prefix=obj3 2024-02-01T10:49:54.522 DEBUG:boto:Data: 2024-02-01T10:49:54.522 DEBUG:boto:Headers: {} 2024-02-01T10:49:54.522 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:54.522 DEBUG:boto:Port: 8000 2024-02-01T10:49:54.522 DEBUG:boto:Params: {} 2024-02-01T10:49:54.522 DEBUG:boto:Token: None 2024-02-01T10:49:54.522 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:54 GMT /vclfcn-62/ 2024-02-01T10:49:54.522 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HhKpJHgR2UFZClMtRS3LPwCK2sA= 2024-02-01T10:49:54.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HhKpJHgR2UFZClMtRS3LPwCK2sA=', 'Content-Length': '0'} 2024-02-01T10:49:54.526 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091c059b30f82e7c9-0065bb7752-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:54.527 DEBUG:boto:b'vclfcn-62obj31000falseobj32024-02-01T10:48:26.384Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:54.527 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:54.527 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:54.527 DEBUG:boto:path=/vclfcn-62/?prefix=obj3 2024-02-01T10:49:54.527 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj3 2024-02-01T10:49:54.528 DEBUG:boto:Method: GET 2024-02-01T10:49:54.528 DEBUG:boto:Path: /vclfcn-62/?prefix=obj3 2024-02-01T10:49:54.528 DEBUG:boto:Data: 2024-02-01T10:49:54.528 DEBUG:boto:Headers: {} 2024-02-01T10:49:54.528 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:54.528 DEBUG:boto:Port: 8000 2024-02-01T10:49:54.528 DEBUG:boto:Params: {} 2024-02-01T10:49:54.528 DEBUG:boto:Token: None 2024-02-01T10:49:54.528 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:54 GMT /vclfcn-62/ 2024-02-01T10:49:54.528 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HhKpJHgR2UFZClMtRS3LPwCK2sA= 2024-02-01T10:49:54.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HhKpJHgR2UFZClMtRS3LPwCK2sA=', 'Content-Length': '0'} 2024-02-01T10:49:54.533 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ddfa8e77e1580ade-0065bb7752-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:54.576 DEBUG:boto:b'vclfcn-62obj31000falseobj32024-02-01T10:48:26.384Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:54.577 DEBUG:boto:path=/vclfcn-62/obj3 2024-02-01T10:49:54.577 DEBUG:boto:auth_path=/vclfcn-62/obj3 2024-02-01T10:49:54.577 DEBUG:boto:Method: HEAD 2024-02-01T10:49:54.577 DEBUG:boto:Path: /vclfcn-62/obj3 2024-02-01T10:49:54.577 DEBUG:boto:Data: 2024-02-01T10:49:54.577 DEBUG:boto:Headers: {} 2024-02-01T10:49:54.577 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:54.577 DEBUG:boto:Port: 8000 2024-02-01T10:49:54.577 DEBUG:boto:Params: {} 2024-02-01T10:49:54.578 DEBUG:boto:Token: None 2024-02-01T10:49:54.578 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:54 GMT /vclfcn-62/obj3 2024-02-01T10:49:54.578 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:bbr0FQUo8bYtBVJZsiYzXByAXOE= 2024-02-01T10:49:54.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:bbr0FQUo8bYtBVJZsiYzXByAXOE=', 'Content-Length': '0'} 2024-02-01T10:49:54.581 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002443e366b8fdd79f-0065bb7752-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:54.581 DEBUG:boto:path=/vclfcn-62/obj3 2024-02-01T10:49:54.581 DEBUG:boto:auth_path=/vclfcn-62/obj3 2024-02-01T10:49:54.581 DEBUG:boto:Method: HEAD 2024-02-01T10:49:54.581 DEBUG:boto:Path: /vclfcn-62/obj3 2024-02-01T10:49:54.581 DEBUG:boto:Data: 2024-02-01T10:49:54.581 DEBUG:boto:Headers: {} 2024-02-01T10:49:54.582 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:54.582 DEBUG:boto:Port: 8000 2024-02-01T10:49:54.582 DEBUG:boto:Params: {} 2024-02-01T10:49:54.582 DEBUG:boto:Token: None 2024-02-01T10:49:54.582 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:54 GMT /vclfcn-62/obj3 2024-02-01T10:49:54.582 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:bbr0FQUo8bYtBVJZsiYzXByAXOE= 2024-02-01T10:49:54.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:bbr0FQUo8bYtBVJZsiYzXByAXOE=', 'Content-Length': '0'} 2024-02-01T10:49:54.585 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009a1bc80d43692605-0065bb7752-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:54.585 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:49:54.585 DEBUG:boto:path=/vclfcn-62/obj3 2024-02-01T10:49:54.586 DEBUG:boto:auth_path=/vclfcn-62/obj3 2024-02-01T10:49:54.586 DEBUG:boto:Method: GET 2024-02-01T10:49:54.586 DEBUG:boto:Path: /vclfcn-62/obj3 2024-02-01T10:49:54.586 DEBUG:boto:Data: 2024-02-01T10:49:54.586 DEBUG:boto:Headers: {} 2024-02-01T10:49:54.586 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:54.587 DEBUG:boto:Port: 8000 2024-02-01T10:49:54.587 DEBUG:boto:Params: {} 2024-02-01T10:49:54.587 DEBUG:boto:Token: None 2024-02-01T10:49:54.587 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:54 GMT /vclfcn-62/obj3 2024-02-01T10:49:54.587 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:eFfqv1Wdg6IEaiWAdYI7TY9YO2s= 2024-02-01T10:49:54.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:eFfqv1Wdg6IEaiWAdYI7TY9YO2s=', 'Content-Length': '0'} 2024-02-01T10:49:54.590 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008868e2ac1a51e12e-0065bb7752-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:54.632 DEBUG:boto:path=/vclfcn-62/obj3 2024-02-01T10:49:54.632 DEBUG:boto:auth_path=/vclfcn-62/obj3 2024-02-01T10:49:54.632 DEBUG:boto:Method: GET 2024-02-01T10:49:54.633 DEBUG:boto:Path: /vclfcn-62/obj3 2024-02-01T10:49:54.633 DEBUG:boto:Data: 2024-02-01T10:49:54.633 DEBUG:boto:Headers: {} 2024-02-01T10:49:54.633 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:54.633 DEBUG:boto:Port: 8000 2024-02-01T10:49:54.633 DEBUG:boto:Params: {} 2024-02-01T10:49:54.633 DEBUG:boto:Token: None 2024-02-01T10:49:54.633 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:54 GMT /vclfcn-62/obj3 2024-02-01T10:49:54.633 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:eFfqv1Wdg6IEaiWAdYI7TY9YO2s= 2024-02-01T10:49:54.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:eFfqv1Wdg6IEaiWAdYI7TY9YO2s=', 'Content-Length': '0'} 2024-02-01T10:49:54.636 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e5c0806e6b61693e-0065bb7752-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:54.637 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:54.637 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:54.637 DEBUG:boto:path=/vclfcn-62/?prefix=obj3 2024-02-01T10:49:54.637 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj3 2024-02-01T10:49:54.637 DEBUG:boto:Method: GET 2024-02-01T10:49:54.637 DEBUG:boto:Path: /vclfcn-62/?prefix=obj3 2024-02-01T10:49:54.637 DEBUG:boto:Data: 2024-02-01T10:49:54.637 DEBUG:boto:Headers: {} 2024-02-01T10:49:54.637 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:54.637 DEBUG:boto:Port: 8000 2024-02-01T10:49:54.638 DEBUG:boto:Params: {} 2024-02-01T10:49:54.638 DEBUG:boto:Token: None 2024-02-01T10:49:54.638 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:54 GMT /vclfcn-62/ 2024-02-01T10:49:54.638 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HhKpJHgR2UFZClMtRS3LPwCK2sA= 2024-02-01T10:49:54.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HhKpJHgR2UFZClMtRS3LPwCK2sA=', 'Content-Length': '0'} 2024-02-01T10:49:54.642 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008000e1b773c432bc-0065bb7752-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:54.684 DEBUG:boto:b'vclfcn-62obj31000falseobj32024-02-01T10:48:26.384Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:54.685 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:54.685 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:54.685 DEBUG:boto:path=/vclfcn-62/?prefix=obj3 2024-02-01T10:49:54.685 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj3 2024-02-01T10:49:54.685 DEBUG:boto:Method: GET 2024-02-01T10:49:54.685 DEBUG:boto:Path: /vclfcn-62/?prefix=obj3 2024-02-01T10:49:54.685 DEBUG:boto:Data: 2024-02-01T10:49:54.685 DEBUG:boto:Headers: {} 2024-02-01T10:49:54.685 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:54.685 DEBUG:boto:Port: 8000 2024-02-01T10:49:54.686 DEBUG:boto:Params: {} 2024-02-01T10:49:54.686 DEBUG:boto:Token: None 2024-02-01T10:49:54.686 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:54 GMT /vclfcn-62/ 2024-02-01T10:49:54.686 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HhKpJHgR2UFZClMtRS3LPwCK2sA= 2024-02-01T10:49:54.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HhKpJHgR2UFZClMtRS3LPwCK2sA=', 'Content-Length': '0'} 2024-02-01T10:49:54.691 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f618ef19e599a34-0065bb7752-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:54.692 DEBUG:boto:b'vclfcn-62obj31000falseobj32024-02-01T10:48:26.384Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:54.692 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:49:54.692 DEBUG:boto:path=/vclfcn-62/obj4 2024-02-01T10:49:54.692 DEBUG:boto:auth_path=/vclfcn-62/obj4 2024-02-01T10:49:54.693 DEBUG:boto:path=/vclfcn-62/obj4?versionId=null 2024-02-01T10:49:54.693 DEBUG:boto:auth_path=/vclfcn-62/obj4?versionId=null 2024-02-01T10:49:54.693 DEBUG:boto:Method: GET 2024-02-01T10:49:54.693 DEBUG:boto:Path: /vclfcn-62/obj4?versionId=null 2024-02-01T10:49:54.693 DEBUG:boto:Data: 2024-02-01T10:49:54.693 DEBUG:boto:Headers: {} 2024-02-01T10:49:54.693 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:54.693 DEBUG:boto:Port: 8000 2024-02-01T10:49:54.693 DEBUG:boto:Params: {} 2024-02-01T10:49:54.693 DEBUG:boto:Token: None 2024-02-01T10:49:54.694 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:54 GMT /vclfcn-62/obj4?versionId=null 2024-02-01T10:49:54.694 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:oTqg8l9jIDHDLdGYn4KvY0X1aOE= 2024-02-01T10:49:54.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:oTqg8l9jIDHDLdGYn4KvY0X1aOE=', 'Content-Length': '0'} 2024-02-01T10:49:54.697 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000060194a34b849fb7d-0065bb7752-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:54.740 DEBUG:boto:path=/vclfcn-62/obj4 2024-02-01T10:49:54.740 DEBUG:boto:auth_path=/vclfcn-62/obj4 2024-02-01T10:49:54.740 DEBUG:boto:path=/vclfcn-62/obj4?versionId=null 2024-02-01T10:49:54.741 DEBUG:boto:auth_path=/vclfcn-62/obj4?versionId=null 2024-02-01T10:49:54.741 DEBUG:boto:Method: GET 2024-02-01T10:49:54.741 DEBUG:boto:Path: /vclfcn-62/obj4?versionId=null 2024-02-01T10:49:54.741 DEBUG:boto:Data: 2024-02-01T10:49:54.741 DEBUG:boto:Headers: {} 2024-02-01T10:49:54.741 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:54.741 DEBUG:boto:Port: 8000 2024-02-01T10:49:54.741 DEBUG:boto:Params: {} 2024-02-01T10:49:54.742 DEBUG:boto:Token: None 2024-02-01T10:49:54.742 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:54 GMT /vclfcn-62/obj4?versionId=null 2024-02-01T10:49:54.742 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:oTqg8l9jIDHDLdGYn4KvY0X1aOE= 2024-02-01T10:49:54.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:oTqg8l9jIDHDLdGYn4KvY0X1aOE=', 'Content-Length': '0'} 2024-02-01T10:49:54.745 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000018b821e4de925633-0065bb7752-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:54.745 DEBUG:boto:path=/vclfcn-62/obj4 2024-02-01T10:49:54.746 DEBUG:boto:auth_path=/vclfcn-62/obj4 2024-02-01T10:49:54.746 DEBUG:boto:path=/vclfcn-62/obj4?versionId=null 2024-02-01T10:49:54.746 DEBUG:boto:auth_path=/vclfcn-62/obj4?versionId=null 2024-02-01T10:49:54.746 DEBUG:boto:Method: HEAD 2024-02-01T10:49:54.746 DEBUG:boto:Path: /vclfcn-62/obj4?versionId=null 2024-02-01T10:49:54.746 DEBUG:boto:Data: 2024-02-01T10:49:54.746 DEBUG:boto:Headers: {} 2024-02-01T10:49:54.746 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:54.746 DEBUG:boto:Port: 8000 2024-02-01T10:49:54.746 DEBUG:boto:Params: {} 2024-02-01T10:49:54.747 DEBUG:boto:Token: None 2024-02-01T10:49:54.747 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:54 GMT /vclfcn-62/obj4?versionId=null 2024-02-01T10:49:54.747 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lOPwEitiWpQxfF7MBsScunbEHOk= 2024-02-01T10:49:54.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lOPwEitiWpQxfF7MBsScunbEHOk=', 'Content-Length': '0'} 2024-02-01T10:49:54.750 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000049034b62ca888375-0065bb7752-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:54.750 DEBUG:boto:path=/vclfcn-62/obj4 2024-02-01T10:49:54.751 DEBUG:boto:auth_path=/vclfcn-62/obj4 2024-02-01T10:49:54.751 DEBUG:boto:path=/vclfcn-62/obj4?versionId=null 2024-02-01T10:49:54.751 DEBUG:boto:auth_path=/vclfcn-62/obj4?versionId=null 2024-02-01T10:49:54.751 DEBUG:boto:Method: HEAD 2024-02-01T10:49:54.751 DEBUG:boto:Path: /vclfcn-62/obj4?versionId=null 2024-02-01T10:49:54.751 DEBUG:boto:Data: 2024-02-01T10:49:54.751 DEBUG:boto:Headers: {} 2024-02-01T10:49:54.751 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:54.751 DEBUG:boto:Port: 8000 2024-02-01T10:49:54.751 DEBUG:boto:Params: {} 2024-02-01T10:49:54.752 DEBUG:boto:Token: None 2024-02-01T10:49:54.752 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:54 GMT /vclfcn-62/obj4?versionId=null 2024-02-01T10:49:54.752 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lOPwEitiWpQxfF7MBsScunbEHOk= 2024-02-01T10:49:54.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lOPwEitiWpQxfF7MBsScunbEHOk=', 'Content-Length': '0'} 2024-02-01T10:49:54.755 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a6fac2c8962b9fec-0065bb7752-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:54.755 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:49:54.755 DEBUG:boto:path=/vclfcn-62/obj4 2024-02-01T10:49:54.755 DEBUG:boto:auth_path=/vclfcn-62/obj4 2024-02-01T10:49:54.755 DEBUG:boto:path=/vclfcn-62/obj4?versionId=null 2024-02-01T10:49:54.755 DEBUG:boto:auth_path=/vclfcn-62/obj4?versionId=null 2024-02-01T10:49:54.756 DEBUG:boto:Method: GET 2024-02-01T10:49:54.756 DEBUG:boto:Path: /vclfcn-62/obj4?versionId=null 2024-02-01T10:49:54.756 DEBUG:boto:Data: 2024-02-01T10:49:54.756 DEBUG:boto:Headers: {} 2024-02-01T10:49:54.756 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:54.756 DEBUG:boto:Port: 8000 2024-02-01T10:49:54.756 DEBUG:boto:Params: {} 2024-02-01T10:49:54.756 DEBUG:boto:Token: None 2024-02-01T10:49:54.756 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:54 GMT /vclfcn-62/obj4?versionId=null 2024-02-01T10:49:54.757 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:oTqg8l9jIDHDLdGYn4KvY0X1aOE= 2024-02-01T10:49:54.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:oTqg8l9jIDHDLdGYn4KvY0X1aOE=', 'Content-Length': '0'} 2024-02-01T10:49:54.759 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000013a4db2009c90d0c-0065bb7752-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:54.800 DEBUG:boto:path=/vclfcn-62/obj4 2024-02-01T10:49:54.800 DEBUG:boto:auth_path=/vclfcn-62/obj4 2024-02-01T10:49:54.801 DEBUG:boto:path=/vclfcn-62/obj4?versionId=null 2024-02-01T10:49:54.801 DEBUG:boto:auth_path=/vclfcn-62/obj4?versionId=null 2024-02-01T10:49:54.801 DEBUG:boto:Method: GET 2024-02-01T10:49:54.801 DEBUG:boto:Path: /vclfcn-62/obj4?versionId=null 2024-02-01T10:49:54.801 DEBUG:boto:Data: 2024-02-01T10:49:54.802 DEBUG:boto:Headers: {} 2024-02-01T10:49:54.802 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:54.802 DEBUG:boto:Port: 8000 2024-02-01T10:49:54.802 DEBUG:boto:Params: {} 2024-02-01T10:49:54.802 DEBUG:boto:Token: None 2024-02-01T10:49:54.803 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:54 GMT /vclfcn-62/obj4?versionId=null 2024-02-01T10:49:54.803 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:oTqg8l9jIDHDLdGYn4KvY0X1aOE= 2024-02-01T10:49:54.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:oTqg8l9jIDHDLdGYn4KvY0X1aOE=', 'Content-Length': '0'} 2024-02-01T10:49:54.806 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a329510ece6e8e3d-0065bb7752-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:54.807 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:54.807 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:54.807 DEBUG:boto:path=/vclfcn-62/?prefix=obj4 2024-02-01T10:49:54.808 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj4 2024-02-01T10:49:54.808 DEBUG:boto:Method: GET 2024-02-01T10:49:54.808 DEBUG:boto:Path: /vclfcn-62/?prefix=obj4 2024-02-01T10:49:54.808 DEBUG:boto:Data: 2024-02-01T10:49:54.808 DEBUG:boto:Headers: {} 2024-02-01T10:49:54.809 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:54.809 DEBUG:boto:Port: 8000 2024-02-01T10:49:54.809 DEBUG:boto:Params: {} 2024-02-01T10:49:54.809 DEBUG:boto:Token: None 2024-02-01T10:49:54.809 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:54 GMT /vclfcn-62/ 2024-02-01T10:49:54.810 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HhKpJHgR2UFZClMtRS3LPwCK2sA= 2024-02-01T10:49:54.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HhKpJHgR2UFZClMtRS3LPwCK2sA=', 'Content-Length': '0'} 2024-02-01T10:49:54.814 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4d499c15025e13b-0065bb7752-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:54.814 DEBUG:boto:b'vclfcn-62obj41000falseobj42024-02-01T10:48:26.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:54.815 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:54.815 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:54.815 DEBUG:boto:path=/vclfcn-62/?prefix=obj4 2024-02-01T10:49:54.815 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj4 2024-02-01T10:49:54.815 DEBUG:boto:Method: GET 2024-02-01T10:49:54.815 DEBUG:boto:Path: /vclfcn-62/?prefix=obj4 2024-02-01T10:49:54.815 DEBUG:boto:Data: 2024-02-01T10:49:54.815 DEBUG:boto:Headers: {} 2024-02-01T10:49:54.816 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:54.816 DEBUG:boto:Port: 8000 2024-02-01T10:49:54.816 DEBUG:boto:Params: {} 2024-02-01T10:49:54.816 DEBUG:boto:Token: None 2024-02-01T10:49:54.816 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:54 GMT /vclfcn-62/ 2024-02-01T10:49:54.816 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HhKpJHgR2UFZClMtRS3LPwCK2sA= 2024-02-01T10:49:54.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HhKpJHgR2UFZClMtRS3LPwCK2sA=', 'Content-Length': '0'} 2024-02-01T10:49:54.820 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7fb8dcac8485587-0065bb7752-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:54.821 DEBUG:boto:b'vclfcn-62obj41000falseobj42024-02-01T10:48:26.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:54.821 DEBUG:boto:path=/vclfcn-62/obj4 2024-02-01T10:49:54.821 DEBUG:boto:auth_path=/vclfcn-62/obj4 2024-02-01T10:49:54.822 DEBUG:boto:Method: HEAD 2024-02-01T10:49:54.822 DEBUG:boto:Path: /vclfcn-62/obj4 2024-02-01T10:49:54.822 DEBUG:boto:Data: 2024-02-01T10:49:54.822 DEBUG:boto:Headers: {} 2024-02-01T10:49:54.822 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:54.823 DEBUG:boto:Port: 8000 2024-02-01T10:49:54.823 DEBUG:boto:Params: {} 2024-02-01T10:49:54.823 DEBUG:boto:Token: None 2024-02-01T10:49:54.823 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:54 GMT /vclfcn-62/obj4 2024-02-01T10:49:54.823 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:P8BS1XQbNN7VY4CS5/W1OBbSKAY= 2024-02-01T10:49:54.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:P8BS1XQbNN7VY4CS5/W1OBbSKAY=', 'Content-Length': '0'} 2024-02-01T10:49:54.826 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000026e59daa3e0e3c55-0065bb7752-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:54.826 DEBUG:boto:path=/vclfcn-62/obj4 2024-02-01T10:49:54.827 DEBUG:boto:auth_path=/vclfcn-62/obj4 2024-02-01T10:49:54.827 DEBUG:boto:Method: HEAD 2024-02-01T10:49:54.827 DEBUG:boto:Path: /vclfcn-62/obj4 2024-02-01T10:49:54.827 DEBUG:boto:Data: 2024-02-01T10:49:54.827 DEBUG:boto:Headers: {} 2024-02-01T10:49:54.827 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:54.827 DEBUG:boto:Port: 8000 2024-02-01T10:49:54.827 DEBUG:boto:Params: {} 2024-02-01T10:49:54.827 DEBUG:boto:Token: None 2024-02-01T10:49:54.827 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:54 GMT /vclfcn-62/obj4 2024-02-01T10:49:54.828 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:P8BS1XQbNN7VY4CS5/W1OBbSKAY= 2024-02-01T10:49:54.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:P8BS1XQbNN7VY4CS5/W1OBbSKAY=', 'Content-Length': '0'} 2024-02-01T10:49:54.830 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000727bba396810fecb-0065bb7752-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:54.831 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:49:54.831 DEBUG:boto:path=/vclfcn-62/obj4 2024-02-01T10:49:54.831 DEBUG:boto:auth_path=/vclfcn-62/obj4 2024-02-01T10:49:54.831 DEBUG:boto:Method: GET 2024-02-01T10:49:54.831 DEBUG:boto:Path: /vclfcn-62/obj4 2024-02-01T10:49:54.832 DEBUG:boto:Data: 2024-02-01T10:49:54.832 DEBUG:boto:Headers: {} 2024-02-01T10:49:54.832 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:54.832 DEBUG:boto:Port: 8000 2024-02-01T10:49:54.832 DEBUG:boto:Params: {} 2024-02-01T10:49:54.832 DEBUG:boto:Token: None 2024-02-01T10:49:54.832 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:54 GMT /vclfcn-62/obj4 2024-02-01T10:49:54.833 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:dpvbyRQqBsOkUTiIQ5t0T60SG4E= 2024-02-01T10:49:54.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dpvbyRQqBsOkUTiIQ5t0T60SG4E=', 'Content-Length': '0'} 2024-02-01T10:49:54.835 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004a24887aa067758e-0065bb7752-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:54.880 DEBUG:boto:path=/vclfcn-62/obj4 2024-02-01T10:49:54.880 DEBUG:boto:auth_path=/vclfcn-62/obj4 2024-02-01T10:49:54.880 DEBUG:boto:Method: GET 2024-02-01T10:49:54.881 DEBUG:boto:Path: /vclfcn-62/obj4 2024-02-01T10:49:54.881 DEBUG:boto:Data: 2024-02-01T10:49:54.881 DEBUG:boto:Headers: {} 2024-02-01T10:49:54.881 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:54.881 DEBUG:boto:Port: 8000 2024-02-01T10:49:54.881 DEBUG:boto:Params: {} 2024-02-01T10:49:54.881 DEBUG:boto:Token: None 2024-02-01T10:49:54.881 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:54 GMT /vclfcn-62/obj4 2024-02-01T10:49:54.882 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:dpvbyRQqBsOkUTiIQ5t0T60SG4E= 2024-02-01T10:49:54.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dpvbyRQqBsOkUTiIQ5t0T60SG4E=', 'Content-Length': '0'} 2024-02-01T10:49:54.885 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cb0fb1233a317de4-0065bb7752-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:54.885 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:54.885 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:54.885 DEBUG:boto:path=/vclfcn-62/?prefix=obj4 2024-02-01T10:49:54.885 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj4 2024-02-01T10:49:54.885 DEBUG:boto:Method: GET 2024-02-01T10:49:54.886 DEBUG:boto:Path: /vclfcn-62/?prefix=obj4 2024-02-01T10:49:54.886 DEBUG:boto:Data: 2024-02-01T10:49:54.886 DEBUG:boto:Headers: {} 2024-02-01T10:49:54.886 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:54.886 DEBUG:boto:Port: 8000 2024-02-01T10:49:54.886 DEBUG:boto:Params: {} 2024-02-01T10:49:54.886 DEBUG:boto:Token: None 2024-02-01T10:49:54.886 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:54 GMT /vclfcn-62/ 2024-02-01T10:49:54.886 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HhKpJHgR2UFZClMtRS3LPwCK2sA= 2024-02-01T10:49:54.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HhKpJHgR2UFZClMtRS3LPwCK2sA=', 'Content-Length': '0'} 2024-02-01T10:49:54.891 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000858caa9c95f9bb44-0065bb7752-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:54.891 DEBUG:boto:b'vclfcn-62obj41000falseobj42024-02-01T10:48:26.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:54.891 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:54.891 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:54.891 DEBUG:boto:path=/vclfcn-62/?prefix=obj4 2024-02-01T10:49:54.892 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj4 2024-02-01T10:49:54.892 DEBUG:boto:Method: GET 2024-02-01T10:49:54.892 DEBUG:boto:Path: /vclfcn-62/?prefix=obj4 2024-02-01T10:49:54.892 DEBUG:boto:Data: 2024-02-01T10:49:54.892 DEBUG:boto:Headers: {} 2024-02-01T10:49:54.892 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:54.892 DEBUG:boto:Port: 8000 2024-02-01T10:49:54.892 DEBUG:boto:Params: {} 2024-02-01T10:49:54.892 DEBUG:boto:Token: None 2024-02-01T10:49:54.892 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:54 GMT /vclfcn-62/ 2024-02-01T10:49:54.893 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HhKpJHgR2UFZClMtRS3LPwCK2sA= 2024-02-01T10:49:54.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HhKpJHgR2UFZClMtRS3LPwCK2sA=', 'Content-Length': '0'} 2024-02-01T10:49:54.897 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f7a9c9f9798f141-0065bb7752-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:54.897 DEBUG:boto:b'vclfcn-62obj41000falseobj42024-02-01T10:48:26.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:54.898 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-02-01T10:49:54.898 DEBUG:boto:path=/vclfcn-62/obj5 2024-02-01T10:49:54.898 DEBUG:boto:auth_path=/vclfcn-62/obj5 2024-02-01T10:49:54.898 DEBUG:boto:path=/vclfcn-62/obj5?versionId=null 2024-02-01T10:49:54.898 DEBUG:boto:auth_path=/vclfcn-62/obj5?versionId=null 2024-02-01T10:49:54.898 DEBUG:boto:Method: GET 2024-02-01T10:49:54.898 DEBUG:boto:Path: /vclfcn-62/obj5?versionId=null 2024-02-01T10:49:54.898 DEBUG:boto:Data: 2024-02-01T10:49:54.899 DEBUG:boto:Headers: {} 2024-02-01T10:49:54.899 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:54.899 DEBUG:boto:Port: 8000 2024-02-01T10:49:54.899 DEBUG:boto:Params: {} 2024-02-01T10:49:54.899 DEBUG:boto:Token: None 2024-02-01T10:49:54.899 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:54 GMT /vclfcn-62/obj5?versionId=null 2024-02-01T10:49:54.899 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fC/3yCtWiuK1gI53z+E9feNEIX4= 2024-02-01T10:49:54.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fC/3yCtWiuK1gI53z+E9feNEIX4=', 'Content-Length': '0'} 2024-02-01T10:49:54.903 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000038330c4a5e0eb8d-0065bb7752-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:54.944 DEBUG:boto:path=/vclfcn-62/obj5 2024-02-01T10:49:54.944 DEBUG:boto:auth_path=/vclfcn-62/obj5 2024-02-01T10:49:54.944 DEBUG:boto:path=/vclfcn-62/obj5?versionId=null 2024-02-01T10:49:54.945 DEBUG:boto:auth_path=/vclfcn-62/obj5?versionId=null 2024-02-01T10:49:54.945 DEBUG:boto:Method: GET 2024-02-01T10:49:54.945 DEBUG:boto:Path: /vclfcn-62/obj5?versionId=null 2024-02-01T10:49:54.945 DEBUG:boto:Data: 2024-02-01T10:49:54.945 DEBUG:boto:Headers: {} 2024-02-01T10:49:54.945 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:54.945 DEBUG:boto:Port: 8000 2024-02-01T10:49:54.945 DEBUG:boto:Params: {} 2024-02-01T10:49:54.945 DEBUG:boto:Token: None 2024-02-01T10:49:54.946 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:54 GMT /vclfcn-62/obj5?versionId=null 2024-02-01T10:49:54.946 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fC/3yCtWiuK1gI53z+E9feNEIX4= 2024-02-01T10:49:54.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fC/3yCtWiuK1gI53z+E9feNEIX4=', 'Content-Length': '0'} 2024-02-01T10:49:54.949 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007576780993ad4ff9-0065bb7752-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:54.949 DEBUG:boto:path=/vclfcn-62/obj5 2024-02-01T10:49:54.949 DEBUG:boto:auth_path=/vclfcn-62/obj5 2024-02-01T10:49:54.949 DEBUG:boto:path=/vclfcn-62/obj5?versionId=null 2024-02-01T10:49:54.949 DEBUG:boto:auth_path=/vclfcn-62/obj5?versionId=null 2024-02-01T10:49:54.949 DEBUG:boto:Method: HEAD 2024-02-01T10:49:54.950 DEBUG:boto:Path: /vclfcn-62/obj5?versionId=null 2024-02-01T10:49:54.950 DEBUG:boto:Data: 2024-02-01T10:49:54.950 DEBUG:boto:Headers: {} 2024-02-01T10:49:54.950 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:54.950 DEBUG:boto:Port: 8000 2024-02-01T10:49:54.950 DEBUG:boto:Params: {} 2024-02-01T10:49:54.950 DEBUG:boto:Token: None 2024-02-01T10:49:54.950 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:54 GMT /vclfcn-62/obj5?versionId=null 2024-02-01T10:49:54.950 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:bpILhS1ysFxNs46ZMLcsFTjmd6Y= 2024-02-01T10:49:54.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:bpILhS1ysFxNs46ZMLcsFTjmd6Y=', 'Content-Length': '0'} 2024-02-01T10:49:54.953 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000904c470d3e6a904f-0065bb7752-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:54.953 DEBUG:boto:path=/vclfcn-62/obj5 2024-02-01T10:49:54.953 DEBUG:boto:auth_path=/vclfcn-62/obj5 2024-02-01T10:49:54.953 DEBUG:boto:path=/vclfcn-62/obj5?versionId=null 2024-02-01T10:49:54.953 DEBUG:boto:auth_path=/vclfcn-62/obj5?versionId=null 2024-02-01T10:49:54.954 DEBUG:boto:Method: HEAD 2024-02-01T10:49:54.954 DEBUG:boto:Path: /vclfcn-62/obj5?versionId=null 2024-02-01T10:49:54.954 DEBUG:boto:Data: 2024-02-01T10:49:54.954 DEBUG:boto:Headers: {} 2024-02-01T10:49:54.954 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:54.954 DEBUG:boto:Port: 8000 2024-02-01T10:49:54.954 DEBUG:boto:Params: {} 2024-02-01T10:49:54.954 DEBUG:boto:Token: None 2024-02-01T10:49:54.954 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:54 GMT /vclfcn-62/obj5?versionId=null 2024-02-01T10:49:54.954 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:bpILhS1ysFxNs46ZMLcsFTjmd6Y= 2024-02-01T10:49:54.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:bpILhS1ysFxNs46ZMLcsFTjmd6Y=', 'Content-Length': '0'} 2024-02-01T10:49:54.957 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006f892c20ef3d5f30-0065bb7752-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:54.957 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-02-01T10:49:54.958 DEBUG:boto:path=/vclfcn-62/obj5 2024-02-01T10:49:54.958 DEBUG:boto:auth_path=/vclfcn-62/obj5 2024-02-01T10:49:54.958 DEBUG:boto:path=/vclfcn-62/obj5?versionId=null 2024-02-01T10:49:54.958 DEBUG:boto:auth_path=/vclfcn-62/obj5?versionId=null 2024-02-01T10:49:54.958 DEBUG:boto:Method: GET 2024-02-01T10:49:54.958 DEBUG:boto:Path: /vclfcn-62/obj5?versionId=null 2024-02-01T10:49:54.958 DEBUG:boto:Data: 2024-02-01T10:49:54.958 DEBUG:boto:Headers: {} 2024-02-01T10:49:54.958 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:54.958 DEBUG:boto:Port: 8000 2024-02-01T10:49:54.959 DEBUG:boto:Params: {} 2024-02-01T10:49:54.959 DEBUG:boto:Token: None 2024-02-01T10:49:54.959 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:54 GMT /vclfcn-62/obj5?versionId=null 2024-02-01T10:49:54.959 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fC/3yCtWiuK1gI53z+E9feNEIX4= 2024-02-01T10:49:54.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:54 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fC/3yCtWiuK1gI53z+E9feNEIX4=', 'Content-Length': '0'} 2024-02-01T10:49:54.962 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002c394c071a783fae-0065bb7752-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:54 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:55.008 DEBUG:boto:path=/vclfcn-62/obj5 2024-02-01T10:49:55.008 DEBUG:boto:auth_path=/vclfcn-62/obj5 2024-02-01T10:49:55.009 DEBUG:boto:path=/vclfcn-62/obj5?versionId=null 2024-02-01T10:49:55.009 DEBUG:boto:auth_path=/vclfcn-62/obj5?versionId=null 2024-02-01T10:49:55.009 DEBUG:boto:Method: GET 2024-02-01T10:49:55.009 DEBUG:boto:Path: /vclfcn-62/obj5?versionId=null 2024-02-01T10:49:55.009 DEBUG:boto:Data: 2024-02-01T10:49:55.009 DEBUG:boto:Headers: {} 2024-02-01T10:49:55.009 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:55.009 DEBUG:boto:Port: 8000 2024-02-01T10:49:55.009 DEBUG:boto:Params: {} 2024-02-01T10:49:55.009 DEBUG:boto:Token: None 2024-02-01T10:49:55.010 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:55 GMT /vclfcn-62/obj5?versionId=null 2024-02-01T10:49:55.010 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:VP1tRfiDEPWabVJ/hEyLeypRr9c= 2024-02-01T10:49:55.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:VP1tRfiDEPWabVJ/hEyLeypRr9c=', 'Content-Length': '0'} 2024-02-01T10:49:55.013 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000390588495057bf88-0065bb7753-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:55.014 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:55.014 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:55.014 DEBUG:boto:path=/vclfcn-62/?prefix=obj5 2024-02-01T10:49:55.014 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj5 2024-02-01T10:49:55.015 DEBUG:boto:Method: GET 2024-02-01T10:49:55.015 DEBUG:boto:Path: /vclfcn-62/?prefix=obj5 2024-02-01T10:49:55.015 DEBUG:boto:Data: 2024-02-01T10:49:55.015 DEBUG:boto:Headers: {} 2024-02-01T10:49:55.015 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:55.016 DEBUG:boto:Port: 8000 2024-02-01T10:49:55.016 DEBUG:boto:Params: {} 2024-02-01T10:49:55.016 DEBUG:boto:Token: None 2024-02-01T10:49:55.016 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:55 GMT /vclfcn-62/ 2024-02-01T10:49:55.017 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:u/xE24Ir1FtV6T/cC18snd7v6wM= 2024-02-01T10:49:55.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:u/xE24Ir1FtV6T/cC18snd7v6wM=', 'Content-Length': '0'} 2024-02-01T10:49:55.021 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000718787822b866e45-0065bb7753-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:55.021 DEBUG:boto:b'vclfcn-62obj51000falseobj52024-02-01T10:49:22.123Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:55.022 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:55.022 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:55.022 DEBUG:boto:path=/vclfcn-62/?prefix=obj5 2024-02-01T10:49:55.022 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj5 2024-02-01T10:49:55.022 DEBUG:boto:Method: GET 2024-02-01T10:49:55.022 DEBUG:boto:Path: /vclfcn-62/?prefix=obj5 2024-02-01T10:49:55.022 DEBUG:boto:Data: 2024-02-01T10:49:55.023 DEBUG:boto:Headers: {} 2024-02-01T10:49:55.023 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:55.023 DEBUG:boto:Port: 8000 2024-02-01T10:49:55.023 DEBUG:boto:Params: {} 2024-02-01T10:49:55.023 DEBUG:boto:Token: None 2024-02-01T10:49:55.023 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:55 GMT /vclfcn-62/ 2024-02-01T10:49:55.023 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:u/xE24Ir1FtV6T/cC18snd7v6wM= 2024-02-01T10:49:55.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:u/xE24Ir1FtV6T/cC18snd7v6wM=', 'Content-Length': '0'} 2024-02-01T10:49:55.028 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba5f878941fb569b-0065bb7753-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:55.072 DEBUG:boto:b'vclfcn-62obj51000falseobj52024-02-01T10:49:22.123Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:55.073 DEBUG:boto:path=/vclfcn-62/obj5 2024-02-01T10:49:55.073 DEBUG:boto:auth_path=/vclfcn-62/obj5 2024-02-01T10:49:55.073 DEBUG:boto:Method: HEAD 2024-02-01T10:49:55.073 DEBUG:boto:Path: /vclfcn-62/obj5 2024-02-01T10:49:55.073 DEBUG:boto:Data: 2024-02-01T10:49:55.073 DEBUG:boto:Headers: {} 2024-02-01T10:49:55.073 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:55.074 DEBUG:boto:Port: 8000 2024-02-01T10:49:55.074 DEBUG:boto:Params: {} 2024-02-01T10:49:55.074 DEBUG:boto:Token: None 2024-02-01T10:49:55.074 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:55 GMT /vclfcn-62/obj5 2024-02-01T10:49:55.074 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cwAC+6ihwtW0e6BxloZRMcVzhx0= 2024-02-01T10:49:55.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cwAC+6ihwtW0e6BxloZRMcVzhx0=', 'Content-Length': '0'} 2024-02-01T10:49:55.077 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b86fbb5cfd2f8a41-0065bb7753-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:55.077 DEBUG:boto:path=/vclfcn-62/obj5 2024-02-01T10:49:55.077 DEBUG:boto:auth_path=/vclfcn-62/obj5 2024-02-01T10:49:55.078 DEBUG:boto:Method: HEAD 2024-02-01T10:49:55.078 DEBUG:boto:Path: /vclfcn-62/obj5 2024-02-01T10:49:55.078 DEBUG:boto:Data: 2024-02-01T10:49:55.078 DEBUG:boto:Headers: {} 2024-02-01T10:49:55.078 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:55.078 DEBUG:boto:Port: 8000 2024-02-01T10:49:55.078 DEBUG:boto:Params: {} 2024-02-01T10:49:55.078 DEBUG:boto:Token: None 2024-02-01T10:49:55.078 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:55 GMT /vclfcn-62/obj5 2024-02-01T10:49:55.078 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cwAC+6ihwtW0e6BxloZRMcVzhx0= 2024-02-01T10:49:55.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cwAC+6ihwtW0e6BxloZRMcVzhx0=', 'Content-Length': '0'} 2024-02-01T10:49:55.081 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000074b158787b8afc45-0065bb7753-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:55.082 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-02-01T10:49:55.082 DEBUG:boto:path=/vclfcn-62/obj5 2024-02-01T10:49:55.082 DEBUG:boto:auth_path=/vclfcn-62/obj5 2024-02-01T10:49:55.082 DEBUG:boto:Method: GET 2024-02-01T10:49:55.082 DEBUG:boto:Path: /vclfcn-62/obj5 2024-02-01T10:49:55.082 DEBUG:boto:Data: 2024-02-01T10:49:55.082 DEBUG:boto:Headers: {} 2024-02-01T10:49:55.082 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:55.082 DEBUG:boto:Port: 8000 2024-02-01T10:49:55.082 DEBUG:boto:Params: {} 2024-02-01T10:49:55.083 DEBUG:boto:Token: None 2024-02-01T10:49:55.083 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:55 GMT /vclfcn-62/obj5 2024-02-01T10:49:55.083 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:LSaTw418kUMe0TeZ5c8D4SbcytA= 2024-02-01T10:49:55.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:LSaTw418kUMe0TeZ5c8D4SbcytA=', 'Content-Length': '0'} 2024-02-01T10:49:55.086 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009ca5078cba93c1fa-0065bb7753-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:55.128 DEBUG:boto:path=/vclfcn-62/obj5 2024-02-01T10:49:55.129 DEBUG:boto:auth_path=/vclfcn-62/obj5 2024-02-01T10:49:55.129 DEBUG:boto:Method: GET 2024-02-01T10:49:55.129 DEBUG:boto:Path: /vclfcn-62/obj5 2024-02-01T10:49:55.129 DEBUG:boto:Data: 2024-02-01T10:49:55.129 DEBUG:boto:Headers: {} 2024-02-01T10:49:55.129 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:55.129 DEBUG:boto:Port: 8000 2024-02-01T10:49:55.129 DEBUG:boto:Params: {} 2024-02-01T10:49:55.130 DEBUG:boto:Token: None 2024-02-01T10:49:55.130 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:55 GMT /vclfcn-62/obj5 2024-02-01T10:49:55.130 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:LSaTw418kUMe0TeZ5c8D4SbcytA= 2024-02-01T10:49:55.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:LSaTw418kUMe0TeZ5c8D4SbcytA=', 'Content-Length': '0'} 2024-02-01T10:49:55.133 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dd17bad58d7af722-0065bb7753-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:55.134 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:55.134 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:55.135 DEBUG:boto:path=/vclfcn-62/?prefix=obj5 2024-02-01T10:49:55.135 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj5 2024-02-01T10:49:55.135 DEBUG:boto:Method: GET 2024-02-01T10:49:55.135 DEBUG:boto:Path: /vclfcn-62/?prefix=obj5 2024-02-01T10:49:55.135 DEBUG:boto:Data: 2024-02-01T10:49:55.135 DEBUG:boto:Headers: {} 2024-02-01T10:49:55.136 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:55.136 DEBUG:boto:Port: 8000 2024-02-01T10:49:55.136 DEBUG:boto:Params: {} 2024-02-01T10:49:55.136 DEBUG:boto:Token: None 2024-02-01T10:49:55.136 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:55 GMT /vclfcn-62/ 2024-02-01T10:49:55.137 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:u/xE24Ir1FtV6T/cC18snd7v6wM= 2024-02-01T10:49:55.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:u/xE24Ir1FtV6T/cC18snd7v6wM=', 'Content-Length': '0'} 2024-02-01T10:49:55.141 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087b2155bda6a55ad-0065bb7753-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:55.188 DEBUG:boto:b'vclfcn-62obj51000falseobj52024-02-01T10:49:22.123Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:55.189 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:55.189 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:55.189 DEBUG:boto:path=/vclfcn-62/?prefix=obj5 2024-02-01T10:49:55.189 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj5 2024-02-01T10:49:55.189 DEBUG:boto:Method: GET 2024-02-01T10:49:55.189 DEBUG:boto:Path: /vclfcn-62/?prefix=obj5 2024-02-01T10:49:55.189 DEBUG:boto:Data: 2024-02-01T10:49:55.189 DEBUG:boto:Headers: {} 2024-02-01T10:49:55.190 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:55.190 DEBUG:boto:Port: 8000 2024-02-01T10:49:55.190 DEBUG:boto:Params: {} 2024-02-01T10:49:55.190 DEBUG:boto:Token: None 2024-02-01T10:49:55.190 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:55 GMT /vclfcn-62/ 2024-02-01T10:49:55.190 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:u/xE24Ir1FtV6T/cC18snd7v6wM= 2024-02-01T10:49:55.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:u/xE24Ir1FtV6T/cC18snd7v6wM=', 'Content-Length': '0'} 2024-02-01T10:49:55.195 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065ad525e9541db5a-0065bb7753-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:55.195 DEBUG:boto:b'vclfcn-62obj51000falseobj52024-02-01T10:49:22.123Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:55.195 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-02-01T10:49:55.195 DEBUG:boto:path=/vclfcn-62/obj6 2024-02-01T10:49:55.196 DEBUG:boto:auth_path=/vclfcn-62/obj6 2024-02-01T10:49:55.196 DEBUG:boto:path=/vclfcn-62/obj6?versionId=null 2024-02-01T10:49:55.196 DEBUG:boto:auth_path=/vclfcn-62/obj6?versionId=null 2024-02-01T10:49:55.196 DEBUG:boto:Method: GET 2024-02-01T10:49:55.196 DEBUG:boto:Path: /vclfcn-62/obj6?versionId=null 2024-02-01T10:49:55.196 DEBUG:boto:Data: 2024-02-01T10:49:55.196 DEBUG:boto:Headers: {} 2024-02-01T10:49:55.196 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:55.196 DEBUG:boto:Port: 8000 2024-02-01T10:49:55.197 DEBUG:boto:Params: {} 2024-02-01T10:49:55.197 DEBUG:boto:Token: None 2024-02-01T10:49:55.197 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:55 GMT /vclfcn-62/obj6?versionId=null 2024-02-01T10:49:55.197 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:VBS/4GnZljB9KH+PJ5cOdKKoktM= 2024-02-01T10:49:55.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:VBS/4GnZljB9KH+PJ5cOdKKoktM=', 'Content-Length': '0'} 2024-02-01T10:49:55.200 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000746c9f233cd6ea6d-0065bb7753-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:55.240 DEBUG:boto:path=/vclfcn-62/obj6 2024-02-01T10:49:55.240 DEBUG:boto:auth_path=/vclfcn-62/obj6 2024-02-01T10:49:55.240 DEBUG:boto:path=/vclfcn-62/obj6?versionId=null 2024-02-01T10:49:55.240 DEBUG:boto:auth_path=/vclfcn-62/obj6?versionId=null 2024-02-01T10:49:55.240 DEBUG:boto:Method: GET 2024-02-01T10:49:55.241 DEBUG:boto:Path: /vclfcn-62/obj6?versionId=null 2024-02-01T10:49:55.241 DEBUG:boto:Data: 2024-02-01T10:49:55.241 DEBUG:boto:Headers: {} 2024-02-01T10:49:55.241 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:55.241 DEBUG:boto:Port: 8000 2024-02-01T10:49:55.241 DEBUG:boto:Params: {} 2024-02-01T10:49:55.241 DEBUG:boto:Token: None 2024-02-01T10:49:55.241 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:55 GMT /vclfcn-62/obj6?versionId=null 2024-02-01T10:49:55.241 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:VBS/4GnZljB9KH+PJ5cOdKKoktM= 2024-02-01T10:49:55.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:VBS/4GnZljB9KH+PJ5cOdKKoktM=', 'Content-Length': '0'} 2024-02-01T10:49:55.244 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fa7148ad5cf39dad-0065bb7753-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:55.245 DEBUG:boto:path=/vclfcn-62/obj6 2024-02-01T10:49:55.245 DEBUG:boto:auth_path=/vclfcn-62/obj6 2024-02-01T10:49:55.245 DEBUG:boto:path=/vclfcn-62/obj6?versionId=null 2024-02-01T10:49:55.245 DEBUG:boto:auth_path=/vclfcn-62/obj6?versionId=null 2024-02-01T10:49:55.245 DEBUG:boto:Method: HEAD 2024-02-01T10:49:55.245 DEBUG:boto:Path: /vclfcn-62/obj6?versionId=null 2024-02-01T10:49:55.245 DEBUG:boto:Data: 2024-02-01T10:49:55.245 DEBUG:boto:Headers: {} 2024-02-01T10:49:55.245 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:55.246 DEBUG:boto:Port: 8000 2024-02-01T10:49:55.246 DEBUG:boto:Params: {} 2024-02-01T10:49:55.246 DEBUG:boto:Token: None 2024-02-01T10:49:55.246 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:55 GMT /vclfcn-62/obj6?versionId=null 2024-02-01T10:49:55.246 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:o/oEhYfeXVBsiDAagQvxed10sPA= 2024-02-01T10:49:55.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:o/oEhYfeXVBsiDAagQvxed10sPA=', 'Content-Length': '0'} 2024-02-01T10:49:55.249 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007364b493e385ad63-0065bb7753-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:55.249 DEBUG:boto:path=/vclfcn-62/obj6 2024-02-01T10:49:55.249 DEBUG:boto:auth_path=/vclfcn-62/obj6 2024-02-01T10:49:55.249 DEBUG:boto:path=/vclfcn-62/obj6?versionId=null 2024-02-01T10:49:55.249 DEBUG:boto:auth_path=/vclfcn-62/obj6?versionId=null 2024-02-01T10:49:55.249 DEBUG:boto:Method: HEAD 2024-02-01T10:49:55.250 DEBUG:boto:Path: /vclfcn-62/obj6?versionId=null 2024-02-01T10:49:55.250 DEBUG:boto:Data: 2024-02-01T10:49:55.250 DEBUG:boto:Headers: {} 2024-02-01T10:49:55.250 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:55.250 DEBUG:boto:Port: 8000 2024-02-01T10:49:55.250 DEBUG:boto:Params: {} 2024-02-01T10:49:55.250 DEBUG:boto:Token: None 2024-02-01T10:49:55.250 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:55 GMT /vclfcn-62/obj6?versionId=null 2024-02-01T10:49:55.250 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:o/oEhYfeXVBsiDAagQvxed10sPA= 2024-02-01T10:49:55.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:o/oEhYfeXVBsiDAagQvxed10sPA=', 'Content-Length': '0'} 2024-02-01T10:49:55.253 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a471c0c43e88ba85-0065bb7753-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:55.254 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-02-01T10:49:55.254 DEBUG:boto:path=/vclfcn-62/obj6 2024-02-01T10:49:55.254 DEBUG:boto:auth_path=/vclfcn-62/obj6 2024-02-01T10:49:55.254 DEBUG:boto:path=/vclfcn-62/obj6?versionId=null 2024-02-01T10:49:55.254 DEBUG:boto:auth_path=/vclfcn-62/obj6?versionId=null 2024-02-01T10:49:55.254 DEBUG:boto:Method: GET 2024-02-01T10:49:55.254 DEBUG:boto:Path: /vclfcn-62/obj6?versionId=null 2024-02-01T10:49:55.254 DEBUG:boto:Data: 2024-02-01T10:49:55.254 DEBUG:boto:Headers: {} 2024-02-01T10:49:55.255 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:55.255 DEBUG:boto:Port: 8000 2024-02-01T10:49:55.255 DEBUG:boto:Params: {} 2024-02-01T10:49:55.255 DEBUG:boto:Token: None 2024-02-01T10:49:55.255 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:55 GMT /vclfcn-62/obj6?versionId=null 2024-02-01T10:49:55.255 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:VBS/4GnZljB9KH+PJ5cOdKKoktM= 2024-02-01T10:49:55.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:VBS/4GnZljB9KH+PJ5cOdKKoktM=', 'Content-Length': '0'} 2024-02-01T10:49:55.258 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c3f1ed66706d6c9b-0065bb7753-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:55.304 DEBUG:boto:path=/vclfcn-62/obj6 2024-02-01T10:49:55.304 DEBUG:boto:auth_path=/vclfcn-62/obj6 2024-02-01T10:49:55.304 DEBUG:boto:path=/vclfcn-62/obj6?versionId=null 2024-02-01T10:49:55.305 DEBUG:boto:auth_path=/vclfcn-62/obj6?versionId=null 2024-02-01T10:49:55.305 DEBUG:boto:Method: GET 2024-02-01T10:49:55.305 DEBUG:boto:Path: /vclfcn-62/obj6?versionId=null 2024-02-01T10:49:55.305 DEBUG:boto:Data: 2024-02-01T10:49:55.305 DEBUG:boto:Headers: {} 2024-02-01T10:49:55.305 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:55.305 DEBUG:boto:Port: 8000 2024-02-01T10:49:55.305 DEBUG:boto:Params: {} 2024-02-01T10:49:55.305 DEBUG:boto:Token: None 2024-02-01T10:49:55.306 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:55 GMT /vclfcn-62/obj6?versionId=null 2024-02-01T10:49:55.306 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:VBS/4GnZljB9KH+PJ5cOdKKoktM= 2024-02-01T10:49:55.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:VBS/4GnZljB9KH+PJ5cOdKKoktM=', 'Content-Length': '0'} 2024-02-01T10:49:55.309 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004e84a17f90fc447d-0065bb7753-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:55.309 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:55.309 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:55.310 DEBUG:boto:path=/vclfcn-62/?prefix=obj6 2024-02-01T10:49:55.310 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj6 2024-02-01T10:49:55.310 DEBUG:boto:Method: GET 2024-02-01T10:49:55.310 DEBUG:boto:Path: /vclfcn-62/?prefix=obj6 2024-02-01T10:49:55.310 DEBUG:boto:Data: 2024-02-01T10:49:55.310 DEBUG:boto:Headers: {} 2024-02-01T10:49:55.310 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:55.310 DEBUG:boto:Port: 8000 2024-02-01T10:49:55.310 DEBUG:boto:Params: {} 2024-02-01T10:49:55.310 DEBUG:boto:Token: None 2024-02-01T10:49:55.311 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:55 GMT /vclfcn-62/ 2024-02-01T10:49:55.311 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:u/xE24Ir1FtV6T/cC18snd7v6wM= 2024-02-01T10:49:55.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:u/xE24Ir1FtV6T/cC18snd7v6wM=', 'Content-Length': '0'} 2024-02-01T10:49:55.316 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3d4a53dac09f81b-0065bb7753-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:55.316 DEBUG:boto:b'vclfcn-62obj61000falseobj62024-02-01T10:49:22.135Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:55.317 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:55.317 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:55.317 DEBUG:boto:path=/vclfcn-62/?prefix=obj6 2024-02-01T10:49:55.317 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj6 2024-02-01T10:49:55.317 DEBUG:boto:Method: GET 2024-02-01T10:49:55.317 DEBUG:boto:Path: /vclfcn-62/?prefix=obj6 2024-02-01T10:49:55.317 DEBUG:boto:Data: 2024-02-01T10:49:55.317 DEBUG:boto:Headers: {} 2024-02-01T10:49:55.317 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:55.317 DEBUG:boto:Port: 8000 2024-02-01T10:49:55.318 DEBUG:boto:Params: {} 2024-02-01T10:49:55.318 DEBUG:boto:Token: None 2024-02-01T10:49:55.318 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:55 GMT /vclfcn-62/ 2024-02-01T10:49:55.318 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:u/xE24Ir1FtV6T/cC18snd7v6wM= 2024-02-01T10:49:55.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:u/xE24Ir1FtV6T/cC18snd7v6wM=', 'Content-Length': '0'} 2024-02-01T10:49:55.322 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b0c3eaa759db6f4-0065bb7753-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:55.364 DEBUG:boto:b'vclfcn-62obj61000falseobj62024-02-01T10:49:22.135Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:55.365 DEBUG:boto:path=/vclfcn-62/obj6 2024-02-01T10:49:55.365 DEBUG:boto:auth_path=/vclfcn-62/obj6 2024-02-01T10:49:55.365 DEBUG:boto:Method: HEAD 2024-02-01T10:49:55.365 DEBUG:boto:Path: /vclfcn-62/obj6 2024-02-01T10:49:55.365 DEBUG:boto:Data: 2024-02-01T10:49:55.365 DEBUG:boto:Headers: {} 2024-02-01T10:49:55.365 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:55.365 DEBUG:boto:Port: 8000 2024-02-01T10:49:55.365 DEBUG:boto:Params: {} 2024-02-01T10:49:55.366 DEBUG:boto:Token: None 2024-02-01T10:49:55.366 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:55 GMT /vclfcn-62/obj6 2024-02-01T10:49:55.366 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:TOZGX/1H0rBBLHKVqLYqy2kOtxU= 2024-02-01T10:49:55.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:TOZGX/1H0rBBLHKVqLYqy2kOtxU=', 'Content-Length': '0'} 2024-02-01T10:49:55.369 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e8f0101aada5f83a-0065bb7753-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:55.369 DEBUG:boto:path=/vclfcn-62/obj6 2024-02-01T10:49:55.369 DEBUG:boto:auth_path=/vclfcn-62/obj6 2024-02-01T10:49:55.369 DEBUG:boto:Method: HEAD 2024-02-01T10:49:55.370 DEBUG:boto:Path: /vclfcn-62/obj6 2024-02-01T10:49:55.370 DEBUG:boto:Data: 2024-02-01T10:49:55.370 DEBUG:boto:Headers: {} 2024-02-01T10:49:55.370 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:55.370 DEBUG:boto:Port: 8000 2024-02-01T10:49:55.370 DEBUG:boto:Params: {} 2024-02-01T10:49:55.370 DEBUG:boto:Token: None 2024-02-01T10:49:55.370 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:55 GMT /vclfcn-62/obj6 2024-02-01T10:49:55.370 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:TOZGX/1H0rBBLHKVqLYqy2kOtxU= 2024-02-01T10:49:55.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:TOZGX/1H0rBBLHKVqLYqy2kOtxU=', 'Content-Length': '0'} 2024-02-01T10:49:55.373 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000db946ba1ebfe748c-0065bb7753-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:55.373 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-02-01T10:49:55.373 DEBUG:boto:path=/vclfcn-62/obj6 2024-02-01T10:49:55.373 DEBUG:boto:auth_path=/vclfcn-62/obj6 2024-02-01T10:49:55.374 DEBUG:boto:Method: GET 2024-02-01T10:49:55.374 DEBUG:boto:Path: /vclfcn-62/obj6 2024-02-01T10:49:55.374 DEBUG:boto:Data: 2024-02-01T10:49:55.374 DEBUG:boto:Headers: {} 2024-02-01T10:49:55.374 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:55.374 DEBUG:boto:Port: 8000 2024-02-01T10:49:55.374 DEBUG:boto:Params: {} 2024-02-01T10:49:55.374 DEBUG:boto:Token: None 2024-02-01T10:49:55.374 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:55 GMT /vclfcn-62/obj6 2024-02-01T10:49:55.375 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vB8ynL8YKN1CObIKjM4Wd+rQZ0w= 2024-02-01T10:49:55.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vB8ynL8YKN1CObIKjM4Wd+rQZ0w=', 'Content-Length': '0'} 2024-02-01T10:49:55.378 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f7ff3fab4d60fa97-0065bb7753-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:55.420 DEBUG:boto:path=/vclfcn-62/obj6 2024-02-01T10:49:55.420 DEBUG:boto:auth_path=/vclfcn-62/obj6 2024-02-01T10:49:55.420 DEBUG:boto:Method: GET 2024-02-01T10:49:55.420 DEBUG:boto:Path: /vclfcn-62/obj6 2024-02-01T10:49:55.421 DEBUG:boto:Data: 2024-02-01T10:49:55.421 DEBUG:boto:Headers: {} 2024-02-01T10:49:55.421 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:55.421 DEBUG:boto:Port: 8000 2024-02-01T10:49:55.421 DEBUG:boto:Params: {} 2024-02-01T10:49:55.421 DEBUG:boto:Token: None 2024-02-01T10:49:55.422 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:55 GMT /vclfcn-62/obj6 2024-02-01T10:49:55.422 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vB8ynL8YKN1CObIKjM4Wd+rQZ0w= 2024-02-01T10:49:55.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vB8ynL8YKN1CObIKjM4Wd+rQZ0w=', 'Content-Length': '0'} 2024-02-01T10:49:55.425 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000028b779c2a85866c8-0065bb7753-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:55.425 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:55.425 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:55.426 DEBUG:boto:path=/vclfcn-62/?prefix=obj6 2024-02-01T10:49:55.426 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj6 2024-02-01T10:49:55.426 DEBUG:boto:Method: GET 2024-02-01T10:49:55.426 DEBUG:boto:Path: /vclfcn-62/?prefix=obj6 2024-02-01T10:49:55.426 DEBUG:boto:Data: 2024-02-01T10:49:55.426 DEBUG:boto:Headers: {} 2024-02-01T10:49:55.426 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:55.426 DEBUG:boto:Port: 8000 2024-02-01T10:49:55.426 DEBUG:boto:Params: {} 2024-02-01T10:49:55.426 DEBUG:boto:Token: None 2024-02-01T10:49:55.427 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:55 GMT /vclfcn-62/ 2024-02-01T10:49:55.427 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:u/xE24Ir1FtV6T/cC18snd7v6wM= 2024-02-01T10:49:55.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:u/xE24Ir1FtV6T/cC18snd7v6wM=', 'Content-Length': '0'} 2024-02-01T10:49:55.431 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a640bb99aeb87e97-0065bb7753-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:55.476 DEBUG:boto:b'vclfcn-62obj61000falseobj62024-02-01T10:49:22.135Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:55.477 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:55.477 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:55.477 DEBUG:boto:path=/vclfcn-62/?prefix=obj6 2024-02-01T10:49:55.477 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj6 2024-02-01T10:49:55.477 DEBUG:boto:Method: GET 2024-02-01T10:49:55.477 DEBUG:boto:Path: /vclfcn-62/?prefix=obj6 2024-02-01T10:49:55.477 DEBUG:boto:Data: 2024-02-01T10:49:55.477 DEBUG:boto:Headers: {} 2024-02-01T10:49:55.477 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:55.477 DEBUG:boto:Port: 8000 2024-02-01T10:49:55.478 DEBUG:boto:Params: {} 2024-02-01T10:49:55.478 DEBUG:boto:Token: None 2024-02-01T10:49:55.478 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:55 GMT /vclfcn-62/ 2024-02-01T10:49:55.478 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:u/xE24Ir1FtV6T/cC18snd7v6wM= 2024-02-01T10:49:55.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:u/xE24Ir1FtV6T/cC18snd7v6wM=', 'Content-Length': '0'} 2024-02-01T10:49:55.483 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b7cad759ecf9f15-0065bb7753-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:55.484 DEBUG:boto:b'vclfcn-62obj61000falseobj62024-02-01T10:49:22.135Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:55.484 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-02-01T10:49:55.484 DEBUG:boto:path=/vclfcn-62/obj7 2024-02-01T10:49:55.485 DEBUG:boto:auth_path=/vclfcn-62/obj7 2024-02-01T10:49:55.485 DEBUG:boto:path=/vclfcn-62/obj7?versionId=null 2024-02-01T10:49:55.485 DEBUG:boto:auth_path=/vclfcn-62/obj7?versionId=null 2024-02-01T10:49:55.485 DEBUG:boto:Method: GET 2024-02-01T10:49:55.485 DEBUG:boto:Path: /vclfcn-62/obj7?versionId=null 2024-02-01T10:49:55.485 DEBUG:boto:Data: 2024-02-01T10:49:55.485 DEBUG:boto:Headers: {} 2024-02-01T10:49:55.485 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:55.485 DEBUG:boto:Port: 8000 2024-02-01T10:49:55.485 DEBUG:boto:Params: {} 2024-02-01T10:49:55.486 DEBUG:boto:Token: None 2024-02-01T10:49:55.486 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:55 GMT /vclfcn-62/obj7?versionId=null 2024-02-01T10:49:55.486 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:S+osVZsu7uwKy/uWiAwspEFlQ58= 2024-02-01T10:49:55.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:S+osVZsu7uwKy/uWiAwspEFlQ58=', 'Content-Length': '0'} 2024-02-01T10:49:55.489 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000040e098a3af0912e7-0065bb7753-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:55.532 DEBUG:boto:path=/vclfcn-62/obj7 2024-02-01T10:49:55.532 DEBUG:boto:auth_path=/vclfcn-62/obj7 2024-02-01T10:49:55.532 DEBUG:boto:path=/vclfcn-62/obj7?versionId=null 2024-02-01T10:49:55.533 DEBUG:boto:auth_path=/vclfcn-62/obj7?versionId=null 2024-02-01T10:49:55.533 DEBUG:boto:Method: GET 2024-02-01T10:49:55.533 DEBUG:boto:Path: /vclfcn-62/obj7?versionId=null 2024-02-01T10:49:55.533 DEBUG:boto:Data: 2024-02-01T10:49:55.533 DEBUG:boto:Headers: {} 2024-02-01T10:49:55.533 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:55.533 DEBUG:boto:Port: 8000 2024-02-01T10:49:55.533 DEBUG:boto:Params: {} 2024-02-01T10:49:55.533 DEBUG:boto:Token: None 2024-02-01T10:49:55.534 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:55 GMT /vclfcn-62/obj7?versionId=null 2024-02-01T10:49:55.535 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:S+osVZsu7uwKy/uWiAwspEFlQ58= 2024-02-01T10:49:55.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:S+osVZsu7uwKy/uWiAwspEFlQ58=', 'Content-Length': '0'} 2024-02-01T10:49:55.538 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000041d6f9d389fcbbc4-0065bb7753-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:55.539 DEBUG:boto:path=/vclfcn-62/obj7 2024-02-01T10:49:55.539 DEBUG:boto:auth_path=/vclfcn-62/obj7 2024-02-01T10:49:55.539 DEBUG:boto:path=/vclfcn-62/obj7?versionId=null 2024-02-01T10:49:55.539 DEBUG:boto:auth_path=/vclfcn-62/obj7?versionId=null 2024-02-01T10:49:55.539 DEBUG:boto:Method: HEAD 2024-02-01T10:49:55.539 DEBUG:boto:Path: /vclfcn-62/obj7?versionId=null 2024-02-01T10:49:55.539 DEBUG:boto:Data: 2024-02-01T10:49:55.539 DEBUG:boto:Headers: {} 2024-02-01T10:49:55.539 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:55.539 DEBUG:boto:Port: 8000 2024-02-01T10:49:55.539 DEBUG:boto:Params: {} 2024-02-01T10:49:55.540 DEBUG:boto:Token: None 2024-02-01T10:49:55.540 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:55 GMT /vclfcn-62/obj7?versionId=null 2024-02-01T10:49:55.540 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fV6X0l4sAgw7iX9Om/Rz7ZGgvYA= 2024-02-01T10:49:55.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fV6X0l4sAgw7iX9Om/Rz7ZGgvYA=', 'Content-Length': '0'} 2024-02-01T10:49:55.543 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cd2541eedd85bf00-0065bb7753-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:55.544 DEBUG:boto:path=/vclfcn-62/obj7 2024-02-01T10:49:55.544 DEBUG:boto:auth_path=/vclfcn-62/obj7 2024-02-01T10:49:55.544 DEBUG:boto:path=/vclfcn-62/obj7?versionId=null 2024-02-01T10:49:55.544 DEBUG:boto:auth_path=/vclfcn-62/obj7?versionId=null 2024-02-01T10:49:55.544 DEBUG:boto:Method: HEAD 2024-02-01T10:49:55.545 DEBUG:boto:Path: /vclfcn-62/obj7?versionId=null 2024-02-01T10:49:55.545 DEBUG:boto:Data: 2024-02-01T10:49:55.545 DEBUG:boto:Headers: {} 2024-02-01T10:49:55.545 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:55.545 DEBUG:boto:Port: 8000 2024-02-01T10:49:55.546 DEBUG:boto:Params: {} 2024-02-01T10:49:55.546 DEBUG:boto:Token: None 2024-02-01T10:49:55.546 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:55 GMT /vclfcn-62/obj7?versionId=null 2024-02-01T10:49:55.546 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fV6X0l4sAgw7iX9Om/Rz7ZGgvYA= 2024-02-01T10:49:55.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fV6X0l4sAgw7iX9Om/Rz7ZGgvYA=', 'Content-Length': '0'} 2024-02-01T10:49:55.549 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007bfbc3b6abeeff51-0065bb7753-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:55.550 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-02-01T10:49:55.550 DEBUG:boto:path=/vclfcn-62/obj7 2024-02-01T10:49:55.551 DEBUG:boto:auth_path=/vclfcn-62/obj7 2024-02-01T10:49:55.551 DEBUG:boto:path=/vclfcn-62/obj7?versionId=null 2024-02-01T10:49:55.551 DEBUG:boto:auth_path=/vclfcn-62/obj7?versionId=null 2024-02-01T10:49:55.551 DEBUG:boto:Method: GET 2024-02-01T10:49:55.551 DEBUG:boto:Path: /vclfcn-62/obj7?versionId=null 2024-02-01T10:49:55.552 DEBUG:boto:Data: 2024-02-01T10:49:55.552 DEBUG:boto:Headers: {} 2024-02-01T10:49:55.552 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:55.552 DEBUG:boto:Port: 8000 2024-02-01T10:49:55.552 DEBUG:boto:Params: {} 2024-02-01T10:49:55.553 DEBUG:boto:Token: None 2024-02-01T10:49:55.553 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:55 GMT /vclfcn-62/obj7?versionId=null 2024-02-01T10:49:55.553 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:S+osVZsu7uwKy/uWiAwspEFlQ58= 2024-02-01T10:49:55.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:S+osVZsu7uwKy/uWiAwspEFlQ58=', 'Content-Length': '0'} 2024-02-01T10:49:55.557 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000876cb3dbc57829b-0065bb7753-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:55.604 DEBUG:boto:path=/vclfcn-62/obj7 2024-02-01T10:49:55.604 DEBUG:boto:auth_path=/vclfcn-62/obj7 2024-02-01T10:49:55.604 DEBUG:boto:path=/vclfcn-62/obj7?versionId=null 2024-02-01T10:49:55.604 DEBUG:boto:auth_path=/vclfcn-62/obj7?versionId=null 2024-02-01T10:49:55.605 DEBUG:boto:Method: GET 2024-02-01T10:49:55.605 DEBUG:boto:Path: /vclfcn-62/obj7?versionId=null 2024-02-01T10:49:55.605 DEBUG:boto:Data: 2024-02-01T10:49:55.605 DEBUG:boto:Headers: {} 2024-02-01T10:49:55.605 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:55.605 DEBUG:boto:Port: 8000 2024-02-01T10:49:55.605 DEBUG:boto:Params: {} 2024-02-01T10:49:55.605 DEBUG:boto:Token: None 2024-02-01T10:49:55.605 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:55 GMT /vclfcn-62/obj7?versionId=null 2024-02-01T10:49:55.606 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:S+osVZsu7uwKy/uWiAwspEFlQ58= 2024-02-01T10:49:55.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:S+osVZsu7uwKy/uWiAwspEFlQ58=', 'Content-Length': '0'} 2024-02-01T10:49:55.608 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000099279b788a0402f8-0065bb7753-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:55.609 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:55.609 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:55.609 DEBUG:boto:path=/vclfcn-62/?prefix=obj7 2024-02-01T10:49:55.609 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj7 2024-02-01T10:49:55.609 DEBUG:boto:Method: GET 2024-02-01T10:49:55.609 DEBUG:boto:Path: /vclfcn-62/?prefix=obj7 2024-02-01T10:49:55.609 DEBUG:boto:Data: 2024-02-01T10:49:55.610 DEBUG:boto:Headers: {} 2024-02-01T10:49:55.610 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:55.610 DEBUG:boto:Port: 8000 2024-02-01T10:49:55.610 DEBUG:boto:Params: {} 2024-02-01T10:49:55.610 DEBUG:boto:Token: None 2024-02-01T10:49:55.610 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:55 GMT /vclfcn-62/ 2024-02-01T10:49:55.610 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:u/xE24Ir1FtV6T/cC18snd7v6wM= 2024-02-01T10:49:55.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:u/xE24Ir1FtV6T/cC18snd7v6wM=', 'Content-Length': '0'} 2024-02-01T10:49:55.614 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b15adcd4c347146-0065bb7753-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:55.660 DEBUG:boto:b'vclfcn-62obj71000falseobj72024-02-01T10:49:22.186Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:55.661 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:55.661 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:55.661 DEBUG:boto:path=/vclfcn-62/?prefix=obj7 2024-02-01T10:49:55.661 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj7 2024-02-01T10:49:55.661 DEBUG:boto:Method: GET 2024-02-01T10:49:55.661 DEBUG:boto:Path: /vclfcn-62/?prefix=obj7 2024-02-01T10:49:55.662 DEBUG:boto:Data: 2024-02-01T10:49:55.662 DEBUG:boto:Headers: {} 2024-02-01T10:49:55.662 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:55.662 DEBUG:boto:Port: 8000 2024-02-01T10:49:55.662 DEBUG:boto:Params: {} 2024-02-01T10:49:55.663 DEBUG:boto:Token: None 2024-02-01T10:49:55.663 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:55 GMT /vclfcn-62/ 2024-02-01T10:49:55.663 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:u/xE24Ir1FtV6T/cC18snd7v6wM= 2024-02-01T10:49:55.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:u/xE24Ir1FtV6T/cC18snd7v6wM=', 'Content-Length': '0'} 2024-02-01T10:49:55.668 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000858d14664c3a6c03-0065bb7753-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:55.668 DEBUG:boto:b'vclfcn-62obj71000falseobj72024-02-01T10:49:22.186Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:55.669 DEBUG:boto:path=/vclfcn-62/obj7 2024-02-01T10:49:55.669 DEBUG:boto:auth_path=/vclfcn-62/obj7 2024-02-01T10:49:55.669 DEBUG:boto:Method: HEAD 2024-02-01T10:49:55.669 DEBUG:boto:Path: /vclfcn-62/obj7 2024-02-01T10:49:55.669 DEBUG:boto:Data: 2024-02-01T10:49:55.669 DEBUG:boto:Headers: {} 2024-02-01T10:49:55.669 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:55.670 DEBUG:boto:Port: 8000 2024-02-01T10:49:55.670 DEBUG:boto:Params: {} 2024-02-01T10:49:55.670 DEBUG:boto:Token: None 2024-02-01T10:49:55.670 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:55 GMT /vclfcn-62/obj7 2024-02-01T10:49:55.670 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:m7Sw5V+ozTq25+bgFEbTUBZIkgc= 2024-02-01T10:49:55.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:m7Sw5V+ozTq25+bgFEbTUBZIkgc=', 'Content-Length': '0'} 2024-02-01T10:49:55.673 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e6c18ea9bff7b853-0065bb7753-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:55.673 DEBUG:boto:path=/vclfcn-62/obj7 2024-02-01T10:49:55.673 DEBUG:boto:auth_path=/vclfcn-62/obj7 2024-02-01T10:49:55.673 DEBUG:boto:Method: HEAD 2024-02-01T10:49:55.673 DEBUG:boto:Path: /vclfcn-62/obj7 2024-02-01T10:49:55.673 DEBUG:boto:Data: 2024-02-01T10:49:55.673 DEBUG:boto:Headers: {} 2024-02-01T10:49:55.674 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:55.674 DEBUG:boto:Port: 8000 2024-02-01T10:49:55.674 DEBUG:boto:Params: {} 2024-02-01T10:49:55.674 DEBUG:boto:Token: None 2024-02-01T10:49:55.674 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:55 GMT /vclfcn-62/obj7 2024-02-01T10:49:55.674 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:m7Sw5V+ozTq25+bgFEbTUBZIkgc= 2024-02-01T10:49:55.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:m7Sw5V+ozTq25+bgFEbTUBZIkgc=', 'Content-Length': '0'} 2024-02-01T10:49:55.677 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d447953ffe144ebb-0065bb7753-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:55.677 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-02-01T10:49:55.678 DEBUG:boto:path=/vclfcn-62/obj7 2024-02-01T10:49:55.678 DEBUG:boto:auth_path=/vclfcn-62/obj7 2024-02-01T10:49:55.678 DEBUG:boto:Method: GET 2024-02-01T10:49:55.678 DEBUG:boto:Path: /vclfcn-62/obj7 2024-02-01T10:49:55.678 DEBUG:boto:Data: 2024-02-01T10:49:55.678 DEBUG:boto:Headers: {} 2024-02-01T10:49:55.678 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:55.678 DEBUG:boto:Port: 8000 2024-02-01T10:49:55.678 DEBUG:boto:Params: {} 2024-02-01T10:49:55.678 DEBUG:boto:Token: None 2024-02-01T10:49:55.679 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:55 GMT /vclfcn-62/obj7 2024-02-01T10:49:55.679 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ER7hQ+dOAK3TnnMvVjTqgyLJyzo= 2024-02-01T10:49:55.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ER7hQ+dOAK3TnnMvVjTqgyLJyzo=', 'Content-Length': '0'} 2024-02-01T10:49:55.681 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000585956f241529ad1-0065bb7753-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:55.724 DEBUG:boto:path=/vclfcn-62/obj7 2024-02-01T10:49:55.724 DEBUG:boto:auth_path=/vclfcn-62/obj7 2024-02-01T10:49:55.724 DEBUG:boto:Method: GET 2024-02-01T10:49:55.725 DEBUG:boto:Path: /vclfcn-62/obj7 2024-02-01T10:49:55.725 DEBUG:boto:Data: 2024-02-01T10:49:55.725 DEBUG:boto:Headers: {} 2024-02-01T10:49:55.725 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:55.725 DEBUG:boto:Port: 8000 2024-02-01T10:49:55.726 DEBUG:boto:Params: {} 2024-02-01T10:49:55.726 DEBUG:boto:Token: None 2024-02-01T10:49:55.726 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:55 GMT /vclfcn-62/obj7 2024-02-01T10:49:55.726 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ER7hQ+dOAK3TnnMvVjTqgyLJyzo= 2024-02-01T10:49:55.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ER7hQ+dOAK3TnnMvVjTqgyLJyzo=', 'Content-Length': '0'} 2024-02-01T10:49:55.730 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bfabd3a254fa2644-0065bb7753-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:55.730 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:55.731 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:55.731 DEBUG:boto:path=/vclfcn-62/?prefix=obj7 2024-02-01T10:49:55.731 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj7 2024-02-01T10:49:55.731 DEBUG:boto:Method: GET 2024-02-01T10:49:55.731 DEBUG:boto:Path: /vclfcn-62/?prefix=obj7 2024-02-01T10:49:55.732 DEBUG:boto:Data: 2024-02-01T10:49:55.732 DEBUG:boto:Headers: {} 2024-02-01T10:49:55.732 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:55.732 DEBUG:boto:Port: 8000 2024-02-01T10:49:55.732 DEBUG:boto:Params: {} 2024-02-01T10:49:55.733 DEBUG:boto:Token: None 2024-02-01T10:49:55.733 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:55 GMT /vclfcn-62/ 2024-02-01T10:49:55.733 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:u/xE24Ir1FtV6T/cC18snd7v6wM= 2024-02-01T10:49:55.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:u/xE24Ir1FtV6T/cC18snd7v6wM=', 'Content-Length': '0'} 2024-02-01T10:49:55.739 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000861d749c5731de33-0065bb7753-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:55.780 DEBUG:boto:b'vclfcn-62obj71000falseobj72024-02-01T10:49:22.186Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:55.781 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:55.781 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:55.781 DEBUG:boto:path=/vclfcn-62/?prefix=obj7 2024-02-01T10:49:55.781 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj7 2024-02-01T10:49:55.781 DEBUG:boto:Method: GET 2024-02-01T10:49:55.782 DEBUG:boto:Path: /vclfcn-62/?prefix=obj7 2024-02-01T10:49:55.782 DEBUG:boto:Data: 2024-02-01T10:49:55.782 DEBUG:boto:Headers: {} 2024-02-01T10:49:55.782 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:55.782 DEBUG:boto:Port: 8000 2024-02-01T10:49:55.783 DEBUG:boto:Params: {} 2024-02-01T10:49:55.783 DEBUG:boto:Token: None 2024-02-01T10:49:55.783 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:55 GMT /vclfcn-62/ 2024-02-01T10:49:55.783 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:u/xE24Ir1FtV6T/cC18snd7v6wM= 2024-02-01T10:49:55.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:u/xE24Ir1FtV6T/cC18snd7v6wM=', 'Content-Length': '0'} 2024-02-01T10:49:55.788 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a65a1ef4b3e6c7ff-0065bb7753-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:55.788 DEBUG:boto:b'vclfcn-62obj71000falseobj72024-02-01T10:49:22.186Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:55.788 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-02-01T10:49:55.789 DEBUG:boto:path=/vclfcn-62/obj8 2024-02-01T10:49:55.789 DEBUG:boto:auth_path=/vclfcn-62/obj8 2024-02-01T10:49:55.789 DEBUG:boto:path=/vclfcn-62/obj8?versionId=null 2024-02-01T10:49:55.789 DEBUG:boto:auth_path=/vclfcn-62/obj8?versionId=null 2024-02-01T10:49:55.789 DEBUG:boto:Method: GET 2024-02-01T10:49:55.789 DEBUG:boto:Path: /vclfcn-62/obj8?versionId=null 2024-02-01T10:49:55.789 DEBUG:boto:Data: 2024-02-01T10:49:55.789 DEBUG:boto:Headers: {} 2024-02-01T10:49:55.789 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:55.789 DEBUG:boto:Port: 8000 2024-02-01T10:49:55.789 DEBUG:boto:Params: {} 2024-02-01T10:49:55.790 DEBUG:boto:Token: None 2024-02-01T10:49:55.790 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:55 GMT /vclfcn-62/obj8?versionId=null 2024-02-01T10:49:55.790 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zvcgwPOGPtKpTnvmEb/0X8LZDVY= 2024-02-01T10:49:55.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zvcgwPOGPtKpTnvmEb/0X8LZDVY=', 'Content-Length': '0'} 2024-02-01T10:49:55.793 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009e225458471818d4-0065bb7753-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:55.840 DEBUG:boto:path=/vclfcn-62/obj8 2024-02-01T10:49:55.840 DEBUG:boto:auth_path=/vclfcn-62/obj8 2024-02-01T10:49:55.840 DEBUG:boto:path=/vclfcn-62/obj8?versionId=null 2024-02-01T10:49:55.840 DEBUG:boto:auth_path=/vclfcn-62/obj8?versionId=null 2024-02-01T10:49:55.841 DEBUG:boto:Method: GET 2024-02-01T10:49:55.841 DEBUG:boto:Path: /vclfcn-62/obj8?versionId=null 2024-02-01T10:49:55.841 DEBUG:boto:Data: 2024-02-01T10:49:55.841 DEBUG:boto:Headers: {} 2024-02-01T10:49:55.841 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:55.841 DEBUG:boto:Port: 8000 2024-02-01T10:49:55.841 DEBUG:boto:Params: {} 2024-02-01T10:49:55.841 DEBUG:boto:Token: None 2024-02-01T10:49:55.841 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:55 GMT /vclfcn-62/obj8?versionId=null 2024-02-01T10:49:55.842 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zvcgwPOGPtKpTnvmEb/0X8LZDVY= 2024-02-01T10:49:55.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zvcgwPOGPtKpTnvmEb/0X8LZDVY=', 'Content-Length': '0'} 2024-02-01T10:49:55.845 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000066aba81284a1be67-0065bb7753-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:55.845 DEBUG:boto:path=/vclfcn-62/obj8 2024-02-01T10:49:55.845 DEBUG:boto:auth_path=/vclfcn-62/obj8 2024-02-01T10:49:55.845 DEBUG:boto:path=/vclfcn-62/obj8?versionId=null 2024-02-01T10:49:55.845 DEBUG:boto:auth_path=/vclfcn-62/obj8?versionId=null 2024-02-01T10:49:55.845 DEBUG:boto:Method: HEAD 2024-02-01T10:49:55.846 DEBUG:boto:Path: /vclfcn-62/obj8?versionId=null 2024-02-01T10:49:55.846 DEBUG:boto:Data: 2024-02-01T10:49:55.846 DEBUG:boto:Headers: {} 2024-02-01T10:49:55.846 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:55.846 DEBUG:boto:Port: 8000 2024-02-01T10:49:55.846 DEBUG:boto:Params: {} 2024-02-01T10:49:55.846 DEBUG:boto:Token: None 2024-02-01T10:49:55.846 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:55 GMT /vclfcn-62/obj8?versionId=null 2024-02-01T10:49:55.846 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Kli+kDXCV9AlLws/vEhLC2BBVFg= 2024-02-01T10:49:55.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Kli+kDXCV9AlLws/vEhLC2BBVFg=', 'Content-Length': '0'} 2024-02-01T10:49:55.849 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006d98666baa78563b-0065bb7753-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:55.849 DEBUG:boto:path=/vclfcn-62/obj8 2024-02-01T10:49:55.850 DEBUG:boto:auth_path=/vclfcn-62/obj8 2024-02-01T10:49:55.850 DEBUG:boto:path=/vclfcn-62/obj8?versionId=null 2024-02-01T10:49:55.850 DEBUG:boto:auth_path=/vclfcn-62/obj8?versionId=null 2024-02-01T10:49:55.850 DEBUG:boto:Method: HEAD 2024-02-01T10:49:55.850 DEBUG:boto:Path: /vclfcn-62/obj8?versionId=null 2024-02-01T10:49:55.850 DEBUG:boto:Data: 2024-02-01T10:49:55.850 DEBUG:boto:Headers: {} 2024-02-01T10:49:55.850 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:55.851 DEBUG:boto:Port: 8000 2024-02-01T10:49:55.851 DEBUG:boto:Params: {} 2024-02-01T10:49:55.851 DEBUG:boto:Token: None 2024-02-01T10:49:55.851 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:55 GMT /vclfcn-62/obj8?versionId=null 2024-02-01T10:49:55.851 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Kli+kDXCV9AlLws/vEhLC2BBVFg= 2024-02-01T10:49:55.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Kli+kDXCV9AlLws/vEhLC2BBVFg=', 'Content-Length': '0'} 2024-02-01T10:49:55.854 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000095758167d53204e9-0065bb7753-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:55.854 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-02-01T10:49:55.854 DEBUG:boto:path=/vclfcn-62/obj8 2024-02-01T10:49:55.854 DEBUG:boto:auth_path=/vclfcn-62/obj8 2024-02-01T10:49:55.855 DEBUG:boto:path=/vclfcn-62/obj8?versionId=null 2024-02-01T10:49:55.855 DEBUG:boto:auth_path=/vclfcn-62/obj8?versionId=null 2024-02-01T10:49:55.855 DEBUG:boto:Method: GET 2024-02-01T10:49:55.855 DEBUG:boto:Path: /vclfcn-62/obj8?versionId=null 2024-02-01T10:49:55.855 DEBUG:boto:Data: 2024-02-01T10:49:55.855 DEBUG:boto:Headers: {} 2024-02-01T10:49:55.855 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:55.856 DEBUG:boto:Port: 8000 2024-02-01T10:49:55.856 DEBUG:boto:Params: {} 2024-02-01T10:49:55.856 DEBUG:boto:Token: None 2024-02-01T10:49:55.856 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:55 GMT /vclfcn-62/obj8?versionId=null 2024-02-01T10:49:55.857 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zvcgwPOGPtKpTnvmEb/0X8LZDVY= 2024-02-01T10:49:55.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zvcgwPOGPtKpTnvmEb/0X8LZDVY=', 'Content-Length': '0'} 2024-02-01T10:49:55.860 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c2159a09a814bad0-0065bb7753-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:55.908 DEBUG:boto:path=/vclfcn-62/obj8 2024-02-01T10:49:55.908 DEBUG:boto:auth_path=/vclfcn-62/obj8 2024-02-01T10:49:55.908 DEBUG:boto:path=/vclfcn-62/obj8?versionId=null 2024-02-01T10:49:55.909 DEBUG:boto:auth_path=/vclfcn-62/obj8?versionId=null 2024-02-01T10:49:55.909 DEBUG:boto:Method: GET 2024-02-01T10:49:55.909 DEBUG:boto:Path: /vclfcn-62/obj8?versionId=null 2024-02-01T10:49:55.909 DEBUG:boto:Data: 2024-02-01T10:49:55.909 DEBUG:boto:Headers: {} 2024-02-01T10:49:55.909 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:55.909 DEBUG:boto:Port: 8000 2024-02-01T10:49:55.910 DEBUG:boto:Params: {} 2024-02-01T10:49:55.910 DEBUG:boto:Token: None 2024-02-01T10:49:55.910 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:55 GMT /vclfcn-62/obj8?versionId=null 2024-02-01T10:49:55.910 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zvcgwPOGPtKpTnvmEb/0X8LZDVY= 2024-02-01T10:49:55.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zvcgwPOGPtKpTnvmEb/0X8LZDVY=', 'Content-Length': '0'} 2024-02-01T10:49:55.913 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cafaa9e1f1672205-0065bb7753-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:55.914 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:55.914 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:55.914 DEBUG:boto:path=/vclfcn-62/?prefix=obj8 2024-02-01T10:49:55.914 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj8 2024-02-01T10:49:55.914 DEBUG:boto:Method: GET 2024-02-01T10:49:55.914 DEBUG:boto:Path: /vclfcn-62/?prefix=obj8 2024-02-01T10:49:55.914 DEBUG:boto:Data: 2024-02-01T10:49:55.914 DEBUG:boto:Headers: {} 2024-02-01T10:49:55.914 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:55.915 DEBUG:boto:Port: 8000 2024-02-01T10:49:55.915 DEBUG:boto:Params: {} 2024-02-01T10:49:55.915 DEBUG:boto:Token: None 2024-02-01T10:49:55.915 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:55 GMT /vclfcn-62/ 2024-02-01T10:49:55.915 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:u/xE24Ir1FtV6T/cC18snd7v6wM= 2024-02-01T10:49:55.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:u/xE24Ir1FtV6T/cC18snd7v6wM=', 'Content-Length': '0'} 2024-02-01T10:49:55.919 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000005a0f85aff9ea2ba-0065bb7753-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:55.920 DEBUG:boto:b'vclfcn-62obj81000falseobj82024-02-01T10:49:22.200Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:55.920 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:55.920 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:55.921 DEBUG:boto:path=/vclfcn-62/?prefix=obj8 2024-02-01T10:49:55.921 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj8 2024-02-01T10:49:55.921 DEBUG:boto:Method: GET 2024-02-01T10:49:55.921 DEBUG:boto:Path: /vclfcn-62/?prefix=obj8 2024-02-01T10:49:55.921 DEBUG:boto:Data: 2024-02-01T10:49:55.921 DEBUG:boto:Headers: {} 2024-02-01T10:49:55.921 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:55.921 DEBUG:boto:Port: 8000 2024-02-01T10:49:55.922 DEBUG:boto:Params: {} 2024-02-01T10:49:55.922 DEBUG:boto:Token: None 2024-02-01T10:49:55.922 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:55 GMT /vclfcn-62/ 2024-02-01T10:49:55.922 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:u/xE24Ir1FtV6T/cC18snd7v6wM= 2024-02-01T10:49:55.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:u/xE24Ir1FtV6T/cC18snd7v6wM=', 'Content-Length': '0'} 2024-02-01T10:49:55.927 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000006299fcd94cea1df-0065bb7753-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:55.927 DEBUG:boto:b'vclfcn-62obj81000falseobj82024-02-01T10:49:22.200Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:55.928 DEBUG:boto:path=/vclfcn-62/obj8 2024-02-01T10:49:55.928 DEBUG:boto:auth_path=/vclfcn-62/obj8 2024-02-01T10:49:55.928 DEBUG:boto:Method: HEAD 2024-02-01T10:49:55.928 DEBUG:boto:Path: /vclfcn-62/obj8 2024-02-01T10:49:55.928 DEBUG:boto:Data: 2024-02-01T10:49:55.929 DEBUG:boto:Headers: {} 2024-02-01T10:49:55.929 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:55.929 DEBUG:boto:Port: 8000 2024-02-01T10:49:55.929 DEBUG:boto:Params: {} 2024-02-01T10:49:55.929 DEBUG:boto:Token: None 2024-02-01T10:49:55.929 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:55 GMT /vclfcn-62/obj8 2024-02-01T10:49:55.930 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FuIq+PAh20nEeJlfPAuHI5iZmns= 2024-02-01T10:49:55.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FuIq+PAh20nEeJlfPAuHI5iZmns=', 'Content-Length': '0'} 2024-02-01T10:49:55.932 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000058bf509b8d8f895-0065bb7753-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:55.933 DEBUG:boto:path=/vclfcn-62/obj8 2024-02-01T10:49:55.933 DEBUG:boto:auth_path=/vclfcn-62/obj8 2024-02-01T10:49:55.934 DEBUG:boto:Method: HEAD 2024-02-01T10:49:55.934 DEBUG:boto:Path: /vclfcn-62/obj8 2024-02-01T10:49:55.934 DEBUG:boto:Data: 2024-02-01T10:49:55.934 DEBUG:boto:Headers: {} 2024-02-01T10:49:55.934 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:55.934 DEBUG:boto:Port: 8000 2024-02-01T10:49:55.934 DEBUG:boto:Params: {} 2024-02-01T10:49:55.934 DEBUG:boto:Token: None 2024-02-01T10:49:55.934 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:55 GMT /vclfcn-62/obj8 2024-02-01T10:49:55.934 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FuIq+PAh20nEeJlfPAuHI5iZmns= 2024-02-01T10:49:55.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FuIq+PAh20nEeJlfPAuHI5iZmns=', 'Content-Length': '0'} 2024-02-01T10:49:55.937 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ed5c9d5b014ed2e9-0065bb7753-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:55.937 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-02-01T10:49:55.938 DEBUG:boto:path=/vclfcn-62/obj8 2024-02-01T10:49:55.938 DEBUG:boto:auth_path=/vclfcn-62/obj8 2024-02-01T10:49:55.938 DEBUG:boto:Method: GET 2024-02-01T10:49:55.938 DEBUG:boto:Path: /vclfcn-62/obj8 2024-02-01T10:49:55.938 DEBUG:boto:Data: 2024-02-01T10:49:55.938 DEBUG:boto:Headers: {} 2024-02-01T10:49:55.938 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:55.938 DEBUG:boto:Port: 8000 2024-02-01T10:49:55.938 DEBUG:boto:Params: {} 2024-02-01T10:49:55.938 DEBUG:boto:Token: None 2024-02-01T10:49:55.939 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:55 GMT /vclfcn-62/obj8 2024-02-01T10:49:55.939 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vS3jrUYhYjks+td0KS0wBslNn08= 2024-02-01T10:49:55.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vS3jrUYhYjks+td0KS0wBslNn08=', 'Content-Length': '0'} 2024-02-01T10:49:55.943 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eed238f71beea380-0065bb7753-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:55.988 DEBUG:boto:path=/vclfcn-62/obj8 2024-02-01T10:49:55.988 DEBUG:boto:auth_path=/vclfcn-62/obj8 2024-02-01T10:49:55.988 DEBUG:boto:Method: GET 2024-02-01T10:49:55.989 DEBUG:boto:Path: /vclfcn-62/obj8 2024-02-01T10:49:55.989 DEBUG:boto:Data: 2024-02-01T10:49:55.989 DEBUG:boto:Headers: {} 2024-02-01T10:49:55.989 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:55.989 DEBUG:boto:Port: 8000 2024-02-01T10:49:55.989 DEBUG:boto:Params: {} 2024-02-01T10:49:55.989 DEBUG:boto:Token: None 2024-02-01T10:49:55.989 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:55 GMT /vclfcn-62/obj8 2024-02-01T10:49:55.989 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vS3jrUYhYjks+td0KS0wBslNn08= 2024-02-01T10:49:55.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vS3jrUYhYjks+td0KS0wBslNn08=', 'Content-Length': '0'} 2024-02-01T10:49:55.992 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cba776a310211580-0065bb7753-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:55 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:55.993 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:55.993 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:55.993 DEBUG:boto:path=/vclfcn-62/?prefix=obj8 2024-02-01T10:49:55.993 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj8 2024-02-01T10:49:55.993 DEBUG:boto:Method: GET 2024-02-01T10:49:55.993 DEBUG:boto:Path: /vclfcn-62/?prefix=obj8 2024-02-01T10:49:55.993 DEBUG:boto:Data: 2024-02-01T10:49:55.993 DEBUG:boto:Headers: {} 2024-02-01T10:49:55.994 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:55.994 DEBUG:boto:Port: 8000 2024-02-01T10:49:55.994 DEBUG:boto:Params: {} 2024-02-01T10:49:55.994 DEBUG:boto:Token: None 2024-02-01T10:49:55.994 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:55 GMT /vclfcn-62/ 2024-02-01T10:49:55.994 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:u/xE24Ir1FtV6T/cC18snd7v6wM= 2024-02-01T10:49:55.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:55 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:u/xE24Ir1FtV6T/cC18snd7v6wM=', 'Content-Length': '0'} 2024-02-01T10:49:55.998 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c37e2f07948dc7ae-0065bb7753-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:55.998 DEBUG:boto:b'vclfcn-62obj81000falseobj82024-02-01T10:49:22.200Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:55.999 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:55.999 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:55.999 DEBUG:boto:path=/vclfcn-62/?prefix=obj8 2024-02-01T10:49:55.999 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj8 2024-02-01T10:49:55.999 DEBUG:boto:Method: GET 2024-02-01T10:49:55.999 DEBUG:boto:Path: /vclfcn-62/?prefix=obj8 2024-02-01T10:49:55.999 DEBUG:boto:Data: 2024-02-01T10:49:56.000 DEBUG:boto:Headers: {} 2024-02-01T10:49:56.000 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:56.000 DEBUG:boto:Port: 8000 2024-02-01T10:49:56.000 DEBUG:boto:Params: {} 2024-02-01T10:49:56.000 DEBUG:boto:Token: None 2024-02-01T10:49:56.000 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:56 GMT /vclfcn-62/ 2024-02-01T10:49:56.000 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:rhqfdyVzV8k/sJU5MGBgCWI9J2U= 2024-02-01T10:49:56.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:rhqfdyVzV8k/sJU5MGBgCWI9J2U=', 'Content-Length': '0'} 2024-02-01T10:49:56.005 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b14a69b253a7b59-0065bb7754-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:56.005 DEBUG:boto:b'vclfcn-62obj81000falseobj82024-02-01T10:49:22.200Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:56.005 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-62 zones={test-zone2, test-zone1} 2024-02-01T10:49:56.006 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-62 zones={test-zone3, test-zone1} 2024-02-01T10:49:56.006 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:56.006 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:56.006 DEBUG:boto:Method: HEAD 2024-02-01T10:49:56.006 DEBUG:boto:Path: /vclfcn-62/ 2024-02-01T10:49:56.006 DEBUG:boto:Data: 2024-02-01T10:49:56.006 DEBUG:boto:Headers: {} 2024-02-01T10:49:56.007 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:56.007 DEBUG:boto:Port: 8001 2024-02-01T10:49:56.007 DEBUG:boto:Params: {} 2024-02-01T10:49:56.007 DEBUG:boto:Token: None 2024-02-01T10:49:56.007 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:56 GMT /vclfcn-62/ 2024-02-01T10:49:56.007 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:smncQBigfbuegWItcqn5gWQwLtA= 2024-02-01T10:49:56.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:smncQBigfbuegWItcqn5gWQwLtA=', 'Content-Length': '0'} 2024-02-01T10:49:56.012 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000077632155bcb86790-0065bb7754-5762-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:49:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:56.012 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:56.012 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:56.012 DEBUG:boto:Method: HEAD 2024-02-01T10:49:56.012 DEBUG:boto:Path: /vclfcn-62/ 2024-02-01T10:49:56.013 DEBUG:boto:Data: 2024-02-01T10:49:56.013 DEBUG:boto:Headers: {} 2024-02-01T10:49:56.013 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:56.013 DEBUG:boto:Port: 8000 2024-02-01T10:49:56.013 DEBUG:boto:Params: {} 2024-02-01T10:49:56.013 DEBUG:boto:Token: None 2024-02-01T10:49:56.014 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:56 GMT /vclfcn-62/ 2024-02-01T10:49:56.014 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:smncQBigfbuegWItcqn5gWQwLtA= 2024-02-01T10:49:56.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:smncQBigfbuegWItcqn5gWQwLtA=', 'Content-Length': '0'} 2024-02-01T10:49:56.018 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002d2f56bbc911b657-0065bb7754-5765-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:49:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:56.018 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:49:56.018 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:56.018 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:56.019 DEBUG:boto:path=/vclfcn-62/?versions&max-keys=999 2024-02-01T10:49:56.019 DEBUG:boto:auth_path=/vclfcn-62/?versions&max-keys=999 2024-02-01T10:49:56.019 DEBUG:boto:Method: GET 2024-02-01T10:49:56.019 DEBUG:boto:Path: /vclfcn-62/?versions&max-keys=999 2024-02-01T10:49:56.019 DEBUG:boto:Data: 2024-02-01T10:49:56.019 DEBUG:boto:Headers: {} 2024-02-01T10:49:56.019 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:56.019 DEBUG:boto:Port: 8001 2024-02-01T10:49:56.019 DEBUG:boto:Params: {} 2024-02-01T10:49:56.019 DEBUG:boto:Token: None 2024-02-01T10:49:56.020 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:56 GMT /vclfcn-62/?versions 2024-02-01T10:49:56.020 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Bum92xXdXJy+5pJz6hRv8lvGQW8= 2024-02-01T10:49:56.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Bum92xXdXJy+5pJz6hRv8lvGQW8=', 'Content-Length': '0'} 2024-02-01T10:49:56.025 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040ddf705d5928138-0065bb7754-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:56.025 DEBUG:boto:b'vclfcn-62999falseobj1nulltrue2024-02-01T10:48:26.315Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:48:26.333Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:48:26.384Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:48:26.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-02-01T10:49:22.123Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-02-01T10:49:22.135Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-02-01T10:49:22.186Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-02-01T10:49:22.200Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:56.026 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-02-01T10:49:56.026 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-02-01T10:49:56.026 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-02-01T10:49:56.026 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-02-01T10:49:56.026 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-02-01T10:49:56.026 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-02-01T10:49:56.026 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-02-01T10:49:56.026 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-02-01T10:49:56.026 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:49:56.026 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:56.027 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:56.027 DEBUG:boto:path=/vclfcn-62/?versions&max-keys=999 2024-02-01T10:49:56.027 DEBUG:boto:auth_path=/vclfcn-62/?versions&max-keys=999 2024-02-01T10:49:56.027 DEBUG:boto:Method: GET 2024-02-01T10:49:56.027 DEBUG:boto:Path: /vclfcn-62/?versions&max-keys=999 2024-02-01T10:49:56.027 DEBUG:boto:Data: 2024-02-01T10:49:56.027 DEBUG:boto:Headers: {} 2024-02-01T10:49:56.027 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:56.027 DEBUG:boto:Port: 8000 2024-02-01T10:49:56.027 DEBUG:boto:Params: {} 2024-02-01T10:49:56.027 DEBUG:boto:Token: None 2024-02-01T10:49:56.028 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:56 GMT /vclfcn-62/?versions 2024-02-01T10:49:56.028 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Bum92xXdXJy+5pJz6hRv8lvGQW8= 2024-02-01T10:49:56.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Bum92xXdXJy+5pJz6hRv8lvGQW8=', 'Content-Length': '0'} 2024-02-01T10:49:56.033 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc73f0ac4dafac73-0065bb7754-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:56.033 DEBUG:boto:b'vclfcn-62999falseobj1nulltrue2024-02-01T10:48:26.315Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:48:26.333Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:48:26.384Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:48:26.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-02-01T10:49:22.123Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-02-01T10:49:22.135Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-02-01T10:49:22.186Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-02-01T10:49:22.200Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:56.034 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-02-01T10:49:56.034 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-02-01T10:49:56.034 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-02-01T10:49:56.034 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-02-01T10:49:56.034 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-02-01T10:49:56.035 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-02-01T10:49:56.035 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-02-01T10:49:56.035 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-02-01T10:49:56.035 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:56.035 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:56.035 DEBUG:boto:path=/vclfcn-62/?versions&max-keys=999 2024-02-01T10:49:56.035 DEBUG:boto:auth_path=/vclfcn-62/?versions&max-keys=999 2024-02-01T10:49:56.035 DEBUG:boto:Method: GET 2024-02-01T10:49:56.035 DEBUG:boto:Path: /vclfcn-62/?versions&max-keys=999 2024-02-01T10:49:56.035 DEBUG:boto:Data: 2024-02-01T10:49:56.035 DEBUG:boto:Headers: {} 2024-02-01T10:49:56.036 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:56.036 DEBUG:boto:Port: 8001 2024-02-01T10:49:56.036 DEBUG:boto:Params: {} 2024-02-01T10:49:56.036 DEBUG:boto:Token: None 2024-02-01T10:49:56.036 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:56 GMT /vclfcn-62/?versions 2024-02-01T10:49:56.036 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Bum92xXdXJy+5pJz6hRv8lvGQW8= 2024-02-01T10:49:56.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Bum92xXdXJy+5pJz6hRv8lvGQW8=', 'Content-Length': '0'} 2024-02-01T10:49:56.041 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a38d7941abf44662-0065bb7754-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:56.042 DEBUG:boto:b'vclfcn-62999falseobj1nulltrue2024-02-01T10:48:26.315Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:48:26.333Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:48:26.384Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:48:26.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-02-01T10:49:22.123Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-02-01T10:49:22.135Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-02-01T10:49:22.186Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-02-01T10:49:22.200Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:56.042 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:56.043 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:56.043 DEBUG:boto:path=/vclfcn-62/?versions&max-keys=999 2024-02-01T10:49:56.043 DEBUG:boto:auth_path=/vclfcn-62/?versions&max-keys=999 2024-02-01T10:49:56.043 DEBUG:boto:Method: GET 2024-02-01T10:49:56.043 DEBUG:boto:Path: /vclfcn-62/?versions&max-keys=999 2024-02-01T10:49:56.043 DEBUG:boto:Data: 2024-02-01T10:49:56.043 DEBUG:boto:Headers: {} 2024-02-01T10:49:56.043 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:56.043 DEBUG:boto:Port: 8000 2024-02-01T10:49:56.043 DEBUG:boto:Params: {} 2024-02-01T10:49:56.043 DEBUG:boto:Token: None 2024-02-01T10:49:56.044 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:56 GMT /vclfcn-62/?versions 2024-02-01T10:49:56.044 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Bum92xXdXJy+5pJz6hRv8lvGQW8= 2024-02-01T10:49:56.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Bum92xXdXJy+5pJz6hRv8lvGQW8=', 'Content-Length': '0'} 2024-02-01T10:49:56.049 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d82316806eacf91-0065bb7754-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:56.049 DEBUG:boto:b'vclfcn-62999falseobj1nulltrue2024-02-01T10:48:26.315Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:48:26.333Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:48:26.384Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:48:26.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-02-01T10:49:22.123Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-02-01T10:49:22.135Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-02-01T10:49:22.186Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-02-01T10:49:22.200Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:56.050 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:49:56.050 DEBUG:boto:path=/vclfcn-62/obj1 2024-02-01T10:49:56.050 DEBUG:boto:auth_path=/vclfcn-62/obj1 2024-02-01T10:49:56.050 DEBUG:boto:path=/vclfcn-62/obj1?versionId=null 2024-02-01T10:49:56.050 DEBUG:boto:auth_path=/vclfcn-62/obj1?versionId=null 2024-02-01T10:49:56.050 DEBUG:boto:Method: GET 2024-02-01T10:49:56.050 DEBUG:boto:Path: /vclfcn-62/obj1?versionId=null 2024-02-01T10:49:56.051 DEBUG:boto:Data: 2024-02-01T10:49:56.051 DEBUG:boto:Headers: {} 2024-02-01T10:49:56.051 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:56.051 DEBUG:boto:Port: 8001 2024-02-01T10:49:56.051 DEBUG:boto:Params: {} 2024-02-01T10:49:56.051 DEBUG:boto:Token: None 2024-02-01T10:49:56.051 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:56 GMT /vclfcn-62/obj1?versionId=null 2024-02-01T10:49:56.051 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BFRuN9mMEIvTKhwkhaUIlWPj0mg= 2024-02-01T10:49:56.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BFRuN9mMEIvTKhwkhaUIlWPj0mg=', 'Content-Length': '0'} 2024-02-01T10:49:56.054 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bb56e4b69e2a8f40-0065bb7754-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:56.100 DEBUG:boto:path=/vclfcn-62/obj1 2024-02-01T10:49:56.100 DEBUG:boto:auth_path=/vclfcn-62/obj1 2024-02-01T10:49:56.100 DEBUG:boto:path=/vclfcn-62/obj1?versionId=null 2024-02-01T10:49:56.100 DEBUG:boto:auth_path=/vclfcn-62/obj1?versionId=null 2024-02-01T10:49:56.101 DEBUG:boto:Method: GET 2024-02-01T10:49:56.101 DEBUG:boto:Path: /vclfcn-62/obj1?versionId=null 2024-02-01T10:49:56.101 DEBUG:boto:Data: 2024-02-01T10:49:56.101 DEBUG:boto:Headers: {} 2024-02-01T10:49:56.101 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:56.101 DEBUG:boto:Port: 8000 2024-02-01T10:49:56.101 DEBUG:boto:Params: {} 2024-02-01T10:49:56.101 DEBUG:boto:Token: None 2024-02-01T10:49:56.101 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:56 GMT /vclfcn-62/obj1?versionId=null 2024-02-01T10:49:56.102 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BFRuN9mMEIvTKhwkhaUIlWPj0mg= 2024-02-01T10:49:56.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BFRuN9mMEIvTKhwkhaUIlWPj0mg=', 'Content-Length': '0'} 2024-02-01T10:49:56.106 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009b9c11d858415d3d-0065bb7754-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:56.106 DEBUG:boto:path=/vclfcn-62/obj1 2024-02-01T10:49:56.106 DEBUG:boto:auth_path=/vclfcn-62/obj1 2024-02-01T10:49:56.106 DEBUG:boto:path=/vclfcn-62/obj1?versionId=null 2024-02-01T10:49:56.106 DEBUG:boto:auth_path=/vclfcn-62/obj1?versionId=null 2024-02-01T10:49:56.107 DEBUG:boto:Method: HEAD 2024-02-01T10:49:56.107 DEBUG:boto:Path: /vclfcn-62/obj1?versionId=null 2024-02-01T10:49:56.107 DEBUG:boto:Data: 2024-02-01T10:49:56.107 DEBUG:boto:Headers: {} 2024-02-01T10:49:56.107 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:56.107 DEBUG:boto:Port: 8001 2024-02-01T10:49:56.107 DEBUG:boto:Params: {} 2024-02-01T10:49:56.107 DEBUG:boto:Token: None 2024-02-01T10:49:56.107 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:56 GMT /vclfcn-62/obj1?versionId=null 2024-02-01T10:49:56.108 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:50HPqNvT9oSxF83Ar8NpKMq6708= 2024-02-01T10:49:56.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:50HPqNvT9oSxF83Ar8NpKMq6708=', 'Content-Length': '0'} 2024-02-01T10:49:56.111 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000983e51a165cba4e3-0065bb7754-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:56.111 DEBUG:boto:path=/vclfcn-62/obj1 2024-02-01T10:49:56.111 DEBUG:boto:auth_path=/vclfcn-62/obj1 2024-02-01T10:49:56.111 DEBUG:boto:path=/vclfcn-62/obj1?versionId=null 2024-02-01T10:49:56.112 DEBUG:boto:auth_path=/vclfcn-62/obj1?versionId=null 2024-02-01T10:49:56.112 DEBUG:boto:Method: HEAD 2024-02-01T10:49:56.112 DEBUG:boto:Path: /vclfcn-62/obj1?versionId=null 2024-02-01T10:49:56.112 DEBUG:boto:Data: 2024-02-01T10:49:56.112 DEBUG:boto:Headers: {} 2024-02-01T10:49:56.112 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:56.112 DEBUG:boto:Port: 8000 2024-02-01T10:49:56.112 DEBUG:boto:Params: {} 2024-02-01T10:49:56.113 DEBUG:boto:Token: None 2024-02-01T10:49:56.113 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:56 GMT /vclfcn-62/obj1?versionId=null 2024-02-01T10:49:56.113 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:50HPqNvT9oSxF83Ar8NpKMq6708= 2024-02-01T10:49:56.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:50HPqNvT9oSxF83Ar8NpKMq6708=', 'Content-Length': '0'} 2024-02-01T10:49:56.116 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000496cc12fe04cb9c9-0065bb7754-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:56.116 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:49:56.116 DEBUG:boto:path=/vclfcn-62/obj1 2024-02-01T10:49:56.116 DEBUG:boto:auth_path=/vclfcn-62/obj1 2024-02-01T10:49:56.116 DEBUG:boto:path=/vclfcn-62/obj1?versionId=null 2024-02-01T10:49:56.117 DEBUG:boto:auth_path=/vclfcn-62/obj1?versionId=null 2024-02-01T10:49:56.117 DEBUG:boto:Method: GET 2024-02-01T10:49:56.117 DEBUG:boto:Path: /vclfcn-62/obj1?versionId=null 2024-02-01T10:49:56.117 DEBUG:boto:Data: 2024-02-01T10:49:56.117 DEBUG:boto:Headers: {} 2024-02-01T10:49:56.117 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:56.117 DEBUG:boto:Port: 8001 2024-02-01T10:49:56.117 DEBUG:boto:Params: {} 2024-02-01T10:49:56.117 DEBUG:boto:Token: None 2024-02-01T10:49:56.117 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:56 GMT /vclfcn-62/obj1?versionId=null 2024-02-01T10:49:56.117 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BFRuN9mMEIvTKhwkhaUIlWPj0mg= 2024-02-01T10:49:56.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BFRuN9mMEIvTKhwkhaUIlWPj0mg=', 'Content-Length': '0'} 2024-02-01T10:49:56.120 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001c8603d452b337d5-0065bb7754-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:56.121 DEBUG:boto:path=/vclfcn-62/obj1 2024-02-01T10:49:56.121 DEBUG:boto:auth_path=/vclfcn-62/obj1 2024-02-01T10:49:56.121 DEBUG:boto:path=/vclfcn-62/obj1?versionId=null 2024-02-01T10:49:56.121 DEBUG:boto:auth_path=/vclfcn-62/obj1?versionId=null 2024-02-01T10:49:56.122 DEBUG:boto:Method: GET 2024-02-01T10:49:56.122 DEBUG:boto:Path: /vclfcn-62/obj1?versionId=null 2024-02-01T10:49:56.122 DEBUG:boto:Data: 2024-02-01T10:49:56.122 DEBUG:boto:Headers: {} 2024-02-01T10:49:56.122 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:56.122 DEBUG:boto:Port: 8000 2024-02-01T10:49:56.122 DEBUG:boto:Params: {} 2024-02-01T10:49:56.123 DEBUG:boto:Token: None 2024-02-01T10:49:56.123 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:56 GMT /vclfcn-62/obj1?versionId=null 2024-02-01T10:49:56.123 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BFRuN9mMEIvTKhwkhaUIlWPj0mg= 2024-02-01T10:49:56.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BFRuN9mMEIvTKhwkhaUIlWPj0mg=', 'Content-Length': '0'} 2024-02-01T10:49:56.126 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000e9e94b7dff175d5-0065bb7754-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:56.126 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:56.127 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:56.127 DEBUG:boto:path=/vclfcn-62/?prefix=obj1 2024-02-01T10:49:56.127 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj1 2024-02-01T10:49:56.127 DEBUG:boto:Method: GET 2024-02-01T10:49:56.127 DEBUG:boto:Path: /vclfcn-62/?prefix=obj1 2024-02-01T10:49:56.127 DEBUG:boto:Data: 2024-02-01T10:49:56.127 DEBUG:boto:Headers: {} 2024-02-01T10:49:56.127 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:56.127 DEBUG:boto:Port: 8001 2024-02-01T10:49:56.127 DEBUG:boto:Params: {} 2024-02-01T10:49:56.128 DEBUG:boto:Token: None 2024-02-01T10:49:56.128 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:56 GMT /vclfcn-62/ 2024-02-01T10:49:56.128 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:rhqfdyVzV8k/sJU5MGBgCWI9J2U= 2024-02-01T10:49:56.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:rhqfdyVzV8k/sJU5MGBgCWI9J2U=', 'Content-Length': '0'} 2024-02-01T10:49:56.132 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc8679e40f429c3e-0065bb7754-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:56.132 DEBUG:boto:b'vclfcn-62obj11000falseobj12024-02-01T10:48:26.315Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:56.133 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:56.133 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:56.133 DEBUG:boto:path=/vclfcn-62/?prefix=obj1 2024-02-01T10:49:56.133 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj1 2024-02-01T10:49:56.133 DEBUG:boto:Method: GET 2024-02-01T10:49:56.134 DEBUG:boto:Path: /vclfcn-62/?prefix=obj1 2024-02-01T10:49:56.134 DEBUG:boto:Data: 2024-02-01T10:49:56.134 DEBUG:boto:Headers: {} 2024-02-01T10:49:56.134 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:56.134 DEBUG:boto:Port: 8000 2024-02-01T10:49:56.134 DEBUG:boto:Params: {} 2024-02-01T10:49:56.134 DEBUG:boto:Token: None 2024-02-01T10:49:56.134 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:56 GMT /vclfcn-62/ 2024-02-01T10:49:56.134 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:rhqfdyVzV8k/sJU5MGBgCWI9J2U= 2024-02-01T10:49:56.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:rhqfdyVzV8k/sJU5MGBgCWI9J2U=', 'Content-Length': '0'} 2024-02-01T10:49:56.139 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005267fb5707ad8745-0065bb7754-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:56.180 DEBUG:boto:b'vclfcn-62obj11000falseobj12024-02-01T10:48:26.315Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:56.181 DEBUG:boto:path=/vclfcn-62/obj1 2024-02-01T10:49:56.181 DEBUG:boto:auth_path=/vclfcn-62/obj1 2024-02-01T10:49:56.181 DEBUG:boto:Method: HEAD 2024-02-01T10:49:56.181 DEBUG:boto:Path: /vclfcn-62/obj1 2024-02-01T10:49:56.181 DEBUG:boto:Data: 2024-02-01T10:49:56.181 DEBUG:boto:Headers: {} 2024-02-01T10:49:56.181 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:56.181 DEBUG:boto:Port: 8001 2024-02-01T10:49:56.181 DEBUG:boto:Params: {} 2024-02-01T10:49:56.182 DEBUG:boto:Token: None 2024-02-01T10:49:56.182 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:56 GMT /vclfcn-62/obj1 2024-02-01T10:49:56.182 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:N4ziXpXXYUKa8dzBb3Br3xUIwc0= 2024-02-01T10:49:56.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:N4ziXpXXYUKa8dzBb3Br3xUIwc0=', 'Content-Length': '0'} 2024-02-01T10:49:56.185 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005d6bb32d1dd37a13-0065bb7754-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:56.185 DEBUG:boto:path=/vclfcn-62/obj1 2024-02-01T10:49:56.185 DEBUG:boto:auth_path=/vclfcn-62/obj1 2024-02-01T10:49:56.185 DEBUG:boto:Method: HEAD 2024-02-01T10:49:56.186 DEBUG:boto:Path: /vclfcn-62/obj1 2024-02-01T10:49:56.186 DEBUG:boto:Data: 2024-02-01T10:49:56.186 DEBUG:boto:Headers: {} 2024-02-01T10:49:56.186 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:56.186 DEBUG:boto:Port: 8000 2024-02-01T10:49:56.186 DEBUG:boto:Params: {} 2024-02-01T10:49:56.187 DEBUG:boto:Token: None 2024-02-01T10:49:56.187 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:56 GMT /vclfcn-62/obj1 2024-02-01T10:49:56.187 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:N4ziXpXXYUKa8dzBb3Br3xUIwc0= 2024-02-01T10:49:56.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:N4ziXpXXYUKa8dzBb3Br3xUIwc0=', 'Content-Length': '0'} 2024-02-01T10:49:56.190 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fef8d401886e2640-0065bb7754-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:56.190 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:49:56.191 DEBUG:boto:path=/vclfcn-62/obj1 2024-02-01T10:49:56.191 DEBUG:boto:auth_path=/vclfcn-62/obj1 2024-02-01T10:49:56.191 DEBUG:boto:Method: GET 2024-02-01T10:49:56.191 DEBUG:boto:Path: /vclfcn-62/obj1 2024-02-01T10:49:56.191 DEBUG:boto:Data: 2024-02-01T10:49:56.191 DEBUG:boto:Headers: {} 2024-02-01T10:49:56.191 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:56.191 DEBUG:boto:Port: 8001 2024-02-01T10:49:56.192 DEBUG:boto:Params: {} 2024-02-01T10:49:56.192 DEBUG:boto:Token: None 2024-02-01T10:49:56.192 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:56 GMT /vclfcn-62/obj1 2024-02-01T10:49:56.192 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:pSxlHb4/WkXNuSzMzrIhX6CS9G0= 2024-02-01T10:49:56.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:pSxlHb4/WkXNuSzMzrIhX6CS9G0=', 'Content-Length': '0'} 2024-02-01T10:49:56.195 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000200017584bf96b8e-0065bb7754-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:56.240 DEBUG:boto:path=/vclfcn-62/obj1 2024-02-01T10:49:56.240 DEBUG:boto:auth_path=/vclfcn-62/obj1 2024-02-01T10:49:56.240 DEBUG:boto:Method: GET 2024-02-01T10:49:56.240 DEBUG:boto:Path: /vclfcn-62/obj1 2024-02-01T10:49:56.241 DEBUG:boto:Data: 2024-02-01T10:49:56.241 DEBUG:boto:Headers: {} 2024-02-01T10:49:56.241 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:56.241 DEBUG:boto:Port: 8000 2024-02-01T10:49:56.241 DEBUG:boto:Params: {} 2024-02-01T10:49:56.241 DEBUG:boto:Token: None 2024-02-01T10:49:56.241 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:56 GMT /vclfcn-62/obj1 2024-02-01T10:49:56.241 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:pSxlHb4/WkXNuSzMzrIhX6CS9G0= 2024-02-01T10:49:56.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:pSxlHb4/WkXNuSzMzrIhX6CS9G0=', 'Content-Length': '0'} 2024-02-01T10:49:56.244 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000edd2c7191ad31e4e-0065bb7754-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:56.245 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:56.245 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:56.245 DEBUG:boto:path=/vclfcn-62/?prefix=obj1 2024-02-01T10:49:56.245 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj1 2024-02-01T10:49:56.245 DEBUG:boto:Method: GET 2024-02-01T10:49:56.246 DEBUG:boto:Path: /vclfcn-62/?prefix=obj1 2024-02-01T10:49:56.246 DEBUG:boto:Data: 2024-02-01T10:49:56.246 DEBUG:boto:Headers: {} 2024-02-01T10:49:56.246 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:56.246 DEBUG:boto:Port: 8001 2024-02-01T10:49:56.246 DEBUG:boto:Params: {} 2024-02-01T10:49:56.246 DEBUG:boto:Token: None 2024-02-01T10:49:56.246 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:56 GMT /vclfcn-62/ 2024-02-01T10:49:56.246 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:rhqfdyVzV8k/sJU5MGBgCWI9J2U= 2024-02-01T10:49:56.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:rhqfdyVzV8k/sJU5MGBgCWI9J2U=', 'Content-Length': '0'} 2024-02-01T10:49:56.250 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c03786e97a199000-0065bb7754-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:56.251 DEBUG:boto:b'vclfcn-62obj11000falseobj12024-02-01T10:48:26.315Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:56.251 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:56.251 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:56.251 DEBUG:boto:path=/vclfcn-62/?prefix=obj1 2024-02-01T10:49:56.251 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj1 2024-02-01T10:49:56.252 DEBUG:boto:Method: GET 2024-02-01T10:49:56.252 DEBUG:boto:Path: /vclfcn-62/?prefix=obj1 2024-02-01T10:49:56.252 DEBUG:boto:Data: 2024-02-01T10:49:56.252 DEBUG:boto:Headers: {} 2024-02-01T10:49:56.252 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:56.252 DEBUG:boto:Port: 8000 2024-02-01T10:49:56.252 DEBUG:boto:Params: {} 2024-02-01T10:49:56.253 DEBUG:boto:Token: None 2024-02-01T10:49:56.253 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:56 GMT /vclfcn-62/ 2024-02-01T10:49:56.253 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:rhqfdyVzV8k/sJU5MGBgCWI9J2U= 2024-02-01T10:49:56.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:rhqfdyVzV8k/sJU5MGBgCWI9J2U=', 'Content-Length': '0'} 2024-02-01T10:49:56.257 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8da10dbe5ac2dfb-0065bb7754-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:56.258 DEBUG:boto:b'vclfcn-62obj11000falseobj12024-02-01T10:48:26.315Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:56.258 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:49:56.258 DEBUG:boto:path=/vclfcn-62/obj2 2024-02-01T10:49:56.259 DEBUG:boto:auth_path=/vclfcn-62/obj2 2024-02-01T10:49:56.259 DEBUG:boto:path=/vclfcn-62/obj2?versionId=null 2024-02-01T10:49:56.259 DEBUG:boto:auth_path=/vclfcn-62/obj2?versionId=null 2024-02-01T10:49:56.259 DEBUG:boto:Method: GET 2024-02-01T10:49:56.259 DEBUG:boto:Path: /vclfcn-62/obj2?versionId=null 2024-02-01T10:49:56.260 DEBUG:boto:Data: 2024-02-01T10:49:56.260 DEBUG:boto:Headers: {} 2024-02-01T10:49:56.260 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:56.260 DEBUG:boto:Port: 8001 2024-02-01T10:49:56.260 DEBUG:boto:Params: {} 2024-02-01T10:49:56.261 DEBUG:boto:Token: None 2024-02-01T10:49:56.261 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:56 GMT /vclfcn-62/obj2?versionId=null 2024-02-01T10:49:56.261 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mKnv+luKN1KtlhOjgI0blC/CMBs= 2024-02-01T10:49:56.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mKnv+luKN1KtlhOjgI0blC/CMBs=', 'Content-Length': '0'} 2024-02-01T10:49:56.264 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007779673c28ed8fc8-0065bb7754-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:56.265 DEBUG:boto:path=/vclfcn-62/obj2 2024-02-01T10:49:56.265 DEBUG:boto:auth_path=/vclfcn-62/obj2 2024-02-01T10:49:56.265 DEBUG:boto:path=/vclfcn-62/obj2?versionId=null 2024-02-01T10:49:56.265 DEBUG:boto:auth_path=/vclfcn-62/obj2?versionId=null 2024-02-01T10:49:56.265 DEBUG:boto:Method: GET 2024-02-01T10:49:56.265 DEBUG:boto:Path: /vclfcn-62/obj2?versionId=null 2024-02-01T10:49:56.265 DEBUG:boto:Data: 2024-02-01T10:49:56.265 DEBUG:boto:Headers: {} 2024-02-01T10:49:56.265 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:56.265 DEBUG:boto:Port: 8000 2024-02-01T10:49:56.266 DEBUG:boto:Params: {} 2024-02-01T10:49:56.266 DEBUG:boto:Token: None 2024-02-01T10:49:56.266 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:56 GMT /vclfcn-62/obj2?versionId=null 2024-02-01T10:49:56.266 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mKnv+luKN1KtlhOjgI0blC/CMBs= 2024-02-01T10:49:56.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mKnv+luKN1KtlhOjgI0blC/CMBs=', 'Content-Length': '0'} 2024-02-01T10:49:56.270 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000089f26e166e6baf0b-0065bb7754-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:56.316 DEBUG:boto:path=/vclfcn-62/obj2 2024-02-01T10:49:56.316 DEBUG:boto:auth_path=/vclfcn-62/obj2 2024-02-01T10:49:56.316 DEBUG:boto:path=/vclfcn-62/obj2?versionId=null 2024-02-01T10:49:56.316 DEBUG:boto:auth_path=/vclfcn-62/obj2?versionId=null 2024-02-01T10:49:56.317 DEBUG:boto:Method: HEAD 2024-02-01T10:49:56.317 DEBUG:boto:Path: /vclfcn-62/obj2?versionId=null 2024-02-01T10:49:56.317 DEBUG:boto:Data: 2024-02-01T10:49:56.317 DEBUG:boto:Headers: {} 2024-02-01T10:49:56.317 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:56.317 DEBUG:boto:Port: 8001 2024-02-01T10:49:56.317 DEBUG:boto:Params: {} 2024-02-01T10:49:56.317 DEBUG:boto:Token: None 2024-02-01T10:49:56.317 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:56 GMT /vclfcn-62/obj2?versionId=null 2024-02-01T10:49:56.318 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:LcyS6mN4J7U2qrWNYNyAyXP949o= 2024-02-01T10:49:56.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:LcyS6mN4J7U2qrWNYNyAyXP949o=', 'Content-Length': '0'} 2024-02-01T10:49:56.320 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004d4b2edc50b7b71e-0065bb7754-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:56.321 DEBUG:boto:path=/vclfcn-62/obj2 2024-02-01T10:49:56.321 DEBUG:boto:auth_path=/vclfcn-62/obj2 2024-02-01T10:49:56.321 DEBUG:boto:path=/vclfcn-62/obj2?versionId=null 2024-02-01T10:49:56.321 DEBUG:boto:auth_path=/vclfcn-62/obj2?versionId=null 2024-02-01T10:49:56.321 DEBUG:boto:Method: HEAD 2024-02-01T10:49:56.321 DEBUG:boto:Path: /vclfcn-62/obj2?versionId=null 2024-02-01T10:49:56.321 DEBUG:boto:Data: 2024-02-01T10:49:56.321 DEBUG:boto:Headers: {} 2024-02-01T10:49:56.322 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:56.322 DEBUG:boto:Port: 8000 2024-02-01T10:49:56.322 DEBUG:boto:Params: {} 2024-02-01T10:49:56.322 DEBUG:boto:Token: None 2024-02-01T10:49:56.322 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:56 GMT /vclfcn-62/obj2?versionId=null 2024-02-01T10:49:56.322 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:LcyS6mN4J7U2qrWNYNyAyXP949o= 2024-02-01T10:49:56.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:LcyS6mN4J7U2qrWNYNyAyXP949o=', 'Content-Length': '0'} 2024-02-01T10:49:56.325 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a34a2c68a82d0463-0065bb7754-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:56.325 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:49:56.326 DEBUG:boto:path=/vclfcn-62/obj2 2024-02-01T10:49:56.326 DEBUG:boto:auth_path=/vclfcn-62/obj2 2024-02-01T10:49:56.326 DEBUG:boto:path=/vclfcn-62/obj2?versionId=null 2024-02-01T10:49:56.326 DEBUG:boto:auth_path=/vclfcn-62/obj2?versionId=null 2024-02-01T10:49:56.326 DEBUG:boto:Method: GET 2024-02-01T10:49:56.326 DEBUG:boto:Path: /vclfcn-62/obj2?versionId=null 2024-02-01T10:49:56.326 DEBUG:boto:Data: 2024-02-01T10:49:56.326 DEBUG:boto:Headers: {} 2024-02-01T10:49:56.326 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:56.326 DEBUG:boto:Port: 8001 2024-02-01T10:49:56.326 DEBUG:boto:Params: {} 2024-02-01T10:49:56.327 DEBUG:boto:Token: None 2024-02-01T10:49:56.327 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:56 GMT /vclfcn-62/obj2?versionId=null 2024-02-01T10:49:56.327 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mKnv+luKN1KtlhOjgI0blC/CMBs= 2024-02-01T10:49:56.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mKnv+luKN1KtlhOjgI0blC/CMBs=', 'Content-Length': '0'} 2024-02-01T10:49:56.330 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a7377235b57b1cac-0065bb7754-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:56.372 DEBUG:boto:path=/vclfcn-62/obj2 2024-02-01T10:49:56.373 DEBUG:boto:auth_path=/vclfcn-62/obj2 2024-02-01T10:49:56.373 DEBUG:boto:path=/vclfcn-62/obj2?versionId=null 2024-02-01T10:49:56.373 DEBUG:boto:auth_path=/vclfcn-62/obj2?versionId=null 2024-02-01T10:49:56.373 DEBUG:boto:Method: GET 2024-02-01T10:49:56.373 DEBUG:boto:Path: /vclfcn-62/obj2?versionId=null 2024-02-01T10:49:56.374 DEBUG:boto:Data: 2024-02-01T10:49:56.374 DEBUG:boto:Headers: {} 2024-02-01T10:49:56.374 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:56.374 DEBUG:boto:Port: 8000 2024-02-01T10:49:56.374 DEBUG:boto:Params: {} 2024-02-01T10:49:56.375 DEBUG:boto:Token: None 2024-02-01T10:49:56.375 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:56 GMT /vclfcn-62/obj2?versionId=null 2024-02-01T10:49:56.375 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mKnv+luKN1KtlhOjgI0blC/CMBs= 2024-02-01T10:49:56.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mKnv+luKN1KtlhOjgI0blC/CMBs=', 'Content-Length': '0'} 2024-02-01T10:49:56.379 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fcb3047c270ed6d4-0065bb7754-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:56.379 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:56.380 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:56.380 DEBUG:boto:path=/vclfcn-62/?prefix=obj2 2024-02-01T10:49:56.380 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj2 2024-02-01T10:49:56.380 DEBUG:boto:Method: GET 2024-02-01T10:49:56.380 DEBUG:boto:Path: /vclfcn-62/?prefix=obj2 2024-02-01T10:49:56.381 DEBUG:boto:Data: 2024-02-01T10:49:56.381 DEBUG:boto:Headers: {} 2024-02-01T10:49:56.381 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:56.381 DEBUG:boto:Port: 8001 2024-02-01T10:49:56.381 DEBUG:boto:Params: {} 2024-02-01T10:49:56.381 DEBUG:boto:Token: None 2024-02-01T10:49:56.382 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:56 GMT /vclfcn-62/ 2024-02-01T10:49:56.382 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:rhqfdyVzV8k/sJU5MGBgCWI9J2U= 2024-02-01T10:49:56.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:rhqfdyVzV8k/sJU5MGBgCWI9J2U=', 'Content-Length': '0'} 2024-02-01T10:49:56.386 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024a8b252ec046c3e-0065bb7754-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:56.386 DEBUG:boto:b'vclfcn-62obj21000falseobj22024-02-01T10:48:26.333Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:56.387 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:56.387 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:56.387 DEBUG:boto:path=/vclfcn-62/?prefix=obj2 2024-02-01T10:49:56.387 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj2 2024-02-01T10:49:56.387 DEBUG:boto:Method: GET 2024-02-01T10:49:56.387 DEBUG:boto:Path: /vclfcn-62/?prefix=obj2 2024-02-01T10:49:56.387 DEBUG:boto:Data: 2024-02-01T10:49:56.388 DEBUG:boto:Headers: {} 2024-02-01T10:49:56.388 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:56.388 DEBUG:boto:Port: 8000 2024-02-01T10:49:56.388 DEBUG:boto:Params: {} 2024-02-01T10:49:56.388 DEBUG:boto:Token: None 2024-02-01T10:49:56.388 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:56 GMT /vclfcn-62/ 2024-02-01T10:49:56.388 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:rhqfdyVzV8k/sJU5MGBgCWI9J2U= 2024-02-01T10:49:56.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:rhqfdyVzV8k/sJU5MGBgCWI9J2U=', 'Content-Length': '0'} 2024-02-01T10:49:56.393 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000879424d67faec01d-0065bb7754-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:56.393 DEBUG:boto:b'vclfcn-62obj21000falseobj22024-02-01T10:48:26.333Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:56.393 DEBUG:boto:path=/vclfcn-62/obj2 2024-02-01T10:49:56.393 DEBUG:boto:auth_path=/vclfcn-62/obj2 2024-02-01T10:49:56.394 DEBUG:boto:Method: HEAD 2024-02-01T10:49:56.394 DEBUG:boto:Path: /vclfcn-62/obj2 2024-02-01T10:49:56.394 DEBUG:boto:Data: 2024-02-01T10:49:56.394 DEBUG:boto:Headers: {} 2024-02-01T10:49:56.394 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:56.394 DEBUG:boto:Port: 8001 2024-02-01T10:49:56.394 DEBUG:boto:Params: {} 2024-02-01T10:49:56.394 DEBUG:boto:Token: None 2024-02-01T10:49:56.394 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:56 GMT /vclfcn-62/obj2 2024-02-01T10:49:56.395 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:AOetY9DvH9QaasoGAHM6JgJZ1EY= 2024-02-01T10:49:56.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:AOetY9DvH9QaasoGAHM6JgJZ1EY=', 'Content-Length': '0'} 2024-02-01T10:49:56.397 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006683d5a71871e8ab-0065bb7754-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:56.398 DEBUG:boto:path=/vclfcn-62/obj2 2024-02-01T10:49:56.398 DEBUG:boto:auth_path=/vclfcn-62/obj2 2024-02-01T10:49:56.398 DEBUG:boto:Method: HEAD 2024-02-01T10:49:56.398 DEBUG:boto:Path: /vclfcn-62/obj2 2024-02-01T10:49:56.398 DEBUG:boto:Data: 2024-02-01T10:49:56.398 DEBUG:boto:Headers: {} 2024-02-01T10:49:56.398 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:56.398 DEBUG:boto:Port: 8000 2024-02-01T10:49:56.398 DEBUG:boto:Params: {} 2024-02-01T10:49:56.399 DEBUG:boto:Token: None 2024-02-01T10:49:56.399 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:56 GMT /vclfcn-62/obj2 2024-02-01T10:49:56.399 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:AOetY9DvH9QaasoGAHM6JgJZ1EY= 2024-02-01T10:49:56.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:AOetY9DvH9QaasoGAHM6JgJZ1EY=', 'Content-Length': '0'} 2024-02-01T10:49:56.401 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000d6810372061244c-0065bb7754-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:56.402 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:49:56.402 DEBUG:boto:path=/vclfcn-62/obj2 2024-02-01T10:49:56.402 DEBUG:boto:auth_path=/vclfcn-62/obj2 2024-02-01T10:49:56.402 DEBUG:boto:Method: GET 2024-02-01T10:49:56.403 DEBUG:boto:Path: /vclfcn-62/obj2 2024-02-01T10:49:56.403 DEBUG:boto:Data: 2024-02-01T10:49:56.403 DEBUG:boto:Headers: {} 2024-02-01T10:49:56.403 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:56.403 DEBUG:boto:Port: 8001 2024-02-01T10:49:56.403 DEBUG:boto:Params: {} 2024-02-01T10:49:56.403 DEBUG:boto:Token: None 2024-02-01T10:49:56.404 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:56 GMT /vclfcn-62/obj2 2024-02-01T10:49:56.404 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:SQorQo6a/FXOnufD8+O2I0RuUyw= 2024-02-01T10:49:56.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:SQorQo6a/FXOnufD8+O2I0RuUyw=', 'Content-Length': '0'} 2024-02-01T10:49:56.407 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000084391c93f22451f3-0065bb7754-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:56.452 DEBUG:boto:path=/vclfcn-62/obj2 2024-02-01T10:49:56.452 DEBUG:boto:auth_path=/vclfcn-62/obj2 2024-02-01T10:49:56.452 DEBUG:boto:Method: GET 2024-02-01T10:49:56.453 DEBUG:boto:Path: /vclfcn-62/obj2 2024-02-01T10:49:56.453 DEBUG:boto:Data: 2024-02-01T10:49:56.453 DEBUG:boto:Headers: {} 2024-02-01T10:49:56.453 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:56.453 DEBUG:boto:Port: 8000 2024-02-01T10:49:56.453 DEBUG:boto:Params: {} 2024-02-01T10:49:56.453 DEBUG:boto:Token: None 2024-02-01T10:49:56.453 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:56 GMT /vclfcn-62/obj2 2024-02-01T10:49:56.453 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:SQorQo6a/FXOnufD8+O2I0RuUyw= 2024-02-01T10:49:56.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:SQorQo6a/FXOnufD8+O2I0RuUyw=', 'Content-Length': '0'} 2024-02-01T10:49:56.456 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001afc2759d506cb58-0065bb7754-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:56.457 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:56.457 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:56.457 DEBUG:boto:path=/vclfcn-62/?prefix=obj2 2024-02-01T10:49:56.457 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj2 2024-02-01T10:49:56.457 DEBUG:boto:Method: GET 2024-02-01T10:49:56.457 DEBUG:boto:Path: /vclfcn-62/?prefix=obj2 2024-02-01T10:49:56.457 DEBUG:boto:Data: 2024-02-01T10:49:56.457 DEBUG:boto:Headers: {} 2024-02-01T10:49:56.457 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:56.458 DEBUG:boto:Port: 8001 2024-02-01T10:49:56.458 DEBUG:boto:Params: {} 2024-02-01T10:49:56.458 DEBUG:boto:Token: None 2024-02-01T10:49:56.458 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:56 GMT /vclfcn-62/ 2024-02-01T10:49:56.458 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:rhqfdyVzV8k/sJU5MGBgCWI9J2U= 2024-02-01T10:49:56.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:rhqfdyVzV8k/sJU5MGBgCWI9J2U=', 'Content-Length': '0'} 2024-02-01T10:49:56.462 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003cb371365ece5ed0-0065bb7754-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:56.462 DEBUG:boto:b'vclfcn-62obj21000falseobj22024-02-01T10:48:26.333Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:56.463 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:56.463 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:56.463 DEBUG:boto:path=/vclfcn-62/?prefix=obj2 2024-02-01T10:49:56.463 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj2 2024-02-01T10:49:56.463 DEBUG:boto:Method: GET 2024-02-01T10:49:56.463 DEBUG:boto:Path: /vclfcn-62/?prefix=obj2 2024-02-01T10:49:56.463 DEBUG:boto:Data: 2024-02-01T10:49:56.463 DEBUG:boto:Headers: {} 2024-02-01T10:49:56.463 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:56.464 DEBUG:boto:Port: 8000 2024-02-01T10:49:56.464 DEBUG:boto:Params: {} 2024-02-01T10:49:56.464 DEBUG:boto:Token: None 2024-02-01T10:49:56.464 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:56 GMT /vclfcn-62/ 2024-02-01T10:49:56.464 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:rhqfdyVzV8k/sJU5MGBgCWI9J2U= 2024-02-01T10:49:56.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:rhqfdyVzV8k/sJU5MGBgCWI9J2U=', 'Content-Length': '0'} 2024-02-01T10:49:56.468 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004035649061e60a14-0065bb7754-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:56.516 DEBUG:boto:b'vclfcn-62obj21000falseobj22024-02-01T10:48:26.333Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:56.516 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:49:56.517 DEBUG:boto:path=/vclfcn-62/obj3 2024-02-01T10:49:56.517 DEBUG:boto:auth_path=/vclfcn-62/obj3 2024-02-01T10:49:56.517 DEBUG:boto:path=/vclfcn-62/obj3?versionId=null 2024-02-01T10:49:56.517 DEBUG:boto:auth_path=/vclfcn-62/obj3?versionId=null 2024-02-01T10:49:56.517 DEBUG:boto:Method: GET 2024-02-01T10:49:56.517 DEBUG:boto:Path: /vclfcn-62/obj3?versionId=null 2024-02-01T10:49:56.517 DEBUG:boto:Data: 2024-02-01T10:49:56.517 DEBUG:boto:Headers: {} 2024-02-01T10:49:56.517 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:56.517 DEBUG:boto:Port: 8001 2024-02-01T10:49:56.518 DEBUG:boto:Params: {} 2024-02-01T10:49:56.518 DEBUG:boto:Token: None 2024-02-01T10:49:56.518 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:56 GMT /vclfcn-62/obj3?versionId=null 2024-02-01T10:49:56.518 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:X5CS2f7KeumIAlYqWRjHrHDkmts= 2024-02-01T10:49:56.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:X5CS2f7KeumIAlYqWRjHrHDkmts=', 'Content-Length': '0'} 2024-02-01T10:49:56.521 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ae6fbb2c9916a07a-0065bb7754-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:56.521 DEBUG:boto:path=/vclfcn-62/obj3 2024-02-01T10:49:56.521 DEBUG:boto:auth_path=/vclfcn-62/obj3 2024-02-01T10:49:56.521 DEBUG:boto:path=/vclfcn-62/obj3?versionId=null 2024-02-01T10:49:56.521 DEBUG:boto:auth_path=/vclfcn-62/obj3?versionId=null 2024-02-01T10:49:56.522 DEBUG:boto:Method: GET 2024-02-01T10:49:56.522 DEBUG:boto:Path: /vclfcn-62/obj3?versionId=null 2024-02-01T10:49:56.522 DEBUG:boto:Data: 2024-02-01T10:49:56.522 DEBUG:boto:Headers: {} 2024-02-01T10:49:56.522 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:56.522 DEBUG:boto:Port: 8000 2024-02-01T10:49:56.522 DEBUG:boto:Params: {} 2024-02-01T10:49:56.522 DEBUG:boto:Token: None 2024-02-01T10:49:56.522 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:56 GMT /vclfcn-62/obj3?versionId=null 2024-02-01T10:49:56.523 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:X5CS2f7KeumIAlYqWRjHrHDkmts= 2024-02-01T10:49:56.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:X5CS2f7KeumIAlYqWRjHrHDkmts=', 'Content-Length': '0'} 2024-02-01T10:49:56.526 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f123461541130c32-0065bb7754-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:56.526 DEBUG:boto:path=/vclfcn-62/obj3 2024-02-01T10:49:56.527 DEBUG:boto:auth_path=/vclfcn-62/obj3 2024-02-01T10:49:56.527 DEBUG:boto:path=/vclfcn-62/obj3?versionId=null 2024-02-01T10:49:56.527 DEBUG:boto:auth_path=/vclfcn-62/obj3?versionId=null 2024-02-01T10:49:56.527 DEBUG:boto:Method: HEAD 2024-02-01T10:49:56.527 DEBUG:boto:Path: /vclfcn-62/obj3?versionId=null 2024-02-01T10:49:56.527 DEBUG:boto:Data: 2024-02-01T10:49:56.527 DEBUG:boto:Headers: {} 2024-02-01T10:49:56.527 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:56.527 DEBUG:boto:Port: 8001 2024-02-01T10:49:56.527 DEBUG:boto:Params: {} 2024-02-01T10:49:56.527 DEBUG:boto:Token: None 2024-02-01T10:49:56.528 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:56 GMT /vclfcn-62/obj3?versionId=null 2024-02-01T10:49:56.528 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fNfHsNw33SVNYjrEMj4/HmDn43w= 2024-02-01T10:49:56.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fNfHsNw33SVNYjrEMj4/HmDn43w=', 'Content-Length': '0'} 2024-02-01T10:49:56.530 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006c461d33ad3782b0-0065bb7754-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:56.531 DEBUG:boto:path=/vclfcn-62/obj3 2024-02-01T10:49:56.531 DEBUG:boto:auth_path=/vclfcn-62/obj3 2024-02-01T10:49:56.531 DEBUG:boto:path=/vclfcn-62/obj3?versionId=null 2024-02-01T10:49:56.531 DEBUG:boto:auth_path=/vclfcn-62/obj3?versionId=null 2024-02-01T10:49:56.531 DEBUG:boto:Method: HEAD 2024-02-01T10:49:56.531 DEBUG:boto:Path: /vclfcn-62/obj3?versionId=null 2024-02-01T10:49:56.531 DEBUG:boto:Data: 2024-02-01T10:49:56.531 DEBUG:boto:Headers: {} 2024-02-01T10:49:56.531 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:56.531 DEBUG:boto:Port: 8000 2024-02-01T10:49:56.531 DEBUG:boto:Params: {} 2024-02-01T10:49:56.532 DEBUG:boto:Token: None 2024-02-01T10:49:56.532 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:56 GMT /vclfcn-62/obj3?versionId=null 2024-02-01T10:49:56.532 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fNfHsNw33SVNYjrEMj4/HmDn43w= 2024-02-01T10:49:56.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fNfHsNw33SVNYjrEMj4/HmDn43w=', 'Content-Length': '0'} 2024-02-01T10:49:56.534 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007f344887063c1864-0065bb7754-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:56.535 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:49:56.535 DEBUG:boto:path=/vclfcn-62/obj3 2024-02-01T10:49:56.535 DEBUG:boto:auth_path=/vclfcn-62/obj3 2024-02-01T10:49:56.535 DEBUG:boto:path=/vclfcn-62/obj3?versionId=null 2024-02-01T10:49:56.535 DEBUG:boto:auth_path=/vclfcn-62/obj3?versionId=null 2024-02-01T10:49:56.535 DEBUG:boto:Method: GET 2024-02-01T10:49:56.535 DEBUG:boto:Path: /vclfcn-62/obj3?versionId=null 2024-02-01T10:49:56.535 DEBUG:boto:Data: 2024-02-01T10:49:56.535 DEBUG:boto:Headers: {} 2024-02-01T10:49:56.536 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:56.536 DEBUG:boto:Port: 8001 2024-02-01T10:49:56.536 DEBUG:boto:Params: {} 2024-02-01T10:49:56.536 DEBUG:boto:Token: None 2024-02-01T10:49:56.536 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:56 GMT /vclfcn-62/obj3?versionId=null 2024-02-01T10:49:56.536 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:X5CS2f7KeumIAlYqWRjHrHDkmts= 2024-02-01T10:49:56.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:X5CS2f7KeumIAlYqWRjHrHDkmts=', 'Content-Length': '0'} 2024-02-01T10:49:56.539 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000a684bd660f4fbd4-0065bb7754-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:56.584 DEBUG:boto:path=/vclfcn-62/obj3 2024-02-01T10:49:56.584 DEBUG:boto:auth_path=/vclfcn-62/obj3 2024-02-01T10:49:56.584 DEBUG:boto:path=/vclfcn-62/obj3?versionId=null 2024-02-01T10:49:56.585 DEBUG:boto:auth_path=/vclfcn-62/obj3?versionId=null 2024-02-01T10:49:56.585 DEBUG:boto:Method: GET 2024-02-01T10:49:56.585 DEBUG:boto:Path: /vclfcn-62/obj3?versionId=null 2024-02-01T10:49:56.585 DEBUG:boto:Data: 2024-02-01T10:49:56.585 DEBUG:boto:Headers: {} 2024-02-01T10:49:56.585 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:56.585 DEBUG:boto:Port: 8000 2024-02-01T10:49:56.585 DEBUG:boto:Params: {} 2024-02-01T10:49:56.586 DEBUG:boto:Token: None 2024-02-01T10:49:56.586 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:56 GMT /vclfcn-62/obj3?versionId=null 2024-02-01T10:49:56.586 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:X5CS2f7KeumIAlYqWRjHrHDkmts= 2024-02-01T10:49:56.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:X5CS2f7KeumIAlYqWRjHrHDkmts=', 'Content-Length': '0'} 2024-02-01T10:49:56.589 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009fd1a64e7ab6adaa-0065bb7754-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:56.589 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:56.590 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:56.590 DEBUG:boto:path=/vclfcn-62/?prefix=obj3 2024-02-01T10:49:56.590 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj3 2024-02-01T10:49:56.590 DEBUG:boto:Method: GET 2024-02-01T10:49:56.590 DEBUG:boto:Path: /vclfcn-62/?prefix=obj3 2024-02-01T10:49:56.590 DEBUG:boto:Data: 2024-02-01T10:49:56.590 DEBUG:boto:Headers: {} 2024-02-01T10:49:56.590 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:56.590 DEBUG:boto:Port: 8001 2024-02-01T10:49:56.590 DEBUG:boto:Params: {} 2024-02-01T10:49:56.591 DEBUG:boto:Token: None 2024-02-01T10:49:56.591 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:56 GMT /vclfcn-62/ 2024-02-01T10:49:56.591 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:rhqfdyVzV8k/sJU5MGBgCWI9J2U= 2024-02-01T10:49:56.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:rhqfdyVzV8k/sJU5MGBgCWI9J2U=', 'Content-Length': '0'} 2024-02-01T10:49:56.595 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000085cc5f9e4203d274-0065bb7754-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:56.595 DEBUG:boto:b'vclfcn-62obj31000falseobj32024-02-01T10:48:26.384Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:56.596 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:56.596 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:56.596 DEBUG:boto:path=/vclfcn-62/?prefix=obj3 2024-02-01T10:49:56.597 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj3 2024-02-01T10:49:56.597 DEBUG:boto:Method: GET 2024-02-01T10:49:56.597 DEBUG:boto:Path: /vclfcn-62/?prefix=obj3 2024-02-01T10:49:56.597 DEBUG:boto:Data: 2024-02-01T10:49:56.597 DEBUG:boto:Headers: {} 2024-02-01T10:49:56.597 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:56.597 DEBUG:boto:Port: 8000 2024-02-01T10:49:56.597 DEBUG:boto:Params: {} 2024-02-01T10:49:56.598 DEBUG:boto:Token: None 2024-02-01T10:49:56.598 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:56 GMT /vclfcn-62/ 2024-02-01T10:49:56.598 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:rhqfdyVzV8k/sJU5MGBgCWI9J2U= 2024-02-01T10:49:56.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:rhqfdyVzV8k/sJU5MGBgCWI9J2U=', 'Content-Length': '0'} 2024-02-01T10:49:56.602 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004cc275f5b2940ecb-0065bb7754-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:56.644 DEBUG:boto:b'vclfcn-62obj31000falseobj32024-02-01T10:48:26.384Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:56.644 DEBUG:boto:path=/vclfcn-62/obj3 2024-02-01T10:49:56.645 DEBUG:boto:auth_path=/vclfcn-62/obj3 2024-02-01T10:49:56.645 DEBUG:boto:Method: HEAD 2024-02-01T10:49:56.645 DEBUG:boto:Path: /vclfcn-62/obj3 2024-02-01T10:49:56.645 DEBUG:boto:Data: 2024-02-01T10:49:56.645 DEBUG:boto:Headers: {} 2024-02-01T10:49:56.645 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:56.645 DEBUG:boto:Port: 8001 2024-02-01T10:49:56.645 DEBUG:boto:Params: {} 2024-02-01T10:49:56.645 DEBUG:boto:Token: None 2024-02-01T10:49:56.646 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:56 GMT /vclfcn-62/obj3 2024-02-01T10:49:56.646 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:KaC/Ykc1PkYKXLaB6eFWZydRbp8= 2024-02-01T10:49:56.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:KaC/Ykc1PkYKXLaB6eFWZydRbp8=', 'Content-Length': '0'} 2024-02-01T10:49:56.648 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005f69b2b12affe446-0065bb7754-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:56.649 DEBUG:boto:path=/vclfcn-62/obj3 2024-02-01T10:49:56.649 DEBUG:boto:auth_path=/vclfcn-62/obj3 2024-02-01T10:49:56.649 DEBUG:boto:Method: HEAD 2024-02-01T10:49:56.649 DEBUG:boto:Path: /vclfcn-62/obj3 2024-02-01T10:49:56.649 DEBUG:boto:Data: 2024-02-01T10:49:56.649 DEBUG:boto:Headers: {} 2024-02-01T10:49:56.649 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:56.649 DEBUG:boto:Port: 8000 2024-02-01T10:49:56.649 DEBUG:boto:Params: {} 2024-02-01T10:49:56.650 DEBUG:boto:Token: None 2024-02-01T10:49:56.650 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:56 GMT /vclfcn-62/obj3 2024-02-01T10:49:56.650 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:KaC/Ykc1PkYKXLaB6eFWZydRbp8= 2024-02-01T10:49:56.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:KaC/Ykc1PkYKXLaB6eFWZydRbp8=', 'Content-Length': '0'} 2024-02-01T10:49:56.653 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000097b048718e096c71-0065bb7754-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:56.653 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:49:56.653 DEBUG:boto:path=/vclfcn-62/obj3 2024-02-01T10:49:56.653 DEBUG:boto:auth_path=/vclfcn-62/obj3 2024-02-01T10:49:56.653 DEBUG:boto:Method: GET 2024-02-01T10:49:56.653 DEBUG:boto:Path: /vclfcn-62/obj3 2024-02-01T10:49:56.654 DEBUG:boto:Data: 2024-02-01T10:49:56.654 DEBUG:boto:Headers: {} 2024-02-01T10:49:56.654 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:56.654 DEBUG:boto:Port: 8001 2024-02-01T10:49:56.654 DEBUG:boto:Params: {} 2024-02-01T10:49:56.654 DEBUG:boto:Token: None 2024-02-01T10:49:56.654 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:56 GMT /vclfcn-62/obj3 2024-02-01T10:49:56.654 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:KJGVSWtoY3TRozzDcqm8iRIae0Q= 2024-02-01T10:49:56.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:KJGVSWtoY3TRozzDcqm8iRIae0Q=', 'Content-Length': '0'} 2024-02-01T10:49:56.657 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003e96b8e93831b37e-0065bb7754-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:56.700 DEBUG:boto:path=/vclfcn-62/obj3 2024-02-01T10:49:56.700 DEBUG:boto:auth_path=/vclfcn-62/obj3 2024-02-01T10:49:56.701 DEBUG:boto:Method: GET 2024-02-01T10:49:56.701 DEBUG:boto:Path: /vclfcn-62/obj3 2024-02-01T10:49:56.701 DEBUG:boto:Data: 2024-02-01T10:49:56.701 DEBUG:boto:Headers: {} 2024-02-01T10:49:56.701 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:56.701 DEBUG:boto:Port: 8000 2024-02-01T10:49:56.701 DEBUG:boto:Params: {} 2024-02-01T10:49:56.701 DEBUG:boto:Token: None 2024-02-01T10:49:56.702 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:56 GMT /vclfcn-62/obj3 2024-02-01T10:49:56.702 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:KJGVSWtoY3TRozzDcqm8iRIae0Q= 2024-02-01T10:49:56.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:KJGVSWtoY3TRozzDcqm8iRIae0Q=', 'Content-Length': '0'} 2024-02-01T10:49:56.705 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000047f86b6decc9a837-0065bb7754-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:56.705 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:56.705 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:56.705 DEBUG:boto:path=/vclfcn-62/?prefix=obj3 2024-02-01T10:49:56.705 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj3 2024-02-01T10:49:56.706 DEBUG:boto:Method: GET 2024-02-01T10:49:56.706 DEBUG:boto:Path: /vclfcn-62/?prefix=obj3 2024-02-01T10:49:56.706 DEBUG:boto:Data: 2024-02-01T10:49:56.706 DEBUG:boto:Headers: {} 2024-02-01T10:49:56.706 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:56.706 DEBUG:boto:Port: 8001 2024-02-01T10:49:56.706 DEBUG:boto:Params: {} 2024-02-01T10:49:56.707 DEBUG:boto:Token: None 2024-02-01T10:49:56.707 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:56 GMT /vclfcn-62/ 2024-02-01T10:49:56.707 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:rhqfdyVzV8k/sJU5MGBgCWI9J2U= 2024-02-01T10:49:56.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:rhqfdyVzV8k/sJU5MGBgCWI9J2U=', 'Content-Length': '0'} 2024-02-01T10:49:56.712 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000190180fd7f715d13-0065bb7754-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:56.712 DEBUG:boto:b'vclfcn-62obj31000falseobj32024-02-01T10:48:26.384Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:56.713 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:56.713 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:56.714 DEBUG:boto:path=/vclfcn-62/?prefix=obj3 2024-02-01T10:49:56.714 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj3 2024-02-01T10:49:56.714 DEBUG:boto:Method: GET 2024-02-01T10:49:56.714 DEBUG:boto:Path: /vclfcn-62/?prefix=obj3 2024-02-01T10:49:56.714 DEBUG:boto:Data: 2024-02-01T10:49:56.714 DEBUG:boto:Headers: {} 2024-02-01T10:49:56.715 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:56.715 DEBUG:boto:Port: 8000 2024-02-01T10:49:56.715 DEBUG:boto:Params: {} 2024-02-01T10:49:56.715 DEBUG:boto:Token: None 2024-02-01T10:49:56.715 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:56 GMT /vclfcn-62/ 2024-02-01T10:49:56.716 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:rhqfdyVzV8k/sJU5MGBgCWI9J2U= 2024-02-01T10:49:56.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:rhqfdyVzV8k/sJU5MGBgCWI9J2U=', 'Content-Length': '0'} 2024-02-01T10:49:56.721 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab722fc47d761959-0065bb7754-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:56.721 DEBUG:boto:b'vclfcn-62obj31000falseobj32024-02-01T10:48:26.384Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:56.722 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:49:56.722 DEBUG:boto:path=/vclfcn-62/obj4 2024-02-01T10:49:56.722 DEBUG:boto:auth_path=/vclfcn-62/obj4 2024-02-01T10:49:56.723 DEBUG:boto:path=/vclfcn-62/obj4?versionId=null 2024-02-01T10:49:56.723 DEBUG:boto:auth_path=/vclfcn-62/obj4?versionId=null 2024-02-01T10:49:56.723 DEBUG:boto:Method: GET 2024-02-01T10:49:56.723 DEBUG:boto:Path: /vclfcn-62/obj4?versionId=null 2024-02-01T10:49:56.723 DEBUG:boto:Data: 2024-02-01T10:49:56.724 DEBUG:boto:Headers: {} 2024-02-01T10:49:56.724 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:56.724 DEBUG:boto:Port: 8001 2024-02-01T10:49:56.724 DEBUG:boto:Params: {} 2024-02-01T10:49:56.725 DEBUG:boto:Token: None 2024-02-01T10:49:56.725 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:56 GMT /vclfcn-62/obj4?versionId=null 2024-02-01T10:49:56.725 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:KtX85YxqdvkmooaQ7a2r2wg/0Nk= 2024-02-01T10:49:56.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:KtX85YxqdvkmooaQ7a2r2wg/0Nk=', 'Content-Length': '0'} 2024-02-01T10:49:56.728 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000254308d212e4ac38-0065bb7754-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:56.729 DEBUG:boto:path=/vclfcn-62/obj4 2024-02-01T10:49:56.729 DEBUG:boto:auth_path=/vclfcn-62/obj4 2024-02-01T10:49:56.729 DEBUG:boto:path=/vclfcn-62/obj4?versionId=null 2024-02-01T10:49:56.729 DEBUG:boto:auth_path=/vclfcn-62/obj4?versionId=null 2024-02-01T10:49:56.729 DEBUG:boto:Method: GET 2024-02-01T10:49:56.729 DEBUG:boto:Path: /vclfcn-62/obj4?versionId=null 2024-02-01T10:49:56.729 DEBUG:boto:Data: 2024-02-01T10:49:56.730 DEBUG:boto:Headers: {} 2024-02-01T10:49:56.730 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:56.730 DEBUG:boto:Port: 8000 2024-02-01T10:49:56.730 DEBUG:boto:Params: {} 2024-02-01T10:49:56.730 DEBUG:boto:Token: None 2024-02-01T10:49:56.730 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:56 GMT /vclfcn-62/obj4?versionId=null 2024-02-01T10:49:56.730 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:KtX85YxqdvkmooaQ7a2r2wg/0Nk= 2024-02-01T10:49:56.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:KtX85YxqdvkmooaQ7a2r2wg/0Nk=', 'Content-Length': '0'} 2024-02-01T10:49:56.736 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000007957e3930cdf0e7-0065bb7754-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:56.780 DEBUG:boto:path=/vclfcn-62/obj4 2024-02-01T10:49:56.780 DEBUG:boto:auth_path=/vclfcn-62/obj4 2024-02-01T10:49:56.780 DEBUG:boto:path=/vclfcn-62/obj4?versionId=null 2024-02-01T10:49:56.780 DEBUG:boto:auth_path=/vclfcn-62/obj4?versionId=null 2024-02-01T10:49:56.780 DEBUG:boto:Method: HEAD 2024-02-01T10:49:56.781 DEBUG:boto:Path: /vclfcn-62/obj4?versionId=null 2024-02-01T10:49:56.781 DEBUG:boto:Data: 2024-02-01T10:49:56.781 DEBUG:boto:Headers: {} 2024-02-01T10:49:56.781 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:56.781 DEBUG:boto:Port: 8001 2024-02-01T10:49:56.781 DEBUG:boto:Params: {} 2024-02-01T10:49:56.781 DEBUG:boto:Token: None 2024-02-01T10:49:56.781 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:56 GMT /vclfcn-62/obj4?versionId=null 2024-02-01T10:49:56.781 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WYmCTyNJpZ2eQnfszUZdoveT66s= 2024-02-01T10:49:56.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WYmCTyNJpZ2eQnfszUZdoveT66s=', 'Content-Length': '0'} 2024-02-01T10:49:56.784 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eba3d70b746acfaa-0065bb7754-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:56.785 DEBUG:boto:path=/vclfcn-62/obj4 2024-02-01T10:49:56.785 DEBUG:boto:auth_path=/vclfcn-62/obj4 2024-02-01T10:49:56.785 DEBUG:boto:path=/vclfcn-62/obj4?versionId=null 2024-02-01T10:49:56.785 DEBUG:boto:auth_path=/vclfcn-62/obj4?versionId=null 2024-02-01T10:49:56.785 DEBUG:boto:Method: HEAD 2024-02-01T10:49:56.785 DEBUG:boto:Path: /vclfcn-62/obj4?versionId=null 2024-02-01T10:49:56.785 DEBUG:boto:Data: 2024-02-01T10:49:56.785 DEBUG:boto:Headers: {} 2024-02-01T10:49:56.785 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:56.785 DEBUG:boto:Port: 8000 2024-02-01T10:49:56.786 DEBUG:boto:Params: {} 2024-02-01T10:49:56.786 DEBUG:boto:Token: None 2024-02-01T10:49:56.786 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:56 GMT /vclfcn-62/obj4?versionId=null 2024-02-01T10:49:56.786 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WYmCTyNJpZ2eQnfszUZdoveT66s= 2024-02-01T10:49:56.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WYmCTyNJpZ2eQnfszUZdoveT66s=', 'Content-Length': '0'} 2024-02-01T10:49:56.789 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e5f63ada794e90ad-0065bb7754-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:56.789 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:49:56.789 DEBUG:boto:path=/vclfcn-62/obj4 2024-02-01T10:49:56.789 DEBUG:boto:auth_path=/vclfcn-62/obj4 2024-02-01T10:49:56.790 DEBUG:boto:path=/vclfcn-62/obj4?versionId=null 2024-02-01T10:49:56.790 DEBUG:boto:auth_path=/vclfcn-62/obj4?versionId=null 2024-02-01T10:49:56.790 DEBUG:boto:Method: GET 2024-02-01T10:49:56.790 DEBUG:boto:Path: /vclfcn-62/obj4?versionId=null 2024-02-01T10:49:56.790 DEBUG:boto:Data: 2024-02-01T10:49:56.790 DEBUG:boto:Headers: {} 2024-02-01T10:49:56.790 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:56.790 DEBUG:boto:Port: 8001 2024-02-01T10:49:56.790 DEBUG:boto:Params: {} 2024-02-01T10:49:56.790 DEBUG:boto:Token: None 2024-02-01T10:49:56.790 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:56 GMT /vclfcn-62/obj4?versionId=null 2024-02-01T10:49:56.791 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:KtX85YxqdvkmooaQ7a2r2wg/0Nk= 2024-02-01T10:49:56.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:KtX85YxqdvkmooaQ7a2r2wg/0Nk=', 'Content-Length': '0'} 2024-02-01T10:49:56.793 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006d7030b60b5cef1b-0065bb7754-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:56.840 DEBUG:boto:path=/vclfcn-62/obj4 2024-02-01T10:49:56.840 DEBUG:boto:auth_path=/vclfcn-62/obj4 2024-02-01T10:49:56.840 DEBUG:boto:path=/vclfcn-62/obj4?versionId=null 2024-02-01T10:49:56.840 DEBUG:boto:auth_path=/vclfcn-62/obj4?versionId=null 2024-02-01T10:49:56.841 DEBUG:boto:Method: GET 2024-02-01T10:49:56.841 DEBUG:boto:Path: /vclfcn-62/obj4?versionId=null 2024-02-01T10:49:56.841 DEBUG:boto:Data: 2024-02-01T10:49:56.841 DEBUG:boto:Headers: {} 2024-02-01T10:49:56.841 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:56.841 DEBUG:boto:Port: 8000 2024-02-01T10:49:56.841 DEBUG:boto:Params: {} 2024-02-01T10:49:56.841 DEBUG:boto:Token: None 2024-02-01T10:49:56.841 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:56 GMT /vclfcn-62/obj4?versionId=null 2024-02-01T10:49:56.842 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:KtX85YxqdvkmooaQ7a2r2wg/0Nk= 2024-02-01T10:49:56.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:KtX85YxqdvkmooaQ7a2r2wg/0Nk=', 'Content-Length': '0'} 2024-02-01T10:49:56.844 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000936edbfe58a135d6-0065bb7754-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:56.845 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:56.845 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:56.845 DEBUG:boto:path=/vclfcn-62/?prefix=obj4 2024-02-01T10:49:56.845 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj4 2024-02-01T10:49:56.845 DEBUG:boto:Method: GET 2024-02-01T10:49:56.846 DEBUG:boto:Path: /vclfcn-62/?prefix=obj4 2024-02-01T10:49:56.846 DEBUG:boto:Data: 2024-02-01T10:49:56.846 DEBUG:boto:Headers: {} 2024-02-01T10:49:56.846 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:56.846 DEBUG:boto:Port: 8001 2024-02-01T10:49:56.847 DEBUG:boto:Params: {} 2024-02-01T10:49:56.847 DEBUG:boto:Token: None 2024-02-01T10:49:56.847 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:56 GMT /vclfcn-62/ 2024-02-01T10:49:56.847 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:rhqfdyVzV8k/sJU5MGBgCWI9J2U= 2024-02-01T10:49:56.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:rhqfdyVzV8k/sJU5MGBgCWI9J2U=', 'Content-Length': '0'} 2024-02-01T10:49:56.852 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071020369b255b527-0065bb7754-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:56.852 DEBUG:boto:b'vclfcn-62obj41000falseobj42024-02-01T10:48:26.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:56.853 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:56.853 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:56.853 DEBUG:boto:path=/vclfcn-62/?prefix=obj4 2024-02-01T10:49:56.853 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj4 2024-02-01T10:49:56.853 DEBUG:boto:Method: GET 2024-02-01T10:49:56.853 DEBUG:boto:Path: /vclfcn-62/?prefix=obj4 2024-02-01T10:49:56.853 DEBUG:boto:Data: 2024-02-01T10:49:56.853 DEBUG:boto:Headers: {} 2024-02-01T10:49:56.854 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:56.854 DEBUG:boto:Port: 8000 2024-02-01T10:49:56.854 DEBUG:boto:Params: {} 2024-02-01T10:49:56.854 DEBUG:boto:Token: None 2024-02-01T10:49:56.854 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:56 GMT /vclfcn-62/ 2024-02-01T10:49:56.855 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:rhqfdyVzV8k/sJU5MGBgCWI9J2U= 2024-02-01T10:49:56.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:rhqfdyVzV8k/sJU5MGBgCWI9J2U=', 'Content-Length': '0'} 2024-02-01T10:49:56.859 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000341b4f13095c4d3c-0065bb7754-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:56.860 DEBUG:boto:b'vclfcn-62obj41000falseobj42024-02-01T10:48:26.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:56.860 DEBUG:boto:path=/vclfcn-62/obj4 2024-02-01T10:49:56.860 DEBUG:boto:auth_path=/vclfcn-62/obj4 2024-02-01T10:49:56.861 DEBUG:boto:Method: HEAD 2024-02-01T10:49:56.861 DEBUG:boto:Path: /vclfcn-62/obj4 2024-02-01T10:49:56.861 DEBUG:boto:Data: 2024-02-01T10:49:56.861 DEBUG:boto:Headers: {} 2024-02-01T10:49:56.861 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:56.862 DEBUG:boto:Port: 8001 2024-02-01T10:49:56.862 DEBUG:boto:Params: {} 2024-02-01T10:49:56.862 DEBUG:boto:Token: None 2024-02-01T10:49:56.862 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:56 GMT /vclfcn-62/obj4 2024-02-01T10:49:56.863 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nzoFzu2j0ZPL6QCXcchlP4Sw/yU= 2024-02-01T10:49:56.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nzoFzu2j0ZPL6QCXcchlP4Sw/yU=', 'Content-Length': '0'} 2024-02-01T10:49:56.866 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003044a5a8149d8f5e-0065bb7754-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:56.866 DEBUG:boto:path=/vclfcn-62/obj4 2024-02-01T10:49:56.866 DEBUG:boto:auth_path=/vclfcn-62/obj4 2024-02-01T10:49:56.866 DEBUG:boto:Method: HEAD 2024-02-01T10:49:56.866 DEBUG:boto:Path: /vclfcn-62/obj4 2024-02-01T10:49:56.867 DEBUG:boto:Data: 2024-02-01T10:49:56.867 DEBUG:boto:Headers: {} 2024-02-01T10:49:56.867 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:56.867 DEBUG:boto:Port: 8000 2024-02-01T10:49:56.867 DEBUG:boto:Params: {} 2024-02-01T10:49:56.867 DEBUG:boto:Token: None 2024-02-01T10:49:56.867 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:56 GMT /vclfcn-62/obj4 2024-02-01T10:49:56.867 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nzoFzu2j0ZPL6QCXcchlP4Sw/yU= 2024-02-01T10:49:56.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nzoFzu2j0ZPL6QCXcchlP4Sw/yU=', 'Content-Length': '0'} 2024-02-01T10:49:56.870 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c2da194fee0257d5-0065bb7754-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:56.870 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:49:56.870 DEBUG:boto:path=/vclfcn-62/obj4 2024-02-01T10:49:56.870 DEBUG:boto:auth_path=/vclfcn-62/obj4 2024-02-01T10:49:56.871 DEBUG:boto:Method: GET 2024-02-01T10:49:56.871 DEBUG:boto:Path: /vclfcn-62/obj4 2024-02-01T10:49:56.871 DEBUG:boto:Data: 2024-02-01T10:49:56.871 DEBUG:boto:Headers: {} 2024-02-01T10:49:56.871 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:56.871 DEBUG:boto:Port: 8001 2024-02-01T10:49:56.871 DEBUG:boto:Params: {} 2024-02-01T10:49:56.871 DEBUG:boto:Token: None 2024-02-01T10:49:56.871 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:56 GMT /vclfcn-62/obj4 2024-02-01T10:49:56.872 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:4SdDPwBp88ap3hXcl7m7BOrDegE= 2024-02-01T10:49:56.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:4SdDPwBp88ap3hXcl7m7BOrDegE=', 'Content-Length': '0'} 2024-02-01T10:49:56.874 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000079b697c71e74bb42-0065bb7754-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:56.916 DEBUG:boto:path=/vclfcn-62/obj4 2024-02-01T10:49:56.916 DEBUG:boto:auth_path=/vclfcn-62/obj4 2024-02-01T10:49:56.916 DEBUG:boto:Method: GET 2024-02-01T10:49:56.917 DEBUG:boto:Path: /vclfcn-62/obj4 2024-02-01T10:49:56.917 DEBUG:boto:Data: 2024-02-01T10:49:56.917 DEBUG:boto:Headers: {} 2024-02-01T10:49:56.917 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:56.917 DEBUG:boto:Port: 8000 2024-02-01T10:49:56.917 DEBUG:boto:Params: {} 2024-02-01T10:49:56.917 DEBUG:boto:Token: None 2024-02-01T10:49:56.917 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:56 GMT /vclfcn-62/obj4 2024-02-01T10:49:56.917 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:4SdDPwBp88ap3hXcl7m7BOrDegE= 2024-02-01T10:49:56.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:4SdDPwBp88ap3hXcl7m7BOrDegE=', 'Content-Length': '0'} 2024-02-01T10:49:56.921 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000078be76cb5dda3d6b-0065bb7754-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:56.921 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:56.921 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:56.921 DEBUG:boto:path=/vclfcn-62/?prefix=obj4 2024-02-01T10:49:56.921 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj4 2024-02-01T10:49:56.921 DEBUG:boto:Method: GET 2024-02-01T10:49:56.922 DEBUG:boto:Path: /vclfcn-62/?prefix=obj4 2024-02-01T10:49:56.922 DEBUG:boto:Data: 2024-02-01T10:49:56.922 DEBUG:boto:Headers: {} 2024-02-01T10:49:56.922 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:56.922 DEBUG:boto:Port: 8001 2024-02-01T10:49:56.922 DEBUG:boto:Params: {} 2024-02-01T10:49:56.922 DEBUG:boto:Token: None 2024-02-01T10:49:56.922 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:56 GMT /vclfcn-62/ 2024-02-01T10:49:56.922 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:rhqfdyVzV8k/sJU5MGBgCWI9J2U= 2024-02-01T10:49:56.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:rhqfdyVzV8k/sJU5MGBgCWI9J2U=', 'Content-Length': '0'} 2024-02-01T10:49:56.927 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd2b37134bb0bbf5-0065bb7754-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:56.968 DEBUG:boto:b'vclfcn-62obj41000falseobj42024-02-01T10:48:26.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:56.968 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:56.969 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:56.969 DEBUG:boto:path=/vclfcn-62/?prefix=obj4 2024-02-01T10:49:56.969 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj4 2024-02-01T10:49:56.969 DEBUG:boto:Method: GET 2024-02-01T10:49:56.969 DEBUG:boto:Path: /vclfcn-62/?prefix=obj4 2024-02-01T10:49:56.969 DEBUG:boto:Data: 2024-02-01T10:49:56.969 DEBUG:boto:Headers: {} 2024-02-01T10:49:56.969 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:56.969 DEBUG:boto:Port: 8000 2024-02-01T10:49:56.969 DEBUG:boto:Params: {} 2024-02-01T10:49:56.969 DEBUG:boto:Token: None 2024-02-01T10:49:56.970 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:56 GMT /vclfcn-62/ 2024-02-01T10:49:56.970 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:rhqfdyVzV8k/sJU5MGBgCWI9J2U= 2024-02-01T10:49:56.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:rhqfdyVzV8k/sJU5MGBgCWI9J2U=', 'Content-Length': '0'} 2024-02-01T10:49:56.975 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c01f44c38943267-0065bb7754-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:56.975 DEBUG:boto:b'vclfcn-62obj41000falseobj42024-02-01T10:48:26.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:56.975 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-02-01T10:49:56.976 DEBUG:boto:path=/vclfcn-62/obj5 2024-02-01T10:49:56.976 DEBUG:boto:auth_path=/vclfcn-62/obj5 2024-02-01T10:49:56.976 DEBUG:boto:path=/vclfcn-62/obj5?versionId=null 2024-02-01T10:49:56.976 DEBUG:boto:auth_path=/vclfcn-62/obj5?versionId=null 2024-02-01T10:49:56.976 DEBUG:boto:Method: GET 2024-02-01T10:49:56.976 DEBUG:boto:Path: /vclfcn-62/obj5?versionId=null 2024-02-01T10:49:56.976 DEBUG:boto:Data: 2024-02-01T10:49:56.976 DEBUG:boto:Headers: {} 2024-02-01T10:49:56.976 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:56.976 DEBUG:boto:Port: 8001 2024-02-01T10:49:56.977 DEBUG:boto:Params: {} 2024-02-01T10:49:56.977 DEBUG:boto:Token: None 2024-02-01T10:49:56.977 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:56 GMT /vclfcn-62/obj5?versionId=null 2024-02-01T10:49:56.977 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:rGZwfONs3cVKxARh6ll+3JCia4c= 2024-02-01T10:49:56.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:56 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:rGZwfONs3cVKxARh6ll+3JCia4c=', 'Content-Length': '0'} 2024-02-01T10:49:56.980 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000076a61fe2ce582290-0065bb7754-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:56 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:57.020 DEBUG:boto:path=/vclfcn-62/obj5 2024-02-01T10:49:57.020 DEBUG:boto:auth_path=/vclfcn-62/obj5 2024-02-01T10:49:57.020 DEBUG:boto:path=/vclfcn-62/obj5?versionId=null 2024-02-01T10:49:57.020 DEBUG:boto:auth_path=/vclfcn-62/obj5?versionId=null 2024-02-01T10:49:57.021 DEBUG:boto:Method: GET 2024-02-01T10:49:57.021 DEBUG:boto:Path: /vclfcn-62/obj5?versionId=null 2024-02-01T10:49:57.021 DEBUG:boto:Data: 2024-02-01T10:49:57.021 DEBUG:boto:Headers: {} 2024-02-01T10:49:57.021 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:57.021 DEBUG:boto:Port: 8000 2024-02-01T10:49:57.021 DEBUG:boto:Params: {} 2024-02-01T10:49:57.021 DEBUG:boto:Token: None 2024-02-01T10:49:57.021 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:57 GMT /vclfcn-62/obj5?versionId=null 2024-02-01T10:49:57.022 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:h6jUlf/fIgxTQMwaDlPThD9C5ug= 2024-02-01T10:49:57.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:h6jUlf/fIgxTQMwaDlPThD9C5ug=', 'Content-Length': '0'} 2024-02-01T10:49:57.025 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000009792cbe3eec7449-0065bb7755-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:57.025 DEBUG:boto:path=/vclfcn-62/obj5 2024-02-01T10:49:57.025 DEBUG:boto:auth_path=/vclfcn-62/obj5 2024-02-01T10:49:57.026 DEBUG:boto:path=/vclfcn-62/obj5?versionId=null 2024-02-01T10:49:57.026 DEBUG:boto:auth_path=/vclfcn-62/obj5?versionId=null 2024-02-01T10:49:57.026 DEBUG:boto:Method: HEAD 2024-02-01T10:49:57.026 DEBUG:boto:Path: /vclfcn-62/obj5?versionId=null 2024-02-01T10:49:57.026 DEBUG:boto:Data: 2024-02-01T10:49:57.026 DEBUG:boto:Headers: {} 2024-02-01T10:49:57.026 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:57.027 DEBUG:boto:Port: 8001 2024-02-01T10:49:57.027 DEBUG:boto:Params: {} 2024-02-01T10:49:57.027 DEBUG:boto:Token: None 2024-02-01T10:49:57.027 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:57 GMT /vclfcn-62/obj5?versionId=null 2024-02-01T10:49:57.027 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:iGhiRf0WDMAFXjg9gO34CwnHwwY= 2024-02-01T10:49:57.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iGhiRf0WDMAFXjg9gO34CwnHwwY=', 'Content-Length': '0'} 2024-02-01T10:49:57.030 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004f78043de2de4c08-0065bb7755-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:57.031 DEBUG:boto:path=/vclfcn-62/obj5 2024-02-01T10:49:57.031 DEBUG:boto:auth_path=/vclfcn-62/obj5 2024-02-01T10:49:57.031 DEBUG:boto:path=/vclfcn-62/obj5?versionId=null 2024-02-01T10:49:57.031 DEBUG:boto:auth_path=/vclfcn-62/obj5?versionId=null 2024-02-01T10:49:57.031 DEBUG:boto:Method: HEAD 2024-02-01T10:49:57.031 DEBUG:boto:Path: /vclfcn-62/obj5?versionId=null 2024-02-01T10:49:57.031 DEBUG:boto:Data: 2024-02-01T10:49:57.031 DEBUG:boto:Headers: {} 2024-02-01T10:49:57.031 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:57.031 DEBUG:boto:Port: 8000 2024-02-01T10:49:57.032 DEBUG:boto:Params: {} 2024-02-01T10:49:57.032 DEBUG:boto:Token: None 2024-02-01T10:49:57.032 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:57 GMT /vclfcn-62/obj5?versionId=null 2024-02-01T10:49:57.032 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:iGhiRf0WDMAFXjg9gO34CwnHwwY= 2024-02-01T10:49:57.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iGhiRf0WDMAFXjg9gO34CwnHwwY=', 'Content-Length': '0'} 2024-02-01T10:49:57.035 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bd573f6d60b02d8e-0065bb7755-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:57.035 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-02-01T10:49:57.035 DEBUG:boto:path=/vclfcn-62/obj5 2024-02-01T10:49:57.035 DEBUG:boto:auth_path=/vclfcn-62/obj5 2024-02-01T10:49:57.035 DEBUG:boto:path=/vclfcn-62/obj5?versionId=null 2024-02-01T10:49:57.035 DEBUG:boto:auth_path=/vclfcn-62/obj5?versionId=null 2024-02-01T10:49:57.035 DEBUG:boto:Method: GET 2024-02-01T10:49:57.036 DEBUG:boto:Path: /vclfcn-62/obj5?versionId=null 2024-02-01T10:49:57.036 DEBUG:boto:Data: 2024-02-01T10:49:57.036 DEBUG:boto:Headers: {} 2024-02-01T10:49:57.036 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:57.036 DEBUG:boto:Port: 8001 2024-02-01T10:49:57.036 DEBUG:boto:Params: {} 2024-02-01T10:49:57.036 DEBUG:boto:Token: None 2024-02-01T10:49:57.036 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:57 GMT /vclfcn-62/obj5?versionId=null 2024-02-01T10:49:57.036 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:h6jUlf/fIgxTQMwaDlPThD9C5ug= 2024-02-01T10:49:57.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:h6jUlf/fIgxTQMwaDlPThD9C5ug=', 'Content-Length': '0'} 2024-02-01T10:49:57.039 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000867616054b6941e8-0065bb7755-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:57.080 DEBUG:boto:path=/vclfcn-62/obj5 2024-02-01T10:49:57.080 DEBUG:boto:auth_path=/vclfcn-62/obj5 2024-02-01T10:49:57.080 DEBUG:boto:path=/vclfcn-62/obj5?versionId=null 2024-02-01T10:49:57.080 DEBUG:boto:auth_path=/vclfcn-62/obj5?versionId=null 2024-02-01T10:49:57.080 DEBUG:boto:Method: GET 2024-02-01T10:49:57.080 DEBUG:boto:Path: /vclfcn-62/obj5?versionId=null 2024-02-01T10:49:57.081 DEBUG:boto:Data: 2024-02-01T10:49:57.081 DEBUG:boto:Headers: {} 2024-02-01T10:49:57.081 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:57.081 DEBUG:boto:Port: 8000 2024-02-01T10:49:57.081 DEBUG:boto:Params: {} 2024-02-01T10:49:57.081 DEBUG:boto:Token: None 2024-02-01T10:49:57.081 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:57 GMT /vclfcn-62/obj5?versionId=null 2024-02-01T10:49:57.081 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:h6jUlf/fIgxTQMwaDlPThD9C5ug= 2024-02-01T10:49:57.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:h6jUlf/fIgxTQMwaDlPThD9C5ug=', 'Content-Length': '0'} 2024-02-01T10:49:57.084 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001e4fd3051c0f30d5-0065bb7755-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:57.084 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:57.085 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:57.085 DEBUG:boto:path=/vclfcn-62/?prefix=obj5 2024-02-01T10:49:57.085 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj5 2024-02-01T10:49:57.085 DEBUG:boto:Method: GET 2024-02-01T10:49:57.085 DEBUG:boto:Path: /vclfcn-62/?prefix=obj5 2024-02-01T10:49:57.085 DEBUG:boto:Data: 2024-02-01T10:49:57.085 DEBUG:boto:Headers: {} 2024-02-01T10:49:57.085 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:57.085 DEBUG:boto:Port: 8001 2024-02-01T10:49:57.086 DEBUG:boto:Params: {} 2024-02-01T10:49:57.086 DEBUG:boto:Token: None 2024-02-01T10:49:57.086 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:57 GMT /vclfcn-62/ 2024-02-01T10:49:57.086 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:DSDA5A9WUbWcAxjJ/UVfwSY/hG8= 2024-02-01T10:49:57.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:DSDA5A9WUbWcAxjJ/UVfwSY/hG8=', 'Content-Length': '0'} 2024-02-01T10:49:57.090 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd334cbeb11d5f98-0065bb7755-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:57.090 DEBUG:boto:b'vclfcn-62obj51000falseobj52024-02-01T10:49:22.123Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:57.091 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:57.091 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:57.091 DEBUG:boto:path=/vclfcn-62/?prefix=obj5 2024-02-01T10:49:57.091 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj5 2024-02-01T10:49:57.091 DEBUG:boto:Method: GET 2024-02-01T10:49:57.091 DEBUG:boto:Path: /vclfcn-62/?prefix=obj5 2024-02-01T10:49:57.091 DEBUG:boto:Data: 2024-02-01T10:49:57.091 DEBUG:boto:Headers: {} 2024-02-01T10:49:57.092 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:57.092 DEBUG:boto:Port: 8000 2024-02-01T10:49:57.092 DEBUG:boto:Params: {} 2024-02-01T10:49:57.092 DEBUG:boto:Token: None 2024-02-01T10:49:57.092 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:57 GMT /vclfcn-62/ 2024-02-01T10:49:57.092 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:DSDA5A9WUbWcAxjJ/UVfwSY/hG8= 2024-02-01T10:49:57.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:DSDA5A9WUbWcAxjJ/UVfwSY/hG8=', 'Content-Length': '0'} 2024-02-01T10:49:57.097 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc3ed750125b3d0d-0065bb7755-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:57.097 DEBUG:boto:b'vclfcn-62obj51000falseobj52024-02-01T10:49:22.123Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:57.097 DEBUG:boto:path=/vclfcn-62/obj5 2024-02-01T10:49:57.097 DEBUG:boto:auth_path=/vclfcn-62/obj5 2024-02-01T10:49:57.098 DEBUG:boto:Method: HEAD 2024-02-01T10:49:57.098 DEBUG:boto:Path: /vclfcn-62/obj5 2024-02-01T10:49:57.098 DEBUG:boto:Data: 2024-02-01T10:49:57.098 DEBUG:boto:Headers: {} 2024-02-01T10:49:57.098 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:57.098 DEBUG:boto:Port: 8001 2024-02-01T10:49:57.098 DEBUG:boto:Params: {} 2024-02-01T10:49:57.098 DEBUG:boto:Token: None 2024-02-01T10:49:57.098 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:57 GMT /vclfcn-62/obj5 2024-02-01T10:49:57.098 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:09EJ2gIYUza59Y/JNPjKpD+LVsw= 2024-02-01T10:49:57.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:09EJ2gIYUza59Y/JNPjKpD+LVsw=', 'Content-Length': '0'} 2024-02-01T10:49:57.101 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000607d3d787aa9d7e6-0065bb7755-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:57.101 DEBUG:boto:path=/vclfcn-62/obj5 2024-02-01T10:49:57.102 DEBUG:boto:auth_path=/vclfcn-62/obj5 2024-02-01T10:49:57.102 DEBUG:boto:Method: HEAD 2024-02-01T10:49:57.102 DEBUG:boto:Path: /vclfcn-62/obj5 2024-02-01T10:49:57.102 DEBUG:boto:Data: 2024-02-01T10:49:57.102 DEBUG:boto:Headers: {} 2024-02-01T10:49:57.102 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:57.102 DEBUG:boto:Port: 8000 2024-02-01T10:49:57.102 DEBUG:boto:Params: {} 2024-02-01T10:49:57.102 DEBUG:boto:Token: None 2024-02-01T10:49:57.102 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:57 GMT /vclfcn-62/obj5 2024-02-01T10:49:57.103 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:09EJ2gIYUza59Y/JNPjKpD+LVsw= 2024-02-01T10:49:57.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:09EJ2gIYUza59Y/JNPjKpD+LVsw=', 'Content-Length': '0'} 2024-02-01T10:49:57.105 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000530a865563c9f0fd-0065bb7755-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:57.106 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-02-01T10:49:57.106 DEBUG:boto:path=/vclfcn-62/obj5 2024-02-01T10:49:57.106 DEBUG:boto:auth_path=/vclfcn-62/obj5 2024-02-01T10:49:57.106 DEBUG:boto:Method: GET 2024-02-01T10:49:57.106 DEBUG:boto:Path: /vclfcn-62/obj5 2024-02-01T10:49:57.106 DEBUG:boto:Data: 2024-02-01T10:49:57.106 DEBUG:boto:Headers: {} 2024-02-01T10:49:57.106 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:57.106 DEBUG:boto:Port: 8001 2024-02-01T10:49:57.106 DEBUG:boto:Params: {} 2024-02-01T10:49:57.107 DEBUG:boto:Token: None 2024-02-01T10:49:57.107 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:57 GMT /vclfcn-62/obj5 2024-02-01T10:49:57.107 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:2K3wQC1zP3h9vR+AWh/oX0jC8Ao= 2024-02-01T10:49:57.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2K3wQC1zP3h9vR+AWh/oX0jC8Ao=', 'Content-Length': '0'} 2024-02-01T10:49:57.109 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000889d58ef82009b67-0065bb7755-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:57.152 DEBUG:boto:path=/vclfcn-62/obj5 2024-02-01T10:49:57.152 DEBUG:boto:auth_path=/vclfcn-62/obj5 2024-02-01T10:49:57.152 DEBUG:boto:Method: GET 2024-02-01T10:49:57.152 DEBUG:boto:Path: /vclfcn-62/obj5 2024-02-01T10:49:57.152 DEBUG:boto:Data: 2024-02-01T10:49:57.153 DEBUG:boto:Headers: {} 2024-02-01T10:49:57.153 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:57.153 DEBUG:boto:Port: 8000 2024-02-01T10:49:57.153 DEBUG:boto:Params: {} 2024-02-01T10:49:57.153 DEBUG:boto:Token: None 2024-02-01T10:49:57.153 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:57 GMT /vclfcn-62/obj5 2024-02-01T10:49:57.153 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:2K3wQC1zP3h9vR+AWh/oX0jC8Ao= 2024-02-01T10:49:57.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2K3wQC1zP3h9vR+AWh/oX0jC8Ao=', 'Content-Length': '0'} 2024-02-01T10:49:57.157 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f106a7e513da808f-0065bb7755-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:57.157 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:57.157 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:57.157 DEBUG:boto:path=/vclfcn-62/?prefix=obj5 2024-02-01T10:49:57.157 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj5 2024-02-01T10:49:57.157 DEBUG:boto:Method: GET 2024-02-01T10:49:57.158 DEBUG:boto:Path: /vclfcn-62/?prefix=obj5 2024-02-01T10:49:57.158 DEBUG:boto:Data: 2024-02-01T10:49:57.158 DEBUG:boto:Headers: {} 2024-02-01T10:49:57.158 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:57.158 DEBUG:boto:Port: 8001 2024-02-01T10:49:57.158 DEBUG:boto:Params: {} 2024-02-01T10:49:57.158 DEBUG:boto:Token: None 2024-02-01T10:49:57.158 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:57 GMT /vclfcn-62/ 2024-02-01T10:49:57.158 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:DSDA5A9WUbWcAxjJ/UVfwSY/hG8= 2024-02-01T10:49:57.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:DSDA5A9WUbWcAxjJ/UVfwSY/hG8=', 'Content-Length': '0'} 2024-02-01T10:49:57.163 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091d58184e900e093-0065bb7755-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:57.163 DEBUG:boto:b'vclfcn-62obj51000falseobj52024-02-01T10:49:22.123Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:57.164 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:57.164 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:57.164 DEBUG:boto:path=/vclfcn-62/?prefix=obj5 2024-02-01T10:49:57.164 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj5 2024-02-01T10:49:57.164 DEBUG:boto:Method: GET 2024-02-01T10:49:57.164 DEBUG:boto:Path: /vclfcn-62/?prefix=obj5 2024-02-01T10:49:57.164 DEBUG:boto:Data: 2024-02-01T10:49:57.164 DEBUG:boto:Headers: {} 2024-02-01T10:49:57.165 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:57.165 DEBUG:boto:Port: 8000 2024-02-01T10:49:57.165 DEBUG:boto:Params: {} 2024-02-01T10:49:57.165 DEBUG:boto:Token: None 2024-02-01T10:49:57.165 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:57 GMT /vclfcn-62/ 2024-02-01T10:49:57.165 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:DSDA5A9WUbWcAxjJ/UVfwSY/hG8= 2024-02-01T10:49:57.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:DSDA5A9WUbWcAxjJ/UVfwSY/hG8=', 'Content-Length': '0'} 2024-02-01T10:49:57.169 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d3c061cca7c1206-0065bb7755-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:57.169 DEBUG:boto:b'vclfcn-62obj51000falseobj52024-02-01T10:49:22.123Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:57.170 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-02-01T10:49:57.170 DEBUG:boto:path=/vclfcn-62/obj6 2024-02-01T10:49:57.170 DEBUG:boto:auth_path=/vclfcn-62/obj6 2024-02-01T10:49:57.170 DEBUG:boto:path=/vclfcn-62/obj6?versionId=null 2024-02-01T10:49:57.170 DEBUG:boto:auth_path=/vclfcn-62/obj6?versionId=null 2024-02-01T10:49:57.170 DEBUG:boto:Method: GET 2024-02-01T10:49:57.170 DEBUG:boto:Path: /vclfcn-62/obj6?versionId=null 2024-02-01T10:49:57.170 DEBUG:boto:Data: 2024-02-01T10:49:57.170 DEBUG:boto:Headers: {} 2024-02-01T10:49:57.171 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:57.171 DEBUG:boto:Port: 8001 2024-02-01T10:49:57.171 DEBUG:boto:Params: {} 2024-02-01T10:49:57.171 DEBUG:boto:Token: None 2024-02-01T10:49:57.171 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:57 GMT /vclfcn-62/obj6?versionId=null 2024-02-01T10:49:57.171 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:j3fT1isJTYhoJ1WZa98L3rA/FQ8= 2024-02-01T10:49:57.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:j3fT1isJTYhoJ1WZa98L3rA/FQ8=', 'Content-Length': '0'} 2024-02-01T10:49:57.175 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007705561d229d7db5-0065bb7755-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:57.176 DEBUG:boto:path=/vclfcn-62/obj6 2024-02-01T10:49:57.176 DEBUG:boto:auth_path=/vclfcn-62/obj6 2024-02-01T10:49:57.176 DEBUG:boto:path=/vclfcn-62/obj6?versionId=null 2024-02-01T10:49:57.176 DEBUG:boto:auth_path=/vclfcn-62/obj6?versionId=null 2024-02-01T10:49:57.176 DEBUG:boto:Method: GET 2024-02-01T10:49:57.176 DEBUG:boto:Path: /vclfcn-62/obj6?versionId=null 2024-02-01T10:49:57.177 DEBUG:boto:Data: 2024-02-01T10:49:57.177 DEBUG:boto:Headers: {} 2024-02-01T10:49:57.177 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:57.177 DEBUG:boto:Port: 8000 2024-02-01T10:49:57.177 DEBUG:boto:Params: {} 2024-02-01T10:49:57.177 DEBUG:boto:Token: None 2024-02-01T10:49:57.177 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:57 GMT /vclfcn-62/obj6?versionId=null 2024-02-01T10:49:57.177 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:j3fT1isJTYhoJ1WZa98L3rA/FQ8= 2024-02-01T10:49:57.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:j3fT1isJTYhoJ1WZa98L3rA/FQ8=', 'Content-Length': '0'} 2024-02-01T10:49:57.180 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008d8c6ba8c5b0b4f0-0065bb7755-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:57.181 DEBUG:boto:path=/vclfcn-62/obj6 2024-02-01T10:49:57.181 DEBUG:boto:auth_path=/vclfcn-62/obj6 2024-02-01T10:49:57.182 DEBUG:boto:path=/vclfcn-62/obj6?versionId=null 2024-02-01T10:49:57.182 DEBUG:boto:auth_path=/vclfcn-62/obj6?versionId=null 2024-02-01T10:49:57.182 DEBUG:boto:Method: HEAD 2024-02-01T10:49:57.182 DEBUG:boto:Path: /vclfcn-62/obj6?versionId=null 2024-02-01T10:49:57.182 DEBUG:boto:Data: 2024-02-01T10:49:57.183 DEBUG:boto:Headers: {} 2024-02-01T10:49:57.183 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:57.183 DEBUG:boto:Port: 8001 2024-02-01T10:49:57.183 DEBUG:boto:Params: {} 2024-02-01T10:49:57.184 DEBUG:boto:Token: None 2024-02-01T10:49:57.184 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:57 GMT /vclfcn-62/obj6?versionId=null 2024-02-01T10:49:57.184 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:UaKWt+eFcwMSyuEwxpLVAqIBG0o= 2024-02-01T10:49:57.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:UaKWt+eFcwMSyuEwxpLVAqIBG0o=', 'Content-Length': '0'} 2024-02-01T10:49:57.187 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002fb9419f98224ada-0065bb7755-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:57.187 DEBUG:boto:path=/vclfcn-62/obj6 2024-02-01T10:49:57.187 DEBUG:boto:auth_path=/vclfcn-62/obj6 2024-02-01T10:49:57.187 DEBUG:boto:path=/vclfcn-62/obj6?versionId=null 2024-02-01T10:49:57.187 DEBUG:boto:auth_path=/vclfcn-62/obj6?versionId=null 2024-02-01T10:49:57.188 DEBUG:boto:Method: HEAD 2024-02-01T10:49:57.188 DEBUG:boto:Path: /vclfcn-62/obj6?versionId=null 2024-02-01T10:49:57.188 DEBUG:boto:Data: 2024-02-01T10:49:57.188 DEBUG:boto:Headers: {} 2024-02-01T10:49:57.188 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:57.188 DEBUG:boto:Port: 8000 2024-02-01T10:49:57.188 DEBUG:boto:Params: {} 2024-02-01T10:49:57.189 DEBUG:boto:Token: None 2024-02-01T10:49:57.189 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:57 GMT /vclfcn-62/obj6?versionId=null 2024-02-01T10:49:57.189 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:UaKWt+eFcwMSyuEwxpLVAqIBG0o= 2024-02-01T10:49:57.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:UaKWt+eFcwMSyuEwxpLVAqIBG0o=', 'Content-Length': '0'} 2024-02-01T10:49:57.193 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000e7c7b77e32a6932-0065bb7755-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:57.194 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-02-01T10:49:57.194 DEBUG:boto:path=/vclfcn-62/obj6 2024-02-01T10:49:57.194 DEBUG:boto:auth_path=/vclfcn-62/obj6 2024-02-01T10:49:57.194 DEBUG:boto:path=/vclfcn-62/obj6?versionId=null 2024-02-01T10:49:57.194 DEBUG:boto:auth_path=/vclfcn-62/obj6?versionId=null 2024-02-01T10:49:57.195 DEBUG:boto:Method: GET 2024-02-01T10:49:57.195 DEBUG:boto:Path: /vclfcn-62/obj6?versionId=null 2024-02-01T10:49:57.195 DEBUG:boto:Data: 2024-02-01T10:49:57.195 DEBUG:boto:Headers: {} 2024-02-01T10:49:57.195 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:57.195 DEBUG:boto:Port: 8001 2024-02-01T10:49:57.195 DEBUG:boto:Params: {} 2024-02-01T10:49:57.195 DEBUG:boto:Token: None 2024-02-01T10:49:57.196 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:57 GMT /vclfcn-62/obj6?versionId=null 2024-02-01T10:49:57.196 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:j3fT1isJTYhoJ1WZa98L3rA/FQ8= 2024-02-01T10:49:57.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:j3fT1isJTYhoJ1WZa98L3rA/FQ8=', 'Content-Length': '0'} 2024-02-01T10:49:57.199 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f72335d274bc15fb-0065bb7755-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:57.240 DEBUG:boto:path=/vclfcn-62/obj6 2024-02-01T10:49:57.240 DEBUG:boto:auth_path=/vclfcn-62/obj6 2024-02-01T10:49:57.240 DEBUG:boto:path=/vclfcn-62/obj6?versionId=null 2024-02-01T10:49:57.241 DEBUG:boto:auth_path=/vclfcn-62/obj6?versionId=null 2024-02-01T10:49:57.241 DEBUG:boto:Method: GET 2024-02-01T10:49:57.241 DEBUG:boto:Path: /vclfcn-62/obj6?versionId=null 2024-02-01T10:49:57.241 DEBUG:boto:Data: 2024-02-01T10:49:57.241 DEBUG:boto:Headers: {} 2024-02-01T10:49:57.241 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:57.241 DEBUG:boto:Port: 8000 2024-02-01T10:49:57.241 DEBUG:boto:Params: {} 2024-02-01T10:49:57.242 DEBUG:boto:Token: None 2024-02-01T10:49:57.242 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:57 GMT /vclfcn-62/obj6?versionId=null 2024-02-01T10:49:57.242 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:j3fT1isJTYhoJ1WZa98L3rA/FQ8= 2024-02-01T10:49:57.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:j3fT1isJTYhoJ1WZa98L3rA/FQ8=', 'Content-Length': '0'} 2024-02-01T10:49:57.245 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009fd5dda82441bbb3-0065bb7755-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:57.246 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:57.246 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:57.246 DEBUG:boto:path=/vclfcn-62/?prefix=obj6 2024-02-01T10:49:57.246 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj6 2024-02-01T10:49:57.247 DEBUG:boto:Method: GET 2024-02-01T10:49:57.247 DEBUG:boto:Path: /vclfcn-62/?prefix=obj6 2024-02-01T10:49:57.247 DEBUG:boto:Data: 2024-02-01T10:49:57.247 DEBUG:boto:Headers: {} 2024-02-01T10:49:57.247 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:57.248 DEBUG:boto:Port: 8001 2024-02-01T10:49:57.248 DEBUG:boto:Params: {} 2024-02-01T10:49:57.248 DEBUG:boto:Token: None 2024-02-01T10:49:57.248 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:57 GMT /vclfcn-62/ 2024-02-01T10:49:57.248 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:DSDA5A9WUbWcAxjJ/UVfwSY/hG8= 2024-02-01T10:49:57.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:DSDA5A9WUbWcAxjJ/UVfwSY/hG8=', 'Content-Length': '0'} 2024-02-01T10:49:57.253 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000423f46758759c903-0065bb7755-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:57.296 DEBUG:boto:b'vclfcn-62obj61000falseobj62024-02-01T10:49:22.135Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:57.297 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:57.297 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:57.297 DEBUG:boto:path=/vclfcn-62/?prefix=obj6 2024-02-01T10:49:57.297 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj6 2024-02-01T10:49:57.297 DEBUG:boto:Method: GET 2024-02-01T10:49:57.297 DEBUG:boto:Path: /vclfcn-62/?prefix=obj6 2024-02-01T10:49:57.297 DEBUG:boto:Data: 2024-02-01T10:49:57.297 DEBUG:boto:Headers: {} 2024-02-01T10:49:57.298 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:57.298 DEBUG:boto:Port: 8000 2024-02-01T10:49:57.298 DEBUG:boto:Params: {} 2024-02-01T10:49:57.298 DEBUG:boto:Token: None 2024-02-01T10:49:57.298 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:57 GMT /vclfcn-62/ 2024-02-01T10:49:57.298 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:DSDA5A9WUbWcAxjJ/UVfwSY/hG8= 2024-02-01T10:49:57.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:DSDA5A9WUbWcAxjJ/UVfwSY/hG8=', 'Content-Length': '0'} 2024-02-01T10:49:57.304 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023d212fcda715a86-0065bb7755-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:57.304 DEBUG:boto:b'vclfcn-62obj61000falseobj62024-02-01T10:49:22.135Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:57.305 DEBUG:boto:path=/vclfcn-62/obj6 2024-02-01T10:49:57.305 DEBUG:boto:auth_path=/vclfcn-62/obj6 2024-02-01T10:49:57.305 DEBUG:boto:Method: HEAD 2024-02-01T10:49:57.305 DEBUG:boto:Path: /vclfcn-62/obj6 2024-02-01T10:49:57.305 DEBUG:boto:Data: 2024-02-01T10:49:57.305 DEBUG:boto:Headers: {} 2024-02-01T10:49:57.305 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:57.306 DEBUG:boto:Port: 8001 2024-02-01T10:49:57.306 DEBUG:boto:Params: {} 2024-02-01T10:49:57.306 DEBUG:boto:Token: None 2024-02-01T10:49:57.306 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:57 GMT /vclfcn-62/obj6 2024-02-01T10:49:57.306 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:qC8tGXyLK5xn8/w7OgFFLIO2vdk= 2024-02-01T10:49:57.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:qC8tGXyLK5xn8/w7OgFFLIO2vdk=', 'Content-Length': '0'} 2024-02-01T10:49:57.309 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000088d9570fc66270e4-0065bb7755-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:57.309 DEBUG:boto:path=/vclfcn-62/obj6 2024-02-01T10:49:57.309 DEBUG:boto:auth_path=/vclfcn-62/obj6 2024-02-01T10:49:57.309 DEBUG:boto:Method: HEAD 2024-02-01T10:49:57.309 DEBUG:boto:Path: /vclfcn-62/obj6 2024-02-01T10:49:57.309 DEBUG:boto:Data: 2024-02-01T10:49:57.309 DEBUG:boto:Headers: {} 2024-02-01T10:49:57.309 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:57.309 DEBUG:boto:Port: 8000 2024-02-01T10:49:57.309 DEBUG:boto:Params: {} 2024-02-01T10:49:57.310 DEBUG:boto:Token: None 2024-02-01T10:49:57.310 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:57 GMT /vclfcn-62/obj6 2024-02-01T10:49:57.310 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:qC8tGXyLK5xn8/w7OgFFLIO2vdk= 2024-02-01T10:49:57.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:qC8tGXyLK5xn8/w7OgFFLIO2vdk=', 'Content-Length': '0'} 2024-02-01T10:49:57.313 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a480a85ee35bbcbd-0065bb7755-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:57.313 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-02-01T10:49:57.314 DEBUG:boto:path=/vclfcn-62/obj6 2024-02-01T10:49:57.314 DEBUG:boto:auth_path=/vclfcn-62/obj6 2024-02-01T10:49:57.314 DEBUG:boto:Method: GET 2024-02-01T10:49:57.314 DEBUG:boto:Path: /vclfcn-62/obj6 2024-02-01T10:49:57.314 DEBUG:boto:Data: 2024-02-01T10:49:57.314 DEBUG:boto:Headers: {} 2024-02-01T10:49:57.314 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:57.314 DEBUG:boto:Port: 8001 2024-02-01T10:49:57.314 DEBUG:boto:Params: {} 2024-02-01T10:49:57.314 DEBUG:boto:Token: None 2024-02-01T10:49:57.315 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:57 GMT /vclfcn-62/obj6 2024-02-01T10:49:57.315 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:MXgrtr32iyG9AjXWFwfJyu8V05I= 2024-02-01T10:49:57.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:MXgrtr32iyG9AjXWFwfJyu8V05I=', 'Content-Length': '0'} 2024-02-01T10:49:57.318 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008c09587942fbd928-0065bb7755-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:57.364 DEBUG:boto:path=/vclfcn-62/obj6 2024-02-01T10:49:57.364 DEBUG:boto:auth_path=/vclfcn-62/obj6 2024-02-01T10:49:57.364 DEBUG:boto:Method: GET 2024-02-01T10:49:57.364 DEBUG:boto:Path: /vclfcn-62/obj6 2024-02-01T10:49:57.364 DEBUG:boto:Data: 2024-02-01T10:49:57.365 DEBUG:boto:Headers: {} 2024-02-01T10:49:57.365 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:57.365 DEBUG:boto:Port: 8000 2024-02-01T10:49:57.365 DEBUG:boto:Params: {} 2024-02-01T10:49:57.365 DEBUG:boto:Token: None 2024-02-01T10:49:57.365 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:57 GMT /vclfcn-62/obj6 2024-02-01T10:49:57.365 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:MXgrtr32iyG9AjXWFwfJyu8V05I= 2024-02-01T10:49:57.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:MXgrtr32iyG9AjXWFwfJyu8V05I=', 'Content-Length': '0'} 2024-02-01T10:49:57.368 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003d78ff8f2fc11055-0065bb7755-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:57.369 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:57.369 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:57.369 DEBUG:boto:path=/vclfcn-62/?prefix=obj6 2024-02-01T10:49:57.369 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj6 2024-02-01T10:49:57.369 DEBUG:boto:Method: GET 2024-02-01T10:49:57.369 DEBUG:boto:Path: /vclfcn-62/?prefix=obj6 2024-02-01T10:49:57.369 DEBUG:boto:Data: 2024-02-01T10:49:57.369 DEBUG:boto:Headers: {} 2024-02-01T10:49:57.369 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:57.369 DEBUG:boto:Port: 8001 2024-02-01T10:49:57.370 DEBUG:boto:Params: {} 2024-02-01T10:49:57.370 DEBUG:boto:Token: None 2024-02-01T10:49:57.370 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:57 GMT /vclfcn-62/ 2024-02-01T10:49:57.370 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:DSDA5A9WUbWcAxjJ/UVfwSY/hG8= 2024-02-01T10:49:57.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:DSDA5A9WUbWcAxjJ/UVfwSY/hG8=', 'Content-Length': '0'} 2024-02-01T10:49:57.379 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc7fc2451d5b2b41-0065bb7755-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:57.416 DEBUG:boto:b'vclfcn-62obj61000falseobj62024-02-01T10:49:22.135Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:57.417 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:57.417 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:57.417 DEBUG:boto:path=/vclfcn-62/?prefix=obj6 2024-02-01T10:49:57.417 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj6 2024-02-01T10:49:57.417 DEBUG:boto:Method: GET 2024-02-01T10:49:57.417 DEBUG:boto:Path: /vclfcn-62/?prefix=obj6 2024-02-01T10:49:57.417 DEBUG:boto:Data: 2024-02-01T10:49:57.417 DEBUG:boto:Headers: {} 2024-02-01T10:49:57.417 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:57.417 DEBUG:boto:Port: 8000 2024-02-01T10:49:57.418 DEBUG:boto:Params: {} 2024-02-01T10:49:57.418 DEBUG:boto:Token: None 2024-02-01T10:49:57.418 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:57 GMT /vclfcn-62/ 2024-02-01T10:49:57.418 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:DSDA5A9WUbWcAxjJ/UVfwSY/hG8= 2024-02-01T10:49:57.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:DSDA5A9WUbWcAxjJ/UVfwSY/hG8=', 'Content-Length': '0'} 2024-02-01T10:49:57.424 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000186d27f8eb0253ae-0065bb7755-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:57.424 DEBUG:boto:b'vclfcn-62obj61000falseobj62024-02-01T10:49:22.135Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:57.424 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-02-01T10:49:57.424 DEBUG:boto:path=/vclfcn-62/obj7 2024-02-01T10:49:57.425 DEBUG:boto:auth_path=/vclfcn-62/obj7 2024-02-01T10:49:57.425 DEBUG:boto:path=/vclfcn-62/obj7?versionId=null 2024-02-01T10:49:57.425 DEBUG:boto:auth_path=/vclfcn-62/obj7?versionId=null 2024-02-01T10:49:57.425 DEBUG:boto:Method: GET 2024-02-01T10:49:57.425 DEBUG:boto:Path: /vclfcn-62/obj7?versionId=null 2024-02-01T10:49:57.425 DEBUG:boto:Data: 2024-02-01T10:49:57.425 DEBUG:boto:Headers: {} 2024-02-01T10:49:57.425 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:57.425 DEBUG:boto:Port: 8001 2024-02-01T10:49:57.425 DEBUG:boto:Params: {} 2024-02-01T10:49:57.426 DEBUG:boto:Token: None 2024-02-01T10:49:57.426 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:57 GMT /vclfcn-62/obj7?versionId=null 2024-02-01T10:49:57.426 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BioP6SgfzFYq8LX14e6obIVEzl8= 2024-02-01T10:49:57.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BioP6SgfzFYq8LX14e6obIVEzl8=', 'Content-Length': '0'} 2024-02-01T10:49:57.429 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000516d6d5a08980422-0065bb7755-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:57.429 DEBUG:boto:path=/vclfcn-62/obj7 2024-02-01T10:49:57.429 DEBUG:boto:auth_path=/vclfcn-62/obj7 2024-02-01T10:49:57.429 DEBUG:boto:path=/vclfcn-62/obj7?versionId=null 2024-02-01T10:49:57.429 DEBUG:boto:auth_path=/vclfcn-62/obj7?versionId=null 2024-02-01T10:49:57.429 DEBUG:boto:Method: GET 2024-02-01T10:49:57.430 DEBUG:boto:Path: /vclfcn-62/obj7?versionId=null 2024-02-01T10:49:57.430 DEBUG:boto:Data: 2024-02-01T10:49:57.430 DEBUG:boto:Headers: {} 2024-02-01T10:49:57.430 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:57.430 DEBUG:boto:Port: 8000 2024-02-01T10:49:57.430 DEBUG:boto:Params: {} 2024-02-01T10:49:57.430 DEBUG:boto:Token: None 2024-02-01T10:49:57.430 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:57 GMT /vclfcn-62/obj7?versionId=null 2024-02-01T10:49:57.430 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BioP6SgfzFYq8LX14e6obIVEzl8= 2024-02-01T10:49:57.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BioP6SgfzFYq8LX14e6obIVEzl8=', 'Content-Length': '0'} 2024-02-01T10:49:57.434 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000014de8912e878af8c-0065bb7755-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:57.435 DEBUG:boto:path=/vclfcn-62/obj7 2024-02-01T10:49:57.435 DEBUG:boto:auth_path=/vclfcn-62/obj7 2024-02-01T10:49:57.435 DEBUG:boto:path=/vclfcn-62/obj7?versionId=null 2024-02-01T10:49:57.435 DEBUG:boto:auth_path=/vclfcn-62/obj7?versionId=null 2024-02-01T10:49:57.435 DEBUG:boto:Method: HEAD 2024-02-01T10:49:57.435 DEBUG:boto:Path: /vclfcn-62/obj7?versionId=null 2024-02-01T10:49:57.435 DEBUG:boto:Data: 2024-02-01T10:49:57.435 DEBUG:boto:Headers: {} 2024-02-01T10:49:57.435 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:57.435 DEBUG:boto:Port: 8001 2024-02-01T10:49:57.436 DEBUG:boto:Params: {} 2024-02-01T10:49:57.436 DEBUG:boto:Token: None 2024-02-01T10:49:57.436 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:57 GMT /vclfcn-62/obj7?versionId=null 2024-02-01T10:49:57.436 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:6bc+Zi/hM8JijBiD56ION5EYAo8= 2024-02-01T10:49:57.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:6bc+Zi/hM8JijBiD56ION5EYAo8=', 'Content-Length': '0'} 2024-02-01T10:49:57.440 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000371ffb794cee9976-0065bb7755-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:57.440 DEBUG:boto:path=/vclfcn-62/obj7 2024-02-01T10:49:57.441 DEBUG:boto:auth_path=/vclfcn-62/obj7 2024-02-01T10:49:57.441 DEBUG:boto:path=/vclfcn-62/obj7?versionId=null 2024-02-01T10:49:57.441 DEBUG:boto:auth_path=/vclfcn-62/obj7?versionId=null 2024-02-01T10:49:57.441 DEBUG:boto:Method: HEAD 2024-02-01T10:49:57.441 DEBUG:boto:Path: /vclfcn-62/obj7?versionId=null 2024-02-01T10:49:57.441 DEBUG:boto:Data: 2024-02-01T10:49:57.442 DEBUG:boto:Headers: {} 2024-02-01T10:49:57.442 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:57.442 DEBUG:boto:Port: 8000 2024-02-01T10:49:57.442 DEBUG:boto:Params: {} 2024-02-01T10:49:57.443 DEBUG:boto:Token: None 2024-02-01T10:49:57.443 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:57 GMT /vclfcn-62/obj7?versionId=null 2024-02-01T10:49:57.443 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:6bc+Zi/hM8JijBiD56ION5EYAo8= 2024-02-01T10:49:57.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:6bc+Zi/hM8JijBiD56ION5EYAo8=', 'Content-Length': '0'} 2024-02-01T10:49:57.446 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000089f1e41527a5061b-0065bb7755-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:57.447 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-02-01T10:49:57.447 DEBUG:boto:path=/vclfcn-62/obj7 2024-02-01T10:49:57.447 DEBUG:boto:auth_path=/vclfcn-62/obj7 2024-02-01T10:49:57.447 DEBUG:boto:path=/vclfcn-62/obj7?versionId=null 2024-02-01T10:49:57.448 DEBUG:boto:auth_path=/vclfcn-62/obj7?versionId=null 2024-02-01T10:49:57.448 DEBUG:boto:Method: GET 2024-02-01T10:49:57.448 DEBUG:boto:Path: /vclfcn-62/obj7?versionId=null 2024-02-01T10:49:57.448 DEBUG:boto:Data: 2024-02-01T10:49:57.448 DEBUG:boto:Headers: {} 2024-02-01T10:49:57.449 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:57.449 DEBUG:boto:Port: 8001 2024-02-01T10:49:57.449 DEBUG:boto:Params: {} 2024-02-01T10:49:57.449 DEBUG:boto:Token: None 2024-02-01T10:49:57.450 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:57 GMT /vclfcn-62/obj7?versionId=null 2024-02-01T10:49:57.450 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BioP6SgfzFYq8LX14e6obIVEzl8= 2024-02-01T10:49:57.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BioP6SgfzFYq8LX14e6obIVEzl8=', 'Content-Length': '0'} 2024-02-01T10:49:57.453 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009fb253feea8a340f-0065bb7755-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:57.496 DEBUG:boto:path=/vclfcn-62/obj7 2024-02-01T10:49:57.496 DEBUG:boto:auth_path=/vclfcn-62/obj7 2024-02-01T10:49:57.496 DEBUG:boto:path=/vclfcn-62/obj7?versionId=null 2024-02-01T10:49:57.497 DEBUG:boto:auth_path=/vclfcn-62/obj7?versionId=null 2024-02-01T10:49:57.497 DEBUG:boto:Method: GET 2024-02-01T10:49:57.497 DEBUG:boto:Path: /vclfcn-62/obj7?versionId=null 2024-02-01T10:49:57.497 DEBUG:boto:Data: 2024-02-01T10:49:57.497 DEBUG:boto:Headers: {} 2024-02-01T10:49:57.497 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:57.497 DEBUG:boto:Port: 8000 2024-02-01T10:49:57.497 DEBUG:boto:Params: {} 2024-02-01T10:49:57.497 DEBUG:boto:Token: None 2024-02-01T10:49:57.498 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:57 GMT /vclfcn-62/obj7?versionId=null 2024-02-01T10:49:57.498 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BioP6SgfzFYq8LX14e6obIVEzl8= 2024-02-01T10:49:57.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BioP6SgfzFYq8LX14e6obIVEzl8=', 'Content-Length': '0'} 2024-02-01T10:49:57.501 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c82288fe861932d7-0065bb7755-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:57.501 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:57.501 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:57.502 DEBUG:boto:path=/vclfcn-62/?prefix=obj7 2024-02-01T10:49:57.502 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj7 2024-02-01T10:49:57.502 DEBUG:boto:Method: GET 2024-02-01T10:49:57.502 DEBUG:boto:Path: /vclfcn-62/?prefix=obj7 2024-02-01T10:49:57.502 DEBUG:boto:Data: 2024-02-01T10:49:57.502 DEBUG:boto:Headers: {} 2024-02-01T10:49:57.502 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:57.502 DEBUG:boto:Port: 8001 2024-02-01T10:49:57.502 DEBUG:boto:Params: {} 2024-02-01T10:49:57.503 DEBUG:boto:Token: None 2024-02-01T10:49:57.503 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:57 GMT /vclfcn-62/ 2024-02-01T10:49:57.503 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:DSDA5A9WUbWcAxjJ/UVfwSY/hG8= 2024-02-01T10:49:57.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:DSDA5A9WUbWcAxjJ/UVfwSY/hG8=', 'Content-Length': '0'} 2024-02-01T10:49:57.507 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007083511bb8784019-0065bb7755-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:57.548 DEBUG:boto:b'vclfcn-62obj71000falseobj72024-02-01T10:49:22.186Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:57.548 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:57.549 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:57.549 DEBUG:boto:path=/vclfcn-62/?prefix=obj7 2024-02-01T10:49:57.549 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj7 2024-02-01T10:49:57.549 DEBUG:boto:Method: GET 2024-02-01T10:49:57.549 DEBUG:boto:Path: /vclfcn-62/?prefix=obj7 2024-02-01T10:49:57.549 DEBUG:boto:Data: 2024-02-01T10:49:57.549 DEBUG:boto:Headers: {} 2024-02-01T10:49:57.549 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:57.549 DEBUG:boto:Port: 8000 2024-02-01T10:49:57.549 DEBUG:boto:Params: {} 2024-02-01T10:49:57.550 DEBUG:boto:Token: None 2024-02-01T10:49:57.550 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:57 GMT /vclfcn-62/ 2024-02-01T10:49:57.550 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:DSDA5A9WUbWcAxjJ/UVfwSY/hG8= 2024-02-01T10:49:57.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:DSDA5A9WUbWcAxjJ/UVfwSY/hG8=', 'Content-Length': '0'} 2024-02-01T10:49:57.554 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed9f325aa6cca388-0065bb7755-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:57.555 DEBUG:boto:b'vclfcn-62obj71000falseobj72024-02-01T10:49:22.186Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:57.555 DEBUG:boto:path=/vclfcn-62/obj7 2024-02-01T10:49:57.555 DEBUG:boto:auth_path=/vclfcn-62/obj7 2024-02-01T10:49:57.555 DEBUG:boto:Method: HEAD 2024-02-01T10:49:57.556 DEBUG:boto:Path: /vclfcn-62/obj7 2024-02-01T10:49:57.556 DEBUG:boto:Data: 2024-02-01T10:49:57.556 DEBUG:boto:Headers: {} 2024-02-01T10:49:57.556 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:57.556 DEBUG:boto:Port: 8001 2024-02-01T10:49:57.556 DEBUG:boto:Params: {} 2024-02-01T10:49:57.556 DEBUG:boto:Token: None 2024-02-01T10:49:57.556 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:57 GMT /vclfcn-62/obj7 2024-02-01T10:49:57.556 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gYV6x1ZJBZgUi7G6leV11QzOook= 2024-02-01T10:49:57.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gYV6x1ZJBZgUi7G6leV11QzOook=', 'Content-Length': '0'} 2024-02-01T10:49:57.559 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e88a8f439f60b4ac-0065bb7755-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:57.560 DEBUG:boto:path=/vclfcn-62/obj7 2024-02-01T10:49:57.560 DEBUG:boto:auth_path=/vclfcn-62/obj7 2024-02-01T10:49:57.560 DEBUG:boto:Method: HEAD 2024-02-01T10:49:57.560 DEBUG:boto:Path: /vclfcn-62/obj7 2024-02-01T10:49:57.560 DEBUG:boto:Data: 2024-02-01T10:49:57.560 DEBUG:boto:Headers: {} 2024-02-01T10:49:57.560 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:57.560 DEBUG:boto:Port: 8000 2024-02-01T10:49:57.560 DEBUG:boto:Params: {} 2024-02-01T10:49:57.560 DEBUG:boto:Token: None 2024-02-01T10:49:57.561 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:57 GMT /vclfcn-62/obj7 2024-02-01T10:49:57.561 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gYV6x1ZJBZgUi7G6leV11QzOook= 2024-02-01T10:49:57.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gYV6x1ZJBZgUi7G6leV11QzOook=', 'Content-Length': '0'} 2024-02-01T10:49:57.563 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b5381e011995f65c-0065bb7755-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:57.564 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-02-01T10:49:57.564 DEBUG:boto:path=/vclfcn-62/obj7 2024-02-01T10:49:57.564 DEBUG:boto:auth_path=/vclfcn-62/obj7 2024-02-01T10:49:57.564 DEBUG:boto:Method: GET 2024-02-01T10:49:57.564 DEBUG:boto:Path: /vclfcn-62/obj7 2024-02-01T10:49:57.564 DEBUG:boto:Data: 2024-02-01T10:49:57.564 DEBUG:boto:Headers: {} 2024-02-01T10:49:57.564 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:57.565 DEBUG:boto:Port: 8001 2024-02-01T10:49:57.565 DEBUG:boto:Params: {} 2024-02-01T10:49:57.565 DEBUG:boto:Token: None 2024-02-01T10:49:57.565 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:57 GMT /vclfcn-62/obj7 2024-02-01T10:49:57.565 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fgT1SJBnZW0lhZqNX7MaBQ/KRgs= 2024-02-01T10:49:57.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fgT1SJBnZW0lhZqNX7MaBQ/KRgs=', 'Content-Length': '0'} 2024-02-01T10:49:57.568 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009bd5b5b267c3487c-0065bb7755-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:57.608 DEBUG:boto:path=/vclfcn-62/obj7 2024-02-01T10:49:57.608 DEBUG:boto:auth_path=/vclfcn-62/obj7 2024-02-01T10:49:57.609 DEBUG:boto:Method: GET 2024-02-01T10:49:57.609 DEBUG:boto:Path: /vclfcn-62/obj7 2024-02-01T10:49:57.609 DEBUG:boto:Data: 2024-02-01T10:49:57.609 DEBUG:boto:Headers: {} 2024-02-01T10:49:57.609 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:57.609 DEBUG:boto:Port: 8000 2024-02-01T10:49:57.610 DEBUG:boto:Params: {} 2024-02-01T10:49:57.610 DEBUG:boto:Token: None 2024-02-01T10:49:57.610 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:57 GMT /vclfcn-62/obj7 2024-02-01T10:49:57.610 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fgT1SJBnZW0lhZqNX7MaBQ/KRgs= 2024-02-01T10:49:57.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fgT1SJBnZW0lhZqNX7MaBQ/KRgs=', 'Content-Length': '0'} 2024-02-01T10:49:57.613 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ba1a544e44c35646-0065bb7755-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:57.613 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:57.614 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:57.614 DEBUG:boto:path=/vclfcn-62/?prefix=obj7 2024-02-01T10:49:57.614 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj7 2024-02-01T10:49:57.614 DEBUG:boto:Method: GET 2024-02-01T10:49:57.614 DEBUG:boto:Path: /vclfcn-62/?prefix=obj7 2024-02-01T10:49:57.614 DEBUG:boto:Data: 2024-02-01T10:49:57.614 DEBUG:boto:Headers: {} 2024-02-01T10:49:57.615 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:57.615 DEBUG:boto:Port: 8001 2024-02-01T10:49:57.615 DEBUG:boto:Params: {} 2024-02-01T10:49:57.615 DEBUG:boto:Token: None 2024-02-01T10:49:57.615 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:57 GMT /vclfcn-62/ 2024-02-01T10:49:57.615 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:DSDA5A9WUbWcAxjJ/UVfwSY/hG8= 2024-02-01T10:49:57.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:DSDA5A9WUbWcAxjJ/UVfwSY/hG8=', 'Content-Length': '0'} 2024-02-01T10:49:57.619 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005fa78f2664b17d57-0065bb7755-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:57.620 DEBUG:boto:b'vclfcn-62obj71000falseobj72024-02-01T10:49:22.186Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:57.620 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:57.620 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:57.620 DEBUG:boto:path=/vclfcn-62/?prefix=obj7 2024-02-01T10:49:57.620 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj7 2024-02-01T10:49:57.621 DEBUG:boto:Method: GET 2024-02-01T10:49:57.621 DEBUG:boto:Path: /vclfcn-62/?prefix=obj7 2024-02-01T10:49:57.621 DEBUG:boto:Data: 2024-02-01T10:49:57.621 DEBUG:boto:Headers: {} 2024-02-01T10:49:57.621 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:57.621 DEBUG:boto:Port: 8000 2024-02-01T10:49:57.621 DEBUG:boto:Params: {} 2024-02-01T10:49:57.621 DEBUG:boto:Token: None 2024-02-01T10:49:57.621 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:57 GMT /vclfcn-62/ 2024-02-01T10:49:57.621 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:DSDA5A9WUbWcAxjJ/UVfwSY/hG8= 2024-02-01T10:49:57.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:DSDA5A9WUbWcAxjJ/UVfwSY/hG8=', 'Content-Length': '0'} 2024-02-01T10:49:57.625 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009eae77f2a5d2738d-0065bb7755-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:57.626 DEBUG:boto:b'vclfcn-62obj71000falseobj72024-02-01T10:49:22.186Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:57.626 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-02-01T10:49:57.626 DEBUG:boto:path=/vclfcn-62/obj8 2024-02-01T10:49:57.626 DEBUG:boto:auth_path=/vclfcn-62/obj8 2024-02-01T10:49:57.627 DEBUG:boto:path=/vclfcn-62/obj8?versionId=null 2024-02-01T10:49:57.627 DEBUG:boto:auth_path=/vclfcn-62/obj8?versionId=null 2024-02-01T10:49:57.627 DEBUG:boto:Method: GET 2024-02-01T10:49:57.627 DEBUG:boto:Path: /vclfcn-62/obj8?versionId=null 2024-02-01T10:49:57.627 DEBUG:boto:Data: 2024-02-01T10:49:57.627 DEBUG:boto:Headers: {} 2024-02-01T10:49:57.627 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:57.627 DEBUG:boto:Port: 8001 2024-02-01T10:49:57.627 DEBUG:boto:Params: {} 2024-02-01T10:49:57.627 DEBUG:boto:Token: None 2024-02-01T10:49:57.628 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:57 GMT /vclfcn-62/obj8?versionId=null 2024-02-01T10:49:57.628 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vWoVRjVLUu/eFdoI6UQ3dtYNJU4= 2024-02-01T10:49:57.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vWoVRjVLUu/eFdoI6UQ3dtYNJU4=', 'Content-Length': '0'} 2024-02-01T10:49:57.631 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000624cd9d0bdae54d7-0065bb7755-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:57.676 DEBUG:boto:path=/vclfcn-62/obj8 2024-02-01T10:49:57.677 DEBUG:boto:auth_path=/vclfcn-62/obj8 2024-02-01T10:49:57.677 DEBUG:boto:path=/vclfcn-62/obj8?versionId=null 2024-02-01T10:49:57.677 DEBUG:boto:auth_path=/vclfcn-62/obj8?versionId=null 2024-02-01T10:49:57.677 DEBUG:boto:Method: GET 2024-02-01T10:49:57.678 DEBUG:boto:Path: /vclfcn-62/obj8?versionId=null 2024-02-01T10:49:57.678 DEBUG:boto:Data: 2024-02-01T10:49:57.678 DEBUG:boto:Headers: {} 2024-02-01T10:49:57.678 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:57.678 DEBUG:boto:Port: 8000 2024-02-01T10:49:57.678 DEBUG:boto:Params: {} 2024-02-01T10:49:57.678 DEBUG:boto:Token: None 2024-02-01T10:49:57.678 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:57 GMT /vclfcn-62/obj8?versionId=null 2024-02-01T10:49:57.678 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vWoVRjVLUu/eFdoI6UQ3dtYNJU4= 2024-02-01T10:49:57.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vWoVRjVLUu/eFdoI6UQ3dtYNJU4=', 'Content-Length': '0'} 2024-02-01T10:49:57.682 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000938eccbe3a0f0d77-0065bb7755-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:57.682 DEBUG:boto:path=/vclfcn-62/obj8 2024-02-01T10:49:57.682 DEBUG:boto:auth_path=/vclfcn-62/obj8 2024-02-01T10:49:57.683 DEBUG:boto:path=/vclfcn-62/obj8?versionId=null 2024-02-01T10:49:57.683 DEBUG:boto:auth_path=/vclfcn-62/obj8?versionId=null 2024-02-01T10:49:57.683 DEBUG:boto:Method: HEAD 2024-02-01T10:49:57.683 DEBUG:boto:Path: /vclfcn-62/obj8?versionId=null 2024-02-01T10:49:57.683 DEBUG:boto:Data: 2024-02-01T10:49:57.683 DEBUG:boto:Headers: {} 2024-02-01T10:49:57.683 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:57.683 DEBUG:boto:Port: 8001 2024-02-01T10:49:57.683 DEBUG:boto:Params: {} 2024-02-01T10:49:57.683 DEBUG:boto:Token: None 2024-02-01T10:49:57.683 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:57 GMT /vclfcn-62/obj8?versionId=null 2024-02-01T10:49:57.684 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:2MZZUM4VFLqprRksuMziaqpjW8A= 2024-02-01T10:49:57.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2MZZUM4VFLqprRksuMziaqpjW8A=', 'Content-Length': '0'} 2024-02-01T10:49:57.686 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001d9ba836b61324eb-0065bb7755-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:57.687 DEBUG:boto:path=/vclfcn-62/obj8 2024-02-01T10:49:57.687 DEBUG:boto:auth_path=/vclfcn-62/obj8 2024-02-01T10:49:57.687 DEBUG:boto:path=/vclfcn-62/obj8?versionId=null 2024-02-01T10:49:57.687 DEBUG:boto:auth_path=/vclfcn-62/obj8?versionId=null 2024-02-01T10:49:57.687 DEBUG:boto:Method: HEAD 2024-02-01T10:49:57.687 DEBUG:boto:Path: /vclfcn-62/obj8?versionId=null 2024-02-01T10:49:57.687 DEBUG:boto:Data: 2024-02-01T10:49:57.688 DEBUG:boto:Headers: {} 2024-02-01T10:49:57.688 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:57.688 DEBUG:boto:Port: 8000 2024-02-01T10:49:57.688 DEBUG:boto:Params: {} 2024-02-01T10:49:57.688 DEBUG:boto:Token: None 2024-02-01T10:49:57.688 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:57 GMT /vclfcn-62/obj8?versionId=null 2024-02-01T10:49:57.688 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:2MZZUM4VFLqprRksuMziaqpjW8A= 2024-02-01T10:49:57.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2MZZUM4VFLqprRksuMziaqpjW8A=', 'Content-Length': '0'} 2024-02-01T10:49:57.691 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e575c03295672399-0065bb7755-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:57.692 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-02-01T10:49:57.692 DEBUG:boto:path=/vclfcn-62/obj8 2024-02-01T10:49:57.692 DEBUG:boto:auth_path=/vclfcn-62/obj8 2024-02-01T10:49:57.692 DEBUG:boto:path=/vclfcn-62/obj8?versionId=null 2024-02-01T10:49:57.692 DEBUG:boto:auth_path=/vclfcn-62/obj8?versionId=null 2024-02-01T10:49:57.692 DEBUG:boto:Method: GET 2024-02-01T10:49:57.692 DEBUG:boto:Path: /vclfcn-62/obj8?versionId=null 2024-02-01T10:49:57.693 DEBUG:boto:Data: 2024-02-01T10:49:57.693 DEBUG:boto:Headers: {} 2024-02-01T10:49:57.693 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:57.693 DEBUG:boto:Port: 8001 2024-02-01T10:49:57.693 DEBUG:boto:Params: {} 2024-02-01T10:49:57.693 DEBUG:boto:Token: None 2024-02-01T10:49:57.693 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:57 GMT /vclfcn-62/obj8?versionId=null 2024-02-01T10:49:57.693 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vWoVRjVLUu/eFdoI6UQ3dtYNJU4= 2024-02-01T10:49:57.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vWoVRjVLUu/eFdoI6UQ3dtYNJU4=', 'Content-Length': '0'} 2024-02-01T10:49:57.696 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007b4d5c30ba8b7fe2-0065bb7755-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:57.744 DEBUG:boto:path=/vclfcn-62/obj8 2024-02-01T10:49:57.744 DEBUG:boto:auth_path=/vclfcn-62/obj8 2024-02-01T10:49:57.744 DEBUG:boto:path=/vclfcn-62/obj8?versionId=null 2024-02-01T10:49:57.744 DEBUG:boto:auth_path=/vclfcn-62/obj8?versionId=null 2024-02-01T10:49:57.745 DEBUG:boto:Method: GET 2024-02-01T10:49:57.745 DEBUG:boto:Path: /vclfcn-62/obj8?versionId=null 2024-02-01T10:49:57.745 DEBUG:boto:Data: 2024-02-01T10:49:57.745 DEBUG:boto:Headers: {} 2024-02-01T10:49:57.745 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:57.745 DEBUG:boto:Port: 8000 2024-02-01T10:49:57.745 DEBUG:boto:Params: {} 2024-02-01T10:49:57.745 DEBUG:boto:Token: None 2024-02-01T10:49:57.745 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:57 GMT /vclfcn-62/obj8?versionId=null 2024-02-01T10:49:57.746 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vWoVRjVLUu/eFdoI6UQ3dtYNJU4= 2024-02-01T10:49:57.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vWoVRjVLUu/eFdoI6UQ3dtYNJU4=', 'Content-Length': '0'} 2024-02-01T10:49:57.749 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e02f6c1a920a8740-0065bb7755-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:57.750 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:57.750 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:57.750 DEBUG:boto:path=/vclfcn-62/?prefix=obj8 2024-02-01T10:49:57.750 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj8 2024-02-01T10:49:57.750 DEBUG:boto:Method: GET 2024-02-01T10:49:57.751 DEBUG:boto:Path: /vclfcn-62/?prefix=obj8 2024-02-01T10:49:57.751 DEBUG:boto:Data: 2024-02-01T10:49:57.751 DEBUG:boto:Headers: {} 2024-02-01T10:49:57.751 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:57.751 DEBUG:boto:Port: 8001 2024-02-01T10:49:57.751 DEBUG:boto:Params: {} 2024-02-01T10:49:57.751 DEBUG:boto:Token: None 2024-02-01T10:49:57.751 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:57 GMT /vclfcn-62/ 2024-02-01T10:49:57.751 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:DSDA5A9WUbWcAxjJ/UVfwSY/hG8= 2024-02-01T10:49:57.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:DSDA5A9WUbWcAxjJ/UVfwSY/hG8=', 'Content-Length': '0'} 2024-02-01T10:49:57.756 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050b6df0ae9007391-0065bb7755-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:57.756 DEBUG:boto:b'vclfcn-62obj81000falseobj82024-02-01T10:49:22.200Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:57.756 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:57.757 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:57.757 DEBUG:boto:path=/vclfcn-62/?prefix=obj8 2024-02-01T10:49:57.757 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj8 2024-02-01T10:49:57.757 DEBUG:boto:Method: GET 2024-02-01T10:49:57.757 DEBUG:boto:Path: /vclfcn-62/?prefix=obj8 2024-02-01T10:49:57.757 DEBUG:boto:Data: 2024-02-01T10:49:57.757 DEBUG:boto:Headers: {} 2024-02-01T10:49:57.757 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:57.757 DEBUG:boto:Port: 8000 2024-02-01T10:49:57.757 DEBUG:boto:Params: {} 2024-02-01T10:49:57.758 DEBUG:boto:Token: None 2024-02-01T10:49:57.758 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:57 GMT /vclfcn-62/ 2024-02-01T10:49:57.758 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:DSDA5A9WUbWcAxjJ/UVfwSY/hG8= 2024-02-01T10:49:57.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:DSDA5A9WUbWcAxjJ/UVfwSY/hG8=', 'Content-Length': '0'} 2024-02-01T10:49:57.762 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de81ff24d727246b-0065bb7755-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:57.762 DEBUG:boto:b'vclfcn-62obj81000falseobj82024-02-01T10:49:22.200Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:57.763 DEBUG:boto:path=/vclfcn-62/obj8 2024-02-01T10:49:57.763 DEBUG:boto:auth_path=/vclfcn-62/obj8 2024-02-01T10:49:57.763 DEBUG:boto:Method: HEAD 2024-02-01T10:49:57.763 DEBUG:boto:Path: /vclfcn-62/obj8 2024-02-01T10:49:57.763 DEBUG:boto:Data: 2024-02-01T10:49:57.763 DEBUG:boto:Headers: {} 2024-02-01T10:49:57.763 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:57.763 DEBUG:boto:Port: 8001 2024-02-01T10:49:57.763 DEBUG:boto:Params: {} 2024-02-01T10:49:57.763 DEBUG:boto:Token: None 2024-02-01T10:49:57.764 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:57 GMT /vclfcn-62/obj8 2024-02-01T10:49:57.764 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HoB7bf2HlgFGdMbiEPhGO8myXRc= 2024-02-01T10:49:57.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HoB7bf2HlgFGdMbiEPhGO8myXRc=', 'Content-Length': '0'} 2024-02-01T10:49:57.766 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b57f7d8b341b742f-0065bb7755-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:57.767 DEBUG:boto:path=/vclfcn-62/obj8 2024-02-01T10:49:57.767 DEBUG:boto:auth_path=/vclfcn-62/obj8 2024-02-01T10:49:57.767 DEBUG:boto:Method: HEAD 2024-02-01T10:49:57.767 DEBUG:boto:Path: /vclfcn-62/obj8 2024-02-01T10:49:57.767 DEBUG:boto:Data: 2024-02-01T10:49:57.767 DEBUG:boto:Headers: {} 2024-02-01T10:49:57.767 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:57.767 DEBUG:boto:Port: 8000 2024-02-01T10:49:57.768 DEBUG:boto:Params: {} 2024-02-01T10:49:57.768 DEBUG:boto:Token: None 2024-02-01T10:49:57.768 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:57 GMT /vclfcn-62/obj8 2024-02-01T10:49:57.768 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HoB7bf2HlgFGdMbiEPhGO8myXRc= 2024-02-01T10:49:57.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HoB7bf2HlgFGdMbiEPhGO8myXRc=', 'Content-Length': '0'} 2024-02-01T10:49:57.771 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000734b7feef64c0697-0065bb7755-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:57.771 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-02-01T10:49:57.771 DEBUG:boto:path=/vclfcn-62/obj8 2024-02-01T10:49:57.771 DEBUG:boto:auth_path=/vclfcn-62/obj8 2024-02-01T10:49:57.772 DEBUG:boto:Method: GET 2024-02-01T10:49:57.772 DEBUG:boto:Path: /vclfcn-62/obj8 2024-02-01T10:49:57.772 DEBUG:boto:Data: 2024-02-01T10:49:57.772 DEBUG:boto:Headers: {} 2024-02-01T10:49:57.772 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:57.772 DEBUG:boto:Port: 8001 2024-02-01T10:49:57.772 DEBUG:boto:Params: {} 2024-02-01T10:49:57.772 DEBUG:boto:Token: None 2024-02-01T10:49:57.772 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:57 GMT /vclfcn-62/obj8 2024-02-01T10:49:57.772 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:KpCfeeNsp8EJ3AFvtXe9FbHuG00= 2024-02-01T10:49:57.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:KpCfeeNsp8EJ3AFvtXe9FbHuG00=', 'Content-Length': '0'} 2024-02-01T10:49:57.775 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c40e6b038cf1131a-0065bb7755-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:57.816 DEBUG:boto:path=/vclfcn-62/obj8 2024-02-01T10:49:57.816 DEBUG:boto:auth_path=/vclfcn-62/obj8 2024-02-01T10:49:57.817 DEBUG:boto:Method: GET 2024-02-01T10:49:57.817 DEBUG:boto:Path: /vclfcn-62/obj8 2024-02-01T10:49:57.817 DEBUG:boto:Data: 2024-02-01T10:49:57.817 DEBUG:boto:Headers: {} 2024-02-01T10:49:57.817 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:57.817 DEBUG:boto:Port: 8000 2024-02-01T10:49:57.817 DEBUG:boto:Params: {} 2024-02-01T10:49:57.818 DEBUG:boto:Token: None 2024-02-01T10:49:57.818 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:57 GMT /vclfcn-62/obj8 2024-02-01T10:49:57.818 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:KpCfeeNsp8EJ3AFvtXe9FbHuG00= 2024-02-01T10:49:57.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:KpCfeeNsp8EJ3AFvtXe9FbHuG00=', 'Content-Length': '0'} 2024-02-01T10:49:57.821 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000086b67e18d349524d-0065bb7755-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:57.821 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:57.821 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:57.822 DEBUG:boto:path=/vclfcn-62/?prefix=obj8 2024-02-01T10:49:57.822 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj8 2024-02-01T10:49:57.822 DEBUG:boto:Method: GET 2024-02-01T10:49:57.822 DEBUG:boto:Path: /vclfcn-62/?prefix=obj8 2024-02-01T10:49:57.822 DEBUG:boto:Data: 2024-02-01T10:49:57.822 DEBUG:boto:Headers: {} 2024-02-01T10:49:57.822 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:57.822 DEBUG:boto:Port: 8001 2024-02-01T10:49:57.822 DEBUG:boto:Params: {} 2024-02-01T10:49:57.822 DEBUG:boto:Token: None 2024-02-01T10:49:57.823 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:57 GMT /vclfcn-62/ 2024-02-01T10:49:57.823 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:DSDA5A9WUbWcAxjJ/UVfwSY/hG8= 2024-02-01T10:49:57.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:DSDA5A9WUbWcAxjJ/UVfwSY/hG8=', 'Content-Length': '0'} 2024-02-01T10:49:57.827 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c943378512da7a60-0065bb7755-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:57.868 DEBUG:boto:b'vclfcn-62obj81000falseobj82024-02-01T10:49:22.200Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:57.869 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:57.869 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:57.869 DEBUG:boto:path=/vclfcn-62/?prefix=obj8 2024-02-01T10:49:57.869 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj8 2024-02-01T10:49:57.869 DEBUG:boto:Method: GET 2024-02-01T10:49:57.869 DEBUG:boto:Path: /vclfcn-62/?prefix=obj8 2024-02-01T10:49:57.869 DEBUG:boto:Data: 2024-02-01T10:49:57.869 DEBUG:boto:Headers: {} 2024-02-01T10:49:57.869 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:49:57.869 DEBUG:boto:Port: 8000 2024-02-01T10:49:57.870 DEBUG:boto:Params: {} 2024-02-01T10:49:57.870 DEBUG:boto:Token: None 2024-02-01T10:49:57.870 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:57 GMT /vclfcn-62/ 2024-02-01T10:49:57.870 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:DSDA5A9WUbWcAxjJ/UVfwSY/hG8= 2024-02-01T10:49:57.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:DSDA5A9WUbWcAxjJ/UVfwSY/hG8=', 'Content-Length': '0'} 2024-02-01T10:49:57.874 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae554ef40c126e8a-0065bb7755-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:57.875 DEBUG:boto:b'vclfcn-62obj81000falseobj82024-02-01T10:49:22.200Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:57.875 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-62 zones={test-zone3, test-zone1} 2024-02-01T10:49:57.875 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-62 zones={test-zone3, test-zone2} 2024-02-01T10:49:57.876 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:57.876 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:57.876 DEBUG:boto:Method: HEAD 2024-02-01T10:49:57.876 DEBUG:boto:Path: /vclfcn-62/ 2024-02-01T10:49:57.876 DEBUG:boto:Data: 2024-02-01T10:49:57.876 DEBUG:boto:Headers: {} 2024-02-01T10:49:57.876 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:57.876 DEBUG:boto:Port: 8001 2024-02-01T10:49:57.876 DEBUG:boto:Params: {} 2024-02-01T10:49:57.877 DEBUG:boto:Token: None 2024-02-01T10:49:57.877 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:57 GMT /vclfcn-62/ 2024-02-01T10:49:57.877 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:MrpQa4EyNRyt1JfWAl/vi3Or/DA= 2024-02-01T10:49:57.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:MrpQa4EyNRyt1JfWAl/vi3Or/DA=', 'Content-Length': '0'} 2024-02-01T10:49:57.881 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000033d675a43a31e785-0065bb7755-5762-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:49:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:57.881 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:57.881 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:57.881 DEBUG:boto:Method: HEAD 2024-02-01T10:49:57.881 DEBUG:boto:Path: /vclfcn-62/ 2024-02-01T10:49:57.881 DEBUG:boto:Data: 2024-02-01T10:49:57.881 DEBUG:boto:Headers: {} 2024-02-01T10:49:57.881 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:57.881 DEBUG:boto:Port: 8000 2024-02-01T10:49:57.882 DEBUG:boto:Params: {} 2024-02-01T10:49:57.882 DEBUG:boto:Token: None 2024-02-01T10:49:57.882 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:57 GMT /vclfcn-62/ 2024-02-01T10:49:57.882 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:MrpQa4EyNRyt1JfWAl/vi3Or/DA= 2024-02-01T10:49:57.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:MrpQa4EyNRyt1JfWAl/vi3Or/DA=', 'Content-Length': '0'} 2024-02-01T10:49:57.886 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000011e5a1e42e38cbe9-0065bb7755-5350-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:49:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:57.887 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:49:57.887 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:57.887 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:57.887 DEBUG:boto:path=/vclfcn-62/?versions&max-keys=999 2024-02-01T10:49:57.887 DEBUG:boto:auth_path=/vclfcn-62/?versions&max-keys=999 2024-02-01T10:49:57.888 DEBUG:boto:Method: GET 2024-02-01T10:49:57.888 DEBUG:boto:Path: /vclfcn-62/?versions&max-keys=999 2024-02-01T10:49:57.888 DEBUG:boto:Data: 2024-02-01T10:49:57.888 DEBUG:boto:Headers: {} 2024-02-01T10:49:57.888 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:57.888 DEBUG:boto:Port: 8001 2024-02-01T10:49:57.888 DEBUG:boto:Params: {} 2024-02-01T10:49:57.888 DEBUG:boto:Token: None 2024-02-01T10:49:57.889 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:57 GMT /vclfcn-62/?versions 2024-02-01T10:49:57.890 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:CN6pa3Bui2BpfrbJN3+ONO4AoMo= 2024-02-01T10:49:57.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:CN6pa3Bui2BpfrbJN3+ONO4AoMo=', 'Content-Length': '0'} 2024-02-01T10:49:57.895 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a4d88d3376dce1aa-0065bb7755-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:57.895 DEBUG:boto:b'vclfcn-62999falseobj1nulltrue2024-02-01T10:48:26.315Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:48:26.333Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:48:26.384Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:48:26.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-02-01T10:49:22.123Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-02-01T10:49:22.135Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-02-01T10:49:22.186Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-02-01T10:49:22.200Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:57.896 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-02-01T10:49:57.896 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-02-01T10:49:57.896 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-02-01T10:49:57.896 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-02-01T10:49:57.896 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-02-01T10:49:57.896 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-02-01T10:49:57.896 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-02-01T10:49:57.896 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-02-01T10:49:57.896 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:49:57.897 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:57.897 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:57.897 DEBUG:boto:path=/vclfcn-62/?versions&max-keys=999 2024-02-01T10:49:57.897 DEBUG:boto:auth_path=/vclfcn-62/?versions&max-keys=999 2024-02-01T10:49:57.897 DEBUG:boto:Method: GET 2024-02-01T10:49:57.897 DEBUG:boto:Path: /vclfcn-62/?versions&max-keys=999 2024-02-01T10:49:57.897 DEBUG:boto:Data: 2024-02-01T10:49:57.897 DEBUG:boto:Headers: {} 2024-02-01T10:49:57.897 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:57.898 DEBUG:boto:Port: 8000 2024-02-01T10:49:57.898 DEBUG:boto:Params: {} 2024-02-01T10:49:57.898 DEBUG:boto:Token: None 2024-02-01T10:49:57.898 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:57 GMT /vclfcn-62/?versions 2024-02-01T10:49:57.898 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:CN6pa3Bui2BpfrbJN3+ONO4AoMo= 2024-02-01T10:49:57.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:CN6pa3Bui2BpfrbJN3+ONO4AoMo=', 'Content-Length': '0'} 2024-02-01T10:49:57.903 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d70c937a191d7c56-0065bb7755-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:57.903 DEBUG:boto:b'vclfcn-62999falseobj1nulltrue2024-02-01T10:48:26.315Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:48:26.333Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:48:26.384Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:48:26.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-02-01T10:49:22.123Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-02-01T10:49:22.135Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-02-01T10:49:22.186Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-02-01T10:49:22.200Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:57.904 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-02-01T10:49:57.904 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-02-01T10:49:57.904 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-02-01T10:49:57.904 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-02-01T10:49:57.904 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-02-01T10:49:57.904 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-02-01T10:49:57.904 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-02-01T10:49:57.904 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-02-01T10:49:57.904 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:57.905 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:57.905 DEBUG:boto:path=/vclfcn-62/?versions&max-keys=999 2024-02-01T10:49:57.905 DEBUG:boto:auth_path=/vclfcn-62/?versions&max-keys=999 2024-02-01T10:49:57.905 DEBUG:boto:Method: GET 2024-02-01T10:49:57.905 DEBUG:boto:Path: /vclfcn-62/?versions&max-keys=999 2024-02-01T10:49:57.905 DEBUG:boto:Data: 2024-02-01T10:49:57.905 DEBUG:boto:Headers: {} 2024-02-01T10:49:57.905 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:57.905 DEBUG:boto:Port: 8001 2024-02-01T10:49:57.905 DEBUG:boto:Params: {} 2024-02-01T10:49:57.906 DEBUG:boto:Token: None 2024-02-01T10:49:57.906 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:57 GMT /vclfcn-62/?versions 2024-02-01T10:49:57.906 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:CN6pa3Bui2BpfrbJN3+ONO4AoMo= 2024-02-01T10:49:57.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:CN6pa3Bui2BpfrbJN3+ONO4AoMo=', 'Content-Length': '0'} 2024-02-01T10:49:57.910 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000094ec80f0125cc88-0065bb7755-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:57.911 DEBUG:boto:b'vclfcn-62999falseobj1nulltrue2024-02-01T10:48:26.315Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:48:26.333Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:48:26.384Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:48:26.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-02-01T10:49:22.123Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-02-01T10:49:22.135Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-02-01T10:49:22.186Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-02-01T10:49:22.200Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:57.912 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:57.912 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:57.912 DEBUG:boto:path=/vclfcn-62/?versions&max-keys=999 2024-02-01T10:49:57.912 DEBUG:boto:auth_path=/vclfcn-62/?versions&max-keys=999 2024-02-01T10:49:57.912 DEBUG:boto:Method: GET 2024-02-01T10:49:57.912 DEBUG:boto:Path: /vclfcn-62/?versions&max-keys=999 2024-02-01T10:49:57.912 DEBUG:boto:Data: 2024-02-01T10:49:57.912 DEBUG:boto:Headers: {} 2024-02-01T10:49:57.912 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:57.912 DEBUG:boto:Port: 8000 2024-02-01T10:49:57.913 DEBUG:boto:Params: {} 2024-02-01T10:49:57.913 DEBUG:boto:Token: None 2024-02-01T10:49:57.913 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:57 GMT /vclfcn-62/?versions 2024-02-01T10:49:57.913 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:CN6pa3Bui2BpfrbJN3+ONO4AoMo= 2024-02-01T10:49:57.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:CN6pa3Bui2BpfrbJN3+ONO4AoMo=', 'Content-Length': '0'} 2024-02-01T10:49:57.918 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e9fde2188e3b805-0065bb7755-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:57.918 DEBUG:boto:b'vclfcn-62999falseobj1nulltrue2024-02-01T10:48:26.315Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:48:26.333Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:48:26.384Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:48:26.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-02-01T10:49:22.123Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-02-01T10:49:22.135Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-02-01T10:49:22.186Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-02-01T10:49:22.200Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:57.919 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:49:57.919 DEBUG:boto:path=/vclfcn-62/obj1 2024-02-01T10:49:57.919 DEBUG:boto:auth_path=/vclfcn-62/obj1 2024-02-01T10:49:57.919 DEBUG:boto:path=/vclfcn-62/obj1?versionId=null 2024-02-01T10:49:57.919 DEBUG:boto:auth_path=/vclfcn-62/obj1?versionId=null 2024-02-01T10:49:57.919 DEBUG:boto:Method: GET 2024-02-01T10:49:57.919 DEBUG:boto:Path: /vclfcn-62/obj1?versionId=null 2024-02-01T10:49:57.919 DEBUG:boto:Data: 2024-02-01T10:49:57.920 DEBUG:boto:Headers: {} 2024-02-01T10:49:57.920 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:57.920 DEBUG:boto:Port: 8001 2024-02-01T10:49:57.920 DEBUG:boto:Params: {} 2024-02-01T10:49:57.920 DEBUG:boto:Token: None 2024-02-01T10:49:57.920 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:57 GMT /vclfcn-62/obj1?versionId=null 2024-02-01T10:49:57.920 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HZPH8tD6LLrB0cemaiItgqye0Fg= 2024-02-01T10:49:57.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HZPH8tD6LLrB0cemaiItgqye0Fg=', 'Content-Length': '0'} 2024-02-01T10:49:57.923 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b18c439299224327-0065bb7755-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:57.964 DEBUG:boto:path=/vclfcn-62/obj1 2024-02-01T10:49:57.964 DEBUG:boto:auth_path=/vclfcn-62/obj1 2024-02-01T10:49:57.965 DEBUG:boto:path=/vclfcn-62/obj1?versionId=null 2024-02-01T10:49:57.965 DEBUG:boto:auth_path=/vclfcn-62/obj1?versionId=null 2024-02-01T10:49:57.965 DEBUG:boto:Method: GET 2024-02-01T10:49:57.965 DEBUG:boto:Path: /vclfcn-62/obj1?versionId=null 2024-02-01T10:49:57.965 DEBUG:boto:Data: 2024-02-01T10:49:57.965 DEBUG:boto:Headers: {} 2024-02-01T10:49:57.965 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:57.965 DEBUG:boto:Port: 8000 2024-02-01T10:49:57.966 DEBUG:boto:Params: {} 2024-02-01T10:49:57.966 DEBUG:boto:Token: None 2024-02-01T10:49:57.966 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:57 GMT /vclfcn-62/obj1?versionId=null 2024-02-01T10:49:57.966 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HZPH8tD6LLrB0cemaiItgqye0Fg= 2024-02-01T10:49:57.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HZPH8tD6LLrB0cemaiItgqye0Fg=', 'Content-Length': '0'} 2024-02-01T10:49:57.969 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000db3427138d15b778-0065bb7755-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:57.970 DEBUG:boto:path=/vclfcn-62/obj1 2024-02-01T10:49:57.970 DEBUG:boto:auth_path=/vclfcn-62/obj1 2024-02-01T10:49:57.970 DEBUG:boto:path=/vclfcn-62/obj1?versionId=null 2024-02-01T10:49:57.970 DEBUG:boto:auth_path=/vclfcn-62/obj1?versionId=null 2024-02-01T10:49:57.970 DEBUG:boto:Method: HEAD 2024-02-01T10:49:57.970 DEBUG:boto:Path: /vclfcn-62/obj1?versionId=null 2024-02-01T10:49:57.970 DEBUG:boto:Data: 2024-02-01T10:49:57.970 DEBUG:boto:Headers: {} 2024-02-01T10:49:57.970 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:57.971 DEBUG:boto:Port: 8001 2024-02-01T10:49:57.971 DEBUG:boto:Params: {} 2024-02-01T10:49:57.971 DEBUG:boto:Token: None 2024-02-01T10:49:57.971 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:57 GMT /vclfcn-62/obj1?versionId=null 2024-02-01T10:49:57.971 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:IWWirlX/BZKVQhioCjksX6LFz4o= 2024-02-01T10:49:57.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:IWWirlX/BZKVQhioCjksX6LFz4o=', 'Content-Length': '0'} 2024-02-01T10:49:57.974 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ec39c4c55867b608-0065bb7755-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:57.975 DEBUG:boto:path=/vclfcn-62/obj1 2024-02-01T10:49:57.975 DEBUG:boto:auth_path=/vclfcn-62/obj1 2024-02-01T10:49:57.975 DEBUG:boto:path=/vclfcn-62/obj1?versionId=null 2024-02-01T10:49:57.975 DEBUG:boto:auth_path=/vclfcn-62/obj1?versionId=null 2024-02-01T10:49:57.975 DEBUG:boto:Method: HEAD 2024-02-01T10:49:57.975 DEBUG:boto:Path: /vclfcn-62/obj1?versionId=null 2024-02-01T10:49:57.975 DEBUG:boto:Data: 2024-02-01T10:49:57.975 DEBUG:boto:Headers: {} 2024-02-01T10:49:57.975 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:57.975 DEBUG:boto:Port: 8000 2024-02-01T10:49:57.976 DEBUG:boto:Params: {} 2024-02-01T10:49:57.976 DEBUG:boto:Token: None 2024-02-01T10:49:57.976 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:57 GMT /vclfcn-62/obj1?versionId=null 2024-02-01T10:49:57.976 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:IWWirlX/BZKVQhioCjksX6LFz4o= 2024-02-01T10:49:57.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:IWWirlX/BZKVQhioCjksX6LFz4o=', 'Content-Length': '0'} 2024-02-01T10:49:57.979 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000007e992cdcfa3e1aa-0065bb7755-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:57.979 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:49:57.979 DEBUG:boto:path=/vclfcn-62/obj1 2024-02-01T10:49:57.979 DEBUG:boto:auth_path=/vclfcn-62/obj1 2024-02-01T10:49:57.980 DEBUG:boto:path=/vclfcn-62/obj1?versionId=null 2024-02-01T10:49:57.980 DEBUG:boto:auth_path=/vclfcn-62/obj1?versionId=null 2024-02-01T10:49:57.980 DEBUG:boto:Method: GET 2024-02-01T10:49:57.980 DEBUG:boto:Path: /vclfcn-62/obj1?versionId=null 2024-02-01T10:49:57.980 DEBUG:boto:Data: 2024-02-01T10:49:57.980 DEBUG:boto:Headers: {} 2024-02-01T10:49:57.980 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:57.980 DEBUG:boto:Port: 8001 2024-02-01T10:49:57.980 DEBUG:boto:Params: {} 2024-02-01T10:49:57.981 DEBUG:boto:Token: None 2024-02-01T10:49:57.981 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:57 GMT /vclfcn-62/obj1?versionId=null 2024-02-01T10:49:57.981 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HZPH8tD6LLrB0cemaiItgqye0Fg= 2024-02-01T10:49:57.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:57 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HZPH8tD6LLrB0cemaiItgqye0Fg=', 'Content-Length': '0'} 2024-02-01T10:49:57.984 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b1b30ce67d5e7197-0065bb7755-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:57 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:58.024 DEBUG:boto:path=/vclfcn-62/obj1 2024-02-01T10:49:58.024 DEBUG:boto:auth_path=/vclfcn-62/obj1 2024-02-01T10:49:58.024 DEBUG:boto:path=/vclfcn-62/obj1?versionId=null 2024-02-01T10:49:58.025 DEBUG:boto:auth_path=/vclfcn-62/obj1?versionId=null 2024-02-01T10:49:58.025 DEBUG:boto:Method: GET 2024-02-01T10:49:58.025 DEBUG:boto:Path: /vclfcn-62/obj1?versionId=null 2024-02-01T10:49:58.025 DEBUG:boto:Data: 2024-02-01T10:49:58.025 DEBUG:boto:Headers: {} 2024-02-01T10:49:58.025 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:58.025 DEBUG:boto:Port: 8000 2024-02-01T10:49:58.025 DEBUG:boto:Params: {} 2024-02-01T10:49:58.025 DEBUG:boto:Token: None 2024-02-01T10:49:58.025 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:58 GMT /vclfcn-62/obj1?versionId=null 2024-02-01T10:49:58.026 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8hAoS5fbSZakfi6p9WwL2Kax+kU= 2024-02-01T10:49:58.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8hAoS5fbSZakfi6p9WwL2Kax+kU=', 'Content-Length': '0'} 2024-02-01T10:49:58.029 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bdfc152769b423fb-0065bb7756-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:58.072 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:58.072 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:58.073 DEBUG:boto:path=/vclfcn-62/?prefix=obj1 2024-02-01T10:49:58.073 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj1 2024-02-01T10:49:58.073 DEBUG:boto:Method: GET 2024-02-01T10:49:58.073 DEBUG:boto:Path: /vclfcn-62/?prefix=obj1 2024-02-01T10:49:58.073 DEBUG:boto:Data: 2024-02-01T10:49:58.073 DEBUG:boto:Headers: {} 2024-02-01T10:49:58.073 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:58.073 DEBUG:boto:Port: 8001 2024-02-01T10:49:58.073 DEBUG:boto:Params: {} 2024-02-01T10:49:58.073 DEBUG:boto:Token: None 2024-02-01T10:49:58.074 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:58 GMT /vclfcn-62/ 2024-02-01T10:49:58.074 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jOxZZx4i6msi/5P4JWwenEFoGbA= 2024-02-01T10:49:58.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jOxZZx4i6msi/5P4JWwenEFoGbA=', 'Content-Length': '0'} 2024-02-01T10:49:58.079 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0b67171f0cde9ff-0065bb7756-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:58.079 DEBUG:boto:b'vclfcn-62obj11000falseobj12024-02-01T10:48:26.315Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:58.079 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:58.079 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:58.080 DEBUG:boto:path=/vclfcn-62/?prefix=obj1 2024-02-01T10:49:58.080 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj1 2024-02-01T10:49:58.080 DEBUG:boto:Method: GET 2024-02-01T10:49:58.080 DEBUG:boto:Path: /vclfcn-62/?prefix=obj1 2024-02-01T10:49:58.080 DEBUG:boto:Data: 2024-02-01T10:49:58.080 DEBUG:boto:Headers: {} 2024-02-01T10:49:58.080 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:58.080 DEBUG:boto:Port: 8000 2024-02-01T10:49:58.080 DEBUG:boto:Params: {} 2024-02-01T10:49:58.080 DEBUG:boto:Token: None 2024-02-01T10:49:58.081 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:58 GMT /vclfcn-62/ 2024-02-01T10:49:58.081 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jOxZZx4i6msi/5P4JWwenEFoGbA= 2024-02-01T10:49:58.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jOxZZx4i6msi/5P4JWwenEFoGbA=', 'Content-Length': '0'} 2024-02-01T10:49:58.085 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004002936d5230acc2-0065bb7756-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:58.086 DEBUG:boto:b'vclfcn-62obj11000falseobj12024-02-01T10:48:26.315Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:58.086 DEBUG:boto:path=/vclfcn-62/obj1 2024-02-01T10:49:58.086 DEBUG:boto:auth_path=/vclfcn-62/obj1 2024-02-01T10:49:58.086 DEBUG:boto:Method: HEAD 2024-02-01T10:49:58.086 DEBUG:boto:Path: /vclfcn-62/obj1 2024-02-01T10:49:58.086 DEBUG:boto:Data: 2024-02-01T10:49:58.086 DEBUG:boto:Headers: {} 2024-02-01T10:49:58.086 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:58.087 DEBUG:boto:Port: 8001 2024-02-01T10:49:58.087 DEBUG:boto:Params: {} 2024-02-01T10:49:58.087 DEBUG:boto:Token: None 2024-02-01T10:49:58.087 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:58 GMT /vclfcn-62/obj1 2024-02-01T10:49:58.087 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:p9b0a4MIm2W+v9zXLWbfTsWIfjU= 2024-02-01T10:49:58.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:p9b0a4MIm2W+v9zXLWbfTsWIfjU=', 'Content-Length': '0'} 2024-02-01T10:49:58.090 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a4b1d9e09de99284-0065bb7756-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:58.090 DEBUG:boto:path=/vclfcn-62/obj1 2024-02-01T10:49:58.090 DEBUG:boto:auth_path=/vclfcn-62/obj1 2024-02-01T10:49:58.091 DEBUG:boto:Method: HEAD 2024-02-01T10:49:58.091 DEBUG:boto:Path: /vclfcn-62/obj1 2024-02-01T10:49:58.091 DEBUG:boto:Data: 2024-02-01T10:49:58.091 DEBUG:boto:Headers: {} 2024-02-01T10:49:58.091 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:58.091 DEBUG:boto:Port: 8000 2024-02-01T10:49:58.091 DEBUG:boto:Params: {} 2024-02-01T10:49:58.091 DEBUG:boto:Token: None 2024-02-01T10:49:58.091 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:58 GMT /vclfcn-62/obj1 2024-02-01T10:49:58.091 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:p9b0a4MIm2W+v9zXLWbfTsWIfjU= 2024-02-01T10:49:58.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:p9b0a4MIm2W+v9zXLWbfTsWIfjU=', 'Content-Length': '0'} 2024-02-01T10:49:58.094 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000017b64f97256de244-0065bb7756-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:58.094 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:49:58.094 DEBUG:boto:path=/vclfcn-62/obj1 2024-02-01T10:49:58.094 DEBUG:boto:auth_path=/vclfcn-62/obj1 2024-02-01T10:49:58.095 DEBUG:boto:Method: GET 2024-02-01T10:49:58.095 DEBUG:boto:Path: /vclfcn-62/obj1 2024-02-01T10:49:58.095 DEBUG:boto:Data: 2024-02-01T10:49:58.095 DEBUG:boto:Headers: {} 2024-02-01T10:49:58.095 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:58.095 DEBUG:boto:Port: 8001 2024-02-01T10:49:58.095 DEBUG:boto:Params: {} 2024-02-01T10:49:58.095 DEBUG:boto:Token: None 2024-02-01T10:49:58.095 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:58 GMT /vclfcn-62/obj1 2024-02-01T10:49:58.095 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:EDPrG8gR+RQ9wKiUkVodYgmQlEc= 2024-02-01T10:49:58.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:EDPrG8gR+RQ9wKiUkVodYgmQlEc=', 'Content-Length': '0'} 2024-02-01T10:49:58.098 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008cb7f3ebcf00a21a-0065bb7756-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:58.140 DEBUG:boto:path=/vclfcn-62/obj1 2024-02-01T10:49:58.140 DEBUG:boto:auth_path=/vclfcn-62/obj1 2024-02-01T10:49:58.140 DEBUG:boto:Method: GET 2024-02-01T10:49:58.140 DEBUG:boto:Path: /vclfcn-62/obj1 2024-02-01T10:49:58.140 DEBUG:boto:Data: 2024-02-01T10:49:58.141 DEBUG:boto:Headers: {} 2024-02-01T10:49:58.141 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:58.141 DEBUG:boto:Port: 8000 2024-02-01T10:49:58.141 DEBUG:boto:Params: {} 2024-02-01T10:49:58.141 DEBUG:boto:Token: None 2024-02-01T10:49:58.141 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:58 GMT /vclfcn-62/obj1 2024-02-01T10:49:58.141 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:EDPrG8gR+RQ9wKiUkVodYgmQlEc= 2024-02-01T10:49:58.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:EDPrG8gR+RQ9wKiUkVodYgmQlEc=', 'Content-Length': '0'} 2024-02-01T10:49:58.144 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002b42a2229a3b3524-0065bb7756-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:58.192 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:58.192 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:58.192 DEBUG:boto:path=/vclfcn-62/?prefix=obj1 2024-02-01T10:49:58.192 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj1 2024-02-01T10:49:58.192 DEBUG:boto:Method: GET 2024-02-01T10:49:58.193 DEBUG:boto:Path: /vclfcn-62/?prefix=obj1 2024-02-01T10:49:58.193 DEBUG:boto:Data: 2024-02-01T10:49:58.193 DEBUG:boto:Headers: {} 2024-02-01T10:49:58.193 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:58.193 DEBUG:boto:Port: 8001 2024-02-01T10:49:58.193 DEBUG:boto:Params: {} 2024-02-01T10:49:58.193 DEBUG:boto:Token: None 2024-02-01T10:49:58.193 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:58 GMT /vclfcn-62/ 2024-02-01T10:49:58.194 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jOxZZx4i6msi/5P4JWwenEFoGbA= 2024-02-01T10:49:58.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jOxZZx4i6msi/5P4JWwenEFoGbA=', 'Content-Length': '0'} 2024-02-01T10:49:58.199 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b277c004dded99e-0065bb7756-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:58.200 DEBUG:boto:b'vclfcn-62obj11000falseobj12024-02-01T10:48:26.315Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:58.200 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:58.200 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:58.201 DEBUG:boto:path=/vclfcn-62/?prefix=obj1 2024-02-01T10:49:58.201 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj1 2024-02-01T10:49:58.201 DEBUG:boto:Method: GET 2024-02-01T10:49:58.201 DEBUG:boto:Path: /vclfcn-62/?prefix=obj1 2024-02-01T10:49:58.201 DEBUG:boto:Data: 2024-02-01T10:49:58.201 DEBUG:boto:Headers: {} 2024-02-01T10:49:58.201 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:58.201 DEBUG:boto:Port: 8000 2024-02-01T10:49:58.201 DEBUG:boto:Params: {} 2024-02-01T10:49:58.201 DEBUG:boto:Token: None 2024-02-01T10:49:58.202 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:58 GMT /vclfcn-62/ 2024-02-01T10:49:58.202 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jOxZZx4i6msi/5P4JWwenEFoGbA= 2024-02-01T10:49:58.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jOxZZx4i6msi/5P4JWwenEFoGbA=', 'Content-Length': '0'} 2024-02-01T10:49:58.207 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007dee0bbd36d326b1-0065bb7756-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:58.207 DEBUG:boto:b'vclfcn-62obj11000falseobj12024-02-01T10:48:26.315Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:58.207 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:49:58.207 DEBUG:boto:path=/vclfcn-62/obj2 2024-02-01T10:49:58.207 DEBUG:boto:auth_path=/vclfcn-62/obj2 2024-02-01T10:49:58.208 DEBUG:boto:path=/vclfcn-62/obj2?versionId=null 2024-02-01T10:49:58.208 DEBUG:boto:auth_path=/vclfcn-62/obj2?versionId=null 2024-02-01T10:49:58.208 DEBUG:boto:Method: GET 2024-02-01T10:49:58.208 DEBUG:boto:Path: /vclfcn-62/obj2?versionId=null 2024-02-01T10:49:58.208 DEBUG:boto:Data: 2024-02-01T10:49:58.208 DEBUG:boto:Headers: {} 2024-02-01T10:49:58.208 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:58.208 DEBUG:boto:Port: 8001 2024-02-01T10:49:58.208 DEBUG:boto:Params: {} 2024-02-01T10:49:58.208 DEBUG:boto:Token: None 2024-02-01T10:49:58.208 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:58 GMT /vclfcn-62/obj2?versionId=null 2024-02-01T10:49:58.209 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:A1WcGVlE8y2JRoJDoclunfxrMd4= 2024-02-01T10:49:58.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:A1WcGVlE8y2JRoJDoclunfxrMd4=', 'Content-Length': '0'} 2024-02-01T10:49:58.211 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dc7d65b9e751086b-0065bb7756-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:58.212 DEBUG:boto:path=/vclfcn-62/obj2 2024-02-01T10:49:58.212 DEBUG:boto:auth_path=/vclfcn-62/obj2 2024-02-01T10:49:58.212 DEBUG:boto:path=/vclfcn-62/obj2?versionId=null 2024-02-01T10:49:58.212 DEBUG:boto:auth_path=/vclfcn-62/obj2?versionId=null 2024-02-01T10:49:58.212 DEBUG:boto:Method: GET 2024-02-01T10:49:58.212 DEBUG:boto:Path: /vclfcn-62/obj2?versionId=null 2024-02-01T10:49:58.212 DEBUG:boto:Data: 2024-02-01T10:49:58.212 DEBUG:boto:Headers: {} 2024-02-01T10:49:58.212 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:58.212 DEBUG:boto:Port: 8000 2024-02-01T10:49:58.212 DEBUG:boto:Params: {} 2024-02-01T10:49:58.213 DEBUG:boto:Token: None 2024-02-01T10:49:58.213 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:58 GMT /vclfcn-62/obj2?versionId=null 2024-02-01T10:49:58.213 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:A1WcGVlE8y2JRoJDoclunfxrMd4= 2024-02-01T10:49:58.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:A1WcGVlE8y2JRoJDoclunfxrMd4=', 'Content-Length': '0'} 2024-02-01T10:49:58.216 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fc8ea87501746603-0065bb7756-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:58.256 DEBUG:boto:path=/vclfcn-62/obj2 2024-02-01T10:49:58.256 DEBUG:boto:auth_path=/vclfcn-62/obj2 2024-02-01T10:49:58.256 DEBUG:boto:path=/vclfcn-62/obj2?versionId=null 2024-02-01T10:49:58.256 DEBUG:boto:auth_path=/vclfcn-62/obj2?versionId=null 2024-02-01T10:49:58.257 DEBUG:boto:Method: HEAD 2024-02-01T10:49:58.257 DEBUG:boto:Path: /vclfcn-62/obj2?versionId=null 2024-02-01T10:49:58.257 DEBUG:boto:Data: 2024-02-01T10:49:58.257 DEBUG:boto:Headers: {} 2024-02-01T10:49:58.257 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:58.257 DEBUG:boto:Port: 8001 2024-02-01T10:49:58.257 DEBUG:boto:Params: {} 2024-02-01T10:49:58.257 DEBUG:boto:Token: None 2024-02-01T10:49:58.257 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:58 GMT /vclfcn-62/obj2?versionId=null 2024-02-01T10:49:58.258 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:b7vv6+jKr6kDW+i8+KzhYmBy3R8= 2024-02-01T10:49:58.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:b7vv6+jKr6kDW+i8+KzhYmBy3R8=', 'Content-Length': '0'} 2024-02-01T10:49:58.260 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a49151a7eecd488e-0065bb7756-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:58.261 DEBUG:boto:path=/vclfcn-62/obj2 2024-02-01T10:49:58.261 DEBUG:boto:auth_path=/vclfcn-62/obj2 2024-02-01T10:49:58.261 DEBUG:boto:path=/vclfcn-62/obj2?versionId=null 2024-02-01T10:49:58.261 DEBUG:boto:auth_path=/vclfcn-62/obj2?versionId=null 2024-02-01T10:49:58.261 DEBUG:boto:Method: HEAD 2024-02-01T10:49:58.261 DEBUG:boto:Path: /vclfcn-62/obj2?versionId=null 2024-02-01T10:49:58.261 DEBUG:boto:Data: 2024-02-01T10:49:58.261 DEBUG:boto:Headers: {} 2024-02-01T10:49:58.261 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:58.261 DEBUG:boto:Port: 8000 2024-02-01T10:49:58.261 DEBUG:boto:Params: {} 2024-02-01T10:49:58.262 DEBUG:boto:Token: None 2024-02-01T10:49:58.262 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:58 GMT /vclfcn-62/obj2?versionId=null 2024-02-01T10:49:58.262 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:b7vv6+jKr6kDW+i8+KzhYmBy3R8= 2024-02-01T10:49:58.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:b7vv6+jKr6kDW+i8+KzhYmBy3R8=', 'Content-Length': '0'} 2024-02-01T10:49:58.264 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d5104dc06e83662e-0065bb7756-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:58.265 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:49:58.265 DEBUG:boto:path=/vclfcn-62/obj2 2024-02-01T10:49:58.265 DEBUG:boto:auth_path=/vclfcn-62/obj2 2024-02-01T10:49:58.265 DEBUG:boto:path=/vclfcn-62/obj2?versionId=null 2024-02-01T10:49:58.265 DEBUG:boto:auth_path=/vclfcn-62/obj2?versionId=null 2024-02-01T10:49:58.265 DEBUG:boto:Method: GET 2024-02-01T10:49:58.265 DEBUG:boto:Path: /vclfcn-62/obj2?versionId=null 2024-02-01T10:49:58.265 DEBUG:boto:Data: 2024-02-01T10:49:58.265 DEBUG:boto:Headers: {} 2024-02-01T10:49:58.266 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:58.266 DEBUG:boto:Port: 8001 2024-02-01T10:49:58.266 DEBUG:boto:Params: {} 2024-02-01T10:49:58.266 DEBUG:boto:Token: None 2024-02-01T10:49:58.266 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:58 GMT /vclfcn-62/obj2?versionId=null 2024-02-01T10:49:58.266 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:A1WcGVlE8y2JRoJDoclunfxrMd4= 2024-02-01T10:49:58.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:A1WcGVlE8y2JRoJDoclunfxrMd4=', 'Content-Length': '0'} 2024-02-01T10:49:58.269 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000007575d07d3dc03d8-0065bb7756-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:58.316 DEBUG:boto:path=/vclfcn-62/obj2 2024-02-01T10:49:58.316 DEBUG:boto:auth_path=/vclfcn-62/obj2 2024-02-01T10:49:58.316 DEBUG:boto:path=/vclfcn-62/obj2?versionId=null 2024-02-01T10:49:58.317 DEBUG:boto:auth_path=/vclfcn-62/obj2?versionId=null 2024-02-01T10:49:58.317 DEBUG:boto:Method: GET 2024-02-01T10:49:58.317 DEBUG:boto:Path: /vclfcn-62/obj2?versionId=null 2024-02-01T10:49:58.317 DEBUG:boto:Data: 2024-02-01T10:49:58.317 DEBUG:boto:Headers: {} 2024-02-01T10:49:58.317 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:58.317 DEBUG:boto:Port: 8000 2024-02-01T10:49:58.317 DEBUG:boto:Params: {} 2024-02-01T10:49:58.318 DEBUG:boto:Token: None 2024-02-01T10:49:58.318 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:58 GMT /vclfcn-62/obj2?versionId=null 2024-02-01T10:49:58.318 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:A1WcGVlE8y2JRoJDoclunfxrMd4= 2024-02-01T10:49:58.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:A1WcGVlE8y2JRoJDoclunfxrMd4=', 'Content-Length': '0'} 2024-02-01T10:49:58.321 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b3d5d6f568d2891e-0065bb7756-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:58.364 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:58.364 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:58.364 DEBUG:boto:path=/vclfcn-62/?prefix=obj2 2024-02-01T10:49:58.364 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj2 2024-02-01T10:49:58.365 DEBUG:boto:Method: GET 2024-02-01T10:49:58.365 DEBUG:boto:Path: /vclfcn-62/?prefix=obj2 2024-02-01T10:49:58.365 DEBUG:boto:Data: 2024-02-01T10:49:58.365 DEBUG:boto:Headers: {} 2024-02-01T10:49:58.365 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:58.365 DEBUG:boto:Port: 8001 2024-02-01T10:49:58.365 DEBUG:boto:Params: {} 2024-02-01T10:49:58.365 DEBUG:boto:Token: None 2024-02-01T10:49:58.365 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:58 GMT /vclfcn-62/ 2024-02-01T10:49:58.366 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jOxZZx4i6msi/5P4JWwenEFoGbA= 2024-02-01T10:49:58.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jOxZZx4i6msi/5P4JWwenEFoGbA=', 'Content-Length': '0'} 2024-02-01T10:49:58.370 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d04e60ba522acb91-0065bb7756-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:58.416 DEBUG:boto:b'vclfcn-62obj21000falseobj22024-02-01T10:48:26.333Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:58.417 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:58.417 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:58.417 DEBUG:boto:path=/vclfcn-62/?prefix=obj2 2024-02-01T10:49:58.417 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj2 2024-02-01T10:49:58.417 DEBUG:boto:Method: GET 2024-02-01T10:49:58.417 DEBUG:boto:Path: /vclfcn-62/?prefix=obj2 2024-02-01T10:49:58.418 DEBUG:boto:Data: 2024-02-01T10:49:58.418 DEBUG:boto:Headers: {} 2024-02-01T10:49:58.418 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:58.418 DEBUG:boto:Port: 8000 2024-02-01T10:49:58.418 DEBUG:boto:Params: {} 2024-02-01T10:49:58.418 DEBUG:boto:Token: None 2024-02-01T10:49:58.418 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:58 GMT /vclfcn-62/ 2024-02-01T10:49:58.418 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jOxZZx4i6msi/5P4JWwenEFoGbA= 2024-02-01T10:49:58.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jOxZZx4i6msi/5P4JWwenEFoGbA=', 'Content-Length': '0'} 2024-02-01T10:49:58.423 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000056a70ab306c41038-0065bb7756-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:58.423 DEBUG:boto:b'vclfcn-62obj21000falseobj22024-02-01T10:48:26.333Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:58.423 DEBUG:boto:path=/vclfcn-62/obj2 2024-02-01T10:49:58.424 DEBUG:boto:auth_path=/vclfcn-62/obj2 2024-02-01T10:49:58.424 DEBUG:boto:Method: HEAD 2024-02-01T10:49:58.424 DEBUG:boto:Path: /vclfcn-62/obj2 2024-02-01T10:49:58.424 DEBUG:boto:Data: 2024-02-01T10:49:58.424 DEBUG:boto:Headers: {} 2024-02-01T10:49:58.424 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:58.424 DEBUG:boto:Port: 8001 2024-02-01T10:49:58.424 DEBUG:boto:Params: {} 2024-02-01T10:49:58.424 DEBUG:boto:Token: None 2024-02-01T10:49:58.425 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:58 GMT /vclfcn-62/obj2 2024-02-01T10:49:58.425 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:do/FQqyK27yD1m5mJ9FAn5hXLc0= 2024-02-01T10:49:58.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:do/FQqyK27yD1m5mJ9FAn5hXLc0=', 'Content-Length': '0'} 2024-02-01T10:49:58.428 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009353fa660f6af625-0065bb7756-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:58.428 DEBUG:boto:path=/vclfcn-62/obj2 2024-02-01T10:49:58.428 DEBUG:boto:auth_path=/vclfcn-62/obj2 2024-02-01T10:49:58.428 DEBUG:boto:Method: HEAD 2024-02-01T10:49:58.429 DEBUG:boto:Path: /vclfcn-62/obj2 2024-02-01T10:49:58.429 DEBUG:boto:Data: 2024-02-01T10:49:58.429 DEBUG:boto:Headers: {} 2024-02-01T10:49:58.429 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:58.429 DEBUG:boto:Port: 8000 2024-02-01T10:49:58.429 DEBUG:boto:Params: {} 2024-02-01T10:49:58.429 DEBUG:boto:Token: None 2024-02-01T10:49:58.430 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:58 GMT /vclfcn-62/obj2 2024-02-01T10:49:58.430 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:do/FQqyK27yD1m5mJ9FAn5hXLc0= 2024-02-01T10:49:58.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:do/FQqyK27yD1m5mJ9FAn5hXLc0=', 'Content-Length': '0'} 2024-02-01T10:49:58.433 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000086a177218aa4e9d2-0065bb7756-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:58.433 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:49:58.433 DEBUG:boto:path=/vclfcn-62/obj2 2024-02-01T10:49:58.433 DEBUG:boto:auth_path=/vclfcn-62/obj2 2024-02-01T10:49:58.433 DEBUG:boto:Method: GET 2024-02-01T10:49:58.433 DEBUG:boto:Path: /vclfcn-62/obj2 2024-02-01T10:49:58.433 DEBUG:boto:Data: 2024-02-01T10:49:58.433 DEBUG:boto:Headers: {} 2024-02-01T10:49:58.434 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:58.434 DEBUG:boto:Port: 8001 2024-02-01T10:49:58.434 DEBUG:boto:Params: {} 2024-02-01T10:49:58.434 DEBUG:boto:Token: None 2024-02-01T10:49:58.434 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:58 GMT /vclfcn-62/obj2 2024-02-01T10:49:58.434 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Mtcq790XdYOHjHV649C44Ep/Q64= 2024-02-01T10:49:58.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Mtcq790XdYOHjHV649C44Ep/Q64=', 'Content-Length': '0'} 2024-02-01T10:49:58.437 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005460b15fdeb8c2e5-0065bb7756-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:58.480 DEBUG:boto:path=/vclfcn-62/obj2 2024-02-01T10:49:58.480 DEBUG:boto:auth_path=/vclfcn-62/obj2 2024-02-01T10:49:58.480 DEBUG:boto:Method: GET 2024-02-01T10:49:58.481 DEBUG:boto:Path: /vclfcn-62/obj2 2024-02-01T10:49:58.481 DEBUG:boto:Data: 2024-02-01T10:49:58.481 DEBUG:boto:Headers: {} 2024-02-01T10:49:58.481 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:58.481 DEBUG:boto:Port: 8000 2024-02-01T10:49:58.481 DEBUG:boto:Params: {} 2024-02-01T10:49:58.481 DEBUG:boto:Token: None 2024-02-01T10:49:58.481 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:58 GMT /vclfcn-62/obj2 2024-02-01T10:49:58.481 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Mtcq790XdYOHjHV649C44Ep/Q64= 2024-02-01T10:49:58.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Mtcq790XdYOHjHV649C44Ep/Q64=', 'Content-Length': '0'} 2024-02-01T10:49:58.484 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000040e2e4abdc9b2539-0065bb7756-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:58.532 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:58.532 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:58.532 DEBUG:boto:path=/vclfcn-62/?prefix=obj2 2024-02-01T10:49:58.532 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj2 2024-02-01T10:49:58.533 DEBUG:boto:Method: GET 2024-02-01T10:49:58.533 DEBUG:boto:Path: /vclfcn-62/?prefix=obj2 2024-02-01T10:49:58.533 DEBUG:boto:Data: 2024-02-01T10:49:58.533 DEBUG:boto:Headers: {} 2024-02-01T10:49:58.533 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:58.533 DEBUG:boto:Port: 8001 2024-02-01T10:49:58.533 DEBUG:boto:Params: {} 2024-02-01T10:49:58.533 DEBUG:boto:Token: None 2024-02-01T10:49:58.533 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:58 GMT /vclfcn-62/ 2024-02-01T10:49:58.534 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jOxZZx4i6msi/5P4JWwenEFoGbA= 2024-02-01T10:49:58.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jOxZZx4i6msi/5P4JWwenEFoGbA=', 'Content-Length': '0'} 2024-02-01T10:49:58.538 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001fbb5f258dda552f-0065bb7756-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:58.584 DEBUG:boto:b'vclfcn-62obj21000falseobj22024-02-01T10:48:26.333Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:58.584 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:58.585 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:58.585 DEBUG:boto:path=/vclfcn-62/?prefix=obj2 2024-02-01T10:49:58.585 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj2 2024-02-01T10:49:58.585 DEBUG:boto:Method: GET 2024-02-01T10:49:58.585 DEBUG:boto:Path: /vclfcn-62/?prefix=obj2 2024-02-01T10:49:58.585 DEBUG:boto:Data: 2024-02-01T10:49:58.585 DEBUG:boto:Headers: {} 2024-02-01T10:49:58.586 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:58.586 DEBUG:boto:Port: 8000 2024-02-01T10:49:58.586 DEBUG:boto:Params: {} 2024-02-01T10:49:58.586 DEBUG:boto:Token: None 2024-02-01T10:49:58.586 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:58 GMT /vclfcn-62/ 2024-02-01T10:49:58.586 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jOxZZx4i6msi/5P4JWwenEFoGbA= 2024-02-01T10:49:58.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jOxZZx4i6msi/5P4JWwenEFoGbA=', 'Content-Length': '0'} 2024-02-01T10:49:58.591 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a20434a8fc4fcab-0065bb7756-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:58.591 DEBUG:boto:b'vclfcn-62obj21000falseobj22024-02-01T10:48:26.333Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:58.592 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:49:58.592 DEBUG:boto:path=/vclfcn-62/obj3 2024-02-01T10:49:58.592 DEBUG:boto:auth_path=/vclfcn-62/obj3 2024-02-01T10:49:58.592 DEBUG:boto:path=/vclfcn-62/obj3?versionId=null 2024-02-01T10:49:58.592 DEBUG:boto:auth_path=/vclfcn-62/obj3?versionId=null 2024-02-01T10:49:58.592 DEBUG:boto:Method: GET 2024-02-01T10:49:58.592 DEBUG:boto:Path: /vclfcn-62/obj3?versionId=null 2024-02-01T10:49:58.593 DEBUG:boto:Data: 2024-02-01T10:49:58.593 DEBUG:boto:Headers: {} 2024-02-01T10:49:58.593 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:58.593 DEBUG:boto:Port: 8001 2024-02-01T10:49:58.593 DEBUG:boto:Params: {} 2024-02-01T10:49:58.593 DEBUG:boto:Token: None 2024-02-01T10:49:58.593 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:58 GMT /vclfcn-62/obj3?versionId=null 2024-02-01T10:49:58.593 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:MEq8OTj2MUaQ2TN9NumxdFSEUYY= 2024-02-01T10:49:58.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:MEq8OTj2MUaQ2TN9NumxdFSEUYY=', 'Content-Length': '0'} 2024-02-01T10:49:58.596 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000050dac6e2d3050026-0065bb7756-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:58.597 DEBUG:boto:path=/vclfcn-62/obj3 2024-02-01T10:49:58.597 DEBUG:boto:auth_path=/vclfcn-62/obj3 2024-02-01T10:49:58.597 DEBUG:boto:path=/vclfcn-62/obj3?versionId=null 2024-02-01T10:49:58.597 DEBUG:boto:auth_path=/vclfcn-62/obj3?versionId=null 2024-02-01T10:49:58.597 DEBUG:boto:Method: GET 2024-02-01T10:49:58.597 DEBUG:boto:Path: /vclfcn-62/obj3?versionId=null 2024-02-01T10:49:58.598 DEBUG:boto:Data: 2024-02-01T10:49:58.598 DEBUG:boto:Headers: {} 2024-02-01T10:49:58.598 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:58.598 DEBUG:boto:Port: 8000 2024-02-01T10:49:58.598 DEBUG:boto:Params: {} 2024-02-01T10:49:58.598 DEBUG:boto:Token: None 2024-02-01T10:49:58.598 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:58 GMT /vclfcn-62/obj3?versionId=null 2024-02-01T10:49:58.598 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:MEq8OTj2MUaQ2TN9NumxdFSEUYY= 2024-02-01T10:49:58.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:MEq8OTj2MUaQ2TN9NumxdFSEUYY=', 'Content-Length': '0'} 2024-02-01T10:49:58.601 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000402fef1a9536f598-0065bb7756-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:58.644 DEBUG:boto:path=/vclfcn-62/obj3 2024-02-01T10:49:58.644 DEBUG:boto:auth_path=/vclfcn-62/obj3 2024-02-01T10:49:58.644 DEBUG:boto:path=/vclfcn-62/obj3?versionId=null 2024-02-01T10:49:58.645 DEBUG:boto:auth_path=/vclfcn-62/obj3?versionId=null 2024-02-01T10:49:58.645 DEBUG:boto:Method: HEAD 2024-02-01T10:49:58.645 DEBUG:boto:Path: /vclfcn-62/obj3?versionId=null 2024-02-01T10:49:58.645 DEBUG:boto:Data: 2024-02-01T10:49:58.645 DEBUG:boto:Headers: {} 2024-02-01T10:49:58.645 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:58.645 DEBUG:boto:Port: 8001 2024-02-01T10:49:58.645 DEBUG:boto:Params: {} 2024-02-01T10:49:58.645 DEBUG:boto:Token: None 2024-02-01T10:49:58.646 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:58 GMT /vclfcn-62/obj3?versionId=null 2024-02-01T10:49:58.646 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Z60wvXa03CsLBbNECfUzfMuWyqI= 2024-02-01T10:49:58.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Z60wvXa03CsLBbNECfUzfMuWyqI=', 'Content-Length': '0'} 2024-02-01T10:49:58.649 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000007c76808c9e0b595-0065bb7756-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:58.650 DEBUG:boto:path=/vclfcn-62/obj3 2024-02-01T10:49:58.650 DEBUG:boto:auth_path=/vclfcn-62/obj3 2024-02-01T10:49:58.650 DEBUG:boto:path=/vclfcn-62/obj3?versionId=null 2024-02-01T10:49:58.650 DEBUG:boto:auth_path=/vclfcn-62/obj3?versionId=null 2024-02-01T10:49:58.650 DEBUG:boto:Method: HEAD 2024-02-01T10:49:58.650 DEBUG:boto:Path: /vclfcn-62/obj3?versionId=null 2024-02-01T10:49:58.650 DEBUG:boto:Data: 2024-02-01T10:49:58.650 DEBUG:boto:Headers: {} 2024-02-01T10:49:58.650 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:58.650 DEBUG:boto:Port: 8000 2024-02-01T10:49:58.650 DEBUG:boto:Params: {} 2024-02-01T10:49:58.651 DEBUG:boto:Token: None 2024-02-01T10:49:58.651 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:58 GMT /vclfcn-62/obj3?versionId=null 2024-02-01T10:49:58.651 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Z60wvXa03CsLBbNECfUzfMuWyqI= 2024-02-01T10:49:58.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Z60wvXa03CsLBbNECfUzfMuWyqI=', 'Content-Length': '0'} 2024-02-01T10:49:58.654 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004ca7319436b46600-0065bb7756-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:58.654 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:49:58.654 DEBUG:boto:path=/vclfcn-62/obj3 2024-02-01T10:49:58.655 DEBUG:boto:auth_path=/vclfcn-62/obj3 2024-02-01T10:49:58.655 DEBUG:boto:path=/vclfcn-62/obj3?versionId=null 2024-02-01T10:49:58.655 DEBUG:boto:auth_path=/vclfcn-62/obj3?versionId=null 2024-02-01T10:49:58.655 DEBUG:boto:Method: GET 2024-02-01T10:49:58.655 DEBUG:boto:Path: /vclfcn-62/obj3?versionId=null 2024-02-01T10:49:58.655 DEBUG:boto:Data: 2024-02-01T10:49:58.655 DEBUG:boto:Headers: {} 2024-02-01T10:49:58.655 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:58.655 DEBUG:boto:Port: 8001 2024-02-01T10:49:58.655 DEBUG:boto:Params: {} 2024-02-01T10:49:58.655 DEBUG:boto:Token: None 2024-02-01T10:49:58.656 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:58 GMT /vclfcn-62/obj3?versionId=null 2024-02-01T10:49:58.656 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:MEq8OTj2MUaQ2TN9NumxdFSEUYY= 2024-02-01T10:49:58.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:MEq8OTj2MUaQ2TN9NumxdFSEUYY=', 'Content-Length': '0'} 2024-02-01T10:49:58.659 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002c85cf1bd7a9e30f-0065bb7756-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:58.704 DEBUG:boto:path=/vclfcn-62/obj3 2024-02-01T10:49:58.704 DEBUG:boto:auth_path=/vclfcn-62/obj3 2024-02-01T10:49:58.704 DEBUG:boto:path=/vclfcn-62/obj3?versionId=null 2024-02-01T10:49:58.704 DEBUG:boto:auth_path=/vclfcn-62/obj3?versionId=null 2024-02-01T10:49:58.705 DEBUG:boto:Method: GET 2024-02-01T10:49:58.705 DEBUG:boto:Path: /vclfcn-62/obj3?versionId=null 2024-02-01T10:49:58.705 DEBUG:boto:Data: 2024-02-01T10:49:58.705 DEBUG:boto:Headers: {} 2024-02-01T10:49:58.705 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:58.705 DEBUG:boto:Port: 8000 2024-02-01T10:49:58.705 DEBUG:boto:Params: {} 2024-02-01T10:49:58.705 DEBUG:boto:Token: None 2024-02-01T10:49:58.706 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:58 GMT /vclfcn-62/obj3?versionId=null 2024-02-01T10:49:58.706 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:MEq8OTj2MUaQ2TN9NumxdFSEUYY= 2024-02-01T10:49:58.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:MEq8OTj2MUaQ2TN9NumxdFSEUYY=', 'Content-Length': '0'} 2024-02-01T10:49:58.709 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000642a223d83b50d43-0065bb7756-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:58.756 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:58.757 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:58.757 DEBUG:boto:path=/vclfcn-62/?prefix=obj3 2024-02-01T10:49:58.757 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj3 2024-02-01T10:49:58.757 DEBUG:boto:Method: GET 2024-02-01T10:49:58.757 DEBUG:boto:Path: /vclfcn-62/?prefix=obj3 2024-02-01T10:49:58.758 DEBUG:boto:Data: 2024-02-01T10:49:58.758 DEBUG:boto:Headers: {} 2024-02-01T10:49:58.758 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:58.758 DEBUG:boto:Port: 8001 2024-02-01T10:49:58.758 DEBUG:boto:Params: {} 2024-02-01T10:49:58.759 DEBUG:boto:Token: None 2024-02-01T10:49:58.759 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:58 GMT /vclfcn-62/ 2024-02-01T10:49:58.759 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jOxZZx4i6msi/5P4JWwenEFoGbA= 2024-02-01T10:49:58.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jOxZZx4i6msi/5P4JWwenEFoGbA=', 'Content-Length': '0'} 2024-02-01T10:49:58.764 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5ddd1075418ae87-0065bb7756-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:58.812 DEBUG:boto:b'vclfcn-62obj31000falseobj32024-02-01T10:48:26.384Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:58.815 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:58.815 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:58.815 DEBUG:boto:path=/vclfcn-62/?prefix=obj3 2024-02-01T10:49:58.815 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj3 2024-02-01T10:49:58.816 DEBUG:boto:Method: GET 2024-02-01T10:49:58.816 DEBUG:boto:Path: /vclfcn-62/?prefix=obj3 2024-02-01T10:49:58.816 DEBUG:boto:Data: 2024-02-01T10:49:58.816 DEBUG:boto:Headers: {} 2024-02-01T10:49:58.816 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:58.816 DEBUG:boto:Port: 8000 2024-02-01T10:49:58.816 DEBUG:boto:Params: {} 2024-02-01T10:49:58.816 DEBUG:boto:Token: None 2024-02-01T10:49:58.816 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:58 GMT /vclfcn-62/ 2024-02-01T10:49:58.817 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jOxZZx4i6msi/5P4JWwenEFoGbA= 2024-02-01T10:49:58.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jOxZZx4i6msi/5P4JWwenEFoGbA=', 'Content-Length': '0'} 2024-02-01T10:49:58.821 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f04dff0de013acac-0065bb7756-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:58.821 DEBUG:boto:b'vclfcn-62obj31000falseobj32024-02-01T10:48:26.384Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:58.822 DEBUG:boto:path=/vclfcn-62/obj3 2024-02-01T10:49:58.822 DEBUG:boto:auth_path=/vclfcn-62/obj3 2024-02-01T10:49:58.822 DEBUG:boto:Method: HEAD 2024-02-01T10:49:58.822 DEBUG:boto:Path: /vclfcn-62/obj3 2024-02-01T10:49:58.822 DEBUG:boto:Data: 2024-02-01T10:49:58.822 DEBUG:boto:Headers: {} 2024-02-01T10:49:58.822 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:58.822 DEBUG:boto:Port: 8001 2024-02-01T10:49:58.822 DEBUG:boto:Params: {} 2024-02-01T10:49:58.823 DEBUG:boto:Token: None 2024-02-01T10:49:58.823 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:58 GMT /vclfcn-62/obj3 2024-02-01T10:49:58.823 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sTu6k8qoCBeeagAdkzzAErq44+M= 2024-02-01T10:49:58.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sTu6k8qoCBeeagAdkzzAErq44+M=', 'Content-Length': '0'} 2024-02-01T10:49:58.826 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000f477ac07e7a4e77-0065bb7756-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:58.826 DEBUG:boto:path=/vclfcn-62/obj3 2024-02-01T10:49:58.826 DEBUG:boto:auth_path=/vclfcn-62/obj3 2024-02-01T10:49:58.826 DEBUG:boto:Method: HEAD 2024-02-01T10:49:58.826 DEBUG:boto:Path: /vclfcn-62/obj3 2024-02-01T10:49:58.826 DEBUG:boto:Data: 2024-02-01T10:49:58.826 DEBUG:boto:Headers: {} 2024-02-01T10:49:58.826 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:58.827 DEBUG:boto:Port: 8000 2024-02-01T10:49:58.827 DEBUG:boto:Params: {} 2024-02-01T10:49:58.827 DEBUG:boto:Token: None 2024-02-01T10:49:58.827 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:58 GMT /vclfcn-62/obj3 2024-02-01T10:49:58.827 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sTu6k8qoCBeeagAdkzzAErq44+M= 2024-02-01T10:49:58.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sTu6k8qoCBeeagAdkzzAErq44+M=', 'Content-Length': '0'} 2024-02-01T10:49:58.830 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009294159ba5df3357-0065bb7756-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:58.830 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:49:58.831 DEBUG:boto:path=/vclfcn-62/obj3 2024-02-01T10:49:58.831 DEBUG:boto:auth_path=/vclfcn-62/obj3 2024-02-01T10:49:58.831 DEBUG:boto:Method: GET 2024-02-01T10:49:58.832 DEBUG:boto:Path: /vclfcn-62/obj3 2024-02-01T10:49:58.832 DEBUG:boto:Data: 2024-02-01T10:49:58.832 DEBUG:boto:Headers: {} 2024-02-01T10:49:58.832 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:58.832 DEBUG:boto:Port: 8001 2024-02-01T10:49:58.832 DEBUG:boto:Params: {} 2024-02-01T10:49:58.832 DEBUG:boto:Token: None 2024-02-01T10:49:58.832 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:58 GMT /vclfcn-62/obj3 2024-02-01T10:49:58.832 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8zLONAlKBcUruJyGCd/DuDfK3iA= 2024-02-01T10:49:58.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8zLONAlKBcUruJyGCd/DuDfK3iA=', 'Content-Length': '0'} 2024-02-01T10:49:58.835 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000984ea61d45bf84cc-0065bb7756-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:58.876 DEBUG:boto:path=/vclfcn-62/obj3 2024-02-01T10:49:58.876 DEBUG:boto:auth_path=/vclfcn-62/obj3 2024-02-01T10:49:58.876 DEBUG:boto:Method: GET 2024-02-01T10:49:58.876 DEBUG:boto:Path: /vclfcn-62/obj3 2024-02-01T10:49:58.877 DEBUG:boto:Data: 2024-02-01T10:49:58.877 DEBUG:boto:Headers: {} 2024-02-01T10:49:58.877 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:58.877 DEBUG:boto:Port: 8000 2024-02-01T10:49:58.877 DEBUG:boto:Params: {} 2024-02-01T10:49:58.877 DEBUG:boto:Token: None 2024-02-01T10:49:58.877 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:58 GMT /vclfcn-62/obj3 2024-02-01T10:49:58.877 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8zLONAlKBcUruJyGCd/DuDfK3iA= 2024-02-01T10:49:58.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8zLONAlKBcUruJyGCd/DuDfK3iA=', 'Content-Length': '0'} 2024-02-01T10:49:58.880 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bafb6558b18aa3b6-0065bb7756-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:58.924 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:58.925 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:58.925 DEBUG:boto:path=/vclfcn-62/?prefix=obj3 2024-02-01T10:49:58.925 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj3 2024-02-01T10:49:58.925 DEBUG:boto:Method: GET 2024-02-01T10:49:58.925 DEBUG:boto:Path: /vclfcn-62/?prefix=obj3 2024-02-01T10:49:58.925 DEBUG:boto:Data: 2024-02-01T10:49:58.925 DEBUG:boto:Headers: {} 2024-02-01T10:49:58.925 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:58.925 DEBUG:boto:Port: 8001 2024-02-01T10:49:58.925 DEBUG:boto:Params: {} 2024-02-01T10:49:58.926 DEBUG:boto:Token: None 2024-02-01T10:49:58.926 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:58 GMT /vclfcn-62/ 2024-02-01T10:49:58.926 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jOxZZx4i6msi/5P4JWwenEFoGbA= 2024-02-01T10:49:58.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jOxZZx4i6msi/5P4JWwenEFoGbA=', 'Content-Length': '0'} 2024-02-01T10:49:58.930 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083d2cabff8b12e42-0065bb7756-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:58.976 DEBUG:boto:b'vclfcn-62obj31000falseobj32024-02-01T10:48:26.384Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:58.977 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:58.977 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:58.977 DEBUG:boto:path=/vclfcn-62/?prefix=obj3 2024-02-01T10:49:58.978 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj3 2024-02-01T10:49:58.978 DEBUG:boto:Method: GET 2024-02-01T10:49:58.978 DEBUG:boto:Path: /vclfcn-62/?prefix=obj3 2024-02-01T10:49:58.978 DEBUG:boto:Data: 2024-02-01T10:49:58.978 DEBUG:boto:Headers: {} 2024-02-01T10:49:58.978 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:58.979 DEBUG:boto:Port: 8000 2024-02-01T10:49:58.979 DEBUG:boto:Params: {} 2024-02-01T10:49:58.979 DEBUG:boto:Token: None 2024-02-01T10:49:58.979 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:58 GMT /vclfcn-62/ 2024-02-01T10:49:58.979 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jOxZZx4i6msi/5P4JWwenEFoGbA= 2024-02-01T10:49:58.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jOxZZx4i6msi/5P4JWwenEFoGbA=', 'Content-Length': '0'} 2024-02-01T10:49:58.984 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f2db59e072813f4-0065bb7756-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:58.985 DEBUG:boto:b'vclfcn-62obj31000falseobj32024-02-01T10:48:26.384Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:58.985 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:49:58.985 DEBUG:boto:path=/vclfcn-62/obj4 2024-02-01T10:49:58.986 DEBUG:boto:auth_path=/vclfcn-62/obj4 2024-02-01T10:49:58.986 DEBUG:boto:path=/vclfcn-62/obj4?versionId=null 2024-02-01T10:49:58.986 DEBUG:boto:auth_path=/vclfcn-62/obj4?versionId=null 2024-02-01T10:49:58.986 DEBUG:boto:Method: GET 2024-02-01T10:49:58.986 DEBUG:boto:Path: /vclfcn-62/obj4?versionId=null 2024-02-01T10:49:58.986 DEBUG:boto:Data: 2024-02-01T10:49:58.986 DEBUG:boto:Headers: {} 2024-02-01T10:49:58.986 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:58.986 DEBUG:boto:Port: 8001 2024-02-01T10:49:58.986 DEBUG:boto:Params: {} 2024-02-01T10:49:58.987 DEBUG:boto:Token: None 2024-02-01T10:49:58.987 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:58 GMT /vclfcn-62/obj4?versionId=null 2024-02-01T10:49:58.987 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:DVP1OK8fqCIfRkOCcSq+yQi29RM= 2024-02-01T10:49:58.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:DVP1OK8fqCIfRkOCcSq+yQi29RM=', 'Content-Length': '0'} 2024-02-01T10:49:58.990 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e4720663e23e7fb8-0065bb7756-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:58.990 DEBUG:boto:path=/vclfcn-62/obj4 2024-02-01T10:49:58.990 DEBUG:boto:auth_path=/vclfcn-62/obj4 2024-02-01T10:49:58.990 DEBUG:boto:path=/vclfcn-62/obj4?versionId=null 2024-02-01T10:49:58.991 DEBUG:boto:auth_path=/vclfcn-62/obj4?versionId=null 2024-02-01T10:49:58.991 DEBUG:boto:Method: GET 2024-02-01T10:49:58.991 DEBUG:boto:Path: /vclfcn-62/obj4?versionId=null 2024-02-01T10:49:58.991 DEBUG:boto:Data: 2024-02-01T10:49:58.991 DEBUG:boto:Headers: {} 2024-02-01T10:49:58.991 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:58.991 DEBUG:boto:Port: 8000 2024-02-01T10:49:58.991 DEBUG:boto:Params: {} 2024-02-01T10:49:58.991 DEBUG:boto:Token: None 2024-02-01T10:49:58.991 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:58 GMT /vclfcn-62/obj4?versionId=null 2024-02-01T10:49:58.992 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:DVP1OK8fqCIfRkOCcSq+yQi29RM= 2024-02-01T10:49:58.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:DVP1OK8fqCIfRkOCcSq+yQi29RM=', 'Content-Length': '0'} 2024-02-01T10:49:58.994 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008e1adb2251e85cc7-0065bb7756-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:59.036 DEBUG:boto:path=/vclfcn-62/obj4 2024-02-01T10:49:59.036 DEBUG:boto:auth_path=/vclfcn-62/obj4 2024-02-01T10:49:59.036 DEBUG:boto:path=/vclfcn-62/obj4?versionId=null 2024-02-01T10:49:59.036 DEBUG:boto:auth_path=/vclfcn-62/obj4?versionId=null 2024-02-01T10:49:59.037 DEBUG:boto:Method: HEAD 2024-02-01T10:49:59.037 DEBUG:boto:Path: /vclfcn-62/obj4?versionId=null 2024-02-01T10:49:59.037 DEBUG:boto:Data: 2024-02-01T10:49:59.037 DEBUG:boto:Headers: {} 2024-02-01T10:49:59.037 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:59.037 DEBUG:boto:Port: 8001 2024-02-01T10:49:59.038 DEBUG:boto:Params: {} 2024-02-01T10:49:59.038 DEBUG:boto:Token: None 2024-02-01T10:49:59.038 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:59 GMT /vclfcn-62/obj4?versionId=null 2024-02-01T10:49:59.038 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NZNuPBQZUpv+lV7DTktSZPdlFtM= 2024-02-01T10:49:59.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NZNuPBQZUpv+lV7DTktSZPdlFtM=', 'Content-Length': '0'} 2024-02-01T10:49:59.041 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000000a31e34a79bdafa-0065bb7757-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:59.042 DEBUG:boto:path=/vclfcn-62/obj4 2024-02-01T10:49:59.042 DEBUG:boto:auth_path=/vclfcn-62/obj4 2024-02-01T10:49:59.042 DEBUG:boto:path=/vclfcn-62/obj4?versionId=null 2024-02-01T10:49:59.042 DEBUG:boto:auth_path=/vclfcn-62/obj4?versionId=null 2024-02-01T10:49:59.042 DEBUG:boto:Method: HEAD 2024-02-01T10:49:59.042 DEBUG:boto:Path: /vclfcn-62/obj4?versionId=null 2024-02-01T10:49:59.042 DEBUG:boto:Data: 2024-02-01T10:49:59.042 DEBUG:boto:Headers: {} 2024-02-01T10:49:59.043 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:59.043 DEBUG:boto:Port: 8000 2024-02-01T10:49:59.043 DEBUG:boto:Params: {} 2024-02-01T10:49:59.043 DEBUG:boto:Token: None 2024-02-01T10:49:59.043 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:59 GMT /vclfcn-62/obj4?versionId=null 2024-02-01T10:49:59.044 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NZNuPBQZUpv+lV7DTktSZPdlFtM= 2024-02-01T10:49:59.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NZNuPBQZUpv+lV7DTktSZPdlFtM=', 'Content-Length': '0'} 2024-02-01T10:49:59.047 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008f49988f1459a4da-0065bb7757-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:59.047 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:49:59.047 DEBUG:boto:path=/vclfcn-62/obj4 2024-02-01T10:49:59.047 DEBUG:boto:auth_path=/vclfcn-62/obj4 2024-02-01T10:49:59.047 DEBUG:boto:path=/vclfcn-62/obj4?versionId=null 2024-02-01T10:49:59.047 DEBUG:boto:auth_path=/vclfcn-62/obj4?versionId=null 2024-02-01T10:49:59.048 DEBUG:boto:Method: GET 2024-02-01T10:49:59.048 DEBUG:boto:Path: /vclfcn-62/obj4?versionId=null 2024-02-01T10:49:59.048 DEBUG:boto:Data: 2024-02-01T10:49:59.048 DEBUG:boto:Headers: {} 2024-02-01T10:49:59.048 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:59.048 DEBUG:boto:Port: 8001 2024-02-01T10:49:59.048 DEBUG:boto:Params: {} 2024-02-01T10:49:59.048 DEBUG:boto:Token: None 2024-02-01T10:49:59.049 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:59 GMT /vclfcn-62/obj4?versionId=null 2024-02-01T10:49:59.049 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zxnhWaSvGGze+76pP37RuTYk3No= 2024-02-01T10:49:59.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zxnhWaSvGGze+76pP37RuTYk3No=', 'Content-Length': '0'} 2024-02-01T10:49:59.052 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a354d2aa1e5497dd-0065bb7757-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:59.092 DEBUG:boto:path=/vclfcn-62/obj4 2024-02-01T10:49:59.092 DEBUG:boto:auth_path=/vclfcn-62/obj4 2024-02-01T10:49:59.092 DEBUG:boto:path=/vclfcn-62/obj4?versionId=null 2024-02-01T10:49:59.092 DEBUG:boto:auth_path=/vclfcn-62/obj4?versionId=null 2024-02-01T10:49:59.093 DEBUG:boto:Method: GET 2024-02-01T10:49:59.093 DEBUG:boto:Path: /vclfcn-62/obj4?versionId=null 2024-02-01T10:49:59.093 DEBUG:boto:Data: 2024-02-01T10:49:59.093 DEBUG:boto:Headers: {} 2024-02-01T10:49:59.093 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:59.093 DEBUG:boto:Port: 8000 2024-02-01T10:49:59.093 DEBUG:boto:Params: {} 2024-02-01T10:49:59.093 DEBUG:boto:Token: None 2024-02-01T10:49:59.093 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:59 GMT /vclfcn-62/obj4?versionId=null 2024-02-01T10:49:59.093 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zxnhWaSvGGze+76pP37RuTYk3No= 2024-02-01T10:49:59.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zxnhWaSvGGze+76pP37RuTYk3No=', 'Content-Length': '0'} 2024-02-01T10:49:59.096 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000289ba6e4e3661a25-0065bb7757-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:59.140 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:59.140 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:59.140 DEBUG:boto:path=/vclfcn-62/?prefix=obj4 2024-02-01T10:49:59.140 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj4 2024-02-01T10:49:59.140 DEBUG:boto:Method: GET 2024-02-01T10:49:59.141 DEBUG:boto:Path: /vclfcn-62/?prefix=obj4 2024-02-01T10:49:59.141 DEBUG:boto:Data: 2024-02-01T10:49:59.141 DEBUG:boto:Headers: {} 2024-02-01T10:49:59.141 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:59.141 DEBUG:boto:Port: 8001 2024-02-01T10:49:59.141 DEBUG:boto:Params: {} 2024-02-01T10:49:59.141 DEBUG:boto:Token: None 2024-02-01T10:49:59.141 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:59 GMT /vclfcn-62/ 2024-02-01T10:49:59.141 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7iURGu0oFNa8Tp/DBTI2JjcIu9A= 2024-02-01T10:49:59.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7iURGu0oFNa8Tp/DBTI2JjcIu9A=', 'Content-Length': '0'} 2024-02-01T10:49:59.146 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000113280c0d07c4e14-0065bb7757-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:59.192 DEBUG:boto:b'vclfcn-62obj41000falseobj42024-02-01T10:48:26.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:59.193 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:59.193 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:59.193 DEBUG:boto:path=/vclfcn-62/?prefix=obj4 2024-02-01T10:49:59.193 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj4 2024-02-01T10:49:59.193 DEBUG:boto:Method: GET 2024-02-01T10:49:59.193 DEBUG:boto:Path: /vclfcn-62/?prefix=obj4 2024-02-01T10:49:59.193 DEBUG:boto:Data: 2024-02-01T10:49:59.193 DEBUG:boto:Headers: {} 2024-02-01T10:49:59.194 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:59.194 DEBUG:boto:Port: 8000 2024-02-01T10:49:59.194 DEBUG:boto:Params: {} 2024-02-01T10:49:59.194 DEBUG:boto:Token: None 2024-02-01T10:49:59.194 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:59 GMT /vclfcn-62/ 2024-02-01T10:49:59.194 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7iURGu0oFNa8Tp/DBTI2JjcIu9A= 2024-02-01T10:49:59.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7iURGu0oFNa8Tp/DBTI2JjcIu9A=', 'Content-Length': '0'} 2024-02-01T10:49:59.199 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d622783f09826b58-0065bb7757-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:59.199 DEBUG:boto:b'vclfcn-62obj41000falseobj42024-02-01T10:48:26.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:59.200 DEBUG:boto:path=/vclfcn-62/obj4 2024-02-01T10:49:59.200 DEBUG:boto:auth_path=/vclfcn-62/obj4 2024-02-01T10:49:59.200 DEBUG:boto:Method: HEAD 2024-02-01T10:49:59.201 DEBUG:boto:Path: /vclfcn-62/obj4 2024-02-01T10:49:59.201 DEBUG:boto:Data: 2024-02-01T10:49:59.201 DEBUG:boto:Headers: {} 2024-02-01T10:49:59.201 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:59.202 DEBUG:boto:Port: 8001 2024-02-01T10:49:59.202 DEBUG:boto:Params: {} 2024-02-01T10:49:59.202 DEBUG:boto:Token: None 2024-02-01T10:49:59.202 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:59 GMT /vclfcn-62/obj4 2024-02-01T10:49:59.203 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZmuwQgXBqaR+SYdy1kHzgC5ezfU= 2024-02-01T10:49:59.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZmuwQgXBqaR+SYdy1kHzgC5ezfU=', 'Content-Length': '0'} 2024-02-01T10:49:59.206 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c9e4a2b6db5e0d7a-0065bb7757-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:59.206 DEBUG:boto:path=/vclfcn-62/obj4 2024-02-01T10:49:59.206 DEBUG:boto:auth_path=/vclfcn-62/obj4 2024-02-01T10:49:59.206 DEBUG:boto:Method: HEAD 2024-02-01T10:49:59.207 DEBUG:boto:Path: /vclfcn-62/obj4 2024-02-01T10:49:59.207 DEBUG:boto:Data: 2024-02-01T10:49:59.207 DEBUG:boto:Headers: {} 2024-02-01T10:49:59.207 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:59.207 DEBUG:boto:Port: 8000 2024-02-01T10:49:59.207 DEBUG:boto:Params: {} 2024-02-01T10:49:59.207 DEBUG:boto:Token: None 2024-02-01T10:49:59.207 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:59 GMT /vclfcn-62/obj4 2024-02-01T10:49:59.208 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZmuwQgXBqaR+SYdy1kHzgC5ezfU= 2024-02-01T10:49:59.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZmuwQgXBqaR+SYdy1kHzgC5ezfU=', 'Content-Length': '0'} 2024-02-01T10:49:59.210 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c4d1a7082d16baaf-0065bb7757-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:59.211 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:49:59.211 DEBUG:boto:path=/vclfcn-62/obj4 2024-02-01T10:49:59.211 DEBUG:boto:auth_path=/vclfcn-62/obj4 2024-02-01T10:49:59.211 DEBUG:boto:Method: GET 2024-02-01T10:49:59.211 DEBUG:boto:Path: /vclfcn-62/obj4 2024-02-01T10:49:59.211 DEBUG:boto:Data: 2024-02-01T10:49:59.212 DEBUG:boto:Headers: {} 2024-02-01T10:49:59.212 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:59.212 DEBUG:boto:Port: 8001 2024-02-01T10:49:59.212 DEBUG:boto:Params: {} 2024-02-01T10:49:59.212 DEBUG:boto:Token: None 2024-02-01T10:49:59.212 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:59 GMT /vclfcn-62/obj4 2024-02-01T10:49:59.212 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GeswpSeJ1XpTcUW72Y8Eiyxihv4= 2024-02-01T10:49:59.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GeswpSeJ1XpTcUW72Y8Eiyxihv4=', 'Content-Length': '0'} 2024-02-01T10:49:59.215 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000442c21c7ca0c26b0-0065bb7757-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:59.256 DEBUG:boto:path=/vclfcn-62/obj4 2024-02-01T10:49:59.256 DEBUG:boto:auth_path=/vclfcn-62/obj4 2024-02-01T10:49:59.256 DEBUG:boto:Method: GET 2024-02-01T10:49:59.256 DEBUG:boto:Path: /vclfcn-62/obj4 2024-02-01T10:49:59.257 DEBUG:boto:Data: 2024-02-01T10:49:59.257 DEBUG:boto:Headers: {} 2024-02-01T10:49:59.257 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:59.257 DEBUG:boto:Port: 8000 2024-02-01T10:49:59.257 DEBUG:boto:Params: {} 2024-02-01T10:49:59.257 DEBUG:boto:Token: None 2024-02-01T10:49:59.257 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:59 GMT /vclfcn-62/obj4 2024-02-01T10:49:59.257 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GeswpSeJ1XpTcUW72Y8Eiyxihv4= 2024-02-01T10:49:59.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GeswpSeJ1XpTcUW72Y8Eiyxihv4=', 'Content-Length': '0'} 2024-02-01T10:49:59.260 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000829dc699fe3e8809-0065bb7757-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:59.304 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:59.304 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:59.304 DEBUG:boto:path=/vclfcn-62/?prefix=obj4 2024-02-01T10:49:59.305 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj4 2024-02-01T10:49:59.305 DEBUG:boto:Method: GET 2024-02-01T10:49:59.305 DEBUG:boto:Path: /vclfcn-62/?prefix=obj4 2024-02-01T10:49:59.305 DEBUG:boto:Data: 2024-02-01T10:49:59.305 DEBUG:boto:Headers: {} 2024-02-01T10:49:59.305 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:59.305 DEBUG:boto:Port: 8001 2024-02-01T10:49:59.305 DEBUG:boto:Params: {} 2024-02-01T10:49:59.305 DEBUG:boto:Token: None 2024-02-01T10:49:59.306 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:59 GMT /vclfcn-62/ 2024-02-01T10:49:59.306 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7iURGu0oFNa8Tp/DBTI2JjcIu9A= 2024-02-01T10:49:59.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7iURGu0oFNa8Tp/DBTI2JjcIu9A=', 'Content-Length': '0'} 2024-02-01T10:49:59.310 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b558d926e2da401-0065bb7757-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:59.352 DEBUG:boto:b'vclfcn-62obj41000falseobj42024-02-01T10:48:26.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:59.353 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:59.353 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:59.354 DEBUG:boto:path=/vclfcn-62/?prefix=obj4 2024-02-01T10:49:59.354 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj4 2024-02-01T10:49:59.354 DEBUG:boto:Method: GET 2024-02-01T10:49:59.354 DEBUG:boto:Path: /vclfcn-62/?prefix=obj4 2024-02-01T10:49:59.354 DEBUG:boto:Data: 2024-02-01T10:49:59.355 DEBUG:boto:Headers: {} 2024-02-01T10:49:59.355 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:59.355 DEBUG:boto:Port: 8000 2024-02-01T10:49:59.355 DEBUG:boto:Params: {} 2024-02-01T10:49:59.355 DEBUG:boto:Token: None 2024-02-01T10:49:59.355 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:59 GMT /vclfcn-62/ 2024-02-01T10:49:59.355 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7iURGu0oFNa8Tp/DBTI2JjcIu9A= 2024-02-01T10:49:59.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7iURGu0oFNa8Tp/DBTI2JjcIu9A=', 'Content-Length': '0'} 2024-02-01T10:49:59.360 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ccf72d3abb9b6523-0065bb7757-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:59.360 DEBUG:boto:b'vclfcn-62obj41000falseobj42024-02-01T10:48:26.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:59.361 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-02-01T10:49:59.361 DEBUG:boto:path=/vclfcn-62/obj5 2024-02-01T10:49:59.361 DEBUG:boto:auth_path=/vclfcn-62/obj5 2024-02-01T10:49:59.361 DEBUG:boto:path=/vclfcn-62/obj5?versionId=null 2024-02-01T10:49:59.361 DEBUG:boto:auth_path=/vclfcn-62/obj5?versionId=null 2024-02-01T10:49:59.361 DEBUG:boto:Method: GET 2024-02-01T10:49:59.361 DEBUG:boto:Path: /vclfcn-62/obj5?versionId=null 2024-02-01T10:49:59.361 DEBUG:boto:Data: 2024-02-01T10:49:59.362 DEBUG:boto:Headers: {} 2024-02-01T10:49:59.362 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:59.362 DEBUG:boto:Port: 8001 2024-02-01T10:49:59.362 DEBUG:boto:Params: {} 2024-02-01T10:49:59.362 DEBUG:boto:Token: None 2024-02-01T10:49:59.362 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:59 GMT /vclfcn-62/obj5?versionId=null 2024-02-01T10:49:59.362 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RdksuyI37Wuaae+sZYbgtQTtmkI= 2024-02-01T10:49:59.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RdksuyI37Wuaae+sZYbgtQTtmkI=', 'Content-Length': '0'} 2024-02-01T10:49:59.365 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007395bc7f08bf3883-0065bb7757-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:59.365 DEBUG:boto:path=/vclfcn-62/obj5 2024-02-01T10:49:59.366 DEBUG:boto:auth_path=/vclfcn-62/obj5 2024-02-01T10:49:59.366 DEBUG:boto:path=/vclfcn-62/obj5?versionId=null 2024-02-01T10:49:59.366 DEBUG:boto:auth_path=/vclfcn-62/obj5?versionId=null 2024-02-01T10:49:59.366 DEBUG:boto:Method: GET 2024-02-01T10:49:59.366 DEBUG:boto:Path: /vclfcn-62/obj5?versionId=null 2024-02-01T10:49:59.366 DEBUG:boto:Data: 2024-02-01T10:49:59.366 DEBUG:boto:Headers: {} 2024-02-01T10:49:59.366 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:59.366 DEBUG:boto:Port: 8000 2024-02-01T10:49:59.367 DEBUG:boto:Params: {} 2024-02-01T10:49:59.367 DEBUG:boto:Token: None 2024-02-01T10:49:59.367 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:59 GMT /vclfcn-62/obj5?versionId=null 2024-02-01T10:49:59.367 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RdksuyI37Wuaae+sZYbgtQTtmkI= 2024-02-01T10:49:59.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RdksuyI37Wuaae+sZYbgtQTtmkI=', 'Content-Length': '0'} 2024-02-01T10:49:59.370 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000034b7186c6e6d4e3b-0065bb7757-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:59.416 DEBUG:boto:path=/vclfcn-62/obj5 2024-02-01T10:49:59.417 DEBUG:boto:auth_path=/vclfcn-62/obj5 2024-02-01T10:49:59.417 DEBUG:boto:path=/vclfcn-62/obj5?versionId=null 2024-02-01T10:49:59.417 DEBUG:boto:auth_path=/vclfcn-62/obj5?versionId=null 2024-02-01T10:49:59.417 DEBUG:boto:Method: HEAD 2024-02-01T10:49:59.418 DEBUG:boto:Path: /vclfcn-62/obj5?versionId=null 2024-02-01T10:49:59.418 DEBUG:boto:Data: 2024-02-01T10:49:59.418 DEBUG:boto:Headers: {} 2024-02-01T10:49:59.418 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:59.418 DEBUG:boto:Port: 8001 2024-02-01T10:49:59.419 DEBUG:boto:Params: {} 2024-02-01T10:49:59.419 DEBUG:boto:Token: None 2024-02-01T10:49:59.419 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:59 GMT /vclfcn-62/obj5?versionId=null 2024-02-01T10:49:59.419 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:LY2VwcltTcpa3X+vPeeK+WRRlk8= 2024-02-01T10:49:59.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:LY2VwcltTcpa3X+vPeeK+WRRlk8=', 'Content-Length': '0'} 2024-02-01T10:49:59.422 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004fa18ee430d3522e-0065bb7757-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:59.423 DEBUG:boto:path=/vclfcn-62/obj5 2024-02-01T10:49:59.423 DEBUG:boto:auth_path=/vclfcn-62/obj5 2024-02-01T10:49:59.423 DEBUG:boto:path=/vclfcn-62/obj5?versionId=null 2024-02-01T10:49:59.423 DEBUG:boto:auth_path=/vclfcn-62/obj5?versionId=null 2024-02-01T10:49:59.424 DEBUG:boto:Method: HEAD 2024-02-01T10:49:59.424 DEBUG:boto:Path: /vclfcn-62/obj5?versionId=null 2024-02-01T10:49:59.424 DEBUG:boto:Data: 2024-02-01T10:49:59.424 DEBUG:boto:Headers: {} 2024-02-01T10:49:59.424 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:59.424 DEBUG:boto:Port: 8000 2024-02-01T10:49:59.424 DEBUG:boto:Params: {} 2024-02-01T10:49:59.424 DEBUG:boto:Token: None 2024-02-01T10:49:59.425 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:59 GMT /vclfcn-62/obj5?versionId=null 2024-02-01T10:49:59.427 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:LY2VwcltTcpa3X+vPeeK+WRRlk8= 2024-02-01T10:49:59.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:LY2VwcltTcpa3X+vPeeK+WRRlk8=', 'Content-Length': '0'} 2024-02-01T10:49:59.430 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000925b2d687699fb5f-0065bb7757-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:59.430 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-02-01T10:49:59.431 DEBUG:boto:path=/vclfcn-62/obj5 2024-02-01T10:49:59.431 DEBUG:boto:auth_path=/vclfcn-62/obj5 2024-02-01T10:49:59.431 DEBUG:boto:path=/vclfcn-62/obj5?versionId=null 2024-02-01T10:49:59.431 DEBUG:boto:auth_path=/vclfcn-62/obj5?versionId=null 2024-02-01T10:49:59.431 DEBUG:boto:Method: GET 2024-02-01T10:49:59.431 DEBUG:boto:Path: /vclfcn-62/obj5?versionId=null 2024-02-01T10:49:59.431 DEBUG:boto:Data: 2024-02-01T10:49:59.431 DEBUG:boto:Headers: {} 2024-02-01T10:49:59.431 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:59.432 DEBUG:boto:Port: 8001 2024-02-01T10:49:59.432 DEBUG:boto:Params: {} 2024-02-01T10:49:59.432 DEBUG:boto:Token: None 2024-02-01T10:49:59.432 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:59 GMT /vclfcn-62/obj5?versionId=null 2024-02-01T10:49:59.432 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RdksuyI37Wuaae+sZYbgtQTtmkI= 2024-02-01T10:49:59.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RdksuyI37Wuaae+sZYbgtQTtmkI=', 'Content-Length': '0'} 2024-02-01T10:49:59.435 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002707b391b91e07e3-0065bb7757-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:59.480 DEBUG:boto:path=/vclfcn-62/obj5 2024-02-01T10:49:59.480 DEBUG:boto:auth_path=/vclfcn-62/obj5 2024-02-01T10:49:59.481 DEBUG:boto:path=/vclfcn-62/obj5?versionId=null 2024-02-01T10:49:59.481 DEBUG:boto:auth_path=/vclfcn-62/obj5?versionId=null 2024-02-01T10:49:59.481 DEBUG:boto:Method: GET 2024-02-01T10:49:59.481 DEBUG:boto:Path: /vclfcn-62/obj5?versionId=null 2024-02-01T10:49:59.481 DEBUG:boto:Data: 2024-02-01T10:49:59.481 DEBUG:boto:Headers: {} 2024-02-01T10:49:59.481 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:59.481 DEBUG:boto:Port: 8000 2024-02-01T10:49:59.481 DEBUG:boto:Params: {} 2024-02-01T10:49:59.481 DEBUG:boto:Token: None 2024-02-01T10:49:59.482 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:59 GMT /vclfcn-62/obj5?versionId=null 2024-02-01T10:49:59.482 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RdksuyI37Wuaae+sZYbgtQTtmkI= 2024-02-01T10:49:59.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RdksuyI37Wuaae+sZYbgtQTtmkI=', 'Content-Length': '0'} 2024-02-01T10:49:59.485 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c583661ad53e6e3e-0065bb7757-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:59.528 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:59.529 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:59.529 DEBUG:boto:path=/vclfcn-62/?prefix=obj5 2024-02-01T10:49:59.529 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj5 2024-02-01T10:49:59.529 DEBUG:boto:Method: GET 2024-02-01T10:49:59.529 DEBUG:boto:Path: /vclfcn-62/?prefix=obj5 2024-02-01T10:49:59.530 DEBUG:boto:Data: 2024-02-01T10:49:59.530 DEBUG:boto:Headers: {} 2024-02-01T10:49:59.530 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:59.530 DEBUG:boto:Port: 8001 2024-02-01T10:49:59.530 DEBUG:boto:Params: {} 2024-02-01T10:49:59.530 DEBUG:boto:Token: None 2024-02-01T10:49:59.531 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:59 GMT /vclfcn-62/ 2024-02-01T10:49:59.531 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7iURGu0oFNa8Tp/DBTI2JjcIu9A= 2024-02-01T10:49:59.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7iURGu0oFNa8Tp/DBTI2JjcIu9A=', 'Content-Length': '0'} 2024-02-01T10:49:59.536 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000075e482d58302d362-0065bb7757-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:59.536 DEBUG:boto:b'vclfcn-62obj51000falseobj52024-02-01T10:49:22.123Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:59.536 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:59.537 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:59.537 DEBUG:boto:path=/vclfcn-62/?prefix=obj5 2024-02-01T10:49:59.537 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj5 2024-02-01T10:49:59.537 DEBUG:boto:Method: GET 2024-02-01T10:49:59.537 DEBUG:boto:Path: /vclfcn-62/?prefix=obj5 2024-02-01T10:49:59.537 DEBUG:boto:Data: 2024-02-01T10:49:59.537 DEBUG:boto:Headers: {} 2024-02-01T10:49:59.537 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:59.537 DEBUG:boto:Port: 8000 2024-02-01T10:49:59.537 DEBUG:boto:Params: {} 2024-02-01T10:49:59.538 DEBUG:boto:Token: None 2024-02-01T10:49:59.538 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:59 GMT /vclfcn-62/ 2024-02-01T10:49:59.538 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7iURGu0oFNa8Tp/DBTI2JjcIu9A= 2024-02-01T10:49:59.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7iURGu0oFNa8Tp/DBTI2JjcIu9A=', 'Content-Length': '0'} 2024-02-01T10:49:59.542 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c759cbd8935fec5-0065bb7757-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:59.542 DEBUG:boto:b'vclfcn-62obj51000falseobj52024-02-01T10:49:22.123Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:59.543 DEBUG:boto:path=/vclfcn-62/obj5 2024-02-01T10:49:59.543 DEBUG:boto:auth_path=/vclfcn-62/obj5 2024-02-01T10:49:59.543 DEBUG:boto:Method: HEAD 2024-02-01T10:49:59.543 DEBUG:boto:Path: /vclfcn-62/obj5 2024-02-01T10:49:59.543 DEBUG:boto:Data: 2024-02-01T10:49:59.543 DEBUG:boto:Headers: {} 2024-02-01T10:49:59.543 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:59.543 DEBUG:boto:Port: 8001 2024-02-01T10:49:59.543 DEBUG:boto:Params: {} 2024-02-01T10:49:59.543 DEBUG:boto:Token: None 2024-02-01T10:49:59.544 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:59 GMT /vclfcn-62/obj5 2024-02-01T10:49:59.544 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lOFb2xTXmuwtFCIAvK/qT/LNs/8= 2024-02-01T10:49:59.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lOFb2xTXmuwtFCIAvK/qT/LNs/8=', 'Content-Length': '0'} 2024-02-01T10:49:59.547 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005c4e0bd91ea11ea6-0065bb7757-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:59.547 DEBUG:boto:path=/vclfcn-62/obj5 2024-02-01T10:49:59.547 DEBUG:boto:auth_path=/vclfcn-62/obj5 2024-02-01T10:49:59.547 DEBUG:boto:Method: HEAD 2024-02-01T10:49:59.548 DEBUG:boto:Path: /vclfcn-62/obj5 2024-02-01T10:49:59.548 DEBUG:boto:Data: 2024-02-01T10:49:59.548 DEBUG:boto:Headers: {} 2024-02-01T10:49:59.548 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:59.549 DEBUG:boto:Port: 8000 2024-02-01T10:49:59.549 DEBUG:boto:Params: {} 2024-02-01T10:49:59.549 DEBUG:boto:Token: None 2024-02-01T10:49:59.549 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:59 GMT /vclfcn-62/obj5 2024-02-01T10:49:59.549 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lOFb2xTXmuwtFCIAvK/qT/LNs/8= 2024-02-01T10:49:59.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lOFb2xTXmuwtFCIAvK/qT/LNs/8=', 'Content-Length': '0'} 2024-02-01T10:49:59.552 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b95d237d3e18461a-0065bb7757-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:59.552 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-02-01T10:49:59.552 DEBUG:boto:path=/vclfcn-62/obj5 2024-02-01T10:49:59.552 DEBUG:boto:auth_path=/vclfcn-62/obj5 2024-02-01T10:49:59.553 DEBUG:boto:Method: GET 2024-02-01T10:49:59.553 DEBUG:boto:Path: /vclfcn-62/obj5 2024-02-01T10:49:59.553 DEBUG:boto:Data: 2024-02-01T10:49:59.553 DEBUG:boto:Headers: {} 2024-02-01T10:49:59.553 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:59.553 DEBUG:boto:Port: 8001 2024-02-01T10:49:59.553 DEBUG:boto:Params: {} 2024-02-01T10:49:59.553 DEBUG:boto:Token: None 2024-02-01T10:49:59.553 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:59 GMT /vclfcn-62/obj5 2024-02-01T10:49:59.553 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:XZkoDKFomdyuIyeNTjftp3R4XHw= 2024-02-01T10:49:59.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:XZkoDKFomdyuIyeNTjftp3R4XHw=', 'Content-Length': '0'} 2024-02-01T10:49:59.557 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000deaa9055a52d2fbf-0065bb7757-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:59.600 DEBUG:boto:path=/vclfcn-62/obj5 2024-02-01T10:49:59.600 DEBUG:boto:auth_path=/vclfcn-62/obj5 2024-02-01T10:49:59.600 DEBUG:boto:Method: GET 2024-02-01T10:49:59.601 DEBUG:boto:Path: /vclfcn-62/obj5 2024-02-01T10:49:59.601 DEBUG:boto:Data: 2024-02-01T10:49:59.601 DEBUG:boto:Headers: {} 2024-02-01T10:49:59.601 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:59.601 DEBUG:boto:Port: 8000 2024-02-01T10:49:59.601 DEBUG:boto:Params: {} 2024-02-01T10:49:59.601 DEBUG:boto:Token: None 2024-02-01T10:49:59.601 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:59 GMT /vclfcn-62/obj5 2024-02-01T10:49:59.601 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:XZkoDKFomdyuIyeNTjftp3R4XHw= 2024-02-01T10:49:59.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:XZkoDKFomdyuIyeNTjftp3R4XHw=', 'Content-Length': '0'} 2024-02-01T10:49:59.604 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000099d34b27168b1799-0065bb7757-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:59.648 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:59.648 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:59.648 DEBUG:boto:path=/vclfcn-62/?prefix=obj5 2024-02-01T10:49:59.648 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj5 2024-02-01T10:49:59.649 DEBUG:boto:Method: GET 2024-02-01T10:49:59.649 DEBUG:boto:Path: /vclfcn-62/?prefix=obj5 2024-02-01T10:49:59.649 DEBUG:boto:Data: 2024-02-01T10:49:59.649 DEBUG:boto:Headers: {} 2024-02-01T10:49:59.649 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:59.649 DEBUG:boto:Port: 8001 2024-02-01T10:49:59.649 DEBUG:boto:Params: {} 2024-02-01T10:49:59.649 DEBUG:boto:Token: None 2024-02-01T10:49:59.649 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:59 GMT /vclfcn-62/ 2024-02-01T10:49:59.650 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7iURGu0oFNa8Tp/DBTI2JjcIu9A= 2024-02-01T10:49:59.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7iURGu0oFNa8Tp/DBTI2JjcIu9A=', 'Content-Length': '0'} 2024-02-01T10:49:59.654 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000037a0c0bf571e4af-0065bb7757-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:59.654 DEBUG:boto:b'vclfcn-62obj51000falseobj52024-02-01T10:49:22.123Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:59.655 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:59.655 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:59.655 DEBUG:boto:path=/vclfcn-62/?prefix=obj5 2024-02-01T10:49:59.655 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj5 2024-02-01T10:49:59.655 DEBUG:boto:Method: GET 2024-02-01T10:49:59.655 DEBUG:boto:Path: /vclfcn-62/?prefix=obj5 2024-02-01T10:49:59.655 DEBUG:boto:Data: 2024-02-01T10:49:59.655 DEBUG:boto:Headers: {} 2024-02-01T10:49:59.655 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:59.655 DEBUG:boto:Port: 8000 2024-02-01T10:49:59.656 DEBUG:boto:Params: {} 2024-02-01T10:49:59.656 DEBUG:boto:Token: None 2024-02-01T10:49:59.656 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:59 GMT /vclfcn-62/ 2024-02-01T10:49:59.656 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7iURGu0oFNa8Tp/DBTI2JjcIu9A= 2024-02-01T10:49:59.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7iURGu0oFNa8Tp/DBTI2JjcIu9A=', 'Content-Length': '0'} 2024-02-01T10:49:59.661 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001729e0a1200daaf2-0065bb7757-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:59.704 DEBUG:boto:b'vclfcn-62obj51000falseobj52024-02-01T10:49:22.123Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:59.704 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-02-01T10:49:59.705 DEBUG:boto:path=/vclfcn-62/obj6 2024-02-01T10:49:59.705 DEBUG:boto:auth_path=/vclfcn-62/obj6 2024-02-01T10:49:59.705 DEBUG:boto:path=/vclfcn-62/obj6?versionId=null 2024-02-01T10:49:59.705 DEBUG:boto:auth_path=/vclfcn-62/obj6?versionId=null 2024-02-01T10:49:59.705 DEBUG:boto:Method: GET 2024-02-01T10:49:59.705 DEBUG:boto:Path: /vclfcn-62/obj6?versionId=null 2024-02-01T10:49:59.705 DEBUG:boto:Data: 2024-02-01T10:49:59.705 DEBUG:boto:Headers: {} 2024-02-01T10:49:59.705 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:59.705 DEBUG:boto:Port: 8001 2024-02-01T10:49:59.706 DEBUG:boto:Params: {} 2024-02-01T10:49:59.706 DEBUG:boto:Token: None 2024-02-01T10:49:59.706 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:59 GMT /vclfcn-62/obj6?versionId=null 2024-02-01T10:49:59.706 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:SIEb0XQq7KI7yyTE9GZyuOpC/X4= 2024-02-01T10:49:59.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:SIEb0XQq7KI7yyTE9GZyuOpC/X4=', 'Content-Length': '0'} 2024-02-01T10:49:59.709 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000096c029aee8e2ffc2-0065bb7757-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:59.710 DEBUG:boto:path=/vclfcn-62/obj6 2024-02-01T10:49:59.710 DEBUG:boto:auth_path=/vclfcn-62/obj6 2024-02-01T10:49:59.710 DEBUG:boto:path=/vclfcn-62/obj6?versionId=null 2024-02-01T10:49:59.710 DEBUG:boto:auth_path=/vclfcn-62/obj6?versionId=null 2024-02-01T10:49:59.710 DEBUG:boto:Method: GET 2024-02-01T10:49:59.710 DEBUG:boto:Path: /vclfcn-62/obj6?versionId=null 2024-02-01T10:49:59.710 DEBUG:boto:Data: 2024-02-01T10:49:59.710 DEBUG:boto:Headers: {} 2024-02-01T10:49:59.710 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:59.710 DEBUG:boto:Port: 8000 2024-02-01T10:49:59.710 DEBUG:boto:Params: {} 2024-02-01T10:49:59.711 DEBUG:boto:Token: None 2024-02-01T10:49:59.711 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:59 GMT /vclfcn-62/obj6?versionId=null 2024-02-01T10:49:59.711 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:SIEb0XQq7KI7yyTE9GZyuOpC/X4= 2024-02-01T10:49:59.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:SIEb0XQq7KI7yyTE9GZyuOpC/X4=', 'Content-Length': '0'} 2024-02-01T10:49:59.714 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cb58aac02704fc91-0065bb7757-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:59.756 DEBUG:boto:path=/vclfcn-62/obj6 2024-02-01T10:49:59.756 DEBUG:boto:auth_path=/vclfcn-62/obj6 2024-02-01T10:49:59.756 DEBUG:boto:path=/vclfcn-62/obj6?versionId=null 2024-02-01T10:49:59.757 DEBUG:boto:auth_path=/vclfcn-62/obj6?versionId=null 2024-02-01T10:49:59.757 DEBUG:boto:Method: HEAD 2024-02-01T10:49:59.757 DEBUG:boto:Path: /vclfcn-62/obj6?versionId=null 2024-02-01T10:49:59.757 DEBUG:boto:Data: 2024-02-01T10:49:59.757 DEBUG:boto:Headers: {} 2024-02-01T10:49:59.757 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:59.757 DEBUG:boto:Port: 8001 2024-02-01T10:49:59.757 DEBUG:boto:Params: {} 2024-02-01T10:49:59.757 DEBUG:boto:Token: None 2024-02-01T10:49:59.757 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:59 GMT /vclfcn-62/obj6?versionId=null 2024-02-01T10:49:59.758 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ya1ywEylDdX3GbrQkdP53DWrpp8= 2024-02-01T10:49:59.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ya1ywEylDdX3GbrQkdP53DWrpp8=', 'Content-Length': '0'} 2024-02-01T10:49:59.761 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000003a5cbc865348533-0065bb7757-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:59.761 DEBUG:boto:path=/vclfcn-62/obj6 2024-02-01T10:49:59.761 DEBUG:boto:auth_path=/vclfcn-62/obj6 2024-02-01T10:49:59.761 DEBUG:boto:path=/vclfcn-62/obj6?versionId=null 2024-02-01T10:49:59.761 DEBUG:boto:auth_path=/vclfcn-62/obj6?versionId=null 2024-02-01T10:49:59.761 DEBUG:boto:Method: HEAD 2024-02-01T10:49:59.761 DEBUG:boto:Path: /vclfcn-62/obj6?versionId=null 2024-02-01T10:49:59.761 DEBUG:boto:Data: 2024-02-01T10:49:59.762 DEBUG:boto:Headers: {} 2024-02-01T10:49:59.762 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:59.762 DEBUG:boto:Port: 8000 2024-02-01T10:49:59.762 DEBUG:boto:Params: {} 2024-02-01T10:49:59.762 DEBUG:boto:Token: None 2024-02-01T10:49:59.762 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:59 GMT /vclfcn-62/obj6?versionId=null 2024-02-01T10:49:59.762 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ya1ywEylDdX3GbrQkdP53DWrpp8= 2024-02-01T10:49:59.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ya1ywEylDdX3GbrQkdP53DWrpp8=', 'Content-Length': '0'} 2024-02-01T10:49:59.765 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009ac942e5bb6b72cc-0065bb7757-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:59.765 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-02-01T10:49:59.765 DEBUG:boto:path=/vclfcn-62/obj6 2024-02-01T10:49:59.765 DEBUG:boto:auth_path=/vclfcn-62/obj6 2024-02-01T10:49:59.765 DEBUG:boto:path=/vclfcn-62/obj6?versionId=null 2024-02-01T10:49:59.766 DEBUG:boto:auth_path=/vclfcn-62/obj6?versionId=null 2024-02-01T10:49:59.766 DEBUG:boto:Method: GET 2024-02-01T10:49:59.766 DEBUG:boto:Path: /vclfcn-62/obj6?versionId=null 2024-02-01T10:49:59.766 DEBUG:boto:Data: 2024-02-01T10:49:59.766 DEBUG:boto:Headers: {} 2024-02-01T10:49:59.766 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:59.766 DEBUG:boto:Port: 8001 2024-02-01T10:49:59.766 DEBUG:boto:Params: {} 2024-02-01T10:49:59.766 DEBUG:boto:Token: None 2024-02-01T10:49:59.766 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:59 GMT /vclfcn-62/obj6?versionId=null 2024-02-01T10:49:59.767 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:SIEb0XQq7KI7yyTE9GZyuOpC/X4= 2024-02-01T10:49:59.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:SIEb0XQq7KI7yyTE9GZyuOpC/X4=', 'Content-Length': '0'} 2024-02-01T10:49:59.770 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fffdd85265169810-0065bb7757-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:59.812 DEBUG:boto:path=/vclfcn-62/obj6 2024-02-01T10:49:59.812 DEBUG:boto:auth_path=/vclfcn-62/obj6 2024-02-01T10:49:59.812 DEBUG:boto:path=/vclfcn-62/obj6?versionId=null 2024-02-01T10:49:59.812 DEBUG:boto:auth_path=/vclfcn-62/obj6?versionId=null 2024-02-01T10:49:59.813 DEBUG:boto:Method: GET 2024-02-01T10:49:59.813 DEBUG:boto:Path: /vclfcn-62/obj6?versionId=null 2024-02-01T10:49:59.813 DEBUG:boto:Data: 2024-02-01T10:49:59.813 DEBUG:boto:Headers: {} 2024-02-01T10:49:59.813 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:59.813 DEBUG:boto:Port: 8000 2024-02-01T10:49:59.813 DEBUG:boto:Params: {} 2024-02-01T10:49:59.813 DEBUG:boto:Token: None 2024-02-01T10:49:59.813 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:59 GMT /vclfcn-62/obj6?versionId=null 2024-02-01T10:49:59.813 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:SIEb0XQq7KI7yyTE9GZyuOpC/X4= 2024-02-01T10:49:59.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:SIEb0XQq7KI7yyTE9GZyuOpC/X4=', 'Content-Length': '0'} 2024-02-01T10:49:59.816 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003026dd70ce8d8a5a-0065bb7757-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:59.860 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:59.861 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:59.861 DEBUG:boto:path=/vclfcn-62/?prefix=obj6 2024-02-01T10:49:59.861 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj6 2024-02-01T10:49:59.861 DEBUG:boto:Method: GET 2024-02-01T10:49:59.861 DEBUG:boto:Path: /vclfcn-62/?prefix=obj6 2024-02-01T10:49:59.861 DEBUG:boto:Data: 2024-02-01T10:49:59.861 DEBUG:boto:Headers: {} 2024-02-01T10:49:59.861 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:59.861 DEBUG:boto:Port: 8001 2024-02-01T10:49:59.861 DEBUG:boto:Params: {} 2024-02-01T10:49:59.862 DEBUG:boto:Token: None 2024-02-01T10:49:59.862 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:59 GMT /vclfcn-62/ 2024-02-01T10:49:59.862 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7iURGu0oFNa8Tp/DBTI2JjcIu9A= 2024-02-01T10:49:59.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7iURGu0oFNa8Tp/DBTI2JjcIu9A=', 'Content-Length': '0'} 2024-02-01T10:49:59.866 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000922629c0c668363e-0065bb7757-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:59.866 DEBUG:boto:b'vclfcn-62obj61000falseobj62024-02-01T10:49:22.135Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:59.866 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:59.867 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:59.867 DEBUG:boto:path=/vclfcn-62/?prefix=obj6 2024-02-01T10:49:59.867 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj6 2024-02-01T10:49:59.867 DEBUG:boto:Method: GET 2024-02-01T10:49:59.867 DEBUG:boto:Path: /vclfcn-62/?prefix=obj6 2024-02-01T10:49:59.867 DEBUG:boto:Data: 2024-02-01T10:49:59.867 DEBUG:boto:Headers: {} 2024-02-01T10:49:59.867 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:59.867 DEBUG:boto:Port: 8000 2024-02-01T10:49:59.867 DEBUG:boto:Params: {} 2024-02-01T10:49:59.867 DEBUG:boto:Token: None 2024-02-01T10:49:59.868 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:59 GMT /vclfcn-62/ 2024-02-01T10:49:59.868 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7iURGu0oFNa8Tp/DBTI2JjcIu9A= 2024-02-01T10:49:59.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7iURGu0oFNa8Tp/DBTI2JjcIu9A=', 'Content-Length': '0'} 2024-02-01T10:49:59.872 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e00264fbfa6638c-0065bb7757-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:59.872 DEBUG:boto:b'vclfcn-62obj61000falseobj62024-02-01T10:49:22.135Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:49:59.873 DEBUG:boto:path=/vclfcn-62/obj6 2024-02-01T10:49:59.873 DEBUG:boto:auth_path=/vclfcn-62/obj6 2024-02-01T10:49:59.873 DEBUG:boto:Method: HEAD 2024-02-01T10:49:59.873 DEBUG:boto:Path: /vclfcn-62/obj6 2024-02-01T10:49:59.873 DEBUG:boto:Data: 2024-02-01T10:49:59.873 DEBUG:boto:Headers: {} 2024-02-01T10:49:59.873 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:59.874 DEBUG:boto:Port: 8001 2024-02-01T10:49:59.874 DEBUG:boto:Params: {} 2024-02-01T10:49:59.874 DEBUG:boto:Token: None 2024-02-01T10:49:59.874 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:59 GMT /vclfcn-62/obj6 2024-02-01T10:49:59.874 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:2oKsb70pjlP1rg206IcYbMSRMjw= 2024-02-01T10:49:59.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2oKsb70pjlP1rg206IcYbMSRMjw=', 'Content-Length': '0'} 2024-02-01T10:49:59.877 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000894b850854e9e4ae-0065bb7757-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:59.877 DEBUG:boto:path=/vclfcn-62/obj6 2024-02-01T10:49:59.877 DEBUG:boto:auth_path=/vclfcn-62/obj6 2024-02-01T10:49:59.878 DEBUG:boto:Method: HEAD 2024-02-01T10:49:59.878 DEBUG:boto:Path: /vclfcn-62/obj6 2024-02-01T10:49:59.878 DEBUG:boto:Data: 2024-02-01T10:49:59.878 DEBUG:boto:Headers: {} 2024-02-01T10:49:59.878 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:59.878 DEBUG:boto:Port: 8000 2024-02-01T10:49:59.878 DEBUG:boto:Params: {} 2024-02-01T10:49:59.878 DEBUG:boto:Token: None 2024-02-01T10:49:59.878 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:49:59 GMT /vclfcn-62/obj6 2024-02-01T10:49:59.878 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:2oKsb70pjlP1rg206IcYbMSRMjw= 2024-02-01T10:49:59.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2oKsb70pjlP1rg206IcYbMSRMjw=', 'Content-Length': '0'} 2024-02-01T10:49:59.881 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000054d27adc39b7b6d7-0065bb7757-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:59.882 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-02-01T10:49:59.882 DEBUG:boto:path=/vclfcn-62/obj6 2024-02-01T10:49:59.882 DEBUG:boto:auth_path=/vclfcn-62/obj6 2024-02-01T10:49:59.882 DEBUG:boto:Method: GET 2024-02-01T10:49:59.882 DEBUG:boto:Path: /vclfcn-62/obj6 2024-02-01T10:49:59.882 DEBUG:boto:Data: 2024-02-01T10:49:59.883 DEBUG:boto:Headers: {} 2024-02-01T10:49:59.883 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:59.883 DEBUG:boto:Port: 8001 2024-02-01T10:49:59.883 DEBUG:boto:Params: {} 2024-02-01T10:49:59.883 DEBUG:boto:Token: None 2024-02-01T10:49:59.883 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:59 GMT /vclfcn-62/obj6 2024-02-01T10:49:59.883 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vBUty1knKPxWezRvkPjDVcvbibU= 2024-02-01T10:49:59.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vBUty1knKPxWezRvkPjDVcvbibU=', 'Content-Length': '0'} 2024-02-01T10:49:59.886 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000071c19c02f4ce5ecb-0065bb7757-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:59.928 DEBUG:boto:path=/vclfcn-62/obj6 2024-02-01T10:49:59.928 DEBUG:boto:auth_path=/vclfcn-62/obj6 2024-02-01T10:49:59.928 DEBUG:boto:Method: GET 2024-02-01T10:49:59.929 DEBUG:boto:Path: /vclfcn-62/obj6 2024-02-01T10:49:59.929 DEBUG:boto:Data: 2024-02-01T10:49:59.929 DEBUG:boto:Headers: {} 2024-02-01T10:49:59.929 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:49:59.929 DEBUG:boto:Port: 8000 2024-02-01T10:49:59.929 DEBUG:boto:Params: {} 2024-02-01T10:49:59.929 DEBUG:boto:Token: None 2024-02-01T10:49:59.929 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:59 GMT /vclfcn-62/obj6 2024-02-01T10:49:59.929 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vBUty1knKPxWezRvkPjDVcvbibU= 2024-02-01T10:49:59.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vBUty1knKPxWezRvkPjDVcvbibU=', 'Content-Length': '0'} 2024-02-01T10:49:59.932 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d6febc1a7fe75c2d-0065bb7757-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:49:59.976 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:49:59.976 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:49:59.976 DEBUG:boto:path=/vclfcn-62/?prefix=obj6 2024-02-01T10:49:59.976 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj6 2024-02-01T10:49:59.977 DEBUG:boto:Method: GET 2024-02-01T10:49:59.977 DEBUG:boto:Path: /vclfcn-62/?prefix=obj6 2024-02-01T10:49:59.977 DEBUG:boto:Data: 2024-02-01T10:49:59.977 DEBUG:boto:Headers: {} 2024-02-01T10:49:59.977 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:49:59.977 DEBUG:boto:Port: 8001 2024-02-01T10:49:59.977 DEBUG:boto:Params: {} 2024-02-01T10:49:59.977 DEBUG:boto:Token: None 2024-02-01T10:49:59.977 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:49:59 GMT /vclfcn-62/ 2024-02-01T10:49:59.977 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7iURGu0oFNa8Tp/DBTI2JjcIu9A= 2024-02-01T10:49:59.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:49:59 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7iURGu0oFNa8Tp/DBTI2JjcIu9A=', 'Content-Length': '0'} 2024-02-01T10:49:59.982 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ae80fea473a5901-0065bb7757-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:49:59 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:00.024 DEBUG:boto:b'vclfcn-62obj61000falseobj62024-02-01T10:49:22.135Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:00.024 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:50:00.025 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:50:00.025 DEBUG:boto:path=/vclfcn-62/?prefix=obj6 2024-02-01T10:50:00.025 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj6 2024-02-01T10:50:00.025 DEBUG:boto:Method: GET 2024-02-01T10:50:00.025 DEBUG:boto:Path: /vclfcn-62/?prefix=obj6 2024-02-01T10:50:00.025 DEBUG:boto:Data: 2024-02-01T10:50:00.025 DEBUG:boto:Headers: {} 2024-02-01T10:50:00.025 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:00.025 DEBUG:boto:Port: 8000 2024-02-01T10:50:00.025 DEBUG:boto:Params: {} 2024-02-01T10:50:00.026 DEBUG:boto:Token: None 2024-02-01T10:50:00.026 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:00 GMT /vclfcn-62/ 2024-02-01T10:50:00.026 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:26jso5R2vPYsHxTEVV9idUwWsNw= 2024-02-01T10:50:00.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:26jso5R2vPYsHxTEVV9idUwWsNw=', 'Content-Length': '0'} 2024-02-01T10:50:00.031 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af095fa264a88d34-0065bb7758-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:00.076 DEBUG:boto:b'vclfcn-62obj61000falseobj62024-02-01T10:49:22.135Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:00.076 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-02-01T10:50:00.077 DEBUG:boto:path=/vclfcn-62/obj7 2024-02-01T10:50:00.077 DEBUG:boto:auth_path=/vclfcn-62/obj7 2024-02-01T10:50:00.077 DEBUG:boto:path=/vclfcn-62/obj7?versionId=null 2024-02-01T10:50:00.077 DEBUG:boto:auth_path=/vclfcn-62/obj7?versionId=null 2024-02-01T10:50:00.077 DEBUG:boto:Method: GET 2024-02-01T10:50:00.077 DEBUG:boto:Path: /vclfcn-62/obj7?versionId=null 2024-02-01T10:50:00.077 DEBUG:boto:Data: 2024-02-01T10:50:00.077 DEBUG:boto:Headers: {} 2024-02-01T10:50:00.077 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:00.077 DEBUG:boto:Port: 8001 2024-02-01T10:50:00.078 DEBUG:boto:Params: {} 2024-02-01T10:50:00.078 DEBUG:boto:Token: None 2024-02-01T10:50:00.078 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:00 GMT /vclfcn-62/obj7?versionId=null 2024-02-01T10:50:00.078 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nOd2CLsSGf2uH9eE3UlphJo5hms= 2024-02-01T10:50:00.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nOd2CLsSGf2uH9eE3UlphJo5hms=', 'Content-Length': '0'} 2024-02-01T10:50:00.081 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000051df8f3e10ed7c7f-0065bb7758-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:00.081 DEBUG:boto:path=/vclfcn-62/obj7 2024-02-01T10:50:00.081 DEBUG:boto:auth_path=/vclfcn-62/obj7 2024-02-01T10:50:00.082 DEBUG:boto:path=/vclfcn-62/obj7?versionId=null 2024-02-01T10:50:00.082 DEBUG:boto:auth_path=/vclfcn-62/obj7?versionId=null 2024-02-01T10:50:00.082 DEBUG:boto:Method: GET 2024-02-01T10:50:00.082 DEBUG:boto:Path: /vclfcn-62/obj7?versionId=null 2024-02-01T10:50:00.082 DEBUG:boto:Data: 2024-02-01T10:50:00.082 DEBUG:boto:Headers: {} 2024-02-01T10:50:00.082 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:00.082 DEBUG:boto:Port: 8000 2024-02-01T10:50:00.082 DEBUG:boto:Params: {} 2024-02-01T10:50:00.082 DEBUG:boto:Token: None 2024-02-01T10:50:00.083 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:00 GMT /vclfcn-62/obj7?versionId=null 2024-02-01T10:50:00.083 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nOd2CLsSGf2uH9eE3UlphJo5hms= 2024-02-01T10:50:00.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nOd2CLsSGf2uH9eE3UlphJo5hms=', 'Content-Length': '0'} 2024-02-01T10:50:00.086 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bc2e6014a0955349-0065bb7758-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:00.132 DEBUG:boto:path=/vclfcn-62/obj7 2024-02-01T10:50:00.132 DEBUG:boto:auth_path=/vclfcn-62/obj7 2024-02-01T10:50:00.132 DEBUG:boto:path=/vclfcn-62/obj7?versionId=null 2024-02-01T10:50:00.132 DEBUG:boto:auth_path=/vclfcn-62/obj7?versionId=null 2024-02-01T10:50:00.133 DEBUG:boto:Method: HEAD 2024-02-01T10:50:00.133 DEBUG:boto:Path: /vclfcn-62/obj7?versionId=null 2024-02-01T10:50:00.133 DEBUG:boto:Data: 2024-02-01T10:50:00.133 DEBUG:boto:Headers: {} 2024-02-01T10:50:00.133 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:00.133 DEBUG:boto:Port: 8001 2024-02-01T10:50:00.133 DEBUG:boto:Params: {} 2024-02-01T10:50:00.133 DEBUG:boto:Token: None 2024-02-01T10:50:00.133 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:00 GMT /vclfcn-62/obj7?versionId=null 2024-02-01T10:50:00.134 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:CehLFcC/ihmLTSSSJntCKREZ6v8= 2024-02-01T10:50:00.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:CehLFcC/ihmLTSSSJntCKREZ6v8=', 'Content-Length': '0'} 2024-02-01T10:50:00.136 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000044a1f5aab78a5b5-0065bb7758-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:00.136 DEBUG:boto:path=/vclfcn-62/obj7 2024-02-01T10:50:00.136 DEBUG:boto:auth_path=/vclfcn-62/obj7 2024-02-01T10:50:00.136 DEBUG:boto:path=/vclfcn-62/obj7?versionId=null 2024-02-01T10:50:00.137 DEBUG:boto:auth_path=/vclfcn-62/obj7?versionId=null 2024-02-01T10:50:00.137 DEBUG:boto:Method: HEAD 2024-02-01T10:50:00.137 DEBUG:boto:Path: /vclfcn-62/obj7?versionId=null 2024-02-01T10:50:00.137 DEBUG:boto:Data: 2024-02-01T10:50:00.137 DEBUG:boto:Headers: {} 2024-02-01T10:50:00.137 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:00.137 DEBUG:boto:Port: 8000 2024-02-01T10:50:00.137 DEBUG:boto:Params: {} 2024-02-01T10:50:00.137 DEBUG:boto:Token: None 2024-02-01T10:50:00.137 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:00 GMT /vclfcn-62/obj7?versionId=null 2024-02-01T10:50:00.138 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:CehLFcC/ihmLTSSSJntCKREZ6v8= 2024-02-01T10:50:00.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:CehLFcC/ihmLTSSSJntCKREZ6v8=', 'Content-Length': '0'} 2024-02-01T10:50:00.140 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000219ec3a8f6046dfc-0065bb7758-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:00.141 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-02-01T10:50:00.141 DEBUG:boto:path=/vclfcn-62/obj7 2024-02-01T10:50:00.141 DEBUG:boto:auth_path=/vclfcn-62/obj7 2024-02-01T10:50:00.141 DEBUG:boto:path=/vclfcn-62/obj7?versionId=null 2024-02-01T10:50:00.141 DEBUG:boto:auth_path=/vclfcn-62/obj7?versionId=null 2024-02-01T10:50:00.141 DEBUG:boto:Method: GET 2024-02-01T10:50:00.141 DEBUG:boto:Path: /vclfcn-62/obj7?versionId=null 2024-02-01T10:50:00.141 DEBUG:boto:Data: 2024-02-01T10:50:00.141 DEBUG:boto:Headers: {} 2024-02-01T10:50:00.141 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:00.142 DEBUG:boto:Port: 8001 2024-02-01T10:50:00.142 DEBUG:boto:Params: {} 2024-02-01T10:50:00.142 DEBUG:boto:Token: None 2024-02-01T10:50:00.142 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:00 GMT /vclfcn-62/obj7?versionId=null 2024-02-01T10:50:00.142 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nOd2CLsSGf2uH9eE3UlphJo5hms= 2024-02-01T10:50:00.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nOd2CLsSGf2uH9eE3UlphJo5hms=', 'Content-Length': '0'} 2024-02-01T10:50:00.145 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005220248b2fff0c03-0065bb7758-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:00.188 DEBUG:boto:path=/vclfcn-62/obj7 2024-02-01T10:50:00.188 DEBUG:boto:auth_path=/vclfcn-62/obj7 2024-02-01T10:50:00.188 DEBUG:boto:path=/vclfcn-62/obj7?versionId=null 2024-02-01T10:50:00.188 DEBUG:boto:auth_path=/vclfcn-62/obj7?versionId=null 2024-02-01T10:50:00.189 DEBUG:boto:Method: GET 2024-02-01T10:50:00.189 DEBUG:boto:Path: /vclfcn-62/obj7?versionId=null 2024-02-01T10:50:00.189 DEBUG:boto:Data: 2024-02-01T10:50:00.189 DEBUG:boto:Headers: {} 2024-02-01T10:50:00.189 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:00.189 DEBUG:boto:Port: 8000 2024-02-01T10:50:00.190 DEBUG:boto:Params: {} 2024-02-01T10:50:00.190 DEBUG:boto:Token: None 2024-02-01T10:50:00.190 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:00 GMT /vclfcn-62/obj7?versionId=null 2024-02-01T10:50:00.190 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nOd2CLsSGf2uH9eE3UlphJo5hms= 2024-02-01T10:50:00.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nOd2CLsSGf2uH9eE3UlphJo5hms=', 'Content-Length': '0'} 2024-02-01T10:50:00.193 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cdf07d17978a9305-0065bb7758-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:00.240 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:50:00.245 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:50:00.245 DEBUG:boto:path=/vclfcn-62/?prefix=obj7 2024-02-01T10:50:00.245 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj7 2024-02-01T10:50:00.245 DEBUG:boto:Method: GET 2024-02-01T10:50:00.245 DEBUG:boto:Path: /vclfcn-62/?prefix=obj7 2024-02-01T10:50:00.245 DEBUG:boto:Data: 2024-02-01T10:50:00.245 DEBUG:boto:Headers: {} 2024-02-01T10:50:00.245 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:00.245 DEBUG:boto:Port: 8001 2024-02-01T10:50:00.246 DEBUG:boto:Params: {} 2024-02-01T10:50:00.246 DEBUG:boto:Token: None 2024-02-01T10:50:00.246 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:00 GMT /vclfcn-62/ 2024-02-01T10:50:00.246 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:26jso5R2vPYsHxTEVV9idUwWsNw= 2024-02-01T10:50:00.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:26jso5R2vPYsHxTEVV9idUwWsNw=', 'Content-Length': '0'} 2024-02-01T10:50:00.251 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004b07de1ab6f0e24-0065bb7758-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:00.252 DEBUG:boto:b'vclfcn-62obj71000falseobj72024-02-01T10:49:22.186Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:00.252 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:50:00.252 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:50:00.252 DEBUG:boto:path=/vclfcn-62/?prefix=obj7 2024-02-01T10:50:00.252 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj7 2024-02-01T10:50:00.253 DEBUG:boto:Method: GET 2024-02-01T10:50:00.253 DEBUG:boto:Path: /vclfcn-62/?prefix=obj7 2024-02-01T10:50:00.253 DEBUG:boto:Data: 2024-02-01T10:50:00.253 DEBUG:boto:Headers: {} 2024-02-01T10:50:00.253 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:00.253 DEBUG:boto:Port: 8000 2024-02-01T10:50:00.253 DEBUG:boto:Params: {} 2024-02-01T10:50:00.253 DEBUG:boto:Token: None 2024-02-01T10:50:00.253 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:00 GMT /vclfcn-62/ 2024-02-01T10:50:00.253 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:26jso5R2vPYsHxTEVV9idUwWsNw= 2024-02-01T10:50:00.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:26jso5R2vPYsHxTEVV9idUwWsNw=', 'Content-Length': '0'} 2024-02-01T10:50:00.258 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a0923174b9939b11-0065bb7758-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:00.258 DEBUG:boto:b'vclfcn-62obj71000falseobj72024-02-01T10:49:22.186Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:00.259 DEBUG:boto:path=/vclfcn-62/obj7 2024-02-01T10:50:00.259 DEBUG:boto:auth_path=/vclfcn-62/obj7 2024-02-01T10:50:00.259 DEBUG:boto:Method: HEAD 2024-02-01T10:50:00.259 DEBUG:boto:Path: /vclfcn-62/obj7 2024-02-01T10:50:00.259 DEBUG:boto:Data: 2024-02-01T10:50:00.259 DEBUG:boto:Headers: {} 2024-02-01T10:50:00.259 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:00.260 DEBUG:boto:Port: 8001 2024-02-01T10:50:00.260 DEBUG:boto:Params: {} 2024-02-01T10:50:00.260 DEBUG:boto:Token: None 2024-02-01T10:50:00.260 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:00 GMT /vclfcn-62/obj7 2024-02-01T10:50:00.260 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:LXST3k+SuFYgCGNQPjpqiMEKFSk= 2024-02-01T10:50:00.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:LXST3k+SuFYgCGNQPjpqiMEKFSk=', 'Content-Length': '0'} 2024-02-01T10:50:00.263 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e2a24f4b918248ed-0065bb7758-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:00.263 DEBUG:boto:path=/vclfcn-62/obj7 2024-02-01T10:50:00.263 DEBUG:boto:auth_path=/vclfcn-62/obj7 2024-02-01T10:50:00.263 DEBUG:boto:Method: HEAD 2024-02-01T10:50:00.264 DEBUG:boto:Path: /vclfcn-62/obj7 2024-02-01T10:50:00.264 DEBUG:boto:Data: 2024-02-01T10:50:00.264 DEBUG:boto:Headers: {} 2024-02-01T10:50:00.264 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:00.264 DEBUG:boto:Port: 8000 2024-02-01T10:50:00.264 DEBUG:boto:Params: {} 2024-02-01T10:50:00.264 DEBUG:boto:Token: None 2024-02-01T10:50:00.264 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:00 GMT /vclfcn-62/obj7 2024-02-01T10:50:00.264 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:LXST3k+SuFYgCGNQPjpqiMEKFSk= 2024-02-01T10:50:00.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:LXST3k+SuFYgCGNQPjpqiMEKFSk=', 'Content-Length': '0'} 2024-02-01T10:50:00.268 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000079041dd316755c5d-0065bb7758-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:00.268 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-02-01T10:50:00.269 DEBUG:boto:path=/vclfcn-62/obj7 2024-02-01T10:50:00.269 DEBUG:boto:auth_path=/vclfcn-62/obj7 2024-02-01T10:50:00.269 DEBUG:boto:Method: GET 2024-02-01T10:50:00.269 DEBUG:boto:Path: /vclfcn-62/obj7 2024-02-01T10:50:00.269 DEBUG:boto:Data: 2024-02-01T10:50:00.269 DEBUG:boto:Headers: {} 2024-02-01T10:50:00.269 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:00.269 DEBUG:boto:Port: 8001 2024-02-01T10:50:00.269 DEBUG:boto:Params: {} 2024-02-01T10:50:00.270 DEBUG:boto:Token: None 2024-02-01T10:50:00.270 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:00 GMT /vclfcn-62/obj7 2024-02-01T10:50:00.270 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wqgEmA08gCL5Y2fTyTNR8b69BFg= 2024-02-01T10:50:00.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wqgEmA08gCL5Y2fTyTNR8b69BFg=', 'Content-Length': '0'} 2024-02-01T10:50:00.273 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009b7afd23e119036e-0065bb7758-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:00.316 DEBUG:boto:path=/vclfcn-62/obj7 2024-02-01T10:50:00.316 DEBUG:boto:auth_path=/vclfcn-62/obj7 2024-02-01T10:50:00.316 DEBUG:boto:Method: GET 2024-02-01T10:50:00.316 DEBUG:boto:Path: /vclfcn-62/obj7 2024-02-01T10:50:00.317 DEBUG:boto:Data: 2024-02-01T10:50:00.317 DEBUG:boto:Headers: {} 2024-02-01T10:50:00.317 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:00.317 DEBUG:boto:Port: 8000 2024-02-01T10:50:00.317 DEBUG:boto:Params: {} 2024-02-01T10:50:00.317 DEBUG:boto:Token: None 2024-02-01T10:50:00.317 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:00 GMT /vclfcn-62/obj7 2024-02-01T10:50:00.317 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wqgEmA08gCL5Y2fTyTNR8b69BFg= 2024-02-01T10:50:00.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wqgEmA08gCL5Y2fTyTNR8b69BFg=', 'Content-Length': '0'} 2024-02-01T10:50:00.320 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f79d16e97625ec1e-0065bb7758-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:00.364 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:50:00.364 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:50:00.364 DEBUG:boto:path=/vclfcn-62/?prefix=obj7 2024-02-01T10:50:00.364 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj7 2024-02-01T10:50:00.365 DEBUG:boto:Method: GET 2024-02-01T10:50:00.365 DEBUG:boto:Path: /vclfcn-62/?prefix=obj7 2024-02-01T10:50:00.365 DEBUG:boto:Data: 2024-02-01T10:50:00.365 DEBUG:boto:Headers: {} 2024-02-01T10:50:00.365 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:00.365 DEBUG:boto:Port: 8001 2024-02-01T10:50:00.365 DEBUG:boto:Params: {} 2024-02-01T10:50:00.365 DEBUG:boto:Token: None 2024-02-01T10:50:00.366 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:00 GMT /vclfcn-62/ 2024-02-01T10:50:00.366 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:26jso5R2vPYsHxTEVV9idUwWsNw= 2024-02-01T10:50:00.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:26jso5R2vPYsHxTEVV9idUwWsNw=', 'Content-Length': '0'} 2024-02-01T10:50:00.370 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1badd9b5f6d12e2-0065bb7758-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:00.370 DEBUG:boto:b'vclfcn-62obj71000falseobj72024-02-01T10:49:22.186Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:00.371 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:50:00.371 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:50:00.371 DEBUG:boto:path=/vclfcn-62/?prefix=obj7 2024-02-01T10:50:00.371 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj7 2024-02-01T10:50:00.371 DEBUG:boto:Method: GET 2024-02-01T10:50:00.371 DEBUG:boto:Path: /vclfcn-62/?prefix=obj7 2024-02-01T10:50:00.371 DEBUG:boto:Data: 2024-02-01T10:50:00.371 DEBUG:boto:Headers: {} 2024-02-01T10:50:00.371 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:00.371 DEBUG:boto:Port: 8000 2024-02-01T10:50:00.371 DEBUG:boto:Params: {} 2024-02-01T10:50:00.372 DEBUG:boto:Token: None 2024-02-01T10:50:00.372 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:00 GMT /vclfcn-62/ 2024-02-01T10:50:00.372 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:26jso5R2vPYsHxTEVV9idUwWsNw= 2024-02-01T10:50:00.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:26jso5R2vPYsHxTEVV9idUwWsNw=', 'Content-Length': '0'} 2024-02-01T10:50:00.377 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000adb00eb0cc6b9183-0065bb7758-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:00.420 DEBUG:boto:b'vclfcn-62obj71000falseobj72024-02-01T10:49:22.186Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:00.420 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-02-01T10:50:00.421 DEBUG:boto:path=/vclfcn-62/obj8 2024-02-01T10:50:00.421 DEBUG:boto:auth_path=/vclfcn-62/obj8 2024-02-01T10:50:00.421 DEBUG:boto:path=/vclfcn-62/obj8?versionId=null 2024-02-01T10:50:00.421 DEBUG:boto:auth_path=/vclfcn-62/obj8?versionId=null 2024-02-01T10:50:00.421 DEBUG:boto:Method: GET 2024-02-01T10:50:00.421 DEBUG:boto:Path: /vclfcn-62/obj8?versionId=null 2024-02-01T10:50:00.421 DEBUG:boto:Data: 2024-02-01T10:50:00.421 DEBUG:boto:Headers: {} 2024-02-01T10:50:00.421 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:00.421 DEBUG:boto:Port: 8001 2024-02-01T10:50:00.421 DEBUG:boto:Params: {} 2024-02-01T10:50:00.422 DEBUG:boto:Token: None 2024-02-01T10:50:00.422 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:00 GMT /vclfcn-62/obj8?versionId=null 2024-02-01T10:50:00.422 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:E29KosB0UfP8oxOUWnErU8Nrt04= 2024-02-01T10:50:00.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:E29KosB0UfP8oxOUWnErU8Nrt04=', 'Content-Length': '0'} 2024-02-01T10:50:00.425 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d8ac1471b037adb1-0065bb7758-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:00.472 DEBUG:boto:path=/vclfcn-62/obj8 2024-02-01T10:50:00.472 DEBUG:boto:auth_path=/vclfcn-62/obj8 2024-02-01T10:50:00.472 DEBUG:boto:path=/vclfcn-62/obj8?versionId=null 2024-02-01T10:50:00.472 DEBUG:boto:auth_path=/vclfcn-62/obj8?versionId=null 2024-02-01T10:50:00.473 DEBUG:boto:Method: GET 2024-02-01T10:50:00.473 DEBUG:boto:Path: /vclfcn-62/obj8?versionId=null 2024-02-01T10:50:00.473 DEBUG:boto:Data: 2024-02-01T10:50:00.473 DEBUG:boto:Headers: {} 2024-02-01T10:50:00.473 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:00.473 DEBUG:boto:Port: 8000 2024-02-01T10:50:00.473 DEBUG:boto:Params: {} 2024-02-01T10:50:00.473 DEBUG:boto:Token: None 2024-02-01T10:50:00.474 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:00 GMT /vclfcn-62/obj8?versionId=null 2024-02-01T10:50:00.474 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:E29KosB0UfP8oxOUWnErU8Nrt04= 2024-02-01T10:50:00.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:E29KosB0UfP8oxOUWnErU8Nrt04=', 'Content-Length': '0'} 2024-02-01T10:50:00.477 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d11a8b7e8711750a-0065bb7758-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:00.524 DEBUG:boto:path=/vclfcn-62/obj8 2024-02-01T10:50:00.524 DEBUG:boto:auth_path=/vclfcn-62/obj8 2024-02-01T10:50:00.524 DEBUG:boto:path=/vclfcn-62/obj8?versionId=null 2024-02-01T10:50:00.524 DEBUG:boto:auth_path=/vclfcn-62/obj8?versionId=null 2024-02-01T10:50:00.524 DEBUG:boto:Method: HEAD 2024-02-01T10:50:00.525 DEBUG:boto:Path: /vclfcn-62/obj8?versionId=null 2024-02-01T10:50:00.525 DEBUG:boto:Data: 2024-02-01T10:50:00.525 DEBUG:boto:Headers: {} 2024-02-01T10:50:00.525 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:00.525 DEBUG:boto:Port: 8001 2024-02-01T10:50:00.525 DEBUG:boto:Params: {} 2024-02-01T10:50:00.525 DEBUG:boto:Token: None 2024-02-01T10:50:00.525 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:00 GMT /vclfcn-62/obj8?versionId=null 2024-02-01T10:50:00.525 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:319EysAhUtOw3HzqMNX5GJz8yOY= 2024-02-01T10:50:00.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:319EysAhUtOw3HzqMNX5GJz8yOY=', 'Content-Length': '0'} 2024-02-01T10:50:00.528 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004d7f14a0ee4bafa0-0065bb7758-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:00.529 DEBUG:boto:path=/vclfcn-62/obj8 2024-02-01T10:50:00.529 DEBUG:boto:auth_path=/vclfcn-62/obj8 2024-02-01T10:50:00.529 DEBUG:boto:path=/vclfcn-62/obj8?versionId=null 2024-02-01T10:50:00.529 DEBUG:boto:auth_path=/vclfcn-62/obj8?versionId=null 2024-02-01T10:50:00.529 DEBUG:boto:Method: HEAD 2024-02-01T10:50:00.529 DEBUG:boto:Path: /vclfcn-62/obj8?versionId=null 2024-02-01T10:50:00.529 DEBUG:boto:Data: 2024-02-01T10:50:00.530 DEBUG:boto:Headers: {} 2024-02-01T10:50:00.530 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:00.530 DEBUG:boto:Port: 8000 2024-02-01T10:50:00.530 DEBUG:boto:Params: {} 2024-02-01T10:50:00.530 DEBUG:boto:Token: None 2024-02-01T10:50:00.530 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:00 GMT /vclfcn-62/obj8?versionId=null 2024-02-01T10:50:00.530 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:319EysAhUtOw3HzqMNX5GJz8yOY= 2024-02-01T10:50:00.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:319EysAhUtOw3HzqMNX5GJz8yOY=', 'Content-Length': '0'} 2024-02-01T10:50:00.533 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007d3e11ef77208670-0065bb7758-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:00.533 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-02-01T10:50:00.533 DEBUG:boto:path=/vclfcn-62/obj8 2024-02-01T10:50:00.533 DEBUG:boto:auth_path=/vclfcn-62/obj8 2024-02-01T10:50:00.534 DEBUG:boto:path=/vclfcn-62/obj8?versionId=null 2024-02-01T10:50:00.534 DEBUG:boto:auth_path=/vclfcn-62/obj8?versionId=null 2024-02-01T10:50:00.534 DEBUG:boto:Method: GET 2024-02-01T10:50:00.534 DEBUG:boto:Path: /vclfcn-62/obj8?versionId=null 2024-02-01T10:50:00.534 DEBUG:boto:Data: 2024-02-01T10:50:00.534 DEBUG:boto:Headers: {} 2024-02-01T10:50:00.534 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:00.534 DEBUG:boto:Port: 8001 2024-02-01T10:50:00.534 DEBUG:boto:Params: {} 2024-02-01T10:50:00.534 DEBUG:boto:Token: None 2024-02-01T10:50:00.535 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:00 GMT /vclfcn-62/obj8?versionId=null 2024-02-01T10:50:00.535 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:E29KosB0UfP8oxOUWnErU8Nrt04= 2024-02-01T10:50:00.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:E29KosB0UfP8oxOUWnErU8Nrt04=', 'Content-Length': '0'} 2024-02-01T10:50:00.537 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d13b075fafd96735-0065bb7758-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:00.580 DEBUG:boto:path=/vclfcn-62/obj8 2024-02-01T10:50:00.580 DEBUG:boto:auth_path=/vclfcn-62/obj8 2024-02-01T10:50:00.580 DEBUG:boto:path=/vclfcn-62/obj8?versionId=null 2024-02-01T10:50:00.581 DEBUG:boto:auth_path=/vclfcn-62/obj8?versionId=null 2024-02-01T10:50:00.581 DEBUG:boto:Method: GET 2024-02-01T10:50:00.581 DEBUG:boto:Path: /vclfcn-62/obj8?versionId=null 2024-02-01T10:50:00.581 DEBUG:boto:Data: 2024-02-01T10:50:00.581 DEBUG:boto:Headers: {} 2024-02-01T10:50:00.581 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:00.581 DEBUG:boto:Port: 8000 2024-02-01T10:50:00.581 DEBUG:boto:Params: {} 2024-02-01T10:50:00.581 DEBUG:boto:Token: None 2024-02-01T10:50:00.582 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:00 GMT /vclfcn-62/obj8?versionId=null 2024-02-01T10:50:00.582 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:E29KosB0UfP8oxOUWnErU8Nrt04= 2024-02-01T10:50:00.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:E29KosB0UfP8oxOUWnErU8Nrt04=', 'Content-Length': '0'} 2024-02-01T10:50:00.584 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002f4066e09d435228-0065bb7758-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:00.628 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:50:00.628 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:50:00.629 DEBUG:boto:path=/vclfcn-62/?prefix=obj8 2024-02-01T10:50:00.629 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj8 2024-02-01T10:50:00.629 DEBUG:boto:Method: GET 2024-02-01T10:50:00.629 DEBUG:boto:Path: /vclfcn-62/?prefix=obj8 2024-02-01T10:50:00.629 DEBUG:boto:Data: 2024-02-01T10:50:00.629 DEBUG:boto:Headers: {} 2024-02-01T10:50:00.629 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:00.629 DEBUG:boto:Port: 8001 2024-02-01T10:50:00.629 DEBUG:boto:Params: {} 2024-02-01T10:50:00.630 DEBUG:boto:Token: None 2024-02-01T10:50:00.630 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:00 GMT /vclfcn-62/ 2024-02-01T10:50:00.630 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:26jso5R2vPYsHxTEVV9idUwWsNw= 2024-02-01T10:50:00.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:26jso5R2vPYsHxTEVV9idUwWsNw=', 'Content-Length': '0'} 2024-02-01T10:50:00.634 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1f3ddc7bafe3789-0065bb7758-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:00.634 DEBUG:boto:b'vclfcn-62obj81000falseobj82024-02-01T10:49:22.200Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:00.635 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:50:00.635 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:50:00.635 DEBUG:boto:path=/vclfcn-62/?prefix=obj8 2024-02-01T10:50:00.635 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj8 2024-02-01T10:50:00.635 DEBUG:boto:Method: GET 2024-02-01T10:50:00.635 DEBUG:boto:Path: /vclfcn-62/?prefix=obj8 2024-02-01T10:50:00.635 DEBUG:boto:Data: 2024-02-01T10:50:00.635 DEBUG:boto:Headers: {} 2024-02-01T10:50:00.635 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:00.636 DEBUG:boto:Port: 8000 2024-02-01T10:50:00.636 DEBUG:boto:Params: {} 2024-02-01T10:50:00.636 DEBUG:boto:Token: None 2024-02-01T10:50:00.636 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:00 GMT /vclfcn-62/ 2024-02-01T10:50:00.636 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:26jso5R2vPYsHxTEVV9idUwWsNw= 2024-02-01T10:50:00.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:26jso5R2vPYsHxTEVV9idUwWsNw=', 'Content-Length': '0'} 2024-02-01T10:50:00.640 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df23cac0d11cd429-0065bb7758-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:00.684 DEBUG:boto:b'vclfcn-62obj81000falseobj82024-02-01T10:49:22.200Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:00.685 DEBUG:boto:path=/vclfcn-62/obj8 2024-02-01T10:50:00.685 DEBUG:boto:auth_path=/vclfcn-62/obj8 2024-02-01T10:50:00.685 DEBUG:boto:Method: HEAD 2024-02-01T10:50:00.685 DEBUG:boto:Path: /vclfcn-62/obj8 2024-02-01T10:50:00.685 DEBUG:boto:Data: 2024-02-01T10:50:00.685 DEBUG:boto:Headers: {} 2024-02-01T10:50:00.685 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:00.685 DEBUG:boto:Port: 8001 2024-02-01T10:50:00.685 DEBUG:boto:Params: {} 2024-02-01T10:50:00.686 DEBUG:boto:Token: None 2024-02-01T10:50:00.686 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:00 GMT /vclfcn-62/obj8 2024-02-01T10:50:00.686 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fhjH7uf5QKJdwtNrg4oGVKaBQK8= 2024-02-01T10:50:00.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fhjH7uf5QKJdwtNrg4oGVKaBQK8=', 'Content-Length': '0'} 2024-02-01T10:50:00.689 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b3f003875532216d-0065bb7758-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:00.689 DEBUG:boto:path=/vclfcn-62/obj8 2024-02-01T10:50:00.690 DEBUG:boto:auth_path=/vclfcn-62/obj8 2024-02-01T10:50:00.690 DEBUG:boto:Method: HEAD 2024-02-01T10:50:00.690 DEBUG:boto:Path: /vclfcn-62/obj8 2024-02-01T10:50:00.690 DEBUG:boto:Data: 2024-02-01T10:50:00.690 DEBUG:boto:Headers: {} 2024-02-01T10:50:00.690 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:00.690 DEBUG:boto:Port: 8000 2024-02-01T10:50:00.690 DEBUG:boto:Params: {} 2024-02-01T10:50:00.690 DEBUG:boto:Token: None 2024-02-01T10:50:00.691 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:00 GMT /vclfcn-62/obj8 2024-02-01T10:50:00.691 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fhjH7uf5QKJdwtNrg4oGVKaBQK8= 2024-02-01T10:50:00.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fhjH7uf5QKJdwtNrg4oGVKaBQK8=', 'Content-Length': '0'} 2024-02-01T10:50:00.694 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c11ba75d78a64e72-0065bb7758-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:00.694 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-02-01T10:50:00.694 DEBUG:boto:path=/vclfcn-62/obj8 2024-02-01T10:50:00.694 DEBUG:boto:auth_path=/vclfcn-62/obj8 2024-02-01T10:50:00.694 DEBUG:boto:Method: GET 2024-02-01T10:50:00.694 DEBUG:boto:Path: /vclfcn-62/obj8 2024-02-01T10:50:00.694 DEBUG:boto:Data: 2024-02-01T10:50:00.695 DEBUG:boto:Headers: {} 2024-02-01T10:50:00.695 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:00.695 DEBUG:boto:Port: 8001 2024-02-01T10:50:00.695 DEBUG:boto:Params: {} 2024-02-01T10:50:00.695 DEBUG:boto:Token: None 2024-02-01T10:50:00.695 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:00 GMT /vclfcn-62/obj8 2024-02-01T10:50:00.695 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ECBAqZQyOrukv62ULnoAojvP8Kc= 2024-02-01T10:50:00.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ECBAqZQyOrukv62ULnoAojvP8Kc=', 'Content-Length': '0'} 2024-02-01T10:50:00.698 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000047f070c655373b9e-0065bb7758-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:00.744 DEBUG:boto:path=/vclfcn-62/obj8 2024-02-01T10:50:00.744 DEBUG:boto:auth_path=/vclfcn-62/obj8 2024-02-01T10:50:00.744 DEBUG:boto:Method: GET 2024-02-01T10:50:00.745 DEBUG:boto:Path: /vclfcn-62/obj8 2024-02-01T10:50:00.745 DEBUG:boto:Data: 2024-02-01T10:50:00.745 DEBUG:boto:Headers: {} 2024-02-01T10:50:00.745 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:00.745 DEBUG:boto:Port: 8000 2024-02-01T10:50:00.745 DEBUG:boto:Params: {} 2024-02-01T10:50:00.745 DEBUG:boto:Token: None 2024-02-01T10:50:00.745 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:00 GMT /vclfcn-62/obj8 2024-02-01T10:50:00.745 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ECBAqZQyOrukv62ULnoAojvP8Kc= 2024-02-01T10:50:00.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ECBAqZQyOrukv62ULnoAojvP8Kc=', 'Content-Length': '0'} 2024-02-01T10:50:00.749 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-62:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000002e1b2cd53bcdadc-0065bb7758-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:00.792 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:50:00.792 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:50:00.792 DEBUG:boto:path=/vclfcn-62/?prefix=obj8 2024-02-01T10:50:00.792 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj8 2024-02-01T10:50:00.793 DEBUG:boto:Method: GET 2024-02-01T10:50:00.793 DEBUG:boto:Path: /vclfcn-62/?prefix=obj8 2024-02-01T10:50:00.793 DEBUG:boto:Data: 2024-02-01T10:50:00.793 DEBUG:boto:Headers: {} 2024-02-01T10:50:00.793 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:00.793 DEBUG:boto:Port: 8001 2024-02-01T10:50:00.793 DEBUG:boto:Params: {} 2024-02-01T10:50:00.793 DEBUG:boto:Token: None 2024-02-01T10:50:00.793 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:00 GMT /vclfcn-62/ 2024-02-01T10:50:00.793 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:26jso5R2vPYsHxTEVV9idUwWsNw= 2024-02-01T10:50:00.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:26jso5R2vPYsHxTEVV9idUwWsNw=', 'Content-Length': '0'} 2024-02-01T10:50:00.798 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8fbdd8fbb313480-0065bb7758-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:00.840 DEBUG:boto:b'vclfcn-62obj81000falseobj82024-02-01T10:49:22.200Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:00.841 DEBUG:boto:path=/vclfcn-62/ 2024-02-01T10:50:00.841 DEBUG:boto:auth_path=/vclfcn-62/ 2024-02-01T10:50:00.841 DEBUG:boto:path=/vclfcn-62/?prefix=obj8 2024-02-01T10:50:00.841 DEBUG:boto:auth_path=/vclfcn-62/?prefix=obj8 2024-02-01T10:50:00.841 DEBUG:boto:Method: GET 2024-02-01T10:50:00.841 DEBUG:boto:Path: /vclfcn-62/?prefix=obj8 2024-02-01T10:50:00.841 DEBUG:boto:Data: 2024-02-01T10:50:00.841 DEBUG:boto:Headers: {} 2024-02-01T10:50:00.842 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:00.842 DEBUG:boto:Port: 8000 2024-02-01T10:50:00.842 DEBUG:boto:Params: {} 2024-02-01T10:50:00.842 DEBUG:boto:Token: None 2024-02-01T10:50:00.842 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:00 GMT /vclfcn-62/ 2024-02-01T10:50:00.842 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:26jso5R2vPYsHxTEVV9idUwWsNw= 2024-02-01T10:50:00.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:00 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:26jso5R2vPYsHxTEVV9idUwWsNw=', 'Content-Length': '0'} 2024-02-01T10:50:00.847 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eff490c182c8a4b3-0065bb7758-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:00 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:00.847 DEBUG:boto:b'vclfcn-62obj81000falseobj82024-02-01T10:49:22.200Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:00.847 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-62 zones={test-zone3, test-zone2} 2024-02-01T10:50:00.848 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=vclfcn-63 2024-02-01T10:50:00.848 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:50:00.848 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:50:00.848 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket vclfcn-63 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-02-01T10:50:00.899 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:50:00.900 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:50:00.967 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:50:00.968+0000 7f9586e04980 1 bucket sync caught up with source: 2024-02-01T10:50:00.967 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [00000000004.52.6, 00000000002.70.1, 00000000004.56.6, 00000000002.61.1, 00000000004.80.6, 00000000002.69.1, 00000000002.116.1, 00000000004.58.6, 00000000002.44.1, 00000000002.56.1, 00000000002.117.1] 2024-02-01T10:50:00.967 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [00000000004.52.6, 00000000002.70.1, 00000000004.56.6, 00000000002.61.1, 00000000004.80.6, 00000000002.69.1, 00000000002.116.1, 00000000004.58.6, 00000000002.44.1, 00000000002.56.1, 00000000002.117.1] 2024-02-01T10:50:00.967 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:50:00.968+0000 7f9586e04980 0 bucket checkpoint complete 2024-02-01T10:50:00.991 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=vclfcn-63 2024-02-01T10:50:00.991 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:50:00.991 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:50:00.991 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-63 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:50:01.041 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:50:01.041 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:50:01.102 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:50:01.102+0000 7fda79fdb980 1 bucket sync caught up with source: 2024-02-01T10:50:01.103 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [00000000004.52.6, 00000000002.70.1, 00000000004.56.6, 00000000002.61.1, 00000000004.80.6, 00000000002.69.1, 00000000002.116.1, 00000000004.58.6, 00000000002.44.1, 00000000002.56.1, 00000000002.117.1] 2024-02-01T10:50:01.103 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [00000000004.52.6, 00000000002.70.1, 00000000004.56.6, 00000000002.61.1, 00000000004.80.6, 00000000002.69.1, 00000000002.116.1, 00000000004.58.6, 00000000002.44.1, 00000000002.56.1, 00000000002.117.1] 2024-02-01T10:50:01.103 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:50:01.102+0000 7fda79fdb980 0 bucket checkpoint complete 2024-02-01T10:50:01.125 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=vclfcn-63 2024-02-01T10:50:01.125 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-63', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:50:01.125 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-63', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:50:01.125 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-63 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:50:01.165 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:50:01.165 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:50:01.226 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:50:01.226+0000 7f22e3745980 1 bucket sync caught up with source: 2024-02-01T10:50:01.226 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [00000000004.257.1, 00000000002.288.1, 00000000004.337.1, 00000000002.219.1, 00000000004.326.1, 00000000002.265.1, 00000000002.298.1, 00000000004.325.1, 00000000002.244.1, 00000000002.326.1, 00000000002.301.1] 2024-02-01T10:50:01.226 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [00000000004.257.1, 00000000002.288.1, 00000000004.337.1, 00000000002.219.1, 00000000004.326.1, 00000000002.265.1, 00000000002.298.1, 00000000004.325.1, 00000000002.244.1, 00000000002.326.1, 00000000002.301.1] 2024-02-01T10:50:01.226 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:50:01.226+0000 7f22e3745980 0 bucket checkpoint complete 2024-02-01T10:50:01.249 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=vclfcn-63 2024-02-01T10:50:01.249 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-63', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:50:01.249 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-63', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:50:01.250 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-63 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:50:01.335 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:50:01.335 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:50:01.396 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:50:01.398+0000 7fc8d8600980 1 bucket sync caught up with source: 2024-02-01T10:50:01.396 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [00000000004.257.1, 00000000002.288.1, 00000000004.337.1, 00000000002.219.1, 00000000004.326.1, 00000000002.265.1, 00000000002.298.1, 00000000004.325.1, 00000000002.244.1, 00000000002.326.1, 00000000002.301.1] 2024-02-01T10:50:01.396 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [00000000004.257.1, 00000000002.288.1, 00000000004.337.1, 00000000002.219.1, 00000000004.326.1, 00000000002.265.1, 00000000002.298.1, 00000000004.325.1, 00000000002.244.1, 00000000002.326.1, 00000000002.301.1] 2024-02-01T10:50:01.396 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:50:01.398+0000 7fc8d8600980 0 bucket checkpoint complete 2024-02-01T10:50:01.419 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=vclfcn-63 2024-02-01T10:50:01.419 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-63', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:50:01.419 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-63', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:50:01.419 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-63 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:50:01.506 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:50:01.507 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:50:01.565 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:50:01.566+0000 7f142cb3e980 1 bucket sync caught up with source: 2024-02-01T10:50:01.565 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [00000000004.233.6, 00000000002.231.1, 00000000004.207.6, 00000000002.113.1, 00000000004.228.6, 00000000002.234.1, 00000000002.365.1, 00000000004.249.6, 00000000002.147.1, 00000000002.245.1, 00000000002.372.1] 2024-02-01T10:50:01.565 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [00000000004.233.6, 00000000002.231.1, 00000000004.207.6, 00000000002.113.1, 00000000004.228.6, 00000000002.234.1, 00000000002.365.1, 00000000004.249.6, 00000000002.147.1, 00000000002.245.1, 00000000002.372.1] 2024-02-01T10:50:01.565 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:50:01.566+0000 7f142cb3e980 0 bucket checkpoint complete 2024-02-01T10:50:01.588 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=vclfcn-63 2024-02-01T10:50:01.588 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-63', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:50:01.589 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-63', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:50:01.589 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket vclfcn-63 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-02-01T10:50:01.640 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:50:01.640 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:50:01.705 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:50:01.704+0000 7ff79c5ac980 1 bucket sync caught up with source: 2024-02-01T10:50:01.705 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [00000000004.233.6, 00000000002.231.1, 00000000004.207.6, 00000000002.113.1, 00000000004.228.6, 00000000002.234.1, 00000000002.365.1, 00000000004.249.6, 00000000002.147.1, 00000000002.245.1, 00000000002.372.1] 2024-02-01T10:50:01.705 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [00000000004.233.6, 00000000002.231.1, 00000000004.207.6, 00000000002.113.1, 00000000004.228.6, 00000000002.234.1, 00000000002.365.1, 00000000004.249.6, 00000000002.147.1, 00000000002.245.1, 00000000002.372.1] 2024-02-01T10:50:01.705 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:50:01.704+0000 7ff79c5ac980 0 bucket checkpoint complete 2024-02-01T10:50:01.728 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-63 zones={test-zone2, test-zone1} 2024-02-01T10:50:01.729 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:01.729 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:01.729 DEBUG:boto:Method: HEAD 2024-02-01T10:50:01.729 DEBUG:boto:Path: /vclfcn-63/ 2024-02-01T10:50:01.729 DEBUG:boto:Data: 2024-02-01T10:50:01.729 DEBUG:boto:Headers: {} 2024-02-01T10:50:01.729 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:01.729 DEBUG:boto:Port: 8000 2024-02-01T10:50:01.729 DEBUG:boto:Params: {} 2024-02-01T10:50:01.730 DEBUG:boto:Token: None 2024-02-01T10:50:01.730 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:01 GMT /vclfcn-63/ 2024-02-01T10:50:01.730 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5h0YNx6qyUWIMrq9g1eY3B0e1Wc= 2024-02-01T10:50:01.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5h0YNx6qyUWIMrq9g1eY3B0e1Wc=', 'Content-Length': '0'} 2024-02-01T10:50:01.733 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000009b4be494b9ea7b4-0065bb7759-5350-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:50:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:01.733 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:01.733 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:01.733 DEBUG:boto:Method: HEAD 2024-02-01T10:50:01.733 DEBUG:boto:Path: /vclfcn-63/ 2024-02-01T10:50:01.733 DEBUG:boto:Data: 2024-02-01T10:50:01.734 DEBUG:boto:Headers: {} 2024-02-01T10:50:01.734 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:01.734 DEBUG:boto:Port: 8000 2024-02-01T10:50:01.734 DEBUG:boto:Params: {} 2024-02-01T10:50:01.734 DEBUG:boto:Token: None 2024-02-01T10:50:01.734 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:01 GMT /vclfcn-63/ 2024-02-01T10:50:01.734 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5h0YNx6qyUWIMrq9g1eY3B0e1Wc= 2024-02-01T10:50:01.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5h0YNx6qyUWIMrq9g1eY3B0e1Wc=', 'Content-Length': '0'} 2024-02-01T10:50:01.738 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c429ec1ea7f53604-0065bb7759-5765-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:50:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:01.738 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:50:01.738 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:01.739 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:01.739 DEBUG:boto:path=/vclfcn-63/?versions&max-keys=999 2024-02-01T10:50:01.739 DEBUG:boto:auth_path=/vclfcn-63/?versions&max-keys=999 2024-02-01T10:50:01.739 DEBUG:boto:Method: GET 2024-02-01T10:50:01.739 DEBUG:boto:Path: /vclfcn-63/?versions&max-keys=999 2024-02-01T10:50:01.739 DEBUG:boto:Data: 2024-02-01T10:50:01.739 DEBUG:boto:Headers: {} 2024-02-01T10:50:01.739 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:01.739 DEBUG:boto:Port: 8000 2024-02-01T10:50:01.739 DEBUG:boto:Params: {} 2024-02-01T10:50:01.740 DEBUG:boto:Token: None 2024-02-01T10:50:01.740 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:01 GMT /vclfcn-63/?versions 2024-02-01T10:50:01.740 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:OV076aAaDMz7gsla9vtCaphXQII= 2024-02-01T10:50:01.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:OV076aAaDMz7gsla9vtCaphXQII=', 'Content-Length': '0'} 2024-02-01T10:50:01.742 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005349f76fed013267-0065bb7759-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:01.743 DEBUG:boto:b'vclfcn-63999falseobj1nulltrue2024-02-01T10:48:26.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:48:26.507Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:48:26.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:48:26.575Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-02-01T10:49:22.255Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-02-01T10:49:22.271Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-02-01T10:49:22.327Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-02-01T10:49:22.343Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:01.744 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-02-01T10:50:01.744 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-02-01T10:50:01.744 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-02-01T10:50:01.745 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-02-01T10:50:01.745 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-02-01T10:50:01.745 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-02-01T10:50:01.745 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-02-01T10:50:01.745 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-02-01T10:50:01.745 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:50:01.745 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:01.745 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:01.746 DEBUG:boto:path=/vclfcn-63/?versions&max-keys=999 2024-02-01T10:50:01.746 DEBUG:boto:auth_path=/vclfcn-63/?versions&max-keys=999 2024-02-01T10:50:01.746 DEBUG:boto:Method: GET 2024-02-01T10:50:01.746 DEBUG:boto:Path: /vclfcn-63/?versions&max-keys=999 2024-02-01T10:50:01.746 DEBUG:boto:Data: 2024-02-01T10:50:01.746 DEBUG:boto:Headers: {} 2024-02-01T10:50:01.746 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:01.746 DEBUG:boto:Port: 8000 2024-02-01T10:50:01.746 DEBUG:boto:Params: {} 2024-02-01T10:50:01.746 DEBUG:boto:Token: None 2024-02-01T10:50:01.747 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:01 GMT /vclfcn-63/?versions 2024-02-01T10:50:01.747 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:OV076aAaDMz7gsla9vtCaphXQII= 2024-02-01T10:50:01.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:OV076aAaDMz7gsla9vtCaphXQII=', 'Content-Length': '0'} 2024-02-01T10:50:01.752 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d2c12cd7c5dfe11-0065bb7759-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:01.752 DEBUG:boto:b'vclfcn-63999falseobj1nulltrue2024-02-01T10:48:26.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:48:26.507Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:48:26.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:48:26.575Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-02-01T10:49:22.255Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-02-01T10:49:22.271Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-02-01T10:49:22.327Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-02-01T10:49:22.343Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:01.752 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-02-01T10:50:01.753 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-02-01T10:50:01.753 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-02-01T10:50:01.753 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-02-01T10:50:01.753 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-02-01T10:50:01.753 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-02-01T10:50:01.753 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-02-01T10:50:01.753 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-02-01T10:50:01.753 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:01.753 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:01.753 DEBUG:boto:path=/vclfcn-63/?versions&max-keys=999 2024-02-01T10:50:01.754 DEBUG:boto:auth_path=/vclfcn-63/?versions&max-keys=999 2024-02-01T10:50:01.754 DEBUG:boto:Method: GET 2024-02-01T10:50:01.754 DEBUG:boto:Path: /vclfcn-63/?versions&max-keys=999 2024-02-01T10:50:01.754 DEBUG:boto:Data: 2024-02-01T10:50:01.754 DEBUG:boto:Headers: {} 2024-02-01T10:50:01.754 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:01.754 DEBUG:boto:Port: 8000 2024-02-01T10:50:01.754 DEBUG:boto:Params: {} 2024-02-01T10:50:01.754 DEBUG:boto:Token: None 2024-02-01T10:50:01.754 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:01 GMT /vclfcn-63/?versions 2024-02-01T10:50:01.755 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:OV076aAaDMz7gsla9vtCaphXQII= 2024-02-01T10:50:01.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:OV076aAaDMz7gsla9vtCaphXQII=', 'Content-Length': '0'} 2024-02-01T10:50:01.757 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050123130746f623f-0065bb7759-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:01.757 DEBUG:boto:b'vclfcn-63999falseobj1nulltrue2024-02-01T10:48:26.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:48:26.507Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:48:26.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:48:26.575Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-02-01T10:49:22.255Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-02-01T10:49:22.271Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-02-01T10:49:22.327Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-02-01T10:49:22.343Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:01.758 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:01.758 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:01.759 DEBUG:boto:path=/vclfcn-63/?versions&max-keys=999 2024-02-01T10:50:01.759 DEBUG:boto:auth_path=/vclfcn-63/?versions&max-keys=999 2024-02-01T10:50:01.759 DEBUG:boto:Method: GET 2024-02-01T10:50:01.759 DEBUG:boto:Path: /vclfcn-63/?versions&max-keys=999 2024-02-01T10:50:01.759 DEBUG:boto:Data: 2024-02-01T10:50:01.759 DEBUG:boto:Headers: {} 2024-02-01T10:50:01.759 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:01.760 DEBUG:boto:Port: 8000 2024-02-01T10:50:01.760 DEBUG:boto:Params: {} 2024-02-01T10:50:01.760 DEBUG:boto:Token: None 2024-02-01T10:50:01.760 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:01 GMT /vclfcn-63/?versions 2024-02-01T10:50:01.761 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:OV076aAaDMz7gsla9vtCaphXQII= 2024-02-01T10:50:01.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:OV076aAaDMz7gsla9vtCaphXQII=', 'Content-Length': '0'} 2024-02-01T10:50:01.765 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a2a6fcc077a73ed-0065bb7759-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:01.766 DEBUG:boto:b'vclfcn-63999falseobj1nulltrue2024-02-01T10:48:26.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:48:26.507Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:48:26.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:48:26.575Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-02-01T10:49:22.255Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-02-01T10:49:22.271Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-02-01T10:49:22.327Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-02-01T10:49:22.343Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:01.767 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:50:01.767 DEBUG:boto:path=/vclfcn-63/obj1 2024-02-01T10:50:01.767 DEBUG:boto:auth_path=/vclfcn-63/obj1 2024-02-01T10:50:01.767 DEBUG:boto:path=/vclfcn-63/obj1?versionId=null 2024-02-01T10:50:01.767 DEBUG:boto:auth_path=/vclfcn-63/obj1?versionId=null 2024-02-01T10:50:01.767 DEBUG:boto:Method: GET 2024-02-01T10:50:01.767 DEBUG:boto:Path: /vclfcn-63/obj1?versionId=null 2024-02-01T10:50:01.768 DEBUG:boto:Data: 2024-02-01T10:50:01.768 DEBUG:boto:Headers: {} 2024-02-01T10:50:01.768 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:01.768 DEBUG:boto:Port: 8000 2024-02-01T10:50:01.768 DEBUG:boto:Params: {} 2024-02-01T10:50:01.768 DEBUG:boto:Token: None 2024-02-01T10:50:01.768 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:01 GMT /vclfcn-63/obj1?versionId=null 2024-02-01T10:50:01.769 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:kR+wt5UJ0P4dGliEJ4Y2inmUJMg= 2024-02-01T10:50:01.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:kR+wt5UJ0P4dGliEJ4Y2inmUJMg=', 'Content-Length': '0'} 2024-02-01T10:50:01.771 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fb99689dbae4ebe2-0065bb7759-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:01.812 DEBUG:boto:path=/vclfcn-63/obj1 2024-02-01T10:50:01.812 DEBUG:boto:auth_path=/vclfcn-63/obj1 2024-02-01T10:50:01.812 DEBUG:boto:path=/vclfcn-63/obj1?versionId=null 2024-02-01T10:50:01.813 DEBUG:boto:auth_path=/vclfcn-63/obj1?versionId=null 2024-02-01T10:50:01.813 DEBUG:boto:Method: GET 2024-02-01T10:50:01.813 DEBUG:boto:Path: /vclfcn-63/obj1?versionId=null 2024-02-01T10:50:01.813 DEBUG:boto:Data: 2024-02-01T10:50:01.813 DEBUG:boto:Headers: {} 2024-02-01T10:50:01.813 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:01.813 DEBUG:boto:Port: 8000 2024-02-01T10:50:01.813 DEBUG:boto:Params: {} 2024-02-01T10:50:01.814 DEBUG:boto:Token: None 2024-02-01T10:50:01.814 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:01 GMT /vclfcn-63/obj1?versionId=null 2024-02-01T10:50:01.814 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:kR+wt5UJ0P4dGliEJ4Y2inmUJMg= 2024-02-01T10:50:01.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:kR+wt5UJ0P4dGliEJ4Y2inmUJMg=', 'Content-Length': '0'} 2024-02-01T10:50:01.817 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000010b35bd19bd511d6-0065bb7759-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:01.818 DEBUG:boto:path=/vclfcn-63/obj1 2024-02-01T10:50:01.818 DEBUG:boto:auth_path=/vclfcn-63/obj1 2024-02-01T10:50:01.818 DEBUG:boto:path=/vclfcn-63/obj1?versionId=null 2024-02-01T10:50:01.818 DEBUG:boto:auth_path=/vclfcn-63/obj1?versionId=null 2024-02-01T10:50:01.818 DEBUG:boto:Method: HEAD 2024-02-01T10:50:01.818 DEBUG:boto:Path: /vclfcn-63/obj1?versionId=null 2024-02-01T10:50:01.818 DEBUG:boto:Data: 2024-02-01T10:50:01.818 DEBUG:boto:Headers: {} 2024-02-01T10:50:01.818 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:01.818 DEBUG:boto:Port: 8000 2024-02-01T10:50:01.819 DEBUG:boto:Params: {} 2024-02-01T10:50:01.819 DEBUG:boto:Token: None 2024-02-01T10:50:01.819 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:01 GMT /vclfcn-63/obj1?versionId=null 2024-02-01T10:50:01.819 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zPB7jgIF1MRMfVTEVAuIFjJuaoI= 2024-02-01T10:50:01.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zPB7jgIF1MRMfVTEVAuIFjJuaoI=', 'Content-Length': '0'} 2024-02-01T10:50:01.822 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a006b732b1b27d6e-0065bb7759-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:01.822 DEBUG:boto:path=/vclfcn-63/obj1 2024-02-01T10:50:01.822 DEBUG:boto:auth_path=/vclfcn-63/obj1 2024-02-01T10:50:01.823 DEBUG:boto:path=/vclfcn-63/obj1?versionId=null 2024-02-01T10:50:01.823 DEBUG:boto:auth_path=/vclfcn-63/obj1?versionId=null 2024-02-01T10:50:01.823 DEBUG:boto:Method: HEAD 2024-02-01T10:50:01.823 DEBUG:boto:Path: /vclfcn-63/obj1?versionId=null 2024-02-01T10:50:01.823 DEBUG:boto:Data: 2024-02-01T10:50:01.823 DEBUG:boto:Headers: {} 2024-02-01T10:50:01.823 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:01.823 DEBUG:boto:Port: 8000 2024-02-01T10:50:01.823 DEBUG:boto:Params: {} 2024-02-01T10:50:01.824 DEBUG:boto:Token: None 2024-02-01T10:50:01.824 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:01 GMT /vclfcn-63/obj1?versionId=null 2024-02-01T10:50:01.824 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zPB7jgIF1MRMfVTEVAuIFjJuaoI= 2024-02-01T10:50:01.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zPB7jgIF1MRMfVTEVAuIFjJuaoI=', 'Content-Length': '0'} 2024-02-01T10:50:01.827 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aca74148fec3f863-0065bb7759-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:01.827 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:50:01.827 DEBUG:boto:path=/vclfcn-63/obj1 2024-02-01T10:50:01.828 DEBUG:boto:auth_path=/vclfcn-63/obj1 2024-02-01T10:50:01.828 DEBUG:boto:path=/vclfcn-63/obj1?versionId=null 2024-02-01T10:50:01.828 DEBUG:boto:auth_path=/vclfcn-63/obj1?versionId=null 2024-02-01T10:50:01.828 DEBUG:boto:Method: GET 2024-02-01T10:50:01.828 DEBUG:boto:Path: /vclfcn-63/obj1?versionId=null 2024-02-01T10:50:01.828 DEBUG:boto:Data: 2024-02-01T10:50:01.828 DEBUG:boto:Headers: {} 2024-02-01T10:50:01.828 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:01.828 DEBUG:boto:Port: 8000 2024-02-01T10:50:01.828 DEBUG:boto:Params: {} 2024-02-01T10:50:01.829 DEBUG:boto:Token: None 2024-02-01T10:50:01.829 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:01 GMT /vclfcn-63/obj1?versionId=null 2024-02-01T10:50:01.829 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:kR+wt5UJ0P4dGliEJ4Y2inmUJMg= 2024-02-01T10:50:01.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:kR+wt5UJ0P4dGliEJ4Y2inmUJMg=', 'Content-Length': '0'} 2024-02-01T10:50:01.831 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008308d401650c1d3c-0065bb7759-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:01.832 DEBUG:boto:path=/vclfcn-63/obj1 2024-02-01T10:50:01.832 DEBUG:boto:auth_path=/vclfcn-63/obj1 2024-02-01T10:50:01.832 DEBUG:boto:path=/vclfcn-63/obj1?versionId=null 2024-02-01T10:50:01.832 DEBUG:boto:auth_path=/vclfcn-63/obj1?versionId=null 2024-02-01T10:50:01.832 DEBUG:boto:Method: GET 2024-02-01T10:50:01.832 DEBUG:boto:Path: /vclfcn-63/obj1?versionId=null 2024-02-01T10:50:01.832 DEBUG:boto:Data: 2024-02-01T10:50:01.833 DEBUG:boto:Headers: {} 2024-02-01T10:50:01.833 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:01.833 DEBUG:boto:Port: 8000 2024-02-01T10:50:01.833 DEBUG:boto:Params: {} 2024-02-01T10:50:01.833 DEBUG:boto:Token: None 2024-02-01T10:50:01.833 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:01 GMT /vclfcn-63/obj1?versionId=null 2024-02-01T10:50:01.833 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:kR+wt5UJ0P4dGliEJ4Y2inmUJMg= 2024-02-01T10:50:01.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:kR+wt5UJ0P4dGliEJ4Y2inmUJMg=', 'Content-Length': '0'} 2024-02-01T10:50:01.836 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c67e3edc7aac5d54-0065bb7759-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:01.837 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:01.837 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:01.837 DEBUG:boto:path=/vclfcn-63/?prefix=obj1 2024-02-01T10:50:01.837 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj1 2024-02-01T10:50:01.837 DEBUG:boto:Method: GET 2024-02-01T10:50:01.837 DEBUG:boto:Path: /vclfcn-63/?prefix=obj1 2024-02-01T10:50:01.837 DEBUG:boto:Data: 2024-02-01T10:50:01.838 DEBUG:boto:Headers: {} 2024-02-01T10:50:01.838 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:01.838 DEBUG:boto:Port: 8000 2024-02-01T10:50:01.838 DEBUG:boto:Params: {} 2024-02-01T10:50:01.838 DEBUG:boto:Token: None 2024-02-01T10:50:01.838 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:01 GMT /vclfcn-63/ 2024-02-01T10:50:01.838 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:EENbchD2J+S08GNTxklj34po+Ac= 2024-02-01T10:50:01.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:EENbchD2J+S08GNTxklj34po+Ac=', 'Content-Length': '0'} 2024-02-01T10:50:01.843 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de5ecf0a4b5dad1e-0065bb7759-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:01.843 DEBUG:boto:b'vclfcn-63obj11000falseobj12024-02-01T10:48:26.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:01.843 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:01.844 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:01.844 DEBUG:boto:path=/vclfcn-63/?prefix=obj1 2024-02-01T10:50:01.844 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj1 2024-02-01T10:50:01.844 DEBUG:boto:Method: GET 2024-02-01T10:50:01.844 DEBUG:boto:Path: /vclfcn-63/?prefix=obj1 2024-02-01T10:50:01.844 DEBUG:boto:Data: 2024-02-01T10:50:01.844 DEBUG:boto:Headers: {} 2024-02-01T10:50:01.844 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:01.844 DEBUG:boto:Port: 8000 2024-02-01T10:50:01.844 DEBUG:boto:Params: {} 2024-02-01T10:50:01.844 DEBUG:boto:Token: None 2024-02-01T10:50:01.845 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:01 GMT /vclfcn-63/ 2024-02-01T10:50:01.845 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:EENbchD2J+S08GNTxklj34po+Ac= 2024-02-01T10:50:01.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:EENbchD2J+S08GNTxklj34po+Ac=', 'Content-Length': '0'} 2024-02-01T10:50:01.849 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c89cbb5e3132a16f-0065bb7759-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:01.849 DEBUG:boto:b'vclfcn-63obj11000falseobj12024-02-01T10:48:26.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:01.850 DEBUG:boto:path=/vclfcn-63/obj1 2024-02-01T10:50:01.850 DEBUG:boto:auth_path=/vclfcn-63/obj1 2024-02-01T10:50:01.850 DEBUG:boto:Method: HEAD 2024-02-01T10:50:01.850 DEBUG:boto:Path: /vclfcn-63/obj1 2024-02-01T10:50:01.850 DEBUG:boto:Data: 2024-02-01T10:50:01.850 DEBUG:boto:Headers: {} 2024-02-01T10:50:01.850 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:01.850 DEBUG:boto:Port: 8000 2024-02-01T10:50:01.851 DEBUG:boto:Params: {} 2024-02-01T10:50:01.851 DEBUG:boto:Token: None 2024-02-01T10:50:01.851 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:01 GMT /vclfcn-63/obj1 2024-02-01T10:50:01.851 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:XyXhXCObuQOSfNNG+lntMwOd14s= 2024-02-01T10:50:01.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:XyXhXCObuQOSfNNG+lntMwOd14s=', 'Content-Length': '0'} 2024-02-01T10:50:01.853 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005c5b455266dca6ad-0065bb7759-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:01.854 DEBUG:boto:path=/vclfcn-63/obj1 2024-02-01T10:50:01.854 DEBUG:boto:auth_path=/vclfcn-63/obj1 2024-02-01T10:50:01.854 DEBUG:boto:Method: HEAD 2024-02-01T10:50:01.854 DEBUG:boto:Path: /vclfcn-63/obj1 2024-02-01T10:50:01.854 DEBUG:boto:Data: 2024-02-01T10:50:01.854 DEBUG:boto:Headers: {} 2024-02-01T10:50:01.854 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:01.854 DEBUG:boto:Port: 8000 2024-02-01T10:50:01.854 DEBUG:boto:Params: {} 2024-02-01T10:50:01.855 DEBUG:boto:Token: None 2024-02-01T10:50:01.855 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:01 GMT /vclfcn-63/obj1 2024-02-01T10:50:01.855 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:XyXhXCObuQOSfNNG+lntMwOd14s= 2024-02-01T10:50:01.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:XyXhXCObuQOSfNNG+lntMwOd14s=', 'Content-Length': '0'} 2024-02-01T10:50:01.857 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000059c33910d8a39495-0065bb7759-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:01.858 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:50:01.858 DEBUG:boto:path=/vclfcn-63/obj1 2024-02-01T10:50:01.858 DEBUG:boto:auth_path=/vclfcn-63/obj1 2024-02-01T10:50:01.858 DEBUG:boto:Method: GET 2024-02-01T10:50:01.858 DEBUG:boto:Path: /vclfcn-63/obj1 2024-02-01T10:50:01.858 DEBUG:boto:Data: 2024-02-01T10:50:01.858 DEBUG:boto:Headers: {} 2024-02-01T10:50:01.858 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:01.858 DEBUG:boto:Port: 8000 2024-02-01T10:50:01.858 DEBUG:boto:Params: {} 2024-02-01T10:50:01.859 DEBUG:boto:Token: None 2024-02-01T10:50:01.859 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:01 GMT /vclfcn-63/obj1 2024-02-01T10:50:01.859 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/YlyVrnAstQi7xDoUbycI3En5KQ= 2024-02-01T10:50:01.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/YlyVrnAstQi7xDoUbycI3En5KQ=', 'Content-Length': '0'} 2024-02-01T10:50:01.862 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006a793d035145e152-0065bb7759-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:01.904 DEBUG:boto:path=/vclfcn-63/obj1 2024-02-01T10:50:01.904 DEBUG:boto:auth_path=/vclfcn-63/obj1 2024-02-01T10:50:01.904 DEBUG:boto:Method: GET 2024-02-01T10:50:01.904 DEBUG:boto:Path: /vclfcn-63/obj1 2024-02-01T10:50:01.905 DEBUG:boto:Data: 2024-02-01T10:50:01.905 DEBUG:boto:Headers: {} 2024-02-01T10:50:01.905 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:01.905 DEBUG:boto:Port: 8000 2024-02-01T10:50:01.905 DEBUG:boto:Params: {} 2024-02-01T10:50:01.905 DEBUG:boto:Token: None 2024-02-01T10:50:01.905 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:01 GMT /vclfcn-63/obj1 2024-02-01T10:50:01.905 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/YlyVrnAstQi7xDoUbycI3En5KQ= 2024-02-01T10:50:01.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/YlyVrnAstQi7xDoUbycI3En5KQ=', 'Content-Length': '0'} 2024-02-01T10:50:01.908 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000042b0323f70a60e35-0065bb7759-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:01.952 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:01.952 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:01.952 DEBUG:boto:path=/vclfcn-63/?prefix=obj1 2024-02-01T10:50:01.953 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj1 2024-02-01T10:50:01.953 DEBUG:boto:Method: GET 2024-02-01T10:50:01.953 DEBUG:boto:Path: /vclfcn-63/?prefix=obj1 2024-02-01T10:50:01.953 DEBUG:boto:Data: 2024-02-01T10:50:01.953 DEBUG:boto:Headers: {} 2024-02-01T10:50:01.953 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:01.953 DEBUG:boto:Port: 8000 2024-02-01T10:50:01.953 DEBUG:boto:Params: {} 2024-02-01T10:50:01.953 DEBUG:boto:Token: None 2024-02-01T10:50:01.953 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:01 GMT /vclfcn-63/ 2024-02-01T10:50:01.954 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:EENbchD2J+S08GNTxklj34po+Ac= 2024-02-01T10:50:01.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:EENbchD2J+S08GNTxklj34po+Ac=', 'Content-Length': '0'} 2024-02-01T10:50:01.958 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c444a175da792645-0065bb7759-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:01.958 DEBUG:boto:b'vclfcn-63obj11000falseobj12024-02-01T10:48:26.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:01.959 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:01.959 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:01.959 DEBUG:boto:path=/vclfcn-63/?prefix=obj1 2024-02-01T10:50:01.959 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj1 2024-02-01T10:50:01.959 DEBUG:boto:Method: GET 2024-02-01T10:50:01.959 DEBUG:boto:Path: /vclfcn-63/?prefix=obj1 2024-02-01T10:50:01.959 DEBUG:boto:Data: 2024-02-01T10:50:01.959 DEBUG:boto:Headers: {} 2024-02-01T10:50:01.959 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:01.959 DEBUG:boto:Port: 8000 2024-02-01T10:50:01.960 DEBUG:boto:Params: {} 2024-02-01T10:50:01.960 DEBUG:boto:Token: None 2024-02-01T10:50:01.960 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:01 GMT /vclfcn-63/ 2024-02-01T10:50:01.960 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:EENbchD2J+S08GNTxklj34po+Ac= 2024-02-01T10:50:01.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:EENbchD2J+S08GNTxklj34po+Ac=', 'Content-Length': '0'} 2024-02-01T10:50:01.964 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018d11708ab3a9eb5-0065bb7759-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:01.964 DEBUG:boto:b'vclfcn-63obj11000falseobj12024-02-01T10:48:26.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:01.965 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:50:01.965 DEBUG:boto:path=/vclfcn-63/obj2 2024-02-01T10:50:01.965 DEBUG:boto:auth_path=/vclfcn-63/obj2 2024-02-01T10:50:01.965 DEBUG:boto:path=/vclfcn-63/obj2?versionId=null 2024-02-01T10:50:01.965 DEBUG:boto:auth_path=/vclfcn-63/obj2?versionId=null 2024-02-01T10:50:01.965 DEBUG:boto:Method: GET 2024-02-01T10:50:01.965 DEBUG:boto:Path: /vclfcn-63/obj2?versionId=null 2024-02-01T10:50:01.965 DEBUG:boto:Data: 2024-02-01T10:50:01.966 DEBUG:boto:Headers: {} 2024-02-01T10:50:01.966 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:01.966 DEBUG:boto:Port: 8000 2024-02-01T10:50:01.966 DEBUG:boto:Params: {} 2024-02-01T10:50:01.966 DEBUG:boto:Token: None 2024-02-01T10:50:01.966 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:01 GMT /vclfcn-63/obj2?versionId=null 2024-02-01T10:50:01.966 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Dss5CPiJmv+MxN6NeuYqnTozR6k= 2024-02-01T10:50:01.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Dss5CPiJmv+MxN6NeuYqnTozR6k=', 'Content-Length': '0'} 2024-02-01T10:50:01.969 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003c9e4a9a05ef1ffd-0065bb7759-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:01.969 DEBUG:boto:path=/vclfcn-63/obj2 2024-02-01T10:50:01.969 DEBUG:boto:auth_path=/vclfcn-63/obj2 2024-02-01T10:50:01.970 DEBUG:boto:path=/vclfcn-63/obj2?versionId=null 2024-02-01T10:50:01.970 DEBUG:boto:auth_path=/vclfcn-63/obj2?versionId=null 2024-02-01T10:50:01.970 DEBUG:boto:Method: GET 2024-02-01T10:50:01.970 DEBUG:boto:Path: /vclfcn-63/obj2?versionId=null 2024-02-01T10:50:01.970 DEBUG:boto:Data: 2024-02-01T10:50:01.970 DEBUG:boto:Headers: {} 2024-02-01T10:50:01.970 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:01.970 DEBUG:boto:Port: 8000 2024-02-01T10:50:01.970 DEBUG:boto:Params: {} 2024-02-01T10:50:01.970 DEBUG:boto:Token: None 2024-02-01T10:50:01.970 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:01 GMT /vclfcn-63/obj2?versionId=null 2024-02-01T10:50:01.971 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Dss5CPiJmv+MxN6NeuYqnTozR6k= 2024-02-01T10:50:01.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Dss5CPiJmv+MxN6NeuYqnTozR6k=', 'Content-Length': '0'} 2024-02-01T10:50:01.974 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000945c11a0ffa4df42-0065bb7759-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:01.974 DEBUG:boto:path=/vclfcn-63/obj2 2024-02-01T10:50:01.974 DEBUG:boto:auth_path=/vclfcn-63/obj2 2024-02-01T10:50:01.974 DEBUG:boto:path=/vclfcn-63/obj2?versionId=null 2024-02-01T10:50:01.975 DEBUG:boto:auth_path=/vclfcn-63/obj2?versionId=null 2024-02-01T10:50:01.975 DEBUG:boto:Method: HEAD 2024-02-01T10:50:01.975 DEBUG:boto:Path: /vclfcn-63/obj2?versionId=null 2024-02-01T10:50:01.975 DEBUG:boto:Data: 2024-02-01T10:50:01.975 DEBUG:boto:Headers: {} 2024-02-01T10:50:01.975 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:01.975 DEBUG:boto:Port: 8000 2024-02-01T10:50:01.975 DEBUG:boto:Params: {} 2024-02-01T10:50:01.975 DEBUG:boto:Token: None 2024-02-01T10:50:01.975 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:01 GMT /vclfcn-63/obj2?versionId=null 2024-02-01T10:50:01.976 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YZtP1JZSJWDxPhI5Fum3gZpBfno= 2024-02-01T10:50:01.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YZtP1JZSJWDxPhI5Fum3gZpBfno=', 'Content-Length': '0'} 2024-02-01T10:50:01.978 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000007d7b60da684d8b-0065bb7759-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:01.979 DEBUG:boto:path=/vclfcn-63/obj2 2024-02-01T10:50:01.979 DEBUG:boto:auth_path=/vclfcn-63/obj2 2024-02-01T10:50:01.979 DEBUG:boto:path=/vclfcn-63/obj2?versionId=null 2024-02-01T10:50:01.979 DEBUG:boto:auth_path=/vclfcn-63/obj2?versionId=null 2024-02-01T10:50:01.979 DEBUG:boto:Method: HEAD 2024-02-01T10:50:01.979 DEBUG:boto:Path: /vclfcn-63/obj2?versionId=null 2024-02-01T10:50:01.979 DEBUG:boto:Data: 2024-02-01T10:50:01.979 DEBUG:boto:Headers: {} 2024-02-01T10:50:01.979 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:01.979 DEBUG:boto:Port: 8000 2024-02-01T10:50:01.979 DEBUG:boto:Params: {} 2024-02-01T10:50:01.980 DEBUG:boto:Token: None 2024-02-01T10:50:01.980 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:01 GMT /vclfcn-63/obj2?versionId=null 2024-02-01T10:50:01.980 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YZtP1JZSJWDxPhI5Fum3gZpBfno= 2024-02-01T10:50:01.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YZtP1JZSJWDxPhI5Fum3gZpBfno=', 'Content-Length': '0'} 2024-02-01T10:50:01.983 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000082d9e34352494c20-0065bb7759-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:01.983 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:50:01.983 DEBUG:boto:path=/vclfcn-63/obj2 2024-02-01T10:50:01.983 DEBUG:boto:auth_path=/vclfcn-63/obj2 2024-02-01T10:50:01.984 DEBUG:boto:path=/vclfcn-63/obj2?versionId=null 2024-02-01T10:50:01.984 DEBUG:boto:auth_path=/vclfcn-63/obj2?versionId=null 2024-02-01T10:50:01.984 DEBUG:boto:Method: GET 2024-02-01T10:50:01.984 DEBUG:boto:Path: /vclfcn-63/obj2?versionId=null 2024-02-01T10:50:01.984 DEBUG:boto:Data: 2024-02-01T10:50:01.984 DEBUG:boto:Headers: {} 2024-02-01T10:50:01.984 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:01.984 DEBUG:boto:Port: 8000 2024-02-01T10:50:01.984 DEBUG:boto:Params: {} 2024-02-01T10:50:01.984 DEBUG:boto:Token: None 2024-02-01T10:50:01.985 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:01 GMT /vclfcn-63/obj2?versionId=null 2024-02-01T10:50:01.985 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Dss5CPiJmv+MxN6NeuYqnTozR6k= 2024-02-01T10:50:01.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Dss5CPiJmv+MxN6NeuYqnTozR6k=', 'Content-Length': '0'} 2024-02-01T10:50:01.987 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000025827fa39f0a3988-0065bb7759-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:01.988 DEBUG:boto:path=/vclfcn-63/obj2 2024-02-01T10:50:01.988 DEBUG:boto:auth_path=/vclfcn-63/obj2 2024-02-01T10:50:01.988 DEBUG:boto:path=/vclfcn-63/obj2?versionId=null 2024-02-01T10:50:01.988 DEBUG:boto:auth_path=/vclfcn-63/obj2?versionId=null 2024-02-01T10:50:01.988 DEBUG:boto:Method: GET 2024-02-01T10:50:01.988 DEBUG:boto:Path: /vclfcn-63/obj2?versionId=null 2024-02-01T10:50:01.988 DEBUG:boto:Data: 2024-02-01T10:50:01.989 DEBUG:boto:Headers: {} 2024-02-01T10:50:01.989 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:01.989 DEBUG:boto:Port: 8000 2024-02-01T10:50:01.989 DEBUG:boto:Params: {} 2024-02-01T10:50:01.989 DEBUG:boto:Token: None 2024-02-01T10:50:01.989 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:01 GMT /vclfcn-63/obj2?versionId=null 2024-02-01T10:50:01.989 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Dss5CPiJmv+MxN6NeuYqnTozR6k= 2024-02-01T10:50:01.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:01 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Dss5CPiJmv+MxN6NeuYqnTozR6k=', 'Content-Length': '0'} 2024-02-01T10:50:01.992 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c45ed840cc16de0e-0065bb7759-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:01 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:02.036 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:02.036 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:02.036 DEBUG:boto:path=/vclfcn-63/?prefix=obj2 2024-02-01T10:50:02.037 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj2 2024-02-01T10:50:02.037 DEBUG:boto:Method: GET 2024-02-01T10:50:02.037 DEBUG:boto:Path: /vclfcn-63/?prefix=obj2 2024-02-01T10:50:02.037 DEBUG:boto:Data: 2024-02-01T10:50:02.037 DEBUG:boto:Headers: {} 2024-02-01T10:50:02.037 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:02.037 DEBUG:boto:Port: 8000 2024-02-01T10:50:02.037 DEBUG:boto:Params: {} 2024-02-01T10:50:02.037 DEBUG:boto:Token: None 2024-02-01T10:50:02.038 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:02 GMT /vclfcn-63/ 2024-02-01T10:50:02.038 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ylIwsGcaSaHAU7+2+RTv2egtcow= 2024-02-01T10:50:02.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ylIwsGcaSaHAU7+2+RTv2egtcow=', 'Content-Length': '0'} 2024-02-01T10:50:02.042 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a430aa705d63e83-0065bb775a-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:02.042 DEBUG:boto:b'vclfcn-63obj21000falseobj22024-02-01T10:48:26.507Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:02.043 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:02.043 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:02.043 DEBUG:boto:path=/vclfcn-63/?prefix=obj2 2024-02-01T10:50:02.043 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj2 2024-02-01T10:50:02.043 DEBUG:boto:Method: GET 2024-02-01T10:50:02.043 DEBUG:boto:Path: /vclfcn-63/?prefix=obj2 2024-02-01T10:50:02.043 DEBUG:boto:Data: 2024-02-01T10:50:02.043 DEBUG:boto:Headers: {} 2024-02-01T10:50:02.044 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:02.044 DEBUG:boto:Port: 8000 2024-02-01T10:50:02.044 DEBUG:boto:Params: {} 2024-02-01T10:50:02.044 DEBUG:boto:Token: None 2024-02-01T10:50:02.044 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:02 GMT /vclfcn-63/ 2024-02-01T10:50:02.044 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ylIwsGcaSaHAU7+2+RTv2egtcow= 2024-02-01T10:50:02.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ylIwsGcaSaHAU7+2+RTv2egtcow=', 'Content-Length': '0'} 2024-02-01T10:50:02.048 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005dc3a7a6b95a520c-0065bb775a-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:02.049 DEBUG:boto:b'vclfcn-63obj21000falseobj22024-02-01T10:48:26.507Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:02.049 DEBUG:boto:path=/vclfcn-63/obj2 2024-02-01T10:50:02.050 DEBUG:boto:auth_path=/vclfcn-63/obj2 2024-02-01T10:50:02.050 DEBUG:boto:Method: HEAD 2024-02-01T10:50:02.050 DEBUG:boto:Path: /vclfcn-63/obj2 2024-02-01T10:50:02.050 DEBUG:boto:Data: 2024-02-01T10:50:02.050 DEBUG:boto:Headers: {} 2024-02-01T10:50:02.050 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:02.050 DEBUG:boto:Port: 8000 2024-02-01T10:50:02.050 DEBUG:boto:Params: {} 2024-02-01T10:50:02.050 DEBUG:boto:Token: None 2024-02-01T10:50:02.051 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:02 GMT /vclfcn-63/obj2 2024-02-01T10:50:02.051 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:oHFFqpFHngj9XNjECHCkRB410X8= 2024-02-01T10:50:02.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:oHFFqpFHngj9XNjECHCkRB410X8=', 'Content-Length': '0'} 2024-02-01T10:50:02.053 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003389bd6161ba5ad8-0065bb775a-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:02.054 DEBUG:boto:path=/vclfcn-63/obj2 2024-02-01T10:50:02.054 DEBUG:boto:auth_path=/vclfcn-63/obj2 2024-02-01T10:50:02.054 DEBUG:boto:Method: HEAD 2024-02-01T10:50:02.054 DEBUG:boto:Path: /vclfcn-63/obj2 2024-02-01T10:50:02.054 DEBUG:boto:Data: 2024-02-01T10:50:02.054 DEBUG:boto:Headers: {} 2024-02-01T10:50:02.054 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:02.055 DEBUG:boto:Port: 8000 2024-02-01T10:50:02.056 DEBUG:boto:Params: {} 2024-02-01T10:50:02.056 DEBUG:boto:Token: None 2024-02-01T10:50:02.056 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:02 GMT /vclfcn-63/obj2 2024-02-01T10:50:02.057 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:oHFFqpFHngj9XNjECHCkRB410X8= 2024-02-01T10:50:02.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:oHFFqpFHngj9XNjECHCkRB410X8=', 'Content-Length': '0'} 2024-02-01T10:50:02.059 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000313930917e633458-0065bb775a-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:02.060 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:50:02.060 DEBUG:boto:path=/vclfcn-63/obj2 2024-02-01T10:50:02.060 DEBUG:boto:auth_path=/vclfcn-63/obj2 2024-02-01T10:50:02.060 DEBUG:boto:Method: GET 2024-02-01T10:50:02.060 DEBUG:boto:Path: /vclfcn-63/obj2 2024-02-01T10:50:02.060 DEBUG:boto:Data: 2024-02-01T10:50:02.060 DEBUG:boto:Headers: {} 2024-02-01T10:50:02.060 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:02.061 DEBUG:boto:Port: 8000 2024-02-01T10:50:02.061 DEBUG:boto:Params: {} 2024-02-01T10:50:02.061 DEBUG:boto:Token: None 2024-02-01T10:50:02.061 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:02 GMT /vclfcn-63/obj2 2024-02-01T10:50:02.061 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:a8PITSX9xXkN1bByrcPrfHkkcP8= 2024-02-01T10:50:02.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:a8PITSX9xXkN1bByrcPrfHkkcP8=', 'Content-Length': '0'} 2024-02-01T10:50:02.064 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c9f793d6923adf28-0065bb775a-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:02.108 DEBUG:boto:path=/vclfcn-63/obj2 2024-02-01T10:50:02.108 DEBUG:boto:auth_path=/vclfcn-63/obj2 2024-02-01T10:50:02.108 DEBUG:boto:Method: GET 2024-02-01T10:50:02.109 DEBUG:boto:Path: /vclfcn-63/obj2 2024-02-01T10:50:02.109 DEBUG:boto:Data: 2024-02-01T10:50:02.109 DEBUG:boto:Headers: {} 2024-02-01T10:50:02.109 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:02.109 DEBUG:boto:Port: 8000 2024-02-01T10:50:02.109 DEBUG:boto:Params: {} 2024-02-01T10:50:02.109 DEBUG:boto:Token: None 2024-02-01T10:50:02.110 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:02 GMT /vclfcn-63/obj2 2024-02-01T10:50:02.110 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:a8PITSX9xXkN1bByrcPrfHkkcP8= 2024-02-01T10:50:02.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:a8PITSX9xXkN1bByrcPrfHkkcP8=', 'Content-Length': '0'} 2024-02-01T10:50:02.113 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fd400aeb7a1d807a-0065bb775a-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:02.160 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:02.160 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:02.160 DEBUG:boto:path=/vclfcn-63/?prefix=obj2 2024-02-01T10:50:02.160 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj2 2024-02-01T10:50:02.161 DEBUG:boto:Method: GET 2024-02-01T10:50:02.161 DEBUG:boto:Path: /vclfcn-63/?prefix=obj2 2024-02-01T10:50:02.161 DEBUG:boto:Data: 2024-02-01T10:50:02.161 DEBUG:boto:Headers: {} 2024-02-01T10:50:02.161 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:02.161 DEBUG:boto:Port: 8000 2024-02-01T10:50:02.161 DEBUG:boto:Params: {} 2024-02-01T10:50:02.161 DEBUG:boto:Token: None 2024-02-01T10:50:02.161 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:02 GMT /vclfcn-63/ 2024-02-01T10:50:02.162 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ylIwsGcaSaHAU7+2+RTv2egtcow= 2024-02-01T10:50:02.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ylIwsGcaSaHAU7+2+RTv2egtcow=', 'Content-Length': '0'} 2024-02-01T10:50:02.166 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000611e84503088dafc-0065bb775a-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:02.166 DEBUG:boto:b'vclfcn-63obj21000falseobj22024-02-01T10:48:26.507Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:02.167 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:02.167 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:02.167 DEBUG:boto:path=/vclfcn-63/?prefix=obj2 2024-02-01T10:50:02.167 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj2 2024-02-01T10:50:02.167 DEBUG:boto:Method: GET 2024-02-01T10:50:02.168 DEBUG:boto:Path: /vclfcn-63/?prefix=obj2 2024-02-01T10:50:02.168 DEBUG:boto:Data: 2024-02-01T10:50:02.168 DEBUG:boto:Headers: {} 2024-02-01T10:50:02.168 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:02.168 DEBUG:boto:Port: 8000 2024-02-01T10:50:02.168 DEBUG:boto:Params: {} 2024-02-01T10:50:02.168 DEBUG:boto:Token: None 2024-02-01T10:50:02.169 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:02 GMT /vclfcn-63/ 2024-02-01T10:50:02.169 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ylIwsGcaSaHAU7+2+RTv2egtcow= 2024-02-01T10:50:02.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ylIwsGcaSaHAU7+2+RTv2egtcow=', 'Content-Length': '0'} 2024-02-01T10:50:02.174 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086ce4a4c87a8f56e-0065bb775a-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:02.216 DEBUG:boto:b'vclfcn-63obj21000falseobj22024-02-01T10:48:26.507Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:02.217 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:50:02.217 DEBUG:boto:path=/vclfcn-63/obj3 2024-02-01T10:50:02.217 DEBUG:boto:auth_path=/vclfcn-63/obj3 2024-02-01T10:50:02.217 DEBUG:boto:path=/vclfcn-63/obj3?versionId=null 2024-02-01T10:50:02.217 DEBUG:boto:auth_path=/vclfcn-63/obj3?versionId=null 2024-02-01T10:50:02.217 DEBUG:boto:Method: GET 2024-02-01T10:50:02.217 DEBUG:boto:Path: /vclfcn-63/obj3?versionId=null 2024-02-01T10:50:02.217 DEBUG:boto:Data: 2024-02-01T10:50:02.217 DEBUG:boto:Headers: {} 2024-02-01T10:50:02.218 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:02.218 DEBUG:boto:Port: 8000 2024-02-01T10:50:02.218 DEBUG:boto:Params: {} 2024-02-01T10:50:02.218 DEBUG:boto:Token: None 2024-02-01T10:50:02.218 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:02 GMT /vclfcn-63/obj3?versionId=null 2024-02-01T10:50:02.218 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mEU9CbjivtDQ8uiBJJXqQzpUXaM= 2024-02-01T10:50:02.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mEU9CbjivtDQ8uiBJJXqQzpUXaM=', 'Content-Length': '0'} 2024-02-01T10:50:02.221 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000112b4dca1f532da6-0065bb775a-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:02.221 DEBUG:boto:path=/vclfcn-63/obj3 2024-02-01T10:50:02.222 DEBUG:boto:auth_path=/vclfcn-63/obj3 2024-02-01T10:50:02.222 DEBUG:boto:path=/vclfcn-63/obj3?versionId=null 2024-02-01T10:50:02.222 DEBUG:boto:auth_path=/vclfcn-63/obj3?versionId=null 2024-02-01T10:50:02.222 DEBUG:boto:Method: GET 2024-02-01T10:50:02.222 DEBUG:boto:Path: /vclfcn-63/obj3?versionId=null 2024-02-01T10:50:02.222 DEBUG:boto:Data: 2024-02-01T10:50:02.222 DEBUG:boto:Headers: {} 2024-02-01T10:50:02.222 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:02.222 DEBUG:boto:Port: 8000 2024-02-01T10:50:02.222 DEBUG:boto:Params: {} 2024-02-01T10:50:02.222 DEBUG:boto:Token: None 2024-02-01T10:50:02.223 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:02 GMT /vclfcn-63/obj3?versionId=null 2024-02-01T10:50:02.223 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mEU9CbjivtDQ8uiBJJXqQzpUXaM= 2024-02-01T10:50:02.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mEU9CbjivtDQ8uiBJJXqQzpUXaM=', 'Content-Length': '0'} 2024-02-01T10:50:02.226 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007f886f772e7c2a1a-0065bb775a-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:02.226 DEBUG:boto:path=/vclfcn-63/obj3 2024-02-01T10:50:02.226 DEBUG:boto:auth_path=/vclfcn-63/obj3 2024-02-01T10:50:02.226 DEBUG:boto:path=/vclfcn-63/obj3?versionId=null 2024-02-01T10:50:02.226 DEBUG:boto:auth_path=/vclfcn-63/obj3?versionId=null 2024-02-01T10:50:02.226 DEBUG:boto:Method: HEAD 2024-02-01T10:50:02.227 DEBUG:boto:Path: /vclfcn-63/obj3?versionId=null 2024-02-01T10:50:02.227 DEBUG:boto:Data: 2024-02-01T10:50:02.227 DEBUG:boto:Headers: {} 2024-02-01T10:50:02.227 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:02.227 DEBUG:boto:Port: 8000 2024-02-01T10:50:02.227 DEBUG:boto:Params: {} 2024-02-01T10:50:02.227 DEBUG:boto:Token: None 2024-02-01T10:50:02.227 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:02 GMT /vclfcn-63/obj3?versionId=null 2024-02-01T10:50:02.228 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZwdQTxxr9s6+V7E3cqmKnks7dII= 2024-02-01T10:50:02.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZwdQTxxr9s6+V7E3cqmKnks7dII=', 'Content-Length': '0'} 2024-02-01T10:50:02.231 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eb68b94ba31af5ed-0065bb775a-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:02.231 DEBUG:boto:path=/vclfcn-63/obj3 2024-02-01T10:50:02.231 DEBUG:boto:auth_path=/vclfcn-63/obj3 2024-02-01T10:50:02.231 DEBUG:boto:path=/vclfcn-63/obj3?versionId=null 2024-02-01T10:50:02.231 DEBUG:boto:auth_path=/vclfcn-63/obj3?versionId=null 2024-02-01T10:50:02.231 DEBUG:boto:Method: HEAD 2024-02-01T10:50:02.232 DEBUG:boto:Path: /vclfcn-63/obj3?versionId=null 2024-02-01T10:50:02.232 DEBUG:boto:Data: 2024-02-01T10:50:02.232 DEBUG:boto:Headers: {} 2024-02-01T10:50:02.232 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:02.232 DEBUG:boto:Port: 8000 2024-02-01T10:50:02.232 DEBUG:boto:Params: {} 2024-02-01T10:50:02.232 DEBUG:boto:Token: None 2024-02-01T10:50:02.233 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:02 GMT /vclfcn-63/obj3?versionId=null 2024-02-01T10:50:02.233 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZwdQTxxr9s6+V7E3cqmKnks7dII= 2024-02-01T10:50:02.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZwdQTxxr9s6+V7E3cqmKnks7dII=', 'Content-Length': '0'} 2024-02-01T10:50:02.236 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008f3f67d75a79a945-0065bb775a-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:02.236 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:50:02.237 DEBUG:boto:path=/vclfcn-63/obj3 2024-02-01T10:50:02.237 DEBUG:boto:auth_path=/vclfcn-63/obj3 2024-02-01T10:50:02.237 DEBUG:boto:path=/vclfcn-63/obj3?versionId=null 2024-02-01T10:50:02.237 DEBUG:boto:auth_path=/vclfcn-63/obj3?versionId=null 2024-02-01T10:50:02.237 DEBUG:boto:Method: GET 2024-02-01T10:50:02.237 DEBUG:boto:Path: /vclfcn-63/obj3?versionId=null 2024-02-01T10:50:02.237 DEBUG:boto:Data: 2024-02-01T10:50:02.237 DEBUG:boto:Headers: {} 2024-02-01T10:50:02.237 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:02.237 DEBUG:boto:Port: 8000 2024-02-01T10:50:02.238 DEBUG:boto:Params: {} 2024-02-01T10:50:02.238 DEBUG:boto:Token: None 2024-02-01T10:50:02.238 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:02 GMT /vclfcn-63/obj3?versionId=null 2024-02-01T10:50:02.238 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mEU9CbjivtDQ8uiBJJXqQzpUXaM= 2024-02-01T10:50:02.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mEU9CbjivtDQ8uiBJJXqQzpUXaM=', 'Content-Length': '0'} 2024-02-01T10:50:02.241 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000020caa7832e4d19cc-0065bb775a-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:02.241 DEBUG:boto:path=/vclfcn-63/obj3 2024-02-01T10:50:02.242 DEBUG:boto:auth_path=/vclfcn-63/obj3 2024-02-01T10:50:02.242 DEBUG:boto:path=/vclfcn-63/obj3?versionId=null 2024-02-01T10:50:02.242 DEBUG:boto:auth_path=/vclfcn-63/obj3?versionId=null 2024-02-01T10:50:02.242 DEBUG:boto:Method: GET 2024-02-01T10:50:02.242 DEBUG:boto:Path: /vclfcn-63/obj3?versionId=null 2024-02-01T10:50:02.242 DEBUG:boto:Data: 2024-02-01T10:50:02.243 DEBUG:boto:Headers: {} 2024-02-01T10:50:02.243 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:02.243 DEBUG:boto:Port: 8000 2024-02-01T10:50:02.243 DEBUG:boto:Params: {} 2024-02-01T10:50:02.243 DEBUG:boto:Token: None 2024-02-01T10:50:02.243 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:02 GMT /vclfcn-63/obj3?versionId=null 2024-02-01T10:50:02.243 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mEU9CbjivtDQ8uiBJJXqQzpUXaM= 2024-02-01T10:50:02.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mEU9CbjivtDQ8uiBJJXqQzpUXaM=', 'Content-Length': '0'} 2024-02-01T10:50:02.246 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c85924a58788a655-0065bb775a-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:02.288 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:02.288 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:02.288 DEBUG:boto:path=/vclfcn-63/?prefix=obj3 2024-02-01T10:50:02.289 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj3 2024-02-01T10:50:02.289 DEBUG:boto:Method: GET 2024-02-01T10:50:02.289 DEBUG:boto:Path: /vclfcn-63/?prefix=obj3 2024-02-01T10:50:02.289 DEBUG:boto:Data: 2024-02-01T10:50:02.289 DEBUG:boto:Headers: {} 2024-02-01T10:50:02.289 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:02.289 DEBUG:boto:Port: 8000 2024-02-01T10:50:02.289 DEBUG:boto:Params: {} 2024-02-01T10:50:02.289 DEBUG:boto:Token: None 2024-02-01T10:50:02.290 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:02 GMT /vclfcn-63/ 2024-02-01T10:50:02.290 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ylIwsGcaSaHAU7+2+RTv2egtcow= 2024-02-01T10:50:02.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ylIwsGcaSaHAU7+2+RTv2egtcow=', 'Content-Length': '0'} 2024-02-01T10:50:02.294 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa9c5ad253ae72e4-0065bb775a-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:02.294 DEBUG:boto:b'vclfcn-63obj31000falseobj32024-02-01T10:48:26.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:02.295 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:02.295 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:02.295 DEBUG:boto:path=/vclfcn-63/?prefix=obj3 2024-02-01T10:50:02.295 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj3 2024-02-01T10:50:02.296 DEBUG:boto:Method: GET 2024-02-01T10:50:02.296 DEBUG:boto:Path: /vclfcn-63/?prefix=obj3 2024-02-01T10:50:02.296 DEBUG:boto:Data: 2024-02-01T10:50:02.296 DEBUG:boto:Headers: {} 2024-02-01T10:50:02.296 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:02.296 DEBUG:boto:Port: 8000 2024-02-01T10:50:02.296 DEBUG:boto:Params: {} 2024-02-01T10:50:02.297 DEBUG:boto:Token: None 2024-02-01T10:50:02.297 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:02 GMT /vclfcn-63/ 2024-02-01T10:50:02.297 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ylIwsGcaSaHAU7+2+RTv2egtcow= 2024-02-01T10:50:02.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ylIwsGcaSaHAU7+2+RTv2egtcow=', 'Content-Length': '0'} 2024-02-01T10:50:02.302 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000075be2e3edaa55c3a-0065bb775a-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:02.303 DEBUG:boto:b'vclfcn-63obj31000falseobj32024-02-01T10:48:26.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:02.303 DEBUG:boto:path=/vclfcn-63/obj3 2024-02-01T10:50:02.304 DEBUG:boto:auth_path=/vclfcn-63/obj3 2024-02-01T10:50:02.304 DEBUG:boto:Method: HEAD 2024-02-01T10:50:02.304 DEBUG:boto:Path: /vclfcn-63/obj3 2024-02-01T10:50:02.304 DEBUG:boto:Data: 2024-02-01T10:50:02.304 DEBUG:boto:Headers: {} 2024-02-01T10:50:02.304 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:02.304 DEBUG:boto:Port: 8000 2024-02-01T10:50:02.304 DEBUG:boto:Params: {} 2024-02-01T10:50:02.304 DEBUG:boto:Token: None 2024-02-01T10:50:02.305 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:02 GMT /vclfcn-63/obj3 2024-02-01T10:50:02.305 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:p2fOus2YZZVXKxP9I3d0SOFpNUA= 2024-02-01T10:50:02.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:p2fOus2YZZVXKxP9I3d0SOFpNUA=', 'Content-Length': '0'} 2024-02-01T10:50:02.309 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000003c5e6525c10cf38-0065bb775a-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:02.310 DEBUG:boto:path=/vclfcn-63/obj3 2024-02-01T10:50:02.310 DEBUG:boto:auth_path=/vclfcn-63/obj3 2024-02-01T10:50:02.310 DEBUG:boto:Method: HEAD 2024-02-01T10:50:02.310 DEBUG:boto:Path: /vclfcn-63/obj3 2024-02-01T10:50:02.310 DEBUG:boto:Data: 2024-02-01T10:50:02.310 DEBUG:boto:Headers: {} 2024-02-01T10:50:02.310 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:02.310 DEBUG:boto:Port: 8000 2024-02-01T10:50:02.310 DEBUG:boto:Params: {} 2024-02-01T10:50:02.310 DEBUG:boto:Token: None 2024-02-01T10:50:02.311 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:02 GMT /vclfcn-63/obj3 2024-02-01T10:50:02.311 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:p2fOus2YZZVXKxP9I3d0SOFpNUA= 2024-02-01T10:50:02.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:p2fOus2YZZVXKxP9I3d0SOFpNUA=', 'Content-Length': '0'} 2024-02-01T10:50:02.313 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006023de2da5112c82-0065bb775a-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:02.314 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:50:02.314 DEBUG:boto:path=/vclfcn-63/obj3 2024-02-01T10:50:02.314 DEBUG:boto:auth_path=/vclfcn-63/obj3 2024-02-01T10:50:02.314 DEBUG:boto:Method: GET 2024-02-01T10:50:02.314 DEBUG:boto:Path: /vclfcn-63/obj3 2024-02-01T10:50:02.314 DEBUG:boto:Data: 2024-02-01T10:50:02.314 DEBUG:boto:Headers: {} 2024-02-01T10:50:02.314 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:02.315 DEBUG:boto:Port: 8000 2024-02-01T10:50:02.315 DEBUG:boto:Params: {} 2024-02-01T10:50:02.315 DEBUG:boto:Token: None 2024-02-01T10:50:02.315 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:02 GMT /vclfcn-63/obj3 2024-02-01T10:50:02.315 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+F4DHzu6Id58o2I0h+tcG/dC1vE= 2024-02-01T10:50:02.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+F4DHzu6Id58o2I0h+tcG/dC1vE=', 'Content-Length': '0'} 2024-02-01T10:50:02.318 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000697e0689fbca01e-0065bb775a-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:02.318 DEBUG:boto:path=/vclfcn-63/obj3 2024-02-01T10:50:02.318 DEBUG:boto:auth_path=/vclfcn-63/obj3 2024-02-01T10:50:02.318 DEBUG:boto:Method: GET 2024-02-01T10:50:02.318 DEBUG:boto:Path: /vclfcn-63/obj3 2024-02-01T10:50:02.318 DEBUG:boto:Data: 2024-02-01T10:50:02.318 DEBUG:boto:Headers: {} 2024-02-01T10:50:02.318 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:02.319 DEBUG:boto:Port: 8000 2024-02-01T10:50:02.319 DEBUG:boto:Params: {} 2024-02-01T10:50:02.319 DEBUG:boto:Token: None 2024-02-01T10:50:02.319 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:02 GMT /vclfcn-63/obj3 2024-02-01T10:50:02.319 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+F4DHzu6Id58o2I0h+tcG/dC1vE= 2024-02-01T10:50:02.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+F4DHzu6Id58o2I0h+tcG/dC1vE=', 'Content-Length': '0'} 2024-02-01T10:50:02.322 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e13648e3e1013567-0065bb775a-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:02.364 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:02.365 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:02.365 DEBUG:boto:path=/vclfcn-63/?prefix=obj3 2024-02-01T10:50:02.366 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj3 2024-02-01T10:50:02.366 DEBUG:boto:Method: GET 2024-02-01T10:50:02.366 DEBUG:boto:Path: /vclfcn-63/?prefix=obj3 2024-02-01T10:50:02.366 DEBUG:boto:Data: 2024-02-01T10:50:02.366 DEBUG:boto:Headers: {} 2024-02-01T10:50:02.366 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:02.366 DEBUG:boto:Port: 8000 2024-02-01T10:50:02.366 DEBUG:boto:Params: {} 2024-02-01T10:50:02.366 DEBUG:boto:Token: None 2024-02-01T10:50:02.367 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:02 GMT /vclfcn-63/ 2024-02-01T10:50:02.367 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ylIwsGcaSaHAU7+2+RTv2egtcow= 2024-02-01T10:50:02.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ylIwsGcaSaHAU7+2+RTv2egtcow=', 'Content-Length': '0'} 2024-02-01T10:50:02.371 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005313016fbd3c162f-0065bb775a-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:02.371 DEBUG:boto:b'vclfcn-63obj31000falseobj32024-02-01T10:48:26.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:02.372 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:02.372 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:02.372 DEBUG:boto:path=/vclfcn-63/?prefix=obj3 2024-02-01T10:50:02.372 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj3 2024-02-01T10:50:02.372 DEBUG:boto:Method: GET 2024-02-01T10:50:02.372 DEBUG:boto:Path: /vclfcn-63/?prefix=obj3 2024-02-01T10:50:02.372 DEBUG:boto:Data: 2024-02-01T10:50:02.373 DEBUG:boto:Headers: {} 2024-02-01T10:50:02.373 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:02.373 DEBUG:boto:Port: 8000 2024-02-01T10:50:02.373 DEBUG:boto:Params: {} 2024-02-01T10:50:02.373 DEBUG:boto:Token: None 2024-02-01T10:50:02.373 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:02 GMT /vclfcn-63/ 2024-02-01T10:50:02.373 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ylIwsGcaSaHAU7+2+RTv2egtcow= 2024-02-01T10:50:02.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ylIwsGcaSaHAU7+2+RTv2egtcow=', 'Content-Length': '0'} 2024-02-01T10:50:02.377 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000839d416303ece229-0065bb775a-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:02.378 DEBUG:boto:b'vclfcn-63obj31000falseobj32024-02-01T10:48:26.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:02.378 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:50:02.378 DEBUG:boto:path=/vclfcn-63/obj4 2024-02-01T10:50:02.378 DEBUG:boto:auth_path=/vclfcn-63/obj4 2024-02-01T10:50:02.378 DEBUG:boto:path=/vclfcn-63/obj4?versionId=null 2024-02-01T10:50:02.378 DEBUG:boto:auth_path=/vclfcn-63/obj4?versionId=null 2024-02-01T10:50:02.378 DEBUG:boto:Method: GET 2024-02-01T10:50:02.379 DEBUG:boto:Path: /vclfcn-63/obj4?versionId=null 2024-02-01T10:50:02.379 DEBUG:boto:Data: 2024-02-01T10:50:02.379 DEBUG:boto:Headers: {} 2024-02-01T10:50:02.379 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:02.379 DEBUG:boto:Port: 8000 2024-02-01T10:50:02.379 DEBUG:boto:Params: {} 2024-02-01T10:50:02.379 DEBUG:boto:Token: None 2024-02-01T10:50:02.379 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:02 GMT /vclfcn-63/obj4?versionId=null 2024-02-01T10:50:02.379 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YxRBvhmNIoKDMP6OzNrGFKU/J+8= 2024-02-01T10:50:02.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YxRBvhmNIoKDMP6OzNrGFKU/J+8=', 'Content-Length': '0'} 2024-02-01T10:50:02.382 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d76e636d42cd339e-0065bb775a-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:02.428 DEBUG:boto:path=/vclfcn-63/obj4 2024-02-01T10:50:02.428 DEBUG:boto:auth_path=/vclfcn-63/obj4 2024-02-01T10:50:02.428 DEBUG:boto:path=/vclfcn-63/obj4?versionId=null 2024-02-01T10:50:02.428 DEBUG:boto:auth_path=/vclfcn-63/obj4?versionId=null 2024-02-01T10:50:02.429 DEBUG:boto:Method: GET 2024-02-01T10:50:02.429 DEBUG:boto:Path: /vclfcn-63/obj4?versionId=null 2024-02-01T10:50:02.429 DEBUG:boto:Data: 2024-02-01T10:50:02.429 DEBUG:boto:Headers: {} 2024-02-01T10:50:02.429 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:02.429 DEBUG:boto:Port: 8000 2024-02-01T10:50:02.429 DEBUG:boto:Params: {} 2024-02-01T10:50:02.429 DEBUG:boto:Token: None 2024-02-01T10:50:02.429 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:02 GMT /vclfcn-63/obj4?versionId=null 2024-02-01T10:50:02.430 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YxRBvhmNIoKDMP6OzNrGFKU/J+8= 2024-02-01T10:50:02.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YxRBvhmNIoKDMP6OzNrGFKU/J+8=', 'Content-Length': '0'} 2024-02-01T10:50:02.433 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dbdfbb958bad2bbc-0065bb775a-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:02.433 DEBUG:boto:path=/vclfcn-63/obj4 2024-02-01T10:50:02.433 DEBUG:boto:auth_path=/vclfcn-63/obj4 2024-02-01T10:50:02.433 DEBUG:boto:path=/vclfcn-63/obj4?versionId=null 2024-02-01T10:50:02.433 DEBUG:boto:auth_path=/vclfcn-63/obj4?versionId=null 2024-02-01T10:50:02.433 DEBUG:boto:Method: HEAD 2024-02-01T10:50:02.434 DEBUG:boto:Path: /vclfcn-63/obj4?versionId=null 2024-02-01T10:50:02.434 DEBUG:boto:Data: 2024-02-01T10:50:02.434 DEBUG:boto:Headers: {} 2024-02-01T10:50:02.434 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:02.434 DEBUG:boto:Port: 8000 2024-02-01T10:50:02.434 DEBUG:boto:Params: {} 2024-02-01T10:50:02.434 DEBUG:boto:Token: None 2024-02-01T10:50:02.434 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:02 GMT /vclfcn-63/obj4?versionId=null 2024-02-01T10:50:02.434 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fVzkqqaGBxX1G8fbCbVsPrry4mA= 2024-02-01T10:50:02.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fVzkqqaGBxX1G8fbCbVsPrry4mA=', 'Content-Length': '0'} 2024-02-01T10:50:02.437 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009b8d77d9db3b9e1d-0065bb775a-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:02.438 DEBUG:boto:path=/vclfcn-63/obj4 2024-02-01T10:50:02.438 DEBUG:boto:auth_path=/vclfcn-63/obj4 2024-02-01T10:50:02.438 DEBUG:boto:path=/vclfcn-63/obj4?versionId=null 2024-02-01T10:50:02.438 DEBUG:boto:auth_path=/vclfcn-63/obj4?versionId=null 2024-02-01T10:50:02.438 DEBUG:boto:Method: HEAD 2024-02-01T10:50:02.438 DEBUG:boto:Path: /vclfcn-63/obj4?versionId=null 2024-02-01T10:50:02.438 DEBUG:boto:Data: 2024-02-01T10:50:02.438 DEBUG:boto:Headers: {} 2024-02-01T10:50:02.439 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:02.439 DEBUG:boto:Port: 8000 2024-02-01T10:50:02.439 DEBUG:boto:Params: {} 2024-02-01T10:50:02.439 DEBUG:boto:Token: None 2024-02-01T10:50:02.439 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:02 GMT /vclfcn-63/obj4?versionId=null 2024-02-01T10:50:02.439 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fVzkqqaGBxX1G8fbCbVsPrry4mA= 2024-02-01T10:50:02.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fVzkqqaGBxX1G8fbCbVsPrry4mA=', 'Content-Length': '0'} 2024-02-01T10:50:02.442 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a343742735f23ced-0065bb775a-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:02.442 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:50:02.442 DEBUG:boto:path=/vclfcn-63/obj4 2024-02-01T10:50:02.442 DEBUG:boto:auth_path=/vclfcn-63/obj4 2024-02-01T10:50:02.442 DEBUG:boto:path=/vclfcn-63/obj4?versionId=null 2024-02-01T10:50:02.442 DEBUG:boto:auth_path=/vclfcn-63/obj4?versionId=null 2024-02-01T10:50:02.442 DEBUG:boto:Method: GET 2024-02-01T10:50:02.442 DEBUG:boto:Path: /vclfcn-63/obj4?versionId=null 2024-02-01T10:50:02.443 DEBUG:boto:Data: 2024-02-01T10:50:02.443 DEBUG:boto:Headers: {} 2024-02-01T10:50:02.443 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:02.443 DEBUG:boto:Port: 8000 2024-02-01T10:50:02.443 DEBUG:boto:Params: {} 2024-02-01T10:50:02.443 DEBUG:boto:Token: None 2024-02-01T10:50:02.443 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:02 GMT /vclfcn-63/obj4?versionId=null 2024-02-01T10:50:02.443 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YxRBvhmNIoKDMP6OzNrGFKU/J+8= 2024-02-01T10:50:02.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YxRBvhmNIoKDMP6OzNrGFKU/J+8=', 'Content-Length': '0'} 2024-02-01T10:50:02.446 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a3be2fe4277e390d-0065bb775a-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:02.447 DEBUG:boto:path=/vclfcn-63/obj4 2024-02-01T10:50:02.447 DEBUG:boto:auth_path=/vclfcn-63/obj4 2024-02-01T10:50:02.447 DEBUG:boto:path=/vclfcn-63/obj4?versionId=null 2024-02-01T10:50:02.447 DEBUG:boto:auth_path=/vclfcn-63/obj4?versionId=null 2024-02-01T10:50:02.447 DEBUG:boto:Method: GET 2024-02-01T10:50:02.447 DEBUG:boto:Path: /vclfcn-63/obj4?versionId=null 2024-02-01T10:50:02.447 DEBUG:boto:Data: 2024-02-01T10:50:02.447 DEBUG:boto:Headers: {} 2024-02-01T10:50:02.447 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:02.447 DEBUG:boto:Port: 8000 2024-02-01T10:50:02.447 DEBUG:boto:Params: {} 2024-02-01T10:50:02.448 DEBUG:boto:Token: None 2024-02-01T10:50:02.448 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:02 GMT /vclfcn-63/obj4?versionId=null 2024-02-01T10:50:02.448 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YxRBvhmNIoKDMP6OzNrGFKU/J+8= 2024-02-01T10:50:02.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YxRBvhmNIoKDMP6OzNrGFKU/J+8=', 'Content-Length': '0'} 2024-02-01T10:50:02.450 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003f2f72fa8b8e71aa-0065bb775a-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:02.492 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:02.492 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:02.492 DEBUG:boto:path=/vclfcn-63/?prefix=obj4 2024-02-01T10:50:02.492 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj4 2024-02-01T10:50:02.493 DEBUG:boto:Method: GET 2024-02-01T10:50:02.493 DEBUG:boto:Path: /vclfcn-63/?prefix=obj4 2024-02-01T10:50:02.493 DEBUG:boto:Data: 2024-02-01T10:50:02.493 DEBUG:boto:Headers: {} 2024-02-01T10:50:02.493 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:02.493 DEBUG:boto:Port: 8000 2024-02-01T10:50:02.493 DEBUG:boto:Params: {} 2024-02-01T10:50:02.493 DEBUG:boto:Token: None 2024-02-01T10:50:02.493 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:02 GMT /vclfcn-63/ 2024-02-01T10:50:02.493 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ylIwsGcaSaHAU7+2+RTv2egtcow= 2024-02-01T10:50:02.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ylIwsGcaSaHAU7+2+RTv2egtcow=', 'Content-Length': '0'} 2024-02-01T10:50:02.503 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000098793a539be5fe93-0065bb775a-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:02.503 DEBUG:boto:b'vclfcn-63obj41000falseobj42024-02-01T10:48:26.575Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:02.503 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:02.504 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:02.504 DEBUG:boto:path=/vclfcn-63/?prefix=obj4 2024-02-01T10:50:02.504 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj4 2024-02-01T10:50:02.504 DEBUG:boto:Method: GET 2024-02-01T10:50:02.504 DEBUG:boto:Path: /vclfcn-63/?prefix=obj4 2024-02-01T10:50:02.504 DEBUG:boto:Data: 2024-02-01T10:50:02.504 DEBUG:boto:Headers: {} 2024-02-01T10:50:02.504 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:02.504 DEBUG:boto:Port: 8000 2024-02-01T10:50:02.504 DEBUG:boto:Params: {} 2024-02-01T10:50:02.505 DEBUG:boto:Token: None 2024-02-01T10:50:02.505 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:02 GMT /vclfcn-63/ 2024-02-01T10:50:02.505 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ylIwsGcaSaHAU7+2+RTv2egtcow= 2024-02-01T10:50:02.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ylIwsGcaSaHAU7+2+RTv2egtcow=', 'Content-Length': '0'} 2024-02-01T10:50:02.510 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f2eb5690a55df0b-0065bb775a-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:02.510 DEBUG:boto:b'vclfcn-63obj41000falseobj42024-02-01T10:48:26.575Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:02.510 DEBUG:boto:path=/vclfcn-63/obj4 2024-02-01T10:50:02.511 DEBUG:boto:auth_path=/vclfcn-63/obj4 2024-02-01T10:50:02.511 DEBUG:boto:Method: HEAD 2024-02-01T10:50:02.511 DEBUG:boto:Path: /vclfcn-63/obj4 2024-02-01T10:50:02.511 DEBUG:boto:Data: 2024-02-01T10:50:02.511 DEBUG:boto:Headers: {} 2024-02-01T10:50:02.511 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:02.511 DEBUG:boto:Port: 8000 2024-02-01T10:50:02.511 DEBUG:boto:Params: {} 2024-02-01T10:50:02.511 DEBUG:boto:Token: None 2024-02-01T10:50:02.511 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:02 GMT /vclfcn-63/obj4 2024-02-01T10:50:02.512 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mjjCTswpghy3iteO/yBaOMcEnWg= 2024-02-01T10:50:02.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mjjCTswpghy3iteO/yBaOMcEnWg=', 'Content-Length': '0'} 2024-02-01T10:50:02.514 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003ed93ede55f8cb94-0065bb775a-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:02.514 DEBUG:boto:path=/vclfcn-63/obj4 2024-02-01T10:50:02.515 DEBUG:boto:auth_path=/vclfcn-63/obj4 2024-02-01T10:50:02.515 DEBUG:boto:Method: HEAD 2024-02-01T10:50:02.515 DEBUG:boto:Path: /vclfcn-63/obj4 2024-02-01T10:50:02.515 DEBUG:boto:Data: 2024-02-01T10:50:02.515 DEBUG:boto:Headers: {} 2024-02-01T10:50:02.515 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:02.515 DEBUG:boto:Port: 8000 2024-02-01T10:50:02.515 DEBUG:boto:Params: {} 2024-02-01T10:50:02.515 DEBUG:boto:Token: None 2024-02-01T10:50:02.516 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:02 GMT /vclfcn-63/obj4 2024-02-01T10:50:02.516 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mjjCTswpghy3iteO/yBaOMcEnWg= 2024-02-01T10:50:02.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mjjCTswpghy3iteO/yBaOMcEnWg=', 'Content-Length': '0'} 2024-02-01T10:50:02.518 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d854ff090716c1fd-0065bb775a-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:02.519 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:50:02.519 DEBUG:boto:path=/vclfcn-63/obj4 2024-02-01T10:50:02.519 DEBUG:boto:auth_path=/vclfcn-63/obj4 2024-02-01T10:50:02.519 DEBUG:boto:Method: GET 2024-02-01T10:50:02.519 DEBUG:boto:Path: /vclfcn-63/obj4 2024-02-01T10:50:02.519 DEBUG:boto:Data: 2024-02-01T10:50:02.519 DEBUG:boto:Headers: {} 2024-02-01T10:50:02.519 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:02.519 DEBUG:boto:Port: 8000 2024-02-01T10:50:02.520 DEBUG:boto:Params: {} 2024-02-01T10:50:02.520 DEBUG:boto:Token: None 2024-02-01T10:50:02.520 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:02 GMT /vclfcn-63/obj4 2024-02-01T10:50:02.520 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+dmogqacIWKkOS1/ZkatmDblvRc= 2024-02-01T10:50:02.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+dmogqacIWKkOS1/ZkatmDblvRc=', 'Content-Length': '0'} 2024-02-01T10:50:02.523 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dcfc60998665ec08-0065bb775a-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:02.564 DEBUG:boto:path=/vclfcn-63/obj4 2024-02-01T10:50:02.564 DEBUG:boto:auth_path=/vclfcn-63/obj4 2024-02-01T10:50:02.564 DEBUG:boto:Method: GET 2024-02-01T10:50:02.564 DEBUG:boto:Path: /vclfcn-63/obj4 2024-02-01T10:50:02.565 DEBUG:boto:Data: 2024-02-01T10:50:02.565 DEBUG:boto:Headers: {} 2024-02-01T10:50:02.565 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:02.565 DEBUG:boto:Port: 8000 2024-02-01T10:50:02.565 DEBUG:boto:Params: {} 2024-02-01T10:50:02.565 DEBUG:boto:Token: None 2024-02-01T10:50:02.565 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:02 GMT /vclfcn-63/obj4 2024-02-01T10:50:02.565 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+dmogqacIWKkOS1/ZkatmDblvRc= 2024-02-01T10:50:02.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+dmogqacIWKkOS1/ZkatmDblvRc=', 'Content-Length': '0'} 2024-02-01T10:50:02.568 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003bb1292669455927-0065bb775a-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:02.612 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:02.612 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:02.612 DEBUG:boto:path=/vclfcn-63/?prefix=obj4 2024-02-01T10:50:02.613 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj4 2024-02-01T10:50:02.613 DEBUG:boto:Method: GET 2024-02-01T10:50:02.613 DEBUG:boto:Path: /vclfcn-63/?prefix=obj4 2024-02-01T10:50:02.613 DEBUG:boto:Data: 2024-02-01T10:50:02.613 DEBUG:boto:Headers: {} 2024-02-01T10:50:02.613 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:02.613 DEBUG:boto:Port: 8000 2024-02-01T10:50:02.613 DEBUG:boto:Params: {} 2024-02-01T10:50:02.613 DEBUG:boto:Token: None 2024-02-01T10:50:02.613 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:02 GMT /vclfcn-63/ 2024-02-01T10:50:02.614 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ylIwsGcaSaHAU7+2+RTv2egtcow= 2024-02-01T10:50:02.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ylIwsGcaSaHAU7+2+RTv2egtcow=', 'Content-Length': '0'} 2024-02-01T10:50:02.618 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000887bd75cd302edc6-0065bb775a-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:02.618 DEBUG:boto:b'vclfcn-63obj41000falseobj42024-02-01T10:48:26.575Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:02.618 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:02.618 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:02.619 DEBUG:boto:path=/vclfcn-63/?prefix=obj4 2024-02-01T10:50:02.619 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj4 2024-02-01T10:50:02.619 DEBUG:boto:Method: GET 2024-02-01T10:50:02.619 DEBUG:boto:Path: /vclfcn-63/?prefix=obj4 2024-02-01T10:50:02.619 DEBUG:boto:Data: 2024-02-01T10:50:02.619 DEBUG:boto:Headers: {} 2024-02-01T10:50:02.619 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:02.619 DEBUG:boto:Port: 8000 2024-02-01T10:50:02.619 DEBUG:boto:Params: {} 2024-02-01T10:50:02.619 DEBUG:boto:Token: None 2024-02-01T10:50:02.620 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:02 GMT /vclfcn-63/ 2024-02-01T10:50:02.620 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ylIwsGcaSaHAU7+2+RTv2egtcow= 2024-02-01T10:50:02.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ylIwsGcaSaHAU7+2+RTv2egtcow=', 'Content-Length': '0'} 2024-02-01T10:50:02.624 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000044830dc4ddeed52b-0065bb775a-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:02.672 DEBUG:boto:b'vclfcn-63obj41000falseobj42024-02-01T10:48:26.575Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:02.672 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-02-01T10:50:02.673 DEBUG:boto:path=/vclfcn-63/obj5 2024-02-01T10:50:02.673 DEBUG:boto:auth_path=/vclfcn-63/obj5 2024-02-01T10:50:02.673 DEBUG:boto:path=/vclfcn-63/obj5?versionId=null 2024-02-01T10:50:02.673 DEBUG:boto:auth_path=/vclfcn-63/obj5?versionId=null 2024-02-01T10:50:02.673 DEBUG:boto:Method: GET 2024-02-01T10:50:02.673 DEBUG:boto:Path: /vclfcn-63/obj5?versionId=null 2024-02-01T10:50:02.673 DEBUG:boto:Data: 2024-02-01T10:50:02.673 DEBUG:boto:Headers: {} 2024-02-01T10:50:02.673 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:02.673 DEBUG:boto:Port: 8000 2024-02-01T10:50:02.674 DEBUG:boto:Params: {} 2024-02-01T10:50:02.674 DEBUG:boto:Token: None 2024-02-01T10:50:02.674 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:02 GMT /vclfcn-63/obj5?versionId=null 2024-02-01T10:50:02.674 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NbgaTxgFkaXRYH2aQ/RCjeIjfCI= 2024-02-01T10:50:02.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NbgaTxgFkaXRYH2aQ/RCjeIjfCI=', 'Content-Length': '0'} 2024-02-01T10:50:02.677 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004da273ac92280b97-0065bb775a-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:02.677 DEBUG:boto:path=/vclfcn-63/obj5 2024-02-01T10:50:02.677 DEBUG:boto:auth_path=/vclfcn-63/obj5 2024-02-01T10:50:02.677 DEBUG:boto:path=/vclfcn-63/obj5?versionId=null 2024-02-01T10:50:02.678 DEBUG:boto:auth_path=/vclfcn-63/obj5?versionId=null 2024-02-01T10:50:02.678 DEBUG:boto:Method: GET 2024-02-01T10:50:02.678 DEBUG:boto:Path: /vclfcn-63/obj5?versionId=null 2024-02-01T10:50:02.678 DEBUG:boto:Data: 2024-02-01T10:50:02.678 DEBUG:boto:Headers: {} 2024-02-01T10:50:02.678 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:02.678 DEBUG:boto:Port: 8000 2024-02-01T10:50:02.678 DEBUG:boto:Params: {} 2024-02-01T10:50:02.678 DEBUG:boto:Token: None 2024-02-01T10:50:02.678 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:02 GMT /vclfcn-63/obj5?versionId=null 2024-02-01T10:50:02.678 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NbgaTxgFkaXRYH2aQ/RCjeIjfCI= 2024-02-01T10:50:02.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NbgaTxgFkaXRYH2aQ/RCjeIjfCI=', 'Content-Length': '0'} 2024-02-01T10:50:02.682 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000070438027fe193ef5-0065bb775a-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:02.682 DEBUG:boto:path=/vclfcn-63/obj5 2024-02-01T10:50:02.682 DEBUG:boto:auth_path=/vclfcn-63/obj5 2024-02-01T10:50:02.682 DEBUG:boto:path=/vclfcn-63/obj5?versionId=null 2024-02-01T10:50:02.682 DEBUG:boto:auth_path=/vclfcn-63/obj5?versionId=null 2024-02-01T10:50:02.682 DEBUG:boto:Method: HEAD 2024-02-01T10:50:02.682 DEBUG:boto:Path: /vclfcn-63/obj5?versionId=null 2024-02-01T10:50:02.682 DEBUG:boto:Data: 2024-02-01T10:50:02.683 DEBUG:boto:Headers: {} 2024-02-01T10:50:02.683 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:02.683 DEBUG:boto:Port: 8000 2024-02-01T10:50:02.683 DEBUG:boto:Params: {} 2024-02-01T10:50:02.683 DEBUG:boto:Token: None 2024-02-01T10:50:02.683 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:02 GMT /vclfcn-63/obj5?versionId=null 2024-02-01T10:50:02.683 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YFfkMPK4BaWqpGx7wyYz6jFCQMQ= 2024-02-01T10:50:02.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YFfkMPK4BaWqpGx7wyYz6jFCQMQ=', 'Content-Length': '0'} 2024-02-01T10:50:02.686 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cf39f0a20ce5d9e0-0065bb775a-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:02.686 DEBUG:boto:path=/vclfcn-63/obj5 2024-02-01T10:50:02.686 DEBUG:boto:auth_path=/vclfcn-63/obj5 2024-02-01T10:50:02.686 DEBUG:boto:path=/vclfcn-63/obj5?versionId=null 2024-02-01T10:50:02.686 DEBUG:boto:auth_path=/vclfcn-63/obj5?versionId=null 2024-02-01T10:50:02.686 DEBUG:boto:Method: HEAD 2024-02-01T10:50:02.686 DEBUG:boto:Path: /vclfcn-63/obj5?versionId=null 2024-02-01T10:50:02.687 DEBUG:boto:Data: 2024-02-01T10:50:02.687 DEBUG:boto:Headers: {} 2024-02-01T10:50:02.687 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:02.687 DEBUG:boto:Port: 8000 2024-02-01T10:50:02.687 DEBUG:boto:Params: {} 2024-02-01T10:50:02.687 DEBUG:boto:Token: None 2024-02-01T10:50:02.687 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:02 GMT /vclfcn-63/obj5?versionId=null 2024-02-01T10:50:02.687 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:YFfkMPK4BaWqpGx7wyYz6jFCQMQ= 2024-02-01T10:50:02.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:YFfkMPK4BaWqpGx7wyYz6jFCQMQ=', 'Content-Length': '0'} 2024-02-01T10:50:02.690 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000168da98f02b7891e-0065bb775a-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:02.690 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-02-01T10:50:02.690 DEBUG:boto:path=/vclfcn-63/obj5 2024-02-01T10:50:02.690 DEBUG:boto:auth_path=/vclfcn-63/obj5 2024-02-01T10:50:02.691 DEBUG:boto:path=/vclfcn-63/obj5?versionId=null 2024-02-01T10:50:02.691 DEBUG:boto:auth_path=/vclfcn-63/obj5?versionId=null 2024-02-01T10:50:02.691 DEBUG:boto:Method: GET 2024-02-01T10:50:02.691 DEBUG:boto:Path: /vclfcn-63/obj5?versionId=null 2024-02-01T10:50:02.691 DEBUG:boto:Data: 2024-02-01T10:50:02.691 DEBUG:boto:Headers: {} 2024-02-01T10:50:02.691 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:02.691 DEBUG:boto:Port: 8000 2024-02-01T10:50:02.691 DEBUG:boto:Params: {} 2024-02-01T10:50:02.691 DEBUG:boto:Token: None 2024-02-01T10:50:02.692 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:02 GMT /vclfcn-63/obj5?versionId=null 2024-02-01T10:50:02.692 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NbgaTxgFkaXRYH2aQ/RCjeIjfCI= 2024-02-01T10:50:02.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NbgaTxgFkaXRYH2aQ/RCjeIjfCI=', 'Content-Length': '0'} 2024-02-01T10:50:02.695 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006abe5ab2c37144d4-0065bb775a-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:02.695 DEBUG:boto:path=/vclfcn-63/obj5 2024-02-01T10:50:02.696 DEBUG:boto:auth_path=/vclfcn-63/obj5 2024-02-01T10:50:02.696 DEBUG:boto:path=/vclfcn-63/obj5?versionId=null 2024-02-01T10:50:02.696 DEBUG:boto:auth_path=/vclfcn-63/obj5?versionId=null 2024-02-01T10:50:02.696 DEBUG:boto:Method: GET 2024-02-01T10:50:02.696 DEBUG:boto:Path: /vclfcn-63/obj5?versionId=null 2024-02-01T10:50:02.696 DEBUG:boto:Data: 2024-02-01T10:50:02.696 DEBUG:boto:Headers: {} 2024-02-01T10:50:02.696 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:02.696 DEBUG:boto:Port: 8000 2024-02-01T10:50:02.696 DEBUG:boto:Params: {} 2024-02-01T10:50:02.697 DEBUG:boto:Token: None 2024-02-01T10:50:02.697 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:02 GMT /vclfcn-63/obj5?versionId=null 2024-02-01T10:50:02.697 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NbgaTxgFkaXRYH2aQ/RCjeIjfCI= 2024-02-01T10:50:02.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NbgaTxgFkaXRYH2aQ/RCjeIjfCI=', 'Content-Length': '0'} 2024-02-01T10:50:02.699 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003cab816fccd07be7-0065bb775a-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:02.744 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:02.744 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:02.744 DEBUG:boto:path=/vclfcn-63/?prefix=obj5 2024-02-01T10:50:02.744 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj5 2024-02-01T10:50:02.745 DEBUG:boto:Method: GET 2024-02-01T10:50:02.745 DEBUG:boto:Path: /vclfcn-63/?prefix=obj5 2024-02-01T10:50:02.745 DEBUG:boto:Data: 2024-02-01T10:50:02.745 DEBUG:boto:Headers: {} 2024-02-01T10:50:02.745 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:02.745 DEBUG:boto:Port: 8000 2024-02-01T10:50:02.745 DEBUG:boto:Params: {} 2024-02-01T10:50:02.745 DEBUG:boto:Token: None 2024-02-01T10:50:02.745 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:02 GMT /vclfcn-63/ 2024-02-01T10:50:02.746 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ylIwsGcaSaHAU7+2+RTv2egtcow= 2024-02-01T10:50:02.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ylIwsGcaSaHAU7+2+RTv2egtcow=', 'Content-Length': '0'} 2024-02-01T10:50:02.750 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df2c53d4b1b9affa-0065bb775a-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:02.750 DEBUG:boto:b'vclfcn-63obj51000falseobj52024-02-01T10:49:22.255Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:02.750 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:02.751 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:02.751 DEBUG:boto:path=/vclfcn-63/?prefix=obj5 2024-02-01T10:50:02.751 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj5 2024-02-01T10:50:02.751 DEBUG:boto:Method: GET 2024-02-01T10:50:02.751 DEBUG:boto:Path: /vclfcn-63/?prefix=obj5 2024-02-01T10:50:02.751 DEBUG:boto:Data: 2024-02-01T10:50:02.751 DEBUG:boto:Headers: {} 2024-02-01T10:50:02.751 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:02.751 DEBUG:boto:Port: 8000 2024-02-01T10:50:02.751 DEBUG:boto:Params: {} 2024-02-01T10:50:02.752 DEBUG:boto:Token: None 2024-02-01T10:50:02.752 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:02 GMT /vclfcn-63/ 2024-02-01T10:50:02.752 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ylIwsGcaSaHAU7+2+RTv2egtcow= 2024-02-01T10:50:02.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ylIwsGcaSaHAU7+2+RTv2egtcow=', 'Content-Length': '0'} 2024-02-01T10:50:02.757 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d14daf6341380471-0065bb775a-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:02.800 DEBUG:boto:b'vclfcn-63obj51000falseobj52024-02-01T10:49:22.255Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:02.800 DEBUG:boto:path=/vclfcn-63/obj5 2024-02-01T10:50:02.801 DEBUG:boto:auth_path=/vclfcn-63/obj5 2024-02-01T10:50:02.801 DEBUG:boto:Method: HEAD 2024-02-01T10:50:02.801 DEBUG:boto:Path: /vclfcn-63/obj5 2024-02-01T10:50:02.801 DEBUG:boto:Data: 2024-02-01T10:50:02.801 DEBUG:boto:Headers: {} 2024-02-01T10:50:02.801 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:02.801 DEBUG:boto:Port: 8000 2024-02-01T10:50:02.801 DEBUG:boto:Params: {} 2024-02-01T10:50:02.801 DEBUG:boto:Token: None 2024-02-01T10:50:02.802 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:02 GMT /vclfcn-63/obj5 2024-02-01T10:50:02.802 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:6gzRP/XTKAGu+RJUC1RPcn5GMJM= 2024-02-01T10:50:02.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:6gzRP/XTKAGu+RJUC1RPcn5GMJM=', 'Content-Length': '0'} 2024-02-01T10:50:02.805 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fdc13aff3b0fa2c7-0065bb775a-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:02.805 DEBUG:boto:path=/vclfcn-63/obj5 2024-02-01T10:50:02.805 DEBUG:boto:auth_path=/vclfcn-63/obj5 2024-02-01T10:50:02.805 DEBUG:boto:Method: HEAD 2024-02-01T10:50:02.805 DEBUG:boto:Path: /vclfcn-63/obj5 2024-02-01T10:50:02.805 DEBUG:boto:Data: 2024-02-01T10:50:02.806 DEBUG:boto:Headers: {} 2024-02-01T10:50:02.806 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:02.806 DEBUG:boto:Port: 8000 2024-02-01T10:50:02.806 DEBUG:boto:Params: {} 2024-02-01T10:50:02.806 DEBUG:boto:Token: None 2024-02-01T10:50:02.806 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:02 GMT /vclfcn-63/obj5 2024-02-01T10:50:02.806 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:6gzRP/XTKAGu+RJUC1RPcn5GMJM= 2024-02-01T10:50:02.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:6gzRP/XTKAGu+RJUC1RPcn5GMJM=', 'Content-Length': '0'} 2024-02-01T10:50:02.809 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f3db88de54833816-0065bb775a-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:02.809 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-02-01T10:50:02.810 DEBUG:boto:path=/vclfcn-63/obj5 2024-02-01T10:50:02.810 DEBUG:boto:auth_path=/vclfcn-63/obj5 2024-02-01T10:50:02.810 DEBUG:boto:Method: GET 2024-02-01T10:50:02.810 DEBUG:boto:Path: /vclfcn-63/obj5 2024-02-01T10:50:02.810 DEBUG:boto:Data: 2024-02-01T10:50:02.810 DEBUG:boto:Headers: {} 2024-02-01T10:50:02.810 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:02.810 DEBUG:boto:Port: 8000 2024-02-01T10:50:02.810 DEBUG:boto:Params: {} 2024-02-01T10:50:02.811 DEBUG:boto:Token: None 2024-02-01T10:50:02.811 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:02 GMT /vclfcn-63/obj5 2024-02-01T10:50:02.811 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:qTNVTIhP3L+P3Hh+hd7ZgrzQLH0= 2024-02-01T10:50:02.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:qTNVTIhP3L+P3Hh+hd7ZgrzQLH0=', 'Content-Length': '0'} 2024-02-01T10:50:02.813 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000384a38c5c0827845-0065bb775a-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:02.856 DEBUG:boto:path=/vclfcn-63/obj5 2024-02-01T10:50:02.856 DEBUG:boto:auth_path=/vclfcn-63/obj5 2024-02-01T10:50:02.856 DEBUG:boto:Method: GET 2024-02-01T10:50:02.857 DEBUG:boto:Path: /vclfcn-63/obj5 2024-02-01T10:50:02.857 DEBUG:boto:Data: 2024-02-01T10:50:02.857 DEBUG:boto:Headers: {} 2024-02-01T10:50:02.857 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:02.857 DEBUG:boto:Port: 8000 2024-02-01T10:50:02.857 DEBUG:boto:Params: {} 2024-02-01T10:50:02.857 DEBUG:boto:Token: None 2024-02-01T10:50:02.857 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:02 GMT /vclfcn-63/obj5 2024-02-01T10:50:02.858 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:qTNVTIhP3L+P3Hh+hd7ZgrzQLH0= 2024-02-01T10:50:02.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:qTNVTIhP3L+P3Hh+hd7ZgrzQLH0=', 'Content-Length': '0'} 2024-02-01T10:50:02.860 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d604c6eacf2f2cf2-0065bb775a-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:02.909 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:02.909 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:02.909 DEBUG:boto:path=/vclfcn-63/?prefix=obj5 2024-02-01T10:50:02.909 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj5 2024-02-01T10:50:02.909 DEBUG:boto:Method: GET 2024-02-01T10:50:02.910 DEBUG:boto:Path: /vclfcn-63/?prefix=obj5 2024-02-01T10:50:02.910 DEBUG:boto:Data: 2024-02-01T10:50:02.910 DEBUG:boto:Headers: {} 2024-02-01T10:50:02.910 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:02.910 DEBUG:boto:Port: 8000 2024-02-01T10:50:02.910 DEBUG:boto:Params: {} 2024-02-01T10:50:02.910 DEBUG:boto:Token: None 2024-02-01T10:50:02.910 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:02 GMT /vclfcn-63/ 2024-02-01T10:50:02.911 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ylIwsGcaSaHAU7+2+RTv2egtcow= 2024-02-01T10:50:02.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ylIwsGcaSaHAU7+2+RTv2egtcow=', 'Content-Length': '0'} 2024-02-01T10:50:02.915 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000331319ca89b8cbb8-0065bb775a-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:02.915 DEBUG:boto:b'vclfcn-63obj51000falseobj52024-02-01T10:49:22.255Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:02.916 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:02.916 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:02.916 DEBUG:boto:path=/vclfcn-63/?prefix=obj5 2024-02-01T10:50:02.916 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj5 2024-02-01T10:50:02.916 DEBUG:boto:Method: GET 2024-02-01T10:50:02.916 DEBUG:boto:Path: /vclfcn-63/?prefix=obj5 2024-02-01T10:50:02.917 DEBUG:boto:Data: 2024-02-01T10:50:02.917 DEBUG:boto:Headers: {} 2024-02-01T10:50:02.917 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:02.917 DEBUG:boto:Port: 8000 2024-02-01T10:50:02.917 DEBUG:boto:Params: {} 2024-02-01T10:50:02.918 DEBUG:boto:Token: None 2024-02-01T10:50:02.918 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:02 GMT /vclfcn-63/ 2024-02-01T10:50:02.918 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ylIwsGcaSaHAU7+2+RTv2egtcow= 2024-02-01T10:50:02.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ylIwsGcaSaHAU7+2+RTv2egtcow=', 'Content-Length': '0'} 2024-02-01T10:50:02.923 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0d80414a88b749a-0065bb775a-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:02.923 DEBUG:boto:b'vclfcn-63obj51000falseobj52024-02-01T10:49:22.255Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:02.923 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-02-01T10:50:02.923 DEBUG:boto:path=/vclfcn-63/obj6 2024-02-01T10:50:02.924 DEBUG:boto:auth_path=/vclfcn-63/obj6 2024-02-01T10:50:02.924 DEBUG:boto:path=/vclfcn-63/obj6?versionId=null 2024-02-01T10:50:02.924 DEBUG:boto:auth_path=/vclfcn-63/obj6?versionId=null 2024-02-01T10:50:02.924 DEBUG:boto:Method: GET 2024-02-01T10:50:02.924 DEBUG:boto:Path: /vclfcn-63/obj6?versionId=null 2024-02-01T10:50:02.924 DEBUG:boto:Data: 2024-02-01T10:50:02.924 DEBUG:boto:Headers: {} 2024-02-01T10:50:02.925 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:02.925 DEBUG:boto:Port: 8000 2024-02-01T10:50:02.925 DEBUG:boto:Params: {} 2024-02-01T10:50:02.925 DEBUG:boto:Token: None 2024-02-01T10:50:02.925 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:02 GMT /vclfcn-63/obj6?versionId=null 2024-02-01T10:50:02.925 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BqBomiwZ4DAk5hsxgTu/1ydzqDA= 2024-02-01T10:50:02.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BqBomiwZ4DAk5hsxgTu/1ydzqDA=', 'Content-Length': '0'} 2024-02-01T10:50:02.928 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000d91c725850c10e8-0065bb775a-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:02.928 DEBUG:boto:path=/vclfcn-63/obj6 2024-02-01T10:50:02.929 DEBUG:boto:auth_path=/vclfcn-63/obj6 2024-02-01T10:50:02.929 DEBUG:boto:path=/vclfcn-63/obj6?versionId=null 2024-02-01T10:50:02.929 DEBUG:boto:auth_path=/vclfcn-63/obj6?versionId=null 2024-02-01T10:50:02.929 DEBUG:boto:Method: GET 2024-02-01T10:50:02.929 DEBUG:boto:Path: /vclfcn-63/obj6?versionId=null 2024-02-01T10:50:02.929 DEBUG:boto:Data: 2024-02-01T10:50:02.929 DEBUG:boto:Headers: {} 2024-02-01T10:50:02.929 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:02.929 DEBUG:boto:Port: 8000 2024-02-01T10:50:02.929 DEBUG:boto:Params: {} 2024-02-01T10:50:02.929 DEBUG:boto:Token: None 2024-02-01T10:50:02.930 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:02 GMT /vclfcn-63/obj6?versionId=null 2024-02-01T10:50:02.930 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BqBomiwZ4DAk5hsxgTu/1ydzqDA= 2024-02-01T10:50:02.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BqBomiwZ4DAk5hsxgTu/1ydzqDA=', 'Content-Length': '0'} 2024-02-01T10:50:02.933 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000011108e59e7a1872b-0065bb775a-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:02.980 DEBUG:boto:path=/vclfcn-63/obj6 2024-02-01T10:50:02.980 DEBUG:boto:auth_path=/vclfcn-63/obj6 2024-02-01T10:50:02.980 DEBUG:boto:path=/vclfcn-63/obj6?versionId=null 2024-02-01T10:50:02.981 DEBUG:boto:auth_path=/vclfcn-63/obj6?versionId=null 2024-02-01T10:50:02.981 DEBUG:boto:Method: HEAD 2024-02-01T10:50:02.981 DEBUG:boto:Path: /vclfcn-63/obj6?versionId=null 2024-02-01T10:50:02.981 DEBUG:boto:Data: 2024-02-01T10:50:02.981 DEBUG:boto:Headers: {} 2024-02-01T10:50:02.981 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:02.981 DEBUG:boto:Port: 8000 2024-02-01T10:50:02.981 DEBUG:boto:Params: {} 2024-02-01T10:50:02.981 DEBUG:boto:Token: None 2024-02-01T10:50:02.982 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:02 GMT /vclfcn-63/obj6?versionId=null 2024-02-01T10:50:02.982 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:q4bo/+wB4FjLO+WMEXVbJZnGaTg= 2024-02-01T10:50:02.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:q4bo/+wB4FjLO+WMEXVbJZnGaTg=', 'Content-Length': '0'} 2024-02-01T10:50:02.984 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000056edf58ce5718185-0065bb775a-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:02.985 DEBUG:boto:path=/vclfcn-63/obj6 2024-02-01T10:50:02.985 DEBUG:boto:auth_path=/vclfcn-63/obj6 2024-02-01T10:50:02.985 DEBUG:boto:path=/vclfcn-63/obj6?versionId=null 2024-02-01T10:50:02.985 DEBUG:boto:auth_path=/vclfcn-63/obj6?versionId=null 2024-02-01T10:50:02.985 DEBUG:boto:Method: HEAD 2024-02-01T10:50:02.985 DEBUG:boto:Path: /vclfcn-63/obj6?versionId=null 2024-02-01T10:50:02.985 DEBUG:boto:Data: 2024-02-01T10:50:02.985 DEBUG:boto:Headers: {} 2024-02-01T10:50:02.985 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:02.985 DEBUG:boto:Port: 8000 2024-02-01T10:50:02.986 DEBUG:boto:Params: {} 2024-02-01T10:50:02.986 DEBUG:boto:Token: None 2024-02-01T10:50:02.986 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:02 GMT /vclfcn-63/obj6?versionId=null 2024-02-01T10:50:02.986 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:q4bo/+wB4FjLO+WMEXVbJZnGaTg= 2024-02-01T10:50:02.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:q4bo/+wB4FjLO+WMEXVbJZnGaTg=', 'Content-Length': '0'} 2024-02-01T10:50:02.989 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003aea4d136d2c01dd-0065bb775a-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:02.989 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-02-01T10:50:02.989 DEBUG:boto:path=/vclfcn-63/obj6 2024-02-01T10:50:02.989 DEBUG:boto:auth_path=/vclfcn-63/obj6 2024-02-01T10:50:02.989 DEBUG:boto:path=/vclfcn-63/obj6?versionId=null 2024-02-01T10:50:02.989 DEBUG:boto:auth_path=/vclfcn-63/obj6?versionId=null 2024-02-01T10:50:02.989 DEBUG:boto:Method: GET 2024-02-01T10:50:02.989 DEBUG:boto:Path: /vclfcn-63/obj6?versionId=null 2024-02-01T10:50:02.990 DEBUG:boto:Data: 2024-02-01T10:50:02.990 DEBUG:boto:Headers: {} 2024-02-01T10:50:02.990 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:02.990 DEBUG:boto:Port: 8000 2024-02-01T10:50:02.990 DEBUG:boto:Params: {} 2024-02-01T10:50:02.990 DEBUG:boto:Token: None 2024-02-01T10:50:02.990 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:02 GMT /vclfcn-63/obj6?versionId=null 2024-02-01T10:50:02.990 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BqBomiwZ4DAk5hsxgTu/1ydzqDA= 2024-02-01T10:50:02.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BqBomiwZ4DAk5hsxgTu/1ydzqDA=', 'Content-Length': '0'} 2024-02-01T10:50:02.993 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000040f3392f3c9674e-0065bb775a-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:02.993 DEBUG:boto:path=/vclfcn-63/obj6 2024-02-01T10:50:02.993 DEBUG:boto:auth_path=/vclfcn-63/obj6 2024-02-01T10:50:02.994 DEBUG:boto:path=/vclfcn-63/obj6?versionId=null 2024-02-01T10:50:02.994 DEBUG:boto:auth_path=/vclfcn-63/obj6?versionId=null 2024-02-01T10:50:02.994 DEBUG:boto:Method: GET 2024-02-01T10:50:02.994 DEBUG:boto:Path: /vclfcn-63/obj6?versionId=null 2024-02-01T10:50:02.994 DEBUG:boto:Data: 2024-02-01T10:50:02.994 DEBUG:boto:Headers: {} 2024-02-01T10:50:02.994 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:02.994 DEBUG:boto:Port: 8000 2024-02-01T10:50:02.994 DEBUG:boto:Params: {} 2024-02-01T10:50:02.994 DEBUG:boto:Token: None 2024-02-01T10:50:02.995 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:02 GMT /vclfcn-63/obj6?versionId=null 2024-02-01T10:50:02.995 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BqBomiwZ4DAk5hsxgTu/1ydzqDA= 2024-02-01T10:50:02.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BqBomiwZ4DAk5hsxgTu/1ydzqDA=', 'Content-Length': '0'} 2024-02-01T10:50:02.998 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009f2f5725bd4a11cf-0065bb775a-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:02 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:03.040 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:03.040 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:03.040 DEBUG:boto:path=/vclfcn-63/?prefix=obj6 2024-02-01T10:50:03.041 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj6 2024-02-01T10:50:03.041 DEBUG:boto:Method: GET 2024-02-01T10:50:03.041 DEBUG:boto:Path: /vclfcn-63/?prefix=obj6 2024-02-01T10:50:03.041 DEBUG:boto:Data: 2024-02-01T10:50:03.041 DEBUG:boto:Headers: {} 2024-02-01T10:50:03.041 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:03.041 DEBUG:boto:Port: 8000 2024-02-01T10:50:03.041 DEBUG:boto:Params: {} 2024-02-01T10:50:03.041 DEBUG:boto:Token: None 2024-02-01T10:50:03.041 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:03 GMT /vclfcn-63/ 2024-02-01T10:50:03.042 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fKmQ3/EOM1XxqeNzBXF9UlpTJn0= 2024-02-01T10:50:03.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fKmQ3/EOM1XxqeNzBXF9UlpTJn0=', 'Content-Length': '0'} 2024-02-01T10:50:03.046 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d65136ef02999a68-0065bb775b-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:03.046 DEBUG:boto:b'vclfcn-63obj61000falseobj62024-02-01T10:49:22.271Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:03.047 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:03.047 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:03.047 DEBUG:boto:path=/vclfcn-63/?prefix=obj6 2024-02-01T10:50:03.047 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj6 2024-02-01T10:50:03.047 DEBUG:boto:Method: GET 2024-02-01T10:50:03.047 DEBUG:boto:Path: /vclfcn-63/?prefix=obj6 2024-02-01T10:50:03.047 DEBUG:boto:Data: 2024-02-01T10:50:03.047 DEBUG:boto:Headers: {} 2024-02-01T10:50:03.047 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:03.047 DEBUG:boto:Port: 8000 2024-02-01T10:50:03.048 DEBUG:boto:Params: {} 2024-02-01T10:50:03.048 DEBUG:boto:Token: None 2024-02-01T10:50:03.048 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:03 GMT /vclfcn-63/ 2024-02-01T10:50:03.048 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fKmQ3/EOM1XxqeNzBXF9UlpTJn0= 2024-02-01T10:50:03.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fKmQ3/EOM1XxqeNzBXF9UlpTJn0=', 'Content-Length': '0'} 2024-02-01T10:50:03.053 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1291095195dbc91-0065bb775b-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:03.100 DEBUG:boto:b'vclfcn-63obj61000falseobj62024-02-01T10:49:22.271Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:03.103 DEBUG:boto:path=/vclfcn-63/obj6 2024-02-01T10:50:03.103 DEBUG:boto:auth_path=/vclfcn-63/obj6 2024-02-01T10:50:03.103 DEBUG:boto:Method: HEAD 2024-02-01T10:50:03.103 DEBUG:boto:Path: /vclfcn-63/obj6 2024-02-01T10:50:03.103 DEBUG:boto:Data: 2024-02-01T10:50:03.104 DEBUG:boto:Headers: {} 2024-02-01T10:50:03.104 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:03.104 DEBUG:boto:Port: 8000 2024-02-01T10:50:03.104 DEBUG:boto:Params: {} 2024-02-01T10:50:03.104 DEBUG:boto:Token: None 2024-02-01T10:50:03.104 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:03 GMT /vclfcn-63/obj6 2024-02-01T10:50:03.104 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:DRhOSn8UI8OaaWyJUJX1dK80kh8= 2024-02-01T10:50:03.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:DRhOSn8UI8OaaWyJUJX1dK80kh8=', 'Content-Length': '0'} 2024-02-01T10:50:03.107 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009be73079dea4dcb9-0065bb775b-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:03.107 DEBUG:boto:path=/vclfcn-63/obj6 2024-02-01T10:50:03.107 DEBUG:boto:auth_path=/vclfcn-63/obj6 2024-02-01T10:50:03.108 DEBUG:boto:Method: HEAD 2024-02-01T10:50:03.108 DEBUG:boto:Path: /vclfcn-63/obj6 2024-02-01T10:50:03.108 DEBUG:boto:Data: 2024-02-01T10:50:03.108 DEBUG:boto:Headers: {} 2024-02-01T10:50:03.108 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:03.108 DEBUG:boto:Port: 8000 2024-02-01T10:50:03.108 DEBUG:boto:Params: {} 2024-02-01T10:50:03.108 DEBUG:boto:Token: None 2024-02-01T10:50:03.108 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:03 GMT /vclfcn-63/obj6 2024-02-01T10:50:03.109 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:DRhOSn8UI8OaaWyJUJX1dK80kh8= 2024-02-01T10:50:03.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:DRhOSn8UI8OaaWyJUJX1dK80kh8=', 'Content-Length': '0'} 2024-02-01T10:50:03.111 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c32d620a244ccd94-0065bb775b-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:03.112 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-02-01T10:50:03.112 DEBUG:boto:path=/vclfcn-63/obj6 2024-02-01T10:50:03.112 DEBUG:boto:auth_path=/vclfcn-63/obj6 2024-02-01T10:50:03.112 DEBUG:boto:Method: GET 2024-02-01T10:50:03.112 DEBUG:boto:Path: /vclfcn-63/obj6 2024-02-01T10:50:03.112 DEBUG:boto:Data: 2024-02-01T10:50:03.112 DEBUG:boto:Headers: {} 2024-02-01T10:50:03.112 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:03.112 DEBUG:boto:Port: 8000 2024-02-01T10:50:03.112 DEBUG:boto:Params: {} 2024-02-01T10:50:03.113 DEBUG:boto:Token: None 2024-02-01T10:50:03.113 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:03 GMT /vclfcn-63/obj6 2024-02-01T10:50:03.113 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BZZXs9Xyuj8o+s+OY65tze7TTaI= 2024-02-01T10:50:03.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BZZXs9Xyuj8o+s+OY65tze7TTaI=', 'Content-Length': '0'} 2024-02-01T10:50:03.116 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f754dcb76b7d8360-0065bb775b-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:03.117 DEBUG:boto:path=/vclfcn-63/obj6 2024-02-01T10:50:03.117 DEBUG:boto:auth_path=/vclfcn-63/obj6 2024-02-01T10:50:03.117 DEBUG:boto:Method: GET 2024-02-01T10:50:03.117 DEBUG:boto:Path: /vclfcn-63/obj6 2024-02-01T10:50:03.117 DEBUG:boto:Data: 2024-02-01T10:50:03.117 DEBUG:boto:Headers: {} 2024-02-01T10:50:03.117 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:03.117 DEBUG:boto:Port: 8000 2024-02-01T10:50:03.118 DEBUG:boto:Params: {} 2024-02-01T10:50:03.118 DEBUG:boto:Token: None 2024-02-01T10:50:03.118 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:03 GMT /vclfcn-63/obj6 2024-02-01T10:50:03.118 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BZZXs9Xyuj8o+s+OY65tze7TTaI= 2024-02-01T10:50:03.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BZZXs9Xyuj8o+s+OY65tze7TTaI=', 'Content-Length': '0'} 2024-02-01T10:50:03.121 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000399ba4964c375564-0065bb775b-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:03.168 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:03.168 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:03.169 DEBUG:boto:path=/vclfcn-63/?prefix=obj6 2024-02-01T10:50:03.169 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj6 2024-02-01T10:50:03.169 DEBUG:boto:Method: GET 2024-02-01T10:50:03.169 DEBUG:boto:Path: /vclfcn-63/?prefix=obj6 2024-02-01T10:50:03.169 DEBUG:boto:Data: 2024-02-01T10:50:03.169 DEBUG:boto:Headers: {} 2024-02-01T10:50:03.169 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:03.169 DEBUG:boto:Port: 8000 2024-02-01T10:50:03.170 DEBUG:boto:Params: {} 2024-02-01T10:50:03.170 DEBUG:boto:Token: None 2024-02-01T10:50:03.170 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:03 GMT /vclfcn-63/ 2024-02-01T10:50:03.170 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fKmQ3/EOM1XxqeNzBXF9UlpTJn0= 2024-02-01T10:50:03.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fKmQ3/EOM1XxqeNzBXF9UlpTJn0=', 'Content-Length': '0'} 2024-02-01T10:50:03.175 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000007eeaa876c40eb74-0065bb775b-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:03.175 DEBUG:boto:b'vclfcn-63obj61000falseobj62024-02-01T10:49:22.271Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:03.176 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:03.176 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:03.176 DEBUG:boto:path=/vclfcn-63/?prefix=obj6 2024-02-01T10:50:03.176 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj6 2024-02-01T10:50:03.176 DEBUG:boto:Method: GET 2024-02-01T10:50:03.177 DEBUG:boto:Path: /vclfcn-63/?prefix=obj6 2024-02-01T10:50:03.177 DEBUG:boto:Data: 2024-02-01T10:50:03.177 DEBUG:boto:Headers: {} 2024-02-01T10:50:03.177 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:03.177 DEBUG:boto:Port: 8000 2024-02-01T10:50:03.177 DEBUG:boto:Params: {} 2024-02-01T10:50:03.177 DEBUG:boto:Token: None 2024-02-01T10:50:03.177 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:03 GMT /vclfcn-63/ 2024-02-01T10:50:03.178 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fKmQ3/EOM1XxqeNzBXF9UlpTJn0= 2024-02-01T10:50:03.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fKmQ3/EOM1XxqeNzBXF9UlpTJn0=', 'Content-Length': '0'} 2024-02-01T10:50:03.183 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d286ec89e60e6653-0065bb775b-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:03.224 DEBUG:boto:b'vclfcn-63obj61000falseobj62024-02-01T10:49:22.271Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:03.225 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-02-01T10:50:03.225 DEBUG:boto:path=/vclfcn-63/obj7 2024-02-01T10:50:03.225 DEBUG:boto:auth_path=/vclfcn-63/obj7 2024-02-01T10:50:03.225 DEBUG:boto:path=/vclfcn-63/obj7?versionId=null 2024-02-01T10:50:03.225 DEBUG:boto:auth_path=/vclfcn-63/obj7?versionId=null 2024-02-01T10:50:03.225 DEBUG:boto:Method: GET 2024-02-01T10:50:03.225 DEBUG:boto:Path: /vclfcn-63/obj7?versionId=null 2024-02-01T10:50:03.225 DEBUG:boto:Data: 2024-02-01T10:50:03.225 DEBUG:boto:Headers: {} 2024-02-01T10:50:03.225 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:03.226 DEBUG:boto:Port: 8000 2024-02-01T10:50:03.226 DEBUG:boto:Params: {} 2024-02-01T10:50:03.226 DEBUG:boto:Token: None 2024-02-01T10:50:03.226 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:03 GMT /vclfcn-63/obj7?versionId=null 2024-02-01T10:50:03.226 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:IQZJGRXuO9FH+NEi1fm87mvfZWY= 2024-02-01T10:50:03.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:IQZJGRXuO9FH+NEi1fm87mvfZWY=', 'Content-Length': '0'} 2024-02-01T10:50:03.229 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d53098017dd82019-0065bb775b-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:03.230 DEBUG:boto:path=/vclfcn-63/obj7 2024-02-01T10:50:03.230 DEBUG:boto:auth_path=/vclfcn-63/obj7 2024-02-01T10:50:03.230 DEBUG:boto:path=/vclfcn-63/obj7?versionId=null 2024-02-01T10:50:03.230 DEBUG:boto:auth_path=/vclfcn-63/obj7?versionId=null 2024-02-01T10:50:03.230 DEBUG:boto:Method: GET 2024-02-01T10:50:03.230 DEBUG:boto:Path: /vclfcn-63/obj7?versionId=null 2024-02-01T10:50:03.230 DEBUG:boto:Data: 2024-02-01T10:50:03.230 DEBUG:boto:Headers: {} 2024-02-01T10:50:03.231 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:03.231 DEBUG:boto:Port: 8000 2024-02-01T10:50:03.231 DEBUG:boto:Params: {} 2024-02-01T10:50:03.231 DEBUG:boto:Token: None 2024-02-01T10:50:03.231 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:03 GMT /vclfcn-63/obj7?versionId=null 2024-02-01T10:50:03.231 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:IQZJGRXuO9FH+NEi1fm87mvfZWY= 2024-02-01T10:50:03.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:IQZJGRXuO9FH+NEi1fm87mvfZWY=', 'Content-Length': '0'} 2024-02-01T10:50:03.234 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007d7ab9709260369e-0065bb775b-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:03.276 DEBUG:boto:path=/vclfcn-63/obj7 2024-02-01T10:50:03.276 DEBUG:boto:auth_path=/vclfcn-63/obj7 2024-02-01T10:50:03.276 DEBUG:boto:path=/vclfcn-63/obj7?versionId=null 2024-02-01T10:50:03.277 DEBUG:boto:auth_path=/vclfcn-63/obj7?versionId=null 2024-02-01T10:50:03.277 DEBUG:boto:Method: HEAD 2024-02-01T10:50:03.277 DEBUG:boto:Path: /vclfcn-63/obj7?versionId=null 2024-02-01T10:50:03.277 DEBUG:boto:Data: 2024-02-01T10:50:03.277 DEBUG:boto:Headers: {} 2024-02-01T10:50:03.277 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:03.277 DEBUG:boto:Port: 8000 2024-02-01T10:50:03.277 DEBUG:boto:Params: {} 2024-02-01T10:50:03.277 DEBUG:boto:Token: None 2024-02-01T10:50:03.278 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:03 GMT /vclfcn-63/obj7?versionId=null 2024-02-01T10:50:03.278 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7PbSBjlAINghkhWXrvgAVXcFGko= 2024-02-01T10:50:03.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7PbSBjlAINghkhWXrvgAVXcFGko=', 'Content-Length': '0'} 2024-02-01T10:50:03.281 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007f7d98a81f6e2539-0065bb775b-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:03.281 DEBUG:boto:path=/vclfcn-63/obj7 2024-02-01T10:50:03.281 DEBUG:boto:auth_path=/vclfcn-63/obj7 2024-02-01T10:50:03.282 DEBUG:boto:path=/vclfcn-63/obj7?versionId=null 2024-02-01T10:50:03.282 DEBUG:boto:auth_path=/vclfcn-63/obj7?versionId=null 2024-02-01T10:50:03.282 DEBUG:boto:Method: HEAD 2024-02-01T10:50:03.282 DEBUG:boto:Path: /vclfcn-63/obj7?versionId=null 2024-02-01T10:50:03.282 DEBUG:boto:Data: 2024-02-01T10:50:03.282 DEBUG:boto:Headers: {} 2024-02-01T10:50:03.282 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:03.282 DEBUG:boto:Port: 8000 2024-02-01T10:50:03.282 DEBUG:boto:Params: {} 2024-02-01T10:50:03.282 DEBUG:boto:Token: None 2024-02-01T10:50:03.282 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:03 GMT /vclfcn-63/obj7?versionId=null 2024-02-01T10:50:03.283 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7PbSBjlAINghkhWXrvgAVXcFGko= 2024-02-01T10:50:03.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7PbSBjlAINghkhWXrvgAVXcFGko=', 'Content-Length': '0'} 2024-02-01T10:50:03.285 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001a97204d85375bd3-0065bb775b-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:03.286 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-02-01T10:50:03.286 DEBUG:boto:path=/vclfcn-63/obj7 2024-02-01T10:50:03.286 DEBUG:boto:auth_path=/vclfcn-63/obj7 2024-02-01T10:50:03.286 DEBUG:boto:path=/vclfcn-63/obj7?versionId=null 2024-02-01T10:50:03.286 DEBUG:boto:auth_path=/vclfcn-63/obj7?versionId=null 2024-02-01T10:50:03.286 DEBUG:boto:Method: GET 2024-02-01T10:50:03.286 DEBUG:boto:Path: /vclfcn-63/obj7?versionId=null 2024-02-01T10:50:03.286 DEBUG:boto:Data: 2024-02-01T10:50:03.286 DEBUG:boto:Headers: {} 2024-02-01T10:50:03.286 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:03.287 DEBUG:boto:Port: 8000 2024-02-01T10:50:03.287 DEBUG:boto:Params: {} 2024-02-01T10:50:03.287 DEBUG:boto:Token: None 2024-02-01T10:50:03.287 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:03 GMT /vclfcn-63/obj7?versionId=null 2024-02-01T10:50:03.287 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:IQZJGRXuO9FH+NEi1fm87mvfZWY= 2024-02-01T10:50:03.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:IQZJGRXuO9FH+NEi1fm87mvfZWY=', 'Content-Length': '0'} 2024-02-01T10:50:03.290 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003f97351b41e330f9-0065bb775b-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:03.336 DEBUG:boto:path=/vclfcn-63/obj7 2024-02-01T10:50:03.337 DEBUG:boto:auth_path=/vclfcn-63/obj7 2024-02-01T10:50:03.337 DEBUG:boto:path=/vclfcn-63/obj7?versionId=null 2024-02-01T10:50:03.337 DEBUG:boto:auth_path=/vclfcn-63/obj7?versionId=null 2024-02-01T10:50:03.337 DEBUG:boto:Method: GET 2024-02-01T10:50:03.337 DEBUG:boto:Path: /vclfcn-63/obj7?versionId=null 2024-02-01T10:50:03.337 DEBUG:boto:Data: 2024-02-01T10:50:03.337 DEBUG:boto:Headers: {} 2024-02-01T10:50:03.337 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:03.337 DEBUG:boto:Port: 8000 2024-02-01T10:50:03.338 DEBUG:boto:Params: {} 2024-02-01T10:50:03.338 DEBUG:boto:Token: None 2024-02-01T10:50:03.338 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:03 GMT /vclfcn-63/obj7?versionId=null 2024-02-01T10:50:03.338 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:IQZJGRXuO9FH+NEi1fm87mvfZWY= 2024-02-01T10:50:03.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:IQZJGRXuO9FH+NEi1fm87mvfZWY=', 'Content-Length': '0'} 2024-02-01T10:50:03.341 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000027180c3b1b366573-0065bb775b-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:03.388 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:03.388 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:03.388 DEBUG:boto:path=/vclfcn-63/?prefix=obj7 2024-02-01T10:50:03.388 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj7 2024-02-01T10:50:03.388 DEBUG:boto:Method: GET 2024-02-01T10:50:03.389 DEBUG:boto:Path: /vclfcn-63/?prefix=obj7 2024-02-01T10:50:03.389 DEBUG:boto:Data: 2024-02-01T10:50:03.389 DEBUG:boto:Headers: {} 2024-02-01T10:50:03.389 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:03.389 DEBUG:boto:Port: 8000 2024-02-01T10:50:03.389 DEBUG:boto:Params: {} 2024-02-01T10:50:03.389 DEBUG:boto:Token: None 2024-02-01T10:50:03.389 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:03 GMT /vclfcn-63/ 2024-02-01T10:50:03.389 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fKmQ3/EOM1XxqeNzBXF9UlpTJn0= 2024-02-01T10:50:03.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fKmQ3/EOM1XxqeNzBXF9UlpTJn0=', 'Content-Length': '0'} 2024-02-01T10:50:03.394 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b47ef01a21917bd8-0065bb775b-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:03.394 DEBUG:boto:b'vclfcn-63obj71000falseobj72024-02-01T10:49:22.327Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:03.394 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:03.394 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:03.394 DEBUG:boto:path=/vclfcn-63/?prefix=obj7 2024-02-01T10:50:03.395 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj7 2024-02-01T10:50:03.395 DEBUG:boto:Method: GET 2024-02-01T10:50:03.395 DEBUG:boto:Path: /vclfcn-63/?prefix=obj7 2024-02-01T10:50:03.395 DEBUG:boto:Data: 2024-02-01T10:50:03.395 DEBUG:boto:Headers: {} 2024-02-01T10:50:03.395 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:03.395 DEBUG:boto:Port: 8000 2024-02-01T10:50:03.395 DEBUG:boto:Params: {} 2024-02-01T10:50:03.395 DEBUG:boto:Token: None 2024-02-01T10:50:03.395 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:03 GMT /vclfcn-63/ 2024-02-01T10:50:03.396 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fKmQ3/EOM1XxqeNzBXF9UlpTJn0= 2024-02-01T10:50:03.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fKmQ3/EOM1XxqeNzBXF9UlpTJn0=', 'Content-Length': '0'} 2024-02-01T10:50:03.400 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b889d7f0615b477f-0065bb775b-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:03.448 DEBUG:boto:b'vclfcn-63obj71000falseobj72024-02-01T10:49:22.327Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:03.449 DEBUG:boto:path=/vclfcn-63/obj7 2024-02-01T10:50:03.449 DEBUG:boto:auth_path=/vclfcn-63/obj7 2024-02-01T10:50:03.449 DEBUG:boto:Method: HEAD 2024-02-01T10:50:03.449 DEBUG:boto:Path: /vclfcn-63/obj7 2024-02-01T10:50:03.449 DEBUG:boto:Data: 2024-02-01T10:50:03.449 DEBUG:boto:Headers: {} 2024-02-01T10:50:03.449 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:03.449 DEBUG:boto:Port: 8000 2024-02-01T10:50:03.449 DEBUG:boto:Params: {} 2024-02-01T10:50:03.449 DEBUG:boto:Token: None 2024-02-01T10:50:03.450 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:03 GMT /vclfcn-63/obj7 2024-02-01T10:50:03.450 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WDdziAvblIqSA+djIqTNOorNoZ4= 2024-02-01T10:50:03.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WDdziAvblIqSA+djIqTNOorNoZ4=', 'Content-Length': '0'} 2024-02-01T10:50:03.453 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000353578fc3e5ee85d-0065bb775b-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:03.453 DEBUG:boto:path=/vclfcn-63/obj7 2024-02-01T10:50:03.453 DEBUG:boto:auth_path=/vclfcn-63/obj7 2024-02-01T10:50:03.453 DEBUG:boto:Method: HEAD 2024-02-01T10:50:03.453 DEBUG:boto:Path: /vclfcn-63/obj7 2024-02-01T10:50:03.454 DEBUG:boto:Data: 2024-02-01T10:50:03.454 DEBUG:boto:Headers: {} 2024-02-01T10:50:03.454 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:03.454 DEBUG:boto:Port: 8000 2024-02-01T10:50:03.454 DEBUG:boto:Params: {} 2024-02-01T10:50:03.454 DEBUG:boto:Token: None 2024-02-01T10:50:03.454 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:03 GMT /vclfcn-63/obj7 2024-02-01T10:50:03.454 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:WDdziAvblIqSA+djIqTNOorNoZ4= 2024-02-01T10:50:03.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:WDdziAvblIqSA+djIqTNOorNoZ4=', 'Content-Length': '0'} 2024-02-01T10:50:03.457 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007ad2bf0a58af5ec3-0065bb775b-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:03.458 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-02-01T10:50:03.458 DEBUG:boto:path=/vclfcn-63/obj7 2024-02-01T10:50:03.458 DEBUG:boto:auth_path=/vclfcn-63/obj7 2024-02-01T10:50:03.458 DEBUG:boto:Method: GET 2024-02-01T10:50:03.458 DEBUG:boto:Path: /vclfcn-63/obj7 2024-02-01T10:50:03.458 DEBUG:boto:Data: 2024-02-01T10:50:03.458 DEBUG:boto:Headers: {} 2024-02-01T10:50:03.459 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:03.459 DEBUG:boto:Port: 8000 2024-02-01T10:50:03.459 DEBUG:boto:Params: {} 2024-02-01T10:50:03.459 DEBUG:boto:Token: None 2024-02-01T10:50:03.459 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:03 GMT /vclfcn-63/obj7 2024-02-01T10:50:03.459 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mQ8kwY8YyKuiuMY3GrJkcEIc/lc= 2024-02-01T10:50:03.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mQ8kwY8YyKuiuMY3GrJkcEIc/lc=', 'Content-Length': '0'} 2024-02-01T10:50:03.463 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000108d98313dcfa570-0065bb775b-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:03.463 DEBUG:boto:path=/vclfcn-63/obj7 2024-02-01T10:50:03.463 DEBUG:boto:auth_path=/vclfcn-63/obj7 2024-02-01T10:50:03.463 DEBUG:boto:Method: GET 2024-02-01T10:50:03.463 DEBUG:boto:Path: /vclfcn-63/obj7 2024-02-01T10:50:03.463 DEBUG:boto:Data: 2024-02-01T10:50:03.463 DEBUG:boto:Headers: {} 2024-02-01T10:50:03.464 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:03.464 DEBUG:boto:Port: 8000 2024-02-01T10:50:03.464 DEBUG:boto:Params: {} 2024-02-01T10:50:03.464 DEBUG:boto:Token: None 2024-02-01T10:50:03.464 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:03 GMT /vclfcn-63/obj7 2024-02-01T10:50:03.464 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mQ8kwY8YyKuiuMY3GrJkcEIc/lc= 2024-02-01T10:50:03.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mQ8kwY8YyKuiuMY3GrJkcEIc/lc=', 'Content-Length': '0'} 2024-02-01T10:50:03.467 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000987358ab97b9cbfd-0065bb775b-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:03.508 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:03.508 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:03.508 DEBUG:boto:path=/vclfcn-63/?prefix=obj7 2024-02-01T10:50:03.509 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj7 2024-02-01T10:50:03.509 DEBUG:boto:Method: GET 2024-02-01T10:50:03.509 DEBUG:boto:Path: /vclfcn-63/?prefix=obj7 2024-02-01T10:50:03.509 DEBUG:boto:Data: 2024-02-01T10:50:03.509 DEBUG:boto:Headers: {} 2024-02-01T10:50:03.509 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:03.509 DEBUG:boto:Port: 8000 2024-02-01T10:50:03.509 DEBUG:boto:Params: {} 2024-02-01T10:50:03.509 DEBUG:boto:Token: None 2024-02-01T10:50:03.509 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:03 GMT /vclfcn-63/ 2024-02-01T10:50:03.510 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fKmQ3/EOM1XxqeNzBXF9UlpTJn0= 2024-02-01T10:50:03.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fKmQ3/EOM1XxqeNzBXF9UlpTJn0=', 'Content-Length': '0'} 2024-02-01T10:50:03.515 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc135d33c64e61bb-0065bb775b-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:03.516 DEBUG:boto:b'vclfcn-63obj71000falseobj72024-02-01T10:49:22.327Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:03.516 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:03.516 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:03.516 DEBUG:boto:path=/vclfcn-63/?prefix=obj7 2024-02-01T10:50:03.516 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj7 2024-02-01T10:50:03.516 DEBUG:boto:Method: GET 2024-02-01T10:50:03.517 DEBUG:boto:Path: /vclfcn-63/?prefix=obj7 2024-02-01T10:50:03.517 DEBUG:boto:Data: 2024-02-01T10:50:03.517 DEBUG:boto:Headers: {} 2024-02-01T10:50:03.517 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:03.517 DEBUG:boto:Port: 8000 2024-02-01T10:50:03.517 DEBUG:boto:Params: {} 2024-02-01T10:50:03.517 DEBUG:boto:Token: None 2024-02-01T10:50:03.517 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:03 GMT /vclfcn-63/ 2024-02-01T10:50:03.517 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fKmQ3/EOM1XxqeNzBXF9UlpTJn0= 2024-02-01T10:50:03.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fKmQ3/EOM1XxqeNzBXF9UlpTJn0=', 'Content-Length': '0'} 2024-02-01T10:50:03.522 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d7a3817f00adb99-0065bb775b-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:03.564 DEBUG:boto:b'vclfcn-63obj71000falseobj72024-02-01T10:49:22.327Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:03.564 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-02-01T10:50:03.565 DEBUG:boto:path=/vclfcn-63/obj8 2024-02-01T10:50:03.565 DEBUG:boto:auth_path=/vclfcn-63/obj8 2024-02-01T10:50:03.565 DEBUG:boto:path=/vclfcn-63/obj8?versionId=null 2024-02-01T10:50:03.565 DEBUG:boto:auth_path=/vclfcn-63/obj8?versionId=null 2024-02-01T10:50:03.565 DEBUG:boto:Method: GET 2024-02-01T10:50:03.565 DEBUG:boto:Path: /vclfcn-63/obj8?versionId=null 2024-02-01T10:50:03.565 DEBUG:boto:Data: 2024-02-01T10:50:03.565 DEBUG:boto:Headers: {} 2024-02-01T10:50:03.565 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:03.566 DEBUG:boto:Port: 8000 2024-02-01T10:50:03.566 DEBUG:boto:Params: {} 2024-02-01T10:50:03.566 DEBUG:boto:Token: None 2024-02-01T10:50:03.566 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:03 GMT /vclfcn-63/obj8?versionId=null 2024-02-01T10:50:03.566 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:KJu+AeInpvoHrTWOtufbvwzR5w8= 2024-02-01T10:50:03.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:KJu+AeInpvoHrTWOtufbvwzR5w8=', 'Content-Length': '0'} 2024-02-01T10:50:03.569 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a4779199ba1458c2-0065bb775b-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:03.569 DEBUG:boto:path=/vclfcn-63/obj8 2024-02-01T10:50:03.569 DEBUG:boto:auth_path=/vclfcn-63/obj8 2024-02-01T10:50:03.570 DEBUG:boto:path=/vclfcn-63/obj8?versionId=null 2024-02-01T10:50:03.570 DEBUG:boto:auth_path=/vclfcn-63/obj8?versionId=null 2024-02-01T10:50:03.570 DEBUG:boto:Method: GET 2024-02-01T10:50:03.570 DEBUG:boto:Path: /vclfcn-63/obj8?versionId=null 2024-02-01T10:50:03.570 DEBUG:boto:Data: 2024-02-01T10:50:03.570 DEBUG:boto:Headers: {} 2024-02-01T10:50:03.570 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:03.570 DEBUG:boto:Port: 8000 2024-02-01T10:50:03.570 DEBUG:boto:Params: {} 2024-02-01T10:50:03.570 DEBUG:boto:Token: None 2024-02-01T10:50:03.571 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:03 GMT /vclfcn-63/obj8?versionId=null 2024-02-01T10:50:03.571 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:KJu+AeInpvoHrTWOtufbvwzR5w8= 2024-02-01T10:50:03.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:KJu+AeInpvoHrTWOtufbvwzR5w8=', 'Content-Length': '0'} 2024-02-01T10:50:03.574 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000075960e28c6a0d3b5-0065bb775b-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:03.574 DEBUG:boto:path=/vclfcn-63/obj8 2024-02-01T10:50:03.575 DEBUG:boto:auth_path=/vclfcn-63/obj8 2024-02-01T10:50:03.575 DEBUG:boto:path=/vclfcn-63/obj8?versionId=null 2024-02-01T10:50:03.575 DEBUG:boto:auth_path=/vclfcn-63/obj8?versionId=null 2024-02-01T10:50:03.575 DEBUG:boto:Method: HEAD 2024-02-01T10:50:03.575 DEBUG:boto:Path: /vclfcn-63/obj8?versionId=null 2024-02-01T10:50:03.575 DEBUG:boto:Data: 2024-02-01T10:50:03.575 DEBUG:boto:Headers: {} 2024-02-01T10:50:03.575 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:03.575 DEBUG:boto:Port: 8000 2024-02-01T10:50:03.575 DEBUG:boto:Params: {} 2024-02-01T10:50:03.576 DEBUG:boto:Token: None 2024-02-01T10:50:03.576 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:03 GMT /vclfcn-63/obj8?versionId=null 2024-02-01T10:50:03.576 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1H+77lB2Zj2uKNmXWnExZyqSROo= 2024-02-01T10:50:03.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1H+77lB2Zj2uKNmXWnExZyqSROo=', 'Content-Length': '0'} 2024-02-01T10:50:03.579 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001ee81580b2179ba4-0065bb775b-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:03.579 DEBUG:boto:path=/vclfcn-63/obj8 2024-02-01T10:50:03.579 DEBUG:boto:auth_path=/vclfcn-63/obj8 2024-02-01T10:50:03.579 DEBUG:boto:path=/vclfcn-63/obj8?versionId=null 2024-02-01T10:50:03.579 DEBUG:boto:auth_path=/vclfcn-63/obj8?versionId=null 2024-02-01T10:50:03.579 DEBUG:boto:Method: HEAD 2024-02-01T10:50:03.580 DEBUG:boto:Path: /vclfcn-63/obj8?versionId=null 2024-02-01T10:50:03.580 DEBUG:boto:Data: 2024-02-01T10:50:03.580 DEBUG:boto:Headers: {} 2024-02-01T10:50:03.580 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:03.580 DEBUG:boto:Port: 8000 2024-02-01T10:50:03.580 DEBUG:boto:Params: {} 2024-02-01T10:50:03.580 DEBUG:boto:Token: None 2024-02-01T10:50:03.580 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:03 GMT /vclfcn-63/obj8?versionId=null 2024-02-01T10:50:03.580 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1H+77lB2Zj2uKNmXWnExZyqSROo= 2024-02-01T10:50:03.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1H+77lB2Zj2uKNmXWnExZyqSROo=', 'Content-Length': '0'} 2024-02-01T10:50:03.583 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bc0de3f877aedd6a-0065bb775b-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:03.584 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-02-01T10:50:03.584 DEBUG:boto:path=/vclfcn-63/obj8 2024-02-01T10:50:03.584 DEBUG:boto:auth_path=/vclfcn-63/obj8 2024-02-01T10:50:03.585 DEBUG:boto:path=/vclfcn-63/obj8?versionId=null 2024-02-01T10:50:03.585 DEBUG:boto:auth_path=/vclfcn-63/obj8?versionId=null 2024-02-01T10:50:03.585 DEBUG:boto:Method: GET 2024-02-01T10:50:03.585 DEBUG:boto:Path: /vclfcn-63/obj8?versionId=null 2024-02-01T10:50:03.585 DEBUG:boto:Data: 2024-02-01T10:50:03.585 DEBUG:boto:Headers: {} 2024-02-01T10:50:03.585 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:03.586 DEBUG:boto:Port: 8000 2024-02-01T10:50:03.586 DEBUG:boto:Params: {} 2024-02-01T10:50:03.586 DEBUG:boto:Token: None 2024-02-01T10:50:03.586 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:03 GMT /vclfcn-63/obj8?versionId=null 2024-02-01T10:50:03.586 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:KJu+AeInpvoHrTWOtufbvwzR5w8= 2024-02-01T10:50:03.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:KJu+AeInpvoHrTWOtufbvwzR5w8=', 'Content-Length': '0'} 2024-02-01T10:50:03.589 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006d89cb182cc3b589-0065bb775b-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:03.589 DEBUG:boto:path=/vclfcn-63/obj8 2024-02-01T10:50:03.589 DEBUG:boto:auth_path=/vclfcn-63/obj8 2024-02-01T10:50:03.590 DEBUG:boto:path=/vclfcn-63/obj8?versionId=null 2024-02-01T10:50:03.590 DEBUG:boto:auth_path=/vclfcn-63/obj8?versionId=null 2024-02-01T10:50:03.590 DEBUG:boto:Method: GET 2024-02-01T10:50:03.590 DEBUG:boto:Path: /vclfcn-63/obj8?versionId=null 2024-02-01T10:50:03.590 DEBUG:boto:Data: 2024-02-01T10:50:03.590 DEBUG:boto:Headers: {} 2024-02-01T10:50:03.590 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:03.590 DEBUG:boto:Port: 8000 2024-02-01T10:50:03.590 DEBUG:boto:Params: {} 2024-02-01T10:50:03.590 DEBUG:boto:Token: None 2024-02-01T10:50:03.591 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:03 GMT /vclfcn-63/obj8?versionId=null 2024-02-01T10:50:03.591 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:KJu+AeInpvoHrTWOtufbvwzR5w8= 2024-02-01T10:50:03.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:KJu+AeInpvoHrTWOtufbvwzR5w8=', 'Content-Length': '0'} 2024-02-01T10:50:03.594 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ff154fae23f32bba-0065bb775b-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:03.636 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:03.636 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:03.636 DEBUG:boto:path=/vclfcn-63/?prefix=obj8 2024-02-01T10:50:03.637 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj8 2024-02-01T10:50:03.637 DEBUG:boto:Method: GET 2024-02-01T10:50:03.637 DEBUG:boto:Path: /vclfcn-63/?prefix=obj8 2024-02-01T10:50:03.637 DEBUG:boto:Data: 2024-02-01T10:50:03.637 DEBUG:boto:Headers: {} 2024-02-01T10:50:03.637 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:03.637 DEBUG:boto:Port: 8000 2024-02-01T10:50:03.637 DEBUG:boto:Params: {} 2024-02-01T10:50:03.637 DEBUG:boto:Token: None 2024-02-01T10:50:03.637 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:03 GMT /vclfcn-63/ 2024-02-01T10:50:03.638 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fKmQ3/EOM1XxqeNzBXF9UlpTJn0= 2024-02-01T10:50:03.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fKmQ3/EOM1XxqeNzBXF9UlpTJn0=', 'Content-Length': '0'} 2024-02-01T10:50:03.642 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000066ecf110ffdb13bb-0065bb775b-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:03.642 DEBUG:boto:b'vclfcn-63obj81000falseobj82024-02-01T10:49:22.343Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:03.643 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:03.643 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:03.643 DEBUG:boto:path=/vclfcn-63/?prefix=obj8 2024-02-01T10:50:03.643 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj8 2024-02-01T10:50:03.643 DEBUG:boto:Method: GET 2024-02-01T10:50:03.643 DEBUG:boto:Path: /vclfcn-63/?prefix=obj8 2024-02-01T10:50:03.643 DEBUG:boto:Data: 2024-02-01T10:50:03.643 DEBUG:boto:Headers: {} 2024-02-01T10:50:03.643 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:03.643 DEBUG:boto:Port: 8000 2024-02-01T10:50:03.644 DEBUG:boto:Params: {} 2024-02-01T10:50:03.644 DEBUG:boto:Token: None 2024-02-01T10:50:03.644 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:03 GMT /vclfcn-63/ 2024-02-01T10:50:03.644 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fKmQ3/EOM1XxqeNzBXF9UlpTJn0= 2024-02-01T10:50:03.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fKmQ3/EOM1XxqeNzBXF9UlpTJn0=', 'Content-Length': '0'} 2024-02-01T10:50:03.648 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9779f981342597d-0065bb775b-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:03.692 DEBUG:boto:b'vclfcn-63obj81000falseobj82024-02-01T10:49:22.343Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:03.693 DEBUG:boto:path=/vclfcn-63/obj8 2024-02-01T10:50:03.693 DEBUG:boto:auth_path=/vclfcn-63/obj8 2024-02-01T10:50:03.693 DEBUG:boto:Method: HEAD 2024-02-01T10:50:03.693 DEBUG:boto:Path: /vclfcn-63/obj8 2024-02-01T10:50:03.693 DEBUG:boto:Data: 2024-02-01T10:50:03.693 DEBUG:boto:Headers: {} 2024-02-01T10:50:03.693 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:03.693 DEBUG:boto:Port: 8000 2024-02-01T10:50:03.693 DEBUG:boto:Params: {} 2024-02-01T10:50:03.694 DEBUG:boto:Token: None 2024-02-01T10:50:03.694 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:03 GMT /vclfcn-63/obj8 2024-02-01T10:50:03.694 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jXZVq4VGo2RXMwniv9azkBor0JM= 2024-02-01T10:50:03.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jXZVq4VGo2RXMwniv9azkBor0JM=', 'Content-Length': '0'} 2024-02-01T10:50:03.697 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a0f3041fe15e01e2-0065bb775b-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:03.697 DEBUG:boto:path=/vclfcn-63/obj8 2024-02-01T10:50:03.697 DEBUG:boto:auth_path=/vclfcn-63/obj8 2024-02-01T10:50:03.697 DEBUG:boto:Method: HEAD 2024-02-01T10:50:03.697 DEBUG:boto:Path: /vclfcn-63/obj8 2024-02-01T10:50:03.697 DEBUG:boto:Data: 2024-02-01T10:50:03.697 DEBUG:boto:Headers: {} 2024-02-01T10:50:03.697 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:03.697 DEBUG:boto:Port: 8000 2024-02-01T10:50:03.698 DEBUG:boto:Params: {} 2024-02-01T10:50:03.698 DEBUG:boto:Token: None 2024-02-01T10:50:03.698 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:03 GMT /vclfcn-63/obj8 2024-02-01T10:50:03.698 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jXZVq4VGo2RXMwniv9azkBor0JM= 2024-02-01T10:50:03.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jXZVq4VGo2RXMwniv9azkBor0JM=', 'Content-Length': '0'} 2024-02-01T10:50:03.701 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000075ebc1c73a50db5e-0065bb775b-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:03.701 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-02-01T10:50:03.701 DEBUG:boto:path=/vclfcn-63/obj8 2024-02-01T10:50:03.701 DEBUG:boto:auth_path=/vclfcn-63/obj8 2024-02-01T10:50:03.701 DEBUG:boto:Method: GET 2024-02-01T10:50:03.701 DEBUG:boto:Path: /vclfcn-63/obj8 2024-02-01T10:50:03.702 DEBUG:boto:Data: 2024-02-01T10:50:03.702 DEBUG:boto:Headers: {} 2024-02-01T10:50:03.702 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:03.702 DEBUG:boto:Port: 8000 2024-02-01T10:50:03.702 DEBUG:boto:Params: {} 2024-02-01T10:50:03.702 DEBUG:boto:Token: None 2024-02-01T10:50:03.702 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:03 GMT /vclfcn-63/obj8 2024-02-01T10:50:03.702 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:hzwm5QvR/5+JYNB1JUpKkTDlYPo= 2024-02-01T10:50:03.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:hzwm5QvR/5+JYNB1JUpKkTDlYPo=', 'Content-Length': '0'} 2024-02-01T10:50:03.705 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001d732f03f45b0303-0065bb775b-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:03.748 DEBUG:boto:path=/vclfcn-63/obj8 2024-02-01T10:50:03.748 DEBUG:boto:auth_path=/vclfcn-63/obj8 2024-02-01T10:50:03.748 DEBUG:boto:Method: GET 2024-02-01T10:50:03.749 DEBUG:boto:Path: /vclfcn-63/obj8 2024-02-01T10:50:03.749 DEBUG:boto:Data: 2024-02-01T10:50:03.749 DEBUG:boto:Headers: {} 2024-02-01T10:50:03.749 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:03.749 DEBUG:boto:Port: 8000 2024-02-01T10:50:03.749 DEBUG:boto:Params: {} 2024-02-01T10:50:03.749 DEBUG:boto:Token: None 2024-02-01T10:50:03.749 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:03 GMT /vclfcn-63/obj8 2024-02-01T10:50:03.749 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:hzwm5QvR/5+JYNB1JUpKkTDlYPo= 2024-02-01T10:50:03.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:hzwm5QvR/5+JYNB1JUpKkTDlYPo=', 'Content-Length': '0'} 2024-02-01T10:50:03.752 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000093674b2ff757ae9d-0065bb775b-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:03.796 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:03.796 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:03.796 DEBUG:boto:path=/vclfcn-63/?prefix=obj8 2024-02-01T10:50:03.796 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj8 2024-02-01T10:50:03.797 DEBUG:boto:Method: GET 2024-02-01T10:50:03.797 DEBUG:boto:Path: /vclfcn-63/?prefix=obj8 2024-02-01T10:50:03.797 DEBUG:boto:Data: 2024-02-01T10:50:03.797 DEBUG:boto:Headers: {} 2024-02-01T10:50:03.797 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:03.797 DEBUG:boto:Port: 8000 2024-02-01T10:50:03.797 DEBUG:boto:Params: {} 2024-02-01T10:50:03.797 DEBUG:boto:Token: None 2024-02-01T10:50:03.797 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:03 GMT /vclfcn-63/ 2024-02-01T10:50:03.798 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fKmQ3/EOM1XxqeNzBXF9UlpTJn0= 2024-02-01T10:50:03.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fKmQ3/EOM1XxqeNzBXF9UlpTJn0=', 'Content-Length': '0'} 2024-02-01T10:50:03.802 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000016cda68553a8cc21-0065bb775b-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:03.802 DEBUG:boto:b'vclfcn-63obj81000falseobj82024-02-01T10:49:22.343Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:03.803 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:03.803 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:03.803 DEBUG:boto:path=/vclfcn-63/?prefix=obj8 2024-02-01T10:50:03.803 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj8 2024-02-01T10:50:03.803 DEBUG:boto:Method: GET 2024-02-01T10:50:03.803 DEBUG:boto:Path: /vclfcn-63/?prefix=obj8 2024-02-01T10:50:03.804 DEBUG:boto:Data: 2024-02-01T10:50:03.804 DEBUG:boto:Headers: {} 2024-02-01T10:50:03.804 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:03.804 DEBUG:boto:Port: 8000 2024-02-01T10:50:03.804 DEBUG:boto:Params: {} 2024-02-01T10:50:03.804 DEBUG:boto:Token: None 2024-02-01T10:50:03.804 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:03 GMT /vclfcn-63/ 2024-02-01T10:50:03.804 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fKmQ3/EOM1XxqeNzBXF9UlpTJn0= 2024-02-01T10:50:03.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fKmQ3/EOM1XxqeNzBXF9UlpTJn0=', 'Content-Length': '0'} 2024-02-01T10:50:03.809 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008944daf469f4c085-0065bb775b-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:03.809 DEBUG:boto:b'vclfcn-63obj81000falseobj82024-02-01T10:49:22.343Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:03.810 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-63 zones={test-zone2, test-zone1} 2024-02-01T10:50:03.810 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-63 zones={test-zone3, test-zone1} 2024-02-01T10:50:03.810 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:03.810 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:03.810 DEBUG:boto:Method: HEAD 2024-02-01T10:50:03.810 DEBUG:boto:Path: /vclfcn-63/ 2024-02-01T10:50:03.811 DEBUG:boto:Data: 2024-02-01T10:50:03.811 DEBUG:boto:Headers: {} 2024-02-01T10:50:03.811 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:03.811 DEBUG:boto:Port: 8001 2024-02-01T10:50:03.811 DEBUG:boto:Params: {} 2024-02-01T10:50:03.811 DEBUG:boto:Token: None 2024-02-01T10:50:03.811 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:03 GMT /vclfcn-63/ 2024-02-01T10:50:03.811 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sSzfamEphUiLNjM2A1j8X/1KMf8= 2024-02-01T10:50:03.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sSzfamEphUiLNjM2A1j8X/1KMf8=', 'Content-Length': '0'} 2024-02-01T10:50:03.815 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001aa9724b6a780952-0065bb775b-5762-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:50:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:03.816 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:03.816 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:03.816 DEBUG:boto:Method: HEAD 2024-02-01T10:50:03.816 DEBUG:boto:Path: /vclfcn-63/ 2024-02-01T10:50:03.816 DEBUG:boto:Data: 2024-02-01T10:50:03.816 DEBUG:boto:Headers: {} 2024-02-01T10:50:03.816 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:03.816 DEBUG:boto:Port: 8000 2024-02-01T10:50:03.817 DEBUG:boto:Params: {} 2024-02-01T10:50:03.817 DEBUG:boto:Token: None 2024-02-01T10:50:03.817 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:03 GMT /vclfcn-63/ 2024-02-01T10:50:03.817 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sSzfamEphUiLNjM2A1j8X/1KMf8= 2024-02-01T10:50:03.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sSzfamEphUiLNjM2A1j8X/1KMf8=', 'Content-Length': '0'} 2024-02-01T10:50:03.821 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007b7465483f7b2755-0065bb775b-5765-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:50:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:03.821 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:50:03.822 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:03.822 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:03.822 DEBUG:boto:path=/vclfcn-63/?versions&max-keys=999 2024-02-01T10:50:03.822 DEBUG:boto:auth_path=/vclfcn-63/?versions&max-keys=999 2024-02-01T10:50:03.822 DEBUG:boto:Method: GET 2024-02-01T10:50:03.822 DEBUG:boto:Path: /vclfcn-63/?versions&max-keys=999 2024-02-01T10:50:03.822 DEBUG:boto:Data: 2024-02-01T10:50:03.822 DEBUG:boto:Headers: {} 2024-02-01T10:50:03.822 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:03.822 DEBUG:boto:Port: 8001 2024-02-01T10:50:03.822 DEBUG:boto:Params: {} 2024-02-01T10:50:03.823 DEBUG:boto:Token: None 2024-02-01T10:50:03.823 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:03 GMT /vclfcn-63/?versions 2024-02-01T10:50:03.823 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:KgRrjmXCZzsPNAbE2NFzVrbwZv8= 2024-02-01T10:50:03.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:KgRrjmXCZzsPNAbE2NFzVrbwZv8=', 'Content-Length': '0'} 2024-02-01T10:50:03.831 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009001c7e409cfd329-0065bb775b-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:03.831 DEBUG:boto:b'vclfcn-63999falseobj1nulltrue2024-02-01T10:48:26.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:48:26.507Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:48:26.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:48:26.575Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-02-01T10:49:22.255Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-02-01T10:49:22.271Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-02-01T10:49:22.327Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-02-01T10:49:22.343Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:03.832 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-02-01T10:50:03.832 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-02-01T10:50:03.832 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-02-01T10:50:03.832 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-02-01T10:50:03.832 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-02-01T10:50:03.833 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-02-01T10:50:03.833 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-02-01T10:50:03.833 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-02-01T10:50:03.833 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:50:03.833 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:03.833 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:03.833 DEBUG:boto:path=/vclfcn-63/?versions&max-keys=999 2024-02-01T10:50:03.833 DEBUG:boto:auth_path=/vclfcn-63/?versions&max-keys=999 2024-02-01T10:50:03.833 DEBUG:boto:Method: GET 2024-02-01T10:50:03.833 DEBUG:boto:Path: /vclfcn-63/?versions&max-keys=999 2024-02-01T10:50:03.834 DEBUG:boto:Data: 2024-02-01T10:50:03.834 DEBUG:boto:Headers: {} 2024-02-01T10:50:03.834 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:03.834 DEBUG:boto:Port: 8000 2024-02-01T10:50:03.834 DEBUG:boto:Params: {} 2024-02-01T10:50:03.834 DEBUG:boto:Token: None 2024-02-01T10:50:03.834 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:03 GMT /vclfcn-63/?versions 2024-02-01T10:50:03.834 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:KgRrjmXCZzsPNAbE2NFzVrbwZv8= 2024-02-01T10:50:03.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:KgRrjmXCZzsPNAbE2NFzVrbwZv8=', 'Content-Length': '0'} 2024-02-01T10:50:03.839 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ffe5104f2f933e19-0065bb775b-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:03.840 DEBUG:boto:b'vclfcn-63999falseobj1nulltrue2024-02-01T10:48:26.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:48:26.507Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:48:26.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:48:26.575Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-02-01T10:49:22.255Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-02-01T10:49:22.271Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-02-01T10:49:22.327Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-02-01T10:49:22.343Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:03.841 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-02-01T10:50:03.841 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-02-01T10:50:03.841 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-02-01T10:50:03.841 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-02-01T10:50:03.841 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-02-01T10:50:03.841 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-02-01T10:50:03.841 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-02-01T10:50:03.842 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-02-01T10:50:03.842 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:03.842 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:03.842 DEBUG:boto:path=/vclfcn-63/?versions&max-keys=999 2024-02-01T10:50:03.842 DEBUG:boto:auth_path=/vclfcn-63/?versions&max-keys=999 2024-02-01T10:50:03.842 DEBUG:boto:Method: GET 2024-02-01T10:50:03.842 DEBUG:boto:Path: /vclfcn-63/?versions&max-keys=999 2024-02-01T10:50:03.842 DEBUG:boto:Data: 2024-02-01T10:50:03.842 DEBUG:boto:Headers: {} 2024-02-01T10:50:03.842 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:03.842 DEBUG:boto:Port: 8001 2024-02-01T10:50:03.843 DEBUG:boto:Params: {} 2024-02-01T10:50:03.843 DEBUG:boto:Token: None 2024-02-01T10:50:03.843 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:03 GMT /vclfcn-63/?versions 2024-02-01T10:50:03.843 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:KgRrjmXCZzsPNAbE2NFzVrbwZv8= 2024-02-01T10:50:03.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:KgRrjmXCZzsPNAbE2NFzVrbwZv8=', 'Content-Length': '0'} 2024-02-01T10:50:03.847 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa10005278ce21fb-0065bb775b-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:03.848 DEBUG:boto:b'vclfcn-63999falseobj1nulltrue2024-02-01T10:48:26.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:48:26.507Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:48:26.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:48:26.575Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-02-01T10:49:22.255Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-02-01T10:49:22.271Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-02-01T10:49:22.327Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-02-01T10:49:22.343Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:03.848 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:03.849 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:03.849 DEBUG:boto:path=/vclfcn-63/?versions&max-keys=999 2024-02-01T10:50:03.849 DEBUG:boto:auth_path=/vclfcn-63/?versions&max-keys=999 2024-02-01T10:50:03.849 DEBUG:boto:Method: GET 2024-02-01T10:50:03.849 DEBUG:boto:Path: /vclfcn-63/?versions&max-keys=999 2024-02-01T10:50:03.849 DEBUG:boto:Data: 2024-02-01T10:50:03.849 DEBUG:boto:Headers: {} 2024-02-01T10:50:03.849 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:03.849 DEBUG:boto:Port: 8000 2024-02-01T10:50:03.849 DEBUG:boto:Params: {} 2024-02-01T10:50:03.850 DEBUG:boto:Token: None 2024-02-01T10:50:03.850 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:03 GMT /vclfcn-63/?versions 2024-02-01T10:50:03.850 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:KgRrjmXCZzsPNAbE2NFzVrbwZv8= 2024-02-01T10:50:03.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:KgRrjmXCZzsPNAbE2NFzVrbwZv8=', 'Content-Length': '0'} 2024-02-01T10:50:03.854 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000057d11359df5806b0-0065bb775b-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:03.855 DEBUG:boto:b'vclfcn-63999falseobj1nulltrue2024-02-01T10:48:26.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:48:26.507Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:48:26.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:48:26.575Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-02-01T10:49:22.255Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-02-01T10:49:22.271Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-02-01T10:49:22.327Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-02-01T10:49:22.343Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:03.856 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:50:03.856 DEBUG:boto:path=/vclfcn-63/obj1 2024-02-01T10:50:03.856 DEBUG:boto:auth_path=/vclfcn-63/obj1 2024-02-01T10:50:03.856 DEBUG:boto:path=/vclfcn-63/obj1?versionId=null 2024-02-01T10:50:03.856 DEBUG:boto:auth_path=/vclfcn-63/obj1?versionId=null 2024-02-01T10:50:03.856 DEBUG:boto:Method: GET 2024-02-01T10:50:03.857 DEBUG:boto:Path: /vclfcn-63/obj1?versionId=null 2024-02-01T10:50:03.857 DEBUG:boto:Data: 2024-02-01T10:50:03.857 DEBUG:boto:Headers: {} 2024-02-01T10:50:03.857 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:03.857 DEBUG:boto:Port: 8001 2024-02-01T10:50:03.857 DEBUG:boto:Params: {} 2024-02-01T10:50:03.857 DEBUG:boto:Token: None 2024-02-01T10:50:03.857 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:03 GMT /vclfcn-63/obj1?versionId=null 2024-02-01T10:50:03.857 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vcFukgHRmIB5oyVY0S4gGCUSt2s= 2024-02-01T10:50:03.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vcFukgHRmIB5oyVY0S4gGCUSt2s=', 'Content-Length': '0'} 2024-02-01T10:50:03.860 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001ef4406def934b32-0065bb775b-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:03.861 DEBUG:boto:path=/vclfcn-63/obj1 2024-02-01T10:50:03.861 DEBUG:boto:auth_path=/vclfcn-63/obj1 2024-02-01T10:50:03.861 DEBUG:boto:path=/vclfcn-63/obj1?versionId=null 2024-02-01T10:50:03.861 DEBUG:boto:auth_path=/vclfcn-63/obj1?versionId=null 2024-02-01T10:50:03.861 DEBUG:boto:Method: GET 2024-02-01T10:50:03.861 DEBUG:boto:Path: /vclfcn-63/obj1?versionId=null 2024-02-01T10:50:03.862 DEBUG:boto:Data: 2024-02-01T10:50:03.862 DEBUG:boto:Headers: {} 2024-02-01T10:50:03.862 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:03.862 DEBUG:boto:Port: 8000 2024-02-01T10:50:03.862 DEBUG:boto:Params: {} 2024-02-01T10:50:03.862 DEBUG:boto:Token: None 2024-02-01T10:50:03.862 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:03 GMT /vclfcn-63/obj1?versionId=null 2024-02-01T10:50:03.862 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vcFukgHRmIB5oyVY0S4gGCUSt2s= 2024-02-01T10:50:03.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vcFukgHRmIB5oyVY0S4gGCUSt2s=', 'Content-Length': '0'} 2024-02-01T10:50:03.865 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b3e21bb8338e3289-0065bb775b-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:03.908 DEBUG:boto:path=/vclfcn-63/obj1 2024-02-01T10:50:03.908 DEBUG:boto:auth_path=/vclfcn-63/obj1 2024-02-01T10:50:03.908 DEBUG:boto:path=/vclfcn-63/obj1?versionId=null 2024-02-01T10:50:03.909 DEBUG:boto:auth_path=/vclfcn-63/obj1?versionId=null 2024-02-01T10:50:03.909 DEBUG:boto:Method: HEAD 2024-02-01T10:50:03.909 DEBUG:boto:Path: /vclfcn-63/obj1?versionId=null 2024-02-01T10:50:03.909 DEBUG:boto:Data: 2024-02-01T10:50:03.909 DEBUG:boto:Headers: {} 2024-02-01T10:50:03.909 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:03.909 DEBUG:boto:Port: 8001 2024-02-01T10:50:03.909 DEBUG:boto:Params: {} 2024-02-01T10:50:03.909 DEBUG:boto:Token: None 2024-02-01T10:50:03.910 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:03 GMT /vclfcn-63/obj1?versionId=null 2024-02-01T10:50:03.910 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8kZj9sLbAUxQ+lXBmnSQy+DKgqM= 2024-02-01T10:50:03.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8kZj9sLbAUxQ+lXBmnSQy+DKgqM=', 'Content-Length': '0'} 2024-02-01T10:50:03.912 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cb875cd899a70156-0065bb775b-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:03.913 DEBUG:boto:path=/vclfcn-63/obj1 2024-02-01T10:50:03.913 DEBUG:boto:auth_path=/vclfcn-63/obj1 2024-02-01T10:50:03.913 DEBUG:boto:path=/vclfcn-63/obj1?versionId=null 2024-02-01T10:50:03.913 DEBUG:boto:auth_path=/vclfcn-63/obj1?versionId=null 2024-02-01T10:50:03.914 DEBUG:boto:Method: HEAD 2024-02-01T10:50:03.914 DEBUG:boto:Path: /vclfcn-63/obj1?versionId=null 2024-02-01T10:50:03.914 DEBUG:boto:Data: 2024-02-01T10:50:03.914 DEBUG:boto:Headers: {} 2024-02-01T10:50:03.914 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:03.914 DEBUG:boto:Port: 8000 2024-02-01T10:50:03.914 DEBUG:boto:Params: {} 2024-02-01T10:50:03.914 DEBUG:boto:Token: None 2024-02-01T10:50:03.914 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:03 GMT /vclfcn-63/obj1?versionId=null 2024-02-01T10:50:03.914 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8kZj9sLbAUxQ+lXBmnSQy+DKgqM= 2024-02-01T10:50:03.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8kZj9sLbAUxQ+lXBmnSQy+DKgqM=', 'Content-Length': '0'} 2024-02-01T10:50:03.917 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b22d67463cfbb1d5-0065bb775b-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:03.918 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:50:03.918 DEBUG:boto:path=/vclfcn-63/obj1 2024-02-01T10:50:03.918 DEBUG:boto:auth_path=/vclfcn-63/obj1 2024-02-01T10:50:03.918 DEBUG:boto:path=/vclfcn-63/obj1?versionId=null 2024-02-01T10:50:03.918 DEBUG:boto:auth_path=/vclfcn-63/obj1?versionId=null 2024-02-01T10:50:03.918 DEBUG:boto:Method: GET 2024-02-01T10:50:03.918 DEBUG:boto:Path: /vclfcn-63/obj1?versionId=null 2024-02-01T10:50:03.918 DEBUG:boto:Data: 2024-02-01T10:50:03.919 DEBUG:boto:Headers: {} 2024-02-01T10:50:03.919 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:03.919 DEBUG:boto:Port: 8001 2024-02-01T10:50:03.919 DEBUG:boto:Params: {} 2024-02-01T10:50:03.919 DEBUG:boto:Token: None 2024-02-01T10:50:03.919 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:03 GMT /vclfcn-63/obj1?versionId=null 2024-02-01T10:50:03.919 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vcFukgHRmIB5oyVY0S4gGCUSt2s= 2024-02-01T10:50:03.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vcFukgHRmIB5oyVY0S4gGCUSt2s=', 'Content-Length': '0'} 2024-02-01T10:50:03.925 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000050993fdd37fbe626-0065bb775b-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:03.964 DEBUG:boto:path=/vclfcn-63/obj1 2024-02-01T10:50:03.964 DEBUG:boto:auth_path=/vclfcn-63/obj1 2024-02-01T10:50:03.964 DEBUG:boto:path=/vclfcn-63/obj1?versionId=null 2024-02-01T10:50:03.965 DEBUG:boto:auth_path=/vclfcn-63/obj1?versionId=null 2024-02-01T10:50:03.965 DEBUG:boto:Method: GET 2024-02-01T10:50:03.965 DEBUG:boto:Path: /vclfcn-63/obj1?versionId=null 2024-02-01T10:50:03.965 DEBUG:boto:Data: 2024-02-01T10:50:03.965 DEBUG:boto:Headers: {} 2024-02-01T10:50:03.965 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:03.965 DEBUG:boto:Port: 8000 2024-02-01T10:50:03.965 DEBUG:boto:Params: {} 2024-02-01T10:50:03.965 DEBUG:boto:Token: None 2024-02-01T10:50:03.966 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:03 GMT /vclfcn-63/obj1?versionId=null 2024-02-01T10:50:03.966 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vcFukgHRmIB5oyVY0S4gGCUSt2s= 2024-02-01T10:50:03.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:03 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vcFukgHRmIB5oyVY0S4gGCUSt2s=', 'Content-Length': '0'} 2024-02-01T10:50:03.969 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002235cc7beb30f495-0065bb775b-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:03 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:04.012 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:04.012 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:04.012 DEBUG:boto:path=/vclfcn-63/?prefix=obj1 2024-02-01T10:50:04.013 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj1 2024-02-01T10:50:04.013 DEBUG:boto:Method: GET 2024-02-01T10:50:04.013 DEBUG:boto:Path: /vclfcn-63/?prefix=obj1 2024-02-01T10:50:04.013 DEBUG:boto:Data: 2024-02-01T10:50:04.013 DEBUG:boto:Headers: {} 2024-02-01T10:50:04.013 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:04.013 DEBUG:boto:Port: 8001 2024-02-01T10:50:04.013 DEBUG:boto:Params: {} 2024-02-01T10:50:04.013 DEBUG:boto:Token: None 2024-02-01T10:50:04.013 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:04 GMT /vclfcn-63/ 2024-02-01T10:50:04.014 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:EMK2gUI0ziEM+Y3QBXP2RhnnetY= 2024-02-01T10:50:04.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:EMK2gUI0ziEM+Y3QBXP2RhnnetY=', 'Content-Length': '0'} 2024-02-01T10:50:04.018 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000098baae86ad47f03a-0065bb775c-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:04.018 DEBUG:boto:b'vclfcn-63obj11000falseobj12024-02-01T10:48:26.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:04.019 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:04.019 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:04.019 DEBUG:boto:path=/vclfcn-63/?prefix=obj1 2024-02-01T10:50:04.019 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj1 2024-02-01T10:50:04.019 DEBUG:boto:Method: GET 2024-02-01T10:50:04.019 DEBUG:boto:Path: /vclfcn-63/?prefix=obj1 2024-02-01T10:50:04.019 DEBUG:boto:Data: 2024-02-01T10:50:04.019 DEBUG:boto:Headers: {} 2024-02-01T10:50:04.019 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:04.019 DEBUG:boto:Port: 8000 2024-02-01T10:50:04.020 DEBUG:boto:Params: {} 2024-02-01T10:50:04.020 DEBUG:boto:Token: None 2024-02-01T10:50:04.020 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:04 GMT /vclfcn-63/ 2024-02-01T10:50:04.020 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:EMK2gUI0ziEM+Y3QBXP2RhnnetY= 2024-02-01T10:50:04.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:EMK2gUI0ziEM+Y3QBXP2RhnnetY=', 'Content-Length': '0'} 2024-02-01T10:50:04.024 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001bf02aa6793fe5ef-0065bb775c-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:04.025 DEBUG:boto:b'vclfcn-63obj11000falseobj12024-02-01T10:48:26.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:04.025 DEBUG:boto:path=/vclfcn-63/obj1 2024-02-01T10:50:04.025 DEBUG:boto:auth_path=/vclfcn-63/obj1 2024-02-01T10:50:04.025 DEBUG:boto:Method: HEAD 2024-02-01T10:50:04.025 DEBUG:boto:Path: /vclfcn-63/obj1 2024-02-01T10:50:04.025 DEBUG:boto:Data: 2024-02-01T10:50:04.026 DEBUG:boto:Headers: {} 2024-02-01T10:50:04.026 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:04.026 DEBUG:boto:Port: 8001 2024-02-01T10:50:04.026 DEBUG:boto:Params: {} 2024-02-01T10:50:04.026 DEBUG:boto:Token: None 2024-02-01T10:50:04.026 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:04 GMT /vclfcn-63/obj1 2024-02-01T10:50:04.026 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HxSam30rhQ9pO4UiHJo4o5ZNqiQ= 2024-02-01T10:50:04.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HxSam30rhQ9pO4UiHJo4o5ZNqiQ=', 'Content-Length': '0'} 2024-02-01T10:50:04.029 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000074a8daefa0d997a2-0065bb775c-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:04.029 DEBUG:boto:path=/vclfcn-63/obj1 2024-02-01T10:50:04.029 DEBUG:boto:auth_path=/vclfcn-63/obj1 2024-02-01T10:50:04.029 DEBUG:boto:Method: HEAD 2024-02-01T10:50:04.029 DEBUG:boto:Path: /vclfcn-63/obj1 2024-02-01T10:50:04.029 DEBUG:boto:Data: 2024-02-01T10:50:04.030 DEBUG:boto:Headers: {} 2024-02-01T10:50:04.030 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:04.030 DEBUG:boto:Port: 8000 2024-02-01T10:50:04.030 DEBUG:boto:Params: {} 2024-02-01T10:50:04.030 DEBUG:boto:Token: None 2024-02-01T10:50:04.030 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:04 GMT /vclfcn-63/obj1 2024-02-01T10:50:04.030 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HxSam30rhQ9pO4UiHJo4o5ZNqiQ= 2024-02-01T10:50:04.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HxSam30rhQ9pO4UiHJo4o5ZNqiQ=', 'Content-Length': '0'} 2024-02-01T10:50:04.033 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000006f0ac8429d9b2ec-0065bb775c-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:04.033 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:50:04.033 DEBUG:boto:path=/vclfcn-63/obj1 2024-02-01T10:50:04.033 DEBUG:boto:auth_path=/vclfcn-63/obj1 2024-02-01T10:50:04.034 DEBUG:boto:Method: GET 2024-02-01T10:50:04.034 DEBUG:boto:Path: /vclfcn-63/obj1 2024-02-01T10:50:04.034 DEBUG:boto:Data: 2024-02-01T10:50:04.034 DEBUG:boto:Headers: {} 2024-02-01T10:50:04.034 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:04.034 DEBUG:boto:Port: 8001 2024-02-01T10:50:04.034 DEBUG:boto:Params: {} 2024-02-01T10:50:04.034 DEBUG:boto:Token: None 2024-02-01T10:50:04.034 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:04 GMT /vclfcn-63/obj1 2024-02-01T10:50:04.034 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:x3auPd4+05cqCSIPw7hd10q4C8k= 2024-02-01T10:50:04.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:x3auPd4+05cqCSIPw7hd10q4C8k=', 'Content-Length': '0'} 2024-02-01T10:50:04.037 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ee8a189e1e87be7c-0065bb775c-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:04.084 DEBUG:boto:path=/vclfcn-63/obj1 2024-02-01T10:50:04.084 DEBUG:boto:auth_path=/vclfcn-63/obj1 2024-02-01T10:50:04.084 DEBUG:boto:Method: GET 2024-02-01T10:50:04.085 DEBUG:boto:Path: /vclfcn-63/obj1 2024-02-01T10:50:04.085 DEBUG:boto:Data: 2024-02-01T10:50:04.085 DEBUG:boto:Headers: {} 2024-02-01T10:50:04.085 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:04.085 DEBUG:boto:Port: 8000 2024-02-01T10:50:04.085 DEBUG:boto:Params: {} 2024-02-01T10:50:04.085 DEBUG:boto:Token: None 2024-02-01T10:50:04.085 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:04 GMT /vclfcn-63/obj1 2024-02-01T10:50:04.085 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:x3auPd4+05cqCSIPw7hd10q4C8k= 2024-02-01T10:50:04.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:x3auPd4+05cqCSIPw7hd10q4C8k=', 'Content-Length': '0'} 2024-02-01T10:50:04.088 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005480ac8145797f35-0065bb775c-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:04.136 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:04.136 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:04.136 DEBUG:boto:path=/vclfcn-63/?prefix=obj1 2024-02-01T10:50:04.137 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj1 2024-02-01T10:50:04.137 DEBUG:boto:Method: GET 2024-02-01T10:50:04.137 DEBUG:boto:Path: /vclfcn-63/?prefix=obj1 2024-02-01T10:50:04.137 DEBUG:boto:Data: 2024-02-01T10:50:04.137 DEBUG:boto:Headers: {} 2024-02-01T10:50:04.137 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:04.137 DEBUG:boto:Port: 8001 2024-02-01T10:50:04.137 DEBUG:boto:Params: {} 2024-02-01T10:50:04.137 DEBUG:boto:Token: None 2024-02-01T10:50:04.138 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:04 GMT /vclfcn-63/ 2024-02-01T10:50:04.138 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:EMK2gUI0ziEM+Y3QBXP2RhnnetY= 2024-02-01T10:50:04.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:EMK2gUI0ziEM+Y3QBXP2RhnnetY=', 'Content-Length': '0'} 2024-02-01T10:50:04.142 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000690452d991a6f355-0065bb775c-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:04.184 DEBUG:boto:b'vclfcn-63obj11000falseobj12024-02-01T10:48:26.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:04.184 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:04.185 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:04.185 DEBUG:boto:path=/vclfcn-63/?prefix=obj1 2024-02-01T10:50:04.185 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj1 2024-02-01T10:50:04.185 DEBUG:boto:Method: GET 2024-02-01T10:50:04.185 DEBUG:boto:Path: /vclfcn-63/?prefix=obj1 2024-02-01T10:50:04.185 DEBUG:boto:Data: 2024-02-01T10:50:04.185 DEBUG:boto:Headers: {} 2024-02-01T10:50:04.185 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:04.185 DEBUG:boto:Port: 8000 2024-02-01T10:50:04.185 DEBUG:boto:Params: {} 2024-02-01T10:50:04.186 DEBUG:boto:Token: None 2024-02-01T10:50:04.186 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:04 GMT /vclfcn-63/ 2024-02-01T10:50:04.186 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:EMK2gUI0ziEM+Y3QBXP2RhnnetY= 2024-02-01T10:50:04.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:EMK2gUI0ziEM+Y3QBXP2RhnnetY=', 'Content-Length': '0'} 2024-02-01T10:50:04.191 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000296737b24956dd37-0065bb775c-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:04.191 DEBUG:boto:b'vclfcn-63obj11000falseobj12024-02-01T10:48:26.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:04.191 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:50:04.192 DEBUG:boto:path=/vclfcn-63/obj2 2024-02-01T10:50:04.192 DEBUG:boto:auth_path=/vclfcn-63/obj2 2024-02-01T10:50:04.192 DEBUG:boto:path=/vclfcn-63/obj2?versionId=null 2024-02-01T10:50:04.192 DEBUG:boto:auth_path=/vclfcn-63/obj2?versionId=null 2024-02-01T10:50:04.192 DEBUG:boto:Method: GET 2024-02-01T10:50:04.192 DEBUG:boto:Path: /vclfcn-63/obj2?versionId=null 2024-02-01T10:50:04.192 DEBUG:boto:Data: 2024-02-01T10:50:04.192 DEBUG:boto:Headers: {} 2024-02-01T10:50:04.192 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:04.192 DEBUG:boto:Port: 8001 2024-02-01T10:50:04.192 DEBUG:boto:Params: {} 2024-02-01T10:50:04.193 DEBUG:boto:Token: None 2024-02-01T10:50:04.193 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:04 GMT /vclfcn-63/obj2?versionId=null 2024-02-01T10:50:04.193 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:AOMI1ZIN8A3TN+XMySnrhWaCSQ4= 2024-02-01T10:50:04.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:AOMI1ZIN8A3TN+XMySnrhWaCSQ4=', 'Content-Length': '0'} 2024-02-01T10:50:04.196 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005e82153e627a871a-0065bb775c-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:04.196 DEBUG:boto:path=/vclfcn-63/obj2 2024-02-01T10:50:04.196 DEBUG:boto:auth_path=/vclfcn-63/obj2 2024-02-01T10:50:04.196 DEBUG:boto:path=/vclfcn-63/obj2?versionId=null 2024-02-01T10:50:04.196 DEBUG:boto:auth_path=/vclfcn-63/obj2?versionId=null 2024-02-01T10:50:04.196 DEBUG:boto:Method: GET 2024-02-01T10:50:04.196 DEBUG:boto:Path: /vclfcn-63/obj2?versionId=null 2024-02-01T10:50:04.196 DEBUG:boto:Data: 2024-02-01T10:50:04.197 DEBUG:boto:Headers: {} 2024-02-01T10:50:04.197 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:04.197 DEBUG:boto:Port: 8000 2024-02-01T10:50:04.197 DEBUG:boto:Params: {} 2024-02-01T10:50:04.197 DEBUG:boto:Token: None 2024-02-01T10:50:04.197 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:04 GMT /vclfcn-63/obj2?versionId=null 2024-02-01T10:50:04.197 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:AOMI1ZIN8A3TN+XMySnrhWaCSQ4= 2024-02-01T10:50:04.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:AOMI1ZIN8A3TN+XMySnrhWaCSQ4=', 'Content-Length': '0'} 2024-02-01T10:50:04.200 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000873308b7b238c1c3-0065bb775c-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:04.200 DEBUG:boto:path=/vclfcn-63/obj2 2024-02-01T10:50:04.200 DEBUG:boto:auth_path=/vclfcn-63/obj2 2024-02-01T10:50:04.200 DEBUG:boto:path=/vclfcn-63/obj2?versionId=null 2024-02-01T10:50:04.200 DEBUG:boto:auth_path=/vclfcn-63/obj2?versionId=null 2024-02-01T10:50:04.200 DEBUG:boto:Method: HEAD 2024-02-01T10:50:04.201 DEBUG:boto:Path: /vclfcn-63/obj2?versionId=null 2024-02-01T10:50:04.201 DEBUG:boto:Data: 2024-02-01T10:50:04.201 DEBUG:boto:Headers: {} 2024-02-01T10:50:04.201 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:04.201 DEBUG:boto:Port: 8001 2024-02-01T10:50:04.201 DEBUG:boto:Params: {} 2024-02-01T10:50:04.201 DEBUG:boto:Token: None 2024-02-01T10:50:04.201 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:04 GMT /vclfcn-63/obj2?versionId=null 2024-02-01T10:50:04.201 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:JHM5RUFO7g3HwY6lz2r4Pv7nYTU= 2024-02-01T10:50:04.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:JHM5RUFO7g3HwY6lz2r4Pv7nYTU=', 'Content-Length': '0'} 2024-02-01T10:50:04.204 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dca0c02c88d1c528-0065bb775c-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:04.205 DEBUG:boto:path=/vclfcn-63/obj2 2024-02-01T10:50:04.205 DEBUG:boto:auth_path=/vclfcn-63/obj2 2024-02-01T10:50:04.205 DEBUG:boto:path=/vclfcn-63/obj2?versionId=null 2024-02-01T10:50:04.205 DEBUG:boto:auth_path=/vclfcn-63/obj2?versionId=null 2024-02-01T10:50:04.205 DEBUG:boto:Method: HEAD 2024-02-01T10:50:04.205 DEBUG:boto:Path: /vclfcn-63/obj2?versionId=null 2024-02-01T10:50:04.205 DEBUG:boto:Data: 2024-02-01T10:50:04.205 DEBUG:boto:Headers: {} 2024-02-01T10:50:04.205 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:04.205 DEBUG:boto:Port: 8000 2024-02-01T10:50:04.205 DEBUG:boto:Params: {} 2024-02-01T10:50:04.205 DEBUG:boto:Token: None 2024-02-01T10:50:04.206 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:04 GMT /vclfcn-63/obj2?versionId=null 2024-02-01T10:50:04.206 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:JHM5RUFO7g3HwY6lz2r4Pv7nYTU= 2024-02-01T10:50:04.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:JHM5RUFO7g3HwY6lz2r4Pv7nYTU=', 'Content-Length': '0'} 2024-02-01T10:50:04.208 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000236640050e51b80d-0065bb775c-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:04.208 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:50:04.209 DEBUG:boto:path=/vclfcn-63/obj2 2024-02-01T10:50:04.209 DEBUG:boto:auth_path=/vclfcn-63/obj2 2024-02-01T10:50:04.209 DEBUG:boto:path=/vclfcn-63/obj2?versionId=null 2024-02-01T10:50:04.209 DEBUG:boto:auth_path=/vclfcn-63/obj2?versionId=null 2024-02-01T10:50:04.209 DEBUG:boto:Method: GET 2024-02-01T10:50:04.209 DEBUG:boto:Path: /vclfcn-63/obj2?versionId=null 2024-02-01T10:50:04.209 DEBUG:boto:Data: 2024-02-01T10:50:04.209 DEBUG:boto:Headers: {} 2024-02-01T10:50:04.209 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:04.209 DEBUG:boto:Port: 8001 2024-02-01T10:50:04.209 DEBUG:boto:Params: {} 2024-02-01T10:50:04.210 DEBUG:boto:Token: None 2024-02-01T10:50:04.210 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:04 GMT /vclfcn-63/obj2?versionId=null 2024-02-01T10:50:04.210 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:AOMI1ZIN8A3TN+XMySnrhWaCSQ4= 2024-02-01T10:50:04.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:AOMI1ZIN8A3TN+XMySnrhWaCSQ4=', 'Content-Length': '0'} 2024-02-01T10:50:04.212 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000022e3ee25b8c0ae6c-0065bb775c-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:04.256 DEBUG:boto:path=/vclfcn-63/obj2 2024-02-01T10:50:04.256 DEBUG:boto:auth_path=/vclfcn-63/obj2 2024-02-01T10:50:04.256 DEBUG:boto:path=/vclfcn-63/obj2?versionId=null 2024-02-01T10:50:04.256 DEBUG:boto:auth_path=/vclfcn-63/obj2?versionId=null 2024-02-01T10:50:04.256 DEBUG:boto:Method: GET 2024-02-01T10:50:04.257 DEBUG:boto:Path: /vclfcn-63/obj2?versionId=null 2024-02-01T10:50:04.257 DEBUG:boto:Data: 2024-02-01T10:50:04.257 DEBUG:boto:Headers: {} 2024-02-01T10:50:04.257 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:04.257 DEBUG:boto:Port: 8000 2024-02-01T10:50:04.257 DEBUG:boto:Params: {} 2024-02-01T10:50:04.257 DEBUG:boto:Token: None 2024-02-01T10:50:04.257 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:04 GMT /vclfcn-63/obj2?versionId=null 2024-02-01T10:50:04.257 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:AOMI1ZIN8A3TN+XMySnrhWaCSQ4= 2024-02-01T10:50:04.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:AOMI1ZIN8A3TN+XMySnrhWaCSQ4=', 'Content-Length': '0'} 2024-02-01T10:50:04.260 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000082700917890bf8a1-0065bb775c-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:04.304 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:04.304 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:04.304 DEBUG:boto:path=/vclfcn-63/?prefix=obj2 2024-02-01T10:50:04.304 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj2 2024-02-01T10:50:04.305 DEBUG:boto:Method: GET 2024-02-01T10:50:04.305 DEBUG:boto:Path: /vclfcn-63/?prefix=obj2 2024-02-01T10:50:04.305 DEBUG:boto:Data: 2024-02-01T10:50:04.305 DEBUG:boto:Headers: {} 2024-02-01T10:50:04.305 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:04.305 DEBUG:boto:Port: 8001 2024-02-01T10:50:04.305 DEBUG:boto:Params: {} 2024-02-01T10:50:04.305 DEBUG:boto:Token: None 2024-02-01T10:50:04.305 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:04 GMT /vclfcn-63/ 2024-02-01T10:50:04.306 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:EMK2gUI0ziEM+Y3QBXP2RhnnetY= 2024-02-01T10:50:04.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:EMK2gUI0ziEM+Y3QBXP2RhnnetY=', 'Content-Length': '0'} 2024-02-01T10:50:04.310 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9934f0be60bbf31-0065bb775c-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:04.352 DEBUG:boto:b'vclfcn-63obj21000falseobj22024-02-01T10:48:26.507Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:04.353 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:04.353 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:04.353 DEBUG:boto:path=/vclfcn-63/?prefix=obj2 2024-02-01T10:50:04.353 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj2 2024-02-01T10:50:04.353 DEBUG:boto:Method: GET 2024-02-01T10:50:04.353 DEBUG:boto:Path: /vclfcn-63/?prefix=obj2 2024-02-01T10:50:04.353 DEBUG:boto:Data: 2024-02-01T10:50:04.353 DEBUG:boto:Headers: {} 2024-02-01T10:50:04.353 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:04.354 DEBUG:boto:Port: 8000 2024-02-01T10:50:04.354 DEBUG:boto:Params: {} 2024-02-01T10:50:04.354 DEBUG:boto:Token: None 2024-02-01T10:50:04.354 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:04 GMT /vclfcn-63/ 2024-02-01T10:50:04.354 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:EMK2gUI0ziEM+Y3QBXP2RhnnetY= 2024-02-01T10:50:04.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:EMK2gUI0ziEM+Y3QBXP2RhnnetY=', 'Content-Length': '0'} 2024-02-01T10:50:04.359 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8a856a2850203aa-0065bb775c-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:04.359 DEBUG:boto:b'vclfcn-63obj21000falseobj22024-02-01T10:48:26.507Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:04.360 DEBUG:boto:path=/vclfcn-63/obj2 2024-02-01T10:50:04.360 DEBUG:boto:auth_path=/vclfcn-63/obj2 2024-02-01T10:50:04.360 DEBUG:boto:Method: HEAD 2024-02-01T10:50:04.360 DEBUG:boto:Path: /vclfcn-63/obj2 2024-02-01T10:50:04.360 DEBUG:boto:Data: 2024-02-01T10:50:04.360 DEBUG:boto:Headers: {} 2024-02-01T10:50:04.360 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:04.361 DEBUG:boto:Port: 8001 2024-02-01T10:50:04.361 DEBUG:boto:Params: {} 2024-02-01T10:50:04.361 DEBUG:boto:Token: None 2024-02-01T10:50:04.361 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:04 GMT /vclfcn-63/obj2 2024-02-01T10:50:04.361 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:4M73uCRKe2JOQpPbayaPCuDxs7o= 2024-02-01T10:50:04.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:4M73uCRKe2JOQpPbayaPCuDxs7o=', 'Content-Length': '0'} 2024-02-01T10:50:04.365 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000001c456fa7b43975f-0065bb775c-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:04.365 DEBUG:boto:path=/vclfcn-63/obj2 2024-02-01T10:50:04.365 DEBUG:boto:auth_path=/vclfcn-63/obj2 2024-02-01T10:50:04.365 DEBUG:boto:Method: HEAD 2024-02-01T10:50:04.365 DEBUG:boto:Path: /vclfcn-63/obj2 2024-02-01T10:50:04.365 DEBUG:boto:Data: 2024-02-01T10:50:04.366 DEBUG:boto:Headers: {} 2024-02-01T10:50:04.366 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:04.366 DEBUG:boto:Port: 8000 2024-02-01T10:50:04.366 DEBUG:boto:Params: {} 2024-02-01T10:50:04.366 DEBUG:boto:Token: None 2024-02-01T10:50:04.366 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:04 GMT /vclfcn-63/obj2 2024-02-01T10:50:04.366 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:4M73uCRKe2JOQpPbayaPCuDxs7o= 2024-02-01T10:50:04.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:4M73uCRKe2JOQpPbayaPCuDxs7o=', 'Content-Length': '0'} 2024-02-01T10:50:04.369 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f11a2422920fe2ea-0065bb775c-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:04.369 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:50:04.369 DEBUG:boto:path=/vclfcn-63/obj2 2024-02-01T10:50:04.369 DEBUG:boto:auth_path=/vclfcn-63/obj2 2024-02-01T10:50:04.369 DEBUG:boto:Method: GET 2024-02-01T10:50:04.369 DEBUG:boto:Path: /vclfcn-63/obj2 2024-02-01T10:50:04.370 DEBUG:boto:Data: 2024-02-01T10:50:04.370 DEBUG:boto:Headers: {} 2024-02-01T10:50:04.370 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:04.370 DEBUG:boto:Port: 8001 2024-02-01T10:50:04.370 DEBUG:boto:Params: {} 2024-02-01T10:50:04.370 DEBUG:boto:Token: None 2024-02-01T10:50:04.370 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:04 GMT /vclfcn-63/obj2 2024-02-01T10:50:04.370 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:EXZJmHGwIYOABkiaqi6p5gI3Ogw= 2024-02-01T10:50:04.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:EXZJmHGwIYOABkiaqi6p5gI3Ogw=', 'Content-Length': '0'} 2024-02-01T10:50:04.373 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009144798807fe19dc-0065bb775c-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:04.416 DEBUG:boto:path=/vclfcn-63/obj2 2024-02-01T10:50:04.416 DEBUG:boto:auth_path=/vclfcn-63/obj2 2024-02-01T10:50:04.417 DEBUG:boto:Method: GET 2024-02-01T10:50:04.417 DEBUG:boto:Path: /vclfcn-63/obj2 2024-02-01T10:50:04.417 DEBUG:boto:Data: 2024-02-01T10:50:04.417 DEBUG:boto:Headers: {} 2024-02-01T10:50:04.417 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:04.417 DEBUG:boto:Port: 8000 2024-02-01T10:50:04.417 DEBUG:boto:Params: {} 2024-02-01T10:50:04.417 DEBUG:boto:Token: None 2024-02-01T10:50:04.417 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:04 GMT /vclfcn-63/obj2 2024-02-01T10:50:04.418 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:EXZJmHGwIYOABkiaqi6p5gI3Ogw= 2024-02-01T10:50:04.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:EXZJmHGwIYOABkiaqi6p5gI3Ogw=', 'Content-Length': '0'} 2024-02-01T10:50:04.421 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005d9ec9a25115d719-0065bb775c-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:04.464 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:04.464 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:04.464 DEBUG:boto:path=/vclfcn-63/?prefix=obj2 2024-02-01T10:50:04.465 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj2 2024-02-01T10:50:04.465 DEBUG:boto:Method: GET 2024-02-01T10:50:04.465 DEBUG:boto:Path: /vclfcn-63/?prefix=obj2 2024-02-01T10:50:04.465 DEBUG:boto:Data: 2024-02-01T10:50:04.465 DEBUG:boto:Headers: {} 2024-02-01T10:50:04.465 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:04.465 DEBUG:boto:Port: 8001 2024-02-01T10:50:04.465 DEBUG:boto:Params: {} 2024-02-01T10:50:04.466 DEBUG:boto:Token: None 2024-02-01T10:50:04.466 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:04 GMT /vclfcn-63/ 2024-02-01T10:50:04.466 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:EMK2gUI0ziEM+Y3QBXP2RhnnetY= 2024-02-01T10:50:04.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:EMK2gUI0ziEM+Y3QBXP2RhnnetY=', 'Content-Length': '0'} 2024-02-01T10:50:04.471 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072e20930600c379a-0065bb775c-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:04.512 DEBUG:boto:b'vclfcn-63obj21000falseobj22024-02-01T10:48:26.507Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:04.513 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:04.513 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:04.513 DEBUG:boto:path=/vclfcn-63/?prefix=obj2 2024-02-01T10:50:04.513 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj2 2024-02-01T10:50:04.513 DEBUG:boto:Method: GET 2024-02-01T10:50:04.513 DEBUG:boto:Path: /vclfcn-63/?prefix=obj2 2024-02-01T10:50:04.513 DEBUG:boto:Data: 2024-02-01T10:50:04.513 DEBUG:boto:Headers: {} 2024-02-01T10:50:04.514 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:04.514 DEBUG:boto:Port: 8000 2024-02-01T10:50:04.514 DEBUG:boto:Params: {} 2024-02-01T10:50:04.514 DEBUG:boto:Token: None 2024-02-01T10:50:04.514 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:04 GMT /vclfcn-63/ 2024-02-01T10:50:04.514 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:EMK2gUI0ziEM+Y3QBXP2RhnnetY= 2024-02-01T10:50:04.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:EMK2gUI0ziEM+Y3QBXP2RhnnetY=', 'Content-Length': '0'} 2024-02-01T10:50:04.519 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007310e49667dcd546-0065bb775c-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:04.560 DEBUG:boto:b'vclfcn-63obj21000falseobj22024-02-01T10:48:26.507Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:04.561 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:50:04.561 DEBUG:boto:path=/vclfcn-63/obj3 2024-02-01T10:50:04.561 DEBUG:boto:auth_path=/vclfcn-63/obj3 2024-02-01T10:50:04.561 DEBUG:boto:path=/vclfcn-63/obj3?versionId=null 2024-02-01T10:50:04.561 DEBUG:boto:auth_path=/vclfcn-63/obj3?versionId=null 2024-02-01T10:50:04.561 DEBUG:boto:Method: GET 2024-02-01T10:50:04.561 DEBUG:boto:Path: /vclfcn-63/obj3?versionId=null 2024-02-01T10:50:04.561 DEBUG:boto:Data: 2024-02-01T10:50:04.561 DEBUG:boto:Headers: {} 2024-02-01T10:50:04.561 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:04.562 DEBUG:boto:Port: 8001 2024-02-01T10:50:04.562 DEBUG:boto:Params: {} 2024-02-01T10:50:04.562 DEBUG:boto:Token: None 2024-02-01T10:50:04.562 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:04 GMT /vclfcn-63/obj3?versionId=null 2024-02-01T10:50:04.562 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zO5MnTHf90tre69qbIhoJ0eksQE= 2024-02-01T10:50:04.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zO5MnTHf90tre69qbIhoJ0eksQE=', 'Content-Length': '0'} 2024-02-01T10:50:04.565 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000159a19d699c4934f-0065bb775c-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:04.566 DEBUG:boto:path=/vclfcn-63/obj3 2024-02-01T10:50:04.566 DEBUG:boto:auth_path=/vclfcn-63/obj3 2024-02-01T10:50:04.566 DEBUG:boto:path=/vclfcn-63/obj3?versionId=null 2024-02-01T10:50:04.566 DEBUG:boto:auth_path=/vclfcn-63/obj3?versionId=null 2024-02-01T10:50:04.566 DEBUG:boto:Method: GET 2024-02-01T10:50:04.566 DEBUG:boto:Path: /vclfcn-63/obj3?versionId=null 2024-02-01T10:50:04.566 DEBUG:boto:Data: 2024-02-01T10:50:04.567 DEBUG:boto:Headers: {} 2024-02-01T10:50:04.567 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:04.567 DEBUG:boto:Port: 8000 2024-02-01T10:50:04.567 DEBUG:boto:Params: {} 2024-02-01T10:50:04.567 DEBUG:boto:Token: None 2024-02-01T10:50:04.567 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:04 GMT /vclfcn-63/obj3?versionId=null 2024-02-01T10:50:04.567 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zO5MnTHf90tre69qbIhoJ0eksQE= 2024-02-01T10:50:04.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zO5MnTHf90tre69qbIhoJ0eksQE=', 'Content-Length': '0'} 2024-02-01T10:50:04.570 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000024ef33a194913a5d-0065bb775c-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:04.571 DEBUG:boto:path=/vclfcn-63/obj3 2024-02-01T10:50:04.571 DEBUG:boto:auth_path=/vclfcn-63/obj3 2024-02-01T10:50:04.571 DEBUG:boto:path=/vclfcn-63/obj3?versionId=null 2024-02-01T10:50:04.571 DEBUG:boto:auth_path=/vclfcn-63/obj3?versionId=null 2024-02-01T10:50:04.572 DEBUG:boto:Method: HEAD 2024-02-01T10:50:04.572 DEBUG:boto:Path: /vclfcn-63/obj3?versionId=null 2024-02-01T10:50:04.572 DEBUG:boto:Data: 2024-02-01T10:50:04.572 DEBUG:boto:Headers: {} 2024-02-01T10:50:04.573 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:04.573 DEBUG:boto:Port: 8001 2024-02-01T10:50:04.573 DEBUG:boto:Params: {} 2024-02-01T10:50:04.573 DEBUG:boto:Token: None 2024-02-01T10:50:04.574 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:04 GMT /vclfcn-63/obj3?versionId=null 2024-02-01T10:50:04.574 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+8/zeRlClr/3Xe/tmymTgMnKFAw= 2024-02-01T10:50:04.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+8/zeRlClr/3Xe/tmymTgMnKFAw=', 'Content-Length': '0'} 2024-02-01T10:50:04.577 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000012878c75b80f50c4-0065bb775c-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:04.578 DEBUG:boto:path=/vclfcn-63/obj3 2024-02-01T10:50:04.578 DEBUG:boto:auth_path=/vclfcn-63/obj3 2024-02-01T10:50:04.578 DEBUG:boto:path=/vclfcn-63/obj3?versionId=null 2024-02-01T10:50:04.578 DEBUG:boto:auth_path=/vclfcn-63/obj3?versionId=null 2024-02-01T10:50:04.578 DEBUG:boto:Method: HEAD 2024-02-01T10:50:04.579 DEBUG:boto:Path: /vclfcn-63/obj3?versionId=null 2024-02-01T10:50:04.579 DEBUG:boto:Data: 2024-02-01T10:50:04.579 DEBUG:boto:Headers: {} 2024-02-01T10:50:04.579 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:04.579 DEBUG:boto:Port: 8000 2024-02-01T10:50:04.579 DEBUG:boto:Params: {} 2024-02-01T10:50:04.580 DEBUG:boto:Token: None 2024-02-01T10:50:04.580 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:04 GMT /vclfcn-63/obj3?versionId=null 2024-02-01T10:50:04.580 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+8/zeRlClr/3Xe/tmymTgMnKFAw= 2024-02-01T10:50:04.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+8/zeRlClr/3Xe/tmymTgMnKFAw=', 'Content-Length': '0'} 2024-02-01T10:50:04.583 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006d068ed95f004f72-0065bb775c-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:04.583 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:50:04.584 DEBUG:boto:path=/vclfcn-63/obj3 2024-02-01T10:50:04.584 DEBUG:boto:auth_path=/vclfcn-63/obj3 2024-02-01T10:50:04.584 DEBUG:boto:path=/vclfcn-63/obj3?versionId=null 2024-02-01T10:50:04.584 DEBUG:boto:auth_path=/vclfcn-63/obj3?versionId=null 2024-02-01T10:50:04.584 DEBUG:boto:Method: GET 2024-02-01T10:50:04.584 DEBUG:boto:Path: /vclfcn-63/obj3?versionId=null 2024-02-01T10:50:04.584 DEBUG:boto:Data: 2024-02-01T10:50:04.584 DEBUG:boto:Headers: {} 2024-02-01T10:50:04.584 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:04.585 DEBUG:boto:Port: 8001 2024-02-01T10:50:04.585 DEBUG:boto:Params: {} 2024-02-01T10:50:04.585 DEBUG:boto:Token: None 2024-02-01T10:50:04.585 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:04 GMT /vclfcn-63/obj3?versionId=null 2024-02-01T10:50:04.585 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zO5MnTHf90tre69qbIhoJ0eksQE= 2024-02-01T10:50:04.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zO5MnTHf90tre69qbIhoJ0eksQE=', 'Content-Length': '0'} 2024-02-01T10:50:04.588 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000090871061e46f421e-0065bb775c-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:04.628 DEBUG:boto:path=/vclfcn-63/obj3 2024-02-01T10:50:04.628 DEBUG:boto:auth_path=/vclfcn-63/obj3 2024-02-01T10:50:04.628 DEBUG:boto:path=/vclfcn-63/obj3?versionId=null 2024-02-01T10:50:04.628 DEBUG:boto:auth_path=/vclfcn-63/obj3?versionId=null 2024-02-01T10:50:04.629 DEBUG:boto:Method: GET 2024-02-01T10:50:04.629 DEBUG:boto:Path: /vclfcn-63/obj3?versionId=null 2024-02-01T10:50:04.629 DEBUG:boto:Data: 2024-02-01T10:50:04.629 DEBUG:boto:Headers: {} 2024-02-01T10:50:04.629 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:04.629 DEBUG:boto:Port: 8000 2024-02-01T10:50:04.629 DEBUG:boto:Params: {} 2024-02-01T10:50:04.629 DEBUG:boto:Token: None 2024-02-01T10:50:04.629 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:04 GMT /vclfcn-63/obj3?versionId=null 2024-02-01T10:50:04.629 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zO5MnTHf90tre69qbIhoJ0eksQE= 2024-02-01T10:50:04.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zO5MnTHf90tre69qbIhoJ0eksQE=', 'Content-Length': '0'} 2024-02-01T10:50:04.632 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ba8220b5005cdfd1-0065bb775c-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:04.676 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:04.676 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:04.676 DEBUG:boto:path=/vclfcn-63/?prefix=obj3 2024-02-01T10:50:04.677 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj3 2024-02-01T10:50:04.677 DEBUG:boto:Method: GET 2024-02-01T10:50:04.677 DEBUG:boto:Path: /vclfcn-63/?prefix=obj3 2024-02-01T10:50:04.677 DEBUG:boto:Data: 2024-02-01T10:50:04.677 DEBUG:boto:Headers: {} 2024-02-01T10:50:04.677 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:04.677 DEBUG:boto:Port: 8001 2024-02-01T10:50:04.677 DEBUG:boto:Params: {} 2024-02-01T10:50:04.677 DEBUG:boto:Token: None 2024-02-01T10:50:04.677 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:04 GMT /vclfcn-63/ 2024-02-01T10:50:04.678 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:EMK2gUI0ziEM+Y3QBXP2RhnnetY= 2024-02-01T10:50:04.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:EMK2gUI0ziEM+Y3QBXP2RhnnetY=', 'Content-Length': '0'} 2024-02-01T10:50:04.682 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086cd0fcd665f0fb7-0065bb775c-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:04.682 DEBUG:boto:b'vclfcn-63obj31000falseobj32024-02-01T10:48:26.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:04.683 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:04.683 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:04.683 DEBUG:boto:path=/vclfcn-63/?prefix=obj3 2024-02-01T10:50:04.683 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj3 2024-02-01T10:50:04.683 DEBUG:boto:Method: GET 2024-02-01T10:50:04.683 DEBUG:boto:Path: /vclfcn-63/?prefix=obj3 2024-02-01T10:50:04.684 DEBUG:boto:Data: 2024-02-01T10:50:04.684 DEBUG:boto:Headers: {} 2024-02-01T10:50:04.684 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:04.684 DEBUG:boto:Port: 8000 2024-02-01T10:50:04.684 DEBUG:boto:Params: {} 2024-02-01T10:50:04.684 DEBUG:boto:Token: None 2024-02-01T10:50:04.684 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:04 GMT /vclfcn-63/ 2024-02-01T10:50:04.684 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:EMK2gUI0ziEM+Y3QBXP2RhnnetY= 2024-02-01T10:50:04.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:EMK2gUI0ziEM+Y3QBXP2RhnnetY=', 'Content-Length': '0'} 2024-02-01T10:50:04.690 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e89d4bf7bb5b5cc7-0065bb775c-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:04.690 DEBUG:boto:b'vclfcn-63obj31000falseobj32024-02-01T10:48:26.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:04.690 DEBUG:boto:path=/vclfcn-63/obj3 2024-02-01T10:50:04.691 DEBUG:boto:auth_path=/vclfcn-63/obj3 2024-02-01T10:50:04.691 DEBUG:boto:Method: HEAD 2024-02-01T10:50:04.691 DEBUG:boto:Path: /vclfcn-63/obj3 2024-02-01T10:50:04.691 DEBUG:boto:Data: 2024-02-01T10:50:04.691 DEBUG:boto:Headers: {} 2024-02-01T10:50:04.691 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:04.691 DEBUG:boto:Port: 8001 2024-02-01T10:50:04.691 DEBUG:boto:Params: {} 2024-02-01T10:50:04.691 DEBUG:boto:Token: None 2024-02-01T10:50:04.692 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:04 GMT /vclfcn-63/obj3 2024-02-01T10:50:04.692 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Q6FTnXPTgJLK3YW77MyoeHveFu8= 2024-02-01T10:50:04.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Q6FTnXPTgJLK3YW77MyoeHveFu8=', 'Content-Length': '0'} 2024-02-01T10:50:04.695 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000084521927429c8d8a-0065bb775c-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:04.695 DEBUG:boto:path=/vclfcn-63/obj3 2024-02-01T10:50:04.695 DEBUG:boto:auth_path=/vclfcn-63/obj3 2024-02-01T10:50:04.695 DEBUG:boto:Method: HEAD 2024-02-01T10:50:04.695 DEBUG:boto:Path: /vclfcn-63/obj3 2024-02-01T10:50:04.695 DEBUG:boto:Data: 2024-02-01T10:50:04.696 DEBUG:boto:Headers: {} 2024-02-01T10:50:04.696 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:04.696 DEBUG:boto:Port: 8000 2024-02-01T10:50:04.696 DEBUG:boto:Params: {} 2024-02-01T10:50:04.696 DEBUG:boto:Token: None 2024-02-01T10:50:04.696 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:04 GMT /vclfcn-63/obj3 2024-02-01T10:50:04.696 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Q6FTnXPTgJLK3YW77MyoeHveFu8= 2024-02-01T10:50:04.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Q6FTnXPTgJLK3YW77MyoeHveFu8=', 'Content-Length': '0'} 2024-02-01T10:50:04.699 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000015c299c3def016a0-0065bb775c-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:04.699 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:50:04.699 DEBUG:boto:path=/vclfcn-63/obj3 2024-02-01T10:50:04.699 DEBUG:boto:auth_path=/vclfcn-63/obj3 2024-02-01T10:50:04.700 DEBUG:boto:Method: GET 2024-02-01T10:50:04.700 DEBUG:boto:Path: /vclfcn-63/obj3 2024-02-01T10:50:04.700 DEBUG:boto:Data: 2024-02-01T10:50:04.700 DEBUG:boto:Headers: {} 2024-02-01T10:50:04.700 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:04.700 DEBUG:boto:Port: 8001 2024-02-01T10:50:04.700 DEBUG:boto:Params: {} 2024-02-01T10:50:04.700 DEBUG:boto:Token: None 2024-02-01T10:50:04.700 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:04 GMT /vclfcn-63/obj3 2024-02-01T10:50:04.701 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:y/n0kAmWWRMjVV4jrG8UAxeE1ZA= 2024-02-01T10:50:04.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:y/n0kAmWWRMjVV4jrG8UAxeE1ZA=', 'Content-Length': '0'} 2024-02-01T10:50:04.703 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d4bd7f65f6fb7030-0065bb775c-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:04.748 DEBUG:boto:path=/vclfcn-63/obj3 2024-02-01T10:50:04.749 DEBUG:boto:auth_path=/vclfcn-63/obj3 2024-02-01T10:50:04.749 DEBUG:boto:Method: GET 2024-02-01T10:50:04.749 DEBUG:boto:Path: /vclfcn-63/obj3 2024-02-01T10:50:04.749 DEBUG:boto:Data: 2024-02-01T10:50:04.749 DEBUG:boto:Headers: {} 2024-02-01T10:50:04.749 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:04.749 DEBUG:boto:Port: 8000 2024-02-01T10:50:04.749 DEBUG:boto:Params: {} 2024-02-01T10:50:04.749 DEBUG:boto:Token: None 2024-02-01T10:50:04.750 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:04 GMT /vclfcn-63/obj3 2024-02-01T10:50:04.750 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:y/n0kAmWWRMjVV4jrG8UAxeE1ZA= 2024-02-01T10:50:04.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:y/n0kAmWWRMjVV4jrG8UAxeE1ZA=', 'Content-Length': '0'} 2024-02-01T10:50:04.753 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dac2f2bafe934bef-0065bb775c-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:04.796 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:04.797 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:04.797 DEBUG:boto:path=/vclfcn-63/?prefix=obj3 2024-02-01T10:50:04.797 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj3 2024-02-01T10:50:04.797 DEBUG:boto:Method: GET 2024-02-01T10:50:04.797 DEBUG:boto:Path: /vclfcn-63/?prefix=obj3 2024-02-01T10:50:04.798 DEBUG:boto:Data: 2024-02-01T10:50:04.798 DEBUG:boto:Headers: {} 2024-02-01T10:50:04.798 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:04.798 DEBUG:boto:Port: 8001 2024-02-01T10:50:04.798 DEBUG:boto:Params: {} 2024-02-01T10:50:04.799 DEBUG:boto:Token: None 2024-02-01T10:50:04.799 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:04 GMT /vclfcn-63/ 2024-02-01T10:50:04.799 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:EMK2gUI0ziEM+Y3QBXP2RhnnetY= 2024-02-01T10:50:04.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:EMK2gUI0ziEM+Y3QBXP2RhnnetY=', 'Content-Length': '0'} 2024-02-01T10:50:04.804 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f5c94dcb0f0e32c-0065bb775c-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:04.848 DEBUG:boto:b'vclfcn-63obj31000falseobj32024-02-01T10:48:26.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:04.849 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:04.849 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:04.849 DEBUG:boto:path=/vclfcn-63/?prefix=obj3 2024-02-01T10:50:04.849 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj3 2024-02-01T10:50:04.850 DEBUG:boto:Method: GET 2024-02-01T10:50:04.850 DEBUG:boto:Path: /vclfcn-63/?prefix=obj3 2024-02-01T10:50:04.850 DEBUG:boto:Data: 2024-02-01T10:50:04.850 DEBUG:boto:Headers: {} 2024-02-01T10:50:04.850 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:04.850 DEBUG:boto:Port: 8000 2024-02-01T10:50:04.850 DEBUG:boto:Params: {} 2024-02-01T10:50:04.850 DEBUG:boto:Token: None 2024-02-01T10:50:04.851 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:04 GMT /vclfcn-63/ 2024-02-01T10:50:04.851 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:EMK2gUI0ziEM+Y3QBXP2RhnnetY= 2024-02-01T10:50:04.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:EMK2gUI0ziEM+Y3QBXP2RhnnetY=', 'Content-Length': '0'} 2024-02-01T10:50:04.855 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a140347cf76d9413-0065bb775c-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:04.856 DEBUG:boto:b'vclfcn-63obj31000falseobj32024-02-01T10:48:26.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:04.856 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:50:04.857 DEBUG:boto:path=/vclfcn-63/obj4 2024-02-01T10:50:04.857 DEBUG:boto:auth_path=/vclfcn-63/obj4 2024-02-01T10:50:04.857 DEBUG:boto:path=/vclfcn-63/obj4?versionId=null 2024-02-01T10:50:04.857 DEBUG:boto:auth_path=/vclfcn-63/obj4?versionId=null 2024-02-01T10:50:04.857 DEBUG:boto:Method: GET 2024-02-01T10:50:04.857 DEBUG:boto:Path: /vclfcn-63/obj4?versionId=null 2024-02-01T10:50:04.857 DEBUG:boto:Data: 2024-02-01T10:50:04.857 DEBUG:boto:Headers: {} 2024-02-01T10:50:04.857 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:04.857 DEBUG:boto:Port: 8001 2024-02-01T10:50:04.858 DEBUG:boto:Params: {} 2024-02-01T10:50:04.858 DEBUG:boto:Token: None 2024-02-01T10:50:04.858 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:04 GMT /vclfcn-63/obj4?versionId=null 2024-02-01T10:50:04.858 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nKaCHhiiX097/eWbdK7oZlMJN3M= 2024-02-01T10:50:04.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nKaCHhiiX097/eWbdK7oZlMJN3M=', 'Content-Length': '0'} 2024-02-01T10:50:04.861 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000072baafc331e9554a-0065bb775c-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:04.905 DEBUG:boto:path=/vclfcn-63/obj4 2024-02-01T10:50:04.905 DEBUG:boto:auth_path=/vclfcn-63/obj4 2024-02-01T10:50:04.905 DEBUG:boto:path=/vclfcn-63/obj4?versionId=null 2024-02-01T10:50:04.905 DEBUG:boto:auth_path=/vclfcn-63/obj4?versionId=null 2024-02-01T10:50:04.905 DEBUG:boto:Method: GET 2024-02-01T10:50:04.905 DEBUG:boto:Path: /vclfcn-63/obj4?versionId=null 2024-02-01T10:50:04.905 DEBUG:boto:Data: 2024-02-01T10:50:04.907 DEBUG:boto:Headers: {} 2024-02-01T10:50:04.907 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:04.908 DEBUG:boto:Port: 8000 2024-02-01T10:50:04.908 DEBUG:boto:Params: {} 2024-02-01T10:50:04.908 DEBUG:boto:Token: None 2024-02-01T10:50:04.909 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:04 GMT /vclfcn-63/obj4?versionId=null 2024-02-01T10:50:04.909 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nKaCHhiiX097/eWbdK7oZlMJN3M= 2024-02-01T10:50:04.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nKaCHhiiX097/eWbdK7oZlMJN3M=', 'Content-Length': '0'} 2024-02-01T10:50:04.912 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000be6f46327d18f29f-0065bb775c-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:04.912 DEBUG:boto:path=/vclfcn-63/obj4 2024-02-01T10:50:04.912 DEBUG:boto:auth_path=/vclfcn-63/obj4 2024-02-01T10:50:04.913 DEBUG:boto:path=/vclfcn-63/obj4?versionId=null 2024-02-01T10:50:04.913 DEBUG:boto:auth_path=/vclfcn-63/obj4?versionId=null 2024-02-01T10:50:04.913 DEBUG:boto:Method: HEAD 2024-02-01T10:50:04.913 DEBUG:boto:Path: /vclfcn-63/obj4?versionId=null 2024-02-01T10:50:04.913 DEBUG:boto:Data: 2024-02-01T10:50:04.913 DEBUG:boto:Headers: {} 2024-02-01T10:50:04.913 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:04.913 DEBUG:boto:Port: 8001 2024-02-01T10:50:04.913 DEBUG:boto:Params: {} 2024-02-01T10:50:04.913 DEBUG:boto:Token: None 2024-02-01T10:50:04.914 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:04 GMT /vclfcn-63/obj4?versionId=null 2024-02-01T10:50:04.914 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RMAmI56nd1TG7TdQLY2IALCW6ko= 2024-02-01T10:50:04.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RMAmI56nd1TG7TdQLY2IALCW6ko=', 'Content-Length': '0'} 2024-02-01T10:50:04.918 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000413361dca3ef68b6-0065bb775c-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:04.918 DEBUG:boto:path=/vclfcn-63/obj4 2024-02-01T10:50:04.918 DEBUG:boto:auth_path=/vclfcn-63/obj4 2024-02-01T10:50:04.918 DEBUG:boto:path=/vclfcn-63/obj4?versionId=null 2024-02-01T10:50:04.918 DEBUG:boto:auth_path=/vclfcn-63/obj4?versionId=null 2024-02-01T10:50:04.918 DEBUG:boto:Method: HEAD 2024-02-01T10:50:04.918 DEBUG:boto:Path: /vclfcn-63/obj4?versionId=null 2024-02-01T10:50:04.918 DEBUG:boto:Data: 2024-02-01T10:50:04.918 DEBUG:boto:Headers: {} 2024-02-01T10:50:04.919 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:04.919 DEBUG:boto:Port: 8000 2024-02-01T10:50:04.919 DEBUG:boto:Params: {} 2024-02-01T10:50:04.919 DEBUG:boto:Token: None 2024-02-01T10:50:04.919 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:04 GMT /vclfcn-63/obj4?versionId=null 2024-02-01T10:50:04.919 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RMAmI56nd1TG7TdQLY2IALCW6ko= 2024-02-01T10:50:04.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RMAmI56nd1TG7TdQLY2IALCW6ko=', 'Content-Length': '0'} 2024-02-01T10:50:04.922 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c5b8f34fcbfb5ec7-0065bb775c-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:04.922 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:50:04.923 DEBUG:boto:path=/vclfcn-63/obj4 2024-02-01T10:50:04.923 DEBUG:boto:auth_path=/vclfcn-63/obj4 2024-02-01T10:50:04.923 DEBUG:boto:path=/vclfcn-63/obj4?versionId=null 2024-02-01T10:50:04.923 DEBUG:boto:auth_path=/vclfcn-63/obj4?versionId=null 2024-02-01T10:50:04.923 DEBUG:boto:Method: GET 2024-02-01T10:50:04.923 DEBUG:boto:Path: /vclfcn-63/obj4?versionId=null 2024-02-01T10:50:04.923 DEBUG:boto:Data: 2024-02-01T10:50:04.923 DEBUG:boto:Headers: {} 2024-02-01T10:50:04.923 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:04.923 DEBUG:boto:Port: 8001 2024-02-01T10:50:04.923 DEBUG:boto:Params: {} 2024-02-01T10:50:04.924 DEBUG:boto:Token: None 2024-02-01T10:50:04.924 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:04 GMT /vclfcn-63/obj4?versionId=null 2024-02-01T10:50:04.924 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nKaCHhiiX097/eWbdK7oZlMJN3M= 2024-02-01T10:50:04.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nKaCHhiiX097/eWbdK7oZlMJN3M=', 'Content-Length': '0'} 2024-02-01T10:50:04.927 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eff877a88311987d-0065bb775c-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:04.968 DEBUG:boto:path=/vclfcn-63/obj4 2024-02-01T10:50:04.969 DEBUG:boto:auth_path=/vclfcn-63/obj4 2024-02-01T10:50:04.969 DEBUG:boto:path=/vclfcn-63/obj4?versionId=null 2024-02-01T10:50:04.970 DEBUG:boto:auth_path=/vclfcn-63/obj4?versionId=null 2024-02-01T10:50:04.970 DEBUG:boto:Method: GET 2024-02-01T10:50:04.970 DEBUG:boto:Path: /vclfcn-63/obj4?versionId=null 2024-02-01T10:50:04.971 DEBUG:boto:Data: 2024-02-01T10:50:04.971 DEBUG:boto:Headers: {} 2024-02-01T10:50:04.971 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:04.971 DEBUG:boto:Port: 8000 2024-02-01T10:50:04.971 DEBUG:boto:Params: {} 2024-02-01T10:50:04.972 DEBUG:boto:Token: None 2024-02-01T10:50:04.972 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:04 GMT /vclfcn-63/obj4?versionId=null 2024-02-01T10:50:04.973 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nKaCHhiiX097/eWbdK7oZlMJN3M= 2024-02-01T10:50:04.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:04 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nKaCHhiiX097/eWbdK7oZlMJN3M=', 'Content-Length': '0'} 2024-02-01T10:50:04.976 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c9a3194f3f1c1d26-0065bb775c-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:04 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:05.016 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:05.017 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:05.017 DEBUG:boto:path=/vclfcn-63/?prefix=obj4 2024-02-01T10:50:05.017 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj4 2024-02-01T10:50:05.017 DEBUG:boto:Method: GET 2024-02-01T10:50:05.018 DEBUG:boto:Path: /vclfcn-63/?prefix=obj4 2024-02-01T10:50:05.018 DEBUG:boto:Data: 2024-02-01T10:50:05.018 DEBUG:boto:Headers: {} 2024-02-01T10:50:05.018 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:05.018 DEBUG:boto:Port: 8001 2024-02-01T10:50:05.019 DEBUG:boto:Params: {} 2024-02-01T10:50:05.019 DEBUG:boto:Token: None 2024-02-01T10:50:05.019 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:05 GMT /vclfcn-63/ 2024-02-01T10:50:05.020 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9WtEsClKkCwkqpqMUXOHtzjNa7g= 2024-02-01T10:50:05.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9WtEsClKkCwkqpqMUXOHtzjNa7g=', 'Content-Length': '0'} 2024-02-01T10:50:05.024 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019ea64ffef708e3b-0065bb775d-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:05.068 DEBUG:boto:b'vclfcn-63obj41000falseobj42024-02-01T10:48:26.575Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:05.069 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:05.069 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:05.069 DEBUG:boto:path=/vclfcn-63/?prefix=obj4 2024-02-01T10:50:05.069 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj4 2024-02-01T10:50:05.069 DEBUG:boto:Method: GET 2024-02-01T10:50:05.070 DEBUG:boto:Path: /vclfcn-63/?prefix=obj4 2024-02-01T10:50:05.070 DEBUG:boto:Data: 2024-02-01T10:50:05.070 DEBUG:boto:Headers: {} 2024-02-01T10:50:05.070 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:05.070 DEBUG:boto:Port: 8000 2024-02-01T10:50:05.070 DEBUG:boto:Params: {} 2024-02-01T10:50:05.070 DEBUG:boto:Token: None 2024-02-01T10:50:05.070 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:05 GMT /vclfcn-63/ 2024-02-01T10:50:05.070 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9WtEsClKkCwkqpqMUXOHtzjNa7g= 2024-02-01T10:50:05.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9WtEsClKkCwkqpqMUXOHtzjNa7g=', 'Content-Length': '0'} 2024-02-01T10:50:05.075 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db6279214e7699b4-0065bb775d-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:05.075 DEBUG:boto:b'vclfcn-63obj41000falseobj42024-02-01T10:48:26.575Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:05.076 DEBUG:boto:path=/vclfcn-63/obj4 2024-02-01T10:50:05.076 DEBUG:boto:auth_path=/vclfcn-63/obj4 2024-02-01T10:50:05.076 DEBUG:boto:Method: HEAD 2024-02-01T10:50:05.077 DEBUG:boto:Path: /vclfcn-63/obj4 2024-02-01T10:50:05.077 DEBUG:boto:Data: 2024-02-01T10:50:05.077 DEBUG:boto:Headers: {} 2024-02-01T10:50:05.077 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:05.077 DEBUG:boto:Port: 8001 2024-02-01T10:50:05.077 DEBUG:boto:Params: {} 2024-02-01T10:50:05.077 DEBUG:boto:Token: None 2024-02-01T10:50:05.077 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:05 GMT /vclfcn-63/obj4 2024-02-01T10:50:05.077 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vygFTOhrc0LG8YHkO8Ol/HGy6zU= 2024-02-01T10:50:05.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vygFTOhrc0LG8YHkO8Ol/HGy6zU=', 'Content-Length': '0'} 2024-02-01T10:50:05.081 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e914f5029972db70-0065bb775d-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:05.081 DEBUG:boto:path=/vclfcn-63/obj4 2024-02-01T10:50:05.081 DEBUG:boto:auth_path=/vclfcn-63/obj4 2024-02-01T10:50:05.081 DEBUG:boto:Method: HEAD 2024-02-01T10:50:05.081 DEBUG:boto:Path: /vclfcn-63/obj4 2024-02-01T10:50:05.081 DEBUG:boto:Data: 2024-02-01T10:50:05.081 DEBUG:boto:Headers: {} 2024-02-01T10:50:05.081 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:05.082 DEBUG:boto:Port: 8000 2024-02-01T10:50:05.082 DEBUG:boto:Params: {} 2024-02-01T10:50:05.082 DEBUG:boto:Token: None 2024-02-01T10:50:05.082 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:05 GMT /vclfcn-63/obj4 2024-02-01T10:50:05.082 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vygFTOhrc0LG8YHkO8Ol/HGy6zU= 2024-02-01T10:50:05.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vygFTOhrc0LG8YHkO8Ol/HGy6zU=', 'Content-Length': '0'} 2024-02-01T10:50:05.085 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007d228ae3591058dc-0065bb775d-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:05.085 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:50:05.085 DEBUG:boto:path=/vclfcn-63/obj4 2024-02-01T10:50:05.085 DEBUG:boto:auth_path=/vclfcn-63/obj4 2024-02-01T10:50:05.086 DEBUG:boto:Method: GET 2024-02-01T10:50:05.086 DEBUG:boto:Path: /vclfcn-63/obj4 2024-02-01T10:50:05.086 DEBUG:boto:Data: 2024-02-01T10:50:05.086 DEBUG:boto:Headers: {} 2024-02-01T10:50:05.086 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:05.086 DEBUG:boto:Port: 8001 2024-02-01T10:50:05.086 DEBUG:boto:Params: {} 2024-02-01T10:50:05.086 DEBUG:boto:Token: None 2024-02-01T10:50:05.086 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:05 GMT /vclfcn-63/obj4 2024-02-01T10:50:05.086 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:rP/KL7N5VKfe6KUTf6M0hCbLXiY= 2024-02-01T10:50:05.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:rP/KL7N5VKfe6KUTf6M0hCbLXiY=', 'Content-Length': '0'} 2024-02-01T10:50:05.089 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000434ccc23c0abc0a-0065bb775d-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:05.132 DEBUG:boto:path=/vclfcn-63/obj4 2024-02-01T10:50:05.132 DEBUG:boto:auth_path=/vclfcn-63/obj4 2024-02-01T10:50:05.133 DEBUG:boto:Method: GET 2024-02-01T10:50:05.133 DEBUG:boto:Path: /vclfcn-63/obj4 2024-02-01T10:50:05.133 DEBUG:boto:Data: 2024-02-01T10:50:05.133 DEBUG:boto:Headers: {} 2024-02-01T10:50:05.133 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:05.133 DEBUG:boto:Port: 8000 2024-02-01T10:50:05.133 DEBUG:boto:Params: {} 2024-02-01T10:50:05.133 DEBUG:boto:Token: None 2024-02-01T10:50:05.133 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:05 GMT /vclfcn-63/obj4 2024-02-01T10:50:05.134 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:rP/KL7N5VKfe6KUTf6M0hCbLXiY= 2024-02-01T10:50:05.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:rP/KL7N5VKfe6KUTf6M0hCbLXiY=', 'Content-Length': '0'} 2024-02-01T10:50:05.137 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000c56efd58ddbe285-0065bb775d-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:05.180 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:05.180 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:05.181 DEBUG:boto:path=/vclfcn-63/?prefix=obj4 2024-02-01T10:50:05.181 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj4 2024-02-01T10:50:05.181 DEBUG:boto:Method: GET 2024-02-01T10:50:05.181 DEBUG:boto:Path: /vclfcn-63/?prefix=obj4 2024-02-01T10:50:05.181 DEBUG:boto:Data: 2024-02-01T10:50:05.181 DEBUG:boto:Headers: {} 2024-02-01T10:50:05.181 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:05.181 DEBUG:boto:Port: 8001 2024-02-01T10:50:05.181 DEBUG:boto:Params: {} 2024-02-01T10:50:05.181 DEBUG:boto:Token: None 2024-02-01T10:50:05.182 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:05 GMT /vclfcn-63/ 2024-02-01T10:50:05.182 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9WtEsClKkCwkqpqMUXOHtzjNa7g= 2024-02-01T10:50:05.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9WtEsClKkCwkqpqMUXOHtzjNa7g=', 'Content-Length': '0'} 2024-02-01T10:50:05.187 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ebe2e669cb825b16-0065bb775d-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:05.228 DEBUG:boto:b'vclfcn-63obj41000falseobj42024-02-01T10:48:26.575Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:05.229 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:05.229 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:05.229 DEBUG:boto:path=/vclfcn-63/?prefix=obj4 2024-02-01T10:50:05.229 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj4 2024-02-01T10:50:05.229 DEBUG:boto:Method: GET 2024-02-01T10:50:05.229 DEBUG:boto:Path: /vclfcn-63/?prefix=obj4 2024-02-01T10:50:05.229 DEBUG:boto:Data: 2024-02-01T10:50:05.229 DEBUG:boto:Headers: {} 2024-02-01T10:50:05.229 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:05.230 DEBUG:boto:Port: 8000 2024-02-01T10:50:05.230 DEBUG:boto:Params: {} 2024-02-01T10:50:05.230 DEBUG:boto:Token: None 2024-02-01T10:50:05.230 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:05 GMT /vclfcn-63/ 2024-02-01T10:50:05.230 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9WtEsClKkCwkqpqMUXOHtzjNa7g= 2024-02-01T10:50:05.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9WtEsClKkCwkqpqMUXOHtzjNa7g=', 'Content-Length': '0'} 2024-02-01T10:50:05.235 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aed3e38876b15f79-0065bb775d-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:05.276 DEBUG:boto:b'vclfcn-63obj41000falseobj42024-02-01T10:48:26.575Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:05.276 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-02-01T10:50:05.277 DEBUG:boto:path=/vclfcn-63/obj5 2024-02-01T10:50:05.277 DEBUG:boto:auth_path=/vclfcn-63/obj5 2024-02-01T10:50:05.277 DEBUG:boto:path=/vclfcn-63/obj5?versionId=null 2024-02-01T10:50:05.277 DEBUG:boto:auth_path=/vclfcn-63/obj5?versionId=null 2024-02-01T10:50:05.277 DEBUG:boto:Method: GET 2024-02-01T10:50:05.277 DEBUG:boto:Path: /vclfcn-63/obj5?versionId=null 2024-02-01T10:50:05.277 DEBUG:boto:Data: 2024-02-01T10:50:05.277 DEBUG:boto:Headers: {} 2024-02-01T10:50:05.277 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:05.277 DEBUG:boto:Port: 8001 2024-02-01T10:50:05.278 DEBUG:boto:Params: {} 2024-02-01T10:50:05.278 DEBUG:boto:Token: None 2024-02-01T10:50:05.278 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:05 GMT /vclfcn-63/obj5?versionId=null 2024-02-01T10:50:05.278 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QdGfz3zJDx8KdI3jbxImtIAdQfE= 2024-02-01T10:50:05.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QdGfz3zJDx8KdI3jbxImtIAdQfE=', 'Content-Length': '0'} 2024-02-01T10:50:05.282 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a749772173354cf7-0065bb775d-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:05.282 DEBUG:boto:path=/vclfcn-63/obj5 2024-02-01T10:50:05.282 DEBUG:boto:auth_path=/vclfcn-63/obj5 2024-02-01T10:50:05.282 DEBUG:boto:path=/vclfcn-63/obj5?versionId=null 2024-02-01T10:50:05.282 DEBUG:boto:auth_path=/vclfcn-63/obj5?versionId=null 2024-02-01T10:50:05.282 DEBUG:boto:Method: GET 2024-02-01T10:50:05.282 DEBUG:boto:Path: /vclfcn-63/obj5?versionId=null 2024-02-01T10:50:05.282 DEBUG:boto:Data: 2024-02-01T10:50:05.283 DEBUG:boto:Headers: {} 2024-02-01T10:50:05.283 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:05.283 DEBUG:boto:Port: 8000 2024-02-01T10:50:05.283 DEBUG:boto:Params: {} 2024-02-01T10:50:05.283 DEBUG:boto:Token: None 2024-02-01T10:50:05.283 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:05 GMT /vclfcn-63/obj5?versionId=null 2024-02-01T10:50:05.283 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QdGfz3zJDx8KdI3jbxImtIAdQfE= 2024-02-01T10:50:05.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QdGfz3zJDx8KdI3jbxImtIAdQfE=', 'Content-Length': '0'} 2024-02-01T10:50:05.286 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b20970384b2a1da4-0065bb775d-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:05.287 DEBUG:boto:path=/vclfcn-63/obj5 2024-02-01T10:50:05.287 DEBUG:boto:auth_path=/vclfcn-63/obj5 2024-02-01T10:50:05.287 DEBUG:boto:path=/vclfcn-63/obj5?versionId=null 2024-02-01T10:50:05.287 DEBUG:boto:auth_path=/vclfcn-63/obj5?versionId=null 2024-02-01T10:50:05.288 DEBUG:boto:Method: HEAD 2024-02-01T10:50:05.288 DEBUG:boto:Path: /vclfcn-63/obj5?versionId=null 2024-02-01T10:50:05.288 DEBUG:boto:Data: 2024-02-01T10:50:05.288 DEBUG:boto:Headers: {} 2024-02-01T10:50:05.288 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:05.289 DEBUG:boto:Port: 8001 2024-02-01T10:50:05.289 DEBUG:boto:Params: {} 2024-02-01T10:50:05.289 DEBUG:boto:Token: None 2024-02-01T10:50:05.289 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:05 GMT /vclfcn-63/obj5?versionId=null 2024-02-01T10:50:05.289 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:uv3MJcHakx6TxEB1OTUYyvlB9uc= 2024-02-01T10:50:05.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:uv3MJcHakx6TxEB1OTUYyvlB9uc=', 'Content-Length': '0'} 2024-02-01T10:50:05.293 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008c760a0d0145ea60-0065bb775d-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:05.293 DEBUG:boto:path=/vclfcn-63/obj5 2024-02-01T10:50:05.293 DEBUG:boto:auth_path=/vclfcn-63/obj5 2024-02-01T10:50:05.293 DEBUG:boto:path=/vclfcn-63/obj5?versionId=null 2024-02-01T10:50:05.294 DEBUG:boto:auth_path=/vclfcn-63/obj5?versionId=null 2024-02-01T10:50:05.294 DEBUG:boto:Method: HEAD 2024-02-01T10:50:05.294 DEBUG:boto:Path: /vclfcn-63/obj5?versionId=null 2024-02-01T10:50:05.294 DEBUG:boto:Data: 2024-02-01T10:50:05.295 DEBUG:boto:Headers: {} 2024-02-01T10:50:05.295 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:05.295 DEBUG:boto:Port: 8000 2024-02-01T10:50:05.295 DEBUG:boto:Params: {} 2024-02-01T10:50:05.295 DEBUG:boto:Token: None 2024-02-01T10:50:05.296 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:05 GMT /vclfcn-63/obj5?versionId=null 2024-02-01T10:50:05.296 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:uv3MJcHakx6TxEB1OTUYyvlB9uc= 2024-02-01T10:50:05.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:uv3MJcHakx6TxEB1OTUYyvlB9uc=', 'Content-Length': '0'} 2024-02-01T10:50:05.299 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000485c803ac05c513a-0065bb775d-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:05.299 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-02-01T10:50:05.300 DEBUG:boto:path=/vclfcn-63/obj5 2024-02-01T10:50:05.300 DEBUG:boto:auth_path=/vclfcn-63/obj5 2024-02-01T10:50:05.300 DEBUG:boto:path=/vclfcn-63/obj5?versionId=null 2024-02-01T10:50:05.300 DEBUG:boto:auth_path=/vclfcn-63/obj5?versionId=null 2024-02-01T10:50:05.300 DEBUG:boto:Method: GET 2024-02-01T10:50:05.300 DEBUG:boto:Path: /vclfcn-63/obj5?versionId=null 2024-02-01T10:50:05.301 DEBUG:boto:Data: 2024-02-01T10:50:05.301 DEBUG:boto:Headers: {} 2024-02-01T10:50:05.301 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:05.301 DEBUG:boto:Port: 8001 2024-02-01T10:50:05.301 DEBUG:boto:Params: {} 2024-02-01T10:50:05.302 DEBUG:boto:Token: None 2024-02-01T10:50:05.302 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:05 GMT /vclfcn-63/obj5?versionId=null 2024-02-01T10:50:05.302 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QdGfz3zJDx8KdI3jbxImtIAdQfE= 2024-02-01T10:50:05.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QdGfz3zJDx8KdI3jbxImtIAdQfE=', 'Content-Length': '0'} 2024-02-01T10:50:05.306 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008f53384003d9610e-0065bb775d-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:05.348 DEBUG:boto:path=/vclfcn-63/obj5 2024-02-01T10:50:05.348 DEBUG:boto:auth_path=/vclfcn-63/obj5 2024-02-01T10:50:05.348 DEBUG:boto:path=/vclfcn-63/obj5?versionId=null 2024-02-01T10:50:05.351 DEBUG:boto:auth_path=/vclfcn-63/obj5?versionId=null 2024-02-01T10:50:05.351 DEBUG:boto:Method: GET 2024-02-01T10:50:05.351 DEBUG:boto:Path: /vclfcn-63/obj5?versionId=null 2024-02-01T10:50:05.351 DEBUG:boto:Data: 2024-02-01T10:50:05.352 DEBUG:boto:Headers: {} 2024-02-01T10:50:05.352 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:05.352 DEBUG:boto:Port: 8000 2024-02-01T10:50:05.352 DEBUG:boto:Params: {} 2024-02-01T10:50:05.353 DEBUG:boto:Token: None 2024-02-01T10:50:05.353 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:05 GMT /vclfcn-63/obj5?versionId=null 2024-02-01T10:50:05.353 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QdGfz3zJDx8KdI3jbxImtIAdQfE= 2024-02-01T10:50:05.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QdGfz3zJDx8KdI3jbxImtIAdQfE=', 'Content-Length': '0'} 2024-02-01T10:50:05.356 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006bc36fe128c030d1-0065bb775d-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:05.400 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:05.400 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:05.400 DEBUG:boto:path=/vclfcn-63/?prefix=obj5 2024-02-01T10:50:05.401 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj5 2024-02-01T10:50:05.401 DEBUG:boto:Method: GET 2024-02-01T10:50:05.401 DEBUG:boto:Path: /vclfcn-63/?prefix=obj5 2024-02-01T10:50:05.401 DEBUG:boto:Data: 2024-02-01T10:50:05.401 DEBUG:boto:Headers: {} 2024-02-01T10:50:05.401 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:05.401 DEBUG:boto:Port: 8001 2024-02-01T10:50:05.401 DEBUG:boto:Params: {} 2024-02-01T10:50:05.401 DEBUG:boto:Token: None 2024-02-01T10:50:05.401 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:05 GMT /vclfcn-63/ 2024-02-01T10:50:05.402 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9WtEsClKkCwkqpqMUXOHtzjNa7g= 2024-02-01T10:50:05.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9WtEsClKkCwkqpqMUXOHtzjNa7g=', 'Content-Length': '0'} 2024-02-01T10:50:05.407 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000278781e2c4a31ed0-0065bb775d-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:05.407 DEBUG:boto:b'vclfcn-63obj51000falseobj52024-02-01T10:49:22.255Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:05.408 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:05.408 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:05.408 DEBUG:boto:path=/vclfcn-63/?prefix=obj5 2024-02-01T10:50:05.408 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj5 2024-02-01T10:50:05.408 DEBUG:boto:Method: GET 2024-02-01T10:50:05.408 DEBUG:boto:Path: /vclfcn-63/?prefix=obj5 2024-02-01T10:50:05.408 DEBUG:boto:Data: 2024-02-01T10:50:05.408 DEBUG:boto:Headers: {} 2024-02-01T10:50:05.408 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:05.408 DEBUG:boto:Port: 8000 2024-02-01T10:50:05.408 DEBUG:boto:Params: {} 2024-02-01T10:50:05.409 DEBUG:boto:Token: None 2024-02-01T10:50:05.409 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:05 GMT /vclfcn-63/ 2024-02-01T10:50:05.409 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9WtEsClKkCwkqpqMUXOHtzjNa7g= 2024-02-01T10:50:05.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9WtEsClKkCwkqpqMUXOHtzjNa7g=', 'Content-Length': '0'} 2024-02-01T10:50:05.414 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071a6059dcedf367d-0065bb775d-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:05.456 DEBUG:boto:b'vclfcn-63obj51000falseobj52024-02-01T10:49:22.255Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:05.456 DEBUG:boto:path=/vclfcn-63/obj5 2024-02-01T10:50:05.457 DEBUG:boto:auth_path=/vclfcn-63/obj5 2024-02-01T10:50:05.457 DEBUG:boto:Method: HEAD 2024-02-01T10:50:05.457 DEBUG:boto:Path: /vclfcn-63/obj5 2024-02-01T10:50:05.457 DEBUG:boto:Data: 2024-02-01T10:50:05.457 DEBUG:boto:Headers: {} 2024-02-01T10:50:05.457 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:05.457 DEBUG:boto:Port: 8001 2024-02-01T10:50:05.457 DEBUG:boto:Params: {} 2024-02-01T10:50:05.457 DEBUG:boto:Token: None 2024-02-01T10:50:05.458 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:05 GMT /vclfcn-63/obj5 2024-02-01T10:50:05.458 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:TExL9hFqm8xZjUYvpHsCDiwW5ec= 2024-02-01T10:50:05.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:TExL9hFqm8xZjUYvpHsCDiwW5ec=', 'Content-Length': '0'} 2024-02-01T10:50:05.460 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000033714be5589718e7-0065bb775d-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:05.461 DEBUG:boto:path=/vclfcn-63/obj5 2024-02-01T10:50:05.461 DEBUG:boto:auth_path=/vclfcn-63/obj5 2024-02-01T10:50:05.461 DEBUG:boto:Method: HEAD 2024-02-01T10:50:05.461 DEBUG:boto:Path: /vclfcn-63/obj5 2024-02-01T10:50:05.461 DEBUG:boto:Data: 2024-02-01T10:50:05.461 DEBUG:boto:Headers: {} 2024-02-01T10:50:05.461 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:05.461 DEBUG:boto:Port: 8000 2024-02-01T10:50:05.462 DEBUG:boto:Params: {} 2024-02-01T10:50:05.462 DEBUG:boto:Token: None 2024-02-01T10:50:05.462 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:05 GMT /vclfcn-63/obj5 2024-02-01T10:50:05.462 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:TExL9hFqm8xZjUYvpHsCDiwW5ec= 2024-02-01T10:50:05.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:TExL9hFqm8xZjUYvpHsCDiwW5ec=', 'Content-Length': '0'} 2024-02-01T10:50:05.465 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003a2d4bfa7ef9e4e1-0065bb775d-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:05.465 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-02-01T10:50:05.465 DEBUG:boto:path=/vclfcn-63/obj5 2024-02-01T10:50:05.465 DEBUG:boto:auth_path=/vclfcn-63/obj5 2024-02-01T10:50:05.465 DEBUG:boto:Method: GET 2024-02-01T10:50:05.465 DEBUG:boto:Path: /vclfcn-63/obj5 2024-02-01T10:50:05.465 DEBUG:boto:Data: 2024-02-01T10:50:05.465 DEBUG:boto:Headers: {} 2024-02-01T10:50:05.466 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:05.466 DEBUG:boto:Port: 8001 2024-02-01T10:50:05.466 DEBUG:boto:Params: {} 2024-02-01T10:50:05.466 DEBUG:boto:Token: None 2024-02-01T10:50:05.466 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:05 GMT /vclfcn-63/obj5 2024-02-01T10:50:05.466 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:6vBsBY2oSnFzEuwnPnEEq7hgiUU= 2024-02-01T10:50:05.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:6vBsBY2oSnFzEuwnPnEEq7hgiUU=', 'Content-Length': '0'} 2024-02-01T10:50:05.469 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000fd5abb5eaa21680-0065bb775d-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:05.512 DEBUG:boto:path=/vclfcn-63/obj5 2024-02-01T10:50:05.513 DEBUG:boto:auth_path=/vclfcn-63/obj5 2024-02-01T10:50:05.513 DEBUG:boto:Method: GET 2024-02-01T10:50:05.513 DEBUG:boto:Path: /vclfcn-63/obj5 2024-02-01T10:50:05.513 DEBUG:boto:Data: 2024-02-01T10:50:05.513 DEBUG:boto:Headers: {} 2024-02-01T10:50:05.513 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:05.513 DEBUG:boto:Port: 8000 2024-02-01T10:50:05.513 DEBUG:boto:Params: {} 2024-02-01T10:50:05.514 DEBUG:boto:Token: None 2024-02-01T10:50:05.514 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:05 GMT /vclfcn-63/obj5 2024-02-01T10:50:05.514 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:6vBsBY2oSnFzEuwnPnEEq7hgiUU= 2024-02-01T10:50:05.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:6vBsBY2oSnFzEuwnPnEEq7hgiUU=', 'Content-Length': '0'} 2024-02-01T10:50:05.517 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fcc82e03016cab4a-0065bb775d-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:05.564 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:05.565 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:05.565 DEBUG:boto:path=/vclfcn-63/?prefix=obj5 2024-02-01T10:50:05.565 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj5 2024-02-01T10:50:05.566 DEBUG:boto:Method: GET 2024-02-01T10:50:05.566 DEBUG:boto:Path: /vclfcn-63/?prefix=obj5 2024-02-01T10:50:05.566 DEBUG:boto:Data: 2024-02-01T10:50:05.566 DEBUG:boto:Headers: {} 2024-02-01T10:50:05.566 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:05.566 DEBUG:boto:Port: 8001 2024-02-01T10:50:05.566 DEBUG:boto:Params: {} 2024-02-01T10:50:05.566 DEBUG:boto:Token: None 2024-02-01T10:50:05.567 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:05 GMT /vclfcn-63/ 2024-02-01T10:50:05.567 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9WtEsClKkCwkqpqMUXOHtzjNa7g= 2024-02-01T10:50:05.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9WtEsClKkCwkqpqMUXOHtzjNa7g=', 'Content-Length': '0'} 2024-02-01T10:50:05.571 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007fdf1b1ca15f78ca-0065bb775d-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:05.571 DEBUG:boto:b'vclfcn-63obj51000falseobj52024-02-01T10:49:22.255Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:05.572 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:05.572 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:05.572 DEBUG:boto:path=/vclfcn-63/?prefix=obj5 2024-02-01T10:50:05.572 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj5 2024-02-01T10:50:05.572 DEBUG:boto:Method: GET 2024-02-01T10:50:05.572 DEBUG:boto:Path: /vclfcn-63/?prefix=obj5 2024-02-01T10:50:05.572 DEBUG:boto:Data: 2024-02-01T10:50:05.572 DEBUG:boto:Headers: {} 2024-02-01T10:50:05.572 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:05.573 DEBUG:boto:Port: 8000 2024-02-01T10:50:05.573 DEBUG:boto:Params: {} 2024-02-01T10:50:05.573 DEBUG:boto:Token: None 2024-02-01T10:50:05.573 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:05 GMT /vclfcn-63/ 2024-02-01T10:50:05.574 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9WtEsClKkCwkqpqMUXOHtzjNa7g= 2024-02-01T10:50:05.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9WtEsClKkCwkqpqMUXOHtzjNa7g=', 'Content-Length': '0'} 2024-02-01T10:50:05.578 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004926108f9f7b899b-0065bb775d-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:05.620 DEBUG:boto:b'vclfcn-63obj51000falseobj52024-02-01T10:49:22.255Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:05.621 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-02-01T10:50:05.621 DEBUG:boto:path=/vclfcn-63/obj6 2024-02-01T10:50:05.621 DEBUG:boto:auth_path=/vclfcn-63/obj6 2024-02-01T10:50:05.621 DEBUG:boto:path=/vclfcn-63/obj6?versionId=null 2024-02-01T10:50:05.621 DEBUG:boto:auth_path=/vclfcn-63/obj6?versionId=null 2024-02-01T10:50:05.621 DEBUG:boto:Method: GET 2024-02-01T10:50:05.621 DEBUG:boto:Path: /vclfcn-63/obj6?versionId=null 2024-02-01T10:50:05.621 DEBUG:boto:Data: 2024-02-01T10:50:05.622 DEBUG:boto:Headers: {} 2024-02-01T10:50:05.622 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:05.622 DEBUG:boto:Port: 8001 2024-02-01T10:50:05.622 DEBUG:boto:Params: {} 2024-02-01T10:50:05.622 DEBUG:boto:Token: None 2024-02-01T10:50:05.622 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:05 GMT /vclfcn-63/obj6?versionId=null 2024-02-01T10:50:05.622 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GTfF29WQ4HIFjxMyKJoypf0n2xQ= 2024-02-01T10:50:05.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GTfF29WQ4HIFjxMyKJoypf0n2xQ=', 'Content-Length': '0'} 2024-02-01T10:50:05.625 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003050ac593f84533a-0065bb775d-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:05.668 DEBUG:boto:path=/vclfcn-63/obj6 2024-02-01T10:50:05.668 DEBUG:boto:auth_path=/vclfcn-63/obj6 2024-02-01T10:50:05.668 DEBUG:boto:path=/vclfcn-63/obj6?versionId=null 2024-02-01T10:50:05.668 DEBUG:boto:auth_path=/vclfcn-63/obj6?versionId=null 2024-02-01T10:50:05.669 DEBUG:boto:Method: GET 2024-02-01T10:50:05.669 DEBUG:boto:Path: /vclfcn-63/obj6?versionId=null 2024-02-01T10:50:05.669 DEBUG:boto:Data: 2024-02-01T10:50:05.669 DEBUG:boto:Headers: {} 2024-02-01T10:50:05.669 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:05.669 DEBUG:boto:Port: 8000 2024-02-01T10:50:05.669 DEBUG:boto:Params: {} 2024-02-01T10:50:05.669 DEBUG:boto:Token: None 2024-02-01T10:50:05.669 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:05 GMT /vclfcn-63/obj6?versionId=null 2024-02-01T10:50:05.670 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GTfF29WQ4HIFjxMyKJoypf0n2xQ= 2024-02-01T10:50:05.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GTfF29WQ4HIFjxMyKJoypf0n2xQ=', 'Content-Length': '0'} 2024-02-01T10:50:05.673 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000de268b95f0958fb7-0065bb775d-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:05.716 DEBUG:boto:path=/vclfcn-63/obj6 2024-02-01T10:50:05.716 DEBUG:boto:auth_path=/vclfcn-63/obj6 2024-02-01T10:50:05.716 DEBUG:boto:path=/vclfcn-63/obj6?versionId=null 2024-02-01T10:50:05.717 DEBUG:boto:auth_path=/vclfcn-63/obj6?versionId=null 2024-02-01T10:50:05.717 DEBUG:boto:Method: HEAD 2024-02-01T10:50:05.717 DEBUG:boto:Path: /vclfcn-63/obj6?versionId=null 2024-02-01T10:50:05.717 DEBUG:boto:Data: 2024-02-01T10:50:05.717 DEBUG:boto:Headers: {} 2024-02-01T10:50:05.717 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:05.717 DEBUG:boto:Port: 8001 2024-02-01T10:50:05.717 DEBUG:boto:Params: {} 2024-02-01T10:50:05.717 DEBUG:boto:Token: None 2024-02-01T10:50:05.717 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:05 GMT /vclfcn-63/obj6?versionId=null 2024-02-01T10:50:05.718 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zrIdb0oa1KyiTzGDM15owto7xus= 2024-02-01T10:50:05.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zrIdb0oa1KyiTzGDM15owto7xus=', 'Content-Length': '0'} 2024-02-01T10:50:05.721 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000da94fba2ebd896d0-0065bb775d-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:05.721 DEBUG:boto:path=/vclfcn-63/obj6 2024-02-01T10:50:05.721 DEBUG:boto:auth_path=/vclfcn-63/obj6 2024-02-01T10:50:05.721 DEBUG:boto:path=/vclfcn-63/obj6?versionId=null 2024-02-01T10:50:05.721 DEBUG:boto:auth_path=/vclfcn-63/obj6?versionId=null 2024-02-01T10:50:05.721 DEBUG:boto:Method: HEAD 2024-02-01T10:50:05.721 DEBUG:boto:Path: /vclfcn-63/obj6?versionId=null 2024-02-01T10:50:05.721 DEBUG:boto:Data: 2024-02-01T10:50:05.722 DEBUG:boto:Headers: {} 2024-02-01T10:50:05.722 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:05.722 DEBUG:boto:Port: 8000 2024-02-01T10:50:05.722 DEBUG:boto:Params: {} 2024-02-01T10:50:05.722 DEBUG:boto:Token: None 2024-02-01T10:50:05.722 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:05 GMT /vclfcn-63/obj6?versionId=null 2024-02-01T10:50:05.722 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zrIdb0oa1KyiTzGDM15owto7xus= 2024-02-01T10:50:05.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zrIdb0oa1KyiTzGDM15owto7xus=', 'Content-Length': '0'} 2024-02-01T10:50:05.725 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004086280d4d478389-0065bb775d-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:05.725 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-02-01T10:50:05.725 DEBUG:boto:path=/vclfcn-63/obj6 2024-02-01T10:50:05.725 DEBUG:boto:auth_path=/vclfcn-63/obj6 2024-02-01T10:50:05.725 DEBUG:boto:path=/vclfcn-63/obj6?versionId=null 2024-02-01T10:50:05.726 DEBUG:boto:auth_path=/vclfcn-63/obj6?versionId=null 2024-02-01T10:50:05.726 DEBUG:boto:Method: GET 2024-02-01T10:50:05.726 DEBUG:boto:Path: /vclfcn-63/obj6?versionId=null 2024-02-01T10:50:05.726 DEBUG:boto:Data: 2024-02-01T10:50:05.726 DEBUG:boto:Headers: {} 2024-02-01T10:50:05.726 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:05.726 DEBUG:boto:Port: 8001 2024-02-01T10:50:05.726 DEBUG:boto:Params: {} 2024-02-01T10:50:05.726 DEBUG:boto:Token: None 2024-02-01T10:50:05.727 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:05 GMT /vclfcn-63/obj6?versionId=null 2024-02-01T10:50:05.727 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GTfF29WQ4HIFjxMyKJoypf0n2xQ= 2024-02-01T10:50:05.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GTfF29WQ4HIFjxMyKJoypf0n2xQ=', 'Content-Length': '0'} 2024-02-01T10:50:05.730 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001be98903929f0570-0065bb775d-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:05.772 DEBUG:boto:path=/vclfcn-63/obj6 2024-02-01T10:50:05.772 DEBUG:boto:auth_path=/vclfcn-63/obj6 2024-02-01T10:50:05.772 DEBUG:boto:path=/vclfcn-63/obj6?versionId=null 2024-02-01T10:50:05.773 DEBUG:boto:auth_path=/vclfcn-63/obj6?versionId=null 2024-02-01T10:50:05.773 DEBUG:boto:Method: GET 2024-02-01T10:50:05.773 DEBUG:boto:Path: /vclfcn-63/obj6?versionId=null 2024-02-01T10:50:05.773 DEBUG:boto:Data: 2024-02-01T10:50:05.773 DEBUG:boto:Headers: {} 2024-02-01T10:50:05.773 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:05.773 DEBUG:boto:Port: 8000 2024-02-01T10:50:05.773 DEBUG:boto:Params: {} 2024-02-01T10:50:05.773 DEBUG:boto:Token: None 2024-02-01T10:50:05.774 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:05 GMT /vclfcn-63/obj6?versionId=null 2024-02-01T10:50:05.774 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GTfF29WQ4HIFjxMyKJoypf0n2xQ= 2024-02-01T10:50:05.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GTfF29WQ4HIFjxMyKJoypf0n2xQ=', 'Content-Length': '0'} 2024-02-01T10:50:05.777 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d3308e74e4f275ef-0065bb775d-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:05.821 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:05.821 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:05.822 DEBUG:boto:path=/vclfcn-63/?prefix=obj6 2024-02-01T10:50:05.822 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj6 2024-02-01T10:50:05.822 DEBUG:boto:Method: GET 2024-02-01T10:50:05.822 DEBUG:boto:Path: /vclfcn-63/?prefix=obj6 2024-02-01T10:50:05.822 DEBUG:boto:Data: 2024-02-01T10:50:05.822 DEBUG:boto:Headers: {} 2024-02-01T10:50:05.822 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:05.822 DEBUG:boto:Port: 8001 2024-02-01T10:50:05.822 DEBUG:boto:Params: {} 2024-02-01T10:50:05.822 DEBUG:boto:Token: None 2024-02-01T10:50:05.823 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:05 GMT /vclfcn-63/ 2024-02-01T10:50:05.823 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9WtEsClKkCwkqpqMUXOHtzjNa7g= 2024-02-01T10:50:05.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9WtEsClKkCwkqpqMUXOHtzjNa7g=', 'Content-Length': '0'} 2024-02-01T10:50:05.827 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9b55499b8b8b251-0065bb775d-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:05.828 DEBUG:boto:b'vclfcn-63obj61000falseobj62024-02-01T10:49:22.271Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:05.828 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:05.828 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:05.828 DEBUG:boto:path=/vclfcn-63/?prefix=obj6 2024-02-01T10:50:05.828 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj6 2024-02-01T10:50:05.829 DEBUG:boto:Method: GET 2024-02-01T10:50:05.829 DEBUG:boto:Path: /vclfcn-63/?prefix=obj6 2024-02-01T10:50:05.829 DEBUG:boto:Data: 2024-02-01T10:50:05.829 DEBUG:boto:Headers: {} 2024-02-01T10:50:05.829 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:05.829 DEBUG:boto:Port: 8000 2024-02-01T10:50:05.829 DEBUG:boto:Params: {} 2024-02-01T10:50:05.829 DEBUG:boto:Token: None 2024-02-01T10:50:05.829 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:05 GMT /vclfcn-63/ 2024-02-01T10:50:05.829 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9WtEsClKkCwkqpqMUXOHtzjNa7g= 2024-02-01T10:50:05.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9WtEsClKkCwkqpqMUXOHtzjNa7g=', 'Content-Length': '0'} 2024-02-01T10:50:05.834 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007dfed6bfd3901cfb-0065bb775d-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:05.876 DEBUG:boto:b'vclfcn-63obj61000falseobj62024-02-01T10:49:22.271Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:05.877 DEBUG:boto:path=/vclfcn-63/obj6 2024-02-01T10:50:05.877 DEBUG:boto:auth_path=/vclfcn-63/obj6 2024-02-01T10:50:05.877 DEBUG:boto:Method: HEAD 2024-02-01T10:50:05.877 DEBUG:boto:Path: /vclfcn-63/obj6 2024-02-01T10:50:05.877 DEBUG:boto:Data: 2024-02-01T10:50:05.877 DEBUG:boto:Headers: {} 2024-02-01T10:50:05.877 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:05.877 DEBUG:boto:Port: 8001 2024-02-01T10:50:05.877 DEBUG:boto:Params: {} 2024-02-01T10:50:05.878 DEBUG:boto:Token: None 2024-02-01T10:50:05.878 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:05 GMT /vclfcn-63/obj6 2024-02-01T10:50:05.878 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:yhZ0RWdoZXSx9zPeb3S9tJU5uzM= 2024-02-01T10:50:05.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:yhZ0RWdoZXSx9zPeb3S9tJU5uzM=', 'Content-Length': '0'} 2024-02-01T10:50:05.881 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f37d894a1f8e5948-0065bb775d-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:05.881 DEBUG:boto:path=/vclfcn-63/obj6 2024-02-01T10:50:05.881 DEBUG:boto:auth_path=/vclfcn-63/obj6 2024-02-01T10:50:05.882 DEBUG:boto:Method: HEAD 2024-02-01T10:50:05.882 DEBUG:boto:Path: /vclfcn-63/obj6 2024-02-01T10:50:05.882 DEBUG:boto:Data: 2024-02-01T10:50:05.882 DEBUG:boto:Headers: {} 2024-02-01T10:50:05.882 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:05.882 DEBUG:boto:Port: 8000 2024-02-01T10:50:05.882 DEBUG:boto:Params: {} 2024-02-01T10:50:05.882 DEBUG:boto:Token: None 2024-02-01T10:50:05.882 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:05 GMT /vclfcn-63/obj6 2024-02-01T10:50:05.883 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:yhZ0RWdoZXSx9zPeb3S9tJU5uzM= 2024-02-01T10:50:05.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:yhZ0RWdoZXSx9zPeb3S9tJU5uzM=', 'Content-Length': '0'} 2024-02-01T10:50:05.885 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bd4ee39e974af855-0065bb775d-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:05.886 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-02-01T10:50:05.886 DEBUG:boto:path=/vclfcn-63/obj6 2024-02-01T10:50:05.886 DEBUG:boto:auth_path=/vclfcn-63/obj6 2024-02-01T10:50:05.886 DEBUG:boto:Method: GET 2024-02-01T10:50:05.886 DEBUG:boto:Path: /vclfcn-63/obj6 2024-02-01T10:50:05.886 DEBUG:boto:Data: 2024-02-01T10:50:05.886 DEBUG:boto:Headers: {} 2024-02-01T10:50:05.886 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:05.886 DEBUG:boto:Port: 8001 2024-02-01T10:50:05.886 DEBUG:boto:Params: {} 2024-02-01T10:50:05.886 DEBUG:boto:Token: None 2024-02-01T10:50:05.887 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:05 GMT /vclfcn-63/obj6 2024-02-01T10:50:05.887 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Z2HxdbieiNYhjc47m1xQi49OILQ= 2024-02-01T10:50:05.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Z2HxdbieiNYhjc47m1xQi49OILQ=', 'Content-Length': '0'} 2024-02-01T10:50:05.889 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000994310571ec1e88b-0065bb775d-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:05.932 DEBUG:boto:path=/vclfcn-63/obj6 2024-02-01T10:50:05.932 DEBUG:boto:auth_path=/vclfcn-63/obj6 2024-02-01T10:50:05.933 DEBUG:boto:Method: GET 2024-02-01T10:50:05.933 DEBUG:boto:Path: /vclfcn-63/obj6 2024-02-01T10:50:05.933 DEBUG:boto:Data: 2024-02-01T10:50:05.933 DEBUG:boto:Headers: {} 2024-02-01T10:50:05.933 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:05.933 DEBUG:boto:Port: 8000 2024-02-01T10:50:05.933 DEBUG:boto:Params: {} 2024-02-01T10:50:05.933 DEBUG:boto:Token: None 2024-02-01T10:50:05.934 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:05 GMT /vclfcn-63/obj6 2024-02-01T10:50:05.934 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Z2HxdbieiNYhjc47m1xQi49OILQ= 2024-02-01T10:50:05.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Z2HxdbieiNYhjc47m1xQi49OILQ=', 'Content-Length': '0'} 2024-02-01T10:50:05.937 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d8a5a5c463b8e703-0065bb775d-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:05.980 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:05.980 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:05.980 DEBUG:boto:path=/vclfcn-63/?prefix=obj6 2024-02-01T10:50:05.980 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj6 2024-02-01T10:50:05.981 DEBUG:boto:Method: GET 2024-02-01T10:50:05.981 DEBUG:boto:Path: /vclfcn-63/?prefix=obj6 2024-02-01T10:50:05.981 DEBUG:boto:Data: 2024-02-01T10:50:05.981 DEBUG:boto:Headers: {} 2024-02-01T10:50:05.981 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:05.981 DEBUG:boto:Port: 8001 2024-02-01T10:50:05.981 DEBUG:boto:Params: {} 2024-02-01T10:50:05.981 DEBUG:boto:Token: None 2024-02-01T10:50:05.981 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:05 GMT /vclfcn-63/ 2024-02-01T10:50:05.982 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9WtEsClKkCwkqpqMUXOHtzjNa7g= 2024-02-01T10:50:05.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9WtEsClKkCwkqpqMUXOHtzjNa7g=', 'Content-Length': '0'} 2024-02-01T10:50:05.986 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000942a1b06e002c88b-0065bb775d-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:05.987 DEBUG:boto:b'vclfcn-63obj61000falseobj62024-02-01T10:49:22.271Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:05.987 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:05.987 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:05.987 DEBUG:boto:path=/vclfcn-63/?prefix=obj6 2024-02-01T10:50:05.987 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj6 2024-02-01T10:50:05.987 DEBUG:boto:Method: GET 2024-02-01T10:50:05.988 DEBUG:boto:Path: /vclfcn-63/?prefix=obj6 2024-02-01T10:50:05.988 DEBUG:boto:Data: 2024-02-01T10:50:05.988 DEBUG:boto:Headers: {} 2024-02-01T10:50:05.988 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:05.988 DEBUG:boto:Port: 8000 2024-02-01T10:50:05.988 DEBUG:boto:Params: {} 2024-02-01T10:50:05.988 DEBUG:boto:Token: None 2024-02-01T10:50:05.988 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:05 GMT /vclfcn-63/ 2024-02-01T10:50:05.988 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9WtEsClKkCwkqpqMUXOHtzjNa7g= 2024-02-01T10:50:05.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:05 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9WtEsClKkCwkqpqMUXOHtzjNa7g=', 'Content-Length': '0'} 2024-02-01T10:50:05.993 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d909abebafc8e555-0065bb775d-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:05 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:06.036 DEBUG:boto:b'vclfcn-63obj61000falseobj62024-02-01T10:49:22.271Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:06.036 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-02-01T10:50:06.037 DEBUG:boto:path=/vclfcn-63/obj7 2024-02-01T10:50:06.037 DEBUG:boto:auth_path=/vclfcn-63/obj7 2024-02-01T10:50:06.037 DEBUG:boto:path=/vclfcn-63/obj7?versionId=null 2024-02-01T10:50:06.037 DEBUG:boto:auth_path=/vclfcn-63/obj7?versionId=null 2024-02-01T10:50:06.037 DEBUG:boto:Method: GET 2024-02-01T10:50:06.037 DEBUG:boto:Path: /vclfcn-63/obj7?versionId=null 2024-02-01T10:50:06.037 DEBUG:boto:Data: 2024-02-01T10:50:06.037 DEBUG:boto:Headers: {} 2024-02-01T10:50:06.038 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:06.038 DEBUG:boto:Port: 8001 2024-02-01T10:50:06.038 DEBUG:boto:Params: {} 2024-02-01T10:50:06.038 DEBUG:boto:Token: None 2024-02-01T10:50:06.038 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:06 GMT /vclfcn-63/obj7?versionId=null 2024-02-01T10:50:06.038 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:dZGBYwh3+H4U4Nx7oFx2BM+I1VQ= 2024-02-01T10:50:06.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dZGBYwh3+H4U4Nx7oFx2BM+I1VQ=', 'Content-Length': '0'} 2024-02-01T10:50:06.041 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a71e7826e64d68e2-0065bb775e-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:06.041 DEBUG:boto:path=/vclfcn-63/obj7 2024-02-01T10:50:06.042 DEBUG:boto:auth_path=/vclfcn-63/obj7 2024-02-01T10:50:06.042 DEBUG:boto:path=/vclfcn-63/obj7?versionId=null 2024-02-01T10:50:06.042 DEBUG:boto:auth_path=/vclfcn-63/obj7?versionId=null 2024-02-01T10:50:06.042 DEBUG:boto:Method: GET 2024-02-01T10:50:06.042 DEBUG:boto:Path: /vclfcn-63/obj7?versionId=null 2024-02-01T10:50:06.042 DEBUG:boto:Data: 2024-02-01T10:50:06.042 DEBUG:boto:Headers: {} 2024-02-01T10:50:06.042 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:06.042 DEBUG:boto:Port: 8000 2024-02-01T10:50:06.042 DEBUG:boto:Params: {} 2024-02-01T10:50:06.043 DEBUG:boto:Token: None 2024-02-01T10:50:06.043 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:06 GMT /vclfcn-63/obj7?versionId=null 2024-02-01T10:50:06.043 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:dZGBYwh3+H4U4Nx7oFx2BM+I1VQ= 2024-02-01T10:50:06.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dZGBYwh3+H4U4Nx7oFx2BM+I1VQ=', 'Content-Length': '0'} 2024-02-01T10:50:06.045 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b0f75dbdbc7c15a2-0065bb775e-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:06.088 DEBUG:boto:path=/vclfcn-63/obj7 2024-02-01T10:50:06.088 DEBUG:boto:auth_path=/vclfcn-63/obj7 2024-02-01T10:50:06.088 DEBUG:boto:path=/vclfcn-63/obj7?versionId=null 2024-02-01T10:50:06.088 DEBUG:boto:auth_path=/vclfcn-63/obj7?versionId=null 2024-02-01T10:50:06.089 DEBUG:boto:Method: HEAD 2024-02-01T10:50:06.089 DEBUG:boto:Path: /vclfcn-63/obj7?versionId=null 2024-02-01T10:50:06.089 DEBUG:boto:Data: 2024-02-01T10:50:06.089 DEBUG:boto:Headers: {} 2024-02-01T10:50:06.089 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:06.089 DEBUG:boto:Port: 8001 2024-02-01T10:50:06.089 DEBUG:boto:Params: {} 2024-02-01T10:50:06.089 DEBUG:boto:Token: None 2024-02-01T10:50:06.089 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:06 GMT /vclfcn-63/obj7?versionId=null 2024-02-01T10:50:06.090 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:XB8t8Et03iX1ZxwK+5WJy+HVKwQ= 2024-02-01T10:50:06.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:XB8t8Et03iX1ZxwK+5WJy+HVKwQ=', 'Content-Length': '0'} 2024-02-01T10:50:06.092 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000428ca312b41ae5b4-0065bb775e-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:06.093 DEBUG:boto:path=/vclfcn-63/obj7 2024-02-01T10:50:06.093 DEBUG:boto:auth_path=/vclfcn-63/obj7 2024-02-01T10:50:06.093 DEBUG:boto:path=/vclfcn-63/obj7?versionId=null 2024-02-01T10:50:06.093 DEBUG:boto:auth_path=/vclfcn-63/obj7?versionId=null 2024-02-01T10:50:06.093 DEBUG:boto:Method: HEAD 2024-02-01T10:50:06.093 DEBUG:boto:Path: /vclfcn-63/obj7?versionId=null 2024-02-01T10:50:06.093 DEBUG:boto:Data: 2024-02-01T10:50:06.093 DEBUG:boto:Headers: {} 2024-02-01T10:50:06.093 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:06.094 DEBUG:boto:Port: 8000 2024-02-01T10:50:06.094 DEBUG:boto:Params: {} 2024-02-01T10:50:06.094 DEBUG:boto:Token: None 2024-02-01T10:50:06.094 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:06 GMT /vclfcn-63/obj7?versionId=null 2024-02-01T10:50:06.094 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:XB8t8Et03iX1ZxwK+5WJy+HVKwQ= 2024-02-01T10:50:06.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:XB8t8Et03iX1ZxwK+5WJy+HVKwQ=', 'Content-Length': '0'} 2024-02-01T10:50:06.097 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d700e21d99f9e20d-0065bb775e-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:06.097 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-02-01T10:50:06.097 DEBUG:boto:path=/vclfcn-63/obj7 2024-02-01T10:50:06.097 DEBUG:boto:auth_path=/vclfcn-63/obj7 2024-02-01T10:50:06.097 DEBUG:boto:path=/vclfcn-63/obj7?versionId=null 2024-02-01T10:50:06.097 DEBUG:boto:auth_path=/vclfcn-63/obj7?versionId=null 2024-02-01T10:50:06.097 DEBUG:boto:Method: GET 2024-02-01T10:50:06.098 DEBUG:boto:Path: /vclfcn-63/obj7?versionId=null 2024-02-01T10:50:06.098 DEBUG:boto:Data: 2024-02-01T10:50:06.098 DEBUG:boto:Headers: {} 2024-02-01T10:50:06.098 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:06.098 DEBUG:boto:Port: 8001 2024-02-01T10:50:06.098 DEBUG:boto:Params: {} 2024-02-01T10:50:06.098 DEBUG:boto:Token: None 2024-02-01T10:50:06.098 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:06 GMT /vclfcn-63/obj7?versionId=null 2024-02-01T10:50:06.098 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:dZGBYwh3+H4U4Nx7oFx2BM+I1VQ= 2024-02-01T10:50:06.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dZGBYwh3+H4U4Nx7oFx2BM+I1VQ=', 'Content-Length': '0'} 2024-02-01T10:50:06.101 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a15d8b19033ab2ef-0065bb775e-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:06.144 DEBUG:boto:path=/vclfcn-63/obj7 2024-02-01T10:50:06.144 DEBUG:boto:auth_path=/vclfcn-63/obj7 2024-02-01T10:50:06.144 DEBUG:boto:path=/vclfcn-63/obj7?versionId=null 2024-02-01T10:50:06.144 DEBUG:boto:auth_path=/vclfcn-63/obj7?versionId=null 2024-02-01T10:50:06.145 DEBUG:boto:Method: GET 2024-02-01T10:50:06.145 DEBUG:boto:Path: /vclfcn-63/obj7?versionId=null 2024-02-01T10:50:06.145 DEBUG:boto:Data: 2024-02-01T10:50:06.145 DEBUG:boto:Headers: {} 2024-02-01T10:50:06.145 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:06.145 DEBUG:boto:Port: 8000 2024-02-01T10:50:06.145 DEBUG:boto:Params: {} 2024-02-01T10:50:06.145 DEBUG:boto:Token: None 2024-02-01T10:50:06.145 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:06 GMT /vclfcn-63/obj7?versionId=null 2024-02-01T10:50:06.146 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:dZGBYwh3+H4U4Nx7oFx2BM+I1VQ= 2024-02-01T10:50:06.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dZGBYwh3+H4U4Nx7oFx2BM+I1VQ=', 'Content-Length': '0'} 2024-02-01T10:50:06.148 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fc918200f799ec7e-0065bb775e-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:06.196 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:06.196 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:06.196 DEBUG:boto:path=/vclfcn-63/?prefix=obj7 2024-02-01T10:50:06.197 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj7 2024-02-01T10:50:06.197 DEBUG:boto:Method: GET 2024-02-01T10:50:06.197 DEBUG:boto:Path: /vclfcn-63/?prefix=obj7 2024-02-01T10:50:06.197 DEBUG:boto:Data: 2024-02-01T10:50:06.197 DEBUG:boto:Headers: {} 2024-02-01T10:50:06.197 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:06.197 DEBUG:boto:Port: 8001 2024-02-01T10:50:06.197 DEBUG:boto:Params: {} 2024-02-01T10:50:06.197 DEBUG:boto:Token: None 2024-02-01T10:50:06.198 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:06 GMT /vclfcn-63/ 2024-02-01T10:50:06.198 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:hPSiqnFPiqiYoo/UABhcN/FSAnU= 2024-02-01T10:50:06.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:hPSiqnFPiqiYoo/UABhcN/FSAnU=', 'Content-Length': '0'} 2024-02-01T10:50:06.202 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000044429dc451fe3344-0065bb775e-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:06.203 DEBUG:boto:b'vclfcn-63obj71000falseobj72024-02-01T10:49:22.327Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:06.203 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:06.203 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:06.203 DEBUG:boto:path=/vclfcn-63/?prefix=obj7 2024-02-01T10:50:06.204 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj7 2024-02-01T10:50:06.204 DEBUG:boto:Method: GET 2024-02-01T10:50:06.204 DEBUG:boto:Path: /vclfcn-63/?prefix=obj7 2024-02-01T10:50:06.204 DEBUG:boto:Data: 2024-02-01T10:50:06.204 DEBUG:boto:Headers: {} 2024-02-01T10:50:06.204 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:06.204 DEBUG:boto:Port: 8000 2024-02-01T10:50:06.204 DEBUG:boto:Params: {} 2024-02-01T10:50:06.204 DEBUG:boto:Token: None 2024-02-01T10:50:06.205 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:06 GMT /vclfcn-63/ 2024-02-01T10:50:06.205 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:hPSiqnFPiqiYoo/UABhcN/FSAnU= 2024-02-01T10:50:06.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:hPSiqnFPiqiYoo/UABhcN/FSAnU=', 'Content-Length': '0'} 2024-02-01T10:50:06.210 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002144651d9d83f872-0065bb775e-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:06.256 DEBUG:boto:b'vclfcn-63obj71000falseobj72024-02-01T10:49:22.327Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:06.257 DEBUG:boto:path=/vclfcn-63/obj7 2024-02-01T10:50:06.257 DEBUG:boto:auth_path=/vclfcn-63/obj7 2024-02-01T10:50:06.257 DEBUG:boto:Method: HEAD 2024-02-01T10:50:06.257 DEBUG:boto:Path: /vclfcn-63/obj7 2024-02-01T10:50:06.257 DEBUG:boto:Data: 2024-02-01T10:50:06.257 DEBUG:boto:Headers: {} 2024-02-01T10:50:06.257 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:06.257 DEBUG:boto:Port: 8001 2024-02-01T10:50:06.257 DEBUG:boto:Params: {} 2024-02-01T10:50:06.258 DEBUG:boto:Token: None 2024-02-01T10:50:06.258 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:06 GMT /vclfcn-63/obj7 2024-02-01T10:50:06.258 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:G49fbiUl2ff6Dz1t/PPle0lfiFE= 2024-02-01T10:50:06.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:G49fbiUl2ff6Dz1t/PPle0lfiFE=', 'Content-Length': '0'} 2024-02-01T10:50:06.261 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f64d89c4dfed70b9-0065bb775e-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:06.261 DEBUG:boto:path=/vclfcn-63/obj7 2024-02-01T10:50:06.261 DEBUG:boto:auth_path=/vclfcn-63/obj7 2024-02-01T10:50:06.262 DEBUG:boto:Method: HEAD 2024-02-01T10:50:06.262 DEBUG:boto:Path: /vclfcn-63/obj7 2024-02-01T10:50:06.262 DEBUG:boto:Data: 2024-02-01T10:50:06.262 DEBUG:boto:Headers: {} 2024-02-01T10:50:06.262 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:06.262 DEBUG:boto:Port: 8000 2024-02-01T10:50:06.262 DEBUG:boto:Params: {} 2024-02-01T10:50:06.262 DEBUG:boto:Token: None 2024-02-01T10:50:06.262 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:06 GMT /vclfcn-63/obj7 2024-02-01T10:50:06.263 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:G49fbiUl2ff6Dz1t/PPle0lfiFE= 2024-02-01T10:50:06.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:G49fbiUl2ff6Dz1t/PPle0lfiFE=', 'Content-Length': '0'} 2024-02-01T10:50:06.265 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000000b1ae389f838831-0065bb775e-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:06.265 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-02-01T10:50:06.265 DEBUG:boto:path=/vclfcn-63/obj7 2024-02-01T10:50:06.266 DEBUG:boto:auth_path=/vclfcn-63/obj7 2024-02-01T10:50:06.266 DEBUG:boto:Method: GET 2024-02-01T10:50:06.266 DEBUG:boto:Path: /vclfcn-63/obj7 2024-02-01T10:50:06.266 DEBUG:boto:Data: 2024-02-01T10:50:06.266 DEBUG:boto:Headers: {} 2024-02-01T10:50:06.266 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:06.266 DEBUG:boto:Port: 8001 2024-02-01T10:50:06.266 DEBUG:boto:Params: {} 2024-02-01T10:50:06.266 DEBUG:boto:Token: None 2024-02-01T10:50:06.266 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:06 GMT /vclfcn-63/obj7 2024-02-01T10:50:06.267 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:UT8wyM7rwwAhaVn7PPLDKEiaVd0= 2024-02-01T10:50:06.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:UT8wyM7rwwAhaVn7PPLDKEiaVd0=', 'Content-Length': '0'} 2024-02-01T10:50:06.270 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bc4bbdc03001297a-0065bb775e-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:06.312 DEBUG:boto:path=/vclfcn-63/obj7 2024-02-01T10:50:06.312 DEBUG:boto:auth_path=/vclfcn-63/obj7 2024-02-01T10:50:06.312 DEBUG:boto:Method: GET 2024-02-01T10:50:06.313 DEBUG:boto:Path: /vclfcn-63/obj7 2024-02-01T10:50:06.313 DEBUG:boto:Data: 2024-02-01T10:50:06.313 DEBUG:boto:Headers: {} 2024-02-01T10:50:06.313 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:06.313 DEBUG:boto:Port: 8000 2024-02-01T10:50:06.313 DEBUG:boto:Params: {} 2024-02-01T10:50:06.313 DEBUG:boto:Token: None 2024-02-01T10:50:06.313 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:06 GMT /vclfcn-63/obj7 2024-02-01T10:50:06.314 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:UT8wyM7rwwAhaVn7PPLDKEiaVd0= 2024-02-01T10:50:06.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:UT8wyM7rwwAhaVn7PPLDKEiaVd0=', 'Content-Length': '0'} 2024-02-01T10:50:06.317 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000058d6ee05a9ef4175-0065bb775e-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:06.360 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:06.360 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:06.360 DEBUG:boto:path=/vclfcn-63/?prefix=obj7 2024-02-01T10:50:06.360 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj7 2024-02-01T10:50:06.361 DEBUG:boto:Method: GET 2024-02-01T10:50:06.361 DEBUG:boto:Path: /vclfcn-63/?prefix=obj7 2024-02-01T10:50:06.361 DEBUG:boto:Data: 2024-02-01T10:50:06.361 DEBUG:boto:Headers: {} 2024-02-01T10:50:06.361 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:06.362 DEBUG:boto:Port: 8001 2024-02-01T10:50:06.362 DEBUG:boto:Params: {} 2024-02-01T10:50:06.362 DEBUG:boto:Token: None 2024-02-01T10:50:06.362 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:06 GMT /vclfcn-63/ 2024-02-01T10:50:06.362 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:hPSiqnFPiqiYoo/UABhcN/FSAnU= 2024-02-01T10:50:06.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:hPSiqnFPiqiYoo/UABhcN/FSAnU=', 'Content-Length': '0'} 2024-02-01T10:50:06.367 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009bbd026f3ce42e05-0065bb775e-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:06.408 DEBUG:boto:b'vclfcn-63obj71000falseobj72024-02-01T10:49:22.327Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:06.409 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:06.409 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:06.409 DEBUG:boto:path=/vclfcn-63/?prefix=obj7 2024-02-01T10:50:06.409 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj7 2024-02-01T10:50:06.409 DEBUG:boto:Method: GET 2024-02-01T10:50:06.409 DEBUG:boto:Path: /vclfcn-63/?prefix=obj7 2024-02-01T10:50:06.409 DEBUG:boto:Data: 2024-02-01T10:50:06.409 DEBUG:boto:Headers: {} 2024-02-01T10:50:06.409 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:06.409 DEBUG:boto:Port: 8000 2024-02-01T10:50:06.409 DEBUG:boto:Params: {} 2024-02-01T10:50:06.410 DEBUG:boto:Token: None 2024-02-01T10:50:06.410 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:06 GMT /vclfcn-63/ 2024-02-01T10:50:06.410 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:hPSiqnFPiqiYoo/UABhcN/FSAnU= 2024-02-01T10:50:06.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:hPSiqnFPiqiYoo/UABhcN/FSAnU=', 'Content-Length': '0'} 2024-02-01T10:50:06.414 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c543a76ca46374b-0065bb775e-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:06.414 DEBUG:boto:b'vclfcn-63obj71000falseobj72024-02-01T10:49:22.327Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:06.415 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-02-01T10:50:06.415 DEBUG:boto:path=/vclfcn-63/obj8 2024-02-01T10:50:06.415 DEBUG:boto:auth_path=/vclfcn-63/obj8 2024-02-01T10:50:06.415 DEBUG:boto:path=/vclfcn-63/obj8?versionId=null 2024-02-01T10:50:06.415 DEBUG:boto:auth_path=/vclfcn-63/obj8?versionId=null 2024-02-01T10:50:06.415 DEBUG:boto:Method: GET 2024-02-01T10:50:06.415 DEBUG:boto:Path: /vclfcn-63/obj8?versionId=null 2024-02-01T10:50:06.415 DEBUG:boto:Data: 2024-02-01T10:50:06.416 DEBUG:boto:Headers: {} 2024-02-01T10:50:06.416 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:06.416 DEBUG:boto:Port: 8001 2024-02-01T10:50:06.416 DEBUG:boto:Params: {} 2024-02-01T10:50:06.416 DEBUG:boto:Token: None 2024-02-01T10:50:06.416 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:06 GMT /vclfcn-63/obj8?versionId=null 2024-02-01T10:50:06.416 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:k5k7aVh9s8PJ/IO4Q5R3gM/dAGU= 2024-02-01T10:50:06.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:k5k7aVh9s8PJ/IO4Q5R3gM/dAGU=', 'Content-Length': '0'} 2024-02-01T10:50:06.419 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004d9b1e2fcf4aa013-0065bb775e-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:06.419 DEBUG:boto:path=/vclfcn-63/obj8 2024-02-01T10:50:06.419 DEBUG:boto:auth_path=/vclfcn-63/obj8 2024-02-01T10:50:06.420 DEBUG:boto:path=/vclfcn-63/obj8?versionId=null 2024-02-01T10:50:06.420 DEBUG:boto:auth_path=/vclfcn-63/obj8?versionId=null 2024-02-01T10:50:06.420 DEBUG:boto:Method: GET 2024-02-01T10:50:06.420 DEBUG:boto:Path: /vclfcn-63/obj8?versionId=null 2024-02-01T10:50:06.420 DEBUG:boto:Data: 2024-02-01T10:50:06.420 DEBUG:boto:Headers: {} 2024-02-01T10:50:06.420 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:06.420 DEBUG:boto:Port: 8000 2024-02-01T10:50:06.420 DEBUG:boto:Params: {} 2024-02-01T10:50:06.420 DEBUG:boto:Token: None 2024-02-01T10:50:06.421 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:06 GMT /vclfcn-63/obj8?versionId=null 2024-02-01T10:50:06.421 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:k5k7aVh9s8PJ/IO4Q5R3gM/dAGU= 2024-02-01T10:50:06.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:k5k7aVh9s8PJ/IO4Q5R3gM/dAGU=', 'Content-Length': '0'} 2024-02-01T10:50:06.423 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e35875bbfbd4f7ad-0065bb775e-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:06.468 DEBUG:boto:path=/vclfcn-63/obj8 2024-02-01T10:50:06.468 DEBUG:boto:auth_path=/vclfcn-63/obj8 2024-02-01T10:50:06.468 DEBUG:boto:path=/vclfcn-63/obj8?versionId=null 2024-02-01T10:50:06.469 DEBUG:boto:auth_path=/vclfcn-63/obj8?versionId=null 2024-02-01T10:50:06.469 DEBUG:boto:Method: HEAD 2024-02-01T10:50:06.469 DEBUG:boto:Path: /vclfcn-63/obj8?versionId=null 2024-02-01T10:50:06.469 DEBUG:boto:Data: 2024-02-01T10:50:06.469 DEBUG:boto:Headers: {} 2024-02-01T10:50:06.469 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:06.469 DEBUG:boto:Port: 8001 2024-02-01T10:50:06.469 DEBUG:boto:Params: {} 2024-02-01T10:50:06.469 DEBUG:boto:Token: None 2024-02-01T10:50:06.469 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:06 GMT /vclfcn-63/obj8?versionId=null 2024-02-01T10:50:06.470 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8EOVtrUn1x7BoMc3xDLXamBXP/0= 2024-02-01T10:50:06.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8EOVtrUn1x7BoMc3xDLXamBXP/0=', 'Content-Length': '0'} 2024-02-01T10:50:06.472 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f5af71454bb11cbc-0065bb775e-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:06.473 DEBUG:boto:path=/vclfcn-63/obj8 2024-02-01T10:50:06.473 DEBUG:boto:auth_path=/vclfcn-63/obj8 2024-02-01T10:50:06.473 DEBUG:boto:path=/vclfcn-63/obj8?versionId=null 2024-02-01T10:50:06.473 DEBUG:boto:auth_path=/vclfcn-63/obj8?versionId=null 2024-02-01T10:50:06.473 DEBUG:boto:Method: HEAD 2024-02-01T10:50:06.473 DEBUG:boto:Path: /vclfcn-63/obj8?versionId=null 2024-02-01T10:50:06.473 DEBUG:boto:Data: 2024-02-01T10:50:06.473 DEBUG:boto:Headers: {} 2024-02-01T10:50:06.473 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:06.474 DEBUG:boto:Port: 8000 2024-02-01T10:50:06.474 DEBUG:boto:Params: {} 2024-02-01T10:50:06.474 DEBUG:boto:Token: None 2024-02-01T10:50:06.474 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:06 GMT /vclfcn-63/obj8?versionId=null 2024-02-01T10:50:06.474 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8EOVtrUn1x7BoMc3xDLXamBXP/0= 2024-02-01T10:50:06.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8EOVtrUn1x7BoMc3xDLXamBXP/0=', 'Content-Length': '0'} 2024-02-01T10:50:06.477 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f52c6db31afe1c5c-0065bb775e-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:06.477 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-02-01T10:50:06.477 DEBUG:boto:path=/vclfcn-63/obj8 2024-02-01T10:50:06.477 DEBUG:boto:auth_path=/vclfcn-63/obj8 2024-02-01T10:50:06.477 DEBUG:boto:path=/vclfcn-63/obj8?versionId=null 2024-02-01T10:50:06.477 DEBUG:boto:auth_path=/vclfcn-63/obj8?versionId=null 2024-02-01T10:50:06.477 DEBUG:boto:Method: GET 2024-02-01T10:50:06.478 DEBUG:boto:Path: /vclfcn-63/obj8?versionId=null 2024-02-01T10:50:06.478 DEBUG:boto:Data: 2024-02-01T10:50:06.478 DEBUG:boto:Headers: {} 2024-02-01T10:50:06.478 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:06.478 DEBUG:boto:Port: 8001 2024-02-01T10:50:06.478 DEBUG:boto:Params: {} 2024-02-01T10:50:06.478 DEBUG:boto:Token: None 2024-02-01T10:50:06.478 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:06 GMT /vclfcn-63/obj8?versionId=null 2024-02-01T10:50:06.478 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:k5k7aVh9s8PJ/IO4Q5R3gM/dAGU= 2024-02-01T10:50:06.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:k5k7aVh9s8PJ/IO4Q5R3gM/dAGU=', 'Content-Length': '0'} 2024-02-01T10:50:06.481 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000083da746866b930d9-0065bb775e-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:06.524 DEBUG:boto:path=/vclfcn-63/obj8 2024-02-01T10:50:06.524 DEBUG:boto:auth_path=/vclfcn-63/obj8 2024-02-01T10:50:06.525 DEBUG:boto:path=/vclfcn-63/obj8?versionId=null 2024-02-01T10:50:06.525 DEBUG:boto:auth_path=/vclfcn-63/obj8?versionId=null 2024-02-01T10:50:06.525 DEBUG:boto:Method: GET 2024-02-01T10:50:06.525 DEBUG:boto:Path: /vclfcn-63/obj8?versionId=null 2024-02-01T10:50:06.525 DEBUG:boto:Data: 2024-02-01T10:50:06.525 DEBUG:boto:Headers: {} 2024-02-01T10:50:06.525 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:06.525 DEBUG:boto:Port: 8000 2024-02-01T10:50:06.525 DEBUG:boto:Params: {} 2024-02-01T10:50:06.525 DEBUG:boto:Token: None 2024-02-01T10:50:06.526 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:06 GMT /vclfcn-63/obj8?versionId=null 2024-02-01T10:50:06.526 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:k5k7aVh9s8PJ/IO4Q5R3gM/dAGU= 2024-02-01T10:50:06.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:k5k7aVh9s8PJ/IO4Q5R3gM/dAGU=', 'Content-Length': '0'} 2024-02-01T10:50:06.529 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000ee565dab16e9196-0065bb775e-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:06.572 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:06.572 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:06.573 DEBUG:boto:path=/vclfcn-63/?prefix=obj8 2024-02-01T10:50:06.573 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj8 2024-02-01T10:50:06.573 DEBUG:boto:Method: GET 2024-02-01T10:50:06.573 DEBUG:boto:Path: /vclfcn-63/?prefix=obj8 2024-02-01T10:50:06.573 DEBUG:boto:Data: 2024-02-01T10:50:06.573 DEBUG:boto:Headers: {} 2024-02-01T10:50:06.573 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:06.573 DEBUG:boto:Port: 8001 2024-02-01T10:50:06.573 DEBUG:boto:Params: {} 2024-02-01T10:50:06.573 DEBUG:boto:Token: None 2024-02-01T10:50:06.574 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:06 GMT /vclfcn-63/ 2024-02-01T10:50:06.574 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:hPSiqnFPiqiYoo/UABhcN/FSAnU= 2024-02-01T10:50:06.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:hPSiqnFPiqiYoo/UABhcN/FSAnU=', 'Content-Length': '0'} 2024-02-01T10:50:06.580 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086963c4048448017-0065bb775e-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:06.620 DEBUG:boto:b'vclfcn-63obj81000falseobj82024-02-01T10:49:22.343Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:06.621 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:06.621 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:06.621 DEBUG:boto:path=/vclfcn-63/?prefix=obj8 2024-02-01T10:50:06.621 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj8 2024-02-01T10:50:06.621 DEBUG:boto:Method: GET 2024-02-01T10:50:06.621 DEBUG:boto:Path: /vclfcn-63/?prefix=obj8 2024-02-01T10:50:06.621 DEBUG:boto:Data: 2024-02-01T10:50:06.621 DEBUG:boto:Headers: {} 2024-02-01T10:50:06.621 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:06.622 DEBUG:boto:Port: 8000 2024-02-01T10:50:06.622 DEBUG:boto:Params: {} 2024-02-01T10:50:06.622 DEBUG:boto:Token: None 2024-02-01T10:50:06.622 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:06 GMT /vclfcn-63/ 2024-02-01T10:50:06.622 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:hPSiqnFPiqiYoo/UABhcN/FSAnU= 2024-02-01T10:50:06.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:hPSiqnFPiqiYoo/UABhcN/FSAnU=', 'Content-Length': '0'} 2024-02-01T10:50:06.627 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000559a0e4513d6cd58-0065bb775e-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:06.627 DEBUG:boto:b'vclfcn-63obj81000falseobj82024-02-01T10:49:22.343Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:06.627 DEBUG:boto:path=/vclfcn-63/obj8 2024-02-01T10:50:06.627 DEBUG:boto:auth_path=/vclfcn-63/obj8 2024-02-01T10:50:06.627 DEBUG:boto:Method: HEAD 2024-02-01T10:50:06.627 DEBUG:boto:Path: /vclfcn-63/obj8 2024-02-01T10:50:06.628 DEBUG:boto:Data: 2024-02-01T10:50:06.628 DEBUG:boto:Headers: {} 2024-02-01T10:50:06.628 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:06.628 DEBUG:boto:Port: 8001 2024-02-01T10:50:06.628 DEBUG:boto:Params: {} 2024-02-01T10:50:06.628 DEBUG:boto:Token: None 2024-02-01T10:50:06.628 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:06 GMT /vclfcn-63/obj8 2024-02-01T10:50:06.628 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:IeDJkXuVs/sB8Ip3tC88jw//ELQ= 2024-02-01T10:50:06.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:IeDJkXuVs/sB8Ip3tC88jw//ELQ=', 'Content-Length': '0'} 2024-02-01T10:50:06.631 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000156937e2080324f0-0065bb775e-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:06.631 DEBUG:boto:path=/vclfcn-63/obj8 2024-02-01T10:50:06.631 DEBUG:boto:auth_path=/vclfcn-63/obj8 2024-02-01T10:50:06.632 DEBUG:boto:Method: HEAD 2024-02-01T10:50:06.632 DEBUG:boto:Path: /vclfcn-63/obj8 2024-02-01T10:50:06.632 DEBUG:boto:Data: 2024-02-01T10:50:06.632 DEBUG:boto:Headers: {} 2024-02-01T10:50:06.632 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:06.632 DEBUG:boto:Port: 8000 2024-02-01T10:50:06.632 DEBUG:boto:Params: {} 2024-02-01T10:50:06.632 DEBUG:boto:Token: None 2024-02-01T10:50:06.632 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:06 GMT /vclfcn-63/obj8 2024-02-01T10:50:06.633 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:IeDJkXuVs/sB8Ip3tC88jw//ELQ= 2024-02-01T10:50:06.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:IeDJkXuVs/sB8Ip3tC88jw//ELQ=', 'Content-Length': '0'} 2024-02-01T10:50:06.635 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f6e773814f067754-0065bb775e-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:06.635 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-02-01T10:50:06.636 DEBUG:boto:path=/vclfcn-63/obj8 2024-02-01T10:50:06.636 DEBUG:boto:auth_path=/vclfcn-63/obj8 2024-02-01T10:50:06.636 DEBUG:boto:Method: GET 2024-02-01T10:50:06.636 DEBUG:boto:Path: /vclfcn-63/obj8 2024-02-01T10:50:06.636 DEBUG:boto:Data: 2024-02-01T10:50:06.636 DEBUG:boto:Headers: {} 2024-02-01T10:50:06.636 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:06.637 DEBUG:boto:Port: 8001 2024-02-01T10:50:06.637 DEBUG:boto:Params: {} 2024-02-01T10:50:06.637 DEBUG:boto:Token: None 2024-02-01T10:50:06.637 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:06 GMT /vclfcn-63/obj8 2024-02-01T10:50:06.637 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:VjY8VlXv6l2nl44RxbYPEddssXc= 2024-02-01T10:50:06.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:VjY8VlXv6l2nl44RxbYPEddssXc=', 'Content-Length': '0'} 2024-02-01T10:50:06.640 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000038a12914d69c5c74-0065bb775e-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:06.684 DEBUG:boto:path=/vclfcn-63/obj8 2024-02-01T10:50:06.684 DEBUG:boto:auth_path=/vclfcn-63/obj8 2024-02-01T10:50:06.684 DEBUG:boto:Method: GET 2024-02-01T10:50:06.684 DEBUG:boto:Path: /vclfcn-63/obj8 2024-02-01T10:50:06.684 DEBUG:boto:Data: 2024-02-01T10:50:06.685 DEBUG:boto:Headers: {} 2024-02-01T10:50:06.685 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:06.685 DEBUG:boto:Port: 8000 2024-02-01T10:50:06.685 DEBUG:boto:Params: {} 2024-02-01T10:50:06.685 DEBUG:boto:Token: None 2024-02-01T10:50:06.685 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:06 GMT /vclfcn-63/obj8 2024-02-01T10:50:06.685 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:VjY8VlXv6l2nl44RxbYPEddssXc= 2024-02-01T10:50:06.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:VjY8VlXv6l2nl44RxbYPEddssXc=', 'Content-Length': '0'} 2024-02-01T10:50:06.688 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c53ef7a7bcf1a7ba-0065bb775e-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:06.736 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:06.736 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:06.736 DEBUG:boto:path=/vclfcn-63/?prefix=obj8 2024-02-01T10:50:06.737 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj8 2024-02-01T10:50:06.737 DEBUG:boto:Method: GET 2024-02-01T10:50:06.737 DEBUG:boto:Path: /vclfcn-63/?prefix=obj8 2024-02-01T10:50:06.737 DEBUG:boto:Data: 2024-02-01T10:50:06.737 DEBUG:boto:Headers: {} 2024-02-01T10:50:06.737 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:06.737 DEBUG:boto:Port: 8001 2024-02-01T10:50:06.737 DEBUG:boto:Params: {} 2024-02-01T10:50:06.737 DEBUG:boto:Token: None 2024-02-01T10:50:06.737 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:06 GMT /vclfcn-63/ 2024-02-01T10:50:06.738 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:hPSiqnFPiqiYoo/UABhcN/FSAnU= 2024-02-01T10:50:06.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:hPSiqnFPiqiYoo/UABhcN/FSAnU=', 'Content-Length': '0'} 2024-02-01T10:50:06.745 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000073fe01228202bab1-0065bb775e-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:06.792 DEBUG:boto:b'vclfcn-63obj81000falseobj82024-02-01T10:49:22.343Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:06.793 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:06.793 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:06.793 DEBUG:boto:path=/vclfcn-63/?prefix=obj8 2024-02-01T10:50:06.793 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj8 2024-02-01T10:50:06.793 DEBUG:boto:Method: GET 2024-02-01T10:50:06.793 DEBUG:boto:Path: /vclfcn-63/?prefix=obj8 2024-02-01T10:50:06.793 DEBUG:boto:Data: 2024-02-01T10:50:06.793 DEBUG:boto:Headers: {} 2024-02-01T10:50:06.793 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:06.793 DEBUG:boto:Port: 8000 2024-02-01T10:50:06.794 DEBUG:boto:Params: {} 2024-02-01T10:50:06.794 DEBUG:boto:Token: None 2024-02-01T10:50:06.794 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:06 GMT /vclfcn-63/ 2024-02-01T10:50:06.794 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:hPSiqnFPiqiYoo/UABhcN/FSAnU= 2024-02-01T10:50:06.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:hPSiqnFPiqiYoo/UABhcN/FSAnU=', 'Content-Length': '0'} 2024-02-01T10:50:06.798 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000034449642b4b87147-0065bb775e-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:06.799 DEBUG:boto:b'vclfcn-63obj81000falseobj82024-02-01T10:49:22.343Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:06.799 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-63 zones={test-zone3, test-zone1} 2024-02-01T10:50:06.799 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-63 zones={test-zone3, test-zone2} 2024-02-01T10:50:06.799 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:06.799 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:06.799 DEBUG:boto:Method: HEAD 2024-02-01T10:50:06.800 DEBUG:boto:Path: /vclfcn-63/ 2024-02-01T10:50:06.800 DEBUG:boto:Data: 2024-02-01T10:50:06.800 DEBUG:boto:Headers: {} 2024-02-01T10:50:06.800 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:06.800 DEBUG:boto:Port: 8001 2024-02-01T10:50:06.800 DEBUG:boto:Params: {} 2024-02-01T10:50:06.800 DEBUG:boto:Token: None 2024-02-01T10:50:06.800 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:06 GMT /vclfcn-63/ 2024-02-01T10:50:06.800 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:29f4Uw5AdZEyYsRleglL92jRQzY= 2024-02-01T10:50:06.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:29f4Uw5AdZEyYsRleglL92jRQzY=', 'Content-Length': '0'} 2024-02-01T10:50:06.804 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006c2a08cdc6a96b17-0065bb775e-5762-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:50:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:06.804 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:06.805 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:06.805 DEBUG:boto:Method: HEAD 2024-02-01T10:50:06.805 DEBUG:boto:Path: /vclfcn-63/ 2024-02-01T10:50:06.805 DEBUG:boto:Data: 2024-02-01T10:50:06.805 DEBUG:boto:Headers: {} 2024-02-01T10:50:06.805 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:06.805 DEBUG:boto:Port: 8000 2024-02-01T10:50:06.805 DEBUG:boto:Params: {} 2024-02-01T10:50:06.805 DEBUG:boto:Token: None 2024-02-01T10:50:06.805 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:06 GMT /vclfcn-63/ 2024-02-01T10:50:06.806 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:29f4Uw5AdZEyYsRleglL92jRQzY= 2024-02-01T10:50:06.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:29f4Uw5AdZEyYsRleglL92jRQzY=', 'Content-Length': '0'} 2024-02-01T10:50:06.810 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e7dde0e3103aea96-0065bb775e-5350-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:50:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:06.810 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:50:06.811 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:06.811 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:06.811 DEBUG:boto:path=/vclfcn-63/?versions&max-keys=999 2024-02-01T10:50:06.811 DEBUG:boto:auth_path=/vclfcn-63/?versions&max-keys=999 2024-02-01T10:50:06.811 DEBUG:boto:Method: GET 2024-02-01T10:50:06.811 DEBUG:boto:Path: /vclfcn-63/?versions&max-keys=999 2024-02-01T10:50:06.811 DEBUG:boto:Data: 2024-02-01T10:50:06.811 DEBUG:boto:Headers: {} 2024-02-01T10:50:06.811 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:06.811 DEBUG:boto:Port: 8001 2024-02-01T10:50:06.811 DEBUG:boto:Params: {} 2024-02-01T10:50:06.812 DEBUG:boto:Token: None 2024-02-01T10:50:06.812 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:06 GMT /vclfcn-63/?versions 2024-02-01T10:50:06.812 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fvol2p33VPPFZ34sOmyYbjVOaf8= 2024-02-01T10:50:06.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fvol2p33VPPFZ34sOmyYbjVOaf8=', 'Content-Length': '0'} 2024-02-01T10:50:06.817 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000004f44cf272012cd-0065bb775e-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:06.817 DEBUG:boto:b'vclfcn-63999falseobj1nulltrue2024-02-01T10:48:26.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:48:26.507Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:48:26.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:48:26.575Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-02-01T10:49:22.255Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-02-01T10:49:22.271Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-02-01T10:49:22.327Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-02-01T10:49:22.343Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:06.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-02-01T10:50:06.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-02-01T10:50:06.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-02-01T10:50:06.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-02-01T10:50:06.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-02-01T10:50:06.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-02-01T10:50:06.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-02-01T10:50:06.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-02-01T10:50:06.818 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:50:06.819 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:06.819 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:06.819 DEBUG:boto:path=/vclfcn-63/?versions&max-keys=999 2024-02-01T10:50:06.819 DEBUG:boto:auth_path=/vclfcn-63/?versions&max-keys=999 2024-02-01T10:50:06.819 DEBUG:boto:Method: GET 2024-02-01T10:50:06.819 DEBUG:boto:Path: /vclfcn-63/?versions&max-keys=999 2024-02-01T10:50:06.819 DEBUG:boto:Data: 2024-02-01T10:50:06.819 DEBUG:boto:Headers: {} 2024-02-01T10:50:06.819 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:06.819 DEBUG:boto:Port: 8000 2024-02-01T10:50:06.819 DEBUG:boto:Params: {} 2024-02-01T10:50:06.820 DEBUG:boto:Token: None 2024-02-01T10:50:06.820 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:06 GMT /vclfcn-63/?versions 2024-02-01T10:50:06.820 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fvol2p33VPPFZ34sOmyYbjVOaf8= 2024-02-01T10:50:06.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fvol2p33VPPFZ34sOmyYbjVOaf8=', 'Content-Length': '0'} 2024-02-01T10:50:06.824 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3693d3136beca07-0065bb775e-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:06.824 DEBUG:boto:b'vclfcn-63999falseobj1nulltrue2024-02-01T10:48:26.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:48:26.507Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:48:26.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:48:26.575Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-02-01T10:49:22.255Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-02-01T10:49:22.271Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-02-01T10:49:22.327Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-02-01T10:49:22.343Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:06.825 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-02-01T10:50:06.825 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-02-01T10:50:06.825 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-02-01T10:50:06.825 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-02-01T10:50:06.825 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-02-01T10:50:06.825 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-02-01T10:50:06.825 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-02-01T10:50:06.825 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-02-01T10:50:06.826 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:06.826 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:06.826 DEBUG:boto:path=/vclfcn-63/?versions&max-keys=999 2024-02-01T10:50:06.826 DEBUG:boto:auth_path=/vclfcn-63/?versions&max-keys=999 2024-02-01T10:50:06.826 DEBUG:boto:Method: GET 2024-02-01T10:50:06.826 DEBUG:boto:Path: /vclfcn-63/?versions&max-keys=999 2024-02-01T10:50:06.826 DEBUG:boto:Data: 2024-02-01T10:50:06.826 DEBUG:boto:Headers: {} 2024-02-01T10:50:06.826 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:06.826 DEBUG:boto:Port: 8001 2024-02-01T10:50:06.826 DEBUG:boto:Params: {} 2024-02-01T10:50:06.827 DEBUG:boto:Token: None 2024-02-01T10:50:06.827 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:06 GMT /vclfcn-63/?versions 2024-02-01T10:50:06.827 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fvol2p33VPPFZ34sOmyYbjVOaf8= 2024-02-01T10:50:06.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fvol2p33VPPFZ34sOmyYbjVOaf8=', 'Content-Length': '0'} 2024-02-01T10:50:06.832 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000074cbbebd058199ad-0065bb775e-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:06.833 DEBUG:boto:b'vclfcn-63999falseobj1nulltrue2024-02-01T10:48:26.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:48:26.507Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:48:26.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:48:26.575Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-02-01T10:49:22.255Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-02-01T10:49:22.271Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-02-01T10:49:22.327Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-02-01T10:49:22.343Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:06.833 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:06.833 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:06.833 DEBUG:boto:path=/vclfcn-63/?versions&max-keys=999 2024-02-01T10:50:06.834 DEBUG:boto:auth_path=/vclfcn-63/?versions&max-keys=999 2024-02-01T10:50:06.834 DEBUG:boto:Method: GET 2024-02-01T10:50:06.834 DEBUG:boto:Path: /vclfcn-63/?versions&max-keys=999 2024-02-01T10:50:06.834 DEBUG:boto:Data: 2024-02-01T10:50:06.834 DEBUG:boto:Headers: {} 2024-02-01T10:50:06.834 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:06.834 DEBUG:boto:Port: 8000 2024-02-01T10:50:06.834 DEBUG:boto:Params: {} 2024-02-01T10:50:06.834 DEBUG:boto:Token: None 2024-02-01T10:50:06.834 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:06 GMT /vclfcn-63/?versions 2024-02-01T10:50:06.835 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:fvol2p33VPPFZ34sOmyYbjVOaf8= 2024-02-01T10:50:06.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:fvol2p33VPPFZ34sOmyYbjVOaf8=', 'Content-Length': '0'} 2024-02-01T10:50:06.839 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077ffb7eb8b0a6fe7-0065bb775e-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:06.839 DEBUG:boto:b'vclfcn-63999falseobj1nulltrue2024-02-01T10:48:26.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:48:26.507Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:48:26.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:48:26.575Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-02-01T10:49:22.255Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-02-01T10:49:22.271Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-02-01T10:49:22.327Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-02-01T10:49:22.343Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:06.840 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:50:06.840 DEBUG:boto:path=/vclfcn-63/obj1 2024-02-01T10:50:06.840 DEBUG:boto:auth_path=/vclfcn-63/obj1 2024-02-01T10:50:06.840 DEBUG:boto:path=/vclfcn-63/obj1?versionId=null 2024-02-01T10:50:06.840 DEBUG:boto:auth_path=/vclfcn-63/obj1?versionId=null 2024-02-01T10:50:06.840 DEBUG:boto:Method: GET 2024-02-01T10:50:06.841 DEBUG:boto:Path: /vclfcn-63/obj1?versionId=null 2024-02-01T10:50:06.841 DEBUG:boto:Data: 2024-02-01T10:50:06.841 DEBUG:boto:Headers: {} 2024-02-01T10:50:06.841 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:06.841 DEBUG:boto:Port: 8001 2024-02-01T10:50:06.841 DEBUG:boto:Params: {} 2024-02-01T10:50:06.841 DEBUG:boto:Token: None 2024-02-01T10:50:06.841 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:06 GMT /vclfcn-63/obj1?versionId=null 2024-02-01T10:50:06.841 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:X8BA+1mGXQ1SwpZW0UonzdB3g6k= 2024-02-01T10:50:06.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:X8BA+1mGXQ1SwpZW0UonzdB3g6k=', 'Content-Length': '0'} 2024-02-01T10:50:06.844 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003bca6e87df144bf0-0065bb775e-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:06.888 DEBUG:boto:path=/vclfcn-63/obj1 2024-02-01T10:50:06.888 DEBUG:boto:auth_path=/vclfcn-63/obj1 2024-02-01T10:50:06.888 DEBUG:boto:path=/vclfcn-63/obj1?versionId=null 2024-02-01T10:50:06.888 DEBUG:boto:auth_path=/vclfcn-63/obj1?versionId=null 2024-02-01T10:50:06.889 DEBUG:boto:Method: GET 2024-02-01T10:50:06.889 DEBUG:boto:Path: /vclfcn-63/obj1?versionId=null 2024-02-01T10:50:06.889 DEBUG:boto:Data: 2024-02-01T10:50:06.889 DEBUG:boto:Headers: {} 2024-02-01T10:50:06.889 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:06.889 DEBUG:boto:Port: 8000 2024-02-01T10:50:06.889 DEBUG:boto:Params: {} 2024-02-01T10:50:06.889 DEBUG:boto:Token: None 2024-02-01T10:50:06.889 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:06 GMT /vclfcn-63/obj1?versionId=null 2024-02-01T10:50:06.890 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:X8BA+1mGXQ1SwpZW0UonzdB3g6k= 2024-02-01T10:50:06.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:X8BA+1mGXQ1SwpZW0UonzdB3g6k=', 'Content-Length': '0'} 2024-02-01T10:50:06.893 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000415300b1db5a5168-0065bb775e-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:06.893 DEBUG:boto:path=/vclfcn-63/obj1 2024-02-01T10:50:06.893 DEBUG:boto:auth_path=/vclfcn-63/obj1 2024-02-01T10:50:06.893 DEBUG:boto:path=/vclfcn-63/obj1?versionId=null 2024-02-01T10:50:06.893 DEBUG:boto:auth_path=/vclfcn-63/obj1?versionId=null 2024-02-01T10:50:06.893 DEBUG:boto:Method: HEAD 2024-02-01T10:50:06.894 DEBUG:boto:Path: /vclfcn-63/obj1?versionId=null 2024-02-01T10:50:06.894 DEBUG:boto:Data: 2024-02-01T10:50:06.894 DEBUG:boto:Headers: {} 2024-02-01T10:50:06.894 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:06.894 DEBUG:boto:Port: 8001 2024-02-01T10:50:06.894 DEBUG:boto:Params: {} 2024-02-01T10:50:06.894 DEBUG:boto:Token: None 2024-02-01T10:50:06.894 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:06 GMT /vclfcn-63/obj1?versionId=null 2024-02-01T10:50:06.894 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:bNsBw0sYf1gwfPsFttXAIus29vM= 2024-02-01T10:50:06.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:bNsBw0sYf1gwfPsFttXAIus29vM=', 'Content-Length': '0'} 2024-02-01T10:50:06.897 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f40f42a55b87c0d1-0065bb775e-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:06.897 DEBUG:boto:path=/vclfcn-63/obj1 2024-02-01T10:50:06.898 DEBUG:boto:auth_path=/vclfcn-63/obj1 2024-02-01T10:50:06.898 DEBUG:boto:path=/vclfcn-63/obj1?versionId=null 2024-02-01T10:50:06.898 DEBUG:boto:auth_path=/vclfcn-63/obj1?versionId=null 2024-02-01T10:50:06.898 DEBUG:boto:Method: HEAD 2024-02-01T10:50:06.898 DEBUG:boto:Path: /vclfcn-63/obj1?versionId=null 2024-02-01T10:50:06.898 DEBUG:boto:Data: 2024-02-01T10:50:06.898 DEBUG:boto:Headers: {} 2024-02-01T10:50:06.898 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:06.898 DEBUG:boto:Port: 8000 2024-02-01T10:50:06.898 DEBUG:boto:Params: {} 2024-02-01T10:50:06.898 DEBUG:boto:Token: None 2024-02-01T10:50:06.899 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:06 GMT /vclfcn-63/obj1?versionId=null 2024-02-01T10:50:06.899 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:bNsBw0sYf1gwfPsFttXAIus29vM= 2024-02-01T10:50:06.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:bNsBw0sYf1gwfPsFttXAIus29vM=', 'Content-Length': '0'} 2024-02-01T10:50:06.902 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000088e32ba351015e11-0065bb775e-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:06.902 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:50:06.902 DEBUG:boto:path=/vclfcn-63/obj1 2024-02-01T10:50:06.902 DEBUG:boto:auth_path=/vclfcn-63/obj1 2024-02-01T10:50:06.902 DEBUG:boto:path=/vclfcn-63/obj1?versionId=null 2024-02-01T10:50:06.902 DEBUG:boto:auth_path=/vclfcn-63/obj1?versionId=null 2024-02-01T10:50:06.903 DEBUG:boto:Method: GET 2024-02-01T10:50:06.903 DEBUG:boto:Path: /vclfcn-63/obj1?versionId=null 2024-02-01T10:50:06.903 DEBUG:boto:Data: 2024-02-01T10:50:06.903 DEBUG:boto:Headers: {} 2024-02-01T10:50:06.903 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:06.903 DEBUG:boto:Port: 8001 2024-02-01T10:50:06.903 DEBUG:boto:Params: {} 2024-02-01T10:50:06.903 DEBUG:boto:Token: None 2024-02-01T10:50:06.903 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:06 GMT /vclfcn-63/obj1?versionId=null 2024-02-01T10:50:06.904 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:X8BA+1mGXQ1SwpZW0UonzdB3g6k= 2024-02-01T10:50:06.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:X8BA+1mGXQ1SwpZW0UonzdB3g6k=', 'Content-Length': '0'} 2024-02-01T10:50:06.906 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f3f34a51035ebac7-0065bb775e-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:06.948 DEBUG:boto:path=/vclfcn-63/obj1 2024-02-01T10:50:06.948 DEBUG:boto:auth_path=/vclfcn-63/obj1 2024-02-01T10:50:06.948 DEBUG:boto:path=/vclfcn-63/obj1?versionId=null 2024-02-01T10:50:06.948 DEBUG:boto:auth_path=/vclfcn-63/obj1?versionId=null 2024-02-01T10:50:06.949 DEBUG:boto:Method: GET 2024-02-01T10:50:06.949 DEBUG:boto:Path: /vclfcn-63/obj1?versionId=null 2024-02-01T10:50:06.949 DEBUG:boto:Data: 2024-02-01T10:50:06.949 DEBUG:boto:Headers: {} 2024-02-01T10:50:06.949 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:06.949 DEBUG:boto:Port: 8000 2024-02-01T10:50:06.949 DEBUG:boto:Params: {} 2024-02-01T10:50:06.949 DEBUG:boto:Token: None 2024-02-01T10:50:06.949 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:06 GMT /vclfcn-63/obj1?versionId=null 2024-02-01T10:50:06.950 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:X8BA+1mGXQ1SwpZW0UonzdB3g6k= 2024-02-01T10:50:06.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:X8BA+1mGXQ1SwpZW0UonzdB3g6k=', 'Content-Length': '0'} 2024-02-01T10:50:06.953 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aeeaea85c5ed1d58-0065bb775e-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:06.953 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:06.953 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:06.953 DEBUG:boto:path=/vclfcn-63/?prefix=obj1 2024-02-01T10:50:06.953 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj1 2024-02-01T10:50:06.954 DEBUG:boto:Method: GET 2024-02-01T10:50:06.954 DEBUG:boto:Path: /vclfcn-63/?prefix=obj1 2024-02-01T10:50:06.954 DEBUG:boto:Data: 2024-02-01T10:50:06.954 DEBUG:boto:Headers: {} 2024-02-01T10:50:06.954 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:06.954 DEBUG:boto:Port: 8001 2024-02-01T10:50:06.954 DEBUG:boto:Params: {} 2024-02-01T10:50:06.954 DEBUG:boto:Token: None 2024-02-01T10:50:06.954 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:06 GMT /vclfcn-63/ 2024-02-01T10:50:06.955 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:hPSiqnFPiqiYoo/UABhcN/FSAnU= 2024-02-01T10:50:06.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:06 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:hPSiqnFPiqiYoo/UABhcN/FSAnU=', 'Content-Length': '0'} 2024-02-01T10:50:06.959 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d908544fff934eb-0065bb775e-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:06 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:07.000 DEBUG:boto:b'vclfcn-63obj11000falseobj12024-02-01T10:48:26.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:07.001 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:07.001 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:07.001 DEBUG:boto:path=/vclfcn-63/?prefix=obj1 2024-02-01T10:50:07.001 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj1 2024-02-01T10:50:07.001 DEBUG:boto:Method: GET 2024-02-01T10:50:07.001 DEBUG:boto:Path: /vclfcn-63/?prefix=obj1 2024-02-01T10:50:07.001 DEBUG:boto:Data: 2024-02-01T10:50:07.001 DEBUG:boto:Headers: {} 2024-02-01T10:50:07.001 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:07.001 DEBUG:boto:Port: 8000 2024-02-01T10:50:07.002 DEBUG:boto:Params: {} 2024-02-01T10:50:07.002 DEBUG:boto:Token: None 2024-02-01T10:50:07.002 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:07 GMT /vclfcn-63/ 2024-02-01T10:50:07.002 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:i9iEv8SZmFIq7dzLkaM60uCok+4= 2024-02-01T10:50:07.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:i9iEv8SZmFIq7dzLkaM60uCok+4=', 'Content-Length': '0'} 2024-02-01T10:50:07.006 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094c83d40f93c5bf1-0065bb775f-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:07.007 DEBUG:boto:b'vclfcn-63obj11000falseobj12024-02-01T10:48:26.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:07.007 DEBUG:boto:path=/vclfcn-63/obj1 2024-02-01T10:50:07.007 DEBUG:boto:auth_path=/vclfcn-63/obj1 2024-02-01T10:50:07.007 DEBUG:boto:Method: HEAD 2024-02-01T10:50:07.007 DEBUG:boto:Path: /vclfcn-63/obj1 2024-02-01T10:50:07.007 DEBUG:boto:Data: 2024-02-01T10:50:07.007 DEBUG:boto:Headers: {} 2024-02-01T10:50:07.007 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:07.008 DEBUG:boto:Port: 8001 2024-02-01T10:50:07.008 DEBUG:boto:Params: {} 2024-02-01T10:50:07.008 DEBUG:boto:Token: None 2024-02-01T10:50:07.008 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:07 GMT /vclfcn-63/obj1 2024-02-01T10:50:07.008 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZKBUPM3vIbsBQnmgFiOPGUTuWpo= 2024-02-01T10:50:07.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZKBUPM3vIbsBQnmgFiOPGUTuWpo=', 'Content-Length': '0'} 2024-02-01T10:50:07.011 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000425f42cf6eba8782-0065bb775f-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:07.011 DEBUG:boto:path=/vclfcn-63/obj1 2024-02-01T10:50:07.011 DEBUG:boto:auth_path=/vclfcn-63/obj1 2024-02-01T10:50:07.011 DEBUG:boto:Method: HEAD 2024-02-01T10:50:07.012 DEBUG:boto:Path: /vclfcn-63/obj1 2024-02-01T10:50:07.012 DEBUG:boto:Data: 2024-02-01T10:50:07.012 DEBUG:boto:Headers: {} 2024-02-01T10:50:07.012 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:07.012 DEBUG:boto:Port: 8000 2024-02-01T10:50:07.012 DEBUG:boto:Params: {} 2024-02-01T10:50:07.012 DEBUG:boto:Token: None 2024-02-01T10:50:07.012 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:07 GMT /vclfcn-63/obj1 2024-02-01T10:50:07.012 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZKBUPM3vIbsBQnmgFiOPGUTuWpo= 2024-02-01T10:50:07.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZKBUPM3vIbsBQnmgFiOPGUTuWpo=', 'Content-Length': '0'} 2024-02-01T10:50:07.015 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009722dbdb86974684-0065bb775f-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:07.015 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:50:07.015 DEBUG:boto:path=/vclfcn-63/obj1 2024-02-01T10:50:07.016 DEBUG:boto:auth_path=/vclfcn-63/obj1 2024-02-01T10:50:07.016 DEBUG:boto:Method: GET 2024-02-01T10:50:07.016 DEBUG:boto:Path: /vclfcn-63/obj1 2024-02-01T10:50:07.016 DEBUG:boto:Data: 2024-02-01T10:50:07.016 DEBUG:boto:Headers: {} 2024-02-01T10:50:07.016 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:07.016 DEBUG:boto:Port: 8001 2024-02-01T10:50:07.016 DEBUG:boto:Params: {} 2024-02-01T10:50:07.016 DEBUG:boto:Token: None 2024-02-01T10:50:07.017 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:07 GMT /vclfcn-63/obj1 2024-02-01T10:50:07.017 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:AxSX0bk+wSlMRyVlx+BSHqyfrWE= 2024-02-01T10:50:07.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:AxSX0bk+wSlMRyVlx+BSHqyfrWE=', 'Content-Length': '0'} 2024-02-01T10:50:07.019 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b3ae161485a1a4eb-0065bb775f-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:07.060 DEBUG:boto:path=/vclfcn-63/obj1 2024-02-01T10:50:07.060 DEBUG:boto:auth_path=/vclfcn-63/obj1 2024-02-01T10:50:07.060 DEBUG:boto:Method: GET 2024-02-01T10:50:07.060 DEBUG:boto:Path: /vclfcn-63/obj1 2024-02-01T10:50:07.061 DEBUG:boto:Data: 2024-02-01T10:50:07.061 DEBUG:boto:Headers: {} 2024-02-01T10:50:07.061 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:07.061 DEBUG:boto:Port: 8000 2024-02-01T10:50:07.061 DEBUG:boto:Params: {} 2024-02-01T10:50:07.061 DEBUG:boto:Token: None 2024-02-01T10:50:07.061 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:07 GMT /vclfcn-63/obj1 2024-02-01T10:50:07.061 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:AxSX0bk+wSlMRyVlx+BSHqyfrWE= 2024-02-01T10:50:07.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:AxSX0bk+wSlMRyVlx+BSHqyfrWE=', 'Content-Length': '0'} 2024-02-01T10:50:07.064 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000060dfffa4a092bea9-0065bb775f-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:07.065 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:07.065 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:07.065 DEBUG:boto:path=/vclfcn-63/?prefix=obj1 2024-02-01T10:50:07.065 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj1 2024-02-01T10:50:07.065 DEBUG:boto:Method: GET 2024-02-01T10:50:07.065 DEBUG:boto:Path: /vclfcn-63/?prefix=obj1 2024-02-01T10:50:07.065 DEBUG:boto:Data: 2024-02-01T10:50:07.065 DEBUG:boto:Headers: {} 2024-02-01T10:50:07.065 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:07.066 DEBUG:boto:Port: 8001 2024-02-01T10:50:07.066 DEBUG:boto:Params: {} 2024-02-01T10:50:07.066 DEBUG:boto:Token: None 2024-02-01T10:50:07.066 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:07 GMT /vclfcn-63/ 2024-02-01T10:50:07.066 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:i9iEv8SZmFIq7dzLkaM60uCok+4= 2024-02-01T10:50:07.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:i9iEv8SZmFIq7dzLkaM60uCok+4=', 'Content-Length': '0'} 2024-02-01T10:50:07.071 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ffd0441a051117b2-0065bb775f-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:07.112 DEBUG:boto:b'vclfcn-63obj11000falseobj12024-02-01T10:48:26.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:07.113 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:07.113 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:07.113 DEBUG:boto:path=/vclfcn-63/?prefix=obj1 2024-02-01T10:50:07.113 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj1 2024-02-01T10:50:07.113 DEBUG:boto:Method: GET 2024-02-01T10:50:07.113 DEBUG:boto:Path: /vclfcn-63/?prefix=obj1 2024-02-01T10:50:07.113 DEBUG:boto:Data: 2024-02-01T10:50:07.113 DEBUG:boto:Headers: {} 2024-02-01T10:50:07.113 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:07.113 DEBUG:boto:Port: 8000 2024-02-01T10:50:07.114 DEBUG:boto:Params: {} 2024-02-01T10:50:07.114 DEBUG:boto:Token: None 2024-02-01T10:50:07.114 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:07 GMT /vclfcn-63/ 2024-02-01T10:50:07.114 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:i9iEv8SZmFIq7dzLkaM60uCok+4= 2024-02-01T10:50:07.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:i9iEv8SZmFIq7dzLkaM60uCok+4=', 'Content-Length': '0'} 2024-02-01T10:50:07.118 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000afc7be1b8607c7f1-0065bb775f-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:07.118 DEBUG:boto:b'vclfcn-63obj11000falseobj12024-02-01T10:48:26.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:07.118 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:50:07.119 DEBUG:boto:path=/vclfcn-63/obj2 2024-02-01T10:50:07.119 DEBUG:boto:auth_path=/vclfcn-63/obj2 2024-02-01T10:50:07.119 DEBUG:boto:path=/vclfcn-63/obj2?versionId=null 2024-02-01T10:50:07.119 DEBUG:boto:auth_path=/vclfcn-63/obj2?versionId=null 2024-02-01T10:50:07.119 DEBUG:boto:Method: GET 2024-02-01T10:50:07.119 DEBUG:boto:Path: /vclfcn-63/obj2?versionId=null 2024-02-01T10:50:07.119 DEBUG:boto:Data: 2024-02-01T10:50:07.119 DEBUG:boto:Headers: {} 2024-02-01T10:50:07.119 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:07.119 DEBUG:boto:Port: 8001 2024-02-01T10:50:07.120 DEBUG:boto:Params: {} 2024-02-01T10:50:07.120 DEBUG:boto:Token: None 2024-02-01T10:50:07.120 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:07 GMT /vclfcn-63/obj2?versionId=null 2024-02-01T10:50:07.120 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:CaEz4UxNkJ9F0iv8NoO9MPzKdEA= 2024-02-01T10:50:07.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:CaEz4UxNkJ9F0iv8NoO9MPzKdEA=', 'Content-Length': '0'} 2024-02-01T10:50:07.123 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000008ac085c8a020ac8-0065bb775f-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:07.164 DEBUG:boto:path=/vclfcn-63/obj2 2024-02-01T10:50:07.164 DEBUG:boto:auth_path=/vclfcn-63/obj2 2024-02-01T10:50:07.164 DEBUG:boto:path=/vclfcn-63/obj2?versionId=null 2024-02-01T10:50:07.164 DEBUG:boto:auth_path=/vclfcn-63/obj2?versionId=null 2024-02-01T10:50:07.165 DEBUG:boto:Method: GET 2024-02-01T10:50:07.165 DEBUG:boto:Path: /vclfcn-63/obj2?versionId=null 2024-02-01T10:50:07.165 DEBUG:boto:Data: 2024-02-01T10:50:07.165 DEBUG:boto:Headers: {} 2024-02-01T10:50:07.165 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:07.165 DEBUG:boto:Port: 8000 2024-02-01T10:50:07.165 DEBUG:boto:Params: {} 2024-02-01T10:50:07.165 DEBUG:boto:Token: None 2024-02-01T10:50:07.166 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:07 GMT /vclfcn-63/obj2?versionId=null 2024-02-01T10:50:07.166 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:CaEz4UxNkJ9F0iv8NoO9MPzKdEA= 2024-02-01T10:50:07.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:CaEz4UxNkJ9F0iv8NoO9MPzKdEA=', 'Content-Length': '0'} 2024-02-01T10:50:07.169 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007844bc6bf04b7ed3-0065bb775f-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:07.169 DEBUG:boto:path=/vclfcn-63/obj2 2024-02-01T10:50:07.169 DEBUG:boto:auth_path=/vclfcn-63/obj2 2024-02-01T10:50:07.169 DEBUG:boto:path=/vclfcn-63/obj2?versionId=null 2024-02-01T10:50:07.169 DEBUG:boto:auth_path=/vclfcn-63/obj2?versionId=null 2024-02-01T10:50:07.169 DEBUG:boto:Method: HEAD 2024-02-01T10:50:07.170 DEBUG:boto:Path: /vclfcn-63/obj2?versionId=null 2024-02-01T10:50:07.170 DEBUG:boto:Data: 2024-02-01T10:50:07.170 DEBUG:boto:Headers: {} 2024-02-01T10:50:07.170 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:07.170 DEBUG:boto:Port: 8001 2024-02-01T10:50:07.170 DEBUG:boto:Params: {} 2024-02-01T10:50:07.170 DEBUG:boto:Token: None 2024-02-01T10:50:07.170 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:07 GMT /vclfcn-63/obj2?versionId=null 2024-02-01T10:50:07.170 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:AIioLWMo0V8vUVbBj92BI0ssME8= 2024-02-01T10:50:07.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:AIioLWMo0V8vUVbBj92BI0ssME8=', 'Content-Length': '0'} 2024-02-01T10:50:07.173 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000005aa61bf534545ab-0065bb775f-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:07.174 DEBUG:boto:path=/vclfcn-63/obj2 2024-02-01T10:50:07.174 DEBUG:boto:auth_path=/vclfcn-63/obj2 2024-02-01T10:50:07.174 DEBUG:boto:path=/vclfcn-63/obj2?versionId=null 2024-02-01T10:50:07.174 DEBUG:boto:auth_path=/vclfcn-63/obj2?versionId=null 2024-02-01T10:50:07.174 DEBUG:boto:Method: HEAD 2024-02-01T10:50:07.174 DEBUG:boto:Path: /vclfcn-63/obj2?versionId=null 2024-02-01T10:50:07.174 DEBUG:boto:Data: 2024-02-01T10:50:07.174 DEBUG:boto:Headers: {} 2024-02-01T10:50:07.174 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:07.175 DEBUG:boto:Port: 8000 2024-02-01T10:50:07.175 DEBUG:boto:Params: {} 2024-02-01T10:50:07.175 DEBUG:boto:Token: None 2024-02-01T10:50:07.175 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:07 GMT /vclfcn-63/obj2?versionId=null 2024-02-01T10:50:07.175 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:AIioLWMo0V8vUVbBj92BI0ssME8= 2024-02-01T10:50:07.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:AIioLWMo0V8vUVbBj92BI0ssME8=', 'Content-Length': '0'} 2024-02-01T10:50:07.178 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000733ce553f8f604ba-0065bb775f-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:07.178 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:50:07.178 DEBUG:boto:path=/vclfcn-63/obj2 2024-02-01T10:50:07.178 DEBUG:boto:auth_path=/vclfcn-63/obj2 2024-02-01T10:50:07.178 DEBUG:boto:path=/vclfcn-63/obj2?versionId=null 2024-02-01T10:50:07.178 DEBUG:boto:auth_path=/vclfcn-63/obj2?versionId=null 2024-02-01T10:50:07.179 DEBUG:boto:Method: GET 2024-02-01T10:50:07.179 DEBUG:boto:Path: /vclfcn-63/obj2?versionId=null 2024-02-01T10:50:07.179 DEBUG:boto:Data: 2024-02-01T10:50:07.179 DEBUG:boto:Headers: {} 2024-02-01T10:50:07.179 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:07.179 DEBUG:boto:Port: 8001 2024-02-01T10:50:07.179 DEBUG:boto:Params: {} 2024-02-01T10:50:07.179 DEBUG:boto:Token: None 2024-02-01T10:50:07.179 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:07 GMT /vclfcn-63/obj2?versionId=null 2024-02-01T10:50:07.179 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:CaEz4UxNkJ9F0iv8NoO9MPzKdEA= 2024-02-01T10:50:07.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:CaEz4UxNkJ9F0iv8NoO9MPzKdEA=', 'Content-Length': '0'} 2024-02-01T10:50:07.183 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000076920bac02322595-0065bb775f-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:07.224 DEBUG:boto:path=/vclfcn-63/obj2 2024-02-01T10:50:07.224 DEBUG:boto:auth_path=/vclfcn-63/obj2 2024-02-01T10:50:07.225 DEBUG:boto:path=/vclfcn-63/obj2?versionId=null 2024-02-01T10:50:07.225 DEBUG:boto:auth_path=/vclfcn-63/obj2?versionId=null 2024-02-01T10:50:07.225 DEBUG:boto:Method: GET 2024-02-01T10:50:07.225 DEBUG:boto:Path: /vclfcn-63/obj2?versionId=null 2024-02-01T10:50:07.225 DEBUG:boto:Data: 2024-02-01T10:50:07.225 DEBUG:boto:Headers: {} 2024-02-01T10:50:07.225 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:07.225 DEBUG:boto:Port: 8000 2024-02-01T10:50:07.225 DEBUG:boto:Params: {} 2024-02-01T10:50:07.226 DEBUG:boto:Token: None 2024-02-01T10:50:07.226 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:07 GMT /vclfcn-63/obj2?versionId=null 2024-02-01T10:50:07.226 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:CaEz4UxNkJ9F0iv8NoO9MPzKdEA= 2024-02-01T10:50:07.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:CaEz4UxNkJ9F0iv8NoO9MPzKdEA=', 'Content-Length': '0'} 2024-02-01T10:50:07.229 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d70bc0e62d1f2333-0065bb775f-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:07.229 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:07.229 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:07.229 DEBUG:boto:path=/vclfcn-63/?prefix=obj2 2024-02-01T10:50:07.229 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj2 2024-02-01T10:50:07.230 DEBUG:boto:Method: GET 2024-02-01T10:50:07.230 DEBUG:boto:Path: /vclfcn-63/?prefix=obj2 2024-02-01T10:50:07.230 DEBUG:boto:Data: 2024-02-01T10:50:07.230 DEBUG:boto:Headers: {} 2024-02-01T10:50:07.230 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:07.230 DEBUG:boto:Port: 8001 2024-02-01T10:50:07.230 DEBUG:boto:Params: {} 2024-02-01T10:50:07.230 DEBUG:boto:Token: None 2024-02-01T10:50:07.230 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:07 GMT /vclfcn-63/ 2024-02-01T10:50:07.231 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:i9iEv8SZmFIq7dzLkaM60uCok+4= 2024-02-01T10:50:07.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:i9iEv8SZmFIq7dzLkaM60uCok+4=', 'Content-Length': '0'} 2024-02-01T10:50:07.235 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068d578e00a914486-0065bb775f-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:07.276 DEBUG:boto:b'vclfcn-63obj21000falseobj22024-02-01T10:48:26.507Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:07.277 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:07.277 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:07.277 DEBUG:boto:path=/vclfcn-63/?prefix=obj2 2024-02-01T10:50:07.277 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj2 2024-02-01T10:50:07.277 DEBUG:boto:Method: GET 2024-02-01T10:50:07.277 DEBUG:boto:Path: /vclfcn-63/?prefix=obj2 2024-02-01T10:50:07.277 DEBUG:boto:Data: 2024-02-01T10:50:07.277 DEBUG:boto:Headers: {} 2024-02-01T10:50:07.278 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:07.278 DEBUG:boto:Port: 8000 2024-02-01T10:50:07.278 DEBUG:boto:Params: {} 2024-02-01T10:50:07.278 DEBUG:boto:Token: None 2024-02-01T10:50:07.278 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:07 GMT /vclfcn-63/ 2024-02-01T10:50:07.278 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:i9iEv8SZmFIq7dzLkaM60uCok+4= 2024-02-01T10:50:07.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:i9iEv8SZmFIq7dzLkaM60uCok+4=', 'Content-Length': '0'} 2024-02-01T10:50:07.283 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064223977754e71ff-0065bb775f-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:07.283 DEBUG:boto:b'vclfcn-63obj21000falseobj22024-02-01T10:48:26.507Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:07.283 DEBUG:boto:path=/vclfcn-63/obj2 2024-02-01T10:50:07.283 DEBUG:boto:auth_path=/vclfcn-63/obj2 2024-02-01T10:50:07.283 DEBUG:boto:Method: HEAD 2024-02-01T10:50:07.284 DEBUG:boto:Path: /vclfcn-63/obj2 2024-02-01T10:50:07.284 DEBUG:boto:Data: 2024-02-01T10:50:07.284 DEBUG:boto:Headers: {} 2024-02-01T10:50:07.284 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:07.284 DEBUG:boto:Port: 8001 2024-02-01T10:50:07.284 DEBUG:boto:Params: {} 2024-02-01T10:50:07.284 DEBUG:boto:Token: None 2024-02-01T10:50:07.284 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:07 GMT /vclfcn-63/obj2 2024-02-01T10:50:07.284 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7UbB2CHnWdJtbBWf2+ujeotVmKo= 2024-02-01T10:50:07.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7UbB2CHnWdJtbBWf2+ujeotVmKo=', 'Content-Length': '0'} 2024-02-01T10:50:07.287 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ea0efa1dd255273e-0065bb775f-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:07.287 DEBUG:boto:path=/vclfcn-63/obj2 2024-02-01T10:50:07.287 DEBUG:boto:auth_path=/vclfcn-63/obj2 2024-02-01T10:50:07.288 DEBUG:boto:Method: HEAD 2024-02-01T10:50:07.288 DEBUG:boto:Path: /vclfcn-63/obj2 2024-02-01T10:50:07.288 DEBUG:boto:Data: 2024-02-01T10:50:07.288 DEBUG:boto:Headers: {} 2024-02-01T10:50:07.288 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:07.288 DEBUG:boto:Port: 8000 2024-02-01T10:50:07.288 DEBUG:boto:Params: {} 2024-02-01T10:50:07.288 DEBUG:boto:Token: None 2024-02-01T10:50:07.288 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:07 GMT /vclfcn-63/obj2 2024-02-01T10:50:07.288 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7UbB2CHnWdJtbBWf2+ujeotVmKo= 2024-02-01T10:50:07.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7UbB2CHnWdJtbBWf2+ujeotVmKo=', 'Content-Length': '0'} 2024-02-01T10:50:07.291 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f981c4e007455ad0-0065bb775f-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:07.291 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:50:07.292 DEBUG:boto:path=/vclfcn-63/obj2 2024-02-01T10:50:07.292 DEBUG:boto:auth_path=/vclfcn-63/obj2 2024-02-01T10:50:07.292 DEBUG:boto:Method: GET 2024-02-01T10:50:07.292 DEBUG:boto:Path: /vclfcn-63/obj2 2024-02-01T10:50:07.292 DEBUG:boto:Data: 2024-02-01T10:50:07.292 DEBUG:boto:Headers: {} 2024-02-01T10:50:07.292 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:07.292 DEBUG:boto:Port: 8001 2024-02-01T10:50:07.293 DEBUG:boto:Params: {} 2024-02-01T10:50:07.293 DEBUG:boto:Token: None 2024-02-01T10:50:07.293 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:07 GMT /vclfcn-63/obj2 2024-02-01T10:50:07.293 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:4CHkPdfd5g+4C2NHhFFI74b/Tm0= 2024-02-01T10:50:07.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:4CHkPdfd5g+4C2NHhFFI74b/Tm0=', 'Content-Length': '0'} 2024-02-01T10:50:07.296 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f1bd91d2626ea872-0065bb775f-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:07.336 DEBUG:boto:path=/vclfcn-63/obj2 2024-02-01T10:50:07.336 DEBUG:boto:auth_path=/vclfcn-63/obj2 2024-02-01T10:50:07.336 DEBUG:boto:Method: GET 2024-02-01T10:50:07.336 DEBUG:boto:Path: /vclfcn-63/obj2 2024-02-01T10:50:07.337 DEBUG:boto:Data: 2024-02-01T10:50:07.337 DEBUG:boto:Headers: {} 2024-02-01T10:50:07.337 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:07.337 DEBUG:boto:Port: 8000 2024-02-01T10:50:07.337 DEBUG:boto:Params: {} 2024-02-01T10:50:07.337 DEBUG:boto:Token: None 2024-02-01T10:50:07.337 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:07 GMT /vclfcn-63/obj2 2024-02-01T10:50:07.337 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:4CHkPdfd5g+4C2NHhFFI74b/Tm0= 2024-02-01T10:50:07.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:4CHkPdfd5g+4C2NHhFFI74b/Tm0=', 'Content-Length': '0'} 2024-02-01T10:50:07.340 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ccfa8d1576a22b75-0065bb775f-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:07.341 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:07.341 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:07.341 DEBUG:boto:path=/vclfcn-63/?prefix=obj2 2024-02-01T10:50:07.341 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj2 2024-02-01T10:50:07.341 DEBUG:boto:Method: GET 2024-02-01T10:50:07.341 DEBUG:boto:Path: /vclfcn-63/?prefix=obj2 2024-02-01T10:50:07.341 DEBUG:boto:Data: 2024-02-01T10:50:07.341 DEBUG:boto:Headers: {} 2024-02-01T10:50:07.341 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:07.341 DEBUG:boto:Port: 8001 2024-02-01T10:50:07.341 DEBUG:boto:Params: {} 2024-02-01T10:50:07.342 DEBUG:boto:Token: None 2024-02-01T10:50:07.342 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:07 GMT /vclfcn-63/ 2024-02-01T10:50:07.342 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:i9iEv8SZmFIq7dzLkaM60uCok+4= 2024-02-01T10:50:07.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:i9iEv8SZmFIq7dzLkaM60uCok+4=', 'Content-Length': '0'} 2024-02-01T10:50:07.346 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000463f7b534a766da4-0065bb775f-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:07.388 DEBUG:boto:b'vclfcn-63obj21000falseobj22024-02-01T10:48:26.507Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:07.389 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:07.389 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:07.389 DEBUG:boto:path=/vclfcn-63/?prefix=obj2 2024-02-01T10:50:07.389 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj2 2024-02-01T10:50:07.389 DEBUG:boto:Method: GET 2024-02-01T10:50:07.389 DEBUG:boto:Path: /vclfcn-63/?prefix=obj2 2024-02-01T10:50:07.389 DEBUG:boto:Data: 2024-02-01T10:50:07.389 DEBUG:boto:Headers: {} 2024-02-01T10:50:07.389 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:07.389 DEBUG:boto:Port: 8000 2024-02-01T10:50:07.389 DEBUG:boto:Params: {} 2024-02-01T10:50:07.390 DEBUG:boto:Token: None 2024-02-01T10:50:07.390 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:07 GMT /vclfcn-63/ 2024-02-01T10:50:07.390 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:i9iEv8SZmFIq7dzLkaM60uCok+4= 2024-02-01T10:50:07.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:i9iEv8SZmFIq7dzLkaM60uCok+4=', 'Content-Length': '0'} 2024-02-01T10:50:07.394 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000030fdb7affe5a758-0065bb775f-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:07.394 DEBUG:boto:b'vclfcn-63obj21000falseobj22024-02-01T10:48:26.507Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:07.394 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:50:07.395 DEBUG:boto:path=/vclfcn-63/obj3 2024-02-01T10:50:07.395 DEBUG:boto:auth_path=/vclfcn-63/obj3 2024-02-01T10:50:07.395 DEBUG:boto:path=/vclfcn-63/obj3?versionId=null 2024-02-01T10:50:07.395 DEBUG:boto:auth_path=/vclfcn-63/obj3?versionId=null 2024-02-01T10:50:07.395 DEBUG:boto:Method: GET 2024-02-01T10:50:07.395 DEBUG:boto:Path: /vclfcn-63/obj3?versionId=null 2024-02-01T10:50:07.395 DEBUG:boto:Data: 2024-02-01T10:50:07.395 DEBUG:boto:Headers: {} 2024-02-01T10:50:07.395 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:07.396 DEBUG:boto:Port: 8001 2024-02-01T10:50:07.396 DEBUG:boto:Params: {} 2024-02-01T10:50:07.396 DEBUG:boto:Token: None 2024-02-01T10:50:07.396 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:07 GMT /vclfcn-63/obj3?versionId=null 2024-02-01T10:50:07.396 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sQ+y4BxlIR4psecKsfhKA+GtIOQ= 2024-02-01T10:50:07.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sQ+y4BxlIR4psecKsfhKA+GtIOQ=', 'Content-Length': '0'} 2024-02-01T10:50:07.399 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d985cd5fd8c8ed5c-0065bb775f-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:07.444 DEBUG:boto:path=/vclfcn-63/obj3 2024-02-01T10:50:07.444 DEBUG:boto:auth_path=/vclfcn-63/obj3 2024-02-01T10:50:07.444 DEBUG:boto:path=/vclfcn-63/obj3?versionId=null 2024-02-01T10:50:07.445 DEBUG:boto:auth_path=/vclfcn-63/obj3?versionId=null 2024-02-01T10:50:07.445 DEBUG:boto:Method: GET 2024-02-01T10:50:07.445 DEBUG:boto:Path: /vclfcn-63/obj3?versionId=null 2024-02-01T10:50:07.445 DEBUG:boto:Data: 2024-02-01T10:50:07.445 DEBUG:boto:Headers: {} 2024-02-01T10:50:07.445 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:07.445 DEBUG:boto:Port: 8000 2024-02-01T10:50:07.445 DEBUG:boto:Params: {} 2024-02-01T10:50:07.445 DEBUG:boto:Token: None 2024-02-01T10:50:07.445 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:07 GMT /vclfcn-63/obj3?versionId=null 2024-02-01T10:50:07.446 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sQ+y4BxlIR4psecKsfhKA+GtIOQ= 2024-02-01T10:50:07.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sQ+y4BxlIR4psecKsfhKA+GtIOQ=', 'Content-Length': '0'} 2024-02-01T10:50:07.449 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000095fc2e7066ab6368-0065bb775f-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:07.449 DEBUG:boto:path=/vclfcn-63/obj3 2024-02-01T10:50:07.449 DEBUG:boto:auth_path=/vclfcn-63/obj3 2024-02-01T10:50:07.449 DEBUG:boto:path=/vclfcn-63/obj3?versionId=null 2024-02-01T10:50:07.449 DEBUG:boto:auth_path=/vclfcn-63/obj3?versionId=null 2024-02-01T10:50:07.449 DEBUG:boto:Method: HEAD 2024-02-01T10:50:07.449 DEBUG:boto:Path: /vclfcn-63/obj3?versionId=null 2024-02-01T10:50:07.449 DEBUG:boto:Data: 2024-02-01T10:50:07.450 DEBUG:boto:Headers: {} 2024-02-01T10:50:07.450 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:07.450 DEBUG:boto:Port: 8001 2024-02-01T10:50:07.450 DEBUG:boto:Params: {} 2024-02-01T10:50:07.450 DEBUG:boto:Token: None 2024-02-01T10:50:07.450 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:07 GMT /vclfcn-63/obj3?versionId=null 2024-02-01T10:50:07.450 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zkzrtZtTs4SqH7fbp/cJB4d6+ss= 2024-02-01T10:50:07.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zkzrtZtTs4SqH7fbp/cJB4d6+ss=', 'Content-Length': '0'} 2024-02-01T10:50:07.453 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000053374923d869ec5c-0065bb775f-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:07.453 DEBUG:boto:path=/vclfcn-63/obj3 2024-02-01T10:50:07.454 DEBUG:boto:auth_path=/vclfcn-63/obj3 2024-02-01T10:50:07.454 DEBUG:boto:path=/vclfcn-63/obj3?versionId=null 2024-02-01T10:50:07.454 DEBUG:boto:auth_path=/vclfcn-63/obj3?versionId=null 2024-02-01T10:50:07.454 DEBUG:boto:Method: HEAD 2024-02-01T10:50:07.454 DEBUG:boto:Path: /vclfcn-63/obj3?versionId=null 2024-02-01T10:50:07.454 DEBUG:boto:Data: 2024-02-01T10:50:07.454 DEBUG:boto:Headers: {} 2024-02-01T10:50:07.454 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:07.454 DEBUG:boto:Port: 8000 2024-02-01T10:50:07.454 DEBUG:boto:Params: {} 2024-02-01T10:50:07.454 DEBUG:boto:Token: None 2024-02-01T10:50:07.455 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:07 GMT /vclfcn-63/obj3?versionId=null 2024-02-01T10:50:07.455 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:zkzrtZtTs4SqH7fbp/cJB4d6+ss= 2024-02-01T10:50:07.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:zkzrtZtTs4SqH7fbp/cJB4d6+ss=', 'Content-Length': '0'} 2024-02-01T10:50:07.457 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d8dc9b1f132922e4-0065bb775f-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:07.458 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:50:07.458 DEBUG:boto:path=/vclfcn-63/obj3 2024-02-01T10:50:07.458 DEBUG:boto:auth_path=/vclfcn-63/obj3 2024-02-01T10:50:07.458 DEBUG:boto:path=/vclfcn-63/obj3?versionId=null 2024-02-01T10:50:07.458 DEBUG:boto:auth_path=/vclfcn-63/obj3?versionId=null 2024-02-01T10:50:07.458 DEBUG:boto:Method: GET 2024-02-01T10:50:07.458 DEBUG:boto:Path: /vclfcn-63/obj3?versionId=null 2024-02-01T10:50:07.458 DEBUG:boto:Data: 2024-02-01T10:50:07.458 DEBUG:boto:Headers: {} 2024-02-01T10:50:07.459 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:07.459 DEBUG:boto:Port: 8001 2024-02-01T10:50:07.459 DEBUG:boto:Params: {} 2024-02-01T10:50:07.459 DEBUG:boto:Token: None 2024-02-01T10:50:07.459 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:07 GMT /vclfcn-63/obj3?versionId=null 2024-02-01T10:50:07.459 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sQ+y4BxlIR4psecKsfhKA+GtIOQ= 2024-02-01T10:50:07.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sQ+y4BxlIR4psecKsfhKA+GtIOQ=', 'Content-Length': '0'} 2024-02-01T10:50:07.462 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005ac59c7a5c334e9d-0065bb775f-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:07.508 DEBUG:boto:path=/vclfcn-63/obj3 2024-02-01T10:50:07.508 DEBUG:boto:auth_path=/vclfcn-63/obj3 2024-02-01T10:50:07.508 DEBUG:boto:path=/vclfcn-63/obj3?versionId=null 2024-02-01T10:50:07.508 DEBUG:boto:auth_path=/vclfcn-63/obj3?versionId=null 2024-02-01T10:50:07.508 DEBUG:boto:Method: GET 2024-02-01T10:50:07.509 DEBUG:boto:Path: /vclfcn-63/obj3?versionId=null 2024-02-01T10:50:07.509 DEBUG:boto:Data: 2024-02-01T10:50:07.509 DEBUG:boto:Headers: {} 2024-02-01T10:50:07.509 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:07.509 DEBUG:boto:Port: 8000 2024-02-01T10:50:07.509 DEBUG:boto:Params: {} 2024-02-01T10:50:07.509 DEBUG:boto:Token: None 2024-02-01T10:50:07.509 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:07 GMT /vclfcn-63/obj3?versionId=null 2024-02-01T10:50:07.509 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sQ+y4BxlIR4psecKsfhKA+GtIOQ= 2024-02-01T10:50:07.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sQ+y4BxlIR4psecKsfhKA+GtIOQ=', 'Content-Length': '0'} 2024-02-01T10:50:07.512 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e4ece33b64c49d01-0065bb775f-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:07.513 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:07.513 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:07.513 DEBUG:boto:path=/vclfcn-63/?prefix=obj3 2024-02-01T10:50:07.513 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj3 2024-02-01T10:50:07.513 DEBUG:boto:Method: GET 2024-02-01T10:50:07.513 DEBUG:boto:Path: /vclfcn-63/?prefix=obj3 2024-02-01T10:50:07.513 DEBUG:boto:Data: 2024-02-01T10:50:07.513 DEBUG:boto:Headers: {} 2024-02-01T10:50:07.514 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:07.514 DEBUG:boto:Port: 8001 2024-02-01T10:50:07.514 DEBUG:boto:Params: {} 2024-02-01T10:50:07.514 DEBUG:boto:Token: None 2024-02-01T10:50:07.514 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:07 GMT /vclfcn-63/ 2024-02-01T10:50:07.514 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:i9iEv8SZmFIq7dzLkaM60uCok+4= 2024-02-01T10:50:07.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:i9iEv8SZmFIq7dzLkaM60uCok+4=', 'Content-Length': '0'} 2024-02-01T10:50:07.519 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f4c6c1847687c65-0065bb775f-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:07.560 DEBUG:boto:b'vclfcn-63obj31000falseobj32024-02-01T10:48:26.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:07.561 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:07.561 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:07.561 DEBUG:boto:path=/vclfcn-63/?prefix=obj3 2024-02-01T10:50:07.561 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj3 2024-02-01T10:50:07.561 DEBUG:boto:Method: GET 2024-02-01T10:50:07.561 DEBUG:boto:Path: /vclfcn-63/?prefix=obj3 2024-02-01T10:50:07.561 DEBUG:boto:Data: 2024-02-01T10:50:07.561 DEBUG:boto:Headers: {} 2024-02-01T10:50:07.561 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:07.561 DEBUG:boto:Port: 8000 2024-02-01T10:50:07.561 DEBUG:boto:Params: {} 2024-02-01T10:50:07.562 DEBUG:boto:Token: None 2024-02-01T10:50:07.562 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:07 GMT /vclfcn-63/ 2024-02-01T10:50:07.562 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:i9iEv8SZmFIq7dzLkaM60uCok+4= 2024-02-01T10:50:07.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:i9iEv8SZmFIq7dzLkaM60uCok+4=', 'Content-Length': '0'} 2024-02-01T10:50:07.566 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4f27027fd7c3e11-0065bb775f-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:07.566 DEBUG:boto:b'vclfcn-63obj31000falseobj32024-02-01T10:48:26.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:07.567 DEBUG:boto:path=/vclfcn-63/obj3 2024-02-01T10:50:07.567 DEBUG:boto:auth_path=/vclfcn-63/obj3 2024-02-01T10:50:07.567 DEBUG:boto:Method: HEAD 2024-02-01T10:50:07.567 DEBUG:boto:Path: /vclfcn-63/obj3 2024-02-01T10:50:07.567 DEBUG:boto:Data: 2024-02-01T10:50:07.567 DEBUG:boto:Headers: {} 2024-02-01T10:50:07.567 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:07.568 DEBUG:boto:Port: 8001 2024-02-01T10:50:07.568 DEBUG:boto:Params: {} 2024-02-01T10:50:07.568 DEBUG:boto:Token: None 2024-02-01T10:50:07.568 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:07 GMT /vclfcn-63/obj3 2024-02-01T10:50:07.568 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wNxtz5JgLJ2uRXuqQ2QT+MOcysM= 2024-02-01T10:50:07.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wNxtz5JgLJ2uRXuqQ2QT+MOcysM=', 'Content-Length': '0'} 2024-02-01T10:50:07.571 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000019af9e254ab14356-0065bb775f-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:07.571 DEBUG:boto:path=/vclfcn-63/obj3 2024-02-01T10:50:07.571 DEBUG:boto:auth_path=/vclfcn-63/obj3 2024-02-01T10:50:07.572 DEBUG:boto:Method: HEAD 2024-02-01T10:50:07.572 DEBUG:boto:Path: /vclfcn-63/obj3 2024-02-01T10:50:07.572 DEBUG:boto:Data: 2024-02-01T10:50:07.572 DEBUG:boto:Headers: {} 2024-02-01T10:50:07.572 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:07.572 DEBUG:boto:Port: 8000 2024-02-01T10:50:07.572 DEBUG:boto:Params: {} 2024-02-01T10:50:07.572 DEBUG:boto:Token: None 2024-02-01T10:50:07.572 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:07 GMT /vclfcn-63/obj3 2024-02-01T10:50:07.572 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wNxtz5JgLJ2uRXuqQ2QT+MOcysM= 2024-02-01T10:50:07.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wNxtz5JgLJ2uRXuqQ2QT+MOcysM=', 'Content-Length': '0'} 2024-02-01T10:50:07.575 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000108b175b7528e591-0065bb775f-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:07.575 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:50:07.576 DEBUG:boto:path=/vclfcn-63/obj3 2024-02-01T10:50:07.576 DEBUG:boto:auth_path=/vclfcn-63/obj3 2024-02-01T10:50:07.576 DEBUG:boto:Method: GET 2024-02-01T10:50:07.576 DEBUG:boto:Path: /vclfcn-63/obj3 2024-02-01T10:50:07.576 DEBUG:boto:Data: 2024-02-01T10:50:07.576 DEBUG:boto:Headers: {} 2024-02-01T10:50:07.576 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:07.576 DEBUG:boto:Port: 8001 2024-02-01T10:50:07.576 DEBUG:boto:Params: {} 2024-02-01T10:50:07.576 DEBUG:boto:Token: None 2024-02-01T10:50:07.577 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:07 GMT /vclfcn-63/obj3 2024-02-01T10:50:07.577 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:iHaf9IVRAKdkIWY6CTA1teZk0zg= 2024-02-01T10:50:07.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iHaf9IVRAKdkIWY6CTA1teZk0zg=', 'Content-Length': '0'} 2024-02-01T10:50:07.580 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002948e54dbc699e8b-0065bb775f-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:07.624 DEBUG:boto:path=/vclfcn-63/obj3 2024-02-01T10:50:07.624 DEBUG:boto:auth_path=/vclfcn-63/obj3 2024-02-01T10:50:07.624 DEBUG:boto:Method: GET 2024-02-01T10:50:07.624 DEBUG:boto:Path: /vclfcn-63/obj3 2024-02-01T10:50:07.625 DEBUG:boto:Data: 2024-02-01T10:50:07.625 DEBUG:boto:Headers: {} 2024-02-01T10:50:07.625 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:07.625 DEBUG:boto:Port: 8000 2024-02-01T10:50:07.625 DEBUG:boto:Params: {} 2024-02-01T10:50:07.625 DEBUG:boto:Token: None 2024-02-01T10:50:07.625 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:07 GMT /vclfcn-63/obj3 2024-02-01T10:50:07.625 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:iHaf9IVRAKdkIWY6CTA1teZk0zg= 2024-02-01T10:50:07.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iHaf9IVRAKdkIWY6CTA1teZk0zg=', 'Content-Length': '0'} 2024-02-01T10:50:07.628 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000024f1ea7978ea9cd0-0065bb775f-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:07.629 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:07.629 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:07.629 DEBUG:boto:path=/vclfcn-63/?prefix=obj3 2024-02-01T10:50:07.629 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj3 2024-02-01T10:50:07.629 DEBUG:boto:Method: GET 2024-02-01T10:50:07.629 DEBUG:boto:Path: /vclfcn-63/?prefix=obj3 2024-02-01T10:50:07.629 DEBUG:boto:Data: 2024-02-01T10:50:07.629 DEBUG:boto:Headers: {} 2024-02-01T10:50:07.630 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:07.630 DEBUG:boto:Port: 8001 2024-02-01T10:50:07.630 DEBUG:boto:Params: {} 2024-02-01T10:50:07.630 DEBUG:boto:Token: None 2024-02-01T10:50:07.630 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:07 GMT /vclfcn-63/ 2024-02-01T10:50:07.630 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:i9iEv8SZmFIq7dzLkaM60uCok+4= 2024-02-01T10:50:07.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:i9iEv8SZmFIq7dzLkaM60uCok+4=', 'Content-Length': '0'} 2024-02-01T10:50:07.634 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb11c41c2a811968-0065bb775f-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:07.676 DEBUG:boto:b'vclfcn-63obj31000falseobj32024-02-01T10:48:26.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:07.676 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:07.677 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:07.677 DEBUG:boto:path=/vclfcn-63/?prefix=obj3 2024-02-01T10:50:07.677 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj3 2024-02-01T10:50:07.677 DEBUG:boto:Method: GET 2024-02-01T10:50:07.677 DEBUG:boto:Path: /vclfcn-63/?prefix=obj3 2024-02-01T10:50:07.677 DEBUG:boto:Data: 2024-02-01T10:50:07.677 DEBUG:boto:Headers: {} 2024-02-01T10:50:07.677 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:07.677 DEBUG:boto:Port: 8000 2024-02-01T10:50:07.677 DEBUG:boto:Params: {} 2024-02-01T10:50:07.678 DEBUG:boto:Token: None 2024-02-01T10:50:07.678 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:07 GMT /vclfcn-63/ 2024-02-01T10:50:07.678 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:i9iEv8SZmFIq7dzLkaM60uCok+4= 2024-02-01T10:50:07.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:i9iEv8SZmFIq7dzLkaM60uCok+4=', 'Content-Length': '0'} 2024-02-01T10:50:07.683 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0a17d6c31e04e2d-0065bb775f-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:07.683 DEBUG:boto:b'vclfcn-63obj31000falseobj32024-02-01T10:48:26.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:07.683 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:50:07.683 DEBUG:boto:path=/vclfcn-63/obj4 2024-02-01T10:50:07.683 DEBUG:boto:auth_path=/vclfcn-63/obj4 2024-02-01T10:50:07.684 DEBUG:boto:path=/vclfcn-63/obj4?versionId=null 2024-02-01T10:50:07.684 DEBUG:boto:auth_path=/vclfcn-63/obj4?versionId=null 2024-02-01T10:50:07.684 DEBUG:boto:Method: GET 2024-02-01T10:50:07.684 DEBUG:boto:Path: /vclfcn-63/obj4?versionId=null 2024-02-01T10:50:07.684 DEBUG:boto:Data: 2024-02-01T10:50:07.684 DEBUG:boto:Headers: {} 2024-02-01T10:50:07.684 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:07.684 DEBUG:boto:Port: 8001 2024-02-01T10:50:07.684 DEBUG:boto:Params: {} 2024-02-01T10:50:07.684 DEBUG:boto:Token: None 2024-02-01T10:50:07.685 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:07 GMT /vclfcn-63/obj4?versionId=null 2024-02-01T10:50:07.685 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:SOe7kO6AlyxRv4/j6QbyHI15kc8= 2024-02-01T10:50:07.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:SOe7kO6AlyxRv4/j6QbyHI15kc8=', 'Content-Length': '0'} 2024-02-01T10:50:07.688 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000093b9c81ac51c0be0-0065bb775f-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:07.732 DEBUG:boto:path=/vclfcn-63/obj4 2024-02-01T10:50:07.732 DEBUG:boto:auth_path=/vclfcn-63/obj4 2024-02-01T10:50:07.732 DEBUG:boto:path=/vclfcn-63/obj4?versionId=null 2024-02-01T10:50:07.732 DEBUG:boto:auth_path=/vclfcn-63/obj4?versionId=null 2024-02-01T10:50:07.732 DEBUG:boto:Method: GET 2024-02-01T10:50:07.732 DEBUG:boto:Path: /vclfcn-63/obj4?versionId=null 2024-02-01T10:50:07.733 DEBUG:boto:Data: 2024-02-01T10:50:07.733 DEBUG:boto:Headers: {} 2024-02-01T10:50:07.733 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:07.733 DEBUG:boto:Port: 8000 2024-02-01T10:50:07.733 DEBUG:boto:Params: {} 2024-02-01T10:50:07.733 DEBUG:boto:Token: None 2024-02-01T10:50:07.733 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:07 GMT /vclfcn-63/obj4?versionId=null 2024-02-01T10:50:07.733 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:SOe7kO6AlyxRv4/j6QbyHI15kc8= 2024-02-01T10:50:07.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:SOe7kO6AlyxRv4/j6QbyHI15kc8=', 'Content-Length': '0'} 2024-02-01T10:50:07.737 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e56248429966a825-0065bb775f-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:07.737 DEBUG:boto:path=/vclfcn-63/obj4 2024-02-01T10:50:07.737 DEBUG:boto:auth_path=/vclfcn-63/obj4 2024-02-01T10:50:07.737 DEBUG:boto:path=/vclfcn-63/obj4?versionId=null 2024-02-01T10:50:07.738 DEBUG:boto:auth_path=/vclfcn-63/obj4?versionId=null 2024-02-01T10:50:07.738 DEBUG:boto:Method: HEAD 2024-02-01T10:50:07.738 DEBUG:boto:Path: /vclfcn-63/obj4?versionId=null 2024-02-01T10:50:07.738 DEBUG:boto:Data: 2024-02-01T10:50:07.738 DEBUG:boto:Headers: {} 2024-02-01T10:50:07.738 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:07.738 DEBUG:boto:Port: 8001 2024-02-01T10:50:07.738 DEBUG:boto:Params: {} 2024-02-01T10:50:07.738 DEBUG:boto:Token: None 2024-02-01T10:50:07.738 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:07 GMT /vclfcn-63/obj4?versionId=null 2024-02-01T10:50:07.739 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:M5lst/+S5jkPCE3ZqXQTFT9yRO0= 2024-02-01T10:50:07.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:M5lst/+S5jkPCE3ZqXQTFT9yRO0=', 'Content-Length': '0'} 2024-02-01T10:50:07.741 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dc70e61bfe664ab2-0065bb775f-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:07.742 DEBUG:boto:path=/vclfcn-63/obj4 2024-02-01T10:50:07.742 DEBUG:boto:auth_path=/vclfcn-63/obj4 2024-02-01T10:50:07.742 DEBUG:boto:path=/vclfcn-63/obj4?versionId=null 2024-02-01T10:50:07.742 DEBUG:boto:auth_path=/vclfcn-63/obj4?versionId=null 2024-02-01T10:50:07.742 DEBUG:boto:Method: HEAD 2024-02-01T10:50:07.742 DEBUG:boto:Path: /vclfcn-63/obj4?versionId=null 2024-02-01T10:50:07.742 DEBUG:boto:Data: 2024-02-01T10:50:07.742 DEBUG:boto:Headers: {} 2024-02-01T10:50:07.742 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:07.742 DEBUG:boto:Port: 8000 2024-02-01T10:50:07.742 DEBUG:boto:Params: {} 2024-02-01T10:50:07.743 DEBUG:boto:Token: None 2024-02-01T10:50:07.743 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:07 GMT /vclfcn-63/obj4?versionId=null 2024-02-01T10:50:07.743 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:M5lst/+S5jkPCE3ZqXQTFT9yRO0= 2024-02-01T10:50:07.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:M5lst/+S5jkPCE3ZqXQTFT9yRO0=', 'Content-Length': '0'} 2024-02-01T10:50:07.745 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d52223e1143ded25-0065bb775f-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:07.746 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:50:07.746 DEBUG:boto:path=/vclfcn-63/obj4 2024-02-01T10:50:07.746 DEBUG:boto:auth_path=/vclfcn-63/obj4 2024-02-01T10:50:07.746 DEBUG:boto:path=/vclfcn-63/obj4?versionId=null 2024-02-01T10:50:07.746 DEBUG:boto:auth_path=/vclfcn-63/obj4?versionId=null 2024-02-01T10:50:07.746 DEBUG:boto:Method: GET 2024-02-01T10:50:07.746 DEBUG:boto:Path: /vclfcn-63/obj4?versionId=null 2024-02-01T10:50:07.747 DEBUG:boto:Data: 2024-02-01T10:50:07.747 DEBUG:boto:Headers: {} 2024-02-01T10:50:07.747 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:07.747 DEBUG:boto:Port: 8001 2024-02-01T10:50:07.747 DEBUG:boto:Params: {} 2024-02-01T10:50:07.747 DEBUG:boto:Token: None 2024-02-01T10:50:07.747 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:07 GMT /vclfcn-63/obj4?versionId=null 2024-02-01T10:50:07.747 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:SOe7kO6AlyxRv4/j6QbyHI15kc8= 2024-02-01T10:50:07.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:SOe7kO6AlyxRv4/j6QbyHI15kc8=', 'Content-Length': '0'} 2024-02-01T10:50:07.750 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000033145a94ae3ce41d-0065bb775f-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:07.796 DEBUG:boto:path=/vclfcn-63/obj4 2024-02-01T10:50:07.797 DEBUG:boto:auth_path=/vclfcn-63/obj4 2024-02-01T10:50:07.797 DEBUG:boto:path=/vclfcn-63/obj4?versionId=null 2024-02-01T10:50:07.797 DEBUG:boto:auth_path=/vclfcn-63/obj4?versionId=null 2024-02-01T10:50:07.797 DEBUG:boto:Method: GET 2024-02-01T10:50:07.797 DEBUG:boto:Path: /vclfcn-63/obj4?versionId=null 2024-02-01T10:50:07.797 DEBUG:boto:Data: 2024-02-01T10:50:07.797 DEBUG:boto:Headers: {} 2024-02-01T10:50:07.797 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:07.797 DEBUG:boto:Port: 8000 2024-02-01T10:50:07.797 DEBUG:boto:Params: {} 2024-02-01T10:50:07.798 DEBUG:boto:Token: None 2024-02-01T10:50:07.798 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:07 GMT /vclfcn-63/obj4?versionId=null 2024-02-01T10:50:07.798 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:SOe7kO6AlyxRv4/j6QbyHI15kc8= 2024-02-01T10:50:07.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:SOe7kO6AlyxRv4/j6QbyHI15kc8=', 'Content-Length': '0'} 2024-02-01T10:50:07.801 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dd162dfec1f63bfa-0065bb775f-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:07.802 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:07.802 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:07.802 DEBUG:boto:path=/vclfcn-63/?prefix=obj4 2024-02-01T10:50:07.802 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj4 2024-02-01T10:50:07.802 DEBUG:boto:Method: GET 2024-02-01T10:50:07.802 DEBUG:boto:Path: /vclfcn-63/?prefix=obj4 2024-02-01T10:50:07.802 DEBUG:boto:Data: 2024-02-01T10:50:07.802 DEBUG:boto:Headers: {} 2024-02-01T10:50:07.802 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:07.803 DEBUG:boto:Port: 8001 2024-02-01T10:50:07.803 DEBUG:boto:Params: {} 2024-02-01T10:50:07.803 DEBUG:boto:Token: None 2024-02-01T10:50:07.803 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:07 GMT /vclfcn-63/ 2024-02-01T10:50:07.803 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:i9iEv8SZmFIq7dzLkaM60uCok+4= 2024-02-01T10:50:07.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:i9iEv8SZmFIq7dzLkaM60uCok+4=', 'Content-Length': '0'} 2024-02-01T10:50:07.808 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003391fb141bae87f9-0065bb775f-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:07.848 DEBUG:boto:b'vclfcn-63obj41000falseobj42024-02-01T10:48:26.575Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:07.849 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:07.849 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:07.849 DEBUG:boto:path=/vclfcn-63/?prefix=obj4 2024-02-01T10:50:07.849 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj4 2024-02-01T10:50:07.849 DEBUG:boto:Method: GET 2024-02-01T10:50:07.849 DEBUG:boto:Path: /vclfcn-63/?prefix=obj4 2024-02-01T10:50:07.849 DEBUG:boto:Data: 2024-02-01T10:50:07.849 DEBUG:boto:Headers: {} 2024-02-01T10:50:07.849 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:07.849 DEBUG:boto:Port: 8000 2024-02-01T10:50:07.849 DEBUG:boto:Params: {} 2024-02-01T10:50:07.850 DEBUG:boto:Token: None 2024-02-01T10:50:07.850 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:07 GMT /vclfcn-63/ 2024-02-01T10:50:07.850 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:i9iEv8SZmFIq7dzLkaM60uCok+4= 2024-02-01T10:50:07.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:i9iEv8SZmFIq7dzLkaM60uCok+4=', 'Content-Length': '0'} 2024-02-01T10:50:07.854 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000153b78b391f515d0-0065bb775f-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:07.855 DEBUG:boto:b'vclfcn-63obj41000falseobj42024-02-01T10:48:26.575Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:07.855 DEBUG:boto:path=/vclfcn-63/obj4 2024-02-01T10:50:07.855 DEBUG:boto:auth_path=/vclfcn-63/obj4 2024-02-01T10:50:07.855 DEBUG:boto:Method: HEAD 2024-02-01T10:50:07.855 DEBUG:boto:Path: /vclfcn-63/obj4 2024-02-01T10:50:07.856 DEBUG:boto:Data: 2024-02-01T10:50:07.856 DEBUG:boto:Headers: {} 2024-02-01T10:50:07.856 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:07.856 DEBUG:boto:Port: 8001 2024-02-01T10:50:07.856 DEBUG:boto:Params: {} 2024-02-01T10:50:07.856 DEBUG:boto:Token: None 2024-02-01T10:50:07.856 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:07 GMT /vclfcn-63/obj4 2024-02-01T10:50:07.856 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:JRtO7Rox8Ewm8t2L5Y/4GHHP8bk= 2024-02-01T10:50:07.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:JRtO7Rox8Ewm8t2L5Y/4GHHP8bk=', 'Content-Length': '0'} 2024-02-01T10:50:07.859 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008633c44a486af096-0065bb775f-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:07.859 DEBUG:boto:path=/vclfcn-63/obj4 2024-02-01T10:50:07.859 DEBUG:boto:auth_path=/vclfcn-63/obj4 2024-02-01T10:50:07.860 DEBUG:boto:Method: HEAD 2024-02-01T10:50:07.860 DEBUG:boto:Path: /vclfcn-63/obj4 2024-02-01T10:50:07.860 DEBUG:boto:Data: 2024-02-01T10:50:07.860 DEBUG:boto:Headers: {} 2024-02-01T10:50:07.860 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:07.861 DEBUG:boto:Port: 8000 2024-02-01T10:50:07.861 DEBUG:boto:Params: {} 2024-02-01T10:50:07.861 DEBUG:boto:Token: None 2024-02-01T10:50:07.861 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:07 GMT /vclfcn-63/obj4 2024-02-01T10:50:07.861 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:JRtO7Rox8Ewm8t2L5Y/4GHHP8bk= 2024-02-01T10:50:07.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:JRtO7Rox8Ewm8t2L5Y/4GHHP8bk=', 'Content-Length': '0'} 2024-02-01T10:50:07.864 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b56f523b6944fefe-0065bb775f-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:07.864 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:50:07.864 DEBUG:boto:path=/vclfcn-63/obj4 2024-02-01T10:50:07.865 DEBUG:boto:auth_path=/vclfcn-63/obj4 2024-02-01T10:50:07.865 DEBUG:boto:Method: GET 2024-02-01T10:50:07.865 DEBUG:boto:Path: /vclfcn-63/obj4 2024-02-01T10:50:07.865 DEBUG:boto:Data: 2024-02-01T10:50:07.865 DEBUG:boto:Headers: {} 2024-02-01T10:50:07.865 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:07.866 DEBUG:boto:Port: 8001 2024-02-01T10:50:07.866 DEBUG:boto:Params: {} 2024-02-01T10:50:07.866 DEBUG:boto:Token: None 2024-02-01T10:50:07.866 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:07 GMT /vclfcn-63/obj4 2024-02-01T10:50:07.866 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:s4MQVbK9ELwdmO0fAhStgNr+Lms= 2024-02-01T10:50:07.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:s4MQVbK9ELwdmO0fAhStgNr+Lms=', 'Content-Length': '0'} 2024-02-01T10:50:07.869 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a0a465e60facd873-0065bb775f-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:07.912 DEBUG:boto:path=/vclfcn-63/obj4 2024-02-01T10:50:07.912 DEBUG:boto:auth_path=/vclfcn-63/obj4 2024-02-01T10:50:07.912 DEBUG:boto:Method: GET 2024-02-01T10:50:07.913 DEBUG:boto:Path: /vclfcn-63/obj4 2024-02-01T10:50:07.913 DEBUG:boto:Data: 2024-02-01T10:50:07.913 DEBUG:boto:Headers: {} 2024-02-01T10:50:07.913 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:07.913 DEBUG:boto:Port: 8000 2024-02-01T10:50:07.913 DEBUG:boto:Params: {} 2024-02-01T10:50:07.913 DEBUG:boto:Token: None 2024-02-01T10:50:07.913 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:07 GMT /vclfcn-63/obj4 2024-02-01T10:50:07.914 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:s4MQVbK9ELwdmO0fAhStgNr+Lms= 2024-02-01T10:50:07.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:s4MQVbK9ELwdmO0fAhStgNr+Lms=', 'Content-Length': '0'} 2024-02-01T10:50:07.916 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ad54836559adaad2-0065bb775f-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:07.917 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:07.917 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:07.917 DEBUG:boto:path=/vclfcn-63/?prefix=obj4 2024-02-01T10:50:07.917 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj4 2024-02-01T10:50:07.917 DEBUG:boto:Method: GET 2024-02-01T10:50:07.917 DEBUG:boto:Path: /vclfcn-63/?prefix=obj4 2024-02-01T10:50:07.918 DEBUG:boto:Data: 2024-02-01T10:50:07.918 DEBUG:boto:Headers: {} 2024-02-01T10:50:07.918 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:07.918 DEBUG:boto:Port: 8001 2024-02-01T10:50:07.918 DEBUG:boto:Params: {} 2024-02-01T10:50:07.918 DEBUG:boto:Token: None 2024-02-01T10:50:07.918 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:07 GMT /vclfcn-63/ 2024-02-01T10:50:07.918 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:i9iEv8SZmFIq7dzLkaM60uCok+4= 2024-02-01T10:50:07.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:i9iEv8SZmFIq7dzLkaM60uCok+4=', 'Content-Length': '0'} 2024-02-01T10:50:07.923 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009651aa70bdd57d65-0065bb775f-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:07.923 DEBUG:boto:b'vclfcn-63obj41000falseobj42024-02-01T10:48:26.575Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:07.923 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:07.923 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:07.923 DEBUG:boto:path=/vclfcn-63/?prefix=obj4 2024-02-01T10:50:07.924 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj4 2024-02-01T10:50:07.924 DEBUG:boto:Method: GET 2024-02-01T10:50:07.924 DEBUG:boto:Path: /vclfcn-63/?prefix=obj4 2024-02-01T10:50:07.924 DEBUG:boto:Data: 2024-02-01T10:50:07.924 DEBUG:boto:Headers: {} 2024-02-01T10:50:07.924 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:07.924 DEBUG:boto:Port: 8000 2024-02-01T10:50:07.924 DEBUG:boto:Params: {} 2024-02-01T10:50:07.924 DEBUG:boto:Token: None 2024-02-01T10:50:07.924 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:07 GMT /vclfcn-63/ 2024-02-01T10:50:07.925 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:i9iEv8SZmFIq7dzLkaM60uCok+4= 2024-02-01T10:50:07.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:i9iEv8SZmFIq7dzLkaM60uCok+4=', 'Content-Length': '0'} 2024-02-01T10:50:07.928 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c1b115a99b60aca-0065bb775f-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:07.929 DEBUG:boto:b'vclfcn-63obj41000falseobj42024-02-01T10:48:26.575Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:07.929 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-02-01T10:50:07.929 DEBUG:boto:path=/vclfcn-63/obj5 2024-02-01T10:50:07.929 DEBUG:boto:auth_path=/vclfcn-63/obj5 2024-02-01T10:50:07.929 DEBUG:boto:path=/vclfcn-63/obj5?versionId=null 2024-02-01T10:50:07.930 DEBUG:boto:auth_path=/vclfcn-63/obj5?versionId=null 2024-02-01T10:50:07.930 DEBUG:boto:Method: GET 2024-02-01T10:50:07.930 DEBUG:boto:Path: /vclfcn-63/obj5?versionId=null 2024-02-01T10:50:07.930 DEBUG:boto:Data: 2024-02-01T10:50:07.930 DEBUG:boto:Headers: {} 2024-02-01T10:50:07.930 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:07.930 DEBUG:boto:Port: 8001 2024-02-01T10:50:07.930 DEBUG:boto:Params: {} 2024-02-01T10:50:07.930 DEBUG:boto:Token: None 2024-02-01T10:50:07.930 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:07 GMT /vclfcn-63/obj5?versionId=null 2024-02-01T10:50:07.931 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gtdIw1KoEd7vDQaHLrFli/qG4fg= 2024-02-01T10:50:07.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gtdIw1KoEd7vDQaHLrFli/qG4fg=', 'Content-Length': '0'} 2024-02-01T10:50:07.934 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dd99344cf40ff90c-0065bb775f-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:07.976 DEBUG:boto:path=/vclfcn-63/obj5 2024-02-01T10:50:07.976 DEBUG:boto:auth_path=/vclfcn-63/obj5 2024-02-01T10:50:07.976 DEBUG:boto:path=/vclfcn-63/obj5?versionId=null 2024-02-01T10:50:07.976 DEBUG:boto:auth_path=/vclfcn-63/obj5?versionId=null 2024-02-01T10:50:07.976 DEBUG:boto:Method: GET 2024-02-01T10:50:07.977 DEBUG:boto:Path: /vclfcn-63/obj5?versionId=null 2024-02-01T10:50:07.977 DEBUG:boto:Data: 2024-02-01T10:50:07.977 DEBUG:boto:Headers: {} 2024-02-01T10:50:07.977 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:07.977 DEBUG:boto:Port: 8000 2024-02-01T10:50:07.977 DEBUG:boto:Params: {} 2024-02-01T10:50:07.977 DEBUG:boto:Token: None 2024-02-01T10:50:07.977 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:07 GMT /vclfcn-63/obj5?versionId=null 2024-02-01T10:50:07.977 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gtdIw1KoEd7vDQaHLrFli/qG4fg= 2024-02-01T10:50:07.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gtdIw1KoEd7vDQaHLrFli/qG4fg=', 'Content-Length': '0'} 2024-02-01T10:50:07.980 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000179404c90f3c96ef-0065bb775f-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:07.980 DEBUG:boto:path=/vclfcn-63/obj5 2024-02-01T10:50:07.981 DEBUG:boto:auth_path=/vclfcn-63/obj5 2024-02-01T10:50:07.981 DEBUG:boto:path=/vclfcn-63/obj5?versionId=null 2024-02-01T10:50:07.981 DEBUG:boto:auth_path=/vclfcn-63/obj5?versionId=null 2024-02-01T10:50:07.981 DEBUG:boto:Method: HEAD 2024-02-01T10:50:07.981 DEBUG:boto:Path: /vclfcn-63/obj5?versionId=null 2024-02-01T10:50:07.981 DEBUG:boto:Data: 2024-02-01T10:50:07.981 DEBUG:boto:Headers: {} 2024-02-01T10:50:07.981 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:07.981 DEBUG:boto:Port: 8001 2024-02-01T10:50:07.981 DEBUG:boto:Params: {} 2024-02-01T10:50:07.981 DEBUG:boto:Token: None 2024-02-01T10:50:07.982 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:07 GMT /vclfcn-63/obj5?versionId=null 2024-02-01T10:50:07.982 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Z6G63mbQ51mAXwXSOy5i+AhSOt4= 2024-02-01T10:50:07.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Z6G63mbQ51mAXwXSOy5i+AhSOt4=', 'Content-Length': '0'} 2024-02-01T10:50:07.985 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008572a2f7bc8a1677-0065bb775f-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:07.985 DEBUG:boto:path=/vclfcn-63/obj5 2024-02-01T10:50:07.985 DEBUG:boto:auth_path=/vclfcn-63/obj5 2024-02-01T10:50:07.985 DEBUG:boto:path=/vclfcn-63/obj5?versionId=null 2024-02-01T10:50:07.985 DEBUG:boto:auth_path=/vclfcn-63/obj5?versionId=null 2024-02-01T10:50:07.985 DEBUG:boto:Method: HEAD 2024-02-01T10:50:07.985 DEBUG:boto:Path: /vclfcn-63/obj5?versionId=null 2024-02-01T10:50:07.986 DEBUG:boto:Data: 2024-02-01T10:50:07.986 DEBUG:boto:Headers: {} 2024-02-01T10:50:07.986 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:07.986 DEBUG:boto:Port: 8000 2024-02-01T10:50:07.986 DEBUG:boto:Params: {} 2024-02-01T10:50:07.986 DEBUG:boto:Token: None 2024-02-01T10:50:07.986 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:07 GMT /vclfcn-63/obj5?versionId=null 2024-02-01T10:50:07.986 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Z6G63mbQ51mAXwXSOy5i+AhSOt4= 2024-02-01T10:50:07.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Z6G63mbQ51mAXwXSOy5i+AhSOt4=', 'Content-Length': '0'} 2024-02-01T10:50:07.989 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000056475308701cb90d-0065bb775f-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:07.989 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-02-01T10:50:07.989 DEBUG:boto:path=/vclfcn-63/obj5 2024-02-01T10:50:07.989 DEBUG:boto:auth_path=/vclfcn-63/obj5 2024-02-01T10:50:07.989 DEBUG:boto:path=/vclfcn-63/obj5?versionId=null 2024-02-01T10:50:07.990 DEBUG:boto:auth_path=/vclfcn-63/obj5?versionId=null 2024-02-01T10:50:07.990 DEBUG:boto:Method: GET 2024-02-01T10:50:07.990 DEBUG:boto:Path: /vclfcn-63/obj5?versionId=null 2024-02-01T10:50:07.990 DEBUG:boto:Data: 2024-02-01T10:50:07.990 DEBUG:boto:Headers: {} 2024-02-01T10:50:07.990 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:07.990 DEBUG:boto:Port: 8001 2024-02-01T10:50:07.990 DEBUG:boto:Params: {} 2024-02-01T10:50:07.990 DEBUG:boto:Token: None 2024-02-01T10:50:07.990 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:07 GMT /vclfcn-63/obj5?versionId=null 2024-02-01T10:50:07.991 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gtdIw1KoEd7vDQaHLrFli/qG4fg= 2024-02-01T10:50:07.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:07 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gtdIw1KoEd7vDQaHLrFli/qG4fg=', 'Content-Length': '0'} 2024-02-01T10:50:07.993 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004be1ccb0fb051298-0065bb775f-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:07 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:08.036 DEBUG:boto:path=/vclfcn-63/obj5 2024-02-01T10:50:08.036 DEBUG:boto:auth_path=/vclfcn-63/obj5 2024-02-01T10:50:08.036 DEBUG:boto:path=/vclfcn-63/obj5?versionId=null 2024-02-01T10:50:08.037 DEBUG:boto:auth_path=/vclfcn-63/obj5?versionId=null 2024-02-01T10:50:08.037 DEBUG:boto:Method: GET 2024-02-01T10:50:08.037 DEBUG:boto:Path: /vclfcn-63/obj5?versionId=null 2024-02-01T10:50:08.037 DEBUG:boto:Data: 2024-02-01T10:50:08.037 DEBUG:boto:Headers: {} 2024-02-01T10:50:08.037 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:08.037 DEBUG:boto:Port: 8000 2024-02-01T10:50:08.037 DEBUG:boto:Params: {} 2024-02-01T10:50:08.037 DEBUG:boto:Token: None 2024-02-01T10:50:08.038 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:08 GMT /vclfcn-63/obj5?versionId=null 2024-02-01T10:50:08.038 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:s99eiWaa1/bC5rR439AKAc01VEo= 2024-02-01T10:50:08.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:s99eiWaa1/bC5rR439AKAc01VEo=', 'Content-Length': '0'} 2024-02-01T10:50:08.041 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aeea76998a79957a-0065bb7760-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:08.041 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:08.041 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:08.041 DEBUG:boto:path=/vclfcn-63/?prefix=obj5 2024-02-01T10:50:08.041 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj5 2024-02-01T10:50:08.041 DEBUG:boto:Method: GET 2024-02-01T10:50:08.042 DEBUG:boto:Path: /vclfcn-63/?prefix=obj5 2024-02-01T10:50:08.042 DEBUG:boto:Data: 2024-02-01T10:50:08.042 DEBUG:boto:Headers: {} 2024-02-01T10:50:08.042 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:08.042 DEBUG:boto:Port: 8001 2024-02-01T10:50:08.042 DEBUG:boto:Params: {} 2024-02-01T10:50:08.042 DEBUG:boto:Token: None 2024-02-01T10:50:08.042 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:08 GMT /vclfcn-63/ 2024-02-01T10:50:08.042 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:SWr7xT6s+p02WaAfyPyEv0WQAGU= 2024-02-01T10:50:08.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:SWr7xT6s+p02WaAfyPyEv0WQAGU=', 'Content-Length': '0'} 2024-02-01T10:50:08.047 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ccbd03456752e516-0065bb7760-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:08.088 DEBUG:boto:b'vclfcn-63obj51000falseobj52024-02-01T10:49:22.255Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:08.089 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:08.089 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:08.089 DEBUG:boto:path=/vclfcn-63/?prefix=obj5 2024-02-01T10:50:08.089 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj5 2024-02-01T10:50:08.089 DEBUG:boto:Method: GET 2024-02-01T10:50:08.089 DEBUG:boto:Path: /vclfcn-63/?prefix=obj5 2024-02-01T10:50:08.089 DEBUG:boto:Data: 2024-02-01T10:50:08.089 DEBUG:boto:Headers: {} 2024-02-01T10:50:08.089 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:08.089 DEBUG:boto:Port: 8000 2024-02-01T10:50:08.090 DEBUG:boto:Params: {} 2024-02-01T10:50:08.090 DEBUG:boto:Token: None 2024-02-01T10:50:08.090 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:08 GMT /vclfcn-63/ 2024-02-01T10:50:08.090 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:SWr7xT6s+p02WaAfyPyEv0WQAGU= 2024-02-01T10:50:08.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:SWr7xT6s+p02WaAfyPyEv0WQAGU=', 'Content-Length': '0'} 2024-02-01T10:50:08.094 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000630c6ebefe456e0d-0065bb7760-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:08.094 DEBUG:boto:b'vclfcn-63obj51000falseobj52024-02-01T10:49:22.255Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:08.095 DEBUG:boto:path=/vclfcn-63/obj5 2024-02-01T10:50:08.095 DEBUG:boto:auth_path=/vclfcn-63/obj5 2024-02-01T10:50:08.095 DEBUG:boto:Method: HEAD 2024-02-01T10:50:08.095 DEBUG:boto:Path: /vclfcn-63/obj5 2024-02-01T10:50:08.095 DEBUG:boto:Data: 2024-02-01T10:50:08.095 DEBUG:boto:Headers: {} 2024-02-01T10:50:08.095 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:08.095 DEBUG:boto:Port: 8001 2024-02-01T10:50:08.095 DEBUG:boto:Params: {} 2024-02-01T10:50:08.096 DEBUG:boto:Token: None 2024-02-01T10:50:08.096 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:08 GMT /vclfcn-63/obj5 2024-02-01T10:50:08.096 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:yUYoanHdVFslpvhJSz8kHHQ8VAk= 2024-02-01T10:50:08.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:yUYoanHdVFslpvhJSz8kHHQ8VAk=', 'Content-Length': '0'} 2024-02-01T10:50:08.099 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a2a51815ff0eec61-0065bb7760-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:08.099 DEBUG:boto:path=/vclfcn-63/obj5 2024-02-01T10:50:08.099 DEBUG:boto:auth_path=/vclfcn-63/obj5 2024-02-01T10:50:08.099 DEBUG:boto:Method: HEAD 2024-02-01T10:50:08.099 DEBUG:boto:Path: /vclfcn-63/obj5 2024-02-01T10:50:08.099 DEBUG:boto:Data: 2024-02-01T10:50:08.100 DEBUG:boto:Headers: {} 2024-02-01T10:50:08.100 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:08.100 DEBUG:boto:Port: 8000 2024-02-01T10:50:08.100 DEBUG:boto:Params: {} 2024-02-01T10:50:08.100 DEBUG:boto:Token: None 2024-02-01T10:50:08.100 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:08 GMT /vclfcn-63/obj5 2024-02-01T10:50:08.100 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:yUYoanHdVFslpvhJSz8kHHQ8VAk= 2024-02-01T10:50:08.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:yUYoanHdVFslpvhJSz8kHHQ8VAk=', 'Content-Length': '0'} 2024-02-01T10:50:08.103 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000153de5ccd07df3f-0065bb7760-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:08.103 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-02-01T10:50:08.103 DEBUG:boto:path=/vclfcn-63/obj5 2024-02-01T10:50:08.103 DEBUG:boto:auth_path=/vclfcn-63/obj5 2024-02-01T10:50:08.103 DEBUG:boto:Method: GET 2024-02-01T10:50:08.104 DEBUG:boto:Path: /vclfcn-63/obj5 2024-02-01T10:50:08.104 DEBUG:boto:Data: 2024-02-01T10:50:08.104 DEBUG:boto:Headers: {} 2024-02-01T10:50:08.104 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:08.104 DEBUG:boto:Port: 8001 2024-02-01T10:50:08.104 DEBUG:boto:Params: {} 2024-02-01T10:50:08.104 DEBUG:boto:Token: None 2024-02-01T10:50:08.104 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:08 GMT /vclfcn-63/obj5 2024-02-01T10:50:08.104 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ImGuBRunVqUMCCyDKpwBBJYHrYc= 2024-02-01T10:50:08.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ImGuBRunVqUMCCyDKpwBBJYHrYc=', 'Content-Length': '0'} 2024-02-01T10:50:08.107 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009127e3945eff26dc-0065bb7760-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:08.148 DEBUG:boto:path=/vclfcn-63/obj5 2024-02-01T10:50:08.148 DEBUG:boto:auth_path=/vclfcn-63/obj5 2024-02-01T10:50:08.148 DEBUG:boto:Method: GET 2024-02-01T10:50:08.149 DEBUG:boto:Path: /vclfcn-63/obj5 2024-02-01T10:50:08.149 DEBUG:boto:Data: 2024-02-01T10:50:08.149 DEBUG:boto:Headers: {} 2024-02-01T10:50:08.149 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:08.149 DEBUG:boto:Port: 8000 2024-02-01T10:50:08.149 DEBUG:boto:Params: {} 2024-02-01T10:50:08.149 DEBUG:boto:Token: None 2024-02-01T10:50:08.149 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:08 GMT /vclfcn-63/obj5 2024-02-01T10:50:08.149 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ImGuBRunVqUMCCyDKpwBBJYHrYc= 2024-02-01T10:50:08.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ImGuBRunVqUMCCyDKpwBBJYHrYc=', 'Content-Length': '0'} 2024-02-01T10:50:08.152 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005b53f6135d328069-0065bb7760-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:08.152 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:08.153 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:08.153 DEBUG:boto:path=/vclfcn-63/?prefix=obj5 2024-02-01T10:50:08.153 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj5 2024-02-01T10:50:08.153 DEBUG:boto:Method: GET 2024-02-01T10:50:08.153 DEBUG:boto:Path: /vclfcn-63/?prefix=obj5 2024-02-01T10:50:08.153 DEBUG:boto:Data: 2024-02-01T10:50:08.153 DEBUG:boto:Headers: {} 2024-02-01T10:50:08.153 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:08.153 DEBUG:boto:Port: 8001 2024-02-01T10:50:08.153 DEBUG:boto:Params: {} 2024-02-01T10:50:08.153 DEBUG:boto:Token: None 2024-02-01T10:50:08.154 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:08 GMT /vclfcn-63/ 2024-02-01T10:50:08.154 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:SWr7xT6s+p02WaAfyPyEv0WQAGU= 2024-02-01T10:50:08.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:SWr7xT6s+p02WaAfyPyEv0WQAGU=', 'Content-Length': '0'} 2024-02-01T10:50:08.158 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000981d58a7fcdc0022-0065bb7760-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:08.200 DEBUG:boto:b'vclfcn-63obj51000falseobj52024-02-01T10:49:22.255Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:08.201 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:08.201 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:08.201 DEBUG:boto:path=/vclfcn-63/?prefix=obj5 2024-02-01T10:50:08.201 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj5 2024-02-01T10:50:08.201 DEBUG:boto:Method: GET 2024-02-01T10:50:08.201 DEBUG:boto:Path: /vclfcn-63/?prefix=obj5 2024-02-01T10:50:08.201 DEBUG:boto:Data: 2024-02-01T10:50:08.201 DEBUG:boto:Headers: {} 2024-02-01T10:50:08.202 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:08.202 DEBUG:boto:Port: 8000 2024-02-01T10:50:08.202 DEBUG:boto:Params: {} 2024-02-01T10:50:08.202 DEBUG:boto:Token: None 2024-02-01T10:50:08.202 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:08 GMT /vclfcn-63/ 2024-02-01T10:50:08.202 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:SWr7xT6s+p02WaAfyPyEv0WQAGU= 2024-02-01T10:50:08.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:SWr7xT6s+p02WaAfyPyEv0WQAGU=', 'Content-Length': '0'} 2024-02-01T10:50:08.206 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060f8feb37250e473-0065bb7760-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:08.207 DEBUG:boto:b'vclfcn-63obj51000falseobj52024-02-01T10:49:22.255Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:08.207 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-02-01T10:50:08.207 DEBUG:boto:path=/vclfcn-63/obj6 2024-02-01T10:50:08.207 DEBUG:boto:auth_path=/vclfcn-63/obj6 2024-02-01T10:50:08.207 DEBUG:boto:path=/vclfcn-63/obj6?versionId=null 2024-02-01T10:50:08.208 DEBUG:boto:auth_path=/vclfcn-63/obj6?versionId=null 2024-02-01T10:50:08.208 DEBUG:boto:Method: GET 2024-02-01T10:50:08.208 DEBUG:boto:Path: /vclfcn-63/obj6?versionId=null 2024-02-01T10:50:08.208 DEBUG:boto:Data: 2024-02-01T10:50:08.208 DEBUG:boto:Headers: {} 2024-02-01T10:50:08.208 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:08.208 DEBUG:boto:Port: 8001 2024-02-01T10:50:08.208 DEBUG:boto:Params: {} 2024-02-01T10:50:08.209 DEBUG:boto:Token: None 2024-02-01T10:50:08.209 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:08 GMT /vclfcn-63/obj6?versionId=null 2024-02-01T10:50:08.209 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:MyUQyVW8fy+usa4c5io2A4+w9nw= 2024-02-01T10:50:08.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:MyUQyVW8fy+usa4c5io2A4+w9nw=', 'Content-Length': '0'} 2024-02-01T10:50:08.212 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c7d2c67c9e8272f1-0065bb7760-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:08.252 DEBUG:boto:path=/vclfcn-63/obj6 2024-02-01T10:50:08.252 DEBUG:boto:auth_path=/vclfcn-63/obj6 2024-02-01T10:50:08.252 DEBUG:boto:path=/vclfcn-63/obj6?versionId=null 2024-02-01T10:50:08.253 DEBUG:boto:auth_path=/vclfcn-63/obj6?versionId=null 2024-02-01T10:50:08.253 DEBUG:boto:Method: GET 2024-02-01T10:50:08.253 DEBUG:boto:Path: /vclfcn-63/obj6?versionId=null 2024-02-01T10:50:08.253 DEBUG:boto:Data: 2024-02-01T10:50:08.253 DEBUG:boto:Headers: {} 2024-02-01T10:50:08.253 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:08.253 DEBUG:boto:Port: 8000 2024-02-01T10:50:08.253 DEBUG:boto:Params: {} 2024-02-01T10:50:08.253 DEBUG:boto:Token: None 2024-02-01T10:50:08.254 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:08 GMT /vclfcn-63/obj6?versionId=null 2024-02-01T10:50:08.254 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:MyUQyVW8fy+usa4c5io2A4+w9nw= 2024-02-01T10:50:08.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:MyUQyVW8fy+usa4c5io2A4+w9nw=', 'Content-Length': '0'} 2024-02-01T10:50:08.257 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000639e5adb544602ba-0065bb7760-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:08.257 DEBUG:boto:path=/vclfcn-63/obj6 2024-02-01T10:50:08.257 DEBUG:boto:auth_path=/vclfcn-63/obj6 2024-02-01T10:50:08.257 DEBUG:boto:path=/vclfcn-63/obj6?versionId=null 2024-02-01T10:50:08.257 DEBUG:boto:auth_path=/vclfcn-63/obj6?versionId=null 2024-02-01T10:50:08.257 DEBUG:boto:Method: HEAD 2024-02-01T10:50:08.257 DEBUG:boto:Path: /vclfcn-63/obj6?versionId=null 2024-02-01T10:50:08.258 DEBUG:boto:Data: 2024-02-01T10:50:08.258 DEBUG:boto:Headers: {} 2024-02-01T10:50:08.258 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:08.258 DEBUG:boto:Port: 8001 2024-02-01T10:50:08.258 DEBUG:boto:Params: {} 2024-02-01T10:50:08.258 DEBUG:boto:Token: None 2024-02-01T10:50:08.258 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:08 GMT /vclfcn-63/obj6?versionId=null 2024-02-01T10:50:08.258 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wFdDFHeql0po8bz769ysjuNPCvQ= 2024-02-01T10:50:08.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wFdDFHeql0po8bz769ysjuNPCvQ=', 'Content-Length': '0'} 2024-02-01T10:50:08.262 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000329d1c7506fad008-0065bb7760-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:08.262 DEBUG:boto:path=/vclfcn-63/obj6 2024-02-01T10:50:08.262 DEBUG:boto:auth_path=/vclfcn-63/obj6 2024-02-01T10:50:08.262 DEBUG:boto:path=/vclfcn-63/obj6?versionId=null 2024-02-01T10:50:08.262 DEBUG:boto:auth_path=/vclfcn-63/obj6?versionId=null 2024-02-01T10:50:08.263 DEBUG:boto:Method: HEAD 2024-02-01T10:50:08.263 DEBUG:boto:Path: /vclfcn-63/obj6?versionId=null 2024-02-01T10:50:08.263 DEBUG:boto:Data: 2024-02-01T10:50:08.263 DEBUG:boto:Headers: {} 2024-02-01T10:50:08.263 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:08.263 DEBUG:boto:Port: 8000 2024-02-01T10:50:08.263 DEBUG:boto:Params: {} 2024-02-01T10:50:08.263 DEBUG:boto:Token: None 2024-02-01T10:50:08.263 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:08 GMT /vclfcn-63/obj6?versionId=null 2024-02-01T10:50:08.263 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:wFdDFHeql0po8bz769ysjuNPCvQ= 2024-02-01T10:50:08.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:wFdDFHeql0po8bz769ysjuNPCvQ=', 'Content-Length': '0'} 2024-02-01T10:50:08.266 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a53da65acff51c57-0065bb7760-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:08.267 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-02-01T10:50:08.267 DEBUG:boto:path=/vclfcn-63/obj6 2024-02-01T10:50:08.267 DEBUG:boto:auth_path=/vclfcn-63/obj6 2024-02-01T10:50:08.267 DEBUG:boto:path=/vclfcn-63/obj6?versionId=null 2024-02-01T10:50:08.267 DEBUG:boto:auth_path=/vclfcn-63/obj6?versionId=null 2024-02-01T10:50:08.267 DEBUG:boto:Method: GET 2024-02-01T10:50:08.267 DEBUG:boto:Path: /vclfcn-63/obj6?versionId=null 2024-02-01T10:50:08.267 DEBUG:boto:Data: 2024-02-01T10:50:08.267 DEBUG:boto:Headers: {} 2024-02-01T10:50:08.268 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:08.268 DEBUG:boto:Port: 8001 2024-02-01T10:50:08.268 DEBUG:boto:Params: {} 2024-02-01T10:50:08.268 DEBUG:boto:Token: None 2024-02-01T10:50:08.268 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:08 GMT /vclfcn-63/obj6?versionId=null 2024-02-01T10:50:08.268 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:MyUQyVW8fy+usa4c5io2A4+w9nw= 2024-02-01T10:50:08.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:MyUQyVW8fy+usa4c5io2A4+w9nw=', 'Content-Length': '0'} 2024-02-01T10:50:08.271 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005af14e49c4d1e345-0065bb7760-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:08.312 DEBUG:boto:path=/vclfcn-63/obj6 2024-02-01T10:50:08.312 DEBUG:boto:auth_path=/vclfcn-63/obj6 2024-02-01T10:50:08.312 DEBUG:boto:path=/vclfcn-63/obj6?versionId=null 2024-02-01T10:50:08.312 DEBUG:boto:auth_path=/vclfcn-63/obj6?versionId=null 2024-02-01T10:50:08.312 DEBUG:boto:Method: GET 2024-02-01T10:50:08.313 DEBUG:boto:Path: /vclfcn-63/obj6?versionId=null 2024-02-01T10:50:08.313 DEBUG:boto:Data: 2024-02-01T10:50:08.313 DEBUG:boto:Headers: {} 2024-02-01T10:50:08.313 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:08.313 DEBUG:boto:Port: 8000 2024-02-01T10:50:08.313 DEBUG:boto:Params: {} 2024-02-01T10:50:08.313 DEBUG:boto:Token: None 2024-02-01T10:50:08.313 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:08 GMT /vclfcn-63/obj6?versionId=null 2024-02-01T10:50:08.314 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:MyUQyVW8fy+usa4c5io2A4+w9nw= 2024-02-01T10:50:08.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:MyUQyVW8fy+usa4c5io2A4+w9nw=', 'Content-Length': '0'} 2024-02-01T10:50:08.316 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c17bfe0360d64cd1-0065bb7760-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:08.317 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:08.317 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:08.317 DEBUG:boto:path=/vclfcn-63/?prefix=obj6 2024-02-01T10:50:08.317 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj6 2024-02-01T10:50:08.317 DEBUG:boto:Method: GET 2024-02-01T10:50:08.317 DEBUG:boto:Path: /vclfcn-63/?prefix=obj6 2024-02-01T10:50:08.317 DEBUG:boto:Data: 2024-02-01T10:50:08.317 DEBUG:boto:Headers: {} 2024-02-01T10:50:08.318 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:08.318 DEBUG:boto:Port: 8001 2024-02-01T10:50:08.318 DEBUG:boto:Params: {} 2024-02-01T10:50:08.318 DEBUG:boto:Token: None 2024-02-01T10:50:08.318 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:08 GMT /vclfcn-63/ 2024-02-01T10:50:08.318 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:SWr7xT6s+p02WaAfyPyEv0WQAGU= 2024-02-01T10:50:08.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:SWr7xT6s+p02WaAfyPyEv0WQAGU=', 'Content-Length': '0'} 2024-02-01T10:50:08.322 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ad3b03b7461bda4-0065bb7760-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:08.368 DEBUG:boto:b'vclfcn-63obj61000falseobj62024-02-01T10:49:22.271Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:08.368 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:08.369 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:08.369 DEBUG:boto:path=/vclfcn-63/?prefix=obj6 2024-02-01T10:50:08.369 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj6 2024-02-01T10:50:08.369 DEBUG:boto:Method: GET 2024-02-01T10:50:08.369 DEBUG:boto:Path: /vclfcn-63/?prefix=obj6 2024-02-01T10:50:08.369 DEBUG:boto:Data: 2024-02-01T10:50:08.369 DEBUG:boto:Headers: {} 2024-02-01T10:50:08.369 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:08.369 DEBUG:boto:Port: 8000 2024-02-01T10:50:08.369 DEBUG:boto:Params: {} 2024-02-01T10:50:08.370 DEBUG:boto:Token: None 2024-02-01T10:50:08.370 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:08 GMT /vclfcn-63/ 2024-02-01T10:50:08.370 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:SWr7xT6s+p02WaAfyPyEv0WQAGU= 2024-02-01T10:50:08.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:SWr7xT6s+p02WaAfyPyEv0WQAGU=', 'Content-Length': '0'} 2024-02-01T10:50:08.374 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e33d1b45f4aae5d-0065bb7760-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:08.375 DEBUG:boto:b'vclfcn-63obj61000falseobj62024-02-01T10:49:22.271Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:08.375 DEBUG:boto:path=/vclfcn-63/obj6 2024-02-01T10:50:08.375 DEBUG:boto:auth_path=/vclfcn-63/obj6 2024-02-01T10:50:08.375 DEBUG:boto:Method: HEAD 2024-02-01T10:50:08.375 DEBUG:boto:Path: /vclfcn-63/obj6 2024-02-01T10:50:08.376 DEBUG:boto:Data: 2024-02-01T10:50:08.376 DEBUG:boto:Headers: {} 2024-02-01T10:50:08.376 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:08.376 DEBUG:boto:Port: 8001 2024-02-01T10:50:08.376 DEBUG:boto:Params: {} 2024-02-01T10:50:08.376 DEBUG:boto:Token: None 2024-02-01T10:50:08.376 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:08 GMT /vclfcn-63/obj6 2024-02-01T10:50:08.376 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nHr00YOnpOO1oWEPMNeRq0pQxdE= 2024-02-01T10:50:08.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nHr00YOnpOO1oWEPMNeRq0pQxdE=', 'Content-Length': '0'} 2024-02-01T10:50:08.379 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000396a8c7a1eb0091a-0065bb7760-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:08.379 DEBUG:boto:path=/vclfcn-63/obj6 2024-02-01T10:50:08.379 DEBUG:boto:auth_path=/vclfcn-63/obj6 2024-02-01T10:50:08.380 DEBUG:boto:Method: HEAD 2024-02-01T10:50:08.380 DEBUG:boto:Path: /vclfcn-63/obj6 2024-02-01T10:50:08.380 DEBUG:boto:Data: 2024-02-01T10:50:08.380 DEBUG:boto:Headers: {} 2024-02-01T10:50:08.380 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:08.380 DEBUG:boto:Port: 8000 2024-02-01T10:50:08.380 DEBUG:boto:Params: {} 2024-02-01T10:50:08.380 DEBUG:boto:Token: None 2024-02-01T10:50:08.381 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:08 GMT /vclfcn-63/obj6 2024-02-01T10:50:08.381 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nHr00YOnpOO1oWEPMNeRq0pQxdE= 2024-02-01T10:50:08.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nHr00YOnpOO1oWEPMNeRq0pQxdE=', 'Content-Length': '0'} 2024-02-01T10:50:08.383 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c093bf5ec5a602d6-0065bb7760-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:08.383 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-02-01T10:50:08.384 DEBUG:boto:path=/vclfcn-63/obj6 2024-02-01T10:50:08.384 DEBUG:boto:auth_path=/vclfcn-63/obj6 2024-02-01T10:50:08.384 DEBUG:boto:Method: GET 2024-02-01T10:50:08.384 DEBUG:boto:Path: /vclfcn-63/obj6 2024-02-01T10:50:08.384 DEBUG:boto:Data: 2024-02-01T10:50:08.384 DEBUG:boto:Headers: {} 2024-02-01T10:50:08.384 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:08.384 DEBUG:boto:Port: 8001 2024-02-01T10:50:08.384 DEBUG:boto:Params: {} 2024-02-01T10:50:08.385 DEBUG:boto:Token: None 2024-02-01T10:50:08.385 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:08 GMT /vclfcn-63/obj6 2024-02-01T10:50:08.385 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:xu+wtvjYiLZSllGoUOFXTIUHC6w= 2024-02-01T10:50:08.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:xu+wtvjYiLZSllGoUOFXTIUHC6w=', 'Content-Length': '0'} 2024-02-01T10:50:08.387 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009684c2a1f1fb8ec3-0065bb7760-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:08.428 DEBUG:boto:path=/vclfcn-63/obj6 2024-02-01T10:50:08.428 DEBUG:boto:auth_path=/vclfcn-63/obj6 2024-02-01T10:50:08.428 DEBUG:boto:Method: GET 2024-02-01T10:50:08.429 DEBUG:boto:Path: /vclfcn-63/obj6 2024-02-01T10:50:08.429 DEBUG:boto:Data: 2024-02-01T10:50:08.429 DEBUG:boto:Headers: {} 2024-02-01T10:50:08.429 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:08.429 DEBUG:boto:Port: 8000 2024-02-01T10:50:08.429 DEBUG:boto:Params: {} 2024-02-01T10:50:08.429 DEBUG:boto:Token: None 2024-02-01T10:50:08.429 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:08 GMT /vclfcn-63/obj6 2024-02-01T10:50:08.429 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:xu+wtvjYiLZSllGoUOFXTIUHC6w= 2024-02-01T10:50:08.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:xu+wtvjYiLZSllGoUOFXTIUHC6w=', 'Content-Length': '0'} 2024-02-01T10:50:08.432 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000089f94d8c74c1a92e-0065bb7760-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:08.433 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:08.433 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:08.433 DEBUG:boto:path=/vclfcn-63/?prefix=obj6 2024-02-01T10:50:08.433 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj6 2024-02-01T10:50:08.433 DEBUG:boto:Method: GET 2024-02-01T10:50:08.433 DEBUG:boto:Path: /vclfcn-63/?prefix=obj6 2024-02-01T10:50:08.433 DEBUG:boto:Data: 2024-02-01T10:50:08.433 DEBUG:boto:Headers: {} 2024-02-01T10:50:08.433 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:08.433 DEBUG:boto:Port: 8001 2024-02-01T10:50:08.434 DEBUG:boto:Params: {} 2024-02-01T10:50:08.434 DEBUG:boto:Token: None 2024-02-01T10:50:08.434 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:08 GMT /vclfcn-63/ 2024-02-01T10:50:08.434 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:SWr7xT6s+p02WaAfyPyEv0WQAGU= 2024-02-01T10:50:08.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:SWr7xT6s+p02WaAfyPyEv0WQAGU=', 'Content-Length': '0'} 2024-02-01T10:50:08.439 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000bf7f5145816a35a-0065bb7760-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:08.439 DEBUG:boto:b'vclfcn-63obj61000falseobj62024-02-01T10:49:22.271Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:08.439 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:08.439 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:08.440 DEBUG:boto:path=/vclfcn-63/?prefix=obj6 2024-02-01T10:50:08.440 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj6 2024-02-01T10:50:08.440 DEBUG:boto:Method: GET 2024-02-01T10:50:08.440 DEBUG:boto:Path: /vclfcn-63/?prefix=obj6 2024-02-01T10:50:08.440 DEBUG:boto:Data: 2024-02-01T10:50:08.440 DEBUG:boto:Headers: {} 2024-02-01T10:50:08.440 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:08.440 DEBUG:boto:Port: 8000 2024-02-01T10:50:08.440 DEBUG:boto:Params: {} 2024-02-01T10:50:08.440 DEBUG:boto:Token: None 2024-02-01T10:50:08.440 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:08 GMT /vclfcn-63/ 2024-02-01T10:50:08.441 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:SWr7xT6s+p02WaAfyPyEv0WQAGU= 2024-02-01T10:50:08.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:SWr7xT6s+p02WaAfyPyEv0WQAGU=', 'Content-Length': '0'} 2024-02-01T10:50:08.445 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a33ee4e45f518a8a-0065bb7760-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:08.445 DEBUG:boto:b'vclfcn-63obj61000falseobj62024-02-01T10:49:22.271Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:08.446 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-02-01T10:50:08.446 DEBUG:boto:path=/vclfcn-63/obj7 2024-02-01T10:50:08.446 DEBUG:boto:auth_path=/vclfcn-63/obj7 2024-02-01T10:50:08.446 DEBUG:boto:path=/vclfcn-63/obj7?versionId=null 2024-02-01T10:50:08.446 DEBUG:boto:auth_path=/vclfcn-63/obj7?versionId=null 2024-02-01T10:50:08.446 DEBUG:boto:Method: GET 2024-02-01T10:50:08.446 DEBUG:boto:Path: /vclfcn-63/obj7?versionId=null 2024-02-01T10:50:08.446 DEBUG:boto:Data: 2024-02-01T10:50:08.446 DEBUG:boto:Headers: {} 2024-02-01T10:50:08.447 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:08.447 DEBUG:boto:Port: 8001 2024-02-01T10:50:08.447 DEBUG:boto:Params: {} 2024-02-01T10:50:08.447 DEBUG:boto:Token: None 2024-02-01T10:50:08.447 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:08 GMT /vclfcn-63/obj7?versionId=null 2024-02-01T10:50:08.447 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:W7xKgd8LrjQ9bacCwY2ghZSTnIw= 2024-02-01T10:50:08.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:W7xKgd8LrjQ9bacCwY2ghZSTnIw=', 'Content-Length': '0'} 2024-02-01T10:50:08.450 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006de7e3f56eb03e71-0065bb7760-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:08.496 DEBUG:boto:path=/vclfcn-63/obj7 2024-02-01T10:50:08.496 DEBUG:boto:auth_path=/vclfcn-63/obj7 2024-02-01T10:50:08.496 DEBUG:boto:path=/vclfcn-63/obj7?versionId=null 2024-02-01T10:50:08.496 DEBUG:boto:auth_path=/vclfcn-63/obj7?versionId=null 2024-02-01T10:50:08.497 DEBUG:boto:Method: GET 2024-02-01T10:50:08.497 DEBUG:boto:Path: /vclfcn-63/obj7?versionId=null 2024-02-01T10:50:08.497 DEBUG:boto:Data: 2024-02-01T10:50:08.497 DEBUG:boto:Headers: {} 2024-02-01T10:50:08.497 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:08.497 DEBUG:boto:Port: 8000 2024-02-01T10:50:08.497 DEBUG:boto:Params: {} 2024-02-01T10:50:08.497 DEBUG:boto:Token: None 2024-02-01T10:50:08.498 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:08 GMT /vclfcn-63/obj7?versionId=null 2024-02-01T10:50:08.498 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:W7xKgd8LrjQ9bacCwY2ghZSTnIw= 2024-02-01T10:50:08.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:W7xKgd8LrjQ9bacCwY2ghZSTnIw=', 'Content-Length': '0'} 2024-02-01T10:50:08.502 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a580af8ab109547c-0065bb7760-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:08.502 DEBUG:boto:path=/vclfcn-63/obj7 2024-02-01T10:50:08.502 DEBUG:boto:auth_path=/vclfcn-63/obj7 2024-02-01T10:50:08.502 DEBUG:boto:path=/vclfcn-63/obj7?versionId=null 2024-02-01T10:50:08.502 DEBUG:boto:auth_path=/vclfcn-63/obj7?versionId=null 2024-02-01T10:50:08.502 DEBUG:boto:Method: HEAD 2024-02-01T10:50:08.502 DEBUG:boto:Path: /vclfcn-63/obj7?versionId=null 2024-02-01T10:50:08.503 DEBUG:boto:Data: 2024-02-01T10:50:08.503 DEBUG:boto:Headers: {} 2024-02-01T10:50:08.503 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:08.503 DEBUG:boto:Port: 8001 2024-02-01T10:50:08.503 DEBUG:boto:Params: {} 2024-02-01T10:50:08.503 DEBUG:boto:Token: None 2024-02-01T10:50:08.503 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:08 GMT /vclfcn-63/obj7?versionId=null 2024-02-01T10:50:08.503 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7x07h8v/fQLdl7V58C+QzcvRrxE= 2024-02-01T10:50:08.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7x07h8v/fQLdl7V58C+QzcvRrxE=', 'Content-Length': '0'} 2024-02-01T10:50:08.506 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006802426338a18c8b-0065bb7760-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:08.506 DEBUG:boto:path=/vclfcn-63/obj7 2024-02-01T10:50:08.507 DEBUG:boto:auth_path=/vclfcn-63/obj7 2024-02-01T10:50:08.507 DEBUG:boto:path=/vclfcn-63/obj7?versionId=null 2024-02-01T10:50:08.507 DEBUG:boto:auth_path=/vclfcn-63/obj7?versionId=null 2024-02-01T10:50:08.507 DEBUG:boto:Method: HEAD 2024-02-01T10:50:08.507 DEBUG:boto:Path: /vclfcn-63/obj7?versionId=null 2024-02-01T10:50:08.507 DEBUG:boto:Data: 2024-02-01T10:50:08.507 DEBUG:boto:Headers: {} 2024-02-01T10:50:08.507 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:08.507 DEBUG:boto:Port: 8000 2024-02-01T10:50:08.507 DEBUG:boto:Params: {} 2024-02-01T10:50:08.508 DEBUG:boto:Token: None 2024-02-01T10:50:08.508 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:08 GMT /vclfcn-63/obj7?versionId=null 2024-02-01T10:50:08.508 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7x07h8v/fQLdl7V58C+QzcvRrxE= 2024-02-01T10:50:08.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7x07h8v/fQLdl7V58C+QzcvRrxE=', 'Content-Length': '0'} 2024-02-01T10:50:08.511 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000db558c2aad77f6e6-0065bb7760-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:08.511 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-02-01T10:50:08.511 DEBUG:boto:path=/vclfcn-63/obj7 2024-02-01T10:50:08.511 DEBUG:boto:auth_path=/vclfcn-63/obj7 2024-02-01T10:50:08.511 DEBUG:boto:path=/vclfcn-63/obj7?versionId=null 2024-02-01T10:50:08.511 DEBUG:boto:auth_path=/vclfcn-63/obj7?versionId=null 2024-02-01T10:50:08.511 DEBUG:boto:Method: GET 2024-02-01T10:50:08.511 DEBUG:boto:Path: /vclfcn-63/obj7?versionId=null 2024-02-01T10:50:08.511 DEBUG:boto:Data: 2024-02-01T10:50:08.512 DEBUG:boto:Headers: {} 2024-02-01T10:50:08.512 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:08.512 DEBUG:boto:Port: 8001 2024-02-01T10:50:08.512 DEBUG:boto:Params: {} 2024-02-01T10:50:08.512 DEBUG:boto:Token: None 2024-02-01T10:50:08.512 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:08 GMT /vclfcn-63/obj7?versionId=null 2024-02-01T10:50:08.512 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:W7xKgd8LrjQ9bacCwY2ghZSTnIw= 2024-02-01T10:50:08.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:W7xKgd8LrjQ9bacCwY2ghZSTnIw=', 'Content-Length': '0'} 2024-02-01T10:50:08.515 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001d42bb4a9b21e458-0065bb7760-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:08.556 DEBUG:boto:path=/vclfcn-63/obj7 2024-02-01T10:50:08.556 DEBUG:boto:auth_path=/vclfcn-63/obj7 2024-02-01T10:50:08.557 DEBUG:boto:path=/vclfcn-63/obj7?versionId=null 2024-02-01T10:50:08.557 DEBUG:boto:auth_path=/vclfcn-63/obj7?versionId=null 2024-02-01T10:50:08.557 DEBUG:boto:Method: GET 2024-02-01T10:50:08.557 DEBUG:boto:Path: /vclfcn-63/obj7?versionId=null 2024-02-01T10:50:08.557 DEBUG:boto:Data: 2024-02-01T10:50:08.557 DEBUG:boto:Headers: {} 2024-02-01T10:50:08.557 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:08.557 DEBUG:boto:Port: 8000 2024-02-01T10:50:08.557 DEBUG:boto:Params: {} 2024-02-01T10:50:08.558 DEBUG:boto:Token: None 2024-02-01T10:50:08.558 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:08 GMT /vclfcn-63/obj7?versionId=null 2024-02-01T10:50:08.558 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:W7xKgd8LrjQ9bacCwY2ghZSTnIw= 2024-02-01T10:50:08.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:W7xKgd8LrjQ9bacCwY2ghZSTnIw=', 'Content-Length': '0'} 2024-02-01T10:50:08.562 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000075582c7e8a4358ac-0065bb7760-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:08.562 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:08.563 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:08.563 DEBUG:boto:path=/vclfcn-63/?prefix=obj7 2024-02-01T10:50:08.563 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj7 2024-02-01T10:50:08.563 DEBUG:boto:Method: GET 2024-02-01T10:50:08.563 DEBUG:boto:Path: /vclfcn-63/?prefix=obj7 2024-02-01T10:50:08.563 DEBUG:boto:Data: 2024-02-01T10:50:08.563 DEBUG:boto:Headers: {} 2024-02-01T10:50:08.563 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:08.563 DEBUG:boto:Port: 8001 2024-02-01T10:50:08.564 DEBUG:boto:Params: {} 2024-02-01T10:50:08.564 DEBUG:boto:Token: None 2024-02-01T10:50:08.564 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:08 GMT /vclfcn-63/ 2024-02-01T10:50:08.565 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:SWr7xT6s+p02WaAfyPyEv0WQAGU= 2024-02-01T10:50:08.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:SWr7xT6s+p02WaAfyPyEv0WQAGU=', 'Content-Length': '0'} 2024-02-01T10:50:08.569 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c461fc1ae85e324-0065bb7760-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:08.569 DEBUG:boto:b'vclfcn-63obj71000falseobj72024-02-01T10:49:22.327Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:08.570 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:08.570 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:08.570 DEBUG:boto:path=/vclfcn-63/?prefix=obj7 2024-02-01T10:50:08.570 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj7 2024-02-01T10:50:08.570 DEBUG:boto:Method: GET 2024-02-01T10:50:08.570 DEBUG:boto:Path: /vclfcn-63/?prefix=obj7 2024-02-01T10:50:08.570 DEBUG:boto:Data: 2024-02-01T10:50:08.570 DEBUG:boto:Headers: {} 2024-02-01T10:50:08.570 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:08.570 DEBUG:boto:Port: 8000 2024-02-01T10:50:08.571 DEBUG:boto:Params: {} 2024-02-01T10:50:08.571 DEBUG:boto:Token: None 2024-02-01T10:50:08.571 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:08 GMT /vclfcn-63/ 2024-02-01T10:50:08.571 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:SWr7xT6s+p02WaAfyPyEv0WQAGU= 2024-02-01T10:50:08.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:SWr7xT6s+p02WaAfyPyEv0WQAGU=', 'Content-Length': '0'} 2024-02-01T10:50:08.575 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c786b6eb9c8ebfb9-0065bb7760-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:08.620 DEBUG:boto:b'vclfcn-63obj71000falseobj72024-02-01T10:49:22.327Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:08.621 DEBUG:boto:path=/vclfcn-63/obj7 2024-02-01T10:50:08.621 DEBUG:boto:auth_path=/vclfcn-63/obj7 2024-02-01T10:50:08.621 DEBUG:boto:Method: HEAD 2024-02-01T10:50:08.621 DEBUG:boto:Path: /vclfcn-63/obj7 2024-02-01T10:50:08.621 DEBUG:boto:Data: 2024-02-01T10:50:08.621 DEBUG:boto:Headers: {} 2024-02-01T10:50:08.621 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:08.621 DEBUG:boto:Port: 8001 2024-02-01T10:50:08.621 DEBUG:boto:Params: {} 2024-02-01T10:50:08.621 DEBUG:boto:Token: None 2024-02-01T10:50:08.622 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:08 GMT /vclfcn-63/obj7 2024-02-01T10:50:08.622 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8WUfnDr1Gwuc7a3lutRM5oBaM7E= 2024-02-01T10:50:08.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8WUfnDr1Gwuc7a3lutRM5oBaM7E=', 'Content-Length': '0'} 2024-02-01T10:50:08.628 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000069eb638f91feeade-0065bb7760-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:08.628 DEBUG:boto:path=/vclfcn-63/obj7 2024-02-01T10:50:08.628 DEBUG:boto:auth_path=/vclfcn-63/obj7 2024-02-01T10:50:08.629 DEBUG:boto:Method: HEAD 2024-02-01T10:50:08.629 DEBUG:boto:Path: /vclfcn-63/obj7 2024-02-01T10:50:08.629 DEBUG:boto:Data: 2024-02-01T10:50:08.629 DEBUG:boto:Headers: {} 2024-02-01T10:50:08.629 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:08.630 DEBUG:boto:Port: 8000 2024-02-01T10:50:08.630 DEBUG:boto:Params: {} 2024-02-01T10:50:08.630 DEBUG:boto:Token: None 2024-02-01T10:50:08.630 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:08 GMT /vclfcn-63/obj7 2024-02-01T10:50:08.631 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8WUfnDr1Gwuc7a3lutRM5oBaM7E= 2024-02-01T10:50:08.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8WUfnDr1Gwuc7a3lutRM5oBaM7E=', 'Content-Length': '0'} 2024-02-01T10:50:08.634 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d93d25fb161fdadf-0065bb7760-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:08.634 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-02-01T10:50:08.635 DEBUG:boto:path=/vclfcn-63/obj7 2024-02-01T10:50:08.635 DEBUG:boto:auth_path=/vclfcn-63/obj7 2024-02-01T10:50:08.635 DEBUG:boto:Method: GET 2024-02-01T10:50:08.635 DEBUG:boto:Path: /vclfcn-63/obj7 2024-02-01T10:50:08.635 DEBUG:boto:Data: 2024-02-01T10:50:08.636 DEBUG:boto:Headers: {} 2024-02-01T10:50:08.636 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:08.636 DEBUG:boto:Port: 8001 2024-02-01T10:50:08.636 DEBUG:boto:Params: {} 2024-02-01T10:50:08.636 DEBUG:boto:Token: None 2024-02-01T10:50:08.637 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:08 GMT /vclfcn-63/obj7 2024-02-01T10:50:08.637 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jl1bFYFyv/MrMUDZwImc7J3Tp7g= 2024-02-01T10:50:08.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jl1bFYFyv/MrMUDZwImc7J3Tp7g=', 'Content-Length': '0'} 2024-02-01T10:50:08.640 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a1bdead070110337-0065bb7760-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:08.688 DEBUG:boto:path=/vclfcn-63/obj7 2024-02-01T10:50:08.689 DEBUG:boto:auth_path=/vclfcn-63/obj7 2024-02-01T10:50:08.689 DEBUG:boto:Method: GET 2024-02-01T10:50:08.689 DEBUG:boto:Path: /vclfcn-63/obj7 2024-02-01T10:50:08.690 DEBUG:boto:Data: 2024-02-01T10:50:08.691 DEBUG:boto:Headers: {} 2024-02-01T10:50:08.691 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:08.691 DEBUG:boto:Port: 8000 2024-02-01T10:50:08.691 DEBUG:boto:Params: {} 2024-02-01T10:50:08.691 DEBUG:boto:Token: None 2024-02-01T10:50:08.691 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:08 GMT /vclfcn-63/obj7 2024-02-01T10:50:08.692 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:jl1bFYFyv/MrMUDZwImc7J3Tp7g= 2024-02-01T10:50:08.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:jl1bFYFyv/MrMUDZwImc7J3Tp7g=', 'Content-Length': '0'} 2024-02-01T10:50:08.695 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008771c293dd760f9b-0065bb7760-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:08.696 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:08.696 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:08.696 DEBUG:boto:path=/vclfcn-63/?prefix=obj7 2024-02-01T10:50:08.696 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj7 2024-02-01T10:50:08.696 DEBUG:boto:Method: GET 2024-02-01T10:50:08.697 DEBUG:boto:Path: /vclfcn-63/?prefix=obj7 2024-02-01T10:50:08.697 DEBUG:boto:Data: 2024-02-01T10:50:08.697 DEBUG:boto:Headers: {} 2024-02-01T10:50:08.697 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:08.697 DEBUG:boto:Port: 8001 2024-02-01T10:50:08.697 DEBUG:boto:Params: {} 2024-02-01T10:50:08.697 DEBUG:boto:Token: None 2024-02-01T10:50:08.697 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:08 GMT /vclfcn-63/ 2024-02-01T10:50:08.697 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:SWr7xT6s+p02WaAfyPyEv0WQAGU= 2024-02-01T10:50:08.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:SWr7xT6s+p02WaAfyPyEv0WQAGU=', 'Content-Length': '0'} 2024-02-01T10:50:08.702 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000005a83861377a367d-0065bb7760-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:08.748 DEBUG:boto:b'vclfcn-63obj71000falseobj72024-02-01T10:49:22.327Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:08.749 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:08.749 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:08.749 DEBUG:boto:path=/vclfcn-63/?prefix=obj7 2024-02-01T10:50:08.749 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj7 2024-02-01T10:50:08.749 DEBUG:boto:Method: GET 2024-02-01T10:50:08.749 DEBUG:boto:Path: /vclfcn-63/?prefix=obj7 2024-02-01T10:50:08.749 DEBUG:boto:Data: 2024-02-01T10:50:08.749 DEBUG:boto:Headers: {} 2024-02-01T10:50:08.749 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:08.749 DEBUG:boto:Port: 8000 2024-02-01T10:50:08.750 DEBUG:boto:Params: {} 2024-02-01T10:50:08.750 DEBUG:boto:Token: None 2024-02-01T10:50:08.750 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:08 GMT /vclfcn-63/ 2024-02-01T10:50:08.750 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:SWr7xT6s+p02WaAfyPyEv0WQAGU= 2024-02-01T10:50:08.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:SWr7xT6s+p02WaAfyPyEv0WQAGU=', 'Content-Length': '0'} 2024-02-01T10:50:08.754 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cbf011a94c71f786-0065bb7760-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:08.754 DEBUG:boto:b'vclfcn-63obj71000falseobj72024-02-01T10:49:22.327Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:08.755 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-02-01T10:50:08.755 DEBUG:boto:path=/vclfcn-63/obj8 2024-02-01T10:50:08.755 DEBUG:boto:auth_path=/vclfcn-63/obj8 2024-02-01T10:50:08.755 DEBUG:boto:path=/vclfcn-63/obj8?versionId=null 2024-02-01T10:50:08.755 DEBUG:boto:auth_path=/vclfcn-63/obj8?versionId=null 2024-02-01T10:50:08.755 DEBUG:boto:Method: GET 2024-02-01T10:50:08.755 DEBUG:boto:Path: /vclfcn-63/obj8?versionId=null 2024-02-01T10:50:08.755 DEBUG:boto:Data: 2024-02-01T10:50:08.755 DEBUG:boto:Headers: {} 2024-02-01T10:50:08.756 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:08.756 DEBUG:boto:Port: 8001 2024-02-01T10:50:08.756 DEBUG:boto:Params: {} 2024-02-01T10:50:08.756 DEBUG:boto:Token: None 2024-02-01T10:50:08.756 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:08 GMT /vclfcn-63/obj8?versionId=null 2024-02-01T10:50:08.756 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ay/Q9gxiQ6nH9E0Du0ePHwSCkaA= 2024-02-01T10:50:08.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ay/Q9gxiQ6nH9E0Du0ePHwSCkaA=', 'Content-Length': '0'} 2024-02-01T10:50:08.759 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b9ac7424eb4b4d56-0065bb7760-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:08.804 DEBUG:boto:path=/vclfcn-63/obj8 2024-02-01T10:50:08.804 DEBUG:boto:auth_path=/vclfcn-63/obj8 2024-02-01T10:50:08.804 DEBUG:boto:path=/vclfcn-63/obj8?versionId=null 2024-02-01T10:50:08.804 DEBUG:boto:auth_path=/vclfcn-63/obj8?versionId=null 2024-02-01T10:50:08.805 DEBUG:boto:Method: GET 2024-02-01T10:50:08.805 DEBUG:boto:Path: /vclfcn-63/obj8?versionId=null 2024-02-01T10:50:08.805 DEBUG:boto:Data: 2024-02-01T10:50:08.805 DEBUG:boto:Headers: {} 2024-02-01T10:50:08.805 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:08.805 DEBUG:boto:Port: 8000 2024-02-01T10:50:08.805 DEBUG:boto:Params: {} 2024-02-01T10:50:08.805 DEBUG:boto:Token: None 2024-02-01T10:50:08.805 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:08 GMT /vclfcn-63/obj8?versionId=null 2024-02-01T10:50:08.806 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ay/Q9gxiQ6nH9E0Du0ePHwSCkaA= 2024-02-01T10:50:08.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ay/Q9gxiQ6nH9E0Du0ePHwSCkaA=', 'Content-Length': '0'} 2024-02-01T10:50:08.809 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b6580e12c4aa92b8-0065bb7760-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:08.809 DEBUG:boto:path=/vclfcn-63/obj8 2024-02-01T10:50:08.809 DEBUG:boto:auth_path=/vclfcn-63/obj8 2024-02-01T10:50:08.809 DEBUG:boto:path=/vclfcn-63/obj8?versionId=null 2024-02-01T10:50:08.809 DEBUG:boto:auth_path=/vclfcn-63/obj8?versionId=null 2024-02-01T10:50:08.809 DEBUG:boto:Method: HEAD 2024-02-01T10:50:08.809 DEBUG:boto:Path: /vclfcn-63/obj8?versionId=null 2024-02-01T10:50:08.810 DEBUG:boto:Data: 2024-02-01T10:50:08.810 DEBUG:boto:Headers: {} 2024-02-01T10:50:08.810 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:08.810 DEBUG:boto:Port: 8001 2024-02-01T10:50:08.810 DEBUG:boto:Params: {} 2024-02-01T10:50:08.810 DEBUG:boto:Token: None 2024-02-01T10:50:08.810 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:08 GMT /vclfcn-63/obj8?versionId=null 2024-02-01T10:50:08.810 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QwXHJJhPQ/BAQ1jBQ/FSkjmYod0= 2024-02-01T10:50:08.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QwXHJJhPQ/BAQ1jBQ/FSkjmYod0=', 'Content-Length': '0'} 2024-02-01T10:50:08.813 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c40be1883fdb22e4-0065bb7760-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:08.813 DEBUG:boto:path=/vclfcn-63/obj8 2024-02-01T10:50:08.813 DEBUG:boto:auth_path=/vclfcn-63/obj8 2024-02-01T10:50:08.813 DEBUG:boto:path=/vclfcn-63/obj8?versionId=null 2024-02-01T10:50:08.814 DEBUG:boto:auth_path=/vclfcn-63/obj8?versionId=null 2024-02-01T10:50:08.814 DEBUG:boto:Method: HEAD 2024-02-01T10:50:08.814 DEBUG:boto:Path: /vclfcn-63/obj8?versionId=null 2024-02-01T10:50:08.814 DEBUG:boto:Data: 2024-02-01T10:50:08.814 DEBUG:boto:Headers: {} 2024-02-01T10:50:08.814 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:08.814 DEBUG:boto:Port: 8000 2024-02-01T10:50:08.814 DEBUG:boto:Params: {} 2024-02-01T10:50:08.814 DEBUG:boto:Token: None 2024-02-01T10:50:08.814 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:08 GMT /vclfcn-63/obj8?versionId=null 2024-02-01T10:50:08.815 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QwXHJJhPQ/BAQ1jBQ/FSkjmYod0= 2024-02-01T10:50:08.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QwXHJJhPQ/BAQ1jBQ/FSkjmYod0=', 'Content-Length': '0'} 2024-02-01T10:50:08.817 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eda6140aae46f06f-0065bb7760-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:08.817 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-02-01T10:50:08.818 DEBUG:boto:path=/vclfcn-63/obj8 2024-02-01T10:50:08.818 DEBUG:boto:auth_path=/vclfcn-63/obj8 2024-02-01T10:50:08.818 DEBUG:boto:path=/vclfcn-63/obj8?versionId=null 2024-02-01T10:50:08.818 DEBUG:boto:auth_path=/vclfcn-63/obj8?versionId=null 2024-02-01T10:50:08.818 DEBUG:boto:Method: GET 2024-02-01T10:50:08.818 DEBUG:boto:Path: /vclfcn-63/obj8?versionId=null 2024-02-01T10:50:08.818 DEBUG:boto:Data: 2024-02-01T10:50:08.818 DEBUG:boto:Headers: {} 2024-02-01T10:50:08.818 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:08.818 DEBUG:boto:Port: 8001 2024-02-01T10:50:08.818 DEBUG:boto:Params: {} 2024-02-01T10:50:08.819 DEBUG:boto:Token: None 2024-02-01T10:50:08.819 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:08 GMT /vclfcn-63/obj8?versionId=null 2024-02-01T10:50:08.819 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ay/Q9gxiQ6nH9E0Du0ePHwSCkaA= 2024-02-01T10:50:08.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ay/Q9gxiQ6nH9E0Du0ePHwSCkaA=', 'Content-Length': '0'} 2024-02-01T10:50:08.822 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006063734540bd4c0e-0065bb7760-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:08.864 DEBUG:boto:path=/vclfcn-63/obj8 2024-02-01T10:50:08.864 DEBUG:boto:auth_path=/vclfcn-63/obj8 2024-02-01T10:50:08.864 DEBUG:boto:path=/vclfcn-63/obj8?versionId=null 2024-02-01T10:50:08.864 DEBUG:boto:auth_path=/vclfcn-63/obj8?versionId=null 2024-02-01T10:50:08.865 DEBUG:boto:Method: GET 2024-02-01T10:50:08.865 DEBUG:boto:Path: /vclfcn-63/obj8?versionId=null 2024-02-01T10:50:08.865 DEBUG:boto:Data: 2024-02-01T10:50:08.865 DEBUG:boto:Headers: {} 2024-02-01T10:50:08.865 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:08.865 DEBUG:boto:Port: 8000 2024-02-01T10:50:08.865 DEBUG:boto:Params: {} 2024-02-01T10:50:08.865 DEBUG:boto:Token: None 2024-02-01T10:50:08.865 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:08 GMT /vclfcn-63/obj8?versionId=null 2024-02-01T10:50:08.866 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ay/Q9gxiQ6nH9E0Du0ePHwSCkaA= 2024-02-01T10:50:08.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ay/Q9gxiQ6nH9E0Du0ePHwSCkaA=', 'Content-Length': '0'} 2024-02-01T10:50:08.869 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000007d23d305815898b-0065bb7760-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:08.869 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:08.869 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:08.869 DEBUG:boto:path=/vclfcn-63/?prefix=obj8 2024-02-01T10:50:08.869 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj8 2024-02-01T10:50:08.870 DEBUG:boto:Method: GET 2024-02-01T10:50:08.870 DEBUG:boto:Path: /vclfcn-63/?prefix=obj8 2024-02-01T10:50:08.870 DEBUG:boto:Data: 2024-02-01T10:50:08.870 DEBUG:boto:Headers: {} 2024-02-01T10:50:08.870 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:08.870 DEBUG:boto:Port: 8001 2024-02-01T10:50:08.870 DEBUG:boto:Params: {} 2024-02-01T10:50:08.870 DEBUG:boto:Token: None 2024-02-01T10:50:08.870 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:08 GMT /vclfcn-63/ 2024-02-01T10:50:08.871 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:SWr7xT6s+p02WaAfyPyEv0WQAGU= 2024-02-01T10:50:08.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:SWr7xT6s+p02WaAfyPyEv0WQAGU=', 'Content-Length': '0'} 2024-02-01T10:50:08.875 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d09b3952bb372fcf-0065bb7760-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:08.875 DEBUG:boto:b'vclfcn-63obj81000falseobj82024-02-01T10:49:22.343Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:08.876 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:08.876 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:08.876 DEBUG:boto:path=/vclfcn-63/?prefix=obj8 2024-02-01T10:50:08.876 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj8 2024-02-01T10:50:08.877 DEBUG:boto:Method: GET 2024-02-01T10:50:08.877 DEBUG:boto:Path: /vclfcn-63/?prefix=obj8 2024-02-01T10:50:08.877 DEBUG:boto:Data: 2024-02-01T10:50:08.877 DEBUG:boto:Headers: {} 2024-02-01T10:50:08.877 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:08.877 DEBUG:boto:Port: 8000 2024-02-01T10:50:08.877 DEBUG:boto:Params: {} 2024-02-01T10:50:08.877 DEBUG:boto:Token: None 2024-02-01T10:50:08.877 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:08 GMT /vclfcn-63/ 2024-02-01T10:50:08.878 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:SWr7xT6s+p02WaAfyPyEv0WQAGU= 2024-02-01T10:50:08.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:SWr7xT6s+p02WaAfyPyEv0WQAGU=', 'Content-Length': '0'} 2024-02-01T10:50:08.882 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000dab789f324a8aa1-0065bb7760-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:08.924 DEBUG:boto:b'vclfcn-63obj81000falseobj82024-02-01T10:49:22.343Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:08.925 DEBUG:boto:path=/vclfcn-63/obj8 2024-02-01T10:50:08.925 DEBUG:boto:auth_path=/vclfcn-63/obj8 2024-02-01T10:50:08.925 DEBUG:boto:Method: HEAD 2024-02-01T10:50:08.925 DEBUG:boto:Path: /vclfcn-63/obj8 2024-02-01T10:50:08.925 DEBUG:boto:Data: 2024-02-01T10:50:08.925 DEBUG:boto:Headers: {} 2024-02-01T10:50:08.925 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:08.925 DEBUG:boto:Port: 8001 2024-02-01T10:50:08.925 DEBUG:boto:Params: {} 2024-02-01T10:50:08.925 DEBUG:boto:Token: None 2024-02-01T10:50:08.926 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:08 GMT /vclfcn-63/obj8 2024-02-01T10:50:08.926 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:pG4AkqrX+Hh8JX6mTP2PKl2LhiU= 2024-02-01T10:50:08.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:pG4AkqrX+Hh8JX6mTP2PKl2LhiU=', 'Content-Length': '0'} 2024-02-01T10:50:08.929 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004a69f7d2bc73ad45-0065bb7760-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:08.929 DEBUG:boto:path=/vclfcn-63/obj8 2024-02-01T10:50:08.929 DEBUG:boto:auth_path=/vclfcn-63/obj8 2024-02-01T10:50:08.929 DEBUG:boto:Method: HEAD 2024-02-01T10:50:08.929 DEBUG:boto:Path: /vclfcn-63/obj8 2024-02-01T10:50:08.929 DEBUG:boto:Data: 2024-02-01T10:50:08.929 DEBUG:boto:Headers: {} 2024-02-01T10:50:08.930 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:08.930 DEBUG:boto:Port: 8000 2024-02-01T10:50:08.930 DEBUG:boto:Params: {} 2024-02-01T10:50:08.930 DEBUG:boto:Token: None 2024-02-01T10:50:08.930 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:08 GMT /vclfcn-63/obj8 2024-02-01T10:50:08.930 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:pG4AkqrX+Hh8JX6mTP2PKl2LhiU= 2024-02-01T10:50:08.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:pG4AkqrX+Hh8JX6mTP2PKl2LhiU=', 'Content-Length': '0'} 2024-02-01T10:50:08.933 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004cf0a0e1dfe37cd8-0065bb7760-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:08.933 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-02-01T10:50:08.933 DEBUG:boto:path=/vclfcn-63/obj8 2024-02-01T10:50:08.933 DEBUG:boto:auth_path=/vclfcn-63/obj8 2024-02-01T10:50:08.933 DEBUG:boto:Method: GET 2024-02-01T10:50:08.934 DEBUG:boto:Path: /vclfcn-63/obj8 2024-02-01T10:50:08.934 DEBUG:boto:Data: 2024-02-01T10:50:08.934 DEBUG:boto:Headers: {} 2024-02-01T10:50:08.934 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:08.934 DEBUG:boto:Port: 8001 2024-02-01T10:50:08.934 DEBUG:boto:Params: {} 2024-02-01T10:50:08.934 DEBUG:boto:Token: None 2024-02-01T10:50:08.934 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:08 GMT /vclfcn-63/obj8 2024-02-01T10:50:08.934 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HeCEEfFeb4Q3Y0q3mppR9O3lHA4= 2024-02-01T10:50:08.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HeCEEfFeb4Q3Y0q3mppR9O3lHA4=', 'Content-Length': '0'} 2024-02-01T10:50:08.937 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '01fb6d04-f0ef-49b2-9ebc-c1b530925485:vclfcn-63:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000288a72be90bd5372-0065bb7760-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:08.980 DEBUG:boto:path=/vclfcn-63/obj8 2024-02-01T10:50:08.980 DEBUG:boto:auth_path=/vclfcn-63/obj8 2024-02-01T10:50:08.980 DEBUG:boto:Method: GET 2024-02-01T10:50:08.981 DEBUG:boto:Path: /vclfcn-63/obj8 2024-02-01T10:50:08.981 DEBUG:boto:Data: 2024-02-01T10:50:08.981 DEBUG:boto:Headers: {} 2024-02-01T10:50:08.981 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:08.981 DEBUG:boto:Port: 8000 2024-02-01T10:50:08.981 DEBUG:boto:Params: {} 2024-02-01T10:50:08.981 DEBUG:boto:Token: None 2024-02-01T10:50:08.981 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:08 GMT /vclfcn-63/obj8 2024-02-01T10:50:08.982 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HeCEEfFeb4Q3Y0q3mppR9O3lHA4= 2024-02-01T10:50:08.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HeCEEfFeb4Q3Y0q3mppR9O3lHA4=', 'Content-Length': '0'} 2024-02-01T10:50:08.984 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eb73a695bdb06b5c-0065bb7760-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:08.985 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:08.985 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:08.985 DEBUG:boto:path=/vclfcn-63/?prefix=obj8 2024-02-01T10:50:08.985 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj8 2024-02-01T10:50:08.985 DEBUG:boto:Method: GET 2024-02-01T10:50:08.985 DEBUG:boto:Path: /vclfcn-63/?prefix=obj8 2024-02-01T10:50:08.986 DEBUG:boto:Data: 2024-02-01T10:50:08.986 DEBUG:boto:Headers: {} 2024-02-01T10:50:08.986 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:08.986 DEBUG:boto:Port: 8001 2024-02-01T10:50:08.986 DEBUG:boto:Params: {} 2024-02-01T10:50:08.986 DEBUG:boto:Token: None 2024-02-01T10:50:08.986 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:08 GMT /vclfcn-63/ 2024-02-01T10:50:08.986 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:SWr7xT6s+p02WaAfyPyEv0WQAGU= 2024-02-01T10:50:08.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:SWr7xT6s+p02WaAfyPyEv0WQAGU=', 'Content-Length': '0'} 2024-02-01T10:50:08.991 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e388d70e4b9e0f92-0065bb7760-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:08.991 DEBUG:boto:b'vclfcn-63obj81000falseobj82024-02-01T10:49:22.343Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:08.991 DEBUG:boto:path=/vclfcn-63/ 2024-02-01T10:50:08.992 DEBUG:boto:auth_path=/vclfcn-63/ 2024-02-01T10:50:08.992 DEBUG:boto:path=/vclfcn-63/?prefix=obj8 2024-02-01T10:50:08.992 DEBUG:boto:auth_path=/vclfcn-63/?prefix=obj8 2024-02-01T10:50:08.992 DEBUG:boto:Method: GET 2024-02-01T10:50:08.992 DEBUG:boto:Path: /vclfcn-63/?prefix=obj8 2024-02-01T10:50:08.992 DEBUG:boto:Data: 2024-02-01T10:50:08.992 DEBUG:boto:Headers: {} 2024-02-01T10:50:08.992 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:08.992 DEBUG:boto:Port: 8000 2024-02-01T10:50:08.993 DEBUG:boto:Params: {} 2024-02-01T10:50:08.993 DEBUG:boto:Token: None 2024-02-01T10:50:08.993 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:08 GMT /vclfcn-63/ 2024-02-01T10:50:08.993 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:SWr7xT6s+p02WaAfyPyEv0WQAGU= 2024-02-01T10:50:08.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:08 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:SWr7xT6s+p02WaAfyPyEv0WQAGU=', 'Content-Length': '0'} 2024-02-01T10:50:08.997 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5e25286a3aa1a41-0065bb7760-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:08.998 DEBUG:boto:b'vclfcn-63obj81000falseobj82024-02-01T10:49:22.343Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:08.998 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-63 zones={test-zone3, test-zone2} 2024-02-01T10:50:08.998 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=vclfcn-64 2024-02-01T10:50:08.998 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:50:08.998 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:50:08.999 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket vclfcn-64 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-02-01T10:50:09.049 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:50:09.049 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:50:09.113 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:50:09.112+0000 7f79106d2980 1 bucket sync caught up with source: 2024-02-01T10:50:09.113 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [00000000004.82.6, 00000000002.65.1, 00000000004.56.6, 00000000002.45.1, 00000000004.56.6, 00000000002.29.1, 00000000002.54.1, 00000000004.65.6, 00000000002.70.1, 00000000002.54.1, 00000000002.54.1] 2024-02-01T10:50:09.113 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [00000000004.82.6, 00000000002.65.1, 00000000004.56.6, 00000000002.45.1, 00000000004.56.6, 00000000002.29.1, 00000000002.54.1, 00000000004.65.6, 00000000002.70.1, 00000000002.54.1, 00000000002.54.1] 2024-02-01T10:50:09.113 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:50:09.112+0000 7f79106d2980 0 bucket checkpoint complete 2024-02-01T10:50:09.138 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=vclfcn-64 2024-02-01T10:50:09.138 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:50:09.138 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:50:09.139 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-64 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:50:09.188 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:50:09.189 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:50:09.250 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:50:09.249+0000 7f0fb2ac4980 1 bucket sync caught up with source: 2024-02-01T10:50:09.250 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [00000000004.82.6, 00000000002.65.1, 00000000004.56.6, 00000000002.45.1, 00000000004.56.6, 00000000002.29.1, 00000000002.54.1, 00000000004.65.6, 00000000002.70.1, 00000000002.54.1, 00000000002.54.1] 2024-02-01T10:50:09.250 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [00000000004.82.6, 00000000002.65.1, 00000000004.56.6, 00000000002.45.1, 00000000004.56.6, 00000000002.29.1, 00000000002.54.1, 00000000004.65.6, 00000000002.70.1, 00000000002.54.1, 00000000002.54.1] 2024-02-01T10:50:09.250 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:50:09.249+0000 7f0fb2ac4980 0 bucket checkpoint complete 2024-02-01T10:50:09.273 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=vclfcn-64 2024-02-01T10:50:09.273 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-64', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:50:09.274 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-64', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:50:09.274 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-64 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:50:09.314 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:50:09.314 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:50:09.375 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:50:09.373+0000 7effd3c60980 1 bucket sync caught up with source: 2024-02-01T10:50:09.375 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [00000000004.337.6, 00000000002.372.1, 00000000004.260.6, 00000000002.211.1, 00000000004.300.6, 00000000002.137.1, 00000000002.292.1, 00000000004.307.6, 00000000002.262.1, 00000000002.243.1, 00000000002.249.1] 2024-02-01T10:50:09.375 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [00000000004.337.6, 00000000002.372.1, 00000000004.260.6, 00000000002.211.1, 00000000004.300.6, 00000000002.137.1, 00000000002.292.1, 00000000004.307.6, 00000000002.262.1, 00000000002.243.1, 00000000002.249.1] 2024-02-01T10:50:09.375 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:50:09.373+0000 7effd3c60980 0 bucket checkpoint complete 2024-02-01T10:50:09.398 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=vclfcn-64 2024-02-01T10:50:09.398 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-64', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:50:09.398 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-64', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:50:09.398 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-64 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:50:09.483 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:50:09.483 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:50:09.548 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:50:09.549+0000 7f8990937980 1 bucket sync caught up with source: 2024-02-01T10:50:09.548 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [00000000004.337.6, 00000000002.372.1, 00000000004.260.6, 00000000002.211.1, 00000000004.300.6, 00000000002.137.1, 00000000002.292.1, 00000000004.307.6, 00000000002.262.1, 00000000002.243.1, 00000000002.249.1] 2024-02-01T10:50:09.548 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [00000000004.337.6, 00000000002.372.1, 00000000004.260.6, 00000000002.211.1, 00000000004.300.6, 00000000002.137.1, 00000000002.292.1, 00000000004.307.6, 00000000002.262.1, 00000000002.243.1, 00000000002.249.1] 2024-02-01T10:50:09.548 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:50:09.549+0000 7f8990937980 0 bucket checkpoint complete 2024-02-01T10:50:09.571 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=vclfcn-64 2024-02-01T10:50:09.571 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-64', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:50:09.571 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-64', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:50:09.571 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket vclfcn-64 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-02-01T10:50:09.656 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:50:09.657 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:50:09.716 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:50:09.717+0000 7f312663e980 1 bucket sync caught up with source: 2024-02-01T10:50:09.716 INFO:teuthology.orchestra.run.smithi148.stderr: local status: [00000000004.222.1, 00000000002.337.1, 00000000004.219.1, 00000000002.162.1, 00000000004.235.1, 00000000002.132.1, 00000000002.236.1, 00000000004.228.1, 00000000002.241.1, 00000000002.158.1, 00000000002.220.1] 2024-02-01T10:50:09.716 INFO:teuthology.orchestra.run.smithi148.stderr: remote markers: [00000000004.222.1, 00000000002.337.1, 00000000004.219.1, 00000000002.162.1, 00000000004.235.1, 00000000002.132.1, 00000000002.236.1, 00000000004.228.1, 00000000002.241.1, 00000000002.158.1, 00000000002.220.1] 2024-02-01T10:50:09.716 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:50:09.717+0000 7f312663e980 0 bucket checkpoint complete 2024-02-01T10:50:09.739 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=vclfcn-64 2024-02-01T10:50:09.739 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-64', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:50:09.739 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-64', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:50:09.740 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket vclfcn-64 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-02-01T10:50:09.790 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:50:09.790 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:50:09.856 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:50:09.856+0000 7f2e8b6d7980 1 bucket sync caught up with source: 2024-02-01T10:50:09.856 INFO:teuthology.orchestra.run.smithi187.stderr: local status: [00000000004.222.1, 00000000002.337.1, 00000000004.219.1, 00000000002.162.1, 00000000004.235.1, 00000000002.132.1, 00000000002.236.1, 00000000004.228.1, 00000000002.241.1, 00000000002.158.1, 00000000002.220.1] 2024-02-01T10:50:09.856 INFO:teuthology.orchestra.run.smithi187.stderr: remote markers: [00000000004.222.1, 00000000002.337.1, 00000000004.219.1, 00000000002.162.1, 00000000004.235.1, 00000000002.132.1, 00000000002.236.1, 00000000004.228.1, 00000000002.241.1, 00000000002.158.1, 00000000002.220.1] 2024-02-01T10:50:09.857 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:50:09.856+0000 7f2e8b6d7980 0 bucket checkpoint complete 2024-02-01T10:50:09.882 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-64 zones={test-zone2, test-zone1} 2024-02-01T10:50:09.883 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:09.883 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:09.883 DEBUG:boto:Method: HEAD 2024-02-01T10:50:09.883 DEBUG:boto:Path: /vclfcn-64/ 2024-02-01T10:50:09.884 DEBUG:boto:Data: 2024-02-01T10:50:09.884 DEBUG:boto:Headers: {} 2024-02-01T10:50:09.884 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:09.884 DEBUG:boto:Port: 8000 2024-02-01T10:50:09.885 DEBUG:boto:Params: {} 2024-02-01T10:50:09.885 DEBUG:boto:Token: None 2024-02-01T10:50:09.885 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:09 GMT /vclfcn-64/ 2024-02-01T10:50:09.886 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:XjHGYcaRZ5+zHLYD52MuMHJ8aRU= 2024-02-01T10:50:09.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:XjHGYcaRZ5+zHLYD52MuMHJ8aRU=', 'Content-Length': '0'} 2024-02-01T10:50:09.889 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000012f1527f76f96c6f-0065bb7761-5350-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:50:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:09.889 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:09.889 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:09.889 DEBUG:boto:Method: HEAD 2024-02-01T10:50:09.890 DEBUG:boto:Path: /vclfcn-64/ 2024-02-01T10:50:09.890 DEBUG:boto:Data: 2024-02-01T10:50:09.890 DEBUG:boto:Headers: {} 2024-02-01T10:50:09.890 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:09.891 DEBUG:boto:Port: 8000 2024-02-01T10:50:09.891 DEBUG:boto:Params: {} 2024-02-01T10:50:09.891 DEBUG:boto:Token: None 2024-02-01T10:50:09.891 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:09 GMT /vclfcn-64/ 2024-02-01T10:50:09.892 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:XjHGYcaRZ5+zHLYD52MuMHJ8aRU= 2024-02-01T10:50:09.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:XjHGYcaRZ5+zHLYD52MuMHJ8aRU=', 'Content-Length': '0'} 2024-02-01T10:50:09.896 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ec0791c019c925bd-0065bb7761-5765-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:50:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:09.896 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:50:09.897 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:09.897 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:09.897 DEBUG:boto:path=/vclfcn-64/?versions&max-keys=999 2024-02-01T10:50:09.897 DEBUG:boto:auth_path=/vclfcn-64/?versions&max-keys=999 2024-02-01T10:50:09.898 DEBUG:boto:Method: GET 2024-02-01T10:50:09.898 DEBUG:boto:Path: /vclfcn-64/?versions&max-keys=999 2024-02-01T10:50:09.898 DEBUG:boto:Data: 2024-02-01T10:50:09.898 DEBUG:boto:Headers: {} 2024-02-01T10:50:09.899 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:09.899 DEBUG:boto:Port: 8000 2024-02-01T10:50:09.899 DEBUG:boto:Params: {} 2024-02-01T10:50:09.899 DEBUG:boto:Token: None 2024-02-01T10:50:09.900 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:09 GMT /vclfcn-64/?versions 2024-02-01T10:50:09.900 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9vR2vmCHT0TxbFgOJKCqNiWd8+0= 2024-02-01T10:50:09.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9vR2vmCHT0TxbFgOJKCqNiWd8+0=', 'Content-Length': '0'} 2024-02-01T10:50:09.903 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e266d5c611252d0-0065bb7761-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:09.904 DEBUG:boto:b'vclfcn-64999falseobj1nulltrue2024-02-01T10:48:26.595Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:48:26.615Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:48:26.634Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:48:26.689Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-02-01T10:49:22.402Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-02-01T10:49:22.455Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-02-01T10:49:22.471Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-02-01T10:49:22.519Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:09.905 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-02-01T10:50:09.905 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-02-01T10:50:09.906 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-02-01T10:50:09.906 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-02-01T10:50:09.906 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-02-01T10:50:09.907 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-02-01T10:50:09.907 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-02-01T10:50:09.907 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-02-01T10:50:09.907 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:50:09.908 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:09.908 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:09.908 DEBUG:boto:path=/vclfcn-64/?versions&max-keys=999 2024-02-01T10:50:09.908 DEBUG:boto:auth_path=/vclfcn-64/?versions&max-keys=999 2024-02-01T10:50:09.909 DEBUG:boto:Method: GET 2024-02-01T10:50:09.909 DEBUG:boto:Path: /vclfcn-64/?versions&max-keys=999 2024-02-01T10:50:09.909 DEBUG:boto:Data: 2024-02-01T10:50:09.909 DEBUG:boto:Headers: {} 2024-02-01T10:50:09.909 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:09.910 DEBUG:boto:Port: 8000 2024-02-01T10:50:09.910 DEBUG:boto:Params: {} 2024-02-01T10:50:09.910 DEBUG:boto:Token: None 2024-02-01T10:50:09.911 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:09 GMT /vclfcn-64/?versions 2024-02-01T10:50:09.911 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9vR2vmCHT0TxbFgOJKCqNiWd8+0= 2024-02-01T10:50:09.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9vR2vmCHT0TxbFgOJKCqNiWd8+0=', 'Content-Length': '0'} 2024-02-01T10:50:09.916 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043b56c8be8fe0ede-0065bb7761-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:09.916 DEBUG:boto:b'vclfcn-64999falseobj1nulltrue2024-02-01T10:48:26.595Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:48:26.615Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:48:26.634Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:48:26.689Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-02-01T10:49:22.402Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-02-01T10:49:22.455Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-02-01T10:49:22.471Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-02-01T10:49:22.519Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:09.917 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-02-01T10:50:09.917 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-02-01T10:50:09.917 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-02-01T10:50:09.917 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-02-01T10:50:09.917 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-02-01T10:50:09.917 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-02-01T10:50:09.918 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-02-01T10:50:09.918 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-02-01T10:50:09.918 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:09.918 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:09.918 DEBUG:boto:path=/vclfcn-64/?versions&max-keys=999 2024-02-01T10:50:09.918 DEBUG:boto:auth_path=/vclfcn-64/?versions&max-keys=999 2024-02-01T10:50:09.918 DEBUG:boto:Method: GET 2024-02-01T10:50:09.918 DEBUG:boto:Path: /vclfcn-64/?versions&max-keys=999 2024-02-01T10:50:09.918 DEBUG:boto:Data: 2024-02-01T10:50:09.918 DEBUG:boto:Headers: {} 2024-02-01T10:50:09.918 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:09.919 DEBUG:boto:Port: 8000 2024-02-01T10:50:09.919 DEBUG:boto:Params: {} 2024-02-01T10:50:09.919 DEBUG:boto:Token: None 2024-02-01T10:50:09.919 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:09 GMT /vclfcn-64/?versions 2024-02-01T10:50:09.919 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9vR2vmCHT0TxbFgOJKCqNiWd8+0= 2024-02-01T10:50:09.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9vR2vmCHT0TxbFgOJKCqNiWd8+0=', 'Content-Length': '0'} 2024-02-01T10:50:09.922 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c43aad8f3fc9b21a-0065bb7761-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:09.923 DEBUG:boto:b'vclfcn-64999falseobj1nulltrue2024-02-01T10:48:26.595Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:48:26.615Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:48:26.634Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:48:26.689Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-02-01T10:49:22.402Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-02-01T10:49:22.455Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-02-01T10:49:22.471Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-02-01T10:49:22.519Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:09.923 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:09.924 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:09.924 DEBUG:boto:path=/vclfcn-64/?versions&max-keys=999 2024-02-01T10:50:09.924 DEBUG:boto:auth_path=/vclfcn-64/?versions&max-keys=999 2024-02-01T10:50:09.924 DEBUG:boto:Method: GET 2024-02-01T10:50:09.924 DEBUG:boto:Path: /vclfcn-64/?versions&max-keys=999 2024-02-01T10:50:09.924 DEBUG:boto:Data: 2024-02-01T10:50:09.924 DEBUG:boto:Headers: {} 2024-02-01T10:50:09.924 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:09.924 DEBUG:boto:Port: 8000 2024-02-01T10:50:09.924 DEBUG:boto:Params: {} 2024-02-01T10:50:09.925 DEBUG:boto:Token: None 2024-02-01T10:50:09.925 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:09 GMT /vclfcn-64/?versions 2024-02-01T10:50:09.925 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9vR2vmCHT0TxbFgOJKCqNiWd8+0= 2024-02-01T10:50:09.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9vR2vmCHT0TxbFgOJKCqNiWd8+0=', 'Content-Length': '0'} 2024-02-01T10:50:09.929 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ff0587e98c4af40-0065bb7761-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:09.930 DEBUG:boto:b'vclfcn-64999falseobj1nulltrue2024-02-01T10:48:26.595Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:48:26.615Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:48:26.634Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:48:26.689Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-02-01T10:49:22.402Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-02-01T10:49:22.455Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-02-01T10:49:22.471Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-02-01T10:49:22.519Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:09.930 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:50:09.930 DEBUG:boto:path=/vclfcn-64/obj1 2024-02-01T10:50:09.930 DEBUG:boto:auth_path=/vclfcn-64/obj1 2024-02-01T10:50:09.931 DEBUG:boto:path=/vclfcn-64/obj1?versionId=null 2024-02-01T10:50:09.931 DEBUG:boto:auth_path=/vclfcn-64/obj1?versionId=null 2024-02-01T10:50:09.931 DEBUG:boto:Method: GET 2024-02-01T10:50:09.931 DEBUG:boto:Path: /vclfcn-64/obj1?versionId=null 2024-02-01T10:50:09.931 DEBUG:boto:Data: 2024-02-01T10:50:09.931 DEBUG:boto:Headers: {} 2024-02-01T10:50:09.931 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:09.931 DEBUG:boto:Port: 8000 2024-02-01T10:50:09.931 DEBUG:boto:Params: {} 2024-02-01T10:50:09.931 DEBUG:boto:Token: None 2024-02-01T10:50:09.931 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:09 GMT /vclfcn-64/obj1?versionId=null 2024-02-01T10:50:09.932 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:MzH50mh31H7UAZfJQQUZhZUdaLc= 2024-02-01T10:50:09.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:MzH50mh31H7UAZfJQQUZhZUdaLc=', 'Content-Length': '0'} 2024-02-01T10:50:09.934 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007b1bb9a946072e04-0065bb7761-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:09.976 DEBUG:boto:path=/vclfcn-64/obj1 2024-02-01T10:50:09.976 DEBUG:boto:auth_path=/vclfcn-64/obj1 2024-02-01T10:50:09.976 DEBUG:boto:path=/vclfcn-64/obj1?versionId=null 2024-02-01T10:50:09.977 DEBUG:boto:auth_path=/vclfcn-64/obj1?versionId=null 2024-02-01T10:50:09.977 DEBUG:boto:Method: GET 2024-02-01T10:50:09.977 DEBUG:boto:Path: /vclfcn-64/obj1?versionId=null 2024-02-01T10:50:09.977 DEBUG:boto:Data: 2024-02-01T10:50:09.977 DEBUG:boto:Headers: {} 2024-02-01T10:50:09.977 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:09.978 DEBUG:boto:Port: 8000 2024-02-01T10:50:09.978 DEBUG:boto:Params: {} 2024-02-01T10:50:09.978 DEBUG:boto:Token: None 2024-02-01T10:50:09.978 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:09 GMT /vclfcn-64/obj1?versionId=null 2024-02-01T10:50:09.978 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:MzH50mh31H7UAZfJQQUZhZUdaLc= 2024-02-01T10:50:09.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:MzH50mh31H7UAZfJQQUZhZUdaLc=', 'Content-Length': '0'} 2024-02-01T10:50:09.981 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a276eb71d0ddd724-0065bb7761-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:09.981 DEBUG:boto:path=/vclfcn-64/obj1 2024-02-01T10:50:09.981 DEBUG:boto:auth_path=/vclfcn-64/obj1 2024-02-01T10:50:09.982 DEBUG:boto:path=/vclfcn-64/obj1?versionId=null 2024-02-01T10:50:09.982 DEBUG:boto:auth_path=/vclfcn-64/obj1?versionId=null 2024-02-01T10:50:09.982 DEBUG:boto:Method: HEAD 2024-02-01T10:50:09.982 DEBUG:boto:Path: /vclfcn-64/obj1?versionId=null 2024-02-01T10:50:09.982 DEBUG:boto:Data: 2024-02-01T10:50:09.982 DEBUG:boto:Headers: {} 2024-02-01T10:50:09.982 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:09.982 DEBUG:boto:Port: 8000 2024-02-01T10:50:09.982 DEBUG:boto:Params: {} 2024-02-01T10:50:09.982 DEBUG:boto:Token: None 2024-02-01T10:50:09.983 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:09 GMT /vclfcn-64/obj1?versionId=null 2024-02-01T10:50:09.983 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Lkjsty705uN7xoVntbYrvI53LkU= 2024-02-01T10:50:09.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Lkjsty705uN7xoVntbYrvI53LkU=', 'Content-Length': '0'} 2024-02-01T10:50:09.985 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000054e311503b809070-0065bb7761-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:09.986 DEBUG:boto:path=/vclfcn-64/obj1 2024-02-01T10:50:09.986 DEBUG:boto:auth_path=/vclfcn-64/obj1 2024-02-01T10:50:09.986 DEBUG:boto:path=/vclfcn-64/obj1?versionId=null 2024-02-01T10:50:09.986 DEBUG:boto:auth_path=/vclfcn-64/obj1?versionId=null 2024-02-01T10:50:09.986 DEBUG:boto:Method: HEAD 2024-02-01T10:50:09.986 DEBUG:boto:Path: /vclfcn-64/obj1?versionId=null 2024-02-01T10:50:09.986 DEBUG:boto:Data: 2024-02-01T10:50:09.986 DEBUG:boto:Headers: {} 2024-02-01T10:50:09.986 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:09.986 DEBUG:boto:Port: 8000 2024-02-01T10:50:09.987 DEBUG:boto:Params: {} 2024-02-01T10:50:09.987 DEBUG:boto:Token: None 2024-02-01T10:50:09.987 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:09 GMT /vclfcn-64/obj1?versionId=null 2024-02-01T10:50:09.987 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Lkjsty705uN7xoVntbYrvI53LkU= 2024-02-01T10:50:09.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Lkjsty705uN7xoVntbYrvI53LkU=', 'Content-Length': '0'} 2024-02-01T10:50:09.990 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008c71ff0e6e7a814b-0065bb7761-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:09.990 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:50:09.990 DEBUG:boto:path=/vclfcn-64/obj1 2024-02-01T10:50:09.990 DEBUG:boto:auth_path=/vclfcn-64/obj1 2024-02-01T10:50:09.990 DEBUG:boto:path=/vclfcn-64/obj1?versionId=null 2024-02-01T10:50:09.990 DEBUG:boto:auth_path=/vclfcn-64/obj1?versionId=null 2024-02-01T10:50:09.990 DEBUG:boto:Method: GET 2024-02-01T10:50:09.990 DEBUG:boto:Path: /vclfcn-64/obj1?versionId=null 2024-02-01T10:50:09.991 DEBUG:boto:Data: 2024-02-01T10:50:09.991 DEBUG:boto:Headers: {} 2024-02-01T10:50:09.991 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:09.991 DEBUG:boto:Port: 8000 2024-02-01T10:50:09.991 DEBUG:boto:Params: {} 2024-02-01T10:50:09.991 DEBUG:boto:Token: None 2024-02-01T10:50:09.991 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:09 GMT /vclfcn-64/obj1?versionId=null 2024-02-01T10:50:09.991 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:MzH50mh31H7UAZfJQQUZhZUdaLc= 2024-02-01T10:50:09.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:MzH50mh31H7UAZfJQQUZhZUdaLc=', 'Content-Length': '0'} 2024-02-01T10:50:09.994 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000102c818df3296f49-0065bb7761-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:09.994 DEBUG:boto:path=/vclfcn-64/obj1 2024-02-01T10:50:09.994 DEBUG:boto:auth_path=/vclfcn-64/obj1 2024-02-01T10:50:09.994 DEBUG:boto:path=/vclfcn-64/obj1?versionId=null 2024-02-01T10:50:09.995 DEBUG:boto:auth_path=/vclfcn-64/obj1?versionId=null 2024-02-01T10:50:09.995 DEBUG:boto:Method: GET 2024-02-01T10:50:09.995 DEBUG:boto:Path: /vclfcn-64/obj1?versionId=null 2024-02-01T10:50:09.995 DEBUG:boto:Data: 2024-02-01T10:50:09.995 DEBUG:boto:Headers: {} 2024-02-01T10:50:09.995 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:09.995 DEBUG:boto:Port: 8000 2024-02-01T10:50:09.995 DEBUG:boto:Params: {} 2024-02-01T10:50:09.995 DEBUG:boto:Token: None 2024-02-01T10:50:09.995 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:09 GMT /vclfcn-64/obj1?versionId=null 2024-02-01T10:50:09.996 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:MzH50mh31H7UAZfJQQUZhZUdaLc= 2024-02-01T10:50:09.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:09 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:MzH50mh31H7UAZfJQQUZhZUdaLc=', 'Content-Length': '0'} 2024-02-01T10:50:09.999 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000006125ebaa7c91949-0065bb7761-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:09 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:10.044 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:10.044 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:10.044 DEBUG:boto:path=/vclfcn-64/?prefix=obj1 2024-02-01T10:50:10.044 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj1 2024-02-01T10:50:10.045 DEBUG:boto:Method: GET 2024-02-01T10:50:10.045 DEBUG:boto:Path: /vclfcn-64/?prefix=obj1 2024-02-01T10:50:10.045 DEBUG:boto:Data: 2024-02-01T10:50:10.045 DEBUG:boto:Headers: {} 2024-02-01T10:50:10.045 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:10.045 DEBUG:boto:Port: 8000 2024-02-01T10:50:10.045 DEBUG:boto:Params: {} 2024-02-01T10:50:10.045 DEBUG:boto:Token: None 2024-02-01T10:50:10.045 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:10 GMT /vclfcn-64/ 2024-02-01T10:50:10.046 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vbNmSpbPy6Szb1Rj2it0MD3AnJ4= 2024-02-01T10:50:10.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vbNmSpbPy6Szb1Rj2it0MD3AnJ4=', 'Content-Length': '0'} 2024-02-01T10:50:10.050 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008deb3433e1b61025-0065bb7762-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:10.050 DEBUG:boto:b'vclfcn-64obj11000falseobj12024-02-01T10:48:26.595Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:10.051 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:10.051 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:10.051 DEBUG:boto:path=/vclfcn-64/?prefix=obj1 2024-02-01T10:50:10.051 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj1 2024-02-01T10:50:10.051 DEBUG:boto:Method: GET 2024-02-01T10:50:10.051 DEBUG:boto:Path: /vclfcn-64/?prefix=obj1 2024-02-01T10:50:10.052 DEBUG:boto:Data: 2024-02-01T10:50:10.052 DEBUG:boto:Headers: {} 2024-02-01T10:50:10.052 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:10.052 DEBUG:boto:Port: 8000 2024-02-01T10:50:10.052 DEBUG:boto:Params: {} 2024-02-01T10:50:10.052 DEBUG:boto:Token: None 2024-02-01T10:50:10.052 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:10 GMT /vclfcn-64/ 2024-02-01T10:50:10.052 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vbNmSpbPy6Szb1Rj2it0MD3AnJ4= 2024-02-01T10:50:10.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vbNmSpbPy6Szb1Rj2it0MD3AnJ4=', 'Content-Length': '0'} 2024-02-01T10:50:10.056 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060db372a2c9a1bbb-0065bb7762-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:10.100 DEBUG:boto:b'vclfcn-64obj11000falseobj12024-02-01T10:48:26.595Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:10.101 DEBUG:boto:path=/vclfcn-64/obj1 2024-02-01T10:50:10.101 DEBUG:boto:auth_path=/vclfcn-64/obj1 2024-02-01T10:50:10.101 DEBUG:boto:Method: HEAD 2024-02-01T10:50:10.101 DEBUG:boto:Path: /vclfcn-64/obj1 2024-02-01T10:50:10.101 DEBUG:boto:Data: 2024-02-01T10:50:10.101 DEBUG:boto:Headers: {} 2024-02-01T10:50:10.102 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:10.102 DEBUG:boto:Port: 8000 2024-02-01T10:50:10.102 DEBUG:boto:Params: {} 2024-02-01T10:50:10.102 DEBUG:boto:Token: None 2024-02-01T10:50:10.102 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:10 GMT /vclfcn-64/obj1 2024-02-01T10:50:10.102 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:6mPWAA7IlAM9lNAFOO9qK23TXX4= 2024-02-01T10:50:10.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:6mPWAA7IlAM9lNAFOO9qK23TXX4=', 'Content-Length': '0'} 2024-02-01T10:50:10.105 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002f50551548c34c8b-0065bb7762-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:10.106 DEBUG:boto:path=/vclfcn-64/obj1 2024-02-01T10:50:10.106 DEBUG:boto:auth_path=/vclfcn-64/obj1 2024-02-01T10:50:10.106 DEBUG:boto:Method: HEAD 2024-02-01T10:50:10.106 DEBUG:boto:Path: /vclfcn-64/obj1 2024-02-01T10:50:10.106 DEBUG:boto:Data: 2024-02-01T10:50:10.106 DEBUG:boto:Headers: {} 2024-02-01T10:50:10.106 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:10.106 DEBUG:boto:Port: 8000 2024-02-01T10:50:10.106 DEBUG:boto:Params: {} 2024-02-01T10:50:10.107 DEBUG:boto:Token: None 2024-02-01T10:50:10.107 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:10 GMT /vclfcn-64/obj1 2024-02-01T10:50:10.107 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:6mPWAA7IlAM9lNAFOO9qK23TXX4= 2024-02-01T10:50:10.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:6mPWAA7IlAM9lNAFOO9qK23TXX4=', 'Content-Length': '0'} 2024-02-01T10:50:10.109 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000864a2348c17ea1be-0065bb7762-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:10.110 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:50:10.110 DEBUG:boto:path=/vclfcn-64/obj1 2024-02-01T10:50:10.110 DEBUG:boto:auth_path=/vclfcn-64/obj1 2024-02-01T10:50:10.110 DEBUG:boto:Method: GET 2024-02-01T10:50:10.110 DEBUG:boto:Path: /vclfcn-64/obj1 2024-02-01T10:50:10.110 DEBUG:boto:Data: 2024-02-01T10:50:10.111 DEBUG:boto:Headers: {} 2024-02-01T10:50:10.111 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:10.111 DEBUG:boto:Port: 8000 2024-02-01T10:50:10.111 DEBUG:boto:Params: {} 2024-02-01T10:50:10.111 DEBUG:boto:Token: None 2024-02-01T10:50:10.111 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:10 GMT /vclfcn-64/obj1 2024-02-01T10:50:10.111 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:AGJq0nYnp7XR4fmVWdT670ExIiE= 2024-02-01T10:50:10.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:AGJq0nYnp7XR4fmVWdT670ExIiE=', 'Content-Length': '0'} 2024-02-01T10:50:10.114 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ab31b5dc6c712407-0065bb7762-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:10.156 DEBUG:boto:path=/vclfcn-64/obj1 2024-02-01T10:50:10.156 DEBUG:boto:auth_path=/vclfcn-64/obj1 2024-02-01T10:50:10.156 DEBUG:boto:Method: GET 2024-02-01T10:50:10.157 DEBUG:boto:Path: /vclfcn-64/obj1 2024-02-01T10:50:10.157 DEBUG:boto:Data: 2024-02-01T10:50:10.157 DEBUG:boto:Headers: {} 2024-02-01T10:50:10.157 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:10.157 DEBUG:boto:Port: 8000 2024-02-01T10:50:10.157 DEBUG:boto:Params: {} 2024-02-01T10:50:10.157 DEBUG:boto:Token: None 2024-02-01T10:50:10.158 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:10 GMT /vclfcn-64/obj1 2024-02-01T10:50:10.158 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:AGJq0nYnp7XR4fmVWdT670ExIiE= 2024-02-01T10:50:10.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:AGJq0nYnp7XR4fmVWdT670ExIiE=', 'Content-Length': '0'} 2024-02-01T10:50:10.161 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000da556c1dd3080b1d-0065bb7762-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:10.208 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:10.208 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:10.208 DEBUG:boto:path=/vclfcn-64/?prefix=obj1 2024-02-01T10:50:10.208 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj1 2024-02-01T10:50:10.208 DEBUG:boto:Method: GET 2024-02-01T10:50:10.209 DEBUG:boto:Path: /vclfcn-64/?prefix=obj1 2024-02-01T10:50:10.209 DEBUG:boto:Data: 2024-02-01T10:50:10.209 DEBUG:boto:Headers: {} 2024-02-01T10:50:10.209 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:10.209 DEBUG:boto:Port: 8000 2024-02-01T10:50:10.209 DEBUG:boto:Params: {} 2024-02-01T10:50:10.209 DEBUG:boto:Token: None 2024-02-01T10:50:10.209 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:10 GMT /vclfcn-64/ 2024-02-01T10:50:10.209 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vbNmSpbPy6Szb1Rj2it0MD3AnJ4= 2024-02-01T10:50:10.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vbNmSpbPy6Szb1Rj2it0MD3AnJ4=', 'Content-Length': '0'} 2024-02-01T10:50:10.214 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6777066bd0da911-0065bb7762-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:10.214 DEBUG:boto:b'vclfcn-64obj11000falseobj12024-02-01T10:48:26.595Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:10.214 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:10.214 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:10.215 DEBUG:boto:path=/vclfcn-64/?prefix=obj1 2024-02-01T10:50:10.215 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj1 2024-02-01T10:50:10.215 DEBUG:boto:Method: GET 2024-02-01T10:50:10.215 DEBUG:boto:Path: /vclfcn-64/?prefix=obj1 2024-02-01T10:50:10.215 DEBUG:boto:Data: 2024-02-01T10:50:10.215 DEBUG:boto:Headers: {} 2024-02-01T10:50:10.215 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:10.215 DEBUG:boto:Port: 8000 2024-02-01T10:50:10.215 DEBUG:boto:Params: {} 2024-02-01T10:50:10.216 DEBUG:boto:Token: None 2024-02-01T10:50:10.216 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:10 GMT /vclfcn-64/ 2024-02-01T10:50:10.216 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vbNmSpbPy6Szb1Rj2it0MD3AnJ4= 2024-02-01T10:50:10.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vbNmSpbPy6Szb1Rj2it0MD3AnJ4=', 'Content-Length': '0'} 2024-02-01T10:50:10.220 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088212a2c522d0be4-0065bb7762-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:10.267 DEBUG:boto:b'vclfcn-64obj11000falseobj12024-02-01T10:48:26.595Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:10.268 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:50:10.268 DEBUG:boto:path=/vclfcn-64/obj2 2024-02-01T10:50:10.268 DEBUG:boto:auth_path=/vclfcn-64/obj2 2024-02-01T10:50:10.268 DEBUG:boto:path=/vclfcn-64/obj2?versionId=null 2024-02-01T10:50:10.268 DEBUG:boto:auth_path=/vclfcn-64/obj2?versionId=null 2024-02-01T10:50:10.268 DEBUG:boto:Method: GET 2024-02-01T10:50:10.268 DEBUG:boto:Path: /vclfcn-64/obj2?versionId=null 2024-02-01T10:50:10.268 DEBUG:boto:Data: 2024-02-01T10:50:10.270 DEBUG:boto:Headers: {} 2024-02-01T10:50:10.270 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:10.271 DEBUG:boto:Port: 8000 2024-02-01T10:50:10.271 DEBUG:boto:Params: {} 2024-02-01T10:50:10.271 DEBUG:boto:Token: None 2024-02-01T10:50:10.271 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:10 GMT /vclfcn-64/obj2?versionId=null 2024-02-01T10:50:10.272 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:VGqNx5ltOFmEy1mGozFz/DORp1Y= 2024-02-01T10:50:10.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:VGqNx5ltOFmEy1mGozFz/DORp1Y=', 'Content-Length': '0'} 2024-02-01T10:50:10.275 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a1b8be1348ad5d90-0065bb7762-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:10.276 DEBUG:boto:path=/vclfcn-64/obj2 2024-02-01T10:50:10.276 DEBUG:boto:auth_path=/vclfcn-64/obj2 2024-02-01T10:50:10.276 DEBUG:boto:path=/vclfcn-64/obj2?versionId=null 2024-02-01T10:50:10.276 DEBUG:boto:auth_path=/vclfcn-64/obj2?versionId=null 2024-02-01T10:50:10.276 DEBUG:boto:Method: GET 2024-02-01T10:50:10.276 DEBUG:boto:Path: /vclfcn-64/obj2?versionId=null 2024-02-01T10:50:10.277 DEBUG:boto:Data: 2024-02-01T10:50:10.277 DEBUG:boto:Headers: {} 2024-02-01T10:50:10.277 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:10.277 DEBUG:boto:Port: 8000 2024-02-01T10:50:10.277 DEBUG:boto:Params: {} 2024-02-01T10:50:10.277 DEBUG:boto:Token: None 2024-02-01T10:50:10.277 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:10 GMT /vclfcn-64/obj2?versionId=null 2024-02-01T10:50:10.277 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:VGqNx5ltOFmEy1mGozFz/DORp1Y= 2024-02-01T10:50:10.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:VGqNx5ltOFmEy1mGozFz/DORp1Y=', 'Content-Length': '0'} 2024-02-01T10:50:10.280 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bb8b03d85929b3a4-0065bb7762-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:10.324 DEBUG:boto:path=/vclfcn-64/obj2 2024-02-01T10:50:10.324 DEBUG:boto:auth_path=/vclfcn-64/obj2 2024-02-01T10:50:10.324 DEBUG:boto:path=/vclfcn-64/obj2?versionId=null 2024-02-01T10:50:10.324 DEBUG:boto:auth_path=/vclfcn-64/obj2?versionId=null 2024-02-01T10:50:10.325 DEBUG:boto:Method: HEAD 2024-02-01T10:50:10.325 DEBUG:boto:Path: /vclfcn-64/obj2?versionId=null 2024-02-01T10:50:10.325 DEBUG:boto:Data: 2024-02-01T10:50:10.325 DEBUG:boto:Headers: {} 2024-02-01T10:50:10.325 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:10.325 DEBUG:boto:Port: 8000 2024-02-01T10:50:10.325 DEBUG:boto:Params: {} 2024-02-01T10:50:10.325 DEBUG:boto:Token: None 2024-02-01T10:50:10.325 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:10 GMT /vclfcn-64/obj2?versionId=null 2024-02-01T10:50:10.326 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:L4Tb6+QSLaw9a/LEzLE++9UNxLo= 2024-02-01T10:50:10.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:L4Tb6+QSLaw9a/LEzLE++9UNxLo=', 'Content-Length': '0'} 2024-02-01T10:50:10.329 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000579902d49eafb45-0065bb7762-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:10.329 DEBUG:boto:path=/vclfcn-64/obj2 2024-02-01T10:50:10.329 DEBUG:boto:auth_path=/vclfcn-64/obj2 2024-02-01T10:50:10.329 DEBUG:boto:path=/vclfcn-64/obj2?versionId=null 2024-02-01T10:50:10.329 DEBUG:boto:auth_path=/vclfcn-64/obj2?versionId=null 2024-02-01T10:50:10.329 DEBUG:boto:Method: HEAD 2024-02-01T10:50:10.330 DEBUG:boto:Path: /vclfcn-64/obj2?versionId=null 2024-02-01T10:50:10.330 DEBUG:boto:Data: 2024-02-01T10:50:10.330 DEBUG:boto:Headers: {} 2024-02-01T10:50:10.330 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:10.330 DEBUG:boto:Port: 8000 2024-02-01T10:50:10.330 DEBUG:boto:Params: {} 2024-02-01T10:50:10.330 DEBUG:boto:Token: None 2024-02-01T10:50:10.330 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:10 GMT /vclfcn-64/obj2?versionId=null 2024-02-01T10:50:10.331 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:L4Tb6+QSLaw9a/LEzLE++9UNxLo= 2024-02-01T10:50:10.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:L4Tb6+QSLaw9a/LEzLE++9UNxLo=', 'Content-Length': '0'} 2024-02-01T10:50:10.334 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000466c5a51cd8b1ee3-0065bb7762-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:10.334 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:50:10.335 DEBUG:boto:path=/vclfcn-64/obj2 2024-02-01T10:50:10.335 DEBUG:boto:auth_path=/vclfcn-64/obj2 2024-02-01T10:50:10.335 DEBUG:boto:path=/vclfcn-64/obj2?versionId=null 2024-02-01T10:50:10.335 DEBUG:boto:auth_path=/vclfcn-64/obj2?versionId=null 2024-02-01T10:50:10.335 DEBUG:boto:Method: GET 2024-02-01T10:50:10.335 DEBUG:boto:Path: /vclfcn-64/obj2?versionId=null 2024-02-01T10:50:10.335 DEBUG:boto:Data: 2024-02-01T10:50:10.336 DEBUG:boto:Headers: {} 2024-02-01T10:50:10.336 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:10.336 DEBUG:boto:Port: 8000 2024-02-01T10:50:10.336 DEBUG:boto:Params: {} 2024-02-01T10:50:10.336 DEBUG:boto:Token: None 2024-02-01T10:50:10.337 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:10 GMT /vclfcn-64/obj2?versionId=null 2024-02-01T10:50:10.337 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:VGqNx5ltOFmEy1mGozFz/DORp1Y= 2024-02-01T10:50:10.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:VGqNx5ltOFmEy1mGozFz/DORp1Y=', 'Content-Length': '0'} 2024-02-01T10:50:10.340 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000641d51dcf15f6fd1-0065bb7762-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:10.384 DEBUG:boto:path=/vclfcn-64/obj2 2024-02-01T10:50:10.384 DEBUG:boto:auth_path=/vclfcn-64/obj2 2024-02-01T10:50:10.384 DEBUG:boto:path=/vclfcn-64/obj2?versionId=null 2024-02-01T10:50:10.384 DEBUG:boto:auth_path=/vclfcn-64/obj2?versionId=null 2024-02-01T10:50:10.385 DEBUG:boto:Method: GET 2024-02-01T10:50:10.385 DEBUG:boto:Path: /vclfcn-64/obj2?versionId=null 2024-02-01T10:50:10.385 DEBUG:boto:Data: 2024-02-01T10:50:10.385 DEBUG:boto:Headers: {} 2024-02-01T10:50:10.385 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:10.385 DEBUG:boto:Port: 8000 2024-02-01T10:50:10.385 DEBUG:boto:Params: {} 2024-02-01T10:50:10.385 DEBUG:boto:Token: None 2024-02-01T10:50:10.385 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:10 GMT /vclfcn-64/obj2?versionId=null 2024-02-01T10:50:10.386 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:VGqNx5ltOFmEy1mGozFz/DORp1Y= 2024-02-01T10:50:10.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:VGqNx5ltOFmEy1mGozFz/DORp1Y=', 'Content-Length': '0'} 2024-02-01T10:50:10.389 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c14a768ffa350abb-0065bb7762-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:10.432 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:10.432 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:10.432 DEBUG:boto:path=/vclfcn-64/?prefix=obj2 2024-02-01T10:50:10.432 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj2 2024-02-01T10:50:10.433 DEBUG:boto:Method: GET 2024-02-01T10:50:10.433 DEBUG:boto:Path: /vclfcn-64/?prefix=obj2 2024-02-01T10:50:10.433 DEBUG:boto:Data: 2024-02-01T10:50:10.433 DEBUG:boto:Headers: {} 2024-02-01T10:50:10.433 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:10.433 DEBUG:boto:Port: 8000 2024-02-01T10:50:10.433 DEBUG:boto:Params: {} 2024-02-01T10:50:10.433 DEBUG:boto:Token: None 2024-02-01T10:50:10.433 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:10 GMT /vclfcn-64/ 2024-02-01T10:50:10.434 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vbNmSpbPy6Szb1Rj2it0MD3AnJ4= 2024-02-01T10:50:10.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vbNmSpbPy6Szb1Rj2it0MD3AnJ4=', 'Content-Length': '0'} 2024-02-01T10:50:10.438 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c8a960f2adab480-0065bb7762-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:10.480 DEBUG:boto:b'vclfcn-64obj21000falseobj22024-02-01T10:48:26.615Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:10.481 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:10.481 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:10.481 DEBUG:boto:path=/vclfcn-64/?prefix=obj2 2024-02-01T10:50:10.481 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj2 2024-02-01T10:50:10.481 DEBUG:boto:Method: GET 2024-02-01T10:50:10.482 DEBUG:boto:Path: /vclfcn-64/?prefix=obj2 2024-02-01T10:50:10.482 DEBUG:boto:Data: 2024-02-01T10:50:10.482 DEBUG:boto:Headers: {} 2024-02-01T10:50:10.482 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:10.482 DEBUG:boto:Port: 8000 2024-02-01T10:50:10.482 DEBUG:boto:Params: {} 2024-02-01T10:50:10.483 DEBUG:boto:Token: None 2024-02-01T10:50:10.483 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:10 GMT /vclfcn-64/ 2024-02-01T10:50:10.483 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vbNmSpbPy6Szb1Rj2it0MD3AnJ4= 2024-02-01T10:50:10.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vbNmSpbPy6Szb1Rj2it0MD3AnJ4=', 'Content-Length': '0'} 2024-02-01T10:50:10.488 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e41f3547e9701cc-0065bb7762-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:10.532 DEBUG:boto:b'vclfcn-64obj21000falseobj22024-02-01T10:48:26.615Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:10.533 DEBUG:boto:path=/vclfcn-64/obj2 2024-02-01T10:50:10.533 DEBUG:boto:auth_path=/vclfcn-64/obj2 2024-02-01T10:50:10.533 DEBUG:boto:Method: HEAD 2024-02-01T10:50:10.533 DEBUG:boto:Path: /vclfcn-64/obj2 2024-02-01T10:50:10.533 DEBUG:boto:Data: 2024-02-01T10:50:10.533 DEBUG:boto:Headers: {} 2024-02-01T10:50:10.533 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:10.533 DEBUG:boto:Port: 8000 2024-02-01T10:50:10.533 DEBUG:boto:Params: {} 2024-02-01T10:50:10.534 DEBUG:boto:Token: None 2024-02-01T10:50:10.534 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:10 GMT /vclfcn-64/obj2 2024-02-01T10:50:10.534 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:E0lRCVP3rJtHZGgbpIZrm1xB9ok= 2024-02-01T10:50:10.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:E0lRCVP3rJtHZGgbpIZrm1xB9ok=', 'Content-Length': '0'} 2024-02-01T10:50:10.537 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000935541c93751836a-0065bb7762-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:10.537 DEBUG:boto:path=/vclfcn-64/obj2 2024-02-01T10:50:10.537 DEBUG:boto:auth_path=/vclfcn-64/obj2 2024-02-01T10:50:10.537 DEBUG:boto:Method: HEAD 2024-02-01T10:50:10.538 DEBUG:boto:Path: /vclfcn-64/obj2 2024-02-01T10:50:10.538 DEBUG:boto:Data: 2024-02-01T10:50:10.538 DEBUG:boto:Headers: {} 2024-02-01T10:50:10.538 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:10.538 DEBUG:boto:Port: 8000 2024-02-01T10:50:10.538 DEBUG:boto:Params: {} 2024-02-01T10:50:10.538 DEBUG:boto:Token: None 2024-02-01T10:50:10.538 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:10 GMT /vclfcn-64/obj2 2024-02-01T10:50:10.538 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:E0lRCVP3rJtHZGgbpIZrm1xB9ok= 2024-02-01T10:50:10.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:E0lRCVP3rJtHZGgbpIZrm1xB9ok=', 'Content-Length': '0'} 2024-02-01T10:50:10.542 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005af015dda498500c-0065bb7762-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:10.543 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:50:10.543 DEBUG:boto:path=/vclfcn-64/obj2 2024-02-01T10:50:10.543 DEBUG:boto:auth_path=/vclfcn-64/obj2 2024-02-01T10:50:10.543 DEBUG:boto:Method: GET 2024-02-01T10:50:10.543 DEBUG:boto:Path: /vclfcn-64/obj2 2024-02-01T10:50:10.543 DEBUG:boto:Data: 2024-02-01T10:50:10.543 DEBUG:boto:Headers: {} 2024-02-01T10:50:10.543 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:10.543 DEBUG:boto:Port: 8000 2024-02-01T10:50:10.543 DEBUG:boto:Params: {} 2024-02-01T10:50:10.544 DEBUG:boto:Token: None 2024-02-01T10:50:10.544 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:10 GMT /vclfcn-64/obj2 2024-02-01T10:50:10.544 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:POmDjUeuMxVRfW6vNTTEwVBdpCc= 2024-02-01T10:50:10.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:POmDjUeuMxVRfW6vNTTEwVBdpCc=', 'Content-Length': '0'} 2024-02-01T10:50:10.547 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000025e40ed2a782aa82-0065bb7762-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:10.588 DEBUG:boto:path=/vclfcn-64/obj2 2024-02-01T10:50:10.588 DEBUG:boto:auth_path=/vclfcn-64/obj2 2024-02-01T10:50:10.588 DEBUG:boto:Method: GET 2024-02-01T10:50:10.588 DEBUG:boto:Path: /vclfcn-64/obj2 2024-02-01T10:50:10.589 DEBUG:boto:Data: 2024-02-01T10:50:10.589 DEBUG:boto:Headers: {} 2024-02-01T10:50:10.589 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:10.589 DEBUG:boto:Port: 8000 2024-02-01T10:50:10.589 DEBUG:boto:Params: {} 2024-02-01T10:50:10.589 DEBUG:boto:Token: None 2024-02-01T10:50:10.589 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:10 GMT /vclfcn-64/obj2 2024-02-01T10:50:10.589 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:POmDjUeuMxVRfW6vNTTEwVBdpCc= 2024-02-01T10:50:10.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:POmDjUeuMxVRfW6vNTTEwVBdpCc=', 'Content-Length': '0'} 2024-02-01T10:50:10.593 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000857e20d821465ae8-0065bb7762-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:10.640 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:10.640 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:10.640 DEBUG:boto:path=/vclfcn-64/?prefix=obj2 2024-02-01T10:50:10.640 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj2 2024-02-01T10:50:10.641 DEBUG:boto:Method: GET 2024-02-01T10:50:10.641 DEBUG:boto:Path: /vclfcn-64/?prefix=obj2 2024-02-01T10:50:10.641 DEBUG:boto:Data: 2024-02-01T10:50:10.641 DEBUG:boto:Headers: {} 2024-02-01T10:50:10.641 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:10.641 DEBUG:boto:Port: 8000 2024-02-01T10:50:10.641 DEBUG:boto:Params: {} 2024-02-01T10:50:10.641 DEBUG:boto:Token: None 2024-02-01T10:50:10.641 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:10 GMT /vclfcn-64/ 2024-02-01T10:50:10.642 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vbNmSpbPy6Szb1Rj2it0MD3AnJ4= 2024-02-01T10:50:10.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vbNmSpbPy6Szb1Rj2it0MD3AnJ4=', 'Content-Length': '0'} 2024-02-01T10:50:10.646 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b4b354760026168-0065bb7762-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:10.692 DEBUG:boto:b'vclfcn-64obj21000falseobj22024-02-01T10:48:26.615Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:10.693 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:10.693 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:10.693 DEBUG:boto:path=/vclfcn-64/?prefix=obj2 2024-02-01T10:50:10.693 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj2 2024-02-01T10:50:10.693 DEBUG:boto:Method: GET 2024-02-01T10:50:10.693 DEBUG:boto:Path: /vclfcn-64/?prefix=obj2 2024-02-01T10:50:10.693 DEBUG:boto:Data: 2024-02-01T10:50:10.693 DEBUG:boto:Headers: {} 2024-02-01T10:50:10.693 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:10.693 DEBUG:boto:Port: 8000 2024-02-01T10:50:10.693 DEBUG:boto:Params: {} 2024-02-01T10:50:10.694 DEBUG:boto:Token: None 2024-02-01T10:50:10.694 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:10 GMT /vclfcn-64/ 2024-02-01T10:50:10.694 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vbNmSpbPy6Szb1Rj2it0MD3AnJ4= 2024-02-01T10:50:10.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vbNmSpbPy6Szb1Rj2it0MD3AnJ4=', 'Content-Length': '0'} 2024-02-01T10:50:10.698 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba689d48eabc9e7f-0065bb7762-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:10.699 DEBUG:boto:b'vclfcn-64obj21000falseobj22024-02-01T10:48:26.615Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:10.699 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:50:10.699 DEBUG:boto:path=/vclfcn-64/obj3 2024-02-01T10:50:10.699 DEBUG:boto:auth_path=/vclfcn-64/obj3 2024-02-01T10:50:10.699 DEBUG:boto:path=/vclfcn-64/obj3?versionId=null 2024-02-01T10:50:10.699 DEBUG:boto:auth_path=/vclfcn-64/obj3?versionId=null 2024-02-01T10:50:10.700 DEBUG:boto:Method: GET 2024-02-01T10:50:10.700 DEBUG:boto:Path: /vclfcn-64/obj3?versionId=null 2024-02-01T10:50:10.700 DEBUG:boto:Data: 2024-02-01T10:50:10.700 DEBUG:boto:Headers: {} 2024-02-01T10:50:10.700 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:10.700 DEBUG:boto:Port: 8000 2024-02-01T10:50:10.700 DEBUG:boto:Params: {} 2024-02-01T10:50:10.700 DEBUG:boto:Token: None 2024-02-01T10:50:10.700 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:10 GMT /vclfcn-64/obj3?versionId=null 2024-02-01T10:50:10.701 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9UejZK4G7dFxOzkzJxy34cvWvp8= 2024-02-01T10:50:10.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9UejZK4G7dFxOzkzJxy34cvWvp8=', 'Content-Length': '0'} 2024-02-01T10:50:10.704 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e0bba4817b6f48f0-0065bb7762-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:10.704 DEBUG:boto:path=/vclfcn-64/obj3 2024-02-01T10:50:10.704 DEBUG:boto:auth_path=/vclfcn-64/obj3 2024-02-01T10:50:10.704 DEBUG:boto:path=/vclfcn-64/obj3?versionId=null 2024-02-01T10:50:10.705 DEBUG:boto:auth_path=/vclfcn-64/obj3?versionId=null 2024-02-01T10:50:10.705 DEBUG:boto:Method: GET 2024-02-01T10:50:10.705 DEBUG:boto:Path: /vclfcn-64/obj3?versionId=null 2024-02-01T10:50:10.705 DEBUG:boto:Data: 2024-02-01T10:50:10.705 DEBUG:boto:Headers: {} 2024-02-01T10:50:10.705 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:10.705 DEBUG:boto:Port: 8000 2024-02-01T10:50:10.705 DEBUG:boto:Params: {} 2024-02-01T10:50:10.705 DEBUG:boto:Token: None 2024-02-01T10:50:10.705 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:10 GMT /vclfcn-64/obj3?versionId=null 2024-02-01T10:50:10.706 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9UejZK4G7dFxOzkzJxy34cvWvp8= 2024-02-01T10:50:10.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9UejZK4G7dFxOzkzJxy34cvWvp8=', 'Content-Length': '0'} 2024-02-01T10:50:10.708 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000096c5d71f5247b3cd-0065bb7762-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:10.752 DEBUG:boto:path=/vclfcn-64/obj3 2024-02-01T10:50:10.752 DEBUG:boto:auth_path=/vclfcn-64/obj3 2024-02-01T10:50:10.752 DEBUG:boto:path=/vclfcn-64/obj3?versionId=null 2024-02-01T10:50:10.752 DEBUG:boto:auth_path=/vclfcn-64/obj3?versionId=null 2024-02-01T10:50:10.753 DEBUG:boto:Method: HEAD 2024-02-01T10:50:10.753 DEBUG:boto:Path: /vclfcn-64/obj3?versionId=null 2024-02-01T10:50:10.753 DEBUG:boto:Data: 2024-02-01T10:50:10.753 DEBUG:boto:Headers: {} 2024-02-01T10:50:10.753 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:10.753 DEBUG:boto:Port: 8000 2024-02-01T10:50:10.753 DEBUG:boto:Params: {} 2024-02-01T10:50:10.753 DEBUG:boto:Token: None 2024-02-01T10:50:10.753 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:10 GMT /vclfcn-64/obj3?versionId=null 2024-02-01T10:50:10.753 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BJA3WamAqVNMgkU+fe7I7tvIerU= 2024-02-01T10:50:10.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BJA3WamAqVNMgkU+fe7I7tvIerU=', 'Content-Length': '0'} 2024-02-01T10:50:10.756 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007377bc28a5e6dbe3-0065bb7762-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:10.757 DEBUG:boto:path=/vclfcn-64/obj3 2024-02-01T10:50:10.757 DEBUG:boto:auth_path=/vclfcn-64/obj3 2024-02-01T10:50:10.757 DEBUG:boto:path=/vclfcn-64/obj3?versionId=null 2024-02-01T10:50:10.757 DEBUG:boto:auth_path=/vclfcn-64/obj3?versionId=null 2024-02-01T10:50:10.757 DEBUG:boto:Method: HEAD 2024-02-01T10:50:10.757 DEBUG:boto:Path: /vclfcn-64/obj3?versionId=null 2024-02-01T10:50:10.757 DEBUG:boto:Data: 2024-02-01T10:50:10.757 DEBUG:boto:Headers: {} 2024-02-01T10:50:10.758 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:10.758 DEBUG:boto:Port: 8000 2024-02-01T10:50:10.758 DEBUG:boto:Params: {} 2024-02-01T10:50:10.758 DEBUG:boto:Token: None 2024-02-01T10:50:10.758 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:10 GMT /vclfcn-64/obj3?versionId=null 2024-02-01T10:50:10.758 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BJA3WamAqVNMgkU+fe7I7tvIerU= 2024-02-01T10:50:10.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BJA3WamAqVNMgkU+fe7I7tvIerU=', 'Content-Length': '0'} 2024-02-01T10:50:10.761 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000037a902e0935472c8-0065bb7762-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:10.762 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:50:10.762 DEBUG:boto:path=/vclfcn-64/obj3 2024-02-01T10:50:10.762 DEBUG:boto:auth_path=/vclfcn-64/obj3 2024-02-01T10:50:10.762 DEBUG:boto:path=/vclfcn-64/obj3?versionId=null 2024-02-01T10:50:10.762 DEBUG:boto:auth_path=/vclfcn-64/obj3?versionId=null 2024-02-01T10:50:10.762 DEBUG:boto:Method: GET 2024-02-01T10:50:10.762 DEBUG:boto:Path: /vclfcn-64/obj3?versionId=null 2024-02-01T10:50:10.762 DEBUG:boto:Data: 2024-02-01T10:50:10.762 DEBUG:boto:Headers: {} 2024-02-01T10:50:10.763 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:10.763 DEBUG:boto:Port: 8000 2024-02-01T10:50:10.763 DEBUG:boto:Params: {} 2024-02-01T10:50:10.763 DEBUG:boto:Token: None 2024-02-01T10:50:10.764 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:10 GMT /vclfcn-64/obj3?versionId=null 2024-02-01T10:50:10.764 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9UejZK4G7dFxOzkzJxy34cvWvp8= 2024-02-01T10:50:10.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9UejZK4G7dFxOzkzJxy34cvWvp8=', 'Content-Length': '0'} 2024-02-01T10:50:10.767 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ff7fa3372b6e6cc1-0065bb7762-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:10.808 DEBUG:boto:path=/vclfcn-64/obj3 2024-02-01T10:50:10.808 DEBUG:boto:auth_path=/vclfcn-64/obj3 2024-02-01T10:50:10.808 DEBUG:boto:path=/vclfcn-64/obj3?versionId=null 2024-02-01T10:50:10.809 DEBUG:boto:auth_path=/vclfcn-64/obj3?versionId=null 2024-02-01T10:50:10.809 DEBUG:boto:Method: GET 2024-02-01T10:50:10.809 DEBUG:boto:Path: /vclfcn-64/obj3?versionId=null 2024-02-01T10:50:10.809 DEBUG:boto:Data: 2024-02-01T10:50:10.809 DEBUG:boto:Headers: {} 2024-02-01T10:50:10.809 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:10.809 DEBUG:boto:Port: 8000 2024-02-01T10:50:10.809 DEBUG:boto:Params: {} 2024-02-01T10:50:10.809 DEBUG:boto:Token: None 2024-02-01T10:50:10.809 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:10 GMT /vclfcn-64/obj3?versionId=null 2024-02-01T10:50:10.810 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9UejZK4G7dFxOzkzJxy34cvWvp8= 2024-02-01T10:50:10.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9UejZK4G7dFxOzkzJxy34cvWvp8=', 'Content-Length': '0'} 2024-02-01T10:50:10.812 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000031ef7a427aca153d-0065bb7762-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:10.856 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:10.857 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:10.857 DEBUG:boto:path=/vclfcn-64/?prefix=obj3 2024-02-01T10:50:10.857 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj3 2024-02-01T10:50:10.857 DEBUG:boto:Method: GET 2024-02-01T10:50:10.857 DEBUG:boto:Path: /vclfcn-64/?prefix=obj3 2024-02-01T10:50:10.857 DEBUG:boto:Data: 2024-02-01T10:50:10.858 DEBUG:boto:Headers: {} 2024-02-01T10:50:10.858 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:10.858 DEBUG:boto:Port: 8000 2024-02-01T10:50:10.858 DEBUG:boto:Params: {} 2024-02-01T10:50:10.858 DEBUG:boto:Token: None 2024-02-01T10:50:10.858 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:10 GMT /vclfcn-64/ 2024-02-01T10:50:10.858 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vbNmSpbPy6Szb1Rj2it0MD3AnJ4= 2024-02-01T10:50:10.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vbNmSpbPy6Szb1Rj2it0MD3AnJ4=', 'Content-Length': '0'} 2024-02-01T10:50:10.862 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006338f14451ed37ac-0065bb7762-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:10.904 DEBUG:boto:b'vclfcn-64obj31000falseobj32024-02-01T10:48:26.634Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:10.905 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:10.906 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:10.906 DEBUG:boto:path=/vclfcn-64/?prefix=obj3 2024-02-01T10:50:10.906 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj3 2024-02-01T10:50:10.907 DEBUG:boto:Method: GET 2024-02-01T10:50:10.907 DEBUG:boto:Path: /vclfcn-64/?prefix=obj3 2024-02-01T10:50:10.907 DEBUG:boto:Data: 2024-02-01T10:50:10.907 DEBUG:boto:Headers: {} 2024-02-01T10:50:10.908 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:10.908 DEBUG:boto:Port: 8000 2024-02-01T10:50:10.908 DEBUG:boto:Params: {} 2024-02-01T10:50:10.908 DEBUG:boto:Token: None 2024-02-01T10:50:10.909 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:10 GMT /vclfcn-64/ 2024-02-01T10:50:10.909 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vbNmSpbPy6Szb1Rj2it0MD3AnJ4= 2024-02-01T10:50:10.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vbNmSpbPy6Szb1Rj2it0MD3AnJ4=', 'Content-Length': '0'} 2024-02-01T10:50:10.914 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aff420ac494c7909-0065bb7762-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:10.914 DEBUG:boto:b'vclfcn-64obj31000falseobj32024-02-01T10:48:26.634Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:10.915 DEBUG:boto:path=/vclfcn-64/obj3 2024-02-01T10:50:10.915 DEBUG:boto:auth_path=/vclfcn-64/obj3 2024-02-01T10:50:10.916 DEBUG:boto:Method: HEAD 2024-02-01T10:50:10.916 DEBUG:boto:Path: /vclfcn-64/obj3 2024-02-01T10:50:10.916 DEBUG:boto:Data: 2024-02-01T10:50:10.916 DEBUG:boto:Headers: {} 2024-02-01T10:50:10.917 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:10.917 DEBUG:boto:Port: 8000 2024-02-01T10:50:10.917 DEBUG:boto:Params: {} 2024-02-01T10:50:10.917 DEBUG:boto:Token: None 2024-02-01T10:50:10.917 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:10 GMT /vclfcn-64/obj3 2024-02-01T10:50:10.918 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QBRX+QwV6ynNL00qSUr2H9dGsGA= 2024-02-01T10:50:10.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QBRX+QwV6ynNL00qSUr2H9dGsGA=', 'Content-Length': '0'} 2024-02-01T10:50:10.921 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000db74dd6dbd1abd64-0065bb7762-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:10.921 DEBUG:boto:path=/vclfcn-64/obj3 2024-02-01T10:50:10.922 DEBUG:boto:auth_path=/vclfcn-64/obj3 2024-02-01T10:50:10.922 DEBUG:boto:Method: HEAD 2024-02-01T10:50:10.922 DEBUG:boto:Path: /vclfcn-64/obj3 2024-02-01T10:50:10.922 DEBUG:boto:Data: 2024-02-01T10:50:10.922 DEBUG:boto:Headers: {} 2024-02-01T10:50:10.922 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:10.922 DEBUG:boto:Port: 8000 2024-02-01T10:50:10.922 DEBUG:boto:Params: {} 2024-02-01T10:50:10.923 DEBUG:boto:Token: None 2024-02-01T10:50:10.923 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:10 GMT /vclfcn-64/obj3 2024-02-01T10:50:10.923 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QBRX+QwV6ynNL00qSUr2H9dGsGA= 2024-02-01T10:50:10.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QBRX+QwV6ynNL00qSUr2H9dGsGA=', 'Content-Length': '0'} 2024-02-01T10:50:10.926 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005dfca296723b6a01-0065bb7762-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:10.926 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:50:10.926 DEBUG:boto:path=/vclfcn-64/obj3 2024-02-01T10:50:10.926 DEBUG:boto:auth_path=/vclfcn-64/obj3 2024-02-01T10:50:10.926 DEBUG:boto:Method: GET 2024-02-01T10:50:10.926 DEBUG:boto:Path: /vclfcn-64/obj3 2024-02-01T10:50:10.926 DEBUG:boto:Data: 2024-02-01T10:50:10.926 DEBUG:boto:Headers: {} 2024-02-01T10:50:10.927 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:10.927 DEBUG:boto:Port: 8000 2024-02-01T10:50:10.927 DEBUG:boto:Params: {} 2024-02-01T10:50:10.927 DEBUG:boto:Token: None 2024-02-01T10:50:10.927 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:10 GMT /vclfcn-64/obj3 2024-02-01T10:50:10.927 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lvhNbwCVn09CoiSFjV5OQcqTnyU= 2024-02-01T10:50:10.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lvhNbwCVn09CoiSFjV5OQcqTnyU=', 'Content-Length': '0'} 2024-02-01T10:50:10.931 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000007973de3966397b2-0065bb7762-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:10.973 DEBUG:boto:path=/vclfcn-64/obj3 2024-02-01T10:50:10.973 DEBUG:boto:auth_path=/vclfcn-64/obj3 2024-02-01T10:50:10.973 DEBUG:boto:Method: GET 2024-02-01T10:50:10.973 DEBUG:boto:Path: /vclfcn-64/obj3 2024-02-01T10:50:10.973 DEBUG:boto:Data: 2024-02-01T10:50:10.974 DEBUG:boto:Headers: {} 2024-02-01T10:50:10.974 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:10.974 DEBUG:boto:Port: 8000 2024-02-01T10:50:10.974 DEBUG:boto:Params: {} 2024-02-01T10:50:10.974 DEBUG:boto:Token: None 2024-02-01T10:50:10.974 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:10 GMT /vclfcn-64/obj3 2024-02-01T10:50:10.974 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lvhNbwCVn09CoiSFjV5OQcqTnyU= 2024-02-01T10:50:10.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lvhNbwCVn09CoiSFjV5OQcqTnyU=', 'Content-Length': '0'} 2024-02-01T10:50:10.977 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000004f04a56637fc592-0065bb7762-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:10 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:11.020 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:11.020 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:11.020 DEBUG:boto:path=/vclfcn-64/?prefix=obj3 2024-02-01T10:50:11.021 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj3 2024-02-01T10:50:11.021 DEBUG:boto:Method: GET 2024-02-01T10:50:11.021 DEBUG:boto:Path: /vclfcn-64/?prefix=obj3 2024-02-01T10:50:11.021 DEBUG:boto:Data: 2024-02-01T10:50:11.021 DEBUG:boto:Headers: {} 2024-02-01T10:50:11.021 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:11.021 DEBUG:boto:Port: 8000 2024-02-01T10:50:11.021 DEBUG:boto:Params: {} 2024-02-01T10:50:11.021 DEBUG:boto:Token: None 2024-02-01T10:50:11.022 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:11 GMT /vclfcn-64/ 2024-02-01T10:50:11.022 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1folidtZMJOkYQ7zp4TORGIFN3I= 2024-02-01T10:50:11.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1folidtZMJOkYQ7zp4TORGIFN3I=', 'Content-Length': '0'} 2024-02-01T10:50:11.026 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ebdbf0bd39c811b2-0065bb7763-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:11.072 DEBUG:boto:b'vclfcn-64obj31000falseobj32024-02-01T10:48:26.634Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:11.072 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:11.073 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:11.073 DEBUG:boto:path=/vclfcn-64/?prefix=obj3 2024-02-01T10:50:11.073 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj3 2024-02-01T10:50:11.073 DEBUG:boto:Method: GET 2024-02-01T10:50:11.073 DEBUG:boto:Path: /vclfcn-64/?prefix=obj3 2024-02-01T10:50:11.073 DEBUG:boto:Data: 2024-02-01T10:50:11.073 DEBUG:boto:Headers: {} 2024-02-01T10:50:11.073 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:11.073 DEBUG:boto:Port: 8000 2024-02-01T10:50:11.073 DEBUG:boto:Params: {} 2024-02-01T10:50:11.074 DEBUG:boto:Token: None 2024-02-01T10:50:11.074 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:11 GMT /vclfcn-64/ 2024-02-01T10:50:11.074 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1folidtZMJOkYQ7zp4TORGIFN3I= 2024-02-01T10:50:11.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1folidtZMJOkYQ7zp4TORGIFN3I=', 'Content-Length': '0'} 2024-02-01T10:50:11.078 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000550b83d9f9ce08d8-0065bb7763-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:11.078 DEBUG:boto:b'vclfcn-64obj31000falseobj32024-02-01T10:48:26.634Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:11.079 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:50:11.079 DEBUG:boto:path=/vclfcn-64/obj4 2024-02-01T10:50:11.079 DEBUG:boto:auth_path=/vclfcn-64/obj4 2024-02-01T10:50:11.079 DEBUG:boto:path=/vclfcn-64/obj4?versionId=null 2024-02-01T10:50:11.079 DEBUG:boto:auth_path=/vclfcn-64/obj4?versionId=null 2024-02-01T10:50:11.079 DEBUG:boto:Method: GET 2024-02-01T10:50:11.079 DEBUG:boto:Path: /vclfcn-64/obj4?versionId=null 2024-02-01T10:50:11.079 DEBUG:boto:Data: 2024-02-01T10:50:11.080 DEBUG:boto:Headers: {} 2024-02-01T10:50:11.080 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:11.080 DEBUG:boto:Port: 8000 2024-02-01T10:50:11.080 DEBUG:boto:Params: {} 2024-02-01T10:50:11.080 DEBUG:boto:Token: None 2024-02-01T10:50:11.080 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:11 GMT /vclfcn-64/obj4?versionId=null 2024-02-01T10:50:11.080 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HV2UYIPiljIlzLcCOVA3MtR5/cw= 2024-02-01T10:50:11.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HV2UYIPiljIlzLcCOVA3MtR5/cw=', 'Content-Length': '0'} 2024-02-01T10:50:11.083 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bdeaf52fa1653fdf-0065bb7763-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:11.083 DEBUG:boto:path=/vclfcn-64/obj4 2024-02-01T10:50:11.084 DEBUG:boto:auth_path=/vclfcn-64/obj4 2024-02-01T10:50:11.084 DEBUG:boto:path=/vclfcn-64/obj4?versionId=null 2024-02-01T10:50:11.089 DEBUG:boto:auth_path=/vclfcn-64/obj4?versionId=null 2024-02-01T10:50:11.089 DEBUG:boto:Method: GET 2024-02-01T10:50:11.089 DEBUG:boto:Path: /vclfcn-64/obj4?versionId=null 2024-02-01T10:50:11.089 DEBUG:boto:Data: 2024-02-01T10:50:11.089 DEBUG:boto:Headers: {} 2024-02-01T10:50:11.089 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:11.090 DEBUG:boto:Port: 8000 2024-02-01T10:50:11.090 DEBUG:boto:Params: {} 2024-02-01T10:50:11.090 DEBUG:boto:Token: None 2024-02-01T10:50:11.090 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:11 GMT /vclfcn-64/obj4?versionId=null 2024-02-01T10:50:11.090 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HV2UYIPiljIlzLcCOVA3MtR5/cw= 2024-02-01T10:50:11.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HV2UYIPiljIlzLcCOVA3MtR5/cw=', 'Content-Length': '0'} 2024-02-01T10:50:11.093 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e100159f951a9788-0065bb7763-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:11.093 DEBUG:boto:path=/vclfcn-64/obj4 2024-02-01T10:50:11.094 DEBUG:boto:auth_path=/vclfcn-64/obj4 2024-02-01T10:50:11.094 DEBUG:boto:path=/vclfcn-64/obj4?versionId=null 2024-02-01T10:50:11.094 DEBUG:boto:auth_path=/vclfcn-64/obj4?versionId=null 2024-02-01T10:50:11.094 DEBUG:boto:Method: HEAD 2024-02-01T10:50:11.094 DEBUG:boto:Path: /vclfcn-64/obj4?versionId=null 2024-02-01T10:50:11.094 DEBUG:boto:Data: 2024-02-01T10:50:11.094 DEBUG:boto:Headers: {} 2024-02-01T10:50:11.094 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:11.094 DEBUG:boto:Port: 8000 2024-02-01T10:50:11.094 DEBUG:boto:Params: {} 2024-02-01T10:50:11.095 DEBUG:boto:Token: None 2024-02-01T10:50:11.095 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:11 GMT /vclfcn-64/obj4?versionId=null 2024-02-01T10:50:11.095 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ev0lPoMHGovZS5b4D6X6LPWf5J8= 2024-02-01T10:50:11.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ev0lPoMHGovZS5b4D6X6LPWf5J8=', 'Content-Length': '0'} 2024-02-01T10:50:11.098 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cd14490c2941a450-0065bb7763-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:11.098 DEBUG:boto:path=/vclfcn-64/obj4 2024-02-01T10:50:11.098 DEBUG:boto:auth_path=/vclfcn-64/obj4 2024-02-01T10:50:11.098 DEBUG:boto:path=/vclfcn-64/obj4?versionId=null 2024-02-01T10:50:11.098 DEBUG:boto:auth_path=/vclfcn-64/obj4?versionId=null 2024-02-01T10:50:11.098 DEBUG:boto:Method: HEAD 2024-02-01T10:50:11.098 DEBUG:boto:Path: /vclfcn-64/obj4?versionId=null 2024-02-01T10:50:11.098 DEBUG:boto:Data: 2024-02-01T10:50:11.099 DEBUG:boto:Headers: {} 2024-02-01T10:50:11.099 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:11.099 DEBUG:boto:Port: 8000 2024-02-01T10:50:11.099 DEBUG:boto:Params: {} 2024-02-01T10:50:11.099 DEBUG:boto:Token: None 2024-02-01T10:50:11.099 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:11 GMT /vclfcn-64/obj4?versionId=null 2024-02-01T10:50:11.099 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ev0lPoMHGovZS5b4D6X6LPWf5J8= 2024-02-01T10:50:11.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ev0lPoMHGovZS5b4D6X6LPWf5J8=', 'Content-Length': '0'} 2024-02-01T10:50:11.102 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d7949531594ee26d-0065bb7763-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:11.102 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:50:11.102 DEBUG:boto:path=/vclfcn-64/obj4 2024-02-01T10:50:11.103 DEBUG:boto:auth_path=/vclfcn-64/obj4 2024-02-01T10:50:11.103 DEBUG:boto:path=/vclfcn-64/obj4?versionId=null 2024-02-01T10:50:11.103 DEBUG:boto:auth_path=/vclfcn-64/obj4?versionId=null 2024-02-01T10:50:11.103 DEBUG:boto:Method: GET 2024-02-01T10:50:11.103 DEBUG:boto:Path: /vclfcn-64/obj4?versionId=null 2024-02-01T10:50:11.103 DEBUG:boto:Data: 2024-02-01T10:50:11.103 DEBUG:boto:Headers: {} 2024-02-01T10:50:11.103 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:11.103 DEBUG:boto:Port: 8000 2024-02-01T10:50:11.103 DEBUG:boto:Params: {} 2024-02-01T10:50:11.103 DEBUG:boto:Token: None 2024-02-01T10:50:11.104 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:11 GMT /vclfcn-64/obj4?versionId=null 2024-02-01T10:50:11.104 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HV2UYIPiljIlzLcCOVA3MtR5/cw= 2024-02-01T10:50:11.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HV2UYIPiljIlzLcCOVA3MtR5/cw=', 'Content-Length': '0'} 2024-02-01T10:50:11.107 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008737eb59339ae8f7-0065bb7763-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:11.152 DEBUG:boto:path=/vclfcn-64/obj4 2024-02-01T10:50:11.152 DEBUG:boto:auth_path=/vclfcn-64/obj4 2024-02-01T10:50:11.152 DEBUG:boto:path=/vclfcn-64/obj4?versionId=null 2024-02-01T10:50:11.152 DEBUG:boto:auth_path=/vclfcn-64/obj4?versionId=null 2024-02-01T10:50:11.153 DEBUG:boto:Method: GET 2024-02-01T10:50:11.153 DEBUG:boto:Path: /vclfcn-64/obj4?versionId=null 2024-02-01T10:50:11.153 DEBUG:boto:Data: 2024-02-01T10:50:11.153 DEBUG:boto:Headers: {} 2024-02-01T10:50:11.153 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:11.153 DEBUG:boto:Port: 8000 2024-02-01T10:50:11.153 DEBUG:boto:Params: {} 2024-02-01T10:50:11.154 DEBUG:boto:Token: None 2024-02-01T10:50:11.154 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:11 GMT /vclfcn-64/obj4?versionId=null 2024-02-01T10:50:11.154 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:HV2UYIPiljIlzLcCOVA3MtR5/cw= 2024-02-01T10:50:11.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:HV2UYIPiljIlzLcCOVA3MtR5/cw=', 'Content-Length': '0'} 2024-02-01T10:50:11.157 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000c7e0cec1d0cceae-0065bb7763-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:11.204 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:11.204 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:11.204 DEBUG:boto:path=/vclfcn-64/?prefix=obj4 2024-02-01T10:50:11.205 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj4 2024-02-01T10:50:11.205 DEBUG:boto:Method: GET 2024-02-01T10:50:11.205 DEBUG:boto:Path: /vclfcn-64/?prefix=obj4 2024-02-01T10:50:11.205 DEBUG:boto:Data: 2024-02-01T10:50:11.205 DEBUG:boto:Headers: {} 2024-02-01T10:50:11.205 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:11.205 DEBUG:boto:Port: 8000 2024-02-01T10:50:11.205 DEBUG:boto:Params: {} 2024-02-01T10:50:11.205 DEBUG:boto:Token: None 2024-02-01T10:50:11.206 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:11 GMT /vclfcn-64/ 2024-02-01T10:50:11.207 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1folidtZMJOkYQ7zp4TORGIFN3I= 2024-02-01T10:50:11.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1folidtZMJOkYQ7zp4TORGIFN3I=', 'Content-Length': '0'} 2024-02-01T10:50:11.211 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000395b9eb676b40262-0065bb7763-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:11.252 DEBUG:boto:b'vclfcn-64obj41000falseobj42024-02-01T10:48:26.689Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:11.253 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:11.253 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:11.253 DEBUG:boto:path=/vclfcn-64/?prefix=obj4 2024-02-01T10:50:11.253 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj4 2024-02-01T10:50:11.254 DEBUG:boto:Method: GET 2024-02-01T10:50:11.254 DEBUG:boto:Path: /vclfcn-64/?prefix=obj4 2024-02-01T10:50:11.254 DEBUG:boto:Data: 2024-02-01T10:50:11.254 DEBUG:boto:Headers: {} 2024-02-01T10:50:11.254 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:11.254 DEBUG:boto:Port: 8000 2024-02-01T10:50:11.254 DEBUG:boto:Params: {} 2024-02-01T10:50:11.254 DEBUG:boto:Token: None 2024-02-01T10:50:11.255 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:11 GMT /vclfcn-64/ 2024-02-01T10:50:11.255 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1folidtZMJOkYQ7zp4TORGIFN3I= 2024-02-01T10:50:11.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1folidtZMJOkYQ7zp4TORGIFN3I=', 'Content-Length': '0'} 2024-02-01T10:50:11.259 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003899ac2fff89133a-0065bb7763-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:11.259 DEBUG:boto:b'vclfcn-64obj41000falseobj42024-02-01T10:48:26.689Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:11.260 DEBUG:boto:path=/vclfcn-64/obj4 2024-02-01T10:50:11.260 DEBUG:boto:auth_path=/vclfcn-64/obj4 2024-02-01T10:50:11.260 DEBUG:boto:Method: HEAD 2024-02-01T10:50:11.260 DEBUG:boto:Path: /vclfcn-64/obj4 2024-02-01T10:50:11.260 DEBUG:boto:Data: 2024-02-01T10:50:11.260 DEBUG:boto:Headers: {} 2024-02-01T10:50:11.260 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:11.260 DEBUG:boto:Port: 8000 2024-02-01T10:50:11.260 DEBUG:boto:Params: {} 2024-02-01T10:50:11.261 DEBUG:boto:Token: None 2024-02-01T10:50:11.261 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:11 GMT /vclfcn-64/obj4 2024-02-01T10:50:11.261 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1bHjqPK5rtyplagjQVss2fTSQo4= 2024-02-01T10:50:11.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1bHjqPK5rtyplagjQVss2fTSQo4=', 'Content-Length': '0'} 2024-02-01T10:50:11.264 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a9713bc36c530c01-0065bb7763-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:11.264 DEBUG:boto:path=/vclfcn-64/obj4 2024-02-01T10:50:11.264 DEBUG:boto:auth_path=/vclfcn-64/obj4 2024-02-01T10:50:11.265 DEBUG:boto:Method: HEAD 2024-02-01T10:50:11.265 DEBUG:boto:Path: /vclfcn-64/obj4 2024-02-01T10:50:11.265 DEBUG:boto:Data: 2024-02-01T10:50:11.265 DEBUG:boto:Headers: {} 2024-02-01T10:50:11.265 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:11.265 DEBUG:boto:Port: 8000 2024-02-01T10:50:11.265 DEBUG:boto:Params: {} 2024-02-01T10:50:11.265 DEBUG:boto:Token: None 2024-02-01T10:50:11.266 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:11 GMT /vclfcn-64/obj4 2024-02-01T10:50:11.266 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1bHjqPK5rtyplagjQVss2fTSQo4= 2024-02-01T10:50:11.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1bHjqPK5rtyplagjQVss2fTSQo4=', 'Content-Length': '0'} 2024-02-01T10:50:11.268 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000915cee7512e52d77-0065bb7763-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:11.269 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:50:11.269 DEBUG:boto:path=/vclfcn-64/obj4 2024-02-01T10:50:11.269 DEBUG:boto:auth_path=/vclfcn-64/obj4 2024-02-01T10:50:11.269 DEBUG:boto:Method: GET 2024-02-01T10:50:11.269 DEBUG:boto:Path: /vclfcn-64/obj4 2024-02-01T10:50:11.269 DEBUG:boto:Data: 2024-02-01T10:50:11.269 DEBUG:boto:Headers: {} 2024-02-01T10:50:11.269 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:11.270 DEBUG:boto:Port: 8000 2024-02-01T10:50:11.270 DEBUG:boto:Params: {} 2024-02-01T10:50:11.270 DEBUG:boto:Token: None 2024-02-01T10:50:11.270 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:11 GMT /vclfcn-64/obj4 2024-02-01T10:50:11.270 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:D/0HU64DKmZQTrvDsaULQ7+pF+Y= 2024-02-01T10:50:11.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:D/0HU64DKmZQTrvDsaULQ7+pF+Y=', 'Content-Length': '0'} 2024-02-01T10:50:11.273 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ca6b2fe55c81a8f5-0065bb7763-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:11.316 DEBUG:boto:path=/vclfcn-64/obj4 2024-02-01T10:50:11.317 DEBUG:boto:auth_path=/vclfcn-64/obj4 2024-02-01T10:50:11.317 DEBUG:boto:Method: GET 2024-02-01T10:50:11.317 DEBUG:boto:Path: /vclfcn-64/obj4 2024-02-01T10:50:11.317 DEBUG:boto:Data: 2024-02-01T10:50:11.317 DEBUG:boto:Headers: {} 2024-02-01T10:50:11.317 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:11.317 DEBUG:boto:Port: 8000 2024-02-01T10:50:11.317 DEBUG:boto:Params: {} 2024-02-01T10:50:11.317 DEBUG:boto:Token: None 2024-02-01T10:50:11.318 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:11 GMT /vclfcn-64/obj4 2024-02-01T10:50:11.318 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:D/0HU64DKmZQTrvDsaULQ7+pF+Y= 2024-02-01T10:50:11.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:D/0HU64DKmZQTrvDsaULQ7+pF+Y=', 'Content-Length': '0'} 2024-02-01T10:50:11.321 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007a85d950e8e70a55-0065bb7763-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:11.364 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:11.364 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:11.364 DEBUG:boto:path=/vclfcn-64/?prefix=obj4 2024-02-01T10:50:11.364 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj4 2024-02-01T10:50:11.365 DEBUG:boto:Method: GET 2024-02-01T10:50:11.365 DEBUG:boto:Path: /vclfcn-64/?prefix=obj4 2024-02-01T10:50:11.365 DEBUG:boto:Data: 2024-02-01T10:50:11.365 DEBUG:boto:Headers: {} 2024-02-01T10:50:11.365 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:11.365 DEBUG:boto:Port: 8000 2024-02-01T10:50:11.365 DEBUG:boto:Params: {} 2024-02-01T10:50:11.365 DEBUG:boto:Token: None 2024-02-01T10:50:11.365 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:11 GMT /vclfcn-64/ 2024-02-01T10:50:11.366 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1folidtZMJOkYQ7zp4TORGIFN3I= 2024-02-01T10:50:11.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1folidtZMJOkYQ7zp4TORGIFN3I=', 'Content-Length': '0'} 2024-02-01T10:50:11.370 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e79aa1e1138cadbd-0065bb7763-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:11.416 DEBUG:boto:b'vclfcn-64obj41000falseobj42024-02-01T10:48:26.689Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:11.417 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:11.417 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:11.417 DEBUG:boto:path=/vclfcn-64/?prefix=obj4 2024-02-01T10:50:11.417 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj4 2024-02-01T10:50:11.417 DEBUG:boto:Method: GET 2024-02-01T10:50:11.417 DEBUG:boto:Path: /vclfcn-64/?prefix=obj4 2024-02-01T10:50:11.417 DEBUG:boto:Data: 2024-02-01T10:50:11.417 DEBUG:boto:Headers: {} 2024-02-01T10:50:11.417 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:11.417 DEBUG:boto:Port: 8000 2024-02-01T10:50:11.418 DEBUG:boto:Params: {} 2024-02-01T10:50:11.418 DEBUG:boto:Token: None 2024-02-01T10:50:11.418 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:11 GMT /vclfcn-64/ 2024-02-01T10:50:11.418 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1folidtZMJOkYQ7zp4TORGIFN3I= 2024-02-01T10:50:11.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1folidtZMJOkYQ7zp4TORGIFN3I=', 'Content-Length': '0'} 2024-02-01T10:50:11.423 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000620565a498943bd6-0065bb7763-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:11.423 DEBUG:boto:b'vclfcn-64obj41000falseobj42024-02-01T10:48:26.689Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:11.423 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-02-01T10:50:11.423 DEBUG:boto:path=/vclfcn-64/obj5 2024-02-01T10:50:11.423 DEBUG:boto:auth_path=/vclfcn-64/obj5 2024-02-01T10:50:11.423 DEBUG:boto:path=/vclfcn-64/obj5?versionId=null 2024-02-01T10:50:11.424 DEBUG:boto:auth_path=/vclfcn-64/obj5?versionId=null 2024-02-01T10:50:11.424 DEBUG:boto:Method: GET 2024-02-01T10:50:11.424 DEBUG:boto:Path: /vclfcn-64/obj5?versionId=null 2024-02-01T10:50:11.424 DEBUG:boto:Data: 2024-02-01T10:50:11.424 DEBUG:boto:Headers: {} 2024-02-01T10:50:11.424 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:11.424 DEBUG:boto:Port: 8000 2024-02-01T10:50:11.424 DEBUG:boto:Params: {} 2024-02-01T10:50:11.424 DEBUG:boto:Token: None 2024-02-01T10:50:11.424 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:11 GMT /vclfcn-64/obj5?versionId=null 2024-02-01T10:50:11.425 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:VZybZn8bQZpRl4zlAqSpwM4Tols= 2024-02-01T10:50:11.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:VZybZn8bQZpRl4zlAqSpwM4Tols=', 'Content-Length': '0'} 2024-02-01T10:50:11.428 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009bc1141153172cdd-0065bb7763-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:11.428 DEBUG:boto:path=/vclfcn-64/obj5 2024-02-01T10:50:11.428 DEBUG:boto:auth_path=/vclfcn-64/obj5 2024-02-01T10:50:11.428 DEBUG:boto:path=/vclfcn-64/obj5?versionId=null 2024-02-01T10:50:11.428 DEBUG:boto:auth_path=/vclfcn-64/obj5?versionId=null 2024-02-01T10:50:11.428 DEBUG:boto:Method: GET 2024-02-01T10:50:11.429 DEBUG:boto:Path: /vclfcn-64/obj5?versionId=null 2024-02-01T10:50:11.429 DEBUG:boto:Data: 2024-02-01T10:50:11.429 DEBUG:boto:Headers: {} 2024-02-01T10:50:11.429 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:11.429 DEBUG:boto:Port: 8000 2024-02-01T10:50:11.429 DEBUG:boto:Params: {} 2024-02-01T10:50:11.429 DEBUG:boto:Token: None 2024-02-01T10:50:11.429 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:11 GMT /vclfcn-64/obj5?versionId=null 2024-02-01T10:50:11.429 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:VZybZn8bQZpRl4zlAqSpwM4Tols= 2024-02-01T10:50:11.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:VZybZn8bQZpRl4zlAqSpwM4Tols=', 'Content-Length': '0'} 2024-02-01T10:50:11.432 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a13a19ac55f12248-0065bb7763-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:11.433 DEBUG:boto:path=/vclfcn-64/obj5 2024-02-01T10:50:11.433 DEBUG:boto:auth_path=/vclfcn-64/obj5 2024-02-01T10:50:11.433 DEBUG:boto:path=/vclfcn-64/obj5?versionId=null 2024-02-01T10:50:11.433 DEBUG:boto:auth_path=/vclfcn-64/obj5?versionId=null 2024-02-01T10:50:11.433 DEBUG:boto:Method: HEAD 2024-02-01T10:50:11.433 DEBUG:boto:Path: /vclfcn-64/obj5?versionId=null 2024-02-01T10:50:11.433 DEBUG:boto:Data: 2024-02-01T10:50:11.433 DEBUG:boto:Headers: {} 2024-02-01T10:50:11.434 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:11.434 DEBUG:boto:Port: 8000 2024-02-01T10:50:11.434 DEBUG:boto:Params: {} 2024-02-01T10:50:11.434 DEBUG:boto:Token: None 2024-02-01T10:50:11.434 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:11 GMT /vclfcn-64/obj5?versionId=null 2024-02-01T10:50:11.434 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+eHA3wLv/F6QzsAkKS6XOGuC8n4= 2024-02-01T10:50:11.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+eHA3wLv/F6QzsAkKS6XOGuC8n4=', 'Content-Length': '0'} 2024-02-01T10:50:11.437 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b13290fc18d19fca-0065bb7763-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:11.437 DEBUG:boto:path=/vclfcn-64/obj5 2024-02-01T10:50:11.437 DEBUG:boto:auth_path=/vclfcn-64/obj5 2024-02-01T10:50:11.437 DEBUG:boto:path=/vclfcn-64/obj5?versionId=null 2024-02-01T10:50:11.437 DEBUG:boto:auth_path=/vclfcn-64/obj5?versionId=null 2024-02-01T10:50:11.437 DEBUG:boto:Method: HEAD 2024-02-01T10:50:11.438 DEBUG:boto:Path: /vclfcn-64/obj5?versionId=null 2024-02-01T10:50:11.438 DEBUG:boto:Data: 2024-02-01T10:50:11.438 DEBUG:boto:Headers: {} 2024-02-01T10:50:11.438 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:11.438 DEBUG:boto:Port: 8000 2024-02-01T10:50:11.438 DEBUG:boto:Params: {} 2024-02-01T10:50:11.438 DEBUG:boto:Token: None 2024-02-01T10:50:11.438 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:11 GMT /vclfcn-64/obj5?versionId=null 2024-02-01T10:50:11.438 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+eHA3wLv/F6QzsAkKS6XOGuC8n4= 2024-02-01T10:50:11.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+eHA3wLv/F6QzsAkKS6XOGuC8n4=', 'Content-Length': '0'} 2024-02-01T10:50:11.441 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004c23eadb63918aa5-0065bb7763-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:11.441 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-02-01T10:50:11.441 DEBUG:boto:path=/vclfcn-64/obj5 2024-02-01T10:50:11.442 DEBUG:boto:auth_path=/vclfcn-64/obj5 2024-02-01T10:50:11.442 DEBUG:boto:path=/vclfcn-64/obj5?versionId=null 2024-02-01T10:50:11.442 DEBUG:boto:auth_path=/vclfcn-64/obj5?versionId=null 2024-02-01T10:50:11.442 DEBUG:boto:Method: GET 2024-02-01T10:50:11.442 DEBUG:boto:Path: /vclfcn-64/obj5?versionId=null 2024-02-01T10:50:11.442 DEBUG:boto:Data: 2024-02-01T10:50:11.442 DEBUG:boto:Headers: {} 2024-02-01T10:50:11.442 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:11.442 DEBUG:boto:Port: 8000 2024-02-01T10:50:11.442 DEBUG:boto:Params: {} 2024-02-01T10:50:11.442 DEBUG:boto:Token: None 2024-02-01T10:50:11.443 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:11 GMT /vclfcn-64/obj5?versionId=null 2024-02-01T10:50:11.443 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:VZybZn8bQZpRl4zlAqSpwM4Tols= 2024-02-01T10:50:11.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:VZybZn8bQZpRl4zlAqSpwM4Tols=', 'Content-Length': '0'} 2024-02-01T10:50:11.445 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c002f64704e4daf7-0065bb7763-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:11.488 DEBUG:boto:path=/vclfcn-64/obj5 2024-02-01T10:50:11.488 DEBUG:boto:auth_path=/vclfcn-64/obj5 2024-02-01T10:50:11.488 DEBUG:boto:path=/vclfcn-64/obj5?versionId=null 2024-02-01T10:50:11.488 DEBUG:boto:auth_path=/vclfcn-64/obj5?versionId=null 2024-02-01T10:50:11.488 DEBUG:boto:Method: GET 2024-02-01T10:50:11.489 DEBUG:boto:Path: /vclfcn-64/obj5?versionId=null 2024-02-01T10:50:11.489 DEBUG:boto:Data: 2024-02-01T10:50:11.489 DEBUG:boto:Headers: {} 2024-02-01T10:50:11.489 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:11.489 DEBUG:boto:Port: 8000 2024-02-01T10:50:11.489 DEBUG:boto:Params: {} 2024-02-01T10:50:11.489 DEBUG:boto:Token: None 2024-02-01T10:50:11.489 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:11 GMT /vclfcn-64/obj5?versionId=null 2024-02-01T10:50:11.489 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:VZybZn8bQZpRl4zlAqSpwM4Tols= 2024-02-01T10:50:11.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:VZybZn8bQZpRl4zlAqSpwM4Tols=', 'Content-Length': '0'} 2024-02-01T10:50:11.492 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000095f1b78a88cd52e1-0065bb7763-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:11.536 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:11.536 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:11.536 DEBUG:boto:path=/vclfcn-64/?prefix=obj5 2024-02-01T10:50:11.537 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj5 2024-02-01T10:50:11.537 DEBUG:boto:Method: GET 2024-02-01T10:50:11.537 DEBUG:boto:Path: /vclfcn-64/?prefix=obj5 2024-02-01T10:50:11.537 DEBUG:boto:Data: 2024-02-01T10:50:11.537 DEBUG:boto:Headers: {} 2024-02-01T10:50:11.537 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:11.537 DEBUG:boto:Port: 8000 2024-02-01T10:50:11.537 DEBUG:boto:Params: {} 2024-02-01T10:50:11.537 DEBUG:boto:Token: None 2024-02-01T10:50:11.538 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:11 GMT /vclfcn-64/ 2024-02-01T10:50:11.538 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1folidtZMJOkYQ7zp4TORGIFN3I= 2024-02-01T10:50:11.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1folidtZMJOkYQ7zp4TORGIFN3I=', 'Content-Length': '0'} 2024-02-01T10:50:11.543 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005944a74aef751ed7-0065bb7763-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:11.588 DEBUG:boto:b'vclfcn-64obj51000falseobj52024-02-01T10:49:22.402Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:11.589 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:11.589 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:11.589 DEBUG:boto:path=/vclfcn-64/?prefix=obj5 2024-02-01T10:50:11.589 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj5 2024-02-01T10:50:11.589 DEBUG:boto:Method: GET 2024-02-01T10:50:11.589 DEBUG:boto:Path: /vclfcn-64/?prefix=obj5 2024-02-01T10:50:11.589 DEBUG:boto:Data: 2024-02-01T10:50:11.589 DEBUG:boto:Headers: {} 2024-02-01T10:50:11.589 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:11.590 DEBUG:boto:Port: 8000 2024-02-01T10:50:11.590 DEBUG:boto:Params: {} 2024-02-01T10:50:11.590 DEBUG:boto:Token: None 2024-02-01T10:50:11.590 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:11 GMT /vclfcn-64/ 2024-02-01T10:50:11.590 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1folidtZMJOkYQ7zp4TORGIFN3I= 2024-02-01T10:50:11.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1folidtZMJOkYQ7zp4TORGIFN3I=', 'Content-Length': '0'} 2024-02-01T10:50:11.595 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd2d0a777b63a066-0065bb7763-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:11.636 DEBUG:boto:b'vclfcn-64obj51000falseobj52024-02-01T10:49:22.402Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:11.636 DEBUG:boto:path=/vclfcn-64/obj5 2024-02-01T10:50:11.637 DEBUG:boto:auth_path=/vclfcn-64/obj5 2024-02-01T10:50:11.637 DEBUG:boto:Method: HEAD 2024-02-01T10:50:11.637 DEBUG:boto:Path: /vclfcn-64/obj5 2024-02-01T10:50:11.637 DEBUG:boto:Data: 2024-02-01T10:50:11.637 DEBUG:boto:Headers: {} 2024-02-01T10:50:11.637 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:11.637 DEBUG:boto:Port: 8000 2024-02-01T10:50:11.637 DEBUG:boto:Params: {} 2024-02-01T10:50:11.637 DEBUG:boto:Token: None 2024-02-01T10:50:11.638 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:11 GMT /vclfcn-64/obj5 2024-02-01T10:50:11.638 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+mDd4Ycnpwn9ZJISf2/Ha7tOreI= 2024-02-01T10:50:11.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+mDd4Ycnpwn9ZJISf2/Ha7tOreI=', 'Content-Length': '0'} 2024-02-01T10:50:11.641 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f780cf12f5366480-0065bb7763-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:11.641 DEBUG:boto:path=/vclfcn-64/obj5 2024-02-01T10:50:11.641 DEBUG:boto:auth_path=/vclfcn-64/obj5 2024-02-01T10:50:11.641 DEBUG:boto:Method: HEAD 2024-02-01T10:50:11.641 DEBUG:boto:Path: /vclfcn-64/obj5 2024-02-01T10:50:11.642 DEBUG:boto:Data: 2024-02-01T10:50:11.642 DEBUG:boto:Headers: {} 2024-02-01T10:50:11.642 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:11.642 DEBUG:boto:Port: 8000 2024-02-01T10:50:11.642 DEBUG:boto:Params: {} 2024-02-01T10:50:11.642 DEBUG:boto:Token: None 2024-02-01T10:50:11.642 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:11 GMT /vclfcn-64/obj5 2024-02-01T10:50:11.642 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+mDd4Ycnpwn9ZJISf2/Ha7tOreI= 2024-02-01T10:50:11.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+mDd4Ycnpwn9ZJISf2/Ha7tOreI=', 'Content-Length': '0'} 2024-02-01T10:50:11.645 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c1997ef8025af2c2-0065bb7763-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:11.646 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-02-01T10:50:11.646 DEBUG:boto:path=/vclfcn-64/obj5 2024-02-01T10:50:11.646 DEBUG:boto:auth_path=/vclfcn-64/obj5 2024-02-01T10:50:11.646 DEBUG:boto:Method: GET 2024-02-01T10:50:11.646 DEBUG:boto:Path: /vclfcn-64/obj5 2024-02-01T10:50:11.646 DEBUG:boto:Data: 2024-02-01T10:50:11.646 DEBUG:boto:Headers: {} 2024-02-01T10:50:11.646 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:11.647 DEBUG:boto:Port: 8000 2024-02-01T10:50:11.647 DEBUG:boto:Params: {} 2024-02-01T10:50:11.647 DEBUG:boto:Token: None 2024-02-01T10:50:11.647 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:11 GMT /vclfcn-64/obj5 2024-02-01T10:50:11.647 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5yHFsIjYGmdN2izZZ6M8/lnhTT4= 2024-02-01T10:50:11.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5yHFsIjYGmdN2izZZ6M8/lnhTT4=', 'Content-Length': '0'} 2024-02-01T10:50:11.650 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000638faa054b8962cf-0065bb7763-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:11.692 DEBUG:boto:path=/vclfcn-64/obj5 2024-02-01T10:50:11.692 DEBUG:boto:auth_path=/vclfcn-64/obj5 2024-02-01T10:50:11.692 DEBUG:boto:Method: GET 2024-02-01T10:50:11.693 DEBUG:boto:Path: /vclfcn-64/obj5 2024-02-01T10:50:11.693 DEBUG:boto:Data: 2024-02-01T10:50:11.693 DEBUG:boto:Headers: {} 2024-02-01T10:50:11.693 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:11.693 DEBUG:boto:Port: 8000 2024-02-01T10:50:11.693 DEBUG:boto:Params: {} 2024-02-01T10:50:11.693 DEBUG:boto:Token: None 2024-02-01T10:50:11.693 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:11 GMT /vclfcn-64/obj5 2024-02-01T10:50:11.694 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5yHFsIjYGmdN2izZZ6M8/lnhTT4= 2024-02-01T10:50:11.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5yHFsIjYGmdN2izZZ6M8/lnhTT4=', 'Content-Length': '0'} 2024-02-01T10:50:11.696 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e0595b141373480a-0065bb7763-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:11.740 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:11.740 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:11.740 DEBUG:boto:path=/vclfcn-64/?prefix=obj5 2024-02-01T10:50:11.741 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj5 2024-02-01T10:50:11.741 DEBUG:boto:Method: GET 2024-02-01T10:50:11.741 DEBUG:boto:Path: /vclfcn-64/?prefix=obj5 2024-02-01T10:50:11.741 DEBUG:boto:Data: 2024-02-01T10:50:11.741 DEBUG:boto:Headers: {} 2024-02-01T10:50:11.741 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:11.741 DEBUG:boto:Port: 8000 2024-02-01T10:50:11.741 DEBUG:boto:Params: {} 2024-02-01T10:50:11.741 DEBUG:boto:Token: None 2024-02-01T10:50:11.741 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:11 GMT /vclfcn-64/ 2024-02-01T10:50:11.742 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1folidtZMJOkYQ7zp4TORGIFN3I= 2024-02-01T10:50:11.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1folidtZMJOkYQ7zp4TORGIFN3I=', 'Content-Length': '0'} 2024-02-01T10:50:11.746 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a3c8d1d51b6f5bc2-0065bb7763-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:11.788 DEBUG:boto:b'vclfcn-64obj51000falseobj52024-02-01T10:49:22.402Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:11.788 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:11.788 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:11.789 DEBUG:boto:path=/vclfcn-64/?prefix=obj5 2024-02-01T10:50:11.789 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj5 2024-02-01T10:50:11.789 DEBUG:boto:Method: GET 2024-02-01T10:50:11.789 DEBUG:boto:Path: /vclfcn-64/?prefix=obj5 2024-02-01T10:50:11.789 DEBUG:boto:Data: 2024-02-01T10:50:11.789 DEBUG:boto:Headers: {} 2024-02-01T10:50:11.789 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:11.789 DEBUG:boto:Port: 8000 2024-02-01T10:50:11.789 DEBUG:boto:Params: {} 2024-02-01T10:50:11.789 DEBUG:boto:Token: None 2024-02-01T10:50:11.790 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:11 GMT /vclfcn-64/ 2024-02-01T10:50:11.790 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1folidtZMJOkYQ7zp4TORGIFN3I= 2024-02-01T10:50:11.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1folidtZMJOkYQ7zp4TORGIFN3I=', 'Content-Length': '0'} 2024-02-01T10:50:11.796 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a1e8cd58ee748d6-0065bb7763-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:11.840 DEBUG:boto:b'vclfcn-64obj51000falseobj52024-02-01T10:49:22.402Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:11.841 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-02-01T10:50:11.841 DEBUG:boto:path=/vclfcn-64/obj6 2024-02-01T10:50:11.841 DEBUG:boto:auth_path=/vclfcn-64/obj6 2024-02-01T10:50:11.841 DEBUG:boto:path=/vclfcn-64/obj6?versionId=null 2024-02-01T10:50:11.841 DEBUG:boto:auth_path=/vclfcn-64/obj6?versionId=null 2024-02-01T10:50:11.841 DEBUG:boto:Method: GET 2024-02-01T10:50:11.841 DEBUG:boto:Path: /vclfcn-64/obj6?versionId=null 2024-02-01T10:50:11.841 DEBUG:boto:Data: 2024-02-01T10:50:11.841 DEBUG:boto:Headers: {} 2024-02-01T10:50:11.841 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:11.842 DEBUG:boto:Port: 8000 2024-02-01T10:50:11.842 DEBUG:boto:Params: {} 2024-02-01T10:50:11.842 DEBUG:boto:Token: None 2024-02-01T10:50:11.842 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:11 GMT /vclfcn-64/obj6?versionId=null 2024-02-01T10:50:11.842 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Q0tNcPiR4Ac7CBtnIvii4IKfMnE= 2024-02-01T10:50:11.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Q0tNcPiR4Ac7CBtnIvii4IKfMnE=', 'Content-Length': '0'} 2024-02-01T10:50:11.846 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000078cd41a473edeb34-0065bb7763-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:11.846 DEBUG:boto:path=/vclfcn-64/obj6 2024-02-01T10:50:11.846 DEBUG:boto:auth_path=/vclfcn-64/obj6 2024-02-01T10:50:11.846 DEBUG:boto:path=/vclfcn-64/obj6?versionId=null 2024-02-01T10:50:11.846 DEBUG:boto:auth_path=/vclfcn-64/obj6?versionId=null 2024-02-01T10:50:11.846 DEBUG:boto:Method: GET 2024-02-01T10:50:11.846 DEBUG:boto:Path: /vclfcn-64/obj6?versionId=null 2024-02-01T10:50:11.846 DEBUG:boto:Data: 2024-02-01T10:50:11.847 DEBUG:boto:Headers: {} 2024-02-01T10:50:11.847 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:11.847 DEBUG:boto:Port: 8000 2024-02-01T10:50:11.847 DEBUG:boto:Params: {} 2024-02-01T10:50:11.848 DEBUG:boto:Token: None 2024-02-01T10:50:11.848 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:11 GMT /vclfcn-64/obj6?versionId=null 2024-02-01T10:50:11.848 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Q0tNcPiR4Ac7CBtnIvii4IKfMnE= 2024-02-01T10:50:11.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Q0tNcPiR4Ac7CBtnIvii4IKfMnE=', 'Content-Length': '0'} 2024-02-01T10:50:11.851 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cb0cfae7d6289fd4-0065bb7763-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:11.896 DEBUG:boto:path=/vclfcn-64/obj6 2024-02-01T10:50:11.896 DEBUG:boto:auth_path=/vclfcn-64/obj6 2024-02-01T10:50:11.896 DEBUG:boto:path=/vclfcn-64/obj6?versionId=null 2024-02-01T10:50:11.896 DEBUG:boto:auth_path=/vclfcn-64/obj6?versionId=null 2024-02-01T10:50:11.897 DEBUG:boto:Method: HEAD 2024-02-01T10:50:11.897 DEBUG:boto:Path: /vclfcn-64/obj6?versionId=null 2024-02-01T10:50:11.897 DEBUG:boto:Data: 2024-02-01T10:50:11.897 DEBUG:boto:Headers: {} 2024-02-01T10:50:11.897 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:11.897 DEBUG:boto:Port: 8000 2024-02-01T10:50:11.897 DEBUG:boto:Params: {} 2024-02-01T10:50:11.897 DEBUG:boto:Token: None 2024-02-01T10:50:11.897 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:11 GMT /vclfcn-64/obj6?versionId=null 2024-02-01T10:50:11.898 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NMep7CH+GpISpHRSlnu/5AOswxk= 2024-02-01T10:50:11.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NMep7CH+GpISpHRSlnu/5AOswxk=', 'Content-Length': '0'} 2024-02-01T10:50:11.901 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fb115285f0404066-0065bb7763-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:11.901 DEBUG:boto:path=/vclfcn-64/obj6 2024-02-01T10:50:11.902 DEBUG:boto:auth_path=/vclfcn-64/obj6 2024-02-01T10:50:11.902 DEBUG:boto:path=/vclfcn-64/obj6?versionId=null 2024-02-01T10:50:11.902 DEBUG:boto:auth_path=/vclfcn-64/obj6?versionId=null 2024-02-01T10:50:11.902 DEBUG:boto:Method: HEAD 2024-02-01T10:50:11.902 DEBUG:boto:Path: /vclfcn-64/obj6?versionId=null 2024-02-01T10:50:11.902 DEBUG:boto:Data: 2024-02-01T10:50:11.902 DEBUG:boto:Headers: {} 2024-02-01T10:50:11.902 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:11.902 DEBUG:boto:Port: 8000 2024-02-01T10:50:11.903 DEBUG:boto:Params: {} 2024-02-01T10:50:11.903 DEBUG:boto:Token: None 2024-02-01T10:50:11.903 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:11 GMT /vclfcn-64/obj6?versionId=null 2024-02-01T10:50:11.903 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NMep7CH+GpISpHRSlnu/5AOswxk= 2024-02-01T10:50:11.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NMep7CH+GpISpHRSlnu/5AOswxk=', 'Content-Length': '0'} 2024-02-01T10:50:11.906 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d8736d94a2745a95-0065bb7763-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:11.907 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-02-01T10:50:11.907 DEBUG:boto:path=/vclfcn-64/obj6 2024-02-01T10:50:11.907 DEBUG:boto:auth_path=/vclfcn-64/obj6 2024-02-01T10:50:11.907 DEBUG:boto:path=/vclfcn-64/obj6?versionId=null 2024-02-01T10:50:11.907 DEBUG:boto:auth_path=/vclfcn-64/obj6?versionId=null 2024-02-01T10:50:11.907 DEBUG:boto:Method: GET 2024-02-01T10:50:11.907 DEBUG:boto:Path: /vclfcn-64/obj6?versionId=null 2024-02-01T10:50:11.907 DEBUG:boto:Data: 2024-02-01T10:50:11.908 DEBUG:boto:Headers: {} 2024-02-01T10:50:11.908 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:11.908 DEBUG:boto:Port: 8000 2024-02-01T10:50:11.908 DEBUG:boto:Params: {} 2024-02-01T10:50:11.908 DEBUG:boto:Token: None 2024-02-01T10:50:11.908 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:11 GMT /vclfcn-64/obj6?versionId=null 2024-02-01T10:50:11.908 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Q0tNcPiR4Ac7CBtnIvii4IKfMnE= 2024-02-01T10:50:11.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Q0tNcPiR4Ac7CBtnIvii4IKfMnE=', 'Content-Length': '0'} 2024-02-01T10:50:11.911 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000004160372f2211e15-0065bb7763-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:11.956 DEBUG:boto:path=/vclfcn-64/obj6 2024-02-01T10:50:11.956 DEBUG:boto:auth_path=/vclfcn-64/obj6 2024-02-01T10:50:11.956 DEBUG:boto:path=/vclfcn-64/obj6?versionId=null 2024-02-01T10:50:11.957 DEBUG:boto:auth_path=/vclfcn-64/obj6?versionId=null 2024-02-01T10:50:11.957 DEBUG:boto:Method: GET 2024-02-01T10:50:11.957 DEBUG:boto:Path: /vclfcn-64/obj6?versionId=null 2024-02-01T10:50:11.957 DEBUG:boto:Data: 2024-02-01T10:50:11.957 DEBUG:boto:Headers: {} 2024-02-01T10:50:11.957 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:11.957 DEBUG:boto:Port: 8000 2024-02-01T10:50:11.957 DEBUG:boto:Params: {} 2024-02-01T10:50:11.957 DEBUG:boto:Token: None 2024-02-01T10:50:11.957 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:11 GMT /vclfcn-64/obj6?versionId=null 2024-02-01T10:50:11.958 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Q0tNcPiR4Ac7CBtnIvii4IKfMnE= 2024-02-01T10:50:11.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:11 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Q0tNcPiR4Ac7CBtnIvii4IKfMnE=', 'Content-Length': '0'} 2024-02-01T10:50:11.961 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f091bd791e1978df-0065bb7763-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:11 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:12.004 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:12.004 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:12.004 DEBUG:boto:path=/vclfcn-64/?prefix=obj6 2024-02-01T10:50:12.005 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj6 2024-02-01T10:50:12.005 DEBUG:boto:Method: GET 2024-02-01T10:50:12.005 DEBUG:boto:Path: /vclfcn-64/?prefix=obj6 2024-02-01T10:50:12.005 DEBUG:boto:Data: 2024-02-01T10:50:12.005 DEBUG:boto:Headers: {} 2024-02-01T10:50:12.005 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:12.005 DEBUG:boto:Port: 8000 2024-02-01T10:50:12.005 DEBUG:boto:Params: {} 2024-02-01T10:50:12.005 DEBUG:boto:Token: None 2024-02-01T10:50:12.006 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:12 GMT /vclfcn-64/ 2024-02-01T10:50:12.006 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:qziLNDSFHlLlEoTowv8AgejDfiA= 2024-02-01T10:50:12.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:qziLNDSFHlLlEoTowv8AgejDfiA=', 'Content-Length': '0'} 2024-02-01T10:50:12.010 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5c8a8fb7a7cf6f9-0065bb7764-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:12.010 DEBUG:boto:b'vclfcn-64obj61000falseobj62024-02-01T10:49:22.455Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:12.011 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:12.011 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:12.011 DEBUG:boto:path=/vclfcn-64/?prefix=obj6 2024-02-01T10:50:12.011 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj6 2024-02-01T10:50:12.011 DEBUG:boto:Method: GET 2024-02-01T10:50:12.011 DEBUG:boto:Path: /vclfcn-64/?prefix=obj6 2024-02-01T10:50:12.011 DEBUG:boto:Data: 2024-02-01T10:50:12.011 DEBUG:boto:Headers: {} 2024-02-01T10:50:12.012 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:12.012 DEBUG:boto:Port: 8000 2024-02-01T10:50:12.012 DEBUG:boto:Params: {} 2024-02-01T10:50:12.012 DEBUG:boto:Token: None 2024-02-01T10:50:12.012 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:12 GMT /vclfcn-64/ 2024-02-01T10:50:12.012 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:qziLNDSFHlLlEoTowv8AgejDfiA= 2024-02-01T10:50:12.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:qziLNDSFHlLlEoTowv8AgejDfiA=', 'Content-Length': '0'} 2024-02-01T10:50:12.016 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc3b07ae1e0258cd-0065bb7764-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:12.060 DEBUG:boto:b'vclfcn-64obj61000falseobj62024-02-01T10:49:22.455Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:12.061 DEBUG:boto:path=/vclfcn-64/obj6 2024-02-01T10:50:12.061 DEBUG:boto:auth_path=/vclfcn-64/obj6 2024-02-01T10:50:12.061 DEBUG:boto:Method: HEAD 2024-02-01T10:50:12.061 DEBUG:boto:Path: /vclfcn-64/obj6 2024-02-01T10:50:12.061 DEBUG:boto:Data: 2024-02-01T10:50:12.061 DEBUG:boto:Headers: {} 2024-02-01T10:50:12.061 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:12.061 DEBUG:boto:Port: 8000 2024-02-01T10:50:12.061 DEBUG:boto:Params: {} 2024-02-01T10:50:12.062 DEBUG:boto:Token: None 2024-02-01T10:50:12.062 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:12 GMT /vclfcn-64/obj6 2024-02-01T10:50:12.062 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vwLmLkV8iGNh8scqFw6IOggmw3I= 2024-02-01T10:50:12.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vwLmLkV8iGNh8scqFw6IOggmw3I=', 'Content-Length': '0'} 2024-02-01T10:50:12.065 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000260520289cf0af80-0065bb7764-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:12.065 DEBUG:boto:path=/vclfcn-64/obj6 2024-02-01T10:50:12.065 DEBUG:boto:auth_path=/vclfcn-64/obj6 2024-02-01T10:50:12.065 DEBUG:boto:Method: HEAD 2024-02-01T10:50:12.065 DEBUG:boto:Path: /vclfcn-64/obj6 2024-02-01T10:50:12.065 DEBUG:boto:Data: 2024-02-01T10:50:12.065 DEBUG:boto:Headers: {} 2024-02-01T10:50:12.065 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:12.066 DEBUG:boto:Port: 8000 2024-02-01T10:50:12.066 DEBUG:boto:Params: {} 2024-02-01T10:50:12.066 DEBUG:boto:Token: None 2024-02-01T10:50:12.066 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:12 GMT /vclfcn-64/obj6 2024-02-01T10:50:12.066 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:vwLmLkV8iGNh8scqFw6IOggmw3I= 2024-02-01T10:50:12.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:vwLmLkV8iGNh8scqFw6IOggmw3I=', 'Content-Length': '0'} 2024-02-01T10:50:12.069 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a7451033b143c226-0065bb7764-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:12.069 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-02-01T10:50:12.069 DEBUG:boto:path=/vclfcn-64/obj6 2024-02-01T10:50:12.070 DEBUG:boto:auth_path=/vclfcn-64/obj6 2024-02-01T10:50:12.070 DEBUG:boto:Method: GET 2024-02-01T10:50:12.070 DEBUG:boto:Path: /vclfcn-64/obj6 2024-02-01T10:50:12.070 DEBUG:boto:Data: 2024-02-01T10:50:12.070 DEBUG:boto:Headers: {} 2024-02-01T10:50:12.070 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:12.070 DEBUG:boto:Port: 8000 2024-02-01T10:50:12.070 DEBUG:boto:Params: {} 2024-02-01T10:50:12.070 DEBUG:boto:Token: None 2024-02-01T10:50:12.071 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:12 GMT /vclfcn-64/obj6 2024-02-01T10:50:12.071 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1Rly7yxsaS4jrue3JWcumh3T1AQ= 2024-02-01T10:50:12.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1Rly7yxsaS4jrue3JWcumh3T1AQ=', 'Content-Length': '0'} 2024-02-01T10:50:12.074 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008551775d39a36c2c-0065bb7764-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:12.120 DEBUG:boto:path=/vclfcn-64/obj6 2024-02-01T10:50:12.120 DEBUG:boto:auth_path=/vclfcn-64/obj6 2024-02-01T10:50:12.121 DEBUG:boto:Method: GET 2024-02-01T10:50:12.121 DEBUG:boto:Path: /vclfcn-64/obj6 2024-02-01T10:50:12.121 DEBUG:boto:Data: 2024-02-01T10:50:12.121 DEBUG:boto:Headers: {} 2024-02-01T10:50:12.121 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:12.121 DEBUG:boto:Port: 8000 2024-02-01T10:50:12.121 DEBUG:boto:Params: {} 2024-02-01T10:50:12.122 DEBUG:boto:Token: None 2024-02-01T10:50:12.122 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:12 GMT /vclfcn-64/obj6 2024-02-01T10:50:12.122 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1Rly7yxsaS4jrue3JWcumh3T1AQ= 2024-02-01T10:50:12.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1Rly7yxsaS4jrue3JWcumh3T1AQ=', 'Content-Length': '0'} 2024-02-01T10:50:12.126 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007441beadbf95e7ca-0065bb7764-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:12.168 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:12.168 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:12.169 DEBUG:boto:path=/vclfcn-64/?prefix=obj6 2024-02-01T10:50:12.169 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj6 2024-02-01T10:50:12.169 DEBUG:boto:Method: GET 2024-02-01T10:50:12.169 DEBUG:boto:Path: /vclfcn-64/?prefix=obj6 2024-02-01T10:50:12.169 DEBUG:boto:Data: 2024-02-01T10:50:12.169 DEBUG:boto:Headers: {} 2024-02-01T10:50:12.169 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:12.169 DEBUG:boto:Port: 8000 2024-02-01T10:50:12.169 DEBUG:boto:Params: {} 2024-02-01T10:50:12.169 DEBUG:boto:Token: None 2024-02-01T10:50:12.170 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:12 GMT /vclfcn-64/ 2024-02-01T10:50:12.170 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:qziLNDSFHlLlEoTowv8AgejDfiA= 2024-02-01T10:50:12.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:qziLNDSFHlLlEoTowv8AgejDfiA=', 'Content-Length': '0'} 2024-02-01T10:50:12.174 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef9e541548cf16a3-0065bb7764-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:12.220 DEBUG:boto:b'vclfcn-64obj61000falseobj62024-02-01T10:49:22.455Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:12.221 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:12.221 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:12.221 DEBUG:boto:path=/vclfcn-64/?prefix=obj6 2024-02-01T10:50:12.221 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj6 2024-02-01T10:50:12.221 DEBUG:boto:Method: GET 2024-02-01T10:50:12.221 DEBUG:boto:Path: /vclfcn-64/?prefix=obj6 2024-02-01T10:50:12.221 DEBUG:boto:Data: 2024-02-01T10:50:12.221 DEBUG:boto:Headers: {} 2024-02-01T10:50:12.221 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:12.221 DEBUG:boto:Port: 8000 2024-02-01T10:50:12.221 DEBUG:boto:Params: {} 2024-02-01T10:50:12.222 DEBUG:boto:Token: None 2024-02-01T10:50:12.222 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:12 GMT /vclfcn-64/ 2024-02-01T10:50:12.222 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:qziLNDSFHlLlEoTowv8AgejDfiA= 2024-02-01T10:50:12.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:qziLNDSFHlLlEoTowv8AgejDfiA=', 'Content-Length': '0'} 2024-02-01T10:50:12.226 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd82ef5e6859f01b-0065bb7764-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:12.226 DEBUG:boto:b'vclfcn-64obj61000falseobj62024-02-01T10:49:22.455Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:12.227 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-02-01T10:50:12.227 DEBUG:boto:path=/vclfcn-64/obj7 2024-02-01T10:50:12.227 DEBUG:boto:auth_path=/vclfcn-64/obj7 2024-02-01T10:50:12.227 DEBUG:boto:path=/vclfcn-64/obj7?versionId=null 2024-02-01T10:50:12.227 DEBUG:boto:auth_path=/vclfcn-64/obj7?versionId=null 2024-02-01T10:50:12.227 DEBUG:boto:Method: GET 2024-02-01T10:50:12.227 DEBUG:boto:Path: /vclfcn-64/obj7?versionId=null 2024-02-01T10:50:12.227 DEBUG:boto:Data: 2024-02-01T10:50:12.227 DEBUG:boto:Headers: {} 2024-02-01T10:50:12.228 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:12.228 DEBUG:boto:Port: 8000 2024-02-01T10:50:12.228 DEBUG:boto:Params: {} 2024-02-01T10:50:12.228 DEBUG:boto:Token: None 2024-02-01T10:50:12.228 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:12 GMT /vclfcn-64/obj7?versionId=null 2024-02-01T10:50:12.228 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nUKfkvISShAmSDV0j9VDgGBX+6w= 2024-02-01T10:50:12.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nUKfkvISShAmSDV0j9VDgGBX+6w=', 'Content-Length': '0'} 2024-02-01T10:50:12.231 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000075bce36bb0fbc23b-0065bb7764-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:12.231 DEBUG:boto:path=/vclfcn-64/obj7 2024-02-01T10:50:12.231 DEBUG:boto:auth_path=/vclfcn-64/obj7 2024-02-01T10:50:12.232 DEBUG:boto:path=/vclfcn-64/obj7?versionId=null 2024-02-01T10:50:12.232 DEBUG:boto:auth_path=/vclfcn-64/obj7?versionId=null 2024-02-01T10:50:12.232 DEBUG:boto:Method: GET 2024-02-01T10:50:12.232 DEBUG:boto:Path: /vclfcn-64/obj7?versionId=null 2024-02-01T10:50:12.232 DEBUG:boto:Data: 2024-02-01T10:50:12.232 DEBUG:boto:Headers: {} 2024-02-01T10:50:12.232 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:12.232 DEBUG:boto:Port: 8000 2024-02-01T10:50:12.232 DEBUG:boto:Params: {} 2024-02-01T10:50:12.232 DEBUG:boto:Token: None 2024-02-01T10:50:12.233 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:12 GMT /vclfcn-64/obj7?versionId=null 2024-02-01T10:50:12.233 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nUKfkvISShAmSDV0j9VDgGBX+6w= 2024-02-01T10:50:12.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nUKfkvISShAmSDV0j9VDgGBX+6w=', 'Content-Length': '0'} 2024-02-01T10:50:12.236 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008a49fa8bd49a2836-0065bb7764-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:12.276 DEBUG:boto:path=/vclfcn-64/obj7 2024-02-01T10:50:12.276 DEBUG:boto:auth_path=/vclfcn-64/obj7 2024-02-01T10:50:12.276 DEBUG:boto:path=/vclfcn-64/obj7?versionId=null 2024-02-01T10:50:12.276 DEBUG:boto:auth_path=/vclfcn-64/obj7?versionId=null 2024-02-01T10:50:12.277 DEBUG:boto:Method: HEAD 2024-02-01T10:50:12.277 DEBUG:boto:Path: /vclfcn-64/obj7?versionId=null 2024-02-01T10:50:12.277 DEBUG:boto:Data: 2024-02-01T10:50:12.277 DEBUG:boto:Headers: {} 2024-02-01T10:50:12.277 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:12.277 DEBUG:boto:Port: 8000 2024-02-01T10:50:12.277 DEBUG:boto:Params: {} 2024-02-01T10:50:12.277 DEBUG:boto:Token: None 2024-02-01T10:50:12.277 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:12 GMT /vclfcn-64/obj7?versionId=null 2024-02-01T10:50:12.278 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:iGlp1K0TEsiXElZHpTqMeLqOsds= 2024-02-01T10:50:12.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iGlp1K0TEsiXElZHpTqMeLqOsds=', 'Content-Length': '0'} 2024-02-01T10:50:12.280 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000049c05526bd2ca2df-0065bb7764-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:12.281 DEBUG:boto:path=/vclfcn-64/obj7 2024-02-01T10:50:12.281 DEBUG:boto:auth_path=/vclfcn-64/obj7 2024-02-01T10:50:12.281 DEBUG:boto:path=/vclfcn-64/obj7?versionId=null 2024-02-01T10:50:12.281 DEBUG:boto:auth_path=/vclfcn-64/obj7?versionId=null 2024-02-01T10:50:12.281 DEBUG:boto:Method: HEAD 2024-02-01T10:50:12.281 DEBUG:boto:Path: /vclfcn-64/obj7?versionId=null 2024-02-01T10:50:12.281 DEBUG:boto:Data: 2024-02-01T10:50:12.281 DEBUG:boto:Headers: {} 2024-02-01T10:50:12.282 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:12.282 DEBUG:boto:Port: 8000 2024-02-01T10:50:12.282 DEBUG:boto:Params: {} 2024-02-01T10:50:12.282 DEBUG:boto:Token: None 2024-02-01T10:50:12.282 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:12 GMT /vclfcn-64/obj7?versionId=null 2024-02-01T10:50:12.282 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:iGlp1K0TEsiXElZHpTqMeLqOsds= 2024-02-01T10:50:12.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iGlp1K0TEsiXElZHpTqMeLqOsds=', 'Content-Length': '0'} 2024-02-01T10:50:12.285 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000076363eb1edc4f6f3-0065bb7764-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:12.285 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-02-01T10:50:12.286 DEBUG:boto:path=/vclfcn-64/obj7 2024-02-01T10:50:12.286 DEBUG:boto:auth_path=/vclfcn-64/obj7 2024-02-01T10:50:12.286 DEBUG:boto:path=/vclfcn-64/obj7?versionId=null 2024-02-01T10:50:12.286 DEBUG:boto:auth_path=/vclfcn-64/obj7?versionId=null 2024-02-01T10:50:12.286 DEBUG:boto:Method: GET 2024-02-01T10:50:12.286 DEBUG:boto:Path: /vclfcn-64/obj7?versionId=null 2024-02-01T10:50:12.286 DEBUG:boto:Data: 2024-02-01T10:50:12.286 DEBUG:boto:Headers: {} 2024-02-01T10:50:12.286 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:12.287 DEBUG:boto:Port: 8000 2024-02-01T10:50:12.287 DEBUG:boto:Params: {} 2024-02-01T10:50:12.287 DEBUG:boto:Token: None 2024-02-01T10:50:12.287 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:12 GMT /vclfcn-64/obj7?versionId=null 2024-02-01T10:50:12.287 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nUKfkvISShAmSDV0j9VDgGBX+6w= 2024-02-01T10:50:12.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nUKfkvISShAmSDV0j9VDgGBX+6w=', 'Content-Length': '0'} 2024-02-01T10:50:12.290 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000098e62e23fc282ac8-0065bb7764-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:12.332 DEBUG:boto:path=/vclfcn-64/obj7 2024-02-01T10:50:12.332 DEBUG:boto:auth_path=/vclfcn-64/obj7 2024-02-01T10:50:12.332 DEBUG:boto:path=/vclfcn-64/obj7?versionId=null 2024-02-01T10:50:12.332 DEBUG:boto:auth_path=/vclfcn-64/obj7?versionId=null 2024-02-01T10:50:12.332 DEBUG:boto:Method: GET 2024-02-01T10:50:12.333 DEBUG:boto:Path: /vclfcn-64/obj7?versionId=null 2024-02-01T10:50:12.333 DEBUG:boto:Data: 2024-02-01T10:50:12.333 DEBUG:boto:Headers: {} 2024-02-01T10:50:12.333 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:12.333 DEBUG:boto:Port: 8000 2024-02-01T10:50:12.333 DEBUG:boto:Params: {} 2024-02-01T10:50:12.333 DEBUG:boto:Token: None 2024-02-01T10:50:12.333 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:12 GMT /vclfcn-64/obj7?versionId=null 2024-02-01T10:50:12.333 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nUKfkvISShAmSDV0j9VDgGBX+6w= 2024-02-01T10:50:12.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nUKfkvISShAmSDV0j9VDgGBX+6w=', 'Content-Length': '0'} 2024-02-01T10:50:12.337 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e6217cae287ed27c-0065bb7764-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:12.380 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:12.380 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:12.380 DEBUG:boto:path=/vclfcn-64/?prefix=obj7 2024-02-01T10:50:12.381 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj7 2024-02-01T10:50:12.381 DEBUG:boto:Method: GET 2024-02-01T10:50:12.381 DEBUG:boto:Path: /vclfcn-64/?prefix=obj7 2024-02-01T10:50:12.381 DEBUG:boto:Data: 2024-02-01T10:50:12.381 DEBUG:boto:Headers: {} 2024-02-01T10:50:12.381 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:12.381 DEBUG:boto:Port: 8000 2024-02-01T10:50:12.381 DEBUG:boto:Params: {} 2024-02-01T10:50:12.381 DEBUG:boto:Token: None 2024-02-01T10:50:12.381 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:12 GMT /vclfcn-64/ 2024-02-01T10:50:12.382 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:qziLNDSFHlLlEoTowv8AgejDfiA= 2024-02-01T10:50:12.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:qziLNDSFHlLlEoTowv8AgejDfiA=', 'Content-Length': '0'} 2024-02-01T10:50:12.386 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aafc2230f263b1bd-0065bb7764-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:12.428 DEBUG:boto:b'vclfcn-64obj71000falseobj72024-02-01T10:49:22.471Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:12.428 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:12.429 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:12.429 DEBUG:boto:path=/vclfcn-64/?prefix=obj7 2024-02-01T10:50:12.429 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj7 2024-02-01T10:50:12.429 DEBUG:boto:Method: GET 2024-02-01T10:50:12.429 DEBUG:boto:Path: /vclfcn-64/?prefix=obj7 2024-02-01T10:50:12.429 DEBUG:boto:Data: 2024-02-01T10:50:12.429 DEBUG:boto:Headers: {} 2024-02-01T10:50:12.429 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:12.429 DEBUG:boto:Port: 8000 2024-02-01T10:50:12.429 DEBUG:boto:Params: {} 2024-02-01T10:50:12.429 DEBUG:boto:Token: None 2024-02-01T10:50:12.430 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:12 GMT /vclfcn-64/ 2024-02-01T10:50:12.430 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:qziLNDSFHlLlEoTowv8AgejDfiA= 2024-02-01T10:50:12.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:qziLNDSFHlLlEoTowv8AgejDfiA=', 'Content-Length': '0'} 2024-02-01T10:50:12.434 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c30567e423bc892-0065bb7764-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:12.434 DEBUG:boto:b'vclfcn-64obj71000falseobj72024-02-01T10:49:22.471Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:12.435 DEBUG:boto:path=/vclfcn-64/obj7 2024-02-01T10:50:12.435 DEBUG:boto:auth_path=/vclfcn-64/obj7 2024-02-01T10:50:12.435 DEBUG:boto:Method: HEAD 2024-02-01T10:50:12.435 DEBUG:boto:Path: /vclfcn-64/obj7 2024-02-01T10:50:12.435 DEBUG:boto:Data: 2024-02-01T10:50:12.435 DEBUG:boto:Headers: {} 2024-02-01T10:50:12.435 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:12.435 DEBUG:boto:Port: 8000 2024-02-01T10:50:12.436 DEBUG:boto:Params: {} 2024-02-01T10:50:12.436 DEBUG:boto:Token: None 2024-02-01T10:50:12.436 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:12 GMT /vclfcn-64/obj7 2024-02-01T10:50:12.436 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Tab5t6OHyKavsTfkeWmL8rFoIg4= 2024-02-01T10:50:12.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Tab5t6OHyKavsTfkeWmL8rFoIg4=', 'Content-Length': '0'} 2024-02-01T10:50:12.439 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fbcf8214798febda-0065bb7764-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:12.439 DEBUG:boto:path=/vclfcn-64/obj7 2024-02-01T10:50:12.439 DEBUG:boto:auth_path=/vclfcn-64/obj7 2024-02-01T10:50:12.439 DEBUG:boto:Method: HEAD 2024-02-01T10:50:12.439 DEBUG:boto:Path: /vclfcn-64/obj7 2024-02-01T10:50:12.440 DEBUG:boto:Data: 2024-02-01T10:50:12.440 DEBUG:boto:Headers: {} 2024-02-01T10:50:12.440 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:12.440 DEBUG:boto:Port: 8000 2024-02-01T10:50:12.440 DEBUG:boto:Params: {} 2024-02-01T10:50:12.440 DEBUG:boto:Token: None 2024-02-01T10:50:12.440 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:12 GMT /vclfcn-64/obj7 2024-02-01T10:50:12.440 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Tab5t6OHyKavsTfkeWmL8rFoIg4= 2024-02-01T10:50:12.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Tab5t6OHyKavsTfkeWmL8rFoIg4=', 'Content-Length': '0'} 2024-02-01T10:50:12.443 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002dcb23f0301ce5a4-0065bb7764-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:12.443 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-02-01T10:50:12.444 DEBUG:boto:path=/vclfcn-64/obj7 2024-02-01T10:50:12.444 DEBUG:boto:auth_path=/vclfcn-64/obj7 2024-02-01T10:50:12.444 DEBUG:boto:Method: GET 2024-02-01T10:50:12.444 DEBUG:boto:Path: /vclfcn-64/obj7 2024-02-01T10:50:12.444 DEBUG:boto:Data: 2024-02-01T10:50:12.444 DEBUG:boto:Headers: {} 2024-02-01T10:50:12.444 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:12.444 DEBUG:boto:Port: 8000 2024-02-01T10:50:12.444 DEBUG:boto:Params: {} 2024-02-01T10:50:12.444 DEBUG:boto:Token: None 2024-02-01T10:50:12.445 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:12 GMT /vclfcn-64/obj7 2024-02-01T10:50:12.445 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mfJP+IrBpGkhUVOQB/+AayIzRcE= 2024-02-01T10:50:12.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mfJP+IrBpGkhUVOQB/+AayIzRcE=', 'Content-Length': '0'} 2024-02-01T10:50:12.448 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000df1674527a074caa-0065bb7764-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:12.492 DEBUG:boto:path=/vclfcn-64/obj7 2024-02-01T10:50:12.492 DEBUG:boto:auth_path=/vclfcn-64/obj7 2024-02-01T10:50:12.492 DEBUG:boto:Method: GET 2024-02-01T10:50:12.492 DEBUG:boto:Path: /vclfcn-64/obj7 2024-02-01T10:50:12.492 DEBUG:boto:Data: 2024-02-01T10:50:12.493 DEBUG:boto:Headers: {} 2024-02-01T10:50:12.493 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:12.493 DEBUG:boto:Port: 8000 2024-02-01T10:50:12.493 DEBUG:boto:Params: {} 2024-02-01T10:50:12.494 DEBUG:boto:Token: None 2024-02-01T10:50:12.494 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:12 GMT /vclfcn-64/obj7 2024-02-01T10:50:12.494 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mfJP+IrBpGkhUVOQB/+AayIzRcE= 2024-02-01T10:50:12.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mfJP+IrBpGkhUVOQB/+AayIzRcE=', 'Content-Length': '0'} 2024-02-01T10:50:12.497 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f190ed45fd533227-0065bb7764-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:12.544 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:12.544 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:12.544 DEBUG:boto:path=/vclfcn-64/?prefix=obj7 2024-02-01T10:50:12.544 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj7 2024-02-01T10:50:12.545 DEBUG:boto:Method: GET 2024-02-01T10:50:12.545 DEBUG:boto:Path: /vclfcn-64/?prefix=obj7 2024-02-01T10:50:12.545 DEBUG:boto:Data: 2024-02-01T10:50:12.545 DEBUG:boto:Headers: {} 2024-02-01T10:50:12.545 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:12.545 DEBUG:boto:Port: 8000 2024-02-01T10:50:12.545 DEBUG:boto:Params: {} 2024-02-01T10:50:12.545 DEBUG:boto:Token: None 2024-02-01T10:50:12.546 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:12 GMT /vclfcn-64/ 2024-02-01T10:50:12.546 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:qziLNDSFHlLlEoTowv8AgejDfiA= 2024-02-01T10:50:12.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:qziLNDSFHlLlEoTowv8AgejDfiA=', 'Content-Length': '0'} 2024-02-01T10:50:12.550 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d5329932851f5df-0065bb7764-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:12.592 DEBUG:boto:b'vclfcn-64obj71000falseobj72024-02-01T10:49:22.471Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:12.593 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:12.593 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:12.593 DEBUG:boto:path=/vclfcn-64/?prefix=obj7 2024-02-01T10:50:12.593 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj7 2024-02-01T10:50:12.593 DEBUG:boto:Method: GET 2024-02-01T10:50:12.593 DEBUG:boto:Path: /vclfcn-64/?prefix=obj7 2024-02-01T10:50:12.593 DEBUG:boto:Data: 2024-02-01T10:50:12.593 DEBUG:boto:Headers: {} 2024-02-01T10:50:12.593 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:12.593 DEBUG:boto:Port: 8000 2024-02-01T10:50:12.594 DEBUG:boto:Params: {} 2024-02-01T10:50:12.594 DEBUG:boto:Token: None 2024-02-01T10:50:12.594 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:12 GMT /vclfcn-64/ 2024-02-01T10:50:12.594 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:qziLNDSFHlLlEoTowv8AgejDfiA= 2024-02-01T10:50:12.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:qziLNDSFHlLlEoTowv8AgejDfiA=', 'Content-Length': '0'} 2024-02-01T10:50:12.598 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012919256dbf726d8-0065bb7764-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:12.599 DEBUG:boto:b'vclfcn-64obj71000falseobj72024-02-01T10:49:22.471Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:12.599 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-02-01T10:50:12.599 DEBUG:boto:path=/vclfcn-64/obj8 2024-02-01T10:50:12.599 DEBUG:boto:auth_path=/vclfcn-64/obj8 2024-02-01T10:50:12.599 DEBUG:boto:path=/vclfcn-64/obj8?versionId=null 2024-02-01T10:50:12.599 DEBUG:boto:auth_path=/vclfcn-64/obj8?versionId=null 2024-02-01T10:50:12.600 DEBUG:boto:Method: GET 2024-02-01T10:50:12.600 DEBUG:boto:Path: /vclfcn-64/obj8?versionId=null 2024-02-01T10:50:12.600 DEBUG:boto:Data: 2024-02-01T10:50:12.600 DEBUG:boto:Headers: {} 2024-02-01T10:50:12.600 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:12.600 DEBUG:boto:Port: 8000 2024-02-01T10:50:12.600 DEBUG:boto:Params: {} 2024-02-01T10:50:12.600 DEBUG:boto:Token: None 2024-02-01T10:50:12.600 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:12 GMT /vclfcn-64/obj8?versionId=null 2024-02-01T10:50:12.600 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:l5MCu6JChkh6gXuteamnusfYEGY= 2024-02-01T10:50:12.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:l5MCu6JChkh6gXuteamnusfYEGY=', 'Content-Length': '0'} 2024-02-01T10:50:12.604 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000069f170edac95e210-0065bb7764-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:12.648 DEBUG:boto:path=/vclfcn-64/obj8 2024-02-01T10:50:12.648 DEBUG:boto:auth_path=/vclfcn-64/obj8 2024-02-01T10:50:12.649 DEBUG:boto:path=/vclfcn-64/obj8?versionId=null 2024-02-01T10:50:12.649 DEBUG:boto:auth_path=/vclfcn-64/obj8?versionId=null 2024-02-01T10:50:12.649 DEBUG:boto:Method: GET 2024-02-01T10:50:12.649 DEBUG:boto:Path: /vclfcn-64/obj8?versionId=null 2024-02-01T10:50:12.649 DEBUG:boto:Data: 2024-02-01T10:50:12.649 DEBUG:boto:Headers: {} 2024-02-01T10:50:12.649 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:12.649 DEBUG:boto:Port: 8000 2024-02-01T10:50:12.649 DEBUG:boto:Params: {} 2024-02-01T10:50:12.649 DEBUG:boto:Token: None 2024-02-01T10:50:12.650 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:12 GMT /vclfcn-64/obj8?versionId=null 2024-02-01T10:50:12.650 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:l5MCu6JChkh6gXuteamnusfYEGY= 2024-02-01T10:50:12.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:l5MCu6JChkh6gXuteamnusfYEGY=', 'Content-Length': '0'} 2024-02-01T10:50:12.653 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000016921eed4c10d1f4-0065bb7764-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:12.653 DEBUG:boto:path=/vclfcn-64/obj8 2024-02-01T10:50:12.653 DEBUG:boto:auth_path=/vclfcn-64/obj8 2024-02-01T10:50:12.653 DEBUG:boto:path=/vclfcn-64/obj8?versionId=null 2024-02-01T10:50:12.653 DEBUG:boto:auth_path=/vclfcn-64/obj8?versionId=null 2024-02-01T10:50:12.653 DEBUG:boto:Method: HEAD 2024-02-01T10:50:12.654 DEBUG:boto:Path: /vclfcn-64/obj8?versionId=null 2024-02-01T10:50:12.654 DEBUG:boto:Data: 2024-02-01T10:50:12.654 DEBUG:boto:Headers: {} 2024-02-01T10:50:12.654 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:12.654 DEBUG:boto:Port: 8000 2024-02-01T10:50:12.654 DEBUG:boto:Params: {} 2024-02-01T10:50:12.654 DEBUG:boto:Token: None 2024-02-01T10:50:12.654 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:12 GMT /vclfcn-64/obj8?versionId=null 2024-02-01T10:50:12.654 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tYhbE0vKQkja9KMuC4PHpf6CXOE= 2024-02-01T10:50:12.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tYhbE0vKQkja9KMuC4PHpf6CXOE=', 'Content-Length': '0'} 2024-02-01T10:50:12.657 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000050f600d14988cab3-0065bb7764-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:12.657 DEBUG:boto:path=/vclfcn-64/obj8 2024-02-01T10:50:12.657 DEBUG:boto:auth_path=/vclfcn-64/obj8 2024-02-01T10:50:12.658 DEBUG:boto:path=/vclfcn-64/obj8?versionId=null 2024-02-01T10:50:12.658 DEBUG:boto:auth_path=/vclfcn-64/obj8?versionId=null 2024-02-01T10:50:12.658 DEBUG:boto:Method: HEAD 2024-02-01T10:50:12.658 DEBUG:boto:Path: /vclfcn-64/obj8?versionId=null 2024-02-01T10:50:12.658 DEBUG:boto:Data: 2024-02-01T10:50:12.658 DEBUG:boto:Headers: {} 2024-02-01T10:50:12.658 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:12.658 DEBUG:boto:Port: 8000 2024-02-01T10:50:12.658 DEBUG:boto:Params: {} 2024-02-01T10:50:12.658 DEBUG:boto:Token: None 2024-02-01T10:50:12.659 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:12 GMT /vclfcn-64/obj8?versionId=null 2024-02-01T10:50:12.659 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:tYhbE0vKQkja9KMuC4PHpf6CXOE= 2024-02-01T10:50:12.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:tYhbE0vKQkja9KMuC4PHpf6CXOE=', 'Content-Length': '0'} 2024-02-01T10:50:12.661 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007ab618e9a764b7c1-0065bb7764-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:12.662 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-02-01T10:50:12.662 DEBUG:boto:path=/vclfcn-64/obj8 2024-02-01T10:50:12.662 DEBUG:boto:auth_path=/vclfcn-64/obj8 2024-02-01T10:50:12.662 DEBUG:boto:path=/vclfcn-64/obj8?versionId=null 2024-02-01T10:50:12.662 DEBUG:boto:auth_path=/vclfcn-64/obj8?versionId=null 2024-02-01T10:50:12.662 DEBUG:boto:Method: GET 2024-02-01T10:50:12.663 DEBUG:boto:Path: /vclfcn-64/obj8?versionId=null 2024-02-01T10:50:12.663 DEBUG:boto:Data: 2024-02-01T10:50:12.663 DEBUG:boto:Headers: {} 2024-02-01T10:50:12.663 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:12.663 DEBUG:boto:Port: 8000 2024-02-01T10:50:12.663 DEBUG:boto:Params: {} 2024-02-01T10:50:12.663 DEBUG:boto:Token: None 2024-02-01T10:50:12.663 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:12 GMT /vclfcn-64/obj8?versionId=null 2024-02-01T10:50:12.663 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:l5MCu6JChkh6gXuteamnusfYEGY= 2024-02-01T10:50:12.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:l5MCu6JChkh6gXuteamnusfYEGY=', 'Content-Length': '0'} 2024-02-01T10:50:12.666 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005115d3ac9ba1cf21-0065bb7764-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:12.708 DEBUG:boto:path=/vclfcn-64/obj8 2024-02-01T10:50:12.708 DEBUG:boto:auth_path=/vclfcn-64/obj8 2024-02-01T10:50:12.708 DEBUG:boto:path=/vclfcn-64/obj8?versionId=null 2024-02-01T10:50:12.709 DEBUG:boto:auth_path=/vclfcn-64/obj8?versionId=null 2024-02-01T10:50:12.709 DEBUG:boto:Method: GET 2024-02-01T10:50:12.709 DEBUG:boto:Path: /vclfcn-64/obj8?versionId=null 2024-02-01T10:50:12.709 DEBUG:boto:Data: 2024-02-01T10:50:12.709 DEBUG:boto:Headers: {} 2024-02-01T10:50:12.709 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:12.709 DEBUG:boto:Port: 8000 2024-02-01T10:50:12.709 DEBUG:boto:Params: {} 2024-02-01T10:50:12.709 DEBUG:boto:Token: None 2024-02-01T10:50:12.709 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:12 GMT /vclfcn-64/obj8?versionId=null 2024-02-01T10:50:12.710 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:l5MCu6JChkh6gXuteamnusfYEGY= 2024-02-01T10:50:12.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:l5MCu6JChkh6gXuteamnusfYEGY=', 'Content-Length': '0'} 2024-02-01T10:50:12.713 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c553c8e9ea0aeae3-0065bb7764-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:12.756 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:12.756 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:12.756 DEBUG:boto:path=/vclfcn-64/?prefix=obj8 2024-02-01T10:50:12.756 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj8 2024-02-01T10:50:12.757 DEBUG:boto:Method: GET 2024-02-01T10:50:12.757 DEBUG:boto:Path: /vclfcn-64/?prefix=obj8 2024-02-01T10:50:12.757 DEBUG:boto:Data: 2024-02-01T10:50:12.757 DEBUG:boto:Headers: {} 2024-02-01T10:50:12.757 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:12.757 DEBUG:boto:Port: 8000 2024-02-01T10:50:12.757 DEBUG:boto:Params: {} 2024-02-01T10:50:12.757 DEBUG:boto:Token: None 2024-02-01T10:50:12.757 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:12 GMT /vclfcn-64/ 2024-02-01T10:50:12.758 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:qziLNDSFHlLlEoTowv8AgejDfiA= 2024-02-01T10:50:12.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:qziLNDSFHlLlEoTowv8AgejDfiA=', 'Content-Length': '0'} 2024-02-01T10:50:12.762 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a70c91ad38305b8b-0065bb7764-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:12.804 DEBUG:boto:b'vclfcn-64obj81000falseobj82024-02-01T10:49:22.519Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:12.805 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:12.805 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:12.805 DEBUG:boto:path=/vclfcn-64/?prefix=obj8 2024-02-01T10:50:12.805 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj8 2024-02-01T10:50:12.805 DEBUG:boto:Method: GET 2024-02-01T10:50:12.805 DEBUG:boto:Path: /vclfcn-64/?prefix=obj8 2024-02-01T10:50:12.805 DEBUG:boto:Data: 2024-02-01T10:50:12.805 DEBUG:boto:Headers: {} 2024-02-01T10:50:12.805 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:12.805 DEBUG:boto:Port: 8000 2024-02-01T10:50:12.806 DEBUG:boto:Params: {} 2024-02-01T10:50:12.806 DEBUG:boto:Token: None 2024-02-01T10:50:12.806 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:12 GMT /vclfcn-64/ 2024-02-01T10:50:12.806 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:qziLNDSFHlLlEoTowv8AgejDfiA= 2024-02-01T10:50:12.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:qziLNDSFHlLlEoTowv8AgejDfiA=', 'Content-Length': '0'} 2024-02-01T10:50:12.811 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005bbf33245c71dac1-0065bb7764-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:12.812 DEBUG:boto:b'vclfcn-64obj81000falseobj82024-02-01T10:49:22.519Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:12.812 DEBUG:boto:path=/vclfcn-64/obj8 2024-02-01T10:50:12.812 DEBUG:boto:auth_path=/vclfcn-64/obj8 2024-02-01T10:50:12.812 DEBUG:boto:Method: HEAD 2024-02-01T10:50:12.812 DEBUG:boto:Path: /vclfcn-64/obj8 2024-02-01T10:50:12.812 DEBUG:boto:Data: 2024-02-01T10:50:12.813 DEBUG:boto:Headers: {} 2024-02-01T10:50:12.813 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:12.813 DEBUG:boto:Port: 8000 2024-02-01T10:50:12.813 DEBUG:boto:Params: {} 2024-02-01T10:50:12.813 DEBUG:boto:Token: None 2024-02-01T10:50:12.813 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:12 GMT /vclfcn-64/obj8 2024-02-01T10:50:12.813 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8sNc/Ym64tHJa+BdTO5H3g+oi60= 2024-02-01T10:50:12.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8sNc/Ym64tHJa+BdTO5H3g+oi60=', 'Content-Length': '0'} 2024-02-01T10:50:12.817 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b65a216d552b7a63-0065bb7764-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:12.817 DEBUG:boto:path=/vclfcn-64/obj8 2024-02-01T10:50:12.817 DEBUG:boto:auth_path=/vclfcn-64/obj8 2024-02-01T10:50:12.817 DEBUG:boto:Method: HEAD 2024-02-01T10:50:12.817 DEBUG:boto:Path: /vclfcn-64/obj8 2024-02-01T10:50:12.817 DEBUG:boto:Data: 2024-02-01T10:50:12.817 DEBUG:boto:Headers: {} 2024-02-01T10:50:12.818 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:12.818 DEBUG:boto:Port: 8000 2024-02-01T10:50:12.818 DEBUG:boto:Params: {} 2024-02-01T10:50:12.818 DEBUG:boto:Token: None 2024-02-01T10:50:12.818 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:12 GMT /vclfcn-64/obj8 2024-02-01T10:50:12.818 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:8sNc/Ym64tHJa+BdTO5H3g+oi60= 2024-02-01T10:50:12.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:8sNc/Ym64tHJa+BdTO5H3g+oi60=', 'Content-Length': '0'} 2024-02-01T10:50:12.821 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009170a4a82ed3f149-0065bb7764-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:12.821 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-02-01T10:50:12.821 DEBUG:boto:path=/vclfcn-64/obj8 2024-02-01T10:50:12.821 DEBUG:boto:auth_path=/vclfcn-64/obj8 2024-02-01T10:50:12.821 DEBUG:boto:Method: GET 2024-02-01T10:50:12.821 DEBUG:boto:Path: /vclfcn-64/obj8 2024-02-01T10:50:12.822 DEBUG:boto:Data: 2024-02-01T10:50:12.822 DEBUG:boto:Headers: {} 2024-02-01T10:50:12.822 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:12.822 DEBUG:boto:Port: 8000 2024-02-01T10:50:12.822 DEBUG:boto:Params: {} 2024-02-01T10:50:12.822 DEBUG:boto:Token: None 2024-02-01T10:50:12.822 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:12 GMT /vclfcn-64/obj8 2024-02-01T10:50:12.822 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:eOsZ3BABZKSYMRHz/OWcmZUc3zA= 2024-02-01T10:50:12.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:eOsZ3BABZKSYMRHz/OWcmZUc3zA=', 'Content-Length': '0'} 2024-02-01T10:50:12.825 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005436b6e9d76bfa9e-0065bb7764-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:12.868 DEBUG:boto:path=/vclfcn-64/obj8 2024-02-01T10:50:12.868 DEBUG:boto:auth_path=/vclfcn-64/obj8 2024-02-01T10:50:12.868 DEBUG:boto:Method: GET 2024-02-01T10:50:12.869 DEBUG:boto:Path: /vclfcn-64/obj8 2024-02-01T10:50:12.869 DEBUG:boto:Data: 2024-02-01T10:50:12.869 DEBUG:boto:Headers: {} 2024-02-01T10:50:12.869 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:12.869 DEBUG:boto:Port: 8000 2024-02-01T10:50:12.869 DEBUG:boto:Params: {} 2024-02-01T10:50:12.869 DEBUG:boto:Token: None 2024-02-01T10:50:12.869 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:12 GMT /vclfcn-64/obj8 2024-02-01T10:50:12.869 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:eOsZ3BABZKSYMRHz/OWcmZUc3zA= 2024-02-01T10:50:12.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:eOsZ3BABZKSYMRHz/OWcmZUc3zA=', 'Content-Length': '0'} 2024-02-01T10:50:12.873 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b9cd5e6c4cb19ca3-0065bb7764-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:12.920 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:12.920 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:12.920 DEBUG:boto:path=/vclfcn-64/?prefix=obj8 2024-02-01T10:50:12.921 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj8 2024-02-01T10:50:12.921 DEBUG:boto:Method: GET 2024-02-01T10:50:12.921 DEBUG:boto:Path: /vclfcn-64/?prefix=obj8 2024-02-01T10:50:12.921 DEBUG:boto:Data: 2024-02-01T10:50:12.921 DEBUG:boto:Headers: {} 2024-02-01T10:50:12.921 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:12.921 DEBUG:boto:Port: 8000 2024-02-01T10:50:12.921 DEBUG:boto:Params: {} 2024-02-01T10:50:12.921 DEBUG:boto:Token: None 2024-02-01T10:50:12.921 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:12 GMT /vclfcn-64/ 2024-02-01T10:50:12.922 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:qziLNDSFHlLlEoTowv8AgejDfiA= 2024-02-01T10:50:12.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:qziLNDSFHlLlEoTowv8AgejDfiA=', 'Content-Length': '0'} 2024-02-01T10:50:12.926 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f885f6c86405c083-0065bb7764-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:12.968 DEBUG:boto:b'vclfcn-64obj81000falseobj82024-02-01T10:49:22.519Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:12.969 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:12.969 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:12.969 DEBUG:boto:path=/vclfcn-64/?prefix=obj8 2024-02-01T10:50:12.969 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj8 2024-02-01T10:50:12.969 DEBUG:boto:Method: GET 2024-02-01T10:50:12.970 DEBUG:boto:Path: /vclfcn-64/?prefix=obj8 2024-02-01T10:50:12.970 DEBUG:boto:Data: 2024-02-01T10:50:12.970 DEBUG:boto:Headers: {} 2024-02-01T10:50:12.970 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:12.970 DEBUG:boto:Port: 8000 2024-02-01T10:50:12.970 DEBUG:boto:Params: {} 2024-02-01T10:50:12.971 DEBUG:boto:Token: None 2024-02-01T10:50:12.971 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:12 GMT /vclfcn-64/ 2024-02-01T10:50:12.971 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:qziLNDSFHlLlEoTowv8AgejDfiA= 2024-02-01T10:50:12.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:12 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:qziLNDSFHlLlEoTowv8AgejDfiA=', 'Content-Length': '0'} 2024-02-01T10:50:12.976 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a90c2ea91968b270-0065bb7764-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:12 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:13.020 DEBUG:boto:b'vclfcn-64obj81000falseobj82024-02-01T10:49:22.519Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:13.021 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-64 zones={test-zone2, test-zone1} 2024-02-01T10:50:13.021 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-64 zones={test-zone3, test-zone1} 2024-02-01T10:50:13.021 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:13.021 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:13.021 DEBUG:boto:Method: HEAD 2024-02-01T10:50:13.021 DEBUG:boto:Path: /vclfcn-64/ 2024-02-01T10:50:13.021 DEBUG:boto:Data: 2024-02-01T10:50:13.021 DEBUG:boto:Headers: {} 2024-02-01T10:50:13.021 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:13.021 DEBUG:boto:Port: 8001 2024-02-01T10:50:13.022 DEBUG:boto:Params: {} 2024-02-01T10:50:13.022 DEBUG:boto:Token: None 2024-02-01T10:50:13.022 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:13 GMT /vclfcn-64/ 2024-02-01T10:50:13.022 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZKUxb+UrUlvkF50cC0/ziRvkeak= 2024-02-01T10:50:13.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZKUxb+UrUlvkF50cC0/ziRvkeak=', 'Content-Length': '0'} 2024-02-01T10:50:13.026 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000883b41fb33e6dad5-0065bb7765-5762-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:50:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:13.026 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:13.026 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:13.027 DEBUG:boto:Method: HEAD 2024-02-01T10:50:13.027 DEBUG:boto:Path: /vclfcn-64/ 2024-02-01T10:50:13.027 DEBUG:boto:Data: 2024-02-01T10:50:13.027 DEBUG:boto:Headers: {} 2024-02-01T10:50:13.027 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:13.027 DEBUG:boto:Port: 8000 2024-02-01T10:50:13.027 DEBUG:boto:Params: {} 2024-02-01T10:50:13.027 DEBUG:boto:Token: None 2024-02-01T10:50:13.028 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:13 GMT /vclfcn-64/ 2024-02-01T10:50:13.028 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZKUxb+UrUlvkF50cC0/ziRvkeak= 2024-02-01T10:50:13.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZKUxb+UrUlvkF50cC0/ziRvkeak=', 'Content-Length': '0'} 2024-02-01T10:50:13.032 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007a5341d4f6c2cc49-0065bb7765-5765-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:50:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:13.032 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:50:13.032 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:13.032 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:13.033 DEBUG:boto:path=/vclfcn-64/?versions&max-keys=999 2024-02-01T10:50:13.033 DEBUG:boto:auth_path=/vclfcn-64/?versions&max-keys=999 2024-02-01T10:50:13.033 DEBUG:boto:Method: GET 2024-02-01T10:50:13.033 DEBUG:boto:Path: /vclfcn-64/?versions&max-keys=999 2024-02-01T10:50:13.033 DEBUG:boto:Data: 2024-02-01T10:50:13.033 DEBUG:boto:Headers: {} 2024-02-01T10:50:13.034 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:13.035 DEBUG:boto:Port: 8001 2024-02-01T10:50:13.035 DEBUG:boto:Params: {} 2024-02-01T10:50:13.035 DEBUG:boto:Token: None 2024-02-01T10:50:13.035 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:13 GMT /vclfcn-64/?versions 2024-02-01T10:50:13.035 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GZCoiGhX6cYqnsN6PDGxpT0uZBg= 2024-02-01T10:50:13.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GZCoiGhX6cYqnsN6PDGxpT0uZBg=', 'Content-Length': '0'} 2024-02-01T10:50:13.039 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000254ad3f492180bb9-0065bb7765-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:13.040 DEBUG:boto:b'vclfcn-64999falseobj1nulltrue2024-02-01T10:48:26.595Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:48:26.615Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:48:26.634Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:48:26.689Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-02-01T10:49:22.402Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-02-01T10:49:22.455Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-02-01T10:49:22.471Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-02-01T10:49:22.519Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:13.040 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-02-01T10:50:13.041 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-02-01T10:50:13.041 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-02-01T10:50:13.041 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-02-01T10:50:13.041 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-02-01T10:50:13.041 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-02-01T10:50:13.041 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-02-01T10:50:13.041 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-02-01T10:50:13.041 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:50:13.041 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:13.041 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:13.042 DEBUG:boto:path=/vclfcn-64/?versions&max-keys=999 2024-02-01T10:50:13.042 DEBUG:boto:auth_path=/vclfcn-64/?versions&max-keys=999 2024-02-01T10:50:13.042 DEBUG:boto:Method: GET 2024-02-01T10:50:13.042 DEBUG:boto:Path: /vclfcn-64/?versions&max-keys=999 2024-02-01T10:50:13.042 DEBUG:boto:Data: 2024-02-01T10:50:13.042 DEBUG:boto:Headers: {} 2024-02-01T10:50:13.042 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:13.042 DEBUG:boto:Port: 8000 2024-02-01T10:50:13.042 DEBUG:boto:Params: {} 2024-02-01T10:50:13.042 DEBUG:boto:Token: None 2024-02-01T10:50:13.043 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:13 GMT /vclfcn-64/?versions 2024-02-01T10:50:13.043 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GZCoiGhX6cYqnsN6PDGxpT0uZBg= 2024-02-01T10:50:13.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GZCoiGhX6cYqnsN6PDGxpT0uZBg=', 'Content-Length': '0'} 2024-02-01T10:50:13.047 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000536411a1cff45af6-0065bb7765-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:13.048 DEBUG:boto:b'vclfcn-64999falseobj1nulltrue2024-02-01T10:48:26.595Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:48:26.615Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:48:26.634Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:48:26.689Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-02-01T10:49:22.402Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-02-01T10:49:22.455Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-02-01T10:49:22.471Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-02-01T10:49:22.519Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:13.048 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-02-01T10:50:13.048 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-02-01T10:50:13.048 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-02-01T10:50:13.049 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-02-01T10:50:13.049 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-02-01T10:50:13.049 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-02-01T10:50:13.049 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-02-01T10:50:13.049 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-02-01T10:50:13.049 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:13.049 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:13.049 DEBUG:boto:path=/vclfcn-64/?versions&max-keys=999 2024-02-01T10:50:13.050 DEBUG:boto:auth_path=/vclfcn-64/?versions&max-keys=999 2024-02-01T10:50:13.050 DEBUG:boto:Method: GET 2024-02-01T10:50:13.050 DEBUG:boto:Path: /vclfcn-64/?versions&max-keys=999 2024-02-01T10:50:13.050 DEBUG:boto:Data: 2024-02-01T10:50:13.050 DEBUG:boto:Headers: {} 2024-02-01T10:50:13.050 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:13.050 DEBUG:boto:Port: 8001 2024-02-01T10:50:13.050 DEBUG:boto:Params: {} 2024-02-01T10:50:13.050 DEBUG:boto:Token: None 2024-02-01T10:50:13.051 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:13 GMT /vclfcn-64/?versions 2024-02-01T10:50:13.051 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GZCoiGhX6cYqnsN6PDGxpT0uZBg= 2024-02-01T10:50:13.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GZCoiGhX6cYqnsN6PDGxpT0uZBg=', 'Content-Length': '0'} 2024-02-01T10:50:13.055 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bbb56b6e186d672c-0065bb7765-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:13.056 DEBUG:boto:b'vclfcn-64999falseobj1nulltrue2024-02-01T10:48:26.595Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:48:26.615Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:48:26.634Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:48:26.689Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-02-01T10:49:22.402Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-02-01T10:49:22.455Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-02-01T10:49:22.471Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-02-01T10:49:22.519Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:13.056 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:13.057 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:13.057 DEBUG:boto:path=/vclfcn-64/?versions&max-keys=999 2024-02-01T10:50:13.057 DEBUG:boto:auth_path=/vclfcn-64/?versions&max-keys=999 2024-02-01T10:50:13.057 DEBUG:boto:Method: GET 2024-02-01T10:50:13.057 DEBUG:boto:Path: /vclfcn-64/?versions&max-keys=999 2024-02-01T10:50:13.057 DEBUG:boto:Data: 2024-02-01T10:50:13.057 DEBUG:boto:Headers: {} 2024-02-01T10:50:13.057 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:13.057 DEBUG:boto:Port: 8000 2024-02-01T10:50:13.057 DEBUG:boto:Params: {} 2024-02-01T10:50:13.058 DEBUG:boto:Token: None 2024-02-01T10:50:13.058 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:13 GMT /vclfcn-64/?versions 2024-02-01T10:50:13.058 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:GZCoiGhX6cYqnsN6PDGxpT0uZBg= 2024-02-01T10:50:13.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:GZCoiGhX6cYqnsN6PDGxpT0uZBg=', 'Content-Length': '0'} 2024-02-01T10:50:13.063 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad83a603860486ae-0065bb7765-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:13.063 DEBUG:boto:b'vclfcn-64999falseobj1nulltrue2024-02-01T10:48:26.595Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:48:26.615Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:48:26.634Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:48:26.689Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-02-01T10:49:22.402Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-02-01T10:49:22.455Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-02-01T10:49:22.471Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-02-01T10:49:22.519Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:13.064 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:50:13.064 DEBUG:boto:path=/vclfcn-64/obj1 2024-02-01T10:50:13.064 DEBUG:boto:auth_path=/vclfcn-64/obj1 2024-02-01T10:50:13.064 DEBUG:boto:path=/vclfcn-64/obj1?versionId=null 2024-02-01T10:50:13.064 DEBUG:boto:auth_path=/vclfcn-64/obj1?versionId=null 2024-02-01T10:50:13.064 DEBUG:boto:Method: GET 2024-02-01T10:50:13.064 DEBUG:boto:Path: /vclfcn-64/obj1?versionId=null 2024-02-01T10:50:13.065 DEBUG:boto:Data: 2024-02-01T10:50:13.065 DEBUG:boto:Headers: {} 2024-02-01T10:50:13.065 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:13.065 DEBUG:boto:Port: 8001 2024-02-01T10:50:13.065 DEBUG:boto:Params: {} 2024-02-01T10:50:13.065 DEBUG:boto:Token: None 2024-02-01T10:50:13.065 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:13 GMT /vclfcn-64/obj1?versionId=null 2024-02-01T10:50:13.065 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FNeIjGrarumvlJ/P/s9brddKTDo= 2024-02-01T10:50:13.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FNeIjGrarumvlJ/P/s9brddKTDo=', 'Content-Length': '0'} 2024-02-01T10:50:13.068 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004deb7a09858579c4-0065bb7765-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:13.112 DEBUG:boto:path=/vclfcn-64/obj1 2024-02-01T10:50:13.112 DEBUG:boto:auth_path=/vclfcn-64/obj1 2024-02-01T10:50:13.112 DEBUG:boto:path=/vclfcn-64/obj1?versionId=null 2024-02-01T10:50:13.112 DEBUG:boto:auth_path=/vclfcn-64/obj1?versionId=null 2024-02-01T10:50:13.112 DEBUG:boto:Method: GET 2024-02-01T10:50:13.113 DEBUG:boto:Path: /vclfcn-64/obj1?versionId=null 2024-02-01T10:50:13.113 DEBUG:boto:Data: 2024-02-01T10:50:13.113 DEBUG:boto:Headers: {} 2024-02-01T10:50:13.113 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:13.113 DEBUG:boto:Port: 8000 2024-02-01T10:50:13.113 DEBUG:boto:Params: {} 2024-02-01T10:50:13.113 DEBUG:boto:Token: None 2024-02-01T10:50:13.113 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:13 GMT /vclfcn-64/obj1?versionId=null 2024-02-01T10:50:13.113 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FNeIjGrarumvlJ/P/s9brddKTDo= 2024-02-01T10:50:13.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FNeIjGrarumvlJ/P/s9brddKTDo=', 'Content-Length': '0'} 2024-02-01T10:50:13.116 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f9fddee4ac756fc2-0065bb7765-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:13.117 DEBUG:boto:path=/vclfcn-64/obj1 2024-02-01T10:50:13.117 DEBUG:boto:auth_path=/vclfcn-64/obj1 2024-02-01T10:50:13.117 DEBUG:boto:path=/vclfcn-64/obj1?versionId=null 2024-02-01T10:50:13.117 DEBUG:boto:auth_path=/vclfcn-64/obj1?versionId=null 2024-02-01T10:50:13.117 DEBUG:boto:Method: HEAD 2024-02-01T10:50:13.117 DEBUG:boto:Path: /vclfcn-64/obj1?versionId=null 2024-02-01T10:50:13.117 DEBUG:boto:Data: 2024-02-01T10:50:13.117 DEBUG:boto:Headers: {} 2024-02-01T10:50:13.117 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:13.117 DEBUG:boto:Port: 8001 2024-02-01T10:50:13.118 DEBUG:boto:Params: {} 2024-02-01T10:50:13.118 DEBUG:boto:Token: None 2024-02-01T10:50:13.118 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:13 GMT /vclfcn-64/obj1?versionId=null 2024-02-01T10:50:13.118 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:p7DldI+D5sp7Pejb82IsDGToKsU= 2024-02-01T10:50:13.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:p7DldI+D5sp7Pejb82IsDGToKsU=', 'Content-Length': '0'} 2024-02-01T10:50:13.121 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ea8bc81153cc4aa1-0065bb7765-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:13.121 DEBUG:boto:path=/vclfcn-64/obj1 2024-02-01T10:50:13.121 DEBUG:boto:auth_path=/vclfcn-64/obj1 2024-02-01T10:50:13.121 DEBUG:boto:path=/vclfcn-64/obj1?versionId=null 2024-02-01T10:50:13.121 DEBUG:boto:auth_path=/vclfcn-64/obj1?versionId=null 2024-02-01T10:50:13.121 DEBUG:boto:Method: HEAD 2024-02-01T10:50:13.121 DEBUG:boto:Path: /vclfcn-64/obj1?versionId=null 2024-02-01T10:50:13.122 DEBUG:boto:Data: 2024-02-01T10:50:13.122 DEBUG:boto:Headers: {} 2024-02-01T10:50:13.122 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:13.122 DEBUG:boto:Port: 8000 2024-02-01T10:50:13.122 DEBUG:boto:Params: {} 2024-02-01T10:50:13.122 DEBUG:boto:Token: None 2024-02-01T10:50:13.122 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:13 GMT /vclfcn-64/obj1?versionId=null 2024-02-01T10:50:13.122 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:p7DldI+D5sp7Pejb82IsDGToKsU= 2024-02-01T10:50:13.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:p7DldI+D5sp7Pejb82IsDGToKsU=', 'Content-Length': '0'} 2024-02-01T10:50:13.125 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006095f81d29523ab2-0065bb7765-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:13.126 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:50:13.126 DEBUG:boto:path=/vclfcn-64/obj1 2024-02-01T10:50:13.126 DEBUG:boto:auth_path=/vclfcn-64/obj1 2024-02-01T10:50:13.126 DEBUG:boto:path=/vclfcn-64/obj1?versionId=null 2024-02-01T10:50:13.126 DEBUG:boto:auth_path=/vclfcn-64/obj1?versionId=null 2024-02-01T10:50:13.126 DEBUG:boto:Method: GET 2024-02-01T10:50:13.126 DEBUG:boto:Path: /vclfcn-64/obj1?versionId=null 2024-02-01T10:50:13.126 DEBUG:boto:Data: 2024-02-01T10:50:13.126 DEBUG:boto:Headers: {} 2024-02-01T10:50:13.126 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:13.126 DEBUG:boto:Port: 8001 2024-02-01T10:50:13.127 DEBUG:boto:Params: {} 2024-02-01T10:50:13.127 DEBUG:boto:Token: None 2024-02-01T10:50:13.127 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:13 GMT /vclfcn-64/obj1?versionId=null 2024-02-01T10:50:13.127 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FNeIjGrarumvlJ/P/s9brddKTDo= 2024-02-01T10:50:13.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FNeIjGrarumvlJ/P/s9brddKTDo=', 'Content-Length': '0'} 2024-02-01T10:50:13.130 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ea7dce77a90897b4-0065bb7765-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:13.130 DEBUG:boto:path=/vclfcn-64/obj1 2024-02-01T10:50:13.130 DEBUG:boto:auth_path=/vclfcn-64/obj1 2024-02-01T10:50:13.130 DEBUG:boto:path=/vclfcn-64/obj1?versionId=null 2024-02-01T10:50:13.130 DEBUG:boto:auth_path=/vclfcn-64/obj1?versionId=null 2024-02-01T10:50:13.130 DEBUG:boto:Method: GET 2024-02-01T10:50:13.131 DEBUG:boto:Path: /vclfcn-64/obj1?versionId=null 2024-02-01T10:50:13.131 DEBUG:boto:Data: 2024-02-01T10:50:13.131 DEBUG:boto:Headers: {} 2024-02-01T10:50:13.131 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:13.131 DEBUG:boto:Port: 8000 2024-02-01T10:50:13.131 DEBUG:boto:Params: {} 2024-02-01T10:50:13.131 DEBUG:boto:Token: None 2024-02-01T10:50:13.131 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:13 GMT /vclfcn-64/obj1?versionId=null 2024-02-01T10:50:13.131 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:FNeIjGrarumvlJ/P/s9brddKTDo= 2024-02-01T10:50:13.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:FNeIjGrarumvlJ/P/s9brddKTDo=', 'Content-Length': '0'} 2024-02-01T10:50:13.134 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a7f55293162dd0f6-0065bb7765-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:13.180 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:13.180 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:13.180 DEBUG:boto:path=/vclfcn-64/?prefix=obj1 2024-02-01T10:50:13.181 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj1 2024-02-01T10:50:13.181 DEBUG:boto:Method: GET 2024-02-01T10:50:13.181 DEBUG:boto:Path: /vclfcn-64/?prefix=obj1 2024-02-01T10:50:13.181 DEBUG:boto:Data: 2024-02-01T10:50:13.181 DEBUG:boto:Headers: {} 2024-02-01T10:50:13.181 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:13.181 DEBUG:boto:Port: 8001 2024-02-01T10:50:13.181 DEBUG:boto:Params: {} 2024-02-01T10:50:13.181 DEBUG:boto:Token: None 2024-02-01T10:50:13.182 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:13 GMT /vclfcn-64/ 2024-02-01T10:50:13.182 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:U845z3jmrl2BY7DUvSM+mO7jCGI= 2024-02-01T10:50:13.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:U845z3jmrl2BY7DUvSM+mO7jCGI=', 'Content-Length': '0'} 2024-02-01T10:50:13.186 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000073d9b25963331972-0065bb7765-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:13.186 DEBUG:boto:b'vclfcn-64obj11000falseobj12024-02-01T10:48:26.595Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:13.187 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:13.187 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:13.187 DEBUG:boto:path=/vclfcn-64/?prefix=obj1 2024-02-01T10:50:13.187 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj1 2024-02-01T10:50:13.187 DEBUG:boto:Method: GET 2024-02-01T10:50:13.187 DEBUG:boto:Path: /vclfcn-64/?prefix=obj1 2024-02-01T10:50:13.188 DEBUG:boto:Data: 2024-02-01T10:50:13.188 DEBUG:boto:Headers: {} 2024-02-01T10:50:13.188 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:13.188 DEBUG:boto:Port: 8000 2024-02-01T10:50:13.188 DEBUG:boto:Params: {} 2024-02-01T10:50:13.188 DEBUG:boto:Token: None 2024-02-01T10:50:13.188 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:13 GMT /vclfcn-64/ 2024-02-01T10:50:13.188 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:U845z3jmrl2BY7DUvSM+mO7jCGI= 2024-02-01T10:50:13.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:U845z3jmrl2BY7DUvSM+mO7jCGI=', 'Content-Length': '0'} 2024-02-01T10:50:13.193 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e60df34ba3ac042-0065bb7765-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:13.236 DEBUG:boto:b'vclfcn-64obj11000falseobj12024-02-01T10:48:26.595Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:13.237 DEBUG:boto:path=/vclfcn-64/obj1 2024-02-01T10:50:13.237 DEBUG:boto:auth_path=/vclfcn-64/obj1 2024-02-01T10:50:13.237 DEBUG:boto:Method: HEAD 2024-02-01T10:50:13.237 DEBUG:boto:Path: /vclfcn-64/obj1 2024-02-01T10:50:13.237 DEBUG:boto:Data: 2024-02-01T10:50:13.237 DEBUG:boto:Headers: {} 2024-02-01T10:50:13.237 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:13.237 DEBUG:boto:Port: 8001 2024-02-01T10:50:13.237 DEBUG:boto:Params: {} 2024-02-01T10:50:13.238 DEBUG:boto:Token: None 2024-02-01T10:50:13.238 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:13 GMT /vclfcn-64/obj1 2024-02-01T10:50:13.238 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NlQvpE2tyFf9TZJQDgPBKCu4fb4= 2024-02-01T10:50:13.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NlQvpE2tyFf9TZJQDgPBKCu4fb4=', 'Content-Length': '0'} 2024-02-01T10:50:13.241 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000061b8f5786515092a-0065bb7765-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:13.242 DEBUG:boto:path=/vclfcn-64/obj1 2024-02-01T10:50:13.242 DEBUG:boto:auth_path=/vclfcn-64/obj1 2024-02-01T10:50:13.242 DEBUG:boto:Method: HEAD 2024-02-01T10:50:13.242 DEBUG:boto:Path: /vclfcn-64/obj1 2024-02-01T10:50:13.242 DEBUG:boto:Data: 2024-02-01T10:50:13.242 DEBUG:boto:Headers: {} 2024-02-01T10:50:13.242 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:13.242 DEBUG:boto:Port: 8000 2024-02-01T10:50:13.242 DEBUG:boto:Params: {} 2024-02-01T10:50:13.243 DEBUG:boto:Token: None 2024-02-01T10:50:13.243 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:13 GMT /vclfcn-64/obj1 2024-02-01T10:50:13.243 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:NlQvpE2tyFf9TZJQDgPBKCu4fb4= 2024-02-01T10:50:13.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:NlQvpE2tyFf9TZJQDgPBKCu4fb4=', 'Content-Length': '0'} 2024-02-01T10:50:13.245 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004ef5355e4d322067-0065bb7765-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:13.246 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:50:13.246 DEBUG:boto:path=/vclfcn-64/obj1 2024-02-01T10:50:13.246 DEBUG:boto:auth_path=/vclfcn-64/obj1 2024-02-01T10:50:13.246 DEBUG:boto:Method: GET 2024-02-01T10:50:13.246 DEBUG:boto:Path: /vclfcn-64/obj1 2024-02-01T10:50:13.246 DEBUG:boto:Data: 2024-02-01T10:50:13.246 DEBUG:boto:Headers: {} 2024-02-01T10:50:13.246 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:13.247 DEBUG:boto:Port: 8001 2024-02-01T10:50:13.247 DEBUG:boto:Params: {} 2024-02-01T10:50:13.247 DEBUG:boto:Token: None 2024-02-01T10:50:13.247 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:13 GMT /vclfcn-64/obj1 2024-02-01T10:50:13.247 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lwMVL1X2H84M+iBc0L6erzU03DE= 2024-02-01T10:50:13.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lwMVL1X2H84M+iBc0L6erzU03DE=', 'Content-Length': '0'} 2024-02-01T10:50:13.250 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005113a5d97d8d9721-0065bb7765-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:13.292 DEBUG:boto:path=/vclfcn-64/obj1 2024-02-01T10:50:13.292 DEBUG:boto:auth_path=/vclfcn-64/obj1 2024-02-01T10:50:13.292 DEBUG:boto:Method: GET 2024-02-01T10:50:13.293 DEBUG:boto:Path: /vclfcn-64/obj1 2024-02-01T10:50:13.293 DEBUG:boto:Data: 2024-02-01T10:50:13.293 DEBUG:boto:Headers: {} 2024-02-01T10:50:13.293 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:13.293 DEBUG:boto:Port: 8000 2024-02-01T10:50:13.293 DEBUG:boto:Params: {} 2024-02-01T10:50:13.293 DEBUG:boto:Token: None 2024-02-01T10:50:13.294 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:13 GMT /vclfcn-64/obj1 2024-02-01T10:50:13.294 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lwMVL1X2H84M+iBc0L6erzU03DE= 2024-02-01T10:50:13.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lwMVL1X2H84M+iBc0L6erzU03DE=', 'Content-Length': '0'} 2024-02-01T10:50:13.297 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d841eb183e5eece5-0065bb7765-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:13.340 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:13.340 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:13.341 DEBUG:boto:path=/vclfcn-64/?prefix=obj1 2024-02-01T10:50:13.341 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj1 2024-02-01T10:50:13.341 DEBUG:boto:Method: GET 2024-02-01T10:50:13.342 DEBUG:boto:Path: /vclfcn-64/?prefix=obj1 2024-02-01T10:50:13.342 DEBUG:boto:Data: 2024-02-01T10:50:13.342 DEBUG:boto:Headers: {} 2024-02-01T10:50:13.342 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:13.342 DEBUG:boto:Port: 8001 2024-02-01T10:50:13.342 DEBUG:boto:Params: {} 2024-02-01T10:50:13.342 DEBUG:boto:Token: None 2024-02-01T10:50:13.343 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:13 GMT /vclfcn-64/ 2024-02-01T10:50:13.343 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:U845z3jmrl2BY7DUvSM+mO7jCGI= 2024-02-01T10:50:13.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:U845z3jmrl2BY7DUvSM+mO7jCGI=', 'Content-Length': '0'} 2024-02-01T10:50:13.347 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3f8ad46c69137e9-0065bb7765-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:13.347 DEBUG:boto:b'vclfcn-64obj11000falseobj12024-02-01T10:48:26.595Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:13.347 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:13.348 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:13.348 DEBUG:boto:path=/vclfcn-64/?prefix=obj1 2024-02-01T10:50:13.348 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj1 2024-02-01T10:50:13.348 DEBUG:boto:Method: GET 2024-02-01T10:50:13.348 DEBUG:boto:Path: /vclfcn-64/?prefix=obj1 2024-02-01T10:50:13.348 DEBUG:boto:Data: 2024-02-01T10:50:13.348 DEBUG:boto:Headers: {} 2024-02-01T10:50:13.348 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:13.348 DEBUG:boto:Port: 8000 2024-02-01T10:50:13.348 DEBUG:boto:Params: {} 2024-02-01T10:50:13.349 DEBUG:boto:Token: None 2024-02-01T10:50:13.349 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:13 GMT /vclfcn-64/ 2024-02-01T10:50:13.349 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:U845z3jmrl2BY7DUvSM+mO7jCGI= 2024-02-01T10:50:13.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:U845z3jmrl2BY7DUvSM+mO7jCGI=', 'Content-Length': '0'} 2024-02-01T10:50:13.353 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c65ef4ec6ecb2dd4-0065bb7765-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:13.353 DEBUG:boto:b'vclfcn-64obj11000falseobj12024-02-01T10:48:26.595Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:13.354 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:50:13.354 DEBUG:boto:path=/vclfcn-64/obj2 2024-02-01T10:50:13.354 DEBUG:boto:auth_path=/vclfcn-64/obj2 2024-02-01T10:50:13.354 DEBUG:boto:path=/vclfcn-64/obj2?versionId=null 2024-02-01T10:50:13.354 DEBUG:boto:auth_path=/vclfcn-64/obj2?versionId=null 2024-02-01T10:50:13.354 DEBUG:boto:Method: GET 2024-02-01T10:50:13.354 DEBUG:boto:Path: /vclfcn-64/obj2?versionId=null 2024-02-01T10:50:13.354 DEBUG:boto:Data: 2024-02-01T10:50:13.354 DEBUG:boto:Headers: {} 2024-02-01T10:50:13.355 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:13.355 DEBUG:boto:Port: 8001 2024-02-01T10:50:13.355 DEBUG:boto:Params: {} 2024-02-01T10:50:13.355 DEBUG:boto:Token: None 2024-02-01T10:50:13.355 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:13 GMT /vclfcn-64/obj2?versionId=null 2024-02-01T10:50:13.355 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Zuva5ky8QI7GZu5XVEJEMzfSeBA= 2024-02-01T10:50:13.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Zuva5ky8QI7GZu5XVEJEMzfSeBA=', 'Content-Length': '0'} 2024-02-01T10:50:13.358 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008bee346257abe70c-0065bb7765-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:13.358 DEBUG:boto:path=/vclfcn-64/obj2 2024-02-01T10:50:13.359 DEBUG:boto:auth_path=/vclfcn-64/obj2 2024-02-01T10:50:13.359 DEBUG:boto:path=/vclfcn-64/obj2?versionId=null 2024-02-01T10:50:13.359 DEBUG:boto:auth_path=/vclfcn-64/obj2?versionId=null 2024-02-01T10:50:13.359 DEBUG:boto:Method: GET 2024-02-01T10:50:13.359 DEBUG:boto:Path: /vclfcn-64/obj2?versionId=null 2024-02-01T10:50:13.359 DEBUG:boto:Data: 2024-02-01T10:50:13.359 DEBUG:boto:Headers: {} 2024-02-01T10:50:13.359 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:13.359 DEBUG:boto:Port: 8000 2024-02-01T10:50:13.360 DEBUG:boto:Params: {} 2024-02-01T10:50:13.360 DEBUG:boto:Token: None 2024-02-01T10:50:13.360 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:13 GMT /vclfcn-64/obj2?versionId=null 2024-02-01T10:50:13.360 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Zuva5ky8QI7GZu5XVEJEMzfSeBA= 2024-02-01T10:50:13.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Zuva5ky8QI7GZu5XVEJEMzfSeBA=', 'Content-Length': '0'} 2024-02-01T10:50:13.363 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002695b1ec4151dd17-0065bb7765-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:13.408 DEBUG:boto:path=/vclfcn-64/obj2 2024-02-01T10:50:13.408 DEBUG:boto:auth_path=/vclfcn-64/obj2 2024-02-01T10:50:13.408 DEBUG:boto:path=/vclfcn-64/obj2?versionId=null 2024-02-01T10:50:13.408 DEBUG:boto:auth_path=/vclfcn-64/obj2?versionId=null 2024-02-01T10:50:13.409 DEBUG:boto:Method: HEAD 2024-02-01T10:50:13.409 DEBUG:boto:Path: /vclfcn-64/obj2?versionId=null 2024-02-01T10:50:13.409 DEBUG:boto:Data: 2024-02-01T10:50:13.409 DEBUG:boto:Headers: {} 2024-02-01T10:50:13.409 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:13.409 DEBUG:boto:Port: 8001 2024-02-01T10:50:13.409 DEBUG:boto:Params: {} 2024-02-01T10:50:13.409 DEBUG:boto:Token: None 2024-02-01T10:50:13.409 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:13 GMT /vclfcn-64/obj2?versionId=null 2024-02-01T10:50:13.409 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mfw5jXvcQ4OR1KuRfre22SzvamA= 2024-02-01T10:50:13.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mfw5jXvcQ4OR1KuRfre22SzvamA=', 'Content-Length': '0'} 2024-02-01T10:50:13.412 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f5866d6142d850aa-0065bb7765-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:13.413 DEBUG:boto:path=/vclfcn-64/obj2 2024-02-01T10:50:13.413 DEBUG:boto:auth_path=/vclfcn-64/obj2 2024-02-01T10:50:13.413 DEBUG:boto:path=/vclfcn-64/obj2?versionId=null 2024-02-01T10:50:13.413 DEBUG:boto:auth_path=/vclfcn-64/obj2?versionId=null 2024-02-01T10:50:13.413 DEBUG:boto:Method: HEAD 2024-02-01T10:50:13.413 DEBUG:boto:Path: /vclfcn-64/obj2?versionId=null 2024-02-01T10:50:13.413 DEBUG:boto:Data: 2024-02-01T10:50:13.413 DEBUG:boto:Headers: {} 2024-02-01T10:50:13.413 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:13.414 DEBUG:boto:Port: 8000 2024-02-01T10:50:13.414 DEBUG:boto:Params: {} 2024-02-01T10:50:13.414 DEBUG:boto:Token: None 2024-02-01T10:50:13.414 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:13 GMT /vclfcn-64/obj2?versionId=null 2024-02-01T10:50:13.414 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mfw5jXvcQ4OR1KuRfre22SzvamA= 2024-02-01T10:50:13.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mfw5jXvcQ4OR1KuRfre22SzvamA=', 'Content-Length': '0'} 2024-02-01T10:50:13.417 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006a7f96ec131e19b3-0065bb7765-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:13.417 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:50:13.417 DEBUG:boto:path=/vclfcn-64/obj2 2024-02-01T10:50:13.418 DEBUG:boto:auth_path=/vclfcn-64/obj2 2024-02-01T10:50:13.418 DEBUG:boto:path=/vclfcn-64/obj2?versionId=null 2024-02-01T10:50:13.418 DEBUG:boto:auth_path=/vclfcn-64/obj2?versionId=null 2024-02-01T10:50:13.418 DEBUG:boto:Method: GET 2024-02-01T10:50:13.418 DEBUG:boto:Path: /vclfcn-64/obj2?versionId=null 2024-02-01T10:50:13.418 DEBUG:boto:Data: 2024-02-01T10:50:13.418 DEBUG:boto:Headers: {} 2024-02-01T10:50:13.418 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:13.418 DEBUG:boto:Port: 8001 2024-02-01T10:50:13.418 DEBUG:boto:Params: {} 2024-02-01T10:50:13.418 DEBUG:boto:Token: None 2024-02-01T10:50:13.419 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:13 GMT /vclfcn-64/obj2?versionId=null 2024-02-01T10:50:13.419 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Zuva5ky8QI7GZu5XVEJEMzfSeBA= 2024-02-01T10:50:13.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Zuva5ky8QI7GZu5XVEJEMzfSeBA=', 'Content-Length': '0'} 2024-02-01T10:50:13.421 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ac895ed92b4133e4-0065bb7765-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:13.422 DEBUG:boto:path=/vclfcn-64/obj2 2024-02-01T10:50:13.422 DEBUG:boto:auth_path=/vclfcn-64/obj2 2024-02-01T10:50:13.422 DEBUG:boto:path=/vclfcn-64/obj2?versionId=null 2024-02-01T10:50:13.422 DEBUG:boto:auth_path=/vclfcn-64/obj2?versionId=null 2024-02-01T10:50:13.422 DEBUG:boto:Method: GET 2024-02-01T10:50:13.422 DEBUG:boto:Path: /vclfcn-64/obj2?versionId=null 2024-02-01T10:50:13.422 DEBUG:boto:Data: 2024-02-01T10:50:13.422 DEBUG:boto:Headers: {} 2024-02-01T10:50:13.422 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:13.422 DEBUG:boto:Port: 8000 2024-02-01T10:50:13.423 DEBUG:boto:Params: {} 2024-02-01T10:50:13.423 DEBUG:boto:Token: None 2024-02-01T10:50:13.423 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:13 GMT /vclfcn-64/obj2?versionId=null 2024-02-01T10:50:13.423 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Zuva5ky8QI7GZu5XVEJEMzfSeBA= 2024-02-01T10:50:13.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Zuva5ky8QI7GZu5XVEJEMzfSeBA=', 'Content-Length': '0'} 2024-02-01T10:50:13.426 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cd820b80737d77ee-0065bb7765-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:13.468 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:13.468 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:13.468 DEBUG:boto:path=/vclfcn-64/?prefix=obj2 2024-02-01T10:50:13.469 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj2 2024-02-01T10:50:13.469 DEBUG:boto:Method: GET 2024-02-01T10:50:13.469 DEBUG:boto:Path: /vclfcn-64/?prefix=obj2 2024-02-01T10:50:13.469 DEBUG:boto:Data: 2024-02-01T10:50:13.469 DEBUG:boto:Headers: {} 2024-02-01T10:50:13.469 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:13.469 DEBUG:boto:Port: 8001 2024-02-01T10:50:13.469 DEBUG:boto:Params: {} 2024-02-01T10:50:13.469 DEBUG:boto:Token: None 2024-02-01T10:50:13.469 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:13 GMT /vclfcn-64/ 2024-02-01T10:50:13.470 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:U845z3jmrl2BY7DUvSM+mO7jCGI= 2024-02-01T10:50:13.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:U845z3jmrl2BY7DUvSM+mO7jCGI=', 'Content-Length': '0'} 2024-02-01T10:50:13.474 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e77d5fb8123ddb0-0065bb7765-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:13.474 DEBUG:boto:b'vclfcn-64obj21000falseobj22024-02-01T10:48:26.615Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:13.475 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:13.475 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:13.475 DEBUG:boto:path=/vclfcn-64/?prefix=obj2 2024-02-01T10:50:13.475 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj2 2024-02-01T10:50:13.475 DEBUG:boto:Method: GET 2024-02-01T10:50:13.475 DEBUG:boto:Path: /vclfcn-64/?prefix=obj2 2024-02-01T10:50:13.475 DEBUG:boto:Data: 2024-02-01T10:50:13.475 DEBUG:boto:Headers: {} 2024-02-01T10:50:13.475 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:13.476 DEBUG:boto:Port: 8000 2024-02-01T10:50:13.476 DEBUG:boto:Params: {} 2024-02-01T10:50:13.476 DEBUG:boto:Token: None 2024-02-01T10:50:13.476 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:13 GMT /vclfcn-64/ 2024-02-01T10:50:13.476 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:U845z3jmrl2BY7DUvSM+mO7jCGI= 2024-02-01T10:50:13.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:U845z3jmrl2BY7DUvSM+mO7jCGI=', 'Content-Length': '0'} 2024-02-01T10:50:13.481 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003be08464b2a6b4e9-0065bb7765-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:13.524 DEBUG:boto:b'vclfcn-64obj21000falseobj22024-02-01T10:48:26.615Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:13.525 DEBUG:boto:path=/vclfcn-64/obj2 2024-02-01T10:50:13.525 DEBUG:boto:auth_path=/vclfcn-64/obj2 2024-02-01T10:50:13.525 DEBUG:boto:Method: HEAD 2024-02-01T10:50:13.525 DEBUG:boto:Path: /vclfcn-64/obj2 2024-02-01T10:50:13.525 DEBUG:boto:Data: 2024-02-01T10:50:13.525 DEBUG:boto:Headers: {} 2024-02-01T10:50:13.525 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:13.525 DEBUG:boto:Port: 8001 2024-02-01T10:50:13.525 DEBUG:boto:Params: {} 2024-02-01T10:50:13.526 DEBUG:boto:Token: None 2024-02-01T10:50:13.526 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:13 GMT /vclfcn-64/obj2 2024-02-01T10:50:13.526 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:M2YLMdhTxS9sTrkNBkzfJt0SzlQ= 2024-02-01T10:50:13.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:M2YLMdhTxS9sTrkNBkzfJt0SzlQ=', 'Content-Length': '0'} 2024-02-01T10:50:13.529 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000662bb69b6715cad8-0065bb7765-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:13.529 DEBUG:boto:path=/vclfcn-64/obj2 2024-02-01T10:50:13.530 DEBUG:boto:auth_path=/vclfcn-64/obj2 2024-02-01T10:50:13.530 DEBUG:boto:Method: HEAD 2024-02-01T10:50:13.530 DEBUG:boto:Path: /vclfcn-64/obj2 2024-02-01T10:50:13.530 DEBUG:boto:Data: 2024-02-01T10:50:13.530 DEBUG:boto:Headers: {} 2024-02-01T10:50:13.530 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:13.530 DEBUG:boto:Port: 8000 2024-02-01T10:50:13.530 DEBUG:boto:Params: {} 2024-02-01T10:50:13.530 DEBUG:boto:Token: None 2024-02-01T10:50:13.530 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:13 GMT /vclfcn-64/obj2 2024-02-01T10:50:13.531 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:M2YLMdhTxS9sTrkNBkzfJt0SzlQ= 2024-02-01T10:50:13.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:M2YLMdhTxS9sTrkNBkzfJt0SzlQ=', 'Content-Length': '0'} 2024-02-01T10:50:13.534 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000027ded09d0b295093-0065bb7765-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:13.534 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:50:13.534 DEBUG:boto:path=/vclfcn-64/obj2 2024-02-01T10:50:13.534 DEBUG:boto:auth_path=/vclfcn-64/obj2 2024-02-01T10:50:13.534 DEBUG:boto:Method: GET 2024-02-01T10:50:13.535 DEBUG:boto:Path: /vclfcn-64/obj2 2024-02-01T10:50:13.535 DEBUG:boto:Data: 2024-02-01T10:50:13.535 DEBUG:boto:Headers: {} 2024-02-01T10:50:13.535 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:13.535 DEBUG:boto:Port: 8001 2024-02-01T10:50:13.535 DEBUG:boto:Params: {} 2024-02-01T10:50:13.535 DEBUG:boto:Token: None 2024-02-01T10:50:13.535 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:13 GMT /vclfcn-64/obj2 2024-02-01T10:50:13.535 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cXFIOwbJreO0dmMxHVo7SxbfBWU= 2024-02-01T10:50:13.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cXFIOwbJreO0dmMxHVo7SxbfBWU=', 'Content-Length': '0'} 2024-02-01T10:50:13.538 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000056647fd9667aa095-0065bb7765-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:13.580 DEBUG:boto:path=/vclfcn-64/obj2 2024-02-01T10:50:13.580 DEBUG:boto:auth_path=/vclfcn-64/obj2 2024-02-01T10:50:13.580 DEBUG:boto:Method: GET 2024-02-01T10:50:13.581 DEBUG:boto:Path: /vclfcn-64/obj2 2024-02-01T10:50:13.581 DEBUG:boto:Data: 2024-02-01T10:50:13.581 DEBUG:boto:Headers: {} 2024-02-01T10:50:13.581 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:13.581 DEBUG:boto:Port: 8000 2024-02-01T10:50:13.581 DEBUG:boto:Params: {} 2024-02-01T10:50:13.581 DEBUG:boto:Token: None 2024-02-01T10:50:13.581 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:13 GMT /vclfcn-64/obj2 2024-02-01T10:50:13.581 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:cXFIOwbJreO0dmMxHVo7SxbfBWU= 2024-02-01T10:50:13.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:cXFIOwbJreO0dmMxHVo7SxbfBWU=', 'Content-Length': '0'} 2024-02-01T10:50:13.584 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000325f6cf6e4589b5d-0065bb7765-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:13.628 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:13.628 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:13.628 DEBUG:boto:path=/vclfcn-64/?prefix=obj2 2024-02-01T10:50:13.628 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj2 2024-02-01T10:50:13.629 DEBUG:boto:Method: GET 2024-02-01T10:50:13.629 DEBUG:boto:Path: /vclfcn-64/?prefix=obj2 2024-02-01T10:50:13.629 DEBUG:boto:Data: 2024-02-01T10:50:13.629 DEBUG:boto:Headers: {} 2024-02-01T10:50:13.629 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:13.629 DEBUG:boto:Port: 8001 2024-02-01T10:50:13.629 DEBUG:boto:Params: {} 2024-02-01T10:50:13.629 DEBUG:boto:Token: None 2024-02-01T10:50:13.629 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:13 GMT /vclfcn-64/ 2024-02-01T10:50:13.629 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:U845z3jmrl2BY7DUvSM+mO7jCGI= 2024-02-01T10:50:13.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:U845z3jmrl2BY7DUvSM+mO7jCGI=', 'Content-Length': '0'} 2024-02-01T10:50:13.634 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d63e1401b248489-0065bb7765-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:13.634 DEBUG:boto:b'vclfcn-64obj21000falseobj22024-02-01T10:48:26.615Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:13.634 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:13.634 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:13.635 DEBUG:boto:path=/vclfcn-64/?prefix=obj2 2024-02-01T10:50:13.635 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj2 2024-02-01T10:50:13.635 DEBUG:boto:Method: GET 2024-02-01T10:50:13.635 DEBUG:boto:Path: /vclfcn-64/?prefix=obj2 2024-02-01T10:50:13.635 DEBUG:boto:Data: 2024-02-01T10:50:13.635 DEBUG:boto:Headers: {} 2024-02-01T10:50:13.635 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:13.635 DEBUG:boto:Port: 8000 2024-02-01T10:50:13.635 DEBUG:boto:Params: {} 2024-02-01T10:50:13.635 DEBUG:boto:Token: None 2024-02-01T10:50:13.635 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:13 GMT /vclfcn-64/ 2024-02-01T10:50:13.636 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:U845z3jmrl2BY7DUvSM+mO7jCGI= 2024-02-01T10:50:13.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:U845z3jmrl2BY7DUvSM+mO7jCGI=', 'Content-Length': '0'} 2024-02-01T10:50:13.640 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043ce39c6212e5161-0065bb7765-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:13.640 DEBUG:boto:b'vclfcn-64obj21000falseobj22024-02-01T10:48:26.615Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:13.640 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:50:13.640 DEBUG:boto:path=/vclfcn-64/obj3 2024-02-01T10:50:13.640 DEBUG:boto:auth_path=/vclfcn-64/obj3 2024-02-01T10:50:13.640 DEBUG:boto:path=/vclfcn-64/obj3?versionId=null 2024-02-01T10:50:13.641 DEBUG:boto:auth_path=/vclfcn-64/obj3?versionId=null 2024-02-01T10:50:13.641 DEBUG:boto:Method: GET 2024-02-01T10:50:13.641 DEBUG:boto:Path: /vclfcn-64/obj3?versionId=null 2024-02-01T10:50:13.641 DEBUG:boto:Data: 2024-02-01T10:50:13.641 DEBUG:boto:Headers: {} 2024-02-01T10:50:13.641 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:13.641 DEBUG:boto:Port: 8001 2024-02-01T10:50:13.641 DEBUG:boto:Params: {} 2024-02-01T10:50:13.641 DEBUG:boto:Token: None 2024-02-01T10:50:13.641 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:13 GMT /vclfcn-64/obj3?versionId=null 2024-02-01T10:50:13.641 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:kQl1f7/ueNLd42DexWvhC22/o7o= 2024-02-01T10:50:13.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:kQl1f7/ueNLd42DexWvhC22/o7o=', 'Content-Length': '0'} 2024-02-01T10:50:13.644 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006f62e54fcdeff39f-0065bb7765-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:13.644 DEBUG:boto:path=/vclfcn-64/obj3 2024-02-01T10:50:13.645 DEBUG:boto:auth_path=/vclfcn-64/obj3 2024-02-01T10:50:13.645 DEBUG:boto:path=/vclfcn-64/obj3?versionId=null 2024-02-01T10:50:13.645 DEBUG:boto:auth_path=/vclfcn-64/obj3?versionId=null 2024-02-01T10:50:13.645 DEBUG:boto:Method: GET 2024-02-01T10:50:13.645 DEBUG:boto:Path: /vclfcn-64/obj3?versionId=null 2024-02-01T10:50:13.645 DEBUG:boto:Data: 2024-02-01T10:50:13.645 DEBUG:boto:Headers: {} 2024-02-01T10:50:13.645 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:13.645 DEBUG:boto:Port: 8000 2024-02-01T10:50:13.645 DEBUG:boto:Params: {} 2024-02-01T10:50:13.646 DEBUG:boto:Token: None 2024-02-01T10:50:13.646 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:13 GMT /vclfcn-64/obj3?versionId=null 2024-02-01T10:50:13.646 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:kQl1f7/ueNLd42DexWvhC22/o7o= 2024-02-01T10:50:13.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:kQl1f7/ueNLd42DexWvhC22/o7o=', 'Content-Length': '0'} 2024-02-01T10:50:13.648 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000925b1adced173c8c-0065bb7765-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:13.649 DEBUG:boto:path=/vclfcn-64/obj3 2024-02-01T10:50:13.649 DEBUG:boto:auth_path=/vclfcn-64/obj3 2024-02-01T10:50:13.650 DEBUG:boto:path=/vclfcn-64/obj3?versionId=null 2024-02-01T10:50:13.650 DEBUG:boto:auth_path=/vclfcn-64/obj3?versionId=null 2024-02-01T10:50:13.650 DEBUG:boto:Method: HEAD 2024-02-01T10:50:13.650 DEBUG:boto:Path: /vclfcn-64/obj3?versionId=null 2024-02-01T10:50:13.650 DEBUG:boto:Data: 2024-02-01T10:50:13.650 DEBUG:boto:Headers: {} 2024-02-01T10:50:13.650 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:13.650 DEBUG:boto:Port: 8001 2024-02-01T10:50:13.650 DEBUG:boto:Params: {} 2024-02-01T10:50:13.650 DEBUG:boto:Token: None 2024-02-01T10:50:13.651 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:13 GMT /vclfcn-64/obj3?versionId=null 2024-02-01T10:50:13.651 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:O4oLWqNvIXWg81t6zmalqIYSAts= 2024-02-01T10:50:13.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:O4oLWqNvIXWg81t6zmalqIYSAts=', 'Content-Length': '0'} 2024-02-01T10:50:13.654 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000037afdc8716af907-0065bb7765-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:13.654 DEBUG:boto:path=/vclfcn-64/obj3 2024-02-01T10:50:13.654 DEBUG:boto:auth_path=/vclfcn-64/obj3 2024-02-01T10:50:13.654 DEBUG:boto:path=/vclfcn-64/obj3?versionId=null 2024-02-01T10:50:13.654 DEBUG:boto:auth_path=/vclfcn-64/obj3?versionId=null 2024-02-01T10:50:13.654 DEBUG:boto:Method: HEAD 2024-02-01T10:50:13.654 DEBUG:boto:Path: /vclfcn-64/obj3?versionId=null 2024-02-01T10:50:13.654 DEBUG:boto:Data: 2024-02-01T10:50:13.655 DEBUG:boto:Headers: {} 2024-02-01T10:50:13.655 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:13.655 DEBUG:boto:Port: 8000 2024-02-01T10:50:13.655 DEBUG:boto:Params: {} 2024-02-01T10:50:13.655 DEBUG:boto:Token: None 2024-02-01T10:50:13.655 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:13 GMT /vclfcn-64/obj3?versionId=null 2024-02-01T10:50:13.655 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:O4oLWqNvIXWg81t6zmalqIYSAts= 2024-02-01T10:50:13.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:O4oLWqNvIXWg81t6zmalqIYSAts=', 'Content-Length': '0'} 2024-02-01T10:50:13.658 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000ef0bfc683ba53da-0065bb7765-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:13.658 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:50:13.658 DEBUG:boto:path=/vclfcn-64/obj3 2024-02-01T10:50:13.658 DEBUG:boto:auth_path=/vclfcn-64/obj3 2024-02-01T10:50:13.659 DEBUG:boto:path=/vclfcn-64/obj3?versionId=null 2024-02-01T10:50:13.659 DEBUG:boto:auth_path=/vclfcn-64/obj3?versionId=null 2024-02-01T10:50:13.659 DEBUG:boto:Method: GET 2024-02-01T10:50:13.659 DEBUG:boto:Path: /vclfcn-64/obj3?versionId=null 2024-02-01T10:50:13.659 DEBUG:boto:Data: 2024-02-01T10:50:13.659 DEBUG:boto:Headers: {} 2024-02-01T10:50:13.659 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:13.659 DEBUG:boto:Port: 8001 2024-02-01T10:50:13.659 DEBUG:boto:Params: {} 2024-02-01T10:50:13.659 DEBUG:boto:Token: None 2024-02-01T10:50:13.660 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:13 GMT /vclfcn-64/obj3?versionId=null 2024-02-01T10:50:13.660 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:kQl1f7/ueNLd42DexWvhC22/o7o= 2024-02-01T10:50:13.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:kQl1f7/ueNLd42DexWvhC22/o7o=', 'Content-Length': '0'} 2024-02-01T10:50:13.662 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000043f7f11e860f9aa1-0065bb7765-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:13.704 DEBUG:boto:path=/vclfcn-64/obj3 2024-02-01T10:50:13.705 DEBUG:boto:auth_path=/vclfcn-64/obj3 2024-02-01T10:50:13.705 DEBUG:boto:path=/vclfcn-64/obj3?versionId=null 2024-02-01T10:50:13.705 DEBUG:boto:auth_path=/vclfcn-64/obj3?versionId=null 2024-02-01T10:50:13.705 DEBUG:boto:Method: GET 2024-02-01T10:50:13.705 DEBUG:boto:Path: /vclfcn-64/obj3?versionId=null 2024-02-01T10:50:13.705 DEBUG:boto:Data: 2024-02-01T10:50:13.705 DEBUG:boto:Headers: {} 2024-02-01T10:50:13.705 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:13.705 DEBUG:boto:Port: 8000 2024-02-01T10:50:13.706 DEBUG:boto:Params: {} 2024-02-01T10:50:13.706 DEBUG:boto:Token: None 2024-02-01T10:50:13.706 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:13 GMT /vclfcn-64/obj3?versionId=null 2024-02-01T10:50:13.706 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:kQl1f7/ueNLd42DexWvhC22/o7o= 2024-02-01T10:50:13.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:kQl1f7/ueNLd42DexWvhC22/o7o=', 'Content-Length': '0'} 2024-02-01T10:50:13.709 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003f73ed42a9eca2c0-0065bb7765-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:13.752 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:13.755 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:13.755 DEBUG:boto:path=/vclfcn-64/?prefix=obj3 2024-02-01T10:50:13.755 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj3 2024-02-01T10:50:13.755 DEBUG:boto:Method: GET 2024-02-01T10:50:13.755 DEBUG:boto:Path: /vclfcn-64/?prefix=obj3 2024-02-01T10:50:13.755 DEBUG:boto:Data: 2024-02-01T10:50:13.755 DEBUG:boto:Headers: {} 2024-02-01T10:50:13.755 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:13.755 DEBUG:boto:Port: 8001 2024-02-01T10:50:13.756 DEBUG:boto:Params: {} 2024-02-01T10:50:13.756 DEBUG:boto:Token: None 2024-02-01T10:50:13.756 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:13 GMT /vclfcn-64/ 2024-02-01T10:50:13.756 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:U845z3jmrl2BY7DUvSM+mO7jCGI= 2024-02-01T10:50:13.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:U845z3jmrl2BY7DUvSM+mO7jCGI=', 'Content-Length': '0'} 2024-02-01T10:50:13.761 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000307d8a80c22ac153-0065bb7765-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:13.762 DEBUG:boto:b'vclfcn-64obj31000falseobj32024-02-01T10:48:26.634Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:13.762 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:13.762 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:13.762 DEBUG:boto:path=/vclfcn-64/?prefix=obj3 2024-02-01T10:50:13.762 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj3 2024-02-01T10:50:13.762 DEBUG:boto:Method: GET 2024-02-01T10:50:13.763 DEBUG:boto:Path: /vclfcn-64/?prefix=obj3 2024-02-01T10:50:13.763 DEBUG:boto:Data: 2024-02-01T10:50:13.763 DEBUG:boto:Headers: {} 2024-02-01T10:50:13.763 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:13.763 DEBUG:boto:Port: 8000 2024-02-01T10:50:13.763 DEBUG:boto:Params: {} 2024-02-01T10:50:13.763 DEBUG:boto:Token: None 2024-02-01T10:50:13.763 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:13 GMT /vclfcn-64/ 2024-02-01T10:50:13.763 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:U845z3jmrl2BY7DUvSM+mO7jCGI= 2024-02-01T10:50:13.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:U845z3jmrl2BY7DUvSM+mO7jCGI=', 'Content-Length': '0'} 2024-02-01T10:50:13.768 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004db2cda3f4ebb3b3-0065bb7765-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:13.768 DEBUG:boto:b'vclfcn-64obj31000falseobj32024-02-01T10:48:26.634Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:13.768 DEBUG:boto:path=/vclfcn-64/obj3 2024-02-01T10:50:13.768 DEBUG:boto:auth_path=/vclfcn-64/obj3 2024-02-01T10:50:13.769 DEBUG:boto:Method: HEAD 2024-02-01T10:50:13.769 DEBUG:boto:Path: /vclfcn-64/obj3 2024-02-01T10:50:13.769 DEBUG:boto:Data: 2024-02-01T10:50:13.769 DEBUG:boto:Headers: {} 2024-02-01T10:50:13.769 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:13.769 DEBUG:boto:Port: 8001 2024-02-01T10:50:13.769 DEBUG:boto:Params: {} 2024-02-01T10:50:13.769 DEBUG:boto:Token: None 2024-02-01T10:50:13.769 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:13 GMT /vclfcn-64/obj3 2024-02-01T10:50:13.769 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:CtRrBsgscXIw1qFfyqo1gSgfOpg= 2024-02-01T10:50:13.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:CtRrBsgscXIw1qFfyqo1gSgfOpg=', 'Content-Length': '0'} 2024-02-01T10:50:13.773 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005a47702ba70e53eb-0065bb7765-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:13.774 DEBUG:boto:path=/vclfcn-64/obj3 2024-02-01T10:50:13.774 DEBUG:boto:auth_path=/vclfcn-64/obj3 2024-02-01T10:50:13.774 DEBUG:boto:Method: HEAD 2024-02-01T10:50:13.774 DEBUG:boto:Path: /vclfcn-64/obj3 2024-02-01T10:50:13.774 DEBUG:boto:Data: 2024-02-01T10:50:13.774 DEBUG:boto:Headers: {} 2024-02-01T10:50:13.774 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:13.774 DEBUG:boto:Port: 8000 2024-02-01T10:50:13.774 DEBUG:boto:Params: {} 2024-02-01T10:50:13.775 DEBUG:boto:Token: None 2024-02-01T10:50:13.775 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:13 GMT /vclfcn-64/obj3 2024-02-01T10:50:13.775 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:CtRrBsgscXIw1qFfyqo1gSgfOpg= 2024-02-01T10:50:13.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:CtRrBsgscXIw1qFfyqo1gSgfOpg=', 'Content-Length': '0'} 2024-02-01T10:50:13.777 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000299e8a48a897c071-0065bb7765-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:13.778 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:50:13.778 DEBUG:boto:path=/vclfcn-64/obj3 2024-02-01T10:50:13.778 DEBUG:boto:auth_path=/vclfcn-64/obj3 2024-02-01T10:50:13.778 DEBUG:boto:Method: GET 2024-02-01T10:50:13.778 DEBUG:boto:Path: /vclfcn-64/obj3 2024-02-01T10:50:13.778 DEBUG:boto:Data: 2024-02-01T10:50:13.778 DEBUG:boto:Headers: {} 2024-02-01T10:50:13.778 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:13.778 DEBUG:boto:Port: 8001 2024-02-01T10:50:13.779 DEBUG:boto:Params: {} 2024-02-01T10:50:13.779 DEBUG:boto:Token: None 2024-02-01T10:50:13.779 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:13 GMT /vclfcn-64/obj3 2024-02-01T10:50:13.779 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:bTZ/+MLkUNFYIm4aPGWvPHMDDio= 2024-02-01T10:50:13.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:bTZ/+MLkUNFYIm4aPGWvPHMDDio=', 'Content-Length': '0'} 2024-02-01T10:50:13.782 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000072a8381d39590ee-0065bb7765-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:13.782 DEBUG:boto:path=/vclfcn-64/obj3 2024-02-01T10:50:13.782 DEBUG:boto:auth_path=/vclfcn-64/obj3 2024-02-01T10:50:13.782 DEBUG:boto:Method: GET 2024-02-01T10:50:13.782 DEBUG:boto:Path: /vclfcn-64/obj3 2024-02-01T10:50:13.782 DEBUG:boto:Data: 2024-02-01T10:50:13.782 DEBUG:boto:Headers: {} 2024-02-01T10:50:13.782 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:13.783 DEBUG:boto:Port: 8000 2024-02-01T10:50:13.783 DEBUG:boto:Params: {} 2024-02-01T10:50:13.783 DEBUG:boto:Token: None 2024-02-01T10:50:13.783 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:13 GMT /vclfcn-64/obj3 2024-02-01T10:50:13.783 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:bTZ/+MLkUNFYIm4aPGWvPHMDDio= 2024-02-01T10:50:13.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:bTZ/+MLkUNFYIm4aPGWvPHMDDio=', 'Content-Length': '0'} 2024-02-01T10:50:13.786 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b2f2b8a5eea916ba-0065bb7765-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:13.828 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:13.828 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:13.828 DEBUG:boto:path=/vclfcn-64/?prefix=obj3 2024-02-01T10:50:13.828 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj3 2024-02-01T10:50:13.829 DEBUG:boto:Method: GET 2024-02-01T10:50:13.829 DEBUG:boto:Path: /vclfcn-64/?prefix=obj3 2024-02-01T10:50:13.829 DEBUG:boto:Data: 2024-02-01T10:50:13.829 DEBUG:boto:Headers: {} 2024-02-01T10:50:13.829 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:13.829 DEBUG:boto:Port: 8001 2024-02-01T10:50:13.829 DEBUG:boto:Params: {} 2024-02-01T10:50:13.829 DEBUG:boto:Token: None 2024-02-01T10:50:13.829 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:13 GMT /vclfcn-64/ 2024-02-01T10:50:13.830 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:U845z3jmrl2BY7DUvSM+mO7jCGI= 2024-02-01T10:50:13.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:U845z3jmrl2BY7DUvSM+mO7jCGI=', 'Content-Length': '0'} 2024-02-01T10:50:13.834 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb429af83691c932-0065bb7765-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:13.834 DEBUG:boto:b'vclfcn-64obj31000falseobj32024-02-01T10:48:26.634Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:13.835 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:13.835 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:13.835 DEBUG:boto:path=/vclfcn-64/?prefix=obj3 2024-02-01T10:50:13.835 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj3 2024-02-01T10:50:13.835 DEBUG:boto:Method: GET 2024-02-01T10:50:13.835 DEBUG:boto:Path: /vclfcn-64/?prefix=obj3 2024-02-01T10:50:13.835 DEBUG:boto:Data: 2024-02-01T10:50:13.835 DEBUG:boto:Headers: {} 2024-02-01T10:50:13.836 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:13.836 DEBUG:boto:Port: 8000 2024-02-01T10:50:13.836 DEBUG:boto:Params: {} 2024-02-01T10:50:13.836 DEBUG:boto:Token: None 2024-02-01T10:50:13.836 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:13 GMT /vclfcn-64/ 2024-02-01T10:50:13.836 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:U845z3jmrl2BY7DUvSM+mO7jCGI= 2024-02-01T10:50:13.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:U845z3jmrl2BY7DUvSM+mO7jCGI=', 'Content-Length': '0'} 2024-02-01T10:50:13.842 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000014fa1add8c0b73b1-0065bb7765-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:13.888 DEBUG:boto:b'vclfcn-64obj31000falseobj32024-02-01T10:48:26.634Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:13.888 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:50:13.889 DEBUG:boto:path=/vclfcn-64/obj4 2024-02-01T10:50:13.889 DEBUG:boto:auth_path=/vclfcn-64/obj4 2024-02-01T10:50:13.889 DEBUG:boto:path=/vclfcn-64/obj4?versionId=null 2024-02-01T10:50:13.889 DEBUG:boto:auth_path=/vclfcn-64/obj4?versionId=null 2024-02-01T10:50:13.889 DEBUG:boto:Method: GET 2024-02-01T10:50:13.889 DEBUG:boto:Path: /vclfcn-64/obj4?versionId=null 2024-02-01T10:50:13.889 DEBUG:boto:Data: 2024-02-01T10:50:13.889 DEBUG:boto:Headers: {} 2024-02-01T10:50:13.889 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:13.889 DEBUG:boto:Port: 8001 2024-02-01T10:50:13.890 DEBUG:boto:Params: {} 2024-02-01T10:50:13.890 DEBUG:boto:Token: None 2024-02-01T10:50:13.890 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:13 GMT /vclfcn-64/obj4?versionId=null 2024-02-01T10:50:13.890 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:LLTU82cjWwBe8XaabAH35ozBxXY= 2024-02-01T10:50:13.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:LLTU82cjWwBe8XaabAH35ozBxXY=', 'Content-Length': '0'} 2024-02-01T10:50:13.893 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000001ffee1c8ad3ad70-0065bb7765-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:13.894 DEBUG:boto:path=/vclfcn-64/obj4 2024-02-01T10:50:13.894 DEBUG:boto:auth_path=/vclfcn-64/obj4 2024-02-01T10:50:13.894 DEBUG:boto:path=/vclfcn-64/obj4?versionId=null 2024-02-01T10:50:13.894 DEBUG:boto:auth_path=/vclfcn-64/obj4?versionId=null 2024-02-01T10:50:13.894 DEBUG:boto:Method: GET 2024-02-01T10:50:13.894 DEBUG:boto:Path: /vclfcn-64/obj4?versionId=null 2024-02-01T10:50:13.894 DEBUG:boto:Data: 2024-02-01T10:50:13.894 DEBUG:boto:Headers: {} 2024-02-01T10:50:13.894 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:13.894 DEBUG:boto:Port: 8000 2024-02-01T10:50:13.895 DEBUG:boto:Params: {} 2024-02-01T10:50:13.895 DEBUG:boto:Token: None 2024-02-01T10:50:13.895 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:13 GMT /vclfcn-64/obj4?versionId=null 2024-02-01T10:50:13.895 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:LLTU82cjWwBe8XaabAH35ozBxXY= 2024-02-01T10:50:13.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:LLTU82cjWwBe8XaabAH35ozBxXY=', 'Content-Length': '0'} 2024-02-01T10:50:13.898 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b764298a4438aad6-0065bb7765-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:13.940 DEBUG:boto:path=/vclfcn-64/obj4 2024-02-01T10:50:13.940 DEBUG:boto:auth_path=/vclfcn-64/obj4 2024-02-01T10:50:13.940 DEBUG:boto:path=/vclfcn-64/obj4?versionId=null 2024-02-01T10:50:13.940 DEBUG:boto:auth_path=/vclfcn-64/obj4?versionId=null 2024-02-01T10:50:13.941 DEBUG:boto:Method: HEAD 2024-02-01T10:50:13.941 DEBUG:boto:Path: /vclfcn-64/obj4?versionId=null 2024-02-01T10:50:13.941 DEBUG:boto:Data: 2024-02-01T10:50:13.941 DEBUG:boto:Headers: {} 2024-02-01T10:50:13.941 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:13.941 DEBUG:boto:Port: 8001 2024-02-01T10:50:13.941 DEBUG:boto:Params: {} 2024-02-01T10:50:13.941 DEBUG:boto:Token: None 2024-02-01T10:50:13.941 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:13 GMT /vclfcn-64/obj4?versionId=null 2024-02-01T10:50:13.942 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:VmxPM5ZaFq+P6u8DC3n98Kdt5xM= 2024-02-01T10:50:13.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:VmxPM5ZaFq+P6u8DC3n98Kdt5xM=', 'Content-Length': '0'} 2024-02-01T10:50:13.945 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006ddfac1eb7bd1542-0065bb7765-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:13.945 DEBUG:boto:path=/vclfcn-64/obj4 2024-02-01T10:50:13.945 DEBUG:boto:auth_path=/vclfcn-64/obj4 2024-02-01T10:50:13.945 DEBUG:boto:path=/vclfcn-64/obj4?versionId=null 2024-02-01T10:50:13.945 DEBUG:boto:auth_path=/vclfcn-64/obj4?versionId=null 2024-02-01T10:50:13.945 DEBUG:boto:Method: HEAD 2024-02-01T10:50:13.945 DEBUG:boto:Path: /vclfcn-64/obj4?versionId=null 2024-02-01T10:50:13.945 DEBUG:boto:Data: 2024-02-01T10:50:13.946 DEBUG:boto:Headers: {} 2024-02-01T10:50:13.946 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:13.946 DEBUG:boto:Port: 8000 2024-02-01T10:50:13.946 DEBUG:boto:Params: {} 2024-02-01T10:50:13.946 DEBUG:boto:Token: None 2024-02-01T10:50:13.946 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:13 GMT /vclfcn-64/obj4?versionId=null 2024-02-01T10:50:13.946 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:VmxPM5ZaFq+P6u8DC3n98Kdt5xM= 2024-02-01T10:50:13.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:VmxPM5ZaFq+P6u8DC3n98Kdt5xM=', 'Content-Length': '0'} 2024-02-01T10:50:13.949 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001d121408277a836d-0065bb7765-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:13.950 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:50:13.950 DEBUG:boto:path=/vclfcn-64/obj4 2024-02-01T10:50:13.950 DEBUG:boto:auth_path=/vclfcn-64/obj4 2024-02-01T10:50:13.951 DEBUG:boto:path=/vclfcn-64/obj4?versionId=null 2024-02-01T10:50:13.951 DEBUG:boto:auth_path=/vclfcn-64/obj4?versionId=null 2024-02-01T10:50:13.951 DEBUG:boto:Method: GET 2024-02-01T10:50:13.951 DEBUG:boto:Path: /vclfcn-64/obj4?versionId=null 2024-02-01T10:50:13.951 DEBUG:boto:Data: 2024-02-01T10:50:13.952 DEBUG:boto:Headers: {} 2024-02-01T10:50:13.952 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:13.952 DEBUG:boto:Port: 8001 2024-02-01T10:50:13.952 DEBUG:boto:Params: {} 2024-02-01T10:50:13.953 DEBUG:boto:Token: None 2024-02-01T10:50:13.953 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:13 GMT /vclfcn-64/obj4?versionId=null 2024-02-01T10:50:13.953 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:LLTU82cjWwBe8XaabAH35ozBxXY= 2024-02-01T10:50:13.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:13 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:LLTU82cjWwBe8XaabAH35ozBxXY=', 'Content-Length': '0'} 2024-02-01T10:50:13.956 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006ab85f1231f366e9-0065bb7765-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:13 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:14.000 DEBUG:boto:path=/vclfcn-64/obj4 2024-02-01T10:50:14.001 DEBUG:boto:auth_path=/vclfcn-64/obj4 2024-02-01T10:50:14.001 DEBUG:boto:path=/vclfcn-64/obj4?versionId=null 2024-02-01T10:50:14.001 DEBUG:boto:auth_path=/vclfcn-64/obj4?versionId=null 2024-02-01T10:50:14.001 DEBUG:boto:Method: GET 2024-02-01T10:50:14.002 DEBUG:boto:Path: /vclfcn-64/obj4?versionId=null 2024-02-01T10:50:14.002 DEBUG:boto:Data: 2024-02-01T10:50:14.002 DEBUG:boto:Headers: {} 2024-02-01T10:50:14.002 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:14.002 DEBUG:boto:Port: 8000 2024-02-01T10:50:14.002 DEBUG:boto:Params: {} 2024-02-01T10:50:14.002 DEBUG:boto:Token: None 2024-02-01T10:50:14.002 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:14 GMT /vclfcn-64/obj4?versionId=null 2024-02-01T10:50:14.003 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:RIrV2MyoeWEHyiGADgN5Hm03P54= 2024-02-01T10:50:14.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:RIrV2MyoeWEHyiGADgN5Hm03P54=', 'Content-Length': '0'} 2024-02-01T10:50:14.006 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009fffe0212faa5dd8-0065bb7766-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:14.052 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:14.052 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:14.053 DEBUG:boto:path=/vclfcn-64/?prefix=obj4 2024-02-01T10:50:14.053 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj4 2024-02-01T10:50:14.053 DEBUG:boto:Method: GET 2024-02-01T10:50:14.053 DEBUG:boto:Path: /vclfcn-64/?prefix=obj4 2024-02-01T10:50:14.053 DEBUG:boto:Data: 2024-02-01T10:50:14.053 DEBUG:boto:Headers: {} 2024-02-01T10:50:14.053 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:14.053 DEBUG:boto:Port: 8001 2024-02-01T10:50:14.053 DEBUG:boto:Params: {} 2024-02-01T10:50:14.054 DEBUG:boto:Token: None 2024-02-01T10:50:14.054 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:14 GMT /vclfcn-64/ 2024-02-01T10:50:14.054 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:AxA3arhtwk68gbwpuLrcUfT7xh4= 2024-02-01T10:50:14.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:AxA3arhtwk68gbwpuLrcUfT7xh4=', 'Content-Length': '0'} 2024-02-01T10:50:14.059 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ca9c9fa4b7107f7-0065bb7766-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:14.059 DEBUG:boto:b'vclfcn-64obj41000falseobj42024-02-01T10:48:26.689Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:14.059 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:14.059 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:14.059 DEBUG:boto:path=/vclfcn-64/?prefix=obj4 2024-02-01T10:50:14.060 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj4 2024-02-01T10:50:14.060 DEBUG:boto:Method: GET 2024-02-01T10:50:14.060 DEBUG:boto:Path: /vclfcn-64/?prefix=obj4 2024-02-01T10:50:14.060 DEBUG:boto:Data: 2024-02-01T10:50:14.060 DEBUG:boto:Headers: {} 2024-02-01T10:50:14.060 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:14.060 DEBUG:boto:Port: 8000 2024-02-01T10:50:14.060 DEBUG:boto:Params: {} 2024-02-01T10:50:14.060 DEBUG:boto:Token: None 2024-02-01T10:50:14.060 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:14 GMT /vclfcn-64/ 2024-02-01T10:50:14.061 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:AxA3arhtwk68gbwpuLrcUfT7xh4= 2024-02-01T10:50:14.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:AxA3arhtwk68gbwpuLrcUfT7xh4=', 'Content-Length': '0'} 2024-02-01T10:50:14.065 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007056ba3d5a6109c0-0065bb7766-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:14.065 DEBUG:boto:b'vclfcn-64obj41000falseobj42024-02-01T10:48:26.689Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:14.065 DEBUG:boto:path=/vclfcn-64/obj4 2024-02-01T10:50:14.065 DEBUG:boto:auth_path=/vclfcn-64/obj4 2024-02-01T10:50:14.066 DEBUG:boto:Method: HEAD 2024-02-01T10:50:14.066 DEBUG:boto:Path: /vclfcn-64/obj4 2024-02-01T10:50:14.066 DEBUG:boto:Data: 2024-02-01T10:50:14.066 DEBUG:boto:Headers: {} 2024-02-01T10:50:14.066 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:14.066 DEBUG:boto:Port: 8001 2024-02-01T10:50:14.066 DEBUG:boto:Params: {} 2024-02-01T10:50:14.066 DEBUG:boto:Token: None 2024-02-01T10:50:14.066 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:14 GMT /vclfcn-64/obj4 2024-02-01T10:50:14.066 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0IQyB65BStOgV0eRz1Kds36lkvc= 2024-02-01T10:50:14.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0IQyB65BStOgV0eRz1Kds36lkvc=', 'Content-Length': '0'} 2024-02-01T10:50:14.069 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005a2e3af139880168-0065bb7766-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:14.069 DEBUG:boto:path=/vclfcn-64/obj4 2024-02-01T10:50:14.069 DEBUG:boto:auth_path=/vclfcn-64/obj4 2024-02-01T10:50:14.069 DEBUG:boto:Method: HEAD 2024-02-01T10:50:14.069 DEBUG:boto:Path: /vclfcn-64/obj4 2024-02-01T10:50:14.069 DEBUG:boto:Data: 2024-02-01T10:50:14.070 DEBUG:boto:Headers: {} 2024-02-01T10:50:14.070 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:14.070 DEBUG:boto:Port: 8000 2024-02-01T10:50:14.070 DEBUG:boto:Params: {} 2024-02-01T10:50:14.070 DEBUG:boto:Token: None 2024-02-01T10:50:14.070 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:14 GMT /vclfcn-64/obj4 2024-02-01T10:50:14.070 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0IQyB65BStOgV0eRz1Kds36lkvc= 2024-02-01T10:50:14.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0IQyB65BStOgV0eRz1Kds36lkvc=', 'Content-Length': '0'} 2024-02-01T10:50:14.073 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000c83f9dc27021c14-0065bb7766-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:14.073 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:50:14.073 DEBUG:boto:path=/vclfcn-64/obj4 2024-02-01T10:50:14.074 DEBUG:boto:auth_path=/vclfcn-64/obj4 2024-02-01T10:50:14.074 DEBUG:boto:Method: GET 2024-02-01T10:50:14.074 DEBUG:boto:Path: /vclfcn-64/obj4 2024-02-01T10:50:14.074 DEBUG:boto:Data: 2024-02-01T10:50:14.074 DEBUG:boto:Headers: {} 2024-02-01T10:50:14.074 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:14.074 DEBUG:boto:Port: 8001 2024-02-01T10:50:14.074 DEBUG:boto:Params: {} 2024-02-01T10:50:14.074 DEBUG:boto:Token: None 2024-02-01T10:50:14.075 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:14 GMT /vclfcn-64/obj4 2024-02-01T10:50:14.075 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:kn/AlbPU22fjtALpjETUP9GHe9U= 2024-02-01T10:50:14.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:kn/AlbPU22fjtALpjETUP9GHe9U=', 'Content-Length': '0'} 2024-02-01T10:50:14.077 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000226110571b6be4b5-0065bb7766-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:14.078 DEBUG:boto:path=/vclfcn-64/obj4 2024-02-01T10:50:14.078 DEBUG:boto:auth_path=/vclfcn-64/obj4 2024-02-01T10:50:14.078 DEBUG:boto:Method: GET 2024-02-01T10:50:14.078 DEBUG:boto:Path: /vclfcn-64/obj4 2024-02-01T10:50:14.078 DEBUG:boto:Data: 2024-02-01T10:50:14.078 DEBUG:boto:Headers: {} 2024-02-01T10:50:14.078 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:14.078 DEBUG:boto:Port: 8000 2024-02-01T10:50:14.078 DEBUG:boto:Params: {} 2024-02-01T10:50:14.079 DEBUG:boto:Token: None 2024-02-01T10:50:14.079 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:14 GMT /vclfcn-64/obj4 2024-02-01T10:50:14.079 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:kn/AlbPU22fjtALpjETUP9GHe9U= 2024-02-01T10:50:14.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:kn/AlbPU22fjtALpjETUP9GHe9U=', 'Content-Length': '0'} 2024-02-01T10:50:14.081 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eab2fdb3ef9434e8-0065bb7766-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:14.124 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:14.124 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:14.124 DEBUG:boto:path=/vclfcn-64/?prefix=obj4 2024-02-01T10:50:14.124 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj4 2024-02-01T10:50:14.125 DEBUG:boto:Method: GET 2024-02-01T10:50:14.125 DEBUG:boto:Path: /vclfcn-64/?prefix=obj4 2024-02-01T10:50:14.125 DEBUG:boto:Data: 2024-02-01T10:50:14.125 DEBUG:boto:Headers: {} 2024-02-01T10:50:14.125 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:14.125 DEBUG:boto:Port: 8001 2024-02-01T10:50:14.125 DEBUG:boto:Params: {} 2024-02-01T10:50:14.125 DEBUG:boto:Token: None 2024-02-01T10:50:14.125 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:14 GMT /vclfcn-64/ 2024-02-01T10:50:14.125 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:AxA3arhtwk68gbwpuLrcUfT7xh4= 2024-02-01T10:50:14.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:AxA3arhtwk68gbwpuLrcUfT7xh4=', 'Content-Length': '0'} 2024-02-01T10:50:14.130 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005baa44c5206a5636-0065bb7766-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:14.130 DEBUG:boto:b'vclfcn-64obj41000falseobj42024-02-01T10:48:26.689Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:14.130 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:14.130 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:14.130 DEBUG:boto:path=/vclfcn-64/?prefix=obj4 2024-02-01T10:50:14.131 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj4 2024-02-01T10:50:14.131 DEBUG:boto:Method: GET 2024-02-01T10:50:14.131 DEBUG:boto:Path: /vclfcn-64/?prefix=obj4 2024-02-01T10:50:14.131 DEBUG:boto:Data: 2024-02-01T10:50:14.131 DEBUG:boto:Headers: {} 2024-02-01T10:50:14.131 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:14.131 DEBUG:boto:Port: 8000 2024-02-01T10:50:14.131 DEBUG:boto:Params: {} 2024-02-01T10:50:14.131 DEBUG:boto:Token: None 2024-02-01T10:50:14.132 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:14 GMT /vclfcn-64/ 2024-02-01T10:50:14.132 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:AxA3arhtwk68gbwpuLrcUfT7xh4= 2024-02-01T10:50:14.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:AxA3arhtwk68gbwpuLrcUfT7xh4=', 'Content-Length': '0'} 2024-02-01T10:50:14.136 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000783d5f9629a96895-0065bb7766-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:14.184 DEBUG:boto:b'vclfcn-64obj41000falseobj42024-02-01T10:48:26.689Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:14.185 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-02-01T10:50:14.185 DEBUG:boto:path=/vclfcn-64/obj5 2024-02-01T10:50:14.185 DEBUG:boto:auth_path=/vclfcn-64/obj5 2024-02-01T10:50:14.185 DEBUG:boto:path=/vclfcn-64/obj5?versionId=null 2024-02-01T10:50:14.185 DEBUG:boto:auth_path=/vclfcn-64/obj5?versionId=null 2024-02-01T10:50:14.185 DEBUG:boto:Method: GET 2024-02-01T10:50:14.185 DEBUG:boto:Path: /vclfcn-64/obj5?versionId=null 2024-02-01T10:50:14.185 DEBUG:boto:Data: 2024-02-01T10:50:14.185 DEBUG:boto:Headers: {} 2024-02-01T10:50:14.186 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:14.186 DEBUG:boto:Port: 8001 2024-02-01T10:50:14.186 DEBUG:boto:Params: {} 2024-02-01T10:50:14.186 DEBUG:boto:Token: None 2024-02-01T10:50:14.186 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:14 GMT /vclfcn-64/obj5?versionId=null 2024-02-01T10:50:14.186 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:r3ZdS4Y20KbbbRm7iMmDdxVy8x0= 2024-02-01T10:50:14.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:r3ZdS4Y20KbbbRm7iMmDdxVy8x0=', 'Content-Length': '0'} 2024-02-01T10:50:14.189 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c144de761bc206d7-0065bb7766-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:14.189 DEBUG:boto:path=/vclfcn-64/obj5 2024-02-01T10:50:14.189 DEBUG:boto:auth_path=/vclfcn-64/obj5 2024-02-01T10:50:14.189 DEBUG:boto:path=/vclfcn-64/obj5?versionId=null 2024-02-01T10:50:14.190 DEBUG:boto:auth_path=/vclfcn-64/obj5?versionId=null 2024-02-01T10:50:14.190 DEBUG:boto:Method: GET 2024-02-01T10:50:14.190 DEBUG:boto:Path: /vclfcn-64/obj5?versionId=null 2024-02-01T10:50:14.190 DEBUG:boto:Data: 2024-02-01T10:50:14.190 DEBUG:boto:Headers: {} 2024-02-01T10:50:14.190 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:14.190 DEBUG:boto:Port: 8000 2024-02-01T10:50:14.190 DEBUG:boto:Params: {} 2024-02-01T10:50:14.190 DEBUG:boto:Token: None 2024-02-01T10:50:14.190 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:14 GMT /vclfcn-64/obj5?versionId=null 2024-02-01T10:50:14.191 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:r3ZdS4Y20KbbbRm7iMmDdxVy8x0= 2024-02-01T10:50:14.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:r3ZdS4Y20KbbbRm7iMmDdxVy8x0=', 'Content-Length': '0'} 2024-02-01T10:50:14.193 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000023924d3d25d1326a-0065bb7766-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:14.236 DEBUG:boto:path=/vclfcn-64/obj5 2024-02-01T10:50:14.236 DEBUG:boto:auth_path=/vclfcn-64/obj5 2024-02-01T10:50:14.236 DEBUG:boto:path=/vclfcn-64/obj5?versionId=null 2024-02-01T10:50:14.237 DEBUG:boto:auth_path=/vclfcn-64/obj5?versionId=null 2024-02-01T10:50:14.237 DEBUG:boto:Method: HEAD 2024-02-01T10:50:14.237 DEBUG:boto:Path: /vclfcn-64/obj5?versionId=null 2024-02-01T10:50:14.237 DEBUG:boto:Data: 2024-02-01T10:50:14.237 DEBUG:boto:Headers: {} 2024-02-01T10:50:14.237 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:14.237 DEBUG:boto:Port: 8001 2024-02-01T10:50:14.237 DEBUG:boto:Params: {} 2024-02-01T10:50:14.237 DEBUG:boto:Token: None 2024-02-01T10:50:14.238 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:14 GMT /vclfcn-64/obj5?versionId=null 2024-02-01T10:50:14.238 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:dW+fz6d/XrlugjsNKk+LAt4GzzA= 2024-02-01T10:50:14.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dW+fz6d/XrlugjsNKk+LAt4GzzA=', 'Content-Length': '0'} 2024-02-01T10:50:14.241 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b57071527c80c0a5-0065bb7766-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:14.241 DEBUG:boto:path=/vclfcn-64/obj5 2024-02-01T10:50:14.241 DEBUG:boto:auth_path=/vclfcn-64/obj5 2024-02-01T10:50:14.241 DEBUG:boto:path=/vclfcn-64/obj5?versionId=null 2024-02-01T10:50:14.241 DEBUG:boto:auth_path=/vclfcn-64/obj5?versionId=null 2024-02-01T10:50:14.241 DEBUG:boto:Method: HEAD 2024-02-01T10:50:14.241 DEBUG:boto:Path: /vclfcn-64/obj5?versionId=null 2024-02-01T10:50:14.241 DEBUG:boto:Data: 2024-02-01T10:50:14.241 DEBUG:boto:Headers: {} 2024-02-01T10:50:14.242 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:14.242 DEBUG:boto:Port: 8000 2024-02-01T10:50:14.242 DEBUG:boto:Params: {} 2024-02-01T10:50:14.242 DEBUG:boto:Token: None 2024-02-01T10:50:14.242 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:14 GMT /vclfcn-64/obj5?versionId=null 2024-02-01T10:50:14.242 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:dW+fz6d/XrlugjsNKk+LAt4GzzA= 2024-02-01T10:50:14.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dW+fz6d/XrlugjsNKk+LAt4GzzA=', 'Content-Length': '0'} 2024-02-01T10:50:14.245 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eedc7c09df38c009-0065bb7766-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:14.245 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-02-01T10:50:14.245 DEBUG:boto:path=/vclfcn-64/obj5 2024-02-01T10:50:14.245 DEBUG:boto:auth_path=/vclfcn-64/obj5 2024-02-01T10:50:14.245 DEBUG:boto:path=/vclfcn-64/obj5?versionId=null 2024-02-01T10:50:14.245 DEBUG:boto:auth_path=/vclfcn-64/obj5?versionId=null 2024-02-01T10:50:14.246 DEBUG:boto:Method: GET 2024-02-01T10:50:14.246 DEBUG:boto:Path: /vclfcn-64/obj5?versionId=null 2024-02-01T10:50:14.246 DEBUG:boto:Data: 2024-02-01T10:50:14.246 DEBUG:boto:Headers: {} 2024-02-01T10:50:14.246 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:14.246 DEBUG:boto:Port: 8001 2024-02-01T10:50:14.246 DEBUG:boto:Params: {} 2024-02-01T10:50:14.246 DEBUG:boto:Token: None 2024-02-01T10:50:14.246 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:14 GMT /vclfcn-64/obj5?versionId=null 2024-02-01T10:50:14.246 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:r3ZdS4Y20KbbbRm7iMmDdxVy8x0= 2024-02-01T10:50:14.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:r3ZdS4Y20KbbbRm7iMmDdxVy8x0=', 'Content-Length': '0'} 2024-02-01T10:50:14.249 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000676f410ad472dedf-0065bb7766-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:14.296 DEBUG:boto:path=/vclfcn-64/obj5 2024-02-01T10:50:14.296 DEBUG:boto:auth_path=/vclfcn-64/obj5 2024-02-01T10:50:14.296 DEBUG:boto:path=/vclfcn-64/obj5?versionId=null 2024-02-01T10:50:14.297 DEBUG:boto:auth_path=/vclfcn-64/obj5?versionId=null 2024-02-01T10:50:14.297 DEBUG:boto:Method: GET 2024-02-01T10:50:14.297 DEBUG:boto:Path: /vclfcn-64/obj5?versionId=null 2024-02-01T10:50:14.297 DEBUG:boto:Data: 2024-02-01T10:50:14.297 DEBUG:boto:Headers: {} 2024-02-01T10:50:14.297 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:14.297 DEBUG:boto:Port: 8000 2024-02-01T10:50:14.297 DEBUG:boto:Params: {} 2024-02-01T10:50:14.297 DEBUG:boto:Token: None 2024-02-01T10:50:14.298 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:14 GMT /vclfcn-64/obj5?versionId=null 2024-02-01T10:50:14.298 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:r3ZdS4Y20KbbbRm7iMmDdxVy8x0= 2024-02-01T10:50:14.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:r3ZdS4Y20KbbbRm7iMmDdxVy8x0=', 'Content-Length': '0'} 2024-02-01T10:50:14.300 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009cb98da5c1215049-0065bb7766-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:14.344 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:14.344 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:14.344 DEBUG:boto:path=/vclfcn-64/?prefix=obj5 2024-02-01T10:50:14.345 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj5 2024-02-01T10:50:14.345 DEBUG:boto:Method: GET 2024-02-01T10:50:14.345 DEBUG:boto:Path: /vclfcn-64/?prefix=obj5 2024-02-01T10:50:14.345 DEBUG:boto:Data: 2024-02-01T10:50:14.345 DEBUG:boto:Headers: {} 2024-02-01T10:50:14.345 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:14.345 DEBUG:boto:Port: 8001 2024-02-01T10:50:14.345 DEBUG:boto:Params: {} 2024-02-01T10:50:14.345 DEBUG:boto:Token: None 2024-02-01T10:50:14.345 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:14 GMT /vclfcn-64/ 2024-02-01T10:50:14.346 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:AxA3arhtwk68gbwpuLrcUfT7xh4= 2024-02-01T10:50:14.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:AxA3arhtwk68gbwpuLrcUfT7xh4=', 'Content-Length': '0'} 2024-02-01T10:50:14.350 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3a1d61f7d5d6d2c-0065bb7766-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:14.350 DEBUG:boto:b'vclfcn-64obj51000falseobj52024-02-01T10:49:22.402Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:14.351 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:14.351 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:14.351 DEBUG:boto:path=/vclfcn-64/?prefix=obj5 2024-02-01T10:50:14.351 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj5 2024-02-01T10:50:14.351 DEBUG:boto:Method: GET 2024-02-01T10:50:14.351 DEBUG:boto:Path: /vclfcn-64/?prefix=obj5 2024-02-01T10:50:14.351 DEBUG:boto:Data: 2024-02-01T10:50:14.352 DEBUG:boto:Headers: {} 2024-02-01T10:50:14.352 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:14.352 DEBUG:boto:Port: 8000 2024-02-01T10:50:14.352 DEBUG:boto:Params: {} 2024-02-01T10:50:14.352 DEBUG:boto:Token: None 2024-02-01T10:50:14.352 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:14 GMT /vclfcn-64/ 2024-02-01T10:50:14.352 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:AxA3arhtwk68gbwpuLrcUfT7xh4= 2024-02-01T10:50:14.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:AxA3arhtwk68gbwpuLrcUfT7xh4=', 'Content-Length': '0'} 2024-02-01T10:50:14.356 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000401bdfd3cb9935fd-0065bb7766-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:14.356 DEBUG:boto:b'vclfcn-64obj51000falseobj52024-02-01T10:49:22.402Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:14.357 DEBUG:boto:path=/vclfcn-64/obj5 2024-02-01T10:50:14.357 DEBUG:boto:auth_path=/vclfcn-64/obj5 2024-02-01T10:50:14.357 DEBUG:boto:Method: HEAD 2024-02-01T10:50:14.357 DEBUG:boto:Path: /vclfcn-64/obj5 2024-02-01T10:50:14.357 DEBUG:boto:Data: 2024-02-01T10:50:14.357 DEBUG:boto:Headers: {} 2024-02-01T10:50:14.357 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:14.357 DEBUG:boto:Port: 8001 2024-02-01T10:50:14.357 DEBUG:boto:Params: {} 2024-02-01T10:50:14.358 DEBUG:boto:Token: None 2024-02-01T10:50:14.358 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:14 GMT /vclfcn-64/obj5 2024-02-01T10:50:14.358 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5y8C3OEZcyWH8a/D9D3PxZ3oJwE= 2024-02-01T10:50:14.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5y8C3OEZcyWH8a/D9D3PxZ3oJwE=', 'Content-Length': '0'} 2024-02-01T10:50:14.360 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000554e1da5a1c1a06f-0065bb7766-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:14.361 DEBUG:boto:path=/vclfcn-64/obj5 2024-02-01T10:50:14.361 DEBUG:boto:auth_path=/vclfcn-64/obj5 2024-02-01T10:50:14.361 DEBUG:boto:Method: HEAD 2024-02-01T10:50:14.361 DEBUG:boto:Path: /vclfcn-64/obj5 2024-02-01T10:50:14.361 DEBUG:boto:Data: 2024-02-01T10:50:14.362 DEBUG:boto:Headers: {} 2024-02-01T10:50:14.362 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:14.362 DEBUG:boto:Port: 8000 2024-02-01T10:50:14.362 DEBUG:boto:Params: {} 2024-02-01T10:50:14.363 DEBUG:boto:Token: None 2024-02-01T10:50:14.363 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:14 GMT /vclfcn-64/obj5 2024-02-01T10:50:14.363 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:5y8C3OEZcyWH8a/D9D3PxZ3oJwE= 2024-02-01T10:50:14.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:5y8C3OEZcyWH8a/D9D3PxZ3oJwE=', 'Content-Length': '0'} 2024-02-01T10:50:14.366 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000095d04c1365265d59-0065bb7766-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:14.367 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-02-01T10:50:14.367 DEBUG:boto:path=/vclfcn-64/obj5 2024-02-01T10:50:14.367 DEBUG:boto:auth_path=/vclfcn-64/obj5 2024-02-01T10:50:14.367 DEBUG:boto:Method: GET 2024-02-01T10:50:14.368 DEBUG:boto:Path: /vclfcn-64/obj5 2024-02-01T10:50:14.368 DEBUG:boto:Data: 2024-02-01T10:50:14.368 DEBUG:boto:Headers: {} 2024-02-01T10:50:14.368 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:14.368 DEBUG:boto:Port: 8001 2024-02-01T10:50:14.369 DEBUG:boto:Params: {} 2024-02-01T10:50:14.371 DEBUG:boto:Token: None 2024-02-01T10:50:14.371 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:14 GMT /vclfcn-64/obj5 2024-02-01T10:50:14.371 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Im5rSkfAmK49J0HIPvBD8xnz9hQ= 2024-02-01T10:50:14.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Im5rSkfAmK49J0HIPvBD8xnz9hQ=', 'Content-Length': '0'} 2024-02-01T10:50:14.375 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000803b4240e40a7737-0065bb7766-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:14.375 DEBUG:boto:path=/vclfcn-64/obj5 2024-02-01T10:50:14.376 DEBUG:boto:auth_path=/vclfcn-64/obj5 2024-02-01T10:50:14.376 DEBUG:boto:Method: GET 2024-02-01T10:50:14.376 DEBUG:boto:Path: /vclfcn-64/obj5 2024-02-01T10:50:14.376 DEBUG:boto:Data: 2024-02-01T10:50:14.376 DEBUG:boto:Headers: {} 2024-02-01T10:50:14.377 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:14.377 DEBUG:boto:Port: 8000 2024-02-01T10:50:14.377 DEBUG:boto:Params: {} 2024-02-01T10:50:14.377 DEBUG:boto:Token: None 2024-02-01T10:50:14.378 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:14 GMT /vclfcn-64/obj5 2024-02-01T10:50:14.378 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Im5rSkfAmK49J0HIPvBD8xnz9hQ= 2024-02-01T10:50:14.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Im5rSkfAmK49J0HIPvBD8xnz9hQ=', 'Content-Length': '0'} 2024-02-01T10:50:14.381 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ef6b7690b2bffefc-0065bb7766-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:14.428 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:14.428 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:14.428 DEBUG:boto:path=/vclfcn-64/?prefix=obj5 2024-02-01T10:50:14.429 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj5 2024-02-01T10:50:14.429 DEBUG:boto:Method: GET 2024-02-01T10:50:14.429 DEBUG:boto:Path: /vclfcn-64/?prefix=obj5 2024-02-01T10:50:14.429 DEBUG:boto:Data: 2024-02-01T10:50:14.429 DEBUG:boto:Headers: {} 2024-02-01T10:50:14.429 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:14.429 DEBUG:boto:Port: 8001 2024-02-01T10:50:14.429 DEBUG:boto:Params: {} 2024-02-01T10:50:14.429 DEBUG:boto:Token: None 2024-02-01T10:50:14.430 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:14 GMT /vclfcn-64/ 2024-02-01T10:50:14.430 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:AxA3arhtwk68gbwpuLrcUfT7xh4= 2024-02-01T10:50:14.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:AxA3arhtwk68gbwpuLrcUfT7xh4=', 'Content-Length': '0'} 2024-02-01T10:50:14.434 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c41864a9aa932f20-0065bb7766-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:14.434 DEBUG:boto:b'vclfcn-64obj51000falseobj52024-02-01T10:49:22.402Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:14.435 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:14.435 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:14.435 DEBUG:boto:path=/vclfcn-64/?prefix=obj5 2024-02-01T10:50:14.435 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj5 2024-02-01T10:50:14.435 DEBUG:boto:Method: GET 2024-02-01T10:50:14.435 DEBUG:boto:Path: /vclfcn-64/?prefix=obj5 2024-02-01T10:50:14.436 DEBUG:boto:Data: 2024-02-01T10:50:14.436 DEBUG:boto:Headers: {} 2024-02-01T10:50:14.436 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:14.436 DEBUG:boto:Port: 8000 2024-02-01T10:50:14.436 DEBUG:boto:Params: {} 2024-02-01T10:50:14.436 DEBUG:boto:Token: None 2024-02-01T10:50:14.436 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:14 GMT /vclfcn-64/ 2024-02-01T10:50:14.436 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:AxA3arhtwk68gbwpuLrcUfT7xh4= 2024-02-01T10:50:14.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:AxA3arhtwk68gbwpuLrcUfT7xh4=', 'Content-Length': '0'} 2024-02-01T10:50:14.441 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028129afc1ab06d3e-0065bb7766-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:14.488 DEBUG:boto:b'vclfcn-64obj51000falseobj52024-02-01T10:49:22.402Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:14.488 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-02-01T10:50:14.489 DEBUG:boto:path=/vclfcn-64/obj6 2024-02-01T10:50:14.489 DEBUG:boto:auth_path=/vclfcn-64/obj6 2024-02-01T10:50:14.489 DEBUG:boto:path=/vclfcn-64/obj6?versionId=null 2024-02-01T10:50:14.489 DEBUG:boto:auth_path=/vclfcn-64/obj6?versionId=null 2024-02-01T10:50:14.489 DEBUG:boto:Method: GET 2024-02-01T10:50:14.489 DEBUG:boto:Path: /vclfcn-64/obj6?versionId=null 2024-02-01T10:50:14.489 DEBUG:boto:Data: 2024-02-01T10:50:14.489 DEBUG:boto:Headers: {} 2024-02-01T10:50:14.490 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:14.490 DEBUG:boto:Port: 8001 2024-02-01T10:50:14.490 DEBUG:boto:Params: {} 2024-02-01T10:50:14.490 DEBUG:boto:Token: None 2024-02-01T10:50:14.490 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:14 GMT /vclfcn-64/obj6?versionId=null 2024-02-01T10:50:14.490 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Q0nhQiiI5mBHufURbBR5qV5M/I8= 2024-02-01T10:50:14.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Q0nhQiiI5mBHufURbBR5qV5M/I8=', 'Content-Length': '0'} 2024-02-01T10:50:14.493 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b4327d776ccf3111-0065bb7766-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:14.494 DEBUG:boto:path=/vclfcn-64/obj6 2024-02-01T10:50:14.494 DEBUG:boto:auth_path=/vclfcn-64/obj6 2024-02-01T10:50:14.494 DEBUG:boto:path=/vclfcn-64/obj6?versionId=null 2024-02-01T10:50:14.494 DEBUG:boto:auth_path=/vclfcn-64/obj6?versionId=null 2024-02-01T10:50:14.494 DEBUG:boto:Method: GET 2024-02-01T10:50:14.494 DEBUG:boto:Path: /vclfcn-64/obj6?versionId=null 2024-02-01T10:50:14.494 DEBUG:boto:Data: 2024-02-01T10:50:14.494 DEBUG:boto:Headers: {} 2024-02-01T10:50:14.495 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:14.495 DEBUG:boto:Port: 8000 2024-02-01T10:50:14.495 DEBUG:boto:Params: {} 2024-02-01T10:50:14.495 DEBUG:boto:Token: None 2024-02-01T10:50:14.495 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:14 GMT /vclfcn-64/obj6?versionId=null 2024-02-01T10:50:14.495 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Q0nhQiiI5mBHufURbBR5qV5M/I8= 2024-02-01T10:50:14.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Q0nhQiiI5mBHufURbBR5qV5M/I8=', 'Content-Length': '0'} 2024-02-01T10:50:14.498 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f68611da5de7c7aa-0065bb7766-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:14.499 DEBUG:boto:path=/vclfcn-64/obj6 2024-02-01T10:50:14.499 DEBUG:boto:auth_path=/vclfcn-64/obj6 2024-02-01T10:50:14.499 DEBUG:boto:path=/vclfcn-64/obj6?versionId=null 2024-02-01T10:50:14.499 DEBUG:boto:auth_path=/vclfcn-64/obj6?versionId=null 2024-02-01T10:50:14.499 DEBUG:boto:Method: HEAD 2024-02-01T10:50:14.499 DEBUG:boto:Path: /vclfcn-64/obj6?versionId=null 2024-02-01T10:50:14.499 DEBUG:boto:Data: 2024-02-01T10:50:14.499 DEBUG:boto:Headers: {} 2024-02-01T10:50:14.499 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:14.499 DEBUG:boto:Port: 8001 2024-02-01T10:50:14.499 DEBUG:boto:Params: {} 2024-02-01T10:50:14.500 DEBUG:boto:Token: None 2024-02-01T10:50:14.500 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:14 GMT /vclfcn-64/obj6?versionId=null 2024-02-01T10:50:14.500 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9/tBUXT2WtroDUk1MD65jJdLJz0= 2024-02-01T10:50:14.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9/tBUXT2WtroDUk1MD65jJdLJz0=', 'Content-Length': '0'} 2024-02-01T10:50:14.503 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000eba9bf2bfbe7c5a-0065bb7766-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:14.503 DEBUG:boto:path=/vclfcn-64/obj6 2024-02-01T10:50:14.503 DEBUG:boto:auth_path=/vclfcn-64/obj6 2024-02-01T10:50:14.503 DEBUG:boto:path=/vclfcn-64/obj6?versionId=null 2024-02-01T10:50:14.503 DEBUG:boto:auth_path=/vclfcn-64/obj6?versionId=null 2024-02-01T10:50:14.503 DEBUG:boto:Method: HEAD 2024-02-01T10:50:14.503 DEBUG:boto:Path: /vclfcn-64/obj6?versionId=null 2024-02-01T10:50:14.503 DEBUG:boto:Data: 2024-02-01T10:50:14.504 DEBUG:boto:Headers: {} 2024-02-01T10:50:14.504 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:14.504 DEBUG:boto:Port: 8000 2024-02-01T10:50:14.504 DEBUG:boto:Params: {} 2024-02-01T10:50:14.504 DEBUG:boto:Token: None 2024-02-01T10:50:14.504 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:14 GMT /vclfcn-64/obj6?versionId=null 2024-02-01T10:50:14.504 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9/tBUXT2WtroDUk1MD65jJdLJz0= 2024-02-01T10:50:14.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9/tBUXT2WtroDUk1MD65jJdLJz0=', 'Content-Length': '0'} 2024-02-01T10:50:14.507 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000268a8785876527eb-0065bb7766-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:14.507 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-02-01T10:50:14.508 DEBUG:boto:path=/vclfcn-64/obj6 2024-02-01T10:50:14.508 DEBUG:boto:auth_path=/vclfcn-64/obj6 2024-02-01T10:50:14.508 DEBUG:boto:path=/vclfcn-64/obj6?versionId=null 2024-02-01T10:50:14.508 DEBUG:boto:auth_path=/vclfcn-64/obj6?versionId=null 2024-02-01T10:50:14.508 DEBUG:boto:Method: GET 2024-02-01T10:50:14.508 DEBUG:boto:Path: /vclfcn-64/obj6?versionId=null 2024-02-01T10:50:14.508 DEBUG:boto:Data: 2024-02-01T10:50:14.508 DEBUG:boto:Headers: {} 2024-02-01T10:50:14.508 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:14.508 DEBUG:boto:Port: 8001 2024-02-01T10:50:14.508 DEBUG:boto:Params: {} 2024-02-01T10:50:14.509 DEBUG:boto:Token: None 2024-02-01T10:50:14.509 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:14 GMT /vclfcn-64/obj6?versionId=null 2024-02-01T10:50:14.509 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Q0nhQiiI5mBHufURbBR5qV5M/I8= 2024-02-01T10:50:14.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Q0nhQiiI5mBHufURbBR5qV5M/I8=', 'Content-Length': '0'} 2024-02-01T10:50:14.511 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000092ee79d30148923f-0065bb7766-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:14.556 DEBUG:boto:path=/vclfcn-64/obj6 2024-02-01T10:50:14.556 DEBUG:boto:auth_path=/vclfcn-64/obj6 2024-02-01T10:50:14.556 DEBUG:boto:path=/vclfcn-64/obj6?versionId=null 2024-02-01T10:50:14.557 DEBUG:boto:auth_path=/vclfcn-64/obj6?versionId=null 2024-02-01T10:50:14.557 DEBUG:boto:Method: GET 2024-02-01T10:50:14.557 DEBUG:boto:Path: /vclfcn-64/obj6?versionId=null 2024-02-01T10:50:14.557 DEBUG:boto:Data: 2024-02-01T10:50:14.557 DEBUG:boto:Headers: {} 2024-02-01T10:50:14.557 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:14.557 DEBUG:boto:Port: 8000 2024-02-01T10:50:14.557 DEBUG:boto:Params: {} 2024-02-01T10:50:14.557 DEBUG:boto:Token: None 2024-02-01T10:50:14.558 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:14 GMT /vclfcn-64/obj6?versionId=null 2024-02-01T10:50:14.558 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Q0nhQiiI5mBHufURbBR5qV5M/I8= 2024-02-01T10:50:14.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Q0nhQiiI5mBHufURbBR5qV5M/I8=', 'Content-Length': '0'} 2024-02-01T10:50:14.560 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ce344880dc3a5c4a-0065bb7766-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:14.608 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:14.608 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:14.608 DEBUG:boto:path=/vclfcn-64/?prefix=obj6 2024-02-01T10:50:14.609 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj6 2024-02-01T10:50:14.609 DEBUG:boto:Method: GET 2024-02-01T10:50:14.609 DEBUG:boto:Path: /vclfcn-64/?prefix=obj6 2024-02-01T10:50:14.609 DEBUG:boto:Data: 2024-02-01T10:50:14.609 DEBUG:boto:Headers: {} 2024-02-01T10:50:14.609 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:14.610 DEBUG:boto:Port: 8001 2024-02-01T10:50:14.610 DEBUG:boto:Params: {} 2024-02-01T10:50:14.610 DEBUG:boto:Token: None 2024-02-01T10:50:14.610 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:14 GMT /vclfcn-64/ 2024-02-01T10:50:14.610 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:AxA3arhtwk68gbwpuLrcUfT7xh4= 2024-02-01T10:50:14.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:AxA3arhtwk68gbwpuLrcUfT7xh4=', 'Content-Length': '0'} 2024-02-01T10:50:14.615 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000621866afd79b2a17-0065bb7766-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:14.615 DEBUG:boto:b'vclfcn-64obj61000falseobj62024-02-01T10:49:22.455Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:14.616 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:14.616 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:14.616 DEBUG:boto:path=/vclfcn-64/?prefix=obj6 2024-02-01T10:50:14.616 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj6 2024-02-01T10:50:14.616 DEBUG:boto:Method: GET 2024-02-01T10:50:14.616 DEBUG:boto:Path: /vclfcn-64/?prefix=obj6 2024-02-01T10:50:14.616 DEBUG:boto:Data: 2024-02-01T10:50:14.616 DEBUG:boto:Headers: {} 2024-02-01T10:50:14.616 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:14.616 DEBUG:boto:Port: 8000 2024-02-01T10:50:14.616 DEBUG:boto:Params: {} 2024-02-01T10:50:14.617 DEBUG:boto:Token: None 2024-02-01T10:50:14.617 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:14 GMT /vclfcn-64/ 2024-02-01T10:50:14.617 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:AxA3arhtwk68gbwpuLrcUfT7xh4= 2024-02-01T10:50:14.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:AxA3arhtwk68gbwpuLrcUfT7xh4=', 'Content-Length': '0'} 2024-02-01T10:50:14.622 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f49134197fa5e8de-0065bb7766-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:14.664 DEBUG:boto:b'vclfcn-64obj61000falseobj62024-02-01T10:49:22.455Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:14.665 DEBUG:boto:path=/vclfcn-64/obj6 2024-02-01T10:50:14.665 DEBUG:boto:auth_path=/vclfcn-64/obj6 2024-02-01T10:50:14.665 DEBUG:boto:Method: HEAD 2024-02-01T10:50:14.665 DEBUG:boto:Path: /vclfcn-64/obj6 2024-02-01T10:50:14.665 DEBUG:boto:Data: 2024-02-01T10:50:14.665 DEBUG:boto:Headers: {} 2024-02-01T10:50:14.665 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:14.665 DEBUG:boto:Port: 8001 2024-02-01T10:50:14.665 DEBUG:boto:Params: {} 2024-02-01T10:50:14.666 DEBUG:boto:Token: None 2024-02-01T10:50:14.666 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:14 GMT /vclfcn-64/obj6 2024-02-01T10:50:14.666 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Y8Vuwobn5KtMp3FstVB+BskZKw0= 2024-02-01T10:50:14.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Y8Vuwobn5KtMp3FstVB+BskZKw0=', 'Content-Length': '0'} 2024-02-01T10:50:14.669 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003c591c080b1af382-0065bb7766-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:14.669 DEBUG:boto:path=/vclfcn-64/obj6 2024-02-01T10:50:14.669 DEBUG:boto:auth_path=/vclfcn-64/obj6 2024-02-01T10:50:14.670 DEBUG:boto:Method: HEAD 2024-02-01T10:50:14.670 DEBUG:boto:Path: /vclfcn-64/obj6 2024-02-01T10:50:14.670 DEBUG:boto:Data: 2024-02-01T10:50:14.670 DEBUG:boto:Headers: {} 2024-02-01T10:50:14.670 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:14.670 DEBUG:boto:Port: 8000 2024-02-01T10:50:14.670 DEBUG:boto:Params: {} 2024-02-01T10:50:14.670 DEBUG:boto:Token: None 2024-02-01T10:50:14.671 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:14 GMT /vclfcn-64/obj6 2024-02-01T10:50:14.671 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Y8Vuwobn5KtMp3FstVB+BskZKw0= 2024-02-01T10:50:14.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Y8Vuwobn5KtMp3FstVB+BskZKw0=', 'Content-Length': '0'} 2024-02-01T10:50:14.673 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000038dabd64de262bcf-0065bb7766-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:14.674 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-02-01T10:50:14.674 DEBUG:boto:path=/vclfcn-64/obj6 2024-02-01T10:50:14.674 DEBUG:boto:auth_path=/vclfcn-64/obj6 2024-02-01T10:50:14.674 DEBUG:boto:Method: GET 2024-02-01T10:50:14.674 DEBUG:boto:Path: /vclfcn-64/obj6 2024-02-01T10:50:14.674 DEBUG:boto:Data: 2024-02-01T10:50:14.674 DEBUG:boto:Headers: {} 2024-02-01T10:50:14.675 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:14.675 DEBUG:boto:Port: 8001 2024-02-01T10:50:14.675 DEBUG:boto:Params: {} 2024-02-01T10:50:14.675 DEBUG:boto:Token: None 2024-02-01T10:50:14.675 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:14 GMT /vclfcn-64/obj6 2024-02-01T10:50:14.675 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sySmCgygObEIarHxqJsrIA49/Y0= 2024-02-01T10:50:14.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sySmCgygObEIarHxqJsrIA49/Y0=', 'Content-Length': '0'} 2024-02-01T10:50:14.678 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004e6f5ce6a8bc252d-0065bb7766-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:14.678 DEBUG:boto:path=/vclfcn-64/obj6 2024-02-01T10:50:14.678 DEBUG:boto:auth_path=/vclfcn-64/obj6 2024-02-01T10:50:14.678 DEBUG:boto:Method: GET 2024-02-01T10:50:14.679 DEBUG:boto:Path: /vclfcn-64/obj6 2024-02-01T10:50:14.679 DEBUG:boto:Data: 2024-02-01T10:50:14.679 DEBUG:boto:Headers: {} 2024-02-01T10:50:14.679 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:14.679 DEBUG:boto:Port: 8000 2024-02-01T10:50:14.679 DEBUG:boto:Params: {} 2024-02-01T10:50:14.679 DEBUG:boto:Token: None 2024-02-01T10:50:14.679 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:14 GMT /vclfcn-64/obj6 2024-02-01T10:50:14.679 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:sySmCgygObEIarHxqJsrIA49/Y0= 2024-02-01T10:50:14.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:sySmCgygObEIarHxqJsrIA49/Y0=', 'Content-Length': '0'} 2024-02-01T10:50:14.682 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007db9929fcdae0b74-0065bb7766-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:14.724 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:14.724 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:14.724 DEBUG:boto:path=/vclfcn-64/?prefix=obj6 2024-02-01T10:50:14.724 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj6 2024-02-01T10:50:14.725 DEBUG:boto:Method: GET 2024-02-01T10:50:14.725 DEBUG:boto:Path: /vclfcn-64/?prefix=obj6 2024-02-01T10:50:14.725 DEBUG:boto:Data: 2024-02-01T10:50:14.725 DEBUG:boto:Headers: {} 2024-02-01T10:50:14.725 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:14.725 DEBUG:boto:Port: 8001 2024-02-01T10:50:14.725 DEBUG:boto:Params: {} 2024-02-01T10:50:14.725 DEBUG:boto:Token: None 2024-02-01T10:50:14.725 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:14 GMT /vclfcn-64/ 2024-02-01T10:50:14.725 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:AxA3arhtwk68gbwpuLrcUfT7xh4= 2024-02-01T10:50:14.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:AxA3arhtwk68gbwpuLrcUfT7xh4=', 'Content-Length': '0'} 2024-02-01T10:50:14.730 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010446d9337da37d3-0065bb7766-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:14.730 DEBUG:boto:b'vclfcn-64obj61000falseobj62024-02-01T10:49:22.455Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:14.731 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:14.731 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:14.731 DEBUG:boto:path=/vclfcn-64/?prefix=obj6 2024-02-01T10:50:14.731 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj6 2024-02-01T10:50:14.731 DEBUG:boto:Method: GET 2024-02-01T10:50:14.731 DEBUG:boto:Path: /vclfcn-64/?prefix=obj6 2024-02-01T10:50:14.731 DEBUG:boto:Data: 2024-02-01T10:50:14.731 DEBUG:boto:Headers: {} 2024-02-01T10:50:14.731 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:14.732 DEBUG:boto:Port: 8000 2024-02-01T10:50:14.732 DEBUG:boto:Params: {} 2024-02-01T10:50:14.732 DEBUG:boto:Token: None 2024-02-01T10:50:14.732 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:14 GMT /vclfcn-64/ 2024-02-01T10:50:14.732 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:AxA3arhtwk68gbwpuLrcUfT7xh4= 2024-02-01T10:50:14.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:AxA3arhtwk68gbwpuLrcUfT7xh4=', 'Content-Length': '0'} 2024-02-01T10:50:14.737 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a0bd90ef77f8e426-0065bb7766-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:14.780 DEBUG:boto:b'vclfcn-64obj61000falseobj62024-02-01T10:49:22.455Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:14.780 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-02-01T10:50:14.781 DEBUG:boto:path=/vclfcn-64/obj7 2024-02-01T10:50:14.781 DEBUG:boto:auth_path=/vclfcn-64/obj7 2024-02-01T10:50:14.781 DEBUG:boto:path=/vclfcn-64/obj7?versionId=null 2024-02-01T10:50:14.781 DEBUG:boto:auth_path=/vclfcn-64/obj7?versionId=null 2024-02-01T10:50:14.781 DEBUG:boto:Method: GET 2024-02-01T10:50:14.781 DEBUG:boto:Path: /vclfcn-64/obj7?versionId=null 2024-02-01T10:50:14.781 DEBUG:boto:Data: 2024-02-01T10:50:14.782 DEBUG:boto:Headers: {} 2024-02-01T10:50:14.782 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:14.782 DEBUG:boto:Port: 8001 2024-02-01T10:50:14.782 DEBUG:boto:Params: {} 2024-02-01T10:50:14.782 DEBUG:boto:Token: None 2024-02-01T10:50:14.782 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:14 GMT /vclfcn-64/obj7?versionId=null 2024-02-01T10:50:14.782 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BnmODJljw6vEHEIn0MzmbXynKZs= 2024-02-01T10:50:14.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BnmODJljw6vEHEIn0MzmbXynKZs=', 'Content-Length': '0'} 2024-02-01T10:50:14.786 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000602794f38cab53c2-0065bb7766-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:14.786 DEBUG:boto:path=/vclfcn-64/obj7 2024-02-01T10:50:14.787 DEBUG:boto:auth_path=/vclfcn-64/obj7 2024-02-01T10:50:14.787 DEBUG:boto:path=/vclfcn-64/obj7?versionId=null 2024-02-01T10:50:14.787 DEBUG:boto:auth_path=/vclfcn-64/obj7?versionId=null 2024-02-01T10:50:14.787 DEBUG:boto:Method: GET 2024-02-01T10:50:14.787 DEBUG:boto:Path: /vclfcn-64/obj7?versionId=null 2024-02-01T10:50:14.787 DEBUG:boto:Data: 2024-02-01T10:50:14.788 DEBUG:boto:Headers: {} 2024-02-01T10:50:14.788 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:14.788 DEBUG:boto:Port: 8000 2024-02-01T10:50:14.788 DEBUG:boto:Params: {} 2024-02-01T10:50:14.788 DEBUG:boto:Token: None 2024-02-01T10:50:14.788 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:14 GMT /vclfcn-64/obj7?versionId=null 2024-02-01T10:50:14.789 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BnmODJljw6vEHEIn0MzmbXynKZs= 2024-02-01T10:50:14.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BnmODJljw6vEHEIn0MzmbXynKZs=', 'Content-Length': '0'} 2024-02-01T10:50:14.792 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000082284df7bf8e8117-0065bb7766-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:14.793 DEBUG:boto:path=/vclfcn-64/obj7 2024-02-01T10:50:14.793 DEBUG:boto:auth_path=/vclfcn-64/obj7 2024-02-01T10:50:14.793 DEBUG:boto:path=/vclfcn-64/obj7?versionId=null 2024-02-01T10:50:14.794 DEBUG:boto:auth_path=/vclfcn-64/obj7?versionId=null 2024-02-01T10:50:14.794 DEBUG:boto:Method: HEAD 2024-02-01T10:50:14.794 DEBUG:boto:Path: /vclfcn-64/obj7?versionId=null 2024-02-01T10:50:14.794 DEBUG:boto:Data: 2024-02-01T10:50:14.794 DEBUG:boto:Headers: {} 2024-02-01T10:50:14.794 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:14.794 DEBUG:boto:Port: 8001 2024-02-01T10:50:14.794 DEBUG:boto:Params: {} 2024-02-01T10:50:14.794 DEBUG:boto:Token: None 2024-02-01T10:50:14.794 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:14 GMT /vclfcn-64/obj7?versionId=null 2024-02-01T10:50:14.795 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1sluKIHRDJuO9eazfWRDzxczTvo= 2024-02-01T10:50:14.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1sluKIHRDJuO9eazfWRDzxczTvo=', 'Content-Length': '0'} 2024-02-01T10:50:14.797 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008da7ed7a6112b1c1-0065bb7766-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:14.798 DEBUG:boto:path=/vclfcn-64/obj7 2024-02-01T10:50:14.798 DEBUG:boto:auth_path=/vclfcn-64/obj7 2024-02-01T10:50:14.798 DEBUG:boto:path=/vclfcn-64/obj7?versionId=null 2024-02-01T10:50:14.798 DEBUG:boto:auth_path=/vclfcn-64/obj7?versionId=null 2024-02-01T10:50:14.798 DEBUG:boto:Method: HEAD 2024-02-01T10:50:14.798 DEBUG:boto:Path: /vclfcn-64/obj7?versionId=null 2024-02-01T10:50:14.798 DEBUG:boto:Data: 2024-02-01T10:50:14.798 DEBUG:boto:Headers: {} 2024-02-01T10:50:14.798 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:14.798 DEBUG:boto:Port: 8000 2024-02-01T10:50:14.798 DEBUG:boto:Params: {} 2024-02-01T10:50:14.799 DEBUG:boto:Token: None 2024-02-01T10:50:14.799 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:14 GMT /vclfcn-64/obj7?versionId=null 2024-02-01T10:50:14.799 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1sluKIHRDJuO9eazfWRDzxczTvo= 2024-02-01T10:50:14.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1sluKIHRDJuO9eazfWRDzxczTvo=', 'Content-Length': '0'} 2024-02-01T10:50:14.802 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f6d4d8e5c48833ce-0065bb7766-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:14.802 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-02-01T10:50:14.802 DEBUG:boto:path=/vclfcn-64/obj7 2024-02-01T10:50:14.802 DEBUG:boto:auth_path=/vclfcn-64/obj7 2024-02-01T10:50:14.802 DEBUG:boto:path=/vclfcn-64/obj7?versionId=null 2024-02-01T10:50:14.802 DEBUG:boto:auth_path=/vclfcn-64/obj7?versionId=null 2024-02-01T10:50:14.803 DEBUG:boto:Method: GET 2024-02-01T10:50:14.803 DEBUG:boto:Path: /vclfcn-64/obj7?versionId=null 2024-02-01T10:50:14.803 DEBUG:boto:Data: 2024-02-01T10:50:14.803 DEBUG:boto:Headers: {} 2024-02-01T10:50:14.803 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:14.803 DEBUG:boto:Port: 8001 2024-02-01T10:50:14.803 DEBUG:boto:Params: {} 2024-02-01T10:50:14.803 DEBUG:boto:Token: None 2024-02-01T10:50:14.803 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:14 GMT /vclfcn-64/obj7?versionId=null 2024-02-01T10:50:14.803 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BnmODJljw6vEHEIn0MzmbXynKZs= 2024-02-01T10:50:14.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BnmODJljw6vEHEIn0MzmbXynKZs=', 'Content-Length': '0'} 2024-02-01T10:50:14.807 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000296906dcb6df7f02-0065bb7766-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:14.852 DEBUG:boto:path=/vclfcn-64/obj7 2024-02-01T10:50:14.852 DEBUG:boto:auth_path=/vclfcn-64/obj7 2024-02-01T10:50:14.852 DEBUG:boto:path=/vclfcn-64/obj7?versionId=null 2024-02-01T10:50:14.852 DEBUG:boto:auth_path=/vclfcn-64/obj7?versionId=null 2024-02-01T10:50:14.853 DEBUG:boto:Method: GET 2024-02-01T10:50:14.853 DEBUG:boto:Path: /vclfcn-64/obj7?versionId=null 2024-02-01T10:50:14.853 DEBUG:boto:Data: 2024-02-01T10:50:14.853 DEBUG:boto:Headers: {} 2024-02-01T10:50:14.853 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:14.853 DEBUG:boto:Port: 8000 2024-02-01T10:50:14.854 DEBUG:boto:Params: {} 2024-02-01T10:50:14.854 DEBUG:boto:Token: None 2024-02-01T10:50:14.854 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:14 GMT /vclfcn-64/obj7?versionId=null 2024-02-01T10:50:14.854 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BnmODJljw6vEHEIn0MzmbXynKZs= 2024-02-01T10:50:14.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BnmODJljw6vEHEIn0MzmbXynKZs=', 'Content-Length': '0'} 2024-02-01T10:50:14.858 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008b370ad16874af66-0065bb7766-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:14.904 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:14.905 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:14.905 DEBUG:boto:path=/vclfcn-64/?prefix=obj7 2024-02-01T10:50:14.905 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj7 2024-02-01T10:50:14.906 DEBUG:boto:Method: GET 2024-02-01T10:50:14.906 DEBUG:boto:Path: /vclfcn-64/?prefix=obj7 2024-02-01T10:50:14.906 DEBUG:boto:Data: 2024-02-01T10:50:14.906 DEBUG:boto:Headers: {} 2024-02-01T10:50:14.906 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:14.906 DEBUG:boto:Port: 8001 2024-02-01T10:50:14.907 DEBUG:boto:Params: {} 2024-02-01T10:50:14.907 DEBUG:boto:Token: None 2024-02-01T10:50:14.907 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:14 GMT /vclfcn-64/ 2024-02-01T10:50:14.907 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:AxA3arhtwk68gbwpuLrcUfT7xh4= 2024-02-01T10:50:14.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:AxA3arhtwk68gbwpuLrcUfT7xh4=', 'Content-Length': '0'} 2024-02-01T10:50:14.912 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000243a9fb4f8ac71c7-0065bb7766-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:14.913 DEBUG:boto:b'vclfcn-64obj71000falseobj72024-02-01T10:49:22.471Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:14.913 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:14.914 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:14.914 DEBUG:boto:path=/vclfcn-64/?prefix=obj7 2024-02-01T10:50:14.914 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj7 2024-02-01T10:50:14.914 DEBUG:boto:Method: GET 2024-02-01T10:50:14.914 DEBUG:boto:Path: /vclfcn-64/?prefix=obj7 2024-02-01T10:50:14.914 DEBUG:boto:Data: 2024-02-01T10:50:14.915 DEBUG:boto:Headers: {} 2024-02-01T10:50:14.915 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:14.915 DEBUG:boto:Port: 8000 2024-02-01T10:50:14.915 DEBUG:boto:Params: {} 2024-02-01T10:50:14.915 DEBUG:boto:Token: None 2024-02-01T10:50:14.915 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:14 GMT /vclfcn-64/ 2024-02-01T10:50:14.915 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:AxA3arhtwk68gbwpuLrcUfT7xh4= 2024-02-01T10:50:14.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:AxA3arhtwk68gbwpuLrcUfT7xh4=', 'Content-Length': '0'} 2024-02-01T10:50:14.919 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1e1c7c9f482e20a-0065bb7766-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:14.964 DEBUG:boto:b'vclfcn-64obj71000falseobj72024-02-01T10:49:22.471Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:14.964 DEBUG:boto:path=/vclfcn-64/obj7 2024-02-01T10:50:14.965 DEBUG:boto:auth_path=/vclfcn-64/obj7 2024-02-01T10:50:14.965 DEBUG:boto:Method: HEAD 2024-02-01T10:50:14.965 DEBUG:boto:Path: /vclfcn-64/obj7 2024-02-01T10:50:14.965 DEBUG:boto:Data: 2024-02-01T10:50:14.965 DEBUG:boto:Headers: {} 2024-02-01T10:50:14.965 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:14.965 DEBUG:boto:Port: 8001 2024-02-01T10:50:14.965 DEBUG:boto:Params: {} 2024-02-01T10:50:14.965 DEBUG:boto:Token: None 2024-02-01T10:50:14.966 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:14 GMT /vclfcn-64/obj7 2024-02-01T10:50:14.966 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZoKSAyHXIzEt78QU3niFKh3R1Vc= 2024-02-01T10:50:14.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZoKSAyHXIzEt78QU3niFKh3R1Vc=', 'Content-Length': '0'} 2024-02-01T10:50:14.968 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004ac7b5ea4fd2a6f0-0065bb7766-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:14.969 DEBUG:boto:path=/vclfcn-64/obj7 2024-02-01T10:50:14.969 DEBUG:boto:auth_path=/vclfcn-64/obj7 2024-02-01T10:50:14.969 DEBUG:boto:Method: HEAD 2024-02-01T10:50:14.969 DEBUG:boto:Path: /vclfcn-64/obj7 2024-02-01T10:50:14.969 DEBUG:boto:Data: 2024-02-01T10:50:14.969 DEBUG:boto:Headers: {} 2024-02-01T10:50:14.969 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:14.969 DEBUG:boto:Port: 8000 2024-02-01T10:50:14.969 DEBUG:boto:Params: {} 2024-02-01T10:50:14.970 DEBUG:boto:Token: None 2024-02-01T10:50:14.970 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:14 GMT /vclfcn-64/obj7 2024-02-01T10:50:14.970 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZoKSAyHXIzEt78QU3niFKh3R1Vc= 2024-02-01T10:50:14.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZoKSAyHXIzEt78QU3niFKh3R1Vc=', 'Content-Length': '0'} 2024-02-01T10:50:14.973 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ab1f117adcefbb45-0065bb7766-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:14.973 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-02-01T10:50:14.973 DEBUG:boto:path=/vclfcn-64/obj7 2024-02-01T10:50:14.973 DEBUG:boto:auth_path=/vclfcn-64/obj7 2024-02-01T10:50:14.973 DEBUG:boto:Method: GET 2024-02-01T10:50:14.973 DEBUG:boto:Path: /vclfcn-64/obj7 2024-02-01T10:50:14.974 DEBUG:boto:Data: 2024-02-01T10:50:14.974 DEBUG:boto:Headers: {} 2024-02-01T10:50:14.974 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:14.974 DEBUG:boto:Port: 8001 2024-02-01T10:50:14.974 DEBUG:boto:Params: {} 2024-02-01T10:50:14.974 DEBUG:boto:Token: None 2024-02-01T10:50:14.974 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:14 GMT /vclfcn-64/obj7 2024-02-01T10:50:14.974 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:acM41OfNf3YfN0JF2rDx/srO3fM= 2024-02-01T10:50:14.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:acM41OfNf3YfN0JF2rDx/srO3fM=', 'Content-Length': '0'} 2024-02-01T10:50:14.977 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006bbae7f848bcac6e-0065bb7766-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:14.977 DEBUG:boto:path=/vclfcn-64/obj7 2024-02-01T10:50:14.978 DEBUG:boto:auth_path=/vclfcn-64/obj7 2024-02-01T10:50:14.978 DEBUG:boto:Method: GET 2024-02-01T10:50:14.978 DEBUG:boto:Path: /vclfcn-64/obj7 2024-02-01T10:50:14.978 DEBUG:boto:Data: 2024-02-01T10:50:14.978 DEBUG:boto:Headers: {} 2024-02-01T10:50:14.978 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:14.978 DEBUG:boto:Port: 8000 2024-02-01T10:50:14.978 DEBUG:boto:Params: {} 2024-02-01T10:50:14.978 DEBUG:boto:Token: None 2024-02-01T10:50:14.979 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:14 GMT /vclfcn-64/obj7 2024-02-01T10:50:14.979 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:acM41OfNf3YfN0JF2rDx/srO3fM= 2024-02-01T10:50:14.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:14 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:acM41OfNf3YfN0JF2rDx/srO3fM=', 'Content-Length': '0'} 2024-02-01T10:50:14.981 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008e0d62a2695a07b9-0065bb7766-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:14 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:15.024 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:15.025 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:15.025 DEBUG:boto:path=/vclfcn-64/?prefix=obj7 2024-02-01T10:50:15.025 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj7 2024-02-01T10:50:15.025 DEBUG:boto:Method: GET 2024-02-01T10:50:15.025 DEBUG:boto:Path: /vclfcn-64/?prefix=obj7 2024-02-01T10:50:15.025 DEBUG:boto:Data: 2024-02-01T10:50:15.025 DEBUG:boto:Headers: {} 2024-02-01T10:50:15.025 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:15.025 DEBUG:boto:Port: 8001 2024-02-01T10:50:15.025 DEBUG:boto:Params: {} 2024-02-01T10:50:15.026 DEBUG:boto:Token: None 2024-02-01T10:50:15.026 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:15 GMT /vclfcn-64/ 2024-02-01T10:50:15.026 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/SJBQ2QgdijbWbUJuIDTkWAWObA= 2024-02-01T10:50:15.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/SJBQ2QgdijbWbUJuIDTkWAWObA=', 'Content-Length': '0'} 2024-02-01T10:50:15.030 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bfe6ee7224c856a1-0065bb7767-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:15.030 DEBUG:boto:b'vclfcn-64obj71000falseobj72024-02-01T10:49:22.471Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:15.031 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:15.031 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:15.031 DEBUG:boto:path=/vclfcn-64/?prefix=obj7 2024-02-01T10:50:15.031 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj7 2024-02-01T10:50:15.032 DEBUG:boto:Method: GET 2024-02-01T10:50:15.032 DEBUG:boto:Path: /vclfcn-64/?prefix=obj7 2024-02-01T10:50:15.032 DEBUG:boto:Data: 2024-02-01T10:50:15.032 DEBUG:boto:Headers: {} 2024-02-01T10:50:15.032 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:15.032 DEBUG:boto:Port: 8000 2024-02-01T10:50:15.032 DEBUG:boto:Params: {} 2024-02-01T10:50:15.032 DEBUG:boto:Token: None 2024-02-01T10:50:15.032 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:15 GMT /vclfcn-64/ 2024-02-01T10:50:15.033 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/SJBQ2QgdijbWbUJuIDTkWAWObA= 2024-02-01T10:50:15.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/SJBQ2QgdijbWbUJuIDTkWAWObA=', 'Content-Length': '0'} 2024-02-01T10:50:15.036 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f56609d4268e7d2-0065bb7767-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:15.080 DEBUG:boto:b'vclfcn-64obj71000falseobj72024-02-01T10:49:22.471Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:15.081 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-02-01T10:50:15.081 DEBUG:boto:path=/vclfcn-64/obj8 2024-02-01T10:50:15.081 DEBUG:boto:auth_path=/vclfcn-64/obj8 2024-02-01T10:50:15.081 DEBUG:boto:path=/vclfcn-64/obj8?versionId=null 2024-02-01T10:50:15.081 DEBUG:boto:auth_path=/vclfcn-64/obj8?versionId=null 2024-02-01T10:50:15.082 DEBUG:boto:Method: GET 2024-02-01T10:50:15.082 DEBUG:boto:Path: /vclfcn-64/obj8?versionId=null 2024-02-01T10:50:15.082 DEBUG:boto:Data: 2024-02-01T10:50:15.082 DEBUG:boto:Headers: {} 2024-02-01T10:50:15.082 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:15.082 DEBUG:boto:Port: 8001 2024-02-01T10:50:15.082 DEBUG:boto:Params: {} 2024-02-01T10:50:15.082 DEBUG:boto:Token: None 2024-02-01T10:50:15.082 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:15 GMT /vclfcn-64/obj8?versionId=null 2024-02-01T10:50:15.082 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:h8lHDrNn0yL0m/U4IY6CjL0pkLM= 2024-02-01T10:50:15.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:h8lHDrNn0yL0m/U4IY6CjL0pkLM=', 'Content-Length': '0'} 2024-02-01T10:50:15.085 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003ae64a65dea47332-0065bb7767-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:15.086 DEBUG:boto:path=/vclfcn-64/obj8 2024-02-01T10:50:15.086 DEBUG:boto:auth_path=/vclfcn-64/obj8 2024-02-01T10:50:15.086 DEBUG:boto:path=/vclfcn-64/obj8?versionId=null 2024-02-01T10:50:15.086 DEBUG:boto:auth_path=/vclfcn-64/obj8?versionId=null 2024-02-01T10:50:15.086 DEBUG:boto:Method: GET 2024-02-01T10:50:15.086 DEBUG:boto:Path: /vclfcn-64/obj8?versionId=null 2024-02-01T10:50:15.086 DEBUG:boto:Data: 2024-02-01T10:50:15.086 DEBUG:boto:Headers: {} 2024-02-01T10:50:15.086 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:15.086 DEBUG:boto:Port: 8000 2024-02-01T10:50:15.087 DEBUG:boto:Params: {} 2024-02-01T10:50:15.087 DEBUG:boto:Token: None 2024-02-01T10:50:15.087 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:15 GMT /vclfcn-64/obj8?versionId=null 2024-02-01T10:50:15.087 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:h8lHDrNn0yL0m/U4IY6CjL0pkLM= 2024-02-01T10:50:15.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:h8lHDrNn0yL0m/U4IY6CjL0pkLM=', 'Content-Length': '0'} 2024-02-01T10:50:15.090 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000022a040cb037b427e-0065bb7767-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:15.090 DEBUG:boto:path=/vclfcn-64/obj8 2024-02-01T10:50:15.090 DEBUG:boto:auth_path=/vclfcn-64/obj8 2024-02-01T10:50:15.090 DEBUG:boto:path=/vclfcn-64/obj8?versionId=null 2024-02-01T10:50:15.091 DEBUG:boto:auth_path=/vclfcn-64/obj8?versionId=null 2024-02-01T10:50:15.091 DEBUG:boto:Method: HEAD 2024-02-01T10:50:15.091 DEBUG:boto:Path: /vclfcn-64/obj8?versionId=null 2024-02-01T10:50:15.091 DEBUG:boto:Data: 2024-02-01T10:50:15.091 DEBUG:boto:Headers: {} 2024-02-01T10:50:15.092 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:15.092 DEBUG:boto:Port: 8001 2024-02-01T10:50:15.092 DEBUG:boto:Params: {} 2024-02-01T10:50:15.092 DEBUG:boto:Token: None 2024-02-01T10:50:15.092 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:15 GMT /vclfcn-64/obj8?versionId=null 2024-02-01T10:50:15.093 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Qid9ZNunfDVusiKRLfaaJLazYjc= 2024-02-01T10:50:15.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Qid9ZNunfDVusiKRLfaaJLazYjc=', 'Content-Length': '0'} 2024-02-01T10:50:15.096 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004207b12a05e2fe77-0065bb7767-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:15.097 DEBUG:boto:path=/vclfcn-64/obj8 2024-02-01T10:50:15.097 DEBUG:boto:auth_path=/vclfcn-64/obj8 2024-02-01T10:50:15.097 DEBUG:boto:path=/vclfcn-64/obj8?versionId=null 2024-02-01T10:50:15.097 DEBUG:boto:auth_path=/vclfcn-64/obj8?versionId=null 2024-02-01T10:50:15.097 DEBUG:boto:Method: HEAD 2024-02-01T10:50:15.098 DEBUG:boto:Path: /vclfcn-64/obj8?versionId=null 2024-02-01T10:50:15.098 DEBUG:boto:Data: 2024-02-01T10:50:15.098 DEBUG:boto:Headers: {} 2024-02-01T10:50:15.098 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:15.098 DEBUG:boto:Port: 8000 2024-02-01T10:50:15.099 DEBUG:boto:Params: {} 2024-02-01T10:50:15.099 DEBUG:boto:Token: None 2024-02-01T10:50:15.099 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:15 GMT /vclfcn-64/obj8?versionId=null 2024-02-01T10:50:15.099 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Qid9ZNunfDVusiKRLfaaJLazYjc= 2024-02-01T10:50:15.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Qid9ZNunfDVusiKRLfaaJLazYjc=', 'Content-Length': '0'} 2024-02-01T10:50:15.103 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003df754fcf9fccafd-0065bb7767-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:15.103 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-02-01T10:50:15.104 DEBUG:boto:path=/vclfcn-64/obj8 2024-02-01T10:50:15.104 DEBUG:boto:auth_path=/vclfcn-64/obj8 2024-02-01T10:50:15.104 DEBUG:boto:path=/vclfcn-64/obj8?versionId=null 2024-02-01T10:50:15.104 DEBUG:boto:auth_path=/vclfcn-64/obj8?versionId=null 2024-02-01T10:50:15.104 DEBUG:boto:Method: GET 2024-02-01T10:50:15.105 DEBUG:boto:Path: /vclfcn-64/obj8?versionId=null 2024-02-01T10:50:15.105 DEBUG:boto:Data: 2024-02-01T10:50:15.105 DEBUG:boto:Headers: {} 2024-02-01T10:50:15.105 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:15.105 DEBUG:boto:Port: 8001 2024-02-01T10:50:15.106 DEBUG:boto:Params: {} 2024-02-01T10:50:15.106 DEBUG:boto:Token: None 2024-02-01T10:50:15.106 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:15 GMT /vclfcn-64/obj8?versionId=null 2024-02-01T10:50:15.106 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:h8lHDrNn0yL0m/U4IY6CjL0pkLM= 2024-02-01T10:50:15.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:h8lHDrNn0yL0m/U4IY6CjL0pkLM=', 'Content-Length': '0'} 2024-02-01T10:50:15.110 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000849c3652c4f63ee9-0065bb7767-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:15.152 DEBUG:boto:path=/vclfcn-64/obj8 2024-02-01T10:50:15.152 DEBUG:boto:auth_path=/vclfcn-64/obj8 2024-02-01T10:50:15.152 DEBUG:boto:path=/vclfcn-64/obj8?versionId=null 2024-02-01T10:50:15.153 DEBUG:boto:auth_path=/vclfcn-64/obj8?versionId=null 2024-02-01T10:50:15.153 DEBUG:boto:Method: GET 2024-02-01T10:50:15.153 DEBUG:boto:Path: /vclfcn-64/obj8?versionId=null 2024-02-01T10:50:15.153 DEBUG:boto:Data: 2024-02-01T10:50:15.153 DEBUG:boto:Headers: {} 2024-02-01T10:50:15.153 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:15.153 DEBUG:boto:Port: 8000 2024-02-01T10:50:15.153 DEBUG:boto:Params: {} 2024-02-01T10:50:15.153 DEBUG:boto:Token: None 2024-02-01T10:50:15.154 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:15 GMT /vclfcn-64/obj8?versionId=null 2024-02-01T10:50:15.154 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:h8lHDrNn0yL0m/U4IY6CjL0pkLM= 2024-02-01T10:50:15.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:h8lHDrNn0yL0m/U4IY6CjL0pkLM=', 'Content-Length': '0'} 2024-02-01T10:50:15.157 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f1d5a5c53b99c5c9-0065bb7767-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:15.200 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:15.200 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:15.200 DEBUG:boto:path=/vclfcn-64/?prefix=obj8 2024-02-01T10:50:15.201 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj8 2024-02-01T10:50:15.201 DEBUG:boto:Method: GET 2024-02-01T10:50:15.201 DEBUG:boto:Path: /vclfcn-64/?prefix=obj8 2024-02-01T10:50:15.201 DEBUG:boto:Data: 2024-02-01T10:50:15.201 DEBUG:boto:Headers: {} 2024-02-01T10:50:15.201 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:15.201 DEBUG:boto:Port: 8001 2024-02-01T10:50:15.201 DEBUG:boto:Params: {} 2024-02-01T10:50:15.201 DEBUG:boto:Token: None 2024-02-01T10:50:15.202 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:15 GMT /vclfcn-64/ 2024-02-01T10:50:15.202 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/SJBQ2QgdijbWbUJuIDTkWAWObA= 2024-02-01T10:50:15.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/SJBQ2QgdijbWbUJuIDTkWAWObA=', 'Content-Length': '0'} 2024-02-01T10:50:15.206 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b4b0e736c17f797-0065bb7767-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:15.207 DEBUG:boto:b'vclfcn-64obj81000falseobj82024-02-01T10:49:22.519Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:15.208 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:15.208 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:15.208 DEBUG:boto:path=/vclfcn-64/?prefix=obj8 2024-02-01T10:50:15.208 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj8 2024-02-01T10:50:15.208 DEBUG:boto:Method: GET 2024-02-01T10:50:15.208 DEBUG:boto:Path: /vclfcn-64/?prefix=obj8 2024-02-01T10:50:15.208 DEBUG:boto:Data: 2024-02-01T10:50:15.208 DEBUG:boto:Headers: {} 2024-02-01T10:50:15.208 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:15.209 DEBUG:boto:Port: 8000 2024-02-01T10:50:15.209 DEBUG:boto:Params: {} 2024-02-01T10:50:15.209 DEBUG:boto:Token: None 2024-02-01T10:50:15.209 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:15 GMT /vclfcn-64/ 2024-02-01T10:50:15.209 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/SJBQ2QgdijbWbUJuIDTkWAWObA= 2024-02-01T10:50:15.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/SJBQ2QgdijbWbUJuIDTkWAWObA=', 'Content-Length': '0'} 2024-02-01T10:50:15.213 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5130797f466b70b-0065bb7767-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:15.256 DEBUG:boto:b'vclfcn-64obj81000falseobj82024-02-01T10:49:22.519Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:15.257 DEBUG:boto:path=/vclfcn-64/obj8 2024-02-01T10:50:15.257 DEBUG:boto:auth_path=/vclfcn-64/obj8 2024-02-01T10:50:15.257 DEBUG:boto:Method: HEAD 2024-02-01T10:50:15.257 DEBUG:boto:Path: /vclfcn-64/obj8 2024-02-01T10:50:15.257 DEBUG:boto:Data: 2024-02-01T10:50:15.257 DEBUG:boto:Headers: {} 2024-02-01T10:50:15.257 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:15.257 DEBUG:boto:Port: 8001 2024-02-01T10:50:15.257 DEBUG:boto:Params: {} 2024-02-01T10:50:15.258 DEBUG:boto:Token: None 2024-02-01T10:50:15.258 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:15 GMT /vclfcn-64/obj8 2024-02-01T10:50:15.258 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:havvfz1bjlcdBrdqniCwK5dvQ34= 2024-02-01T10:50:15.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:havvfz1bjlcdBrdqniCwK5dvQ34=', 'Content-Length': '0'} 2024-02-01T10:50:15.261 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d36989731ddafd95-0065bb7767-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:15.261 DEBUG:boto:path=/vclfcn-64/obj8 2024-02-01T10:50:15.261 DEBUG:boto:auth_path=/vclfcn-64/obj8 2024-02-01T10:50:15.261 DEBUG:boto:Method: HEAD 2024-02-01T10:50:15.261 DEBUG:boto:Path: /vclfcn-64/obj8 2024-02-01T10:50:15.261 DEBUG:boto:Data: 2024-02-01T10:50:15.261 DEBUG:boto:Headers: {} 2024-02-01T10:50:15.261 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:15.262 DEBUG:boto:Port: 8000 2024-02-01T10:50:15.262 DEBUG:boto:Params: {} 2024-02-01T10:50:15.262 DEBUG:boto:Token: None 2024-02-01T10:50:15.262 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:15 GMT /vclfcn-64/obj8 2024-02-01T10:50:15.262 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:havvfz1bjlcdBrdqniCwK5dvQ34= 2024-02-01T10:50:15.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:havvfz1bjlcdBrdqniCwK5dvQ34=', 'Content-Length': '0'} 2024-02-01T10:50:15.265 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d49cd9eab8dda981-0065bb7767-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:15.265 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-02-01T10:50:15.265 DEBUG:boto:path=/vclfcn-64/obj8 2024-02-01T10:50:15.265 DEBUG:boto:auth_path=/vclfcn-64/obj8 2024-02-01T10:50:15.266 DEBUG:boto:Method: GET 2024-02-01T10:50:15.266 DEBUG:boto:Path: /vclfcn-64/obj8 2024-02-01T10:50:15.266 DEBUG:boto:Data: 2024-02-01T10:50:15.266 DEBUG:boto:Headers: {} 2024-02-01T10:50:15.266 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:15.266 DEBUG:boto:Port: 8001 2024-02-01T10:50:15.266 DEBUG:boto:Params: {} 2024-02-01T10:50:15.266 DEBUG:boto:Token: None 2024-02-01T10:50:15.266 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:15 GMT /vclfcn-64/obj8 2024-02-01T10:50:15.266 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:XRTMTw4jofQZDm8l5y7Ix0EWEjQ= 2024-02-01T10:50:15.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:XRTMTw4jofQZDm8l5y7Ix0EWEjQ=', 'Content-Length': '0'} 2024-02-01T10:50:15.270 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a18982ea0fc04886-0065bb7767-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:15.270 DEBUG:boto:path=/vclfcn-64/obj8 2024-02-01T10:50:15.270 DEBUG:boto:auth_path=/vclfcn-64/obj8 2024-02-01T10:50:15.270 DEBUG:boto:Method: GET 2024-02-01T10:50:15.270 DEBUG:boto:Path: /vclfcn-64/obj8 2024-02-01T10:50:15.270 DEBUG:boto:Data: 2024-02-01T10:50:15.270 DEBUG:boto:Headers: {} 2024-02-01T10:50:15.270 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:15.271 DEBUG:boto:Port: 8000 2024-02-01T10:50:15.271 DEBUG:boto:Params: {} 2024-02-01T10:50:15.271 DEBUG:boto:Token: None 2024-02-01T10:50:15.271 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:15 GMT /vclfcn-64/obj8 2024-02-01T10:50:15.271 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:XRTMTw4jofQZDm8l5y7Ix0EWEjQ= 2024-02-01T10:50:15.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:XRTMTw4jofQZDm8l5y7Ix0EWEjQ=', 'Content-Length': '0'} 2024-02-01T10:50:15.274 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000001f090dca9aaf8e0-0065bb7767-5765-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:15.316 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:15.316 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:15.316 DEBUG:boto:path=/vclfcn-64/?prefix=obj8 2024-02-01T10:50:15.316 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj8 2024-02-01T10:50:15.317 DEBUG:boto:Method: GET 2024-02-01T10:50:15.317 DEBUG:boto:Path: /vclfcn-64/?prefix=obj8 2024-02-01T10:50:15.317 DEBUG:boto:Data: 2024-02-01T10:50:15.317 DEBUG:boto:Headers: {} 2024-02-01T10:50:15.317 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:15.317 DEBUG:boto:Port: 8001 2024-02-01T10:50:15.317 DEBUG:boto:Params: {} 2024-02-01T10:50:15.317 DEBUG:boto:Token: None 2024-02-01T10:50:15.318 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:15 GMT /vclfcn-64/ 2024-02-01T10:50:15.318 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/SJBQ2QgdijbWbUJuIDTkWAWObA= 2024-02-01T10:50:15.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/SJBQ2QgdijbWbUJuIDTkWAWObA=', 'Content-Length': '0'} 2024-02-01T10:50:15.322 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040264739a1844b84-0065bb7767-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:15.322 DEBUG:boto:b'vclfcn-64obj81000falseobj82024-02-01T10:49:22.519Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:15.323 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:15.323 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:15.323 DEBUG:boto:path=/vclfcn-64/?prefix=obj8 2024-02-01T10:50:15.323 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj8 2024-02-01T10:50:15.324 DEBUG:boto:Method: GET 2024-02-01T10:50:15.324 DEBUG:boto:Path: /vclfcn-64/?prefix=obj8 2024-02-01T10:50:15.324 DEBUG:boto:Data: 2024-02-01T10:50:15.324 DEBUG:boto:Headers: {} 2024-02-01T10:50:15.324 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:15.324 DEBUG:boto:Port: 8000 2024-02-01T10:50:15.324 DEBUG:boto:Params: {} 2024-02-01T10:50:15.325 DEBUG:boto:Token: None 2024-02-01T10:50:15.325 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:15 GMT /vclfcn-64/ 2024-02-01T10:50:15.325 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/SJBQ2QgdijbWbUJuIDTkWAWObA= 2024-02-01T10:50:15.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/SJBQ2QgdijbWbUJuIDTkWAWObA=', 'Content-Length': '0'} 2024-02-01T10:50:15.330 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ade1893c2c6e1ee3-0065bb7767-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:15.372 DEBUG:boto:b'vclfcn-64obj81000falseobj82024-02-01T10:49:22.519Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:15.373 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-64 zones={test-zone3, test-zone1} 2024-02-01T10:50:15.373 INFO:tasks.rgw_multi.zone_rados:comparing bucket=vclfcn-64 zones={test-zone3, test-zone2} 2024-02-01T10:50:15.373 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:15.373 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:15.373 DEBUG:boto:Method: HEAD 2024-02-01T10:50:15.373 DEBUG:boto:Path: /vclfcn-64/ 2024-02-01T10:50:15.373 DEBUG:boto:Data: 2024-02-01T10:50:15.373 DEBUG:boto:Headers: {} 2024-02-01T10:50:15.373 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:15.373 DEBUG:boto:Port: 8001 2024-02-01T10:50:15.374 DEBUG:boto:Params: {} 2024-02-01T10:50:15.374 DEBUG:boto:Token: None 2024-02-01T10:50:15.374 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:15 GMT /vclfcn-64/ 2024-02-01T10:50:15.374 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:bR0PUU82d25w39OdMYA6IMeYvCg= 2024-02-01T10:50:15.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:bR0PUU82d25w39OdMYA6IMeYvCg=', 'Content-Length': '0'} 2024-02-01T10:50:15.379 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000098c099b7b657ff6a-0065bb7767-5762-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:15.379 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:15.379 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:15.379 DEBUG:boto:Method: HEAD 2024-02-01T10:50:15.379 DEBUG:boto:Path: /vclfcn-64/ 2024-02-01T10:50:15.379 DEBUG:boto:Data: 2024-02-01T10:50:15.379 DEBUG:boto:Headers: {} 2024-02-01T10:50:15.379 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:15.379 DEBUG:boto:Port: 8000 2024-02-01T10:50:15.379 DEBUG:boto:Params: {} 2024-02-01T10:50:15.380 DEBUG:boto:Token: None 2024-02-01T10:50:15.380 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:15 GMT /vclfcn-64/ 2024-02-01T10:50:15.380 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:bR0PUU82d25w39OdMYA6IMeYvCg= 2024-02-01T10:50:15.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:bR0PUU82d25w39OdMYA6IMeYvCg=', 'Content-Length': '0'} 2024-02-01T10:50:15.385 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fab257d33a8270ec-0065bb7767-5350-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:15.385 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-02-01T10:50:15.385 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:15.385 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:15.385 DEBUG:boto:path=/vclfcn-64/?versions&max-keys=999 2024-02-01T10:50:15.385 DEBUG:boto:auth_path=/vclfcn-64/?versions&max-keys=999 2024-02-01T10:50:15.385 DEBUG:boto:Method: GET 2024-02-01T10:50:15.385 DEBUG:boto:Path: /vclfcn-64/?versions&max-keys=999 2024-02-01T10:50:15.386 DEBUG:boto:Data: 2024-02-01T10:50:15.386 DEBUG:boto:Headers: {} 2024-02-01T10:50:15.386 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:15.386 DEBUG:boto:Port: 8001 2024-02-01T10:50:15.386 DEBUG:boto:Params: {} 2024-02-01T10:50:15.386 DEBUG:boto:Token: None 2024-02-01T10:50:15.386 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:15 GMT /vclfcn-64/?versions 2024-02-01T10:50:15.386 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:4QuvydvLaroXMwObk92zRTMl/Io= 2024-02-01T10:50:15.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:4QuvydvLaroXMwObk92zRTMl/Io=', 'Content-Length': '0'} 2024-02-01T10:50:15.391 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb0b2f1fca5cd2e4-0065bb7767-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:15.391 DEBUG:boto:b'vclfcn-64999falseobj1nulltrue2024-02-01T10:48:26.595Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:48:26.615Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:48:26.634Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:48:26.689Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-02-01T10:49:22.402Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-02-01T10:49:22.455Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-02-01T10:49:22.471Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-02-01T10:49:22.519Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:15.395 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-02-01T10:50:15.395 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-02-01T10:50:15.395 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-02-01T10:50:15.395 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-02-01T10:50:15.395 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-02-01T10:50:15.395 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-02-01T10:50:15.396 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-02-01T10:50:15.396 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-02-01T10:50:15.396 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-02-01T10:50:15.396 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:15.396 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:15.396 DEBUG:boto:path=/vclfcn-64/?versions&max-keys=999 2024-02-01T10:50:15.396 DEBUG:boto:auth_path=/vclfcn-64/?versions&max-keys=999 2024-02-01T10:50:15.396 DEBUG:boto:Method: GET 2024-02-01T10:50:15.396 DEBUG:boto:Path: /vclfcn-64/?versions&max-keys=999 2024-02-01T10:50:15.397 DEBUG:boto:Data: 2024-02-01T10:50:15.397 DEBUG:boto:Headers: {} 2024-02-01T10:50:15.397 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:15.397 DEBUG:boto:Port: 8000 2024-02-01T10:50:15.397 DEBUG:boto:Params: {} 2024-02-01T10:50:15.397 DEBUG:boto:Token: None 2024-02-01T10:50:15.397 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:15 GMT /vclfcn-64/?versions 2024-02-01T10:50:15.397 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:4QuvydvLaroXMwObk92zRTMl/Io= 2024-02-01T10:50:15.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:4QuvydvLaroXMwObk92zRTMl/Io=', 'Content-Length': '0'} 2024-02-01T10:50:15.402 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000904a4932b8f05fa4-0065bb7767-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:15.402 DEBUG:boto:b'vclfcn-64999falseobj1nulltrue2024-02-01T10:48:26.595Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:48:26.615Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:48:26.634Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:48:26.689Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-02-01T10:49:22.402Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-02-01T10:49:22.455Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-02-01T10:49:22.471Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-02-01T10:49:22.519Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:15.403 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-02-01T10:50:15.403 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-02-01T10:50:15.403 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-02-01T10:50:15.403 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-02-01T10:50:15.403 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-02-01T10:50:15.403 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-02-01T10:50:15.404 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-02-01T10:50:15.404 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-02-01T10:50:15.404 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:15.404 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:15.404 DEBUG:boto:path=/vclfcn-64/?versions&max-keys=999 2024-02-01T10:50:15.404 DEBUG:boto:auth_path=/vclfcn-64/?versions&max-keys=999 2024-02-01T10:50:15.404 DEBUG:boto:Method: GET 2024-02-01T10:50:15.404 DEBUG:boto:Path: /vclfcn-64/?versions&max-keys=999 2024-02-01T10:50:15.404 DEBUG:boto:Data: 2024-02-01T10:50:15.404 DEBUG:boto:Headers: {} 2024-02-01T10:50:15.404 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:15.405 DEBUG:boto:Port: 8001 2024-02-01T10:50:15.405 DEBUG:boto:Params: {} 2024-02-01T10:50:15.405 DEBUG:boto:Token: None 2024-02-01T10:50:15.405 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:15 GMT /vclfcn-64/?versions 2024-02-01T10:50:15.405 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:4QuvydvLaroXMwObk92zRTMl/Io= 2024-02-01T10:50:15.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:4QuvydvLaroXMwObk92zRTMl/Io=', 'Content-Length': '0'} 2024-02-01T10:50:15.410 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d455fcda614250f3-0065bb7767-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:15.411 DEBUG:boto:b'vclfcn-64999falseobj1nulltrue2024-02-01T10:48:26.595Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:48:26.615Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:48:26.634Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:48:26.689Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-02-01T10:49:22.402Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-02-01T10:49:22.455Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-02-01T10:49:22.471Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-02-01T10:49:22.519Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:15.411 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:15.412 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:15.412 DEBUG:boto:path=/vclfcn-64/?versions&max-keys=999 2024-02-01T10:50:15.412 DEBUG:boto:auth_path=/vclfcn-64/?versions&max-keys=999 2024-02-01T10:50:15.412 DEBUG:boto:Method: GET 2024-02-01T10:50:15.412 DEBUG:boto:Path: /vclfcn-64/?versions&max-keys=999 2024-02-01T10:50:15.412 DEBUG:boto:Data: 2024-02-01T10:50:15.412 DEBUG:boto:Headers: {} 2024-02-01T10:50:15.413 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:15.413 DEBUG:boto:Port: 8000 2024-02-01T10:50:15.413 DEBUG:boto:Params: {} 2024-02-01T10:50:15.413 DEBUG:boto:Token: None 2024-02-01T10:50:15.413 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:15 GMT /vclfcn-64/?versions 2024-02-01T10:50:15.413 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:4QuvydvLaroXMwObk92zRTMl/Io= 2024-02-01T10:50:15.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:4QuvydvLaroXMwObk92zRTMl/Io=', 'Content-Length': '0'} 2024-02-01T10:50:15.418 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009cb09676a80ccfaa-0065bb7767-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:15.418 DEBUG:boto:b'vclfcn-64999falseobj1nulltrue2024-02-01T10:48:26.595Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-02-01T10:48:26.615Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-02-01T10:48:26.634Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-02-01T10:48:26.689Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-02-01T10:49:22.402Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-02-01T10:49:22.455Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-02-01T10:49:22.471Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-02-01T10:49:22.519Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:15.419 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:50:15.419 DEBUG:boto:path=/vclfcn-64/obj1 2024-02-01T10:50:15.419 DEBUG:boto:auth_path=/vclfcn-64/obj1 2024-02-01T10:50:15.419 DEBUG:boto:path=/vclfcn-64/obj1?versionId=null 2024-02-01T10:50:15.419 DEBUG:boto:auth_path=/vclfcn-64/obj1?versionId=null 2024-02-01T10:50:15.419 DEBUG:boto:Method: GET 2024-02-01T10:50:15.419 DEBUG:boto:Path: /vclfcn-64/obj1?versionId=null 2024-02-01T10:50:15.419 DEBUG:boto:Data: 2024-02-01T10:50:15.419 DEBUG:boto:Headers: {} 2024-02-01T10:50:15.420 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:15.420 DEBUG:boto:Port: 8001 2024-02-01T10:50:15.420 DEBUG:boto:Params: {} 2024-02-01T10:50:15.420 DEBUG:boto:Token: None 2024-02-01T10:50:15.420 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:15 GMT /vclfcn-64/obj1?versionId=null 2024-02-01T10:50:15.420 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:55xRY0N+7CIJ3BxY7Jd0R//b1gQ= 2024-02-01T10:50:15.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:55xRY0N+7CIJ3BxY7Jd0R//b1gQ=', 'Content-Length': '0'} 2024-02-01T10:50:15.424 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000601194624404b51c-0065bb7767-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:15.464 DEBUG:boto:path=/vclfcn-64/obj1 2024-02-01T10:50:15.465 DEBUG:boto:auth_path=/vclfcn-64/obj1 2024-02-01T10:50:15.465 DEBUG:boto:path=/vclfcn-64/obj1?versionId=null 2024-02-01T10:50:15.465 DEBUG:boto:auth_path=/vclfcn-64/obj1?versionId=null 2024-02-01T10:50:15.465 DEBUG:boto:Method: GET 2024-02-01T10:50:15.465 DEBUG:boto:Path: /vclfcn-64/obj1?versionId=null 2024-02-01T10:50:15.465 DEBUG:boto:Data: 2024-02-01T10:50:15.465 DEBUG:boto:Headers: {} 2024-02-01T10:50:15.465 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:15.465 DEBUG:boto:Port: 8000 2024-02-01T10:50:15.465 DEBUG:boto:Params: {} 2024-02-01T10:50:15.466 DEBUG:boto:Token: None 2024-02-01T10:50:15.466 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:15 GMT /vclfcn-64/obj1?versionId=null 2024-02-01T10:50:15.466 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:55xRY0N+7CIJ3BxY7Jd0R//b1gQ= 2024-02-01T10:50:15.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:55xRY0N+7CIJ3BxY7Jd0R//b1gQ=', 'Content-Length': '0'} 2024-02-01T10:50:15.469 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003cd0478ba8d21425-0065bb7767-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:15.469 DEBUG:boto:path=/vclfcn-64/obj1 2024-02-01T10:50:15.469 DEBUG:boto:auth_path=/vclfcn-64/obj1 2024-02-01T10:50:15.470 DEBUG:boto:path=/vclfcn-64/obj1?versionId=null 2024-02-01T10:50:15.470 DEBUG:boto:auth_path=/vclfcn-64/obj1?versionId=null 2024-02-01T10:50:15.470 DEBUG:boto:Method: HEAD 2024-02-01T10:50:15.470 DEBUG:boto:Path: /vclfcn-64/obj1?versionId=null 2024-02-01T10:50:15.470 DEBUG:boto:Data: 2024-02-01T10:50:15.470 DEBUG:boto:Headers: {} 2024-02-01T10:50:15.470 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:15.470 DEBUG:boto:Port: 8001 2024-02-01T10:50:15.470 DEBUG:boto:Params: {} 2024-02-01T10:50:15.470 DEBUG:boto:Token: None 2024-02-01T10:50:15.471 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:15 GMT /vclfcn-64/obj1?versionId=null 2024-02-01T10:50:15.471 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:rc75Q+9e12+BW5/Il83Eu9zVlMY= 2024-02-01T10:50:15.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:rc75Q+9e12+BW5/Il83Eu9zVlMY=', 'Content-Length': '0'} 2024-02-01T10:50:15.474 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006484a2e46db37140-0065bb7767-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:15.474 DEBUG:boto:path=/vclfcn-64/obj1 2024-02-01T10:50:15.474 DEBUG:boto:auth_path=/vclfcn-64/obj1 2024-02-01T10:50:15.474 DEBUG:boto:path=/vclfcn-64/obj1?versionId=null 2024-02-01T10:50:15.474 DEBUG:boto:auth_path=/vclfcn-64/obj1?versionId=null 2024-02-01T10:50:15.474 DEBUG:boto:Method: HEAD 2024-02-01T10:50:15.474 DEBUG:boto:Path: /vclfcn-64/obj1?versionId=null 2024-02-01T10:50:15.474 DEBUG:boto:Data: 2024-02-01T10:50:15.474 DEBUG:boto:Headers: {} 2024-02-01T10:50:15.475 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:15.475 DEBUG:boto:Port: 8000 2024-02-01T10:50:15.475 DEBUG:boto:Params: {} 2024-02-01T10:50:15.475 DEBUG:boto:Token: None 2024-02-01T10:50:15.475 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:15 GMT /vclfcn-64/obj1?versionId=null 2024-02-01T10:50:15.475 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:rc75Q+9e12+BW5/Il83Eu9zVlMY= 2024-02-01T10:50:15.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:rc75Q+9e12+BW5/Il83Eu9zVlMY=', 'Content-Length': '0'} 2024-02-01T10:50:15.478 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000080030c4010289959-0065bb7767-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:15.478 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:50:15.479 DEBUG:boto:path=/vclfcn-64/obj1 2024-02-01T10:50:15.479 DEBUG:boto:auth_path=/vclfcn-64/obj1 2024-02-01T10:50:15.479 DEBUG:boto:path=/vclfcn-64/obj1?versionId=null 2024-02-01T10:50:15.479 DEBUG:boto:auth_path=/vclfcn-64/obj1?versionId=null 2024-02-01T10:50:15.479 DEBUG:boto:Method: GET 2024-02-01T10:50:15.479 DEBUG:boto:Path: /vclfcn-64/obj1?versionId=null 2024-02-01T10:50:15.479 DEBUG:boto:Data: 2024-02-01T10:50:15.479 DEBUG:boto:Headers: {} 2024-02-01T10:50:15.479 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:15.480 DEBUG:boto:Port: 8001 2024-02-01T10:50:15.480 DEBUG:boto:Params: {} 2024-02-01T10:50:15.480 DEBUG:boto:Token: None 2024-02-01T10:50:15.480 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:15 GMT /vclfcn-64/obj1?versionId=null 2024-02-01T10:50:15.480 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:55xRY0N+7CIJ3BxY7Jd0R//b1gQ= 2024-02-01T10:50:15.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:55xRY0N+7CIJ3BxY7Jd0R//b1gQ=', 'Content-Length': '0'} 2024-02-01T10:50:15.483 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b5b7ef9326c64113-0065bb7767-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:15.484 DEBUG:boto:path=/vclfcn-64/obj1 2024-02-01T10:50:15.484 DEBUG:boto:auth_path=/vclfcn-64/obj1 2024-02-01T10:50:15.484 DEBUG:boto:path=/vclfcn-64/obj1?versionId=null 2024-02-01T10:50:15.485 DEBUG:boto:auth_path=/vclfcn-64/obj1?versionId=null 2024-02-01T10:50:15.485 DEBUG:boto:Method: GET 2024-02-01T10:50:15.485 DEBUG:boto:Path: /vclfcn-64/obj1?versionId=null 2024-02-01T10:50:15.485 DEBUG:boto:Data: 2024-02-01T10:50:15.485 DEBUG:boto:Headers: {} 2024-02-01T10:50:15.486 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:15.486 DEBUG:boto:Port: 8000 2024-02-01T10:50:15.486 DEBUG:boto:Params: {} 2024-02-01T10:50:15.486 DEBUG:boto:Token: None 2024-02-01T10:50:15.486 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:15 GMT /vclfcn-64/obj1?versionId=null 2024-02-01T10:50:15.487 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:55xRY0N+7CIJ3BxY7Jd0R//b1gQ= 2024-02-01T10:50:15.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:55xRY0N+7CIJ3BxY7Jd0R//b1gQ=', 'Content-Length': '0'} 2024-02-01T10:50:15.490 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e4729dd52fe95b08-0065bb7767-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:15.532 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:15.533 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:15.533 DEBUG:boto:path=/vclfcn-64/?prefix=obj1 2024-02-01T10:50:15.533 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj1 2024-02-01T10:50:15.533 DEBUG:boto:Method: GET 2024-02-01T10:50:15.533 DEBUG:boto:Path: /vclfcn-64/?prefix=obj1 2024-02-01T10:50:15.533 DEBUG:boto:Data: 2024-02-01T10:50:15.533 DEBUG:boto:Headers: {} 2024-02-01T10:50:15.533 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:15.533 DEBUG:boto:Port: 8001 2024-02-01T10:50:15.534 DEBUG:boto:Params: {} 2024-02-01T10:50:15.534 DEBUG:boto:Token: None 2024-02-01T10:50:15.534 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:15 GMT /vclfcn-64/ 2024-02-01T10:50:15.534 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/SJBQ2QgdijbWbUJuIDTkWAWObA= 2024-02-01T10:50:15.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/SJBQ2QgdijbWbUJuIDTkWAWObA=', 'Content-Length': '0'} 2024-02-01T10:50:15.539 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eabcfc418fba3259-0065bb7767-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:15.539 DEBUG:boto:b'vclfcn-64obj11000falseobj12024-02-01T10:48:26.595Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:15.539 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:15.540 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:15.540 DEBUG:boto:path=/vclfcn-64/?prefix=obj1 2024-02-01T10:50:15.540 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj1 2024-02-01T10:50:15.540 DEBUG:boto:Method: GET 2024-02-01T10:50:15.540 DEBUG:boto:Path: /vclfcn-64/?prefix=obj1 2024-02-01T10:50:15.540 DEBUG:boto:Data: 2024-02-01T10:50:15.540 DEBUG:boto:Headers: {} 2024-02-01T10:50:15.540 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:15.540 DEBUG:boto:Port: 8000 2024-02-01T10:50:15.540 DEBUG:boto:Params: {} 2024-02-01T10:50:15.541 DEBUG:boto:Token: None 2024-02-01T10:50:15.541 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:15 GMT /vclfcn-64/ 2024-02-01T10:50:15.541 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/SJBQ2QgdijbWbUJuIDTkWAWObA= 2024-02-01T10:50:15.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/SJBQ2QgdijbWbUJuIDTkWAWObA=', 'Content-Length': '0'} 2024-02-01T10:50:15.546 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2c8cc7a2eb0f1f6-0065bb7767-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:15.546 DEBUG:boto:b'vclfcn-64obj11000falseobj12024-02-01T10:48:26.595Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:15.547 DEBUG:boto:path=/vclfcn-64/obj1 2024-02-01T10:50:15.547 DEBUG:boto:auth_path=/vclfcn-64/obj1 2024-02-01T10:50:15.547 DEBUG:boto:Method: HEAD 2024-02-01T10:50:15.548 DEBUG:boto:Path: /vclfcn-64/obj1 2024-02-01T10:50:15.548 DEBUG:boto:Data: 2024-02-01T10:50:15.548 DEBUG:boto:Headers: {} 2024-02-01T10:50:15.548 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:15.548 DEBUG:boto:Port: 8001 2024-02-01T10:50:15.549 DEBUG:boto:Params: {} 2024-02-01T10:50:15.549 DEBUG:boto:Token: None 2024-02-01T10:50:15.549 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:15 GMT /vclfcn-64/obj1 2024-02-01T10:50:15.550 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0WsGtQDVtdILGteqoL+2wx5SvXE= 2024-02-01T10:50:15.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0WsGtQDVtdILGteqoL+2wx5SvXE=', 'Content-Length': '0'} 2024-02-01T10:50:15.553 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001d20ce363501dec8-0065bb7767-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:15.553 DEBUG:boto:path=/vclfcn-64/obj1 2024-02-01T10:50:15.554 DEBUG:boto:auth_path=/vclfcn-64/obj1 2024-02-01T10:50:15.554 DEBUG:boto:Method: HEAD 2024-02-01T10:50:15.554 DEBUG:boto:Path: /vclfcn-64/obj1 2024-02-01T10:50:15.554 DEBUG:boto:Data: 2024-02-01T10:50:15.554 DEBUG:boto:Headers: {} 2024-02-01T10:50:15.555 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:15.555 DEBUG:boto:Port: 8000 2024-02-01T10:50:15.555 DEBUG:boto:Params: {} 2024-02-01T10:50:15.555 DEBUG:boto:Token: None 2024-02-01T10:50:15.556 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:15 GMT /vclfcn-64/obj1 2024-02-01T10:50:15.556 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0WsGtQDVtdILGteqoL+2wx5SvXE= 2024-02-01T10:50:15.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0WsGtQDVtdILGteqoL+2wx5SvXE=', 'Content-Length': '0'} 2024-02-01T10:50:15.559 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b5fbf36488639068-0065bb7767-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:15.560 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-02-01T10:50:15.560 DEBUG:boto:path=/vclfcn-64/obj1 2024-02-01T10:50:15.560 DEBUG:boto:auth_path=/vclfcn-64/obj1 2024-02-01T10:50:15.561 DEBUG:boto:Method: GET 2024-02-01T10:50:15.561 DEBUG:boto:Path: /vclfcn-64/obj1 2024-02-01T10:50:15.561 DEBUG:boto:Data: 2024-02-01T10:50:15.561 DEBUG:boto:Headers: {} 2024-02-01T10:50:15.561 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:15.562 DEBUG:boto:Port: 8001 2024-02-01T10:50:15.562 DEBUG:boto:Params: {} 2024-02-01T10:50:15.562 DEBUG:boto:Token: None 2024-02-01T10:50:15.562 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:15 GMT /vclfcn-64/obj1 2024-02-01T10:50:15.563 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:T26ep3vJ2G/Qgn3CI+yVM+dj8ck= 2024-02-01T10:50:15.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:T26ep3vJ2G/Qgn3CI+yVM+dj8ck=', 'Content-Length': '0'} 2024-02-01T10:50:15.566 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a5e0df268533ebf7-0065bb7767-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:15.608 DEBUG:boto:path=/vclfcn-64/obj1 2024-02-01T10:50:15.608 DEBUG:boto:auth_path=/vclfcn-64/obj1 2024-02-01T10:50:15.608 DEBUG:boto:Method: GET 2024-02-01T10:50:15.609 DEBUG:boto:Path: /vclfcn-64/obj1 2024-02-01T10:50:15.609 DEBUG:boto:Data: 2024-02-01T10:50:15.609 DEBUG:boto:Headers: {} 2024-02-01T10:50:15.609 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:15.609 DEBUG:boto:Port: 8000 2024-02-01T10:50:15.609 DEBUG:boto:Params: {} 2024-02-01T10:50:15.609 DEBUG:boto:Token: None 2024-02-01T10:50:15.609 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:15 GMT /vclfcn-64/obj1 2024-02-01T10:50:15.610 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:T26ep3vJ2G/Qgn3CI+yVM+dj8ck= 2024-02-01T10:50:15.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:T26ep3vJ2G/Qgn3CI+yVM+dj8ck=', 'Content-Length': '0'} 2024-02-01T10:50:15.613 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f4d62b8fde3714e6-0065bb7767-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:15.656 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:15.656 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:15.656 DEBUG:boto:path=/vclfcn-64/?prefix=obj1 2024-02-01T10:50:15.657 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj1 2024-02-01T10:50:15.657 DEBUG:boto:Method: GET 2024-02-01T10:50:15.657 DEBUG:boto:Path: /vclfcn-64/?prefix=obj1 2024-02-01T10:50:15.657 DEBUG:boto:Data: 2024-02-01T10:50:15.657 DEBUG:boto:Headers: {} 2024-02-01T10:50:15.657 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:15.657 DEBUG:boto:Port: 8001 2024-02-01T10:50:15.657 DEBUG:boto:Params: {} 2024-02-01T10:50:15.657 DEBUG:boto:Token: None 2024-02-01T10:50:15.658 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:15 GMT /vclfcn-64/ 2024-02-01T10:50:15.658 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/SJBQ2QgdijbWbUJuIDTkWAWObA= 2024-02-01T10:50:15.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/SJBQ2QgdijbWbUJuIDTkWAWObA=', 'Content-Length': '0'} 2024-02-01T10:50:15.662 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c2a31161b52cdac-0065bb7767-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:15.663 DEBUG:boto:b'vclfcn-64obj11000falseobj12024-02-01T10:48:26.595Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:15.663 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:15.663 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:15.663 DEBUG:boto:path=/vclfcn-64/?prefix=obj1 2024-02-01T10:50:15.663 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj1 2024-02-01T10:50:15.664 DEBUG:boto:Method: GET 2024-02-01T10:50:15.664 DEBUG:boto:Path: /vclfcn-64/?prefix=obj1 2024-02-01T10:50:15.664 DEBUG:boto:Data: 2024-02-01T10:50:15.664 DEBUG:boto:Headers: {} 2024-02-01T10:50:15.664 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:15.664 DEBUG:boto:Port: 8000 2024-02-01T10:50:15.664 DEBUG:boto:Params: {} 2024-02-01T10:50:15.664 DEBUG:boto:Token: None 2024-02-01T10:50:15.664 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:15 GMT /vclfcn-64/ 2024-02-01T10:50:15.665 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/SJBQ2QgdijbWbUJuIDTkWAWObA= 2024-02-01T10:50:15.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/SJBQ2QgdijbWbUJuIDTkWAWObA=', 'Content-Length': '0'} 2024-02-01T10:50:15.669 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050acdf3c4835d656-0065bb7767-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:15.712 DEBUG:boto:b'vclfcn-64obj11000falseobj12024-02-01T10:48:26.595Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:15.712 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:50:15.713 DEBUG:boto:path=/vclfcn-64/obj2 2024-02-01T10:50:15.713 DEBUG:boto:auth_path=/vclfcn-64/obj2 2024-02-01T10:50:15.713 DEBUG:boto:path=/vclfcn-64/obj2?versionId=null 2024-02-01T10:50:15.713 DEBUG:boto:auth_path=/vclfcn-64/obj2?versionId=null 2024-02-01T10:50:15.713 DEBUG:boto:Method: GET 2024-02-01T10:50:15.714 DEBUG:boto:Path: /vclfcn-64/obj2?versionId=null 2024-02-01T10:50:15.714 DEBUG:boto:Data: 2024-02-01T10:50:15.714 DEBUG:boto:Headers: {} 2024-02-01T10:50:15.714 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:15.714 DEBUG:boto:Port: 8001 2024-02-01T10:50:15.714 DEBUG:boto:Params: {} 2024-02-01T10:50:15.714 DEBUG:boto:Token: None 2024-02-01T10:50:15.714 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:15 GMT /vclfcn-64/obj2?versionId=null 2024-02-01T10:50:15.715 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:73uLtwWOa27ouzusEmPNjSlQWuM= 2024-02-01T10:50:15.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:73uLtwWOa27ouzusEmPNjSlQWuM=', 'Content-Length': '0'} 2024-02-01T10:50:15.718 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d53ca1fb2a6daec4-0065bb7767-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:15.718 DEBUG:boto:path=/vclfcn-64/obj2 2024-02-01T10:50:15.718 DEBUG:boto:auth_path=/vclfcn-64/obj2 2024-02-01T10:50:15.718 DEBUG:boto:path=/vclfcn-64/obj2?versionId=null 2024-02-01T10:50:15.718 DEBUG:boto:auth_path=/vclfcn-64/obj2?versionId=null 2024-02-01T10:50:15.718 DEBUG:boto:Method: GET 2024-02-01T10:50:15.719 DEBUG:boto:Path: /vclfcn-64/obj2?versionId=null 2024-02-01T10:50:15.719 DEBUG:boto:Data: 2024-02-01T10:50:15.719 DEBUG:boto:Headers: {} 2024-02-01T10:50:15.719 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:15.719 DEBUG:boto:Port: 8000 2024-02-01T10:50:15.719 DEBUG:boto:Params: {} 2024-02-01T10:50:15.719 DEBUG:boto:Token: None 2024-02-01T10:50:15.719 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:15 GMT /vclfcn-64/obj2?versionId=null 2024-02-01T10:50:15.719 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:73uLtwWOa27ouzusEmPNjSlQWuM= 2024-02-01T10:50:15.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:73uLtwWOa27ouzusEmPNjSlQWuM=', 'Content-Length': '0'} 2024-02-01T10:50:15.723 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009c533cf2c8426322-0065bb7767-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:15.724 DEBUG:boto:path=/vclfcn-64/obj2 2024-02-01T10:50:15.724 DEBUG:boto:auth_path=/vclfcn-64/obj2 2024-02-01T10:50:15.724 DEBUG:boto:path=/vclfcn-64/obj2?versionId=null 2024-02-01T10:50:15.724 DEBUG:boto:auth_path=/vclfcn-64/obj2?versionId=null 2024-02-01T10:50:15.724 DEBUG:boto:Method: HEAD 2024-02-01T10:50:15.725 DEBUG:boto:Path: /vclfcn-64/obj2?versionId=null 2024-02-01T10:50:15.725 DEBUG:boto:Data: 2024-02-01T10:50:15.725 DEBUG:boto:Headers: {} 2024-02-01T10:50:15.725 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:15.725 DEBUG:boto:Port: 8001 2024-02-01T10:50:15.726 DEBUG:boto:Params: {} 2024-02-01T10:50:15.726 DEBUG:boto:Token: None 2024-02-01T10:50:15.726 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:15 GMT /vclfcn-64/obj2?versionId=null 2024-02-01T10:50:15.727 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:c0gCgsaNIBvoF4YJeD7kYIcGC90= 2024-02-01T10:50:15.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:c0gCgsaNIBvoF4YJeD7kYIcGC90=', 'Content-Length': '0'} 2024-02-01T10:50:15.730 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005237f83368317451-0065bb7767-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:15.731 DEBUG:boto:path=/vclfcn-64/obj2 2024-02-01T10:50:15.731 DEBUG:boto:auth_path=/vclfcn-64/obj2 2024-02-01T10:50:15.731 DEBUG:boto:path=/vclfcn-64/obj2?versionId=null 2024-02-01T10:50:15.731 DEBUG:boto:auth_path=/vclfcn-64/obj2?versionId=null 2024-02-01T10:50:15.731 DEBUG:boto:Method: HEAD 2024-02-01T10:50:15.732 DEBUG:boto:Path: /vclfcn-64/obj2?versionId=null 2024-02-01T10:50:15.732 DEBUG:boto:Data: 2024-02-01T10:50:15.732 DEBUG:boto:Headers: {} 2024-02-01T10:50:15.732 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:15.732 DEBUG:boto:Port: 8000 2024-02-01T10:50:15.733 DEBUG:boto:Params: {} 2024-02-01T10:50:15.733 DEBUG:boto:Token: None 2024-02-01T10:50:15.733 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:15 GMT /vclfcn-64/obj2?versionId=null 2024-02-01T10:50:15.734 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:c0gCgsaNIBvoF4YJeD7kYIcGC90= 2024-02-01T10:50:15.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:c0gCgsaNIBvoF4YJeD7kYIcGC90=', 'Content-Length': '0'} 2024-02-01T10:50:15.737 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000847b527d66de2290-0065bb7767-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:15.738 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:50:15.738 DEBUG:boto:path=/vclfcn-64/obj2 2024-02-01T10:50:15.738 DEBUG:boto:auth_path=/vclfcn-64/obj2 2024-02-01T10:50:15.738 DEBUG:boto:path=/vclfcn-64/obj2?versionId=null 2024-02-01T10:50:15.738 DEBUG:boto:auth_path=/vclfcn-64/obj2?versionId=null 2024-02-01T10:50:15.739 DEBUG:boto:Method: GET 2024-02-01T10:50:15.739 DEBUG:boto:Path: /vclfcn-64/obj2?versionId=null 2024-02-01T10:50:15.739 DEBUG:boto:Data: 2024-02-01T10:50:15.739 DEBUG:boto:Headers: {} 2024-02-01T10:50:15.739 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:15.740 DEBUG:boto:Port: 8001 2024-02-01T10:50:15.740 DEBUG:boto:Params: {} 2024-02-01T10:50:15.740 DEBUG:boto:Token: None 2024-02-01T10:50:15.741 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:15 GMT /vclfcn-64/obj2?versionId=null 2024-02-01T10:50:15.741 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:73uLtwWOa27ouzusEmPNjSlQWuM= 2024-02-01T10:50:15.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:73uLtwWOa27ouzusEmPNjSlQWuM=', 'Content-Length': '0'} 2024-02-01T10:50:15.744 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000023615cbd0c9b33d-0065bb7767-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:15.744 DEBUG:boto:path=/vclfcn-64/obj2 2024-02-01T10:50:15.744 DEBUG:boto:auth_path=/vclfcn-64/obj2 2024-02-01T10:50:15.744 DEBUG:boto:path=/vclfcn-64/obj2?versionId=null 2024-02-01T10:50:15.745 DEBUG:boto:auth_path=/vclfcn-64/obj2?versionId=null 2024-02-01T10:50:15.745 DEBUG:boto:Method: GET 2024-02-01T10:50:15.745 DEBUG:boto:Path: /vclfcn-64/obj2?versionId=null 2024-02-01T10:50:15.745 DEBUG:boto:Data: 2024-02-01T10:50:15.745 DEBUG:boto:Headers: {} 2024-02-01T10:50:15.745 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:15.745 DEBUG:boto:Port: 8000 2024-02-01T10:50:15.745 DEBUG:boto:Params: {} 2024-02-01T10:50:15.745 DEBUG:boto:Token: None 2024-02-01T10:50:15.745 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:15 GMT /vclfcn-64/obj2?versionId=null 2024-02-01T10:50:15.746 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:73uLtwWOa27ouzusEmPNjSlQWuM= 2024-02-01T10:50:15.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:73uLtwWOa27ouzusEmPNjSlQWuM=', 'Content-Length': '0'} 2024-02-01T10:50:15.749 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000b20480e3c344b1b-0065bb7767-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:15.792 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:15.792 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:15.792 DEBUG:boto:path=/vclfcn-64/?prefix=obj2 2024-02-01T10:50:15.792 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj2 2024-02-01T10:50:15.793 DEBUG:boto:Method: GET 2024-02-01T10:50:15.793 DEBUG:boto:Path: /vclfcn-64/?prefix=obj2 2024-02-01T10:50:15.793 DEBUG:boto:Data: 2024-02-01T10:50:15.793 DEBUG:boto:Headers: {} 2024-02-01T10:50:15.793 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:15.793 DEBUG:boto:Port: 8001 2024-02-01T10:50:15.793 DEBUG:boto:Params: {} 2024-02-01T10:50:15.793 DEBUG:boto:Token: None 2024-02-01T10:50:15.793 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:15 GMT /vclfcn-64/ 2024-02-01T10:50:15.794 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/SJBQ2QgdijbWbUJuIDTkWAWObA= 2024-02-01T10:50:15.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/SJBQ2QgdijbWbUJuIDTkWAWObA=', 'Content-Length': '0'} 2024-02-01T10:50:15.798 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d161efd3a4213db-0065bb7767-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:15.799 DEBUG:boto:b'vclfcn-64obj21000falseobj22024-02-01T10:48:26.615Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:15.799 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:15.800 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:15.800 DEBUG:boto:path=/vclfcn-64/?prefix=obj2 2024-02-01T10:50:15.800 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj2 2024-02-01T10:50:15.800 DEBUG:boto:Method: GET 2024-02-01T10:50:15.800 DEBUG:boto:Path: /vclfcn-64/?prefix=obj2 2024-02-01T10:50:15.801 DEBUG:boto:Data: 2024-02-01T10:50:15.801 DEBUG:boto:Headers: {} 2024-02-01T10:50:15.801 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:15.801 DEBUG:boto:Port: 8000 2024-02-01T10:50:15.801 DEBUG:boto:Params: {} 2024-02-01T10:50:15.801 DEBUG:boto:Token: None 2024-02-01T10:50:15.802 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:15 GMT /vclfcn-64/ 2024-02-01T10:50:15.802 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/SJBQ2QgdijbWbUJuIDTkWAWObA= 2024-02-01T10:50:15.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/SJBQ2QgdijbWbUJuIDTkWAWObA=', 'Content-Length': '0'} 2024-02-01T10:50:15.806 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ebb99aae1dd4d9a-0065bb7767-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:15.848 DEBUG:boto:b'vclfcn-64obj21000falseobj22024-02-01T10:48:26.615Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:15.849 DEBUG:boto:path=/vclfcn-64/obj2 2024-02-01T10:50:15.849 DEBUG:boto:auth_path=/vclfcn-64/obj2 2024-02-01T10:50:15.849 DEBUG:boto:Method: HEAD 2024-02-01T10:50:15.849 DEBUG:boto:Path: /vclfcn-64/obj2 2024-02-01T10:50:15.849 DEBUG:boto:Data: 2024-02-01T10:50:15.849 DEBUG:boto:Headers: {} 2024-02-01T10:50:15.849 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:15.849 DEBUG:boto:Port: 8001 2024-02-01T10:50:15.849 DEBUG:boto:Params: {} 2024-02-01T10:50:15.849 DEBUG:boto:Token: None 2024-02-01T10:50:15.850 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:15 GMT /vclfcn-64/obj2 2024-02-01T10:50:15.850 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:EiJUEu/KNKsenEpbbz1HD19lFOg= 2024-02-01T10:50:15.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:EiJUEu/KNKsenEpbbz1HD19lFOg=', 'Content-Length': '0'} 2024-02-01T10:50:15.853 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000034a6abc88215fac2-0065bb7767-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:15.853 DEBUG:boto:path=/vclfcn-64/obj2 2024-02-01T10:50:15.853 DEBUG:boto:auth_path=/vclfcn-64/obj2 2024-02-01T10:50:15.853 DEBUG:boto:Method: HEAD 2024-02-01T10:50:15.853 DEBUG:boto:Path: /vclfcn-64/obj2 2024-02-01T10:50:15.854 DEBUG:boto:Data: 2024-02-01T10:50:15.854 DEBUG:boto:Headers: {} 2024-02-01T10:50:15.854 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:15.854 DEBUG:boto:Port: 8000 2024-02-01T10:50:15.854 DEBUG:boto:Params: {} 2024-02-01T10:50:15.854 DEBUG:boto:Token: None 2024-02-01T10:50:15.854 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:15 GMT /vclfcn-64/obj2 2024-02-01T10:50:15.854 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:EiJUEu/KNKsenEpbbz1HD19lFOg= 2024-02-01T10:50:15.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:EiJUEu/KNKsenEpbbz1HD19lFOg=', 'Content-Length': '0'} 2024-02-01T10:50:15.857 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c26ecb3e660ad1d8-0065bb7767-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:15.857 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-02-01T10:50:15.857 DEBUG:boto:path=/vclfcn-64/obj2 2024-02-01T10:50:15.858 DEBUG:boto:auth_path=/vclfcn-64/obj2 2024-02-01T10:50:15.858 DEBUG:boto:Method: GET 2024-02-01T10:50:15.858 DEBUG:boto:Path: /vclfcn-64/obj2 2024-02-01T10:50:15.858 DEBUG:boto:Data: 2024-02-01T10:50:15.858 DEBUG:boto:Headers: {} 2024-02-01T10:50:15.858 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:15.858 DEBUG:boto:Port: 8001 2024-02-01T10:50:15.858 DEBUG:boto:Params: {} 2024-02-01T10:50:15.858 DEBUG:boto:Token: None 2024-02-01T10:50:15.858 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:15 GMT /vclfcn-64/obj2 2024-02-01T10:50:15.859 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lO8yMLToyPjHBjwgEX0GAeEJCG8= 2024-02-01T10:50:15.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lO8yMLToyPjHBjwgEX0GAeEJCG8=', 'Content-Length': '0'} 2024-02-01T10:50:15.861 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001fd16d89dd97aa2d-0065bb7767-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:15.904 DEBUG:boto:path=/vclfcn-64/obj2 2024-02-01T10:50:15.905 DEBUG:boto:auth_path=/vclfcn-64/obj2 2024-02-01T10:50:15.905 DEBUG:boto:Method: GET 2024-02-01T10:50:15.905 DEBUG:boto:Path: /vclfcn-64/obj2 2024-02-01T10:50:15.905 DEBUG:boto:Data: 2024-02-01T10:50:15.905 DEBUG:boto:Headers: {} 2024-02-01T10:50:15.906 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:15.906 DEBUG:boto:Port: 8000 2024-02-01T10:50:15.906 DEBUG:boto:Params: {} 2024-02-01T10:50:15.906 DEBUG:boto:Token: None 2024-02-01T10:50:15.907 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:15 GMT /vclfcn-64/obj2 2024-02-01T10:50:15.907 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:lO8yMLToyPjHBjwgEX0GAeEJCG8= 2024-02-01T10:50:15.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:lO8yMLToyPjHBjwgEX0GAeEJCG8=', 'Content-Length': '0'} 2024-02-01T10:50:15.910 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e7d79509d598b130-0065bb7767-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:15.952 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:15.953 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:15.953 DEBUG:boto:path=/vclfcn-64/?prefix=obj2 2024-02-01T10:50:15.953 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj2 2024-02-01T10:50:15.953 DEBUG:boto:Method: GET 2024-02-01T10:50:15.953 DEBUG:boto:Path: /vclfcn-64/?prefix=obj2 2024-02-01T10:50:15.953 DEBUG:boto:Data: 2024-02-01T10:50:15.953 DEBUG:boto:Headers: {} 2024-02-01T10:50:15.953 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:15.954 DEBUG:boto:Port: 8001 2024-02-01T10:50:15.954 DEBUG:boto:Params: {} 2024-02-01T10:50:15.954 DEBUG:boto:Token: None 2024-02-01T10:50:15.954 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:15 GMT /vclfcn-64/ 2024-02-01T10:50:15.954 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/SJBQ2QgdijbWbUJuIDTkWAWObA= 2024-02-01T10:50:15.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/SJBQ2QgdijbWbUJuIDTkWAWObA=', 'Content-Length': '0'} 2024-02-01T10:50:15.959 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009f9079850df45d0-0065bb7767-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:15.959 DEBUG:boto:b'vclfcn-64obj21000falseobj22024-02-01T10:48:26.615Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:15.959 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:15.960 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:15.960 DEBUG:boto:path=/vclfcn-64/?prefix=obj2 2024-02-01T10:50:15.960 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj2 2024-02-01T10:50:15.960 DEBUG:boto:Method: GET 2024-02-01T10:50:15.960 DEBUG:boto:Path: /vclfcn-64/?prefix=obj2 2024-02-01T10:50:15.960 DEBUG:boto:Data: 2024-02-01T10:50:15.960 DEBUG:boto:Headers: {} 2024-02-01T10:50:15.960 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:15.960 DEBUG:boto:Port: 8000 2024-02-01T10:50:15.960 DEBUG:boto:Params: {} 2024-02-01T10:50:15.961 DEBUG:boto:Token: None 2024-02-01T10:50:15.961 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:15 GMT /vclfcn-64/ 2024-02-01T10:50:15.961 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/SJBQ2QgdijbWbUJuIDTkWAWObA= 2024-02-01T10:50:15.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:15 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/SJBQ2QgdijbWbUJuIDTkWAWObA=', 'Content-Length': '0'} 2024-02-01T10:50:15.967 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068576fbc67757542-0065bb7767-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:15 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:16.012 DEBUG:boto:b'vclfcn-64obj21000falseobj22024-02-01T10:48:26.615Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:16.013 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:50:16.014 DEBUG:boto:path=/vclfcn-64/obj3 2024-02-01T10:50:16.015 DEBUG:boto:auth_path=/vclfcn-64/obj3 2024-02-01T10:50:16.015 DEBUG:boto:path=/vclfcn-64/obj3?versionId=null 2024-02-01T10:50:16.015 DEBUG:boto:auth_path=/vclfcn-64/obj3?versionId=null 2024-02-01T10:50:16.015 DEBUG:boto:Method: GET 2024-02-01T10:50:16.015 DEBUG:boto:Path: /vclfcn-64/obj3?versionId=null 2024-02-01T10:50:16.015 DEBUG:boto:Data: 2024-02-01T10:50:16.015 DEBUG:boto:Headers: {} 2024-02-01T10:50:16.015 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:16.015 DEBUG:boto:Port: 8001 2024-02-01T10:50:16.016 DEBUG:boto:Params: {} 2024-02-01T10:50:16.016 DEBUG:boto:Token: None 2024-02-01T10:50:16.016 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:16 GMT /vclfcn-64/obj3?versionId=null 2024-02-01T10:50:16.016 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nhi4rW1gx8oXUaSUYmiHLb844yg= 2024-02-01T10:50:16.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nhi4rW1gx8oXUaSUYmiHLb844yg=', 'Content-Length': '0'} 2024-02-01T10:50:16.019 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000929bf3a32ec3380a-0065bb7768-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:16.019 DEBUG:boto:path=/vclfcn-64/obj3 2024-02-01T10:50:16.019 DEBUG:boto:auth_path=/vclfcn-64/obj3 2024-02-01T10:50:16.020 DEBUG:boto:path=/vclfcn-64/obj3?versionId=null 2024-02-01T10:50:16.020 DEBUG:boto:auth_path=/vclfcn-64/obj3?versionId=null 2024-02-01T10:50:16.020 DEBUG:boto:Method: GET 2024-02-01T10:50:16.020 DEBUG:boto:Path: /vclfcn-64/obj3?versionId=null 2024-02-01T10:50:16.020 DEBUG:boto:Data: 2024-02-01T10:50:16.020 DEBUG:boto:Headers: {} 2024-02-01T10:50:16.020 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:16.020 DEBUG:boto:Port: 8000 2024-02-01T10:50:16.020 DEBUG:boto:Params: {} 2024-02-01T10:50:16.020 DEBUG:boto:Token: None 2024-02-01T10:50:16.021 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:16 GMT /vclfcn-64/obj3?versionId=null 2024-02-01T10:50:16.021 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nhi4rW1gx8oXUaSUYmiHLb844yg= 2024-02-01T10:50:16.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nhi4rW1gx8oXUaSUYmiHLb844yg=', 'Content-Length': '0'} 2024-02-01T10:50:16.025 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d9c302b1bcc34d4f-0065bb7768-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:16.068 DEBUG:boto:path=/vclfcn-64/obj3 2024-02-01T10:50:16.068 DEBUG:boto:auth_path=/vclfcn-64/obj3 2024-02-01T10:50:16.068 DEBUG:boto:path=/vclfcn-64/obj3?versionId=null 2024-02-01T10:50:16.069 DEBUG:boto:auth_path=/vclfcn-64/obj3?versionId=null 2024-02-01T10:50:16.069 DEBUG:boto:Method: HEAD 2024-02-01T10:50:16.069 DEBUG:boto:Path: /vclfcn-64/obj3?versionId=null 2024-02-01T10:50:16.069 DEBUG:boto:Data: 2024-02-01T10:50:16.069 DEBUG:boto:Headers: {} 2024-02-01T10:50:16.069 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:16.070 DEBUG:boto:Port: 8001 2024-02-01T10:50:16.070 DEBUG:boto:Params: {} 2024-02-01T10:50:16.070 DEBUG:boto:Token: None 2024-02-01T10:50:16.070 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:16 GMT /vclfcn-64/obj3?versionId=null 2024-02-01T10:50:16.070 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9e8eNAMiuLa4TetIWwtYZrTDfSs= 2024-02-01T10:50:16.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9e8eNAMiuLa4TetIWwtYZrTDfSs=', 'Content-Length': '0'} 2024-02-01T10:50:16.073 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e88bf348d5f62d1f-0065bb7768-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:16.074 DEBUG:boto:path=/vclfcn-64/obj3 2024-02-01T10:50:16.074 DEBUG:boto:auth_path=/vclfcn-64/obj3 2024-02-01T10:50:16.074 DEBUG:boto:path=/vclfcn-64/obj3?versionId=null 2024-02-01T10:50:16.074 DEBUG:boto:auth_path=/vclfcn-64/obj3?versionId=null 2024-02-01T10:50:16.074 DEBUG:boto:Method: HEAD 2024-02-01T10:50:16.074 DEBUG:boto:Path: /vclfcn-64/obj3?versionId=null 2024-02-01T10:50:16.074 DEBUG:boto:Data: 2024-02-01T10:50:16.074 DEBUG:boto:Headers: {} 2024-02-01T10:50:16.074 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:16.074 DEBUG:boto:Port: 8000 2024-02-01T10:50:16.074 DEBUG:boto:Params: {} 2024-02-01T10:50:16.075 DEBUG:boto:Token: None 2024-02-01T10:50:16.075 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:16 GMT /vclfcn-64/obj3?versionId=null 2024-02-01T10:50:16.075 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9e8eNAMiuLa4TetIWwtYZrTDfSs= 2024-02-01T10:50:16.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9e8eNAMiuLa4TetIWwtYZrTDfSs=', 'Content-Length': '0'} 2024-02-01T10:50:16.078 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003013a62b73b6ffaf-0065bb7768-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:16.078 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:50:16.078 DEBUG:boto:path=/vclfcn-64/obj3 2024-02-01T10:50:16.078 DEBUG:boto:auth_path=/vclfcn-64/obj3 2024-02-01T10:50:16.078 DEBUG:boto:path=/vclfcn-64/obj3?versionId=null 2024-02-01T10:50:16.078 DEBUG:boto:auth_path=/vclfcn-64/obj3?versionId=null 2024-02-01T10:50:16.078 DEBUG:boto:Method: GET 2024-02-01T10:50:16.078 DEBUG:boto:Path: /vclfcn-64/obj3?versionId=null 2024-02-01T10:50:16.079 DEBUG:boto:Data: 2024-02-01T10:50:16.079 DEBUG:boto:Headers: {} 2024-02-01T10:50:16.079 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:16.079 DEBUG:boto:Port: 8001 2024-02-01T10:50:16.079 DEBUG:boto:Params: {} 2024-02-01T10:50:16.079 DEBUG:boto:Token: None 2024-02-01T10:50:16.079 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:16 GMT /vclfcn-64/obj3?versionId=null 2024-02-01T10:50:16.079 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nhi4rW1gx8oXUaSUYmiHLb844yg= 2024-02-01T10:50:16.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nhi4rW1gx8oXUaSUYmiHLb844yg=', 'Content-Length': '0'} 2024-02-01T10:50:16.082 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bb45efdbebf7b6e0-0065bb7768-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:16.082 DEBUG:boto:path=/vclfcn-64/obj3 2024-02-01T10:50:16.083 DEBUG:boto:auth_path=/vclfcn-64/obj3 2024-02-01T10:50:16.083 DEBUG:boto:path=/vclfcn-64/obj3?versionId=null 2024-02-01T10:50:16.083 DEBUG:boto:auth_path=/vclfcn-64/obj3?versionId=null 2024-02-01T10:50:16.084 DEBUG:boto:Method: GET 2024-02-01T10:50:16.084 DEBUG:boto:Path: /vclfcn-64/obj3?versionId=null 2024-02-01T10:50:16.084 DEBUG:boto:Data: 2024-02-01T10:50:16.084 DEBUG:boto:Headers: {} 2024-02-01T10:50:16.084 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:16.084 DEBUG:boto:Port: 8000 2024-02-01T10:50:16.084 DEBUG:boto:Params: {} 2024-02-01T10:50:16.085 DEBUG:boto:Token: None 2024-02-01T10:50:16.085 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:16 GMT /vclfcn-64/obj3?versionId=null 2024-02-01T10:50:16.085 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:nhi4rW1gx8oXUaSUYmiHLb844yg= 2024-02-01T10:50:16.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:nhi4rW1gx8oXUaSUYmiHLb844yg=', 'Content-Length': '0'} 2024-02-01T10:50:16.088 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cdcaec60d3a15a9e-0065bb7768-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:16.132 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:16.132 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:16.132 DEBUG:boto:path=/vclfcn-64/?prefix=obj3 2024-02-01T10:50:16.132 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj3 2024-02-01T10:50:16.132 DEBUG:boto:Method: GET 2024-02-01T10:50:16.133 DEBUG:boto:Path: /vclfcn-64/?prefix=obj3 2024-02-01T10:50:16.133 DEBUG:boto:Data: 2024-02-01T10:50:16.133 DEBUG:boto:Headers: {} 2024-02-01T10:50:16.133 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:16.133 DEBUG:boto:Port: 8001 2024-02-01T10:50:16.133 DEBUG:boto:Params: {} 2024-02-01T10:50:16.133 DEBUG:boto:Token: None 2024-02-01T10:50:16.133 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:16 GMT /vclfcn-64/ 2024-02-01T10:50:16.133 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+KF5bKSSGsCmEAAkRMVXIkM3IK8= 2024-02-01T10:50:16.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+KF5bKSSGsCmEAAkRMVXIkM3IK8=', 'Content-Length': '0'} 2024-02-01T10:50:16.138 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010e6847c586e5479-0065bb7768-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:16.139 DEBUG:boto:b'vclfcn-64obj31000falseobj32024-02-01T10:48:26.634Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:16.139 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:16.139 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:16.139 DEBUG:boto:path=/vclfcn-64/?prefix=obj3 2024-02-01T10:50:16.139 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj3 2024-02-01T10:50:16.140 DEBUG:boto:Method: GET 2024-02-01T10:50:16.140 DEBUG:boto:Path: /vclfcn-64/?prefix=obj3 2024-02-01T10:50:16.140 DEBUG:boto:Data: 2024-02-01T10:50:16.140 DEBUG:boto:Headers: {} 2024-02-01T10:50:16.140 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:16.140 DEBUG:boto:Port: 8000 2024-02-01T10:50:16.140 DEBUG:boto:Params: {} 2024-02-01T10:50:16.140 DEBUG:boto:Token: None 2024-02-01T10:50:16.141 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:16 GMT /vclfcn-64/ 2024-02-01T10:50:16.141 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+KF5bKSSGsCmEAAkRMVXIkM3IK8= 2024-02-01T10:50:16.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+KF5bKSSGsCmEAAkRMVXIkM3IK8=', 'Content-Length': '0'} 2024-02-01T10:50:16.145 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e85e9d99961f10a-0065bb7768-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:16.188 DEBUG:boto:b'vclfcn-64obj31000falseobj32024-02-01T10:48:26.634Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:16.188 DEBUG:boto:path=/vclfcn-64/obj3 2024-02-01T10:50:16.188 DEBUG:boto:auth_path=/vclfcn-64/obj3 2024-02-01T10:50:16.189 DEBUG:boto:Method: HEAD 2024-02-01T10:50:16.189 DEBUG:boto:Path: /vclfcn-64/obj3 2024-02-01T10:50:16.189 DEBUG:boto:Data: 2024-02-01T10:50:16.189 DEBUG:boto:Headers: {} 2024-02-01T10:50:16.189 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:16.189 DEBUG:boto:Port: 8001 2024-02-01T10:50:16.189 DEBUG:boto:Params: {} 2024-02-01T10:50:16.189 DEBUG:boto:Token: None 2024-02-01T10:50:16.189 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:16 GMT /vclfcn-64/obj3 2024-02-01T10:50:16.190 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Z2YIHhVG+28/ec2wA99vZEgP3pI= 2024-02-01T10:50:16.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Z2YIHhVG+28/ec2wA99vZEgP3pI=', 'Content-Length': '0'} 2024-02-01T10:50:16.193 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a05d14077c2a10b3-0065bb7768-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:16.193 DEBUG:boto:path=/vclfcn-64/obj3 2024-02-01T10:50:16.193 DEBUG:boto:auth_path=/vclfcn-64/obj3 2024-02-01T10:50:16.193 DEBUG:boto:Method: HEAD 2024-02-01T10:50:16.193 DEBUG:boto:Path: /vclfcn-64/obj3 2024-02-01T10:50:16.193 DEBUG:boto:Data: 2024-02-01T10:50:16.193 DEBUG:boto:Headers: {} 2024-02-01T10:50:16.194 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:16.194 DEBUG:boto:Port: 8000 2024-02-01T10:50:16.194 DEBUG:boto:Params: {} 2024-02-01T10:50:16.194 DEBUG:boto:Token: None 2024-02-01T10:50:16.194 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:16 GMT /vclfcn-64/obj3 2024-02-01T10:50:16.194 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Z2YIHhVG+28/ec2wA99vZEgP3pI= 2024-02-01T10:50:16.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Z2YIHhVG+28/ec2wA99vZEgP3pI=', 'Content-Length': '0'} 2024-02-01T10:50:16.197 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000048388b4b4fa0f55a-0065bb7768-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:16.197 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-02-01T10:50:16.198 DEBUG:boto:path=/vclfcn-64/obj3 2024-02-01T10:50:16.198 DEBUG:boto:auth_path=/vclfcn-64/obj3 2024-02-01T10:50:16.198 DEBUG:boto:Method: GET 2024-02-01T10:50:16.198 DEBUG:boto:Path: /vclfcn-64/obj3 2024-02-01T10:50:16.198 DEBUG:boto:Data: 2024-02-01T10:50:16.198 DEBUG:boto:Headers: {} 2024-02-01T10:50:16.199 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:16.199 DEBUG:boto:Port: 8001 2024-02-01T10:50:16.199 DEBUG:boto:Params: {} 2024-02-01T10:50:16.199 DEBUG:boto:Token: None 2024-02-01T10:50:16.199 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:16 GMT /vclfcn-64/obj3 2024-02-01T10:50:16.199 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:VOs7W3ZdbpA8i1Yfu/YbOyyy0t4= 2024-02-01T10:50:16.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:VOs7W3ZdbpA8i1Yfu/YbOyyy0t4=', 'Content-Length': '0'} 2024-02-01T10:50:16.202 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b1f03fc6e018083b-0065bb7768-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:16.202 DEBUG:boto:path=/vclfcn-64/obj3 2024-02-01T10:50:16.203 DEBUG:boto:auth_path=/vclfcn-64/obj3 2024-02-01T10:50:16.203 DEBUG:boto:Method: GET 2024-02-01T10:50:16.203 DEBUG:boto:Path: /vclfcn-64/obj3 2024-02-01T10:50:16.203 DEBUG:boto:Data: 2024-02-01T10:50:16.203 DEBUG:boto:Headers: {} 2024-02-01T10:50:16.203 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:16.203 DEBUG:boto:Port: 8000 2024-02-01T10:50:16.203 DEBUG:boto:Params: {} 2024-02-01T10:50:16.203 DEBUG:boto:Token: None 2024-02-01T10:50:16.204 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:16 GMT /vclfcn-64/obj3 2024-02-01T10:50:16.204 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:VOs7W3ZdbpA8i1Yfu/YbOyyy0t4= 2024-02-01T10:50:16.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:VOs7W3ZdbpA8i1Yfu/YbOyyy0t4=', 'Content-Length': '0'} 2024-02-01T10:50:16.207 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f6331b4ed1c32082-0065bb7768-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:16.248 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:16.248 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:16.248 DEBUG:boto:path=/vclfcn-64/?prefix=obj3 2024-02-01T10:50:16.249 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj3 2024-02-01T10:50:16.249 DEBUG:boto:Method: GET 2024-02-01T10:50:16.249 DEBUG:boto:Path: /vclfcn-64/?prefix=obj3 2024-02-01T10:50:16.249 DEBUG:boto:Data: 2024-02-01T10:50:16.249 DEBUG:boto:Headers: {} 2024-02-01T10:50:16.249 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:16.249 DEBUG:boto:Port: 8001 2024-02-01T10:50:16.249 DEBUG:boto:Params: {} 2024-02-01T10:50:16.249 DEBUG:boto:Token: None 2024-02-01T10:50:16.249 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:16 GMT /vclfcn-64/ 2024-02-01T10:50:16.250 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+KF5bKSSGsCmEAAkRMVXIkM3IK8= 2024-02-01T10:50:16.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+KF5bKSSGsCmEAAkRMVXIkM3IK8=', 'Content-Length': '0'} 2024-02-01T10:50:16.254 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000652939eb568faeef-0065bb7768-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:16.254 DEBUG:boto:b'vclfcn-64obj31000falseobj32024-02-01T10:48:26.634Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:16.255 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:16.255 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:16.255 DEBUG:boto:path=/vclfcn-64/?prefix=obj3 2024-02-01T10:50:16.255 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj3 2024-02-01T10:50:16.255 DEBUG:boto:Method: GET 2024-02-01T10:50:16.255 DEBUG:boto:Path: /vclfcn-64/?prefix=obj3 2024-02-01T10:50:16.255 DEBUG:boto:Data: 2024-02-01T10:50:16.255 DEBUG:boto:Headers: {} 2024-02-01T10:50:16.255 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:16.255 DEBUG:boto:Port: 8000 2024-02-01T10:50:16.256 DEBUG:boto:Params: {} 2024-02-01T10:50:16.256 DEBUG:boto:Token: None 2024-02-01T10:50:16.256 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:16 GMT /vclfcn-64/ 2024-02-01T10:50:16.256 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+KF5bKSSGsCmEAAkRMVXIkM3IK8= 2024-02-01T10:50:16.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+KF5bKSSGsCmEAAkRMVXIkM3IK8=', 'Content-Length': '0'} 2024-02-01T10:50:16.260 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e25ab86eda9d95c1-0065bb7768-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:16.304 DEBUG:boto:b'vclfcn-64obj31000falseobj32024-02-01T10:48:26.634Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:16.305 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:50:16.305 DEBUG:boto:path=/vclfcn-64/obj4 2024-02-01T10:50:16.305 DEBUG:boto:auth_path=/vclfcn-64/obj4 2024-02-01T10:50:16.306 DEBUG:boto:path=/vclfcn-64/obj4?versionId=null 2024-02-01T10:50:16.306 DEBUG:boto:auth_path=/vclfcn-64/obj4?versionId=null 2024-02-01T10:50:16.306 DEBUG:boto:Method: GET 2024-02-01T10:50:16.306 DEBUG:boto:Path: /vclfcn-64/obj4?versionId=null 2024-02-01T10:50:16.306 DEBUG:boto:Data: 2024-02-01T10:50:16.307 DEBUG:boto:Headers: {} 2024-02-01T10:50:16.307 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:16.307 DEBUG:boto:Port: 8001 2024-02-01T10:50:16.307 DEBUG:boto:Params: {} 2024-02-01T10:50:16.307 DEBUG:boto:Token: None 2024-02-01T10:50:16.308 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:16 GMT /vclfcn-64/obj4?versionId=null 2024-02-01T10:50:16.308 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gP8/IDz2AyGKpnzleuqMCpY/uj4= 2024-02-01T10:50:16.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gP8/IDz2AyGKpnzleuqMCpY/uj4=', 'Content-Length': '0'} 2024-02-01T10:50:16.312 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d042e826f0267662-0065bb7768-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:16.312 DEBUG:boto:path=/vclfcn-64/obj4 2024-02-01T10:50:16.312 DEBUG:boto:auth_path=/vclfcn-64/obj4 2024-02-01T10:50:16.313 DEBUG:boto:path=/vclfcn-64/obj4?versionId=null 2024-02-01T10:50:16.313 DEBUG:boto:auth_path=/vclfcn-64/obj4?versionId=null 2024-02-01T10:50:16.313 DEBUG:boto:Method: GET 2024-02-01T10:50:16.313 DEBUG:boto:Path: /vclfcn-64/obj4?versionId=null 2024-02-01T10:50:16.313 DEBUG:boto:Data: 2024-02-01T10:50:16.314 DEBUG:boto:Headers: {} 2024-02-01T10:50:16.314 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:16.314 DEBUG:boto:Port: 8000 2024-02-01T10:50:16.314 DEBUG:boto:Params: {} 2024-02-01T10:50:16.315 DEBUG:boto:Token: None 2024-02-01T10:50:16.315 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:16 GMT /vclfcn-64/obj4?versionId=null 2024-02-01T10:50:16.315 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gP8/IDz2AyGKpnzleuqMCpY/uj4= 2024-02-01T10:50:16.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gP8/IDz2AyGKpnzleuqMCpY/uj4=', 'Content-Length': '0'} 2024-02-01T10:50:16.319 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000042985601340caa31-0065bb7768-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:16.320 DEBUG:boto:path=/vclfcn-64/obj4 2024-02-01T10:50:16.320 DEBUG:boto:auth_path=/vclfcn-64/obj4 2024-02-01T10:50:16.320 DEBUG:boto:path=/vclfcn-64/obj4?versionId=null 2024-02-01T10:50:16.320 DEBUG:boto:auth_path=/vclfcn-64/obj4?versionId=null 2024-02-01T10:50:16.320 DEBUG:boto:Method: HEAD 2024-02-01T10:50:16.320 DEBUG:boto:Path: /vclfcn-64/obj4?versionId=null 2024-02-01T10:50:16.320 DEBUG:boto:Data: 2024-02-01T10:50:16.321 DEBUG:boto:Headers: {} 2024-02-01T10:50:16.321 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:16.321 DEBUG:boto:Port: 8001 2024-02-01T10:50:16.321 DEBUG:boto:Params: {} 2024-02-01T10:50:16.321 DEBUG:boto:Token: None 2024-02-01T10:50:16.321 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:16 GMT /vclfcn-64/obj4?versionId=null 2024-02-01T10:50:16.321 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:U5RxmujxVycN7U5KOdbnr0C2jf8= 2024-02-01T10:50:16.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:U5RxmujxVycN7U5KOdbnr0C2jf8=', 'Content-Length': '0'} 2024-02-01T10:50:16.324 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001c3b5a46085a48d7-0065bb7768-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:16.324 DEBUG:boto:path=/vclfcn-64/obj4 2024-02-01T10:50:16.325 DEBUG:boto:auth_path=/vclfcn-64/obj4 2024-02-01T10:50:16.325 DEBUG:boto:path=/vclfcn-64/obj4?versionId=null 2024-02-01T10:50:16.325 DEBUG:boto:auth_path=/vclfcn-64/obj4?versionId=null 2024-02-01T10:50:16.325 DEBUG:boto:Method: HEAD 2024-02-01T10:50:16.325 DEBUG:boto:Path: /vclfcn-64/obj4?versionId=null 2024-02-01T10:50:16.326 DEBUG:boto:Data: 2024-02-01T10:50:16.326 DEBUG:boto:Headers: {} 2024-02-01T10:50:16.326 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:16.326 DEBUG:boto:Port: 8000 2024-02-01T10:50:16.326 DEBUG:boto:Params: {} 2024-02-01T10:50:16.327 DEBUG:boto:Token: None 2024-02-01T10:50:16.327 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:16 GMT /vclfcn-64/obj4?versionId=null 2024-02-01T10:50:16.327 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:U5RxmujxVycN7U5KOdbnr0C2jf8= 2024-02-01T10:50:16.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:U5RxmujxVycN7U5KOdbnr0C2jf8=', 'Content-Length': '0'} 2024-02-01T10:50:16.330 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000096d0b80be685bdb5-0065bb7768-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:16.330 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:50:16.330 DEBUG:boto:path=/vclfcn-64/obj4 2024-02-01T10:50:16.330 DEBUG:boto:auth_path=/vclfcn-64/obj4 2024-02-01T10:50:16.331 DEBUG:boto:path=/vclfcn-64/obj4?versionId=null 2024-02-01T10:50:16.331 DEBUG:boto:auth_path=/vclfcn-64/obj4?versionId=null 2024-02-01T10:50:16.331 DEBUG:boto:Method: GET 2024-02-01T10:50:16.331 DEBUG:boto:Path: /vclfcn-64/obj4?versionId=null 2024-02-01T10:50:16.331 DEBUG:boto:Data: 2024-02-01T10:50:16.331 DEBUG:boto:Headers: {} 2024-02-01T10:50:16.331 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:16.331 DEBUG:boto:Port: 8001 2024-02-01T10:50:16.331 DEBUG:boto:Params: {} 2024-02-01T10:50:16.331 DEBUG:boto:Token: None 2024-02-01T10:50:16.332 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:16 GMT /vclfcn-64/obj4?versionId=null 2024-02-01T10:50:16.332 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gP8/IDz2AyGKpnzleuqMCpY/uj4= 2024-02-01T10:50:16.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gP8/IDz2AyGKpnzleuqMCpY/uj4=', 'Content-Length': '0'} 2024-02-01T10:50:16.335 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c78b05ee523bddf6-0065bb7768-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:16.335 DEBUG:boto:path=/vclfcn-64/obj4 2024-02-01T10:50:16.335 DEBUG:boto:auth_path=/vclfcn-64/obj4 2024-02-01T10:50:16.336 DEBUG:boto:path=/vclfcn-64/obj4?versionId=null 2024-02-01T10:50:16.336 DEBUG:boto:auth_path=/vclfcn-64/obj4?versionId=null 2024-02-01T10:50:16.336 DEBUG:boto:Method: GET 2024-02-01T10:50:16.336 DEBUG:boto:Path: /vclfcn-64/obj4?versionId=null 2024-02-01T10:50:16.336 DEBUG:boto:Data: 2024-02-01T10:50:16.337 DEBUG:boto:Headers: {} 2024-02-01T10:50:16.337 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:16.337 DEBUG:boto:Port: 8000 2024-02-01T10:50:16.337 DEBUG:boto:Params: {} 2024-02-01T10:50:16.337 DEBUG:boto:Token: None 2024-02-01T10:50:16.338 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:16 GMT /vclfcn-64/obj4?versionId=null 2024-02-01T10:50:16.338 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gP8/IDz2AyGKpnzleuqMCpY/uj4= 2024-02-01T10:50:16.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gP8/IDz2AyGKpnzleuqMCpY/uj4=', 'Content-Length': '0'} 2024-02-01T10:50:16.341 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a9885bbac85dae7e-0065bb7768-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:16.384 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:16.384 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:16.384 DEBUG:boto:path=/vclfcn-64/?prefix=obj4 2024-02-01T10:50:16.385 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj4 2024-02-01T10:50:16.385 DEBUG:boto:Method: GET 2024-02-01T10:50:16.385 DEBUG:boto:Path: /vclfcn-64/?prefix=obj4 2024-02-01T10:50:16.385 DEBUG:boto:Data: 2024-02-01T10:50:16.385 DEBUG:boto:Headers: {} 2024-02-01T10:50:16.385 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:16.385 DEBUG:boto:Port: 8001 2024-02-01T10:50:16.385 DEBUG:boto:Params: {} 2024-02-01T10:50:16.385 DEBUG:boto:Token: None 2024-02-01T10:50:16.385 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:16 GMT /vclfcn-64/ 2024-02-01T10:50:16.386 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+KF5bKSSGsCmEAAkRMVXIkM3IK8= 2024-02-01T10:50:16.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+KF5bKSSGsCmEAAkRMVXIkM3IK8=', 'Content-Length': '0'} 2024-02-01T10:50:16.390 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000036544f1bd8daed1e-0065bb7768-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:16.391 DEBUG:boto:b'vclfcn-64obj41000falseobj42024-02-01T10:48:26.689Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:16.391 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:16.391 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:16.391 DEBUG:boto:path=/vclfcn-64/?prefix=obj4 2024-02-01T10:50:16.392 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj4 2024-02-01T10:50:16.392 DEBUG:boto:Method: GET 2024-02-01T10:50:16.392 DEBUG:boto:Path: /vclfcn-64/?prefix=obj4 2024-02-01T10:50:16.392 DEBUG:boto:Data: 2024-02-01T10:50:16.392 DEBUG:boto:Headers: {} 2024-02-01T10:50:16.392 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:16.392 DEBUG:boto:Port: 8000 2024-02-01T10:50:16.392 DEBUG:boto:Params: {} 2024-02-01T10:50:16.392 DEBUG:boto:Token: None 2024-02-01T10:50:16.392 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:16 GMT /vclfcn-64/ 2024-02-01T10:50:16.393 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+KF5bKSSGsCmEAAkRMVXIkM3IK8= 2024-02-01T10:50:16.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+KF5bKSSGsCmEAAkRMVXIkM3IK8=', 'Content-Length': '0'} 2024-02-01T10:50:16.397 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aafcacf2398c9087-0065bb7768-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:16.444 DEBUG:boto:b'vclfcn-64obj41000falseobj42024-02-01T10:48:26.689Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:16.445 DEBUG:boto:path=/vclfcn-64/obj4 2024-02-01T10:50:16.445 DEBUG:boto:auth_path=/vclfcn-64/obj4 2024-02-01T10:50:16.445 DEBUG:boto:Method: HEAD 2024-02-01T10:50:16.445 DEBUG:boto:Path: /vclfcn-64/obj4 2024-02-01T10:50:16.445 DEBUG:boto:Data: 2024-02-01T10:50:16.445 DEBUG:boto:Headers: {} 2024-02-01T10:50:16.445 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:16.445 DEBUG:boto:Port: 8001 2024-02-01T10:50:16.445 DEBUG:boto:Params: {} 2024-02-01T10:50:16.445 DEBUG:boto:Token: None 2024-02-01T10:50:16.446 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:16 GMT /vclfcn-64/obj4 2024-02-01T10:50:16.446 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Iy0XnNudaiB2+9y3iyVLKMYC8TI= 2024-02-01T10:50:16.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Iy0XnNudaiB2+9y3iyVLKMYC8TI=', 'Content-Length': '0'} 2024-02-01T10:50:16.448 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000029837208c359fdd5-0065bb7768-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:16.449 DEBUG:boto:path=/vclfcn-64/obj4 2024-02-01T10:50:16.449 DEBUG:boto:auth_path=/vclfcn-64/obj4 2024-02-01T10:50:16.449 DEBUG:boto:Method: HEAD 2024-02-01T10:50:16.449 DEBUG:boto:Path: /vclfcn-64/obj4 2024-02-01T10:50:16.449 DEBUG:boto:Data: 2024-02-01T10:50:16.449 DEBUG:boto:Headers: {} 2024-02-01T10:50:16.449 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:16.449 DEBUG:boto:Port: 8000 2024-02-01T10:50:16.450 DEBUG:boto:Params: {} 2024-02-01T10:50:16.450 DEBUG:boto:Token: None 2024-02-01T10:50:16.450 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:16 GMT /vclfcn-64/obj4 2024-02-01T10:50:16.450 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Iy0XnNudaiB2+9y3iyVLKMYC8TI= 2024-02-01T10:50:16.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Iy0XnNudaiB2+9y3iyVLKMYC8TI=', 'Content-Length': '0'} 2024-02-01T10:50:16.453 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bd6c4fb7ae1d02ad-0065bb7768-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:16.453 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-02-01T10:50:16.454 DEBUG:boto:path=/vclfcn-64/obj4 2024-02-01T10:50:16.454 DEBUG:boto:auth_path=/vclfcn-64/obj4 2024-02-01T10:50:16.454 DEBUG:boto:Method: GET 2024-02-01T10:50:16.454 DEBUG:boto:Path: /vclfcn-64/obj4 2024-02-01T10:50:16.454 DEBUG:boto:Data: 2024-02-01T10:50:16.454 DEBUG:boto:Headers: {} 2024-02-01T10:50:16.454 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:16.454 DEBUG:boto:Port: 8001 2024-02-01T10:50:16.455 DEBUG:boto:Params: {} 2024-02-01T10:50:16.455 DEBUG:boto:Token: None 2024-02-01T10:50:16.455 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:16 GMT /vclfcn-64/obj4 2024-02-01T10:50:16.455 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7ndF/LylwcwPH9d87YPTxszHmIc= 2024-02-01T10:50:16.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7ndF/LylwcwPH9d87YPTxszHmIc=', 'Content-Length': '0'} 2024-02-01T10:50:16.458 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000069f9030e0e8cc0fd-0065bb7768-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:16.500 DEBUG:boto:path=/vclfcn-64/obj4 2024-02-01T10:50:16.501 DEBUG:boto:auth_path=/vclfcn-64/obj4 2024-02-01T10:50:16.501 DEBUG:boto:Method: GET 2024-02-01T10:50:16.501 DEBUG:boto:Path: /vclfcn-64/obj4 2024-02-01T10:50:16.502 DEBUG:boto:Data: 2024-02-01T10:50:16.502 DEBUG:boto:Headers: {} 2024-02-01T10:50:16.502 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:16.502 DEBUG:boto:Port: 8000 2024-02-01T10:50:16.502 DEBUG:boto:Params: {} 2024-02-01T10:50:16.503 DEBUG:boto:Token: None 2024-02-01T10:50:16.503 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:16 GMT /vclfcn-64/obj4 2024-02-01T10:50:16.503 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:7ndF/LylwcwPH9d87YPTxszHmIc= 2024-02-01T10:50:16.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:7ndF/LylwcwPH9d87YPTxszHmIc=', 'Content-Length': '0'} 2024-02-01T10:50:16.506 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:48:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008d3952c927f12580-0065bb7768-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:16.552 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:16.552 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:16.552 DEBUG:boto:path=/vclfcn-64/?prefix=obj4 2024-02-01T10:50:16.553 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj4 2024-02-01T10:50:16.553 DEBUG:boto:Method: GET 2024-02-01T10:50:16.553 DEBUG:boto:Path: /vclfcn-64/?prefix=obj4 2024-02-01T10:50:16.553 DEBUG:boto:Data: 2024-02-01T10:50:16.553 DEBUG:boto:Headers: {} 2024-02-01T10:50:16.553 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:16.553 DEBUG:boto:Port: 8001 2024-02-01T10:50:16.553 DEBUG:boto:Params: {} 2024-02-01T10:50:16.553 DEBUG:boto:Token: None 2024-02-01T10:50:16.554 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:16 GMT /vclfcn-64/ 2024-02-01T10:50:16.554 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+KF5bKSSGsCmEAAkRMVXIkM3IK8= 2024-02-01T10:50:16.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+KF5bKSSGsCmEAAkRMVXIkM3IK8=', 'Content-Length': '0'} 2024-02-01T10:50:16.558 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042a1930df0f5b557-0065bb7768-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:16.558 DEBUG:boto:b'vclfcn-64obj41000falseobj42024-02-01T10:48:26.689Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:16.559 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:16.559 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:16.559 DEBUG:boto:path=/vclfcn-64/?prefix=obj4 2024-02-01T10:50:16.559 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj4 2024-02-01T10:50:16.559 DEBUG:boto:Method: GET 2024-02-01T10:50:16.559 DEBUG:boto:Path: /vclfcn-64/?prefix=obj4 2024-02-01T10:50:16.560 DEBUG:boto:Data: 2024-02-01T10:50:16.560 DEBUG:boto:Headers: {} 2024-02-01T10:50:16.560 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:16.560 DEBUG:boto:Port: 8000 2024-02-01T10:50:16.560 DEBUG:boto:Params: {} 2024-02-01T10:50:16.560 DEBUG:boto:Token: None 2024-02-01T10:50:16.560 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:16 GMT /vclfcn-64/ 2024-02-01T10:50:16.560 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+KF5bKSSGsCmEAAkRMVXIkM3IK8= 2024-02-01T10:50:16.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+KF5bKSSGsCmEAAkRMVXIkM3IK8=', 'Content-Length': '0'} 2024-02-01T10:50:16.565 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d23965e8a3c0d6d4-0065bb7768-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:16.565 DEBUG:boto:b'vclfcn-64obj41000falseobj42024-02-01T10:48:26.689Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:16.565 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-02-01T10:50:16.565 DEBUG:boto:path=/vclfcn-64/obj5 2024-02-01T10:50:16.566 DEBUG:boto:auth_path=/vclfcn-64/obj5 2024-02-01T10:50:16.566 DEBUG:boto:path=/vclfcn-64/obj5?versionId=null 2024-02-01T10:50:16.566 DEBUG:boto:auth_path=/vclfcn-64/obj5?versionId=null 2024-02-01T10:50:16.566 DEBUG:boto:Method: GET 2024-02-01T10:50:16.566 DEBUG:boto:Path: /vclfcn-64/obj5?versionId=null 2024-02-01T10:50:16.566 DEBUG:boto:Data: 2024-02-01T10:50:16.566 DEBUG:boto:Headers: {} 2024-02-01T10:50:16.566 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:16.566 DEBUG:boto:Port: 8001 2024-02-01T10:50:16.566 DEBUG:boto:Params: {} 2024-02-01T10:50:16.566 DEBUG:boto:Token: None 2024-02-01T10:50:16.567 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:16 GMT /vclfcn-64/obj5?versionId=null 2024-02-01T10:50:16.567 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/kNvdCfioM9R6kcoJhvaqq6UBXY= 2024-02-01T10:50:16.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/kNvdCfioM9R6kcoJhvaqq6UBXY=', 'Content-Length': '0'} 2024-02-01T10:50:16.570 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002b9f9650c60d54cf-0065bb7768-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:16.570 DEBUG:boto:path=/vclfcn-64/obj5 2024-02-01T10:50:16.570 DEBUG:boto:auth_path=/vclfcn-64/obj5 2024-02-01T10:50:16.570 DEBUG:boto:path=/vclfcn-64/obj5?versionId=null 2024-02-01T10:50:16.570 DEBUG:boto:auth_path=/vclfcn-64/obj5?versionId=null 2024-02-01T10:50:16.570 DEBUG:boto:Method: GET 2024-02-01T10:50:16.570 DEBUG:boto:Path: /vclfcn-64/obj5?versionId=null 2024-02-01T10:50:16.570 DEBUG:boto:Data: 2024-02-01T10:50:16.571 DEBUG:boto:Headers: {} 2024-02-01T10:50:16.571 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:16.571 DEBUG:boto:Port: 8000 2024-02-01T10:50:16.571 DEBUG:boto:Params: {} 2024-02-01T10:50:16.571 DEBUG:boto:Token: None 2024-02-01T10:50:16.571 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:16 GMT /vclfcn-64/obj5?versionId=null 2024-02-01T10:50:16.571 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/kNvdCfioM9R6kcoJhvaqq6UBXY= 2024-02-01T10:50:16.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/kNvdCfioM9R6kcoJhvaqq6UBXY=', 'Content-Length': '0'} 2024-02-01T10:50:16.574 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005a257ffabf55d29e-0065bb7768-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:16.620 DEBUG:boto:path=/vclfcn-64/obj5 2024-02-01T10:50:16.620 DEBUG:boto:auth_path=/vclfcn-64/obj5 2024-02-01T10:50:16.620 DEBUG:boto:path=/vclfcn-64/obj5?versionId=null 2024-02-01T10:50:16.620 DEBUG:boto:auth_path=/vclfcn-64/obj5?versionId=null 2024-02-01T10:50:16.621 DEBUG:boto:Method: HEAD 2024-02-01T10:50:16.621 DEBUG:boto:Path: /vclfcn-64/obj5?versionId=null 2024-02-01T10:50:16.621 DEBUG:boto:Data: 2024-02-01T10:50:16.621 DEBUG:boto:Headers: {} 2024-02-01T10:50:16.621 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:16.621 DEBUG:boto:Port: 8001 2024-02-01T10:50:16.621 DEBUG:boto:Params: {} 2024-02-01T10:50:16.621 DEBUG:boto:Token: None 2024-02-01T10:50:16.623 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:16 GMT /vclfcn-64/obj5?versionId=null 2024-02-01T10:50:16.623 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:enEr7wnrP5VF4kwB8mpVVrIBiGM= 2024-02-01T10:50:16.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:enEr7wnrP5VF4kwB8mpVVrIBiGM=', 'Content-Length': '0'} 2024-02-01T10:50:16.626 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000070a8af49e2774531-0065bb7768-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:16.626 DEBUG:boto:path=/vclfcn-64/obj5 2024-02-01T10:50:16.627 DEBUG:boto:auth_path=/vclfcn-64/obj5 2024-02-01T10:50:16.627 DEBUG:boto:path=/vclfcn-64/obj5?versionId=null 2024-02-01T10:50:16.627 DEBUG:boto:auth_path=/vclfcn-64/obj5?versionId=null 2024-02-01T10:50:16.627 DEBUG:boto:Method: HEAD 2024-02-01T10:50:16.627 DEBUG:boto:Path: /vclfcn-64/obj5?versionId=null 2024-02-01T10:50:16.627 DEBUG:boto:Data: 2024-02-01T10:50:16.627 DEBUG:boto:Headers: {} 2024-02-01T10:50:16.627 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:16.627 DEBUG:boto:Port: 8000 2024-02-01T10:50:16.627 DEBUG:boto:Params: {} 2024-02-01T10:50:16.628 DEBUG:boto:Token: None 2024-02-01T10:50:16.628 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:16 GMT /vclfcn-64/obj5?versionId=null 2024-02-01T10:50:16.628 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:enEr7wnrP5VF4kwB8mpVVrIBiGM= 2024-02-01T10:50:16.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:enEr7wnrP5VF4kwB8mpVVrIBiGM=', 'Content-Length': '0'} 2024-02-01T10:50:16.631 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002dc6a18860143d35-0065bb7768-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:16.631 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-02-01T10:50:16.632 DEBUG:boto:path=/vclfcn-64/obj5 2024-02-01T10:50:16.632 DEBUG:boto:auth_path=/vclfcn-64/obj5 2024-02-01T10:50:16.632 DEBUG:boto:path=/vclfcn-64/obj5?versionId=null 2024-02-01T10:50:16.632 DEBUG:boto:auth_path=/vclfcn-64/obj5?versionId=null 2024-02-01T10:50:16.632 DEBUG:boto:Method: GET 2024-02-01T10:50:16.632 DEBUG:boto:Path: /vclfcn-64/obj5?versionId=null 2024-02-01T10:50:16.632 DEBUG:boto:Data: 2024-02-01T10:50:16.632 DEBUG:boto:Headers: {} 2024-02-01T10:50:16.632 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:16.633 DEBUG:boto:Port: 8001 2024-02-01T10:50:16.633 DEBUG:boto:Params: {} 2024-02-01T10:50:16.633 DEBUG:boto:Token: None 2024-02-01T10:50:16.633 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:16 GMT /vclfcn-64/obj5?versionId=null 2024-02-01T10:50:16.633 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/kNvdCfioM9R6kcoJhvaqq6UBXY= 2024-02-01T10:50:16.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/kNvdCfioM9R6kcoJhvaqq6UBXY=', 'Content-Length': '0'} 2024-02-01T10:50:16.636 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e23229b50ddb6779-0065bb7768-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:16.636 DEBUG:boto:path=/vclfcn-64/obj5 2024-02-01T10:50:16.637 DEBUG:boto:auth_path=/vclfcn-64/obj5 2024-02-01T10:50:16.637 DEBUG:boto:path=/vclfcn-64/obj5?versionId=null 2024-02-01T10:50:16.637 DEBUG:boto:auth_path=/vclfcn-64/obj5?versionId=null 2024-02-01T10:50:16.637 DEBUG:boto:Method: GET 2024-02-01T10:50:16.637 DEBUG:boto:Path: /vclfcn-64/obj5?versionId=null 2024-02-01T10:50:16.637 DEBUG:boto:Data: 2024-02-01T10:50:16.637 DEBUG:boto:Headers: {} 2024-02-01T10:50:16.637 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:16.637 DEBUG:boto:Port: 8000 2024-02-01T10:50:16.637 DEBUG:boto:Params: {} 2024-02-01T10:50:16.638 DEBUG:boto:Token: None 2024-02-01T10:50:16.638 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:16 GMT /vclfcn-64/obj5?versionId=null 2024-02-01T10:50:16.638 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:/kNvdCfioM9R6kcoJhvaqq6UBXY= 2024-02-01T10:50:16.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:/kNvdCfioM9R6kcoJhvaqq6UBXY=', 'Content-Length': '0'} 2024-02-01T10:50:16.641 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bb602fc8e244787d-0065bb7768-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:16.684 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:16.684 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:16.684 DEBUG:boto:path=/vclfcn-64/?prefix=obj5 2024-02-01T10:50:16.685 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj5 2024-02-01T10:50:16.685 DEBUG:boto:Method: GET 2024-02-01T10:50:16.685 DEBUG:boto:Path: /vclfcn-64/?prefix=obj5 2024-02-01T10:50:16.685 DEBUG:boto:Data: 2024-02-01T10:50:16.685 DEBUG:boto:Headers: {} 2024-02-01T10:50:16.685 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:16.685 DEBUG:boto:Port: 8001 2024-02-01T10:50:16.685 DEBUG:boto:Params: {} 2024-02-01T10:50:16.685 DEBUG:boto:Token: None 2024-02-01T10:50:16.686 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:16 GMT /vclfcn-64/ 2024-02-01T10:50:16.686 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+KF5bKSSGsCmEAAkRMVXIkM3IK8= 2024-02-01T10:50:16.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+KF5bKSSGsCmEAAkRMVXIkM3IK8=', 'Content-Length': '0'} 2024-02-01T10:50:16.690 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b69f11be301c655-0065bb7768-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:16.691 DEBUG:boto:b'vclfcn-64obj51000falseobj52024-02-01T10:49:22.402Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:16.691 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:16.691 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:16.691 DEBUG:boto:path=/vclfcn-64/?prefix=obj5 2024-02-01T10:50:16.692 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj5 2024-02-01T10:50:16.692 DEBUG:boto:Method: GET 2024-02-01T10:50:16.692 DEBUG:boto:Path: /vclfcn-64/?prefix=obj5 2024-02-01T10:50:16.692 DEBUG:boto:Data: 2024-02-01T10:50:16.692 DEBUG:boto:Headers: {} 2024-02-01T10:50:16.692 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:16.692 DEBUG:boto:Port: 8000 2024-02-01T10:50:16.692 DEBUG:boto:Params: {} 2024-02-01T10:50:16.692 DEBUG:boto:Token: None 2024-02-01T10:50:16.692 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:16 GMT /vclfcn-64/ 2024-02-01T10:50:16.693 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+KF5bKSSGsCmEAAkRMVXIkM3IK8= 2024-02-01T10:50:16.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+KF5bKSSGsCmEAAkRMVXIkM3IK8=', 'Content-Length': '0'} 2024-02-01T10:50:16.697 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed091543aa857548-0065bb7768-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:16.698 DEBUG:boto:b'vclfcn-64obj51000falseobj52024-02-01T10:49:22.402Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:16.698 DEBUG:boto:path=/vclfcn-64/obj5 2024-02-01T10:50:16.698 DEBUG:boto:auth_path=/vclfcn-64/obj5 2024-02-01T10:50:16.698 DEBUG:boto:Method: HEAD 2024-02-01T10:50:16.698 DEBUG:boto:Path: /vclfcn-64/obj5 2024-02-01T10:50:16.698 DEBUG:boto:Data: 2024-02-01T10:50:16.698 DEBUG:boto:Headers: {} 2024-02-01T10:50:16.699 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:16.699 DEBUG:boto:Port: 8001 2024-02-01T10:50:16.699 DEBUG:boto:Params: {} 2024-02-01T10:50:16.699 DEBUG:boto:Token: None 2024-02-01T10:50:16.699 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:16 GMT /vclfcn-64/obj5 2024-02-01T10:50:16.699 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZX+zP+u3SwUyk+rUk7hwe//hAFE= 2024-02-01T10:50:16.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZX+zP+u3SwUyk+rUk7hwe//hAFE=', 'Content-Length': '0'} 2024-02-01T10:50:16.702 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002c4264cdc4c6fc98-0065bb7768-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:16.702 DEBUG:boto:path=/vclfcn-64/obj5 2024-02-01T10:50:16.702 DEBUG:boto:auth_path=/vclfcn-64/obj5 2024-02-01T10:50:16.702 DEBUG:boto:Method: HEAD 2024-02-01T10:50:16.703 DEBUG:boto:Path: /vclfcn-64/obj5 2024-02-01T10:50:16.703 DEBUG:boto:Data: 2024-02-01T10:50:16.703 DEBUG:boto:Headers: {} 2024-02-01T10:50:16.703 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:16.703 DEBUG:boto:Port: 8000 2024-02-01T10:50:16.703 DEBUG:boto:Params: {} 2024-02-01T10:50:16.703 DEBUG:boto:Token: None 2024-02-01T10:50:16.703 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:16 GMT /vclfcn-64/obj5 2024-02-01T10:50:16.703 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:ZX+zP+u3SwUyk+rUk7hwe//hAFE= 2024-02-01T10:50:16.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:ZX+zP+u3SwUyk+rUk7hwe//hAFE=', 'Content-Length': '0'} 2024-02-01T10:50:16.706 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000026880a20c08239b6-0065bb7768-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:16.706 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-02-01T10:50:16.707 DEBUG:boto:path=/vclfcn-64/obj5 2024-02-01T10:50:16.707 DEBUG:boto:auth_path=/vclfcn-64/obj5 2024-02-01T10:50:16.707 DEBUG:boto:Method: GET 2024-02-01T10:50:16.707 DEBUG:boto:Path: /vclfcn-64/obj5 2024-02-01T10:50:16.707 DEBUG:boto:Data: 2024-02-01T10:50:16.707 DEBUG:boto:Headers: {} 2024-02-01T10:50:16.707 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:16.707 DEBUG:boto:Port: 8001 2024-02-01T10:50:16.707 DEBUG:boto:Params: {} 2024-02-01T10:50:16.707 DEBUG:boto:Token: None 2024-02-01T10:50:16.708 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:16 GMT /vclfcn-64/obj5 2024-02-01T10:50:16.708 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gVlSUeX/U0jpFWRsKu376S3hd/w= 2024-02-01T10:50:16.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gVlSUeX/U0jpFWRsKu376S3hd/w=', 'Content-Length': '0'} 2024-02-01T10:50:16.711 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000048c55f9488f7efaa-0065bb7768-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:16.752 DEBUG:boto:path=/vclfcn-64/obj5 2024-02-01T10:50:16.752 DEBUG:boto:auth_path=/vclfcn-64/obj5 2024-02-01T10:50:16.752 DEBUG:boto:Method: GET 2024-02-01T10:50:16.753 DEBUG:boto:Path: /vclfcn-64/obj5 2024-02-01T10:50:16.753 DEBUG:boto:Data: 2024-02-01T10:50:16.753 DEBUG:boto:Headers: {} 2024-02-01T10:50:16.753 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:16.753 DEBUG:boto:Port: 8000 2024-02-01T10:50:16.753 DEBUG:boto:Params: {} 2024-02-01T10:50:16.754 DEBUG:boto:Token: None 2024-02-01T10:50:16.754 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:16 GMT /vclfcn-64/obj5 2024-02-01T10:50:16.754 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gVlSUeX/U0jpFWRsKu376S3hd/w= 2024-02-01T10:50:16.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gVlSUeX/U0jpFWRsKu376S3hd/w=', 'Content-Length': '0'} 2024-02-01T10:50:16.757 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008dbce430a1612472-0065bb7768-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:16.800 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:16.800 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:16.801 DEBUG:boto:path=/vclfcn-64/?prefix=obj5 2024-02-01T10:50:16.801 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj5 2024-02-01T10:50:16.801 DEBUG:boto:Method: GET 2024-02-01T10:50:16.801 DEBUG:boto:Path: /vclfcn-64/?prefix=obj5 2024-02-01T10:50:16.801 DEBUG:boto:Data: 2024-02-01T10:50:16.801 DEBUG:boto:Headers: {} 2024-02-01T10:50:16.801 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:16.801 DEBUG:boto:Port: 8001 2024-02-01T10:50:16.801 DEBUG:boto:Params: {} 2024-02-01T10:50:16.801 DEBUG:boto:Token: None 2024-02-01T10:50:16.802 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:16 GMT /vclfcn-64/ 2024-02-01T10:50:16.802 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+KF5bKSSGsCmEAAkRMVXIkM3IK8= 2024-02-01T10:50:16.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+KF5bKSSGsCmEAAkRMVXIkM3IK8=', 'Content-Length': '0'} 2024-02-01T10:50:16.807 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ab81a7ceec702c8-0065bb7768-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:16.807 DEBUG:boto:b'vclfcn-64obj51000falseobj52024-02-01T10:49:22.402Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:16.807 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:16.808 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:16.808 DEBUG:boto:path=/vclfcn-64/?prefix=obj5 2024-02-01T10:50:16.808 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj5 2024-02-01T10:50:16.808 DEBUG:boto:Method: GET 2024-02-01T10:50:16.808 DEBUG:boto:Path: /vclfcn-64/?prefix=obj5 2024-02-01T10:50:16.808 DEBUG:boto:Data: 2024-02-01T10:50:16.808 DEBUG:boto:Headers: {} 2024-02-01T10:50:16.808 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:16.808 DEBUG:boto:Port: 8000 2024-02-01T10:50:16.808 DEBUG:boto:Params: {} 2024-02-01T10:50:16.809 DEBUG:boto:Token: None 2024-02-01T10:50:16.809 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:16 GMT /vclfcn-64/ 2024-02-01T10:50:16.809 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+KF5bKSSGsCmEAAkRMVXIkM3IK8= 2024-02-01T10:50:16.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+KF5bKSSGsCmEAAkRMVXIkM3IK8=', 'Content-Length': '0'} 2024-02-01T10:50:16.813 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091e290231b41f44c-0065bb7768-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:16.860 DEBUG:boto:b'vclfcn-64obj51000falseobj52024-02-01T10:49:22.402Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:16.861 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-02-01T10:50:16.861 DEBUG:boto:path=/vclfcn-64/obj6 2024-02-01T10:50:16.861 DEBUG:boto:auth_path=/vclfcn-64/obj6 2024-02-01T10:50:16.861 DEBUG:boto:path=/vclfcn-64/obj6?versionId=null 2024-02-01T10:50:16.861 DEBUG:boto:auth_path=/vclfcn-64/obj6?versionId=null 2024-02-01T10:50:16.861 DEBUG:boto:Method: GET 2024-02-01T10:50:16.861 DEBUG:boto:Path: /vclfcn-64/obj6?versionId=null 2024-02-01T10:50:16.862 DEBUG:boto:Data: 2024-02-01T10:50:16.862 DEBUG:boto:Headers: {} 2024-02-01T10:50:16.862 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:16.862 DEBUG:boto:Port: 8001 2024-02-01T10:50:16.862 DEBUG:boto:Params: {} 2024-02-01T10:50:16.862 DEBUG:boto:Token: None 2024-02-01T10:50:16.862 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:16 GMT /vclfcn-64/obj6?versionId=null 2024-02-01T10:50:16.862 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BJbXnmpdhOIT+pCwu1VJP5XgFbw= 2024-02-01T10:50:16.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BJbXnmpdhOIT+pCwu1VJP5XgFbw=', 'Content-Length': '0'} 2024-02-01T10:50:16.865 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000da96ae8ecae5130a-0065bb7768-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:16.866 DEBUG:boto:path=/vclfcn-64/obj6 2024-02-01T10:50:16.866 DEBUG:boto:auth_path=/vclfcn-64/obj6 2024-02-01T10:50:16.866 DEBUG:boto:path=/vclfcn-64/obj6?versionId=null 2024-02-01T10:50:16.866 DEBUG:boto:auth_path=/vclfcn-64/obj6?versionId=null 2024-02-01T10:50:16.866 DEBUG:boto:Method: GET 2024-02-01T10:50:16.866 DEBUG:boto:Path: /vclfcn-64/obj6?versionId=null 2024-02-01T10:50:16.866 DEBUG:boto:Data: 2024-02-01T10:50:16.867 DEBUG:boto:Headers: {} 2024-02-01T10:50:16.867 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:16.867 DEBUG:boto:Port: 8000 2024-02-01T10:50:16.867 DEBUG:boto:Params: {} 2024-02-01T10:50:16.867 DEBUG:boto:Token: None 2024-02-01T10:50:16.867 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:16 GMT /vclfcn-64/obj6?versionId=null 2024-02-01T10:50:16.867 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BJbXnmpdhOIT+pCwu1VJP5XgFbw= 2024-02-01T10:50:16.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BJbXnmpdhOIT+pCwu1VJP5XgFbw=', 'Content-Length': '0'} 2024-02-01T10:50:16.870 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000075934fe0de76fae2-0065bb7768-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:16.916 DEBUG:boto:path=/vclfcn-64/obj6 2024-02-01T10:50:16.916 DEBUG:boto:auth_path=/vclfcn-64/obj6 2024-02-01T10:50:16.916 DEBUG:boto:path=/vclfcn-64/obj6?versionId=null 2024-02-01T10:50:16.916 DEBUG:boto:auth_path=/vclfcn-64/obj6?versionId=null 2024-02-01T10:50:16.917 DEBUG:boto:Method: HEAD 2024-02-01T10:50:16.917 DEBUG:boto:Path: /vclfcn-64/obj6?versionId=null 2024-02-01T10:50:16.917 DEBUG:boto:Data: 2024-02-01T10:50:16.917 DEBUG:boto:Headers: {} 2024-02-01T10:50:16.917 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:16.917 DEBUG:boto:Port: 8001 2024-02-01T10:50:16.917 DEBUG:boto:Params: {} 2024-02-01T10:50:16.917 DEBUG:boto:Token: None 2024-02-01T10:50:16.917 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:16 GMT /vclfcn-64/obj6?versionId=null 2024-02-01T10:50:16.918 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:hUXPuMc7LPRa56R7t2Gbl6nYpLE= 2024-02-01T10:50:16.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:hUXPuMc7LPRa56R7t2Gbl6nYpLE=', 'Content-Length': '0'} 2024-02-01T10:50:16.921 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f53a048513192b1e-0065bb7768-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:16.921 DEBUG:boto:path=/vclfcn-64/obj6 2024-02-01T10:50:16.921 DEBUG:boto:auth_path=/vclfcn-64/obj6 2024-02-01T10:50:16.921 DEBUG:boto:path=/vclfcn-64/obj6?versionId=null 2024-02-01T10:50:16.921 DEBUG:boto:auth_path=/vclfcn-64/obj6?versionId=null 2024-02-01T10:50:16.921 DEBUG:boto:Method: HEAD 2024-02-01T10:50:16.922 DEBUG:boto:Path: /vclfcn-64/obj6?versionId=null 2024-02-01T10:50:16.922 DEBUG:boto:Data: 2024-02-01T10:50:16.922 DEBUG:boto:Headers: {} 2024-02-01T10:50:16.922 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:16.922 DEBUG:boto:Port: 8000 2024-02-01T10:50:16.922 DEBUG:boto:Params: {} 2024-02-01T10:50:16.922 DEBUG:boto:Token: None 2024-02-01T10:50:16.922 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:16 GMT /vclfcn-64/obj6?versionId=null 2024-02-01T10:50:16.922 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:hUXPuMc7LPRa56R7t2Gbl6nYpLE= 2024-02-01T10:50:16.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:hUXPuMc7LPRa56R7t2Gbl6nYpLE=', 'Content-Length': '0'} 2024-02-01T10:50:16.925 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000076e44ab330393495-0065bb7768-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:16.925 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-02-01T10:50:16.926 DEBUG:boto:path=/vclfcn-64/obj6 2024-02-01T10:50:16.926 DEBUG:boto:auth_path=/vclfcn-64/obj6 2024-02-01T10:50:16.926 DEBUG:boto:path=/vclfcn-64/obj6?versionId=null 2024-02-01T10:50:16.926 DEBUG:boto:auth_path=/vclfcn-64/obj6?versionId=null 2024-02-01T10:50:16.926 DEBUG:boto:Method: GET 2024-02-01T10:50:16.926 DEBUG:boto:Path: /vclfcn-64/obj6?versionId=null 2024-02-01T10:50:16.926 DEBUG:boto:Data: 2024-02-01T10:50:16.926 DEBUG:boto:Headers: {} 2024-02-01T10:50:16.926 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:16.926 DEBUG:boto:Port: 8001 2024-02-01T10:50:16.927 DEBUG:boto:Params: {} 2024-02-01T10:50:16.927 DEBUG:boto:Token: None 2024-02-01T10:50:16.927 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:16 GMT /vclfcn-64/obj6?versionId=null 2024-02-01T10:50:16.927 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BJbXnmpdhOIT+pCwu1VJP5XgFbw= 2024-02-01T10:50:16.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BJbXnmpdhOIT+pCwu1VJP5XgFbw=', 'Content-Length': '0'} 2024-02-01T10:50:16.929 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004cdb61137a185f91-0065bb7768-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:16.930 DEBUG:boto:path=/vclfcn-64/obj6 2024-02-01T10:50:16.930 DEBUG:boto:auth_path=/vclfcn-64/obj6 2024-02-01T10:50:16.930 DEBUG:boto:path=/vclfcn-64/obj6?versionId=null 2024-02-01T10:50:16.930 DEBUG:boto:auth_path=/vclfcn-64/obj6?versionId=null 2024-02-01T10:50:16.930 DEBUG:boto:Method: GET 2024-02-01T10:50:16.930 DEBUG:boto:Path: /vclfcn-64/obj6?versionId=null 2024-02-01T10:50:16.930 DEBUG:boto:Data: 2024-02-01T10:50:16.930 DEBUG:boto:Headers: {} 2024-02-01T10:50:16.930 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:16.931 DEBUG:boto:Port: 8000 2024-02-01T10:50:16.931 DEBUG:boto:Params: {} 2024-02-01T10:50:16.931 DEBUG:boto:Token: None 2024-02-01T10:50:16.931 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:16 GMT /vclfcn-64/obj6?versionId=null 2024-02-01T10:50:16.931 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:BJbXnmpdhOIT+pCwu1VJP5XgFbw= 2024-02-01T10:50:16.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:BJbXnmpdhOIT+pCwu1VJP5XgFbw=', 'Content-Length': '0'} 2024-02-01T10:50:16.934 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a2fcb0bd6b7e8200-0065bb7768-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:16.976 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:16.976 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:16.976 DEBUG:boto:path=/vclfcn-64/?prefix=obj6 2024-02-01T10:50:16.976 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj6 2024-02-01T10:50:16.977 DEBUG:boto:Method: GET 2024-02-01T10:50:16.977 DEBUG:boto:Path: /vclfcn-64/?prefix=obj6 2024-02-01T10:50:16.977 DEBUG:boto:Data: 2024-02-01T10:50:16.977 DEBUG:boto:Headers: {} 2024-02-01T10:50:16.977 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:16.977 DEBUG:boto:Port: 8001 2024-02-01T10:50:16.977 DEBUG:boto:Params: {} 2024-02-01T10:50:16.977 DEBUG:boto:Token: None 2024-02-01T10:50:16.977 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:16 GMT /vclfcn-64/ 2024-02-01T10:50:16.978 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+KF5bKSSGsCmEAAkRMVXIkM3IK8= 2024-02-01T10:50:16.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+KF5bKSSGsCmEAAkRMVXIkM3IK8=', 'Content-Length': '0'} 2024-02-01T10:50:16.982 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e7abec1bf18db67-0065bb7768-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:16.983 DEBUG:boto:b'vclfcn-64obj61000falseobj62024-02-01T10:49:22.455Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:16.983 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:16.983 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:16.983 DEBUG:boto:path=/vclfcn-64/?prefix=obj6 2024-02-01T10:50:16.983 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj6 2024-02-01T10:50:16.984 DEBUG:boto:Method: GET 2024-02-01T10:50:16.984 DEBUG:boto:Path: /vclfcn-64/?prefix=obj6 2024-02-01T10:50:16.984 DEBUG:boto:Data: 2024-02-01T10:50:16.984 DEBUG:boto:Headers: {} 2024-02-01T10:50:16.984 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:16.984 DEBUG:boto:Port: 8000 2024-02-01T10:50:16.984 DEBUG:boto:Params: {} 2024-02-01T10:50:16.984 DEBUG:boto:Token: None 2024-02-01T10:50:16.984 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:16 GMT /vclfcn-64/ 2024-02-01T10:50:16.985 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+KF5bKSSGsCmEAAkRMVXIkM3IK8= 2024-02-01T10:50:16.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+KF5bKSSGsCmEAAkRMVXIkM3IK8=', 'Content-Length': '0'} 2024-02-01T10:50:16.989 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6ba9a994f110c16-0065bb7768-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:16.989 DEBUG:boto:b'vclfcn-64obj61000falseobj62024-02-01T10:49:22.455Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:16.989 DEBUG:boto:path=/vclfcn-64/obj6 2024-02-01T10:50:16.989 DEBUG:boto:auth_path=/vclfcn-64/obj6 2024-02-01T10:50:16.990 DEBUG:boto:Method: HEAD 2024-02-01T10:50:16.990 DEBUG:boto:Path: /vclfcn-64/obj6 2024-02-01T10:50:16.990 DEBUG:boto:Data: 2024-02-01T10:50:16.990 DEBUG:boto:Headers: {} 2024-02-01T10:50:16.990 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:16.990 DEBUG:boto:Port: 8001 2024-02-01T10:50:16.990 DEBUG:boto:Params: {} 2024-02-01T10:50:16.990 DEBUG:boto:Token: None 2024-02-01T10:50:16.990 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:16 GMT /vclfcn-64/obj6 2024-02-01T10:50:16.991 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:DLoquoIf7qTb94kgvSf8L2jzhoU= 2024-02-01T10:50:16.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:DLoquoIf7qTb94kgvSf8L2jzhoU=', 'Content-Length': '0'} 2024-02-01T10:50:16.993 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000371041943c42ae34-0065bb7768-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:16.993 DEBUG:boto:path=/vclfcn-64/obj6 2024-02-01T10:50:16.993 DEBUG:boto:auth_path=/vclfcn-64/obj6 2024-02-01T10:50:16.994 DEBUG:boto:Method: HEAD 2024-02-01T10:50:16.994 DEBUG:boto:Path: /vclfcn-64/obj6 2024-02-01T10:50:16.994 DEBUG:boto:Data: 2024-02-01T10:50:16.994 DEBUG:boto:Headers: {} 2024-02-01T10:50:16.994 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:16.994 DEBUG:boto:Port: 8000 2024-02-01T10:50:16.994 DEBUG:boto:Params: {} 2024-02-01T10:50:16.994 DEBUG:boto:Token: None 2024-02-01T10:50:16.994 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:16 GMT /vclfcn-64/obj6 2024-02-01T10:50:16.995 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:DLoquoIf7qTb94kgvSf8L2jzhoU= 2024-02-01T10:50:16.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:DLoquoIf7qTb94kgvSf8L2jzhoU=', 'Content-Length': '0'} 2024-02-01T10:50:16.997 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aad588c9a6bb632a-0065bb7768-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:16 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:16.997 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-02-01T10:50:16.997 DEBUG:boto:path=/vclfcn-64/obj6 2024-02-01T10:50:16.998 DEBUG:boto:auth_path=/vclfcn-64/obj6 2024-02-01T10:50:16.998 DEBUG:boto:Method: GET 2024-02-01T10:50:16.998 DEBUG:boto:Path: /vclfcn-64/obj6 2024-02-01T10:50:16.998 DEBUG:boto:Data: 2024-02-01T10:50:16.998 DEBUG:boto:Headers: {} 2024-02-01T10:50:16.998 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:16.998 DEBUG:boto:Port: 8001 2024-02-01T10:50:16.998 DEBUG:boto:Params: {} 2024-02-01T10:50:16.998 DEBUG:boto:Token: None 2024-02-01T10:50:16.999 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:16 GMT /vclfcn-64/obj6 2024-02-01T10:50:16.999 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:OofwZHyHfFAXKXcqrhuXV7TNGSo= 2024-02-01T10:50:16.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:16 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:OofwZHyHfFAXKXcqrhuXV7TNGSo=', 'Content-Length': '0'} 2024-02-01T10:50:17.001 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000062c49da72983b4d3-0065bb7769-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:17.044 DEBUG:boto:path=/vclfcn-64/obj6 2024-02-01T10:50:17.044 DEBUG:boto:auth_path=/vclfcn-64/obj6 2024-02-01T10:50:17.044 DEBUG:boto:Method: GET 2024-02-01T10:50:17.044 DEBUG:boto:Path: /vclfcn-64/obj6 2024-02-01T10:50:17.045 DEBUG:boto:Data: 2024-02-01T10:50:17.045 DEBUG:boto:Headers: {} 2024-02-01T10:50:17.045 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:17.045 DEBUG:boto:Port: 8000 2024-02-01T10:50:17.045 DEBUG:boto:Params: {} 2024-02-01T10:50:17.045 DEBUG:boto:Token: None 2024-02-01T10:50:17.045 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:17 GMT /vclfcn-64/obj6 2024-02-01T10:50:17.045 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:aLUgdSaRUvKiVPBCeM4jDLv0L/c= 2024-02-01T10:50:17.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:aLUgdSaRUvKiVPBCeM4jDLv0L/c=', 'Content-Length': '0'} 2024-02-01T10:50:17.048 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003e116be95b4270f5-0065bb7769-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:17.092 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:17.092 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:17.092 DEBUG:boto:path=/vclfcn-64/?prefix=obj6 2024-02-01T10:50:17.093 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj6 2024-02-01T10:50:17.093 DEBUG:boto:Method: GET 2024-02-01T10:50:17.093 DEBUG:boto:Path: /vclfcn-64/?prefix=obj6 2024-02-01T10:50:17.093 DEBUG:boto:Data: 2024-02-01T10:50:17.093 DEBUG:boto:Headers: {} 2024-02-01T10:50:17.093 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:17.093 DEBUG:boto:Port: 8001 2024-02-01T10:50:17.093 DEBUG:boto:Params: {} 2024-02-01T10:50:17.093 DEBUG:boto:Token: None 2024-02-01T10:50:17.094 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:17 GMT /vclfcn-64/ 2024-02-01T10:50:17.094 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:T1kufTqfIzhONjfIcG5BfyLd17s= 2024-02-01T10:50:17.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:T1kufTqfIzhONjfIcG5BfyLd17s=', 'Content-Length': '0'} 2024-02-01T10:50:17.098 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d33e3ed5ea3c0f0-0065bb7769-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:17.099 DEBUG:boto:b'vclfcn-64obj61000falseobj62024-02-01T10:49:22.455Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:17.099 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:17.099 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:17.099 DEBUG:boto:path=/vclfcn-64/?prefix=obj6 2024-02-01T10:50:17.099 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj6 2024-02-01T10:50:17.100 DEBUG:boto:Method: GET 2024-02-01T10:50:17.100 DEBUG:boto:Path: /vclfcn-64/?prefix=obj6 2024-02-01T10:50:17.100 DEBUG:boto:Data: 2024-02-01T10:50:17.100 DEBUG:boto:Headers: {} 2024-02-01T10:50:17.100 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:17.100 DEBUG:boto:Port: 8000 2024-02-01T10:50:17.100 DEBUG:boto:Params: {} 2024-02-01T10:50:17.100 DEBUG:boto:Token: None 2024-02-01T10:50:17.100 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:17 GMT /vclfcn-64/ 2024-02-01T10:50:17.101 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:T1kufTqfIzhONjfIcG5BfyLd17s= 2024-02-01T10:50:17.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:T1kufTqfIzhONjfIcG5BfyLd17s=', 'Content-Length': '0'} 2024-02-01T10:50:17.105 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000feb39c0f08cae260-0065bb7769-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:17.148 DEBUG:boto:b'vclfcn-64obj61000falseobj62024-02-01T10:49:22.455Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:17.149 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-02-01T10:50:17.149 DEBUG:boto:path=/vclfcn-64/obj7 2024-02-01T10:50:17.149 DEBUG:boto:auth_path=/vclfcn-64/obj7 2024-02-01T10:50:17.150 DEBUG:boto:path=/vclfcn-64/obj7?versionId=null 2024-02-01T10:50:17.150 DEBUG:boto:auth_path=/vclfcn-64/obj7?versionId=null 2024-02-01T10:50:17.150 DEBUG:boto:Method: GET 2024-02-01T10:50:17.150 DEBUG:boto:Path: /vclfcn-64/obj7?versionId=null 2024-02-01T10:50:17.150 DEBUG:boto:Data: 2024-02-01T10:50:17.150 DEBUG:boto:Headers: {} 2024-02-01T10:50:17.150 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:17.150 DEBUG:boto:Port: 8001 2024-02-01T10:50:17.150 DEBUG:boto:Params: {} 2024-02-01T10:50:17.151 DEBUG:boto:Token: None 2024-02-01T10:50:17.151 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:17 GMT /vclfcn-64/obj7?versionId=null 2024-02-01T10:50:17.151 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:f9jsM4hgCdpKv9dXQaQhOE1PAFY= 2024-02-01T10:50:17.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:f9jsM4hgCdpKv9dXQaQhOE1PAFY=', 'Content-Length': '0'} 2024-02-01T10:50:17.154 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000005c90dcd0e2e2bd7-0065bb7769-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:17.154 DEBUG:boto:path=/vclfcn-64/obj7 2024-02-01T10:50:17.154 DEBUG:boto:auth_path=/vclfcn-64/obj7 2024-02-01T10:50:17.154 DEBUG:boto:path=/vclfcn-64/obj7?versionId=null 2024-02-01T10:50:17.154 DEBUG:boto:auth_path=/vclfcn-64/obj7?versionId=null 2024-02-01T10:50:17.155 DEBUG:boto:Method: GET 2024-02-01T10:50:17.155 DEBUG:boto:Path: /vclfcn-64/obj7?versionId=null 2024-02-01T10:50:17.155 DEBUG:boto:Data: 2024-02-01T10:50:17.155 DEBUG:boto:Headers: {} 2024-02-01T10:50:17.155 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:17.155 DEBUG:boto:Port: 8000 2024-02-01T10:50:17.155 DEBUG:boto:Params: {} 2024-02-01T10:50:17.155 DEBUG:boto:Token: None 2024-02-01T10:50:17.155 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:17 GMT /vclfcn-64/obj7?versionId=null 2024-02-01T10:50:17.155 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:f9jsM4hgCdpKv9dXQaQhOE1PAFY= 2024-02-01T10:50:17.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:f9jsM4hgCdpKv9dXQaQhOE1PAFY=', 'Content-Length': '0'} 2024-02-01T10:50:17.159 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f95eae8008adc2ec-0065bb7769-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:17.159 DEBUG:boto:path=/vclfcn-64/obj7 2024-02-01T10:50:17.159 DEBUG:boto:auth_path=/vclfcn-64/obj7 2024-02-01T10:50:17.159 DEBUG:boto:path=/vclfcn-64/obj7?versionId=null 2024-02-01T10:50:17.159 DEBUG:boto:auth_path=/vclfcn-64/obj7?versionId=null 2024-02-01T10:50:17.160 DEBUG:boto:Method: HEAD 2024-02-01T10:50:17.160 DEBUG:boto:Path: /vclfcn-64/obj7?versionId=null 2024-02-01T10:50:17.160 DEBUG:boto:Data: 2024-02-01T10:50:17.160 DEBUG:boto:Headers: {} 2024-02-01T10:50:17.160 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:17.160 DEBUG:boto:Port: 8001 2024-02-01T10:50:17.160 DEBUG:boto:Params: {} 2024-02-01T10:50:17.161 DEBUG:boto:Token: None 2024-02-01T10:50:17.161 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:17 GMT /vclfcn-64/obj7?versionId=null 2024-02-01T10:50:17.161 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:A4nd+2WE0Sw2Ej354paYoOtOMYo= 2024-02-01T10:50:17.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:A4nd+2WE0Sw2Ej354paYoOtOMYo=', 'Content-Length': '0'} 2024-02-01T10:50:17.163 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ffb7efa68ad193b6-0065bb7769-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:17.164 DEBUG:boto:path=/vclfcn-64/obj7 2024-02-01T10:50:17.164 DEBUG:boto:auth_path=/vclfcn-64/obj7 2024-02-01T10:50:17.164 DEBUG:boto:path=/vclfcn-64/obj7?versionId=null 2024-02-01T10:50:17.164 DEBUG:boto:auth_path=/vclfcn-64/obj7?versionId=null 2024-02-01T10:50:17.164 DEBUG:boto:Method: HEAD 2024-02-01T10:50:17.164 DEBUG:boto:Path: /vclfcn-64/obj7?versionId=null 2024-02-01T10:50:17.164 DEBUG:boto:Data: 2024-02-01T10:50:17.164 DEBUG:boto:Headers: {} 2024-02-01T10:50:17.164 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:17.165 DEBUG:boto:Port: 8000 2024-02-01T10:50:17.165 DEBUG:boto:Params: {} 2024-02-01T10:50:17.165 DEBUG:boto:Token: None 2024-02-01T10:50:17.165 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:17 GMT /vclfcn-64/obj7?versionId=null 2024-02-01T10:50:17.165 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:A4nd+2WE0Sw2Ej354paYoOtOMYo= 2024-02-01T10:50:17.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:A4nd+2WE0Sw2Ej354paYoOtOMYo=', 'Content-Length': '0'} 2024-02-01T10:50:17.168 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003ffcf53df09e62da-0065bb7769-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:17.168 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-02-01T10:50:17.168 DEBUG:boto:path=/vclfcn-64/obj7 2024-02-01T10:50:17.169 DEBUG:boto:auth_path=/vclfcn-64/obj7 2024-02-01T10:50:17.169 DEBUG:boto:path=/vclfcn-64/obj7?versionId=null 2024-02-01T10:50:17.169 DEBUG:boto:auth_path=/vclfcn-64/obj7?versionId=null 2024-02-01T10:50:17.169 DEBUG:boto:Method: GET 2024-02-01T10:50:17.169 DEBUG:boto:Path: /vclfcn-64/obj7?versionId=null 2024-02-01T10:50:17.169 DEBUG:boto:Data: 2024-02-01T10:50:17.169 DEBUG:boto:Headers: {} 2024-02-01T10:50:17.169 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:17.169 DEBUG:boto:Port: 8001 2024-02-01T10:50:17.169 DEBUG:boto:Params: {} 2024-02-01T10:50:17.169 DEBUG:boto:Token: None 2024-02-01T10:50:17.170 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:17 GMT /vclfcn-64/obj7?versionId=null 2024-02-01T10:50:17.170 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:f9jsM4hgCdpKv9dXQaQhOE1PAFY= 2024-02-01T10:50:17.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:f9jsM4hgCdpKv9dXQaQhOE1PAFY=', 'Content-Length': '0'} 2024-02-01T10:50:17.173 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003afabb97da79c1bb-0065bb7769-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:17.173 DEBUG:boto:path=/vclfcn-64/obj7 2024-02-01T10:50:17.173 DEBUG:boto:auth_path=/vclfcn-64/obj7 2024-02-01T10:50:17.173 DEBUG:boto:path=/vclfcn-64/obj7?versionId=null 2024-02-01T10:50:17.173 DEBUG:boto:auth_path=/vclfcn-64/obj7?versionId=null 2024-02-01T10:50:17.173 DEBUG:boto:Method: GET 2024-02-01T10:50:17.174 DEBUG:boto:Path: /vclfcn-64/obj7?versionId=null 2024-02-01T10:50:17.174 DEBUG:boto:Data: 2024-02-01T10:50:17.174 DEBUG:boto:Headers: {} 2024-02-01T10:50:17.174 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:17.174 DEBUG:boto:Port: 8000 2024-02-01T10:50:17.174 DEBUG:boto:Params: {} 2024-02-01T10:50:17.174 DEBUG:boto:Token: None 2024-02-01T10:50:17.174 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:17 GMT /vclfcn-64/obj7?versionId=null 2024-02-01T10:50:17.174 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:f9jsM4hgCdpKv9dXQaQhOE1PAFY= 2024-02-01T10:50:17.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:f9jsM4hgCdpKv9dXQaQhOE1PAFY=', 'Content-Length': '0'} 2024-02-01T10:50:17.178 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000996eb72881bd1421-0065bb7769-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:17.220 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:17.220 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:17.220 DEBUG:boto:path=/vclfcn-64/?prefix=obj7 2024-02-01T10:50:17.221 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj7 2024-02-01T10:50:17.221 DEBUG:boto:Method: GET 2024-02-01T10:50:17.221 DEBUG:boto:Path: /vclfcn-64/?prefix=obj7 2024-02-01T10:50:17.221 DEBUG:boto:Data: 2024-02-01T10:50:17.221 DEBUG:boto:Headers: {} 2024-02-01T10:50:17.221 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:17.221 DEBUG:boto:Port: 8001 2024-02-01T10:50:17.221 DEBUG:boto:Params: {} 2024-02-01T10:50:17.222 DEBUG:boto:Token: None 2024-02-01T10:50:17.222 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:17 GMT /vclfcn-64/ 2024-02-01T10:50:17.222 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:T1kufTqfIzhONjfIcG5BfyLd17s= 2024-02-01T10:50:17.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:T1kufTqfIzhONjfIcG5BfyLd17s=', 'Content-Length': '0'} 2024-02-01T10:50:17.226 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a0f8c792d60c5a26-0065bb7769-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:17.227 DEBUG:boto:b'vclfcn-64obj71000falseobj72024-02-01T10:49:22.471Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:17.227 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:17.227 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:17.228 DEBUG:boto:path=/vclfcn-64/?prefix=obj7 2024-02-01T10:50:17.228 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj7 2024-02-01T10:50:17.228 DEBUG:boto:Method: GET 2024-02-01T10:50:17.228 DEBUG:boto:Path: /vclfcn-64/?prefix=obj7 2024-02-01T10:50:17.228 DEBUG:boto:Data: 2024-02-01T10:50:17.228 DEBUG:boto:Headers: {} 2024-02-01T10:50:17.228 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:17.228 DEBUG:boto:Port: 8000 2024-02-01T10:50:17.228 DEBUG:boto:Params: {} 2024-02-01T10:50:17.229 DEBUG:boto:Token: None 2024-02-01T10:50:17.229 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:17 GMT /vclfcn-64/ 2024-02-01T10:50:17.229 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:T1kufTqfIzhONjfIcG5BfyLd17s= 2024-02-01T10:50:17.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:T1kufTqfIzhONjfIcG5BfyLd17s=', 'Content-Length': '0'} 2024-02-01T10:50:17.233 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b05e2f5d2d7da7a-0065bb7769-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:17.276 DEBUG:boto:b'vclfcn-64obj71000falseobj72024-02-01T10:49:22.471Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:17.277 DEBUG:boto:path=/vclfcn-64/obj7 2024-02-01T10:50:17.277 DEBUG:boto:auth_path=/vclfcn-64/obj7 2024-02-01T10:50:17.277 DEBUG:boto:Method: HEAD 2024-02-01T10:50:17.277 DEBUG:boto:Path: /vclfcn-64/obj7 2024-02-01T10:50:17.277 DEBUG:boto:Data: 2024-02-01T10:50:17.277 DEBUG:boto:Headers: {} 2024-02-01T10:50:17.277 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:17.277 DEBUG:boto:Port: 8001 2024-02-01T10:50:17.277 DEBUG:boto:Params: {} 2024-02-01T10:50:17.278 DEBUG:boto:Token: None 2024-02-01T10:50:17.278 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:17 GMT /vclfcn-64/obj7 2024-02-01T10:50:17.278 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:DOx+CuS+oLTRGQwUCCNavHRG/nU= 2024-02-01T10:50:17.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:DOx+CuS+oLTRGQwUCCNavHRG/nU=', 'Content-Length': '0'} 2024-02-01T10:50:17.281 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008b7a642ef2a75d32-0065bb7769-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:17.281 DEBUG:boto:path=/vclfcn-64/obj7 2024-02-01T10:50:17.281 DEBUG:boto:auth_path=/vclfcn-64/obj7 2024-02-01T10:50:17.281 DEBUG:boto:Method: HEAD 2024-02-01T10:50:17.282 DEBUG:boto:Path: /vclfcn-64/obj7 2024-02-01T10:50:17.282 DEBUG:boto:Data: 2024-02-01T10:50:17.282 DEBUG:boto:Headers: {} 2024-02-01T10:50:17.282 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:17.282 DEBUG:boto:Port: 8000 2024-02-01T10:50:17.282 DEBUG:boto:Params: {} 2024-02-01T10:50:17.282 DEBUG:boto:Token: None 2024-02-01T10:50:17.282 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:17 GMT /vclfcn-64/obj7 2024-02-01T10:50:17.282 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:DOx+CuS+oLTRGQwUCCNavHRG/nU= 2024-02-01T10:50:17.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:DOx+CuS+oLTRGQwUCCNavHRG/nU=', 'Content-Length': '0'} 2024-02-01T10:50:17.285 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009e9a8c496e7897f9-0065bb7769-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:17.286 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-02-01T10:50:17.286 DEBUG:boto:path=/vclfcn-64/obj7 2024-02-01T10:50:17.286 DEBUG:boto:auth_path=/vclfcn-64/obj7 2024-02-01T10:50:17.286 DEBUG:boto:Method: GET 2024-02-01T10:50:17.286 DEBUG:boto:Path: /vclfcn-64/obj7 2024-02-01T10:50:17.286 DEBUG:boto:Data: 2024-02-01T10:50:17.286 DEBUG:boto:Headers: {} 2024-02-01T10:50:17.286 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:17.286 DEBUG:boto:Port: 8001 2024-02-01T10:50:17.287 DEBUG:boto:Params: {} 2024-02-01T10:50:17.287 DEBUG:boto:Token: None 2024-02-01T10:50:17.287 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:17 GMT /vclfcn-64/obj7 2024-02-01T10:50:17.287 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1rBhUm+ZHnQ8Nta6X6eFZrkk4+0= 2024-02-01T10:50:17.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1rBhUm+ZHnQ8Nta6X6eFZrkk4+0=', 'Content-Length': '0'} 2024-02-01T10:50:17.290 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e75a15d179384280-0065bb7769-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:17.290 DEBUG:boto:path=/vclfcn-64/obj7 2024-02-01T10:50:17.290 DEBUG:boto:auth_path=/vclfcn-64/obj7 2024-02-01T10:50:17.291 DEBUG:boto:Method: GET 2024-02-01T10:50:17.291 DEBUG:boto:Path: /vclfcn-64/obj7 2024-02-01T10:50:17.291 DEBUG:boto:Data: 2024-02-01T10:50:17.291 DEBUG:boto:Headers: {} 2024-02-01T10:50:17.291 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:17.291 DEBUG:boto:Port: 8000 2024-02-01T10:50:17.291 DEBUG:boto:Params: {} 2024-02-01T10:50:17.292 DEBUG:boto:Token: None 2024-02-01T10:50:17.292 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:17 GMT /vclfcn-64/obj7 2024-02-01T10:50:17.292 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:1rBhUm+ZHnQ8Nta6X6eFZrkk4+0= 2024-02-01T10:50:17.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:1rBhUm+ZHnQ8Nta6X6eFZrkk4+0=', 'Content-Length': '0'} 2024-02-01T10:50:17.295 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000013492234ffa557c0-0065bb7769-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:17.336 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:17.336 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:17.336 DEBUG:boto:path=/vclfcn-64/?prefix=obj7 2024-02-01T10:50:17.336 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj7 2024-02-01T10:50:17.336 DEBUG:boto:Method: GET 2024-02-01T10:50:17.337 DEBUG:boto:Path: /vclfcn-64/?prefix=obj7 2024-02-01T10:50:17.337 DEBUG:boto:Data: 2024-02-01T10:50:17.337 DEBUG:boto:Headers: {} 2024-02-01T10:50:17.337 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:17.337 DEBUG:boto:Port: 8001 2024-02-01T10:50:17.337 DEBUG:boto:Params: {} 2024-02-01T10:50:17.337 DEBUG:boto:Token: None 2024-02-01T10:50:17.337 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:17 GMT /vclfcn-64/ 2024-02-01T10:50:17.338 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:T1kufTqfIzhONjfIcG5BfyLd17s= 2024-02-01T10:50:17.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:T1kufTqfIzhONjfIcG5BfyLd17s=', 'Content-Length': '0'} 2024-02-01T10:50:17.342 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c794b768cae3514-0065bb7769-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:17.342 DEBUG:boto:b'vclfcn-64obj71000falseobj72024-02-01T10:49:22.471Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:17.343 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:17.343 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:17.343 DEBUG:boto:path=/vclfcn-64/?prefix=obj7 2024-02-01T10:50:17.343 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj7 2024-02-01T10:50:17.343 DEBUG:boto:Method: GET 2024-02-01T10:50:17.343 DEBUG:boto:Path: /vclfcn-64/?prefix=obj7 2024-02-01T10:50:17.343 DEBUG:boto:Data: 2024-02-01T10:50:17.344 DEBUG:boto:Headers: {} 2024-02-01T10:50:17.344 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:17.344 DEBUG:boto:Port: 8000 2024-02-01T10:50:17.344 DEBUG:boto:Params: {} 2024-02-01T10:50:17.344 DEBUG:boto:Token: None 2024-02-01T10:50:17.344 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:17 GMT /vclfcn-64/ 2024-02-01T10:50:17.344 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:T1kufTqfIzhONjfIcG5BfyLd17s= 2024-02-01T10:50:17.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:T1kufTqfIzhONjfIcG5BfyLd17s=', 'Content-Length': '0'} 2024-02-01T10:50:17.348 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004230290646eb3838-0065bb7769-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:17.392 DEBUG:boto:b'vclfcn-64obj71000falseobj72024-02-01T10:49:22.471Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:17.392 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-02-01T10:50:17.393 DEBUG:boto:path=/vclfcn-64/obj8 2024-02-01T10:50:17.393 DEBUG:boto:auth_path=/vclfcn-64/obj8 2024-02-01T10:50:17.393 DEBUG:boto:path=/vclfcn-64/obj8?versionId=null 2024-02-01T10:50:17.393 DEBUG:boto:auth_path=/vclfcn-64/obj8?versionId=null 2024-02-01T10:50:17.393 DEBUG:boto:Method: GET 2024-02-01T10:50:17.394 DEBUG:boto:Path: /vclfcn-64/obj8?versionId=null 2024-02-01T10:50:17.394 DEBUG:boto:Data: 2024-02-01T10:50:17.394 DEBUG:boto:Headers: {} 2024-02-01T10:50:17.394 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:17.394 DEBUG:boto:Port: 8001 2024-02-01T10:50:17.394 DEBUG:boto:Params: {} 2024-02-01T10:50:17.394 DEBUG:boto:Token: None 2024-02-01T10:50:17.395 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:17 GMT /vclfcn-64/obj8?versionId=null 2024-02-01T10:50:17.395 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gTC7TwJHOIKjkV1Y5Lg/DeDyikU= 2024-02-01T10:50:17.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gTC7TwJHOIKjkV1Y5Lg/DeDyikU=', 'Content-Length': '0'} 2024-02-01T10:50:17.398 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000dabe354e904853e-0065bb7769-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:17.398 DEBUG:boto:path=/vclfcn-64/obj8 2024-02-01T10:50:17.399 DEBUG:boto:auth_path=/vclfcn-64/obj8 2024-02-01T10:50:17.399 DEBUG:boto:path=/vclfcn-64/obj8?versionId=null 2024-02-01T10:50:17.399 DEBUG:boto:auth_path=/vclfcn-64/obj8?versionId=null 2024-02-01T10:50:17.399 DEBUG:boto:Method: GET 2024-02-01T10:50:17.399 DEBUG:boto:Path: /vclfcn-64/obj8?versionId=null 2024-02-01T10:50:17.399 DEBUG:boto:Data: 2024-02-01T10:50:17.399 DEBUG:boto:Headers: {} 2024-02-01T10:50:17.399 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:17.399 DEBUG:boto:Port: 8000 2024-02-01T10:50:17.399 DEBUG:boto:Params: {} 2024-02-01T10:50:17.400 DEBUG:boto:Token: None 2024-02-01T10:50:17.400 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:17 GMT /vclfcn-64/obj8?versionId=null 2024-02-01T10:50:17.400 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gTC7TwJHOIKjkV1Y5Lg/DeDyikU= 2024-02-01T10:50:17.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gTC7TwJHOIKjkV1Y5Lg/DeDyikU=', 'Content-Length': '0'} 2024-02-01T10:50:17.403 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008ef81697c748ae21-0065bb7769-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:17.403 DEBUG:boto:path=/vclfcn-64/obj8 2024-02-01T10:50:17.403 DEBUG:boto:auth_path=/vclfcn-64/obj8 2024-02-01T10:50:17.404 DEBUG:boto:path=/vclfcn-64/obj8?versionId=null 2024-02-01T10:50:17.404 DEBUG:boto:auth_path=/vclfcn-64/obj8?versionId=null 2024-02-01T10:50:17.404 DEBUG:boto:Method: HEAD 2024-02-01T10:50:17.404 DEBUG:boto:Path: /vclfcn-64/obj8?versionId=null 2024-02-01T10:50:17.404 DEBUG:boto:Data: 2024-02-01T10:50:17.404 DEBUG:boto:Headers: {} 2024-02-01T10:50:17.404 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:17.404 DEBUG:boto:Port: 8001 2024-02-01T10:50:17.404 DEBUG:boto:Params: {} 2024-02-01T10:50:17.405 DEBUG:boto:Token: None 2024-02-01T10:50:17.405 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:17 GMT /vclfcn-64/obj8?versionId=null 2024-02-01T10:50:17.405 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0PtXypja3mq9/I/4UckuSHbsufc= 2024-02-01T10:50:17.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0PtXypja3mq9/I/4UckuSHbsufc=', 'Content-Length': '0'} 2024-02-01T10:50:17.408 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000946dacb8071878fa-0065bb7769-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:17.408 DEBUG:boto:path=/vclfcn-64/obj8 2024-02-01T10:50:17.408 DEBUG:boto:auth_path=/vclfcn-64/obj8 2024-02-01T10:50:17.408 DEBUG:boto:path=/vclfcn-64/obj8?versionId=null 2024-02-01T10:50:17.408 DEBUG:boto:auth_path=/vclfcn-64/obj8?versionId=null 2024-02-01T10:50:17.408 DEBUG:boto:Method: HEAD 2024-02-01T10:50:17.409 DEBUG:boto:Path: /vclfcn-64/obj8?versionId=null 2024-02-01T10:50:17.409 DEBUG:boto:Data: 2024-02-01T10:50:17.409 DEBUG:boto:Headers: {} 2024-02-01T10:50:17.409 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:17.409 DEBUG:boto:Port: 8000 2024-02-01T10:50:17.409 DEBUG:boto:Params: {} 2024-02-01T10:50:17.409 DEBUG:boto:Token: None 2024-02-01T10:50:17.409 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:17 GMT /vclfcn-64/obj8?versionId=null 2024-02-01T10:50:17.409 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0PtXypja3mq9/I/4UckuSHbsufc= 2024-02-01T10:50:17.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0PtXypja3mq9/I/4UckuSHbsufc=', 'Content-Length': '0'} 2024-02-01T10:50:17.412 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000045d420773bef959e-0065bb7769-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:17.412 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-02-01T10:50:17.413 DEBUG:boto:path=/vclfcn-64/obj8 2024-02-01T10:50:17.413 DEBUG:boto:auth_path=/vclfcn-64/obj8 2024-02-01T10:50:17.413 DEBUG:boto:path=/vclfcn-64/obj8?versionId=null 2024-02-01T10:50:17.413 DEBUG:boto:auth_path=/vclfcn-64/obj8?versionId=null 2024-02-01T10:50:17.413 DEBUG:boto:Method: GET 2024-02-01T10:50:17.413 DEBUG:boto:Path: /vclfcn-64/obj8?versionId=null 2024-02-01T10:50:17.413 DEBUG:boto:Data: 2024-02-01T10:50:17.413 DEBUG:boto:Headers: {} 2024-02-01T10:50:17.414 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:17.414 DEBUG:boto:Port: 8001 2024-02-01T10:50:17.414 DEBUG:boto:Params: {} 2024-02-01T10:50:17.414 DEBUG:boto:Token: None 2024-02-01T10:50:17.414 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:17 GMT /vclfcn-64/obj8?versionId=null 2024-02-01T10:50:17.414 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gTC7TwJHOIKjkV1Y5Lg/DeDyikU= 2024-02-01T10:50:17.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gTC7TwJHOIKjkV1Y5Lg/DeDyikU=', 'Content-Length': '0'} 2024-02-01T10:50:17.417 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000079b3e6f33d3121f7-0065bb7769-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:17.460 DEBUG:boto:path=/vclfcn-64/obj8 2024-02-01T10:50:17.460 DEBUG:boto:auth_path=/vclfcn-64/obj8 2024-02-01T10:50:17.460 DEBUG:boto:path=/vclfcn-64/obj8?versionId=null 2024-02-01T10:50:17.460 DEBUG:boto:auth_path=/vclfcn-64/obj8?versionId=null 2024-02-01T10:50:17.461 DEBUG:boto:Method: GET 2024-02-01T10:50:17.461 DEBUG:boto:Path: /vclfcn-64/obj8?versionId=null 2024-02-01T10:50:17.461 DEBUG:boto:Data: 2024-02-01T10:50:17.461 DEBUG:boto:Headers: {} 2024-02-01T10:50:17.461 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:17.461 DEBUG:boto:Port: 8000 2024-02-01T10:50:17.461 DEBUG:boto:Params: {} 2024-02-01T10:50:17.461 DEBUG:boto:Token: None 2024-02-01T10:50:17.461 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:17 GMT /vclfcn-64/obj8?versionId=null 2024-02-01T10:50:17.462 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:gTC7TwJHOIKjkV1Y5Lg/DeDyikU= 2024-02-01T10:50:17.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:gTC7TwJHOIKjkV1Y5Lg/DeDyikU=', 'Content-Length': '0'} 2024-02-01T10:50:17.465 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a69bf2c6f652ba9c-0065bb7769-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:17.509 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:17.509 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:17.509 DEBUG:boto:path=/vclfcn-64/?prefix=obj8 2024-02-01T10:50:17.509 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj8 2024-02-01T10:50:17.509 DEBUG:boto:Method: GET 2024-02-01T10:50:17.509 DEBUG:boto:Path: /vclfcn-64/?prefix=obj8 2024-02-01T10:50:17.509 DEBUG:boto:Data: 2024-02-01T10:50:17.509 DEBUG:boto:Headers: {} 2024-02-01T10:50:17.510 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:17.510 DEBUG:boto:Port: 8001 2024-02-01T10:50:17.510 DEBUG:boto:Params: {} 2024-02-01T10:50:17.510 DEBUG:boto:Token: None 2024-02-01T10:50:17.510 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:17 GMT /vclfcn-64/ 2024-02-01T10:50:17.510 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:T1kufTqfIzhONjfIcG5BfyLd17s= 2024-02-01T10:50:17.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:T1kufTqfIzhONjfIcG5BfyLd17s=', 'Content-Length': '0'} 2024-02-01T10:50:17.514 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f644cbb16422848d-0065bb7769-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:17.514 DEBUG:boto:b'vclfcn-64obj81000falseobj82024-02-01T10:49:22.519Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:17.515 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:17.515 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:17.515 DEBUG:boto:path=/vclfcn-64/?prefix=obj8 2024-02-01T10:50:17.515 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj8 2024-02-01T10:50:17.515 DEBUG:boto:Method: GET 2024-02-01T10:50:17.516 DEBUG:boto:Path: /vclfcn-64/?prefix=obj8 2024-02-01T10:50:17.516 DEBUG:boto:Data: 2024-02-01T10:50:17.516 DEBUG:boto:Headers: {} 2024-02-01T10:50:17.516 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:17.516 DEBUG:boto:Port: 8000 2024-02-01T10:50:17.516 DEBUG:boto:Params: {} 2024-02-01T10:50:17.516 DEBUG:boto:Token: None 2024-02-01T10:50:17.516 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:17 GMT /vclfcn-64/ 2024-02-01T10:50:17.516 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:T1kufTqfIzhONjfIcG5BfyLd17s= 2024-02-01T10:50:17.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:T1kufTqfIzhONjfIcG5BfyLd17s=', 'Content-Length': '0'} 2024-02-01T10:50:17.520 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fee1323069cb2b18-0065bb7769-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:17.521 DEBUG:boto:b'vclfcn-64obj81000falseobj82024-02-01T10:49:22.519Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:17.521 DEBUG:boto:path=/vclfcn-64/obj8 2024-02-01T10:50:17.521 DEBUG:boto:auth_path=/vclfcn-64/obj8 2024-02-01T10:50:17.521 DEBUG:boto:Method: HEAD 2024-02-01T10:50:17.521 DEBUG:boto:Path: /vclfcn-64/obj8 2024-02-01T10:50:17.522 DEBUG:boto:Data: 2024-02-01T10:50:17.522 DEBUG:boto:Headers: {} 2024-02-01T10:50:17.522 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:17.522 DEBUG:boto:Port: 8001 2024-02-01T10:50:17.522 DEBUG:boto:Params: {} 2024-02-01T10:50:17.522 DEBUG:boto:Token: None 2024-02-01T10:50:17.522 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:17 GMT /vclfcn-64/obj8 2024-02-01T10:50:17.522 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Bd9DNqzmMjDMV4Rt1K+XReVjwZA= 2024-02-01T10:50:17.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Bd9DNqzmMjDMV4Rt1K+XReVjwZA=', 'Content-Length': '0'} 2024-02-01T10:50:17.525 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007ad8375b26d59933-0065bb7769-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:17.525 DEBUG:boto:path=/vclfcn-64/obj8 2024-02-01T10:50:17.525 DEBUG:boto:auth_path=/vclfcn-64/obj8 2024-02-01T10:50:17.525 DEBUG:boto:Method: HEAD 2024-02-01T10:50:17.525 DEBUG:boto:Path: /vclfcn-64/obj8 2024-02-01T10:50:17.525 DEBUG:boto:Data: 2024-02-01T10:50:17.526 DEBUG:boto:Headers: {} 2024-02-01T10:50:17.526 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:17.526 DEBUG:boto:Port: 8000 2024-02-01T10:50:17.526 DEBUG:boto:Params: {} 2024-02-01T10:50:17.526 DEBUG:boto:Token: None 2024-02-01T10:50:17.526 DEBUG:boto:StringToSign: HEAD Thu, 01 Feb 2024 10:50:17 GMT /vclfcn-64/obj8 2024-02-01T10:50:17.526 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Bd9DNqzmMjDMV4Rt1K+XReVjwZA= 2024-02-01T10:50:17.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Bd9DNqzmMjDMV4Rt1K+XReVjwZA=', 'Content-Length': '0'} 2024-02-01T10:50:17.529 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000574a3576b4d0b988-0065bb7769-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:17.529 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-02-01T10:50:17.529 DEBUG:boto:path=/vclfcn-64/obj8 2024-02-01T10:50:17.529 DEBUG:boto:auth_path=/vclfcn-64/obj8 2024-02-01T10:50:17.529 DEBUG:boto:Method: GET 2024-02-01T10:50:17.530 DEBUG:boto:Path: /vclfcn-64/obj8 2024-02-01T10:50:17.530 DEBUG:boto:Data: 2024-02-01T10:50:17.530 DEBUG:boto:Headers: {} 2024-02-01T10:50:17.530 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:17.530 DEBUG:boto:Port: 8001 2024-02-01T10:50:17.530 DEBUG:boto:Params: {} 2024-02-01T10:50:17.530 DEBUG:boto:Token: None 2024-02-01T10:50:17.530 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:17 GMT /vclfcn-64/obj8 2024-02-01T10:50:17.530 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:PRJ5f1TH/EM+DQpQZh9vSpRifro= 2024-02-01T10:50:17.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:PRJ5f1TH/EM+DQpQZh9vSpRifro=', 'Content-Length': '0'} 2024-02-01T10:50:17.533 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000ccb814155f83035-0065bb7769-5762-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:17.533 DEBUG:boto:path=/vclfcn-64/obj8 2024-02-01T10:50:17.533 DEBUG:boto:auth_path=/vclfcn-64/obj8 2024-02-01T10:50:17.534 DEBUG:boto:Method: GET 2024-02-01T10:50:17.534 DEBUG:boto:Path: /vclfcn-64/obj8 2024-02-01T10:50:17.534 DEBUG:boto:Data: 2024-02-01T10:50:17.534 DEBUG:boto:Headers: {} 2024-02-01T10:50:17.534 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:17.534 DEBUG:boto:Port: 8000 2024-02-01T10:50:17.534 DEBUG:boto:Params: {} 2024-02-01T10:50:17.534 DEBUG:boto:Token: None 2024-02-01T10:50:17.534 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:17 GMT /vclfcn-64/obj8 2024-02-01T10:50:17.534 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:PRJ5f1TH/EM+DQpQZh9vSpRifro= 2024-02-01T10:50:17.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:PRJ5f1TH/EM+DQpQZh9vSpRifro=', 'Content-Length': '0'} 2024-02-01T10:50:17.537 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 01 Feb 2024 10:49:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '2ea581c3-3f48-49d7-a989-68024c5b8eac:vclfcn-64:ae00a0f7-b2ea-4f6c-be42-8cb92c1dbb01.5765.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007cd91cdbb088b3e6-0065bb7769-5350-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:17.580 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:17.580 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:17.580 DEBUG:boto:path=/vclfcn-64/?prefix=obj8 2024-02-01T10:50:17.580 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj8 2024-02-01T10:50:17.580 DEBUG:boto:Method: GET 2024-02-01T10:50:17.581 DEBUG:boto:Path: /vclfcn-64/?prefix=obj8 2024-02-01T10:50:17.581 DEBUG:boto:Data: 2024-02-01T10:50:17.581 DEBUG:boto:Headers: {} 2024-02-01T10:50:17.581 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:17.581 DEBUG:boto:Port: 8001 2024-02-01T10:50:17.581 DEBUG:boto:Params: {} 2024-02-01T10:50:17.581 DEBUG:boto:Token: None 2024-02-01T10:50:17.581 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:17 GMT /vclfcn-64/ 2024-02-01T10:50:17.581 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:T1kufTqfIzhONjfIcG5BfyLd17s= 2024-02-01T10:50:17.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:T1kufTqfIzhONjfIcG5BfyLd17s=', 'Content-Length': '0'} 2024-02-01T10:50:17.585 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018b23dcfd927a91b-0065bb7769-5762-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:17.586 DEBUG:boto:b'vclfcn-64obj81000falseobj82024-02-01T10:49:22.519Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:17.586 DEBUG:boto:path=/vclfcn-64/ 2024-02-01T10:50:17.586 DEBUG:boto:auth_path=/vclfcn-64/ 2024-02-01T10:50:17.586 DEBUG:boto:path=/vclfcn-64/?prefix=obj8 2024-02-01T10:50:17.586 DEBUG:boto:auth_path=/vclfcn-64/?prefix=obj8 2024-02-01T10:50:17.586 DEBUG:boto:Method: GET 2024-02-01T10:50:17.587 DEBUG:boto:Path: /vclfcn-64/?prefix=obj8 2024-02-01T10:50:17.587 DEBUG:boto:Data: 2024-02-01T10:50:17.587 DEBUG:boto:Headers: {} 2024-02-01T10:50:17.587 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:17.587 DEBUG:boto:Port: 8000 2024-02-01T10:50:17.587 DEBUG:boto:Params: {} 2024-02-01T10:50:17.587 DEBUG:boto:Token: None 2024-02-01T10:50:17.587 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:17 GMT /vclfcn-64/ 2024-02-01T10:50:17.587 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:T1kufTqfIzhONjfIcG5BfyLd17s= 2024-02-01T10:50:17.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:17 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:T1kufTqfIzhONjfIcG5BfyLd17s=', 'Content-Length': '0'} 2024-02-01T10:50:17.591 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b41432ed63e9070e-0065bb7769-5350-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:17 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:17.592 DEBUG:boto:b'vclfcn-64obj81000falseobj82024-02-01T10:49:22.519Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-02-01T10:50:17.592 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=vclfcn-64 zones={test-zone3, test-zone2} 2024-02-01T10:50:17.592 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone1 target=test-zone2 2024-02-01T10:50:17.592 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:50:17.592 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:50:17.593 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:50:17.634 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:50:17.634 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:50:17.721 INFO:teuthology.orchestra.run.smithi148.stdout:[{"marker":"00000000000000000000:00000000000000008120","last_update":"2024-02-01T10:49:36.087170Z"},{"marker":"00000000000000000000:00000000000000014600","last_update":"2024-02-01T10:49:36.080232Z"},{"marker":"00000000000000000000:00000000000000012626","last_update":"2024-02-01T10:49:23.019163Z"},{"marker":"00000000000000000000:00000000000000008840","last_update":"2024-02-01T10:49:36.035545Z"}] 2024-02-01T10:50:17.721 DEBUG:rgw_multi.tests:data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000008120', 1: '00000000000000000000:00000000000000014600', 2: '00000000000000000000:00000000000000012626', 3: '00000000000000000000:00000000000000008840'} 2024-02-01T10:50:17.722 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-02-01T10:50:17.722 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:50:17.722 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:50:17.722 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:50:17.767 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:50:17.768 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:50:17.856 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":12760245309917388665},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000008120","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:49:36.087170Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000014600","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:49:36.080232Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000012626","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:49:23.019163Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000008840","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:49:36.035545Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-02-01T10:50:17Z"} 2024-02-01T10:50:17.857 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":12760245309917388665},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000008120","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:49:36.087170Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000014600","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:49:36.080232Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000012626","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:49:23.019163Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000008840","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:49:36.035545Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-02-01T10:50:17Z"} 2024-02-01T10:50:17.857 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008120', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:49:36.087170Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000014600', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:49:36.080232Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012626', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:49:23.019163Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008840', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:49:36.035545Z'}}] 2024-02-01T10:50:17.857 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000008120', 1: '00000000000000000000:00000000000000014600', 2: '00000000000000000000:00000000000000012626', 3: '00000000000000000000:00000000000000008840'} 2024-02-01T10:50:17.857 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000008120', 1: '00000000000000000000:00000000000000014600', 2: '00000000000000000000:00000000000000012626', 3: '00000000000000000000:00000000000000008840'} 2024-02-01T10:50:17.857 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-02-01T10:50:17.857 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone1 target=test-zone3 2024-02-01T10:50:17.858 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:50:17.858 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:50:17.858 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:50:17.901 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:50:17.901 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:50:17.988 INFO:teuthology.orchestra.run.smithi148.stdout:[{"marker":"00000000000000000000:00000000000000008120","last_update":"2024-02-01T10:49:36.087170Z"},{"marker":"00000000000000000000:00000000000000014600","last_update":"2024-02-01T10:49:36.080232Z"},{"marker":"00000000000000000000:00000000000000012626","last_update":"2024-02-01T10:49:23.019163Z"},{"marker":"00000000000000000000:00000000000000008840","last_update":"2024-02-01T10:49:36.035545Z"}] 2024-02-01T10:50:17.989 DEBUG:rgw_multi.tests:data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000008120', 1: '00000000000000000000:00000000000000014600', 2: '00000000000000000000:00000000000000012626', 3: '00000000000000000000:00000000000000008840'} 2024-02-01T10:50:17.989 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2024-02-01T10:50:17.989 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:50:17.989 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:50:17.989 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:50:18.071 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:50:18.071 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:50:18.147 INFO:teuthology.orchestra.run.smithi148.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":8365149014541277257},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000008120","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:49:36.087170Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000014600","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:49:36.080232Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000012626","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:49:23.019163Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000008840","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:49:36.035545Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-02-01T10:50:18Z"} 2024-02-01T10:50:18.147 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":8365149014541277257},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000008120","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:49:36.087170Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000014600","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:49:36.080232Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000012626","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:49:23.019163Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000008840","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:49:36.035545Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-02-01T10:50:18Z"} 2024-02-01T10:50:18.147 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008120', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:49:36.087170Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000014600', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:49:36.080232Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012626', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:49:23.019163Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008840', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:49:36.035545Z'}}] 2024-02-01T10:50:18.148 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000008120', 1: '00000000000000000000:00000000000000014600', 2: '00000000000000000000:00000000000000012626', 3: '00000000000000000000:00000000000000008840'} 2024-02-01T10:50:18.148 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000008120', 1: '00000000000000000000:00000000000000014600', 2: '00000000000000000000:00000000000000012626', 3: '00000000000000000000:00000000000000008840'} 2024-02-01T10:50:18.148 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2024-02-01T10:50:18.148 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone2 target=test-zone1 2024-02-01T10:50:18.148 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:50:18.148 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:50:18.148 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:50:18.190 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:50:18.191 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:50:18.277 INFO:teuthology.orchestra.run.smithi187.stdout:[{"marker":"00000000000000000000:00000000000000006710","last_update":"2024-02-01T10:49:43.063692Z"},{"marker":"00000000000000000000:00000000000000012203","last_update":"2024-02-01T10:49:43.067052Z"},{"marker":"00000000000000000000:00000000000000009385","last_update":"2024-02-01T10:49:43.070232Z"},{"marker":"00000000000000000000:00000000000000006579","last_update":"2024-02-01T10:49:43.073118Z"}] 2024-02-01T10:50:18.277 DEBUG:rgw_multi.tests:data markers for zone=test-zone2 markers={0: '00000000000000000000:00000000000000006710', 1: '00000000000000000000:00000000000000012203', 2: '00000000000000000000:00000000000000009385', 3: '00000000000000000000:00000000000000006579'} 2024-02-01T10:50:18.277 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2024-02-01T10:50:18.277 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:50:18.278 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:50:18.278 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:50:18.323 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:50:18.324 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:50:18.407 INFO:teuthology.orchestra.run.smithi148.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":8595394196901466828},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000006710","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:49:43.063692Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000012203","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:49:43.067052Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000009385","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:49:43.070232Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000006579","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:49:43.073118Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-02-01T10:50:18Z"} 2024-02-01T10:50:18.407 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":8595394196901466828},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000006710","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:49:43.063692Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000012203","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:49:43.067052Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000009385","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:49:43.070232Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000006579","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:49:43.073118Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-02-01T10:50:18Z"} 2024-02-01T10:50:18.408 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006710', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:49:43.063692Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012203', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:49:43.067052Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000009385', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:49:43.070232Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006579', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:49:43.073118Z'}}] 2024-02-01T10:50:18.408 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000006710', 1: '00000000000000000000:00000000000000012203', 2: '00000000000000000000:00000000000000009385', 3: '00000000000000000000:00000000000000006579'} 2024-02-01T10:50:18.408 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000006710', 1: '00000000000000000000:00000000000000012203', 2: '00000000000000000000:00000000000000009385', 3: '00000000000000000000:00000000000000006579'} 2024-02-01T10:50:18.408 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2024-02-01T10:50:18.408 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone2 target=test-zone3 2024-02-01T10:50:18.408 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:50:18.408 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:50:18.409 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:50:18.452 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:50:18.452 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:50:18.546 INFO:teuthology.orchestra.run.smithi187.stdout:[{"marker":"00000000000000000000:00000000000000006710","last_update":"2024-02-01T10:49:43.063692Z"},{"marker":"00000000000000000000:00000000000000012203","last_update":"2024-02-01T10:49:43.067052Z"},{"marker":"00000000000000000000:00000000000000009385","last_update":"2024-02-01T10:49:43.070232Z"},{"marker":"00000000000000000000:00000000000000006579","last_update":"2024-02-01T10:49:43.073118Z"}] 2024-02-01T10:50:18.546 DEBUG:rgw_multi.tests:data markers for zone=test-zone2 markers={0: '00000000000000000000:00000000000000006710', 1: '00000000000000000000:00000000000000012203', 2: '00000000000000000000:00000000000000009385', 3: '00000000000000000000:00000000000000006579'} 2024-02-01T10:50:18.546 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2024-02-01T10:50:18.546 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:50:18.546 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:50:18.547 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:50:18.591 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:50:18.591 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:50:18.670 INFO:teuthology.orchestra.run.smithi148.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4461638765891857795},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000006710","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:49:43.063692Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000012203","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:49:43.067052Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000009385","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:49:43.070232Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000006579","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:49:43.073118Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-02-01T10:50:18Z"} 2024-02-01T10:50:18.671 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4461638765891857795},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000006710","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:49:43.063692Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000012203","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:49:43.067052Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000009385","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:49:43.070232Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000006579","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:49:43.073118Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-02-01T10:50:18Z"} 2024-02-01T10:50:18.671 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006710', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:49:43.063692Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012203', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:49:43.067052Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000009385', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:49:43.070232Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006579', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:49:43.073118Z'}}] 2024-02-01T10:50:18.671 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000006710', 1: '00000000000000000000:00000000000000012203', 2: '00000000000000000000:00000000000000009385', 3: '00000000000000000000:00000000000000006579'} 2024-02-01T10:50:18.671 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000006710', 1: '00000000000000000000:00000000000000012203', 2: '00000000000000000000:00000000000000009385', 3: '00000000000000000000:00000000000000006579'} 2024-02-01T10:50:18.671 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2024-02-01T10:50:18.671 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone3 target=test-zone1 2024-02-01T10:50:18.671 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:50:18.672 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:50:18.672 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:50:18.746 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:50:18.746 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:50:18.832 INFO:teuthology.orchestra.run.smithi148.stdout:[{"marker":"00000000000000000000:00000000000000007697","last_update":"2024-02-01T10:49:43.039031Z"},{"marker":"00000000000000000000:00000000000000012767","last_update":"2024-02-01T10:49:43.042621Z"},{"marker":"00000000000000000000:00000000000000011358","last_update":"2024-02-01T10:49:43.046508Z"},{"marker":"00000000000000000000:00000000000000007568","last_update":"2024-02-01T10:49:43.049955Z"}] 2024-02-01T10:50:18.832 DEBUG:rgw_multi.tests:data markers for zone=test-zone3 markers={0: '00000000000000000000:00000000000000007697', 1: '00000000000000000000:00000000000000012767', 2: '00000000000000000000:00000000000000011358', 3: '00000000000000000000:00000000000000007568'} 2024-02-01T10:50:18.832 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2024-02-01T10:50:18.833 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:50:18.833 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:50:18.833 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:50:18.911 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:50:18.911 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:50:18.991 INFO:teuthology.orchestra.run.smithi148.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":8579032341556541049},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000007697","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:49:43.039031Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000012767","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:49:43.042621Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000011358","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:49:43.046508Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000007568","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:49:43.049955Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-02-01T10:50:18Z"} 2024-02-01T10:50:18.991 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":8579032341556541049},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000007697","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:49:43.039031Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000012767","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:49:43.042621Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000011358","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:49:43.046508Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000007568","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:49:43.049955Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-02-01T10:50:18Z"} 2024-02-01T10:50:18.992 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007697', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:49:43.039031Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012767', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:49:43.042621Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011358', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:49:43.046508Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007568', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:49:43.049955Z'}}] 2024-02-01T10:50:18.992 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000007697', 1: '00000000000000000000:00000000000000012767', 2: '00000000000000000000:00000000000000011358', 3: '00000000000000000000:00000000000000007568'} 2024-02-01T10:50:18.992 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000007697', 1: '00000000000000000000:00000000000000012767', 2: '00000000000000000000:00000000000000011358', 3: '00000000000000000000:00000000000000007568'} 2024-02-01T10:50:18.992 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2024-02-01T10:50:18.992 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone3 target=test-zone2 2024-02-01T10:50:18.992 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:50:18.992 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:50:18.993 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:50:19.075 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:50:19.075 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:50:19.160 INFO:teuthology.orchestra.run.smithi148.stdout:[{"marker":"00000000000000000000:00000000000000007697","last_update":"2024-02-01T10:49:43.039031Z"},{"marker":"00000000000000000000:00000000000000012767","last_update":"2024-02-01T10:49:43.042621Z"},{"marker":"00000000000000000000:00000000000000011358","last_update":"2024-02-01T10:49:43.046508Z"},{"marker":"00000000000000000000:00000000000000007568","last_update":"2024-02-01T10:49:43.049955Z"}] 2024-02-01T10:50:19.160 DEBUG:rgw_multi.tests:data markers for zone=test-zone3 markers={0: '00000000000000000000:00000000000000007697', 1: '00000000000000000000:00000000000000012767', 2: '00000000000000000000:00000000000000011358', 3: '00000000000000000000:00000000000000007568'} 2024-02-01T10:50:19.161 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2024-02-01T10:50:19.161 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:50:19.161 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:50:19.161 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:50:19.206 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:50:19.206 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:50:19.291 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":843725689139632598},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000007697","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:49:43.039031Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000012767","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:49:43.042621Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000011358","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:49:43.046508Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000007568","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:49:43.049955Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-02-01T10:50:19Z"} 2024-02-01T10:50:19.292 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":843725689139632598},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000007697","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:49:43.039031Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000012767","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:49:43.042621Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000011358","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:49:43.046508Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000007568","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:49:43.049955Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-02-01T10:50:19Z"} 2024-02-01T10:50:19.292 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007697', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:49:43.039031Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012767', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:49:43.042621Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011358', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:49:43.046508Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007568', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:49:43.049955Z'}}] 2024-02-01T10:50:19.292 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000007697', 1: '00000000000000000000:00000000000000012767', 2: '00000000000000000000:00000000000000011358', 3: '00000000000000000000:00000000000000007568'} 2024-02-01T10:50:19.292 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000007697', 1: '00000000000000000000:00000000000000012767', 2: '00000000000000000000:00000000000000011358', 3: '00000000000000000000:00000000000000007568'} 2024-02-01T10:50:19.292 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2024-02-01T10:50:19.295 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_disable_enable ... ok 2024-02-01T10:50:19.298 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=vclfcn-65 2024-02-01T10:50:19.298 DEBUG:boto:path=/vclfcn-65/ 2024-02-01T10:50:19.299 DEBUG:boto:auth_path=/vclfcn-65/ 2024-02-01T10:50:19.299 DEBUG:boto:Method: PUT 2024-02-01T10:50:19.299 DEBUG:boto:Path: /vclfcn-65/ 2024-02-01T10:50:19.299 DEBUG:boto:Data: 2024-02-01T10:50:19.299 DEBUG:boto:Headers: {} 2024-02-01T10:50:19.299 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:19.299 DEBUG:boto:Port: 8000 2024-02-01T10:50:19.299 DEBUG:boto:Params: {} 2024-02-01T10:50:19.300 DEBUG:boto:Token: None 2024-02-01T10:50:19.300 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:50:19 GMT /vclfcn-65/ 2024-02-01T10:50:19.300 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+OXp7r+uCbWCoaoBf5aitrvsav0= 2024-02-01T10:50:19.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+OXp7r+uCbWCoaoBf5aitrvsav0=', 'Content-Length': '0'} 2024-02-01T10:50:19.332 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000bbb34bbc819827b5-0065bb776b-5765-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:50:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:19.332 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=vclfcn-66 2024-02-01T10:50:19.333 DEBUG:boto:path=/vclfcn-66/ 2024-02-01T10:50:19.333 DEBUG:boto:auth_path=/vclfcn-66/ 2024-02-01T10:50:19.333 DEBUG:boto:Method: PUT 2024-02-01T10:50:19.333 DEBUG:boto:Path: /vclfcn-66/ 2024-02-01T10:50:19.333 DEBUG:boto:Data: 2024-02-01T10:50:19.333 DEBUG:boto:Headers: {} 2024-02-01T10:50:19.333 DEBUG:boto:Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T10:50:19.334 DEBUG:boto:Port: 8000 2024-02-01T10:50:19.334 DEBUG:boto:Params: {} 2024-02-01T10:50:19.334 DEBUG:boto:Token: None 2024-02-01T10:50:19.334 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:50:19 GMT /vclfcn-66/ 2024-02-01T10:50:19.334 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:X56HkM98+NybjkWb+LZyVVfye60= 2024-02-01T10:50:19.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:X56HkM98+NybjkWb+LZyVVfye60=', 'Content-Length': '0'} 2024-02-01T10:50:19.405 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008d55c01c3c1a389b-0065bb776b-5350-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:50:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:19.406 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=vclfcn-67 2024-02-01T10:50:19.406 DEBUG:boto:path=/vclfcn-67/ 2024-02-01T10:50:19.406 DEBUG:boto:auth_path=/vclfcn-67/ 2024-02-01T10:50:19.406 DEBUG:boto:Method: PUT 2024-02-01T10:50:19.406 DEBUG:boto:Path: /vclfcn-67/ 2024-02-01T10:50:19.406 DEBUG:boto:Data: 2024-02-01T10:50:19.406 DEBUG:boto:Headers: {} 2024-02-01T10:50:19.406 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T10:50:19.406 DEBUG:boto:Port: 8001 2024-02-01T10:50:19.406 DEBUG:boto:Params: {} 2024-02-01T10:50:19.407 DEBUG:boto:Token: None 2024-02-01T10:50:19.407 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:50:19 GMT /vclfcn-67/ 2024-02-01T10:50:19.407 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+N7G8MxPqf0YS/ihTKKkPQ63XVE= 2024-02-01T10:50:19.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+N7G8MxPqf0YS/ihTKKkPQ63XVE=', 'Content-Length': '0'} 2024-02-01T10:50:19.474 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f80d697590db6d09-0065bb776b-5762-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:50:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:19.475 DEBUG:boto:path=/vclfcn-65/MULTIPART 2024-02-01T10:50:19.475 DEBUG:boto:auth_path=/vclfcn-65/MULTIPART 2024-02-01T10:50:19.475 DEBUG:boto:path=/vclfcn-65/MULTIPART?uploads 2024-02-01T10:50:19.475 DEBUG:boto:auth_path=/vclfcn-65/MULTIPART?uploads 2024-02-01T10:50:19.475 DEBUG:boto:Method: POST 2024-02-01T10:50:19.475 DEBUG:boto:Path: /vclfcn-65/MULTIPART?uploads 2024-02-01T10:50:19.475 DEBUG:boto:Data: 2024-02-01T10:50:19.475 DEBUG:boto:Headers: {} 2024-02-01T10:50:19.475 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:19.476 DEBUG:boto:Port: 8000 2024-02-01T10:50:19.476 DEBUG:boto:Params: {} 2024-02-01T10:50:19.476 DEBUG:boto:Token: None 2024-02-01T10:50:19.476 DEBUG:boto:StringToSign: POST Thu, 01 Feb 2024 10:50:19 GMT /vclfcn-65/MULTIPART?uploads 2024-02-01T10:50:19.476 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:0xJ3MWPuSJjbfKXLyoICmIafuKw= 2024-02-01T10:50:19.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0xJ3MWPuSJjbfKXLyoICmIafuKw=', 'Content-Length': '0'} 2024-02-01T10:50:21.293 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000362315371a0001b9-0065bb776b-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '249'), ('Date', 'Thu, 01 Feb 2024 10:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:21.294 DEBUG:boto:b'vclfcn-65MULTIPART2~D-K6AESwISRPEo9y0bUvx9MhCOlpttu' 2024-02-01T10:50:21.328 DEBUG:boto:path=/vclfcn-65/MULTIPART 2024-02-01T10:50:21.329 DEBUG:boto:auth_path=/vclfcn-65/MULTIPART 2024-02-01T10:50:21.329 DEBUG:boto:path=/vclfcn-65/MULTIPART?uploadId=2~D-K6AESwISRPEo9y0bUvx9MhCOlpttu&partNumber=1 2024-02-01T10:50:21.329 DEBUG:boto:auth_path=/vclfcn-65/MULTIPART?uploadId=2~D-K6AESwISRPEo9y0bUvx9MhCOlpttu&partNumber=1 2024-02-01T10:50:21.329 DEBUG:boto:Method: PUT 2024-02-01T10:50:21.330 DEBUG:boto:Path: /vclfcn-65/MULTIPART?uploadId=2~D-K6AESwISRPEo9y0bUvx9MhCOlpttu&partNumber=1 2024-02-01T10:50:21.330 DEBUG:boto:Data: 2024-02-01T10:50:21.330 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ebKBBg0ze5srhMzzkK3PdA==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2024-02-01T10:50:21.330 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:21.330 DEBUG:boto:Port: 8000 2024-02-01T10:50:21.330 DEBUG:boto:Params: {} 2024-02-01T10:50:21.331 DEBUG:boto:Token: None 2024-02-01T10:50:21.331 DEBUG:boto:StringToSign: PUT ebKBBg0ze5srhMzzkK3PdA== application/octet-stream Thu, 01 Feb 2024 10:50:21 GMT /vclfcn-65/MULTIPART?partNumber=1&uploadId=2~D-K6AESwISRPEo9y0bUvx9MhCOlpttu 2024-02-01T10:50:21.331 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:QAR+AvANfgYW4VMmq6jbCXae+yo= 2024-02-01T10:50:21.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ebKBBg0ze5srhMzzkK3PdA==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:50:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QAR+AvANfgYW4VMmq6jbCXae+yo='} 2024-02-01T10:50:21.388 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"79b281060d337b9b2b84ccf390adcf74"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000bfcffdb6e0f95e5-0065bb776d-5765-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:21.426 DEBUG:boto:path=/vclfcn-65/MULTIPART 2024-02-01T10:50:21.426 DEBUG:boto:auth_path=/vclfcn-65/MULTIPART 2024-02-01T10:50:21.426 DEBUG:boto:path=/vclfcn-65/MULTIPART?uploadId=2~D-K6AESwISRPEo9y0bUvx9MhCOlpttu&partNumber=2 2024-02-01T10:50:21.426 DEBUG:boto:auth_path=/vclfcn-65/MULTIPART?uploadId=2~D-K6AESwISRPEo9y0bUvx9MhCOlpttu&partNumber=2 2024-02-01T10:50:21.426 DEBUG:boto:Method: PUT 2024-02-01T10:50:21.426 DEBUG:boto:Path: /vclfcn-65/MULTIPART?uploadId=2~D-K6AESwISRPEo9y0bUvx9MhCOlpttu&partNumber=2 2024-02-01T10:50:21.426 DEBUG:boto:Data: 2024-02-01T10:50:21.426 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'dIQ6OrGTo4m87YmUAtmdXw==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2024-02-01T10:50:21.427 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:21.427 DEBUG:boto:Port: 8000 2024-02-01T10:50:21.427 DEBUG:boto:Params: {} 2024-02-01T10:50:21.427 DEBUG:boto:Token: None 2024-02-01T10:50:21.427 DEBUG:boto:StringToSign: PUT dIQ6OrGTo4m87YmUAtmdXw== application/octet-stream Thu, 01 Feb 2024 10:50:21 GMT /vclfcn-65/MULTIPART?partNumber=2&uploadId=2~D-K6AESwISRPEo9y0bUvx9MhCOlpttu 2024-02-01T10:50:21.427 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:emyyiXLPhIEUSyQ5MN0ijnmcwF4= 2024-02-01T10:50:21.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'dIQ6OrGTo4m87YmUAtmdXw==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:50:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:emyyiXLPhIEUSyQ5MN0ijnmcwF4='} 2024-02-01T10:50:21.485 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"74843a3ab193a389bced899402d99d5f"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c5fcdf3ba1575545-0065bb776d-5765-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:21.507 DEBUG:boto:path=/vclfcn-65/MULTIPART 2024-02-01T10:50:21.507 DEBUG:boto:auth_path=/vclfcn-65/MULTIPART 2024-02-01T10:50:21.507 DEBUG:boto:path=/vclfcn-65/MULTIPART?uploadId=2~D-K6AESwISRPEo9y0bUvx9MhCOlpttu&partNumber=3 2024-02-01T10:50:21.507 DEBUG:boto:auth_path=/vclfcn-65/MULTIPART?uploadId=2~D-K6AESwISRPEo9y0bUvx9MhCOlpttu&partNumber=3 2024-02-01T10:50:21.508 DEBUG:boto:Method: PUT 2024-02-01T10:50:21.508 DEBUG:boto:Path: /vclfcn-65/MULTIPART?uploadId=2~D-K6AESwISRPEo9y0bUvx9MhCOlpttu&partNumber=3 2024-02-01T10:50:21.508 DEBUG:boto:Data: 2024-02-01T10:50:21.508 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'e4RW4edMN49Fhh9TYZ51tg==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2024-02-01T10:50:21.508 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:21.508 DEBUG:boto:Port: 8000 2024-02-01T10:50:21.508 DEBUG:boto:Params: {} 2024-02-01T10:50:21.508 DEBUG:boto:Token: None 2024-02-01T10:50:21.509 DEBUG:boto:StringToSign: PUT e4RW4edMN49Fhh9TYZ51tg== application/octet-stream Thu, 01 Feb 2024 10:50:21 GMT /vclfcn-65/MULTIPART?partNumber=3&uploadId=2~D-K6AESwISRPEo9y0bUvx9MhCOlpttu 2024-02-01T10:50:21.509 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:j4U5CwnEAQMnizs2/QENj0ICTyM= 2024-02-01T10:50:21.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'e4RW4edMN49Fhh9TYZ51tg==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:50:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:j4U5CwnEAQMnizs2/QENj0ICTyM='} 2024-02-01T10:50:21.582 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"7b8456e1e74c378f45861f53619e75b6"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bdf8e42246b26986-0065bb776d-5765-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:21.604 DEBUG:boto:path=/vclfcn-65/MULTIPART 2024-02-01T10:50:21.604 DEBUG:boto:auth_path=/vclfcn-65/MULTIPART 2024-02-01T10:50:21.604 DEBUG:boto:path=/vclfcn-65/MULTIPART?uploadId=2~D-K6AESwISRPEo9y0bUvx9MhCOlpttu&partNumber=4 2024-02-01T10:50:21.604 DEBUG:boto:auth_path=/vclfcn-65/MULTIPART?uploadId=2~D-K6AESwISRPEo9y0bUvx9MhCOlpttu&partNumber=4 2024-02-01T10:50:21.604 DEBUG:boto:Method: PUT 2024-02-01T10:50:21.605 DEBUG:boto:Path: /vclfcn-65/MULTIPART?uploadId=2~D-K6AESwISRPEo9y0bUvx9MhCOlpttu&partNumber=4 2024-02-01T10:50:21.605 DEBUG:boto:Data: 2024-02-01T10:50:21.605 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'l+jDi4yPcIlVvEiQwDQZTQ==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2024-02-01T10:50:21.605 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:21.605 DEBUG:boto:Port: 8000 2024-02-01T10:50:21.605 DEBUG:boto:Params: {} 2024-02-01T10:50:21.605 DEBUG:boto:Token: None 2024-02-01T10:50:21.605 DEBUG:boto:StringToSign: PUT l+jDi4yPcIlVvEiQwDQZTQ== application/octet-stream Thu, 01 Feb 2024 10:50:21 GMT /vclfcn-65/MULTIPART?partNumber=4&uploadId=2~D-K6AESwISRPEo9y0bUvx9MhCOlpttu 2024-02-01T10:50:21.606 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Q8sHtvhpMHqxNMlvTAOQftFOR4Y= 2024-02-01T10:50:21.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'l+jDi4yPcIlVvEiQwDQZTQ==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:50:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Q8sHtvhpMHqxNMlvTAOQftFOR4Y='} 2024-02-01T10:50:21.645 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"97e8c38b8c8f708955bc4890c034194d"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000210ba675038ca203-0065bb776d-5765-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:21.645 DEBUG:boto:path=/vclfcn-65/MULTIPART 2024-02-01T10:50:21.645 DEBUG:boto:auth_path=/vclfcn-65/MULTIPART 2024-02-01T10:50:21.645 DEBUG:boto:path=/vclfcn-65/MULTIPART?uploadId=2~D-K6AESwISRPEo9y0bUvx9MhCOlpttu 2024-02-01T10:50:21.645 DEBUG:boto:auth_path=/vclfcn-65/MULTIPART?uploadId=2~D-K6AESwISRPEo9y0bUvx9MhCOlpttu 2024-02-01T10:50:21.645 DEBUG:boto:Method: GET 2024-02-01T10:50:21.646 DEBUG:boto:Path: /vclfcn-65/MULTIPART?uploadId=2~D-K6AESwISRPEo9y0bUvx9MhCOlpttu 2024-02-01T10:50:21.646 DEBUG:boto:Data: 2024-02-01T10:50:21.646 DEBUG:boto:Headers: {} 2024-02-01T10:50:21.646 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:21.646 DEBUG:boto:Port: 8000 2024-02-01T10:50:21.646 DEBUG:boto:Params: {} 2024-02-01T10:50:21.646 DEBUG:boto:Token: None 2024-02-01T10:50:21.646 DEBUG:boto:StringToSign: GET Thu, 01 Feb 2024 10:50:21 GMT /vclfcn-65/MULTIPART?uploadId=2~D-K6AESwISRPEo9y0bUvx9MhCOlpttu 2024-02-01T10:50:21.647 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:2Zo6K7xOJiskfkh3p8IHIXOexzs= 2024-02-01T10:50:21.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2Zo6K7xOJiskfkh3p8IHIXOexzs=', 'Content-Length': '0'} 2024-02-01T10:50:21.651 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009233e65651e13a03-0065bb776d-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:21.697 DEBUG:boto:path=/vclfcn-65/MULTIPART 2024-02-01T10:50:21.697 DEBUG:boto:auth_path=/vclfcn-65/MULTIPART 2024-02-01T10:50:21.697 DEBUG:boto:path=/vclfcn-65/MULTIPART?uploadId=2~D-K6AESwISRPEo9y0bUvx9MhCOlpttu 2024-02-01T10:50:21.697 DEBUG:boto:auth_path=/vclfcn-65/MULTIPART?uploadId=2~D-K6AESwISRPEo9y0bUvx9MhCOlpttu 2024-02-01T10:50:21.698 DEBUG:boto:Method: POST 2024-02-01T10:50:21.698 DEBUG:boto:Path: /vclfcn-65/MULTIPART?uploadId=2~D-K6AESwISRPEo9y0bUvx9MhCOlpttu 2024-02-01T10:50:21.698 DEBUG:boto:Data: 1 "79b281060d337b9b2b84ccf390adcf74" 2 "74843a3ab193a389bced899402d99d5f" 3 "7b8456e1e74c378f45861f53619e75b6" 4 "97e8c38b8c8f708955bc4890c034194d" 2024-02-01T10:50:21.698 DEBUG:boto:Headers: {'Content-Type': 'text/xml'} 2024-02-01T10:50:21.698 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:50:21.698 DEBUG:boto:Port: 8000 2024-02-01T10:50:21.699 DEBUG:boto:Params: {} 2024-02-01T10:50:21.699 DEBUG:boto:Token: None 2024-02-01T10:50:21.699 DEBUG:boto:StringToSign: POST text/xml Thu, 01 Feb 2024 10:50:21 GMT /vclfcn-65/MULTIPART?uploadId=2~D-K6AESwISRPEo9y0bUvx9MhCOlpttu 2024-02-01T10:50:21.699 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:2cXixmeEPv/Ryw+bwvoArGL714E= 2024-02-01T10:50:21.699 DEBUG:boto:Final headers: {'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2cXixmeEPv/Ryw+bwvoArGL714E=', 'Content-Length': '460'} 2024-02-01T10:50:21.724 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000318cbbf3d6dce6a7-0065bb776d-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '325'), ('Date', 'Thu, 01 Feb 2024 10:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:50:21.724 DEBUG:boto:http://smithi148.front.sepia.ceph.com:8000/vclfcn-65/MULTIPARTvclfcn-65MULTIPARTd07d411772772538cd4b657da660bfb5-4 2024-02-01T10:50:21.725 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:50:21.725 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:50:21.725 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:50:21.769 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:50:21.769 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:50:21.829 INFO:teuthology.orchestra.run.smithi148.stderr:No --period given, using current period=f2e6c290-e7aa-4279-b77e-15403b18d61f 2024-02-01T10:50:21.856 INFO:teuthology.orchestra.run.smithi148.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1706784619.449972_360.1","last_update":"2024-02-01T10:50:19.449972Z"},{"marker":"","last_update":"0.000000"}] 2024-02-01T10:50:21.857 DEBUG:rgw_multi.tests:master meta markers={0: '', 1: '', 2: '1_1706784619.449972_360.1', 3: ''} 2024-02-01T10:50:21.857 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-02-01T10:50:21.857 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:50:21.857 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:50:21.857 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:50:21.900 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:50:21.901 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:50:21.956 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:50:21.955+0000 7fdd7a950980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:50:21.956 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:50:21.955+0000 7fdd7a950980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:50:21.981 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706784619.320199_349.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:50:19.320199Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:50:21Z"} 2024-02-01T10:50:21.981 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706784619.320199_349.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:50:19.320199Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:50:21Z"} 2024-02-01T10:50:21.981 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706784619.320199_349.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:50:19.320199Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:50:21.981 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1706784619.449972_360.1', 3: ''} 2024-02-01T10:50:21.981 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1706784619.320199_349.1', 3: ''} 2024-02-01T10:50:21.981 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1706784619.449972_360.1 target=1_1706784619.320199_349.1 2024-02-01T10:50:26.702 INFO:tasks.rgw.c1.client.1.smithi148.stderr:daemon-helper: command crashed with signal 11 2024-02-01T10:50:26.982 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:50:26.982 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:50:26.983 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:50:27.025 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:50:27.025 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:50:27.094 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:50:27.091+0000 7ff779ce4980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:50:27.094 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:50:27.095+0000 7ff779ce4980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:50:27.118 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706784619.320199_349.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:50:19.320199Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:50:27Z"} 2024-02-01T10:50:27.119 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706784619.320199_349.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:50:19.320199Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:50:27Z"} 2024-02-01T10:50:27.119 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706784619.320199_349.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:50:19.320199Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:50:27.119 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1706784619.449972_360.1', 3: ''} 2024-02-01T10:50:27.119 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1706784619.320199_349.1', 3: ''} 2024-02-01T10:50:27.119 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1706784619.449972_360.1 target=1_1706784619.320199_349.1 2024-02-01T10:50:27.761 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2024-02-01T10:50:32.120 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:50:32.121 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:50:32.121 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:50:32.164 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:50:32.164 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:50:32.220 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:50:32.219+0000 7feea4fe2980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:50:32.221 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:50:32.219+0000 7feea4fe2980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:50:32.246 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706784619.449972_360.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:50:19.449972Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:50:32Z"} 2024-02-01T10:50:32.246 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706784619.449972_360.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:50:19.449972Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:50:32Z"} 2024-02-01T10:50:32.247 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706784619.449972_360.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:50:19.449972Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:50:32.247 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1706784619.449972_360.1', 3: ''} 2024-02-01T10:50:32.247 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1706784619.449972_360.1', 3: ''} 2024-02-01T10:50:32.247 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-02-01T10:50:32.248 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-02-01T10:50:32.248 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:50:32.248 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:50:32.249 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:50:32.292 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:50:32.292 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:50:32.346 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:50:32.345+0000 7f311308f980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:50:32.347 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T10:50:32.345+0000 7f311308f980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:50:32.371 INFO:teuthology.orchestra.run.smithi148.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706784619.449972_360.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:50:19.449972Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:50:32Z"} 2024-02-01T10:50:32.372 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706784619.449972_360.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:50:19.449972Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:50:32Z"} 2024-02-01T10:50:32.372 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706784619.449972_360.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:50:19.449972Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:50:32.372 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1706784619.449972_360.1', 3: ''} 2024-02-01T10:50:32.372 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1706784619.449972_360.1', 3: ''} 2024-02-01T10:50:32.373 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-02-01T10:50:32.373 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=vclfcn-65 2024-02-01T10:50:32.373 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:50:32.373 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T10:50:32.374 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket vclfcn-65 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-02-01T10:50:32.424 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:50:32.424 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:50:32.488 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:50:32.487+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T10:50:33.367 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~6s 2024-02-01T10:50:37.489 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:50:37.487+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T10:50:37.857 INFO:tasks.rgw.c2.client.0.smithi187.stderr:daemon-helper: command crashed with signal 11 2024-02-01T10:50:38.973 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~11s 2024-02-01T10:50:41.817 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2024-02-01T10:50:42.492 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:50:42.491+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T10:50:44.578 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~17s 2024-02-01T10:50:47.423 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~6s 2024-02-01T10:50:47.493 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:50:47.491+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T10:50:50.184 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~22s 2024-02-01T10:50:52.494 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:50:52.491+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T10:50:53.027 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~11s 2024-02-01T10:50:55.790 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~28s 2024-02-01T10:50:57.496 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:50:57.495+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T10:50:58.633 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~17s 2024-02-01T10:51:01.396 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~34s 2024-02-01T10:51:02.497 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:51:02.495+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T10:51:04.239 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~22s 2024-02-01T10:51:07.002 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~39s 2024-02-01T10:51:07.499 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:51:07.499+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T10:51:09.845 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~28s 2024-02-01T10:51:12.500 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:51:12.499+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T10:51:12.608 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~45s 2024-02-01T10:51:15.452 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~34s 2024-02-01T10:51:17.501 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:51:17.499+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T10:51:18.214 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~50s 2024-02-01T10:51:21.058 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~39s 2024-02-01T10:51:22.502 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:51:22.499+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T10:51:23.819 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~56s 2024-02-01T10:51:26.664 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~45s 2024-02-01T10:51:27.503 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:51:27.502+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T10:51:29.428 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~62s 2024-02-01T10:51:32.270 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~50s 2024-02-01T10:51:32.505 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:51:32.502+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T10:51:35.034 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~67s 2024-02-01T10:51:37.506 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:51:37.502+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T10:51:37.875 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~56s 2024-02-01T10:51:40.641 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~73s 2024-02-01T10:51:42.507 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:51:42.506+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T10:51:43.482 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~62s 2024-02-01T10:51:46.246 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~78s 2024-02-01T10:51:47.508 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:51:47.506+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T10:51:49.088 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~67s 2024-02-01T10:51:51.852 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~84s 2024-02-01T10:51:52.509 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:51:52.506+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T10:51:54.694 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~73s 2024-02-01T10:51:57.458 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~90s 2024-02-01T10:51:57.510 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:51:57.506+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T10:52:00.300 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~78s 2024-02-01T10:52:02.511 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:52:02.510+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T10:52:03.064 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~95s 2024-02-01T10:52:05.906 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~84s 2024-02-01T10:52:07.512 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:52:07.510+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T10:52:08.669 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~101s 2024-02-01T10:52:11.512 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~90s 2024-02-01T10:52:12.513 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:52:12.510+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T10:52:14.274 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~107s 2024-02-01T10:52:17.118 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~95s 2024-02-01T10:52:17.514 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:52:17.509+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T10:52:19.880 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~112s 2024-02-01T10:52:22.515 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:52:22.513+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T10:52:22.724 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~101s 2024-02-01T10:52:25.486 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~118s 2024-02-01T10:52:27.516 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:52:27.513+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T10:52:28.333 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~107s 2024-02-01T10:52:31.092 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~123s 2024-02-01T10:52:32.517 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:52:32.513+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T10:52:33.939 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~112s 2024-02-01T10:52:36.697 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~129s 2024-02-01T10:52:37.519 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:52:37.517+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T10:52:39.545 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~118s 2024-02-01T10:52:42.303 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~135s 2024-02-01T10:52:42.520 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:52:42.517+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T10:52:45.153 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~123s 2024-02-01T10:52:47.521 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:52:47.521+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T10:52:47.908 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~140s 2024-02-01T10:52:50.758 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~129s 2024-02-01T10:52:52.523 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:52:52.517+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T10:52:53.514 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~146s 2024-02-01T10:52:56.364 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~135s 2024-02-01T10:52:57.524 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:52:57.521+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T10:52:59.119 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~151s 2024-02-01T10:53:01.971 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~140s 2024-02-01T10:53:02.525 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:53:02.521+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T10:53:04.724 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~157s 2024-02-01T10:53:07.526 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:53:07.521+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T10:53:07.576 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~146s 2024-02-01T10:53:10.353 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~163s 2024-02-01T10:53:12.527 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:53:12.524+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T10:53:13.182 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~151s 2024-02-01T10:53:15.959 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~168s 2024-02-01T10:53:17.528 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:53:17.524+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T10:53:18.787 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~157s 2024-02-01T10:53:21.564 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~174s 2024-02-01T10:53:22.530 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:53:22.528+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T10:53:24.394 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~163s 2024-02-01T10:53:27.175 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~179s 2024-02-01T10:53:27.530 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:53:27.528+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T10:53:30.000 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~168s 2024-02-01T10:53:32.532 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:53:32.532+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T10:53:32.782 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~185s 2024-02-01T10:53:35.606 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~174s 2024-02-01T10:53:37.533 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:53:37.532+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T10:53:38.388 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~191s 2024-02-01T10:53:41.213 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~179s 2024-02-01T10:53:42.534 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:53:42.532+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T10:53:43.994 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~196s 2024-02-01T10:53:46.818 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~185s 2024-02-01T10:53:47.535 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:53:47.532+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T10:53:49.600 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~202s 2024-02-01T10:53:52.424 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~191s 2024-02-01T10:53:52.536 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:53:52.536+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T10:53:55.207 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~207s 2024-02-01T10:53:57.537 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:53:57.536+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T10:53:58.029 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~196s 2024-02-01T10:54:00.813 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~213s 2024-02-01T10:54:02.538 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:54:02.536+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T10:54:03.641 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~202s 2024-02-01T10:54:06.419 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~219s 2024-02-01T10:54:07.539 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:54:07.539+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T10:54:09.247 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~207s 2024-02-01T10:54:12.024 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~224s 2024-02-01T10:54:12.540 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:54:12.539+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T10:54:14.852 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~213s 2024-02-01T10:54:17.541 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:54:17.539+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T10:54:17.630 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~230s 2024-02-01T10:54:20.458 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~219s 2024-02-01T10:54:22.543 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:54:22.539+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T10:54:23.236 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~235s 2024-02-01T10:54:26.064 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~224s 2024-02-01T10:54:27.544 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:54:27.543+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T10:54:28.843 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~241s 2024-02-01T10:54:31.670 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~230s 2024-02-01T10:54:32.545 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:54:32.543+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T10:54:34.448 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~247s 2024-02-01T10:54:37.275 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~235s 2024-02-01T10:54:37.546 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:54:37.543+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T10:54:40.053 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~252s 2024-02-01T10:54:42.548 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:54:42.547+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T10:54:42.881 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~241s 2024-02-01T10:54:45.659 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~258s 2024-02-01T10:54:47.549 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:54:47.547+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T10:54:48.487 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~247s 2024-02-01T10:54:51.265 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~264s 2024-02-01T10:54:52.550 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:54:52.547+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T10:54:54.092 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~252s 2024-02-01T10:54:56.871 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~269s 2024-02-01T10:54:57.551 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:54:57.550+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T10:54:59.698 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~258s 2024-02-01T10:55:02.476 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~275s 2024-02-01T10:55:02.552 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:55:02.550+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T10:55:05.303 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~263s 2024-02-01T10:55:07.554 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:55:07.550+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T10:55:08.081 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~280s 2024-02-01T10:55:10.908 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~269s 2024-02-01T10:55:12.555 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:55:12.550+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T10:55:13.687 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~286s 2024-02-01T10:55:16.514 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~275s 2024-02-01T10:55:17.556 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:55:17.554+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T10:55:19.292 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~292s 2024-02-01T10:55:22.122 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~280s 2024-02-01T10:55:22.557 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:55:22.554+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T10:55:24.897 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~297s 2024-02-01T10:55:27.558 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:55:27.554+0000 7f518ad76980 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-02-01T10:55:27.559 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:55:27.554+0000 7f518ad76980 0 bucket sync checkpoint failed: (110) Connection timed out 2024-02-01T10:55:27.559 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:55:27.554+0000 7f518ad76980 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-02-01T10:55:27.606 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-02-01T10:55:27.609 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multipart_object_sync ... FAIL 2024-02-01T10:55:27.611 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=vclfcn-68 2024-02-01T10:55:27.611 DEBUG:boto:path=/vclfcn-68/ 2024-02-01T10:55:27.611 DEBUG:boto:auth_path=/vclfcn-68/ 2024-02-01T10:55:27.612 DEBUG:boto:Method: PUT 2024-02-01T10:55:27.612 DEBUG:boto:Path: /vclfcn-68/ 2024-02-01T10:55:27.612 DEBUG:boto:Data: 2024-02-01T10:55:27.612 DEBUG:boto:Headers: {} 2024-02-01T10:55:27.612 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:55:27.612 DEBUG:boto:Port: 8000 2024-02-01T10:55:27.612 DEBUG:boto:Params: {} 2024-02-01T10:55:27.613 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T10:55:27.613 DEBUG:boto:Token: None 2024-02-01T10:55:27.613 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 10:55:27 GMT /vclfcn-68/ 2024-02-01T10:55:27.613 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:+1o6iZ15GVPxc8A+OLJyieOwbjc= 2024-02-01T10:55:27.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:55:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+1o6iZ15GVPxc8A+OLJyieOwbjc=', 'Content-Length': '0'} 2024-02-01T10:55:27.656 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000685607002e74b943-0065bb789f-5765-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:55:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:55:27.656 DEBUG:boto:path=/vclfcn-68/testobj-sse-c 2024-02-01T10:55:27.656 DEBUG:boto:auth_path=/vclfcn-68/testobj-sse-c 2024-02-01T10:55:27.656 DEBUG:boto:Method: PUT 2024-02-01T10:55:27.657 DEBUG:boto:Path: /vclfcn-68/testobj-sse-c 2024-02-01T10:55:27.657 DEBUG:boto:Data: 2024-02-01T10:55:27.657 DEBUG:boto:Headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2024-02-01T10:55:27.657 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:55:27.657 DEBUG:boto:Port: 8000 2024-02-01T10:55:27.657 DEBUG:boto:Params: {} 2024-02-01T10:55:27.657 DEBUG:boto:Token: None 2024-02-01T10:55:27.657 DEBUG:boto:StringToSign: PUT 3FCGuEcom6i4veFJuDiBdQ== application/octet-stream Thu, 01 Feb 2024 10:55:27 GMT x-amz-server-side-encryption-customer-algorithm:AES256 x-amz-server-side-encryption-customer-key:pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs= x-amz-server-side-encryption-customer-key-md5:DWygnHRtgiJ77HCm+1rvHw== /vclfcn-68/testobj-sse-c 2024-02-01T10:55:27.657 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:37C09I1eu+J3RNjmDqnwKIw/a7o= 2024-02-01T10:55:27.658 DEBUG:boto:Final headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:55:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:37C09I1eu+J3RNjmDqnwKIw/a7o='} 2024-02-01T10:55:27.672 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-server-side-encryption-customer-algorithm', 'AES256'), ('x-amz-server-side-encryption-customer-key-MD5', 'DWygnHRtgiJ77HCm+1rvHw=='), ('x-amz-request-id', 'tx000005bf238a48701f1f5-0065bb789f-5765-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:55:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:55:27.672 DEBUG:boto:path=/vclfcn-68/testobj-sse-kms 2024-02-01T10:55:27.673 DEBUG:boto:auth_path=/vclfcn-68/testobj-sse-kms 2024-02-01T10:55:27.673 DEBUG:boto:Method: PUT 2024-02-01T10:55:27.673 DEBUG:boto:Path: /vclfcn-68/testobj-sse-kms 2024-02-01T10:55:27.673 DEBUG:boto:Data: 2024-02-01T10:55:27.673 DEBUG:boto:Headers: {'x-amz-server-side-encryption': 'aws:kms', 'x-amz-server-side-encryption-aws-kms-key-id': 'testkey-1', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2024-02-01T10:55:27.673 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T10:55:27.673 DEBUG:boto:Port: 8000 2024-02-01T10:55:27.673 DEBUG:boto:Params: {} 2024-02-01T10:55:27.673 DEBUG:boto:Token: None 2024-02-01T10:55:27.674 DEBUG:boto:StringToSign: PUT 3FCGuEcom6i4veFJuDiBdQ== application/octet-stream Thu, 01 Feb 2024 10:55:27 GMT x-amz-server-side-encryption:aws:kms x-amz-server-side-encryption-aws-kms-key-id:testkey-1 /vclfcn-68/testobj-sse-kms 2024-02-01T10:55:27.674 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:p+V/Wi+CGgOgWryyx1cVnewDu4U= 2024-02-01T10:55:27.674 DEBUG:boto:Final headers: {'x-amz-server-side-encryption': 'aws:kms', 'x-amz-server-side-encryption-aws-kms-key-id': 'testkey-1', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:55:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:p+V/Wi+CGgOgWryyx1cVnewDu4U='} 2024-02-01T10:55:27.700 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-server-side-encryption', 'aws:kms'), ('x-amz-server-side-encryption-aws-kms-key-id', 'testkey-1'), ('x-amz-server-side-encryption-context', 'eyJhd3M6czM6YXJuIjoiYXJuOmF3czpzMzo6cmd3LW11bHRpc2l0ZS10ZXN0LXVzZXI6dmNsZmNuLTY4L3Rlc3RvYmotc3NlLWttcyJ9'), ('x-amz-request-id', 'tx000001050489dcc5e2a0e-0065bb789f-5765-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:55:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T10:55:27.701 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:55:27.701 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:55:27.701 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:55:27.727 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~286s 2024-02-01T10:55:27.741 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:55:27.741 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:55:27.802 INFO:teuthology.orchestra.run.smithi148.stderr:No --period given, using current period=f2e6c290-e7aa-4279-b77e-15403b18d61f 2024-02-01T10:55:27.827 INFO:teuthology.orchestra.run.smithi148.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1706784927.652136_370.1","last_update":"2024-02-01T10:55:27.652136Z"},{"marker":"","last_update":"0.000000"}] 2024-02-01T10:55:27.828 DEBUG:rgw_multi.tests:master meta markers={0: '', 1: '', 2: '1_1706784927.652136_370.1', 3: ''} 2024-02-01T10:55:27.828 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-02-01T10:55:27.828 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:55:27.828 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:55:27.829 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:55:27.872 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:55:27.872 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:55:27.932 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:55:27.930+0000 7f433efcc980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:55:27.933 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:55:27.930+0000 7f433efcc980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:55:27.957 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706784619.449972_360.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:50:19.449972Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:55:27Z"} 2024-02-01T10:55:27.958 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706784619.449972_360.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:50:19.449972Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:55:27Z"} 2024-02-01T10:55:27.958 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706784619.449972_360.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:50:19.449972Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:55:27.958 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1706784927.652136_370.1', 3: ''} 2024-02-01T10:55:27.958 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1706784619.449972_360.1', 3: ''} 2024-02-01T10:55:27.958 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1706784927.652136_370.1 target=1_1706784619.449972_360.1 2024-02-01T10:55:30.503 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~303s 2024-02-01T10:55:30.504 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2024-02-01T10:55:31.107 INFO:tasks.ceph.c1.osd.0:Sent signal 15 2024-02-01T10:55:31.108 INFO:tasks.ceph.c1.osd.1:Sent signal 15 2024-02-01T10:55:31.108 INFO:tasks.ceph.c1.osd.2:Sent signal 15 2024-02-01T10:55:31.109 INFO:tasks.ceph.c1.mon.a:Sent signal 15 2024-02-01T10:55:31.109 INFO:tasks.rgw.c1.client.0:Sent signal 15 2024-02-01T10:55:31.109 INFO:tasks.ceph.c1.mgr.x:Sent signal 15 2024-02-01T10:55:31.110 INFO:tasks.ceph.c1.osd.0.smithi148.stderr:2024-02-01T10:55:31.109+0000 7f15af050640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster c1 -i 0 (PID: 22575) UID: 0 2024-02-01T10:55:31.110 INFO:tasks.ceph.c1.osd.0.smithi148.stderr:2024-02-01T10:55:31.109+0000 7f15af050640 -1 osd.0 59 *** Got signal Terminated *** 2024-02-01T10:55:31.110 INFO:tasks.ceph.c1.osd.0.smithi148.stderr:2024-02-01T10:55:31.109+0000 7f15af050640 -1 osd.0 59 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-01T10:55:31.110 INFO:tasks.ceph.c1.osd.1.smithi148.stderr:2024-02-01T10:55:31.109+0000 7faf02b24640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster c1 -i 1 (PID: 22578) UID: 0 2024-02-01T10:55:31.110 INFO:tasks.ceph.c1.osd.1.smithi148.stderr:2024-02-01T10:55:31.109+0000 7faf02b24640 -1 osd.1 59 *** Got signal Terminated *** 2024-02-01T10:55:31.110 INFO:tasks.ceph.c1.osd.1.smithi148.stderr:2024-02-01T10:55:31.109+0000 7faf02b24640 -1 osd.1 59 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-01T10:55:31.111 INFO:tasks.ceph.c1.mon.a.smithi148.stderr:2024-02-01T10:55:31.109+0000 7f12cda22640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mon -f --cluster c1 -i a (PID: 22403) UID: 0 2024-02-01T10:55:31.111 INFO:tasks.ceph.c1.mon.a.smithi148.stderr:2024-02-01T10:55:31.109+0000 7f12cda22640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-02-01T10:55:31.111 INFO:tasks.ceph.c1.osd.2.smithi148.stderr:2024-02-01T10:55:31.109+0000 7f9dc250b640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster c1 -i 2 (PID: 22582) UID: 0 2024-02-01T10:55:31.111 INFO:tasks.ceph.c1.osd.2.smithi148.stderr:2024-02-01T10:55:31.109+0000 7f9dc250b640 -1 osd.2 59 *** Got signal Terminated *** 2024-02-01T10:55:31.111 INFO:tasks.ceph.c1.osd.2.smithi148.stderr:2024-02-01T10:55:31.109+0000 7f9dc250b640 -1 osd.2 59 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-01T10:55:31.112 INFO:tasks.ceph.c1.mgr.x.smithi148.stderr:2024-02-01T10:55:31.109+0000 7f7508cba640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mgr -f --cluster c1 -i x (PID: 22405) UID: 0 2024-02-01T10:55:31.112 INFO:tasks.ceph.c1.mgr.x.smithi148.stderr:2024-02-01T10:55:31.109+0000 7f7508cba640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-02-01T10:55:31.112 INFO:tasks.rgw.c1.client.0.smithi148.stdout:2024-02-01T10:55:31.109+0000 7f606358f640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 33468) UID: 0 2024-02-01T10:55:31.112 INFO:tasks.rgw.c1.client.0.smithi148.stdout:2024-02-01T10:55:31.109+0000 7f6066dd7a40 -1 shutting down 2024-02-01T10:55:32.959 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:55:32.960 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:55:32.960 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:55:33.003 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:55:33.003 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:55:33.061 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:55:33.058+0000 7f8f2578d980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:55:33.062 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:55:33.058+0000 7f8f2578d980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:55:33.087 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706784619.449972_360.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:50:19.449972Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:55:33Z"} 2024-02-01T10:55:33.087 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706784619.449972_360.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:50:19.449972Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:55:33Z"} 2024-02-01T10:55:33.088 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706784619.449972_360.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:50:19.449972Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:55:33.088 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1706784927.652136_370.1', 3: ''} 2024-02-01T10:55:33.088 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1706784619.449972_360.1', 3: ''} 2024-02-01T10:55:33.088 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1706784927.652136_370.1 target=1_1706784619.449972_360.1 2024-02-01T10:55:33.333 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~292s 2024-02-01T10:55:38.089 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:55:38.089 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:55:38.089 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:55:38.131 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:55:38.131 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:55:38.186 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:55:38.182+0000 7f592cad5980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:55:38.187 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:55:38.186+0000 7f592cad5980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:55:38.211 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706784619.449972_360.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:50:19.449972Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:55:38Z"} 2024-02-01T10:55:38.211 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706784619.449972_360.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:50:19.449972Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:55:38Z"} 2024-02-01T10:55:38.211 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706784619.449972_360.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:50:19.449972Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:55:38.211 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1706784927.652136_370.1', 3: ''} 2024-02-01T10:55:38.212 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1706784619.449972_360.1', 3: ''} 2024-02-01T10:55:38.212 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1706784927.652136_370.1 target=1_1706784619.449972_360.1 2024-02-01T10:55:38.939 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~297s 2024-02-01T10:55:43.212 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:55:43.212 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:55:43.213 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:55:43.256 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:55:43.256 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T10:55:43.313 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:55:43.310+0000 7f34c7484980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:55:43.314 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T10:55:43.310+0000 7f34c7484980 20 RGW-SYNC:meta: read sync status 2024-02-01T10:55:43.338 INFO:teuthology.orchestra.run.smithi187.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706784619.449972_360.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:50:19.449972Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:55:43Z"} 2024-02-01T10:55:43.338 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706784619.449972_360.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:50:19.449972Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:55:43Z"} 2024-02-01T10:55:43.338 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706784619.449972_360.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:50:19.449972Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T10:55:43.339 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1706784927.652136_370.1', 3: ''} 2024-02-01T10:55:43.339 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1706784619.449972_360.1', 3: ''} 2024-02-01T10:55:43.339 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1706784927.652136_370.1 target=1_1706784619.449972_360.1 2024-02-01T10:55:44.545 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~303s 2024-02-01T10:55:44.545 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2024-02-01T10:55:45.151 INFO:tasks.ceph.c2.osd.0:Sent signal 15 2024-02-01T10:55:45.152 INFO:tasks.ceph.c2.osd.1:Sent signal 15 2024-02-01T10:55:45.152 INFO:tasks.ceph.c2.osd.2:Sent signal 15 2024-02-01T10:55:45.152 INFO:tasks.ceph.c2.mon.a:Sent signal 15 2024-02-01T10:55:45.153 INFO:tasks.rgw.c2.client.1:Sent signal 15 2024-02-01T10:55:45.153 INFO:tasks.ceph.c2.mgr.x:Sent signal 15 2024-02-01T10:55:45.154 INFO:tasks.ceph.c2.osd.0.smithi187.stderr:2024-02-01T10:55:45.150+0000 7efcfacc9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster c2 -i 0 (PID: 22597) UID: 0 2024-02-01T10:55:45.154 INFO:tasks.ceph.c2.osd.0.smithi187.stderr:2024-02-01T10:55:45.150+0000 7efcfacc9640 -1 osd.0 52 *** Got signal Terminated *** 2024-02-01T10:55:45.154 INFO:tasks.ceph.c2.osd.0.smithi187.stderr:2024-02-01T10:55:45.150+0000 7efcfacc9640 -1 osd.0 52 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-01T10:55:45.154 INFO:tasks.ceph.c2.osd.1.smithi187.stderr:2024-02-01T10:55:45.150+0000 7fd237fa3640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster c2 -i 1 (PID: 22600) UID: 0 2024-02-01T10:55:45.154 INFO:tasks.ceph.c2.osd.1.smithi187.stderr:2024-02-01T10:55:45.150+0000 7fd237fa3640 -1 osd.1 52 *** Got signal Terminated *** 2024-02-01T10:55:45.155 INFO:tasks.ceph.c2.osd.1.smithi187.stderr:2024-02-01T10:55:45.150+0000 7fd237fa3640 -1 osd.1 52 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-01T10:55:45.155 INFO:tasks.ceph.c2.osd.2.smithi187.stderr:2024-02-01T10:55:45.150+0000 7f6bbac68640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster c2 -i 2 (PID: 22599) UID: 0 2024-02-01T10:55:45.155 INFO:tasks.ceph.c2.osd.2.smithi187.stderr:2024-02-01T10:55:45.150+0000 7f6bbac68640 -1 osd.2 52 *** Got signal Terminated *** 2024-02-01T10:55:45.155 INFO:tasks.ceph.c2.osd.2.smithi187.stderr:2024-02-01T10:55:45.150+0000 7f6bbac68640 -1 osd.2 52 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-01T10:55:45.156 INFO:tasks.ceph.c2.mon.a.smithi187.stderr:2024-02-01T10:55:45.150+0000 7f0efca01640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mon -f --cluster c2 -i a (PID: 22424) UID: 0 2024-02-01T10:55:45.156 INFO:tasks.ceph.c2.mon.a.smithi187.stderr:2024-02-01T10:55:45.150+0000 7f0efca01640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-02-01T10:55:45.156 INFO:tasks.rgw.c2.client.1.smithi187.stdout:2024-02-01T10:55:45.150+0000 7f340883c640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground (PID: 28963) UID: 0 2024-02-01T10:55:45.156 INFO:tasks.rgw.c2.client.1.smithi187.stdout:2024-02-01T10:55:45.150+0000 7f340c084a40 -1 shutting down 2024-02-01T10:55:45.157 INFO:tasks.ceph.c2.mgr.x.smithi187.stderr:2024-02-01T10:55:45.150+0000 7f8f82f7d640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mgr -f --cluster c2 -i x (PID: 22426) UID: 0 2024-02-01T10:55:45.157 INFO:tasks.ceph.c2.mgr.x.smithi187.stderr:2024-02-01T10:55:45.150+0000 7f8f82f7d640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-02-01T10:55:48.340 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:55:48.340 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T10:55:48.341 DEBUG:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T10:55:48.383 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setuser ceph since I am not root 2024-02-01T10:55:48.383 INFO:teuthology.orchestra.run.smithi187.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T11:00:48.385 INFO:teuthology.orchestra.run.smithi187.stderr:2024-02-01T11:00:48.381+0000 7f04ec427980 0 monclient(hunting): authenticate timed out after 300 2024-02-01T11:00:48.386 INFO:teuthology.orchestra.run.smithi187.stderr:failed to fetch mon config (--no-mon-config to skip) 2024-02-01T11:00:48.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-01T11:00:48.389 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_encrypted_object_sync ... FAIL 2024-02-01T11:00:48.391 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=vclfcn-69 2024-02-01T11:00:48.391 DEBUG:boto:path=/vclfcn-69/ 2024-02-01T11:00:48.391 DEBUG:boto:auth_path=/vclfcn-69/ 2024-02-01T11:00:48.391 DEBUG:boto:Method: PUT 2024-02-01T11:00:48.391 DEBUG:boto:Path: /vclfcn-69/ 2024-02-01T11:00:48.391 DEBUG:boto:Data: 2024-02-01T11:00:48.391 DEBUG:boto:Headers: {} 2024-02-01T11:00:48.392 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T11:00:48.392 DEBUG:boto:Port: 8000 2024-02-01T11:00:48.392 DEBUG:boto:Params: {} 2024-02-01T11:00:48.392 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:00:48.392 DEBUG:boto:Token: None 2024-02-01T11:00:48.393 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 11:00:48 GMT /vclfcn-69/ 2024-02-01T11:00:48.393 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:aY1JEqvh4MlozEoELLurVH8ZpEg= 2024-02-01T11:00:48.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:00:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:aY1JEqvh4MlozEoELLurVH8ZpEg=', 'Content-Length': '0'} 2024-02-01T11:00:48.396 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:00:48.396 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:00:48.401 DEBUG:boto:Token: None 2024-02-01T11:00:48.401 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 11:00:48 GMT /vclfcn-69/ 2024-02-01T11:00:48.401 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:aY1JEqvh4MlozEoELLurVH8ZpEg= 2024-02-01T11:00:48.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:00:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:aY1JEqvh4MlozEoELLurVH8ZpEg=', 'Content-Length': '0'} 2024-02-01T11:00:48.403 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:00:48.403 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:00:49.463 DEBUG:boto:Token: None 2024-02-01T11:00:49.463 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 11:00:48 GMT /vclfcn-69/ 2024-02-01T11:00:49.463 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:aY1JEqvh4MlozEoELLurVH8ZpEg= 2024-02-01T11:00:49.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:00:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:aY1JEqvh4MlozEoELLurVH8ZpEg=', 'Content-Length': '0'} 2024-02-01T11:00:49.465 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:00:49.465 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:00:53.270 DEBUG:boto:Token: None 2024-02-01T11:00:53.271 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 11:00:48 GMT /vclfcn-69/ 2024-02-01T11:00:53.271 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:aY1JEqvh4MlozEoELLurVH8ZpEg= 2024-02-01T11:00:53.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:00:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:aY1JEqvh4MlozEoELLurVH8ZpEg=', 'Content-Length': '0'} 2024-02-01T11:00:53.273 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:00:53.274 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:00:59.732 DEBUG:boto:Token: None 2024-02-01T11:00:59.733 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 11:00:48 GMT /vclfcn-69/ 2024-02-01T11:00:59.733 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:aY1JEqvh4MlozEoELLurVH8ZpEg= 2024-02-01T11:00:59.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:00:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:aY1JEqvh4MlozEoELLurVH8ZpEg=', 'Content-Length': '0'} 2024-02-01T11:00:59.735 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:00:59.735 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:01:08.631 DEBUG:boto:Token: None 2024-02-01T11:01:08.631 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 11:00:48 GMT /vclfcn-69/ 2024-02-01T11:01:08.631 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:aY1JEqvh4MlozEoELLurVH8ZpEg= 2024-02-01T11:01:08.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:00:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:aY1JEqvh4MlozEoELLurVH8ZpEg=', 'Content-Length': '0'} 2024-02-01T11:01:08.633 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:01:08.634 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:01:22.451 DEBUG:boto:Token: None 2024-02-01T11:01:22.452 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 11:00:48 GMT /vclfcn-69/ 2024-02-01T11:01:22.452 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:aY1JEqvh4MlozEoELLurVH8ZpEg= 2024-02-01T11:01:22.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:00:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:aY1JEqvh4MlozEoELLurVH8ZpEg=', 'Content-Length': '0'} 2024-02-01T11:01:22.454 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:01:22.454 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:01:58.798 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_index_log_trim ... ERROR 2024-02-01T11:01:58.801 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=vclfcn-70 2024-02-01T11:01:58.802 DEBUG:boto:path=/vclfcn-70/ 2024-02-01T11:01:58.802 DEBUG:boto:auth_path=/vclfcn-70/ 2024-02-01T11:01:58.802 DEBUG:boto:Method: PUT 2024-02-01T11:01:58.802 DEBUG:boto:Path: /vclfcn-70/ 2024-02-01T11:01:58.802 DEBUG:boto:Data: 2024-02-01T11:01:58.803 DEBUG:boto:Headers: {} 2024-02-01T11:01:58.803 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T11:01:58.803 DEBUG:boto:Port: 8000 2024-02-01T11:01:58.803 DEBUG:boto:Params: {} 2024-02-01T11:01:58.803 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:01:58.803 DEBUG:boto:Token: None 2024-02-01T11:01:58.803 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 11:01:58 GMT /vclfcn-70/ 2024-02-01T11:01:58.804 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9/yOre7A8udHqCFoiuRpT/ehCO8= 2024-02-01T11:01:58.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:01:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9/yOre7A8udHqCFoiuRpT/ehCO8=', 'Content-Length': '0'} 2024-02-01T11:01:58.807 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:01:58.807 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:01:59.613 DEBUG:boto:Token: None 2024-02-01T11:01:59.613 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 11:01:58 GMT /vclfcn-70/ 2024-02-01T11:01:59.613 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9/yOre7A8udHqCFoiuRpT/ehCO8= 2024-02-01T11:01:59.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:01:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9/yOre7A8udHqCFoiuRpT/ehCO8=', 'Content-Length': '0'} 2024-02-01T11:01:59.615 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:01:59.615 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:02:00.182 DEBUG:boto:Token: None 2024-02-01T11:02:00.182 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 11:01:58 GMT /vclfcn-70/ 2024-02-01T11:02:00.183 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9/yOre7A8udHqCFoiuRpT/ehCO8= 2024-02-01T11:02:00.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:01:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9/yOre7A8udHqCFoiuRpT/ehCO8=', 'Content-Length': '0'} 2024-02-01T11:02:00.184 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:02:00.184 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:02:00.736 DEBUG:boto:Token: None 2024-02-01T11:02:00.737 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 11:01:58 GMT /vclfcn-70/ 2024-02-01T11:02:00.737 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9/yOre7A8udHqCFoiuRpT/ehCO8= 2024-02-01T11:02:00.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:01:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9/yOre7A8udHqCFoiuRpT/ehCO8=', 'Content-Length': '0'} 2024-02-01T11:02:00.739 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:02:00.739 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:02:06.720 DEBUG:boto:Token: None 2024-02-01T11:02:06.720 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 11:01:58 GMT /vclfcn-70/ 2024-02-01T11:02:06.721 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9/yOre7A8udHqCFoiuRpT/ehCO8= 2024-02-01T11:02:06.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:01:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9/yOre7A8udHqCFoiuRpT/ehCO8=', 'Content-Length': '0'} 2024-02-01T11:02:06.723 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:02:06.723 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:02:13.427 DEBUG:boto:Token: None 2024-02-01T11:02:13.427 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 11:01:58 GMT /vclfcn-70/ 2024-02-01T11:02:13.428 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9/yOre7A8udHqCFoiuRpT/ehCO8= 2024-02-01T11:02:13.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:01:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9/yOre7A8udHqCFoiuRpT/ehCO8=', 'Content-Length': '0'} 2024-02-01T11:02:13.430 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:02:13.430 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:02:28.679 DEBUG:boto:Token: None 2024-02-01T11:02:28.680 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 11:01:58 GMT /vclfcn-70/ 2024-02-01T11:02:28.680 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:9/yOre7A8udHqCFoiuRpT/ehCO8= 2024-02-01T11:02:28.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:01:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9/yOre7A8udHqCFoiuRpT/ehCO8=', 'Content-Length': '0'} 2024-02-01T11:02:28.682 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:02:28.682 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:02:43.307 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_reshard_index_log_trim ... ERROR 2024-02-01T11:02:43.309 DEBUG:boto:path=/vclfcn-71/ 2024-02-01T11:02:43.309 DEBUG:boto:auth_path=/vclfcn-71/ 2024-02-01T11:02:43.309 DEBUG:boto:Method: PUT 2024-02-01T11:02:43.309 DEBUG:boto:Path: /vclfcn-71/ 2024-02-01T11:02:43.309 DEBUG:boto:Data: 2024-02-01T11:02:43.309 DEBUG:boto:Headers: {} 2024-02-01T11:02:43.309 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T11:02:43.309 DEBUG:boto:Port: 8000 2024-02-01T11:02:43.309 DEBUG:boto:Params: {} 2024-02-01T11:02:43.310 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:02:43.310 DEBUG:boto:Token: None 2024-02-01T11:02:43.310 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 11:02:43 GMT /vclfcn-71/ 2024-02-01T11:02:43.310 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:6oTFxRxGk1893HCPszeUut1oEso= 2024-02-01T11:02:43.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:02:43 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:6oTFxRxGk1893HCPszeUut1oEso=', 'Content-Length': '0'} 2024-02-01T11:02:43.312 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:02:43.312 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:02:43.759 DEBUG:boto:Token: None 2024-02-01T11:02:43.759 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 11:02:43 GMT /vclfcn-71/ 2024-02-01T11:02:43.760 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:6oTFxRxGk1893HCPszeUut1oEso= 2024-02-01T11:02:43.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:02:43 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:6oTFxRxGk1893HCPszeUut1oEso=', 'Content-Length': '0'} 2024-02-01T11:02:43.761 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:02:43.761 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:02:44.373 DEBUG:boto:Token: None 2024-02-01T11:02:44.373 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 11:02:43 GMT /vclfcn-71/ 2024-02-01T11:02:44.373 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:6oTFxRxGk1893HCPszeUut1oEso= 2024-02-01T11:02:44.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:02:43 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:6oTFxRxGk1893HCPszeUut1oEso=', 'Content-Length': '0'} 2024-02-01T11:02:44.375 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:02:44.375 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:02:46.411 DEBUG:boto:Token: None 2024-02-01T11:02:46.411 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 11:02:43 GMT /vclfcn-71/ 2024-02-01T11:02:46.411 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:6oTFxRxGk1893HCPszeUut1oEso= 2024-02-01T11:02:46.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:02:43 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:6oTFxRxGk1893HCPszeUut1oEso=', 'Content-Length': '0'} 2024-02-01T11:02:46.413 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:02:46.414 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:02:46.732 DEBUG:boto:Token: None 2024-02-01T11:02:46.732 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 11:02:43 GMT /vclfcn-71/ 2024-02-01T11:02:46.732 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:6oTFxRxGk1893HCPszeUut1oEso= 2024-02-01T11:02:46.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:02:43 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:6oTFxRxGk1893HCPszeUut1oEso=', 'Content-Length': '0'} 2024-02-01T11:02:46.734 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:02:46.734 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:02:47.180 DEBUG:boto:Token: None 2024-02-01T11:02:47.181 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 11:02:43 GMT /vclfcn-71/ 2024-02-01T11:02:47.181 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:6oTFxRxGk1893HCPszeUut1oEso= 2024-02-01T11:02:47.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:02:43 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:6oTFxRxGk1893HCPszeUut1oEso=', 'Content-Length': '0'} 2024-02-01T11:02:47.183 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:02:47.183 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:02:50.627 DEBUG:boto:Token: None 2024-02-01T11:02:50.627 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 11:02:43 GMT /vclfcn-71/ 2024-02-01T11:02:50.627 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:6oTFxRxGk1893HCPszeUut1oEso= 2024-02-01T11:02:50.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:02:43 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:6oTFxRxGk1893HCPszeUut1oEso=', 'Content-Length': '0'} 2024-02-01T11:02:50.629 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:02:50.629 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:03:32.296 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_reshard_incremental ... ERROR 2024-02-01T11:03:32.298 DEBUG:boto:path=/vclfcn-72/ 2024-02-01T11:03:32.298 DEBUG:boto:auth_path=/vclfcn-72/ 2024-02-01T11:03:32.298 DEBUG:boto:Method: PUT 2024-02-01T11:03:32.298 DEBUG:boto:Path: /vclfcn-72/ 2024-02-01T11:03:32.298 DEBUG:boto:Data: 2024-02-01T11:03:32.299 DEBUG:boto:Headers: {} 2024-02-01T11:03:32.299 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T11:03:32.299 DEBUG:boto:Port: 8000 2024-02-01T11:03:32.299 DEBUG:boto:Params: {} 2024-02-01T11:03:32.299 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:03:32.299 DEBUG:boto:Token: None 2024-02-01T11:03:32.299 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 11:03:32 GMT /vclfcn-72/ 2024-02-01T11:03:32.300 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:iu+A5XLfHaQ4nqroFeZRdiPj6Ck= 2024-02-01T11:03:32.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:03:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iu+A5XLfHaQ4nqroFeZRdiPj6Ck=', 'Content-Length': '0'} 2024-02-01T11:03:32.302 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:03:32.302 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:03:32.703 DEBUG:boto:Token: None 2024-02-01T11:03:32.703 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 11:03:32 GMT /vclfcn-72/ 2024-02-01T11:03:32.704 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:iu+A5XLfHaQ4nqroFeZRdiPj6Ck= 2024-02-01T11:03:32.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:03:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iu+A5XLfHaQ4nqroFeZRdiPj6Ck=', 'Content-Length': '0'} 2024-02-01T11:03:32.706 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:03:32.706 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:03:34.103 DEBUG:boto:Token: None 2024-02-01T11:03:34.103 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 11:03:32 GMT /vclfcn-72/ 2024-02-01T11:03:34.103 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:iu+A5XLfHaQ4nqroFeZRdiPj6Ck= 2024-02-01T11:03:34.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:03:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iu+A5XLfHaQ4nqroFeZRdiPj6Ck=', 'Content-Length': '0'} 2024-02-01T11:03:34.106 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:03:34.106 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:03:37.317 DEBUG:boto:Token: None 2024-02-01T11:03:37.318 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 11:03:32 GMT /vclfcn-72/ 2024-02-01T11:03:37.318 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:iu+A5XLfHaQ4nqroFeZRdiPj6Ck= 2024-02-01T11:03:37.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:03:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iu+A5XLfHaQ4nqroFeZRdiPj6Ck=', 'Content-Length': '0'} 2024-02-01T11:03:37.320 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:03:37.320 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:03:38.110 DEBUG:boto:Token: None 2024-02-01T11:03:38.110 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 11:03:32 GMT /vclfcn-72/ 2024-02-01T11:03:38.111 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:iu+A5XLfHaQ4nqroFeZRdiPj6Ck= 2024-02-01T11:03:38.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:03:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iu+A5XLfHaQ4nqroFeZRdiPj6Ck=', 'Content-Length': '0'} 2024-02-01T11:03:38.114 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:03:38.114 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:03:44.711 DEBUG:boto:Token: None 2024-02-01T11:03:44.712 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 11:03:32 GMT /vclfcn-72/ 2024-02-01T11:03:44.712 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:iu+A5XLfHaQ4nqroFeZRdiPj6Ck= 2024-02-01T11:03:44.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:03:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iu+A5XLfHaQ4nqroFeZRdiPj6Ck=', 'Content-Length': '0'} 2024-02-01T11:03:44.714 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:03:44.714 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:04:11.391 DEBUG:boto:Token: None 2024-02-01T11:04:11.392 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 11:03:32 GMT /vclfcn-72/ 2024-02-01T11:04:11.392 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:iu+A5XLfHaQ4nqroFeZRdiPj6Ck= 2024-02-01T11:04:11.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:03:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iu+A5XLfHaQ4nqroFeZRdiPj6Ck=', 'Content-Length': '0'} 2024-02-01T11:04:11.395 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:04:11.395 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:05:02.010 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_reshard_full ... ERROR 2024-02-01T11:05:02.012 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T11:05:02.012 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T11:05:02.012 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T11:05:02.053 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T11:05:02.053 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T11:10:02.055 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T11:10:02.057+0000 7f8657491980 0 monclient(hunting): authenticate timed out after 300 2024-02-01T11:10:02.055 INFO:teuthology.orchestra.run.smithi148.stderr:failed to fetch mon config (--no-mon-config to skip) 2024-02-01T11:10:02.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-01T11:10:02.059 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_creation_time ... FAIL 2024-02-01T11:10:02.060 DEBUG:boto:path=/vclfcn-73/ 2024-02-01T11:10:02.060 DEBUG:boto:auth_path=/vclfcn-73/ 2024-02-01T11:10:02.060 DEBUG:boto:Method: PUT 2024-02-01T11:10:02.060 DEBUG:boto:Path: /vclfcn-73/ 2024-02-01T11:10:02.061 DEBUG:boto:Data: 2024-02-01T11:10:02.061 DEBUG:boto:Headers: {} 2024-02-01T11:10:02.061 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T11:10:02.061 DEBUG:boto:Port: 8000 2024-02-01T11:10:02.061 DEBUG:boto:Params: {} 2024-02-01T11:10:02.061 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:10:02.061 DEBUG:boto:Token: None 2024-02-01T11:10:02.062 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 11:10:02 GMT /vclfcn-73/ 2024-02-01T11:10:02.062 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Nk0JW4/RC7ZEn+dQH6V9FAc0DDw= 2024-02-01T11:10:02.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:10:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Nk0JW4/RC7ZEn+dQH6V9FAc0DDw=', 'Content-Length': '0'} 2024-02-01T11:10:02.064 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:10:02.064 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:10:02.681 DEBUG:boto:Token: None 2024-02-01T11:10:02.681 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 11:10:02 GMT /vclfcn-73/ 2024-02-01T11:10:02.681 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Nk0JW4/RC7ZEn+dQH6V9FAc0DDw= 2024-02-01T11:10:02.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:10:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Nk0JW4/RC7ZEn+dQH6V9FAc0DDw=', 'Content-Length': '0'} 2024-02-01T11:10:02.683 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:10:02.683 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:10:04.195 DEBUG:boto:Token: None 2024-02-01T11:10:04.196 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 11:10:02 GMT /vclfcn-73/ 2024-02-01T11:10:04.196 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Nk0JW4/RC7ZEn+dQH6V9FAc0DDw= 2024-02-01T11:10:04.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:10:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Nk0JW4/RC7ZEn+dQH6V9FAc0DDw=', 'Content-Length': '0'} 2024-02-01T11:10:04.198 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:10:04.198 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:10:06.305 DEBUG:boto:Token: None 2024-02-01T11:10:06.305 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 11:10:02 GMT /vclfcn-73/ 2024-02-01T11:10:06.306 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Nk0JW4/RC7ZEn+dQH6V9FAc0DDw= 2024-02-01T11:10:06.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:10:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Nk0JW4/RC7ZEn+dQH6V9FAc0DDw=', 'Content-Length': '0'} 2024-02-01T11:10:06.307 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:10:06.308 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:10:06.865 DEBUG:boto:Token: None 2024-02-01T11:10:06.865 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 11:10:02 GMT /vclfcn-73/ 2024-02-01T11:10:06.865 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Nk0JW4/RC7ZEn+dQH6V9FAc0DDw= 2024-02-01T11:10:06.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:10:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Nk0JW4/RC7ZEn+dQH6V9FAc0DDw=', 'Content-Length': '0'} 2024-02-01T11:10:06.867 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:10:06.867 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:10:14.179 DEBUG:boto:Token: None 2024-02-01T11:10:14.180 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 11:10:02 GMT /vclfcn-73/ 2024-02-01T11:10:14.180 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Nk0JW4/RC7ZEn+dQH6V9FAc0DDw= 2024-02-01T11:10:14.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:10:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Nk0JW4/RC7ZEn+dQH6V9FAc0DDw=', 'Content-Length': '0'} 2024-02-01T11:10:14.182 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:10:14.182 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:10:26.146 DEBUG:boto:Token: None 2024-02-01T11:10:26.147 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 11:10:02 GMT /vclfcn-73/ 2024-02-01T11:10:26.147 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Nk0JW4/RC7ZEn+dQH6V9FAc0DDw= 2024-02-01T11:10:26.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:10:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Nk0JW4/RC7ZEn+dQH6V9FAc0DDw=', 'Content-Length': '0'} 2024-02-01T11:10:26.149 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:10:26.149 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:11:10.945 INFO:tasks.rgw_multisite_tests:Create several generations of objects, then run bucket sync ... ERROR 2024-02-01T11:11:10.946 DEBUG:boto:path=/vclfcn-74/ 2024-02-01T11:11:10.946 DEBUG:boto:auth_path=/vclfcn-74/ 2024-02-01T11:11:10.947 DEBUG:boto:Method: PUT 2024-02-01T11:11:10.947 DEBUG:boto:Path: /vclfcn-74/ 2024-02-01T11:11:10.947 DEBUG:boto:Data: 2024-02-01T11:11:10.947 DEBUG:boto:Headers: {} 2024-02-01T11:11:10.947 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T11:11:10.947 DEBUG:boto:Port: 8000 2024-02-01T11:11:10.947 DEBUG:boto:Params: {} 2024-02-01T11:11:10.947 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:11:10.948 DEBUG:boto:Token: None 2024-02-01T11:11:10.948 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 11:11:10 GMT /vclfcn-74/ 2024-02-01T11:11:10.948 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Q5DbJjQBCIunGCiTMGvLPH32xgw= 2024-02-01T11:11:10.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:11:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Q5DbJjQBCIunGCiTMGvLPH32xgw=', 'Content-Length': '0'} 2024-02-01T11:11:10.950 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:11:10.951 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:11:11.283 DEBUG:boto:Token: None 2024-02-01T11:11:11.283 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 11:11:10 GMT /vclfcn-74/ 2024-02-01T11:11:11.283 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Q5DbJjQBCIunGCiTMGvLPH32xgw= 2024-02-01T11:11:11.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:11:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Q5DbJjQBCIunGCiTMGvLPH32xgw=', 'Content-Length': '0'} 2024-02-01T11:11:11.285 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:11:11.286 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:11:12.303 DEBUG:boto:Token: None 2024-02-01T11:11:12.303 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 11:11:10 GMT /vclfcn-74/ 2024-02-01T11:11:12.303 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Q5DbJjQBCIunGCiTMGvLPH32xgw= 2024-02-01T11:11:12.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:11:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Q5DbJjQBCIunGCiTMGvLPH32xgw=', 'Content-Length': '0'} 2024-02-01T11:11:12.305 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:11:12.305 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:11:15.283 DEBUG:boto:Token: None 2024-02-01T11:11:15.283 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 11:11:10 GMT /vclfcn-74/ 2024-02-01T11:11:15.283 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Q5DbJjQBCIunGCiTMGvLPH32xgw= 2024-02-01T11:11:15.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:11:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Q5DbJjQBCIunGCiTMGvLPH32xgw=', 'Content-Length': '0'} 2024-02-01T11:11:15.285 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:11:15.285 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:11:21.377 DEBUG:boto:Token: None 2024-02-01T11:11:21.377 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 11:11:10 GMT /vclfcn-74/ 2024-02-01T11:11:21.378 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Q5DbJjQBCIunGCiTMGvLPH32xgw= 2024-02-01T11:11:21.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:11:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Q5DbJjQBCIunGCiTMGvLPH32xgw=', 'Content-Length': '0'} 2024-02-01T11:11:21.380 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:11:21.380 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:11:34.037 DEBUG:boto:Token: None 2024-02-01T11:11:34.037 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 11:11:10 GMT /vclfcn-74/ 2024-02-01T11:11:34.037 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Q5DbJjQBCIunGCiTMGvLPH32xgw= 2024-02-01T11:11:34.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:11:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Q5DbJjQBCIunGCiTMGvLPH32xgw=', 'Content-Length': '0'} 2024-02-01T11:11:34.040 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:11:34.040 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:12:04.449 DEBUG:boto:Token: None 2024-02-01T11:12:04.449 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 11:11:10 GMT /vclfcn-74/ 2024-02-01T11:12:04.450 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:Q5DbJjQBCIunGCiTMGvLPH32xgw= 2024-02-01T11:12:04.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:11:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Q5DbJjQBCIunGCiTMGvLPH32xgw=', 'Content-Length': '0'} 2024-02-01T11:12:04.452 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:12:04.452 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:12:15.369 INFO:tasks.rgw_multisite_tests:Create several generations of objects, trash them, then run bucket sync init ... ERROR 2024-02-01T11:12:15.371 INFO:rgw_multi.tests:create role zone=test-zone1 name=roles-vclfcn-1 2024-02-01T11:12:15.372 DEBUG:boto:Method: POST 2024-02-01T11:12:15.372 DEBUG:boto:Path: / 2024-02-01T11:12:15.372 DEBUG:boto:Data: 2024-02-01T11:12:15.372 DEBUG:boto:Headers: {} 2024-02-01T11:12:15.373 DEBUG:boto:Host: smithi148.front.sepia.ceph.com 2024-02-01T11:12:15.373 DEBUG:boto:Port: 8000 2024-02-01T11:12:15.373 DEBUG:boto:Params: {'RoleName': 'roles-vclfcn-1', 'AssumeRolePolicyDocument': '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}', 'Path': '', 'Action': 'CreateRole', 'Version': '2010-05-08'} 2024-02-01T11:12:15.373 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:12:15.374 DEBUG:boto:Token: None 2024-02-01T11:12:15.374 DEBUG:boto:CanonicalRequest: POST / host:smithi148.front.sepia.ceph.com:8000 x-amz-date:20240201T111215Z host;x-amz-date 8344f44cf7b239e92f6078048bc20e4d5df2bc02ecccd3ac62b2071b0d761648 2024-02-01T11:12:15.374 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20240201T111215Z 20240201/front/smithi148/aws4_request 2a426675aadbf734174a6957771e048270b98bf28514fed19577f78df44cc9e7 2024-02-01T11:12:15.375 DEBUG:boto:Signature: 5b648e3e39d84e6f96fd62f755e3b5f6a91460668140eb8756f9572e06326002 2024-02-01T11:12:15.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240201T111215Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=1IMYR3NXFV2H92HDXL01/20240201/front/smithi148/aws4_request,SignedHeaders=host;x-amz-date,Signature=5b648e3e39d84e6f96fd62f755e3b5f6a91460668140eb8756f9572e06326002', 'Host': 'smithi148.front.sepia.ceph.com:8000'} 2024-02-01T11:12:15.377 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:12:15.377 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:12:15.941 DEBUG:boto:Token: None 2024-02-01T11:12:15.941 DEBUG:boto:CanonicalRequest: POST / host:smithi148.front.sepia.ceph.com:8000 x-amz-date:20240201T111215Z host;x-amz-date 8344f44cf7b239e92f6078048bc20e4d5df2bc02ecccd3ac62b2071b0d761648 2024-02-01T11:12:15.941 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20240201T111215Z 20240201/front/smithi148/aws4_request 2a426675aadbf734174a6957771e048270b98bf28514fed19577f78df44cc9e7 2024-02-01T11:12:15.942 DEBUG:boto:Signature: 5b648e3e39d84e6f96fd62f755e3b5f6a91460668140eb8756f9572e06326002 2024-02-01T11:12:15.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240201T111215Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=1IMYR3NXFV2H92HDXL01/20240201/front/smithi148/aws4_request,SignedHeaders=host;x-amz-date,Signature=5b648e3e39d84e6f96fd62f755e3b5f6a91460668140eb8756f9572e06326002', 'Host': 'smithi148.front.sepia.ceph.com:8000'} 2024-02-01T11:12:15.943 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:12:15.944 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:12:16.911 DEBUG:boto:Token: None 2024-02-01T11:12:16.911 DEBUG:boto:CanonicalRequest: POST / host:smithi148.front.sepia.ceph.com:8000 x-amz-date:20240201T111216Z host;x-amz-date 8344f44cf7b239e92f6078048bc20e4d5df2bc02ecccd3ac62b2071b0d761648 2024-02-01T11:12:16.912 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20240201T111216Z 20240201/front/smithi148/aws4_request b561e90ed06656214a7e5a3f7d1afbd1cad84b7a495bbec78091e4b6579b7f77 2024-02-01T11:12:16.912 DEBUG:boto:Signature: ff01a21c966c94915255726be1842abb62cb1f4f6f80d25057282893b88887aa 2024-02-01T11:12:16.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240201T111216Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=1IMYR3NXFV2H92HDXL01/20240201/front/smithi148/aws4_request,SignedHeaders=host;x-amz-date,Signature=ff01a21c966c94915255726be1842abb62cb1f4f6f80d25057282893b88887aa', 'Host': 'smithi148.front.sepia.ceph.com:8000'} 2024-02-01T11:12:16.914 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:12:16.914 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:12:18.954 DEBUG:boto:Token: None 2024-02-01T11:12:18.955 DEBUG:boto:CanonicalRequest: POST / host:smithi148.front.sepia.ceph.com:8000 x-amz-date:20240201T111218Z host;x-amz-date 8344f44cf7b239e92f6078048bc20e4d5df2bc02ecccd3ac62b2071b0d761648 2024-02-01T11:12:18.955 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20240201T111218Z 20240201/front/smithi148/aws4_request 3eda4d4b740830b6c636f11c9bc42b9360ef91d78a9b3565fb2129fa62ac9fca 2024-02-01T11:12:18.955 DEBUG:boto:Signature: fba1ed5876bfd2a1058a0906a867b4deeae277c2bd2d726d6cd231bd9d59e5a6 2024-02-01T11:12:18.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240201T111218Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=1IMYR3NXFV2H92HDXL01/20240201/front/smithi148/aws4_request,SignedHeaders=host;x-amz-date,Signature=fba1ed5876bfd2a1058a0906a867b4deeae277c2bd2d726d6cd231bd9d59e5a6', 'Host': 'smithi148.front.sepia.ceph.com:8000'} 2024-02-01T11:12:18.957 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:12:18.957 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:12:25.165 DEBUG:boto:Token: None 2024-02-01T11:12:25.166 DEBUG:boto:CanonicalRequest: POST / host:smithi148.front.sepia.ceph.com:8000 x-amz-date:20240201T111225Z host;x-amz-date 8344f44cf7b239e92f6078048bc20e4d5df2bc02ecccd3ac62b2071b0d761648 2024-02-01T11:12:25.166 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20240201T111225Z 20240201/front/smithi148/aws4_request 33ea75450219a29b273a93965fe0fbaa518153ddb3c933593bf18014585b869a 2024-02-01T11:12:25.166 DEBUG:boto:Signature: 86b552d44e71a7c81f94b9cb3e2a81af341c96ab1bec84ffbc316e0fc05f9ef1 2024-02-01T11:12:25.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240201T111225Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=1IMYR3NXFV2H92HDXL01/20240201/front/smithi148/aws4_request,SignedHeaders=host;x-amz-date,Signature=86b552d44e71a7c81f94b9cb3e2a81af341c96ab1bec84ffbc316e0fc05f9ef1', 'Host': 'smithi148.front.sepia.ceph.com:8000'} 2024-02-01T11:12:25.168 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:12:25.168 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:12:32.712 DEBUG:boto:Token: None 2024-02-01T11:12:32.713 DEBUG:boto:CanonicalRequest: POST / host:smithi148.front.sepia.ceph.com:8000 x-amz-date:20240201T111232Z host;x-amz-date 8344f44cf7b239e92f6078048bc20e4d5df2bc02ecccd3ac62b2071b0d761648 2024-02-01T11:12:32.713 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20240201T111232Z 20240201/front/smithi148/aws4_request 424feb93c3a4cce025a9c1d7936e9e8c9f94cd8f67a906be5ade47d6329fb4fd 2024-02-01T11:12:32.713 DEBUG:boto:Signature: dac3013101a42415a25ad679c892053a9192580cbf078de1e055bf7af092241e 2024-02-01T11:12:32.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240201T111232Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=1IMYR3NXFV2H92HDXL01/20240201/front/smithi148/aws4_request,SignedHeaders=host;x-amz-date,Signature=dac3013101a42415a25ad679c892053a9192580cbf078de1e055bf7af092241e', 'Host': 'smithi148.front.sepia.ceph.com:8000'} 2024-02-01T11:12:32.715 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:12:32.715 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:12:37.454 DEBUG:boto:Token: None 2024-02-01T11:12:37.454 DEBUG:boto:CanonicalRequest: POST / host:smithi148.front.sepia.ceph.com:8000 x-amz-date:20240201T111237Z host;x-amz-date 8344f44cf7b239e92f6078048bc20e4d5df2bc02ecccd3ac62b2071b0d761648 2024-02-01T11:12:37.454 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20240201T111237Z 20240201/front/smithi148/aws4_request 0030fee67c701869c4f96ff412d3727783b54141f9354972575c095cbb8b49f6 2024-02-01T11:12:37.454 DEBUG:boto:Signature: fc0facb4cf5e2a571a4723f31838fe2e7475dbf6d10179812059ccea944f1092 2024-02-01T11:12:37.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240201T111237Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=1IMYR3NXFV2H92HDXL01/20240201/front/smithi148/aws4_request,SignedHeaders=host;x-amz-date,Signature=fc0facb4cf5e2a571a4723f31838fe2e7475dbf6d10179812059ccea944f1092', 'Host': 'smithi148.front.sepia.ceph.com:8000'} 2024-02-01T11:12:37.456 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:12:37.456 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:13:25.680 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_role_sync ... ERROR 2024-02-01T11:13:25.681 INFO:rgw_multi.tests:create role zone=test-zone1 name=roles-vclfcn-2 2024-02-01T11:13:25.681 DEBUG:boto:Method: POST 2024-02-01T11:13:25.682 DEBUG:boto:Path: / 2024-02-01T11:13:25.682 DEBUG:boto:Data: 2024-02-01T11:13:25.682 DEBUG:boto:Headers: {} 2024-02-01T11:13:25.682 DEBUG:boto:Host: smithi148.front.sepia.ceph.com 2024-02-01T11:13:25.682 DEBUG:boto:Port: 8000 2024-02-01T11:13:25.682 DEBUG:boto:Params: {'RoleName': 'roles-vclfcn-2', 'AssumeRolePolicyDocument': '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}', 'Path': '', 'Action': 'CreateRole', 'Version': '2010-05-08'} 2024-02-01T11:13:25.682 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:13:25.683 DEBUG:boto:Token: None 2024-02-01T11:13:25.683 DEBUG:boto:CanonicalRequest: POST / host:smithi148.front.sepia.ceph.com:8000 x-amz-date:20240201T111325Z host;x-amz-date 4efa4a360bd87db21c71e7ea0c35b095f1f1059d6445a9281c4f4753dce8b410 2024-02-01T11:13:25.683 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20240201T111325Z 20240201/front/smithi148/aws4_request 3993b16311684068362030f255ed473bd51a2dd96e21bf18bb566975e30774a0 2024-02-01T11:13:25.683 DEBUG:boto:Signature: d3bfd428f34f075af3fd2e3a84486390da3b05bdbec6bef488ccf410e5fed81c 2024-02-01T11:13:25.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240201T111325Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=1IMYR3NXFV2H92HDXL01/20240201/front/smithi148/aws4_request,SignedHeaders=host;x-amz-date,Signature=d3bfd428f34f075af3fd2e3a84486390da3b05bdbec6bef488ccf410e5fed81c', 'Host': 'smithi148.front.sepia.ceph.com:8000'} 2024-02-01T11:13:25.686 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:13:25.686 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:13:25.801 DEBUG:boto:Token: None 2024-02-01T11:13:25.801 DEBUG:boto:CanonicalRequest: POST / host:smithi148.front.sepia.ceph.com:8000 x-amz-date:20240201T111325Z host;x-amz-date 4efa4a360bd87db21c71e7ea0c35b095f1f1059d6445a9281c4f4753dce8b410 2024-02-01T11:13:25.801 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20240201T111325Z 20240201/front/smithi148/aws4_request 3993b16311684068362030f255ed473bd51a2dd96e21bf18bb566975e30774a0 2024-02-01T11:13:25.801 DEBUG:boto:Signature: d3bfd428f34f075af3fd2e3a84486390da3b05bdbec6bef488ccf410e5fed81c 2024-02-01T11:13:25.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240201T111325Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=1IMYR3NXFV2H92HDXL01/20240201/front/smithi148/aws4_request,SignedHeaders=host;x-amz-date,Signature=d3bfd428f34f075af3fd2e3a84486390da3b05bdbec6bef488ccf410e5fed81c', 'Host': 'smithi148.front.sepia.ceph.com:8000'} 2024-02-01T11:13:25.803 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:13:25.804 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:13:26.463 DEBUG:boto:Token: None 2024-02-01T11:13:26.464 DEBUG:boto:CanonicalRequest: POST / host:smithi148.front.sepia.ceph.com:8000 x-amz-date:20240201T111326Z host;x-amz-date 4efa4a360bd87db21c71e7ea0c35b095f1f1059d6445a9281c4f4753dce8b410 2024-02-01T11:13:26.464 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20240201T111326Z 20240201/front/smithi148/aws4_request af68e20ca666426330ce53cf6c6f72ec4e4cab6ae705ecf18babd51b03aa70d2 2024-02-01T11:13:26.464 DEBUG:boto:Signature: 593b777b1fd42f3f9c866f13215cbccab75e6138b22309204417fb2e4492710c 2024-02-01T11:13:26.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240201T111326Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=1IMYR3NXFV2H92HDXL01/20240201/front/smithi148/aws4_request,SignedHeaders=host;x-amz-date,Signature=593b777b1fd42f3f9c866f13215cbccab75e6138b22309204417fb2e4492710c', 'Host': 'smithi148.front.sepia.ceph.com:8000'} 2024-02-01T11:13:26.467 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:13:26.467 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:13:28.108 DEBUG:boto:Token: None 2024-02-01T11:13:28.109 DEBUG:boto:CanonicalRequest: POST / host:smithi148.front.sepia.ceph.com:8000 x-amz-date:20240201T111328Z host;x-amz-date 4efa4a360bd87db21c71e7ea0c35b095f1f1059d6445a9281c4f4753dce8b410 2024-02-01T11:13:28.110 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20240201T111328Z 20240201/front/smithi148/aws4_request 61c147c0534dc08320c2e6cbc51d550df462a463fb543d1d6e3aecdae3742f05 2024-02-01T11:13:28.110 DEBUG:boto:Signature: df1fe91fdf6acdad2cac783c030c9cc2b7b9a57a8ae25b70a8b9df118c1d6f70 2024-02-01T11:13:28.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240201T111328Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=1IMYR3NXFV2H92HDXL01/20240201/front/smithi148/aws4_request,SignedHeaders=host;x-amz-date,Signature=df1fe91fdf6acdad2cac783c030c9cc2b7b9a57a8ae25b70a8b9df118c1d6f70', 'Host': 'smithi148.front.sepia.ceph.com:8000'} 2024-02-01T11:13:28.112 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:13:28.112 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:13:30.240 DEBUG:boto:Token: None 2024-02-01T11:13:30.240 DEBUG:boto:CanonicalRequest: POST / host:smithi148.front.sepia.ceph.com:8000 x-amz-date:20240201T111330Z host;x-amz-date 4efa4a360bd87db21c71e7ea0c35b095f1f1059d6445a9281c4f4753dce8b410 2024-02-01T11:13:30.241 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20240201T111330Z 20240201/front/smithi148/aws4_request c5f9f151405c1f796c0b09e95932aeabd7a7b6cbef82e394d49cca73759b5113 2024-02-01T11:13:30.241 DEBUG:boto:Signature: bbb493ac48537d91387c677da94a31bcb1df0b86647a67686736156bff59908d 2024-02-01T11:13:30.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240201T111330Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=1IMYR3NXFV2H92HDXL01/20240201/front/smithi148/aws4_request,SignedHeaders=host;x-amz-date,Signature=bbb493ac48537d91387c677da94a31bcb1df0b86647a67686736156bff59908d', 'Host': 'smithi148.front.sepia.ceph.com:8000'} 2024-02-01T11:13:30.242 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:13:30.243 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:13:31.355 DEBUG:boto:Token: None 2024-02-01T11:13:31.355 DEBUG:boto:CanonicalRequest: POST / host:smithi148.front.sepia.ceph.com:8000 x-amz-date:20240201T111331Z host;x-amz-date 4efa4a360bd87db21c71e7ea0c35b095f1f1059d6445a9281c4f4753dce8b410 2024-02-01T11:13:31.355 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20240201T111331Z 20240201/front/smithi148/aws4_request 57f3e3d11b0131548af4d75f7431f79bb7ba19be38c2cd99aafb8ec33479d044 2024-02-01T11:13:31.355 DEBUG:boto:Signature: d97ebbca0c0a758b5a4dd217fe3ab7185bfff3de495bf0123f7ed91c997ba5e1 2024-02-01T11:13:31.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240201T111331Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=1IMYR3NXFV2H92HDXL01/20240201/front/smithi148/aws4_request,SignedHeaders=host;x-amz-date,Signature=d97ebbca0c0a758b5a4dd217fe3ab7185bfff3de495bf0123f7ed91c997ba5e1', 'Host': 'smithi148.front.sepia.ceph.com:8000'} 2024-02-01T11:13:31.357 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:13:31.357 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:13:52.759 DEBUG:boto:Token: None 2024-02-01T11:13:52.760 DEBUG:boto:CanonicalRequest: POST / host:smithi148.front.sepia.ceph.com:8000 x-amz-date:20240201T111352Z host;x-amz-date 4efa4a360bd87db21c71e7ea0c35b095f1f1059d6445a9281c4f4753dce8b410 2024-02-01T11:13:52.760 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20240201T111352Z 20240201/front/smithi148/aws4_request 39e61bb7f406c0343cc117f7381b12c374d274ed3f90df170d31f04adf78a119 2024-02-01T11:13:52.761 DEBUG:boto:Signature: 41efdaf5f718a5779ab3be1e67aa2008e749c1c2915c9c90973114c64ad55b20 2024-02-01T11:13:52.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240201T111352Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=1IMYR3NXFV2H92HDXL01/20240201/front/smithi148/aws4_request,SignedHeaders=host;x-amz-date,Signature=41efdaf5f718a5779ab3be1e67aa2008e749c1c2915c9c90973114c64ad55b20', 'Host': 'smithi148.front.sepia.ceph.com:8000'} 2024-02-01T11:13:52.763 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:13:52.763 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:14:51.247 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_role_delete_sync ... ERROR 2024-02-01T11:14:51.249 DEBUG:boto:path=/vclfcn-75/ 2024-02-01T11:14:51.249 DEBUG:boto:auth_path=/vclfcn-75/ 2024-02-01T11:14:51.249 DEBUG:boto:Method: PUT 2024-02-01T11:14:51.249 DEBUG:boto:Path: /vclfcn-75/ 2024-02-01T11:14:51.250 DEBUG:boto:Data: 2024-02-01T11:14:51.250 DEBUG:boto:Headers: {} 2024-02-01T11:14:51.250 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T11:14:51.250 DEBUG:boto:Port: 8000 2024-02-01T11:14:51.250 DEBUG:boto:Params: {} 2024-02-01T11:14:51.250 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:14:51.251 DEBUG:boto:Token: None 2024-02-01T11:14:51.251 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 11:14:51 GMT /vclfcn-75/ 2024-02-01T11:14:51.251 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:iPQhg6SV30aUjzFfDinkG8+tvQM= 2024-02-01T11:14:51.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:14:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iPQhg6SV30aUjzFfDinkG8+tvQM=', 'Content-Length': '0'} 2024-02-01T11:14:51.254 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:14:51.254 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:14:51.550 DEBUG:boto:Token: None 2024-02-01T11:14:51.550 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 11:14:51 GMT /vclfcn-75/ 2024-02-01T11:14:51.551 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:iPQhg6SV30aUjzFfDinkG8+tvQM= 2024-02-01T11:14:51.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:14:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iPQhg6SV30aUjzFfDinkG8+tvQM=', 'Content-Length': '0'} 2024-02-01T11:14:51.553 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:14:51.553 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:14:52.671 DEBUG:boto:Token: None 2024-02-01T11:14:52.672 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 11:14:51 GMT /vclfcn-75/ 2024-02-01T11:14:52.672 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:iPQhg6SV30aUjzFfDinkG8+tvQM= 2024-02-01T11:14:52.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:14:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iPQhg6SV30aUjzFfDinkG8+tvQM=', 'Content-Length': '0'} 2024-02-01T11:14:52.674 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:14:52.674 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:14:54.443 DEBUG:boto:Token: None 2024-02-01T11:14:54.443 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 11:14:51 GMT /vclfcn-75/ 2024-02-01T11:14:54.444 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:iPQhg6SV30aUjzFfDinkG8+tvQM= 2024-02-01T11:14:54.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:14:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iPQhg6SV30aUjzFfDinkG8+tvQM=', 'Content-Length': '0'} 2024-02-01T11:14:54.445 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:14:54.446 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:14:55.166 DEBUG:boto:Token: None 2024-02-01T11:14:55.166 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 11:14:51 GMT /vclfcn-75/ 2024-02-01T11:14:55.166 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:iPQhg6SV30aUjzFfDinkG8+tvQM= 2024-02-01T11:14:55.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:14:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iPQhg6SV30aUjzFfDinkG8+tvQM=', 'Content-Length': '0'} 2024-02-01T11:14:55.168 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:14:55.168 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:15:10.487 DEBUG:boto:Token: None 2024-02-01T11:15:10.488 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 11:14:51 GMT /vclfcn-75/ 2024-02-01T11:15:10.488 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:iPQhg6SV30aUjzFfDinkG8+tvQM= 2024-02-01T11:15:10.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:14:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iPQhg6SV30aUjzFfDinkG8+tvQM=', 'Content-Length': '0'} 2024-02-01T11:15:10.490 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:15:10.490 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:15:34.493 DEBUG:boto:Token: None 2024-02-01T11:15:34.494 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 11:14:51 GMT /vclfcn-75/ 2024-02-01T11:15:34.494 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:iPQhg6SV30aUjzFfDinkG8+tvQM= 2024-02-01T11:15:34.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:14:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iPQhg6SV30aUjzFfDinkG8+tvQM=', 'Content-Length': '0'} 2024-02-01T11:15:34.497 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:15:34.497 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:15:48.669 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_full_sync_after_data_sync_init ... ERROR 2024-02-01T11:15:48.671 DEBUG:boto:path=/vclfcn-76/ 2024-02-01T11:15:48.671 DEBUG:boto:auth_path=/vclfcn-76/ 2024-02-01T11:15:48.671 DEBUG:boto:Method: PUT 2024-02-01T11:15:48.671 DEBUG:boto:Path: /vclfcn-76/ 2024-02-01T11:15:48.671 DEBUG:boto:Data: 2024-02-01T11:15:48.671 DEBUG:boto:Headers: {} 2024-02-01T11:15:48.671 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T11:15:48.671 DEBUG:boto:Port: 8000 2024-02-01T11:15:48.672 DEBUG:boto:Params: {} 2024-02-01T11:15:48.672 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:15:48.672 DEBUG:boto:Token: None 2024-02-01T11:15:48.672 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 11:15:48 GMT /vclfcn-76/ 2024-02-01T11:15:48.672 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:3THkG+IhweOFI9+k19IXyy8xxbE= 2024-02-01T11:15:48.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:15:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:3THkG+IhweOFI9+k19IXyy8xxbE=', 'Content-Length': '0'} 2024-02-01T11:15:48.674 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:15:48.677 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:15:48.762 DEBUG:boto:Token: None 2024-02-01T11:15:48.762 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 11:15:48 GMT /vclfcn-76/ 2024-02-01T11:15:48.762 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:3THkG+IhweOFI9+k19IXyy8xxbE= 2024-02-01T11:15:48.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:15:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:3THkG+IhweOFI9+k19IXyy8xxbE=', 'Content-Length': '0'} 2024-02-01T11:15:48.764 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:15:48.764 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:15:50.601 DEBUG:boto:Token: None 2024-02-01T11:15:50.601 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 11:15:48 GMT /vclfcn-76/ 2024-02-01T11:15:50.601 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:3THkG+IhweOFI9+k19IXyy8xxbE= 2024-02-01T11:15:50.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:15:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:3THkG+IhweOFI9+k19IXyy8xxbE=', 'Content-Length': '0'} 2024-02-01T11:15:50.603 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:15:50.603 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:15:52.016 DEBUG:boto:Token: None 2024-02-01T11:15:52.017 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 11:15:48 GMT /vclfcn-76/ 2024-02-01T11:15:52.017 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:3THkG+IhweOFI9+k19IXyy8xxbE= 2024-02-01T11:15:52.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:15:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:3THkG+IhweOFI9+k19IXyy8xxbE=', 'Content-Length': '0'} 2024-02-01T11:15:52.019 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:15:52.019 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:15:59.622 DEBUG:boto:Token: None 2024-02-01T11:15:59.622 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 11:15:48 GMT /vclfcn-76/ 2024-02-01T11:15:59.622 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:3THkG+IhweOFI9+k19IXyy8xxbE= 2024-02-01T11:15:59.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:15:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:3THkG+IhweOFI9+k19IXyy8xxbE=', 'Content-Length': '0'} 2024-02-01T11:15:59.625 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:15:59.625 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:16:06.284 DEBUG:boto:Token: None 2024-02-01T11:16:06.285 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 11:15:48 GMT /vclfcn-76/ 2024-02-01T11:16:06.285 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:3THkG+IhweOFI9+k19IXyy8xxbE= 2024-02-01T11:16:06.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:15:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:3THkG+IhweOFI9+k19IXyy8xxbE=', 'Content-Length': '0'} 2024-02-01T11:16:06.287 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:16:06.287 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:16:12.109 DEBUG:boto:Token: None 2024-02-01T11:16:12.109 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 11:15:48 GMT /vclfcn-76/ 2024-02-01T11:16:12.109 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:3THkG+IhweOFI9+k19IXyy8xxbE= 2024-02-01T11:16:12.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:15:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:3THkG+IhweOFI9+k19IXyy8xxbE=', 'Content-Length': '0'} 2024-02-01T11:16:12.112 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:16:12.112 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:16:38.569 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_resharded_bucket_full_sync_after_data_sync_init ... ERROR 2024-02-01T11:16:38.570 DEBUG:boto:path=/vclfcn-77/ 2024-02-01T11:16:38.571 DEBUG:boto:auth_path=/vclfcn-77/ 2024-02-01T11:16:38.571 DEBUG:boto:Method: PUT 2024-02-01T11:16:38.571 DEBUG:boto:Path: /vclfcn-77/ 2024-02-01T11:16:38.571 DEBUG:boto:Data: 2024-02-01T11:16:38.571 DEBUG:boto:Headers: {} 2024-02-01T11:16:38.571 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T11:16:38.571 DEBUG:boto:Port: 8000 2024-02-01T11:16:38.571 DEBUG:boto:Params: {} 2024-02-01T11:16:38.571 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:16:38.572 DEBUG:boto:Token: None 2024-02-01T11:16:38.572 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 11:16:38 GMT /vclfcn-77/ 2024-02-01T11:16:38.572 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:dYaBPDFTdsOPyzBhPQMARzX8lu4= 2024-02-01T11:16:38.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:16:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dYaBPDFTdsOPyzBhPQMARzX8lu4=', 'Content-Length': '0'} 2024-02-01T11:16:38.574 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:16:38.574 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:16:38.865 DEBUG:boto:Token: None 2024-02-01T11:16:38.865 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 11:16:38 GMT /vclfcn-77/ 2024-02-01T11:16:38.865 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:dYaBPDFTdsOPyzBhPQMARzX8lu4= 2024-02-01T11:16:38.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:16:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dYaBPDFTdsOPyzBhPQMARzX8lu4=', 'Content-Length': '0'} 2024-02-01T11:16:38.867 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:16:38.867 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:16:40.340 DEBUG:boto:Token: None 2024-02-01T11:16:40.340 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 11:16:38 GMT /vclfcn-77/ 2024-02-01T11:16:40.340 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:dYaBPDFTdsOPyzBhPQMARzX8lu4= 2024-02-01T11:16:40.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:16:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dYaBPDFTdsOPyzBhPQMARzX8lu4=', 'Content-Length': '0'} 2024-02-01T11:16:40.342 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:16:40.342 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:16:43.120 DEBUG:boto:Token: None 2024-02-01T11:16:43.120 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 11:16:38 GMT /vclfcn-77/ 2024-02-01T11:16:43.120 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:dYaBPDFTdsOPyzBhPQMARzX8lu4= 2024-02-01T11:16:43.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:16:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dYaBPDFTdsOPyzBhPQMARzX8lu4=', 'Content-Length': '0'} 2024-02-01T11:16:43.123 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:16:43.123 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:16:45.192 DEBUG:boto:Token: None 2024-02-01T11:16:45.193 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 11:16:38 GMT /vclfcn-77/ 2024-02-01T11:16:45.193 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:dYaBPDFTdsOPyzBhPQMARzX8lu4= 2024-02-01T11:16:45.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:16:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dYaBPDFTdsOPyzBhPQMARzX8lu4=', 'Content-Length': '0'} 2024-02-01T11:16:45.195 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:16:45.196 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:16:49.987 DEBUG:boto:Token: None 2024-02-01T11:16:49.988 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 11:16:38 GMT /vclfcn-77/ 2024-02-01T11:16:49.988 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:dYaBPDFTdsOPyzBhPQMARzX8lu4= 2024-02-01T11:16:49.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:16:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dYaBPDFTdsOPyzBhPQMARzX8lu4=', 'Content-Length': '0'} 2024-02-01T11:16:49.991 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:16:49.991 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:16:52.443 DEBUG:boto:Token: None 2024-02-01T11:16:52.443 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 11:16:38 GMT /vclfcn-77/ 2024-02-01T11:16:52.443 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:dYaBPDFTdsOPyzBhPQMARzX8lu4= 2024-02-01T11:16:52.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:16:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dYaBPDFTdsOPyzBhPQMARzX8lu4=', 'Content-Length': '0'} 2024-02-01T11:16:52.445 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:16:52.445 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:17:37.969 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_incremental_sync_after_data_sync_init ... ERROR 2024-02-01T11:17:37.971 DEBUG:boto:path=/vclfcn-78/ 2024-02-01T11:17:37.971 DEBUG:boto:auth_path=/vclfcn-78/ 2024-02-01T11:17:37.971 DEBUG:boto:Method: PUT 2024-02-01T11:17:37.971 DEBUG:boto:Path: /vclfcn-78/ 2024-02-01T11:17:37.971 DEBUG:boto:Data: 2024-02-01T11:17:37.972 DEBUG:boto:Headers: {} 2024-02-01T11:17:37.972 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T11:17:37.972 DEBUG:boto:Port: 8000 2024-02-01T11:17:37.972 DEBUG:boto:Params: {} 2024-02-01T11:17:37.972 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:17:37.972 DEBUG:boto:Token: None 2024-02-01T11:17:37.973 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 11:17:37 GMT /vclfcn-78/ 2024-02-01T11:17:37.973 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:civvKzBwcFUkJ7sJqPCCSryUAww= 2024-02-01T11:17:37.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:17:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:civvKzBwcFUkJ7sJqPCCSryUAww=', 'Content-Length': '0'} 2024-02-01T11:17:37.975 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:17:37.976 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:17:38.480 DEBUG:boto:Token: None 2024-02-01T11:17:38.480 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 11:17:37 GMT /vclfcn-78/ 2024-02-01T11:17:38.480 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:civvKzBwcFUkJ7sJqPCCSryUAww= 2024-02-01T11:17:38.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:17:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:civvKzBwcFUkJ7sJqPCCSryUAww=', 'Content-Length': '0'} 2024-02-01T11:17:38.482 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:17:38.483 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:17:39.554 DEBUG:boto:Token: None 2024-02-01T11:17:39.554 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 11:17:37 GMT /vclfcn-78/ 2024-02-01T11:17:39.555 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:civvKzBwcFUkJ7sJqPCCSryUAww= 2024-02-01T11:17:39.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:17:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:civvKzBwcFUkJ7sJqPCCSryUAww=', 'Content-Length': '0'} 2024-02-01T11:17:39.556 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:17:39.556 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:17:41.551 DEBUG:boto:Token: None 2024-02-01T11:17:41.552 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 11:17:37 GMT /vclfcn-78/ 2024-02-01T11:17:41.552 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:civvKzBwcFUkJ7sJqPCCSryUAww= 2024-02-01T11:17:41.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:17:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:civvKzBwcFUkJ7sJqPCCSryUAww=', 'Content-Length': '0'} 2024-02-01T11:17:41.554 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:17:41.554 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:17:43.545 DEBUG:boto:Token: None 2024-02-01T11:17:43.545 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 11:17:37 GMT /vclfcn-78/ 2024-02-01T11:17:43.545 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:civvKzBwcFUkJ7sJqPCCSryUAww= 2024-02-01T11:17:43.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:17:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:civvKzBwcFUkJ7sJqPCCSryUAww=', 'Content-Length': '0'} 2024-02-01T11:17:43.547 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:17:43.547 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:17:46.367 DEBUG:boto:Token: None 2024-02-01T11:17:46.367 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 11:17:37 GMT /vclfcn-78/ 2024-02-01T11:17:46.367 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:civvKzBwcFUkJ7sJqPCCSryUAww= 2024-02-01T11:17:46.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:17:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:civvKzBwcFUkJ7sJqPCCSryUAww=', 'Content-Length': '0'} 2024-02-01T11:17:46.369 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:17:46.369 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:18:07.962 DEBUG:boto:Token: None 2024-02-01T11:18:07.963 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 11:17:37 GMT /vclfcn-78/ 2024-02-01T11:18:07.963 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:civvKzBwcFUkJ7sJqPCCSryUAww= 2024-02-01T11:18:07.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:17:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:civvKzBwcFUkJ7sJqPCCSryUAww=', 'Content-Length': '0'} 2024-02-01T11:18:07.966 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:18:07.967 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:18:46.915 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_resharded_bucket_incremental_sync_latest_after_data_sync_init ... ERROR 2024-02-01T11:18:46.917 DEBUG:boto:path=/vclfcn-79/ 2024-02-01T11:18:46.917 DEBUG:boto:auth_path=/vclfcn-79/ 2024-02-01T11:18:46.917 DEBUG:boto:Method: PUT 2024-02-01T11:18:46.917 DEBUG:boto:Path: /vclfcn-79/ 2024-02-01T11:18:46.918 DEBUG:boto:Data: 2024-02-01T11:18:46.918 DEBUG:boto:Headers: {} 2024-02-01T11:18:46.918 DEBUG:boto:Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T11:18:46.918 DEBUG:boto:Port: 8000 2024-02-01T11:18:46.918 DEBUG:boto:Params: {} 2024-02-01T11:18:46.918 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:18:46.918 DEBUG:boto:Token: None 2024-02-01T11:18:46.919 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 11:18:46 GMT /vclfcn-79/ 2024-02-01T11:18:46.919 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mWUqgNYFSRyiwBwFGSsJimsjaVM= 2024-02-01T11:18:46.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:18:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mWUqgNYFSRyiwBwFGSsJimsjaVM=', 'Content-Length': '0'} 2024-02-01T11:18:46.922 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:18:46.922 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:18:47.777 DEBUG:boto:Token: None 2024-02-01T11:18:47.777 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 11:18:46 GMT /vclfcn-79/ 2024-02-01T11:18:47.777 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mWUqgNYFSRyiwBwFGSsJimsjaVM= 2024-02-01T11:18:47.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:18:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mWUqgNYFSRyiwBwFGSsJimsjaVM=', 'Content-Length': '0'} 2024-02-01T11:18:47.779 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:18:47.779 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:18:49.638 DEBUG:boto:Token: None 2024-02-01T11:18:49.638 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 11:18:46 GMT /vclfcn-79/ 2024-02-01T11:18:49.638 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mWUqgNYFSRyiwBwFGSsJimsjaVM= 2024-02-01T11:18:49.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:18:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mWUqgNYFSRyiwBwFGSsJimsjaVM=', 'Content-Length': '0'} 2024-02-01T11:18:49.641 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:18:49.641 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:18:50.839 DEBUG:boto:Token: None 2024-02-01T11:18:50.840 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 11:18:46 GMT /vclfcn-79/ 2024-02-01T11:18:50.840 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mWUqgNYFSRyiwBwFGSsJimsjaVM= 2024-02-01T11:18:50.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:18:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mWUqgNYFSRyiwBwFGSsJimsjaVM=', 'Content-Length': '0'} 2024-02-01T11:18:50.842 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:18:50.842 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:18:55.286 DEBUG:boto:Token: None 2024-02-01T11:18:55.287 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 11:18:46 GMT /vclfcn-79/ 2024-02-01T11:18:55.287 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mWUqgNYFSRyiwBwFGSsJimsjaVM= 2024-02-01T11:18:55.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:18:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mWUqgNYFSRyiwBwFGSsJimsjaVM=', 'Content-Length': '0'} 2024-02-01T11:18:55.289 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:18:55.289 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:19:03.659 DEBUG:boto:Token: None 2024-02-01T11:19:03.660 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 11:18:46 GMT /vclfcn-79/ 2024-02-01T11:19:03.660 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mWUqgNYFSRyiwBwFGSsJimsjaVM= 2024-02-01T11:19:03.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:18:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mWUqgNYFSRyiwBwFGSsJimsjaVM=', 'Content-Length': '0'} 2024-02-01T11:19:03.662 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:19:03.662 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:19:30.620 DEBUG:boto:Token: None 2024-02-01T11:19:30.620 DEBUG:boto:StringToSign: PUT Thu, 01 Feb 2024 11:18:46 GMT /vclfcn-79/ 2024-02-01T11:19:30.621 DEBUG:boto:Signature: AWS 1IMYR3NXFV2H92HDXL01:mWUqgNYFSRyiwBwFGSsJimsjaVM= 2024-02-01T11:19:30.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:18:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mWUqgNYFSRyiwBwFGSsJimsjaVM=', 'Content-Length': '0'} 2024-02-01T11:19:30.623 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2024-02-01T11:19:30.623 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T11:20:07.255 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_resharded_bucket_incremental_sync_oldest_after_data_sync_init ... ERROR 2024-02-01T11:20:07.256 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T11:20:07.257 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T11:20:07.257 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T11:20:07.299 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T11:20:07.299 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T11:25:07.301 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T11:25:07.299+0000 7f22342b4980 0 monclient(hunting): authenticate timed out after 300 2024-02-01T11:25:07.302 INFO:teuthology.orchestra.run.smithi148.stderr:failed to fetch mon config (--no-mon-config to skip) 2024-02-01T11:25:07.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-01T11:25:07.305 INFO:tasks.rgw_multisite_tests:test_sync_policy_config_zonegroup: ... FAIL 2024-02-01T11:25:07.306 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T11:25:07.306 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T11:25:07.306 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T11:25:07.386 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T11:25:07.386 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T11:30:07.389 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T11:30:07.388+0000 7f9742cf1980 0 monclient(hunting): authenticate timed out after 300 2024-02-01T11:30:07.389 INFO:teuthology.orchestra.run.smithi148.stderr:failed to fetch mon config (--no-mon-config to skip) 2024-02-01T11:30:07.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-01T11:30:07.393 INFO:tasks.rgw_multisite_tests:test_sync_flow_symmetrical_zonegroup_all: ... FAIL 2024-02-01T11:30:07.395 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T11:30:07.395 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T11:30:07.396 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T11:30:07.473 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T11:30:07.473 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T11:35:07.475 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T11:35:07.477+0000 7f95e170a980 0 monclient(hunting): authenticate timed out after 300 2024-02-01T11:35:07.476 INFO:teuthology.orchestra.run.smithi148.stderr:failed to fetch mon config (--no-mon-config to skip) 2024-02-01T11:35:07.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-01T11:35:07.479 INFO:tasks.rgw_multisite_tests:test_sync_flow_symmetrical_zonegroup_select: ... FAIL 2024-02-01T11:35:07.480 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T11:35:07.481 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T11:35:07.481 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T11:35:07.562 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T11:35:07.562 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T11:40:07.565 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T11:40:07.566+0000 7fa12f3bb980 0 monclient(hunting): authenticate timed out after 300 2024-02-01T11:40:08.045 INFO:teuthology.orchestra.run.smithi148.stderr:failed to fetch mon config (--no-mon-config to skip) 2024-02-01T11:40:08.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-01T11:40:08.048 INFO:tasks.rgw_multisite_tests:test_sync_flow_directional_zonegroup_select: ... FAIL 2024-02-01T11:40:08.049 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T11:40:08.049 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T11:40:08.050 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T11:40:08.131 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T11:40:08.132 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T11:45:08.134 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T11:45:08.135+0000 7fcabd39d980 0 monclient(hunting): authenticate timed out after 300 2024-02-01T11:45:08.135 INFO:teuthology.orchestra.run.smithi148.stderr:failed to fetch mon config (--no-mon-config to skip) 2024-02-01T11:45:08.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-01T11:45:08.138 INFO:tasks.rgw_multisite_tests:test_sync_single_bucket: ... FAIL 2024-02-01T11:45:08.139 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T11:45:08.139 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T11:45:08.140 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T11:45:08.217 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T11:45:08.218 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T11:50:08.220 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T11:50:08.220+0000 7fd34d050980 0 monclient(hunting): authenticate timed out after 300 2024-02-01T11:50:08.220 INFO:teuthology.orchestra.run.smithi148.stderr:failed to fetch mon config (--no-mon-config to skip) 2024-02-01T11:50:08.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-01T11:50:08.223 INFO:tasks.rgw_multisite_tests:test_sync_different_buckets: ... FAIL 2024-02-01T11:50:08.224 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T11:50:08.224 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T11:50:08.225 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T11:50:08.307 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T11:50:08.307 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T11:55:08.308 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T11:55:08.309+0000 7fef7cbdb980 0 monclient(hunting): authenticate timed out after 300 2024-02-01T11:55:08.309 INFO:teuthology.orchestra.run.smithi148.stderr:failed to fetch mon config (--no-mon-config to skip) 2024-02-01T11:55:08.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-01T11:55:08.312 INFO:tasks.rgw_multisite_tests:test_sync_multiple_buckets_to_single: ... FAIL 2024-02-01T11:55:08.313 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T11:55:08.313 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T11:55:08.313 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T11:55:08.398 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T11:55:08.399 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T12:00:08.400 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T12:00:08.399+0000 7f58a9a5d980 0 monclient(hunting): authenticate timed out after 300 2024-02-01T12:00:08.401 INFO:teuthology.orchestra.run.smithi148.stderr:failed to fetch mon config (--no-mon-config to skip) 2024-02-01T12:00:08.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-01T12:00:08.404 INFO:tasks.rgw_multisite_tests:test_sync_single_bucket_to_multiple: ... FAIL 2024-02-01T12:00:08.405 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_create_rgw_down ... SKIP: test_bucket_create_rgw_down skipped. More than one rgw needed in any one or multiple zone(s). 2024-02-01T12:00:08.406 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_remove_rgw_down ... SKIP: test_bucket_remove_rgw_down skipped. More than one rgw needed in any one or multiple zone(s). 2024-02-01T12:00:08.406 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_object_sync_rgw_down ... SKIP: test_object_sync_rgw_down skipped. More than one rgw needed in any one or multiple zone(s). 2024-02-01T12:00:08.406 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_object_delete_rgw_down ... SKIP: test_object_delete_rgw_down skipped. More than one rgw needed in any one or multiple zone(s). 2024-02-01T12:00:08.407 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_concurrent_versioned_object_incremental_sync_rgw_down ... SKIP: test_concurrent_versioned_object_incremental_sync_rgw_down skipped. More than one rgw needed in any one or multiple zone(s). 2024-02-01T12:00:08.407 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_suspended_delete_marker_full_sync_rgw_down ... SKIP: test_suspended_delete_marker_full_sync_rgw_down skipped. More than one rgw needed in any one or multiple zone(s). 2024-02-01T12:00:08.407 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_acl_rgw_down ... SKIP: test_bucket_acl_rgw_down skipped. More than one rgw needed in any one or multiple zone(s). 2024-02-01T12:00:08.408 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_enable_right_after_disable_rgw_down ... SKIP: test_bucket_sync_enable_right_after_disable_rgw_down skipped. More than one rgw needed in any one or multiple zone(s). 2024-02-01T12:00:08.408 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multipart_object_sync_rgw_down ... SKIP: test_multipart_object_sync_rgw_down skipped. More than one rgw needed in any one or multiple zone(s). 2024-02-01T12:00:08.408 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_run_basic_incremental_rgw_down ... SKIP: test_bucket_sync_run_basic_incremental_rgw_down skipped. More than one rgw needed in any one or multiple zone(s). 2024-02-01T12:00:08.409 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_role_sync_rgw_down ... SKIP: test_role_sync_rgw_down skipped. More than one rgw needed in any one or multiple zone(s). 2024-02-01T12:00:08.409 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_full_sync_after_data_sync_init_rgw_down ... SKIP: test_bucket_full_sync_after_data_sync_init_rgw_down skipped. More than one rgw needed in any one or multiple zone(s). 2024-02-01T12:00:08.409 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_sync_policy_config_zonegroup_rgw_down ... SKIP: test_sync_policy_config_zonegroup_rgw_down skipped. More than one rgw needed in any one or multiple zone(s). 2024-02-01T12:00:08.410 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_sync_flow_symmetrical_zonegroup_all_rgw_down ... SKIP: test_sync_flow_symmetrical_zonegroup_all_rgw_down skipped. More than one rgw needed in any one or multiple zone(s). 2024-02-01T12:00:08.410 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.410 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-02-01T12:00:08.410 INFO:tasks.rgw_multisite_tests:ERROR: rgw_multi.tests.test_bucket_index_log_trim 2024-02-01T12:00:08.410 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-02-01T12:00:08.411 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-02-01T12:00:08.411 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-02-01T12:00:08.411 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-02-01T12:00:08.411 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/../src/test/rgw/rgw_multi/tests.py", line 1312, in test_bucket_index_log_trim 2024-02-01T12:00:08.411 INFO:tasks.rgw_multisite_tests: cold_bucket = make_test_bucket() 2024-02-01T12:00:08.411 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/../src/test/rgw/rgw_multi/tests.py", line 1303, in make_test_bucket 2024-02-01T12:00:08.411 INFO:tasks.rgw_multisite_tests: bucket = zone.conn.create_bucket(name) 2024-02-01T12:00:08.411 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/boto/s3/connection.py", line 618, in create_bucket 2024-02-01T12:00:08.411 INFO:tasks.rgw_multisite_tests: response = self.make_request('PUT', bucket_name, headers=headers, 2024-02-01T12:00:08.411 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/boto/s3/connection.py", line 667, in make_request 2024-02-01T12:00:08.411 INFO:tasks.rgw_multisite_tests: return super(S3Connection, self).make_request( 2024-02-01T12:00:08.411 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 1070, in make_request 2024-02-01T12:00:08.411 INFO:tasks.rgw_multisite_tests: return self._mexe(http_request, sender, override_num_retries, 2024-02-01T12:00:08.412 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 1030, in _mexe 2024-02-01T12:00:08.412 INFO:tasks.rgw_multisite_tests: raise ex 2024-02-01T12:00:08.412 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 942, in _mexe 2024-02-01T12:00:08.412 INFO:tasks.rgw_multisite_tests: connection.request(request.method, request.path, 2024-02-01T12:00:08.412 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1256, in request 2024-02-01T12:00:08.412 INFO:tasks.rgw_multisite_tests: self._send_request(method, url, body, headers, encode_chunked) 2024-02-01T12:00:08.412 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1302, in _send_request 2024-02-01T12:00:08.412 INFO:tasks.rgw_multisite_tests: self.endheaders(body, encode_chunked=encode_chunked) 2024-02-01T12:00:08.412 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1251, in endheaders 2024-02-01T12:00:08.412 INFO:tasks.rgw_multisite_tests: self._send_output(message_body, encode_chunked=encode_chunked) 2024-02-01T12:00:08.412 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1011, in _send_output 2024-02-01T12:00:08.412 INFO:tasks.rgw_multisite_tests: self.send(msg) 2024-02-01T12:00:08.412 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 951, in send 2024-02-01T12:00:08.413 INFO:tasks.rgw_multisite_tests: self.connect() 2024-02-01T12:00:08.413 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 922, in connect 2024-02-01T12:00:08.413 INFO:tasks.rgw_multisite_tests: self.sock = self._create_connection( 2024-02-01T12:00:08.413 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/gevent/socket.py", line 115, in create_connection 2024-02-01T12:00:08.413 INFO:tasks.rgw_multisite_tests: sock.connect(sa) 2024-02-01T12:00:08.413 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 590, in connect 2024-02-01T12:00:08.413 INFO:tasks.rgw_multisite_tests: self._internal_connect(address) 2024-02-01T12:00:08.413 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 634, in _internal_connect 2024-02-01T12:00:08.413 INFO:tasks.rgw_multisite_tests: raise _SocketError(err, strerror(err)) 2024-02-01T12:00:08.413 INFO:tasks.rgw_multisite_tests:ConnectionRefusedError: [Errno 111] Connection refused 2024-02-01T12:00:08.413 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-02-01T12:00:08.413 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone1 name=vclfcn-69 2024-02-01T12:00:08.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vclfcn-69/ 2024-02-01T12:00:08.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vclfcn-69/ 2024-02-01T12:00:08.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-02-01T12:00:08.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vclfcn-69/ 2024-02-01T12:00:08.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-02-01T12:00:08.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-02-01T12:00:08.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T12:00:08.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-02-01T12:00:08.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-02-01T12:00:08.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.415 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.415 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.415 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.415 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 11:00:48 GMT 2024-02-01T12:00:08.415 INFO:tasks.rgw_multisite_tests:/vclfcn-69/ 2024-02-01T12:00:08.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.415 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:aY1JEqvh4MlozEoELLurVH8ZpEg= 2024-02-01T12:00:08.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:00:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:aY1JEqvh4MlozEoELLurVH8ZpEg=', 'Content-Length': '0'} 2024-02-01T12:00:08.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.416 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.416 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.416 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.416 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 11:00:48 GMT 2024-02-01T12:00:08.416 INFO:tasks.rgw_multisite_tests:/vclfcn-69/ 2024-02-01T12:00:08.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.416 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:aY1JEqvh4MlozEoELLurVH8ZpEg= 2024-02-01T12:00:08.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:00:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:aY1JEqvh4MlozEoELLurVH8ZpEg=', 'Content-Length': '0'} 2024-02-01T12:00:08.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.417 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.417 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.417 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.417 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 11:00:48 GMT 2024-02-01T12:00:08.417 INFO:tasks.rgw_multisite_tests:/vclfcn-69/ 2024-02-01T12:00:08.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.417 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:aY1JEqvh4MlozEoELLurVH8ZpEg= 2024-02-01T12:00:08.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:00:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:aY1JEqvh4MlozEoELLurVH8ZpEg=', 'Content-Length': '0'} 2024-02-01T12:00:08.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.417 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.418 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.418 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.418 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 11:00:48 GMT 2024-02-01T12:00:08.418 INFO:tasks.rgw_multisite_tests:/vclfcn-69/ 2024-02-01T12:00:08.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.418 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:aY1JEqvh4MlozEoELLurVH8ZpEg= 2024-02-01T12:00:08.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:00:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:aY1JEqvh4MlozEoELLurVH8ZpEg=', 'Content-Length': '0'} 2024-02-01T12:00:08.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.418 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.418 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.419 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.419 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 11:00:48 GMT 2024-02-01T12:00:08.419 INFO:tasks.rgw_multisite_tests:/vclfcn-69/ 2024-02-01T12:00:08.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.419 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:aY1JEqvh4MlozEoELLurVH8ZpEg= 2024-02-01T12:00:08.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:00:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:aY1JEqvh4MlozEoELLurVH8ZpEg=', 'Content-Length': '0'} 2024-02-01T12:00:08.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.419 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.419 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.419 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.420 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 11:00:48 GMT 2024-02-01T12:00:08.420 INFO:tasks.rgw_multisite_tests:/vclfcn-69/ 2024-02-01T12:00:08.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.420 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:aY1JEqvh4MlozEoELLurVH8ZpEg= 2024-02-01T12:00:08.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:00:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:aY1JEqvh4MlozEoELLurVH8ZpEg=', 'Content-Length': '0'} 2024-02-01T12:00:08.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.420 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.420 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.420 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.420 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 11:00:48 GMT 2024-02-01T12:00:08.420 INFO:tasks.rgw_multisite_tests:/vclfcn-69/ 2024-02-01T12:00:08.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.421 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:aY1JEqvh4MlozEoELLurVH8ZpEg= 2024-02-01T12:00:08.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:00:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:aY1JEqvh4MlozEoELLurVH8ZpEg=', 'Content-Length': '0'} 2024-02-01T12:00:08.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.421 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-02-01T12:00:08.422 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.422 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-02-01T12:00:08.422 INFO:tasks.rgw_multisite_tests:ERROR: rgw_multi.tests.test_bucket_reshard_index_log_trim 2024-02-01T12:00:08.422 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-02-01T12:00:08.422 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-02-01T12:00:08.422 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-02-01T12:00:08.422 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-02-01T12:00:08.422 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/../src/test/rgw/rgw_multi/tests.py", line 1366, in test_bucket_reshard_index_log_trim 2024-02-01T12:00:08.422 INFO:tasks.rgw_multisite_tests: test_bucket = make_test_bucket() 2024-02-01T12:00:08.423 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/../src/test/rgw/rgw_multi/tests.py", line 1357, in make_test_bucket 2024-02-01T12:00:08.423 INFO:tasks.rgw_multisite_tests: bucket = zone.conn.create_bucket(name) 2024-02-01T12:00:08.423 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/boto/s3/connection.py", line 618, in create_bucket 2024-02-01T12:00:08.423 INFO:tasks.rgw_multisite_tests: response = self.make_request('PUT', bucket_name, headers=headers, 2024-02-01T12:00:08.423 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/boto/s3/connection.py", line 667, in make_request 2024-02-01T12:00:08.423 INFO:tasks.rgw_multisite_tests: return super(S3Connection, self).make_request( 2024-02-01T12:00:08.423 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 1070, in make_request 2024-02-01T12:00:08.423 INFO:tasks.rgw_multisite_tests: return self._mexe(http_request, sender, override_num_retries, 2024-02-01T12:00:08.423 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 1030, in _mexe 2024-02-01T12:00:08.423 INFO:tasks.rgw_multisite_tests: raise ex 2024-02-01T12:00:08.423 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 942, in _mexe 2024-02-01T12:00:08.423 INFO:tasks.rgw_multisite_tests: connection.request(request.method, request.path, 2024-02-01T12:00:08.423 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1256, in request 2024-02-01T12:00:08.423 INFO:tasks.rgw_multisite_tests: self._send_request(method, url, body, headers, encode_chunked) 2024-02-01T12:00:08.424 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1302, in _send_request 2024-02-01T12:00:08.424 INFO:tasks.rgw_multisite_tests: self.endheaders(body, encode_chunked=encode_chunked) 2024-02-01T12:00:08.424 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1251, in endheaders 2024-02-01T12:00:08.424 INFO:tasks.rgw_multisite_tests: self._send_output(message_body, encode_chunked=encode_chunked) 2024-02-01T12:00:08.424 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1011, in _send_output 2024-02-01T12:00:08.424 INFO:tasks.rgw_multisite_tests: self.send(msg) 2024-02-01T12:00:08.424 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 951, in send 2024-02-01T12:00:08.424 INFO:tasks.rgw_multisite_tests: self.connect() 2024-02-01T12:00:08.424 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 922, in connect 2024-02-01T12:00:08.424 INFO:tasks.rgw_multisite_tests: self.sock = self._create_connection( 2024-02-01T12:00:08.424 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/gevent/socket.py", line 115, in create_connection 2024-02-01T12:00:08.424 INFO:tasks.rgw_multisite_tests: sock.connect(sa) 2024-02-01T12:00:08.425 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 590, in connect 2024-02-01T12:00:08.425 INFO:tasks.rgw_multisite_tests: self._internal_connect(address) 2024-02-01T12:00:08.425 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 634, in _internal_connect 2024-02-01T12:00:08.425 INFO:tasks.rgw_multisite_tests: raise _SocketError(err, strerror(err)) 2024-02-01T12:00:08.425 INFO:tasks.rgw_multisite_tests:ConnectionRefusedError: [Errno 111] Connection refused 2024-02-01T12:00:08.425 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-02-01T12:00:08.425 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone1 name=vclfcn-70 2024-02-01T12:00:08.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vclfcn-70/ 2024-02-01T12:00:08.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vclfcn-70/ 2024-02-01T12:00:08.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-02-01T12:00:08.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vclfcn-70/ 2024-02-01T12:00:08.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-02-01T12:00:08.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-02-01T12:00:08.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T12:00:08.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-02-01T12:00:08.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-02-01T12:00:08.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.426 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.426 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.426 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.426 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 11:01:58 GMT 2024-02-01T12:00:08.426 INFO:tasks.rgw_multisite_tests:/vclfcn-70/ 2024-02-01T12:00:08.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.426 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:9/yOre7A8udHqCFoiuRpT/ehCO8= 2024-02-01T12:00:08.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:01:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9/yOre7A8udHqCFoiuRpT/ehCO8=', 'Content-Length': '0'} 2024-02-01T12:00:08.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.427 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.427 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.427 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.427 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 11:01:58 GMT 2024-02-01T12:00:08.427 INFO:tasks.rgw_multisite_tests:/vclfcn-70/ 2024-02-01T12:00:08.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.427 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:9/yOre7A8udHqCFoiuRpT/ehCO8= 2024-02-01T12:00:08.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:01:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9/yOre7A8udHqCFoiuRpT/ehCO8=', 'Content-Length': '0'} 2024-02-01T12:00:08.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.428 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.428 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.428 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.428 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 11:01:58 GMT 2024-02-01T12:00:08.428 INFO:tasks.rgw_multisite_tests:/vclfcn-70/ 2024-02-01T12:00:08.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.428 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:9/yOre7A8udHqCFoiuRpT/ehCO8= 2024-02-01T12:00:08.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:01:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9/yOre7A8udHqCFoiuRpT/ehCO8=', 'Content-Length': '0'} 2024-02-01T12:00:08.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.429 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.429 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.429 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.429 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 11:01:58 GMT 2024-02-01T12:00:08.429 INFO:tasks.rgw_multisite_tests:/vclfcn-70/ 2024-02-01T12:00:08.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.429 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:9/yOre7A8udHqCFoiuRpT/ehCO8= 2024-02-01T12:00:08.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:01:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9/yOre7A8udHqCFoiuRpT/ehCO8=', 'Content-Length': '0'} 2024-02-01T12:00:08.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.430 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.430 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.430 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.430 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 11:01:58 GMT 2024-02-01T12:00:08.430 INFO:tasks.rgw_multisite_tests:/vclfcn-70/ 2024-02-01T12:00:08.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.430 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:9/yOre7A8udHqCFoiuRpT/ehCO8= 2024-02-01T12:00:08.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:01:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9/yOre7A8udHqCFoiuRpT/ehCO8=', 'Content-Length': '0'} 2024-02-01T12:00:08.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.430 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.431 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.431 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.431 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 11:01:58 GMT 2024-02-01T12:00:08.431 INFO:tasks.rgw_multisite_tests:/vclfcn-70/ 2024-02-01T12:00:08.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.431 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:9/yOre7A8udHqCFoiuRpT/ehCO8= 2024-02-01T12:00:08.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:01:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9/yOre7A8udHqCFoiuRpT/ehCO8=', 'Content-Length': '0'} 2024-02-01T12:00:08.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.431 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.431 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.431 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.432 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 11:01:58 GMT 2024-02-01T12:00:08.432 INFO:tasks.rgw_multisite_tests:/vclfcn-70/ 2024-02-01T12:00:08.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.432 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:9/yOre7A8udHqCFoiuRpT/ehCO8= 2024-02-01T12:00:08.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:01:58 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:9/yOre7A8udHqCFoiuRpT/ehCO8=', 'Content-Length': '0'} 2024-02-01T12:00:08.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.432 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-02-01T12:00:08.432 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.432 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-02-01T12:00:08.432 INFO:tasks.rgw_multisite_tests:ERROR: rgw_multi.tests.test_bucket_reshard_incremental 2024-02-01T12:00:08.432 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-02-01T12:00:08.433 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-02-01T12:00:08.433 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-02-01T12:00:08.433 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-02-01T12:00:08.433 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/../src/test/rgw/rgw_multi/tests.py", line 1441, in test_bucket_reshard_incremental 2024-02-01T12:00:08.433 INFO:tasks.rgw_multisite_tests: bucket = zone.create_bucket(gen_bucket_name()) 2024-02-01T12:00:08.433 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/tasks/rgw_multi/zone_rados.py", line 56, in create_bucket 2024-02-01T12:00:08.433 INFO:tasks.rgw_multisite_tests: return self.conn.create_bucket(name) 2024-02-01T12:00:08.433 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/boto/s3/connection.py", line 618, in create_bucket 2024-02-01T12:00:08.433 INFO:tasks.rgw_multisite_tests: response = self.make_request('PUT', bucket_name, headers=headers, 2024-02-01T12:00:08.433 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/boto/s3/connection.py", line 667, in make_request 2024-02-01T12:00:08.433 INFO:tasks.rgw_multisite_tests: return super(S3Connection, self).make_request( 2024-02-01T12:00:08.433 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 1070, in make_request 2024-02-01T12:00:08.434 INFO:tasks.rgw_multisite_tests: return self._mexe(http_request, sender, override_num_retries, 2024-02-01T12:00:08.434 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 1030, in _mexe 2024-02-01T12:00:08.434 INFO:tasks.rgw_multisite_tests: raise ex 2024-02-01T12:00:08.434 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 942, in _mexe 2024-02-01T12:00:08.434 INFO:tasks.rgw_multisite_tests: connection.request(request.method, request.path, 2024-02-01T12:00:08.434 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1256, in request 2024-02-01T12:00:08.434 INFO:tasks.rgw_multisite_tests: self._send_request(method, url, body, headers, encode_chunked) 2024-02-01T12:00:08.434 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1302, in _send_request 2024-02-01T12:00:08.434 INFO:tasks.rgw_multisite_tests: self.endheaders(body, encode_chunked=encode_chunked) 2024-02-01T12:00:08.434 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1251, in endheaders 2024-02-01T12:00:08.434 INFO:tasks.rgw_multisite_tests: self._send_output(message_body, encode_chunked=encode_chunked) 2024-02-01T12:00:08.434 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1011, in _send_output 2024-02-01T12:00:08.434 INFO:tasks.rgw_multisite_tests: self.send(msg) 2024-02-01T12:00:08.434 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 951, in send 2024-02-01T12:00:08.435 INFO:tasks.rgw_multisite_tests: self.connect() 2024-02-01T12:00:08.435 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 922, in connect 2024-02-01T12:00:08.435 INFO:tasks.rgw_multisite_tests: self.sock = self._create_connection( 2024-02-01T12:00:08.435 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/gevent/socket.py", line 115, in create_connection 2024-02-01T12:00:08.435 INFO:tasks.rgw_multisite_tests: sock.connect(sa) 2024-02-01T12:00:08.435 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 590, in connect 2024-02-01T12:00:08.435 INFO:tasks.rgw_multisite_tests: self._internal_connect(address) 2024-02-01T12:00:08.435 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 634, in _internal_connect 2024-02-01T12:00:08.435 INFO:tasks.rgw_multisite_tests: raise _SocketError(err, strerror(err)) 2024-02-01T12:00:08.435 INFO:tasks.rgw_multisite_tests:ConnectionRefusedError: [Errno 111] Connection refused 2024-02-01T12:00:08.435 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-02-01T12:00:08.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vclfcn-71/ 2024-02-01T12:00:08.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vclfcn-71/ 2024-02-01T12:00:08.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-02-01T12:00:08.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vclfcn-71/ 2024-02-01T12:00:08.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-02-01T12:00:08.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-02-01T12:00:08.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T12:00:08.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-02-01T12:00:08.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-02-01T12:00:08.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.436 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.436 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.436 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.437 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 11:02:43 GMT 2024-02-01T12:00:08.437 INFO:tasks.rgw_multisite_tests:/vclfcn-71/ 2024-02-01T12:00:08.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.437 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:6oTFxRxGk1893HCPszeUut1oEso= 2024-02-01T12:00:08.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:02:43 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:6oTFxRxGk1893HCPszeUut1oEso=', 'Content-Length': '0'} 2024-02-01T12:00:08.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.437 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.437 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.437 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.437 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 11:02:43 GMT 2024-02-01T12:00:08.437 INFO:tasks.rgw_multisite_tests:/vclfcn-71/ 2024-02-01T12:00:08.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.438 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:6oTFxRxGk1893HCPszeUut1oEso= 2024-02-01T12:00:08.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:02:43 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:6oTFxRxGk1893HCPszeUut1oEso=', 'Content-Length': '0'} 2024-02-01T12:00:08.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.438 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.438 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.438 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.438 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 11:02:43 GMT 2024-02-01T12:00:08.438 INFO:tasks.rgw_multisite_tests:/vclfcn-71/ 2024-02-01T12:00:08.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.439 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:6oTFxRxGk1893HCPszeUut1oEso= 2024-02-01T12:00:08.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:02:43 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:6oTFxRxGk1893HCPszeUut1oEso=', 'Content-Length': '0'} 2024-02-01T12:00:08.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.439 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.439 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.439 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.439 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 11:02:43 GMT 2024-02-01T12:00:08.439 INFO:tasks.rgw_multisite_tests:/vclfcn-71/ 2024-02-01T12:00:08.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.439 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:6oTFxRxGk1893HCPszeUut1oEso= 2024-02-01T12:00:08.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:02:43 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:6oTFxRxGk1893HCPszeUut1oEso=', 'Content-Length': '0'} 2024-02-01T12:00:08.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.440 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.440 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.440 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.440 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 11:02:43 GMT 2024-02-01T12:00:08.440 INFO:tasks.rgw_multisite_tests:/vclfcn-71/ 2024-02-01T12:00:08.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.440 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:6oTFxRxGk1893HCPszeUut1oEso= 2024-02-01T12:00:08.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:02:43 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:6oTFxRxGk1893HCPszeUut1oEso=', 'Content-Length': '0'} 2024-02-01T12:00:08.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.441 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.441 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.441 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.441 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 11:02:43 GMT 2024-02-01T12:00:08.441 INFO:tasks.rgw_multisite_tests:/vclfcn-71/ 2024-02-01T12:00:08.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.441 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:6oTFxRxGk1893HCPszeUut1oEso= 2024-02-01T12:00:08.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:02:43 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:6oTFxRxGk1893HCPszeUut1oEso=', 'Content-Length': '0'} 2024-02-01T12:00:08.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.442 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.442 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.442 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.442 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 11:02:43 GMT 2024-02-01T12:00:08.442 INFO:tasks.rgw_multisite_tests:/vclfcn-71/ 2024-02-01T12:00:08.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.442 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:6oTFxRxGk1893HCPszeUut1oEso= 2024-02-01T12:00:08.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:02:43 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:6oTFxRxGk1893HCPszeUut1oEso=', 'Content-Length': '0'} 2024-02-01T12:00:08.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.442 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-02-01T12:00:08.442 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.443 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-02-01T12:00:08.443 INFO:tasks.rgw_multisite_tests:ERROR: rgw_multi.tests.test_bucket_reshard_full 2024-02-01T12:00:08.443 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-02-01T12:00:08.443 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-02-01T12:00:08.443 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-02-01T12:00:08.443 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-02-01T12:00:08.443 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/../src/test/rgw/rgw_multi/tests.py", line 1471, in test_bucket_reshard_full 2024-02-01T12:00:08.443 INFO:tasks.rgw_multisite_tests: bucket = zone.create_bucket(gen_bucket_name()) 2024-02-01T12:00:08.443 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/tasks/rgw_multi/zone_rados.py", line 56, in create_bucket 2024-02-01T12:00:08.443 INFO:tasks.rgw_multisite_tests: return self.conn.create_bucket(name) 2024-02-01T12:00:08.443 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/boto/s3/connection.py", line 618, in create_bucket 2024-02-01T12:00:08.443 INFO:tasks.rgw_multisite_tests: response = self.make_request('PUT', bucket_name, headers=headers, 2024-02-01T12:00:08.443 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/boto/s3/connection.py", line 667, in make_request 2024-02-01T12:00:08.444 INFO:tasks.rgw_multisite_tests: return super(S3Connection, self).make_request( 2024-02-01T12:00:08.444 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 1070, in make_request 2024-02-01T12:00:08.444 INFO:tasks.rgw_multisite_tests: return self._mexe(http_request, sender, override_num_retries, 2024-02-01T12:00:08.444 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 1030, in _mexe 2024-02-01T12:00:08.444 INFO:tasks.rgw_multisite_tests: raise ex 2024-02-01T12:00:08.444 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 942, in _mexe 2024-02-01T12:00:08.444 INFO:tasks.rgw_multisite_tests: connection.request(request.method, request.path, 2024-02-01T12:00:08.444 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1256, in request 2024-02-01T12:00:08.444 INFO:tasks.rgw_multisite_tests: self._send_request(method, url, body, headers, encode_chunked) 2024-02-01T12:00:08.444 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1302, in _send_request 2024-02-01T12:00:08.444 INFO:tasks.rgw_multisite_tests: self.endheaders(body, encode_chunked=encode_chunked) 2024-02-01T12:00:08.444 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1251, in endheaders 2024-02-01T12:00:08.444 INFO:tasks.rgw_multisite_tests: self._send_output(message_body, encode_chunked=encode_chunked) 2024-02-01T12:00:08.445 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1011, in _send_output 2024-02-01T12:00:08.445 INFO:tasks.rgw_multisite_tests: self.send(msg) 2024-02-01T12:00:08.445 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 951, in send 2024-02-01T12:00:08.445 INFO:tasks.rgw_multisite_tests: self.connect() 2024-02-01T12:00:08.445 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 922, in connect 2024-02-01T12:00:08.445 INFO:tasks.rgw_multisite_tests: self.sock = self._create_connection( 2024-02-01T12:00:08.445 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/gevent/socket.py", line 115, in create_connection 2024-02-01T12:00:08.445 INFO:tasks.rgw_multisite_tests: sock.connect(sa) 2024-02-01T12:00:08.445 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 590, in connect 2024-02-01T12:00:08.445 INFO:tasks.rgw_multisite_tests: self._internal_connect(address) 2024-02-01T12:00:08.445 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 634, in _internal_connect 2024-02-01T12:00:08.445 INFO:tasks.rgw_multisite_tests: raise _SocketError(err, strerror(err)) 2024-02-01T12:00:08.445 INFO:tasks.rgw_multisite_tests:ConnectionRefusedError: [Errno 111] Connection refused 2024-02-01T12:00:08.446 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-02-01T12:00:08.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vclfcn-72/ 2024-02-01T12:00:08.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vclfcn-72/ 2024-02-01T12:00:08.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-02-01T12:00:08.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vclfcn-72/ 2024-02-01T12:00:08.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-02-01T12:00:08.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-02-01T12:00:08.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T12:00:08.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-02-01T12:00:08.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-02-01T12:00:08.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.446 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.447 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.447 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.447 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 11:03:32 GMT 2024-02-01T12:00:08.447 INFO:tasks.rgw_multisite_tests:/vclfcn-72/ 2024-02-01T12:00:08.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.447 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:iu+A5XLfHaQ4nqroFeZRdiPj6Ck= 2024-02-01T12:00:08.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:03:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iu+A5XLfHaQ4nqroFeZRdiPj6Ck=', 'Content-Length': '0'} 2024-02-01T12:00:08.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.447 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.447 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.447 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.448 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 11:03:32 GMT 2024-02-01T12:00:08.448 INFO:tasks.rgw_multisite_tests:/vclfcn-72/ 2024-02-01T12:00:08.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.448 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:iu+A5XLfHaQ4nqroFeZRdiPj6Ck= 2024-02-01T12:00:08.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:03:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iu+A5XLfHaQ4nqroFeZRdiPj6Ck=', 'Content-Length': '0'} 2024-02-01T12:00:08.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.448 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.448 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.448 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.448 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 11:03:32 GMT 2024-02-01T12:00:08.449 INFO:tasks.rgw_multisite_tests:/vclfcn-72/ 2024-02-01T12:00:08.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.449 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:iu+A5XLfHaQ4nqroFeZRdiPj6Ck= 2024-02-01T12:00:08.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:03:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iu+A5XLfHaQ4nqroFeZRdiPj6Ck=', 'Content-Length': '0'} 2024-02-01T12:00:08.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.449 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.449 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.449 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.449 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 11:03:32 GMT 2024-02-01T12:00:08.449 INFO:tasks.rgw_multisite_tests:/vclfcn-72/ 2024-02-01T12:00:08.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.450 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:iu+A5XLfHaQ4nqroFeZRdiPj6Ck= 2024-02-01T12:00:08.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:03:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iu+A5XLfHaQ4nqroFeZRdiPj6Ck=', 'Content-Length': '0'} 2024-02-01T12:00:08.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.450 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.450 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.450 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.450 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 11:03:32 GMT 2024-02-01T12:00:08.450 INFO:tasks.rgw_multisite_tests:/vclfcn-72/ 2024-02-01T12:00:08.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.450 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:iu+A5XLfHaQ4nqroFeZRdiPj6Ck= 2024-02-01T12:00:08.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:03:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iu+A5XLfHaQ4nqroFeZRdiPj6Ck=', 'Content-Length': '0'} 2024-02-01T12:00:08.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.451 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.451 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.451 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.451 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 11:03:32 GMT 2024-02-01T12:00:08.451 INFO:tasks.rgw_multisite_tests:/vclfcn-72/ 2024-02-01T12:00:08.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.451 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:iu+A5XLfHaQ4nqroFeZRdiPj6Ck= 2024-02-01T12:00:08.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:03:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iu+A5XLfHaQ4nqroFeZRdiPj6Ck=', 'Content-Length': '0'} 2024-02-01T12:00:08.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.452 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.452 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.452 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.452 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 11:03:32 GMT 2024-02-01T12:00:08.452 INFO:tasks.rgw_multisite_tests:/vclfcn-72/ 2024-02-01T12:00:08.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.452 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:iu+A5XLfHaQ4nqroFeZRdiPj6Ck= 2024-02-01T12:00:08.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:03:32 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iu+A5XLfHaQ4nqroFeZRdiPj6Ck=', 'Content-Length': '0'} 2024-02-01T12:00:08.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.453 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-02-01T12:00:08.453 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.453 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-02-01T12:00:08.453 INFO:tasks.rgw_multisite_tests:ERROR: Create several generations of objects, then run bucket sync 2024-02-01T12:00:08.453 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-02-01T12:00:08.454 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-02-01T12:00:08.454 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-02-01T12:00:08.454 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-02-01T12:00:08.454 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/../src/test/rgw/rgw_multi/tests.py", line 1589, in test_bucket_sync_run_basic_incremental 2024-02-01T12:00:08.454 INFO:tasks.rgw_multisite_tests: bucket = primary.create_bucket(gen_bucket_name()) 2024-02-01T12:00:08.454 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/tasks/rgw_multi/zone_rados.py", line 56, in create_bucket 2024-02-01T12:00:08.454 INFO:tasks.rgw_multisite_tests: return self.conn.create_bucket(name) 2024-02-01T12:00:08.455 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/boto/s3/connection.py", line 618, in create_bucket 2024-02-01T12:00:08.455 INFO:tasks.rgw_multisite_tests: response = self.make_request('PUT', bucket_name, headers=headers, 2024-02-01T12:00:08.455 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/boto/s3/connection.py", line 667, in make_request 2024-02-01T12:00:08.455 INFO:tasks.rgw_multisite_tests: return super(S3Connection, self).make_request( 2024-02-01T12:00:08.455 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 1070, in make_request 2024-02-01T12:00:08.455 INFO:tasks.rgw_multisite_tests: return self._mexe(http_request, sender, override_num_retries, 2024-02-01T12:00:08.455 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 1030, in _mexe 2024-02-01T12:00:08.455 INFO:tasks.rgw_multisite_tests: raise ex 2024-02-01T12:00:08.455 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 942, in _mexe 2024-02-01T12:00:08.455 INFO:tasks.rgw_multisite_tests: connection.request(request.method, request.path, 2024-02-01T12:00:08.455 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1256, in request 2024-02-01T12:00:08.455 INFO:tasks.rgw_multisite_tests: self._send_request(method, url, body, headers, encode_chunked) 2024-02-01T12:00:08.456 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1302, in _send_request 2024-02-01T12:00:08.456 INFO:tasks.rgw_multisite_tests: self.endheaders(body, encode_chunked=encode_chunked) 2024-02-01T12:00:08.456 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1251, in endheaders 2024-02-01T12:00:08.456 INFO:tasks.rgw_multisite_tests: self._send_output(message_body, encode_chunked=encode_chunked) 2024-02-01T12:00:08.456 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1011, in _send_output 2024-02-01T12:00:08.456 INFO:tasks.rgw_multisite_tests: self.send(msg) 2024-02-01T12:00:08.456 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 951, in send 2024-02-01T12:00:08.456 INFO:tasks.rgw_multisite_tests: self.connect() 2024-02-01T12:00:08.456 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 922, in connect 2024-02-01T12:00:08.456 INFO:tasks.rgw_multisite_tests: self.sock = self._create_connection( 2024-02-01T12:00:08.456 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/gevent/socket.py", line 115, in create_connection 2024-02-01T12:00:08.456 INFO:tasks.rgw_multisite_tests: sock.connect(sa) 2024-02-01T12:00:08.457 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 590, in connect 2024-02-01T12:00:08.457 INFO:tasks.rgw_multisite_tests: self._internal_connect(address) 2024-02-01T12:00:08.457 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 634, in _internal_connect 2024-02-01T12:00:08.457 INFO:tasks.rgw_multisite_tests: raise _SocketError(err, strerror(err)) 2024-02-01T12:00:08.457 INFO:tasks.rgw_multisite_tests:ConnectionRefusedError: [Errno 111] Connection refused 2024-02-01T12:00:08.457 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-02-01T12:00:08.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vclfcn-73/ 2024-02-01T12:00:08.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vclfcn-73/ 2024-02-01T12:00:08.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-02-01T12:00:08.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vclfcn-73/ 2024-02-01T12:00:08.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-02-01T12:00:08.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-02-01T12:00:08.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T12:00:08.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-02-01T12:00:08.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-02-01T12:00:08.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.458 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.458 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.458 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.458 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 11:10:02 GMT 2024-02-01T12:00:08.458 INFO:tasks.rgw_multisite_tests:/vclfcn-73/ 2024-02-01T12:00:08.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.458 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:Nk0JW4/RC7ZEn+dQH6V9FAc0DDw= 2024-02-01T12:00:08.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:10:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Nk0JW4/RC7ZEn+dQH6V9FAc0DDw=', 'Content-Length': '0'} 2024-02-01T12:00:08.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.459 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.459 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.459 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.459 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 11:10:02 GMT 2024-02-01T12:00:08.459 INFO:tasks.rgw_multisite_tests:/vclfcn-73/ 2024-02-01T12:00:08.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.459 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:Nk0JW4/RC7ZEn+dQH6V9FAc0DDw= 2024-02-01T12:00:08.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:10:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Nk0JW4/RC7ZEn+dQH6V9FAc0DDw=', 'Content-Length': '0'} 2024-02-01T12:00:08.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.460 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.460 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.460 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.460 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 11:10:02 GMT 2024-02-01T12:00:08.460 INFO:tasks.rgw_multisite_tests:/vclfcn-73/ 2024-02-01T12:00:08.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.460 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:Nk0JW4/RC7ZEn+dQH6V9FAc0DDw= 2024-02-01T12:00:08.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:10:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Nk0JW4/RC7ZEn+dQH6V9FAc0DDw=', 'Content-Length': '0'} 2024-02-01T12:00:08.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.461 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.461 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.461 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.461 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 11:10:02 GMT 2024-02-01T12:00:08.461 INFO:tasks.rgw_multisite_tests:/vclfcn-73/ 2024-02-01T12:00:08.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.461 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:Nk0JW4/RC7ZEn+dQH6V9FAc0DDw= 2024-02-01T12:00:08.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:10:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Nk0JW4/RC7ZEn+dQH6V9FAc0DDw=', 'Content-Length': '0'} 2024-02-01T12:00:08.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.461 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.462 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.462 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.462 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 11:10:02 GMT 2024-02-01T12:00:08.462 INFO:tasks.rgw_multisite_tests:/vclfcn-73/ 2024-02-01T12:00:08.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.462 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:Nk0JW4/RC7ZEn+dQH6V9FAc0DDw= 2024-02-01T12:00:08.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:10:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Nk0JW4/RC7ZEn+dQH6V9FAc0DDw=', 'Content-Length': '0'} 2024-02-01T12:00:08.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.462 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.462 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.463 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.463 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 11:10:02 GMT 2024-02-01T12:00:08.463 INFO:tasks.rgw_multisite_tests:/vclfcn-73/ 2024-02-01T12:00:08.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.463 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:Nk0JW4/RC7ZEn+dQH6V9FAc0DDw= 2024-02-01T12:00:08.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:10:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Nk0JW4/RC7ZEn+dQH6V9FAc0DDw=', 'Content-Length': '0'} 2024-02-01T12:00:08.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.463 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.463 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.463 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.464 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 11:10:02 GMT 2024-02-01T12:00:08.464 INFO:tasks.rgw_multisite_tests:/vclfcn-73/ 2024-02-01T12:00:08.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.464 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:Nk0JW4/RC7ZEn+dQH6V9FAc0DDw= 2024-02-01T12:00:08.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:10:02 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Nk0JW4/RC7ZEn+dQH6V9FAc0DDw=', 'Content-Length': '0'} 2024-02-01T12:00:08.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.464 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-02-01T12:00:08.464 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.464 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-02-01T12:00:08.464 INFO:tasks.rgw_multisite_tests:ERROR: Create several generations of objects, trash them, then run bucket sync init 2024-02-01T12:00:08.464 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-02-01T12:00:08.465 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-02-01T12:00:08.465 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-02-01T12:00:08.465 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-02-01T12:00:08.465 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/../src/test/rgw/rgw_multi/tests.py", line 1652, in test_zap_init_bucket_sync_run 2024-02-01T12:00:08.465 INFO:tasks.rgw_multisite_tests: bucket = primary.create_bucket(gen_bucket_name()) 2024-02-01T12:00:08.465 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/tasks/rgw_multi/zone_rados.py", line 56, in create_bucket 2024-02-01T12:00:08.465 INFO:tasks.rgw_multisite_tests: return self.conn.create_bucket(name) 2024-02-01T12:00:08.465 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/boto/s3/connection.py", line 618, in create_bucket 2024-02-01T12:00:08.465 INFO:tasks.rgw_multisite_tests: response = self.make_request('PUT', bucket_name, headers=headers, 2024-02-01T12:00:08.465 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/boto/s3/connection.py", line 667, in make_request 2024-02-01T12:00:08.465 INFO:tasks.rgw_multisite_tests: return super(S3Connection, self).make_request( 2024-02-01T12:00:08.465 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 1070, in make_request 2024-02-01T12:00:08.466 INFO:tasks.rgw_multisite_tests: return self._mexe(http_request, sender, override_num_retries, 2024-02-01T12:00:08.466 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 1030, in _mexe 2024-02-01T12:00:08.466 INFO:tasks.rgw_multisite_tests: raise ex 2024-02-01T12:00:08.466 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 942, in _mexe 2024-02-01T12:00:08.466 INFO:tasks.rgw_multisite_tests: connection.request(request.method, request.path, 2024-02-01T12:00:08.466 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1256, in request 2024-02-01T12:00:08.466 INFO:tasks.rgw_multisite_tests: self._send_request(method, url, body, headers, encode_chunked) 2024-02-01T12:00:08.466 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1302, in _send_request 2024-02-01T12:00:08.466 INFO:tasks.rgw_multisite_tests: self.endheaders(body, encode_chunked=encode_chunked) 2024-02-01T12:00:08.466 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1251, in endheaders 2024-02-01T12:00:08.467 INFO:tasks.rgw_multisite_tests: self._send_output(message_body, encode_chunked=encode_chunked) 2024-02-01T12:00:08.467 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1011, in _send_output 2024-02-01T12:00:08.467 INFO:tasks.rgw_multisite_tests: self.send(msg) 2024-02-01T12:00:08.467 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 951, in send 2024-02-01T12:00:08.467 INFO:tasks.rgw_multisite_tests: self.connect() 2024-02-01T12:00:08.467 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 922, in connect 2024-02-01T12:00:08.467 INFO:tasks.rgw_multisite_tests: self.sock = self._create_connection( 2024-02-01T12:00:08.467 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/gevent/socket.py", line 115, in create_connection 2024-02-01T12:00:08.467 INFO:tasks.rgw_multisite_tests: sock.connect(sa) 2024-02-01T12:00:08.467 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 590, in connect 2024-02-01T12:00:08.467 INFO:tasks.rgw_multisite_tests: self._internal_connect(address) 2024-02-01T12:00:08.467 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 634, in _internal_connect 2024-02-01T12:00:08.467 INFO:tasks.rgw_multisite_tests: raise _SocketError(err, strerror(err)) 2024-02-01T12:00:08.468 INFO:tasks.rgw_multisite_tests:ConnectionRefusedError: [Errno 111] Connection refused 2024-02-01T12:00:08.468 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-02-01T12:00:08.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vclfcn-74/ 2024-02-01T12:00:08.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vclfcn-74/ 2024-02-01T12:00:08.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-02-01T12:00:08.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vclfcn-74/ 2024-02-01T12:00:08.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-02-01T12:00:08.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-02-01T12:00:08.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T12:00:08.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-02-01T12:00:08.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-02-01T12:00:08.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.469 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.469 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.469 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.469 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 11:11:10 GMT 2024-02-01T12:00:08.469 INFO:tasks.rgw_multisite_tests:/vclfcn-74/ 2024-02-01T12:00:08.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.469 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:Q5DbJjQBCIunGCiTMGvLPH32xgw= 2024-02-01T12:00:08.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:11:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Q5DbJjQBCIunGCiTMGvLPH32xgw=', 'Content-Length': '0'} 2024-02-01T12:00:08.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.470 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.470 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.470 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.470 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 11:11:10 GMT 2024-02-01T12:00:08.470 INFO:tasks.rgw_multisite_tests:/vclfcn-74/ 2024-02-01T12:00:08.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.470 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:Q5DbJjQBCIunGCiTMGvLPH32xgw= 2024-02-01T12:00:08.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:11:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Q5DbJjQBCIunGCiTMGvLPH32xgw=', 'Content-Length': '0'} 2024-02-01T12:00:08.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.470 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.471 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.471 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.471 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 11:11:10 GMT 2024-02-01T12:00:08.471 INFO:tasks.rgw_multisite_tests:/vclfcn-74/ 2024-02-01T12:00:08.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.471 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:Q5DbJjQBCIunGCiTMGvLPH32xgw= 2024-02-01T12:00:08.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:11:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Q5DbJjQBCIunGCiTMGvLPH32xgw=', 'Content-Length': '0'} 2024-02-01T12:00:08.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.471 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.471 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.472 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.472 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 11:11:10 GMT 2024-02-01T12:00:08.472 INFO:tasks.rgw_multisite_tests:/vclfcn-74/ 2024-02-01T12:00:08.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.472 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:Q5DbJjQBCIunGCiTMGvLPH32xgw= 2024-02-01T12:00:08.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:11:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Q5DbJjQBCIunGCiTMGvLPH32xgw=', 'Content-Length': '0'} 2024-02-01T12:00:08.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.472 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.472 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.472 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.472 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 11:11:10 GMT 2024-02-01T12:00:08.473 INFO:tasks.rgw_multisite_tests:/vclfcn-74/ 2024-02-01T12:00:08.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.473 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:Q5DbJjQBCIunGCiTMGvLPH32xgw= 2024-02-01T12:00:08.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:11:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Q5DbJjQBCIunGCiTMGvLPH32xgw=', 'Content-Length': '0'} 2024-02-01T12:00:08.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.473 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.473 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.473 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.473 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 11:11:10 GMT 2024-02-01T12:00:08.473 INFO:tasks.rgw_multisite_tests:/vclfcn-74/ 2024-02-01T12:00:08.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.474 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:Q5DbJjQBCIunGCiTMGvLPH32xgw= 2024-02-01T12:00:08.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:11:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Q5DbJjQBCIunGCiTMGvLPH32xgw=', 'Content-Length': '0'} 2024-02-01T12:00:08.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.474 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.474 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.474 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.474 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 11:11:10 GMT 2024-02-01T12:00:08.474 INFO:tasks.rgw_multisite_tests:/vclfcn-74/ 2024-02-01T12:00:08.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.474 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:Q5DbJjQBCIunGCiTMGvLPH32xgw= 2024-02-01T12:00:08.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:11:10 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Q5DbJjQBCIunGCiTMGvLPH32xgw=', 'Content-Length': '0'} 2024-02-01T12:00:08.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.475 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-02-01T12:00:08.475 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.475 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-02-01T12:00:08.475 INFO:tasks.rgw_multisite_tests:ERROR: rgw_multi.tests.test_role_sync 2024-02-01T12:00:08.475 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-02-01T12:00:08.475 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-02-01T12:00:08.475 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-02-01T12:00:08.475 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-02-01T12:00:08.476 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/../src/test/rgw/rgw_multi/tests.py", line 1699, in test_role_sync 2024-02-01T12:00:08.476 INFO:tasks.rgw_multisite_tests: roles, zone_role = create_role_per_zone(zonegroup_conns) 2024-02-01T12:00:08.476 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/../src/test/rgw/rgw_multi/tests.py", line 513, in create_role_per_zone 2024-02-01T12:00:08.476 INFO:tasks.rgw_multisite_tests: role = zone.create_role("", role_name, policy_document, "") 2024-02-01T12:00:08.476 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/tasks/rgw_multi/zone_rados.py", line 133, in create_role 2024-02-01T12:00:08.476 INFO:tasks.rgw_multisite_tests: return self.iam_conn.create_role(rolename, policy_document, path) 2024-02-01T12:00:08.476 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/boto/iam/connection.py", line 1155, in create_role 2024-02-01T12:00:08.476 INFO:tasks.rgw_multisite_tests: return self.get_response('CreateRole', params) 2024-02-01T12:00:08.476 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/boto/iam/connection.py", line 85, in get_response 2024-02-01T12:00:08.476 INFO:tasks.rgw_multisite_tests: response = self.make_request(action, params, path, verb) 2024-02-01T12:00:08.476 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 1116, in make_request 2024-02-01T12:00:08.477 INFO:tasks.rgw_multisite_tests: return self._mexe(http_request) 2024-02-01T12:00:08.477 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 1030, in _mexe 2024-02-01T12:00:08.477 INFO:tasks.rgw_multisite_tests: raise ex 2024-02-01T12:00:08.477 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 942, in _mexe 2024-02-01T12:00:08.477 INFO:tasks.rgw_multisite_tests: connection.request(request.method, request.path, 2024-02-01T12:00:08.477 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1256, in request 2024-02-01T12:00:08.477 INFO:tasks.rgw_multisite_tests: self._send_request(method, url, body, headers, encode_chunked) 2024-02-01T12:00:08.477 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1302, in _send_request 2024-02-01T12:00:08.477 INFO:tasks.rgw_multisite_tests: self.endheaders(body, encode_chunked=encode_chunked) 2024-02-01T12:00:08.477 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1251, in endheaders 2024-02-01T12:00:08.477 INFO:tasks.rgw_multisite_tests: self._send_output(message_body, encode_chunked=encode_chunked) 2024-02-01T12:00:08.477 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1011, in _send_output 2024-02-01T12:00:08.477 INFO:tasks.rgw_multisite_tests: self.send(msg) 2024-02-01T12:00:08.478 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 951, in send 2024-02-01T12:00:08.478 INFO:tasks.rgw_multisite_tests: self.connect() 2024-02-01T12:00:08.478 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 922, in connect 2024-02-01T12:00:08.478 INFO:tasks.rgw_multisite_tests: self.sock = self._create_connection( 2024-02-01T12:00:08.478 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/gevent/socket.py", line 115, in create_connection 2024-02-01T12:00:08.478 INFO:tasks.rgw_multisite_tests: sock.connect(sa) 2024-02-01T12:00:08.478 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 590, in connect 2024-02-01T12:00:08.478 INFO:tasks.rgw_multisite_tests: self._internal_connect(address) 2024-02-01T12:00:08.478 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 634, in _internal_connect 2024-02-01T12:00:08.478 INFO:tasks.rgw_multisite_tests: raise _SocketError(err, strerror(err)) 2024-02-01T12:00:08.478 INFO:tasks.rgw_multisite_tests:ConnectionRefusedError: [Errno 111] Connection refused 2024-02-01T12:00:08.478 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-02-01T12:00:08.478 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create role zone=test-zone1 name=roles-vclfcn-1 2024-02-01T12:00:08.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: POST 2024-02-01T12:00:08.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: / 2024-02-01T12:00:08.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-02-01T12:00:08.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-02-01T12:00:08.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi148.front.sepia.ceph.com 2024-02-01T12:00:08.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-02-01T12:00:08.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {'RoleName': 'roles-vclfcn-1', 'AssumeRolePolicyDocument': '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}', 'Path': '', 'Action': 'CreateRole', 'Version': '2010-05-08'} 2024-02-01T12:00:08.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: CanonicalRequest: 2024-02-01T12:00:08.479 INFO:tasks.rgw_multisite_tests:POST 2024-02-01T12:00:08.479 INFO:tasks.rgw_multisite_tests:/ 2024-02-01T12:00:08.479 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.480 INFO:tasks.rgw_multisite_tests:host:smithi148.front.sepia.ceph.com:8000 2024-02-01T12:00:08.480 INFO:tasks.rgw_multisite_tests:x-amz-date:20240201T111215Z 2024-02-01T12:00:08.480 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.480 INFO:tasks.rgw_multisite_tests:host;x-amz-date 2024-02-01T12:00:08.480 INFO:tasks.rgw_multisite_tests:8344f44cf7b239e92f6078048bc20e4d5df2bc02ecccd3ac62b2071b0d761648 2024-02-01T12:00:08.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.480 INFO:tasks.rgw_multisite_tests:AWS4-HMAC-SHA256 2024-02-01T12:00:08.480 INFO:tasks.rgw_multisite_tests:20240201T111215Z 2024-02-01T12:00:08.480 INFO:tasks.rgw_multisite_tests:20240201/front/smithi148/aws4_request 2024-02-01T12:00:08.480 INFO:tasks.rgw_multisite_tests:2a426675aadbf734174a6957771e048270b98bf28514fed19577f78df44cc9e7 2024-02-01T12:00:08.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.480 INFO:tasks.rgw_multisite_tests:5b648e3e39d84e6f96fd62f755e3b5f6a91460668140eb8756f9572e06326002 2024-02-01T12:00:08.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240201T111215Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=1IMYR3NXFV2H92HDXL01/20240201/front/smithi148/aws4_request,SignedHeaders=host;x-amz-date,Signature=5b648e3e39d84e6f96fd62f755e3b5f6a91460668140eb8756f9572e06326002', 'Host': 'smithi148.front.sepia.ceph.com:8000'} 2024-02-01T12:00:08.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: CanonicalRequest: 2024-02-01T12:00:08.481 INFO:tasks.rgw_multisite_tests:POST 2024-02-01T12:00:08.481 INFO:tasks.rgw_multisite_tests:/ 2024-02-01T12:00:08.481 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.481 INFO:tasks.rgw_multisite_tests:host:smithi148.front.sepia.ceph.com:8000 2024-02-01T12:00:08.481 INFO:tasks.rgw_multisite_tests:x-amz-date:20240201T111215Z 2024-02-01T12:00:08.481 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.481 INFO:tasks.rgw_multisite_tests:host;x-amz-date 2024-02-01T12:00:08.481 INFO:tasks.rgw_multisite_tests:8344f44cf7b239e92f6078048bc20e4d5df2bc02ecccd3ac62b2071b0d761648 2024-02-01T12:00:08.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.482 INFO:tasks.rgw_multisite_tests:AWS4-HMAC-SHA256 2024-02-01T12:00:08.482 INFO:tasks.rgw_multisite_tests:20240201T111215Z 2024-02-01T12:00:08.482 INFO:tasks.rgw_multisite_tests:20240201/front/smithi148/aws4_request 2024-02-01T12:00:08.482 INFO:tasks.rgw_multisite_tests:2a426675aadbf734174a6957771e048270b98bf28514fed19577f78df44cc9e7 2024-02-01T12:00:08.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.482 INFO:tasks.rgw_multisite_tests:5b648e3e39d84e6f96fd62f755e3b5f6a91460668140eb8756f9572e06326002 2024-02-01T12:00:08.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240201T111215Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=1IMYR3NXFV2H92HDXL01/20240201/front/smithi148/aws4_request,SignedHeaders=host;x-amz-date,Signature=5b648e3e39d84e6f96fd62f755e3b5f6a91460668140eb8756f9572e06326002', 'Host': 'smithi148.front.sepia.ceph.com:8000'} 2024-02-01T12:00:08.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: CanonicalRequest: 2024-02-01T12:00:08.482 INFO:tasks.rgw_multisite_tests:POST 2024-02-01T12:00:08.482 INFO:tasks.rgw_multisite_tests:/ 2024-02-01T12:00:08.483 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.483 INFO:tasks.rgw_multisite_tests:host:smithi148.front.sepia.ceph.com:8000 2024-02-01T12:00:08.483 INFO:tasks.rgw_multisite_tests:x-amz-date:20240201T111216Z 2024-02-01T12:00:08.483 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.483 INFO:tasks.rgw_multisite_tests:host;x-amz-date 2024-02-01T12:00:08.483 INFO:tasks.rgw_multisite_tests:8344f44cf7b239e92f6078048bc20e4d5df2bc02ecccd3ac62b2071b0d761648 2024-02-01T12:00:08.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.483 INFO:tasks.rgw_multisite_tests:AWS4-HMAC-SHA256 2024-02-01T12:00:08.483 INFO:tasks.rgw_multisite_tests:20240201T111216Z 2024-02-01T12:00:08.483 INFO:tasks.rgw_multisite_tests:20240201/front/smithi148/aws4_request 2024-02-01T12:00:08.483 INFO:tasks.rgw_multisite_tests:b561e90ed06656214a7e5a3f7d1afbd1cad84b7a495bbec78091e4b6579b7f77 2024-02-01T12:00:08.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.483 INFO:tasks.rgw_multisite_tests:ff01a21c966c94915255726be1842abb62cb1f4f6f80d25057282893b88887aa 2024-02-01T12:00:08.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240201T111216Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=1IMYR3NXFV2H92HDXL01/20240201/front/smithi148/aws4_request,SignedHeaders=host;x-amz-date,Signature=ff01a21c966c94915255726be1842abb62cb1f4f6f80d25057282893b88887aa', 'Host': 'smithi148.front.sepia.ceph.com:8000'} 2024-02-01T12:00:08.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: CanonicalRequest: 2024-02-01T12:00:08.484 INFO:tasks.rgw_multisite_tests:POST 2024-02-01T12:00:08.484 INFO:tasks.rgw_multisite_tests:/ 2024-02-01T12:00:08.484 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.484 INFO:tasks.rgw_multisite_tests:host:smithi148.front.sepia.ceph.com:8000 2024-02-01T12:00:08.484 INFO:tasks.rgw_multisite_tests:x-amz-date:20240201T111218Z 2024-02-01T12:00:08.484 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.484 INFO:tasks.rgw_multisite_tests:host;x-amz-date 2024-02-01T12:00:08.484 INFO:tasks.rgw_multisite_tests:8344f44cf7b239e92f6078048bc20e4d5df2bc02ecccd3ac62b2071b0d761648 2024-02-01T12:00:08.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.485 INFO:tasks.rgw_multisite_tests:AWS4-HMAC-SHA256 2024-02-01T12:00:08.485 INFO:tasks.rgw_multisite_tests:20240201T111218Z 2024-02-01T12:00:08.485 INFO:tasks.rgw_multisite_tests:20240201/front/smithi148/aws4_request 2024-02-01T12:00:08.485 INFO:tasks.rgw_multisite_tests:3eda4d4b740830b6c636f11c9bc42b9360ef91d78a9b3565fb2129fa62ac9fca 2024-02-01T12:00:08.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.485 INFO:tasks.rgw_multisite_tests:fba1ed5876bfd2a1058a0906a867b4deeae277c2bd2d726d6cd231bd9d59e5a6 2024-02-01T12:00:08.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240201T111218Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=1IMYR3NXFV2H92HDXL01/20240201/front/smithi148/aws4_request,SignedHeaders=host;x-amz-date,Signature=fba1ed5876bfd2a1058a0906a867b4deeae277c2bd2d726d6cd231bd9d59e5a6', 'Host': 'smithi148.front.sepia.ceph.com:8000'} 2024-02-01T12:00:08.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: CanonicalRequest: 2024-02-01T12:00:08.485 INFO:tasks.rgw_multisite_tests:POST 2024-02-01T12:00:08.485 INFO:tasks.rgw_multisite_tests:/ 2024-02-01T12:00:08.486 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.486 INFO:tasks.rgw_multisite_tests:host:smithi148.front.sepia.ceph.com:8000 2024-02-01T12:00:08.486 INFO:tasks.rgw_multisite_tests:x-amz-date:20240201T111225Z 2024-02-01T12:00:08.486 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.486 INFO:tasks.rgw_multisite_tests:host;x-amz-date 2024-02-01T12:00:08.486 INFO:tasks.rgw_multisite_tests:8344f44cf7b239e92f6078048bc20e4d5df2bc02ecccd3ac62b2071b0d761648 2024-02-01T12:00:08.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.486 INFO:tasks.rgw_multisite_tests:AWS4-HMAC-SHA256 2024-02-01T12:00:08.486 INFO:tasks.rgw_multisite_tests:20240201T111225Z 2024-02-01T12:00:08.486 INFO:tasks.rgw_multisite_tests:20240201/front/smithi148/aws4_request 2024-02-01T12:00:08.486 INFO:tasks.rgw_multisite_tests:33ea75450219a29b273a93965fe0fbaa518153ddb3c933593bf18014585b869a 2024-02-01T12:00:08.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.486 INFO:tasks.rgw_multisite_tests:86b552d44e71a7c81f94b9cb3e2a81af341c96ab1bec84ffbc316e0fc05f9ef1 2024-02-01T12:00:08.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240201T111225Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=1IMYR3NXFV2H92HDXL01/20240201/front/smithi148/aws4_request,SignedHeaders=host;x-amz-date,Signature=86b552d44e71a7c81f94b9cb3e2a81af341c96ab1bec84ffbc316e0fc05f9ef1', 'Host': 'smithi148.front.sepia.ceph.com:8000'} 2024-02-01T12:00:08.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: CanonicalRequest: 2024-02-01T12:00:08.487 INFO:tasks.rgw_multisite_tests:POST 2024-02-01T12:00:08.487 INFO:tasks.rgw_multisite_tests:/ 2024-02-01T12:00:08.487 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.487 INFO:tasks.rgw_multisite_tests:host:smithi148.front.sepia.ceph.com:8000 2024-02-01T12:00:08.487 INFO:tasks.rgw_multisite_tests:x-amz-date:20240201T111232Z 2024-02-01T12:00:08.487 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.487 INFO:tasks.rgw_multisite_tests:host;x-amz-date 2024-02-01T12:00:08.487 INFO:tasks.rgw_multisite_tests:8344f44cf7b239e92f6078048bc20e4d5df2bc02ecccd3ac62b2071b0d761648 2024-02-01T12:00:08.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.488 INFO:tasks.rgw_multisite_tests:AWS4-HMAC-SHA256 2024-02-01T12:00:08.488 INFO:tasks.rgw_multisite_tests:20240201T111232Z 2024-02-01T12:00:08.488 INFO:tasks.rgw_multisite_tests:20240201/front/smithi148/aws4_request 2024-02-01T12:00:08.488 INFO:tasks.rgw_multisite_tests:424feb93c3a4cce025a9c1d7936e9e8c9f94cd8f67a906be5ade47d6329fb4fd 2024-02-01T12:00:08.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.488 INFO:tasks.rgw_multisite_tests:dac3013101a42415a25ad679c892053a9192580cbf078de1e055bf7af092241e 2024-02-01T12:00:08.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240201T111232Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=1IMYR3NXFV2H92HDXL01/20240201/front/smithi148/aws4_request,SignedHeaders=host;x-amz-date,Signature=dac3013101a42415a25ad679c892053a9192580cbf078de1e055bf7af092241e', 'Host': 'smithi148.front.sepia.ceph.com:8000'} 2024-02-01T12:00:08.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: CanonicalRequest: 2024-02-01T12:00:08.488 INFO:tasks.rgw_multisite_tests:POST 2024-02-01T12:00:08.489 INFO:tasks.rgw_multisite_tests:/ 2024-02-01T12:00:08.489 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.489 INFO:tasks.rgw_multisite_tests:host:smithi148.front.sepia.ceph.com:8000 2024-02-01T12:00:08.489 INFO:tasks.rgw_multisite_tests:x-amz-date:20240201T111237Z 2024-02-01T12:00:08.489 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.489 INFO:tasks.rgw_multisite_tests:host;x-amz-date 2024-02-01T12:00:08.489 INFO:tasks.rgw_multisite_tests:8344f44cf7b239e92f6078048bc20e4d5df2bc02ecccd3ac62b2071b0d761648 2024-02-01T12:00:08.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.489 INFO:tasks.rgw_multisite_tests:AWS4-HMAC-SHA256 2024-02-01T12:00:08.489 INFO:tasks.rgw_multisite_tests:20240201T111237Z 2024-02-01T12:00:08.489 INFO:tasks.rgw_multisite_tests:20240201/front/smithi148/aws4_request 2024-02-01T12:00:08.489 INFO:tasks.rgw_multisite_tests:0030fee67c701869c4f96ff412d3727783b54141f9354972575c095cbb8b49f6 2024-02-01T12:00:08.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.489 INFO:tasks.rgw_multisite_tests:fc0facb4cf5e2a571a4723f31838fe2e7475dbf6d10179812059ccea944f1092 2024-02-01T12:00:08.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240201T111237Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=1IMYR3NXFV2H92HDXL01/20240201/front/smithi148/aws4_request,SignedHeaders=host;x-amz-date,Signature=fc0facb4cf5e2a571a4723f31838fe2e7475dbf6d10179812059ccea944f1092', 'Host': 'smithi148.front.sepia.ceph.com:8000'} 2024-02-01T12:00:08.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.490 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-02-01T12:00:08.490 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.490 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-02-01T12:00:08.490 INFO:tasks.rgw_multisite_tests:ERROR: rgw_multi.tests.test_role_delete_sync 2024-02-01T12:00:08.490 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-02-01T12:00:08.490 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-02-01T12:00:08.490 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-02-01T12:00:08.491 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-02-01T12:00:08.491 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/../src/test/rgw/rgw_multi/tests.py", line 1715, in test_role_delete_sync 2024-02-01T12:00:08.491 INFO:tasks.rgw_multisite_tests: zonegroup_conns.master_zone.create_role("", role_name, None, "") 2024-02-01T12:00:08.491 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/tasks/rgw_multi/zone_rados.py", line 133, in create_role 2024-02-01T12:00:08.491 INFO:tasks.rgw_multisite_tests: return self.iam_conn.create_role(rolename, policy_document, path) 2024-02-01T12:00:08.491 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/boto/iam/connection.py", line 1155, in create_role 2024-02-01T12:00:08.491 INFO:tasks.rgw_multisite_tests: return self.get_response('CreateRole', params) 2024-02-01T12:00:08.491 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/boto/iam/connection.py", line 85, in get_response 2024-02-01T12:00:08.491 INFO:tasks.rgw_multisite_tests: response = self.make_request(action, params, path, verb) 2024-02-01T12:00:08.491 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 1116, in make_request 2024-02-01T12:00:08.491 INFO:tasks.rgw_multisite_tests: return self._mexe(http_request) 2024-02-01T12:00:08.491 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 1030, in _mexe 2024-02-01T12:00:08.491 INFO:tasks.rgw_multisite_tests: raise ex 2024-02-01T12:00:08.492 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 942, in _mexe 2024-02-01T12:00:08.492 INFO:tasks.rgw_multisite_tests: connection.request(request.method, request.path, 2024-02-01T12:00:08.492 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1256, in request 2024-02-01T12:00:08.492 INFO:tasks.rgw_multisite_tests: self._send_request(method, url, body, headers, encode_chunked) 2024-02-01T12:00:08.492 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1302, in _send_request 2024-02-01T12:00:08.492 INFO:tasks.rgw_multisite_tests: self.endheaders(body, encode_chunked=encode_chunked) 2024-02-01T12:00:08.492 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1251, in endheaders 2024-02-01T12:00:08.492 INFO:tasks.rgw_multisite_tests: self._send_output(message_body, encode_chunked=encode_chunked) 2024-02-01T12:00:08.492 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1011, in _send_output 2024-02-01T12:00:08.492 INFO:tasks.rgw_multisite_tests: self.send(msg) 2024-02-01T12:00:08.492 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 951, in send 2024-02-01T12:00:08.492 INFO:tasks.rgw_multisite_tests: self.connect() 2024-02-01T12:00:08.492 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 922, in connect 2024-02-01T12:00:08.493 INFO:tasks.rgw_multisite_tests: self.sock = self._create_connection( 2024-02-01T12:00:08.493 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/gevent/socket.py", line 115, in create_connection 2024-02-01T12:00:08.493 INFO:tasks.rgw_multisite_tests: sock.connect(sa) 2024-02-01T12:00:08.493 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 590, in connect 2024-02-01T12:00:08.493 INFO:tasks.rgw_multisite_tests: self._internal_connect(address) 2024-02-01T12:00:08.493 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 634, in _internal_connect 2024-02-01T12:00:08.493 INFO:tasks.rgw_multisite_tests: raise _SocketError(err, strerror(err)) 2024-02-01T12:00:08.493 INFO:tasks.rgw_multisite_tests:ConnectionRefusedError: [Errno 111] Connection refused 2024-02-01T12:00:08.493 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-02-01T12:00:08.493 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create role zone=test-zone1 name=roles-vclfcn-2 2024-02-01T12:00:08.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: POST 2024-02-01T12:00:08.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: / 2024-02-01T12:00:08.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-02-01T12:00:08.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-02-01T12:00:08.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi148.front.sepia.ceph.com 2024-02-01T12:00:08.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-02-01T12:00:08.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {'RoleName': 'roles-vclfcn-2', 'AssumeRolePolicyDocument': '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}', 'Path': '', 'Action': 'CreateRole', 'Version': '2010-05-08'} 2024-02-01T12:00:08.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: CanonicalRequest: 2024-02-01T12:00:08.494 INFO:tasks.rgw_multisite_tests:POST 2024-02-01T12:00:08.494 INFO:tasks.rgw_multisite_tests:/ 2024-02-01T12:00:08.494 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.494 INFO:tasks.rgw_multisite_tests:host:smithi148.front.sepia.ceph.com:8000 2024-02-01T12:00:08.494 INFO:tasks.rgw_multisite_tests:x-amz-date:20240201T111325Z 2024-02-01T12:00:08.495 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.495 INFO:tasks.rgw_multisite_tests:host;x-amz-date 2024-02-01T12:00:08.495 INFO:tasks.rgw_multisite_tests:4efa4a360bd87db21c71e7ea0c35b095f1f1059d6445a9281c4f4753dce8b410 2024-02-01T12:00:08.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.495 INFO:tasks.rgw_multisite_tests:AWS4-HMAC-SHA256 2024-02-01T12:00:08.495 INFO:tasks.rgw_multisite_tests:20240201T111325Z 2024-02-01T12:00:08.495 INFO:tasks.rgw_multisite_tests:20240201/front/smithi148/aws4_request 2024-02-01T12:00:08.495 INFO:tasks.rgw_multisite_tests:3993b16311684068362030f255ed473bd51a2dd96e21bf18bb566975e30774a0 2024-02-01T12:00:08.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.495 INFO:tasks.rgw_multisite_tests:d3bfd428f34f075af3fd2e3a84486390da3b05bdbec6bef488ccf410e5fed81c 2024-02-01T12:00:08.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240201T111325Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=1IMYR3NXFV2H92HDXL01/20240201/front/smithi148/aws4_request,SignedHeaders=host;x-amz-date,Signature=d3bfd428f34f075af3fd2e3a84486390da3b05bdbec6bef488ccf410e5fed81c', 'Host': 'smithi148.front.sepia.ceph.com:8000'} 2024-02-01T12:00:08.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: CanonicalRequest: 2024-02-01T12:00:08.496 INFO:tasks.rgw_multisite_tests:POST 2024-02-01T12:00:08.496 INFO:tasks.rgw_multisite_tests:/ 2024-02-01T12:00:08.496 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.496 INFO:tasks.rgw_multisite_tests:host:smithi148.front.sepia.ceph.com:8000 2024-02-01T12:00:08.496 INFO:tasks.rgw_multisite_tests:x-amz-date:20240201T111325Z 2024-02-01T12:00:08.496 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.496 INFO:tasks.rgw_multisite_tests:host;x-amz-date 2024-02-01T12:00:08.496 INFO:tasks.rgw_multisite_tests:4efa4a360bd87db21c71e7ea0c35b095f1f1059d6445a9281c4f4753dce8b410 2024-02-01T12:00:08.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.496 INFO:tasks.rgw_multisite_tests:AWS4-HMAC-SHA256 2024-02-01T12:00:08.497 INFO:tasks.rgw_multisite_tests:20240201T111325Z 2024-02-01T12:00:08.497 INFO:tasks.rgw_multisite_tests:20240201/front/smithi148/aws4_request 2024-02-01T12:00:08.497 INFO:tasks.rgw_multisite_tests:3993b16311684068362030f255ed473bd51a2dd96e21bf18bb566975e30774a0 2024-02-01T12:00:08.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.497 INFO:tasks.rgw_multisite_tests:d3bfd428f34f075af3fd2e3a84486390da3b05bdbec6bef488ccf410e5fed81c 2024-02-01T12:00:08.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240201T111325Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=1IMYR3NXFV2H92HDXL01/20240201/front/smithi148/aws4_request,SignedHeaders=host;x-amz-date,Signature=d3bfd428f34f075af3fd2e3a84486390da3b05bdbec6bef488ccf410e5fed81c', 'Host': 'smithi148.front.sepia.ceph.com:8000'} 2024-02-01T12:00:08.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: CanonicalRequest: 2024-02-01T12:00:08.497 INFO:tasks.rgw_multisite_tests:POST 2024-02-01T12:00:08.497 INFO:tasks.rgw_multisite_tests:/ 2024-02-01T12:00:08.497 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.498 INFO:tasks.rgw_multisite_tests:host:smithi148.front.sepia.ceph.com:8000 2024-02-01T12:00:08.498 INFO:tasks.rgw_multisite_tests:x-amz-date:20240201T111326Z 2024-02-01T12:00:08.498 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.498 INFO:tasks.rgw_multisite_tests:host;x-amz-date 2024-02-01T12:00:08.498 INFO:tasks.rgw_multisite_tests:4efa4a360bd87db21c71e7ea0c35b095f1f1059d6445a9281c4f4753dce8b410 2024-02-01T12:00:08.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.498 INFO:tasks.rgw_multisite_tests:AWS4-HMAC-SHA256 2024-02-01T12:00:08.498 INFO:tasks.rgw_multisite_tests:20240201T111326Z 2024-02-01T12:00:08.498 INFO:tasks.rgw_multisite_tests:20240201/front/smithi148/aws4_request 2024-02-01T12:00:08.498 INFO:tasks.rgw_multisite_tests:af68e20ca666426330ce53cf6c6f72ec4e4cab6ae705ecf18babd51b03aa70d2 2024-02-01T12:00:08.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.498 INFO:tasks.rgw_multisite_tests:593b777b1fd42f3f9c866f13215cbccab75e6138b22309204417fb2e4492710c 2024-02-01T12:00:08.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240201T111326Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=1IMYR3NXFV2H92HDXL01/20240201/front/smithi148/aws4_request,SignedHeaders=host;x-amz-date,Signature=593b777b1fd42f3f9c866f13215cbccab75e6138b22309204417fb2e4492710c', 'Host': 'smithi148.front.sepia.ceph.com:8000'} 2024-02-01T12:00:08.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: CanonicalRequest: 2024-02-01T12:00:08.499 INFO:tasks.rgw_multisite_tests:POST 2024-02-01T12:00:08.499 INFO:tasks.rgw_multisite_tests:/ 2024-02-01T12:00:08.499 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.499 INFO:tasks.rgw_multisite_tests:host:smithi148.front.sepia.ceph.com:8000 2024-02-01T12:00:08.499 INFO:tasks.rgw_multisite_tests:x-amz-date:20240201T111328Z 2024-02-01T12:00:08.499 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.499 INFO:tasks.rgw_multisite_tests:host;x-amz-date 2024-02-01T12:00:08.499 INFO:tasks.rgw_multisite_tests:4efa4a360bd87db21c71e7ea0c35b095f1f1059d6445a9281c4f4753dce8b410 2024-02-01T12:00:08.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.500 INFO:tasks.rgw_multisite_tests:AWS4-HMAC-SHA256 2024-02-01T12:00:08.500 INFO:tasks.rgw_multisite_tests:20240201T111328Z 2024-02-01T12:00:08.500 INFO:tasks.rgw_multisite_tests:20240201/front/smithi148/aws4_request 2024-02-01T12:00:08.500 INFO:tasks.rgw_multisite_tests:61c147c0534dc08320c2e6cbc51d550df462a463fb543d1d6e3aecdae3742f05 2024-02-01T12:00:08.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.500 INFO:tasks.rgw_multisite_tests:df1fe91fdf6acdad2cac783c030c9cc2b7b9a57a8ae25b70a8b9df118c1d6f70 2024-02-01T12:00:08.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240201T111328Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=1IMYR3NXFV2H92HDXL01/20240201/front/smithi148/aws4_request,SignedHeaders=host;x-amz-date,Signature=df1fe91fdf6acdad2cac783c030c9cc2b7b9a57a8ae25b70a8b9df118c1d6f70', 'Host': 'smithi148.front.sepia.ceph.com:8000'} 2024-02-01T12:00:08.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: CanonicalRequest: 2024-02-01T12:00:08.500 INFO:tasks.rgw_multisite_tests:POST 2024-02-01T12:00:08.501 INFO:tasks.rgw_multisite_tests:/ 2024-02-01T12:00:08.501 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.501 INFO:tasks.rgw_multisite_tests:host:smithi148.front.sepia.ceph.com:8000 2024-02-01T12:00:08.501 INFO:tasks.rgw_multisite_tests:x-amz-date:20240201T111330Z 2024-02-01T12:00:08.501 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.501 INFO:tasks.rgw_multisite_tests:host;x-amz-date 2024-02-01T12:00:08.501 INFO:tasks.rgw_multisite_tests:4efa4a360bd87db21c71e7ea0c35b095f1f1059d6445a9281c4f4753dce8b410 2024-02-01T12:00:08.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.501 INFO:tasks.rgw_multisite_tests:AWS4-HMAC-SHA256 2024-02-01T12:00:08.501 INFO:tasks.rgw_multisite_tests:20240201T111330Z 2024-02-01T12:00:08.501 INFO:tasks.rgw_multisite_tests:20240201/front/smithi148/aws4_request 2024-02-01T12:00:08.501 INFO:tasks.rgw_multisite_tests:c5f9f151405c1f796c0b09e95932aeabd7a7b6cbef82e394d49cca73759b5113 2024-02-01T12:00:08.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.502 INFO:tasks.rgw_multisite_tests:bbb493ac48537d91387c677da94a31bcb1df0b86647a67686736156bff59908d 2024-02-01T12:00:08.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240201T111330Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=1IMYR3NXFV2H92HDXL01/20240201/front/smithi148/aws4_request,SignedHeaders=host;x-amz-date,Signature=bbb493ac48537d91387c677da94a31bcb1df0b86647a67686736156bff59908d', 'Host': 'smithi148.front.sepia.ceph.com:8000'} 2024-02-01T12:00:08.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: CanonicalRequest: 2024-02-01T12:00:08.502 INFO:tasks.rgw_multisite_tests:POST 2024-02-01T12:00:08.502 INFO:tasks.rgw_multisite_tests:/ 2024-02-01T12:00:08.502 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.502 INFO:tasks.rgw_multisite_tests:host:smithi148.front.sepia.ceph.com:8000 2024-02-01T12:00:08.502 INFO:tasks.rgw_multisite_tests:x-amz-date:20240201T111331Z 2024-02-01T12:00:08.502 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.503 INFO:tasks.rgw_multisite_tests:host;x-amz-date 2024-02-01T12:00:08.503 INFO:tasks.rgw_multisite_tests:4efa4a360bd87db21c71e7ea0c35b095f1f1059d6445a9281c4f4753dce8b410 2024-02-01T12:00:08.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.503 INFO:tasks.rgw_multisite_tests:AWS4-HMAC-SHA256 2024-02-01T12:00:08.503 INFO:tasks.rgw_multisite_tests:20240201T111331Z 2024-02-01T12:00:08.503 INFO:tasks.rgw_multisite_tests:20240201/front/smithi148/aws4_request 2024-02-01T12:00:08.503 INFO:tasks.rgw_multisite_tests:57f3e3d11b0131548af4d75f7431f79bb7ba19be38c2cd99aafb8ec33479d044 2024-02-01T12:00:08.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.503 INFO:tasks.rgw_multisite_tests:d97ebbca0c0a758b5a4dd217fe3ab7185bfff3de495bf0123f7ed91c997ba5e1 2024-02-01T12:00:08.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240201T111331Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=1IMYR3NXFV2H92HDXL01/20240201/front/smithi148/aws4_request,SignedHeaders=host;x-amz-date,Signature=d97ebbca0c0a758b5a4dd217fe3ab7185bfff3de495bf0123f7ed91c997ba5e1', 'Host': 'smithi148.front.sepia.ceph.com:8000'} 2024-02-01T12:00:08.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: CanonicalRequest: 2024-02-01T12:00:08.504 INFO:tasks.rgw_multisite_tests:POST 2024-02-01T12:00:08.504 INFO:tasks.rgw_multisite_tests:/ 2024-02-01T12:00:08.504 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.504 INFO:tasks.rgw_multisite_tests:host:smithi148.front.sepia.ceph.com:8000 2024-02-01T12:00:08.504 INFO:tasks.rgw_multisite_tests:x-amz-date:20240201T111352Z 2024-02-01T12:00:08.504 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.504 INFO:tasks.rgw_multisite_tests:host;x-amz-date 2024-02-01T12:00:08.504 INFO:tasks.rgw_multisite_tests:4efa4a360bd87db21c71e7ea0c35b095f1f1059d6445a9281c4f4753dce8b410 2024-02-01T12:00:08.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.504 INFO:tasks.rgw_multisite_tests:AWS4-HMAC-SHA256 2024-02-01T12:00:08.505 INFO:tasks.rgw_multisite_tests:20240201T111352Z 2024-02-01T12:00:08.505 INFO:tasks.rgw_multisite_tests:20240201/front/smithi148/aws4_request 2024-02-01T12:00:08.505 INFO:tasks.rgw_multisite_tests:39e61bb7f406c0343cc117f7381b12c374d274ed3f90df170d31f04adf78a119 2024-02-01T12:00:08.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.505 INFO:tasks.rgw_multisite_tests:41efdaf5f718a5779ab3be1e67aa2008e749c1c2915c9c90973114c64ad55b20 2024-02-01T12:00:08.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240201T111352Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=1IMYR3NXFV2H92HDXL01/20240201/front/smithi148/aws4_request,SignedHeaders=host;x-amz-date,Signature=41efdaf5f718a5779ab3be1e67aa2008e749c1c2915c9c90973114c64ad55b20', 'Host': 'smithi148.front.sepia.ceph.com:8000'} 2024-02-01T12:00:08.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.505 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-02-01T12:00:08.505 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.505 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-02-01T12:00:08.505 INFO:tasks.rgw_multisite_tests:ERROR: rgw_multi.tests.test_bucket_full_sync_after_data_sync_init 2024-02-01T12:00:08.506 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-02-01T12:00:08.506 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-02-01T12:00:08.506 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-02-01T12:00:08.506 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-02-01T12:00:08.506 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/../src/test/rgw/rgw_multi/tests.py", line 1741, in test_bucket_full_sync_after_data_sync_init 2024-02-01T12:00:08.506 INFO:tasks.rgw_multisite_tests: bucket = primary.create_bucket(gen_bucket_name()) 2024-02-01T12:00:08.506 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/tasks/rgw_multi/zone_rados.py", line 56, in create_bucket 2024-02-01T12:00:08.506 INFO:tasks.rgw_multisite_tests: return self.conn.create_bucket(name) 2024-02-01T12:00:08.506 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/boto/s3/connection.py", line 618, in create_bucket 2024-02-01T12:00:08.506 INFO:tasks.rgw_multisite_tests: response = self.make_request('PUT', bucket_name, headers=headers, 2024-02-01T12:00:08.506 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/boto/s3/connection.py", line 667, in make_request 2024-02-01T12:00:08.506 INFO:tasks.rgw_multisite_tests: return super(S3Connection, self).make_request( 2024-02-01T12:00:08.507 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 1070, in make_request 2024-02-01T12:00:08.507 INFO:tasks.rgw_multisite_tests: return self._mexe(http_request, sender, override_num_retries, 2024-02-01T12:00:08.507 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 1030, in _mexe 2024-02-01T12:00:08.507 INFO:tasks.rgw_multisite_tests: raise ex 2024-02-01T12:00:08.507 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 942, in _mexe 2024-02-01T12:00:08.507 INFO:tasks.rgw_multisite_tests: connection.request(request.method, request.path, 2024-02-01T12:00:08.507 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1256, in request 2024-02-01T12:00:08.507 INFO:tasks.rgw_multisite_tests: self._send_request(method, url, body, headers, encode_chunked) 2024-02-01T12:00:08.507 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1302, in _send_request 2024-02-01T12:00:08.507 INFO:tasks.rgw_multisite_tests: self.endheaders(body, encode_chunked=encode_chunked) 2024-02-01T12:00:08.507 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1251, in endheaders 2024-02-01T12:00:08.507 INFO:tasks.rgw_multisite_tests: self._send_output(message_body, encode_chunked=encode_chunked) 2024-02-01T12:00:08.508 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1011, in _send_output 2024-02-01T12:00:08.508 INFO:tasks.rgw_multisite_tests: self.send(msg) 2024-02-01T12:00:08.508 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 951, in send 2024-02-01T12:00:08.508 INFO:tasks.rgw_multisite_tests: self.connect() 2024-02-01T12:00:08.508 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 922, in connect 2024-02-01T12:00:08.508 INFO:tasks.rgw_multisite_tests: self.sock = self._create_connection( 2024-02-01T12:00:08.508 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/gevent/socket.py", line 115, in create_connection 2024-02-01T12:00:08.508 INFO:tasks.rgw_multisite_tests: sock.connect(sa) 2024-02-01T12:00:08.508 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 590, in connect 2024-02-01T12:00:08.508 INFO:tasks.rgw_multisite_tests: self._internal_connect(address) 2024-02-01T12:00:08.508 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 634, in _internal_connect 2024-02-01T12:00:08.508 INFO:tasks.rgw_multisite_tests: raise _SocketError(err, strerror(err)) 2024-02-01T12:00:08.508 INFO:tasks.rgw_multisite_tests:ConnectionRefusedError: [Errno 111] Connection refused 2024-02-01T12:00:08.509 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-02-01T12:00:08.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vclfcn-75/ 2024-02-01T12:00:08.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vclfcn-75/ 2024-02-01T12:00:08.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-02-01T12:00:08.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vclfcn-75/ 2024-02-01T12:00:08.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-02-01T12:00:08.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-02-01T12:00:08.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T12:00:08.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-02-01T12:00:08.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-02-01T12:00:08.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.510 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.510 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.510 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.510 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 11:14:51 GMT 2024-02-01T12:00:08.510 INFO:tasks.rgw_multisite_tests:/vclfcn-75/ 2024-02-01T12:00:08.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.510 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:iPQhg6SV30aUjzFfDinkG8+tvQM= 2024-02-01T12:00:08.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:14:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iPQhg6SV30aUjzFfDinkG8+tvQM=', 'Content-Length': '0'} 2024-02-01T12:00:08.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.510 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.511 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.511 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.511 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 11:14:51 GMT 2024-02-01T12:00:08.511 INFO:tasks.rgw_multisite_tests:/vclfcn-75/ 2024-02-01T12:00:08.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.511 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:iPQhg6SV30aUjzFfDinkG8+tvQM= 2024-02-01T12:00:08.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:14:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iPQhg6SV30aUjzFfDinkG8+tvQM=', 'Content-Length': '0'} 2024-02-01T12:00:08.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.511 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.511 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.511 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.512 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 11:14:51 GMT 2024-02-01T12:00:08.512 INFO:tasks.rgw_multisite_tests:/vclfcn-75/ 2024-02-01T12:00:08.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.512 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:iPQhg6SV30aUjzFfDinkG8+tvQM= 2024-02-01T12:00:08.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:14:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iPQhg6SV30aUjzFfDinkG8+tvQM=', 'Content-Length': '0'} 2024-02-01T12:00:08.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.512 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.512 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.512 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.512 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 11:14:51 GMT 2024-02-01T12:00:08.513 INFO:tasks.rgw_multisite_tests:/vclfcn-75/ 2024-02-01T12:00:08.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.513 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:iPQhg6SV30aUjzFfDinkG8+tvQM= 2024-02-01T12:00:08.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:14:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iPQhg6SV30aUjzFfDinkG8+tvQM=', 'Content-Length': '0'} 2024-02-01T12:00:08.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.513 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.513 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.513 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.513 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 11:14:51 GMT 2024-02-01T12:00:08.513 INFO:tasks.rgw_multisite_tests:/vclfcn-75/ 2024-02-01T12:00:08.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.514 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:iPQhg6SV30aUjzFfDinkG8+tvQM= 2024-02-01T12:00:08.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:14:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iPQhg6SV30aUjzFfDinkG8+tvQM=', 'Content-Length': '0'} 2024-02-01T12:00:08.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.514 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.514 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.514 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.514 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 11:14:51 GMT 2024-02-01T12:00:08.514 INFO:tasks.rgw_multisite_tests:/vclfcn-75/ 2024-02-01T12:00:08.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.514 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:iPQhg6SV30aUjzFfDinkG8+tvQM= 2024-02-01T12:00:08.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:14:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iPQhg6SV30aUjzFfDinkG8+tvQM=', 'Content-Length': '0'} 2024-02-01T12:00:08.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.515 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.515 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.515 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.515 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 11:14:51 GMT 2024-02-01T12:00:08.515 INFO:tasks.rgw_multisite_tests:/vclfcn-75/ 2024-02-01T12:00:08.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.515 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:iPQhg6SV30aUjzFfDinkG8+tvQM= 2024-02-01T12:00:08.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:14:51 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:iPQhg6SV30aUjzFfDinkG8+tvQM=', 'Content-Length': '0'} 2024-02-01T12:00:08.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.516 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-02-01T12:00:08.516 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.516 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-02-01T12:00:08.516 INFO:tasks.rgw_multisite_tests:ERROR: rgw_multi.tests.test_resharded_bucket_full_sync_after_data_sync_init 2024-02-01T12:00:08.516 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-02-01T12:00:08.516 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-02-01T12:00:08.516 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-02-01T12:00:08.516 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-02-01T12:00:08.517 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/../src/test/rgw/rgw_multi/tests.py", line 1773, in test_resharded_bucket_full_sync_after_data_sync_init 2024-02-01T12:00:08.517 INFO:tasks.rgw_multisite_tests: bucket = primary.create_bucket(gen_bucket_name()) 2024-02-01T12:00:08.517 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/tasks/rgw_multi/zone_rados.py", line 56, in create_bucket 2024-02-01T12:00:08.517 INFO:tasks.rgw_multisite_tests: return self.conn.create_bucket(name) 2024-02-01T12:00:08.517 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/boto/s3/connection.py", line 618, in create_bucket 2024-02-01T12:00:08.517 INFO:tasks.rgw_multisite_tests: response = self.make_request('PUT', bucket_name, headers=headers, 2024-02-01T12:00:08.517 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/boto/s3/connection.py", line 667, in make_request 2024-02-01T12:00:08.517 INFO:tasks.rgw_multisite_tests: return super(S3Connection, self).make_request( 2024-02-01T12:00:08.517 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 1070, in make_request 2024-02-01T12:00:08.517 INFO:tasks.rgw_multisite_tests: return self._mexe(http_request, sender, override_num_retries, 2024-02-01T12:00:08.518 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 1030, in _mexe 2024-02-01T12:00:08.518 INFO:tasks.rgw_multisite_tests: raise ex 2024-02-01T12:00:08.518 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 942, in _mexe 2024-02-01T12:00:08.518 INFO:tasks.rgw_multisite_tests: connection.request(request.method, request.path, 2024-02-01T12:00:08.518 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1256, in request 2024-02-01T12:00:08.518 INFO:tasks.rgw_multisite_tests: self._send_request(method, url, body, headers, encode_chunked) 2024-02-01T12:00:08.518 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1302, in _send_request 2024-02-01T12:00:08.518 INFO:tasks.rgw_multisite_tests: self.endheaders(body, encode_chunked=encode_chunked) 2024-02-01T12:00:08.518 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1251, in endheaders 2024-02-01T12:00:08.518 INFO:tasks.rgw_multisite_tests: self._send_output(message_body, encode_chunked=encode_chunked) 2024-02-01T12:00:08.518 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1011, in _send_output 2024-02-01T12:00:08.518 INFO:tasks.rgw_multisite_tests: self.send(msg) 2024-02-01T12:00:08.518 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 951, in send 2024-02-01T12:00:08.518 INFO:tasks.rgw_multisite_tests: self.connect() 2024-02-01T12:00:08.519 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 922, in connect 2024-02-01T12:00:08.519 INFO:tasks.rgw_multisite_tests: self.sock = self._create_connection( 2024-02-01T12:00:08.519 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/gevent/socket.py", line 115, in create_connection 2024-02-01T12:00:08.519 INFO:tasks.rgw_multisite_tests: sock.connect(sa) 2024-02-01T12:00:08.519 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 590, in connect 2024-02-01T12:00:08.519 INFO:tasks.rgw_multisite_tests: self._internal_connect(address) 2024-02-01T12:00:08.519 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 634, in _internal_connect 2024-02-01T12:00:08.519 INFO:tasks.rgw_multisite_tests: raise _SocketError(err, strerror(err)) 2024-02-01T12:00:08.519 INFO:tasks.rgw_multisite_tests:ConnectionRefusedError: [Errno 111] Connection refused 2024-02-01T12:00:08.519 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-02-01T12:00:08.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vclfcn-76/ 2024-02-01T12:00:08.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vclfcn-76/ 2024-02-01T12:00:08.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-02-01T12:00:08.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vclfcn-76/ 2024-02-01T12:00:08.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-02-01T12:00:08.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-02-01T12:00:08.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T12:00:08.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-02-01T12:00:08.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-02-01T12:00:08.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.520 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.520 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.520 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.520 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 11:15:48 GMT 2024-02-01T12:00:08.521 INFO:tasks.rgw_multisite_tests:/vclfcn-76/ 2024-02-01T12:00:08.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.521 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:3THkG+IhweOFI9+k19IXyy8xxbE= 2024-02-01T12:00:08.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:15:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:3THkG+IhweOFI9+k19IXyy8xxbE=', 'Content-Length': '0'} 2024-02-01T12:00:08.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.521 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.521 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.521 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.521 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 11:15:48 GMT 2024-02-01T12:00:08.521 INFO:tasks.rgw_multisite_tests:/vclfcn-76/ 2024-02-01T12:00:08.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.522 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:3THkG+IhweOFI9+k19IXyy8xxbE= 2024-02-01T12:00:08.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:15:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:3THkG+IhweOFI9+k19IXyy8xxbE=', 'Content-Length': '0'} 2024-02-01T12:00:08.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.522 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.522 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.522 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.522 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 11:15:48 GMT 2024-02-01T12:00:08.522 INFO:tasks.rgw_multisite_tests:/vclfcn-76/ 2024-02-01T12:00:08.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.522 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:3THkG+IhweOFI9+k19IXyy8xxbE= 2024-02-01T12:00:08.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:15:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:3THkG+IhweOFI9+k19IXyy8xxbE=', 'Content-Length': '0'} 2024-02-01T12:00:08.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.523 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.523 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.523 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.523 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 11:15:48 GMT 2024-02-01T12:00:08.523 INFO:tasks.rgw_multisite_tests:/vclfcn-76/ 2024-02-01T12:00:08.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.523 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:3THkG+IhweOFI9+k19IXyy8xxbE= 2024-02-01T12:00:08.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:15:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:3THkG+IhweOFI9+k19IXyy8xxbE=', 'Content-Length': '0'} 2024-02-01T12:00:08.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.524 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.524 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.524 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.524 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 11:15:48 GMT 2024-02-01T12:00:08.524 INFO:tasks.rgw_multisite_tests:/vclfcn-76/ 2024-02-01T12:00:08.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.524 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:3THkG+IhweOFI9+k19IXyy8xxbE= 2024-02-01T12:00:08.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:15:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:3THkG+IhweOFI9+k19IXyy8xxbE=', 'Content-Length': '0'} 2024-02-01T12:00:08.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.525 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.525 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.525 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.525 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 11:15:48 GMT 2024-02-01T12:00:08.525 INFO:tasks.rgw_multisite_tests:/vclfcn-76/ 2024-02-01T12:00:08.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.525 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:3THkG+IhweOFI9+k19IXyy8xxbE= 2024-02-01T12:00:08.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:15:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:3THkG+IhweOFI9+k19IXyy8xxbE=', 'Content-Length': '0'} 2024-02-01T12:00:08.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.526 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.526 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.526 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.526 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 11:15:48 GMT 2024-02-01T12:00:08.526 INFO:tasks.rgw_multisite_tests:/vclfcn-76/ 2024-02-01T12:00:08.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.526 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:3THkG+IhweOFI9+k19IXyy8xxbE= 2024-02-01T12:00:08.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:15:48 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:3THkG+IhweOFI9+k19IXyy8xxbE=', 'Content-Length': '0'} 2024-02-01T12:00:08.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.526 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-02-01T12:00:08.526 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.526 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-02-01T12:00:08.527 INFO:tasks.rgw_multisite_tests:ERROR: rgw_multi.tests.test_bucket_incremental_sync_after_data_sync_init 2024-02-01T12:00:08.527 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-02-01T12:00:08.527 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-02-01T12:00:08.527 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-02-01T12:00:08.527 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-02-01T12:00:08.527 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/../src/test/rgw/rgw_multi/tests.py", line 1812, in test_bucket_incremental_sync_after_data_sync_init 2024-02-01T12:00:08.527 INFO:tasks.rgw_multisite_tests: bucket = primary.create_bucket(gen_bucket_name()) 2024-02-01T12:00:08.527 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/tasks/rgw_multi/zone_rados.py", line 56, in create_bucket 2024-02-01T12:00:08.527 INFO:tasks.rgw_multisite_tests: return self.conn.create_bucket(name) 2024-02-01T12:00:08.527 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/boto/s3/connection.py", line 618, in create_bucket 2024-02-01T12:00:08.527 INFO:tasks.rgw_multisite_tests: response = self.make_request('PUT', bucket_name, headers=headers, 2024-02-01T12:00:08.527 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/boto/s3/connection.py", line 667, in make_request 2024-02-01T12:00:08.527 INFO:tasks.rgw_multisite_tests: return super(S3Connection, self).make_request( 2024-02-01T12:00:08.528 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 1070, in make_request 2024-02-01T12:00:08.528 INFO:tasks.rgw_multisite_tests: return self._mexe(http_request, sender, override_num_retries, 2024-02-01T12:00:08.528 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 1030, in _mexe 2024-02-01T12:00:08.528 INFO:tasks.rgw_multisite_tests: raise ex 2024-02-01T12:00:08.528 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 942, in _mexe 2024-02-01T12:00:08.528 INFO:tasks.rgw_multisite_tests: connection.request(request.method, request.path, 2024-02-01T12:00:08.528 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1256, in request 2024-02-01T12:00:08.528 INFO:tasks.rgw_multisite_tests: self._send_request(method, url, body, headers, encode_chunked) 2024-02-01T12:00:08.528 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1302, in _send_request 2024-02-01T12:00:08.528 INFO:tasks.rgw_multisite_tests: self.endheaders(body, encode_chunked=encode_chunked) 2024-02-01T12:00:08.528 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1251, in endheaders 2024-02-01T12:00:08.528 INFO:tasks.rgw_multisite_tests: self._send_output(message_body, encode_chunked=encode_chunked) 2024-02-01T12:00:08.528 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1011, in _send_output 2024-02-01T12:00:08.529 INFO:tasks.rgw_multisite_tests: self.send(msg) 2024-02-01T12:00:08.529 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 951, in send 2024-02-01T12:00:08.529 INFO:tasks.rgw_multisite_tests: self.connect() 2024-02-01T12:00:08.529 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 922, in connect 2024-02-01T12:00:08.529 INFO:tasks.rgw_multisite_tests: self.sock = self._create_connection( 2024-02-01T12:00:08.529 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/gevent/socket.py", line 115, in create_connection 2024-02-01T12:00:08.529 INFO:tasks.rgw_multisite_tests: sock.connect(sa) 2024-02-01T12:00:08.529 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 590, in connect 2024-02-01T12:00:08.531 INFO:tasks.rgw_multisite_tests: self._internal_connect(address) 2024-02-01T12:00:08.531 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 634, in _internal_connect 2024-02-01T12:00:08.531 INFO:tasks.rgw_multisite_tests: raise _SocketError(err, strerror(err)) 2024-02-01T12:00:08.531 INFO:tasks.rgw_multisite_tests:ConnectionRefusedError: [Errno 111] Connection refused 2024-02-01T12:00:08.531 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-02-01T12:00:08.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vclfcn-77/ 2024-02-01T12:00:08.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vclfcn-77/ 2024-02-01T12:00:08.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-02-01T12:00:08.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vclfcn-77/ 2024-02-01T12:00:08.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-02-01T12:00:08.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-02-01T12:00:08.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T12:00:08.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-02-01T12:00:08.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-02-01T12:00:08.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.532 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.532 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.533 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.533 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 11:16:38 GMT 2024-02-01T12:00:08.533 INFO:tasks.rgw_multisite_tests:/vclfcn-77/ 2024-02-01T12:00:08.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.533 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:dYaBPDFTdsOPyzBhPQMARzX8lu4= 2024-02-01T12:00:08.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:16:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dYaBPDFTdsOPyzBhPQMARzX8lu4=', 'Content-Length': '0'} 2024-02-01T12:00:08.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.533 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.533 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.533 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.533 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 11:16:38 GMT 2024-02-01T12:00:08.534 INFO:tasks.rgw_multisite_tests:/vclfcn-77/ 2024-02-01T12:00:08.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.534 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:dYaBPDFTdsOPyzBhPQMARzX8lu4= 2024-02-01T12:00:08.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:16:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dYaBPDFTdsOPyzBhPQMARzX8lu4=', 'Content-Length': '0'} 2024-02-01T12:00:08.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.534 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.534 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.534 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.534 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 11:16:38 GMT 2024-02-01T12:00:08.534 INFO:tasks.rgw_multisite_tests:/vclfcn-77/ 2024-02-01T12:00:08.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.535 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:dYaBPDFTdsOPyzBhPQMARzX8lu4= 2024-02-01T12:00:08.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:16:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dYaBPDFTdsOPyzBhPQMARzX8lu4=', 'Content-Length': '0'} 2024-02-01T12:00:08.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.535 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.535 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.535 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.535 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 11:16:38 GMT 2024-02-01T12:00:08.535 INFO:tasks.rgw_multisite_tests:/vclfcn-77/ 2024-02-01T12:00:08.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.535 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:dYaBPDFTdsOPyzBhPQMARzX8lu4= 2024-02-01T12:00:08.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:16:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dYaBPDFTdsOPyzBhPQMARzX8lu4=', 'Content-Length': '0'} 2024-02-01T12:00:08.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.536 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.536 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.536 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.536 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 11:16:38 GMT 2024-02-01T12:00:08.536 INFO:tasks.rgw_multisite_tests:/vclfcn-77/ 2024-02-01T12:00:08.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.536 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:dYaBPDFTdsOPyzBhPQMARzX8lu4= 2024-02-01T12:00:08.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:16:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dYaBPDFTdsOPyzBhPQMARzX8lu4=', 'Content-Length': '0'} 2024-02-01T12:00:08.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.537 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.537 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.537 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.537 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 11:16:38 GMT 2024-02-01T12:00:08.537 INFO:tasks.rgw_multisite_tests:/vclfcn-77/ 2024-02-01T12:00:08.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.537 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:dYaBPDFTdsOPyzBhPQMARzX8lu4= 2024-02-01T12:00:08.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:16:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dYaBPDFTdsOPyzBhPQMARzX8lu4=', 'Content-Length': '0'} 2024-02-01T12:00:08.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.538 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.538 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.538 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.538 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 11:16:38 GMT 2024-02-01T12:00:08.538 INFO:tasks.rgw_multisite_tests:/vclfcn-77/ 2024-02-01T12:00:08.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.538 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:dYaBPDFTdsOPyzBhPQMARzX8lu4= 2024-02-01T12:00:08.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:16:38 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:dYaBPDFTdsOPyzBhPQMARzX8lu4=', 'Content-Length': '0'} 2024-02-01T12:00:08.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.538 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-02-01T12:00:08.539 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.539 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-02-01T12:00:08.539 INFO:tasks.rgw_multisite_tests:ERROR: rgw_multi.tests.test_resharded_bucket_incremental_sync_latest_after_data_sync_init 2024-02-01T12:00:08.539 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-02-01T12:00:08.539 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-02-01T12:00:08.539 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-02-01T12:00:08.539 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-02-01T12:00:08.539 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/../src/test/rgw/rgw_multi/tests.py", line 1850, in test_resharded_bucket_incremental_sync_latest_after_data_sync_init 2024-02-01T12:00:08.539 INFO:tasks.rgw_multisite_tests: bucket = primary.create_bucket(gen_bucket_name()) 2024-02-01T12:00:08.539 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/tasks/rgw_multi/zone_rados.py", line 56, in create_bucket 2024-02-01T12:00:08.539 INFO:tasks.rgw_multisite_tests: return self.conn.create_bucket(name) 2024-02-01T12:00:08.540 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/boto/s3/connection.py", line 618, in create_bucket 2024-02-01T12:00:08.540 INFO:tasks.rgw_multisite_tests: response = self.make_request('PUT', bucket_name, headers=headers, 2024-02-01T12:00:08.540 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/boto/s3/connection.py", line 667, in make_request 2024-02-01T12:00:08.540 INFO:tasks.rgw_multisite_tests: return super(S3Connection, self).make_request( 2024-02-01T12:00:08.540 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 1070, in make_request 2024-02-01T12:00:08.540 INFO:tasks.rgw_multisite_tests: return self._mexe(http_request, sender, override_num_retries, 2024-02-01T12:00:08.540 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 1030, in _mexe 2024-02-01T12:00:08.540 INFO:tasks.rgw_multisite_tests: raise ex 2024-02-01T12:00:08.540 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 942, in _mexe 2024-02-01T12:00:08.540 INFO:tasks.rgw_multisite_tests: connection.request(request.method, request.path, 2024-02-01T12:00:08.540 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1256, in request 2024-02-01T12:00:08.540 INFO:tasks.rgw_multisite_tests: self._send_request(method, url, body, headers, encode_chunked) 2024-02-01T12:00:08.541 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1302, in _send_request 2024-02-01T12:00:08.541 INFO:tasks.rgw_multisite_tests: self.endheaders(body, encode_chunked=encode_chunked) 2024-02-01T12:00:08.541 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1251, in endheaders 2024-02-01T12:00:08.541 INFO:tasks.rgw_multisite_tests: self._send_output(message_body, encode_chunked=encode_chunked) 2024-02-01T12:00:08.541 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1011, in _send_output 2024-02-01T12:00:08.541 INFO:tasks.rgw_multisite_tests: self.send(msg) 2024-02-01T12:00:08.541 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 951, in send 2024-02-01T12:00:08.541 INFO:tasks.rgw_multisite_tests: self.connect() 2024-02-01T12:00:08.541 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 922, in connect 2024-02-01T12:00:08.541 INFO:tasks.rgw_multisite_tests: self.sock = self._create_connection( 2024-02-01T12:00:08.541 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/gevent/socket.py", line 115, in create_connection 2024-02-01T12:00:08.541 INFO:tasks.rgw_multisite_tests: sock.connect(sa) 2024-02-01T12:00:08.541 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 590, in connect 2024-02-01T12:00:08.542 INFO:tasks.rgw_multisite_tests: self._internal_connect(address) 2024-02-01T12:00:08.542 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 634, in _internal_connect 2024-02-01T12:00:08.542 INFO:tasks.rgw_multisite_tests: raise _SocketError(err, strerror(err)) 2024-02-01T12:00:08.542 INFO:tasks.rgw_multisite_tests:ConnectionRefusedError: [Errno 111] Connection refused 2024-02-01T12:00:08.542 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-02-01T12:00:08.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vclfcn-78/ 2024-02-01T12:00:08.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vclfcn-78/ 2024-02-01T12:00:08.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-02-01T12:00:08.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vclfcn-78/ 2024-02-01T12:00:08.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-02-01T12:00:08.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-02-01T12:00:08.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T12:00:08.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-02-01T12:00:08.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-02-01T12:00:08.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.543 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.543 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.543 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.543 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 11:17:37 GMT 2024-02-01T12:00:08.543 INFO:tasks.rgw_multisite_tests:/vclfcn-78/ 2024-02-01T12:00:08.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.543 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:civvKzBwcFUkJ7sJqPCCSryUAww= 2024-02-01T12:00:08.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:17:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:civvKzBwcFUkJ7sJqPCCSryUAww=', 'Content-Length': '0'} 2024-02-01T12:00:08.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.544 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.544 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.544 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.544 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 11:17:37 GMT 2024-02-01T12:00:08.544 INFO:tasks.rgw_multisite_tests:/vclfcn-78/ 2024-02-01T12:00:08.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.544 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:civvKzBwcFUkJ7sJqPCCSryUAww= 2024-02-01T12:00:08.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:17:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:civvKzBwcFUkJ7sJqPCCSryUAww=', 'Content-Length': '0'} 2024-02-01T12:00:08.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.545 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.545 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.545 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.545 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 11:17:37 GMT 2024-02-01T12:00:08.545 INFO:tasks.rgw_multisite_tests:/vclfcn-78/ 2024-02-01T12:00:08.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.545 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:civvKzBwcFUkJ7sJqPCCSryUAww= 2024-02-01T12:00:08.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:17:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:civvKzBwcFUkJ7sJqPCCSryUAww=', 'Content-Length': '0'} 2024-02-01T12:00:08.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.545 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.545 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.546 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.546 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 11:17:37 GMT 2024-02-01T12:00:08.546 INFO:tasks.rgw_multisite_tests:/vclfcn-78/ 2024-02-01T12:00:08.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.546 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:civvKzBwcFUkJ7sJqPCCSryUAww= 2024-02-01T12:00:08.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:17:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:civvKzBwcFUkJ7sJqPCCSryUAww=', 'Content-Length': '0'} 2024-02-01T12:00:08.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.546 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.546 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.546 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.547 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 11:17:37 GMT 2024-02-01T12:00:08.547 INFO:tasks.rgw_multisite_tests:/vclfcn-78/ 2024-02-01T12:00:08.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.547 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:civvKzBwcFUkJ7sJqPCCSryUAww= 2024-02-01T12:00:08.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:17:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:civvKzBwcFUkJ7sJqPCCSryUAww=', 'Content-Length': '0'} 2024-02-01T12:00:08.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.547 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.547 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.547 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.547 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 11:17:37 GMT 2024-02-01T12:00:08.547 INFO:tasks.rgw_multisite_tests:/vclfcn-78/ 2024-02-01T12:00:08.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.548 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:civvKzBwcFUkJ7sJqPCCSryUAww= 2024-02-01T12:00:08.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:17:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:civvKzBwcFUkJ7sJqPCCSryUAww=', 'Content-Length': '0'} 2024-02-01T12:00:08.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.548 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.548 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.548 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.548 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 11:17:37 GMT 2024-02-01T12:00:08.548 INFO:tasks.rgw_multisite_tests:/vclfcn-78/ 2024-02-01T12:00:08.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.549 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:civvKzBwcFUkJ7sJqPCCSryUAww= 2024-02-01T12:00:08.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:17:37 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:civvKzBwcFUkJ7sJqPCCSryUAww=', 'Content-Length': '0'} 2024-02-01T12:00:08.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.549 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-02-01T12:00:08.549 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.549 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-02-01T12:00:08.549 INFO:tasks.rgw_multisite_tests:ERROR: rgw_multi.tests.test_resharded_bucket_incremental_sync_oldest_after_data_sync_init 2024-02-01T12:00:08.549 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-02-01T12:00:08.549 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-02-01T12:00:08.549 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-02-01T12:00:08.549 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-02-01T12:00:08.550 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/../src/test/rgw/rgw_multi/tests.py", line 1898, in test_resharded_bucket_incremental_sync_oldest_after_data_sync_init 2024-02-01T12:00:08.550 INFO:tasks.rgw_multisite_tests: bucket = primary.create_bucket(gen_bucket_name()) 2024-02-01T12:00:08.550 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/tasks/rgw_multi/zone_rados.py", line 56, in create_bucket 2024-02-01T12:00:08.550 INFO:tasks.rgw_multisite_tests: return self.conn.create_bucket(name) 2024-02-01T12:00:08.550 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/boto/s3/connection.py", line 618, in create_bucket 2024-02-01T12:00:08.550 INFO:tasks.rgw_multisite_tests: response = self.make_request('PUT', bucket_name, headers=headers, 2024-02-01T12:00:08.550 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/boto/s3/connection.py", line 667, in make_request 2024-02-01T12:00:08.550 INFO:tasks.rgw_multisite_tests: return super(S3Connection, self).make_request( 2024-02-01T12:00:08.550 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 1070, in make_request 2024-02-01T12:00:08.550 INFO:tasks.rgw_multisite_tests: return self._mexe(http_request, sender, override_num_retries, 2024-02-01T12:00:08.550 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 1030, in _mexe 2024-02-01T12:00:08.550 INFO:tasks.rgw_multisite_tests: raise ex 2024-02-01T12:00:08.550 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/boto/connection.py", line 942, in _mexe 2024-02-01T12:00:08.551 INFO:tasks.rgw_multisite_tests: connection.request(request.method, request.path, 2024-02-01T12:00:08.551 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1256, in request 2024-02-01T12:00:08.551 INFO:tasks.rgw_multisite_tests: self._send_request(method, url, body, headers, encode_chunked) 2024-02-01T12:00:08.551 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1302, in _send_request 2024-02-01T12:00:08.551 INFO:tasks.rgw_multisite_tests: self.endheaders(body, encode_chunked=encode_chunked) 2024-02-01T12:00:08.551 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1251, in endheaders 2024-02-01T12:00:08.551 INFO:tasks.rgw_multisite_tests: self._send_output(message_body, encode_chunked=encode_chunked) 2024-02-01T12:00:08.551 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 1011, in _send_output 2024-02-01T12:00:08.551 INFO:tasks.rgw_multisite_tests: self.send(msg) 2024-02-01T12:00:08.551 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 951, in send 2024-02-01T12:00:08.551 INFO:tasks.rgw_multisite_tests: self.connect() 2024-02-01T12:00:08.551 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.8/http/client.py", line 922, in connect 2024-02-01T12:00:08.551 INFO:tasks.rgw_multisite_tests: self.sock = self._create_connection( 2024-02-01T12:00:08.552 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/gevent/socket.py", line 115, in create_connection 2024-02-01T12:00:08.552 INFO:tasks.rgw_multisite_tests: sock.connect(sa) 2024-02-01T12:00:08.552 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 590, in connect 2024-02-01T12:00:08.552 INFO:tasks.rgw_multisite_tests: self._internal_connect(address) 2024-02-01T12:00:08.552 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 634, in _internal_connect 2024-02-01T12:00:08.552 INFO:tasks.rgw_multisite_tests: raise _SocketError(err, strerror(err)) 2024-02-01T12:00:08.552 INFO:tasks.rgw_multisite_tests:ConnectionRefusedError: [Errno 111] Connection refused 2024-02-01T12:00:08.552 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-02-01T12:00:08.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vclfcn-79/ 2024-02-01T12:00:08.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vclfcn-79/ 2024-02-01T12:00:08.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-02-01T12:00:08.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vclfcn-79/ 2024-02-01T12:00:08.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-02-01T12:00:08.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-02-01T12:00:08.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T12:00:08.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-02-01T12:00:08.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-02-01T12:00:08.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.553 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.553 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.553 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.553 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 11:18:46 GMT 2024-02-01T12:00:08.553 INFO:tasks.rgw_multisite_tests:/vclfcn-79/ 2024-02-01T12:00:08.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.554 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:mWUqgNYFSRyiwBwFGSsJimsjaVM= 2024-02-01T12:00:08.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:18:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mWUqgNYFSRyiwBwFGSsJimsjaVM=', 'Content-Length': '0'} 2024-02-01T12:00:08.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.554 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.554 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.554 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.554 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 11:18:46 GMT 2024-02-01T12:00:08.554 INFO:tasks.rgw_multisite_tests:/vclfcn-79/ 2024-02-01T12:00:08.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.554 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:mWUqgNYFSRyiwBwFGSsJimsjaVM= 2024-02-01T12:00:08.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:18:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mWUqgNYFSRyiwBwFGSsJimsjaVM=', 'Content-Length': '0'} 2024-02-01T12:00:08.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.555 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.555 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.555 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.555 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 11:18:46 GMT 2024-02-01T12:00:08.555 INFO:tasks.rgw_multisite_tests:/vclfcn-79/ 2024-02-01T12:00:08.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.555 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:mWUqgNYFSRyiwBwFGSsJimsjaVM= 2024-02-01T12:00:08.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:18:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mWUqgNYFSRyiwBwFGSsJimsjaVM=', 'Content-Length': '0'} 2024-02-01T12:00:08.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.556 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.556 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.556 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.556 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 11:18:46 GMT 2024-02-01T12:00:08.556 INFO:tasks.rgw_multisite_tests:/vclfcn-79/ 2024-02-01T12:00:08.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.556 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:mWUqgNYFSRyiwBwFGSsJimsjaVM= 2024-02-01T12:00:08.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:18:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mWUqgNYFSRyiwBwFGSsJimsjaVM=', 'Content-Length': '0'} 2024-02-01T12:00:08.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.557 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.557 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.557 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.557 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 11:18:46 GMT 2024-02-01T12:00:08.557 INFO:tasks.rgw_multisite_tests:/vclfcn-79/ 2024-02-01T12:00:08.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.557 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:mWUqgNYFSRyiwBwFGSsJimsjaVM= 2024-02-01T12:00:08.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:18:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mWUqgNYFSRyiwBwFGSsJimsjaVM=', 'Content-Length': '0'} 2024-02-01T12:00:08.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.557 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.558 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.558 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.558 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 11:18:46 GMT 2024-02-01T12:00:08.558 INFO:tasks.rgw_multisite_tests:/vclfcn-79/ 2024-02-01T12:00:08.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.558 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:mWUqgNYFSRyiwBwFGSsJimsjaVM= 2024-02-01T12:00:08.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:18:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mWUqgNYFSRyiwBwFGSsJimsjaVM=', 'Content-Length': '0'} 2024-02-01T12:00:08.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.558 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.558 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.559 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.559 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 11:18:46 GMT 2024-02-01T12:00:08.559 INFO:tasks.rgw_multisite_tests:/vclfcn-79/ 2024-02-01T12:00:08.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.559 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:mWUqgNYFSRyiwBwFGSsJimsjaVM= 2024-02-01T12:00:08.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 11:18:46 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:mWUqgNYFSRyiwBwFGSsJimsjaVM=', 'Content-Length': '0'} 2024-02-01T12:00:08.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-02-01T12:00:08.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.559 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-02-01T12:00:08.559 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.559 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-02-01T12:00:08.559 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_multipart_object_sync 2024-02-01T12:00:08.560 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-02-01T12:00:08.560 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-02-01T12:00:08.560 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-02-01T12:00:08.560 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-02-01T12:00:08.560 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/../src/test/rgw/rgw_multi/tests.py", line 1246, in test_multipart_object_sync 2024-02-01T12:00:08.560 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) 2024-02-01T12:00:08.560 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/../src/test/rgw/rgw_multi/tests.py", line 397, in zonegroup_bucket_checkpoint 2024-02-01T12:00:08.560 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2024-02-01T12:00:08.560 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/../src/test/rgw/rgw_multi/tests.py", line 389, in zone_bucket_checkpoint 2024-02-01T12:00:08.560 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2024-02-01T12:00:08.561 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/tasks/rgw_multisite.py", line 197, in admin 2024-02-01T12:00:08.561 INFO:tasks.rgw_multisite_tests: assert r == 0 2024-02-01T12:00:08.561 INFO:tasks.rgw_multisite_tests:AssertionError: 2024-02-01T12:00:08.561 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-02-01T12:00:08.561 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone1 name=vclfcn-65 2024-02-01T12:00:08.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vclfcn-65/ 2024-02-01T12:00:08.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vclfcn-65/ 2024-02-01T12:00:08.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-02-01T12:00:08.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vclfcn-65/ 2024-02-01T12:00:08.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-02-01T12:00:08.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-02-01T12:00:08.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T12:00:08.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-02-01T12:00:08.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-02-01T12:00:08.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.562 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.562 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.562 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.562 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 10:50:19 GMT 2024-02-01T12:00:08.562 INFO:tasks.rgw_multisite_tests:/vclfcn-65/ 2024-02-01T12:00:08.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.562 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:+OXp7r+uCbWCoaoBf5aitrvsav0= 2024-02-01T12:00:08.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+OXp7r+uCbWCoaoBf5aitrvsav0=', 'Content-Length': '0'} 2024-02-01T12:00:08.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000bbb34bbc819827b5-0065bb776b-5765-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:50:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T12:00:08.563 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone2 name=vclfcn-66 2024-02-01T12:00:08.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vclfcn-66/ 2024-02-01T12:00:08.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vclfcn-66/ 2024-02-01T12:00:08.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-02-01T12:00:08.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vclfcn-66/ 2024-02-01T12:00:08.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-02-01T12:00:08.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-02-01T12:00:08.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi187.front.sepia.ceph.com:8000 2024-02-01T12:00:08.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-02-01T12:00:08.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-02-01T12:00:08.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.563 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.563 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.564 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.564 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 10:50:19 GMT 2024-02-01T12:00:08.564 INFO:tasks.rgw_multisite_tests:/vclfcn-66/ 2024-02-01T12:00:08.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.564 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:X56HkM98+NybjkWb+LZyVVfye60= 2024-02-01T12:00:08.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:X56HkM98+NybjkWb+LZyVVfye60=', 'Content-Length': '0'} 2024-02-01T12:00:08.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000008d55c01c3c1a389b-0065bb776b-5350-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:50:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T12:00:08.564 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone3 name=vclfcn-67 2024-02-01T12:00:08.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vclfcn-67/ 2024-02-01T12:00:08.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vclfcn-67/ 2024-02-01T12:00:08.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-02-01T12:00:08.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vclfcn-67/ 2024-02-01T12:00:08.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-02-01T12:00:08.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-02-01T12:00:08.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi148.front.sepia.ceph.com:8001 2024-02-01T12:00:08.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-02-01T12:00:08.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-02-01T12:00:08.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.565 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.565 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.565 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.565 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 10:50:19 GMT 2024-02-01T12:00:08.565 INFO:tasks.rgw_multisite_tests:/vclfcn-67/ 2024-02-01T12:00:08.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.565 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:+N7G8MxPqf0YS/ihTKKkPQ63XVE= 2024-02-01T12:00:08.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+N7G8MxPqf0YS/ihTKKkPQ63XVE=', 'Content-Length': '0'} 2024-02-01T12:00:08.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000f80d697590db6d09-0065bb776b-5762-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:50:19 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T12:00:08.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vclfcn-65/MULTIPART 2024-02-01T12:00:08.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vclfcn-65/MULTIPART 2024-02-01T12:00:08.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vclfcn-65/MULTIPART?uploads 2024-02-01T12:00:08.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vclfcn-65/MULTIPART?uploads 2024-02-01T12:00:08.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: POST 2024-02-01T12:00:08.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vclfcn-65/MULTIPART?uploads 2024-02-01T12:00:08.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-02-01T12:00:08.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-02-01T12:00:08.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T12:00:08.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-02-01T12:00:08.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-02-01T12:00:08.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.567 INFO:tasks.rgw_multisite_tests:POST 2024-02-01T12:00:08.567 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.567 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.567 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 10:50:19 GMT 2024-02-01T12:00:08.567 INFO:tasks.rgw_multisite_tests:/vclfcn-65/MULTIPART?uploads 2024-02-01T12:00:08.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.567 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:0xJ3MWPuSJjbfKXLyoICmIafuKw= 2024-02-01T12:00:08.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:19 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:0xJ3MWPuSJjbfKXLyoICmIafuKw=', 'Content-Length': '0'} 2024-02-01T12:00:08.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000362315371a0001b9-0065bb776b-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '249'), ('Date', 'Thu, 01 Feb 2024 10:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T12:00:08.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'vclfcn-65MULTIPART2~D-K6AESwISRPEo9y0bUvx9MhCOlpttu' 2024-02-01T12:00:08.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vclfcn-65/MULTIPART 2024-02-01T12:00:08.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vclfcn-65/MULTIPART 2024-02-01T12:00:08.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vclfcn-65/MULTIPART?uploadId=2~D-K6AESwISRPEo9y0bUvx9MhCOlpttu&partNumber=1 2024-02-01T12:00:08.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vclfcn-65/MULTIPART?uploadId=2~D-K6AESwISRPEo9y0bUvx9MhCOlpttu&partNumber=1 2024-02-01T12:00:08.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-02-01T12:00:08.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vclfcn-65/MULTIPART?uploadId=2~D-K6AESwISRPEo9y0bUvx9MhCOlpttu&partNumber=1 2024-02-01T12:00:08.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-02-01T12:00:08.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ebKBBg0ze5srhMzzkK3PdA==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2024-02-01T12:00:08.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T12:00:08.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-02-01T12:00:08.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-02-01T12:00:08.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.568 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.568 INFO:tasks.rgw_multisite_tests:ebKBBg0ze5srhMzzkK3PdA== 2024-02-01T12:00:08.568 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-02-01T12:00:08.569 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 10:50:21 GMT 2024-02-01T12:00:08.569 INFO:tasks.rgw_multisite_tests:/vclfcn-65/MULTIPART?partNumber=1&uploadId=2~D-K6AESwISRPEo9y0bUvx9MhCOlpttu 2024-02-01T12:00:08.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.569 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:QAR+AvANfgYW4VMmq6jbCXae+yo= 2024-02-01T12:00:08.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ebKBBg0ze5srhMzzkK3PdA==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:50:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:QAR+AvANfgYW4VMmq6jbCXae+yo='} 2024-02-01T12:00:08.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"79b281060d337b9b2b84ccf390adcf74"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000bfcffdb6e0f95e5-0065bb776d-5765-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T12:00:08.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vclfcn-65/MULTIPART 2024-02-01T12:00:08.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vclfcn-65/MULTIPART 2024-02-01T12:00:08.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vclfcn-65/MULTIPART?uploadId=2~D-K6AESwISRPEo9y0bUvx9MhCOlpttu&partNumber=2 2024-02-01T12:00:08.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vclfcn-65/MULTIPART?uploadId=2~D-K6AESwISRPEo9y0bUvx9MhCOlpttu&partNumber=2 2024-02-01T12:00:08.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-02-01T12:00:08.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vclfcn-65/MULTIPART?uploadId=2~D-K6AESwISRPEo9y0bUvx9MhCOlpttu&partNumber=2 2024-02-01T12:00:08.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-02-01T12:00:08.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'dIQ6OrGTo4m87YmUAtmdXw==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2024-02-01T12:00:08.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T12:00:08.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-02-01T12:00:08.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-02-01T12:00:08.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.570 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.570 INFO:tasks.rgw_multisite_tests:dIQ6OrGTo4m87YmUAtmdXw== 2024-02-01T12:00:08.570 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-02-01T12:00:08.570 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 10:50:21 GMT 2024-02-01T12:00:08.570 INFO:tasks.rgw_multisite_tests:/vclfcn-65/MULTIPART?partNumber=2&uploadId=2~D-K6AESwISRPEo9y0bUvx9MhCOlpttu 2024-02-01T12:00:08.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.570 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:emyyiXLPhIEUSyQ5MN0ijnmcwF4= 2024-02-01T12:00:08.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'dIQ6OrGTo4m87YmUAtmdXw==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:50:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:emyyiXLPhIEUSyQ5MN0ijnmcwF4='} 2024-02-01T12:00:08.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"74843a3ab193a389bced899402d99d5f"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c5fcdf3ba1575545-0065bb776d-5765-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T12:00:08.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vclfcn-65/MULTIPART 2024-02-01T12:00:08.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vclfcn-65/MULTIPART 2024-02-01T12:00:08.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vclfcn-65/MULTIPART?uploadId=2~D-K6AESwISRPEo9y0bUvx9MhCOlpttu&partNumber=3 2024-02-01T12:00:08.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vclfcn-65/MULTIPART?uploadId=2~D-K6AESwISRPEo9y0bUvx9MhCOlpttu&partNumber=3 2024-02-01T12:00:08.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-02-01T12:00:08.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vclfcn-65/MULTIPART?uploadId=2~D-K6AESwISRPEo9y0bUvx9MhCOlpttu&partNumber=3 2024-02-01T12:00:08.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-02-01T12:00:08.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'e4RW4edMN49Fhh9TYZ51tg==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2024-02-01T12:00:08.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T12:00:08.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-02-01T12:00:08.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-02-01T12:00:08.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.572 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.572 INFO:tasks.rgw_multisite_tests:e4RW4edMN49Fhh9TYZ51tg== 2024-02-01T12:00:08.572 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-02-01T12:00:08.572 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 10:50:21 GMT 2024-02-01T12:00:08.572 INFO:tasks.rgw_multisite_tests:/vclfcn-65/MULTIPART?partNumber=3&uploadId=2~D-K6AESwISRPEo9y0bUvx9MhCOlpttu 2024-02-01T12:00:08.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.572 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:j4U5CwnEAQMnizs2/QENj0ICTyM= 2024-02-01T12:00:08.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'e4RW4edMN49Fhh9TYZ51tg==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:50:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:j4U5CwnEAQMnizs2/QENj0ICTyM='} 2024-02-01T12:00:08.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"7b8456e1e74c378f45861f53619e75b6"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bdf8e42246b26986-0065bb776d-5765-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T12:00:08.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vclfcn-65/MULTIPART 2024-02-01T12:00:08.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vclfcn-65/MULTIPART 2024-02-01T12:00:08.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vclfcn-65/MULTIPART?uploadId=2~D-K6AESwISRPEo9y0bUvx9MhCOlpttu&partNumber=4 2024-02-01T12:00:08.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vclfcn-65/MULTIPART?uploadId=2~D-K6AESwISRPEo9y0bUvx9MhCOlpttu&partNumber=4 2024-02-01T12:00:08.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-02-01T12:00:08.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vclfcn-65/MULTIPART?uploadId=2~D-K6AESwISRPEo9y0bUvx9MhCOlpttu&partNumber=4 2024-02-01T12:00:08.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-02-01T12:00:08.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'l+jDi4yPcIlVvEiQwDQZTQ==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2024-02-01T12:00:08.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T12:00:08.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-02-01T12:00:08.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-02-01T12:00:08.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.573 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.573 INFO:tasks.rgw_multisite_tests:l+jDi4yPcIlVvEiQwDQZTQ== 2024-02-01T12:00:08.573 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-02-01T12:00:08.573 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 10:50:21 GMT 2024-02-01T12:00:08.574 INFO:tasks.rgw_multisite_tests:/vclfcn-65/MULTIPART?partNumber=4&uploadId=2~D-K6AESwISRPEo9y0bUvx9MhCOlpttu 2024-02-01T12:00:08.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.574 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:Q8sHtvhpMHqxNMlvTAOQftFOR4Y= 2024-02-01T12:00:08.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'l+jDi4yPcIlVvEiQwDQZTQ==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:50:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:Q8sHtvhpMHqxNMlvTAOQftFOR4Y='} 2024-02-01T12:00:08.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"97e8c38b8c8f708955bc4890c034194d"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000210ba675038ca203-0065bb776d-5765-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T12:00:08.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vclfcn-65/MULTIPART 2024-02-01T12:00:08.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vclfcn-65/MULTIPART 2024-02-01T12:00:08.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vclfcn-65/MULTIPART?uploadId=2~D-K6AESwISRPEo9y0bUvx9MhCOlpttu 2024-02-01T12:00:08.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vclfcn-65/MULTIPART?uploadId=2~D-K6AESwISRPEo9y0bUvx9MhCOlpttu 2024-02-01T12:00:08.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-02-01T12:00:08.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vclfcn-65/MULTIPART?uploadId=2~D-K6AESwISRPEo9y0bUvx9MhCOlpttu 2024-02-01T12:00:08.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-02-01T12:00:08.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-02-01T12:00:08.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T12:00:08.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-02-01T12:00:08.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-02-01T12:00:08.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.575 INFO:tasks.rgw_multisite_tests:GET 2024-02-01T12:00:08.575 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.575 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.575 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 10:50:21 GMT 2024-02-01T12:00:08.575 INFO:tasks.rgw_multisite_tests:/vclfcn-65/MULTIPART?uploadId=2~D-K6AESwISRPEo9y0bUvx9MhCOlpttu 2024-02-01T12:00:08.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.575 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:2Zo6K7xOJiskfkh3p8IHIXOexzs= 2024-02-01T12:00:08.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2Zo6K7xOJiskfkh3p8IHIXOexzs=', 'Content-Length': '0'} 2024-02-01T12:00:08.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009233e65651e13a03-0065bb776d-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T12:00:08.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vclfcn-65/MULTIPART 2024-02-01T12:00:08.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vclfcn-65/MULTIPART 2024-02-01T12:00:08.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vclfcn-65/MULTIPART?uploadId=2~D-K6AESwISRPEo9y0bUvx9MhCOlpttu 2024-02-01T12:00:08.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vclfcn-65/MULTIPART?uploadId=2~D-K6AESwISRPEo9y0bUvx9MhCOlpttu 2024-02-01T12:00:08.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: POST 2024-02-01T12:00:08.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vclfcn-65/MULTIPART?uploadId=2~D-K6AESwISRPEo9y0bUvx9MhCOlpttu 2024-02-01T12:00:08.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-02-01T12:00:08.576 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.576 INFO:tasks.rgw_multisite_tests: 1 2024-02-01T12:00:08.576 INFO:tasks.rgw_multisite_tests: "79b281060d337b9b2b84ccf390adcf74" 2024-02-01T12:00:08.576 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.576 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.577 INFO:tasks.rgw_multisite_tests: 2 2024-02-01T12:00:08.577 INFO:tasks.rgw_multisite_tests: "74843a3ab193a389bced899402d99d5f" 2024-02-01T12:00:08.577 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.577 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.577 INFO:tasks.rgw_multisite_tests: 3 2024-02-01T12:00:08.577 INFO:tasks.rgw_multisite_tests: "7b8456e1e74c378f45861f53619e75b6" 2024-02-01T12:00:08.577 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.577 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.577 INFO:tasks.rgw_multisite_tests: 4 2024-02-01T12:00:08.577 INFO:tasks.rgw_multisite_tests: "97e8c38b8c8f708955bc4890c034194d" 2024-02-01T12:00:08.577 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.577 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'Content-Type': 'text/xml'} 2024-02-01T12:00:08.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T12:00:08.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-02-01T12:00:08.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-02-01T12:00:08.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.578 INFO:tasks.rgw_multisite_tests:POST 2024-02-01T12:00:08.578 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.578 INFO:tasks.rgw_multisite_tests:text/xml 2024-02-01T12:00:08.578 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 10:50:21 GMT 2024-02-01T12:00:08.578 INFO:tasks.rgw_multisite_tests:/vclfcn-65/MULTIPART?uploadId=2~D-K6AESwISRPEo9y0bUvx9MhCOlpttu 2024-02-01T12:00:08.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.578 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:2cXixmeEPv/Ryw+bwvoArGL714E= 2024-02-01T12:00:08.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:50:21 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:2cXixmeEPv/Ryw+bwvoArGL714E=', 'Content-Length': '460'} 2024-02-01T12:00:08.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000318cbbf3d6dce6a7-0065bb776d-5765-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '325'), ('Date', 'Thu, 01 Feb 2024 10:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T12:00:08.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: http://smithi148.front.sepia.ceph.com:8000/vclfcn-65/MULTIPARTvclfcn-65MULTIPARTd07d411772772538cd4b657da660bfb5-4 2024-02-01T12:00:08.579 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T12:00:08.579 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T12:00:08.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi148: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T12:00:08.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi148.stderr: INFO: ignoring --setuser ceph since I am not root 2024-02-01T12:00:08.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi148.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-02-01T12:00:08.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi148.stderr: INFO: No --period given, using current period=f2e6c290-e7aa-4279-b77e-15403b18d61f 2024-02-01T12:00:08.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi148.stdout: INFO: [{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1706784619.449972_360.1","last_update":"2024-02-01T10:50:19.449972Z"},{"marker":"","last_update":"0.000000"}] 2024-02-01T12:00:08.579 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '', 1: '', 2: '1_1706784619.449972_360.1', 3: ''} 2024-02-01T12:00:08.579 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-02-01T12:00:08.579 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T12:00:08.579 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T12:00:08.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T12:00:08.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: ignoring --setuser ceph since I am not root 2024-02-01T12:00:08.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-02-01T12:00:08.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: 2024-02-01T10:50:21.955+0000 7fdd7a950980 20 RGW-SYNC:meta: read sync status 2024-02-01T12:00:08.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: 2024-02-01T10:50:21.955+0000 7fdd7a950980 20 RGW-SYNC:meta: read sync status 2024-02-01T12:00:08.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706784619.320199_349.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:50:19.320199Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:50:21Z"} 2024-02-01T12:00:08.580 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706784619.320199_349.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:50:19.320199Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:50:21Z"} 2024-02-01T12:00:08.580 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706784619.320199_349.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:50:19.320199Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T12:00:08.580 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '', 1: '', 2: '1_1706784619.449972_360.1', 3: ''} 2024-02-01T12:00:08.580 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '', 1: '', 2: '1_1706784619.320199_349.1', 3: ''} 2024-02-01T12:00:08.580 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1706784619.449972_360.1 target=1_1706784619.320199_349.1 2024-02-01T12:00:08.580 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi148.stderr: INFO: daemon-helper: command crashed with signal 11 2024-02-01T12:00:08.580 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T12:00:08.581 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T12:00:08.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T12:00:08.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: ignoring --setuser ceph since I am not root 2024-02-01T12:00:08.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-02-01T12:00:08.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: 2024-02-01T10:50:27.091+0000 7ff779ce4980 20 RGW-SYNC:meta: read sync status 2024-02-01T12:00:08.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: 2024-02-01T10:50:27.095+0000 7ff779ce4980 20 RGW-SYNC:meta: read sync status 2024-02-01T12:00:08.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706784619.320199_349.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:50:19.320199Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:50:27Z"} 2024-02-01T12:00:08.581 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706784619.320199_349.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:50:19.320199Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:50:27Z"} 2024-02-01T12:00:08.581 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706784619.320199_349.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:50:19.320199Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T12:00:08.581 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '', 1: '', 2: '1_1706784619.449972_360.1', 3: ''} 2024-02-01T12:00:08.581 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '', 1: '', 2: '1_1706784619.320199_349.1', 3: ''} 2024-02-01T12:00:08.581 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1706784619.449972_360.1 target=1_1706784619.320199_349.1 2024-02-01T12:00:08.582 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~0s 2024-02-01T12:00:08.582 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T12:00:08.582 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T12:00:08.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T12:00:08.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: ignoring --setuser ceph since I am not root 2024-02-01T12:00:08.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-02-01T12:00:08.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: 2024-02-01T10:50:32.219+0000 7feea4fe2980 20 RGW-SYNC:meta: read sync status 2024-02-01T12:00:08.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: 2024-02-01T10:50:32.219+0000 7feea4fe2980 20 RGW-SYNC:meta: read sync status 2024-02-01T12:00:08.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706784619.449972_360.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:50:19.449972Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:50:32Z"} 2024-02-01T12:00:08.582 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706784619.449972_360.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:50:19.449972Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:50:32Z"} 2024-02-01T12:00:08.582 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706784619.449972_360.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:50:19.449972Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T12:00:08.582 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '', 1: '', 2: '1_1706784619.449972_360.1', 3: ''} 2024-02-01T12:00:08.582 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '', 1: '', 2: '1_1706784619.449972_360.1', 3: ''} 2024-02-01T12:00:08.583 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone2 2024-02-01T12:00:08.583 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone3 2024-02-01T12:00:08.583 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T12:00:08.583 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T12:00:08.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi148: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T12:00:08.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi148.stderr: INFO: ignoring --setuser ceph since I am not root 2024-02-01T12:00:08.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi148.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-02-01T12:00:08.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi148.stderr: INFO: 2024-02-01T10:50:32.345+0000 7f311308f980 20 RGW-SYNC:meta: read sync status 2024-02-01T12:00:08.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi148.stderr: INFO: 2024-02-01T10:50:32.345+0000 7f311308f980 20 RGW-SYNC:meta: read sync status 2024-02-01T12:00:08.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi148.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706784619.449972_360.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:50:19.449972Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:50:32Z"} 2024-02-01T12:00:08.583 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706784619.449972_360.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:50:19.449972Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:50:32Z"} 2024-02-01T12:00:08.583 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706784619.449972_360.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:50:19.449972Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T12:00:08.584 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '', 1: '', 2: '1_1706784619.449972_360.1', 3: ''} 2024-02-01T12:00:08.584 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '', 1: '', 2: '1_1706784619.449972_360.1', 3: ''} 2024-02-01T12:00:08.584 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone3 2024-02-01T12:00:08.584 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone1 target=test-zone2 bucket=vclfcn-65 2024-02-01T12:00:08.584 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T12:00:08.584 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'vclfcn-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-02-01T12:00:08.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket vclfcn-65 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-02-01T12:00:08.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: ignoring --setuser ceph since I am not root 2024-02-01T12:00:08.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-02-01T12:00:08.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: 2024-02-01T10:50:32.487+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T12:00:08.584 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~6s 2024-02-01T12:00:08.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: 2024-02-01T10:50:37.487+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T12:00:08.585 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi187.stderr: INFO: daemon-helper: command crashed with signal 11 2024-02-01T12:00:08.585 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~11s 2024-02-01T12:00:08.585 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~0s 2024-02-01T12:00:08.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: 2024-02-01T10:50:42.491+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T12:00:08.585 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~17s 2024-02-01T12:00:08.585 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~6s 2024-02-01T12:00:08.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: 2024-02-01T10:50:47.491+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T12:00:08.585 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~22s 2024-02-01T12:00:08.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: 2024-02-01T10:50:52.491+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T12:00:08.585 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~11s 2024-02-01T12:00:08.585 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~28s 2024-02-01T12:00:08.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: 2024-02-01T10:50:57.495+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T12:00:08.585 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~17s 2024-02-01T12:00:08.585 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~34s 2024-02-01T12:00:08.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: 2024-02-01T10:51:02.495+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T12:00:08.586 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~22s 2024-02-01T12:00:08.586 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~39s 2024-02-01T12:00:08.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: 2024-02-01T10:51:07.499+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T12:00:08.586 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~28s 2024-02-01T12:00:08.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: 2024-02-01T10:51:12.499+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T12:00:08.586 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~45s 2024-02-01T12:00:08.586 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~34s 2024-02-01T12:00:08.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: 2024-02-01T10:51:17.499+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T12:00:08.586 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~50s 2024-02-01T12:00:08.586 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~39s 2024-02-01T12:00:08.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: 2024-02-01T10:51:22.499+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T12:00:08.587 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~56s 2024-02-01T12:00:08.587 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~45s 2024-02-01T12:00:08.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: 2024-02-01T10:51:27.502+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T12:00:08.587 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~62s 2024-02-01T12:00:08.587 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~50s 2024-02-01T12:00:08.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: 2024-02-01T10:51:32.502+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T12:00:08.587 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~67s 2024-02-01T12:00:08.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: 2024-02-01T10:51:37.502+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T12:00:08.587 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~56s 2024-02-01T12:00:08.587 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~73s 2024-02-01T12:00:08.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: 2024-02-01T10:51:42.506+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T12:00:08.587 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~62s 2024-02-01T12:00:08.588 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~78s 2024-02-01T12:00:08.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: 2024-02-01T10:51:47.506+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T12:00:08.588 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~67s 2024-02-01T12:00:08.588 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~84s 2024-02-01T12:00:08.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: 2024-02-01T10:51:52.506+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T12:00:08.588 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~73s 2024-02-01T12:00:08.588 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~90s 2024-02-01T12:00:08.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: 2024-02-01T10:51:57.506+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T12:00:08.588 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~78s 2024-02-01T12:00:08.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: 2024-02-01T10:52:02.510+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T12:00:08.588 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~95s 2024-02-01T12:00:08.588 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~84s 2024-02-01T12:00:08.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: 2024-02-01T10:52:07.510+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T12:00:08.589 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~101s 2024-02-01T12:00:08.589 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~90s 2024-02-01T12:00:08.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: 2024-02-01T10:52:12.510+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T12:00:08.589 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~107s 2024-02-01T12:00:08.589 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~95s 2024-02-01T12:00:08.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: 2024-02-01T10:52:17.509+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T12:00:08.589 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~112s 2024-02-01T12:00:08.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: 2024-02-01T10:52:22.513+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T12:00:08.589 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~101s 2024-02-01T12:00:08.589 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~118s 2024-02-01T12:00:08.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: 2024-02-01T10:52:27.513+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T12:00:08.589 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~107s 2024-02-01T12:00:08.590 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~123s 2024-02-01T12:00:08.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: 2024-02-01T10:52:32.513+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T12:00:08.590 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~112s 2024-02-01T12:00:08.590 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~129s 2024-02-01T12:00:08.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: 2024-02-01T10:52:37.517+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T12:00:08.590 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~118s 2024-02-01T12:00:08.590 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~135s 2024-02-01T12:00:08.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: 2024-02-01T10:52:42.517+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T12:00:08.590 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~123s 2024-02-01T12:00:08.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: 2024-02-01T10:52:47.521+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T12:00:08.590 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~140s 2024-02-01T12:00:08.590 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~129s 2024-02-01T12:00:08.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: 2024-02-01T10:52:52.517+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T12:00:08.591 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~146s 2024-02-01T12:00:08.591 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~135s 2024-02-01T12:00:08.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: 2024-02-01T10:52:57.521+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T12:00:08.591 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~151s 2024-02-01T12:00:08.591 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~140s 2024-02-01T12:00:08.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: 2024-02-01T10:53:02.521+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T12:00:08.591 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~157s 2024-02-01T12:00:08.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: 2024-02-01T10:53:07.521+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T12:00:08.591 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~146s 2024-02-01T12:00:08.591 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~163s 2024-02-01T12:00:08.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: 2024-02-01T10:53:12.524+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T12:00:08.591 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~151s 2024-02-01T12:00:08.591 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~168s 2024-02-01T12:00:08.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: 2024-02-01T10:53:17.524+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T12:00:08.592 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~157s 2024-02-01T12:00:08.592 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~174s 2024-02-01T12:00:08.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: 2024-02-01T10:53:22.528+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T12:00:08.592 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~163s 2024-02-01T12:00:08.592 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~179s 2024-02-01T12:00:08.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: 2024-02-01T10:53:27.528+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T12:00:08.592 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~168s 2024-02-01T12:00:08.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: 2024-02-01T10:53:32.532+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T12:00:08.592 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~185s 2024-02-01T12:00:08.592 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~174s 2024-02-01T12:00:08.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: 2024-02-01T10:53:37.532+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T12:00:08.592 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~191s 2024-02-01T12:00:08.593 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~179s 2024-02-01T12:00:08.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: 2024-02-01T10:53:42.532+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T12:00:08.593 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~196s 2024-02-01T12:00:08.593 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~185s 2024-02-01T12:00:08.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: 2024-02-01T10:53:47.532+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T12:00:08.593 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~202s 2024-02-01T12:00:08.593 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~191s 2024-02-01T12:00:08.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: 2024-02-01T10:53:52.536+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T12:00:08.593 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~207s 2024-02-01T12:00:08.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: 2024-02-01T10:53:57.536+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T12:00:08.593 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~196s 2024-02-01T12:00:08.593 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~213s 2024-02-01T12:00:08.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: 2024-02-01T10:54:02.536+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T12:00:08.594 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~202s 2024-02-01T12:00:08.594 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~219s 2024-02-01T12:00:08.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: 2024-02-01T10:54:07.539+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T12:00:08.594 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~207s 2024-02-01T12:00:08.594 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~224s 2024-02-01T12:00:08.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: 2024-02-01T10:54:12.539+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T12:00:08.594 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~213s 2024-02-01T12:00:08.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: 2024-02-01T10:54:17.539+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T12:00:08.594 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~230s 2024-02-01T12:00:08.594 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~219s 2024-02-01T12:00:08.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: 2024-02-01T10:54:22.539+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T12:00:08.594 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~235s 2024-02-01T12:00:08.595 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~224s 2024-02-01T12:00:08.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: 2024-02-01T10:54:27.543+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T12:00:08.595 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~241s 2024-02-01T12:00:08.595 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~230s 2024-02-01T12:00:08.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: 2024-02-01T10:54:32.543+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T12:00:08.595 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~247s 2024-02-01T12:00:08.595 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~235s 2024-02-01T12:00:08.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: 2024-02-01T10:54:37.543+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T12:00:08.595 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~252s 2024-02-01T12:00:08.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: 2024-02-01T10:54:42.547+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T12:00:08.595 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~241s 2024-02-01T12:00:08.595 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~258s 2024-02-01T12:00:08.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: 2024-02-01T10:54:47.547+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T12:00:08.595 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~247s 2024-02-01T12:00:08.596 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~264s 2024-02-01T12:00:08.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: 2024-02-01T10:54:52.547+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T12:00:08.596 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~252s 2024-02-01T12:00:08.596 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~269s 2024-02-01T12:00:08.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: 2024-02-01T10:54:57.550+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T12:00:08.596 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~258s 2024-02-01T12:00:08.596 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~275s 2024-02-01T12:00:08.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: 2024-02-01T10:55:02.550+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T12:00:08.596 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~263s 2024-02-01T12:00:08.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: 2024-02-01T10:55:07.550+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T12:00:08.596 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~280s 2024-02-01T12:00:08.596 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~269s 2024-02-01T12:00:08.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: 2024-02-01T10:55:12.550+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T12:00:08.597 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~286s 2024-02-01T12:00:08.597 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~275s 2024-02-01T12:00:08.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: 2024-02-01T10:55:17.554+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T12:00:08.597 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~292s 2024-02-01T12:00:08.597 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~280s 2024-02-01T12:00:08.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: 2024-02-01T10:55:22.554+0000 7f518ad76980 1 waiting to reach incremental sync.. 2024-02-01T12:00:08.597 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~297s 2024-02-01T12:00:08.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: 2024-02-01T10:55:27.554+0000 7f518ad76980 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-02-01T12:00:08.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: 2024-02-01T10:55:27.554+0000 7f518ad76980 0 bucket sync checkpoint failed: (110) Connection timed out 2024-02-01T12:00:08.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: 2024-02-01T10:55:27.554+0000 7f518ad76980 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-02-01T12:00:08.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 110 2024-02-01T12:00:08.597 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-02-01T12:00:08.598 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.598 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-02-01T12:00:08.598 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_encrypted_object_sync 2024-02-01T12:00:08.598 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-02-01T12:00:08.598 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-02-01T12:00:08.598 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-02-01T12:00:08.598 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-02-01T12:00:08.598 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/../src/test/rgw/rgw_multi/tests.py", line 1282, in test_encrypted_object_sync 2024-02-01T12:00:08.598 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-02-01T12:00:08.598 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/../src/test/rgw/rgw_multi/tests.py", line 210, in zonegroup_meta_checkpoint 2024-02-01T12:00:08.598 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-02-01T12:00:08.599 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/../src/test/rgw/rgw_multi/tests.py", line 185, in zone_meta_checkpoint 2024-02-01T12:00:08.599 INFO:tasks.rgw_multisite_tests: global_status, period, realm_epoch, num_shards, sync_status = meta_sync_status(zone) 2024-02-01T12:00:08.599 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/../src/test/rgw/rgw_multi/tests.py", line 142, in meta_sync_status 2024-02-01T12:00:08.599 INFO:tasks.rgw_multisite_tests: assert(retcode == 2) # ENOENT 2024-02-01T12:00:08.599 INFO:tasks.rgw_multisite_tests:AssertionError: 2024-02-01T12:00:08.599 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-02-01T12:00:08.599 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone1 name=vclfcn-68 2024-02-01T12:00:08.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vclfcn-68/ 2024-02-01T12:00:08.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vclfcn-68/ 2024-02-01T12:00:08.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-02-01T12:00:08.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vclfcn-68/ 2024-02-01T12:00:08.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-02-01T12:00:08.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-02-01T12:00:08.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T12:00:08.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-02-01T12:00:08.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-02-01T12:00:08.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-02-01T12:00:08.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.600 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.600 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.600 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.600 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 10:55:27 GMT 2024-02-01T12:00:08.600 INFO:tasks.rgw_multisite_tests:/vclfcn-68/ 2024-02-01T12:00:08.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.600 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:+1o6iZ15GVPxc8A+OLJyieOwbjc= 2024-02-01T12:00:08.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Thu, 01 Feb 2024 10:55:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:+1o6iZ15GVPxc8A+OLJyieOwbjc=', 'Content-Length': '0'} 2024-02-01T12:00:08.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000685607002e74b943-0065bb789f-5765-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Thu, 01 Feb 2024 10:55:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T12:00:08.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vclfcn-68/testobj-sse-c 2024-02-01T12:00:08.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vclfcn-68/testobj-sse-c 2024-02-01T12:00:08.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-02-01T12:00:08.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vclfcn-68/testobj-sse-c 2024-02-01T12:00:08.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-02-01T12:00:08.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2024-02-01T12:00:08.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T12:00:08.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-02-01T12:00:08.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-02-01T12:00:08.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.601 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.601 INFO:tasks.rgw_multisite_tests:3FCGuEcom6i4veFJuDiBdQ== 2024-02-01T12:00:08.602 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-02-01T12:00:08.602 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 10:55:27 GMT 2024-02-01T12:00:08.602 INFO:tasks.rgw_multisite_tests:x-amz-server-side-encryption-customer-algorithm:AES256 2024-02-01T12:00:08.602 INFO:tasks.rgw_multisite_tests:x-amz-server-side-encryption-customer-key:pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs= 2024-02-01T12:00:08.602 INFO:tasks.rgw_multisite_tests:x-amz-server-side-encryption-customer-key-md5:DWygnHRtgiJ77HCm+1rvHw== 2024-02-01T12:00:08.602 INFO:tasks.rgw_multisite_tests:/vclfcn-68/testobj-sse-c 2024-02-01T12:00:08.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.602 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:37C09I1eu+J3RNjmDqnwKIw/a7o= 2024-02-01T12:00:08.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:55:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:37C09I1eu+J3RNjmDqnwKIw/a7o='} 2024-02-01T12:00:08.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-server-side-encryption-customer-algorithm', 'AES256'), ('x-amz-server-side-encryption-customer-key-MD5', 'DWygnHRtgiJ77HCm+1rvHw=='), ('x-amz-request-id', 'tx000005bf238a48701f1f5-0065bb789f-5765-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:55:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T12:00:08.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/vclfcn-68/testobj-sse-kms 2024-02-01T12:00:08.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/vclfcn-68/testobj-sse-kms 2024-02-01T12:00:08.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-02-01T12:00:08.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /vclfcn-68/testobj-sse-kms 2024-02-01T12:00:08.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-02-01T12:00:08.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'x-amz-server-side-encryption': 'aws:kms', 'x-amz-server-side-encryption-aws-kms-key-id': 'testkey-1', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2024-02-01T12:00:08.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi148.front.sepia.ceph.com:8000 2024-02-01T12:00:08.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-02-01T12:00:08.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-02-01T12:00:08.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-02-01T12:00:08.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-02-01T12:00:08.603 INFO:tasks.rgw_multisite_tests:PUT 2024-02-01T12:00:08.603 INFO:tasks.rgw_multisite_tests:3FCGuEcom6i4veFJuDiBdQ== 2024-02-01T12:00:08.603 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-02-01T12:00:08.603 INFO:tasks.rgw_multisite_tests:Thu, 01 Feb 2024 10:55:27 GMT 2024-02-01T12:00:08.603 INFO:tasks.rgw_multisite_tests:x-amz-server-side-encryption:aws:kms 2024-02-01T12:00:08.603 INFO:tasks.rgw_multisite_tests:x-amz-server-side-encryption-aws-kms-key-id:testkey-1 2024-02-01T12:00:08.604 INFO:tasks.rgw_multisite_tests:/vclfcn-68/testobj-sse-kms 2024-02-01T12:00:08.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-02-01T12:00:08.604 INFO:tasks.rgw_multisite_tests:AWS 1IMYR3NXFV2H92HDXL01:p+V/Wi+CGgOgWryyx1cVnewDu4U= 2024-02-01T12:00:08.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'x-amz-server-side-encryption': 'aws:kms', 'x-amz-server-side-encryption-aws-kms-key-id': 'testkey-1', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Thu, 01 Feb 2024 10:55:27 GMT', 'Authorization': 'AWS 1IMYR3NXFV2H92HDXL01:p+V/Wi+CGgOgWryyx1cVnewDu4U='} 2024-02-01T12:00:08.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-server-side-encryption', 'aws:kms'), ('x-amz-server-side-encryption-aws-kms-key-id', 'testkey-1'), ('x-amz-server-side-encryption-context', 'eyJhd3M6czM6YXJuIjoiYXJuOmF3czpzMzo6cmd3LW11bHRpc2l0ZS10ZXN0LXVzZXI6dmNsZmNuLTY4L3Rlc3RvYmotc3NlLWttcyJ9'), ('x-amz-request-id', 'tx000001050489dcc5e2a0e-0065bb789f-5765-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Thu, 01 Feb 2024 10:55:27 GMT'), ('Connection', 'Keep-Alive')] 2024-02-01T12:00:08.604 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T12:00:08.604 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T12:00:08.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi148: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T12:00:08.604 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~286s 2024-02-01T12:00:08.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi148.stderr: INFO: ignoring --setuser ceph since I am not root 2024-02-01T12:00:08.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi148.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-02-01T12:00:08.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi148.stderr: INFO: No --period given, using current period=f2e6c290-e7aa-4279-b77e-15403b18d61f 2024-02-01T12:00:08.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi148.stdout: INFO: [{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1706784927.652136_370.1","last_update":"2024-02-01T10:55:27.652136Z"},{"marker":"","last_update":"0.000000"}] 2024-02-01T12:00:08.605 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '', 1: '', 2: '1_1706784927.652136_370.1', 3: ''} 2024-02-01T12:00:08.605 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-02-01T12:00:08.605 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T12:00:08.605 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T12:00:08.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T12:00:08.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: ignoring --setuser ceph since I am not root 2024-02-01T12:00:08.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-02-01T12:00:08.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: 2024-02-01T10:55:27.930+0000 7f433efcc980 20 RGW-SYNC:meta: read sync status 2024-02-01T12:00:08.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: 2024-02-01T10:55:27.930+0000 7f433efcc980 20 RGW-SYNC:meta: read sync status 2024-02-01T12:00:08.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706784619.449972_360.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:50:19.449972Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:55:27Z"} 2024-02-01T12:00:08.605 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706784619.449972_360.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:50:19.449972Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:55:27Z"} 2024-02-01T12:00:08.605 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706784619.449972_360.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:50:19.449972Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T12:00:08.606 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '', 1: '', 2: '1_1706784927.652136_370.1', 3: ''} 2024-02-01T12:00:08.606 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '', 1: '', 2: '1_1706784619.449972_360.1', 3: ''} 2024-02-01T12:00:08.606 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1706784927.652136_370.1 target=1_1706784619.449972_360.1 2024-02-01T12:00:08.606 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~303s 2024-02-01T12:00:08.606 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: BARK! unmounting mounts and killing all daemons 2024-02-01T12:00:08.606 INFO:tasks.rgw_multisite_tests:tasks.ceph.c1.osd.0: INFO: Sent signal 15 2024-02-01T12:00:08.606 INFO:tasks.rgw_multisite_tests:tasks.ceph.c1.osd.1: INFO: Sent signal 15 2024-02-01T12:00:08.606 INFO:tasks.rgw_multisite_tests:tasks.ceph.c1.osd.2: INFO: Sent signal 15 2024-02-01T12:00:08.606 INFO:tasks.rgw_multisite_tests:tasks.ceph.c1.mon.a: INFO: Sent signal 15 2024-02-01T12:00:08.606 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0: INFO: Sent signal 15 2024-02-01T12:00:08.606 INFO:tasks.rgw_multisite_tests:tasks.ceph.c1.mgr.x: INFO: Sent signal 15 2024-02-01T12:00:08.606 INFO:tasks.rgw_multisite_tests:tasks.ceph.c1.osd.0.smithi148.stderr: INFO: 2024-02-01T10:55:31.109+0000 7f15af050640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster c1 -i 0 (PID: 22575) UID: 0 2024-02-01T12:00:08.606 INFO:tasks.rgw_multisite_tests:tasks.ceph.c1.osd.0.smithi148.stderr: INFO: 2024-02-01T10:55:31.109+0000 7f15af050640 -1 osd.0 59 *** Got signal Terminated *** 2024-02-01T12:00:08.607 INFO:tasks.rgw_multisite_tests:tasks.ceph.c1.osd.0.smithi148.stderr: INFO: 2024-02-01T10:55:31.109+0000 7f15af050640 -1 osd.0 59 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-01T12:00:08.607 INFO:tasks.rgw_multisite_tests:tasks.ceph.c1.osd.1.smithi148.stderr: INFO: 2024-02-01T10:55:31.109+0000 7faf02b24640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster c1 -i 1 (PID: 22578) UID: 0 2024-02-01T12:00:08.607 INFO:tasks.rgw_multisite_tests:tasks.ceph.c1.osd.1.smithi148.stderr: INFO: 2024-02-01T10:55:31.109+0000 7faf02b24640 -1 osd.1 59 *** Got signal Terminated *** 2024-02-01T12:00:08.607 INFO:tasks.rgw_multisite_tests:tasks.ceph.c1.osd.1.smithi148.stderr: INFO: 2024-02-01T10:55:31.109+0000 7faf02b24640 -1 osd.1 59 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-01T12:00:08.607 INFO:tasks.rgw_multisite_tests:tasks.ceph.c1.mon.a.smithi148.stderr: INFO: 2024-02-01T10:55:31.109+0000 7f12cda22640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mon -f --cluster c1 -i a (PID: 22403) UID: 0 2024-02-01T12:00:08.607 INFO:tasks.rgw_multisite_tests:tasks.ceph.c1.mon.a.smithi148.stderr: INFO: 2024-02-01T10:55:31.109+0000 7f12cda22640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-02-01T12:00:08.607 INFO:tasks.rgw_multisite_tests:tasks.ceph.c1.osd.2.smithi148.stderr: INFO: 2024-02-01T10:55:31.109+0000 7f9dc250b640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster c1 -i 2 (PID: 22582) UID: 0 2024-02-01T12:00:08.607 INFO:tasks.rgw_multisite_tests:tasks.ceph.c1.osd.2.smithi148.stderr: INFO: 2024-02-01T10:55:31.109+0000 7f9dc250b640 -1 osd.2 59 *** Got signal Terminated *** 2024-02-01T12:00:08.607 INFO:tasks.rgw_multisite_tests:tasks.ceph.c1.osd.2.smithi148.stderr: INFO: 2024-02-01T10:55:31.109+0000 7f9dc250b640 -1 osd.2 59 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-01T12:00:08.607 INFO:tasks.rgw_multisite_tests:tasks.ceph.c1.mgr.x.smithi148.stderr: INFO: 2024-02-01T10:55:31.109+0000 7f7508cba640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mgr -f --cluster c1 -i x (PID: 22405) UID: 0 2024-02-01T12:00:08.607 INFO:tasks.rgw_multisite_tests:tasks.ceph.c1.mgr.x.smithi148.stderr: INFO: 2024-02-01T10:55:31.109+0000 7f7508cba640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-02-01T12:00:08.607 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi148.stdout: INFO: 2024-02-01T10:55:31.109+0000 7f606358f640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 33468) UID: 0 2024-02-01T12:00:08.607 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi148.stdout: INFO: 2024-02-01T10:55:31.109+0000 7f6066dd7a40 -1 shutting down 2024-02-01T12:00:08.607 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T12:00:08.608 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T12:00:08.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T12:00:08.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: ignoring --setuser ceph since I am not root 2024-02-01T12:00:08.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-02-01T12:00:08.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: 2024-02-01T10:55:33.058+0000 7f8f2578d980 20 RGW-SYNC:meta: read sync status 2024-02-01T12:00:08.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: 2024-02-01T10:55:33.058+0000 7f8f2578d980 20 RGW-SYNC:meta: read sync status 2024-02-01T12:00:08.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706784619.449972_360.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:50:19.449972Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:55:33Z"} 2024-02-01T12:00:08.608 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706784619.449972_360.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:50:19.449972Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:55:33Z"} 2024-02-01T12:00:08.608 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706784619.449972_360.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:50:19.449972Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T12:00:08.608 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '', 1: '', 2: '1_1706784927.652136_370.1', 3: ''} 2024-02-01T12:00:08.608 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '', 1: '', 2: '1_1706784619.449972_360.1', 3: ''} 2024-02-01T12:00:08.608 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1706784927.652136_370.1 target=1_1706784619.449972_360.1 2024-02-01T12:00:08.609 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~292s 2024-02-01T12:00:08.609 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T12:00:08.609 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T12:00:08.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T12:00:08.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: ignoring --setuser ceph since I am not root 2024-02-01T12:00:08.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-02-01T12:00:08.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: 2024-02-01T10:55:38.182+0000 7f592cad5980 20 RGW-SYNC:meta: read sync status 2024-02-01T12:00:08.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: 2024-02-01T10:55:38.186+0000 7f592cad5980 20 RGW-SYNC:meta: read sync status 2024-02-01T12:00:08.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706784619.449972_360.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:50:19.449972Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:55:38Z"} 2024-02-01T12:00:08.609 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706784619.449972_360.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:50:19.449972Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:55:38Z"} 2024-02-01T12:00:08.609 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706784619.449972_360.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:50:19.449972Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T12:00:08.609 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '', 1: '', 2: '1_1706784927.652136_370.1', 3: ''} 2024-02-01T12:00:08.609 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '', 1: '', 2: '1_1706784619.449972_360.1', 3: ''} 2024-02-01T12:00:08.610 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1706784927.652136_370.1 target=1_1706784619.449972_360.1 2024-02-01T12:00:08.610 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~297s 2024-02-01T12:00:08.610 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T12:00:08.610 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T12:00:08.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T12:00:08.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: ignoring --setuser ceph since I am not root 2024-02-01T12:00:08.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-02-01T12:00:08.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: 2024-02-01T10:55:43.310+0000 7f34c7484980 20 RGW-SYNC:meta: read sync status 2024-02-01T12:00:08.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: 2024-02-01T10:55:43.310+0000 7f34c7484980 20 RGW-SYNC:meta: read sync status 2024-02-01T12:00:08.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706784619.449972_360.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:50:19.449972Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:55:43Z"} 2024-02-01T12:00:08.610 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2e6c290-e7aa-4279-b77e-15403b18d61f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1706782453.132619_30.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-02-01T10:14:13.132619Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1706784619.449972_360.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:50:19.449972Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1706782595.670952_121.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-02-01T10:16:35.670952Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-02-01T10:55:43Z"} 2024-02-01T12:00:08.610 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1706782453.132619_30.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-02-01T10:14:13.132619Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1706784619.449972_360.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:50:19.449972Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1706782595.670952_121.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-02-01T10:16:35.670952Z', 'realm_epoch': 2}}] 2024-02-01T12:00:08.610 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '', 1: '', 2: '1_1706784927.652136_370.1', 3: ''} 2024-02-01T12:00:08.611 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '', 1: '', 2: '1_1706784619.449972_360.1', 3: ''} 2024-02-01T12:00:08.611 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1706784927.652136_370.1 target=1_1706784619.449972_360.1 2024-02-01T12:00:08.611 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~303s 2024-02-01T12:00:08.611 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: BARK! unmounting mounts and killing all daemons 2024-02-01T12:00:08.611 INFO:tasks.rgw_multisite_tests:tasks.ceph.c2.osd.0: INFO: Sent signal 15 2024-02-01T12:00:08.611 INFO:tasks.rgw_multisite_tests:tasks.ceph.c2.osd.1: INFO: Sent signal 15 2024-02-01T12:00:08.611 INFO:tasks.rgw_multisite_tests:tasks.ceph.c2.osd.2: INFO: Sent signal 15 2024-02-01T12:00:08.611 INFO:tasks.rgw_multisite_tests:tasks.ceph.c2.mon.a: INFO: Sent signal 15 2024-02-01T12:00:08.611 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.1: INFO: Sent signal 15 2024-02-01T12:00:08.611 INFO:tasks.rgw_multisite_tests:tasks.ceph.c2.mgr.x: INFO: Sent signal 15 2024-02-01T12:00:08.611 INFO:tasks.rgw_multisite_tests:tasks.ceph.c2.osd.0.smithi187.stderr: INFO: 2024-02-01T10:55:45.150+0000 7efcfacc9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster c2 -i 0 (PID: 22597) UID: 0 2024-02-01T12:00:08.611 INFO:tasks.rgw_multisite_tests:tasks.ceph.c2.osd.0.smithi187.stderr: INFO: 2024-02-01T10:55:45.150+0000 7efcfacc9640 -1 osd.0 52 *** Got signal Terminated *** 2024-02-01T12:00:08.611 INFO:tasks.rgw_multisite_tests:tasks.ceph.c2.osd.0.smithi187.stderr: INFO: 2024-02-01T10:55:45.150+0000 7efcfacc9640 -1 osd.0 52 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-01T12:00:08.612 INFO:tasks.rgw_multisite_tests:tasks.ceph.c2.osd.1.smithi187.stderr: INFO: 2024-02-01T10:55:45.150+0000 7fd237fa3640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster c2 -i 1 (PID: 22600) UID: 0 2024-02-01T12:00:08.612 INFO:tasks.rgw_multisite_tests:tasks.ceph.c2.osd.1.smithi187.stderr: INFO: 2024-02-01T10:55:45.150+0000 7fd237fa3640 -1 osd.1 52 *** Got signal Terminated *** 2024-02-01T12:00:08.612 INFO:tasks.rgw_multisite_tests:tasks.ceph.c2.osd.1.smithi187.stderr: INFO: 2024-02-01T10:55:45.150+0000 7fd237fa3640 -1 osd.1 52 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-01T12:00:08.612 INFO:tasks.rgw_multisite_tests:tasks.ceph.c2.osd.2.smithi187.stderr: INFO: 2024-02-01T10:55:45.150+0000 7f6bbac68640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster c2 -i 2 (PID: 22599) UID: 0 2024-02-01T12:00:08.612 INFO:tasks.rgw_multisite_tests:tasks.ceph.c2.osd.2.smithi187.stderr: INFO: 2024-02-01T10:55:45.150+0000 7f6bbac68640 -1 osd.2 52 *** Got signal Terminated *** 2024-02-01T12:00:08.612 INFO:tasks.rgw_multisite_tests:tasks.ceph.c2.osd.2.smithi187.stderr: INFO: 2024-02-01T10:55:45.150+0000 7f6bbac68640 -1 osd.2 52 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-01T12:00:08.612 INFO:tasks.rgw_multisite_tests:tasks.ceph.c2.mon.a.smithi187.stderr: INFO: 2024-02-01T10:55:45.150+0000 7f0efca01640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mon -f --cluster c2 -i a (PID: 22424) UID: 0 2024-02-01T12:00:08.612 INFO:tasks.rgw_multisite_tests:tasks.ceph.c2.mon.a.smithi187.stderr: INFO: 2024-02-01T10:55:45.150+0000 7f0efca01640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-02-01T12:00:08.612 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.1.smithi187.stdout: INFO: 2024-02-01T10:55:45.150+0000 7f340883c640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground (PID: 28963) UID: 0 2024-02-01T12:00:08.612 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.1.smithi187.stdout: INFO: 2024-02-01T10:55:45.150+0000 7f340c084a40 -1 shutting down 2024-02-01T12:00:08.612 INFO:tasks.rgw_multisite_tests:tasks.ceph.c2.mgr.x.smithi187.stderr: INFO: 2024-02-01T10:55:45.150+0000 7f8f82f7d640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mgr -f --cluster c2 -i x (PID: 22426) UID: 0 2024-02-01T12:00:08.612 INFO:tasks.rgw_multisite_tests:tasks.ceph.c2.mgr.x.smithi187.stderr: INFO: 2024-02-01T10:55:45.150+0000 7f8f82f7d640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-02-01T12:00:08.613 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T12:00:08.613 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T12:00:08.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T12:00:08.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: ignoring --setuser ceph since I am not root 2024-02-01T12:00:08.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-02-01T12:00:08.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: 2024-02-01T11:00:48.381+0000 7f04ec427980 0 monclient(hunting): authenticate timed out after 300 2024-02-01T12:00:08.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi187.stderr: INFO: failed to fetch mon config (--no-mon-config to skip) 2024-02-01T12:00:08.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 1 2024-02-01T12:00:08.613 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-02-01T12:00:08.613 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.613 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-02-01T12:00:08.614 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_creation_time 2024-02-01T12:00:08.614 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-02-01T12:00:08.614 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-02-01T12:00:08.614 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-02-01T12:00:08.614 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-02-01T12:00:08.614 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/../src/test/rgw/rgw_multi/tests.py", line 1506, in test_bucket_creation_time 2024-02-01T12:00:08.614 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-02-01T12:00:08.614 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/../src/test/rgw/rgw_multi/tests.py", line 205, in zonegroup_meta_checkpoint 2024-02-01T12:00:08.614 INFO:tasks.rgw_multisite_tests: master_status = meta_master_log_status(meta_master_zone) 2024-02-01T12:00:08.614 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/../src/test/rgw/rgw_multi/tests.py", line 149, in meta_master_log_status 2024-02-01T12:00:08.614 INFO:tasks.rgw_multisite_tests: mdlog_status_json, retcode = master_zone.cluster.admin(cmd, read_only=True) 2024-02-01T12:00:08.614 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/tasks/rgw_multisite.py", line 197, in admin 2024-02-01T12:00:08.614 INFO:tasks.rgw_multisite_tests: assert r == 0 2024-02-01T12:00:08.615 INFO:tasks.rgw_multisite_tests:AssertionError: 2024-02-01T12:00:08.615 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-02-01T12:00:08.615 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T12:00:08.615 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T12:00:08.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi148: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T12:00:08.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi148.stderr: INFO: ignoring --setuser ceph since I am not root 2024-02-01T12:00:08.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi148.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-02-01T12:00:08.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi148.stderr: INFO: 2024-02-01T11:10:02.057+0000 7f8657491980 0 monclient(hunting): authenticate timed out after 300 2024-02-01T12:00:08.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi148.stderr: INFO: failed to fetch mon config (--no-mon-config to skip) 2024-02-01T12:00:08.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 1 2024-02-01T12:00:08.615 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-02-01T12:00:08.615 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.615 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-02-01T12:00:08.616 INFO:tasks.rgw_multisite_tests:FAIL: test_sync_policy_config_zonegroup: 2024-02-01T12:00:08.616 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-02-01T12:00:08.616 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-02-01T12:00:08.616 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-02-01T12:00:08.616 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-02-01T12:00:08.616 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/../src/test/rgw/rgw_multi/tests.py", line 2090, in test_sync_policy_config_zonegroup 2024-02-01T12:00:08.616 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-02-01T12:00:08.616 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/../src/test/rgw/rgw_multi/tests.py", line 205, in zonegroup_meta_checkpoint 2024-02-01T12:00:08.616 INFO:tasks.rgw_multisite_tests: master_status = meta_master_log_status(meta_master_zone) 2024-02-01T12:00:08.616 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/../src/test/rgw/rgw_multi/tests.py", line 149, in meta_master_log_status 2024-02-01T12:00:08.616 INFO:tasks.rgw_multisite_tests: mdlog_status_json, retcode = master_zone.cluster.admin(cmd, read_only=True) 2024-02-01T12:00:08.617 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/tasks/rgw_multisite.py", line 197, in admin 2024-02-01T12:00:08.617 INFO:tasks.rgw_multisite_tests: assert r == 0 2024-02-01T12:00:08.617 INFO:tasks.rgw_multisite_tests:AssertionError: 2024-02-01T12:00:08.617 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-02-01T12:00:08.617 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T12:00:08.617 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T12:00:08.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi148: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T12:00:08.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi148.stderr: INFO: ignoring --setuser ceph since I am not root 2024-02-01T12:00:08.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi148.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-02-01T12:00:08.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi148.stderr: INFO: 2024-02-01T11:25:07.299+0000 7f22342b4980 0 monclient(hunting): authenticate timed out after 300 2024-02-01T12:00:08.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi148.stderr: INFO: failed to fetch mon config (--no-mon-config to skip) 2024-02-01T12:00:08.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 1 2024-02-01T12:00:08.617 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-02-01T12:00:08.618 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.618 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-02-01T12:00:08.618 INFO:tasks.rgw_multisite_tests:FAIL: test_sync_flow_symmetrical_zonegroup_all: 2024-02-01T12:00:08.618 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-02-01T12:00:08.618 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-02-01T12:00:08.618 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-02-01T12:00:08.618 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-02-01T12:00:08.618 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/../src/test/rgw/rgw_multi/tests.py", line 2162, in test_sync_flow_symmetrical_zonegroup_all 2024-02-01T12:00:08.618 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-02-01T12:00:08.618 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/../src/test/rgw/rgw_multi/tests.py", line 205, in zonegroup_meta_checkpoint 2024-02-01T12:00:08.618 INFO:tasks.rgw_multisite_tests: master_status = meta_master_log_status(meta_master_zone) 2024-02-01T12:00:08.618 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/../src/test/rgw/rgw_multi/tests.py", line 149, in meta_master_log_status 2024-02-01T12:00:08.619 INFO:tasks.rgw_multisite_tests: mdlog_status_json, retcode = master_zone.cluster.admin(cmd, read_only=True) 2024-02-01T12:00:08.619 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/tasks/rgw_multisite.py", line 197, in admin 2024-02-01T12:00:08.619 INFO:tasks.rgw_multisite_tests: assert r == 0 2024-02-01T12:00:08.619 INFO:tasks.rgw_multisite_tests:AssertionError: 2024-02-01T12:00:08.619 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-02-01T12:00:08.619 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T12:00:08.619 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T12:00:08.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi148: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T12:00:08.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi148.stderr: INFO: ignoring --setuser ceph since I am not root 2024-02-01T12:00:08.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi148.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-02-01T12:00:08.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi148.stderr: INFO: 2024-02-01T11:30:07.388+0000 7f9742cf1980 0 monclient(hunting): authenticate timed out after 300 2024-02-01T12:00:08.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi148.stderr: INFO: failed to fetch mon config (--no-mon-config to skip) 2024-02-01T12:00:08.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 1 2024-02-01T12:00:08.620 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-02-01T12:00:08.620 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.620 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-02-01T12:00:08.620 INFO:tasks.rgw_multisite_tests:FAIL: test_sync_flow_symmetrical_zonegroup_select: 2024-02-01T12:00:08.620 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-02-01T12:00:08.620 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-02-01T12:00:08.620 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-02-01T12:00:08.620 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-02-01T12:00:08.620 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/../src/test/rgw/rgw_multi/tests.py", line 2225, in test_sync_flow_symmetrical_zonegroup_select 2024-02-01T12:00:08.620 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-02-01T12:00:08.620 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/../src/test/rgw/rgw_multi/tests.py", line 205, in zonegroup_meta_checkpoint 2024-02-01T12:00:08.621 INFO:tasks.rgw_multisite_tests: master_status = meta_master_log_status(meta_master_zone) 2024-02-01T12:00:08.621 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/../src/test/rgw/rgw_multi/tests.py", line 149, in meta_master_log_status 2024-02-01T12:00:08.621 INFO:tasks.rgw_multisite_tests: mdlog_status_json, retcode = master_zone.cluster.admin(cmd, read_only=True) 2024-02-01T12:00:08.621 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/tasks/rgw_multisite.py", line 197, in admin 2024-02-01T12:00:08.621 INFO:tasks.rgw_multisite_tests: assert r == 0 2024-02-01T12:00:08.621 INFO:tasks.rgw_multisite_tests:AssertionError: 2024-02-01T12:00:08.621 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-02-01T12:00:08.621 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T12:00:08.621 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T12:00:08.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi148: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T12:00:08.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi148.stderr: INFO: ignoring --setuser ceph since I am not root 2024-02-01T12:00:08.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi148.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-02-01T12:00:08.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi148.stderr: INFO: 2024-02-01T11:35:07.477+0000 7f95e170a980 0 monclient(hunting): authenticate timed out after 300 2024-02-01T12:00:08.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi148.stderr: INFO: failed to fetch mon config (--no-mon-config to skip) 2024-02-01T12:00:08.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 1 2024-02-01T12:00:08.622 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-02-01T12:00:08.622 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.622 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-02-01T12:00:08.622 INFO:tasks.rgw_multisite_tests:FAIL: test_sync_flow_directional_zonegroup_select: 2024-02-01T12:00:08.622 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-02-01T12:00:08.622 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-02-01T12:00:08.622 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-02-01T12:00:08.622 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-02-01T12:00:08.622 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/../src/test/rgw/rgw_multi/tests.py", line 2295, in test_sync_flow_directional_zonegroup_select 2024-02-01T12:00:08.622 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-02-01T12:00:08.623 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/../src/test/rgw/rgw_multi/tests.py", line 205, in zonegroup_meta_checkpoint 2024-02-01T12:00:08.623 INFO:tasks.rgw_multisite_tests: master_status = meta_master_log_status(meta_master_zone) 2024-02-01T12:00:08.623 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/../src/test/rgw/rgw_multi/tests.py", line 149, in meta_master_log_status 2024-02-01T12:00:08.623 INFO:tasks.rgw_multisite_tests: mdlog_status_json, retcode = master_zone.cluster.admin(cmd, read_only=True) 2024-02-01T12:00:08.623 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/tasks/rgw_multisite.py", line 197, in admin 2024-02-01T12:00:08.623 INFO:tasks.rgw_multisite_tests: assert r == 0 2024-02-01T12:00:08.623 INFO:tasks.rgw_multisite_tests:AssertionError: 2024-02-01T12:00:08.623 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-02-01T12:00:08.623 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T12:00:08.623 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T12:00:08.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi148: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T12:00:08.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi148.stderr: INFO: ignoring --setuser ceph since I am not root 2024-02-01T12:00:08.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi148.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-02-01T12:00:08.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi148.stderr: INFO: 2024-02-01T11:40:07.566+0000 7fa12f3bb980 0 monclient(hunting): authenticate timed out after 300 2024-02-01T12:00:08.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi148.stderr: INFO: failed to fetch mon config (--no-mon-config to skip) 2024-02-01T12:00:08.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 1 2024-02-01T12:00:08.624 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-02-01T12:00:08.624 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.624 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-02-01T12:00:08.624 INFO:tasks.rgw_multisite_tests:FAIL: test_sync_single_bucket: 2024-02-01T12:00:08.624 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-02-01T12:00:08.624 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-02-01T12:00:08.624 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-02-01T12:00:08.624 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-02-01T12:00:08.625 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/../src/test/rgw/rgw_multi/tests.py", line 2408, in test_sync_single_bucket 2024-02-01T12:00:08.625 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-02-01T12:00:08.625 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/../src/test/rgw/rgw_multi/tests.py", line 205, in zonegroup_meta_checkpoint 2024-02-01T12:00:08.625 INFO:tasks.rgw_multisite_tests: master_status = meta_master_log_status(meta_master_zone) 2024-02-01T12:00:08.625 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/../src/test/rgw/rgw_multi/tests.py", line 149, in meta_master_log_status 2024-02-01T12:00:08.625 INFO:tasks.rgw_multisite_tests: mdlog_status_json, retcode = master_zone.cluster.admin(cmd, read_only=True) 2024-02-01T12:00:08.625 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/tasks/rgw_multisite.py", line 197, in admin 2024-02-01T12:00:08.625 INFO:tasks.rgw_multisite_tests: assert r == 0 2024-02-01T12:00:08.625 INFO:tasks.rgw_multisite_tests:AssertionError: 2024-02-01T12:00:08.625 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-02-01T12:00:08.625 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T12:00:08.625 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T12:00:08.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi148: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T12:00:08.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi148.stderr: INFO: ignoring --setuser ceph since I am not root 2024-02-01T12:00:08.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi148.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-02-01T12:00:08.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi148.stderr: INFO: 2024-02-01T11:45:08.135+0000 7fcabd39d980 0 monclient(hunting): authenticate timed out after 300 2024-02-01T12:00:08.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi148.stderr: INFO: failed to fetch mon config (--no-mon-config to skip) 2024-02-01T12:00:08.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 1 2024-02-01T12:00:08.626 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-02-01T12:00:08.626 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.626 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-02-01T12:00:08.626 INFO:tasks.rgw_multisite_tests:FAIL: test_sync_different_buckets: 2024-02-01T12:00:08.626 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-02-01T12:00:08.626 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-02-01T12:00:08.626 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-02-01T12:00:08.627 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-02-01T12:00:08.627 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/../src/test/rgw/rgw_multi/tests.py", line 2525, in test_sync_different_buckets 2024-02-01T12:00:08.627 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-02-01T12:00:08.627 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/../src/test/rgw/rgw_multi/tests.py", line 205, in zonegroup_meta_checkpoint 2024-02-01T12:00:08.627 INFO:tasks.rgw_multisite_tests: master_status = meta_master_log_status(meta_master_zone) 2024-02-01T12:00:08.627 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/../src/test/rgw/rgw_multi/tests.py", line 149, in meta_master_log_status 2024-02-01T12:00:08.627 INFO:tasks.rgw_multisite_tests: mdlog_status_json, retcode = master_zone.cluster.admin(cmd, read_only=True) 2024-02-01T12:00:08.627 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/tasks/rgw_multisite.py", line 197, in admin 2024-02-01T12:00:08.627 INFO:tasks.rgw_multisite_tests: assert r == 0 2024-02-01T12:00:08.627 INFO:tasks.rgw_multisite_tests:AssertionError: 2024-02-01T12:00:08.627 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-02-01T12:00:08.627 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T12:00:08.627 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T12:00:08.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi148: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T12:00:08.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi148.stderr: INFO: ignoring --setuser ceph since I am not root 2024-02-01T12:00:08.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi148.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-02-01T12:00:08.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi148.stderr: INFO: 2024-02-01T11:50:08.220+0000 7fd34d050980 0 monclient(hunting): authenticate timed out after 300 2024-02-01T12:00:08.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi148.stderr: INFO: failed to fetch mon config (--no-mon-config to skip) 2024-02-01T12:00:08.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 1 2024-02-01T12:00:08.628 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-02-01T12:00:08.628 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.628 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-02-01T12:00:08.628 INFO:tasks.rgw_multisite_tests:FAIL: test_sync_multiple_buckets_to_single: 2024-02-01T12:00:08.628 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-02-01T12:00:08.629 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-02-01T12:00:08.629 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-02-01T12:00:08.629 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-02-01T12:00:08.629 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/../src/test/rgw/rgw_multi/tests.py", line 2671, in test_sync_multiple_buckets_to_single 2024-02-01T12:00:08.629 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-02-01T12:00:08.629 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/../src/test/rgw/rgw_multi/tests.py", line 205, in zonegroup_meta_checkpoint 2024-02-01T12:00:08.629 INFO:tasks.rgw_multisite_tests: master_status = meta_master_log_status(meta_master_zone) 2024-02-01T12:00:08.629 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/../src/test/rgw/rgw_multi/tests.py", line 149, in meta_master_log_status 2024-02-01T12:00:08.629 INFO:tasks.rgw_multisite_tests: mdlog_status_json, retcode = master_zone.cluster.admin(cmd, read_only=True) 2024-02-01T12:00:08.629 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/tasks/rgw_multisite.py", line 197, in admin 2024-02-01T12:00:08.629 INFO:tasks.rgw_multisite_tests: assert r == 0 2024-02-01T12:00:08.629 INFO:tasks.rgw_multisite_tests:AssertionError: 2024-02-01T12:00:08.629 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-02-01T12:00:08.629 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T12:00:08.630 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T12:00:08.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi148: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T12:00:08.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi148.stderr: INFO: ignoring --setuser ceph since I am not root 2024-02-01T12:00:08.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi148.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-02-01T12:00:08.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi148.stderr: INFO: 2024-02-01T11:55:08.309+0000 7fef7cbdb980 0 monclient(hunting): authenticate timed out after 300 2024-02-01T12:00:08.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi148.stderr: INFO: failed to fetch mon config (--no-mon-config to skip) 2024-02-01T12:00:08.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 1 2024-02-01T12:00:08.630 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-02-01T12:00:08.630 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.630 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-02-01T12:00:08.630 INFO:tasks.rgw_multisite_tests:FAIL: test_sync_single_bucket_to_multiple: 2024-02-01T12:00:08.630 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-02-01T12:00:08.631 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-02-01T12:00:08.631 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-02-01T12:00:08.631 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-02-01T12:00:08.631 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/../src/test/rgw/rgw_multi/tests.py", line 2792, in test_sync_single_bucket_to_multiple 2024-02-01T12:00:08.631 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-02-01T12:00:08.631 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/../src/test/rgw/rgw_multi/tests.py", line 205, in zonegroup_meta_checkpoint 2024-02-01T12:00:08.631 INFO:tasks.rgw_multisite_tests: master_status = meta_master_log_status(meta_master_zone) 2024-02-01T12:00:08.631 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/../src/test/rgw/rgw_multi/tests.py", line 149, in meta_master_log_status 2024-02-01T12:00:08.631 INFO:tasks.rgw_multisite_tests: mdlog_status_json, retcode = master_zone.cluster.admin(cmd, read_only=True) 2024-02-01T12:00:08.631 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/tasks/rgw_multisite.py", line 197, in admin 2024-02-01T12:00:08.631 INFO:tasks.rgw_multisite_tests: assert r == 0 2024-02-01T12:00:08.631 INFO:tasks.rgw_multisite_tests:AssertionError: 2024-02-01T12:00:08.631 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-02-01T12:00:08.632 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T12:00:08.632 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-02-01T12:00:08.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi148: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-02-01T12:00:08.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi148.stderr: INFO: ignoring --setuser ceph since I am not root 2024-02-01T12:00:08.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi148.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-02-01T12:00:08.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi148.stderr: INFO: 2024-02-01T12:00:08.399+0000 7f58a9a5d980 0 monclient(hunting): authenticate timed out after 300 2024-02-01T12:00:08.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi148.stderr: INFO: failed to fetch mon config (--no-mon-config to skip) 2024-02-01T12:00:08.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 1 2024-02-01T12:00:08.632 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-02-01T12:00:08.632 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.632 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-02-01T12:00:08.632 INFO:tasks.rgw_multisite_tests:Ran 62 tests in 6236.215s 2024-02-01T12:00:08.633 INFO:tasks.rgw_multisite_tests: 2024-02-01T12:00:08.633 INFO:tasks.rgw_multisite_tests:FAILED (SKIP=15, errors=13, failures=11) 2024-02-01T12:00:08.633 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/tasks/rgw_multisite_tests.py", line 95, in begin assert nose.run(argv=argv, config=conf), 'rgw multisite test failures' AssertionError: rgw multisite test failures 2024-02-01T12:00:12.643 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=c5305a7f2be442c6bc6819c93ab98551 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/tasks/rgw_multisite_tests.py", line 95, in begin assert nose.run(argv=argv, config=conf), 'rgw multisite test failures' AssertionError: rgw multisite test failures 2024-02-01T12:00:12.646 DEBUG:teuthology.run_tasks:Unwinding manager rgw-multisite-tests 2024-02-01T12:00:12.657 DEBUG:teuthology.run_tasks:Unwinding manager rgw-multisite 2024-02-01T12:00:12.667 DEBUG:teuthology.run_tasks:Unwinding manager rgw 2024-02-01T12:00:12.678 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2024-02-01T12:00:12.678 INFO:teuthology.orchestra.run:waiting for 300 2024-02-01T12:00:12.680 INFO:tasks.rgw.c1.client.0:Stopped 2024-02-01T12:00:12.680 DEBUG:teuthology.orchestra.run.smithi148:> rm -f /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock 2024-02-01T12:00:12.694 DEBUG:teuthology.orchestra.run.smithi148:> sudo rm -f /etc/ceph/vault-root-token 2024-02-01T12:00:12.749 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['gc', 'process', '--include-all'] 2024-02-01T12:00:12.750 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'gc', 'process', '--include-all'] 2024-02-01T12:00:12.750 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 gc process --include-all 2024-02-01T12:00:12.829 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setuser ceph since I am not root 2024-02-01T12:00:12.829 INFO:teuthology.orchestra.run.smithi148.stderr:ignoring --setgroup ceph since I am not root 2024-02-01T12:05:12.831 INFO:teuthology.orchestra.run.smithi148.stderr:2024-02-01T12:05:12.832+0000 7f597fb56980 0 monclient(hunting): authenticate timed out after 300 2024-02-01T12:05:12.832 INFO:teuthology.orchestra.run.smithi148.stderr:failed to fetch mon config (--no-mon-config to skip) 2024-02-01T12:05:12.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-01T12:05:12.835 ERROR:teuthology.run_tasks:Manager failed: rgw Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/tasks/rgw.py", line 473, in task File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/tasks/rgw.py", line 300, in create_pools File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/tasks/rgw.py", line 242, in start_rgw File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/tasks/util/rgw.py", line 34, in rgwadmin File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi148 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 gc process --include-all' 2024-02-01T12:05:12.851 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-02-01T12:05:12.865 INFO:tasks.ceph.ceph_manager.c2:waiting for clean 2024-02-01T12:05:12.865 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2024-02-01T12:07:12.894 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-02-01T12:07:12.896 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/tasks/ceph.py", line 1918, in task File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/tasks/rgw.py", line 473, in task File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/tasks/rgw.py", line 300, in create_pools File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/tasks/rgw.py", line 242, in start_rgw File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/tasks/util/rgw.py", line 34, in rgwadmin File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi148 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 gc process --include-all' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/tasks/ceph.py", line 1928, in task File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/tasks/ceph_manager.py", line 2774, in wait_for_clean File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/tasks/ceph_manager.py", line 2606, in get_num_active_clean File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/tasks/ceph_manager.py", line 2372, in get_pg_stats File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/tasks/ceph_manager.py", line 1627, in raw_cluster_cmd File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/tasks/ceph_manager.py", line 1618, in run_cluster_cmd File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi187 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json' 2024-02-01T12:07:12.910 INFO:teuthology.misc:Shutting down mds daemons... 2024-02-01T12:07:12.910 INFO:teuthology.misc:Shutting down osd daemons... 2024-02-01T12:07:12.910 DEBUG:tasks.ceph.c2.osd.0:waiting for process to exit 2024-02-01T12:07:12.910 INFO:teuthology.orchestra.run:waiting for 300 2024-02-01T12:07:12.914 INFO:tasks.ceph.c2.osd.0:Stopped 2024-02-01T12:07:12.914 DEBUG:tasks.ceph.c2.osd.1:waiting for process to exit 2024-02-01T12:07:12.914 INFO:teuthology.orchestra.run:waiting for 300 2024-02-01T12:07:12.915 INFO:tasks.ceph.c2.osd.1:Stopped 2024-02-01T12:07:12.915 DEBUG:tasks.ceph.c2.osd.2:waiting for process to exit 2024-02-01T12:07:12.915 INFO:teuthology.orchestra.run:waiting for 300 2024-02-01T12:07:12.915 INFO:tasks.ceph.c2.osd.2:Stopped 2024-02-01T12:07:12.916 INFO:teuthology.misc:Shutting down mgr daemons... 2024-02-01T12:07:12.916 DEBUG:tasks.ceph.c2.mgr.x:waiting for process to exit 2024-02-01T12:07:12.916 INFO:teuthology.orchestra.run:waiting for 300 2024-02-01T12:07:12.916 INFO:tasks.ceph.c2.mgr.x:Stopped 2024-02-01T12:07:12.916 INFO:teuthology.misc:Shutting down mon daemons... 2024-02-01T12:07:12.916 DEBUG:tasks.ceph.c2.mon.a:waiting for process to exit 2024-02-01T12:07:12.917 INFO:teuthology.orchestra.run:waiting for 300 2024-02-01T12:07:12.917 INFO:tasks.ceph.c2.mon.a:Stopped 2024-02-01T12:07:12.917 INFO:tasks.ceph:Checking cluster log for badness... 2024-02-01T12:07:12.917 DEBUG:teuthology.orchestra.run.smithi187:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/c2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'not have an application enabled' | head -n 1 2024-02-01T12:07:12.935 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-0 on ubuntu@smithi187.front.sepia.ceph.com 2024-02-01T12:07:12.935 DEBUG:teuthology.orchestra.run.smithi187:> sync && sudo umount -f /var/lib/ceph/osd/c2-0 2024-02-01T12:07:13.410 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-1 on ubuntu@smithi187.front.sepia.ceph.com 2024-02-01T12:07:13.411 DEBUG:teuthology.orchestra.run.smithi187:> sync && sudo umount -f /var/lib/ceph/osd/c2-1 2024-02-01T12:07:13.600 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-2 on ubuntu@smithi187.front.sepia.ceph.com 2024-02-01T12:07:13.600 DEBUG:teuthology.orchestra.run.smithi187:> sync && sudo umount -f /var/lib/ceph/osd/c2-2 2024-02-01T12:07:13.759 INFO:tasks.ceph:Archiving mon data... 2024-02-01T12:07:13.760 DEBUG:teuthology.misc:Transferring archived files from smithi187:/var/lib/ceph/mon/c2-a to /home/teuthworker/archive/yuvalif-2024-01-30_08:46:48-rgw-wip-end2end-tracing-distro-default-smithi/7537587/data/c2.mon.a.tgz 2024-02-01T12:07:13.760 DEBUG:teuthology.orchestra.run.smithi187:> mktemp 2024-02-01T12:07:13.766 INFO:teuthology.orchestra.run.smithi187.stdout:/tmp/tmp.b9QN2DsxuC 2024-02-01T12:07:13.766 DEBUG:teuthology.orchestra.run.smithi187:> sudo tar cz -f - -C /var/lib/ceph/mon/c2-a -- . > /tmp/tmp.b9QN2DsxuC 2024-02-01T12:07:13.902 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod 0666 /tmp/tmp.b9QN2DsxuC 2024-02-01T12:07:13.963 DEBUG:teuthology.orchestra.remote:smithi187:/tmp/tmp.b9QN2DsxuC is 305KB 2024-02-01T12:07:14.021 DEBUG:teuthology.orchestra.run.smithi187:> rm -fr /tmp/tmp.b9QN2DsxuC 2024-02-01T12:07:14.027 INFO:tasks.ceph:Cleaning ceph cluster... 2024-02-01T12:07:14.027 DEBUG:teuthology.orchestra.run.smithi148:> sudo rm -rf -- /etc/ceph/c2.conf /etc/ceph/c2.keyring /home/ubuntu/cephtest/c2.data /home/ubuntu/cephtest/c2.monmap /home/ubuntu/cephtest/../*.pid 2024-02-01T12:07:14.030 DEBUG:teuthology.orchestra.run.smithi187:> sudo rm -rf -- /etc/ceph/c2.conf /etc/ceph/c2.keyring /home/ubuntu/cephtest/c2.data /home/ubuntu/cephtest/c2.monmap /home/ubuntu/cephtest/../*.pid 2024-02-01T12:07:14.084 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/tasks/ceph.py", line 1918, in task File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/tasks/rgw.py", line 473, in task File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/tasks/rgw.py", line 300, in create_pools File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/tasks/rgw.py", line 242, in start_rgw File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/tasks/util/rgw.py", line 34, in rgwadmin File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi148 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 gc process --include-all' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/tasks/ceph.py", line 1928, in task File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/tasks/ceph_manager.py", line 2774, in wait_for_clean File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/tasks/ceph_manager.py", line 2606, in get_num_active_clean File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/tasks/ceph_manager.py", line 2372, in get_pg_stats File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/tasks/ceph_manager.py", line 1627, in raw_cluster_cmd File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/tasks/ceph_manager.py", line 1618, in run_cluster_cmd File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi187 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json' 2024-02-01T12:07:14.089 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-02-01T12:07:14.098 INFO:tasks.ceph.ceph_manager.c1:waiting for clean 2024-02-01T12:07:14.099 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2024-02-01T12:09:14.117 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-02-01T12:09:14.118 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/tasks/ceph.py", line 1918, in task File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/tasks/rgw.py", line 473, in task File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/tasks/rgw.py", line 300, in create_pools File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/tasks/rgw.py", line 242, in start_rgw File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/tasks/util/rgw.py", line 34, in rgwadmin File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi148 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 gc process --include-all' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/tasks/ceph.py", line 1928, in task File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/tasks/ceph_manager.py", line 2774, in wait_for_clean File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/tasks/ceph_manager.py", line 2606, in get_num_active_clean File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/tasks/ceph_manager.py", line 2372, in get_pg_stats File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/tasks/ceph_manager.py", line 1627, in raw_cluster_cmd File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/tasks/ceph_manager.py", line 1618, in run_cluster_cmd File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi148 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json' 2024-02-01T12:09:14.133 INFO:teuthology.misc:Shutting down mds daemons... 2024-02-01T12:09:14.133 INFO:teuthology.misc:Shutting down osd daemons... 2024-02-01T12:09:14.133 DEBUG:tasks.ceph.c1.osd.0:waiting for process to exit 2024-02-01T12:09:14.133 INFO:teuthology.orchestra.run:waiting for 300 2024-02-01T12:09:14.137 INFO:tasks.ceph.c1.osd.0:Stopped 2024-02-01T12:09:14.138 DEBUG:tasks.ceph.c1.osd.1:waiting for process to exit 2024-02-01T12:09:14.138 INFO:teuthology.orchestra.run:waiting for 300 2024-02-01T12:09:14.138 INFO:tasks.ceph.c1.osd.1:Stopped 2024-02-01T12:09:14.138 DEBUG:tasks.ceph.c1.osd.2:waiting for process to exit 2024-02-01T12:09:14.138 INFO:teuthology.orchestra.run:waiting for 300 2024-02-01T12:09:14.139 INFO:tasks.ceph.c1.osd.2:Stopped 2024-02-01T12:09:14.139 INFO:teuthology.misc:Shutting down mgr daemons... 2024-02-01T12:09:14.139 DEBUG:tasks.ceph.c1.mgr.x:waiting for process to exit 2024-02-01T12:09:14.139 INFO:teuthology.orchestra.run:waiting for 300 2024-02-01T12:09:14.140 INFO:tasks.ceph.c1.mgr.x:Stopped 2024-02-01T12:09:14.140 INFO:teuthology.misc:Shutting down mon daemons... 2024-02-01T12:09:14.140 DEBUG:tasks.ceph.c1.mon.a:waiting for process to exit 2024-02-01T12:09:14.140 INFO:teuthology.orchestra.run:waiting for 300 2024-02-01T12:09:14.140 INFO:tasks.ceph.c1.mon.a:Stopped 2024-02-01T12:09:14.141 INFO:tasks.ceph:Checking cluster log for badness... 2024-02-01T12:09:14.141 DEBUG:teuthology.orchestra.run.smithi148:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/c1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'not have an application enabled' | head -n 1 2024-02-01T12:09:14.167 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-0 on ubuntu@smithi148.front.sepia.ceph.com 2024-02-01T12:09:14.168 DEBUG:teuthology.orchestra.run.smithi148:> sync && sudo umount -f /var/lib/ceph/osd/c1-0 2024-02-01T12:09:14.336 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-1 on ubuntu@smithi148.front.sepia.ceph.com 2024-02-01T12:09:14.336 DEBUG:teuthology.orchestra.run.smithi148:> sync && sudo umount -f /var/lib/ceph/osd/c1-1 2024-02-01T12:09:14.516 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-2 on ubuntu@smithi148.front.sepia.ceph.com 2024-02-01T12:09:14.516 DEBUG:teuthology.orchestra.run.smithi148:> sync && sudo umount -f /var/lib/ceph/osd/c1-2 2024-02-01T12:09:14.707 INFO:tasks.ceph:Archiving mon data... 2024-02-01T12:09:14.708 DEBUG:teuthology.misc:Transferring archived files from smithi148:/var/lib/ceph/mon/c1-a to /home/teuthworker/archive/yuvalif-2024-01-30_08:46:48-rgw-wip-end2end-tracing-distro-default-smithi/7537587/data/c1.mon.a.tgz 2024-02-01T12:09:14.708 DEBUG:teuthology.orchestra.run.smithi148:> mktemp 2024-02-01T12:09:14.713 INFO:teuthology.orchestra.run.smithi148.stdout:/tmp/tmp.1TvKhU7V58 2024-02-01T12:09:14.714 DEBUG:teuthology.orchestra.run.smithi148:> sudo tar cz -f - -C /var/lib/ceph/mon/c1-a -- . > /tmp/tmp.1TvKhU7V58 2024-02-01T12:09:14.858 DEBUG:teuthology.orchestra.run.smithi148:> sudo chmod 0666 /tmp/tmp.1TvKhU7V58 2024-02-01T12:09:14.922 DEBUG:teuthology.orchestra.remote:smithi148:/tmp/tmp.1TvKhU7V58 is 361KB 2024-02-01T12:09:14.984 DEBUG:teuthology.orchestra.run.smithi148:> rm -fr /tmp/tmp.1TvKhU7V58 2024-02-01T12:09:14.990 INFO:tasks.ceph:Cleaning ceph cluster... 2024-02-01T12:09:14.990 DEBUG:teuthology.orchestra.run.smithi148:> sudo rm -rf -- /etc/ceph/c1.conf /etc/ceph/c1.keyring /home/ubuntu/cephtest/c1.data /home/ubuntu/cephtest/c1.monmap /home/ubuntu/cephtest/../*.pid 2024-02-01T12:09:15.035 DEBUG:teuthology.orchestra.run.smithi187:> sudo rm -rf -- /etc/ceph/c1.conf /etc/ceph/c1.keyring /home/ubuntu/cephtest/c1.data /home/ubuntu/cephtest/c1.monmap /home/ubuntu/cephtest/../*.pid 2024-02-01T12:09:15.095 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-02-01T12:09:15.100 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-02-01T12:09:15.100 INFO:tasks.ceph:Archiving crash dumps... 2024-02-01T12:09:15.102 DEBUG:teuthology.misc:Transferring archived files from smithi148:/var/lib/ceph/crash to /home/teuthworker/archive/yuvalif-2024-01-30_08:46:48-rgw-wip-end2end-tracing-distro-default-smithi/7537587/remote/smithi148/crash 2024-02-01T12:09:15.103 DEBUG:teuthology.orchestra.run.smithi148:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-02-01T12:09:15.151 DEBUG:teuthology.misc:Transferring archived files from smithi187:/var/lib/ceph/crash to /home/teuthworker/archive/yuvalif-2024-01-30_08:46:48-rgw-wip-end2end-tracing-distro-default-smithi/7537587/remote/smithi187/crash 2024-02-01T12:09:15.153 DEBUG:teuthology.orchestra.run.smithi187:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-02-01T12:09:15.166 INFO:tasks.ceph:Compressing logs... 2024-02-01T12:09:15.166 DEBUG:teuthology.orchestra.run.smithi148:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-01T12:09:15.200 DEBUG:teuthology.orchestra.run.smithi187:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-01T12:09:15.220 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.146689.log 2024-02-01T12:09:15.220 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.admin.26233.log 2024-02-01T12:09:15.226 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.41495.log 2024-02-01T12:09:15.226 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.146689.log: /var/log/ceph/c1-client.admin.26233.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.146689.log.gz 2024-02-01T12:09:15.226 INFO:teuthology.orchestra.run.smithi148.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.26233.log.gz 2024-02-01T12:09:15.226 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.51573.log 2024-02-01T12:09:15.226 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.41495.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.22501.log 2024-02-01T12:09:15.226 INFO:teuthology.orchestra.run.smithi148.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.41495.log.gz 2024-02-01T12:09:15.227 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.51573.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.135345.log 2024-02-01T12:09:15.227 INFO:teuthology.orchestra.run.smithi148.stderr: 47.4% -- replaced with /var/log/ceph/c1-client.0.51573.log.gz 2024-02-01T12:09:15.227 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.admin.22501.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.38526.log 2024-02-01T12:09:15.227 INFO:teuthology.orchestra.run.smithi148.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.22501.log.gz 2024-02-01T12:09:15.227 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.135345.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.135345.log.gz 2024-02-01T12:09:15.227 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.40311.log 2024-02-01T12:09:15.228 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.38526.log: 47.4% -- replaced with /var/log/ceph/c1-client.0.38526.log.gz 2024-02-01T12:09:15.228 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.182211.log 2024-02-01T12:09:15.228 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.40311.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.135940.log 2024-02-01T12:09:15.228 INFO:teuthology.orchestra.run.smithi148.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.40311.log.gz 2024-02-01T12:09:15.228 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.182211.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.106082.log 2024-02-01T12:09:15.228 INFO:teuthology.orchestra.run.smithi148.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.182211.log.gz 2024-02-01T12:09:15.229 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.135940.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.135940.log.gz 2024-02-01T12:09:15.229 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.62829.log 2024-02-01T12:09:15.229 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.106082.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.31063.log 2024-02-01T12:09:15.229 INFO:teuthology.orchestra.run.smithi148.stderr: 47.4% -- replaced with /var/log/ceph/c1-client.0.106082.log.gz 2024-02-01T12:09:15.230 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.62829.log: gzip -5 --verbose -- 87.7% -- replaced with /var/log/ceph/c1-client.0.62829.log.gz /var/log/ceph/c1-client.0.86556.log 2024-02-01T12:09:15.230 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-02-01T12:09:15.247 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.31063.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.112583.log 2024-02-01T12:09:15.247 INFO:teuthology.orchestra.run.smithi148.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.31063.log.gz 2024-02-01T12:09:15.248 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.86556.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.151225.log 2024-02-01T12:09:15.248 INFO:teuthology.orchestra.run.smithi148.stderr: 50.4% -- replaced with /var/log/ceph/c1-client.0.86556.log.gz 2024-02-01T12:09:15.248 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.112583.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.199353.log 2024-02-01T12:09:15.248 INFO:teuthology.orchestra.run.smithi148.stderr: 46.7% -- replaced with /var/log/ceph/c1-client.0.112583.log.gz 2024-02-01T12:09:15.248 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.151225.log: 26.6% -- replaced with /var/log/ceph/c1-client.0.151225.log.gz 2024-02-01T12:09:15.249 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.1.28009.log 2024-02-01T12:09:15.249 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.199353.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.58087.log 2024-02-01T12:09:15.249 INFO:teuthology.orchestra.run.smithi148.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.199353.log.gz 2024-02-01T12:09:15.249 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.1.28009.log: gzip -5 --verbose -- /var/log/ceph/c1-mgr.x.log 2024-02-01T12:09:15.249 INFO:teuthology.orchestra.run.smithi148.stderr: 83.3% -- replaced with /var/log/ceph/c1-client.1.28009.log.gz 2024-02-01T12:09:15.249 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.58087.log: 64.3% -- replaced with /var/log/ceph/c1-client.0.58087.log.gz 2024-02-01T12:09:15.250 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.91881.log 2024-02-01T12:09:15.250 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.26591.log 2024-02-01T12:09:15.250 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.91881.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.214209.log 46.3% 2024-02-01T12:09:15.250 INFO:teuthology.orchestra.run.smithi148.stderr: -- replaced with /var/log/ceph/c1-client.0.91881.log.gz 2024-02-01T12:09:15.250 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.admin.26591.log: 0.0% -- replaced with /var/log/ceph/c1-client.admin.26591.log.gz 2024-02-01T12:09:15.250 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.152473.log 2024-02-01T12:09:15.251 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.197577.log 2024-02-01T12:09:15.251 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.214209.log: /var/log/ceph/c1-client.0.152473.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.152473.log.gz 2024-02-01T12:09:15.251 INFO:teuthology.orchestra.run.smithi148.stderr: 8.2% -- replaced with /var/log/ceph/c1-client.0.214209.log.gz 2024-02-01T12:09:15.251 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.48616.log 2024-02-01T12:09:15.251 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.197577.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.96027.log 2024-02-01T12:09:15.251 INFO:teuthology.orchestra.run.smithi148.stderr: 45.5% -- replaced with /var/log/ceph/c1-client.0.197577.log.gz 2024-02-01T12:09:15.252 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.48616.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.26804.log 2024-02-01T12:09:15.252 INFO:teuthology.orchestra.run.smithi148.stderr: 56.6% -- replaced with /var/log/ceph/c1-client.0.48616.log.gz 2024-02-01T12:09:15.252 INFO:teuthology.orchestra.run.smithi148.stderr:gzip/var/log/ceph/c1-client.0.96027.log: -5 --verbose -- /var/log/ceph/c1-client.0.137117.log 2024-02-01T12:09:15.252 INFO:teuthology.orchestra.run.smithi148.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.96027.log.gz 2024-02-01T12:09:15.252 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.admin.26804.log: 0.0% -- replaced with /var/log/ceph/c1-client.admin.26804.log.gz 2024-02-01T12:09:15.252 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.45044.log 2024-02-01T12:09:15.253 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.137117.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.99581.log 2024-02-01T12:09:15.253 INFO:teuthology.orchestra.run.smithi148.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.137117.log.gz 2024-02-01T12:09:15.253 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.45044.log: gzip 0.0% -- replaced with /var/log/ceph/c1-client.0.45044.log.gz 2024-02-01T12:09:15.253 INFO:teuthology.orchestra.run.smithi148.stderr: -5 --verbose -- /var/log/ceph/c1-client.0.160337.log 2024-02-01T12:09:15.253 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.114358.log 2024-02-01T12:09:15.253 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.99581.log: /var/log/ceph/c1-client.0.160337.log: 46.7% -- replaced with /var/log/ceph/c1-client.0.99581.log.gz 2024-02-01T12:09:15.253 INFO:teuthology.orchestra.run.smithi148.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.160337.log.gz 2024-02-01T12:09:15.254 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.110218.log 2024-02-01T12:09:15.254 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.114358.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.130598.log 2024-02-01T12:09:15.254 INFO:teuthology.orchestra.run.smithi148.stderr: 47.4% -- replaced with /var/log/ceph/c1-client.0.114358.log.gz 2024-02-01T12:09:15.254 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.110218.log: gzip -5 --verbose 46.3% -- -- replaced with /var/log/ceph/c1-client.0.110218.log.gz 2024-02-01T12:09:15.254 INFO:teuthology.orchestra.run.smithi148.stderr: /var/log/ceph/c1-client.0.162111.log 2024-02-01T12:09:15.254 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.62240.log/var/log/ceph/c1-client.0.130598.log: 2024-02-01T12:09:15.255 INFO:teuthology.orchestra.run.smithi148.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.130598.log.gz 2024-02-01T12:09:15.255 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.162111.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.162111.log.gz 2024-02-01T12:09:15.255 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.214164.log 2024-02-01T12:09:15.255 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.62240.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.195217.log 2024-02-01T12:09:15.255 INFO:teuthology.orchestra.run.smithi148.stderr: 53.6% -- replaced with /var/log/ceph/c1-client.0.62240.log.gz 2024-02-01T12:09:15.255 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.214164.log: gzip -5 --verbose -- /var/log/ceph/c1-osd.2.log 2024-02-01T12:09:15.255 INFO:teuthology.orchestra.run.smithi148.stderr: 9.2% -- replaced with /var/log/ceph/c1-client.0.214164.log.gz 2024-02-01T12:09:15.256 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose --/var/log/ceph/c1-client.0.195217.log: /var/log/ceph/c1-client.0.163887.log 2024-02-01T12:09:15.256 INFO:teuthology.orchestra.run.smithi148.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.195217.log.gz 2024-02-01T12:09:15.256 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-osd.2.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.26235.log 2024-02-01T12:09:15.256 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.163887.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.26305.log 2024-02-01T12:09:15.256 INFO:teuthology.orchestra.run.smithi148.stderr: 47.4% -- replaced with /var/log/ceph/c1-client.0.163887.log.gz 2024-02-01T12:09:15.256 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.admin.26235.log: 0.0% -- replaced with /var/log/ceph/c1-client.admin.26235.log.gz 2024-02-01T12:09:15.257 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.153232.log 2024-02-01T12:09:15.257 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.admin.26305.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.186352.log 2024-02-01T12:09:15.257 INFO:teuthology.orchestra.run.smithi148.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.26305.log.gz 2024-02-01T12:09:15.257 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.153232.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.49800.log 2024-02-01T12:09:15.257 INFO:teuthology.orchestra.run.smithi148.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.153232.log.gz 2024-02-01T12:09:15.257 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.186352.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.186352.log.gz 2024-02-01T12:09:15.258 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.69926.log 2024-02-01T12:09:15.258 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.49800.log: gzip -5 --verbose -- /var/log/ceph/rgw.c1.client.1.log 2024-02-01T12:09:15.258 INFO:teuthology.orchestra.run.smithi148.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.49800.log.gz 2024-02-01T12:09:15.258 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.69926.log: 52.7% -- replaced with /var/log/ceph/c1-client.0.69926.log.gz 2024-02-01T12:09:15.258 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.209403.log 2024-02-01T12:09:15.258 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/rgw.c1.client.1.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.77665.log 2024-02-01T12:09:15.258 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.209403.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.209403.log.gz 2024-02-01T12:09:15.259 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.154424.log 2024-02-01T12:09:15.259 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.124695.log 2024-02-01T12:09:15.259 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.77665.log: 46.7% -- replaced with /var/log/ceph/c1-client.0.77665.log.gz 2024-02-01T12:09:15.259 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.154424.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.205269.log 2024-02-01T12:09:15.259 INFO:teuthology.orchestra.run.smithi148.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.154424.log.gz 2024-02-01T12:09:15.259 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.124695.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.124695.log.gz 2024-02-01T12:09:15.260 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.198170.log 2024-02-01T12:09:15.260 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.165667.log 2024-02-01T12:09:15.260 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.205269.log: /var/log/ceph/c1-client.0.198170.log: 70.3% -- replaced with /var/log/ceph/c1-client.0.205269.log.gz 2024-02-01T12:09:15.260 INFO:teuthology.orchestra.run.smithi148.stderr: 45.5% -- replaced with /var/log/ceph/c1-client.0.198170.log.gz 2024-02-01T12:09:15.260 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.110807.log 2024-02-01T12:09:15.260 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.157963.log 2024-02-01T12:09:15.260 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.165667.log: /var/log/ceph/c1-client.0.110807.log: 57.0% -- replaced with /var/log/ceph/c1-client.0.165667.log.gz 2024-02-01T12:09:15.261 INFO:teuthology.orchestra.run.smithi148.stderr: 46.7% -- replaced with /var/log/ceph/c1-client.0.110807.log.gz 2024-02-01T12:09:15.261 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.188125.log 2024-02-01T12:09:15.261 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.214184.log 2024-02-01T12:09:15.261 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.157963.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.157963.log.gz 2024-02-01T12:09:15.261 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.188125.log: 56.8% -- replaced with /var/log/ceph/c1-client.0.188125.log.gz 2024-02-01T12:09:15.261 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.189901.log 2024-02-01T12:09:15.262 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.148023.log 2024-02-01T12:09:15.262 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.214184.log: /var/log/ceph/c1-client.0.189901.log: 7.1% -- replaced with /var/log/ceph/c1-client.0.214184.log.gz 2024-02-01T12:09:15.262 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.admin.22584.log 2024-02-01T12:09:15.262 INFO:teuthology.orchestra.run.smithi148.stderr: 57.6% -- replaced with /var/log/ceph/c1-client.0.189901.log.gz 2024-02-01T12:09:15.262 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.148023.log: 14.6% -- replaced with /var/log/ceph/c1-client.0.148023.log.gz 2024-02-01T12:09:15.262 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.admin.22549.log 2024-02-01T12:09:15.263 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.208221.log 2024-02-01T12:09:15.263 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.admin.22584.log: 0.0% -- replaced with /var/log/ceph/c1-client.admin.22584.log.gz 2024-02-01T12:09:15.263 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.admin.22549.log: 0.0% -- replaced with /var/log/ceph/c1-client.admin.22549.log.gz 2024-02-01T12:09:15.263 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.50983.log 2024-02-01T12:09:15.263 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.208221.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.34779.log 2024-02-01T12:09:15.263 INFO:teuthology.orchestra.run.smithi148.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.208221.log.gz 2024-02-01T12:09:15.418 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.50983.log: 50.1% -- replaced with /var/log/ceph/c1-client.0.50983.log.gz 2024-02-01T12:09:15.418 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.163296.log 2024-02-01T12:09:15.418 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.34779.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.206450.log 2024-02-01T12:09:15.418 INFO:teuthology.orchestra.run.smithi148.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.34779.log.gz 2024-02-01T12:09:15.419 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.163296.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.163296.log.gz 2024-02-01T12:09:15.419 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.191082.log 2024-02-01T12:09:15.419 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.196397.log 2024-02-01T12:09:15.419 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.206450.log: /var/log/ceph/c1-client.0.191082.log: 69.8% -- replaced with /var/log/ceph/c1-client.0.206450.log.gz 2024-02-01T12:09:15.419 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.133571.log 2024-02-01T12:09:15.419 INFO:teuthology.orchestra.run.smithi148.stderr: 58.3% -- replaced with /var/log/ceph/c1-client.0.191082.log.gz 2024-02-01T12:09:15.420 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.196397.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.194036.log 2024-02-01T12:09:15.420 INFO:teuthology.orchestra.run.smithi148.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.196397.log.gz 2024-02-01T12:09:15.420 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.133571.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.133571.log.gz 2024-02-01T12:09:15.420 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.169210.log 2024-02-01T12:09:15.420 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.35413.log 2024-02-01T12:09:15.420 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.194036.log: /var/log/ceph/c1-client.0.169210.log: 57.5% -- replaced with /var/log/ceph/c1-client.0.194036.log.gz 2024-02-01T12:09:15.421 INFO:teuthology.orchestra.run.smithi148.stderr: 57.1% -- replaced with /var/log/ceph/c1-client.0.169210.log.gz 2024-02-01T12:09:15.421 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.104305.log 2024-02-01T12:09:15.421 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.73469.log 2024-02-01T12:09:15.421 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.35413.log: /var/log/ceph/c1-client.0.104305.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.35413.log.gz 2024-02-01T12:09:15.421 INFO:teuthology.orchestra.run.smithi148.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.104305.log.gz 2024-02-01T12:09:15.421 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.180438.log 2024-02-01T12:09:15.421 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.73469.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.26035.log 2024-02-01T12:09:15.422 INFO:teuthology.orchestra.run.smithi148.stderr: 52.9% -- replaced with /var/log/ceph/c1-client.0.73469.log.gz 2024-02-01T12:09:15.422 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.180438.log: 57.5% -- replaced with /var/log/ceph/c1-client.0.180438.log.gz 2024-02-01T12:09:15.422 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.96617.log 2024-02-01T12:09:15.422 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.117400.log 2024-02-01T12:09:15.422 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.admin.26035.log: 0.0% -- replaced with /var/log/ceph/c1-client.admin.26035.log.gz 2024-02-01T12:09:15.422 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.96617.log: 45.5% -- replaced with /var/log/ceph/c1-client.0.96617.log.gz 2024-02-01T12:09:15.423 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.119963.log 2024-02-01T12:09:15.423 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.117400.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.26391.log 2024-02-01T12:09:15.423 INFO:teuthology.orchestra.run.smithi148.stderr: 47.4% -- replaced with /var/log/ceph/c1-client.0.117400.log.gz 2024-02-01T12:09:15.423 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.119963.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.119963.log.gz 2024-02-01T12:09:15.423 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.37340.log 2024-02-01T12:09:15.423 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.195807.log 2024-02-01T12:09:15.423 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.admin.26391.log: 0.0% -- replaced with /var/log/ceph/c1-client.admin.26391.log.gz 2024-02-01T12:09:15.424 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.37340.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.37340.log.gz 2024-02-01T12:09:15.424 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.admin.26115.log 2024-02-01T12:09:15.424 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.195807.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.192263.log 2024-02-01T12:09:15.424 INFO:teuthology.orchestra.run.smithi148.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.195807.log.gz 2024-02-01T12:09:15.424 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.admin.26115.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.212954.log 2024-02-01T12:09:15.424 INFO:teuthology.orchestra.run.smithi148.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.26115.log.gz 2024-02-01T12:09:15.425 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.192263.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.26469.log 2024-02-01T12:09:15.425 INFO:teuthology.orchestra.run.smithi148.stderr: 57.5% -- replaced with /var/log/ceph/c1-client.0.192263.log.gz 2024-02-01T12:09:15.425 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.212954.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.131194.log 2024-02-01T12:09:15.425 INFO:teuthology.orchestra.run.smithi148.stderr: 47.4% -- replaced with /var/log/ceph/c1-client.0.212954.log.gz 2024-02-01T12:09:15.425 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.admin.26469.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.124099.log 2024-02-01T12:09:15.425 INFO:teuthology.orchestra.run.smithi148.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.26469.log.gz 2024-02-01T12:09:15.426 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.131194.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.131194.log.gz 2024-02-01T12:09:15.426 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.173342.log 2024-02-01T12:09:15.426 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.124099.log: gzip -5 --verbose -- /var/log/ceph/ops-log-c1-client.1.log 2024-02-01T12:09:15.426 INFO:teuthology.orchestra.run.smithi148.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.124099.log.gz 2024-02-01T12:09:15.426 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.173342.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.173342.log.gz 2024-02-01T12:09:15.426 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.204676.log 2024-02-01T12:09:15.426 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.90694.log 2024-02-01T12:09:15.427 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ops-log-c1-client.1.log: /var/log/ceph/c1-client.0.204676.log: 69.6% -- replaced with /var/log/ceph/c1-client.0.204676.log.gz 2024-02-01T12:09:15.427 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.1.28599.log 2024-02-01T12:09:15.427 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.90694.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.171572.log 2024-02-01T12:09:15.427 INFO:teuthology.orchestra.run.smithi148.stderr: 81.2% -- replaced with /var/log/ceph/c1-client.0.90694.log.gz 2024-02-01T12:09:15.427 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.1.28599.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.130008.log 2024-02-01T12:09:15.427 INFO:teuthology.orchestra.run.smithi148.stderr: 44.9% -- replaced with /var/log/ceph/c1-client.1.28599.log.gz 2024-02-01T12:09:15.428 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.171572.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.168618.log 2024-02-01T12:09:15.428 INFO:teuthology.orchestra.run.smithi148.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.171572.log.gz 2024-02-01T12:09:15.428 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.130008.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.130008.log.gz 2024-02-01T12:09:15.428 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.125285.log 2024-02-01T12:09:15.428 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.72880.log 2024-02-01T12:09:15.428 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.168618.log: 57.8% -- replaced with /var/log/ceph/c1-client.0.168618.log.gz 2024-02-01T12:09:15.428 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.125285.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.48012.log 2024-02-01T12:09:15.429 INFO:teuthology.orchestra.run.smithi148.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.125285.log.gz 2024-02-01T12:09:15.429 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.72880.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.214310.log 2024-02-01T12:09:15.429 INFO:teuthology.orchestra.run.smithi148.stderr: 52.4% -- replaced with /var/log/ceph/c1-client.0.72880.log.gz 2024-02-01T12:09:15.429 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.48012.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.109626.log 2024-02-01T12:09:15.429 INFO:teuthology.orchestra.run.smithi148.stderr: 84.2% -- replaced with /var/log/ceph/c1-client.0.48012.log.gz 2024-02-01T12:09:15.429 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.admin.214310.log: 0.0% -- replaced with /var/log/ceph/c1-client.admin.214310.log.gz 2024-02-01T12:09:15.430 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.175112.log 2024-02-01T12:09:15.430 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.109626.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.47423.log 2024-02-01T12:09:15.430 INFO:teuthology.orchestra.run.smithi148.stderr: 46.3% -- replaced with /var/log/ceph/c1-client.0.109626.log.gz 2024-02-01T12:09:15.430 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.175112.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.202906.log 2024-02-01T12:09:15.430 INFO:teuthology.orchestra.run.smithi148.stderr: 82.6% -- replaced with /var/log/ceph/c1-client.0.175112.log.gz 2024-02-01T12:09:15.430 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.47423.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.26539.log 2024-02-01T12:09:15.430 INFO:teuthology.orchestra.run.smithi148.stderr: 56.9% -- replaced with /var/log/ceph/c1-client.0.47423.log.gz 2024-02-01T12:09:15.431 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.202906.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.55721.log 2024-02-01T12:09:15.431 INFO:teuthology.orchestra.run.smithi148.stderr: 69.4% -- replaced with /var/log/ceph/c1-client.0.202906.log.gz 2024-02-01T12:09:15.431 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.admin.26539.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.97214.log 2024-02-01T12:09:15.431 INFO:teuthology.orchestra.run.smithi148.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.26539.log.gz 2024-02-01T12:09:15.431 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.55721.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.55721.log.gz 2024-02-01T12:09:15.431 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.46227.log 2024-02-01T12:09:15.431 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.61648.log 2024-02-01T12:09:15.432 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.97214.log: 50.4% -- replaced with /var/log/ceph/c1-client.0.97214.log.gz 2024-02-01T12:09:15.432 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.46227.log: gzip -5 --verbose -- 0.0% /var/log/ceph/c1-client.0.111402.log 2024-02-01T12:09:15.432 INFO:teuthology.orchestra.run.smithi148.stderr: -- replaced with /var/log/ceph/c1-client.0.46227.log.gz 2024-02-01T12:09:15.432 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.61648.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.155014.log 2024-02-01T12:09:15.432 INFO:teuthology.orchestra.run.smithi148.stderr: 53.3% -- replaced with /var/log/ceph/c1-client.0.61648.log.gz 2024-02-01T12:09:15.432 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.111402.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.198763.log 2024-02-01T12:09:15.433 INFO:teuthology.orchestra.run.smithi148.stderr: 46.7% -- replaced with /var/log/ceph/c1-client.0.111402.log.gz 2024-02-01T12:09:15.433 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.155014.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.209998.log 2024-02-01T12:09:15.433 INFO:teuthology.orchestra.run.smithi148.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.155014.log.gz 2024-02-01T12:09:15.433 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/rgw.c1.client.0.log 2024-02-01T12:09:15.433 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.198763.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.198763.log.gz 2024-02-01T12:09:15.433 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.209998.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.101351.log 2024-02-01T12:09:15.433 INFO:teuthology.orchestra.run.smithi148.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.209998.log.gz 2024-02-01T12:09:15.434 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/rgw.c1.client.0.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.90105.log 2024-02-01T12:09:15.434 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.101351.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.26461.log 2024-02-01T12:09:15.434 INFO:teuthology.orchestra.run.smithi148.stderr: 46.7% -- replaced with /var/log/ceph/c1-client.0.101351.log.gz 2024-02-01T12:09:15.434 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.90105.log: gzip -5 --verbose -- 46.3% /var/log/ceph/c1-client.0.196987.log -- replaced with /var/log/ceph/c1-client.0.90105.log.gz 2024-02-01T12:09:15.434 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-02-01T12:09:15.434 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.admin.26461.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.120553.log 2024-02-01T12:09:15.435 INFO:teuthology.orchestra.run.smithi148.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.26461.log.gz 2024-02-01T12:09:15.435 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose/var/log/ceph/c1-client.0.196987.log: -- /var/log/ceph/c1-client.0.61055.log 2024-02-01T12:09:15.435 INFO:teuthology.orchestra.run.smithi148.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.196987.log.gz 2024-02-01T12:09:15.435 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.120553.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.120553.log.gz 2024-02-01T12:09:15.435 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.113172.log 2024-02-01T12:09:15.435 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.61055.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.74654.log 2024-02-01T12:09:15.435 INFO:teuthology.orchestra.run.smithi148.stderr: 75.9% -- replaced with /var/log/ceph/c1-client.0.61055.log.gz 2024-02-01T12:09:15.436 INFO:teuthology.orchestra.run.smithi148.stderr:gzip/var/log/ceph/c1-client.0.113172.log: -5 --verbose -- /var/log/ceph/c1-client.0.75247.log 2024-02-01T12:09:15.436 INFO:teuthology.orchestra.run.smithi148.stderr: 46.7% -- replaced with /var/log/ceph/c1-client.0.113172.log.gz 2024-02-01T12:09:15.436 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.74654.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.179849.log 2024-02-01T12:09:15.436 INFO:teuthology.orchestra.run.smithi148.stderr: 47.4% -- replaced with /var/log/ceph/c1-client.0.74654.log.gz 2024-02-01T12:09:15.436 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.141853.log 2024-02-01T12:09:15.436 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.75247.log: /var/log/ceph/c1-client.0.179849.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.174522.log 2024-02-01T12:09:15.437 INFO:teuthology.orchestra.run.smithi148.stderr: 47.4% -- replaced with /var/log/ceph/c1-client.0.75247.log.gz 2024-02-01T12:09:15.437 INFO:teuthology.orchestra.run.smithi148.stderr: 68.9% -- replaced with /var/log/ceph/c1-client.0.179849.log.gz/var/log/ceph/c1-client.0.141853.log: 2024-02-01T12:09:15.437 INFO:teuthology.orchestra.run.smithi148.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.141853.log.gzgzip -5 --verbose -- /var/log/ceph/c1-client.0.166848.log 2024-02-01T12:09:15.437 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-02-01T12:09:15.437 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.174522.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.174522.log.gz 2024-02-01T12:09:15.437 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.94248.log 2024-02-01T12:09:15.438 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.166848.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.175710.log 2024-02-01T12:09:15.438 INFO:teuthology.orchestra.run.smithi148.stderr: 57.7% -- replaced with /var/log/ceph/c1-client.0.166848.log.gz 2024-02-01T12:09:15.438 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.94248.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.173932.log 2024-02-01T12:09:15.438 INFO:teuthology.orchestra.run.smithi148.stderr: 80.9% -- replaced with /var/log/ceph/c1-client.0.94248.log.gz/var/log/ceph/c1-client.0.175710.log: 2024-02-01T12:09:15.438 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.207039.log 2024-02-01T12:09:15.438 INFO:teuthology.orchestra.run.smithi148.stderr: 57.8% -- replaced with /var/log/ceph/c1-client.0.175710.log.gz 2024-02-01T12:09:15.438 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.173932.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.173932.log.gz 2024-02-01T12:09:15.439 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.49205.log 2024-02-01T12:09:15.439 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.207039.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.159145.log 2024-02-01T12:09:15.439 INFO:teuthology.orchestra.run.smithi148.stderr: 70.6% -- replaced with /var/log/ceph/c1-client.0.207039.log.gz 2024-02-01T12:09:15.439 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.49205.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.92473.log 2024-02-01T12:09:15.439 INFO:teuthology.orchestra.run.smithi148.stderr: 56.9% -- replaced with /var/log/ceph/c1-client.0.49205.log.gz/var/log/ceph/c1-client.0.159145.log: 2024-02-01T12:09:15.439 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.81232.log 2024-02-01T12:09:15.439 INFO:teuthology.orchestra.run.smithi148.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.159145.log.gz 2024-02-01T12:09:15.440 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.92473.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.92473.log.gz 2024-02-01T12:09:15.440 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.102532.log 2024-02-01T12:09:15.440 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.81232.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.57498.log 2024-02-01T12:09:15.440 INFO:teuthology.orchestra.run.smithi148.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.81232.log.gz 2024-02-01T12:09:15.440 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.102532.log: gzip 47.0% -5 -- replaced with /var/log/ceph/c1-client.0.102532.log.gz --verbose 2024-02-01T12:09:15.440 INFO:teuthology.orchestra.run.smithi148.stderr: -- /var/log/ceph/c1-client.0.119370.log 2024-02-01T12:09:15.441 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.57498.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.128825.log 2024-02-01T12:09:15.441 INFO:teuthology.orchestra.run.smithi148.stderr: 50.7% -- replaced with /var/log/ceph/c1-client.0.57498.log.gz/var/log/ceph/c1-client.0.119370.log: 2024-02-01T12:09:15.441 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.admin.26682.log 2024-02-01T12:09:15.441 INFO:teuthology.orchestra.run.smithi148.stderr: 47.4% -- replaced with /var/log/ceph/c1-client.0.119370.log.gz 2024-02-01T12:09:15.441 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.128825.log: 47.4% -- replaced with /var/log/ceph/c1-client.0.128825.log.gz 2024-02-01T12:09:15.441 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.admin.26313.log 2024-02-01T12:09:15.441 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.admin.26682.log: 0.0%gzip -5 --verbose -- /var/log/ceph/c1-client.0.116816.log 2024-02-01T12:09:15.442 INFO:teuthology.orchestra.run.smithi148.stderr: -- replaced with /var/log/ceph/c1-client.admin.26682.log.gz 2024-02-01T12:09:15.442 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.172162.log 2024-02-01T12:09:15.442 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.admin.26313.log: 0.0% -- replaced with /var/log/ceph/c1-client.admin.26313.log.gz 2024-02-01T12:09:15.442 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.116816.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.214223.log 2024-02-01T12:09:15.442 INFO:teuthology.orchestra.run.smithi148.stderr: 7.0% -- replaced with /var/log/ceph/c1-client.0.116816.log.gz 2024-02-01T12:09:15.442 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.172162.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.172162.log.gz 2024-02-01T12:09:15.443 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.43268.log 2024-02-01T12:09:15.443 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.140077.log 2024-02-01T12:09:15.443 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.214223.log: /var/log/ceph/c1-client.0.43268.log: 9.2% -- replaced with /var/log/ceph/c1-client.0.214223.log.gz 2024-02-01T12:09:15.443 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.156194.log 2024-02-01T12:09:15.443 INFO:teuthology.orchestra.run.smithi148.stderr: 47.4% -- replaced with /var/log/ceph/c1-client.0.43268.log.gz 2024-02-01T12:09:15.443 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.181030.log 2024-02-01T12:09:15.444 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.140077.log: /var/log/ceph/c1-client.0.156194.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.140077.log.gz 2024-02-01T12:09:15.444 INFO:teuthology.orchestra.run.smithi148.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.156194.log.gz 2024-02-01T12:09:15.444 INFO:teuthology.orchestra.run.smithi148.stderr: 94.5% -- replaced with /var/log/ceph/ops-log-c1-client.1.log.gz 2024-02-01T12:09:15.444 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.66973.log 2024-02-01T12:09:15.444 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.181030.log: gzip -5 --verbose -- /var/log/ceph/c1-osd.0.log 2024-02-01T12:09:15.444 INFO:teuthology.orchestra.run.smithi148.stderr: 57.5% -- replaced with /var/log/ceph/c1-client.0.181030.log.gz 2024-02-01T12:09:15.445 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.66973.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.107856.log 2024-02-01T12:09:15.445 INFO:teuthology.orchestra.run.smithi148.stderr: 66.4% -- replaced with /var/log/ceph/c1-client.0.66973.log.gz 2024-02-01T12:09:15.445 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-osd.0.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.65789.log 2024-02-01T12:09:15.445 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.107856.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.214274.log 2024-02-01T12:09:15.445 INFO:teuthology.orchestra.run.smithi148.stderr: 46.7% -- replaced with /var/log/ceph/c1-client.0.107856.log.gz 2024-02-01T12:09:15.445 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.65789.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.186941.log 2024-02-01T12:09:15.445 INFO:teuthology.orchestra.run.smithi148.stderr: 52.7% -- replaced with /var/log/ceph/c1-client.0.65789.log.gz 2024-02-01T12:09:15.446 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.85963.log 2024-02-01T12:09:15.446 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.214274.log: 10.2%/var/log/ceph/c1-client.0.186941.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.83601.log 2024-02-01T12:09:15.446 INFO:teuthology.orchestra.run.smithi148.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.186941.log.gz 2024-02-01T12:09:15.446 INFO:teuthology.orchestra.run.smithi148.stderr: -- replaced with /var/log/ceph/c1-client.0.214274.log.gz 2024-02-01T12:09:15.446 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.85963.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.97808.log 2024-02-01T12:09:15.446 INFO:teuthology.orchestra.run.smithi148.stderr: 44.8% -- replaced with /var/log/ceph/c1-client.0.85963.log.gz 2024-02-01T12:09:15.447 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.83601.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.121143.log 2024-02-01T12:09:15.447 INFO:teuthology.orchestra.run.smithi148.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.83601.log.gz 2024-02-01T12:09:15.447 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1.log 2024-02-01T12:09:15.447 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.97808.log: /var/log/ceph/c1-client.0.121143.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.192855.log 2024-02-01T12:09:15.447 INFO:teuthology.orchestra.run.smithi148.stderr: 63.4% -- replaced with /var/log/ceph/c1-client.0.97808.log.gz 2024-02-01T12:09:15.447 INFO:teuthology.orchestra.run.smithi148.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.121143.log.gz 2024-02-01T12:09:15.448 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.156784.log 2024-02-01T12:09:15.448 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.52754.log 2024-02-01T12:09:15.448 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.192855.log: 57.4% -- replaced with /var/log/ceph/c1-client.0.192855.log.gz 2024-02-01T12:09:15.448 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.156784.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.156784.log.gz 2024-02-01T12:09:15.448 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.admin.32258.log 2024-02-01T12:09:15.448 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.52754.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.103713.log 2024-02-01T12:09:15.449 INFO:teuthology.orchestra.run.smithi148.stderr: 47.4%/var/log/ceph/c1-client.admin.32258.log: -- replaced with /var/log/ceph/c1-client.0.52754.log.gz 2024-02-01T12:09:15.449 INFO:teuthology.orchestra.run.smithi148.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.32258.log.gz 2024-02-01T12:09:15.449 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.26827.log 2024-02-01T12:09:15.449 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.211177.log 2024-02-01T12:09:15.449 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.103713.log: /var/log/ceph/c1-client.0.26827.log: 46.3% -- replaced with /var/log/ceph/c1-client.0.103713.log.gz 2024-02-01T12:09:15.449 INFO:teuthology.orchestra.run.smithi148.stderr: 84.3% -- replaced with /var/log/ceph/c1-client.0.26827.log.gz 2024-02-01T12:09:15.450 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.103124.log 2024-02-01T12:09:15.450 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.211177.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.211177.log.gz 2024-02-01T12:09:15.450 INFO:teuthology.orchestra.run.smithi148.stderr: 88.2% -- replaced with /var/log/ceph/c1.log.gz 2024-02-01T12:09:15.450 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.admin.26756.log 2024-02-01T12:09:15.450 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.72288.log 2024-02-01T12:09:15.450 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.103124.log: /var/log/ceph/c1-client.admin.26756.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.32232.log 2024-02-01T12:09:15.450 INFO:teuthology.orchestra.run.smithi148.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.26756.log.gz 2024-02-01T12:09:15.451 INFO:teuthology.orchestra.run.smithi148.stderr: 46.3% -- replaced with /var/log/ceph/c1-client.0.103124.log.gz 2024-02-01T12:09:15.451 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.72288.log: 52.7% -- replaced with /var/log/ceph/c1-client.0.72288.log.gz 2024-02-01T12:09:15.451 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.141263.log 2024-02-01T12:09:15.451 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.admin.22525.log 2024-02-01T12:09:15.451 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.admin.32232.log: /var/log/ceph/c1-client.0.141263.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.65197.log 2024-02-01T12:09:15.451 INFO:teuthology.orchestra.run.smithi148.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.141263.log.gz 0.0% -- replaced with /var/log/ceph/c1-client.admin.32232.log.gz 2024-02-01T12:09:15.452 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-02-01T12:09:15.452 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.admin.22525.log: 0.0% -- replaced with /var/log/ceph/c1-client.admin.22525.log.gz 2024-02-01T12:09:15.452 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.208811.log 2024-02-01T12:09:15.452 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.191674.log 2024-02-01T12:09:15.452 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.65197.log: /var/log/ceph/c1-client.0.208811.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.149265.log 2024-02-01T12:09:15.452 INFO:teuthology.orchestra.run.smithi148.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.208811.log.gz 2024-02-01T12:09:15.453 INFO:teuthology.orchestra.run.smithi148.stderr: 52.7% -- replaced with /var/log/ceph/c1-client.0.65197.log.gz 2024-02-01T12:09:15.453 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.191674.log: 58.3% -- replaced with /var/log/ceph/c1-client.0.191674.log.gz 2024-02-01T12:09:15.453 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.184578.log 2024-02-01T12:09:15.453 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.108445.log 2024-02-01T12:09:15.453 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.149265.log: /var/log/ceph/c1-client.0.184578.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.115541.log 2024-02-01T12:09:15.453 INFO:teuthology.orchestra.run.smithi148.stderr: 0.0% 6.3% -- replaced with /var/log/ceph/c1-client.0.149265.log.gz 2024-02-01T12:09:15.454 INFO:teuthology.orchestra.run.smithi148.stderr: -- replaced with /var/log/ceph/c1-client.0.184578.log.gz 2024-02-01T12:09:15.454 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.108445.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.45634.log 46.7% 2024-02-01T12:09:15.454 INFO:teuthology.orchestra.run.smithi148.stderr: -- replaced with /var/log/ceph/c1-client.0.108445.log.gz 2024-02-01T12:09:15.454 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.214260.log 2024-02-01T12:09:15.454 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.115541.log: /var/log/ceph/c1-client.0.45634.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.210590.log 2024-02-01T12:09:15.454 INFO:teuthology.orchestra.run.smithi148.stderr: 47.4% -- replaced with /var/log/ceph/c1-client.0.115541.log.gz 2024-02-01T12:09:15.455 INFO:teuthology.orchestra.run.smithi148.stderr: 47.4% -- replaced with /var/log/ceph/c1-client.0.45634.log.gz 2024-02-01T12:09:15.455 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.214260.log: 7.1% -- replaced with /var/log/ceph/c1-client.0.214260.log.gz 2024-02-01T12:09:15.455 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.194625.log 2024-02-01T12:09:15.455 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.admin.35390.log 2024-02-01T12:09:15.455 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.210590.log: /var/log/ceph/c1-client.0.194625.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.107264.log 2024-02-01T12:09:15.455 INFO:teuthology.orchestra.run.smithi148.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.210590.log.gz 2024-02-01T12:09:15.455 INFO:teuthology.orchestra.run.smithi148.stderr: 59.1% -- replaced with /var/log/ceph/c1-client.0.194625.log.gz/var/log/ceph/c1-client.admin.35390.log: 2024-02-01T12:09:15.456 INFO:teuthology.orchestra.run.smithi148.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.35390.log.gz 2024-02-01T12:09:15.456 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.134758.log 2024-02-01T12:09:15.456 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.128235.log 2024-02-01T12:09:15.456 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.107264.log: /var/log/ceph/c1-client.0.134758.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.26060.log 2024-02-01T12:09:15.456 INFO:teuthology.orchestra.run.smithi148.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.134758.log.gz 2024-02-01T12:09:15.456 INFO:teuthology.orchestra.run.smithi148.stderr: 63.3% -- replaced with /var/log/ceph/c1-client.0.107264.log.gz 2024-02-01T12:09:15.457 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.128235.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.128235.log.gz 2024-02-01T12:09:15.457 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.31647.log 2024-02-01T12:09:15.457 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.137713.log 2024-02-01T12:09:15.457 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.admin.26060.log: /var/log/ceph/c1-client.0.31647.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.123503.log 2024-02-01T12:09:15.457 INFO:teuthology.orchestra.run.smithi148.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.26060.log.gz 2024-02-01T12:09:15.457 INFO:teuthology.orchestra.run.smithi148.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.31647.log.gz 2024-02-01T12:09:15.458 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.137713.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.137713.log.gz 2024-02-01T12:09:15.458 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.81822.log 2024-02-01T12:09:15.458 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.39709.log 2024-02-01T12:09:15.458 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.123503.log: /var/log/ceph/c1-client.0.81822.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.26732.log 2024-02-01T12:09:15.458 INFO:teuthology.orchestra.run.smithi148.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.123503.log.gz 2024-02-01T12:09:15.458 INFO:teuthology.orchestra.run.smithi148.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.81822.log.gz 2024-02-01T12:09:15.459 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.39709.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.26465.log 2024-02-01T12:09:15.459 INFO:teuthology.orchestra.run.smithi148.stderr: 47.4% -- replaced with /var/log/ceph/c1-client.0.39709.log.gz 2024-02-01T12:09:15.459 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.35997.log 2024-02-01T12:09:15.459 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.admin.26732.log: /var/log/ceph/c1-client.admin.26465.log: 0.0%gzip -5 --verbose -- /var/log/ceph/c1-client.0.199943.log 2024-02-01T12:09:15.459 INFO:teuthology.orchestra.run.smithi148.stderr: -- replaced with /var/log/ceph/c1-client.admin.26732.log.gz 2024-02-01T12:09:15.459 INFO:teuthology.orchestra.run.smithi148.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.26465.log.gz 2024-02-01T12:09:15.459 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.35997.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.26166.log 2024-02-01T12:09:15.460 INFO:teuthology.orchestra.run.smithi148.stderr: 18.5% -- replaced with /var/log/ceph/c1-client.0.35997.log.gz 2024-02-01T12:09:15.460 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.199943.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.199943.log.gz 2024-02-01T12:09:15.460 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.211772.log 2024-02-01T12:09:15.460 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.148618.log 2024-02-01T12:09:15.460 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.admin.26166.log: /var/log/ceph/c1-client.0.211772.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.69337.log 2024-02-01T12:09:15.460 INFO:teuthology.orchestra.run.smithi148.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.26166.log.gz 2024-02-01T12:09:15.461 INFO:teuthology.orchestra.run.smithi148.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.211772.log.gz 2024-02-01T12:09:15.461 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.148618.log: gzip 0.0% -5 --verbose -- replaced with /var/log/ceph/c1-client.0.148618.log.gz -- 2024-02-01T12:09:15.461 INFO:teuthology.orchestra.run.smithi148.stderr: /var/log/ceph/c1-client.0.185170.log 2024-02-01T12:09:15.461 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.84188.log 2024-02-01T12:09:15.461 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.69337.log: /var/log/ceph/c1-client.0.185170.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.32281.log 2024-02-01T12:09:15.461 INFO:teuthology.orchestra.run.smithi148.stderr: 53.0% -- replaced with /var/log/ceph/c1-client.0.69337.log.gz 2024-02-01T12:09:15.462 INFO:teuthology.orchestra.run.smithi148.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.185170.log.gz 2024-02-01T12:09:15.462 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.84188.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.84188.log.gz 2024-02-01T12:09:15.462 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.40902.log 2024-02-01T12:09:15.462 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.165075.log 2024-02-01T12:09:15.462 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.32281.log: /var/log/ceph/c1-client.0.40902.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.176302.log 2024-02-01T12:09:15.462 INFO:teuthology.orchestra.run.smithi148.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.32281.log.gz 2024-02-01T12:09:15.463 INFO:teuthology.orchestra.run.smithi148.stderr: 47.4% -- replaced with /var/log/ceph/c1-client.0.40902.log.gz 2024-02-01T12:09:15.463 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.165075.log: 57.8% -- replaced with /var/log/ceph/c1-client.0.165075.log.gz 2024-02-01T12:09:15.463 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.98989.log 2024-02-01T12:09:15.463 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.122913.log 2024-02-01T12:09:15.463 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.176302.log: /var/log/ceph/c1-client.0.98989.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.161524.log 2024-02-01T12:09:15.463 INFO:teuthology.orchestra.run.smithi148.stderr: 57.8% -- replaced with /var/log/ceph/c1-client.0.176302.log.gz 2024-02-01T12:09:15.464 INFO:teuthology.orchestra.run.smithi148.stderr: 46.7% -- replaced with /var/log/ceph/c1-client.0.98989.log.gz/var/log/ceph/c1-client.0.122913.log: 2024-02-01T12:09:15.464 INFO:teuthology.orchestra.run.smithi148.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.122913.log.gz 2024-02-01T12:09:15.464 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.127055.log 2024-02-01T12:09:15.464 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.164483.log 2024-02-01T12:09:15.464 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.161524.log: /var/log/ceph/c1-client.0.127055.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.74061.log 2024-02-01T12:09:15.464 INFO:teuthology.orchestra.run.smithi148.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.161524.log.gz 2024-02-01T12:09:15.465 INFO:teuthology.orchestra.run.smithi148.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.127055.log.gz 2024-02-01T12:09:15.465 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.164483.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.200537.log 2024-02-01T12:09:15.465 INFO:teuthology.orchestra.run.smithi148.stderr: 63.4% -- replaced with /var/log/ceph/c1-client.0.164483.log.gz 2024-02-01T12:09:15.465 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.100170.log 2024-02-01T12:09:15.465 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.74061.log: /var/log/ceph/c1-client.0.200537.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.19928.log 2024-02-01T12:09:15.465 INFO:teuthology.orchestra.run.smithi148.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.74061.log.gz 2024-02-01T12:09:15.466 INFO:teuthology.orchestra.run.smithi148.stderr: 69.6% -- replaced with /var/log/ceph/c1-client.0.200537.log.gz 2024-02-01T12:09:15.466 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.100170.log: 46.3% -- replaced with /var/log/ceph/c1-client.0.100170.log.gz 2024-02-01T12:09:15.466 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.170391.log 2024-02-01T12:09:15.466 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.79460.log 2024-02-01T12:09:15.466 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph.tmp-client.admin.19928.log: /var/log/ceph/c1-client.0.170391.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.138893.log 2024-02-01T12:09:15.466 INFO:teuthology.orchestra.run.smithi148.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.19928.log.gz 2024-02-01T12:09:15.467 INFO:teuthology.orchestra.run.smithi148.stderr: 57.5% -- replaced with /var/log/ceph/c1-client.0.170391.log.gz/var/log/ceph/c1-client.0.79460.log: 2024-02-01T12:09:15.467 INFO:teuthology.orchestra.run.smithi148.stderr: 64.8%gzip -5 --verbose -- /var/log/ceph/c1-client.0.71107.log 2024-02-01T12:09:15.467 INFO:teuthology.orchestra.run.smithi148.stderr: -- replaced with /var/log/ceph/c1-client.0.79460.log.gz 2024-02-01T12:09:15.467 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.153831.log 2024-02-01T12:09:15.467 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.138893.log: /var/log/ceph/c1-client.0.71107.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.138893.log.gz 2024-02-01T12:09:15.467 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.88329.log 2024-02-01T12:09:15.468 INFO:teuthology.orchestra.run.smithi148.stderr: 53.8% -- replaced with /var/log/ceph/c1-client.0.71107.log.gz 2024-02-01T12:09:15.468 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.153831.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.70518.log 2024-02-01T12:09:15.468 INFO:teuthology.orchestra.run.smithi148.stderr: 45.5% -- replaced with /var/log/ceph/c1-client.0.153831.log.gz 2024-02-01T12:09:15.468 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.167437.log 2024-02-01T12:09:15.468 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.88329.log: /var/log/ceph/c1-client.0.70518.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.181619.log 2024-02-01T12:09:15.469 INFO:teuthology.orchestra.run.smithi148.stderr: 46.3% -- replaced with /var/log/ceph/c1-client.0.88329.log.gz 2024-02-01T12:09:15.469 INFO:teuthology.orchestra.run.smithi148.stderr: 52.7% -- replaced with /var/log/ceph/c1-client.0.70518.log.gz/var/log/ceph/c1-client.0.167437.log: 2024-02-01T12:09:15.469 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.182801.log 2024-02-01T12:09:15.469 INFO:teuthology.orchestra.run.smithi148.stderr: 57.8% -- replaced with /var/log/ceph/c1-client.0.167437.log.gz 2024-02-01T12:09:15.469 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.84783.log 2024-02-01T12:09:15.469 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.181619.log: /var/log/ceph/c1-client.0.182801.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.22407.log 2024-02-01T12:09:15.469 INFO:teuthology.orchestra.run.smithi148.stderr: 69.9% -- replaced with /var/log/ceph/c1-client.0.181619.log.gz 2024-02-01T12:09:15.470 INFO:teuthology.orchestra.run.smithi148.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.182801.log.gz 2024-02-01T12:09:15.470 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.84783.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.84783.log.gz 2024-02-01T12:09:15.470 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.132976.log 2024-02-01T12:09:15.470 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.60459.log 2024-02-01T12:09:15.470 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.admin.22407.log: /var/log/ceph/c1-client.0.132976.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.26312.log 2024-02-01T12:09:15.470 INFO:teuthology.orchestra.run.smithi148.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.22407.log.gz 2024-02-01T12:09:15.471 INFO:teuthology.orchestra.run.smithi148.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.132976.log.gz 2024-02-01T12:09:15.471 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.60459.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.172752.log 2024-02-01T12:09:15.471 INFO:teuthology.orchestra.run.smithi148.stderr: 74.9% -- replaced with /var/log/ceph/c1-client.0.60459.log.gz 2024-02-01T12:09:15.471 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.admin.26312.log: 0.0% -- replaced with /var/log/ceph/c1-client.admin.26312.log.gz 2024-02-01T12:09:15.471 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.admin.26088.log 2024-02-01T12:09:15.471 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.admin.26658.log 2024-02-01T12:09:15.472 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.172752.log: /var/log/ceph/c1-client.admin.26088.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.158553.log 2024-02-01T12:09:15.472 INFO:teuthology.orchestra.run.smithi148.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.172752.log.gz 2024-02-01T12:09:15.472 INFO:teuthology.orchestra.run.smithi148.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.26088.log.gz 2024-02-01T12:09:15.472 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.admin.26658.log: 0.0% -- replaced with /var/log/ceph/c1-client.admin.26658.log.gzgzip 2024-02-01T12:09:15.472 INFO:teuthology.orchestra.run.smithi148.stderr: -5 --verbose -- /var/log/ceph/c1-client.0.43861.log 2024-02-01T12:09:15.472 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.89511.log 2024-02-01T12:09:15.473 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.158553.log: /var/log/ceph/c1-client.0.43861.log: gzip -5 --verbose -- /var/log/ceph/c1-osd.1.log 2024-02-01T12:09:15.473 INFO:teuthology.orchestra.run.smithi148.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.158553.log.gz 2024-02-01T12:09:15.473 INFO:teuthology.orchestra.run.smithi148.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.43861.log.gz 2024-02-01T12:09:15.473 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.89511.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.98400.log 2024-02-01T12:09:15.473 INFO:teuthology.orchestra.run.smithi148.stderr: 44.2% -- replaced with /var/log/ceph/c1-client.0.89511.log.gz 2024-02-01T12:09:15.473 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.213565.log 2024-02-01T12:09:15.473 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-osd.1.log: /var/log/ceph/c1-client.0.98400.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.26780.log 2024-02-01T12:09:15.474 INFO:teuthology.orchestra.run.smithi148.stderr: 46.7% -- replaced with /var/log/ceph/c1-client.0.98400.log.gz 2024-02-01T12:09:15.474 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.213565.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.213565.log.gz 2024-02-01T12:09:15.474 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.admin.26632.log 2024-02-01T12:09:15.474 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.admin.26780.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.214249.log 2024-02-01T12:09:15.474 INFO:teuthology.orchestra.run.smithi148.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.26780.log.gz 2024-02-01T12:09:15.474 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.admin.26632.log: 0.0% -- replaced with /var/log/ceph/c1-client.admin.26632.log.gz 2024-02-01T12:09:15.475 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.170980.log 2024-02-01T12:09:15.475 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.87740.log 2024-02-01T12:09:15.475 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.214249.log: /var/log/ceph/c1-client.0.170980.log: 10.2% -- replaced with /var/log/ceph/c1-client.0.214249.log.gz 2024-02-01T12:09:15.475 INFO:teuthology.orchestra.run.smithi148.stderr: 57.2% -- replaced with /var/log/ceph/c1-client.0.170980.log.gz 2024-02-01T12:09:15.475 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.201725.log 2024-02-01T12:09:15.475 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.admin.26708.log 2024-02-01T12:09:15.475 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.87740.log: /var/log/ceph/c1-client.0.201725.log: 46.3% -- replaced with /var/log/ceph/c1-client.0.87740.log.gz 2024-02-01T12:09:15.476 INFO:teuthology.orchestra.run.smithi148.stderr: 69.4% -- replaced with /var/log/ceph/c1-client.0.201725.log.gz 2024-02-01T12:09:15.476 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.201126.log 2024-02-01T12:09:15.476 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.193444.log 2024-02-01T12:09:15.476 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.admin.26708.log: 0.0% -- replaced with /var/log/ceph/c1-client.admin.26708.log.gz 2024-02-01T12:09:15.476 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.201126.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.66378.log 2024-02-01T12:09:15.476 INFO:teuthology.orchestra.run.smithi148.stderr: 87.4% -- replaced with /var/log/ceph/c1-client.0.201126.log.gz 2024-02-01T12:09:15.477 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.193444.log: 57.5% -- replaced with /var/log/ceph/c1-client.0.193444.log.gz 2024-02-01T12:09:15.477 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.34104.log 2024-02-01T12:09:15.477 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.189312.log 2024-02-01T12:09:15.477 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.66378.log: 53.8% -- replaced with /var/log/ceph/c1-client.0.66378.log.gz 2024-02-01T12:09:15.477 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.34104.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.34104.log.gz 2024-02-01T12:09:15.477 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.100762.log 2024-02-01T12:09:15.478 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.189312.log: 57.8% -- replaced with /var/log/ceph/c1-client.0.189312.log.gz 2024-02-01T12:09:15.478 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.139483.log 2024-02-01T12:09:15.478 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.32865.log 2024-02-01T12:09:15.478 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.100762.log: 46.7% -- replaced with /var/log/ceph/c1-client.0.100762.log.gz 2024-02-01T12:09:15.478 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.139483.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.139483.log.gz 2024-02-01T12:09:15.478 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.168029.log 2024-02-01T12:09:15.478 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.32865.log: 18.5% -- replaced with /var/log/ceph/c1-client.0.32865.log.gz 2024-02-01T12:09:15.479 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.68745.log 2024-02-01T12:09:15.479 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.94844.log 2024-02-01T12:09:15.479 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.168029.log: 57.8%/var/log/ceph/c1-client.0.68745.log: -- replaced with /var/log/ceph/c1-client.0.168029.log.gz 2024-02-01T12:09:15.479 INFO:teuthology.orchestra.run.smithi148.stderr: 53.3% -- replaced with /var/log/ceph/c1-client.0.68745.log.gz 2024-02-01T12:09:15.479 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.83009.log 2024-02-01T12:09:15.479 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.94844.log: 59.7% -- replaced with /var/log/ceph/c1-client.0.94844.log.gz 2024-02-01T12:09:15.480 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.151889.log 2024-02-01T12:09:15.480 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.157371.log 2024-02-01T12:09:15.480 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.83009.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.83009.log.gz 2024-02-01T12:09:15.480 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.151889.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.151889.log.gzgzip -5 --verbose -- /var/log/ceph/c1-client.0.143790.log 2024-02-01T12:09:15.480 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-02-01T12:09:15.480 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.157371.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.157371.log.gz 2024-02-01T12:09:15.480 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.111991.log 2024-02-01T12:09:15.481 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.54532.log 2024-02-01T12:09:15.481 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.143790.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.143790.log.gz 2024-02-01T12:09:15.481 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.111991.log: 46.7% -- replaced with /var/log/ceph/c1-client.0.111991.log.gzgzip -5 --verbose -- /var/log/ceph/c1-client.0.125875.log 2024-02-01T12:09:15.481 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-02-01T12:09:15.481 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.54532.log: 47.4% -- replaced with /var/log/ceph/c1-client.0.54532.log.gz 2024-02-01T12:09:15.481 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.39119.log 2024-02-01T12:09:15.482 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.46817.log 2024-02-01T12:09:15.482 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.125875.log: /var/log/ceph/c1-client.0.39119.log: 0.0% 0.0% -- replaced with /var/log/ceph/c1-client.0.39119.log.gz -- replaced with /var/log/ceph/c1-client.0.125875.log.gz 2024-02-01T12:09:15.482 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-02-01T12:09:15.482 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.admin.35364.log 2024-02-01T12:09:15.482 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.42085.log 2024-02-01T12:09:15.482 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.46817.log: /var/log/ceph/c1-client.admin.35364.log: 47.4% -- replaced with /var/log/ceph/c1-client.0.46817.log.gz 2024-02-01T12:09:15.483 INFO:teuthology.orchestra.run.smithi148.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.35364.log.gz 2024-02-01T12:09:15.483 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.37930.log 2024-02-01T12:09:15.483 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.admin.26383.log 2024-02-01T12:09:15.483 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.42085.log: /var/log/ceph/c1-client.0.37930.log: 45.5% -- replaced with /var/log/ceph/c1-client.0.42085.log.gz 2024-02-01T12:09:15.483 INFO:teuthology.orchestra.run.smithi148.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.37930.log.gz 2024-02-01T12:09:15.483 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.82414.log 2024-02-01T12:09:15.484 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ops-log-c1-client.0.log 2024-02-01T12:09:15.484 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.admin.26383.log: /var/log/ceph/c1-client.0.82414.log: 0.0% -- replaced with /var/log/ceph/c1-client.admin.26383.log.gz 2024-02-01T12:09:15.484 INFO:teuthology.orchestra.run.smithi148.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.82414.log.gz 2024-02-01T12:09:15.484 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.185757.log 2024-02-01T12:09:15.484 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.105488.log 2024-02-01T12:09:15.484 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ops-log-c1-client.0.log: /var/log/ceph/c1-client.0.185757.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.185757.log.gz 2024-02-01T12:09:15.485 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.58681.log 2024-02-01T12:09:15.485 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.105488.log: 47.4% -- replaced with /var/log/ceph/c1-client.0.105488.log.gz 2024-02-01T12:09:15.485 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.50390.log 2024-02-01T12:09:15.485 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.58681.log: 50.4% -- replaced with /var/log/ceph/c1-client.0.58681.log.gz 2024-02-01T12:09:15.485 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.127645.log 2024-02-01T12:09:15.485 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.50390.log: 47.4% -- replaced with /var/log/ceph/c1-client.0.50390.log.gz 2024-02-01T12:09:15.485 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.55121.log 2024-02-01T12:09:15.486 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.127645.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.127645.log.gz 2024-02-01T12:09:15.486 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.132381.log 2024-02-01T12:09:15.486 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.55121.log: 81.1% -- replaced with /var/log/ceph/c1-client.0.55121.log.gz 2024-02-01T12:09:15.486 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.admin.26234.log 2024-02-01T12:09:15.486 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.132381.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.132381.log.gz 2024-02-01T12:09:15.486 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.71699.log 2024-02-01T12:09:15.487 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.admin.26234.log: 0.0% -- replaced with /var/log/ceph/c1-client.admin.26234.log.gz 2024-02-01T12:09:15.487 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.52165.log 2024-02-01T12:09:15.487 INFO:teuthology.orchestra.run.smithi148.stderr: 94.4% -- replaced with /var/log/ceph/ops-log-c1-client.0.log.gz 2024-02-01T12:09:15.487 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.71699.log: 53.0%gzip -5 --verbose -- /var/log/ceph/c1-client.0.146091.log 2024-02-01T12:09:15.487 INFO:teuthology.orchestra.run.smithi148.stderr: -- replaced with /var/log/ceph/c1-client.0.71699.log.gz 2024-02-01T12:09:15.487 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.52165.log: 47.4%gzip -5 --verbose -- /var/log/ceph/c1-client.0.160932.log 2024-02-01T12:09:15.488 INFO:teuthology.orchestra.run.smithi148.stderr: -- replaced with /var/log/ceph/c1-client.0.52165.log.gz 2024-02-01T12:09:15.488 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.146091.log: 45.5% -- replaced with /var/log/ceph/c1-client.0.146091.log.gz 2024-02-01T12:09:15.488 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.129418.log 2024-02-01T12:09:15.488 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.104895.log 2024-02-01T12:09:15.488 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.160932.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.160932.log.gz 2024-02-01T12:09:15.488 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.129418.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.129418.log.gz 2024-02-01T12:09:15.489 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.78266.log 2024-02-01T12:09:15.489 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.104895.log: 47.4% -- replaced with /var/log/ceph/c1-client.0.104895.log.gz 2024-02-01T12:09:15.489 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.207631.log 2024-02-01T12:09:15.489 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.56904.log 2024-02-01T12:09:15.489 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.78266.log: /var/log/ceph/c1-client.0.207631.log: 84.7% -- replaced with /var/log/ceph/c1-client.0.78266.log.gz 2024-02-01T12:09:15.489 INFO:teuthology.orchestra.run.smithi148.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.207631.log.gz 2024-02-01T12:09:15.489 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.44451.log 2024-02-01T12:09:15.490 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.56904.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.202314.log 2024-02-01T12:09:15.490 INFO:teuthology.orchestra.run.smithi148.stderr: 44.8% -- replaced with /var/log/ceph/c1-client.0.56904.log.gz 2024-02-01T12:09:15.490 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.44451.log: 47.4% -- replaced with /var/log/ceph/c1-client.0.44451.log.gz 2024-02-01T12:09:15.490 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.101943.log 2024-02-01T12:09:15.490 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.59869.log 2024-02-01T12:09:15.490 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.202314.log: /var/log/ceph/c1-client.0.101943.log: 70.3% -- replaced with /var/log/ceph/c1-client.0.202314.log.gz 2024-02-01T12:09:15.490 INFO:teuthology.orchestra.run.smithi148.stderr: 46.7% -- replaced with /var/log/ceph/c1-client.0.101943.log.gz 2024-02-01T12:09:15.491 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.30479.log 2024-02-01T12:09:15.491 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.87146.log 2024-02-01T12:09:15.491 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.59869.log: /var/log/ceph/c1-client.0.30479.log: 64.5% -- replaced with /var/log/ceph/c1-client.0.59869.log.gz 2024-02-01T12:09:15.491 INFO:teuthology.orchestra.run.smithi148.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.30479.log.gz 2024-02-01T12:09:15.491 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.190493.log 2024-02-01T12:09:15.491 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.42678.log 2024-02-01T12:09:15.492 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.87146.log: /var/log/ceph/c1-client.0.190493.log: 58.3% -- replaced with /var/log/ceph/c1-client.0.87146.log.gz 2024-02-01T12:09:15.492 INFO:teuthology.orchestra.run.smithi148.stderr: 57.7% -- replaced with /var/log/ceph/c1-client.0.190493.log.gz 2024-02-01T12:09:15.492 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.169799.log 2024-02-01T12:09:15.492 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.140667.log 2024-02-01T12:09:15.492 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.42678.log: /var/log/ceph/c1-client.0.169799.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.42678.log.gz 2024-02-01T12:09:15.493 INFO:teuthology.orchestra.run.smithi148.stderr: 57.5% -- replaced with /var/log/ceph/c1-client.0.169799.log.gz 2024-02-01T12:09:15.493 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.188714.log 2024-02-01T12:09:15.493 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.183391.log 2024-02-01T12:09:15.493 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.140667.log: /var/log/ceph/c1-client.0.188714.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.140667.log.gz 2024-02-01T12:09:15.493 INFO:teuthology.orchestra.run.smithi148.stderr: 63.6% -- replaced with /var/log/ceph/c1-client.0.188714.log.gz 2024-02-01T12:09:15.493 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.144395.log 2024-02-01T12:09:15.494 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.155604.log 2024-02-01T12:09:15.494 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.183391.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.183391.log.gz 2024-02-01T12:09:15.494 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.144395.log: 49.4% -- replaced with /var/log/ceph/c1-client.0.144395.log.gzgzip -5 --verbose -- /var/log/ceph/c1-client.0.162706.log 2024-02-01T12:09:15.494 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-02-01T12:09:15.494 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.155604.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.155604.log.gz 2024-02-01T12:09:15.494 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.131786.log 2024-02-01T12:09:15.494 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.75888.log 2024-02-01T12:09:15.495 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.162706.log: /var/log/ceph/c1-client.0.131786.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.162706.log.gz 2024-02-01T12:09:15.495 INFO:teuthology.orchestra.run.smithi148.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.131786.log.gz 2024-02-01T12:09:15.495 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.93065.log 2024-02-01T12:09:15.495 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.59271.log 2024-02-01T12:09:15.495 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.75888.log: /var/log/ceph/c1-client.0.93065.log: 47.1% -- replaced with /var/log/ceph/c1-client.0.75888.log.gz 2024-02-01T12:09:15.495 INFO:teuthology.orchestra.run.smithi148.stderr: 47.4% -- replaced with /var/log/ceph/c1-client.0.93065.log.gz 2024-02-01T12:09:15.495 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.64016.log 2024-02-01T12:09:15.496 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.78864.log 2024-02-01T12:09:15.496 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.59271.log: 64.7% -- replaced with /var/log/ceph/c1-client.0.59271.log.gz 2024-02-01T12:09:15.496 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.64016.log: 53.5% -- replaced with /var/log/ceph/c1-client.0.64016.log.gz 2024-02-01T12:09:15.496 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.176891.log 2024-02-01T12:09:15.496 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.78864.log: 79.9% -- replaced with /var/log/ceph/c1-client.0.78864.log.gz 2024-02-01T12:09:15.496 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.64608.log 2024-02-01T12:09:15.496 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.204087.log 2024-02-01T12:09:15.497 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.176891.log: /var/log/ceph/c1-client.0.64608.log: 87.4% -- replaced with /var/log/ceph/c1-client.0.176891.log.gz 2024-02-01T12:09:15.497 INFO:teuthology.orchestra.run.smithi148.stderr: 53.0% -- replaced with /var/log/ceph/c1-client.0.64608.log.gz 2024-02-01T12:09:15.497 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.159740.log 2024-02-01T12:09:15.497 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.138303.log 2024-02-01T12:09:15.497 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.204087.log: /var/log/ceph/c1-client.0.159740.log: 70.4% -- replaced with /var/log/ceph/c1-client.0.204087.log.gz 2024-02-01T12:09:15.497 INFO:teuthology.orchestra.run.smithi148.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.159740.log.gz 2024-02-01T12:09:15.498 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.187531.log 2024-02-01T12:09:15.498 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.214288.log 2024-02-01T12:09:15.498 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.138303.log: /var/log/ceph/c1-client.0.187531.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.138303.log.gz 2024-02-01T12:09:15.498 INFO:teuthology.orchestra.run.smithi148.stderr: 44.8% -- replaced with /var/log/ceph/c1-client.0.187531.log.gz 2024-02-01T12:09:15.498 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.68156.log 2024-02-01T12:09:15.498 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.admin.26201.log 2024-02-01T12:09:15.499 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.214288.log: /var/log/ceph/c1-client.0.68156.log: 6.1% -- replaced with /var/log/ceph/c1-client.0.214288.log.gz 2024-02-01T12:09:15.499 INFO:teuthology.orchestra.run.smithi148.stderr: 52.4% -- replaced with /var/log/ceph/c1-client.0.68156.log.gz 2024-02-01T12:09:15.499 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.80050.log 2024-02-01T12:09:15.499 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.77076.log 2024-02-01T12:09:15.499 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.admin.26201.log: /var/log/ceph/c1-client.0.80050.log: 0.0% -- replaced with /var/log/ceph/c1-client.admin.26201.log.gz 2024-02-01T12:09:15.499 INFO:teuthology.orchestra.run.smithi148.stderr: 51.6% -- replaced with /var/log/ceph/c1-client.0.80050.log.gz 2024-02-01T12:09:15.500 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.56311.log 2024-02-01T12:09:15.500 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.214197.log 2024-02-01T12:09:15.500 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.77076.log: 47.1% -- replaced with /var/log/ceph/c1-client.0.77076.log.gz 2024-02-01T12:09:15.500 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.56311.log: 47.4% -- replaced with /var/log/ceph/c1-client.0.56311.log.gzgzip 2024-02-01T12:09:15.500 INFO:teuthology.orchestra.run.smithi148.stderr: -5 --verbose -- /var/log/ceph/c1-mon.a.log 2024-02-01T12:09:15.501 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.214197.log: 10.2% -- replaced with /var/log/ceph/c1-client.0.214197.log.gz 2024-02-01T12:09:15.501 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.95434.log 2024-02-01T12:09:15.501 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.93659.log 2024-02-01T12:09:15.501 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-mon.a.log: /var/log/ceph/c1-client.0.95434.log: 46.0% -- replaced with /var/log/ceph/c1-client.0.95434.log.gz 2024-02-01T12:09:15.548 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.126465.log 2024-02-01T12:09:15.548 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.93659.log: 46.7% -- replaced with /var/log/ceph/c1-client.0.93659.log.gz 2024-02-01T12:09:15.548 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.214235.log 2024-02-01T12:09:15.549 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.126465.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.126465.log.gz 2024-02-01T12:09:15.549 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.114951.log 2024-02-01T12:09:15.549 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.214235.log: 9.2% -- replaced with /var/log/ceph/c1-client.0.214235.log.gz 2024-02-01T12:09:15.549 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1.audit.log 2024-02-01T12:09:15.549 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.114951.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.114951.log.gz 2024-02-01T12:09:15.549 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.143033.log 2024-02-01T12:09:15.549 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1.audit.log: 89.2% -- replaced with /var/log/ceph/c1.audit.log.gz 2024-02-01T12:09:15.550 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.67564.log 2024-02-01T12:09:15.550 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.143033.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.143033.log.gz 2024-02-01T12:09:15.550 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.113768.log 2024-02-01T12:09:15.550 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.67564.log: 52.7% -- replaced with /var/log/ceph/c1-client.0.67564.log.gz 2024-02-01T12:09:15.550 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.109037.log 2024-02-01T12:09:15.550 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.113768.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.113768.log.gz 2024-02-01T12:09:15.550 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.142443.log 2024-02-01T12:09:15.551 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.109037.log: 50.4% -- replaced with /var/log/ceph/c1-client.0.109037.log.gz 2024-02-01T12:09:15.551 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.27425.log 2024-02-01T12:09:15.551 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.142443.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.142443.log.gz 2024-02-01T12:09:15.551 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.53347.log 2024-02-01T12:09:15.551 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.27425.log: 45.3% -- replaced with /var/log/ceph/c1-client.0.27425.log.gz 2024-02-01T12:09:15.551 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.admin.22646.log 2024-02-01T12:09:15.552 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.53347.log: 47.0% -- replaced with /var/log/ceph/c1-client.0.53347.log.gz 2024-02-01T12:09:15.552 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.178668.log 2024-02-01T12:09:15.552 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.admin.22646.log: 0.0% -- replaced with /var/log/ceph/c1-client.admin.22646.log.gz 2024-02-01T12:09:15.552 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.166256.log 2024-02-01T12:09:15.552 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.178668.log: 70.3% -- replaced with /var/log/ceph/c1-client.0.178668.log.gz 2024-02-01T12:09:15.552 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.150543.log 2024-02-01T12:09:15.553 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.166256.log: 62.9% -- replaced with /var/log/ceph/c1-client.0.166256.log.gz 2024-02-01T12:09:15.553 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.179257.log 2024-02-01T12:09:15.553 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.150543.log: 41.9% -- replaced with /var/log/ceph/c1-client.0.150543.log.gz 2024-02-01T12:09:15.553 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.205858.log 2024-02-01T12:09:15.553 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.179257.log: 57.5% -- replaced with /var/log/ceph/c1-client.0.179257.log.gz 2024-02-01T12:09:15.553 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.212364.log 2024-02-01T12:09:15.554 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.205858.log: 69.8% -- replaced with /var/log/ceph/c1-client.0.205858.log.gz 2024-02-01T12:09:15.554 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.147439.log 2024-02-01T12:09:15.554 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.212364.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.212364.log.gz 2024-02-01T12:09:15.554 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.136527.log 2024-02-01T12:09:15.554 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.147439.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.147439.log.gz 2024-02-01T12:09:15.554 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.122323.log 2024-02-01T12:09:15.555 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.136527.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.136527.log.gz 2024-02-01T12:09:15.555 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.118779.log 2024-02-01T12:09:15.555 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.122323.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.122323.log.gz 2024-02-01T12:09:15.555 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.admin.26387.log 2024-02-01T12:09:15.555 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.118779.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.118779.log.gz 2024-02-01T12:09:15.555 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.53936.log 2024-02-01T12:09:15.556 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.admin.26387.log: 0.0% -- replaced with /var/log/ceph/c1-client.admin.26387.log.gz 2024-02-01T12:09:15.556 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.88921.log 2024-02-01T12:09:15.556 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.53936.log: 77.5% -- replaced with /var/log/ceph/c1-client.0.53936.log.gz 2024-02-01T12:09:15.556 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.203495.log 2024-02-01T12:09:15.556 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.88921.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.88921.log.gz 2024-02-01T12:09:15.556 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.106675.log 2024-02-01T12:09:15.557 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.203495.log: 70.6% -- replaced with /var/log/ceph/c1-client.0.203495.log.gz 2024-02-01T12:09:15.557 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.183983.log 2024-02-01T12:09:15.557 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.106675.log: 50.1% -- replaced with /var/log/ceph/c1-client.0.106675.log.gz 2024-02-01T12:09:15.557 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.admin.26565.log 2024-02-01T12:09:15.557 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.183983.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.183983.log.gz 2024-02-01T12:09:15.558 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.177487.log 2024-02-01T12:09:15.558 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.admin.26565.log: 0.0% -- replaced with /var/log/ceph/c1-client.admin.26565.log.gz 2024-02-01T12:09:15.558 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.63427.log 2024-02-01T12:09:15.558 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.177487.log: 69.2% -- replaced with /var/log/ceph/c1-client.0.177487.log.gz 2024-02-01T12:09:15.558 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.85370.log 2024-02-01T12:09:15.558 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.63427.log: 52.4% -- replaced with /var/log/ceph/c1-client.0.63427.log.gz 2024-02-01T12:09:15.559 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.178076.log 2024-02-01T12:09:15.559 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.85370.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.85370.log.gz 2024-02-01T12:09:15.559 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.145501.log 2024-02-01T12:09:15.559 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.178076.log: 70.3% -- replaced with /var/log/ceph/c1-client.0.178076.log.gz 2024-02-01T12:09:15.559 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.149948.log 2024-02-01T12:09:15.559 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.145501.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.145501.log.gz 2024-02-01T12:09:15.560 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.91292.log 2024-02-01T12:09:15.560 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.149948.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.149948.log.gz 2024-02-01T12:09:15.560 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.134166.log 2024-02-01T12:09:15.560 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.91292.log: 46.7% -- replaced with /var/log/ceph/c1-client.0.91292.log.gz 2024-02-01T12:09:15.560 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.121733.log 2024-02-01T12:09:15.561 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.134166.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.134166.log.gz 2024-02-01T12:09:15.561 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.116223.log 2024-02-01T12:09:15.561 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.121733.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.121733.log.gz 2024-02-01T12:09:15.561 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.80642.log 2024-02-01T12:09:15.561 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.116223.log: 47.4% -- replaced with /var/log/ceph/c1-client.0.116223.log.gz 2024-02-01T12:09:15.561 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.76477.log 2024-02-01T12:09:15.562 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.80642.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.80642.log.gz 2024-02-01T12:09:15.562 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/c1-client.0.76477.log: 87.5% -- replaced with /var/log/ceph/c1-client.0.76477.log.gz 2024-02-01T12:09:15.591 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.62459.log 2024-02-01T12:09:15.592 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.145529.log 2024-02-01T12:09:15.592 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.50598.log/var/log/ceph/c2-client.0.62459.log: 2024-02-01T12:09:15.592 INFO:teuthology.orchestra.run.smithi187.stderr: 53.3% -- replaced with /var/log/ceph/c2-client.0.62459.log.gz 2024-02-01T12:09:15.592 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.145529.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.142576.log 2024-02-01T12:09:15.593 INFO:teuthology.orchestra.run.smithi187.stderr: 56.8% -- replaced with /var/log/ceph/c2-client.0.145529.log.gz 2024-02-01T12:09:15.593 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.50598.log: 47.4% -- replaced with /var/log/ceph/c2-client.0.50598.log.gz 2024-02-01T12:09:15.593 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.admin.26022.log 2024-02-01T12:09:15.593 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.142576.log: 45.5% -- replaced with /var/log/ceph/c2-client.0.142576.log.gz 2024-02-01T12:09:15.593 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.80839.log 2024-02-01T12:09:15.593 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.admin.26022.log: 0.0% -- replaced with /var/log/ceph/c2-client.admin.26022.log.gz 2024-02-01T12:09:15.594 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.136074.log 2024-02-01T12:09:15.594 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.80839.log: 47.4% -- replaced with /var/log/ceph/c2-client.0.80839.log.gz 2024-02-01T12:09:15.594 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.admin.26402.log 2024-02-01T12:09:15.594 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.136074.log: 47.4% -- replaced with /var/log/ceph/c2-client.0.136074.log.gz 2024-02-01T12:09:15.594 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.35758.log 2024-02-01T12:09:15.594 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.admin.26402.log: 0.0% -- replaced with /var/log/ceph/c2-client.admin.26402.log.gz 2024-02-01T12:09:15.595 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.120049.log 2024-02-01T12:09:15.595 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.35758.log: 47.4% -- replaced with /var/log/ceph/c2-client.0.35758.log.gz 2024-02-01T12:09:15.595 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.115418.log 2024-02-01T12:09:15.595 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.120049.log: 47.4% -- replaced with /var/log/ceph/c2-client.0.120049.log.gz 2024-02-01T12:09:15.595 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.118529.log 2024-02-01T12:09:15.595 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.115418.log: 0.0% -- replaced with /var/log/ceph/c2-client.0.115418.log.gz 2024-02-01T12:09:15.596 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.75472.log 2024-02-01T12:09:15.596 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.118529.log: 18.5% -- replaced with /var/log/ceph/c2-client.0.118529.log.gz 2024-02-01T12:09:15.596 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.90294.log 2024-02-01T12:09:15.596 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.75472.log: 47.4% -- replaced with /var/log/ceph/c2-client.0.75472.log.gz 2024-02-01T12:09:15.596 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.162641.log 2024-02-01T12:09:15.596 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.90294.log: 7.0% -- replaced with /var/log/ceph/c2-client.0.90294.log.gz 2024-02-01T12:09:15.597 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.admin.26170.log 2024-02-01T12:09:15.597 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.162641.log: 0.0% -- replaced with /var/log/ceph/c2-client.0.162641.log.gz 2024-02-01T12:09:15.597 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.45842.log 2024-02-01T12:09:15.597 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.admin.26170.log: 0.0% -- replaced with /var/log/ceph/c2-client.admin.26170.log.gz 2024-02-01T12:09:15.597 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.38130.log 2024-02-01T12:09:15.598 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.45842.log: 45.5% -- replaced with /var/log/ceph/c2-client.0.45842.log.gz 2024-02-01T12:09:15.598 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.128956.log 2024-02-01T12:09:15.598 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.38130.log: 44.8% -- replaced with /var/log/ceph/c2-client.0.38130.log.gz 2024-02-01T12:09:15.598 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.112400.log 2024-02-01T12:09:15.598 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.128956.log: 45.5% -- replaced with /var/log/ceph/c2-client.0.128956.log.gz 2024-02-01T12:09:15.598 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.admin.26574.log 2024-02-01T12:09:15.598 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.112400.log: gzip 47.4% -- replaced with /var/log/ceph/c2-client.0.112400.log.gz 2024-02-01T12:09:15.599 INFO:teuthology.orchestra.run.smithi187.stderr: -5 --verbose -- /var/log/ceph/c2-client.0.128363.log 2024-02-01T12:09:15.599 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.admin.26574.log: 0.0% -- replaced with /var/log/ceph/c2-client.admin.26574.log.gz 2024-02-01T12:09:15.599 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.19949.log 2024-02-01T12:09:15.599 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.128363.log: 47.4% -- replaced with /var/log/ceph/c2-client.0.128363.log.gz 2024-02-01T12:09:15.599 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.admin.26452.log 2024-02-01T12:09:15.599 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph.tmp-client.admin.19949.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.19949.log.gz 2024-02-01T12:09:15.600 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.admin.31443.log 2024-02-01T12:09:15.600 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.admin.26452.log: 0.0% -- replaced with /var/log/ceph/c2-client.admin.26452.log.gz 2024-02-01T12:09:15.600 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.admin.26550.log 2024-02-01T12:09:15.600 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.admin.31443.log: 0.0% -- replaced with /var/log/ceph/c2-client.admin.31443.log.gz 2024-02-01T12:09:15.600 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.164408.log 2024-02-01T12:09:15.600 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.admin.26550.log: 0.0% -- replaced with /var/log/ceph/c2-client.admin.26550.log.gz 2024-02-01T12:09:15.601 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.59497.log 2024-02-01T12:09:15.601 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.164408.log: 47.4% -- replaced with /var/log/ceph/c2-client.0.164408.log.gz 2024-02-01T12:09:15.601 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.106466.log 2024-02-01T12:09:15.601 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.59497.log: gzip -5 45.5% --verbose -- replaced with /var/log/ceph/c2-client.0.59497.log.gz 2024-02-01T12:09:15.601 INFO:teuthology.orchestra.run.smithi187.stderr: -- /var/log/ceph/c2-client.0.27195.log 2024-02-01T12:09:15.601 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.106466.log: 45.5% -- replaced with /var/log/ceph/c2-client.0.106466.log.gz 2024-02-01T12:09:15.602 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.157915.log 2024-02-01T12:09:15.602 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.27195.log: 43.9% -- replaced with /var/log/ceph/c2-client.0.27195.log.gz 2024-02-01T12:09:15.602 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.104036.log 2024-02-01T12:09:15.602 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.157915.log: 47.4% -- replaced with /var/log/ceph/c2-client.0.157915.log.gz 2024-02-01T12:09:15.602 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.91561.log 2024-02-01T12:09:15.602 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.104036.log: 0.0% -- replaced with /var/log/ceph/c2-client.0.104036.log.gz 2024-02-01T12:09:15.602 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.86144.log 2024-02-01T12:09:15.603 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.91561.log: 0.0% -- replaced with /var/log/ceph/c2-client.0.91561.log.gz 2024-02-01T12:09:15.603 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.61272.log 2024-02-01T12:09:15.603 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.86144.log: 84.4% -- replaced with /var/log/ceph/c2-client.0.86144.log.gz 2024-02-01T12:09:15.603 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.141983.log 2024-02-01T12:09:15.603 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.61272.log: 81.5% -- replaced with /var/log/ceph/c2-client.0.61272.log.gz 2024-02-01T12:09:15.603 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.121235.log 2024-02-01T12:09:15.604 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.141983.log: gzip 48.7% -- replaced with /var/log/ceph/c2-client.0.141983.log.gz 2024-02-01T12:09:15.604 INFO:teuthology.orchestra.run.smithi187.stderr: -5 --verbose -- /var/log/ceph/c2-client.0.30832.log 2024-02-01T12:09:15.604 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.121235.log: 45.5% -- replaced with /var/log/ceph/c2-client.0.121235.log.gz 2024-02-01T12:09:15.604 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.admin.26099.log 2024-02-01T12:09:15.604 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.30832.log: 0.0% -- replaced with /var/log/ceph/c2-client.0.30832.log.gz 2024-02-01T12:09:15.604 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.134295.log 2024-02-01T12:09:15.605 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.admin.26099.log: 0.0% -- replaced with /var/log/ceph/c2-client.admin.26099.log.gz 2024-02-01T12:09:15.605 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.152009.log 2024-02-01T12:09:15.605 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.134295.log: 47.4% -- replaced with /var/log/ceph/c2-client.0.134295.log.gz 2024-02-01T12:09:15.605 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-osd.1.log 2024-02-01T12:09:15.605 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.152009.log: 0.0% -- replaced with /var/log/ceph/c2-client.0.152009.log.gz 2024-02-01T12:09:15.605 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.34566.log 2024-02-01T12:09:15.605 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-osd.1.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.159692.log 2024-02-01T12:09:15.605 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.34566.log: 44.8% -- replaced with /var/log/ceph/c2-client.0.34566.log.gz 2024-02-01T12:09:15.606 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.35165.log 2024-02-01T12:09:15.606 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.159692.log: gzip -5 --verbose 69.4% -- -- replaced with /var/log/ceph/c2-client.0.159692.log.gz 2024-02-01T12:09:15.606 INFO:teuthology.orchestra.run.smithi187.stderr: /var/log/ceph/c2-client.0.55942.log 2024-02-01T12:09:15.606 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.35165.log: 47.4% -- replaced with /var/log/ceph/c2-client.0.35165.log.gz 2024-02-01T12:09:15.606 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.137846.log 2024-02-01T12:09:15.606 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.55942.log: 45.5% -- replaced with /var/log/ceph/c2-client.0.55942.log.gz 2024-02-01T12:09:15.607 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.169154.log 2024-02-01T12:09:15.607 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.137846.log: 0.0% -- replaced with /var/log/ceph/c2-client.0.137846.log.gz 2024-02-01T12:09:15.607 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.132514.log 2024-02-01T12:09:15.607 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.169154.log: gzip -5 --verbose -- 10.2% -- replaced with /var/log/ceph/c2-client.0.169154.log.gz /var/log/ceph/c2-client.admin.26526.log 2024-02-01T12:09:15.607 INFO:teuthology.orchestra.run.smithi187.stderr: 2024-02-01T12:09:15.607 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.132514.log: 44.8% -- replaced with /var/log/ceph/c2-client.0.132514.log.gz 2024-02-01T12:09:15.608 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.39912.log 2024-02-01T12:09:15.608 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.admin.26526.log: 0.0% -- replaced with /var/log/ceph/c2-client.admin.26526.log.gz 2024-02-01T12:09:15.608 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.109433.log 2024-02-01T12:09:15.608 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.39912.log: 47.4% -- replaced with /var/log/ceph/c2-client.0.39912.log.gz 2024-02-01T12:09:15.608 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.82606.log 2024-02-01T12:09:15.608 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.109433.log: gzip 47.4% -5 -- replaced with /var/log/ceph/c2-client.0.109433.log.gz 2024-02-01T12:09:15.609 INFO:teuthology.orchestra.run.smithi187.stderr: --verbose -- /var/log/ceph/c2-client.0.52970.log 2024-02-01T12:09:15.609 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.82606.log: 46.7% -- replaced with /var/log/ceph/c2-client.0.82606.log.gz 2024-02-01T12:09:15.609 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/rgw.c2.client.0.log 2024-02-01T12:09:15.609 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.52970.log: 45.5% -- replaced with /var/log/ceph/c2-client.0.52970.log.gz 2024-02-01T12:09:15.609 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.44063.log 2024-02-01T12:09:15.609 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/rgw.c2.client.0.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.112993.log 2024-02-01T12:09:15.610 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.44063.log: 47.4% -- replaced with /var/log/ceph/c2-client.0.44063.log.gz 2024-02-01T12:09:15.610 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.156732.log 2024-02-01T12:09:15.610 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.63048.log 2024-02-01T12:09:15.610 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.112993.log: /var/log/ceph/c2-client.0.156732.log: 59.4% 49.4% -- replaced with /var/log/ceph/c2-client.0.156732.log.gz -- replaced with /var/log/ceph/c2-client.0.112993.log.gz 2024-02-01T12:09:15.610 INFO:teuthology.orchestra.run.smithi187.stderr: 2024-02-01T12:09:15.610 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.93429.log 2024-02-01T12:09:15.611 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.63048.log: 53.5% -- replaced with /var/log/ceph/c2-client.0.63048.log.gz 2024-02-01T12:09:15.611 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.127174.log 2024-02-01T12:09:15.611 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.80247.log 2024-02-01T12:09:15.611 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.93429.log: 0.0% -- replaced with /var/log/ceph/c2-client.0.93429.log.gz 2024-02-01T12:09:15.611 INFO:teuthology.orchestra.run.smithi187.stderr:gzip/var/log/ceph/c2-client.0.127174.log: -5 --verbose -- /var/log/ceph/c2-client.0.141390.log 2024-02-01T12:09:15.611 INFO:teuthology.orchestra.run.smithi187.stderr: 47.4% -- replaced with /var/log/ceph/c2-client.0.127174.log.gz 2024-02-01T12:09:15.612 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.80247.log: 50.7% -- replaced with /var/log/ceph/c2-client.0.80247.log.gz 2024-02-01T12:09:15.612 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.52377.log 2024-02-01T12:09:15.612 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.141390.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.84377.log 2024-02-01T12:09:15.612 INFO:teuthology.orchestra.run.smithi187.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.141390.log.gz 2024-02-01T12:09:15.612 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.52377.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.105873.log 2024-02-01T12:09:15.612 INFO:teuthology.orchestra.run.smithi187.stderr: 47.4% -- replaced with /var/log/ceph/c2-client.0.52377.log.gz 2024-02-01T12:09:15.613 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.84377.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.125981.log 2024-02-01T12:09:15.613 INFO:teuthology.orchestra.run.smithi187.stderr: 50.1% -- replaced with /var/log/ceph/c2-client.0.84377.log.gz 2024-02-01T12:09:15.613 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.105873.log: 44.8% -- replaced with /var/log/ceph/c2-client.0.105873.log.gz 2024-02-01T12:09:15.613 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.50005.log 2024-02-01T12:09:15.613 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.101684.log 2024-02-01T12:09:15.613 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.125981.log: 47.4% -- replaced with /var/log/ceph/c2-client.0.125981.log.gz 2024-02-01T12:09:15.614 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.50005.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.55349.log 2024-02-01T12:09:15.614 INFO:teuthology.orchestra.run.smithi187.stderr: 47.4% -- replaced with /var/log/ceph/c2-client.0.50005.log.gz 2024-02-01T12:09:15.614 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.101684.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.99327.log 2024-02-01T12:09:15.614 INFO:teuthology.orchestra.run.smithi187.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.101684.log.gz 2024-02-01T12:09:15.614 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.55349.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.103449.log 2024-02-01T12:09:15.614 INFO:teuthology.orchestra.run.smithi187.stderr: 47.4% -- replaced with /var/log/ceph/c2-client.0.55349.log.gz 2024-02-01T12:09:15.615 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.99327.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.114834.log 2024-02-01T12:09:15.615 INFO:teuthology.orchestra.run.smithi187.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.99327.log.gz 2024-02-01T12:09:15.615 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.103449.log: 0.0% -- replaced with /var/log/ceph/c2-client.0.103449.log.gz 2024-02-01T12:09:15.615 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.90878.log 2024-02-01T12:09:15.615 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.114834.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.167373.log 2024-02-01T12:09:15.615 INFO:teuthology.orchestra.run.smithi187.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.114834.log.gz 2024-02-01T12:09:15.615 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.90878.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.81428.log 2024-02-01T12:09:15.616 INFO:teuthology.orchestra.run.smithi187.stderr: 47.4% -- replaced with /var/log/ceph/c2-client.0.90878.log.gz 2024-02-01T12:09:15.616 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.167373.log: 47.4% -- replaced with /var/log/ceph/c2-client.0.167373.log.gz 2024-02-01T12:09:15.616 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.71340.log 2024-02-01T12:09:15.616 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.81428.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.136667.log 2024-02-01T12:09:15.616 INFO:teuthology.orchestra.run.smithi187.stderr: 68.4% -- replaced with /var/log/ceph/c2-client.0.81428.log.gz 2024-02-01T12:09:15.616 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.71340.log: 0.0%gzip -- replaced with /var/log/ceph/c2-client.0.71340.log.gz 2024-02-01T12:09:15.617 INFO:teuthology.orchestra.run.smithi187.stderr: -5 --verbose -- /var/log/ceph/c2-client.0.100503.log 2024-02-01T12:09:15.617 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.136667.log: gzip -5 --verbose -- /var/log/ceph/c2-client.1.28369.log 2024-02-01T12:09:15.617 INFO:teuthology.orchestra.run.smithi187.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.136667.log.gz 2024-02-01T12:09:15.617 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.100503.log: gzip -5 0.0% --verbose -- replaced with /var/log/ceph/c2-client.0.100503.log.gz -- 2024-02-01T12:09:15.617 INFO:teuthology.orchestra.run.smithi187.stderr: /var/log/ceph/c2-client.admin.25847.log 2024-02-01T12:09:15.617 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.1.28369.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.140214.log 2024-02-01T12:09:15.617 INFO:teuthology.orchestra.run.smithi187.stderr: 43.9% -- replaced with /var/log/ceph/c2-client.1.28369.log.gz 2024-02-01T12:09:15.618 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.admin.25847.log: gzip -5 0.0% --verbose -- replaced with /var/log/ceph/c2-client.admin.25847.log.gz -- 2024-02-01T12:09:15.618 INFO:teuthology.orchestra.run.smithi187.stderr: /var/log/ceph/c2-client.0.166780.log 2024-02-01T12:09:15.618 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.140214.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.86736.log 2024-02-01T12:09:15.618 INFO:teuthology.orchestra.run.smithi187.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.140214.log.gz 2024-02-01T12:09:15.618 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.166780.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.147887.log 2024-02-01T12:09:15.618 INFO:teuthology.orchestra.run.smithi187.stderr: 47.4% -- replaced with /var/log/ceph/c2-client.0.166780.log.gz 2024-02-01T12:09:15.619 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.86736.log: 46.7% -- replaced with /var/log/ceph/c2-client.0.86736.log.gz 2024-02-01T12:09:15.619 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.admin.169256.log 2024-02-01T12:09:15.619 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.147887.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.37537.log 2024-02-01T12:09:15.619 INFO:teuthology.orchestra.run.smithi187.stderr: 69.2% -- replaced with /var/log/ceph/c2-client.0.147887.log.gz 2024-02-01T12:09:15.619 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.admin.169256.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.78468.log 2024-02-01T12:09:15.619 INFO:teuthology.orchestra.run.smithi187.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.169256.log.gz 2024-02-01T12:09:15.619 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.37537.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.117352.log 2024-02-01T12:09:15.620 INFO:teuthology.orchestra.run.smithi187.stderr: 47.4% -- replaced with /var/log/ceph/c2-client.0.37537.log.gz 2024-02-01T12:09:15.620 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.78468.log: 46.3% -- replaced with /var/log/ceph/c2-client.0.78468.log.gz 2024-02-01T12:09:15.620 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.admin.22522.log 2024-02-01T12:09:15.620 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.117352.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.42284.log 2024-02-01T12:09:15.620 INFO:teuthology.orchestra.run.smithi187.stderr: 44.8% -- replaced with /var/log/ceph/c2-client.0.117352.log.gz 2024-02-01T12:09:15.620 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.admin.22522.log: 0.0% -- replaced with /var/log/ceph/c2-client.admin.22522.log.gz 2024-02-01T12:09:15.621 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.150835.log 2024-02-01T12:09:15.621 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.42284.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.129549.log 2024-02-01T12:09:15.621 INFO:teuthology.orchestra.run.smithi187.stderr: 47.4% -- replaced with /var/log/ceph/c2-client.0.42284.log.gz 2024-02-01T12:09:15.621 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.150835.log: 0.0% -- replaced with /var/log/ceph/c2-client.0.150835.log.gz 2024-02-01T12:09:15.621 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.72519.log 2024-02-01T12:09:15.621 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.129549.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.102275.log 2024-02-01T12:09:15.622 INFO:teuthology.orchestra.run.smithi187.stderr: 47.4% -- replaced with /var/log/ceph/c2-client.0.129549.log.gz 2024-02-01T12:09:15.622 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.72519.log: 0.0% -- replaced with /var/log/ceph/c2-client.0.72519.log.gz 2024-02-01T12:09:15.622 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.76065.log 2024-02-01T12:09:15.622 INFO:teuthology.orchestra.run.smithi187.stderr:gzip/var/log/ceph/c2-client.0.102275.log: -5 --verbose -- /var/log/ceph/c2-client.0.117945.log 2024-02-01T12:09:15.622 INFO:teuthology.orchestra.run.smithi187.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.102275.log.gz 2024-02-01T12:09:15.622 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.76065.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.133107.log 2024-02-01T12:09:15.623 INFO:teuthology.orchestra.run.smithi187.stderr: 94.8% -- replaced with /var/log/ceph/c2-client.0.76065.log.gz 2024-02-01T12:09:15.623 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.117945.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.77874.log 2024-02-01T12:09:15.623 INFO:teuthology.orchestra.run.smithi187.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.117945.log.gz 2024-02-01T12:09:15.623 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.133107.log: 47.4% -- replaced with /var/log/ceph/c2-client.0.133107.log.gz 2024-02-01T12:09:15.623 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.98731.log 2024-02-01T12:09:15.623 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.77874.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.26597.log 2024-02-01T12:09:15.623 INFO:teuthology.orchestra.run.smithi187.stderr: 80.5% -- replaced with /var/log/ceph/c2-client.0.77874.log.gz 2024-02-01T12:09:15.624 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.98731.log: 0.0% -- replaced with /var/log/ceph/c2-client.0.98731.log.gz 2024-02-01T12:09:15.624 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.58904.log 2024-02-01T12:09:15.624 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.26597.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.139622.log 2024-02-01T12:09:15.624 INFO:teuthology.orchestra.run.smithi187.stderr: 84.2% -- replaced with /var/log/ceph/c2-client.0.26597.log.gz 2024-02-01T12:09:15.624 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.58904.log: 47.4% -- replaced with /var/log/ceph/c2-client.0.58904.log.gz 2024-02-01T12:09:15.624 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.41691.log 2024-02-01T12:09:15.624 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.admin.25947.log 2024-02-01T12:09:15.625 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.139622.log: 0.0% -- replaced with /var/log/ceph/c2-client.0.139622.log.gz 2024-02-01T12:09:15.625 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.41691.log: gzip -5 --verbose -- 47.4% /var/log/ceph/c2-client.admin.26256.log 2024-02-01T12:09:15.625 INFO:teuthology.orchestra.run.smithi187.stderr: -- replaced with /var/log/ceph/c2-client.0.41691.log.gz 2024-02-01T12:09:15.625 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.admin.25947.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.36351.log 2024-02-01T12:09:15.625 INFO:teuthology.orchestra.run.smithi187.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.25947.log.gz 2024-02-01T12:09:15.625 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.admin.26256.log: 0.0% -- replaced with /var/log/ceph/c2-client.admin.26256.log.gz 2024-02-01T12:09:15.626 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.124201.log 2024-02-01T12:09:15.626 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.36351.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.146118.log 2024-02-01T12:09:15.626 INFO:teuthology.orchestra.run.smithi187.stderr: 47.4% -- replaced with /var/log/ceph/c2-client.0.36351.log.gz 2024-02-01T12:09:15.626 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.124201.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.130142.log 2024-02-01T12:09:15.626 INFO:teuthology.orchestra.run.smithi187.stderr: 47.4% -- replaced with /var/log/ceph/c2-client.0.124201.log.gz 2024-02-01T12:09:15.626 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.146118.log: gzip -5 --verbose 57.2% -- -- replaced with /var/log/ceph/c2-client.0.146118.log.gz /var/log/ceph/c2-client.0.97545.log 2024-02-01T12:09:15.626 INFO:teuthology.orchestra.run.smithi187.stderr: 2024-02-01T12:09:15.627 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.130142.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.166187.log 2024-02-01T12:09:15.627 INFO:teuthology.orchestra.run.smithi187.stderr: 47.4% -- replaced with /var/log/ceph/c2-client.0.130142.log.gz 2024-02-01T12:09:15.627 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.97545.log: 47.4% -- replaced with /var/log/ceph/c2-client.0.97545.log.gz 2024-02-01T12:09:15.627 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.125387.log 2024-02-01T12:09:15.627 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.166187.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.25902.log 2024-02-01T12:09:15.627 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.125387.log: 92.7% -- replaced with /var/log/ceph/c2-client.0.166187.log.gz 2024-02-01T12:09:15.628 INFO:teuthology.orchestra.run.smithi187.stderr: 47.4%gzip -- replaced with /var/log/ceph/c2-client.0.125387.log.gz 2024-02-01T12:09:15.628 INFO:teuthology.orchestra.run.smithi187.stderr: -5 --verbose -- /var/log/ceph/ops-log-c2-client.1.log 2024-02-01T12:09:15.628 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.143169.log 2024-02-01T12:09:15.628 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.admin.25902.log: 0.0% -- replaced with /var/log/ceph/c2-client.admin.25902.log.gz 2024-02-01T12:09:15.628 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ops-log-c2-client.1.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.161460.log 2024-02-01T12:09:15.628 INFO:teuthology.orchestra.run.smithi187.stderr: 35.9% -- replaced with /var/log/ceph/ops-log-c2-client.1.log.gz 2024-02-01T12:09:15.628 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.143169.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.165001.log 2024-02-01T12:09:15.629 INFO:teuthology.orchestra.run.smithi187.stderr: 57.9% -- replaced with /var/log/ceph/c2-client.0.143169.log.gz 2024-02-01T12:09:15.629 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.161460.log: 70.3% -- replaced with /var/log/ceph/c2-client.0.161460.log.gz 2024-02-01T12:09:15.629 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.70751.log 2024-02-01T12:09:15.629 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.165001.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.152601.log 2024-02-01T12:09:15.629 INFO:teuthology.orchestra.run.smithi187.stderr: 45.5% -- replaced with /var/log/ceph/c2-client.0.165001.log.gz 2024-02-01T12:09:15.629 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.70751.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.139030.log 2024-02-01T12:09:15.629 INFO:teuthology.orchestra.run.smithi187.stderr: 51.3% -- replaced with /var/log/ceph/c2-client.0.70751.log.gz 2024-02-01T12:09:15.630 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.152601.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.69548.log 2024-02-01T12:09:15.630 INFO:teuthology.orchestra.run.smithi187.stderr: 47.4% -- replaced with /var/log/ceph/c2-client.0.152601.log.gz 2024-02-01T12:09:15.630 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.139030.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.84966.log 2024-02-01T12:09:15.630 INFO:teuthology.orchestra.run.smithi187.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.139030.log.gz 2024-02-01T12:09:15.630 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.69548.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.26428.log 2024-02-01T12:09:15.630 INFO:teuthology.orchestra.run.smithi187.stderr: 47.1% -- replaced with /var/log/ceph/c2-client.0.69548.log.gz 2024-02-01T12:09:15.631 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.84966.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.159103.log 2024-02-01T12:09:15.631 INFO:teuthology.orchestra.run.smithi187.stderr: 46.3% -- replaced with /var/log/ceph/c2-client.0.84966.log.gz 2024-02-01T12:09:15.631 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.admin.26428.log: gzip -5 --verbose -- /var/log/ceph/c2.log 2024-02-01T12:09:15.631 INFO:teuthology.orchestra.run.smithi187.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.26428.log.gz 2024-02-01T12:09:15.631 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.159103.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.95192.log 2024-02-01T12:09:15.631 INFO:teuthology.orchestra.run.smithi187.stderr: 70.4% -- replaced with /var/log/ceph/c2-client.0.159103.log.gz 2024-02-01T12:09:15.632 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.111807.log 2024-02-01T12:09:15.632 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.95192.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.87325.log 2024-02-01T12:09:15.632 INFO:teuthology.orchestra.run.smithi187.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.95192.log.gz 2024-02-01T12:09:15.632 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.89106.log 2024-02-01T12:09:15.632 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.111807.log: 47.4% -- replaced with /var/log/ceph/c2-client.0.111807.log.gz 2024-02-01T12:09:15.632 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.87325.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.149065.log 2024-02-01T12:09:15.633 INFO:teuthology.orchestra.run.smithi187.stderr: 46.7% -- replaced with /var/log/ceph/c2-client.0.87325.log.gz 2024-02-01T12:09:15.633 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.89106.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.33973.log 2024-02-01T12:09:15.633 INFO:teuthology.orchestra.run.smithi187.stderr: 44.8% -- replaced with /var/log/ceph/c2-client.0.89106.log.gz 2024-02-01T12:09:15.633 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.149065.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.135481.log 2024-02-01T12:09:15.633 INFO:teuthology.orchestra.run.smithi187.stderr: 69.1% -- replaced with /var/log/ceph/c2-client.0.149065.log.gz 2024-02-01T12:09:15.633 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.33973.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.48819.log 2024-02-01T12:09:15.634 INFO:teuthology.orchestra.run.smithi187.stderr: 47.4% -- replaced with /var/log/ceph/c2-client.0.33973.log.gz 2024-02-01T12:09:15.634 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.135481.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.26251.log 2024-02-01T12:09:15.634 INFO:teuthology.orchestra.run.smithi187.stderr: 47.4% -- replaced with /var/log/ceph/c2-client.0.135481.log.gz 2024-02-01T12:09:15.634 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.48819.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.65410.log 2024-02-01T12:09:15.634 INFO:teuthology.orchestra.run.smithi187.stderr: 47.4% -- replaced with /var/log/ceph/c2-client.0.48819.log.gz 2024-02-01T12:09:15.634 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.admin.26251.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.167968.log 2024-02-01T12:09:15.634 INFO:teuthology.orchestra.run.smithi187.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.26251.log.gz 2024-02-01T12:09:15.635 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.94016.log 2024-02-01T12:09:15.635 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.65410.log: /var/log/ceph/c2-client.0.167968.log: 81.0% -- replaced with /var/log/ceph/c2-client.0.65410.log.gz 2024-02-01T12:09:15.635 INFO:teuthology.orchestra.run.smithi187.stderr: 45.5%gzip -- replaced with /var/log/ceph/c2-client.0.167968.log.gz -5 2024-02-01T12:09:15.635 INFO:teuthology.orchestra.run.smithi187.stderr: --verbose -- /var/log/ceph/c2-client.0.108839.log 2024-02-01T12:09:15.635 INFO:teuthology.orchestra.run.smithi187.stderr: 88.5% -- replaced with /var/log/ceph/c2.log.gz 2024-02-01T12:09:15.635 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.64815.log 2024-02-01T12:09:15.635 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.94016.log: 0.0% -- replaced with /var/log/ceph/c2-client.0.94016.log.gz 2024-02-01T12:09:15.636 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.108839.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.26326.log 2024-02-01T12:09:15.636 INFO:teuthology.orchestra.run.smithi187.stderr: 47.4% -- replaced with /var/log/ceph/c2-client.0.108839.log.gz 2024-02-01T12:09:15.636 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.64815.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.157321.log 2024-02-01T12:09:15.636 INFO:teuthology.orchestra.run.smithi187.stderr: 80.8% -- replaced with /var/log/ceph/c2-client.0.64815.log.gz 2024-02-01T12:09:15.636 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.admin.26326.log: gzip -5 --verbose -- 0.0% /var/log/ceph/c2-client.0.107652.log 2024-02-01T12:09:15.636 INFO:teuthology.orchestra.run.smithi187.stderr: -- replaced with /var/log/ceph/c2-client.admin.26326.log.gz 2024-02-01T12:09:15.637 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.157321.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.99916.log 2024-02-01T12:09:15.637 INFO:teuthology.orchestra.run.smithi187.stderr: 47.4% -- replaced with /var/log/ceph/c2-client.0.157321.log.gz 2024-02-01T12:09:15.637 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.107652.log: 45.5% -- replaced with /var/log/ceph/c2-client.0.107652.log.gz 2024-02-01T12:09:15.637 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.40505.log 2024-02-01T12:09:15.637 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.99916.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.63637.log 2024-02-01T12:09:15.637 INFO:teuthology.orchestra.run.smithi187.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.99916.log.gz 2024-02-01T12:09:15.637 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.40505.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.163228.log 2024-02-01T12:09:15.638 INFO:teuthology.orchestra.run.smithi187.stderr: 47.4% -- replaced with /var/log/ceph/c2-client.0.40505.log.gz 2024-02-01T12:09:15.638 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.63637.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.38723.log 2024-02-01T12:09:15.638 INFO:teuthology.orchestra.run.smithi187.stderr: 53.0% -- replaced with /var/log/ceph/c2-client.0.63637.log.gz 2024-02-01T12:09:15.638 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.163228.log: 0.0%gzip -- replaced with /var/log/ceph/c2-client.0.163228.log.gz 2024-02-01T12:09:15.638 INFO:teuthology.orchestra.run.smithi187.stderr: -5 --verbose -- /var/log/ceph/c2-client.0.127767.log 2024-02-01T12:09:15.638 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.38723.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.49412.log 2024-02-01T12:09:15.639 INFO:teuthology.orchestra.run.smithi187.stderr: 44.8% -- replaced with /var/log/ceph/c2-client.0.38723.log.gz 2024-02-01T12:09:15.639 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.127767.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.89699.log 2024-02-01T12:09:15.639 INFO:teuthology.orchestra.run.smithi187.stderr: 47.4% -- replaced with /var/log/ceph/c2-client.0.127767.log.gz 2024-02-01T12:09:15.639 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.49412.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.131921.log 2024-02-01T12:09:15.639 INFO:teuthology.orchestra.run.smithi187.stderr: 45.5% -- replaced with /var/log/ceph/c2-client.0.49412.log.gz 2024-02-01T12:09:15.639 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.89699.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.36944.log 2024-02-01T12:09:15.640 INFO:teuthology.orchestra.run.smithi187.stderr: 47.4% -- replaced with /var/log/ceph/c2-client.0.89699.log.gz 2024-02-01T12:09:15.640 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.131921.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.116098.log 2024-02-01T12:09:15.640 INFO:teuthology.orchestra.run.smithi187.stderr: 47.4% -- replaced with /var/log/ceph/c2-client.0.131921.log.gz 2024-02-01T12:09:15.640 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.36944.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.158508.log 2024-02-01T12:09:15.640 INFO:teuthology.orchestra.run.smithi187.stderr: 47.4% -- replaced with /var/log/ceph/c2-client.0.36944.log.gz 2024-02-01T12:09:15.640 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.116098.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.26021.log 2024-02-01T12:09:15.641 INFO:teuthology.orchestra.run.smithi187.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.116098.log.gz 2024-02-01T12:09:15.641 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.158508.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.149654.log 2024-02-01T12:09:15.641 INFO:teuthology.orchestra.run.smithi187.stderr: 73.7% -- replaced with /var/log/ceph/c2-client.0.158508.log.gz 2024-02-01T12:09:15.641 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.admin.26021.log: 0.0%gzip -- replaced with /var/log/ceph/c2-client.admin.26021.log.gz 2024-02-01T12:09:15.641 INFO:teuthology.orchestra.run.smithi187.stderr: -5 --verbose -- /var/log/ceph/c2-client.0.120642.log 2024-02-01T12:09:15.641 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.149654.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.96371.log 2024-02-01T12:09:15.642 INFO:teuthology.orchestra.run.smithi187.stderr: 69.9% -- replaced with /var/log/ceph/c2-client.0.149654.log.gz 2024-02-01T12:09:15.642 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.120642.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.124794.log 2024-02-01T12:09:15.642 INFO:teuthology.orchestra.run.smithi187.stderr: 44.8% -- replaced with /var/log/ceph/c2-client.0.120642.log.gz 2024-02-01T12:09:15.642 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.96371.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.96958.log 2024-02-01T12:09:15.642 INFO:teuthology.orchestra.run.smithi187.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.96371.log.gz 2024-02-01T12:09:15.642 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.124794.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.32050.log 2024-02-01T12:09:15.642 INFO:teuthology.orchestra.run.smithi187.stderr: 47.4% -- replaced with /var/log/ceph/c2-client.0.124794.log.gz 2024-02-01T12:09:15.643 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.96958.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.66591.log 2024-02-01T12:09:15.643 INFO:teuthology.orchestra.run.smithi187.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.96958.log.gz 2024-02-01T12:09:15.643 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.32050.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.51784.log 2024-02-01T12:09:15.643 INFO:teuthology.orchestra.run.smithi187.stderr: 19.1% -- replaced with /var/log/ceph/c2-client.0.32050.log.gz 2024-02-01T12:09:15.643 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.66591.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.25875.log 2024-02-01T12:09:15.643 INFO:teuthology.orchestra.run.smithi187.stderr: 53.5% -- replaced with /var/log/ceph/c2-client.0.66591.log.gz 2024-02-01T12:09:15.644 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.51784.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.31417.log 2024-02-01T12:09:15.644 INFO:teuthology.orchestra.run.smithi187.stderr: 47.4% -- replaced with /var/log/ceph/c2-client.0.51784.log.gz 2024-02-01T12:09:15.644 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.admin.25875.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.26352.log 2024-02-01T12:09:15.644 INFO:teuthology.orchestra.run.smithi187.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.25875.log.gz 2024-02-01T12:09:15.644 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.admin.31417.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.54758.log 2024-02-01T12:09:15.644 INFO:teuthology.orchestra.run.smithi187.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.31417.log.gz 2024-02-01T12:09:15.645 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.admin.26352.log: gzip 0.0% -- replaced with /var/log/ceph/c2-client.admin.26352.log.gz -5 2024-02-01T12:09:15.645 INFO:teuthology.orchestra.run.smithi187.stderr: --verbose -- /var/log/ceph/c2-client.0.83784.log 2024-02-01T12:09:15.645 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.54758.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.131328.log 2024-02-01T12:09:15.645 INFO:teuthology.orchestra.run.smithi187.stderr: 56.9% -- replaced with /var/log/ceph/c2-client.0.54758.log.gz 2024-02-01T12:09:15.645 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.83784.log: gzip -5 --verbose -- /var/log/ceph/rgw.c2.client.1.log 2024-02-01T12:09:15.645 INFO:teuthology.orchestra.run.smithi187.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.83784.log.gz 2024-02-01T12:09:15.646 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.131328.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.151422.log 2024-02-01T12:09:15.646 INFO:teuthology.orchestra.run.smithi187.stderr: 47.4% -- replaced with /var/log/ceph/c2-client.0.131328.log.gz 2024-02-01T12:09:15.646 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/rgw.c2.client.1.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.77281.log 2024-02-01T12:09:15.646 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.151422.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.66002.log 2024-02-01T12:09:15.646 INFO:teuthology.orchestra.run.smithi187.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.151422.log.gz 2024-02-01T12:09:15.646 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.77281.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.22428.log 2024-02-01T12:09:15.646 INFO:teuthology.orchestra.run.smithi187.stderr: 47.4% -- replaced with /var/log/ceph/c2-client.0.77281.log.gz 2024-02-01T12:09:15.647 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.66002.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.64226.log 2024-02-01T12:09:15.647 INFO:teuthology.orchestra.run.smithi187.stderr: 53.3% -- replaced with /var/log/ceph/c2-client.0.66002.log.gz 2024-02-01T12:09:15.647 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.admin.22428.log: gzip 0.0% -5 -- replaced with /var/log/ceph/c2-client.admin.22428.log.gz --verbose 2024-02-01T12:09:15.647 INFO:teuthology.orchestra.run.smithi187.stderr: -- /var/log/ceph/c2-client.0.121828.log 2024-02-01T12:09:15.647 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.144351.log 2024-02-01T12:09:15.647 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.64226.log: 53.5% -- replaced with /var/log/ceph/c2-client.0.64226.log.gz 2024-02-01T12:09:15.648 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.121828.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.61870.log 2024-02-01T12:09:15.648 INFO:teuthology.orchestra.run.smithi187.stderr: 45.5% -- replaced with /var/log/ceph/c2-client.0.121828.log.gz 2024-02-01T12:09:15.648 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.144351.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.26020.log 2024-02-01T12:09:15.648 INFO:teuthology.orchestra.run.smithi187.stderr: 57.4% -- replaced with /var/log/ceph/c2-client.0.144351.log.gz 2024-02-01T12:09:15.648 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.61870.log: gzip 93.7% -5 --verbose -- -- replaced with /var/log/ceph/rgw.c2.client.1.log.gz /var/log/ceph/c2-client.0.150243.log 2024-02-01T12:09:15.648 INFO:teuthology.orchestra.run.smithi187.stderr: 2024-02-01T12:09:15.649 INFO:teuthology.orchestra.run.smithi187.stderr: 53.6% -- replaced with /var/log/ceph/c2-client.0.61870.log.gz 2024-02-01T12:09:15.649 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.admin.26020.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.47626.log 2024-02-01T12:09:15.649 INFO:teuthology.orchestra.run.smithi187.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.26020.log.gz 2024-02-01T12:09:15.649 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.150243.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.148476.log 2024-02-01T12:09:15.649 INFO:teuthology.orchestra.run.smithi187.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.150243.log.gz 2024-02-01T12:09:15.649 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.47626.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.68953.log 2024-02-01T12:09:15.650 INFO:teuthology.orchestra.run.smithi187.stderr: 47.4% -- replaced with /var/log/ceph/c2-client.0.47626.log.gz 2024-02-01T12:09:15.650 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.148476.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.133700.log 2024-02-01T12:09:15.650 INFO:teuthology.orchestra.run.smithi187.stderr: 57.5% -- replaced with /var/log/ceph/c2-client.0.148476.log.gz 2024-02-01T12:09:15.650 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.68953.log: gzip -5 --verbose -- /var/log/ceph/ops-log-c2-client.0.log 2024-02-01T12:09:15.650 INFO:teuthology.orchestra.run.smithi187.stderr: 68.1% -- replaced with /var/log/ceph/c2-client.0.68953.log.gz 2024-02-01T12:09:15.650 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.133700.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.57126.log 2024-02-01T12:09:15.651 INFO:teuthology.orchestra.run.smithi187.stderr: 47.4% -- replaced with /var/log/ceph/c2-client.0.133700.log.gz 2024-02-01T12:09:15.651 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ops-log-c2-client.0.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.107059.log 2024-02-01T12:09:15.651 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.57126.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.102862.log 2024-02-01T12:09:15.651 INFO:teuthology.orchestra.run.smithi187.stderr: 47.4% -- replaced with /var/log/ceph/c2-client.0.57126.log.gz 2024-02-01T12:09:15.651 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.73698.log 2024-02-01T12:09:15.651 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.107059.log: 47.4% -- replaced with /var/log/ceph/c2-client.0.107059.log.gz 2024-02-01T12:09:15.651 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.102862.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.160871.log 2024-02-01T12:09:15.652 INFO:teuthology.orchestra.run.smithi187.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.102862.log.gz 2024-02-01T12:09:15.652 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.73698.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.88513.log 2024-02-01T12:09:15.652 INFO:teuthology.orchestra.run.smithi187.stderr: 45.5% -- replaced with /var/log/ceph/c2-client.0.73698.log.gz 2024-02-01T12:09:15.652 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.160871.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.126574.log 2024-02-01T12:09:15.652 INFO:teuthology.orchestra.run.smithi187.stderr: 70.3% -- replaced with /var/log/ceph/c2-client.0.160871.log.gz 2024-02-01T12:09:15.652 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.88513.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.39319.log 2024-02-01T12:09:15.653 INFO:teuthology.orchestra.run.smithi187.stderr: 47.4% -- replaced with /var/log/ceph/c2-client.0.88513.log.gz 2024-02-01T12:09:15.653 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose/var/log/ceph/c2-client.0.126574.log: -- /var/log/ceph/c2-client.0.137254.log 2024-02-01T12:09:15.653 INFO:teuthology.orchestra.run.smithi187.stderr: 47.4% -- replaced with /var/log/ceph/c2-client.0.126574.log.gz 2024-02-01T12:09:15.653 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.39319.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.138438.log 2024-02-01T12:09:15.653 INFO:teuthology.orchestra.run.smithi187.stderr: 47.4% -- replaced with /var/log/ceph/c2-client.0.39319.log.gz 2024-02-01T12:09:15.653 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.137254.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.60090.log 2024-02-01T12:09:15.654 INFO:teuthology.orchestra.run.smithi187.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.137254.log.gz 2024-02-01T12:09:15.654 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.138438.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.45249.log 2024-02-01T12:09:15.654 INFO:teuthology.orchestra.run.smithi187.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.138438.log.gz 2024-02-01T12:09:15.654 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.60090.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.155554.log 2024-02-01T12:09:15.654 INFO:teuthology.orchestra.run.smithi187.stderr: 57.8% -- replaced with /var/log/ceph/c2-client.0.60090.log.gz 2024-02-01T12:09:15.654 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.45249.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.168561.log 2024-02-01T12:09:15.655 INFO:teuthology.orchestra.run.smithi187.stderr: 47.4% -- replaced with /var/log/ceph/c2-client.0.45249.log.gz 2024-02-01T12:09:15.655 INFO:teuthology.orchestra.run.smithi187.stderr:gzip/var/log/ceph/c2-client.0.155554.log: -5 --verbose -- /var/log/ceph/c2-client.admin.26252.log 2024-02-01T12:09:15.655 INFO:teuthology.orchestra.run.smithi187.stderr: 57.8% -- replaced with /var/log/ceph/c2-client.0.155554.log.gz 2024-02-01T12:09:15.655 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.168561.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.85555.log 2024-02-01T12:09:15.655 INFO:teuthology.orchestra.run.smithi187.stderr: 47.4% -- replaced with /var/log/ceph/c2-client.0.168561.log.gz 2024-02-01T12:09:15.655 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2.audit.log 2024-02-01T12:09:15.656 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.admin.26252.log: 0.0% -- replaced with /var/log/ceph/c2-client.admin.26252.log.gz 2024-02-01T12:09:15.656 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.85555.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.79060.log 2024-02-01T12:09:15.656 INFO:teuthology.orchestra.run.smithi187.stderr: 47.1% -- replaced with /var/log/ceph/c2-client.0.85555.log.gz 2024-02-01T12:09:15.656 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2.audit.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.26178.log 2024-02-01T12:09:15.656 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.79060.log: gzip -5 --verbose -- /var/log/ceph/c2-mgr.x.log 2024-02-01T12:09:15.656 INFO:teuthology.orchestra.run.smithi187.stderr: 47.4% -- replaced with /var/log/ceph/c2-client.0.79060.log.gz 2024-02-01T12:09:15.656 INFO:teuthology.orchestra.run.smithi187.stderr: 89.2% -- replaced with /var/log/ceph/c2.audit.log.gz 2024-02-01T12:09:15.657 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.43470.log 2024-02-01T12:09:15.657 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.admin.26178.log: 0.0% -- replaced with /var/log/ceph/c2-client.admin.26178.log.gz 2024-02-01T12:09:15.657 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.144940.log 2024-02-01T12:09:15.657 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.105209.log 2024-02-01T12:09:15.657 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.43470.log: 47.4% -- replaced with /var/log/ceph/c2-client.0.43470.log.gz 2024-02-01T12:09:15.657 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.144940.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.110028.log 2024-02-01T12:09:15.658 INFO:teuthology.orchestra.run.smithi187.stderr: 57.8% -- replaced with /var/log/ceph/c2-client.0.144940.log.gz 2024-02-01T12:09:15.658 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.105209.log: 0.0% -- replaced with /var/log/ceph/c2-client.0.105209.log.gz 2024-02-01T12:09:15.658 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.admin.25988.log 2024-02-01T12:09:15.658 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.94603.log 2024-02-01T12:09:15.658 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.110028.log: 47.4% -- replaced with /var/log/ceph/c2-client.0.110028.log.gz/var/log/ceph/c2-client.admin.25988.log: 2024-02-01T12:09:15.658 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.76692.log 2024-02-01T12:09:15.658 INFO:teuthology.orchestra.run.smithi187.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.25988.log.gz 2024-02-01T12:09:15.659 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.94603.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.22570.log 2024-02-01T12:09:15.659 INFO:teuthology.orchestra.run.smithi187.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.94603.log.gz 2024-02-01T12:09:15.659 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.76692.log: 46.3% -- replaced with /var/log/ceph/c2-client.0.76692.log.gz 2024-02-01T12:09:15.659 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.87920.log 2024-02-01T12:09:15.659 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.41098.log 2024-02-01T12:09:15.659 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.admin.22570.log: 0.0% -- replaced with /var/log/ceph/c2-client.admin.22570.log.gz/var/log/ceph/c2-client.0.87920.log: 2024-02-01T12:09:15.659 INFO:teuthology.orchestra.run.smithi187.stderr:gzip 45.5% -- replaced with /var/log/ceph/c2-client.0.87920.log.gz -5 2024-02-01T12:09:15.660 INFO:teuthology.orchestra.run.smithi187.stderr: --verbose -- /var/log/ceph/c2-client.0.95784.log 2024-02-01T12:09:15.660 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.41098.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.92835.log 2024-02-01T12:09:15.660 INFO:teuthology.orchestra.run.smithi187.stderr: 47.4% -- replaced with /var/log/ceph/c2-client.0.41098.log.gz 2024-02-01T12:09:15.660 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.95784.log: gzip -5 --verbose -- /var/log/ceph/c2-mon.a.log 2024-02-01T12:09:15.660 INFO:teuthology.orchestra.run.smithi187.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.95784.log.gz 2024-02-01T12:09:15.661 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.74881.log 2024-02-01T12:09:15.661 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.92835.log: 45.5% -- replaced with /var/log/ceph/c2-client.0.92835.log.gz/var/log/ceph/c2-mon.a.log: 2024-02-01T12:09:15.661 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.31466.log 2024-02-01T12:09:15.661 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.74881.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.153787.log 2024-02-01T12:09:15.661 INFO:teuthology.orchestra.run.smithi187.stderr: 50.1% -- replaced with /var/log/ceph/c2-client.0.74881.log.gz 2024-02-01T12:09:15.661 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.31466.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/c2-client.0.31466.log.gz /var/log/ceph/c2-client.0.114170.log 2024-02-01T12:09:15.662 INFO:teuthology.orchestra.run.smithi187.stderr: 2024-02-01T12:09:15.662 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.153787.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.68360.log 2024-02-01T12:09:15.662 INFO:teuthology.orchestra.run.smithi187.stderr: 58.2% -- replaced with /var/log/ceph/c2-client.0.153787.log.gz 2024-02-01T12:09:15.662 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.114170.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.54156.log 2024-02-01T12:09:15.662 INFO:teuthology.orchestra.run.smithi187.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.114170.log.gz 2024-02-01T12:09:15.662 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.68360.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.22546.log 2024-02-01T12:09:15.662 INFO:teuthology.orchestra.run.smithi187.stderr: 47.4% -- replaced with /var/log/ceph/c2-client.0.68360.log.gz 2024-02-01T12:09:15.663 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.54156.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.146707.log 2024-02-01T12:09:15.663 INFO:teuthology.orchestra.run.smithi187.stderr: 62.7% -- replaced with /var/log/ceph/c2-client.0.54156.log.gz 2024-02-01T12:09:15.663 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.admin.22546.log: 0.0% -- replaced with /var/log/ceph/c2-client.admin.22546.log.gz 2024-02-01T12:09:15.663 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.110621.log 2024-02-01T12:09:15.663 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.146707.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.113586.log 2024-02-01T12:09:15.663 INFO:teuthology.orchestra.run.smithi187.stderr: 76.5% -- replaced with /var/log/ceph/c2-client.0.146707.log.gz 2024-02-01T12:09:15.664 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.110621.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.46435.log 2024-02-01T12:09:15.664 INFO:teuthology.orchestra.run.smithi187.stderr: 44.8% -- replaced with /var/log/ceph/c2-client.0.110621.log.gz 2024-02-01T12:09:15.664 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.113586.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.92242.log 2024-02-01T12:09:15.664 INFO:teuthology.orchestra.run.smithi187.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.113586.log.gz 2024-02-01T12:09:15.664 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.46435.log: 47.4% -- replaced with /var/log/ceph/c2-client.0.46435.log.gz 2024-02-01T12:09:15.665 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.163815.log 2024-02-01T12:09:15.665 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.92242.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.111214.log 2024-02-01T12:09:15.665 INFO:teuthology.orchestra.run.smithi187.stderr: 47.4% -- replaced with /var/log/ceph/c2-client.0.92242.log.gz 2024-02-01T12:09:15.665 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.163815.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.70137.log 2024-02-01T12:09:15.665 INFO:teuthology.orchestra.run.smithi187.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.163815.log.gz 2024-02-01T12:09:15.666 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.111214.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.47033.log 2024-02-01T12:09:15.666 INFO:teuthology.orchestra.run.smithi187.stderr: 44.8% -- replaced with /var/log/ceph/c2-client.0.111214.log.gz 2024-02-01T12:09:15.666 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.70137.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.74292.log 2024-02-01T12:09:15.666 INFO:teuthology.orchestra.run.smithi187.stderr: 92.7% -- replaced with /var/log/ceph/c2-client.0.70137.log.gz 2024-02-01T12:09:15.666 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.47033.log: 47.4% -- replaced with /var/log/ceph/c2-client.0.47033.log.gz 2024-02-01T12:09:15.666 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.admin.25822.log 2024-02-01T12:09:15.666 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.83195.log 2024-02-01T12:09:15.667 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.74292.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.165594.log 2024-02-01T12:09:15.667 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.admin.25822.log: 0.0% 57.4% -- replaced with /var/log/ceph/c2-client.0.74292.log.gz 2024-02-01T12:09:15.667 INFO:teuthology.orchestra.run.smithi187.stderr: -- replaced with /var/log/ceph/c2-client.admin.25822.log.gz 2024-02-01T12:09:15.667 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.83195.log: 46.3% -- replaced with /var/log/ceph/c2-client.0.83195.log.gz 2024-02-01T12:09:15.667 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.48226.log 2024-02-01T12:09:15.667 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.42877.log 2024-02-01T12:09:15.667 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.165594.log: /var/log/ceph/c2-client.0.48226.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.79654.log 2024-02-01T12:09:15.668 INFO:teuthology.orchestra.run.smithi187.stderr: 47.4% -- replaced with /var/log/ceph/c2-client.0.165594.log.gz 2024-02-01T12:09:15.668 INFO:teuthology.orchestra.run.smithi187.stderr: 47.4% -- replaced with /var/log/ceph/c2-client.0.48226.log.gz/var/log/ceph/c2-client.0.42877.log: 2024-02-01T12:09:15.668 INFO:teuthology.orchestra.run.smithi187.stderr: 47.4% -- replaced with /var/log/ceph/c2-client.0.42877.log.gzgzip 2024-02-01T12:09:15.668 INFO:teuthology.orchestra.run.smithi187.stderr: -5 --verbose -- /var/log/ceph/c2-osd.2.log 2024-02-01T12:09:15.668 INFO:teuthology.orchestra.run.smithi187.stderr: 94.5% -- replaced with /var/log/ceph/ops-log-c2-client.0.log.gz 2024-02-01T12:09:15.668 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.123014.log 2024-02-01T12:09:15.669 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.79654.log: 45.5% -- replaced with /var/log/ceph/c2-client.0.79654.log.gz 2024-02-01T12:09:15.669 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-osd.2.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.134888.log 2024-02-01T12:09:15.669 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.153194.log 2024-02-01T12:09:15.669 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.123014.log: 48.7% -- replaced with /var/log/ceph/c2-client.0.123014.log.gz 2024-02-01T12:09:15.669 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.134888.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.156143.log 2024-02-01T12:09:15.669 INFO:teuthology.orchestra.run.smithi187.stderr: 47.4% -- replaced with /var/log/ceph/c2-client.0.134888.log.gz 2024-02-01T12:09:15.670 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.153194.log: gzip 47.4% -5 --verbose -- replaced with /var/log/ceph/c2-client.0.153194.log.gz -- 2024-02-01T12:09:15.670 INFO:teuthology.orchestra.run.smithi187.stderr: /var/log/ceph/c2-client.0.143762.log 2024-02-01T12:09:15.670 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.156143.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.56535.log 2024-02-01T12:09:15.670 INFO:teuthology.orchestra.run.smithi187.stderr: 57.4% -- replaced with /var/log/ceph/c2-client.0.156143.log.gz 2024-02-01T12:09:15.670 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.143762.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.147298.log 2024-02-01T12:09:15.670 INFO:teuthology.orchestra.run.smithi187.stderr: 56.7% -- replaced with /var/log/ceph/c2-client.0.143762.log.gz 2024-02-01T12:09:15.671 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.44656.log 2024-02-01T12:09:15.671 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.56535.log: /var/log/ceph/c2-client.0.147298.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.130735.log 2024-02-01T12:09:15.671 INFO:teuthology.orchestra.run.smithi187.stderr: 69.8% -- replaced with /var/log/ceph/c2-client.0.147298.log.gz 2024-02-01T12:09:15.671 INFO:teuthology.orchestra.run.smithi187.stderr: 58.3% -- replaced with /var/log/ceph/c2-client.0.56535.log.gz 2024-02-01T12:09:15.671 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.44656.log: 45.5% -- replaced with /var/log/ceph/c2-client.0.44656.log.gz 2024-02-01T12:09:15.671 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.154376.log 2024-02-01T12:09:15.672 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.130735.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.71932.log 2024-02-01T12:09:15.672 INFO:teuthology.orchestra.run.smithi187.stderr: 47.4% -- replaced with /var/log/ceph/c2-client.0.130735.log.gz 2024-02-01T12:09:15.672 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.154376.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.67769.log 2024-02-01T12:09:15.672 INFO:teuthology.orchestra.run.smithi187.stderr: 57.6% -- replaced with /var/log/ceph/c2-client.0.154376.log.gz 2024-02-01T12:09:15.672 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.71932.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.60683.log 2024-02-01T12:09:15.672 INFO:teuthology.orchestra.run.smithi187.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.71932.log.gz 2024-02-01T12:09:15.673 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.67769.log: 52.9% -- replaced with /var/log/ceph/c2-client.0.67769.log.gz 2024-02-01T12:09:15.673 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.1.27779.log 2024-02-01T12:09:15.673 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.60683.log: gzip -5 --verbose -- /var/log/ceph/c2-osd.0.log 2024-02-01T12:09:15.673 INFO:teuthology.orchestra.run.smithi187.stderr: 47.1% -- replaced with /var/log/ceph/c2-client.0.60683.log.gz 2024-02-01T12:09:15.673 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.1.27779.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.30247.log 2024-02-01T12:09:15.673 INFO:teuthology.orchestra.run.smithi187.stderr: 83.0% -- replaced with /var/log/ceph/c2-client.1.27779.log.gz 2024-02-01T12:09:15.674 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-osd.0.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.108245.log 2024-02-01T12:09:15.674 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.58312.log 2024-02-01T12:09:15.674 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.30247.log: /var/log/ceph/c2-client.0.108245.log: 0.0% -- replaced with /var/log/ceph/c2-client.0.30247.log.gz 2024-02-01T12:09:15.674 INFO:teuthology.orchestra.run.smithi187.stderr: 47.4% -- replaced with /var/log/ceph/c2-client.0.108245.log.gz 2024-02-01T12:09:15.674 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.98138.log 2024-02-01T12:09:15.674 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.162049.log 2024-02-01T12:09:15.675 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.58312.log: /var/log/ceph/c2-client.0.98138.log: 47.0% -- replaced with /var/log/ceph/c2-client.0.58312.log.gz 2024-02-01T12:09:15.675 INFO:teuthology.orchestra.run.smithi187.stderr: 45.5% -- replaced with /var/log/ceph/c2-client.0.98138.log.gz 2024-02-01T12:09:15.675 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.140803.log 2024-02-01T12:09:15.675 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.122421.log 2024-02-01T12:09:15.675 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.162049.log: /var/log/ceph/c2-client.0.140803.log: 0.0% -- replaced with /var/log/ceph/c2-client.0.162049.log.gz 2024-02-01T12:09:15.675 INFO:teuthology.orchestra.run.smithi187.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.140803.log.gz 2024-02-01T12:09:15.676 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.admin.26478.log 2024-02-01T12:09:15.676 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.67180.log 2024-02-01T12:09:15.676 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.122421.log: /var/log/ceph/c2-client.admin.26478.log: 47.4% -- replaced with /var/log/ceph/c2-client.0.122421.log.gz 2024-02-01T12:09:15.676 INFO:teuthology.orchestra.run.smithi187.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.26478.log.gz 2024-02-01T12:09:15.676 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.123608.log 2024-02-01T12:09:15.676 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.53563.log 2024-02-01T12:09:15.677 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.67180.log: /var/log/ceph/c2-client.0.123608.log: 53.0% -- replaced with /var/log/ceph/c2-client.0.67180.log.gz 2024-02-01T12:09:15.677 INFO:teuthology.orchestra.run.smithi187.stderr: 47.4% -- replaced with /var/log/ceph/c2-client.0.123608.log.gz 2024-02-01T12:09:15.677 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.51191.log 2024-02-01T12:09:15.677 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.33380.log 2024-02-01T12:09:15.677 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.53563.log: /var/log/ceph/c2-client.0.51191.log: 47.4% -- replaced with /var/log/ceph/c2-client.0.53563.log.gz 2024-02-01T12:09:15.677 INFO:teuthology.orchestra.run.smithi187.stderr: 47.4% -- replaced with /var/log/ceph/c2-client.0.51191.log.gz 2024-02-01T12:09:15.677 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.82017.log 2024-02-01T12:09:15.678 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.73106.log 2024-02-01T12:09:15.678 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.33380.log: /var/log/ceph/c2-client.0.82017.log: 47.4% -- replaced with /var/log/ceph/c2-client.0.33380.log.gz 2024-02-01T12:09:15.678 INFO:teuthology.orchestra.run.smithi187.stderr: 46.7% -- replaced with /var/log/ceph/c2-client.0.82017.log.gz 2024-02-01T12:09:15.678 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.104625.log 2024-02-01T12:09:15.678 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.57715.log 2024-02-01T12:09:15.678 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.73106.log: /var/log/ceph/c2-client.0.104625.log: 0.0% -- replaced with /var/log/ceph/c2-client.0.73106.log.gz 2024-02-01T12:09:15.678 INFO:teuthology.orchestra.run.smithi187.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.104625.log.gz 2024-02-01T12:09:15.679 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.admin.26177.log 2024-02-01T12:09:15.679 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.admin.26100.log 2024-02-01T12:09:15.679 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.57715.log: /var/log/ceph/c2-client.admin.26177.log: 87.1% -- replaced with /var/log/ceph/c2-client.0.57715.log.gz 2024-02-01T12:09:15.679 INFO:teuthology.orchestra.run.smithi187.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.26177.log.gz 2024-02-01T12:09:15.679 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.admin.22667.log 2024-02-01T12:09:15.679 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.admin.26092.log 2024-02-01T12:09:15.679 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.admin.26100.log: 0.0% -- replaced with /var/log/ceph/c2-client.admin.26100.log.gz 2024-02-01T12:09:15.680 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.admin.22667.log: 0.0%gzip -- replaced with /var/log/ceph/c2-client.admin.22667.log.gz -5 --verbose -- /var/log/ceph/c2-client.admin.26378.log 2024-02-01T12:09:15.680 INFO:teuthology.orchestra.run.smithi187.stderr: 2024-02-01T12:09:15.680 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.admin.26092.log: 0.0% -- replaced with /var/log/ceph/c2-client.admin.26092.log.gz 2024-02-01T12:09:15.680 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.admin.26502.log 2024-02-01T12:09:15.680 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.admin.22605.log 2024-02-01T12:09:15.680 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.admin.26378.log: 0.0% -- replaced with /var/log/ceph/c2-client.admin.26378.log.gz 2024-02-01T12:09:15.681 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.admin.26502.log: 0.0% -- replaced with /var/log/ceph/c2-client.admin.26502.log.gzgzip -5 --verbose -- /var/log/ceph/c2-client.0.116682.log 2024-02-01T12:09:15.681 INFO:teuthology.orchestra.run.smithi187.stderr: 2024-02-01T12:09:15.681 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.admin.22605.log: 0.0% -- replaced with /var/log/ceph/c2-client.admin.22605.log.gz 2024-02-01T12:09:15.681 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.119206.log 2024-02-01T12:09:15.681 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.160281.log 2024-02-01T12:09:15.681 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.116682.log: /var/log/ceph/c2-client.0.119206.log: 0.0% -- replaced with /var/log/ceph/c2-client.0.116682.log.gz 2024-02-01T12:09:15.682 INFO:teuthology.orchestra.run.smithi187.stderr: 30.2% -- replaced with /var/log/ceph/c2-client.0.119206.log.gz 2024-02-01T12:09:15.682 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.101097.log 2024-02-01T12:09:15.682 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.154965.log 2024-02-01T12:09:15.682 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.160281.log: /var/log/ceph/c2-client.0.101097.log: 69.6% -- replaced with /var/log/ceph/c2-client.0.160281.log.gz 2024-02-01T12:09:15.682 INFO:teuthology.orchestra.run.smithi187.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.101097.log.gz 2024-02-01T12:09:15.682 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c2-client.0.154965.log: 62.7% -- replaced with /var/log/ceph/c2-client.0.154965.log.gz 2024-02-01T12:09:15.683 INFO:teuthology.orchestra.run.smithi148.stderr: 91.7% -- replaced with /var/log/ceph/c1-mgr.x.log.gz 2024-02-01T12:09:15.752 INFO:teuthology.orchestra.run.smithi187.stderr: 91.5% -- replaced with /var/log/ceph/c2-mgr.x.log.gz 2024-02-01T12:09:16.363 INFO:teuthology.orchestra.run.smithi148.stderr: 93.6% -- replaced with /var/log/ceph/rgw.c1.client.1.log.gz 2024-02-01T12:09:16.586 INFO:teuthology.orchestra.run.smithi187.stderr: 93.7% -- replaced with /var/log/ceph/rgw.c2.client.0.log.gz 2024-02-01T12:09:16.790 INFO:teuthology.orchestra.run.smithi187.stderr: 91.5% -- replaced with /var/log/ceph/c2-mon.a.log.gz 2024-02-01T12:09:16.817 INFO:teuthology.orchestra.run.smithi148.stderr: 92.2% -- replaced with /var/log/ceph/c1-mon.a.log.gz 2024-02-01T12:09:19.133 INFO:teuthology.orchestra.run.smithi148.stderr: 96.2% -- replaced with /var/log/ceph/rgw.c1.client.0.log.gz 2024-02-01T12:09:25.083 INFO:teuthology.orchestra.run.smithi187.stderr: 94.0% -- replaced with /var/log/ceph/c2-osd.1.log.gz 2024-02-01T12:09:25.323 INFO:teuthology.orchestra.run.smithi187.stderr: 94.0% -- replaced with /var/log/ceph/c2-osd.2.log.gz 2024-02-01T12:09:25.901 INFO:teuthology.orchestra.run.smithi187.stderr: 93.9% -- replaced with /var/log/ceph/c2-osd.0.log.gz 2024-02-01T12:09:25.902 INFO:teuthology.orchestra.run.smithi187.stderr: 2024-02-01T12:09:25.903 INFO:teuthology.orchestra.run.smithi187.stderr:real 0m10.689s 2024-02-01T12:09:25.903 INFO:teuthology.orchestra.run.smithi187.stderr:user 0m32.119s 2024-02-01T12:09:25.903 INFO:teuthology.orchestra.run.smithi187.stderr:sys 0m1.338s 2024-02-01T12:09:29.773 INFO:teuthology.orchestra.run.smithi148.stderr: 94.0% -- replaced with /var/log/ceph/c1-osd.2.log.gz 2024-02-01T12:09:29.867 INFO:teuthology.orchestra.run.smithi148.stderr: 93.9% -- replaced with /var/log/ceph/c1-osd.1.log.gz 2024-02-01T12:09:30.940 INFO:teuthology.orchestra.run.smithi148.stderr: 93.9% -- replaced with /var/log/ceph/c1-osd.0.log.gz 2024-02-01T12:09:30.941 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-02-01T12:09:30.941 INFO:teuthology.orchestra.run.smithi148.stderr:real 0m15.740s 2024-02-01T12:09:30.941 INFO:teuthology.orchestra.run.smithi148.stderr:user 0m50.075s 2024-02-01T12:09:30.942 INFO:teuthology.orchestra.run.smithi148.stderr:sys 0m1.860s 2024-02-01T12:09:30.943 INFO:tasks.ceph:Archiving logs... 2024-02-01T12:09:30.943 DEBUG:teuthology.misc:Transferring archived files from smithi148:/var/log/ceph to /home/teuthworker/archive/yuvalif-2024-01-30_08:46:48-rgw-wip-end2end-tracing-distro-default-smithi/7537587/remote/smithi148/log 2024-02-01T12:09:30.944 DEBUG:teuthology.orchestra.run.smithi148:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-01T12:09:37.255 DEBUG:teuthology.misc:Transferring archived files from smithi187:/var/log/ceph to /home/teuthworker/archive/yuvalif-2024-01-30_08:46:48-rgw-wip-end2end-tracing-distro-default-smithi/7537587/remote/smithi187/log 2024-02-01T12:09:37.256 DEBUG:teuthology.orchestra.run.smithi187:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-01T12:09:41.243 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/tasks/ceph.py", line 1918, in task File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/tasks/rgw.py", line 473, in task File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/tasks/rgw.py", line 300, in create_pools File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/tasks/rgw.py", line 242, in start_rgw File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/tasks/util/rgw.py", line 34, in rgwadmin File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi148 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 gc process --include-all' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/tasks/ceph.py", line 1928, in task File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/tasks/ceph_manager.py", line 2774, in wait_for_clean File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/tasks/ceph_manager.py", line 2606, in get_num_active_clean File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/tasks/ceph_manager.py", line 2372, in get_pg_stats File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/tasks/ceph_manager.py", line 1627, in raw_cluster_cmd File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/tasks/ceph_manager.py", line 1618, in run_cluster_cmd File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi148 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json' 2024-02-01T12:09:41.247 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-02-01T12:09:41.258 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/tasks/rgw.py", line 473, in task File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/tasks/rgw.py", line 300, in create_pools File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/tasks/rgw.py", line 242, in start_rgw File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/tasks/util/rgw.py", line 34, in rgwadmin File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi148 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 gc process --include-all' 2024-02-01T12:09:41.268 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-02-01T12:09:41.268 DEBUG:teuthology.orchestra.run.smithi148:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-01T12:09:41.274 DEBUG:teuthology.orchestra.run.smithi187:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-01T12:09:41.342 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-02-01T12:09:41.351 INFO:teuthology.task.clock:Checking final clock skew... 2024-02-01T12:09:41.352 DEBUG:teuthology.orchestra.run.smithi148:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-01T12:09:41.355 DEBUG:teuthology.orchestra.run.smithi187:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-01T12:09:41.368 INFO:teuthology.orchestra.run.smithi148.stdout: remote refid st t when poll reach delay offset jitter 2024-02-01T12:09:41.368 INFO:teuthology.orchestra.run.smithi148.stdout:============================================================================== 2024-02-01T12:09:41.368 INFO:teuthology.orchestra.run.smithi148.stdout:*hv01.front.sepi 44.4.53.2 2 u 80 256 377 0.190 -1.499 1.083 2024-02-01T12:09:41.369 INFO:teuthology.orchestra.run.smithi148.stdout:+hv02.front.sepi 63.231.80.2 3 u 230 256 377 0.141 -2.480 1.435 2024-02-01T12:09:41.369 INFO:teuthology.orchestra.run.smithi148.stdout:+hv03.front.sepi 74.6.168.72 3 u 222 256 377 0.187 -5.031 1.954 2024-02-01T12:09:41.369 INFO:teuthology.orchestra.run.smithi148.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-02-01T12:09:41.372 INFO:teuthology.orchestra.run.smithi187.stdout: remote refid st t when poll reach delay offset jitter 2024-02-01T12:09:41.372 INFO:teuthology.orchestra.run.smithi187.stdout:============================================================================== 2024-02-01T12:09:41.372 INFO:teuthology.orchestra.run.smithi187.stdout:*hv01.front.sepi 44.4.53.2 2 u 35 256 377 0.220 +1.854 0.299 2024-02-01T12:09:41.373 INFO:teuthology.orchestra.run.smithi187.stdout:+hv02.front.sepi 63.231.80.2 3 u 95 256 377 0.219 -0.200 0.127 2024-02-01T12:09:41.373 INFO:teuthology.orchestra.run.smithi187.stdout:+hv03.front.sepi 74.6.168.72 3 u 60 256 377 0.150 -3.552 0.360 2024-02-01T12:09:41.373 INFO:teuthology.orchestra.run.smithi187.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-02-01T12:09:41.374 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-02-01T12:09:41.384 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-02-01T12:09:41.384 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-02-01T12:09:41.394 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-02-01T12:09:41.403 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-02-01T12:09:41.412 INFO:teuthology.task.internal:Duration was 7590.812471 seconds 2024-02-01T12:09:41.412 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-02-01T12:09:41.422 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-02-01T12:09:41.422 DEBUG:teuthology.orchestra.run.smithi148:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-01T12:09:41.425 DEBUG:teuthology.orchestra.run.smithi187:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-01T12:09:41.462 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-02-01T12:09:41.462 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi148.front.sepia.ceph.com 2024-02-01T12:09:41.463 DEBUG:teuthology.orchestra.run.smithi148:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-01T12:09:41.510 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi187.front.sepia.ceph.com 2024-02-01T12:09:41.511 DEBUG:teuthology.orchestra.run.smithi187:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-01T12:09:41.520 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-02-01T12:09:41.521 DEBUG:teuthology.orchestra.run.smithi148:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-01T12:09:41.555 DEBUG:teuthology.orchestra.run.smithi187:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-01T12:09:41.589 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-02-01T12:09:41.589 DEBUG:teuthology.orchestra.run.smithi148:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-01T12:09:41.623 DEBUG:teuthology.orchestra.run.smithi187:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-01T12:09:41.686 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-02-01T12:09:41.697 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-02-01T12:09:41.698 DEBUG:teuthology.orchestra.run.smithi148:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-01T12:09:41.712 DEBUG:teuthology.orchestra.run.smithi187:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-01T12:09:41.722 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-02-01T12:09:41.733 DEBUG:teuthology.orchestra.run.smithi148:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-01T12:09:41.755 DEBUG:teuthology.orchestra.run.smithi187:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-01T12:09:41.764 INFO:teuthology.orchestra.run.smithi148.stdout:kernel.core_pattern = core 2024-02-01T12:09:41.773 INFO:teuthology.orchestra.run.smithi187.stdout:kernel.core_pattern = core 2024-02-01T12:09:41.881 DEBUG:teuthology.orchestra.run.smithi148:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-01T12:09:41.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-01T12:09:41.930 DEBUG:teuthology.orchestra.run.smithi187:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-01T12:09:41.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-01T12:09:41.935 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-02-01T12:09:41.947 INFO:teuthology.task.internal:Transferring archived files... 2024-02-01T12:09:41.948 DEBUG:teuthology.misc:Transferring archived files from smithi148:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuvalif-2024-01-30_08:46:48-rgw-wip-end2end-tracing-distro-default-smithi/7537587/remote/smithi148 2024-02-01T12:09:41.948 DEBUG:teuthology.orchestra.run.smithi148:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-01T12:09:41.993 DEBUG:teuthology.misc:Transferring archived files from smithi187:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuvalif-2024-01-30_08:46:48-rgw-wip-end2end-tracing-distro-default-smithi/7537587/remote/smithi187 2024-02-01T12:09:41.993 DEBUG:teuthology.orchestra.run.smithi187:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-01T12:09:42.015 INFO:teuthology.task.internal:Removing archive directory... 2024-02-01T12:09:42.015 DEBUG:teuthology.orchestra.run.smithi148:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-01T12:09:42.031 DEBUG:teuthology.orchestra.run.smithi187:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-01T12:09:42.057 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-02-01T12:09:42.071 INFO:teuthology.task.internal:Not uploading archives. 2024-02-01T12:09:42.071 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-02-01T12:09:42.144 INFO:teuthology.task.internal:Tidying up after the test... 2024-02-01T12:09:42.144 DEBUG:teuthology.orchestra.run.smithi148:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-01T12:09:42.147 DEBUG:teuthology.orchestra.run.smithi187:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-01T12:09:42.150 INFO:teuthology.orchestra.run.smithi148.stdout: 655472 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 1 12:09 /home/ubuntu/cephtest 2024-02-01T12:09:42.150 INFO:teuthology.orchestra.run.smithi148.stdout: 674572 0 srwxr-xr-x 1 root root 0 Feb 1 10:37 /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock 2024-02-01T12:09:42.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-01T12:09:42.152 INFO:teuthology.orchestra.run.smithi148.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-02-01T12:09:42.152 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/tasks/rgw.py", line 473, in task File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/tasks/rgw.py", line 300, in create_pools File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/tasks/rgw.py", line 242, in start_rgw File "/home/teuthworker/src/git.ceph.com_ceph-c_5a7faa111fd8a5e84a4602ed7dfe97ac5e4e3d24/qa/tasks/util/rgw.py", line 34, in rgwadmin File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi148 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 gc process --include-all' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi148 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-02-01T12:09:42.164 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-02-01T12:09:42.176 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-02-01T12:09:42.187 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi148 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 gc process --include-all' 2024-02-01T12:09:42.187 INFO:teuthology.nuke:Checking targets against current locks 2024-02-01T12:09:42.194 INFO:teuthology.orchestra.run.smithi187.stdout: 655472 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 1 12:09 /home/ubuntu/cephtest 2024-02-01T12:09:42.194 INFO:teuthology.orchestra.run.smithi187.stdout: 674396 0 srwxr-xr-x 1 root root 0 Feb 1 10:13 /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock 2024-02-01T12:09:42.195 INFO:teuthology.orchestra.run.smithi187.stdout: 674395 0 srwxr-xr-x 1 root root 0 Feb 1 10:13 /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock 2024-02-01T12:09:42.195 INFO:teuthology.orchestra.run.smithi187.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-02-01T12:09:42.218 DEBUG:teuthology.nuke:shortname: smithi148 2024-02-01T12:09:42.218 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-01T12:09:42.241 DEBUG:teuthology.nuke:shortname: smithi187 2024-02-01T12:09:42.241 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-01T12:09:42.248 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi148.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuvalif-2024-01-30_08:46:48-rgw-wip-end2end-tracing-distro-default-smithi/7537587', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-01 09:54:39.089625', 'locked_by': 'scheduled_yuvalif@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBH5ce3/FCp+PGNOXXdX8il8f4jkJaHpplftAcDQSFX3tHjR+D85kVrtsDWCNIGnihkTWiK/q2ovxDNYhkN4nd0g='} 2024-02-01T12:09:42.261 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi187.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuvalif-2024-01-30_08:46:48-rgw-wip-end2end-tracing-distro-default-smithi/7537587', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-01 09:54:39.088225', 'locked_by': 'scheduled_yuvalif@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLmrn1ce5MSq4PQkPQy/SrK5OHVanVF/Lx8xwqrWintcWDpX5+8boxWTRV5B7sthtEqb55YGOwOBChsT+yODK3Y='} 2024-02-01T12:09:42.299 INFO:teuthology.orchestra.console.smithi148:Power off 2024-02-01T12:09:42.299 DEBUG:teuthology.orchestra.console.smithi148:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-01T12:09:42.317 INFO:teuthology.orchestra.console.smithi187:Power off 2024-02-01T12:09:42.317 DEBUG:teuthology.orchestra.console.smithi187:pexpect command: ipmitool -H smithi187.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-01T12:09:42.331 DEBUG:teuthology.orchestra.console.smithi148:power off output: Chassis Power Control: Down/Off 2024-02-01T12:09:42.332 DEBUG:teuthology.orchestra.console.smithi148:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-01T12:09:42.348 DEBUG:teuthology.orchestra.console.smithi187:power off output: Chassis Power Control: Down/Off 2024-02-01T12:09:42.349 DEBUG:teuthology.orchestra.console.smithi187:pexpect command: ipmitool -H smithi187.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-01T12:09:42.361 DEBUG:teuthology.orchestra.console.smithi148:check power output: Chassis Power is on 2024-02-01T12:09:42.374 DEBUG:teuthology.orchestra.console.smithi187:check power output: Chassis Power is on 2024-02-01T12:09:46.363 DEBUG:teuthology.orchestra.console.smithi148:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-01T12:09:46.374 DEBUG:teuthology.orchestra.console.smithi187:pexpect command: ipmitool -H smithi187.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-01T12:09:46.476 DEBUG:teuthology.orchestra.console.smithi148:check power output: Chassis Power is on 2024-02-01T12:09:46.486 DEBUG:teuthology.orchestra.console.smithi187:check power output: Chassis Power is on 2024-02-01T12:09:50.482 DEBUG:teuthology.orchestra.console.smithi148:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-01T12:09:50.491 DEBUG:teuthology.orchestra.console.smithi187:pexpect command: ipmitool -H smithi187.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-01T12:09:50.593 DEBUG:teuthology.orchestra.console.smithi148:check power output: Chassis Power is off 2024-02-01T12:09:50.605 DEBUG:teuthology.orchestra.console.smithi187:check power output: Chassis Power is on 2024-02-01T12:09:50.695 INFO:teuthology.orchestra.console.smithi148:Power off completed 2024-02-01T12:09:54.608 DEBUG:teuthology.orchestra.console.smithi187:pexpect command: ipmitool -H smithi187.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-01T12:09:54.719 DEBUG:teuthology.orchestra.console.smithi187:check power output: Chassis Power is off 2024-02-01T12:09:54.820 INFO:teuthology.orchestra.console.smithi187:Power off completed 2024-02-01T12:09:54.924 INFO:teuthology.run:Summary data: description: rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zones supported-random-distro$/{ubuntu_latest} tasks/test_multi} duration: 7590.812470674515 failure_reason: rgw multisite test failures flavor: default owner: scheduled_yuvalif@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=c5305a7f2be442c6bc6819c93ab98551 status: fail success: false 2024-02-01T12:09:54.925 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-01T12:09:55.004 INFO:teuthology.run:FAIL